2024-03-04T09:11:53.895 INFO:root:teuthology version: 0.0.1.dev260+gb1dac55 2024-03-04T09:11:53.895 DEBUG:teuthology.run:Teuthology command: teuthology --description fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/strays} --name leonidus-2024-03-04_07:05:03-fs-wip-lusov-qdb-distro-default-smithi --archive /home/teuthworker/archive/leonidus-2024-03-04_07:05:03-fs-wip-lusov-qdb-distro-default-smithi/7580856 --verbose --owner scheduled_leonidus@teuthology -- /home/teuthworker/archive/leonidus-2024-03-04_07:05:03-fs-wip-lusov-qdb-distro-default-smithi/7580856/orig.config.yaml 2024-03-04T09:11:53.945 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-04T09:11:54.063 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/leonidus-2024-03-04_07:05:03-fs-wip-lusov-qdb-distro-default-smithi/7580856 branch: wip-lusov-qdb description: fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/strays} email: null first_in_suite: false job_id: '7580856' kernel: client: branch: testing kdb: true sha1: distro ktype: upstream last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: leonidus-2024-03-04_07:05:03-fs-wip-lusov-qdb-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-lusov-qdb ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 pid file: '' rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding sha1: 1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd kclient: syntax: v2 subvolume_version: 2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-lusov-qdb sha1: 1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd owner: scheduled_leonidus@teuthology priority: 80 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - client.2 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - client.3 seed: 5072 sha1: 1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd sleep_before_teardown: 0 subset: 246/10000 suite: fs suite_branch: wip-lusov-qdb suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi006.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBC5g9n3VlEpyNSiCHwg9pTGH2AJTQjN0eGNKVhnZzlrs/IkxoO07GCYbY1l3lSh6BcN6HkmG4HgB/CfyXrWW3Vs= smithi146.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHFfcb7jz8+ewcln2HSzgDFE300DWYZ0fvE9UQNbHDxKP7HaNMFdFaAGEb1IL6TJrNBpNkhlR8i/gVXYccJEAwU= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - kclient: null - cephfs_test_runner: modules: - tasks.cephfs.test_strays teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: b1dac5519c57c269ea98a22fb7729016a1d0e4d2 timestamp: 2024-03-04_07:05:03 tube: smithi user: leonidus verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-04T09:11:54.064 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/qa; will attempt to use it 2024-03-04T09:11:54.065 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/qa/tasks 2024-03-04T09:11:54.065 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-04T09:11:54.066 INFO:teuthology.task.internal:Checking packages... 2024-03-04T09:11:54.094 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd' 2024-03-04T09:11:54.094 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-04T09:11:54.094 INFO:teuthology.packaging:ref: None 2024-03-04T09:11:54.094 INFO:teuthology.packaging:tag: None 2024-03-04T09:11:54.094 INFO:teuthology.packaging:branch: wip-lusov-qdb 2024-03-04T09:11:54.094 INFO:teuthology.packaging:sha1: 1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd 2024-03-04T09:11:54.094 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-lusov-qdb 2024-03-04T09:11:54.355 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1739.g1bc250c7 2024-03-04T09:11:54.357 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-04T09:11:54.363 INFO:teuthology.task.internal:no buildpackages task found 2024-03-04T09:11:54.363 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-04T09:11:54.375 INFO:teuthology.task.internal:Saving configuration 2024-03-04T09:11:54.390 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-04T09:11:54.396 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-04T09:11:54.429 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi006.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-03-04_07:05:03-fs-wip-lusov-qdb-distro-default-smithi/7580856', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-04 09:06:35.906898', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBC5g9n3VlEpyNSiCHwg9pTGH2AJTQjN0eGNKVhnZzlrs/IkxoO07GCYbY1l3lSh6BcN6HkmG4HgB/CfyXrWW3Vs='} 2024-03-04T09:11:54.461 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi146.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-03-04_07:05:03-fs-wip-lusov-qdb-distro-default-smithi/7580856', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-04 09:06:35.904958', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHFfcb7jz8+ewcln2HSzgDFE300DWYZ0fvE9UQNbHDxKP7HaNMFdFaAGEb1IL6TJrNBpNkhlR8i/gVXYccJEAwU='} 2024-03-04T09:11:54.461 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-04T09:11:54.467 INFO:teuthology.task.internal:roles: ubuntu@smithi006.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2024-03-04T09:11:54.467 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2024-03-04T09:11:54.467 INFO:teuthology.run_tasks:Running task console_log... 2024-03-04T09:11:54.556 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f3f82549f70>, signals=[15]) 2024-03-04T09:11:54.556 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-04T09:11:54.563 INFO:teuthology.task.internal:Opening connections... 2024-03-04T09:11:54.563 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi006.front.sepia.ceph.com 2024-03-04T09:11:54.564 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi006.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T09:11:54.677 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi146.front.sepia.ceph.com 2024-03-04T09:11:54.678 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T09:11:54.781 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-04T09:11:54.787 DEBUG:teuthology.orchestra.run.smithi006:> uname -m 2024-03-04T09:11:54.805 INFO:teuthology.orchestra.run.smithi006.stdout:x86_64 2024-03-04T09:11:54.805 DEBUG:teuthology.orchestra.run.smithi006:> cat /etc/os-release 2024-03-04T09:11:54.863 INFO:teuthology.orchestra.run.smithi006.stdout:NAME="CentOS Stream" 2024-03-04T09:11:54.863 INFO:teuthology.orchestra.run.smithi006.stdout:VERSION="9" 2024-03-04T09:11:54.863 INFO:teuthology.orchestra.run.smithi006.stdout:ID="centos" 2024-03-04T09:11:54.863 INFO:teuthology.orchestra.run.smithi006.stdout:ID_LIKE="rhel fedora" 2024-03-04T09:11:54.863 INFO:teuthology.orchestra.run.smithi006.stdout:VERSION_ID="9" 2024-03-04T09:11:54.863 INFO:teuthology.orchestra.run.smithi006.stdout:PLATFORM_ID="platform:el9" 2024-03-04T09:11:54.863 INFO:teuthology.orchestra.run.smithi006.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-04T09:11:54.863 INFO:teuthology.orchestra.run.smithi006.stdout:ANSI_COLOR="0;31" 2024-03-04T09:11:54.863 INFO:teuthology.orchestra.run.smithi006.stdout:LOGO="fedora-logo-icon" 2024-03-04T09:11:54.863 INFO:teuthology.orchestra.run.smithi006.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-04T09:11:54.864 INFO:teuthology.orchestra.run.smithi006.stdout:HOME_URL="https://centos.org/" 2024-03-04T09:11:54.864 INFO:teuthology.orchestra.run.smithi006.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-04T09:11:54.864 INFO:teuthology.orchestra.run.smithi006.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-04T09:11:54.864 INFO:teuthology.orchestra.run.smithi006.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-04T09:11:54.865 INFO:teuthology.lock.ops:Updating smithi006.front.sepia.ceph.com on lock server 2024-03-04T09:11:54.895 DEBUG:teuthology.orchestra.run.smithi146:> uname -m 2024-03-04T09:11:54.912 INFO:teuthology.orchestra.run.smithi146.stdout:x86_64 2024-03-04T09:11:54.913 DEBUG:teuthology.orchestra.run.smithi146:> cat /etc/os-release 2024-03-04T09:11:54.973 INFO:teuthology.orchestra.run.smithi146.stdout:NAME="CentOS Stream" 2024-03-04T09:11:54.973 INFO:teuthology.orchestra.run.smithi146.stdout:VERSION="9" 2024-03-04T09:11:54.973 INFO:teuthology.orchestra.run.smithi146.stdout:ID="centos" 2024-03-04T09:11:54.973 INFO:teuthology.orchestra.run.smithi146.stdout:ID_LIKE="rhel fedora" 2024-03-04T09:11:54.973 INFO:teuthology.orchestra.run.smithi146.stdout:VERSION_ID="9" 2024-03-04T09:11:54.973 INFO:teuthology.orchestra.run.smithi146.stdout:PLATFORM_ID="platform:el9" 2024-03-04T09:11:54.973 INFO:teuthology.orchestra.run.smithi146.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-04T09:11:54.973 INFO:teuthology.orchestra.run.smithi146.stdout:ANSI_COLOR="0;31" 2024-03-04T09:11:54.973 INFO:teuthology.orchestra.run.smithi146.stdout:LOGO="fedora-logo-icon" 2024-03-04T09:11:54.973 INFO:teuthology.orchestra.run.smithi146.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-04T09:11:54.973 INFO:teuthology.orchestra.run.smithi146.stdout:HOME_URL="https://centos.org/" 2024-03-04T09:11:54.974 INFO:teuthology.orchestra.run.smithi146.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-04T09:11:54.974 INFO:teuthology.orchestra.run.smithi146.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-04T09:11:54.974 INFO:teuthology.orchestra.run.smithi146.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-04T09:11:54.974 INFO:teuthology.lock.ops:Updating smithi146.front.sepia.ceph.com on lock server 2024-03-04T09:11:55.003 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-04T09:11:55.012 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-04T09:11:55.059 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-04T09:11:55.059 DEBUG:teuthology.orchestra.run.smithi006:> test '!' -e /home/ubuntu/cephtest 2024-03-04T09:11:55.062 DEBUG:teuthology.orchestra.run.smithi146:> test '!' -e /home/ubuntu/cephtest 2024-03-04T09:11:55.078 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-04T09:11:55.083 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-04T09:11:55.084 DEBUG:teuthology.orchestra.run.smithi006:> test -z $(ls -A /var/lib/ceph) 2024-03-04T09:11:55.119 DEBUG:teuthology.orchestra.run.smithi146:> test -z $(ls -A /var/lib/ceph) 2024-03-04T09:11:55.209 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-04T09:11:55.279 INFO:teuthology.run_tasks:Running task kernel... 2024-03-04T09:11:55.293 INFO:teuthology.task.kernel:normalize config orig: {'client': {'branch': 'testing'}, 'kdb': True, 'sha1': 'distro'} 2024-03-04T09:11:55.293 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'branch': 'testing'}, 'client.2': {'branch': 'testing'}, 'client.1': {'branch': 'testing'}, 'client.3': {'branch': 'testing'}} 2024-03-04T09:11:55.294 INFO:teuthology.task.kernel:config {'client.0': {'branch': 'testing'}, 'client.1': {'branch': 'testing'}}, timeout 300 2024-03-04T09:11:55.294 DEBUG:teuthology.orchestra.run.smithi006:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-04T09:11:55.294 DEBUG:teuthology.orchestra.run.smithi146:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-04T09:11:55.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T09:11:55.311 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&ref=testing 2024-03-04T09:11:55.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T09:11:55.323 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&ref=testing 2024-03-04T09:11:55.562 DEBUG:teuthology.task.kernel:sha1 for client.0 is 786939134d7eee4dc7a805245d3f595929af37d7 2024-03-04T09:11:55.562 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "786939134d7eee4dc7a805245d3f595929af37d7"... 2024-03-04T09:11:55.562 DEBUG:teuthology.orchestra.run.smithi006:> uname -r 2024-03-04T09:11:55.581 INFO:teuthology.orchestra.run.smithi006.stdout:5.14.0-361.el9.x86_64 2024-03-04T09:11:55.581 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-361.el9.x86_64 vs 786939134d7eee4dc7a805245d3f595929af37d7 2024-03-04T09:11:55.582 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-03-04T09:11:55.583 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum upgrade -y linux-firmware 2024-03-04T09:11:55.652 DEBUG:teuthology.task.kernel:sha1 for client.1 is 786939134d7eee4dc7a805245d3f595929af37d7 2024-03-04T09:11:55.652 INFO:teuthology.task.kernel:Checking kernel version of client.1, want "786939134d7eee4dc7a805245d3f595929af37d7"... 2024-03-04T09:11:55.653 DEBUG:teuthology.orchestra.run.smithi146:> uname -r 2024-03-04T09:11:55.670 INFO:teuthology.orchestra.run.smithi146.stdout:5.14.0-361.el9.x86_64 2024-03-04T09:11:55.670 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-361.el9.x86_64 vs 786939134d7eee4dc7a805245d3f595929af37d7 2024-03-04T09:11:55.670 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-03-04T09:11:55.671 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum upgrade -y linux-firmware 2024-03-04T09:11:57.809 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-04T09:11:57.860 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-04T09:11:59.219 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS-9 - BaseOS 7.8 MB/s | 2.0 MB 00:00 2024-03-04T09:11:59.243 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS-9 - BaseOS 7.4 MB/s | 2.0 MB 00:00 2024-03-04T09:12:02.684 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS Stream 9 - BaseOS 2.6 MB/s | 8.0 MB 00:03 2024-03-04T09:12:04.935 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS Stream 9 - AppStream 17 MB/s | 19 MB 00:01 2024-03-04T09:12:09.018 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS Stream 9 - CRB 15 MB/s | 6.1 MB 00:00 2024-03-04T09:12:10.619 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS Stream 9 - Extras packages 33 kB/s | 15 kB 00:00 2024-03-04T09:12:11.396 INFO:teuthology.orchestra.run.smithi146.stdout:Extra Packages for Enterprise Linux 31 MB/s | 20 MB 00:00 2024-03-04T09:12:12.116 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS Stream 9 - BaseOS 656 kB/s | 8.0 MB 00:12 2024-03-04T09:12:13.900 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS Stream 9 - AppStream 29 MB/s | 19 MB 00:00 2024-03-04T09:12:16.396 INFO:teuthology.orchestra.run.smithi146.stdout:lab-extras 11 kB/s | 1.7 kB 00:00 2024-03-04T09:12:18.784 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-03-04T09:12:18.788 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-04T09:12:18.788 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repository Size 2024-03-04T09:12:18.789 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-04T09:12:18.789 INFO:teuthology.orchestra.run.smithi146.stdout:Upgrading: 2024-03-04T09:12:18.789 INFO:teuthology.orchestra.run.smithi146.stdout: linux-firmware noarch 20240219-143.el9 CentOS-BaseOS 375 M 2024-03-04T09:12:18.789 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:12:18.789 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-03-04T09:12:18.789 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-04T09:12:18.789 INFO:teuthology.orchestra.run.smithi146.stdout:Upgrade 1 Package 2024-03-04T09:12:18.789 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:12:18.790 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 375 M 2024-03-04T09:12:18.790 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-03-04T09:12:19.226 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS Stream 9 - CRB 3.9 MB/s | 6.1 MB 00:01 2024-03-04T09:12:21.277 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS Stream 9 - Extras packages 17 kB/s | 15 kB 00:00 2024-03-04T09:12:23.005 INFO:teuthology.orchestra.run.smithi006.stdout:Extra Packages for Enterprise Linux 13 MB/s | 20 MB 00:01 2024-03-04T09:12:26.446 INFO:teuthology.orchestra.run.smithi146.stdout:linux-firmware-20240219-143.el9.noarch.rpm 49 MB/s | 375 MB 00:07 2024-03-04T09:12:26.447 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-03-04T09:12:26.448 INFO:teuthology.orchestra.run.smithi146.stdout:Total 49 MB/s | 375 MB 00:07 2024-03-04T09:12:26.448 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-03-04T09:12:26.544 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-03-04T09:12:26.544 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-03-04T09:12:28.120 INFO:teuthology.orchestra.run.smithi006.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-03-04T09:12:28.962 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-03-04T09:12:28.962 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-03-04T09:12:30.576 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-03-04T09:12:30.590 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-04T09:12:30.590 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repository Size 2024-03-04T09:12:30.590 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-04T09:12:30.590 INFO:teuthology.orchestra.run.smithi006.stdout:Upgrading: 2024-03-04T09:12:30.590 INFO:teuthology.orchestra.run.smithi006.stdout: linux-firmware noarch 20240219-143.el9 CentOS-BaseOS 375 M 2024-03-04T09:12:30.590 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:12:30.591 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2024-03-04T09:12:30.591 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-04T09:12:30.591 INFO:teuthology.orchestra.run.smithi006.stdout:Upgrade 1 Package 2024-03-04T09:12:30.591 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:12:30.591 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 375 M 2024-03-04T09:12:30.591 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2024-03-04T09:12:35.512 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-03-04T09:12:35.748 INFO:teuthology.orchestra.run.smithi146.stdout: Upgrading : linux-firmware-20240219-143.el9.noarch 1/2 2024-03-04T09:12:35.830 INFO:teuthology.orchestra.run.smithi146.stdout: Cleanup : linux-firmware-20230814-139.el9.noarch 2/2 2024-03-04T09:12:37.230 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: linux-firmware-20230814-139.el9.noarch 2/2 2024-03-04T09:12:37.230 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : linux-firmware-20240219-143.el9.noarch 1/2 2024-03-04T09:12:37.634 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : linux-firmware-20230814-139.el9.noarch 2/2 2024-03-04T09:12:37.635 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:12:37.635 INFO:teuthology.orchestra.run.smithi146.stdout:Upgraded: 2024-03-04T09:12:37.635 INFO:teuthology.orchestra.run.smithi146.stdout: linux-firmware-20240219-143.el9.noarch 2024-03-04T09:12:37.635 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:12:37.635 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-03-04T09:12:37.967 INFO:teuthology.task.kernel:Downloading kernel 786939134d7eee4dc7a805245d3f595929af37d7 on client.1... 2024-03-04T09:12:37.968 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=786939134d7eee4dc7a805245d3f595929af37d7 2024-03-04T09:12:38.201 INFO:teuthology.task.kernel:fetching, builder baseurl is https://3.chacra.ceph.com/r/kernel/testing/786939134d7eee4dc7a805245d3f595929af37d7/centos/9/flavors/default/x86_64/ 2024-03-04T09:12:38.201 DEBUG:teuthology.orchestra.run.smithi146:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.8.0_rc6_g786939134d7e-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://3.chacra.ceph.com/r/kernel/testing/786939134d7eee4dc7a805245d3f595929af37d7/centos/9/flavors/default/x86_64/ --input-file=- 2024-03-04T09:12:38.204 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi146.front.sepia.ceph.com to complete... 2024-03-04T09:12:38.921 INFO:teuthology.orchestra.run.smithi006.stdout:linux-firmware-20240219-143.el9.noarch.rpm 45 MB/s | 375 MB 00:08 2024-03-04T09:12:38.922 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2024-03-04T09:12:38.922 INFO:teuthology.orchestra.run.smithi006.stdout:Total 45 MB/s | 375 MB 00:08 2024-03-04T09:12:38.923 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2024-03-04T09:12:39.014 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2024-03-04T09:12:39.014 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2024-03-04T09:12:41.520 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2024-03-04T09:12:41.520 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2024-03-04T09:12:48.080 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2024-03-04T09:12:48.312 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : linux-firmware-20240219-143.el9.noarch 1/2 2024-03-04T09:12:48.407 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : linux-firmware-20230814-139.el9.noarch 2/2 2024-03-04T09:12:49.590 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04 09:12:49 URL:https://3.chacra.ceph.com/r/kernel/testing/786939134d7eee4dc7a805245d3f595929af37d7/centos/9/flavors/default/x86_64/kernel-6.8.0_rc6_g786939134d7e-1.x86_64.rpm [336049686/336049686] -> "/tmp/kernel.x86_64.rpm" [1] 2024-03-04T09:12:49.590 INFO:teuthology.orchestra.run.smithi146.stderr:FINISHED --2024-03-04 09:12:49-- 2024-03-04T09:12:49.590 INFO:teuthology.orchestra.run.smithi146.stderr:Total wall clock time: 11s 2024-03-04T09:12:49.590 INFO:teuthology.orchestra.run.smithi146.stderr:Downloaded: 1 files, 320M in 11s (28.9 MB/s) 2024-03-04T09:12:49.593 INFO:teuthology.task.kernel:Installing kernel 786939134d7eee4dc7a805245d3f595929af37d7 on client.1... 2024-03-04T09:12:49.594 DEBUG:teuthology.orchestra.run.smithi146:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-03-04T09:12:49.774 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: linux-firmware-20230814-139.el9.noarch 2/2 2024-03-04T09:12:49.774 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : linux-firmware-20240219-143.el9.noarch 1/2 2024-03-04T09:12:50.175 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : linux-firmware-20230814-139.el9.noarch 2/2 2024-03-04T09:12:50.175 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:12:50.176 INFO:teuthology.orchestra.run.smithi006.stdout:Upgraded: 2024-03-04T09:12:50.176 INFO:teuthology.orchestra.run.smithi006.stdout: linux-firmware-20240219-143.el9.noarch 2024-03-04T09:12:50.176 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:12:50.176 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-03-04T09:12:50.470 INFO:teuthology.task.kernel:Downloading kernel 786939134d7eee4dc7a805245d3f595929af37d7 on client.0... 2024-03-04T09:12:50.470 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=786939134d7eee4dc7a805245d3f595929af37d7 2024-03-04T09:12:50.626 INFO:teuthology.task.kernel:fetching, builder baseurl is https://3.chacra.ceph.com/r/kernel/testing/786939134d7eee4dc7a805245d3f595929af37d7/centos/9/flavors/default/x86_64/ 2024-03-04T09:12:50.626 DEBUG:teuthology.orchestra.run.smithi006:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.8.0_rc6_g786939134d7e-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://3.chacra.ceph.com/r/kernel/testing/786939134d7eee4dc7a805245d3f595929af37d7/centos/9/flavors/default/x86_64/ --input-file=- 2024-03-04T09:12:50.629 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi006.front.sepia.ceph.com to complete... 2024-03-04T09:12:51.527 INFO:teuthology.orchestra.run.smithi146.stdout:Verifying... ######################################## 2024-03-04T09:12:51.663 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing... ######################################## 2024-03-04T09:12:51.745 INFO:teuthology.orchestra.run.smithi146.stdout:Updating / installing... 2024-03-04T09:12:59.340 INFO:teuthology.orchestra.run.smithi146.stdout:kernel-6.8.0_rc6_g786939134d7e-1 ######################################## 2024-03-04T09:13:02.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04 09:13:02 URL:https://3.chacra.ceph.com/r/kernel/testing/786939134d7eee4dc7a805245d3f595929af37d7/centos/9/flavors/default/x86_64/kernel-6.8.0_rc6_g786939134d7e-1.x86_64.rpm [336049686/336049686] -> "/tmp/kernel.x86_64.rpm" [1] 2024-03-04T09:13:02.046 INFO:teuthology.orchestra.run.smithi006.stderr:FINISHED --2024-03-04 09:13:02-- 2024-03-04T09:13:02.046 INFO:teuthology.orchestra.run.smithi006.stderr:Total wall clock time: 11s 2024-03-04T09:13:02.046 INFO:teuthology.orchestra.run.smithi006.stderr:Downloaded: 1 files, 320M in 11s (28.9 MB/s) 2024-03-04T09:13:02.049 INFO:teuthology.task.kernel:Installing kernel 786939134d7eee4dc7a805245d3f595929af37d7 on client.0... 2024-03-04T09:13:02.049 DEBUG:teuthology.orchestra.run.smithi006:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-03-04T09:13:04.086 INFO:teuthology.orchestra.run.smithi006.stdout:Verifying... ######################################## 2024-03-04T09:13:04.147 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing... ######################################## 2024-03-04T09:13:04.232 INFO:teuthology.orchestra.run.smithi006.stdout:Updating / installing... 2024-03-04T09:13:12.136 INFO:teuthology.orchestra.run.smithi006.stdout:kernel-6.8.0_rc6_g786939134d7e-1 ######################################## 2024-03-04T09:13:15.064 INFO:teuthology.orchestra.run.smithi146.stderr:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-04T09:13:28.258 INFO:teuthology.orchestra.run.smithi006.stderr:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-04T09:13:37.552 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi146.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-03-04T09:13:37.552 DEBUG:teuthology.orchestra.run.smithi146:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-03-04T09:13:37.636 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e 2024-03-04T09:13:37.637 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/System.map 2024-03-04T09:13:37.637 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/config 2024-03-04T09:13:37.637 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel 2024-03-04T09:13:37.637 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/arch 2024-03-04T09:13:37.637 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/arch/x86 2024-03-04T09:13:37.637 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/arch/x86/crypto 2024-03-04T09:13:37.637 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-03-04T09:13:37.637 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-03-04T09:13:37.637 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-03-04T09:13:37.637 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-03-04T09:13:37.637 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-03-04T09:13:37.637 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-03-04T09:13:37.637 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-03-04T09:13:37.638 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/arch/x86/crypto/crc32c-intel.ko 2024-03-04T09:13:37.638 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-03-04T09:13:37.638 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-03-04T09:13:37.638 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-03-04T09:13:37.638 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-03-04T09:13:37.638 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/arch/x86/crypto/serpent-avx2.ko 2024-03-04T09:13:37.638 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-03-04T09:13:37.638 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-03-04T09:13:37.638 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-03-04T09:13:37.638 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-03-04T09:13:37.638 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-03-04T09:13:37.638 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-03-04T09:13:37.638 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-03-04T09:13:37.638 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/arch/x86/kernel 2024-03-04T09:13:37.639 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/arch/x86/kernel/cpu 2024-03-04T09:13:37.639 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/arch/x86/kernel/cpu/mce 2024-03-04T09:13:37.639 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-03-04T09:13:37.639 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/arch/x86/kvm 2024-03-04T09:13:37.639 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/arch/x86/kvm/kvm-amd.ko 2024-03-04T09:13:37.639 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/arch/x86/kvm/kvm-intel.ko 2024-03-04T09:13:37.639 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/arch/x86/kvm/kvm.ko 2024-03-04T09:13:37.639 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto 2024-03-04T09:13:37.639 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/ansi_cprng.ko 2024-03-04T09:13:37.639 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/anubis.ko 2024-03-04T09:13:37.639 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/arc4.ko 2024-03-04T09:13:37.639 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/async_tx 2024-03-04T09:13:37.639 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/async_tx/async_memcpy.ko 2024-03-04T09:13:37.639 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/async_tx/async_pq.ko 2024-03-04T09:13:37.640 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/async_tx/async_raid6_recov.ko 2024-03-04T09:13:37.640 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/async_tx/async_tx.ko 2024-03-04T09:13:37.640 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/async_tx/async_xor.ko 2024-03-04T09:13:37.640 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/async_tx/raid6test.ko 2024-03-04T09:13:37.640 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/authenc.ko 2024-03-04T09:13:37.640 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/authencesn.ko 2024-03-04T09:13:37.640 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/blake2b_generic.ko 2024-03-04T09:13:37.640 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/blowfish_common.ko 2024-03-04T09:13:37.640 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/blowfish_generic.ko 2024-03-04T09:13:37.640 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/camellia_generic.ko 2024-03-04T09:13:37.640 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/cast5_generic.ko 2024-03-04T09:13:37.640 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/cast6_generic.ko 2024-03-04T09:13:37.640 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/cast_common.ko 2024-03-04T09:13:37.640 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/ccm.ko 2024-03-04T09:13:37.641 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/cmac.ko 2024-03-04T09:13:37.641 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/crc32_generic.ko 2024-03-04T09:13:37.641 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/crypto_engine.ko 2024-03-04T09:13:37.641 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/crypto_user.ko 2024-03-04T09:13:37.641 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/cts.ko 2024-03-04T09:13:37.641 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/des_generic.ko 2024-03-04T09:13:37.641 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/dh_generic.ko 2024-03-04T09:13:37.641 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/echainiv.ko 2024-03-04T09:13:37.641 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/essiv.ko 2024-03-04T09:13:37.641 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/fcrypt.ko 2024-03-04T09:13:37.641 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/khazad.ko 2024-03-04T09:13:37.641 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/lz4.ko 2024-03-04T09:13:37.641 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/lz4hc.ko 2024-03-04T09:13:37.641 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/md4.ko 2024-03-04T09:13:37.642 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/michael_mic.ko 2024-03-04T09:13:37.642 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/pcbc.ko 2024-03-04T09:13:37.642 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/pcrypt.ko 2024-03-04T09:13:37.642 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/rmd160.ko 2024-03-04T09:13:37.642 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/seed.ko 2024-03-04T09:13:37.642 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/serpent_generic.ko 2024-03-04T09:13:37.642 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/tcrypt.ko 2024-03-04T09:13:37.642 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/tea.ko 2024-03-04T09:13:37.642 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/twofish_common.ko 2024-03-04T09:13:37.642 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/twofish_generic.ko 2024-03-04T09:13:37.642 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/vmac.ko 2024-03-04T09:13:37.642 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/wp512.ko 2024-03-04T09:13:37.642 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/xcbc.ko 2024-03-04T09:13:37.642 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/xor.ko 2024-03-04T09:13:37.643 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/xxhash_generic.ko 2024-03-04T09:13:37.643 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers 2024-03-04T09:13:37.643 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/acpi 2024-03-04T09:13:37.643 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/acpi/acpi_ipmi.ko 2024-03-04T09:13:37.643 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/acpi/acpi_pad.ko 2024-03-04T09:13:37.643 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/acpi/custom_method.ko 2024-03-04T09:13:37.643 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/acpi/ec_sys.ko 2024-03-04T09:13:37.643 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/acpi/platform_profile.ko 2024-03-04T09:13:37.643 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/acpi/sbs.ko 2024-03-04T09:13:37.643 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/acpi/sbshc.ko 2024-03-04T09:13:37.643 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/acpi/video.ko 2024-03-04T09:13:37.643 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata 2024-03-04T09:13:37.643 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/acard-ahci.ko 2024-03-04T09:13:37.643 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/ahci_platform.ko 2024-03-04T09:13:37.643 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/ata_generic.ko 2024-03-04T09:13:37.644 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/libahci_platform.ko 2024-03-04T09:13:37.644 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/pata_acpi.ko 2024-03-04T09:13:37.644 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/pata_ali.ko 2024-03-04T09:13:37.644 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/pata_amd.ko 2024-03-04T09:13:37.644 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/pata_artop.ko 2024-03-04T09:13:37.644 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/pata_atiixp.ko 2024-03-04T09:13:37.644 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/pata_atp867x.ko 2024-03-04T09:13:37.644 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/pata_cmd640.ko 2024-03-04T09:13:37.644 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/pata_cmd64x.ko 2024-03-04T09:13:37.644 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/pata_cypress.ko 2024-03-04T09:13:37.644 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/pata_efar.ko 2024-03-04T09:13:37.644 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/pata_hpt366.ko 2024-03-04T09:13:37.644 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/pata_hpt37x.ko 2024-03-04T09:13:37.644 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/pata_hpt3x2n.ko 2024-03-04T09:13:37.644 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/pata_hpt3x3.ko 2024-03-04T09:13:37.645 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/pata_it8213.ko 2024-03-04T09:13:37.645 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/pata_it821x.ko 2024-03-04T09:13:37.645 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/pata_jmicron.ko 2024-03-04T09:13:37.645 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/pata_marvell.ko 2024-03-04T09:13:37.645 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/pata_mpiix.ko 2024-03-04T09:13:37.645 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/pata_netcell.ko 2024-03-04T09:13:37.645 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/pata_ninja32.ko 2024-03-04T09:13:37.645 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/pata_ns87410.ko 2024-03-04T09:13:37.645 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/pata_ns87415.ko 2024-03-04T09:13:37.645 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/pata_oldpiix.ko 2024-03-04T09:13:37.645 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/pata_opti.ko 2024-03-04T09:13:37.645 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/pata_optidma.ko 2024-03-04T09:13:37.646 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/pata_pcmcia.ko 2024-03-04T09:13:37.646 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/pata_pdc2027x.ko 2024-03-04T09:13:37.646 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-03-04T09:13:37.646 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/pata_piccolo.ko 2024-03-04T09:13:37.646 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/pata_rdc.ko 2024-03-04T09:13:37.646 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/pata_sch.ko 2024-03-04T09:13:37.646 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/pata_serverworks.ko 2024-03-04T09:13:37.646 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/pata_sil680.ko 2024-03-04T09:13:37.646 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/pata_sis.ko 2024-03-04T09:13:37.646 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/pata_sl82c105.ko 2024-03-04T09:13:37.646 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/pata_triflex.ko 2024-03-04T09:13:37.646 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/pata_via.ko 2024-03-04T09:13:37.646 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/pdc_adma.ko 2024-03-04T09:13:37.646 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/sata_inic162x.ko 2024-03-04T09:13:37.646 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/sata_mv.ko 2024-03-04T09:13:37.647 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/sata_nv.ko 2024-03-04T09:13:37.647 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/sata_promise.ko 2024-03-04T09:13:37.647 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/sata_qstor.ko 2024-03-04T09:13:37.647 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/sata_sil.ko 2024-03-04T09:13:37.647 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/sata_sil24.ko 2024-03-04T09:13:37.647 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/sata_sis.ko 2024-03-04T09:13:37.647 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/sata_svw.ko 2024-03-04T09:13:37.647 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/sata_sx4.ko 2024-03-04T09:13:37.647 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/sata_uli.ko 2024-03-04T09:13:37.647 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/sata_via.ko 2024-03-04T09:13:37.647 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/sata_vsc.ko 2024-03-04T09:13:37.647 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/auxdisplay 2024-03-04T09:13:37.647 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/auxdisplay/ks0108.ko 2024-03-04T09:13:37.647 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/base 2024-03-04T09:13:37.647 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/base/regmap 2024-03-04T09:13:37.648 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/base/regmap/regmap-i2c.ko 2024-03-04T09:13:37.648 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/base/regmap/regmap-mmio.ko 2024-03-04T09:13:37.648 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/bcma 2024-03-04T09:13:37.648 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/bcma/bcma.ko 2024-03-04T09:13:37.648 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/block 2024-03-04T09:13:37.648 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/block/aoe 2024-03-04T09:13:37.648 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/block/aoe/aoe.ko 2024-03-04T09:13:37.648 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/block/brd.ko 2024-03-04T09:13:37.648 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/block/drbd 2024-03-04T09:13:37.648 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/block/drbd/drbd.ko 2024-03-04T09:13:37.648 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/block/floppy.ko 2024-03-04T09:13:37.648 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/block/loop.ko 2024-03-04T09:13:37.648 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/block/mtip32xx 2024-03-04T09:13:37.648 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-03-04T09:13:37.649 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/block/nbd.ko 2024-03-04T09:13:37.649 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/block/null_blk 2024-03-04T09:13:37.649 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/block/null_blk/null_blk.ko 2024-03-04T09:13:37.649 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/block/pktcdvd.ko 2024-03-04T09:13:37.649 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/block/rbd.ko 2024-03-04T09:13:37.649 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/block/virtio_blk.ko 2024-03-04T09:13:37.649 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/block/xen-blkback 2024-03-04T09:13:37.649 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-03-04T09:13:37.649 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/block/xen-blkfront.ko 2024-03-04T09:13:37.649 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/block/zram 2024-03-04T09:13:37.649 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/block/zram/zram.ko 2024-03-04T09:13:37.649 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/char 2024-03-04T09:13:37.649 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/char/hangcheck-timer.ko 2024-03-04T09:13:37.650 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/char/hw_random 2024-03-04T09:13:37.650 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/char/hw_random/amd-rng.ko 2024-03-04T09:13:37.650 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/char/hw_random/intel-rng.ko 2024-03-04T09:13:37.650 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-03-04T09:13:37.650 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/char/hw_random/via-rng.ko 2024-03-04T09:13:37.650 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/char/hw_random/virtio-rng.ko 2024-03-04T09:13:37.650 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/char/ipmi 2024-03-04T09:13:37.650 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-03-04T09:13:37.650 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-03-04T09:13:37.651 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-03-04T09:13:37.651 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/char/ipmi/ipmi_si.ko 2024-03-04T09:13:37.651 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-03-04T09:13:37.651 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-03-04T09:13:37.651 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/char/lp.ko 2024-03-04T09:13:37.651 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/char/mwave 2024-03-04T09:13:37.651 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/char/mwave/mwave.ko 2024-03-04T09:13:37.652 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/char/ppdev.ko 2024-03-04T09:13:37.652 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/char/tlclk.ko 2024-03-04T09:13:37.652 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/char/tpm 2024-03-04T09:13:37.652 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/char/tpm/tpm.ko 2024-03-04T09:13:37.652 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/char/tpm/tpm_atmel.ko 2024-03-04T09:13:37.653 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/char/tpm/tpm_infineon.ko 2024-03-04T09:13:37.653 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/char/tpm/tpm_nsc.ko 2024-03-04T09:13:37.653 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/char/tpm/tpm_tis.ko 2024-03-04T09:13:37.653 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-03-04T09:13:37.653 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/char/uv_mmtimer.ko 2024-03-04T09:13:37.653 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/char/virtio_console.ko 2024-03-04T09:13:37.653 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/cpufreq 2024-03-04T09:13:37.653 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-03-04T09:13:37.654 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-03-04T09:13:37.654 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/cpufreq/p4-clockmod.ko 2024-03-04T09:13:37.654 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-03-04T09:13:37.654 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/cpufreq/powernow-k8.ko 2024-03-04T09:13:37.654 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/cpufreq/speedstep-lib.ko 2024-03-04T09:13:37.654 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/crypto 2024-03-04T09:13:37.654 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/crypto/ccp 2024-03-04T09:13:37.654 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-03-04T09:13:37.654 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/crypto/ccp/ccp.ko 2024-03-04T09:13:37.654 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/crypto/intel 2024-03-04T09:13:37.654 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/crypto/intel/qat 2024-03-04T09:13:37.654 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/crypto/intel/qat/qat_common 2024-03-04T09:13:37.654 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-03-04T09:13:37.655 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-03-04T09:13:37.655 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-03-04T09:13:37.655 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/crypto/padlock-aes.ko 2024-03-04T09:13:37.655 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/crypto/padlock-sha.ko 2024-03-04T09:13:37.655 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/crypto/virtio 2024-03-04T09:13:37.655 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-03-04T09:13:37.655 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/dca 2024-03-04T09:13:37.655 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/dca/dca.ko 2024-03-04T09:13:37.655 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/devfreq 2024-03-04T09:13:37.655 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-03-04T09:13:37.655 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/dma 2024-03-04T09:13:37.655 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/dma/dw 2024-03-04T09:13:37.656 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/dma/dw/dw_dmac.ko 2024-03-04T09:13:37.656 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/dma/ioat 2024-03-04T09:13:37.656 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/dma/ioat/ioatdma.ko 2024-03-04T09:13:37.656 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/edac 2024-03-04T09:13:37.656 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/edac/amd64_edac.ko 2024-03-04T09:13:37.656 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/edac/e752x_edac.ko 2024-03-04T09:13:37.656 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/edac/edac_mce_amd.ko 2024-03-04T09:13:37.656 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/edac/i3000_edac.ko 2024-03-04T09:13:37.656 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/edac/i3200_edac.ko 2024-03-04T09:13:37.656 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/edac/i5100_edac.ko 2024-03-04T09:13:37.656 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/edac/i5400_edac.ko 2024-03-04T09:13:37.656 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/edac/i7300_edac.ko 2024-03-04T09:13:37.656 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/edac/i7core_edac.ko 2024-03-04T09:13:37.656 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/edac/i82975x_edac.ko 2024-03-04T09:13:37.657 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/edac/ie31200_edac.ko 2024-03-04T09:13:37.657 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/edac/sb_edac.ko 2024-03-04T09:13:37.657 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/edac/x38_edac.ko 2024-03-04T09:13:37.657 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/firmware 2024-03-04T09:13:37.657 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/firmware/edd.ko 2024-03-04T09:13:37.657 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/firmware/iscsi_ibft.ko 2024-03-04T09:13:37.657 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/gpio 2024-03-04T09:13:37.657 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/gpio/gpio-ich.ko 2024-03-04T09:13:37.657 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/gpio/gpio-viperboard.ko 2024-03-04T09:13:37.657 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid 2024-03-04T09:13:37.657 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-appleir.ko 2024-03-04T09:13:37.657 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-aureal.ko 2024-03-04T09:13:37.657 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-axff.ko 2024-03-04T09:13:37.657 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-dr.ko 2024-03-04T09:13:37.658 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-elecom.ko 2024-03-04T09:13:37.658 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-elo.ko 2024-03-04T09:13:37.658 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-emsff.ko 2024-03-04T09:13:37.658 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-gaff.ko 2024-03-04T09:13:37.658 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-gt683r.ko 2024-03-04T09:13:37.658 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-gyration.ko 2024-03-04T09:13:37.658 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-holtek-kbd.ko 2024-03-04T09:13:37.658 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-holtek-mouse.ko 2024-03-04T09:13:37.658 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-holtekff.ko 2024-03-04T09:13:37.658 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-hyperv.ko 2024-03-04T09:13:37.658 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-icade.ko 2024-03-04T09:13:37.658 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-keytouch.ko 2024-03-04T09:13:37.658 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-kye.ko 2024-03-04T09:13:37.658 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-lcpower.ko 2024-03-04T09:13:37.659 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-led.ko 2024-03-04T09:13:37.659 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-lenovo.ko 2024-03-04T09:13:37.659 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-logitech-dj.ko 2024-03-04T09:13:37.659 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-03-04T09:13:37.659 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-multitouch.ko 2024-03-04T09:13:37.659 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-ortek.ko 2024-03-04T09:13:37.659 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-penmount.ko 2024-03-04T09:13:37.659 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-petalynx.ko 2024-03-04T09:13:37.659 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-picolcd.ko 2024-03-04T09:13:37.659 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-pl.ko 2024-03-04T09:13:37.659 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-plantronics.ko 2024-03-04T09:13:37.659 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-primax.ko 2024-03-04T09:13:37.659 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-rmi.ko 2024-03-04T09:13:37.659 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-roccat-arvo.ko 2024-03-04T09:13:37.659 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-roccat-common.ko 2024-03-04T09:13:37.660 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-roccat-isku.ko 2024-03-04T09:13:37.660 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-roccat-kone.ko 2024-03-04T09:13:37.660 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-03-04T09:13:37.660 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-roccat-konepure.ko 2024-03-04T09:13:37.660 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-03-04T09:13:37.660 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-roccat-lua.ko 2024-03-04T09:13:37.660 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-roccat-pyra.ko 2024-03-04T09:13:37.660 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-roccat-ryos.ko 2024-03-04T09:13:37.660 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-roccat-savu.ko 2024-03-04T09:13:37.660 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-roccat.ko 2024-03-04T09:13:37.660 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-saitek.ko 2024-03-04T09:13:37.660 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-samsung.ko 2024-03-04T09:13:37.660 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-sensor-hub.ko 2024-03-04T09:13:37.660 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-sjoy.ko 2024-03-04T09:13:37.661 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-sony.ko 2024-03-04T09:13:37.661 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-speedlink.ko 2024-03-04T09:13:37.661 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-steelseries.ko 2024-03-04T09:13:37.661 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-sunplus.ko 2024-03-04T09:13:37.661 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-thrustmaster.ko 2024-03-04T09:13:37.661 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-tivo.ko 2024-03-04T09:13:37.661 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-tmff.ko 2024-03-04T09:13:37.661 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-topseed.ko 2024-03-04T09:13:37.661 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-twinhan.ko 2024-03-04T09:13:37.661 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-uclogic.ko 2024-03-04T09:13:37.661 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-waltop.ko 2024-03-04T09:13:37.661 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-wiimote.ko 2024-03-04T09:13:37.662 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-xinmo.ko 2024-03-04T09:13:37.662 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-zpff.ko 2024-03-04T09:13:37.662 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-zydacron.ko 2024-03-04T09:13:37.662 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/uhid.ko 2024-03-04T09:13:37.662 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/wacom.ko 2024-03-04T09:13:37.662 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hv 2024-03-04T09:13:37.662 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hv/hv_balloon.ko 2024-03-04T09:13:37.662 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hv/hv_utils.ko 2024-03-04T09:13:37.662 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hv/hv_vmbus.ko 2024-03-04T09:13:37.662 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon 2024-03-04T09:13:37.662 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/abituguru.ko 2024-03-04T09:13:37.662 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/abituguru3.ko 2024-03-04T09:13:37.663 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/acpi_power_meter.ko 2024-03-04T09:13:37.663 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/ad7414.ko 2024-03-04T09:13:37.663 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/ad7418.ko 2024-03-04T09:13:37.663 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/adc128d818.ko 2024-03-04T09:13:37.663 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/adm1025.ko 2024-03-04T09:13:37.663 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/adm1026.ko 2024-03-04T09:13:37.663 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/adm1029.ko 2024-03-04T09:13:37.663 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/adm1031.ko 2024-03-04T09:13:37.663 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/adm9240.ko 2024-03-04T09:13:37.663 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/ads7828.ko 2024-03-04T09:13:37.663 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/adt7410.ko 2024-03-04T09:13:37.663 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/adt7411.ko 2024-03-04T09:13:37.663 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/adt7462.ko 2024-03-04T09:13:37.663 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/adt7470.ko 2024-03-04T09:13:37.663 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/adt7475.ko 2024-03-04T09:13:37.664 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/adt7x10.ko 2024-03-04T09:13:37.664 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/amc6821.ko 2024-03-04T09:13:37.664 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/applesmc.ko 2024-03-04T09:13:37.664 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/asb100.ko 2024-03-04T09:13:37.664 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/asc7621.ko 2024-03-04T09:13:37.664 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/asus_atk0110.ko 2024-03-04T09:13:37.664 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/atxp1.ko 2024-03-04T09:13:37.664 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/coretemp.ko 2024-03-04T09:13:37.664 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-03-04T09:13:37.664 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/dme1737.ko 2024-03-04T09:13:37.664 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/ds1621.ko 2024-03-04T09:13:37.664 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/ds620.ko 2024-03-04T09:13:37.664 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/emc1403.ko 2024-03-04T09:13:37.664 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/emc6w201.ko 2024-03-04T09:13:37.665 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/f71805f.ko 2024-03-04T09:13:37.665 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/f71882fg.ko 2024-03-04T09:13:37.665 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/f75375s.ko 2024-03-04T09:13:37.665 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/fam15h_power.ko 2024-03-04T09:13:37.665 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/fschmd.ko 2024-03-04T09:13:37.665 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/g760a.ko 2024-03-04T09:13:37.665 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/g762.ko 2024-03-04T09:13:37.665 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/gl518sm.ko 2024-03-04T09:13:37.665 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/gl520sm.ko 2024-03-04T09:13:37.665 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/hwmon-vid.ko 2024-03-04T09:13:37.665 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/i5500_temp.ko 2024-03-04T09:13:37.665 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/i5k_amb.ko 2024-03-04T09:13:37.665 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/ibmaem.ko 2024-03-04T09:13:37.665 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/ibmpex.ko 2024-03-04T09:13:37.665 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/ina209.ko 2024-03-04T09:13:37.666 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/ina2xx.ko 2024-03-04T09:13:37.666 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/it87.ko 2024-03-04T09:13:37.666 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/k10temp.ko 2024-03-04T09:13:37.666 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/k8temp.ko 2024-03-04T09:13:37.666 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/lineage-pem.ko 2024-03-04T09:13:37.666 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/lm63.ko 2024-03-04T09:13:37.666 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/lm73.ko 2024-03-04T09:13:37.666 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/lm75.ko 2024-03-04T09:13:37.666 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/lm77.ko 2024-03-04T09:13:37.666 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/lm78.ko 2024-03-04T09:13:37.666 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/lm80.ko 2024-03-04T09:13:37.666 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/lm83.ko 2024-03-04T09:13:37.666 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/lm85.ko 2024-03-04T09:13:37.666 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/lm87.ko 2024-03-04T09:13:37.666 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/lm90.ko 2024-03-04T09:13:37.667 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/lm92.ko 2024-03-04T09:13:37.667 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/lm93.ko 2024-03-04T09:13:37.667 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/lm95234.ko 2024-03-04T09:13:37.667 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/lm95241.ko 2024-03-04T09:13:37.667 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/lm95245.ko 2024-03-04T09:13:37.667 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/ltc2945.ko 2024-03-04T09:13:37.667 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/ltc4151.ko 2024-03-04T09:13:37.667 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/ltc4215.ko 2024-03-04T09:13:37.667 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/ltc4222.ko 2024-03-04T09:13:37.667 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/ltc4245.ko 2024-03-04T09:13:37.667 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/ltc4260.ko 2024-03-04T09:13:37.667 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/ltc4261.ko 2024-03-04T09:13:37.667 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/max16065.ko 2024-03-04T09:13:37.667 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/max1619.ko 2024-03-04T09:13:37.667 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/max1668.ko 2024-03-04T09:13:37.667 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/max197.ko 2024-03-04T09:13:37.668 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/max6639.ko 2024-03-04T09:13:37.668 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/max6650.ko 2024-03-04T09:13:37.668 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/max6697.ko 2024-03-04T09:13:37.668 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/mcp3021.ko 2024-03-04T09:13:37.668 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/nct6683.ko 2024-03-04T09:13:37.668 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/nct6775-core.ko 2024-03-04T09:13:37.668 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/nct6775.ko 2024-03-04T09:13:37.668 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/nct7802.ko 2024-03-04T09:13:37.668 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/pc87360.ko 2024-03-04T09:13:37.668 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/pc87427.ko 2024-03-04T09:13:37.668 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/pcf8591.ko 2024-03-04T09:13:37.668 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/pmbus 2024-03-04T09:13:37.668 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-03-04T09:13:37.668 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-03-04T09:13:37.669 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-03-04T09:13:37.669 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/pmbus/max16064.ko 2024-03-04T09:13:37.669 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/pmbus/max34440.ko 2024-03-04T09:13:37.669 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/pmbus/max8688.ko 2024-03-04T09:13:37.669 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-03-04T09:13:37.669 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-03-04T09:13:37.669 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-03-04T09:13:37.669 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-03-04T09:13:37.669 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-03-04T09:13:37.669 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-03-04T09:13:37.669 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/powr1220.ko 2024-03-04T09:13:37.669 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/sch5627.ko 2024-03-04T09:13:37.669 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/sch5636.ko 2024-03-04T09:13:37.669 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/sch56xx-common.ko 2024-03-04T09:13:37.669 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/sht15.ko 2024-03-04T09:13:37.669 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/sht21.ko 2024-03-04T09:13:37.670 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/shtc1.ko 2024-03-04T09:13:37.670 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/sis5595.ko 2024-03-04T09:13:37.670 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/smsc47b397.ko 2024-03-04T09:13:37.670 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/smsc47m1.ko 2024-03-04T09:13:37.670 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/smsc47m192.ko 2024-03-04T09:13:37.670 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/thmc50.ko 2024-03-04T09:13:37.670 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/tmp102.ko 2024-03-04T09:13:37.670 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/tmp103.ko 2024-03-04T09:13:37.670 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/tmp401.ko 2024-03-04T09:13:37.670 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/tmp421.ko 2024-03-04T09:13:37.670 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/via-cputemp.ko 2024-03-04T09:13:37.670 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/via686a.ko 2024-03-04T09:13:37.671 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/vt1211.ko 2024-03-04T09:13:37.671 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/vt8231.ko 2024-03-04T09:13:37.671 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/w83627ehf.ko 2024-03-04T09:13:37.671 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/w83627hf.ko 2024-03-04T09:13:37.671 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/w83781d.ko 2024-03-04T09:13:37.671 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/w83791d.ko 2024-03-04T09:13:37.671 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/w83792d.ko 2024-03-04T09:13:37.671 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/w83793.ko 2024-03-04T09:13:37.671 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/w83795.ko 2024-03-04T09:13:37.671 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/w83l785ts.ko 2024-03-04T09:13:37.671 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/w83l786ng.ko 2024-03-04T09:13:37.671 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/i2c 2024-03-04T09:13:37.671 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/i2c/algos 2024-03-04T09:13:37.671 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-03-04T09:13:37.671 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-03-04T09:13:37.672 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/i2c/busses 2024-03-04T09:13:37.672 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-03-04T09:13:37.672 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-03-04T09:13:37.672 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-03-04T09:13:37.672 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-03-04T09:13:37.672 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-03-04T09:13:37.672 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-03-04T09:13:37.672 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-03-04T09:13:37.672 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-03-04T09:13:37.672 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/i2c/busses/i2c-i801.ko 2024-03-04T09:13:37.672 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/i2c/busses/i2c-isch.ko 2024-03-04T09:13:37.673 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-03-04T09:13:37.673 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-03-04T09:13:37.673 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-03-04T09:13:37.673 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/i2c/busses/i2c-parport.ko 2024-03-04T09:13:37.673 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-03-04T09:13:37.673 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-03-04T09:13:37.673 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-03-04T09:13:37.673 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-03-04T09:13:37.673 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-03-04T09:13:37.673 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-03-04T09:13:37.673 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/i2c/busses/i2c-via.ko 2024-03-04T09:13:37.673 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-03-04T09:13:37.673 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-03-04T09:13:37.673 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/i2c/i2c-dev.ko 2024-03-04T09:13:37.674 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/i2c/i2c-mux.ko 2024-03-04T09:13:37.674 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-03-04T09:13:37.674 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/i2c/i2c-smbus.ko 2024-03-04T09:13:37.674 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/i2c/i2c-stub.ko 2024-03-04T09:13:37.674 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/infiniband 2024-03-04T09:13:37.674 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/infiniband/core 2024-03-04T09:13:37.674 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/infiniband/core/ib_cm.ko 2024-03-04T09:13:37.674 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/infiniband/core/ib_core.ko 2024-03-04T09:13:37.674 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/infiniband/core/ib_umad.ko 2024-03-04T09:13:37.674 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-03-04T09:13:37.674 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/infiniband/core/iw_cm.ko 2024-03-04T09:13:37.674 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/infiniband/core/rdma_cm.ko 2024-03-04T09:13:37.675 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-03-04T09:13:37.675 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/infiniband/hw 2024-03-04T09:13:37.675 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/infiniband/hw/cxgb4 2024-03-04T09:13:37.675 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-03-04T09:13:37.675 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/infiniband/hw/mlx4 2024-03-04T09:13:37.675 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-03-04T09:13:37.675 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/infiniband/hw/mlx5 2024-03-04T09:13:37.675 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-03-04T09:13:37.675 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/infiniband/hw/mthca 2024-03-04T09:13:37.675 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-03-04T09:13:37.675 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/infiniband/ulp 2024-03-04T09:13:37.675 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/infiniband/ulp/ipoib 2024-03-04T09:13:37.675 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-03-04T09:13:37.675 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/infiniband/ulp/iser 2024-03-04T09:13:37.675 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-03-04T09:13:37.676 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/infiniband/ulp/isert 2024-03-04T09:13:37.676 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-03-04T09:13:37.676 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/infiniband/ulp/srp 2024-03-04T09:13:37.676 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-03-04T09:13:37.676 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/infiniband/ulp/srpt 2024-03-04T09:13:37.676 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-03-04T09:13:37.676 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/input 2024-03-04T09:13:37.676 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/input/gameport 2024-03-04T09:13:37.676 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-03-04T09:13:37.676 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/input/gameport/fm801-gp.ko 2024-03-04T09:13:37.676 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/input/gameport/gameport.ko 2024-03-04T09:13:37.677 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/input/joydev.ko 2024-03-04T09:13:37.677 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/input/keyboard 2024-03-04T09:13:37.677 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/input/keyboard/gpio_keys.ko 2024-03-04T09:13:37.677 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/input/mouse 2024-03-04T09:13:37.677 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/input/mouse/appletouch.ko 2024-03-04T09:13:37.677 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/input/mouse/bcm5974.ko 2024-03-04T09:13:37.677 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/input/mouse/cyapatp.ko 2024-03-04T09:13:37.678 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/input/mouse/elan_i2c.ko 2024-03-04T09:13:37.678 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/input/mouse/sermouse.ko 2024-03-04T09:13:37.678 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-03-04T09:13:37.678 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/input/mouse/synaptics_usb.ko 2024-03-04T09:13:37.678 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/input/mouse/vsxxxaa.ko 2024-03-04T09:13:37.678 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/input/rmi4 2024-03-04T09:13:37.678 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/input/rmi4/rmi_core.ko 2024-03-04T09:13:37.678 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/input/serio 2024-03-04T09:13:37.678 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/input/serio/altera_ps2.ko 2024-03-04T09:13:37.678 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/input/serio/arc_ps2.ko 2024-03-04T09:13:37.678 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-03-04T09:13:37.678 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/input/serio/serio_raw.ko 2024-03-04T09:13:37.679 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/input/sparse-keymap.ko 2024-03-04T09:13:37.679 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/leds 2024-03-04T09:13:37.679 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/leds/leds-blinkm.ko 2024-03-04T09:13:37.679 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/leds/leds-lm3530.ko 2024-03-04T09:13:37.679 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/leds/leds-lp3944.ko 2024-03-04T09:13:37.679 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/leds/leds-ss4200.ko 2024-03-04T09:13:37.679 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/leds/trigger 2024-03-04T09:13:37.680 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-03-04T09:13:37.680 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-03-04T09:13:37.680 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-03-04T09:13:37.680 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-03-04T09:13:37.680 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-03-04T09:13:37.680 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-03-04T09:13:37.680 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-03-04T09:13:37.680 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-03-04T09:13:37.680 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-03-04T09:13:37.680 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/md 2024-03-04T09:13:37.680 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/md/bcache 2024-03-04T09:13:37.681 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/md/bcache/bcache.ko 2024-03-04T09:13:37.681 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/md/dm-bio-prison.ko 2024-03-04T09:13:37.681 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/md/dm-cache-smq.ko 2024-03-04T09:13:37.681 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/md/dm-cache.ko 2024-03-04T09:13:37.681 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/md/dm-crypt.ko 2024-03-04T09:13:37.681 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/md/dm-delay.ko 2024-03-04T09:13:37.681 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/md/dm-flakey.ko 2024-03-04T09:13:37.681 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/md/dm-log-userspace.ko 2024-03-04T09:13:37.681 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/md/dm-multipath.ko 2024-03-04T09:13:37.681 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/md/dm-queue-length.ko 2024-03-04T09:13:37.681 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/md/dm-raid.ko 2024-03-04T09:13:37.681 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/md/dm-round-robin.ko 2024-03-04T09:13:37.682 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/md/dm-service-time.ko 2024-03-04T09:13:37.682 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/md/dm-switch.ko 2024-03-04T09:13:37.682 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/md/dm-thin-pool.ko 2024-03-04T09:13:37.682 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/md/dm-verity.ko 2024-03-04T09:13:37.682 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/md/persistent-data 2024-03-04T09:13:37.682 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-03-04T09:13:37.682 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/md/raid0.ko 2024-03-04T09:13:37.682 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/md/raid1.ko 2024-03-04T09:13:37.682 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/md/raid10.ko 2024-03-04T09:13:37.682 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/md/raid456.ko 2024-03-04T09:13:37.682 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/message 2024-03-04T09:13:37.682 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/message/fusion 2024-03-04T09:13:37.682 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/message/fusion/mptbase.ko 2024-03-04T09:13:37.682 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/message/fusion/mptctl.ko 2024-03-04T09:13:37.683 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/message/fusion/mptfc.ko 2024-03-04T09:13:37.683 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/message/fusion/mptlan.ko 2024-03-04T09:13:37.683 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/message/fusion/mptsas.ko 2024-03-04T09:13:37.683 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/message/fusion/mptscsih.ko 2024-03-04T09:13:37.683 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/message/fusion/mptspi.ko 2024-03-04T09:13:37.683 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/mfd 2024-03-04T09:13:37.683 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/mfd/lpc_ich.ko 2024-03-04T09:13:37.683 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/mfd/lpc_sch.ko 2024-03-04T09:13:37.683 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/mfd/mfd-core.ko 2024-03-04T09:13:37.683 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/mfd/sm501.ko 2024-03-04T09:13:37.683 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/mfd/viperboard.ko 2024-03-04T09:13:37.683 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/mfd/vx855.ko 2024-03-04T09:13:37.683 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/mfd/wl1273-core.ko 2024-03-04T09:13:37.683 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/misc 2024-03-04T09:13:37.683 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/misc/altera-stapl 2024-03-04T09:13:37.684 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-03-04T09:13:37.684 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/misc/apds9802als.ko 2024-03-04T09:13:37.684 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/misc/apds990x.ko 2024-03-04T09:13:37.684 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/misc/bh1770glc.ko 2024-03-04T09:13:37.684 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/misc/cb710 2024-03-04T09:13:37.684 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/misc/cb710/cb710.ko 2024-03-04T09:13:37.684 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/misc/eeprom 2024-03-04T09:13:37.684 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/misc/eeprom/at24.ko 2024-03-04T09:13:37.684 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-03-04T09:13:37.684 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/misc/eeprom/max6875.ko 2024-03-04T09:13:37.684 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/misc/enclosure.ko 2024-03-04T09:13:37.685 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/misc/hpilo.ko 2024-03-04T09:13:37.685 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/misc/isl29003.ko 2024-03-04T09:13:37.685 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/misc/isl29020.ko 2024-03-04T09:13:37.685 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/misc/lis3lv02d 2024-03-04T09:13:37.685 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-03-04T09:13:37.685 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-03-04T09:13:37.685 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/misc/mei 2024-03-04T09:13:37.685 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/misc/mei/mei-me.ko 2024-03-04T09:13:37.685 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/misc/mei/mei-txe.ko 2024-03-04T09:13:37.685 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/misc/mei/mei.ko 2024-03-04T09:13:37.685 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/misc/sgi-gru 2024-03-04T09:13:37.685 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/misc/sgi-gru/gru.ko 2024-03-04T09:13:37.685 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/misc/sgi-xp 2024-03-04T09:13:37.686 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/misc/sgi-xp/xp.ko 2024-03-04T09:13:37.686 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/misc/sgi-xp/xpc.ko 2024-03-04T09:13:37.686 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-03-04T09:13:37.686 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/misc/tifm_7xx1.ko 2024-03-04T09:13:37.686 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/misc/tifm_core.ko 2024-03-04T09:13:37.686 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/misc/tsl2550.ko 2024-03-04T09:13:37.686 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/misc/vmw_balloon.ko 2024-03-04T09:13:37.686 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/misc/vmw_vmci 2024-03-04T09:13:37.686 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-03-04T09:13:37.686 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net 2024-03-04T09:13:37.686 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/bonding 2024-03-04T09:13:37.686 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/bonding/bonding.ko 2024-03-04T09:13:37.687 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/dsa 2024-03-04T09:13:37.687 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/dsa/b53 2024-03-04T09:13:37.687 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/dsa/b53/b53_common.ko 2024-03-04T09:13:37.687 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/dsa/bcm-sf2.ko 2024-03-04T09:13:37.687 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/dsa/mv88e6060.ko 2024-03-04T09:13:37.687 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/dsa/mv88e6xxx 2024-03-04T09:13:37.687 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-03-04T09:13:37.687 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/dummy.ko 2024-03-04T09:13:37.687 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/eql.ko 2024-03-04T09:13:37.687 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet 2024-03-04T09:13:37.687 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/3com 2024-03-04T09:13:37.687 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-03-04T09:13:37.687 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-03-04T09:13:37.688 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-03-04T09:13:37.688 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-03-04T09:13:37.688 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/8390 2024-03-04T09:13:37.688 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/8390/8390.ko 2024-03-04T09:13:37.688 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-03-04T09:13:37.688 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-03-04T09:13:37.688 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-03-04T09:13:37.688 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/adaptec 2024-03-04T09:13:37.688 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-03-04T09:13:37.688 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/agere 2024-03-04T09:13:37.688 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/agere/et131x.ko 2024-03-04T09:13:37.688 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/alteon 2024-03-04T09:13:37.688 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-03-04T09:13:37.688 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/altera 2024-03-04T09:13:37.688 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-03-04T09:13:37.689 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/amd 2024-03-04T09:13:37.689 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-03-04T09:13:37.689 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-03-04T09:13:37.689 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-03-04T09:13:37.689 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/atheros 2024-03-04T09:13:37.689 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/atheros/alx 2024-03-04T09:13:37.689 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-03-04T09:13:37.689 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/atheros/atl1c 2024-03-04T09:13:37.689 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-03-04T09:13:37.689 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/atheros/atl1e 2024-03-04T09:13:37.689 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-03-04T09:13:37.689 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/atheros/atlx 2024-03-04T09:13:37.689 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-03-04T09:13:37.689 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-03-04T09:13:37.690 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/broadcom 2024-03-04T09:13:37.690 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-03-04T09:13:37.690 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-03-04T09:13:37.690 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-03-04T09:13:37.690 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-03-04T09:13:37.690 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-03-04T09:13:37.690 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/broadcom/genet 2024-03-04T09:13:37.690 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-03-04T09:13:37.690 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-03-04T09:13:37.690 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/brocade 2024-03-04T09:13:37.690 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/brocade/bna 2024-03-04T09:13:37.690 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-03-04T09:13:37.690 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/chelsio 2024-03-04T09:13:37.691 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/chelsio/cxgb 2024-03-04T09:13:37.691 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-03-04T09:13:37.691 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-03-04T09:13:37.691 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-03-04T09:13:37.691 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-03-04T09:13:37.691 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-03-04T09:13:37.692 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-03-04T09:13:37.692 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-03-04T09:13:37.692 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-03-04T09:13:37.692 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-03-04T09:13:37.692 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/cisco 2024-03-04T09:13:37.692 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/cisco/enic 2024-03-04T09:13:37.692 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-03-04T09:13:37.692 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/dec 2024-03-04T09:13:37.692 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/dec/tulip 2024-03-04T09:13:37.692 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-03-04T09:13:37.692 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-03-04T09:13:37.692 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-03-04T09:13:37.693 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-03-04T09:13:37.693 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-03-04T09:13:37.693 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-03-04T09:13:37.693 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/dlink 2024-03-04T09:13:37.693 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-03-04T09:13:37.693 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-03-04T09:13:37.693 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/dnet.ko 2024-03-04T09:13:37.693 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/emulex 2024-03-04T09:13:37.693 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/emulex/benet 2024-03-04T09:13:37.693 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-03-04T09:13:37.693 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/ethoc.ko 2024-03-04T09:13:37.693 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/fealnx.ko 2024-03-04T09:13:37.693 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/intel 2024-03-04T09:13:37.693 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/intel/e100.ko 2024-03-04T09:13:37.694 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/intel/e1000 2024-03-04T09:13:37.694 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-03-04T09:13:37.694 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/intel/e1000e 2024-03-04T09:13:37.694 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-03-04T09:13:37.694 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/intel/fm10k 2024-03-04T09:13:37.694 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-03-04T09:13:37.694 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/intel/i40e 2024-03-04T09:13:37.694 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-03-04T09:13:37.694 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/intel/iavf 2024-03-04T09:13:37.694 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-03-04T09:13:37.694 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/intel/igb 2024-03-04T09:13:37.694 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-03-04T09:13:37.694 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/intel/igbvf 2024-03-04T09:13:37.694 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-03-04T09:13:37.694 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/intel/ixgbe 2024-03-04T09:13:37.695 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-03-04T09:13:37.695 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/intel/ixgbevf 2024-03-04T09:13:37.695 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-03-04T09:13:37.695 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/jme.ko 2024-03-04T09:13:37.695 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/marvell 2024-03-04T09:13:37.695 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-03-04T09:13:37.695 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/marvell/skge.ko 2024-03-04T09:13:37.695 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-03-04T09:13:37.695 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/mellanox 2024-03-04T09:13:37.695 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/mellanox/mlx4 2024-03-04T09:13:37.695 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-03-04T09:13:37.696 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-03-04T09:13:37.696 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/mellanox/mlx5 2024-03-04T09:13:37.696 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-03-04T09:13:37.696 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-03-04T09:13:37.696 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/micrel 2024-03-04T09:13:37.696 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-03-04T09:13:37.696 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/myricom 2024-03-04T09:13:37.696 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/myricom/myri10ge 2024-03-04T09:13:37.696 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-03-04T09:13:37.696 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/natsemi 2024-03-04T09:13:37.696 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-03-04T09:13:37.696 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-03-04T09:13:37.696 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/neterion 2024-03-04T09:13:37.697 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-03-04T09:13:37.697 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/nvidia 2024-03-04T09:13:37.697 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-03-04T09:13:37.697 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/packetengines 2024-03-04T09:13:37.697 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-03-04T09:13:37.697 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-03-04T09:13:37.697 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/qlogic 2024-03-04T09:13:37.697 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/qlogic/netxen 2024-03-04T09:13:37.697 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-03-04T09:13:37.697 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-03-04T09:13:37.697 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-03-04T09:13:37.697 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-03-04T09:13:37.697 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/rdc 2024-03-04T09:13:37.697 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-03-04T09:13:37.697 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/realtek 2024-03-04T09:13:37.697 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-03-04T09:13:37.698 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-03-04T09:13:37.698 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/realtek/atp.ko 2024-03-04T09:13:37.698 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-03-04T09:13:37.698 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/rocker 2024-03-04T09:13:37.698 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-03-04T09:13:37.698 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/sfc 2024-03-04T09:13:37.698 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-03-04T09:13:37.698 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/silan 2024-03-04T09:13:37.698 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-03-04T09:13:37.698 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/sis 2024-03-04T09:13:37.698 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/sis/sis190.ko 2024-03-04T09:13:37.698 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/sis/sis900.ko 2024-03-04T09:13:37.698 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/smsc 2024-03-04T09:13:37.699 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-03-04T09:13:37.699 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-03-04T09:13:37.699 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-03-04T09:13:37.699 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-03-04T09:13:37.699 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/stmicro 2024-03-04T09:13:37.699 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/stmicro/stmmac 2024-03-04T09:13:37.699 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-03-04T09:13:37.699 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-03-04T09:13:37.699 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/sun 2024-03-04T09:13:37.699 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/sun/cassini.ko 2024-03-04T09:13:37.699 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/sun/niu.ko 2024-03-04T09:13:37.699 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/sun/sungem.ko 2024-03-04T09:13:37.699 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-03-04T09:13:37.699 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/tehuti 2024-03-04T09:13:37.699 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-03-04T09:13:37.700 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/ti 2024-03-04T09:13:37.700 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/ti/tlan.ko 2024-03-04T09:13:37.700 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/via 2024-03-04T09:13:37.700 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-03-04T09:13:37.700 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-03-04T09:13:37.700 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/wiznet 2024-03-04T09:13:37.700 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-03-04T09:13:37.700 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-03-04T09:13:37.700 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/xircom 2024-03-04T09:13:37.700 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-03-04T09:13:37.700 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/geneve.ko 2024-03-04T09:13:37.700 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/hyperv 2024-03-04T09:13:37.700 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-03-04T09:13:37.700 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ieee802154 2024-03-04T09:13:37.701 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ieee802154/fakelb.ko 2024-03-04T09:13:37.701 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ifb.ko 2024-03-04T09:13:37.701 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ipvlan 2024-03-04T09:13:37.701 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ipvlan/ipvlan.ko 2024-03-04T09:13:37.701 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/macvlan.ko 2024-03-04T09:13:37.701 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/macvtap.ko 2024-03-04T09:13:37.701 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/mdio 2024-03-04T09:13:37.701 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/mdio.ko 2024-03-04T09:13:37.701 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-03-04T09:13:37.701 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-03-04T09:13:37.701 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/mdio/mdio-regmap.ko 2024-03-04T09:13:37.701 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/mii.ko 2024-03-04T09:13:37.701 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/net_failover.ko 2024-03-04T09:13:37.701 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/netconsole.ko 2024-03-04T09:13:37.701 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/nlmon.ko 2024-03-04T09:13:37.701 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/pcs 2024-03-04T09:13:37.702 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/pcs/pcs-lynx.ko 2024-03-04T09:13:37.702 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-03-04T09:13:37.702 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/phy 2024-03-04T09:13:37.702 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/phy/amd.ko 2024-03-04T09:13:37.702 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-03-04T09:13:37.702 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-03-04T09:13:37.702 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/phy/bcm7xxx.ko 2024-03-04T09:13:37.702 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/phy/bcm87xx.ko 2024-03-04T09:13:37.702 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/phy/broadcom.ko 2024-03-04T09:13:37.702 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/phy/cicada.ko 2024-03-04T09:13:37.702 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/phy/davicom.ko 2024-03-04T09:13:37.702 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/phy/dp83640.ko 2024-03-04T09:13:37.702 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/phy/et1011c.ko 2024-03-04T09:13:37.702 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/phy/icplus.ko 2024-03-04T09:13:37.703 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/phy/lxt.ko 2024-03-04T09:13:37.703 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/phy/marvell.ko 2024-03-04T09:13:37.703 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/phy/micrel.ko 2024-03-04T09:13:37.703 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/phy/national.ko 2024-03-04T09:13:37.703 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/phy/phylink.ko 2024-03-04T09:13:37.703 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/phy/qsemi.ko 2024-03-04T09:13:37.703 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/phy/realtek.ko 2024-03-04T09:13:37.703 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/phy/smsc.ko 2024-03-04T09:13:37.703 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/phy/ste10Xp.ko 2024-03-04T09:13:37.703 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/phy/vitesse.ko 2024-03-04T09:13:37.703 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ppp 2024-03-04T09:13:37.703 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ppp/bsd_comp.ko 2024-03-04T09:13:37.703 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ppp/ppp_async.ko 2024-03-04T09:13:37.704 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ppp/ppp_deflate.ko 2024-03-04T09:13:37.704 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ppp/ppp_generic.ko 2024-03-04T09:13:37.704 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ppp/ppp_mppe.ko 2024-03-04T09:13:37.704 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ppp/ppp_synctty.ko 2024-03-04T09:13:37.704 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ppp/pppoe.ko 2024-03-04T09:13:37.704 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ppp/pppox.ko 2024-03-04T09:13:37.704 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ppp/pptp.ko 2024-03-04T09:13:37.704 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/slip 2024-03-04T09:13:37.704 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/slip/slhc.ko 2024-03-04T09:13:37.704 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/slip/slip.ko 2024-03-04T09:13:37.704 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/sungem_phy.ko 2024-03-04T09:13:37.705 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/tap.ko 2024-03-04T09:13:37.705 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/team 2024-03-04T09:13:37.705 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/team/team.ko 2024-03-04T09:13:37.705 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/team/team_mode_activebackup.ko 2024-03-04T09:13:37.705 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/team/team_mode_broadcast.ko 2024-03-04T09:13:37.705 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-03-04T09:13:37.705 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/team/team_mode_random.ko 2024-03-04T09:13:37.705 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-03-04T09:13:37.705 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/tun.ko 2024-03-04T09:13:37.706 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/veth.ko 2024-03-04T09:13:37.706 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/virtio_net.ko 2024-03-04T09:13:37.706 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/vmxnet3 2024-03-04T09:13:37.706 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-03-04T09:13:37.706 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/vxlan 2024-03-04T09:13:37.706 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/vxlan/vxlan.ko 2024-03-04T09:13:37.706 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/xen-netback 2024-03-04T09:13:37.706 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/xen-netback/xen-netback.ko 2024-03-04T09:13:37.706 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/xen-netfront.ko 2024-03-04T09:13:37.706 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ntb 2024-03-04T09:13:37.706 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ntb/ntb.ko 2024-03-04T09:13:37.706 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/nvme 2024-03-04T09:13:37.706 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/nvme/host 2024-03-04T09:13:37.706 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/nvme/host/nvme-core.ko 2024-03-04T09:13:37.706 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/nvme/host/nvme.ko 2024-03-04T09:13:37.706 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/parport 2024-03-04T09:13:37.707 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/parport/parport.ko 2024-03-04T09:13:37.707 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/parport/parport_cs.ko 2024-03-04T09:13:37.707 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/parport/parport_pc.ko 2024-03-04T09:13:37.707 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/parport/parport_serial.ko 2024-03-04T09:13:37.707 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/pci 2024-03-04T09:13:37.707 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/pci/hotplug 2024-03-04T09:13:37.707 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-03-04T09:13:37.707 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/pci/pcie 2024-03-04T09:13:37.707 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/pci/pcie/aer_inject.ko 2024-03-04T09:13:37.708 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/pci/xen-pcifront.ko 2024-03-04T09:13:37.708 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/pcmcia 2024-03-04T09:13:37.708 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/pcmcia/i82092.ko 2024-03-04T09:13:37.708 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/pcmcia/pd6729.ko 2024-03-04T09:13:37.708 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/pcmcia/yenta_socket.ko 2024-03-04T09:13:37.708 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/pinctrl 2024-03-04T09:13:37.708 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/pinctrl/intel 2024-03-04T09:13:37.708 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-03-04T09:13:37.708 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/platform 2024-03-04T09:13:37.709 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/platform/x86 2024-03-04T09:13:37.709 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/platform/x86/acer-wmi.ko 2024-03-04T09:13:37.709 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/platform/x86/acerhdf.ko 2024-03-04T09:13:37.709 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/platform/x86/apple-gmux.ko 2024-03-04T09:13:37.709 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/platform/x86/asus-laptop.ko 2024-03-04T09:13:37.709 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-03-04T09:13:37.709 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/platform/x86/asus-wmi.ko 2024-03-04T09:13:37.709 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/platform/x86/classmate-laptop.ko 2024-03-04T09:13:37.709 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-03-04T09:13:37.709 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-03-04T09:13:37.709 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-03-04T09:13:37.709 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-03-04T09:13:37.709 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/platform/x86/hdaps.ko 2024-03-04T09:13:37.709 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/platform/x86/intel 2024-03-04T09:13:37.709 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-03-04T09:13:37.709 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/platform/x86/intel_ips.ko 2024-03-04T09:13:37.710 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/platform/x86/msi-wmi.ko 2024-03-04T09:13:37.710 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/platform/x86/mxm-wmi.ko 2024-03-04T09:13:37.710 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-03-04T09:13:37.710 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/platform/x86/samsung-laptop.ko 2024-03-04T09:13:37.710 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/platform/x86/samsung-q10.ko 2024-03-04T09:13:37.710 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/platform/x86/topstar-laptop.ko 2024-03-04T09:13:37.710 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-03-04T09:13:37.710 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/platform/x86/toshiba_haps.ko 2024-03-04T09:13:37.710 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/platform/x86/wmi-bmof.ko 2024-03-04T09:13:37.710 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/platform/x86/wmi.ko 2024-03-04T09:13:37.710 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/powercap 2024-03-04T09:13:37.710 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/powercap/intel_rapl_common.ko 2024-03-04T09:13:37.710 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/powercap/intel_rapl_msr.ko 2024-03-04T09:13:37.710 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/pps 2024-03-04T09:13:37.710 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/pps/clients 2024-03-04T09:13:37.711 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/pps/clients/pps-gpio.ko 2024-03-04T09:13:37.711 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/pps/clients/pps-ldisc.ko 2024-03-04T09:13:37.711 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/pps/clients/pps_parport.ko 2024-03-04T09:13:37.711 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/pps/pps_core.ko 2024-03-04T09:13:37.711 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ptp 2024-03-04T09:13:37.711 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ptp/ptp.ko 2024-03-04T09:13:37.711 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ptp/ptp_kvm.ko 2024-03-04T09:13:37.711 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/rtc 2024-03-04T09:13:37.711 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/rtc/rtc-bq32k.ko 2024-03-04T09:13:37.711 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/rtc/rtc-ds1286.ko 2024-03-04T09:13:37.711 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/rtc/rtc-ds1307.ko 2024-03-04T09:13:37.711 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/rtc/rtc-ds1374.ko 2024-03-04T09:13:37.711 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/rtc/rtc-ds1511.ko 2024-03-04T09:13:37.711 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/rtc/rtc-ds1553.ko 2024-03-04T09:13:37.711 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/rtc/rtc-ds1672.ko 2024-03-04T09:13:37.712 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/rtc/rtc-ds1742.ko 2024-03-04T09:13:37.712 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/rtc/rtc-ds2404.ko 2024-03-04T09:13:37.712 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/rtc/rtc-ds3232.ko 2024-03-04T09:13:37.712 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/rtc/rtc-em3027.ko 2024-03-04T09:13:37.712 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/rtc/rtc-fm3130.ko 2024-03-04T09:13:37.712 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/rtc/rtc-isl12022.ko 2024-03-04T09:13:37.712 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/rtc/rtc-isl1208.ko 2024-03-04T09:13:37.712 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/rtc/rtc-m41t80.ko 2024-03-04T09:13:37.712 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/rtc/rtc-m48t35.ko 2024-03-04T09:13:37.712 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/rtc/rtc-m48t59.ko 2024-03-04T09:13:37.712 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/rtc/rtc-max6900.ko 2024-03-04T09:13:37.712 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/rtc/rtc-msm6242.ko 2024-03-04T09:13:37.712 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/rtc/rtc-pcf2127.ko 2024-03-04T09:13:37.713 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/rtc/rtc-pcf85063.ko 2024-03-04T09:13:37.713 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/rtc/rtc-pcf8523.ko 2024-03-04T09:13:37.713 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/rtc/rtc-pcf8563.ko 2024-03-04T09:13:37.713 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/rtc/rtc-pcf8583.ko 2024-03-04T09:13:37.713 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/rtc/rtc-rp5c01.ko 2024-03-04T09:13:37.713 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/rtc/rtc-rs5c372.ko 2024-03-04T09:13:37.713 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-03-04T09:13:37.713 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/rtc/rtc-rx8025.ko 2024-03-04T09:13:37.713 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/rtc/rtc-rx8581.ko 2024-03-04T09:13:37.713 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-03-04T09:13:37.713 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/rtc/rtc-x1205.ko 2024-03-04T09:13:37.713 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi 2024-03-04T09:13:37.713 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/3w-9xxx.ko 2024-03-04T09:13:37.713 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/3w-sas.ko 2024-03-04T09:13:37.714 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/3w-xxxx.ko 2024-03-04T09:13:37.714 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/BusLogic.ko 2024-03-04T09:13:37.714 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/a100u2w.ko 2024-03-04T09:13:37.714 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/aacraid 2024-03-04T09:13:37.714 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/aacraid/aacraid.ko 2024-03-04T09:13:37.714 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/advansys.ko 2024-03-04T09:13:37.714 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/aic7xxx 2024-03-04T09:13:37.714 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-03-04T09:13:37.714 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-03-04T09:13:37.714 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/am53c974.ko 2024-03-04T09:13:37.714 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/arcmsr 2024-03-04T09:13:37.714 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-03-04T09:13:37.714 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/atp870u.ko 2024-03-04T09:13:37.714 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/be2iscsi 2024-03-04T09:13:37.714 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-03-04T09:13:37.715 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/bfa 2024-03-04T09:13:37.715 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/bfa/bfa.ko 2024-03-04T09:13:37.715 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/bnx2fc 2024-03-04T09:13:37.715 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-03-04T09:13:37.715 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/bnx2i 2024-03-04T09:13:37.715 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-03-04T09:13:37.715 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/ch.ko 2024-03-04T09:13:37.715 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/csiostor 2024-03-04T09:13:37.715 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/csiostor/csiostor.ko 2024-03-04T09:13:37.715 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/cxgbi 2024-03-04T09:13:37.715 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/cxgbi/cxgb3i 2024-03-04T09:13:37.715 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-03-04T09:13:37.715 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/cxgbi/cxgb4i 2024-03-04T09:13:37.715 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-03-04T09:13:37.715 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-03-04T09:13:37.716 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/dc395x.ko 2024-03-04T09:13:37.716 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/device_handler 2024-03-04T09:13:37.716 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-03-04T09:13:37.716 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-03-04T09:13:37.716 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-03-04T09:13:37.716 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-03-04T09:13:37.716 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/dmx3191d.ko 2024-03-04T09:13:37.716 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/esas2r 2024-03-04T09:13:37.716 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/esas2r/esas2r.ko 2024-03-04T09:13:37.716 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/esp_scsi.ko 2024-03-04T09:13:37.716 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/fcoe 2024-03-04T09:13:37.716 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/fcoe/fcoe.ko 2024-03-04T09:13:37.716 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-03-04T09:13:37.717 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/fnic 2024-03-04T09:13:37.717 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/fnic/fnic.ko 2024-03-04T09:13:37.717 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/hpsa.ko 2024-03-04T09:13:37.717 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/hptiop.ko 2024-03-04T09:13:37.717 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/hv_storvsc.ko 2024-03-04T09:13:37.717 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/initio.ko 2024-03-04T09:13:37.717 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/ipr.ko 2024-03-04T09:13:37.717 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/ips.ko 2024-03-04T09:13:37.717 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/isci 2024-03-04T09:13:37.717 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/isci/isci.ko 2024-03-04T09:13:37.717 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-03-04T09:13:37.717 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/iscsi_tcp.ko 2024-03-04T09:13:37.717 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/libfc 2024-03-04T09:13:37.717 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/libfc/libfc.ko 2024-03-04T09:13:37.717 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/libiscsi.ko 2024-03-04T09:13:37.718 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/libiscsi_tcp.ko 2024-03-04T09:13:37.718 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/libsas 2024-03-04T09:13:37.718 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/libsas/libsas.ko 2024-03-04T09:13:37.718 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/lpfc 2024-03-04T09:13:37.718 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/lpfc/lpfc.ko 2024-03-04T09:13:37.718 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/megaraid 2024-03-04T09:13:37.718 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/megaraid.ko 2024-03-04T09:13:37.718 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-03-04T09:13:37.718 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-03-04T09:13:37.718 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-03-04T09:13:37.718 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/mpt3sas 2024-03-04T09:13:37.718 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-03-04T09:13:37.718 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/mvsas 2024-03-04T09:13:37.718 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/mvsas/mvsas.ko 2024-03-04T09:13:37.718 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/mvumi.ko 2024-03-04T09:13:37.719 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/pm8001 2024-03-04T09:13:37.719 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-03-04T09:13:37.719 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/pmcraid.ko 2024-03-04T09:13:37.719 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/qla1280.ko 2024-03-04T09:13:37.719 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/qla2xxx 2024-03-04T09:13:37.719 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-03-04T09:13:37.719 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-03-04T09:13:37.719 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/qla4xxx 2024-03-04T09:13:37.719 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-03-04T09:13:37.719 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/raid_class.ko 2024-03-04T09:13:37.719 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/scsi_debug.ko 2024-03-04T09:13:37.719 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/scsi_transport_fc.ko 2024-03-04T09:13:37.719 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-03-04T09:13:37.719 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/scsi_transport_sas.ko 2024-03-04T09:13:37.719 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/scsi_transport_spi.ko 2024-03-04T09:13:37.720 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/scsi_transport_srp.ko 2024-03-04T09:13:37.720 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/ses.ko 2024-03-04T09:13:37.720 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/st.ko 2024-03-04T09:13:37.720 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/stex.ko 2024-03-04T09:13:37.720 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/sym53c8xx_2 2024-03-04T09:13:37.720 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-03-04T09:13:37.720 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/virtio_scsi.ko 2024-03-04T09:13:37.720 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/vmw_pvscsi.ko 2024-03-04T09:13:37.720 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/wd719x.ko 2024-03-04T09:13:37.720 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ssb 2024-03-04T09:13:37.720 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ssb/ssb.ko 2024-03-04T09:13:37.720 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/target 2024-03-04T09:13:37.720 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/target/iscsi 2024-03-04T09:13:37.721 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-03-04T09:13:37.721 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/target/loopback 2024-03-04T09:13:37.721 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/target/loopback/tcm_loop.ko 2024-03-04T09:13:37.721 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/target/target_core_file.ko 2024-03-04T09:13:37.721 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/target/target_core_iblock.ko 2024-03-04T09:13:37.721 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/target/target_core_mod.ko 2024-03-04T09:13:37.721 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/target/target_core_pscsi.ko 2024-03-04T09:13:37.721 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/target/target_core_user.ko 2024-03-04T09:13:37.721 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/target/tcm_fc 2024-03-04T09:13:37.721 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-03-04T09:13:37.721 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/thermal 2024-03-04T09:13:37.721 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/thermal/intel 2024-03-04T09:13:37.721 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/thermal/intel/int340x_thermal 2024-03-04T09:13:37.721 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-03-04T09:13:37.721 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-03-04T09:13:37.722 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-03-04T09:13:37.722 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-03-04T09:13:37.722 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-03-04T09:13:37.722 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-03-04T09:13:37.722 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-03-04T09:13:37.722 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-03-04T09:13:37.722 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-03-04T09:13:37.722 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-03-04T09:13:37.722 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-03-04T09:13:37.722 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-03-04T09:13:37.722 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-03-04T09:13:37.722 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-03-04T09:13:37.722 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-03-04T09:13:37.722 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-03-04T09:13:37.723 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-03-04T09:13:37.723 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-03-04T09:13:37.723 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/tty 2024-03-04T09:13:37.723 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/tty/ipwireless 2024-03-04T09:13:37.723 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-03-04T09:13:37.723 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/tty/n_gsm.ko 2024-03-04T09:13:37.723 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/tty/n_hdlc.ko 2024-03-04T09:13:37.723 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/tty/nozomi.ko 2024-03-04T09:13:37.723 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/tty/serial 2024-03-04T09:13:37.723 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/tty/serial/8250 2024-03-04T09:13:37.723 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-03-04T09:13:37.723 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/tty/serial/arc_uart.ko 2024-03-04T09:13:37.723 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/tty/serial/jsm 2024-03-04T09:13:37.724 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/tty/serial/jsm/jsm.ko 2024-03-04T09:13:37.724 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ufs 2024-03-04T09:13:37.724 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ufs/core 2024-03-04T09:13:37.724 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ufs/core/ufshcd-core.ko 2024-03-04T09:13:37.724 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ufs/host 2024-03-04T09:13:37.724 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-03-04T09:13:37.724 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/uio 2024-03-04T09:13:37.724 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/uio/uio.ko 2024-03-04T09:13:37.724 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/uio/uio_aec.ko 2024-03-04T09:13:37.724 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/uio/uio_cif.ko 2024-03-04T09:13:37.724 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/uio/uio_pci_generic.ko 2024-03-04T09:13:37.724 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/uio/uio_sercos3.ko 2024-03-04T09:13:37.724 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb 2024-03-04T09:13:37.725 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/class 2024-03-04T09:13:37.725 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/class/cdc-acm.ko 2024-03-04T09:13:37.725 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/class/cdc-wdm.ko 2024-03-04T09:13:37.725 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/class/usblp.ko 2024-03-04T09:13:37.725 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/class/usbtmc.ko 2024-03-04T09:13:37.725 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/host 2024-03-04T09:13:37.725 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/host/sl811-hcd.ko 2024-03-04T09:13:37.725 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/image 2024-03-04T09:13:37.725 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/image/mdc800.ko 2024-03-04T09:13:37.725 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/image/microtek.ko 2024-03-04T09:13:37.725 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/misc 2024-03-04T09:13:37.726 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/misc/adutux.ko 2024-03-04T09:13:37.726 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/misc/appledisplay.ko 2024-03-04T09:13:37.726 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/misc/emi26.ko 2024-03-04T09:13:37.726 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/misc/emi62.ko 2024-03-04T09:13:37.726 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/misc/ezusb.ko 2024-03-04T09:13:37.726 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/misc/idmouse.ko 2024-03-04T09:13:37.726 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/misc/iowarrior.ko 2024-03-04T09:13:37.726 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/misc/isight_firmware.ko 2024-03-04T09:13:37.726 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/misc/ldusb.ko 2024-03-04T09:13:37.726 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/misc/legousbtower.ko 2024-03-04T09:13:37.726 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/misc/sisusbvga 2024-03-04T09:13:37.726 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-03-04T09:13:37.726 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/misc/trancevibrator.ko 2024-03-04T09:13:37.727 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/misc/usb3503.ko 2024-03-04T09:13:37.727 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/misc/usblcd.ko 2024-03-04T09:13:37.727 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/misc/usbsevseg.ko 2024-03-04T09:13:37.727 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/misc/uss720.ko 2024-03-04T09:13:37.727 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/misc/yurex.ko 2024-03-04T09:13:37.727 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/phy 2024-03-04T09:13:37.727 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/phy/phy-generic.ko 2024-03-04T09:13:37.727 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial 2024-03-04T09:13:37.727 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/aircable.ko 2024-03-04T09:13:37.727 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/ark3116.ko 2024-03-04T09:13:37.727 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/belkin_sa.ko 2024-03-04T09:13:37.727 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/ch341.ko 2024-03-04T09:13:37.727 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/cp210x.ko 2024-03-04T09:13:37.727 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/cyberjack.ko 2024-03-04T09:13:37.728 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/cypress_m8.ko 2024-03-04T09:13:37.728 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/digi_acceleport.ko 2024-03-04T09:13:37.728 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/empeg.ko 2024-03-04T09:13:37.728 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/ftdi_sio.ko 2024-03-04T09:13:37.728 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/garmin_gps.ko 2024-03-04T09:13:37.728 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/io_edgeport.ko 2024-03-04T09:13:37.728 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/io_ti.ko 2024-03-04T09:13:37.728 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/ipaq.ko 2024-03-04T09:13:37.728 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/ipw.ko 2024-03-04T09:13:37.728 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/ir-usb.ko 2024-03-04T09:13:37.728 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-03-04T09:13:37.729 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/keyspan.ko 2024-03-04T09:13:37.729 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/keyspan_pda.ko 2024-03-04T09:13:37.729 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/kl5kusb105.ko 2024-03-04T09:13:37.729 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/kobil_sct.ko 2024-03-04T09:13:37.729 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/mct_u232.ko 2024-03-04T09:13:37.729 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/mos7720.ko 2024-03-04T09:13:37.729 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/mos7840.ko 2024-03-04T09:13:37.729 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/navman.ko 2024-03-04T09:13:37.729 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/omninet.ko 2024-03-04T09:13:37.729 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/opticon.ko 2024-03-04T09:13:37.729 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/option.ko 2024-03-04T09:13:37.729 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/oti6858.ko 2024-03-04T09:13:37.729 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/pl2303.ko 2024-03-04T09:13:37.729 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/qcaux.ko 2024-03-04T09:13:37.730 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/qcserial.ko 2024-03-04T09:13:37.730 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/quatech2.ko 2024-03-04T09:13:37.730 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/safe_serial.ko 2024-03-04T09:13:37.730 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/sierra.ko 2024-03-04T09:13:37.730 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/spcp8x5.ko 2024-03-04T09:13:37.730 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/ssu100.ko 2024-03-04T09:13:37.730 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/symbolserial.ko 2024-03-04T09:13:37.730 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-03-04T09:13:37.730 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-03-04T09:13:37.730 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/usb_debug.ko 2024-03-04T09:13:37.730 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/usb_wwan.ko 2024-03-04T09:13:37.731 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/visor.ko 2024-03-04T09:13:37.731 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/whiteheat.ko 2024-03-04T09:13:37.731 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/xsens_mt.ko 2024-03-04T09:13:37.731 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/storage 2024-03-04T09:13:37.731 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/storage/uas.ko 2024-03-04T09:13:37.731 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/storage/ums-alauda.ko 2024-03-04T09:13:37.731 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/storage/ums-cypress.ko 2024-03-04T09:13:37.731 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/storage/ums-datafab.ko 2024-03-04T09:13:37.731 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-03-04T09:13:37.731 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/storage/ums-freecom.ko 2024-03-04T09:13:37.731 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/storage/ums-isd200.ko 2024-03-04T09:13:37.731 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-03-04T09:13:37.731 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/storage/ums-karma.ko 2024-03-04T09:13:37.732 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/storage/ums-onetouch.ko 2024-03-04T09:13:37.732 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/storage/ums-realtek.ko 2024-03-04T09:13:37.732 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/storage/ums-sddr09.ko 2024-03-04T09:13:37.732 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/storage/ums-sddr55.ko 2024-03-04T09:13:37.732 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/storage/ums-usbat.ko 2024-03-04T09:13:37.732 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/storage/usb-storage.ko 2024-03-04T09:13:37.732 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/usbip 2024-03-04T09:13:37.732 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/usbip/usbip-core.ko 2024-03-04T09:13:37.732 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/usbip/usbip-host.ko 2024-03-04T09:13:37.732 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-03-04T09:13:37.732 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/vfio 2024-03-04T09:13:37.732 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/vfio/pci 2024-03-04T09:13:37.732 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-03-04T09:13:37.733 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/vfio/pci/vfio-pci.ko 2024-03-04T09:13:37.733 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/vfio/vfio.ko 2024-03-04T09:13:37.733 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-03-04T09:13:37.733 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/vhost 2024-03-04T09:13:37.733 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/vhost/vhost.ko 2024-03-04T09:13:37.733 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/vhost/vhost_iotlb.ko 2024-03-04T09:13:37.733 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/vhost/vhost_net.ko 2024-03-04T09:13:37.733 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/vhost/vhost_scsi.ko 2024-03-04T09:13:37.733 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/video 2024-03-04T09:13:37.733 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/video/backlight 2024-03-04T09:13:37.733 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/video/backlight/apple_bl.ko 2024-03-04T09:13:37.733 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/video/backlight/lcd.ko 2024-03-04T09:13:37.733 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/video/backlight/platform_lcd.ko 2024-03-04T09:13:37.733 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/virtio 2024-03-04T09:13:37.733 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/virtio/virtio.ko 2024-03-04T09:13:37.733 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/virtio/virtio_balloon.ko 2024-03-04T09:13:37.734 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/virtio/virtio_mmio.ko 2024-03-04T09:13:37.734 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/virtio/virtio_pci.ko 2024-03-04T09:13:37.734 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-03-04T09:13:37.734 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-03-04T09:13:37.734 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/virtio/virtio_ring.ko 2024-03-04T09:13:37.734 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/w1 2024-03-04T09:13:37.734 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/w1/masters 2024-03-04T09:13:37.734 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/w1/masters/ds2482.ko 2024-03-04T09:13:37.734 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/w1/masters/ds2490.ko 2024-03-04T09:13:37.734 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/w1/slaves 2024-03-04T09:13:37.734 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-03-04T09:13:37.734 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-03-04T09:13:37.734 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-03-04T09:13:37.734 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-03-04T09:13:37.734 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-03-04T09:13:37.734 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-03-04T09:13:37.735 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-03-04T09:13:37.735 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-03-04T09:13:37.735 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-03-04T09:13:37.735 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/w1/slaves/w1_smem.ko 2024-03-04T09:13:37.735 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/w1/slaves/w1_therm.ko 2024-03-04T09:13:37.735 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/w1/wire.ko 2024-03-04T09:13:37.735 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/watchdog 2024-03-04T09:13:37.735 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/watchdog/alim1535_wdt.ko 2024-03-04T09:13:37.735 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/watchdog/alim7101_wdt.ko 2024-03-04T09:13:37.735 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/watchdog/f71808e_wdt.ko 2024-03-04T09:13:37.735 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/watchdog/hpwdt.ko 2024-03-04T09:13:37.735 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/watchdog/i6300esb.ko 2024-03-04T09:13:37.735 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-03-04T09:13:37.735 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/watchdog/iTCO_wdt.ko 2024-03-04T09:13:37.735 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/watchdog/ib700wdt.ko 2024-03-04T09:13:37.736 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/watchdog/ibmasr.ko 2024-03-04T09:13:37.736 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-03-04T09:13:37.736 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/watchdog/it8712f_wdt.ko 2024-03-04T09:13:37.736 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/watchdog/it87_wdt.ko 2024-03-04T09:13:37.736 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/watchdog/machzwd.ko 2024-03-04T09:13:37.736 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/watchdog/nv_tco.ko 2024-03-04T09:13:37.736 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/watchdog/pcwd_pci.ko 2024-03-04T09:13:37.736 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/watchdog/pcwd_usb.ko 2024-03-04T09:13:37.736 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-03-04T09:13:37.736 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/watchdog/sch311x_wdt.ko 2024-03-04T09:13:37.736 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/watchdog/softdog.ko 2024-03-04T09:13:37.736 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/watchdog/sp5100_tco.ko 2024-03-04T09:13:37.736 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/watchdog/via_wdt.ko 2024-03-04T09:13:37.737 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-03-04T09:13:37.737 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/watchdog/w83877f_wdt.ko 2024-03-04T09:13:37.737 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/watchdog/w83977f_wdt.ko 2024-03-04T09:13:37.737 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/watchdog/wdt_pci.ko 2024-03-04T09:13:37.737 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/watchdog/xen_wdt.ko 2024-03-04T09:13:37.737 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/xen 2024-03-04T09:13:37.737 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/xen/xen-acpi-processor.ko 2024-03-04T09:13:37.737 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/xen/xen-evtchn.ko 2024-03-04T09:13:37.737 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/xen/xen-gntalloc.ko 2024-03-04T09:13:37.737 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/xen/xen-gntdev.ko 2024-03-04T09:13:37.737 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/xen/xen-pciback 2024-03-04T09:13:37.737 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-03-04T09:13:37.737 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/xen/xen-privcmd.ko 2024-03-04T09:13:37.737 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/xen/xenfs 2024-03-04T09:13:37.737 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/xen/xenfs/xenfs.ko 2024-03-04T09:13:37.738 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs 2024-03-04T09:13:37.738 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/9p 2024-03-04T09:13:37.738 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/9p/9p.ko 2024-03-04T09:13:37.738 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/affs 2024-03-04T09:13:37.738 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/affs/affs.ko 2024-03-04T09:13:37.738 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/befs 2024-03-04T09:13:37.738 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/befs/befs.ko 2024-03-04T09:13:37.738 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/binfmt_misc.ko 2024-03-04T09:13:37.738 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/btrfs 2024-03-04T09:13:37.738 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/btrfs/btrfs.ko 2024-03-04T09:13:37.738 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/cachefiles 2024-03-04T09:13:37.738 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/cachefiles/cachefiles.ko 2024-03-04T09:13:37.738 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/ceph 2024-03-04T09:13:37.738 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/ceph/ceph.ko 2024-03-04T09:13:37.738 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/coda 2024-03-04T09:13:37.738 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/coda/coda.ko 2024-03-04T09:13:37.739 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/cramfs 2024-03-04T09:13:37.739 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/cramfs/cramfs.ko 2024-03-04T09:13:37.739 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/dlm 2024-03-04T09:13:37.739 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/dlm/dlm.ko 2024-03-04T09:13:37.739 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/ecryptfs 2024-03-04T09:13:37.739 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/ecryptfs/ecryptfs.ko 2024-03-04T09:13:37.739 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/f2fs 2024-03-04T09:13:37.739 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/f2fs/f2fs.ko 2024-03-04T09:13:37.739 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/fat 2024-03-04T09:13:37.739 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/fat/fat.ko 2024-03-04T09:13:37.739 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/fat/msdos.ko 2024-03-04T09:13:37.739 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/fat/vfat.ko 2024-03-04T09:13:37.739 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/fuse 2024-03-04T09:13:37.739 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/fuse/cuse.ko 2024-03-04T09:13:37.739 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/fuse/fuse.ko 2024-03-04T09:13:37.739 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/gfs2 2024-03-04T09:13:37.739 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/gfs2/gfs2.ko 2024-03-04T09:13:37.740 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/hfs 2024-03-04T09:13:37.740 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/hfs/hfs.ko 2024-03-04T09:13:37.740 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/hfsplus 2024-03-04T09:13:37.740 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/hfsplus/hfsplus.ko 2024-03-04T09:13:37.740 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/isofs 2024-03-04T09:13:37.740 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/isofs/isofs.ko 2024-03-04T09:13:37.740 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/jfs 2024-03-04T09:13:37.740 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/jfs/jfs.ko 2024-03-04T09:13:37.740 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/lockd 2024-03-04T09:13:37.740 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/lockd/lockd.ko 2024-03-04T09:13:37.740 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/minix 2024-03-04T09:13:37.740 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/minix/minix.ko 2024-03-04T09:13:37.740 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/netfs 2024-03-04T09:13:37.741 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/netfs/netfs.ko 2024-03-04T09:13:37.741 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nfs 2024-03-04T09:13:37.741 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nfs/blocklayout 2024-03-04T09:13:37.741 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-03-04T09:13:37.741 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nfs/filelayout 2024-03-04T09:13:37.741 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-03-04T09:13:37.741 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nfs/flexfilelayout 2024-03-04T09:13:37.741 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-03-04T09:13:37.741 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nfs/nfs.ko 2024-03-04T09:13:37.741 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nfs/nfsv3.ko 2024-03-04T09:13:37.741 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nfs/nfsv4.ko 2024-03-04T09:13:37.741 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nfs_common 2024-03-04T09:13:37.741 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nfs_common/grace.ko 2024-03-04T09:13:37.741 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nfs_common/nfs_acl.ko 2024-03-04T09:13:37.741 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nfsd 2024-03-04T09:13:37.741 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nfsd/nfsd.ko 2024-03-04T09:13:37.742 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nilfs2 2024-03-04T09:13:37.742 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nilfs2/nilfs2.ko 2024-03-04T09:13:37.742 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls 2024-03-04T09:13:37.742 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/mac-celtic.ko 2024-03-04T09:13:37.742 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/mac-centeuro.ko 2024-03-04T09:13:37.742 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/mac-croatian.ko 2024-03-04T09:13:37.742 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/mac-cyrillic.ko 2024-03-04T09:13:37.742 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/mac-gaelic.ko 2024-03-04T09:13:37.742 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/mac-greek.ko 2024-03-04T09:13:37.742 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/mac-iceland.ko 2024-03-04T09:13:37.742 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/mac-inuit.ko 2024-03-04T09:13:37.742 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/mac-roman.ko 2024-03-04T09:13:37.742 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/mac-romanian.ko 2024-03-04T09:13:37.742 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/mac-turkish.ko 2024-03-04T09:13:37.742 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/nls_cp1250.ko 2024-03-04T09:13:37.742 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/nls_cp1251.ko 2024-03-04T09:13:37.743 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/nls_cp1255.ko 2024-03-04T09:13:37.743 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/nls_cp737.ko 2024-03-04T09:13:37.743 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/nls_cp775.ko 2024-03-04T09:13:37.743 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/nls_cp850.ko 2024-03-04T09:13:37.743 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/nls_cp852.ko 2024-03-04T09:13:37.743 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/nls_cp855.ko 2024-03-04T09:13:37.743 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/nls_cp857.ko 2024-03-04T09:13:37.743 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/nls_cp860.ko 2024-03-04T09:13:37.743 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/nls_cp861.ko 2024-03-04T09:13:37.743 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/nls_cp862.ko 2024-03-04T09:13:37.743 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/nls_cp863.ko 2024-03-04T09:13:37.743 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/nls_cp864.ko 2024-03-04T09:13:37.743 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/nls_cp865.ko 2024-03-04T09:13:37.743 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/nls_cp866.ko 2024-03-04T09:13:37.743 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/nls_cp869.ko 2024-03-04T09:13:37.743 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/nls_cp874.ko 2024-03-04T09:13:37.744 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/nls_cp932.ko 2024-03-04T09:13:37.744 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/nls_cp936.ko 2024-03-04T09:13:37.744 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/nls_cp949.ko 2024-03-04T09:13:37.744 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/nls_cp950.ko 2024-03-04T09:13:37.744 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/nls_euc-jp.ko 2024-03-04T09:13:37.744 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/nls_iso8859-1.ko 2024-03-04T09:13:37.744 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/nls_iso8859-13.ko 2024-03-04T09:13:37.744 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/nls_iso8859-14.ko 2024-03-04T09:13:37.744 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/nls_iso8859-15.ko 2024-03-04T09:13:37.744 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/nls_iso8859-2.ko 2024-03-04T09:13:37.744 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/nls_iso8859-3.ko 2024-03-04T09:13:37.744 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/nls_iso8859-4.ko 2024-03-04T09:13:37.744 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/nls_iso8859-5.ko 2024-03-04T09:13:37.745 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/nls_iso8859-6.ko 2024-03-04T09:13:37.745 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/nls_iso8859-7.ko 2024-03-04T09:13:37.745 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/nls_iso8859-9.ko 2024-03-04T09:13:37.745 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/nls_koi8-r.ko 2024-03-04T09:13:37.745 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/nls_koi8-ru.ko 2024-03-04T09:13:37.745 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/nls_koi8-u.ko 2024-03-04T09:13:37.745 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/nls_ucs2_utils.ko 2024-03-04T09:13:37.745 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/nls_utf8.ko 2024-03-04T09:13:37.745 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/ocfs2 2024-03-04T09:13:37.745 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/ocfs2/cluster 2024-03-04T09:13:37.745 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-03-04T09:13:37.745 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/ocfs2/dlm 2024-03-04T09:13:37.745 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-03-04T09:13:37.745 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/ocfs2/dlmfs 2024-03-04T09:13:37.745 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-03-04T09:13:37.746 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/ocfs2/ocfs2.ko 2024-03-04T09:13:37.746 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-03-04T09:13:37.746 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-03-04T09:13:37.746 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-03-04T09:13:37.746 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/overlayfs 2024-03-04T09:13:37.746 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/overlayfs/overlay.ko 2024-03-04T09:13:37.746 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/pstore 2024-03-04T09:13:37.746 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/pstore/ramoops.ko 2024-03-04T09:13:37.746 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/reiserfs 2024-03-04T09:13:37.746 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/reiserfs/reiserfs.ko 2024-03-04T09:13:37.746 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/romfs 2024-03-04T09:13:37.746 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/romfs/romfs.ko 2024-03-04T09:13:37.746 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/smb 2024-03-04T09:13:37.746 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/smb/client 2024-03-04T09:13:37.746 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/smb/client/cifs.ko 2024-03-04T09:13:37.747 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/smb/common 2024-03-04T09:13:37.747 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/smb/common/cifs_arc4.ko 2024-03-04T09:13:37.747 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/smb/common/cifs_md4.ko 2024-03-04T09:13:37.747 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/squashfs 2024-03-04T09:13:37.747 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/squashfs/squashfs.ko 2024-03-04T09:13:37.747 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/sysv 2024-03-04T09:13:37.747 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/sysv/sysv.ko 2024-03-04T09:13:37.747 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/udf 2024-03-04T09:13:37.747 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/udf/udf.ko 2024-03-04T09:13:37.747 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/ufs 2024-03-04T09:13:37.747 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/ufs/ufs.ko 2024-03-04T09:13:37.747 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/xfs 2024-03-04T09:13:37.747 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/xfs/xfs.ko 2024-03-04T09:13:37.747 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/kernel 2024-03-04T09:13:37.747 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/kernel/rcu 2024-03-04T09:13:37.747 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/kernel/rcu/rcutorture.ko 2024-03-04T09:13:37.748 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/kernel/torture.ko 2024-03-04T09:13:37.748 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/kernel/trace 2024-03-04T09:13:37.748 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-03-04T09:13:37.748 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/lib 2024-03-04T09:13:37.748 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/lib/asn1_encoder.ko 2024-03-04T09:13:37.748 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/lib/crc-itu-t.ko 2024-03-04T09:13:37.748 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/lib/crc8.ko 2024-03-04T09:13:37.748 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/lib/crypto 2024-03-04T09:13:37.748 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/lib/crypto/libarc4.ko 2024-03-04T09:13:37.748 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/lib/crypto/libdes.ko 2024-03-04T09:13:37.748 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/lib/libcrc32c.ko 2024-03-04T09:13:37.748 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/lib/lru_cache.ko 2024-03-04T09:13:37.748 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/lib/lz4 2024-03-04T09:13:37.749 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/lib/lz4/lz4_compress.ko 2024-03-04T09:13:37.749 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/lib/lz4/lz4hc_compress.ko 2024-03-04T09:13:37.749 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/lib/math 2024-03-04T09:13:37.749 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/lib/math/cordic.ko 2024-03-04T09:13:37.749 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/lib/raid6 2024-03-04T09:13:37.749 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/lib/raid6/raid6_pq.ko 2024-03-04T09:13:37.749 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/lib/reed_solomon 2024-03-04T09:13:37.749 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/lib/reed_solomon/reed_solomon.ko 2024-03-04T09:13:37.749 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/lib/ts_bm.ko 2024-03-04T09:13:37.749 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/lib/ts_fsm.ko 2024-03-04T09:13:37.749 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/lib/ts_kmp.ko 2024-03-04T09:13:37.749 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/lib/zstd 2024-03-04T09:13:37.749 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/lib/zstd/zstd_compress.ko 2024-03-04T09:13:37.749 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/mm 2024-03-04T09:13:37.750 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/mm/hwpoison-inject.ko 2024-03-04T09:13:37.750 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net 2024-03-04T09:13:37.750 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/6lowpan 2024-03-04T09:13:37.750 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/6lowpan/6lowpan.ko 2024-03-04T09:13:37.750 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/6lowpan/nhc_dest.ko 2024-03-04T09:13:37.750 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/6lowpan/nhc_fragment.ko 2024-03-04T09:13:37.750 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/6lowpan/nhc_hop.ko 2024-03-04T09:13:37.750 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/6lowpan/nhc_ipv6.ko 2024-03-04T09:13:37.750 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/6lowpan/nhc_mobility.ko 2024-03-04T09:13:37.750 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/6lowpan/nhc_routing.ko 2024-03-04T09:13:37.750 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/6lowpan/nhc_udp.ko 2024-03-04T09:13:37.750 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/802 2024-03-04T09:13:37.750 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/802/garp.ko 2024-03-04T09:13:37.750 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/802/mrp.ko 2024-03-04T09:13:37.750 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/802/p8022.ko 2024-03-04T09:13:37.751 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/802/psnap.ko 2024-03-04T09:13:37.751 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/802/stp.ko 2024-03-04T09:13:37.751 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/8021q 2024-03-04T09:13:37.751 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/8021q/8021q.ko 2024-03-04T09:13:37.751 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/9p 2024-03-04T09:13:37.751 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/9p/9pnet.ko 2024-03-04T09:13:37.751 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/9p/9pnet_fd.ko 2024-03-04T09:13:37.751 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/9p/9pnet_rdma.ko 2024-03-04T09:13:37.751 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/9p/9pnet_virtio.ko 2024-03-04T09:13:37.751 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/batman-adv 2024-03-04T09:13:37.751 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/batman-adv/batman-adv.ko 2024-03-04T09:13:37.751 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/bridge 2024-03-04T09:13:37.751 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/bridge/br_netfilter.ko 2024-03-04T09:13:37.751 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/bridge/bridge.ko 2024-03-04T09:13:37.751 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/bridge/netfilter 2024-03-04T09:13:37.751 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-03-04T09:13:37.752 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/bridge/netfilter/ebt_among.ko 2024-03-04T09:13:37.752 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/bridge/netfilter/ebt_arp.ko 2024-03-04T09:13:37.752 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-03-04T09:13:37.752 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-03-04T09:13:37.752 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/bridge/netfilter/ebt_ip.ko 2024-03-04T09:13:37.752 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-03-04T09:13:37.752 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/bridge/netfilter/ebt_limit.ko 2024-03-04T09:13:37.752 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/bridge/netfilter/ebt_log.ko 2024-03-04T09:13:37.752 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/bridge/netfilter/ebt_mark.ko 2024-03-04T09:13:37.752 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-03-04T09:13:37.752 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-03-04T09:13:37.752 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-03-04T09:13:37.752 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-03-04T09:13:37.752 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/bridge/netfilter/ebt_snat.ko 2024-03-04T09:13:37.753 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/bridge/netfilter/ebt_stp.ko 2024-03-04T09:13:37.753 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-03-04T09:13:37.753 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-03-04T09:13:37.753 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-03-04T09:13:37.753 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-03-04T09:13:37.753 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/bridge/netfilter/ebtables.ko 2024-03-04T09:13:37.753 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-03-04T09:13:37.753 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-03-04T09:13:37.753 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-03-04T09:13:37.753 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ceph 2024-03-04T09:13:37.753 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ceph/libceph.ko 2024-03-04T09:13:37.753 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/core 2024-03-04T09:13:37.753 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/core/failover.ko 2024-03-04T09:13:37.753 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/core/pktgen.ko 2024-03-04T09:13:37.753 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/dccp 2024-03-04T09:13:37.753 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/dccp/dccp.ko 2024-03-04T09:13:37.754 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/dccp/dccp_diag.ko 2024-03-04T09:13:37.754 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/dccp/dccp_ipv4.ko 2024-03-04T09:13:37.754 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/dccp/dccp_ipv6.ko 2024-03-04T09:13:37.754 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/dns_resolver 2024-03-04T09:13:37.754 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/dns_resolver/dns_resolver.ko 2024-03-04T09:13:37.754 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/dsa 2024-03-04T09:13:37.754 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/dsa/dsa_core.ko 2024-03-04T09:13:37.754 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/dsa/tag_brcm.ko 2024-03-04T09:13:37.754 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/dsa/tag_dsa.ko 2024-03-04T09:13:37.754 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/dsa/tag_none.ko 2024-03-04T09:13:37.754 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/dsa/tag_trailer.ko 2024-03-04T09:13:37.754 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ieee802154 2024-03-04T09:13:37.754 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ieee802154/6lowpan 2024-03-04T09:13:37.754 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-03-04T09:13:37.754 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ieee802154/ieee802154.ko 2024-03-04T09:13:37.755 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ieee802154/ieee802154_socket.ko 2024-03-04T09:13:37.755 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4 2024-03-04T09:13:37.755 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/ah4.ko 2024-03-04T09:13:37.755 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/esp4.ko 2024-03-04T09:13:37.755 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/fou.ko 2024-03-04T09:13:37.755 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/gre.ko 2024-03-04T09:13:37.755 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/inet_diag.ko 2024-03-04T09:13:37.755 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/ip_gre.ko 2024-03-04T09:13:37.755 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/ip_tunnel.ko 2024-03-04T09:13:37.755 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/ip_vti.ko 2024-03-04T09:13:37.755 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/ipcomp.ko 2024-03-04T09:13:37.755 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/ipip.ko 2024-03-04T09:13:37.755 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/netfilter 2024-03-04T09:13:37.755 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/netfilter/arp_tables.ko 2024-03-04T09:13:37.756 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-03-04T09:13:37.756 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-03-04T09:13:37.756 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-03-04T09:13:37.756 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-03-04T09:13:37.756 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-03-04T09:13:37.756 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-03-04T09:13:37.756 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-03-04T09:13:37.756 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-03-04T09:13:37.756 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-03-04T09:13:37.756 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/netfilter/iptable_security.ko 2024-03-04T09:13:37.756 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-03-04T09:13:37.756 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-03-04T09:13:37.756 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-03-04T09:13:37.756 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-03-04T09:13:37.757 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-03-04T09:13:37.757 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-03-04T09:13:37.757 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-03-04T09:13:37.757 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-03-04T09:13:37.757 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-03-04T09:13:37.757 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-03-04T09:13:37.757 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/tcp_bic.ko 2024-03-04T09:13:37.757 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/tcp_dctcp.ko 2024-03-04T09:13:37.757 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/tcp_diag.ko 2024-03-04T09:13:37.757 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/tcp_highspeed.ko 2024-03-04T09:13:37.757 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/tcp_htcp.ko 2024-03-04T09:13:37.757 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/tcp_hybla.ko 2024-03-04T09:13:37.757 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/tcp_illinois.ko 2024-03-04T09:13:37.757 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/tcp_lp.ko 2024-03-04T09:13:37.758 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/tcp_scalable.ko 2024-03-04T09:13:37.758 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/tcp_vegas.ko 2024-03-04T09:13:37.758 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/tcp_veno.ko 2024-03-04T09:13:37.758 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/tcp_westwood.ko 2024-03-04T09:13:37.758 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/tcp_yeah.ko 2024-03-04T09:13:37.758 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/tunnel4.ko 2024-03-04T09:13:37.758 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/udp_diag.ko 2024-03-04T09:13:37.758 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/udp_tunnel.ko 2024-03-04T09:13:37.758 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/xfrm4_tunnel.ko 2024-03-04T09:13:37.758 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv6 2024-03-04T09:13:37.758 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv6/ah6.ko 2024-03-04T09:13:37.758 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv6/esp6.ko 2024-03-04T09:13:37.758 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv6/fou6.ko 2024-03-04T09:13:37.758 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv6/ip6_tunnel.ko 2024-03-04T09:13:37.759 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-03-04T09:13:37.759 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv6/ip6_vti.ko 2024-03-04T09:13:37.759 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv6/ipcomp6.ko 2024-03-04T09:13:37.759 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv6/netfilter 2024-03-04T09:13:37.759 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-03-04T09:13:37.759 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-03-04T09:13:37.759 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-03-04T09:13:37.759 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-03-04T09:13:37.759 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-03-04T09:13:37.759 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-03-04T09:13:37.759 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-03-04T09:13:37.759 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-03-04T09:13:37.759 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-03-04T09:13:37.759 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-03-04T09:13:37.760 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-03-04T09:13:37.760 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-03-04T09:13:37.760 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-03-04T09:13:37.760 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-03-04T09:13:37.760 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-03-04T09:13:37.760 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-03-04T09:13:37.760 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-03-04T09:13:37.760 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-03-04T09:13:37.760 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-03-04T09:13:37.760 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-03-04T09:13:37.760 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-03-04T09:13:37.760 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-03-04T09:13:37.760 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-03-04T09:13:37.761 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-03-04T09:13:37.761 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv6/sit.ko 2024-03-04T09:13:37.761 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv6/tunnel6.ko 2024-03-04T09:13:37.761 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv6/xfrm6_tunnel.ko 2024-03-04T09:13:37.761 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/key 2024-03-04T09:13:37.761 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/key/af_key.ko 2024-03-04T09:13:37.761 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/l2tp 2024-03-04T09:13:37.761 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/l2tp/l2tp_core.ko 2024-03-04T09:13:37.761 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/l2tp/l2tp_debugfs.ko 2024-03-04T09:13:37.761 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/l2tp/l2tp_eth.ko 2024-03-04T09:13:37.761 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/l2tp/l2tp_ip.ko 2024-03-04T09:13:37.761 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/l2tp/l2tp_ip6.ko 2024-03-04T09:13:37.761 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/l2tp/l2tp_netlink.ko 2024-03-04T09:13:37.761 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/l2tp/l2tp_ppp.ko 2024-03-04T09:13:37.761 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/llc 2024-03-04T09:13:37.762 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/llc/llc.ko 2024-03-04T09:13:37.762 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/mac802154 2024-03-04T09:13:37.762 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/mac802154/mac802154.ko 2024-03-04T09:13:37.762 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/mpls 2024-03-04T09:13:37.762 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/mpls/mpls_gso.ko 2024-03-04T09:13:37.762 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter 2024-03-04T09:13:37.762 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/ipset 2024-03-04T09:13:37.762 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/ipset/ip_set.ko 2024-03-04T09:13:37.762 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-03-04T09:13:37.762 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-03-04T09:13:37.762 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-03-04T09:13:37.763 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-03-04T09:13:37.763 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-03-04T09:13:37.763 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-03-04T09:13:37.763 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-03-04T09:13:37.763 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-03-04T09:13:37.763 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-03-04T09:13:37.763 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-03-04T09:13:37.763 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-03-04T09:13:37.763 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-03-04T09:13:37.763 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-03-04T09:13:37.763 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-03-04T09:13:37.763 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-03-04T09:13:37.763 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/ipvs 2024-03-04T09:13:37.764 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/ipvs/ip_vs.ko 2024-03-04T09:13:37.764 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-03-04T09:13:37.764 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-03-04T09:13:37.764 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-03-04T09:13:37.764 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-03-04T09:13:37.764 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-03-04T09:13:37.764 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-03-04T09:13:37.764 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-03-04T09:13:37.764 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-03-04T09:13:37.764 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-03-04T09:13:37.764 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-03-04T09:13:37.764 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-03-04T09:13:37.764 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-03-04T09:13:37.765 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-03-04T09:13:37.765 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nf_conncount.ko 2024-03-04T09:13:37.765 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nf_conntrack.ko 2024-03-04T09:13:37.765 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-03-04T09:13:37.765 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-03-04T09:13:37.765 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-03-04T09:13:37.765 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nf_conntrack_h323.ko 2024-03-04T09:13:37.765 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nf_conntrack_irc.ko 2024-03-04T09:13:37.765 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-03-04T09:13:37.765 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-03-04T09:13:37.765 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-03-04T09:13:37.765 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nf_conntrack_sane.ko 2024-03-04T09:13:37.765 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nf_conntrack_sip.ko 2024-03-04T09:13:37.766 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-03-04T09:13:37.766 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-03-04T09:13:37.766 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nf_dup_netdev.ko 2024-03-04T09:13:37.766 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nf_flow_table.ko 2024-03-04T09:13:37.766 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nf_flow_table_inet.ko 2024-03-04T09:13:37.766 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nf_log_syslog.ko 2024-03-04T09:13:37.766 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nf_nat.ko 2024-03-04T09:13:37.766 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nf_nat_amanda.ko 2024-03-04T09:13:37.766 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nf_nat_ftp.ko 2024-03-04T09:13:37.766 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nf_nat_irc.ko 2024-03-04T09:13:37.766 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nf_nat_sip.ko 2024-03-04T09:13:37.766 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nf_nat_tftp.ko 2024-03-04T09:13:37.766 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nf_synproxy_core.ko 2024-03-04T09:13:37.766 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nf_tables.ko 2024-03-04T09:13:37.767 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nfnetlink.ko 2024-03-04T09:13:37.767 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nfnetlink_acct.ko 2024-03-04T09:13:37.767 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nfnetlink_log.ko 2024-03-04T09:13:37.767 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nfnetlink_osf.ko 2024-03-04T09:13:37.767 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nfnetlink_queue.ko 2024-03-04T09:13:37.767 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nft_chain_nat.ko 2024-03-04T09:13:37.767 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nft_compat.ko 2024-03-04T09:13:37.767 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nft_ct.ko 2024-03-04T09:13:37.767 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nft_dup_netdev.ko 2024-03-04T09:13:37.767 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nft_fib.ko 2024-03-04T09:13:37.767 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nft_fwd_netdev.ko 2024-03-04T09:13:37.767 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nft_hash.ko 2024-03-04T09:13:37.767 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nft_limit.ko 2024-03-04T09:13:37.767 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nft_log.ko 2024-03-04T09:13:37.767 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nft_masq.ko 2024-03-04T09:13:37.768 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nft_nat.ko 2024-03-04T09:13:37.768 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nft_numgen.ko 2024-03-04T09:13:37.768 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nft_queue.ko 2024-03-04T09:13:37.768 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nft_quota.ko 2024-03-04T09:13:37.768 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nft_redir.ko 2024-03-04T09:13:37.768 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nft_reject.ko 2024-03-04T09:13:37.768 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nft_reject_inet.ko 2024-03-04T09:13:37.768 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nft_socket.ko 2024-03-04T09:13:37.768 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nft_tunnel.ko 2024-03-04T09:13:37.768 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_AUDIT.ko 2024-03-04T09:13:37.768 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_CHECKSUM.ko 2024-03-04T09:13:37.768 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_CLASSIFY.ko 2024-03-04T09:13:37.768 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-03-04T09:13:37.769 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_CT.ko 2024-03-04T09:13:37.769 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_DSCP.ko 2024-03-04T09:13:37.769 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_HL.ko 2024-03-04T09:13:37.769 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_HMARK.ko 2024-03-04T09:13:37.769 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_IDLETIMER.ko 2024-03-04T09:13:37.769 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_LED.ko 2024-03-04T09:13:37.769 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_LOG.ko 2024-03-04T09:13:37.769 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_MASQUERADE.ko 2024-03-04T09:13:37.769 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_NETMAP.ko 2024-03-04T09:13:37.769 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_NFLOG.ko 2024-03-04T09:13:37.769 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_NFQUEUE.ko 2024-03-04T09:13:37.769 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_RATEEST.ko 2024-03-04T09:13:37.769 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_REDIRECT.ko 2024-03-04T09:13:37.769 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_SECMARK.ko 2024-03-04T09:13:37.769 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_TCPMSS.ko 2024-03-04T09:13:37.769 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-03-04T09:13:37.770 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_TEE.ko 2024-03-04T09:13:37.770 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_TPROXY.ko 2024-03-04T09:13:37.770 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_TRACE.ko 2024-03-04T09:13:37.770 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_addrtype.ko 2024-03-04T09:13:37.770 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_bpf.ko 2024-03-04T09:13:37.770 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_cgroup.ko 2024-03-04T09:13:37.770 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_cluster.ko 2024-03-04T09:13:37.770 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_comment.ko 2024-03-04T09:13:37.770 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_connbytes.ko 2024-03-04T09:13:37.770 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_connlabel.ko 2024-03-04T09:13:37.770 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_connlimit.ko 2024-03-04T09:13:37.771 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_connmark.ko 2024-03-04T09:13:37.771 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_conntrack.ko 2024-03-04T09:13:37.771 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_cpu.ko 2024-03-04T09:13:37.771 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_dccp.ko 2024-03-04T09:13:37.771 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_devgroup.ko 2024-03-04T09:13:37.771 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_dscp.ko 2024-03-04T09:13:37.771 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_ecn.ko 2024-03-04T09:13:37.771 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_esp.ko 2024-03-04T09:13:37.771 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_hashlimit.ko 2024-03-04T09:13:37.771 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_helper.ko 2024-03-04T09:13:37.771 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_hl.ko 2024-03-04T09:13:37.771 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_ipcomp.ko 2024-03-04T09:13:37.771 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_iprange.ko 2024-03-04T09:13:37.771 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_ipvs.ko 2024-03-04T09:13:37.771 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_l2tp.ko 2024-03-04T09:13:37.771 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_length.ko 2024-03-04T09:13:37.772 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_limit.ko 2024-03-04T09:13:37.772 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_mac.ko 2024-03-04T09:13:37.772 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_mark.ko 2024-03-04T09:13:37.772 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_multiport.ko 2024-03-04T09:13:37.772 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_nat.ko 2024-03-04T09:13:37.772 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_nfacct.ko 2024-03-04T09:13:37.772 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_osf.ko 2024-03-04T09:13:37.772 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_owner.ko 2024-03-04T09:13:37.772 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_physdev.ko 2024-03-04T09:13:37.772 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_pkttype.ko 2024-03-04T09:13:37.772 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_policy.ko 2024-03-04T09:13:37.772 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_quota.ko 2024-03-04T09:13:37.772 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_rateest.ko 2024-03-04T09:13:37.772 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_realm.ko 2024-03-04T09:13:37.772 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_recent.ko 2024-03-04T09:13:37.773 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_sctp.ko 2024-03-04T09:13:37.773 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_set.ko 2024-03-04T09:13:37.773 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_socket.ko 2024-03-04T09:13:37.773 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_state.ko 2024-03-04T09:13:37.773 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_statistic.ko 2024-03-04T09:13:37.773 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_string.ko 2024-03-04T09:13:37.773 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_tcpmss.ko 2024-03-04T09:13:37.773 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_time.ko 2024-03-04T09:13:37.773 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_u32.ko 2024-03-04T09:13:37.773 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netlink 2024-03-04T09:13:37.773 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netlink/netlink_diag.ko 2024-03-04T09:13:37.773 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/nsh 2024-03-04T09:13:37.773 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/nsh/nsh.ko 2024-03-04T09:13:37.773 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/openvswitch 2024-03-04T09:13:37.773 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/openvswitch/openvswitch.ko 2024-03-04T09:13:37.773 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/openvswitch/vport-geneve.ko 2024-03-04T09:13:37.774 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/openvswitch/vport-gre.ko 2024-03-04T09:13:37.774 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/openvswitch/vport-vxlan.ko 2024-03-04T09:13:37.774 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/packet 2024-03-04T09:13:37.774 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/packet/af_packet_diag.ko 2024-03-04T09:13:37.774 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/rds 2024-03-04T09:13:37.774 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/rds/rds.ko 2024-03-04T09:13:37.774 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/rds/rds_rdma.ko 2024-03-04T09:13:37.774 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/rds/rds_tcp.ko 2024-03-04T09:13:37.774 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched 2024-03-04T09:13:37.774 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/act_csum.ko 2024-03-04T09:13:37.774 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/act_gact.ko 2024-03-04T09:13:37.774 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/act_mirred.ko 2024-03-04T09:13:37.774 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/act_nat.ko 2024-03-04T09:13:37.774 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/act_pedit.ko 2024-03-04T09:13:37.775 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/act_police.ko 2024-03-04T09:13:37.775 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/act_simple.ko 2024-03-04T09:13:37.775 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/act_skbedit.ko 2024-03-04T09:13:37.775 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/act_vlan.ko 2024-03-04T09:13:37.775 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/cls_basic.ko 2024-03-04T09:13:37.775 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/cls_bpf.ko 2024-03-04T09:13:37.775 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/cls_flow.ko 2024-03-04T09:13:37.775 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/cls_fw.ko 2024-03-04T09:13:37.775 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/cls_route.ko 2024-03-04T09:13:37.775 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/cls_u32.ko 2024-03-04T09:13:37.775 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/em_cmp.ko 2024-03-04T09:13:37.775 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/em_ipset.ko 2024-03-04T09:13:37.775 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/em_meta.ko 2024-03-04T09:13:37.776 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/em_nbyte.ko 2024-03-04T09:13:37.776 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/em_text.ko 2024-03-04T09:13:37.776 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/em_u32.ko 2024-03-04T09:13:37.776 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/sch_choke.ko 2024-03-04T09:13:37.776 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/sch_codel.ko 2024-03-04T09:13:37.776 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/sch_drr.ko 2024-03-04T09:13:37.776 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/sch_fq.ko 2024-03-04T09:13:37.776 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/sch_gred.ko 2024-03-04T09:13:37.776 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/sch_hfsc.ko 2024-03-04T09:13:37.776 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/sch_hhf.ko 2024-03-04T09:13:37.776 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/sch_htb.ko 2024-03-04T09:13:37.776 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/sch_ingress.ko 2024-03-04T09:13:37.777 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/sch_mqprio.ko 2024-03-04T09:13:37.777 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/sch_mqprio_lib.ko 2024-03-04T09:13:37.777 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/sch_multiq.ko 2024-03-04T09:13:37.777 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/sch_netem.ko 2024-03-04T09:13:37.777 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/sch_pie.ko 2024-03-04T09:13:37.777 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/sch_plug.ko 2024-03-04T09:13:37.777 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/sch_prio.ko 2024-03-04T09:13:37.777 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/sch_qfq.ko 2024-03-04T09:13:37.777 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/sch_red.ko 2024-03-04T09:13:37.777 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/sch_sfb.ko 2024-03-04T09:13:37.777 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/sch_sfq.ko 2024-03-04T09:13:37.777 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/sch_tbf.ko 2024-03-04T09:13:37.777 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/sch_teql.ko 2024-03-04T09:13:37.777 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sctp 2024-03-04T09:13:37.777 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sctp/sctp.ko 2024-03-04T09:13:37.778 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sctp/sctp_diag.ko 2024-03-04T09:13:37.778 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sunrpc 2024-03-04T09:13:37.778 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sunrpc/auth_gss 2024-03-04T09:13:37.778 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-03-04T09:13:37.778 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-03-04T09:13:37.778 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sunrpc/sunrpc.ko 2024-03-04T09:13:37.778 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sunrpc/xprtrdma 2024-03-04T09:13:37.778 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-03-04T09:13:37.778 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/tipc 2024-03-04T09:13:37.778 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/tipc/diag.ko 2024-03-04T09:13:37.778 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/tipc/tipc.ko 2024-03-04T09:13:37.778 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/unix 2024-03-04T09:13:37.778 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/unix/unix_diag.ko 2024-03-04T09:13:37.778 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/vmw_vsock 2024-03-04T09:13:37.778 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-03-04T09:13:37.779 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-03-04T09:13:37.779 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/vmw_vsock/vsock.ko 2024-03-04T09:13:37.779 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/vmw_vsock/vsock_diag.ko 2024-03-04T09:13:37.779 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/vmw_vsock/vsock_loopback.ko 2024-03-04T09:13:37.779 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/xfrm 2024-03-04T09:13:37.779 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/xfrm/xfrm_ipcomp.ko 2024-03-04T09:13:37.779 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/security 2024-03-04T09:13:37.779 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/security/keys 2024-03-04T09:13:37.779 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/security/keys/encrypted-keys 2024-03-04T09:13:37.779 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-03-04T09:13:37.779 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/security/keys/trusted-keys 2024-03-04T09:13:37.779 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/security/keys/trusted-keys/trusted.ko 2024-03-04T09:13:37.779 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/virt 2024-03-04T09:13:37.779 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/virt/lib 2024-03-04T09:13:37.779 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/virt/lib/irqbypass.ko 2024-03-04T09:13:37.780 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/modules.alias 2024-03-04T09:13:37.780 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/modules.alias.bin 2024-03-04T09:13:37.780 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/modules.builtin 2024-03-04T09:13:37.780 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/modules.builtin.alias.bin 2024-03-04T09:13:37.780 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/modules.builtin.bin 2024-03-04T09:13:37.780 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/modules.builtin.modinfo 2024-03-04T09:13:37.780 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/modules.dep 2024-03-04T09:13:37.780 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/modules.dep.bin 2024-03-04T09:13:37.780 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/modules.devname 2024-03-04T09:13:37.780 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/modules.order 2024-03-04T09:13:37.780 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/modules.softdep 2024-03-04T09:13:37.780 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/modules.symbols 2024-03-04T09:13:37.781 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/modules.symbols.bin 2024-03-04T09:13:37.781 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/vmlinuz 2024-03-04T09:13:37.785 DEBUG:teuthology.task.kernel:get_image_version: 6.8.0-rc6-g786939134d7e 2024-03-04T09:13:37.785 DEBUG:teuthology.orchestra.run.smithi146:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-03-04T09:13:37.807 INFO:teuthology.orchestra.run.smithi146.stdout:postinstall scriptlet (using /bin/sh): 2024-03-04T09:13:37.808 INFO:teuthology.orchestra.run.smithi146.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-03-04T09:13:37.808 INFO:teuthology.orchestra.run.smithi146.stdout: /usr/bin/kernel-install add 6.8.0-rc6-g786939134d7e /lib/modules/6.8.0-rc6-g786939134d7e/vmlinuz 2024-03-04T09:13:37.808 INFO:teuthology.orchestra.run.smithi146.stdout:fi 2024-03-04T09:13:37.808 INFO:teuthology.orchestra.run.smithi146.stdout:for file in vmlinuz System.map config; do 2024-03-04T09:13:37.808 INFO:teuthology.orchestra.run.smithi146.stdout: if ! cmp --silent "/lib/modules/6.8.0-rc6-g786939134d7e/${file}" "/boot/${file}-6.8.0-rc6-g786939134d7e"; then 2024-03-04T09:13:37.808 INFO:teuthology.orchestra.run.smithi146.stdout: cp "/lib/modules/6.8.0-rc6-g786939134d7e/${file}" "/boot/${file}-6.8.0-rc6-g786939134d7e" 2024-03-04T09:13:37.808 INFO:teuthology.orchestra.run.smithi146.stdout: fi 2024-03-04T09:13:37.808 INFO:teuthology.orchestra.run.smithi146.stdout:done 2024-03-04T09:13:37.808 INFO:teuthology.orchestra.run.smithi146.stdout:preuninstall scriptlet (using /bin/sh): 2024-03-04T09:13:37.808 INFO:teuthology.orchestra.run.smithi146.stdout:if [ -x /sbin/new-kernel-pkg ]; then 2024-03-04T09:13:37.808 INFO:teuthology.orchestra.run.smithi146.stdout:new-kernel-pkg --remove 6.8.0-rc6-g786939134d7e --rminitrd --initrdfile=/boot/initramfs-6.8.0-rc6-g786939134d7e.img 2024-03-04T09:13:37.809 INFO:teuthology.orchestra.run.smithi146.stdout:elif [ -x /usr/bin/kernel-install ]; then 2024-03-04T09:13:37.809 INFO:teuthology.orchestra.run.smithi146.stdout:kernel-install remove 6.8.0-rc6-g786939134d7e 2024-03-04T09:13:37.809 INFO:teuthology.orchestra.run.smithi146.stdout:fi 2024-03-04T09:13:37.809 INFO:teuthology.orchestra.run.smithi146.stdout:postuninstall scriptlet (using /bin/sh): 2024-03-04T09:13:37.809 INFO:teuthology.orchestra.run.smithi146.stdout:if [ -x /sbin/update-bootloader ]; then 2024-03-04T09:13:37.809 INFO:teuthology.orchestra.run.smithi146.stdout:/sbin/update-bootloader --remove 6.8.0-rc6-g786939134d7e 2024-03-04T09:13:37.809 INFO:teuthology.orchestra.run.smithi146.stdout:fi 2024-03-04T09:13:37.810 DEBUG:teuthology.orchestra.run.smithi146:> sudo rpm -qi grub2-tools 2024-03-04T09:13:37.883 INFO:teuthology.orchestra.run.smithi146.stdout:Name : grub2-tools 2024-03-04T09:13:37.883 INFO:teuthology.orchestra.run.smithi146.stdout:Epoch : 1 2024-03-04T09:13:37.883 INFO:teuthology.orchestra.run.smithi146.stdout:Version : 2.06 2024-03-04T09:13:37.884 INFO:teuthology.orchestra.run.smithi146.stdout:Release : 68.el9 2024-03-04T09:13:37.884 INFO:teuthology.orchestra.run.smithi146.stdout:Architecture: x86_64 2024-03-04T09:13:37.884 INFO:teuthology.orchestra.run.smithi146.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-04T09:13:37.884 INFO:teuthology.orchestra.run.smithi146.stdout:Group : Unspecified 2024-03-04T09:13:37.884 INFO:teuthology.orchestra.run.smithi146.stdout:Size : 8264254 2024-03-04T09:13:37.884 INFO:teuthology.orchestra.run.smithi146.stdout:License : GPLv3+ 2024-03-04T09:13:37.884 INFO:teuthology.orchestra.run.smithi146.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-04T09:13:37.884 INFO:teuthology.orchestra.run.smithi146.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-04T09:13:37.884 INFO:teuthology.orchestra.run.smithi146.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-04T09:13:37.884 INFO:teuthology.orchestra.run.smithi146.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-04T09:13:37.884 INFO:teuthology.orchestra.run.smithi146.stdout:Packager : builder@centos.org 2024-03-04T09:13:37.884 INFO:teuthology.orchestra.run.smithi146.stdout:Vendor : CentOS 2024-03-04T09:13:37.884 INFO:teuthology.orchestra.run.smithi146.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-04T09:13:37.884 INFO:teuthology.orchestra.run.smithi146.stdout:Summary : Support tools for GRUB. 2024-03-04T09:13:37.885 INFO:teuthology.orchestra.run.smithi146.stdout:Description : 2024-03-04T09:13:37.885 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:13:37.885 INFO:teuthology.orchestra.run.smithi146.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-04T09:13:37.885 INFO:teuthology.orchestra.run.smithi146.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-04T09:13:37.885 INFO:teuthology.orchestra.run.smithi146.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-04T09:13:37.885 INFO:teuthology.orchestra.run.smithi146.stdout:hardware devices. 2024-03-04T09:13:37.885 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:13:37.885 INFO:teuthology.orchestra.run.smithi146.stdout:This subpackage provides tools for support of all platforms. 2024-03-04T09:13:37.886 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-04T09:13:37.886 INFO:teuthology.task.kernel:Updating grub on smithi146 to boot 6.8.0-rc6-g786939134d7e 2024-03-04T09:13:37.886 DEBUG:teuthology.orchestra.run.smithi146:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-04T09:13:38.465 INFO:teuthology.orchestra.run.smithi146.stderr:Generating grub configuration file ... 2024-03-04T09:13:39.858 INFO:teuthology.orchestra.run.smithi146.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-04T09:13:39.890 INFO:teuthology.orchestra.run.smithi146.stderr:done 2024-03-04T09:13:39.893 DEBUG:teuthology.orchestra.run.smithi146:> mktemp 2024-03-04T09:13:39.907 INFO:teuthology.orchestra.run.smithi146.stdout:/tmp/tmp.pIvWgRwBXD 2024-03-04T09:13:39.907 DEBUG:teuthology.orchestra.run.smithi146:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.pIvWgRwBXD 2024-03-04T09:13:39.996 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod 0666 /tmp/tmp.pIvWgRwBXD 2024-03-04T09:13:40.099 DEBUG:teuthology.orchestra.remote:smithi146:/tmp/tmp.pIvWgRwBXD is 6KB 2024-03-04T09:13:40.149 DEBUG:teuthology.orchestra.run.smithi146:> rm -fr /tmp/tmp.pIvWgRwBXD 2024-03-04T09:13:40.164 DEBUG:teuthology.orchestra.run.smithi146:> sudo /bin/ls /boot/loader/entries || true 2024-03-04T09:13:40.234 INFO:teuthology.orchestra.run.smithi146.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-04T09:13:40.234 INFO:teuthology.orchestra.run.smithi146.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-04T09:13:40.234 INFO:teuthology.orchestra.run.smithi146.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-04T09:13:40.234 INFO:teuthology.orchestra.run.smithi146.stdout:15949e3c88704300882482eddfd7baae-6.8.0-rc6-g786939134d7e.conf 2024-03-04T09:13:40.235 DEBUG:teuthology.orchestra.run.smithi146:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-6.8.0-rc6-g786939134d7e 2024-03-04T09:13:40.385 DEBUG:teuthology.orchestra.run.smithi146:> sudo shutdown -r now 2024-03-04T09:13:51.440 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi006.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-03-04T09:13:51.441 DEBUG:teuthology.orchestra.run.smithi006:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-03-04T09:13:51.511 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e 2024-03-04T09:13:51.511 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/System.map 2024-03-04T09:13:51.511 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/config 2024-03-04T09:13:51.511 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel 2024-03-04T09:13:51.511 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/arch 2024-03-04T09:13:51.511 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/arch/x86 2024-03-04T09:13:51.511 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/arch/x86/crypto 2024-03-04T09:13:51.511 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-03-04T09:13:51.512 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-03-04T09:13:51.512 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-03-04T09:13:51.512 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-03-04T09:13:51.512 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-03-04T09:13:51.512 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-03-04T09:13:51.512 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-03-04T09:13:51.512 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/arch/x86/crypto/crc32c-intel.ko 2024-03-04T09:13:51.512 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-03-04T09:13:51.512 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-03-04T09:13:51.512 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-03-04T09:13:51.512 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-03-04T09:13:51.512 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/arch/x86/crypto/serpent-avx2.ko 2024-03-04T09:13:51.512 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-03-04T09:13:51.512 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-03-04T09:13:51.513 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-03-04T09:13:51.513 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-03-04T09:13:51.513 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-03-04T09:13:51.513 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-03-04T09:13:51.513 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-03-04T09:13:51.513 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/arch/x86/kernel 2024-03-04T09:13:51.513 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/arch/x86/kernel/cpu 2024-03-04T09:13:51.513 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/arch/x86/kernel/cpu/mce 2024-03-04T09:13:51.513 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-03-04T09:13:51.513 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/arch/x86/kvm 2024-03-04T09:13:51.513 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/arch/x86/kvm/kvm-amd.ko 2024-03-04T09:13:51.513 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/arch/x86/kvm/kvm-intel.ko 2024-03-04T09:13:51.513 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/arch/x86/kvm/kvm.ko 2024-03-04T09:13:51.513 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto 2024-03-04T09:13:51.513 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/ansi_cprng.ko 2024-03-04T09:13:51.514 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/anubis.ko 2024-03-04T09:13:51.514 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/arc4.ko 2024-03-04T09:13:51.514 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/async_tx 2024-03-04T09:13:51.514 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/async_tx/async_memcpy.ko 2024-03-04T09:13:51.514 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/async_tx/async_pq.ko 2024-03-04T09:13:51.514 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/async_tx/async_raid6_recov.ko 2024-03-04T09:13:51.514 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/async_tx/async_tx.ko 2024-03-04T09:13:51.514 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/async_tx/async_xor.ko 2024-03-04T09:13:51.514 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/async_tx/raid6test.ko 2024-03-04T09:13:51.514 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/authenc.ko 2024-03-04T09:13:51.514 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/authencesn.ko 2024-03-04T09:13:51.515 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/blake2b_generic.ko 2024-03-04T09:13:51.515 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/blowfish_common.ko 2024-03-04T09:13:51.515 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/blowfish_generic.ko 2024-03-04T09:13:51.515 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/camellia_generic.ko 2024-03-04T09:13:51.515 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/cast5_generic.ko 2024-03-04T09:13:51.515 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/cast6_generic.ko 2024-03-04T09:13:51.515 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/cast_common.ko 2024-03-04T09:13:51.515 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/ccm.ko 2024-03-04T09:13:51.515 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/cmac.ko 2024-03-04T09:13:51.515 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/crc32_generic.ko 2024-03-04T09:13:51.515 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/crypto_engine.ko 2024-03-04T09:13:51.515 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/crypto_user.ko 2024-03-04T09:13:51.515 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/cts.ko 2024-03-04T09:13:51.515 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/des_generic.ko 2024-03-04T09:13:51.516 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/dh_generic.ko 2024-03-04T09:13:51.516 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/echainiv.ko 2024-03-04T09:13:51.516 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/essiv.ko 2024-03-04T09:13:51.516 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/fcrypt.ko 2024-03-04T09:13:51.516 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/khazad.ko 2024-03-04T09:13:51.516 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/lz4.ko 2024-03-04T09:13:51.516 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/lz4hc.ko 2024-03-04T09:13:51.516 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/md4.ko 2024-03-04T09:13:51.516 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/michael_mic.ko 2024-03-04T09:13:51.516 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/pcbc.ko 2024-03-04T09:13:51.516 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/pcrypt.ko 2024-03-04T09:13:51.516 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/rmd160.ko 2024-03-04T09:13:51.516 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/seed.ko 2024-03-04T09:13:51.516 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/serpent_generic.ko 2024-03-04T09:13:51.517 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/tcrypt.ko 2024-03-04T09:13:51.517 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/tea.ko 2024-03-04T09:13:51.517 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/twofish_common.ko 2024-03-04T09:13:51.517 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/twofish_generic.ko 2024-03-04T09:13:51.517 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/vmac.ko 2024-03-04T09:13:51.517 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/wp512.ko 2024-03-04T09:13:51.517 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/xcbc.ko 2024-03-04T09:13:51.517 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/xor.ko 2024-03-04T09:13:51.517 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/xxhash_generic.ko 2024-03-04T09:13:51.517 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers 2024-03-04T09:13:51.518 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/acpi 2024-03-04T09:13:51.518 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/acpi/acpi_ipmi.ko 2024-03-04T09:13:51.518 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/acpi/acpi_pad.ko 2024-03-04T09:13:51.518 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/acpi/custom_method.ko 2024-03-04T09:13:51.518 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/acpi/ec_sys.ko 2024-03-04T09:13:51.518 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/acpi/platform_profile.ko 2024-03-04T09:13:51.518 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/acpi/sbs.ko 2024-03-04T09:13:51.518 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/acpi/sbshc.ko 2024-03-04T09:13:51.519 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/acpi/video.ko 2024-03-04T09:13:51.519 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata 2024-03-04T09:13:51.519 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/acard-ahci.ko 2024-03-04T09:13:51.519 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/ahci_platform.ko 2024-03-04T09:13:51.519 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/ata_generic.ko 2024-03-04T09:13:51.519 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/libahci_platform.ko 2024-03-04T09:13:51.519 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/pata_acpi.ko 2024-03-04T09:13:51.519 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/pata_ali.ko 2024-03-04T09:13:51.519 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/pata_amd.ko 2024-03-04T09:13:51.519 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/pata_artop.ko 2024-03-04T09:13:51.520 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/pata_atiixp.ko 2024-03-04T09:13:51.520 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/pata_atp867x.ko 2024-03-04T09:13:51.520 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/pata_cmd640.ko 2024-03-04T09:13:51.520 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/pata_cmd64x.ko 2024-03-04T09:13:51.520 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/pata_cypress.ko 2024-03-04T09:13:51.520 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/pata_efar.ko 2024-03-04T09:13:51.520 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/pata_hpt366.ko 2024-03-04T09:13:51.520 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/pata_hpt37x.ko 2024-03-04T09:13:51.520 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/pata_hpt3x2n.ko 2024-03-04T09:13:51.520 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/pata_hpt3x3.ko 2024-03-04T09:13:51.520 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/pata_it8213.ko 2024-03-04T09:13:51.520 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/pata_it821x.ko 2024-03-04T09:13:51.520 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/pata_jmicron.ko 2024-03-04T09:13:51.520 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/pata_marvell.ko 2024-03-04T09:13:51.521 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/pata_mpiix.ko 2024-03-04T09:13:51.521 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/pata_netcell.ko 2024-03-04T09:13:51.521 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/pata_ninja32.ko 2024-03-04T09:13:51.521 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/pata_ns87410.ko 2024-03-04T09:13:51.521 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/pata_ns87415.ko 2024-03-04T09:13:51.521 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/pata_oldpiix.ko 2024-03-04T09:13:51.521 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/pata_opti.ko 2024-03-04T09:13:51.521 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/pata_optidma.ko 2024-03-04T09:13:51.521 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/pata_pcmcia.ko 2024-03-04T09:13:51.521 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/pata_pdc2027x.ko 2024-03-04T09:13:51.521 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-03-04T09:13:51.521 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/pata_piccolo.ko 2024-03-04T09:13:51.521 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/pata_rdc.ko 2024-03-04T09:13:51.521 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/pata_sch.ko 2024-03-04T09:13:51.522 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/pata_serverworks.ko 2024-03-04T09:13:51.522 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/pata_sil680.ko 2024-03-04T09:13:51.522 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/pata_sis.ko 2024-03-04T09:13:51.522 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/pata_sl82c105.ko 2024-03-04T09:13:51.522 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/pata_triflex.ko 2024-03-04T09:13:51.522 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/pata_via.ko 2024-03-04T09:13:51.522 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/pdc_adma.ko 2024-03-04T09:13:51.522 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/sata_inic162x.ko 2024-03-04T09:13:51.522 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/sata_mv.ko 2024-03-04T09:13:51.522 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/sata_nv.ko 2024-03-04T09:13:51.522 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/sata_promise.ko 2024-03-04T09:13:51.522 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/sata_qstor.ko 2024-03-04T09:13:51.522 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/sata_sil.ko 2024-03-04T09:13:51.522 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/sata_sil24.ko 2024-03-04T09:13:51.523 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/sata_sis.ko 2024-03-04T09:13:51.523 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/sata_svw.ko 2024-03-04T09:13:51.523 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/sata_sx4.ko 2024-03-04T09:13:51.523 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/sata_uli.ko 2024-03-04T09:13:51.523 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/sata_via.ko 2024-03-04T09:13:51.523 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/sata_vsc.ko 2024-03-04T09:13:51.523 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/auxdisplay 2024-03-04T09:13:51.523 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/auxdisplay/ks0108.ko 2024-03-04T09:13:51.523 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/base 2024-03-04T09:13:51.523 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/base/regmap 2024-03-04T09:13:51.523 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/base/regmap/regmap-i2c.ko 2024-03-04T09:13:51.523 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/base/regmap/regmap-mmio.ko 2024-03-04T09:13:51.523 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/bcma 2024-03-04T09:13:51.523 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/bcma/bcma.ko 2024-03-04T09:13:51.524 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/block 2024-03-04T09:13:51.524 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/block/aoe 2024-03-04T09:13:51.524 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/block/aoe/aoe.ko 2024-03-04T09:13:51.524 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/block/brd.ko 2024-03-04T09:13:51.524 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/block/drbd 2024-03-04T09:13:51.524 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/block/drbd/drbd.ko 2024-03-04T09:13:51.524 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/block/floppy.ko 2024-03-04T09:13:51.524 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/block/loop.ko 2024-03-04T09:13:51.524 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/block/mtip32xx 2024-03-04T09:13:51.524 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-03-04T09:13:51.524 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/block/nbd.ko 2024-03-04T09:13:51.524 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/block/null_blk 2024-03-04T09:13:51.524 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/block/null_blk/null_blk.ko 2024-03-04T09:13:51.525 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/block/pktcdvd.ko 2024-03-04T09:13:51.525 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/block/rbd.ko 2024-03-04T09:13:51.525 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/block/virtio_blk.ko 2024-03-04T09:13:51.525 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/block/xen-blkback 2024-03-04T09:13:51.525 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-03-04T09:13:51.525 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/block/xen-blkfront.ko 2024-03-04T09:13:51.525 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/block/zram 2024-03-04T09:13:51.525 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/block/zram/zram.ko 2024-03-04T09:13:51.525 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/char 2024-03-04T09:13:51.525 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/char/hangcheck-timer.ko 2024-03-04T09:13:51.525 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/char/hw_random 2024-03-04T09:13:51.525 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/char/hw_random/amd-rng.ko 2024-03-04T09:13:51.525 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/char/hw_random/intel-rng.ko 2024-03-04T09:13:51.525 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-03-04T09:13:51.525 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/char/hw_random/via-rng.ko 2024-03-04T09:13:51.526 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/char/hw_random/virtio-rng.ko 2024-03-04T09:13:51.526 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/char/ipmi 2024-03-04T09:13:51.526 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-03-04T09:13:51.526 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-03-04T09:13:51.526 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-03-04T09:13:51.526 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/char/ipmi/ipmi_si.ko 2024-03-04T09:13:51.526 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-03-04T09:13:51.526 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-03-04T09:13:51.526 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/char/lp.ko 2024-03-04T09:13:51.526 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/char/mwave 2024-03-04T09:13:51.526 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/char/mwave/mwave.ko 2024-03-04T09:13:51.526 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/char/ppdev.ko 2024-03-04T09:13:51.526 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/char/tlclk.ko 2024-03-04T09:13:51.526 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/char/tpm 2024-03-04T09:13:51.527 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/char/tpm/tpm.ko 2024-03-04T09:13:51.527 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/char/tpm/tpm_atmel.ko 2024-03-04T09:13:51.527 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/char/tpm/tpm_infineon.ko 2024-03-04T09:13:51.527 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/char/tpm/tpm_nsc.ko 2024-03-04T09:13:51.527 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/char/tpm/tpm_tis.ko 2024-03-04T09:13:51.527 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-03-04T09:13:51.527 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/char/uv_mmtimer.ko 2024-03-04T09:13:51.527 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/char/virtio_console.ko 2024-03-04T09:13:51.527 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/cpufreq 2024-03-04T09:13:51.527 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-03-04T09:13:51.527 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-03-04T09:13:51.527 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/cpufreq/p4-clockmod.ko 2024-03-04T09:13:51.527 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-03-04T09:13:51.527 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/cpufreq/powernow-k8.ko 2024-03-04T09:13:51.528 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/cpufreq/speedstep-lib.ko 2024-03-04T09:13:51.528 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/crypto 2024-03-04T09:13:51.528 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/crypto/ccp 2024-03-04T09:13:51.528 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-03-04T09:13:51.528 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/crypto/ccp/ccp.ko 2024-03-04T09:13:51.528 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/crypto/intel 2024-03-04T09:13:51.528 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/crypto/intel/qat 2024-03-04T09:13:51.528 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/crypto/intel/qat/qat_common 2024-03-04T09:13:51.528 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-03-04T09:13:51.528 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-03-04T09:13:51.528 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-03-04T09:13:51.528 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/crypto/padlock-aes.ko 2024-03-04T09:13:51.528 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/crypto/padlock-sha.ko 2024-03-04T09:13:51.528 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/crypto/virtio 2024-03-04T09:13:51.528 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-03-04T09:13:51.529 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/dca 2024-03-04T09:13:51.529 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/dca/dca.ko 2024-03-04T09:13:51.529 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/devfreq 2024-03-04T09:13:51.529 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-03-04T09:13:51.529 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/dma 2024-03-04T09:13:51.529 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/dma/dw 2024-03-04T09:13:51.529 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/dma/dw/dw_dmac.ko 2024-03-04T09:13:51.529 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/dma/ioat 2024-03-04T09:13:51.529 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/dma/ioat/ioatdma.ko 2024-03-04T09:13:51.529 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/edac 2024-03-04T09:13:51.529 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/edac/amd64_edac.ko 2024-03-04T09:13:51.530 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/edac/e752x_edac.ko 2024-03-04T09:13:51.530 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/edac/edac_mce_amd.ko 2024-03-04T09:13:51.530 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/edac/i3000_edac.ko 2024-03-04T09:13:51.530 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/edac/i3200_edac.ko 2024-03-04T09:13:51.530 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/edac/i5100_edac.ko 2024-03-04T09:13:51.530 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/edac/i5400_edac.ko 2024-03-04T09:13:51.530 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/edac/i7300_edac.ko 2024-03-04T09:13:51.530 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/edac/i7core_edac.ko 2024-03-04T09:13:51.530 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/edac/i82975x_edac.ko 2024-03-04T09:13:51.530 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/edac/ie31200_edac.ko 2024-03-04T09:13:51.530 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/edac/sb_edac.ko 2024-03-04T09:13:51.530 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/edac/x38_edac.ko 2024-03-04T09:13:51.530 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/firmware 2024-03-04T09:13:51.530 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/firmware/edd.ko 2024-03-04T09:13:51.531 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/firmware/iscsi_ibft.ko 2024-03-04T09:13:51.531 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/gpio 2024-03-04T09:13:51.531 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/gpio/gpio-ich.ko 2024-03-04T09:13:51.531 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/gpio/gpio-viperboard.ko 2024-03-04T09:13:51.531 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid 2024-03-04T09:13:51.531 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-appleir.ko 2024-03-04T09:13:51.531 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-aureal.ko 2024-03-04T09:13:51.531 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-axff.ko 2024-03-04T09:13:51.531 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-dr.ko 2024-03-04T09:13:51.531 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-elecom.ko 2024-03-04T09:13:51.531 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-elo.ko 2024-03-04T09:13:51.531 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-emsff.ko 2024-03-04T09:13:51.531 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-gaff.ko 2024-03-04T09:13:51.531 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-gt683r.ko 2024-03-04T09:13:51.532 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-gyration.ko 2024-03-04T09:13:51.532 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-holtek-kbd.ko 2024-03-04T09:13:51.532 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-holtek-mouse.ko 2024-03-04T09:13:51.532 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-holtekff.ko 2024-03-04T09:13:51.532 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-hyperv.ko 2024-03-04T09:13:51.532 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-icade.ko 2024-03-04T09:13:51.532 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-keytouch.ko 2024-03-04T09:13:51.532 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-kye.ko 2024-03-04T09:13:51.532 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-lcpower.ko 2024-03-04T09:13:51.532 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-led.ko 2024-03-04T09:13:51.532 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-lenovo.ko 2024-03-04T09:13:51.532 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-logitech-dj.ko 2024-03-04T09:13:51.532 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-03-04T09:13:51.532 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-multitouch.ko 2024-03-04T09:13:51.533 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-ortek.ko 2024-03-04T09:13:51.533 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-penmount.ko 2024-03-04T09:13:51.533 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-petalynx.ko 2024-03-04T09:13:51.533 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-picolcd.ko 2024-03-04T09:13:51.533 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-pl.ko 2024-03-04T09:13:51.533 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-plantronics.ko 2024-03-04T09:13:51.533 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-primax.ko 2024-03-04T09:13:51.533 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-rmi.ko 2024-03-04T09:13:51.533 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-roccat-arvo.ko 2024-03-04T09:13:51.533 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-roccat-common.ko 2024-03-04T09:13:51.533 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-roccat-isku.ko 2024-03-04T09:13:51.533 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-roccat-kone.ko 2024-03-04T09:13:51.533 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-03-04T09:13:51.533 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-roccat-konepure.ko 2024-03-04T09:13:51.534 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-03-04T09:13:51.534 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-roccat-lua.ko 2024-03-04T09:13:51.534 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-roccat-pyra.ko 2024-03-04T09:13:51.534 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-roccat-ryos.ko 2024-03-04T09:13:51.534 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-roccat-savu.ko 2024-03-04T09:13:51.534 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-roccat.ko 2024-03-04T09:13:51.534 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-saitek.ko 2024-03-04T09:13:51.534 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-samsung.ko 2024-03-04T09:13:51.534 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-sensor-hub.ko 2024-03-04T09:13:51.534 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-sjoy.ko 2024-03-04T09:13:51.534 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-sony.ko 2024-03-04T09:13:51.534 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-speedlink.ko 2024-03-04T09:13:51.534 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-steelseries.ko 2024-03-04T09:13:51.534 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-sunplus.ko 2024-03-04T09:13:51.535 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-thrustmaster.ko 2024-03-04T09:13:51.535 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-tivo.ko 2024-03-04T09:13:51.535 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-tmff.ko 2024-03-04T09:13:51.535 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-topseed.ko 2024-03-04T09:13:51.535 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-twinhan.ko 2024-03-04T09:13:51.535 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-uclogic.ko 2024-03-04T09:13:51.535 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-waltop.ko 2024-03-04T09:13:51.535 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-wiimote.ko 2024-03-04T09:13:51.535 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-xinmo.ko 2024-03-04T09:13:51.535 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-zpff.ko 2024-03-04T09:13:51.535 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-zydacron.ko 2024-03-04T09:13:51.535 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/uhid.ko 2024-03-04T09:13:51.535 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/wacom.ko 2024-03-04T09:13:51.535 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hv 2024-03-04T09:13:51.536 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hv/hv_balloon.ko 2024-03-04T09:13:51.536 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hv/hv_utils.ko 2024-03-04T09:13:51.536 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hv/hv_vmbus.ko 2024-03-04T09:13:51.536 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon 2024-03-04T09:13:51.536 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/abituguru.ko 2024-03-04T09:13:51.536 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/abituguru3.ko 2024-03-04T09:13:51.536 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/acpi_power_meter.ko 2024-03-04T09:13:51.536 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/ad7414.ko 2024-03-04T09:13:51.536 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/ad7418.ko 2024-03-04T09:13:51.536 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/adc128d818.ko 2024-03-04T09:13:51.536 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/adm1025.ko 2024-03-04T09:13:51.536 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/adm1026.ko 2024-03-04T09:13:51.536 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/adm1029.ko 2024-03-04T09:13:51.536 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/adm1031.ko 2024-03-04T09:13:51.537 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/adm9240.ko 2024-03-04T09:13:51.537 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/ads7828.ko 2024-03-04T09:13:51.537 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/adt7410.ko 2024-03-04T09:13:51.537 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/adt7411.ko 2024-03-04T09:13:51.537 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/adt7462.ko 2024-03-04T09:13:51.537 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/adt7470.ko 2024-03-04T09:13:51.537 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/adt7475.ko 2024-03-04T09:13:51.537 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/adt7x10.ko 2024-03-04T09:13:51.537 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/amc6821.ko 2024-03-04T09:13:51.537 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/applesmc.ko 2024-03-04T09:13:51.537 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/asb100.ko 2024-03-04T09:13:51.537 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/asc7621.ko 2024-03-04T09:13:51.537 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/asus_atk0110.ko 2024-03-04T09:13:51.537 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/atxp1.ko 2024-03-04T09:13:51.537 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/coretemp.ko 2024-03-04T09:13:51.538 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-03-04T09:13:51.538 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/dme1737.ko 2024-03-04T09:13:51.538 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/ds1621.ko 2024-03-04T09:13:51.538 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/ds620.ko 2024-03-04T09:13:51.538 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/emc1403.ko 2024-03-04T09:13:51.538 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/emc6w201.ko 2024-03-04T09:13:51.538 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/f71805f.ko 2024-03-04T09:13:51.538 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/f71882fg.ko 2024-03-04T09:13:51.538 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/f75375s.ko 2024-03-04T09:13:51.538 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/fam15h_power.ko 2024-03-04T09:13:51.538 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/fschmd.ko 2024-03-04T09:13:51.538 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/g760a.ko 2024-03-04T09:13:51.538 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/g762.ko 2024-03-04T09:13:51.538 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/gl518sm.ko 2024-03-04T09:13:51.538 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/gl520sm.ko 2024-03-04T09:13:51.539 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/hwmon-vid.ko 2024-03-04T09:13:51.539 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/i5500_temp.ko 2024-03-04T09:13:51.539 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/i5k_amb.ko 2024-03-04T09:13:51.539 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/ibmaem.ko 2024-03-04T09:13:51.539 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/ibmpex.ko 2024-03-04T09:13:51.539 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/ina209.ko 2024-03-04T09:13:51.539 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/ina2xx.ko 2024-03-04T09:13:51.539 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/it87.ko 2024-03-04T09:13:51.539 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/k10temp.ko 2024-03-04T09:13:51.539 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/k8temp.ko 2024-03-04T09:13:51.539 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/lineage-pem.ko 2024-03-04T09:13:51.539 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/lm63.ko 2024-03-04T09:13:51.539 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/lm73.ko 2024-03-04T09:13:51.540 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/lm75.ko 2024-03-04T09:13:51.540 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/lm77.ko 2024-03-04T09:13:51.540 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/lm78.ko 2024-03-04T09:13:51.540 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/lm80.ko 2024-03-04T09:13:51.540 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/lm83.ko 2024-03-04T09:13:51.540 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/lm85.ko 2024-03-04T09:13:51.540 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/lm87.ko 2024-03-04T09:13:51.540 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/lm90.ko 2024-03-04T09:13:51.540 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/lm92.ko 2024-03-04T09:13:51.540 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/lm93.ko 2024-03-04T09:13:51.540 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/lm95234.ko 2024-03-04T09:13:51.540 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/lm95241.ko 2024-03-04T09:13:51.540 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/lm95245.ko 2024-03-04T09:13:51.540 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/ltc2945.ko 2024-03-04T09:13:51.540 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/ltc4151.ko 2024-03-04T09:13:51.541 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/ltc4215.ko 2024-03-04T09:13:51.541 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/ltc4222.ko 2024-03-04T09:13:51.541 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/ltc4245.ko 2024-03-04T09:13:51.541 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/ltc4260.ko 2024-03-04T09:13:51.541 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/ltc4261.ko 2024-03-04T09:13:51.541 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/max16065.ko 2024-03-04T09:13:51.541 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/max1619.ko 2024-03-04T09:13:51.541 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/max1668.ko 2024-03-04T09:13:51.541 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/max197.ko 2024-03-04T09:13:51.541 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/max6639.ko 2024-03-04T09:13:51.541 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/max6650.ko 2024-03-04T09:13:51.541 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/max6697.ko 2024-03-04T09:13:51.541 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/mcp3021.ko 2024-03-04T09:13:51.541 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/nct6683.ko 2024-03-04T09:13:51.542 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/nct6775-core.ko 2024-03-04T09:13:51.542 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/nct6775.ko 2024-03-04T09:13:51.542 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/nct7802.ko 2024-03-04T09:13:51.542 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/pc87360.ko 2024-03-04T09:13:51.542 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/pc87427.ko 2024-03-04T09:13:51.542 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/pcf8591.ko 2024-03-04T09:13:51.542 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/pmbus 2024-03-04T09:13:51.542 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-03-04T09:13:51.542 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-03-04T09:13:51.542 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-03-04T09:13:51.542 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/pmbus/max16064.ko 2024-03-04T09:13:51.542 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/pmbus/max34440.ko 2024-03-04T09:13:51.542 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/pmbus/max8688.ko 2024-03-04T09:13:51.542 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-03-04T09:13:51.543 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-03-04T09:13:51.543 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-03-04T09:13:51.543 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-03-04T09:13:51.543 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-03-04T09:13:51.543 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-03-04T09:13:51.543 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/powr1220.ko 2024-03-04T09:13:51.543 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/sch5627.ko 2024-03-04T09:13:51.543 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/sch5636.ko 2024-03-04T09:13:51.543 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/sch56xx-common.ko 2024-03-04T09:13:51.543 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/sht15.ko 2024-03-04T09:13:51.543 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/sht21.ko 2024-03-04T09:13:51.543 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/shtc1.ko 2024-03-04T09:13:51.543 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/sis5595.ko 2024-03-04T09:13:51.543 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/smsc47b397.ko 2024-03-04T09:13:51.543 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/smsc47m1.ko 2024-03-04T09:13:51.544 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/smsc47m192.ko 2024-03-04T09:13:51.544 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/thmc50.ko 2024-03-04T09:13:51.544 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/tmp102.ko 2024-03-04T09:13:51.544 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/tmp103.ko 2024-03-04T09:13:51.544 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/tmp401.ko 2024-03-04T09:13:51.544 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/tmp421.ko 2024-03-04T09:13:51.544 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/via-cputemp.ko 2024-03-04T09:13:51.544 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/via686a.ko 2024-03-04T09:13:51.544 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/vt1211.ko 2024-03-04T09:13:51.544 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/vt8231.ko 2024-03-04T09:13:51.544 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/w83627ehf.ko 2024-03-04T09:13:51.544 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/w83627hf.ko 2024-03-04T09:13:51.544 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/w83781d.ko 2024-03-04T09:13:51.544 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/w83791d.ko 2024-03-04T09:13:51.545 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/w83792d.ko 2024-03-04T09:13:51.545 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/w83793.ko 2024-03-04T09:13:51.545 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/w83795.ko 2024-03-04T09:13:51.545 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/w83l785ts.ko 2024-03-04T09:13:51.545 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/w83l786ng.ko 2024-03-04T09:13:51.545 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/i2c 2024-03-04T09:13:51.545 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/i2c/algos 2024-03-04T09:13:51.545 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-03-04T09:13:51.545 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-03-04T09:13:51.545 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/i2c/busses 2024-03-04T09:13:51.545 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-03-04T09:13:51.545 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-03-04T09:13:51.545 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-03-04T09:13:51.545 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-03-04T09:13:51.545 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-03-04T09:13:51.546 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-03-04T09:13:51.546 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-03-04T09:13:51.546 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-03-04T09:13:51.546 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/i2c/busses/i2c-i801.ko 2024-03-04T09:13:51.546 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/i2c/busses/i2c-isch.ko 2024-03-04T09:13:51.546 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-03-04T09:13:51.546 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-03-04T09:13:51.546 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-03-04T09:13:51.546 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/i2c/busses/i2c-parport.ko 2024-03-04T09:13:51.546 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-03-04T09:13:51.546 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-03-04T09:13:51.546 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-03-04T09:13:51.546 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-03-04T09:13:51.546 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-03-04T09:13:51.546 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-03-04T09:13:51.547 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/i2c/busses/i2c-via.ko 2024-03-04T09:13:51.547 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-03-04T09:13:51.547 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-03-04T09:13:51.547 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/i2c/i2c-dev.ko 2024-03-04T09:13:51.547 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/i2c/i2c-mux.ko 2024-03-04T09:13:51.547 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-03-04T09:13:51.547 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/i2c/i2c-smbus.ko 2024-03-04T09:13:51.547 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/i2c/i2c-stub.ko 2024-03-04T09:13:51.547 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/infiniband 2024-03-04T09:13:51.547 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/infiniband/core 2024-03-04T09:13:51.547 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/infiniband/core/ib_cm.ko 2024-03-04T09:13:51.547 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/infiniband/core/ib_core.ko 2024-03-04T09:13:51.547 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/infiniband/core/ib_umad.ko 2024-03-04T09:13:51.547 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-03-04T09:13:51.547 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/infiniband/core/iw_cm.ko 2024-03-04T09:13:51.548 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/infiniband/core/rdma_cm.ko 2024-03-04T09:13:51.548 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-03-04T09:13:51.548 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/infiniband/hw 2024-03-04T09:13:51.548 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/infiniband/hw/cxgb4 2024-03-04T09:13:51.548 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-03-04T09:13:51.548 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/infiniband/hw/mlx4 2024-03-04T09:13:51.548 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-03-04T09:13:51.548 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/infiniband/hw/mlx5 2024-03-04T09:13:51.548 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-03-04T09:13:51.548 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/infiniband/hw/mthca 2024-03-04T09:13:51.548 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-03-04T09:13:51.548 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/infiniband/ulp 2024-03-04T09:13:51.548 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/infiniband/ulp/ipoib 2024-03-04T09:13:51.548 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-03-04T09:13:51.549 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/infiniband/ulp/iser 2024-03-04T09:13:51.549 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-03-04T09:13:51.549 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/infiniband/ulp/isert 2024-03-04T09:13:51.549 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-03-04T09:13:51.549 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/infiniband/ulp/srp 2024-03-04T09:13:51.549 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-03-04T09:13:51.549 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/infiniband/ulp/srpt 2024-03-04T09:13:51.549 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-03-04T09:13:51.549 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/input 2024-03-04T09:13:51.549 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/input/gameport 2024-03-04T09:13:51.549 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-03-04T09:13:51.549 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/input/gameport/fm801-gp.ko 2024-03-04T09:13:51.549 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/input/gameport/gameport.ko 2024-03-04T09:13:51.549 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/input/joydev.ko 2024-03-04T09:13:51.549 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/input/keyboard 2024-03-04T09:13:51.550 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/input/keyboard/gpio_keys.ko 2024-03-04T09:13:51.550 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/input/mouse 2024-03-04T09:13:51.550 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/input/mouse/appletouch.ko 2024-03-04T09:13:51.550 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/input/mouse/bcm5974.ko 2024-03-04T09:13:51.550 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/input/mouse/cyapatp.ko 2024-03-04T09:13:51.550 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/input/mouse/elan_i2c.ko 2024-03-04T09:13:51.550 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/input/mouse/sermouse.ko 2024-03-04T09:13:51.550 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-03-04T09:13:51.550 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/input/mouse/synaptics_usb.ko 2024-03-04T09:13:51.550 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/input/mouse/vsxxxaa.ko 2024-03-04T09:13:51.550 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/input/rmi4 2024-03-04T09:13:51.550 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/input/rmi4/rmi_core.ko 2024-03-04T09:13:51.550 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/input/serio 2024-03-04T09:13:51.550 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/input/serio/altera_ps2.ko 2024-03-04T09:13:51.550 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/input/serio/arc_ps2.ko 2024-03-04T09:13:51.551 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-03-04T09:13:51.551 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/input/serio/serio_raw.ko 2024-03-04T09:13:51.551 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/input/sparse-keymap.ko 2024-03-04T09:13:51.551 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/leds 2024-03-04T09:13:51.551 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/leds/leds-blinkm.ko 2024-03-04T09:13:51.551 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/leds/leds-lm3530.ko 2024-03-04T09:13:51.551 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/leds/leds-lp3944.ko 2024-03-04T09:13:51.551 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/leds/leds-ss4200.ko 2024-03-04T09:13:51.551 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/leds/trigger 2024-03-04T09:13:51.551 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-03-04T09:13:51.551 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-03-04T09:13:51.551 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-03-04T09:13:51.551 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-03-04T09:13:51.551 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-03-04T09:13:51.551 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-03-04T09:13:51.551 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-03-04T09:13:51.552 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-03-04T09:13:51.552 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-03-04T09:13:51.552 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/md 2024-03-04T09:13:51.552 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/md/bcache 2024-03-04T09:13:51.552 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/md/bcache/bcache.ko 2024-03-04T09:13:51.552 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/md/dm-bio-prison.ko 2024-03-04T09:13:51.552 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/md/dm-cache-smq.ko 2024-03-04T09:13:51.552 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/md/dm-cache.ko 2024-03-04T09:13:51.552 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/md/dm-crypt.ko 2024-03-04T09:13:51.552 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/md/dm-delay.ko 2024-03-04T09:13:51.552 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/md/dm-flakey.ko 2024-03-04T09:13:51.552 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/md/dm-log-userspace.ko 2024-03-04T09:13:51.552 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/md/dm-multipath.ko 2024-03-04T09:13:51.552 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/md/dm-queue-length.ko 2024-03-04T09:13:51.553 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/md/dm-raid.ko 2024-03-04T09:13:51.553 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/md/dm-round-robin.ko 2024-03-04T09:13:51.553 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/md/dm-service-time.ko 2024-03-04T09:13:51.553 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/md/dm-switch.ko 2024-03-04T09:13:51.553 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/md/dm-thin-pool.ko 2024-03-04T09:13:51.553 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/md/dm-verity.ko 2024-03-04T09:13:51.553 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/md/persistent-data 2024-03-04T09:13:51.553 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-03-04T09:13:51.553 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/md/raid0.ko 2024-03-04T09:13:51.553 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/md/raid1.ko 2024-03-04T09:13:51.553 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/md/raid10.ko 2024-03-04T09:13:51.553 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/md/raid456.ko 2024-03-04T09:13:51.553 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/message 2024-03-04T09:13:51.553 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/message/fusion 2024-03-04T09:13:51.553 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/message/fusion/mptbase.ko 2024-03-04T09:13:51.554 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/message/fusion/mptctl.ko 2024-03-04T09:13:51.554 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/message/fusion/mptfc.ko 2024-03-04T09:13:51.554 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/message/fusion/mptlan.ko 2024-03-04T09:13:51.554 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/message/fusion/mptsas.ko 2024-03-04T09:13:51.554 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/message/fusion/mptscsih.ko 2024-03-04T09:13:51.554 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/message/fusion/mptspi.ko 2024-03-04T09:13:51.554 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/mfd 2024-03-04T09:13:51.554 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/mfd/lpc_ich.ko 2024-03-04T09:13:51.554 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/mfd/lpc_sch.ko 2024-03-04T09:13:51.554 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/mfd/mfd-core.ko 2024-03-04T09:13:51.554 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/mfd/sm501.ko 2024-03-04T09:13:51.554 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/mfd/viperboard.ko 2024-03-04T09:13:51.554 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/mfd/vx855.ko 2024-03-04T09:13:51.554 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/mfd/wl1273-core.ko 2024-03-04T09:13:51.554 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/misc 2024-03-04T09:13:51.555 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/misc/altera-stapl 2024-03-04T09:13:51.555 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-03-04T09:13:51.555 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/misc/apds9802als.ko 2024-03-04T09:13:51.555 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/misc/apds990x.ko 2024-03-04T09:13:51.555 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/misc/bh1770glc.ko 2024-03-04T09:13:51.555 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/misc/cb710 2024-03-04T09:13:51.555 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/misc/cb710/cb710.ko 2024-03-04T09:13:51.555 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/misc/eeprom 2024-03-04T09:13:51.555 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/misc/eeprom/at24.ko 2024-03-04T09:13:51.555 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-03-04T09:13:51.555 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/misc/eeprom/max6875.ko 2024-03-04T09:13:51.555 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/misc/enclosure.ko 2024-03-04T09:13:51.555 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/misc/hpilo.ko 2024-03-04T09:13:51.555 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/misc/isl29003.ko 2024-03-04T09:13:51.555 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/misc/isl29020.ko 2024-03-04T09:13:51.556 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/misc/lis3lv02d 2024-03-04T09:13:51.556 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-03-04T09:13:51.556 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-03-04T09:13:51.556 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/misc/mei 2024-03-04T09:13:51.556 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/misc/mei/mei-me.ko 2024-03-04T09:13:51.556 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/misc/mei/mei-txe.ko 2024-03-04T09:13:51.556 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/misc/mei/mei.ko 2024-03-04T09:13:51.556 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/misc/sgi-gru 2024-03-04T09:13:51.556 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/misc/sgi-gru/gru.ko 2024-03-04T09:13:51.556 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/misc/sgi-xp 2024-03-04T09:13:51.556 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/misc/sgi-xp/xp.ko 2024-03-04T09:13:51.556 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/misc/sgi-xp/xpc.ko 2024-03-04T09:13:51.556 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-03-04T09:13:51.556 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/misc/tifm_7xx1.ko 2024-03-04T09:13:51.556 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/misc/tifm_core.ko 2024-03-04T09:13:51.557 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/misc/tsl2550.ko 2024-03-04T09:13:51.557 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/misc/vmw_balloon.ko 2024-03-04T09:13:51.557 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/misc/vmw_vmci 2024-03-04T09:13:51.557 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-03-04T09:13:51.557 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net 2024-03-04T09:13:51.557 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/bonding 2024-03-04T09:13:51.557 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/bonding/bonding.ko 2024-03-04T09:13:51.557 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/dsa 2024-03-04T09:13:51.557 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/dsa/b53 2024-03-04T09:13:51.557 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/dsa/b53/b53_common.ko 2024-03-04T09:13:51.557 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/dsa/bcm-sf2.ko 2024-03-04T09:13:51.557 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/dsa/mv88e6060.ko 2024-03-04T09:13:51.557 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/dsa/mv88e6xxx 2024-03-04T09:13:51.557 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-03-04T09:13:51.557 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/dummy.ko 2024-03-04T09:13:51.558 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/eql.ko 2024-03-04T09:13:51.558 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet 2024-03-04T09:13:51.558 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/3com 2024-03-04T09:13:51.558 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-03-04T09:13:51.558 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-03-04T09:13:51.558 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-03-04T09:13:51.558 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-03-04T09:13:51.558 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/8390 2024-03-04T09:13:51.558 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/8390/8390.ko 2024-03-04T09:13:51.558 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-03-04T09:13:51.558 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-03-04T09:13:51.558 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-03-04T09:13:51.558 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/adaptec 2024-03-04T09:13:51.558 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-03-04T09:13:51.558 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/agere 2024-03-04T09:13:51.559 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/agere/et131x.ko 2024-03-04T09:13:51.559 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/alteon 2024-03-04T09:13:51.559 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-03-04T09:13:51.559 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/altera 2024-03-04T09:13:51.559 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-03-04T09:13:51.559 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/amd 2024-03-04T09:13:51.559 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-03-04T09:13:51.559 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-03-04T09:13:51.559 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-03-04T09:13:51.559 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/atheros 2024-03-04T09:13:51.559 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/atheros/alx 2024-03-04T09:13:51.559 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-03-04T09:13:51.559 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/atheros/atl1c 2024-03-04T09:13:51.559 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-03-04T09:13:51.559 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/atheros/atl1e 2024-03-04T09:13:51.560 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-03-04T09:13:51.560 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/atheros/atlx 2024-03-04T09:13:51.560 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-03-04T09:13:51.560 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-03-04T09:13:51.560 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/broadcom 2024-03-04T09:13:51.560 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-03-04T09:13:51.560 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-03-04T09:13:51.560 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-03-04T09:13:51.560 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-03-04T09:13:51.560 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-03-04T09:13:51.560 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/broadcom/genet 2024-03-04T09:13:51.560 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-03-04T09:13:51.560 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-03-04T09:13:51.560 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/brocade 2024-03-04T09:13:51.560 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/brocade/bna 2024-03-04T09:13:51.561 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-03-04T09:13:51.561 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/chelsio 2024-03-04T09:13:51.561 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/chelsio/cxgb 2024-03-04T09:13:51.561 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-03-04T09:13:51.561 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-03-04T09:13:51.561 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-03-04T09:13:51.561 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-03-04T09:13:51.561 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-03-04T09:13:51.561 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-03-04T09:13:51.561 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-03-04T09:13:51.561 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-03-04T09:13:51.561 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-03-04T09:13:51.561 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/cisco 2024-03-04T09:13:51.561 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/cisco/enic 2024-03-04T09:13:51.561 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-03-04T09:13:51.562 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/dec 2024-03-04T09:13:51.562 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/dec/tulip 2024-03-04T09:13:51.562 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-03-04T09:13:51.562 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-03-04T09:13:51.562 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-03-04T09:13:51.562 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-03-04T09:13:51.562 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-03-04T09:13:51.562 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-03-04T09:13:51.562 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/dlink 2024-03-04T09:13:51.562 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-03-04T09:13:51.562 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-03-04T09:13:51.562 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/dnet.ko 2024-03-04T09:13:51.562 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/emulex 2024-03-04T09:13:51.562 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/emulex/benet 2024-03-04T09:13:51.562 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-03-04T09:13:51.562 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/ethoc.ko 2024-03-04T09:13:51.563 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/fealnx.ko 2024-03-04T09:13:51.563 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/intel 2024-03-04T09:13:51.563 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/intel/e100.ko 2024-03-04T09:13:51.563 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/intel/e1000 2024-03-04T09:13:51.563 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-03-04T09:13:51.563 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/intel/e1000e 2024-03-04T09:13:51.563 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-03-04T09:13:51.563 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/intel/fm10k 2024-03-04T09:13:51.563 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-03-04T09:13:51.563 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/intel/i40e 2024-03-04T09:13:51.563 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-03-04T09:13:51.563 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/intel/iavf 2024-03-04T09:13:51.563 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-03-04T09:13:51.563 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/intel/igb 2024-03-04T09:13:51.564 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-03-04T09:13:51.564 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/intel/igbvf 2024-03-04T09:13:51.564 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-03-04T09:13:51.564 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/intel/ixgbe 2024-03-04T09:13:51.564 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-03-04T09:13:51.564 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/intel/ixgbevf 2024-03-04T09:13:51.564 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-03-04T09:13:51.564 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/jme.ko 2024-03-04T09:13:51.564 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/marvell 2024-03-04T09:13:51.564 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-03-04T09:13:51.564 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/marvell/skge.ko 2024-03-04T09:13:51.564 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-03-04T09:13:51.564 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/mellanox 2024-03-04T09:13:51.564 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/mellanox/mlx4 2024-03-04T09:13:51.564 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-03-04T09:13:51.565 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-03-04T09:13:51.565 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/mellanox/mlx5 2024-03-04T09:13:51.565 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-03-04T09:13:51.565 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-03-04T09:13:51.565 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/micrel 2024-03-04T09:13:51.565 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-03-04T09:13:51.565 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/myricom 2024-03-04T09:13:51.565 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/myricom/myri10ge 2024-03-04T09:13:51.565 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-03-04T09:13:51.565 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/natsemi 2024-03-04T09:13:51.565 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-03-04T09:13:51.565 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-03-04T09:13:51.565 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/neterion 2024-03-04T09:13:51.565 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-03-04T09:13:51.565 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/nvidia 2024-03-04T09:13:51.566 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-03-04T09:13:51.566 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/packetengines 2024-03-04T09:13:51.566 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-03-04T09:13:51.566 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-03-04T09:13:51.566 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/qlogic 2024-03-04T09:13:51.566 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/qlogic/netxen 2024-03-04T09:13:51.566 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-03-04T09:13:51.566 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-03-04T09:13:51.566 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-03-04T09:13:51.566 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-03-04T09:13:51.566 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/rdc 2024-03-04T09:13:51.566 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-03-04T09:13:51.566 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/realtek 2024-03-04T09:13:51.566 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-03-04T09:13:51.566 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-03-04T09:13:51.567 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/realtek/atp.ko 2024-03-04T09:13:51.567 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-03-04T09:13:51.567 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/rocker 2024-03-04T09:13:51.567 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-03-04T09:13:51.567 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/sfc 2024-03-04T09:13:51.567 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-03-04T09:13:51.567 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/silan 2024-03-04T09:13:51.567 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-03-04T09:13:51.567 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/sis 2024-03-04T09:13:51.567 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/sis/sis190.ko 2024-03-04T09:13:51.567 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/sis/sis900.ko 2024-03-04T09:13:51.567 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/smsc 2024-03-04T09:13:51.567 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-03-04T09:13:51.567 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-03-04T09:13:51.567 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-03-04T09:13:51.568 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-03-04T09:13:51.568 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/stmicro 2024-03-04T09:13:51.568 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/stmicro/stmmac 2024-03-04T09:13:51.568 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-03-04T09:13:51.568 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-03-04T09:13:51.568 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/sun 2024-03-04T09:13:51.568 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/sun/cassini.ko 2024-03-04T09:13:51.568 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/sun/niu.ko 2024-03-04T09:13:51.568 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/sun/sungem.ko 2024-03-04T09:13:51.568 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-03-04T09:13:51.568 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/tehuti 2024-03-04T09:13:51.568 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-03-04T09:13:51.568 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/ti 2024-03-04T09:13:51.568 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/ti/tlan.ko 2024-03-04T09:13:51.569 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/via 2024-03-04T09:13:51.569 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-03-04T09:13:51.569 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-03-04T09:13:51.569 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/wiznet 2024-03-04T09:13:51.569 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-03-04T09:13:51.569 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-03-04T09:13:51.569 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/xircom 2024-03-04T09:13:51.569 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-03-04T09:13:51.569 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/geneve.ko 2024-03-04T09:13:51.569 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/hyperv 2024-03-04T09:13:51.569 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-03-04T09:13:51.569 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ieee802154 2024-03-04T09:13:51.569 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ieee802154/fakelb.ko 2024-03-04T09:13:51.569 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ifb.ko 2024-03-04T09:13:51.569 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ipvlan 2024-03-04T09:13:51.569 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ipvlan/ipvlan.ko 2024-03-04T09:13:51.570 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/macvlan.ko 2024-03-04T09:13:51.570 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/macvtap.ko 2024-03-04T09:13:51.570 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/mdio 2024-03-04T09:13:51.570 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/mdio.ko 2024-03-04T09:13:51.570 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-03-04T09:13:51.570 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-03-04T09:13:51.570 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/mdio/mdio-regmap.ko 2024-03-04T09:13:51.570 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/mii.ko 2024-03-04T09:13:51.570 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/net_failover.ko 2024-03-04T09:13:51.570 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/netconsole.ko 2024-03-04T09:13:51.570 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/nlmon.ko 2024-03-04T09:13:51.570 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/pcs 2024-03-04T09:13:51.570 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/pcs/pcs-lynx.ko 2024-03-04T09:13:51.570 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-03-04T09:13:51.571 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/phy 2024-03-04T09:13:51.571 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/phy/amd.ko 2024-03-04T09:13:51.571 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-03-04T09:13:51.571 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-03-04T09:13:51.571 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/phy/bcm7xxx.ko 2024-03-04T09:13:51.571 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/phy/bcm87xx.ko 2024-03-04T09:13:51.571 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/phy/broadcom.ko 2024-03-04T09:13:51.571 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/phy/cicada.ko 2024-03-04T09:13:51.571 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/phy/davicom.ko 2024-03-04T09:13:51.571 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/phy/dp83640.ko 2024-03-04T09:13:51.571 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/phy/et1011c.ko 2024-03-04T09:13:51.571 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/phy/icplus.ko 2024-03-04T09:13:51.571 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/phy/lxt.ko 2024-03-04T09:13:51.571 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/phy/marvell.ko 2024-03-04T09:13:51.571 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/phy/micrel.ko 2024-03-04T09:13:51.571 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/phy/national.ko 2024-03-04T09:13:51.572 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/phy/phylink.ko 2024-03-04T09:13:51.572 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/phy/qsemi.ko 2024-03-04T09:13:51.572 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/phy/realtek.ko 2024-03-04T09:13:51.572 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/phy/smsc.ko 2024-03-04T09:13:51.572 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/phy/ste10Xp.ko 2024-03-04T09:13:51.572 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/phy/vitesse.ko 2024-03-04T09:13:51.572 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ppp 2024-03-04T09:13:51.572 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ppp/bsd_comp.ko 2024-03-04T09:13:51.572 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ppp/ppp_async.ko 2024-03-04T09:13:51.572 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ppp/ppp_deflate.ko 2024-03-04T09:13:51.572 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ppp/ppp_generic.ko 2024-03-04T09:13:51.572 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ppp/ppp_mppe.ko 2024-03-04T09:13:51.572 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ppp/ppp_synctty.ko 2024-03-04T09:13:51.572 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ppp/pppoe.ko 2024-03-04T09:13:51.573 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ppp/pppox.ko 2024-03-04T09:13:51.573 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ppp/pptp.ko 2024-03-04T09:13:51.573 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/slip 2024-03-04T09:13:51.573 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/slip/slhc.ko 2024-03-04T09:13:51.573 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/slip/slip.ko 2024-03-04T09:13:51.573 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/sungem_phy.ko 2024-03-04T09:13:51.573 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/tap.ko 2024-03-04T09:13:51.573 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/team 2024-03-04T09:13:51.573 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/team/team.ko 2024-03-04T09:13:51.573 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/team/team_mode_activebackup.ko 2024-03-04T09:13:51.573 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/team/team_mode_broadcast.ko 2024-03-04T09:13:51.573 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-03-04T09:13:51.573 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/team/team_mode_random.ko 2024-03-04T09:13:51.573 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-03-04T09:13:51.573 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/tun.ko 2024-03-04T09:13:51.573 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/veth.ko 2024-03-04T09:13:51.574 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/virtio_net.ko 2024-03-04T09:13:51.574 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/vmxnet3 2024-03-04T09:13:51.574 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-03-04T09:13:51.574 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/vxlan 2024-03-04T09:13:51.574 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/vxlan/vxlan.ko 2024-03-04T09:13:51.574 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/xen-netback 2024-03-04T09:13:51.574 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/xen-netback/xen-netback.ko 2024-03-04T09:13:51.574 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/xen-netfront.ko 2024-03-04T09:13:51.574 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ntb 2024-03-04T09:13:51.574 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ntb/ntb.ko 2024-03-04T09:13:51.574 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/nvme 2024-03-04T09:13:51.574 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/nvme/host 2024-03-04T09:13:51.574 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/nvme/host/nvme-core.ko 2024-03-04T09:13:51.574 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/nvme/host/nvme.ko 2024-03-04T09:13:51.574 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/parport 2024-03-04T09:13:51.574 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/parport/parport.ko 2024-03-04T09:13:51.575 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/parport/parport_cs.ko 2024-03-04T09:13:51.575 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/parport/parport_pc.ko 2024-03-04T09:13:51.575 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/parport/parport_serial.ko 2024-03-04T09:13:51.575 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/pci 2024-03-04T09:13:51.575 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/pci/hotplug 2024-03-04T09:13:51.575 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-03-04T09:13:51.575 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/pci/pcie 2024-03-04T09:13:51.575 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/pci/pcie/aer_inject.ko 2024-03-04T09:13:51.575 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/pci/xen-pcifront.ko 2024-03-04T09:13:51.575 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/pcmcia 2024-03-04T09:13:51.575 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/pcmcia/i82092.ko 2024-03-04T09:13:51.575 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/pcmcia/pd6729.ko 2024-03-04T09:13:51.575 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/pcmcia/yenta_socket.ko 2024-03-04T09:13:51.575 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/pinctrl 2024-03-04T09:13:51.576 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/pinctrl/intel 2024-03-04T09:13:51.576 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-03-04T09:13:51.576 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/platform 2024-03-04T09:13:51.576 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/platform/x86 2024-03-04T09:13:51.576 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/platform/x86/acer-wmi.ko 2024-03-04T09:13:51.576 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/platform/x86/acerhdf.ko 2024-03-04T09:13:51.576 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/platform/x86/apple-gmux.ko 2024-03-04T09:13:51.576 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/platform/x86/asus-laptop.ko 2024-03-04T09:13:51.576 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-03-04T09:13:51.576 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/platform/x86/asus-wmi.ko 2024-03-04T09:13:51.576 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/platform/x86/classmate-laptop.ko 2024-03-04T09:13:51.576 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-03-04T09:13:51.576 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-03-04T09:13:51.576 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-03-04T09:13:51.576 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-03-04T09:13:51.577 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/platform/x86/hdaps.ko 2024-03-04T09:13:51.577 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/platform/x86/intel 2024-03-04T09:13:51.577 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-03-04T09:13:51.577 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/platform/x86/intel_ips.ko 2024-03-04T09:13:51.577 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/platform/x86/msi-wmi.ko 2024-03-04T09:13:51.577 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/platform/x86/mxm-wmi.ko 2024-03-04T09:13:51.577 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-03-04T09:13:51.577 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/platform/x86/samsung-laptop.ko 2024-03-04T09:13:51.577 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/platform/x86/samsung-q10.ko 2024-03-04T09:13:51.577 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/platform/x86/topstar-laptop.ko 2024-03-04T09:13:51.577 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-03-04T09:13:51.577 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/platform/x86/toshiba_haps.ko 2024-03-04T09:13:51.577 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/platform/x86/wmi-bmof.ko 2024-03-04T09:13:51.577 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/platform/x86/wmi.ko 2024-03-04T09:13:51.578 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/powercap 2024-03-04T09:13:51.578 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/powercap/intel_rapl_common.ko 2024-03-04T09:13:51.578 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/powercap/intel_rapl_msr.ko 2024-03-04T09:13:51.578 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/pps 2024-03-04T09:13:51.578 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/pps/clients 2024-03-04T09:13:51.578 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/pps/clients/pps-gpio.ko 2024-03-04T09:13:51.578 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/pps/clients/pps-ldisc.ko 2024-03-04T09:13:51.578 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/pps/clients/pps_parport.ko 2024-03-04T09:13:51.578 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/pps/pps_core.ko 2024-03-04T09:13:51.578 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ptp 2024-03-04T09:13:51.578 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ptp/ptp.ko 2024-03-04T09:13:51.578 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ptp/ptp_kvm.ko 2024-03-04T09:13:51.578 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/rtc 2024-03-04T09:13:51.578 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/rtc/rtc-bq32k.ko 2024-03-04T09:13:51.578 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/rtc/rtc-ds1286.ko 2024-03-04T09:13:51.578 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/rtc/rtc-ds1307.ko 2024-03-04T09:13:51.579 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/rtc/rtc-ds1374.ko 2024-03-04T09:13:51.579 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/rtc/rtc-ds1511.ko 2024-03-04T09:13:51.579 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/rtc/rtc-ds1553.ko 2024-03-04T09:13:51.579 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/rtc/rtc-ds1672.ko 2024-03-04T09:13:51.579 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/rtc/rtc-ds1742.ko 2024-03-04T09:13:51.579 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/rtc/rtc-ds2404.ko 2024-03-04T09:13:51.579 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/rtc/rtc-ds3232.ko 2024-03-04T09:13:51.579 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/rtc/rtc-em3027.ko 2024-03-04T09:13:51.579 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/rtc/rtc-fm3130.ko 2024-03-04T09:13:51.579 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/rtc/rtc-isl12022.ko 2024-03-04T09:13:51.579 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/rtc/rtc-isl1208.ko 2024-03-04T09:13:51.579 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/rtc/rtc-m41t80.ko 2024-03-04T09:13:51.579 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/rtc/rtc-m48t35.ko 2024-03-04T09:13:51.579 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/rtc/rtc-m48t59.ko 2024-03-04T09:13:51.579 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/rtc/rtc-max6900.ko 2024-03-04T09:13:51.580 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/rtc/rtc-msm6242.ko 2024-03-04T09:13:51.580 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/rtc/rtc-pcf2127.ko 2024-03-04T09:13:51.580 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/rtc/rtc-pcf85063.ko 2024-03-04T09:13:51.580 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/rtc/rtc-pcf8523.ko 2024-03-04T09:13:51.580 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/rtc/rtc-pcf8563.ko 2024-03-04T09:13:51.580 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/rtc/rtc-pcf8583.ko 2024-03-04T09:13:51.580 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/rtc/rtc-rp5c01.ko 2024-03-04T09:13:51.580 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/rtc/rtc-rs5c372.ko 2024-03-04T09:13:51.580 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-03-04T09:13:51.580 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/rtc/rtc-rx8025.ko 2024-03-04T09:13:51.580 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/rtc/rtc-rx8581.ko 2024-03-04T09:13:51.580 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-03-04T09:13:51.580 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/rtc/rtc-x1205.ko 2024-03-04T09:13:51.580 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi 2024-03-04T09:13:51.580 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/3w-9xxx.ko 2024-03-04T09:13:51.581 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/3w-sas.ko 2024-03-04T09:13:51.581 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/3w-xxxx.ko 2024-03-04T09:13:51.581 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/BusLogic.ko 2024-03-04T09:13:51.581 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/a100u2w.ko 2024-03-04T09:13:51.581 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/aacraid 2024-03-04T09:13:51.581 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/aacraid/aacraid.ko 2024-03-04T09:13:51.581 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/advansys.ko 2024-03-04T09:13:51.581 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/aic7xxx 2024-03-04T09:13:51.581 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-03-04T09:13:51.581 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-03-04T09:13:51.581 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/am53c974.ko 2024-03-04T09:13:51.581 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/arcmsr 2024-03-04T09:13:51.581 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-03-04T09:13:51.581 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/atp870u.ko 2024-03-04T09:13:51.581 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/be2iscsi 2024-03-04T09:13:51.582 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-03-04T09:13:51.582 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/bfa 2024-03-04T09:13:51.582 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/bfa/bfa.ko 2024-03-04T09:13:51.582 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/bnx2fc 2024-03-04T09:13:51.582 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-03-04T09:13:51.582 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/bnx2i 2024-03-04T09:13:51.582 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-03-04T09:13:51.582 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/ch.ko 2024-03-04T09:13:51.582 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/csiostor 2024-03-04T09:13:51.582 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/csiostor/csiostor.ko 2024-03-04T09:13:51.582 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/cxgbi 2024-03-04T09:13:51.582 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/cxgbi/cxgb3i 2024-03-04T09:13:51.582 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-03-04T09:13:51.582 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/cxgbi/cxgb4i 2024-03-04T09:13:51.582 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-03-04T09:13:51.583 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-03-04T09:13:51.583 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/dc395x.ko 2024-03-04T09:13:51.583 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/device_handler 2024-03-04T09:13:51.583 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-03-04T09:13:51.583 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-03-04T09:13:51.583 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-03-04T09:13:51.583 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-03-04T09:13:51.583 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/dmx3191d.ko 2024-03-04T09:13:51.583 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/esas2r 2024-03-04T09:13:51.583 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/esas2r/esas2r.ko 2024-03-04T09:13:51.583 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/esp_scsi.ko 2024-03-04T09:13:51.583 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/fcoe 2024-03-04T09:13:51.583 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/fcoe/fcoe.ko 2024-03-04T09:13:51.583 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-03-04T09:13:51.584 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/fnic 2024-03-04T09:13:51.584 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/fnic/fnic.ko 2024-03-04T09:13:51.584 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/hpsa.ko 2024-03-04T09:13:51.584 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/hptiop.ko 2024-03-04T09:13:51.584 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/hv_storvsc.ko 2024-03-04T09:13:51.584 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/initio.ko 2024-03-04T09:13:51.584 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/ipr.ko 2024-03-04T09:13:51.584 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/ips.ko 2024-03-04T09:13:51.584 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/isci 2024-03-04T09:13:51.584 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/isci/isci.ko 2024-03-04T09:13:51.584 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-03-04T09:13:51.584 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/iscsi_tcp.ko 2024-03-04T09:13:51.584 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/libfc 2024-03-04T09:13:51.584 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/libfc/libfc.ko 2024-03-04T09:13:51.584 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/libiscsi.ko 2024-03-04T09:13:51.585 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/libiscsi_tcp.ko 2024-03-04T09:13:51.585 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/libsas 2024-03-04T09:13:51.585 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/libsas/libsas.ko 2024-03-04T09:13:51.585 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/lpfc 2024-03-04T09:13:51.585 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/lpfc/lpfc.ko 2024-03-04T09:13:51.585 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/megaraid 2024-03-04T09:13:51.585 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/megaraid.ko 2024-03-04T09:13:51.585 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-03-04T09:13:51.585 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-03-04T09:13:51.585 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-03-04T09:13:51.585 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/mpt3sas 2024-03-04T09:13:51.585 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-03-04T09:13:51.585 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/mvsas 2024-03-04T09:13:51.585 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/mvsas/mvsas.ko 2024-03-04T09:13:51.586 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/mvumi.ko 2024-03-04T09:13:51.586 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/pm8001 2024-03-04T09:13:51.586 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-03-04T09:13:51.586 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/pmcraid.ko 2024-03-04T09:13:51.586 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/qla1280.ko 2024-03-04T09:13:51.586 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/qla2xxx 2024-03-04T09:13:51.586 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-03-04T09:13:51.586 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-03-04T09:13:51.586 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/qla4xxx 2024-03-04T09:13:51.586 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-03-04T09:13:51.586 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/raid_class.ko 2024-03-04T09:13:51.586 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/scsi_debug.ko 2024-03-04T09:13:51.586 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/scsi_transport_fc.ko 2024-03-04T09:13:51.586 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-03-04T09:13:51.586 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/scsi_transport_sas.ko 2024-03-04T09:13:51.587 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/scsi_transport_spi.ko 2024-03-04T09:13:51.587 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/scsi_transport_srp.ko 2024-03-04T09:13:51.587 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/ses.ko 2024-03-04T09:13:51.587 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/st.ko 2024-03-04T09:13:51.587 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/stex.ko 2024-03-04T09:13:51.587 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/sym53c8xx_2 2024-03-04T09:13:51.587 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-03-04T09:13:51.587 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/virtio_scsi.ko 2024-03-04T09:13:51.587 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/vmw_pvscsi.ko 2024-03-04T09:13:51.587 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/wd719x.ko 2024-03-04T09:13:51.587 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ssb 2024-03-04T09:13:51.587 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ssb/ssb.ko 2024-03-04T09:13:51.587 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/target 2024-03-04T09:13:51.587 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/target/iscsi 2024-03-04T09:13:51.587 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-03-04T09:13:51.588 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/target/loopback 2024-03-04T09:13:51.588 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/target/loopback/tcm_loop.ko 2024-03-04T09:13:51.588 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/target/target_core_file.ko 2024-03-04T09:13:51.588 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/target/target_core_iblock.ko 2024-03-04T09:13:51.588 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/target/target_core_mod.ko 2024-03-04T09:13:51.588 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/target/target_core_pscsi.ko 2024-03-04T09:13:51.588 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/target/target_core_user.ko 2024-03-04T09:13:51.588 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/target/tcm_fc 2024-03-04T09:13:51.588 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-03-04T09:13:51.588 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/thermal 2024-03-04T09:13:51.588 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/thermal/intel 2024-03-04T09:13:51.588 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/thermal/intel/int340x_thermal 2024-03-04T09:13:51.588 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-03-04T09:13:51.588 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-03-04T09:13:51.589 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-03-04T09:13:51.589 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-03-04T09:13:51.589 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-03-04T09:13:51.589 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-03-04T09:13:51.589 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-03-04T09:13:51.589 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-03-04T09:13:51.589 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-03-04T09:13:51.589 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-03-04T09:13:51.589 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-03-04T09:13:51.589 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-03-04T09:13:51.589 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-03-04T09:13:51.589 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-03-04T09:13:51.589 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-03-04T09:13:51.589 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-03-04T09:13:51.589 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-03-04T09:13:51.590 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-03-04T09:13:51.590 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/tty 2024-03-04T09:13:51.590 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/tty/ipwireless 2024-03-04T09:13:51.590 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-03-04T09:13:51.590 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/tty/n_gsm.ko 2024-03-04T09:13:51.590 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/tty/n_hdlc.ko 2024-03-04T09:13:51.590 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/tty/nozomi.ko 2024-03-04T09:13:51.590 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/tty/serial 2024-03-04T09:13:51.590 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/tty/serial/8250 2024-03-04T09:13:51.590 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-03-04T09:13:51.590 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/tty/serial/arc_uart.ko 2024-03-04T09:13:51.590 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/tty/serial/jsm 2024-03-04T09:13:51.590 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/tty/serial/jsm/jsm.ko 2024-03-04T09:13:51.590 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ufs 2024-03-04T09:13:51.590 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ufs/core 2024-03-04T09:13:51.591 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ufs/core/ufshcd-core.ko 2024-03-04T09:13:51.591 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ufs/host 2024-03-04T09:13:51.591 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-03-04T09:13:51.591 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/uio 2024-03-04T09:13:51.591 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/uio/uio.ko 2024-03-04T09:13:51.591 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/uio/uio_aec.ko 2024-03-04T09:13:51.591 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/uio/uio_cif.ko 2024-03-04T09:13:51.591 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/uio/uio_pci_generic.ko 2024-03-04T09:13:51.591 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/uio/uio_sercos3.ko 2024-03-04T09:13:51.591 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb 2024-03-04T09:13:51.591 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/class 2024-03-04T09:13:51.591 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/class/cdc-acm.ko 2024-03-04T09:13:51.591 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/class/cdc-wdm.ko 2024-03-04T09:13:51.591 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/class/usblp.ko 2024-03-04T09:13:51.591 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/class/usbtmc.ko 2024-03-04T09:13:51.592 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/host 2024-03-04T09:13:51.592 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/host/sl811-hcd.ko 2024-03-04T09:13:51.592 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/image 2024-03-04T09:13:51.592 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/image/mdc800.ko 2024-03-04T09:13:51.592 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/image/microtek.ko 2024-03-04T09:13:51.592 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/misc 2024-03-04T09:13:51.592 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/misc/adutux.ko 2024-03-04T09:13:51.592 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/misc/appledisplay.ko 2024-03-04T09:13:51.592 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/misc/emi26.ko 2024-03-04T09:13:51.592 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/misc/emi62.ko 2024-03-04T09:13:51.592 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/misc/ezusb.ko 2024-03-04T09:13:51.592 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/misc/idmouse.ko 2024-03-04T09:13:51.592 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/misc/iowarrior.ko 2024-03-04T09:13:51.592 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/misc/isight_firmware.ko 2024-03-04T09:13:51.593 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/misc/ldusb.ko 2024-03-04T09:13:51.593 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/misc/legousbtower.ko 2024-03-04T09:13:51.593 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/misc/sisusbvga 2024-03-04T09:13:51.593 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-03-04T09:13:51.593 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/misc/trancevibrator.ko 2024-03-04T09:13:51.593 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/misc/usb3503.ko 2024-03-04T09:13:51.593 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/misc/usblcd.ko 2024-03-04T09:13:51.593 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/misc/usbsevseg.ko 2024-03-04T09:13:51.593 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/misc/uss720.ko 2024-03-04T09:13:51.593 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/misc/yurex.ko 2024-03-04T09:13:51.593 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/phy 2024-03-04T09:13:51.593 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/phy/phy-generic.ko 2024-03-04T09:13:51.593 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial 2024-03-04T09:13:51.593 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/aircable.ko 2024-03-04T09:13:51.593 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/ark3116.ko 2024-03-04T09:13:51.594 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/belkin_sa.ko 2024-03-04T09:13:51.594 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/ch341.ko 2024-03-04T09:13:51.594 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/cp210x.ko 2024-03-04T09:13:51.594 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/cyberjack.ko 2024-03-04T09:13:51.594 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/cypress_m8.ko 2024-03-04T09:13:51.594 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/digi_acceleport.ko 2024-03-04T09:13:51.594 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/empeg.ko 2024-03-04T09:13:51.594 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/ftdi_sio.ko 2024-03-04T09:13:51.594 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/garmin_gps.ko 2024-03-04T09:13:51.594 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/io_edgeport.ko 2024-03-04T09:13:51.594 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/io_ti.ko 2024-03-04T09:13:51.594 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/ipaq.ko 2024-03-04T09:13:51.594 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/ipw.ko 2024-03-04T09:13:51.594 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/ir-usb.ko 2024-03-04T09:13:51.595 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-03-04T09:13:51.595 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/keyspan.ko 2024-03-04T09:13:51.595 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/keyspan_pda.ko 2024-03-04T09:13:51.595 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/kl5kusb105.ko 2024-03-04T09:13:51.595 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/kobil_sct.ko 2024-03-04T09:13:51.595 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/mct_u232.ko 2024-03-04T09:13:51.595 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/mos7720.ko 2024-03-04T09:13:51.595 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/mos7840.ko 2024-03-04T09:13:51.595 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/navman.ko 2024-03-04T09:13:51.595 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/omninet.ko 2024-03-04T09:13:51.595 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/opticon.ko 2024-03-04T09:13:51.595 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/option.ko 2024-03-04T09:13:51.595 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/oti6858.ko 2024-03-04T09:13:51.595 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/pl2303.ko 2024-03-04T09:13:51.595 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/qcaux.ko 2024-03-04T09:13:51.596 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/qcserial.ko 2024-03-04T09:13:51.596 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/quatech2.ko 2024-03-04T09:13:51.596 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/safe_serial.ko 2024-03-04T09:13:51.596 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/sierra.ko 2024-03-04T09:13:51.596 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/spcp8x5.ko 2024-03-04T09:13:51.596 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/ssu100.ko 2024-03-04T09:13:51.596 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/symbolserial.ko 2024-03-04T09:13:51.596 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-03-04T09:13:51.596 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-03-04T09:13:51.596 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/usb_debug.ko 2024-03-04T09:13:51.596 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/usb_wwan.ko 2024-03-04T09:13:51.596 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/visor.ko 2024-03-04T09:13:51.596 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/whiteheat.ko 2024-03-04T09:13:51.596 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/xsens_mt.ko 2024-03-04T09:13:51.596 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/storage 2024-03-04T09:13:51.597 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/storage/uas.ko 2024-03-04T09:13:51.597 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/storage/ums-alauda.ko 2024-03-04T09:13:51.597 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/storage/ums-cypress.ko 2024-03-04T09:13:51.597 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/storage/ums-datafab.ko 2024-03-04T09:13:51.597 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-03-04T09:13:51.597 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/storage/ums-freecom.ko 2024-03-04T09:13:51.597 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/storage/ums-isd200.ko 2024-03-04T09:13:51.597 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-03-04T09:13:51.597 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/storage/ums-karma.ko 2024-03-04T09:13:51.597 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/storage/ums-onetouch.ko 2024-03-04T09:13:51.597 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/storage/ums-realtek.ko 2024-03-04T09:13:51.597 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/storage/ums-sddr09.ko 2024-03-04T09:13:51.597 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/storage/ums-sddr55.ko 2024-03-04T09:13:51.597 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/storage/ums-usbat.ko 2024-03-04T09:13:51.598 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/storage/usb-storage.ko 2024-03-04T09:13:51.598 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/usbip 2024-03-04T09:13:51.598 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/usbip/usbip-core.ko 2024-03-04T09:13:51.598 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/usbip/usbip-host.ko 2024-03-04T09:13:51.598 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-03-04T09:13:51.598 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/vfio 2024-03-04T09:13:51.598 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/vfio/pci 2024-03-04T09:13:51.598 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-03-04T09:13:51.598 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/vfio/pci/vfio-pci.ko 2024-03-04T09:13:51.598 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/vfio/vfio.ko 2024-03-04T09:13:51.598 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-03-04T09:13:51.598 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/vhost 2024-03-04T09:13:51.598 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/vhost/vhost.ko 2024-03-04T09:13:51.598 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/vhost/vhost_iotlb.ko 2024-03-04T09:13:51.599 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/vhost/vhost_net.ko 2024-03-04T09:13:51.599 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/vhost/vhost_scsi.ko 2024-03-04T09:13:51.599 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/video 2024-03-04T09:13:51.599 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/video/backlight 2024-03-04T09:13:51.599 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/video/backlight/apple_bl.ko 2024-03-04T09:13:51.599 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/video/backlight/lcd.ko 2024-03-04T09:13:51.599 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/video/backlight/platform_lcd.ko 2024-03-04T09:13:51.599 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/virtio 2024-03-04T09:13:51.599 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/virtio/virtio.ko 2024-03-04T09:13:51.599 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/virtio/virtio_balloon.ko 2024-03-04T09:13:51.599 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/virtio/virtio_mmio.ko 2024-03-04T09:13:51.599 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/virtio/virtio_pci.ko 2024-03-04T09:13:51.599 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-03-04T09:13:51.599 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-03-04T09:13:51.599 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/virtio/virtio_ring.ko 2024-03-04T09:13:51.600 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/w1 2024-03-04T09:13:51.600 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/w1/masters 2024-03-04T09:13:51.600 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/w1/masters/ds2482.ko 2024-03-04T09:13:51.600 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/w1/masters/ds2490.ko 2024-03-04T09:13:51.600 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/w1/slaves 2024-03-04T09:13:51.600 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-03-04T09:13:51.600 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-03-04T09:13:51.600 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-03-04T09:13:51.600 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-03-04T09:13:51.600 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-03-04T09:13:51.600 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-03-04T09:13:51.600 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-03-04T09:13:51.600 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-03-04T09:13:51.600 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-03-04T09:13:51.601 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/w1/slaves/w1_smem.ko 2024-03-04T09:13:51.601 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/w1/slaves/w1_therm.ko 2024-03-04T09:13:51.601 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/w1/wire.ko 2024-03-04T09:13:51.601 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/watchdog 2024-03-04T09:13:51.601 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/watchdog/alim1535_wdt.ko 2024-03-04T09:13:51.601 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/watchdog/alim7101_wdt.ko 2024-03-04T09:13:51.601 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/watchdog/f71808e_wdt.ko 2024-03-04T09:13:51.601 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/watchdog/hpwdt.ko 2024-03-04T09:13:51.601 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/watchdog/i6300esb.ko 2024-03-04T09:13:51.601 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-03-04T09:13:51.601 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/watchdog/iTCO_wdt.ko 2024-03-04T09:13:51.601 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/watchdog/ib700wdt.ko 2024-03-04T09:13:51.601 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/watchdog/ibmasr.ko 2024-03-04T09:13:51.601 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-03-04T09:13:51.602 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/watchdog/it8712f_wdt.ko 2024-03-04T09:13:51.602 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/watchdog/it87_wdt.ko 2024-03-04T09:13:51.602 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/watchdog/machzwd.ko 2024-03-04T09:13:51.602 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/watchdog/nv_tco.ko 2024-03-04T09:13:51.602 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/watchdog/pcwd_pci.ko 2024-03-04T09:13:51.602 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/watchdog/pcwd_usb.ko 2024-03-04T09:13:51.602 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-03-04T09:13:51.602 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/watchdog/sch311x_wdt.ko 2024-03-04T09:13:51.602 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/watchdog/softdog.ko 2024-03-04T09:13:51.602 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/watchdog/sp5100_tco.ko 2024-03-04T09:13:51.602 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/watchdog/via_wdt.ko 2024-03-04T09:13:51.602 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-03-04T09:13:51.602 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/watchdog/w83877f_wdt.ko 2024-03-04T09:13:51.603 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/watchdog/w83977f_wdt.ko 2024-03-04T09:13:51.603 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/watchdog/wdt_pci.ko 2024-03-04T09:13:51.603 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/watchdog/xen_wdt.ko 2024-03-04T09:13:51.603 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/xen 2024-03-04T09:13:51.603 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/xen/xen-acpi-processor.ko 2024-03-04T09:13:51.603 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/xen/xen-evtchn.ko 2024-03-04T09:13:51.603 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/xen/xen-gntalloc.ko 2024-03-04T09:13:51.603 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/xen/xen-gntdev.ko 2024-03-04T09:13:51.603 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/xen/xen-pciback 2024-03-04T09:13:51.603 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-03-04T09:13:51.603 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/xen/xen-privcmd.ko 2024-03-04T09:13:51.603 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/xen/xenfs 2024-03-04T09:13:51.603 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/xen/xenfs/xenfs.ko 2024-03-04T09:13:51.603 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs 2024-03-04T09:13:51.604 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/9p 2024-03-04T09:13:51.604 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/9p/9p.ko 2024-03-04T09:13:51.604 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/affs 2024-03-04T09:13:51.604 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/affs/affs.ko 2024-03-04T09:13:51.604 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/befs 2024-03-04T09:13:51.604 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/befs/befs.ko 2024-03-04T09:13:51.604 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/binfmt_misc.ko 2024-03-04T09:13:51.604 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/btrfs 2024-03-04T09:13:51.604 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/btrfs/btrfs.ko 2024-03-04T09:13:51.604 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/cachefiles 2024-03-04T09:13:51.604 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/cachefiles/cachefiles.ko 2024-03-04T09:13:51.604 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/ceph 2024-03-04T09:13:51.604 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/ceph/ceph.ko 2024-03-04T09:13:51.604 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/coda 2024-03-04T09:13:51.605 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/coda/coda.ko 2024-03-04T09:13:51.605 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/cramfs 2024-03-04T09:13:51.605 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/cramfs/cramfs.ko 2024-03-04T09:13:51.605 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/dlm 2024-03-04T09:13:51.605 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/dlm/dlm.ko 2024-03-04T09:13:51.605 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/ecryptfs 2024-03-04T09:13:51.605 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/ecryptfs/ecryptfs.ko 2024-03-04T09:13:51.605 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/f2fs 2024-03-04T09:13:51.605 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/f2fs/f2fs.ko 2024-03-04T09:13:51.605 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/fat 2024-03-04T09:13:51.605 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/fat/fat.ko 2024-03-04T09:13:51.605 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/fat/msdos.ko 2024-03-04T09:13:51.605 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/fat/vfat.ko 2024-03-04T09:13:51.605 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/fuse 2024-03-04T09:13:51.605 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/fuse/cuse.ko 2024-03-04T09:13:51.606 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/fuse/fuse.ko 2024-03-04T09:13:51.606 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/gfs2 2024-03-04T09:13:51.606 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/gfs2/gfs2.ko 2024-03-04T09:13:51.606 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/hfs 2024-03-04T09:13:51.606 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/hfs/hfs.ko 2024-03-04T09:13:51.606 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/hfsplus 2024-03-04T09:13:51.606 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/hfsplus/hfsplus.ko 2024-03-04T09:13:51.606 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/isofs 2024-03-04T09:13:51.606 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/isofs/isofs.ko 2024-03-04T09:13:51.606 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/jfs 2024-03-04T09:13:51.606 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/jfs/jfs.ko 2024-03-04T09:13:51.606 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/lockd 2024-03-04T09:13:51.606 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/lockd/lockd.ko 2024-03-04T09:13:51.606 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/minix 2024-03-04T09:13:51.606 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/minix/minix.ko 2024-03-04T09:13:51.607 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/netfs 2024-03-04T09:13:51.607 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/netfs/netfs.ko 2024-03-04T09:13:51.607 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nfs 2024-03-04T09:13:51.607 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nfs/blocklayout 2024-03-04T09:13:51.607 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-03-04T09:13:51.607 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nfs/filelayout 2024-03-04T09:13:51.607 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-03-04T09:13:51.607 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nfs/flexfilelayout 2024-03-04T09:13:51.607 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-03-04T09:13:51.607 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nfs/nfs.ko 2024-03-04T09:13:51.607 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nfs/nfsv3.ko 2024-03-04T09:13:51.607 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nfs/nfsv4.ko 2024-03-04T09:13:51.607 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nfs_common 2024-03-04T09:13:51.607 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nfs_common/grace.ko 2024-03-04T09:13:51.608 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nfs_common/nfs_acl.ko 2024-03-04T09:13:51.608 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nfsd 2024-03-04T09:13:51.608 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nfsd/nfsd.ko 2024-03-04T09:13:51.608 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nilfs2 2024-03-04T09:13:51.608 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nilfs2/nilfs2.ko 2024-03-04T09:13:51.608 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls 2024-03-04T09:13:51.608 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/mac-celtic.ko 2024-03-04T09:13:51.608 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/mac-centeuro.ko 2024-03-04T09:13:51.608 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/mac-croatian.ko 2024-03-04T09:13:51.608 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/mac-cyrillic.ko 2024-03-04T09:13:51.608 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/mac-gaelic.ko 2024-03-04T09:13:51.608 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/mac-greek.ko 2024-03-04T09:13:51.608 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/mac-iceland.ko 2024-03-04T09:13:51.608 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/mac-inuit.ko 2024-03-04T09:13:51.608 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/mac-roman.ko 2024-03-04T09:13:51.609 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/mac-romanian.ko 2024-03-04T09:13:51.609 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/mac-turkish.ko 2024-03-04T09:13:51.609 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/nls_cp1250.ko 2024-03-04T09:13:51.609 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/nls_cp1251.ko 2024-03-04T09:13:51.609 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/nls_cp1255.ko 2024-03-04T09:13:51.609 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/nls_cp737.ko 2024-03-04T09:13:51.609 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/nls_cp775.ko 2024-03-04T09:13:51.609 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/nls_cp850.ko 2024-03-04T09:13:51.609 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/nls_cp852.ko 2024-03-04T09:13:51.609 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/nls_cp855.ko 2024-03-04T09:13:51.609 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/nls_cp857.ko 2024-03-04T09:13:51.609 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/nls_cp860.ko 2024-03-04T09:13:51.609 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/nls_cp861.ko 2024-03-04T09:13:51.609 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/nls_cp862.ko 2024-03-04T09:13:51.609 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/nls_cp863.ko 2024-03-04T09:13:51.610 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/nls_cp864.ko 2024-03-04T09:13:51.610 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/nls_cp865.ko 2024-03-04T09:13:51.610 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/nls_cp866.ko 2024-03-04T09:13:51.610 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/nls_cp869.ko 2024-03-04T09:13:51.610 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/nls_cp874.ko 2024-03-04T09:13:51.610 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/nls_cp932.ko 2024-03-04T09:13:51.610 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/nls_cp936.ko 2024-03-04T09:13:51.610 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/nls_cp949.ko 2024-03-04T09:13:51.610 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/nls_cp950.ko 2024-03-04T09:13:51.610 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/nls_euc-jp.ko 2024-03-04T09:13:51.610 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/nls_iso8859-1.ko 2024-03-04T09:13:51.610 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/nls_iso8859-13.ko 2024-03-04T09:13:51.610 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/nls_iso8859-14.ko 2024-03-04T09:13:51.610 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/nls_iso8859-15.ko 2024-03-04T09:13:51.610 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/nls_iso8859-2.ko 2024-03-04T09:13:51.611 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/nls_iso8859-3.ko 2024-03-04T09:13:51.611 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/nls_iso8859-4.ko 2024-03-04T09:13:51.611 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/nls_iso8859-5.ko 2024-03-04T09:13:51.611 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/nls_iso8859-6.ko 2024-03-04T09:13:51.611 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/nls_iso8859-7.ko 2024-03-04T09:13:51.611 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/nls_iso8859-9.ko 2024-03-04T09:13:51.611 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/nls_koi8-r.ko 2024-03-04T09:13:51.611 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/nls_koi8-ru.ko 2024-03-04T09:13:51.611 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/nls_koi8-u.ko 2024-03-04T09:13:51.611 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/nls_ucs2_utils.ko 2024-03-04T09:13:51.611 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/nls_utf8.ko 2024-03-04T09:13:51.611 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/ocfs2 2024-03-04T09:13:51.611 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/ocfs2/cluster 2024-03-04T09:13:51.611 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-03-04T09:13:51.611 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/ocfs2/dlm 2024-03-04T09:13:51.612 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-03-04T09:13:51.612 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/ocfs2/dlmfs 2024-03-04T09:13:51.612 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-03-04T09:13:51.612 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/ocfs2/ocfs2.ko 2024-03-04T09:13:51.612 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-03-04T09:13:51.612 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-03-04T09:13:51.612 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-03-04T09:13:51.612 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/overlayfs 2024-03-04T09:13:51.612 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/overlayfs/overlay.ko 2024-03-04T09:13:51.612 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/pstore 2024-03-04T09:13:51.612 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/pstore/ramoops.ko 2024-03-04T09:13:51.612 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/reiserfs 2024-03-04T09:13:51.612 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/reiserfs/reiserfs.ko 2024-03-04T09:13:51.612 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/romfs 2024-03-04T09:13:51.612 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/romfs/romfs.ko 2024-03-04T09:13:51.613 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/smb 2024-03-04T09:13:51.613 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/smb/client 2024-03-04T09:13:51.613 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/smb/client/cifs.ko 2024-03-04T09:13:51.613 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/smb/common 2024-03-04T09:13:51.613 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/smb/common/cifs_arc4.ko 2024-03-04T09:13:51.613 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/smb/common/cifs_md4.ko 2024-03-04T09:13:51.613 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/squashfs 2024-03-04T09:13:51.613 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/squashfs/squashfs.ko 2024-03-04T09:13:51.613 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/sysv 2024-03-04T09:13:51.613 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/sysv/sysv.ko 2024-03-04T09:13:51.613 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/udf 2024-03-04T09:13:51.613 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/udf/udf.ko 2024-03-04T09:13:51.613 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/ufs 2024-03-04T09:13:51.613 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/ufs/ufs.ko 2024-03-04T09:13:51.613 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/xfs 2024-03-04T09:13:51.614 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/xfs/xfs.ko 2024-03-04T09:13:51.614 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/kernel 2024-03-04T09:13:51.614 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/kernel/rcu 2024-03-04T09:13:51.614 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/kernel/rcu/rcutorture.ko 2024-03-04T09:13:51.614 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/kernel/torture.ko 2024-03-04T09:13:51.614 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/kernel/trace 2024-03-04T09:13:51.614 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-03-04T09:13:51.614 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/lib 2024-03-04T09:13:51.614 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/lib/asn1_encoder.ko 2024-03-04T09:13:51.614 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/lib/crc-itu-t.ko 2024-03-04T09:13:51.614 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/lib/crc8.ko 2024-03-04T09:13:51.614 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/lib/crypto 2024-03-04T09:13:51.614 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/lib/crypto/libarc4.ko 2024-03-04T09:13:51.614 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/lib/crypto/libdes.ko 2024-03-04T09:13:51.614 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/lib/libcrc32c.ko 2024-03-04T09:13:51.615 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/lib/lru_cache.ko 2024-03-04T09:13:51.615 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/lib/lz4 2024-03-04T09:13:51.615 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/lib/lz4/lz4_compress.ko 2024-03-04T09:13:51.615 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/lib/lz4/lz4hc_compress.ko 2024-03-04T09:13:51.615 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/lib/math 2024-03-04T09:13:51.615 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/lib/math/cordic.ko 2024-03-04T09:13:51.615 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/lib/raid6 2024-03-04T09:13:51.615 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/lib/raid6/raid6_pq.ko 2024-03-04T09:13:51.615 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/lib/reed_solomon 2024-03-04T09:13:51.615 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/lib/reed_solomon/reed_solomon.ko 2024-03-04T09:13:51.615 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/lib/ts_bm.ko 2024-03-04T09:13:51.615 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/lib/ts_fsm.ko 2024-03-04T09:13:51.615 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/lib/ts_kmp.ko 2024-03-04T09:13:51.615 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/lib/zstd 2024-03-04T09:13:51.615 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/lib/zstd/zstd_compress.ko 2024-03-04T09:13:51.616 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/mm 2024-03-04T09:13:51.616 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/mm/hwpoison-inject.ko 2024-03-04T09:13:51.616 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net 2024-03-04T09:13:51.616 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/6lowpan 2024-03-04T09:13:51.616 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/6lowpan/6lowpan.ko 2024-03-04T09:13:51.616 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/6lowpan/nhc_dest.ko 2024-03-04T09:13:51.616 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/6lowpan/nhc_fragment.ko 2024-03-04T09:13:51.616 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/6lowpan/nhc_hop.ko 2024-03-04T09:13:51.616 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/6lowpan/nhc_ipv6.ko 2024-03-04T09:13:51.616 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/6lowpan/nhc_mobility.ko 2024-03-04T09:13:51.616 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/6lowpan/nhc_routing.ko 2024-03-04T09:13:51.616 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/6lowpan/nhc_udp.ko 2024-03-04T09:13:51.616 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/802 2024-03-04T09:13:51.616 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/802/garp.ko 2024-03-04T09:13:51.616 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/802/mrp.ko 2024-03-04T09:13:51.617 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/802/p8022.ko 2024-03-04T09:13:51.617 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/802/psnap.ko 2024-03-04T09:13:51.617 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/802/stp.ko 2024-03-04T09:13:51.617 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/8021q 2024-03-04T09:13:51.617 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/8021q/8021q.ko 2024-03-04T09:13:51.617 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/9p 2024-03-04T09:13:51.617 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/9p/9pnet.ko 2024-03-04T09:13:51.617 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/9p/9pnet_fd.ko 2024-03-04T09:13:51.617 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/9p/9pnet_rdma.ko 2024-03-04T09:13:51.617 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/9p/9pnet_virtio.ko 2024-03-04T09:13:51.617 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/batman-adv 2024-03-04T09:13:51.617 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/batman-adv/batman-adv.ko 2024-03-04T09:13:51.617 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/bridge 2024-03-04T09:13:51.617 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/bridge/br_netfilter.ko 2024-03-04T09:13:51.617 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/bridge/bridge.ko 2024-03-04T09:13:51.618 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/bridge/netfilter 2024-03-04T09:13:51.618 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-03-04T09:13:51.618 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/bridge/netfilter/ebt_among.ko 2024-03-04T09:13:51.618 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/bridge/netfilter/ebt_arp.ko 2024-03-04T09:13:51.618 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-03-04T09:13:51.618 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-03-04T09:13:51.618 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/bridge/netfilter/ebt_ip.ko 2024-03-04T09:13:51.618 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-03-04T09:13:51.618 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/bridge/netfilter/ebt_limit.ko 2024-03-04T09:13:51.618 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/bridge/netfilter/ebt_log.ko 2024-03-04T09:13:51.618 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/bridge/netfilter/ebt_mark.ko 2024-03-04T09:13:51.618 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-03-04T09:13:51.618 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-03-04T09:13:51.618 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-03-04T09:13:51.619 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-03-04T09:13:51.619 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/bridge/netfilter/ebt_snat.ko 2024-03-04T09:13:51.619 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/bridge/netfilter/ebt_stp.ko 2024-03-04T09:13:51.619 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-03-04T09:13:51.619 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-03-04T09:13:51.619 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-03-04T09:13:51.619 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-03-04T09:13:51.619 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/bridge/netfilter/ebtables.ko 2024-03-04T09:13:51.619 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-03-04T09:13:51.619 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-03-04T09:13:51.619 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-03-04T09:13:51.619 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ceph 2024-03-04T09:13:51.619 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ceph/libceph.ko 2024-03-04T09:13:51.619 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/core 2024-03-04T09:13:51.620 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/core/failover.ko 2024-03-04T09:13:51.620 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/core/pktgen.ko 2024-03-04T09:13:51.620 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/dccp 2024-03-04T09:13:51.620 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/dccp/dccp.ko 2024-03-04T09:13:51.620 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/dccp/dccp_diag.ko 2024-03-04T09:13:51.620 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/dccp/dccp_ipv4.ko 2024-03-04T09:13:51.620 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/dccp/dccp_ipv6.ko 2024-03-04T09:13:51.620 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/dns_resolver 2024-03-04T09:13:51.620 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/dns_resolver/dns_resolver.ko 2024-03-04T09:13:51.620 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/dsa 2024-03-04T09:13:51.620 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/dsa/dsa_core.ko 2024-03-04T09:13:51.620 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/dsa/tag_brcm.ko 2024-03-04T09:13:51.620 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/dsa/tag_dsa.ko 2024-03-04T09:13:51.620 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/dsa/tag_none.ko 2024-03-04T09:13:51.621 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/dsa/tag_trailer.ko 2024-03-04T09:13:51.621 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ieee802154 2024-03-04T09:13:51.621 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ieee802154/6lowpan 2024-03-04T09:13:51.621 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-03-04T09:13:51.621 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ieee802154/ieee802154.ko 2024-03-04T09:13:51.621 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ieee802154/ieee802154_socket.ko 2024-03-04T09:13:51.621 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4 2024-03-04T09:13:51.621 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/ah4.ko 2024-03-04T09:13:51.621 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/esp4.ko 2024-03-04T09:13:51.621 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/fou.ko 2024-03-04T09:13:51.621 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/gre.ko 2024-03-04T09:13:51.621 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/inet_diag.ko 2024-03-04T09:13:51.621 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/ip_gre.ko 2024-03-04T09:13:51.621 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/ip_tunnel.ko 2024-03-04T09:13:51.621 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/ip_vti.ko 2024-03-04T09:13:51.622 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/ipcomp.ko 2024-03-04T09:13:51.622 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/ipip.ko 2024-03-04T09:13:51.622 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/netfilter 2024-03-04T09:13:51.622 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/netfilter/arp_tables.ko 2024-03-04T09:13:51.622 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-03-04T09:13:51.622 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-03-04T09:13:51.622 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-03-04T09:13:51.622 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-03-04T09:13:51.622 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-03-04T09:13:51.622 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-03-04T09:13:51.622 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-03-04T09:13:51.622 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-03-04T09:13:51.622 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-03-04T09:13:51.622 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/netfilter/iptable_security.ko 2024-03-04T09:13:51.622 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-03-04T09:13:51.623 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-03-04T09:13:51.623 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-03-04T09:13:51.623 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-03-04T09:13:51.623 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-03-04T09:13:51.623 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-03-04T09:13:51.623 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-03-04T09:13:51.623 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-03-04T09:13:51.623 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-03-04T09:13:51.623 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-03-04T09:13:51.623 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/tcp_bic.ko 2024-03-04T09:13:51.623 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/tcp_dctcp.ko 2024-03-04T09:13:51.623 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/tcp_diag.ko 2024-03-04T09:13:51.623 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/tcp_highspeed.ko 2024-03-04T09:13:51.623 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/tcp_htcp.ko 2024-03-04T09:13:51.624 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/tcp_hybla.ko 2024-03-04T09:13:51.624 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/tcp_illinois.ko 2024-03-04T09:13:51.624 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/tcp_lp.ko 2024-03-04T09:13:51.624 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/tcp_scalable.ko 2024-03-04T09:13:51.624 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/tcp_vegas.ko 2024-03-04T09:13:51.624 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/tcp_veno.ko 2024-03-04T09:13:51.624 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/tcp_westwood.ko 2024-03-04T09:13:51.624 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/tcp_yeah.ko 2024-03-04T09:13:51.624 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/tunnel4.ko 2024-03-04T09:13:51.624 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/udp_diag.ko 2024-03-04T09:13:51.624 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/udp_tunnel.ko 2024-03-04T09:13:51.624 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/xfrm4_tunnel.ko 2024-03-04T09:13:51.624 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv6 2024-03-04T09:13:51.624 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv6/ah6.ko 2024-03-04T09:13:51.624 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv6/esp6.ko 2024-03-04T09:13:51.625 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv6/fou6.ko 2024-03-04T09:13:51.625 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv6/ip6_tunnel.ko 2024-03-04T09:13:51.625 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-03-04T09:13:51.625 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv6/ip6_vti.ko 2024-03-04T09:13:51.625 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv6/ipcomp6.ko 2024-03-04T09:13:51.625 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv6/netfilter 2024-03-04T09:13:51.625 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-03-04T09:13:51.625 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-03-04T09:13:51.625 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-03-04T09:13:51.625 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-03-04T09:13:51.625 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-03-04T09:13:51.625 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-03-04T09:13:51.625 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-03-04T09:13:51.625 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-03-04T09:13:51.626 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-03-04T09:13:51.626 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-03-04T09:13:51.626 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-03-04T09:13:51.626 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-03-04T09:13:51.626 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-03-04T09:13:51.626 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-03-04T09:13:51.626 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-03-04T09:13:51.626 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-03-04T09:13:51.626 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-03-04T09:13:51.626 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-03-04T09:13:51.626 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-03-04T09:13:51.626 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-03-04T09:13:51.626 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-03-04T09:13:51.626 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-03-04T09:13:51.627 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-03-04T09:13:51.627 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-03-04T09:13:51.627 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv6/sit.ko 2024-03-04T09:13:51.627 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv6/tunnel6.ko 2024-03-04T09:13:51.627 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv6/xfrm6_tunnel.ko 2024-03-04T09:13:51.627 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/key 2024-03-04T09:13:51.627 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/key/af_key.ko 2024-03-04T09:13:51.627 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/l2tp 2024-03-04T09:13:51.627 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/l2tp/l2tp_core.ko 2024-03-04T09:13:51.627 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/l2tp/l2tp_debugfs.ko 2024-03-04T09:13:51.627 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/l2tp/l2tp_eth.ko 2024-03-04T09:13:51.627 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/l2tp/l2tp_ip.ko 2024-03-04T09:13:51.627 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/l2tp/l2tp_ip6.ko 2024-03-04T09:13:51.627 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/l2tp/l2tp_netlink.ko 2024-03-04T09:13:51.627 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/l2tp/l2tp_ppp.ko 2024-03-04T09:13:51.628 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/llc 2024-03-04T09:13:51.628 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/llc/llc.ko 2024-03-04T09:13:51.628 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/mac802154 2024-03-04T09:13:51.628 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/mac802154/mac802154.ko 2024-03-04T09:13:51.628 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/mpls 2024-03-04T09:13:51.628 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/mpls/mpls_gso.ko 2024-03-04T09:13:51.628 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter 2024-03-04T09:13:51.628 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/ipset 2024-03-04T09:13:51.628 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/ipset/ip_set.ko 2024-03-04T09:13:51.628 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-03-04T09:13:51.628 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-03-04T09:13:51.628 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-03-04T09:13:51.628 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-03-04T09:13:51.628 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-03-04T09:13:51.629 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-03-04T09:13:51.629 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-03-04T09:13:51.629 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-03-04T09:13:51.629 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-03-04T09:13:51.629 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-03-04T09:13:51.629 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-03-04T09:13:51.629 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-03-04T09:13:51.629 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-03-04T09:13:51.629 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-03-04T09:13:51.629 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-03-04T09:13:51.629 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/ipvs 2024-03-04T09:13:51.629 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/ipvs/ip_vs.ko 2024-03-04T09:13:51.629 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-03-04T09:13:51.629 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-03-04T09:13:51.630 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-03-04T09:13:51.630 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-03-04T09:13:51.630 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-03-04T09:13:51.630 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-03-04T09:13:51.630 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-03-04T09:13:51.630 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-03-04T09:13:51.630 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-03-04T09:13:51.630 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-03-04T09:13:51.630 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-03-04T09:13:51.630 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-03-04T09:13:51.630 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-03-04T09:13:51.630 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nf_conncount.ko 2024-03-04T09:13:51.630 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nf_conntrack.ko 2024-03-04T09:13:51.630 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-03-04T09:13:51.630 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-03-04T09:13:51.630 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-03-04T09:13:51.631 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nf_conntrack_h323.ko 2024-03-04T09:13:51.631 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nf_conntrack_irc.ko 2024-03-04T09:13:51.631 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-03-04T09:13:51.631 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-03-04T09:13:51.631 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-03-04T09:13:51.631 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nf_conntrack_sane.ko 2024-03-04T09:13:51.631 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nf_conntrack_sip.ko 2024-03-04T09:13:51.631 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-03-04T09:13:51.631 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-03-04T09:13:51.631 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nf_dup_netdev.ko 2024-03-04T09:13:51.631 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nf_flow_table.ko 2024-03-04T09:13:51.631 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nf_flow_table_inet.ko 2024-03-04T09:13:51.631 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nf_log_syslog.ko 2024-03-04T09:13:51.631 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nf_nat.ko 2024-03-04T09:13:51.632 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nf_nat_amanda.ko 2024-03-04T09:13:51.632 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nf_nat_ftp.ko 2024-03-04T09:13:51.632 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nf_nat_irc.ko 2024-03-04T09:13:51.632 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nf_nat_sip.ko 2024-03-04T09:13:51.632 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nf_nat_tftp.ko 2024-03-04T09:13:51.632 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nf_synproxy_core.ko 2024-03-04T09:13:51.632 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nf_tables.ko 2024-03-04T09:13:51.632 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nfnetlink.ko 2024-03-04T09:13:51.632 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nfnetlink_acct.ko 2024-03-04T09:13:51.632 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nfnetlink_log.ko 2024-03-04T09:13:51.632 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nfnetlink_osf.ko 2024-03-04T09:13:51.632 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nfnetlink_queue.ko 2024-03-04T09:13:51.632 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nft_chain_nat.ko 2024-03-04T09:13:51.632 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nft_compat.ko 2024-03-04T09:13:51.632 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nft_ct.ko 2024-03-04T09:13:51.633 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nft_dup_netdev.ko 2024-03-04T09:13:51.633 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nft_fib.ko 2024-03-04T09:13:51.633 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nft_fwd_netdev.ko 2024-03-04T09:13:51.633 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nft_hash.ko 2024-03-04T09:13:51.633 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nft_limit.ko 2024-03-04T09:13:51.633 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nft_log.ko 2024-03-04T09:13:51.633 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nft_masq.ko 2024-03-04T09:13:51.633 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nft_nat.ko 2024-03-04T09:13:51.633 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nft_numgen.ko 2024-03-04T09:13:51.633 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nft_queue.ko 2024-03-04T09:13:51.633 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nft_quota.ko 2024-03-04T09:13:51.633 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nft_redir.ko 2024-03-04T09:13:51.633 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nft_reject.ko 2024-03-04T09:13:51.633 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nft_reject_inet.ko 2024-03-04T09:13:51.633 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nft_socket.ko 2024-03-04T09:13:51.634 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nft_tunnel.ko 2024-03-04T09:13:51.634 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_AUDIT.ko 2024-03-04T09:13:51.634 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_CHECKSUM.ko 2024-03-04T09:13:51.634 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_CLASSIFY.ko 2024-03-04T09:13:51.634 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-03-04T09:13:51.634 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_CT.ko 2024-03-04T09:13:51.634 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_DSCP.ko 2024-03-04T09:13:51.634 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_HL.ko 2024-03-04T09:13:51.634 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_HMARK.ko 2024-03-04T09:13:51.634 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_IDLETIMER.ko 2024-03-04T09:13:51.634 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_LED.ko 2024-03-04T09:13:51.634 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_LOG.ko 2024-03-04T09:13:51.634 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_MASQUERADE.ko 2024-03-04T09:13:51.634 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_NETMAP.ko 2024-03-04T09:13:51.634 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_NFLOG.ko 2024-03-04T09:13:51.635 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_NFQUEUE.ko 2024-03-04T09:13:51.635 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_RATEEST.ko 2024-03-04T09:13:51.635 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_REDIRECT.ko 2024-03-04T09:13:51.635 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_SECMARK.ko 2024-03-04T09:13:51.635 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_TCPMSS.ko 2024-03-04T09:13:51.635 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-03-04T09:13:51.635 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_TEE.ko 2024-03-04T09:13:51.635 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_TPROXY.ko 2024-03-04T09:13:51.635 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_TRACE.ko 2024-03-04T09:13:51.635 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_addrtype.ko 2024-03-04T09:13:51.635 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_bpf.ko 2024-03-04T09:13:51.635 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_cgroup.ko 2024-03-04T09:13:51.635 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_cluster.ko 2024-03-04T09:13:51.635 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_comment.ko 2024-03-04T09:13:51.635 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_connbytes.ko 2024-03-04T09:13:51.636 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_connlabel.ko 2024-03-04T09:13:51.636 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_connlimit.ko 2024-03-04T09:13:51.636 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_connmark.ko 2024-03-04T09:13:51.636 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_conntrack.ko 2024-03-04T09:13:51.636 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_cpu.ko 2024-03-04T09:13:51.636 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_dccp.ko 2024-03-04T09:13:51.636 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_devgroup.ko 2024-03-04T09:13:51.636 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_dscp.ko 2024-03-04T09:13:51.636 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_ecn.ko 2024-03-04T09:13:51.636 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_esp.ko 2024-03-04T09:13:51.636 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_hashlimit.ko 2024-03-04T09:13:51.636 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_helper.ko 2024-03-04T09:13:51.636 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_hl.ko 2024-03-04T09:13:51.636 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_ipcomp.ko 2024-03-04T09:13:51.636 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_iprange.ko 2024-03-04T09:13:51.636 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_ipvs.ko 2024-03-04T09:13:51.637 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_l2tp.ko 2024-03-04T09:13:51.637 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_length.ko 2024-03-04T09:13:51.637 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_limit.ko 2024-03-04T09:13:51.637 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_mac.ko 2024-03-04T09:13:51.637 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_mark.ko 2024-03-04T09:13:51.637 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_multiport.ko 2024-03-04T09:13:51.637 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_nat.ko 2024-03-04T09:13:51.637 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_nfacct.ko 2024-03-04T09:13:51.637 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_osf.ko 2024-03-04T09:13:51.637 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_owner.ko 2024-03-04T09:13:51.637 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_physdev.ko 2024-03-04T09:13:51.637 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_pkttype.ko 2024-03-04T09:13:51.637 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_policy.ko 2024-03-04T09:13:51.637 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_quota.ko 2024-03-04T09:13:51.637 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_rateest.ko 2024-03-04T09:13:51.638 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_realm.ko 2024-03-04T09:13:51.638 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_recent.ko 2024-03-04T09:13:51.638 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_sctp.ko 2024-03-04T09:13:51.638 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_set.ko 2024-03-04T09:13:51.638 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_socket.ko 2024-03-04T09:13:51.638 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_state.ko 2024-03-04T09:13:51.638 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_statistic.ko 2024-03-04T09:13:51.638 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_string.ko 2024-03-04T09:13:51.638 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_tcpmss.ko 2024-03-04T09:13:51.638 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_time.ko 2024-03-04T09:13:51.638 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_u32.ko 2024-03-04T09:13:51.638 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netlink 2024-03-04T09:13:51.638 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netlink/netlink_diag.ko 2024-03-04T09:13:51.638 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/nsh 2024-03-04T09:13:51.638 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/nsh/nsh.ko 2024-03-04T09:13:51.639 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/openvswitch 2024-03-04T09:13:51.639 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/openvswitch/openvswitch.ko 2024-03-04T09:13:51.639 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/openvswitch/vport-geneve.ko 2024-03-04T09:13:51.639 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/openvswitch/vport-gre.ko 2024-03-04T09:13:51.639 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/openvswitch/vport-vxlan.ko 2024-03-04T09:13:51.639 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/packet 2024-03-04T09:13:51.639 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/packet/af_packet_diag.ko 2024-03-04T09:13:51.639 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/rds 2024-03-04T09:13:51.639 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/rds/rds.ko 2024-03-04T09:13:51.639 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/rds/rds_rdma.ko 2024-03-04T09:13:51.639 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/rds/rds_tcp.ko 2024-03-04T09:13:51.639 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched 2024-03-04T09:13:51.639 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/act_csum.ko 2024-03-04T09:13:51.639 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/act_gact.ko 2024-03-04T09:13:51.640 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/act_mirred.ko 2024-03-04T09:13:51.640 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/act_nat.ko 2024-03-04T09:13:51.640 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/act_pedit.ko 2024-03-04T09:13:51.640 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/act_police.ko 2024-03-04T09:13:51.640 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/act_simple.ko 2024-03-04T09:13:51.640 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/act_skbedit.ko 2024-03-04T09:13:51.640 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/act_vlan.ko 2024-03-04T09:13:51.640 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/cls_basic.ko 2024-03-04T09:13:51.640 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/cls_bpf.ko 2024-03-04T09:13:51.640 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/cls_flow.ko 2024-03-04T09:13:51.640 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/cls_fw.ko 2024-03-04T09:13:51.640 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/cls_route.ko 2024-03-04T09:13:51.640 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/cls_u32.ko 2024-03-04T09:13:51.640 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/em_cmp.ko 2024-03-04T09:13:51.640 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/em_ipset.ko 2024-03-04T09:13:51.641 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/em_meta.ko 2024-03-04T09:13:51.641 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/em_nbyte.ko 2024-03-04T09:13:51.641 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/em_text.ko 2024-03-04T09:13:51.641 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/em_u32.ko 2024-03-04T09:13:51.641 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/sch_choke.ko 2024-03-04T09:13:51.641 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/sch_codel.ko 2024-03-04T09:13:51.641 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/sch_drr.ko 2024-03-04T09:13:51.641 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/sch_fq.ko 2024-03-04T09:13:51.641 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/sch_gred.ko 2024-03-04T09:13:51.641 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/sch_hfsc.ko 2024-03-04T09:13:51.641 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/sch_hhf.ko 2024-03-04T09:13:51.641 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/sch_htb.ko 2024-03-04T09:13:51.641 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/sch_ingress.ko 2024-03-04T09:13:51.641 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/sch_mqprio.ko 2024-03-04T09:13:51.641 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/sch_mqprio_lib.ko 2024-03-04T09:13:51.642 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/sch_multiq.ko 2024-03-04T09:13:51.642 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/sch_netem.ko 2024-03-04T09:13:51.642 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/sch_pie.ko 2024-03-04T09:13:51.642 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/sch_plug.ko 2024-03-04T09:13:51.642 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/sch_prio.ko 2024-03-04T09:13:51.642 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/sch_qfq.ko 2024-03-04T09:13:51.642 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/sch_red.ko 2024-03-04T09:13:51.642 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/sch_sfb.ko 2024-03-04T09:13:51.642 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/sch_sfq.ko 2024-03-04T09:13:51.642 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/sch_tbf.ko 2024-03-04T09:13:51.642 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/sch_teql.ko 2024-03-04T09:13:51.642 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sctp 2024-03-04T09:13:51.642 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sctp/sctp.ko 2024-03-04T09:13:51.642 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sctp/sctp_diag.ko 2024-03-04T09:13:51.642 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sunrpc 2024-03-04T09:13:51.643 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sunrpc/auth_gss 2024-03-04T09:13:51.643 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-03-04T09:13:51.643 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-03-04T09:13:51.643 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sunrpc/sunrpc.ko 2024-03-04T09:13:51.643 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sunrpc/xprtrdma 2024-03-04T09:13:51.643 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-03-04T09:13:51.643 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/tipc 2024-03-04T09:13:51.643 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/tipc/diag.ko 2024-03-04T09:13:51.643 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/tipc/tipc.ko 2024-03-04T09:13:51.643 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/unix 2024-03-04T09:13:51.643 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/unix/unix_diag.ko 2024-03-04T09:13:51.643 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/vmw_vsock 2024-03-04T09:13:51.643 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-03-04T09:13:51.643 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-03-04T09:13:51.643 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/vmw_vsock/vsock.ko 2024-03-04T09:13:51.644 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/vmw_vsock/vsock_diag.ko 2024-03-04T09:13:51.644 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/vmw_vsock/vsock_loopback.ko 2024-03-04T09:13:51.644 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/xfrm 2024-03-04T09:13:51.644 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/xfrm/xfrm_ipcomp.ko 2024-03-04T09:13:51.644 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/security 2024-03-04T09:13:51.644 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/security/keys 2024-03-04T09:13:51.644 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/security/keys/encrypted-keys 2024-03-04T09:13:51.644 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-03-04T09:13:51.644 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/security/keys/trusted-keys 2024-03-04T09:13:51.644 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/security/keys/trusted-keys/trusted.ko 2024-03-04T09:13:51.644 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/virt 2024-03-04T09:13:51.644 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/virt/lib 2024-03-04T09:13:51.644 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/virt/lib/irqbypass.ko 2024-03-04T09:13:51.644 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/modules.alias 2024-03-04T09:13:51.644 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/modules.alias.bin 2024-03-04T09:13:51.645 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/modules.builtin 2024-03-04T09:13:51.645 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/modules.builtin.alias.bin 2024-03-04T09:13:51.645 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/modules.builtin.bin 2024-03-04T09:13:51.645 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/modules.builtin.modinfo 2024-03-04T09:13:51.645 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/modules.dep 2024-03-04T09:13:51.645 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/modules.dep.bin 2024-03-04T09:13:51.645 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/modules.devname 2024-03-04T09:13:51.645 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/modules.order 2024-03-04T09:13:51.645 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/modules.softdep 2024-03-04T09:13:51.645 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/modules.symbols 2024-03-04T09:13:51.645 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/modules.symbols.bin 2024-03-04T09:13:51.645 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/vmlinuz 2024-03-04T09:13:51.649 DEBUG:teuthology.task.kernel:get_image_version: 6.8.0-rc6-g786939134d7e 2024-03-04T09:13:51.649 DEBUG:teuthology.orchestra.run.smithi006:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-03-04T09:13:51.672 INFO:teuthology.orchestra.run.smithi006.stdout:postinstall scriptlet (using /bin/sh): 2024-03-04T09:13:51.672 INFO:teuthology.orchestra.run.smithi006.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-03-04T09:13:51.672 INFO:teuthology.orchestra.run.smithi006.stdout: /usr/bin/kernel-install add 6.8.0-rc6-g786939134d7e /lib/modules/6.8.0-rc6-g786939134d7e/vmlinuz 2024-03-04T09:13:51.672 INFO:teuthology.orchestra.run.smithi006.stdout:fi 2024-03-04T09:13:51.673 INFO:teuthology.orchestra.run.smithi006.stdout:for file in vmlinuz System.map config; do 2024-03-04T09:13:51.673 INFO:teuthology.orchestra.run.smithi006.stdout: if ! cmp --silent "/lib/modules/6.8.0-rc6-g786939134d7e/${file}" "/boot/${file}-6.8.0-rc6-g786939134d7e"; then 2024-03-04T09:13:51.673 INFO:teuthology.orchestra.run.smithi006.stdout: cp "/lib/modules/6.8.0-rc6-g786939134d7e/${file}" "/boot/${file}-6.8.0-rc6-g786939134d7e" 2024-03-04T09:13:51.673 INFO:teuthology.orchestra.run.smithi006.stdout: fi 2024-03-04T09:13:51.673 INFO:teuthology.orchestra.run.smithi006.stdout:done 2024-03-04T09:13:51.673 INFO:teuthology.orchestra.run.smithi006.stdout:preuninstall scriptlet (using /bin/sh): 2024-03-04T09:13:51.673 INFO:teuthology.orchestra.run.smithi006.stdout:if [ -x /sbin/new-kernel-pkg ]; then 2024-03-04T09:13:51.673 INFO:teuthology.orchestra.run.smithi006.stdout:new-kernel-pkg --remove 6.8.0-rc6-g786939134d7e --rminitrd --initrdfile=/boot/initramfs-6.8.0-rc6-g786939134d7e.img 2024-03-04T09:13:51.673 INFO:teuthology.orchestra.run.smithi006.stdout:elif [ -x /usr/bin/kernel-install ]; then 2024-03-04T09:13:51.673 INFO:teuthology.orchestra.run.smithi006.stdout:kernel-install remove 6.8.0-rc6-g786939134d7e 2024-03-04T09:13:51.673 INFO:teuthology.orchestra.run.smithi006.stdout:fi 2024-03-04T09:13:51.673 INFO:teuthology.orchestra.run.smithi006.stdout:postuninstall scriptlet (using /bin/sh): 2024-03-04T09:13:51.673 INFO:teuthology.orchestra.run.smithi006.stdout:if [ -x /sbin/update-bootloader ]; then 2024-03-04T09:13:51.673 INFO:teuthology.orchestra.run.smithi006.stdout:/sbin/update-bootloader --remove 6.8.0-rc6-g786939134d7e 2024-03-04T09:13:51.673 INFO:teuthology.orchestra.run.smithi006.stdout:fi 2024-03-04T09:13:51.674 DEBUG:teuthology.orchestra.run.smithi006:> sudo rpm -qi grub2-tools 2024-03-04T09:13:51.754 INFO:teuthology.orchestra.run.smithi006.stdout:Name : grub2-tools 2024-03-04T09:13:51.755 INFO:teuthology.orchestra.run.smithi006.stdout:Epoch : 1 2024-03-04T09:13:51.755 INFO:teuthology.orchestra.run.smithi006.stdout:Version : 2.06 2024-03-04T09:13:51.755 INFO:teuthology.orchestra.run.smithi006.stdout:Release : 68.el9 2024-03-04T09:13:51.755 INFO:teuthology.orchestra.run.smithi006.stdout:Architecture: x86_64 2024-03-04T09:13:51.755 INFO:teuthology.orchestra.run.smithi006.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-04T09:13:51.755 INFO:teuthology.orchestra.run.smithi006.stdout:Group : Unspecified 2024-03-04T09:13:51.755 INFO:teuthology.orchestra.run.smithi006.stdout:Size : 8264254 2024-03-04T09:13:51.755 INFO:teuthology.orchestra.run.smithi006.stdout:License : GPLv3+ 2024-03-04T09:13:51.755 INFO:teuthology.orchestra.run.smithi006.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-04T09:13:51.755 INFO:teuthology.orchestra.run.smithi006.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-04T09:13:51.755 INFO:teuthology.orchestra.run.smithi006.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-04T09:13:51.756 INFO:teuthology.orchestra.run.smithi006.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-04T09:13:51.756 INFO:teuthology.orchestra.run.smithi006.stdout:Packager : builder@centos.org 2024-03-04T09:13:51.756 INFO:teuthology.orchestra.run.smithi006.stdout:Vendor : CentOS 2024-03-04T09:13:51.756 INFO:teuthology.orchestra.run.smithi006.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-04T09:13:51.756 INFO:teuthology.orchestra.run.smithi006.stdout:Summary : Support tools for GRUB. 2024-03-04T09:13:51.757 INFO:teuthology.orchestra.run.smithi006.stdout:Description : 2024-03-04T09:13:51.757 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:13:51.757 INFO:teuthology.orchestra.run.smithi006.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-04T09:13:51.757 INFO:teuthology.orchestra.run.smithi006.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-04T09:13:51.757 INFO:teuthology.orchestra.run.smithi006.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-04T09:13:51.757 INFO:teuthology.orchestra.run.smithi006.stdout:hardware devices. 2024-03-04T09:13:51.757 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:13:51.757 INFO:teuthology.orchestra.run.smithi006.stdout:This subpackage provides tools for support of all platforms. 2024-03-04T09:13:51.758 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-04T09:13:51.758 INFO:teuthology.task.kernel:Updating grub on smithi006 to boot 6.8.0-rc6-g786939134d7e 2024-03-04T09:13:51.759 DEBUG:teuthology.orchestra.run.smithi006:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-04T09:13:52.390 INFO:teuthology.orchestra.run.smithi006.stderr:Generating grub configuration file ... 2024-03-04T09:13:53.839 INFO:teuthology.orchestra.run.smithi006.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-04T09:13:53.883 INFO:teuthology.orchestra.run.smithi006.stderr:done 2024-03-04T09:13:53.886 DEBUG:teuthology.orchestra.run.smithi006:> mktemp 2024-03-04T09:13:53.901 INFO:teuthology.orchestra.run.smithi006.stdout:/tmp/tmp.TBQoFRT5Y0 2024-03-04T09:13:53.901 DEBUG:teuthology.orchestra.run.smithi006:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.TBQoFRT5Y0 2024-03-04T09:13:53.972 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 0666 /tmp/tmp.TBQoFRT5Y0 2024-03-04T09:13:54.115 DEBUG:teuthology.orchestra.remote:smithi006:/tmp/tmp.TBQoFRT5Y0 is 6KB 2024-03-04T09:13:54.164 DEBUG:teuthology.orchestra.run.smithi006:> rm -fr /tmp/tmp.TBQoFRT5Y0 2024-03-04T09:13:54.179 DEBUG:teuthology.orchestra.run.smithi006:> sudo /bin/ls /boot/loader/entries || true 2024-03-04T09:13:54.247 INFO:teuthology.orchestra.run.smithi006.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-04T09:13:54.247 INFO:teuthology.orchestra.run.smithi006.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-04T09:13:54.247 INFO:teuthology.orchestra.run.smithi006.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-04T09:13:54.247 INFO:teuthology.orchestra.run.smithi006.stdout:15949e3c88704300882482eddfd7baae-6.8.0-rc6-g786939134d7e.conf 2024-03-04T09:13:54.248 DEBUG:teuthology.orchestra.run.smithi006:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-6.8.0-rc6-g786939134d7e 2024-03-04T09:13:54.391 DEBUG:teuthology.orchestra.run.smithi006:> sudo shutdown -r now 2024-03-04T09:14:10.389 INFO:teuthology.task.kernel:Checking client client.1 for new kernel version... 2024-03-04T09:14:10.390 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-03-04T09:14:10.391 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T09:14:24.408 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-03-04T09:14:24.408 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi006.front.sepia.ceph.com' 2024-03-04T09:14:24.409 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi006.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T09:14:44.928 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.146 2024-03-04T09:14:53.930 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-03-04T09:14:53.932 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T09:15:12.316 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.146 2024-03-04T09:15:24.329 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-03-04T09:15:24.330 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T09:15:24.419 DEBUG:teuthology.orchestra.remote:timed out 2024-03-04T09:15:27.388 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.146 2024-03-04T09:15:33.424 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi006.front.sepia.ceph.com' 2024-03-04T09:15:33.425 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi006.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T09:15:34.020 DEBUG:teuthology.orchestra.run.smithi006:> true 2024-03-04T09:15:34.684 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi006.front.sepia.ceph.com' 2024-03-04T09:15:34.684 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "6.8.0-rc6-g786939134d7e"... 2024-03-04T09:15:34.685 DEBUG:teuthology.orchestra.run.smithi006:> uname -r 2024-03-04T09:15:34.743 INFO:teuthology.orchestra.run.smithi006.stdout:6.8.0-rc6-g786939134d7e 2024-03-04T09:15:34.743 DEBUG:teuthology.task.kernel:current kernel version is 6.8.0-rc6-g786939134d7e vs 6.8.0-rc6-g786939134d7e 2024-03-04T09:15:34.743 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-04T09:15:34.743 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-04T09:15:35.748 DEBUG:teuthology.parallel:result is None 2024-03-04T09:15:42.389 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-03-04T09:15:42.390 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T09:15:42.780 DEBUG:teuthology.orchestra.run.smithi146:> true 2024-03-04T09:15:43.418 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-03-04T09:15:43.418 INFO:teuthology.task.kernel:Checking kernel version of client.1, want "6.8.0-rc6-g786939134d7e"... 2024-03-04T09:15:43.418 DEBUG:teuthology.orchestra.run.smithi146:> uname -r 2024-03-04T09:15:43.455 INFO:teuthology.orchestra.run.smithi146.stdout:6.8.0-rc6-g786939134d7e 2024-03-04T09:15:43.456 DEBUG:teuthology.task.kernel:current kernel version is 6.8.0-rc6-g786939134d7e vs 6.8.0-rc6-g786939134d7e 2024-03-04T09:15:43.456 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-04T09:15:43.456 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-04T09:15:44.457 DEBUG:teuthology.parallel:result is None 2024-03-04T09:15:44.458 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-04T09:15:44.464 INFO:teuthology.task.internal:Creating test directory... 2024-03-04T09:15:44.465 DEBUG:teuthology.orchestra.run.smithi006:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-04T09:15:44.471 DEBUG:teuthology.orchestra.run.smithi146:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-04T09:15:44.513 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-04T09:15:44.520 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-04T09:15:44.526 INFO:teuthology.task.internal:Creating archive directory... 2024-03-04T09:15:44.527 DEBUG:teuthology.orchestra.run.smithi006:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-04T09:15:44.552 DEBUG:teuthology.orchestra.run.smithi146:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-04T09:15:44.609 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-04T09:15:44.615 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-04T09:15:44.615 DEBUG:teuthology.orchestra.run.smithi006:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-04T09:15:44.634 DEBUG:teuthology.orchestra.run.smithi146:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-04T09:15:44.767 INFO:teuthology.orchestra.run.smithi146.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-04T09:15:44.787 INFO:teuthology.orchestra.run.smithi006.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-04T09:15:44.813 INFO:teuthology.orchestra.run.smithi146.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-04T09:15:44.843 INFO:teuthology.orchestra.run.smithi006.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-04T09:15:44.847 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-04T09:15:44.854 INFO:teuthology.task.internal:Configuring sudo... 2024-03-04T09:15:44.854 DEBUG:teuthology.orchestra.run.smithi006:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-04T09:15:44.858 DEBUG:teuthology.orchestra.run.smithi146:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-04T09:15:44.933 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-04T09:15:44.942 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-04T09:15:44.943 DEBUG:teuthology.orchestra.run.smithi006:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-04T09:15:44.948 DEBUG:teuthology.orchestra.run.smithi146:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-04T09:15:44.990 DEBUG:teuthology.orchestra.run.smithi006:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-04T09:15:45.102 DEBUG:teuthology.orchestra.run.smithi006:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-04T09:15:45.185 DEBUG:teuthology.orchestra.run.smithi006:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-04T09:15:45.268 DEBUG:teuthology.orchestra.run.smithi006:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-04T09:15:45.348 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-04T09:15:45.348 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-04T09:15:45.424 DEBUG:teuthology.orchestra.run.smithi146:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-04T09:15:45.513 DEBUG:teuthology.orchestra.run.smithi146:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-04T09:15:45.607 DEBUG:teuthology.orchestra.run.smithi146:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-04T09:15:45.681 DEBUG:teuthology.orchestra.run.smithi146:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-04T09:15:45.762 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-04T09:15:45.762 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-04T09:15:45.825 DEBUG:teuthology.orchestra.run.smithi006:> sudo service rsyslog restart 2024-03-04T09:15:45.833 DEBUG:teuthology.orchestra.run.smithi146:> sudo service rsyslog restart 2024-03-04T09:15:45.955 INFO:teuthology.orchestra.run.smithi006.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-04T09:15:45.969 INFO:teuthology.orchestra.run.smithi146.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-04T09:15:46.251 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-04T09:15:46.260 INFO:teuthology.task.internal:Starting timer... 2024-03-04T09:15:46.260 INFO:teuthology.run_tasks:Running task pcp... 2024-03-04T09:15:46.270 INFO:teuthology.run_tasks:Running task selinux... 2024-03-04T09:15:46.278 DEBUG:teuthology.orchestra.run.smithi006:> sudo service auditd rotate 2024-03-04T09:15:46.427 INFO:teuthology.orchestra.run.smithi006.stdout:Rotating logs: 2024-03-04T09:15:46.432 DEBUG:teuthology.orchestra.run.smithi146:> sudo service auditd rotate 2024-03-04T09:15:46.531 INFO:teuthology.orchestra.run.smithi146.stdout:Rotating logs: 2024-03-04T09:15:46.535 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-04T09:15:46.535 DEBUG:teuthology.orchestra.run.smithi006:> /usr/sbin/getenforce 2024-03-04T09:15:46.582 INFO:teuthology.orchestra.run.smithi006.stdout:Permissive 2024-03-04T09:15:46.582 DEBUG:teuthology.orchestra.run.smithi146:> /usr/sbin/getenforce 2024-03-04T09:15:46.632 INFO:teuthology.orchestra.run.smithi146.stdout:Permissive 2024-03-04T09:15:46.632 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi006.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi146.front.sepia.ceph.com': 'permissive'} 2024-03-04T09:15:46.632 DEBUG:teuthology.orchestra.run.smithi006:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-04T09:15:46.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T09:15:46.706 DEBUG:teuthology.orchestra.run.smithi146:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-04T09:15:46.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T09:15:46.774 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-04T09:15:46.774 DEBUG:teuthology.orchestra.run.smithi006:> sudo /usr/sbin/setenforce permissive 2024-03-04T09:15:46.853 DEBUG:teuthology.orchestra.run.smithi146:> sudo /usr/sbin/setenforce permissive 2024-03-04T09:15:46.928 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-04T09:15:46.937 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-03-04T09:15:46.943 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-03-04T09:15:47.100 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-04T09:15:47.114 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-04T09:15:47.115 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi006.front.sepia.ceph.com,smithi146.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-04T09:21:51.918 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi006.front.sepia.ceph.com'), Remote(name='ubuntu@smithi146.front.sepia.ceph.com')] 2024-03-04T09:21:51.919 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi006.front.sepia.ceph.com' 2024-03-04T09:21:51.921 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi006.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T09:21:52.071 DEBUG:teuthology.orchestra.run.smithi006:> true 2024-03-04T09:21:52.228 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi006.front.sepia.ceph.com' 2024-03-04T09:21:52.228 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-03-04T09:21:52.228 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T09:21:52.375 DEBUG:teuthology.orchestra.run.smithi146:> true 2024-03-04T09:21:52.523 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-03-04T09:21:52.523 INFO:teuthology.run_tasks:Running task clock... 2024-03-04T09:21:52.534 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-04T09:21:52.534 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-04T09:21:52.535 DEBUG:teuthology.orchestra.run.smithi006:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-04T09:21:52.542 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-04T09:21:52.542 DEBUG:teuthology.orchestra.run.smithi146:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-04T09:21:52.642 INFO:teuthology.orchestra.run.smithi146.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-04T09:21:52.647 INFO:teuthology.orchestra.run.smithi006.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-04T09:21:52.693 INFO:teuthology.orchestra.run.smithi146.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-04T09:21:52.702 INFO:teuthology.orchestra.run.smithi006.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-04T09:21:52.784 INFO:teuthology.orchestra.run.smithi146.stderr:sudo: ntpd: command not found 2024-03-04T09:21:52.804 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: ntpd: command not found 2024-03-04T09:21:52.828 INFO:teuthology.orchestra.run.smithi146.stdout:506 Cannot talk to daemon 2024-03-04T09:21:52.851 INFO:teuthology.orchestra.run.smithi006.stdout:506 Cannot talk to daemon 2024-03-04T09:21:52.880 INFO:teuthology.orchestra.run.smithi146.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-04T09:21:52.906 INFO:teuthology.orchestra.run.smithi006.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-04T09:21:52.934 INFO:teuthology.orchestra.run.smithi146.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-04T09:21:52.966 INFO:teuthology.orchestra.run.smithi006.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-04T09:21:53.065 INFO:teuthology.orchestra.run.smithi146.stderr:bash: line 1: ntpq: command not found 2024-03-04T09:21:53.073 INFO:teuthology.orchestra.run.smithi146.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-04T09:21:53.073 INFO:teuthology.orchestra.run.smithi146.stdout:=============================================================================== 2024-03-04T09:21:53.073 INFO:teuthology.orchestra.run.smithi146.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-04T09:21:53.073 INFO:teuthology.orchestra.run.smithi146.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-04T09:21:53.073 INFO:teuthology.orchestra.run.smithi146.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-04T09:21:53.073 INFO:teuthology.orchestra.run.smithi146.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-04T09:21:53.104 INFO:teuthology.orchestra.run.smithi006.stderr:bash: line 1: ntpq: command not found 2024-03-04T09:21:53.113 INFO:teuthology.orchestra.run.smithi006.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-04T09:21:53.114 INFO:teuthology.orchestra.run.smithi006.stdout:=============================================================================== 2024-03-04T09:21:53.114 INFO:teuthology.orchestra.run.smithi006.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-04T09:21:53.114 INFO:teuthology.orchestra.run.smithi006.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-04T09:21:53.114 INFO:teuthology.orchestra.run.smithi006.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-04T09:21:53.114 INFO:teuthology.orchestra.run.smithi006.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-04T09:21:53.115 INFO:teuthology.run_tasks:Running task install... 2024-03-04T09:21:53.123 DEBUG:teuthology.task.install:project ceph 2024-03-04T09:21:53.123 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd'}} 2024-03-04T09:21:53.124 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd'} 2024-03-04T09:21:53.124 INFO:teuthology.task.install:Using flavor: default 2024-03-04T09:21:53.130 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-04T09:21:53.130 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-03-04T09:21:53.130 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd', 'tag': None, 'wait_for_package': False} 2024-03-04T09:21:53.131 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd 2024-03-04T09:21:53.133 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd', 'tag': None, 'wait_for_package': False} 2024-03-04T09:21:53.133 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd 2024-03-04T09:21:53.503 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/centos/9/flavors/default/ 2024-03-04T09:21:53.503 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1739.g1bc250c7 2024-03-04T09:21:53.547 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/centos/9/flavors/default/ 2024-03-04T09:21:53.547 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1739.g1bc250c7 2024-03-04T09:21:53.754 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-04T09:21:53.754 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-04T09:21:53.754 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-04T09:21:53.783 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-04T09:21:53.784 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-04T09:21:53.784 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-04T09:21:53.852 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-04T09:21:53.853 DEBUG:teuthology.orchestra.run.smithi146:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-04T09:21:53.888 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-04T09:21:53.888 DEBUG:teuthology.orchestra.run.smithi006:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-04T09:21:53.990 DEBUG:teuthology.orchestra.run.smithi146:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-04T09:21:54.030 DEBUG:teuthology.orchestra.run.smithi006:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-04T09:21:54.118 DEBUG:teuthology.orchestra.run.smithi146:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-04T09:21:54.170 DEBUG:teuthology.orchestra.run.smithi006:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-04T09:21:54.208 INFO:teuthology.orchestra.run.smithi146.stdout:check_obsoletes = 1 2024-03-04T09:21:54.213 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum clean all 2024-03-04T09:21:54.264 INFO:teuthology.orchestra.run.smithi006.stdout:check_obsoletes = 1 2024-03-04T09:21:54.269 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum clean all 2024-03-04T09:21:54.562 INFO:teuthology.orchestra.run.smithi146.stdout:74 files removed 2024-03-04T09:21:54.601 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-04T09:21:54.630 INFO:teuthology.orchestra.run.smithi006.stdout:74 files removed 2024-03-04T09:21:54.670 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-04T09:21:55.555 INFO:teuthology.orchestra.run.smithi146.stdout:ceph packages for x86_64 184 kB/s | 83 kB 00:00 2024-03-04T09:21:55.657 INFO:teuthology.orchestra.run.smithi006.stdout:ceph packages for x86_64 180 kB/s | 83 kB 00:00 2024-03-04T09:21:55.997 INFO:teuthology.orchestra.run.smithi146.stdout:ceph noarch packages 29 kB/s | 10 kB 00:00 2024-03-04T09:21:56.208 INFO:teuthology.orchestra.run.smithi006.stdout:ceph noarch packages 22 kB/s | 10 kB 00:00 2024-03-04T09:21:56.576 INFO:teuthology.orchestra.run.smithi146.stdout:ceph source packages 3.7 kB/s | 1.9 kB 00:00 2024-03-04T09:21:56.802 INFO:teuthology.orchestra.run.smithi006.stdout:ceph source packages 3.7 kB/s | 1.9 kB 00:00 2024-03-04T09:21:57.318 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS-9 - AppStream 10 MB/s | 6.7 MB 00:00 2024-03-04T09:21:57.513 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS-9 - AppStream 11 MB/s | 6.7 MB 00:00 2024-03-04T09:21:59.144 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS-9 - BaseOS 4.0 MB/s | 2.0 MB 00:00 2024-03-04T09:21:59.248 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS-9 - BaseOS 4.6 MB/s | 2.0 MB 00:00 2024-03-04T09:22:00.394 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.0 MB 00:00 2024-03-04T09:22:04.711 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS Stream 9 - AppStream 6.3 MB/s | 19 MB 00:03 2024-03-04T09:22:09.655 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS Stream 9 - BaseOS 813 kB/s | 8.0 MB 00:10 2024-03-04T09:22:10.312 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS Stream 9 - CRB 3.8 MB/s | 6.1 MB 00:01 2024-03-04T09:22:12.204 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS Stream 9 - Extras packages 29 kB/s | 15 kB 00:00 2024-03-04T09:22:12.706 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS Stream 9 - AppStream 10 MB/s | 19 MB 00:01 2024-03-04T09:22:13.631 INFO:teuthology.orchestra.run.smithi006.stdout:Extra Packages for Enterprise Linux 17 MB/s | 20 MB 00:01 2024-03-04T09:22:17.266 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS Stream 9 - CRB 9.2 MB/s | 6.1 MB 00:00 2024-03-04T09:22:19.049 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS Stream 9 - Extras packages 32 kB/s | 15 kB 00:00 2024-03-04T09:22:19.540 INFO:teuthology.orchestra.run.smithi006.stdout:lab-extras 28 kB/s | 1.7 kB 00:00 2024-03-04T09:22:21.065 INFO:teuthology.orchestra.run.smithi146.stdout:Extra Packages for Enterprise Linux 11 MB/s | 20 MB 00:01 2024-03-04T09:22:22.453 INFO:teuthology.orchestra.run.smithi006.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-04T09:22:22.455 INFO:teuthology.orchestra.run.smithi006.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-04T09:22:22.480 INFO:teuthology.orchestra.run.smithi006.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-04T09:22:22.481 INFO:teuthology.orchestra.run.smithi006.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-04T09:22:22.483 INFO:teuthology.orchestra.run.smithi006.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-04T09:22:22.485 INFO:teuthology.orchestra.run.smithi006.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-04T09:22:22.488 INFO:teuthology.orchestra.run.smithi006.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-04T09:22:22.492 INFO:teuthology.orchestra.run.smithi006.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-04T09:22:22.493 INFO:teuthology.orchestra.run.smithi006.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-04T09:22:22.589 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-03-04T09:22:22.606 INFO:teuthology.orchestra.run.smithi006.stdout:============================================================================================ 2024-03-04T09:22:22.606 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repository Size 2024-03-04T09:22:22.606 INFO:teuthology.orchestra.run.smithi006.stdout:============================================================================================ 2024-03-04T09:22:22.606 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2024-03-04T09:22:22.606 INFO:teuthology.orchestra.run.smithi006.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-04T09:22:22.606 INFO:teuthology.orchestra.run.smithi006.stdout: ceph x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 6.6 k 2024-03-04T09:22:22.606 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-base x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 5.3 M 2024-03-04T09:22:22.606 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-fuse x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 888 k 2024-03-04T09:22:22.607 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 145 k 2024-03-04T09:22:22.607 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 1.5 M 2024-03-04T09:22:22.607 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 138 k 2024-03-04T09:22:22.607 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 1.8 M 2024-03-04T09:22:22.607 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 7.4 M 2024-03-04T09:22:22.607 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-rook noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 49 k 2024-03-04T09:22:22.607 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-radosgw x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 9.8 M 2024-03-04T09:22:22.607 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-test x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 47 M 2024-03-04T09:22:22.607 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-volume noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 264 k 2024-03-04T09:22:22.607 INFO:teuthology.orchestra.run.smithi006.stdout: cephadm noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 748 k 2024-03-04T09:22:22.608 INFO:teuthology.orchestra.run.smithi006.stdout: cephfs-mirror x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 220 k 2024-03-04T09:22:22.608 INFO:teuthology.orchestra.run.smithi006.stdout: cephfs-shell x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 27 k 2024-03-04T09:22:22.608 INFO:teuthology.orchestra.run.smithi006.stdout: cephfs-top noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 18 k 2024-03-04T09:22:22.608 INFO:teuthology.orchestra.run.smithi006.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 CentOS-AppStream 20 k 2024-03-04T09:22:22.608 INFO:teuthology.orchestra.run.smithi006.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-04T09:22:22.608 INFO:teuthology.orchestra.run.smithi006.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-04T09:22:22.608 INFO:teuthology.orchestra.run.smithi006.stdout: libcephfs-devel x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 31 k 2024-03-04T09:22:22.608 INFO:teuthology.orchestra.run.smithi006.stdout: libcephfs2 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 747 k 2024-03-04T09:22:22.608 INFO:teuthology.orchestra.run.smithi006.stdout: librados-devel x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 127 k 2024-03-04T09:22:22.608 INFO:teuthology.orchestra.run.smithi006.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-04T09:22:22.608 INFO:teuthology.orchestra.run.smithi006.stdout: libuuid-devel x86_64 2.37.4-18.el9 CentOS-AppStream 24 k 2024-03-04T09:22:22.609 INFO:teuthology.orchestra.run.smithi006.stdout: openssl-devel x86_64 1:3.0.7-27.el9 CentOS-AppStream 3.0 M 2024-03-04T09:22:22.609 INFO:teuthology.orchestra.run.smithi006.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-04T09:22:22.609 INFO:teuthology.orchestra.run.smithi006.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-04T09:22:22.609 INFO:teuthology.orchestra.run.smithi006.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-04T09:22:22.609 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cephfs x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 162 k 2024-03-04T09:22:22.609 INFO:teuthology.orchestra.run.smithi006.stdout: python3-rados x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 321 k 2024-03-04T09:22:22.609 INFO:teuthology.orchestra.run.smithi006.stdout: python3-rbd x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 298 k 2024-03-04T09:22:22.609 INFO:teuthology.orchestra.run.smithi006.stdout: python3-rgw x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 99 k 2024-03-04T09:22:22.609 INFO:teuthology.orchestra.run.smithi006.stdout: rbd-fuse x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 85 k 2024-03-04T09:22:22.609 INFO:teuthology.orchestra.run.smithi006.stdout: rbd-mirror x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 3.1 M 2024-03-04T09:22:22.610 INFO:teuthology.orchestra.run.smithi006.stdout: rbd-nbd x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 171 k 2024-03-04T09:22:22.610 INFO:teuthology.orchestra.run.smithi006.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 CentOS-AppStream 87 k 2024-03-04T09:22:22.610 INFO:teuthology.orchestra.run.smithi006.stdout:Upgrading: 2024-03-04T09:22:22.610 INFO:teuthology.orchestra.run.smithi006.stdout: NetworkManager x86_64 1:1.46.0-1.el9 CentOS-BaseOS 2.3 M 2024-03-04T09:22:22.610 INFO:teuthology.orchestra.run.smithi006.stdout: NetworkManager-libnm x86_64 1:1.46.0-1.el9 CentOS-BaseOS 1.8 M 2024-03-04T09:22:22.610 INFO:teuthology.orchestra.run.smithi006.stdout: NetworkManager-team x86_64 1:1.46.0-1.el9 CentOS-BaseOS 41 k 2024-03-04T09:22:22.610 INFO:teuthology.orchestra.run.smithi006.stdout: NetworkManager-tui x86_64 1:1.46.0-1.el9 CentOS-BaseOS 247 k 2024-03-04T09:22:22.610 INFO:teuthology.orchestra.run.smithi006.stdout: acl x86_64 2.3.1-4.el9 CentOS-BaseOS 69 k 2024-03-04T09:22:22.610 INFO:teuthology.orchestra.run.smithi006.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 CentOS-BaseOS 35 k 2024-03-04T09:22:22.610 INFO:teuthology.orchestra.run.smithi006.stdout: elfutils-libelf x86_64 0.190-2.el9 CentOS-BaseOS 191 k 2024-03-04T09:22:22.610 INFO:teuthology.orchestra.run.smithi006.stdout: elfutils-libs x86_64 0.190-2.el9 CentOS-BaseOS 254 k 2024-03-04T09:22:22.610 INFO:teuthology.orchestra.run.smithi006.stdout: iproute x86_64 6.7.0-2.el9 CentOS-BaseOS 802 k 2024-03-04T09:22:22.610 INFO:teuthology.orchestra.run.smithi006.stdout: iproute-tc x86_64 6.7.0-2.el9 CentOS-BaseOS 433 k 2024-03-04T09:22:22.611 INFO:teuthology.orchestra.run.smithi006.stdout: libacl x86_64 2.3.1-4.el9 CentOS-BaseOS 22 k 2024-03-04T09:22:22.611 INFO:teuthology.orchestra.run.smithi006.stdout: libacl-devel x86_64 2.3.1-4.el9 CentOS-AppStream 68 k 2024-03-04T09:22:22.611 INFO:teuthology.orchestra.run.smithi006.stdout: libblkid x86_64 2.37.4-18.el9 CentOS-BaseOS 107 k 2024-03-04T09:22:22.611 INFO:teuthology.orchestra.run.smithi006.stdout: libfdisk x86_64 2.37.4-18.el9 CentOS-BaseOS 154 k 2024-03-04T09:22:22.611 INFO:teuthology.orchestra.run.smithi006.stdout: libmount x86_64 2.37.4-18.el9 CentOS-BaseOS 135 k 2024-03-04T09:22:22.611 INFO:teuthology.orchestra.run.smithi006.stdout: librados2 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 3.4 M 2024-03-04T09:22:22.611 INFO:teuthology.orchestra.run.smithi006.stdout: librbd1 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 3.1 M 2024-03-04T09:22:22.611 INFO:teuthology.orchestra.run.smithi006.stdout: libsmartcols x86_64 2.37.4-18.el9 CentOS-BaseOS 63 k 2024-03-04T09:22:22.611 INFO:teuthology.orchestra.run.smithi006.stdout: libuuid x86_64 2.37.4-18.el9 CentOS-BaseOS 28 k 2024-03-04T09:22:22.611 INFO:teuthology.orchestra.run.smithi006.stdout: openssl x86_64 1:3.0.7-27.el9 CentOS-BaseOS 1.2 M 2024-03-04T09:22:22.611 INFO:teuthology.orchestra.run.smithi006.stdout: openssl-libs x86_64 1:3.0.7-27.el9 CentOS-BaseOS 2.2 M 2024-03-04T09:22:22.611 INFO:teuthology.orchestra.run.smithi006.stdout: util-linux x86_64 2.37.4-18.el9 CentOS-BaseOS 2.2 M 2024-03-04T09:22:22.611 INFO:teuthology.orchestra.run.smithi006.stdout: util-linux-core x86_64 2.37.4-18.el9 CentOS-BaseOS 438 k 2024-03-04T09:22:22.612 INFO:teuthology.orchestra.run.smithi006.stdout: util-linux-user x86_64 2.37.4-18.el9 CentOS-BaseOS 30 k 2024-03-04T09:22:22.612 INFO:teuthology.orchestra.run.smithi006.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-04T09:22:22.612 INFO:teuthology.orchestra.run.smithi006.stdout: xfsprogs x86_64 6.3.0-1.el9 CentOS-BaseOS 1.1 M 2024-03-04T09:22:22.612 INFO:teuthology.orchestra.run.smithi006.stdout:Installing dependencies: 2024-03-04T09:22:22.612 INFO:teuthology.orchestra.run.smithi006.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-04T09:22:22.612 INFO:teuthology.orchestra.run.smithi006.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-04T09:22:22.612 INFO:teuthology.orchestra.run.smithi006.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-04T09:22:22.612 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-common x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 20 M 2024-03-04T09:22:22.612 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 31 k 2024-03-04T09:22:22.612 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mds x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 2.3 M 2024-03-04T09:22:22.612 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 248 k 2024-03-04T09:22:22.612 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mon x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 4.7 M 2024-03-04T09:22:22.613 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-osd x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 17 M 2024-03-04T09:22:22.613 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 15 k 2024-03-04T09:22:22.613 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-selinux x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 25 k 2024-03-04T09:22:22.613 INFO:teuthology.orchestra.run.smithi006.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-04T09:22:22.613 INFO:teuthology.orchestra.run.smithi006.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-04T09:22:22.613 INFO:teuthology.orchestra.run.smithi006.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-04T09:22:22.613 INFO:teuthology.orchestra.run.smithi006.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-04T09:22:22.613 INFO:teuthology.orchestra.run.smithi006.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-04T09:22:22.613 INFO:teuthology.orchestra.run.smithi006.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-04T09:22:22.613 INFO:teuthology.orchestra.run.smithi006.stdout: jq x86_64 1.6-16.el9 CentOS-BaseOS 186 k 2024-03-04T09:22:22.613 INFO:teuthology.orchestra.run.smithi006.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-04T09:22:22.613 INFO:teuthology.orchestra.run.smithi006.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-04T09:22:22.613 INFO:teuthology.orchestra.run.smithi006.stdout: libcephsqlite x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 163 k 2024-03-04T09:22:22.614 INFO:teuthology.orchestra.run.smithi006.stdout: libgfortran x86_64 11.4.1-3.el9 CentOS-BaseOS 802 k 2024-03-04T09:22:22.614 INFO:teuthology.orchestra.run.smithi006.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-04T09:22:22.614 INFO:teuthology.orchestra.run.smithi006.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-04T09:22:22.614 INFO:teuthology.orchestra.run.smithi006.stdout: libquadmath x86_64 11.4.1-3.el9 CentOS-BaseOS 194 k 2024-03-04T09:22:22.614 INFO:teuthology.orchestra.run.smithi006.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-04T09:22:22.614 INFO:teuthology.orchestra.run.smithi006.stdout: libradosstriper1 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 501 k 2024-03-04T09:22:22.614 INFO:teuthology.orchestra.run.smithi006.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-04T09:22:22.614 INFO:teuthology.orchestra.run.smithi006.stdout: librgw2 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 5.1 M 2024-03-04T09:22:22.614 INFO:teuthology.orchestra.run.smithi006.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-04T09:22:22.614 INFO:teuthology.orchestra.run.smithi006.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-04T09:22:22.614 INFO:teuthology.orchestra.run.smithi006.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-04T09:22:22.614 INFO:teuthology.orchestra.run.smithi006.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-04T09:22:22.615 INFO:teuthology.orchestra.run.smithi006.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-04T09:22:22.615 INFO:teuthology.orchestra.run.smithi006.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-04T09:22:22.615 INFO:teuthology.orchestra.run.smithi006.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-BaseOS 217 k 2024-03-04T09:22:22.615 INFO:teuthology.orchestra.run.smithi006.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-04T09:22:22.615 INFO:teuthology.orchestra.run.smithi006.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-04T09:22:22.615 INFO:teuthology.orchestra.run.smithi006.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-04T09:22:22.615 INFO:teuthology.orchestra.run.smithi006.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-04T09:22:22.615 INFO:teuthology.orchestra.run.smithi006.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-04T09:22:22.615 INFO:teuthology.orchestra.run.smithi006.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-04T09:22:22.615 INFO:teuthology.orchestra.run.smithi006.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-04T09:22:22.615 INFO:teuthology.orchestra.run.smithi006.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-04T09:22:22.615 INFO:teuthology.orchestra.run.smithi006.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-04T09:22:22.615 INFO:teuthology.orchestra.run.smithi006.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-04T09:22:22.615 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-04T09:22:22.616 INFO:teuthology.orchestra.run.smithi006.stdout: python3-ceph-argparse x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 45 k 2024-03-04T09:22:22.616 INFO:teuthology.orchestra.run.smithi006.stdout: python3-ceph-common x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 131 k 2024-03-04T09:22:22.616 INFO:teuthology.orchestra.run.smithi006.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-04T09:22:22.616 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-04T09:22:22.616 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-04T09:22:22.616 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-04T09:22:22.616 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-04T09:22:22.616 INFO:teuthology.orchestra.run.smithi006.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-04T09:22:22.617 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-04T09:22:22.617 INFO:teuthology.orchestra.run.smithi006.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-04T09:22:22.617 INFO:teuthology.orchestra.run.smithi006.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-04T09:22:22.617 INFO:teuthology.orchestra.run.smithi006.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-04T09:22:22.617 INFO:teuthology.orchestra.run.smithi006.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-04T09:22:22.617 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-04T09:22:22.617 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-04T09:22:22.617 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-04T09:22:22.617 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-04T09:22:22.617 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-04T09:22:22.617 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jinja2 noarch 2.11.3-5.el9 CentOS-AppStream 227 k 2024-03-04T09:22:22.618 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-04T09:22:22.618 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-04T09:22:22.618 INFO:teuthology.orchestra.run.smithi006.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-04T09:22:22.618 INFO:teuthology.orchestra.run.smithi006.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-04T09:22:22.618 INFO:teuthology.orchestra.run.smithi006.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-04T09:22:22.618 INFO:teuthology.orchestra.run.smithi006.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-04T09:22:22.618 INFO:teuthology.orchestra.run.smithi006.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-04T09:22:22.618 INFO:teuthology.orchestra.run.smithi006.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-04T09:22:22.618 INFO:teuthology.orchestra.run.smithi006.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-04T09:22:22.618 INFO:teuthology.orchestra.run.smithi006.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-04T09:22:22.618 INFO:teuthology.orchestra.run.smithi006.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-04T09:22:22.618 INFO:teuthology.orchestra.run.smithi006.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-04T09:22:22.619 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-04T09:22:22.619 INFO:teuthology.orchestra.run.smithi006.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-04T09:22:22.619 INFO:teuthology.orchestra.run.smithi006.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-04T09:22:22.619 INFO:teuthology.orchestra.run.smithi006.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-04T09:22:22.619 INFO:teuthology.orchestra.run.smithi006.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-04T09:22:22.619 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-04T09:22:22.619 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-04T09:22:22.619 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-04T09:22:22.619 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-04T09:22:22.619 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-04T09:22:22.619 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-04T09:22:22.619 INFO:teuthology.orchestra.run.smithi006.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-04T09:22:22.619 INFO:teuthology.orchestra.run.smithi006.stdout: python3-requests noarch 2.25.1-8.el9 CentOS-BaseOS 113 k 2024-03-04T09:22:22.620 INFO:teuthology.orchestra.run.smithi006.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-04T09:22:22.620 INFO:teuthology.orchestra.run.smithi006.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-04T09:22:22.620 INFO:teuthology.orchestra.run.smithi006.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-04T09:22:22.620 INFO:teuthology.orchestra.run.smithi006.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-04T09:22:22.620 INFO:teuthology.orchestra.run.smithi006.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-04T09:22:22.620 INFO:teuthology.orchestra.run.smithi006.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-04T09:22:22.620 INFO:teuthology.orchestra.run.smithi006.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-04T09:22:22.620 INFO:teuthology.orchestra.run.smithi006.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-04T09:22:22.620 INFO:teuthology.orchestra.run.smithi006.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-04T09:22:22.620 INFO:teuthology.orchestra.run.smithi006.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-04T09:22:22.620 INFO:teuthology.orchestra.run.smithi006.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-04T09:22:22.620 INFO:teuthology.orchestra.run.smithi006.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-04T09:22:22.620 INFO:teuthology.orchestra.run.smithi006.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-04T09:22:22.621 INFO:teuthology.orchestra.run.smithi006.stdout: qatlib x86_64 23.11.0-1.el9 CentOS-AppStream 215 k 2024-03-04T09:22:22.621 INFO:teuthology.orchestra.run.smithi006.stdout: qatzip-libs x86_64 1.1.2-2.el9 CentOS-AppStream 43 k 2024-03-04T09:22:22.621 INFO:teuthology.orchestra.run.smithi006.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-04T09:22:22.621 INFO:teuthology.orchestra.run.smithi006.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-04T09:22:22.621 INFO:teuthology.orchestra.run.smithi006.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-04T09:22:22.621 INFO:teuthology.orchestra.run.smithi006.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-04T09:22:22.621 INFO:teuthology.orchestra.run.smithi006.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-04T09:22:22.621 INFO:teuthology.orchestra.run.smithi006.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-04T09:22:22.621 INFO:teuthology.orchestra.run.smithi006.stdout:Installing weak dependencies: 2024-03-04T09:22:22.621 INFO:teuthology.orchestra.run.smithi006.stdout: qatlib-service x86_64 23.11.0-1.el9 CentOS-AppStream 34 k 2024-03-04T09:22:22.621 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:22:22.621 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2024-03-04T09:22:22.622 INFO:teuthology.orchestra.run.smithi006.stdout:============================================================================================ 2024-03-04T09:22:22.622 INFO:teuthology.orchestra.run.smithi006.stdout:Install 150 Packages 2024-03-04T09:22:22.622 INFO:teuthology.orchestra.run.smithi006.stdout:Upgrade 26 Packages 2024-03-04T09:22:22.622 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:22:22.622 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 224 M 2024-03-04T09:22:22.622 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2024-03-04T09:22:23.916 INFO:teuthology.orchestra.run.smithi006.stdout:(1/176): ceph-19.0.0-1739.g1bc250c7.el9.x86_64. 13 kB/s | 6.6 kB 00:00 2024-03-04T09:22:25.800 INFO:teuthology.orchestra.run.smithi006.stdout:(2/176): ceph-base-19.0.0-1739.g1bc250c7.el9.x8 2.2 MB/s | 5.3 MB 00:02 2024-03-04T09:22:25.891 INFO:teuthology.orchestra.run.smithi006.stdout:(3/176): ceph-immutable-object-cache-19.0.0-173 1.6 MB/s | 145 kB 00:00 2024-03-04T09:22:26.108 INFO:teuthology.orchestra.run.smithi006.stdout:(4/176): ceph-fuse-19.0.0-1739.g1bc250c7.el9.x8 406 kB/s | 888 kB 00:02 2024-03-04T09:22:26.809 INFO:teuthology.orchestra.run.smithi146.stdout:lab-extras 38 kB/s | 1.7 kB 00:00 2024-03-04T09:22:26.817 INFO:teuthology.orchestra.run.smithi006.stdout:(5/176): ceph-mds-19.0.0-1739.g1bc250c7.el9.x86 2.5 MB/s | 2.3 MB 00:00 2024-03-04T09:22:28.242 INFO:teuthology.orchestra.run.smithi006.stdout:(6/176): ceph-mgr-19.0.0-1739.g1bc250c7.el9.x86 717 kB/s | 1.5 MB 00:02 2024-03-04T09:22:28.517 INFO:teuthology.orchestra.run.smithi006.stdout:(7/176): ceph-mon-19.0.0-1739.g1bc250c7.el9.x86 2.8 MB/s | 4.7 MB 00:01 2024-03-04T09:22:29.476 INFO:teuthology.orchestra.run.smithi006.stdout:(8/176): ceph-common-19.0.0-1739.g1bc250c7.el9. 3.4 MB/s | 20 MB 00:06 2024-03-04T09:22:29.576 INFO:teuthology.orchestra.run.smithi006.stdout:(9/176): ceph-selinux-19.0.0-1739.g1bc250c7.el9 254 kB/s | 25 kB 00:00 2024-03-04T09:22:29.593 INFO:teuthology.orchestra.run.smithi146.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-04T09:22:29.595 INFO:teuthology.orchestra.run.smithi146.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-04T09:22:29.619 INFO:teuthology.orchestra.run.smithi146.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-04T09:22:29.620 INFO:teuthology.orchestra.run.smithi146.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-04T09:22:29.622 INFO:teuthology.orchestra.run.smithi146.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-04T09:22:29.623 INFO:teuthology.orchestra.run.smithi146.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-04T09:22:29.626 INFO:teuthology.orchestra.run.smithi146.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-04T09:22:29.630 INFO:teuthology.orchestra.run.smithi146.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-04T09:22:29.631 INFO:teuthology.orchestra.run.smithi146.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-04T09:22:29.722 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-03-04T09:22:29.746 INFO:teuthology.orchestra.run.smithi146.stdout:============================================================================================ 2024-03-04T09:22:29.746 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repository Size 2024-03-04T09:22:29.746 INFO:teuthology.orchestra.run.smithi146.stdout:============================================================================================ 2024-03-04T09:22:29.746 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-03-04T09:22:29.746 INFO:teuthology.orchestra.run.smithi146.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-04T09:22:29.746 INFO:teuthology.orchestra.run.smithi146.stdout: ceph x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 6.6 k 2024-03-04T09:22:29.746 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-base x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 5.3 M 2024-03-04T09:22:29.747 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-fuse x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 888 k 2024-03-04T09:22:29.747 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 145 k 2024-03-04T09:22:29.747 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 1.5 M 2024-03-04T09:22:29.747 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 138 k 2024-03-04T09:22:29.747 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 1.8 M 2024-03-04T09:22:29.747 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 7.4 M 2024-03-04T09:22:29.747 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-rook noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 49 k 2024-03-04T09:22:29.747 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-radosgw x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 9.8 M 2024-03-04T09:22:29.747 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-test x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 47 M 2024-03-04T09:22:29.747 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-volume noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 264 k 2024-03-04T09:22:29.747 INFO:teuthology.orchestra.run.smithi146.stdout: cephadm noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 748 k 2024-03-04T09:22:29.747 INFO:teuthology.orchestra.run.smithi146.stdout: cephfs-mirror x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 220 k 2024-03-04T09:22:29.747 INFO:teuthology.orchestra.run.smithi146.stdout: cephfs-shell x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 27 k 2024-03-04T09:22:29.747 INFO:teuthology.orchestra.run.smithi146.stdout: cephfs-top noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 18 k 2024-03-04T09:22:29.747 INFO:teuthology.orchestra.run.smithi146.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 CentOS-AppStream 20 k 2024-03-04T09:22:29.748 INFO:teuthology.orchestra.run.smithi146.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-04T09:22:29.748 INFO:teuthology.orchestra.run.smithi146.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-04T09:22:29.748 INFO:teuthology.orchestra.run.smithi146.stdout: libcephfs-devel x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 31 k 2024-03-04T09:22:29.748 INFO:teuthology.orchestra.run.smithi146.stdout: libcephfs2 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 747 k 2024-03-04T09:22:29.748 INFO:teuthology.orchestra.run.smithi146.stdout: librados-devel x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 127 k 2024-03-04T09:22:29.748 INFO:teuthology.orchestra.run.smithi146.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-04T09:22:29.748 INFO:teuthology.orchestra.run.smithi146.stdout: libuuid-devel x86_64 2.37.4-18.el9 CentOS-AppStream 24 k 2024-03-04T09:22:29.748 INFO:teuthology.orchestra.run.smithi146.stdout: openssl-devel x86_64 1:3.0.7-27.el9 CentOS-AppStream 3.0 M 2024-03-04T09:22:29.748 INFO:teuthology.orchestra.run.smithi146.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-04T09:22:29.748 INFO:teuthology.orchestra.run.smithi146.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-04T09:22:29.748 INFO:teuthology.orchestra.run.smithi146.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-04T09:22:29.748 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cephfs x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 162 k 2024-03-04T09:22:29.748 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rados x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 321 k 2024-03-04T09:22:29.749 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rbd x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 298 k 2024-03-04T09:22:29.749 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rgw x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 99 k 2024-03-04T09:22:29.749 INFO:teuthology.orchestra.run.smithi146.stdout: rbd-fuse x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 85 k 2024-03-04T09:22:29.749 INFO:teuthology.orchestra.run.smithi146.stdout: rbd-mirror x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 3.1 M 2024-03-04T09:22:29.749 INFO:teuthology.orchestra.run.smithi146.stdout: rbd-nbd x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 171 k 2024-03-04T09:22:29.749 INFO:teuthology.orchestra.run.smithi146.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 CentOS-AppStream 87 k 2024-03-04T09:22:29.749 INFO:teuthology.orchestra.run.smithi146.stdout:Upgrading: 2024-03-04T09:22:29.749 INFO:teuthology.orchestra.run.smithi146.stdout: NetworkManager x86_64 1:1.46.0-1.el9 CentOS-BaseOS 2.3 M 2024-03-04T09:22:29.749 INFO:teuthology.orchestra.run.smithi146.stdout: NetworkManager-libnm x86_64 1:1.46.0-1.el9 CentOS-BaseOS 1.8 M 2024-03-04T09:22:29.749 INFO:teuthology.orchestra.run.smithi146.stdout: NetworkManager-team x86_64 1:1.46.0-1.el9 CentOS-BaseOS 41 k 2024-03-04T09:22:29.749 INFO:teuthology.orchestra.run.smithi146.stdout: NetworkManager-tui x86_64 1:1.46.0-1.el9 CentOS-BaseOS 247 k 2024-03-04T09:22:29.749 INFO:teuthology.orchestra.run.smithi146.stdout: acl x86_64 2.3.1-4.el9 CentOS-BaseOS 69 k 2024-03-04T09:22:29.749 INFO:teuthology.orchestra.run.smithi146.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 CentOS-BaseOS 35 k 2024-03-04T09:22:29.749 INFO:teuthology.orchestra.run.smithi146.stdout: elfutils-libelf x86_64 0.190-2.el9 CentOS-BaseOS 191 k 2024-03-04T09:22:29.750 INFO:teuthology.orchestra.run.smithi146.stdout: elfutils-libs x86_64 0.190-2.el9 CentOS-BaseOS 254 k 2024-03-04T09:22:29.750 INFO:teuthology.orchestra.run.smithi146.stdout: iproute x86_64 6.7.0-2.el9 CentOS-BaseOS 802 k 2024-03-04T09:22:29.750 INFO:teuthology.orchestra.run.smithi146.stdout: iproute-tc x86_64 6.7.0-2.el9 CentOS-BaseOS 433 k 2024-03-04T09:22:29.750 INFO:teuthology.orchestra.run.smithi146.stdout: libacl x86_64 2.3.1-4.el9 CentOS-BaseOS 22 k 2024-03-04T09:22:29.750 INFO:teuthology.orchestra.run.smithi146.stdout: libacl-devel x86_64 2.3.1-4.el9 CentOS-AppStream 68 k 2024-03-04T09:22:29.750 INFO:teuthology.orchestra.run.smithi146.stdout: libblkid x86_64 2.37.4-18.el9 CentOS-BaseOS 107 k 2024-03-04T09:22:29.750 INFO:teuthology.orchestra.run.smithi146.stdout: libfdisk x86_64 2.37.4-18.el9 CentOS-BaseOS 154 k 2024-03-04T09:22:29.750 INFO:teuthology.orchestra.run.smithi146.stdout: libmount x86_64 2.37.4-18.el9 CentOS-BaseOS 135 k 2024-03-04T09:22:29.750 INFO:teuthology.orchestra.run.smithi146.stdout: librados2 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 3.4 M 2024-03-04T09:22:29.750 INFO:teuthology.orchestra.run.smithi146.stdout: librbd1 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 3.1 M 2024-03-04T09:22:29.750 INFO:teuthology.orchestra.run.smithi146.stdout: libsmartcols x86_64 2.37.4-18.el9 CentOS-BaseOS 63 k 2024-03-04T09:22:29.750 INFO:teuthology.orchestra.run.smithi146.stdout: libuuid x86_64 2.37.4-18.el9 CentOS-BaseOS 28 k 2024-03-04T09:22:29.750 INFO:teuthology.orchestra.run.smithi146.stdout: openssl x86_64 1:3.0.7-27.el9 CentOS-BaseOS 1.2 M 2024-03-04T09:22:29.750 INFO:teuthology.orchestra.run.smithi146.stdout: openssl-libs x86_64 1:3.0.7-27.el9 CentOS-BaseOS 2.2 M 2024-03-04T09:22:29.751 INFO:teuthology.orchestra.run.smithi146.stdout: util-linux x86_64 2.37.4-18.el9 CentOS-BaseOS 2.2 M 2024-03-04T09:22:29.751 INFO:teuthology.orchestra.run.smithi146.stdout: util-linux-core x86_64 2.37.4-18.el9 CentOS-BaseOS 438 k 2024-03-04T09:22:29.751 INFO:teuthology.orchestra.run.smithi146.stdout: util-linux-user x86_64 2.37.4-18.el9 CentOS-BaseOS 30 k 2024-03-04T09:22:29.751 INFO:teuthology.orchestra.run.smithi146.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-04T09:22:29.751 INFO:teuthology.orchestra.run.smithi146.stdout: xfsprogs x86_64 6.3.0-1.el9 CentOS-BaseOS 1.1 M 2024-03-04T09:22:29.751 INFO:teuthology.orchestra.run.smithi146.stdout:Installing dependencies: 2024-03-04T09:22:29.751 INFO:teuthology.orchestra.run.smithi146.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-04T09:22:29.751 INFO:teuthology.orchestra.run.smithi146.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-04T09:22:29.751 INFO:teuthology.orchestra.run.smithi146.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-04T09:22:29.751 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-common x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 20 M 2024-03-04T09:22:29.751 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 31 k 2024-03-04T09:22:29.751 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mds x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 2.3 M 2024-03-04T09:22:29.751 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 248 k 2024-03-04T09:22:29.751 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mon x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 4.7 M 2024-03-04T09:22:29.752 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-osd x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 17 M 2024-03-04T09:22:29.752 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 15 k 2024-03-04T09:22:29.752 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-selinux x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 25 k 2024-03-04T09:22:29.752 INFO:teuthology.orchestra.run.smithi146.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-04T09:22:29.752 INFO:teuthology.orchestra.run.smithi146.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-04T09:22:29.752 INFO:teuthology.orchestra.run.smithi146.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-04T09:22:29.752 INFO:teuthology.orchestra.run.smithi146.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-04T09:22:29.752 INFO:teuthology.orchestra.run.smithi146.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-04T09:22:29.752 INFO:teuthology.orchestra.run.smithi146.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-04T09:22:29.752 INFO:teuthology.orchestra.run.smithi146.stdout: jq x86_64 1.6-16.el9 CentOS-BaseOS 186 k 2024-03-04T09:22:29.752 INFO:teuthology.orchestra.run.smithi146.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-04T09:22:29.752 INFO:teuthology.orchestra.run.smithi146.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-04T09:22:29.752 INFO:teuthology.orchestra.run.smithi146.stdout: libcephsqlite x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 163 k 2024-03-04T09:22:29.753 INFO:teuthology.orchestra.run.smithi146.stdout: libgfortran x86_64 11.4.1-3.el9 CentOS-BaseOS 802 k 2024-03-04T09:22:29.753 INFO:teuthology.orchestra.run.smithi146.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-04T09:22:29.753 INFO:teuthology.orchestra.run.smithi146.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-04T09:22:29.753 INFO:teuthology.orchestra.run.smithi146.stdout: libquadmath x86_64 11.4.1-3.el9 CentOS-BaseOS 194 k 2024-03-04T09:22:29.753 INFO:teuthology.orchestra.run.smithi146.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-04T09:22:29.753 INFO:teuthology.orchestra.run.smithi146.stdout: libradosstriper1 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 501 k 2024-03-04T09:22:29.753 INFO:teuthology.orchestra.run.smithi146.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-04T09:22:29.753 INFO:teuthology.orchestra.run.smithi146.stdout: librgw2 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 5.1 M 2024-03-04T09:22:29.753 INFO:teuthology.orchestra.run.smithi146.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-04T09:22:29.753 INFO:teuthology.orchestra.run.smithi146.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-04T09:22:29.753 INFO:teuthology.orchestra.run.smithi146.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-04T09:22:29.753 INFO:teuthology.orchestra.run.smithi146.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-04T09:22:29.753 INFO:teuthology.orchestra.run.smithi146.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-04T09:22:29.753 INFO:teuthology.orchestra.run.smithi146.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-04T09:22:29.754 INFO:teuthology.orchestra.run.smithi146.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-BaseOS 217 k 2024-03-04T09:22:29.754 INFO:teuthology.orchestra.run.smithi146.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-04T09:22:29.754 INFO:teuthology.orchestra.run.smithi146.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-04T09:22:29.754 INFO:teuthology.orchestra.run.smithi146.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-04T09:22:29.754 INFO:teuthology.orchestra.run.smithi146.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-04T09:22:29.754 INFO:teuthology.orchestra.run.smithi146.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-04T09:22:29.754 INFO:teuthology.orchestra.run.smithi146.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-04T09:22:29.754 INFO:teuthology.orchestra.run.smithi146.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-04T09:22:29.754 INFO:teuthology.orchestra.run.smithi146.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-04T09:22:29.754 INFO:teuthology.orchestra.run.smithi146.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-04T09:22:29.754 INFO:teuthology.orchestra.run.smithi146.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-04T09:22:29.754 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-04T09:22:29.755 INFO:teuthology.orchestra.run.smithi146.stdout: python3-ceph-argparse x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 45 k 2024-03-04T09:22:29.755 INFO:teuthology.orchestra.run.smithi146.stdout: python3-ceph-common x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 131 k 2024-03-04T09:22:29.755 INFO:teuthology.orchestra.run.smithi146.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-04T09:22:29.755 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-04T09:22:29.755 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-04T09:22:29.755 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-04T09:22:29.755 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-04T09:22:29.755 INFO:teuthology.orchestra.run.smithi146.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-04T09:22:29.755 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-04T09:22:29.755 INFO:teuthology.orchestra.run.smithi146.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-04T09:22:29.755 INFO:teuthology.orchestra.run.smithi146.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-04T09:22:29.755 INFO:teuthology.orchestra.run.smithi146.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-04T09:22:29.755 INFO:teuthology.orchestra.run.smithi146.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-04T09:22:29.756 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-04T09:22:29.756 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-04T09:22:29.756 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-04T09:22:29.756 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-04T09:22:29.756 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-04T09:22:29.756 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jinja2 noarch 2.11.3-5.el9 CentOS-AppStream 227 k 2024-03-04T09:22:29.756 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-04T09:22:29.756 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-04T09:22:29.756 INFO:teuthology.orchestra.run.smithi146.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-04T09:22:29.756 INFO:teuthology.orchestra.run.smithi146.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-04T09:22:29.756 INFO:teuthology.orchestra.run.smithi146.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-04T09:22:29.756 INFO:teuthology.orchestra.run.smithi146.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-04T09:22:29.756 INFO:teuthology.orchestra.run.smithi146.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-04T09:22:29.757 INFO:teuthology.orchestra.run.smithi146.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-04T09:22:29.757 INFO:teuthology.orchestra.run.smithi146.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-04T09:22:29.757 INFO:teuthology.orchestra.run.smithi146.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-04T09:22:29.757 INFO:teuthology.orchestra.run.smithi146.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-04T09:22:29.757 INFO:teuthology.orchestra.run.smithi146.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-04T09:22:29.757 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-04T09:22:29.757 INFO:teuthology.orchestra.run.smithi146.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-04T09:22:29.757 INFO:teuthology.orchestra.run.smithi146.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-04T09:22:29.757 INFO:teuthology.orchestra.run.smithi146.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-04T09:22:29.757 INFO:teuthology.orchestra.run.smithi146.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-04T09:22:29.757 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-04T09:22:29.757 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-04T09:22:29.757 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-04T09:22:29.758 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-04T09:22:29.758 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-04T09:22:29.758 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-04T09:22:29.758 INFO:teuthology.orchestra.run.smithi146.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-04T09:22:29.758 INFO:teuthology.orchestra.run.smithi146.stdout: python3-requests noarch 2.25.1-8.el9 CentOS-BaseOS 113 k 2024-03-04T09:22:29.758 INFO:teuthology.orchestra.run.smithi146.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-04T09:22:29.758 INFO:teuthology.orchestra.run.smithi146.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-04T09:22:29.758 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-04T09:22:29.758 INFO:teuthology.orchestra.run.smithi146.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-04T09:22:29.758 INFO:teuthology.orchestra.run.smithi146.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-04T09:22:29.758 INFO:teuthology.orchestra.run.smithi146.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-04T09:22:29.758 INFO:teuthology.orchestra.run.smithi146.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-04T09:22:29.758 INFO:teuthology.orchestra.run.smithi146.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-04T09:22:29.758 INFO:teuthology.orchestra.run.smithi146.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-04T09:22:29.759 INFO:teuthology.orchestra.run.smithi146.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-04T09:22:29.759 INFO:teuthology.orchestra.run.smithi146.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-04T09:22:29.759 INFO:teuthology.orchestra.run.smithi146.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-04T09:22:29.759 INFO:teuthology.orchestra.run.smithi146.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-04T09:22:29.759 INFO:teuthology.orchestra.run.smithi146.stdout: qatlib x86_64 23.11.0-1.el9 CentOS-AppStream 215 k 2024-03-04T09:22:29.759 INFO:teuthology.orchestra.run.smithi146.stdout: qatzip-libs x86_64 1.1.2-2.el9 CentOS-AppStream 43 k 2024-03-04T09:22:29.759 INFO:teuthology.orchestra.run.smithi146.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-04T09:22:29.759 INFO:teuthology.orchestra.run.smithi146.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-04T09:22:29.759 INFO:teuthology.orchestra.run.smithi146.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-04T09:22:29.759 INFO:teuthology.orchestra.run.smithi146.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-04T09:22:29.759 INFO:teuthology.orchestra.run.smithi146.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-04T09:22:29.759 INFO:teuthology.orchestra.run.smithi146.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-04T09:22:29.760 INFO:teuthology.orchestra.run.smithi146.stdout:Installing weak dependencies: 2024-03-04T09:22:29.760 INFO:teuthology.orchestra.run.smithi146.stdout: qatlib-service x86_64 23.11.0-1.el9 CentOS-AppStream 34 k 2024-03-04T09:22:29.760 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:22:29.760 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-03-04T09:22:29.760 INFO:teuthology.orchestra.run.smithi146.stdout:============================================================================================ 2024-03-04T09:22:29.760 INFO:teuthology.orchestra.run.smithi146.stdout:Install 150 Packages 2024-03-04T09:22:29.760 INFO:teuthology.orchestra.run.smithi146.stdout:Upgrade 26 Packages 2024-03-04T09:22:29.760 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:22:29.761 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 224 M 2024-03-04T09:22:29.761 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-03-04T09:22:30.827 INFO:teuthology.orchestra.run.smithi146.stdout:(1/176): ceph-19.0.0-1739.g1bc250c7.el9.x86_64. 19 kB/s | 6.6 kB 00:00 2024-03-04T09:22:31.831 INFO:teuthology.orchestra.run.smithi146.stdout:(2/176): ceph-fuse-19.0.0-1739.g1bc250c7.el9.x8 886 kB/s | 888 kB 00:01 2024-03-04T09:22:31.986 INFO:teuthology.orchestra.run.smithi146.stdout:(3/176): ceph-immutable-object-cache-19.0.0-173 945 kB/s | 145 kB 00:00 2024-03-04T09:22:32.010 INFO:teuthology.orchestra.run.smithi006.stdout:(10/176): ceph-radosgw-19.0.0-1739.g1bc250c7.el 2.8 MB/s | 9.8 MB 00:03 2024-03-04T09:22:32.185 INFO:teuthology.orchestra.run.smithi006.stdout:(11/176): cephfs-mirror-19.0.0-1739.g1bc250c7.e 1.2 MB/s | 220 kB 00:00 2024-03-04T09:22:32.293 INFO:teuthology.orchestra.run.smithi006.stdout:(12/176): cephfs-shell-19.0.0-1739.g1bc250c7.el 251 kB/s | 27 kB 00:00 2024-03-04T09:22:32.403 INFO:teuthology.orchestra.run.smithi006.stdout:(13/176): libcephfs-devel-19.0.0-1739.g1bc250c7 293 kB/s | 31 kB 00:00 2024-03-04T09:22:32.702 INFO:teuthology.orchestra.run.smithi006.stdout:(14/176): libcephfs2-19.0.0-1739.g1bc250c7.el9. 2.4 MB/s | 747 kB 00:00 2024-03-04T09:22:32.802 INFO:teuthology.orchestra.run.smithi006.stdout:(15/176): libcephsqlite-19.0.0-1739.g1bc250c7.e 1.6 MB/s | 163 kB 00:00 2024-03-04T09:22:32.911 INFO:teuthology.orchestra.run.smithi006.stdout:(16/176): librados-devel-19.0.0-1739.g1bc250c7. 1.1 MB/s | 127 kB 00:00 2024-03-04T09:22:33.010 INFO:teuthology.orchestra.run.smithi146.stdout:(4/176): ceph-mds-19.0.0-1739.g1bc250c7.el9.x86 2.3 MB/s | 2.3 MB 00:01 2024-03-04T09:22:33.153 INFO:teuthology.orchestra.run.smithi006.stdout:(17/176): libradosstriper1-19.0.0-1739.g1bc250c 2.0 MB/s | 501 kB 00:00 2024-03-04T09:22:33.461 INFO:teuthology.orchestra.run.smithi146.stdout:(5/176): ceph-base-19.0.0-1739.g1bc250c7.el9.x8 1.8 MB/s | 5.3 MB 00:02 2024-03-04T09:22:33.578 INFO:teuthology.orchestra.run.smithi146.stdout:(6/176): ceph-mgr-19.0.0-1739.g1bc250c7.el9.x86 2.6 MB/s | 1.5 MB 00:00 2024-03-04T09:22:34.429 INFO:teuthology.orchestra.run.smithi146.stdout:(7/176): ceph-mon-19.0.0-1739.g1bc250c7.el9.x86 4.9 MB/s | 4.7 MB 00:00 2024-03-04T09:22:35.287 INFO:teuthology.orchestra.run.smithi006.stdout:(18/176): librgw2-19.0.0-1739.g1bc250c7.el9.x86 2.4 MB/s | 5.1 MB 00:02 2024-03-04T09:22:35.379 INFO:teuthology.orchestra.run.smithi006.stdout:(19/176): python3-ceph-argparse-19.0.0-1739.g1b 497 kB/s | 45 kB 00:00 2024-03-04T09:22:35.487 INFO:teuthology.orchestra.run.smithi006.stdout:(20/176): python3-ceph-common-19.0.0-1739.g1bc2 1.2 MB/s | 131 kB 00:00 2024-03-04T09:22:35.588 INFO:teuthology.orchestra.run.smithi006.stdout:(21/176): python3-cephfs-19.0.0-1739.g1bc250c7. 1.6 MB/s | 162 kB 00:00 2024-03-04T09:22:35.788 INFO:teuthology.orchestra.run.smithi006.stdout:(22/176): python3-rados-19.0.0-1739.g1bc250c7.e 1.6 MB/s | 321 kB 00:00 2024-03-04T09:22:35.963 INFO:teuthology.orchestra.run.smithi006.stdout:(23/176): python3-rbd-19.0.0-1739.g1bc250c7.el9 1.7 MB/s | 298 kB 00:00 2024-03-04T09:22:36.055 INFO:teuthology.orchestra.run.smithi006.stdout:(24/176): python3-rgw-19.0.0-1739.g1bc250c7.el9 1.1 MB/s | 99 kB 00:00 2024-03-04T09:22:36.147 INFO:teuthology.orchestra.run.smithi006.stdout:(25/176): rbd-fuse-19.0.0-1739.g1bc250c7.el9.x8 939 kB/s | 85 kB 00:00 2024-03-04T09:22:36.529 INFO:teuthology.orchestra.run.smithi146.stdout:(8/176): ceph-radosgw-19.0.0-1739.g1bc250c7.el9 4.7 MB/s | 9.8 MB 00:02 2024-03-04T09:22:36.637 INFO:teuthology.orchestra.run.smithi146.stdout:(9/176): ceph-selinux-19.0.0-1739.g1bc250c7.el9 234 kB/s | 25 kB 00:00 2024-03-04T09:22:36.904 INFO:teuthology.orchestra.run.smithi146.stdout:(10/176): ceph-common-19.0.0-1739.g1bc250c7.el9 3.2 MB/s | 20 MB 00:06 2024-03-04T09:22:37.021 INFO:teuthology.orchestra.run.smithi146.stdout:(11/176): cephfs-mirror-19.0.0-1739.g1bc250c7.e 1.9 MB/s | 220 kB 00:00 2024-03-04T09:22:37.139 INFO:teuthology.orchestra.run.smithi006.stdout:(26/176): rbd-mirror-19.0.0-1739.g1bc250c7.el9. 3.1 MB/s | 3.1 MB 00:00 2024-03-04T09:22:37.146 INFO:teuthology.orchestra.run.smithi146.stdout:(12/176): cephfs-shell-19.0.0-1739.g1bc250c7.el 217 kB/s | 27 kB 00:00 2024-03-04T09:22:37.214 INFO:teuthology.orchestra.run.smithi006.stdout:(27/176): rbd-nbd-19.0.0-1739.g1bc250c7.el9.x86 2.3 MB/s | 171 kB 00:00 2024-03-04T09:22:37.271 INFO:teuthology.orchestra.run.smithi146.stdout:(13/176): libcephfs-devel-19.0.0-1739.g1bc250c7 252 kB/s | 31 kB 00:00 2024-03-04T09:22:37.298 INFO:teuthology.orchestra.run.smithi006.stdout:(28/176): ceph-grafana-dashboards-19.0.0-1739.g 381 kB/s | 31 kB 00:00 2024-03-04T09:22:37.390 INFO:teuthology.orchestra.run.smithi006.stdout:(29/176): ceph-mgr-cephadm-19.0.0-1739.g1bc250c 1.5 MB/s | 138 kB 00:00 2024-03-04T09:22:37.480 INFO:teuthology.orchestra.run.smithi146.stdout:(14/176): libcephfs2-19.0.0-1739.g1bc250c7.el9. 3.5 MB/s | 747 kB 00:00 2024-03-04T09:22:37.588 INFO:teuthology.orchestra.run.smithi146.stdout:(15/176): libcephsqlite-19.0.0-1739.g1bc250c7.e 1.5 MB/s | 163 kB 00:00 2024-03-04T09:22:37.689 INFO:teuthology.orchestra.run.smithi146.stdout:(16/176): librados-devel-19.0.0-1739.g1bc250c7. 1.2 MB/s | 127 kB 00:00 2024-03-04T09:22:37.814 INFO:teuthology.orchestra.run.smithi146.stdout:(17/176): libradosstriper1-19.0.0-1739.g1bc250c 3.9 MB/s | 501 kB 00:00 2024-03-04T09:22:38.040 INFO:teuthology.orchestra.run.smithi006.stdout:(30/176): ceph-mgr-dashboard-19.0.0-1739.g1bc25 2.7 MB/s | 1.8 MB 00:00 2024-03-04T09:22:38.647 INFO:teuthology.orchestra.run.smithi146.stdout:(18/176): librgw2-19.0.0-1739.g1bc250c7.el9.x86 6.2 MB/s | 5.1 MB 00:00 2024-03-04T09:22:38.748 INFO:teuthology.orchestra.run.smithi146.stdout:(19/176): python3-ceph-argparse-19.0.0-1739.g1b 454 kB/s | 45 kB 00:00 2024-03-04T09:22:38.856 INFO:teuthology.orchestra.run.smithi146.stdout:(20/176): python3-ceph-common-19.0.0-1739.g1bc2 1.2 MB/s | 131 kB 00:00 2024-03-04T09:22:38.956 INFO:teuthology.orchestra.run.smithi146.stdout:(21/176): python3-cephfs-19.0.0-1739.g1bc250c7. 1.6 MB/s | 162 kB 00:00 2024-03-04T09:22:39.065 INFO:teuthology.orchestra.run.smithi146.stdout:(22/176): python3-rados-19.0.0-1739.g1bc250c7.e 2.9 MB/s | 321 kB 00:00 2024-03-04T09:22:39.174 INFO:teuthology.orchestra.run.smithi146.stdout:(23/176): python3-rbd-19.0.0-1739.g1bc250c7.el9 2.7 MB/s | 298 kB 00:00 2024-03-04T09:22:39.282 INFO:teuthology.orchestra.run.smithi146.stdout:(24/176): python3-rgw-19.0.0-1739.g1bc250c7.el9 923 kB/s | 99 kB 00:00 2024-03-04T09:22:39.383 INFO:teuthology.orchestra.run.smithi146.stdout:(25/176): rbd-fuse-19.0.0-1739.g1bc250c7.el9.x8 858 kB/s | 85 kB 00:00 2024-03-04T09:22:39.683 INFO:teuthology.orchestra.run.smithi146.stdout:(26/176): ceph-osd-19.0.0-1739.g1bc250c7.el9.x8 2.8 MB/s | 17 MB 00:06 2024-03-04T09:22:39.866 INFO:teuthology.orchestra.run.smithi146.stdout:(27/176): rbd-nbd-19.0.0-1739.g1bc250c7.el9.x86 936 kB/s | 171 kB 00:00 2024-03-04T09:22:39.942 INFO:teuthology.orchestra.run.smithi146.stdout:(28/176): rbd-mirror-19.0.0-1739.g1bc250c7.el9. 5.5 MB/s | 3.1 MB 00:00 2024-03-04T09:22:39.958 INFO:teuthology.orchestra.run.smithi146.stdout:(29/176): ceph-grafana-dashboards-19.0.0-1739.g 345 kB/s | 31 kB 00:00 2024-03-04T09:22:40.042 INFO:teuthology.orchestra.run.smithi146.stdout:(30/176): ceph-mgr-cephadm-19.0.0-1739.g1bc250c 1.4 MB/s | 138 kB 00:00 2024-03-04T09:22:40.232 INFO:teuthology.orchestra.run.smithi006.stdout:(31/176): ceph-mgr-diskprediction-local-19.0.0- 3.4 MB/s | 7.4 MB 00:02 2024-03-04T09:22:40.449 INFO:teuthology.orchestra.run.smithi006.stdout:(32/176): ceph-mgr-modules-core-19.0.0-1739.g1b 1.1 MB/s | 248 kB 00:00 2024-03-04T09:22:40.532 INFO:teuthology.orchestra.run.smithi006.stdout:(33/176): ceph-mgr-rook-19.0.0-1739.g1bc250c7.e 599 kB/s | 49 kB 00:00 2024-03-04T09:22:40.624 INFO:teuthology.orchestra.run.smithi006.stdout:(34/176): ceph-prometheus-alerts-19.0.0-1739.g1 168 kB/s | 15 kB 00:00 2024-03-04T09:22:40.791 INFO:teuthology.orchestra.run.smithi006.stdout:(35/176): ceph-volume-19.0.0-1739.g1bc250c7.el9 1.6 MB/s | 264 kB 00:00 2024-03-04T09:22:40.892 INFO:teuthology.orchestra.run.smithi146.stdout:(31/176): ceph-mgr-dashboard-19.0.0-1739.g1bc25 1.9 MB/s | 1.8 MB 00:00 2024-03-04T09:22:41.125 INFO:teuthology.orchestra.run.smithi006.stdout:(36/176): cephadm-19.0.0-1739.g1bc250c7.el9.noa 2.2 MB/s | 748 kB 00:00 2024-03-04T09:22:41.167 INFO:teuthology.orchestra.run.smithi146.stdout:(32/176): ceph-mgr-diskprediction-local-19.0.0- 6.6 MB/s | 7.4 MB 00:01 2024-03-04T09:22:41.192 INFO:teuthology.orchestra.run.smithi006.stdout:(37/176): cephfs-top-19.0.0-1739.g1bc250c7.el9. 277 kB/s | 18 kB 00:00 2024-03-04T09:22:41.193 INFO:teuthology.orchestra.run.smithi146.stdout:(33/176): ceph-mgr-modules-core-19.0.0-1739.g1b 830 kB/s | 248 kB 00:00 2024-03-04T09:22:41.226 INFO:teuthology.orchestra.run.smithi146.stdout:(34/176): ceph-mgr-rook-19.0.0-1739.g1bc250c7.e 856 kB/s | 49 kB 00:00 2024-03-04T09:22:41.277 INFO:teuthology.orchestra.run.smithi146.stdout:(35/176): ceph-prometheus-alerts-19.0.0-1739.g1 184 kB/s | 15 kB 00:00 2024-03-04T09:22:41.392 INFO:teuthology.orchestra.run.smithi006.stdout:(38/176): automake-1.16.2-8.el9.noarch.rpm 3.3 MB/s | 662 kB 00:00 2024-03-04T09:22:41.419 INFO:teuthology.orchestra.run.smithi146.stdout:(36/176): ceph-volume-19.0.0-1739.g1bc250c7.el9 1.3 MB/s | 264 kB 00:00 2024-03-04T09:22:41.452 INFO:teuthology.orchestra.run.smithi146.stdout:(37/176): cephadm-19.0.0-1739.g1bc250c7.el9.noa 4.2 MB/s | 748 kB 00:00 2024-03-04T09:22:41.467 INFO:teuthology.orchestra.run.smithi006.stdout:(39/176): bison-3.7.4-5.el9.x86_64.rpm 12 MB/s | 920 kB 00:00 2024-03-04T09:22:41.478 INFO:teuthology.orchestra.run.smithi146.stdout:(38/176): cephfs-top-19.0.0-1739.g1bc250c7.el9. 316 kB/s | 18 kB 00:00 2024-03-04T09:22:41.519 INFO:teuthology.orchestra.run.smithi006.stdout:(40/176): boost-program-options-1.75.0-8.el9.x8 2.1 MB/s | 106 kB 00:00 2024-03-04T09:22:41.559 INFO:teuthology.orchestra.run.smithi006.stdout:(41/176): elfutils-libelf-devel-0.190-2.el9.x86 516 kB/s | 20 kB 00:00 2024-03-04T09:22:41.602 INFO:teuthology.orchestra.run.smithi006.stdout:(42/176): flex-2.6.4-9.el9.x86_64.rpm 7.4 MB/s | 307 kB 00:00 2024-03-04T09:22:41.643 INFO:teuthology.orchestra.run.smithi006.stdout:(43/176): flexiblas-3.0.4-8.el9.x86_64.rpm 777 kB/s | 32 kB 00:00 2024-03-04T09:22:41.720 INFO:teuthology.orchestra.run.smithi146.stdout:(39/176): automake-1.16.2-8.el9.noarch.rpm 2.4 MB/s | 662 kB 00:00 2024-03-04T09:22:41.769 INFO:teuthology.orchestra.run.smithi006.stdout:(44/176): flexiblas-netlib-3.0.4-8.el9.x86_64.r 24 MB/s | 3.0 MB 00:00 2024-03-04T09:22:41.786 INFO:teuthology.orchestra.run.smithi146.stdout:(40/176): bison-3.7.4-5.el9.x86_64.rpm 3.7 MB/s | 920 kB 00:00 2024-03-04T09:22:41.803 INFO:teuthology.orchestra.run.smithi006.stdout:(45/176): flexiblas-openblas-openmp-3.0.4-8.el9 530 kB/s | 17 kB 00:00 2024-03-04T09:22:41.812 INFO:teuthology.orchestra.run.smithi146.stdout:(41/176): boost-program-options-1.75.0-8.el9.x8 1.2 MB/s | 106 kB 00:00 2024-03-04T09:22:41.836 INFO:teuthology.orchestra.run.smithi006.stdout:(46/176): libaio-devel-0.3.111-13.el9.x86_64.rp 311 kB/s | 10 kB 00:00 2024-03-04T09:22:41.837 INFO:teuthology.orchestra.run.smithi146.stdout:(42/176): elfutils-libelf-devel-0.190-2.el9.x86 413 kB/s | 20 kB 00:00 2024-03-04T09:22:41.871 INFO:teuthology.orchestra.run.smithi146.stdout:(43/176): flex-2.6.4-9.el9.x86_64.rpm 5.2 MB/s | 307 kB 00:00 2024-03-04T09:22:41.878 INFO:teuthology.orchestra.run.smithi006.stdout:(47/176): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.8 MB/s | 159 kB 00:00 2024-03-04T09:22:41.896 INFO:teuthology.orchestra.run.smithi146.stdout:(44/176): flexiblas-3.0.4-8.el9.x86_64.rpm 573 kB/s | 32 kB 00:00 2024-03-04T09:22:41.920 INFO:teuthology.orchestra.run.smithi006.stdout:(48/176): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.1 MB/s | 44 kB 00:00 2024-03-04T09:22:41.979 INFO:teuthology.orchestra.run.smithi006.stdout:(49/176): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 661 kB 00:00 2024-03-04T09:22:42.004 INFO:teuthology.orchestra.run.smithi146.stdout:(45/176): flexiblas-netlib-3.0.4-8.el9.x86_64.r 23 MB/s | 3.0 MB 00:00 2024-03-04T09:22:42.029 INFO:teuthology.orchestra.run.smithi006.stdout:(50/176): libtool-2.4.6-45.el9.x86_64.rpm 12 MB/s | 578 kB 00:00 2024-03-04T09:22:42.030 INFO:teuthology.orchestra.run.smithi146.stdout:(46/176): flexiblas-openblas-openmp-3.0.4-8.el9 133 kB/s | 17 kB 00:00 2024-03-04T09:22:42.063 INFO:teuthology.orchestra.run.smithi006.stdout:(51/176): libuuid-devel-2.37.4-18.el9.x86_64.rp 753 kB/s | 24 kB 00:00 2024-03-04T09:22:42.072 INFO:teuthology.orchestra.run.smithi146.stdout:(47/176): libaio-devel-0.3.111-13.el9.x86_64.rp 150 kB/s | 10 kB 00:00 2024-03-04T09:22:42.096 INFO:teuthology.orchestra.run.smithi006.stdout:(52/176): libzstd-devel-1.5.1-2.el9.x86_64.rpm 1.4 MB/s | 46 kB 00:00 2024-03-04T09:22:42.105 INFO:teuthology.orchestra.run.smithi146.stdout:(48/176): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.1 MB/s | 159 kB 00:00 2024-03-04T09:22:42.131 INFO:teuthology.orchestra.run.smithi146.stdout:(49/176): librabbitmq-0.11.0-7.el9.x86_64.rpm 794 kB/s | 44 kB 00:00 2024-03-04T09:22:42.138 INFO:teuthology.orchestra.run.smithi006.stdout:(53/176): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.8 MB/s | 282 kB 00:00 2024-03-04T09:22:42.172 INFO:teuthology.orchestra.run.smithi146.stdout:(50/176): librdkafka-1.6.1-102.el9.x86_64.rpm 9.8 MB/s | 661 kB 00:00 2024-03-04T09:22:42.214 INFO:teuthology.orchestra.run.smithi146.stdout:(51/176): libtool-2.4.6-45.el9.x86_64.rpm 7.3 MB/s | 578 kB 00:00 2024-03-04T09:22:42.232 INFO:teuthology.orchestra.run.smithi146.stdout:(52/176): libuuid-devel-2.37.4-18.el9.x86_64.rp 460 kB/s | 24 kB 00:00 2024-03-04T09:22:42.248 INFO:teuthology.orchestra.run.smithi146.stdout:(53/176): libzstd-devel-1.5.1-2.el9.x86_64.rpm 1.4 MB/s | 46 kB 00:00 2024-03-04T09:22:42.282 INFO:teuthology.orchestra.run.smithi146.stdout:(54/176): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.6 MB/s | 282 kB 00:00 2024-03-04T09:22:42.297 INFO:teuthology.orchestra.run.smithi006.stdout:(54/176): lua-5.4.4-4.el9.x86_64.rpm 1.2 MB/s | 187 kB 00:00 2024-03-04T09:22:42.316 INFO:teuthology.orchestra.run.smithi146.stdout:(55/176): lua-5.4.4-4.el9.x86_64.rpm 2.9 MB/s | 187 kB 00:00 2024-03-04T09:22:42.331 INFO:teuthology.orchestra.run.smithi006.stdout:(55/176): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2024-03-04T09:22:42.333 INFO:teuthology.orchestra.run.smithi146.stdout:(56/176): openblas-0.3.21-2.el9.x86_64.rpm 716 kB/s | 34 kB 00:00 2024-03-04T09:22:42.475 INFO:teuthology.orchestra.run.smithi146.stdout:(57/176): openblas-openmp-0.3.21-2.el9.x86_64.r 30 MB/s | 4.7 MB 00:00 2024-03-04T09:22:42.573 INFO:teuthology.orchestra.run.smithi006.stdout:(56/176): openblas-openmp-0.3.21-2.el9.x86_64.r 20 MB/s | 4.7 MB 00:00 2024-03-04T09:22:42.591 INFO:teuthology.orchestra.run.smithi146.stdout:(58/176): openssl-devel-3.0.7-27.el9.x86_64.rpm 12 MB/s | 3.0 MB 00:00 2024-03-04T09:22:42.650 INFO:teuthology.orchestra.run.smithi146.stdout:(59/176): postgresql-13.11-1.el9.x86_64.rpm 9.5 MB/s | 1.5 MB 00:00 2024-03-04T09:22:42.689 INFO:teuthology.orchestra.run.smithi006.stdout:(57/176): openssl-devel-3.0.7-27.el9.x86_64.rpm 26 MB/s | 3.0 MB 00:00 2024-03-04T09:22:42.700 INFO:teuthology.orchestra.run.smithi146.stdout:(60/176): postgresql-contrib-13.11-1.el9.x86_64 8.2 MB/s | 811 kB 00:00 2024-03-04T09:22:42.765 INFO:teuthology.orchestra.run.smithi006.stdout:(58/176): postgresql-13.11-1.el9.x86_64.rpm 21 MB/s | 1.5 MB 00:00 2024-03-04T09:22:42.775 INFO:teuthology.orchestra.run.smithi146.stdout:(61/176): postgresql-private-libs-13.11-1.el9.x 1.1 MB/s | 134 kB 00:00 2024-03-04T09:22:42.915 INFO:teuthology.orchestra.run.smithi006.stdout:(59/176): postgresql-contrib-13.11-1.el9.x86_64 5.3 MB/s | 811 kB 00:00 2024-03-04T09:22:42.917 INFO:teuthology.orchestra.run.smithi146.stdout:(62/176): postgresql-server-13.11-1.el9.x86_64. 26 MB/s | 5.7 MB 00:00 2024-03-04T09:22:42.957 INFO:teuthology.orchestra.run.smithi006.stdout:(60/176): postgresql-private-libs-13.11-1.el9.x 3.2 MB/s | 134 kB 00:00 2024-03-04T09:22:42.959 INFO:teuthology.orchestra.run.smithi146.stdout:(63/176): protobuf-3.14.0-13.el9.x86_64.rpm 5.9 MB/s | 1.0 MB 00:00 2024-03-04T09:22:43.018 INFO:teuthology.orchestra.run.smithi146.stdout:(64/176): python3-attrs-20.3.0-7.el9.noarch.rpm 1.3 MB/s | 78 kB 00:00 2024-03-04T09:22:43.202 INFO:teuthology.orchestra.run.smithi146.stdout:(65/176): python3-babel-2.9.1-2.el9.noarch.rpm 24 MB/s | 5.8 MB 00:00 2024-03-04T09:22:43.207 INFO:teuthology.orchestra.run.smithi006.stdout:(61/176): postgresql-server-13.11-1.el9.x86_64. 23 MB/s | 5.7 MB 00:00 2024-03-04T09:22:43.227 INFO:teuthology.orchestra.run.smithi146.stdout:(66/176): python3-jinja2-2.11.3-5.el9.noarch.rp 1.1 MB/s | 227 kB 00:00 2024-03-04T09:22:43.261 INFO:teuthology.orchestra.run.smithi146.stdout:(67/176): python3-jsonpatch-1.21-16.el9.noarch. 756 kB/s | 24 kB 00:00 2024-03-04T09:22:43.274 INFO:teuthology.orchestra.run.smithi006.stdout:(62/176): protobuf-3.14.0-13.el9.x86_64.rpm 15 MB/s | 1.0 MB 00:00 2024-03-04T09:22:43.286 INFO:teuthology.orchestra.run.smithi146.stdout:(68/176): python3-jsonpointer-2.0-4.el9.noarch. 315 kB/s | 18 kB 00:00 2024-03-04T09:22:43.311 INFO:teuthology.orchestra.run.smithi146.stdout:(69/176): python3-mako-1.1.4-6.el9.noarch.rpm 2.9 MB/s | 144 kB 00:00 2024-03-04T09:22:43.316 INFO:teuthology.orchestra.run.smithi006.stdout:(63/176): python3-attrs-20.3.0-7.el9.noarch.rpm 1.9 MB/s | 78 kB 00:00 2024-03-04T09:22:43.336 INFO:teuthology.orchestra.run.smithi146.stdout:(70/176): python3-markupsafe-1.1.1-12.el9.x86_6 658 kB/s | 32 kB 00:00 2024-03-04T09:22:43.545 INFO:teuthology.orchestra.run.smithi146.stdout:(71/176): python3-numpy-1.20.1-5.el9.x86_64.rpm 21 MB/s | 4.8 MB 00:00 2024-03-04T09:22:43.550 INFO:teuthology.orchestra.run.smithi006.stdout:(64/176): python3-babel-2.9.1-2.el9.noarch.rpm 25 MB/s | 5.8 MB 00:00 2024-03-04T09:22:43.570 INFO:teuthology.orchestra.run.smithi146.stdout:(72/176): python3-numpy-f2py-1.20.1-5.el9.x86_6 965 kB/s | 217 kB 00:00 2024-03-04T09:22:43.620 INFO:teuthology.orchestra.run.smithi146.stdout:(73/176): python3-oauthlib-3.1.1-5.el9.noarch.r 3.5 MB/s | 174 kB 00:00 2024-03-04T09:22:43.645 INFO:teuthology.orchestra.run.smithi146.stdout:(74/176): python3-packaging-20.9-5.el9.noarch.r 935 kB/s | 69 kB 00:00 2024-03-04T09:22:43.687 INFO:teuthology.orchestra.run.smithi146.stdout:(75/176): python3-prettytable-0.7.2-27.el9.noar 1.0 MB/s | 41 kB 00:00 2024-03-04T09:22:43.721 INFO:teuthology.orchestra.run.smithi146.stdout:(76/176): python3-protobuf-3.14.0-13.el9.noarch 3.1 MB/s | 238 kB 00:00 2024-03-04T09:22:43.746 INFO:teuthology.orchestra.run.smithi146.stdout:(77/176): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.3 MB/s | 132 kB 00:00 2024-03-04T09:22:43.780 INFO:teuthology.orchestra.run.smithi146.stdout:(78/176): python3-pyasn1-modules-0.4.8-6.el9.no 3.7 MB/s | 211 kB 00:00 2024-03-04T09:22:43.805 INFO:teuthology.orchestra.run.smithi146.stdout:(79/176): python3-pytz-2021.1-5.el9.noarch.rpm 847 kB/s | 47 kB 00:00 2024-03-04T09:22:43.830 INFO:teuthology.orchestra.run.smithi146.stdout:(80/176): python3-requests-oauthlib-1.3.0-12.el 869 kB/s | 43 kB 00:00 2024-03-04T09:22:43.900 INFO:teuthology.orchestra.run.smithi006.stdout:(65/176): python3-jinja2-2.11.3-5.el9.noarch.rp 650 kB/s | 227 kB 00:00 2024-03-04T09:22:44.056 INFO:teuthology.orchestra.run.smithi146.stdout:(81/176): python3-toml-0.10.2-6.el9.noarch.rpm 181 kB/s | 39 kB 00:00 2024-03-04T09:22:44.134 INFO:teuthology.orchestra.run.smithi006.stdout:(66/176): ceph-osd-19.0.0-1739.g1bc250c7.el9.x8 1.1 MB/s | 17 MB 00:15 2024-03-04T09:22:44.159 INFO:teuthology.orchestra.run.smithi006.stdout:(67/176): python3-jsonpatch-1.21-16.el9.noarch. 95 kB/s | 24 kB 00:00 2024-03-04T09:22:44.184 INFO:teuthology.orchestra.run.smithi006.stdout:(68/176): python3-jsonpointer-2.0-4.el9.noarch. 371 kB/s | 18 kB 00:00 2024-03-04T09:22:44.209 INFO:teuthology.orchestra.run.smithi006.stdout:(69/176): python3-mako-1.1.4-6.el9.noarch.rpm 2.9 MB/s | 144 kB 00:00 2024-03-04T09:22:44.235 INFO:teuthology.orchestra.run.smithi006.stdout:(70/176): python3-markupsafe-1.1.1-12.el9.x86_6 658 kB/s | 32 kB 00:00 2024-03-04T09:22:44.289 INFO:teuthology.orchestra.run.smithi146.stdout:(82/176): python3-scipy-1.6.2-8.el9.x86_64.rpm 32 MB/s | 16 MB 00:00 2024-03-04T09:22:44.314 INFO:teuthology.orchestra.run.smithi146.stdout:(83/176): python3-wcwidth-0.2.5-8.el9.noarch.rp 166 kB/s | 41 kB 00:00 2024-03-04T09:22:44.357 INFO:teuthology.orchestra.run.smithi146.stdout:(84/176): qatlib-23.11.0-1.el9.x86_64.rpm 5.2 MB/s | 215 kB 00:00 2024-03-04T09:22:44.368 INFO:teuthology.orchestra.run.smithi006.stdout:(71/176): python3-numpy-1.20.1-5.el9.x86_64.rpm 30 MB/s | 4.8 MB 00:00 2024-03-04T09:22:44.373 INFO:teuthology.orchestra.run.smithi146.stdout:(85/176): qatlib-service-23.11.0-1.el9.x86_64.r 592 kB/s | 34 kB 00:00 2024-03-04T09:22:44.419 INFO:teuthology.orchestra.run.smithi006.stdout:(72/176): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.3 MB/s | 217 kB 00:00 2024-03-04T09:22:44.424 INFO:teuthology.orchestra.run.smithi146.stdout:(86/176): qatzip-libs-1.1.2-2.el9.x86_64.rpm 661 kB/s | 43 kB 00:00 2024-03-04T09:22:44.457 INFO:teuthology.orchestra.run.smithi146.stdout:(87/176): socat-1.7.4.1-5.el9.x86_64.rpm 3.6 MB/s | 300 kB 00:00 2024-03-04T09:22:44.461 INFO:teuthology.orchestra.run.smithi006.stdout:(73/176): python3-oauthlib-3.1.1-5.el9.noarch.r 1.9 MB/s | 174 kB 00:00 2024-03-04T09:22:44.474 INFO:teuthology.orchestra.run.smithi146.stdout:(88/176): uuid-1.6.2-55.el9.x86_64.rpm 1.2 MB/s | 56 kB 00:00 2024-03-04T09:22:44.486 INFO:teuthology.orchestra.run.smithi006.stdout:(74/176): python3-packaging-20.9-5.el9.noarch.r 1.0 MB/s | 69 kB 00:00 2024-03-04T09:22:44.508 INFO:teuthology.orchestra.run.smithi146.stdout:(89/176): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.7 MB/s | 87 kB 00:00 2024-03-04T09:22:44.511 INFO:teuthology.orchestra.run.smithi006.stdout:(75/176): python3-prettytable-0.7.2-27.el9.noar 843 kB/s | 41 kB 00:00 2024-03-04T09:22:44.524 INFO:teuthology.orchestra.run.smithi146.stdout:(90/176): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.3 MB/s | 63 kB 00:00 2024-03-04T09:22:44.545 INFO:teuthology.orchestra.run.smithi006.stdout:(76/176): python3-protobuf-3.14.0-13.el9.noarch 4.1 MB/s | 238 kB 00:00 2024-03-04T09:22:44.550 INFO:teuthology.orchestra.run.smithi146.stdout:(91/176): zlib-devel-1.2.11-40.el9.x86_64.rpm 1.1 MB/s | 44 kB 00:00 2024-03-04T09:22:44.570 INFO:teuthology.orchestra.run.smithi006.stdout:(77/176): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.3 MB/s | 132 kB 00:00 2024-03-04T09:22:44.575 INFO:teuthology.orchestra.run.smithi146.stdout:(92/176): fuse-2.9.9-15.el9.x86_64.rpm 1.6 MB/s | 79 kB 00:00 2024-03-04T09:22:44.604 INFO:teuthology.orchestra.run.smithi006.stdout:(78/176): python3-pyasn1-modules-0.4.8-6.el9.no 3.7 MB/s | 211 kB 00:00 2024-03-04T09:22:44.609 INFO:teuthology.orchestra.run.smithi146.stdout:(93/176): jq-1.6-16.el9.x86_64.rpm 3.2 MB/s | 186 kB 00:00 2024-03-04T09:22:44.629 INFO:teuthology.orchestra.run.smithi006.stdout:(79/176): python3-pytz-2021.1-5.el9.noarch.rpm 848 kB/s | 47 kB 00:00 2024-03-04T09:22:44.651 INFO:teuthology.orchestra.run.smithi146.stdout:(94/176): libgfortran-11.4.1-3.el9.x86_64.rpm 11 MB/s | 802 kB 00:00 2024-03-04T09:22:44.654 INFO:teuthology.orchestra.run.smithi006.stdout:(80/176): python3-requests-oauthlib-1.3.0-12.el 876 kB/s | 43 kB 00:00 2024-03-04T09:22:44.684 INFO:teuthology.orchestra.run.smithi146.stdout:(95/176): libquadmath-11.4.1-3.el9.x86_64.rpm 2.8 MB/s | 194 kB 00:00 2024-03-04T09:22:44.726 INFO:teuthology.orchestra.run.smithi146.stdout:(96/176): oniguruma-6.9.6-1.el9.5.x86_64.rpm 2.9 MB/s | 217 kB 00:00 2024-03-04T09:22:44.769 INFO:teuthology.orchestra.run.smithi146.stdout:(97/176): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.8 MB/s | 241 kB 00:00 2024-03-04T09:22:44.827 INFO:teuthology.orchestra.run.smithi146.stdout:(98/176): python3-cryptography-36.0.1-4.el9.x86 11 MB/s | 1.1 MB 00:00 2024-03-04T09:22:44.869 INFO:teuthology.orchestra.run.smithi146.stdout:(99/176): python3-ply-3.11-14.el9.noarch.rpm 1.1 MB/s | 103 kB 00:00 2024-03-04T09:22:44.894 INFO:teuthology.orchestra.run.smithi146.stdout:(100/176): python3-pycparser-2.20-6.el9.noarch. 1.9 MB/s | 124 kB 00:00 2024-03-04T09:22:44.945 INFO:teuthology.orchestra.run.smithi146.stdout:(101/176): python3-requests-2.25.1-8.el9.noarch 1.5 MB/s | 113 kB 00:00 2024-03-04T09:22:44.978 INFO:teuthology.orchestra.run.smithi146.stdout:(102/176): python3-urllib3-1.26.5-5.el9.noarch. 2.2 MB/s | 187 kB 00:00 2024-03-04T09:22:45.005 INFO:teuthology.orchestra.run.smithi006.stdout:(81/176): python3-toml-0.10.2-6.el9.noarch.rpm 116 kB/s | 39 kB 00:00 2024-03-04T09:22:45.120 INFO:teuthology.orchestra.run.smithi146.stdout:(103/176): python3-devel-3.9.17-2.el9.x86_64.rp 1.4 MB/s | 245 kB 00:00 2024-03-04T09:22:45.214 INFO:teuthology.orchestra.run.smithi006.stdout:(82/176): python3-scipy-1.6.2-8.el9.x86_64.rpm 27 MB/s | 16 MB 00:00 2024-03-04T09:22:45.239 INFO:teuthology.orchestra.run.smithi006.stdout:(83/176): python3-wcwidth-0.2.5-8.el9.noarch.rp 185 kB/s | 41 kB 00:00 2024-03-04T09:22:45.272 INFO:teuthology.orchestra.run.smithi006.stdout:(84/176): qatlib-23.11.0-1.el9.x86_64.rpm 3.7 MB/s | 215 kB 00:00 2024-03-04T09:22:45.279 INFO:teuthology.orchestra.run.smithi146.stdout:(104/176): lua-devel-5.4.4-4.el9.x86_64.rpm 74 kB/s | 22 kB 00:00 2024-03-04T09:22:45.297 INFO:teuthology.orchestra.run.smithi006.stdout:(85/176): qatlib-service-23.11.0-1.el9.x86_64.r 605 kB/s | 34 kB 00:00 2024-03-04T09:22:45.323 INFO:teuthology.orchestra.run.smithi006.stdout:(86/176): qatzip-libs-1.1.2-2.el9.x86_64.rpm 887 kB/s | 43 kB 00:00 2024-03-04T09:22:45.382 INFO:teuthology.orchestra.run.smithi006.stdout:(87/176): socat-1.7.4.1-5.el9.x86_64.rpm 3.6 MB/s | 300 kB 00:00 2024-03-04T09:22:45.407 INFO:teuthology.orchestra.run.smithi006.stdout:(88/176): uuid-1.6.2-55.el9.x86_64.rpm 704 kB/s | 56 kB 00:00 2024-03-04T09:22:45.432 INFO:teuthology.orchestra.run.smithi006.stdout:(89/176): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.7 MB/s | 87 kB 00:00 2024-03-04T09:22:45.457 INFO:teuthology.orchestra.run.smithi006.stdout:(90/176): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.3 MB/s | 63 kB 00:00 2024-03-04T09:22:45.483 INFO:teuthology.orchestra.run.smithi006.stdout:(91/176): zlib-devel-1.2.11-40.el9.x86_64.rpm 904 kB/s | 44 kB 00:00 2024-03-04T09:22:45.508 INFO:teuthology.orchestra.run.smithi006.stdout:(92/176): fuse-2.9.9-15.el9.x86_64.rpm 1.6 MB/s | 79 kB 00:00 2024-03-04T09:22:45.542 INFO:teuthology.orchestra.run.smithi006.stdout:(93/176): jq-1.6-16.el9.x86_64.rpm 3.2 MB/s | 186 kB 00:00 2024-03-04T09:22:45.584 INFO:teuthology.orchestra.run.smithi006.stdout:(94/176): libgfortran-11.4.1-3.el9.x86_64.rpm 11 MB/s | 802 kB 00:00 2024-03-04T09:22:45.617 INFO:teuthology.orchestra.run.smithi006.stdout:(95/176): libquadmath-11.4.1-3.el9.x86_64.rpm 2.8 MB/s | 194 kB 00:00 2024-03-04T09:22:45.643 INFO:teuthology.orchestra.run.smithi006.stdout:(96/176): oniguruma-6.9.6-1.el9.5.x86_64.rpm 3.8 MB/s | 217 kB 00:00 2024-03-04T09:22:45.676 INFO:teuthology.orchestra.run.smithi006.stdout:(97/176): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.1 MB/s | 241 kB 00:00 2024-03-04T09:22:45.735 INFO:teuthology.orchestra.run.smithi006.stdout:(98/176): python3-cryptography-36.0.1-4.el9.x86 13 MB/s | 1.1 MB 00:00 2024-03-04T09:22:45.760 INFO:teuthology.orchestra.run.smithi006.stdout:(99/176): python3-ply-3.11-14.el9.noarch.rpm 1.4 MB/s | 103 kB 00:00 2024-03-04T09:22:45.785 INFO:teuthology.orchestra.run.smithi006.stdout:(100/176): python3-pycparser-2.20-6.el9.noarch. 2.5 MB/s | 124 kB 00:00 2024-03-04T09:22:45.811 INFO:teuthology.orchestra.run.smithi006.stdout:(101/176): python3-requests-2.25.1-8.el9.noarch 2.3 MB/s | 113 kB 00:00 2024-03-04T09:22:45.844 INFO:teuthology.orchestra.run.smithi006.stdout:(102/176): python3-urllib3-1.26.5-5.el9.noarch. 3.2 MB/s | 187 kB 00:00 2024-03-04T09:22:45.879 INFO:teuthology.orchestra.run.smithi146.stdout:(105/176): ceph-test-19.0.0-1739.g1bc250c7.el9. 5.1 MB/s | 47 MB 00:09 2024-03-04T09:22:46.138 INFO:teuthology.orchestra.run.smithi146.stdout:(106/176): gperftools-libs-2.9.1-2.el9.x86_64.r 1.2 MB/s | 309 kB 00:00 2024-03-04T09:22:46.221 INFO:teuthology.orchestra.run.smithi146.stdout:(107/176): grpc-data-1.46.7-10.el9.noarch.rpm 235 kB/s | 19 kB 00:00 2024-03-04T09:22:46.322 INFO:teuthology.orchestra.run.smithi146.stdout:(108/176): abseil-cpp-20211102.0-3.el9.x86_64.r 526 kB/s | 548 kB 00:01 2024-03-04T09:22:46.353 INFO:teuthology.orchestra.run.smithi006.stdout:(103/176): ceph-test-19.0.0-1739.g1bc250c7.el9. 2.8 MB/s | 47 MB 00:16 2024-03-04T09:22:46.379 INFO:teuthology.orchestra.run.smithi006.stdout:(104/176): lua-devel-5.4.4-4.el9.x86_64.rpm 42 kB/s | 22 kB 00:00 2024-03-04T09:22:46.405 INFO:teuthology.orchestra.run.smithi146.stdout:(109/176): protobuf-compiler-3.14.0-13.el9.x86_ 672 kB/s | 863 kB 00:01 2024-03-04T09:22:46.430 INFO:teuthology.orchestra.run.smithi146.stdout:(110/176): libarrow-doc-9.0.0-10.el9.noarch.rpm 241 kB/s | 26 kB 00:00 2024-03-04T09:22:46.462 INFO:teuthology.orchestra.run.smithi006.stdout:(105/176): python3-devel-3.9.17-2.el9.x86_64.rp 379 kB/s | 245 kB 00:00 2024-03-04T09:22:46.556 INFO:teuthology.orchestra.run.smithi146.stdout:(111/176): libarrow-9.0.0-10.el9.x86_64.rpm 13 MB/s | 4.4 MB 00:00 2024-03-04T09:22:46.581 INFO:teuthology.orchestra.run.smithi146.stdout:(112/176): liboath-2.6.7-2.el9.x86_64.rpm 278 kB/s | 49 kB 00:00 2024-03-04T09:22:46.606 INFO:teuthology.orchestra.run.smithi146.stdout:(113/176): libunwind-1.6.2-1.el9.x86_64.rpm 385 kB/s | 67 kB 00:00 2024-03-04T09:22:46.631 INFO:teuthology.orchestra.run.smithi146.stdout:(114/176): luarocks-3.9.2-1.el9.noarch.rpm 2.0 MB/s | 151 kB 00:00 2024-03-04T09:22:46.687 INFO:teuthology.orchestra.run.smithi006.stdout:(106/176): protobuf-compiler-3.14.0-13.el9.x86_ 2.5 MB/s | 863 kB 00:00 2024-03-04T09:22:46.698 INFO:teuthology.orchestra.run.smithi146.stdout:(115/176): python3-bcrypt-3.2.2-1.el9.x86_64.rp 659 kB/s | 43 kB 00:00 2024-03-04T09:22:46.765 INFO:teuthology.orchestra.run.smithi146.stdout:(116/176): python3-cachetools-4.2.4-1.el9.noarc 489 kB/s | 32 kB 00:00 2024-03-04T09:22:46.807 INFO:teuthology.orchestra.run.smithi146.stdout:(117/176): python3-asyncssh-2.13.2-4.el9.noarch 2.7 MB/s | 548 kB 00:00 2024-03-04T09:22:46.832 INFO:teuthology.orchestra.run.smithi146.stdout:(118/176): python3-certifi-2021.10.8-2.el9.noar 227 kB/s | 15 kB 00:00 2024-03-04T09:22:46.874 INFO:teuthology.orchestra.run.smithi146.stdout:(119/176): python3-cheroot-8.6.0-4.el9.noarch.r 2.5 MB/s | 172 kB 00:00 2024-03-04T09:22:46.907 INFO:teuthology.orchestra.run.smithi146.stdout:(120/176): python3-cherrypy-18.6.1-2.el9.noarch 4.7 MB/s | 358 kB 00:00 2024-03-04T09:22:46.950 INFO:teuthology.orchestra.run.smithi146.stdout:(121/176): python3-cmd2-2.4.2-2.el9.noarch.rpm 4.0 MB/s | 303 kB 00:00 2024-03-04T09:22:46.971 INFO:teuthology.orchestra.run.smithi006.stdout:(107/176): grpc-data-1.46.7-10.el9.noarch.rpm 69 kB/s | 19 kB 00:00 2024-03-04T09:22:46.991 INFO:teuthology.orchestra.run.smithi146.stdout:(122/176): python3-colorama-0.4.4-12.el9.noarch 425 kB/s | 35 kB 00:00 2024-03-04T09:22:47.013 INFO:teuthology.orchestra.run.smithi006.stdout:(108/176): abseil-cpp-20211102.0-3.el9.x86_64.r 866 kB/s | 548 kB 00:00 2024-03-04T09:22:47.025 INFO:teuthology.orchestra.run.smithi146.stdout:(123/176): python3-google-auth-2.28.0-1.el9.noa 2.7 MB/s | 203 kB 00:00 2024-03-04T09:22:47.038 INFO:teuthology.orchestra.run.smithi006.stdout:(109/176): gperftools-libs-2.9.1-2.el9.x86_64.r 612 kB/s | 309 kB 00:00 2024-03-04T09:22:47.059 INFO:teuthology.orchestra.run.smithi146.stdout:(124/176): parquet-libs-9.0.0-10.el9.x86_64.rpm 1.7 MB/s | 838 kB 00:00 2024-03-04T09:22:47.088 INFO:teuthology.orchestra.run.smithi006.stdout:(110/176): libarrow-doc-9.0.0-10.el9.noarch.rpm 349 kB/s | 26 kB 00:00 2024-03-04T09:22:47.092 INFO:teuthology.orchestra.run.smithi146.stdout:(125/176): python3-grpcio-tools-1.46.7-10.el9.x 2.1 MB/s | 144 kB 00:00 2024-03-04T09:22:47.113 INFO:teuthology.orchestra.run.smithi006.stdout:(111/176): liboath-2.6.7-2.el9.x86_64.rpm 652 kB/s | 49 kB 00:00 2024-03-04T09:22:47.143 INFO:teuthology.orchestra.run.smithi146.stdout:(126/176): python3-jaraco-8.2.1-3.el9.noarch.rp 129 kB/s | 11 kB 00:00 2024-03-04T09:22:47.164 INFO:teuthology.orchestra.run.smithi006.stdout:(112/176): libunwind-1.6.2-1.el9.x86_64.rpm 908 kB/s | 67 kB 00:00 2024-03-04T09:22:47.185 INFO:teuthology.orchestra.run.smithi146.stdout:(127/176): python3-jaraco-classes-3.2.1-5.el9.n 193 kB/s | 18 kB 00:00 2024-03-04T09:22:47.206 INFO:teuthology.orchestra.run.smithi006.stdout:(113/176): luarocks-3.9.2-1.el9.noarch.rpm 1.6 MB/s | 151 kB 00:00 2024-03-04T09:22:47.252 INFO:teuthology.orchestra.run.smithi146.stdout:(128/176): python3-jaraco-collections-3.0.0-8.e 214 kB/s | 23 kB 00:00 2024-03-04T09:22:47.294 INFO:teuthology.orchestra.run.smithi146.stdout:(129/176): python3-jaraco-functools-3.5.0-2.el9 180 kB/s | 19 kB 00:00 2024-03-04T09:22:47.306 INFO:teuthology.orchestra.run.smithi006.stdout:(114/176): parquet-libs-9.0.0-10.el9.x86_64.rpm 5.8 MB/s | 838 kB 00:00 2024-03-04T09:22:47.336 INFO:teuthology.orchestra.run.smithi146.stdout:(130/176): python3-jaraco-text-3.2.0-6.el9.noar 237 kB/s | 20 kB 00:00 2024-03-04T09:22:47.389 INFO:teuthology.orchestra.run.smithi006.stdout:(115/176): python3-asyncssh-2.13.2-4.el9.noarch 2.9 MB/s | 548 kB 00:00 2024-03-04T09:22:47.394 INFO:teuthology.orchestra.run.smithi146.stdout:(131/176): python3-kubernetes-26.1.0-2.el9.noar 10 MB/s | 1.0 MB 00:00 2024-03-04T09:22:47.415 INFO:teuthology.orchestra.run.smithi006.stdout:(116/176): python3-bcrypt-3.2.2-1.el9.x86_64.rp 404 kB/s | 43 kB 00:00 2024-03-04T09:22:47.445 INFO:teuthology.orchestra.run.smithi146.stdout:(132/176): python3-logutils-0.3.5-21.el9.noarch 430 kB/s | 46 kB 00:00 2024-03-04T09:22:47.465 INFO:teuthology.orchestra.run.smithi006.stdout:(117/176): python3-cachetools-4.2.4-1.el9.noarc 432 kB/s | 32 kB 00:00 2024-03-04T09:22:47.491 INFO:teuthology.orchestra.run.smithi006.stdout:(118/176): python3-certifi-2021.10.8-2.el9.noar 201 kB/s | 15 kB 00:00 2024-03-04T09:22:47.503 INFO:teuthology.orchestra.run.smithi146.stdout:(133/176): python3-grpcio-1.46.7-10.el9.x86_64. 4.0 MB/s | 2.0 MB 00:00 2024-03-04T09:22:47.529 INFO:teuthology.orchestra.run.smithi146.stdout:(134/176): python3-more-itertools-8.12.0-2.el9. 593 kB/s | 79 kB 00:00 2024-03-04T09:22:47.554 INFO:teuthology.orchestra.run.smithi146.stdout:(135/176): python3-natsort-7.1.1-5.el9.noarch.r 535 kB/s | 58 kB 00:00 2024-03-04T09:22:47.587 INFO:teuthology.orchestra.run.smithi146.stdout:(136/176): python3-pecan-1.4.2-3.el9.noarch.rpm 3.2 MB/s | 272 kB 00:00 2024-03-04T09:22:47.590 INFO:teuthology.orchestra.run.smithi006.stdout:(119/176): libarrow-9.0.0-10.el9.x86_64.rpm 7.1 MB/s | 4.4 MB 00:00 2024-03-04T09:22:47.613 INFO:teuthology.orchestra.run.smithi146.stdout:(137/176): python3-portend-3.1.0-2.el9.noarch.r 198 kB/s | 16 kB 00:00 2024-03-04T09:22:47.625 INFO:teuthology.orchestra.run.smithi006.stdout:(120/176): python3-cheroot-8.6.0-4.el9.noarch.r 1.1 MB/s | 172 kB 00:00 2024-03-04T09:22:47.646 INFO:teuthology.orchestra.run.smithi146.stdout:(138/176): python3-pyOpenSSL-21.0.0-1.el9.noarc 984 kB/s | 90 kB 00:00 2024-03-04T09:22:47.658 INFO:teuthology.orchestra.run.smithi006.stdout:(121/176): python3-cherrypy-18.6.1-2.el9.noarch 2.1 MB/s | 358 kB 00:00 2024-03-04T09:22:47.692 INFO:teuthology.orchestra.run.smithi006.stdout:(122/176): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.0 MB/s | 303 kB 00:00 2024-03-04T09:22:47.696 INFO:teuthology.orchestra.run.smithi146.stdout:(139/176): python3-pyperclip-1.8.2-2.el9.noarch 260 kB/s | 28 kB 00:00 2024-03-04T09:22:47.722 INFO:teuthology.orchestra.run.smithi146.stdout:(140/176): python3-repoze-lru-0.7-16.el9.noarch 284 kB/s | 31 kB 00:00 2024-03-04T09:22:47.725 INFO:teuthology.orchestra.run.smithi006.stdout:(123/176): python3-colorama-0.4.4-12.el9.noarch 354 kB/s | 35 kB 00:00 2024-03-04T09:22:47.755 INFO:teuthology.orchestra.run.smithi146.stdout:(141/176): python3-routes-2.5.1-5.el9.noarch.rp 1.7 MB/s | 188 kB 00:00 2024-03-04T09:22:47.759 INFO:teuthology.orchestra.run.smithi006.stdout:(124/176): python3-google-auth-2.28.0-1.el9.noa 2.0 MB/s | 203 kB 00:00 2024-03-04T09:22:47.781 INFO:teuthology.orchestra.run.smithi146.stdout:(142/176): python3-rsa-4.9-2.el9.noarch.rpm 712 kB/s | 59 kB 00:00 2024-03-04T09:22:47.801 INFO:teuthology.orchestra.run.smithi006.stdout:(125/176): python3-grpcio-tools-1.46.7-10.el9.x 1.9 MB/s | 144 kB 00:00 2024-03-04T09:22:47.806 INFO:teuthology.orchestra.run.smithi146.stdout:(143/176): python3-tempora-5.0.0-2.el9.noarch.r 430 kB/s | 36 kB 00:00 2024-03-04T09:22:47.826 INFO:teuthology.orchestra.run.smithi006.stdout:(126/176): python3-jaraco-8.2.1-3.el9.noarch.rp 162 kB/s | 11 kB 00:00 2024-03-04T09:22:47.887 INFO:teuthology.orchestra.run.smithi146.stdout:(144/176): python3-typing-extensions-4.4.0-2.el 391 kB/s | 51 kB 00:00 2024-03-04T09:22:47.901 INFO:teuthology.orchestra.run.smithi006.stdout:(127/176): python3-grpcio-1.46.7-10.el9.x86_64. 9.8 MB/s | 2.0 MB 00:00 2024-03-04T09:22:47.918 INFO:teuthology.orchestra.run.smithi006.stdout:(128/176): python3-jaraco-classes-3.2.1-5.el9.n 152 kB/s | 18 kB 00:00 2024-03-04T09:22:47.935 INFO:teuthology.orchestra.run.smithi006.stdout:(129/176): python3-jaraco-collections-3.0.0-8.e 214 kB/s | 23 kB 00:00 2024-03-04T09:22:47.969 INFO:teuthology.orchestra.run.smithi006.stdout:(130/176): python3-jaraco-functools-3.5.0-2.el9 294 kB/s | 19 kB 00:00 2024-03-04T09:22:48.003 INFO:teuthology.orchestra.run.smithi006.stdout:(131/176): python3-jaraco-text-3.2.0-6.el9.noar 297 kB/s | 20 kB 00:00 2024-03-04T09:22:48.045 INFO:teuthology.orchestra.run.smithi006.stdout:(132/176): python3-logutils-0.3.5-21.el9.noarch 618 kB/s | 46 kB 00:00 2024-03-04T09:22:48.078 INFO:teuthology.orchestra.run.smithi006.stdout:(133/176): python3-more-itertools-8.12.0-2.el9. 1.0 MB/s | 79 kB 00:00 2024-03-04T09:22:48.120 INFO:teuthology.orchestra.run.smithi006.stdout:(134/176): python3-kubernetes-26.1.0-2.el9.noar 5.6 MB/s | 1.0 MB 00:00 2024-03-04T09:22:48.145 INFO:teuthology.orchestra.run.smithi006.stdout:(135/176): python3-natsort-7.1.1-5.el9.noarch.r 580 kB/s | 58 kB 00:00 2024-03-04T09:22:48.156 INFO:teuthology.orchestra.run.smithi146.stdout:(145/176): python3-websocket-client-1.2.3-2.el9 256 kB/s | 90 kB 00:00 2024-03-04T09:22:48.179 INFO:teuthology.orchestra.run.smithi006.stdout:(136/176): python3-pecan-1.4.2-3.el9.noarch.rpm 2.7 MB/s | 272 kB 00:00 2024-03-04T09:22:48.190 INFO:teuthology.orchestra.run.smithi146.stdout:(146/176): python3-webob-1.8.7-6.el9.noarch.rpm 562 kB/s | 230 kB 00:00 2024-03-04T09:22:48.204 INFO:teuthology.orchestra.run.smithi006.stdout:(137/176): python3-portend-3.1.0-2.el9.noarch.r 198 kB/s | 16 kB 00:00 2024-03-04T09:22:48.224 INFO:teuthology.orchestra.run.smithi146.stdout:(147/176): python3-werkzeug-2.0.3-3.el9.1.noarc 1.2 MB/s | 427 kB 00:00 2024-03-04T09:22:48.229 INFO:teuthology.orchestra.run.smithi006.stdout:(138/176): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.1 MB/s | 90 kB 00:00 2024-03-04T09:22:48.249 INFO:teuthology.orchestra.run.smithi146.stdout:(148/176): python3-zc-lockfile-2.0-10.el9.noarc 219 kB/s | 20 kB 00:00 2024-03-04T09:22:48.255 INFO:teuthology.orchestra.run.smithi006.stdout:(139/176): python3-pyperclip-1.8.2-2.el9.noarch 376 kB/s | 28 kB 00:00 2024-03-04T09:22:48.280 INFO:teuthology.orchestra.run.smithi006.stdout:(140/176): python3-repoze-lru-0.7-16.el9.noarch 412 kB/s | 31 kB 00:00 2024-03-04T09:22:48.282 INFO:teuthology.orchestra.run.smithi146.stdout:(149/176): re2-20211101-3.el9.x86_64.rpm 2.0 MB/s | 192 kB 00:00 2024-03-04T09:22:48.330 INFO:teuthology.orchestra.run.smithi006.stdout:(141/176): python3-routes-2.5.1-5.el9.noarch.rp 1.8 MB/s | 188 kB 00:00 2024-03-04T09:22:48.356 INFO:teuthology.orchestra.run.smithi006.stdout:(142/176): python3-tempora-5.0.0-2.el9.noarch.r 480 kB/s | 36 kB 00:00 2024-03-04T09:22:48.381 INFO:teuthology.orchestra.run.smithi006.stdout:(143/176): python3-rsa-4.9-2.el9.noarch.rpm 473 kB/s | 59 kB 00:00 2024-03-04T09:22:48.406 INFO:teuthology.orchestra.run.smithi006.stdout:(144/176): python3-typing-extensions-4.4.0-2.el 685 kB/s | 51 kB 00:00 2024-03-04T09:22:48.440 INFO:teuthology.orchestra.run.smithi006.stdout:(145/176): python3-webob-1.8.7-6.el9.noarch.rpm 2.7 MB/s | 230 kB 00:00 2024-03-04T09:22:48.465 INFO:teuthology.orchestra.run.smithi006.stdout:(146/176): python3-websocket-client-1.2.3-2.el9 1.1 MB/s | 90 kB 00:00 2024-03-04T09:22:48.507 INFO:teuthology.orchestra.run.smithi006.stdout:(147/176): python3-werkzeug-2.0.3-3.el9.1.noarc 4.2 MB/s | 427 kB 00:00 2024-03-04T09:22:48.532 INFO:teuthology.orchestra.run.smithi006.stdout:(148/176): python3-zc-lockfile-2.0-10.el9.noarc 218 kB/s | 20 kB 00:00 2024-03-04T09:22:48.566 INFO:teuthology.orchestra.run.smithi006.stdout:(149/176): re2-20211101-3.el9.x86_64.rpm 1.9 MB/s | 192 kB 00:00 2024-03-04T09:22:48.641 INFO:teuthology.orchestra.run.smithi006.stdout:(150/176): thrift-0.15.0-2.el9.x86_64.rpm 12 MB/s | 1.6 MB 00:00 2024-03-04T09:22:48.683 INFO:teuthology.orchestra.run.smithi006.stdout:(151/176): libacl-devel-2.3.1-4.el9.x86_64.rpm 1.6 MB/s | 68 kB 00:00 2024-03-04T09:22:48.858 INFO:teuthology.orchestra.run.smithi146.stdout:(150/176): thrift-0.15.0-2.el9.x86_64.rpm 2.5 MB/s | 1.6 MB 00:00 2024-03-04T09:22:48.900 INFO:teuthology.orchestra.run.smithi146.stdout:(151/176): libacl-devel-2.3.1-4.el9.x86_64.rpm 1.6 MB/s | 68 kB 00:00 2024-03-04T09:22:48.967 INFO:teuthology.orchestra.run.smithi006.stdout:(152/176): NetworkManager-1.46.0-1.el9.x86_64.r 8.0 MB/s | 2.3 MB 00:00 2024-03-04T09:22:49.087 INFO:teuthology.orchestra.run.smithi146.stdout:(152/176): NetworkManager-1.46.0-1.el9.x86_64.r 12 MB/s | 2.3 MB 00:00 2024-03-04T09:22:49.134 INFO:teuthology.orchestra.run.smithi006.stdout:(153/176): NetworkManager-libnm-1.46.0-1.el9.x8 11 MB/s | 1.8 MB 00:00 2024-03-04T09:22:49.167 INFO:teuthology.orchestra.run.smithi006.stdout:(154/176): NetworkManager-team-1.46.0-1.el9.x86 1.3 MB/s | 41 kB 00:00 2024-03-04T09:22:49.200 INFO:teuthology.orchestra.run.smithi146.stdout:(153/176): librbd1-19.0.0-1739.g1bc250c7.el9.x8 3.4 MB/s | 3.1 MB 00:00 2024-03-04T09:22:49.209 INFO:teuthology.orchestra.run.smithi006.stdout:(155/176): NetworkManager-tui-1.46.0-1.el9.x86_ 5.9 MB/s | 247 kB 00:00 2024-03-04T09:22:49.259 INFO:teuthology.orchestra.run.smithi146.stdout:(154/176): NetworkManager-libnm-1.46.0-1.el9.x8 11 MB/s | 1.8 MB 00:00 2024-03-04T09:22:49.276 INFO:teuthology.orchestra.run.smithi006.stdout:(156/176): acl-2.3.1-4.el9.x86_64.rpm 1.0 MB/s | 69 kB 00:00 2024-03-04T09:22:49.284 INFO:teuthology.orchestra.run.smithi146.stdout:(155/176): NetworkManager-team-1.46.0-1.el9.x86 533 kB/s | 41 kB 00:00 2024-03-04T09:22:49.318 INFO:teuthology.orchestra.run.smithi146.stdout:(156/176): NetworkManager-tui-1.46.0-1.el9.x86_ 4.2 MB/s | 247 kB 00:00 2024-03-04T09:22:49.318 INFO:teuthology.orchestra.run.smithi006.stdout:(157/176): elfutils-debuginfod-client-0.190-2.e 871 kB/s | 35 kB 00:00 2024-03-04T09:22:49.343 INFO:teuthology.orchestra.run.smithi146.stdout:(157/176): acl-2.3.1-4.el9.x86_64.rpm 1.2 MB/s | 69 kB 00:00 2024-03-04T09:22:49.360 INFO:teuthology.orchestra.run.smithi006.stdout:(158/176): elfutils-libelf-0.190-2.el9.x86_64.r 4.6 MB/s | 191 kB 00:00 2024-03-04T09:22:49.377 INFO:teuthology.orchestra.run.smithi146.stdout:(158/176): elfutils-debuginfod-client-0.190-2.e 616 kB/s | 35 kB 00:00 2024-03-04T09:22:49.410 INFO:teuthology.orchestra.run.smithi146.stdout:(159/176): elfutils-libelf-0.190-2.el9.x86_64.r 2.8 MB/s | 191 kB 00:00 2024-03-04T09:22:49.419 INFO:teuthology.orchestra.run.smithi006.stdout:(159/176): elfutils-libs-0.190-2.el9.x86_64.rpm 4.3 MB/s | 254 kB 00:00 2024-03-04T09:22:49.444 INFO:teuthology.orchestra.run.smithi146.stdout:(160/176): elfutils-libs-0.190-2.el9.x86_64.rpm 3.9 MB/s | 254 kB 00:00 2024-03-04T09:22:49.486 INFO:teuthology.orchestra.run.smithi146.stdout:(161/176): iproute-6.7.0-2.el9.x86_64.rpm 11 MB/s | 802 kB 00:00 2024-03-04T09:22:49.503 INFO:teuthology.orchestra.run.smithi006.stdout:(160/176): iproute-6.7.0-2.el9.x86_64.rpm 9.5 MB/s | 802 kB 00:00 2024-03-04T09:22:49.548 INFO:teuthology.orchestra.run.smithi146.stdout:(162/176): iproute-tc-6.7.0-2.el9.x86_64.rpm 4.4 MB/s | 433 kB 00:00 2024-03-04T09:22:49.578 INFO:teuthology.orchestra.run.smithi146.stdout:(163/176): libacl-2.3.1-4.el9.x86_64.rpm 246 kB/s | 22 kB 00:00 2024-03-04T09:22:49.595 INFO:teuthology.orchestra.run.smithi006.stdout:(161/176): iproute-tc-6.7.0-2.el9.x86_64.rpm 4.7 MB/s | 433 kB 00:00 2024-03-04T09:22:49.620 INFO:teuthology.orchestra.run.smithi146.stdout:(164/176): libblkid-2.37.4-18.el9.x86_64.rpm 1.5 MB/s | 107 kB 00:00 2024-03-04T09:22:49.653 INFO:teuthology.orchestra.run.smithi146.stdout:(165/176): libfdisk-2.37.4-18.el9.x86_64.rpm 2.0 MB/s | 154 kB 00:00 2024-03-04T09:22:49.670 INFO:teuthology.orchestra.run.smithi006.stdout:(162/176): librbd1-19.0.0-1739.g1bc250c7.el9.x8 2.8 MB/s | 3.1 MB 00:01 2024-03-04T09:22:49.678 INFO:teuthology.orchestra.run.smithi146.stdout:(166/176): libmount-2.37.4-18.el9.x86_64.rpm 2.3 MB/s | 135 kB 00:00 2024-03-04T09:22:49.695 INFO:teuthology.orchestra.run.smithi006.stdout:(163/176): libacl-2.3.1-4.el9.x86_64.rpm 221 kB/s | 22 kB 00:00 2024-03-04T09:22:49.696 INFO:teuthology.orchestra.run.smithi146.stdout:(167/176): libsmartcols-2.37.4-18.el9.x86_64.rp 1.5 MB/s | 63 kB 00:00 2024-03-04T09:22:49.729 INFO:teuthology.orchestra.run.smithi006.stdout:(164/176): libblkid-2.37.4-18.el9.x86_64.rpm 1.8 MB/s | 107 kB 00:00 2024-03-04T09:22:49.754 INFO:teuthology.orchestra.run.smithi006.stdout:(165/176): libfdisk-2.37.4-18.el9.x86_64.rpm 2.6 MB/s | 154 kB 00:00 2024-03-04T09:22:49.779 INFO:teuthology.orchestra.run.smithi146.stdout:(168/176): librados2-19.0.0-1739.g1bc250c7.el9. 2.2 MB/s | 3.4 MB 00:01 2024-03-04T09:22:49.807 INFO:teuthology.orchestra.run.smithi146.stdout:(169/176): libuuid-2.37.4-18.el9.x86_64.rpm 222 kB/s | 28 kB 00:00 2024-03-04T09:22:49.813 INFO:teuthology.orchestra.run.smithi006.stdout:(166/176): libmount-2.37.4-18.el9.x86_64.rpm 1.6 MB/s | 135 kB 00:00 2024-03-04T09:22:49.830 INFO:teuthology.orchestra.run.smithi006.stdout:(167/176): libsmartcols-2.37.4-18.el9.x86_64.rp 842 kB/s | 63 kB 00:00 2024-03-04T09:22:49.863 INFO:teuthology.orchestra.run.smithi146.stdout:(170/176): openssl-3.0.7-27.el9.x86_64.rpm 6.9 MB/s | 1.2 MB 00:00 2024-03-04T09:22:49.864 INFO:teuthology.orchestra.run.smithi006.stdout:(168/176): libuuid-2.37.4-18.el9.x86_64.rpm 567 kB/s | 28 kB 00:00 2024-03-04T09:22:49.939 INFO:teuthology.orchestra.run.smithi146.stdout:(171/176): openssl-libs-3.0.7-27.el9.x86_64.rpm 15 MB/s | 2.2 MB 00:00 2024-03-04T09:22:49.947 INFO:teuthology.orchestra.run.smithi006.stdout:(169/176): openssl-3.0.7-27.el9.x86_64.rpm 9.9 MB/s | 1.2 MB 00:00 2024-03-04T09:22:50.114 INFO:teuthology.orchestra.run.smithi146.stdout:(172/176): util-linux-core-2.37.4-18.el9.x86_64 1.8 MB/s | 438 kB 00:00 2024-03-04T09:22:50.139 INFO:teuthology.orchestra.run.smithi006.stdout:(170/176): openssl-libs-3.0.7-27.el9.x86_64.rpm 7.9 MB/s | 2.2 MB 00:00 2024-03-04T09:22:50.172 INFO:teuthology.orchestra.run.smithi146.stdout:(173/176): util-linux-2.37.4-18.el9.x86_64.rpm 7.4 MB/s | 2.2 MB 00:00 2024-03-04T09:22:50.197 INFO:teuthology.orchestra.run.smithi146.stdout:(174/176): util-linux-user-2.37.4-18.el9.x86_64 119 kB/s | 30 kB 00:00 2024-03-04T09:22:50.231 INFO:teuthology.orchestra.run.smithi146.stdout:(175/176): xfsdump-3.1.12-4.el9.x86_64.rpm 2.7 MB/s | 321 kB 00:00 2024-03-04T09:22:50.232 INFO:teuthology.orchestra.run.smithi006.stdout:(171/176): util-linux-2.37.4-18.el9.x86_64.rpm 8.0 MB/s | 2.2 MB 00:00 2024-03-04T09:22:50.315 INFO:teuthology.orchestra.run.smithi006.stdout:(172/176): util-linux-core-2.37.4-18.el9.x86_64 2.7 MB/s | 438 kB 00:00 2024-03-04T09:22:50.340 INFO:teuthology.orchestra.run.smithi006.stdout:(173/176): util-linux-user-2.37.4-18.el9.x86_64 283 kB/s | 30 kB 00:00 2024-03-04T09:22:50.364 INFO:teuthology.orchestra.run.smithi146.stdout:(176/176): xfsprogs-6.3.0-1.el9.x86_64.rpm 6.5 MB/s | 1.1 MB 00:00 2024-03-04T09:22:50.373 INFO:teuthology.orchestra.run.smithi006.stdout:(174/176): xfsdump-3.1.12-4.el9.x86_64.rpm 5.5 MB/s | 321 kB 00:00 2024-03-04T09:22:50.373 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-03-04T09:22:50.374 INFO:teuthology.orchestra.run.smithi146.stdout:Total 11 MB/s | 224 MB 00:20 2024-03-04T09:22:50.474 INFO:teuthology.orchestra.run.smithi006.stdout:(175/176): xfsprogs-6.3.0-1.el9.x86_64.rpm 8.2 MB/s | 1.1 MB 00:00 2024-03-04T09:22:50.507 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-03-04T09:22:50.884 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-03-04T09:22:50.885 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-03-04T09:22:51.007 INFO:teuthology.orchestra.run.smithi006.stdout:(176/176): librados2-19.0.0-1739.g1bc250c7.el9. 1.4 MB/s | 3.4 MB 00:02 2024-03-04T09:22:51.021 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2024-03-04T09:22:51.021 INFO:teuthology.orchestra.run.smithi006.stdout:Total 7.9 MB/s | 224 MB 00:28 2024-03-04T09:22:51.157 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2024-03-04T09:22:51.554 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2024-03-04T09:22:51.554 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2024-03-04T09:22:53.265 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-03-04T09:22:53.266 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-03-04T09:22:54.089 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2024-03-04T09:22:54.090 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2024-03-04T09:23:00.831 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-03-04T09:23:00.939 INFO:teuthology.orchestra.run.smithi146.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/202 2024-03-04T09:23:00.978 INFO:teuthology.orchestra.run.smithi146.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/202 2024-03-04T09:23:01.025 INFO:teuthology.orchestra.run.smithi146.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/202 2024-03-04T09:23:01.117 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/202 2024-03-04T09:23:01.207 INFO:teuthology.orchestra.run.smithi146.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 4/202 2024-03-04T09:23:01.298 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 5/202 2024-03-04T09:23:01.415 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/202 2024-03-04T09:23:01.478 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : thrift-0.15.0-2.el9.x86_64 7/202 2024-03-04T09:23:01.508 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 8/202 2024-03-04T09:23:01.536 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2024-03-04T09:23:01.600 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 9/202 2024-03-04T09:23:01.629 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/202 2024-03-04T09:23:01.677 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/202 2024-03-04T09:23:01.717 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/202 2024-03-04T09:23:01.802 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/202 2024-03-04T09:23:01.877 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 4/202 2024-03-04T09:23:01.966 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 5/202 2024-03-04T09:23:02.024 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 10/202 2024-03-04T09:23:02.064 INFO:teuthology.orchestra.run.smithi146.stdout: Upgrading : librados2-2:19.0.0-1739.g1bc250c7.el9.x86_64 11/202 2024-03-04T09:23:02.187 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/202 2024-03-04T09:23:02.247 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: librados2-2:19.0.0-1739.g1bc250c7.el9.x86_64 11/202 2024-03-04T09:23:02.248 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : thrift-0.15.0-2.el9.x86_64 7/202 2024-03-04T09:23:02.273 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libcephfs2-2:19.0.0-1739.g1bc250c7.el9.x86_64 12/202 2024-03-04T09:23:02.283 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 8/202 2024-03-04T09:23:02.377 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: libcephfs2-2:19.0.0-1739.g1bc250c7.el9.x86_64 12/202 2024-03-04T09:23:02.377 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 9/202 2024-03-04T09:23:02.417 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-rados-2:19.0.0-1739.g1bc250c7.el9.x86_64 13/202 2024-03-04T09:23:02.662 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 14/202 2024-03-04T09:23:02.729 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/202 2024-03-04T09:23:02.760 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libcephsqlite-2:19.0.0-1739.g1bc250c7.el9.x86_64 16/202 2024-03-04T09:23:02.806 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 10/202 2024-03-04T09:23:02.849 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : librados2-2:19.0.0-1739.g1bc250c7.el9.x86_64 11/202 2024-03-04T09:23:02.912 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1739.g1bc250c7.el9.x86_64 16/202 2024-03-04T09:23:02.952 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libradosstriper1-2:19.0.0-1739.g1bc250c7.el9.x86 17/202 2024-03-04T09:23:03.008 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: librados2-2:19.0.0-1739.g1bc250c7.el9.x86_64 11/202 2024-03-04T09:23:03.040 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libcephfs2-2:19.0.0-1739.g1bc250c7.el9.x86_64 12/202 2024-03-04T09:23:03.044 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1739.g1bc250c7.el9.x86 17/202 2024-03-04T09:23:03.105 INFO:teuthology.orchestra.run.smithi146.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/202 2024-03-04T09:23:03.146 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/202 2024-03-04T09:23:03.164 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: libcephfs2-2:19.0.0-1739.g1bc250c7.el9.x86_64 12/202 2024-03-04T09:23:03.199 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-rados-2:19.0.0-1739.g1bc250c7.el9.x86_64 13/202 2024-03-04T09:23:03.207 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/202 2024-03-04T09:23:03.445 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 14/202 2024-03-04T09:23:03.503 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/202 2024-03-04T09:23:03.528 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libcephsqlite-2:19.0.0-1739.g1bc250c7.el9.x86_64 16/202 2024-03-04T09:23:03.627 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/202 2024-03-04T09:23:03.679 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/202 2024-03-04T09:23:03.714 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1739.g1bc250c7.el9.x86_64 16/202 2024-03-04T09:23:03.721 INFO:teuthology.orchestra.run.smithi146.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/202 2024-03-04T09:23:03.745 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libradosstriper1-2:19.0.0-1739.g1bc250c7.el9.x86 17/202 2024-03-04T09:23:03.855 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1739.g1bc250c7.el9.x86 17/202 2024-03-04T09:23:03.909 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/202 2024-03-04T09:23:03.933 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/202 2024-03-04T09:23:03.999 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/202 2024-03-04T09:23:04.051 INFO:teuthology.orchestra.run.smithi146.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/202 2024-03-04T09:23:04.178 INFO:teuthology.orchestra.run.smithi146.stdout: Upgrading : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/202 2024-03-04T09:23:04.266 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/202 2024-03-04T09:23:04.277 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/202 2024-03-04T09:23:04.324 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/202 2024-03-04T09:23:04.375 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/202 2024-03-04T09:23:04.467 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/202 2024-03-04T09:23:04.547 INFO:teuthology.orchestra.run.smithi146.stdout: Upgrading : NetworkManager-1:1.46.0-1.el9.x86_64 24/202 2024-03-04T09:23:04.758 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/202 2024-03-04T09:23:04.869 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/202 2024-03-04T09:23:04.932 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/202 2024-03-04T09:23:05.021 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/202 2024-03-04T09:23:05.056 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/202 2024-03-04T09:23:05.181 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/202 2024-03-04T09:23:05.220 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/202 2024-03-04T09:23:05.231 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/202 2024-03-04T09:23:05.285 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : NetworkManager-1:1.46.0-1.el9.x86_64 24/202 2024-03-04T09:23:05.318 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/202 2024-03-04T09:23:05.382 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 29/202 2024-03-04T09:23:05.434 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 30/202 2024-03-04T09:23:05.502 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 31/202 2024-03-04T09:23:05.554 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 32/202 2024-03-04T09:23:05.667 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/202 2024-03-04T09:23:05.686 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/202 2024-03-04T09:23:05.708 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/202 2024-03-04T09:23:05.773 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/202 2024-03-04T09:23:05.822 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/202 2024-03-04T09:23:05.875 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/202 2024-03-04T09:23:05.891 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/202 2024-03-04T09:23:06.021 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/202 2024-03-04T09:23:06.021 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/202 2024-03-04T09:23:06.057 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-ceph-common-2:19.0.0-1739.g1bc250c7.el9. 39/202 2024-03-04T09:23:06.066 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/202 2024-03-04T09:23:06.109 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-ceph-argparse-2:19.0.0-1739.g1bc250c7.el 40/202 2024-03-04T09:23:06.147 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-cephfs-2:19.0.0-1739.g1bc250c7.el9.x86_6 41/202 2024-03-04T09:23:06.178 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/202 2024-03-04T09:23:06.240 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 29/202 2024-03-04T09:23:06.265 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/202 2024-03-04T09:23:06.278 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 30/202 2024-03-04T09:23:06.326 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/202 2024-03-04T09:23:06.335 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 31/202 2024-03-04T09:23:06.391 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 32/202 2024-03-04T09:23:06.445 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/202 2024-03-04T09:23:06.517 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/202 2024-03-04T09:23:06.591 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/202 2024-03-04T09:23:06.623 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/202 2024-03-04T09:23:06.663 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/202 2024-03-04T09:23:06.685 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/202 2024-03-04T09:23:06.714 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/202 2024-03-04T09:23:06.752 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/202 2024-03-04T09:23:06.875 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/202 2024-03-04T09:23:06.914 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-ceph-common-2:19.0.0-1739.g1bc250c7.el9. 39/202 2024-03-04T09:23:06.968 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-ceph-argparse-2:19.0.0-1739.g1bc250c7.el 40/202 2024-03-04T09:23:07.003 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-cephfs-2:19.0.0-1739.g1bc250c7.el9.x86_6 41/202 2024-03-04T09:23:07.127 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/202 2024-03-04T09:23:07.191 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/202 2024-03-04T09:23:07.305 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/202 2024-03-04T09:23:07.390 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/202 2024-03-04T09:23:07.481 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/202 2024-03-04T09:23:07.488 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/202 2024-03-04T09:23:07.568 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/202 2024-03-04T09:23:07.620 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/202 2024-03-04T09:23:07.667 INFO:teuthology.orchestra.run.smithi146.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/202 2024-03-04T09:23:07.694 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/202 2024-03-04T09:23:07.773 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/202 2024-03-04T09:23:07.834 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : librados-devel-2:19.0.0-1739.g1bc250c7.el9.x86_6 52/202 2024-03-04T09:23:07.881 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 53/202 2024-03-04T09:23:08.053 INFO:teuthology.orchestra.run.smithi146.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 54/202 2024-03-04T09:23:08.096 INFO:teuthology.orchestra.run.smithi146.stdout: Upgrading : iproute-6.7.0-2.el9.x86_64 55/202 2024-03-04T09:23:08.291 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/202 2024-03-04T09:23:08.403 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/202 2024-03-04T09:23:08.520 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/202 2024-03-04T09:23:08.558 INFO:teuthology.orchestra.run.smithi146.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 56/202 2024-03-04T09:23:08.606 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/202 2024-03-04T09:23:08.631 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/202 2024-03-04T09:23:08.688 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/202 2024-03-04T09:23:08.740 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : librados-devel-2:19.0.0-1739.g1bc250c7.el9.x86_6 52/202 2024-03-04T09:23:08.788 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 53/202 2024-03-04T09:23:08.996 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 54/202 2024-03-04T09:23:08.996 INFO:teuthology.orchestra.run.smithi146.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 57/202 2024-03-04T09:23:09.054 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : iproute-6.7.0-2.el9.x86_64 55/202 2024-03-04T09:23:09.095 INFO:teuthology.orchestra.run.smithi146.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 58/202 2024-03-04T09:23:09.163 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 59/202 2024-03-04T09:23:09.412 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 60/202 2024-03-04T09:23:09.414 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 56/202 2024-03-04T09:23:09.522 INFO:teuthology.orchestra.run.smithi146.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 61/202 2024-03-04T09:23:09.601 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 62/202 2024-03-04T09:23:09.621 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 57/202 2024-03-04T09:23:09.703 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 63/202 2024-03-04T09:23:09.729 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 58/202 2024-03-04T09:23:09.738 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 64/202 2024-03-04T09:23:09.777 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 65/202 2024-03-04T09:23:09.861 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 66/202 2024-03-04T09:23:09.902 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 67/202 2024-03-04T09:23:09.966 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 68/202 2024-03-04T09:23:10.033 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 69/202 2024-03-04T09:23:10.089 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 59/202 2024-03-04T09:23:10.223 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 70/202 2024-03-04T09:23:10.301 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 71/202 2024-03-04T09:23:10.331 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 60/202 2024-03-04T09:23:10.338 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 72/202 2024-03-04T09:23:10.397 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 73/202 2024-03-04T09:23:10.412 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 61/202 2024-03-04T09:23:10.498 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 62/202 2024-03-04T09:23:10.535 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 74/202 2024-03-04T09:23:10.569 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 75/202 2024-03-04T09:23:10.597 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 63/202 2024-03-04T09:23:10.614 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 76/202 2024-03-04T09:23:10.636 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 64/202 2024-03-04T09:23:10.652 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 77/202 2024-03-04T09:23:10.676 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 78/202 2024-03-04T09:23:10.676 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 65/202 2024-03-04T09:23:10.748 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 66/202 2024-03-04T09:23:10.792 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 67/202 2024-03-04T09:23:10.855 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 79/202 2024-03-04T09:23:10.856 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 68/202 2024-03-04T09:23:10.933 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 69/202 2024-03-04T09:23:11.038 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 80/202 2024-03-04T09:23:11.141 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 70/202 2024-03-04T09:23:11.182 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 81/202 2024-03-04T09:23:11.198 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 71/202 2024-03-04T09:23:11.236 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 72/202 2024-03-04T09:23:11.254 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 82/202 2024-03-04T09:23:11.296 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 73/202 2024-03-04T09:23:11.318 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-ply-3.11-14.el9.noarch 83/202 2024-03-04T09:23:11.391 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 84/202 2024-03-04T09:23:11.442 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 74/202 2024-03-04T09:23:11.484 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 75/202 2024-03-04T09:23:11.536 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 76/202 2024-03-04T09:23:11.567 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 77/202 2024-03-04T09:23:11.591 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 78/202 2024-03-04T09:23:11.646 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 85/202 2024-03-04T09:23:11.725 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 86/202 2024-03-04T09:23:11.748 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 79/202 2024-03-04T09:23:11.896 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 87/202 2024-03-04T09:23:11.937 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 80/202 2024-03-04T09:23:11.938 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 88/202 2024-03-04T09:23:11.979 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 89/202 2024-03-04T09:23:12.019 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 90/202 2024-03-04T09:23:12.058 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : jq-1.6-16.el9.x86_64 91/202 2024-03-04T09:23:12.080 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 81/202 2024-03-04T09:23:12.095 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : fuse-2.9.9-15.el9.x86_64 92/202 2024-03-04T09:23:12.127 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 82/202 2024-03-04T09:23:12.162 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 93/202 2024-03-04T09:23:12.189 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-ply-3.11-14.el9.noarch 83/202 2024-03-04T09:23:12.201 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 94/202 2024-03-04T09:23:12.265 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 84/202 2024-03-04T09:23:12.266 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : uuid-1.6.2-55.el9.x86_64 95/202 2024-03-04T09:23:12.322 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 96/202 2024-03-04T09:23:12.363 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 97/202 2024-03-04T09:23:12.489 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 98/202 2024-03-04T09:23:12.519 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 85/202 2024-03-04T09:23:12.533 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 99/202 2024-03-04T09:23:12.574 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 100/202 2024-03-04T09:23:12.581 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 86/202 2024-03-04T09:23:12.656 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 101/202 2024-03-04T09:23:12.696 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 102/202 2024-03-04T09:23:12.757 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 87/202 2024-03-04T09:23:12.802 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 88/202 2024-03-04T09:23:12.840 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 103/202 2024-03-04T09:23:12.842 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 89/202 2024-03-04T09:23:12.883 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 90/202 2024-03-04T09:23:12.924 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : jq-1.6-16.el9.x86_64 91/202 2024-03-04T09:23:12.954 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 104/202 2024-03-04T09:23:12.959 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : fuse-2.9.9-15.el9.x86_64 92/202 2024-03-04T09:23:13.002 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 93/202 2024-03-04T09:23:13.008 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 105/202 2024-03-04T09:23:13.053 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 106/202 2024-03-04T09:23:13.073 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 94/202 2024-03-04T09:23:13.135 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 107/202 2024-03-04T09:23:13.143 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : uuid-1.6.2-55.el9.x86_64 95/202 2024-03-04T09:23:13.199 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 96/202 2024-03-04T09:23:13.245 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 97/202 2024-03-04T09:23:13.275 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-packaging-20.9-5.el9.noarch 108/202 2024-03-04T09:23:13.343 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 109/202 2024-03-04T09:23:13.377 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 98/202 2024-03-04T09:23:13.423 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 99/202 2024-03-04T09:23:13.455 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 100/202 2024-03-04T09:23:13.513 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 101/202 2024-03-04T09:23:13.553 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 102/202 2024-03-04T09:23:13.690 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 103/202 2024-03-04T09:23:13.811 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 104/202 2024-03-04T09:23:13.866 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 105/202 2024-03-04T09:23:13.918 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 106/202 2024-03-04T09:23:14.089 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 107/202 2024-03-04T09:23:14.247 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-packaging-20.9-5.el9.noarch 108/202 2024-03-04T09:23:14.306 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 110/202 2024-03-04T09:23:14.341 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 109/202 2024-03-04T09:23:14.367 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 111/202 2024-03-04T09:23:14.400 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 112/202 2024-03-04T09:23:14.458 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 113/202 2024-03-04T09:23:14.577 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 114/202 2024-03-04T09:23:14.618 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 115/202 2024-03-04T09:23:14.846 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : openblas-0.3.21-2.el9.x86_64 116/202 2024-03-04T09:23:14.873 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 117/202 2024-03-04T09:23:14.965 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 118/202 2024-03-04T09:23:15.363 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 110/202 2024-03-04T09:23:15.443 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 111/202 2024-03-04T09:23:15.476 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 112/202 2024-03-04T09:23:15.542 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 113/202 2024-03-04T09:23:15.662 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 114/202 2024-03-04T09:23:15.703 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 115/202 2024-03-04T09:23:15.768 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 119/202 2024-03-04T09:23:15.901 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 120/202 2024-03-04T09:23:15.948 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : openblas-0.3.21-2.el9.x86_64 116/202 2024-03-04T09:23:15.980 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 117/202 2024-03-04T09:23:16.080 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 118/202 2024-03-04T09:23:16.910 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 119/202 2024-03-04T09:23:17.047 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 120/202 2024-03-04T09:23:17.417 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 121/202 2024-03-04T09:23:17.511 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 122/202 2024-03-04T09:23:17.546 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 123/202 2024-03-04T09:23:17.904 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 124/202 2024-03-04T09:23:17.932 INFO:teuthology.orchestra.run.smithi146.stdout: Upgrading : librbd1-2:19.0.0-1739.g1bc250c7.el9.x86_64 125/202 2024-03-04T09:23:18.051 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: librbd1-2:19.0.0-1739.g1bc250c7.el9.x86_64 125/202 2024-03-04T09:23:18.079 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-rbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 126/202 2024-03-04T09:23:18.134 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 127/202 2024-03-04T09:23:18.680 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 128/202 2024-03-04T09:23:18.683 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 121/202 2024-03-04T09:23:18.710 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : librgw2-2:19.0.0-1739.g1bc250c7.el9.x86_64 129/202 2024-03-04T09:23:18.769 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 122/202 2024-03-04T09:23:18.796 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 123/202 2024-03-04T09:23:18.801 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: librgw2-2:19.0.0-1739.g1bc250c7.el9.x86_64 129/202 2024-03-04T09:23:18.828 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-rgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 130/202 2024-03-04T09:23:19.165 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 124/202 2024-03-04T09:23:19.198 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : librbd1-2:19.0.0-1739.g1bc250c7.el9.x86_64 125/202 2024-03-04T09:23:19.319 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: librbd1-2:19.0.0-1739.g1bc250c7.el9.x86_64 125/202 2024-03-04T09:23:19.354 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-rbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 126/202 2024-03-04T09:23:19.410 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 127/202 2024-03-04T09:23:19.944 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 128/202 2024-03-04T09:23:19.976 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : librgw2-2:19.0.0-1739.g1bc250c7.el9.x86_64 129/202 2024-03-04T09:23:20.076 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: librgw2-2:19.0.0-1739.g1bc250c7.el9.x86_64 129/202 2024-03-04T09:23:20.128 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-rgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 130/202 2024-03-04T09:23:21.418 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 131/202 2024-03-04T09:23:21.458 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 131/202 2024-03-04T09:23:22.079 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 131/202 2024-03-04T09:23:22.120 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-base-2:19.0.0-1739.g1bc250c7.el9.x86_64 132/202 2024-03-04T09:23:22.256 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-base-2:19.0.0-1739.g1bc250c7.el9.x86_64 132/202 2024-03-04T09:23:22.256 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-04T09:23:22.256 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-04T09:23:22.256 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:23:22.296 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-selinux-2:19.0.0-1739.g1bc250c7.el9.x86_64 133/202 2024-03-04T09:23:22.791 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 131/202 2024-03-04T09:23:22.824 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 131/202 2024-03-04T09:23:23.478 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 131/202 2024-03-04T09:23:23.519 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-base-2:19.0.0-1739.g1bc250c7.el9.x86_64 132/202 2024-03-04T09:23:23.685 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-base-2:19.0.0-1739.g1bc250c7.el9.x86_64 132/202 2024-03-04T09:23:23.685 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-04T09:23:23.685 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-04T09:23:23.685 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:23:23.729 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-selinux-2:19.0.0-1739.g1bc250c7.el9.x86_64 133/202 2024-03-04T09:23:35.122 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1739.g1bc250c7.el9.x86_64 133/202 2024-03-04T09:23:35.122 INFO:teuthology.orchestra.run.smithi146.stdout:skipping the directory /sys 2024-03-04T09:23:35.123 INFO:teuthology.orchestra.run.smithi146.stdout:skipping the directory /proc 2024-03-04T09:23:35.123 INFO:teuthology.orchestra.run.smithi146.stdout:skipping the directory /mnt 2024-03-04T09:23:35.123 INFO:teuthology.orchestra.run.smithi146.stdout:skipping the directory /var/tmp 2024-03-04T09:23:35.123 INFO:teuthology.orchestra.run.smithi146.stdout:skipping the directory /home 2024-03-04T09:23:35.123 INFO:teuthology.orchestra.run.smithi146.stdout:skipping the directory /root 2024-03-04T09:23:35.123 INFO:teuthology.orchestra.run.smithi146.stdout:skipping the directory /tmp 2024-03-04T09:23:35.123 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:23:35.375 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-volume-2:19.0.0-1739.g1bc250c7.el9.noarch 134/202 2024-03-04T09:23:35.435 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-volume-2:19.0.0-1739.g1bc250c7.el9.noarch 134/202 2024-03-04T09:23:35.435 INFO:teuthology.orchestra.run.smithi146.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T09:23:35.435 INFO:teuthology.orchestra.run.smithi146.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-04T09:23:35.436 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:23:37.059 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-osd-2:19.0.0-1739.g1bc250c7.el9.x86_64 135/202 2024-03-04T09:23:37.162 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-osd-2:19.0.0-1739.g1bc250c7.el9.x86_64 135/202 2024-03-04T09:23:37.162 INFO:teuthology.orchestra.run.smithi146.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T09:23:37.163 INFO:teuthology.orchestra.run.smithi146.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-04T09:23:37.163 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-04T09:23:37.163 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-04T09:23:37.163 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:23:37.401 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-mds-2:19.0.0-1739.g1bc250c7.el9.x86_64 136/202 2024-03-04T09:23:37.490 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-mds-2:19.0.0-1739.g1bc250c7.el9.x86_64 136/202 2024-03-04T09:23:37.490 INFO:teuthology.orchestra.run.smithi146.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T09:23:37.490 INFO:teuthology.orchestra.run.smithi146.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-04T09:23:37.490 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-04T09:23:37.490 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-04T09:23:37.490 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:23:37.735 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1739.g1bc250c7.el9.x86_64 133/202 2024-03-04T09:23:37.735 INFO:teuthology.orchestra.run.smithi006.stdout:skipping the directory /sys 2024-03-04T09:23:37.735 INFO:teuthology.orchestra.run.smithi006.stdout:skipping the directory /proc 2024-03-04T09:23:37.735 INFO:teuthology.orchestra.run.smithi006.stdout:skipping the directory /mnt 2024-03-04T09:23:37.735 INFO:teuthology.orchestra.run.smithi006.stdout:skipping the directory /var/tmp 2024-03-04T09:23:37.735 INFO:teuthology.orchestra.run.smithi006.stdout:skipping the directory /home 2024-03-04T09:23:37.735 INFO:teuthology.orchestra.run.smithi006.stdout:skipping the directory /root 2024-03-04T09:23:37.735 INFO:teuthology.orchestra.run.smithi006.stdout:skipping the directory /tmp 2024-03-04T09:23:37.735 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:23:37.968 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-mon-2:19.0.0-1739.g1bc250c7.el9.x86_64 137/202 2024-03-04T09:23:37.995 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-volume-2:19.0.0-1739.g1bc250c7.el9.noarch 134/202 2024-03-04T09:23:38.046 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-volume-2:19.0.0-1739.g1bc250c7.el9.noarch 134/202 2024-03-04T09:23:38.046 INFO:teuthology.orchestra.run.smithi006.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T09:23:38.046 INFO:teuthology.orchestra.run.smithi006.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-04T09:23:38.047 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:23:38.063 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-mon-2:19.0.0-1739.g1bc250c7.el9.x86_64 137/202 2024-03-04T09:23:38.063 INFO:teuthology.orchestra.run.smithi146.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T09:23:38.064 INFO:teuthology.orchestra.run.smithi146.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-04T09:23:38.064 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-04T09:23:38.064 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-04T09:23:38.064 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:23:38.178 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : automake-1.16.2-8.el9.noarch 138/202 2024-03-04T09:23:38.700 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 139/202 2024-03-04T09:23:38.729 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 139/202 2024-03-04T09:23:38.760 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1739.g1bc250c7.e 140/202 2024-03-04T09:23:38.857 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1739.g1bc250c7. 141/202 2024-03-04T09:23:38.900 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1739.g1bc250c7.el9.noa 142/202 2024-03-04T09:23:39.226 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1739.g1bc250c7.el9.noa 142/202 2024-03-04T09:23:39.267 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1739.g1bc250c7.el9.n 143/202 2024-03-04T09:23:39.765 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-osd-2:19.0.0-1739.g1bc250c7.el9.x86_64 135/202 2024-03-04T09:23:39.847 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-osd-2:19.0.0-1739.g1bc250c7.el9.x86_64 135/202 2024-03-04T09:23:39.848 INFO:teuthology.orchestra.run.smithi006.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T09:23:39.848 INFO:teuthology.orchestra.run.smithi006.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-04T09:23:39.848 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-04T09:23:39.848 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-04T09:23:39.848 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:23:40.098 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-mds-2:19.0.0-1739.g1bc250c7.el9.x86_64 136/202 2024-03-04T09:23:40.173 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-mds-2:19.0.0-1739.g1bc250c7.el9.x86_64 136/202 2024-03-04T09:23:40.173 INFO:teuthology.orchestra.run.smithi006.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T09:23:40.173 INFO:teuthology.orchestra.run.smithi006.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-04T09:23:40.173 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-04T09:23:40.173 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-04T09:23:40.173 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:23:40.384 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1739.g1bc250c7.el9.n 143/202 2024-03-04T09:23:40.417 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1739.g1bc 144/202 2024-03-04T09:23:40.559 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1739.g1bc 144/202 2024-03-04T09:23:40.699 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-mon-2:19.0.0-1739.g1bc250c7.el9.x86_64 137/202 2024-03-04T09:23:40.731 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1739.g1bc250c7.el 145/202 2024-03-04T09:23:40.769 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-mgr-2:19.0.0-1739.g1bc250c7.el9.x86_64 146/202 2024-03-04T09:23:40.797 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-mon-2:19.0.0-1739.g1bc250c7.el9.x86_64 137/202 2024-03-04T09:23:40.797 INFO:teuthology.orchestra.run.smithi006.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T09:23:40.797 INFO:teuthology.orchestra.run.smithi006.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-04T09:23:40.797 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-04T09:23:40.798 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-04T09:23:40.798 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:23:40.863 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1739.g1bc250c7.el9.x86_64 146/202 2024-03-04T09:23:40.863 INFO:teuthology.orchestra.run.smithi146.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T09:23:40.863 INFO:teuthology.orchestra.run.smithi146.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-04T09:23:40.863 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-04T09:23:40.864 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-04T09:23:40.864 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:23:40.920 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-mgr-rook-2:19.0.0-1739.g1bc250c7.el9.noarch 147/202 2024-03-04T09:23:40.926 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : automake-1.16.2-8.el9.noarch 138/202 2024-03-04T09:23:40.959 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1739.g1bc250c7.el9.noarch 147/202 2024-03-04T09:23:41.029 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-2:19.0.0-1739.g1bc250c7.el9.x86_64 148/202 2024-03-04T09:23:41.524 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 139/202 2024-03-04T09:23:41.552 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 139/202 2024-03-04T09:23:41.601 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1739.g1bc250c7.e 140/202 2024-03-04T09:23:41.705 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1739.g1bc250c7. 141/202 2024-03-04T09:23:41.761 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1739.g1bc250c7.el9.noa 142/202 2024-03-04T09:23:41.959 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libtool-2.4.6-45.el9.x86_64 149/202 2024-03-04T09:23:41.988 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-radosgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 150/202 2024-03-04T09:23:42.081 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 150/202 2024-03-04T09:23:42.081 INFO:teuthology.orchestra.run.smithi146.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T09:23:42.081 INFO:teuthology.orchestra.run.smithi146.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-04T09:23:42.082 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-04T09:23:42.082 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-04T09:23:42.082 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:23:42.098 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1739.g1bc250c7.el9.noa 142/202 2024-03-04T09:23:42.131 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1739.g1bc250c7.el9.n 143/202 2024-03-04T09:23:42.156 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1739.g1bc25 151/202 2024-03-04T09:23:42.251 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1739.g1bc25 151/202 2024-03-04T09:23:42.251 INFO:teuthology.orchestra.run.smithi146.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T09:23:42.251 INFO:teuthology.orchestra.run.smithi146.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-04T09:23:42.251 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:23:42.332 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : cephfs-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 152/202 2024-03-04T09:23:42.436 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 152/202 2024-03-04T09:23:42.436 INFO:teuthology.orchestra.run.smithi146.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T09:23:42.436 INFO:teuthology.orchestra.run.smithi146.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-04T09:23:42.436 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-04T09:23:42.437 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-04T09:23:42.437 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:23:42.832 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : rbd-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 153/202 2024-03-04T09:23:42.929 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 153/202 2024-03-04T09:23:42.929 INFO:teuthology.orchestra.run.smithi146.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T09:23:42.930 INFO:teuthology.orchestra.run.smithi146.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-04T09:23:42.930 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-04T09:23:42.930 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-04T09:23:42.930 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:23:43.301 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1739.g1bc250c7.el9.n 143/202 2024-03-04T09:23:43.332 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1739.g1bc 144/202 2024-03-04T09:23:43.480 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1739.g1bc 144/202 2024-03-04T09:23:43.662 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1739.g1bc250c7.el 145/202 2024-03-04T09:23:43.691 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-mgr-2:19.0.0-1739.g1bc250c7.el9.x86_64 146/202 2024-03-04T09:23:43.786 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1739.g1bc250c7.el9.x86_64 146/202 2024-03-04T09:23:43.786 INFO:teuthology.orchestra.run.smithi006.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T09:23:43.786 INFO:teuthology.orchestra.run.smithi006.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-04T09:23:43.786 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-04T09:23:43.786 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-04T09:23:43.786 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:23:43.851 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-mgr-rook-2:19.0.0-1739.g1bc250c7.el9.noarch 147/202 2024-03-04T09:23:43.891 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1739.g1bc250c7.el9.noarch 147/202 2024-03-04T09:23:44.107 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-2:19.0.0-1739.g1bc250c7.el9.x86_64 148/202 2024-03-04T09:23:45.151 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libtool-2.4.6-45.el9.x86_64 149/202 2024-03-04T09:23:45.186 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-radosgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 150/202 2024-03-04T09:23:45.259 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 150/202 2024-03-04T09:23:45.259 INFO:teuthology.orchestra.run.smithi006.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T09:23:45.259 INFO:teuthology.orchestra.run.smithi006.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-04T09:23:45.259 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-04T09:23:45.260 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-04T09:23:45.260 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:23:45.320 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1739.g1bc25 151/202 2024-03-04T09:23:45.396 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1739.g1bc25 151/202 2024-03-04T09:23:45.396 INFO:teuthology.orchestra.run.smithi006.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T09:23:45.396 INFO:teuthology.orchestra.run.smithi006.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-04T09:23:45.396 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:23:45.480 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : cephfs-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 152/202 2024-03-04T09:23:45.562 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 152/202 2024-03-04T09:23:45.562 INFO:teuthology.orchestra.run.smithi006.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T09:23:45.562 INFO:teuthology.orchestra.run.smithi006.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-04T09:23:45.562 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-04T09:23:45.563 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-04T09:23:45.563 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:23:45.963 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : rbd-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 153/202 2024-03-04T09:23:46.064 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 153/202 2024-03-04T09:23:46.064 INFO:teuthology.orchestra.run.smithi006.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T09:23:46.064 INFO:teuthology.orchestra.run.smithi006.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-04T09:23:46.064 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-04T09:23:46.064 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-04T09:23:46.064 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:23:47.696 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-test-2:19.0.0-1739.g1bc250c7.el9.x86_64 154/202 2024-03-04T09:23:47.736 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : rbd-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 155/202 2024-03-04T09:23:47.768 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : rbd-nbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 156/202 2024-03-04T09:23:47.839 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 157/202 2024-03-04T09:23:48.105 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : cephfs-shell-2:19.0.0-1739.g1bc250c7.el9.x86_64 158/202 2024-03-04T09:23:48.231 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 159/202 2024-03-04T09:23:48.308 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 160/202 2024-03-04T09:23:48.368 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 161/202 2024-03-04T09:23:48.419 INFO:teuthology.orchestra.run.smithi146.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 162/202 2024-03-04T09:23:48.533 INFO:teuthology.orchestra.run.smithi146.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 163/202 2024-03-04T09:23:48.571 INFO:teuthology.orchestra.run.smithi146.stdout: Upgrading : iproute-tc-6.7.0-2.el9.x86_64 164/202 2024-03-04T09:23:48.615 INFO:teuthology.orchestra.run.smithi146.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 165/202 2024-03-04T09:23:48.644 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libcephfs-devel-2:19.0.0-1739.g1bc250c7.el9.x86_ 166/202 2024-03-04T09:23:48.696 INFO:teuthology.orchestra.run.smithi146.stdout: Upgrading : NetworkManager-team-1:1.46.0-1.el9.x86_64 167/202 2024-03-04T09:23:48.849 INFO:teuthology.orchestra.run.smithi146.stdout: Upgrading : NetworkManager-tui-1:1.46.0-1.el9.x86_64 168/202 2024-03-04T09:23:48.918 INFO:teuthology.orchestra.run.smithi146.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 169/202 2024-03-04T09:23:48.963 INFO:teuthology.orchestra.run.smithi146.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 170/202 2024-03-04T09:23:49.813 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 171/202 2024-03-04T09:23:49.860 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 171/202 2024-03-04T09:23:49.926 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 171/202 2024-03-04T09:23:51.111 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-test-2:19.0.0-1739.g1bc250c7.el9.x86_64 154/202 2024-03-04T09:23:51.153 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : rbd-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 155/202 2024-03-04T09:23:51.183 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : rbd-nbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 156/202 2024-03-04T09:23:51.221 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 157/202 2024-03-04T09:23:51.237 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : cephfs-top-2:19.0.0-1739.g1bc250c7.el9.noarch 172/202 2024-03-04T09:23:51.306 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 173/202 2024-03-04T09:23:51.356 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 174/202 2024-03-04T09:23:51.481 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : flex-2.6.4-9.el9.x86_64 175/202 2024-03-04T09:23:51.513 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : cephfs-shell-2:19.0.0-1739.g1bc250c7.el9.x86_64 158/202 2024-03-04T09:23:51.516 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : bison-3.7.4-5.el9.x86_64 176/202 2024-03-04T09:23:51.574 INFO:teuthology.orchestra.run.smithi146.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 177/202 2024-03-04T09:23:51.617 INFO:teuthology.orchestra.run.smithi146.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 178/202 2024-03-04T09:23:51.642 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 159/202 2024-03-04T09:23:51.659 INFO:teuthology.orchestra.run.smithi146.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 179/202 2024-03-04T09:23:51.659 INFO:teuthology.orchestra.run.smithi146.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 180/202 2024-03-04T09:23:51.694 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 160/202 2024-03-04T09:23:51.754 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 161/202 2024-03-04T09:23:51.792 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 180/202 2024-03-04T09:23:51.818 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 162/202 2024-03-04T09:23:51.843 INFO:teuthology.orchestra.run.smithi146.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 181/202 2024-03-04T09:23:51.888 INFO:teuthology.orchestra.run.smithi146.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 182/202 2024-03-04T09:23:51.888 INFO:teuthology.orchestra.run.smithi146.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 183/202 2024-03-04T09:23:51.940 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 163/202 2024-03-04T09:23:51.987 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : iproute-tc-6.7.0-2.el9.x86_64 164/202 2024-03-04T09:23:52.035 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 165/202 2024-03-04T09:23:52.043 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 183/202 2024-03-04T09:23:52.069 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libcephfs-devel-2:19.0.0-1739.g1bc250c7.el9.x86_ 166/202 2024-03-04T09:23:52.096 INFO:teuthology.orchestra.run.smithi146.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 184/202 2024-03-04T09:23:52.122 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : NetworkManager-team-1:1.46.0-1.el9.x86_64 167/202 2024-03-04T09:23:52.152 INFO:teuthology.orchestra.run.smithi146.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 185/202 2024-03-04T09:23:52.191 INFO:teuthology.orchestra.run.smithi146.stdout: Cleanup : iproute-tc-6.2.0-5.el9.x86_64 186/202 2024-03-04T09:23:52.237 INFO:teuthology.orchestra.run.smithi146.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 187/202 2024-03-04T09:23:52.240 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : NetworkManager-tui-1:1.46.0-1.el9.x86_64 168/202 2024-03-04T09:23:52.284 INFO:teuthology.orchestra.run.smithi146.stdout: Cleanup : iproute-6.2.0-5.el9.x86_64 188/202 2024-03-04T09:23:52.308 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 169/202 2024-03-04T09:23:52.345 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 170/202 2024-03-04T09:23:52.353 INFO:teuthology.orchestra.run.smithi146.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 189/202 2024-03-04T09:23:52.429 INFO:teuthology.orchestra.run.smithi146.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 190/202 2024-03-04T09:23:52.501 INFO:teuthology.orchestra.run.smithi146.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 191/202 2024-03-04T09:23:52.538 INFO:teuthology.orchestra.run.smithi146.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 192/202 2024-03-04T09:23:52.571 INFO:teuthology.orchestra.run.smithi146.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 193/202 2024-03-04T09:23:52.605 INFO:teuthology.orchestra.run.smithi146.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 194/202 2024-03-04T09:23:52.638 INFO:teuthology.orchestra.run.smithi146.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 195/202 2024-03-04T09:23:52.675 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 196/202 2024-03-04T09:23:52.675 INFO:teuthology.orchestra.run.smithi146.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 196/202 2024-03-04T09:23:52.996 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 196/202 2024-03-04T09:23:52.997 INFO:teuthology.orchestra.run.smithi146.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 197/202 2024-03-04T09:23:53.128 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 197/202 2024-03-04T09:23:53.179 INFO:teuthology.orchestra.run.smithi146.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 198/202 2024-03-04T09:23:53.196 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 171/202 2024-03-04T09:23:53.208 INFO:teuthology.orchestra.run.smithi146.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 199/202 2024-03-04T09:23:53.240 INFO:teuthology.orchestra.run.smithi146.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 200/202 2024-03-04T09:23:53.242 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 171/202 2024-03-04T09:23:53.277 INFO:teuthology.orchestra.run.smithi146.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 201/202 2024-03-04T09:23:53.312 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 171/202 2024-03-04T09:23:53.403 INFO:teuthology.orchestra.run.smithi146.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 202/202 2024-03-04T09:23:53.669 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: openssl-libs-1:3.0.7-24.el9.x86_64 202/202 2024-03-04T09:23:53.669 INFO:teuthology.orchestra.run.smithi146.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-03-04T09:23:53.669 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:23:54.730 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : cephfs-top-2:19.0.0-1739.g1bc250c7.el9.noarch 172/202 2024-03-04T09:23:54.815 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 173/202 2024-03-04T09:23:54.871 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 174/202 2024-03-04T09:23:55.005 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : flex-2.6.4-9.el9.x86_64 175/202 2024-03-04T09:23:55.041 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : bison-3.7.4-5.el9.x86_64 176/202 2024-03-04T09:23:55.083 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 177/202 2024-03-04T09:23:55.118 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 178/202 2024-03-04T09:23:55.159 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 179/202 2024-03-04T09:23:55.159 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 180/202 2024-03-04T09:23:55.308 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 180/202 2024-03-04T09:23:55.359 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 181/202 2024-03-04T09:23:55.404 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 182/202 2024-03-04T09:23:55.405 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 183/202 2024-03-04T09:23:55.543 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 183/202 2024-03-04T09:23:55.587 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 184/202 2024-03-04T09:23:55.644 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 185/202 2024-03-04T09:23:55.683 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : iproute-tc-6.2.0-5.el9.x86_64 186/202 2024-03-04T09:23:55.729 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 187/202 2024-03-04T09:23:55.775 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : iproute-6.2.0-5.el9.x86_64 188/202 2024-03-04T09:23:55.843 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 189/202 2024-03-04T09:23:55.913 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 190/202 2024-03-04T09:23:55.963 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 191/202 2024-03-04T09:23:55.996 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 192/202 2024-03-04T09:23:56.030 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 193/202 2024-03-04T09:23:56.063 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 194/202 2024-03-04T09:23:56.096 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 195/202 2024-03-04T09:23:56.134 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 196/202 2024-03-04T09:23:56.134 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 196/202 2024-03-04T09:23:56.441 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 196/202 2024-03-04T09:23:56.442 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 197/202 2024-03-04T09:23:56.580 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 197/202 2024-03-04T09:23:56.632 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 198/202 2024-03-04T09:23:56.661 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 199/202 2024-03-04T09:23:56.693 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 200/202 2024-03-04T09:23:56.730 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 201/202 2024-03-04T09:23:56.860 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 202/202 2024-03-04T09:23:57.130 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: openssl-libs-1:3.0.7-24.el9.x86_64 202/202 2024-03-04T09:23:57.130 INFO:teuthology.orchestra.run.smithi006.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-03-04T09:23:57.130 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:23:58.477 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-2:19.0.0-1739.g1bc250c7.el9.x86_64 1/202 2024-03-04T09:23:58.477 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-base-2:19.0.0-1739.g1bc250c7.el9.x86_64 2/202 2024-03-04T09:23:58.477 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 3/202 2024-03-04T09:23:58.477 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 4/202 2024-03-04T09:23:58.478 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1739.g1bc25 5/202 2024-03-04T09:23:58.478 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-mds-2:19.0.0-1739.g1bc250c7.el9.x86_64 6/202 2024-03-04T09:23:58.478 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-mgr-2:19.0.0-1739.g1bc250c7.el9.x86_64 7/202 2024-03-04T09:23:58.478 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-mon-2:19.0.0-1739.g1bc250c7.el9.x86_64 8/202 2024-03-04T09:23:58.478 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-osd-2:19.0.0-1739.g1bc250c7.el9.x86_64 9/202 2024-03-04T09:23:58.478 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-radosgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 10/202 2024-03-04T09:23:58.478 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-selinux-2:19.0.0-1739.g1bc250c7.el9.x86_64 11/202 2024-03-04T09:23:58.478 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-test-2:19.0.0-1739.g1bc250c7.el9.x86_64 12/202 2024-03-04T09:23:58.478 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : cephfs-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 13/202 2024-03-04T09:23:58.478 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : cephfs-shell-2:19.0.0-1739.g1bc250c7.el9.x86_64 14/202 2024-03-04T09:23:58.478 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libcephfs-devel-2:19.0.0-1739.g1bc250c7.el9.x86_ 15/202 2024-03-04T09:23:58.478 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libcephfs2-2:19.0.0-1739.g1bc250c7.el9.x86_64 16/202 2024-03-04T09:23:58.478 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libcephsqlite-2:19.0.0-1739.g1bc250c7.el9.x86_64 17/202 2024-03-04T09:23:58.478 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : librados-devel-2:19.0.0-1739.g1bc250c7.el9.x86_6 18/202 2024-03-04T09:23:58.479 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libradosstriper1-2:19.0.0-1739.g1bc250c7.el9.x86 19/202 2024-03-04T09:23:58.479 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : librgw2-2:19.0.0-1739.g1bc250c7.el9.x86_64 20/202 2024-03-04T09:23:58.479 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1739.g1bc250c7.el 21/202 2024-03-04T09:23:58.479 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-ceph-common-2:19.0.0-1739.g1bc250c7.el9. 22/202 2024-03-04T09:23:58.479 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-cephfs-2:19.0.0-1739.g1bc250c7.el9.x86_6 23/202 2024-03-04T09:23:58.479 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-rados-2:19.0.0-1739.g1bc250c7.el9.x86_64 24/202 2024-03-04T09:23:58.479 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-rbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 25/202 2024-03-04T09:23:58.479 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-rgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 26/202 2024-03-04T09:23:58.479 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : rbd-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 27/202 2024-03-04T09:23:58.479 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : rbd-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 28/202 2024-03-04T09:23:58.479 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : rbd-nbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 29/202 2024-03-04T09:23:58.479 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1739.g1bc250c7. 30/202 2024-03-04T09:23:58.479 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1739.g1bc250c7.el9.noa 31/202 2024-03-04T09:23:58.480 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1739.g1bc250c7.el9.n 32/202 2024-03-04T09:23:58.480 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1739.g1bc 33/202 2024-03-04T09:23:58.480 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1739.g1bc250c7.el 34/202 2024-03-04T09:23:58.480 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1739.g1bc250c7.el9.noarch 35/202 2024-03-04T09:23:58.480 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1739.g1bc250c7.e 36/202 2024-03-04T09:23:58.480 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-volume-2:19.0.0-1739.g1bc250c7.el9.noarch 37/202 2024-03-04T09:23:58.480 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 38/202 2024-03-04T09:23:58.480 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : cephfs-top-2:19.0.0-1739.g1bc250c7.el9.noarch 39/202 2024-03-04T09:23:58.480 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/202 2024-03-04T09:23:58.480 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/202 2024-03-04T09:23:58.480 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/202 2024-03-04T09:23:58.480 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 43/202 2024-03-04T09:23:58.481 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : flex-2.6.4-9.el9.x86_64 44/202 2024-03-04T09:23:58.481 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 45/202 2024-03-04T09:23:58.481 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 46/202 2024-03-04T09:23:58.481 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 47/202 2024-03-04T09:23:58.481 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 48/202 2024-03-04T09:23:58.481 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 49/202 2024-03-04T09:23:58.481 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 50/202 2024-03-04T09:23:58.481 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 51/202 2024-03-04T09:23:58.481 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 52/202 2024-03-04T09:23:58.481 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 53/202 2024-03-04T09:23:58.481 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 54/202 2024-03-04T09:23:58.482 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 55/202 2024-03-04T09:23:58.482 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : lua-5.4.4-4.el9.x86_64 56/202 2024-03-04T09:23:58.482 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 57/202 2024-03-04T09:23:58.482 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 58/202 2024-03-04T09:23:58.482 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 59/202 2024-03-04T09:23:58.482 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : postgresql-13.11-1.el9.x86_64 60/202 2024-03-04T09:23:58.482 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 61/202 2024-03-04T09:23:58.482 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 62/202 2024-03-04T09:23:58.482 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 63/202 2024-03-04T09:23:58.482 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 64/202 2024-03-04T09:23:58.482 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 65/202 2024-03-04T09:23:58.482 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 66/202 2024-03-04T09:23:58.482 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 67/202 2024-03-04T09:23:58.483 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 68/202 2024-03-04T09:23:58.483 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 69/202 2024-03-04T09:23:58.483 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 70/202 2024-03-04T09:23:58.483 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 71/202 2024-03-04T09:23:58.483 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 72/202 2024-03-04T09:23:58.483 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 73/202 2024-03-04T09:23:58.483 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 74/202 2024-03-04T09:23:58.483 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 75/202 2024-03-04T09:23:58.483 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 76/202 2024-03-04T09:23:58.483 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 77/202 2024-03-04T09:23:58.483 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 78/202 2024-03-04T09:23:58.483 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 79/202 2024-03-04T09:23:58.483 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 80/202 2024-03-04T09:23:58.483 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 81/202 2024-03-04T09:23:58.484 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 82/202 2024-03-04T09:23:58.484 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 83/202 2024-03-04T09:23:58.484 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 84/202 2024-03-04T09:23:58.484 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 85/202 2024-03-04T09:23:58.484 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 86/202 2024-03-04T09:23:58.484 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 87/202 2024-03-04T09:23:58.484 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 88/202 2024-03-04T09:23:58.484 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 89/202 2024-03-04T09:23:58.484 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 90/202 2024-03-04T09:23:58.484 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 91/202 2024-03-04T09:23:58.484 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 92/202 2024-03-04T09:23:58.484 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 93/202 2024-03-04T09:23:58.484 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : jq-1.6-16.el9.x86_64 94/202 2024-03-04T09:23:58.484 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 95/202 2024-03-04T09:23:58.484 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 96/202 2024-03-04T09:23:58.485 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 97/202 2024-03-04T09:23:58.485 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 98/202 2024-03-04T09:23:58.485 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 99/202 2024-03-04T09:23:58.485 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-ply-3.11-14.el9.noarch 100/202 2024-03-04T09:23:58.485 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 101/202 2024-03-04T09:23:58.485 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 102/202 2024-03-04T09:23:58.485 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 103/202 2024-03-04T09:23:58.485 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 104/202 2024-03-04T09:23:58.485 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/202 2024-03-04T09:23:58.485 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/202 2024-03-04T09:23:58.485 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/202 2024-03-04T09:23:58.485 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/202 2024-03-04T09:23:58.485 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/202 2024-03-04T09:23:58.485 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/202 2024-03-04T09:23:58.485 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/202 2024-03-04T09:23:58.486 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/202 2024-03-04T09:23:58.486 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/202 2024-03-04T09:23:58.486 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/202 2024-03-04T09:23:58.486 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/202 2024-03-04T09:23:58.486 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 116/202 2024-03-04T09:23:58.486 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/202 2024-03-04T09:23:58.486 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/202 2024-03-04T09:23:58.486 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/202 2024-03-04T09:23:58.486 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/202 2024-03-04T09:23:58.486 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/202 2024-03-04T09:23:58.486 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/202 2024-03-04T09:23:58.486 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/202 2024-03-04T09:23:58.486 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 124/202 2024-03-04T09:23:58.486 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/202 2024-03-04T09:23:58.486 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/202 2024-03-04T09:23:58.486 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/202 2024-03-04T09:23:58.487 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/202 2024-03-04T09:23:58.487 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/202 2024-03-04T09:23:58.487 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/202 2024-03-04T09:23:58.487 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/202 2024-03-04T09:23:58.487 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/202 2024-03-04T09:23:58.487 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/202 2024-03-04T09:23:58.487 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/202 2024-03-04T09:23:58.487 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/202 2024-03-04T09:23:58.487 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/202 2024-03-04T09:23:58.487 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/202 2024-03-04T09:23:58.487 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/202 2024-03-04T09:23:58.487 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/202 2024-03-04T09:23:58.487 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/202 2024-03-04T09:23:58.487 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/202 2024-03-04T09:23:58.487 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/202 2024-03-04T09:23:58.487 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/202 2024-03-04T09:23:58.488 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/202 2024-03-04T09:23:58.488 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/202 2024-03-04T09:23:58.488 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/202 2024-03-04T09:23:58.488 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/202 2024-03-04T09:23:58.488 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/202 2024-03-04T09:23:58.488 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/202 2024-03-04T09:23:58.488 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/202 2024-03-04T09:23:58.488 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : librados2-2:19.0.0-1739.g1bc250c7.el9.x86_64 151/202 2024-03-04T09:23:58.488 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/202 2024-03-04T09:23:58.488 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : librbd1-2:19.0.0-1739.g1bc250c7.el9.x86_64 153/202 2024-03-04T09:23:58.488 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/202 2024-03-04T09:23:58.488 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 155/202 2024-03-04T09:23:58.489 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 156/202 2024-03-04T09:23:58.489 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : NetworkManager-1:1.46.0-1.el9.x86_64 157/202 2024-03-04T09:23:58.489 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/202 2024-03-04T09:23:58.489 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 159/202 2024-03-04T09:23:58.489 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/202 2024-03-04T09:23:58.489 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : NetworkManager-team-1:1.46.0-1.el9.x86_64 161/202 2024-03-04T09:23:58.489 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/202 2024-03-04T09:23:58.489 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : NetworkManager-tui-1:1.46.0-1.el9.x86_64 163/202 2024-03-04T09:23:58.489 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/202 2024-03-04T09:23:58.489 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/202 2024-03-04T09:23:58.489 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/202 2024-03-04T09:23:58.489 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/202 2024-03-04T09:23:58.489 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/202 2024-03-04T09:23:58.489 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/202 2024-03-04T09:23:58.490 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/202 2024-03-04T09:23:58.490 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/202 2024-03-04T09:23:58.490 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/202 2024-03-04T09:23:58.490 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : iproute-6.7.0-2.el9.x86_64 173/202 2024-03-04T09:23:58.490 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : iproute-6.2.0-5.el9.x86_64 174/202 2024-03-04T09:23:58.490 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : iproute-tc-6.7.0-2.el9.x86_64 175/202 2024-03-04T09:23:58.490 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : iproute-tc-6.2.0-5.el9.x86_64 176/202 2024-03-04T09:23:58.490 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 177/202 2024-03-04T09:23:58.490 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 178/202 2024-03-04T09:23:58.490 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 179/202 2024-03-04T09:23:58.490 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 180/202 2024-03-04T09:23:58.490 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 181/202 2024-03-04T09:23:58.490 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 182/202 2024-03-04T09:23:58.490 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 183/202 2024-03-04T09:23:58.490 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 184/202 2024-03-04T09:23:58.490 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 185/202 2024-03-04T09:23:58.491 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 186/202 2024-03-04T09:23:58.491 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 187/202 2024-03-04T09:23:58.491 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 188/202 2024-03-04T09:23:58.491 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 189/202 2024-03-04T09:23:58.491 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 190/202 2024-03-04T09:23:58.491 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 191/202 2024-03-04T09:23:58.491 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 192/202 2024-03-04T09:23:58.491 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 193/202 2024-03-04T09:23:58.491 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 194/202 2024-03-04T09:23:58.491 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 195/202 2024-03-04T09:23:58.491 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 196/202 2024-03-04T09:23:58.491 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 197/202 2024-03-04T09:23:58.491 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 198/202 2024-03-04T09:23:58.491 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 199/202 2024-03-04T09:23:58.491 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 200/202 2024-03-04T09:23:58.491 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 201/202 2024-03-04T09:24:01.177 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 202/202 2024-03-04T09:24:01.178 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:24:01.178 INFO:teuthology.orchestra.run.smithi146.stdout:Upgraded: 2024-03-04T09:24:01.178 INFO:teuthology.orchestra.run.smithi146.stdout: NetworkManager-1:1.46.0-1.el9.x86_64 2024-03-04T09:24:01.178 INFO:teuthology.orchestra.run.smithi146.stdout: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 2024-03-04T09:24:01.178 INFO:teuthology.orchestra.run.smithi146.stdout: NetworkManager-team-1:1.46.0-1.el9.x86_64 2024-03-04T09:24:01.178 INFO:teuthology.orchestra.run.smithi146.stdout: NetworkManager-tui-1:1.46.0-1.el9.x86_64 2024-03-04T09:24:01.178 INFO:teuthology.orchestra.run.smithi146.stdout: acl-2.3.1-4.el9.x86_64 2024-03-04T09:24:01.178 INFO:teuthology.orchestra.run.smithi146.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-04T09:24:01.178 INFO:teuthology.orchestra.run.smithi146.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-04T09:24:01.178 INFO:teuthology.orchestra.run.smithi146.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-04T09:24:01.178 INFO:teuthology.orchestra.run.smithi146.stdout: iproute-6.7.0-2.el9.x86_64 2024-03-04T09:24:01.179 INFO:teuthology.orchestra.run.smithi146.stdout: iproute-tc-6.7.0-2.el9.x86_64 2024-03-04T09:24:01.179 INFO:teuthology.orchestra.run.smithi146.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-04T09:24:01.179 INFO:teuthology.orchestra.run.smithi146.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-04T09:24:01.179 INFO:teuthology.orchestra.run.smithi146.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-04T09:24:01.179 INFO:teuthology.orchestra.run.smithi146.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-04T09:24:01.179 INFO:teuthology.orchestra.run.smithi146.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-04T09:24:01.179 INFO:teuthology.orchestra.run.smithi146.stdout: librados2-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T09:24:01.179 INFO:teuthology.orchestra.run.smithi146.stdout: librbd1-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T09:24:01.179 INFO:teuthology.orchestra.run.smithi146.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-04T09:24:01.179 INFO:teuthology.orchestra.run.smithi146.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-04T09:24:01.179 INFO:teuthology.orchestra.run.smithi146.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-04T09:24:01.179 INFO:teuthology.orchestra.run.smithi146.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-04T09:24:01.179 INFO:teuthology.orchestra.run.smithi146.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-04T09:24:01.179 INFO:teuthology.orchestra.run.smithi146.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-04T09:24:01.179 INFO:teuthology.orchestra.run.smithi146.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-04T09:24:01.180 INFO:teuthology.orchestra.run.smithi146.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-04T09:24:01.180 INFO:teuthology.orchestra.run.smithi146.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-04T09:24:01.180 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-03-04T09:24:01.180 INFO:teuthology.orchestra.run.smithi146.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-04T09:24:01.180 INFO:teuthology.orchestra.run.smithi146.stdout: automake-1.16.2-8.el9.noarch 2024-03-04T09:24:01.180 INFO:teuthology.orchestra.run.smithi146.stdout: bison-3.7.4-5.el9.x86_64 2024-03-04T09:24:01.180 INFO:teuthology.orchestra.run.smithi146.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-04T09:24:01.180 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T09:24:01.180 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-base-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T09:24:01.180 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T09:24:01.180 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T09:24:01.180 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-grafana-dashboards-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-04T09:24:01.180 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-immutable-object-cache-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T09:24:01.180 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mds-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T09:24:01.181 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T09:24:01.181 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-04T09:24:01.181 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-dashboard-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-04T09:24:01.181 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-04T09:24:01.181 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-modules-core-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-04T09:24:01.181 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-rook-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-04T09:24:01.181 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mon-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T09:24:01.181 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-osd-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T09:24:01.181 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-prometheus-alerts-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-04T09:24:01.181 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-radosgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T09:24:01.181 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-selinux-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T09:24:01.181 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-test-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T09:24:01.181 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-volume-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-04T09:24:01.182 INFO:teuthology.orchestra.run.smithi146.stdout: cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-04T09:24:01.182 INFO:teuthology.orchestra.run.smithi146.stdout: cephfs-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T09:24:01.182 INFO:teuthology.orchestra.run.smithi146.stdout: cephfs-shell-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T09:24:01.182 INFO:teuthology.orchestra.run.smithi146.stdout: cephfs-top-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-04T09:24:01.182 INFO:teuthology.orchestra.run.smithi146.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-04T09:24:01.182 INFO:teuthology.orchestra.run.smithi146.stdout: flex-2.6.4-9.el9.x86_64 2024-03-04T09:24:01.182 INFO:teuthology.orchestra.run.smithi146.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-04T09:24:01.182 INFO:teuthology.orchestra.run.smithi146.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-04T09:24:01.182 INFO:teuthology.orchestra.run.smithi146.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-04T09:24:01.182 INFO:teuthology.orchestra.run.smithi146.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-04T09:24:01.182 INFO:teuthology.orchestra.run.smithi146.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-04T09:24:01.182 INFO:teuthology.orchestra.run.smithi146.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-04T09:24:01.182 INFO:teuthology.orchestra.run.smithi146.stdout: jq-1.6-16.el9.x86_64 2024-03-04T09:24:01.182 INFO:teuthology.orchestra.run.smithi146.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-04T09:24:01.182 INFO:teuthology.orchestra.run.smithi146.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-04T09:24:01.183 INFO:teuthology.orchestra.run.smithi146.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-04T09:24:01.183 INFO:teuthology.orchestra.run.smithi146.stdout: libcephfs-devel-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T09:24:01.183 INFO:teuthology.orchestra.run.smithi146.stdout: libcephfs2-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T09:24:01.183 INFO:teuthology.orchestra.run.smithi146.stdout: libcephsqlite-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T09:24:01.183 INFO:teuthology.orchestra.run.smithi146.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-04T09:24:01.183 INFO:teuthology.orchestra.run.smithi146.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-04T09:24:01.183 INFO:teuthology.orchestra.run.smithi146.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-04T09:24:01.183 INFO:teuthology.orchestra.run.smithi146.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-04T09:24:01.183 INFO:teuthology.orchestra.run.smithi146.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-04T09:24:01.183 INFO:teuthology.orchestra.run.smithi146.stdout: librados-devel-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T09:24:01.183 INFO:teuthology.orchestra.run.smithi146.stdout: libradosstriper1-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T09:24:01.183 INFO:teuthology.orchestra.run.smithi146.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-04T09:24:01.183 INFO:teuthology.orchestra.run.smithi146.stdout: librgw2-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T09:24:01.183 INFO:teuthology.orchestra.run.smithi146.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-04T09:24:01.183 INFO:teuthology.orchestra.run.smithi146.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-04T09:24:01.184 INFO:teuthology.orchestra.run.smithi146.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-04T09:24:01.184 INFO:teuthology.orchestra.run.smithi146.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-04T09:24:01.184 INFO:teuthology.orchestra.run.smithi146.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-04T09:24:01.184 INFO:teuthology.orchestra.run.smithi146.stdout: lua-5.4.4-4.el9.x86_64 2024-03-04T09:24:01.184 INFO:teuthology.orchestra.run.smithi146.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-04T09:24:01.184 INFO:teuthology.orchestra.run.smithi146.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-04T09:24:01.184 INFO:teuthology.orchestra.run.smithi146.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-04T09:24:01.184 INFO:teuthology.orchestra.run.smithi146.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-04T09:24:01.184 INFO:teuthology.orchestra.run.smithi146.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-04T09:24:01.184 INFO:teuthology.orchestra.run.smithi146.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-04T09:24:01.184 INFO:teuthology.orchestra.run.smithi146.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-04T09:24:01.184 INFO:teuthology.orchestra.run.smithi146.stdout: postgresql-13.11-1.el9.x86_64 2024-03-04T09:24:01.184 INFO:teuthology.orchestra.run.smithi146.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-04T09:24:01.185 INFO:teuthology.orchestra.run.smithi146.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-04T09:24:01.185 INFO:teuthology.orchestra.run.smithi146.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-04T09:24:01.185 INFO:teuthology.orchestra.run.smithi146.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-04T09:24:01.185 INFO:teuthology.orchestra.run.smithi146.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-04T09:24:01.185 INFO:teuthology.orchestra.run.smithi146.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-04T09:24:01.185 INFO:teuthology.orchestra.run.smithi146.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-04T09:24:01.185 INFO:teuthology.orchestra.run.smithi146.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-04T09:24:01.185 INFO:teuthology.orchestra.run.smithi146.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-04T09:24:01.185 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-04T09:24:01.185 INFO:teuthology.orchestra.run.smithi146.stdout: python3-ceph-argparse-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T09:24:01.185 INFO:teuthology.orchestra.run.smithi146.stdout: python3-ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T09:24:01.185 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cephfs-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T09:24:01.185 INFO:teuthology.orchestra.run.smithi146.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-04T09:24:01.185 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-04T09:24:01.186 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-04T09:24:01.186 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-04T09:24:01.186 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-04T09:24:01.186 INFO:teuthology.orchestra.run.smithi146.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-04T09:24:01.186 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-04T09:24:01.186 INFO:teuthology.orchestra.run.smithi146.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-04T09:24:01.186 INFO:teuthology.orchestra.run.smithi146.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-04T09:24:01.186 INFO:teuthology.orchestra.run.smithi146.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-04T09:24:01.186 INFO:teuthology.orchestra.run.smithi146.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-04T09:24:01.186 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-04T09:24:01.186 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-04T09:24:01.186 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-04T09:24:01.186 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-04T09:24:01.186 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-04T09:24:01.187 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-04T09:24:01.187 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-04T09:24:01.187 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-04T09:24:01.187 INFO:teuthology.orchestra.run.smithi146.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-04T09:24:01.187 INFO:teuthology.orchestra.run.smithi146.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-04T09:24:01.187 INFO:teuthology.orchestra.run.smithi146.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-04T09:24:01.187 INFO:teuthology.orchestra.run.smithi146.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-04T09:24:01.187 INFO:teuthology.orchestra.run.smithi146.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-04T09:24:01.187 INFO:teuthology.orchestra.run.smithi146.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-04T09:24:01.187 INFO:teuthology.orchestra.run.smithi146.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-04T09:24:01.187 INFO:teuthology.orchestra.run.smithi146.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-04T09:24:01.187 INFO:teuthology.orchestra.run.smithi146.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-04T09:24:01.187 INFO:teuthology.orchestra.run.smithi146.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-04T09:24:01.187 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-04T09:24:01.187 INFO:teuthology.orchestra.run.smithi146.stdout: python3-ply-3.11-14.el9.noarch 2024-03-04T09:24:01.188 INFO:teuthology.orchestra.run.smithi146.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-04T09:24:01.188 INFO:teuthology.orchestra.run.smithi146.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-04T09:24:01.188 INFO:teuthology.orchestra.run.smithi146.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-04T09:24:01.188 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-04T09:24:01.188 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-04T09:24:01.188 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-04T09:24:01.188 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-04T09:24:01.188 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-04T09:24:01.188 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-04T09:24:01.188 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rados-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T09:24:01.188 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T09:24:01.188 INFO:teuthology.orchestra.run.smithi146.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-04T09:24:01.188 INFO:teuthology.orchestra.run.smithi146.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-04T09:24:01.189 INFO:teuthology.orchestra.run.smithi146.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-04T09:24:01.189 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T09:24:01.189 INFO:teuthology.orchestra.run.smithi146.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-04T09:24:01.189 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-04T09:24:01.189 INFO:teuthology.orchestra.run.smithi146.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-04T09:24:01.189 INFO:teuthology.orchestra.run.smithi146.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-04T09:24:01.189 INFO:teuthology.orchestra.run.smithi146.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-04T09:24:01.189 INFO:teuthology.orchestra.run.smithi146.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-04T09:24:01.189 INFO:teuthology.orchestra.run.smithi146.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-04T09:24:01.189 INFO:teuthology.orchestra.run.smithi146.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-04T09:24:01.189 INFO:teuthology.orchestra.run.smithi146.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-04T09:24:01.189 INFO:teuthology.orchestra.run.smithi146.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-04T09:24:01.189 INFO:teuthology.orchestra.run.smithi146.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-04T09:24:01.189 INFO:teuthology.orchestra.run.smithi146.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-04T09:24:01.190 INFO:teuthology.orchestra.run.smithi146.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-04T09:24:01.190 INFO:teuthology.orchestra.run.smithi146.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-04T09:24:01.190 INFO:teuthology.orchestra.run.smithi146.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-04T09:24:01.190 INFO:teuthology.orchestra.run.smithi146.stdout: rbd-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T09:24:01.190 INFO:teuthology.orchestra.run.smithi146.stdout: rbd-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T09:24:01.190 INFO:teuthology.orchestra.run.smithi146.stdout: rbd-nbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T09:24:01.190 INFO:teuthology.orchestra.run.smithi146.stdout: re2-1:20211101-3.el9.x86_64 2024-03-04T09:24:01.190 INFO:teuthology.orchestra.run.smithi146.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-04T09:24:01.190 INFO:teuthology.orchestra.run.smithi146.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-04T09:24:01.190 INFO:teuthology.orchestra.run.smithi146.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-04T09:24:01.190 INFO:teuthology.orchestra.run.smithi146.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-04T09:24:01.190 INFO:teuthology.orchestra.run.smithi146.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-04T09:24:01.191 INFO:teuthology.orchestra.run.smithi146.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-04T09:24:01.191 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:24:01.191 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-03-04T09:24:01.837 DEBUG:teuthology.parallel:result is None 2024-03-04T09:24:02.091 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-2:19.0.0-1739.g1bc250c7.el9.x86_64 1/202 2024-03-04T09:24:02.091 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-base-2:19.0.0-1739.g1bc250c7.el9.x86_64 2/202 2024-03-04T09:24:02.091 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 3/202 2024-03-04T09:24:02.091 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 4/202 2024-03-04T09:24:02.091 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1739.g1bc25 5/202 2024-03-04T09:24:02.091 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-mds-2:19.0.0-1739.g1bc250c7.el9.x86_64 6/202 2024-03-04T09:24:02.091 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-mgr-2:19.0.0-1739.g1bc250c7.el9.x86_64 7/202 2024-03-04T09:24:02.092 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-mon-2:19.0.0-1739.g1bc250c7.el9.x86_64 8/202 2024-03-04T09:24:02.092 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-osd-2:19.0.0-1739.g1bc250c7.el9.x86_64 9/202 2024-03-04T09:24:02.092 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-radosgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 10/202 2024-03-04T09:24:02.092 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-selinux-2:19.0.0-1739.g1bc250c7.el9.x86_64 11/202 2024-03-04T09:24:02.092 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-test-2:19.0.0-1739.g1bc250c7.el9.x86_64 12/202 2024-03-04T09:24:02.092 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : cephfs-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 13/202 2024-03-04T09:24:02.092 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : cephfs-shell-2:19.0.0-1739.g1bc250c7.el9.x86_64 14/202 2024-03-04T09:24:02.092 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libcephfs-devel-2:19.0.0-1739.g1bc250c7.el9.x86_ 15/202 2024-03-04T09:24:02.092 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libcephfs2-2:19.0.0-1739.g1bc250c7.el9.x86_64 16/202 2024-03-04T09:24:02.092 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libcephsqlite-2:19.0.0-1739.g1bc250c7.el9.x86_64 17/202 2024-03-04T09:24:02.092 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : librados-devel-2:19.0.0-1739.g1bc250c7.el9.x86_6 18/202 2024-03-04T09:24:02.092 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libradosstriper1-2:19.0.0-1739.g1bc250c7.el9.x86 19/202 2024-03-04T09:24:02.092 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : librgw2-2:19.0.0-1739.g1bc250c7.el9.x86_64 20/202 2024-03-04T09:24:02.092 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1739.g1bc250c7.el 21/202 2024-03-04T09:24:02.093 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-ceph-common-2:19.0.0-1739.g1bc250c7.el9. 22/202 2024-03-04T09:24:02.093 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-cephfs-2:19.0.0-1739.g1bc250c7.el9.x86_6 23/202 2024-03-04T09:24:02.093 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-rados-2:19.0.0-1739.g1bc250c7.el9.x86_64 24/202 2024-03-04T09:24:02.093 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-rbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 25/202 2024-03-04T09:24:02.093 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-rgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 26/202 2024-03-04T09:24:02.093 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : rbd-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 27/202 2024-03-04T09:24:02.093 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : rbd-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 28/202 2024-03-04T09:24:02.093 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : rbd-nbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 29/202 2024-03-04T09:24:02.093 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1739.g1bc250c7. 30/202 2024-03-04T09:24:02.093 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1739.g1bc250c7.el9.noa 31/202 2024-03-04T09:24:02.108 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1739.g1bc250c7.el9.n 32/202 2024-03-04T09:24:02.109 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1739.g1bc 33/202 2024-03-04T09:24:02.109 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1739.g1bc250c7.el 34/202 2024-03-04T09:24:02.109 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1739.g1bc250c7.el9.noarch 35/202 2024-03-04T09:24:02.109 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1739.g1bc250c7.e 36/202 2024-03-04T09:24:02.109 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-volume-2:19.0.0-1739.g1bc250c7.el9.noarch 37/202 2024-03-04T09:24:02.109 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 38/202 2024-03-04T09:24:02.109 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : cephfs-top-2:19.0.0-1739.g1bc250c7.el9.noarch 39/202 2024-03-04T09:24:02.109 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/202 2024-03-04T09:24:02.109 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/202 2024-03-04T09:24:02.109 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/202 2024-03-04T09:24:02.109 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 43/202 2024-03-04T09:24:02.109 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : flex-2.6.4-9.el9.x86_64 44/202 2024-03-04T09:24:02.109 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 45/202 2024-03-04T09:24:02.109 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 46/202 2024-03-04T09:24:02.110 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 47/202 2024-03-04T09:24:02.110 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 48/202 2024-03-04T09:24:02.110 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 49/202 2024-03-04T09:24:02.110 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 50/202 2024-03-04T09:24:02.110 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 51/202 2024-03-04T09:24:02.110 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 52/202 2024-03-04T09:24:02.110 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 53/202 2024-03-04T09:24:02.110 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 54/202 2024-03-04T09:24:02.110 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 55/202 2024-03-04T09:24:02.110 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : lua-5.4.4-4.el9.x86_64 56/202 2024-03-04T09:24:02.110 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 57/202 2024-03-04T09:24:02.110 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 58/202 2024-03-04T09:24:02.110 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 59/202 2024-03-04T09:24:02.110 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : postgresql-13.11-1.el9.x86_64 60/202 2024-03-04T09:24:02.110 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 61/202 2024-03-04T09:24:02.111 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 62/202 2024-03-04T09:24:02.111 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 63/202 2024-03-04T09:24:02.111 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 64/202 2024-03-04T09:24:02.111 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 65/202 2024-03-04T09:24:02.111 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 66/202 2024-03-04T09:24:02.111 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 67/202 2024-03-04T09:24:02.111 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 68/202 2024-03-04T09:24:02.111 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 69/202 2024-03-04T09:24:02.111 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 70/202 2024-03-04T09:24:02.111 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 71/202 2024-03-04T09:24:02.111 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 72/202 2024-03-04T09:24:02.111 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 73/202 2024-03-04T09:24:02.112 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 74/202 2024-03-04T09:24:02.112 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 75/202 2024-03-04T09:24:02.112 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 76/202 2024-03-04T09:24:02.112 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 77/202 2024-03-04T09:24:02.112 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 78/202 2024-03-04T09:24:02.112 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 79/202 2024-03-04T09:24:02.112 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 80/202 2024-03-04T09:24:02.112 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 81/202 2024-03-04T09:24:02.112 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 82/202 2024-03-04T09:24:02.112 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 83/202 2024-03-04T09:24:02.112 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 84/202 2024-03-04T09:24:02.112 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 85/202 2024-03-04T09:24:02.112 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 86/202 2024-03-04T09:24:02.112 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 87/202 2024-03-04T09:24:02.113 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 88/202 2024-03-04T09:24:02.113 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 89/202 2024-03-04T09:24:02.113 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 90/202 2024-03-04T09:24:02.113 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 91/202 2024-03-04T09:24:02.113 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 92/202 2024-03-04T09:24:02.113 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 93/202 2024-03-04T09:24:02.113 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : jq-1.6-16.el9.x86_64 94/202 2024-03-04T09:24:02.113 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 95/202 2024-03-04T09:24:02.113 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 96/202 2024-03-04T09:24:02.113 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 97/202 2024-03-04T09:24:02.113 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 98/202 2024-03-04T09:24:02.113 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 99/202 2024-03-04T09:24:02.113 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-ply-3.11-14.el9.noarch 100/202 2024-03-04T09:24:02.113 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 101/202 2024-03-04T09:24:02.113 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 102/202 2024-03-04T09:24:02.114 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 103/202 2024-03-04T09:24:02.114 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 104/202 2024-03-04T09:24:02.114 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/202 2024-03-04T09:24:02.114 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/202 2024-03-04T09:24:02.114 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/202 2024-03-04T09:24:02.114 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/202 2024-03-04T09:24:02.114 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/202 2024-03-04T09:24:02.114 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/202 2024-03-04T09:24:02.114 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/202 2024-03-04T09:24:02.114 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/202 2024-03-04T09:24:02.114 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/202 2024-03-04T09:24:02.114 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/202 2024-03-04T09:24:02.114 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/202 2024-03-04T09:24:02.114 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 116/202 2024-03-04T09:24:02.115 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/202 2024-03-04T09:24:02.115 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/202 2024-03-04T09:24:02.115 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/202 2024-03-04T09:24:02.115 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/202 2024-03-04T09:24:02.115 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/202 2024-03-04T09:24:02.115 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/202 2024-03-04T09:24:02.115 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/202 2024-03-04T09:24:02.115 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 124/202 2024-03-04T09:24:02.115 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/202 2024-03-04T09:24:02.115 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/202 2024-03-04T09:24:02.115 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/202 2024-03-04T09:24:02.115 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/202 2024-03-04T09:24:02.115 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/202 2024-03-04T09:24:02.115 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/202 2024-03-04T09:24:02.116 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/202 2024-03-04T09:24:02.116 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/202 2024-03-04T09:24:02.116 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/202 2024-03-04T09:24:02.116 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/202 2024-03-04T09:24:02.116 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/202 2024-03-04T09:24:02.116 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/202 2024-03-04T09:24:02.116 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/202 2024-03-04T09:24:02.116 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/202 2024-03-04T09:24:02.116 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/202 2024-03-04T09:24:02.116 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/202 2024-03-04T09:24:02.116 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/202 2024-03-04T09:24:02.116 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/202 2024-03-04T09:24:02.116 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/202 2024-03-04T09:24:02.116 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/202 2024-03-04T09:24:02.116 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/202 2024-03-04T09:24:02.117 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/202 2024-03-04T09:24:02.117 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/202 2024-03-04T09:24:02.117 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/202 2024-03-04T09:24:02.117 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/202 2024-03-04T09:24:02.117 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/202 2024-03-04T09:24:02.117 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : librados2-2:19.0.0-1739.g1bc250c7.el9.x86_64 151/202 2024-03-04T09:24:02.117 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/202 2024-03-04T09:24:02.117 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : librbd1-2:19.0.0-1739.g1bc250c7.el9.x86_64 153/202 2024-03-04T09:24:02.117 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/202 2024-03-04T09:24:02.117 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 155/202 2024-03-04T09:24:02.117 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 156/202 2024-03-04T09:24:02.117 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : NetworkManager-1:1.46.0-1.el9.x86_64 157/202 2024-03-04T09:24:02.117 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/202 2024-03-04T09:24:02.117 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 159/202 2024-03-04T09:24:02.118 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/202 2024-03-04T09:24:02.118 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : NetworkManager-team-1:1.46.0-1.el9.x86_64 161/202 2024-03-04T09:24:02.118 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/202 2024-03-04T09:24:02.118 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : NetworkManager-tui-1:1.46.0-1.el9.x86_64 163/202 2024-03-04T09:24:02.118 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/202 2024-03-04T09:24:02.118 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/202 2024-03-04T09:24:02.118 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/202 2024-03-04T09:24:02.118 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/202 2024-03-04T09:24:02.118 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/202 2024-03-04T09:24:02.118 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/202 2024-03-04T09:24:02.118 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/202 2024-03-04T09:24:02.118 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/202 2024-03-04T09:24:02.118 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/202 2024-03-04T09:24:02.118 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : iproute-6.7.0-2.el9.x86_64 173/202 2024-03-04T09:24:02.118 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : iproute-6.2.0-5.el9.x86_64 174/202 2024-03-04T09:24:02.119 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : iproute-tc-6.7.0-2.el9.x86_64 175/202 2024-03-04T09:24:02.119 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : iproute-tc-6.2.0-5.el9.x86_64 176/202 2024-03-04T09:24:02.119 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 177/202 2024-03-04T09:24:02.119 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 178/202 2024-03-04T09:24:02.119 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 179/202 2024-03-04T09:24:02.119 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 180/202 2024-03-04T09:24:02.119 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 181/202 2024-03-04T09:24:02.119 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 182/202 2024-03-04T09:24:02.119 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 183/202 2024-03-04T09:24:02.119 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 184/202 2024-03-04T09:24:02.119 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 185/202 2024-03-04T09:24:02.119 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 186/202 2024-03-04T09:24:02.119 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 187/202 2024-03-04T09:24:02.119 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 188/202 2024-03-04T09:24:02.119 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 189/202 2024-03-04T09:24:02.119 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 190/202 2024-03-04T09:24:02.120 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 191/202 2024-03-04T09:24:02.120 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 192/202 2024-03-04T09:24:02.120 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 193/202 2024-03-04T09:24:02.120 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 194/202 2024-03-04T09:24:02.120 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 195/202 2024-03-04T09:24:02.120 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 196/202 2024-03-04T09:24:02.120 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 197/202 2024-03-04T09:24:02.120 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 198/202 2024-03-04T09:24:02.120 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 199/202 2024-03-04T09:24:02.120 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 200/202 2024-03-04T09:24:02.120 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 201/202 2024-03-04T09:24:04.752 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 202/202 2024-03-04T09:24:04.753 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:24:04.753 INFO:teuthology.orchestra.run.smithi006.stdout:Upgraded: 2024-03-04T09:24:04.753 INFO:teuthology.orchestra.run.smithi006.stdout: NetworkManager-1:1.46.0-1.el9.x86_64 2024-03-04T09:24:04.753 INFO:teuthology.orchestra.run.smithi006.stdout: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 2024-03-04T09:24:04.753 INFO:teuthology.orchestra.run.smithi006.stdout: NetworkManager-team-1:1.46.0-1.el9.x86_64 2024-03-04T09:24:04.753 INFO:teuthology.orchestra.run.smithi006.stdout: NetworkManager-tui-1:1.46.0-1.el9.x86_64 2024-03-04T09:24:04.753 INFO:teuthology.orchestra.run.smithi006.stdout: acl-2.3.1-4.el9.x86_64 2024-03-04T09:24:04.753 INFO:teuthology.orchestra.run.smithi006.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-04T09:24:04.753 INFO:teuthology.orchestra.run.smithi006.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-04T09:24:04.753 INFO:teuthology.orchestra.run.smithi006.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-04T09:24:04.753 INFO:teuthology.orchestra.run.smithi006.stdout: iproute-6.7.0-2.el9.x86_64 2024-03-04T09:24:04.753 INFO:teuthology.orchestra.run.smithi006.stdout: iproute-tc-6.7.0-2.el9.x86_64 2024-03-04T09:24:04.753 INFO:teuthology.orchestra.run.smithi006.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-04T09:24:04.754 INFO:teuthology.orchestra.run.smithi006.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-04T09:24:04.754 INFO:teuthology.orchestra.run.smithi006.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-04T09:24:04.754 INFO:teuthology.orchestra.run.smithi006.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-04T09:24:04.754 INFO:teuthology.orchestra.run.smithi006.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-04T09:24:04.754 INFO:teuthology.orchestra.run.smithi006.stdout: librados2-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T09:24:04.754 INFO:teuthology.orchestra.run.smithi006.stdout: librbd1-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T09:24:04.754 INFO:teuthology.orchestra.run.smithi006.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-04T09:24:04.754 INFO:teuthology.orchestra.run.smithi006.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-04T09:24:04.754 INFO:teuthology.orchestra.run.smithi006.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-04T09:24:04.754 INFO:teuthology.orchestra.run.smithi006.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-04T09:24:04.754 INFO:teuthology.orchestra.run.smithi006.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-04T09:24:04.754 INFO:teuthology.orchestra.run.smithi006.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-04T09:24:04.754 INFO:teuthology.orchestra.run.smithi006.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-04T09:24:04.755 INFO:teuthology.orchestra.run.smithi006.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-04T09:24:04.755 INFO:teuthology.orchestra.run.smithi006.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-04T09:24:04.755 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2024-03-04T09:24:04.755 INFO:teuthology.orchestra.run.smithi006.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-04T09:24:04.755 INFO:teuthology.orchestra.run.smithi006.stdout: automake-1.16.2-8.el9.noarch 2024-03-04T09:24:04.755 INFO:teuthology.orchestra.run.smithi006.stdout: bison-3.7.4-5.el9.x86_64 2024-03-04T09:24:04.755 INFO:teuthology.orchestra.run.smithi006.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-04T09:24:04.755 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T09:24:04.755 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-base-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T09:24:04.755 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T09:24:04.755 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T09:24:04.755 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-grafana-dashboards-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-04T09:24:04.755 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-immutable-object-cache-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T09:24:04.755 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mds-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T09:24:04.756 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T09:24:04.756 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-04T09:24:04.756 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-dashboard-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-04T09:24:04.756 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-04T09:24:04.756 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-modules-core-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-04T09:24:04.756 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-rook-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-04T09:24:04.756 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mon-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T09:24:04.756 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-osd-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T09:24:04.756 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-prometheus-alerts-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-04T09:24:04.756 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-radosgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T09:24:04.756 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-selinux-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T09:24:04.756 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-test-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T09:24:04.756 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-volume-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-04T09:24:04.756 INFO:teuthology.orchestra.run.smithi006.stdout: cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-04T09:24:04.757 INFO:teuthology.orchestra.run.smithi006.stdout: cephfs-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T09:24:04.757 INFO:teuthology.orchestra.run.smithi006.stdout: cephfs-shell-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T09:24:04.757 INFO:teuthology.orchestra.run.smithi006.stdout: cephfs-top-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-04T09:24:04.757 INFO:teuthology.orchestra.run.smithi006.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-04T09:24:04.757 INFO:teuthology.orchestra.run.smithi006.stdout: flex-2.6.4-9.el9.x86_64 2024-03-04T09:24:04.757 INFO:teuthology.orchestra.run.smithi006.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-04T09:24:04.757 INFO:teuthology.orchestra.run.smithi006.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-04T09:24:04.757 INFO:teuthology.orchestra.run.smithi006.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-04T09:24:04.757 INFO:teuthology.orchestra.run.smithi006.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-04T09:24:04.757 INFO:teuthology.orchestra.run.smithi006.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-04T09:24:04.757 INFO:teuthology.orchestra.run.smithi006.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-04T09:24:04.757 INFO:teuthology.orchestra.run.smithi006.stdout: jq-1.6-16.el9.x86_64 2024-03-04T09:24:04.757 INFO:teuthology.orchestra.run.smithi006.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-04T09:24:04.757 INFO:teuthology.orchestra.run.smithi006.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-04T09:24:04.757 INFO:teuthology.orchestra.run.smithi006.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-04T09:24:04.758 INFO:teuthology.orchestra.run.smithi006.stdout: libcephfs-devel-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T09:24:04.758 INFO:teuthology.orchestra.run.smithi006.stdout: libcephfs2-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T09:24:04.758 INFO:teuthology.orchestra.run.smithi006.stdout: libcephsqlite-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T09:24:04.758 INFO:teuthology.orchestra.run.smithi006.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-04T09:24:04.758 INFO:teuthology.orchestra.run.smithi006.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-04T09:24:04.758 INFO:teuthology.orchestra.run.smithi006.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-04T09:24:04.758 INFO:teuthology.orchestra.run.smithi006.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-04T09:24:04.758 INFO:teuthology.orchestra.run.smithi006.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-04T09:24:04.758 INFO:teuthology.orchestra.run.smithi006.stdout: librados-devel-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T09:24:04.758 INFO:teuthology.orchestra.run.smithi006.stdout: libradosstriper1-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T09:24:04.758 INFO:teuthology.orchestra.run.smithi006.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-04T09:24:04.758 INFO:teuthology.orchestra.run.smithi006.stdout: librgw2-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T09:24:04.758 INFO:teuthology.orchestra.run.smithi006.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-04T09:24:04.758 INFO:teuthology.orchestra.run.smithi006.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-04T09:24:04.758 INFO:teuthology.orchestra.run.smithi006.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-04T09:24:04.759 INFO:teuthology.orchestra.run.smithi006.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-04T09:24:04.759 INFO:teuthology.orchestra.run.smithi006.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-04T09:24:04.759 INFO:teuthology.orchestra.run.smithi006.stdout: lua-5.4.4-4.el9.x86_64 2024-03-04T09:24:04.759 INFO:teuthology.orchestra.run.smithi006.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-04T09:24:04.759 INFO:teuthology.orchestra.run.smithi006.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-04T09:24:04.759 INFO:teuthology.orchestra.run.smithi006.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-04T09:24:04.759 INFO:teuthology.orchestra.run.smithi006.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-04T09:24:04.759 INFO:teuthology.orchestra.run.smithi006.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-04T09:24:04.759 INFO:teuthology.orchestra.run.smithi006.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-04T09:24:04.759 INFO:teuthology.orchestra.run.smithi006.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-04T09:24:04.759 INFO:teuthology.orchestra.run.smithi006.stdout: postgresql-13.11-1.el9.x86_64 2024-03-04T09:24:04.759 INFO:teuthology.orchestra.run.smithi006.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-04T09:24:04.759 INFO:teuthology.orchestra.run.smithi006.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-04T09:24:04.759 INFO:teuthology.orchestra.run.smithi006.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-04T09:24:04.759 INFO:teuthology.orchestra.run.smithi006.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-04T09:24:04.760 INFO:teuthology.orchestra.run.smithi006.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-04T09:24:04.760 INFO:teuthology.orchestra.run.smithi006.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-04T09:24:04.760 INFO:teuthology.orchestra.run.smithi006.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-04T09:24:04.760 INFO:teuthology.orchestra.run.smithi006.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-04T09:24:04.760 INFO:teuthology.orchestra.run.smithi006.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-04T09:24:04.760 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-04T09:24:04.760 INFO:teuthology.orchestra.run.smithi006.stdout: python3-ceph-argparse-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T09:24:04.760 INFO:teuthology.orchestra.run.smithi006.stdout: python3-ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T09:24:04.760 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cephfs-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T09:24:04.760 INFO:teuthology.orchestra.run.smithi006.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-04T09:24:04.760 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-04T09:24:04.760 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-04T09:24:04.760 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-04T09:24:04.760 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-04T09:24:04.761 INFO:teuthology.orchestra.run.smithi006.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-04T09:24:04.761 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-04T09:24:04.761 INFO:teuthology.orchestra.run.smithi006.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-04T09:24:04.761 INFO:teuthology.orchestra.run.smithi006.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-04T09:24:04.761 INFO:teuthology.orchestra.run.smithi006.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-04T09:24:04.761 INFO:teuthology.orchestra.run.smithi006.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-04T09:24:04.761 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-04T09:24:04.761 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-04T09:24:04.761 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-04T09:24:04.761 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-04T09:24:04.761 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-04T09:24:04.761 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-04T09:24:04.761 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-04T09:24:04.761 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-04T09:24:04.761 INFO:teuthology.orchestra.run.smithi006.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-04T09:24:04.762 INFO:teuthology.orchestra.run.smithi006.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-04T09:24:04.762 INFO:teuthology.orchestra.run.smithi006.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-04T09:24:04.762 INFO:teuthology.orchestra.run.smithi006.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-04T09:24:04.762 INFO:teuthology.orchestra.run.smithi006.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-04T09:24:04.762 INFO:teuthology.orchestra.run.smithi006.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-04T09:24:04.762 INFO:teuthology.orchestra.run.smithi006.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-04T09:24:04.762 INFO:teuthology.orchestra.run.smithi006.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-04T09:24:04.762 INFO:teuthology.orchestra.run.smithi006.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-04T09:24:04.762 INFO:teuthology.orchestra.run.smithi006.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-04T09:24:04.762 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-04T09:24:04.762 INFO:teuthology.orchestra.run.smithi006.stdout: python3-ply-3.11-14.el9.noarch 2024-03-04T09:24:04.762 INFO:teuthology.orchestra.run.smithi006.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-04T09:24:04.762 INFO:teuthology.orchestra.run.smithi006.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-04T09:24:04.762 INFO:teuthology.orchestra.run.smithi006.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-04T09:24:04.763 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-04T09:24:04.763 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-04T09:24:04.763 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-04T09:24:04.763 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-04T09:24:04.763 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-04T09:24:04.763 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-04T09:24:04.763 INFO:teuthology.orchestra.run.smithi006.stdout: python3-rados-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T09:24:04.763 INFO:teuthology.orchestra.run.smithi006.stdout: python3-rbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T09:24:04.763 INFO:teuthology.orchestra.run.smithi006.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-04T09:24:04.763 INFO:teuthology.orchestra.run.smithi006.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-04T09:24:04.763 INFO:teuthology.orchestra.run.smithi006.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-04T09:24:04.763 INFO:teuthology.orchestra.run.smithi006.stdout: python3-rgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T09:24:04.763 INFO:teuthology.orchestra.run.smithi006.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-04T09:24:04.763 INFO:teuthology.orchestra.run.smithi006.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-04T09:24:04.763 INFO:teuthology.orchestra.run.smithi006.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-04T09:24:04.764 INFO:teuthology.orchestra.run.smithi006.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-04T09:24:04.764 INFO:teuthology.orchestra.run.smithi006.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-04T09:24:04.764 INFO:teuthology.orchestra.run.smithi006.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-04T09:24:04.764 INFO:teuthology.orchestra.run.smithi006.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-04T09:24:04.764 INFO:teuthology.orchestra.run.smithi006.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-04T09:24:04.764 INFO:teuthology.orchestra.run.smithi006.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-04T09:24:04.764 INFO:teuthology.orchestra.run.smithi006.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-04T09:24:04.764 INFO:teuthology.orchestra.run.smithi006.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-04T09:24:04.764 INFO:teuthology.orchestra.run.smithi006.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-04T09:24:04.764 INFO:teuthology.orchestra.run.smithi006.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-04T09:24:04.764 INFO:teuthology.orchestra.run.smithi006.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-04T09:24:04.764 INFO:teuthology.orchestra.run.smithi006.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-04T09:24:04.764 INFO:teuthology.orchestra.run.smithi006.stdout: rbd-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T09:24:04.765 INFO:teuthology.orchestra.run.smithi006.stdout: rbd-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T09:24:04.765 INFO:teuthology.orchestra.run.smithi006.stdout: rbd-nbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T09:24:04.765 INFO:teuthology.orchestra.run.smithi006.stdout: re2-1:20211101-3.el9.x86_64 2024-03-04T09:24:04.765 INFO:teuthology.orchestra.run.smithi006.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-04T09:24:04.765 INFO:teuthology.orchestra.run.smithi006.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-04T09:24:04.765 INFO:teuthology.orchestra.run.smithi006.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-04T09:24:04.765 INFO:teuthology.orchestra.run.smithi006.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-04T09:24:04.765 INFO:teuthology.orchestra.run.smithi006.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-04T09:24:04.765 INFO:teuthology.orchestra.run.smithi006.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-04T09:24:04.765 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:24:04.766 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-03-04T09:24:05.404 DEBUG:teuthology.parallel:result is None 2024-03-04T09:24:05.404 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd 2024-03-04T09:24:05.555 DEBUG:teuthology.orchestra.run.smithi006:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-04T09:24:05.611 INFO:teuthology.orchestra.run.smithi006.stdout:19.0.0-1739.g1bc250c7.el9 2024-03-04T09:24:05.611 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1739.g1bc250c7.el9 2024-03-04T09:24:05.611 INFO:teuthology.task.install:The correct ceph version 19.0.0-1739.g1bc250c7 is installed. 2024-03-04T09:24:05.613 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd 2024-03-04T09:24:05.761 DEBUG:teuthology.orchestra.run.smithi146:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-04T09:24:05.821 INFO:teuthology.orchestra.run.smithi146.stdout:19.0.0-1739.g1bc250c7.el9 2024-03-04T09:24:05.822 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1739.g1bc250c7.el9 2024-03-04T09:24:05.822 INFO:teuthology.task.install:The correct ceph version 19.0.0-1739.g1bc250c7 is installed. 2024-03-04T09:24:05.824 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-04T09:24:05.825 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-04T09:24:05.825 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-04T09:24:05.904 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-04T09:24:05.905 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-04T09:24:05.980 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-04T09:24:05.981 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-04T09:24:05.981 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/usr/bin/daemon-helper 2024-03-04T09:24:06.061 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-04T09:24:06.173 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-04T09:24:06.173 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/usr/bin/daemon-helper 2024-03-04T09:24:06.256 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-04T09:24:06.367 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-04T09:24:06.367 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-04T09:24:06.367 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-04T09:24:06.468 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-04T09:24:06.584 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-04T09:24:06.584 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-04T09:24:06.663 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-04T09:24:06.775 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-04T09:24:06.775 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-04T09:24:06.775 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/usr/bin/stdin-killer 2024-03-04T09:24:06.853 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-04T09:24:06.969 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-04T09:24:06.969 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/usr/bin/stdin-killer 2024-03-04T09:24:07.072 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-04T09:24:07.161 INFO:teuthology.run_tasks:Running task ceph... 2024-03-04T09:24:07.248 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-04T09:24:07.249 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 777 /var/log/ceph 2024-03-04T09:24:07.254 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod 777 /var/log/ceph 2024-03-04T09:24:07.351 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-04T09:24:07.351 DEBUG:teuthology.orchestra.run.smithi006:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-04T09:24:07.356 DEBUG:teuthology.orchestra.run.smithi146:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-04T09:24:07.445 INFO:tasks.ceph:Creating extra log directories... 2024-03-04T09:24:07.446 DEBUG:teuthology.orchestra.run.smithi006:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-04T09:24:07.450 DEBUG:teuthology.orchestra.run.smithi146:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-04T09:24:07.550 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2024-03-04T09:24:07.551 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2024-03-04T09:24:07.551 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2024-03-04T09:24:07.551 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-04T09:24:07.551 DEBUG:teuthology.orchestra.run.smithi006:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-03-04T09:24:07.593 DEBUG:teuthology.orchestra.run.smithi006:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-03-04T09:24:07.814 DEBUG:teuthology.orchestra.run.smithi006:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-03-04T09:24:07.883 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-04T09:24:07.884 DEBUG:teuthology.orchestra.run.smithi146:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-03-04T09:24:07.939 DEBUG:teuthology.orchestra.run.smithi146:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-03-04T09:24:08.200 DEBUG:teuthology.orchestra.run.smithi146:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-03-04T09:24:08.305 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-04T09:24:08.305 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-03-04T09:24:08.305 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/leonidus-2024-03-04_07:05:03-fs-wip-lusov-qdb-distro-default-smithi/7580856', 'branch': 'wip-lusov-qdb', 'description': 'fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/strays}', 'email': None, 'first_in_suite': False, 'job_id': '7580856', 'kernel': {'client': {'branch': 'testing'}, 'kdb': True, 'sha1': 'distro'}, 'ktype': 'upstream', 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'leonidus-2024-03-04_07:05:03-fs-wip-lusov-qdb-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-lusov-qdb'}, 'ceph': {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd'}}, 'kclient': {'syntax': 'v2'}, 'subvolume_version': 2, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-lusov-qdb', 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd'}}, 'owner': 'scheduled_leonidus@teuthology', 'priority': 80, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3']], 'seed': 5072, 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd', 'sleep_before_teardown': 0, 'subset': '246/10000', 'suite': 'fs', 'suite_branch': 'wip-lusov-qdb', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi006.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBC5g9n3VlEpyNSiCHwg9pTGH2AJTQjN0eGNKVhnZzlrs/IkxoO07GCYbY1l3lSh6BcN6HkmG4HgB/CfyXrWW3Vs=', 'smithi146.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHFfcb7jz8+ewcln2HSzgDFE300DWYZ0fvE9UQNbHDxKP7HaNMFdFaAGEb1IL6TJrNBpNkhlR8i/gVXYccJEAwU='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'client': {'branch': 'testing'}, 'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd'}}, {'ceph': {'create_rbd_pool': False, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd', 'cluster': 'ceph'}}, {'kclient': None}, {'cephfs_test_runner': {'modules': ['tasks.cephfs.test_strays']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ["local function is_kupstream() return yaml.ktype == 'upstream' end", "local function is_kdistro() return yaml.ktype == 'distro' end", 'local function is_fuse() return false end', 'local function syntax_version() return yaml.overrides.kclient.syntax end']}, 'teuthology_branch': 'main', 'teuthology_sha1': 'b1dac5519c57c269ea98a22fb7729016a1d0e4d2', 'timestamp': '2024-03-04_07:05:03', 'tube': 'smithi', 'user': 'leonidus', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-04T09:24:08.306 DEBUG:teuthology.orchestra.run.smithi006:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-04T09:24:08.310 DEBUG:teuthology.orchestra.run.smithi146:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-04T09:24:08.365 DEBUG:teuthology.orchestra.run.smithi006:> sudo install -d -m0777 -- /var/run/ceph 2024-03-04T09:24:08.395 DEBUG:teuthology.orchestra.run.smithi146:> sudo install -d -m0777 -- /var/run/ceph 2024-03-04T09:24:08.490 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-04T09:24:08.490 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/scratch_devs of=/dev/stdout 2024-03-04T09:24:08.544 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-04T09:24:08.544 DEBUG:teuthology.orchestra.run.smithi006:> stat /dev/vg_nvme/lv_1 2024-03-04T09:24:08.592 INFO:teuthology.orchestra.run.smithi006.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-04T09:24:08.592 INFO:teuthology.orchestra.run.smithi006.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-04T09:24:08.592 INFO:teuthology.orchestra.run.smithi006.stdout:Device: 5h/5d Inode: 632 Links: 1 2024-03-04T09:24:08.592 INFO:teuthology.orchestra.run.smithi006.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-04T09:24:08.592 INFO:teuthology.orchestra.run.smithi006.stdout:Context: system_u:object_r:device_t:s0 2024-03-04T09:24:08.593 INFO:teuthology.orchestra.run.smithi006.stdout:Access: 2024-03-04 09:23:57.641477003 +0000 2024-03-04T09:24:08.593 INFO:teuthology.orchestra.run.smithi006.stdout:Modify: 2024-03-04 09:21:19.701001747 +0000 2024-03-04T09:24:08.593 INFO:teuthology.orchestra.run.smithi006.stdout:Change: 2024-03-04 09:21:19.701001747 +0000 2024-03-04T09:24:08.593 INFO:teuthology.orchestra.run.smithi006.stdout: Birth: 2024-03-04 09:21:19.701001747 +0000 2024-03-04T09:24:08.593 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-04T09:24:08.712 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records in 2024-03-04T09:24:08.713 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records out 2024-03-04T09:24:08.713 INFO:teuthology.orchestra.run.smithi006.stderr:512 bytes copied, 0.000376585 s, 1.4 MB/s 2024-03-04T09:24:08.715 DEBUG:teuthology.orchestra.run.smithi006:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-04T09:24:08.758 DEBUG:teuthology.orchestra.run.smithi006:> stat /dev/vg_nvme/lv_2 2024-03-04T09:24:08.841 INFO:teuthology.orchestra.run.smithi006.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-04T09:24:08.841 INFO:teuthology.orchestra.run.smithi006.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-04T09:24:08.841 INFO:teuthology.orchestra.run.smithi006.stdout:Device: 5h/5d Inode: 653 Links: 1 2024-03-04T09:24:08.841 INFO:teuthology.orchestra.run.smithi006.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-04T09:24:08.841 INFO:teuthology.orchestra.run.smithi006.stdout:Context: system_u:object_r:device_t:s0 2024-03-04T09:24:08.841 INFO:teuthology.orchestra.run.smithi006.stdout:Access: 2024-03-04 09:23:57.642476991 +0000 2024-03-04T09:24:08.841 INFO:teuthology.orchestra.run.smithi006.stdout:Modify: 2024-03-04 09:21:20.331994289 +0000 2024-03-04T09:24:08.841 INFO:teuthology.orchestra.run.smithi006.stdout:Change: 2024-03-04 09:21:20.331994289 +0000 2024-03-04T09:24:08.841 INFO:teuthology.orchestra.run.smithi006.stdout: Birth: 2024-03-04 09:21:20.331994289 +0000 2024-03-04T09:24:08.841 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-04T09:24:08.949 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records in 2024-03-04T09:24:08.949 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records out 2024-03-04T09:24:08.949 INFO:teuthology.orchestra.run.smithi006.stderr:512 bytes copied, 0.000281692 s, 1.8 MB/s 2024-03-04T09:24:08.952 DEBUG:teuthology.orchestra.run.smithi006:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-04T09:24:08.991 DEBUG:teuthology.orchestra.run.smithi006:> stat /dev/vg_nvme/lv_3 2024-03-04T09:24:09.072 INFO:teuthology.orchestra.run.smithi006.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-04T09:24:09.072 INFO:teuthology.orchestra.run.smithi006.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-04T09:24:09.072 INFO:teuthology.orchestra.run.smithi006.stdout:Device: 5h/5d Inode: 669 Links: 1 2024-03-04T09:24:09.072 INFO:teuthology.orchestra.run.smithi006.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-04T09:24:09.072 INFO:teuthology.orchestra.run.smithi006.stdout:Context: system_u:object_r:device_t:s0 2024-03-04T09:24:09.072 INFO:teuthology.orchestra.run.smithi006.stdout:Access: 2024-03-04 09:23:57.643476979 +0000 2024-03-04T09:24:09.072 INFO:teuthology.orchestra.run.smithi006.stdout:Modify: 2024-03-04 09:21:20.929987221 +0000 2024-03-04T09:24:09.072 INFO:teuthology.orchestra.run.smithi006.stdout:Change: 2024-03-04 09:21:20.929987221 +0000 2024-03-04T09:24:09.072 INFO:teuthology.orchestra.run.smithi006.stdout: Birth: 2024-03-04 09:21:20.929987221 +0000 2024-03-04T09:24:09.073 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-04T09:24:09.187 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records in 2024-03-04T09:24:09.187 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records out 2024-03-04T09:24:09.187 INFO:teuthology.orchestra.run.smithi006.stderr:512 bytes copied, 0.000295643 s, 1.7 MB/s 2024-03-04T09:24:09.190 DEBUG:teuthology.orchestra.run.smithi006:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-04T09:24:09.231 DEBUG:teuthology.orchestra.run.smithi006:> stat /dev/vg_nvme/lv_4 2024-03-04T09:24:09.313 INFO:teuthology.orchestra.run.smithi006.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-04T09:24:09.313 INFO:teuthology.orchestra.run.smithi006.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-04T09:24:09.313 INFO:teuthology.orchestra.run.smithi006.stdout:Device: 5h/5d Inode: 691 Links: 1 2024-03-04T09:24:09.313 INFO:teuthology.orchestra.run.smithi006.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-04T09:24:09.313 INFO:teuthology.orchestra.run.smithi006.stdout:Context: system_u:object_r:device_t:s0 2024-03-04T09:24:09.313 INFO:teuthology.orchestra.run.smithi006.stdout:Access: 2024-03-04 09:23:57.644476968 +0000 2024-03-04T09:24:09.313 INFO:teuthology.orchestra.run.smithi006.stdout:Modify: 2024-03-04 09:21:21.520980237 +0000 2024-03-04T09:24:09.313 INFO:teuthology.orchestra.run.smithi006.stdout:Change: 2024-03-04 09:21:21.520980237 +0000 2024-03-04T09:24:09.314 INFO:teuthology.orchestra.run.smithi006.stdout: Birth: 2024-03-04 09:21:21.520980237 +0000 2024-03-04T09:24:09.314 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-04T09:24:09.423 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records in 2024-03-04T09:24:09.424 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records out 2024-03-04T09:24:09.424 INFO:teuthology.orchestra.run.smithi006.stderr:512 bytes copied, 0.000293844 s, 1.7 MB/s 2024-03-04T09:24:09.427 DEBUG:teuthology.orchestra.run.smithi006:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-04T09:24:09.469 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-04T09:24:09.469 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-04T09:24:09.469 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/scratch_devs of=/dev/stdout 2024-03-04T09:24:09.526 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-04T09:24:09.527 DEBUG:teuthology.orchestra.run.smithi146:> stat /dev/vg_nvme/lv_1 2024-03-04T09:24:09.577 INFO:teuthology.orchestra.run.smithi146.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-04T09:24:09.578 INFO:teuthology.orchestra.run.smithi146.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-04T09:24:09.578 INFO:teuthology.orchestra.run.smithi146.stdout:Device: 5h/5d Inode: 631 Links: 1 2024-03-04T09:24:09.578 INFO:teuthology.orchestra.run.smithi146.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-04T09:24:09.578 INFO:teuthology.orchestra.run.smithi146.stdout:Context: system_u:object_r:device_t:s0 2024-03-04T09:24:09.578 INFO:teuthology.orchestra.run.smithi146.stdout:Access: 2024-03-04 09:23:54.127275625 +0000 2024-03-04T09:24:09.578 INFO:teuthology.orchestra.run.smithi146.stdout:Modify: 2024-03-04 09:21:09.959383309 +0000 2024-03-04T09:24:09.578 INFO:teuthology.orchestra.run.smithi146.stdout:Change: 2024-03-04 09:21:09.959383309 +0000 2024-03-04T09:24:09.578 INFO:teuthology.orchestra.run.smithi146.stdout: Birth: 2024-03-04 09:21:09.959383309 +0000 2024-03-04T09:24:09.578 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-04T09:24:09.663 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records in 2024-03-04T09:24:09.663 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records out 2024-03-04T09:24:09.663 INFO:teuthology.orchestra.run.smithi146.stderr:512 bytes copied, 0.000330119 s, 1.6 MB/s 2024-03-04T09:24:09.668 DEBUG:teuthology.orchestra.run.smithi146:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-04T09:24:09.723 DEBUG:teuthology.orchestra.run.smithi146:> stat /dev/vg_nvme/lv_2 2024-03-04T09:24:09.773 INFO:teuthology.orchestra.run.smithi146.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-04T09:24:09.773 INFO:teuthology.orchestra.run.smithi146.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-04T09:24:09.773 INFO:teuthology.orchestra.run.smithi146.stdout:Device: 5h/5d Inode: 652 Links: 1 2024-03-04T09:24:09.773 INFO:teuthology.orchestra.run.smithi146.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-04T09:24:09.773 INFO:teuthology.orchestra.run.smithi146.stdout:Context: system_u:object_r:device_t:s0 2024-03-04T09:24:09.773 INFO:teuthology.orchestra.run.smithi146.stdout:Access: 2024-03-04 09:23:54.128275609 +0000 2024-03-04T09:24:09.773 INFO:teuthology.orchestra.run.smithi146.stdout:Modify: 2024-03-04 09:21:10.536374449 +0000 2024-03-04T09:24:09.773 INFO:teuthology.orchestra.run.smithi146.stdout:Change: 2024-03-04 09:21:10.536374449 +0000 2024-03-04T09:24:09.773 INFO:teuthology.orchestra.run.smithi146.stdout: Birth: 2024-03-04 09:21:10.536374449 +0000 2024-03-04T09:24:09.774 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-04T09:24:09.854 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records in 2024-03-04T09:24:09.854 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records out 2024-03-04T09:24:09.855 INFO:teuthology.orchestra.run.smithi146.stderr:512 bytes copied, 0.000341103 s, 1.5 MB/s 2024-03-04T09:24:09.857 DEBUG:teuthology.orchestra.run.smithi146:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-04T09:24:09.909 DEBUG:teuthology.orchestra.run.smithi146:> stat /dev/vg_nvme/lv_3 2024-03-04T09:24:09.956 INFO:teuthology.orchestra.run.smithi146.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-04T09:24:09.956 INFO:teuthology.orchestra.run.smithi146.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-04T09:24:09.956 INFO:teuthology.orchestra.run.smithi146.stdout:Device: 5h/5d Inode: 672 Links: 1 2024-03-04T09:24:09.956 INFO:teuthology.orchestra.run.smithi146.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-04T09:24:09.956 INFO:teuthology.orchestra.run.smithi146.stdout:Context: system_u:object_r:device_t:s0 2024-03-04T09:24:09.956 INFO:teuthology.orchestra.run.smithi146.stdout:Access: 2024-03-04 09:23:54.129275594 +0000 2024-03-04T09:24:09.956 INFO:teuthology.orchestra.run.smithi146.stdout:Modify: 2024-03-04 09:21:11.087365989 +0000 2024-03-04T09:24:09.957 INFO:teuthology.orchestra.run.smithi146.stdout:Change: 2024-03-04 09:21:11.087365989 +0000 2024-03-04T09:24:09.957 INFO:teuthology.orchestra.run.smithi146.stdout: Birth: 2024-03-04 09:21:11.087365989 +0000 2024-03-04T09:24:09.957 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-04T09:24:10.032 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records in 2024-03-04T09:24:10.032 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records out 2024-03-04T09:24:10.032 INFO:teuthology.orchestra.run.smithi146.stderr:512 bytes copied, 0.000306569 s, 1.7 MB/s 2024-03-04T09:24:10.035 DEBUG:teuthology.orchestra.run.smithi146:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-04T09:24:10.083 DEBUG:teuthology.orchestra.run.smithi146:> stat /dev/vg_nvme/lv_4 2024-03-04T09:24:10.129 INFO:teuthology.orchestra.run.smithi146.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-04T09:24:10.129 INFO:teuthology.orchestra.run.smithi146.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-04T09:24:10.129 INFO:teuthology.orchestra.run.smithi146.stdout:Device: 5h/5d Inode: 688 Links: 1 2024-03-04T09:24:10.129 INFO:teuthology.orchestra.run.smithi146.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-04T09:24:10.129 INFO:teuthology.orchestra.run.smithi146.stdout:Context: system_u:object_r:device_t:s0 2024-03-04T09:24:10.129 INFO:teuthology.orchestra.run.smithi146.stdout:Access: 2024-03-04 09:23:54.129275594 +0000 2024-03-04T09:24:10.129 INFO:teuthology.orchestra.run.smithi146.stdout:Modify: 2024-03-04 09:21:11.640357498 +0000 2024-03-04T09:24:10.130 INFO:teuthology.orchestra.run.smithi146.stdout:Change: 2024-03-04 09:21:11.640357498 +0000 2024-03-04T09:24:10.130 INFO:teuthology.orchestra.run.smithi146.stdout: Birth: 2024-03-04 09:21:11.640357498 +0000 2024-03-04T09:24:10.130 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-04T09:24:10.261 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records in 2024-03-04T09:24:10.261 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records out 2024-03-04T09:24:10.262 INFO:teuthology.orchestra.run.smithi146.stderr:512 bytes copied, 0.00034628 s, 1.5 MB/s 2024-03-04T09:24:10.266 DEBUG:teuthology.orchestra.run.smithi146:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-04T09:24:10.320 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-04T09:24:10.321 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi006.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi146.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-04T09:24:10.321 INFO:tasks.ceph:Generating config... 2024-03-04T09:24:10.322 INFO:tasks.ceph:[client] client mount timeout = 600 2024-03-04T09:24:10.322 INFO:tasks.ceph:[client] debug client = 20 2024-03-04T09:24:10.322 INFO:tasks.ceph:[client] debug ms = 1 2024-03-04T09:24:10.322 INFO:tasks.ceph:[client] pid file = 2024-03-04T09:24:10.322 INFO:tasks.ceph:[client] rados mon op timeout = 900 2024-03-04T09:24:10.322 INFO:tasks.ceph:[client] rados osd op timeout = 900 2024-03-04T09:24:10.322 INFO:tasks.ceph:[global] ms die on skipped message = False 2024-03-04T09:24:10.322 INFO:tasks.ceph:[mds] debug mds = 20 2024-03-04T09:24:10.322 INFO:tasks.ceph:[mds] debug mds balancer = 20 2024-03-04T09:24:10.323 INFO:tasks.ceph:[mds] debug ms = 1 2024-03-04T09:24:10.323 INFO:tasks.ceph:[mds] mds debug frag = True 2024-03-04T09:24:10.323 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2024-03-04T09:24:10.323 INFO:tasks.ceph:[mds] mds op complaint time = 180 2024-03-04T09:24:10.323 INFO:tasks.ceph:[mds] mds verify scatter = True 2024-03-04T09:24:10.323 INFO:tasks.ceph:[mds] osd op complaint time = 180 2024-03-04T09:24:10.323 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2024-03-04T09:24:10.323 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2024-03-04T09:24:10.323 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-04T09:24:10.323 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-04T09:24:10.323 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-04T09:24:10.323 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-04T09:24:10.323 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-04T09:24:10.323 INFO:tasks.ceph:[mon] mon op complaint time = 120 2024-03-04T09:24:10.323 INFO:tasks.ceph:[osd] bdev async discard = True 2024-03-04T09:24:10.323 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-03-04T09:24:10.324 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2024-03-04T09:24:10.324 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-04T09:24:10.324 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-04T09:24:10.324 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-03-04T09:24:10.324 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-03-04T09:24:10.324 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-04T09:24:10.324 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-04T09:24:10.324 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-03-04T09:24:10.324 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-04T09:24:10.324 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-04T09:24:10.324 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-04T09:24:10.324 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-04T09:24:10.324 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-04T09:24:10.324 INFO:tasks.ceph:[osd] osd op complaint time = 180 2024-03-04T09:24:10.325 INFO:tasks.ceph:Setting up mon.a... 2024-03-04T09:24:10.325 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-04T09:24:10.486 INFO:teuthology.orchestra.run.smithi006.stdout:creating /etc/ceph/ceph.keyring 2024-03-04T09:24:10.495 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-04T09:24:10.641 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-04T09:24:10.733 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.6'), ('mon.b', '172.21.15.146'), ('mon.c', '[v2:172.21.15.146:3301,v1:172.21.15.146:6790]')] 2024-03-04T09:24:10.733 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.6,172.21.15.146,[v2:172.21.15.146:3301,v1:172.21.15.146:6790]', 'ms die on skipped message': False}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds.d': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2024-03-04T09:24:10.734 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-04T09:24:10.734 DEBUG:teuthology.orchestra.run.smithi006:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-04T09:24:10.792 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.6 --add b 172.21.15.146 --addv c '[v2:172.21.15.146:3301,v1:172.21.15.146:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-04T09:24:10.952 INFO:teuthology.orchestra.run.smithi006.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-04T09:24:10.952 INFO:teuthology.orchestra.run.smithi006.stdout:monmaptool: generated fsid edce0578-5ae4-4402-b45a-552bdef84d7c 2024-03-04T09:24:10.952 INFO:teuthology.orchestra.run.smithi006.stdout:setting min_mon_release = quincy 2024-03-04T09:24:10.953 INFO:teuthology.orchestra.run.smithi006.stdout:epoch 0 2024-03-04T09:24:10.953 INFO:teuthology.orchestra.run.smithi006.stdout:fsid edce0578-5ae4-4402-b45a-552bdef84d7c 2024-03-04T09:24:10.953 INFO:teuthology.orchestra.run.smithi006.stdout:last_changed 2024-03-04T09:24:10.956887+0000 2024-03-04T09:24:10.953 INFO:teuthology.orchestra.run.smithi006.stdout:created 2024-03-04T09:24:10.956887+0000 2024-03-04T09:24:10.953 INFO:teuthology.orchestra.run.smithi006.stdout:min_mon_release 17 (quincy) 2024-03-04T09:24:10.953 INFO:teuthology.orchestra.run.smithi006.stdout:election_strategy: 1 2024-03-04T09:24:10.953 INFO:teuthology.orchestra.run.smithi006.stdout:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2024-03-04T09:24:10.953 INFO:teuthology.orchestra.run.smithi006.stdout:1: [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] mon.b 2024-03-04T09:24:10.953 INFO:teuthology.orchestra.run.smithi006.stdout:2: [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] mon.c 2024-03-04T09:24:10.953 INFO:teuthology.orchestra.run.smithi006.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-04T09:24:10.956 DEBUG:teuthology.orchestra.run.smithi006:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-04T09:24:11.009 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID edce0578-5ae4-4402-b45a-552bdef84d7c... 2024-03-04T09:24:11.010 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-04T09:24:11.015 DEBUG:teuthology.orchestra.run.smithi146:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-04T09:24:11.176 INFO:teuthology.orchestra.run.smithi146.stdout:[global] 2024-03-04T09:24:11.176 INFO:teuthology.orchestra.run.smithi146.stdout: chdir = "" 2024-03-04T09:24:11.176 INFO:teuthology.orchestra.run.smithi146.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-04T09:24:11.176 INFO:teuthology.orchestra.run.smithi146.stdout: auth supported = cephx 2024-03-04T09:24:11.176 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:24:11.176 INFO:teuthology.orchestra.run.smithi146.stdout: filestore xattr use omap = true 2024-03-04T09:24:11.176 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:24:11.177 INFO:teuthology.orchestra.run.smithi146.stdout: mon clock drift allowed = 1.000 2024-03-04T09:24:11.177 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:24:11.177 INFO:teuthology.orchestra.run.smithi146.stdout: osd crush chooseleaf type = 0 2024-03-04T09:24:11.177 INFO:teuthology.orchestra.run.smithi146.stdout: auth debug = true 2024-03-04T09:24:11.177 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:24:11.177 INFO:teuthology.orchestra.run.smithi146.stdout: ms die on old message = true 2024-03-04T09:24:11.177 INFO:teuthology.orchestra.run.smithi146.stdout: ms die on bug = true 2024-03-04T09:24:11.177 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:24:11.177 INFO:teuthology.orchestra.run.smithi146.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-04T09:24:11.177 INFO:teuthology.orchestra.run.smithi146.stdout: mon pg warn max object skew = 0 2024-03-04T09:24:11.177 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:24:11.177 INFO:teuthology.orchestra.run.smithi146.stdout: # disable pg_autoscaler by default for new pools 2024-03-04T09:24:11.177 INFO:teuthology.orchestra.run.smithi146.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-04T09:24:11.177 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:24:11.177 INFO:teuthology.orchestra.run.smithi146.stdout: osd pool default size = 2 2024-03-04T09:24:11.178 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:24:11.178 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd allow primary affinity = true 2024-03-04T09:24:11.178 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd allow pg remap = true 2024-03-04T09:24:11.178 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on legacy crush tunables = false 2024-03-04T09:24:11.178 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on crush straw calc version zero = false 2024-03-04T09:24:11.178 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on no sortbitwise = false 2024-03-04T09:24:11.178 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on osd down out interval zero = false 2024-03-04T09:24:11.178 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on too few osds = false 2024-03-04T09:24:11.178 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-04T09:24:11.178 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-04T09:24:11.178 INFO:teuthology.orchestra.run.smithi146.stdout: mon_allow_pool_size_one = true 2024-03-04T09:24:11.178 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:24:11.178 INFO:teuthology.orchestra.run.smithi146.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-04T09:24:11.178 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:24:11.178 INFO:teuthology.orchestra.run.smithi146.stdout: osd default data pool replay window = 5 2024-03-04T09:24:11.179 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:24:11.179 INFO:teuthology.orchestra.run.smithi146.stdout: mon allow pool delete = true 2024-03-04T09:24:11.179 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:24:11.179 INFO:teuthology.orchestra.run.smithi146.stdout: mon cluster log file level = debug 2024-03-04T09:24:11.179 INFO:teuthology.orchestra.run.smithi146.stdout: debug asserts on shutdown = true 2024-03-04T09:24:11.179 INFO:teuthology.orchestra.run.smithi146.stdout: mon health detail to clog = false 2024-03-04T09:24:11.179 INFO:teuthology.orchestra.run.smithi146.stdout: mon host = "172.21.15.6,172.21.15.146,[v2:172.21.15.146:3301,v1:172.21.15.146:6790]" 2024-03-04T09:24:11.179 INFO:teuthology.orchestra.run.smithi146.stdout: ms die on skipped message = False 2024-03-04T09:24:11.179 INFO:teuthology.orchestra.run.smithi146.stdout: fsid = edce0578-5ae4-4402-b45a-552bdef84d7c 2024-03-04T09:24:11.179 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:24:11.179 INFO:teuthology.orchestra.run.smithi146.stdout:[osd] 2024-03-04T09:24:11.179 INFO:teuthology.orchestra.run.smithi146.stdout: osd journal size = 100 2024-03-04T09:24:11.179 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:24:11.179 INFO:teuthology.orchestra.run.smithi146.stdout: osd scrub load threshold = 5.0 2024-03-04T09:24:11.179 INFO:teuthology.orchestra.run.smithi146.stdout: osd scrub max interval = 600 2024-03-04T09:24:11.180 INFO:teuthology.orchestra.run.smithi146.stdout: osd mclock profile = high_recovery_ops 2024-03-04T09:24:11.180 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:24:11.180 INFO:teuthology.orchestra.run.smithi146.stdout: osd recover clone overlap = true 2024-03-04T09:24:11.180 INFO:teuthology.orchestra.run.smithi146.stdout: osd recovery max chunk = 1048576 2024-03-04T09:24:11.180 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:24:11.180 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug shutdown = true 2024-03-04T09:24:11.180 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug op order = true 2024-03-04T09:24:11.180 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug verify stray on activate = true 2024-03-04T09:24:11.180 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug trim objects = true 2024-03-04T09:24:11.180 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:24:11.180 INFO:teuthology.orchestra.run.smithi146.stdout: osd open classes on start = true 2024-03-04T09:24:11.180 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug pg log writeout = true 2024-03-04T09:24:11.180 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:24:11.180 INFO:teuthology.orchestra.run.smithi146.stdout: osd deep scrub update digest min age = 30 2024-03-04T09:24:11.180 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:24:11.181 INFO:teuthology.orchestra.run.smithi146.stdout: osd map max advance = 10 2024-03-04T09:24:11.181 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:24:11.181 INFO:teuthology.orchestra.run.smithi146.stdout: journal zero on create = true 2024-03-04T09:24:11.181 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:24:11.181 INFO:teuthology.orchestra.run.smithi146.stdout: filestore ondisk finisher threads = 3 2024-03-04T09:24:11.181 INFO:teuthology.orchestra.run.smithi146.stdout: filestore apply finisher threads = 3 2024-03-04T09:24:11.181 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:24:11.181 INFO:teuthology.orchestra.run.smithi146.stdout: bdev debug aio = true 2024-03-04T09:24:11.181 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug misdirected ops = true 2024-03-04T09:24:11.181 INFO:teuthology.orchestra.run.smithi146.stdout: bdev async discard = True 2024-03-04T09:24:11.181 INFO:teuthology.orchestra.run.smithi146.stdout: bdev enable discard = True 2024-03-04T09:24:11.181 INFO:teuthology.orchestra.run.smithi146.stdout: bluestore allocator = bitmap 2024-03-04T09:24:11.181 INFO:teuthology.orchestra.run.smithi146.stdout: bluestore block size = 96636764160 2024-03-04T09:24:11.181 INFO:teuthology.orchestra.run.smithi146.stdout: bluestore fsck on mount = True 2024-03-04T09:24:11.181 INFO:teuthology.orchestra.run.smithi146.stdout: debug bluefs = 1/20 2024-03-04T09:24:11.182 INFO:teuthology.orchestra.run.smithi146.stdout: debug bluestore = 1/20 2024-03-04T09:24:11.182 INFO:teuthology.orchestra.run.smithi146.stdout: debug ms = 1 2024-03-04T09:24:11.182 INFO:teuthology.orchestra.run.smithi146.stdout: debug osd = 20 2024-03-04T09:24:11.182 INFO:teuthology.orchestra.run.smithi146.stdout: debug rocksdb = 4/10 2024-03-04T09:24:11.182 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd backfillfull_ratio = 0.85 2024-03-04T09:24:11.182 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd full ratio = 0.9 2024-03-04T09:24:11.182 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd nearfull ratio = 0.8 2024-03-04T09:24:11.182 INFO:teuthology.orchestra.run.smithi146.stdout: osd failsafe full ratio = 0.95 2024-03-04T09:24:11.182 INFO:teuthology.orchestra.run.smithi146.stdout: osd objectstore = bluestore 2024-03-04T09:24:11.182 INFO:teuthology.orchestra.run.smithi146.stdout: osd op complaint time = 180 2024-03-04T09:24:11.182 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:24:11.182 INFO:teuthology.orchestra.run.smithi146.stdout:[mgr] 2024-03-04T09:24:11.182 INFO:teuthology.orchestra.run.smithi146.stdout: debug ms = 1 2024-03-04T09:24:11.182 INFO:teuthology.orchestra.run.smithi146.stdout: debug mgr = 20 2024-03-04T09:24:11.182 INFO:teuthology.orchestra.run.smithi146.stdout: debug mon = 20 2024-03-04T09:24:11.182 INFO:teuthology.orchestra.run.smithi146.stdout: debug auth = 20 2024-03-04T09:24:11.183 INFO:teuthology.orchestra.run.smithi146.stdout: mon reweight min pgs per osd = 4 2024-03-04T09:24:11.183 INFO:teuthology.orchestra.run.smithi146.stdout: mon reweight min bytes per osd = 10 2024-03-04T09:24:11.183 INFO:teuthology.orchestra.run.smithi146.stdout: mgr/telemetry/nag = false 2024-03-04T09:24:11.183 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:24:11.183 INFO:teuthology.orchestra.run.smithi146.stdout:[mon] 2024-03-04T09:24:11.183 INFO:teuthology.orchestra.run.smithi146.stdout: debug ms = 1 2024-03-04T09:24:11.183 INFO:teuthology.orchestra.run.smithi146.stdout: debug mon = 20 2024-03-04T09:24:11.183 INFO:teuthology.orchestra.run.smithi146.stdout: debug paxos = 20 2024-03-04T09:24:11.183 INFO:teuthology.orchestra.run.smithi146.stdout: debug auth = 20 2024-03-04T09:24:11.183 INFO:teuthology.orchestra.run.smithi146.stdout: mon data avail warn = 5 2024-03-04T09:24:11.183 INFO:teuthology.orchestra.run.smithi146.stdout: mon mgr mkfs grace = 240 2024-03-04T09:24:11.183 INFO:teuthology.orchestra.run.smithi146.stdout: mon reweight min pgs per osd = 4 2024-03-04T09:24:11.183 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd reporter subtree level = osd 2024-03-04T09:24:11.183 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd prime pg temp = true 2024-03-04T09:24:11.183 INFO:teuthology.orchestra.run.smithi146.stdout: mon reweight min bytes per osd = 10 2024-03-04T09:24:11.184 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:24:11.184 INFO:teuthology.orchestra.run.smithi146.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-04T09:24:11.184 INFO:teuthology.orchestra.run.smithi146.stdout: auth mon ticket ttl = 660 # 11m 2024-03-04T09:24:11.184 INFO:teuthology.orchestra.run.smithi146.stdout: auth service ticket ttl = 240 # 4m 2024-03-04T09:24:11.184 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:24:11.184 INFO:teuthology.orchestra.run.smithi146.stdout: # don't complain about insecure global_id in the test suite 2024-03-04T09:24:11.184 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-04T09:24:11.184 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-04T09:24:11.184 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:24:11.184 INFO:teuthology.orchestra.run.smithi146.stdout: # 1m isn't quite enough 2024-03-04T09:24:11.184 INFO:teuthology.orchestra.run.smithi146.stdout: mon_down_mkfs_grace = 2m 2024-03-04T09:24:11.184 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:24:11.184 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_filestore_osds = false 2024-03-04T09:24:11.184 INFO:teuthology.orchestra.run.smithi146.stdout: mon op complaint time = 120 2024-03-04T09:24:11.184 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:24:11.185 INFO:teuthology.orchestra.run.smithi146.stdout:[client] 2024-03-04T09:24:11.185 INFO:teuthology.orchestra.run.smithi146.stdout: rgw cache enabled = true 2024-03-04T09:24:11.185 INFO:teuthology.orchestra.run.smithi146.stdout: rgw enable ops log = true 2024-03-04T09:24:11.185 INFO:teuthology.orchestra.run.smithi146.stdout: rgw enable usage log = true 2024-03-04T09:24:11.185 INFO:teuthology.orchestra.run.smithi146.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-04T09:24:11.185 INFO:teuthology.orchestra.run.smithi146.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-04T09:24:11.185 INFO:teuthology.orchestra.run.smithi146.stdout: client mount timeout = 600 2024-03-04T09:24:11.185 INFO:teuthology.orchestra.run.smithi146.stdout: debug client = 20 2024-03-04T09:24:11.185 INFO:teuthology.orchestra.run.smithi146.stdout: debug ms = 1 2024-03-04T09:24:11.185 INFO:teuthology.orchestra.run.smithi146.stdout: pid file = "" 2024-03-04T09:24:11.185 INFO:teuthology.orchestra.run.smithi146.stdout: rados mon op timeout = 900 2024-03-04T09:24:11.185 INFO:teuthology.orchestra.run.smithi146.stdout: rados osd op timeout = 900 2024-03-04T09:24:11.185 INFO:teuthology.orchestra.run.smithi146.stdout:[mon.a] 2024-03-04T09:24:11.185 INFO:teuthology.orchestra.run.smithi146.stdout:[mon.b] 2024-03-04T09:24:11.186 INFO:teuthology.orchestra.run.smithi146.stdout:[mon.c] 2024-03-04T09:24:11.186 INFO:teuthology.orchestra.run.smithi146.stdout:[mds.a] 2024-03-04T09:24:11.186 INFO:teuthology.orchestra.run.smithi146.stdout:[mds.c] 2024-03-04T09:24:11.186 INFO:teuthology.orchestra.run.smithi146.stdout:[mds.b] 2024-03-04T09:24:11.186 INFO:teuthology.orchestra.run.smithi146.stdout:[mds.d] 2024-03-04T09:24:11.186 INFO:teuthology.orchestra.run.smithi146.stdout:[mds] 2024-03-04T09:24:11.186 INFO:teuthology.orchestra.run.smithi146.stdout: debug mds = 20 2024-03-04T09:24:11.186 INFO:teuthology.orchestra.run.smithi146.stdout: debug mds balancer = 20 2024-03-04T09:24:11.186 INFO:teuthology.orchestra.run.smithi146.stdout: debug ms = 1 2024-03-04T09:24:11.186 INFO:teuthology.orchestra.run.smithi146.stdout: mds debug frag = True 2024-03-04T09:24:11.186 INFO:teuthology.orchestra.run.smithi146.stdout: mds debug scatterstat = True 2024-03-04T09:24:11.186 INFO:teuthology.orchestra.run.smithi146.stdout: mds op complaint time = 180 2024-03-04T09:24:11.186 INFO:teuthology.orchestra.run.smithi146.stdout: mds verify scatter = True 2024-03-04T09:24:11.186 INFO:teuthology.orchestra.run.smithi146.stdout: osd op complaint time = 180 2024-03-04T09:24:11.186 INFO:teuthology.orchestra.run.smithi146.stdout: rados mon op timeout = 900 2024-03-04T09:24:11.186 INFO:teuthology.orchestra.run.smithi146.stdout: rados osd op timeout = 900 2024-03-04T09:24:11.190 INFO:teuthology.orchestra.run.smithi006.stdout:[global] 2024-03-04T09:24:11.190 INFO:teuthology.orchestra.run.smithi006.stdout: chdir = "" 2024-03-04T09:24:11.190 INFO:teuthology.orchestra.run.smithi006.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-04T09:24:11.190 INFO:teuthology.orchestra.run.smithi006.stdout: auth supported = cephx 2024-03-04T09:24:11.190 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:24:11.190 INFO:teuthology.orchestra.run.smithi006.stdout: filestore xattr use omap = true 2024-03-04T09:24:11.190 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:24:11.190 INFO:teuthology.orchestra.run.smithi006.stdout: mon clock drift allowed = 1.000 2024-03-04T09:24:11.190 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:24:11.190 INFO:teuthology.orchestra.run.smithi006.stdout: osd crush chooseleaf type = 0 2024-03-04T09:24:11.191 INFO:teuthology.orchestra.run.smithi006.stdout: auth debug = true 2024-03-04T09:24:11.191 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:24:11.191 INFO:teuthology.orchestra.run.smithi006.stdout: ms die on old message = true 2024-03-04T09:24:11.191 INFO:teuthology.orchestra.run.smithi006.stdout: ms die on bug = true 2024-03-04T09:24:11.191 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:24:11.191 INFO:teuthology.orchestra.run.smithi006.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-04T09:24:11.191 INFO:teuthology.orchestra.run.smithi006.stdout: mon pg warn max object skew = 0 2024-03-04T09:24:11.191 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:24:11.191 INFO:teuthology.orchestra.run.smithi006.stdout: # disable pg_autoscaler by default for new pools 2024-03-04T09:24:11.191 INFO:teuthology.orchestra.run.smithi006.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-04T09:24:11.191 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:24:11.191 INFO:teuthology.orchestra.run.smithi006.stdout: osd pool default size = 2 2024-03-04T09:24:11.191 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:24:11.191 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd allow primary affinity = true 2024-03-04T09:24:11.191 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd allow pg remap = true 2024-03-04T09:24:11.192 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on legacy crush tunables = false 2024-03-04T09:24:11.192 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on crush straw calc version zero = false 2024-03-04T09:24:11.192 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on no sortbitwise = false 2024-03-04T09:24:11.192 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on osd down out interval zero = false 2024-03-04T09:24:11.192 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on too few osds = false 2024-03-04T09:24:11.192 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-04T09:24:11.192 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-04T09:24:11.192 INFO:teuthology.orchestra.run.smithi006.stdout: mon_allow_pool_size_one = true 2024-03-04T09:24:11.192 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:24:11.192 INFO:teuthology.orchestra.run.smithi006.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-04T09:24:11.192 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:24:11.192 INFO:teuthology.orchestra.run.smithi006.stdout: osd default data pool replay window = 5 2024-03-04T09:24:11.192 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:24:11.192 INFO:teuthology.orchestra.run.smithi006.stdout: mon allow pool delete = true 2024-03-04T09:24:11.192 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:24:11.193 INFO:teuthology.orchestra.run.smithi006.stdout: mon cluster log file level = debug 2024-03-04T09:24:11.193 INFO:teuthology.orchestra.run.smithi006.stdout: debug asserts on shutdown = true 2024-03-04T09:24:11.193 INFO:teuthology.orchestra.run.smithi006.stdout: mon health detail to clog = false 2024-03-04T09:24:11.193 INFO:teuthology.orchestra.run.smithi006.stdout: mon host = "172.21.15.6,172.21.15.146,[v2:172.21.15.146:3301,v1:172.21.15.146:6790]" 2024-03-04T09:24:11.193 INFO:teuthology.orchestra.run.smithi006.stdout: ms die on skipped message = False 2024-03-04T09:24:11.193 INFO:teuthology.orchestra.run.smithi006.stdout: fsid = edce0578-5ae4-4402-b45a-552bdef84d7c 2024-03-04T09:24:11.193 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:24:11.193 INFO:teuthology.orchestra.run.smithi006.stdout:[osd] 2024-03-04T09:24:11.193 INFO:teuthology.orchestra.run.smithi006.stdout: osd journal size = 100 2024-03-04T09:24:11.193 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:24:11.193 INFO:teuthology.orchestra.run.smithi006.stdout: osd scrub load threshold = 5.0 2024-03-04T09:24:11.193 INFO:teuthology.orchestra.run.smithi006.stdout: osd scrub max interval = 600 2024-03-04T09:24:11.193 INFO:teuthology.orchestra.run.smithi006.stdout: osd mclock profile = high_recovery_ops 2024-03-04T09:24:11.193 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:24:11.193 INFO:teuthology.orchestra.run.smithi006.stdout: osd recover clone overlap = true 2024-03-04T09:24:11.193 INFO:teuthology.orchestra.run.smithi006.stdout: osd recovery max chunk = 1048576 2024-03-04T09:24:11.194 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:24:11.194 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug shutdown = true 2024-03-04T09:24:11.194 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug op order = true 2024-03-04T09:24:11.194 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug verify stray on activate = true 2024-03-04T09:24:11.194 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug trim objects = true 2024-03-04T09:24:11.194 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:24:11.194 INFO:teuthology.orchestra.run.smithi006.stdout: osd open classes on start = true 2024-03-04T09:24:11.194 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug pg log writeout = true 2024-03-04T09:24:11.194 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:24:11.194 INFO:teuthology.orchestra.run.smithi006.stdout: osd deep scrub update digest min age = 30 2024-03-04T09:24:11.194 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:24:11.194 INFO:teuthology.orchestra.run.smithi006.stdout: osd map max advance = 10 2024-03-04T09:24:11.194 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:24:11.194 INFO:teuthology.orchestra.run.smithi006.stdout: journal zero on create = true 2024-03-04T09:24:11.194 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:24:11.195 INFO:teuthology.orchestra.run.smithi006.stdout: filestore ondisk finisher threads = 3 2024-03-04T09:24:11.195 INFO:teuthology.orchestra.run.smithi006.stdout: filestore apply finisher threads = 3 2024-03-04T09:24:11.195 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:24:11.195 INFO:teuthology.orchestra.run.smithi006.stdout: bdev debug aio = true 2024-03-04T09:24:11.195 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug misdirected ops = true 2024-03-04T09:24:11.195 INFO:teuthology.orchestra.run.smithi006.stdout: bdev async discard = True 2024-03-04T09:24:11.195 INFO:teuthology.orchestra.run.smithi006.stdout: bdev enable discard = True 2024-03-04T09:24:11.195 INFO:teuthology.orchestra.run.smithi006.stdout: bluestore allocator = bitmap 2024-03-04T09:24:11.195 INFO:teuthology.orchestra.run.smithi006.stdout: bluestore block size = 96636764160 2024-03-04T09:24:11.195 INFO:teuthology.orchestra.run.smithi006.stdout: bluestore fsck on mount = True 2024-03-04T09:24:11.195 INFO:teuthology.orchestra.run.smithi006.stdout: debug bluefs = 1/20 2024-03-04T09:24:11.195 INFO:teuthology.orchestra.run.smithi006.stdout: debug bluestore = 1/20 2024-03-04T09:24:11.195 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2024-03-04T09:24:11.195 INFO:teuthology.orchestra.run.smithi006.stdout: debug osd = 20 2024-03-04T09:24:11.195 INFO:teuthology.orchestra.run.smithi006.stdout: debug rocksdb = 4/10 2024-03-04T09:24:11.195 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd backfillfull_ratio = 0.85 2024-03-04T09:24:11.196 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd full ratio = 0.9 2024-03-04T09:24:11.196 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd nearfull ratio = 0.8 2024-03-04T09:24:11.196 INFO:teuthology.orchestra.run.smithi006.stdout: osd failsafe full ratio = 0.95 2024-03-04T09:24:11.196 INFO:teuthology.orchestra.run.smithi006.stdout: osd objectstore = bluestore 2024-03-04T09:24:11.196 INFO:teuthology.orchestra.run.smithi006.stdout: osd op complaint time = 180 2024-03-04T09:24:11.196 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:24:11.196 INFO:teuthology.orchestra.run.smithi006.stdout:[mgr] 2024-03-04T09:24:11.196 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2024-03-04T09:24:11.196 INFO:teuthology.orchestra.run.smithi006.stdout: debug mgr = 20 2024-03-04T09:24:11.196 INFO:teuthology.orchestra.run.smithi006.stdout: debug mon = 20 2024-03-04T09:24:11.196 INFO:teuthology.orchestra.run.smithi006.stdout: debug auth = 20 2024-03-04T09:24:11.196 INFO:teuthology.orchestra.run.smithi006.stdout: mon reweight min pgs per osd = 4 2024-03-04T09:24:11.196 INFO:teuthology.orchestra.run.smithi006.stdout: mon reweight min bytes per osd = 10 2024-03-04T09:24:11.196 INFO:teuthology.orchestra.run.smithi006.stdout: mgr/telemetry/nag = false 2024-03-04T09:24:11.196 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:24:11.197 INFO:teuthology.orchestra.run.smithi006.stdout:[mon] 2024-03-04T09:24:11.197 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2024-03-04T09:24:11.197 INFO:teuthology.orchestra.run.smithi006.stdout: debug mon = 20 2024-03-04T09:24:11.197 INFO:teuthology.orchestra.run.smithi006.stdout: debug paxos = 20 2024-03-04T09:24:11.197 INFO:teuthology.orchestra.run.smithi006.stdout: debug auth = 20 2024-03-04T09:24:11.197 INFO:teuthology.orchestra.run.smithi006.stdout: mon data avail warn = 5 2024-03-04T09:24:11.197 INFO:teuthology.orchestra.run.smithi006.stdout: mon mgr mkfs grace = 240 2024-03-04T09:24:11.197 INFO:teuthology.orchestra.run.smithi006.stdout: mon reweight min pgs per osd = 4 2024-03-04T09:24:11.197 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd reporter subtree level = osd 2024-03-04T09:24:11.197 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd prime pg temp = true 2024-03-04T09:24:11.197 INFO:teuthology.orchestra.run.smithi006.stdout: mon reweight min bytes per osd = 10 2024-03-04T09:24:11.197 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:24:11.197 INFO:teuthology.orchestra.run.smithi006.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-04T09:24:11.197 INFO:teuthology.orchestra.run.smithi006.stdout: auth mon ticket ttl = 660 # 11m 2024-03-04T09:24:11.197 INFO:teuthology.orchestra.run.smithi006.stdout: auth service ticket ttl = 240 # 4m 2024-03-04T09:24:11.198 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:24:11.198 INFO:teuthology.orchestra.run.smithi006.stdout: # don't complain about insecure global_id in the test suite 2024-03-04T09:24:11.198 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-04T09:24:11.198 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-04T09:24:11.198 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:24:11.198 INFO:teuthology.orchestra.run.smithi006.stdout: # 1m isn't quite enough 2024-03-04T09:24:11.198 INFO:teuthology.orchestra.run.smithi006.stdout: mon_down_mkfs_grace = 2m 2024-03-04T09:24:11.198 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:24:11.198 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_filestore_osds = false 2024-03-04T09:24:11.198 INFO:teuthology.orchestra.run.smithi006.stdout: mon op complaint time = 120 2024-03-04T09:24:11.198 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:24:11.198 INFO:teuthology.orchestra.run.smithi006.stdout:[client] 2024-03-04T09:24:11.198 INFO:teuthology.orchestra.run.smithi006.stdout: rgw cache enabled = true 2024-03-04T09:24:11.198 INFO:teuthology.orchestra.run.smithi006.stdout: rgw enable ops log = true 2024-03-04T09:24:11.198 INFO:teuthology.orchestra.run.smithi006.stdout: rgw enable usage log = true 2024-03-04T09:24:11.199 INFO:teuthology.orchestra.run.smithi006.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-04T09:24:11.199 INFO:teuthology.orchestra.run.smithi006.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-04T09:24:11.199 INFO:teuthology.orchestra.run.smithi006.stdout: client mount timeout = 600 2024-03-04T09:24:11.199 INFO:teuthology.orchestra.run.smithi006.stdout: debug client = 20 2024-03-04T09:24:11.199 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2024-03-04T09:24:11.199 INFO:teuthology.orchestra.run.smithi006.stdout: pid file = "" 2024-03-04T09:24:11.199 INFO:teuthology.orchestra.run.smithi006.stdout: rados mon op timeout = 900 2024-03-04T09:24:11.199 INFO:teuthology.orchestra.run.smithi006.stdout: rados osd op timeout = 900 2024-03-04T09:24:11.199 INFO:teuthology.orchestra.run.smithi006.stdout:[mon.a] 2024-03-04T09:24:11.199 INFO:teuthology.orchestra.run.smithi006.stdout:[mon.b] 2024-03-04T09:24:11.199 INFO:teuthology.orchestra.run.smithi006.stdout:[mon.c] 2024-03-04T09:24:11.199 INFO:teuthology.orchestra.run.smithi006.stdout:[mds.a] 2024-03-04T09:24:11.199 INFO:teuthology.orchestra.run.smithi006.stdout:[mds.c] 2024-03-04T09:24:11.199 INFO:teuthology.orchestra.run.smithi006.stdout:[mds.b] 2024-03-04T09:24:11.199 INFO:teuthology.orchestra.run.smithi006.stdout:[mds.d] 2024-03-04T09:24:11.199 INFO:teuthology.orchestra.run.smithi006.stdout:[mds] 2024-03-04T09:24:11.200 INFO:teuthology.orchestra.run.smithi006.stdout: debug mds = 20 2024-03-04T09:24:11.200 INFO:teuthology.orchestra.run.smithi006.stdout: debug mds balancer = 20 2024-03-04T09:24:11.200 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2024-03-04T09:24:11.200 INFO:teuthology.orchestra.run.smithi006.stdout: mds debug frag = True 2024-03-04T09:24:11.200 INFO:teuthology.orchestra.run.smithi006.stdout: mds debug scatterstat = True 2024-03-04T09:24:11.200 INFO:teuthology.orchestra.run.smithi006.stdout: mds op complaint time = 180 2024-03-04T09:24:11.200 INFO:teuthology.orchestra.run.smithi006.stdout: mds verify scatter = True 2024-03-04T09:24:11.200 INFO:teuthology.orchestra.run.smithi006.stdout: osd op complaint time = 180 2024-03-04T09:24:11.200 INFO:teuthology.orchestra.run.smithi006.stdout: rados mon op timeout = 900 2024-03-04T09:24:11.200 INFO:teuthology.orchestra.run.smithi006.stdout: rados osd op timeout = 900 2024-03-04T09:24:11.239 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-04T09:24:11.239 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-04T09:24:11.385 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-04T09:24:11.385 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-04T09:24:11.385 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-04T09:24:11.441 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-04T09:24:11.441 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-04T09:24:11.493 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi006.front.sepia.ceph.com 2024-03-04T09:24:11.493 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-04T09:24:11.493 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-04T09:24:11.493 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-04T09:24:11.635 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-04T09:24:11.635 DEBUG:teuthology.orchestra.run.smithi006:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-04T09:24:11.732 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi146.front.sepia.ceph.com 2024-03-04T09:24:11.732 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-04T09:24:11.732 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-04T09:24:11.732 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-04T09:24:11.862 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-04T09:24:11.862 DEBUG:teuthology.orchestra.run.smithi146:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-04T09:24:11.957 INFO:tasks.ceph:Setting up mon nodes... 2024-03-04T09:24:11.958 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-04T09:24:11.958 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-04T09:24:12.138 INFO:teuthology.orchestra.run.smithi006.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-04T09:24:12.149 DEBUG:teuthology.orchestra.run.smithi146:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-04T09:24:12.342 INFO:teuthology.orchestra.run.smithi146.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-04T09:24:12.353 INFO:tasks.ceph:Setting up mds nodes... 2024-03-04T09:24:12.354 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2024-03-04T09:24:12.546 INFO:teuthology.orchestra.run.smithi006.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2024-03-04T09:24:12.555 DEBUG:teuthology.orchestra.run.smithi006:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2024-03-04T09:24:12.657 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2024-03-04T09:24:12.848 INFO:teuthology.orchestra.run.smithi006.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2024-03-04T09:24:12.857 DEBUG:teuthology.orchestra.run.smithi006:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2024-03-04T09:24:12.956 DEBUG:teuthology.orchestra.run.smithi146:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2024-03-04T09:24:13.144 INFO:teuthology.orchestra.run.smithi146.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2024-03-04T09:24:13.154 DEBUG:teuthology.orchestra.run.smithi146:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2024-03-04T09:24:13.251 DEBUG:teuthology.orchestra.run.smithi146:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2024-03-04T09:24:13.425 INFO:teuthology.orchestra.run.smithi146.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2024-03-04T09:24:13.434 DEBUG:teuthology.orchestra.run.smithi146:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2024-03-04T09:24:13.525 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-04T09:24:13.525 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-04T09:24:13.661 INFO:teuthology.orchestra.run.smithi006.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-04T09:24:13.721 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.2 /etc/ceph/ceph.client.2.keyring && sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2024-03-04T09:24:13.851 INFO:teuthology.orchestra.run.smithi006.stdout:creating /etc/ceph/ceph.client.2.keyring 2024-03-04T09:24:13.901 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-04T09:24:14.030 INFO:teuthology.orchestra.run.smithi146.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-04T09:24:14.083 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.3 /etc/ceph/ceph.client.3.keyring && sudo chmod 0644 /etc/ceph/ceph.client.3.keyring 2024-03-04T09:24:14.202 INFO:teuthology.orchestra.run.smithi146.stdout:creating /etc/ceph/ceph.client.3.keyring 2024-03-04T09:24:14.256 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-04T09:24:14.256 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi006.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi146.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-04T09:24:14.256 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-04T09:24:14.361 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-04T09:24:14.361 INFO:tasks.ceph:role: osd.0 2024-03-04T09:24:14.361 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi006.front.sepia.ceph.com 2024-03-04T09:24:14.361 DEBUG:teuthology.orchestra.run.smithi006:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-04T09:24:14.466 INFO:teuthology.orchestra.run.smithi006.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-04T09:24:14.467 INFO:teuthology.orchestra.run.smithi006.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-04T09:24:14.467 INFO:teuthology.orchestra.run.smithi006.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-04T09:24:14.467 INFO:teuthology.orchestra.run.smithi006.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-04T09:24:14.467 INFO:teuthology.orchestra.run.smithi006.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-04T09:24:14.467 INFO:teuthology.orchestra.run.smithi006.stdout: = sunit=0 swidth=0 blks 2024-03-04T09:24:14.467 INFO:teuthology.orchestra.run.smithi006.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-04T09:24:14.467 INFO:teuthology.orchestra.run.smithi006.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-04T09:24:14.467 INFO:teuthology.orchestra.run.smithi006.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-04T09:24:14.467 INFO:teuthology.orchestra.run.smithi006.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-04T09:24:14.964 INFO:teuthology.orchestra.run.smithi006.stdout:Discarding blocks...Done. 2024-03-04T09:24:14.973 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi006.front.sepia.ceph.com -o noatime 2024-03-04T09:24:14.973 DEBUG:teuthology.orchestra.run.smithi006:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-04T09:24:15.145 DEBUG:teuthology.orchestra.run.smithi006:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-04T09:24:15.254 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-04T09:24:15.351 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-04T09:24:15.351 INFO:tasks.ceph:role: osd.1 2024-03-04T09:24:15.351 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi006.front.sepia.ceph.com 2024-03-04T09:24:15.351 DEBUG:teuthology.orchestra.run.smithi006:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-04T09:24:15.454 INFO:teuthology.orchestra.run.smithi006.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-04T09:24:15.454 INFO:teuthology.orchestra.run.smithi006.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-04T09:24:15.454 INFO:teuthology.orchestra.run.smithi006.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-04T09:24:15.454 INFO:teuthology.orchestra.run.smithi006.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-04T09:24:15.454 INFO:teuthology.orchestra.run.smithi006.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-04T09:24:15.454 INFO:teuthology.orchestra.run.smithi006.stdout: = sunit=0 swidth=0 blks 2024-03-04T09:24:15.454 INFO:teuthology.orchestra.run.smithi006.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-04T09:24:15.454 INFO:teuthology.orchestra.run.smithi006.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-04T09:24:15.454 INFO:teuthology.orchestra.run.smithi006.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-04T09:24:15.455 INFO:teuthology.orchestra.run.smithi006.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-04T09:24:15.939 INFO:teuthology.orchestra.run.smithi006.stdout:Discarding blocks...Done. 2024-03-04T09:24:15.947 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi006.front.sepia.ceph.com -o noatime 2024-03-04T09:24:15.947 DEBUG:teuthology.orchestra.run.smithi006:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-04T09:24:16.094 DEBUG:teuthology.orchestra.run.smithi006:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-04T09:24:16.189 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-04T09:24:16.280 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-04T09:24:16.280 INFO:tasks.ceph:role: osd.2 2024-03-04T09:24:16.280 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi006.front.sepia.ceph.com 2024-03-04T09:24:16.280 DEBUG:teuthology.orchestra.run.smithi006:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-04T09:24:16.381 INFO:teuthology.orchestra.run.smithi006.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-04T09:24:16.381 INFO:teuthology.orchestra.run.smithi006.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-04T09:24:16.381 INFO:teuthology.orchestra.run.smithi006.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-04T09:24:16.381 INFO:teuthology.orchestra.run.smithi006.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-04T09:24:16.381 INFO:teuthology.orchestra.run.smithi006.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-04T09:24:16.381 INFO:teuthology.orchestra.run.smithi006.stdout: = sunit=0 swidth=0 blks 2024-03-04T09:24:16.381 INFO:teuthology.orchestra.run.smithi006.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-04T09:24:16.381 INFO:teuthology.orchestra.run.smithi006.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-04T09:24:16.381 INFO:teuthology.orchestra.run.smithi006.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-04T09:24:16.382 INFO:teuthology.orchestra.run.smithi006.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-04T09:24:16.900 INFO:teuthology.orchestra.run.smithi006.stdout:Discarding blocks...Done. 2024-03-04T09:24:16.907 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi006.front.sepia.ceph.com -o noatime 2024-03-04T09:24:16.907 DEBUG:teuthology.orchestra.run.smithi006:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-04T09:24:17.109 DEBUG:teuthology.orchestra.run.smithi006:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-04T09:24:17.222 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-04T09:24:17.324 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-04T09:24:17.324 INFO:tasks.ceph:role: osd.3 2024-03-04T09:24:17.324 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi006.front.sepia.ceph.com 2024-03-04T09:24:17.325 DEBUG:teuthology.orchestra.run.smithi006:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-04T09:24:17.428 INFO:teuthology.orchestra.run.smithi006.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-04T09:24:17.428 INFO:teuthology.orchestra.run.smithi006.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-04T09:24:17.428 INFO:teuthology.orchestra.run.smithi006.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-04T09:24:17.428 INFO:teuthology.orchestra.run.smithi006.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-04T09:24:17.428 INFO:teuthology.orchestra.run.smithi006.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-04T09:24:17.428 INFO:teuthology.orchestra.run.smithi006.stdout: = sunit=0 swidth=0 blks 2024-03-04T09:24:17.428 INFO:teuthology.orchestra.run.smithi006.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-04T09:24:17.428 INFO:teuthology.orchestra.run.smithi006.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-04T09:24:17.429 INFO:teuthology.orchestra.run.smithi006.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-04T09:24:17.429 INFO:teuthology.orchestra.run.smithi006.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-04T09:24:17.903 INFO:teuthology.orchestra.run.smithi006.stdout:Discarding blocks...Done. 2024-03-04T09:24:17.909 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi006.front.sepia.ceph.com -o noatime 2024-03-04T09:24:17.909 DEBUG:teuthology.orchestra.run.smithi006:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-04T09:24:18.059 DEBUG:teuthology.orchestra.run.smithi006:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-04T09:24:18.159 DEBUG:teuthology.orchestra.run.smithi006:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-04T09:24:18.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:18.318+0000 7f2272006640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-04T09:24:18.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:18.320+0000 7f2272006640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-04T09:24:18.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:18.320+0000 7f2272006640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-04T09:24:18.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:18.321+0000 7f2272006640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-04T09:24:18.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:18.321+0000 7f2272006640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-04T09:24:18.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:18.332+0000 7f2272006640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-04T09:24:21.497 DEBUG:teuthology.orchestra.run.smithi006:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-04T09:24:21.601 DEBUG:teuthology.orchestra.run.smithi006:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-04T09:24:21.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:21.751+0000 7f436bbdd640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-04T09:24:21.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:21.752+0000 7f436bbdd640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-04T09:24:21.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:21.752+0000 7f436bbdd640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-04T09:24:21.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:21.752+0000 7f436bbdd640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-04T09:24:21.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:21.752+0000 7f436bbdd640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-04T09:24:21.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:21.754+0000 7f436bbdd640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-04T09:24:24.936 DEBUG:teuthology.orchestra.run.smithi006:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-04T09:24:25.037 DEBUG:teuthology.orchestra.run.smithi006:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-04T09:24:25.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:25.190+0000 7fc14059a640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-04T09:24:25.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:25.190+0000 7fc14059a640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-04T09:24:25.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:25.190+0000 7fc14059a640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-04T09:24:25.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:25.191+0000 7fc14059a640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-04T09:24:25.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:25.191+0000 7fc14059a640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-04T09:24:25.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:25.193+0000 7fc14059a640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-04T09:24:28.631 DEBUG:teuthology.orchestra.run.smithi006:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-04T09:24:28.723 DEBUG:teuthology.orchestra.run.smithi006:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-04T09:24:28.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:28.874+0000 7fa526149640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-04T09:24:28.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:28.875+0000 7fa526149640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-04T09:24:28.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:28.875+0000 7fa526149640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-04T09:24:28.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:28.875+0000 7fa526149640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-04T09:24:28.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:28.875+0000 7fa526149640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-04T09:24:28.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:28.877+0000 7fa526149640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-04T09:24:32.058 DEBUG:teuthology.orchestra.run.smithi006:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-04T09:24:32.161 DEBUG:teuthology.orchestra.run.smithi146:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-04T09:24:32.263 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-04T09:24:32.263 INFO:tasks.ceph:role: osd.4 2024-03-04T09:24:32.263 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi146.front.sepia.ceph.com 2024-03-04T09:24:32.263 DEBUG:teuthology.orchestra.run.smithi146:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-04T09:24:32.362 INFO:teuthology.orchestra.run.smithi146.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-04T09:24:32.362 INFO:teuthology.orchestra.run.smithi146.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-04T09:24:32.362 INFO:teuthology.orchestra.run.smithi146.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-04T09:24:32.362 INFO:teuthology.orchestra.run.smithi146.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-04T09:24:32.362 INFO:teuthology.orchestra.run.smithi146.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-04T09:24:32.363 INFO:teuthology.orchestra.run.smithi146.stdout: = sunit=0 swidth=0 blks 2024-03-04T09:24:32.363 INFO:teuthology.orchestra.run.smithi146.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-04T09:24:32.363 INFO:teuthology.orchestra.run.smithi146.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-04T09:24:32.363 INFO:teuthology.orchestra.run.smithi146.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-04T09:24:32.363 INFO:teuthology.orchestra.run.smithi146.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-04T09:24:32.889 INFO:teuthology.orchestra.run.smithi146.stdout:Discarding blocks...Done. 2024-03-04T09:24:32.895 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi146.front.sepia.ceph.com -o noatime 2024-03-04T09:24:32.896 DEBUG:teuthology.orchestra.run.smithi146:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-04T09:24:33.081 DEBUG:teuthology.orchestra.run.smithi146:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-04T09:24:33.167 DEBUG:teuthology.orchestra.run.smithi146:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-04T09:24:33.242 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-04T09:24:33.242 INFO:tasks.ceph:role: osd.5 2024-03-04T09:24:33.242 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi146.front.sepia.ceph.com 2024-03-04T09:24:33.242 DEBUG:teuthology.orchestra.run.smithi146:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-04T09:24:33.322 INFO:teuthology.orchestra.run.smithi146.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-04T09:24:33.322 INFO:teuthology.orchestra.run.smithi146.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-04T09:24:33.322 INFO:teuthology.orchestra.run.smithi146.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-04T09:24:33.322 INFO:teuthology.orchestra.run.smithi146.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-04T09:24:33.322 INFO:teuthology.orchestra.run.smithi146.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-04T09:24:33.322 INFO:teuthology.orchestra.run.smithi146.stdout: = sunit=0 swidth=0 blks 2024-03-04T09:24:33.322 INFO:teuthology.orchestra.run.smithi146.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-04T09:24:33.322 INFO:teuthology.orchestra.run.smithi146.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-04T09:24:33.322 INFO:teuthology.orchestra.run.smithi146.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-04T09:24:33.322 INFO:teuthology.orchestra.run.smithi146.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-04T09:24:33.802 INFO:teuthology.orchestra.run.smithi146.stdout:Discarding blocks...Done. 2024-03-04T09:24:33.808 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi146.front.sepia.ceph.com -o noatime 2024-03-04T09:24:33.808 DEBUG:teuthology.orchestra.run.smithi146:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-04T09:24:33.925 DEBUG:teuthology.orchestra.run.smithi146:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-04T09:24:34.008 DEBUG:teuthology.orchestra.run.smithi146:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-04T09:24:34.081 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-04T09:24:34.081 INFO:tasks.ceph:role: osd.6 2024-03-04T09:24:34.081 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi146.front.sepia.ceph.com 2024-03-04T09:24:34.081 DEBUG:teuthology.orchestra.run.smithi146:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-04T09:24:34.162 INFO:teuthology.orchestra.run.smithi146.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-04T09:24:34.162 INFO:teuthology.orchestra.run.smithi146.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-04T09:24:34.162 INFO:teuthology.orchestra.run.smithi146.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-04T09:24:34.162 INFO:teuthology.orchestra.run.smithi146.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-04T09:24:34.163 INFO:teuthology.orchestra.run.smithi146.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-04T09:24:34.163 INFO:teuthology.orchestra.run.smithi146.stdout: = sunit=0 swidth=0 blks 2024-03-04T09:24:34.163 INFO:teuthology.orchestra.run.smithi146.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-04T09:24:34.163 INFO:teuthology.orchestra.run.smithi146.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-04T09:24:34.163 INFO:teuthology.orchestra.run.smithi146.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-04T09:24:34.163 INFO:teuthology.orchestra.run.smithi146.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-04T09:24:34.690 INFO:teuthology.orchestra.run.smithi146.stdout:Discarding blocks...Done. 2024-03-04T09:24:34.697 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi146.front.sepia.ceph.com -o noatime 2024-03-04T09:24:34.697 DEBUG:teuthology.orchestra.run.smithi146:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-04T09:24:34.880 DEBUG:teuthology.orchestra.run.smithi146:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-04T09:24:34.983 DEBUG:teuthology.orchestra.run.smithi146:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-04T09:24:35.080 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-04T09:24:35.080 INFO:tasks.ceph:role: osd.7 2024-03-04T09:24:35.080 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi146.front.sepia.ceph.com 2024-03-04T09:24:35.080 DEBUG:teuthology.orchestra.run.smithi146:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-04T09:24:35.175 INFO:teuthology.orchestra.run.smithi146.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-04T09:24:35.176 INFO:teuthology.orchestra.run.smithi146.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-04T09:24:35.176 INFO:teuthology.orchestra.run.smithi146.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-04T09:24:35.176 INFO:teuthology.orchestra.run.smithi146.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-04T09:24:35.176 INFO:teuthology.orchestra.run.smithi146.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-04T09:24:35.176 INFO:teuthology.orchestra.run.smithi146.stdout: = sunit=0 swidth=0 blks 2024-03-04T09:24:35.176 INFO:teuthology.orchestra.run.smithi146.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-04T09:24:35.176 INFO:teuthology.orchestra.run.smithi146.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-04T09:24:35.176 INFO:teuthology.orchestra.run.smithi146.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-04T09:24:35.176 INFO:teuthology.orchestra.run.smithi146.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-04T09:24:35.641 INFO:teuthology.orchestra.run.smithi146.stdout:Discarding blocks...Done. 2024-03-04T09:24:35.647 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi146.front.sepia.ceph.com -o noatime 2024-03-04T09:24:35.647 DEBUG:teuthology.orchestra.run.smithi146:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-04T09:24:35.792 DEBUG:teuthology.orchestra.run.smithi146:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-04T09:24:35.876 DEBUG:teuthology.orchestra.run.smithi146:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-04T09:24:36.016 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:36.020+0000 7f74a340e640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-04T09:24:36.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:36.023+0000 7f74a340e640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-04T09:24:36.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:36.023+0000 7f74a340e640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-04T09:24:36.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:36.023+0000 7f74a340e640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-04T09:24:36.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:36.023+0000 7f74a340e640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-04T09:24:36.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:36.030+0000 7f74a340e640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-04T09:24:38.307 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:24:38.315 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:24:39.209 DEBUG:teuthology.orchestra.run.smithi146:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-04T09:24:39.305 DEBUG:teuthology.orchestra.run.smithi146:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-04T09:24:39.441 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:39.444+0000 7f6242815640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-04T09:24:39.441 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:39.445+0000 7f6242815640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-04T09:24:39.442 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:39.445+0000 7f6242815640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-04T09:24:39.442 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:39.445+0000 7f6242815640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-04T09:24:39.442 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:39.445+0000 7f6242815640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-04T09:24:39.443 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:39.447+0000 7f6242815640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-04T09:24:42.369 DEBUG:teuthology.orchestra.run.smithi146:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-04T09:24:42.467 DEBUG:teuthology.orchestra.run.smithi146:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-04T09:24:42.608 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:42.611+0000 7f1e0e013640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-04T09:24:42.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:42.612+0000 7f1e0e013640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-04T09:24:42.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:42.612+0000 7f1e0e013640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-04T09:24:42.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:42.612+0000 7f1e0e013640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-04T09:24:42.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:42.612+0000 7f1e0e013640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-04T09:24:42.611 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:42.614+0000 7f1e0e013640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-04T09:24:45.543 DEBUG:teuthology.orchestra.run.smithi146:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-04T09:24:45.635 DEBUG:teuthology.orchestra.run.smithi146:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-04T09:24:45.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:45.774+0000 7f53409bb640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-04T09:24:45.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:45.775+0000 7f53409bb640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-04T09:24:45.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:45.775+0000 7f53409bb640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-04T09:24:45.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:45.775+0000 7f53409bb640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-04T09:24:45.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:45.775+0000 7f53409bb640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-04T09:24:45.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:45.777+0000 7f53409bb640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-04T09:24:48.699 DEBUG:teuthology.orchestra.run.smithi146:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-04T09:24:48.794 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-04T09:24:48.794 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-04T09:24:48.794 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-04T09:24:48.903 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-04T09:24:48.904 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2024-03-04T09:24:49.008 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-04T09:24:49.008 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2024-03-04T09:24:49.109 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-04T09:24:49.110 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-04T09:24:49.200 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-04T09:24:49.201 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-04T09:24:49.290 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-04T09:24:49.290 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-04T09:24:49.392 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-04T09:24:49.392 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-04T09:24:49.494 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-04T09:24:49.494 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-04T09:24:49.591 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-04T09:24:49.591 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2024-03-04T09:24:49.690 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-04T09:24:49.691 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2024-03-04T09:24:49.782 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-04T09:24:49.782 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-04T09:24:49.880 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-04T09:24:49.880 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-04T09:24:49.977 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-04T09:24:49.978 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-04T09:24:50.074 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-04T09:24:50.074 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-04T09:24:50.166 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-04T09:24:50.166 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-04T09:24:50.226 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-04T09:24:50.226 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/etc/ceph/ceph.client.2.keyring of=/dev/stdout 2024-03-04T09:24:50.281 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-04T09:24:50.281 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-04T09:24:50.336 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-04T09:24:50.336 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/etc/ceph/ceph.client.3.keyring of=/dev/stdout 2024-03-04T09:24:50.386 INFO:tasks.ceph:Adding keys to all mons... 2024-03-04T09:24:50.386 DEBUG:teuthology.orchestra.run.smithi006:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-04T09:24:50.394 DEBUG:teuthology.orchestra.run.smithi146:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-04T09:24:50.481 INFO:teuthology.orchestra.run.smithi146.stdout:[mgr.y] 2024-03-04T09:24:50.481 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQA8k+VlEA2ACBAAXofDj3gmqItVTJ1pYLqZyQ== 2024-03-04T09:24:50.481 INFO:teuthology.orchestra.run.smithi146.stdout:[mds.a] 2024-03-04T09:24:50.481 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQA8k+VlHCzFIBAAnKYlmCMDw45LqnSjxnZY5Q== 2024-03-04T09:24:50.481 INFO:teuthology.orchestra.run.smithi146.stdout:[mds.c] 2024-03-04T09:24:50.481 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQA8k+Vl0pjOMhAA7WcTw8awv4EAsrpP4yncNA== 2024-03-04T09:24:50.481 INFO:teuthology.orchestra.run.smithi146.stdout:[osd.0] 2024-03-04T09:24:50.481 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQBCk+VlyjMMExAAqb2FJ9n5PWehfx/6PB19ig== 2024-03-04T09:24:50.481 INFO:teuthology.orchestra.run.smithi146.stdout:[osd.1] 2024-03-04T09:24:50.481 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQBFk+VlXD3cLBAAGclSivcYWDslqN8MLlj7eQ== 2024-03-04T09:24:50.482 INFO:teuthology.orchestra.run.smithi146.stdout:[osd.2] 2024-03-04T09:24:50.482 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQBJk+Vl7d9nCxAAnb3W4EI3Yf3XY8ht3VnKEQ== 2024-03-04T09:24:50.482 INFO:teuthology.orchestra.run.smithi146.stdout:[osd.3] 2024-03-04T09:24:50.482 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQBMk+Vlv+Y1NBAAiRYSUqov2i2sj3iInfpHvA== 2024-03-04T09:24:50.482 INFO:teuthology.orchestra.run.smithi146.stdout:[mgr.x] 2024-03-04T09:24:50.482 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQA8k+VlPFatFBAA/o3Ps/uJrcg1738HOGy/FA== 2024-03-04T09:24:50.482 INFO:teuthology.orchestra.run.smithi146.stdout:[mds.b] 2024-03-04T09:24:50.482 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQA9k+VlNY7fCBAAX3z8X72/6LkQD6mwGYT69g== 2024-03-04T09:24:50.482 INFO:teuthology.orchestra.run.smithi146.stdout:[mds.d] 2024-03-04T09:24:50.482 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQA9k+VlbladGRAABNgUcaqTVYuJfCqy5c/G0g== 2024-03-04T09:24:50.482 INFO:teuthology.orchestra.run.smithi146.stdout:[osd.4] 2024-03-04T09:24:50.482 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQBUk+Vl1OlAARAAxMV4GUGS+RJ6XlrRhrVKcg== 2024-03-04T09:24:50.482 INFO:teuthology.orchestra.run.smithi146.stdout:[osd.5] 2024-03-04T09:24:50.482 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQBXk+Vl1lKLGhAAHlsiRr+1+OSRWhkB+weIJg== 2024-03-04T09:24:50.482 INFO:teuthology.orchestra.run.smithi146.stdout:[osd.6] 2024-03-04T09:24:50.482 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQBak+VlwB+GJBAA3f9KUz+zZlQuUR/ywQfkDA== 2024-03-04T09:24:50.483 INFO:teuthology.orchestra.run.smithi146.stdout:[osd.7] 2024-03-04T09:24:50.483 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQBdk+Vle4o8LhAAko+iH5VqqCsfWkNfKZjGVA== 2024-03-04T09:24:50.483 INFO:teuthology.orchestra.run.smithi146.stdout:[client.0] 2024-03-04T09:24:50.483 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQA9k+Vl4L+rJxAACaDbQ4/Bybp5hAZAnJzlVw== 2024-03-04T09:24:50.483 INFO:teuthology.orchestra.run.smithi146.stdout:[client.2] 2024-03-04T09:24:50.483 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQA9k+VlUc8DMxAA8pGUtIh+BO6UKrrYnCqVgQ== 2024-03-04T09:24:50.483 INFO:teuthology.orchestra.run.smithi146.stdout:[client.1] 2024-03-04T09:24:50.483 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQA+k+VlGjkOAhAAJNzB8+tQl3mqCnTgMn9EVw== 2024-03-04T09:24:50.483 INFO:teuthology.orchestra.run.smithi146.stdout:[client.3] 2024-03-04T09:24:50.483 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQA+k+VlzMFKDBAAGr713I2hcmGMuHMIjNknYA== 2024-03-04T09:24:50.486 INFO:teuthology.orchestra.run.smithi006.stdout:[mgr.y] 2024-03-04T09:24:50.486 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQA8k+VlEA2ACBAAXofDj3gmqItVTJ1pYLqZyQ== 2024-03-04T09:24:50.486 INFO:teuthology.orchestra.run.smithi006.stdout:[mds.a] 2024-03-04T09:24:50.486 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQA8k+VlHCzFIBAAnKYlmCMDw45LqnSjxnZY5Q== 2024-03-04T09:24:50.487 INFO:teuthology.orchestra.run.smithi006.stdout:[mds.c] 2024-03-04T09:24:50.487 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQA8k+Vl0pjOMhAA7WcTw8awv4EAsrpP4yncNA== 2024-03-04T09:24:50.487 INFO:teuthology.orchestra.run.smithi006.stdout:[osd.0] 2024-03-04T09:24:50.487 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQBCk+VlyjMMExAAqb2FJ9n5PWehfx/6PB19ig== 2024-03-04T09:24:50.487 INFO:teuthology.orchestra.run.smithi006.stdout:[osd.1] 2024-03-04T09:24:50.487 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQBFk+VlXD3cLBAAGclSivcYWDslqN8MLlj7eQ== 2024-03-04T09:24:50.487 INFO:teuthology.orchestra.run.smithi006.stdout:[osd.2] 2024-03-04T09:24:50.487 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQBJk+Vl7d9nCxAAnb3W4EI3Yf3XY8ht3VnKEQ== 2024-03-04T09:24:50.487 INFO:teuthology.orchestra.run.smithi006.stdout:[osd.3] 2024-03-04T09:24:50.487 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQBMk+Vlv+Y1NBAAiRYSUqov2i2sj3iInfpHvA== 2024-03-04T09:24:50.487 INFO:teuthology.orchestra.run.smithi006.stdout:[mgr.x] 2024-03-04T09:24:50.487 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQA8k+VlPFatFBAA/o3Ps/uJrcg1738HOGy/FA== 2024-03-04T09:24:50.487 INFO:teuthology.orchestra.run.smithi006.stdout:[mds.b] 2024-03-04T09:24:50.487 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQA9k+VlNY7fCBAAX3z8X72/6LkQD6mwGYT69g== 2024-03-04T09:24:50.488 INFO:teuthology.orchestra.run.smithi006.stdout:[mds.d] 2024-03-04T09:24:50.488 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQA9k+VlbladGRAABNgUcaqTVYuJfCqy5c/G0g== 2024-03-04T09:24:50.488 INFO:teuthology.orchestra.run.smithi006.stdout:[osd.4] 2024-03-04T09:24:50.488 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQBUk+Vl1OlAARAAxMV4GUGS+RJ6XlrRhrVKcg== 2024-03-04T09:24:50.488 INFO:teuthology.orchestra.run.smithi006.stdout:[osd.5] 2024-03-04T09:24:50.488 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQBXk+Vl1lKLGhAAHlsiRr+1+OSRWhkB+weIJg== 2024-03-04T09:24:50.488 INFO:teuthology.orchestra.run.smithi006.stdout:[osd.6] 2024-03-04T09:24:50.488 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQBak+VlwB+GJBAA3f9KUz+zZlQuUR/ywQfkDA== 2024-03-04T09:24:50.488 INFO:teuthology.orchestra.run.smithi006.stdout:[osd.7] 2024-03-04T09:24:50.488 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQBdk+Vle4o8LhAAko+iH5VqqCsfWkNfKZjGVA== 2024-03-04T09:24:50.488 INFO:teuthology.orchestra.run.smithi006.stdout:[client.0] 2024-03-04T09:24:50.488 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQA9k+Vl4L+rJxAACaDbQ4/Bybp5hAZAnJzlVw== 2024-03-04T09:24:50.488 INFO:teuthology.orchestra.run.smithi006.stdout:[client.2] 2024-03-04T09:24:50.488 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQA9k+VlUc8DMxAA8pGUtIh+BO6UKrrYnCqVgQ== 2024-03-04T09:24:50.488 INFO:teuthology.orchestra.run.smithi006.stdout:[client.1] 2024-03-04T09:24:50.488 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQA+k+VlGjkOAhAAJNzB8+tQl3mqCnTgMn9EVw== 2024-03-04T09:24:50.488 INFO:teuthology.orchestra.run.smithi006.stdout:[client.3] 2024-03-04T09:24:50.489 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQA+k+VlzMFKDBAAGr713I2hcmGMuHMIjNknYA== 2024-03-04T09:24:50.491 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-04T09:24:50.499 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-04T09:24:50.671 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-04T09:24:50.676 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-04T09:24:50.825 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-04T09:24:50.832 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-04T09:24:50.980 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-04T09:24:50.985 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-04T09:24:51.131 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-04T09:24:51.138 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-04T09:24:51.283 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-04T09:24:51.289 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-04T09:24:51.438 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-04T09:24:51.443 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-04T09:24:51.587 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-04T09:24:51.595 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-04T09:24:51.740 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-04T09:24:51.745 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-04T09:24:51.888 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-04T09:24:51.894 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-04T09:24:52.040 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-04T09:24:52.045 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-04T09:24:52.189 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-04T09:24:52.193 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-04T09:24:52.343 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-04T09:24:52.350 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-04T09:24:52.496 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-04T09:24:52.500 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-04T09:24:52.641 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-04T09:24:52.646 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-04T09:24:52.795 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-04T09:24:52.801 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-04T09:24:52.945 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-04T09:24:52.950 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-04T09:24:53.097 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-04T09:24:53.103 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-04T09:24:53.251 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-04T09:24:53.251 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-04T09:24:53.345 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-04T09:24:53.499 DEBUG:teuthology.orchestra.run.smithi006:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-04T09:24:53.596 DEBUG:teuthology.orchestra.run.smithi146:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-04T09:24:53.697 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-04T09:24:53.843 DEBUG:teuthology.orchestra.run.smithi146:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-04T09:24:53.925 DEBUG:teuthology.orchestra.run.smithi146:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-04T09:24:54.021 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-04T09:24:54.177 DEBUG:teuthology.orchestra.run.smithi146:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-04T09:24:54.250 DEBUG:teuthology.orchestra.run.smithi006:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-04T09:24:54.258 DEBUG:teuthology.orchestra.run.smithi146:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-04T09:24:54.312 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-04T09:24:54.312 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-04T09:24:54.312 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-03-04T09:24:54.319 INFO:tasks.ceph.mon.a:Started 2024-03-04T09:24:54.320 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-04T09:24:54.320 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-03-04T09:24:54.324 INFO:tasks.ceph.mon.b:Started 2024-03-04T09:24:54.325 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-04T09:24:54.325 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-03-04T09:24:54.329 INFO:tasks.ceph.mon.c:Started 2024-03-04T09:24:54.329 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-04T09:24:54.329 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-04T09:24:54.329 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-03-04T09:24:54.335 INFO:tasks.ceph.mgr.y:Started 2024-03-04T09:24:54.335 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-04T09:24:54.335 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-03-04T09:24:54.340 INFO:tasks.ceph.mgr.x:Started 2024-03-04T09:24:54.340 INFO:tasks.ceph:Setting crush tunables to default 2024-03-04T09:24:54.340 DEBUG:teuthology.orchestra.run.smithi006:> sudo ceph --cluster ceph osd crush tunables default 2024-03-04T09:24:54.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:54.527+0000 7fc9f7200640 1 Processor -- start 2024-03-04T09:24:54.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:54.528+0000 7fc9f7200640 1 -- start start 2024-03-04T09:24:54.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:54.529+0000 7fc9f7200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc9f80d4860 0x7fc9f80d0630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:24:54.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:54.529+0000 7fc9f7200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9f80ceca0 con 0x7fc9f815dee0 2024-03-04T09:24:54.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:54.529+0000 7fc9f7200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9f80cef80 con 0x7fc9f80cfa60 2024-03-04T09:24:54.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:54.529+0000 7fc9f7200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9f80cf5d0 con 0x7fc9f80d4860 2024-03-04T09:24:54.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:54.529+0000 7fc9f5400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc9f80d4860 0x7fc9f80d0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:24:54.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:54.529+0000 7fc9f5400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc9f80d4860 0x7fc9f80d0630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.6:39248/0 (socket says 172.21.15.6:39248) 2024-03-04T09:24:54.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:54.529+0000 7fc9f5400640 1 -- 172.21.15.6:0/66210296 learned_addr learned my addr 172.21.15.6:0/66210296 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:24:54.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:54.529+0000 7fc9f5400640 1 -- 172.21.15.6:0/66210296 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc9f80d4860 msgr2=0x7fc9f80d0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-04T09:24:54.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:54.529+0000 7fc9f5400640 1 -- 172.21.15.6:0/66210296 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc9f80d4860 msgr2=0x7fc9f80d0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-04T09:24:54.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:54.529+0000 7fc9f5400640 1 --2- 172.21.15.6:0/66210296 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc9f80d4860 0x7fc9f80d0630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-04T09:24:54.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:54.529+0000 7fc9f5400640 1 --2- 172.21.15.6:0/66210296 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc9f80d4860 0x7fc9f80d0630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-04T09:24:54.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:54.540+0000 7fc9f4a00640 1 -- 172.21.15.6:0/66210296 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1695775354 0 0) 0x7fc9f80ceca0 con 0x7fc9f815dee0 2024-03-04T09:24:54.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:54.540+0000 7fc9f4a00640 1 -- 172.21.15.6:0/66210296 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9dc003520 con 0x7fc9f815dee0 2024-03-04T09:24:54.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:54.541+0000 7fc9f4a00640 1 -- 172.21.15.6:0/66210296 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3882447727 0 0) 0x7fc9dc003520 con 0x7fc9f815dee0 2024-03-04T09:24:54.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:54.541+0000 7fc9f4a00640 1 -- 172.21.15.6:0/66210296 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc9f80d4860 msgr2=0x7fc9f80d0630 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-04T09:24:54.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:54.541+0000 7fc9f4a00640 1 --2- 172.21.15.6:0/66210296 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc9f80d4860 0x7fc9f80d0630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:24:54.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:54.541+0000 7fc9f4a00640 1 -- 172.21.15.6:0/66210296 >> v1:172.21.15.146:6789/0 conn(0x7fc9f80cfa60 legacy=0x7fc9f80b3510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:24:54.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:54.541+0000 7fc9f4a00640 1 -- 172.21.15.6:0/66210296 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9f80cf260 con 0x7fc9f815dee0 2024-03-04T09:24:54.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:54.541+0000 7fc9f4a00640 1 -- 172.21.15.6:0/66210296 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7fc9e8001670 con 0x7fc9f815dee0 2024-03-04T09:24:54.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:54.541+0000 7fc9f4a00640 1 -- 172.21.15.6:0/66210296 >> v1:172.21.15.6:6789/0 conn(0x7fc9f815dee0 legacy=0x7fc9f80d18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:24:54.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:54.541+0000 7fc9f4a00640 1 --2- 172.21.15.6:0/66210296 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc9dc0039e0 0x7fc9dc003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:24:54.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:54.541+0000 7fc9f4a00640 1 --2- 172.21.15.6:0/66210296 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc9dc004800 0x7fc9dc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:24:54.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:54.541+0000 7fc9f4a00640 1 --2- 172.21.15.6:0/66210296 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc9f80d4860 0x7fc9dc005250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:24:54.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:54.541+0000 7fc9f4a00640 1 -- 172.21.15.6:0/66210296 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9f80cf5d0 con 0x7fc9dc004800 2024-03-04T09:24:54.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:54.541+0000 7fc9f4a00640 1 -- 172.21.15.6:0/66210296 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9dc00bb50 con 0x7fc9dc0039e0 2024-03-04T09:24:54.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:54.541+0000 7fc9f4a00640 1 -- 172.21.15.6:0/66210296 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9dc00bd50 con 0x7fc9f80d4860 2024-03-04T09:24:54.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:54.541+0000 7fc9f5e00640 1 --2- 172.21.15.6:0/66210296 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc9dc0039e0 0x7fc9dc003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:24:54.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:54.541+0000 7fc9f5400640 1 --2- 172.21.15.6:0/66210296 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc9dc004800 0x7fc9dc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:24:54.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:54.541+0000 7fc9f6800640 1 --2- 172.21.15.6:0/66210296 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc9f80d4860 0x7fc9dc005250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:24:54.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:54.542+0000 7fc9f5400640 1 -- 172.21.15.6:0/66210296 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc9f80d4860 msgr2=0x7fc9dc005250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:24:54.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:54.542+0000 7fc9f5400640 1 --2- 172.21.15.6:0/66210296 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc9f80d4860 0x7fc9dc005250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:24:54.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:54.542+0000 7fc9f5400640 1 -- 172.21.15.6:0/66210296 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc9dc0039e0 msgr2=0x7fc9dc003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:24:54.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:54.542+0000 7fc9f5400640 1 --2- 172.21.15.6:0/66210296 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc9dc0039e0 0x7fc9dc003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:24:54.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:54.542+0000 7fc9f5400640 1 -- 172.21.15.6:0/66210296 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9dc003520 con 0x7fc9dc004800 2024-03-04T09:24:54.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:54.542+0000 7fc9f5e00640 1 --2- 172.21.15.6:0/66210296 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc9dc0039e0 0x7fc9dc003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T09:24:54.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:54.543+0000 7fc9f5400640 1 --2- 172.21.15.6:0/66210296 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc9dc004800 0x7fc9dc004c80 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fc9e4009100 tx=0x7fc9dc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=888fb9c985a0285e server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:24:54.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:54.543+0000 7fc9f4a00640 1 -- 172.21.15.6:0/66210296 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9e4034070 con 0x7fc9dc004800 2024-03-04T09:24:54.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:54.543+0000 7fc9f4a00640 1 -- 172.21.15.6:0/66210296 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9e4030630 con 0x7fc9dc004800 2024-03-04T09:24:54.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:54.543+0000 7fc9f4a00640 1 -- 172.21.15.6:0/66210296 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9e400f040 con 0x7fc9dc004800 2024-03-04T09:24:54.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:54.544+0000 7fc9f7200640 1 -- 172.21.15.6:0/66210296 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc9dc004800 msgr2=0x7fc9dc004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:24:54.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:54.544+0000 7fc9f7200640 1 --2- 172.21.15.6:0/66210296 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc9dc004800 0x7fc9dc004c80 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fc9e4009100 tx=0x7fc9dc005830 comp rx=0 tx=0).stop 2024-03-04T09:24:54.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:54.544+0000 7fc9f6800640 1 -- 172.21.15.6:0/66210296 reap_dead start 2024-03-04T09:24:54.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:54.544+0000 7fc9f7200640 1 -- 172.21.15.6:0/66210296 shutdown_connections 2024-03-04T09:24:54.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:54.544+0000 7fc9f7200640 1 -- 172.21.15.6:0/66210296 >> 172.21.15.6:0/66210296 conn(0x7fc9f80d3cd0 msgr2=0x7fc9f80d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:24:54.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:54.544+0000 7fc9f7200640 1 -- 172.21.15.6:0/66210296 shutdown_connections 2024-03-04T09:24:54.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:54.544+0000 7fc9f7200640 1 -- 172.21.15.6:0/66210296 wait complete. 2024-03-04T09:24:54.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:54.546+0000 7fc9f7200640 1 Processor -- start 2024-03-04T09:24:54.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:54.546+0000 7fc9f7200640 1 -- start start 2024-03-04T09:24:54.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:54.547+0000 7fc9f7200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc9dc004800 0x7fc9f81f7d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:24:54.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:54.547+0000 7fc9f7200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc9f815edc0 0x7fc9f806bf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:24:54.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:54.547+0000 7fc9f7200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc9f8158c90 0x7fc9f8159120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:24:54.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:54.547+0000 7fc9f7200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9f80b3c20 con 0x7fc9f8158c90 2024-03-04T09:24:54.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:54.547+0000 7fc9f7200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9f815e2c0 con 0x7fc9dc004800 2024-03-04T09:24:54.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:54.547+0000 7fc9f7200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9f80b3da0 con 0x7fc9f815edc0 2024-03-04T09:24:54.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:54.548+0000 7fc9f6800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc9f8158c90 0x7fc9f8159120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:24:54.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:54.548+0000 7fc9f6800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc9f8158c90 0x7fc9f8159120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45040/0 (socket says 172.21.15.6:45040) 2024-03-04T09:24:54.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:54.548+0000 7fc9f6800640 1 -- 172.21.15.6:0/3190420468 learned_addr learned my addr 172.21.15.6:0/3190420468 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:24:54.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:54.548+0000 7fc9f5e00640 1 --2- 172.21.15.6:0/3190420468 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc9dc004800 0x7fc9f81f7d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:24:54.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:54.548+0000 7fc9f5400640 1 --2- 172.21.15.6:0/3190420468 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc9f815edc0 0x7fc9f806bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:24:54.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:54.548+0000 7fc9f6800640 1 -- 172.21.15.6:0/3190420468 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc9f815edc0 msgr2=0x7fc9f806bf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:24:54.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:54.548+0000 7fc9f6800640 1 --2- 172.21.15.6:0/3190420468 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc9f815edc0 0x7fc9f806bf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:24:54.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:54.548+0000 7fc9f6800640 1 -- 172.21.15.6:0/3190420468 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc9dc004800 msgr2=0x7fc9f81f7d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:24:54.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:54.548+0000 7fc9f6800640 1 --2- 172.21.15.6:0/3190420468 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc9dc004800 0x7fc9f81f7d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:24:54.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:54.548+0000 7fc9f6800640 1 -- 172.21.15.6:0/3190420468 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9f81f82d0 con 0x7fc9f8158c90 2024-03-04T09:24:54.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:54.548+0000 7fc9f6800640 1 --2- 172.21.15.6:0/3190420468 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc9f8158c90 0x7fc9f8159120 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fc9f80cad50 tx=0x7fc9f000fc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:24:54.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:54.548+0000 7fc9ee000640 1 -- 172.21.15.6:0/3190420468 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9f001a070 con 0x7fc9f8158c90 2024-03-04T09:24:54.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:54.549+0000 7fc9ee000640 1 -- 172.21.15.6:0/3190420468 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9f0006d60 con 0x7fc9f8158c90 2024-03-04T09:24:54.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:54.549+0000 7fc9ee000640 1 -- 172.21.15.6:0/3190420468 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9e8007b70 con 0x7fc9f8158c90 2024-03-04T09:24:54.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:54.549+0000 7fc9f7200640 1 -- 172.21.15.6:0/3190420468 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9f815f330 con 0x7fc9f8158c90 2024-03-04T09:24:54.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:54.549+0000 7fc9f7200640 1 -- 172.21.15.6:0/3190420468 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9f80c8930 con 0x7fc9f8158c90 2024-03-04T09:24:54.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:54.549+0000 7fc9ee000640 1 -- 172.21.15.6:0/3190420468 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fc9f0003b90 con 0x7fc9f8158c90 2024-03-04T09:24:54.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:54.549+0000 7fc9ee000640 1 -- 172.21.15.6:0/3190420468 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fc9f000d760 con 0x7fc9f8158c90 2024-03-04T09:24:54.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:54.550+0000 7fc9f7200640 1 -- 172.21.15.6:0/3190420468 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9f804f330 con 0x7fc9f8158c90 2024-03-04T09:24:54.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:54.553+0000 7fc9ee000640 1 -- 172.21.15.6:0/3190420468 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fc9f0035dd0 con 0x7fc9f8158c90 2024-03-04T09:24:54.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:54.626+0000 7fc9f7200640 1 -- 172.21.15.6:0/3190420468 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fc9f808d7d0 con 0x7fc9f8158c90 2024-03-04T09:24:54.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:54.629+0000 7fc9ee000640 1 -- 172.21.15.6:0/3190420468 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7fc9f0024020 con 0x7fc9f8158c90 2024-03-04T09:24:54.626 INFO:teuthology.orchestra.run.smithi006.stderr:adjusted tunables profile to default 2024-03-04T09:24:54.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:54.631+0000 7fc9f7200640 1 -- 172.21.15.6:0/3190420468 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc9f8158c90 msgr2=0x7fc9f8159120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:24:54.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:54.631+0000 7fc9f7200640 1 --2- 172.21.15.6:0/3190420468 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc9f8158c90 0x7fc9f8159120 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fc9f80cad50 tx=0x7fc9f000fc50 comp rx=0 tx=0).stop 2024-03-04T09:24:54.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:54.631+0000 7fc9f7200640 1 -- 172.21.15.6:0/3190420468 shutdown_connections 2024-03-04T09:24:54.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:54.631+0000 7fc9f7200640 1 --2- 172.21.15.6:0/3190420468 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc9f8158c90 0x7fc9f8159120 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:24:54.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:54.631+0000 7fc9f7200640 1 --2- 172.21.15.6:0/3190420468 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc9f815edc0 0x7fc9f806bf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:24:54.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:54.631+0000 7fc9f7200640 1 --2- 172.21.15.6:0/3190420468 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc9dc004800 0x7fc9f81f7d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:24:54.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:54.631+0000 7fc9f7200640 1 -- 172.21.15.6:0/3190420468 >> 172.21.15.6:0/3190420468 conn(0x7fc9f80d3cd0 msgr2=0x7fc9f806ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:24:54.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:54.631+0000 7fc9f7200640 1 -- 172.21.15.6:0/3190420468 shutdown_connections 2024-03-04T09:24:54.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:54.631+0000 7fc9f7200640 1 -- 172.21.15.6:0/3190420468 wait complete. 2024-03-04T09:24:54.647 INFO:tasks.ceph:check_enable_crimson: False 2024-03-04T09:24:54.647 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-04T09:24:54.647 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-04T09:24:54.647 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-04T09:24:54.730 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-04T09:24:54.730 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-04T09:24:54.864 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-04T09:24:54.864 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-04T09:24:54.946 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-04T09:24:54.946 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-04T09:24:55.083 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-04T09:24:55.083 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-04T09:24:55.169 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-04T09:24:55.170 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-04T09:24:55.250 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-04T09:24:55.250 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-04T09:24:55.286 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-03-04T09:24:55.290+0000 7f43eb238040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-04T09:24:55.312 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-04T09:24:55.315+0000 7fe53068f040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-04T09:24:55.352 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-03-04T09:24:55.355+0000 7f43eb238040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-04T09:24:55.370 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-04T09:24:55.370 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-04T09:24:55.382 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-04T09:24:55.385+0000 7fe53068f040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-04T09:24:55.497 DEBUG:teuthology.orchestra.run.smithi146:> sudo ceph --cluster ceph osd new 7a96574f-81f1-4bc6-8463-757520b84a44 0 2024-03-04T09:24:55.733 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:55.736+0000 7f833fe00640 1 Processor -- start 2024-03-04T09:24:55.733 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:55.736+0000 7f833fe00640 1 -- start start 2024-03-04T09:24:55.735 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:55.738+0000 7f833fe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f8340161600 0x7f83400cc780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:24:55.735 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:55.738+0000 7f833fe00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f834015e680 0x7f83400ce440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:24:55.735 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:55.738+0000 7f833fe00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83400cd190 con 0x7f83400cccc0 2024-03-04T09:24:55.735 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:55.738+0000 7f833fe00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83400cd450 con 0x7f8340161600 2024-03-04T09:24:55.735 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:55.738+0000 7f833fe00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83400cd730 con 0x7f834015e680 2024-03-04T09:24:55.735 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:55.738+0000 7f8336000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f834015e680 0x7f83400ce440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:24:55.735 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:55.738+0000 7f8336000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f834015e680 0x7f83400ce440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.146:54978/0 (socket says 172.21.15.146:54978) 2024-03-04T09:24:55.736 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:55.738+0000 7f8336000640 1 -- 172.21.15.146:0/3053416700 learned_addr learned my addr 172.21.15.146:0/3053416700 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-03-04T09:24:55.736 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:55.738+0000 7f833f400640 1 --1- 172.21.15.146:0/3053416700 >> v1:172.21.15.146:6789/0 conn(0x7f83400cccc0 0x7f834015d730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:44734/0 (socket says 172.21.15.146:44734) 2024-03-04T09:24:55.736 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:55.738+0000 7f833ea00640 1 --2- 172.21.15.146:0/3053416700 >> v2:172.21.15.6:3300/0 conn(0x7f8340161600 0x7f83400cc780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:24:55.736 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:55.738+0000 7f8336000640 1 -- 172.21.15.146:0/3053416700 >> v2:172.21.15.6:3300/0 conn(0x7f8340161600 msgr2=0x7f83400cc780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:24:55.736 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:55.738+0000 7f8336000640 1 --2- 172.21.15.146:0/3053416700 >> v2:172.21.15.6:3300/0 conn(0x7f8340161600 0x7f83400cc780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:24:55.736 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:55.738+0000 7f8336000640 1 -- 172.21.15.146:0/3053416700 >> v1:172.21.15.146:6789/0 conn(0x7f83400cccc0 legacy=0x7f834015d730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:24:55.736 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:55.738+0000 7f8336000640 1 -- 172.21.15.146:0/3053416700 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83400cda10 con 0x7f834015e680 2024-03-04T09:24:55.736 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:55.739+0000 7f833ea00640 1 --2- 172.21.15.146:0/3053416700 >> v2:172.21.15.6:3300/0 conn(0x7f8340161600 0x7f83400cc780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:24:55.736 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:55.740+0000 7f8336000640 1 --2- 172.21.15.146:0/3053416700 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f834015e680 0x7f83400ce440 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f8330009870 tx=0x7f833002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7fb3814bc50f0f34 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:24:55.737 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:55.740+0000 7f833e000640 1 -- 172.21.15.146:0/3053416700 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8330034070 con 0x7f834015e680 2024-03-04T09:24:55.737 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:55.740+0000 7f833e000640 1 -- 172.21.15.146:0/3053416700 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8330030840 con 0x7f834015e680 2024-03-04T09:24:55.737 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:55.740+0000 7f833e000640 1 -- 172.21.15.146:0/3053416700 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8330030c10 con 0x7f834015e680 2024-03-04T09:24:55.738 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:55.741+0000 7f833fe00640 1 -- 172.21.15.146:0/3053416700 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f834015e680 msgr2=0x7f83400ce440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:24:55.738 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:55.741+0000 7f833fe00640 1 --2- 172.21.15.146:0/3053416700 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f834015e680 0x7f83400ce440 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f8330009870 tx=0x7f833002b980 comp rx=0 tx=0).stop 2024-03-04T09:24:55.738 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:55.741+0000 7f833fe00640 1 -- 172.21.15.146:0/3053416700 shutdown_connections 2024-03-04T09:24:55.738 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:55.741+0000 7f833fe00640 1 --2- 172.21.15.146:0/3053416700 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f834015e680 0x7f83400ce440 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:24:55.738 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:55.741+0000 7f833fe00640 1 --2- 172.21.15.146:0/3053416700 >> v2:172.21.15.6:3300/0 conn(0x7f8340161600 0x7f83400cc780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:24:55.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:55.741+0000 7f833fe00640 1 -- 172.21.15.146:0/3053416700 >> 172.21.15.146:0/3053416700 conn(0x7f83400b35d0 msgr2=0x7f83400b39e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:24:55.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:55.741+0000 7f833fe00640 1 -- 172.21.15.146:0/3053416700 shutdown_connections 2024-03-04T09:24:55.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:55.741+0000 7f833fe00640 1 -- 172.21.15.146:0/3053416700 wait complete. 2024-03-04T09:24:55.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:55.743+0000 7f833fe00640 1 Processor -- start 2024-03-04T09:24:55.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:55.743+0000 7f833fe00640 1 -- start start 2024-03-04T09:24:55.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:55.744+0000 7f833fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f83400cccc0 0x7f834014ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:24:55.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:55.744+0000 7f833fe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834015e680 0x7f834014ff70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:24:55.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:55.744+0000 7f833fe00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8340161600 0x7f834014d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:24:55.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:55.744+0000 7f833fe00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f834015fa20 con 0x7f83400cccc0 2024-03-04T09:24:55.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:55.744+0000 7f833fe00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f834015f8a0 con 0x7f834015e680 2024-03-04T09:24:55.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:55.744+0000 7f833fe00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83400ce980 con 0x7f8340161600 2024-03-04T09:24:55.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:55.744+0000 7f8336000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834015e680 0x7f834014ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:24:55.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:55.744+0000 7f833f400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8340161600 0x7f834014d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:24:55.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:55.744+0000 7f8336000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834015e680 0x7f834014ff70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:53244/0 (socket says 172.21.15.146:53244) 2024-03-04T09:24:55.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:55.744+0000 7f8336000640 1 -- 172.21.15.146:0/139834012 learned_addr learned my addr 172.21.15.146:0/139834012 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-03-04T09:24:55.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:55.744+0000 7f833ea00640 1 --2- 172.21.15.146:0/139834012 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f83400cccc0 0x7f834014ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:24:55.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:55.745+0000 7f833f400640 1 -- 172.21.15.146:0/139834012 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834015e680 msgr2=0x7f834014ff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:24:55.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:55.745+0000 7f833f400640 1 --2- 172.21.15.146:0/139834012 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834015e680 0x7f834014ff70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:24:55.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:55.745+0000 7f833f400640 1 -- 172.21.15.146:0/139834012 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f83400cccc0 msgr2=0x7f834014ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:24:55.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:55.745+0000 7f833f400640 1 --2- 172.21.15.146:0/139834012 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f83400cccc0 0x7f834014ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:24:55.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:55.745+0000 7f833f400640 1 -- 172.21.15.146:0/139834012 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83400c8d30 con 0x7f8340161600 2024-03-04T09:24:55.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:55.745+0000 7f8336000640 1 --2- 172.21.15.146:0/139834012 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834015e680 0x7f834014ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T09:24:55.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:55.745+0000 7f833f400640 1 --2- 172.21.15.146:0/139834012 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8340161600 0x7f834014d530 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f83380012e0 tx=0x7f833800cb90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:24:55.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:55.745+0000 7f8337400640 1 -- 172.21.15.146:0/139834012 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8338018070 con 0x7f8340161600 2024-03-04T09:24:55.744 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:55.745+0000 7f8337400640 1 -- 172.21.15.146:0/139834012 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8338004d60 con 0x7f8340161600 2024-03-04T09:24:55.744 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:55.745+0000 7f8337400640 1 -- 172.21.15.146:0/139834012 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833800a820 con 0x7f8340161600 2024-03-04T09:24:55.744 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:55.745+0000 7f833fe00640 1 -- 172.21.15.146:0/139834012 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83400c9010 con 0x7f8340161600 2024-03-04T09:24:55.744 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:55.745+0000 7f833fe00640 1 -- 172.21.15.146:0/139834012 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83400cacd0 con 0x7f8340161600 2024-03-04T09:24:55.744 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:55.746+0000 7f8337400640 1 -- 172.21.15.146:0/139834012 <== mon.2 v2:172.21.15.146:3301/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f833801b070 con 0x7f8340161600 2024-03-04T09:24:55.744 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:55.746+0000 7f8337400640 1 -- 172.21.15.146:0/139834012 <== mon.2 v2:172.21.15.146:3301/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (secure 0 0 0) 0x7f833801d440 con 0x7f8340161600 2024-03-04T09:24:55.744 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:55.746+0000 7f833fe00640 1 -- 172.21.15.146:0/139834012 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83400ca7e0 con 0x7f8340161600 2024-03-04T09:24:55.746 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:55.749+0000 7f8337400640 1 -- 172.21.15.146:0/139834012 <== mon.2 v2:172.21.15.146:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f8338007070 con 0x7f8340161600 2024-03-04T09:24:55.825 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:55.828+0000 7f833fe00640 1 -- 172.21.15.146:0/139834012 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_command({"prefix": "osd new", "uuid": "7a96574f-81f1-4bc6-8463-757520b84a44", "id": 0} v 0) v1 -- 0x7f834004f330 con 0x7f8340161600 2024-03-04T09:24:55.832 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:55.836+0000 7f8337400640 1 -- 172.21.15.146:0/139834012 <== mon.2 v2:172.21.15.146:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "7a96574f-81f1-4bc6-8463-757520b84a44", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7f8338007250 con 0x7f8340161600 2024-03-04T09:24:55.832 INFO:teuthology.orchestra.run.smithi146.stdout:0 2024-03-04T09:24:55.834 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:55.837+0000 7f833fe00640 1 -- 172.21.15.146:0/139834012 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8340161600 msgr2=0x7f834014d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:24:55.834 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:55.837+0000 7f833fe00640 1 --2- 172.21.15.146:0/139834012 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8340161600 0x7f834014d530 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f83380012e0 tx=0x7f833800cb90 comp rx=0 tx=0).stop 2024-03-04T09:24:55.834 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:55.837+0000 7f833fe00640 1 -- 172.21.15.146:0/139834012 shutdown_connections 2024-03-04T09:24:55.835 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:55.837+0000 7f833fe00640 1 --2- 172.21.15.146:0/139834012 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8340161600 0x7f834014d530 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:24:55.835 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:55.837+0000 7f833fe00640 1 --2- 172.21.15.146:0/139834012 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834015e680 0x7f834014ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:24:55.835 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:55.837+0000 7f833fe00640 1 --2- 172.21.15.146:0/139834012 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f83400cccc0 0x7f834014ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:24:55.835 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:55.837+0000 7f833fe00640 1 -- 172.21.15.146:0/139834012 >> 172.21.15.146:0/139834012 conn(0x7f83400b35d0 msgr2=0x7f8340160820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:24:55.835 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:55.838+0000 7f833fe00640 1 -- 172.21.15.146:0/139834012 shutdown_connections 2024-03-04T09:24:55.835 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:55.838+0000 7f833fe00640 1 -- 172.21.15.146:0/139834012 wait complete. 2024-03-04T09:24:55.852 DEBUG:teuthology.orchestra.run.smithi146:> sudo ceph --cluster ceph osd new 7d79b9c1-6b87-476a-bc36-7db5ee64cd90 1 2024-03-04T09:24:55.986 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-03-04T09:24:55.989+0000 7f43eb238040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-04T09:24:56.012 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-04T09:24:56.015+0000 7fe53068f040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-04T09:24:56.043 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.046+0000 7f2d60a00640 1 Processor -- start 2024-03-04T09:24:56.043 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.046+0000 7f2d60a00640 1 -- start start 2024-03-04T09:24:56.044 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.048+0000 7f2d60a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f2d5c161960 0x7f2d5c0b3490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:24:56.044 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.048+0000 7f2d60a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d5c0b3150 con 0x7f2d5c161580 2024-03-04T09:24:56.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.048+0000 7f2d60a00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d5c0b2b90 con 0x7f2d5c0d4800 2024-03-04T09:24:56.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.048+0000 7f2d60a00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d5c0d26f0 con 0x7f2d5c161960 2024-03-04T09:24:56.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.048+0000 7f2d5b200640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f2d5c0d4800 0x7f2d5c0b39d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:44736/0 (socket says 172.21.15.146:44736) 2024-03-04T09:24:56.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.048+0000 7f2d5b200640 1 -- 172.21.15.146:0/2558756093 learned_addr learned my addr 172.21.15.146:0/2558756093 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-03-04T09:24:56.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.048+0000 7f2d59e00640 1 --2- 172.21.15.146:0/2558756093 >> v2:172.21.15.6:3300/0 conn(0x7f2d5c161960 0x7f2d5c0b3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:24:56.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.048+0000 7f2d59400640 1 -- 172.21.15.146:0/2558756093 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1738350430 0 0) 0x7f2d5c0b2b90 con 0x7f2d5c0d4800 2024-03-04T09:24:56.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.049+0000 7f2d59400640 1 -- 172.21.15.146:0/2558756093 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d40003520 con 0x7f2d5c0d4800 2024-03-04T09:24:56.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.049+0000 7f2d59400640 1 -- 172.21.15.146:0/2558756093 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1908135872 0 0) 0x7f2d40003520 con 0x7f2d5c0d4800 2024-03-04T09:24:56.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.049+0000 7f2d59400640 1 -- 172.21.15.146:0/2558756093 >> v2:172.21.15.6:3300/0 conn(0x7f2d5c161960 msgr2=0x7f2d5c0b3490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:24:56.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.049+0000 7f2d59400640 1 --2- 172.21.15.146:0/2558756093 >> v2:172.21.15.6:3300/0 conn(0x7f2d5c161960 0x7f2d5c0b3490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:24:56.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.049+0000 7f2d59400640 1 -- 172.21.15.146:0/2558756093 >> v1:172.21.15.6:6789/0 conn(0x7f2d5c161580 legacy=0x7f2d5c0b2240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:24:56.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.049+0000 7f2d59400640 1 -- 172.21.15.146:0/2558756093 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d5c0b2e70 con 0x7f2d5c0d4800 2024-03-04T09:24:56.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.049+0000 7f2d59400640 1 -- 172.21.15.146:0/2558756093 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7f2d540019f0 con 0x7f2d5c0d4800 2024-03-04T09:24:56.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.049+0000 7f2d59400640 1 -- 172.21.15.146:0/2558756093 >> v1:172.21.15.146:6789/0 conn(0x7f2d5c0d4800 legacy=0x7f2d5c0b39d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:24:56.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.049+0000 7f2d59400640 1 --2- 172.21.15.146:0/2558756093 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2d400039b0 0x7f2d40003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:24:56.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.049+0000 7f2d59400640 1 --2- 172.21.15.146:0/2558756093 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2d40004800 0x7f2d40004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:24:56.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.049+0000 7f2d59400640 1 --2- 172.21.15.146:0/2558756093 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2d40005210 0x7f2d4000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:24:56.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.049+0000 7f2d59400640 1 -- 172.21.15.146:0/2558756093 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d5c0d26f0 con 0x7f2d40005210 2024-03-04T09:24:56.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.049+0000 7f2d59400640 1 -- 172.21.15.146:0/2558756093 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d4000c140 con 0x7f2d40004800 2024-03-04T09:24:56.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.049+0000 7f2d59400640 1 -- 172.21.15.146:0/2558756093 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d4000c340 con 0x7f2d400039b0 2024-03-04T09:24:56.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.049+0000 7f2d59e00640 1 --2- 172.21.15.146:0/2558756093 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2d40004800 0x7f2d40004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:24:56.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.049+0000 7f2d5a800640 1 --2- 172.21.15.146:0/2558756093 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2d400039b0 0x7f2d40003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:24:56.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.050+0000 7f2d59e00640 1 -- 172.21.15.146:0/2558756093 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2d400039b0 msgr2=0x7f2d40003df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:24:56.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.050+0000 7f2d59e00640 1 --2- 172.21.15.146:0/2558756093 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2d400039b0 0x7f2d40003df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:24:56.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.050+0000 7f2d5b200640 1 --2- 172.21.15.146:0/2558756093 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2d40005210 0x7f2d4000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:24:56.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.050+0000 7f2d59e00640 1 -- 172.21.15.146:0/2558756093 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2d40005210 msgr2=0x7f2d4000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:24:56.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.050+0000 7f2d59e00640 1 --2- 172.21.15.146:0/2558756093 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2d40005210 0x7f2d4000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:24:56.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.050+0000 7f2d59e00640 1 -- 172.21.15.146:0/2558756093 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d40003520 con 0x7f2d40004800 2024-03-04T09:24:56.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.050+0000 7f2d5a800640 1 --2- 172.21.15.146:0/2558756093 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2d400039b0 0x7f2d40003df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T09:24:56.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.050+0000 7f2d5b200640 1 --2- 172.21.15.146:0/2558756093 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2d40005210 0x7f2d4000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:24:56.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.050+0000 7f2d5b200640 1 -- 172.21.15.146:0/2558756093 reap_dead start 2024-03-04T09:24:56.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.051+0000 7f2d59e00640 1 --2- 172.21.15.146:0/2558756093 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2d40004800 0x7f2d40004c80 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f2d48009a90 tx=0x7f2d40005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=461dd8498a4f7044 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:24:56.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.051+0000 7f2d59400640 1 -- 172.21.15.146:0/2558756093 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d48039070 con 0x7f2d40004800 2024-03-04T09:24:56.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.051+0000 7f2d59400640 1 -- 172.21.15.146:0/2558756093 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d4802c680 con 0x7f2d40004800 2024-03-04T09:24:56.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.051+0000 7f2d59400640 1 -- 172.21.15.146:0/2558756093 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d48009690 con 0x7f2d40004800 2024-03-04T09:24:56.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.052+0000 7f2d60a00640 1 -- 172.21.15.146:0/2558756093 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2d40004800 msgr2=0x7f2d40004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:24:56.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.052+0000 7f2d60a00640 1 --2- 172.21.15.146:0/2558756093 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2d40004800 0x7f2d40004c80 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f2d48009a90 tx=0x7f2d40005830 comp rx=0 tx=0).stop 2024-03-04T09:24:56.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.052+0000 7f2d60a00640 1 -- 172.21.15.146:0/2558756093 shutdown_connections 2024-03-04T09:24:56.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.052+0000 7f2d60a00640 1 --2- 172.21.15.146:0/2558756093 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2d40004800 0x7f2d40004c80 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:24:56.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.052+0000 7f2d60a00640 1 -- 172.21.15.146:0/2558756093 >> 172.21.15.146:0/2558756093 conn(0x7f2d5c0d3c70 msgr2=0x7f2d5c0d4080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:24:56.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.052+0000 7f2d60a00640 1 -- 172.21.15.146:0/2558756093 shutdown_connections 2024-03-04T09:24:56.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.052+0000 7f2d60a00640 1 -- 172.21.15.146:0/2558756093 wait complete. 2024-03-04T09:24:56.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.054+0000 7f2d60a00640 1 Processor -- start 2024-03-04T09:24:56.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.054+0000 7f2d60a00640 1 -- start start 2024-03-04T09:24:56.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.055+0000 7f2d60a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2d40004800 0x7f2d5c0b7280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:24:56.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.055+0000 7f2d60a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2d5c0cd240 0x7f2d5c0cbe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:24:56.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.055+0000 7f2d60a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2d5c0cc3c0 0x7f2d5c0cc840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:24:56.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.055+0000 7f2d60a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d5c0b23c0 con 0x7f2d5c0cd240 2024-03-04T09:24:56.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.055+0000 7f2d60a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d5c0b2240 con 0x7f2d40004800 2024-03-04T09:24:56.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.055+0000 7f2d60a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d5c0b2540 con 0x7f2d5c0cc3c0 2024-03-04T09:24:56.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.055+0000 7f2d5a800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2d40004800 0x7f2d5c0b7280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:24:56.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.055+0000 7f2d5b200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2d5c0cc3c0 0x7f2d5c0cc840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:24:56.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.055+0000 7f2d5a800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2d40004800 0x7f2d5c0b7280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:53266/0 (socket says 172.21.15.146:53266) 2024-03-04T09:24:56.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.055+0000 7f2d5a800640 1 -- 172.21.15.146:0/3250680755 learned_addr learned my addr 172.21.15.146:0/3250680755 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-03-04T09:24:56.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.055+0000 7f2d59e00640 1 --2- 172.21.15.146:0/3250680755 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2d5c0cd240 0x7f2d5c0cbe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:24:56.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.055+0000 7f2d5a800640 1 -- 172.21.15.146:0/3250680755 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2d5c0cc3c0 msgr2=0x7f2d5c0cc840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:24:56.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.055+0000 7f2d5a800640 1 --2- 172.21.15.146:0/3250680755 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2d5c0cc3c0 0x7f2d5c0cc840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:24:56.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.055+0000 7f2d5a800640 1 -- 172.21.15.146:0/3250680755 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2d5c0cd240 msgr2=0x7f2d5c0cbe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:24:56.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.055+0000 7f2d5a800640 1 --2- 172.21.15.146:0/3250680755 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2d5c0cd240 0x7f2d5c0cbe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:24:56.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.055+0000 7f2d5a800640 1 -- 172.21.15.146:0/3250680755 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d5c0c77a0 con 0x7f2d40004800 2024-03-04T09:24:56.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.056+0000 7f2d5a800640 1 --2- 172.21.15.146:0/3250680755 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2d40004800 0x7f2d5c0b7280 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f2d4c00e480 tx=0x7f2d4c00e940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:24:56.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.056+0000 7f2d52a00640 1 -- 172.21.15.146:0/3250680755 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d4c0052e0 con 0x7f2d40004800 2024-03-04T09:24:56.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.056+0000 7f2d52a00640 1 -- 172.21.15.146:0/3250680755 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d4c007020 con 0x7f2d40004800 2024-03-04T09:24:56.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.056+0000 7f2d52a00640 1 -- 172.21.15.146:0/3250680755 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d4c007210 con 0x7f2d40004800 2024-03-04T09:24:56.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.056+0000 7f2d60a00640 1 -- 172.21.15.146:0/3250680755 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d5c0ccd80 con 0x7f2d40004800 2024-03-04T09:24:56.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.056+0000 7f2d60a00640 1 -- 172.21.15.146:0/3250680755 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d5c0b7e20 con 0x7f2d40004800 2024-03-04T09:24:56.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.056+0000 7f2d52a00640 1 -- 172.21.15.146:0/3250680755 <== mon.1 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f2d4c01a020 con 0x7f2d40004800 2024-03-04T09:24:56.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.056+0000 7f2d52a00640 1 -- 172.21.15.146:0/3250680755 <== mon.1 v2:172.21.15.146:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (secure 0 0 0) 0x7f2d4c0038f0 con 0x7f2d40004800 2024-03-04T09:24:56.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.058+0000 7f2d60a00640 1 -- 172.21.15.146:0/3250680755 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d5c04f330 con 0x7f2d40004800 2024-03-04T09:24:56.056 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-03-04T09:24:56.058+0000 7f43eb238040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-04T09:24:56.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.060+0000 7f2d52a00640 1 -- 172.21.15.146:0/3250680755 <== mon.1 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f2d4c003af0 con 0x7f2d40004800 2024-03-04T09:24:56.079 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-04T09:24:56.082+0000 7fe53068f040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-04T09:24:56.124 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-03-04T09:24:56.128+0000 7f43eb238040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-04T09:24:56.132 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-04T09:24:56.136+0000 7fe53068f040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-04T09:24:56.150 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.153+0000 7f2d60a00640 1 -- 172.21.15.146:0/3250680755 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd new", "uuid": "7d79b9c1-6b87-476a-bc36-7db5ee64cd90", "id": 1} v 0) v1 -- 0x7f2d5c06c9b0 con 0x7f2d40004800 2024-03-04T09:24:56.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.160+0000 7f2d52a00640 1 -- 172.21.15.146:0/3250680755 <== mon.1 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "7d79b9c1-6b87-476a-bc36-7db5ee64cd90", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7f2d4c003cd0 con 0x7f2d40004800 2024-03-04T09:24:56.157 INFO:teuthology.orchestra.run.smithi146.stdout:1 2024-03-04T09:24:56.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.162+0000 7f2d60a00640 1 -- 172.21.15.146:0/3250680755 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2d40004800 msgr2=0x7f2d5c0b7280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:24:56.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.162+0000 7f2d60a00640 1 --2- 172.21.15.146:0/3250680755 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2d40004800 0x7f2d5c0b7280 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f2d4c00e480 tx=0x7f2d4c00e940 comp rx=0 tx=0).stop 2024-03-04T09:24:56.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.162+0000 7f2d60a00640 1 -- 172.21.15.146:0/3250680755 shutdown_connections 2024-03-04T09:24:56.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.162+0000 7f2d60a00640 1 --2- 172.21.15.146:0/3250680755 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2d5c0cc3c0 0x7f2d5c0cc840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:24:56.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.162+0000 7f2d60a00640 1 --2- 172.21.15.146:0/3250680755 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2d5c0cd240 0x7f2d5c0cbe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:24:56.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.162+0000 7f2d60a00640 1 --2- 172.21.15.146:0/3250680755 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2d40004800 0x7f2d5c0b7280 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:24:56.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.162+0000 7f2d60a00640 1 -- 172.21.15.146:0/3250680755 >> 172.21.15.146:0/3250680755 conn(0x7f2d5c0d3c70 msgr2=0x7f2d5c0b3470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:24:56.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.162+0000 7f2d60a00640 1 -- 172.21.15.146:0/3250680755 shutdown_connections 2024-03-04T09:24:56.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.162+0000 7f2d60a00640 1 -- 172.21.15.146:0/3250680755 wait complete. 2024-03-04T09:24:56.174 DEBUG:teuthology.orchestra.run.smithi146:> sudo ceph --cluster ceph osd new 492331b9-4ee9-40b6-bacd-85d44cd6c45a 2 2024-03-04T09:24:56.196 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-03-04T09:24:56.199+0000 7f43eb238040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-04T09:24:56.199 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-04T09:24:56.202+0000 7fe53068f040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-04T09:24:56.251 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-04T09:24:56.254+0000 7fe53068f040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-04T09:24:56.251 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-03-04T09:24:56.255+0000 7f43eb238040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-04T09:24:56.365 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.368+0000 7f7519200640 1 Processor -- start 2024-03-04T09:24:56.365 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.369+0000 7f7519200640 1 -- start start 2024-03-04T09:24:56.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.370+0000 7f7519200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f75140c8f10 0x7f75140cf1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:24:56.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.370+0000 7f7519200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75140c7a00 con 0x7f7514161840 2024-03-04T09:24:56.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.370+0000 7f7519200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7514161c20 con 0x7f75140d0f00 2024-03-04T09:24:56.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.370+0000 7f7519200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75140d2530 con 0x7f75140c8f10 2024-03-04T09:24:56.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.370+0000 7f7512800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f75140c8f10 0x7f75140cf1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:24:56.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.370+0000 7f7512800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f75140c8f10 0x7f75140cf1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:53272/0 (socket says 172.21.15.146:53272) 2024-03-04T09:24:56.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.370+0000 7f7512800640 1 -- 172.21.15.146:0/251538393 learned_addr learned my addr 172.21.15.146:0/251538393 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-03-04T09:24:56.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.371+0000 7f7512800640 1 -- 172.21.15.146:0/251538393 >> v1:172.21.15.146:6789/0 conn(0x7f75140d0f00 legacy=0x7f75140ceac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:24:56.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.371+0000 7f7512800640 1 -- 172.21.15.146:0/251538393 >> v1:172.21.15.6:6789/0 conn(0x7f7514161840 legacy=0x7f75140d07f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:24:56.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.371+0000 7f7512800640 1 -- 172.21.15.146:0/251538393 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75140d2810 con 0x7f75140c8f10 2024-03-04T09:24:56.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.372+0000 7f7512800640 1 --2- 172.21.15.146:0/251538393 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f75140c8f10 0x7f75140cf1d0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f750c009870 tx=0x7f750c02b9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2c41967c5af7e62f server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:24:56.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.372+0000 7f7511400640 1 -- 172.21.15.146:0/251538393 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f750c034070 con 0x7f75140c8f10 2024-03-04T09:24:56.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.372+0000 7f7511400640 1 -- 172.21.15.146:0/251538393 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f750c030860 con 0x7f75140c8f10 2024-03-04T09:24:56.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.372+0000 7f7511400640 1 -- 172.21.15.146:0/251538393 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f750c030c30 con 0x7f75140c8f10 2024-03-04T09:24:56.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.373+0000 7f7519200640 1 -- 172.21.15.146:0/251538393 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f75140c8f10 msgr2=0x7f75140cf1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:24:56.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.373+0000 7f7519200640 1 --2- 172.21.15.146:0/251538393 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f75140c8f10 0x7f75140cf1d0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f750c009870 tx=0x7f750c02b9a0 comp rx=0 tx=0).stop 2024-03-04T09:24:56.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.373+0000 7f7519200640 1 -- 172.21.15.146:0/251538393 shutdown_connections 2024-03-04T09:24:56.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.373+0000 7f7519200640 1 --2- 172.21.15.146:0/251538393 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f75140c8f10 0x7f75140cf1d0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:24:56.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.373+0000 7f7519200640 1 -- 172.21.15.146:0/251538393 >> 172.21.15.146:0/251538393 conn(0x7f75140d3eb0 msgr2=0x7f75140d42c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:24:56.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.373+0000 7f7519200640 1 -- 172.21.15.146:0/251538393 shutdown_connections 2024-03-04T09:24:56.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.373+0000 7f7519200640 1 -- 172.21.15.146:0/251538393 wait complete. 2024-03-04T09:24:56.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.375+0000 7f7519200640 1 Processor -- start 2024-03-04T09:24:56.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.375+0000 7f7519200640 1 -- start start 2024-03-04T09:24:56.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.376+0000 7f7519200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f75140c8f10 0x7f75140b0ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:24:56.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.376+0000 7f7519200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f75140d0f00 0x7f75140b3ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:24:56.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.376+0000 7f7519200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7514161840 0x7f7514155210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:24:56.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.376+0000 7f7519200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75140d1750 con 0x7f75140d0f00 2024-03-04T09:24:56.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.376+0000 7f7519200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75140d2dd0 con 0x7f75140c8f10 2024-03-04T09:24:56.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.376+0000 7f7519200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75140d4a40 con 0x7f7514161840 2024-03-04T09:24:56.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.376+0000 7f7512800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7514161840 0x7f7514155210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:24:56.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.376+0000 7f7511e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f75140c8f10 0x7f75140b0ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:24:56.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.376+0000 7f7512800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7514161840 0x7f7514155210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.146:55008/0 (socket says 172.21.15.146:55008) 2024-03-04T09:24:56.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.376+0000 7f7511e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f75140c8f10 0x7f75140b0ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:53284/0 (socket says 172.21.15.146:53284) 2024-03-04T09:24:56.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.376+0000 7f7512800640 1 -- 172.21.15.146:0/1951442842 learned_addr learned my addr 172.21.15.146:0/1951442842 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-03-04T09:24:56.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.376+0000 7f7509400640 1 --2- 172.21.15.146:0/1951442842 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f75140d0f00 0x7f75140b3ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:24:56.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.376+0000 7f7511e00640 1 -- 172.21.15.146:0/1951442842 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7514161840 msgr2=0x7f7514155210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:24:56.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.376+0000 7f7511e00640 1 --2- 172.21.15.146:0/1951442842 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7514161840 0x7f7514155210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:24:56.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.376+0000 7f7511e00640 1 -- 172.21.15.146:0/1951442842 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f75140d0f00 msgr2=0x7f75140b3ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:24:56.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.376+0000 7f7511e00640 1 --2- 172.21.15.146:0/1951442842 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f75140d0f00 0x7f75140b3ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:24:56.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.376+0000 7f7511e00640 1 -- 172.21.15.146:0/1951442842 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75140b4170 con 0x7f75140c8f10 2024-03-04T09:24:56.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.376+0000 7f7512800640 1 --2- 172.21.15.146:0/1951442842 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7514161840 0x7f7514155210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T09:24:56.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.377+0000 7f7511e00640 1 --2- 172.21.15.146:0/1951442842 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f75140c8f10 0x7f75140b0ac0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f750400c700 tx=0x7f750400cbc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:24:56.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.377+0000 7f750aa00640 1 -- 172.21.15.146:0/1951442842 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7504003b70 con 0x7f75140c8f10 2024-03-04T09:24:56.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.377+0000 7f750aa00640 1 -- 172.21.15.146:0/1951442842 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7504003d10 con 0x7f75140c8f10 2024-03-04T09:24:56.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.377+0000 7f750aa00640 1 -- 172.21.15.146:0/1951442842 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7504005310 con 0x7f75140c8f10 2024-03-04T09:24:56.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.377+0000 7f7519200640 1 -- 172.21.15.146:0/1951442842 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75140b4450 con 0x7f75140c8f10 2024-03-04T09:24:56.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.377+0000 7f7519200640 1 -- 172.21.15.146:0/1951442842 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75140cc8f0 con 0x7f75140c8f10 2024-03-04T09:24:56.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.377+0000 7f750aa00640 1 -- 172.21.15.146:0/1951442842 <== mon.1 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f75040054b0 con 0x7f75140c8f10 2024-03-04T09:24:56.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.377+0000 7f750aa00640 1 -- 172.21.15.146:0/1951442842 <== mon.1 v2:172.21.15.146:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (secure 0 0 0) 0x7f75040057a0 con 0x7f75140c8f10 2024-03-04T09:24:56.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.378+0000 7f7519200640 1 -- 172.21.15.146:0/1951442842 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7514068010 con 0x7f75140c8f10 2024-03-04T09:24:56.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.381+0000 7f750aa00640 1 -- 172.21.15.146:0/1951442842 <== mon.1 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f750400a640 con 0x7f75140c8f10 2024-03-04T09:24:56.386 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-04T09:24:56.390+0000 7fe53068f040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-04T09:24:56.396 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-03-04T09:24:56.399+0000 7f43eb238040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-04T09:24:56.456 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.459+0000 7f7519200640 1 -- 172.21.15.146:0/1951442842 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd new", "uuid": "492331b9-4ee9-40b6-bacd-85d44cd6c45a", "id": 2} v 0) v1 -- 0x7f75140ca750 con 0x7f75140c8f10 2024-03-04T09:24:56.464 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.467+0000 7f750aa00640 1 -- 172.21.15.146:0/1951442842 <== mon.1 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "492331b9-4ee9-40b6-bacd-85d44cd6c45a", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7f750400a900 con 0x7f75140c8f10 2024-03-04T09:24:56.465 INFO:teuthology.orchestra.run.smithi146.stdout:2 2024-03-04T09:24:56.466 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.469+0000 7f7519200640 1 -- 172.21.15.146:0/1951442842 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f75140c8f10 msgr2=0x7f75140b0ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:24:56.466 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.469+0000 7f7519200640 1 --2- 172.21.15.146:0/1951442842 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f75140c8f10 0x7f75140b0ac0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f750400c700 tx=0x7f750400cbc0 comp rx=0 tx=0).stop 2024-03-04T09:24:56.466 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.469+0000 7f7519200640 1 -- 172.21.15.146:0/1951442842 shutdown_connections 2024-03-04T09:24:56.466 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.469+0000 7f7519200640 1 --2- 172.21.15.146:0/1951442842 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7514161840 0x7f7514155210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:24:56.466 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.469+0000 7f7519200640 1 --2- 172.21.15.146:0/1951442842 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f75140d0f00 0x7f75140b3ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:24:56.466 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.469+0000 7f7519200640 1 --2- 172.21.15.146:0/1951442842 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f75140c8f10 0x7f75140b0ac0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:24:56.466 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.469+0000 7f7519200640 1 -- 172.21.15.146:0/1951442842 >> 172.21.15.146:0/1951442842 conn(0x7f75140d3eb0 msgr2=0x7f751406c110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:24:56.466 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.469+0000 7f7519200640 1 -- 172.21.15.146:0/1951442842 shutdown_connections 2024-03-04T09:24:56.466 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.469+0000 7f7519200640 1 -- 172.21.15.146:0/1951442842 wait complete. 2024-03-04T09:24:56.471 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-04T09:24:56.474+0000 7fe53068f040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-04T09:24:56.482 DEBUG:teuthology.orchestra.run.smithi146:> sudo ceph --cluster ceph osd new 21bb2efe-9ba7-4bdc-bc31-180cd1102e21 3 2024-03-04T09:24:56.486 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-03-04T09:24:56.489+0000 7f43eb238040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-04T09:24:56.527 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-04T09:24:56.531+0000 7fe53068f040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-04T09:24:56.544 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-03-04T09:24:56.548+0000 7f43eb238040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-04T09:24:56.592 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-04T09:24:56.595+0000 7fe53068f040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-04T09:24:56.609 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-03-04T09:24:56.612+0000 7f43eb238040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-04T09:24:56.649 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-04T09:24:56.653+0000 7fe53068f040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-04T09:24:56.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.671+0000 7f6fb5800640 1 Processor -- start 2024-03-04T09:24:56.668 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-03-04T09:24:56.671+0000 7f43eb238040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-04T09:24:56.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.671+0000 7f6fb5800640 1 -- start start 2024-03-04T09:24:56.674 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.672+0000 7f6fb5800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f6fb00d0b40 0x7f6fb00b4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:24:56.674 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.672+0000 7f6fb5800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6fb00d4680 0x7f6fb0161370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:24:56.674 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.672+0000 7f6fb5800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f6fb00d0f20 0x7f6fb01618b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:24:56.674 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.672+0000 7f6fb5800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fb00cf590 con 0x7f6fb00d0b40 2024-03-04T09:24:56.674 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.672+0000 7f6fb5800640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fb00b6dc0 con 0x7f6fb00d0f20 2024-03-04T09:24:56.674 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.672+0000 7f6fb5800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fb00cf7e0 con 0x7f6fb00d4680 2024-03-04T09:24:56.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.673+0000 7f6fa7e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6fb00d4680 0x7f6fb0161370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:24:56.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.673+0000 7f6faf400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f6fb00d0f20 0x7f6fb01618b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:24:56.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.673+0000 7f6faf400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f6fb00d0f20 0x7f6fb01618b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:53292/0 (socket says 172.21.15.146:53292) 2024-03-04T09:24:56.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.673+0000 7f6fa7e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6fb00d4680 0x7f6fb0161370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.146:55010/0 (socket says 172.21.15.146:55010) 2024-03-04T09:24:56.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.673+0000 7f6faf400640 1 -- 172.21.15.146:0/190608292 learned_addr learned my addr 172.21.15.146:0/190608292 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-03-04T09:24:56.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.673+0000 7f6faea00640 1 --2- 172.21.15.146:0/190608292 >> v2:172.21.15.6:3300/0 conn(0x7f6fb00d0b40 0x7f6fb00b4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:24:56.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.673+0000 7f6faf400640 1 -- 172.21.15.146:0/190608292 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6fb00d4680 msgr2=0x7f6fb0161370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:24:56.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.673+0000 7f6faf400640 1 --2- 172.21.15.146:0/190608292 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6fb00d4680 0x7f6fb0161370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:24:56.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.673+0000 7f6faf400640 1 -- 172.21.15.146:0/190608292 >> v2:172.21.15.6:3300/0 conn(0x7f6fb00d0b40 msgr2=0x7f6fb00b4ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:24:56.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.673+0000 7f6faf400640 1 --2- 172.21.15.146:0/190608292 >> v2:172.21.15.6:3300/0 conn(0x7f6fb00d0b40 0x7f6fb00b4ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:24:56.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.673+0000 7f6faf400640 1 -- 172.21.15.146:0/190608292 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fb00cfac0 con 0x7f6fb00d0f20 2024-03-04T09:24:56.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.675+0000 7f6faf400640 1 --2- 172.21.15.146:0/190608292 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6fb00d0f20 0x7f6fb01618b0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f6fb008e130 tx=0x7f6fa802be40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=42b1d7e6c8e4a723 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:24:56.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.675+0000 7f6fae000640 1 -- 172.21.15.146:0/190608292 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fa802c9e0 con 0x7f6fb00d0f20 2024-03-04T09:24:56.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.675+0000 7f6fae000640 1 -- 172.21.15.146:0/190608292 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fa802cb80 con 0x7f6fb00d0f20 2024-03-04T09:24:56.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.675+0000 7f6fae000640 1 -- 172.21.15.146:0/190608292 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fa80343f0 con 0x7f6fb00d0f20 2024-03-04T09:24:56.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.676+0000 7f6fb5800640 1 -- 172.21.15.146:0/190608292 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6fb00d0f20 msgr2=0x7f6fb01618b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:24:56.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.676+0000 7f6fb5800640 1 --2- 172.21.15.146:0/190608292 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6fb00d0f20 0x7f6fb01618b0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f6fb008e130 tx=0x7f6fa802be40 comp rx=0 tx=0).stop 2024-03-04T09:24:56.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.676+0000 7f6fb5800640 1 -- 172.21.15.146:0/190608292 shutdown_connections 2024-03-04T09:24:56.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.676+0000 7f6fb5800640 1 --2- 172.21.15.146:0/190608292 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6fb00d0f20 0x7f6fb01618b0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:24:56.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.676+0000 7f6fb5800640 1 --2- 172.21.15.146:0/190608292 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6fb00d4680 0x7f6fb0161370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:24:56.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.676+0000 7f6fb5800640 1 --2- 172.21.15.146:0/190608292 >> v2:172.21.15.6:3300/0 conn(0x7f6fb00d0b40 0x7f6fb00b4ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:24:56.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.676+0000 7f6fb5800640 1 -- 172.21.15.146:0/190608292 >> 172.21.15.146:0/190608292 conn(0x7f6fb009d280 msgr2=0x7f6fb00d3f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:24:56.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.676+0000 7f6fb5800640 1 -- 172.21.15.146:0/190608292 shutdown_connections 2024-03-04T09:24:56.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.676+0000 7f6fb5800640 1 -- 172.21.15.146:0/190608292 wait complete. 2024-03-04T09:24:56.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.678+0000 7f6fb5800640 1 Processor -- start 2024-03-04T09:24:56.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.678+0000 7f6fb5800640 1 -- start start 2024-03-04T09:24:56.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.679+0000 7f6fb5800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6fb00d0b40 0x7f6fb014c490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:24:56.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.679+0000 7f6fb5800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6fb00d0f20 0x7f6fb015bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:24:56.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.679+0000 7f6fb5800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6fb00d4680 0x7f6fb015c120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:24:56.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.679+0000 7f6fb5800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fb00d1b40 con 0x7f6fb00d4680 2024-03-04T09:24:56.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.679+0000 7f6fb5800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fb00b53e0 con 0x7f6fb00d0f20 2024-03-04T09:24:56.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.679+0000 7f6fb5800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fb00d1cc0 con 0x7f6fb00d0b40 2024-03-04T09:24:56.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.679+0000 7f6fa7e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6fb00d0f20 0x7f6fb015bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:24:56.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.679+0000 7f6faea00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6fb00d0b40 0x7f6fb014c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:24:56.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.679+0000 7f6fa7e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6fb00d0f20 0x7f6fb015bbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:53300/0 (socket says 172.21.15.146:53300) 2024-03-04T09:24:56.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.679+0000 7f6fa7e00640 1 -- 172.21.15.146:0/812810389 learned_addr learned my addr 172.21.15.146:0/812810389 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-03-04T09:24:56.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.679+0000 7f6faf400640 1 --2- 172.21.15.146:0/812810389 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6fb00d4680 0x7f6fb015c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:24:56.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.679+0000 7f6faea00640 1 -- 172.21.15.146:0/812810389 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6fb00d0f20 msgr2=0x7f6fb015bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:24:56.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.679+0000 7f6faea00640 1 --2- 172.21.15.146:0/812810389 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6fb00d0f20 0x7f6fb015bbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:24:56.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.679+0000 7f6faea00640 1 -- 172.21.15.146:0/812810389 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6fb00d4680 msgr2=0x7f6fb015c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:24:56.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.679+0000 7f6faea00640 1 --2- 172.21.15.146:0/812810389 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6fb00d4680 0x7f6fb015c120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:24:56.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.679+0000 7f6faea00640 1 -- 172.21.15.146:0/812810389 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fb0157180 con 0x7f6fb00d0b40 2024-03-04T09:24:56.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.679+0000 7f6fa7e00640 1 --2- 172.21.15.146:0/812810389 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6fb00d0f20 0x7f6fb015bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T09:24:56.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.679+0000 7f6faf400640 1 --2- 172.21.15.146:0/812810389 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6fb00d4680 0x7f6fb015c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:24:56.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.680+0000 7f6faea00640 1 --2- 172.21.15.146:0/812810389 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6fb00d0b40 0x7f6fb014c490 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f6fa000d620 tx=0x7f6fa000dae0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:24:56.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.680+0000 7f6fa6a00640 1 -- 172.21.15.146:0/812810389 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fa000b620 con 0x7f6fb00d0b40 2024-03-04T09:24:56.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.680+0000 7f6fa6a00640 1 -- 172.21.15.146:0/812810389 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fa0019070 con 0x7f6fb00d0b40 2024-03-04T09:24:56.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.680+0000 7f6fa6a00640 1 -- 172.21.15.146:0/812810389 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fa000b9e0 con 0x7f6fb00d0b40 2024-03-04T09:24:56.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.680+0000 7f6fb5800640 1 -- 172.21.15.146:0/812810389 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fb00cc740 con 0x7f6fb00d0b40 2024-03-04T09:24:56.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.680+0000 7f6fb5800640 1 -- 172.21.15.146:0/812810389 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fb01576e0 con 0x7f6fb00d0b40 2024-03-04T09:24:56.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.680+0000 7f6fa6a00640 1 -- 172.21.15.146:0/812810389 <== mon.2 v2:172.21.15.146:3301/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f6fa000bb80 con 0x7f6fb00d0b40 2024-03-04T09:24:56.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.680+0000 7f6fa6a00640 1 -- 172.21.15.146:0/812810389 <== mon.2 v2:172.21.15.146:3301/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (secure 0 0 0) 0x7f6fa001e8f0 con 0x7f6fb00d0b40 2024-03-04T09:24:56.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.681+0000 7f6fb5800640 1 -- 172.21.15.146:0/812810389 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fb0068720 con 0x7f6fb00d0b40 2024-03-04T09:24:56.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.684+0000 7f6fa6a00640 1 -- 172.21.15.146:0/812810389 <== mon.2 v2:172.21.15.146:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f6fb0157180 con 0x7f6fb00d0b40 2024-03-04T09:24:56.758 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.762+0000 7f6fb5800640 1 -- 172.21.15.146:0/812810389 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_command({"prefix": "osd new", "uuid": "21bb2efe-9ba7-4bdc-bc31-180cd1102e21", "id": 3} v 0) v1 -- 0x7f6fb00d3d50 con 0x7f6fb00d0b40 2024-03-04T09:24:56.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.769+0000 7f6fa6a00640 1 -- 172.21.15.146:0/812810389 <== mon.2 v2:172.21.15.146:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "21bb2efe-9ba7-4bdc-bc31-180cd1102e21", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7f6fa001e460 con 0x7f6fb00d0b40 2024-03-04T09:24:56.766 INFO:teuthology.orchestra.run.smithi146.stdout:3 2024-03-04T09:24:56.767 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.770+0000 7f6fb5800640 1 -- 172.21.15.146:0/812810389 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6fb00d0b40 msgr2=0x7f6fb014c490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:24:56.767 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.770+0000 7f6fb5800640 1 --2- 172.21.15.146:0/812810389 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6fb00d0b40 0x7f6fb014c490 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f6fa000d620 tx=0x7f6fa000dae0 comp rx=0 tx=0).stop 2024-03-04T09:24:56.767 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.770+0000 7f6fb5800640 1 -- 172.21.15.146:0/812810389 shutdown_connections 2024-03-04T09:24:56.767 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.770+0000 7f6fb5800640 1 --2- 172.21.15.146:0/812810389 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6fb00d4680 0x7f6fb015c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:24:56.767 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.770+0000 7f6fb5800640 1 --2- 172.21.15.146:0/812810389 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6fb00d0f20 0x7f6fb015bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:24:56.767 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.770+0000 7f6fb5800640 1 --2- 172.21.15.146:0/812810389 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6fb00d0b40 0x7f6fb014c490 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:24:56.767 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.770+0000 7f6fb5800640 1 -- 172.21.15.146:0/812810389 >> 172.21.15.146:0/812810389 conn(0x7f6fb009d280 msgr2=0x7f6fb0160430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:24:56.767 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.770+0000 7f6fb5800640 1 -- 172.21.15.146:0/812810389 shutdown_connections 2024-03-04T09:24:56.768 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.770+0000 7f6fb5800640 1 -- 172.21.15.146:0/812810389 wait complete. 2024-03-04T09:24:56.783 DEBUG:teuthology.orchestra.run.smithi146:> sudo ceph --cluster ceph osd new 6e2c4b71-19d9-4737-83d9-558705d8644a 4 2024-03-04T09:24:56.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.968+0000 7fa446800640 1 Processor -- start 2024-03-04T09:24:56.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.968+0000 7fa446800640 1 -- start start 2024-03-04T09:24:56.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.969+0000 7fa446800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fa4400d2180 0x7fa4400d1980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:24:56.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.969+0000 7fa446800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa44015e1e0 0x7fa4400b6a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:24:56.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.970+0000 7fa446800640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4400b1480 con 0x7fa4400b3040 2024-03-04T09:24:56.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.970+0000 7fa446800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4400b1230 con 0x7fa44015e1e0 2024-03-04T09:24:56.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.970+0000 7fa446800640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4400b1db0 con 0x7fa4400d2180 2024-03-04T09:24:56.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.970+0000 7fa445400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fa4400d2180 0x7fa4400d1980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:24:56.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.970+0000 7fa445400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fa4400d2180 0x7fa4400d1980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:53306/0 (socket says 172.21.15.146:53306) 2024-03-04T09:24:56.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.970+0000 7fa445400640 1 -- 172.21.15.146:0/3266284730 learned_addr learned my addr 172.21.15.146:0/3266284730 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-03-04T09:24:56.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.970+0000 7fa444a00640 1 --2- 172.21.15.146:0/3266284730 >> v2:172.21.15.6:3300/0 conn(0x7fa44015e1e0 0x7fa4400b6a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:24:56.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.970+0000 7fa445400640 1 -- 172.21.15.146:0/3266284730 >> v2:172.21.15.6:3300/0 conn(0x7fa44015e1e0 msgr2=0x7fa4400b6a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:24:56.968 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.970+0000 7fa445400640 1 --2- 172.21.15.146:0/3266284730 >> v2:172.21.15.6:3300/0 conn(0x7fa44015e1e0 0x7fa4400b6a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:24:56.968 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.970+0000 7fa445400640 1 -- 172.21.15.146:0/3266284730 >> v1:172.21.15.146:6789/0 conn(0x7fa4400b3040 legacy=0x7fa4400d2560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:24:56.968 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.970+0000 7fa445400640 1 -- 172.21.15.146:0/3266284730 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4400b1760 con 0x7fa4400d2180 2024-03-04T09:24:56.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.971+0000 7fa445400640 1 --2- 172.21.15.146:0/3266284730 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa4400d2180 0x7fa4400d1980 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fa44008cd90 tx=0x7fa43402b9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8daeae8b57d233b3 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:24:56.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.972+0000 7fa43fe00640 1 -- 172.21.15.146:0/3266284730 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa434034070 con 0x7fa4400d2180 2024-03-04T09:24:56.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.972+0000 7fa43fe00640 1 -- 172.21.15.146:0/3266284730 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa434030860 con 0x7fa4400d2180 2024-03-04T09:24:56.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.972+0000 7fa43fe00640 1 -- 172.21.15.146:0/3266284730 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa434030c30 con 0x7fa4400d2180 2024-03-04T09:24:56.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.972+0000 7fa446800640 1 -- 172.21.15.146:0/3266284730 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa4400d2180 msgr2=0x7fa4400d1980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:24:56.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.972+0000 7fa446800640 1 --2- 172.21.15.146:0/3266284730 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa4400d2180 0x7fa4400d1980 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fa44008cd90 tx=0x7fa43402b9a0 comp rx=0 tx=0).stop 2024-03-04T09:24:56.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.972+0000 7fa446800640 1 -- 172.21.15.146:0/3266284730 shutdown_connections 2024-03-04T09:24:56.970 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.972+0000 7fa446800640 1 --2- 172.21.15.146:0/3266284730 >> v2:172.21.15.6:3300/0 conn(0x7fa44015e1e0 0x7fa4400b6a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:24:56.970 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.972+0000 7fa446800640 1 --2- 172.21.15.146:0/3266284730 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa4400d2180 0x7fa4400d1980 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:24:56.970 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.972+0000 7fa446800640 1 -- 172.21.15.146:0/3266284730 >> 172.21.15.146:0/3266284730 conn(0x7fa4400d3c30 msgr2=0x7fa4400d4040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:24:56.970 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.972+0000 7fa446800640 1 -- 172.21.15.146:0/3266284730 shutdown_connections 2024-03-04T09:24:56.970 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.972+0000 7fa446800640 1 -- 172.21.15.146:0/3266284730 wait complete. 2024-03-04T09:24:56.970 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.974+0000 7fa446800640 1 Processor -- start 2024-03-04T09:24:56.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.974+0000 7fa446800640 1 -- start start 2024-03-04T09:24:56.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.975+0000 7fa446800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa4400b3040 0x7fa4400b5800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:24:56.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.975+0000 7fa446800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4400d2180 0x7fa4400b3420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:24:56.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.975+0000 7fa446800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa44015e1e0 0x7fa440160a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:24:56.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.975+0000 7fa446800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4400cfae0 con 0x7fa4400d2180 2024-03-04T09:24:56.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.975+0000 7fa446800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4400b6fc0 con 0x7fa4400b3040 2024-03-04T09:24:56.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.975+0000 7fa446800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4400b2240 con 0x7fa44015e1e0 2024-03-04T09:24:56.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.975+0000 7fa445400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa4400b3040 0x7fa4400b5800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:24:56.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.975+0000 7fa445400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa4400b3040 0x7fa4400b5800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:53310/0 (socket says 172.21.15.146:53310) 2024-03-04T09:24:56.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.975+0000 7fa445400640 1 -- 172.21.15.146:0/555629571 learned_addr learned my addr 172.21.15.146:0/555629571 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-03-04T09:24:56.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.975+0000 7fa445e00640 1 --2- 172.21.15.146:0/555629571 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa44015e1e0 0x7fa440160a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:24:56.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.976+0000 7fa444a00640 1 --2- 172.21.15.146:0/555629571 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4400d2180 0x7fa4400b3420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:24:56.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.976+0000 7fa445400640 1 -- 172.21.15.146:0/555629571 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa44015e1e0 msgr2=0x7fa440160a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:24:56.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.976+0000 7fa445400640 1 --2- 172.21.15.146:0/555629571 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa44015e1e0 0x7fa440160a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:24:56.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.976+0000 7fa445400640 1 -- 172.21.15.146:0/555629571 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4400d2180 msgr2=0x7fa4400b3420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:24:56.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.976+0000 7fa445400640 1 --2- 172.21.15.146:0/555629571 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4400d2180 0x7fa4400b3420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:24:56.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.976+0000 7fa445400640 1 -- 172.21.15.146:0/555629571 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4400b5d40 con 0x7fa4400b3040 2024-03-04T09:24:56.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.976+0000 7fa444a00640 1 --2- 172.21.15.146:0/555629571 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4400d2180 0x7fa4400b3420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:24:56.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.976+0000 7fa445400640 1 --2- 172.21.15.146:0/555629571 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa4400b3040 0x7fa4400b5800 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fa434000c00 tx=0x7fa434004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:24:56.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.976+0000 7fa43d600640 1 -- 172.21.15.146:0/555629571 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa434048070 con 0x7fa4400b3040 2024-03-04T09:24:56.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.976+0000 7fa43d600640 1 -- 172.21.15.146:0/555629571 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa434030660 con 0x7fa4400b3040 2024-03-04T09:24:56.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.976+0000 7fa446800640 1 -- 172.21.15.146:0/555629571 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4400b3e00 con 0x7fa4400b3040 2024-03-04T09:24:56.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.976+0000 7fa43d600640 1 -- 172.21.15.146:0/555629571 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa434034040 con 0x7fa4400b3040 2024-03-04T09:24:56.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.976+0000 7fa446800640 1 -- 172.21.15.146:0/555629571 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4400b43c0 con 0x7fa4400b3040 2024-03-04T09:24:56.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.977+0000 7fa43d600640 1 -- 172.21.15.146:0/555629571 <== mon.1 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fa43404b070 con 0x7fa4400b3040 2024-03-04T09:24:56.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.977+0000 7fa43d600640 1 -- 172.21.15.146:0/555629571 <== mon.1 v2:172.21.15.146:3300/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (secure 0 0 0) 0x7fa434043900 con 0x7fa4400b3040 2024-03-04T09:24:56.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.977+0000 7fa446800640 1 -- 172.21.15.146:0/555629571 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa44004f330 con 0x7fa4400b3040 2024-03-04T09:24:56.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:56.980+0000 7fa43d600640 1 -- 172.21.15.146:0/555629571 <== mon.1 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fa434030830 con 0x7fa4400b3040 2024-03-04T09:24:57.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.059+0000 7fa446800640 1 -- 172.21.15.146:0/555629571 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd new", "uuid": "6e2c4b71-19d9-4737-83d9-558705d8644a", "id": 4} v 0) v1 -- 0x7fa440074800 con 0x7fa4400b3040 2024-03-04T09:24:57.062 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.065+0000 7fa43d600640 1 -- 172.21.15.146:0/555629571 <== mon.1 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "6e2c4b71-19d9-4737-83d9-558705d8644a", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7fa434043b50 con 0x7fa4400b3040 2024-03-04T09:24:57.062 INFO:teuthology.orchestra.run.smithi146.stdout:4 2024-03-04T09:24:57.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.067+0000 7fa446800640 1 -- 172.21.15.146:0/555629571 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa4400b3040 msgr2=0x7fa4400b5800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:24:57.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.067+0000 7fa446800640 1 --2- 172.21.15.146:0/555629571 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa4400b3040 0x7fa4400b5800 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fa434000c00 tx=0x7fa434004730 comp rx=0 tx=0).stop 2024-03-04T09:24:57.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.067+0000 7fa446800640 1 -- 172.21.15.146:0/555629571 shutdown_connections 2024-03-04T09:24:57.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.067+0000 7fa446800640 1 --2- 172.21.15.146:0/555629571 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa44015e1e0 0x7fa440160a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:24:57.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.067+0000 7fa446800640 1 --2- 172.21.15.146:0/555629571 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4400d2180 0x7fa4400b3420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:24:57.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.067+0000 7fa446800640 1 --2- 172.21.15.146:0/555629571 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa4400b3040 0x7fa4400b5800 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:24:57.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.067+0000 7fa446800640 1 -- 172.21.15.146:0/555629571 >> 172.21.15.146:0/555629571 conn(0x7fa4400d3c30 msgr2=0x7fa44006cb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:24:57.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.067+0000 7fa446800640 1 -- 172.21.15.146:0/555629571 shutdown_connections 2024-03-04T09:24:57.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.067+0000 7fa446800640 1 -- 172.21.15.146:0/555629571 wait complete. 2024-03-04T09:24:57.079 DEBUG:teuthology.orchestra.run.smithi146:> sudo ceph --cluster ceph osd new fb69c525-f1be-4b7d-ab32-6c29b0d6fddb 5 2024-03-04T09:24:57.120 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-04T09:24:57.123+0000 7fe53068f040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-04T09:24:57.148 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-03-04T09:24:57.152+0000 7f43eb238040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-04T09:24:57.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.269+0000 7f902be00640 1 Processor -- start 2024-03-04T09:24:57.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.270+0000 7f902be00640 1 -- start start 2024-03-04T09:24:57.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.271+0000 7f902be00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f902c0d2180 0x7f902c0d1980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:24:57.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.271+0000 7f902be00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f902c15e1e0 0x7f902c0b6a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:24:57.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.271+0000 7f902be00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f902c0b1480 con 0x7f902c0b3040 2024-03-04T09:24:57.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.271+0000 7f902be00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f902c0b1230 con 0x7f902c0d2180 2024-03-04T09:24:57.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.271+0000 7f902be00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f902c0b1db0 con 0x7f902c15e1e0 2024-03-04T09:24:57.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.271+0000 7f9023e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f902c15e1e0 0x7f902c0b6a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:24:57.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.271+0000 7f9023e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f902c15e1e0 0x7f902c0b6a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:53320/0 (socket says 172.21.15.146:53320) 2024-03-04T09:24:57.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.271+0000 7f9023e00640 1 -- 172.21.15.146:0/1747065990 learned_addr learned my addr 172.21.15.146:0/1747065990 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-03-04T09:24:57.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.272+0000 7f902aa00640 1 --2- 172.21.15.146:0/1747065990 >> v2:172.21.15.6:3300/0 conn(0x7f902c0d2180 0x7f902c0d1980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:24:57.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.272+0000 7f9023e00640 1 -- 172.21.15.146:0/1747065990 >> v2:172.21.15.6:3300/0 conn(0x7f902c0d2180 msgr2=0x7f902c0d1980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:24:57.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.272+0000 7f9023e00640 1 --2- 172.21.15.146:0/1747065990 >> v2:172.21.15.6:3300/0 conn(0x7f902c0d2180 0x7f902c0d1980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:24:57.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.272+0000 7f9023e00640 1 -- 172.21.15.146:0/1747065990 >> v1:172.21.15.6:6789/0 conn(0x7f902c0b3040 legacy=0x7f902c0d2560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:24:57.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.272+0000 7f9023e00640 1 -- 172.21.15.146:0/1747065990 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f902c0b1760 con 0x7f902c15e1e0 2024-03-04T09:24:57.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.273+0000 7f9023e00640 1 --2- 172.21.15.146:0/1747065990 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f902c15e1e0 0x7f902c0b6a80 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f902c08dc50 tx=0x7f901402b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=256523061028db64 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:24:57.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.273+0000 7f902a000640 1 -- 172.21.15.146:0/1747065990 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9014035020 con 0x7f902c15e1e0 2024-03-04T09:24:57.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.273+0000 7f902a000640 1 -- 172.21.15.146:0/1747065990 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9014031870 con 0x7f902c15e1e0 2024-03-04T09:24:57.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.273+0000 7f902a000640 1 -- 172.21.15.146:0/1747065990 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9014031c40 con 0x7f902c15e1e0 2024-03-04T09:24:57.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.274+0000 7f902be00640 1 -- 172.21.15.146:0/1747065990 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f902c15e1e0 msgr2=0x7f902c0b6a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:24:57.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.274+0000 7f902be00640 1 --2- 172.21.15.146:0/1747065990 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f902c15e1e0 0x7f902c0b6a80 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f902c08dc50 tx=0x7f901402b910 comp rx=0 tx=0).stop 2024-03-04T09:24:57.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.274+0000 7f902be00640 1 -- 172.21.15.146:0/1747065990 shutdown_connections 2024-03-04T09:24:57.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.274+0000 7f902be00640 1 --2- 172.21.15.146:0/1747065990 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f902c15e1e0 0x7f902c0b6a80 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:24:57.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.274+0000 7f902be00640 1 --2- 172.21.15.146:0/1747065990 >> v2:172.21.15.6:3300/0 conn(0x7f902c0d2180 0x7f902c0d1980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:24:57.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.274+0000 7f902be00640 1 -- 172.21.15.146:0/1747065990 >> 172.21.15.146:0/1747065990 conn(0x7f902c0d3c30 msgr2=0x7f902c0d4040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:24:57.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.274+0000 7f902be00640 1 -- 172.21.15.146:0/1747065990 shutdown_connections 2024-03-04T09:24:57.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.274+0000 7f902be00640 1 -- 172.21.15.146:0/1747065990 wait complete. 2024-03-04T09:24:57.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.276+0000 7f902be00640 1 Processor -- start 2024-03-04T09:24:57.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.276+0000 7f902be00640 1 -- start start 2024-03-04T09:24:57.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.277+0000 7f902be00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f902c0b3040 0x7f902c1f1020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:24:57.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.277+0000 7f902be00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f902c0d2180 0x7f902c1f4420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:24:57.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.277+0000 7f902be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f902c15e1e0 0x7f902c0cce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:24:57.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.277+0000 7f902be00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f902c0cfae0 con 0x7f902c15e1e0 2024-03-04T09:24:57.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.277+0000 7f902be00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f902c0b6fc0 con 0x7f902c0b3040 2024-03-04T09:24:57.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.277+0000 7f902be00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f902c0b2240 con 0x7f902c0d2180 2024-03-04T09:24:57.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.277+0000 7f902aa00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f902c0b3040 0x7f902c1f1020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:24:57.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.277+0000 7f9023e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f902c0d2180 0x7f902c1f4420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:24:57.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.277+0000 7f902aa00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f902c0b3040 0x7f902c1f1020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:53328/0 (socket says 172.21.15.146:53328) 2024-03-04T09:24:57.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.277+0000 7f902aa00640 1 -- 172.21.15.146:0/448001278 learned_addr learned my addr 172.21.15.146:0/448001278 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-03-04T09:24:57.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.277+0000 7f902b400640 1 --2- 172.21.15.146:0/448001278 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f902c15e1e0 0x7f902c0cce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:24:57.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.277+0000 7f902aa00640 1 -- 172.21.15.146:0/448001278 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f902c0d2180 msgr2=0x7f902c1f4420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:24:57.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.277+0000 7f902aa00640 1 --2- 172.21.15.146:0/448001278 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f902c0d2180 0x7f902c1f4420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:24:57.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.277+0000 7f902aa00640 1 -- 172.21.15.146:0/448001278 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f902c15e1e0 msgr2=0x7f902c0cce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:24:57.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.277+0000 7f902aa00640 1 --2- 172.21.15.146:0/448001278 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f902c15e1e0 0x7f902c0cce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:24:57.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.277+0000 7f902aa00640 1 -- 172.21.15.146:0/448001278 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f902c0c9fd0 con 0x7f902c0b3040 2024-03-04T09:24:57.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.277+0000 7f902b400640 1 --2- 172.21.15.146:0/448001278 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f902c15e1e0 0x7f902c0cce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:24:57.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.278+0000 7f902aa00640 1 --2- 172.21.15.146:0/448001278 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f902c0b3040 0x7f902c1f1020 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f901c00c7f0 tx=0x7f901c00ccb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:24:57.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.278+0000 7f9022a00640 1 -- 172.21.15.146:0/448001278 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f901c015070 con 0x7f902c0b3040 2024-03-04T09:24:57.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.278+0000 7f9022a00640 1 -- 172.21.15.146:0/448001278 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f901c004240 con 0x7f902c0b3040 2024-03-04T09:24:57.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.278+0000 7f9022a00640 1 -- 172.21.15.146:0/448001278 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f901c007620 con 0x7f902c0b3040 2024-03-04T09:24:57.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.278+0000 7f902be00640 1 -- 172.21.15.146:0/448001278 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f902c0cd670 con 0x7f902c0b3040 2024-03-04T09:24:57.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.278+0000 7f902be00640 1 -- 172.21.15.146:0/448001278 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f902c1f3c20 con 0x7f902c0b3040 2024-03-04T09:24:57.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.278+0000 7f9022a00640 1 -- 172.21.15.146:0/448001278 <== mon.1 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f901c0079d0 con 0x7f902c0b3040 2024-03-04T09:24:57.277 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.278+0000 7f9022a00640 1 -- 172.21.15.146:0/448001278 <== mon.1 v2:172.21.15.146:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (secure 0 0 0) 0x7f901c003b90 con 0x7f902c0b3040 2024-03-04T09:24:57.277 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.279+0000 7f902be00640 1 -- 172.21.15.146:0/448001278 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f902c04f330 con 0x7f902c0b3040 2024-03-04T09:24:57.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.281+0000 7f9022a00640 1 -- 172.21.15.146:0/448001278 <== mon.1 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f901c007cc0 con 0x7f902c0b3040 2024-03-04T09:24:57.289 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-04T09:24:57.292+0000 7fe53068f040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-04T09:24:57.316 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-03-04T09:24:57.320+0000 7f43eb238040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-04T09:24:57.356 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.359+0000 7f902be00640 1 -- 172.21.15.146:0/448001278 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd new", "uuid": "fb69c525-f1be-4b7d-ab32-6c29b0d6fddb", "id": 5} v 0) v1 -- 0x7f902c073f70 con 0x7f902c0b3040 2024-03-04T09:24:57.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.366+0000 7f9022a00640 1 -- 172.21.15.146:0/448001278 <== mon.1 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "fb69c525-f1be-4b7d-ab32-6c29b0d6fddb", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7f901c003610 con 0x7f902c0b3040 2024-03-04T09:24:57.363 INFO:teuthology.orchestra.run.smithi146.stdout:5 2024-03-04T09:24:57.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.367+0000 7f902be00640 1 -- 172.21.15.146:0/448001278 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f902c0b3040 msgr2=0x7f902c1f1020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:24:57.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.367+0000 7f902be00640 1 --2- 172.21.15.146:0/448001278 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f902c0b3040 0x7f902c1f1020 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f901c00c7f0 tx=0x7f901c00ccb0 comp rx=0 tx=0).stop 2024-03-04T09:24:57.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.367+0000 7f902be00640 1 -- 172.21.15.146:0/448001278 shutdown_connections 2024-03-04T09:24:57.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.367+0000 7f902be00640 1 --2- 172.21.15.146:0/448001278 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f902c15e1e0 0x7f902c0cce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:24:57.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.367+0000 7f902be00640 1 --2- 172.21.15.146:0/448001278 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f902c0d2180 0x7f902c1f4420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:24:57.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.367+0000 7f902be00640 1 --2- 172.21.15.146:0/448001278 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f902c0b3040 0x7f902c1f1020 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:24:57.365 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.367+0000 7f902be00640 1 -- 172.21.15.146:0/448001278 >> 172.21.15.146:0/448001278 conn(0x7f902c0d3c30 msgr2=0x7f902c0b2630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:24:57.365 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.368+0000 7f902be00640 1 -- 172.21.15.146:0/448001278 shutdown_connections 2024-03-04T09:24:57.365 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.368+0000 7f902be00640 1 -- 172.21.15.146:0/448001278 wait complete. 2024-03-04T09:24:57.381 DEBUG:teuthology.orchestra.run.smithi146:> sudo ceph --cluster ceph osd new c32e7fed-370d-4401-8ce0-6a4335b0d168 6 2024-03-04T09:24:57.420 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-04T09:24:57.424+0000 7fe53068f040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-04T09:24:57.453 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-03-04T09:24:57.457+0000 7f43eb238040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-04T09:24:57.476 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-04T09:24:57.479+0000 7fe53068f040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-04T09:24:57.510 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-03-04T09:24:57.513+0000 7f43eb238040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-04T09:24:57.566 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.569+0000 7f591ec00640 1 Processor -- start 2024-03-04T09:24:57.566 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.569+0000 7f591ec00640 1 -- start start 2024-03-04T09:24:57.568 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.571+0000 7f591ec00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f59180d0f00 0x7f59180ceac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:24:57.568 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.571+0000 7f591ec00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f59180c8f10 0x7f59180cf000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:24:57.568 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.571+0000 7f591ec00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59180c7a00 con 0x7f5918161840 2024-03-04T09:24:57.568 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.571+0000 7f591ec00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5918161c20 con 0x7f59180c8f10 2024-03-04T09:24:57.568 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.571+0000 7f591ec00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59180d2530 con 0x7f59180d0f00 2024-03-04T09:24:57.568 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.571+0000 7f590f400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f59180d0f00 0x7f59180ceac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:24:57.568 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.571+0000 7f5917e00640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f5918161840 0x7f59180d07f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:44760/0 (socket says 172.21.15.146:44760) 2024-03-04T09:24:57.568 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.571+0000 7f5917e00640 1 -- 172.21.15.146:0/1373301648 learned_addr learned my addr 172.21.15.146:0/1373301648 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-03-04T09:24:57.568 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.571+0000 7f591cc00640 1 --2- 172.21.15.146:0/1373301648 >> v2:172.21.15.6:3300/0 conn(0x7f59180c8f10 0x7f59180cf000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:24:57.569 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.571+0000 7f5917400640 1 -- 172.21.15.146:0/1373301648 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1415511717 0 0) 0x7f59180c7a00 con 0x7f5918161840 2024-03-04T09:24:57.569 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.571+0000 7f590f400640 1 -- 172.21.15.146:0/1373301648 >> v2:172.21.15.6:3300/0 conn(0x7f59180c8f10 msgr2=0x7f59180cf000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:24:57.569 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.571+0000 7f590f400640 1 --2- 172.21.15.146:0/1373301648 >> v2:172.21.15.6:3300/0 conn(0x7f59180c8f10 0x7f59180cf000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:24:57.569 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.571+0000 7f590f400640 1 -- 172.21.15.146:0/1373301648 >> v1:172.21.15.146:6789/0 conn(0x7f5918161840 legacy=0x7f59180d07f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:24:57.569 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.571+0000 7f590f400640 1 -- 172.21.15.146:0/1373301648 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59180d2810 con 0x7f59180d0f00 2024-03-04T09:24:57.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.573+0000 7f590f400640 1 --2- 172.21.15.146:0/1373301648 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f59180d0f00 0x7f59180ceac0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f59000097d0 tx=0x7f590002b810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fe471648e0f7a310 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:24:57.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.573+0000 7f5917400640 1 -- 172.21.15.146:0/1373301648 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5900031770 con 0x7f59180d0f00 2024-03-04T09:24:57.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.573+0000 7f5917400640 1 -- 172.21.15.146:0/1373301648 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5900037070 con 0x7f59180d0f00 2024-03-04T09:24:57.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.573+0000 7f5917400640 1 -- 172.21.15.146:0/1373301648 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5900031b30 con 0x7f59180d0f00 2024-03-04T09:24:57.571 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.574+0000 7f591ec00640 1 -- 172.21.15.146:0/1373301648 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f59180d0f00 msgr2=0x7f59180ceac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:24:57.571 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.574+0000 7f591ec00640 1 --2- 172.21.15.146:0/1373301648 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f59180d0f00 0x7f59180ceac0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f59000097d0 tx=0x7f590002b810 comp rx=0 tx=0).stop 2024-03-04T09:24:57.571 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.574+0000 7f591ec00640 1 -- 172.21.15.146:0/1373301648 shutdown_connections 2024-03-04T09:24:57.571 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.574+0000 7f591ec00640 1 --2- 172.21.15.146:0/1373301648 >> v2:172.21.15.6:3300/0 conn(0x7f59180c8f10 0x7f59180cf000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:24:57.571 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.574+0000 7f591ec00640 1 --2- 172.21.15.146:0/1373301648 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f59180d0f00 0x7f59180ceac0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:24:57.571 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.574+0000 7f591ec00640 1 -- 172.21.15.146:0/1373301648 >> 172.21.15.146:0/1373301648 conn(0x7f59180d3eb0 msgr2=0x7f59180d42c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:24:57.571 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.574+0000 7f591ec00640 1 -- 172.21.15.146:0/1373301648 shutdown_connections 2024-03-04T09:24:57.571 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.574+0000 7f591ec00640 1 -- 172.21.15.146:0/1373301648 wait complete. 2024-03-04T09:24:57.572 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.576+0000 7f591ec00640 1 Processor -- start 2024-03-04T09:24:57.572 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.576+0000 7f591ec00640 1 -- start start 2024-03-04T09:24:57.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.577+0000 7f591ec00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f59180c8f10 0x7f59180cd9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:24:57.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.577+0000 7f591ec00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f59180d0f00 0x7f5918150380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:24:57.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.577+0000 7f591ec00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5918161840 0x7f59180a1390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:24:57.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.577+0000 7f591ec00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59180d1750 con 0x7f5918161840 2024-03-04T09:24:57.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.577+0000 7f591ec00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59180d2dd0 con 0x7f59180c8f10 2024-03-04T09:24:57.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.577+0000 7f591ec00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59180d4a40 con 0x7f59180d0f00 2024-03-04T09:24:57.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.577+0000 7f5917e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f59180c8f10 0x7f59180cd9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:24:57.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.577+0000 7f590f400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f59180d0f00 0x7f5918150380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:24:57.576 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.577+0000 7f590f400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f59180d0f00 0x7f5918150380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.146:55032/0 (socket says 172.21.15.146:55032) 2024-03-04T09:24:57.576 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.577+0000 7f590f400640 1 -- 172.21.15.146:0/1459854927 learned_addr learned my addr 172.21.15.146:0/1459854927 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-03-04T09:24:57.576 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.577+0000 7f5917e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f59180c8f10 0x7f59180cd9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:53338/0 (socket says 172.21.15.146:53338) 2024-03-04T09:24:57.576 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.577+0000 7f591cc00640 1 --2- 172.21.15.146:0/1459854927 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5918161840 0x7f59180a1390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:24:57.576 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.577+0000 7f590f400640 1 -- 172.21.15.146:0/1459854927 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f59180c8f10 msgr2=0x7f59180cd9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:24:57.576 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.577+0000 7f590f400640 1 --2- 172.21.15.146:0/1459854927 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f59180c8f10 0x7f59180cd9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:24:57.576 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.577+0000 7f590f400640 1 -- 172.21.15.146:0/1459854927 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5918161840 msgr2=0x7f59180a1390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:24:57.576 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.577+0000 7f590f400640 1 --2- 172.21.15.146:0/1459854927 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5918161840 0x7f59180a1390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:24:57.576 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.577+0000 7f590f400640 1 -- 172.21.15.146:0/1459854927 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59181519e0 con 0x7f59180d0f00 2024-03-04T09:24:57.576 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.577+0000 7f5917e00640 1 --2- 172.21.15.146:0/1459854927 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f59180c8f10 0x7f59180cd9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T09:24:57.576 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.577+0000 7f591cc00640 1 --2- 172.21.15.146:0/1459854927 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5918161840 0x7f59180a1390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:24:57.576 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.578+0000 7f590f400640 1 --2- 172.21.15.146:0/1459854927 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f59180d0f00 0x7f5918150380 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f59000098f0 tx=0x7f59000046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:24:57.576 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.578+0000 7f5914c00640 1 -- 172.21.15.146:0/1459854927 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5900004230 con 0x7f59180d0f00 2024-03-04T09:24:57.576 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.578+0000 7f5914c00640 1 -- 172.21.15.146:0/1459854927 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59000043d0 con 0x7f59180d0f00 2024-03-04T09:24:57.577 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.578+0000 7f5914c00640 1 -- 172.21.15.146:0/1459854927 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5900003810 con 0x7f59180d0f00 2024-03-04T09:24:57.577 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.578+0000 7f591ec00640 1 -- 172.21.15.146:0/1459854927 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5918150c20 con 0x7f59180d0f00 2024-03-04T09:24:57.577 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.578+0000 7f591ec00640 1 -- 172.21.15.146:0/1459854927 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59181511e0 con 0x7f59180d0f00 2024-03-04T09:24:57.577 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.578+0000 7f5914c00640 1 -- 172.21.15.146:0/1459854927 <== mon.2 v2:172.21.15.146:3301/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f5900031070 con 0x7f59180d0f00 2024-03-04T09:24:57.577 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.579+0000 7f5914c00640 1 -- 172.21.15.146:0/1459854927 <== mon.2 v2:172.21.15.146:3301/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (secure 0 0 0) 0x7f590004fa30 con 0x7f59180d0f00 2024-03-04T09:24:57.577 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.579+0000 7f591ec00640 1 -- 172.21.15.146:0/1459854927 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5918068010 con 0x7f59180d0f00 2024-03-04T09:24:57.578 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.582+0000 7f5914c00640 1 -- 172.21.15.146:0/1459854927 <== mon.2 v2:172.21.15.146:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f5900041070 con 0x7f59180d0f00 2024-03-04T09:24:57.609 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-04T09:24:57.612+0000 7fe53068f040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-04T09:24:57.647 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-03-04T09:24:57.650+0000 7f43eb238040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-04T09:24:57.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.661+0000 7f591ec00640 1 -- 172.21.15.146:0/1459854927 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_command({"prefix": "osd new", "uuid": "c32e7fed-370d-4401-8ce0-6a4335b0d168", "id": 6} v 0) v1 -- 0x7f59180d03c0 con 0x7f59180d0f00 2024-03-04T09:24:57.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.668+0000 7f5914c00640 1 -- 172.21.15.146:0/1459854927 <== mon.2 v2:172.21.15.146:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "c32e7fed-370d-4401-8ce0-6a4335b0d168", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7f590004f440 con 0x7f59180d0f00 2024-03-04T09:24:57.665 INFO:teuthology.orchestra.run.smithi146.stdout:6 2024-03-04T09:24:57.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.670+0000 7f591ec00640 1 -- 172.21.15.146:0/1459854927 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f59180d0f00 msgr2=0x7f5918150380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:24:57.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.670+0000 7f591ec00640 1 --2- 172.21.15.146:0/1459854927 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f59180d0f00 0x7f5918150380 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f59000098f0 tx=0x7f59000046c0 comp rx=0 tx=0).stop 2024-03-04T09:24:57.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.670+0000 7f591ec00640 1 -- 172.21.15.146:0/1459854927 shutdown_connections 2024-03-04T09:24:57.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.670+0000 7f591ec00640 1 --2- 172.21.15.146:0/1459854927 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5918161840 0x7f59180a1390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:24:57.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.670+0000 7f591ec00640 1 --2- 172.21.15.146:0/1459854927 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f59180d0f00 0x7f5918150380 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:24:57.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.670+0000 7f591ec00640 1 --2- 172.21.15.146:0/1459854927 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f59180c8f10 0x7f59180cd9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:24:57.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.670+0000 7f591ec00640 1 -- 172.21.15.146:0/1459854927 >> 172.21.15.146:0/1459854927 conn(0x7f59180d3eb0 msgr2=0x7f591806bb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:24:57.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.670+0000 7f591ec00640 1 -- 172.21.15.146:0/1459854927 shutdown_connections 2024-03-04T09:24:57.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.670+0000 7f591ec00640 1 -- 172.21.15.146:0/1459854927 wait complete. 2024-03-04T09:24:57.683 DEBUG:teuthology.orchestra.run.smithi146:> sudo ceph --cluster ceph osd new 75a79b44-fa5f-426d-a7d3-71f21f78835c 7 2024-03-04T09:24:57.726 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-04T09:24:57.729+0000 7fe53068f040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-04T09:24:57.767 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-03-04T09:24:57.770+0000 7f43eb238040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-04T09:24:57.871 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.875+0000 7ffbdc800640 1 Processor -- start 2024-03-04T09:24:57.872 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.875+0000 7ffbdc800640 1 -- start start 2024-03-04T09:24:57.873 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.876+0000 7ffbdc800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ffbd41616b0 0x7ffbd40d07f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:24:57.874 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.876+0000 7ffbdc800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ffbd40d0d30 0x7ffbd40c7d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:24:57.874 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.876+0000 7ffbdc800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ffbd40c82b0 0x7ffbd40c8700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:24:57.874 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.876+0000 7ffbdc800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffbd40d11a0 con 0x7ffbd41616b0 2024-03-04T09:24:57.874 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.876+0000 7ffbdc800640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffbd40d1800 con 0x7ffbd40d0d30 2024-03-04T09:24:57.874 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.876+0000 7ffbdc800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbd40d2630 con 0x7ffbd40c82b0 2024-03-04T09:24:57.874 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.876+0000 7ffbd9400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ffbd40d0d30 0x7ffbd40c7d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:24:57.874 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.877+0000 7ffbda800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ffbd40c82b0 0x7ffbd40c8700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:24:57.874 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.877+0000 7ffbd9400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ffbd40d0d30 0x7ffbd40c7d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:53342/0 (socket says 172.21.15.146:53342) 2024-03-04T09:24:57.874 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.877+0000 7ffbd9400640 1 -- 172.21.15.146:0/1135491596 learned_addr learned my addr 172.21.15.146:0/1135491596 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-03-04T09:24:57.874 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.877+0000 7ffbd9e00640 1 --2- 172.21.15.146:0/1135491596 >> v2:172.21.15.6:3300/0 conn(0x7ffbd41616b0 0x7ffbd40d07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:24:57.874 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.877+0000 7ffbd9400640 1 -- 172.21.15.146:0/1135491596 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ffbd40c82b0 msgr2=0x7ffbd40c8700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:24:57.874 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.877+0000 7ffbd9400640 1 --2- 172.21.15.146:0/1135491596 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ffbd40c82b0 0x7ffbd40c8700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:24:57.874 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.877+0000 7ffbd9400640 1 -- 172.21.15.146:0/1135491596 >> v2:172.21.15.6:3300/0 conn(0x7ffbd41616b0 msgr2=0x7ffbd40d07f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:24:57.874 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.877+0000 7ffbd9400640 1 --2- 172.21.15.146:0/1135491596 >> v2:172.21.15.6:3300/0 conn(0x7ffbd41616b0 0x7ffbd40d07f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:24:57.874 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.877+0000 7ffbd9400640 1 -- 172.21.15.146:0/1135491596 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbd40d2910 con 0x7ffbd40d0d30 2024-03-04T09:24:57.875 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.877+0000 7ffbda800640 1 --2- 172.21.15.146:0/1135491596 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ffbd40c82b0 0x7ffbd40c8700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T09:24:57.876 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.878+0000 7ffbd9400640 1 --2- 172.21.15.146:0/1135491596 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffbd40d0d30 0x7ffbd40c7d70 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7ffbd408d390 tx=0x7ffbc402b870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c9d0ecd03f97cd9 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:24:57.876 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.878+0000 7ffbd8a00640 1 -- 172.21.15.146:0/1135491596 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbc40317d0 con 0x7ffbd40d0d30 2024-03-04T09:24:57.876 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.878+0000 7ffbd8a00640 1 -- 172.21.15.146:0/1135491596 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbc4037070 con 0x7ffbd40d0d30 2024-03-04T09:24:57.876 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.878+0000 7ffbd8a00640 1 -- 172.21.15.146:0/1135491596 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbc4031b90 con 0x7ffbd40d0d30 2024-03-04T09:24:57.876 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.879+0000 7ffbdc800640 1 -- 172.21.15.146:0/1135491596 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffbd40d0d30 msgr2=0x7ffbd40c7d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:24:57.876 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.879+0000 7ffbdc800640 1 --2- 172.21.15.146:0/1135491596 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffbd40d0d30 0x7ffbd40c7d70 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7ffbd408d390 tx=0x7ffbc402b870 comp rx=0 tx=0).stop 2024-03-04T09:24:57.876 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.879+0000 7ffbdc800640 1 -- 172.21.15.146:0/1135491596 shutdown_connections 2024-03-04T09:24:57.876 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.879+0000 7ffbdc800640 1 --2- 172.21.15.146:0/1135491596 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ffbd40c82b0 0x7ffbd40c8700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:24:57.876 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.879+0000 7ffbdc800640 1 --2- 172.21.15.146:0/1135491596 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffbd40d0d30 0x7ffbd40c7d70 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:24:57.876 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.879+0000 7ffbdc800640 1 --2- 172.21.15.146:0/1135491596 >> v2:172.21.15.6:3300/0 conn(0x7ffbd41616b0 0x7ffbd40d07f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:24:57.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.879+0000 7ffbdc800640 1 -- 172.21.15.146:0/1135491596 >> 172.21.15.146:0/1135491596 conn(0x7ffbd40d3eb0 msgr2=0x7ffbd40d42c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:24:57.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.879+0000 7ffbdc800640 1 -- 172.21.15.146:0/1135491596 shutdown_connections 2024-03-04T09:24:57.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.879+0000 7ffbdc800640 1 -- 172.21.15.146:0/1135491596 wait complete. 2024-03-04T09:24:57.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.881+0000 7ffbdc800640 1 Processor -- start 2024-03-04T09:24:57.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.881+0000 7ffbdc800640 1 -- start start 2024-03-04T09:24:57.880 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.882+0000 7ffbdc800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffbd40c82b0 0x7ffbd415af70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:24:57.880 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.882+0000 7ffbdc800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ffbd40d0d30 0x7ffbd40caf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:24:57.881 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.882+0000 7ffbdc800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffbd41616b0 0x7ffbd41f78a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:24:57.881 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.882+0000 7ffbdc800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbd40d4bc0 con 0x7ffbd41616b0 2024-03-04T09:24:57.881 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.882+0000 7ffbdc800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbd40d4a40 con 0x7ffbd40c82b0 2024-03-04T09:24:57.881 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.882+0000 7ffbdc800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbd40cac40 con 0x7ffbd40d0d30 2024-03-04T09:24:57.881 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.882+0000 7ffbd9e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffbd40c82b0 0x7ffbd415af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:24:57.881 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.882+0000 7ffbd9e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffbd40c82b0 0x7ffbd415af70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:53354/0 (socket says 172.21.15.146:53354) 2024-03-04T09:24:57.881 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.882+0000 7ffbd9e00640 1 -- 172.21.15.146:0/3821835537 learned_addr learned my addr 172.21.15.146:0/3821835537 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-03-04T09:24:57.881 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.882+0000 7ffbd9400640 1 --2- 172.21.15.146:0/3821835537 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ffbd40d0d30 0x7ffbd40caf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:24:57.881 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.883+0000 7ffbda800640 1 --2- 172.21.15.146:0/3821835537 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffbd41616b0 0x7ffbd41f78a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:24:57.881 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.883+0000 7ffbd9e00640 1 -- 172.21.15.146:0/3821835537 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ffbd40d0d30 msgr2=0x7ffbd40caf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:24:57.881 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.883+0000 7ffbd9e00640 1 --2- 172.21.15.146:0/3821835537 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ffbd40d0d30 0x7ffbd40caf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:24:57.881 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.883+0000 7ffbd9e00640 1 -- 172.21.15.146:0/3821835537 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffbd41616b0 msgr2=0x7ffbd41f78a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:24:57.881 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.883+0000 7ffbd9e00640 1 --2- 172.21.15.146:0/3821835537 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffbd41616b0 0x7ffbd41f78a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:24:57.881 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.883+0000 7ffbd9e00640 1 -- 172.21.15.146:0/3821835537 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbd40cb4c0 con 0x7ffbd40c82b0 2024-03-04T09:24:57.881 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.883+0000 7ffbda800640 1 --2- 172.21.15.146:0/3821835537 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffbd41616b0 0x7ffbd41f78a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:24:57.882 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.883+0000 7ffbd9e00640 1 --2- 172.21.15.146:0/3821835537 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffbd40c82b0 0x7ffbd415af70 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7ffbc800d800 tx=0x7ffbc800dcc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:24:57.882 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.883+0000 7ffbce000640 1 -- 172.21.15.146:0/3821835537 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbc800b6c0 con 0x7ffbd40c82b0 2024-03-04T09:24:57.882 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.883+0000 7ffbce000640 1 -- 172.21.15.146:0/3821835537 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbc8019070 con 0x7ffbd40c82b0 2024-03-04T09:24:57.882 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.883+0000 7ffbce000640 1 -- 172.21.15.146:0/3821835537 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbc800ba80 con 0x7ffbd40c82b0 2024-03-04T09:24:57.882 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.883+0000 7ffbdc800640 1 -- 172.21.15.146:0/3821835537 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbd41f80d0 con 0x7ffbd40c82b0 2024-03-04T09:24:57.882 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.883+0000 7ffbdc800640 1 -- 172.21.15.146:0/3821835537 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbd41f8690 con 0x7ffbd40c82b0 2024-03-04T09:24:57.882 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.883+0000 7ffbce000640 1 -- 172.21.15.146:0/3821835537 <== mon.1 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7ffbc800bc20 con 0x7ffbd40c82b0 2024-03-04T09:24:57.882 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.884+0000 7ffbce000640 1 -- 172.21.15.146:0/3821835537 <== mon.1 v2:172.21.15.146:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (secure 0 0 0) 0x7ffbc801eac0 con 0x7ffbd40c82b0 2024-03-04T09:24:57.882 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.885+0000 7ffbdc800640 1 -- 172.21.15.146:0/3821835537 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbd4161a90 con 0x7ffbd40c82b0 2024-03-04T09:24:57.883 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.887+0000 7ffbce000640 1 -- 172.21.15.146:0/3821835537 <== mon.1 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7ffbd4161a90 con 0x7ffbd40c82b0 2024-03-04T09:24:57.910 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-04T09:24:57.913+0000 7fe53068f040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-04T09:24:57.963 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-04T09:24:57.966+0000 7fe53068f040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-04T09:24:57.973 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-03-04T09:24:57.976+0000 7f43eb238040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-04T09:24:57.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.979+0000 7ffbdc800640 1 -- 172.21.15.146:0/3821835537 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd new", "uuid": "75a79b44-fa5f-426d-a7d3-71f21f78835c", "id": 7} v 0) v1 -- 0x7ffbd4073210 con 0x7ffbd40c82b0 2024-03-04T09:24:57.984 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.988+0000 7ffbce000640 1 -- 172.21.15.146:0/3821835537 <== mon.1 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "75a79b44-fa5f-426d-a7d3-71f21f78835c", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7ffbd4073210 con 0x7ffbd40c82b0 2024-03-04T09:24:57.985 INFO:teuthology.orchestra.run.smithi146.stdout:7 2024-03-04T09:24:57.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.989+0000 7ffbdc800640 1 -- 172.21.15.146:0/3821835537 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffbd40c82b0 msgr2=0x7ffbd415af70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:24:57.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.989+0000 7ffbdc800640 1 --2- 172.21.15.146:0/3821835537 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffbd40c82b0 0x7ffbd415af70 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7ffbc800d800 tx=0x7ffbc800dcc0 comp rx=0 tx=0).stop 2024-03-04T09:24:57.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.989+0000 7ffbdc800640 1 -- 172.21.15.146:0/3821835537 shutdown_connections 2024-03-04T09:24:57.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.989+0000 7ffbdc800640 1 --2- 172.21.15.146:0/3821835537 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffbd41616b0 0x7ffbd41f78a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:24:57.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.989+0000 7ffbdc800640 1 --2- 172.21.15.146:0/3821835537 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ffbd40d0d30 0x7ffbd40caf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:24:57.987 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.989+0000 7ffbdc800640 1 --2- 172.21.15.146:0/3821835537 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffbd40c82b0 0x7ffbd415af70 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:24:57.987 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.989+0000 7ffbdc800640 1 -- 172.21.15.146:0/3821835537 >> 172.21.15.146:0/3821835537 conn(0x7ffbd40d3eb0 msgr2=0x7ffbd40c9fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:24:57.987 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.989+0000 7ffbdc800640 1 -- 172.21.15.146:0/3821835537 shutdown_connections 2024-03-04T09:24:57.987 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-04T09:24:57.989+0000 7ffbdc800640 1 -- 172.21.15.146:0/3821835537 wait complete. 2024-03-04T09:24:58.003 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-04T09:24:58.003 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-03-04T09:24:58.011 INFO:tasks.ceph.osd.0:Started 2024-03-04T09:24:58.011 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-04T09:24:58.011 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-03-04T09:24:58.018 INFO:tasks.ceph.osd.1:Started 2024-03-04T09:24:58.018 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-04T09:24:58.018 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-03-04T09:24:58.022 INFO:tasks.ceph.osd.2:Started 2024-03-04T09:24:58.023 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-04T09:24:58.023 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-03-04T09:24:58.027 INFO:tasks.ceph.osd.3:Started 2024-03-04T09:24:58.028 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-04T09:24:58.028 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-03-04T09:24:58.029 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-03-04T09:24:58.032+0000 7f43eb238040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-04T09:24:58.035 INFO:tasks.ceph.osd.4:Started 2024-03-04T09:24:58.035 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-04T09:24:58.035 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-03-04T09:24:58.042 INFO:tasks.ceph.osd.5:Started 2024-03-04T09:24:58.042 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-04T09:24:58.042 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-03-04T09:24:58.049 INFO:tasks.ceph.osd.6:Started 2024-03-04T09:24:58.049 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-04T09:24:58.049 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-03-04T09:24:58.055 INFO:tasks.ceph.osd.7:Started 2024-03-04T09:24:58.056 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T09:24:58.212 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-04T09:24:58.216+0000 7fe53068f040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-04T09:24:58.251 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-03-04T09:24:58.254+0000 7f43eb238040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-04T09:24:58.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:58.274+0000 7fd5e1800640 1 Processor -- start 2024-03-04T09:24:58.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:58.275+0000 7fd5e1800640 1 -- start start 2024-03-04T09:24:58.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:58.276+0000 7fd5e1800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd5dc15e0b0 0x7fd5dc0cf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:24:58.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:58.276+0000 7fd5e1800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd5dc0d4a10 0x7fd5dc0d0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:24:58.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:58.276+0000 7fd5e1800640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5dc0b3710 con 0x7fd5dc0d0c80 2024-03-04T09:24:58.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:58.276+0000 7fd5e1800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5dc0d10f0 con 0x7fd5dc15e0b0 2024-03-04T09:24:58.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:58.276+0000 7fd5e1800640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5dc0b09e0 con 0x7fd5dc0d4a10 2024-03-04T09:24:58.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:58.276+0000 7fd5dbe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd5dc15e0b0 0x7fd5dc0cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:24:58.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:58.276+0000 7fd5dbe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd5dc15e0b0 0x7fd5dc0cf310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45092/0 (socket says 172.21.15.6:45092) 2024-03-04T09:24:58.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:58.276+0000 7fd5dbe00640 1 -- 172.21.15.6:0/3763190323 learned_addr learned my addr 172.21.15.6:0/3763190323 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:24:58.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:58.276+0000 7fd5d3e00640 1 --2- 172.21.15.6:0/3763190323 >> v2:172.21.15.146:3300/0 conn(0x7fd5dc0d4a10 0x7fd5dc0d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:24:58.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:58.276+0000 7fd5dbe00640 1 -- 172.21.15.6:0/3763190323 >> v2:172.21.15.146:3300/0 conn(0x7fd5dc0d4a10 msgr2=0x7fd5dc0d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:24:58.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:58.276+0000 7fd5dbe00640 1 --2- 172.21.15.6:0/3763190323 >> v2:172.21.15.146:3300/0 conn(0x7fd5dc0d4a10 0x7fd5dc0d0740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:24:58.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:58.276+0000 7fd5dbe00640 1 -- 172.21.15.6:0/3763190323 >> v1:172.21.15.146:6789/0 conn(0x7fd5dc0d0c80 legacy=0x7fd5dc0d1900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:24:58.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:58.276+0000 7fd5dbe00640 1 -- 172.21.15.6:0/3763190323 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5dc0b39f0 con 0x7fd5dc15e0b0 2024-03-04T09:24:58.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:58.277+0000 7fd5dbe00640 1 --2- 172.21.15.6:0/3763190323 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd5dc15e0b0 0x7fd5dc0cf310 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fd5dc06b8c0 tx=0x7fd5cc02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1afd802f85dfb991 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:24:58.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:58.278+0000 7fd5db400640 1 -- 172.21.15.6:0/3763190323 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5cc034070 con 0x7fd5dc15e0b0 2024-03-04T09:24:58.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:58.278+0000 7fd5db400640 1 -- 172.21.15.6:0/3763190323 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5cc030860 con 0x7fd5dc15e0b0 2024-03-04T09:24:58.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:58.278+0000 7fd5db400640 1 -- 172.21.15.6:0/3763190323 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5cc030c30 con 0x7fd5dc15e0b0 2024-03-04T09:24:58.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:58.278+0000 7fd5e1800640 1 -- 172.21.15.6:0/3763190323 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd5dc15e0b0 msgr2=0x7fd5dc0cf310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:24:58.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:58.278+0000 7fd5e1800640 1 --2- 172.21.15.6:0/3763190323 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd5dc15e0b0 0x7fd5dc0cf310 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fd5dc06b8c0 tx=0x7fd5cc02b9a0 comp rx=0 tx=0).stop 2024-03-04T09:24:58.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:58.278+0000 7fd5e1800640 1 -- 172.21.15.6:0/3763190323 shutdown_connections 2024-03-04T09:24:58.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:58.278+0000 7fd5e1800640 1 --2- 172.21.15.6:0/3763190323 >> v2:172.21.15.146:3300/0 conn(0x7fd5dc0d4a10 0x7fd5dc0d0740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:24:58.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:58.278+0000 7fd5e1800640 1 --2- 172.21.15.6:0/3763190323 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd5dc15e0b0 0x7fd5dc0cf310 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:24:58.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:58.278+0000 7fd5e1800640 1 -- 172.21.15.6:0/3763190323 >> 172.21.15.6:0/3763190323 conn(0x7fd5dc0d3d30 msgr2=0x7fd5dc0d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:24:58.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:58.279+0000 7fd5e1800640 1 -- 172.21.15.6:0/3763190323 shutdown_connections 2024-03-04T09:24:58.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:58.279+0000 7fd5e1800640 1 -- 172.21.15.6:0/3763190323 wait complete. 2024-03-04T09:24:58.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:58.280+0000 7fd5e1800640 1 Processor -- start 2024-03-04T09:24:58.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:58.280+0000 7fd5e1800640 1 -- start start 2024-03-04T09:24:58.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:58.281+0000 7fd5e1800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd5dc0d0c80 0x7fd5dc160da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:24:58.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:58.281+0000 7fd5e1800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd5dc0d4a10 0x7fd5dc0cc220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:24:58.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:58.281+0000 7fd5e1800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd5dc15e0b0 0x7fd5dc0b7540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:24:58.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:58.281+0000 7fd5e1800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5dc15e490 con 0x7fd5dc0d4a10 2024-03-04T09:24:58.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:58.281+0000 7fd5e1800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5dc0b3fb0 con 0x7fd5dc0d0c80 2024-03-04T09:24:58.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:58.281+0000 7fd5e1800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5dc0cfc60 con 0x7fd5dc15e0b0 2024-03-04T09:24:58.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:58.281+0000 7fd5d3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd5dc0d4a10 0x7fd5dc0cc220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:24:58.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:58.281+0000 7fd5dbe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd5dc0d0c80 0x7fd5dc160da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:24:58.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:58.281+0000 7fd5d3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd5dc0d4a10 0x7fd5dc0cc220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45098/0 (socket says 172.21.15.6:45098) 2024-03-04T09:24:58.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:58.281+0000 7fd5dbe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd5dc0d0c80 0x7fd5dc160da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.6:56360/0 (socket says 172.21.15.6:56360) 2024-03-04T09:24:58.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:58.281+0000 7fd5d3e00640 1 -- 172.21.15.6:0/4108504843 learned_addr learned my addr 172.21.15.6:0/4108504843 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:24:58.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:58.281+0000 7fd5e0e00640 1 --2- 172.21.15.6:0/4108504843 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd5dc15e0b0 0x7fd5dc0b7540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:24:58.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:58.282+0000 7fd5dbe00640 1 -- 172.21.15.6:0/4108504843 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd5dc15e0b0 msgr2=0x7fd5dc0b7540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:24:58.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:58.282+0000 7fd5dbe00640 1 --2- 172.21.15.6:0/4108504843 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd5dc15e0b0 0x7fd5dc0b7540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:24:58.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:58.282+0000 7fd5dbe00640 1 -- 172.21.15.6:0/4108504843 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd5dc0d4a10 msgr2=0x7fd5dc0cc220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:24:58.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:58.282+0000 7fd5dbe00640 1 --2- 172.21.15.6:0/4108504843 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd5dc0d4a10 0x7fd5dc0cc220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:24:58.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:58.282+0000 7fd5dbe00640 1 -- 172.21.15.6:0/4108504843 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5dc15a7f0 con 0x7fd5dc0d0c80 2024-03-04T09:24:58.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:58.282+0000 7fd5dbe00640 1 --2- 172.21.15.6:0/4108504843 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd5dc0d0c80 0x7fd5dc160da0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fd5cc033000 tx=0x7fd5cc0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:24:58.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:58.282+0000 7fd5d8c00640 1 -- 172.21.15.6:0/4108504843 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5cc034040 con 0x7fd5dc0d0c80 2024-03-04T09:24:58.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:58.282+0000 7fd5d8c00640 1 -- 172.21.15.6:0/4108504843 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5cc039d90 con 0x7fd5dc0d0c80 2024-03-04T09:24:58.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:58.282+0000 7fd5d8c00640 1 -- 172.21.15.6:0/4108504843 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5cc043610 con 0x7fd5dc0d0c80 2024-03-04T09:24:58.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:58.283+0000 7fd5e1800640 1 -- 172.21.15.6:0/4108504843 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5dc161660 con 0x7fd5dc0d0c80 2024-03-04T09:24:58.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:58.283+0000 7fd5e1800640 1 -- 172.21.15.6:0/4108504843 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5dc0b6a60 con 0x7fd5dc0d0c80 2024-03-04T09:24:58.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:58.283+0000 7fd5d8c00640 1 -- 172.21.15.6:0/4108504843 <== mon.1 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fd5cc04a060 con 0x7fd5dc0d0c80 2024-03-04T09:24:58.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:58.283+0000 7fd5d8c00640 1 -- 172.21.15.6:0/4108504843 <== mon.1 v2:172.21.15.146:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7fd5cc02fa80 con 0x7fd5dc0d0c80 2024-03-04T09:24:58.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:58.284+0000 7fd5e1800640 1 -- 172.21.15.6:0/4108504843 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5dc04f330 con 0x7fd5dc0d0c80 2024-03-04T09:24:58.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:58.286+0000 7fd5d8c00640 1 -- 172.21.15.6:0/4108504843 <== mon.1 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fd5cc03e070 con 0x7fd5dc0d0c80 2024-03-04T09:24:58.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:58.351+0000 7fd5e1800640 1 -- 172.21.15.6:0/4108504843 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd5dc074920 con 0x7fd5dc0d0c80 2024-03-04T09:24:58.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:58.353+0000 7fd5d8c00640 1 -- 172.21.15.6:0/4108504843 <== mon.1 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (secure 0 0 0) 0x7fd5cc02fd80 con 0x7fd5dc0d0c80 2024-03-04T09:24:58.350 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:24:58.350 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":10,"fsid":"edce0578-5ae4-4402-b45a-552bdef84d7c","created":"2024-03-04T09:24:54.523780+0000","modified":"2024-03-04T09:24:57.982077+0000","last_up_change":"0.000000","last_in_change":"2024-03-04T09:24:57.982077+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7a96574f-81f1-4bc6-8463-757520b84a44","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"7d79b9c1-6b87-476a-bc36-7db5ee64cd90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"492331b9-4ee9-40b6-bacd-85d44cd6c45a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"21bb2efe-9ba7-4bdc-bc31-180cd1102e21","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6e2c4b71-19d9-4737-83d9-558705d8644a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"fb69c525-f1be-4b7d-ab32-6c29b0d6fddb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c32e7fed-370d-4401-8ce0-6a4335b0d168","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"75a79b44-fa5f-426d-a7d3-71f21f78835c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T09:24:58.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:58.354+0000 7fd5e1800640 1 -- 172.21.15.6:0/4108504843 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd5dc0d0c80 msgr2=0x7fd5dc160da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:24:58.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:58.354+0000 7fd5e1800640 1 --2- 172.21.15.6:0/4108504843 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd5dc0d0c80 0x7fd5dc160da0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fd5cc033000 tx=0x7fd5cc0046c0 comp rx=0 tx=0).stop 2024-03-04T09:24:58.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:58.354+0000 7fd5e1800640 1 -- 172.21.15.6:0/4108504843 shutdown_connections 2024-03-04T09:24:58.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:58.354+0000 7fd5e1800640 1 --2- 172.21.15.6:0/4108504843 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd5dc15e0b0 0x7fd5dc0b7540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:24:58.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:58.354+0000 7fd5e1800640 1 --2- 172.21.15.6:0/4108504843 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd5dc0d4a10 0x7fd5dc0cc220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:24:58.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:58.354+0000 7fd5e1800640 1 --2- 172.21.15.6:0/4108504843 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd5dc0d0c80 0x7fd5dc160da0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:24:58.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:58.354+0000 7fd5e1800640 1 -- 172.21.15.6:0/4108504843 >> 172.21.15.6:0/4108504843 conn(0x7fd5dc0d3d30 msgr2=0x7fd5dc0b5ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:24:58.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:58.354+0000 7fd5e1800640 1 -- 172.21.15.6:0/4108504843 shutdown_connections 2024-03-04T09:24:58.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:24:58.355+0000 7fd5e1800640 1 -- 172.21.15.6:0/4108504843 wait complete. 2024-03-04T09:24:58.366 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-04T09:24:58.367 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-04T09:24:58.707 INFO:tasks.ceph.osd.3.smithi006.stderr:2024-03-04T09:24:58.710+0000 7f76dd56f640 -1 Falling back to public interface 2024-03-04T09:24:58.738 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-03-04T09:24:58.741+0000 7f5ff7815640 -1 Falling back to public interface 2024-03-04T09:24:58.741 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-03-04T09:24:58.745+0000 7f8ba2611640 -1 Falling back to public interface 2024-03-04T09:24:58.750 INFO:tasks.ceph.osd.1.smithi006.stderr:2024-03-04T09:24:58.753+0000 7fa110410640 -1 Falling back to public interface 2024-03-04T09:24:58.751 INFO:tasks.ceph.osd.2.smithi006.stderr:2024-03-04T09:24:58.754+0000 7f146ef49640 -1 Falling back to public interface 2024-03-04T09:24:58.755 INFO:tasks.ceph.osd.7.smithi146.stderr:2024-03-04T09:24:58.759+0000 7faf51cb3640 -1 Falling back to public interface 2024-03-04T09:24:58.759 INFO:tasks.ceph.osd.6.smithi146.stderr:2024-03-04T09:24:58.763+0000 7f3e404d3640 -1 Falling back to public interface 2024-03-04T09:24:58.761 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-03-04T09:24:58.764+0000 7fd38dd9a640 -1 Falling back to public interface 2024-03-04T09:24:59.173 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-04T09:24:59.324 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-03-04T09:24:59.327+0000 7f43eb238040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-04T09:24:59.340 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-04T09:24:59.343+0000 7fe53068f040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-04T09:24:59.348 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:24:59.352+0000 7f7605e00640 1 Processor -- start 2024-03-04T09:24:59.348 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:24:59.352+0000 7f7605e00640 1 -- start start 2024-03-04T09:24:59.352 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:24:59.354+0000 7f7605e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f760015e2c0 0x7f760015f310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:24:59.352 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:24:59.354+0000 7f7605e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7600071e50 0x7f76000b1bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:24:59.352 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:24:59.354+0000 7f7605e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f76000b2110 0x7f76000ce600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:24:59.352 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:24:59.354+0000 7f7605e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f760015f850 con 0x7f76000b2110 2024-03-04T09:24:59.352 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:24:59.354+0000 7f7605e00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f760015e6a0 con 0x7f760015e2c0 2024-03-04T09:24:59.352 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:24:59.354+0000 7f7605e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7600071390 con 0x7f7600071e50 2024-03-04T09:24:59.352 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:24:59.354+0000 7f75ff400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f76000b2110 0x7f76000ce600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:24:59.352 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:24:59.354+0000 7f75fea00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f760015e2c0 0x7f760015f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:24:59.352 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:24:59.354+0000 7f75fe000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7600071e50 0x7f76000b1bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:24:59.352 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:24:59.354+0000 7f75ff400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f76000b2110 0x7f76000ce600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60252/0 (socket says 172.21.15.6:60252) 2024-03-04T09:24:59.352 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:24:59.354+0000 7f75ff400640 1 -- 172.21.15.6:0/3896849359 learned_addr learned my addr 172.21.15.6:0/3896849359 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:24:59.353 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:24:59.355+0000 7f75ff400640 1 -- 172.21.15.6:0/3896849359 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7600071e50 msgr2=0x7f76000b1bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:24:59.353 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:24:59.355+0000 7f75ff400640 1 --2- 172.21.15.6:0/3896849359 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7600071e50 0x7f76000b1bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:24:59.353 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:24:59.355+0000 7f75ff400640 1 -- 172.21.15.6:0/3896849359 >> v2:172.21.15.146:3300/0 conn(0x7f760015e2c0 msgr2=0x7f760015f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:24:59.353 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:24:59.355+0000 7f75ff400640 1 --2- 172.21.15.6:0/3896849359 >> v2:172.21.15.146:3300/0 conn(0x7f760015e2c0 0x7f760015f310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:24:59.353 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:24:59.355+0000 7f75ff400640 1 -- 172.21.15.6:0/3896849359 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76000ccc90 con 0x7f76000b2110 2024-03-04T09:24:59.354 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:24:59.356+0000 7f75ff400640 1 --2- 172.21.15.6:0/3896849359 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f76000b2110 0x7f76000ce600 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f760006f1f0 tx=0x7f75f802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a617b21823018e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:24:59.354 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:24:59.356+0000 7f75fd600640 1 -- 172.21.15.6:0/3896849359 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75f802c9e0 con 0x7f76000b2110 2024-03-04T09:24:59.354 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:24:59.356+0000 7f75fd600640 1 -- 172.21.15.6:0/3896849359 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75f802cb80 con 0x7f76000b2110 2024-03-04T09:24:59.354 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:24:59.356+0000 7f75fd600640 1 -- 172.21.15.6:0/3896849359 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75f80343f0 con 0x7f76000b2110 2024-03-04T09:24:59.354 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:24:59.357+0000 7f7605e00640 1 -- 172.21.15.6:0/3896849359 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f76000b2110 msgr2=0x7f76000ce600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:24:59.354 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:24:59.357+0000 7f7605e00640 1 --2- 172.21.15.6:0/3896849359 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f76000b2110 0x7f76000ce600 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f760006f1f0 tx=0x7f75f802be40 comp rx=0 tx=0).stop 2024-03-04T09:24:59.354 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:24:59.357+0000 7f7605e00640 1 -- 172.21.15.6:0/3896849359 shutdown_connections 2024-03-04T09:24:59.355 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:24:59.357+0000 7f7605e00640 1 --2- 172.21.15.6:0/3896849359 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f76000b2110 0x7f76000ce600 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:24:59.355 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:24:59.357+0000 7f7605e00640 1 --2- 172.21.15.6:0/3896849359 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7600071e50 0x7f76000b1bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:24:59.355 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:24:59.357+0000 7f7605e00640 1 --2- 172.21.15.6:0/3896849359 >> v2:172.21.15.146:3300/0 conn(0x7f760015e2c0 0x7f760015f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:24:59.355 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:24:59.357+0000 7f7605e00640 1 -- 172.21.15.6:0/3896849359 >> 172.21.15.6:0/3896849359 conn(0x7f760006af70 msgr2=0x7f760006b350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:24:59.355 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:24:59.357+0000 7f7605e00640 1 -- 172.21.15.6:0/3896849359 shutdown_connections 2024-03-04T09:24:59.355 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:24:59.357+0000 7f7605e00640 1 -- 172.21.15.6:0/3896849359 wait complete. 2024-03-04T09:24:59.355 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:24:59.359+0000 7f7605e00640 1 Processor -- start 2024-03-04T09:24:59.356 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:24:59.359+0000 7f7605e00640 1 -- start start 2024-03-04T09:24:59.358 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:24:59.360+0000 7f7605e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7600071e50 0x7f76001f3540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:24:59.359 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:24:59.360+0000 7f7605e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f76000b2110 0x7f7600158e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:24:59.359 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:24:59.360+0000 7f7605e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f760015e2c0 0x7f7600159340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:24:59.359 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:24:59.360+0000 7f7605e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7600071930 con 0x7f76000b2110 2024-03-04T09:24:59.359 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:24:59.360+0000 7f7605e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7600071650 con 0x7f760015e2c0 2024-03-04T09:24:59.359 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:24:59.360+0000 7f7605e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7600073170 con 0x7f7600071e50 2024-03-04T09:24:59.359 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:24:59.360+0000 7f75fe000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f76000b2110 0x7f7600158e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:24:59.359 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:24:59.360+0000 7f75fe000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f76000b2110 0x7f7600158e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60266/0 (socket says 172.21.15.6:60266) 2024-03-04T09:24:59.359 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:24:59.360+0000 7f75fe000640 1 -- 172.21.15.6:0/2021840952 learned_addr learned my addr 172.21.15.6:0/2021840952 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:24:59.359 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:24:59.360+0000 7f75fea00640 1 --2- 172.21.15.6:0/2021840952 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7600071e50 0x7f76001f3540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:24:59.359 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:24:59.360+0000 7f75ff400640 1 --2- 172.21.15.6:0/2021840952 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f760015e2c0 0x7f7600159340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:24:59.359 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:24:59.361+0000 7f75fe000640 1 -- 172.21.15.6:0/2021840952 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7600071e50 msgr2=0x7f76001f3540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:24:59.359 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:24:59.361+0000 7f75fe000640 1 --2- 172.21.15.6:0/2021840952 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7600071e50 0x7f76001f3540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:24:59.359 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:24:59.361+0000 7f75fe000640 1 -- 172.21.15.6:0/2021840952 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f760015e2c0 msgr2=0x7f7600159340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:24:59.360 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:24:59.361+0000 7f75fe000640 1 --2- 172.21.15.6:0/2021840952 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f760015e2c0 0x7f7600159340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:24:59.360 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:24:59.361+0000 7f75fe000640 1 -- 172.21.15.6:0/2021840952 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76001f3e20 con 0x7f76000b2110 2024-03-04T09:24:59.360 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:24:59.361+0000 7f75fe000640 1 --2- 172.21.15.6:0/2021840952 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f76000b2110 0x7f7600158e00 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f75ec00e890 tx=0x7f75ec00ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:24:59.360 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:24:59.361+0000 7f75f7400640 1 -- 172.21.15.6:0/2021840952 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75ec017070 con 0x7f76000b2110 2024-03-04T09:24:59.360 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:24:59.361+0000 7f75f7400640 1 -- 172.21.15.6:0/2021840952 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75ec00c940 con 0x7f76000b2110 2024-03-04T09:24:59.360 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:24:59.361+0000 7f7605e00640 1 -- 172.21.15.6:0/2021840952 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76001f4100 con 0x7f76000b2110 2024-03-04T09:24:59.360 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:24:59.361+0000 7f75f7400640 1 -- 172.21.15.6:0/2021840952 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75ec00cd10 con 0x7f76000b2110 2024-03-04T09:24:59.360 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:24:59.361+0000 7f7605e00640 1 -- 172.21.15.6:0/2021840952 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76000a9d20 con 0x7f76000b2110 2024-03-04T09:24:59.360 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:24:59.362+0000 7f75f7400640 1 -- 172.21.15.6:0/2021840952 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f75ec01a070 con 0x7f76000b2110 2024-03-04T09:24:59.360 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:24:59.362+0000 7f75f7400640 1 -- 172.21.15.6:0/2021840952 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f75ec006020 con 0x7f76000b2110 2024-03-04T09:24:59.361 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:24:59.363+0000 7f7605e00640 1 -- 172.21.15.6:0/2021840952 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f760004f330 con 0x7f76000b2110 2024-03-04T09:24:59.362 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:24:59.365+0000 7f75f7400640 1 -- 172.21.15.6:0/2021840952 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f75ec006520 con 0x7f76000b2110 2024-03-04T09:24:59.429 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:24:59.433+0000 7f7605e00640 1 -- 172.21.15.6:0/2021840952 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f76000a96f0 con 0x7f76000b2110 2024-03-04T09:24:59.430 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:24:59.434+0000 7f75f7400640 1 -- 172.21.15.6:0/2021840952 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (secure 0 0 0) 0x7f75ec006700 con 0x7f76000b2110 2024-03-04T09:24:59.431 INFO:teuthology.misc.health.smithi006.stdout: 2024-03-04T09:24:59.431 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":10,"fsid":"edce0578-5ae4-4402-b45a-552bdef84d7c","created":"2024-03-04T09:24:54.523780+0000","modified":"2024-03-04T09:24:57.982077+0000","last_up_change":"0.000000","last_in_change":"2024-03-04T09:24:57.982077+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7a96574f-81f1-4bc6-8463-757520b84a44","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"7d79b9c1-6b87-476a-bc36-7db5ee64cd90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"492331b9-4ee9-40b6-bacd-85d44cd6c45a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"21bb2efe-9ba7-4bdc-bc31-180cd1102e21","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6e2c4b71-19d9-4737-83d9-558705d8644a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"fb69c525-f1be-4b7d-ab32-6c29b0d6fddb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c32e7fed-370d-4401-8ce0-6a4335b0d168","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"75a79b44-fa5f-426d-a7d3-71f21f78835c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T09:24:59.432 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:24:59.435+0000 7f7605e00640 1 -- 172.21.15.6:0/2021840952 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f76000b2110 msgr2=0x7f7600158e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:24:59.432 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:24:59.435+0000 7f7605e00640 1 --2- 172.21.15.6:0/2021840952 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f76000b2110 0x7f7600158e00 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f75ec00e890 tx=0x7f75ec00ed50 comp rx=0 tx=0).stop 2024-03-04T09:24:59.432 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:24:59.435+0000 7f7605e00640 1 -- 172.21.15.6:0/2021840952 shutdown_connections 2024-03-04T09:24:59.432 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:24:59.435+0000 7f7605e00640 1 --2- 172.21.15.6:0/2021840952 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f760015e2c0 0x7f7600159340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:24:59.432 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:24:59.435+0000 7f7605e00640 1 --2- 172.21.15.6:0/2021840952 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f76000b2110 0x7f7600158e00 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:24:59.432 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:24:59.435+0000 7f7605e00640 1 --2- 172.21.15.6:0/2021840952 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7600071e50 0x7f76001f3540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:24:59.432 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:24:59.435+0000 7f7605e00640 1 -- 172.21.15.6:0/2021840952 >> 172.21.15.6:0/2021840952 conn(0x7f760006af70 msgr2=0x7f7600158080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:24:59.432 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:24:59.435+0000 7f7605e00640 1 -- 172.21.15.6:0/2021840952 shutdown_connections 2024-03-04T09:24:59.433 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:24:59.435+0000 7f7605e00640 1 -- 172.21.15.6:0/2021840952 wait complete. 2024-03-04T09:24:59.444 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-04T09:24:59.641 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-03-04T09:24:59.644+0000 7f43eb238040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-04T09:24:59.672 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-04T09:24:59.675+0000 7fe53068f040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-04T09:24:59.700 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-03-04T09:24:59.704+0000 7f43eb238040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-04T09:24:59.737 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-04T09:24:59.740+0000 7fe53068f040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-04T09:24:59.829 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-03-04T09:24:59.832+0000 7f43eb238040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-04T09:24:59.876 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-04T09:24:59.879+0000 7fe53068f040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-04T09:25:00.876 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-03-04T09:25:00.880+0000 7f438bc00640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-04T09:25:00.877 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-03-04T09:25:00.881+0000 7f438bc00640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-04T09:25:02.280 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-03-04T09:25:02.283+0000 7f5ff7815640 -1 osd.4 0 log_to_monitors true 2024-03-04T09:25:02.294 INFO:tasks.ceph.osd.1.smithi006.stderr:2024-03-04T09:25:02.298+0000 7fa110410640 -1 osd.1 0 log_to_monitors true 2024-03-04T09:25:02.310 INFO:tasks.ceph.osd.2.smithi006.stderr:2024-03-04T09:25:02.314+0000 7f146ef49640 -1 osd.2 0 log_to_monitors true 2024-03-04T09:25:02.329 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-03-04T09:25:02.333+0000 7fd38dd9a640 -1 osd.0 0 log_to_monitors true 2024-03-04T09:25:02.330 INFO:tasks.ceph.osd.6.smithi146.stderr:2024-03-04T09:25:02.334+0000 7f3e404d3640 -1 osd.6 0 log_to_monitors true 2024-03-04T09:25:02.526 INFO:tasks.ceph.osd.3.smithi006.stderr:2024-03-04T09:25:02.529+0000 7f76dd56f640 -1 osd.3 0 log_to_monitors true 2024-03-04T09:25:02.576 INFO:tasks.ceph.osd.7.smithi146.stderr:2024-03-04T09:25:02.578+0000 7faf51cb3640 -1 osd.7 0 log_to_monitors true 2024-03-04T09:25:02.576 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-03-04T09:25:02.580+0000 7f8ba2611640 -1 osd.5 0 log_to_monitors true 2024-03-04T09:25:06.089 INFO:tasks.ceph.osd.1.smithi006.stderr:2024-03-04T09:25:06.092+0000 7fa10a200640 -1 osd.1 0 waiting for initial osdmap 2024-03-04T09:25:06.096 INFO:tasks.ceph.osd.1.smithi006.stderr:2024-03-04T09:25:06.099+0000 7fa100c00640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-04T09:25:06.100 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-03-04T09:25:06.103+0000 7f8b9e800640 -1 osd.5 0 waiting for initial osdmap 2024-03-04T09:25:06.109 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-03-04T09:25:06.112+0000 7f8b93400640 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-04T09:25:06.185 INFO:tasks.ceph.osd.2.smithi006.stderr:2024-03-04T09:25:06.188+0000 7f1469400640 -1 osd.2 0 waiting for initial osdmap 2024-03-04T09:25:06.191 INFO:tasks.ceph.osd.2.smithi006.stderr:2024-03-04T09:25:06.194+0000 7f145fe00640 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-04T09:25:06.198 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-03-04T09:25:06.201+0000 7fd38a000640 -1 osd.0 0 waiting for initial osdmap 2024-03-04T09:25:06.205 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-03-04T09:25:06.208+0000 7fd37ec00640 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-04T09:25:06.213 INFO:tasks.ceph.osd.7.smithi146.stderr:2024-03-04T09:25:06.216+0000 7faf4c200640 -1 osd.7 0 waiting for initial osdmap 2024-03-04T09:25:06.219 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-03-04T09:25:06.223+0000 7f5ff1600640 -1 osd.4 0 waiting for initial osdmap 2024-03-04T09:25:06.222 INFO:tasks.ceph.osd.7.smithi146.stderr:2024-03-04T09:25:06.225+0000 7faf42c00640 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-04T09:25:06.224 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-03-04T09:25:06.228+0000 7f5fe8000640 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-04T09:25:06.233 INFO:tasks.ceph.osd.6.smithi146.stderr:2024-03-04T09:25:06.236+0000 7f3e3c800640 -1 osd.6 0 waiting for initial osdmap 2024-03-04T09:25:06.240 INFO:tasks.ceph.osd.6.smithi146.stderr:2024-03-04T09:25:06.244+0000 7f3e31400640 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-04T09:25:06.247 INFO:tasks.ceph.osd.3.smithi006.stderr:2024-03-04T09:25:06.251+0000 7f76d9800640 -1 osd.3 0 waiting for initial osdmap 2024-03-04T09:25:06.250 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-04T09:25:06.267 INFO:tasks.ceph.osd.3.smithi006.stderr:2024-03-04T09:25:06.270+0000 7f76ce400640 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-04T09:25:06.399 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:06.402+0000 7ff612400640 1 Processor -- start 2024-03-04T09:25:06.399 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:06.402+0000 7ff612400640 1 -- start start 2024-03-04T09:25:06.401 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:06.404+0000 7ff612400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff60c15bef0 0x7ff60c0cc920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:06.401 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:06.404+0000 7ff612400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ff60c070c50 0x7ff60c0cce60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:06.401 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:06.404+0000 7ff612400640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff60c06d9d0 con 0x7ff60c155440 2024-03-04T09:25:06.401 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:06.404+0000 7ff612400640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff60c06ced0 con 0x7ff60c15bef0 2024-03-04T09:25:06.402 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:06.404+0000 7ff612400640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff60c06d130 con 0x7ff60c070c50 2024-03-04T09:25:06.402 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:06.404+0000 7ff60be00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff60c15bef0 0x7ff60c0cc920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:06.402 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:06.404+0000 7ff60be00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff60c15bef0 0x7ff60c0cc920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60302/0 (socket says 172.21.15.6:60302) 2024-03-04T09:25:06.402 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:06.404+0000 7ff60be00640 1 -- 172.21.15.6:0/1778089945 learned_addr learned my addr 172.21.15.6:0/1778089945 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:25:06.402 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:06.404+0000 7ff611a00640 1 --2- 172.21.15.6:0/1778089945 >> v2:172.21.15.146:3300/0 conn(0x7ff60c070c50 0x7ff60c0cce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:06.402 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:06.404+0000 7ff60be00640 1 -- 172.21.15.6:0/1778089945 >> v2:172.21.15.146:3300/0 conn(0x7ff60c070c50 msgr2=0x7ff60c0cce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:06.402 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:06.404+0000 7ff60be00640 1 --2- 172.21.15.6:0/1778089945 >> v2:172.21.15.146:3300/0 conn(0x7ff60c070c50 0x7ff60c0cce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:06.402 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:06.404+0000 7ff60be00640 1 -- 172.21.15.6:0/1778089945 >> v1:172.21.15.146:6789/0 conn(0x7ff60c155440 legacy=0x7ff60c06c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:06.402 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:06.404+0000 7ff60be00640 1 -- 172.21.15.6:0/1778089945 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff60c06d410 con 0x7ff60c15bef0 2024-03-04T09:25:06.403 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:06.405+0000 7ff60be00640 1 --2- 172.21.15.6:0/1778089945 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff60c15bef0 0x7ff60c0cc920 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ff60c0b50e0 tx=0x7ff5f802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70377cc954e6ea19 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:06.403 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:06.406+0000 7ff60b400640 1 -- 172.21.15.6:0/1778089945 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5f8035020 con 0x7ff60c15bef0 2024-03-04T09:25:06.404 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:06.406+0000 7ff60b400640 1 -- 172.21.15.6:0/1778089945 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5f8031870 con 0x7ff60c15bef0 2024-03-04T09:25:06.404 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:06.406+0000 7ff60b400640 1 -- 172.21.15.6:0/1778089945 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5f8031c40 con 0x7ff60c15bef0 2024-03-04T09:25:06.404 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:06.406+0000 7ff612400640 1 -- 172.21.15.6:0/1778089945 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff60c15bef0 msgr2=0x7ff60c0cc920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:06.404 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:06.406+0000 7ff612400640 1 --2- 172.21.15.6:0/1778089945 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff60c15bef0 0x7ff60c0cc920 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ff60c0b50e0 tx=0x7ff5f802b910 comp rx=0 tx=0).stop 2024-03-04T09:25:06.404 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:06.406+0000 7ff612400640 1 -- 172.21.15.6:0/1778089945 shutdown_connections 2024-03-04T09:25:06.404 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:06.406+0000 7ff612400640 1 --2- 172.21.15.6:0/1778089945 >> v2:172.21.15.146:3300/0 conn(0x7ff60c070c50 0x7ff60c0cce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:06.404 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:06.406+0000 7ff612400640 1 --2- 172.21.15.6:0/1778089945 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff60c15bef0 0x7ff60c0cc920 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:06.404 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:06.406+0000 7ff612400640 1 -- 172.21.15.6:0/1778089945 >> 172.21.15.6:0/1778089945 conn(0x7ff60c06b390 msgr2=0x7ff60c06b7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:06.404 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:06.406+0000 7ff612400640 1 -- 172.21.15.6:0/1778089945 shutdown_connections 2024-03-04T09:25:06.404 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:06.407+0000 7ff612400640 1 -- 172.21.15.6:0/1778089945 wait complete. 2024-03-04T09:25:06.405 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:06.408+0000 7ff612400640 1 Processor -- start 2024-03-04T09:25:06.405 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:06.408+0000 7ff612400640 1 -- start start 2024-03-04T09:25:06.407 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:06.409+0000 7ff612400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff60c070c50 0x7ff60c155870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:06.407 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:06.409+0000 7ff612400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff60c155440 0x7ff60c148220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:06.407 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:06.409+0000 7ff612400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff60c15bef0 0x7ff60c159cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:06.407 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:06.409+0000 7ff612400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff60c0b3e80 con 0x7ff60c155440 2024-03-04T09:25:06.408 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:06.409+0000 7ff612400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff60c0b3960 con 0x7ff60c070c50 2024-03-04T09:25:06.408 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:06.409+0000 7ff612400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff60c0b4000 con 0x7ff60c15bef0 2024-03-04T09:25:06.408 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:06.409+0000 7ff60be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff60c155440 0x7ff60c148220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:06.408 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:06.409+0000 7ff60be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff60c155440 0x7ff60c148220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60310/0 (socket says 172.21.15.6:60310) 2024-03-04T09:25:06.408 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:06.409+0000 7ff60be00640 1 -- 172.21.15.6:0/2684015214 learned_addr learned my addr 172.21.15.6:0/2684015214 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:25:06.408 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:06.410+0000 7ff60be00640 1 -- 172.21.15.6:0/2684015214 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff60c15bef0 msgr2=0x7ff60c159cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:06.408 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:06.410+0000 7ff60be00640 1 --2- 172.21.15.6:0/2684015214 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff60c15bef0 0x7ff60c159cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:06.408 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:06.410+0000 7ff60be00640 1 -- 172.21.15.6:0/2684015214 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff60c070c50 msgr2=0x7ff60c155870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:06.408 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:06.410+0000 7ff611000640 1 --2- 172.21.15.6:0/2684015214 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff60c070c50 0x7ff60c155870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:06.408 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:06.410+0000 7ff60be00640 1 --2- 172.21.15.6:0/2684015214 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff60c070c50 0x7ff60c155870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:06.408 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:06.410+0000 7ff60be00640 1 -- 172.21.15.6:0/2684015214 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff60c155e50 con 0x7ff60c155440 2024-03-04T09:25:06.408 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:06.410+0000 7ff60be00640 1 --2- 172.21.15.6:0/2684015214 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff60c155440 0x7ff60c148220 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7ff5f802b950 tx=0x7ff5f80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:06.408 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:06.410+0000 7ff608c00640 1 -- 172.21.15.6:0/2684015214 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5f8040070 con 0x7ff60c155440 2024-03-04T09:25:06.409 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:06.410+0000 7ff608c00640 1 -- 172.21.15.6:0/2684015214 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5f8030d00 con 0x7ff60c155440 2024-03-04T09:25:06.409 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:06.411+0000 7ff608c00640 1 -- 172.21.15.6:0/2684015214 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5f8035040 con 0x7ff60c155440 2024-03-04T09:25:06.409 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:06.411+0000 7ff612400640 1 -- 172.21.15.6:0/2684015214 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff60c1f74d0 con 0x7ff60c155440 2024-03-04T09:25:06.409 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:06.411+0000 7ff612400640 1 -- 172.21.15.6:0/2684015214 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff60c1f7a90 con 0x7ff60c155440 2024-03-04T09:25:06.409 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:06.412+0000 7ff612400640 1 -- 172.21.15.6:0/2684015214 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff60c0753a0 con 0x7ff60c155440 2024-03-04T09:25:06.409 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:06.412+0000 7ff608c00640 1 -- 172.21.15.6:0/2684015214 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 4) v1 ==== 89724+0+0 (secure 0 0 0) 0x7ff5f804b060 con 0x7ff60c155440 2024-03-04T09:25:06.410 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:06.413+0000 7ff608c00640 1 --2- 172.21.15.6:0/2684015214 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff5ec06b440 0x7ff5ec06d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:06.410 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:06.413+0000 7ff608c00640 1 -- 172.21.15.6:0/2684015214 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(12..12 src has 1..12) v4 ==== 2231+0+0 (secure 0 0 0) 0x7ff5f80ab190 con 0x7ff60c155440 2024-03-04T09:25:06.410 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:06.414+0000 7ff611000640 1 --2- 172.21.15.6:0/2684015214 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff5ec06b440 0x7ff5ec06d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:06.411 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:06.415+0000 7ff611000640 1 --2- 172.21.15.6:0/2684015214 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff5ec06b440 0x7ff5ec06d900 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7ff600003d90 tx=0x7ff600009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:06.415 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:06.418+0000 7ff608c00640 1 -- 172.21.15.6:0/2684015214 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff5f807ad60 con 0x7ff60c155440 2024-03-04T09:25:06.605 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:06.608+0000 7ff612400640 1 -- 172.21.15.6:0/2684015214 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff60c0cf4e0 con 0x7ff60c155440 2024-03-04T09:25:06.606 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:06.610+0000 7ff608c00640 1 -- 172.21.15.6:0/2684015214 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) v1 ==== 74+0+7058 (secure 0 0 0) 0x7ff5f807a660 con 0x7ff60c155440 2024-03-04T09:25:06.607 INFO:teuthology.misc.health.smithi006.stdout: 2024-03-04T09:25:06.607 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":12,"fsid":"edce0578-5ae4-4402-b45a-552bdef84d7c","created":"2024-03-04T09:24:54.523780+0000","modified":"2024-03-04T09:25:03.875166+0000","last_up_change":"0.000000","last_in_change":"2024-03-04T09:24:57.982077+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7a96574f-81f1-4bc6-8463-757520b84a44","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"7d79b9c1-6b87-476a-bc36-7db5ee64cd90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"492331b9-4ee9-40b6-bacd-85d44cd6c45a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"21bb2efe-9ba7-4bdc-bc31-180cd1102e21","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6e2c4b71-19d9-4737-83d9-558705d8644a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"fb69c525-f1be-4b7d-ab32-6c29b0d6fddb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c32e7fed-370d-4401-8ce0-6a4335b0d168","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"75a79b44-fa5f-426d-a7d3-71f21f78835c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T09:25:06.609 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:06.612+0000 7ff612400640 1 -- 172.21.15.6:0/2684015214 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff5ec06b440 msgr2=0x7ff5ec06d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:06.609 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:06.612+0000 7ff612400640 1 --2- 172.21.15.6:0/2684015214 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff5ec06b440 0x7ff5ec06d900 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7ff600003d90 tx=0x7ff600009040 comp rx=0 tx=0).stop 2024-03-04T09:25:06.609 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:06.612+0000 7ff612400640 1 -- 172.21.15.6:0/2684015214 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff60c155440 msgr2=0x7ff60c148220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:06.609 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:06.612+0000 7ff612400640 1 --2- 172.21.15.6:0/2684015214 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff60c155440 0x7ff60c148220 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7ff5f802b950 tx=0x7ff5f80046c0 comp rx=0 tx=0).stop 2024-03-04T09:25:06.609 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:06.612+0000 7ff612400640 1 -- 172.21.15.6:0/2684015214 shutdown_connections 2024-03-04T09:25:06.609 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:06.612+0000 7ff612400640 1 --2- 172.21.15.6:0/2684015214 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff5ec06b440 0x7ff5ec06d900 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:06.609 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:06.612+0000 7ff612400640 1 --2- 172.21.15.6:0/2684015214 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff60c15bef0 0x7ff60c159cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:06.609 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:06.612+0000 7ff612400640 1 --2- 172.21.15.6:0/2684015214 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff60c155440 0x7ff60c148220 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:06.609 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:06.612+0000 7ff612400640 1 --2- 172.21.15.6:0/2684015214 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff60c070c50 0x7ff60c155870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:06.610 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:06.612+0000 7ff612400640 1 -- 172.21.15.6:0/2684015214 >> 172.21.15.6:0/2684015214 conn(0x7ff60c06b390 msgr2=0x7ff60c153440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:06.610 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:06.612+0000 7ff612400640 1 -- 172.21.15.6:0/2684015214 shutdown_connections 2024-03-04T09:25:06.610 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:06.612+0000 7ff612400640 1 -- 172.21.15.6:0/2684015214 wait complete. 2024-03-04T09:25:06.622 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-04T09:25:08.469 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:25:08.477 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:25:13.429 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-04T09:25:13.622 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:13.625+0000 7fbe04400640 1 Processor -- start 2024-03-04T09:25:13.622 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:13.626+0000 7fbe04400640 1 -- start start 2024-03-04T09:25:13.625 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:13.627+0000 7fbe04400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fbdfc0cc450 0x7fbdfc145ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:13.625 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:13.627+0000 7fbe04400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fbdfc162b90 0x7fbdfc163290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:13.625 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:13.627+0000 7fbe04400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbdfc0b1bd0 0x7fbdfc0b2040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:13.625 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:13.627+0000 7fbe04400640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbdfc0b2580 con 0x7fbdfc162b90 2024-03-04T09:25:13.625 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:13.627+0000 7fbe04400640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbdfc071c70 con 0x7fbdfc0cc450 2024-03-04T09:25:13.625 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:13.627+0000 7fbe04400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdfc145800 con 0x7fbdfc0b1bd0 2024-03-04T09:25:13.625 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:13.628+0000 7fbe01000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fbdfc162b90 0x7fbdfc163290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:13.625 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:13.628+0000 7fbe01000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fbdfc162b90 0x7fbdfc163290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41550/0 (socket says 172.21.15.6:41550) 2024-03-04T09:25:13.625 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:13.628+0000 7fbe01000640 1 -- 172.21.15.6:0/1819083452 learned_addr learned my addr 172.21.15.6:0/1819083452 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:25:13.626 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:13.628+0000 7fbe02400640 1 --2- 172.21.15.6:0/1819083452 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbdfc0b1bd0 0x7fbdfc0b2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:13.626 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:13.628+0000 7fbe01a00640 1 --2- 172.21.15.6:0/1819083452 >> v2:172.21.15.146:3300/0 conn(0x7fbdfc0cc450 0x7fbdfc145ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:13.626 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:13.628+0000 7fbe01000640 1 -- 172.21.15.6:0/1819083452 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbdfc0b1bd0 msgr2=0x7fbdfc0b2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:13.626 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:13.628+0000 7fbe01000640 1 --2- 172.21.15.6:0/1819083452 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbdfc0b1bd0 0x7fbdfc0b2040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:13.626 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:13.628+0000 7fbe01000640 1 -- 172.21.15.6:0/1819083452 >> v2:172.21.15.146:3300/0 conn(0x7fbdfc0cc450 msgr2=0x7fbdfc145ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:13.626 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:13.628+0000 7fbe01000640 1 --2- 172.21.15.6:0/1819083452 >> v2:172.21.15.146:3300/0 conn(0x7fbdfc0cc450 0x7fbdfc145ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:13.626 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:13.628+0000 7fbe01000640 1 -- 172.21.15.6:0/1819083452 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdfc071060 con 0x7fbdfc162b90 2024-03-04T09:25:13.626 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:13.628+0000 7fbe01a00640 1 --2- 172.21.15.6:0/1819083452 >> v2:172.21.15.146:3300/0 conn(0x7fbdfc0cc450 0x7fbdfc145ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:25:13.627 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:13.629+0000 7fbe01000640 1 --2- 172.21.15.6:0/1819083452 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbdfc162b90 0x7fbdfc163290 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fbdfc074a20 tx=0x7fbdec02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7acdb74744425761 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:13.627 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:13.630+0000 7fbdfbe00640 1 -- 172.21.15.6:0/1819083452 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdec035020 con 0x7fbdfc162b90 2024-03-04T09:25:13.627 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:13.630+0000 7fbdfbe00640 1 -- 172.21.15.6:0/1819083452 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdec031640 con 0x7fbdfc162b90 2024-03-04T09:25:13.627 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:13.630+0000 7fbdfbe00640 1 -- 172.21.15.6:0/1819083452 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdec02d040 con 0x7fbdfc162b90 2024-03-04T09:25:13.627 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:13.630+0000 7fbe04400640 1 -- 172.21.15.6:0/1819083452 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbdfc162b90 msgr2=0x7fbdfc163290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:13.627 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:13.630+0000 7fbe04400640 1 --2- 172.21.15.6:0/1819083452 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbdfc162b90 0x7fbdfc163290 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fbdfc074a20 tx=0x7fbdec02b8f0 comp rx=0 tx=0).stop 2024-03-04T09:25:13.628 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:13.630+0000 7fbe04400640 1 -- 172.21.15.6:0/1819083452 shutdown_connections 2024-03-04T09:25:13.628 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:13.630+0000 7fbe04400640 1 --2- 172.21.15.6:0/1819083452 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbdfc0b1bd0 0x7fbdfc0b2040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:13.628 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:13.631+0000 7fbe04400640 1 --2- 172.21.15.6:0/1819083452 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbdfc162b90 0x7fbdfc163290 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:13.628 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:13.631+0000 7fbe04400640 1 --2- 172.21.15.6:0/1819083452 >> v2:172.21.15.146:3300/0 conn(0x7fbdfc0cc450 0x7fbdfc145ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:13.628 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:13.631+0000 7fbe04400640 1 -- 172.21.15.6:0/1819083452 >> 172.21.15.6:0/1819083452 conn(0x7fbdfc06af70 msgr2=0x7fbdfc06b350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:13.628 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:13.631+0000 7fbe04400640 1 -- 172.21.15.6:0/1819083452 shutdown_connections 2024-03-04T09:25:13.628 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:13.631+0000 7fbe04400640 1 -- 172.21.15.6:0/1819083452 wait complete. 2024-03-04T09:25:13.629 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:13.632+0000 7fbe04400640 1 Processor -- start 2024-03-04T09:25:13.629 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:13.633+0000 7fbe04400640 1 -- start start 2024-03-04T09:25:13.630 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:13.633+0000 7fbe04400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbdfc0b1bd0 0x7fbdfc15bc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:13.630 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:13.633+0000 7fbe04400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbdfc0cc450 0x7fbdfc136cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:13.630 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:13.633+0000 7fbe04400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbdfc162b90 0x7fbdfc137210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:13.631 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:13.633+0000 7fbe04400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdfc071900 con 0x7fbdfc0cc450 2024-03-04T09:25:13.631 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:13.633+0000 7fbe04400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdfc071620 con 0x7fbdfc0b1bd0 2024-03-04T09:25:13.631 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:13.633+0000 7fbe04400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdfc146010 con 0x7fbdfc162b90 2024-03-04T09:25:13.631 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:13.634+0000 7fbe01000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbdfc0cc450 0x7fbdfc136cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:13.631 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:13.634+0000 7fbe01000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbdfc0cc450 0x7fbdfc136cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41554/0 (socket says 172.21.15.6:41554) 2024-03-04T09:25:13.631 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:13.634+0000 7fbe01000640 1 -- 172.21.15.6:0/723492872 learned_addr learned my addr 172.21.15.6:0/723492872 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:25:13.632 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:13.634+0000 7fbe02400640 1 --2- 172.21.15.6:0/723492872 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbdfc162b90 0x7fbdfc137210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:13.632 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:13.634+0000 7fbe01a00640 1 --2- 172.21.15.6:0/723492872 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbdfc0b1bd0 0x7fbdfc15bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:13.632 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:13.634+0000 7fbe01000640 1 -- 172.21.15.6:0/723492872 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbdfc162b90 msgr2=0x7fbdfc137210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:13.632 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:13.634+0000 7fbe01000640 1 --2- 172.21.15.6:0/723492872 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbdfc162b90 0x7fbdfc137210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:13.632 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:13.634+0000 7fbe01000640 1 -- 172.21.15.6:0/723492872 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbdfc0b1bd0 msgr2=0x7fbdfc15bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:13.632 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:13.634+0000 7fbe01000640 1 --2- 172.21.15.6:0/723492872 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbdfc0b1bd0 0x7fbdfc15bc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:13.632 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:13.634+0000 7fbe01000640 1 -- 172.21.15.6:0/723492872 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdfc135ca0 con 0x7fbdfc0cc450 2024-03-04T09:25:13.632 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:13.634+0000 7fbe02400640 1 --2- 172.21.15.6:0/723492872 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbdfc162b90 0x7fbdfc137210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:25:13.633 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:13.635+0000 7fbe01000640 1 --2- 172.21.15.6:0/723492872 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbdfc0cc450 0x7fbdfc136cd0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fbdec000c00 tx=0x7fbdec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:13.633 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:13.635+0000 7fbdf9600640 1 -- 172.21.15.6:0/723492872 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdec040070 con 0x7fbdfc0cc450 2024-03-04T09:25:13.633 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:13.635+0000 7fbdf9600640 1 -- 172.21.15.6:0/723492872 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdec039bf0 con 0x7fbdfc0cc450 2024-03-04T09:25:13.633 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:13.635+0000 7fbdf9600640 1 -- 172.21.15.6:0/723492872 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdec035040 con 0x7fbdfc0cc450 2024-03-04T09:25:13.633 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:13.635+0000 7fbe04400640 1 -- 172.21.15.6:0/723492872 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdfc135f80 con 0x7fbdfc0cc450 2024-03-04T09:25:13.633 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:13.635+0000 7fbe04400640 1 -- 172.21.15.6:0/723492872 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdfc136540 con 0x7fbdfc0cc450 2024-03-04T09:25:13.633 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:13.637+0000 7fbe04400640 1 -- 172.21.15.6:0/723492872 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdfc04f330 con 0x7fbdfc0cc450 2024-03-04T09:25:13.634 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:13.637+0000 7fbdf9600640 1 -- 172.21.15.6:0/723492872 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbdec04b060 con 0x7fbdfc0cc450 2024-03-04T09:25:13.636 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:13.639+0000 7fbdf9600640 1 --2- 172.21.15.6:0/723492872 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fbde006b4d0 0x7fbde006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:13.636 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:13.639+0000 7fbdf9600640 1 -- 172.21.15.6:0/723492872 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fbdec0ab830 con 0x7fbdfc0cc450 2024-03-04T09:25:13.636 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:13.640+0000 7fbe01a00640 1 --2- 172.21.15.6:0/723492872 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fbde006b4d0 0x7fbde006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:13.637 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:13.641+0000 7fbe01a00640 1 --2- 172.21.15.6:0/723492872 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fbde006b4d0 0x7fbde006d990 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fbdf000a990 tx=0x7fbdf0008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:13.641 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:13.644+0000 7fbdf9600640 1 -- 172.21.15.6:0/723492872 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbdec07a960 con 0x7fbdfc0cc450 2024-03-04T09:25:13.840 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:13.844+0000 7fbe04400640 1 -- 172.21.15.6:0/723492872 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbdfc1664c0 con 0x7fbdfc0cc450 2024-03-04T09:25:13.842 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:13.846+0000 7fbdf9600640 1 -- 172.21.15.6:0/723492872 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+12836 (secure 0 0 0) 0x7fbdec07a260 con 0x7fbdfc0cc450 2024-03-04T09:25:13.843 INFO:teuthology.misc.health.smithi006.stdout: 2024-03-04T09:25:13.843 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":16,"fsid":"edce0578-5ae4-4402-b45a-552bdef84d7c","created":"2024-03-04T09:24:54.523780+0000","modified":"2024-03-04T09:25:10.136308+0000","last_up_change":"2024-03-04T09:25:07.089049+0000","last_in_change":"2024-03-04T09:24:57.982077+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T09:25:07.935198+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a96574f-81f1-4bc6-8463-757520b84a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6825","nonce":1364449401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6827","nonce":1364449401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6831","nonce":1364449401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6829","nonce":1364449401}]},"public_addr":"172.21.15.6:6825/1364449401","cluster_addr":"172.21.15.6:6827/1364449401","heartbeat_back_addr":"172.21.15.6:6831/1364449401","heartbeat_front_addr":"172.21.15.6:6829/1364449401","state":["exists","up"]},{"osd":1,"uuid":"7d79b9c1-6b87-476a-bc36-7db5ee64cd90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6809","nonce":2494103078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6811","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6813","nonce":2494103078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6821","nonce":2494103078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6817","nonce":2494103078}]},"public_addr":"172.21.15.6:6809/2494103078","cluster_addr":"172.21.15.6:6813/2494103078","heartbeat_back_addr":"172.21.15.6:6821/2494103078","heartbeat_front_addr":"172.21.15.6:6817/2494103078","state":["exists","up"]},{"osd":2,"uuid":"492331b9-4ee9-40b6-bacd-85d44cd6c45a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6812","nonce":1500066681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6816","nonce":1500066681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6824","nonce":1500066681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6820","nonce":1500066681}]},"public_addr":"172.21.15.6:6812/1500066681","cluster_addr":"172.21.15.6:6816/1500066681","heartbeat_back_addr":"172.21.15.6:6824/1500066681","heartbeat_front_addr":"172.21.15.6:6820/1500066681","state":["exists","up"]},{"osd":3,"uuid":"21bb2efe-9ba7-4bdc-bc31-180cd1102e21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6801","nonce":1558806123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6803","nonce":1558806123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6807","nonce":1558806123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6805","nonce":1558806123}]},"public_addr":"172.21.15.6:6801/1558806123","cluster_addr":"172.21.15.6:6803/1558806123","heartbeat_back_addr":"172.21.15.6:6807/1558806123","heartbeat_front_addr":"172.21.15.6:6805/1558806123","state":["exists","up"]},{"osd":4,"uuid":"6e2c4b71-19d9-4737-83d9-558705d8644a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6801","nonce":1526135780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6803","nonce":1526135780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6807","nonce":1526135780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6805","nonce":1526135780}]},"public_addr":"172.21.15.146:6801/1526135780","cluster_addr":"172.21.15.146:6803/1526135780","heartbeat_back_addr":"172.21.15.146:6807/1526135780","heartbeat_front_addr":"172.21.15.146:6805/1526135780","state":["exists","up"]},{"osd":5,"uuid":"fb69c525-f1be-4b7d-ab32-6c29b0d6fddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6809","nonce":460262507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6811","nonce":460262507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6815","nonce":460262507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6813","nonce":460262507}]},"public_addr":"172.21.15.146:6809/460262507","cluster_addr":"172.21.15.146:6811/460262507","heartbeat_back_addr":"172.21.15.146:6815/460262507","heartbeat_front_addr":"172.21.15.146:6813/460262507","state":["exists","up"]},{"osd":6,"uuid":"c32e7fed-370d-4401-8ce0-6a4335b0d168","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6823","nonce":1392897493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6827","nonce":1392897493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6831","nonce":1392897493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6829","nonce":1392897493}]},"public_addr":"172.21.15.146:6823/1392897493","cluster_addr":"172.21.15.146:6827/1392897493","heartbeat_back_addr":"172.21.15.146:6831/1392897493","heartbeat_front_addr":"172.21.15.146:6829/1392897493","state":["exists","up"]},{"osd":7,"uuid":"75a79b44-fa5f-426d-a7d3-71f21f78835c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6817","nonce":3268323156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6819","nonce":3268323156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6825","nonce":3268323156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6822","nonce":3268323156}]},"public_addr":"172.21.15.146:6817/3268323156","cluster_addr":"172.21.15.146:6819/3268323156","heartbeat_back_addr":"172.21.15.146:6825/3268323156","heartbeat_front_addr":"172.21.15.146:6822/3268323156","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.310437+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.295850+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.294674+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.526855+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.300023+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.595160+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.346046+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.558741+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T09:25:13.845 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:13.848+0000 7fbe04400640 1 -- 172.21.15.6:0/723492872 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fbde006b4d0 msgr2=0x7fbde006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:13.845 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:13.848+0000 7fbe04400640 1 --2- 172.21.15.6:0/723492872 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fbde006b4d0 0x7fbde006d990 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fbdf000a990 tx=0x7fbdf0008040 comp rx=0 tx=0).stop 2024-03-04T09:25:13.845 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:13.848+0000 7fbe04400640 1 -- 172.21.15.6:0/723492872 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbdfc0cc450 msgr2=0x7fbdfc136cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:13.845 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:13.848+0000 7fbe04400640 1 --2- 172.21.15.6:0/723492872 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbdfc0cc450 0x7fbdfc136cd0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fbdec000c00 tx=0x7fbdec0046c0 comp rx=0 tx=0).stop 2024-03-04T09:25:13.845 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:13.848+0000 7fbe04400640 1 -- 172.21.15.6:0/723492872 shutdown_connections 2024-03-04T09:25:13.846 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:13.848+0000 7fbe04400640 1 --2- 172.21.15.6:0/723492872 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fbde006b4d0 0x7fbde006d990 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:13.846 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:13.848+0000 7fbe04400640 1 --2- 172.21.15.6:0/723492872 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbdfc162b90 0x7fbdfc137210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:13.846 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:13.848+0000 7fbe04400640 1 --2- 172.21.15.6:0/723492872 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbdfc0cc450 0x7fbdfc136cd0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:13.846 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:13.848+0000 7fbe04400640 1 --2- 172.21.15.6:0/723492872 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbdfc0b1bd0 0x7fbdfc15bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:13.846 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:13.848+0000 7fbe04400640 1 -- 172.21.15.6:0/723492872 >> 172.21.15.6:0/723492872 conn(0x7fbdfc06af70 msgr2=0x7fbdfc0b2ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:13.846 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:13.848+0000 7fbe04400640 1 -- 172.21.15.6:0/723492872 shutdown_connections 2024-03-04T09:25:13.846 INFO:teuthology.misc.health.smithi006.stderr:2024-03-04T09:25:13.849+0000 7fbe04400640 1 -- 172.21.15.6:0/723492872 wait complete. 2024-03-04T09:25:13.860 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-03-04T09:25:13.860 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-03-04T09:25:13.860 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-04T09:25:13.860 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-04T09:25:13.867 INFO:tasks.ceph.mds.a:Started 2024-03-04T09:25:13.868 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-04T09:25:13.868 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-04T09:25:13.874 INFO:tasks.ceph.mds.c:Started 2024-03-04T09:25:13.874 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-04T09:25:13.874 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-04T09:25:13.882 INFO:tasks.ceph.mds.b:Started 2024-03-04T09:25:13.882 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-04T09:25:13.882 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-04T09:25:13.889 INFO:tasks.ceph.mds.d:Started 2024-03-04T09:25:13.890 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2024-03-04T09:25:13.892 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T09:25:14.025 INFO:tasks.ceph.mds.a.smithi006.stdout:starting mds.a at 2024-03-04T09:25:14.041 INFO:tasks.ceph.mds.c.smithi006.stdout:starting mds.c at 2024-03-04T09:25:14.050 INFO:tasks.ceph.mds.b.smithi146.stdout:starting mds.b at 2024-03-04T09:25:14.051 INFO:tasks.ceph.mds.d.smithi146.stdout:starting mds.d at 2024-03-04T09:25:14.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.085+0000 7ff1caa00640 1 Processor -- start 2024-03-04T09:25:14.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.086+0000 7ff1caa00640 1 -- start start 2024-03-04T09:25:14.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.087+0000 7ff1caa00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ff1c40af1a0 0x7ff1c415c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:14.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.087+0000 7ff1caa00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff1c40af580 0x7ff1c40b4c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:14.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.087+0000 7ff1caa00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff1c40cdc20 0x7ff1c40b2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:14.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.087+0000 7ff1caa00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1c415cc40 con 0x7ff1c40af580 2024-03-04T09:25:14.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.087+0000 7ff1caa00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1c415cf20 con 0x7ff1c40af1a0 2024-03-04T09:25:14.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.087+0000 7ff1caa00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1c415d200 con 0x7ff1c40cdc20 2024-03-04T09:25:14.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.087+0000 7ff1c3400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff1c40af580 0x7ff1c40b4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:14.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.087+0000 7ff1c3400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff1c40af580 0x7ff1c40b4c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41582/0 (socket says 172.21.15.6:41582) 2024-03-04T09:25:14.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.087+0000 7ff1c3400640 1 -- 172.21.15.6:0/1331869820 learned_addr learned my addr 172.21.15.6:0/1331869820 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:25:14.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.087+0000 7ff1c8a00640 1 --2- 172.21.15.6:0/1331869820 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff1c40cdc20 0x7ff1c40b2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:14.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.087+0000 7ff1c3e00640 1 --2- 172.21.15.6:0/1331869820 >> v2:172.21.15.146:3300/0 conn(0x7ff1c40af1a0 0x7ff1c415c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:14.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.088+0000 7ff1c3400640 1 -- 172.21.15.6:0/1331869820 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff1c40cdc20 msgr2=0x7ff1c40b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:14.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.088+0000 7ff1c3400640 1 --2- 172.21.15.6:0/1331869820 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff1c40cdc20 0x7ff1c40b2dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:14.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.088+0000 7ff1c3400640 1 -- 172.21.15.6:0/1331869820 >> v2:172.21.15.146:3300/0 conn(0x7ff1c40af1a0 msgr2=0x7ff1c415c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:14.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.088+0000 7ff1c3400640 1 --2- 172.21.15.6:0/1331869820 >> v2:172.21.15.146:3300/0 conn(0x7ff1c40af1a0 0x7ff1c415c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:14.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.088+0000 7ff1c3400640 1 -- 172.21.15.6:0/1331869820 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1c415d4e0 con 0x7ff1c40af580 2024-03-04T09:25:14.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.088+0000 7ff1c8a00640 1 --2- 172.21.15.6:0/1331869820 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff1c40cdc20 0x7ff1c40b2dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T09:25:14.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.089+0000 7ff1c3400640 1 --2- 172.21.15.6:0/1331869820 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff1c40af580 0x7ff1c40b4c20 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7ff1c4074960 tx=0x7ff1ac02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca104de740971987 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:14.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.089+0000 7ff1c2a00640 1 -- 172.21.15.6:0/1331869820 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1ac02e070 con 0x7ff1c40af580 2024-03-04T09:25:14.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.089+0000 7ff1c2a00640 1 -- 172.21.15.6:0/1331869820 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1ac037070 con 0x7ff1c40af580 2024-03-04T09:25:14.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.089+0000 7ff1c2a00640 1 -- 172.21.15.6:0/1331869820 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1ac031a00 con 0x7ff1c40af580 2024-03-04T09:25:14.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.090+0000 7ff1caa00640 1 -- 172.21.15.6:0/1331869820 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff1c40af580 msgr2=0x7ff1c40b4c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:14.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.090+0000 7ff1caa00640 1 --2- 172.21.15.6:0/1331869820 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff1c40af580 0x7ff1c40b4c20 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7ff1c4074960 tx=0x7ff1ac02b880 comp rx=0 tx=0).stop 2024-03-04T09:25:14.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.090+0000 7ff1caa00640 1 -- 172.21.15.6:0/1331869820 shutdown_connections 2024-03-04T09:25:14.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.090+0000 7ff1caa00640 1 --2- 172.21.15.6:0/1331869820 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff1c40cdc20 0x7ff1c40b2dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:14.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.090+0000 7ff1caa00640 1 --2- 172.21.15.6:0/1331869820 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff1c40af580 0x7ff1c40b4c20 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:14.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.090+0000 7ff1caa00640 1 --2- 172.21.15.6:0/1331869820 >> v2:172.21.15.146:3300/0 conn(0x7ff1c40af1a0 0x7ff1c415c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:14.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.090+0000 7ff1caa00640 1 -- 172.21.15.6:0/1331869820 >> 172.21.15.6:0/1331869820 conn(0x7ff1c406afb0 msgr2=0x7ff1c406b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:14.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.090+0000 7ff1caa00640 1 -- 172.21.15.6:0/1331869820 shutdown_connections 2024-03-04T09:25:14.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.090+0000 7ff1caa00640 1 -- 172.21.15.6:0/1331869820 wait complete. 2024-03-04T09:25:14.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.092+0000 7ff1caa00640 1 Processor -- start 2024-03-04T09:25:14.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.092+0000 7ff1caa00640 1 -- start start 2024-03-04T09:25:14.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.093+0000 7ff1caa00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff1c40af1a0 0x7ff1c414a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:14.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.093+0000 7ff1caa00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff1c40af580 0x7ff1c415b630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:14.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.093+0000 7ff1caa00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff1c40cdc20 0x7ff1c40b0a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:14.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.093+0000 7ff1caa00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1c40b3490 con 0x7ff1c40cdc20 2024-03-04T09:25:14.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.093+0000 7ff1caa00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1c40b3310 con 0x7ff1c40af580 2024-03-04T09:25:14.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.093+0000 7ff1caa00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1c40b3610 con 0x7ff1c40af1a0 2024-03-04T09:25:14.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.093+0000 7ff1c8a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff1c40cdc20 0x7ff1c40b0a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:14.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.093+0000 7ff1c8a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff1c40cdc20 0x7ff1c40b0a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41586/0 (socket says 172.21.15.6:41586) 2024-03-04T09:25:14.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.093+0000 7ff1c8a00640 1 -- 172.21.15.6:0/276727068 learned_addr learned my addr 172.21.15.6:0/276727068 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:25:14.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.093+0000 7ff1c3e00640 1 --2- 172.21.15.6:0/276727068 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff1c40af1a0 0x7ff1c414a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:14.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.093+0000 7ff1c3400640 1 --2- 172.21.15.6:0/276727068 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff1c40af580 0x7ff1c415b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:14.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.094+0000 7ff1c8a00640 1 -- 172.21.15.6:0/276727068 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff1c40af1a0 msgr2=0x7ff1c414a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:14.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.094+0000 7ff1c8a00640 1 --2- 172.21.15.6:0/276727068 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff1c40af1a0 0x7ff1c414a250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:14.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.094+0000 7ff1c8a00640 1 -- 172.21.15.6:0/276727068 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff1c40af580 msgr2=0x7ff1c415b630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:14.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.094+0000 7ff1c8a00640 1 --2- 172.21.15.6:0/276727068 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff1c40af580 0x7ff1c415b630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:14.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.094+0000 7ff1c8a00640 1 -- 172.21.15.6:0/276727068 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1c4156680 con 0x7ff1c40cdc20 2024-03-04T09:25:14.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.094+0000 7ff1c8a00640 1 --2- 172.21.15.6:0/276727068 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff1c40cdc20 0x7ff1c40b0a20 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ff1bc00e880 tx=0x7ff1bc00ed40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:14.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.094+0000 7ff1bbe00640 1 -- 172.21.15.6:0/276727068 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1bc017070 con 0x7ff1c40cdc20 2024-03-04T09:25:14.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.094+0000 7ff1bbe00640 1 -- 172.21.15.6:0/276727068 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1bc004020 con 0x7ff1c40cdc20 2024-03-04T09:25:14.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.094+0000 7ff1bbe00640 1 -- 172.21.15.6:0/276727068 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1bc0043f0 con 0x7ff1c40cdc20 2024-03-04T09:25:14.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.094+0000 7ff1caa00640 1 -- 172.21.15.6:0/276727068 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1c4156960 con 0x7ff1c40cdc20 2024-03-04T09:25:14.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.094+0000 7ff1caa00640 1 -- 172.21.15.6:0/276727068 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1c4156f20 con 0x7ff1c40cdc20 2024-03-04T09:25:14.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.096+0000 7ff1caa00640 1 -- 172.21.15.6:0/276727068 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1c4069510 con 0x7ff1c40cdc20 2024-03-04T09:25:14.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.097+0000 7ff1bbe00640 1 -- 172.21.15.6:0/276727068 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff1bc00c940 con 0x7ff1c40cdc20 2024-03-04T09:25:14.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.098+0000 7ff1bbe00640 1 --2- 172.21.15.6:0/276727068 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff1a006b4d0 0x7ff1a006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:14.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.098+0000 7ff1bbe00640 1 -- 172.21.15.6:0/276727068 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7ff1bc089f60 con 0x7ff1c40cdc20 2024-03-04T09:25:14.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.099+0000 7ff1c3e00640 1 --2- 172.21.15.6:0/276727068 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff1a006b4d0 0x7ff1a006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:14.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.099+0000 7ff1c3e00640 1 --2- 172.21.15.6:0/276727068 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff1a006b4d0 0x7ff1a006d990 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff1b4011360 tx=0x7ff1b400d040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:14.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.103+0000 7ff1bbe00640 1 -- 172.21.15.6:0/276727068 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff1bc058ae0 con 0x7ff1c40cdc20 2024-03-04T09:25:14.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.301+0000 7ff1caa00640 1 -- 172.21.15.6:0/276727068 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff1c415c690 con 0x7ff1c40cdc20 2024-03-04T09:25:14.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.303+0000 7ff1bbe00640 1 -- 172.21.15.6:0/276727068 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+12836 (secure 0 0 0) 0x7ff1bc05cb80 con 0x7ff1c40cdc20 2024-03-04T09:25:14.300 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:25:14.300 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":16,"fsid":"edce0578-5ae4-4402-b45a-552bdef84d7c","created":"2024-03-04T09:24:54.523780+0000","modified":"2024-03-04T09:25:10.136308+0000","last_up_change":"2024-03-04T09:25:07.089049+0000","last_in_change":"2024-03-04T09:24:57.982077+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T09:25:07.935198+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a96574f-81f1-4bc6-8463-757520b84a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6825","nonce":1364449401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6827","nonce":1364449401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6831","nonce":1364449401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6829","nonce":1364449401}]},"public_addr":"172.21.15.6:6825/1364449401","cluster_addr":"172.21.15.6:6827/1364449401","heartbeat_back_addr":"172.21.15.6:6831/1364449401","heartbeat_front_addr":"172.21.15.6:6829/1364449401","state":["exists","up"]},{"osd":1,"uuid":"7d79b9c1-6b87-476a-bc36-7db5ee64cd90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6809","nonce":2494103078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6811","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6813","nonce":2494103078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6821","nonce":2494103078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6817","nonce":2494103078}]},"public_addr":"172.21.15.6:6809/2494103078","cluster_addr":"172.21.15.6:6813/2494103078","heartbeat_back_addr":"172.21.15.6:6821/2494103078","heartbeat_front_addr":"172.21.15.6:6817/2494103078","state":["exists","up"]},{"osd":2,"uuid":"492331b9-4ee9-40b6-bacd-85d44cd6c45a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6812","nonce":1500066681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6816","nonce":1500066681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6824","nonce":1500066681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6820","nonce":1500066681}]},"public_addr":"172.21.15.6:6812/1500066681","cluster_addr":"172.21.15.6:6816/1500066681","heartbeat_back_addr":"172.21.15.6:6824/1500066681","heartbeat_front_addr":"172.21.15.6:6820/1500066681","state":["exists","up"]},{"osd":3,"uuid":"21bb2efe-9ba7-4bdc-bc31-180cd1102e21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6801","nonce":1558806123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6803","nonce":1558806123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6807","nonce":1558806123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6805","nonce":1558806123}]},"public_addr":"172.21.15.6:6801/1558806123","cluster_addr":"172.21.15.6:6803/1558806123","heartbeat_back_addr":"172.21.15.6:6807/1558806123","heartbeat_front_addr":"172.21.15.6:6805/1558806123","state":["exists","up"]},{"osd":4,"uuid":"6e2c4b71-19d9-4737-83d9-558705d8644a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6801","nonce":1526135780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6803","nonce":1526135780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6807","nonce":1526135780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6805","nonce":1526135780}]},"public_addr":"172.21.15.146:6801/1526135780","cluster_addr":"172.21.15.146:6803/1526135780","heartbeat_back_addr":"172.21.15.146:6807/1526135780","heartbeat_front_addr":"172.21.15.146:6805/1526135780","state":["exists","up"]},{"osd":5,"uuid":"fb69c525-f1be-4b7d-ab32-6c29b0d6fddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6809","nonce":460262507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6811","nonce":460262507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6815","nonce":460262507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6813","nonce":460262507}]},"public_addr":"172.21.15.146:6809/460262507","cluster_addr":"172.21.15.146:6811/460262507","heartbeat_back_addr":"172.21.15.146:6815/460262507","heartbeat_front_addr":"172.21.15.146:6813/460262507","state":["exists","up"]},{"osd":6,"uuid":"c32e7fed-370d-4401-8ce0-6a4335b0d168","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6823","nonce":1392897493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6827","nonce":1392897493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6831","nonce":1392897493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6829","nonce":1392897493}]},"public_addr":"172.21.15.146:6823/1392897493","cluster_addr":"172.21.15.146:6827/1392897493","heartbeat_back_addr":"172.21.15.146:6831/1392897493","heartbeat_front_addr":"172.21.15.146:6829/1392897493","state":["exists","up"]},{"osd":7,"uuid":"75a79b44-fa5f-426d-a7d3-71f21f78835c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6817","nonce":3268323156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6819","nonce":3268323156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6825","nonce":3268323156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6822","nonce":3268323156}]},"public_addr":"172.21.15.146:6817/3268323156","cluster_addr":"172.21.15.146:6819/3268323156","heartbeat_back_addr":"172.21.15.146:6825/3268323156","heartbeat_front_addr":"172.21.15.146:6822/3268323156","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.310437+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.295850+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.294674+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.526855+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.300023+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.595160+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.346046+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.558741+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T09:25:14.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.305+0000 7ff1caa00640 1 -- 172.21.15.6:0/276727068 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff1a006b4d0 msgr2=0x7ff1a006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:14.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.305+0000 7ff1caa00640 1 --2- 172.21.15.6:0/276727068 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff1a006b4d0 0x7ff1a006d990 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff1b4011360 tx=0x7ff1b400d040 comp rx=0 tx=0).stop 2024-03-04T09:25:14.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.305+0000 7ff1caa00640 1 -- 172.21.15.6:0/276727068 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff1c40cdc20 msgr2=0x7ff1c40b0a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:14.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.305+0000 7ff1caa00640 1 --2- 172.21.15.6:0/276727068 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff1c40cdc20 0x7ff1c40b0a20 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ff1bc00e880 tx=0x7ff1bc00ed40 comp rx=0 tx=0).stop 2024-03-04T09:25:14.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.305+0000 7ff1caa00640 1 -- 172.21.15.6:0/276727068 shutdown_connections 2024-03-04T09:25:14.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.305+0000 7ff1caa00640 1 --2- 172.21.15.6:0/276727068 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff1c40cdc20 0x7ff1c40b0a20 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:14.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.305+0000 7ff1caa00640 1 --2- 172.21.15.6:0/276727068 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff1c40af580 0x7ff1c415b630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:14.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.305+0000 7ff1caa00640 1 --2- 172.21.15.6:0/276727068 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff1a006b4d0 0x7ff1a006d990 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:14.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.305+0000 7ff1caa00640 1 --2- 172.21.15.6:0/276727068 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff1c40af1a0 0x7ff1c414a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:14.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.305+0000 7ff1caa00640 1 -- 172.21.15.6:0/276727068 >> 172.21.15.6:0/276727068 conn(0x7ff1c406afb0 msgr2=0x7ff1c406b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:14.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.305+0000 7ff1caa00640 1 -- 172.21.15.6:0/276727068 shutdown_connections 2024-03-04T09:25:14.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.305+0000 7ff1caa00640 1 -- 172.21.15.6:0/276727068 wait complete. 2024-03-04T09:25:14.318 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T09:25:07.935198+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-04T09:25:14.319 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-04T09:25:14.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.540+0000 7f6162600640 1 Processor -- start 2024-03-04T09:25:14.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.540+0000 7f6162600640 1 -- start start 2024-03-04T09:25:14.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.541+0000 7f6162600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f615c0729d0 0x7f615c0b4000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:14.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.541+0000 7f6162600640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f615c071930 con 0x7f615c0b1b40 2024-03-04T09:25:14.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.541+0000 7f6162600640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f615c071c10 con 0x7f615c0b3c20 2024-03-04T09:25:14.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.541+0000 7f6162600640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f615c15bfb0 con 0x7f615c0729d0 2024-03-04T09:25:14.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.542+0000 7f615be00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f615c0729d0 0x7f615c0b4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:14.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.542+0000 7f6161c00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f615c0b1b40 0x7f615c15c220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:51516/0 (socket says 172.21.15.6:51516) 2024-03-04T09:25:14.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.542+0000 7f6161c00640 1 -- 172.21.15.6:0/2004973785 learned_addr learned my addr 172.21.15.6:0/2004973785 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:25:14.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.542+0000 7f615be00640 1 -- 172.21.15.6:0/2004973785 >> v1:172.21.15.146:6789/0 conn(0x7f615c0b3c20 legacy=0x7f615c15b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:14.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.542+0000 7f615be00640 1 -- 172.21.15.6:0/2004973785 >> v1:172.21.15.6:6789/0 conn(0x7f615c0b1b40 legacy=0x7f615c15c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:14.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.542+0000 7f615be00640 1 -- 172.21.15.6:0/2004973785 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f615c071ef0 con 0x7f615c0729d0 2024-03-04T09:25:14.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.542+0000 7f615b400640 1 -- 172.21.15.6:0/2004973785 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1013778299 0 0) 0x7f615c071930 con 0x7f615c0b1b40 2024-03-04T09:25:14.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.543+0000 7f615be00640 1 --2- 172.21.15.6:0/2004973785 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f615c0729d0 0x7f615c0b4000 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f615c15c130 tx=0x7f614802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5338191fc313ff36 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:14.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.544+0000 7f615b400640 1 -- 172.21.15.6:0/2004973785 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6148034070 con 0x7f615c0729d0 2024-03-04T09:25:14.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.544+0000 7f615b400640 1 -- 172.21.15.6:0/2004973785 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6148030610 con 0x7f615c0729d0 2024-03-04T09:25:14.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.544+0000 7f615b400640 1 -- 172.21.15.6:0/2004973785 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61480309e0 con 0x7f615c0729d0 2024-03-04T09:25:14.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.544+0000 7f6162600640 1 -- 172.21.15.6:0/2004973785 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f615c0729d0 msgr2=0x7f615c0b4000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:14.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.544+0000 7f6162600640 1 --2- 172.21.15.6:0/2004973785 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f615c0729d0 0x7f615c0b4000 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f615c15c130 tx=0x7f614802b960 comp rx=0 tx=0).stop 2024-03-04T09:25:14.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.544+0000 7f6162600640 1 -- 172.21.15.6:0/2004973785 shutdown_connections 2024-03-04T09:25:14.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.544+0000 7f6162600640 1 --2- 172.21.15.6:0/2004973785 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f615c0729d0 0x7f615c0b4000 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:14.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.544+0000 7f6162600640 1 -- 172.21.15.6:0/2004973785 >> 172.21.15.6:0/2004973785 conn(0x7f615c06b3a0 msgr2=0x7f615c06b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:14.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.544+0000 7f6162600640 1 -- 172.21.15.6:0/2004973785 shutdown_connections 2024-03-04T09:25:14.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.544+0000 7f6162600640 1 -- 172.21.15.6:0/2004973785 wait complete. 2024-03-04T09:25:14.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.546+0000 7f6162600640 1 Processor -- start 2024-03-04T09:25:14.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.546+0000 7f6162600640 1 -- start start 2024-03-04T09:25:14.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.547+0000 7f6162600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f615c0729d0 0x7f615c1f29b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:14.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.547+0000 7f6162600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f615c0b1b40 0x7f615c1f5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:14.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.547+0000 7f6162600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f615c0b3c20 0x7f615c155720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:14.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.547+0000 7f6162600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f615c0b1fa0 con 0x7f615c0b1b40 2024-03-04T09:25:14.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.547+0000 7f6162600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f615c0724b0 con 0x7f615c0729d0 2024-03-04T09:25:14.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.547+0000 7f6162600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f615c0b2120 con 0x7f615c0b3c20 2024-03-04T09:25:14.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.547+0000 7f615be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f615c0b1b40 0x7f615c1f5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:14.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.547+0000 7f615be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f615c0b1b40 0x7f615c1f5470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41602/0 (socket says 172.21.15.6:41602) 2024-03-04T09:25:14.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.547+0000 7f615be00640 1 -- 172.21.15.6:0/409306419 learned_addr learned my addr 172.21.15.6:0/409306419 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:25:14.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.548+0000 7f6161c00640 1 --2- 172.21.15.6:0/409306419 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f615c0b3c20 0x7f615c155720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:14.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.548+0000 7f6161200640 1 --2- 172.21.15.6:0/409306419 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f615c0729d0 0x7f615c1f29b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:14.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.548+0000 7f615be00640 1 -- 172.21.15.6:0/409306419 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f615c0b3c20 msgr2=0x7f615c155720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:14.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.548+0000 7f615be00640 1 --2- 172.21.15.6:0/409306419 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f615c0b3c20 0x7f615c155720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:14.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.548+0000 7f615be00640 1 -- 172.21.15.6:0/409306419 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f615c0729d0 msgr2=0x7f615c1f29b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:14.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.548+0000 7f615be00640 1 --2- 172.21.15.6:0/409306419 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f615c0729d0 0x7f615c1f29b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:14.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.548+0000 7f615be00640 1 -- 172.21.15.6:0/409306419 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f615c1f5c10 con 0x7f615c0b1b40 2024-03-04T09:25:14.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.548+0000 7f6161200640 1 --2- 172.21.15.6:0/409306419 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f615c0729d0 0x7f615c1f29b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:25:14.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.548+0000 7f615be00640 1 --2- 172.21.15.6:0/409306419 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f615c0b1b40 0x7f615c1f5470 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f6148009890 tx=0x7f61480046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:14.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.549+0000 7f6158c00640 1 -- 172.21.15.6:0/409306419 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6148034040 con 0x7f615c0b1b40 2024-03-04T09:25:14.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.549+0000 7f6158c00640 1 -- 172.21.15.6:0/409306419 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6148039a10 con 0x7f615c0b1b40 2024-03-04T09:25:14.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.549+0000 7f6158c00640 1 -- 172.21.15.6:0/409306419 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6148039e00 con 0x7f615c0b1b40 2024-03-04T09:25:14.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.549+0000 7f6162600640 1 -- 172.21.15.6:0/409306419 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f615c1f5ef0 con 0x7f615c0b1b40 2024-03-04T09:25:14.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.549+0000 7f6162600640 1 -- 172.21.15.6:0/409306419 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f615c1e4c40 con 0x7f615c0b1b40 2024-03-04T09:25:14.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.550+0000 7f6162600640 1 -- 172.21.15.6:0/409306419 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f615c15b0a0 con 0x7f615c0b1b40 2024-03-04T09:25:14.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.551+0000 7f6158c00640 1 -- 172.21.15.6:0/409306419 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6148037070 con 0x7f615c0b1b40 2024-03-04T09:25:14.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.552+0000 7f6158c00640 1 --2- 172.21.15.6:0/409306419 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f613c06b4d0 0x7f613c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:14.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.552+0000 7f6158c00640 1 -- 172.21.15.6:0/409306419 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f61480ac170 con 0x7f615c0b1b40 2024-03-04T09:25:14.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.552+0000 7f6161200640 1 --2- 172.21.15.6:0/409306419 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f613c06b4d0 0x7f613c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:14.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.554+0000 7f6161200640 1 --2- 172.21.15.6:0/409306419 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f613c06b4d0 0x7f613c06d990 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f614c0012e0 tx=0x7f614c009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:14.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.557+0000 7f6158c00640 1 -- 172.21.15.6:0/409306419 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f615c15b0a0 con 0x7f615c0b1b40 2024-03-04T09:25:14.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.756+0000 7f6162600640 1 -- 172.21.15.6:0/409306419 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f615c04f330 con 0x7f615c0b1b40 2024-03-04T09:25:14.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.757+0000 7f6158c00640 1 -- 172.21.15.6:0/409306419 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v16) v1 ==== 93+0+10 (secure 0 0 0) 0x7f614807b210 con 0x7f615c0b1b40 2024-03-04T09:25:14.754 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-04T09:25:14.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.760+0000 7f6162600640 1 -- 172.21.15.6:0/409306419 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f613c06b4d0 msgr2=0x7f613c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:14.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.760+0000 7f6162600640 1 --2- 172.21.15.6:0/409306419 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f613c06b4d0 0x7f613c06d990 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f614c0012e0 tx=0x7f614c009040 comp rx=0 tx=0).stop 2024-03-04T09:25:14.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.760+0000 7f6162600640 1 -- 172.21.15.6:0/409306419 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f615c0b1b40 msgr2=0x7f615c1f5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:14.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.760+0000 7f6162600640 1 --2- 172.21.15.6:0/409306419 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f615c0b1b40 0x7f615c1f5470 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f6148009890 tx=0x7f61480046c0 comp rx=0 tx=0).stop 2024-03-04T09:25:14.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.760+0000 7f6162600640 1 -- 172.21.15.6:0/409306419 shutdown_connections 2024-03-04T09:25:14.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.760+0000 7f6162600640 1 --2- 172.21.15.6:0/409306419 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f613c06b4d0 0x7f613c06d990 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:14.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.760+0000 7f6162600640 1 --2- 172.21.15.6:0/409306419 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f615c0b3c20 0x7f615c155720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:14.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.760+0000 7f6162600640 1 --2- 172.21.15.6:0/409306419 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f615c0b1b40 0x7f615c1f5470 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:14.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.760+0000 7f6162600640 1 --2- 172.21.15.6:0/409306419 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f615c0729d0 0x7f615c1f29b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:14.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.760+0000 7f6162600640 1 -- 172.21.15.6:0/409306419 >> 172.21.15.6:0/409306419 conn(0x7f615c06b3a0 msgr2=0x7f615c15c200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:14.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.760+0000 7f6162600640 1 -- 172.21.15.6:0/409306419 shutdown_connections 2024-03-04T09:25:14.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.761+0000 7f6162600640 1 -- 172.21.15.6:0/409306419 wait complete. 2024-03-04T09:25:14.775 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:25:14.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.989+0000 7efc5f200640 1 Processor -- start 2024-03-04T09:25:14.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.989+0000 7efc5f200640 1 -- start start 2024-03-04T09:25:14.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.990+0000 7efc5f200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7efc600d4a10 0x7efc600d0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:14.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.990+0000 7efc5f200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efc600d0c80 0x7efc600b44d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:14.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.990+0000 7efc5f200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc600b3710 con 0x7efc6015e0b0 2024-03-04T09:25:14.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.990+0000 7efc5f200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc600b09e0 con 0x7efc600d4a10 2024-03-04T09:25:14.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.990+0000 7efc5f200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc600cfa20 con 0x7efc600d0c80 2024-03-04T09:25:14.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.990+0000 7efc57e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7efc600d4a10 0x7efc600d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:14.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.990+0000 7efc57e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7efc600d4a10 0x7efc600d0740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41608/0 (socket says 172.21.15.6:41608) 2024-03-04T09:25:14.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.990+0000 7efc57e00640 1 -- 172.21.15.6:0/4179026041 learned_addr learned my addr 172.21.15.6:0/4179026041 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:25:14.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.991+0000 7efc5e800640 1 --2- 172.21.15.6:0/4179026041 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efc600d0c80 0x7efc600b44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:14.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.991+0000 7efc57e00640 1 -- 172.21.15.6:0/4179026041 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efc600d0c80 msgr2=0x7efc600b44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:14.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.991+0000 7efc57e00640 1 --2- 172.21.15.6:0/4179026041 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efc600d0c80 0x7efc600b44d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:14.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.991+0000 7efc57e00640 1 -- 172.21.15.6:0/4179026041 >> v1:172.21.15.146:6789/0 conn(0x7efc6015e0b0 legacy=0x7efc600cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:14.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.991+0000 7efc57e00640 1 -- 172.21.15.6:0/4179026041 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc600b39f0 con 0x7efc600d4a10 2024-03-04T09:25:14.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.991+0000 7efc5e800640 1 --2- 172.21.15.6:0/4179026041 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efc600d0c80 0x7efc600b44d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:25:14.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.992+0000 7efc57e00640 1 --2- 172.21.15.6:0/4179026041 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efc600d4a10 0x7efc600d0740 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7efc48009870 tx=0x7efc4802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9e133f9d8b5b64e server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:14.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.992+0000 7efc5d400640 1 -- 172.21.15.6:0/4179026041 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc48035020 con 0x7efc600d4a10 2024-03-04T09:25:14.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.992+0000 7efc5d400640 1 -- 172.21.15.6:0/4179026041 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc48031640 con 0x7efc600d4a10 2024-03-04T09:25:14.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.992+0000 7efc5d400640 1 -- 172.21.15.6:0/4179026041 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc4802d040 con 0x7efc600d4a10 2024-03-04T09:25:14.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.993+0000 7efc5f200640 1 -- 172.21.15.6:0/4179026041 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efc600d4a10 msgr2=0x7efc600d0740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:14.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.993+0000 7efc5f200640 1 --2- 172.21.15.6:0/4179026041 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efc600d4a10 0x7efc600d0740 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7efc48009870 tx=0x7efc4802b8f0 comp rx=0 tx=0).stop 2024-03-04T09:25:14.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.993+0000 7efc5f200640 1 -- 172.21.15.6:0/4179026041 shutdown_connections 2024-03-04T09:25:14.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.993+0000 7efc5f200640 1 --2- 172.21.15.6:0/4179026041 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efc600d0c80 0x7efc600b44d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:14.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.993+0000 7efc5f200640 1 --2- 172.21.15.6:0/4179026041 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efc600d4a10 0x7efc600d0740 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:14.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.993+0000 7efc5f200640 1 -- 172.21.15.6:0/4179026041 >> 172.21.15.6:0/4179026041 conn(0x7efc600d3d30 msgr2=0x7efc600d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:14.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.993+0000 7efc5f200640 1 -- 172.21.15.6:0/4179026041 shutdown_connections 2024-03-04T09:25:14.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.993+0000 7efc5f200640 1 -- 172.21.15.6:0/4179026041 wait complete. 2024-03-04T09:25:14.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.995+0000 7efc5f200640 1 Processor -- start 2024-03-04T09:25:14.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.995+0000 7efc5f200640 1 -- start start 2024-03-04T09:25:14.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.996+0000 7efc5f200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efc600d0c80 0x7efc601f7800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:14.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.996+0000 7efc5f200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efc600d4a10 0x7efc600cb0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:14.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.996+0000 7efc5f200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efc6015e0b0 0x7efc600cd4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:14.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.996+0000 7efc5f200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc6015e490 con 0x7efc600d0c80 2024-03-04T09:25:14.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.996+0000 7efc5f200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc600b3fb0 con 0x7efc6015e0b0 2024-03-04T09:25:14.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.996+0000 7efc5f200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc600cfc60 con 0x7efc600d4a10 2024-03-04T09:25:14.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.996+0000 7efc5de00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efc600d0c80 0x7efc601f7800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:14.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.996+0000 7efc5de00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efc600d0c80 0x7efc601f7800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41624/0 (socket says 172.21.15.6:41624) 2024-03-04T09:25:14.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.996+0000 7efc5de00640 1 -- 172.21.15.6:0/1942233972 learned_addr learned my addr 172.21.15.6:0/1942233972 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:25:14.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.997+0000 7efc57e00640 1 --2- 172.21.15.6:0/1942233972 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efc600d4a10 0x7efc600cb0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:14.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.997+0000 7efc5e800640 1 --2- 172.21.15.6:0/1942233972 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efc6015e0b0 0x7efc600cd4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:14.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.997+0000 7efc5de00640 1 -- 172.21.15.6:0/1942233972 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efc600d4a10 msgr2=0x7efc600cb0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:14.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.997+0000 7efc5de00640 1 --2- 172.21.15.6:0/1942233972 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efc600d4a10 0x7efc600cb0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:14.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.997+0000 7efc5de00640 1 -- 172.21.15.6:0/1942233972 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efc6015e0b0 msgr2=0x7efc600cd4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:14.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.997+0000 7efc5de00640 1 --2- 172.21.15.6:0/1942233972 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efc6015e0b0 0x7efc600cd4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:14.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.997+0000 7efc5de00640 1 -- 172.21.15.6:0/1942233972 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc600cda10 con 0x7efc600d0c80 2024-03-04T09:25:14.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.997+0000 7efc5de00640 1 --2- 172.21.15.6:0/1942233972 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efc600d0c80 0x7efc601f7800 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7efc5000c460 tx=0x7efc5000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:14.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.997+0000 7efc56000640 1 -- 172.21.15.6:0/1942233972 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc50016020 con 0x7efc600d0c80 2024-03-04T09:25:14.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.997+0000 7efc56000640 1 -- 172.21.15.6:0/1942233972 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc50012870 con 0x7efc600d0c80 2024-03-04T09:25:14.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.997+0000 7efc56000640 1 -- 172.21.15.6:0/1942233972 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc50012c60 con 0x7efc600d0c80 2024-03-04T09:25:14.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.998+0000 7efc5f200640 1 -- 172.21.15.6:0/1942233972 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc600cdcb0 con 0x7efc600d0c80 2024-03-04T09:25:14.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.998+0000 7efc5f200640 1 -- 172.21.15.6:0/1942233972 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc601f7000 con 0x7efc600d0c80 2024-03-04T09:25:14.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:14.999+0000 7efc5f200640 1 -- 172.21.15.6:0/1942233972 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc6004f330 con 0x7efc600d0c80 2024-03-04T09:25:14.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.000+0000 7efc56000640 1 -- 172.21.15.6:0/1942233972 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efc50019020 con 0x7efc600d0c80 2024-03-04T09:25:14.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.001+0000 7efc56000640 1 --2- 172.21.15.6:0/1942233972 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7efc3c06b4d0 0x7efc3c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:14.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.001+0000 7efc56000640 1 -- 172.21.15.6:0/1942233972 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7efc50089940 con 0x7efc600d0c80 2024-03-04T09:25:14.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.002+0000 7efc57e00640 1 --2- 172.21.15.6:0/1942233972 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7efc3c06b4d0 0x7efc3c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:14.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.003+0000 7efc57e00640 1 --2- 172.21.15.6:0/1942233972 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7efc3c06b4d0 0x7efc3c06d990 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7efc48036000 tx=0x7efc4803c040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:15.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.006+0000 7efc56000640 1 -- 172.21.15.6:0/1942233972 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7efc50020070 con 0x7efc600d0c80 2024-03-04T09:25:15.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.237+0000 7efc5f200640 1 -- 172.21.15.6:0/1942233972 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efc601f6db0 con 0x7efc600d0c80 2024-03-04T09:25:15.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.238+0000 7efc56000640 1 -- 172.21.15.6:0/1942233972 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+3694 (secure 0 0 0) 0x7efc50058b00 con 0x7efc600d0c80 2024-03-04T09:25:15.235 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:25:15.235 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4248,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/2702058134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2702058134},{"type":"v1","addr":"172.21.15.6:6833","nonce":2702058134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4263,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/3866998852","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":3866998852},{"type":"v1","addr":"172.21.15.6:6835","nonce":3866998852}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4285,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/3523455455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":3523455455},{"type":"v1","addr":"172.21.15.146:6837","nonce":3523455455}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4288,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6836/1946537763","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":1946537763},{"type":"v1","addr":"172.21.15.146:6836","nonce":1946537763}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[]} 2024-03-04T09:25:15.235 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 3 2024-03-04T09:25:15.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.240+0000 7efc5f200640 1 -- 172.21.15.6:0/1942233972 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7efc3c06b4d0 msgr2=0x7efc3c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:15.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.240+0000 7efc5f200640 1 --2- 172.21.15.6:0/1942233972 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7efc3c06b4d0 0x7efc3c06d990 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7efc48036000 tx=0x7efc4803c040 comp rx=0 tx=0).stop 2024-03-04T09:25:15.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.240+0000 7efc5f200640 1 -- 172.21.15.6:0/1942233972 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efc600d0c80 msgr2=0x7efc601f7800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:15.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.240+0000 7efc5f200640 1 --2- 172.21.15.6:0/1942233972 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efc600d0c80 0x7efc601f7800 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7efc5000c460 tx=0x7efc5000c920 comp rx=0 tx=0).stop 2024-03-04T09:25:15.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.241+0000 7efc5f200640 1 -- 172.21.15.6:0/1942233972 shutdown_connections 2024-03-04T09:25:15.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.241+0000 7efc5f200640 1 --2- 172.21.15.6:0/1942233972 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efc6015e0b0 0x7efc600cd4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:15.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.241+0000 7efc5f200640 1 --2- 172.21.15.6:0/1942233972 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7efc3c06b4d0 0x7efc3c06d990 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:15.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.241+0000 7efc5f200640 1 --2- 172.21.15.6:0/1942233972 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efc600d4a10 0x7efc600cb0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:15.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.241+0000 7efc5f200640 1 --2- 172.21.15.6:0/1942233972 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efc600d0c80 0x7efc601f7800 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:15.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.241+0000 7efc5f200640 1 -- 172.21.15.6:0/1942233972 >> 172.21.15.6:0/1942233972 conn(0x7efc600d3d30 msgr2=0x7efc600b0b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:15.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.241+0000 7efc5f200640 1 -- 172.21.15.6:0/1942233972 shutdown_connections 2024-03-04T09:25:15.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.241+0000 7efc5f200640 1 -- 172.21.15.6:0/1942233972 wait complete. 2024-03-04T09:25:15.255 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T09:25:15.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.473+0000 7f47fa000640 1 Processor -- start 2024-03-04T09:25:15.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.473+0000 7f47fa000640 1 -- start start 2024-03-04T09:25:15.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.475+0000 7f47fa000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f47f40cd060 0x7f47f415b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:15.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.475+0000 7f47fa000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47f415ed10 con 0x7f47f415e810 2024-03-04T09:25:15.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.475+0000 7f47fa000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47f415eff0 con 0x7f47f41552b0 2024-03-04T09:25:15.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.475+0000 7f47fa000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47f415f2d0 con 0x7f47f40cd060 2024-03-04T09:25:15.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.475+0000 7f47f9600640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f47f415e810 0x7f47f40b3cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:51528/0 (socket says 172.21.15.6:51528) 2024-03-04T09:25:15.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.475+0000 7f47f9600640 1 -- 172.21.15.6:0/79628686 learned_addr learned my addr 172.21.15.6:0/79628686 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:25:15.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.475+0000 7f47f3e00640 1 --2- 172.21.15.6:0/79628686 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f47f40cd060 0x7f47f415b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:15.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.475+0000 7f47f3400640 1 -- 172.21.15.6:0/79628686 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1933148244 0 0) 0x7f47f415ed10 con 0x7f47f415e810 2024-03-04T09:25:15.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.476+0000 7f47f3400640 1 -- 172.21.15.6:0/79628686 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47d8003520 con 0x7f47f415e810 2024-03-04T09:25:15.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.476+0000 7f47f3400640 1 -- 172.21.15.6:0/79628686 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1864514081 0 0) 0x7f47f415eff0 con 0x7f47f41552b0 2024-03-04T09:25:15.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.476+0000 7f47f3400640 1 -- 172.21.15.6:0/79628686 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47f415ed10 con 0x7f47f41552b0 2024-03-04T09:25:15.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.476+0000 7f47f3400640 1 -- 172.21.15.6:0/79628686 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3453688900 0 0) 0x7f47d8003520 con 0x7f47f415e810 2024-03-04T09:25:15.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.476+0000 7f47f3400640 1 -- 172.21.15.6:0/79628686 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f47f40cd060 msgr2=0x7f47f415b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:15.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.476+0000 7f47f3400640 1 --2- 172.21.15.6:0/79628686 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f47f40cd060 0x7f47f415b1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:15.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.476+0000 7f47f3400640 1 -- 172.21.15.6:0/79628686 >> v1:172.21.15.146:6789/0 conn(0x7f47f41552b0 legacy=0x7f47f40cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:15.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.476+0000 7f47f3400640 1 -- 172.21.15.6:0/79628686 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47f415f5b0 con 0x7f47f415e810 2024-03-04T09:25:15.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.476+0000 7f47f3400640 1 -- 172.21.15.6:0/79628686 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7f47ec0019f0 con 0x7f47f415e810 2024-03-04T09:25:15.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.476+0000 7f47f3400640 1 -- 172.21.15.6:0/79628686 >> v1:172.21.15.6:6789/0 conn(0x7f47f415e810 legacy=0x7f47f40b3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:15.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.476+0000 7f47f3400640 1 --2- 172.21.15.6:0/79628686 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f47d8004bd0 0x7f47d8004fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:15.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.476+0000 7f47f3400640 1 --2- 172.21.15.6:0/79628686 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f47d80037a0 0x7f47d8003c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:15.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.476+0000 7f47f3400640 1 --2- 172.21.15.6:0/79628686 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f47f40cd060 0x7f47d80041c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:15.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.476+0000 7f47f3400640 1 -- 172.21.15.6:0/79628686 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47f415f2d0 con 0x7f47d8004bd0 2024-03-04T09:25:15.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.476+0000 7f47f3400640 1 -- 172.21.15.6:0/79628686 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47d800dd60 con 0x7f47d80037a0 2024-03-04T09:25:15.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.476+0000 7f47f3400640 1 -- 172.21.15.6:0/79628686 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47d800df30 con 0x7f47f40cd060 2024-03-04T09:25:15.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.476+0000 7f47f8c00640 1 --2- 172.21.15.6:0/79628686 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f47d8004bd0 0x7f47d8004fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:15.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.477+0000 7f47f3e00640 1 --2- 172.21.15.6:0/79628686 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f47d80037a0 0x7f47d8003c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:15.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.477+0000 7f47f9600640 1 --2- 172.21.15.6:0/79628686 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f47f40cd060 0x7f47d80041c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:15.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.477+0000 7f47f8c00640 1 -- 172.21.15.6:0/79628686 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f47f40cd060 msgr2=0x7f47d80041c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:15.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.477+0000 7f47f8c00640 1 --2- 172.21.15.6:0/79628686 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f47f40cd060 0x7f47d80041c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:15.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.477+0000 7f47f8c00640 1 -- 172.21.15.6:0/79628686 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f47d80037a0 msgr2=0x7f47d8003c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:15.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.477+0000 7f47f8c00640 1 --2- 172.21.15.6:0/79628686 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f47d80037a0 0x7f47d8003c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:15.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.477+0000 7f47f8c00640 1 -- 172.21.15.6:0/79628686 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47d8003520 con 0x7f47d8004bd0 2024-03-04T09:25:15.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.477+0000 7f47f9600640 1 --2- 172.21.15.6:0/79628686 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f47f40cd060 0x7f47d80041c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:25:15.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.478+0000 7f47f8c00640 1 --2- 172.21.15.6:0/79628686 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f47d8004bd0 0x7f47d8004fd0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f47e4001900 tx=0x7f47d8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dce0b40ad5191f3c server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:15.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.478+0000 7f47f3400640 1 -- 172.21.15.6:0/79628686 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47e4037070 con 0x7f47d8004bd0 2024-03-04T09:25:15.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.478+0000 7f47f3400640 1 -- 172.21.15.6:0/79628686 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47e40052d0 con 0x7f47d8004bd0 2024-03-04T09:25:15.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.478+0000 7f47f3400640 1 -- 172.21.15.6:0/79628686 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47e40014f0 con 0x7f47d8004bd0 2024-03-04T09:25:15.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.479+0000 7f47fa000640 1 -- 172.21.15.6:0/79628686 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f47d8004bd0 msgr2=0x7f47d8004fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:15.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.479+0000 7f47fa000640 1 --2- 172.21.15.6:0/79628686 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f47d8004bd0 0x7f47d8004fd0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f47e4001900 tx=0x7f47d8005830 comp rx=0 tx=0).stop 2024-03-04T09:25:15.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.479+0000 7f47f9600640 1 -- 172.21.15.6:0/79628686 reap_dead start 2024-03-04T09:25:15.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.479+0000 7f47fa000640 1 -- 172.21.15.6:0/79628686 shutdown_connections 2024-03-04T09:25:15.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.479+0000 7f47fa000640 1 -- 172.21.15.6:0/79628686 >> 172.21.15.6:0/79628686 conn(0x7f47f406b3d0 msgr2=0x7f47f406b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:15.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.479+0000 7f47fa000640 1 -- 172.21.15.6:0/79628686 shutdown_connections 2024-03-04T09:25:15.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.479+0000 7f47fa000640 1 -- 172.21.15.6:0/79628686 wait complete. 2024-03-04T09:25:15.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.481+0000 7f47fa000640 1 Processor -- start 2024-03-04T09:25:15.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.481+0000 7f47fa000640 1 -- start start 2024-03-04T09:25:15.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.482+0000 7f47fa000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f47d8004bd0 0x7f47f4157b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:15.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.482+0000 7f47fa000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f47f415a590 0x7f47f41558a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:15.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.482+0000 7f47fa000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f47f415aa10 0x7f47f4155de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:15.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.482+0000 7f47fa000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47f406dd30 con 0x7f47f415a590 2024-03-04T09:25:15.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.482+0000 7f47fa000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47f40b3760 con 0x7f47f415aa10 2024-03-04T09:25:15.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.482+0000 7f47fa000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47f406deb0 con 0x7f47d8004bd0 2024-03-04T09:25:15.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.482+0000 7f47f3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f47f415a590 0x7f47f41558a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:15.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.482+0000 7f47f3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f47f415a590 0x7f47f41558a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41644/0 (socket says 172.21.15.6:41644) 2024-03-04T09:25:15.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.482+0000 7f47f3e00640 1 -- 172.21.15.6:0/2449310834 learned_addr learned my addr 172.21.15.6:0/2449310834 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:25:15.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.482+0000 7f47f9600640 1 --2- 172.21.15.6:0/2449310834 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f47f415aa10 0x7f47f4155de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:15.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.482+0000 7f47f8c00640 1 --2- 172.21.15.6:0/2449310834 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f47d8004bd0 0x7f47f4157b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:15.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.482+0000 7f47f3e00640 1 -- 172.21.15.6:0/2449310834 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f47d8004bd0 msgr2=0x7f47f4157b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:15.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.482+0000 7f47f3e00640 1 --2- 172.21.15.6:0/2449310834 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f47d8004bd0 0x7f47f4157b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:15.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.482+0000 7f47f3e00640 1 -- 172.21.15.6:0/2449310834 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f47f415aa10 msgr2=0x7f47f4155de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:15.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.482+0000 7f47f3e00640 1 --2- 172.21.15.6:0/2449310834 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f47f415aa10 0x7f47f4155de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:15.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.482+0000 7f47f3e00640 1 -- 172.21.15.6:0/2449310834 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47f4158070 con 0x7f47f415a590 2024-03-04T09:25:15.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.483+0000 7f47f3e00640 1 --2- 172.21.15.6:0/2449310834 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f47f415a590 0x7f47f41558a0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f47e0011600 tx=0x7f47e0011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:15.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.483+0000 7f47f0c00640 1 -- 172.21.15.6:0/2449310834 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47e000f920 con 0x7f47f415a590 2024-03-04T09:25:15.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.483+0000 7f47f0c00640 1 -- 172.21.15.6:0/2449310834 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47e000fac0 con 0x7f47f415a590 2024-03-04T09:25:15.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.483+0000 7f47fa000640 1 -- 172.21.15.6:0/2449310834 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47f4156480 con 0x7f47f415a590 2024-03-04T09:25:15.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.483+0000 7f47f0c00640 1 -- 172.21.15.6:0/2449310834 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47e000fe70 con 0x7f47f415a590 2024-03-04T09:25:15.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.483+0000 7f47fa000640 1 -- 172.21.15.6:0/2449310834 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47f414e530 con 0x7f47f415a590 2024-03-04T09:25:15.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.484+0000 7f47fa000640 1 -- 172.21.15.6:0/2449310834 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47f414e040 con 0x7f47f415a590 2024-03-04T09:25:15.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.485+0000 7f47f0c00640 1 -- 172.21.15.6:0/2449310834 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f47e0015460 con 0x7f47f415a590 2024-03-04T09:25:15.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.486+0000 7f47f0c00640 1 --2- 172.21.15.6:0/2449310834 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f47d406b4d0 0x7f47d406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:15.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.486+0000 7f47f0c00640 1 -- 172.21.15.6:0/2449310834 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f47e008a8c0 con 0x7f47f415a590 2024-03-04T09:25:15.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.487+0000 7f47f8c00640 1 --2- 172.21.15.6:0/2449310834 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f47d406b4d0 0x7f47d406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:15.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.488+0000 7f47f8c00640 1 --2- 172.21.15.6:0/2449310834 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f47d406b4d0 0x7f47d406d990 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f47e4034000 tx=0x7f47e4031000 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:15.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.491+0000 7f47f0c00640 1 -- 172.21.15.6:0/2449310834 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f47e0019030 con 0x7f47f415a590 2024-03-04T09:25:15.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.690+0000 7f47fa000640 1 -- 172.21.15.6:0/2449310834 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f47f404f330 con 0x7f47f415a590 2024-03-04T09:25:15.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.692+0000 7f47f0c00640 1 -- 172.21.15.6:0/2449310834 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+12836 (secure 0 0 0) 0x7f47e00599f0 con 0x7f47f415a590 2024-03-04T09:25:15.689 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:25:15.689 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":16,"fsid":"edce0578-5ae4-4402-b45a-552bdef84d7c","created":"2024-03-04T09:24:54.523780+0000","modified":"2024-03-04T09:25:10.136308+0000","last_up_change":"2024-03-04T09:25:07.089049+0000","last_in_change":"2024-03-04T09:24:57.982077+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T09:25:07.935198+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a96574f-81f1-4bc6-8463-757520b84a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6825","nonce":1364449401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6827","nonce":1364449401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6831","nonce":1364449401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6829","nonce":1364449401}]},"public_addr":"172.21.15.6:6825/1364449401","cluster_addr":"172.21.15.6:6827/1364449401","heartbeat_back_addr":"172.21.15.6:6831/1364449401","heartbeat_front_addr":"172.21.15.6:6829/1364449401","state":["exists","up"]},{"osd":1,"uuid":"7d79b9c1-6b87-476a-bc36-7db5ee64cd90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6809","nonce":2494103078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6811","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6813","nonce":2494103078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6821","nonce":2494103078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6817","nonce":2494103078}]},"public_addr":"172.21.15.6:6809/2494103078","cluster_addr":"172.21.15.6:6813/2494103078","heartbeat_back_addr":"172.21.15.6:6821/2494103078","heartbeat_front_addr":"172.21.15.6:6817/2494103078","state":["exists","up"]},{"osd":2,"uuid":"492331b9-4ee9-40b6-bacd-85d44cd6c45a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6812","nonce":1500066681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6816","nonce":1500066681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6824","nonce":1500066681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6820","nonce":1500066681}]},"public_addr":"172.21.15.6:6812/1500066681","cluster_addr":"172.21.15.6:6816/1500066681","heartbeat_back_addr":"172.21.15.6:6824/1500066681","heartbeat_front_addr":"172.21.15.6:6820/1500066681","state":["exists","up"]},{"osd":3,"uuid":"21bb2efe-9ba7-4bdc-bc31-180cd1102e21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6801","nonce":1558806123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6803","nonce":1558806123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6807","nonce":1558806123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6805","nonce":1558806123}]},"public_addr":"172.21.15.6:6801/1558806123","cluster_addr":"172.21.15.6:6803/1558806123","heartbeat_back_addr":"172.21.15.6:6807/1558806123","heartbeat_front_addr":"172.21.15.6:6805/1558806123","state":["exists","up"]},{"osd":4,"uuid":"6e2c4b71-19d9-4737-83d9-558705d8644a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6801","nonce":1526135780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6803","nonce":1526135780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6807","nonce":1526135780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6805","nonce":1526135780}]},"public_addr":"172.21.15.146:6801/1526135780","cluster_addr":"172.21.15.146:6803/1526135780","heartbeat_back_addr":"172.21.15.146:6807/1526135780","heartbeat_front_addr":"172.21.15.146:6805/1526135780","state":["exists","up"]},{"osd":5,"uuid":"fb69c525-f1be-4b7d-ab32-6c29b0d6fddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6809","nonce":460262507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6811","nonce":460262507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6815","nonce":460262507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6813","nonce":460262507}]},"public_addr":"172.21.15.146:6809/460262507","cluster_addr":"172.21.15.146:6811/460262507","heartbeat_back_addr":"172.21.15.146:6815/460262507","heartbeat_front_addr":"172.21.15.146:6813/460262507","state":["exists","up"]},{"osd":6,"uuid":"c32e7fed-370d-4401-8ce0-6a4335b0d168","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6823","nonce":1392897493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6827","nonce":1392897493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6831","nonce":1392897493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6829","nonce":1392897493}]},"public_addr":"172.21.15.146:6823/1392897493","cluster_addr":"172.21.15.146:6827/1392897493","heartbeat_back_addr":"172.21.15.146:6831/1392897493","heartbeat_front_addr":"172.21.15.146:6829/1392897493","state":["exists","up"]},{"osd":7,"uuid":"75a79b44-fa5f-426d-a7d3-71f21f78835c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6817","nonce":3268323156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6819","nonce":3268323156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6825","nonce":3268323156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6822","nonce":3268323156}]},"public_addr":"172.21.15.146:6817/3268323156","cluster_addr":"172.21.15.146:6819/3268323156","heartbeat_back_addr":"172.21.15.146:6825/3268323156","heartbeat_front_addr":"172.21.15.146:6822/3268323156","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.310437+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.295850+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.294674+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.526855+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.300023+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.595160+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.346046+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.558741+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T09:25:15.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.694+0000 7f47fa000640 1 -- 172.21.15.6:0/2449310834 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f47d406b4d0 msgr2=0x7f47d406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:15.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.694+0000 7f47fa000640 1 --2- 172.21.15.6:0/2449310834 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f47d406b4d0 0x7f47d406d990 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f47e4034000 tx=0x7f47e4031000 comp rx=0 tx=0).stop 2024-03-04T09:25:15.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.694+0000 7f47fa000640 1 -- 172.21.15.6:0/2449310834 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f47f415a590 msgr2=0x7f47f41558a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:15.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.694+0000 7f47fa000640 1 --2- 172.21.15.6:0/2449310834 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f47f415a590 0x7f47f41558a0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f47e0011600 tx=0x7f47e0011ac0 comp rx=0 tx=0).stop 2024-03-04T09:25:15.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.694+0000 7f47fa000640 1 -- 172.21.15.6:0/2449310834 shutdown_connections 2024-03-04T09:25:15.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.694+0000 7f47fa000640 1 --2- 172.21.15.6:0/2449310834 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f47f415aa10 0x7f47f4155de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:15.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.694+0000 7f47fa000640 1 --2- 172.21.15.6:0/2449310834 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f47f415a590 0x7f47f41558a0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:15.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.694+0000 7f47fa000640 1 --2- 172.21.15.6:0/2449310834 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f47d406b4d0 0x7f47d406d990 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:15.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.694+0000 7f47fa000640 1 --2- 172.21.15.6:0/2449310834 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f47d8004bd0 0x7f47f4157b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:15.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.694+0000 7f47fa000640 1 -- 172.21.15.6:0/2449310834 >> 172.21.15.6:0/2449310834 conn(0x7f47f406b3d0 msgr2=0x7f47f40ce260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:15.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.695+0000 7f47fa000640 1 -- 172.21.15.6:0/2449310834 shutdown_connections 2024-03-04T09:25:15.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.695+0000 7f47fa000640 1 -- 172.21.15.6:0/2449310834 wait complete. 2024-03-04T09:25:15.711 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T09:25:07.935198+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-04T09:25:15.711 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-04T09:25:15.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.930+0000 7fb737e00640 1 Processor -- start 2024-03-04T09:25:15.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.930+0000 7fb737e00640 1 -- start start 2024-03-04T09:25:15.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.932+0000 7fb737e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fb73815f810 0x7fb7380b1b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:15.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.932+0000 7fb737e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb7380b2080 0x7fb7380b2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:15.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.932+0000 7fb737e00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb73815fe40 con 0x7fb738145430 2024-03-04T09:25:15.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.932+0000 7fb737e00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb738160120 con 0x7fb73815f810 2024-03-04T09:25:15.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.932+0000 7fb737e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb738160400 con 0x7fb7380b2080 2024-03-04T09:25:15.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.932+0000 7fb737400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb7380b2080 0x7fb7380b2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:15.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.932+0000 7fb737400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb7380b2080 0x7fb7380b2fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.6:59204/0 (socket says 172.21.15.6:59204) 2024-03-04T09:25:15.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.932+0000 7fb737400640 1 -- 172.21.15.6:0/3542225311 learned_addr learned my addr 172.21.15.6:0/3542225311 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:25:15.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.933+0000 7fb736000640 1 --2- 172.21.15.6:0/3542225311 >> v2:172.21.15.146:3300/0 conn(0x7fb73815f810 0x7fb7380b1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:15.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.933+0000 7fb736000640 1 -- 172.21.15.6:0/3542225311 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb7380b2080 msgr2=0x7fb7380b2fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:15.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.933+0000 7fb736000640 1 --2- 172.21.15.6:0/3542225311 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb7380b2080 0x7fb7380b2fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:15.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.933+0000 7fb736000640 1 -- 172.21.15.6:0/3542225311 >> v1:172.21.15.146:6789/0 conn(0x7fb738145430 legacy=0x7fb73815cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:15.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.933+0000 7fb736000640 1 -- 172.21.15.6:0/3542225311 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7381606e0 con 0x7fb73815f810 2024-03-04T09:25:15.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.933+0000 7fb737400640 1 --2- 172.21.15.6:0/3542225311 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb7380b2080 0x7fb7380b2fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T09:25:15.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.935+0000 7fb736000640 1 --2- 172.21.15.6:0/3542225311 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb73815f810 0x7fb7380b1b40 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fb7240097a0 tx=0x7fb72402b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=abd7435e53b59246 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:15.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.935+0000 7fb735600640 1 -- 172.21.15.6:0/3542225311 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb724035020 con 0x7fb73815f810 2024-03-04T09:25:15.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.935+0000 7fb735600640 1 -- 172.21.15.6:0/3542225311 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb724031810 con 0x7fb73815f810 2024-03-04T09:25:15.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.935+0000 7fb735600640 1 -- 172.21.15.6:0/3542225311 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb724031be0 con 0x7fb73815f810 2024-03-04T09:25:15.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.936+0000 7fb737e00640 1 -- 172.21.15.6:0/3542225311 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb73815f810 msgr2=0x7fb7380b1b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:15.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.936+0000 7fb737e00640 1 --2- 172.21.15.6:0/3542225311 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb73815f810 0x7fb7380b1b40 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fb7240097a0 tx=0x7fb72402b8b0 comp rx=0 tx=0).stop 2024-03-04T09:25:15.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.936+0000 7fb737e00640 1 -- 172.21.15.6:0/3542225311 shutdown_connections 2024-03-04T09:25:15.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.936+0000 7fb737e00640 1 --2- 172.21.15.6:0/3542225311 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb7380b2080 0x7fb7380b2fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:15.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.936+0000 7fb737e00640 1 --2- 172.21.15.6:0/3542225311 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb73815f810 0x7fb7380b1b40 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:15.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.936+0000 7fb737e00640 1 -- 172.21.15.6:0/3542225311 >> 172.21.15.6:0/3542225311 conn(0x7fb73806b3a0 msgr2=0x7fb73806b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:15.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.936+0000 7fb737e00640 1 -- 172.21.15.6:0/3542225311 shutdown_connections 2024-03-04T09:25:15.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.936+0000 7fb737e00640 1 -- 172.21.15.6:0/3542225311 wait complete. 2024-03-04T09:25:15.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.937+0000 7fb737e00640 1 Processor -- start 2024-03-04T09:25:15.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.938+0000 7fb737e00640 1 -- start start 2024-03-04T09:25:15.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.938+0000 7fb737e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb7380b2080 0x7fb7381f4380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:15.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.939+0000 7fb737e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb738145430 0x7fb73815c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:15.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.939+0000 7fb737e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb73815f810 0x7fb73815c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:15.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.939+0000 7fb737e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7380cdf80 con 0x7fb738145430 2024-03-04T09:25:15.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.939+0000 7fb737e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7380cde00 con 0x7fb73815f810 2024-03-04T09:25:15.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.939+0000 7fb737e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7380b3510 con 0x7fb7380b2080 2024-03-04T09:25:15.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.939+0000 7fb736000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb738145430 0x7fb73815c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:15.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.939+0000 7fb736000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb738145430 0x7fb73815c1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41650/0 (socket says 172.21.15.6:41650) 2024-03-04T09:25:15.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.939+0000 7fb736000640 1 -- 172.21.15.6:0/1081389638 learned_addr learned my addr 172.21.15.6:0/1081389638 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:25:15.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.939+0000 7fb736a00640 1 --2- 172.21.15.6:0/1081389638 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb7380b2080 0x7fb7381f4380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:15.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.939+0000 7fb737400640 1 --2- 172.21.15.6:0/1081389638 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb73815f810 0x7fb73815c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:15.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.939+0000 7fb736000640 1 -- 172.21.15.6:0/1081389638 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb7380b2080 msgr2=0x7fb7381f4380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:15.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.939+0000 7fb736000640 1 --2- 172.21.15.6:0/1081389638 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb7380b2080 0x7fb7381f4380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:15.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.939+0000 7fb736000640 1 -- 172.21.15.6:0/1081389638 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb73815f810 msgr2=0x7fb73815c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:15.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.939+0000 7fb736000640 1 --2- 172.21.15.6:0/1081389638 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb73815f810 0x7fb73815c720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:15.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.940+0000 7fb736000640 1 -- 172.21.15.6:0/1081389638 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb738142460 con 0x7fb738145430 2024-03-04T09:25:15.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.940+0000 7fb736000640 1 --2- 172.21.15.6:0/1081389638 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb738145430 0x7fb73815c1e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fb7240098c0 tx=0x7fb7240046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:15.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.940+0000 7fb72ea00640 1 -- 172.21.15.6:0/1081389638 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb724035040 con 0x7fb738145430 2024-03-04T09:25:15.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.940+0000 7fb72ea00640 1 -- 172.21.15.6:0/1081389638 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb724039cf0 con 0x7fb738145430 2024-03-04T09:25:15.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.940+0000 7fb737e00640 1 -- 172.21.15.6:0/1081389638 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb738142740 con 0x7fb738145430 2024-03-04T09:25:15.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.940+0000 7fb72ea00640 1 -- 172.21.15.6:0/1081389638 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb724039ea0 con 0x7fb738145430 2024-03-04T09:25:15.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.940+0000 7fb737e00640 1 -- 172.21.15.6:0/1081389638 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb738142d00 con 0x7fb738145430 2024-03-04T09:25:15.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.942+0000 7fb737e00640 1 -- 172.21.15.6:0/1081389638 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb73815dc80 con 0x7fb738145430 2024-03-04T09:25:15.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.943+0000 7fb72ea00640 1 -- 172.21.15.6:0/1081389638 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb724038020 con 0x7fb738145430 2024-03-04T09:25:15.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.943+0000 7fb72ea00640 1 --2- 172.21.15.6:0/1081389638 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb71806b500 0x7fb71806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:15.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.944+0000 7fb72ea00640 1 -- 172.21.15.6:0/1081389638 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fb7240abfe0 con 0x7fb738145430 2024-03-04T09:25:15.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.944+0000 7fb736a00640 1 --2- 172.21.15.6:0/1081389638 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb71806b500 0x7fb71806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:15.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.945+0000 7fb736a00640 1 --2- 172.21.15.6:0/1081389638 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb71806b500 0x7fb71806d9c0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fb7280012e0 tx=0x7fb728009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:15.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:15.948+0000 7fb72ea00640 1 -- 172.21.15.6:0/1081389638 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb73815dc80 con 0x7fb738145430 2024-03-04T09:25:16.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.140+0000 7fb737e00640 1 -- 172.21.15.6:0/1081389638 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb73804f330 con 0x7fb738145430 2024-03-04T09:25:16.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.141+0000 7fb72ea00640 1 -- 172.21.15.6:0/1081389638 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v16) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb72407b110 con 0x7fb738145430 2024-03-04T09:25:16.138 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-04T09:25:16.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.143+0000 7fb737e00640 1 -- 172.21.15.6:0/1081389638 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb71806b500 msgr2=0x7fb71806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:16.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.143+0000 7fb737e00640 1 --2- 172.21.15.6:0/1081389638 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb71806b500 0x7fb71806d9c0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fb7280012e0 tx=0x7fb728009040 comp rx=0 tx=0).stop 2024-03-04T09:25:16.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.143+0000 7fb737e00640 1 -- 172.21.15.6:0/1081389638 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb738145430 msgr2=0x7fb73815c1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:16.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.143+0000 7fb737e00640 1 --2- 172.21.15.6:0/1081389638 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb738145430 0x7fb73815c1e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fb7240098c0 tx=0x7fb7240046c0 comp rx=0 tx=0).stop 2024-03-04T09:25:16.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.144+0000 7fb737e00640 1 -- 172.21.15.6:0/1081389638 shutdown_connections 2024-03-04T09:25:16.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.144+0000 7fb737e00640 1 --2- 172.21.15.6:0/1081389638 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb73815f810 0x7fb73815c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:16.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.144+0000 7fb737e00640 1 --2- 172.21.15.6:0/1081389638 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb738145430 0x7fb73815c1e0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:16.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.144+0000 7fb737e00640 1 --2- 172.21.15.6:0/1081389638 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb71806b500 0x7fb71806d9c0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:16.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.144+0000 7fb737e00640 1 --2- 172.21.15.6:0/1081389638 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb7380b2080 0x7fb7381f4380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:16.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.144+0000 7fb737e00640 1 -- 172.21.15.6:0/1081389638 >> 172.21.15.6:0/1081389638 conn(0x7fb73806b3a0 msgr2=0x7fb73815d570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:16.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.144+0000 7fb737e00640 1 -- 172.21.15.6:0/1081389638 shutdown_connections 2024-03-04T09:25:16.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.144+0000 7fb737e00640 1 -- 172.21.15.6:0/1081389638 wait complete. 2024-03-04T09:25:16.158 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-04T09:25:16.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.382+0000 7f1796200640 1 Processor -- start 2024-03-04T09:25:16.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.383+0000 7f1796200640 1 -- start start 2024-03-04T09:25:16.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.384+0000 7f1796200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1790161b40 0x7f17900cfd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:16.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.384+0000 7f1796200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f17900d2930 0x7f17900b3740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:16.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.384+0000 7f1796200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17900cea50 con 0x7f17900b5060 2024-03-04T09:25:16.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.384+0000 7f1796200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17900cf900 con 0x7f17900d2930 2024-03-04T09:25:16.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.384+0000 7f1796200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17900cecf0 con 0x7f1790161b40 2024-03-04T09:25:16.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.385+0000 7f178ea00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f17900b5060 0x7f17900d2220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:51538/0 (socket says 172.21.15.6:51538) 2024-03-04T09:25:16.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.385+0000 7f178ea00640 1 -- 172.21.15.6:0/2133632579 learned_addr learned my addr 172.21.15.6:0/2133632579 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:25:16.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.385+0000 7f178f400640 1 --2- 172.21.15.6:0/2133632579 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1790161b40 0x7f17900cfd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:16.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.385+0000 7f178e000640 1 -- 172.21.15.6:0/2133632579 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 520678098 0 0) 0x7f17900cea50 con 0x7f17900b5060 2024-03-04T09:25:16.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.385+0000 7f178fe00640 1 --2- 172.21.15.6:0/2133632579 >> v2:172.21.15.146:3300/0 conn(0x7f17900d2930 0x7f17900b3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:16.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.385+0000 7f178e000640 1 -- 172.21.15.6:0/2133632579 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1774003520 con 0x7f17900b5060 2024-03-04T09:25:16.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.385+0000 7f178e000640 1 -- 172.21.15.6:0/2133632579 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2233718917 0 0) 0x7f1774003520 con 0x7f17900b5060 2024-03-04T09:25:16.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.385+0000 7f178e000640 1 -- 172.21.15.6:0/2133632579 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1790161b40 msgr2=0x7f17900cfd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:16.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.385+0000 7f178e000640 1 --2- 172.21.15.6:0/2133632579 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1790161b40 0x7f17900cfd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:16.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.385+0000 7f178e000640 1 -- 172.21.15.6:0/2133632579 >> v2:172.21.15.146:3300/0 conn(0x7f17900d2930 msgr2=0x7f17900b3740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:16.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.385+0000 7f178e000640 1 --2- 172.21.15.6:0/2133632579 >> v2:172.21.15.146:3300/0 conn(0x7f17900d2930 0x7f17900b3740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:16.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.385+0000 7f178e000640 1 -- 172.21.15.6:0/2133632579 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17900cefd0 con 0x7f17900b5060 2024-03-04T09:25:16.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.385+0000 7f178e000640 1 -- 172.21.15.6:0/2133632579 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7f1778001670 con 0x7f17900b5060 2024-03-04T09:25:16.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.385+0000 7f178f400640 1 --2- 172.21.15.6:0/2133632579 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1790161b40 0x7f17900cfd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T09:25:16.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.386+0000 7f178e000640 1 -- 172.21.15.6:0/2133632579 >> v1:172.21.15.6:6789/0 conn(0x7f17900b5060 legacy=0x7f17900d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:16.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.386+0000 7f178e000640 1 --2- 172.21.15.6:0/2133632579 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1774003a20 0x7f1774003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:16.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.386+0000 7f178e000640 1 --2- 172.21.15.6:0/2133632579 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1774004800 0x7f1774004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:16.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.386+0000 7f178e000640 1 --2- 172.21.15.6:0/2133632579 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1790161b40 0x7f17740051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:16.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.386+0000 7f178e000640 1 -- 172.21.15.6:0/2133632579 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17900cf900 con 0x7f1774003a20 2024-03-04T09:25:16.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.386+0000 7f178e000640 1 -- 172.21.15.6:0/2133632579 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17900cecf0 con 0x7f1774004800 2024-03-04T09:25:16.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.386+0000 7f178e000640 1 -- 172.21.15.6:0/2133632579 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f177400baa0 con 0x7f1790161b40 2024-03-04T09:25:16.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.386+0000 7f178f400640 1 --2- 172.21.15.6:0/2133632579 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1774003a20 0x7f1774003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:16.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.386+0000 7f178f400640 1 -- 172.21.15.6:0/2133632579 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1790161b40 msgr2=0x7f17740051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:16.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.386+0000 7f178f400640 1 --2- 172.21.15.6:0/2133632579 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1790161b40 0x7f17740051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:16.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.386+0000 7f178f400640 1 -- 172.21.15.6:0/2133632579 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1774004800 msgr2=0x7f1774004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:16.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.386+0000 7f178f400640 1 --2- 172.21.15.6:0/2133632579 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1774004800 0x7f1774004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:16.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.386+0000 7f178f400640 1 -- 172.21.15.6:0/2133632579 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1774003520 con 0x7f1774003a20 2024-03-04T09:25:16.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.387+0000 7f178f400640 1 --2- 172.21.15.6:0/2133632579 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1774003a20 0x7f1774003e40 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f1780006fd0 tx=0x7f1774005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fa086d0c9644d82 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:16.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.388+0000 7f178e000640 1 -- 172.21.15.6:0/2133632579 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f178000f040 con 0x7f1774003a20 2024-03-04T09:25:16.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.388+0000 7f178e000640 1 -- 172.21.15.6:0/2133632579 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1780004230 con 0x7f1774003a20 2024-03-04T09:25:16.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.388+0000 7f178e000640 1 -- 172.21.15.6:0/2133632579 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1780003610 con 0x7f1774003a20 2024-03-04T09:25:16.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.388+0000 7f1796200640 1 -- 172.21.15.6:0/2133632579 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1774003a20 msgr2=0x7f1774003e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:16.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.388+0000 7f1796200640 1 --2- 172.21.15.6:0/2133632579 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1774003a20 0x7f1774003e40 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f1780006fd0 tx=0x7f1774005830 comp rx=0 tx=0).stop 2024-03-04T09:25:16.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.389+0000 7f178fe00640 1 -- 172.21.15.6:0/2133632579 reap_dead start 2024-03-04T09:25:16.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.389+0000 7f1796200640 1 -- 172.21.15.6:0/2133632579 shutdown_connections 2024-03-04T09:25:16.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.389+0000 7f1796200640 1 -- 172.21.15.6:0/2133632579 >> 172.21.15.6:0/2133632579 conn(0x7f179009d220 msgr2=0x7f17900d3ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:16.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.389+0000 7f1796200640 1 -- 172.21.15.6:0/2133632579 shutdown_connections 2024-03-04T09:25:16.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.389+0000 7f1796200640 1 -- 172.21.15.6:0/2133632579 wait complete. 2024-03-04T09:25:16.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.391+0000 7f1796200640 1 Processor -- start 2024-03-04T09:25:16.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.391+0000 7f1796200640 1 -- start start 2024-03-04T09:25:16.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.392+0000 7f1796200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1774003a20 0x7f1790148440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:16.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.392+0000 7f1796200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f17900cafc0 0x7f17900cb440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:16.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.392+0000 7f1796200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f179015d1a0 0x7f179015d620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:16.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.392+0000 7f1796200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17900b2ff0 con 0x7f1774003a20 2024-03-04T09:25:16.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.392+0000 7f1796200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17900cf590 con 0x7f179015d1a0 2024-03-04T09:25:16.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.392+0000 7f1796200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17900b3170 con 0x7f17900cafc0 2024-03-04T09:25:16.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.392+0000 7f178f400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1774003a20 0x7f1790148440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:16.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.392+0000 7f178f400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1774003a20 0x7f1790148440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41662/0 (socket says 172.21.15.6:41662) 2024-03-04T09:25:16.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.392+0000 7f178f400640 1 -- 172.21.15.6:0/3795640311 learned_addr learned my addr 172.21.15.6:0/3795640311 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:25:16.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.393+0000 7f178ea00640 1 --2- 172.21.15.6:0/3795640311 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f17900cafc0 0x7f17900cb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:16.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.393+0000 7f178fe00640 1 --2- 172.21.15.6:0/3795640311 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f179015d1a0 0x7f179015d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:16.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.393+0000 7f178f400640 1 -- 172.21.15.6:0/3795640311 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f17900cafc0 msgr2=0x7f17900cb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:16.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.393+0000 7f178f400640 1 --2- 172.21.15.6:0/3795640311 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f17900cafc0 0x7f17900cb440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:16.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.393+0000 7f178f400640 1 -- 172.21.15.6:0/3795640311 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f179015d1a0 msgr2=0x7f179015d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:16.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.393+0000 7f178f400640 1 --2- 172.21.15.6:0/3795640311 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f179015d1a0 0x7f179015d620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:16.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.393+0000 7f178f400640 1 -- 172.21.15.6:0/3795640311 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1790157b90 con 0x7f1774003a20 2024-03-04T09:25:16.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.393+0000 7f178fe00640 1 --2- 172.21.15.6:0/3795640311 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f179015d1a0 0x7f179015d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:25:16.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.393+0000 7f178ea00640 1 --2- 172.21.15.6:0/3795640311 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f17900cafc0 0x7f17900cb440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:25:16.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.393+0000 7f178f400640 1 --2- 172.21.15.6:0/3795640311 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1774003a20 0x7f1790148440 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f1780009990 tx=0x7f1780004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:16.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.393+0000 7f1787400640 1 -- 172.21.15.6:0/3795640311 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1780048070 con 0x7f1774003a20 2024-03-04T09:25:16.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.393+0000 7f1787400640 1 -- 172.21.15.6:0/3795640311 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1780030430 con 0x7f1774003a20 2024-03-04T09:25:16.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.393+0000 7f1787400640 1 -- 172.21.15.6:0/3795640311 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1780030820 con 0x7f1774003a20 2024-03-04T09:25:16.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.393+0000 7f1796200640 1 -- 172.21.15.6:0/3795640311 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1790157e70 con 0x7f1774003a20 2024-03-04T09:25:16.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.394+0000 7f1796200640 1 -- 172.21.15.6:0/3795640311 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1790158430 con 0x7f1774003a20 2024-03-04T09:25:16.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.395+0000 7f1796200640 1 -- 172.21.15.6:0/3795640311 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f179004f3a0 con 0x7f1774003a20 2024-03-04T09:25:16.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.396+0000 7f1787400640 1 -- 172.21.15.6:0/3795640311 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f178004b070 con 0x7f1774003a20 2024-03-04T09:25:16.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.396+0000 7f1787400640 1 --2- 172.21.15.6:0/3795640311 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f176c06b500 0x7f176c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:16.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.397+0000 7f1787400640 1 -- 172.21.15.6:0/3795640311 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f17800ab940 con 0x7f1774003a20 2024-03-04T09:25:16.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.397+0000 7f178ea00640 1 --2- 172.21.15.6:0/3795640311 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f176c06b500 0x7f176c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:16.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.398+0000 7f178ea00640 1 --2- 172.21.15.6:0/3795640311 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f176c06b500 0x7f176c06d9c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f1778001820 tx=0x7f1778009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:16.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.401+0000 7f1787400640 1 -- 172.21.15.6:0/3795640311 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1780034020 con 0x7f1774003a20 2024-03-04T09:25:16.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.603+0000 7f1796200640 1 -- 172.21.15.6:0/3795640311 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f1790160b40 con 0x7f1774003a20 2024-03-04T09:25:16.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.604+0000 7f1787400640 1 -- 172.21.15.6:0/3795640311 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v16) v1 ==== 84+0+65 (secure 0 0 0) 0x7f178007aa70 con 0x7f1774003a20 2024-03-04T09:25:16.601 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:25:16.601 INFO:teuthology.orchestra.run.smithi006.stdout:[ 2024-03-04T09:25:16.601 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:25:16.601 INFO:teuthology.orchestra.run.smithi006.stdout: "poolnum": 1, 2024-03-04T09:25:16.601 INFO:teuthology.orchestra.run.smithi006.stdout: "poolname": ".mgr" 2024-03-04T09:25:16.601 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:25:16.602 INFO:teuthology.orchestra.run.smithi006.stdout:] 2024-03-04T09:25:16.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.607+0000 7f1796200640 1 -- 172.21.15.6:0/3795640311 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f176c06b500 msgr2=0x7f176c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:16.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.607+0000 7f1796200640 1 --2- 172.21.15.6:0/3795640311 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f176c06b500 0x7f176c06d9c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f1778001820 tx=0x7f1778009040 comp rx=0 tx=0).stop 2024-03-04T09:25:16.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.607+0000 7f1796200640 1 -- 172.21.15.6:0/3795640311 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1774003a20 msgr2=0x7f1790148440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:16.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.607+0000 7f1796200640 1 --2- 172.21.15.6:0/3795640311 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1774003a20 0x7f1790148440 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f1780009990 tx=0x7f1780004730 comp rx=0 tx=0).stop 2024-03-04T09:25:16.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.607+0000 7f1796200640 1 -- 172.21.15.6:0/3795640311 shutdown_connections 2024-03-04T09:25:16.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.607+0000 7f1796200640 1 --2- 172.21.15.6:0/3795640311 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f179015d1a0 0x7f179015d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:16.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.607+0000 7f1796200640 1 --2- 172.21.15.6:0/3795640311 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f176c06b500 0x7f176c06d9c0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:16.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.607+0000 7f1796200640 1 --2- 172.21.15.6:0/3795640311 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f17900cafc0 0x7f17900cb440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:16.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.607+0000 7f1796200640 1 --2- 172.21.15.6:0/3795640311 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1774003a20 0x7f1790148440 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:16.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.607+0000 7f1796200640 1 -- 172.21.15.6:0/3795640311 >> 172.21.15.6:0/3795640311 conn(0x7f179009d220 msgr2=0x7f179006c000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:16.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.607+0000 7f1796200640 1 -- 172.21.15.6:0/3795640311 shutdown_connections 2024-03-04T09:25:16.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.607+0000 7f1796200640 1 -- 172.21.15.6:0/3795640311 wait complete. 2024-03-04T09:25:16.624 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-04T09:25:16.624 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-04T09:25:16.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.844+0000 7f2105400640 1 Processor -- start 2024-03-04T09:25:16.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.844+0000 7f2105400640 1 -- start start 2024-03-04T09:25:16.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.846+0000 7f2105400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f21000aced0 0x7f21000b3640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:16.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.846+0000 7f2105400640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21001531f0 con 0x7f210015cdc0 2024-03-04T09:25:16.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.846+0000 7f2105400640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21000b2b80 con 0x7f21000ae7b0 2024-03-04T09:25:16.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.846+0000 7f2105400640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21000b3b80 con 0x7f21000aced0 2024-03-04T09:25:16.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.846+0000 7f20ff200640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f210015cdc0 0x7f210006cc70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:51540/0 (socket says 172.21.15.6:51540) 2024-03-04T09:25:16.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.846+0000 7f20ff200640 1 -- 172.21.15.6:0/833440751 learned_addr learned my addr 172.21.15.6:0/833440751 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:25:16.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.847+0000 7f20fde00640 1 --2- 172.21.15.6:0/833440751 >> v2:172.21.15.146:3300/0 conn(0x7f21000aced0 0x7f21000b3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:16.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.847+0000 7f20fd400640 1 -- 172.21.15.6:0/833440751 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3297001055 0 0) 0x7f21001531f0 con 0x7f210015cdc0 2024-03-04T09:25:16.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.847+0000 7f20fd400640 1 -- 172.21.15.6:0/833440751 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20e4003520 con 0x7f210015cdc0 2024-03-04T09:25:16.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.847+0000 7f20fd400640 1 -- 172.21.15.6:0/833440751 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4041675041 0 0) 0x7f20e4003520 con 0x7f210015cdc0 2024-03-04T09:25:16.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.847+0000 7f20fd400640 1 -- 172.21.15.6:0/833440751 >> v2:172.21.15.146:3300/0 conn(0x7f21000aced0 msgr2=0x7f21000b3640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:16.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.847+0000 7f20fd400640 1 --2- 172.21.15.6:0/833440751 >> v2:172.21.15.146:3300/0 conn(0x7f21000aced0 0x7f21000b3640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:16.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.847+0000 7f20fd400640 1 -- 172.21.15.6:0/833440751 >> v1:172.21.15.146:6789/0 conn(0x7f21000ae7b0 legacy=0x7f21000aeb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:16.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.847+0000 7f20fd400640 1 -- 172.21.15.6:0/833440751 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f210006ebd0 con 0x7f210015cdc0 2024-03-04T09:25:16.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.847+0000 7f20fd400640 1 -- 172.21.15.6:0/833440751 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7f20f8001670 con 0x7f210015cdc0 2024-03-04T09:25:16.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.847+0000 7f20fd400640 1 -- 172.21.15.6:0/833440751 >> v1:172.21.15.6:6789/0 conn(0x7f210015cdc0 legacy=0x7f210006cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:16.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.848+0000 7f20fd400640 1 --2- 172.21.15.6:0/833440751 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f20e40039b0 0x7f20e4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:16.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.848+0000 7f20fd400640 1 --2- 172.21.15.6:0/833440751 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f20e4004800 0x7f20e4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:16.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.848+0000 7f20fd400640 1 --2- 172.21.15.6:0/833440751 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f20e40051c0 0x7f20e400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:16.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.848+0000 7f20fd400640 1 -- 172.21.15.6:0/833440751 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21000b3b80 con 0x7f20e40051c0 2024-03-04T09:25:16.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.848+0000 7f20fd400640 1 -- 172.21.15.6:0/833440751 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20e400c060 con 0x7f20e4004800 2024-03-04T09:25:16.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.848+0000 7f20fd400640 1 -- 172.21.15.6:0/833440751 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20e400c260 con 0x7f20e40039b0 2024-03-04T09:25:16.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.848+0000 7f20ff200640 1 --2- 172.21.15.6:0/833440751 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f20e40051c0 0x7f20e400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:16.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.848+0000 7f20fe800640 1 --2- 172.21.15.6:0/833440751 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f20e40039b0 0x7f20e4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:16.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.848+0000 7f20fde00640 1 --2- 172.21.15.6:0/833440751 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f20e4004800 0x7f20e4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:16.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.848+0000 7f20ff200640 1 -- 172.21.15.6:0/833440751 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f20e40039b0 msgr2=0x7f20e4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:16.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.849+0000 7f20ff200640 1 --2- 172.21.15.6:0/833440751 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f20e40039b0 0x7f20e4003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:16.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.849+0000 7f20ff200640 1 -- 172.21.15.6:0/833440751 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f20e4004800 msgr2=0x7f20e4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:16.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.849+0000 7f20ff200640 1 --2- 172.21.15.6:0/833440751 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f20e4004800 0x7f20e4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:16.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.849+0000 7f20ff200640 1 -- 172.21.15.6:0/833440751 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20e4003520 con 0x7f20e40051c0 2024-03-04T09:25:16.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.849+0000 7f20fde00640 1 --2- 172.21.15.6:0/833440751 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f20e4004800 0x7f20e4004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T09:25:16.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.850+0000 7f20ff200640 1 -- 172.21.15.6:0/833440751 reap_dead start 2024-03-04T09:25:16.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.850+0000 7f20ff200640 1 --2- 172.21.15.6:0/833440751 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f20e40051c0 0x7f20e400b9c0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f20f800c240 tx=0x7f20e4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ffdfd6078146455 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:16.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.850+0000 7f20fd400640 1 -- 172.21.15.6:0/833440751 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20f8007cf0 con 0x7f20e40051c0 2024-03-04T09:25:16.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.850+0000 7f20fd400640 1 -- 172.21.15.6:0/833440751 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20f8030070 con 0x7f20e40051c0 2024-03-04T09:25:16.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.850+0000 7f20fd400640 1 -- 172.21.15.6:0/833440751 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20f8005d30 con 0x7f20e40051c0 2024-03-04T09:25:16.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.851+0000 7f2105400640 1 -- 172.21.15.6:0/833440751 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f20e40051c0 msgr2=0x7f20e400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:16.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.851+0000 7f2105400640 1 --2- 172.21.15.6:0/833440751 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f20e40051c0 0x7f20e400b9c0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f20f800c240 tx=0x7f20e4005830 comp rx=0 tx=0).stop 2024-03-04T09:25:16.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.851+0000 7f2105400640 1 -- 172.21.15.6:0/833440751 shutdown_connections 2024-03-04T09:25:16.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.851+0000 7f2105400640 1 --2- 172.21.15.6:0/833440751 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f20e40051c0 0x7f20e400b9c0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:16.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.851+0000 7f2105400640 1 -- 172.21.15.6:0/833440751 >> 172.21.15.6:0/833440751 conn(0x7f210006afb0 msgr2=0x7f210006b3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:16.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.851+0000 7f2105400640 1 -- 172.21.15.6:0/833440751 shutdown_connections 2024-03-04T09:25:16.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.851+0000 7f2105400640 1 -- 172.21.15.6:0/833440751 wait complete. 2024-03-04T09:25:16.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.854+0000 7f2105400640 1 Processor -- start 2024-03-04T09:25:16.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.854+0000 7f2105400640 1 -- start start 2024-03-04T09:25:16.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.855+0000 7f2105400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f20e40051c0 0x7f210013e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:16.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.855+0000 7f2105400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f210015e4c0 0x7f210015e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:16.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.855+0000 7f2105400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2100148f40 0x7f21001493c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:16.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.855+0000 7f2105400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21000b2e40 con 0x7f210015e4c0 2024-03-04T09:25:16.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.855+0000 7f2105400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21000d04d0 con 0x7f2100148f40 2024-03-04T09:25:16.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.855+0000 7f2105400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21000b3120 con 0x7f20e40051c0 2024-03-04T09:25:16.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.855+0000 7f20fde00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f210015e4c0 0x7f210015e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:16.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.855+0000 7f20fde00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f210015e4c0 0x7f210015e940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41680/0 (socket says 172.21.15.6:41680) 2024-03-04T09:25:16.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.855+0000 7f20fde00640 1 -- 172.21.15.6:0/4291214126 learned_addr learned my addr 172.21.15.6:0/4291214126 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:25:16.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.856+0000 7f20fe800640 1 --2- 172.21.15.6:0/4291214126 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f20e40051c0 0x7f210013e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:16.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.856+0000 7f20fde00640 1 -- 172.21.15.6:0/4291214126 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f20e40051c0 msgr2=0x7f210013e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:16.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.856+0000 7f20ff200640 1 --2- 172.21.15.6:0/4291214126 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2100148f40 0x7f21001493c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:16.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.856+0000 7f20fde00640 1 --2- 172.21.15.6:0/4291214126 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f20e40051c0 0x7f210013e560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:16.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.856+0000 7f20fde00640 1 -- 172.21.15.6:0/4291214126 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2100148f40 msgr2=0x7f21001493c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:16.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.856+0000 7f20fde00640 1 --2- 172.21.15.6:0/4291214126 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2100148f40 0x7f21001493c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:16.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.856+0000 7f20fde00640 1 -- 172.21.15.6:0/4291214126 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2100149900 con 0x7f210015e4c0 2024-03-04T09:25:16.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.856+0000 7f20fe800640 1 --2- 172.21.15.6:0/4291214126 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f20e40051c0 0x7f210013e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:25:16.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.856+0000 7f20fde00640 1 --2- 172.21.15.6:0/4291214126 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f210015e4c0 0x7f210015e940 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f20e80115a0 tx=0x7f20e8011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:16.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.856+0000 7f20f6a00640 1 -- 172.21.15.6:0/4291214126 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20e8015870 con 0x7f210015e4c0 2024-03-04T09:25:16.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.856+0000 7f20f6a00640 1 -- 172.21.15.6:0/4291214126 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20e8015a10 con 0x7f210015e4c0 2024-03-04T09:25:16.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.856+0000 7f20f6a00640 1 -- 172.21.15.6:0/4291214126 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20e8015dc0 con 0x7f210015e4c0 2024-03-04T09:25:16.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.856+0000 7f2105400640 1 -- 172.21.15.6:0/4291214126 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2100149bc0 con 0x7f210015e4c0 2024-03-04T09:25:16.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.857+0000 7f2105400640 1 -- 172.21.15.6:0/4291214126 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f210014a180 con 0x7f210015e4c0 2024-03-04T09:25:16.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.858+0000 7f2105400640 1 -- 172.21.15.6:0/4291214126 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2100068010 con 0x7f210015e4c0 2024-03-04T09:25:16.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.859+0000 7f20f6a00640 1 -- 172.21.15.6:0/4291214126 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f20e800f480 con 0x7f210015e4c0 2024-03-04T09:25:16.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.860+0000 7f20f6a00640 1 --2- 172.21.15.6:0/4291214126 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f20dc06b4d0 0x7f20dc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:16.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.860+0000 7f20f6a00640 1 -- 172.21.15.6:0/4291214126 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f20e808a870 con 0x7f210015e4c0 2024-03-04T09:25:16.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.860+0000 7f20fe800640 1 --2- 172.21.15.6:0/4291214126 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f20dc06b4d0 0x7f20dc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:16.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.861+0000 7f20fe800640 1 --2- 172.21.15.6:0/4291214126 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f20dc06b4d0 0x7f20dc06d990 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f20f0007fd0 tx=0x7f20f0009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:16.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:16.865+0000 7f20f6a00640 1 -- 172.21.15.6:0/4291214126 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f20e8019030 con 0x7f210015e4c0 2024-03-04T09:25:17.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:17.065+0000 7f2105400640 1 -- 172.21.15.6:0/4291214126 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f21000b4670 con 0x7f210015e4c0 2024-03-04T09:25:17.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:17.416+0000 7f20f6a00640 1 -- 172.21.15.6:0/4291214126 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v17) v1 ==== 138+0+0 (secure 0 0 0) 0x7f20e80599a0 con 0x7f210015e4c0 2024-03-04T09:25:17.413 INFO:teuthology.orchestra.run.smithi006.stderr:pool 'cephfs_metadata' created 2024-03-04T09:25:17.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:17.420+0000 7f2105400640 1 -- 172.21.15.6:0/4291214126 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f20dc06b4d0 msgr2=0x7f20dc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:17.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:17.420+0000 7f2105400640 1 --2- 172.21.15.6:0/4291214126 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f20dc06b4d0 0x7f20dc06d990 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f20f0007fd0 tx=0x7f20f0009040 comp rx=0 tx=0).stop 2024-03-04T09:25:17.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:17.420+0000 7f2105400640 1 -- 172.21.15.6:0/4291214126 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f210015e4c0 msgr2=0x7f210015e940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:17.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:17.420+0000 7f2105400640 1 --2- 172.21.15.6:0/4291214126 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f210015e4c0 0x7f210015e940 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f20e80115a0 tx=0x7f20e8011a60 comp rx=0 tx=0).stop 2024-03-04T09:25:17.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:17.420+0000 7f2105400640 1 -- 172.21.15.6:0/4291214126 shutdown_connections 2024-03-04T09:25:17.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:17.420+0000 7f2105400640 1 --2- 172.21.15.6:0/4291214126 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2100148f40 0x7f21001493c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:17.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:17.420+0000 7f2105400640 1 --2- 172.21.15.6:0/4291214126 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f210015e4c0 0x7f210015e940 secure :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f20e80115a0 tx=0x7f20e8011a60 comp rx=0 tx=0).stop 2024-03-04T09:25:17.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:17.420+0000 7f2105400640 1 --2- 172.21.15.6:0/4291214126 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f20dc06b4d0 0x7f20dc06d990 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:17.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:17.420+0000 7f2105400640 1 --2- 172.21.15.6:0/4291214126 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f20e40051c0 0x7f210013e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:17.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:17.420+0000 7f2105400640 1 -- 172.21.15.6:0/4291214126 >> 172.21.15.6:0/4291214126 conn(0x7f210006afb0 msgr2=0x7f210015fef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:17.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:17.421+0000 7f2105400640 1 -- 172.21.15.6:0/4291214126 shutdown_connections 2024-03-04T09:25:17.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:17.421+0000 7f2105400640 1 -- 172.21.15.6:0/4291214126 wait complete. 2024-03-04T09:25:17.447 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-04T09:25:17.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:17.654+0000 7f4c62c00640 1 Processor -- start 2024-03-04T09:25:17.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:17.654+0000 7f4c62c00640 1 -- start start 2024-03-04T09:25:17.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:17.656+0000 7f4c62c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4c5c0ad6f0 0x7f4c5c15c370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:17.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:17.656+0000 7f4c62c00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c5c156830 con 0x7f4c5c06d5d0 2024-03-04T09:25:17.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:17.656+0000 7f4c62c00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c5c160120 con 0x7f4c5c15f960 2024-03-04T09:25:17.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:17.656+0000 7f4c62c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c5c06ce60 con 0x7f4c5c0ad6f0 2024-03-04T09:25:17.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:17.656+0000 7f4c60c00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f4c5c06d5d0 0x7f4c5c06da40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:51548/0 (socket says 172.21.15.6:51548) 2024-03-04T09:25:17.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:17.656+0000 7f4c60c00640 1 -- 172.21.15.6:0/1819399943 learned_addr learned my addr 172.21.15.6:0/1819399943 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:25:17.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:17.657+0000 7f4c5be00640 1 --2- 172.21.15.6:0/1819399943 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4c5c0ad6f0 0x7f4c5c15c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:17.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:17.657+0000 7f4c5b400640 1 -- 172.21.15.6:0/1819399943 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3393582704 0 0) 0x7f4c5c156830 con 0x7f4c5c06d5d0 2024-03-04T09:25:17.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:17.657+0000 7f4c5b400640 1 -- 172.21.15.6:0/1819399943 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c40003520 con 0x7f4c5c06d5d0 2024-03-04T09:25:17.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:17.657+0000 7f4c5b400640 1 -- 172.21.15.6:0/1819399943 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 29533686 0 0) 0x7f4c40003520 con 0x7f4c5c06d5d0 2024-03-04T09:25:17.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:17.657+0000 7f4c5b400640 1 -- 172.21.15.6:0/1819399943 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4c5c0ad6f0 msgr2=0x7f4c5c15c370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:17.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:17.657+0000 7f4c5b400640 1 --2- 172.21.15.6:0/1819399943 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4c5c0ad6f0 0x7f4c5c15c370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:17.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:17.657+0000 7f4c5b400640 1 -- 172.21.15.6:0/1819399943 >> v1:172.21.15.146:6789/0 conn(0x7f4c5c15f960 legacy=0x7f4c5c0adad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:17.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:17.657+0000 7f4c5b400640 1 -- 172.21.15.6:0/1819399943 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c5c160400 con 0x7f4c5c06d5d0 2024-03-04T09:25:17.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:17.657+0000 7f4c5b400640 1 -- 172.21.15.6:0/1819399943 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7f4c54001670 con 0x7f4c5c06d5d0 2024-03-04T09:25:17.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:17.657+0000 7f4c5b400640 1 -- 172.21.15.6:0/1819399943 >> v1:172.21.15.6:6789/0 conn(0x7f4c5c06d5d0 legacy=0x7f4c5c06da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:17.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:17.657+0000 7f4c5b400640 1 --2- 172.21.15.6:0/1819399943 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4c400039e0 0x7f4c40003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:17.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:17.657+0000 7f4c5b400640 1 --2- 172.21.15.6:0/1819399943 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4c40004800 0x7f4c40004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:17.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:17.657+0000 7f4c5b400640 1 --2- 172.21.15.6:0/1819399943 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4c40005210 0x7f4c4000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:17.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:17.657+0000 7f4c5b400640 1 -- 172.21.15.6:0/1819399943 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c5c06ce60 con 0x7f4c40004800 2024-03-04T09:25:17.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:17.657+0000 7f4c5b400640 1 -- 172.21.15.6:0/1819399943 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c4000c0c0 con 0x7f4c400039e0 2024-03-04T09:25:17.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:17.657+0000 7f4c5b400640 1 -- 172.21.15.6:0/1819399943 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c4000c2c0 con 0x7f4c40005210 2024-03-04T09:25:17.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:17.657+0000 7f4c5be00640 1 --2- 172.21.15.6:0/1819399943 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4c5c0ad6f0 0x7f4c5c15c370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T09:25:17.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:17.658+0000 7f4c53400640 1 --2- 172.21.15.6:0/1819399943 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4c40004800 0x7f4c40004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:17.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:17.658+0000 7f4c60c00640 1 --2- 172.21.15.6:0/1819399943 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4c40005210 0x7f4c4000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:17.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:17.658+0000 7f4c5be00640 1 --2- 172.21.15.6:0/1819399943 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4c400039e0 0x7f4c40003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:17.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:17.658+0000 7f4c53400640 1 -- 172.21.15.6:0/1819399943 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4c40005210 msgr2=0x7f4c4000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:17.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:17.658+0000 7f4c53400640 1 --2- 172.21.15.6:0/1819399943 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4c40005210 0x7f4c4000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:17.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:17.658+0000 7f4c53400640 1 -- 172.21.15.6:0/1819399943 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4c400039e0 msgr2=0x7f4c40003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:17.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:17.658+0000 7f4c53400640 1 --2- 172.21.15.6:0/1819399943 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4c400039e0 0x7f4c40003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:17.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:17.658+0000 7f4c53400640 1 -- 172.21.15.6:0/1819399943 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c40003520 con 0x7f4c40004800 2024-03-04T09:25:17.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:17.659+0000 7f4c53400640 1 --2- 172.21.15.6:0/1819399943 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4c40004800 0x7f4c40004c80 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f4c44001750 tx=0x7f4c40005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43c8d1bedb6268b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:17.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:17.660+0000 7f4c5b400640 1 -- 172.21.15.6:0/1819399943 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c44001370 con 0x7f4c40004800 2024-03-04T09:25:17.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:17.660+0000 7f4c5b400640 1 -- 172.21.15.6:0/1819399943 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c440052d0 con 0x7f4c40004800 2024-03-04T09:25:17.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:17.660+0000 7f4c5b400640 1 -- 172.21.15.6:0/1819399943 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c44003830 con 0x7f4c40004800 2024-03-04T09:25:17.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:17.660+0000 7f4c62c00640 1 -- 172.21.15.6:0/1819399943 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4c40004800 msgr2=0x7f4c40004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:17.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:17.660+0000 7f4c62c00640 1 --2- 172.21.15.6:0/1819399943 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4c40004800 0x7f4c40004c80 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f4c44001750 tx=0x7f4c40005830 comp rx=0 tx=0).stop 2024-03-04T09:25:17.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:17.660+0000 7f4c60c00640 1 -- 172.21.15.6:0/1819399943 reap_dead start 2024-03-04T09:25:17.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:17.660+0000 7f4c62c00640 1 -- 172.21.15.6:0/1819399943 shutdown_connections 2024-03-04T09:25:17.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:17.660+0000 7f4c62c00640 1 -- 172.21.15.6:0/1819399943 >> 172.21.15.6:0/1819399943 conn(0x7f4c5c06b3d0 msgr2=0x7f4c5c06b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:17.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:17.660+0000 7f4c62c00640 1 -- 172.21.15.6:0/1819399943 shutdown_connections 2024-03-04T09:25:17.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:17.661+0000 7f4c62c00640 1 -- 172.21.15.6:0/1819399943 wait complete. 2024-03-04T09:25:17.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:17.662+0000 7f4c62c00640 1 Processor -- start 2024-03-04T09:25:17.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:17.662+0000 7f4c62c00640 1 -- start start 2024-03-04T09:25:17.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:17.663+0000 7f4c62c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4c40004800 0x7f4c5c0cef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:17.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:17.663+0000 7f4c62c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4c5c1f5fc0 0x7f4c5c1e56d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:17.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:17.663+0000 7f4c62c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4c5c1e7a40 0x7f4c5c1547a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:17.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:17.663+0000 7f4c62c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c5c15fdc0 con 0x7f4c40004800 2024-03-04T09:25:17.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:17.663+0000 7f4c62c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c5c06d0b0 con 0x7f4c5c1f5fc0 2024-03-04T09:25:17.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:17.663+0000 7f4c62c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c5c15c370 con 0x7f4c5c1e7a40 2024-03-04T09:25:17.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:17.664+0000 7f4c5be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4c40004800 0x7f4c5c0cef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:17.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:17.664+0000 7f4c5be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4c40004800 0x7f4c5c0cef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41690/0 (socket says 172.21.15.6:41690) 2024-03-04T09:25:17.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:17.664+0000 7f4c5be00640 1 -- 172.21.15.6:0/1285286481 learned_addr learned my addr 172.21.15.6:0/1285286481 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:25:17.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:17.664+0000 7f4c60c00640 1 --2- 172.21.15.6:0/1285286481 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4c5c1e7a40 0x7f4c5c1547a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:17.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:17.664+0000 7f4c5be00640 1 -- 172.21.15.6:0/1285286481 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4c5c1e7a40 msgr2=0x7f4c5c1547a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:17.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:17.664+0000 7f4c5be00640 1 --2- 172.21.15.6:0/1285286481 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4c5c1e7a40 0x7f4c5c1547a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:17.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:17.664+0000 7f4c5be00640 1 -- 172.21.15.6:0/1285286481 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4c5c1f5fc0 msgr2=0x7f4c5c1e56d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:17.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:17.664+0000 7f4c53400640 1 --2- 172.21.15.6:0/1285286481 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4c5c1f5fc0 0x7f4c5c1e56d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:17.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:17.664+0000 7f4c5be00640 1 --2- 172.21.15.6:0/1285286481 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4c5c1f5fc0 0x7f4c5c1e56d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:17.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:17.664+0000 7f4c5be00640 1 -- 172.21.15.6:0/1285286481 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c5c0cf5c0 con 0x7f4c40004800 2024-03-04T09:25:17.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:17.664+0000 7f4c5be00640 1 --2- 172.21.15.6:0/1285286481 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4c40004800 0x7f4c5c0cef00 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c004690 tx=0x7f4c4c007700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:17.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:17.665+0000 7f4c58c00640 1 -- 172.21.15.6:0/1285286481 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c4c014610 con 0x7f4c40004800 2024-03-04T09:25:17.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:17.665+0000 7f4c58c00640 1 -- 172.21.15.6:0/1285286481 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c4c019070 con 0x7f4c40004800 2024-03-04T09:25:17.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:17.665+0000 7f4c62c00640 1 -- 172.21.15.6:0/1285286481 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c5c0cc700 con 0x7f4c40004800 2024-03-04T09:25:17.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:17.665+0000 7f4c58c00640 1 -- 172.21.15.6:0/1285286481 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c4c0149b0 con 0x7f4c40004800 2024-03-04T09:25:17.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:17.665+0000 7f4c62c00640 1 -- 172.21.15.6:0/1285286481 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c5c0cccc0 con 0x7f4c40004800 2024-03-04T09:25:17.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:17.666+0000 7f4c62c00640 1 -- 172.21.15.6:0/1285286481 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c5c068010 con 0x7f4c40004800 2024-03-04T09:25:17.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:17.667+0000 7f4c58c00640 1 -- 172.21.15.6:0/1285286481 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4c4c005d10 con 0x7f4c40004800 2024-03-04T09:25:17.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:17.668+0000 7f4c58c00640 1 --2- 172.21.15.6:0/1285286481 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f4c3806b4d0 0x7f4c3806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:17.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:17.668+0000 7f4c58c00640 1 -- 172.21.15.6:0/1285286481 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f4c4c08b510 con 0x7f4c40004800 2024-03-04T09:25:17.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:17.669+0000 7f4c53400640 1 --2- 172.21.15.6:0/1285286481 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f4c3806b4d0 0x7f4c3806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:17.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:17.670+0000 7f4c53400640 1 --2- 172.21.15.6:0/1285286481 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f4c3806b4d0 0x7f4c3806d990 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f4c44033000 tx=0x7f4c44009030 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:17.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:17.672+0000 7f4c58c00640 1 -- 172.21.15.6:0/1285286481 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4c4c05a550 con 0x7f4c40004800 2024-03-04T09:25:17.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:17.874+0000 7f4c62c00640 1 -- 172.21.15.6:0/1285286481 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f4c5c0cc440 con 0x7f4c40004800 2024-03-04T09:25:18.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:18.431+0000 7f4c58c00640 1 -- 172.21.15.6:0/1285286481 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v18) v1 ==== 170+0+0 (secure 0 0 0) 0x7f4c4c059e50 con 0x7f4c40004800 2024-03-04T09:25:18.428 INFO:teuthology.orchestra.run.smithi006.stderr:pool 'cephfs_data' created 2024-03-04T09:25:18.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:18.434+0000 7f4c62c00640 1 -- 172.21.15.6:0/1285286481 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f4c3806b4d0 msgr2=0x7f4c3806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:18.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:18.434+0000 7f4c62c00640 1 --2- 172.21.15.6:0/1285286481 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f4c3806b4d0 0x7f4c3806d990 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f4c44033000 tx=0x7f4c44009030 comp rx=0 tx=0).stop 2024-03-04T09:25:18.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:18.435+0000 7f4c62c00640 1 -- 172.21.15.6:0/1285286481 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4c40004800 msgr2=0x7f4c5c0cef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:18.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:18.435+0000 7f4c62c00640 1 --2- 172.21.15.6:0/1285286481 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4c40004800 0x7f4c5c0cef00 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c004690 tx=0x7f4c4c007700 comp rx=0 tx=0).stop 2024-03-04T09:25:18.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:18.435+0000 7f4c62c00640 1 -- 172.21.15.6:0/1285286481 shutdown_connections 2024-03-04T09:25:18.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:18.435+0000 7f4c62c00640 1 --2- 172.21.15.6:0/1285286481 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f4c3806b4d0 0x7f4c3806d990 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:18.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:18.435+0000 7f4c62c00640 1 --2- 172.21.15.6:0/1285286481 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4c5c1e7a40 0x7f4c5c1547a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:18.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:18.435+0000 7f4c62c00640 1 --2- 172.21.15.6:0/1285286481 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4c5c1f5fc0 0x7f4c5c1e56d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:18.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:18.435+0000 7f4c62c00640 1 --2- 172.21.15.6:0/1285286481 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4c40004800 0x7f4c5c0cef00 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:18.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:18.435+0000 7f4c62c00640 1 -- 172.21.15.6:0/1285286481 >> 172.21.15.6:0/1285286481 conn(0x7f4c5c06b3d0 msgr2=0x7f4c5c0b0ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:18.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:18.435+0000 7f4c62c00640 1 -- 172.21.15.6:0/1285286481 shutdown_connections 2024-03-04T09:25:18.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:18.436+0000 7f4c62c00640 1 -- 172.21.15.6:0/1285286481 wait complete. 2024-03-04T09:25:18.462 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-04T09:25:18.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:18.650+0000 7fde30c00640 1 Processor -- start 2024-03-04T09:25:18.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:18.651+0000 7fde30c00640 1 -- start start 2024-03-04T09:25:18.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:18.652+0000 7fde30c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde2c0d0d40 0x7fde2c0b3510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:18.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:18.652+0000 7fde30c00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde2c0ceca0 con 0x7fde2c0d4860 2024-03-04T09:25:18.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:18.652+0000 7fde30c00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde2c0cef80 con 0x7fde2c15dee0 2024-03-04T09:25:18.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:18.652+0000 7fde30c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde2c0cf5d0 con 0x7fde2c0d0d40 2024-03-04T09:25:18.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:18.652+0000 7fde2aa00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fde2c0d4860 0x7fde2c0d0630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:51554/0 (socket says 172.21.15.6:51554) 2024-03-04T09:25:18.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:18.652+0000 7fde2aa00640 1 -- 172.21.15.6:0/111772353 learned_addr learned my addr 172.21.15.6:0/111772353 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:25:18.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:18.653+0000 7fde2be00640 1 --2- 172.21.15.6:0/111772353 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde2c0d0d40 0x7fde2c0b3510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:18.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:18.653+0000 7fde2a000640 1 -- 172.21.15.6:0/111772353 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1573463732 0 0) 0x7fde2c0ceca0 con 0x7fde2c0d4860 2024-03-04T09:25:18.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:18.653+0000 7fde2a000640 1 -- 172.21.15.6:0/111772353 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde10003520 con 0x7fde2c0d4860 2024-03-04T09:25:18.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:18.653+0000 7fde2a000640 1 -- 172.21.15.6:0/111772353 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1396702757 0 0) 0x7fde2c0cef80 con 0x7fde2c15dee0 2024-03-04T09:25:18.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:18.653+0000 7fde2a000640 1 -- 172.21.15.6:0/111772353 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde2c0ceca0 con 0x7fde2c15dee0 2024-03-04T09:25:18.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:18.653+0000 7fde2a000640 1 -- 172.21.15.6:0/111772353 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 62707191 0 0) 0x7fde10003520 con 0x7fde2c0d4860 2024-03-04T09:25:18.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:18.653+0000 7fde2a000640 1 -- 172.21.15.6:0/111772353 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde2c0d0d40 msgr2=0x7fde2c0b3510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:18.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:18.653+0000 7fde2a000640 1 --2- 172.21.15.6:0/111772353 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde2c0d0d40 0x7fde2c0b3510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:18.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:18.653+0000 7fde2a000640 1 -- 172.21.15.6:0/111772353 >> v1:172.21.15.146:6789/0 conn(0x7fde2c15dee0 legacy=0x7fde2c0d18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:18.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:18.653+0000 7fde2a000640 1 -- 172.21.15.6:0/111772353 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde2c0cf260 con 0x7fde2c0d4860 2024-03-04T09:25:18.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:18.653+0000 7fde2a000640 1 -- 172.21.15.6:0/111772353 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7fde1c001a10 con 0x7fde2c0d4860 2024-03-04T09:25:18.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:18.653+0000 7fde2a000640 1 -- 172.21.15.6:0/111772353 >> v1:172.21.15.6:6789/0 conn(0x7fde2c0d4860 legacy=0x7fde2c0d0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:18.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:18.653+0000 7fde2a000640 1 --2- 172.21.15.6:0/111772353 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fde10003b70 0x7fde10003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:18.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:18.653+0000 7fde2a000640 1 --2- 172.21.15.6:0/111772353 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fde10004800 0x7fde10004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:18.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:18.654+0000 7fde2a000640 1 --2- 172.21.15.6:0/111772353 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde2c0d0d40 0x7fde100051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:18.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:18.654+0000 7fde2a000640 1 -- 172.21.15.6:0/111772353 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde2c0cf5d0 con 0x7fde10003b70 2024-03-04T09:25:18.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:18.654+0000 7fde2a000640 1 -- 172.21.15.6:0/111772353 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde1000bad0 con 0x7fde10004800 2024-03-04T09:25:18.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:18.654+0000 7fde2a000640 1 -- 172.21.15.6:0/111772353 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde1000bcd0 con 0x7fde2c0d0d40 2024-03-04T09:25:18.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:18.654+0000 7fde2b400640 1 --2- 172.21.15.6:0/111772353 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fde10003b70 0x7fde10003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:18.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:18.654+0000 7fde2aa00640 1 --2- 172.21.15.6:0/111772353 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fde10004800 0x7fde10004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:18.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:18.654+0000 7fde2be00640 1 --2- 172.21.15.6:0/111772353 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde2c0d0d40 0x7fde100051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:18.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:18.654+0000 7fde2b400640 1 -- 172.21.15.6:0/111772353 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde2c0d0d40 msgr2=0x7fde100051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:18.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:18.654+0000 7fde2b400640 1 --2- 172.21.15.6:0/111772353 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde2c0d0d40 0x7fde100051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:18.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:18.654+0000 7fde2b400640 1 -- 172.21.15.6:0/111772353 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fde10004800 msgr2=0x7fde10004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:18.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:18.654+0000 7fde2b400640 1 --2- 172.21.15.6:0/111772353 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fde10004800 0x7fde10004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:18.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:18.654+0000 7fde2b400640 1 -- 172.21.15.6:0/111772353 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde10003520 con 0x7fde10003b70 2024-03-04T09:25:18.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:18.655+0000 7fde2b400640 1 --2- 172.21.15.6:0/111772353 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fde10003b70 0x7fde10003f70 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fde140018d0 tx=0x7fde10005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdcb430d8014adfc server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:18.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:18.656+0000 7fde2a000640 1 -- 172.21.15.6:0/111772353 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde14037070 con 0x7fde10003b70 2024-03-04T09:25:18.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:18.656+0000 7fde2a000640 1 -- 172.21.15.6:0/111772353 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde140052d0 con 0x7fde10003b70 2024-03-04T09:25:18.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:18.656+0000 7fde2a000640 1 -- 172.21.15.6:0/111772353 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde140014f0 con 0x7fde10003b70 2024-03-04T09:25:18.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:18.656+0000 7fde30c00640 1 -- 172.21.15.6:0/111772353 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fde10003b70 msgr2=0x7fde10003f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:18.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:18.656+0000 7fde30c00640 1 --2- 172.21.15.6:0/111772353 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fde10003b70 0x7fde10003f70 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fde140018d0 tx=0x7fde10005830 comp rx=0 tx=0).stop 2024-03-04T09:25:18.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:18.656+0000 7fde2be00640 1 -- 172.21.15.6:0/111772353 reap_dead start 2024-03-04T09:25:18.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:18.656+0000 7fde30c00640 1 -- 172.21.15.6:0/111772353 shutdown_connections 2024-03-04T09:25:18.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:18.656+0000 7fde30c00640 1 -- 172.21.15.6:0/111772353 >> 172.21.15.6:0/111772353 conn(0x7fde2c0d3cd0 msgr2=0x7fde2c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:18.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:18.656+0000 7fde30c00640 1 -- 172.21.15.6:0/111772353 shutdown_connections 2024-03-04T09:25:18.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:18.656+0000 7fde30c00640 1 -- 172.21.15.6:0/111772353 wait complete. 2024-03-04T09:25:18.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:18.658+0000 7fde30c00640 1 Processor -- start 2024-03-04T09:25:18.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:18.658+0000 7fde30c00640 1 -- start start 2024-03-04T09:25:18.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:18.659+0000 7fde30c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde10003b70 0x7fde2c14f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:18.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:18.659+0000 7fde30c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fde2c0cdb40 0x7fde2c0cada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:18.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:18.659+0000 7fde30c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fde2c160490 0x7fde2c160920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:18.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:18.659+0000 7fde30c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde2c0cfa60 con 0x7fde2c160490 2024-03-04T09:25:18.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:18.659+0000 7fde30c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde2c15e2c0 con 0x7fde2c0cdb40 2024-03-04T09:25:18.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:18.659+0000 7fde30c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde2c0cfbe0 con 0x7fde10003b70 2024-03-04T09:25:18.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:18.659+0000 7fde2be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fde2c160490 0x7fde2c160920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:18.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:18.660+0000 7fde2be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fde2c160490 0x7fde2c160920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41706/0 (socket says 172.21.15.6:41706) 2024-03-04T09:25:18.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:18.660+0000 7fde2be00640 1 -- 172.21.15.6:0/857265619 learned_addr learned my addr 172.21.15.6:0/857265619 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:25:18.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:18.660+0000 7fde2aa00640 1 --2- 172.21.15.6:0/857265619 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fde2c0cdb40 0x7fde2c0cada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:18.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:18.660+0000 7fde2b400640 1 --2- 172.21.15.6:0/857265619 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde10003b70 0x7fde2c14f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:18.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:18.660+0000 7fde2be00640 1 -- 172.21.15.6:0/857265619 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde10003b70 msgr2=0x7fde2c14f220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:18.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:18.660+0000 7fde2be00640 1 --2- 172.21.15.6:0/857265619 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde10003b70 0x7fde2c14f220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:18.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:18.660+0000 7fde2be00640 1 -- 172.21.15.6:0/857265619 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fde2c0cdb40 msgr2=0x7fde2c0cada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:18.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:18.660+0000 7fde2be00640 1 --2- 172.21.15.6:0/857265619 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fde2c0cdb40 0x7fde2c0cada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:18.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:18.660+0000 7fde2be00640 1 -- 172.21.15.6:0/857265619 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde2c0cb710 con 0x7fde2c160490 2024-03-04T09:25:18.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:18.660+0000 7fde2aa00640 1 --2- 172.21.15.6:0/857265619 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fde2c0cdb40 0x7fde2c0cada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:25:18.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:18.660+0000 7fde2be00640 1 --2- 172.21.15.6:0/857265619 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fde2c160490 0x7fde2c160920 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fde2c06b2f0 tx=0x7fde24010c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:18.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:18.660+0000 7fde23400640 1 -- 172.21.15.6:0/857265619 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde2400f040 con 0x7fde2c160490 2024-03-04T09:25:18.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:18.661+0000 7fde23400640 1 -- 172.21.15.6:0/857265619 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde2401c070 con 0x7fde2c160490 2024-03-04T09:25:18.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:18.661+0000 7fde23400640 1 -- 172.21.15.6:0/857265619 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde2400eb00 con 0x7fde2c160490 2024-03-04T09:25:18.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:18.661+0000 7fde30c00640 1 -- 172.21.15.6:0/857265619 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde2c0cb9f0 con 0x7fde2c160490 2024-03-04T09:25:18.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:18.661+0000 7fde30c00640 1 -- 172.21.15.6:0/857265619 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde2c0cbfb0 con 0x7fde2c160490 2024-03-04T09:25:18.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:18.662+0000 7fde30c00640 1 -- 172.21.15.6:0/857265619 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde2c04f330 con 0x7fde2c160490 2024-03-04T09:25:18.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:18.663+0000 7fde23400640 1 -- 172.21.15.6:0/857265619 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fde2400eca0 con 0x7fde2c160490 2024-03-04T09:25:18.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:18.664+0000 7fde23400640 1 --2- 172.21.15.6:0/857265619 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fde0806b4d0 0x7fde0806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:18.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:18.664+0000 7fde23400640 1 -- 172.21.15.6:0/857265619 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5609+0+0 (secure 0 0 0) 0x7fde24089e20 con 0x7fde2c160490 2024-03-04T09:25:18.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:18.665+0000 7fde2b400640 1 --2- 172.21.15.6:0/857265619 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fde0806b4d0 0x7fde0806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:18.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:18.666+0000 7fde2b400640 1 --2- 172.21.15.6:0/857265619 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fde0806b4d0 0x7fde0806d990 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fde14034000 tx=0x7fde14031000 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:18.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:18.669+0000 7fde23400640 1 -- 172.21.15.6:0/857265619 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fde24026020 con 0x7fde2c160490 2024-03-04T09:25:18.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:18.910+0000 7fde30c00640 1 -- 172.21.15.6:0/857265619 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fde2c08d7d0 con 0x7fde2c160490 2024-03-04T09:25:19.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:19.446+0000 7fde23400640 1 -- 172.21.15.6:0/857265619 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) v1 ==== 172+0+0 (secure 0 0 0) 0x7fde24058c60 con 0x7fde2c160490 2024-03-04T09:25:19.451 INFO:teuthology.orchestra.run.smithi006.stderr:new fs with metadata pool 2 and data pool 3 2024-03-04T09:25:19.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:19.457+0000 7fde30c00640 1 -- 172.21.15.6:0/857265619 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fde0806b4d0 msgr2=0x7fde0806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:19.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:19.457+0000 7fde30c00640 1 --2- 172.21.15.6:0/857265619 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fde0806b4d0 0x7fde0806d990 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fde14034000 tx=0x7fde14031000 comp rx=0 tx=0).stop 2024-03-04T09:25:19.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:19.457+0000 7fde30c00640 1 -- 172.21.15.6:0/857265619 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fde2c160490 msgr2=0x7fde2c160920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:19.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:19.457+0000 7fde30c00640 1 --2- 172.21.15.6:0/857265619 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fde2c160490 0x7fde2c160920 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fde2c06b2f0 tx=0x7fde24010c20 comp rx=0 tx=0).stop 2024-03-04T09:25:19.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:19.460+0000 7fde30c00640 1 -- 172.21.15.6:0/857265619 shutdown_connections 2024-03-04T09:25:19.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:19.460+0000 7fde30c00640 1 --2- 172.21.15.6:0/857265619 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fde2c160490 0x7fde2c160920 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:19.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:19.460+0000 7fde30c00640 1 --2- 172.21.15.6:0/857265619 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fde2c0cdb40 0x7fde2c0cada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:19.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:19.460+0000 7fde30c00640 1 --2- 172.21.15.6:0/857265619 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fde0806b4d0 0x7fde0806d990 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:19.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:19.460+0000 7fde30c00640 1 --2- 172.21.15.6:0/857265619 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde10003b70 0x7fde2c14f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:19.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:19.460+0000 7fde30c00640 1 -- 172.21.15.6:0/857265619 >> 172.21.15.6:0/857265619 conn(0x7fde2c0d3cd0 msgr2=0x7fde2c0d0110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:19.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:19.460+0000 7fde30c00640 1 -- 172.21.15.6:0/857265619 shutdown_connections 2024-03-04T09:25:19.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:19.460+0000 7fde30c00640 1 -- 172.21.15.6:0/857265619 wait complete. 2024-03-04T09:25:19.489 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T09:25:19.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:19.669+0000 7f5267c00640 1 Processor -- start 2024-03-04T09:25:19.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:19.669+0000 7f5267c00640 1 -- start start 2024-03-04T09:25:19.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:19.671+0000 7f5267c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f52600af580 0x7f526015c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:19.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:19.671+0000 7f5267c00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f52600cdc20 0x7f52600b2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:19.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:19.671+0000 7f5267c00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f526015cc40 con 0x7f52600af1a0 2024-03-04T09:25:19.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:19.671+0000 7f5267c00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f526015cf20 con 0x7f52600af580 2024-03-04T09:25:19.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:19.671+0000 7f5267c00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f526015d200 con 0x7f52600cdc20 2024-03-04T09:25:19.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:19.671+0000 7f525fe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f52600af580 0x7f526015c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:19.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:19.671+0000 7f525fe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f52600af580 0x7f526015c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:44952/0 (socket says 172.21.15.6:44952) 2024-03-04T09:25:19.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:19.671+0000 7f525fe00640 1 -- 172.21.15.6:0/425107797 learned_addr learned my addr 172.21.15.6:0/425107797 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:25:19.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:19.671+0000 7f525fe00640 1 -- 172.21.15.6:0/425107797 >> v2:172.21.15.146:3300/0 conn(0x7f52600cdc20 msgr2=0x7f52600b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:19.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:19.671+0000 7f525fe00640 1 --2- 172.21.15.6:0/425107797 >> v2:172.21.15.146:3300/0 conn(0x7f52600cdc20 0x7f52600b2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:19.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:19.671+0000 7f525fe00640 1 -- 172.21.15.6:0/425107797 >> v1:172.21.15.146:6789/0 conn(0x7f52600af1a0 legacy=0x7f52600b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:19.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:19.672+0000 7f525fe00640 1 -- 172.21.15.6:0/425107797 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f526015d4e0 con 0x7f52600af580 2024-03-04T09:25:19.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:19.673+0000 7f525fe00640 1 --2- 172.21.15.6:0/425107797 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f52600af580 0x7f526015c600 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f526006c0c0 tx=0x7f524c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f899288dbc0a342f server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:19.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:19.673+0000 7f525f400640 1 -- 172.21.15.6:0/425107797 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f524c02e070 con 0x7f52600af580 2024-03-04T09:25:19.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:19.673+0000 7f525f400640 1 -- 172.21.15.6:0/425107797 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f524c037070 con 0x7f52600af580 2024-03-04T09:25:19.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:19.673+0000 7f525f400640 1 -- 172.21.15.6:0/425107797 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f524c031a00 con 0x7f52600af580 2024-03-04T09:25:19.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:19.674+0000 7f5267c00640 1 -- 172.21.15.6:0/425107797 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f52600af580 msgr2=0x7f526015c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:19.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:19.674+0000 7f5267c00640 1 --2- 172.21.15.6:0/425107797 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f52600af580 0x7f526015c600 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f526006c0c0 tx=0x7f524c02b880 comp rx=0 tx=0).stop 2024-03-04T09:25:19.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:19.674+0000 7f5267c00640 1 -- 172.21.15.6:0/425107797 shutdown_connections 2024-03-04T09:25:19.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:19.674+0000 7f5267c00640 1 --2- 172.21.15.6:0/425107797 >> v2:172.21.15.146:3300/0 conn(0x7f52600cdc20 0x7f52600b2dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:19.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:19.674+0000 7f5267c00640 1 --2- 172.21.15.6:0/425107797 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f52600af580 0x7f526015c600 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:19.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:19.674+0000 7f5267c00640 1 -- 172.21.15.6:0/425107797 >> 172.21.15.6:0/425107797 conn(0x7f526006afb0 msgr2=0x7f526006b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:19.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:19.674+0000 7f5267c00640 1 -- 172.21.15.6:0/425107797 shutdown_connections 2024-03-04T09:25:19.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:19.674+0000 7f5267c00640 1 -- 172.21.15.6:0/425107797 wait complete. 2024-03-04T09:25:19.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:19.676+0000 7f5267c00640 1 Processor -- start 2024-03-04T09:25:19.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:19.676+0000 7f5267c00640 1 -- start start 2024-03-04T09:25:19.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:19.677+0000 7f5267c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f52600af1a0 0x7f52601f2d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:19.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:19.677+0000 7f5267c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f52600af580 0x7f5260157ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:19.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:19.677+0000 7f5267c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f52600cdc20 0x7f52601f26b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:19.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:19.677+0000 7f5267c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52600b36a0 con 0x7f52600cdc20 2024-03-04T09:25:19.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:19.677+0000 7f5267c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52600b3520 con 0x7f52600af580 2024-03-04T09:25:19.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:19.677+0000 7f5267c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52600b3820 con 0x7f52600af1a0 2024-03-04T09:25:19.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:19.677+0000 7f5265c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f52600cdc20 0x7f52601f26b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:19.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:19.677+0000 7f5265c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f52600cdc20 0x7f52601f26b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:44964/0 (socket says 172.21.15.6:44964) 2024-03-04T09:25:19.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:19.677+0000 7f5265c00640 1 -- 172.21.15.6:0/2500101447 learned_addr learned my addr 172.21.15.6:0/2500101447 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:25:19.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:19.678+0000 7f5265200640 1 --2- 172.21.15.6:0/2500101447 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f52600af1a0 0x7f52601f2d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:19.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:19.678+0000 7f5265c00640 1 -- 172.21.15.6:0/2500101447 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f52600af1a0 msgr2=0x7f52601f2d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:19.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:19.678+0000 7f5265c00640 1 --2- 172.21.15.6:0/2500101447 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f52600af1a0 0x7f52601f2d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:19.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:19.678+0000 7f5265c00640 1 -- 172.21.15.6:0/2500101447 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f52600af580 msgr2=0x7f5260157ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:19.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:19.678+0000 7f5265c00640 1 --2- 172.21.15.6:0/2500101447 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f52600af580 0x7f5260157ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:19.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:19.678+0000 7f5265c00640 1 -- 172.21.15.6:0/2500101447 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52601583e0 con 0x7f52600cdc20 2024-03-04T09:25:19.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:19.678+0000 7f5265c00640 1 --2- 172.21.15.6:0/2500101447 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f52600cdc20 0x7f52601f26b0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f525800b9a0 tx=0x7f525800be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:19.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:19.678+0000 7f525cc00640 1 -- 172.21.15.6:0/2500101447 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5258019070 con 0x7f52600cdc20 2024-03-04T09:25:19.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:19.678+0000 7f525cc00640 1 -- 172.21.15.6:0/2500101447 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f525800cbc0 con 0x7f52600cdc20 2024-03-04T09:25:19.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:19.678+0000 7f525cc00640 1 -- 172.21.15.6:0/2500101447 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5258014440 con 0x7f52600cdc20 2024-03-04T09:25:19.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:19.678+0000 7f5267c00640 1 -- 172.21.15.6:0/2500101447 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52601586c0 con 0x7f52600cdc20 2024-03-04T09:25:19.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:19.679+0000 7f5267c00640 1 -- 172.21.15.6:0/2500101447 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5260158c80 con 0x7f52600cdc20 2024-03-04T09:25:19.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:19.680+0000 7f5267c00640 1 -- 172.21.15.6:0/2500101447 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5260069510 con 0x7f52600cdc20 2024-03-04T09:25:19.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:19.681+0000 7f525cc00640 1 -- 172.21.15.6:0/2500101447 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f525801c070 con 0x7f52600cdc20 2024-03-04T09:25:19.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:19.681+0000 7f525cc00640 1 --2- 172.21.15.6:0/2500101447 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f524006b4d0 0x7f524006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:19.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:19.681+0000 7f525cc00640 1 -- 172.21.15.6:0/2500101447 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5258089fc0 con 0x7f52600cdc20 2024-03-04T09:25:19.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:19.682+0000 7f5265200640 1 --2- 172.21.15.6:0/2500101447 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f524006b4d0 0x7f524006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:19.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:19.683+0000 7f5265200640 1 --2- 172.21.15.6:0/2500101447 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f524006b4d0 0x7f524006d990 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f5254007fd0 tx=0x7f5254009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:19.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:19.686+0000 7f525cc00640 1 -- 172.21.15.6:0/2500101447 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5258058da0 con 0x7f52600cdc20 2024-03-04T09:25:19.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:19.885+0000 7f5267c00640 1 -- 172.21.15.6:0/2500101447 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f526004f3a0 con 0x7f52600cdc20 2024-03-04T09:25:19.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:19.887+0000 7f525cc00640 1 -- 172.21.15.6:0/2500101447 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) v1 ==== 74+0+16570 (secure 0 0 0) 0x7f52580586a0 con 0x7f52600cdc20 2024-03-04T09:25:19.885 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:25:19.885 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":19,"fsid":"edce0578-5ae4-4402-b45a-552bdef84d7c","created":"2024-03-04T09:24:54.523780+0000","modified":"2024-03-04T09:25:18.911920+0000","last_up_change":"2024-03-04T09:25:07.089049+0000","last_in_change":"2024-03-04T09:24:57.982077+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T09:25:07.935198+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-04T09:25:17.118573+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-04T09:25:17.930536+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a96574f-81f1-4bc6-8463-757520b84a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6825","nonce":1364449401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6827","nonce":1364449401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6831","nonce":1364449401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6829","nonce":1364449401}]},"public_addr":"172.21.15.6:6825/1364449401","cluster_addr":"172.21.15.6:6827/1364449401","heartbeat_back_addr":"172.21.15.6:6831/1364449401","heartbeat_front_addr":"172.21.15.6:6829/1364449401","state":["exists","up"]},{"osd":1,"uuid":"7d79b9c1-6b87-476a-bc36-7db5ee64cd90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6809","nonce":2494103078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6811","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6813","nonce":2494103078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6821","nonce":2494103078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6817","nonce":2494103078}]},"public_addr":"172.21.15.6:6809/2494103078","cluster_addr":"172.21.15.6:6813/2494103078","heartbeat_back_addr":"172.21.15.6:6821/2494103078","heartbeat_front_addr":"172.21.15.6:6817/2494103078","state":["exists","up"]},{"osd":2,"uuid":"492331b9-4ee9-40b6-bacd-85d44cd6c45a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6812","nonce":1500066681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6816","nonce":1500066681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6824","nonce":1500066681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6820","nonce":1500066681}]},"public_addr":"172.21.15.6:6812/1500066681","cluster_addr":"172.21.15.6:6816/1500066681","heartbeat_back_addr":"172.21.15.6:6824/1500066681","heartbeat_front_addr":"172.21.15.6:6820/1500066681","state":["exists","up"]},{"osd":3,"uuid":"21bb2efe-9ba7-4bdc-bc31-180cd1102e21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6801","nonce":1558806123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6803","nonce":1558806123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6807","nonce":1558806123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6805","nonce":1558806123}]},"public_addr":"172.21.15.6:6801/1558806123","cluster_addr":"172.21.15.6:6803/1558806123","heartbeat_back_addr":"172.21.15.6:6807/1558806123","heartbeat_front_addr":"172.21.15.6:6805/1558806123","state":["exists","up"]},{"osd":4,"uuid":"6e2c4b71-19d9-4737-83d9-558705d8644a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6801","nonce":1526135780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6803","nonce":1526135780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6807","nonce":1526135780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6805","nonce":1526135780}]},"public_addr":"172.21.15.146:6801/1526135780","cluster_addr":"172.21.15.146:6803/1526135780","heartbeat_back_addr":"172.21.15.146:6807/1526135780","heartbeat_front_addr":"172.21.15.146:6805/1526135780","state":["exists","up"]},{"osd":5,"uuid":"fb69c525-f1be-4b7d-ab32-6c29b0d6fddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6809","nonce":460262507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6811","nonce":460262507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6815","nonce":460262507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6813","nonce":460262507}]},"public_addr":"172.21.15.146:6809/460262507","cluster_addr":"172.21.15.146:6811/460262507","heartbeat_back_addr":"172.21.15.146:6815/460262507","heartbeat_front_addr":"172.21.15.146:6813/460262507","state":["exists","up"]},{"osd":6,"uuid":"c32e7fed-370d-4401-8ce0-6a4335b0d168","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6823","nonce":1392897493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6827","nonce":1392897493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6831","nonce":1392897493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6829","nonce":1392897493}]},"public_addr":"172.21.15.146:6823/1392897493","cluster_addr":"172.21.15.146:6827/1392897493","heartbeat_back_addr":"172.21.15.146:6831/1392897493","heartbeat_front_addr":"172.21.15.146:6829/1392897493","state":["exists","up"]},{"osd":7,"uuid":"75a79b44-fa5f-426d-a7d3-71f21f78835c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6817","nonce":3268323156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6819","nonce":3268323156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6825","nonce":3268323156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6822","nonce":3268323156}]},"public_addr":"172.21.15.146:6817/3268323156","cluster_addr":"172.21.15.146:6819/3268323156","heartbeat_back_addr":"172.21.15.146:6825/3268323156","heartbeat_front_addr":"172.21.15.146:6822/3268323156","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.310437+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.295850+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.294674+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.526855+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.300023+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.595160+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.346046+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.558741+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T09:25:19.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:19.890+0000 7f5267c00640 1 -- 172.21.15.6:0/2500101447 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f524006b4d0 msgr2=0x7f524006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:19.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:19.890+0000 7f5267c00640 1 --2- 172.21.15.6:0/2500101447 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f524006b4d0 0x7f524006d990 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f5254007fd0 tx=0x7f5254009040 comp rx=0 tx=0).stop 2024-03-04T09:25:19.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:19.890+0000 7f5267c00640 1 -- 172.21.15.6:0/2500101447 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f52600cdc20 msgr2=0x7f52601f26b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:19.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:19.890+0000 7f5267c00640 1 --2- 172.21.15.6:0/2500101447 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f52600cdc20 0x7f52601f26b0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f525800b9a0 tx=0x7f525800be60 comp rx=0 tx=0).stop 2024-03-04T09:25:19.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:19.890+0000 7f5267c00640 1 -- 172.21.15.6:0/2500101447 shutdown_connections 2024-03-04T09:25:19.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:19.890+0000 7f5267c00640 1 --2- 172.21.15.6:0/2500101447 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f52600cdc20 0x7f52601f26b0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:19.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:19.890+0000 7f5267c00640 1 --2- 172.21.15.6:0/2500101447 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f52600af580 0x7f5260157ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:19.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:19.890+0000 7f5267c00640 1 --2- 172.21.15.6:0/2500101447 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f524006b4d0 0x7f524006d990 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:19.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:19.890+0000 7f5267c00640 1 --2- 172.21.15.6:0/2500101447 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f52600af1a0 0x7f52601f2d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:19.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:19.890+0000 7f5267c00640 1 -- 172.21.15.6:0/2500101447 >> 172.21.15.6:0/2500101447 conn(0x7f526006afb0 msgr2=0x7f526006b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:19.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:19.891+0000 7f5267c00640 1 -- 172.21.15.6:0/2500101447 shutdown_connections 2024-03-04T09:25:19.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:19.891+0000 7f5267c00640 1 -- 172.21.15.6:0/2500101447 wait complete. 2024-03-04T09:25:19.905 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T09:25:20.052 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T09:25:20.054+0000 7f9f6f200640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-04T09:25:20.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.123+0000 7fee5d000640 1 Processor -- start 2024-03-04T09:25:20.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.124+0000 7fee5d000640 1 -- start start 2024-03-04T09:25:20.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.125+0000 7fee5d000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fee580d4a10 0x7fee580d0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:20.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.125+0000 7fee5d000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fee580d0c80 0x7fee580b44d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:20.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.125+0000 7fee5d000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee580b3710 con 0x7fee5815e0b0 2024-03-04T09:25:20.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.125+0000 7fee5d000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee580b09e0 con 0x7fee580d4a10 2024-03-04T09:25:20.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.125+0000 7fee5d000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee580cfa20 con 0x7fee580d0c80 2024-03-04T09:25:20.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.126+0000 7fee57e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fee580d0c80 0x7fee580b44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:20.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.126+0000 7fee57e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fee580d0c80 0x7fee580b44d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.6:33030/0 (socket says 172.21.15.6:33030) 2024-03-04T09:25:20.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.126+0000 7fee57e00640 1 -- 172.21.15.6:0/713796341 learned_addr learned my addr 172.21.15.6:0/713796341 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:25:20.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.126+0000 7fee4fe00640 1 --2- 172.21.15.6:0/713796341 >> v2:172.21.15.146:3300/0 conn(0x7fee580d4a10 0x7fee580d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:20.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.126+0000 7fee57e00640 1 -- 172.21.15.6:0/713796341 >> v2:172.21.15.146:3300/0 conn(0x7fee580d4a10 msgr2=0x7fee580d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:20.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.126+0000 7fee57e00640 1 --2- 172.21.15.6:0/713796341 >> v2:172.21.15.146:3300/0 conn(0x7fee580d4a10 0x7fee580d0740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:20.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.126+0000 7fee57e00640 1 -- 172.21.15.6:0/713796341 >> v1:172.21.15.146:6789/0 conn(0x7fee5815e0b0 legacy=0x7fee580cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:20.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.126+0000 7fee57e00640 1 -- 172.21.15.6:0/713796341 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee580b39f0 con 0x7fee580d0c80 2024-03-04T09:25:20.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.128+0000 7fee57e00640 1 --2- 172.21.15.6:0/713796341 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fee580d0c80 0x7fee580b44d0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fee50009e50 tx=0x7fee5002bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9e1e228220612ef5 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:20.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.128+0000 7fee56a00640 1 -- 172.21.15.6:0/713796341 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee5002cb00 con 0x7fee580d0c80 2024-03-04T09:25:20.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.128+0000 7fee56a00640 1 -- 172.21.15.6:0/713796341 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee5002cca0 con 0x7fee580d0c80 2024-03-04T09:25:20.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.128+0000 7fee56a00640 1 -- 172.21.15.6:0/713796341 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee50034610 con 0x7fee580d0c80 2024-03-04T09:25:20.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.129+0000 7fee5d000640 1 -- 172.21.15.6:0/713796341 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fee580d0c80 msgr2=0x7fee580b44d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:20.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.129+0000 7fee5d000640 1 --2- 172.21.15.6:0/713796341 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fee580d0c80 0x7fee580b44d0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fee50009e50 tx=0x7fee5002bf60 comp rx=0 tx=0).stop 2024-03-04T09:25:20.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.129+0000 7fee5d000640 1 -- 172.21.15.6:0/713796341 shutdown_connections 2024-03-04T09:25:20.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.129+0000 7fee5d000640 1 --2- 172.21.15.6:0/713796341 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fee580d0c80 0x7fee580b44d0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:20.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.129+0000 7fee5d000640 1 --2- 172.21.15.6:0/713796341 >> v2:172.21.15.146:3300/0 conn(0x7fee580d4a10 0x7fee580d0740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:20.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.129+0000 7fee5d000640 1 -- 172.21.15.6:0/713796341 >> 172.21.15.6:0/713796341 conn(0x7fee580d3d30 msgr2=0x7fee580d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:20.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.129+0000 7fee5d000640 1 -- 172.21.15.6:0/713796341 shutdown_connections 2024-03-04T09:25:20.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.129+0000 7fee5d000640 1 -- 172.21.15.6:0/713796341 wait complete. 2024-03-04T09:25:20.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.131+0000 7fee5d000640 1 Processor -- start 2024-03-04T09:25:20.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.131+0000 7fee5d000640 1 -- start start 2024-03-04T09:25:20.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.132+0000 7fee5d000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fee580d0c80 0x7fee581521b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:20.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.132+0000 7fee5d000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fee580d4a10 0x7fee581f3c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:20.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.132+0000 7fee5d000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fee5815e0b0 0x7fee5815ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:20.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.132+0000 7fee5d000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee5815e490 con 0x7fee580d0c80 2024-03-04T09:25:20.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.132+0000 7fee5d000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee580b3fb0 con 0x7fee5815e0b0 2024-03-04T09:25:20.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.132+0000 7fee5d000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee580cfc60 con 0x7fee580d4a10 2024-03-04T09:25:20.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.132+0000 7fee57400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fee580d0c80 0x7fee581521b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:20.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.132+0000 7fee57400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fee580d0c80 0x7fee581521b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:44974/0 (socket says 172.21.15.6:44974) 2024-03-04T09:25:20.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.132+0000 7fee57400640 1 -- 172.21.15.6:0/4152596763 learned_addr learned my addr 172.21.15.6:0/4152596763 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:25:20.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.133+0000 7fee4fe00640 1 --2- 172.21.15.6:0/4152596763 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fee580d4a10 0x7fee581f3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:20.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.133+0000 7fee57e00640 1 --2- 172.21.15.6:0/4152596763 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fee5815e0b0 0x7fee5815ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:20.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.133+0000 7fee57400640 1 -- 172.21.15.6:0/4152596763 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fee580d4a10 msgr2=0x7fee581f3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:20.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.133+0000 7fee57400640 1 --2- 172.21.15.6:0/4152596763 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fee580d4a10 0x7fee581f3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:20.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.133+0000 7fee57400640 1 -- 172.21.15.6:0/4152596763 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fee5815e0b0 msgr2=0x7fee5815ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:20.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.133+0000 7fee57400640 1 --2- 172.21.15.6:0/4152596763 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fee5815e0b0 0x7fee5815ee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:20.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.133+0000 7fee57400640 1 -- 172.21.15.6:0/4152596763 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee580b7ed0 con 0x7fee580d0c80 2024-03-04T09:25:20.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.133+0000 7fee57e00640 1 --2- 172.21.15.6:0/4152596763 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fee5815e0b0 0x7fee5815ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:25:20.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.133+0000 7fee57400640 1 --2- 172.21.15.6:0/4152596763 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fee580d0c80 0x7fee581521b0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fee4800c400 tx=0x7fee4800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:20.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.133+0000 7fee4f400640 1 -- 172.21.15.6:0/4152596763 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee48017070 con 0x7fee580d0c80 2024-03-04T09:25:20.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.133+0000 7fee4f400640 1 -- 172.21.15.6:0/4152596763 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee480052d0 con 0x7fee580d0c80 2024-03-04T09:25:20.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.133+0000 7fee4f400640 1 -- 172.21.15.6:0/4152596763 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee48003830 con 0x7fee580d0c80 2024-03-04T09:25:20.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.133+0000 7fee5d000640 1 -- 172.21.15.6:0/4152596763 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee580b81b0 con 0x7fee580d0c80 2024-03-04T09:25:20.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.134+0000 7fee5d000640 1 -- 172.21.15.6:0/4152596763 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee580b8770 con 0x7fee580d0c80 2024-03-04T09:25:20.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.135+0000 7fee5d000640 1 -- 172.21.15.6:0/4152596763 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee5804f330 con 0x7fee580d0c80 2024-03-04T09:25:20.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.136+0000 7fee4f400640 1 -- 172.21.15.6:0/4152596763 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fee4801a070 con 0x7fee580d0c80 2024-03-04T09:25:20.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.136+0000 7fee4f400640 1 --2- 172.21.15.6:0/4152596763 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fee3406b4d0 0x7fee3406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:20.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.137+0000 7fee4f400640 1 -- 172.21.15.6:0/4152596763 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fee48089570 con 0x7fee580d0c80 2024-03-04T09:25:20.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.137+0000 7fee4fe00640 1 --2- 172.21.15.6:0/4152596763 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fee3406b4d0 0x7fee3406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:20.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.138+0000 7fee4fe00640 1 --2- 172.21.15.6:0/4152596763 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fee3406b4d0 0x7fee3406d990 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fee5815bdf0 tx=0x7fee40008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:20.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.141+0000 7fee4f400640 1 -- 172.21.15.6:0/4152596763 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fee480583e0 con 0x7fee580d0c80 2024-03-04T09:25:20.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.340+0000 7fee5d000640 1 -- 172.21.15.6:0/4152596763 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fee580b1720 con 0x7fee580d0c80 2024-03-04T09:25:20.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.342+0000 7fee4f400640 1 -- 172.21.15.6:0/4152596763 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) v1 ==== 74+0+16570 (secure 0 0 0) 0x7fee48087e20 con 0x7fee580d0c80 2024-03-04T09:25:20.339 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:25:20.339 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":19,"fsid":"edce0578-5ae4-4402-b45a-552bdef84d7c","created":"2024-03-04T09:24:54.523780+0000","modified":"2024-03-04T09:25:18.911920+0000","last_up_change":"2024-03-04T09:25:07.089049+0000","last_in_change":"2024-03-04T09:24:57.982077+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T09:25:07.935198+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-04T09:25:17.118573+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-04T09:25:17.930536+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a96574f-81f1-4bc6-8463-757520b84a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6825","nonce":1364449401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6827","nonce":1364449401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6831","nonce":1364449401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6829","nonce":1364449401}]},"public_addr":"172.21.15.6:6825/1364449401","cluster_addr":"172.21.15.6:6827/1364449401","heartbeat_back_addr":"172.21.15.6:6831/1364449401","heartbeat_front_addr":"172.21.15.6:6829/1364449401","state":["exists","up"]},{"osd":1,"uuid":"7d79b9c1-6b87-476a-bc36-7db5ee64cd90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6809","nonce":2494103078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6811","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6813","nonce":2494103078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6821","nonce":2494103078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6817","nonce":2494103078}]},"public_addr":"172.21.15.6:6809/2494103078","cluster_addr":"172.21.15.6:6813/2494103078","heartbeat_back_addr":"172.21.15.6:6821/2494103078","heartbeat_front_addr":"172.21.15.6:6817/2494103078","state":["exists","up"]},{"osd":2,"uuid":"492331b9-4ee9-40b6-bacd-85d44cd6c45a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6812","nonce":1500066681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6816","nonce":1500066681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6824","nonce":1500066681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6820","nonce":1500066681}]},"public_addr":"172.21.15.6:6812/1500066681","cluster_addr":"172.21.15.6:6816/1500066681","heartbeat_back_addr":"172.21.15.6:6824/1500066681","heartbeat_front_addr":"172.21.15.6:6820/1500066681","state":["exists","up"]},{"osd":3,"uuid":"21bb2efe-9ba7-4bdc-bc31-180cd1102e21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6801","nonce":1558806123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6803","nonce":1558806123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6807","nonce":1558806123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6805","nonce":1558806123}]},"public_addr":"172.21.15.6:6801/1558806123","cluster_addr":"172.21.15.6:6803/1558806123","heartbeat_back_addr":"172.21.15.6:6807/1558806123","heartbeat_front_addr":"172.21.15.6:6805/1558806123","state":["exists","up"]},{"osd":4,"uuid":"6e2c4b71-19d9-4737-83d9-558705d8644a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6801","nonce":1526135780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6803","nonce":1526135780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6807","nonce":1526135780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6805","nonce":1526135780}]},"public_addr":"172.21.15.146:6801/1526135780","cluster_addr":"172.21.15.146:6803/1526135780","heartbeat_back_addr":"172.21.15.146:6807/1526135780","heartbeat_front_addr":"172.21.15.146:6805/1526135780","state":["exists","up"]},{"osd":5,"uuid":"fb69c525-f1be-4b7d-ab32-6c29b0d6fddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6809","nonce":460262507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6811","nonce":460262507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6815","nonce":460262507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6813","nonce":460262507}]},"public_addr":"172.21.15.146:6809/460262507","cluster_addr":"172.21.15.146:6811/460262507","heartbeat_back_addr":"172.21.15.146:6815/460262507","heartbeat_front_addr":"172.21.15.146:6813/460262507","state":["exists","up"]},{"osd":6,"uuid":"c32e7fed-370d-4401-8ce0-6a4335b0d168","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6823","nonce":1392897493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6827","nonce":1392897493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6831","nonce":1392897493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6829","nonce":1392897493}]},"public_addr":"172.21.15.146:6823/1392897493","cluster_addr":"172.21.15.146:6827/1392897493","heartbeat_back_addr":"172.21.15.146:6831/1392897493","heartbeat_front_addr":"172.21.15.146:6829/1392897493","state":["exists","up"]},{"osd":7,"uuid":"75a79b44-fa5f-426d-a7d3-71f21f78835c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6817","nonce":3268323156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6819","nonce":3268323156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6825","nonce":3268323156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6822","nonce":3268323156}]},"public_addr":"172.21.15.146:6817/3268323156","cluster_addr":"172.21.15.146:6819/3268323156","heartbeat_back_addr":"172.21.15.146:6825/3268323156","heartbeat_front_addr":"172.21.15.146:6822/3268323156","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.310437+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.295850+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.294674+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.526855+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.300023+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.595160+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.346046+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.558741+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T09:25:20.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.344+0000 7fee5d000640 1 -- 172.21.15.6:0/4152596763 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fee3406b4d0 msgr2=0x7fee3406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:20.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.344+0000 7fee5d000640 1 --2- 172.21.15.6:0/4152596763 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fee3406b4d0 0x7fee3406d990 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fee5815bdf0 tx=0x7fee40008040 comp rx=0 tx=0).stop 2024-03-04T09:25:20.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.345+0000 7fee5d000640 1 -- 172.21.15.6:0/4152596763 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fee580d0c80 msgr2=0x7fee581521b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:20.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.345+0000 7fee5d000640 1 --2- 172.21.15.6:0/4152596763 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fee580d0c80 0x7fee581521b0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fee4800c400 tx=0x7fee4800c8c0 comp rx=0 tx=0).stop 2024-03-04T09:25:20.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.345+0000 7fee5d000640 1 -- 172.21.15.6:0/4152596763 shutdown_connections 2024-03-04T09:25:20.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.345+0000 7fee5d000640 1 --2- 172.21.15.6:0/4152596763 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fee5815e0b0 0x7fee5815ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:20.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.345+0000 7fee5d000640 1 --2- 172.21.15.6:0/4152596763 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fee3406b4d0 0x7fee3406d990 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:20.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.345+0000 7fee5d000640 1 --2- 172.21.15.6:0/4152596763 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fee580d4a10 0x7fee581f3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:20.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.345+0000 7fee5d000640 1 --2- 172.21.15.6:0/4152596763 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fee580d0c80 0x7fee581521b0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:20.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.345+0000 7fee5d000640 1 -- 172.21.15.6:0/4152596763 >> 172.21.15.6:0/4152596763 conn(0x7fee580d3d30 msgr2=0x7fee580b0b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:20.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.345+0000 7fee5d000640 1 -- 172.21.15.6:0/4152596763 shutdown_connections 2024-03-04T09:25:20.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.345+0000 7fee5d000640 1 -- 172.21.15.6:0/4152596763 wait complete. 2024-03-04T09:25:20.362 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-04T09:25:20.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.593+0000 7f3362200640 1 Processor -- start 2024-03-04T09:25:20.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.594+0000 7f3362200640 1 -- start start 2024-03-04T09:25:20.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.595+0000 7f3362200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f335c162280 0x7f335c165d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:20.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.595+0000 7f3362200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f335c166cb0 con 0x7f335c0b4a90 2024-03-04T09:25:20.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.595+0000 7f3362200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f335c166410 con 0x7f335c0b1b40 2024-03-04T09:25:20.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.595+0000 7f3362200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f335c1666f0 con 0x7f335c162280 2024-03-04T09:25:20.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.596+0000 7f335be00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f335c0b4a90 0x7f335c0cc430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:60156/0 (socket says 172.21.15.6:60156) 2024-03-04T09:25:20.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.596+0000 7f335be00640 1 -- 172.21.15.6:0/1402658337 learned_addr learned my addr 172.21.15.6:0/1402658337 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:25:20.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.596+0000 7f3360e00640 1 --2- 172.21.15.6:0/1402658337 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f335c162280 0x7f335c165d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:20.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.596+0000 7f335b400640 1 -- 172.21.15.6:0/1402658337 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2454977730 0 0) 0x7f335c166cb0 con 0x7f335c0b4a90 2024-03-04T09:25:20.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.596+0000 7f335b400640 1 -- 172.21.15.6:0/1402658337 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3344003520 con 0x7f335c0b4a90 2024-03-04T09:25:20.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.597+0000 7f335b400640 1 -- 172.21.15.6:0/1402658337 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3688173061 0 0) 0x7f3344003520 con 0x7f335c0b4a90 2024-03-04T09:25:20.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.597+0000 7f335b400640 1 -- 172.21.15.6:0/1402658337 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f335c162280 msgr2=0x7f335c165d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:20.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.597+0000 7f335b400640 1 --2- 172.21.15.6:0/1402658337 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f335c162280 0x7f335c165d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:20.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.597+0000 7f335b400640 1 -- 172.21.15.6:0/1402658337 >> v1:172.21.15.146:6789/0 conn(0x7f335c0b1b40 legacy=0x7f335c071560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:20.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.597+0000 7f335b400640 1 -- 172.21.15.6:0/1402658337 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f335c1669d0 con 0x7f335c0b4a90 2024-03-04T09:25:20.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.597+0000 7f335b400640 1 -- 172.21.15.6:0/1402658337 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7f3348001670 con 0x7f335c0b4a90 2024-03-04T09:25:20.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.597+0000 7f335b400640 1 -- 172.21.15.6:0/1402658337 >> v1:172.21.15.6:6789/0 conn(0x7f335c0b4a90 legacy=0x7f335c0cc430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:20.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.597+0000 7f335b400640 1 --2- 172.21.15.6:0/1402658337 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f335c162280 0x7f33440039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:20.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.597+0000 7f335b400640 1 --2- 172.21.15.6:0/1402658337 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3344003fb0 0x7f3344004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:20.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.597+0000 7f335b400640 1 --2- 172.21.15.6:0/1402658337 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3344004d40 0x7f33440051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:20.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.597+0000 7f335b400640 1 -- 172.21.15.6:0/1402658337 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f335c1666f0 con 0x7f3344003fb0 2024-03-04T09:25:20.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.597+0000 7f335b400640 1 -- 172.21.15.6:0/1402658337 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f334400bbb0 con 0x7f3344004d40 2024-03-04T09:25:20.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.597+0000 7f335b400640 1 -- 172.21.15.6:0/1402658337 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f334400bdb0 con 0x7f335c162280 2024-03-04T09:25:20.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.597+0000 7f335be00640 1 --2- 172.21.15.6:0/1402658337 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3344003fb0 0x7f3344004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:20.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.597+0000 7f3360e00640 1 --2- 172.21.15.6:0/1402658337 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f335c162280 0x7f33440039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:20.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.598+0000 7f335be00640 1 -- 172.21.15.6:0/1402658337 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f335c162280 msgr2=0x7f33440039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:20.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.598+0000 7f335be00640 1 --2- 172.21.15.6:0/1402658337 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f335c162280 0x7f33440039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:20.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.598+0000 7f335be00640 1 -- 172.21.15.6:0/1402658337 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3344004d40 msgr2=0x7f33440051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:20.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.598+0000 7f3361800640 1 --2- 172.21.15.6:0/1402658337 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3344004d40 0x7f33440051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:20.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.598+0000 7f335be00640 1 --2- 172.21.15.6:0/1402658337 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3344004d40 0x7f33440051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:20.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.598+0000 7f335be00640 1 -- 172.21.15.6:0/1402658337 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3344003520 con 0x7f3344003fb0 2024-03-04T09:25:20.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.599+0000 7f335be00640 1 --2- 172.21.15.6:0/1402658337 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3344003fb0 0x7f3344004800 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f334800c0e0 tx=0x7f3344005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ca2e78303904a13 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:20.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.599+0000 7f335b400640 1 -- 172.21.15.6:0/1402658337 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3348007c20 con 0x7f3344003fb0 2024-03-04T09:25:20.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.599+0000 7f335b400640 1 -- 172.21.15.6:0/1402658337 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3348030070 con 0x7f3344003fb0 2024-03-04T09:25:20.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.599+0000 7f335b400640 1 -- 172.21.15.6:0/1402658337 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3348005d70 con 0x7f3344003fb0 2024-03-04T09:25:20.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.600+0000 7f3362200640 1 -- 172.21.15.6:0/1402658337 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3344003fb0 msgr2=0x7f3344004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:20.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.600+0000 7f3362200640 1 --2- 172.21.15.6:0/1402658337 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3344003fb0 0x7f3344004800 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f334800c0e0 tx=0x7f3344005830 comp rx=0 tx=0).stop 2024-03-04T09:25:20.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.600+0000 7f3361800640 1 -- 172.21.15.6:0/1402658337 reap_dead start 2024-03-04T09:25:20.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.600+0000 7f3362200640 1 -- 172.21.15.6:0/1402658337 shutdown_connections 2024-03-04T09:25:20.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.600+0000 7f3362200640 1 -- 172.21.15.6:0/1402658337 >> 172.21.15.6:0/1402658337 conn(0x7f335c06b3a0 msgr2=0x7f335c06b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:20.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.600+0000 7f3362200640 1 -- 172.21.15.6:0/1402658337 shutdown_connections 2024-03-04T09:25:20.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.600+0000 7f3362200640 1 -- 172.21.15.6:0/1402658337 wait complete. 2024-03-04T09:25:20.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.602+0000 7f3362200640 1 Processor -- start 2024-03-04T09:25:20.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.602+0000 7f3362200640 1 -- start start 2024-03-04T09:25:20.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.603+0000 7f3362200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3344003fb0 0x7f335c153520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:20.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.603+0000 7f3362200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f335c162660 0x7f335c06c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:20.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.603+0000 7f3362200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f335c162ae0 0x7f335c06cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:20.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.603+0000 7f3362200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f335c165ed0 con 0x7f335c162660 2024-03-04T09:25:20.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.603+0000 7f3362200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f335c165d50 con 0x7f3344003fb0 2024-03-04T09:25:20.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.603+0000 7f3362200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f335c166050 con 0x7f335c162ae0 2024-03-04T09:25:20.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.603+0000 7f335be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f335c162660 0x7f335c06c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:20.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.603+0000 7f335be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f335c162660 0x7f335c06c800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45000/0 (socket says 172.21.15.6:45000) 2024-03-04T09:25:20.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.603+0000 7f335be00640 1 -- 172.21.15.6:0/316160820 learned_addr learned my addr 172.21.15.6:0/316160820 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:25:20.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.603+0000 7f3360e00640 1 --2- 172.21.15.6:0/316160820 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3344003fb0 0x7f335c153520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:20.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.603+0000 7f3361800640 1 --2- 172.21.15.6:0/316160820 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f335c162ae0 0x7f335c06cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:20.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.604+0000 7f335be00640 1 -- 172.21.15.6:0/316160820 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f335c162ae0 msgr2=0x7f335c06cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:20.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.604+0000 7f335be00640 1 --2- 172.21.15.6:0/316160820 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f335c162ae0 0x7f335c06cd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:20.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.604+0000 7f335be00640 1 -- 172.21.15.6:0/316160820 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3344003fb0 msgr2=0x7f335c153520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:20.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.604+0000 7f335be00640 1 --2- 172.21.15.6:0/316160820 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3344003fb0 0x7f335c153520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:20.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.604+0000 7f335be00640 1 -- 172.21.15.6:0/316160820 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f335c163000 con 0x7f335c162660 2024-03-04T09:25:20.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.604+0000 7f335be00640 1 --2- 172.21.15.6:0/316160820 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f335c162660 0x7f335c06c800 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f334800c200 tx=0x7f335c1665e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:20.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.604+0000 7f3358c00640 1 -- 172.21.15.6:0/316160820 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3348004bf0 con 0x7f335c162660 2024-03-04T09:25:20.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.604+0000 7f3358c00640 1 -- 172.21.15.6:0/316160820 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3348000ed0 con 0x7f335c162660 2024-03-04T09:25:20.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.604+0000 7f3358c00640 1 -- 172.21.15.6:0/316160820 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3348007070 con 0x7f335c162660 2024-03-04T09:25:20.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.604+0000 7f3362200640 1 -- 172.21.15.6:0/316160820 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f335c1632e0 con 0x7f335c162660 2024-03-04T09:25:20.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.605+0000 7f3362200640 1 -- 172.21.15.6:0/316160820 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f335c1638a0 con 0x7f335c162660 2024-03-04T09:25:20.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.606+0000 7f3362200640 1 -- 172.21.15.6:0/316160820 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f335c04f330 con 0x7f335c162660 2024-03-04T09:25:20.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.607+0000 7f3358c00640 1 -- 172.21.15.6:0/316160820 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3348047020 con 0x7f335c162660 2024-03-04T09:25:20.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.607+0000 7f3358c00640 1 --2- 172.21.15.6:0/316160820 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f333c06b4d0 0x7f333c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:20.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.608+0000 7f3358c00640 1 -- 172.21.15.6:0/316160820 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f33480ab3d0 con 0x7f335c162660 2024-03-04T09:25:20.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.608+0000 7f3360e00640 1 --2- 172.21.15.6:0/316160820 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f333c06b4d0 0x7f333c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:20.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.609+0000 7f3360e00640 1 --2- 172.21.15.6:0/316160820 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f333c06b4d0 0x7f333c06d990 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f3350009a10 tx=0x7f3350008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:20.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.612+0000 7f3358c00640 1 -- 172.21.15.6:0/316160820 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f334807a240 con 0x7f335c162660 2024-03-04T09:25:20.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:20.851+0000 7f3362200640 1 -- 172.21.15.6:0/316160820 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f335c06e490 con 0x7f335c162660 2024-03-04T09:25:21.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:21.511+0000 7f3358c00640 1 -- 172.21.15.6:0/316160820 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v6) v1 ==== 118+0+0 (secure 0 0 0) 0x7f3348079b40 con 0x7f335c162660 2024-03-04T09:25:21.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:21.515+0000 7f3362200640 1 -- 172.21.15.6:0/316160820 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f333c06b4d0 msgr2=0x7f333c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:21.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:21.515+0000 7f3362200640 1 --2- 172.21.15.6:0/316160820 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f333c06b4d0 0x7f333c06d990 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f3350009a10 tx=0x7f3350008040 comp rx=0 tx=0).stop 2024-03-04T09:25:21.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:21.515+0000 7f3362200640 1 -- 172.21.15.6:0/316160820 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f335c162660 msgr2=0x7f335c06c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:21.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:21.515+0000 7f3362200640 1 --2- 172.21.15.6:0/316160820 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f335c162660 0x7f335c06c800 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f334800c200 tx=0x7f335c1665e0 comp rx=0 tx=0).stop 2024-03-04T09:25:21.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:21.515+0000 7f3362200640 1 -- 172.21.15.6:0/316160820 shutdown_connections 2024-03-04T09:25:21.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:21.515+0000 7f3362200640 1 --2- 172.21.15.6:0/316160820 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f333c06b4d0 0x7f333c06d990 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:21.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:21.515+0000 7f3362200640 1 --2- 172.21.15.6:0/316160820 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f335c162ae0 0x7f335c06cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:21.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:21.515+0000 7f3362200640 1 --2- 172.21.15.6:0/316160820 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f335c162660 0x7f335c06c800 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:21.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:21.515+0000 7f3362200640 1 --2- 172.21.15.6:0/316160820 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3344003fb0 0x7f335c153520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:21.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:21.515+0000 7f3362200640 1 -- 172.21.15.6:0/316160820 >> 172.21.15.6:0/316160820 conn(0x7f335c06b3a0 msgr2=0x7f335c0cefc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:21.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:21.515+0000 7f3362200640 1 -- 172.21.15.6:0/316160820 shutdown_connections 2024-03-04T09:25:21.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:21.515+0000 7f3362200640 1 -- 172.21.15.6:0/316160820 wait complete. 2024-03-04T09:25:21.534 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-04T09:25:21.534 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-04T09:25:21.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:21.758+0000 7f88baa00640 1 Processor -- start 2024-03-04T09:25:21.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:21.758+0000 7f88baa00640 1 -- start start 2024-03-04T09:25:21.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:21.759+0000 7f88baa00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f88b4072520 0x7f88b40cdfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:21.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:21.759+0000 7f88baa00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f88b4071e20 0x7f88b40ce4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:21.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:21.759+0000 7f88baa00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88b4072df0 con 0x7f88b415bf20 2024-03-04T09:25:21.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:21.759+0000 7f88baa00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88b40cdab0 con 0x7f88b4072520 2024-03-04T09:25:21.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:21.759+0000 7f88baa00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88b40b4d70 con 0x7f88b4071e20 2024-03-04T09:25:21.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:21.760+0000 7f88b8a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f88b4071e20 0x7f88b40ce4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:21.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:21.760+0000 7f88b8a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f88b4071e20 0x7f88b40ce4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.6:33090/0 (socket says 172.21.15.6:33090) 2024-03-04T09:25:21.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:21.760+0000 7f88b8a00640 1 -- 172.21.15.6:0/104931792 learned_addr learned my addr 172.21.15.6:0/104931792 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:25:21.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:21.760+0000 7f88b3400640 1 --2- 172.21.15.6:0/104931792 >> v2:172.21.15.146:3300/0 conn(0x7f88b4072520 0x7f88b40cdfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:21.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:21.761+0000 7f88b3400640 1 -- 172.21.15.6:0/104931792 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f88b4071e20 msgr2=0x7f88b40ce4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:21.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:21.761+0000 7f88b3400640 1 --2- 172.21.15.6:0/104931792 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f88b4071e20 0x7f88b40ce4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:21.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:21.761+0000 7f88b3400640 1 -- 172.21.15.6:0/104931792 >> v1:172.21.15.146:6789/0 conn(0x7f88b415bf20 legacy=0x7f88b40acd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:21.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:21.761+0000 7f88b3400640 1 -- 172.21.15.6:0/104931792 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88b40b5050 con 0x7f88b4072520 2024-03-04T09:25:21.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:21.761+0000 7f88b8a00640 1 --2- 172.21.15.6:0/104931792 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f88b4071e20 0x7f88b40ce4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T09:25:21.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:21.762+0000 7f88b3400640 1 --2- 172.21.15.6:0/104931792 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f88b4072520 0x7f88b40cdfb0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f889c0097a0 tx=0x7f889c02b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b6d1cd06b492a1be server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:21.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:21.762+0000 7f88b2a00640 1 -- 172.21.15.6:0/104931792 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f889c035020 con 0x7f88b4072520 2024-03-04T09:25:21.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:21.763+0000 7f88b2a00640 1 -- 172.21.15.6:0/104931792 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f889c031810 con 0x7f88b4072520 2024-03-04T09:25:21.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:21.763+0000 7f88b2a00640 1 -- 172.21.15.6:0/104931792 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f889c031be0 con 0x7f88b4072520 2024-03-04T09:25:21.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:21.763+0000 7f88baa00640 1 -- 172.21.15.6:0/104931792 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f88b4072520 msgr2=0x7f88b40cdfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:21.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:21.763+0000 7f88baa00640 1 --2- 172.21.15.6:0/104931792 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f88b4072520 0x7f88b40cdfb0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f889c0097a0 tx=0x7f889c02b8b0 comp rx=0 tx=0).stop 2024-03-04T09:25:21.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:21.763+0000 7f88baa00640 1 -- 172.21.15.6:0/104931792 shutdown_connections 2024-03-04T09:25:21.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:21.763+0000 7f88baa00640 1 --2- 172.21.15.6:0/104931792 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f88b4071e20 0x7f88b40ce4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:21.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:21.763+0000 7f88baa00640 1 --2- 172.21.15.6:0/104931792 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f88b4072520 0x7f88b40cdfb0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:21.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:21.763+0000 7f88baa00640 1 -- 172.21.15.6:0/104931792 >> 172.21.15.6:0/104931792 conn(0x7f88b406b3d0 msgr2=0x7f88b406b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:21.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:21.763+0000 7f88baa00640 1 -- 172.21.15.6:0/104931792 shutdown_connections 2024-03-04T09:25:21.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:21.763+0000 7f88baa00640 1 -- 172.21.15.6:0/104931792 wait complete. 2024-03-04T09:25:21.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:21.765+0000 7f88baa00640 1 Processor -- start 2024-03-04T09:25:21.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:21.765+0000 7f88baa00640 1 -- start start 2024-03-04T09:25:21.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:21.766+0000 7f88baa00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f88b4071e20 0x7f88b414b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:21.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:21.766+0000 7f88baa00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f88b4072520 0x7f88b409d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:21.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:21.766+0000 7f88baa00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f88b415bf20 0x7f88b415a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:21.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:21.766+0000 7f88baa00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88b40ad450 con 0x7f88b4071e20 2024-03-04T09:25:21.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:21.766+0000 7f88baa00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88b40b4990 con 0x7f88b415bf20 2024-03-04T09:25:21.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:21.766+0000 7f88baa00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88b414ba10 con 0x7f88b4072520 2024-03-04T09:25:21.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:21.766+0000 7f88b3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f88b4071e20 0x7f88b414b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:21.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:21.766+0000 7f88b3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f88b4071e20 0x7f88b414b4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45008/0 (socket says 172.21.15.6:45008) 2024-03-04T09:25:21.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:21.766+0000 7f88b3e00640 1 -- 172.21.15.6:0/2129126034 learned_addr learned my addr 172.21.15.6:0/2129126034 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:25:21.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:21.767+0000 7f88b8a00640 1 --2- 172.21.15.6:0/2129126034 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f88b415bf20 0x7f88b415a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:21.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:21.767+0000 7f88b3e00640 1 -- 172.21.15.6:0/2129126034 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f88b4072520 msgr2=0x7f88b409d2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:21.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:21.767+0000 7f88b3e00640 1 --2- 172.21.15.6:0/2129126034 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f88b4072520 0x7f88b409d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:21.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:21.767+0000 7f88b3e00640 1 -- 172.21.15.6:0/2129126034 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f88b415bf20 msgr2=0x7f88b415a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:21.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:21.767+0000 7f88b3e00640 1 --2- 172.21.15.6:0/2129126034 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f88b415bf20 0x7f88b415a850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:21.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:21.767+0000 7f88b3e00640 1 -- 172.21.15.6:0/2129126034 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88b41f56c0 con 0x7f88b4071e20 2024-03-04T09:25:21.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:21.767+0000 7f88b3e00640 1 --2- 172.21.15.6:0/2129126034 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f88b4071e20 0x7f88b414b4d0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f88a400c790 tx=0x7f88a400cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:21.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:21.767+0000 7f88abe00640 1 -- 172.21.15.6:0/2129126034 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88a4016020 con 0x7f88b4071e20 2024-03-04T09:25:21.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:21.767+0000 7f88abe00640 1 -- 172.21.15.6:0/2129126034 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88a4006d50 con 0x7f88b4071e20 2024-03-04T09:25:21.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:21.767+0000 7f88abe00640 1 -- 172.21.15.6:0/2129126034 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88a4003bb0 con 0x7f88b4071e20 2024-03-04T09:25:21.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:21.767+0000 7f88baa00640 1 -- 172.21.15.6:0/2129126034 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88b41f59a0 con 0x7f88b4071e20 2024-03-04T09:25:21.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:21.768+0000 7f88baa00640 1 -- 172.21.15.6:0/2129126034 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88b41f5f60 con 0x7f88b4071e20 2024-03-04T09:25:21.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:21.769+0000 7f88baa00640 1 -- 172.21.15.6:0/2129126034 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88b404f330 con 0x7f88b4071e20 2024-03-04T09:25:21.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:21.770+0000 7f88abe00640 1 -- 172.21.15.6:0/2129126034 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f88a400a850 con 0x7f88b4071e20 2024-03-04T09:25:21.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:21.771+0000 7f88abe00640 1 --2- 172.21.15.6:0/2129126034 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f889006b4d0 0x7f889006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:21.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:21.771+0000 7f88abe00640 1 -- 172.21.15.6:0/2129126034 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f88a4089f80 con 0x7f88b4071e20 2024-03-04T09:25:21.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:21.771+0000 7f88b3400640 1 --2- 172.21.15.6:0/2129126034 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f889006b4d0 0x7f889006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:21.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:21.772+0000 7f88b3400640 1 --2- 172.21.15.6:0/2129126034 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f889006b4d0 0x7f889006d990 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f889c036000 tx=0x7f889c030000 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:21.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:21.775+0000 7f88abe00640 1 -- 172.21.15.6:0/2129126034 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f88a4020020 con 0x7f88b4071e20 2024-03-04T09:25:22.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:22.014+0000 7f88baa00640 1 -- 172.21.15.6:0/2129126034 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f88b4150b90 con 0x7f88b4071e20 2024-03-04T09:25:22.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:22.522+0000 7f88abe00640 1 -- 172.21.15.6:0/2129126034 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v7) v1 ==== 122+0+0 (secure 0 0 0) 0x7f88a4019310 con 0x7f88b4071e20 2024-03-04T09:25:22.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:22.525+0000 7f88baa00640 1 -- 172.21.15.6:0/2129126034 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f889006b4d0 msgr2=0x7f889006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:22.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:22.525+0000 7f88baa00640 1 --2- 172.21.15.6:0/2129126034 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f889006b4d0 0x7f889006d990 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f889c036000 tx=0x7f889c030000 comp rx=0 tx=0).stop 2024-03-04T09:25:22.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:22.526+0000 7f88baa00640 1 -- 172.21.15.6:0/2129126034 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f88b4071e20 msgr2=0x7f88b414b4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:22.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:22.526+0000 7f88baa00640 1 --2- 172.21.15.6:0/2129126034 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f88b4071e20 0x7f88b414b4d0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f88a400c790 tx=0x7f88a400cc50 comp rx=0 tx=0).stop 2024-03-04T09:25:22.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:22.526+0000 7f88baa00640 1 -- 172.21.15.6:0/2129126034 shutdown_connections 2024-03-04T09:25:22.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:22.526+0000 7f88baa00640 1 --2- 172.21.15.6:0/2129126034 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f88b415bf20 0x7f88b415a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:22.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:22.526+0000 7f88baa00640 1 --2- 172.21.15.6:0/2129126034 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f889006b4d0 0x7f889006d990 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:22.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:22.526+0000 7f88baa00640 1 --2- 172.21.15.6:0/2129126034 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f88b4072520 0x7f88b409d2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:22.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:22.526+0000 7f88baa00640 1 --2- 172.21.15.6:0/2129126034 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f88b4071e20 0x7f88b414b4d0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:22.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:22.526+0000 7f88baa00640 1 -- 172.21.15.6:0/2129126034 >> 172.21.15.6:0/2129126034 conn(0x7f88b406b3d0 msgr2=0x7f88b40cdeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:22.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:22.526+0000 7f88baa00640 1 -- 172.21.15.6:0/2129126034 shutdown_connections 2024-03-04T09:25:22.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:22.526+0000 7f88baa00640 1 -- 172.21.15.6:0/2129126034 wait complete. 2024-03-04T09:25:22.544 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-04T09:25:22.544 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:25:22.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:22.773+0000 7f8769c00640 1 Processor -- start 2024-03-04T09:25:22.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:22.773+0000 7f8769c00640 1 -- start start 2024-03-04T09:25:22.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:22.775+0000 7f8769c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f87640af1a0 0x7f876415c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:22.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:22.775+0000 7f8769c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f87640cdc20 0x7f87640b2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:22.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:22.775+0000 7f8769c00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f876415cc40 con 0x7f87640af580 2024-03-04T09:25:22.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:22.775+0000 7f8769c00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f876415cf20 con 0x7f87640cdc20 2024-03-04T09:25:22.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:22.775+0000 7f8769c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f876415d200 con 0x7f87640af1a0 2024-03-04T09:25:22.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:22.775+0000 7f8762000640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f87640af580 0x7f87640b4c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:60158/0 (socket says 172.21.15.6:60158) 2024-03-04T09:25:22.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:22.775+0000 7f8762000640 1 -- 172.21.15.6:0/3580037235 learned_addr learned my addr 172.21.15.6:0/3580037235 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:25:22.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:22.776+0000 7f8763400640 1 --2- 172.21.15.6:0/3580037235 >> v2:172.21.15.6:3300/0 conn(0x7f87640cdc20 0x7f87640b2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:22.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:22.776+0000 7f8762a00640 1 --2- 172.21.15.6:0/3580037235 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f87640af1a0 0x7f876415c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:22.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:22.776+0000 7f8761600640 1 -- 172.21.15.6:0/3580037235 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2819860647 0 0) 0x7f876415cc40 con 0x7f87640af580 2024-03-04T09:25:22.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:22.776+0000 7f8761600640 1 -- 172.21.15.6:0/3580037235 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8748003520 con 0x7f87640af580 2024-03-04T09:25:22.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:22.776+0000 7f8763400640 1 -- 172.21.15.6:0/3580037235 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f87640af1a0 msgr2=0x7f876415c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:22.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:22.776+0000 7f8763400640 1 --2- 172.21.15.6:0/3580037235 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f87640af1a0 0x7f876415c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:22.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:22.776+0000 7f8763400640 1 -- 172.21.15.6:0/3580037235 >> v1:172.21.15.6:6789/0 conn(0x7f87640af580 legacy=0x7f87640b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:22.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:22.776+0000 7f8763400640 1 -- 172.21.15.6:0/3580037235 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f876415d4e0 con 0x7f87640cdc20 2024-03-04T09:25:22.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:22.778+0000 7f8763400640 1 --2- 172.21.15.6:0/3580037235 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f87640cdc20 0x7f87640b2dd0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f875c0097a0 tx=0x7f875c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53643ba8e58215fd server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:22.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:22.778+0000 7f8761600640 1 -- 172.21.15.6:0/3580037235 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f875c0315a0 con 0x7f87640cdc20 2024-03-04T09:25:22.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:22.778+0000 7f8761600640 1 -- 172.21.15.6:0/3580037235 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f875c037070 con 0x7f87640cdc20 2024-03-04T09:25:22.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:22.778+0000 7f8761600640 1 -- 172.21.15.6:0/3580037235 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f875c031940 con 0x7f87640cdc20 2024-03-04T09:25:22.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:22.778+0000 7f8769c00640 1 -- 172.21.15.6:0/3580037235 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f87640cdc20 msgr2=0x7f87640b2dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:22.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:22.778+0000 7f8769c00640 1 --2- 172.21.15.6:0/3580037235 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f87640cdc20 0x7f87640b2dd0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f875c0097a0 tx=0x7f875c02b840 comp rx=0 tx=0).stop 2024-03-04T09:25:22.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:22.779+0000 7f8769c00640 1 -- 172.21.15.6:0/3580037235 shutdown_connections 2024-03-04T09:25:22.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:22.779+0000 7f8769c00640 1 --2- 172.21.15.6:0/3580037235 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f87640cdc20 0x7f87640b2dd0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:22.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:22.779+0000 7f8769c00640 1 --2- 172.21.15.6:0/3580037235 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f87640af1a0 0x7f876415c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:22.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:22.779+0000 7f8769c00640 1 -- 172.21.15.6:0/3580037235 >> 172.21.15.6:0/3580037235 conn(0x7f876406afb0 msgr2=0x7f876406b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:22.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:22.779+0000 7f8769c00640 1 -- 172.21.15.6:0/3580037235 shutdown_connections 2024-03-04T09:25:22.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:22.779+0000 7f8769c00640 1 -- 172.21.15.6:0/3580037235 wait complete. 2024-03-04T09:25:22.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:22.780+0000 7f8769c00640 1 Processor -- start 2024-03-04T09:25:22.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:22.780+0000 7f8769c00640 1 -- start start 2024-03-04T09:25:22.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:22.781+0000 7f8769c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f87640af1a0 0x7f87640cafd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:22.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:22.781+0000 7f8769c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f87640af580 0x7f876414d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:22.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:22.781+0000 7f8769c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f87640cdc20 0x7f876406da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:22.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:22.781+0000 7f8769c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87640b36a0 con 0x7f87640cdc20 2024-03-04T09:25:22.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:22.781+0000 7f8769c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87640b3520 con 0x7f87640af580 2024-03-04T09:25:22.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:22.781+0000 7f8769c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87640b3820 con 0x7f87640af1a0 2024-03-04T09:25:22.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:22.782+0000 7f8763400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f87640cdc20 0x7f876406da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:22.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:22.782+0000 7f8763400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f87640cdc20 0x7f876406da50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45030/0 (socket says 172.21.15.6:45030) 2024-03-04T09:25:22.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:22.782+0000 7f8763400640 1 -- 172.21.15.6:0/4104228325 learned_addr learned my addr 172.21.15.6:0/4104228325 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:25:22.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:22.782+0000 7f8762000640 1 --2- 172.21.15.6:0/4104228325 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f87640af580 0x7f876414d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:22.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:22.782+0000 7f8762a00640 1 --2- 172.21.15.6:0/4104228325 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f87640af1a0 0x7f87640cafd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:22.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:22.782+0000 7f8763400640 1 -- 172.21.15.6:0/4104228325 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f87640af1a0 msgr2=0x7f87640cafd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:22.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:22.782+0000 7f8763400640 1 --2- 172.21.15.6:0/4104228325 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f87640af1a0 0x7f87640cafd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:22.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:22.782+0000 7f8763400640 1 -- 172.21.15.6:0/4104228325 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f87640af580 msgr2=0x7f876414d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:22.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:22.782+0000 7f8763400640 1 --2- 172.21.15.6:0/4104228325 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f87640af580 0x7f876414d660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:22.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:22.782+0000 7f8763400640 1 -- 172.21.15.6:0/4104228325 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87640c9f30 con 0x7f87640cdc20 2024-03-04T09:25:22.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:22.783+0000 7f8763400640 1 --2- 172.21.15.6:0/4104228325 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f87640cdc20 0x7f876406da50 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f875c034040 tx=0x7f875c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:22.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:22.783+0000 7f875aa00640 1 -- 172.21.15.6:0/4104228325 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f875c006e10 con 0x7f87640cdc20 2024-03-04T09:25:22.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:22.783+0000 7f875aa00640 1 -- 172.21.15.6:0/4104228325 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f875c0317a0 con 0x7f87640cdc20 2024-03-04T09:25:22.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:22.783+0000 7f875aa00640 1 -- 172.21.15.6:0/4104228325 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f875c031b90 con 0x7f87640cdc20 2024-03-04T09:25:22.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:22.783+0000 7f8769c00640 1 -- 172.21.15.6:0/4104228325 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87640ca210 con 0x7f87640cdc20 2024-03-04T09:25:22.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:22.783+0000 7f8769c00640 1 -- 172.21.15.6:0/4104228325 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87640caab0 con 0x7f87640cdc20 2024-03-04T09:25:22.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:22.784+0000 7f8769c00640 1 -- 172.21.15.6:0/4104228325 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87640ca600 con 0x7f87640cdc20 2024-03-04T09:25:22.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:22.785+0000 7f875aa00640 1 -- 172.21.15.6:0/4104228325 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f875c031d30 con 0x7f87640cdc20 2024-03-04T09:25:22.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:22.786+0000 7f875aa00640 1 --2- 172.21.15.6:0/4104228325 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f874006b500 0x7f874006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:22.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:22.786+0000 7f875aa00640 1 -- 172.21.15.6:0/4104228325 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f875c02f020 con 0x7f87640cdc20 2024-03-04T09:25:22.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:22.787+0000 7f8762a00640 1 --2- 172.21.15.6:0/4104228325 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f874006b500 0x7f874006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:22.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:22.788+0000 7f8762a00640 1 --2- 172.21.15.6:0/4104228325 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f874006b500 0x7f874006d9c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f87640b5490 tx=0x7f875400b040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:22.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:22.791+0000 7f875aa00640 1 -- 172.21.15.6:0/4104228325 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f875c07b610 con 0x7f87640cdc20 2024-03-04T09:25:23.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.029+0000 7f8769c00640 1 -- 172.21.15.6:0/4104228325 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f876404f3a0 con 0x7f87640cdc20 2024-03-04T09:25:23.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.031+0000 7f875aa00640 1 -- 172.21.15.6:0/4104228325 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4971 (secure 0 0 0) 0x7f875c07af10 con 0x7f87640cdc20 2024-03-04T09:25:23.028 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:25:23.028 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4248,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/2702058134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2702058134},{"type":"v1","addr":"172.21.15.6:6833","nonce":2702058134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4263,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/3866998852","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":3866998852},{"type":"v1","addr":"172.21.15.6:6835","nonce":3866998852}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4285,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/3523455455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":3523455455},{"type":"v1","addr":"172.21.15.146:6837","nonce":3523455455}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:25:18.912808+0000","modified":"2024-03-04T09:25:22.016493+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4288},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4288":{"gid":4288,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.146:6836/1946537763","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":1946537763},{"type":"v1","addr":"172.21.15.146:6836","nonce":1946537763}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4288,"qdb_cluster":[4288]},"id":1}]} 2024-03-04T09:25:23.028 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 7 2024-03-04T09:25:23.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.033+0000 7f8769c00640 1 -- 172.21.15.6:0/4104228325 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f874006b500 msgr2=0x7f874006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:23.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.033+0000 7f8769c00640 1 --2- 172.21.15.6:0/4104228325 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f874006b500 0x7f874006d9c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f87640b5490 tx=0x7f875400b040 comp rx=0 tx=0).stop 2024-03-04T09:25:23.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.033+0000 7f8769c00640 1 -- 172.21.15.6:0/4104228325 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f87640cdc20 msgr2=0x7f876406da50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:23.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.033+0000 7f8769c00640 1 --2- 172.21.15.6:0/4104228325 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f87640cdc20 0x7f876406da50 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f875c034040 tx=0x7f875c0046c0 comp rx=0 tx=0).stop 2024-03-04T09:25:23.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.034+0000 7f8769c00640 1 -- 172.21.15.6:0/4104228325 shutdown_connections 2024-03-04T09:25:23.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.034+0000 7f8769c00640 1 --2- 172.21.15.6:0/4104228325 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f87640cdc20 0x7f876406da50 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:23.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.034+0000 7f8769c00640 1 --2- 172.21.15.6:0/4104228325 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f87640af580 0x7f876414d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:23.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.034+0000 7f8769c00640 1 --2- 172.21.15.6:0/4104228325 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f874006b500 0x7f874006d9c0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:23.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.034+0000 7f8769c00640 1 --2- 172.21.15.6:0/4104228325 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f87640af1a0 0x7f87640cafd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:23.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.034+0000 7f8769c00640 1 -- 172.21.15.6:0/4104228325 >> 172.21.15.6:0/4104228325 conn(0x7f876406afb0 msgr2=0x7f876415a490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:23.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.034+0000 7f8769c00640 1 -- 172.21.15.6:0/4104228325 shutdown_connections 2024-03-04T09:25:23.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.034+0000 7f8769c00640 1 -- 172.21.15.6:0/4104228325 wait complete. 2024-03-04T09:25:23.050 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T09:25:23.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.272+0000 7fce5de00640 1 Processor -- start 2024-03-04T09:25:23.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.272+0000 7fce5de00640 1 -- start start 2024-03-04T09:25:23.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.274+0000 7fce5de00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fce580af580 0x7fce5815c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:23.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.274+0000 7fce5de00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce5815cc40 con 0x7fce580cdc20 2024-03-04T09:25:23.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.274+0000 7fce5de00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce5815cf20 con 0x7fce580af1a0 2024-03-04T09:25:23.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.274+0000 7fce5de00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce5815d200 con 0x7fce580af580 2024-03-04T09:25:23.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.274+0000 7fce57400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fce580cdc20 0x7fce580b2dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:60172/0 (socket says 172.21.15.6:60172) 2024-03-04T09:25:23.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.274+0000 7fce57400640 1 -- 172.21.15.6:0/994725207 learned_addr learned my addr 172.21.15.6:0/994725207 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:25:23.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.274+0000 7fce55600640 1 -- 172.21.15.6:0/994725207 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1712511796 0 0) 0x7fce5815cc40 con 0x7fce580cdc20 2024-03-04T09:25:23.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.274+0000 7fce56000640 1 --2- 172.21.15.6:0/994725207 >> v2:172.21.15.146:3300/0 conn(0x7fce580af580 0x7fce5815c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:23.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.275+0000 7fce55600640 1 -- 172.21.15.6:0/994725207 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce3c003520 con 0x7fce580cdc20 2024-03-04T09:25:23.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.275+0000 7fce55600640 1 -- 172.21.15.6:0/994725207 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3046401236 0 0) 0x7fce3c003520 con 0x7fce580cdc20 2024-03-04T09:25:23.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.275+0000 7fce55600640 1 -- 172.21.15.6:0/994725207 >> v2:172.21.15.146:3300/0 conn(0x7fce580af580 msgr2=0x7fce5815c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:23.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.275+0000 7fce55600640 1 --2- 172.21.15.6:0/994725207 >> v2:172.21.15.146:3300/0 conn(0x7fce580af580 0x7fce5815c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:23.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.275+0000 7fce55600640 1 -- 172.21.15.6:0/994725207 >> v1:172.21.15.146:6789/0 conn(0x7fce580af1a0 legacy=0x7fce580b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:23.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.275+0000 7fce55600640 1 -- 172.21.15.6:0/994725207 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce5815d4e0 con 0x7fce580cdc20 2024-03-04T09:25:23.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.275+0000 7fce55600640 1 -- 172.21.15.6:0/994725207 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7fce500019f0 con 0x7fce580cdc20 2024-03-04T09:25:23.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.275+0000 7fce55600640 1 -- 172.21.15.6:0/994725207 >> v1:172.21.15.6:6789/0 conn(0x7fce580cdc20 legacy=0x7fce580b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:23.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.275+0000 7fce55600640 1 --2- 172.21.15.6:0/994725207 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fce3c003a70 0x7fce3c003e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:23.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.275+0000 7fce55600640 1 --2- 172.21.15.6:0/994725207 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fce3c004800 0x7fce3c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:23.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.275+0000 7fce55600640 1 --2- 172.21.15.6:0/994725207 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fce3c0052e0 0x7fce3c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:23.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.275+0000 7fce55600640 1 -- 172.21.15.6:0/994725207 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce5815d200 con 0x7fce3c004800 2024-03-04T09:25:23.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.275+0000 7fce55600640 1 -- 172.21.15.6:0/994725207 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce3c00c1b0 con 0x7fce3c0052e0 2024-03-04T09:25:23.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.275+0000 7fce55600640 1 -- 172.21.15.6:0/994725207 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce3c00c3b0 con 0x7fce3c003a70 2024-03-04T09:25:23.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.275+0000 7fce56000640 1 --2- 172.21.15.6:0/994725207 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fce3c004800 0x7fce3c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:23.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.276+0000 7fce56a00640 1 --2- 172.21.15.6:0/994725207 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fce3c003a70 0x7fce3c003e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:23.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.276+0000 7fce56000640 1 -- 172.21.15.6:0/994725207 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fce3c003a70 msgr2=0x7fce3c003e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:23.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.276+0000 7fce56000640 1 --2- 172.21.15.6:0/994725207 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fce3c003a70 0x7fce3c003e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:23.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.276+0000 7fce56000640 1 -- 172.21.15.6:0/994725207 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fce3c0052e0 msgr2=0x7fce3c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:23.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.276+0000 7fce56000640 1 --2- 172.21.15.6:0/994725207 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fce3c0052e0 0x7fce3c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:23.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.276+0000 7fce56000640 1 -- 172.21.15.6:0/994725207 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce3c003520 con 0x7fce3c004800 2024-03-04T09:25:23.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.276+0000 7fce57400640 1 -- 172.21.15.6:0/994725207 reap_dead start 2024-03-04T09:25:23.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.277+0000 7fce56000640 1 --2- 172.21.15.6:0/994725207 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fce3c004800 0x7fce3c004c80 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fce44009a90 tx=0x7fce3c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddacbbacf7fb3ae1 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:23.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.277+0000 7fce55600640 1 -- 172.21.15.6:0/994725207 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce44039070 con 0x7fce3c004800 2024-03-04T09:25:23.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.277+0000 7fce55600640 1 -- 172.21.15.6:0/994725207 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce4402c6b0 con 0x7fce3c004800 2024-03-04T09:25:23.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.277+0000 7fce55600640 1 -- 172.21.15.6:0/994725207 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce44009690 con 0x7fce3c004800 2024-03-04T09:25:23.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.278+0000 7fce5de00640 1 -- 172.21.15.6:0/994725207 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fce3c004800 msgr2=0x7fce3c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:23.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.278+0000 7fce5de00640 1 --2- 172.21.15.6:0/994725207 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fce3c004800 0x7fce3c004c80 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fce44009a90 tx=0x7fce3c005830 comp rx=0 tx=0).stop 2024-03-04T09:25:23.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.278+0000 7fce5de00640 1 -- 172.21.15.6:0/994725207 shutdown_connections 2024-03-04T09:25:23.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.278+0000 7fce5de00640 1 --2- 172.21.15.6:0/994725207 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fce3c004800 0x7fce3c004c80 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:23.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.278+0000 7fce5de00640 1 -- 172.21.15.6:0/994725207 >> 172.21.15.6:0/994725207 conn(0x7fce5806afb0 msgr2=0x7fce5806b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:23.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.278+0000 7fce5de00640 1 -- 172.21.15.6:0/994725207 shutdown_connections 2024-03-04T09:25:23.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.278+0000 7fce5de00640 1 -- 172.21.15.6:0/994725207 wait complete. 2024-03-04T09:25:23.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.280+0000 7fce5de00640 1 Processor -- start 2024-03-04T09:25:23.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.280+0000 7fce5de00640 1 -- start start 2024-03-04T09:25:23.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.281+0000 7fce5de00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fce3c004800 0x7fce580c9310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:23.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.281+0000 7fce5de00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fce58159f70 0x7fce580b0e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:23.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.281+0000 7fce5de00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fce58147fc0 0x7fce58148440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:23.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.281+0000 7fce5de00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce5815c990 con 0x7fce58147fc0 2024-03-04T09:25:23.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.281+0000 7fce5de00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce5815c810 con 0x7fce3c004800 2024-03-04T09:25:23.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.281+0000 7fce5de00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce580b4c20 con 0x7fce58159f70 2024-03-04T09:25:23.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.281+0000 7fce57400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fce58147fc0 0x7fce58148440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:23.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.281+0000 7fce57400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fce58147fc0 0x7fce58148440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45038/0 (socket says 172.21.15.6:45038) 2024-03-04T09:25:23.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.281+0000 7fce57400640 1 -- 172.21.15.6:0/3763957964 learned_addr learned my addr 172.21.15.6:0/3763957964 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:25:23.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.281+0000 7fce56000640 1 --2- 172.21.15.6:0/3763957964 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fce58159f70 0x7fce580b0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:23.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.281+0000 7fce56a00640 1 --2- 172.21.15.6:0/3763957964 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fce3c004800 0x7fce580c9310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:23.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.281+0000 7fce57400640 1 -- 172.21.15.6:0/3763957964 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fce58159f70 msgr2=0x7fce580b0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:23.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.281+0000 7fce57400640 1 --2- 172.21.15.6:0/3763957964 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fce58159f70 0x7fce580b0e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:23.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.281+0000 7fce57400640 1 -- 172.21.15.6:0/3763957964 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fce3c004800 msgr2=0x7fce580c9310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:23.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.281+0000 7fce57400640 1 --2- 172.21.15.6:0/3763957964 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fce3c004800 0x7fce580c9310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:23.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.281+0000 7fce57400640 1 -- 172.21.15.6:0/3763957964 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce580cbd70 con 0x7fce58147fc0 2024-03-04T09:25:23.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.282+0000 7fce56000640 1 --2- 172.21.15.6:0/3763957964 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fce58159f70 0x7fce580b0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:25:23.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.282+0000 7fce57400640 1 --2- 172.21.15.6:0/3763957964 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fce58147fc0 0x7fce58148440 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fce580cda30 tx=0x7fce5000cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:23.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.282+0000 7fce4f400640 1 -- 172.21.15.6:0/3763957964 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce5000ce50 con 0x7fce58147fc0 2024-03-04T09:25:23.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.282+0000 7fce4f400640 1 -- 172.21.15.6:0/3763957964 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce5000a610 con 0x7fce58147fc0 2024-03-04T09:25:23.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.282+0000 7fce4f400640 1 -- 172.21.15.6:0/3763957964 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce5000aa00 con 0x7fce58147fc0 2024-03-04T09:25:23.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.282+0000 7fce5de00640 1 -- 172.21.15.6:0/3763957964 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce580cafb0 con 0x7fce58147fc0 2024-03-04T09:25:23.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.282+0000 7fce5de00640 1 -- 172.21.15.6:0/3763957964 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce580cb570 con 0x7fce58147fc0 2024-03-04T09:25:23.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.283+0000 7fce5de00640 1 -- 172.21.15.6:0/3763957964 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce5804f3a0 con 0x7fce58147fc0 2024-03-04T09:25:23.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.284+0000 7fce4f400640 1 -- 172.21.15.6:0/3763957964 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fce5000aba0 con 0x7fce58147fc0 2024-03-04T09:25:23.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.285+0000 7fce4f400640 1 --2- 172.21.15.6:0/3763957964 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fce3806b520 0x7fce3806d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:23.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.285+0000 7fce4f400640 1 -- 172.21.15.6:0/3763957964 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fce50089a90 con 0x7fce58147fc0 2024-03-04T09:25:23.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.286+0000 7fce56a00640 1 --2- 172.21.15.6:0/3763957964 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fce3806b520 0x7fce3806d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:23.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.287+0000 7fce56a00640 1 --2- 172.21.15.6:0/3763957964 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fce3806b520 0x7fce3806d9e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fce48007fd0 tx=0x7fce48009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:23.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.290+0000 7fce4f400640 1 -- 172.21.15.6:0/3763957964 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fce50058970 con 0x7fce58147fc0 2024-03-04T09:25:23.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.490+0000 7fce5de00640 1 -- 172.21.15.6:0/3763957964 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fce580ad2e0 con 0x7fce58147fc0 2024-03-04T09:25:23.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.491+0000 7fce4f400640 1 -- 172.21.15.6:0/3763957964 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16557 (secure 0 0 0) 0x7fce50058270 con 0x7fce58147fc0 2024-03-04T09:25:23.489 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:25:23.489 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":20,"fsid":"edce0578-5ae4-4402-b45a-552bdef84d7c","created":"2024-03-04T09:24:54.523780+0000","modified":"2024-03-04T09:25:20.446333+0000","last_up_change":"2024-03-04T09:25:07.089049+0000","last_in_change":"2024-03-04T09:24:57.982077+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T09:25:07.935198+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-04T09:25:17.118573+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-04T09:25:17.930536+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a96574f-81f1-4bc6-8463-757520b84a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6825","nonce":1364449401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6827","nonce":1364449401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6831","nonce":1364449401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6829","nonce":1364449401}]},"public_addr":"172.21.15.6:6825/1364449401","cluster_addr":"172.21.15.6:6827/1364449401","heartbeat_back_addr":"172.21.15.6:6831/1364449401","heartbeat_front_addr":"172.21.15.6:6829/1364449401","state":["exists","up"]},{"osd":1,"uuid":"7d79b9c1-6b87-476a-bc36-7db5ee64cd90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6809","nonce":2494103078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6811","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6813","nonce":2494103078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6821","nonce":2494103078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6817","nonce":2494103078}]},"public_addr":"172.21.15.6:6809/2494103078","cluster_addr":"172.21.15.6:6813/2494103078","heartbeat_back_addr":"172.21.15.6:6821/2494103078","heartbeat_front_addr":"172.21.15.6:6817/2494103078","state":["exists","up"]},{"osd":2,"uuid":"492331b9-4ee9-40b6-bacd-85d44cd6c45a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6812","nonce":1500066681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6816","nonce":1500066681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6824","nonce":1500066681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6820","nonce":1500066681}]},"public_addr":"172.21.15.6:6812/1500066681","cluster_addr":"172.21.15.6:6816/1500066681","heartbeat_back_addr":"172.21.15.6:6824/1500066681","heartbeat_front_addr":"172.21.15.6:6820/1500066681","state":["exists","up"]},{"osd":3,"uuid":"21bb2efe-9ba7-4bdc-bc31-180cd1102e21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6801","nonce":1558806123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6803","nonce":1558806123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6807","nonce":1558806123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6805","nonce":1558806123}]},"public_addr":"172.21.15.6:6801/1558806123","cluster_addr":"172.21.15.6:6803/1558806123","heartbeat_back_addr":"172.21.15.6:6807/1558806123","heartbeat_front_addr":"172.21.15.6:6805/1558806123","state":["exists","up"]},{"osd":4,"uuid":"6e2c4b71-19d9-4737-83d9-558705d8644a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6801","nonce":1526135780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6803","nonce":1526135780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6807","nonce":1526135780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6805","nonce":1526135780}]},"public_addr":"172.21.15.146:6801/1526135780","cluster_addr":"172.21.15.146:6803/1526135780","heartbeat_back_addr":"172.21.15.146:6807/1526135780","heartbeat_front_addr":"172.21.15.146:6805/1526135780","state":["exists","up"]},{"osd":5,"uuid":"fb69c525-f1be-4b7d-ab32-6c29b0d6fddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6809","nonce":460262507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6811","nonce":460262507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6815","nonce":460262507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6813","nonce":460262507}]},"public_addr":"172.21.15.146:6809/460262507","cluster_addr":"172.21.15.146:6811/460262507","heartbeat_back_addr":"172.21.15.146:6815/460262507","heartbeat_front_addr":"172.21.15.146:6813/460262507","state":["exists","up"]},{"osd":6,"uuid":"c32e7fed-370d-4401-8ce0-6a4335b0d168","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6823","nonce":1392897493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6827","nonce":1392897493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6831","nonce":1392897493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6829","nonce":1392897493}]},"public_addr":"172.21.15.146:6823/1392897493","cluster_addr":"172.21.15.146:6827/1392897493","heartbeat_back_addr":"172.21.15.146:6831/1392897493","heartbeat_front_addr":"172.21.15.146:6829/1392897493","state":["exists","up"]},{"osd":7,"uuid":"75a79b44-fa5f-426d-a7d3-71f21f78835c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6817","nonce":3268323156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6819","nonce":3268323156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6825","nonce":3268323156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6822","nonce":3268323156}]},"public_addr":"172.21.15.146:6817/3268323156","cluster_addr":"172.21.15.146:6819/3268323156","heartbeat_back_addr":"172.21.15.146:6825/3268323156","heartbeat_front_addr":"172.21.15.146:6822/3268323156","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.310437+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.295850+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.294674+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.526855+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.300023+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.595160+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.346046+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.558741+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T09:25:23.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.494+0000 7fce5de00640 1 -- 172.21.15.6:0/3763957964 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fce3806b520 msgr2=0x7fce3806d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:23.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.494+0000 7fce5de00640 1 --2- 172.21.15.6:0/3763957964 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fce3806b520 0x7fce3806d9e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fce48007fd0 tx=0x7fce48009040 comp rx=0 tx=0).stop 2024-03-04T09:25:23.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.494+0000 7fce5de00640 1 -- 172.21.15.6:0/3763957964 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fce58147fc0 msgr2=0x7fce58148440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:23.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.494+0000 7fce5de00640 1 --2- 172.21.15.6:0/3763957964 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fce58147fc0 0x7fce58148440 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fce580cda30 tx=0x7fce5000cc00 comp rx=0 tx=0).stop 2024-03-04T09:25:23.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.495+0000 7fce5de00640 1 -- 172.21.15.6:0/3763957964 shutdown_connections 2024-03-04T09:25:23.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.495+0000 7fce5de00640 1 --2- 172.21.15.6:0/3763957964 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fce58147fc0 0x7fce58148440 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:23.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.495+0000 7fce5de00640 1 --2- 172.21.15.6:0/3763957964 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fce3806b520 0x7fce3806d9e0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:23.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.495+0000 7fce5de00640 1 --2- 172.21.15.6:0/3763957964 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fce58159f70 0x7fce580b0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:23.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.495+0000 7fce5de00640 1 --2- 172.21.15.6:0/3763957964 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fce3c004800 0x7fce580c9310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:23.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.495+0000 7fce5de00640 1 -- 172.21.15.6:0/3763957964 >> 172.21.15.6:0/3763957964 conn(0x7fce5806afb0 msgr2=0x7fce5815b120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:23.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.495+0000 7fce5de00640 1 -- 172.21.15.6:0/3763957964 shutdown_connections 2024-03-04T09:25:23.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.495+0000 7fce5de00640 1 -- 172.21.15.6:0/3763957964 wait complete. 2024-03-04T09:25:23.512 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-04T09:25:23.512 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T09:25:23.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.736+0000 7f5967e00640 1 Processor -- start 2024-03-04T09:25:23.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.736+0000 7f5967e00640 1 -- start start 2024-03-04T09:25:23.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.738+0000 7f5967e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f596815efc0 0x7f59680cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:23.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.738+0000 7f5967e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f59680cce90 0x7f5968161ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:23.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.738+0000 7f5967e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5968162000 0x7f59680ad880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:23.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.738+0000 7f5967e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f596815f3a0 con 0x7f59680cce90 2024-03-04T09:25:23.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.738+0000 7f5967e00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f596806d190 con 0x7f596815efc0 2024-03-04T09:25:23.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.738+0000 7f5967e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59681627f0 con 0x7f5968162000 2024-03-04T09:25:23.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.738+0000 7f5966000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f59680cce90 0x7f5968161ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:23.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.738+0000 7f5966000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f59680cce90 0x7f5968161ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45048/0 (socket says 172.21.15.6:45048) 2024-03-04T09:25:23.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.738+0000 7f5966000640 1 -- 172.21.15.6:0/1018291443 learned_addr learned my addr 172.21.15.6:0/1018291443 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:25:23.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.738+0000 7f5967400640 1 --2- 172.21.15.6:0/1018291443 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5968162000 0x7f59680ad880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:23.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.738+0000 7f5966a00640 1 --2- 172.21.15.6:0/1018291443 >> v2:172.21.15.146:3300/0 conn(0x7f596815efc0 0x7f59680cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:23.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.739+0000 7f5966000640 1 -- 172.21.15.6:0/1018291443 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5968162000 msgr2=0x7f59680ad880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:23.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.739+0000 7f5966000640 1 --2- 172.21.15.6:0/1018291443 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5968162000 0x7f59680ad880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:23.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.739+0000 7f5966000640 1 -- 172.21.15.6:0/1018291443 >> v2:172.21.15.146:3300/0 conn(0x7f596815efc0 msgr2=0x7f59680cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:23.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.739+0000 7f5966000640 1 --2- 172.21.15.6:0/1018291443 >> v2:172.21.15.146:3300/0 conn(0x7f596815efc0 0x7f59680cc950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:23.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.739+0000 7f5966000640 1 -- 172.21.15.6:0/1018291443 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f596806c860 con 0x7f59680cce90 2024-03-04T09:25:23.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.740+0000 7f5966000640 1 --2- 172.21.15.6:0/1018291443 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f59680cce90 0x7f5968161ac0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f59680cd770 tx=0x7f595402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9989caef64afa0c4 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:23.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.741+0000 7f5965600640 1 -- 172.21.15.6:0/1018291443 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5954035020 con 0x7f59680cce90 2024-03-04T09:25:23.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.741+0000 7f5965600640 1 -- 172.21.15.6:0/1018291443 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5954031870 con 0x7f59680cce90 2024-03-04T09:25:23.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.741+0000 7f5965600640 1 -- 172.21.15.6:0/1018291443 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5954031c40 con 0x7f59680cce90 2024-03-04T09:25:23.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.741+0000 7f5967e00640 1 -- 172.21.15.6:0/1018291443 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f59680cce90 msgr2=0x7f5968161ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:23.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.741+0000 7f5967e00640 1 --2- 172.21.15.6:0/1018291443 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f59680cce90 0x7f5968161ac0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f59680cd770 tx=0x7f595402b910 comp rx=0 tx=0).stop 2024-03-04T09:25:23.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.742+0000 7f5967e00640 1 -- 172.21.15.6:0/1018291443 shutdown_connections 2024-03-04T09:25:23.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.742+0000 7f5967e00640 1 --2- 172.21.15.6:0/1018291443 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5968162000 0x7f59680ad880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:23.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.742+0000 7f5967e00640 1 --2- 172.21.15.6:0/1018291443 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f59680cce90 0x7f5968161ac0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:23.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.742+0000 7f5967e00640 1 --2- 172.21.15.6:0/1018291443 >> v2:172.21.15.146:3300/0 conn(0x7f596815efc0 0x7f59680cc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:23.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.742+0000 7f5967e00640 1 -- 172.21.15.6:0/1018291443 >> 172.21.15.6:0/1018291443 conn(0x7f596806b3d0 msgr2=0x7f596806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:23.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.742+0000 7f5967e00640 1 -- 172.21.15.6:0/1018291443 shutdown_connections 2024-03-04T09:25:23.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.742+0000 7f5967e00640 1 -- 172.21.15.6:0/1018291443 wait complete. 2024-03-04T09:25:23.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.744+0000 7f5967e00640 1 Processor -- start 2024-03-04T09:25:23.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.744+0000 7f5967e00640 1 -- start start 2024-03-04T09:25:23.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.745+0000 7f5967e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f59680cce90 0x7f5968139d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:23.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.745+0000 7f5967e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f596815efc0 0x7f596815cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:23.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.745+0000 7f5967e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5968162000 0x7f596815a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:23.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.745+0000 7f5967e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5968162490 con 0x7f5968162000 2024-03-04T09:25:23.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.745+0000 7f5967e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f596806ce20 con 0x7f596815efc0 2024-03-04T09:25:23.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.745+0000 7f5967e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f596806d620 con 0x7f59680cce90 2024-03-04T09:25:23.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.746+0000 7f5967400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5968162000 0x7f596815a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:23.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.746+0000 7f5967400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5968162000 0x7f596815a480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45058/0 (socket says 172.21.15.6:45058) 2024-03-04T09:25:23.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.746+0000 7f5967400640 1 -- 172.21.15.6:0/2407017774 learned_addr learned my addr 172.21.15.6:0/2407017774 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:25:23.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.746+0000 7f5966a00640 1 --2- 172.21.15.6:0/2407017774 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f59680cce90 0x7f5968139d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:23.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.746+0000 7f5967400640 1 -- 172.21.15.6:0/2407017774 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f59680cce90 msgr2=0x7f5968139d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:23.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.746+0000 7f5967400640 1 --2- 172.21.15.6:0/2407017774 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f59680cce90 0x7f5968139d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:23.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.746+0000 7f5967400640 1 -- 172.21.15.6:0/2407017774 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f596815efc0 msgr2=0x7f596815cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:23.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.746+0000 7f5967400640 1 --2- 172.21.15.6:0/2407017774 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f596815efc0 0x7f596815cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:23.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.746+0000 7f5967400640 1 -- 172.21.15.6:0/2407017774 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f596815d980 con 0x7f5968162000 2024-03-04T09:25:23.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.746+0000 7f5967400640 1 --2- 172.21.15.6:0/2407017774 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5968162000 0x7f596815a480 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f596000efc0 tx=0x7f596000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:23.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.747+0000 7f595ea00640 1 -- 172.21.15.6:0/2407017774 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5960018020 con 0x7f5968162000 2024-03-04T09:25:23.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.747+0000 7f595ea00640 1 -- 172.21.15.6:0/2407017774 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5960014480 con 0x7f5968162000 2024-03-04T09:25:23.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.747+0000 7f595ea00640 1 -- 172.21.15.6:0/2407017774 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5960014870 con 0x7f5968162000 2024-03-04T09:25:23.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.747+0000 7f5967e00640 1 -- 172.21.15.6:0/2407017774 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f596815dc60 con 0x7f5968162000 2024-03-04T09:25:23.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.747+0000 7f5967e00640 1 -- 172.21.15.6:0/2407017774 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59681e7c70 con 0x7f5968162000 2024-03-04T09:25:23.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.748+0000 7f5967e00640 1 -- 172.21.15.6:0/2407017774 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f596804f330 con 0x7f5968162000 2024-03-04T09:25:23.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.749+0000 7f595ea00640 1 -- 172.21.15.6:0/2407017774 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5960005d10 con 0x7f5968162000 2024-03-04T09:25:23.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.750+0000 7f595ea00640 1 --2- 172.21.15.6:0/2407017774 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f594806b550 0x7f594806da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:23.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.751+0000 7f595ea00640 1 -- 172.21.15.6:0/2407017774 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f596008ac50 con 0x7f5968162000 2024-03-04T09:25:23.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.751+0000 7f5966a00640 1 --2- 172.21.15.6:0/2407017774 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f594806b550 0x7f594806da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:23.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.752+0000 7f5966a00640 1 --2- 172.21.15.6:0/2407017774 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f594806b550 0x7f594806da10 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f5958006fd0 tx=0x7f5958008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:23.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.755+0000 7f595ea00640 1 -- 172.21.15.6:0/2407017774 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5960022070 con 0x7f5968162000 2024-03-04T09:25:23.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.953+0000 7f5967e00640 1 -- 172.21.15.6:0/2407017774 --> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f59680af2f0 con 0x7f594806b550 2024-03-04T09:25:23.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.976+0000 7f595ea00640 1 -- 172.21.15.6:0/2407017774 <== mgr.4113 v2:172.21.15.146:6832/1788613334 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251952 (secure 0 0 0) 0x7f59680af2f0 con 0x7f594806b550 2024-03-04T09:25:23.976 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:25:23.977 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-03-04T09:25:23.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.981+0000 7f5967e00640 1 -- 172.21.15.6:0/2407017774 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f594806b550 msgr2=0x7f594806da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:23.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.981+0000 7f5967e00640 1 --2- 172.21.15.6:0/2407017774 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f594806b550 0x7f594806da10 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f5958006fd0 tx=0x7f5958008040 comp rx=0 tx=0).stop 2024-03-04T09:25:23.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.981+0000 7f5967e00640 1 -- 172.21.15.6:0/2407017774 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5968162000 msgr2=0x7f596815a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:23.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.981+0000 7f5967e00640 1 --2- 172.21.15.6:0/2407017774 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5968162000 0x7f596815a480 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f596000efc0 tx=0x7f596000c490 comp rx=0 tx=0).stop 2024-03-04T09:25:23.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.982+0000 7f5967e00640 1 -- 172.21.15.6:0/2407017774 shutdown_connections 2024-03-04T09:25:23.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.982+0000 7f5967e00640 1 --2- 172.21.15.6:0/2407017774 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5968162000 0x7f596815a480 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:23.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.982+0000 7f5967e00640 1 --2- 172.21.15.6:0/2407017774 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f596815efc0 0x7f596815cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:23.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.982+0000 7f5967e00640 1 --2- 172.21.15.6:0/2407017774 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f594806b550 0x7f594806da10 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:23.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.982+0000 7f5967e00640 1 --2- 172.21.15.6:0/2407017774 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f59680cce90 0x7f5968139d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:23.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.982+0000 7f5967e00640 1 -- 172.21.15.6:0/2407017774 >> 172.21.15.6:0/2407017774 conn(0x7f596806b3d0 msgr2=0x7f59680b0950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:23.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.982+0000 7f5967e00640 1 -- 172.21.15.6:0/2407017774 shutdown_connections 2024-03-04T09:25:23.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:23.982+0000 7f5967e00640 1 -- 172.21.15.6:0/2407017774 wait complete. 2024-03-04T09:25:23.999 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-03-04T09:25:23.859677+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218988,"kb_used_data":2824,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755732,"statfs":{"total":773094113280,"available":772869869568,"internally_reserved":0,"allocated":2891776,"data_stored":1753028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":44,"apply_latency_ms":44,"commit_latency_ns":44000000,"apply_latency_ns":44000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002122"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.152203+0000","last_change":"2024-03-04T09:25:19.480195+0000","last_active":"2024-03-04T09:25:21.152203+0000","last_peered":"2024-03-04T09:25:21.152203+0000","last_clean":"2024-03-04T09:25:21.152203+0000","last_became_active":"2024-03-04T09:25:19.479496+0000","last_became_peered":"2024-03-04T09:25:19.479496+0000","last_unstale":"2024-03-04T09:25:21.152203+0000","last_undegraded":"2024-03-04T09:25:21.152203+0000","last_fullsized":"2024-03-04T09:25:21.152203+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:36:55.714408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.457360+0000","last_change":"2024-03-04T09:25:19.457568+0000","last_active":"2024-03-04T09:25:20.457360+0000","last_peered":"2024-03-04T09:25:20.457360+0000","last_clean":"2024-03-04T09:25:20.457360+0000","last_became_active":"2024-03-04T09:25:19.457288+0000","last_became_peered":"2024-03-04T09:25:19.457288+0000","last_unstale":"2024-03-04T09:25:20.457360+0000","last_undegraded":"2024-03-04T09:25:20.457360+0000","last_fullsized":"2024-03-04T09:25:20.457360+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:33:16.368258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.149774+0000","last_change":"2024-03-04T09:25:19.465855+0000","last_active":"2024-03-04T09:25:21.149774+0000","last_peered":"2024-03-04T09:25:21.149774+0000","last_clean":"2024-03-04T09:25:21.149774+0000","last_became_active":"2024-03-04T09:25:19.465489+0000","last_became_peered":"2024-03-04T09:25:19.465489+0000","last_unstale":"2024-03-04T09:25:21.149774+0000","last_undegraded":"2024-03-04T09:25:21.149774+0000","last_fullsized":"2024-03-04T09:25:21.149774+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:05:57.624897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.497998+0000","last_change":"2024-03-04T09:25:19.472849+0000","last_active":"2024-03-04T09:25:20.497998+0000","last_peered":"2024-03-04T09:25:20.497998+0000","last_clean":"2024-03-04T09:25:20.497998+0000","last_became_active":"2024-03-04T09:25:19.468252+0000","last_became_peered":"2024-03-04T09:25:19.468252+0000","last_unstale":"2024-03-04T09:25:20.497998+0000","last_undegraded":"2024-03-04T09:25:20.497998+0000","last_fullsized":"2024-03-04T09:25:20.497998+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:20:25.135538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.151031+0000","last_change":"2024-03-04T09:25:19.471747+0000","last_active":"2024-03-04T09:25:21.151031+0000","last_peered":"2024-03-04T09:25:21.151031+0000","last_clean":"2024-03-04T09:25:21.151031+0000","last_became_active":"2024-03-04T09:25:19.469589+0000","last_became_peered":"2024-03-04T09:25:19.469589+0000","last_unstale":"2024-03-04T09:25:21.151031+0000","last_undegraded":"2024-03-04T09:25:21.151031+0000","last_fullsized":"2024-03-04T09:25:21.151031+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:36:58.654226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.497539+0000","last_change":"2024-03-04T09:25:19.458354+0000","last_active":"2024-03-04T09:25:20.497539+0000","last_peered":"2024-03-04T09:25:20.497539+0000","last_clean":"2024-03-04T09:25:20.497539+0000","last_became_active":"2024-03-04T09:25:19.456693+0000","last_became_peered":"2024-03-04T09:25:19.456693+0000","last_unstale":"2024-03-04T09:25:20.497539+0000","last_undegraded":"2024-03-04T09:25:20.497539+0000","last_fullsized":"2024-03-04T09:25:20.497539+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:11:11.501102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.150488+0000","last_change":"2024-03-04T09:25:19.476465+0000","last_active":"2024-03-04T09:25:21.150488+0000","last_peered":"2024-03-04T09:25:21.150488+0000","last_clean":"2024-03-04T09:25:21.150488+0000","last_became_active":"2024-03-04T09:25:19.471519+0000","last_became_peered":"2024-03-04T09:25:19.471519+0000","last_unstale":"2024-03-04T09:25:21.150488+0000","last_undegraded":"2024-03-04T09:25:21.150488+0000","last_fullsized":"2024-03-04T09:25:21.150488+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:01:33.498760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.455119+0000","last_change":"2024-03-04T09:25:19.471301+0000","last_active":"2024-03-04T09:25:20.455119+0000","last_peered":"2024-03-04T09:25:20.455119+0000","last_clean":"2024-03-04T09:25:20.455119+0000","last_became_active":"2024-03-04T09:25:19.470983+0000","last_became_peered":"2024-03-04T09:25:19.470983+0000","last_unstale":"2024-03-04T09:25:20.455119+0000","last_undegraded":"2024-03-04T09:25:20.455119+0000","last_fullsized":"2024-03-04T09:25:20.455119+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:50:18.265156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.152495+0000","last_change":"2024-03-04T09:25:19.485751+0000","last_active":"2024-03-04T09:25:21.152495+0000","last_peered":"2024-03-04T09:25:21.152495+0000","last_clean":"2024-03-04T09:25:21.152495+0000","last_became_active":"2024-03-04T09:25:19.485488+0000","last_became_peered":"2024-03-04T09:25:19.485488+0000","last_unstale":"2024-03-04T09:25:21.152495+0000","last_undegraded":"2024-03-04T09:25:21.152495+0000","last_fullsized":"2024-03-04T09:25:21.152495+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:45:23.780438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.464582+0000","last_change":"2024-03-04T09:25:19.466385+0000","last_active":"2024-03-04T09:25:20.464582+0000","last_peered":"2024-03-04T09:25:20.464582+0000","last_clean":"2024-03-04T09:25:20.464582+0000","last_became_active":"2024-03-04T09:25:19.466078+0000","last_became_peered":"2024-03-04T09:25:19.466078+0000","last_unstale":"2024-03-04T09:25:20.464582+0000","last_undegraded":"2024-03-04T09:25:20.464582+0000","last_fullsized":"2024-03-04T09:25:20.464582+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:56:18.833810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.464502+0000","last_change":"2024-03-04T09:25:19.458668+0000","last_active":"2024-03-04T09:25:20.464502+0000","last_peered":"2024-03-04T09:25:20.464502+0000","last_clean":"2024-03-04T09:25:20.464502+0000","last_became_active":"2024-03-04T09:25:19.458080+0000","last_became_peered":"2024-03-04T09:25:19.458080+0000","last_unstale":"2024-03-04T09:25:20.464502+0000","last_undegraded":"2024-03-04T09:25:20.464502+0000","last_fullsized":"2024-03-04T09:25:20.464502+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:44:37.465937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.152136+0000","last_change":"2024-03-04T09:25:19.470591+0000","last_active":"2024-03-04T09:25:21.152136+0000","last_peered":"2024-03-04T09:25:21.152136+0000","last_clean":"2024-03-04T09:25:21.152136+0000","last_became_active":"2024-03-04T09:25:19.463604+0000","last_became_peered":"2024-03-04T09:25:19.463604+0000","last_unstale":"2024-03-04T09:25:21.152136+0000","last_undegraded":"2024-03-04T09:25:21.152136+0000","last_fullsized":"2024-03-04T09:25:21.152136+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:02:45.778608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.456030+0000","last_change":"2024-03-04T09:25:19.462024+0000","last_active":"2024-03-04T09:25:20.456030+0000","last_peered":"2024-03-04T09:25:20.456030+0000","last_clean":"2024-03-04T09:25:20.456030+0000","last_became_active":"2024-03-04T09:25:19.461767+0000","last_became_peered":"2024-03-04T09:25:19.461767+0000","last_unstale":"2024-03-04T09:25:20.456030+0000","last_undegraded":"2024-03-04T09:25:20.456030+0000","last_fullsized":"2024-03-04T09:25:20.456030+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:47:26.445286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.149510+0000","last_change":"2024-03-04T09:25:19.468514+0000","last_active":"2024-03-04T09:25:21.149510+0000","last_peered":"2024-03-04T09:25:21.149510+0000","last_clean":"2024-03-04T09:25:21.149510+0000","last_became_active":"2024-03-04T09:25:19.468029+0000","last_became_peered":"2024-03-04T09:25:19.468029+0000","last_unstale":"2024-03-04T09:25:21.149510+0000","last_undegraded":"2024-03-04T09:25:21.149510+0000","last_fullsized":"2024-03-04T09:25:21.149510+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:45:15.160192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.455915+0000","last_change":"2024-03-04T09:25:19.448212+0000","last_active":"2024-03-04T09:25:20.455915+0000","last_peered":"2024-03-04T09:25:20.455915+0000","last_clean":"2024-03-04T09:25:20.455915+0000","last_became_active":"2024-03-04T09:25:19.447824+0000","last_became_peered":"2024-03-04T09:25:19.447824+0000","last_unstale":"2024-03-04T09:25:20.455915+0000","last_undegraded":"2024-03-04T09:25:20.455915+0000","last_fullsized":"2024-03-04T09:25:20.455915+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:44:37.465937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.497265+0000","last_change":"2024-03-04T09:25:19.471682+0000","last_active":"2024-03-04T09:25:20.497265+0000","last_peered":"2024-03-04T09:25:20.497265+0000","last_clean":"2024-03-04T09:25:20.497265+0000","last_became_active":"2024-03-04T09:25:19.471416+0000","last_became_peered":"2024-03-04T09:25:19.471416+0000","last_unstale":"2024-03-04T09:25:20.497265+0000","last_undegraded":"2024-03-04T09:25:20.497265+0000","last_fullsized":"2024-03-04T09:25:20.497265+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:01:33.498760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.499060+0000","last_change":"2024-03-04T09:25:19.484010+0000","last_active":"2024-03-04T09:25:20.499060+0000","last_peered":"2024-03-04T09:25:20.499060+0000","last_clean":"2024-03-04T09:25:20.499060+0000","last_became_active":"2024-03-04T09:25:19.477995+0000","last_became_peered":"2024-03-04T09:25:19.477995+0000","last_unstale":"2024-03-04T09:25:20.499060+0000","last_undegraded":"2024-03-04T09:25:20.499060+0000","last_fullsized":"2024-03-04T09:25:20.499060+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:36:31.167806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.458074+0000","last_change":"2024-03-04T09:25:19.456238+0000","last_active":"2024-03-04T09:25:20.458074+0000","last_peered":"2024-03-04T09:25:20.458074+0000","last_clean":"2024-03-04T09:25:20.458074+0000","last_became_active":"2024-03-04T09:25:19.455908+0000","last_became_peered":"2024-03-04T09:25:19.455908+0000","last_unstale":"2024-03-04T09:25:20.458074+0000","last_undegraded":"2024-03-04T09:25:20.458074+0000","last_fullsized":"2024-03-04T09:25:20.458074+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:32:59.918382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.151921+0000","last_change":"2024-03-04T09:25:19.479250+0000","last_active":"2024-03-04T09:25:21.151921+0000","last_peered":"2024-03-04T09:25:21.151921+0000","last_clean":"2024-03-04T09:25:21.151921+0000","last_became_active":"2024-03-04T09:25:19.478971+0000","last_became_peered":"2024-03-04T09:25:19.478971+0000","last_unstale":"2024-03-04T09:25:21.151921+0000","last_undegraded":"2024-03-04T09:25:21.151921+0000","last_fullsized":"2024-03-04T09:25:21.151921+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:12:45.452669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.464032+0000","last_change":"2024-03-04T09:25:19.471817+0000","last_active":"2024-03-04T09:25:20.464032+0000","last_peered":"2024-03-04T09:25:20.464032+0000","last_clean":"2024-03-04T09:25:20.464032+0000","last_became_active":"2024-03-04T09:25:19.471551+0000","last_became_peered":"2024-03-04T09:25:19.471551+0000","last_unstale":"2024-03-04T09:25:20.464032+0000","last_undegraded":"2024-03-04T09:25:20.464032+0000","last_fullsized":"2024-03-04T09:25:20.464032+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:45:59.096795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.149784+0000","last_change":"2024-03-04T09:25:19.464870+0000","last_active":"2024-03-04T09:25:21.149784+0000","last_peered":"2024-03-04T09:25:21.149784+0000","last_clean":"2024-03-04T09:25:21.149784+0000","last_became_active":"2024-03-04T09:25:19.464544+0000","last_became_peered":"2024-03-04T09:25:19.464544+0000","last_unstale":"2024-03-04T09:25:21.149784+0000","last_undegraded":"2024-03-04T09:25:21.149784+0000","last_fullsized":"2024-03-04T09:25:21.149784+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:45:59.096795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.497519+0000","last_change":"2024-03-04T09:25:19.459890+0000","last_active":"2024-03-04T09:25:20.497519+0000","last_peered":"2024-03-04T09:25:20.497519+0000","last_clean":"2024-03-04T09:25:20.497519+0000","last_became_active":"2024-03-04T09:25:19.459528+0000","last_became_peered":"2024-03-04T09:25:19.459528+0000","last_unstale":"2024-03-04T09:25:20.497519+0000","last_undegraded":"2024-03-04T09:25:20.497519+0000","last_fullsized":"2024-03-04T09:25:20.497519+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:36:58.654226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.456364+0000","last_change":"2024-03-04T09:25:19.462338+0000","last_active":"2024-03-04T09:25:20.456364+0000","last_peered":"2024-03-04T09:25:20.456364+0000","last_clean":"2024-03-04T09:25:20.456364+0000","last_became_active":"2024-03-04T09:25:19.461998+0000","last_became_peered":"2024-03-04T09:25:19.461998+0000","last_unstale":"2024-03-04T09:25:20.456364+0000","last_undegraded":"2024-03-04T09:25:20.456364+0000","last_fullsized":"2024-03-04T09:25:20.456364+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:56:18.833810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.497149+0000","last_change":"2024-03-04T09:25:19.473080+0000","last_active":"2024-03-04T09:25:20.497149+0000","last_peered":"2024-03-04T09:25:20.497149+0000","last_clean":"2024-03-04T09:25:20.497149+0000","last_became_active":"2024-03-04T09:25:19.467315+0000","last_became_peered":"2024-03-04T09:25:19.467315+0000","last_unstale":"2024-03-04T09:25:20.497149+0000","last_undegraded":"2024-03-04T09:25:20.497149+0000","last_fullsized":"2024-03-04T09:25:20.497149+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:33:16.368258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.458010+0000","last_change":"2024-03-04T09:25:19.472788+0000","last_active":"2024-03-04T09:25:20.458010+0000","last_peered":"2024-03-04T09:25:20.458010+0000","last_clean":"2024-03-04T09:25:20.458010+0000","last_became_active":"2024-03-04T09:25:19.472496+0000","last_became_peered":"2024-03-04T09:25:19.472496+0000","last_unstale":"2024-03-04T09:25:20.458010+0000","last_undegraded":"2024-03-04T09:25:20.458010+0000","last_fullsized":"2024-03-04T09:25:20.458010+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:01:33.498760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.464735+0000","last_change":"2024-03-04T09:25:19.461265+0000","last_active":"2024-03-04T09:25:20.464735+0000","last_peered":"2024-03-04T09:25:20.464735+0000","last_clean":"2024-03-04T09:25:20.464735+0000","last_became_active":"2024-03-04T09:25:19.458576+0000","last_became_peered":"2024-03-04T09:25:19.458576+0000","last_unstale":"2024-03-04T09:25:20.464735+0000","last_undegraded":"2024-03-04T09:25:20.464735+0000","last_fullsized":"2024-03-04T09:25:20.464735+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:33:58.076780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.456949+0000","last_change":"2024-03-04T09:25:19.467165+0000","last_active":"2024-03-04T09:25:20.456949+0000","last_peered":"2024-03-04T09:25:20.456949+0000","last_clean":"2024-03-04T09:25:20.456949+0000","last_became_active":"2024-03-04T09:25:19.466793+0000","last_became_peered":"2024-03-04T09:25:19.466793+0000","last_unstale":"2024-03-04T09:25:20.456949+0000","last_undegraded":"2024-03-04T09:25:20.456949+0000","last_fullsized":"2024-03-04T09:25:20.456949+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:33:58.076780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.457770+0000","last_change":"2024-03-04T09:25:19.474283+0000","last_active":"2024-03-04T09:25:20.457770+0000","last_peered":"2024-03-04T09:25:20.457770+0000","last_clean":"2024-03-04T09:25:20.457770+0000","last_became_active":"2024-03-04T09:25:19.474124+0000","last_became_peered":"2024-03-04T09:25:19.474124+0000","last_unstale":"2024-03-04T09:25:20.457770+0000","last_undegraded":"2024-03-04T09:25:20.457770+0000","last_fullsized":"2024-03-04T09:25:20.457770+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:33:16.368258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.150029+0000","last_change":"2024-03-04T09:25:19.461996+0000","last_active":"2024-03-04T09:25:21.150029+0000","last_peered":"2024-03-04T09:25:21.150029+0000","last_clean":"2024-03-04T09:25:21.150029+0000","last_became_active":"2024-03-04T09:25:19.461683+0000","last_became_peered":"2024-03-04T09:25:19.461683+0000","last_unstale":"2024-03-04T09:25:21.150029+0000","last_undegraded":"2024-03-04T09:25:21.150029+0000","last_fullsized":"2024-03-04T09:25:21.150029+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:47:26.445286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.497702+0000","last_change":"2024-03-04T09:25:19.466189+0000","last_active":"2024-03-04T09:25:20.497702+0000","last_peered":"2024-03-04T09:25:20.497702+0000","last_clean":"2024-03-04T09:25:20.497702+0000","last_became_active":"2024-03-04T09:25:19.465903+0000","last_became_peered":"2024-03-04T09:25:19.465903+0000","last_unstale":"2024-03-04T09:25:20.497702+0000","last_undegraded":"2024-03-04T09:25:20.497702+0000","last_fullsized":"2024-03-04T09:25:20.497702+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:04:48.675219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.457411+0000","last_change":"2024-03-04T09:25:19.458477+0000","last_active":"2024-03-04T09:25:20.457411+0000","last_peered":"2024-03-04T09:25:20.457411+0000","last_clean":"2024-03-04T09:25:20.457411+0000","last_became_active":"2024-03-04T09:25:19.455994+0000","last_became_peered":"2024-03-04T09:25:19.455994+0000","last_unstale":"2024-03-04T09:25:20.457411+0000","last_undegraded":"2024-03-04T09:25:20.457411+0000","last_fullsized":"2024-03-04T09:25:20.457411+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:11:52.240719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.463964+0000","last_change":"2024-03-04T09:25:19.464315+0000","last_active":"2024-03-04T09:25:20.463964+0000","last_peered":"2024-03-04T09:25:20.463964+0000","last_clean":"2024-03-04T09:25:20.463964+0000","last_became_active":"2024-03-04T09:25:19.463942+0000","last_became_peered":"2024-03-04T09:25:19.463942+0000","last_unstale":"2024-03-04T09:25:20.463964+0000","last_undegraded":"2024-03-04T09:25:20.463964+0000","last_fullsized":"2024-03-04T09:25:20.463964+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:05:57.624897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.149802+0000","last_change":"2024-03-04T09:25:19.448874+0000","last_active":"2024-03-04T09:25:21.149802+0000","last_peered":"2024-03-04T09:25:21.149802+0000","last_clean":"2024-03-04T09:25:21.149802+0000","last_became_active":"2024-03-04T09:25:19.445047+0000","last_became_peered":"2024-03-04T09:25:19.445047+0000","last_unstale":"2024-03-04T09:25:21.149802+0000","last_undegraded":"2024-03-04T09:25:21.149802+0000","last_fullsized":"2024-03-04T09:25:21.149802+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:23:35.947152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.456151+0000","last_change":"2024-03-04T09:25:18.469771+0000","last_active":"2024-03-04T09:25:20.456151+0000","last_peered":"2024-03-04T09:25:20.456151+0000","last_clean":"2024-03-04T09:25:20.456151+0000","last_became_active":"2024-03-04T09:25:18.469445+0000","last_became_peered":"2024-03-04T09:25:18.469445+0000","last_unstale":"2024-03-04T09:25:20.456151+0000","last_undegraded":"2024-03-04T09:25:20.456151+0000","last_fullsized":"2024-03-04T09:25:20.456151+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:39:20.774577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.150269+0000","last_change":"2024-03-04T09:25:19.463486+0000","last_active":"2024-03-04T09:25:21.150269+0000","last_peered":"2024-03-04T09:25:21.150269+0000","last_clean":"2024-03-04T09:25:21.150269+0000","last_became_active":"2024-03-04T09:25:19.463204+0000","last_became_peered":"2024-03-04T09:25:19.463204+0000","last_unstale":"2024-03-04T09:25:21.150269+0000","last_undegraded":"2024-03-04T09:25:21.150269+0000","last_fullsized":"2024-03-04T09:25:21.150269+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:33:58.076780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.463976+0000","last_change":"2024-03-04T09:25:18.472794+0000","last_active":"2024-03-04T09:25:20.463976+0000","last_peered":"2024-03-04T09:25:20.463976+0000","last_clean":"2024-03-04T09:25:20.463976+0000","last_became_active":"2024-03-04T09:25:18.472449+0000","last_became_peered":"2024-03-04T09:25:18.472449+0000","last_unstale":"2024-03-04T09:25:20.463976+0000","last_undegraded":"2024-03-04T09:25:20.463976+0000","last_fullsized":"2024-03-04T09:25:20.463976+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:23:34.932850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.456455+0000","last_change":"2024-03-04T09:25:18.474050+0000","last_active":"2024-03-04T09:25:20.456455+0000","last_peered":"2024-03-04T09:25:20.456455+0000","last_clean":"2024-03-04T09:25:20.456455+0000","last_became_active":"2024-03-04T09:25:18.473760+0000","last_became_peered":"2024-03-04T09:25:18.473760+0000","last_unstale":"2024-03-04T09:25:20.456455+0000","last_undegraded":"2024-03-04T09:25:20.456455+0000","last_fullsized":"2024-03-04T09:25:20.456455+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:31:15.318756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.455119+0000","last_change":"2024-03-04T09:25:19.464447+0000","last_active":"2024-03-04T09:25:20.455119+0000","last_peered":"2024-03-04T09:25:20.455119+0000","last_clean":"2024-03-04T09:25:20.455119+0000","last_became_active":"2024-03-04T09:25:19.463201+0000","last_became_peered":"2024-03-04T09:25:19.463201+0000","last_unstale":"2024-03-04T09:25:20.455119+0000","last_undegraded":"2024-03-04T09:25:20.455119+0000","last_fullsized":"2024-03-04T09:25:20.455119+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:05:57.624897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.497223+0000","last_change":"2024-03-04T09:25:18.476149+0000","last_active":"2024-03-04T09:25:20.497223+0000","last_peered":"2024-03-04T09:25:20.497223+0000","last_clean":"2024-03-04T09:25:20.497223+0000","last_became_active":"2024-03-04T09:25:18.475664+0000","last_became_peered":"2024-03-04T09:25:18.475664+0000","last_unstale":"2024-03-04T09:25:20.497223+0000","last_undegraded":"2024-03-04T09:25:20.497223+0000","last_fullsized":"2024-03-04T09:25:20.497223+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:33:35.151096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.455640+0000","last_change":"2024-03-04T09:25:19.467051+0000","last_active":"2024-03-04T09:25:20.455640+0000","last_peered":"2024-03-04T09:25:20.455640+0000","last_clean":"2024-03-04T09:25:20.455640+0000","last_became_active":"2024-03-04T09:25:19.466755+0000","last_became_peered":"2024-03-04T09:25:19.466755+0000","last_unstale":"2024-03-04T09:25:20.455640+0000","last_undegraded":"2024-03-04T09:25:20.455640+0000","last_fullsized":"2024-03-04T09:25:20.455640+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:47:26.445286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.464438+0000","last_change":"2024-03-04T09:25:18.479294+0000","last_active":"2024-03-04T09:25:20.464438+0000","last_peered":"2024-03-04T09:25:20.464438+0000","last_clean":"2024-03-04T09:25:20.464438+0000","last_became_active":"2024-03-04T09:25:18.478561+0000","last_became_peered":"2024-03-04T09:25:18.478561+0000","last_unstale":"2024-03-04T09:25:20.464438+0000","last_undegraded":"2024-03-04T09:25:20.464438+0000","last_fullsized":"2024-03-04T09:25:20.464438+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:33:35.151096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.498563+0000","last_change":"2024-03-04T09:25:19.459933+0000","last_active":"2024-03-04T09:25:20.498563+0000","last_peered":"2024-03-04T09:25:20.498563+0000","last_clean":"2024-03-04T09:25:20.498563+0000","last_became_active":"2024-03-04T09:25:19.459622+0000","last_became_peered":"2024-03-04T09:25:19.459622+0000","last_unstale":"2024-03-04T09:25:20.498563+0000","last_undegraded":"2024-03-04T09:25:20.498563+0000","last_fullsized":"2024-03-04T09:25:20.498563+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:45:23.780438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.498673+0000","last_change":"2024-03-04T09:25:18.476726+0000","last_active":"2024-03-04T09:25:20.498673+0000","last_peered":"2024-03-04T09:25:20.498673+0000","last_clean":"2024-03-04T09:25:20.498673+0000","last_became_active":"2024-03-04T09:25:18.475636+0000","last_became_peered":"2024-03-04T09:25:18.475636+0000","last_unstale":"2024-03-04T09:25:20.498673+0000","last_undegraded":"2024-03-04T09:25:20.498673+0000","last_fullsized":"2024-03-04T09:25:20.498673+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:54:01.985207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.150507+0000","last_change":"2024-03-04T09:25:19.473920+0000","last_active":"2024-03-04T09:25:21.150507+0000","last_peered":"2024-03-04T09:25:21.150507+0000","last_clean":"2024-03-04T09:25:21.150507+0000","last_became_active":"2024-03-04T09:25:19.473457+0000","last_became_peered":"2024-03-04T09:25:19.473457+0000","last_unstale":"2024-03-04T09:25:21.150507+0000","last_undegraded":"2024-03-04T09:25:21.150507+0000","last_fullsized":"2024-03-04T09:25:21.150507+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:33:16.368258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.457014+0000","last_change":"2024-03-04T09:25:18.471507+0000","last_active":"2024-03-04T09:25:20.457014+0000","last_peered":"2024-03-04T09:25:20.457014+0000","last_clean":"2024-03-04T09:25:20.457014+0000","last_became_active":"2024-03-04T09:25:18.470138+0000","last_became_peered":"2024-03-04T09:25:18.470138+0000","last_unstale":"2024-03-04T09:25:20.457014+0000","last_undegraded":"2024-03-04T09:25:20.457014+0000","last_fullsized":"2024-03-04T09:25:20.457014+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:54:01.985207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.151586+0000","last_change":"2024-03-04T09:25:19.471146+0000","last_active":"2024-03-04T09:25:21.151586+0000","last_peered":"2024-03-04T09:25:21.151586+0000","last_clean":"2024-03-04T09:25:21.151586+0000","last_became_active":"2024-03-04T09:25:19.470904+0000","last_became_peered":"2024-03-04T09:25:19.470904+0000","last_unstale":"2024-03-04T09:25:21.151586+0000","last_undegraded":"2024-03-04T09:25:21.151586+0000","last_fullsized":"2024-03-04T09:25:21.151586+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:32:59.918382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"19'6","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.463922+0000","last_change":"2024-03-04T09:25:18.474956+0000","last_active":"2024-03-04T09:25:20.463922+0000","last_peered":"2024-03-04T09:25:20.463922+0000","last_clean":"2024-03-04T09:25:20.463922+0000","last_became_active":"2024-03-04T09:25:18.474011+0000","last_became_peered":"2024-03-04T09:25:18.474011+0000","last_unstale":"2024-03-04T09:25:20.463922+0000","last_undegraded":"2024-03-04T09:25:20.463922+0000","last_fullsized":"2024-03-04T09:25:20.463922+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:54:01.985207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.151254+0000","last_change":"2024-03-04T09:25:19.477899+0000","last_active":"2024-03-04T09:25:21.151254+0000","last_peered":"2024-03-04T09:25:21.151254+0000","last_clean":"2024-03-04T09:25:21.151254+0000","last_became_active":"2024-03-04T09:25:19.477117+0000","last_became_peered":"2024-03-04T09:25:19.477117+0000","last_unstale":"2024-03-04T09:25:21.151254+0000","last_undegraded":"2024-03-04T09:25:21.151254+0000","last_fullsized":"2024-03-04T09:25:21.151254+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:04:48.675219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.150191+0000","last_change":"2024-03-04T09:25:18.469651+0000","last_active":"2024-03-04T09:25:21.150191+0000","last_peered":"2024-03-04T09:25:21.150191+0000","last_clean":"2024-03-04T09:25:21.150191+0000","last_became_active":"2024-03-04T09:25:18.469259+0000","last_became_peered":"2024-03-04T09:25:18.469259+0000","last_unstale":"2024-03-04T09:25:21.150191+0000","last_undegraded":"2024-03-04T09:25:21.150191+0000","last_fullsized":"2024-03-04T09:25:21.150191+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:57:52.768999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.149691+0000","last_change":"2024-03-04T09:25:19.465339+0000","last_active":"2024-03-04T09:25:21.149691+0000","last_peered":"2024-03-04T09:25:21.149691+0000","last_clean":"2024-03-04T09:25:21.149691+0000","last_became_active":"2024-03-04T09:25:19.464957+0000","last_became_peered":"2024-03-04T09:25:19.464957+0000","last_unstale":"2024-03-04T09:25:21.149691+0000","last_undegraded":"2024-03-04T09:25:21.149691+0000","last_fullsized":"2024-03-04T09:25:21.149691+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:50:18.265156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.497632+0000","last_change":"2024-03-04T09:25:18.470800+0000","last_active":"2024-03-04T09:25:20.497632+0000","last_peered":"2024-03-04T09:25:20.497632+0000","last_clean":"2024-03-04T09:25:20.497632+0000","last_became_active":"2024-03-04T09:25:18.470431+0000","last_became_peered":"2024-03-04T09:25:18.470431+0000","last_unstale":"2024-03-04T09:25:20.497632+0000","last_undegraded":"2024-03-04T09:25:20.497632+0000","last_fullsized":"2024-03-04T09:25:20.497632+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:39:20.774577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.151318+0000","last_change":"2024-03-04T09:25:19.478795+0000","last_active":"2024-03-04T09:25:21.151318+0000","last_peered":"2024-03-04T09:25:21.151318+0000","last_clean":"2024-03-04T09:25:21.151318+0000","last_became_active":"2024-03-04T09:25:19.478635+0000","last_became_peered":"2024-03-04T09:25:19.478635+0000","last_unstale":"2024-03-04T09:25:21.151318+0000","last_undegraded":"2024-03-04T09:25:21.151318+0000","last_fullsized":"2024-03-04T09:25:21.151318+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:04:20.349240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.463677+0000","last_change":"2024-03-04T09:25:18.474750+0000","last_active":"2024-03-04T09:25:20.463677+0000","last_peered":"2024-03-04T09:25:20.463677+0000","last_clean":"2024-03-04T09:25:20.463677+0000","last_became_active":"2024-03-04T09:25:18.473463+0000","last_became_peered":"2024-03-04T09:25:18.473463+0000","last_unstale":"2024-03-04T09:25:20.463677+0000","last_undegraded":"2024-03-04T09:25:20.463677+0000","last_fullsized":"2024-03-04T09:25:20.463677+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:38:16.906069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.463789+0000","last_change":"2024-03-04T09:25:19.472274+0000","last_active":"2024-03-04T09:25:20.463789+0000","last_peered":"2024-03-04T09:25:20.463789+0000","last_clean":"2024-03-04T09:25:20.463789+0000","last_became_active":"2024-03-04T09:25:19.470669+0000","last_became_peered":"2024-03-04T09:25:19.470669+0000","last_unstale":"2024-03-04T09:25:20.463789+0000","last_undegraded":"2024-03-04T09:25:20.463789+0000","last_fullsized":"2024-03-04T09:25:20.463789+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:50:18.265156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":22,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.456829+0000","last_change":"2024-03-04T09:25:18.462460+0000","last_active":"2024-03-04T09:25:20.456829+0000","last_peered":"2024-03-04T09:25:20.456829+0000","last_clean":"2024-03-04T09:25:20.456829+0000","last_became_active":"2024-03-04T09:25:18.460692+0000","last_became_peered":"2024-03-04T09:25:18.460692+0000","last_unstale":"2024-03-04T09:25:20.456829+0000","last_undegraded":"2024-03-04T09:25:20.456829+0000","last_fullsized":"2024-03-04T09:25:20.456829+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:38:16.906069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.150513+0000","last_change":"2024-03-04T09:25:19.478013+0000","last_active":"2024-03-04T09:25:21.150513+0000","last_peered":"2024-03-04T09:25:21.150513+0000","last_clean":"2024-03-04T09:25:21.150513+0000","last_became_active":"2024-03-04T09:25:19.477150+0000","last_became_peered":"2024-03-04T09:25:19.477150+0000","last_unstale":"2024-03-04T09:25:21.150513+0000","last_undegraded":"2024-03-04T09:25:21.150513+0000","last_fullsized":"2024-03-04T09:25:21.150513+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:11:11.501102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.152408+0000","last_change":"2024-03-04T09:25:18.468798+0000","last_active":"2024-03-04T09:25:21.152408+0000","last_peered":"2024-03-04T09:25:21.152408+0000","last_clean":"2024-03-04T09:25:21.152408+0000","last_became_active":"2024-03-04T09:25:18.468412+0000","last_became_peered":"2024-03-04T09:25:18.468412+0000","last_unstale":"2024-03-04T09:25:21.152408+0000","last_undegraded":"2024-03-04T09:25:21.152408+0000","last_fullsized":"2024-03-04T09:25:21.152408+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:17:37.720962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.151303+0000","last_change":"2024-03-04T09:25:19.478586+0000","last_active":"2024-03-04T09:25:21.151303+0000","last_peered":"2024-03-04T09:25:21.151303+0000","last_clean":"2024-03-04T09:25:21.151303+0000","last_became_active":"2024-03-04T09:25:19.478339+0000","last_became_peered":"2024-03-04T09:25:19.478339+0000","last_unstale":"2024-03-04T09:25:21.151303+0000","last_undegraded":"2024-03-04T09:25:21.151303+0000","last_fullsized":"2024-03-04T09:25:21.151303+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:20:25.135538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.458139+0000","last_change":"2024-03-04T09:25:18.459079+0000","last_active":"2024-03-04T09:25:20.458139+0000","last_peered":"2024-03-04T09:25:20.458139+0000","last_clean":"2024-03-04T09:25:20.458139+0000","last_became_active":"2024-03-04T09:25:18.458767+0000","last_became_peered":"2024-03-04T09:25:18.458767+0000","last_unstale":"2024-03-04T09:25:20.458139+0000","last_undegraded":"2024-03-04T09:25:20.458139+0000","last_fullsized":"2024-03-04T09:25:20.458139+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:33:35.151096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.152708+0000","last_change":"2024-03-04T09:25:19.481294+0000","last_active":"2024-03-04T09:25:21.152708+0000","last_peered":"2024-03-04T09:25:21.152708+0000","last_clean":"2024-03-04T09:25:21.152708+0000","last_became_active":"2024-03-04T09:25:19.479163+0000","last_became_peered":"2024-03-04T09:25:19.479163+0000","last_unstale":"2024-03-04T09:25:21.152708+0000","last_undegraded":"2024-03-04T09:25:21.152708+0000","last_fullsized":"2024-03-04T09:25:21.152708+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:36:31.167806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.457070+0000","last_change":"2024-03-04T09:25:18.464660+0000","last_active":"2024-03-04T09:25:20.457070+0000","last_peered":"2024-03-04T09:25:20.457070+0000","last_clean":"2024-03-04T09:25:20.457070+0000","last_became_active":"2024-03-04T09:25:18.464412+0000","last_became_peered":"2024-03-04T09:25:18.464412+0000","last_unstale":"2024-03-04T09:25:20.457070+0000","last_undegraded":"2024-03-04T09:25:20.457070+0000","last_fullsized":"2024-03-04T09:25:20.457070+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:32:58.904081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.498417+0000","last_change":"2024-03-04T09:25:19.480725+0000","last_active":"2024-03-04T09:25:20.498417+0000","last_peered":"2024-03-04T09:25:20.498417+0000","last_clean":"2024-03-04T09:25:20.498417+0000","last_became_active":"2024-03-04T09:25:19.468626+0000","last_became_peered":"2024-03-04T09:25:19.468626+0000","last_unstale":"2024-03-04T09:25:20.498417+0000","last_undegraded":"2024-03-04T09:25:20.498417+0000","last_fullsized":"2024-03-04T09:25:20.498417+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:36:55.714408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.457915+0000","last_change":"2024-03-04T09:25:18.479022+0000","last_active":"2024-03-04T09:25:20.457915+0000","last_peered":"2024-03-04T09:25:20.457915+0000","last_clean":"2024-03-04T09:25:20.457915+0000","last_became_active":"2024-03-04T09:25:18.478697+0000","last_became_peered":"2024-03-04T09:25:18.478697+0000","last_unstale":"2024-03-04T09:25:20.457915+0000","last_undegraded":"2024-03-04T09:25:20.457915+0000","last_fullsized":"2024-03-04T09:25:20.457915+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:39:20.774577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.151536+0000","last_change":"2024-03-04T09:25:19.485083+0000","last_active":"2024-03-04T09:25:21.151536+0000","last_peered":"2024-03-04T09:25:21.151536+0000","last_clean":"2024-03-04T09:25:21.151536+0000","last_became_active":"2024-03-04T09:25:19.484619+0000","last_became_peered":"2024-03-04T09:25:19.484619+0000","last_unstale":"2024-03-04T09:25:21.151536+0000","last_undegraded":"2024-03-04T09:25:21.151536+0000","last_fullsized":"2024-03-04T09:25:21.151536+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:39:25.865493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":170,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.150247+0000","last_change":"2024-03-04T09:25:09.316159+0000","last_active":"2024-03-04T09:25:21.150247+0000","last_peered":"2024-03-04T09:25:21.150247+0000","last_clean":"2024-03-04T09:25:21.150247+0000","last_became_active":"2024-03-04T09:25:09.313767+0000","last_became_peered":"2024-03-04T09:25:09.313767+0000","last_unstale":"2024-03-04T09:25:21.150247+0000","last_undegraded":"2024-03-04T09:25:21.150247+0000","last_fullsized":"2024-03-04T09:25:21.150247+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:08.110953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:08.110953+0000","last_clean_scrub_stamp":"2024-03-04T09:25:08.110953+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:34:21.745390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.497328+0000","last_change":"2024-03-04T09:25:18.464859+0000","last_active":"2024-03-04T09:25:20.497328+0000","last_peered":"2024-03-04T09:25:20.497328+0000","last_clean":"2024-03-04T09:25:20.497328+0000","last_became_active":"2024-03-04T09:25:18.463515+0000","last_became_peered":"2024-03-04T09:25:18.463515+0000","last_unstale":"2024-03-04T09:25:20.497328+0000","last_undegraded":"2024-03-04T09:25:20.497328+0000","last_fullsized":"2024-03-04T09:25:20.497328+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:38:16.906069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.456773+0000","last_change":"2024-03-04T09:25:19.460896+0000","last_active":"2024-03-04T09:25:20.456773+0000","last_peered":"2024-03-04T09:25:20.456773+0000","last_clean":"2024-03-04T09:25:20.456773+0000","last_became_active":"2024-03-04T09:25:19.460598+0000","last_became_peered":"2024-03-04T09:25:19.460598+0000","last_unstale":"2024-03-04T09:25:20.456773+0000","last_undegraded":"2024-03-04T09:25:20.456773+0000","last_fullsized":"2024-03-04T09:25:20.456773+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:33:58.076780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.149994+0000","last_change":"2024-03-04T09:25:18.480278+0000","last_active":"2024-03-04T09:25:21.149994+0000","last_peered":"2024-03-04T09:25:21.149994+0000","last_clean":"2024-03-04T09:25:21.149994+0000","last_became_active":"2024-03-04T09:25:18.478848+0000","last_became_peered":"2024-03-04T09:25:18.478848+0000","last_unstale":"2024-03-04T09:25:21.149994+0000","last_undegraded":"2024-03-04T09:25:21.149994+0000","last_fullsized":"2024-03-04T09:25:21.149994+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:54:01.985207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.149481+0000","last_change":"2024-03-04T09:25:19.473908+0000","last_active":"2024-03-04T09:25:21.149481+0000","last_peered":"2024-03-04T09:25:21.149481+0000","last_clean":"2024-03-04T09:25:21.149481+0000","last_became_active":"2024-03-04T09:25:19.472415+0000","last_became_peered":"2024-03-04T09:25:19.472415+0000","last_unstale":"2024-03-04T09:25:21.149481+0000","last_undegraded":"2024-03-04T09:25:21.149481+0000","last_fullsized":"2024-03-04T09:25:21.149481+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:46:44.342046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.457503+0000","last_change":"2024-03-04T09:25:18.462877+0000","last_active":"2024-03-04T09:25:20.457503+0000","last_peered":"2024-03-04T09:25:20.457503+0000","last_clean":"2024-03-04T09:25:20.457503+0000","last_became_active":"2024-03-04T09:25:18.459553+0000","last_became_peered":"2024-03-04T09:25:18.459553+0000","last_unstale":"2024-03-04T09:25:20.457503+0000","last_undegraded":"2024-03-04T09:25:20.457503+0000","last_fullsized":"2024-03-04T09:25:20.457503+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:49:03.118927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.456676+0000","last_change":"2024-03-04T09:25:19.452298+0000","last_active":"2024-03-04T09:25:20.456676+0000","last_peered":"2024-03-04T09:25:20.456676+0000","last_clean":"2024-03-04T09:25:20.456676+0000","last_became_active":"2024-03-04T09:25:19.451945+0000","last_became_peered":"2024-03-04T09:25:19.451945+0000","last_unstale":"2024-03-04T09:25:20.456676+0000","last_undegraded":"2024-03-04T09:25:20.456676+0000","last_fullsized":"2024-03-04T09:25:20.456676+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:56:18.833810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.497812+0000","last_change":"2024-03-04T09:25:18.473872+0000","last_active":"2024-03-04T09:25:20.497812+0000","last_peered":"2024-03-04T09:25:20.497812+0000","last_clean":"2024-03-04T09:25:20.497812+0000","last_became_active":"2024-03-04T09:25:18.473390+0000","last_became_peered":"2024-03-04T09:25:18.473390+0000","last_unstale":"2024-03-04T09:25:20.497812+0000","last_undegraded":"2024-03-04T09:25:20.497812+0000","last_fullsized":"2024-03-04T09:25:20.497812+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:23:34.932850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.150539+0000","last_change":"2024-03-04T09:25:19.467593+0000","last_active":"2024-03-04T09:25:21.150539+0000","last_peered":"2024-03-04T09:25:21.150539+0000","last_clean":"2024-03-04T09:25:21.150539+0000","last_became_active":"2024-03-04T09:25:19.467079+0000","last_became_peered":"2024-03-04T09:25:19.467079+0000","last_unstale":"2024-03-04T09:25:21.150539+0000","last_undegraded":"2024-03-04T09:25:21.150539+0000","last_fullsized":"2024-03-04T09:25:21.150539+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:11:52.240719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.456743+0000","last_change":"2024-03-04T09:25:18.465721+0000","last_active":"2024-03-04T09:25:20.456743+0000","last_peered":"2024-03-04T09:25:20.456743+0000","last_clean":"2024-03-04T09:25:20.456743+0000","last_became_active":"2024-03-04T09:25:18.465438+0000","last_became_peered":"2024-03-04T09:25:18.465438+0000","last_unstale":"2024-03-04T09:25:20.456743+0000","last_undegraded":"2024-03-04T09:25:20.456743+0000","last_fullsized":"2024-03-04T09:25:20.456743+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:39:20.774577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.498362+0000","last_change":"2024-03-04T09:25:19.472271+0000","last_active":"2024-03-04T09:25:20.498362+0000","last_peered":"2024-03-04T09:25:20.498362+0000","last_clean":"2024-03-04T09:25:20.498362+0000","last_became_active":"2024-03-04T09:25:19.467596+0000","last_became_peered":"2024-03-04T09:25:19.467596+0000","last_unstale":"2024-03-04T09:25:20.498362+0000","last_undegraded":"2024-03-04T09:25:20.498362+0000","last_fullsized":"2024-03-04T09:25:20.498362+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:12:45.452669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.151714+0000","last_change":"2024-03-04T09:25:18.481607+0000","last_active":"2024-03-04T09:25:21.151714+0000","last_peered":"2024-03-04T09:25:21.151714+0000","last_clean":"2024-03-04T09:25:21.151714+0000","last_became_active":"2024-03-04T09:25:18.481270+0000","last_became_peered":"2024-03-04T09:25:18.481270+0000","last_unstale":"2024-03-04T09:25:21.151714+0000","last_undegraded":"2024-03-04T09:25:21.151714+0000","last_fullsized":"2024-03-04T09:25:21.151714+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:52:36.632272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.455410+0000","last_change":"2024-03-04T09:25:19.450231+0000","last_active":"2024-03-04T09:25:20.455410+0000","last_peered":"2024-03-04T09:25:20.455410+0000","last_clean":"2024-03-04T09:25:20.455410+0000","last_became_active":"2024-03-04T09:25:19.449920+0000","last_became_peered":"2024-03-04T09:25:19.449920+0000","last_unstale":"2024-03-04T09:25:20.455410+0000","last_undegraded":"2024-03-04T09:25:20.455410+0000","last_fullsized":"2024-03-04T09:25:20.455410+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:45:59.096795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.457374+0000","last_change":"2024-03-04T09:25:18.472718+0000","last_active":"2024-03-04T09:25:20.457374+0000","last_peered":"2024-03-04T09:25:20.457374+0000","last_clean":"2024-03-04T09:25:20.457374+0000","last_became_active":"2024-03-04T09:25:18.472426+0000","last_became_peered":"2024-03-04T09:25:18.472426+0000","last_unstale":"2024-03-04T09:25:20.457374+0000","last_undegraded":"2024-03-04T09:25:20.457374+0000","last_fullsized":"2024-03-04T09:25:20.457374+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:23:34.932850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.458202+0000","last_change":"2024-03-04T09:25:19.470531+0000","last_active":"2024-03-04T09:25:20.458202+0000","last_peered":"2024-03-04T09:25:20.458202+0000","last_clean":"2024-03-04T09:25:20.458202+0000","last_became_active":"2024-03-04T09:25:19.469607+0000","last_became_peered":"2024-03-04T09:25:19.469607+0000","last_unstale":"2024-03-04T09:25:20.458202+0000","last_undegraded":"2024-03-04T09:25:20.458202+0000","last_fullsized":"2024-03-04T09:25:20.458202+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:11:11.501102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.464561+0000","last_change":"2024-03-04T09:25:18.473250+0000","last_active":"2024-03-04T09:25:20.464561+0000","last_peered":"2024-03-04T09:25:20.464561+0000","last_clean":"2024-03-04T09:25:20.464561+0000","last_became_active":"2024-03-04T09:25:18.472968+0000","last_became_peered":"2024-03-04T09:25:18.472968+0000","last_unstale":"2024-03-04T09:25:20.464561+0000","last_undegraded":"2024-03-04T09:25:20.464561+0000","last_fullsized":"2024-03-04T09:25:20.464561+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:39:20.774577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.150075+0000","last_change":"2024-03-04T09:25:19.467410+0000","last_active":"2024-03-04T09:25:21.150075+0000","last_peered":"2024-03-04T09:25:21.150075+0000","last_clean":"2024-03-04T09:25:21.150075+0000","last_became_active":"2024-03-04T09:25:19.467101+0000","last_became_peered":"2024-03-04T09:25:19.467101+0000","last_unstale":"2024-03-04T09:25:21.150075+0000","last_undegraded":"2024-03-04T09:25:21.150075+0000","last_fullsized":"2024-03-04T09:25:21.150075+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:44:37.465937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.149988+0000","last_change":"2024-03-04T09:25:18.475507+0000","last_active":"2024-03-04T09:25:21.149988+0000","last_peered":"2024-03-04T09:25:21.149988+0000","last_clean":"2024-03-04T09:25:21.149988+0000","last_became_active":"2024-03-04T09:25:18.475159+0000","last_became_peered":"2024-03-04T09:25:18.475159+0000","last_unstale":"2024-03-04T09:25:21.149988+0000","last_undegraded":"2024-03-04T09:25:21.149988+0000","last_fullsized":"2024-03-04T09:25:21.149988+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:23:34.932850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.149845+0000","last_change":"2024-03-04T09:25:19.453119+0000","last_active":"2024-03-04T09:25:21.149845+0000","last_peered":"2024-03-04T09:25:21.149845+0000","last_clean":"2024-03-04T09:25:21.149845+0000","last_became_active":"2024-03-04T09:25:19.452764+0000","last_became_peered":"2024-03-04T09:25:19.452764+0000","last_unstale":"2024-03-04T09:25:21.149845+0000","last_undegraded":"2024-03-04T09:25:21.149845+0000","last_fullsized":"2024-03-04T09:25:21.149845+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:54:02.999509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.456033+0000","last_change":"2024-03-04T09:25:18.473872+0000","last_active":"2024-03-04T09:25:20.456033+0000","last_peered":"2024-03-04T09:25:20.456033+0000","last_clean":"2024-03-04T09:25:20.456033+0000","last_became_active":"2024-03-04T09:25:18.473554+0000","last_became_peered":"2024-03-04T09:25:18.473554+0000","last_unstale":"2024-03-04T09:25:20.456033+0000","last_undegraded":"2024-03-04T09:25:20.456033+0000","last_fullsized":"2024-03-04T09:25:20.456033+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:11:51.226417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.455488+0000","last_change":"2024-03-04T09:25:19.458923+0000","last_active":"2024-03-04T09:25:20.455488+0000","last_peered":"2024-03-04T09:25:20.455488+0000","last_clean":"2024-03-04T09:25:20.455488+0000","last_became_active":"2024-03-04T09:25:19.458562+0000","last_became_peered":"2024-03-04T09:25:19.458562+0000","last_unstale":"2024-03-04T09:25:20.455488+0000","last_undegraded":"2024-03-04T09:25:20.455488+0000","last_fullsized":"2024-03-04T09:25:20.455488+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:45:59.096795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.456036+0000","last_change":"2024-03-04T09:25:18.458201+0000","last_active":"2024-03-04T09:25:20.456036+0000","last_peered":"2024-03-04T09:25:20.456036+0000","last_clean":"2024-03-04T09:25:20.456036+0000","last_became_active":"2024-03-04T09:25:18.455667+0000","last_became_peered":"2024-03-04T09:25:18.455667+0000","last_unstale":"2024-03-04T09:25:20.456036+0000","last_undegraded":"2024-03-04T09:25:20.456036+0000","last_fullsized":"2024-03-04T09:25:20.456036+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:38:16.906069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.456043+0000","last_change":"2024-03-04T09:25:19.464390+0000","last_active":"2024-03-04T09:25:20.456043+0000","last_peered":"2024-03-04T09:25:20.456043+0000","last_clean":"2024-03-04T09:25:20.456043+0000","last_became_active":"2024-03-04T09:25:19.463511+0000","last_became_peered":"2024-03-04T09:25:19.463511+0000","last_unstale":"2024-03-04T09:25:20.456043+0000","last_undegraded":"2024-03-04T09:25:20.456043+0000","last_fullsized":"2024-03-04T09:25:20.456043+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:44:37.465937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.151481+0000","last_change":"2024-03-04T09:25:18.479732+0000","last_active":"2024-03-04T09:25:21.151481+0000","last_peered":"2024-03-04T09:25:21.151481+0000","last_clean":"2024-03-04T09:25:21.151481+0000","last_became_active":"2024-03-04T09:25:18.478014+0000","last_became_peered":"2024-03-04T09:25:18.478014+0000","last_unstale":"2024-03-04T09:25:21.151481+0000","last_undegraded":"2024-03-04T09:25:21.151481+0000","last_fullsized":"2024-03-04T09:25:21.151481+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:32:58.904081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.464302+0000","last_change":"2024-03-04T09:25:19.455001+0000","last_active":"2024-03-04T09:25:20.464302+0000","last_peered":"2024-03-04T09:25:20.464302+0000","last_clean":"2024-03-04T09:25:20.464302+0000","last_became_active":"2024-03-04T09:25:19.454695+0000","last_became_peered":"2024-03-04T09:25:19.454695+0000","last_unstale":"2024-03-04T09:25:20.464302+0000","last_undegraded":"2024-03-04T09:25:20.464302+0000","last_fullsized":"2024-03-04T09:25:20.464302+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:47:26.445286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.498167+0000","last_change":"2024-03-04T09:25:19.452288+0000","last_active":"2024-03-04T09:25:20.498167+0000","last_peered":"2024-03-04T09:25:20.498167+0000","last_clean":"2024-03-04T09:25:20.498167+0000","last_became_active":"2024-03-04T09:25:19.451982+0000","last_became_peered":"2024-03-04T09:25:19.451982+0000","last_unstale":"2024-03-04T09:25:20.498167+0000","last_undegraded":"2024-03-04T09:25:20.498167+0000","last_fullsized":"2024-03-04T09:25:20.498167+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:39:25.865493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.151027+0000","last_change":"2024-03-04T09:25:18.481511+0000","last_active":"2024-03-04T09:25:21.151027+0000","last_peered":"2024-03-04T09:25:21.151027+0000","last_clean":"2024-03-04T09:25:21.151027+0000","last_became_active":"2024-03-04T09:25:18.481207+0000","last_became_peered":"2024-03-04T09:25:18.481207+0000","last_unstale":"2024-03-04T09:25:21.151027+0000","last_undegraded":"2024-03-04T09:25:21.151027+0000","last_fullsized":"2024-03-04T09:25:21.151027+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:11:51.226417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.498103+0000","last_change":"2024-03-04T09:25:19.453592+0000","last_active":"2024-03-04T09:25:20.498103+0000","last_peered":"2024-03-04T09:25:20.498103+0000","last_clean":"2024-03-04T09:25:20.498103+0000","last_became_active":"2024-03-04T09:25:19.453318+0000","last_became_peered":"2024-03-04T09:25:19.453318+0000","last_unstale":"2024-03-04T09:25:20.498103+0000","last_undegraded":"2024-03-04T09:25:20.498103+0000","last_fullsized":"2024-03-04T09:25:20.498103+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:04:20.349240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.151412+0000","last_change":"2024-03-04T09:25:18.483252+0000","last_active":"2024-03-04T09:25:21.151412+0000","last_peered":"2024-03-04T09:25:21.151412+0000","last_clean":"2024-03-04T09:25:21.151412+0000","last_became_active":"2024-03-04T09:25:18.482987+0000","last_became_peered":"2024-03-04T09:25:18.482987+0000","last_unstale":"2024-03-04T09:25:21.151412+0000","last_undegraded":"2024-03-04T09:25:21.151412+0000","last_fullsized":"2024-03-04T09:25:21.151412+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:02:44.764307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.150110+0000","last_change":"2024-03-04T09:25:19.473423+0000","last_active":"2024-03-04T09:25:21.150110+0000","last_peered":"2024-03-04T09:25:21.150110+0000","last_clean":"2024-03-04T09:25:21.150110+0000","last_became_active":"2024-03-04T09:25:19.473030+0000","last_became_peered":"2024-03-04T09:25:19.473030+0000","last_unstale":"2024-03-04T09:25:21.150110+0000","last_undegraded":"2024-03-04T09:25:21.150110+0000","last_fullsized":"2024-03-04T09:25:21.150110+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:56:18.833810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.458390+0000","last_change":"2024-03-04T09:25:18.466373+0000","last_active":"2024-03-04T09:25:20.458390+0000","last_peered":"2024-03-04T09:25:20.458390+0000","last_clean":"2024-03-04T09:25:20.458390+0000","last_became_active":"2024-03-04T09:25:18.466069+0000","last_became_peered":"2024-03-04T09:25:18.466069+0000","last_unstale":"2024-03-04T09:25:20.458390+0000","last_undegraded":"2024-03-04T09:25:20.458390+0000","last_fullsized":"2024-03-04T09:25:20.458390+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:11:51.226417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.458472+0000","last_change":"2024-03-04T09:25:19.459312+0000","last_active":"2024-03-04T09:25:20.458472+0000","last_peered":"2024-03-04T09:25:20.458472+0000","last_clean":"2024-03-04T09:25:20.458472+0000","last_became_active":"2024-03-04T09:25:19.458993+0000","last_became_peered":"2024-03-04T09:25:19.458993+0000","last_unstale":"2024-03-04T09:25:20.458472+0000","last_undegraded":"2024-03-04T09:25:20.458472+0000","last_fullsized":"2024-03-04T09:25:20.458472+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:36:58.654226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"19'4","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.150271+0000","last_change":"2024-03-04T09:25:18.473489+0000","last_active":"2024-03-04T09:25:21.150271+0000","last_peered":"2024-03-04T09:25:21.150271+0000","last_clean":"2024-03-04T09:25:21.150271+0000","last_became_active":"2024-03-04T09:25:18.473154+0000","last_became_peered":"2024-03-04T09:25:18.473154+0000","last_unstale":"2024-03-04T09:25:21.150271+0000","last_undegraded":"2024-03-04T09:25:21.150271+0000","last_fullsized":"2024-03-04T09:25:21.150271+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:31:15.318756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574853,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27804,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344036,"statfs":{"total":96636764160,"available":96608292864,"internally_reserved":0,"allocated":802816,"data_stored":661884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574853,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":72546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574855,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":208896,"data_stored":71024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574852,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":71060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574854,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":72546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574853,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":70934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574854,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":212992,"data_stored":71094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574853,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27816,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344024,"statfs":{"total":96636764160,"available":96608280576,"internally_reserved":0,"allocated":811008,"data_stored":661940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-04T09:25:24.004 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T09:25:24.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.265+0000 7fe7e4e00640 1 Processor -- start 2024-03-04T09:25:24.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.266+0000 7fe7e4e00640 1 -- start start 2024-03-04T09:25:24.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.267+0000 7fe7e4e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe7e015e0b0 0x7fe7e00cf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:24.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.267+0000 7fe7e4e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fe7e00d0e50 0x7fe7e00b44d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:24.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.267+0000 7fe7e4e00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7e00b3710 con 0x7fe7e00d4a10 2024-03-04T09:25:24.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.268+0000 7fe7e4e00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7e00b09e0 con 0x7fe7e00d0e50 2024-03-04T09:25:24.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.268+0000 7fe7e4e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7e00b39f0 con 0x7fe7e015e0b0 2024-03-04T09:25:24.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.268+0000 7fe7df400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe7e015e0b0 0x7fe7e00cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:24.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.268+0000 7fe7df400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe7e015e0b0 0x7fe7e00cf310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.6:33142/0 (socket says 172.21.15.6:33142) 2024-03-04T09:25:24.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.268+0000 7fe7df400640 1 -- 172.21.15.6:0/3053832623 learned_addr learned my addr 172.21.15.6:0/3053832623 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:25:24.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.268+0000 7fe7dfe00640 1 --2- 172.21.15.6:0/3053832623 >> v2:172.21.15.146:3300/0 conn(0x7fe7e00d0e50 0x7fe7e00b44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:24.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.269+0000 7fe7df400640 1 -- 172.21.15.6:0/3053832623 >> v2:172.21.15.146:3300/0 conn(0x7fe7e00d0e50 msgr2=0x7fe7e00b44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:24.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.269+0000 7fe7df400640 1 --2- 172.21.15.6:0/3053832623 >> v2:172.21.15.146:3300/0 conn(0x7fe7e00d0e50 0x7fe7e00b44d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:24.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.269+0000 7fe7df400640 1 -- 172.21.15.6:0/3053832623 >> v1:172.21.15.146:6789/0 conn(0x7fe7e00d4a10 legacy=0x7fe7e00d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:24.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.269+0000 7fe7df400640 1 -- 172.21.15.6:0/3053832623 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7e00b3cd0 con 0x7fe7e015e0b0 2024-03-04T09:25:24.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.270+0000 7fe7df400640 1 --2- 172.21.15.6:0/3053832623 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe7e015e0b0 0x7fe7e00cf310 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fe7d0009e50 tx=0x7fe7d002bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5a356832e7c39adf server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:24.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.271+0000 7fe7de000640 1 -- 172.21.15.6:0/3053832623 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7d002cb00 con 0x7fe7e015e0b0 2024-03-04T09:25:24.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.271+0000 7fe7de000640 1 -- 172.21.15.6:0/3053832623 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7d002cca0 con 0x7fe7e015e0b0 2024-03-04T09:25:24.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.271+0000 7fe7de000640 1 -- 172.21.15.6:0/3053832623 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7d0034610 con 0x7fe7e015e0b0 2024-03-04T09:25:24.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.271+0000 7fe7e4e00640 1 -- 172.21.15.6:0/3053832623 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe7e015e0b0 msgr2=0x7fe7e00cf310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:24.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.271+0000 7fe7e4e00640 1 --2- 172.21.15.6:0/3053832623 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe7e015e0b0 0x7fe7e00cf310 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fe7d0009e50 tx=0x7fe7d002bf60 comp rx=0 tx=0).stop 2024-03-04T09:25:24.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.271+0000 7fe7e4e00640 1 -- 172.21.15.6:0/3053832623 shutdown_connections 2024-03-04T09:25:24.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.271+0000 7fe7e4e00640 1 --2- 172.21.15.6:0/3053832623 >> v2:172.21.15.146:3300/0 conn(0x7fe7e00d0e50 0x7fe7e00b44d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:24.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.271+0000 7fe7e4e00640 1 --2- 172.21.15.6:0/3053832623 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe7e015e0b0 0x7fe7e00cf310 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:24.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.271+0000 7fe7e4e00640 1 -- 172.21.15.6:0/3053832623 >> 172.21.15.6:0/3053832623 conn(0x7fe7e00d3d30 msgr2=0x7fe7e00d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:24.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.271+0000 7fe7e4e00640 1 -- 172.21.15.6:0/3053832623 shutdown_connections 2024-03-04T09:25:24.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.271+0000 7fe7e4e00640 1 -- 172.21.15.6:0/3053832623 wait complete. 2024-03-04T09:25:24.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.273+0000 7fe7e4e00640 1 Processor -- start 2024-03-04T09:25:24.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.273+0000 7fe7e4e00640 1 -- start start 2024-03-04T09:25:24.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.274+0000 7fe7e4e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe7e00d0e50 0x7fe7e00cae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:24.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.274+0000 7fe7e4e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe7e00d4a10 0x7fe7e0158260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:24.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.274+0000 7fe7e4e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe7e015e0b0 0x7fe7e015ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:24.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.274+0000 7fe7e4e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7e00cfc60 con 0x7fe7e00d0e50 2024-03-04T09:25:24.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.274+0000 7fe7e4e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7e015e490 con 0x7fe7e015e0b0 2024-03-04T09:25:24.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.274+0000 7fe7e4e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7e00cf850 con 0x7fe7e00d4a10 2024-03-04T09:25:24.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.275+0000 7fe7df400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe7e00d0e50 0x7fe7e00cae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:24.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.275+0000 7fe7df400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe7e00d0e50 0x7fe7e00cae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45062/0 (socket says 172.21.15.6:45062) 2024-03-04T09:25:24.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.275+0000 7fe7df400640 1 -- 172.21.15.6:0/1021268533 learned_addr learned my addr 172.21.15.6:0/1021268533 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:25:24.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.275+0000 7fe7dfe00640 1 --2- 172.21.15.6:0/1021268533 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe7e015e0b0 0x7fe7e015ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:24.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.275+0000 7fe7dea00640 1 --2- 172.21.15.6:0/1021268533 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe7e00d4a10 0x7fe7e0158260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:24.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.275+0000 7fe7df400640 1 -- 172.21.15.6:0/1021268533 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe7e00d4a10 msgr2=0x7fe7e0158260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:24.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.275+0000 7fe7df400640 1 --2- 172.21.15.6:0/1021268533 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe7e00d4a10 0x7fe7e0158260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:24.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.275+0000 7fe7df400640 1 -- 172.21.15.6:0/1021268533 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe7e015e0b0 msgr2=0x7fe7e015ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:24.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.275+0000 7fe7df400640 1 --2- 172.21.15.6:0/1021268533 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe7e015e0b0 0x7fe7e015ef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:24.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.275+0000 7fe7df400640 1 -- 172.21.15.6:0/1021268533 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7e01f4fa0 con 0x7fe7e00d0e50 2024-03-04T09:25:24.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.276+0000 7fe7df400640 1 --2- 172.21.15.6:0/1021268533 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe7e00d0e50 0x7fe7e00cae00 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe7d0036040 tx=0x7fe7d0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:24.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.276+0000 7fe7d7400640 1 -- 172.21.15.6:0/1021268533 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7d0043020 con 0x7fe7e00d0e50 2024-03-04T09:25:24.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.276+0000 7fe7d7400640 1 -- 172.21.15.6:0/1021268533 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7d002ce50 con 0x7fe7e00d0e50 2024-03-04T09:25:24.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.276+0000 7fe7d7400640 1 -- 172.21.15.6:0/1021268533 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7d0009510 con 0x7fe7e00d0e50 2024-03-04T09:25:24.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.276+0000 7fe7e4e00640 1 -- 172.21.15.6:0/1021268533 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7e01f52e0 con 0x7fe7e00d0e50 2024-03-04T09:25:24.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.276+0000 7fe7e4e00640 1 -- 172.21.15.6:0/1021268533 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7e01f5b20 con 0x7fe7e00d0e50 2024-03-04T09:25:24.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.277+0000 7fe7e4e00640 1 -- 172.21.15.6:0/1021268533 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7e004f330 con 0x7fe7e00d0e50 2024-03-04T09:25:24.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.278+0000 7fe7d7400640 1 -- 172.21.15.6:0/1021268533 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe7d0046020 con 0x7fe7e00d0e50 2024-03-04T09:25:24.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.279+0000 7fe7d7400640 1 --2- 172.21.15.6:0/1021268533 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fe7c006b4d0 0x7fe7c006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:24.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.279+0000 7fe7d7400640 1 -- 172.21.15.6:0/1021268533 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe7d00ac980 con 0x7fe7e00d0e50 2024-03-04T09:25:24.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.280+0000 7fe7dea00640 1 --2- 172.21.15.6:0/1021268533 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fe7c006b4d0 0x7fe7c006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:24.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.281+0000 7fe7dea00640 1 --2- 172.21.15.6:0/1021268533 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fe7c006b4d0 0x7fe7c006d990 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fe7e00b5220 tx=0x7fe7cc009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:24.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.284+0000 7fe7d7400640 1 -- 172.21.15.6:0/1021268533 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe7d004d030 con 0x7fe7e00d0e50 2024-03-04T09:25:24.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.481+0000 7fe7e4e00640 1 -- 172.21.15.6:0/1021268533 --> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe7e00b2220 con 0x7fe7c006b4d0 2024-03-04T09:25:24.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.501+0000 7fe7d7400640 1 -- 172.21.15.6:0/1021268533 <== mgr.4113 v2:172.21.15.146:6832/1788613334 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251952 (secure 0 0 0) 0x7fe7e00b2220 con 0x7fe7c006b4d0 2024-03-04T09:25:24.502 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:25:24.504 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-03-04T09:25:24.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.506+0000 7fe7e4e00640 1 -- 172.21.15.6:0/1021268533 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fe7c006b4d0 msgr2=0x7fe7c006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:24.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.506+0000 7fe7e4e00640 1 --2- 172.21.15.6:0/1021268533 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fe7c006b4d0 0x7fe7c006d990 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fe7e00b5220 tx=0x7fe7cc009040 comp rx=0 tx=0).stop 2024-03-04T09:25:24.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.506+0000 7fe7e4e00640 1 -- 172.21.15.6:0/1021268533 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe7e00d0e50 msgr2=0x7fe7e00cae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:24.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.506+0000 7fe7e4e00640 1 --2- 172.21.15.6:0/1021268533 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe7e00d0e50 0x7fe7e00cae00 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe7d0036040 tx=0x7fe7d0004a40 comp rx=0 tx=0).stop 2024-03-04T09:25:24.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.506+0000 7fe7e4e00640 1 -- 172.21.15.6:0/1021268533 shutdown_connections 2024-03-04T09:25:24.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.506+0000 7fe7e4e00640 1 --2- 172.21.15.6:0/1021268533 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe7e015e0b0 0x7fe7e015ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:24.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.506+0000 7fe7e4e00640 1 --2- 172.21.15.6:0/1021268533 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fe7c006b4d0 0x7fe7c006d990 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:24.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.506+0000 7fe7e4e00640 1 --2- 172.21.15.6:0/1021268533 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe7e00d4a10 0x7fe7e0158260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:24.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.506+0000 7fe7e4e00640 1 --2- 172.21.15.6:0/1021268533 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe7e00d0e50 0x7fe7e00cae00 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:24.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.506+0000 7fe7e4e00640 1 -- 172.21.15.6:0/1021268533 >> 172.21.15.6:0/1021268533 conn(0x7fe7e00d3d30 msgr2=0x7fe7e00b2bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:24.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.507+0000 7fe7e4e00640 1 -- 172.21.15.6:0/1021268533 shutdown_connections 2024-03-04T09:25:24.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.507+0000 7fe7e4e00640 1 -- 172.21.15.6:0/1021268533 wait complete. 2024-03-04T09:25:24.522 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-03-04T09:25:23.859677+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218988,"kb_used_data":2824,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755732,"statfs":{"total":773094113280,"available":772869869568,"internally_reserved":0,"allocated":2891776,"data_stored":1753028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":44,"apply_latency_ms":44,"commit_latency_ns":44000000,"apply_latency_ns":44000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002122"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.152203+0000","last_change":"2024-03-04T09:25:19.480195+0000","last_active":"2024-03-04T09:25:21.152203+0000","last_peered":"2024-03-04T09:25:21.152203+0000","last_clean":"2024-03-04T09:25:21.152203+0000","last_became_active":"2024-03-04T09:25:19.479496+0000","last_became_peered":"2024-03-04T09:25:19.479496+0000","last_unstale":"2024-03-04T09:25:21.152203+0000","last_undegraded":"2024-03-04T09:25:21.152203+0000","last_fullsized":"2024-03-04T09:25:21.152203+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:36:55.714408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.457360+0000","last_change":"2024-03-04T09:25:19.457568+0000","last_active":"2024-03-04T09:25:20.457360+0000","last_peered":"2024-03-04T09:25:20.457360+0000","last_clean":"2024-03-04T09:25:20.457360+0000","last_became_active":"2024-03-04T09:25:19.457288+0000","last_became_peered":"2024-03-04T09:25:19.457288+0000","last_unstale":"2024-03-04T09:25:20.457360+0000","last_undegraded":"2024-03-04T09:25:20.457360+0000","last_fullsized":"2024-03-04T09:25:20.457360+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:33:16.368258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.149774+0000","last_change":"2024-03-04T09:25:19.465855+0000","last_active":"2024-03-04T09:25:21.149774+0000","last_peered":"2024-03-04T09:25:21.149774+0000","last_clean":"2024-03-04T09:25:21.149774+0000","last_became_active":"2024-03-04T09:25:19.465489+0000","last_became_peered":"2024-03-04T09:25:19.465489+0000","last_unstale":"2024-03-04T09:25:21.149774+0000","last_undegraded":"2024-03-04T09:25:21.149774+0000","last_fullsized":"2024-03-04T09:25:21.149774+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:05:57.624897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.497998+0000","last_change":"2024-03-04T09:25:19.472849+0000","last_active":"2024-03-04T09:25:20.497998+0000","last_peered":"2024-03-04T09:25:20.497998+0000","last_clean":"2024-03-04T09:25:20.497998+0000","last_became_active":"2024-03-04T09:25:19.468252+0000","last_became_peered":"2024-03-04T09:25:19.468252+0000","last_unstale":"2024-03-04T09:25:20.497998+0000","last_undegraded":"2024-03-04T09:25:20.497998+0000","last_fullsized":"2024-03-04T09:25:20.497998+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:20:25.135538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.151031+0000","last_change":"2024-03-04T09:25:19.471747+0000","last_active":"2024-03-04T09:25:21.151031+0000","last_peered":"2024-03-04T09:25:21.151031+0000","last_clean":"2024-03-04T09:25:21.151031+0000","last_became_active":"2024-03-04T09:25:19.469589+0000","last_became_peered":"2024-03-04T09:25:19.469589+0000","last_unstale":"2024-03-04T09:25:21.151031+0000","last_undegraded":"2024-03-04T09:25:21.151031+0000","last_fullsized":"2024-03-04T09:25:21.151031+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:36:58.654226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.497539+0000","last_change":"2024-03-04T09:25:19.458354+0000","last_active":"2024-03-04T09:25:20.497539+0000","last_peered":"2024-03-04T09:25:20.497539+0000","last_clean":"2024-03-04T09:25:20.497539+0000","last_became_active":"2024-03-04T09:25:19.456693+0000","last_became_peered":"2024-03-04T09:25:19.456693+0000","last_unstale":"2024-03-04T09:25:20.497539+0000","last_undegraded":"2024-03-04T09:25:20.497539+0000","last_fullsized":"2024-03-04T09:25:20.497539+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:11:11.501102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.150488+0000","last_change":"2024-03-04T09:25:19.476465+0000","last_active":"2024-03-04T09:25:21.150488+0000","last_peered":"2024-03-04T09:25:21.150488+0000","last_clean":"2024-03-04T09:25:21.150488+0000","last_became_active":"2024-03-04T09:25:19.471519+0000","last_became_peered":"2024-03-04T09:25:19.471519+0000","last_unstale":"2024-03-04T09:25:21.150488+0000","last_undegraded":"2024-03-04T09:25:21.150488+0000","last_fullsized":"2024-03-04T09:25:21.150488+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:01:33.498760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.455119+0000","last_change":"2024-03-04T09:25:19.471301+0000","last_active":"2024-03-04T09:25:20.455119+0000","last_peered":"2024-03-04T09:25:20.455119+0000","last_clean":"2024-03-04T09:25:20.455119+0000","last_became_active":"2024-03-04T09:25:19.470983+0000","last_became_peered":"2024-03-04T09:25:19.470983+0000","last_unstale":"2024-03-04T09:25:20.455119+0000","last_undegraded":"2024-03-04T09:25:20.455119+0000","last_fullsized":"2024-03-04T09:25:20.455119+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:50:18.265156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.152495+0000","last_change":"2024-03-04T09:25:19.485751+0000","last_active":"2024-03-04T09:25:21.152495+0000","last_peered":"2024-03-04T09:25:21.152495+0000","last_clean":"2024-03-04T09:25:21.152495+0000","last_became_active":"2024-03-04T09:25:19.485488+0000","last_became_peered":"2024-03-04T09:25:19.485488+0000","last_unstale":"2024-03-04T09:25:21.152495+0000","last_undegraded":"2024-03-04T09:25:21.152495+0000","last_fullsized":"2024-03-04T09:25:21.152495+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:45:23.780438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.464582+0000","last_change":"2024-03-04T09:25:19.466385+0000","last_active":"2024-03-04T09:25:20.464582+0000","last_peered":"2024-03-04T09:25:20.464582+0000","last_clean":"2024-03-04T09:25:20.464582+0000","last_became_active":"2024-03-04T09:25:19.466078+0000","last_became_peered":"2024-03-04T09:25:19.466078+0000","last_unstale":"2024-03-04T09:25:20.464582+0000","last_undegraded":"2024-03-04T09:25:20.464582+0000","last_fullsized":"2024-03-04T09:25:20.464582+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:56:18.833810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.464502+0000","last_change":"2024-03-04T09:25:19.458668+0000","last_active":"2024-03-04T09:25:20.464502+0000","last_peered":"2024-03-04T09:25:20.464502+0000","last_clean":"2024-03-04T09:25:20.464502+0000","last_became_active":"2024-03-04T09:25:19.458080+0000","last_became_peered":"2024-03-04T09:25:19.458080+0000","last_unstale":"2024-03-04T09:25:20.464502+0000","last_undegraded":"2024-03-04T09:25:20.464502+0000","last_fullsized":"2024-03-04T09:25:20.464502+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:44:37.465937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.152136+0000","last_change":"2024-03-04T09:25:19.470591+0000","last_active":"2024-03-04T09:25:21.152136+0000","last_peered":"2024-03-04T09:25:21.152136+0000","last_clean":"2024-03-04T09:25:21.152136+0000","last_became_active":"2024-03-04T09:25:19.463604+0000","last_became_peered":"2024-03-04T09:25:19.463604+0000","last_unstale":"2024-03-04T09:25:21.152136+0000","last_undegraded":"2024-03-04T09:25:21.152136+0000","last_fullsized":"2024-03-04T09:25:21.152136+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:02:45.778608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.456030+0000","last_change":"2024-03-04T09:25:19.462024+0000","last_active":"2024-03-04T09:25:20.456030+0000","last_peered":"2024-03-04T09:25:20.456030+0000","last_clean":"2024-03-04T09:25:20.456030+0000","last_became_active":"2024-03-04T09:25:19.461767+0000","last_became_peered":"2024-03-04T09:25:19.461767+0000","last_unstale":"2024-03-04T09:25:20.456030+0000","last_undegraded":"2024-03-04T09:25:20.456030+0000","last_fullsized":"2024-03-04T09:25:20.456030+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:47:26.445286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.149510+0000","last_change":"2024-03-04T09:25:19.468514+0000","last_active":"2024-03-04T09:25:21.149510+0000","last_peered":"2024-03-04T09:25:21.149510+0000","last_clean":"2024-03-04T09:25:21.149510+0000","last_became_active":"2024-03-04T09:25:19.468029+0000","last_became_peered":"2024-03-04T09:25:19.468029+0000","last_unstale":"2024-03-04T09:25:21.149510+0000","last_undegraded":"2024-03-04T09:25:21.149510+0000","last_fullsized":"2024-03-04T09:25:21.149510+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:45:15.160192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.455915+0000","last_change":"2024-03-04T09:25:19.448212+0000","last_active":"2024-03-04T09:25:20.455915+0000","last_peered":"2024-03-04T09:25:20.455915+0000","last_clean":"2024-03-04T09:25:20.455915+0000","last_became_active":"2024-03-04T09:25:19.447824+0000","last_became_peered":"2024-03-04T09:25:19.447824+0000","last_unstale":"2024-03-04T09:25:20.455915+0000","last_undegraded":"2024-03-04T09:25:20.455915+0000","last_fullsized":"2024-03-04T09:25:20.455915+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:44:37.465937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.497265+0000","last_change":"2024-03-04T09:25:19.471682+0000","last_active":"2024-03-04T09:25:20.497265+0000","last_peered":"2024-03-04T09:25:20.497265+0000","last_clean":"2024-03-04T09:25:20.497265+0000","last_became_active":"2024-03-04T09:25:19.471416+0000","last_became_peered":"2024-03-04T09:25:19.471416+0000","last_unstale":"2024-03-04T09:25:20.497265+0000","last_undegraded":"2024-03-04T09:25:20.497265+0000","last_fullsized":"2024-03-04T09:25:20.497265+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:01:33.498760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.499060+0000","last_change":"2024-03-04T09:25:19.484010+0000","last_active":"2024-03-04T09:25:20.499060+0000","last_peered":"2024-03-04T09:25:20.499060+0000","last_clean":"2024-03-04T09:25:20.499060+0000","last_became_active":"2024-03-04T09:25:19.477995+0000","last_became_peered":"2024-03-04T09:25:19.477995+0000","last_unstale":"2024-03-04T09:25:20.499060+0000","last_undegraded":"2024-03-04T09:25:20.499060+0000","last_fullsized":"2024-03-04T09:25:20.499060+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:36:31.167806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.458074+0000","last_change":"2024-03-04T09:25:19.456238+0000","last_active":"2024-03-04T09:25:20.458074+0000","last_peered":"2024-03-04T09:25:20.458074+0000","last_clean":"2024-03-04T09:25:20.458074+0000","last_became_active":"2024-03-04T09:25:19.455908+0000","last_became_peered":"2024-03-04T09:25:19.455908+0000","last_unstale":"2024-03-04T09:25:20.458074+0000","last_undegraded":"2024-03-04T09:25:20.458074+0000","last_fullsized":"2024-03-04T09:25:20.458074+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:32:59.918382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.151921+0000","last_change":"2024-03-04T09:25:19.479250+0000","last_active":"2024-03-04T09:25:21.151921+0000","last_peered":"2024-03-04T09:25:21.151921+0000","last_clean":"2024-03-04T09:25:21.151921+0000","last_became_active":"2024-03-04T09:25:19.478971+0000","last_became_peered":"2024-03-04T09:25:19.478971+0000","last_unstale":"2024-03-04T09:25:21.151921+0000","last_undegraded":"2024-03-04T09:25:21.151921+0000","last_fullsized":"2024-03-04T09:25:21.151921+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:12:45.452669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.464032+0000","last_change":"2024-03-04T09:25:19.471817+0000","last_active":"2024-03-04T09:25:20.464032+0000","last_peered":"2024-03-04T09:25:20.464032+0000","last_clean":"2024-03-04T09:25:20.464032+0000","last_became_active":"2024-03-04T09:25:19.471551+0000","last_became_peered":"2024-03-04T09:25:19.471551+0000","last_unstale":"2024-03-04T09:25:20.464032+0000","last_undegraded":"2024-03-04T09:25:20.464032+0000","last_fullsized":"2024-03-04T09:25:20.464032+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:45:59.096795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.149784+0000","last_change":"2024-03-04T09:25:19.464870+0000","last_active":"2024-03-04T09:25:21.149784+0000","last_peered":"2024-03-04T09:25:21.149784+0000","last_clean":"2024-03-04T09:25:21.149784+0000","last_became_active":"2024-03-04T09:25:19.464544+0000","last_became_peered":"2024-03-04T09:25:19.464544+0000","last_unstale":"2024-03-04T09:25:21.149784+0000","last_undegraded":"2024-03-04T09:25:21.149784+0000","last_fullsized":"2024-03-04T09:25:21.149784+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:45:59.096795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.497519+0000","last_change":"2024-03-04T09:25:19.459890+0000","last_active":"2024-03-04T09:25:20.497519+0000","last_peered":"2024-03-04T09:25:20.497519+0000","last_clean":"2024-03-04T09:25:20.497519+0000","last_became_active":"2024-03-04T09:25:19.459528+0000","last_became_peered":"2024-03-04T09:25:19.459528+0000","last_unstale":"2024-03-04T09:25:20.497519+0000","last_undegraded":"2024-03-04T09:25:20.497519+0000","last_fullsized":"2024-03-04T09:25:20.497519+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:36:58.654226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.456364+0000","last_change":"2024-03-04T09:25:19.462338+0000","last_active":"2024-03-04T09:25:20.456364+0000","last_peered":"2024-03-04T09:25:20.456364+0000","last_clean":"2024-03-04T09:25:20.456364+0000","last_became_active":"2024-03-04T09:25:19.461998+0000","last_became_peered":"2024-03-04T09:25:19.461998+0000","last_unstale":"2024-03-04T09:25:20.456364+0000","last_undegraded":"2024-03-04T09:25:20.456364+0000","last_fullsized":"2024-03-04T09:25:20.456364+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:56:18.833810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.497149+0000","last_change":"2024-03-04T09:25:19.473080+0000","last_active":"2024-03-04T09:25:20.497149+0000","last_peered":"2024-03-04T09:25:20.497149+0000","last_clean":"2024-03-04T09:25:20.497149+0000","last_became_active":"2024-03-04T09:25:19.467315+0000","last_became_peered":"2024-03-04T09:25:19.467315+0000","last_unstale":"2024-03-04T09:25:20.497149+0000","last_undegraded":"2024-03-04T09:25:20.497149+0000","last_fullsized":"2024-03-04T09:25:20.497149+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:33:16.368258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.458010+0000","last_change":"2024-03-04T09:25:19.472788+0000","last_active":"2024-03-04T09:25:20.458010+0000","last_peered":"2024-03-04T09:25:20.458010+0000","last_clean":"2024-03-04T09:25:20.458010+0000","last_became_active":"2024-03-04T09:25:19.472496+0000","last_became_peered":"2024-03-04T09:25:19.472496+0000","last_unstale":"2024-03-04T09:25:20.458010+0000","last_undegraded":"2024-03-04T09:25:20.458010+0000","last_fullsized":"2024-03-04T09:25:20.458010+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:01:33.498760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.464735+0000","last_change":"2024-03-04T09:25:19.461265+0000","last_active":"2024-03-04T09:25:20.464735+0000","last_peered":"2024-03-04T09:25:20.464735+0000","last_clean":"2024-03-04T09:25:20.464735+0000","last_became_active":"2024-03-04T09:25:19.458576+0000","last_became_peered":"2024-03-04T09:25:19.458576+0000","last_unstale":"2024-03-04T09:25:20.464735+0000","last_undegraded":"2024-03-04T09:25:20.464735+0000","last_fullsized":"2024-03-04T09:25:20.464735+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:33:58.076780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.456949+0000","last_change":"2024-03-04T09:25:19.467165+0000","last_active":"2024-03-04T09:25:20.456949+0000","last_peered":"2024-03-04T09:25:20.456949+0000","last_clean":"2024-03-04T09:25:20.456949+0000","last_became_active":"2024-03-04T09:25:19.466793+0000","last_became_peered":"2024-03-04T09:25:19.466793+0000","last_unstale":"2024-03-04T09:25:20.456949+0000","last_undegraded":"2024-03-04T09:25:20.456949+0000","last_fullsized":"2024-03-04T09:25:20.456949+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:33:58.076780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.457770+0000","last_change":"2024-03-04T09:25:19.474283+0000","last_active":"2024-03-04T09:25:20.457770+0000","last_peered":"2024-03-04T09:25:20.457770+0000","last_clean":"2024-03-04T09:25:20.457770+0000","last_became_active":"2024-03-04T09:25:19.474124+0000","last_became_peered":"2024-03-04T09:25:19.474124+0000","last_unstale":"2024-03-04T09:25:20.457770+0000","last_undegraded":"2024-03-04T09:25:20.457770+0000","last_fullsized":"2024-03-04T09:25:20.457770+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:33:16.368258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.150029+0000","last_change":"2024-03-04T09:25:19.461996+0000","last_active":"2024-03-04T09:25:21.150029+0000","last_peered":"2024-03-04T09:25:21.150029+0000","last_clean":"2024-03-04T09:25:21.150029+0000","last_became_active":"2024-03-04T09:25:19.461683+0000","last_became_peered":"2024-03-04T09:25:19.461683+0000","last_unstale":"2024-03-04T09:25:21.150029+0000","last_undegraded":"2024-03-04T09:25:21.150029+0000","last_fullsized":"2024-03-04T09:25:21.150029+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:47:26.445286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.497702+0000","last_change":"2024-03-04T09:25:19.466189+0000","last_active":"2024-03-04T09:25:20.497702+0000","last_peered":"2024-03-04T09:25:20.497702+0000","last_clean":"2024-03-04T09:25:20.497702+0000","last_became_active":"2024-03-04T09:25:19.465903+0000","last_became_peered":"2024-03-04T09:25:19.465903+0000","last_unstale":"2024-03-04T09:25:20.497702+0000","last_undegraded":"2024-03-04T09:25:20.497702+0000","last_fullsized":"2024-03-04T09:25:20.497702+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:04:48.675219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.457411+0000","last_change":"2024-03-04T09:25:19.458477+0000","last_active":"2024-03-04T09:25:20.457411+0000","last_peered":"2024-03-04T09:25:20.457411+0000","last_clean":"2024-03-04T09:25:20.457411+0000","last_became_active":"2024-03-04T09:25:19.455994+0000","last_became_peered":"2024-03-04T09:25:19.455994+0000","last_unstale":"2024-03-04T09:25:20.457411+0000","last_undegraded":"2024-03-04T09:25:20.457411+0000","last_fullsized":"2024-03-04T09:25:20.457411+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:11:52.240719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.463964+0000","last_change":"2024-03-04T09:25:19.464315+0000","last_active":"2024-03-04T09:25:20.463964+0000","last_peered":"2024-03-04T09:25:20.463964+0000","last_clean":"2024-03-04T09:25:20.463964+0000","last_became_active":"2024-03-04T09:25:19.463942+0000","last_became_peered":"2024-03-04T09:25:19.463942+0000","last_unstale":"2024-03-04T09:25:20.463964+0000","last_undegraded":"2024-03-04T09:25:20.463964+0000","last_fullsized":"2024-03-04T09:25:20.463964+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:05:57.624897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.149802+0000","last_change":"2024-03-04T09:25:19.448874+0000","last_active":"2024-03-04T09:25:21.149802+0000","last_peered":"2024-03-04T09:25:21.149802+0000","last_clean":"2024-03-04T09:25:21.149802+0000","last_became_active":"2024-03-04T09:25:19.445047+0000","last_became_peered":"2024-03-04T09:25:19.445047+0000","last_unstale":"2024-03-04T09:25:21.149802+0000","last_undegraded":"2024-03-04T09:25:21.149802+0000","last_fullsized":"2024-03-04T09:25:21.149802+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:23:35.947152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.456151+0000","last_change":"2024-03-04T09:25:18.469771+0000","last_active":"2024-03-04T09:25:20.456151+0000","last_peered":"2024-03-04T09:25:20.456151+0000","last_clean":"2024-03-04T09:25:20.456151+0000","last_became_active":"2024-03-04T09:25:18.469445+0000","last_became_peered":"2024-03-04T09:25:18.469445+0000","last_unstale":"2024-03-04T09:25:20.456151+0000","last_undegraded":"2024-03-04T09:25:20.456151+0000","last_fullsized":"2024-03-04T09:25:20.456151+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:39:20.774577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.150269+0000","last_change":"2024-03-04T09:25:19.463486+0000","last_active":"2024-03-04T09:25:21.150269+0000","last_peered":"2024-03-04T09:25:21.150269+0000","last_clean":"2024-03-04T09:25:21.150269+0000","last_became_active":"2024-03-04T09:25:19.463204+0000","last_became_peered":"2024-03-04T09:25:19.463204+0000","last_unstale":"2024-03-04T09:25:21.150269+0000","last_undegraded":"2024-03-04T09:25:21.150269+0000","last_fullsized":"2024-03-04T09:25:21.150269+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:33:58.076780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.463976+0000","last_change":"2024-03-04T09:25:18.472794+0000","last_active":"2024-03-04T09:25:20.463976+0000","last_peered":"2024-03-04T09:25:20.463976+0000","last_clean":"2024-03-04T09:25:20.463976+0000","last_became_active":"2024-03-04T09:25:18.472449+0000","last_became_peered":"2024-03-04T09:25:18.472449+0000","last_unstale":"2024-03-04T09:25:20.463976+0000","last_undegraded":"2024-03-04T09:25:20.463976+0000","last_fullsized":"2024-03-04T09:25:20.463976+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:23:34.932850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.456455+0000","last_change":"2024-03-04T09:25:18.474050+0000","last_active":"2024-03-04T09:25:20.456455+0000","last_peered":"2024-03-04T09:25:20.456455+0000","last_clean":"2024-03-04T09:25:20.456455+0000","last_became_active":"2024-03-04T09:25:18.473760+0000","last_became_peered":"2024-03-04T09:25:18.473760+0000","last_unstale":"2024-03-04T09:25:20.456455+0000","last_undegraded":"2024-03-04T09:25:20.456455+0000","last_fullsized":"2024-03-04T09:25:20.456455+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:31:15.318756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.455119+0000","last_change":"2024-03-04T09:25:19.464447+0000","last_active":"2024-03-04T09:25:20.455119+0000","last_peered":"2024-03-04T09:25:20.455119+0000","last_clean":"2024-03-04T09:25:20.455119+0000","last_became_active":"2024-03-04T09:25:19.463201+0000","last_became_peered":"2024-03-04T09:25:19.463201+0000","last_unstale":"2024-03-04T09:25:20.455119+0000","last_undegraded":"2024-03-04T09:25:20.455119+0000","last_fullsized":"2024-03-04T09:25:20.455119+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:05:57.624897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.497223+0000","last_change":"2024-03-04T09:25:18.476149+0000","last_active":"2024-03-04T09:25:20.497223+0000","last_peered":"2024-03-04T09:25:20.497223+0000","last_clean":"2024-03-04T09:25:20.497223+0000","last_became_active":"2024-03-04T09:25:18.475664+0000","last_became_peered":"2024-03-04T09:25:18.475664+0000","last_unstale":"2024-03-04T09:25:20.497223+0000","last_undegraded":"2024-03-04T09:25:20.497223+0000","last_fullsized":"2024-03-04T09:25:20.497223+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:33:35.151096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.455640+0000","last_change":"2024-03-04T09:25:19.467051+0000","last_active":"2024-03-04T09:25:20.455640+0000","last_peered":"2024-03-04T09:25:20.455640+0000","last_clean":"2024-03-04T09:25:20.455640+0000","last_became_active":"2024-03-04T09:25:19.466755+0000","last_became_peered":"2024-03-04T09:25:19.466755+0000","last_unstale":"2024-03-04T09:25:20.455640+0000","last_undegraded":"2024-03-04T09:25:20.455640+0000","last_fullsized":"2024-03-04T09:25:20.455640+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:47:26.445286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.464438+0000","last_change":"2024-03-04T09:25:18.479294+0000","last_active":"2024-03-04T09:25:20.464438+0000","last_peered":"2024-03-04T09:25:20.464438+0000","last_clean":"2024-03-04T09:25:20.464438+0000","last_became_active":"2024-03-04T09:25:18.478561+0000","last_became_peered":"2024-03-04T09:25:18.478561+0000","last_unstale":"2024-03-04T09:25:20.464438+0000","last_undegraded":"2024-03-04T09:25:20.464438+0000","last_fullsized":"2024-03-04T09:25:20.464438+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:33:35.151096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.498563+0000","last_change":"2024-03-04T09:25:19.459933+0000","last_active":"2024-03-04T09:25:20.498563+0000","last_peered":"2024-03-04T09:25:20.498563+0000","last_clean":"2024-03-04T09:25:20.498563+0000","last_became_active":"2024-03-04T09:25:19.459622+0000","last_became_peered":"2024-03-04T09:25:19.459622+0000","last_unstale":"2024-03-04T09:25:20.498563+0000","last_undegraded":"2024-03-04T09:25:20.498563+0000","last_fullsized":"2024-03-04T09:25:20.498563+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:45:23.780438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.498673+0000","last_change":"2024-03-04T09:25:18.476726+0000","last_active":"2024-03-04T09:25:20.498673+0000","last_peered":"2024-03-04T09:25:20.498673+0000","last_clean":"2024-03-04T09:25:20.498673+0000","last_became_active":"2024-03-04T09:25:18.475636+0000","last_became_peered":"2024-03-04T09:25:18.475636+0000","last_unstale":"2024-03-04T09:25:20.498673+0000","last_undegraded":"2024-03-04T09:25:20.498673+0000","last_fullsized":"2024-03-04T09:25:20.498673+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:54:01.985207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.150507+0000","last_change":"2024-03-04T09:25:19.473920+0000","last_active":"2024-03-04T09:25:21.150507+0000","last_peered":"2024-03-04T09:25:21.150507+0000","last_clean":"2024-03-04T09:25:21.150507+0000","last_became_active":"2024-03-04T09:25:19.473457+0000","last_became_peered":"2024-03-04T09:25:19.473457+0000","last_unstale":"2024-03-04T09:25:21.150507+0000","last_undegraded":"2024-03-04T09:25:21.150507+0000","last_fullsized":"2024-03-04T09:25:21.150507+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:33:16.368258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.457014+0000","last_change":"2024-03-04T09:25:18.471507+0000","last_active":"2024-03-04T09:25:20.457014+0000","last_peered":"2024-03-04T09:25:20.457014+0000","last_clean":"2024-03-04T09:25:20.457014+0000","last_became_active":"2024-03-04T09:25:18.470138+0000","last_became_peered":"2024-03-04T09:25:18.470138+0000","last_unstale":"2024-03-04T09:25:20.457014+0000","last_undegraded":"2024-03-04T09:25:20.457014+0000","last_fullsized":"2024-03-04T09:25:20.457014+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:54:01.985207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.151586+0000","last_change":"2024-03-04T09:25:19.471146+0000","last_active":"2024-03-04T09:25:21.151586+0000","last_peered":"2024-03-04T09:25:21.151586+0000","last_clean":"2024-03-04T09:25:21.151586+0000","last_became_active":"2024-03-04T09:25:19.470904+0000","last_became_peered":"2024-03-04T09:25:19.470904+0000","last_unstale":"2024-03-04T09:25:21.151586+0000","last_undegraded":"2024-03-04T09:25:21.151586+0000","last_fullsized":"2024-03-04T09:25:21.151586+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:32:59.918382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"19'6","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.463922+0000","last_change":"2024-03-04T09:25:18.474956+0000","last_active":"2024-03-04T09:25:20.463922+0000","last_peered":"2024-03-04T09:25:20.463922+0000","last_clean":"2024-03-04T09:25:20.463922+0000","last_became_active":"2024-03-04T09:25:18.474011+0000","last_became_peered":"2024-03-04T09:25:18.474011+0000","last_unstale":"2024-03-04T09:25:20.463922+0000","last_undegraded":"2024-03-04T09:25:20.463922+0000","last_fullsized":"2024-03-04T09:25:20.463922+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:54:01.985207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.151254+0000","last_change":"2024-03-04T09:25:19.477899+0000","last_active":"2024-03-04T09:25:21.151254+0000","last_peered":"2024-03-04T09:25:21.151254+0000","last_clean":"2024-03-04T09:25:21.151254+0000","last_became_active":"2024-03-04T09:25:19.477117+0000","last_became_peered":"2024-03-04T09:25:19.477117+0000","last_unstale":"2024-03-04T09:25:21.151254+0000","last_undegraded":"2024-03-04T09:25:21.151254+0000","last_fullsized":"2024-03-04T09:25:21.151254+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:04:48.675219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.150191+0000","last_change":"2024-03-04T09:25:18.469651+0000","last_active":"2024-03-04T09:25:21.150191+0000","last_peered":"2024-03-04T09:25:21.150191+0000","last_clean":"2024-03-04T09:25:21.150191+0000","last_became_active":"2024-03-04T09:25:18.469259+0000","last_became_peered":"2024-03-04T09:25:18.469259+0000","last_unstale":"2024-03-04T09:25:21.150191+0000","last_undegraded":"2024-03-04T09:25:21.150191+0000","last_fullsized":"2024-03-04T09:25:21.150191+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:57:52.768999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.149691+0000","last_change":"2024-03-04T09:25:19.465339+0000","last_active":"2024-03-04T09:25:21.149691+0000","last_peered":"2024-03-04T09:25:21.149691+0000","last_clean":"2024-03-04T09:25:21.149691+0000","last_became_active":"2024-03-04T09:25:19.464957+0000","last_became_peered":"2024-03-04T09:25:19.464957+0000","last_unstale":"2024-03-04T09:25:21.149691+0000","last_undegraded":"2024-03-04T09:25:21.149691+0000","last_fullsized":"2024-03-04T09:25:21.149691+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:50:18.265156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.497632+0000","last_change":"2024-03-04T09:25:18.470800+0000","last_active":"2024-03-04T09:25:20.497632+0000","last_peered":"2024-03-04T09:25:20.497632+0000","last_clean":"2024-03-04T09:25:20.497632+0000","last_became_active":"2024-03-04T09:25:18.470431+0000","last_became_peered":"2024-03-04T09:25:18.470431+0000","last_unstale":"2024-03-04T09:25:20.497632+0000","last_undegraded":"2024-03-04T09:25:20.497632+0000","last_fullsized":"2024-03-04T09:25:20.497632+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:39:20.774577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.151318+0000","last_change":"2024-03-04T09:25:19.478795+0000","last_active":"2024-03-04T09:25:21.151318+0000","last_peered":"2024-03-04T09:25:21.151318+0000","last_clean":"2024-03-04T09:25:21.151318+0000","last_became_active":"2024-03-04T09:25:19.478635+0000","last_became_peered":"2024-03-04T09:25:19.478635+0000","last_unstale":"2024-03-04T09:25:21.151318+0000","last_undegraded":"2024-03-04T09:25:21.151318+0000","last_fullsized":"2024-03-04T09:25:21.151318+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:04:20.349240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.463677+0000","last_change":"2024-03-04T09:25:18.474750+0000","last_active":"2024-03-04T09:25:20.463677+0000","last_peered":"2024-03-04T09:25:20.463677+0000","last_clean":"2024-03-04T09:25:20.463677+0000","last_became_active":"2024-03-04T09:25:18.473463+0000","last_became_peered":"2024-03-04T09:25:18.473463+0000","last_unstale":"2024-03-04T09:25:20.463677+0000","last_undegraded":"2024-03-04T09:25:20.463677+0000","last_fullsized":"2024-03-04T09:25:20.463677+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:38:16.906069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.463789+0000","last_change":"2024-03-04T09:25:19.472274+0000","last_active":"2024-03-04T09:25:20.463789+0000","last_peered":"2024-03-04T09:25:20.463789+0000","last_clean":"2024-03-04T09:25:20.463789+0000","last_became_active":"2024-03-04T09:25:19.470669+0000","last_became_peered":"2024-03-04T09:25:19.470669+0000","last_unstale":"2024-03-04T09:25:20.463789+0000","last_undegraded":"2024-03-04T09:25:20.463789+0000","last_fullsized":"2024-03-04T09:25:20.463789+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:50:18.265156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":22,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.456829+0000","last_change":"2024-03-04T09:25:18.462460+0000","last_active":"2024-03-04T09:25:20.456829+0000","last_peered":"2024-03-04T09:25:20.456829+0000","last_clean":"2024-03-04T09:25:20.456829+0000","last_became_active":"2024-03-04T09:25:18.460692+0000","last_became_peered":"2024-03-04T09:25:18.460692+0000","last_unstale":"2024-03-04T09:25:20.456829+0000","last_undegraded":"2024-03-04T09:25:20.456829+0000","last_fullsized":"2024-03-04T09:25:20.456829+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:38:16.906069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.150513+0000","last_change":"2024-03-04T09:25:19.478013+0000","last_active":"2024-03-04T09:25:21.150513+0000","last_peered":"2024-03-04T09:25:21.150513+0000","last_clean":"2024-03-04T09:25:21.150513+0000","last_became_active":"2024-03-04T09:25:19.477150+0000","last_became_peered":"2024-03-04T09:25:19.477150+0000","last_unstale":"2024-03-04T09:25:21.150513+0000","last_undegraded":"2024-03-04T09:25:21.150513+0000","last_fullsized":"2024-03-04T09:25:21.150513+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:11:11.501102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.152408+0000","last_change":"2024-03-04T09:25:18.468798+0000","last_active":"2024-03-04T09:25:21.152408+0000","last_peered":"2024-03-04T09:25:21.152408+0000","last_clean":"2024-03-04T09:25:21.152408+0000","last_became_active":"2024-03-04T09:25:18.468412+0000","last_became_peered":"2024-03-04T09:25:18.468412+0000","last_unstale":"2024-03-04T09:25:21.152408+0000","last_undegraded":"2024-03-04T09:25:21.152408+0000","last_fullsized":"2024-03-04T09:25:21.152408+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:17:37.720962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.151303+0000","last_change":"2024-03-04T09:25:19.478586+0000","last_active":"2024-03-04T09:25:21.151303+0000","last_peered":"2024-03-04T09:25:21.151303+0000","last_clean":"2024-03-04T09:25:21.151303+0000","last_became_active":"2024-03-04T09:25:19.478339+0000","last_became_peered":"2024-03-04T09:25:19.478339+0000","last_unstale":"2024-03-04T09:25:21.151303+0000","last_undegraded":"2024-03-04T09:25:21.151303+0000","last_fullsized":"2024-03-04T09:25:21.151303+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:20:25.135538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.458139+0000","last_change":"2024-03-04T09:25:18.459079+0000","last_active":"2024-03-04T09:25:20.458139+0000","last_peered":"2024-03-04T09:25:20.458139+0000","last_clean":"2024-03-04T09:25:20.458139+0000","last_became_active":"2024-03-04T09:25:18.458767+0000","last_became_peered":"2024-03-04T09:25:18.458767+0000","last_unstale":"2024-03-04T09:25:20.458139+0000","last_undegraded":"2024-03-04T09:25:20.458139+0000","last_fullsized":"2024-03-04T09:25:20.458139+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:33:35.151096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.152708+0000","last_change":"2024-03-04T09:25:19.481294+0000","last_active":"2024-03-04T09:25:21.152708+0000","last_peered":"2024-03-04T09:25:21.152708+0000","last_clean":"2024-03-04T09:25:21.152708+0000","last_became_active":"2024-03-04T09:25:19.479163+0000","last_became_peered":"2024-03-04T09:25:19.479163+0000","last_unstale":"2024-03-04T09:25:21.152708+0000","last_undegraded":"2024-03-04T09:25:21.152708+0000","last_fullsized":"2024-03-04T09:25:21.152708+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:36:31.167806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.457070+0000","last_change":"2024-03-04T09:25:18.464660+0000","last_active":"2024-03-04T09:25:20.457070+0000","last_peered":"2024-03-04T09:25:20.457070+0000","last_clean":"2024-03-04T09:25:20.457070+0000","last_became_active":"2024-03-04T09:25:18.464412+0000","last_became_peered":"2024-03-04T09:25:18.464412+0000","last_unstale":"2024-03-04T09:25:20.457070+0000","last_undegraded":"2024-03-04T09:25:20.457070+0000","last_fullsized":"2024-03-04T09:25:20.457070+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:32:58.904081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.498417+0000","last_change":"2024-03-04T09:25:19.480725+0000","last_active":"2024-03-04T09:25:20.498417+0000","last_peered":"2024-03-04T09:25:20.498417+0000","last_clean":"2024-03-04T09:25:20.498417+0000","last_became_active":"2024-03-04T09:25:19.468626+0000","last_became_peered":"2024-03-04T09:25:19.468626+0000","last_unstale":"2024-03-04T09:25:20.498417+0000","last_undegraded":"2024-03-04T09:25:20.498417+0000","last_fullsized":"2024-03-04T09:25:20.498417+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:36:55.714408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.457915+0000","last_change":"2024-03-04T09:25:18.479022+0000","last_active":"2024-03-04T09:25:20.457915+0000","last_peered":"2024-03-04T09:25:20.457915+0000","last_clean":"2024-03-04T09:25:20.457915+0000","last_became_active":"2024-03-04T09:25:18.478697+0000","last_became_peered":"2024-03-04T09:25:18.478697+0000","last_unstale":"2024-03-04T09:25:20.457915+0000","last_undegraded":"2024-03-04T09:25:20.457915+0000","last_fullsized":"2024-03-04T09:25:20.457915+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:39:20.774577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.151536+0000","last_change":"2024-03-04T09:25:19.485083+0000","last_active":"2024-03-04T09:25:21.151536+0000","last_peered":"2024-03-04T09:25:21.151536+0000","last_clean":"2024-03-04T09:25:21.151536+0000","last_became_active":"2024-03-04T09:25:19.484619+0000","last_became_peered":"2024-03-04T09:25:19.484619+0000","last_unstale":"2024-03-04T09:25:21.151536+0000","last_undegraded":"2024-03-04T09:25:21.151536+0000","last_fullsized":"2024-03-04T09:25:21.151536+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:39:25.865493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":170,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.150247+0000","last_change":"2024-03-04T09:25:09.316159+0000","last_active":"2024-03-04T09:25:21.150247+0000","last_peered":"2024-03-04T09:25:21.150247+0000","last_clean":"2024-03-04T09:25:21.150247+0000","last_became_active":"2024-03-04T09:25:09.313767+0000","last_became_peered":"2024-03-04T09:25:09.313767+0000","last_unstale":"2024-03-04T09:25:21.150247+0000","last_undegraded":"2024-03-04T09:25:21.150247+0000","last_fullsized":"2024-03-04T09:25:21.150247+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:08.110953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:08.110953+0000","last_clean_scrub_stamp":"2024-03-04T09:25:08.110953+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:34:21.745390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.497328+0000","last_change":"2024-03-04T09:25:18.464859+0000","last_active":"2024-03-04T09:25:20.497328+0000","last_peered":"2024-03-04T09:25:20.497328+0000","last_clean":"2024-03-04T09:25:20.497328+0000","last_became_active":"2024-03-04T09:25:18.463515+0000","last_became_peered":"2024-03-04T09:25:18.463515+0000","last_unstale":"2024-03-04T09:25:20.497328+0000","last_undegraded":"2024-03-04T09:25:20.497328+0000","last_fullsized":"2024-03-04T09:25:20.497328+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:38:16.906069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.456773+0000","last_change":"2024-03-04T09:25:19.460896+0000","last_active":"2024-03-04T09:25:20.456773+0000","last_peered":"2024-03-04T09:25:20.456773+0000","last_clean":"2024-03-04T09:25:20.456773+0000","last_became_active":"2024-03-04T09:25:19.460598+0000","last_became_peered":"2024-03-04T09:25:19.460598+0000","last_unstale":"2024-03-04T09:25:20.456773+0000","last_undegraded":"2024-03-04T09:25:20.456773+0000","last_fullsized":"2024-03-04T09:25:20.456773+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:33:58.076780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.149994+0000","last_change":"2024-03-04T09:25:18.480278+0000","last_active":"2024-03-04T09:25:21.149994+0000","last_peered":"2024-03-04T09:25:21.149994+0000","last_clean":"2024-03-04T09:25:21.149994+0000","last_became_active":"2024-03-04T09:25:18.478848+0000","last_became_peered":"2024-03-04T09:25:18.478848+0000","last_unstale":"2024-03-04T09:25:21.149994+0000","last_undegraded":"2024-03-04T09:25:21.149994+0000","last_fullsized":"2024-03-04T09:25:21.149994+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:54:01.985207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.149481+0000","last_change":"2024-03-04T09:25:19.473908+0000","last_active":"2024-03-04T09:25:21.149481+0000","last_peered":"2024-03-04T09:25:21.149481+0000","last_clean":"2024-03-04T09:25:21.149481+0000","last_became_active":"2024-03-04T09:25:19.472415+0000","last_became_peered":"2024-03-04T09:25:19.472415+0000","last_unstale":"2024-03-04T09:25:21.149481+0000","last_undegraded":"2024-03-04T09:25:21.149481+0000","last_fullsized":"2024-03-04T09:25:21.149481+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:46:44.342046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.457503+0000","last_change":"2024-03-04T09:25:18.462877+0000","last_active":"2024-03-04T09:25:20.457503+0000","last_peered":"2024-03-04T09:25:20.457503+0000","last_clean":"2024-03-04T09:25:20.457503+0000","last_became_active":"2024-03-04T09:25:18.459553+0000","last_became_peered":"2024-03-04T09:25:18.459553+0000","last_unstale":"2024-03-04T09:25:20.457503+0000","last_undegraded":"2024-03-04T09:25:20.457503+0000","last_fullsized":"2024-03-04T09:25:20.457503+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:49:03.118927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.456676+0000","last_change":"2024-03-04T09:25:19.452298+0000","last_active":"2024-03-04T09:25:20.456676+0000","last_peered":"2024-03-04T09:25:20.456676+0000","last_clean":"2024-03-04T09:25:20.456676+0000","last_became_active":"2024-03-04T09:25:19.451945+0000","last_became_peered":"2024-03-04T09:25:19.451945+0000","last_unstale":"2024-03-04T09:25:20.456676+0000","last_undegraded":"2024-03-04T09:25:20.456676+0000","last_fullsized":"2024-03-04T09:25:20.456676+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:56:18.833810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.497812+0000","last_change":"2024-03-04T09:25:18.473872+0000","last_active":"2024-03-04T09:25:20.497812+0000","last_peered":"2024-03-04T09:25:20.497812+0000","last_clean":"2024-03-04T09:25:20.497812+0000","last_became_active":"2024-03-04T09:25:18.473390+0000","last_became_peered":"2024-03-04T09:25:18.473390+0000","last_unstale":"2024-03-04T09:25:20.497812+0000","last_undegraded":"2024-03-04T09:25:20.497812+0000","last_fullsized":"2024-03-04T09:25:20.497812+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:23:34.932850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.150539+0000","last_change":"2024-03-04T09:25:19.467593+0000","last_active":"2024-03-04T09:25:21.150539+0000","last_peered":"2024-03-04T09:25:21.150539+0000","last_clean":"2024-03-04T09:25:21.150539+0000","last_became_active":"2024-03-04T09:25:19.467079+0000","last_became_peered":"2024-03-04T09:25:19.467079+0000","last_unstale":"2024-03-04T09:25:21.150539+0000","last_undegraded":"2024-03-04T09:25:21.150539+0000","last_fullsized":"2024-03-04T09:25:21.150539+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:11:52.240719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.456743+0000","last_change":"2024-03-04T09:25:18.465721+0000","last_active":"2024-03-04T09:25:20.456743+0000","last_peered":"2024-03-04T09:25:20.456743+0000","last_clean":"2024-03-04T09:25:20.456743+0000","last_became_active":"2024-03-04T09:25:18.465438+0000","last_became_peered":"2024-03-04T09:25:18.465438+0000","last_unstale":"2024-03-04T09:25:20.456743+0000","last_undegraded":"2024-03-04T09:25:20.456743+0000","last_fullsized":"2024-03-04T09:25:20.456743+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:39:20.774577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.498362+0000","last_change":"2024-03-04T09:25:19.472271+0000","last_active":"2024-03-04T09:25:20.498362+0000","last_peered":"2024-03-04T09:25:20.498362+0000","last_clean":"2024-03-04T09:25:20.498362+0000","last_became_active":"2024-03-04T09:25:19.467596+0000","last_became_peered":"2024-03-04T09:25:19.467596+0000","last_unstale":"2024-03-04T09:25:20.498362+0000","last_undegraded":"2024-03-04T09:25:20.498362+0000","last_fullsized":"2024-03-04T09:25:20.498362+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:12:45.452669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.151714+0000","last_change":"2024-03-04T09:25:18.481607+0000","last_active":"2024-03-04T09:25:21.151714+0000","last_peered":"2024-03-04T09:25:21.151714+0000","last_clean":"2024-03-04T09:25:21.151714+0000","last_became_active":"2024-03-04T09:25:18.481270+0000","last_became_peered":"2024-03-04T09:25:18.481270+0000","last_unstale":"2024-03-04T09:25:21.151714+0000","last_undegraded":"2024-03-04T09:25:21.151714+0000","last_fullsized":"2024-03-04T09:25:21.151714+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:52:36.632272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.455410+0000","last_change":"2024-03-04T09:25:19.450231+0000","last_active":"2024-03-04T09:25:20.455410+0000","last_peered":"2024-03-04T09:25:20.455410+0000","last_clean":"2024-03-04T09:25:20.455410+0000","last_became_active":"2024-03-04T09:25:19.449920+0000","last_became_peered":"2024-03-04T09:25:19.449920+0000","last_unstale":"2024-03-04T09:25:20.455410+0000","last_undegraded":"2024-03-04T09:25:20.455410+0000","last_fullsized":"2024-03-04T09:25:20.455410+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:45:59.096795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.457374+0000","last_change":"2024-03-04T09:25:18.472718+0000","last_active":"2024-03-04T09:25:20.457374+0000","last_peered":"2024-03-04T09:25:20.457374+0000","last_clean":"2024-03-04T09:25:20.457374+0000","last_became_active":"2024-03-04T09:25:18.472426+0000","last_became_peered":"2024-03-04T09:25:18.472426+0000","last_unstale":"2024-03-04T09:25:20.457374+0000","last_undegraded":"2024-03-04T09:25:20.457374+0000","last_fullsized":"2024-03-04T09:25:20.457374+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:23:34.932850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.458202+0000","last_change":"2024-03-04T09:25:19.470531+0000","last_active":"2024-03-04T09:25:20.458202+0000","last_peered":"2024-03-04T09:25:20.458202+0000","last_clean":"2024-03-04T09:25:20.458202+0000","last_became_active":"2024-03-04T09:25:19.469607+0000","last_became_peered":"2024-03-04T09:25:19.469607+0000","last_unstale":"2024-03-04T09:25:20.458202+0000","last_undegraded":"2024-03-04T09:25:20.458202+0000","last_fullsized":"2024-03-04T09:25:20.458202+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:11:11.501102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.464561+0000","last_change":"2024-03-04T09:25:18.473250+0000","last_active":"2024-03-04T09:25:20.464561+0000","last_peered":"2024-03-04T09:25:20.464561+0000","last_clean":"2024-03-04T09:25:20.464561+0000","last_became_active":"2024-03-04T09:25:18.472968+0000","last_became_peered":"2024-03-04T09:25:18.472968+0000","last_unstale":"2024-03-04T09:25:20.464561+0000","last_undegraded":"2024-03-04T09:25:20.464561+0000","last_fullsized":"2024-03-04T09:25:20.464561+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:39:20.774577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.150075+0000","last_change":"2024-03-04T09:25:19.467410+0000","last_active":"2024-03-04T09:25:21.150075+0000","last_peered":"2024-03-04T09:25:21.150075+0000","last_clean":"2024-03-04T09:25:21.150075+0000","last_became_active":"2024-03-04T09:25:19.467101+0000","last_became_peered":"2024-03-04T09:25:19.467101+0000","last_unstale":"2024-03-04T09:25:21.150075+0000","last_undegraded":"2024-03-04T09:25:21.150075+0000","last_fullsized":"2024-03-04T09:25:21.150075+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:44:37.465937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.149988+0000","last_change":"2024-03-04T09:25:18.475507+0000","last_active":"2024-03-04T09:25:21.149988+0000","last_peered":"2024-03-04T09:25:21.149988+0000","last_clean":"2024-03-04T09:25:21.149988+0000","last_became_active":"2024-03-04T09:25:18.475159+0000","last_became_peered":"2024-03-04T09:25:18.475159+0000","last_unstale":"2024-03-04T09:25:21.149988+0000","last_undegraded":"2024-03-04T09:25:21.149988+0000","last_fullsized":"2024-03-04T09:25:21.149988+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:23:34.932850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.149845+0000","last_change":"2024-03-04T09:25:19.453119+0000","last_active":"2024-03-04T09:25:21.149845+0000","last_peered":"2024-03-04T09:25:21.149845+0000","last_clean":"2024-03-04T09:25:21.149845+0000","last_became_active":"2024-03-04T09:25:19.452764+0000","last_became_peered":"2024-03-04T09:25:19.452764+0000","last_unstale":"2024-03-04T09:25:21.149845+0000","last_undegraded":"2024-03-04T09:25:21.149845+0000","last_fullsized":"2024-03-04T09:25:21.149845+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:54:02.999509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.456033+0000","last_change":"2024-03-04T09:25:18.473872+0000","last_active":"2024-03-04T09:25:20.456033+0000","last_peered":"2024-03-04T09:25:20.456033+0000","last_clean":"2024-03-04T09:25:20.456033+0000","last_became_active":"2024-03-04T09:25:18.473554+0000","last_became_peered":"2024-03-04T09:25:18.473554+0000","last_unstale":"2024-03-04T09:25:20.456033+0000","last_undegraded":"2024-03-04T09:25:20.456033+0000","last_fullsized":"2024-03-04T09:25:20.456033+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:11:51.226417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.455488+0000","last_change":"2024-03-04T09:25:19.458923+0000","last_active":"2024-03-04T09:25:20.455488+0000","last_peered":"2024-03-04T09:25:20.455488+0000","last_clean":"2024-03-04T09:25:20.455488+0000","last_became_active":"2024-03-04T09:25:19.458562+0000","last_became_peered":"2024-03-04T09:25:19.458562+0000","last_unstale":"2024-03-04T09:25:20.455488+0000","last_undegraded":"2024-03-04T09:25:20.455488+0000","last_fullsized":"2024-03-04T09:25:20.455488+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:45:59.096795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.456036+0000","last_change":"2024-03-04T09:25:18.458201+0000","last_active":"2024-03-04T09:25:20.456036+0000","last_peered":"2024-03-04T09:25:20.456036+0000","last_clean":"2024-03-04T09:25:20.456036+0000","last_became_active":"2024-03-04T09:25:18.455667+0000","last_became_peered":"2024-03-04T09:25:18.455667+0000","last_unstale":"2024-03-04T09:25:20.456036+0000","last_undegraded":"2024-03-04T09:25:20.456036+0000","last_fullsized":"2024-03-04T09:25:20.456036+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:38:16.906069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.456043+0000","last_change":"2024-03-04T09:25:19.464390+0000","last_active":"2024-03-04T09:25:20.456043+0000","last_peered":"2024-03-04T09:25:20.456043+0000","last_clean":"2024-03-04T09:25:20.456043+0000","last_became_active":"2024-03-04T09:25:19.463511+0000","last_became_peered":"2024-03-04T09:25:19.463511+0000","last_unstale":"2024-03-04T09:25:20.456043+0000","last_undegraded":"2024-03-04T09:25:20.456043+0000","last_fullsized":"2024-03-04T09:25:20.456043+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:44:37.465937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.151481+0000","last_change":"2024-03-04T09:25:18.479732+0000","last_active":"2024-03-04T09:25:21.151481+0000","last_peered":"2024-03-04T09:25:21.151481+0000","last_clean":"2024-03-04T09:25:21.151481+0000","last_became_active":"2024-03-04T09:25:18.478014+0000","last_became_peered":"2024-03-04T09:25:18.478014+0000","last_unstale":"2024-03-04T09:25:21.151481+0000","last_undegraded":"2024-03-04T09:25:21.151481+0000","last_fullsized":"2024-03-04T09:25:21.151481+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:32:58.904081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.464302+0000","last_change":"2024-03-04T09:25:19.455001+0000","last_active":"2024-03-04T09:25:20.464302+0000","last_peered":"2024-03-04T09:25:20.464302+0000","last_clean":"2024-03-04T09:25:20.464302+0000","last_became_active":"2024-03-04T09:25:19.454695+0000","last_became_peered":"2024-03-04T09:25:19.454695+0000","last_unstale":"2024-03-04T09:25:20.464302+0000","last_undegraded":"2024-03-04T09:25:20.464302+0000","last_fullsized":"2024-03-04T09:25:20.464302+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:47:26.445286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.498167+0000","last_change":"2024-03-04T09:25:19.452288+0000","last_active":"2024-03-04T09:25:20.498167+0000","last_peered":"2024-03-04T09:25:20.498167+0000","last_clean":"2024-03-04T09:25:20.498167+0000","last_became_active":"2024-03-04T09:25:19.451982+0000","last_became_peered":"2024-03-04T09:25:19.451982+0000","last_unstale":"2024-03-04T09:25:20.498167+0000","last_undegraded":"2024-03-04T09:25:20.498167+0000","last_fullsized":"2024-03-04T09:25:20.498167+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:39:25.865493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.151027+0000","last_change":"2024-03-04T09:25:18.481511+0000","last_active":"2024-03-04T09:25:21.151027+0000","last_peered":"2024-03-04T09:25:21.151027+0000","last_clean":"2024-03-04T09:25:21.151027+0000","last_became_active":"2024-03-04T09:25:18.481207+0000","last_became_peered":"2024-03-04T09:25:18.481207+0000","last_unstale":"2024-03-04T09:25:21.151027+0000","last_undegraded":"2024-03-04T09:25:21.151027+0000","last_fullsized":"2024-03-04T09:25:21.151027+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:11:51.226417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.498103+0000","last_change":"2024-03-04T09:25:19.453592+0000","last_active":"2024-03-04T09:25:20.498103+0000","last_peered":"2024-03-04T09:25:20.498103+0000","last_clean":"2024-03-04T09:25:20.498103+0000","last_became_active":"2024-03-04T09:25:19.453318+0000","last_became_peered":"2024-03-04T09:25:19.453318+0000","last_unstale":"2024-03-04T09:25:20.498103+0000","last_undegraded":"2024-03-04T09:25:20.498103+0000","last_fullsized":"2024-03-04T09:25:20.498103+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:04:20.349240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.151412+0000","last_change":"2024-03-04T09:25:18.483252+0000","last_active":"2024-03-04T09:25:21.151412+0000","last_peered":"2024-03-04T09:25:21.151412+0000","last_clean":"2024-03-04T09:25:21.151412+0000","last_became_active":"2024-03-04T09:25:18.482987+0000","last_became_peered":"2024-03-04T09:25:18.482987+0000","last_unstale":"2024-03-04T09:25:21.151412+0000","last_undegraded":"2024-03-04T09:25:21.151412+0000","last_fullsized":"2024-03-04T09:25:21.151412+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:02:44.764307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.150110+0000","last_change":"2024-03-04T09:25:19.473423+0000","last_active":"2024-03-04T09:25:21.150110+0000","last_peered":"2024-03-04T09:25:21.150110+0000","last_clean":"2024-03-04T09:25:21.150110+0000","last_became_active":"2024-03-04T09:25:19.473030+0000","last_became_peered":"2024-03-04T09:25:19.473030+0000","last_unstale":"2024-03-04T09:25:21.150110+0000","last_undegraded":"2024-03-04T09:25:21.150110+0000","last_fullsized":"2024-03-04T09:25:21.150110+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:56:18.833810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.458390+0000","last_change":"2024-03-04T09:25:18.466373+0000","last_active":"2024-03-04T09:25:20.458390+0000","last_peered":"2024-03-04T09:25:20.458390+0000","last_clean":"2024-03-04T09:25:20.458390+0000","last_became_active":"2024-03-04T09:25:18.466069+0000","last_became_peered":"2024-03-04T09:25:18.466069+0000","last_unstale":"2024-03-04T09:25:20.458390+0000","last_undegraded":"2024-03-04T09:25:20.458390+0000","last_fullsized":"2024-03-04T09:25:20.458390+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:11:51.226417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.458472+0000","last_change":"2024-03-04T09:25:19.459312+0000","last_active":"2024-03-04T09:25:20.458472+0000","last_peered":"2024-03-04T09:25:20.458472+0000","last_clean":"2024-03-04T09:25:20.458472+0000","last_became_active":"2024-03-04T09:25:19.458993+0000","last_became_peered":"2024-03-04T09:25:19.458993+0000","last_unstale":"2024-03-04T09:25:20.458472+0000","last_undegraded":"2024-03-04T09:25:20.458472+0000","last_fullsized":"2024-03-04T09:25:20.458472+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:36:58.654226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"19'4","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.150271+0000","last_change":"2024-03-04T09:25:18.473489+0000","last_active":"2024-03-04T09:25:21.150271+0000","last_peered":"2024-03-04T09:25:21.150271+0000","last_clean":"2024-03-04T09:25:21.150271+0000","last_became_active":"2024-03-04T09:25:18.473154+0000","last_became_peered":"2024-03-04T09:25:18.473154+0000","last_unstale":"2024-03-04T09:25:21.150271+0000","last_undegraded":"2024-03-04T09:25:21.150271+0000","last_fullsized":"2024-03-04T09:25:21.150271+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:31:15.318756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574853,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27804,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344036,"statfs":{"total":96636764160,"available":96608292864,"internally_reserved":0,"allocated":802816,"data_stored":661884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574853,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":72546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574855,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":208896,"data_stored":71024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574852,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":71060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574854,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":72546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574853,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":70934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574854,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":212992,"data_stored":71094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574853,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27816,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344024,"statfs":{"total":96636764160,"available":96608280576,"internally_reserved":0,"allocated":811008,"data_stored":661940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-04T09:25:24.526 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-04T09:25:24.527 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-04T09:25:24.527 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-03-04T09:25:24.527 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-03-04T09:25:24.527 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-03-04T09:25:24.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.792+0000 7f244d800640 1 Processor -- start 2024-03-04T09:25:24.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.792+0000 7f244d800640 1 -- start start 2024-03-04T09:25:24.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.793+0000 7f244d800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f2448161b40 0x7f24480cfd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:24.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.793+0000 7f244d800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f24480b5060 0x7f24480d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:24.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.793+0000 7f244d800640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24480cea50 con 0x7f24480b54a0 2024-03-04T09:25:24.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.793+0000 7f244d800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24480cf900 con 0x7f2448161b40 2024-03-04T09:25:24.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.793+0000 7f244d800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24480cecf0 con 0x7f24480b5060 2024-03-04T09:25:24.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.793+0000 7f2446a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f2448161b40 0x7f24480cfd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:24.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.793+0000 7f2446a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f2448161b40 0x7f24480cfd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45072/0 (socket says 172.21.15.6:45072) 2024-03-04T09:25:24.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.793+0000 7f2446a00640 1 -- 172.21.15.6:0/406823569 learned_addr learned my addr 172.21.15.6:0/406823569 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:25:24.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.794+0000 7f2446000640 1 --2- 172.21.15.6:0/406823569 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f24480b5060 0x7f24480d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:24.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.794+0000 7f2446a00640 1 -- 172.21.15.6:0/406823569 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f24480b5060 msgr2=0x7f24480d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:24.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.794+0000 7f2446a00640 1 --2- 172.21.15.6:0/406823569 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f24480b5060 0x7f24480d2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:24.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.794+0000 7f2446a00640 1 -- 172.21.15.6:0/406823569 >> v1:172.21.15.146:6789/0 conn(0x7f24480b54a0 legacy=0x7f24480b3740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:24.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.794+0000 7f2446a00640 1 -- 172.21.15.6:0/406823569 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24480cefd0 con 0x7f2448161b40 2024-03-04T09:25:24.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.794+0000 7f2446000640 1 --2- 172.21.15.6:0/406823569 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f24480b5060 0x7f24480d2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:25:24.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.795+0000 7f2446a00640 1 --2- 172.21.15.6:0/406823569 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2448161b40 0x7f24480cfd90 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f2438009e50 tx=0x7f243802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70570f1105e1c964 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:24.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.795+0000 7f2445600640 1 -- 172.21.15.6:0/406823569 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f243802ca80 con 0x7f2448161b40 2024-03-04T09:25:24.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.796+0000 7f2445600640 1 -- 172.21.15.6:0/406823569 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f243802cc20 con 0x7f2448161b40 2024-03-04T09:25:24.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.796+0000 7f2445600640 1 -- 172.21.15.6:0/406823569 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24380343f0 con 0x7f2448161b40 2024-03-04T09:25:24.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.796+0000 7f244d800640 1 -- 172.21.15.6:0/406823569 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2448161b40 msgr2=0x7f24480cfd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:24.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.796+0000 7f244d800640 1 --2- 172.21.15.6:0/406823569 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2448161b40 0x7f24480cfd90 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f2438009e50 tx=0x7f243802bee0 comp rx=0 tx=0).stop 2024-03-04T09:25:24.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.796+0000 7f244d800640 1 -- 172.21.15.6:0/406823569 shutdown_connections 2024-03-04T09:25:24.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.796+0000 7f244d800640 1 --2- 172.21.15.6:0/406823569 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f24480b5060 0x7f24480d2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:24.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.796+0000 7f244d800640 1 --2- 172.21.15.6:0/406823569 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2448161b40 0x7f24480cfd90 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:24.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.796+0000 7f244d800640 1 -- 172.21.15.6:0/406823569 >> 172.21.15.6:0/406823569 conn(0x7f244809d220 msgr2=0x7f24480d3ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:24.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.796+0000 7f244d800640 1 -- 172.21.15.6:0/406823569 shutdown_connections 2024-03-04T09:25:24.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.796+0000 7f244d800640 1 -- 172.21.15.6:0/406823569 wait complete. 2024-03-04T09:25:24.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.798+0000 7f244d800640 1 Processor -- start 2024-03-04T09:25:24.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.798+0000 7f244d800640 1 -- start start 2024-03-04T09:25:24.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.799+0000 7f244d800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f24480b5060 0x7f24480b9500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:24.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.799+0000 7f244d800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f24480b54a0 0x7f244815d070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:24.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.799+0000 7f244d800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2448161b40 0x7f2448151f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:24.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.799+0000 7f244d800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24480b2ff0 con 0x7f24480b54a0 2024-03-04T09:25:24.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.799+0000 7f244d800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24480cf590 con 0x7f24480b5060 2024-03-04T09:25:24.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.799+0000 7f244d800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24480b3170 con 0x7f2448161b40 2024-03-04T09:25:24.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.799+0000 7f2446000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f24480b54a0 0x7f244815d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:24.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.799+0000 7f2446000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f24480b54a0 0x7f244815d070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45078/0 (socket says 172.21.15.6:45078) 2024-03-04T09:25:24.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.799+0000 7f2446000640 1 -- 172.21.15.6:0/328031832 learned_addr learned my addr 172.21.15.6:0/328031832 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:25:24.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.800+0000 7f2446a00640 1 --2- 172.21.15.6:0/328031832 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f24480b5060 0x7f24480b9500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:24.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.800+0000 7f2447400640 1 --2- 172.21.15.6:0/328031832 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2448161b40 0x7f2448151f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:24.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.800+0000 7f2446000640 1 -- 172.21.15.6:0/328031832 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2448161b40 msgr2=0x7f2448151f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:24.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.800+0000 7f2446000640 1 --2- 172.21.15.6:0/328031832 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2448161b40 0x7f2448151f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:24.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.800+0000 7f2446000640 1 -- 172.21.15.6:0/328031832 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f24480b5060 msgr2=0x7f24480b9500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:24.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.800+0000 7f2446000640 1 --2- 172.21.15.6:0/328031832 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f24480b5060 0x7f24480b9500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:24.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.800+0000 7f2446000640 1 -- 172.21.15.6:0/328031832 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24480b8460 con 0x7f24480b54a0 2024-03-04T09:25:24.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.800+0000 7f2446000640 1 --2- 172.21.15.6:0/328031832 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f24480b54a0 0x7f244815d070 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f243000c7f0 tx=0x7f243000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:24.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.800+0000 7f243ea00640 1 -- 172.21.15.6:0/328031832 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2430015070 con 0x7f24480b54a0 2024-03-04T09:25:24.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.800+0000 7f243ea00640 1 -- 172.21.15.6:0/328031832 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2430007820 con 0x7f24480b54a0 2024-03-04T09:25:24.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.800+0000 7f243ea00640 1 -- 172.21.15.6:0/328031832 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2430007bd0 con 0x7f24480b54a0 2024-03-04T09:25:24.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.801+0000 7f244d800640 1 -- 172.21.15.6:0/328031832 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24480b8740 con 0x7f24480b54a0 2024-03-04T09:25:24.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.801+0000 7f244d800640 1 -- 172.21.15.6:0/328031832 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24480b8d00 con 0x7f24480b54a0 2024-03-04T09:25:24.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.802+0000 7f244d800640 1 -- 172.21.15.6:0/328031832 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f244804f3a0 con 0x7f24480b54a0 2024-03-04T09:25:24.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.803+0000 7f243ea00640 1 -- 172.21.15.6:0/328031832 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2430018070 con 0x7f24480b54a0 2024-03-04T09:25:24.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.803+0000 7f243ea00640 1 --2- 172.21.15.6:0/328031832 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f242406b4d0 0x7f242406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:24.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.804+0000 7f243ea00640 1 -- 172.21.15.6:0/328031832 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f243008a370 con 0x7f24480b54a0 2024-03-04T09:25:24.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.804+0000 7f2446a00640 1 --2- 172.21.15.6:0/328031832 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f242406b4d0 0x7f242406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:24.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.805+0000 7f2446a00640 1 --2- 172.21.15.6:0/328031832 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f242406b4d0 0x7f242406d990 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f2438035000 tx=0x7f2438030000 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:24.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:24.808+0000 7f243ea00640 1 -- 172.21.15.6:0/328031832 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f24300591f0 con 0x7f24480b54a0 2024-03-04T09:25:25.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.054+0000 7f244d800640 1 -- 172.21.15.6:0/328031832 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f24480d1df0 con 0x7f24480b54a0 2024-03-04T09:25:25.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.059+0000 7f243ea00640 1 -- 172.21.15.6:0/328031832 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+171789 (secure 0 0 0) 0x7f2430058af0 con 0x7f24480b54a0 2024-03-04T09:25:25.059 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:25:25.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.064+0000 7f244d800640 1 -- 172.21.15.6:0/328031832 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f242406b4d0 msgr2=0x7f242406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:25.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.064+0000 7f244d800640 1 --2- 172.21.15.6:0/328031832 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f242406b4d0 0x7f242406d990 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f2438035000 tx=0x7f2438030000 comp rx=0 tx=0).stop 2024-03-04T09:25:25.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.064+0000 7f244d800640 1 -- 172.21.15.6:0/328031832 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f24480b54a0 msgr2=0x7f244815d070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:25.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.064+0000 7f244d800640 1 --2- 172.21.15.6:0/328031832 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f24480b54a0 0x7f244815d070 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f243000c7f0 tx=0x7f243000ccb0 comp rx=0 tx=0).stop 2024-03-04T09:25:25.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.064+0000 7f244d800640 1 -- 172.21.15.6:0/328031832 shutdown_connections 2024-03-04T09:25:25.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.064+0000 7f244d800640 1 --2- 172.21.15.6:0/328031832 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f242406b4d0 0x7f242406d990 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:25.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.064+0000 7f244d800640 1 --2- 172.21.15.6:0/328031832 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2448161b40 0x7f2448151f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:25.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.064+0000 7f244d800640 1 --2- 172.21.15.6:0/328031832 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f24480b54a0 0x7f244815d070 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:25.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.064+0000 7f244d800640 1 --2- 172.21.15.6:0/328031832 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f24480b5060 0x7f24480b9500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:25.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.064+0000 7f244d800640 1 -- 172.21.15.6:0/328031832 >> 172.21.15.6:0/328031832 conn(0x7f244809d220 msgr2=0x7f24480b3870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:25.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.064+0000 7f244d800640 1 -- 172.21.15.6:0/328031832 shutdown_connections 2024-03-04T09:25:25.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.064+0000 7f244d800640 1 -- 172.21.15.6:0/328031832 wait complete. 2024-03-04T09:25:25.079 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":5,"flags":0,"active_gid":4113,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":1788613334},{"type":"v1","addr":"172.21.15.146:6833","nonce":1788613334}]},"active_addr":"172.21.15.146:6833/1788613334","active_change":"2024-03-04T09:24:59.836721+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":4116,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.146:0","nonce":284794247}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.146:0","nonce":147540649}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.146:0","nonce":871549788}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.146:0","nonce":2449230551}]}]} 2024-03-04T09:25:25.083 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-03-04T09:25:25.083 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-03-04T09:25:25.083 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T09:25:25.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.301+0000 7efc07200640 1 Processor -- start 2024-03-04T09:25:25.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.301+0000 7efc07200640 1 -- start start 2024-03-04T09:25:25.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.302+0000 7efc07200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7efc00070f10 0x7efc000cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:25.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.303+0000 7efc07200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efc000cce90 0x7efc0015c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:25.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.303+0000 7efc07200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc000714d0 con 0x7efc00145300 2024-03-04T09:25:25.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.303+0000 7efc07200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc000cd300 con 0x7efc00070f10 2024-03-04T09:25:25.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.303+0000 7efc07200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc000717b0 con 0x7efc000cce90 2024-03-04T09:25:25.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.303+0000 7efc05400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7efc00070f10 0x7efc000cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:25.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.303+0000 7efc05e00640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7efc00145300 0x7efc000b49a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.6:50106/0 (socket says 172.21.15.6:50106) 2024-03-04T09:25:25.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.303+0000 7efc05e00640 1 -- 172.21.15.6:0/2152893249 learned_addr learned my addr 172.21.15.6:0/2152893249 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:25:25.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.303+0000 7efc06800640 1 --2- 172.21.15.6:0/2152893249 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efc000cce90 0x7efc0015c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:25.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.304+0000 7efc06800640 1 -- 172.21.15.6:0/2152893249 >> v2:172.21.15.146:3300/0 conn(0x7efc00070f10 msgr2=0x7efc000cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:25.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.304+0000 7efc06800640 1 --2- 172.21.15.6:0/2152893249 >> v2:172.21.15.146:3300/0 conn(0x7efc00070f10 0x7efc000cc950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:25.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.304+0000 7efc06800640 1 -- 172.21.15.6:0/2152893249 >> v1:172.21.15.146:6789/0 conn(0x7efc00145300 legacy=0x7efc000b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:25.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.304+0000 7efc06800640 1 -- 172.21.15.6:0/2152893249 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc00071a90 con 0x7efc000cce90 2024-03-04T09:25:25.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.306+0000 7efc06800640 1 --2- 172.21.15.6:0/2152893249 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efc000cce90 0x7efc0015c570 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7efbfc009d90 tx=0x7efbfc02be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=53046913410cb5e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:25.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.307+0000 7efc04a00640 1 -- 172.21.15.6:0/2152893249 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbfc039070 con 0x7efc000cce90 2024-03-04T09:25:25.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.307+0000 7efc04a00640 1 -- 172.21.15.6:0/2152893249 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efbfc02cbc0 con 0x7efc000cce90 2024-03-04T09:25:25.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.307+0000 7efc04a00640 1 -- 172.21.15.6:0/2152893249 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbfc034440 con 0x7efc000cce90 2024-03-04T09:25:25.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.307+0000 7efc07200640 1 -- 172.21.15.6:0/2152893249 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efc000cce90 msgr2=0x7efc0015c570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:25.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.307+0000 7efc07200640 1 --2- 172.21.15.6:0/2152893249 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efc000cce90 0x7efc0015c570 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7efbfc009d90 tx=0x7efbfc02be10 comp rx=0 tx=0).stop 2024-03-04T09:25:25.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.307+0000 7efc07200640 1 -- 172.21.15.6:0/2152893249 shutdown_connections 2024-03-04T09:25:25.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.307+0000 7efc07200640 1 --2- 172.21.15.6:0/2152893249 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efc000cce90 0x7efc0015c570 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:25.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.307+0000 7efc07200640 1 --2- 172.21.15.6:0/2152893249 >> v2:172.21.15.146:3300/0 conn(0x7efc00070f10 0x7efc000cc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:25.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.307+0000 7efc07200640 1 -- 172.21.15.6:0/2152893249 >> 172.21.15.6:0/2152893249 conn(0x7efc0006b3d0 msgr2=0x7efc0006b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:25.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.307+0000 7efc07200640 1 -- 172.21.15.6:0/2152893249 shutdown_connections 2024-03-04T09:25:25.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.307+0000 7efc07200640 1 -- 172.21.15.6:0/2152893249 wait complete. 2024-03-04T09:25:25.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.309+0000 7efc07200640 1 Processor -- start 2024-03-04T09:25:25.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.309+0000 7efc07200640 1 -- start start 2024-03-04T09:25:25.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.310+0000 7efc07200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efc00070f10 0x7efc0006d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:25.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.310+0000 7efc07200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efc000cce90 0x7efc0006dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:25.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.310+0000 7efc07200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efc00145300 0x7efc000ae400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:25.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.310+0000 7efc07200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc0015cab0 con 0x7efc000cce90 2024-03-04T09:25:25.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.310+0000 7efc07200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc001456e0 con 0x7efc00145300 2024-03-04T09:25:25.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.310+0000 7efc07200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc0015cc30 con 0x7efc00070f10 2024-03-04T09:25:25.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.310+0000 7efc05400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efc000cce90 0x7efc0006dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:25.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.310+0000 7efc05400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efc000cce90 0x7efc0006dc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45082/0 (socket says 172.21.15.6:45082) 2024-03-04T09:25:25.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.311+0000 7efc05400640 1 -- 172.21.15.6:0/1611682949 learned_addr learned my addr 172.21.15.6:0/1611682949 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:25:25.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.311+0000 7efc05e00640 1 --2- 172.21.15.6:0/1611682949 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efc00070f10 0x7efc0006d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:25.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.311+0000 7efc06800640 1 --2- 172.21.15.6:0/1611682949 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efc00145300 0x7efc000ae400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:25.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.311+0000 7efc05400640 1 -- 172.21.15.6:0/1611682949 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efc00070f10 msgr2=0x7efc0006d6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:25.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.311+0000 7efc05400640 1 --2- 172.21.15.6:0/1611682949 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efc00070f10 0x7efc0006d6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:25.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.311+0000 7efc05400640 1 -- 172.21.15.6:0/1611682949 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efc00145300 msgr2=0x7efc000ae400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:25.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.311+0000 7efc05400640 1 --2- 172.21.15.6:0/1611682949 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efc00145300 0x7efc000ae400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:25.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.311+0000 7efc05400640 1 -- 172.21.15.6:0/1611682949 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc0006e440 con 0x7efc000cce90 2024-03-04T09:25:25.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.311+0000 7efc05e00640 1 --2- 172.21.15.6:0/1611682949 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efc00070f10 0x7efc0006d6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T09:25:25.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.311+0000 7efc06800640 1 --2- 172.21.15.6:0/1611682949 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efc00145300 0x7efc000ae400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:25:25.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.311+0000 7efc05400640 1 --2- 172.21.15.6:0/1611682949 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efc000cce90 0x7efc0006dc10 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7efbf000e9b0 tx=0x7efbf000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:25.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.312+0000 7efbfa000640 1 -- 172.21.15.6:0/1611682949 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbf0017070 con 0x7efc000cce90 2024-03-04T09:25:25.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.312+0000 7efbfa000640 1 -- 172.21.15.6:0/1611682949 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efbf000c9a0 con 0x7efc000cce90 2024-03-04T09:25:25.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.312+0000 7efbfa000640 1 -- 172.21.15.6:0/1611682949 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbf000cd90 con 0x7efc000cce90 2024-03-04T09:25:25.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.312+0000 7efc07200640 1 -- 172.21.15.6:0/1611682949 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc0006e720 con 0x7efc000cce90 2024-03-04T09:25:25.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.312+0000 7efc07200640 1 -- 172.21.15.6:0/1611682949 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc000adee0 con 0x7efc000cce90 2024-03-04T09:25:25.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.313+0000 7efc07200640 1 -- 172.21.15.6:0/1611682949 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc0004f330 con 0x7efc000cce90 2024-03-04T09:25:25.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.314+0000 7efbfa000640 1 -- 172.21.15.6:0/1611682949 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efbf001a070 con 0x7efc000cce90 2024-03-04T09:25:25.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.315+0000 7efbfa000640 1 --2- 172.21.15.6:0/1611682949 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7efbe406b4d0 0x7efbe406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:25.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.315+0000 7efbfa000640 1 -- 172.21.15.6:0/1611682949 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efbf008a680 con 0x7efc000cce90 2024-03-04T09:25:25.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.316+0000 7efc05e00640 1 --2- 172.21.15.6:0/1611682949 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7efbe406b4d0 0x7efbe406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:25.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.317+0000 7efc05e00640 1 --2- 172.21.15.6:0/1611682949 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7efbe406b4d0 0x7efbe406d990 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7efc000ce360 tx=0x7efbf4009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:25.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.320+0000 7efbfa000640 1 -- 172.21.15.6:0/1611682949 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7efbf0059460 con 0x7efc000cce90 2024-03-04T09:25:25.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.522+0000 7efc07200640 1 -- 172.21.15.6:0/1611682949 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efc0006e150 con 0x7efc000cce90 2024-03-04T09:25:25.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.525+0000 7efbfa000640 1 -- 172.21.15.6:0/1611682949 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16557 (secure 0 0 0) 0x7efbf0058d60 con 0x7efc000cce90 2024-03-04T09:25:25.522 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:25:25.522 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":20,"fsid":"edce0578-5ae4-4402-b45a-552bdef84d7c","created":"2024-03-04T09:24:54.523780+0000","modified":"2024-03-04T09:25:20.446333+0000","last_up_change":"2024-03-04T09:25:07.089049+0000","last_in_change":"2024-03-04T09:24:57.982077+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T09:25:07.935198+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-04T09:25:17.118573+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-04T09:25:17.930536+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a96574f-81f1-4bc6-8463-757520b84a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6825","nonce":1364449401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6827","nonce":1364449401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6831","nonce":1364449401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6829","nonce":1364449401}]},"public_addr":"172.21.15.6:6825/1364449401","cluster_addr":"172.21.15.6:6827/1364449401","heartbeat_back_addr":"172.21.15.6:6831/1364449401","heartbeat_front_addr":"172.21.15.6:6829/1364449401","state":["exists","up"]},{"osd":1,"uuid":"7d79b9c1-6b87-476a-bc36-7db5ee64cd90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6809","nonce":2494103078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6811","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6813","nonce":2494103078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6821","nonce":2494103078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6817","nonce":2494103078}]},"public_addr":"172.21.15.6:6809/2494103078","cluster_addr":"172.21.15.6:6813/2494103078","heartbeat_back_addr":"172.21.15.6:6821/2494103078","heartbeat_front_addr":"172.21.15.6:6817/2494103078","state":["exists","up"]},{"osd":2,"uuid":"492331b9-4ee9-40b6-bacd-85d44cd6c45a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6812","nonce":1500066681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6816","nonce":1500066681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6824","nonce":1500066681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6820","nonce":1500066681}]},"public_addr":"172.21.15.6:6812/1500066681","cluster_addr":"172.21.15.6:6816/1500066681","heartbeat_back_addr":"172.21.15.6:6824/1500066681","heartbeat_front_addr":"172.21.15.6:6820/1500066681","state":["exists","up"]},{"osd":3,"uuid":"21bb2efe-9ba7-4bdc-bc31-180cd1102e21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6801","nonce":1558806123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6803","nonce":1558806123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6807","nonce":1558806123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6805","nonce":1558806123}]},"public_addr":"172.21.15.6:6801/1558806123","cluster_addr":"172.21.15.6:6803/1558806123","heartbeat_back_addr":"172.21.15.6:6807/1558806123","heartbeat_front_addr":"172.21.15.6:6805/1558806123","state":["exists","up"]},{"osd":4,"uuid":"6e2c4b71-19d9-4737-83d9-558705d8644a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6801","nonce":1526135780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6803","nonce":1526135780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6807","nonce":1526135780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6805","nonce":1526135780}]},"public_addr":"172.21.15.146:6801/1526135780","cluster_addr":"172.21.15.146:6803/1526135780","heartbeat_back_addr":"172.21.15.146:6807/1526135780","heartbeat_front_addr":"172.21.15.146:6805/1526135780","state":["exists","up"]},{"osd":5,"uuid":"fb69c525-f1be-4b7d-ab32-6c29b0d6fddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6809","nonce":460262507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6811","nonce":460262507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6815","nonce":460262507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6813","nonce":460262507}]},"public_addr":"172.21.15.146:6809/460262507","cluster_addr":"172.21.15.146:6811/460262507","heartbeat_back_addr":"172.21.15.146:6815/460262507","heartbeat_front_addr":"172.21.15.146:6813/460262507","state":["exists","up"]},{"osd":6,"uuid":"c32e7fed-370d-4401-8ce0-6a4335b0d168","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6823","nonce":1392897493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6827","nonce":1392897493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6831","nonce":1392897493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6829","nonce":1392897493}]},"public_addr":"172.21.15.146:6823/1392897493","cluster_addr":"172.21.15.146:6827/1392897493","heartbeat_back_addr":"172.21.15.146:6831/1392897493","heartbeat_front_addr":"172.21.15.146:6829/1392897493","state":["exists","up"]},{"osd":7,"uuid":"75a79b44-fa5f-426d-a7d3-71f21f78835c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6817","nonce":3268323156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6819","nonce":3268323156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6825","nonce":3268323156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6822","nonce":3268323156}]},"public_addr":"172.21.15.146:6817/3268323156","cluster_addr":"172.21.15.146:6819/3268323156","heartbeat_back_addr":"172.21.15.146:6825/3268323156","heartbeat_front_addr":"172.21.15.146:6822/3268323156","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.310437+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.295850+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.294674+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.526855+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.300023+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.595160+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.346046+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.558741+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T09:25:25.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.528+0000 7efc07200640 1 -- 172.21.15.6:0/1611682949 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7efbe406b4d0 msgr2=0x7efbe406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:25.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.528+0000 7efc07200640 1 --2- 172.21.15.6:0/1611682949 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7efbe406b4d0 0x7efbe406d990 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7efc000ce360 tx=0x7efbf4009040 comp rx=0 tx=0).stop 2024-03-04T09:25:25.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.528+0000 7efc07200640 1 -- 172.21.15.6:0/1611682949 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efc000cce90 msgr2=0x7efc0006dc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:25.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.528+0000 7efc07200640 1 --2- 172.21.15.6:0/1611682949 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efc000cce90 0x7efc0006dc10 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7efbf000e9b0 tx=0x7efbf000ee70 comp rx=0 tx=0).stop 2024-03-04T09:25:25.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.528+0000 7efc07200640 1 -- 172.21.15.6:0/1611682949 shutdown_connections 2024-03-04T09:25:25.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.528+0000 7efc07200640 1 --2- 172.21.15.6:0/1611682949 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efc00145300 0x7efc000ae400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:25.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.528+0000 7efc07200640 1 --2- 172.21.15.6:0/1611682949 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efc000cce90 0x7efc0006dc10 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:25.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.528+0000 7efc07200640 1 --2- 172.21.15.6:0/1611682949 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7efbe406b4d0 0x7efbe406d990 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:25.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.528+0000 7efc07200640 1 --2- 172.21.15.6:0/1611682949 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efc00070f10 0x7efc0006d6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:25.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.528+0000 7efc07200640 1 -- 172.21.15.6:0/1611682949 >> 172.21.15.6:0/1611682949 conn(0x7efc0006b3d0 msgr2=0x7efc000b0960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:25.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.528+0000 7efc07200640 1 -- 172.21.15.6:0/1611682949 shutdown_connections 2024-03-04T09:25:25.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.528+0000 7efc07200640 1 -- 172.21.15.6:0/1611682949 wait complete. 2024-03-04T09:25:25.543 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-03-04T09:25:25.543 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T09:25:25.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.751+0000 7ff3dea00640 1 Processor -- start 2024-03-04T09:25:25.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.751+0000 7ff3dea00640 1 -- start start 2024-03-04T09:25:25.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.753+0000 7ff3dea00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff3d80cd060 0x7ff3d815b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:25.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.753+0000 7ff3dea00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff3d815e810 0x7ff3d80b3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:25.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.753+0000 7ff3dea00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3d815ed10 con 0x7ff3d81552b0 2024-03-04T09:25:25.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.753+0000 7ff3dea00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3d815eff0 con 0x7ff3d815e810 2024-03-04T09:25:25.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.753+0000 7ff3dea00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3d815f2d0 con 0x7ff3d80cd060 2024-03-04T09:25:25.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.753+0000 7ff3de000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff3d815e810 0x7ff3d80b3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:25.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.753+0000 7ff3de000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff3d815e810 0x7ff3d80b3cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45094/0 (socket says 172.21.15.6:45094) 2024-03-04T09:25:25.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.753+0000 7ff3de000640 1 -- 172.21.15.6:0/1133331949 learned_addr learned my addr 172.21.15.6:0/1133331949 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:25:25.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.753+0000 7ff3dcc00640 1 --2- 172.21.15.6:0/1133331949 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff3d80cd060 0x7ff3d815b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:25.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.754+0000 7ff3de000640 1 -- 172.21.15.6:0/1133331949 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff3d80cd060 msgr2=0x7ff3d815b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:25.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.754+0000 7ff3de000640 1 --2- 172.21.15.6:0/1133331949 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff3d80cd060 0x7ff3d815b1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:25.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.754+0000 7ff3de000640 1 -- 172.21.15.6:0/1133331949 >> v1:172.21.15.146:6789/0 conn(0x7ff3d81552b0 legacy=0x7ff3d80cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:25.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.754+0000 7ff3de000640 1 -- 172.21.15.6:0/1133331949 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3d815f5b0 con 0x7ff3d815e810 2024-03-04T09:25:25.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.755+0000 7ff3de000640 1 --2- 172.21.15.6:0/1133331949 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff3d815e810 0x7ff3d80b3cb0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7ff3d0009870 tx=0x7ff3d002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63adbe1f38848073 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:25.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.755+0000 7ff3d7e00640 1 -- 172.21.15.6:0/1133331949 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3d0035020 con 0x7ff3d815e810 2024-03-04T09:25:25.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.755+0000 7ff3d7e00640 1 -- 172.21.15.6:0/1133331949 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3d0031640 con 0x7ff3d815e810 2024-03-04T09:25:25.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.755+0000 7ff3d7e00640 1 -- 172.21.15.6:0/1133331949 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3d002d040 con 0x7ff3d815e810 2024-03-04T09:25:25.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.755+0000 7ff3dea00640 1 -- 172.21.15.6:0/1133331949 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff3d815e810 msgr2=0x7ff3d80b3cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:25.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.755+0000 7ff3dea00640 1 --2- 172.21.15.6:0/1133331949 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff3d815e810 0x7ff3d80b3cb0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7ff3d0009870 tx=0x7ff3d002b8f0 comp rx=0 tx=0).stop 2024-03-04T09:25:25.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.756+0000 7ff3dea00640 1 -- 172.21.15.6:0/1133331949 shutdown_connections 2024-03-04T09:25:25.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.756+0000 7ff3dea00640 1 --2- 172.21.15.6:0/1133331949 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff3d815e810 0x7ff3d80b3cb0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:25.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.756+0000 7ff3dea00640 1 --2- 172.21.15.6:0/1133331949 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff3d80cd060 0x7ff3d815b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:25.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.756+0000 7ff3dea00640 1 -- 172.21.15.6:0/1133331949 >> 172.21.15.6:0/1133331949 conn(0x7ff3d806b3d0 msgr2=0x7ff3d806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:25.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.756+0000 7ff3dea00640 1 -- 172.21.15.6:0/1133331949 shutdown_connections 2024-03-04T09:25:25.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.756+0000 7ff3dea00640 1 -- 172.21.15.6:0/1133331949 wait complete. 2024-03-04T09:25:25.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.757+0000 7ff3dea00640 1 Processor -- start 2024-03-04T09:25:25.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.757+0000 7ff3dea00640 1 -- start start 2024-03-04T09:25:25.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.758+0000 7ff3dea00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff3d80cd060 0x7ff3d80ae770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:25.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.758+0000 7ff3dea00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff3d81552b0 0x7ff3d80a9470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:25.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.758+0000 7ff3dea00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff3d815e810 0x7ff3d814c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:25.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.758+0000 7ff3dea00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3d806dd30 con 0x7ff3d815e810 2024-03-04T09:25:25.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.758+0000 7ff3dea00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3d80b3760 con 0x7ff3d81552b0 2024-03-04T09:25:25.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.758+0000 7ff3dea00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3d806deb0 con 0x7ff3d80cd060 2024-03-04T09:25:25.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.759+0000 7ff3de000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff3d815e810 0x7ff3d814c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:25.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.759+0000 7ff3de000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff3d815e810 0x7ff3d814c3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45098/0 (socket says 172.21.15.6:45098) 2024-03-04T09:25:25.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.759+0000 7ff3de000640 1 -- 172.21.15.6:0/2884233828 learned_addr learned my addr 172.21.15.6:0/2884233828 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:25:25.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.759+0000 7ff3dd600640 1 --2- 172.21.15.6:0/2884233828 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff3d80cd060 0x7ff3d80ae770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:25.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.759+0000 7ff3dcc00640 1 --2- 172.21.15.6:0/2884233828 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff3d81552b0 0x7ff3d80a9470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:25.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.759+0000 7ff3de000640 1 -- 172.21.15.6:0/2884233828 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff3d80cd060 msgr2=0x7ff3d80ae770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:25.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.759+0000 7ff3de000640 1 --2- 172.21.15.6:0/2884233828 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff3d80cd060 0x7ff3d80ae770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:25.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.759+0000 7ff3de000640 1 -- 172.21.15.6:0/2884233828 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff3d81552b0 msgr2=0x7ff3d80a9470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:25.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.759+0000 7ff3de000640 1 --2- 172.21.15.6:0/2884233828 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff3d81552b0 0x7ff3d80a9470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:25.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.759+0000 7ff3de000640 1 -- 172.21.15.6:0/2884233828 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3d80aee70 con 0x7ff3d815e810 2024-03-04T09:25:25.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.759+0000 7ff3dd600640 1 --2- 172.21.15.6:0/2884233828 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff3d80cd060 0x7ff3d80ae770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:25:25.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.759+0000 7ff3dcc00640 1 --2- 172.21.15.6:0/2884233828 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff3d81552b0 0x7ff3d80a9470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:25:25.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.759+0000 7ff3de000640 1 --2- 172.21.15.6:0/2884233828 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff3d815e810 0x7ff3d814c3e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7ff3d0009870 tx=0x7ff3d00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:25.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.760+0000 7ff3d5600640 1 -- 172.21.15.6:0/2884233828 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3d0040070 con 0x7ff3d815e810 2024-03-04T09:25:25.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.760+0000 7ff3d5600640 1 -- 172.21.15.6:0/2884233828 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3d00399f0 con 0x7ff3d815e810 2024-03-04T09:25:25.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.760+0000 7ff3d5600640 1 -- 172.21.15.6:0/2884233828 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3d0035040 con 0x7ff3d815e810 2024-03-04T09:25:25.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.760+0000 7ff3dea00640 1 -- 172.21.15.6:0/2884233828 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3d80af150 con 0x7ff3d815e810 2024-03-04T09:25:25.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.760+0000 7ff3dea00640 1 -- 172.21.15.6:0/2884233828 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3d80afbd0 con 0x7ff3d815e810 2024-03-04T09:25:25.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.761+0000 7ff3dea00640 1 -- 172.21.15.6:0/2884233828 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3d8068720 con 0x7ff3d815e810 2024-03-04T09:25:25.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.762+0000 7ff3d5600640 1 -- 172.21.15.6:0/2884233828 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff3d004b060 con 0x7ff3d815e810 2024-03-04T09:25:25.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.763+0000 7ff3d5600640 1 --2- 172.21.15.6:0/2884233828 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff3b806b4d0 0x7ff3b806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:25.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.763+0000 7ff3d5600640 1 -- 172.21.15.6:0/2884233828 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff3d00aba80 con 0x7ff3d815e810 2024-03-04T09:25:25.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.764+0000 7ff3dd600640 1 --2- 172.21.15.6:0/2884233828 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff3b806b4d0 0x7ff3b806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:25.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.765+0000 7ff3dd600640 1 --2- 172.21.15.6:0/2884233828 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff3b806b4d0 0x7ff3b806d990 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7ff3c4001680 tx=0x7ff3c4009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:25.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.767+0000 7ff3d5600640 1 -- 172.21.15.6:0/2884233828 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff3d8068720 con 0x7ff3d815e810 2024-03-04T09:25:25.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.969+0000 7ff3dea00640 1 -- 172.21.15.6:0/2884233828 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff3d804f330 con 0x7ff3d815e810 2024-03-04T09:25:25.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.972+0000 7ff3d5600640 1 -- 172.21.15.6:0/2884233828 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16557 (secure 0 0 0) 0x7ff3d00ade20 con 0x7ff3d815e810 2024-03-04T09:25:25.970 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:25:25.970 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":20,"fsid":"edce0578-5ae4-4402-b45a-552bdef84d7c","created":"2024-03-04T09:24:54.523780+0000","modified":"2024-03-04T09:25:20.446333+0000","last_up_change":"2024-03-04T09:25:07.089049+0000","last_in_change":"2024-03-04T09:24:57.982077+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T09:25:07.935198+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-04T09:25:17.118573+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-04T09:25:17.930536+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a96574f-81f1-4bc6-8463-757520b84a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6825","nonce":1364449401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6827","nonce":1364449401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6831","nonce":1364449401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6829","nonce":1364449401}]},"public_addr":"172.21.15.6:6825/1364449401","cluster_addr":"172.21.15.6:6827/1364449401","heartbeat_back_addr":"172.21.15.6:6831/1364449401","heartbeat_front_addr":"172.21.15.6:6829/1364449401","state":["exists","up"]},{"osd":1,"uuid":"7d79b9c1-6b87-476a-bc36-7db5ee64cd90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6809","nonce":2494103078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6811","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6813","nonce":2494103078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6821","nonce":2494103078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6817","nonce":2494103078}]},"public_addr":"172.21.15.6:6809/2494103078","cluster_addr":"172.21.15.6:6813/2494103078","heartbeat_back_addr":"172.21.15.6:6821/2494103078","heartbeat_front_addr":"172.21.15.6:6817/2494103078","state":["exists","up"]},{"osd":2,"uuid":"492331b9-4ee9-40b6-bacd-85d44cd6c45a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6812","nonce":1500066681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6816","nonce":1500066681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6824","nonce":1500066681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6820","nonce":1500066681}]},"public_addr":"172.21.15.6:6812/1500066681","cluster_addr":"172.21.15.6:6816/1500066681","heartbeat_back_addr":"172.21.15.6:6824/1500066681","heartbeat_front_addr":"172.21.15.6:6820/1500066681","state":["exists","up"]},{"osd":3,"uuid":"21bb2efe-9ba7-4bdc-bc31-180cd1102e21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6801","nonce":1558806123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6803","nonce":1558806123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6807","nonce":1558806123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6805","nonce":1558806123}]},"public_addr":"172.21.15.6:6801/1558806123","cluster_addr":"172.21.15.6:6803/1558806123","heartbeat_back_addr":"172.21.15.6:6807/1558806123","heartbeat_front_addr":"172.21.15.6:6805/1558806123","state":["exists","up"]},{"osd":4,"uuid":"6e2c4b71-19d9-4737-83d9-558705d8644a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6801","nonce":1526135780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6803","nonce":1526135780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6807","nonce":1526135780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6805","nonce":1526135780}]},"public_addr":"172.21.15.146:6801/1526135780","cluster_addr":"172.21.15.146:6803/1526135780","heartbeat_back_addr":"172.21.15.146:6807/1526135780","heartbeat_front_addr":"172.21.15.146:6805/1526135780","state":["exists","up"]},{"osd":5,"uuid":"fb69c525-f1be-4b7d-ab32-6c29b0d6fddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6809","nonce":460262507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6811","nonce":460262507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6815","nonce":460262507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6813","nonce":460262507}]},"public_addr":"172.21.15.146:6809/460262507","cluster_addr":"172.21.15.146:6811/460262507","heartbeat_back_addr":"172.21.15.146:6815/460262507","heartbeat_front_addr":"172.21.15.146:6813/460262507","state":["exists","up"]},{"osd":6,"uuid":"c32e7fed-370d-4401-8ce0-6a4335b0d168","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6823","nonce":1392897493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6827","nonce":1392897493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6831","nonce":1392897493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6829","nonce":1392897493}]},"public_addr":"172.21.15.146:6823/1392897493","cluster_addr":"172.21.15.146:6827/1392897493","heartbeat_back_addr":"172.21.15.146:6831/1392897493","heartbeat_front_addr":"172.21.15.146:6829/1392897493","state":["exists","up"]},{"osd":7,"uuid":"75a79b44-fa5f-426d-a7d3-71f21f78835c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6817","nonce":3268323156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6819","nonce":3268323156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6825","nonce":3268323156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6822","nonce":3268323156}]},"public_addr":"172.21.15.146:6817/3268323156","cluster_addr":"172.21.15.146:6819/3268323156","heartbeat_back_addr":"172.21.15.146:6825/3268323156","heartbeat_front_addr":"172.21.15.146:6822/3268323156","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.310437+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.295850+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.294674+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.526855+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.300023+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.595160+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.346046+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.558741+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T09:25:25.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.975+0000 7ff3dea00640 1 -- 172.21.15.6:0/2884233828 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff3b806b4d0 msgr2=0x7ff3b806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:25.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.975+0000 7ff3dea00640 1 --2- 172.21.15.6:0/2884233828 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff3b806b4d0 0x7ff3b806d990 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7ff3c4001680 tx=0x7ff3c4009040 comp rx=0 tx=0).stop 2024-03-04T09:25:25.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.975+0000 7ff3dea00640 1 -- 172.21.15.6:0/2884233828 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff3d815e810 msgr2=0x7ff3d814c3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:25.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.975+0000 7ff3dea00640 1 --2- 172.21.15.6:0/2884233828 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff3d815e810 0x7ff3d814c3e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7ff3d0009870 tx=0x7ff3d00046c0 comp rx=0 tx=0).stop 2024-03-04T09:25:25.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.975+0000 7ff3dea00640 1 -- 172.21.15.6:0/2884233828 shutdown_connections 2024-03-04T09:25:25.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.975+0000 7ff3dea00640 1 --2- 172.21.15.6:0/2884233828 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff3d815e810 0x7ff3d814c3e0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:25.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.975+0000 7ff3dea00640 1 --2- 172.21.15.6:0/2884233828 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff3d81552b0 0x7ff3d80a9470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:25.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.975+0000 7ff3dea00640 1 --2- 172.21.15.6:0/2884233828 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff3b806b4d0 0x7ff3b806d990 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:25.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.975+0000 7ff3dea00640 1 --2- 172.21.15.6:0/2884233828 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff3d80cd060 0x7ff3d80ae770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:25.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.975+0000 7ff3dea00640 1 -- 172.21.15.6:0/2884233828 >> 172.21.15.6:0/2884233828 conn(0x7ff3d806b3d0 msgr2=0x7ff3d80cc920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:25.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.975+0000 7ff3dea00640 1 -- 172.21.15.6:0/2884233828 shutdown_connections 2024-03-04T09:25:25.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:25.975+0000 7ff3dea00640 1 -- 172.21.15.6:0/2884233828 wait complete. 2024-03-04T09:25:25.992 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-03-04T09:25:25.993 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-03-04T09:25:25.993 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-03-04T09:25:25.994 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-03-04T09:25:25.995 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-03-04T09:25:25.995 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-03-04T09:25:25.996 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-03-04T09:25:25.996 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-03-04T09:25:26.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.295+0000 7f0adb000640 1 Processor -- start 2024-03-04T09:25:26.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.296+0000 7f0adb000640 1 -- start start 2024-03-04T09:25:26.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.299+0000 7f0adb000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0ad41607e0 0x7f0ad40d4a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:26.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.299+0000 7f0adb000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f0ad40b3f60 0x7f0ad40cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:26.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.300+0000 7f0ad3400640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f0ad40d27d0 0x7f0ad40b3850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.6:50120/0 (socket says 172.21.15.6:50120) 2024-03-04T09:25:26.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.300+0000 7f0ad3400640 1 -- 172.21.15.6:0/2226362000 learned_addr learned my addr 172.21.15.6:0/2226362000 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:25:26.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.300+0000 7f0ad3e00640 1 --2- 172.21.15.6:0/2226362000 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0ad41607e0 0x7f0ad40d4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:26.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.300+0000 7f0adb000640 1 -- 172.21.15.6:0/2226362000 --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ad40ceac0 con 0x7f0ad40d27d0 2024-03-04T09:25:26.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.300+0000 7f0adb000640 1 -- 172.21.15.6:0/2226362000 --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ad40cf690 con 0x7f0ad40b3f60 2024-03-04T09:25:26.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.300+0000 7f0adb000640 1 -- 172.21.15.6:0/2226362000 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ad40d0060 con 0x7f0ad41607e0 2024-03-04T09:25:26.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.301+0000 7f0ad9000640 1 --2- 172.21.15.6:0/2226362000 >> v2:172.21.15.6:3300/0 conn(0x7f0ad40b3f60 0x7f0ad40cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:26.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.303+0000 7f0ad3e00640 1 -- 172.21.15.6:0/2226362000 >> v2:172.21.15.6:3300/0 conn(0x7f0ad40b3f60 msgr2=0x7f0ad40cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:26.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.303+0000 7f0ad3e00640 1 --2- 172.21.15.6:0/2226362000 >> v2:172.21.15.6:3300/0 conn(0x7f0ad40b3f60 0x7f0ad40cfb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:26.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.303+0000 7f0ad3e00640 1 -- 172.21.15.6:0/2226362000 >> v1:172.21.15.146:6789/0 conn(0x7f0ad40d27d0 legacy=0x7f0ad40b3850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:26.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.303+0000 7f0ad3e00640 1 -- 172.21.15.6:0/2226362000 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ad40ced60 con 0x7f0ad41607e0 2024-03-04T09:25:26.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.305+0000 7f0ad3e00640 1 --2- 172.21.15.6:0/2226362000 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0ad41607e0 0x7f0ad40d4a10 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f0acc0097a0 tx=0x7f0acc02b710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dc49afcc6245e910 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:26.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.306+0000 7f0ad2a00640 1 -- 172.21.15.6:0/2226362000 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0acc037070 con 0x7f0ad41607e0 2024-03-04T09:25:26.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.306+0000 7f0ad2a00640 1 -- 172.21.15.6:0/2226362000 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0acc030710 con 0x7f0ad41607e0 2024-03-04T09:25:26.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.306+0000 7f0ad2a00640 1 -- 172.21.15.6:0/2226362000 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0acc030ae0 con 0x7f0ad41607e0 2024-03-04T09:25:26.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.307+0000 7fa477800640 1 Processor -- start 2024-03-04T09:25:26.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.307+0000 7fe058600640 1 Processor -- start 2024-03-04T09:25:26.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.307+0000 7f0adb000640 1 -- 172.21.15.6:0/2226362000 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0ad41607e0 msgr2=0x7f0ad40d4a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:26.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.307+0000 7f0adb000640 1 --2- 172.21.15.6:0/2226362000 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0ad41607e0 0x7f0ad40d4a10 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f0acc0097a0 tx=0x7f0acc02b710 comp rx=0 tx=0).stop 2024-03-04T09:25:26.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.307+0000 7f0adb000640 1 -- 172.21.15.6:0/2226362000 shutdown_connections 2024-03-04T09:25:26.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.307+0000 7f0adb000640 1 --2- 172.21.15.6:0/2226362000 >> v2:172.21.15.6:3300/0 conn(0x7f0ad40b3f60 0x7f0ad40cfb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:26.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.307+0000 7f0adb000640 1 --2- 172.21.15.6:0/2226362000 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0ad41607e0 0x7f0ad40d4a10 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:26.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.308+0000 7f0adb000640 1 -- 172.21.15.6:0/2226362000 >> 172.21.15.6:0/2226362000 conn(0x7f0ad40d3ef0 msgr2=0x7f0ad40d4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:26.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.307+0000 7fa477800640 1 -- start start 2024-03-04T09:25:26.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.307+0000 7fe058600640 1 -- start start 2024-03-04T09:25:26.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.308+0000 7f0adb000640 1 -- 172.21.15.6:0/2226362000 shutdown_connections 2024-03-04T09:25:26.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.308+0000 7f0adb000640 1 -- 172.21.15.6:0/2226362000 wait complete. 2024-03-04T09:25:26.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.309+0000 7fa477800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fa470161b40 0x7fa4700cfd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:26.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.309+0000 7fa477800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa4700b5060 0x7fa4700d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:26.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.309+0000 7fa477800640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4700cea50 con 0x7fa4700b54d0 2024-03-04T09:25:26.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.309+0000 7fa477800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4700cf900 con 0x7fa4700b5060 2024-03-04T09:25:26.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.309+0000 7fe058600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe05015e2e0 0x7fe0500d0830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:26.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.309+0000 7fe058600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fe0500b50e0 0x7fe0500d11d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:26.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.310+0000 7fe058600640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0500b3940 con 0x7fe0500d0d70 2024-03-04T09:25:26.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.310+0000 7fe058600640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0500d4e70 con 0x7fe0500b50e0 2024-03-04T09:25:26.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.310+0000 7fe058600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0500b3c40 con 0x7fe05015e2e0 2024-03-04T09:25:26.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.309+0000 7fa477800640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4700cecf0 con 0x7fa470161b40 2024-03-04T09:25:26.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.310+0000 7fa46fe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa4700b5060 0x7fa4700d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:26.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.310+0000 7fa46fe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa4700b5060 0x7fa4700d2220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45108/0 (socket says 172.21.15.6:45108) 2024-03-04T09:25:26.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.310+0000 7fa46fe00640 1 -- 172.21.15.6:0/2243291809 learned_addr learned my addr 172.21.15.6:0/2243291809 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:25:26.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.310+0000 7fa46fe00640 1 -- 172.21.15.6:0/2243291809 >> v2:172.21.15.146:3300/0 conn(0x7fa470161b40 msgr2=0x7fa4700cfd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:26.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.311+0000 7f0adb000640 1 Processor -- start 2024-03-04T09:25:26.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.311+0000 7fa46fe00640 1 --2- 172.21.15.6:0/2243291809 >> v2:172.21.15.146:3300/0 conn(0x7fa470161b40 0x7fa4700cfd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:26.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.311+0000 7fa46fe00640 1 -- 172.21.15.6:0/2243291809 >> v1:172.21.15.6:6789/0 conn(0x7fa4700b54d0 legacy=0x7fa4700b3740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:26.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.311+0000 7f0adb000640 1 -- start start 2024-03-04T09:25:26.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.311+0000 7fa46fe00640 1 -- 172.21.15.6:0/2243291809 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4700cefd0 con 0x7fa4700b5060 2024-03-04T09:25:26.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.311+0000 7fe056600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fe0500b50e0 0x7fe0500d11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:26.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.311+0000 7fe055200640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fe0500d0d70 0x7fe0500d1b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.6:50130/0 (socket says 172.21.15.6:50130) 2024-03-04T09:25:26.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.311+0000 7fe055200640 1 -- 172.21.15.6:0/2679010447 learned_addr learned my addr 172.21.15.6:0/2679010447 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:25:26.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.312+0000 7fe055c00640 1 --2- 172.21.15.6:0/2679010447 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe05015e2e0 0x7fe0500d0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:26.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.313+0000 7fa46fe00640 1 --2- 172.21.15.6:0/2243291809 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4700b5060 0x7fa4700d2220 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fa47008db00 tx=0x7fa45c02b9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a8ce5f54669838a server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:26.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.313+0000 7fe04fe00640 1 -- 172.21.15.6:0/2679010447 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2739265687 0 0) 0x7fe0500b3940 con 0x7fe0500d0d70 2024-03-04T09:25:26.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.313+0000 7fe04fe00640 1 -- 172.21.15.6:0/2679010447 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe038003520 con 0x7fe0500d0d70 2024-03-04T09:25:26.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.313+0000 7fa46f400640 1 -- 172.21.15.6:0/2243291809 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa45c034070 con 0x7fa4700b5060 2024-03-04T09:25:26.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.313+0000 7f0adb000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0ad40b3f60 0x7f0ad41f3aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:26.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.314+0000 7f0adb000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0ad40d27d0 0x7f0ad40c8430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:26.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.314+0000 7f0adb000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0ad41607e0 0x7f0ad40c9890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:26.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.314+0000 7fa46f400640 1 -- 172.21.15.6:0/2243291809 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa45c030880 con 0x7fa4700b5060 2024-03-04T09:25:26.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.314+0000 7f0ad3400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0ad40d27d0 0x7f0ad40c8430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:26.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.314+0000 7f0adb000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ad40b43c0 con 0x7f0ad40d27d0 2024-03-04T09:25:26.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.314+0000 7fa46f400640 1 -- 172.21.15.6:0/2243291809 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa45c030c90 con 0x7fa4700b5060 2024-03-04T09:25:26.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.315+0000 7f0adb000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ad40cf320 con 0x7f0ad40b3f60 2024-03-04T09:25:26.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.315+0000 7f0adb000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ad40d2c10 con 0x7f0ad41607e0 2024-03-04T09:25:26.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.314+0000 7f0ad3400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0ad40d27d0 0x7f0ad40c8430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45138/0 (socket says 172.21.15.6:45138) 2024-03-04T09:25:26.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.315+0000 7f0ad3400640 1 -- 172.21.15.6:0/1619855630 learned_addr learned my addr 172.21.15.6:0/1619855630 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:25:26.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.313+0000 7fe055c00640 1 -- 172.21.15.6:0/2679010447 >> v2:172.21.15.6:3300/0 conn(0x7fe0500b50e0 msgr2=0x7fe0500d11d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:26.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.313+0000 7fe055c00640 1 --2- 172.21.15.6:0/2679010447 >> v2:172.21.15.6:3300/0 conn(0x7fe0500b50e0 0x7fe0500d11d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:26.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.313+0000 7fe055c00640 1 -- 172.21.15.6:0/2679010447 >> v1:172.21.15.146:6789/0 conn(0x7fe0500d0d70 legacy=0x7fe0500d1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:26.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.313+0000 7fe055c00640 1 -- 172.21.15.6:0/2679010447 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0500cf580 con 0x7fe05015e2e0 2024-03-04T09:25:26.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.315+0000 7fe055c00640 1 --2- 172.21.15.6:0/2679010447 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe05015e2e0 0x7fe0500d0830 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe0440097a0 tx=0x7fe04402b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2589d0605f78058d server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:26.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.316+0000 7f0ad3400640 1 -- 172.21.15.6:0/1619855630 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0ad41607e0 msgr2=0x7f0ad40c9890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-04T09:25:26.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.316+0000 7fa477800640 1 -- 172.21.15.6:0/2243291809 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4700b5060 msgr2=0x7fa4700d2220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:26.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.316+0000 7fa477800640 1 --2- 172.21.15.6:0/2243291809 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4700b5060 0x7fa4700d2220 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fa47008db00 tx=0x7fa45c02b9c0 comp rx=0 tx=0).stop 2024-03-04T09:25:26.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.316+0000 7fe04fe00640 1 -- 172.21.15.6:0/2679010447 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe044035020 con 0x7fe05015e2e0 2024-03-04T09:25:26.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.316+0000 7f0ad3400640 1 --2- 172.21.15.6:0/1619855630 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0ad41607e0 0x7f0ad40c9890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:26.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.316+0000 7fe04fe00640 1 -- 172.21.15.6:0/2679010447 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe044031640 con 0x7fe05015e2e0 2024-03-04T09:25:26.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.316+0000 7f0ad3400640 1 -- 172.21.15.6:0/1619855630 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0ad40b3f60 msgr2=0x7f0ad41f3aa0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-04T09:25:26.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.316+0000 7f0ad3400640 1 --2- 172.21.15.6:0/1619855630 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0ad40b3f60 0x7f0ad41f3aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:26.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.316+0000 7fe04fe00640 1 -- 172.21.15.6:0/2679010447 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe04402d040 con 0x7fe05015e2e0 2024-03-04T09:25:26.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.316+0000 7f0ad3400640 1 -- 172.21.15.6:0/1619855630 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ad40cb060 con 0x7f0ad40d27d0 2024-03-04T09:25:26.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.317+0000 7f3fb3a00640 1 Processor -- start 2024-03-04T09:25:26.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.317+0000 7f0ad3400640 1 --2- 172.21.15.6:0/1619855630 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0ad40d27d0 0x7f0ad40c8430 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f0ac000c5f0 tx=0x7f0ac000cab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:26.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.317+0000 7f3fb3a00640 1 -- start start 2024-03-04T09:25:26.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.317+0000 7fe058600640 1 -- 172.21.15.6:0/2679010447 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe05015e2e0 msgr2=0x7fe0500d0830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:26.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.317+0000 7fe058600640 1 --2- 172.21.15.6:0/2679010447 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe05015e2e0 0x7fe0500d0830 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe0440097a0 tx=0x7fe04402b8f0 comp rx=0 tx=0).stop 2024-03-04T09:25:26.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.317+0000 7fe058600640 1 -- 172.21.15.6:0/2679010447 shutdown_connections 2024-03-04T09:25:26.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.317+0000 7fe058600640 1 --2- 172.21.15.6:0/2679010447 >> v2:172.21.15.6:3300/0 conn(0x7fe0500b50e0 0x7fe0500d11d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:26.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.317+0000 7fe058600640 1 --2- 172.21.15.6:0/2679010447 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe05015e2e0 0x7fe0500d0830 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:26.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.317+0000 7fe058600640 1 -- 172.21.15.6:0/2679010447 >> 172.21.15.6:0/2679010447 conn(0x7fe0500d3f60 msgr2=0x7fe0500d4370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:26.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.317+0000 7f0acbe00640 1 -- 172.21.15.6:0/1619855630 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ac0018070 con 0x7f0ad40d27d0 2024-03-04T09:25:26.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.318+0000 7fe058600640 1 -- 172.21.15.6:0/2679010447 shutdown_connections 2024-03-04T09:25:26.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.318+0000 7fe058600640 1 -- 172.21.15.6:0/2679010447 wait complete. 2024-03-04T09:25:26.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.318+0000 7fa477800640 1 -- 172.21.15.6:0/2243291809 shutdown_connections 2024-03-04T09:25:26.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.318+0000 7fa477800640 1 --2- 172.21.15.6:0/2243291809 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4700b5060 0x7fa4700d2220 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:26.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.318+0000 7fa477800640 1 --2- 172.21.15.6:0/2243291809 >> v2:172.21.15.146:3300/0 conn(0x7fa470161b40 0x7fa4700cfd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:26.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.318+0000 7fa477800640 1 -- 172.21.15.6:0/2243291809 >> 172.21.15.6:0/2243291809 conn(0x7fa47009d220 msgr2=0x7fa4700d3ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:26.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.318+0000 7f0adb000640 1 -- 172.21.15.6:0/1619855630 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ad41f3fe0 con 0x7f0ad40d27d0 2024-03-04T09:25:26.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.318+0000 7f0adb000640 1 -- 172.21.15.6:0/1619855630 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ad40c9190 con 0x7f0ad40d27d0 2024-03-04T09:25:26.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.318+0000 7fa477800640 1 -- 172.21.15.6:0/2243291809 shutdown_connections 2024-03-04T09:25:26.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.319+0000 7f6e9be00640 1 Processor -- start 2024-03-04T09:25:26.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.319+0000 7f3fb3a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3fac15e120 0x7f3fac070c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:26.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.319+0000 7f3fb3a00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f3fac0ce670 0x7f3fac0ceb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:26.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.319+0000 7f3fb3a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fac06d180 con 0x7f3fac0b2dd0 2024-03-04T09:25:26.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.319+0000 7f3fb3a00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fac0cdc20 con 0x7f3fac0ce670 2024-03-04T09:25:26.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.319+0000 7f3fb3a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fac0cde70 con 0x7f3fac15e120 2024-03-04T09:25:26.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.320+0000 7f3fabe00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f3fac0b2dd0 0x7f3fac0711b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:60186/0 (socket says 172.21.15.6:60186) 2024-03-04T09:25:26.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.320+0000 7f3fabe00640 1 -- 172.21.15.6:0/1154962132 learned_addr learned my addr 172.21.15.6:0/1154962132 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:25:26.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.319+0000 7f0acbe00640 1 -- 172.21.15.6:0/1619855630 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ac00074e0 con 0x7f0ad40d27d0 2024-03-04T09:25:26.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.319+0000 7f0acbe00640 1 -- 172.21.15.6:0/1619855630 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ac0004a00 con 0x7f0ad40d27d0 2024-03-04T09:25:26.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.320+0000 7f6e9be00640 1 -- start start 2024-03-04T09:25:26.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.320+0000 7fe058600640 1 Processor -- start 2024-03-04T09:25:26.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.320+0000 7f0adb000640 1 -- 172.21.15.6:0/1619855630 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f0aac001030 con 0x7f0ad40d27d0 2024-03-04T09:25:26.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.321+0000 7fe058600640 1 -- start start 2024-03-04T09:25:26.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.321+0000 7f3fab400640 1 -- 172.21.15.6:0/1154962132 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 274207965 0 0) 0x7f3fac06d180 con 0x7f3fac0b2dd0 2024-03-04T09:25:26.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.321+0000 7f3fab400640 1 -- 172.21.15.6:0/1154962132 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f90003520 con 0x7f3fac0b2dd0 2024-03-04T09:25:26.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.321+0000 7f3fb1000640 1 --2- 172.21.15.6:0/1154962132 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3fac15e120 0x7f3fac070c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:26.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.321+0000 7f3fab400640 1 -- 172.21.15.6:0/1154962132 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2106188787 0 0) 0x7f3f90003520 con 0x7f3fac0b2dd0 2024-03-04T09:25:26.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.321+0000 7f3fab400640 1 -- 172.21.15.6:0/1154962132 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3fac15e120 msgr2=0x7f3fac070c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:26.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.321+0000 7f3fab400640 1 --2- 172.21.15.6:0/1154962132 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3fac15e120 0x7f3fac070c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:26.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.321+0000 7f3fab400640 1 -- 172.21.15.6:0/1154962132 >> v2:172.21.15.146:3300/0 conn(0x7f3fac0ce670 msgr2=0x7f3fac0ceb00 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-04T09:25:26.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.322+0000 7f3fab400640 1 --2- 172.21.15.6:0/1154962132 >> v2:172.21.15.146:3300/0 conn(0x7f3fac0ce670 0x7f3fac0ceb00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:26.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.322+0000 7f3fab400640 1 -- 172.21.15.6:0/1154962132 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fac0ce150 con 0x7f3fac0b2dd0 2024-03-04T09:25:26.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.322+0000 7f3fab400640 1 -- 172.21.15.6:0/1154962132 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7f3f98001670 con 0x7f3fac0b2dd0 2024-03-04T09:25:26.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.322+0000 7f3fab400640 1 -- 172.21.15.6:0/1154962132 >> v1:172.21.15.6:6789/0 conn(0x7f3fac0b2dd0 legacy=0x7f3fac0711b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:26.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.322+0000 7f3fab400640 1 --2- 172.21.15.6:0/1154962132 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3f90003a20 0x7f3f90003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:26.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.322+0000 7f3fab400640 1 --2- 172.21.15.6:0/1154962132 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3f90004800 0x7f3f90004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:26.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.322+0000 7f3fab400640 1 --2- 172.21.15.6:0/1154962132 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3f900052e0 0x7f3f9000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:26.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.322+0000 7f3fab400640 1 -- 172.21.15.6:0/1154962132 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fac0cdc20 con 0x7f3f900052e0 2024-03-04T09:25:26.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.322+0000 7f3fab400640 1 -- 172.21.15.6:0/1154962132 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fac0cde70 con 0x7f3f90003a20 2024-03-04T09:25:26.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.322+0000 7f3fab400640 1 -- 172.21.15.6:0/1154962132 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f9000c280 con 0x7f3f90004800 2024-03-04T09:25:26.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.322+0000 7f3fb1a00640 1 --2- 172.21.15.6:0/1154962132 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3f900052e0 0x7f3f9000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:26.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.322+0000 7f6e9be00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f6e8c0061f0 0x7f6e8c0066a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:26.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.322+0000 7f6e9be00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6e8c006be0 0x7f6e8c03c120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:26.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.322+0000 7f6e9be00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e8c0e36e0 con 0x7f6e8c0e3300 2024-03-04T09:25:26.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.322+0000 7f6e9be00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e8c007010 con 0x7f6e8c0061f0 2024-03-04T09:25:26.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.322+0000 7f6e9be00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e8c057b40 con 0x7f6e8c006be0 2024-03-04T09:25:26.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.323+0000 7f6e99400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f6e8c0e3300 0x7f6e8c0572f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:60196/0 (socket says 172.21.15.6:60196) 2024-03-04T09:25:26.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.323+0000 7f6e99400640 1 -- 172.21.15.6:0/2654352826 learned_addr learned my addr 172.21.15.6:0/2654352826 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:25:26.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.323+0000 7f3fabe00640 1 --2- 172.21.15.6:0/1154962132 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3f90004800 0x7f3f90004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:26.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.323+0000 7fe058600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe0500b50e0 0x7fe0500ca2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:26.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.323+0000 7fe058600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe0500d0d70 0x7fe050200810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:26.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.323+0000 7f3fb1000640 1 --2- 172.21.15.6:0/1154962132 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3f90003a20 0x7f3f90003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:26.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.323+0000 7f6e99e00640 1 --2- 172.21.15.6:0/2654352826 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6e8c006be0 0x7f6e8c03c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:26.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.323+0000 7f6e98a00640 1 --2- 172.21.15.6:0/2654352826 >> v2:172.21.15.6:3300/0 conn(0x7f6e8c0061f0 0x7f6e8c0066a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:26.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.322+0000 7fa477800640 1 -- 172.21.15.6:0/2243291809 wait complete. 2024-03-04T09:25:26.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.324+0000 7fe058600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe05015e2e0 0x7fe050171db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:26.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.324+0000 7fe058600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0500cfe20 con 0x7fe05015e2e0 2024-03-04T09:25:26.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.324+0000 7fe058600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0500cfb40 con 0x7fe0500b50e0 2024-03-04T09:25:26.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.324+0000 7fe058600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0500d2240 con 0x7fe0500d0d70 2024-03-04T09:25:26.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.324+0000 7f3fabe00640 1 -- 172.21.15.6:0/1154962132 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3f90003a20 msgr2=0x7f3f90003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:26.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.324+0000 7f3fabe00640 1 --2- 172.21.15.6:0/1154962132 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3f90003a20 0x7f3f90003e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:26.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.324+0000 7f3fabe00640 1 -- 172.21.15.6:0/1154962132 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3f900052e0 msgr2=0x7f3f9000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:26.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.324+0000 7f3fabe00640 1 --2- 172.21.15.6:0/1154962132 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3f900052e0 0x7f3f9000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:26.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.324+0000 7f3fabe00640 1 -- 172.21.15.6:0/1154962132 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fac0ce150 con 0x7f3f90004800 2024-03-04T09:25:26.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.324+0000 7fe055200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe0500d0d70 0x7fe050200810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:26.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.324+0000 7fe055c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe0500b50e0 0x7fe0500ca2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:26.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.324+0000 7fe055c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe0500b50e0 0x7fe0500ca2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.6:47178/0 (socket says 172.21.15.6:47178) 2024-03-04T09:25:26.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.324+0000 7fa477800640 1 Processor -- start 2024-03-04T09:25:26.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.324+0000 7fe055c00640 1 -- 172.21.15.6:0/3656045867 learned_addr learned my addr 172.21.15.6:0/3656045867 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:25:26.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.325+0000 7fa477800640 1 -- start start 2024-03-04T09:25:26.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.324+0000 7f6e98a00640 1 -- 172.21.15.6:0/2654352826 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6e8c006be0 msgr2=0x7f6e8c03c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:26.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.324+0000 7f6e98a00640 1 --2- 172.21.15.6:0/2654352826 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6e8c006be0 0x7f6e8c03c120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:26.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.324+0000 7f6e98a00640 1 -- 172.21.15.6:0/2654352826 >> v1:172.21.15.6:6789/0 conn(0x7f6e8c0e3300 legacy=0x7f6e8c0572f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:26.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.324+0000 7f6e98a00640 1 -- 172.21.15.6:0/2654352826 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e8c057e20 con 0x7f6e8c0061f0 2024-03-04T09:25:26.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.325+0000 7f0acbe00640 1 -- 172.21.15.6:0/1619855630 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0ac001b070 con 0x7f0ad40d27d0 2024-03-04T09:25:26.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.326+0000 7fe055200640 1 -- 172.21.15.6:0/3656045867 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe0500b50e0 msgr2=0x7fe0500ca2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:26.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.326+0000 7f6e98a00640 1 --2- 172.21.15.6:0/2654352826 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6e8c0061f0 0x7f6e8c0066a0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f6e800097a0 tx=0x7f6e8002b780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=966b89d982a28b35 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:26.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.326+0000 7fe055200640 1 --2- 172.21.15.6:0/3656045867 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe0500b50e0 0x7fe0500ca2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:26.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.326+0000 7fe055200640 1 -- 172.21.15.6:0/3656045867 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe05015e2e0 msgr2=0x7fe050171db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:26.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.326+0000 7fe055200640 1 --2- 172.21.15.6:0/3656045867 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe05015e2e0 0x7fe050171db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:26.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.326+0000 7fe055200640 1 -- 172.21.15.6:0/3656045867 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe050200d50 con 0x7fe0500d0d70 2024-03-04T09:25:26.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.326+0000 7f6e93e00640 1 -- 172.21.15.6:0/2654352826 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e80037070 con 0x7f6e8c0061f0 2024-03-04T09:25:26.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.326+0000 7fe055200640 1 --2- 172.21.15.6:0/3656045867 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe0500d0d70 0x7fe050200810 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fe03c00c430 tx=0x7fe03c00c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:26.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.326+0000 7fe055c00640 1 --2- 172.21.15.6:0/3656045867 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe0500b50e0 0x7fe0500ca2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T09:25:26.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.326+0000 7f6e93e00640 1 -- 172.21.15.6:0/2654352826 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e80030710 con 0x7f6e8c0061f0 2024-03-04T09:25:26.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.326+0000 7f6e93e00640 1 -- 172.21.15.6:0/2654352826 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e80030b20 con 0x7f6e8c0061f0 2024-03-04T09:25:26.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.327+0000 7f3fabe00640 1 --2- 172.21.15.6:0/1154962132 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3f90004800 0x7f3f90004c80 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f3f9800c290 tx=0x7f3f90005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7443b7b842984456 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:26.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.327+0000 7fe04d600640 1 -- 172.21.15.6:0/3656045867 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe03c017070 con 0x7fe0500d0d70 2024-03-04T09:25:26.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.327+0000 7fa477800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa4700b5060 0x7fa470153d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:26.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.327+0000 7f0acbe00640 1 --2- 172.21.15.6:0/1619855630 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f0ab406b4d0 0x7f0ab406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:26.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.327+0000 7fa477800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa4700b54d0 0x7fa470171870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:26.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.327+0000 7fe058600640 1 -- 172.21.15.6:0/3656045867 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe050200ff0 con 0x7fe0500d0d70 2024-03-04T09:25:26.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.327+0000 7fe058600640 1 -- 172.21.15.6:0/3656045867 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0502015b0 con 0x7fe0500d0d70 2024-03-04T09:25:26.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.327+0000 7fe04d600640 1 -- 172.21.15.6:0/3656045867 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe03c005ce0 con 0x7fe0500d0d70 2024-03-04T09:25:26.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.327+0000 7fe04d600640 1 -- 172.21.15.6:0/3656045867 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe03c0072d0 con 0x7fe0500d0d70 2024-03-04T09:25:26.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.327+0000 7fa477800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa470161b40 0x7fa47006bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:26.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.327+0000 7f6e9be00640 1 -- 172.21.15.6:0/2654352826 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6e8c0061f0 msgr2=0x7f6e8c0066a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:26.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.327+0000 7f6e9be00640 1 --2- 172.21.15.6:0/2654352826 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6e8c0061f0 0x7f6e8c0066a0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f6e800097a0 tx=0x7f6e8002b780 comp rx=0 tx=0).stop 2024-03-04T09:25:26.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.328+0000 7f3fab400640 1 -- 172.21.15.6:0/1154962132 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f98007d80 con 0x7f3f90004800 2024-03-04T09:25:26.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.328+0000 7f6e9be00640 1 -- 172.21.15.6:0/2654352826 shutdown_connections 2024-03-04T09:25:26.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.328+0000 7f6e9be00640 1 --2- 172.21.15.6:0/2654352826 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6e8c006be0 0x7f6e8c03c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:26.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.328+0000 7f6e9be00640 1 --2- 172.21.15.6:0/2654352826 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6e8c0061f0 0x7f6e8c0066a0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:26.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.328+0000 7f6e9be00640 1 -- 172.21.15.6:0/2654352826 >> 172.21.15.6:0/2654352826 conn(0x7f6e8c058ef0 msgr2=0x7f6e8c059300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:26.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.328+0000 7f0acbe00640 1 -- 172.21.15.6:0/1619855630 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0ac00896e0 con 0x7f0ad40d27d0 2024-03-04T09:25:26.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.328+0000 7f0acbe00640 1 --2- 172.21.15.6:0/1619855630 >> [v2:172.21.15.6:6823/1364449401,v1:172.21.15.6:6825/1364449401] conn(0x7f0ab40736f0 0x7f0ab4075b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:26.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.328+0000 7f0acbe00640 1 -- 172.21.15.6:0/1619855630 --> [v2:172.21.15.6:6823/1364449401,v1:172.21.15.6:6825/1364449401] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0ac0015ec0 con 0x7f0ab40736f0 2024-03-04T09:25:26.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.328+0000 7f0acbe00640 1 -- 172.21.15.6:0/1619855630 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0ac0090050 con 0x7f0ad40d27d0 2024-03-04T09:25:26.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.328+0000 7f0ad3e00640 1 --2- 172.21.15.6:0/1619855630 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f0ab406b4d0 0x7f0ab406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:26.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.328+0000 7fa477800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4700b2ff0 con 0x7fa470161b40 2024-03-04T09:25:26.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.328+0000 7fa475800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa470161b40 0x7fa47006bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:26.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.328+0000 7fa474e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa4700b5060 0x7fa470153d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:26.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.328+0000 7fa475800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa470161b40 0x7fa47006bb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45164/0 (socket says 172.21.15.6:45164) 2024-03-04T09:25:26.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.328+0000 7fa475800640 1 -- 172.21.15.6:0/130796589 learned_addr learned my addr 172.21.15.6:0/130796589 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:25:26.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.328+0000 7fa46fe00640 1 --2- 172.21.15.6:0/130796589 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa4700b54d0 0x7fa470171870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:26.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.328+0000 7f6e9be00640 1 -- 172.21.15.6:0/2654352826 shutdown_connections 2024-03-04T09:25:26.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.329+0000 7fa477800640 1 -- 172.21.15.6:0/130796589 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4700cf590 con 0x7fa4700b54d0 2024-03-04T09:25:26.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.328+0000 7f0ad9000640 1 --2- 172.21.15.6:0/1619855630 >> [v2:172.21.15.6:6823/1364449401,v1:172.21.15.6:6825/1364449401] conn(0x7f0ab40736f0 0x7f0ab4075b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:26.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.328+0000 7f3fab400640 1 -- 172.21.15.6:0/1154962132 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f98030070 con 0x7f3f90004800 2024-03-04T09:25:26.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.328+0000 7f3fab400640 1 -- 172.21.15.6:0/1154962132 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f9802cd80 con 0x7f3f90004800 2024-03-04T09:25:26.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.329+0000 7fa477800640 1 -- 172.21.15.6:0/130796589 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4700b3170 con 0x7fa4700b5060 2024-03-04T09:25:26.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.329+0000 7fe058600640 1 -- 172.21.15.6:0/3656045867 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe02c001030 con 0x7fe0500d0d70 2024-03-04T09:25:26.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.330+0000 7f0ad9000640 1 --2- 172.21.15.6:0/1619855630 >> [v2:172.21.15.6:6823/1364449401,v1:172.21.15.6:6825/1364449401] conn(0x7f0ab40736f0 0x7f0ab4075b10 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:26.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.328+0000 7f6e9be00640 1 -- 172.21.15.6:0/2654352826 wait complete. 2024-03-04T09:25:26.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.330+0000 7f3fb3a00640 1 -- 172.21.15.6:0/1154962132 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3f90004800 msgr2=0x7f3f90004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:26.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.330+0000 7f3fb3a00640 1 --2- 172.21.15.6:0/1154962132 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3f90004800 0x7f3f90004c80 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f3f9800c290 tx=0x7f3f90005830 comp rx=0 tx=0).stop 2024-03-04T09:25:26.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.330+0000 7f3fb1a00640 1 -- 172.21.15.6:0/1154962132 reap_dead start 2024-03-04T09:25:26.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.330+0000 7f3fb3a00640 1 -- 172.21.15.6:0/1154962132 shutdown_connections 2024-03-04T09:25:26.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.330+0000 7f3fb3a00640 1 -- 172.21.15.6:0/1154962132 >> 172.21.15.6:0/1154962132 conn(0x7f3fac06afb0 msgr2=0x7f3fac06b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:26.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.330+0000 7f3fb3a00640 1 -- 172.21.15.6:0/1154962132 shutdown_connections 2024-03-04T09:25:26.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.330+0000 7f3fb3a00640 1 -- 172.21.15.6:0/1154962132 wait complete. 2024-03-04T09:25:26.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.331+0000 7fa475800640 1 -- 172.21.15.6:0/130796589 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa4700b5060 msgr2=0x7fa470153d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:26.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.331+0000 7fa475800640 1 --2- 172.21.15.6:0/130796589 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa4700b5060 0x7fa470153d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:26.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.331+0000 7fa475800640 1 -- 172.21.15.6:0/130796589 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa4700b54d0 msgr2=0x7fa470171870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:26.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.331+0000 7fa475800640 1 --2- 172.21.15.6:0/130796589 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa4700b54d0 0x7fa470171870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:26.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.331+0000 7fa475800640 1 -- 172.21.15.6:0/130796589 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4700c7fc0 con 0x7fa470161b40 2024-03-04T09:25:26.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.331+0000 7fa46fe00640 1 --2- 172.21.15.6:0/130796589 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa4700b54d0 0x7fa470171870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T09:25:26.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.331+0000 7fa474e00640 1 --2- 172.21.15.6:0/130796589 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa4700b5060 0x7fa470153d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T09:25:26.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.331+0000 7f0ad3e00640 1 --2- 172.21.15.6:0/1619855630 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f0ab406b4d0 0x7f0ab406d990 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f0acc034000 tx=0x7f0acc008030 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:26.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.332+0000 7f6e9be00640 1 Processor -- start 2024-03-04T09:25:26.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.332+0000 7fe04d600640 1 -- 172.21.15.6:0/3656045867 <== mon.2 v2:172.21.15.146:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe03c01a070 con 0x7fe0500d0d70 2024-03-04T09:25:26.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.332+0000 7f0acbe00640 1 -- 172.21.15.6:0/1619855630 <== osd.0 v2:172.21.15.6:6823/1364449401 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f0ac0015ec0 con 0x7f0ab40736f0 2024-03-04T09:25:26.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.332+0000 7fa475800640 1 --2- 172.21.15.6:0/130796589 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa470161b40 0x7fa47006bb10 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fa46800c760 tx=0x7fa46800cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:26.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.333+0000 7fa46cc00640 1 -- 172.21.15.6:0/130796589 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa468016020 con 0x7fa470161b40 2024-03-04T09:25:26.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.333+0000 7fa46cc00640 1 -- 172.21.15.6:0/130796589 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa468006d50 con 0x7fa470161b40 2024-03-04T09:25:26.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.333+0000 7fa46cc00640 1 -- 172.21.15.6:0/130796589 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa468003bb0 con 0x7fa470161b40 2024-03-04T09:25:26.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.333+0000 7fa477800640 1 -- 172.21.15.6:0/130796589 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4700c82a0 con 0x7fa470161b40 2024-03-04T09:25:26.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.333+0000 7fa477800640 1 -- 172.21.15.6:0/130796589 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4700c8b40 con 0x7fa470161b40 2024-03-04T09:25:26.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.333+0000 7f3fb3a00640 1 Processor -- start 2024-03-04T09:25:26.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.333+0000 7f3fb3a00640 1 -- start start 2024-03-04T09:25:26.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.333+0000 7fe04d600640 1 --2- 172.21.15.6:0/3656045867 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fe03006b4d0 0x7fe03006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:26.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.333+0000 7fe04d600640 1 -- 172.21.15.6:0/3656045867 <== mon.2 v2:172.21.15.146:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe03c08a010 con 0x7fe0500d0d70 2024-03-04T09:25:26.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.334+0000 7fe04d600640 1 --2- 172.21.15.6:0/3656045867 >> [v2:172.21.15.6:6810/1500066681,v1:172.21.15.6:6812/1500066681] conn(0x7fe030073750 0x7fe030075b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:26.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.334+0000 7fe04d600640 1 -- 172.21.15.6:0/3656045867 --> [v2:172.21.15.6:6810/1500066681,v1:172.21.15.6:6812/1500066681] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe03c020e50 con 0x7fe030073750 2024-03-04T09:25:26.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.334+0000 7fe04d600640 1 -- 172.21.15.6:0/3656045867 <== mon.2 v2:172.21.15.146:3301/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe03c08a400 con 0x7fe0500d0d70 2024-03-04T09:25:26.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.334+0000 7f911ee00640 1 Processor -- start 2024-03-04T09:25:26.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.335+0000 7feb7d400640 1 Processor -- start 2024-03-04T09:25:26.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.335+0000 7f3fb3a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3fac2039b0 0x7f3fac154f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:26.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.335+0000 7f3fb3a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3fac156cb0 0x7f3fac157130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:26.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.335+0000 7f911ee00640 1 -- start start 2024-03-04T09:25:26.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.335+0000 7f3fb3a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3fac15fbf0 0x7f3fac160070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:26.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.335+0000 7f3fb3a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fac06df00 con 0x7f3fac15fbf0 2024-03-04T09:25:26.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.335+0000 7fa477800640 1 -- 172.21.15.6:0/130796589 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa4700c86a0 con 0x7fa470161b40 2024-03-04T09:25:26.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.335+0000 7f3fb3a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fac06e3f0 con 0x7f3fac2039b0 2024-03-04T09:25:26.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.335+0000 7f3fb3a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fac0718c0 con 0x7f3fac156cb0 2024-03-04T09:25:26.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.336+0000 7f3fabe00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3fac156cb0 0x7f3fac157130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:26.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.336+0000 7f3fb1000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3fac2039b0 0x7f3fac154f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:26.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.336+0000 7f3fb1000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3fac2039b0 0x7f3fac154f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.6:47208/0 (socket says 172.21.15.6:47208) 2024-03-04T09:25:26.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.336+0000 7f3fb1000640 1 -- 172.21.15.6:0/3804822420 learned_addr learned my addr 172.21.15.6:0/3804822420 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:25:26.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.336+0000 7f3fb1a00640 1 --2- 172.21.15.6:0/3804822420 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3fac15fbf0 0x7f3fac160070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:26.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.336+0000 7fe056600640 1 --2- 172.21.15.6:0/3656045867 >> [v2:172.21.15.6:6810/1500066681,v1:172.21.15.6:6812/1500066681] conn(0x7fe030073750 0x7fe030075b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:26.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.336+0000 7fe055c00640 1 --2- 172.21.15.6:0/3656045867 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fe03006b4d0 0x7fe03006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:26.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.332+0000 7f6e9be00640 1 -- start start 2024-03-04T09:25:26.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.334+0000 7f6e9be00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6e8c006be0 0x7f6e8c0e5d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:26.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.334+0000 7f6e9be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6e8c0e3300 0x7f6e8c04ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:26.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.334+0000 7f6e9be00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6e8c03c760 0x7f6e8c1816f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:26.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.334+0000 7f6e9be00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e8c058be0 con 0x7f6e8c0e3300 2024-03-04T09:25:26.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.334+0000 7f6e9be00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e8c0583e0 con 0x7f6e8c006be0 2024-03-04T09:25:26.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.334+0000 7f6e9be00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e8c04d0b0 con 0x7f6e8c03c760 2024-03-04T09:25:26.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.334+0000 7f6e99400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6e8c006be0 0x7f6e8c0e5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:26.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.334+0000 7f6e99400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6e8c006be0 0x7f6e8c0e5d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.6:47194/0 (socket says 172.21.15.6:47194) 2024-03-04T09:25:26.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.334+0000 7f6e99400640 1 -- 172.21.15.6:0/1665148162 learned_addr learned my addr 172.21.15.6:0/1665148162 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:25:26.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.335+0000 7f6e99e00640 1 --2- 172.21.15.6:0/1665148162 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6e8c03c760 0x7f6e8c1816f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:26.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.335+0000 7f6e99400640 1 -- 172.21.15.6:0/1665148162 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6e8c03c760 msgr2=0x7f6e8c1816f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:26.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.335+0000 7f6e99400640 1 --2- 172.21.15.6:0/1665148162 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6e8c03c760 0x7f6e8c1816f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:26.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.335+0000 7f6e99400640 1 -- 172.21.15.6:0/1665148162 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6e8c0e3300 msgr2=0x7f6e8c04ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:26.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.335+0000 7f6e99400640 1 --2- 172.21.15.6:0/1665148162 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6e8c0e3300 0x7f6e8c04ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:26.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.335+0000 7f6e99400640 1 -- 172.21.15.6:0/1665148162 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e8c04f610 con 0x7f6e8c006be0 2024-03-04T09:25:26.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.337+0000 7f3fb1a00640 1 -- 172.21.15.6:0/3804822420 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3fac156cb0 msgr2=0x7f3fac157130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:26.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.337+0000 7f3fb1a00640 1 --2- 172.21.15.6:0/3804822420 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3fac156cb0 0x7f3fac157130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:26.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.337+0000 7f3fb1a00640 1 -- 172.21.15.6:0/3804822420 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3fac2039b0 msgr2=0x7f3fac154f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:26.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.337+0000 7f3fb1a00640 1 --2- 172.21.15.6:0/3804822420 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3fac2039b0 0x7f3fac154f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:26.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.337+0000 7f6e99400640 1 --2- 172.21.15.6:0/1665148162 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6e8c006be0 0x7f6e8c0e5d10 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f6e8800c4a0 tx=0x7f6e8800c960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:26.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.337+0000 7f3fb1a00640 1 -- 172.21.15.6:0/3804822420 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fac09c9f0 con 0x7f3fac15fbf0 2024-03-04T09:25:26.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.337+0000 7fe056600640 1 --2- 172.21.15.6:0/3656045867 >> [v2:172.21.15.6:6810/1500066681,v1:172.21.15.6:6812/1500066681] conn(0x7fe030073750 0x7fe030075b70 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:26.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.337+0000 7f3fb1000640 1 --2- 172.21.15.6:0/3804822420 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3fac2039b0 0x7f3fac154f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T09:25:26.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.337+0000 7f3fb1a00640 1 --2- 172.21.15.6:0/3804822420 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3fac15fbf0 0x7f3fac160070 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f3fac06c3a0 tx=0x7f3f9c00e7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:26.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.337+0000 7f6e91600640 1 -- 172.21.15.6:0/1665148162 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e88039070 con 0x7f6e8c006be0 2024-03-04T09:25:26.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.337+0000 7f6e91600640 1 -- 172.21.15.6:0/1665148162 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e88005ae0 con 0x7f6e8c006be0 2024-03-04T09:25:26.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.338+0000 7f6e9be00640 1 -- 172.21.15.6:0/1665148162 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e8c04ca60 con 0x7f6e8c006be0 2024-03-04T09:25:26.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.338+0000 7f3fa8c00640 1 -- 172.21.15.6:0/3804822420 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f9c017070 con 0x7f3fac15fbf0 2024-03-04T09:25:26.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.338+0000 7f3fa8c00640 1 -- 172.21.15.6:0/3804822420 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f9c013680 con 0x7f3fac15fbf0 2024-03-04T09:25:26.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.338+0000 7f3fa8c00640 1 -- 172.21.15.6:0/3804822420 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f9c013a70 con 0x7f3fac15fbf0 2024-03-04T09:25:26.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.338+0000 7f3fb3a00640 1 -- 172.21.15.6:0/3804822420 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fac09ccd0 con 0x7f3fac15fbf0 2024-03-04T09:25:26.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.338+0000 7f6e9be00640 1 -- 172.21.15.6:0/1665148162 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e8c04d540 con 0x7f6e8c006be0 2024-03-04T09:25:26.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.338+0000 7f6e91600640 1 -- 172.21.15.6:0/1665148162 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e88007070 con 0x7f6e8c006be0 2024-03-04T09:25:26.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.338+0000 7f3fb3a00640 1 -- 172.21.15.6:0/3804822420 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fac09d290 con 0x7f3fac15fbf0 2024-03-04T09:25:26.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.337+0000 7f911ee00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f9118161b40 0x7f91180cfd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:26.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.337+0000 7f911ee00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f91180d2930 0x7f91180b3740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:26.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.337+0000 7f911ee00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91180cea50 con 0x7f91180b5060 2024-03-04T09:25:26.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.337+0000 7f911ee00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91180cf900 con 0x7f9118161b40 2024-03-04T09:25:26.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.337+0000 7f911ee00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91180cecf0 con 0x7f91180d2930 2024-03-04T09:25:26.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.338+0000 7f911ce00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f91180d2930 0x7f91180b3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:26.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.338+0000 7f9117400640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f91180b5060 0x7f91180d2220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.6:50134/0 (socket says 172.21.15.6:50134) 2024-03-04T09:25:26.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.338+0000 7f9117400640 1 -- 172.21.15.6:0/3489517879 learned_addr learned my addr 172.21.15.6:0/3489517879 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:25:26.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.339+0000 7f911ce00640 1 -- 172.21.15.6:0/3489517879 >> v2:172.21.15.6:3300/0 conn(0x7f9118161b40 msgr2=0x7f91180cfd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:26.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.339+0000 7f911ce00640 1 --2- 172.21.15.6:0/3489517879 >> v2:172.21.15.6:3300/0 conn(0x7f9118161b40 0x7f91180cfd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:26.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.339+0000 7f911ce00640 1 -- 172.21.15.6:0/3489517879 >> v1:172.21.15.146:6789/0 conn(0x7f91180b5060 legacy=0x7f91180d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:26.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.339+0000 7f911ce00640 1 -- 172.21.15.6:0/3489517879 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91180cefd0 con 0x7f91180d2930 2024-03-04T09:25:26.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.339+0000 7fe055c00640 1 --2- 172.21.15.6:0/3656045867 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fe03006b4d0 0x7fe03006d990 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe0440097a0 tx=0x7fe04403c040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:26.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.339+0000 7f3fb3a00640 1 -- 172.21.15.6:0/3804822420 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f3fac09d600 con 0x7f3fac15fbf0 2024-03-04T09:25:26.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.339+0000 7f6e9be00640 1 -- 172.21.15.6:0/1665148162 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f6e8c051fd0 con 0x7f6e8c006be0 2024-03-04T09:25:26.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.340+0000 7f911ce00640 1 --2- 172.21.15.6:0/3489517879 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f91180d2930 0x7f91180b3740 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f91080097a0 tx=0x7f910802b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f4845a5c68834130 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:26.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.339+0000 7fa46cc00640 1 -- 172.21.15.6:0/130796589 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa46800a850 con 0x7fa470161b40 2024-03-04T09:25:26.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.341+0000 7fa46cc00640 1 --2- 172.21.15.6:0/130796589 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa45006b4d0 0x7fa45006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:26.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.341+0000 7fa46cc00640 1 -- 172.21.15.6:0/130796589 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa468089ea0 con 0x7fa470161b40 2024-03-04T09:25:26.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.341+0000 7fa46cc00640 1 --2- 172.21.15.6:0/130796589 >> [v2:172.21.15.6:6808/2494103078,v1:172.21.15.6:6809/2494103078] conn(0x7fa4500736f0 0x7fa450075b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:26.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.341+0000 7fa46cc00640 1 -- 172.21.15.6:0/130796589 --> [v2:172.21.15.6:6808/2494103078,v1:172.21.15.6:6809/2494103078] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa46801eec0 con 0x7fa4500736f0 2024-03-04T09:25:26.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.341+0000 7fa46cc00640 1 -- 172.21.15.6:0/130796589 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa46808c500 con 0x7fa470161b40 2024-03-04T09:25:26.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.337+0000 7feb7d400640 1 -- start start 2024-03-04T09:25:26.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.339+0000 7feb7d400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feb78070ee0 0x7feb78162c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:26.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.339+0000 7feb7d400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7feb78072da0 0x7feb780cf130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:26.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.339+0000 7feb7d400640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb780ce090 con 0x7feb780712c0 2024-03-04T09:25:26.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.339+0000 7feb7d400640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb780ceca0 con 0x7feb78072da0 2024-03-04T09:25:26.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.340+0000 7feb7d400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb780ce370 con 0x7feb78070ee0 2024-03-04T09:25:26.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.340+0000 7feb76800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feb78070ee0 0x7feb78162c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:26.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.340+0000 7feb76800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feb78070ee0 0x7feb78162c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.6:33278/0 (socket says 172.21.15.6:33278) 2024-03-04T09:25:26.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.340+0000 7feb76800640 1 -- 172.21.15.6:0/3683437454 learned_addr learned my addr 172.21.15.6:0/3683437454 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:25:26.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.341+0000 7feb75e00640 1 --2- 172.21.15.6:0/3683437454 >> v2:172.21.15.6:3300/0 conn(0x7feb78072da0 0x7feb780cf130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:26.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.341+0000 7feb76800640 1 -- 172.21.15.6:0/3683437454 >> v2:172.21.15.6:3300/0 conn(0x7feb78072da0 msgr2=0x7feb780cf130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:26.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.341+0000 7feb76800640 1 --2- 172.21.15.6:0/3683437454 >> v2:172.21.15.6:3300/0 conn(0x7feb78072da0 0x7feb780cf130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:26.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.341+0000 7feb76800640 1 -- 172.21.15.6:0/3683437454 >> v1:172.21.15.6:6789/0 conn(0x7feb780712c0 legacy=0x7feb7806e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:26.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.341+0000 7feb76800640 1 -- 172.21.15.6:0/3683437454 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb780ce650 con 0x7feb78070ee0 2024-03-04T09:25:26.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.341+0000 7feb75e00640 1 --2- 172.21.15.6:0/3683437454 >> v2:172.21.15.6:3300/0 conn(0x7feb78072da0 0x7feb780cf130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T09:25:26.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.341+0000 7f9116a00640 1 -- 172.21.15.6:0/3489517879 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9108035020 con 0x7f91180d2930 2024-03-04T09:25:26.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.341+0000 7f9116a00640 1 -- 172.21.15.6:0/3489517879 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9108031810 con 0x7f91180d2930 2024-03-04T09:25:26.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.341+0000 7f9116a00640 1 -- 172.21.15.6:0/3489517879 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9108031be0 con 0x7f91180d2930 2024-03-04T09:25:26.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.342+0000 7fa474e00640 1 --2- 172.21.15.6:0/130796589 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa45006b4d0 0x7fa45006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:26.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.342+0000 7fa46fe00640 1 --2- 172.21.15.6:0/130796589 >> [v2:172.21.15.6:6808/2494103078,v1:172.21.15.6:6809/2494103078] conn(0x7fa4500736f0 0x7fa450075b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:26.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.342+0000 7f911ee00640 1 -- 172.21.15.6:0/3489517879 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f91180d2930 msgr2=0x7f91180b3740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:26.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.342+0000 7f911ee00640 1 --2- 172.21.15.6:0/3489517879 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f91180d2930 0x7f91180b3740 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f91080097a0 tx=0x7f910802b8b0 comp rx=0 tx=0).stop 2024-03-04T09:25:26.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.343+0000 7f3fa8c00640 1 -- 172.21.15.6:0/3804822420 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3f9c01a070 con 0x7f3fac15fbf0 2024-03-04T09:25:26.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.343+0000 7f911ee00640 1 -- 172.21.15.6:0/3489517879 shutdown_connections 2024-03-04T09:25:26.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.343+0000 7f911ee00640 1 --2- 172.21.15.6:0/3489517879 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f91180d2930 0x7f91180b3740 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:26.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.343+0000 7f911ee00640 1 --2- 172.21.15.6:0/3489517879 >> v2:172.21.15.6:3300/0 conn(0x7f9118161b40 0x7f91180cfd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:26.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.343+0000 7f911ee00640 1 -- 172.21.15.6:0/3489517879 >> 172.21.15.6:0/3489517879 conn(0x7f911809d220 msgr2=0x7f91180d3ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:26.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.343+0000 7fe04d600640 1 -- 172.21.15.6:0/3656045867 <== osd.2 v2:172.21.15.6:6810/1500066681 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fe03c020e50 con 0x7fe030073750 2024-03-04T09:25:26.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.343+0000 7fa46fe00640 1 --2- 172.21.15.6:0/130796589 >> [v2:172.21.15.6:6808/2494103078,v1:172.21.15.6:6809/2494103078] conn(0x7fa4500736f0 0x7fa450075b10 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:26.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.343+0000 7f911ee00640 1 -- 172.21.15.6:0/3489517879 shutdown_connections 2024-03-04T09:25:26.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.343+0000 7f911ee00640 1 -- 172.21.15.6:0/3489517879 wait complete. 2024-03-04T09:25:26.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.343+0000 7f8528400640 1 Processor -- start 2024-03-04T09:25:26.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.343+0000 7feb76800640 1 --2- 172.21.15.6:0/3683437454 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feb78070ee0 0x7feb78162c40 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7feb70009e50 tx=0x7feb7002bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b8d0145d6694864a server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:26.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.343+0000 7f8528400640 1 -- start start 2024-03-04T09:25:26.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.344+0000 7fa474e00640 1 --2- 172.21.15.6:0/130796589 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa45006b4d0 0x7fa45006d990 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fa464009890 tx=0x7fa464008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:26.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.344+0000 7feb75400640 1 -- 172.21.15.6:0/3683437454 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb7002cb00 con 0x7feb78070ee0 2024-03-04T09:25:26.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.344+0000 7feb75400640 1 -- 172.21.15.6:0/3683437454 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb7002cca0 con 0x7feb78070ee0 2024-03-04T09:25:26.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.344+0000 7feb75400640 1 -- 172.21.15.6:0/3683437454 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb70034610 con 0x7feb78070ee0 2024-03-04T09:25:26.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.344+0000 7f3fa8c00640 1 --2- 172.21.15.6:0/3804822420 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f3f8c06b4d0 0x7f3f8c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:26.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.344+0000 7fa46cc00640 1 -- 172.21.15.6:0/130796589 <== osd.1 v2:172.21.15.6:6808/2494103078 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fa46801eec0 con 0x7fa4500736f0 2024-03-04T09:25:26.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.344+0000 7f3fa8c00640 1 -- 172.21.15.6:0/3804822420 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3f9c08a180 con 0x7f3fac15fbf0 2024-03-04T09:25:26.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.345+0000 7f3fa8c00640 1 --2- 172.21.15.6:0/3804822420 >> [v2:172.21.15.6:6800/1558806123,v1:172.21.15.6:6801/1558806123] conn(0x7f3f8c073750 0x7f3f8c075b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:26.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.345+0000 7f3fa8c00640 1 -- 172.21.15.6:0/3804822420 --> [v2:172.21.15.6:6800/1558806123,v1:172.21.15.6:6801/1558806123] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3f8c076220 con 0x7f3f8c073750 2024-03-04T09:25:26.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.345+0000 7f6e91600640 1 -- 172.21.15.6:0/1665148162 <== mon.1 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6e8803c070 con 0x7f6e8c006be0 2024-03-04T09:25:26.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.345+0000 7f3fa8c00640 1 -- 172.21.15.6:0/3804822420 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3f9c08a570 con 0x7f3fac15fbf0 2024-03-04T09:25:26.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.345+0000 7f3fabe00640 1 --2- 172.21.15.6:0/3804822420 >> [v2:172.21.15.6:6800/1558806123,v1:172.21.15.6:6801/1558806123] conn(0x7f3f8c073750 0x7f3f8c075b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:26.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.346+0000 7f911ee00640 1 Processor -- start 2024-03-04T09:25:26.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.346+0000 7f3fabe00640 1 --2- 172.21.15.6:0/3804822420 >> [v2:172.21.15.6:6800/1558806123,v1:172.21.15.6:6801/1558806123] conn(0x7f3f8c073750 0x7f3f8c075b70 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:26.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.346+0000 7f911ee00640 1 -- start start 2024-03-04T09:25:26.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.346+0000 7f6e91600640 1 --2- 172.21.15.6:0/1665148162 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f6e7406b550 0x7f6e7406da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:26.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.346+0000 7f3fb1000640 1 --2- 172.21.15.6:0/3804822420 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f3f8c06b4d0 0x7f3f8c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:26.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.346+0000 7feb7d400640 1 -- 172.21.15.6:0/3683437454 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feb78070ee0 msgr2=0x7feb78162c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:26.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.346+0000 7feb7d400640 1 --2- 172.21.15.6:0/3683437454 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feb78070ee0 0x7feb78162c40 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7feb70009e50 tx=0x7feb7002bf60 comp rx=0 tx=0).stop 2024-03-04T09:25:26.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.346+0000 7f8528400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f85200af580 0x7f852015c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:26.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.347+0000 7f8528400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f85200cdc20 0x7f85200b2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:26.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.347+0000 7f8528400640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f852015cc40 con 0x7f85200af1a0 2024-03-04T09:25:26.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.347+0000 7f8528400640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f852015cf20 con 0x7f85200af580 2024-03-04T09:25:26.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.347+0000 7f8528400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f852015d200 con 0x7f85200cdc20 2024-03-04T09:25:26.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.347+0000 7feb7d400640 1 -- 172.21.15.6:0/3683437454 shutdown_connections 2024-03-04T09:25:26.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.347+0000 7feb7d400640 1 --2- 172.21.15.6:0/3683437454 >> v2:172.21.15.6:3300/0 conn(0x7feb78072da0 0x7feb780cf130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:26.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.347+0000 7feb7d400640 1 --2- 172.21.15.6:0/3683437454 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feb78070ee0 0x7feb78162c40 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:26.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.347+0000 7feb7d400640 1 -- 172.21.15.6:0/3683437454 >> 172.21.15.6:0/3683437454 conn(0x7feb7806afb0 msgr2=0x7feb7806b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:26.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.347+0000 7f911ee00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f91180b5060 0x7f91180c7a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:26.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.347+0000 7f911ee00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f91180d2930 0x7f911814a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:26.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.347+0000 7feb7d400640 1 -- 172.21.15.6:0/3683437454 shutdown_connections 2024-03-04T09:25:26.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.347+0000 7f911ee00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9118161b40 0x7f91180cca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:26.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.347+0000 7f911ee00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91180b2ff0 con 0x7f9118161b40 2024-03-04T09:25:26.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.347+0000 7f911ee00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91180cf590 con 0x7f91180d2930 2024-03-04T09:25:26.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.347+0000 7f911ee00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91180b3170 con 0x7f91180b5060 2024-03-04T09:25:26.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.348+0000 7f3fb1000640 1 --2- 172.21.15.6:0/3804822420 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f3f8c06b4d0 0x7f3f8c06d990 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f3fac0af220 tx=0x7f3fa4003cf0 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:26.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.347+0000 7f8525a00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f85200af1a0 0x7f85200b4c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:60212/0 (socket says 172.21.15.6:60212) 2024-03-04T09:25:26.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.347+0000 7f8525a00640 1 -- 172.21.15.6:0/662977465 learned_addr learned my addr 172.21.15.6:0/662977465 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:25:26.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.347+0000 7f8525000640 1 --2- 172.21.15.6:0/662977465 >> v2:172.21.15.146:3300/0 conn(0x7f85200af580 0x7f852015c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:26.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.347+0000 7f8526400640 1 --2- 172.21.15.6:0/662977465 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f85200cdc20 0x7f85200b2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:26.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.348+0000 7f6e91600640 1 -- 172.21.15.6:0/1665148162 <== mon.1 v2:172.21.15.146:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6e880ab5c0 con 0x7f6e8c006be0 2024-03-04T09:25:26.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.348+0000 7f3fa8c00640 1 -- 172.21.15.6:0/3804822420 <== osd.3 v2:172.21.15.6:6800/1558806123 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f3f8c076220 con 0x7f3f8c073750 2024-03-04T09:25:26.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.348+0000 7f851fe00640 1 -- 172.21.15.6:0/662977465 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3546160714 0 0) 0x7f852015cc40 con 0x7f85200af1a0 2024-03-04T09:25:26.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.348+0000 7f851fe00640 1 -- 172.21.15.6:0/662977465 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8508003520 con 0x7f85200af1a0 2024-03-04T09:25:26.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.348+0000 7feb7d400640 1 -- 172.21.15.6:0/3683437454 wait complete. 2024-03-04T09:25:26.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.348+0000 7f9117e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f91180b5060 0x7f91180c7a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:26.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.348+0000 7f911ce00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9118161b40 0x7f91180cca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:26.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.349+0000 7f911ce00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9118161b40 0x7f91180cca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45208/0 (socket says 172.21.15.6:45208) 2024-03-04T09:25:26.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.349+0000 7f911ce00640 1 -- 172.21.15.6:0/71005028 learned_addr learned my addr 172.21.15.6:0/71005028 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:25:26.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.349+0000 7f6e98a00640 1 --2- 172.21.15.6:0/1665148162 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f6e7406b550 0x7f6e7406da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:26.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.349+0000 7f6e91600640 1 --2- 172.21.15.6:0/1665148162 >> [v2:172.21.15.146:6800/1526135780,v1:172.21.15.146:6801/1526135780] conn(0x7f6e74073750 0x7f6e74075b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:26.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.349+0000 7f6e91600640 1 -- 172.21.15.6:0/1665148162 --> [v2:172.21.15.146:6800/1526135780,v1:172.21.15.146:6801/1526135780] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6e8802fe50 con 0x7f6e74073750 2024-03-04T09:25:26.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.349+0000 7f6e91600640 1 -- 172.21.15.6:0/1665148162 <== mon.1 v2:172.21.15.146:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6e880aba50 con 0x7f6e8c006be0 2024-03-04T09:25:26.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.349+0000 7f911ce00640 1 -- 172.21.15.6:0/71005028 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f91180b5060 msgr2=0x7f91180c7a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:26.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.348+0000 7f9117e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f91180b5060 0x7f91180c7a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.6:33298/0 (socket says 172.21.15.6:33298) 2024-03-04T09:25:26.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.349+0000 7f911ce00640 1 --2- 172.21.15.6:0/71005028 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f91180b5060 0x7f91180c7a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:26.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.349+0000 7f911ce00640 1 -- 172.21.15.6:0/71005028 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f91180d2930 msgr2=0x7f911814a310 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-04T09:25:26.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.350+0000 7f6e99e00640 1 --2- 172.21.15.6:0/1665148162 >> [v2:172.21.15.146:6800/1526135780,v1:172.21.15.146:6801/1526135780] conn(0x7f6e74073750 0x7f6e74075b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:26.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.349+0000 7f911ce00640 1 --2- 172.21.15.6:0/71005028 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f91180d2930 0x7f911814a310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:26.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.349+0000 7f911ce00640 1 -- 172.21.15.6:0/71005028 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91180d0b70 con 0x7f9118161b40 2024-03-04T09:25:26.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.349+0000 7f9117e00640 1 --2- 172.21.15.6:0/71005028 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f91180b5060 0x7f91180c7a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:25:26.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.350+0000 7f6e98a00640 1 --2- 172.21.15.6:0/1665148162 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f6e7406b550 0x7f6e7406da10 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f6e80034000 tx=0x7f6e8002f000 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:26.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.350+0000 7feb7d400640 1 Processor -- start 2024-03-04T09:25:26.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.350+0000 7f911ce00640 1 --2- 172.21.15.6:0/71005028 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9118161b40 0x7f91180cca20 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f91080096f0 tx=0x7f91080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:26.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.350+0000 7feb7d400640 1 -- start start 2024-03-04T09:25:26.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.351+0000 7f910fe00640 1 -- 172.21.15.6:0/71005028 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9108041070 con 0x7f9118161b40 2024-03-04T09:25:26.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.351+0000 7f910fe00640 1 -- 172.21.15.6:0/71005028 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f910803ac80 con 0x7f9118161b40 2024-03-04T09:25:26.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.351+0000 7f911ee00640 1 -- 172.21.15.6:0/71005028 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9118149250 con 0x7f9118161b40 2024-03-04T09:25:26.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.352+0000 7f6e99e00640 1 --2- 172.21.15.6:0/1665148162 >> [v2:172.21.15.146:6800/1526135780,v1:172.21.15.146:6801/1526135780] conn(0x7f6e74073750 0x7f6e74075b70 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:26.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.351+0000 7f910fe00640 1 -- 172.21.15.6:0/71005028 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9108035040 con 0x7f9118161b40 2024-03-04T09:25:26.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.351+0000 7f911ee00640 1 -- 172.21.15.6:0/71005028 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91180ca540 con 0x7f9118161b40 2024-03-04T09:25:26.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.352+0000 7feb7d400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feb78070ee0 0x7feb78150380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:26.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.352+0000 7feb7d400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feb780712c0 0x7feb78149800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:26.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.352+0000 7feb7d400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feb78072da0 0x7feb7814a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:26.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.352+0000 7feb7d400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb780cdda0 con 0x7feb780712c0 2024-03-04T09:25:26.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.352+0000 7feb7d400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb780cdc20 con 0x7feb78070ee0 2024-03-04T09:25:26.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.352+0000 7feb7d400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb7806e7d0 con 0x7feb78072da0 2024-03-04T09:25:26.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.353+0000 7feb76800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feb78070ee0 0x7feb78150380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:26.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.353+0000 7feb76800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feb78070ee0 0x7feb78150380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.6:47222/0 (socket says 172.21.15.6:47222) 2024-03-04T09:25:26.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.353+0000 7feb76800640 1 -- 172.21.15.6:0/1476726874 learned_addr learned my addr 172.21.15.6:0/1476726874 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:25:26.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.353+0000 7feb75e00640 1 --2- 172.21.15.6:0/1476726874 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feb780712c0 0x7feb78149800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:26.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.353+0000 7feb77200640 1 --2- 172.21.15.6:0/1476726874 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feb78072da0 0x7feb7814a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:26.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.352+0000 7f911ee00640 1 -- 172.21.15.6:0/71005028 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f911804f3a0 con 0x7f9118161b40 2024-03-04T09:25:26.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.348+0000 7f8525000640 1 -- 172.21.15.6:0/662977465 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f85200cdc20 msgr2=0x7f85200b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:26.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.348+0000 7f8525000640 1 --2- 172.21.15.6:0/662977465 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f85200cdc20 0x7f85200b2dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:26.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.348+0000 7f8525000640 1 -- 172.21.15.6:0/662977465 >> v1:172.21.15.6:6789/0 conn(0x7f85200af1a0 legacy=0x7f85200b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:26.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.348+0000 7f8525000640 1 -- 172.21.15.6:0/662977465 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f852015d4e0 con 0x7f85200af580 2024-03-04T09:25:26.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.354+0000 7feb76800640 1 -- 172.21.15.6:0/1476726874 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feb78072da0 msgr2=0x7feb7814a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:26.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.354+0000 7feb76800640 1 --2- 172.21.15.6:0/1476726874 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feb78072da0 0x7feb7814a2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:26.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.354+0000 7feb76800640 1 -- 172.21.15.6:0/1476726874 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feb780712c0 msgr2=0x7feb78149800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:26.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.354+0000 7f8525000640 1 --2- 172.21.15.6:0/662977465 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f85200af580 0x7f852015c600 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f850c0097a0 tx=0x7f850c02b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e73b302e20c9858 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:26.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.354+0000 7feb76800640 1 --2- 172.21.15.6:0/1476726874 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feb780712c0 0x7feb78149800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:26.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.354+0000 7feb76800640 1 -- 172.21.15.6:0/1476726874 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb7809b390 con 0x7feb78070ee0 2024-03-04T09:25:26.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.354+0000 7feb75e00640 1 --2- 172.21.15.6:0/1476726874 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feb780712c0 0x7feb78149800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T09:25:26.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.355+0000 7feb76800640 1 --2- 172.21.15.6:0/1476726874 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feb78070ee0 0x7feb78150380 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7feb70009f70 tx=0x7feb70004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:26.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.355+0000 7f851fe00640 1 -- 172.21.15.6:0/662977465 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f850c031780 con 0x7f85200af580 2024-03-04T09:25:26.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.355+0000 7f851fe00640 1 -- 172.21.15.6:0/662977465 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f850c037070 con 0x7f85200af580 2024-03-04T09:25:26.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.355+0000 7f851fe00640 1 -- 172.21.15.6:0/662977465 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f850c031b20 con 0x7f85200af580 2024-03-04T09:25:26.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.355+0000 7f6e91600640 1 -- 172.21.15.6:0/1665148162 <== osd.4 v2:172.21.15.146:6800/1526135780 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f6e8802fe50 con 0x7f6e74073750 2024-03-04T09:25:26.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.355+0000 7feb6ea00640 1 -- 172.21.15.6:0/1476726874 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb70043020 con 0x7feb78070ee0 2024-03-04T09:25:26.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.355+0000 7feb6ea00640 1 -- 172.21.15.6:0/1476726874 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb7002ce50 con 0x7feb78070ee0 2024-03-04T09:25:26.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.355+0000 7feb6ea00640 1 -- 172.21.15.6:0/1476726874 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb70009510 con 0x7feb78070ee0 2024-03-04T09:25:26.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.355+0000 7feb7d400640 1 -- 172.21.15.6:0/1476726874 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb7809b670 con 0x7feb78070ee0 2024-03-04T09:25:26.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.356+0000 7f8528400640 1 -- 172.21.15.6:0/662977465 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f85200af580 msgr2=0x7f852015c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:26.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.356+0000 7f8528400640 1 --2- 172.21.15.6:0/662977465 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f85200af580 0x7f852015c600 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f850c0097a0 tx=0x7f850c02b820 comp rx=0 tx=0).stop 2024-03-04T09:25:26.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.356+0000 7f910fe00640 1 -- 172.21.15.6:0/71005028 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f910804c060 con 0x7f9118161b40 2024-03-04T09:25:26.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.357+0000 7feb7d400640 1 -- 172.21.15.6:0/1476726874 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb7809bc30 con 0x7feb78070ee0 2024-03-04T09:25:26.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.358+0000 7f910fe00640 1 --2- 172.21.15.6:0/71005028 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f90f806b520 0x7f90f806d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:26.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.358+0000 7f910fe00640 1 -- 172.21.15.6:0/71005028 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f91080abf60 con 0x7f9118161b40 2024-03-04T09:25:26.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.358+0000 7f910fe00640 1 --2- 172.21.15.6:0/71005028 >> [v2:172.21.15.146:6821/1392897493,v1:172.21.15.146:6823/1392897493] conn(0x7f90f8073740 0x7f90f8075b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:26.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.358+0000 7f910fe00640 1 -- 172.21.15.6:0/71005028 --> [v2:172.21.15.146:6821/1392897493,v1:172.21.15.146:6823/1392897493] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9108031810 con 0x7f90f8073740 2024-03-04T09:25:26.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.358+0000 7f910fe00640 1 -- 172.21.15.6:0/71005028 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f91080ae580 con 0x7f9118161b40 2024-03-04T09:25:26.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.358+0000 7f8528400640 1 -- 172.21.15.6:0/662977465 shutdown_connections 2024-03-04T09:25:26.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.358+0000 7f8528400640 1 --2- 172.21.15.6:0/662977465 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f85200cdc20 0x7f85200b2dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:26.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.358+0000 7f8528400640 1 --2- 172.21.15.6:0/662977465 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f85200af580 0x7f852015c600 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:26.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.358+0000 7f8528400640 1 -- 172.21.15.6:0/662977465 >> 172.21.15.6:0/662977465 conn(0x7f852006afb0 msgr2=0x7f852006b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:26.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.358+0000 7f9117e00640 1 --2- 172.21.15.6:0/71005028 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f90f806b520 0x7f90f806d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:26.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.359+0000 7f9117400640 1 --2- 172.21.15.6:0/71005028 >> [v2:172.21.15.146:6821/1392897493,v1:172.21.15.146:6823/1392897493] conn(0x7f90f8073740 0x7f90f8075b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:26.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.359+0000 7f9117e00640 1 --2- 172.21.15.6:0/71005028 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f90f806b520 0x7f90f806d9e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f910400a870 tx=0x7f9104008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:26.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.359+0000 7feb7d400640 1 -- 172.21.15.6:0/1476726874 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7feb7809c010 con 0x7feb78070ee0 2024-03-04T09:25:26.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.359+0000 7feb6ea00640 1 -- 172.21.15.6:0/1476726874 <== mon.1 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7feb70046020 con 0x7feb78070ee0 2024-03-04T09:25:26.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.361+0000 7f9117400640 1 --2- 172.21.15.6:0/71005028 >> [v2:172.21.15.146:6821/1392897493,v1:172.21.15.146:6823/1392897493] conn(0x7f90f8073740 0x7f90f8075b60 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:26.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.361+0000 7feb6ea00640 1 --2- 172.21.15.6:0/1476726874 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7feb5806b500 0x7feb5806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:26.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.361+0000 7feb6ea00640 1 -- 172.21.15.6:0/1476726874 <== mon.1 v2:172.21.15.146:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7feb700ac730 con 0x7feb78070ee0 2024-03-04T09:25:26.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.361+0000 7feb6ea00640 1 --2- 172.21.15.6:0/1476726874 >> [v2:172.21.15.146:6808/460262507,v1:172.21.15.146:6809/460262507] conn(0x7feb580736a0 0x7feb58075ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:26.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.361+0000 7feb6ea00640 1 -- 172.21.15.6:0/1476726874 --> [v2:172.21.15.146:6808/460262507,v1:172.21.15.146:6809/460262507] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7feb58076170 con 0x7feb580736a0 2024-03-04T09:25:26.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.361+0000 7feb6ea00640 1 -- 172.21.15.6:0/1476726874 <== mon.1 v2:172.21.15.146:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7feb70077310 con 0x7feb78070ee0 2024-03-04T09:25:26.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.362+0000 7feb75e00640 1 --2- 172.21.15.6:0/1476726874 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7feb5806b500 0x7feb5806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:26.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.362+0000 7f8528400640 1 -- 172.21.15.6:0/662977465 shutdown_connections 2024-03-04T09:25:26.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.362+0000 7f8528400640 1 -- 172.21.15.6:0/662977465 wait complete. 2024-03-04T09:25:26.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.362+0000 7feb77200640 1 --2- 172.21.15.6:0/1476726874 >> [v2:172.21.15.146:6808/460262507,v1:172.21.15.146:6809/460262507] conn(0x7feb580736a0 0x7feb58075ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:26.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.363+0000 7feb75e00640 1 --2- 172.21.15.6:0/1476726874 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7feb5806b500 0x7feb5806d9c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7feb7814e1d0 tx=0x7feb64008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:26.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.363+0000 7f910fe00640 1 -- 172.21.15.6:0/71005028 <== osd.6 v2:172.21.15.146:6821/1392897493 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f9108031810 con 0x7f90f8073740 2024-03-04T09:25:26.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.363+0000 7feb77200640 1 --2- 172.21.15.6:0/1476726874 >> [v2:172.21.15.146:6808/460262507,v1:172.21.15.146:6809/460262507] conn(0x7feb580736a0 0x7feb58075ac0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:26.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.364+0000 7f8528400640 1 Processor -- start 2024-03-04T09:25:26.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.365+0000 7f8528400640 1 -- start start 2024-03-04T09:25:26.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.366+0000 7feb6ea00640 1 -- 172.21.15.6:0/1476726874 <== osd.5 v2:172.21.15.146:6808/460262507 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7feb58076170 con 0x7feb580736a0 2024-03-04T09:25:26.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.367+0000 7f0adb000640 1 -- 172.21.15.6:0/1619855630 --> [v2:172.21.15.6:6823/1364449401,v1:172.21.15.6:6825/1364449401] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f0aac002dd0 con 0x7f0ab40736f0 2024-03-04T09:25:26.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.366+0000 7f8528400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f85200af1a0 0x7f852014a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:26.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.366+0000 7f8528400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f85200af580 0x7f85201f7db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:26.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.367+0000 7f8528400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f85200cdc20 0x7f852009d140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:26.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.367+0000 7f8528400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85200b36e0 con 0x7f85200cdc20 2024-03-04T09:25:26.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.367+0000 7f8528400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85200b3560 con 0x7f85200af580 2024-03-04T09:25:26.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.367+0000 7f8528400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8520155e70 con 0x7f85200af1a0 2024-03-04T09:25:26.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.367+0000 7f8526400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f85200cdc20 0x7f852009d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:26.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.367+0000 7f8526400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f85200cdc20 0x7f852009d140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45226/0 (socket says 172.21.15.6:45226) 2024-03-04T09:25:26.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.367+0000 7f8526400640 1 -- 172.21.15.6:0/1107542005 learned_addr learned my addr 172.21.15.6:0/1107542005 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:25:26.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.368+0000 7f8526400640 1 -- 172.21.15.6:0/1107542005 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f85200af1a0 msgr2=0x7f852014a9e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-04T09:25:26.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.368+0000 7f8526400640 1 --2- 172.21.15.6:0/1107542005 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f85200af1a0 0x7f852014a9e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:26.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.368+0000 7f8526400640 1 -- 172.21.15.6:0/1107542005 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f85200af580 msgr2=0x7f85201f7db0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-04T09:25:26.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.368+0000 7f8526400640 1 --2- 172.21.15.6:0/1107542005 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f85200af580 0x7f85201f7db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:26.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.368+0000 7f8526400640 1 -- 172.21.15.6:0/1107542005 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85201f8720 con 0x7f85200cdc20 2024-03-04T09:25:26.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.368+0000 7f8526400640 1 --2- 172.21.15.6:0/1107542005 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f85200cdc20 0x7f852009d140 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f8518007e90 tx=0x7f851800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:26.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.369+0000 7f851d600640 1 -- 172.21.15.6:0/1107542005 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f851800edd0 con 0x7f85200cdc20 2024-03-04T09:25:26.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.369+0000 7f851d600640 1 -- 172.21.15.6:0/1107542005 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f851802a070 con 0x7f85200cdc20 2024-03-04T09:25:26.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.369+0000 7f851d600640 1 -- 172.21.15.6:0/1107542005 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8518025670 con 0x7f85200cdc20 2024-03-04T09:25:26.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.370+0000 7f0acbe00640 1 -- 172.21.15.6:0/1619855630 <== osd.0 v2:172.21.15.6:6823/1364449401 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f0aac002dd0 con 0x7f0ab40736f0 2024-03-04T09:25:26.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.370+0000 7f8528400640 1 -- 172.21.15.6:0/1107542005 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85201f8a00 con 0x7f85200cdc20 2024-03-04T09:25:26.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.370+0000 7f8528400640 1 -- 172.21.15.6:0/1107542005 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f852009cc20 con 0x7f85200cdc20 2024-03-04T09:25:26.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.371+0000 7f8528400640 1 -- 172.21.15.6:0/1107542005 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f852004f3a0 con 0x7f85200cdc20 2024-03-04T09:25:26.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.373+0000 7f851d600640 1 -- 172.21.15.6:0/1107542005 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8518025810 con 0x7f85200cdc20 2024-03-04T09:25:26.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.373+0000 7fa477800640 1 -- 172.21.15.6:0/130796589 --> [v2:172.21.15.6:6808/2494103078,v1:172.21.15.6:6809/2494103078] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fa47004f3a0 con 0x7fa4500736f0 2024-03-04T09:25:26.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.374+0000 7f0adb000640 1 -- 172.21.15.6:0/1619855630 >> [v2:172.21.15.6:6823/1364449401,v1:172.21.15.6:6825/1364449401] conn(0x7f0ab40736f0 msgr2=0x7f0ab4075b10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:26.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.374+0000 7f0adb000640 1 --2- 172.21.15.6:0/1619855630 >> [v2:172.21.15.6:6823/1364449401,v1:172.21.15.6:6825/1364449401] conn(0x7f0ab40736f0 0x7f0ab4075b10 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:26.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.374+0000 7fa46cc00640 1 -- 172.21.15.6:0/130796589 <== osd.1 v2:172.21.15.6:6808/2494103078 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fa47004f3a0 con 0x7fa4500736f0 2024-03-04T09:25:26.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.374+0000 7f0adb000640 1 -- 172.21.15.6:0/1619855630 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f0ab406b4d0 msgr2=0x7f0ab406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:26.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.374+0000 7f0adb000640 1 --2- 172.21.15.6:0/1619855630 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f0ab406b4d0 0x7f0ab406d990 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f0acc034000 tx=0x7f0acc008030 comp rx=0 tx=0).stop 2024-03-04T09:25:26.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.374+0000 7f0adb000640 1 -- 172.21.15.6:0/1619855630 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0ad40d27d0 msgr2=0x7f0ad40c8430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:26.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.374+0000 7f0adb000640 1 --2- 172.21.15.6:0/1619855630 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0ad40d27d0 0x7f0ad40c8430 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f0ac000c5f0 tx=0x7f0ac000cab0 comp rx=0 tx=0).stop 2024-03-04T09:25:26.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.374+0000 7f0ad9000640 1 -- 172.21.15.6:0/1619855630 reap_dead start 2024-03-04T09:25:26.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.374+0000 7f0adb000640 1 -- 172.21.15.6:0/1619855630 shutdown_connections 2024-03-04T09:25:26.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.374+0000 7f0adb000640 1 -- 172.21.15.6:0/1619855630 >> 172.21.15.6:0/1619855630 conn(0x7f0ad40d3ef0 msgr2=0x7f0ad40d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:26.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.375+0000 7f851d600640 1 --2- 172.21.15.6:0/1107542005 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f850006b520 0x7f850006d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:26.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.375+0000 7fa477800640 1 -- 172.21.15.6:0/130796589 >> [v2:172.21.15.6:6808/2494103078,v1:172.21.15.6:6809/2494103078] conn(0x7fa4500736f0 msgr2=0x7fa450075b10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:26.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.375+0000 7fa477800640 1 --2- 172.21.15.6:0/130796589 >> [v2:172.21.15.6:6808/2494103078,v1:172.21.15.6:6809/2494103078] conn(0x7fa4500736f0 0x7fa450075b10 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:26.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.375+0000 7f851d600640 1 -- 172.21.15.6:0/1107542005 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f851809b860 con 0x7f85200cdc20 2024-03-04T09:25:26.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.375+0000 7fa477800640 1 -- 172.21.15.6:0/130796589 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa45006b4d0 msgr2=0x7fa45006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:26.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.375+0000 7fa477800640 1 --2- 172.21.15.6:0/130796589 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa45006b4d0 0x7fa45006d990 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fa464009890 tx=0x7fa464008040 comp rx=0 tx=0).stop 2024-03-04T09:25:26.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.375+0000 7fa477800640 1 -- 172.21.15.6:0/130796589 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa470161b40 msgr2=0x7fa47006bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:26.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.375+0000 7fa477800640 1 --2- 172.21.15.6:0/130796589 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa470161b40 0x7fa47006bb10 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fa46800c760 tx=0x7fa46800cc20 comp rx=0 tx=0).stop 2024-03-04T09:25:26.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.375+0000 7f851d600640 1 --2- 172.21.15.6:0/1107542005 >> [v2:172.21.15.146:6816/3268323156,v1:172.21.15.146:6817/3268323156] conn(0x7f8500073740 0x7f8500075b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:26.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.375+0000 7f851d600640 1 -- 172.21.15.6:0/1107542005 --> [v2:172.21.15.146:6816/3268323156,v1:172.21.15.146:6817/3268323156] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8518029ec0 con 0x7f8500073740 2024-03-04T09:25:26.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.375+0000 7f851d600640 1 -- 172.21.15.6:0/1107542005 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f85180a2050 con 0x7f85200cdc20 2024-03-04T09:25:26.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.375+0000 7f8525a00640 1 --2- 172.21.15.6:0/1107542005 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f850006b520 0x7f850006d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:26.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.375+0000 7fa475800640 1 -- 172.21.15.6:0/130796589 reap_dead start 2024-03-04T09:25:26.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.376+0000 7fa477800640 1 -- 172.21.15.6:0/130796589 shutdown_connections 2024-03-04T09:25:26.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.376+0000 7fa477800640 1 -- 172.21.15.6:0/130796589 >> 172.21.15.6:0/130796589 conn(0x7fa47009d220 msgr2=0x7fa4700b1750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:26.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.376+0000 7f0adb000640 1 -- 172.21.15.6:0/1619855630 shutdown_connections 2024-03-04T09:25:26.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.376+0000 7fa477800640 1 -- 172.21.15.6:0/130796589 shutdown_connections 2024-03-04T09:25:26.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.376+0000 7f0adb000640 1 -- 172.21.15.6:0/1619855630 wait complete. 2024-03-04T09:25:26.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.376+0000 7f8525000640 1 --2- 172.21.15.6:0/1107542005 >> [v2:172.21.15.146:6816/3268323156,v1:172.21.15.146:6817/3268323156] conn(0x7f8500073740 0x7f8500075b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:26.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.376+0000 7fa477800640 1 -- 172.21.15.6:0/130796589 wait complete. 2024-03-04T09:25:26.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.376+0000 7f8525a00640 1 --2- 172.21.15.6:0/1107542005 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f850006b520 0x7f850006d9e0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f852015c8d0 tx=0x7f85140078f0 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:26.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.379+0000 7fe058600640 1 -- 172.21.15.6:0/3656045867 --> [v2:172.21.15.6:6810/1500066681,v1:172.21.15.6:6812/1500066681] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fe02c002dd0 con 0x7fe030073750 2024-03-04T09:25:26.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.379+0000 7f8525000640 1 --2- 172.21.15.6:0/1107542005 >> [v2:172.21.15.146:6816/3268323156,v1:172.21.15.146:6817/3268323156] conn(0x7f8500073740 0x7f8500075b60 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:26.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.380+0000 7fe04d600640 1 -- 172.21.15.6:0/3656045867 <== osd.2 v2:172.21.15.6:6810/1500066681 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fe02c002dd0 con 0x7fe030073750 2024-03-04T09:25:26.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.382+0000 7fe058600640 1 -- 172.21.15.6:0/3656045867 >> [v2:172.21.15.6:6810/1500066681,v1:172.21.15.6:6812/1500066681] conn(0x7fe030073750 msgr2=0x7fe030075b70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:26.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.382+0000 7fe058600640 1 --2- 172.21.15.6:0/3656045867 >> [v2:172.21.15.6:6810/1500066681,v1:172.21.15.6:6812/1500066681] conn(0x7fe030073750 0x7fe030075b70 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:26.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.382+0000 7fe058600640 1 -- 172.21.15.6:0/3656045867 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fe03006b4d0 msgr2=0x7fe03006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:26.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.382+0000 7fe058600640 1 --2- 172.21.15.6:0/3656045867 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fe03006b4d0 0x7fe03006d990 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe0440097a0 tx=0x7fe04403c040 comp rx=0 tx=0).stop 2024-03-04T09:25:26.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.382+0000 7fe058600640 1 -- 172.21.15.6:0/3656045867 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe0500d0d70 msgr2=0x7fe050200810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:26.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.382+0000 7fe058600640 1 --2- 172.21.15.6:0/3656045867 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe0500d0d70 0x7fe050200810 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fe03c00c430 tx=0x7fe03c00c8f0 comp rx=0 tx=0).stop 2024-03-04T09:25:26.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.384+0000 7fe056600640 1 -- 172.21.15.6:0/3656045867 reap_dead start 2024-03-04T09:25:26.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.384+0000 7fe058600640 1 -- 172.21.15.6:0/3656045867 shutdown_connections 2024-03-04T09:25:26.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.384+0000 7f851d600640 1 -- 172.21.15.6:0/1107542005 <== osd.7 v2:172.21.15.146:6816/3268323156 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f8518029ec0 con 0x7f8500073740 2024-03-04T09:25:26.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.384+0000 7fe058600640 1 -- 172.21.15.6:0/3656045867 >> 172.21.15.6:0/3656045867 conn(0x7fe0500d3f60 msgr2=0x7fe0500d1b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:26.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.384+0000 7fe058600640 1 -- 172.21.15.6:0/3656045867 shutdown_connections 2024-03-04T09:25:26.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.384+0000 7fe058600640 1 -- 172.21.15.6:0/3656045867 wait complete. 2024-03-04T09:25:26.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.388+0000 7f6e9be00640 1 -- 172.21.15.6:0/1665148162 --> [v2:172.21.15.146:6800/1526135780,v1:172.21.15.146:6801/1526135780] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f6e8c0069d0 con 0x7f6e74073750 2024-03-04T09:25:26.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.388+0000 7f3fb3a00640 1 -- 172.21.15.6:0/3804822420 --> [v2:172.21.15.6:6800/1558806123,v1:172.21.15.6:6801/1558806123] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f3fac04f3a0 con 0x7f3f8c073750 2024-03-04T09:25:26.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.389+0000 7f3fa8c00640 1 -- 172.21.15.6:0/3804822420 <== osd.3 v2:172.21.15.6:6800/1558806123 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f3fac04f3a0 con 0x7f3f8c073750 2024-03-04T09:25:26.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.390+0000 7f3fb3a00640 1 -- 172.21.15.6:0/3804822420 >> [v2:172.21.15.6:6800/1558806123,v1:172.21.15.6:6801/1558806123] conn(0x7f3f8c073750 msgr2=0x7f3f8c075b70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:26.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.390+0000 7f3fb3a00640 1 --2- 172.21.15.6:0/3804822420 >> [v2:172.21.15.6:6800/1558806123,v1:172.21.15.6:6801/1558806123] conn(0x7f3f8c073750 0x7f3f8c075b70 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:26.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.391+0000 7f3fb3a00640 1 -- 172.21.15.6:0/3804822420 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f3f8c06b4d0 msgr2=0x7f3f8c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:26.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.391+0000 7f3fb3a00640 1 --2- 172.21.15.6:0/3804822420 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f3f8c06b4d0 0x7f3f8c06d990 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f3fac0af220 tx=0x7f3fa4003cf0 comp rx=0 tx=0).stop 2024-03-04T09:25:26.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.391+0000 7f3fb3a00640 1 -- 172.21.15.6:0/3804822420 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3fac15fbf0 msgr2=0x7f3fac160070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:26.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.391+0000 7f3fb3a00640 1 --2- 172.21.15.6:0/3804822420 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3fac15fbf0 0x7f3fac160070 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f3fac06c3a0 tx=0x7f3f9c00e7d0 comp rx=0 tx=0).stop 2024-03-04T09:25:26.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.391+0000 7f3fb1a00640 1 -- 172.21.15.6:0/3804822420 reap_dead start 2024-03-04T09:25:26.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.391+0000 7f3fb3a00640 1 -- 172.21.15.6:0/3804822420 shutdown_connections 2024-03-04T09:25:26.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.392+0000 7f3fb3a00640 1 -- 172.21.15.6:0/3804822420 >> 172.21.15.6:0/3804822420 conn(0x7f3fac06afb0 msgr2=0x7f3fac0ae640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:26.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.390+0000 7f6e91600640 1 -- 172.21.15.6:0/1665148162 <== osd.4 v2:172.21.15.146:6800/1526135780 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f6e8c0069d0 con 0x7f6e74073750 2024-03-04T09:25:26.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.391+0000 7f6e9be00640 1 -- 172.21.15.6:0/1665148162 >> [v2:172.21.15.146:6800/1526135780,v1:172.21.15.146:6801/1526135780] conn(0x7f6e74073750 msgr2=0x7f6e74075b70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:26.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.391+0000 7f6e9be00640 1 --2- 172.21.15.6:0/1665148162 >> [v2:172.21.15.146:6800/1526135780,v1:172.21.15.146:6801/1526135780] conn(0x7f6e74073750 0x7f6e74075b70 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:26.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.391+0000 7f6e9be00640 1 -- 172.21.15.6:0/1665148162 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f6e7406b550 msgr2=0x7f6e7406da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:26.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.391+0000 7f6e9be00640 1 --2- 172.21.15.6:0/1665148162 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f6e7406b550 0x7f6e7406da10 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f6e80034000 tx=0x7f6e8002f000 comp rx=0 tx=0).stop 2024-03-04T09:25:26.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.392+0000 7f3fb3a00640 1 -- 172.21.15.6:0/3804822420 shutdown_connections 2024-03-04T09:25:26.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.393+0000 7f3fb3a00640 1 -- 172.21.15.6:0/3804822420 wait complete. 2024-03-04T09:25:26.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.391+0000 7f6e9be00640 1 -- 172.21.15.6:0/1665148162 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6e8c006be0 msgr2=0x7f6e8c0e5d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:26.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.391+0000 7f6e9be00640 1 --2- 172.21.15.6:0/1665148162 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6e8c006be0 0x7f6e8c0e5d10 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f6e8800c4a0 tx=0x7f6e8800c960 comp rx=0 tx=0).stop 2024-03-04T09:25:26.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.392+0000 7f6e99e00640 1 -- 172.21.15.6:0/1665148162 reap_dead start 2024-03-04T09:25:26.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.392+0000 7f6e9be00640 1 -- 172.21.15.6:0/1665148162 shutdown_connections 2024-03-04T09:25:26.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.392+0000 7f6e9be00640 1 -- 172.21.15.6:0/1665148162 >> 172.21.15.6:0/1665148162 conn(0x7f6e8c058ef0 msgr2=0x7f6e8c054120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:26.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.392+0000 7f6e9be00640 1 -- 172.21.15.6:0/1665148162 shutdown_connections 2024-03-04T09:25:26.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.392+0000 7f6e9be00640 1 -- 172.21.15.6:0/1665148162 wait complete. 2024-03-04T09:25:26.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.394+0000 7feb7d400640 1 -- 172.21.15.6:0/1476726874 --> [v2:172.21.15.146:6808/460262507,v1:172.21.15.146:6809/460262507] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7feb7804f3a0 con 0x7feb580736a0 2024-03-04T09:25:26.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.395+0000 7feb6ea00640 1 -- 172.21.15.6:0/1476726874 <== osd.5 v2:172.21.15.146:6808/460262507 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7feb7804f3a0 con 0x7feb580736a0 2024-03-04T09:25:26.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.398+0000 7feb7d400640 1 -- 172.21.15.6:0/1476726874 >> [v2:172.21.15.146:6808/460262507,v1:172.21.15.146:6809/460262507] conn(0x7feb580736a0 msgr2=0x7feb58075ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:26.416 INFO:teuthology.orchestra.run.smithi006.stdout:55834574855 2024-03-04T09:25:26.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.398+0000 7feb7d400640 1 --2- 172.21.15.6:0/1476726874 >> [v2:172.21.15.146:6808/460262507,v1:172.21.15.146:6809/460262507] conn(0x7feb580736a0 0x7feb58075ac0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:26.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.398+0000 7feb7d400640 1 -- 172.21.15.6:0/1476726874 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7feb5806b500 msgr2=0x7feb5806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:26.417 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-03-04T09:25:26.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.398+0000 7feb7d400640 1 --2- 172.21.15.6:0/1476726874 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7feb5806b500 0x7feb5806d9c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7feb7814e1d0 tx=0x7feb64008040 comp rx=0 tx=0).stop 2024-03-04T09:25:26.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.399+0000 7feb7d400640 1 -- 172.21.15.6:0/1476726874 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feb78070ee0 msgr2=0x7feb78150380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:26.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.399+0000 7feb7d400640 1 --2- 172.21.15.6:0/1476726874 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feb78070ee0 0x7feb78150380 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7feb70009f70 tx=0x7feb70004a40 comp rx=0 tx=0).stop 2024-03-04T09:25:26.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.399+0000 7feb77200640 1 -- 172.21.15.6:0/1476726874 reap_dead start 2024-03-04T09:25:26.419 INFO:teuthology.orchestra.run.smithi006.stdout:55834574854 2024-03-04T09:25:26.419 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-03-04T09:25:26.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.401+0000 7feb7d400640 1 -- 172.21.15.6:0/1476726874 shutdown_connections 2024-03-04T09:25:26.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.401+0000 7feb7d400640 1 -- 172.21.15.6:0/1476726874 >> 172.21.15.6:0/1476726874 conn(0x7feb7806afb0 msgr2=0x7feb7806b490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:26.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.401+0000 7feb7d400640 1 -- 172.21.15.6:0/1476726874 shutdown_connections 2024-03-04T09:25:26.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.402+0000 7feb7d400640 1 -- 172.21.15.6:0/1476726874 wait complete. 2024-03-04T09:25:26.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.403+0000 7f911ee00640 1 -- 172.21.15.6:0/71005028 --> [v2:172.21.15.146:6821/1392897493,v1:172.21.15.146:6823/1392897493] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f91180b9d80 con 0x7f90f8073740 2024-03-04T09:25:26.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.404+0000 7f910fe00640 1 -- 172.21.15.6:0/71005028 <== osd.6 v2:172.21.15.146:6821/1392897493 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f91180b9d80 con 0x7f90f8073740 2024-03-04T09:25:26.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.405+0000 7f911ee00640 1 -- 172.21.15.6:0/71005028 >> [v2:172.21.15.146:6821/1392897493,v1:172.21.15.146:6823/1392897493] conn(0x7f90f8073740 msgr2=0x7f90f8075b60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:26.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.405+0000 7f911ee00640 1 --2- 172.21.15.6:0/71005028 >> [v2:172.21.15.146:6821/1392897493,v1:172.21.15.146:6823/1392897493] conn(0x7f90f8073740 0x7f90f8075b60 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:26.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.405+0000 7f911ee00640 1 -- 172.21.15.6:0/71005028 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f90f806b520 msgr2=0x7f90f806d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:26.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.405+0000 7f911ee00640 1 --2- 172.21.15.6:0/71005028 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f90f806b520 0x7f90f806d9e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f910400a870 tx=0x7f9104008040 comp rx=0 tx=0).stop 2024-03-04T09:25:26.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.406+0000 7f911ee00640 1 -- 172.21.15.6:0/71005028 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9118161b40 msgr2=0x7f91180cca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:26.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.406+0000 7f911ee00640 1 --2- 172.21.15.6:0/71005028 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9118161b40 0x7f91180cca20 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f91080096f0 tx=0x7f91080046c0 comp rx=0 tx=0).stop 2024-03-04T09:25:26.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.406+0000 7f911ce00640 1 -- 172.21.15.6:0/71005028 reap_dead start 2024-03-04T09:25:26.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.406+0000 7f911ee00640 1 -- 172.21.15.6:0/71005028 shutdown_connections 2024-03-04T09:25:26.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.406+0000 7f911ee00640 1 -- 172.21.15.6:0/71005028 >> 172.21.15.6:0/71005028 conn(0x7f911809d220 msgr2=0x7f91180b0b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:26.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.406+0000 7f911ee00640 1 -- 172.21.15.6:0/71005028 shutdown_connections 2024-03-04T09:25:26.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.406+0000 7f911ee00640 1 -- 172.21.15.6:0/71005028 wait complete. 2024-03-04T09:25:26.423 INFO:teuthology.orchestra.run.smithi006.stdout:55834574854 2024-03-04T09:25:26.423 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-03-04T09:25:26.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.413+0000 7f8528400640 1 -- 172.21.15.6:0/1107542005 --> [v2:172.21.15.146:6816/3268323156,v1:172.21.15.146:6817/3268323156] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f85200ce280 con 0x7f8500073740 2024-03-04T09:25:26.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.414+0000 7f851d600640 1 -- 172.21.15.6:0/1107542005 <== osd.7 v2:172.21.15.146:6816/3268323156 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f85200ce280 con 0x7f8500073740 2024-03-04T09:25:26.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.414+0000 7f8528400640 1 -- 172.21.15.6:0/1107542005 >> [v2:172.21.15.146:6816/3268323156,v1:172.21.15.146:6817/3268323156] conn(0x7f8500073740 msgr2=0x7f8500075b60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:26.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.415+0000 7f8528400640 1 --2- 172.21.15.6:0/1107542005 >> [v2:172.21.15.146:6816/3268323156,v1:172.21.15.146:6817/3268323156] conn(0x7f8500073740 0x7f8500075b60 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:26.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.415+0000 7f8528400640 1 -- 172.21.15.6:0/1107542005 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f850006b520 msgr2=0x7f850006d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:26.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.415+0000 7f8528400640 1 --2- 172.21.15.6:0/1107542005 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f850006b520 0x7f850006d9e0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f852015c8d0 tx=0x7f85140078f0 comp rx=0 tx=0).stop 2024-03-04T09:25:26.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.415+0000 7f8528400640 1 -- 172.21.15.6:0/1107542005 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f85200cdc20 msgr2=0x7f852009d140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:26.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.415+0000 7f8528400640 1 --2- 172.21.15.6:0/1107542005 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f85200cdc20 0x7f852009d140 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f8518007e90 tx=0x7f851800e4b0 comp rx=0 tx=0).stop 2024-03-04T09:25:26.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.415+0000 7f8526400640 1 -- 172.21.15.6:0/1107542005 reap_dead start 2024-03-04T09:25:26.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.415+0000 7f8528400640 1 -- 172.21.15.6:0/1107542005 shutdown_connections 2024-03-04T09:25:26.426 INFO:teuthology.orchestra.run.smithi006.stdout:55834574853 2024-03-04T09:25:26.426 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-03-04T09:25:26.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.415+0000 7f8528400640 1 -- 172.21.15.6:0/1107542005 >> 172.21.15.6:0/1107542005 conn(0x7f852006afb0 msgr2=0x7f852006b3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:26.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.415+0000 7f8528400640 1 -- 172.21.15.6:0/1107542005 shutdown_connections 2024-03-04T09:25:26.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.415+0000 7f8528400640 1 -- 172.21.15.6:0/1107542005 wait complete. 2024-03-04T09:25:26.427 INFO:teuthology.orchestra.run.smithi006.stdout:55834574855 2024-03-04T09:25:26.428 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-03-04T09:25:26.429 INFO:teuthology.orchestra.run.smithi006.stdout:55834574856 2024-03-04T09:25:26.429 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-03-04T09:25:26.430 INFO:teuthology.orchestra.run.smithi006.stdout:55834574854 2024-03-04T09:25:26.430 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-03-04T09:25:26.431 INFO:teuthology.orchestra.run.smithi006.stdout:55834574854 2024-03-04T09:25:26.431 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-03-04T09:25:26.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.725+0000 7f2419000640 1 Processor -- start 2024-03-04T09:25:26.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.725+0000 7f2419000640 1 -- start start 2024-03-04T09:25:26.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.727+0000 7f2419000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f240c053c40 0x7f240c007660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:26.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.727+0000 7f2419000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f240c0589c0 0x7f240c03bd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:26.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.727+0000 7f2419000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f240c0e79d0 con 0x7f240c0e75f0 2024-03-04T09:25:26.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.727+0000 7f2419000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f240c007ba0 con 0x7f240c0589c0 2024-03-04T09:25:26.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.727+0000 7f2419000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f240c0065c0 con 0x7f240c053c40 2024-03-04T09:25:26.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.728+0000 7f2412800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f240c0589c0 0x7f240c03bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:26.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.728+0000 7f2411e00640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f240c0e75f0 0x7f240c053530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.6:50146/0 (socket says 172.21.15.6:50146) 2024-03-04T09:25:26.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.728+0000 7f2411e00640 1 -- 172.21.15.6:0/492306916 learned_addr learned my addr 172.21.15.6:0/492306916 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:25:26.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.729+0000 7faaf7000640 1 Processor -- start 2024-03-04T09:25:26.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.729+0000 7f2411400640 1 --2- 172.21.15.6:0/492306916 >> v2:172.21.15.146:3300/0 conn(0x7f240c053c40 0x7f240c007660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:26.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.729+0000 7f2410a00640 1 -- 172.21.15.6:0/492306916 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4115584567 0 0) 0x7f240c0e79d0 con 0x7f240c0e75f0 2024-03-04T09:25:26.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.730+0000 7f2410a00640 1 -- 172.21.15.6:0/492306916 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23f8003520 con 0x7f240c0e75f0 2024-03-04T09:25:26.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.731+0000 7faaf7000640 1 -- start start 2024-03-04T09:25:26.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.731+0000 7f00a7e00640 1 Processor -- start 2024-03-04T09:25:26.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.731+0000 7f2412800640 1 -- 172.21.15.6:0/492306916 >> v2:172.21.15.146:3300/0 conn(0x7f240c053c40 msgr2=0x7f240c007660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:26.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.731+0000 7f2412800640 1 --2- 172.21.15.6:0/492306916 >> v2:172.21.15.146:3300/0 conn(0x7f240c053c40 0x7f240c007660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:26.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.731+0000 7f2412800640 1 -- 172.21.15.6:0/492306916 >> v1:172.21.15.146:6789/0 conn(0x7f240c0e75f0 legacy=0x7f240c053530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:26.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.731+0000 7f2412800640 1 -- 172.21.15.6:0/492306916 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f240c0068a0 con 0x7f240c0589c0 2024-03-04T09:25:26.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.732+0000 7f2412800640 1 --2- 172.21.15.6:0/492306916 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f240c0589c0 0x7f240c03bd30 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f24140505a0 tx=0x7f24140a6d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43f7cd4573ad32e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:26.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.731+0000 7f00a7e00640 1 -- start start 2024-03-04T09:25:26.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.732+0000 7faaf7000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7faaf015e0b0 0x7faaf00cf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:26.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.732+0000 7faaf7000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faaf00d0e50 0x7faaf00b44d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:26.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.733+0000 7f2410a00640 1 -- 172.21.15.6:0/492306916 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24140b4070 con 0x7f240c0589c0 2024-03-04T09:25:26.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.732+0000 7faaf7000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faaf00b3710 con 0x7faaf00d4a10 2024-03-04T09:25:26.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.732+0000 7faaf7000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faaf00b09e0 con 0x7faaf015e0b0 2024-03-04T09:25:26.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.733+0000 7f2410a00640 1 -- 172.21.15.6:0/492306916 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24140a7960 con 0x7f240c0589c0 2024-03-04T09:25:26.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.733+0000 7f2410a00640 1 -- 172.21.15.6:0/492306916 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24140a7d50 con 0x7f240c0589c0 2024-03-04T09:25:26.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.732+0000 7faaf7000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faaf00b39f0 con 0x7faaf00d0e50 2024-03-04T09:25:26.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.733+0000 7faaf5c00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7faaf015e0b0 0x7faaf00cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:26.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.733+0000 7faaf5c00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7faaf015e0b0 0x7faaf00cf310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.6:47230/0 (socket says 172.21.15.6:47230) 2024-03-04T09:25:26.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.733+0000 7faaf5c00640 1 -- 172.21.15.6:0/773472028 learned_addr learned my addr 172.21.15.6:0/773472028 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:25:26.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.734+0000 7f00a7e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f00a00e3300 0x7f00a00572f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:26.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.734+0000 7f00a7e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f00a0053750 0x7f00a0053bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:26.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.734+0000 7f00a7e00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00a00e36e0 con 0x7f00a0054110 2024-03-04T09:25:26.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.734+0000 7f00a7e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00a0054560 con 0x7f00a0053750 2024-03-04T09:25:26.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.734+0000 7f00a7e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00a0057b40 con 0x7f00a00e3300 2024-03-04T09:25:26.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.734+0000 7f2419000640 1 -- 172.21.15.6:0/492306916 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f240c0589c0 msgr2=0x7f240c03bd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:26.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.734+0000 7f4a44a00640 1 Processor -- start 2024-03-04T09:25:26.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.734+0000 7faaf5c00640 1 -- 172.21.15.6:0/773472028 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faaf00d0e50 msgr2=0x7faaf00b44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:26.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.734+0000 7faaf5c00640 1 --2- 172.21.15.6:0/773472028 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faaf00d0e50 0x7faaf00b44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:26.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.734+0000 7faaf5c00640 1 -- 172.21.15.6:0/773472028 >> v1:172.21.15.146:6789/0 conn(0x7faaf00d4a10 legacy=0x7faaf00d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:26.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.734+0000 7faaf5c00640 1 -- 172.21.15.6:0/773472028 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faaf00b3cd0 con 0x7faaf015e0b0 2024-03-04T09:25:26.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.734+0000 7f00a6a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f00a00e3300 0x7f00a00572f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:26.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.734+0000 7f00a6a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f00a00e3300 0x7f00a00572f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.6:33308/0 (socket says 172.21.15.6:33308) 2024-03-04T09:25:26.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.734+0000 7f00a6000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f00a0053750 0x7f00a0053bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:26.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.734+0000 7f00a6a00640 1 -- 172.21.15.6:0/1138343230 learned_addr learned my addr 172.21.15.6:0/1138343230 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:25:26.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.734+0000 7f2419000640 1 --2- 172.21.15.6:0/492306916 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f240c0589c0 0x7f240c03bd30 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f24140505a0 tx=0x7f24140a6d00 comp rx=0 tx=0).stop 2024-03-04T09:25:26.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.735+0000 7f2419000640 1 -- 172.21.15.6:0/492306916 shutdown_connections 2024-03-04T09:25:26.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.735+0000 7f00a6000640 1 -- 172.21.15.6:0/1138343230 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f00a00e3300 msgr2=0x7f00a00572f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:26.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.735+0000 7f2419000640 1 --2- 172.21.15.6:0/492306916 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f240c0589c0 0x7f240c03bd30 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:26.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.735+0000 7f2419000640 1 --2- 172.21.15.6:0/492306916 >> v2:172.21.15.146:3300/0 conn(0x7f240c053c40 0x7f240c007660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:26.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.735+0000 7f2419000640 1 -- 172.21.15.6:0/492306916 >> 172.21.15.6:0/492306916 conn(0x7f240c058ee0 msgr2=0x7f240c0592f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:26.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.735+0000 7f00a6000640 1 --2- 172.21.15.6:0/1138343230 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f00a00e3300 0x7f00a00572f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:26.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.735+0000 7f00a6000640 1 -- 172.21.15.6:0/1138343230 >> v1:172.21.15.146:6789/0 conn(0x7f00a0054110 legacy=0x7f00a00061f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:26.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.735+0000 7f00a6000640 1 -- 172.21.15.6:0/1138343230 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00a0057e20 con 0x7f00a0053750 2024-03-04T09:25:26.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.735+0000 7f2419000640 1 -- 172.21.15.6:0/492306916 shutdown_connections 2024-03-04T09:25:26.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.735+0000 7f2419000640 1 -- 172.21.15.6:0/492306916 wait complete. 2024-03-04T09:25:26.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.735+0000 7f4a44a00640 1 -- start start 2024-03-04T09:25:26.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.736+0000 7faaf5c00640 1 --2- 172.21.15.6:0/773472028 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faaf015e0b0 0x7faaf00cf310 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7faaf00b68c0 tx=0x7faae402b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e1735d63b426a2a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:26.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.737+0000 7faaefe00640 1 -- 172.21.15.6:0/773472028 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faae4035020 con 0x7faaf015e0b0 2024-03-04T09:25:26.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.737+0000 7faaefe00640 1 -- 172.21.15.6:0/773472028 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faae4031640 con 0x7faaf015e0b0 2024-03-04T09:25:26.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.737+0000 7faaefe00640 1 -- 172.21.15.6:0/773472028 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faae402d040 con 0x7faaf015e0b0 2024-03-04T09:25:26.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.737+0000 7f4a44a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f4a3804a320 0x7f4a38039b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:26.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.737+0000 7f4a44a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4a380529f0 0x7f4a38052e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:26.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.737+0000 7f00a6000640 1 --2- 172.21.15.6:0/1138343230 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f00a0053750 0x7f00a0053bd0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f00980097a0 tx=0x7f009802b6d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4647ecf31f3eec6 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:26.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.738+0000 7f4a44a00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a3803b340 con 0x7f4a380361b0 2024-03-04T09:25:26.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.738+0000 7f4a44a00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a3803b5e0 con 0x7f4a3804a320 2024-03-04T09:25:26.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.738+0000 7f4a44a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a3803b8c0 con 0x7f4a380529f0 2024-03-04T09:25:26.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.738+0000 7f00a5600640 1 -- 172.21.15.6:0/1138343230 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0098037070 con 0x7f00a0053750 2024-03-04T09:25:26.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.738+0000 7f2419000640 1 Processor -- start 2024-03-04T09:25:26.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.738+0000 7f4a3f200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4a380529f0 0x7f4a38052e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:26.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.738+0000 7f00a5600640 1 -- 172.21.15.6:0/1138343230 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0098030710 con 0x7f00a0053750 2024-03-04T09:25:26.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.738+0000 7f00a5600640 1 -- 172.21.15.6:0/1138343230 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0098030ae0 con 0x7f00a0053750 2024-03-04T09:25:26.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.738+0000 7faaf7000640 1 -- 172.21.15.6:0/773472028 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faaf015e0b0 msgr2=0x7faaf00cf310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:26.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.738+0000 7faaf7000640 1 --2- 172.21.15.6:0/773472028 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faaf015e0b0 0x7faaf00cf310 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7faaf00b68c0 tx=0x7faae402b8f0 comp rx=0 tx=0).stop 2024-03-04T09:25:26.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.738+0000 7f4a3f200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4a380529f0 0x7f4a38052e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.6:33330/0 (socket says 172.21.15.6:33330) 2024-03-04T09:25:26.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.738+0000 7f4a3f200640 1 -- 172.21.15.6:0/3495772973 learned_addr learned my addr 172.21.15.6:0/3495772973 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:25:26.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.738+0000 7f2419000640 1 -- start start 2024-03-04T09:25:26.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.739+0000 7f4a3e800640 1 --2- 172.21.15.6:0/3495772973 >> v2:172.21.15.6:3300/0 conn(0x7f4a3804a320 0x7f4a38039b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:26.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.739+0000 7f4a3f200640 1 -- 172.21.15.6:0/3495772973 >> v2:172.21.15.6:3300/0 conn(0x7f4a3804a320 msgr2=0x7f4a38039b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:26.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.739+0000 7f4a3f200640 1 --2- 172.21.15.6:0/3495772973 >> v2:172.21.15.6:3300/0 conn(0x7f4a3804a320 0x7f4a38039b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:26.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.739+0000 7f4a3f200640 1 -- 172.21.15.6:0/3495772973 >> v1:172.21.15.146:6789/0 conn(0x7f4a380361b0 legacy=0x7f4a3803c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:26.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.739+0000 7f00a7e00640 1 -- 172.21.15.6:0/1138343230 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f00a0053750 msgr2=0x7f00a0053bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:26.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.739+0000 7f00a7e00640 1 --2- 172.21.15.6:0/1138343230 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f00a0053750 0x7f00a0053bd0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f00980097a0 tx=0x7f009802b6d0 comp rx=0 tx=0).stop 2024-03-04T09:25:26.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.740+0000 7f2419000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f240c053c40 0x7f240c009d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:26.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.740+0000 7f00a7e00640 1 -- 172.21.15.6:0/1138343230 shutdown_connections 2024-03-04T09:25:26.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.740+0000 7f00a7e00640 1 --2- 172.21.15.6:0/1138343230 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f00a0053750 0x7f00a0053bd0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:26.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.740+0000 7f00a7e00640 1 --2- 172.21.15.6:0/1138343230 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f00a00e3300 0x7f00a00572f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:26.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.740+0000 7f00a7e00640 1 -- 172.21.15.6:0/1138343230 >> 172.21.15.6:0/1138343230 conn(0x7f00a0058ef0 msgr2=0x7f00a0059300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:26.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.739+0000 7f4a3f200640 1 -- 172.21.15.6:0/3495772973 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a3803bba0 con 0x7f4a380529f0 2024-03-04T09:25:26.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.740+0000 7f4a3e800640 1 --2- 172.21.15.6:0/3495772973 >> v2:172.21.15.6:3300/0 conn(0x7f4a3804a320 0x7f4a38039b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T09:25:26.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.740+0000 7f2419000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f240c0589c0 0x7f240c0d72b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:26.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.740+0000 7f2419000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f240c0e75f0 0x7f240c00d250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:26.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.740+0000 7f2419000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f240c007140 con 0x7f240c0589c0 2024-03-04T09:25:26.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.740+0000 7f2419000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f240c006e60 con 0x7f240c0e75f0 2024-03-04T09:25:26.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.740+0000 7f2419000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f240c0061e0 con 0x7f240c053c40 2024-03-04T09:25:26.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.740+0000 7f00a7e00640 1 -- 172.21.15.6:0/1138343230 shutdown_connections 2024-03-04T09:25:26.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.741+0000 7f2411400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f240c0589c0 0x7f240c0d72b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:26.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.741+0000 7f00a7e00640 1 -- 172.21.15.6:0/1138343230 wait complete. 2024-03-04T09:25:26.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.741+0000 7faaf7000640 1 -- 172.21.15.6:0/773472028 shutdown_connections 2024-03-04T09:25:26.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.741+0000 7faaf7000640 1 --2- 172.21.15.6:0/773472028 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faaf00d0e50 0x7faaf00b44d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:26.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.741+0000 7faaf7000640 1 --2- 172.21.15.6:0/773472028 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faaf015e0b0 0x7faaf00cf310 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:26.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.741+0000 7f4a3f200640 1 --2- 172.21.15.6:0/3495772973 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4a380529f0 0x7f4a38052e80 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f4a400697f0 tx=0x7f4a400a6b00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dcc7d7bb952d034d server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:26.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.741+0000 7f2411e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f240c053c40 0x7f240c009d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:26.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.741+0000 7f2411400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f240c0589c0 0x7f240c0d72b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45276/0 (socket says 172.21.15.6:45276) 2024-03-04T09:25:26.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.741+0000 7f2411400640 1 -- 172.21.15.6:0/2724195700 learned_addr learned my addr 172.21.15.6:0/2724195700 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:25:26.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.741+0000 7f2412800640 1 --2- 172.21.15.6:0/2724195700 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f240c0e75f0 0x7f240c00d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:26.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.741+0000 7f2411400640 1 -- 172.21.15.6:0/2724195700 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f240c053c40 msgr2=0x7f240c009d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:26.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.741+0000 7f2411400640 1 --2- 172.21.15.6:0/2724195700 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f240c053c40 0x7f240c009d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:26.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.741+0000 7f2411400640 1 -- 172.21.15.6:0/2724195700 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f240c0e75f0 msgr2=0x7f240c00d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:26.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.741+0000 7f2411400640 1 --2- 172.21.15.6:0/2724195700 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f240c0e75f0 0x7f240c00d250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:26.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.741+0000 7f2411400640 1 -- 172.21.15.6:0/2724195700 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f240c050050 con 0x7f240c0589c0 2024-03-04T09:25:26.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.741+0000 7faaf7000640 1 -- 172.21.15.6:0/773472028 >> 172.21.15.6:0/773472028 conn(0x7faaf00d3d30 msgr2=0x7faaf00d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:26.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.741+0000 7f2412800640 1 --2- 172.21.15.6:0/2724195700 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f240c0e75f0 0x7f240c00d250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:25:26.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.741+0000 7f4a3d400640 1 -- 172.21.15.6:0/3495772973 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a400a8980 con 0x7f4a380529f0 2024-03-04T09:25:26.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.741+0000 7f4a3d400640 1 -- 172.21.15.6:0/3495772973 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a400a8b20 con 0x7f4a380529f0 2024-03-04T09:25:26.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.742+0000 7f2411400640 1 --2- 172.21.15.6:0/2724195700 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f240c0589c0 0x7f240c0d72b0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f240000e9b0 tx=0x7f240000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:26.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.742+0000 7f240a000640 1 -- 172.21.15.6:0/2724195700 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2400017070 con 0x7f240c0589c0 2024-03-04T09:25:26.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.742+0000 7f240a000640 1 -- 172.21.15.6:0/2724195700 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f240000c9a0 con 0x7f240c0589c0 2024-03-04T09:25:26.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.742+0000 7f240a000640 1 -- 172.21.15.6:0/2724195700 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f240000cd90 con 0x7f240c0589c0 2024-03-04T09:25:26.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.742+0000 7f2419000640 1 -- 172.21.15.6:0/2724195700 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f240c050330 con 0x7f240c0589c0 2024-03-04T09:25:26.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.742+0000 7f2419000640 1 -- 172.21.15.6:0/2724195700 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f240c0508f0 con 0x7f240c0589c0 2024-03-04T09:25:26.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.742+0000 7f4a3d400640 1 -- 172.21.15.6:0/3495772973 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a400b0420 con 0x7f4a380529f0 2024-03-04T09:25:26.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.743+0000 7faaf7000640 1 -- 172.21.15.6:0/773472028 shutdown_connections 2024-03-04T09:25:26.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.743+0000 7f00a7e00640 1 Processor -- start 2024-03-04T09:25:26.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.743+0000 7f4a44a00640 1 -- 172.21.15.6:0/3495772973 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4a380529f0 msgr2=0x7f4a38052e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:26.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.743+0000 7f4a44a00640 1 --2- 172.21.15.6:0/3495772973 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4a380529f0 0x7f4a38052e80 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f4a400697f0 tx=0x7f4a400a6b00 comp rx=0 tx=0).stop 2024-03-04T09:25:26.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.743+0000 7f4a44a00640 1 -- 172.21.15.6:0/3495772973 shutdown_connections 2024-03-04T09:25:26.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.743+0000 7f4a44a00640 1 --2- 172.21.15.6:0/3495772973 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4a380529f0 0x7f4a38052e80 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:26.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.743+0000 7f4a44a00640 1 --2- 172.21.15.6:0/3495772973 >> v2:172.21.15.6:3300/0 conn(0x7f4a3804a320 0x7f4a38039b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:26.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.743+0000 7f4a44a00640 1 -- 172.21.15.6:0/3495772973 >> 172.21.15.6:0/3495772973 conn(0x7f4a380537b0 msgr2=0x7f4a38053b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:26.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.743+0000 7f6ae9400640 1 Processor -- start 2024-03-04T09:25:26.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.743+0000 7faaf7000640 1 -- 172.21.15.6:0/773472028 wait complete. 2024-03-04T09:25:26.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.743+0000 7f6ae9400640 1 -- start start 2024-03-04T09:25:26.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.744+0000 7f2419000640 1 -- 172.21.15.6:0/2724195700 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23ec005290 con 0x7f240c0589c0 2024-03-04T09:25:26.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.745+0000 7f00a7e00640 1 -- start start 2024-03-04T09:25:26.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.745+0000 7f6936600640 1 Processor -- start 2024-03-04T09:25:26.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.745+0000 7f6936600640 1 -- start start 2024-03-04T09:25:26.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.745+0000 7f6ae9400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f6ae4070ee0 0x7f6ae4162c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:26.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.745+0000 7f6ae9400640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ae40ce090 con 0x7f6ae40712c0 2024-03-04T09:25:26.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.745+0000 7f6ae9400640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ae40ce370 con 0x7f6ae4072da0 2024-03-04T09:25:26.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.745+0000 7f6ae9400640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ae40ceca0 con 0x7f6ae4070ee0 2024-03-04T09:25:26.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.745+0000 7f4a44a00640 1 -- 172.21.15.6:0/3495772973 shutdown_connections 2024-03-04T09:25:26.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.745+0000 7f4a44a00640 1 -- 172.21.15.6:0/3495772973 wait complete. 2024-03-04T09:25:26.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.745+0000 7fa51e200640 1 Processor -- start 2024-03-04T09:25:26.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.746+0000 7f6ae2800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f6ae4070ee0 0x7f6ae4162c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:26.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.746+0000 7f6ae3200640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f6ae40712c0 0x7f6ae406cfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:60220/0 (socket says 172.21.15.6:60220) 2024-03-04T09:25:26.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.747+0000 7f6ae3200640 1 -- 172.21.15.6:0/2291416662 learned_addr learned my addr 172.21.15.6:0/2291416662 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:25:26.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.747+0000 7faaf7000640 1 Processor -- start 2024-03-04T09:25:26.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.747+0000 7faaf7000640 1 -- start start 2024-03-04T09:25:26.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.747+0000 7f240a000640 1 -- 172.21.15.6:0/2724195700 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f240001a070 con 0x7f240c0589c0 2024-03-04T09:25:26.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.747+0000 7f6936600640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f69300b3c90 0x7f69300cdc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:26.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.746+0000 7fa51e200640 1 -- start start 2024-03-04T09:25:26.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.748+0000 7f6936600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6930071050 0x7f6930071490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:26.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.748+0000 7f6936600640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69300b4860 con 0x7f69300cf5c0 2024-03-04T09:25:26.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.748+0000 7f6936600640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69300ce160 con 0x7f69300b3c90 2024-03-04T09:25:26.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.748+0000 7f6936600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69300b4b40 con 0x7f6930071050 2024-03-04T09:25:26.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.748+0000 7f4a44a00640 1 Processor -- start 2024-03-04T09:25:26.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.748+0000 7f00a7e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f00a0054110 0x7f00a00ddf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:26.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.748+0000 7f00a7e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f00a00e3300 0x7f00a00d9300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:26.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.748+0000 7f00a7e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f00a00e5b80 0x7f00a00e6010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:26.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.748+0000 7f4a44a00640 1 -- start start 2024-03-04T09:25:26.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.748+0000 7f00a7e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00a0057830 con 0x7f00a0054110 2024-03-04T09:25:26.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.748+0000 7f00a7e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00a00583e0 con 0x7f00a00e5b80 2024-03-04T09:25:26.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.748+0000 7f00a7e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00a003d130 con 0x7f00a00e3300 2024-03-04T09:25:26.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.749+0000 7faaf7000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faaf00d0e50 0x7faaf00b8240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:26.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.749+0000 7faaf7000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faaf00d4a10 0x7faaf00b6950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:26.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.749+0000 7faaf7000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faaf015e0b0 0x7faaf015c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:26.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.749+0000 7faaf7000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaf00cfc60 con 0x7faaf015e0b0 2024-03-04T09:25:26.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.749+0000 7f240a000640 1 --2- 172.21.15.6:0/2724195700 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f23f406b5a0 0x7f23f406da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:26.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.749+0000 7f240a000640 1 -- 172.21.15.6:0/2724195700 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f240008a4e0 con 0x7f240c0589c0 2024-03-04T09:25:26.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.749+0000 7fa51e200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa510005ee0 0x7fa510006320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:26.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.749+0000 7fa51ce00640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fa5100ead00 0x7fa51004fa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.6:50200/0 (socket says 172.21.15.6:50200) 2024-03-04T09:25:26.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.749+0000 7fa51ce00640 1 -- 172.21.15.6:0/2461347178 learned_addr learned my addr 172.21.15.6:0/2461347178 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:25:26.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.749+0000 7f692f400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f69300b3c90 0x7f69300cdc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:26.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.750+0000 7faaf7000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaf015e490 con 0x7faaf00d0e50 2024-03-04T09:25:26.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.748+0000 7f6ae2800640 1 -- 172.21.15.6:0/2291416662 >> v1:172.21.15.146:6789/0 conn(0x7f6ae4072da0 legacy=0x7f6ae406e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:26.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.748+0000 7f6ae2800640 1 -- 172.21.15.6:0/2291416662 >> v1:172.21.15.6:6789/0 conn(0x7f6ae40712c0 legacy=0x7f6ae406cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:26.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.748+0000 7f6ae2800640 1 -- 172.21.15.6:0/2291416662 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ae40ce650 con 0x7f6ae4070ee0 2024-03-04T09:25:26.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.750+0000 7fa51e200640 1 -- 172.21.15.6:0/2461347178 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa510053720 con 0x7fa510006860 2024-03-04T09:25:26.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.750+0000 7faaf7000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faaf00cf850 con 0x7faaf00d4a10 2024-03-04T09:25:26.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.750+0000 7fa51e200640 1 -- 172.21.15.6:0/2461347178 --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa510053a00 con 0x7fa5100ead00 2024-03-04T09:25:26.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.750+0000 7faaf5200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faaf00d4a10 0x7faaf00b6950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:26.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.750+0000 7f00a6000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f00a00e3300 0x7f00a00d9300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:26.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.750+0000 7f00a7400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f00a00e5b80 0x7f00a00e6010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:26.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.750+0000 7f00a7400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f00a00e5b80 0x7f00a00e6010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.6:47254/0 (socket says 172.21.15.6:47254) 2024-03-04T09:25:26.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.750+0000 7f00a7400640 1 -- 172.21.15.6:0/3320596365 learned_addr learned my addr 172.21.15.6:0/3320596365 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:25:26.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.750+0000 7fa51e200640 1 -- 172.21.15.6:0/2461347178 --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa510054330 con 0x7fa510005ee0 2024-03-04T09:25:26.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.750+0000 7f00a6a00640 1 --2- 172.21.15.6:0/3320596365 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f00a0054110 0x7f00a00ddf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:26.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.750+0000 7fa517e00640 1 --2- 172.21.15.6:0/2461347178 >> v2:172.21.15.6:3300/0 conn(0x7fa510005ee0 0x7fa510006320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:26.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.751+0000 7f6ae2800640 1 --2- 172.21.15.6:0/2291416662 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6ae4070ee0 0x7f6ae4162c40 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f6ae40cee20 tx=0x7f6ad402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2c0eb9644548f7e server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:26.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.750+0000 7f2411e00640 1 --2- 172.21.15.6:0/2724195700 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f23f406b5a0 0x7f23f406da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:26.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.751+0000 7f4a44a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4a380361b0 0x7f4a3817bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:26.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.751+0000 7f4a44a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4a3804a320 0x7f4a3817a680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:26.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.751+0000 7fa517400640 1 -- 172.21.15.6:0/2461347178 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 21172211 0 0) 0x7fa510053a00 con 0x7fa5100ead00 2024-03-04T09:25:26.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.751+0000 7fa517400640 1 -- 172.21.15.6:0/2461347178 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa500003520 con 0x7fa5100ead00 2024-03-04T09:25:26.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.751+0000 7f4a44a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4a380529f0 0x7f4a3804d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:26.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.750+0000 7f692fe00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6930071050 0x7f6930071490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:26.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.751+0000 7fa517e00640 1 -- 172.21.15.6:0/2461347178 >> v1:172.21.15.146:6789/0 conn(0x7fa5100ead00 legacy=0x7fa51004fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:26.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.751+0000 7fa517e00640 1 -- 172.21.15.6:0/2461347178 >> v1:172.21.15.6:6789/0 conn(0x7fa510006860 legacy=0x7fa510006cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:26.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.750+0000 7faaf5200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faaf00d4a10 0x7faaf00b6950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.6:33362/0 (socket says 172.21.15.6:33362) 2024-03-04T09:25:26.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.750+0000 7faaf5200640 1 -- 172.21.15.6:0/3065817601 learned_addr learned my addr 172.21.15.6:0/3065817601 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:25:26.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.751+0000 7fa517e00640 1 -- 172.21.15.6:0/2461347178 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa510053ce0 con 0x7fa510005ee0 2024-03-04T09:25:26.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.751+0000 7faaf5c00640 1 --2- 172.21.15.6:0/3065817601 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faaf00d0e50 0x7faaf00b8240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:26.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.750+0000 7faaf6600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faaf015e0b0 0x7faaf015c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:26.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.751+0000 7f692fe00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6930071050 0x7f6930071490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.6:33354/0 (socket says 172.21.15.6:33354) 2024-03-04T09:25:26.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.751+0000 7f692fe00640 1 -- 172.21.15.6:0/348919022 learned_addr learned my addr 172.21.15.6:0/348919022 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:25:26.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.750+0000 7f692ea00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f69300cf5c0 0x7f69300b4070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:60234/0 (socket says 172.21.15.6:60234) 2024-03-04T09:25:26.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.751+0000 7f4a44a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a380eb4d0 con 0x7f4a380361b0 2024-03-04T09:25:26.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.751+0000 7f4a44a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a380eb350 con 0x7f4a380529f0 2024-03-04T09:25:26.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.751+0000 7f4a44a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a38054870 con 0x7f4a3804a320 2024-03-04T09:25:26.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.752+0000 7f4a3e800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4a380361b0 0x7f4a3817bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:26.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.752+0000 7f692fe00640 1 -- 172.21.15.6:0/348919022 >> v2:172.21.15.146:3300/0 conn(0x7f69300b3c90 msgr2=0x7f69300cdc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:26.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.752+0000 7f692fe00640 1 --2- 172.21.15.6:0/348919022 >> v2:172.21.15.146:3300/0 conn(0x7f69300b3c90 0x7f69300cdc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:26.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.752+0000 7f692fe00640 1 -- 172.21.15.6:0/348919022 >> v1:172.21.15.6:6789/0 conn(0x7f69300cf5c0 legacy=0x7f69300b4070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:26.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.752+0000 7f692fe00640 1 -- 172.21.15.6:0/348919022 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69300b4e20 con 0x7f6930071050 2024-03-04T09:25:26.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.752+0000 7fa517e00640 1 --2- 172.21.15.6:0/2461347178 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa510005ee0 0x7fa510006320 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fa5040097a0 tx=0x7fa50402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e32c35feca6c9239 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:26.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.752+0000 7faaf5200640 1 -- 172.21.15.6:0/3065817601 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faaf00d0e50 msgr2=0x7faaf00b8240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:26.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.752+0000 7faaf5200640 1 --2- 172.21.15.6:0/3065817601 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faaf00d0e50 0x7faaf00b8240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:26.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.752+0000 7faaf5200640 1 -- 172.21.15.6:0/3065817601 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faaf015e0b0 msgr2=0x7faaf015c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:26.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.752+0000 7faaf5200640 1 --2- 172.21.15.6:0/3065817601 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faaf015e0b0 0x7faaf015c8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:26.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.752+0000 7faaf5200640 1 -- 172.21.15.6:0/3065817601 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faaf00b71a0 con 0x7faaf00d4a10 2024-03-04T09:25:26.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.752+0000 7f4a3e800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4a380361b0 0x7f4a3817bfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45324/0 (socket says 172.21.15.6:45324) 2024-03-04T09:25:26.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.752+0000 7f4a3e800640 1 -- 172.21.15.6:0/537460095 learned_addr learned my addr 172.21.15.6:0/537460095 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:25:26.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.752+0000 7f4a3f200640 1 --2- 172.21.15.6:0/537460095 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4a380529f0 0x7f4a3804d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:26.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.753+0000 7f2411e00640 1 --2- 172.21.15.6:0/2724195700 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f23f406b5a0 0x7f23f406da60 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f24040011e0 tx=0x7f2404007d30 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:26.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.753+0000 7faaf5200640 1 --2- 172.21.15.6:0/3065817601 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faaf00d4a10 0x7faaf00b6950 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7faadc00c400 tx=0x7faadc00c8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:26.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.751+0000 7f00a7400640 1 -- 172.21.15.6:0/3320596365 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f00a00e3300 msgr2=0x7f00a00d9300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:26.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.751+0000 7f00a7400640 1 --2- 172.21.15.6:0/3320596365 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f00a00e3300 0x7f00a00d9300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:26.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.751+0000 7f00a7400640 1 -- 172.21.15.6:0/3320596365 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f00a0054110 msgr2=0x7f00a00ddf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:26.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.751+0000 7f00a7400640 1 --2- 172.21.15.6:0/3320596365 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f00a0054110 0x7f00a00ddf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:26.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.751+0000 7f00a7400640 1 -- 172.21.15.6:0/3320596365 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00a00d9840 con 0x7f00a00e5b80 2024-03-04T09:25:26.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.753+0000 7f00a7400640 1 --2- 172.21.15.6:0/3320596365 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f00a00e5b80 0x7f00a00e6010 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f009400c3c0 tx=0x7f009400c880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:26.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.753+0000 7fda87600640 1 Processor -- start 2024-03-04T09:25:26.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.753+0000 7fda87600640 1 -- start start 2024-03-04T09:25:26.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.753+0000 7f4a3de00640 1 --2- 172.21.15.6:0/537460095 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4a3804a320 0x7f4a3817a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:26.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.753+0000 7f4a3e800640 1 -- 172.21.15.6:0/537460095 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4a3804a320 msgr2=0x7f4a3817a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:26.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.753+0000 7f4a3e800640 1 --2- 172.21.15.6:0/537460095 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4a3804a320 0x7f4a3817a680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:26.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.753+0000 7f4a3e800640 1 -- 172.21.15.6:0/537460095 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4a380529f0 msgr2=0x7f4a3804d180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:26.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.753+0000 7f4a3e800640 1 --2- 172.21.15.6:0/537460095 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4a380529f0 0x7f4a3804d180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:26.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.753+0000 7f4a3e800640 1 -- 172.21.15.6:0/537460095 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a3804c050 con 0x7f4a380361b0 2024-03-04T09:25:26.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.753+0000 7f4a3e800640 1 --2- 172.21.15.6:0/537460095 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4a380361b0 0x7f4a3817bfa0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f4a3000e810 tx=0x7f4a3000ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:26.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.753+0000 7f009f400640 1 -- 172.21.15.6:0/3320596365 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0094017070 con 0x7f00a00e5b80 2024-03-04T09:25:26.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.754+0000 7f009f400640 1 -- 172.21.15.6:0/3320596365 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00940035f0 con 0x7f00a00e5b80 2024-03-04T09:25:26.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.754+0000 7f009f400640 1 -- 172.21.15.6:0/3320596365 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00940039e0 con 0x7f00a00e5b80 2024-03-04T09:25:26.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.754+0000 7f00a7e00640 1 -- 172.21.15.6:0/3320596365 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00a0050c40 con 0x7f00a00e5b80 2024-03-04T09:25:26.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.754+0000 7f00a7e00640 1 -- 172.21.15.6:0/3320596365 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00a00dd760 con 0x7f00a00e5b80 2024-03-04T09:25:26.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.754+0000 7f692fe00640 1 --2- 172.21.15.6:0/348919022 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6930071050 0x7f6930071490 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f692800bef0 tx=0x7f69280093b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e6c4b9addfe5337f server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:26.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.754+0000 7faaed600640 1 -- 172.21.15.6:0/3065817601 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faadc017070 con 0x7faaf00d4a10 2024-03-04T09:25:26.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.754+0000 7fda87600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fda80165960 0x7fda80070c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:26.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.754+0000 7faaed600640 1 -- 172.21.15.6:0/3065817601 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faadc0035f0 con 0x7faaf00d4a10 2024-03-04T09:25:26.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.754+0000 7faaed600640 1 -- 172.21.15.6:0/3065817601 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faadc0039e0 con 0x7faaf00d4a10 2024-03-04T09:25:26.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.754+0000 7faaf7000640 1 -- 172.21.15.6:0/3065817601 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faaf00b7480 con 0x7faaf00d4a10 2024-03-04T09:25:26.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.754+0000 7faaf7000640 1 -- 172.21.15.6:0/3065817601 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faaf00b7a40 con 0x7faaf00d4a10 2024-03-04T09:25:26.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.754+0000 7fda87600640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda80166f90 con 0x7fda8006e3d0 2024-03-04T09:25:26.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.755+0000 7fda87600640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda8006da90 con 0x7fda8006d520 2024-03-04T09:25:26.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.755+0000 7fda87600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda800cdc20 con 0x7fda80165960 2024-03-04T09:25:26.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.755+0000 7fda84c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fda80165960 0x7fda80070c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:26.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.755+0000 7fda84c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fda80165960 0x7fda80070c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.6:33386/0 (socket says 172.21.15.6:33386) 2024-03-04T09:25:26.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.755+0000 7fda84c00640 1 -- 172.21.15.6:0/3690164114 learned_addr learned my addr 172.21.15.6:0/3690164114 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:25:26.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.755+0000 7f692e000640 1 -- 172.21.15.6:0/348919022 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f692802c920 con 0x7f6930071050 2024-03-04T09:25:26.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.755+0000 7f692e000640 1 -- 172.21.15.6:0/348919022 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f692802cac0 con 0x7f6930071050 2024-03-04T09:25:26.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.755+0000 7f00a7e00640 1 -- 172.21.15.6:0/3320596365 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00a0005710 con 0x7f00a00e5b80 2024-03-04T09:25:26.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.756+0000 7f6936600640 1 -- 172.21.15.6:0/348919022 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6930071050 msgr2=0x7f6930071490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:26.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.756+0000 7f692e000640 1 -- 172.21.15.6:0/348919022 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6928009600 con 0x7f6930071050 2024-03-04T09:25:26.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.756+0000 7f6936600640 1 --2- 172.21.15.6:0/348919022 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6930071050 0x7f6930071490 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f692800bef0 tx=0x7f69280093b0 comp rx=0 tx=0).stop 2024-03-04T09:25:26.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.756+0000 7fda84c00640 1 -- 172.21.15.6:0/3690164114 >> v1:172.21.15.146:6789/0 conn(0x7fda8006d520 legacy=0x7fda80166170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:26.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.756+0000 7fda84c00640 1 -- 172.21.15.6:0/3690164114 >> v1:172.21.15.6:6789/0 conn(0x7fda8006e3d0 legacy=0x7fda80166880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:26.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.756+0000 7f6936600640 1 -- 172.21.15.6:0/348919022 shutdown_connections 2024-03-04T09:25:26.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.756+0000 7fda84c00640 1 -- 172.21.15.6:0/3690164114 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda8006dd70 con 0x7fda80165960 2024-03-04T09:25:26.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.756+0000 7f6936600640 1 --2- 172.21.15.6:0/348919022 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6930071050 0x7f6930071490 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:26.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.756+0000 7f6936600640 1 --2- 172.21.15.6:0/348919022 >> v2:172.21.15.146:3300/0 conn(0x7f69300b3c90 0x7f69300cdc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:26.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.756+0000 7f6936600640 1 -- 172.21.15.6:0/348919022 >> 172.21.15.6:0/348919022 conn(0x7f693006afb0 msgr2=0x7f693006b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:26.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.756+0000 7f6936600640 1 -- 172.21.15.6:0/348919022 shutdown_connections 2024-03-04T09:25:26.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.756+0000 7f6936600640 1 -- 172.21.15.6:0/348919022 wait complete. 2024-03-04T09:25:26.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.756+0000 7faaf7000640 1 -- 172.21.15.6:0/3065817601 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faacc005290 con 0x7faaf00d4a10 2024-03-04T09:25:26.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.757+0000 7fa517400640 1 -- 172.21.15.6:0/2461347178 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa504035020 con 0x7fa510005ee0 2024-03-04T09:25:26.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.757+0000 7f009f400640 1 -- 172.21.15.6:0/3320596365 <== mon.1 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0094003b80 con 0x7f00a00e5b80 2024-03-04T09:25:26.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.757+0000 7f6ae1e00640 1 -- 172.21.15.6:0/2291416662 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ad4035020 con 0x7f6ae4070ee0 2024-03-04T09:25:26.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.757+0000 7fda84c00640 1 --2- 172.21.15.6:0/3690164114 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fda80165960 0x7fda80070c70 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fda740098a0 tx=0x7fda7402b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4b93ed100b130abf server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:26.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.758+0000 7fa517400640 1 -- 172.21.15.6:0/2461347178 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa504031830 con 0x7fa510005ee0 2024-03-04T09:25:26.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.758+0000 7fa517400640 1 -- 172.21.15.6:0/2461347178 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa504031be0 con 0x7fa510005ee0 2024-03-04T09:25:26.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.758+0000 7f240a000640 1 -- 172.21.15.6:0/2724195700 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2400058cf0 con 0x7f240c0589c0 2024-03-04T09:25:26.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.758+0000 7f6ae1e00640 1 -- 172.21.15.6:0/2291416662 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ad4031830 con 0x7f6ae4070ee0 2024-03-04T09:25:26.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.758+0000 7f4a36a00640 1 -- 172.21.15.6:0/537460095 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a3000f040 con 0x7f4a380361b0 2024-03-04T09:25:26.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.758+0000 7f6ae1e00640 1 -- 172.21.15.6:0/2291416662 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ad4031c40 con 0x7f6ae4070ee0 2024-03-04T09:25:26.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.758+0000 7f4a36a00640 1 -- 172.21.15.6:0/537460095 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a30004030 con 0x7f4a380361b0 2024-03-04T09:25:26.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.758+0000 7f4a36a00640 1 -- 172.21.15.6:0/537460095 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a30004400 con 0x7f4a380361b0 2024-03-04T09:25:26.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.758+0000 7f4a44a00640 1 -- 172.21.15.6:0/537460095 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a3804b290 con 0x7f4a380361b0 2024-03-04T09:25:26.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.758+0000 7f4a44a00640 1 -- 172.21.15.6:0/537460095 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a3804b850 con 0x7f4a380361b0 2024-03-04T09:25:26.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.759+0000 7faaed600640 1 -- 172.21.15.6:0/3065817601 <== mon.2 v2:172.21.15.146:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7faadc01a070 con 0x7faaf00d4a10 2024-03-04T09:25:26.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.759+0000 7f6936600640 1 Processor -- start 2024-03-04T09:25:26.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.759+0000 7fa51e200640 1 -- 172.21.15.6:0/2461347178 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa510005ee0 msgr2=0x7fa510006320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:26.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.759+0000 7fa51e200640 1 --2- 172.21.15.6:0/2461347178 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa510005ee0 0x7fa510006320 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fa5040097a0 tx=0x7fa50402b8d0 comp rx=0 tx=0).stop 2024-03-04T09:25:26.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.759+0000 7f009f400640 1 --2- 172.21.15.6:0/3320596365 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f008806b4d0 0x7f008806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:26.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.759+0000 7f009f400640 1 -- 172.21.15.6:0/3320596365 <== mon.1 v2:172.21.15.146:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0094089570 con 0x7f00a00e5b80 2024-03-04T09:25:26.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.760+0000 7fa51e200640 1 -- 172.21.15.6:0/2461347178 shutdown_connections 2024-03-04T09:25:26.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.760+0000 7fa51e200640 1 --2- 172.21.15.6:0/2461347178 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa510005ee0 0x7fa510006320 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:26.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.760+0000 7fa51e200640 1 -- 172.21.15.6:0/2461347178 >> 172.21.15.6:0/2461347178 conn(0x7fa510009fb0 msgr2=0x7fa51000a390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:26.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.760+0000 7f6ae9400640 1 -- 172.21.15.6:0/2291416662 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6ae4070ee0 msgr2=0x7f6ae4162c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:26.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.760+0000 7f6ae9400640 1 --2- 172.21.15.6:0/2291416662 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6ae4070ee0 0x7f6ae4162c40 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f6ae40cee20 tx=0x7f6ad402b8d0 comp rx=0 tx=0).stop 2024-03-04T09:25:26.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.759+0000 7f6936600640 1 -- start start 2024-03-04T09:25:26.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.760+0000 7f6ae9400640 1 -- 172.21.15.6:0/2291416662 shutdown_connections 2024-03-04T09:25:26.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.760+0000 7f6ae9400640 1 --2- 172.21.15.6:0/2291416662 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6ae4070ee0 0x7f6ae4162c40 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:26.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.760+0000 7f6ae9400640 1 -- 172.21.15.6:0/2291416662 >> 172.21.15.6:0/2291416662 conn(0x7f6ae406afb0 msgr2=0x7f6ae406b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:26.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.760+0000 7f00a6a00640 1 --2- 172.21.15.6:0/3320596365 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f008806b4d0 0x7f008806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:26.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.760+0000 7f4a44a00640 1 -- 172.21.15.6:0/537460095 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a38005710 con 0x7f4a380361b0 2024-03-04T09:25:26.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.760+0000 7fa51e200640 1 -- 172.21.15.6:0/2461347178 shutdown_connections 2024-03-04T09:25:26.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.760+0000 7fa51e200640 1 -- 172.21.15.6:0/2461347178 wait complete. 2024-03-04T09:25:26.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.760+0000 7faaed600640 1 --2- 172.21.15.6:0/3065817601 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7faad006b5a0 0x7faad006da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:26.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.760+0000 7faaed600640 1 -- 172.21.15.6:0/3065817601 <== mon.2 v2:172.21.15.146:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7faadc089aa0 con 0x7faaf00d4a10 2024-03-04T09:25:26.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.761+0000 7f6ae9400640 1 -- 172.21.15.6:0/2291416662 shutdown_connections 2024-03-04T09:25:26.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.761+0000 7f6936600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6930071050 0x7f693009aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:26.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.761+0000 7f6936600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f69300b3c90 0x7f6930151d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:26.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.761+0000 7f6936600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f69300cf5c0 0x7f6930155ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:26.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.761+0000 7f6936600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69300719d0 con 0x7f6930071050 2024-03-04T09:25:26.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.761+0000 7f6936600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f693015b340 con 0x7f69300cf5c0 2024-03-04T09:25:26.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.761+0000 7f6936600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6930071b50 con 0x7f69300b3c90 2024-03-04T09:25:26.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.761+0000 7faaf5c00640 1 --2- 172.21.15.6:0/3065817601 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7faad006b5a0 0x7faad006da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:26.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.762+0000 7f692f400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6930071050 0x7f693009aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:26.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.761+0000 7f6ae9400640 1 -- 172.21.15.6:0/2291416662 wait complete. 2024-03-04T09:25:26.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.762+0000 7f692f400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6930071050 0x7f693009aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45338/0 (socket says 172.21.15.6:45338) 2024-03-04T09:25:26.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.762+0000 7f692f400640 1 -- 172.21.15.6:0/558902544 learned_addr learned my addr 172.21.15.6:0/558902544 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:25:26.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.762+0000 7f00a6a00640 1 --2- 172.21.15.6:0/3320596365 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f008806b4d0 0x7f008806d990 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f00a806a870 tx=0x7f00a808d040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:26.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.762+0000 7f692fe00640 1 --2- 172.21.15.6:0/558902544 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f69300cf5c0 0x7f6930155ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:26.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.762+0000 7f692f400640 1 -- 172.21.15.6:0/558902544 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f69300b3c90 msgr2=0x7f6930151d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:26.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.762+0000 7f692f400640 1 --2- 172.21.15.6:0/558902544 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f69300b3c90 0x7f6930151d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:26.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.762+0000 7f692f400640 1 -- 172.21.15.6:0/558902544 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f69300cf5c0 msgr2=0x7f6930155ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:26.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.762+0000 7f692f400640 1 --2- 172.21.15.6:0/558902544 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f69300cf5c0 0x7f6930155ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:26.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.762+0000 7f692f400640 1 -- 172.21.15.6:0/558902544 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69301546f0 con 0x7f6930071050 2024-03-04T09:25:26.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.762+0000 7f692fe00640 1 --2- 172.21.15.6:0/558902544 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f69300cf5c0 0x7f6930155ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:25:26.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.762+0000 7fa51e200640 1 Processor -- start 2024-03-04T09:25:26.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.763+0000 7fa51e200640 1 -- start start 2024-03-04T09:25:26.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.763+0000 7f692f400640 1 --2- 172.21.15.6:0/558902544 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6930071050 0x7f693009aa10 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f692000bed0 tx=0x7f692000f480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:26.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.763+0000 7f6ae9400640 1 Processor -- start 2024-03-04T09:25:26.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.763+0000 7f6927400640 1 -- 172.21.15.6:0/558902544 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f692001b070 con 0x7f6930071050 2024-03-04T09:25:26.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.763+0000 7f6927400640 1 -- 172.21.15.6:0/558902544 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f692000fd30 con 0x7f6930071050 2024-03-04T09:25:26.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.763+0000 7f6927400640 1 -- 172.21.15.6:0/558902544 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f692000dd50 con 0x7f6930071050 2024-03-04T09:25:26.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.763+0000 7f4a36a00640 1 -- 172.21.15.6:0/537460095 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4a3000c6d0 con 0x7f4a380361b0 2024-03-04T09:25:26.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.764+0000 7faaf5c00640 1 --2- 172.21.15.6:0/3065817601 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7faad006b5a0 0x7faad006da60 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7faae4036000 tx=0x7faae403c040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:26.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.764+0000 7f4a36a00640 1 --2- 172.21.15.6:0/537460095 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f4a2006b520 0x7f4a2006d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:26.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.764+0000 7f4a36a00640 1 -- 172.21.15.6:0/537460095 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4a30089ec0 con 0x7f4a380361b0 2024-03-04T09:25:26.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.764+0000 7f6936600640 1 -- 172.21.15.6:0/558902544 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69301549d0 con 0x7f6930071050 2024-03-04T09:25:26.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.764+0000 7f6936600640 1 -- 172.21.15.6:0/558902544 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6930154f90 con 0x7f6930071050 2024-03-04T09:25:26.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.763+0000 7f6ae9400640 1 -- start start 2024-03-04T09:25:26.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.765+0000 7fa51e200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa510005ee0 0x7fa5100dc0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:26.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.764+0000 7f6ae9400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6ae40712c0 0x7f6ae415e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:26.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.765+0000 7f6ae9400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6ae4072da0 0x7f6ae409ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:26.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.765+0000 7f6ae9400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6ae41504e0 0x7f6ae414f950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:26.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.765+0000 7f6ae9400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ae40cdda0 con 0x7f6ae41504e0 2024-03-04T09:25:26.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.765+0000 7f6ae9400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ae40cdc20 con 0x7f6ae4072da0 2024-03-04T09:25:26.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.765+0000 7f6ae9400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ae406e7d0 con 0x7f6ae40712c0 2024-03-04T09:25:26.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.765+0000 7fa51e200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa510006860 0x7fa510035e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:26.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.765+0000 7fa51e200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa5100ead00 0x7fa51004a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:26.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.765+0000 7f6ae3200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6ae41504e0 0x7f6ae414f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:26.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.765+0000 7f4a3de00640 1 --2- 172.21.15.6:0/537460095 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f4a2006b520 0x7f4a2006d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:26.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.765+0000 7f6ae3200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6ae41504e0 0x7f6ae414f950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45354/0 (socket says 172.21.15.6:45354) 2024-03-04T09:25:26.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.765+0000 7f6ae3200640 1 -- 172.21.15.6:0/1899106332 learned_addr learned my addr 172.21.15.6:0/1899106332 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:25:26.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.765+0000 7fa51d800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa5100ead00 0x7fa51004a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:26.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.765+0000 7fa51d800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa5100ead00 0x7fa51004a920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45370/0 (socket says 172.21.15.6:45370) 2024-03-04T09:25:26.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.765+0000 7fa51ce00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa510005ee0 0x7fa5100dc0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:26.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.765+0000 7fa51d800640 1 -- 172.21.15.6:0/2576357262 learned_addr learned my addr 172.21.15.6:0/2576357262 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:25:26.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.765+0000 7fa51e200640 1 -- 172.21.15.6:0/2576357262 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5100547c0 con 0x7fa5100ead00 2024-03-04T09:25:26.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.766+0000 7fa51e200640 1 -- 172.21.15.6:0/2576357262 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5100501a0 con 0x7fa510005ee0 2024-03-04T09:25:26.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.766+0000 7fa51e200640 1 -- 172.21.15.6:0/2576357262 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa510054940 con 0x7fa510006860 2024-03-04T09:25:26.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.766+0000 7fa517e00640 1 --2- 172.21.15.6:0/2576357262 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa510006860 0x7fa510035e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:26.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.766+0000 7f6ae2800640 1 --2- 172.21.15.6:0/1899106332 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6ae40712c0 0x7f6ae415e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:26.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.766+0000 7f6ad9e00640 1 --2- 172.21.15.6:0/1899106332 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6ae4072da0 0x7f6ae409ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:26.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.766+0000 7f6936600640 1 -- 172.21.15.6:0/558902544 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6930069510 con 0x7f6930071050 2024-03-04T09:25:26.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.766+0000 7f6ae3200640 1 -- 172.21.15.6:0/1899106332 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6ae40712c0 msgr2=0x7f6ae415e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:26.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.766+0000 7f6ae3200640 1 --2- 172.21.15.6:0/1899106332 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6ae40712c0 0x7f6ae415e9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:26.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.766+0000 7f6ae3200640 1 -- 172.21.15.6:0/1899106332 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6ae4072da0 msgr2=0x7f6ae409ac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:26.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.766+0000 7f6ae3200640 1 --2- 172.21.15.6:0/1899106332 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6ae4072da0 0x7f6ae409ac80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:26.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.766+0000 7f6ae3200640 1 -- 172.21.15.6:0/1899106332 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ae415f120 con 0x7f6ae41504e0 2024-03-04T09:25:26.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.766+0000 7fa51d800640 1 -- 172.21.15.6:0/2576357262 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa510006860 msgr2=0x7fa510035e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:26.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.766+0000 7fa51d800640 1 --2- 172.21.15.6:0/2576357262 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa510006860 0x7fa510035e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:26.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.766+0000 7f6ae3200640 1 --2- 172.21.15.6:0/1899106332 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6ae41504e0 0x7f6ae414f950 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f6adc00caa0 tx=0x7f6adc00cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:26.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.766+0000 7fa51d800640 1 -- 172.21.15.6:0/2576357262 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa510005ee0 msgr2=0x7fa5100dc0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:26.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.766+0000 7fa51d800640 1 --2- 172.21.15.6:0/2576357262 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa510005ee0 0x7fa5100dc0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:26.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.767+0000 7fa51d800640 1 -- 172.21.15.6:0/2576357262 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5100363c0 con 0x7fa5100ead00 2024-03-04T09:25:26.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.767+0000 7f4a3de00640 1 --2- 172.21.15.6:0/537460095 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f4a2006b520 0x7f4a2006d9e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f4a380569a0 tx=0x7f4a2c009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:26.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.767+0000 7fa51d800640 1 --2- 172.21.15.6:0/2576357262 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa5100ead00 0x7fa51004a920 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fa518069a10 tx=0x7fa518069430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:26.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.773+0000 7f4a36a00640 1 -- 172.21.15.6:0/537460095 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4a30017020 con 0x7f4a380361b0 2024-03-04T09:25:26.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.774+0000 7fa514c00640 1 -- 172.21.15.6:0/2576357262 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa518090070 con 0x7fa5100ead00 2024-03-04T09:25:26.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.774+0000 7fa514c00640 1 -- 172.21.15.6:0/2576357262 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa518073260 con 0x7fa5100ead00 2024-03-04T09:25:26.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.774+0000 7fa51e200640 1 -- 172.21.15.6:0/2576357262 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5100369d0 con 0x7fa5100ead00 2024-03-04T09:25:26.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.774+0000 7f6adb400640 1 -- 172.21.15.6:0/1899106332 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6adc017070 con 0x7f6ae41504e0 2024-03-04T09:25:26.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.774+0000 7f6adb400640 1 -- 172.21.15.6:0/1899106332 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6adc00ac00 con 0x7f6ae41504e0 2024-03-04T09:25:26.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.774+0000 7f6adb400640 1 -- 172.21.15.6:0/1899106332 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6adc012400 con 0x7f6ae41504e0 2024-03-04T09:25:26.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.774+0000 7f6ae9400640 1 -- 172.21.15.6:0/1899106332 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ae415f400 con 0x7f6ae41504e0 2024-03-04T09:25:26.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.774+0000 7fa51e200640 1 -- 172.21.15.6:0/2576357262 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5100472d0 con 0x7fa5100ead00 2024-03-04T09:25:26.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.775+0000 7f6ae9400640 1 -- 172.21.15.6:0/1899106332 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ae4160a20 con 0x7f6ae41504e0 2024-03-04T09:25:26.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.775+0000 7fa514c00640 1 -- 172.21.15.6:0/2576357262 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa518070630 con 0x7fa5100ead00 2024-03-04T09:25:26.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.776+0000 7f6927400640 1 -- 172.21.15.6:0/558902544 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f692001e070 con 0x7f6930071050 2024-03-04T09:25:26.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.776+0000 7f009f400640 1 -- 172.21.15.6:0/3320596365 <== mon.1 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0094057e10 con 0x7f00a00e5b80 2024-03-04T09:25:26.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.777+0000 7f6ae9400640 1 -- 172.21.15.6:0/1899106332 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ae4069510 con 0x7f6ae41504e0 2024-03-04T09:25:26.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.777+0000 7fa51e200640 1 -- 172.21.15.6:0/2576357262 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4f4005290 con 0x7fa5100ead00 2024-03-04T09:25:26.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.778+0000 7f6927400640 1 --2- 172.21.15.6:0/558902544 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f690c06b5a0 0x7f690c06da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:26.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.778+0000 7f6927400640 1 -- 172.21.15.6:0/558902544 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6920089e50 con 0x7f6930071050 2024-03-04T09:25:26.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.779+0000 7f692ea00640 1 --2- 172.21.15.6:0/558902544 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f690c06b5a0 0x7f690c06da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:26.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.780+0000 7f692ea00640 1 --2- 172.21.15.6:0/558902544 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f690c06b5a0 0x7f690c06da60 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f6918006fd0 tx=0x7f6918006bc0 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:26.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.779+0000 7fda7f400640 1 -- 172.21.15.6:0/3690164114 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda74035020 con 0x7fda80165960 2024-03-04T09:25:26.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.780+0000 7fda7f400640 1 -- 172.21.15.6:0/3690164114 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda74031640 con 0x7fda80165960 2024-03-04T09:25:26.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.780+0000 7fda7f400640 1 -- 172.21.15.6:0/3690164114 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda7402d040 con 0x7fda80165960 2024-03-04T09:25:26.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.780+0000 7faaed600640 1 -- 172.21.15.6:0/3065817601 <== mon.2 v2:172.21.15.146:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7faadc0582b0 con 0x7faaf00d4a10 2024-03-04T09:25:26.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.781+0000 7fda87600640 1 -- 172.21.15.6:0/3690164114 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fda80165960 msgr2=0x7fda80070c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:26.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.781+0000 7fda87600640 1 --2- 172.21.15.6:0/3690164114 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fda80165960 0x7fda80070c70 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fda740098a0 tx=0x7fda7402b8f0 comp rx=0 tx=0).stop 2024-03-04T09:25:26.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.781+0000 7fda87600640 1 -- 172.21.15.6:0/3690164114 shutdown_connections 2024-03-04T09:25:26.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.781+0000 7fda87600640 1 --2- 172.21.15.6:0/3690164114 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fda80165960 0x7fda80070c70 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:26.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.781+0000 7fda87600640 1 -- 172.21.15.6:0/3690164114 >> 172.21.15.6:0/3690164114 conn(0x7fda8006afb0 msgr2=0x7fda8006b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:26.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.781+0000 7fda87600640 1 -- 172.21.15.6:0/3690164114 shutdown_connections 2024-03-04T09:25:26.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.781+0000 7fda87600640 1 -- 172.21.15.6:0/3690164114 wait complete. 2024-03-04T09:25:26.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.784+0000 7fda87600640 1 Processor -- start 2024-03-04T09:25:26.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.784+0000 7fda87600640 1 -- start start 2024-03-04T09:25:26.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.785+0000 7f6927400640 1 -- 172.21.15.6:0/558902544 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6920025020 con 0x7f6930071050 2024-03-04T09:25:26.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.786+0000 7fda87600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fda8006d520 0x7fda80159ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:26.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.786+0000 7fda87600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fda8006e3d0 0x7fda8015ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:26.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.786+0000 7fda87600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fda80165960 0x7fda8014ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:26.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.786+0000 7fda87600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda80071810 con 0x7fda8006e3d0 2024-03-04T09:25:26.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.786+0000 7fda87600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda800cde70 con 0x7fda8006d520 2024-03-04T09:25:26.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.786+0000 7fda87600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda80071990 con 0x7fda80165960 2024-03-04T09:25:26.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.787+0000 7fda84c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fda8006d520 0x7fda80159ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:26.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.787+0000 7fda7fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fda8006e3d0 0x7fda8015ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:26.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.787+0000 7fda85600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fda80165960 0x7fda8014ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:26.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.787+0000 7fda85600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fda80165960 0x7fda8014ae10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.6:33414/0 (socket says 172.21.15.6:33414) 2024-03-04T09:25:26.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.787+0000 7fda85600640 1 -- 172.21.15.6:0/524534614 learned_addr learned my addr 172.21.15.6:0/524534614 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:25:26.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.788+0000 7fda7fe00640 1 -- 172.21.15.6:0/524534614 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fda80165960 msgr2=0x7fda8014ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:26.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.789+0000 7fda7fe00640 1 --2- 172.21.15.6:0/524534614 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fda80165960 0x7fda8014ae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:26.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.789+0000 7fda7fe00640 1 -- 172.21.15.6:0/524534614 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fda8006d520 msgr2=0x7fda80159ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:26.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.789+0000 7fda7fe00640 1 --2- 172.21.15.6:0/524534614 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fda8006d520 0x7fda80159ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:26.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.789+0000 7fda7fe00640 1 -- 172.21.15.6:0/524534614 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda800b3460 con 0x7fda8006e3d0 2024-03-04T09:25:26.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.789+0000 7fda84c00640 1 --2- 172.21.15.6:0/524534614 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fda8006d520 0x7fda80159ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T09:25:26.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.790+0000 7fda7fe00640 1 --2- 172.21.15.6:0/524534614 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fda8006e3d0 0x7fda8015ab20 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fda6c00c410 tx=0x7fda6c00c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:26.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.788+0000 7f6adb400640 1 -- 172.21.15.6:0/1899106332 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6adc01a070 con 0x7f6ae41504e0 2024-03-04T09:25:26.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.790+0000 7f6adb400640 1 --2- 172.21.15.6:0/1899106332 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f6ac006b4d0 0x7f6ac006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:26.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.790+0000 7f6adb400640 1 -- 172.21.15.6:0/1899106332 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6adc08a700 con 0x7f6ae41504e0 2024-03-04T09:25:26.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.792+0000 7f6ae2800640 1 --2- 172.21.15.6:0/1899106332 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f6ac006b4d0 0x7f6ac006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:26.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.792+0000 7fa514c00640 1 -- 172.21.15.6:0/2576357262 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa518093070 con 0x7fa5100ead00 2024-03-04T09:25:26.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.793+0000 7fa514c00640 1 --2- 172.21.15.6:0/2576357262 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa4f806b4d0 0x7fa4f806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:26.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.794+0000 7f6ae2800640 1 --2- 172.21.15.6:0/1899106332 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f6ac006b4d0 0x7f6ac006d990 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f6ad4036000 tx=0x7f6ad4008030 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:26.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.794+0000 7fa514c00640 1 -- 172.21.15.6:0/2576357262 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa518104be0 con 0x7fa5100ead00 2024-03-04T09:25:26.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.794+0000 7fa51ce00640 1 --2- 172.21.15.6:0/2576357262 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa4f806b4d0 0x7fa4f806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:26.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.795+0000 7fa51ce00640 1 --2- 172.21.15.6:0/2576357262 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa4f806b4d0 0x7fa4f806d990 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fa50c005880 tx=0x7fa50c009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:26.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.797+0000 7fa514c00640 1 -- 172.21.15.6:0/2576357262 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa5180d3480 con 0x7fa5100ead00 2024-03-04T09:25:26.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.805+0000 7fda7cc00640 1 -- 172.21.15.6:0/524534614 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda6c017070 con 0x7fda8006e3d0 2024-03-04T09:25:26.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.806+0000 7fda7cc00640 1 -- 172.21.15.6:0/524534614 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda6c0035f0 con 0x7fda8006e3d0 2024-03-04T09:25:26.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.806+0000 7fda7cc00640 1 -- 172.21.15.6:0/524534614 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda6c0039e0 con 0x7fda8006e3d0 2024-03-04T09:25:26.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.807+0000 7fda87600640 1 -- 172.21.15.6:0/524534614 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda800b3740 con 0x7fda8006e3d0 2024-03-04T09:25:26.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.808+0000 7fda87600640 1 -- 172.21.15.6:0/524534614 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda800b3d00 con 0x7fda8006e3d0 2024-03-04T09:25:26.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.809+0000 7fda87600640 1 -- 172.21.15.6:0/524534614 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda8004f3a0 con 0x7fda8006e3d0 2024-03-04T09:25:26.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.809+0000 7f6adb400640 1 -- 172.21.15.6:0/1899106332 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6adc058f10 con 0x7f6ae41504e0 2024-03-04T09:25:26.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.812+0000 7fda7cc00640 1 -- 172.21.15.6:0/524534614 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fda6c003b80 con 0x7fda8006e3d0 2024-03-04T09:25:26.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.813+0000 7fda7cc00640 1 --2- 172.21.15.6:0/524534614 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fda6006b4d0 0x7fda6006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:26.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.813+0000 7fda7cc00640 1 -- 172.21.15.6:0/524534614 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fda6c0893f0 con 0x7fda8006e3d0 2024-03-04T09:25:26.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.817+0000 7fda84c00640 1 --2- 172.21.15.6:0/524534614 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fda6006b4d0 0x7fda6006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:26.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.818+0000 7fda84c00640 1 --2- 172.21.15.6:0/524534614 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fda6006b4d0 0x7fda6006d990 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fda74036000 tx=0x7fda7403c040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:26.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:26.822+0000 7fda7cc00640 1 -- 172.21.15.6:0/524534614 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fda6c057cd0 con 0x7fda8006e3d0 2024-03-04T09:25:27.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:27.176+0000 7f2419000640 1 -- 172.21.15.6:0/2724195700 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f23ec005580 con 0x7f240c0589c0 2024-03-04T09:25:27.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:27.177+0000 7f240a000640 1 -- 172.21.15.6:0/2724195700 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f240005cd90 con 0x7f240c0589c0 2024-03-04T09:25:27.174 INFO:teuthology.orchestra.run.smithi006.stdout:55834574853 2024-03-04T09:25:27.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:27.180+0000 7f2419000640 1 -- 172.21.15.6:0/2724195700 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f23f406b5a0 msgr2=0x7f23f406da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:27.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:27.181+0000 7f2419000640 1 --2- 172.21.15.6:0/2724195700 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f23f406b5a0 0x7f23f406da60 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f24040011e0 tx=0x7f2404007d30 comp rx=0 tx=0).stop 2024-03-04T09:25:27.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:27.181+0000 7f2419000640 1 -- 172.21.15.6:0/2724195700 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f240c0589c0 msgr2=0x7f240c0d72b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:27.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:27.181+0000 7f2419000640 1 --2- 172.21.15.6:0/2724195700 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f240c0589c0 0x7f240c0d72b0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f240000e9b0 tx=0x7f240000ee70 comp rx=0 tx=0).stop 2024-03-04T09:25:27.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:27.181+0000 7f2419000640 1 -- 172.21.15.6:0/2724195700 shutdown_connections 2024-03-04T09:25:27.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:27.181+0000 7f2419000640 1 --2- 172.21.15.6:0/2724195700 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f240c0e75f0 0x7f240c00d250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:27.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:27.181+0000 7f2419000640 1 --2- 172.21.15.6:0/2724195700 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f240c0589c0 0x7f240c0d72b0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:27.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:27.181+0000 7f2419000640 1 --2- 172.21.15.6:0/2724195700 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f23f406b5a0 0x7f23f406da60 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:27.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:27.181+0000 7f2419000640 1 --2- 172.21.15.6:0/2724195700 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f240c053c40 0x7f240c009d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:27.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:27.181+0000 7f2419000640 1 -- 172.21.15.6:0/2724195700 >> 172.21.15.6:0/2724195700 conn(0x7f240c058ee0 msgr2=0x7f240c039bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:27.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:27.182+0000 7f2419000640 1 -- 172.21.15.6:0/2724195700 shutdown_connections 2024-03-04T09:25:27.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:27.182+0000 7f2419000640 1 -- 172.21.15.6:0/2724195700 wait complete. 2024-03-04T09:25:27.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:27.206+0000 7f4a44a00640 1 -- 172.21.15.6:0/537460095 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f4a38054410 con 0x7f4a380361b0 2024-03-04T09:25:27.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:27.206+0000 7f00a7e00640 1 -- 172.21.15.6:0/3320596365 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f00a003e5c0 con 0x7f00a00e5b80 2024-03-04T09:25:27.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:27.206+0000 7f4a36a00640 1 -- 172.21.15.6:0/537460095 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f4a30058760 con 0x7f4a380361b0 2024-03-04T09:25:27.203 INFO:teuthology.orchestra.run.smithi006.stdout:55834574852 2024-03-04T09:25:27.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:27.207+0000 7f009f400640 1 -- 172.21.15.6:0/3320596365 <== mon.1 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f009405beb0 con 0x7f00a00e5b80 2024-03-04T09:25:27.205 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.2 2024-03-04T09:25:27.205 INFO:teuthology.orchestra.run.smithi006.stdout:55834574854 2024-03-04T09:25:27.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:27.210+0000 7f4a44a00640 1 -- 172.21.15.6:0/537460095 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f4a2006b520 msgr2=0x7f4a2006d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:27.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:27.210+0000 7f4a44a00640 1 --2- 172.21.15.6:0/537460095 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f4a2006b520 0x7f4a2006d9e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f4a380569a0 tx=0x7f4a2c009040 comp rx=0 tx=0).stop 2024-03-04T09:25:27.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:27.210+0000 7f4a44a00640 1 -- 172.21.15.6:0/537460095 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4a380361b0 msgr2=0x7f4a3817bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:27.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:27.210+0000 7f4a44a00640 1 --2- 172.21.15.6:0/537460095 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4a380361b0 0x7f4a3817bfa0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f4a3000e810 tx=0x7f4a3000ecd0 comp rx=0 tx=0).stop 2024-03-04T09:25:27.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:27.210+0000 7f4a44a00640 1 -- 172.21.15.6:0/537460095 shutdown_connections 2024-03-04T09:25:27.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:27.210+0000 7f4a44a00640 1 --2- 172.21.15.6:0/537460095 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4a380529f0 0x7f4a3804d180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:27.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:27.210+0000 7f4a44a00640 1 --2- 172.21.15.6:0/537460095 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f4a2006b520 0x7f4a2006d9e0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:27.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:27.210+0000 7f4a44a00640 1 --2- 172.21.15.6:0/537460095 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4a3804a320 0x7f4a3817a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:27.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:27.210+0000 7f4a44a00640 1 --2- 172.21.15.6:0/537460095 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4a380361b0 0x7f4a3817bfa0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:27.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:27.210+0000 7f4a44a00640 1 -- 172.21.15.6:0/537460095 >> 172.21.15.6:0/537460095 conn(0x7f4a380537b0 msgr2=0x7f4a3803c1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:27.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:27.211+0000 7f00a7e00640 1 -- 172.21.15.6:0/3320596365 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f008806b4d0 msgr2=0x7f008806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:27.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:27.211+0000 7f00a7e00640 1 --2- 172.21.15.6:0/3320596365 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f008806b4d0 0x7f008806d990 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f00a806a870 tx=0x7f00a808d040 comp rx=0 tx=0).stop 2024-03-04T09:25:27.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:27.211+0000 7f00a7e00640 1 -- 172.21.15.6:0/3320596365 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f00a00e5b80 msgr2=0x7f00a00e6010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:27.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:27.211+0000 7f00a7e00640 1 --2- 172.21.15.6:0/3320596365 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f00a00e5b80 0x7f00a00e6010 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f009400c3c0 tx=0x7f009400c880 comp rx=0 tx=0).stop 2024-03-04T09:25:27.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:27.211+0000 7f4a44a00640 1 -- 172.21.15.6:0/537460095 shutdown_connections 2024-03-04T09:25:27.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:27.211+0000 7f00a7e00640 1 -- 172.21.15.6:0/3320596365 shutdown_connections 2024-03-04T09:25:27.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:27.211+0000 7f00a7e00640 1 --2- 172.21.15.6:0/3320596365 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f00a00e5b80 0x7f00a00e6010 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:27.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:27.211+0000 7f00a7e00640 1 --2- 172.21.15.6:0/3320596365 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f008806b4d0 0x7f008806d990 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:27.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:27.211+0000 7f00a7e00640 1 --2- 172.21.15.6:0/3320596365 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f00a00e3300 0x7f00a00d9300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:27.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:27.211+0000 7f00a7e00640 1 --2- 172.21.15.6:0/3320596365 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f00a0054110 0x7f00a00ddf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:27.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:27.211+0000 7f00a7e00640 1 -- 172.21.15.6:0/3320596365 >> 172.21.15.6:0/3320596365 conn(0x7f00a0058ef0 msgr2=0x7f00a0005b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:27.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:27.211+0000 7f4a44a00640 1 -- 172.21.15.6:0/537460095 wait complete. 2024-03-04T09:25:27.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:27.211+0000 7f00a7e00640 1 -- 172.21.15.6:0/3320596365 shutdown_connections 2024-03-04T09:25:27.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:27.211+0000 7f00a7e00640 1 -- 172.21.15.6:0/3320596365 wait complete. 2024-03-04T09:25:27.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:27.222+0000 7faaf7000640 1 -- 172.21.15.6:0/3065817601 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7faacc005850 con 0x7faaf00d4a10 2024-03-04T09:25:27.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:27.223+0000 7faaed600640 1 -- 172.21.15.6:0/3065817601 <== mon.2 v2:172.21.15.146:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7faadc05c350 con 0x7faaf00d4a10 2024-03-04T09:25:27.220 INFO:teuthology.orchestra.run.smithi006.stdout:55834574853 2024-03-04T09:25:27.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:27.225+0000 7f6936600640 1 -- 172.21.15.6:0/558902544 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f693004f3a0 con 0x7f6930071050 2024-03-04T09:25:27.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:27.226+0000 7fa51e200640 1 -- 172.21.15.6:0/2576357262 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fa4f4005850 con 0x7fa5100ead00 2024-03-04T09:25:27.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:27.226+0000 7f6927400640 1 -- 172.21.15.6:0/558902544 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f6920058d40 con 0x7f6930071050 2024-03-04T09:25:27.223 INFO:teuthology.orchestra.run.smithi006.stdout:55834574853 2024-03-04T09:25:27.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:27.226+0000 7fa514c00640 1 -- 172.21.15.6:0/2576357262 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fa5180d7520 con 0x7fa5100ead00 2024-03-04T09:25:27.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:27.227+0000 7faaf7000640 1 -- 172.21.15.6:0/3065817601 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7faad006b5a0 msgr2=0x7faad006da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:27.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:27.227+0000 7faaf7000640 1 --2- 172.21.15.6:0/3065817601 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7faad006b5a0 0x7faad006da60 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7faae4036000 tx=0x7faae403c040 comp rx=0 tx=0).stop 2024-03-04T09:25:27.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:27.227+0000 7faaf7000640 1 -- 172.21.15.6:0/3065817601 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faaf00d4a10 msgr2=0x7faaf00b6950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:27.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:27.227+0000 7faaf7000640 1 --2- 172.21.15.6:0/3065817601 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faaf00d4a10 0x7faaf00b6950 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7faadc00c400 tx=0x7faadc00c8c0 comp rx=0 tx=0).stop 2024-03-04T09:25:27.224 INFO:teuthology.orchestra.run.smithi006.stdout:55834574855 2024-03-04T09:25:27.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:27.227+0000 7faaf7000640 1 -- 172.21.15.6:0/3065817601 shutdown_connections 2024-03-04T09:25:27.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:27.227+0000 7faaf7000640 1 --2- 172.21.15.6:0/3065817601 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faaf015e0b0 0x7faaf015c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:27.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:27.227+0000 7faaf7000640 1 --2- 172.21.15.6:0/3065817601 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7faad006b5a0 0x7faad006da60 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:27.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:27.227+0000 7faaf7000640 1 --2- 172.21.15.6:0/3065817601 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faaf00d4a10 0x7faaf00b6950 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:27.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:27.227+0000 7faaf7000640 1 --2- 172.21.15.6:0/3065817601 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faaf00d0e50 0x7faaf00b8240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:27.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:27.227+0000 7faaf7000640 1 -- 172.21.15.6:0/3065817601 >> 172.21.15.6:0/3065817601 conn(0x7faaf00d3d30 msgr2=0x7faaf00b5bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:27.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:27.227+0000 7faaf7000640 1 -- 172.21.15.6:0/3065817601 shutdown_connections 2024-03-04T09:25:27.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:27.227+0000 7faaf7000640 1 -- 172.21.15.6:0/3065817601 wait complete. 2024-03-04T09:25:27.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:27.230+0000 7f6936600640 1 -- 172.21.15.6:0/558902544 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f690c06b5a0 msgr2=0x7f690c06da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:27.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:27.230+0000 7f6936600640 1 --2- 172.21.15.6:0/558902544 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f690c06b5a0 0x7f690c06da60 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f6918006fd0 tx=0x7f6918006bc0 comp rx=0 tx=0).stop 2024-03-04T09:25:27.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:27.229+0000 7fa51e200640 1 -- 172.21.15.6:0/2576357262 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa4f806b4d0 msgr2=0x7fa4f806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:27.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:27.229+0000 7fa51e200640 1 --2- 172.21.15.6:0/2576357262 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa4f806b4d0 0x7fa4f806d990 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fa50c005880 tx=0x7fa50c009040 comp rx=0 tx=0).stop 2024-03-04T09:25:27.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:27.229+0000 7fa51e200640 1 -- 172.21.15.6:0/2576357262 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa5100ead00 msgr2=0x7fa51004a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:27.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:27.229+0000 7fa51e200640 1 --2- 172.21.15.6:0/2576357262 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa5100ead00 0x7fa51004a920 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fa518069a10 tx=0x7fa518069430 comp rx=0 tx=0).stop 2024-03-04T09:25:27.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:27.229+0000 7fa51e200640 1 -- 172.21.15.6:0/2576357262 shutdown_connections 2024-03-04T09:25:27.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:27.229+0000 7fa51e200640 1 --2- 172.21.15.6:0/2576357262 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa5100ead00 0x7fa51004a920 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:27.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:27.229+0000 7fa51e200640 1 --2- 172.21.15.6:0/2576357262 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa4f806b4d0 0x7fa4f806d990 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:27.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:27.229+0000 7fa51e200640 1 --2- 172.21.15.6:0/2576357262 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa510006860 0x7fa510035e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:27.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:27.230+0000 7fa51e200640 1 --2- 172.21.15.6:0/2576357262 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa510005ee0 0x7fa5100dc0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:27.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:27.230+0000 7fa51e200640 1 -- 172.21.15.6:0/2576357262 >> 172.21.15.6:0/2576357262 conn(0x7fa510009fb0 msgr2=0x7fa51004f9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:27.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:27.230+0000 7fa51e200640 1 -- 172.21.15.6:0/2576357262 shutdown_connections 2024-03-04T09:25:27.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:27.230+0000 7fa51e200640 1 -- 172.21.15.6:0/2576357262 wait complete. 2024-03-04T09:25:27.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:27.230+0000 7f6936600640 1 -- 172.21.15.6:0/558902544 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6930071050 msgr2=0x7f693009aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:27.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:27.230+0000 7f6936600640 1 --2- 172.21.15.6:0/558902544 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6930071050 0x7f693009aa10 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f692000bed0 tx=0x7f692000f480 comp rx=0 tx=0).stop 2024-03-04T09:25:27.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:27.230+0000 7f6936600640 1 -- 172.21.15.6:0/558902544 shutdown_connections 2024-03-04T09:25:27.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:27.230+0000 7f6936600640 1 --2- 172.21.15.6:0/558902544 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f69300cf5c0 0x7f6930155ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:27.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:27.230+0000 7f6936600640 1 --2- 172.21.15.6:0/558902544 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f690c06b5a0 0x7f690c06da60 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:27.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:27.230+0000 7f6936600640 1 --2- 172.21.15.6:0/558902544 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f69300b3c90 0x7f6930151d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:27.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:27.230+0000 7f6936600640 1 --2- 172.21.15.6:0/558902544 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6930071050 0x7f693009aa10 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:27.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:27.230+0000 7f6936600640 1 -- 172.21.15.6:0/558902544 >> 172.21.15.6:0/558902544 conn(0x7f693006afb0 msgr2=0x7f69300acf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:27.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:27.230+0000 7f6936600640 1 -- 172.21.15.6:0/558902544 shutdown_connections 2024-03-04T09:25:27.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:27.231+0000 7f6936600640 1 -- 172.21.15.6:0/558902544 wait complete. 2024-03-04T09:25:27.235 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574852 for osd.4 2024-03-04T09:25:27.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:27.241+0000 7f6ae9400640 1 -- 172.21.15.6:0/1899106332 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f6ae404f3a0 con 0x7f6ae41504e0 2024-03-04T09:25:27.239 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574854 for osd.1 2024-03-04T09:25:27.239 INFO:teuthology.orchestra.run.smithi006.stdout:55834574854 2024-03-04T09:25:27.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:27.242+0000 7f6adb400640 1 -- 172.21.15.6:0/1899106332 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f6adc05cfb0 con 0x7f6ae41504e0 2024-03-04T09:25:27.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:27.245+0000 7f6ae9400640 1 -- 172.21.15.6:0/1899106332 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f6ac006b4d0 msgr2=0x7f6ac006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:27.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:27.245+0000 7f6ae9400640 1 --2- 172.21.15.6:0/1899106332 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f6ac006b4d0 0x7f6ac006d990 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f6ad4036000 tx=0x7f6ad4008030 comp rx=0 tx=0).stop 2024-03-04T09:25:27.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:27.245+0000 7f6ae9400640 1 -- 172.21.15.6:0/1899106332 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6ae41504e0 msgr2=0x7f6ae414f950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:27.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:27.245+0000 7f6ae9400640 1 --2- 172.21.15.6:0/1899106332 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6ae41504e0 0x7f6ae414f950 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f6adc00caa0 tx=0x7f6adc00cf60 comp rx=0 tx=0).stop 2024-03-04T09:25:27.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:27.246+0000 7f6ae9400640 1 -- 172.21.15.6:0/1899106332 shutdown_connections 2024-03-04T09:25:27.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:27.246+0000 7f6ae9400640 1 --2- 172.21.15.6:0/1899106332 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6ae41504e0 0x7f6ae414f950 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:27.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:27.246+0000 7f6ae9400640 1 --2- 172.21.15.6:0/1899106332 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6ae4072da0 0x7f6ae409ac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:27.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:27.246+0000 7f6ae9400640 1 --2- 172.21.15.6:0/1899106332 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f6ac006b4d0 0x7f6ac006d990 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:27.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:27.246+0000 7f6ae9400640 1 --2- 172.21.15.6:0/1899106332 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6ae40712c0 0x7f6ae415e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:27.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:27.246+0000 7f6ae9400640 1 -- 172.21.15.6:0/1899106332 >> 172.21.15.6:0/1899106332 conn(0x7f6ae406afb0 msgr2=0x7f6ae40afe90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:27.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:27.246+0000 7f6ae9400640 1 -- 172.21.15.6:0/1899106332 shutdown_connections 2024-03-04T09:25:27.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:27.246+0000 7f6ae9400640 1 -- 172.21.15.6:0/1899106332 wait complete. 2024-03-04T09:25:27.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:27.247+0000 7fda87600640 1 -- 172.21.15.6:0/524534614 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7fda8014bbf0 con 0x7fda8006e3d0 2024-03-04T09:25:27.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:27.247+0000 7fda7cc00640 1 -- 172.21.15.6:0/524534614 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fda6c05bd70 con 0x7fda8006e3d0 2024-03-04T09:25:27.244 INFO:teuthology.orchestra.run.smithi006.stdout:55834574853 2024-03-04T09:25:27.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:27.249+0000 7fda87600640 1 -- 172.21.15.6:0/524534614 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fda6006b4d0 msgr2=0x7fda6006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:27.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:27.249+0000 7fda87600640 1 --2- 172.21.15.6:0/524534614 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fda6006b4d0 0x7fda6006d990 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fda74036000 tx=0x7fda7403c040 comp rx=0 tx=0).stop 2024-03-04T09:25:27.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:27.249+0000 7fda87600640 1 -- 172.21.15.6:0/524534614 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fda8006e3d0 msgr2=0x7fda8015ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:27.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:27.249+0000 7fda87600640 1 --2- 172.21.15.6:0/524534614 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fda8006e3d0 0x7fda8015ab20 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fda6c00c410 tx=0x7fda6c00c810 comp rx=0 tx=0).stop 2024-03-04T09:25:27.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:27.250+0000 7fda87600640 1 -- 172.21.15.6:0/524534614 shutdown_connections 2024-03-04T09:25:27.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:27.250+0000 7fda87600640 1 --2- 172.21.15.6:0/524534614 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fda6006b4d0 0x7fda6006d990 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:27.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:27.250+0000 7fda87600640 1 --2- 172.21.15.6:0/524534614 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fda80165960 0x7fda8014ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:27.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:27.250+0000 7fda87600640 1 --2- 172.21.15.6:0/524534614 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fda8006e3d0 0x7fda8015ab20 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:27.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:27.250+0000 7fda87600640 1 --2- 172.21.15.6:0/524534614 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fda8006d520 0x7fda80159ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:27.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:27.250+0000 7fda87600640 1 -- 172.21.15.6:0/524534614 >> 172.21.15.6:0/524534614 conn(0x7fda8006afb0 msgr2=0x7fda800cf080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:27.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:27.250+0000 7fda87600640 1 -- 172.21.15.6:0/524534614 shutdown_connections 2024-03-04T09:25:27.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:27.250+0000 7fda87600640 1 -- 172.21.15.6:0/524534614 wait complete. 2024-03-04T09:25:27.248 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.6 2024-03-04T09:25:27.251 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.0 2024-03-04T09:25:27.252 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574855 for osd.5 2024-03-04T09:25:27.264 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574854 for osd.3 2024-03-04T09:25:27.265 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.7 2024-03-04T09:25:28.205 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-03-04T09:25:28.236 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-03-04T09:25:28.241 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-03-04T09:25:28.249 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-03-04T09:25:28.252 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-03-04T09:25:28.252 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-03-04T09:25:28.265 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-03-04T09:25:28.265 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-03-04T09:25:28.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.504+0000 7f1de8000640 1 Processor -- start 2024-03-04T09:25:28.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.504+0000 7f1de8000640 1 -- start start 2024-03-04T09:25:28.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.506+0000 7f1de8000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1de0140640 0x7f1de0071480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:28.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.507+0000 7f1de5600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1de0140640 0x7f1de0071480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:28.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.508+0000 7f1de5600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1de0140640 0x7f1de0071480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.6:33420/0 (socket says 172.21.15.6:33420) 2024-03-04T09:25:28.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.508+0000 7f1de5600640 1 -- 172.21.15.6:0/1311484165 learned_addr learned my addr 172.21.15.6:0/1311484165 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:25:28.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.508+0000 7f1de8000640 1 -- 172.21.15.6:0/1311484165 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1de00ce750 con 0x7f1de006de00 2024-03-04T09:25:28.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.509+0000 7f1de8000640 1 -- 172.21.15.6:0/1311484165 --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1de015de30 con 0x7f1de00cca20 2024-03-04T09:25:28.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.509+0000 7f1de8000640 1 -- 172.21.15.6:0/1311484165 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1de015e110 con 0x7f1de0140640 2024-03-04T09:25:28.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.510+0000 7f1ddfe00640 1 -- 172.21.15.6:0/1311484165 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 77263603 0 0) 0x7f1de015de30 con 0x7f1de00cca20 2024-03-04T09:25:28.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.510+0000 7f1ddfe00640 1 -- 172.21.15.6:0/1311484165 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1dc8003520 con 0x7f1de00cca20 2024-03-04T09:25:28.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.511+0000 7f1ddfe00640 1 -- 172.21.15.6:0/1311484165 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 721222206 0 0) 0x7f1dc8003520 con 0x7f1de00cca20 2024-03-04T09:25:28.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.511+0000 7f1de5600640 1 -- 172.21.15.6:0/1311484165 >> v1:172.21.15.146:6789/0 conn(0x7f1de00cca20 legacy=0x7f1de00cd930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:28.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.511+0000 7f1de5600640 1 -- 172.21.15.6:0/1311484165 >> v1:172.21.15.6:6789/0 conn(0x7f1de006de00 legacy=0x7f1de00ce040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:28.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.511+0000 7f1de5600640 1 -- 172.21.15.6:0/1311484165 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1de015e3f0 con 0x7f1de0140640 2024-03-04T09:25:28.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.513+0000 7f1de5600640 1 --2- 172.21.15.6:0/1311484165 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1de0140640 0x7f1de0071480 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f1dd4007390 tx=0x7f1dd402c860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4219a4463a7a98b server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:28.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.514+0000 7f1ddfe00640 1 -- 172.21.15.6:0/1311484165 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dd4039070 con 0x7f1de0140640 2024-03-04T09:25:28.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.514+0000 7f1ddfe00640 1 -- 172.21.15.6:0/1311484165 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1dd4008bf0 con 0x7f1de0140640 2024-03-04T09:25:28.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.516+0000 7f1ddfe00640 1 -- 172.21.15.6:0/1311484165 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dd4030050 con 0x7f1de0140640 2024-03-04T09:25:28.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.517+0000 7f1de8000640 1 -- 172.21.15.6:0/1311484165 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1de0140640 msgr2=0x7f1de0071480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:28.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.517+0000 7f1de8000640 1 --2- 172.21.15.6:0/1311484165 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1de0140640 0x7f1de0071480 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f1dd4007390 tx=0x7f1dd402c860 comp rx=0 tx=0).stop 2024-03-04T09:25:28.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.517+0000 7f1de8000640 1 -- 172.21.15.6:0/1311484165 shutdown_connections 2024-03-04T09:25:28.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.517+0000 7f1de8000640 1 --2- 172.21.15.6:0/1311484165 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1de0140640 0x7f1de0071480 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:28.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.517+0000 7f1de8000640 1 -- 172.21.15.6:0/1311484165 >> 172.21.15.6:0/1311484165 conn(0x7f1de006afb0 msgr2=0x7f1de006b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:28.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.517+0000 7f1de8000640 1 -- 172.21.15.6:0/1311484165 shutdown_connections 2024-03-04T09:25:28.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.517+0000 7f1de8000640 1 -- 172.21.15.6:0/1311484165 wait complete. 2024-03-04T09:25:28.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.520+0000 7f1de8000640 1 Processor -- start 2024-03-04T09:25:28.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.520+0000 7f1de8000640 1 -- start start 2024-03-04T09:25:28.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.521+0000 7f1de8000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1de006de00 0x7f1de013f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:28.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.521+0000 7f1de8000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1de00cca20 0x7f1de0143110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:28.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.521+0000 7f1de8000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1de0140640 0x7f1de015c240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:28.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.522+0000 7f1de4c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1de00cca20 0x7f1de0143110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:28.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.522+0000 7f1de4c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1de00cca20 0x7f1de0143110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.6:47310/0 (socket says 172.21.15.6:47310) 2024-03-04T09:25:28.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.522+0000 7f1de5600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1de006de00 0x7f1de013f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:28.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.522+0000 7f1de5600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1de006de00 0x7f1de013f1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45388/0 (socket says 172.21.15.6:45388) 2024-03-04T09:25:28.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.523+0000 7f1de4c00640 1 -- 172.21.15.6:0/2776489773 learned_addr learned my addr 172.21.15.6:0/2776489773 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:25:28.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.523+0000 7f1de8000640 1 -- 172.21.15.6:0/2776489773 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1de015dbf0 con 0x7f1de006de00 2024-03-04T09:25:28.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.523+0000 7f1de8000640 1 -- 172.21.15.6:0/2776489773 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1de015da70 con 0x7f1de00cca20 2024-03-04T09:25:28.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.523+0000 7f1de8000640 1 -- 172.21.15.6:0/2776489773 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1de0070e90 con 0x7f1de0140640 2024-03-04T09:25:28.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.523+0000 7f1de6000640 1 --2- 172.21.15.6:0/2776489773 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1de0140640 0x7f1de015c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:28.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.524+0000 7f1de4c00640 1 -- 172.21.15.6:0/2776489773 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1de0140640 msgr2=0x7f1de015c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:28.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.524+0000 7f1de4c00640 1 --2- 172.21.15.6:0/2776489773 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1de0140640 0x7f1de015c240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:28.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.524+0000 7f1de4c00640 1 -- 172.21.15.6:0/2776489773 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1de006de00 msgr2=0x7f1de013f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:28.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.524+0000 7f1de4c00640 1 --2- 172.21.15.6:0/2776489773 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1de006de00 0x7f1de013f1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:28.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.524+0000 7f1de4c00640 1 -- 172.21.15.6:0/2776489773 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1de015a740 con 0x7f1de00cca20 2024-03-04T09:25:28.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.525+0000 7f1de4c00640 1 --2- 172.21.15.6:0/2776489773 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1de00cca20 0x7f1de0143110 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f1dcc00c7b0 tx=0x7f1dcc00cc70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:28.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.525+0000 7f1ddd600640 1 -- 172.21.15.6:0/2776489773 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dcc016020 con 0x7f1de00cca20 2024-03-04T09:25:28.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.526+0000 7f1de8000640 1 -- 172.21.15.6:0/2776489773 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1de015aa20 con 0x7f1de00cca20 2024-03-04T09:25:28.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.526+0000 7f1de8000640 1 -- 172.21.15.6:0/2776489773 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1de015afe0 con 0x7f1de00cca20 2024-03-04T09:25:28.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.527+0000 7f1ddd600640 1 -- 172.21.15.6:0/2776489773 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1dcc00aa50 con 0x7f1de00cca20 2024-03-04T09:25:28.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.527+0000 7f1dd2a00640 1 -- 172.21.15.6:0/2776489773 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1dbc005290 con 0x7f1de00cca20 2024-03-04T09:25:28.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.528+0000 7f1ddd600640 1 -- 172.21.15.6:0/2776489773 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dcc016020 con 0x7f1de00cca20 2024-03-04T09:25:28.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.532+0000 7f1ddd600640 1 -- 172.21.15.6:0/2776489773 <== mon.1 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1dcc019020 con 0x7f1de00cca20 2024-03-04T09:25:28.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.534+0000 7f1ddd600640 1 --2- 172.21.15.6:0/2776489773 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f1dc006b900 0x7f1dc006ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:28.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.534+0000 7f1ddd600640 1 -- 172.21.15.6:0/2776489773 <== mon.1 v2:172.21.15.146:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1dcc00a850 con 0x7f1de00cca20 2024-03-04T09:25:28.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.535+0000 7f1de5600640 1 --2- 172.21.15.6:0/2776489773 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f1dc006b900 0x7f1dc006ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:28.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.540+0000 7f1de5600640 1 --2- 172.21.15.6:0/2776489773 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f1dc006b900 0x7f1dc006ddc0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f1dd4036000 tx=0x7f1dd400a000 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:28.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.550+0000 7fe49e200640 1 Processor -- start 2024-03-04T09:25:28.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.551+0000 7fe49e200640 1 -- start start 2024-03-04T09:25:28.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.552+0000 7fe49e200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe49815e2e0 0x7fe4980d0830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:28.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.552+0000 7fe49e200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fe4980d0d70 0x7fe4980d11d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:28.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.552+0000 7fe49e200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fe4980d1b30 0x7fe4980d1fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:28.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.552+0000 7fe49e200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4980d4e70 con 0x7fe4980d1b30 2024-03-04T09:25:28.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.553+0000 7fe49e200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4980b3c40 con 0x7fe4980d0d70 2024-03-04T09:25:28.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.553+0000 7fe49e200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4980b3940 con 0x7fe49815e2e0 2024-03-04T09:25:28.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.553+0000 7fe497400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe49815e2e0 0x7fe4980d0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:28.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.553+0000 7fe497400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe49815e2e0 0x7fe4980d0830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.6:33438/0 (socket says 172.21.15.6:33438) 2024-03-04T09:25:28.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.553+0000 7fe497400640 1 -- 172.21.15.6:0/1614251160 learned_addr learned my addr 172.21.15.6:0/1614251160 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:25:28.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.554+0000 7fe496a00640 1 --2- 172.21.15.6:0/1614251160 >> v2:172.21.15.146:3300/0 conn(0x7fe4980d0d70 0x7fe4980d11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:28.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.554+0000 7fe497e00640 1 --2- 172.21.15.6:0/1614251160 >> v2:172.21.15.6:3300/0 conn(0x7fe4980d1b30 0x7fe4980d1fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:28.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.555+0000 7f1ddd600640 1 -- 172.21.15.6:0/2776489773 <== mon.1 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1dcc020020 con 0x7f1de00cca20 2024-03-04T09:25:28.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.555+0000 7fea3e800640 1 Processor -- start 2024-03-04T09:25:28.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.554+0000 7fe497400640 1 -- 172.21.15.6:0/1614251160 >> v2:172.21.15.146:3300/0 conn(0x7fe4980d0d70 msgr2=0x7fe4980d11d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:28.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.554+0000 7fe497400640 1 --2- 172.21.15.6:0/1614251160 >> v2:172.21.15.146:3300/0 conn(0x7fe4980d0d70 0x7fe4980d11d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:28.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.554+0000 7fe497400640 1 -- 172.21.15.6:0/1614251160 >> v2:172.21.15.6:3300/0 conn(0x7fe4980d1b30 msgr2=0x7fe4980d1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:28.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.554+0000 7fe497400640 1 --2- 172.21.15.6:0/1614251160 >> v2:172.21.15.6:3300/0 conn(0x7fe4980d1b30 0x7fe4980d1fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:28.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.554+0000 7fe497400640 1 -- 172.21.15.6:0/1614251160 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4980cf580 con 0x7fe49815e2e0 2024-03-04T09:25:28.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.555+0000 7fea3e800640 1 -- start start 2024-03-04T09:25:28.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.557+0000 7fea3e800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fea300e3300 0x7fea300572f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:28.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.557+0000 7fe497400640 1 --2- 172.21.15.6:0/1614251160 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe49815e2e0 0x7fe4980d0830 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fe4980b6b50 tx=0x7fe48002b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a540ae86de03fd13 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:28.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.557+0000 7fea3e800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fea30053750 0x7fea30053bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:28.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.557+0000 7fea3e800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fea30054110 0x7fea300061f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:28.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.557+0000 7fea3e800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea300e36e0 con 0x7fea30054110 2024-03-04T09:25:28.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.557+0000 7fea3e800640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea30054560 con 0x7fea30053750 2024-03-04T09:25:28.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.557+0000 7fea3e800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea30057b40 con 0x7fea300e3300 2024-03-04T09:25:28.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.558+0000 7fea3d400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fea300e3300 0x7fea300572f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:28.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.559+0000 7fea3de00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fea30054110 0x7fea300061f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:28.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.559+0000 7fea3d400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fea300e3300 0x7fea300572f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.6:33444/0 (socket says 172.21.15.6:33444) 2024-03-04T09:25:28.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.559+0000 7fea3d400640 1 -- 172.21.15.6:0/1494139331 learned_addr learned my addr 172.21.15.6:0/1494139331 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:25:28.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.559+0000 7fea3ca00640 1 --2- 172.21.15.6:0/1494139331 >> v2:172.21.15.146:3300/0 conn(0x7fea30053750 0x7fea30053bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:28.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.559+0000 7fea3de00640 1 -- 172.21.15.6:0/1494139331 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fea300e3300 msgr2=0x7fea300572f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:28.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.559+0000 7fb706000640 1 Processor -- start 2024-03-04T09:25:28.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.559+0000 7fea3de00640 1 --2- 172.21.15.6:0/1494139331 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fea300e3300 0x7fea300572f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:28.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.559+0000 7fea3de00640 1 -- 172.21.15.6:0/1494139331 >> v2:172.21.15.146:3300/0 conn(0x7fea30053750 msgr2=0x7fea30053bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:28.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.559+0000 7fea3de00640 1 --2- 172.21.15.6:0/1494139331 >> v2:172.21.15.146:3300/0 conn(0x7fea30053750 0x7fea30053bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:28.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.559+0000 7fea3de00640 1 -- 172.21.15.6:0/1494139331 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea30057e20 con 0x7fea30054110 2024-03-04T09:25:28.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.560+0000 7fb706000640 1 -- start start 2024-03-04T09:25:28.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.560+0000 7f9334a00640 1 Processor -- start 2024-03-04T09:25:28.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.560+0000 7f9334a00640 1 -- start start 2024-03-04T09:25:28.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.561+0000 7fea3de00640 1 --2- 172.21.15.6:0/1494139331 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fea30054110 0x7fea300061f0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fea3802d580 tx=0x7fea380a69b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68e2cd79e0b68719 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:28.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.561+0000 7fea37e00640 1 -- 172.21.15.6:0/1494139331 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea380af070 con 0x7fea30054110 2024-03-04T09:25:28.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.561+0000 7fea37e00640 1 -- 172.21.15.6:0/1494139331 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea380ab930 con 0x7fea30054110 2024-03-04T09:25:28.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.561+0000 7fea37e00640 1 -- 172.21.15.6:0/1494139331 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea380af070 con 0x7fea30054110 2024-03-04T09:25:28.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.562+0000 7fe496000640 1 -- 172.21.15.6:0/1614251160 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe480034070 con 0x7fe49815e2e0 2024-03-04T09:25:28.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.562+0000 7fe496000640 1 -- 172.21.15.6:0/1614251160 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe480030610 con 0x7fe49815e2e0 2024-03-04T09:25:28.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.562+0000 7fe496000640 1 -- 172.21.15.6:0/1614251160 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe480030a20 con 0x7fe49815e2e0 2024-03-04T09:25:28.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.562+0000 7fb706000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fb70015e200 0x7fb7000b3de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:28.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.562+0000 7fb706000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fb70006afb0 0x7fb70006b460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:28.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.562+0000 7fb706000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb70006bec0 con 0x7fb7000afdf0 2024-03-04T09:25:28.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.562+0000 7fb706000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7000b0670 con 0x7fb70015e200 2024-03-04T09:25:28.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.562+0000 7fb706000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7000ae380 con 0x7fb70006afb0 2024-03-04T09:25:28.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.563+0000 7fb6f7e00640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fb7000afdf0 0x7fb7000af140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.6:50222/0 (socket says 172.21.15.6:50222) 2024-03-04T09:25:28.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.563+0000 7fb6f7e00640 1 -- 172.21.15.6:0/3465417075 learned_addr learned my addr 172.21.15.6:0/3465417075 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:25:28.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.563+0000 7fea3e800640 1 -- 172.21.15.6:0/1494139331 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fea30054110 msgr2=0x7fea300061f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:28.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.563+0000 7fea3e800640 1 --2- 172.21.15.6:0/1494139331 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fea30054110 0x7fea300061f0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fea3802d580 tx=0x7fea380a69b0 comp rx=0 tx=0).stop 2024-03-04T09:25:28.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.563+0000 7fb6ffe00640 1 --2- 172.21.15.6:0/3465417075 >> v2:172.21.15.146:3300/0 conn(0x7fb70006afb0 0x7fb70006b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:28.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.562+0000 7f9334a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f933015e120 0x7f9330070c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:28.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.562+0000 7f9334a00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f93300ce670 0x7f93300ceb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:28.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.563+0000 7f9334a00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f933006d180 con 0x7f93300b2dd0 2024-03-04T09:25:28.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.563+0000 7f9334a00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93300cdc20 con 0x7f93300ce670 2024-03-04T09:25:28.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.563+0000 7f9334a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93300cde70 con 0x7f933015e120 2024-03-04T09:25:28.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.563+0000 7fe49e200640 1 -- 172.21.15.6:0/1614251160 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe49815e2e0 msgr2=0x7fe4980d0830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:28.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.563+0000 7fe49e200640 1 --2- 172.21.15.6:0/1614251160 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe49815e2e0 0x7fe4980d0830 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fe4980b6b50 tx=0x7fe48002b960 comp rx=0 tx=0).stop 2024-03-04T09:25:28.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.563+0000 7fea3e800640 1 -- 172.21.15.6:0/1494139331 shutdown_connections 2024-03-04T09:25:28.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.563+0000 7fea3e800640 1 --2- 172.21.15.6:0/1494139331 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fea30054110 0x7fea300061f0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:28.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.563+0000 7fea3e800640 1 --2- 172.21.15.6:0/1494139331 >> v2:172.21.15.146:3300/0 conn(0x7fea30053750 0x7fea30053bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:28.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.563+0000 7fea3e800640 1 --2- 172.21.15.6:0/1494139331 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fea300e3300 0x7fea300572f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:28.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.564+0000 7fea3e800640 1 -- 172.21.15.6:0/1494139331 >> 172.21.15.6:0/1494139331 conn(0x7fea30058ef0 msgr2=0x7fea30059300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:28.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.564+0000 7fe49e200640 1 -- 172.21.15.6:0/1614251160 shutdown_connections 2024-03-04T09:25:28.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.564+0000 7fea3e800640 1 -- 172.21.15.6:0/1494139331 shutdown_connections 2024-03-04T09:25:28.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.564+0000 7fea3e800640 1 -- 172.21.15.6:0/1494139331 wait complete. 2024-03-04T09:25:28.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.564+0000 7fe49e200640 1 --2- 172.21.15.6:0/1614251160 >> v2:172.21.15.6:3300/0 conn(0x7fe4980d1b30 0x7fe4980d1fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:28.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.564+0000 7fe49e200640 1 --2- 172.21.15.6:0/1614251160 >> v2:172.21.15.146:3300/0 conn(0x7fe4980d0d70 0x7fe4980d11d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:28.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.564+0000 7fe49e200640 1 --2- 172.21.15.6:0/1614251160 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe49815e2e0 0x7fe4980d0830 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:28.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.564+0000 7fe49e200640 1 -- 172.21.15.6:0/1614251160 >> 172.21.15.6:0/1614251160 conn(0x7fe4980d3f60 msgr2=0x7fe4980d4370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:28.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.564+0000 7fe49e200640 1 -- 172.21.15.6:0/1614251160 shutdown_connections 2024-03-04T09:25:28.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.563+0000 7f932de00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f933015e120 0x7f9330070c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:28.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.563+0000 7f932de00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f933015e120 0x7f9330070c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.6:33448/0 (socket says 172.21.15.6:33448) 2024-03-04T09:25:28.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.563+0000 7f932de00640 1 -- 172.21.15.6:0/3048787746 learned_addr learned my addr 172.21.15.6:0/3048787746 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:25:28.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.564+0000 7f932de00640 1 -- 172.21.15.6:0/3048787746 >> v2:172.21.15.146:3300/0 conn(0x7f93300ce670 msgr2=0x7f93300ceb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:28.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.564+0000 7f932de00640 1 --2- 172.21.15.6:0/3048787746 >> v2:172.21.15.146:3300/0 conn(0x7f93300ce670 0x7f93300ceb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:28.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.564+0000 7f932de00640 1 -- 172.21.15.6:0/3048787746 >> v1:172.21.15.146:6789/0 conn(0x7f93300b2dd0 legacy=0x7f93300711b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:28.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.564+0000 7f932de00640 1 -- 172.21.15.6:0/3048787746 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93300ce150 con 0x7f933015e120 2024-03-04T09:25:28.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.564+0000 7fe49e200640 1 -- 172.21.15.6:0/1614251160 wait complete. 2024-03-04T09:25:28.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.565+0000 7fb6ffe00640 1 -- 172.21.15.6:0/3465417075 >> v2:172.21.15.6:3300/0 conn(0x7fb70015e200 msgr2=0x7fb7000b3de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:28.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.565+0000 7fb6ffe00640 1 --2- 172.21.15.6:0/3465417075 >> v2:172.21.15.6:3300/0 conn(0x7fb70015e200 0x7fb7000b3de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:28.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.565+0000 7fb6ffe00640 1 -- 172.21.15.6:0/3465417075 >> v1:172.21.15.146:6789/0 conn(0x7fb7000afdf0 legacy=0x7fb7000af140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:28.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.565+0000 7fb6ffe00640 1 -- 172.21.15.6:0/3465417075 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7000ae660 con 0x7fb70006afb0 2024-03-04T09:25:28.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.564+0000 7fb6fea00640 1 -- 172.21.15.6:0/3465417075 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1222895837 0 0) 0x7fb70006bec0 con 0x7fb7000afdf0 2024-03-04T09:25:28.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.566+0000 7f91b6000640 1 Processor -- start 2024-03-04T09:25:28.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.566+0000 7fb6ffe00640 1 --2- 172.21.15.6:0/3465417075 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb70006afb0 0x7fb70006b460 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fb700162c90 tx=0x7fb6f802bee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ba79ee84cc4fcdea server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:28.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.566+0000 7f932de00640 1 --2- 172.21.15.6:0/3048787746 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f933015e120 0x7f9330070c70 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f9320009870 tx=0x7f9320034090 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7e32a1b8e366342b server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:28.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.566+0000 7f91b6000640 1 -- start start 2024-03-04T09:25:28.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.567+0000 7fb6fea00640 1 -- 172.21.15.6:0/3465417075 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6f8039070 con 0x7fb70006afb0 2024-03-04T09:25:28.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.567+0000 7fb6fea00640 1 -- 172.21.15.6:0/3465417075 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6f802cc90 con 0x7fb70006afb0 2024-03-04T09:25:28.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.567+0000 7fb6fea00640 1 -- 172.21.15.6:0/3465417075 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6f8039070 con 0x7fb70006afb0 2024-03-04T09:25:28.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.567+0000 7f932ca00640 1 -- 172.21.15.6:0/3048787746 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9320041070 con 0x7f933015e120 2024-03-04T09:25:28.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.567+0000 7f932ca00640 1 -- 172.21.15.6:0/3048787746 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9320034e40 con 0x7f933015e120 2024-03-04T09:25:28.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.567+0000 7f932ca00640 1 -- 172.21.15.6:0/3048787746 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f932003c640 con 0x7f933015e120 2024-03-04T09:25:28.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.567+0000 7fe49e200640 1 Processor -- start 2024-03-04T09:25:28.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.567+0000 7fe49e200640 1 -- start start 2024-03-04T09:25:28.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.568+0000 7fea3e800640 1 Processor -- start 2024-03-04T09:25:28.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.568+0000 7f91b6000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f91b00cfd90 0x7f91b00d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:28.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.568+0000 7f91b6000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f91b00d2760 0x7f91b00b3740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:28.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.568+0000 7f91b6000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91b00cea50 con 0x7f91b0161b40 2024-03-04T09:25:28.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.568+0000 7f91b6000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91b00b5770 con 0x7f91b00d2760 2024-03-04T09:25:28.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.568+0000 7f91b6000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91b00cf900 con 0x7f91b00cfd90 2024-03-04T09:25:28.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.568+0000 7fb706000640 1 -- 172.21.15.6:0/3465417075 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb70006afb0 msgr2=0x7fb70006b460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:28.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.568+0000 7fb706000640 1 --2- 172.21.15.6:0/3465417075 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb70006afb0 0x7fb70006b460 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fb700162c90 tx=0x7fb6f802bee0 comp rx=0 tx=0).stop 2024-03-04T09:25:28.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.568+0000 7f9334a00640 1 -- 172.21.15.6:0/3048787746 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f933015e120 msgr2=0x7f9330070c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:28.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.568+0000 7f9334a00640 1 --2- 172.21.15.6:0/3048787746 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f933015e120 0x7f9330070c70 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f9320009870 tx=0x7f9320034090 comp rx=0 tx=0).stop 2024-03-04T09:25:28.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.568+0000 7fea3e800640 1 -- start start 2024-03-04T09:25:28.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.568+0000 7f91aea00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f91b00cfd90 0x7f91b00d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:28.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.568+0000 7f9334a00640 1 -- 172.21.15.6:0/3048787746 shutdown_connections 2024-03-04T09:25:28.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.568+0000 7f9334a00640 1 --2- 172.21.15.6:0/3048787746 >> v2:172.21.15.146:3300/0 conn(0x7f93300ce670 0x7f93300ceb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:28.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.568+0000 7f9334a00640 1 --2- 172.21.15.6:0/3048787746 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f933015e120 0x7f9330070c70 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:28.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.568+0000 7f9334a00640 1 -- 172.21.15.6:0/3048787746 >> 172.21.15.6:0/3048787746 conn(0x7f933006afb0 msgr2=0x7f933006b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:28.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.568+0000 7f91aea00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f91b00cfd90 0x7f91b00d2220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.6:33462/0 (socket says 172.21.15.6:33462) 2024-03-04T09:25:28.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.568+0000 7f91aea00640 1 -- 172.21.15.6:0/272699298 learned_addr learned my addr 172.21.15.6:0/272699298 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:25:28.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.568+0000 7f91afe00640 1 --2- 172.21.15.6:0/272699298 >> v2:172.21.15.146:3300/0 conn(0x7f91b00d2760 0x7f91b00b3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:28.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.569+0000 7f9334a00640 1 -- 172.21.15.6:0/3048787746 shutdown_connections 2024-03-04T09:25:28.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.569+0000 7f9334a00640 1 -- 172.21.15.6:0/3048787746 wait complete. 2024-03-04T09:25:28.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.569+0000 7fe49e200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe4980d0d70 0x7fe49814f260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:28.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.570+0000 7fe49e200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe4980d1b30 0x7fe4980cd040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:28.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.570+0000 7fe49e200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe49815e2e0 0x7fe4980cd580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:28.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.570+0000 7fe49e200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4980cfe20 con 0x7fe4980d0d70 2024-03-04T09:25:28.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.571+0000 7fe49e200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4980cfb40 con 0x7fe49815e2e0 2024-03-04T09:25:28.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.571+0000 7fe49e200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4980d2500 con 0x7fe4980d1b30 2024-03-04T09:25:28.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.571+0000 7fe497e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe49815e2e0 0x7fe4980cd580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:28.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.571+0000 7fe497e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe49815e2e0 0x7fe4980cd580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.6:47366/0 (socket says 172.21.15.6:47366) 2024-03-04T09:25:28.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.571+0000 7fe497e00640 1 -- 172.21.15.6:0/1216053169 learned_addr learned my addr 172.21.15.6:0/1216053169 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:25:28.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.571+0000 7fe497400640 1 --2- 172.21.15.6:0/1216053169 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe4980d0d70 0x7fe49814f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:28.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.570+0000 7fea3e800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fea30053750 0x7fea300523e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:28.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.571+0000 7fea3e800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fea30054110 0x7fea300cba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:28.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.571+0000 7fea3e800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fea300e3300 0x7fea300e3860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:28.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.571+0000 7fea3e800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea30057830 con 0x7fea30054110 2024-03-04T09:25:28.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.571+0000 7fea3e800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea300583e0 con 0x7fea300e3300 2024-03-04T09:25:28.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.571+0000 7fea3e800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea30058be0 con 0x7fea30053750 2024-03-04T09:25:28.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.571+0000 7fea3ca00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fea30054110 0x7fea300cba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:28.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.571+0000 7fea3ca00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fea30054110 0x7fea300cba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45438/0 (socket says 172.21.15.6:45438) 2024-03-04T09:25:28.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.571+0000 7fea3ca00640 1 -- 172.21.15.6:0/1922340932 learned_addr learned my addr 172.21.15.6:0/1922340932 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:25:28.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.571+0000 7fea3d400640 1 --2- 172.21.15.6:0/1922340932 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fea30053750 0x7fea300523e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:28.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.572+0000 7fea3de00640 1 --2- 172.21.15.6:0/1922340932 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fea300e3300 0x7fea300e3860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:28.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.569+0000 7f91afe00640 1 -- 172.21.15.6:0/272699298 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f91b00cfd90 msgr2=0x7f91b00d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:28.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.569+0000 7f91afe00640 1 --2- 172.21.15.6:0/272699298 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f91b00cfd90 0x7f91b00d2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:28.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.569+0000 7f91afe00640 1 -- 172.21.15.6:0/272699298 >> v1:172.21.15.146:6789/0 conn(0x7f91b0161b40 legacy=0x7f91b00b5060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:28.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.569+0000 7f91afe00640 1 -- 172.21.15.6:0/272699298 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91b00cecf0 con 0x7f91b00d2760 2024-03-04T09:25:28.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.572+0000 7f91afe00640 1 --2- 172.21.15.6:0/272699298 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f91b00d2760 0x7f91b00b3740 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f91a80097a0 tx=0x7f91a802b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ef3a85a619525b82 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:28.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.569+0000 7fb706000640 1 -- 172.21.15.6:0/3465417075 shutdown_connections 2024-03-04T09:25:28.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.569+0000 7fb706000640 1 --2- 172.21.15.6:0/3465417075 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb70006afb0 0x7fb70006b460 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:28.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.569+0000 7fb706000640 1 --2- 172.21.15.6:0/3465417075 >> v2:172.21.15.6:3300/0 conn(0x7fb70015e200 0x7fb7000b3de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:28.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.569+0000 7fb706000640 1 -- 172.21.15.6:0/3465417075 >> 172.21.15.6:0/3465417075 conn(0x7fb7000b2ae0 msgr2=0x7fb7000b2ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:28.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.569+0000 7fb706000640 1 -- 172.21.15.6:0/3465417075 shutdown_connections 2024-03-04T09:25:28.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.571+0000 7f9334a00640 1 Processor -- start 2024-03-04T09:25:28.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.572+0000 7f9334a00640 1 -- start start 2024-03-04T09:25:28.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.569+0000 7fb706000640 1 -- 172.21.15.6:0/3465417075 wait complete. 2024-03-04T09:25:28.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.572+0000 7fb706000640 1 Processor -- start 2024-03-04T09:25:28.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.572+0000 7fea3ca00640 1 -- 172.21.15.6:0/1922340932 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fea30053750 msgr2=0x7fea300523e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:28.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.572+0000 7fea3ca00640 1 --2- 172.21.15.6:0/1922340932 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fea30053750 0x7fea300523e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:28.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.572+0000 7fea3ca00640 1 -- 172.21.15.6:0/1922340932 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fea300e3300 msgr2=0x7fea300e3860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:28.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.572+0000 7fea3ca00640 1 --2- 172.21.15.6:0/1922340932 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fea300e3300 0x7fea300e3860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:28.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.572+0000 7fea3ca00640 1 -- 172.21.15.6:0/1922340932 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea30052920 con 0x7fea30054110 2024-03-04T09:25:28.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.572+0000 7fe496a00640 1 --2- 172.21.15.6:0/1216053169 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe4980d1b30 0x7fe4980cd040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:28.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.573+0000 7fea3ca00640 1 --2- 172.21.15.6:0/1922340932 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fea30054110 0x7fea300cba80 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fea2400e8a0 tx=0x7fea2400ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:28.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.573+0000 7fb706000640 1 -- start start 2024-03-04T09:25:28.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.573+0000 7f91ae000640 1 -- 172.21.15.6:0/272699298 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91a8031780 con 0x7f91b00d2760 2024-03-04T09:25:28.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.573+0000 7fe497e00640 1 -- 172.21.15.6:0/1216053169 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe4980d1b30 msgr2=0x7fe4980cd040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:28.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.573+0000 7fe497e00640 1 --2- 172.21.15.6:0/1216053169 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe4980d1b30 0x7fe4980cd040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:28.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.573+0000 7fe497e00640 1 -- 172.21.15.6:0/1216053169 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe4980d0d70 msgr2=0x7fe49814f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:28.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.573+0000 7fe497e00640 1 --2- 172.21.15.6:0/1216053169 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe4980d0d70 0x7fe49814f260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:28.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.573+0000 7fe497e00640 1 -- 172.21.15.6:0/1216053169 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4980c8f60 con 0x7fe49815e2e0 2024-03-04T09:25:28.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.573+0000 7fe497400640 1 --2- 172.21.15.6:0/1216053169 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe4980d0d70 0x7fe49814f260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:25:28.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.573+0000 7fea35600640 1 -- 172.21.15.6:0/1922340932 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea24017070 con 0x7fea30054110 2024-03-04T09:25:28.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.573+0000 7fe496a00640 1 --2- 172.21.15.6:0/1216053169 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe4980d1b30 0x7fe4980cd040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:25:28.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.573+0000 7fea35600640 1 -- 172.21.15.6:0/1922340932 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea2400ee90 con 0x7fea30054110 2024-03-04T09:25:28.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.573+0000 7fea35600640 1 -- 172.21.15.6:0/1922340932 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea2400cb80 con 0x7fea30054110 2024-03-04T09:25:28.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.573+0000 7f91ae000640 1 -- 172.21.15.6:0/272699298 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91a8037070 con 0x7f91b00d2760 2024-03-04T09:25:28.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.573+0000 7f91ae000640 1 -- 172.21.15.6:0/272699298 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91a8031b40 con 0x7f91b00d2760 2024-03-04T09:25:28.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.574+0000 7fe497e00640 1 --2- 172.21.15.6:0/1216053169 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe49815e2e0 0x7fe4980cd580 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fe49000efc0 tx=0x7fe49000c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:28.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.574+0000 7f91b6000640 1 -- 172.21.15.6:0/272699298 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f91b00d2760 msgr2=0x7f91b00b3740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:28.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.574+0000 7f91b6000640 1 --2- 172.21.15.6:0/272699298 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f91b00d2760 0x7f91b00b3740 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f91a80097a0 tx=0x7f91a802b820 comp rx=0 tx=0).stop 2024-03-04T09:25:28.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.574+0000 7f91b6000640 1 -- 172.21.15.6:0/272699298 shutdown_connections 2024-03-04T09:25:28.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.574+0000 7f91b6000640 1 --2- 172.21.15.6:0/272699298 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f91b00d2760 0x7f91b00b3740 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:28.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.574+0000 7f91b6000640 1 --2- 172.21.15.6:0/272699298 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f91b00cfd90 0x7f91b00d2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:28.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.574+0000 7f91b6000640 1 -- 172.21.15.6:0/272699298 >> 172.21.15.6:0/272699298 conn(0x7f91b009d220 msgr2=0x7f91b00d3ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:28.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.574+0000 7fea3e800640 1 -- 172.21.15.6:0/1922340932 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea30051340 con 0x7fea30054110 2024-03-04T09:25:28.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.574+0000 7fe48f400640 1 -- 172.21.15.6:0/1216053169 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe490018020 con 0x7fe49815e2e0 2024-03-04T09:25:28.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.575+0000 7f91b6000640 1 -- 172.21.15.6:0/272699298 shutdown_connections 2024-03-04T09:25:28.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.575+0000 7f9334a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f93300b2dd0 0x7f933014f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:28.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.575+0000 7fb706000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb70006afb0 0x7fb7001fad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:28.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.575+0000 7fea3e800640 1 -- 172.21.15.6:0/1922340932 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea30051900 con 0x7fea30054110 2024-03-04T09:25:28.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.575+0000 7fe49e200640 1 -- 172.21.15.6:0/1216053169 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4980c8c60 con 0x7fe49815e2e0 2024-03-04T09:25:28.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.575+0000 7fe48f400640 1 -- 172.21.15.6:0/1216053169 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe490014480 con 0x7fe49815e2e0 2024-03-04T09:25:28.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.575+0000 7fe48f400640 1 -- 172.21.15.6:0/1216053169 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe490014870 con 0x7fe49815e2e0 2024-03-04T09:25:28.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.575+0000 7fe49e200640 1 -- 172.21.15.6:0/1216053169 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4980c8980 con 0x7fe49815e2e0 2024-03-04T09:25:28.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.576+0000 7fea3e800640 1 -- 172.21.15.6:0/1922340932 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea30005710 con 0x7fea30054110 2024-03-04T09:25:28.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.575+0000 7f91b6000640 1 -- 172.21.15.6:0/272699298 wait complete. 2024-03-04T09:25:28.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.577+0000 7fb706000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb7000afdf0 0x7fb700160d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:28.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.577+0000 7fb706000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb70015e200 0x7fb700202080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:28.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.577+0000 7f9334a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f93300ce670 0x7f933009c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:28.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.577+0000 7f9334a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f933015e120 0x7f9330158140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:28.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.577+0000 7f932d400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f93300ce670 0x7f933009c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:28.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.577+0000 7fb6f7e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb7000afdf0 0x7fb700160d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:28.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.578+0000 7fb6f7e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb7000afdf0 0x7fb700160d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.6:33504/0 (socket says 172.21.15.6:33504) 2024-03-04T09:25:28.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.578+0000 7fb6ff400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb70006afb0 0x7fb7001fad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:28.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.578+0000 7fb6ffe00640 1 --2- 172.21.15.6:0/1153631372 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb70015e200 0x7fb700202080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:28.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.578+0000 7fb6f7e00640 1 -- 172.21.15.6:0/1153631372 learned_addr learned my addr 172.21.15.6:0/1153631372 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:25:28.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.578+0000 7fb706000640 1 -- 172.21.15.6:0/1153631372 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb70006dbb0 con 0x7fb70006afb0 2024-03-04T09:25:28.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.578+0000 7fb706000640 1 -- 172.21.15.6:0/1153631372 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7000aec20 con 0x7fb70015e200 2024-03-04T09:25:28.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.578+0000 7fb706000640 1 -- 172.21.15.6:0/1153631372 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb70006dd30 con 0x7fb7000afdf0 2024-03-04T09:25:28.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.578+0000 7f932d400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f93300ce670 0x7f933009c7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45460/0 (socket says 172.21.15.6:45460) 2024-03-04T09:25:28.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.578+0000 7f932d400640 1 -- 172.21.15.6:0/1737186671 learned_addr learned my addr 172.21.15.6:0/1737186671 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:25:28.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.578+0000 7f932e800640 1 --2- 172.21.15.6:0/1737186671 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f933015e120 0x7f9330158140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:28.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.578+0000 7f9334a00640 1 -- 172.21.15.6:0/1737186671 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f933006df00 con 0x7f93300ce670 2024-03-04T09:25:28.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.578+0000 7f9334a00640 1 -- 172.21.15.6:0/1737186671 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f933006e3f0 con 0x7f93300b2dd0 2024-03-04T09:25:28.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.578+0000 7f9334a00640 1 -- 172.21.15.6:0/1737186671 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93300718c0 con 0x7f933015e120 2024-03-04T09:25:28.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.578+0000 7f91b6000640 1 Processor -- start 2024-03-04T09:25:28.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.579+0000 7fe49e200640 1 -- 172.21.15.6:0/1216053169 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe49804f3a0 con 0x7fe49815e2e0 2024-03-04T09:25:28.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.579+0000 7f932de00640 1 --2- 172.21.15.6:0/1737186671 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f93300b2dd0 0x7f933014f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:28.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.579+0000 7f932d400640 1 -- 172.21.15.6:0/1737186671 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f933015e120 msgr2=0x7f9330158140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:28.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.579+0000 7f932d400640 1 --2- 172.21.15.6:0/1737186671 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f933015e120 0x7f9330158140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:28.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.579+0000 7f932d400640 1 -- 172.21.15.6:0/1737186671 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f93300b2dd0 msgr2=0x7f933014f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:28.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.579+0000 7f932d400640 1 --2- 172.21.15.6:0/1737186671 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f93300b2dd0 0x7f933014f7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:28.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.579+0000 7f932d400640 1 -- 172.21.15.6:0/1737186671 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f933009d250 con 0x7f93300ce670 2024-03-04T09:25:28.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.579+0000 7f932de00640 1 --2- 172.21.15.6:0/1737186671 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f93300b2dd0 0x7f933014f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T09:25:28.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.580+0000 7f932d400640 1 --2- 172.21.15.6:0/1737186671 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f93300ce670 0x7f933009c7d0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f931c00c400 tx=0x7f931c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:28.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.580+0000 7fb6ffe00640 1 -- 172.21.15.6:0/1153631372 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb7000afdf0 msgr2=0x7fb700160d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:28.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.580+0000 7fb6ffe00640 1 --2- 172.21.15.6:0/1153631372 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb7000afdf0 0x7fb700160d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:28.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.580+0000 7fb6ffe00640 1 -- 172.21.15.6:0/1153631372 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb70006afb0 msgr2=0x7fb7001fad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:28.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.581+0000 7fea35600640 1 -- 172.21.15.6:0/1922340932 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fea2401a070 con 0x7fea30054110 2024-03-04T09:25:28.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.580+0000 7fb6ffe00640 1 --2- 172.21.15.6:0/1153631372 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb70006afb0 0x7fb7001fad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:28.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.580+0000 7fb6ffe00640 1 -- 172.21.15.6:0/1153631372 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb700150fd0 con 0x7fb70015e200 2024-03-04T09:25:28.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.580+0000 7fb6f7e00640 1 --2- 172.21.15.6:0/1153631372 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb7000afdf0 0x7fb700160d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T09:25:28.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.581+0000 7fb6ff400640 1 --2- 172.21.15.6:0/1153631372 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb70006afb0 0x7fb7001fad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T09:25:28.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.581+0000 7fb6ffe00640 1 --2- 172.21.15.6:0/1153631372 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb70015e200 0x7fb700202080 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fb70006c090 tx=0x7fb6f8004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:28.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.579+0000 7f91b6000640 1 -- start start 2024-03-04T09:25:28.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.582+0000 7fb6f7400640 1 -- 172.21.15.6:0/1153631372 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6f8043020 con 0x7fb70015e200 2024-03-04T09:25:28.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.582+0000 7fea35600640 1 --2- 172.21.15.6:0/1922340932 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fea1806b5a0 0x7fea1806da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:28.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.582+0000 7fea35600640 1 -- 172.21.15.6:0/1922340932 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fea2408a7f0 con 0x7fea30054110 2024-03-04T09:25:28.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.582+0000 7fb706000640 1 -- 172.21.15.6:0/1153631372 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7001512b0 con 0x7fb70015e200 2024-03-04T09:25:28.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.583+0000 7fea3d400640 1 --2- 172.21.15.6:0/1922340932 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fea1806b5a0 0x7fea1806da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:28.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.583+0000 7fb706000640 1 -- 172.21.15.6:0/1153631372 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb700151870 con 0x7fb70015e200 2024-03-04T09:25:28.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.583+0000 7fb6f7400640 1 -- 172.21.15.6:0/1153631372 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6f80046f0 con 0x7fb70015e200 2024-03-04T09:25:28.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.583+0000 7f91b6000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f91b00cfd90 0x7f91b0151bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:28.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.583+0000 7fb6f7400640 1 -- 172.21.15.6:0/1153631372 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6f8039040 con 0x7fb70015e200 2024-03-04T09:25:28.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.583+0000 7f91b6000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f91b00d2760 0x7f91b00cc010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:28.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.583+0000 7f91b6000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f91b0161b40 0x7f91b014e750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:28.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.583+0000 7f91b6000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91b00cf590 con 0x7f91b0161b40 2024-03-04T09:25:28.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.583+0000 7f91b6000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91b00cf2b0 con 0x7f91b00d2760 2024-03-04T09:25:28.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.583+0000 7f91b6000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91b00b2ff0 con 0x7f91b00cfd90 2024-03-04T09:25:28.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.582+0000 7fe48f400640 1 -- 172.21.15.6:0/1216053169 <== mon.1 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe490005d10 con 0x7fe49815e2e0 2024-03-04T09:25:28.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.583+0000 7f91afe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f91b0161b40 0x7f91b014e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:28.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.583+0000 7f91af400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f91b00cfd90 0x7f91b0151bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:28.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.584+0000 7f91af400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f91b00cfd90 0x7f91b0151bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.6:33516/0 (socket says 172.21.15.6:33516) 2024-03-04T09:25:28.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.584+0000 7f91af400640 1 -- 172.21.15.6:0/338755159 learned_addr learned my addr 172.21.15.6:0/338755159 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:25:28.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.584+0000 7f91aea00640 1 --2- 172.21.15.6:0/338755159 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f91b00d2760 0x7f91b00cc010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:28.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.584+0000 7fe48f400640 1 --2- 172.21.15.6:0/1216053169 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fe47406b5d0 0x7fe47406da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:28.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.584+0000 7f91af400640 1 -- 172.21.15.6:0/338755159 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f91b00d2760 msgr2=0x7f91b00cc010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:28.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.584+0000 7f91af400640 1 --2- 172.21.15.6:0/338755159 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f91b00d2760 0x7f91b00cc010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:28.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.584+0000 7f91af400640 1 -- 172.21.15.6:0/338755159 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f91b0161b40 msgr2=0x7f91b014e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:28.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.584+0000 7f91af400640 1 --2- 172.21.15.6:0/338755159 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f91b0161b40 0x7f91b014e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:28.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.584+0000 7f91af400640 1 -- 172.21.15.6:0/338755159 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91b00cc7b0 con 0x7f91b00cfd90 2024-03-04T09:25:28.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.584+0000 7fe48f400640 1 -- 172.21.15.6:0/1216053169 <== mon.1 v2:172.21.15.146:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe49008b050 con 0x7fe49815e2e0 2024-03-04T09:25:28.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.585+0000 7fea3d400640 1 --2- 172.21.15.6:0/1922340932 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fea1806b5a0 0x7fea1806da60 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fea2c009b30 tx=0x7fea2c009240 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:28.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.585+0000 7f91af400640 1 --2- 172.21.15.6:0/338755159 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f91b00cfd90 0x7f91b0151bf0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f91a000c7f0 tx=0x7f91a000ccb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:28.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.585+0000 7fb706000640 1 -- 172.21.15.6:0/1153631372 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6d8005290 con 0x7fb70015e200 2024-03-04T09:25:28.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.585+0000 7fe497400640 1 --2- 172.21.15.6:0/1216053169 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fe47406b5d0 0x7fe47406da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:28.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.586+0000 7f9326000640 1 -- 172.21.15.6:0/1737186671 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f931c017070 con 0x7f93300ce670 2024-03-04T09:25:28.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.586+0000 7f9326000640 1 -- 172.21.15.6:0/1737186671 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f931c0052d0 con 0x7f93300ce670 2024-03-04T09:25:28.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.586+0000 7f9334a00640 1 -- 172.21.15.6:0/1737186671 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f933015e6a0 con 0x7f93300ce670 2024-03-04T09:25:28.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.586+0000 7f9334a00640 1 -- 172.21.15.6:0/1737186671 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93300ae9e0 con 0x7f93300ce670 2024-03-04T09:25:28.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.586+0000 7f91a7400640 1 -- 172.21.15.6:0/338755159 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91a0016020 con 0x7f91b00cfd90 2024-03-04T09:25:28.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.586+0000 7f91a7400640 1 -- 172.21.15.6:0/338755159 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91a000aa50 con 0x7f91b00cfd90 2024-03-04T09:25:28.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.586+0000 7f91b6000640 1 -- 172.21.15.6:0/338755159 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91b00cca90 con 0x7f91b00cfd90 2024-03-04T09:25:28.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.586+0000 7f91a7400640 1 -- 172.21.15.6:0/338755159 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91a000ae40 con 0x7f91b00cfd90 2024-03-04T09:25:28.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.586+0000 7f91b6000640 1 -- 172.21.15.6:0/338755159 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91b00cd050 con 0x7f91b00cfd90 2024-03-04T09:25:28.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.586+0000 7fea35600640 1 -- 172.21.15.6:0/1922340932 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fea24059000 con 0x7fea30054110 2024-03-04T09:25:28.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.586+0000 7fe497400640 1 --2- 172.21.15.6:0/1216053169 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fe47406b5d0 0x7fe47406da90 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fe480036000 tx=0x7fe48003b040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:28.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.587+0000 7f9326000640 1 -- 172.21.15.6:0/1737186671 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f931c003830 con 0x7f93300ce670 2024-03-04T09:25:28.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.588+0000 7f9334a00640 1 -- 172.21.15.6:0/1737186671 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9308005290 con 0x7f93300ce670 2024-03-04T09:25:28.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.588+0000 7f91b6000640 1 -- 172.21.15.6:0/338755159 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91b004f3a0 con 0x7f91b00cfd90 2024-03-04T09:25:28.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.590+0000 7f91a7400640 1 -- 172.21.15.6:0/338755159 <== mon.2 v2:172.21.15.146:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f91a0019020 con 0x7f91b00cfd90 2024-03-04T09:25:28.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.590+0000 7f9326000640 1 -- 172.21.15.6:0/1737186671 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f931c01a070 con 0x7f93300ce670 2024-03-04T09:25:28.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.591+0000 7f91a7400640 1 --2- 172.21.15.6:0/338755159 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f919006b4d0 0x7f919006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:28.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.592+0000 7f91a7400640 1 -- 172.21.15.6:0/338755159 <== mon.2 v2:172.21.15.146:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f91a0089f70 con 0x7f91b00cfd90 2024-03-04T09:25:28.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.592+0000 7f9326000640 1 --2- 172.21.15.6:0/1737186671 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f931006b520 0x7f931006d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:28.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.592+0000 7f91aea00640 1 --2- 172.21.15.6:0/338755159 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f919006b4d0 0x7f919006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:28.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.593+0000 7f9326000640 1 -- 172.21.15.6:0/1737186671 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f931c089400 con 0x7f93300ce670 2024-03-04T09:25:28.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.593+0000 7f932de00640 1 --2- 172.21.15.6:0/1737186671 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f931006b520 0x7f931006d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:28.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.594+0000 7f91aea00640 1 --2- 172.21.15.6:0/338755159 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f919006b4d0 0x7f919006d990 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f91b00b10d0 tx=0x7f919c009240 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:28.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.594+0000 7f932de00640 1 --2- 172.21.15.6:0/1737186671 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f931006b520 0x7f931006d9e0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f932003e000 tx=0x7f9320038000 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:28.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.599+0000 7f9326000640 1 -- 172.21.15.6:0/1737186671 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f931c057d10 con 0x7f93300ce670 2024-03-04T09:25:28.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.603+0000 7fe48f400640 1 -- 172.21.15.6:0/1216053169 <== mon.1 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe490022030 con 0x7fe49815e2e0 2024-03-04T09:25:28.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.607+0000 7fb6f7400640 1 -- 172.21.15.6:0/1153631372 <== mon.1 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb6f8046020 con 0x7fb70015e200 2024-03-04T09:25:28.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.608+0000 7fb6f7400640 1 --2- 172.21.15.6:0/1153631372 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb6dc06b520 0x7fb6dc06d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:28.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.608+0000 7fb6f7400640 1 -- 172.21.15.6:0/1153631372 <== mon.1 v2:172.21.15.146:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb6f80acbf0 con 0x7fb70015e200 2024-03-04T09:25:28.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.609+0000 7fb6ff400640 1 --2- 172.21.15.6:0/1153631372 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb6dc06b520 0x7fb6dc06d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:28.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.611+0000 7fb6ff400640 1 --2- 172.21.15.6:0/1153631372 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb6dc06b520 0x7fb6dc06d9e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fb6f0009880 tx=0x7fb6f0008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:28.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.611+0000 7f33cfe00640 1 Processor -- start 2024-03-04T09:25:28.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.612+0000 7f33cfe00640 1 -- start start 2024-03-04T09:25:28.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.614+0000 7f33cfe00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f33c815d870 0x7f33c8066d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:28.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.614+0000 7f33cfe00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f33c806bfc0 0x7f33c806c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:28.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.614+0000 7f33cfe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f33c806ea30 0x7f33c806eec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:28.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.614+0000 7f33cfe00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33c806c960 con 0x7f33c806ea30 2024-03-04T09:25:28.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.614+0000 7f33cfe00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33c806f400 con 0x7f33c815d870 2024-03-04T09:25:28.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.614+0000 7f33cfe00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33c815dc50 con 0x7f33c806bfc0 2024-03-04T09:25:28.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.614+0000 7f33cde00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f33c806ea30 0x7f33c806eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:28.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.614+0000 7f33cde00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f33c806ea30 0x7f33c806eec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45472/0 (socket says 172.21.15.6:45472) 2024-03-04T09:25:28.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.614+0000 7f33cde00640 1 -- 172.21.15.6:0/2255962127 learned_addr learned my addr 172.21.15.6:0/2255962127 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:25:28.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.614+0000 7f33cca00640 1 --2- 172.21.15.6:0/2255962127 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f33c806bfc0 0x7f33c806c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:28.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.614+0000 7f33cd400640 1 --2- 172.21.15.6:0/2255962127 >> v2:172.21.15.146:3300/0 conn(0x7f33c815d870 0x7f33c8066d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:28.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.615+0000 7f33cde00640 1 -- 172.21.15.6:0/2255962127 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f33c806bfc0 msgr2=0x7f33c806c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:28.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.615+0000 7f33cde00640 1 --2- 172.21.15.6:0/2255962127 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f33c806bfc0 0x7f33c806c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:28.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.615+0000 7f33cde00640 1 -- 172.21.15.6:0/2255962127 >> v2:172.21.15.146:3300/0 conn(0x7f33c815d870 msgr2=0x7f33c8066d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:28.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.615+0000 7f33cde00640 1 --2- 172.21.15.6:0/2255962127 >> v2:172.21.15.146:3300/0 conn(0x7f33c815d870 0x7f33c8066d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:28.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.615+0000 7f33cde00640 1 -- 172.21.15.6:0/2255962127 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33c80b3650 con 0x7f33c806ea30 2024-03-04T09:25:28.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.617+0000 7f33cde00640 1 --2- 172.21.15.6:0/2255962127 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f33c806ea30 0x7f33c806eec0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f33c8071110 tx=0x7f33c002e060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=145a134af9e94b35 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:28.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.617+0000 7f33c7e00640 1 -- 172.21.15.6:0/2255962127 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33c002ec00 con 0x7f33c806ea30 2024-03-04T09:25:28.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.617+0000 7f33c7e00640 1 -- 172.21.15.6:0/2255962127 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33c002eda0 con 0x7f33c806ea30 2024-03-04T09:25:28.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.617+0000 7f33c7e00640 1 -- 172.21.15.6:0/2255962127 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33c0036610 con 0x7f33c806ea30 2024-03-04T09:25:28.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.618+0000 7f33cfe00640 1 -- 172.21.15.6:0/2255962127 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f33c806ea30 msgr2=0x7f33c806eec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:28.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.618+0000 7f33cfe00640 1 --2- 172.21.15.6:0/2255962127 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f33c806ea30 0x7f33c806eec0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f33c8071110 tx=0x7f33c002e060 comp rx=0 tx=0).stop 2024-03-04T09:25:28.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.619+0000 7f33cfe00640 1 -- 172.21.15.6:0/2255962127 shutdown_connections 2024-03-04T09:25:28.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.619+0000 7f91a7400640 1 -- 172.21.15.6:0/338755159 <== mon.2 v2:172.21.15.146:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f91a0020070 con 0x7f91b00cfd90 2024-03-04T09:25:28.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.619+0000 7f33cfe00640 1 --2- 172.21.15.6:0/2255962127 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f33c806ea30 0x7f33c806eec0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:28.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.619+0000 7f33cfe00640 1 --2- 172.21.15.6:0/2255962127 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f33c806bfc0 0x7f33c806c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:28.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.619+0000 7f33cfe00640 1 --2- 172.21.15.6:0/2255962127 >> v2:172.21.15.146:3300/0 conn(0x7f33c815d870 0x7f33c8066d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:28.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.619+0000 7f33cfe00640 1 -- 172.21.15.6:0/2255962127 >> 172.21.15.6:0/2255962127 conn(0x7f33c80714d0 msgr2=0x7f33c80718b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:28.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.621+0000 7f2e10400640 1 Processor -- start 2024-03-04T09:25:28.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.621+0000 7f2e10400640 1 -- start start 2024-03-04T09:25:28.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.619+0000 7f33cfe00640 1 -- 172.21.15.6:0/2255962127 shutdown_connections 2024-03-04T09:25:28.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.619+0000 7f33cfe00640 1 -- 172.21.15.6:0/2255962127 wait complete. 2024-03-04T09:25:28.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.622+0000 7f33cfe00640 1 Processor -- start 2024-03-04T09:25:28.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.623+0000 7fb6f7400640 1 -- 172.21.15.6:0/1153631372 <== mon.1 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb6f804d030 con 0x7fb70015e200 2024-03-04T09:25:28.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.623+0000 7f2e10400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2e0815e2e0 0x7f2e080d0830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:28.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.623+0000 7f2e10400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f2e080d0d70 0x7f2e080d11d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:28.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.623+0000 7f2e10400640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e080b3940 con 0x7f2e080d1b30 2024-03-04T09:25:28.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.623+0000 7f2e10400640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e080d4e70 con 0x7f2e080d0d70 2024-03-04T09:25:28.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.623+0000 7f2e10400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e080b3c40 con 0x7f2e0815e2e0 2024-03-04T09:25:28.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.624+0000 7f2e0d000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f2e080d0d70 0x7f2e080d11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:28.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.624+0000 7f2e0e400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f2e080d1b30 0x7f2e080d1fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:60262/0 (socket says 172.21.15.6:60262) 2024-03-04T09:25:28.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.622+0000 7f33cfe00640 1 -- start start 2024-03-04T09:25:28.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.624+0000 7f33cfe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f33c806bfc0 0x7f33c814f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:28.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.624+0000 7f33cfe00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f33c806ea30 0x7f33c81f8190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:28.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.624+0000 7f33cfe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f33c815d870 0x7f33c815de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:28.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.624+0000 7f2e0e400640 1 -- 172.21.15.6:0/3165351002 learned_addr learned my addr 172.21.15.6:0/3165351002 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:25:28.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.625+0000 7f2e07e00640 1 -- 172.21.15.6:0/3165351002 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2757272611 0 0) 0x7f2e080b3940 con 0x7f2e080d1b30 2024-03-04T09:25:28.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.625+0000 7f2e07e00640 1 -- 172.21.15.6:0/3165351002 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2df0003520 con 0x7f2e080d1b30 2024-03-04T09:25:28.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.625+0000 7f2e0d000640 1 -- 172.21.15.6:0/3165351002 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2e0815e2e0 msgr2=0x7f2e080d0830 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-04T09:25:28.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.625+0000 7f2e0d000640 1 --2- 172.21.15.6:0/3165351002 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2e0815e2e0 0x7f2e080d0830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:28.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.625+0000 7f2e0d000640 1 -- 172.21.15.6:0/3165351002 >> v1:172.21.15.6:6789/0 conn(0x7f2e080d1b30 legacy=0x7f2e080d1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:28.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.625+0000 7f2e0d000640 1 -- 172.21.15.6:0/3165351002 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e080cf580 con 0x7f2e080d0d70 2024-03-04T09:25:28.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.625+0000 7f33cca00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f33c806ea30 0x7f33c81f8190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:28.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.625+0000 7f33cca00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f33c806ea30 0x7f33c81f8190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.6:33536/0 (socket says 172.21.15.6:33536) 2024-03-04T09:25:28.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.625+0000 7f33cd400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f33c806bfc0 0x7f33c814f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:28.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.625+0000 7f33cd400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f33c806bfc0 0x7f33c814f2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.6:47412/0 (socket says 172.21.15.6:47412) 2024-03-04T09:25:28.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.626+0000 7f33cca00640 1 -- 172.21.15.6:0/1116624762 learned_addr learned my addr 172.21.15.6:0/1116624762 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:25:28.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.626+0000 7f33cfe00640 1 -- 172.21.15.6:0/1116624762 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33c80b3ef0 con 0x7f33c815d870 2024-03-04T09:25:28.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.626+0000 7f33cfe00640 1 -- 172.21.15.6:0/1116624762 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33c80b3c10 con 0x7f33c806bfc0 2024-03-04T09:25:28.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.626+0000 7f33cfe00640 1 -- 172.21.15.6:0/1116624762 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33c80b41d0 con 0x7f33c806ea30 2024-03-04T09:25:28.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.627+0000 7f33cde00640 1 --2- 172.21.15.6:0/1116624762 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f33c815d870 0x7f33c815de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:28.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.627+0000 7f33cca00640 1 -- 172.21.15.6:0/1116624762 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f33c806bfc0 msgr2=0x7f33c814f2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:28.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.627+0000 7f33cca00640 1 --2- 172.21.15.6:0/1116624762 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f33c806bfc0 0x7f33c814f2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:28.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.627+0000 7f33cca00640 1 -- 172.21.15.6:0/1116624762 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f33c815d870 msgr2=0x7f33c815de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:28.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.627+0000 7f33cca00640 1 --2- 172.21.15.6:0/1116624762 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f33c815d870 0x7f33c815de60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:28.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.627+0000 7f33cca00640 1 -- 172.21.15.6:0/1116624762 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33c81f8b00 con 0x7f33c806ea30 2024-03-04T09:25:28.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.627+0000 7f33cde00640 1 --2- 172.21.15.6:0/1116624762 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f33c815d870 0x7f33c815de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T09:25:28.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.628+0000 7f33cd400640 1 --2- 172.21.15.6:0/1116624762 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f33c806bfc0 0x7f33c814f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T09:25:28.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.628+0000 7f2e0d000640 1 --2- 172.21.15.6:0/3165351002 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2e080d0d70 0x7f2e080d11d0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f2dfc0097a0 tx=0x7f2dfc02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95a29791f491eb3c server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:28.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.628+0000 7f33cca00640 1 --2- 172.21.15.6:0/1116624762 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f33c806ea30 0x7f33c81f8190 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f33b400d800 tx=0x7f33b400dcc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:28.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.628+0000 7f2e07e00640 1 -- 172.21.15.6:0/3165351002 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dfc035020 con 0x7f2e080d0d70 2024-03-04T09:25:28.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.628+0000 7f2e07e00640 1 -- 172.21.15.6:0/3165351002 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2dfc031640 con 0x7f2e080d0d70 2024-03-04T09:25:28.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.628+0000 7f2e07e00640 1 -- 172.21.15.6:0/3165351002 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dfc02d040 con 0x7f2e080d0d70 2024-03-04T09:25:28.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.629+0000 7f33c5600640 1 -- 172.21.15.6:0/1116624762 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33b4018020 con 0x7f33c806ea30 2024-03-04T09:25:28.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.629+0000 7f33c5600640 1 -- 172.21.15.6:0/1116624762 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33b40096a0 con 0x7f33c806ea30 2024-03-04T09:25:28.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.629+0000 7f33c5600640 1 -- 172.21.15.6:0/1116624762 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33b4009a70 con 0x7f33c806ea30 2024-03-04T09:25:28.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.629+0000 7f33cfe00640 1 -- 172.21.15.6:0/1116624762 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33c81f8de0 con 0x7f33c806ea30 2024-03-04T09:25:28.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.629+0000 7f33cfe00640 1 -- 172.21.15.6:0/1116624762 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33c81f93f0 con 0x7f33c806ea30 2024-03-04T09:25:28.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.629+0000 7f2e10400640 1 -- 172.21.15.6:0/3165351002 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2e080d0d70 msgr2=0x7f2e080d11d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:28.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.629+0000 7f2e10400640 1 --2- 172.21.15.6:0/3165351002 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2e080d0d70 0x7f2e080d11d0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f2dfc0097a0 tx=0x7f2dfc02b8f0 comp rx=0 tx=0).stop 2024-03-04T09:25:28.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.630+0000 7f2e10400640 1 -- 172.21.15.6:0/3165351002 shutdown_connections 2024-03-04T09:25:28.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.630+0000 7f2e10400640 1 --2- 172.21.15.6:0/3165351002 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2e080d0d70 0x7f2e080d11d0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:28.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.630+0000 7f2e10400640 1 --2- 172.21.15.6:0/3165351002 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2e0815e2e0 0x7f2e080d0830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:28.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.630+0000 7f2e10400640 1 -- 172.21.15.6:0/3165351002 >> 172.21.15.6:0/3165351002 conn(0x7f2e080d3f60 msgr2=0x7f2e080d4370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:28.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.630+0000 7f2e10400640 1 -- 172.21.15.6:0/3165351002 shutdown_connections 2024-03-04T09:25:28.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.630+0000 7f2e10400640 1 -- 172.21.15.6:0/3165351002 wait complete. 2024-03-04T09:25:28.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.633+0000 7f2e10400640 1 Processor -- start 2024-03-04T09:25:28.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.631+0000 7f33cfe00640 1 -- 172.21.15.6:0/1116624762 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33c804f3a0 con 0x7f33c806ea30 2024-03-04T09:25:28.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.633+0000 7f2e10400640 1 -- start start 2024-03-04T09:25:28.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.634+0000 7f33c5600640 1 -- 172.21.15.6:0/1116624762 <== mon.2 v2:172.21.15.146:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f33b401b020 con 0x7f33c806ea30 2024-03-04T09:25:28.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.635+0000 7f33c5600640 1 --2- 172.21.15.6:0/1116624762 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f33a806b4d0 0x7f33a806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:28.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.635+0000 7f2e10400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2e080d0d70 0x7f2e080b98a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:28.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.636+0000 7f33c5600640 1 -- 172.21.15.6:0/1116624762 <== mon.2 v2:172.21.15.146:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f33b408ad00 con 0x7f33c806ea30 2024-03-04T09:25:28.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.636+0000 7f2e10400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2e080d1b30 0x7f2e081533f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:28.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.636+0000 7f2e10400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2e0815e2e0 0x7f2e08153c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:28.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.636+0000 7f33cd400640 1 --2- 172.21.15.6:0/1116624762 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f33a806b4d0 0x7f33a806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:28.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.636+0000 7f2e10400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e080cfe20 con 0x7f2e080d0d70 2024-03-04T09:25:28.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.637+0000 7f2e0e400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2e0815e2e0 0x7f2e08153c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:28.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.637+0000 7f2e0e400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2e0815e2e0 0x7f2e08153c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.6:47422/0 (socket says 172.21.15.6:47422) 2024-03-04T09:25:28.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.637+0000 7f2e0e400640 1 -- 172.21.15.6:0/366206193 learned_addr learned my addr 172.21.15.6:0/366206193 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:25:28.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.637+0000 7f2e0d000640 1 --2- 172.21.15.6:0/366206193 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2e080d1b30 0x7f2e081533f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:28.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.637+0000 7f2e0da00640 1 --2- 172.21.15.6:0/366206193 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2e080d0d70 0x7f2e080b98a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:28.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.637+0000 7f2e10400640 1 -- 172.21.15.6:0/366206193 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e080cfb40 con 0x7f2e0815e2e0 2024-03-04T09:25:28.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.637+0000 7f2e10400640 1 -- 172.21.15.6:0/366206193 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e080b50e0 con 0x7f2e080d1b30 2024-03-04T09:25:28.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.638+0000 7f33cd400640 1 --2- 172.21.15.6:0/1116624762 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f33a806b4d0 0x7f33a806d990 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f33bc006fd0 tx=0x7f33bc008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:28.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.638+0000 7f2e0da00640 1 -- 172.21.15.6:0/366206193 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2e080d1b30 msgr2=0x7f2e081533f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:28.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.638+0000 7f2e0da00640 1 --2- 172.21.15.6:0/366206193 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2e080d1b30 0x7f2e081533f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:28.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.638+0000 7f2e0da00640 1 -- 172.21.15.6:0/366206193 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2e0815e2e0 msgr2=0x7f2e08153c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:28.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.638+0000 7f2e0da00640 1 --2- 172.21.15.6:0/366206193 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2e0815e2e0 0x7f2e08153c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:28.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.638+0000 7f2e0da00640 1 -- 172.21.15.6:0/366206193 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e08153930 con 0x7f2e080d0d70 2024-03-04T09:25:28.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.639+0000 7f2e0da00640 1 --2- 172.21.15.6:0/366206193 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2e080d0d70 0x7f2e080b98a0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f2df400b410 tx=0x7f2df400b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:28.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.639+0000 7f2e05600640 1 -- 172.21.15.6:0/366206193 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2df4015020 con 0x7f2e080d0d70 2024-03-04T09:25:28.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.639+0000 7f2e10400640 1 -- 172.21.15.6:0/366206193 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e080b6f30 con 0x7f2e080d0d70 2024-03-04T09:25:28.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.639+0000 7f2e10400640 1 -- 172.21.15.6:0/366206193 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e081603b0 con 0x7f2e080d0d70 2024-03-04T09:25:28.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.640+0000 7f2e05600640 1 -- 172.21.15.6:0/366206193 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2df4011830 con 0x7f2e080d0d70 2024-03-04T09:25:28.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.640+0000 7f2e05600640 1 -- 172.21.15.6:0/366206193 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2df4015020 con 0x7f2e080d0d70 2024-03-04T09:25:28.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.641+0000 7f2e10400640 1 -- 172.21.15.6:0/366206193 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e0815fee0 con 0x7f2e080d0d70 2024-03-04T09:25:28.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.643+0000 7f2e05600640 1 -- 172.21.15.6:0/366206193 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2df4018020 con 0x7f2e080d0d70 2024-03-04T09:25:28.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.645+0000 7f2e05600640 1 --2- 172.21.15.6:0/366206193 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2de806b6d0 0x7f2de806db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:28.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.645+0000 7f2e05600640 1 -- 172.21.15.6:0/366206193 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2df40897e0 con 0x7f2e080d0d70 2024-03-04T09:25:28.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.646+0000 7f33c5600640 1 -- 172.21.15.6:0/1116624762 <== mon.2 v2:172.21.15.146:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f33b4022030 con 0x7f33c806ea30 2024-03-04T09:25:28.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.646+0000 7f2e0d000640 1 --2- 172.21.15.6:0/366206193 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2de806b6d0 0x7f2de806db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:28.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.647+0000 7f2e0d000640 1 --2- 172.21.15.6:0/366206193 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2de806b6d0 0x7f2de806db90 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f2dfc036000 tx=0x7f2dfc03c040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:28.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.654+0000 7f2e05600640 1 -- 172.21.15.6:0/366206193 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2df401f030 con 0x7f2e080d0d70 2024-03-04T09:25:28.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.985+0000 7f1dd2a00640 1 -- 172.21.15.6:0/2776489773 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f1dbc005850 con 0x7f1de00cca20 2024-03-04T09:25:28.986 INFO:teuthology.orchestra.run.smithi006.stdout:55834574854 2024-03-04T09:25:28.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.986+0000 7f1ddd600640 1 -- 172.21.15.6:0/2776489773 <== mon.1 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f1dcc058ba0 con 0x7f1de00cca20 2024-03-04T09:25:28.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.991+0000 7f1de8000640 1 -- 172.21.15.6:0/2776489773 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f1dc006b900 msgr2=0x7f1dc006ddc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:28.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.991+0000 7f1de8000640 1 --2- 172.21.15.6:0/2776489773 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f1dc006b900 0x7f1dc006ddc0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f1dd4036000 tx=0x7f1dd400a000 comp rx=0 tx=0).stop 2024-03-04T09:25:28.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.991+0000 7f1de8000640 1 -- 172.21.15.6:0/2776489773 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1de00cca20 msgr2=0x7f1de0143110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:28.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.991+0000 7f1de8000640 1 --2- 172.21.15.6:0/2776489773 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1de00cca20 0x7f1de0143110 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f1dcc00c7b0 tx=0x7f1dcc00cc70 comp rx=0 tx=0).stop 2024-03-04T09:25:28.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.992+0000 7f1de8000640 1 -- 172.21.15.6:0/2776489773 shutdown_connections 2024-03-04T09:25:28.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.992+0000 7f1de8000640 1 --2- 172.21.15.6:0/2776489773 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f1dc006b900 0x7f1dc006ddc0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:28.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.992+0000 7f1de8000640 1 --2- 172.21.15.6:0/2776489773 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1de0140640 0x7f1de015c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:28.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.992+0000 7f1de8000640 1 --2- 172.21.15.6:0/2776489773 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1de00cca20 0x7f1de0143110 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:28.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.992+0000 7f1de8000640 1 --2- 172.21.15.6:0/2776489773 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1de006de00 0x7f1de013f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:28.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.992+0000 7f1de8000640 1 -- 172.21.15.6:0/2776489773 >> 172.21.15.6:0/2776489773 conn(0x7f1de006afb0 msgr2=0x7f1de00b1ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:28.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.992+0000 7f1de8000640 1 -- 172.21.15.6:0/2776489773 shutdown_connections 2024-03-04T09:25:28.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:28.993+0000 7f1de8000640 1 -- 172.21.15.6:0/2776489773 wait complete. 2024-03-04T09:25:28.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.002+0000 7fea3e800640 1 -- 172.21.15.6:0/1922340932 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fea300e29c0 con 0x7fea30054110 2024-03-04T09:25:29.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.003+0000 7fea35600640 1 -- 172.21.15.6:0/1922340932 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fea2405d0a0 con 0x7fea30054110 2024-03-04T09:25:29.003 INFO:teuthology.orchestra.run.smithi006.stdout:55834574855 2024-03-04T09:25:29.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.007+0000 7fea3e800640 1 -- 172.21.15.6:0/1922340932 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fea1806b5a0 msgr2=0x7fea1806da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:29.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.007+0000 7fea3e800640 1 --2- 172.21.15.6:0/1922340932 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fea1806b5a0 0x7fea1806da60 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fea2c009b30 tx=0x7fea2c009240 comp rx=0 tx=0).stop 2024-03-04T09:25:29.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.007+0000 7fea3e800640 1 -- 172.21.15.6:0/1922340932 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fea30054110 msgr2=0x7fea300cba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:29.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.007+0000 7fea3e800640 1 --2- 172.21.15.6:0/1922340932 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fea30054110 0x7fea300cba80 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fea2400e8a0 tx=0x7fea2400ed60 comp rx=0 tx=0).stop 2024-03-04T09:25:29.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.008+0000 7fea3e800640 1 -- 172.21.15.6:0/1922340932 shutdown_connections 2024-03-04T09:25:29.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.008+0000 7fea3e800640 1 --2- 172.21.15.6:0/1922340932 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fea300e3300 0x7fea300e3860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:29.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.008+0000 7fea3e800640 1 --2- 172.21.15.6:0/1922340932 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fea30054110 0x7fea300cba80 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:29.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.008+0000 7fea3e800640 1 --2- 172.21.15.6:0/1922340932 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fea1806b5a0 0x7fea1806da60 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:29.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.008+0000 7fea3e800640 1 --2- 172.21.15.6:0/1922340932 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fea30053750 0x7fea300523e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:29.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.008+0000 7fea3e800640 1 -- 172.21.15.6:0/1922340932 >> 172.21.15.6:0/1922340932 conn(0x7fea30058ef0 msgr2=0x7fea3003c960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:29.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.009+0000 7fea3e800640 1 -- 172.21.15.6:0/1922340932 shutdown_connections 2024-03-04T09:25:29.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.009+0000 7fea3e800640 1 -- 172.21.15.6:0/1922340932 wait complete. 2024-03-04T09:25:29.016 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.2 2024-03-04T09:25:29.016 DEBUG:teuthology.parallel:result is None 2024-03-04T09:25:29.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.026+0000 7f9334a00640 1 -- 172.21.15.6:0/1737186671 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f9308005850 con 0x7f93300ce670 2024-03-04T09:25:29.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.027+0000 7f9326000640 1 -- 172.21.15.6:0/1737186671 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f931c05bdb0 con 0x7f93300ce670 2024-03-04T09:25:29.024 INFO:teuthology.orchestra.run.smithi006.stdout:55834574856 2024-03-04T09:25:29.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.029+0000 7f9334a00640 1 -- 172.21.15.6:0/1737186671 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f931006b520 msgr2=0x7f931006d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:29.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.029+0000 7f9334a00640 1 --2- 172.21.15.6:0/1737186671 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f931006b520 0x7f931006d9e0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f932003e000 tx=0x7f9320038000 comp rx=0 tx=0).stop 2024-03-04T09:25:29.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.029+0000 7f9334a00640 1 -- 172.21.15.6:0/1737186671 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f93300ce670 msgr2=0x7f933009c7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:29.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.029+0000 7f9334a00640 1 --2- 172.21.15.6:0/1737186671 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f93300ce670 0x7f933009c7d0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f931c00c400 tx=0x7f931c00c8c0 comp rx=0 tx=0).stop 2024-03-04T09:25:29.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.029+0000 7f9334a00640 1 -- 172.21.15.6:0/1737186671 shutdown_connections 2024-03-04T09:25:29.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.029+0000 7f9334a00640 1 --2- 172.21.15.6:0/1737186671 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f931006b520 0x7f931006d9e0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:29.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.029+0000 7f9334a00640 1 --2- 172.21.15.6:0/1737186671 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f933015e120 0x7f9330158140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:29.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.029+0000 7f9334a00640 1 --2- 172.21.15.6:0/1737186671 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f93300ce670 0x7f933009c7d0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:29.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.029+0000 7f9334a00640 1 --2- 172.21.15.6:0/1737186671 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f93300b2dd0 0x7f933014f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:29.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.029+0000 7f9334a00640 1 -- 172.21.15.6:0/1737186671 >> 172.21.15.6:0/1737186671 conn(0x7f933006afb0 msgr2=0x7f93300af550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:29.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.030+0000 7f9334a00640 1 -- 172.21.15.6:0/1737186671 shutdown_connections 2024-03-04T09:25:29.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.030+0000 7f9334a00640 1 -- 172.21.15.6:0/1737186671 wait complete. 2024-03-04T09:25:29.032 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.1 2024-03-04T09:25:29.032 DEBUG:teuthology.parallel:result is None 2024-03-04T09:25:29.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.040+0000 7fe49e200640 1 -- 172.21.15.6:0/1216053169 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7fe4980b5080 con 0x7fe49815e2e0 2024-03-04T09:25:29.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.041+0000 7fe48f400640 1 -- 172.21.15.6:0/1216053169 <== mon.1 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fe490010040 con 0x7fe49815e2e0 2024-03-04T09:25:29.038 INFO:teuthology.orchestra.run.smithi006.stdout:55834574853 2024-03-04T09:25:29.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.046+0000 7fe49e200640 1 -- 172.21.15.6:0/1216053169 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fe47406b5d0 msgr2=0x7fe47406da90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:29.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.046+0000 7fe49e200640 1 --2- 172.21.15.6:0/1216053169 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fe47406b5d0 0x7fe47406da90 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fe480036000 tx=0x7fe48003b040 comp rx=0 tx=0).stop 2024-03-04T09:25:29.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.046+0000 7fe49e200640 1 -- 172.21.15.6:0/1216053169 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe49815e2e0 msgr2=0x7fe4980cd580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:29.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.046+0000 7fe49e200640 1 --2- 172.21.15.6:0/1216053169 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe49815e2e0 0x7fe4980cd580 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fe49000efc0 tx=0x7fe49000c490 comp rx=0 tx=0).stop 2024-03-04T09:25:29.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.046+0000 7fe49e200640 1 -- 172.21.15.6:0/1216053169 shutdown_connections 2024-03-04T09:25:29.044 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.5 2024-03-04T09:25:29.044 DEBUG:teuthology.parallel:result is None 2024-03-04T09:25:29.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.046+0000 7fe49e200640 1 --2- 172.21.15.6:0/1216053169 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe49815e2e0 0x7fe4980cd580 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:29.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.046+0000 7fe49e200640 1 --2- 172.21.15.6:0/1216053169 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fe47406b5d0 0x7fe47406da90 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:29.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.046+0000 7fe49e200640 1 --2- 172.21.15.6:0/1216053169 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe4980d1b30 0x7fe4980cd040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:29.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.046+0000 7fe49e200640 1 --2- 172.21.15.6:0/1216053169 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe4980d0d70 0x7fe49814f260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:29.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.046+0000 7fe49e200640 1 -- 172.21.15.6:0/1216053169 >> 172.21.15.6:0/1216053169 conn(0x7fe4980d3f60 msgr2=0x7fe4980b2c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:29.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.046+0000 7fe49e200640 1 -- 172.21.15.6:0/1216053169 shutdown_connections 2024-03-04T09:25:29.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.046+0000 7fe49e200640 1 -- 172.21.15.6:0/1216053169 wait complete. 2024-03-04T09:25:29.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.048+0000 7fb706000640 1 -- 172.21.15.6:0/1153631372 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fb6d8005850 con 0x7fb70015e200 2024-03-04T09:25:29.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.049+0000 7fb6f7400640 1 -- 172.21.15.6:0/1153631372 <== mon.1 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fb6f807b590 con 0x7fb70015e200 2024-03-04T09:25:29.046 INFO:teuthology.orchestra.run.smithi006.stdout:55834574854 2024-03-04T09:25:29.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.051+0000 7fb706000640 1 -- 172.21.15.6:0/1153631372 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb6dc06b520 msgr2=0x7fb6dc06d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:29.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.051+0000 7fb706000640 1 --2- 172.21.15.6:0/1153631372 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb6dc06b520 0x7fb6dc06d9e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fb6f0009880 tx=0x7fb6f0008040 comp rx=0 tx=0).stop 2024-03-04T09:25:29.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.051+0000 7fb706000640 1 -- 172.21.15.6:0/1153631372 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb70015e200 msgr2=0x7fb700202080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:29.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.051+0000 7fb706000640 1 --2- 172.21.15.6:0/1153631372 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb70015e200 0x7fb700202080 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fb70006c090 tx=0x7fb6f8004a40 comp rx=0 tx=0).stop 2024-03-04T09:25:29.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.051+0000 7fb706000640 1 -- 172.21.15.6:0/1153631372 shutdown_connections 2024-03-04T09:25:29.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.051+0000 7fb706000640 1 --2- 172.21.15.6:0/1153631372 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb70015e200 0x7fb700202080 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:29.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.051+0000 7fb706000640 1 --2- 172.21.15.6:0/1153631372 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb6dc06b520 0x7fb6dc06d9e0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:29.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.051+0000 7fb706000640 1 --2- 172.21.15.6:0/1153631372 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb7000afdf0 0x7fb700160d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:29.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.051+0000 7fb706000640 1 --2- 172.21.15.6:0/1153631372 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb70006afb0 0x7fb7001fad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:29.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.051+0000 7fb706000640 1 -- 172.21.15.6:0/1153631372 >> 172.21.15.6:0/1153631372 conn(0x7fb7000b2ae0 msgr2=0x7fb700156af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:29.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.051+0000 7fb706000640 1 -- 172.21.15.6:0/1153631372 shutdown_connections 2024-03-04T09:25:29.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.052+0000 7fb706000640 1 -- 172.21.15.6:0/1153631372 wait complete. 2024-03-04T09:25:29.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.055+0000 7f91b6000640 1 -- 172.21.15.6:0/338755159 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f91b01612f0 con 0x7f91b00cfd90 2024-03-04T09:25:29.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.056+0000 7f91a7400640 1 -- 172.21.15.6:0/338755159 <== mon.2 v2:172.21.15.146:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f91a0058680 con 0x7f91b00cfd90 2024-03-04T09:25:29.053 INFO:teuthology.orchestra.run.smithi006.stdout:55834574854 2024-03-04T09:25:29.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.060+0000 7f91b6000640 1 -- 172.21.15.6:0/338755159 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f919006b4d0 msgr2=0x7f919006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:29.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.060+0000 7f91b6000640 1 --2- 172.21.15.6:0/338755159 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f919006b4d0 0x7f919006d990 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f91b00b10d0 tx=0x7f919c009240 comp rx=0 tx=0).stop 2024-03-04T09:25:29.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.060+0000 7f91b6000640 1 -- 172.21.15.6:0/338755159 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f91b00cfd90 msgr2=0x7f91b0151bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:29.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.060+0000 7f91b6000640 1 --2- 172.21.15.6:0/338755159 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f91b00cfd90 0x7f91b0151bf0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f91a000c7f0 tx=0x7f91a000ccb0 comp rx=0 tx=0).stop 2024-03-04T09:25:29.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.060+0000 7f91b6000640 1 -- 172.21.15.6:0/338755159 shutdown_connections 2024-03-04T09:25:29.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.060+0000 7f91b6000640 1 --2- 172.21.15.6:0/338755159 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f91b0161b40 0x7f91b014e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:29.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.060+0000 7f91b6000640 1 --2- 172.21.15.6:0/338755159 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f91b00d2760 0x7f91b00cc010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:29.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.060+0000 7f91b6000640 1 --2- 172.21.15.6:0/338755159 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f919006b4d0 0x7f919006d990 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:29.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.060+0000 7f91b6000640 1 --2- 172.21.15.6:0/338755159 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f91b00cfd90 0x7f91b0151bf0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:29.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.060+0000 7f91b6000640 1 -- 172.21.15.6:0/338755159 >> 172.21.15.6:0/338755159 conn(0x7f91b009d220 msgr2=0x7f91b00b15e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:29.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.060+0000 7f91b6000640 1 -- 172.21.15.6:0/338755159 shutdown_connections 2024-03-04T09:25:29.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.060+0000 7f91b6000640 1 -- 172.21.15.6:0/338755159 wait complete. 2024-03-04T09:25:29.065 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.0 2024-03-04T09:25:29.065 DEBUG:teuthology.parallel:result is None 2024-03-04T09:25:29.068 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574853 for osd.4 2024-03-04T09:25:29.068 DEBUG:teuthology.parallel:result is None 2024-03-04T09:25:29.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.075+0000 7f2e10400640 1 -- 172.21.15.6:0/366206193 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f2e0804f3a0 con 0x7f2e080d0d70 2024-03-04T09:25:29.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.076+0000 7f2e05600640 1 -- 172.21.15.6:0/366206193 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f2e08153930 con 0x7f2e080d0d70 2024-03-04T09:25:29.073 INFO:teuthology.orchestra.run.smithi006.stdout:55834574854 2024-03-04T09:25:29.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.078+0000 7f2e10400640 1 -- 172.21.15.6:0/366206193 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2de806b6d0 msgr2=0x7f2de806db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:29.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.078+0000 7f2e10400640 1 --2- 172.21.15.6:0/366206193 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2de806b6d0 0x7f2de806db90 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f2dfc036000 tx=0x7f2dfc03c040 comp rx=0 tx=0).stop 2024-03-04T09:25:29.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.078+0000 7f2e10400640 1 -- 172.21.15.6:0/366206193 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2e080d0d70 msgr2=0x7f2e080b98a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:29.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.078+0000 7f2e10400640 1 --2- 172.21.15.6:0/366206193 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2e080d0d70 0x7f2e080b98a0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f2df400b410 tx=0x7f2df400b8d0 comp rx=0 tx=0).stop 2024-03-04T09:25:29.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.078+0000 7f2e10400640 1 -- 172.21.15.6:0/366206193 shutdown_connections 2024-03-04T09:25:29.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.078+0000 7f2e10400640 1 --2- 172.21.15.6:0/366206193 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2e0815e2e0 0x7f2e08153c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:29.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.078+0000 7f2e10400640 1 --2- 172.21.15.6:0/366206193 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2de806b6d0 0x7f2de806db90 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:29.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.078+0000 7f2e10400640 1 --2- 172.21.15.6:0/366206193 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2e080d1b30 0x7f2e081533f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:29.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.078+0000 7f2e10400640 1 --2- 172.21.15.6:0/366206193 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2e080d0d70 0x7f2e080b98a0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:29.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.078+0000 7f2e10400640 1 -- 172.21.15.6:0/366206193 >> 172.21.15.6:0/366206193 conn(0x7f2e080d3f60 msgr2=0x7f2e0806d040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:29.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.078+0000 7f2e10400640 1 -- 172.21.15.6:0/366206193 shutdown_connections 2024-03-04T09:25:29.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.078+0000 7f2e10400640 1 -- 172.21.15.6:0/366206193 wait complete. 2024-03-04T09:25:29.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.081+0000 7f33cfe00640 1 -- 172.21.15.6:0/1116624762 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f33c8152310 con 0x7f33c806ea30 2024-03-04T09:25:29.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.082+0000 7f33c5600640 1 -- 172.21.15.6:0/1116624762 <== mon.2 v2:172.21.15.146:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f33b4059be0 con 0x7f33c806ea30 2024-03-04T09:25:29.079 INFO:teuthology.orchestra.run.smithi006.stdout:55834574855 2024-03-04T09:25:29.080 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.6 2024-03-04T09:25:29.080 DEBUG:teuthology.parallel:result is None 2024-03-04T09:25:29.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.085+0000 7f33cfe00640 1 -- 172.21.15.6:0/1116624762 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f33a806b4d0 msgr2=0x7f33a806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:29.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.085+0000 7f33cfe00640 1 --2- 172.21.15.6:0/1116624762 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f33a806b4d0 0x7f33a806d990 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f33bc006fd0 tx=0x7f33bc008040 comp rx=0 tx=0).stop 2024-03-04T09:25:29.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.085+0000 7f33cfe00640 1 -- 172.21.15.6:0/1116624762 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f33c806ea30 msgr2=0x7f33c81f8190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:29.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.085+0000 7f33cfe00640 1 --2- 172.21.15.6:0/1116624762 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f33c806ea30 0x7f33c81f8190 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f33b400d800 tx=0x7f33b400dcc0 comp rx=0 tx=0).stop 2024-03-04T09:25:29.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.085+0000 7f33cfe00640 1 -- 172.21.15.6:0/1116624762 shutdown_connections 2024-03-04T09:25:29.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.085+0000 7f33cfe00640 1 --2- 172.21.15.6:0/1116624762 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f33c815d870 0x7f33c815de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:29.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.085+0000 7f33cfe00640 1 --2- 172.21.15.6:0/1116624762 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f33a806b4d0 0x7f33a806d990 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:29.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.085+0000 7f33cfe00640 1 --2- 172.21.15.6:0/1116624762 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f33c806ea30 0x7f33c81f8190 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:29.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.085+0000 7f33cfe00640 1 --2- 172.21.15.6:0/1116624762 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f33c806bfc0 0x7f33c814f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:29.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.085+0000 7f33cfe00640 1 -- 172.21.15.6:0/1116624762 >> 172.21.15.6:0/1116624762 conn(0x7f33c80714d0 msgr2=0x7f33c815c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:29.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.085+0000 7f33cfe00640 1 -- 172.21.15.6:0/1116624762 shutdown_connections 2024-03-04T09:25:29.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.085+0000 7f33cfe00640 1 -- 172.21.15.6:0/1116624762 wait complete. 2024-03-04T09:25:29.092 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.7 2024-03-04T09:25:29.092 DEBUG:teuthology.parallel:result is None 2024-03-04T09:25:29.096 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.3 2024-03-04T09:25:29.096 DEBUG:teuthology.parallel:result is None 2024-03-04T09:25:29.097 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-03-04T09:25:29.097 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T09:25:29.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.355+0000 7febc7e00640 1 Processor -- start 2024-03-04T09:25:29.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.355+0000 7febc7e00640 1 -- start start 2024-03-04T09:25:29.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.357+0000 7febc7e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7febc8070f10 0x7febc80cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:29.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.357+0000 7febc7e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7febc80cce90 0x7febc815c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:29.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.357+0000 7febc7e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febc80714d0 con 0x7febc8145300 2024-03-04T09:25:29.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.357+0000 7febc7e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febc80cd300 con 0x7febc80cce90 2024-03-04T09:25:29.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.357+0000 7febc7e00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febc80717b0 con 0x7febc8070f10 2024-03-04T09:25:29.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.357+0000 7febc6a00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7febc8145300 0x7febc80b49a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:36790/0 (socket says 172.21.15.6:36790) 2024-03-04T09:25:29.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.357+0000 7febc6a00640 1 -- 172.21.15.6:0/4270835416 learned_addr learned my addr 172.21.15.6:0/4270835416 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:25:29.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.357+0000 7febc7400640 1 --2- 172.21.15.6:0/4270835416 >> v2:172.21.15.6:3300/0 conn(0x7febc80cce90 0x7febc815c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:29.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.358+0000 7febc6000640 1 --2- 172.21.15.6:0/4270835416 >> v2:172.21.15.146:3300/0 conn(0x7febc8070f10 0x7febc80cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:29.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.358+0000 7febc7400640 1 -- 172.21.15.6:0/4270835416 >> v2:172.21.15.146:3300/0 conn(0x7febc8070f10 msgr2=0x7febc80cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:29.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.358+0000 7febc7400640 1 --2- 172.21.15.6:0/4270835416 >> v2:172.21.15.146:3300/0 conn(0x7febc8070f10 0x7febc80cc950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:29.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.358+0000 7febc7400640 1 -- 172.21.15.6:0/4270835416 >> v1:172.21.15.6:6789/0 conn(0x7febc8145300 legacy=0x7febc80b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:29.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.358+0000 7febc5600640 1 -- 172.21.15.6:0/4270835416 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3891661224 0 0) 0x7febc80714d0 con 0x7febc8145300 2024-03-04T09:25:29.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.358+0000 7febc7400640 1 -- 172.21.15.6:0/4270835416 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febc8071a90 con 0x7febc80cce90 2024-03-04T09:25:29.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.359+0000 7febc7400640 1 --2- 172.21.15.6:0/4270835416 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7febc80cce90 0x7febc815c570 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7febc80b5150 tx=0x7febc002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85d6942d7ddc727b server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:29.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.359+0000 7febc5600640 1 -- 172.21.15.6:0/4270835416 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febc0039070 con 0x7febc80cce90 2024-03-04T09:25:29.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.359+0000 7febc5600640 1 -- 172.21.15.6:0/4270835416 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febc002cbf0 con 0x7febc80cce90 2024-03-04T09:25:29.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.359+0000 7febc5600640 1 -- 172.21.15.6:0/4270835416 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febc0034440 con 0x7febc80cce90 2024-03-04T09:25:29.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.360+0000 7febc7e00640 1 -- 172.21.15.6:0/4270835416 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7febc80cce90 msgr2=0x7febc815c570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:29.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.360+0000 7febc7e00640 1 --2- 172.21.15.6:0/4270835416 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7febc80cce90 0x7febc815c570 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7febc80b5150 tx=0x7febc002be40 comp rx=0 tx=0).stop 2024-03-04T09:25:29.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.360+0000 7febc7e00640 1 -- 172.21.15.6:0/4270835416 shutdown_connections 2024-03-04T09:25:29.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.360+0000 7febc7e00640 1 --2- 172.21.15.6:0/4270835416 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7febc80cce90 0x7febc815c570 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:29.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.360+0000 7febc7e00640 1 --2- 172.21.15.6:0/4270835416 >> v2:172.21.15.146:3300/0 conn(0x7febc8070f10 0x7febc80cc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:29.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.360+0000 7febc7e00640 1 -- 172.21.15.6:0/4270835416 >> 172.21.15.6:0/4270835416 conn(0x7febc806b3d0 msgr2=0x7febc806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:29.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.360+0000 7febc7e00640 1 -- 172.21.15.6:0/4270835416 shutdown_connections 2024-03-04T09:25:29.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.360+0000 7febc7e00640 1 -- 172.21.15.6:0/4270835416 wait complete. 2024-03-04T09:25:29.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.361+0000 7febc7e00640 1 Processor -- start 2024-03-04T09:25:29.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.362+0000 7febc7e00640 1 -- start start 2024-03-04T09:25:29.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.363+0000 7febc7e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7febc8070f10 0x7febc815b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:29.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.363+0000 7febc7e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7febc80cce90 0x7febc80c6af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:29.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.363+0000 7febc7e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7febc8145300 0x7febc80c7030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:29.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.363+0000 7febc7e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febc815cab0 con 0x7febc8145300 2024-03-04T09:25:29.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.363+0000 7febc7e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febc81456e0 con 0x7febc8070f10 2024-03-04T09:25:29.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.363+0000 7febc7e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febc815cc30 con 0x7febc80cce90 2024-03-04T09:25:29.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.363+0000 7febc7400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7febc8145300 0x7febc80c7030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:29.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.363+0000 7febc7400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7febc8145300 0x7febc80c7030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45680/0 (socket says 172.21.15.6:45680) 2024-03-04T09:25:29.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.363+0000 7febc7400640 1 -- 172.21.15.6:0/557060508 learned_addr learned my addr 172.21.15.6:0/557060508 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:25:29.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.363+0000 7febc6a00640 1 --2- 172.21.15.6:0/557060508 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7febc8070f10 0x7febc815b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:29.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.363+0000 7febc6000640 1 --2- 172.21.15.6:0/557060508 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7febc80cce90 0x7febc80c6af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:29.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.363+0000 7febc7400640 1 -- 172.21.15.6:0/557060508 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7febc80cce90 msgr2=0x7febc80c6af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:29.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.363+0000 7febc7400640 1 --2- 172.21.15.6:0/557060508 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7febc80cce90 0x7febc80c6af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:29.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.363+0000 7febc7400640 1 -- 172.21.15.6:0/557060508 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7febc8070f10 msgr2=0x7febc815b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:29.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.363+0000 7febc7400640 1 --2- 172.21.15.6:0/557060508 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7febc8070f10 0x7febc815b2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:29.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.363+0000 7febc7400640 1 -- 172.21.15.6:0/557060508 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febc8141150 con 0x7febc8145300 2024-03-04T09:25:29.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.364+0000 7febc7400640 1 --2- 172.21.15.6:0/557060508 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7febc8145300 0x7febc80c7030 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7febc0009e80 tx=0x7febc0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:29.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.364+0000 7febbea00640 1 -- 172.21.15.6:0/557060508 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febc0039050 con 0x7febc8145300 2024-03-04T09:25:29.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.364+0000 7febbea00640 1 -- 172.21.15.6:0/557060508 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febc0045070 con 0x7febc8145300 2024-03-04T09:25:29.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.364+0000 7febbea00640 1 -- 172.21.15.6:0/557060508 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febc003e830 con 0x7febc8145300 2024-03-04T09:25:29.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.364+0000 7febc7e00640 1 -- 172.21.15.6:0/557060508 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febc8141430 con 0x7febc8145300 2024-03-04T09:25:29.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.364+0000 7febc7e00640 1 -- 172.21.15.6:0/557060508 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febc81419f0 con 0x7febc8145300 2024-03-04T09:25:29.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.365+0000 7febc7e00640 1 -- 172.21.15.6:0/557060508 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febc804f330 con 0x7febc8145300 2024-03-04T09:25:29.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.366+0000 7febbea00640 1 -- 172.21.15.6:0/557060508 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7febc003e9d0 con 0x7febc8145300 2024-03-04T09:25:29.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.367+0000 7febbea00640 1 --2- 172.21.15.6:0/557060508 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7feba806b4d0 0x7feba806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:29.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.368+0000 7febbea00640 1 -- 172.21.15.6:0/557060508 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7febc00ad270 con 0x7febc8145300 2024-03-04T09:25:29.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.368+0000 7febc6a00640 1 --2- 172.21.15.6:0/557060508 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7feba806b4d0 0x7feba806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:29.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.369+0000 7febc6a00640 1 --2- 172.21.15.6:0/557060508 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7feba806b4d0 0x7feba806d990 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7febb8000f60 tx=0x7febb8007c00 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:29.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.372+0000 7febbea00640 1 -- 172.21.15.6:0/557060508 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7febc007c210 con 0x7febc8145300 2024-03-04T09:25:29.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.560+0000 7febc7e00640 1 -- 172.21.15.6:0/557060508 --> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7febc815c690 con 0x7feba806b4d0 2024-03-04T09:25:29.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.583+0000 7febbea00640 1 -- 172.21.15.6:0/557060508 <== mgr.4113 v2:172.21.15.146:6832/1788613334 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251836 (secure 0 0 0) 0x7febc815c690 con 0x7feba806b4d0 2024-03-04T09:25:29.581 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:25:29.585 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-03-04T09:25:29.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.588+0000 7febc7e00640 1 -- 172.21.15.6:0/557060508 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7feba806b4d0 msgr2=0x7feba806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:29.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.588+0000 7febc7e00640 1 --2- 172.21.15.6:0/557060508 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7feba806b4d0 0x7feba806d990 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7febb8000f60 tx=0x7febb8007c00 comp rx=0 tx=0).stop 2024-03-04T09:25:29.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.588+0000 7febc7e00640 1 -- 172.21.15.6:0/557060508 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7febc8145300 msgr2=0x7febc80c7030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:29.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.588+0000 7febc7e00640 1 --2- 172.21.15.6:0/557060508 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7febc8145300 0x7febc80c7030 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7febc0009e80 tx=0x7febc0004a40 comp rx=0 tx=0).stop 2024-03-04T09:25:29.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.589+0000 7febc7e00640 1 -- 172.21.15.6:0/557060508 shutdown_connections 2024-03-04T09:25:29.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.589+0000 7febc7e00640 1 --2- 172.21.15.6:0/557060508 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7febc8145300 0x7febc80c7030 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:29.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.589+0000 7febc7e00640 1 --2- 172.21.15.6:0/557060508 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7feba806b4d0 0x7feba806d990 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:29.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.589+0000 7febc7e00640 1 --2- 172.21.15.6:0/557060508 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7febc80cce90 0x7febc80c6af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:29.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.589+0000 7febc7e00640 1 --2- 172.21.15.6:0/557060508 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7febc8070f10 0x7febc815b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:29.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.589+0000 7febc7e00640 1 -- 172.21.15.6:0/557060508 >> 172.21.15.6:0/557060508 conn(0x7febc806b3d0 msgr2=0x7febc8072680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:29.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.589+0000 7febc7e00640 1 -- 172.21.15.6:0/557060508 shutdown_connections 2024-03-04T09:25:29.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.589+0000 7febc7e00640 1 -- 172.21.15.6:0/557060508 wait complete. 2024-03-04T09:25:29.603 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-03-04T09:25:27.861453+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218988,"kb_used_data":2824,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755732,"statfs":{"total":773094113280,"available":772869869568,"internally_reserved":0,"allocated":2891776,"data_stored":1753028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.433118"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.152203+0000","last_change":"2024-03-04T09:25:19.480195+0000","last_active":"2024-03-04T09:25:21.152203+0000","last_peered":"2024-03-04T09:25:21.152203+0000","last_clean":"2024-03-04T09:25:21.152203+0000","last_became_active":"2024-03-04T09:25:19.479496+0000","last_became_peered":"2024-03-04T09:25:19.479496+0000","last_unstale":"2024-03-04T09:25:21.152203+0000","last_undegraded":"2024-03-04T09:25:21.152203+0000","last_fullsized":"2024-03-04T09:25:21.152203+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:36:55.714408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.457360+0000","last_change":"2024-03-04T09:25:19.457568+0000","last_active":"2024-03-04T09:25:20.457360+0000","last_peered":"2024-03-04T09:25:20.457360+0000","last_clean":"2024-03-04T09:25:20.457360+0000","last_became_active":"2024-03-04T09:25:19.457288+0000","last_became_peered":"2024-03-04T09:25:19.457288+0000","last_unstale":"2024-03-04T09:25:20.457360+0000","last_undegraded":"2024-03-04T09:25:20.457360+0000","last_fullsized":"2024-03-04T09:25:20.457360+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:33:16.368258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.149774+0000","last_change":"2024-03-04T09:25:19.465855+0000","last_active":"2024-03-04T09:25:21.149774+0000","last_peered":"2024-03-04T09:25:21.149774+0000","last_clean":"2024-03-04T09:25:21.149774+0000","last_became_active":"2024-03-04T09:25:19.465489+0000","last_became_peered":"2024-03-04T09:25:19.465489+0000","last_unstale":"2024-03-04T09:25:21.149774+0000","last_undegraded":"2024-03-04T09:25:21.149774+0000","last_fullsized":"2024-03-04T09:25:21.149774+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:05:57.624897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.497998+0000","last_change":"2024-03-04T09:25:19.472849+0000","last_active":"2024-03-04T09:25:20.497998+0000","last_peered":"2024-03-04T09:25:20.497998+0000","last_clean":"2024-03-04T09:25:20.497998+0000","last_became_active":"2024-03-04T09:25:19.468252+0000","last_became_peered":"2024-03-04T09:25:19.468252+0000","last_unstale":"2024-03-04T09:25:20.497998+0000","last_undegraded":"2024-03-04T09:25:20.497998+0000","last_fullsized":"2024-03-04T09:25:20.497998+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:20:25.135538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.151031+0000","last_change":"2024-03-04T09:25:19.471747+0000","last_active":"2024-03-04T09:25:21.151031+0000","last_peered":"2024-03-04T09:25:21.151031+0000","last_clean":"2024-03-04T09:25:21.151031+0000","last_became_active":"2024-03-04T09:25:19.469589+0000","last_became_peered":"2024-03-04T09:25:19.469589+0000","last_unstale":"2024-03-04T09:25:21.151031+0000","last_undegraded":"2024-03-04T09:25:21.151031+0000","last_fullsized":"2024-03-04T09:25:21.151031+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:36:58.654226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.497539+0000","last_change":"2024-03-04T09:25:19.458354+0000","last_active":"2024-03-04T09:25:20.497539+0000","last_peered":"2024-03-04T09:25:20.497539+0000","last_clean":"2024-03-04T09:25:20.497539+0000","last_became_active":"2024-03-04T09:25:19.456693+0000","last_became_peered":"2024-03-04T09:25:19.456693+0000","last_unstale":"2024-03-04T09:25:20.497539+0000","last_undegraded":"2024-03-04T09:25:20.497539+0000","last_fullsized":"2024-03-04T09:25:20.497539+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:11:11.501102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.150488+0000","last_change":"2024-03-04T09:25:19.476465+0000","last_active":"2024-03-04T09:25:21.150488+0000","last_peered":"2024-03-04T09:25:21.150488+0000","last_clean":"2024-03-04T09:25:21.150488+0000","last_became_active":"2024-03-04T09:25:19.471519+0000","last_became_peered":"2024-03-04T09:25:19.471519+0000","last_unstale":"2024-03-04T09:25:21.150488+0000","last_undegraded":"2024-03-04T09:25:21.150488+0000","last_fullsized":"2024-03-04T09:25:21.150488+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:01:33.498760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.455119+0000","last_change":"2024-03-04T09:25:19.471301+0000","last_active":"2024-03-04T09:25:20.455119+0000","last_peered":"2024-03-04T09:25:20.455119+0000","last_clean":"2024-03-04T09:25:20.455119+0000","last_became_active":"2024-03-04T09:25:19.470983+0000","last_became_peered":"2024-03-04T09:25:19.470983+0000","last_unstale":"2024-03-04T09:25:20.455119+0000","last_undegraded":"2024-03-04T09:25:20.455119+0000","last_fullsized":"2024-03-04T09:25:20.455119+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:50:18.265156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.152495+0000","last_change":"2024-03-04T09:25:19.485751+0000","last_active":"2024-03-04T09:25:21.152495+0000","last_peered":"2024-03-04T09:25:21.152495+0000","last_clean":"2024-03-04T09:25:21.152495+0000","last_became_active":"2024-03-04T09:25:19.485488+0000","last_became_peered":"2024-03-04T09:25:19.485488+0000","last_unstale":"2024-03-04T09:25:21.152495+0000","last_undegraded":"2024-03-04T09:25:21.152495+0000","last_fullsized":"2024-03-04T09:25:21.152495+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:45:23.780438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.464582+0000","last_change":"2024-03-04T09:25:19.466385+0000","last_active":"2024-03-04T09:25:20.464582+0000","last_peered":"2024-03-04T09:25:20.464582+0000","last_clean":"2024-03-04T09:25:20.464582+0000","last_became_active":"2024-03-04T09:25:19.466078+0000","last_became_peered":"2024-03-04T09:25:19.466078+0000","last_unstale":"2024-03-04T09:25:20.464582+0000","last_undegraded":"2024-03-04T09:25:20.464582+0000","last_fullsized":"2024-03-04T09:25:20.464582+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:56:18.833810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.464502+0000","last_change":"2024-03-04T09:25:19.458668+0000","last_active":"2024-03-04T09:25:20.464502+0000","last_peered":"2024-03-04T09:25:20.464502+0000","last_clean":"2024-03-04T09:25:20.464502+0000","last_became_active":"2024-03-04T09:25:19.458080+0000","last_became_peered":"2024-03-04T09:25:19.458080+0000","last_unstale":"2024-03-04T09:25:20.464502+0000","last_undegraded":"2024-03-04T09:25:20.464502+0000","last_fullsized":"2024-03-04T09:25:20.464502+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:44:37.465937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.152136+0000","last_change":"2024-03-04T09:25:19.470591+0000","last_active":"2024-03-04T09:25:21.152136+0000","last_peered":"2024-03-04T09:25:21.152136+0000","last_clean":"2024-03-04T09:25:21.152136+0000","last_became_active":"2024-03-04T09:25:19.463604+0000","last_became_peered":"2024-03-04T09:25:19.463604+0000","last_unstale":"2024-03-04T09:25:21.152136+0000","last_undegraded":"2024-03-04T09:25:21.152136+0000","last_fullsized":"2024-03-04T09:25:21.152136+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:02:45.778608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.456030+0000","last_change":"2024-03-04T09:25:19.462024+0000","last_active":"2024-03-04T09:25:20.456030+0000","last_peered":"2024-03-04T09:25:20.456030+0000","last_clean":"2024-03-04T09:25:20.456030+0000","last_became_active":"2024-03-04T09:25:19.461767+0000","last_became_peered":"2024-03-04T09:25:19.461767+0000","last_unstale":"2024-03-04T09:25:20.456030+0000","last_undegraded":"2024-03-04T09:25:20.456030+0000","last_fullsized":"2024-03-04T09:25:20.456030+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:47:26.445286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.149510+0000","last_change":"2024-03-04T09:25:19.468514+0000","last_active":"2024-03-04T09:25:21.149510+0000","last_peered":"2024-03-04T09:25:21.149510+0000","last_clean":"2024-03-04T09:25:21.149510+0000","last_became_active":"2024-03-04T09:25:19.468029+0000","last_became_peered":"2024-03-04T09:25:19.468029+0000","last_unstale":"2024-03-04T09:25:21.149510+0000","last_undegraded":"2024-03-04T09:25:21.149510+0000","last_fullsized":"2024-03-04T09:25:21.149510+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:45:15.160192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.455915+0000","last_change":"2024-03-04T09:25:19.448212+0000","last_active":"2024-03-04T09:25:20.455915+0000","last_peered":"2024-03-04T09:25:20.455915+0000","last_clean":"2024-03-04T09:25:20.455915+0000","last_became_active":"2024-03-04T09:25:19.447824+0000","last_became_peered":"2024-03-04T09:25:19.447824+0000","last_unstale":"2024-03-04T09:25:20.455915+0000","last_undegraded":"2024-03-04T09:25:20.455915+0000","last_fullsized":"2024-03-04T09:25:20.455915+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:44:37.465937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.497265+0000","last_change":"2024-03-04T09:25:19.471682+0000","last_active":"2024-03-04T09:25:20.497265+0000","last_peered":"2024-03-04T09:25:20.497265+0000","last_clean":"2024-03-04T09:25:20.497265+0000","last_became_active":"2024-03-04T09:25:19.471416+0000","last_became_peered":"2024-03-04T09:25:19.471416+0000","last_unstale":"2024-03-04T09:25:20.497265+0000","last_undegraded":"2024-03-04T09:25:20.497265+0000","last_fullsized":"2024-03-04T09:25:20.497265+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:01:33.498760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.499060+0000","last_change":"2024-03-04T09:25:19.484010+0000","last_active":"2024-03-04T09:25:20.499060+0000","last_peered":"2024-03-04T09:25:20.499060+0000","last_clean":"2024-03-04T09:25:20.499060+0000","last_became_active":"2024-03-04T09:25:19.477995+0000","last_became_peered":"2024-03-04T09:25:19.477995+0000","last_unstale":"2024-03-04T09:25:20.499060+0000","last_undegraded":"2024-03-04T09:25:20.499060+0000","last_fullsized":"2024-03-04T09:25:20.499060+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:36:31.167806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.458074+0000","last_change":"2024-03-04T09:25:19.456238+0000","last_active":"2024-03-04T09:25:20.458074+0000","last_peered":"2024-03-04T09:25:20.458074+0000","last_clean":"2024-03-04T09:25:20.458074+0000","last_became_active":"2024-03-04T09:25:19.455908+0000","last_became_peered":"2024-03-04T09:25:19.455908+0000","last_unstale":"2024-03-04T09:25:20.458074+0000","last_undegraded":"2024-03-04T09:25:20.458074+0000","last_fullsized":"2024-03-04T09:25:20.458074+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:32:59.918382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.151921+0000","last_change":"2024-03-04T09:25:19.479250+0000","last_active":"2024-03-04T09:25:21.151921+0000","last_peered":"2024-03-04T09:25:21.151921+0000","last_clean":"2024-03-04T09:25:21.151921+0000","last_became_active":"2024-03-04T09:25:19.478971+0000","last_became_peered":"2024-03-04T09:25:19.478971+0000","last_unstale":"2024-03-04T09:25:21.151921+0000","last_undegraded":"2024-03-04T09:25:21.151921+0000","last_fullsized":"2024-03-04T09:25:21.151921+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:12:45.452669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.464032+0000","last_change":"2024-03-04T09:25:19.471817+0000","last_active":"2024-03-04T09:25:20.464032+0000","last_peered":"2024-03-04T09:25:20.464032+0000","last_clean":"2024-03-04T09:25:20.464032+0000","last_became_active":"2024-03-04T09:25:19.471551+0000","last_became_peered":"2024-03-04T09:25:19.471551+0000","last_unstale":"2024-03-04T09:25:20.464032+0000","last_undegraded":"2024-03-04T09:25:20.464032+0000","last_fullsized":"2024-03-04T09:25:20.464032+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:45:59.096795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.149784+0000","last_change":"2024-03-04T09:25:19.464870+0000","last_active":"2024-03-04T09:25:21.149784+0000","last_peered":"2024-03-04T09:25:21.149784+0000","last_clean":"2024-03-04T09:25:21.149784+0000","last_became_active":"2024-03-04T09:25:19.464544+0000","last_became_peered":"2024-03-04T09:25:19.464544+0000","last_unstale":"2024-03-04T09:25:21.149784+0000","last_undegraded":"2024-03-04T09:25:21.149784+0000","last_fullsized":"2024-03-04T09:25:21.149784+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:45:59.096795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.497519+0000","last_change":"2024-03-04T09:25:19.459890+0000","last_active":"2024-03-04T09:25:20.497519+0000","last_peered":"2024-03-04T09:25:20.497519+0000","last_clean":"2024-03-04T09:25:20.497519+0000","last_became_active":"2024-03-04T09:25:19.459528+0000","last_became_peered":"2024-03-04T09:25:19.459528+0000","last_unstale":"2024-03-04T09:25:20.497519+0000","last_undegraded":"2024-03-04T09:25:20.497519+0000","last_fullsized":"2024-03-04T09:25:20.497519+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:36:58.654226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.456364+0000","last_change":"2024-03-04T09:25:19.462338+0000","last_active":"2024-03-04T09:25:20.456364+0000","last_peered":"2024-03-04T09:25:20.456364+0000","last_clean":"2024-03-04T09:25:20.456364+0000","last_became_active":"2024-03-04T09:25:19.461998+0000","last_became_peered":"2024-03-04T09:25:19.461998+0000","last_unstale":"2024-03-04T09:25:20.456364+0000","last_undegraded":"2024-03-04T09:25:20.456364+0000","last_fullsized":"2024-03-04T09:25:20.456364+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:56:18.833810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.497149+0000","last_change":"2024-03-04T09:25:19.473080+0000","last_active":"2024-03-04T09:25:20.497149+0000","last_peered":"2024-03-04T09:25:20.497149+0000","last_clean":"2024-03-04T09:25:20.497149+0000","last_became_active":"2024-03-04T09:25:19.467315+0000","last_became_peered":"2024-03-04T09:25:19.467315+0000","last_unstale":"2024-03-04T09:25:20.497149+0000","last_undegraded":"2024-03-04T09:25:20.497149+0000","last_fullsized":"2024-03-04T09:25:20.497149+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:33:16.368258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.458010+0000","last_change":"2024-03-04T09:25:19.472788+0000","last_active":"2024-03-04T09:25:20.458010+0000","last_peered":"2024-03-04T09:25:20.458010+0000","last_clean":"2024-03-04T09:25:20.458010+0000","last_became_active":"2024-03-04T09:25:19.472496+0000","last_became_peered":"2024-03-04T09:25:19.472496+0000","last_unstale":"2024-03-04T09:25:20.458010+0000","last_undegraded":"2024-03-04T09:25:20.458010+0000","last_fullsized":"2024-03-04T09:25:20.458010+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:01:33.498760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.464735+0000","last_change":"2024-03-04T09:25:19.461265+0000","last_active":"2024-03-04T09:25:20.464735+0000","last_peered":"2024-03-04T09:25:20.464735+0000","last_clean":"2024-03-04T09:25:20.464735+0000","last_became_active":"2024-03-04T09:25:19.458576+0000","last_became_peered":"2024-03-04T09:25:19.458576+0000","last_unstale":"2024-03-04T09:25:20.464735+0000","last_undegraded":"2024-03-04T09:25:20.464735+0000","last_fullsized":"2024-03-04T09:25:20.464735+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:33:58.076780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.456949+0000","last_change":"2024-03-04T09:25:19.467165+0000","last_active":"2024-03-04T09:25:20.456949+0000","last_peered":"2024-03-04T09:25:20.456949+0000","last_clean":"2024-03-04T09:25:20.456949+0000","last_became_active":"2024-03-04T09:25:19.466793+0000","last_became_peered":"2024-03-04T09:25:19.466793+0000","last_unstale":"2024-03-04T09:25:20.456949+0000","last_undegraded":"2024-03-04T09:25:20.456949+0000","last_fullsized":"2024-03-04T09:25:20.456949+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:33:58.076780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.457770+0000","last_change":"2024-03-04T09:25:19.474283+0000","last_active":"2024-03-04T09:25:20.457770+0000","last_peered":"2024-03-04T09:25:20.457770+0000","last_clean":"2024-03-04T09:25:20.457770+0000","last_became_active":"2024-03-04T09:25:19.474124+0000","last_became_peered":"2024-03-04T09:25:19.474124+0000","last_unstale":"2024-03-04T09:25:20.457770+0000","last_undegraded":"2024-03-04T09:25:20.457770+0000","last_fullsized":"2024-03-04T09:25:20.457770+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:33:16.368258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.150029+0000","last_change":"2024-03-04T09:25:19.461996+0000","last_active":"2024-03-04T09:25:21.150029+0000","last_peered":"2024-03-04T09:25:21.150029+0000","last_clean":"2024-03-04T09:25:21.150029+0000","last_became_active":"2024-03-04T09:25:19.461683+0000","last_became_peered":"2024-03-04T09:25:19.461683+0000","last_unstale":"2024-03-04T09:25:21.150029+0000","last_undegraded":"2024-03-04T09:25:21.150029+0000","last_fullsized":"2024-03-04T09:25:21.150029+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:47:26.445286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.497702+0000","last_change":"2024-03-04T09:25:19.466189+0000","last_active":"2024-03-04T09:25:20.497702+0000","last_peered":"2024-03-04T09:25:20.497702+0000","last_clean":"2024-03-04T09:25:20.497702+0000","last_became_active":"2024-03-04T09:25:19.465903+0000","last_became_peered":"2024-03-04T09:25:19.465903+0000","last_unstale":"2024-03-04T09:25:20.497702+0000","last_undegraded":"2024-03-04T09:25:20.497702+0000","last_fullsized":"2024-03-04T09:25:20.497702+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:04:48.675219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.457411+0000","last_change":"2024-03-04T09:25:19.458477+0000","last_active":"2024-03-04T09:25:20.457411+0000","last_peered":"2024-03-04T09:25:20.457411+0000","last_clean":"2024-03-04T09:25:20.457411+0000","last_became_active":"2024-03-04T09:25:19.455994+0000","last_became_peered":"2024-03-04T09:25:19.455994+0000","last_unstale":"2024-03-04T09:25:20.457411+0000","last_undegraded":"2024-03-04T09:25:20.457411+0000","last_fullsized":"2024-03-04T09:25:20.457411+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:11:52.240719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.463964+0000","last_change":"2024-03-04T09:25:19.464315+0000","last_active":"2024-03-04T09:25:20.463964+0000","last_peered":"2024-03-04T09:25:20.463964+0000","last_clean":"2024-03-04T09:25:20.463964+0000","last_became_active":"2024-03-04T09:25:19.463942+0000","last_became_peered":"2024-03-04T09:25:19.463942+0000","last_unstale":"2024-03-04T09:25:20.463964+0000","last_undegraded":"2024-03-04T09:25:20.463964+0000","last_fullsized":"2024-03-04T09:25:20.463964+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:05:57.624897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.149802+0000","last_change":"2024-03-04T09:25:19.448874+0000","last_active":"2024-03-04T09:25:21.149802+0000","last_peered":"2024-03-04T09:25:21.149802+0000","last_clean":"2024-03-04T09:25:21.149802+0000","last_became_active":"2024-03-04T09:25:19.445047+0000","last_became_peered":"2024-03-04T09:25:19.445047+0000","last_unstale":"2024-03-04T09:25:21.149802+0000","last_undegraded":"2024-03-04T09:25:21.149802+0000","last_fullsized":"2024-03-04T09:25:21.149802+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:23:35.947152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.456151+0000","last_change":"2024-03-04T09:25:18.469771+0000","last_active":"2024-03-04T09:25:20.456151+0000","last_peered":"2024-03-04T09:25:20.456151+0000","last_clean":"2024-03-04T09:25:20.456151+0000","last_became_active":"2024-03-04T09:25:18.469445+0000","last_became_peered":"2024-03-04T09:25:18.469445+0000","last_unstale":"2024-03-04T09:25:20.456151+0000","last_undegraded":"2024-03-04T09:25:20.456151+0000","last_fullsized":"2024-03-04T09:25:20.456151+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:39:20.774577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.150269+0000","last_change":"2024-03-04T09:25:19.463486+0000","last_active":"2024-03-04T09:25:21.150269+0000","last_peered":"2024-03-04T09:25:21.150269+0000","last_clean":"2024-03-04T09:25:21.150269+0000","last_became_active":"2024-03-04T09:25:19.463204+0000","last_became_peered":"2024-03-04T09:25:19.463204+0000","last_unstale":"2024-03-04T09:25:21.150269+0000","last_undegraded":"2024-03-04T09:25:21.150269+0000","last_fullsized":"2024-03-04T09:25:21.150269+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:33:58.076780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.463976+0000","last_change":"2024-03-04T09:25:18.472794+0000","last_active":"2024-03-04T09:25:20.463976+0000","last_peered":"2024-03-04T09:25:20.463976+0000","last_clean":"2024-03-04T09:25:20.463976+0000","last_became_active":"2024-03-04T09:25:18.472449+0000","last_became_peered":"2024-03-04T09:25:18.472449+0000","last_unstale":"2024-03-04T09:25:20.463976+0000","last_undegraded":"2024-03-04T09:25:20.463976+0000","last_fullsized":"2024-03-04T09:25:20.463976+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:23:34.932850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.456455+0000","last_change":"2024-03-04T09:25:18.474050+0000","last_active":"2024-03-04T09:25:20.456455+0000","last_peered":"2024-03-04T09:25:20.456455+0000","last_clean":"2024-03-04T09:25:20.456455+0000","last_became_active":"2024-03-04T09:25:18.473760+0000","last_became_peered":"2024-03-04T09:25:18.473760+0000","last_unstale":"2024-03-04T09:25:20.456455+0000","last_undegraded":"2024-03-04T09:25:20.456455+0000","last_fullsized":"2024-03-04T09:25:20.456455+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:31:15.318756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.455119+0000","last_change":"2024-03-04T09:25:19.464447+0000","last_active":"2024-03-04T09:25:20.455119+0000","last_peered":"2024-03-04T09:25:20.455119+0000","last_clean":"2024-03-04T09:25:20.455119+0000","last_became_active":"2024-03-04T09:25:19.463201+0000","last_became_peered":"2024-03-04T09:25:19.463201+0000","last_unstale":"2024-03-04T09:25:20.455119+0000","last_undegraded":"2024-03-04T09:25:20.455119+0000","last_fullsized":"2024-03-04T09:25:20.455119+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:05:57.624897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.497223+0000","last_change":"2024-03-04T09:25:18.476149+0000","last_active":"2024-03-04T09:25:20.497223+0000","last_peered":"2024-03-04T09:25:20.497223+0000","last_clean":"2024-03-04T09:25:20.497223+0000","last_became_active":"2024-03-04T09:25:18.475664+0000","last_became_peered":"2024-03-04T09:25:18.475664+0000","last_unstale":"2024-03-04T09:25:20.497223+0000","last_undegraded":"2024-03-04T09:25:20.497223+0000","last_fullsized":"2024-03-04T09:25:20.497223+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:33:35.151096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.455640+0000","last_change":"2024-03-04T09:25:19.467051+0000","last_active":"2024-03-04T09:25:20.455640+0000","last_peered":"2024-03-04T09:25:20.455640+0000","last_clean":"2024-03-04T09:25:20.455640+0000","last_became_active":"2024-03-04T09:25:19.466755+0000","last_became_peered":"2024-03-04T09:25:19.466755+0000","last_unstale":"2024-03-04T09:25:20.455640+0000","last_undegraded":"2024-03-04T09:25:20.455640+0000","last_fullsized":"2024-03-04T09:25:20.455640+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:47:26.445286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.464438+0000","last_change":"2024-03-04T09:25:18.479294+0000","last_active":"2024-03-04T09:25:20.464438+0000","last_peered":"2024-03-04T09:25:20.464438+0000","last_clean":"2024-03-04T09:25:20.464438+0000","last_became_active":"2024-03-04T09:25:18.478561+0000","last_became_peered":"2024-03-04T09:25:18.478561+0000","last_unstale":"2024-03-04T09:25:20.464438+0000","last_undegraded":"2024-03-04T09:25:20.464438+0000","last_fullsized":"2024-03-04T09:25:20.464438+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:33:35.151096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.498563+0000","last_change":"2024-03-04T09:25:19.459933+0000","last_active":"2024-03-04T09:25:20.498563+0000","last_peered":"2024-03-04T09:25:20.498563+0000","last_clean":"2024-03-04T09:25:20.498563+0000","last_became_active":"2024-03-04T09:25:19.459622+0000","last_became_peered":"2024-03-04T09:25:19.459622+0000","last_unstale":"2024-03-04T09:25:20.498563+0000","last_undegraded":"2024-03-04T09:25:20.498563+0000","last_fullsized":"2024-03-04T09:25:20.498563+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:45:23.780438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.498673+0000","last_change":"2024-03-04T09:25:18.476726+0000","last_active":"2024-03-04T09:25:20.498673+0000","last_peered":"2024-03-04T09:25:20.498673+0000","last_clean":"2024-03-04T09:25:20.498673+0000","last_became_active":"2024-03-04T09:25:18.475636+0000","last_became_peered":"2024-03-04T09:25:18.475636+0000","last_unstale":"2024-03-04T09:25:20.498673+0000","last_undegraded":"2024-03-04T09:25:20.498673+0000","last_fullsized":"2024-03-04T09:25:20.498673+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:54:01.985207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.150507+0000","last_change":"2024-03-04T09:25:19.473920+0000","last_active":"2024-03-04T09:25:21.150507+0000","last_peered":"2024-03-04T09:25:21.150507+0000","last_clean":"2024-03-04T09:25:21.150507+0000","last_became_active":"2024-03-04T09:25:19.473457+0000","last_became_peered":"2024-03-04T09:25:19.473457+0000","last_unstale":"2024-03-04T09:25:21.150507+0000","last_undegraded":"2024-03-04T09:25:21.150507+0000","last_fullsized":"2024-03-04T09:25:21.150507+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:33:16.368258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.457014+0000","last_change":"2024-03-04T09:25:18.471507+0000","last_active":"2024-03-04T09:25:20.457014+0000","last_peered":"2024-03-04T09:25:20.457014+0000","last_clean":"2024-03-04T09:25:20.457014+0000","last_became_active":"2024-03-04T09:25:18.470138+0000","last_became_peered":"2024-03-04T09:25:18.470138+0000","last_unstale":"2024-03-04T09:25:20.457014+0000","last_undegraded":"2024-03-04T09:25:20.457014+0000","last_fullsized":"2024-03-04T09:25:20.457014+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:54:01.985207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.151586+0000","last_change":"2024-03-04T09:25:19.471146+0000","last_active":"2024-03-04T09:25:21.151586+0000","last_peered":"2024-03-04T09:25:21.151586+0000","last_clean":"2024-03-04T09:25:21.151586+0000","last_became_active":"2024-03-04T09:25:19.470904+0000","last_became_peered":"2024-03-04T09:25:19.470904+0000","last_unstale":"2024-03-04T09:25:21.151586+0000","last_undegraded":"2024-03-04T09:25:21.151586+0000","last_fullsized":"2024-03-04T09:25:21.151586+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:32:59.918382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"19'6","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.463922+0000","last_change":"2024-03-04T09:25:18.474956+0000","last_active":"2024-03-04T09:25:20.463922+0000","last_peered":"2024-03-04T09:25:20.463922+0000","last_clean":"2024-03-04T09:25:20.463922+0000","last_became_active":"2024-03-04T09:25:18.474011+0000","last_became_peered":"2024-03-04T09:25:18.474011+0000","last_unstale":"2024-03-04T09:25:20.463922+0000","last_undegraded":"2024-03-04T09:25:20.463922+0000","last_fullsized":"2024-03-04T09:25:20.463922+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:54:01.985207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.151254+0000","last_change":"2024-03-04T09:25:19.477899+0000","last_active":"2024-03-04T09:25:21.151254+0000","last_peered":"2024-03-04T09:25:21.151254+0000","last_clean":"2024-03-04T09:25:21.151254+0000","last_became_active":"2024-03-04T09:25:19.477117+0000","last_became_peered":"2024-03-04T09:25:19.477117+0000","last_unstale":"2024-03-04T09:25:21.151254+0000","last_undegraded":"2024-03-04T09:25:21.151254+0000","last_fullsized":"2024-03-04T09:25:21.151254+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:04:48.675219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.150191+0000","last_change":"2024-03-04T09:25:18.469651+0000","last_active":"2024-03-04T09:25:21.150191+0000","last_peered":"2024-03-04T09:25:21.150191+0000","last_clean":"2024-03-04T09:25:21.150191+0000","last_became_active":"2024-03-04T09:25:18.469259+0000","last_became_peered":"2024-03-04T09:25:18.469259+0000","last_unstale":"2024-03-04T09:25:21.150191+0000","last_undegraded":"2024-03-04T09:25:21.150191+0000","last_fullsized":"2024-03-04T09:25:21.150191+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:57:52.768999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.149691+0000","last_change":"2024-03-04T09:25:19.465339+0000","last_active":"2024-03-04T09:25:21.149691+0000","last_peered":"2024-03-04T09:25:21.149691+0000","last_clean":"2024-03-04T09:25:21.149691+0000","last_became_active":"2024-03-04T09:25:19.464957+0000","last_became_peered":"2024-03-04T09:25:19.464957+0000","last_unstale":"2024-03-04T09:25:21.149691+0000","last_undegraded":"2024-03-04T09:25:21.149691+0000","last_fullsized":"2024-03-04T09:25:21.149691+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:50:18.265156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.497632+0000","last_change":"2024-03-04T09:25:18.470800+0000","last_active":"2024-03-04T09:25:20.497632+0000","last_peered":"2024-03-04T09:25:20.497632+0000","last_clean":"2024-03-04T09:25:20.497632+0000","last_became_active":"2024-03-04T09:25:18.470431+0000","last_became_peered":"2024-03-04T09:25:18.470431+0000","last_unstale":"2024-03-04T09:25:20.497632+0000","last_undegraded":"2024-03-04T09:25:20.497632+0000","last_fullsized":"2024-03-04T09:25:20.497632+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:39:20.774577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.151318+0000","last_change":"2024-03-04T09:25:19.478795+0000","last_active":"2024-03-04T09:25:21.151318+0000","last_peered":"2024-03-04T09:25:21.151318+0000","last_clean":"2024-03-04T09:25:21.151318+0000","last_became_active":"2024-03-04T09:25:19.478635+0000","last_became_peered":"2024-03-04T09:25:19.478635+0000","last_unstale":"2024-03-04T09:25:21.151318+0000","last_undegraded":"2024-03-04T09:25:21.151318+0000","last_fullsized":"2024-03-04T09:25:21.151318+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:04:20.349240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.463677+0000","last_change":"2024-03-04T09:25:18.474750+0000","last_active":"2024-03-04T09:25:20.463677+0000","last_peered":"2024-03-04T09:25:20.463677+0000","last_clean":"2024-03-04T09:25:20.463677+0000","last_became_active":"2024-03-04T09:25:18.473463+0000","last_became_peered":"2024-03-04T09:25:18.473463+0000","last_unstale":"2024-03-04T09:25:20.463677+0000","last_undegraded":"2024-03-04T09:25:20.463677+0000","last_fullsized":"2024-03-04T09:25:20.463677+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:38:16.906069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.463789+0000","last_change":"2024-03-04T09:25:19.472274+0000","last_active":"2024-03-04T09:25:20.463789+0000","last_peered":"2024-03-04T09:25:20.463789+0000","last_clean":"2024-03-04T09:25:20.463789+0000","last_became_active":"2024-03-04T09:25:19.470669+0000","last_became_peered":"2024-03-04T09:25:19.470669+0000","last_unstale":"2024-03-04T09:25:20.463789+0000","last_undegraded":"2024-03-04T09:25:20.463789+0000","last_fullsized":"2024-03-04T09:25:20.463789+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:50:18.265156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":22,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.456829+0000","last_change":"2024-03-04T09:25:18.462460+0000","last_active":"2024-03-04T09:25:20.456829+0000","last_peered":"2024-03-04T09:25:20.456829+0000","last_clean":"2024-03-04T09:25:20.456829+0000","last_became_active":"2024-03-04T09:25:18.460692+0000","last_became_peered":"2024-03-04T09:25:18.460692+0000","last_unstale":"2024-03-04T09:25:20.456829+0000","last_undegraded":"2024-03-04T09:25:20.456829+0000","last_fullsized":"2024-03-04T09:25:20.456829+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:38:16.906069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.150513+0000","last_change":"2024-03-04T09:25:19.478013+0000","last_active":"2024-03-04T09:25:21.150513+0000","last_peered":"2024-03-04T09:25:21.150513+0000","last_clean":"2024-03-04T09:25:21.150513+0000","last_became_active":"2024-03-04T09:25:19.477150+0000","last_became_peered":"2024-03-04T09:25:19.477150+0000","last_unstale":"2024-03-04T09:25:21.150513+0000","last_undegraded":"2024-03-04T09:25:21.150513+0000","last_fullsized":"2024-03-04T09:25:21.150513+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:11:11.501102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.152408+0000","last_change":"2024-03-04T09:25:18.468798+0000","last_active":"2024-03-04T09:25:21.152408+0000","last_peered":"2024-03-04T09:25:21.152408+0000","last_clean":"2024-03-04T09:25:21.152408+0000","last_became_active":"2024-03-04T09:25:18.468412+0000","last_became_peered":"2024-03-04T09:25:18.468412+0000","last_unstale":"2024-03-04T09:25:21.152408+0000","last_undegraded":"2024-03-04T09:25:21.152408+0000","last_fullsized":"2024-03-04T09:25:21.152408+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:17:37.720962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.151303+0000","last_change":"2024-03-04T09:25:19.478586+0000","last_active":"2024-03-04T09:25:21.151303+0000","last_peered":"2024-03-04T09:25:21.151303+0000","last_clean":"2024-03-04T09:25:21.151303+0000","last_became_active":"2024-03-04T09:25:19.478339+0000","last_became_peered":"2024-03-04T09:25:19.478339+0000","last_unstale":"2024-03-04T09:25:21.151303+0000","last_undegraded":"2024-03-04T09:25:21.151303+0000","last_fullsized":"2024-03-04T09:25:21.151303+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:20:25.135538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.458139+0000","last_change":"2024-03-04T09:25:18.459079+0000","last_active":"2024-03-04T09:25:20.458139+0000","last_peered":"2024-03-04T09:25:20.458139+0000","last_clean":"2024-03-04T09:25:20.458139+0000","last_became_active":"2024-03-04T09:25:18.458767+0000","last_became_peered":"2024-03-04T09:25:18.458767+0000","last_unstale":"2024-03-04T09:25:20.458139+0000","last_undegraded":"2024-03-04T09:25:20.458139+0000","last_fullsized":"2024-03-04T09:25:20.458139+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:33:35.151096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.152708+0000","last_change":"2024-03-04T09:25:19.481294+0000","last_active":"2024-03-04T09:25:21.152708+0000","last_peered":"2024-03-04T09:25:21.152708+0000","last_clean":"2024-03-04T09:25:21.152708+0000","last_became_active":"2024-03-04T09:25:19.479163+0000","last_became_peered":"2024-03-04T09:25:19.479163+0000","last_unstale":"2024-03-04T09:25:21.152708+0000","last_undegraded":"2024-03-04T09:25:21.152708+0000","last_fullsized":"2024-03-04T09:25:21.152708+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:36:31.167806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.457070+0000","last_change":"2024-03-04T09:25:18.464660+0000","last_active":"2024-03-04T09:25:20.457070+0000","last_peered":"2024-03-04T09:25:20.457070+0000","last_clean":"2024-03-04T09:25:20.457070+0000","last_became_active":"2024-03-04T09:25:18.464412+0000","last_became_peered":"2024-03-04T09:25:18.464412+0000","last_unstale":"2024-03-04T09:25:20.457070+0000","last_undegraded":"2024-03-04T09:25:20.457070+0000","last_fullsized":"2024-03-04T09:25:20.457070+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:32:58.904081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.498417+0000","last_change":"2024-03-04T09:25:19.480725+0000","last_active":"2024-03-04T09:25:20.498417+0000","last_peered":"2024-03-04T09:25:20.498417+0000","last_clean":"2024-03-04T09:25:20.498417+0000","last_became_active":"2024-03-04T09:25:19.468626+0000","last_became_peered":"2024-03-04T09:25:19.468626+0000","last_unstale":"2024-03-04T09:25:20.498417+0000","last_undegraded":"2024-03-04T09:25:20.498417+0000","last_fullsized":"2024-03-04T09:25:20.498417+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:36:55.714408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.457915+0000","last_change":"2024-03-04T09:25:18.479022+0000","last_active":"2024-03-04T09:25:20.457915+0000","last_peered":"2024-03-04T09:25:20.457915+0000","last_clean":"2024-03-04T09:25:20.457915+0000","last_became_active":"2024-03-04T09:25:18.478697+0000","last_became_peered":"2024-03-04T09:25:18.478697+0000","last_unstale":"2024-03-04T09:25:20.457915+0000","last_undegraded":"2024-03-04T09:25:20.457915+0000","last_fullsized":"2024-03-04T09:25:20.457915+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:39:20.774577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.151536+0000","last_change":"2024-03-04T09:25:19.485083+0000","last_active":"2024-03-04T09:25:21.151536+0000","last_peered":"2024-03-04T09:25:21.151536+0000","last_clean":"2024-03-04T09:25:21.151536+0000","last_became_active":"2024-03-04T09:25:19.484619+0000","last_became_peered":"2024-03-04T09:25:19.484619+0000","last_unstale":"2024-03-04T09:25:21.151536+0000","last_undegraded":"2024-03-04T09:25:21.151536+0000","last_fullsized":"2024-03-04T09:25:21.151536+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:39:25.865493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":170,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.150247+0000","last_change":"2024-03-04T09:25:09.316159+0000","last_active":"2024-03-04T09:25:21.150247+0000","last_peered":"2024-03-04T09:25:21.150247+0000","last_clean":"2024-03-04T09:25:21.150247+0000","last_became_active":"2024-03-04T09:25:09.313767+0000","last_became_peered":"2024-03-04T09:25:09.313767+0000","last_unstale":"2024-03-04T09:25:21.150247+0000","last_undegraded":"2024-03-04T09:25:21.150247+0000","last_fullsized":"2024-03-04T09:25:21.150247+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:08.110953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:08.110953+0000","last_clean_scrub_stamp":"2024-03-04T09:25:08.110953+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:34:21.745390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.497328+0000","last_change":"2024-03-04T09:25:18.464859+0000","last_active":"2024-03-04T09:25:20.497328+0000","last_peered":"2024-03-04T09:25:20.497328+0000","last_clean":"2024-03-04T09:25:20.497328+0000","last_became_active":"2024-03-04T09:25:18.463515+0000","last_became_peered":"2024-03-04T09:25:18.463515+0000","last_unstale":"2024-03-04T09:25:20.497328+0000","last_undegraded":"2024-03-04T09:25:20.497328+0000","last_fullsized":"2024-03-04T09:25:20.497328+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:38:16.906069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.456773+0000","last_change":"2024-03-04T09:25:19.460896+0000","last_active":"2024-03-04T09:25:20.456773+0000","last_peered":"2024-03-04T09:25:20.456773+0000","last_clean":"2024-03-04T09:25:20.456773+0000","last_became_active":"2024-03-04T09:25:19.460598+0000","last_became_peered":"2024-03-04T09:25:19.460598+0000","last_unstale":"2024-03-04T09:25:20.456773+0000","last_undegraded":"2024-03-04T09:25:20.456773+0000","last_fullsized":"2024-03-04T09:25:20.456773+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:33:58.076780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.149994+0000","last_change":"2024-03-04T09:25:18.480278+0000","last_active":"2024-03-04T09:25:21.149994+0000","last_peered":"2024-03-04T09:25:21.149994+0000","last_clean":"2024-03-04T09:25:21.149994+0000","last_became_active":"2024-03-04T09:25:18.478848+0000","last_became_peered":"2024-03-04T09:25:18.478848+0000","last_unstale":"2024-03-04T09:25:21.149994+0000","last_undegraded":"2024-03-04T09:25:21.149994+0000","last_fullsized":"2024-03-04T09:25:21.149994+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:54:01.985207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.149481+0000","last_change":"2024-03-04T09:25:19.473908+0000","last_active":"2024-03-04T09:25:21.149481+0000","last_peered":"2024-03-04T09:25:21.149481+0000","last_clean":"2024-03-04T09:25:21.149481+0000","last_became_active":"2024-03-04T09:25:19.472415+0000","last_became_peered":"2024-03-04T09:25:19.472415+0000","last_unstale":"2024-03-04T09:25:21.149481+0000","last_undegraded":"2024-03-04T09:25:21.149481+0000","last_fullsized":"2024-03-04T09:25:21.149481+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:46:44.342046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.457503+0000","last_change":"2024-03-04T09:25:18.462877+0000","last_active":"2024-03-04T09:25:20.457503+0000","last_peered":"2024-03-04T09:25:20.457503+0000","last_clean":"2024-03-04T09:25:20.457503+0000","last_became_active":"2024-03-04T09:25:18.459553+0000","last_became_peered":"2024-03-04T09:25:18.459553+0000","last_unstale":"2024-03-04T09:25:20.457503+0000","last_undegraded":"2024-03-04T09:25:20.457503+0000","last_fullsized":"2024-03-04T09:25:20.457503+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:49:03.118927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.456676+0000","last_change":"2024-03-04T09:25:19.452298+0000","last_active":"2024-03-04T09:25:20.456676+0000","last_peered":"2024-03-04T09:25:20.456676+0000","last_clean":"2024-03-04T09:25:20.456676+0000","last_became_active":"2024-03-04T09:25:19.451945+0000","last_became_peered":"2024-03-04T09:25:19.451945+0000","last_unstale":"2024-03-04T09:25:20.456676+0000","last_undegraded":"2024-03-04T09:25:20.456676+0000","last_fullsized":"2024-03-04T09:25:20.456676+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:56:18.833810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.497812+0000","last_change":"2024-03-04T09:25:18.473872+0000","last_active":"2024-03-04T09:25:20.497812+0000","last_peered":"2024-03-04T09:25:20.497812+0000","last_clean":"2024-03-04T09:25:20.497812+0000","last_became_active":"2024-03-04T09:25:18.473390+0000","last_became_peered":"2024-03-04T09:25:18.473390+0000","last_unstale":"2024-03-04T09:25:20.497812+0000","last_undegraded":"2024-03-04T09:25:20.497812+0000","last_fullsized":"2024-03-04T09:25:20.497812+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:23:34.932850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.150539+0000","last_change":"2024-03-04T09:25:19.467593+0000","last_active":"2024-03-04T09:25:21.150539+0000","last_peered":"2024-03-04T09:25:21.150539+0000","last_clean":"2024-03-04T09:25:21.150539+0000","last_became_active":"2024-03-04T09:25:19.467079+0000","last_became_peered":"2024-03-04T09:25:19.467079+0000","last_unstale":"2024-03-04T09:25:21.150539+0000","last_undegraded":"2024-03-04T09:25:21.150539+0000","last_fullsized":"2024-03-04T09:25:21.150539+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:11:52.240719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.456743+0000","last_change":"2024-03-04T09:25:18.465721+0000","last_active":"2024-03-04T09:25:20.456743+0000","last_peered":"2024-03-04T09:25:20.456743+0000","last_clean":"2024-03-04T09:25:20.456743+0000","last_became_active":"2024-03-04T09:25:18.465438+0000","last_became_peered":"2024-03-04T09:25:18.465438+0000","last_unstale":"2024-03-04T09:25:20.456743+0000","last_undegraded":"2024-03-04T09:25:20.456743+0000","last_fullsized":"2024-03-04T09:25:20.456743+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:39:20.774577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.498362+0000","last_change":"2024-03-04T09:25:19.472271+0000","last_active":"2024-03-04T09:25:20.498362+0000","last_peered":"2024-03-04T09:25:20.498362+0000","last_clean":"2024-03-04T09:25:20.498362+0000","last_became_active":"2024-03-04T09:25:19.467596+0000","last_became_peered":"2024-03-04T09:25:19.467596+0000","last_unstale":"2024-03-04T09:25:20.498362+0000","last_undegraded":"2024-03-04T09:25:20.498362+0000","last_fullsized":"2024-03-04T09:25:20.498362+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:12:45.452669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.151714+0000","last_change":"2024-03-04T09:25:18.481607+0000","last_active":"2024-03-04T09:25:21.151714+0000","last_peered":"2024-03-04T09:25:21.151714+0000","last_clean":"2024-03-04T09:25:21.151714+0000","last_became_active":"2024-03-04T09:25:18.481270+0000","last_became_peered":"2024-03-04T09:25:18.481270+0000","last_unstale":"2024-03-04T09:25:21.151714+0000","last_undegraded":"2024-03-04T09:25:21.151714+0000","last_fullsized":"2024-03-04T09:25:21.151714+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:52:36.632272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.455410+0000","last_change":"2024-03-04T09:25:19.450231+0000","last_active":"2024-03-04T09:25:20.455410+0000","last_peered":"2024-03-04T09:25:20.455410+0000","last_clean":"2024-03-04T09:25:20.455410+0000","last_became_active":"2024-03-04T09:25:19.449920+0000","last_became_peered":"2024-03-04T09:25:19.449920+0000","last_unstale":"2024-03-04T09:25:20.455410+0000","last_undegraded":"2024-03-04T09:25:20.455410+0000","last_fullsized":"2024-03-04T09:25:20.455410+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:45:59.096795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.457374+0000","last_change":"2024-03-04T09:25:18.472718+0000","last_active":"2024-03-04T09:25:20.457374+0000","last_peered":"2024-03-04T09:25:20.457374+0000","last_clean":"2024-03-04T09:25:20.457374+0000","last_became_active":"2024-03-04T09:25:18.472426+0000","last_became_peered":"2024-03-04T09:25:18.472426+0000","last_unstale":"2024-03-04T09:25:20.457374+0000","last_undegraded":"2024-03-04T09:25:20.457374+0000","last_fullsized":"2024-03-04T09:25:20.457374+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:23:34.932850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.458202+0000","last_change":"2024-03-04T09:25:19.470531+0000","last_active":"2024-03-04T09:25:20.458202+0000","last_peered":"2024-03-04T09:25:20.458202+0000","last_clean":"2024-03-04T09:25:20.458202+0000","last_became_active":"2024-03-04T09:25:19.469607+0000","last_became_peered":"2024-03-04T09:25:19.469607+0000","last_unstale":"2024-03-04T09:25:20.458202+0000","last_undegraded":"2024-03-04T09:25:20.458202+0000","last_fullsized":"2024-03-04T09:25:20.458202+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:11:11.501102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.464561+0000","last_change":"2024-03-04T09:25:18.473250+0000","last_active":"2024-03-04T09:25:20.464561+0000","last_peered":"2024-03-04T09:25:20.464561+0000","last_clean":"2024-03-04T09:25:20.464561+0000","last_became_active":"2024-03-04T09:25:18.472968+0000","last_became_peered":"2024-03-04T09:25:18.472968+0000","last_unstale":"2024-03-04T09:25:20.464561+0000","last_undegraded":"2024-03-04T09:25:20.464561+0000","last_fullsized":"2024-03-04T09:25:20.464561+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:39:20.774577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.150075+0000","last_change":"2024-03-04T09:25:19.467410+0000","last_active":"2024-03-04T09:25:21.150075+0000","last_peered":"2024-03-04T09:25:21.150075+0000","last_clean":"2024-03-04T09:25:21.150075+0000","last_became_active":"2024-03-04T09:25:19.467101+0000","last_became_peered":"2024-03-04T09:25:19.467101+0000","last_unstale":"2024-03-04T09:25:21.150075+0000","last_undegraded":"2024-03-04T09:25:21.150075+0000","last_fullsized":"2024-03-04T09:25:21.150075+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:44:37.465937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.149988+0000","last_change":"2024-03-04T09:25:18.475507+0000","last_active":"2024-03-04T09:25:21.149988+0000","last_peered":"2024-03-04T09:25:21.149988+0000","last_clean":"2024-03-04T09:25:21.149988+0000","last_became_active":"2024-03-04T09:25:18.475159+0000","last_became_peered":"2024-03-04T09:25:18.475159+0000","last_unstale":"2024-03-04T09:25:21.149988+0000","last_undegraded":"2024-03-04T09:25:21.149988+0000","last_fullsized":"2024-03-04T09:25:21.149988+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:23:34.932850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.149845+0000","last_change":"2024-03-04T09:25:19.453119+0000","last_active":"2024-03-04T09:25:21.149845+0000","last_peered":"2024-03-04T09:25:21.149845+0000","last_clean":"2024-03-04T09:25:21.149845+0000","last_became_active":"2024-03-04T09:25:19.452764+0000","last_became_peered":"2024-03-04T09:25:19.452764+0000","last_unstale":"2024-03-04T09:25:21.149845+0000","last_undegraded":"2024-03-04T09:25:21.149845+0000","last_fullsized":"2024-03-04T09:25:21.149845+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:54:02.999509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.456033+0000","last_change":"2024-03-04T09:25:18.473872+0000","last_active":"2024-03-04T09:25:20.456033+0000","last_peered":"2024-03-04T09:25:20.456033+0000","last_clean":"2024-03-04T09:25:20.456033+0000","last_became_active":"2024-03-04T09:25:18.473554+0000","last_became_peered":"2024-03-04T09:25:18.473554+0000","last_unstale":"2024-03-04T09:25:20.456033+0000","last_undegraded":"2024-03-04T09:25:20.456033+0000","last_fullsized":"2024-03-04T09:25:20.456033+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:11:51.226417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.455488+0000","last_change":"2024-03-04T09:25:19.458923+0000","last_active":"2024-03-04T09:25:20.455488+0000","last_peered":"2024-03-04T09:25:20.455488+0000","last_clean":"2024-03-04T09:25:20.455488+0000","last_became_active":"2024-03-04T09:25:19.458562+0000","last_became_peered":"2024-03-04T09:25:19.458562+0000","last_unstale":"2024-03-04T09:25:20.455488+0000","last_undegraded":"2024-03-04T09:25:20.455488+0000","last_fullsized":"2024-03-04T09:25:20.455488+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:45:59.096795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.456036+0000","last_change":"2024-03-04T09:25:18.458201+0000","last_active":"2024-03-04T09:25:20.456036+0000","last_peered":"2024-03-04T09:25:20.456036+0000","last_clean":"2024-03-04T09:25:20.456036+0000","last_became_active":"2024-03-04T09:25:18.455667+0000","last_became_peered":"2024-03-04T09:25:18.455667+0000","last_unstale":"2024-03-04T09:25:20.456036+0000","last_undegraded":"2024-03-04T09:25:20.456036+0000","last_fullsized":"2024-03-04T09:25:20.456036+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:38:16.906069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.456043+0000","last_change":"2024-03-04T09:25:19.464390+0000","last_active":"2024-03-04T09:25:20.456043+0000","last_peered":"2024-03-04T09:25:20.456043+0000","last_clean":"2024-03-04T09:25:20.456043+0000","last_became_active":"2024-03-04T09:25:19.463511+0000","last_became_peered":"2024-03-04T09:25:19.463511+0000","last_unstale":"2024-03-04T09:25:20.456043+0000","last_undegraded":"2024-03-04T09:25:20.456043+0000","last_fullsized":"2024-03-04T09:25:20.456043+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:44:37.465937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.151481+0000","last_change":"2024-03-04T09:25:18.479732+0000","last_active":"2024-03-04T09:25:21.151481+0000","last_peered":"2024-03-04T09:25:21.151481+0000","last_clean":"2024-03-04T09:25:21.151481+0000","last_became_active":"2024-03-04T09:25:18.478014+0000","last_became_peered":"2024-03-04T09:25:18.478014+0000","last_unstale":"2024-03-04T09:25:21.151481+0000","last_undegraded":"2024-03-04T09:25:21.151481+0000","last_fullsized":"2024-03-04T09:25:21.151481+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:32:58.904081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.464302+0000","last_change":"2024-03-04T09:25:19.455001+0000","last_active":"2024-03-04T09:25:20.464302+0000","last_peered":"2024-03-04T09:25:20.464302+0000","last_clean":"2024-03-04T09:25:20.464302+0000","last_became_active":"2024-03-04T09:25:19.454695+0000","last_became_peered":"2024-03-04T09:25:19.454695+0000","last_unstale":"2024-03-04T09:25:20.464302+0000","last_undegraded":"2024-03-04T09:25:20.464302+0000","last_fullsized":"2024-03-04T09:25:20.464302+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:47:26.445286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.498167+0000","last_change":"2024-03-04T09:25:19.452288+0000","last_active":"2024-03-04T09:25:20.498167+0000","last_peered":"2024-03-04T09:25:20.498167+0000","last_clean":"2024-03-04T09:25:20.498167+0000","last_became_active":"2024-03-04T09:25:19.451982+0000","last_became_peered":"2024-03-04T09:25:19.451982+0000","last_unstale":"2024-03-04T09:25:20.498167+0000","last_undegraded":"2024-03-04T09:25:20.498167+0000","last_fullsized":"2024-03-04T09:25:20.498167+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:39:25.865493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.151027+0000","last_change":"2024-03-04T09:25:18.481511+0000","last_active":"2024-03-04T09:25:21.151027+0000","last_peered":"2024-03-04T09:25:21.151027+0000","last_clean":"2024-03-04T09:25:21.151027+0000","last_became_active":"2024-03-04T09:25:18.481207+0000","last_became_peered":"2024-03-04T09:25:18.481207+0000","last_unstale":"2024-03-04T09:25:21.151027+0000","last_undegraded":"2024-03-04T09:25:21.151027+0000","last_fullsized":"2024-03-04T09:25:21.151027+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:11:51.226417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.498103+0000","last_change":"2024-03-04T09:25:19.453592+0000","last_active":"2024-03-04T09:25:20.498103+0000","last_peered":"2024-03-04T09:25:20.498103+0000","last_clean":"2024-03-04T09:25:20.498103+0000","last_became_active":"2024-03-04T09:25:19.453318+0000","last_became_peered":"2024-03-04T09:25:19.453318+0000","last_unstale":"2024-03-04T09:25:20.498103+0000","last_undegraded":"2024-03-04T09:25:20.498103+0000","last_fullsized":"2024-03-04T09:25:20.498103+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:04:20.349240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.151412+0000","last_change":"2024-03-04T09:25:18.483252+0000","last_active":"2024-03-04T09:25:21.151412+0000","last_peered":"2024-03-04T09:25:21.151412+0000","last_clean":"2024-03-04T09:25:21.151412+0000","last_became_active":"2024-03-04T09:25:18.482987+0000","last_became_peered":"2024-03-04T09:25:18.482987+0000","last_unstale":"2024-03-04T09:25:21.151412+0000","last_undegraded":"2024-03-04T09:25:21.151412+0000","last_fullsized":"2024-03-04T09:25:21.151412+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:02:44.764307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.150110+0000","last_change":"2024-03-04T09:25:19.473423+0000","last_active":"2024-03-04T09:25:21.150110+0000","last_peered":"2024-03-04T09:25:21.150110+0000","last_clean":"2024-03-04T09:25:21.150110+0000","last_became_active":"2024-03-04T09:25:19.473030+0000","last_became_peered":"2024-03-04T09:25:19.473030+0000","last_unstale":"2024-03-04T09:25:21.150110+0000","last_undegraded":"2024-03-04T09:25:21.150110+0000","last_fullsized":"2024-03-04T09:25:21.150110+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:56:18.833810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.458390+0000","last_change":"2024-03-04T09:25:18.466373+0000","last_active":"2024-03-04T09:25:20.458390+0000","last_peered":"2024-03-04T09:25:20.458390+0000","last_clean":"2024-03-04T09:25:20.458390+0000","last_became_active":"2024-03-04T09:25:18.466069+0000","last_became_peered":"2024-03-04T09:25:18.466069+0000","last_unstale":"2024-03-04T09:25:20.458390+0000","last_undegraded":"2024-03-04T09:25:20.458390+0000","last_fullsized":"2024-03-04T09:25:20.458390+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:11:51.226417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.458472+0000","last_change":"2024-03-04T09:25:19.459312+0000","last_active":"2024-03-04T09:25:20.458472+0000","last_peered":"2024-03-04T09:25:20.458472+0000","last_clean":"2024-03-04T09:25:20.458472+0000","last_became_active":"2024-03-04T09:25:19.458993+0000","last_became_peered":"2024-03-04T09:25:19.458993+0000","last_unstale":"2024-03-04T09:25:20.458472+0000","last_undegraded":"2024-03-04T09:25:20.458472+0000","last_fullsized":"2024-03-04T09:25:20.458472+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:36:58.654226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"19'4","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.150271+0000","last_change":"2024-03-04T09:25:18.473489+0000","last_active":"2024-03-04T09:25:21.150271+0000","last_peered":"2024-03-04T09:25:21.150271+0000","last_clean":"2024-03-04T09:25:21.150271+0000","last_became_active":"2024-03-04T09:25:18.473154+0000","last_became_peered":"2024-03-04T09:25:18.473154+0000","last_unstale":"2024-03-04T09:25:21.150271+0000","last_undegraded":"2024-03-04T09:25:21.150271+0000","last_fullsized":"2024-03-04T09:25:21.150271+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:31:15.318756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27804,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344036,"statfs":{"total":96636764160,"available":96608292864,"internally_reserved":0,"allocated":802816,"data_stored":661884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574854,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":72546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574856,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":208896,"data_stored":71024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574853,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":71060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574855,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":72546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574854,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":70934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574855,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":212992,"data_stored":71094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574854,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27816,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344024,"statfs":{"total":96636764160,"available":96608280576,"internally_reserved":0,"allocated":811008,"data_stored":661940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-04T09:25:29.607 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T09:25:29.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.871+0000 7fb021600640 1 Processor -- start 2024-03-04T09:25:29.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.871+0000 7fb021600640 1 -- start start 2024-03-04T09:25:29.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.873+0000 7fb021600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb01c145300 0x7fb01c0b49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:29.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.873+0000 7fb021600640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fb01c070f10 0x7fb01c0cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:29.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.873+0000 7fb021600640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb01c0714d0 con 0x7fb01c0cce90 2024-03-04T09:25:29.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.873+0000 7fb021600640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb01c0cd300 con 0x7fb01c070f10 2024-03-04T09:25:29.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.873+0000 7fb021600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb01c0717b0 con 0x7fb01c145300 2024-03-04T09:25:29.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.873+0000 7fb020c00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fb01c0cce90 0x7fb01c0cd930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:36802/0 (socket says 172.21.15.6:36802) 2024-03-04T09:25:29.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.873+0000 7fb020c00640 1 -- 172.21.15.6:0/1540845052 learned_addr learned my addr 172.21.15.6:0/1540845052 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:25:29.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.874+0000 7fb01be00640 1 --2- 172.21.15.6:0/1540845052 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb01c145300 0x7fb01c0b49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:29.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.874+0000 7fb013e00640 1 --2- 172.21.15.6:0/1540845052 >> v2:172.21.15.146:3300/0 conn(0x7fb01c070f10 0x7fb01c0cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:29.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.874+0000 7fb01b400640 1 -- 172.21.15.6:0/1540845052 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1728952961 0 0) 0x7fb01c0714d0 con 0x7fb01c0cce90 2024-03-04T09:25:29.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.874+0000 7fb01b400640 1 -- 172.21.15.6:0/1540845052 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb000003520 con 0x7fb01c0cce90 2024-03-04T09:25:29.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.874+0000 7fb01b400640 1 -- 172.21.15.6:0/1540845052 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2644011534 0 0) 0x7fb000003520 con 0x7fb01c0cce90 2024-03-04T09:25:29.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.874+0000 7fb01b400640 1 -- 172.21.15.6:0/1540845052 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb01c145300 msgr2=0x7fb01c0b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:29.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.874+0000 7fb01b400640 1 --2- 172.21.15.6:0/1540845052 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb01c145300 0x7fb01c0b49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:29.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.874+0000 7fb01b400640 1 -- 172.21.15.6:0/1540845052 >> v2:172.21.15.146:3300/0 conn(0x7fb01c070f10 msgr2=0x7fb01c0cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:29.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.874+0000 7fb01b400640 1 --2- 172.21.15.6:0/1540845052 >> v2:172.21.15.146:3300/0 conn(0x7fb01c070f10 0x7fb01c0cc950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:29.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.874+0000 7fb01b400640 1 -- 172.21.15.6:0/1540845052 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb01c071a90 con 0x7fb01c0cce90 2024-03-04T09:25:29.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.874+0000 7fb01b400640 1 -- 172.21.15.6:0/1540845052 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7fb0140019f0 con 0x7fb01c0cce90 2024-03-04T09:25:29.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.874+0000 7fb01b400640 1 -- 172.21.15.6:0/1540845052 >> v1:172.21.15.6:6789/0 conn(0x7fb01c0cce90 legacy=0x7fb01c0cd930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:29.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.874+0000 7fb01b400640 1 --2- 172.21.15.6:0/1540845052 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb0000039b0 0x7fb000003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:29.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.874+0000 7fb01b400640 1 --2- 172.21.15.6:0/1540845052 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb000004800 0x7fb000004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:29.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.874+0000 7fb01b400640 1 --2- 172.21.15.6:0/1540845052 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb01c145300 0x7fb0000051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:29.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.874+0000 7fb01b400640 1 -- 172.21.15.6:0/1540845052 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb01c0cd300 con 0x7fb0000039b0 2024-03-04T09:25:29.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.874+0000 7fb01b400640 1 -- 172.21.15.6:0/1540845052 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb01c0717b0 con 0x7fb000004800 2024-03-04T09:25:29.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.874+0000 7fb01b400640 1 -- 172.21.15.6:0/1540845052 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb00000ba10 con 0x7fb01c145300 2024-03-04T09:25:29.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.874+0000 7fb013e00640 1 --2- 172.21.15.6:0/1540845052 >> v2:172.21.15.146:3300/0 conn(0x7fb01c070f10 0x7fb01c0cc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T09:25:29.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.874+0000 7fb01be00640 1 --2- 172.21.15.6:0/1540845052 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb0000039b0 0x7fb000003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:29.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.875+0000 7fb01be00640 1 -- 172.21.15.6:0/1540845052 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb01c145300 msgr2=0x7fb0000051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:29.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.875+0000 7fb01be00640 1 --2- 172.21.15.6:0/1540845052 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb01c145300 0x7fb0000051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:29.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.875+0000 7fb01be00640 1 -- 172.21.15.6:0/1540845052 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb000004800 msgr2=0x7fb000004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:29.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.875+0000 7fb01be00640 1 --2- 172.21.15.6:0/1540845052 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb000004800 0x7fb000004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:29.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.875+0000 7fb01be00640 1 -- 172.21.15.6:0/1540845052 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb000003520 con 0x7fb0000039b0 2024-03-04T09:25:29.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.876+0000 7fb01be00640 1 --2- 172.21.15.6:0/1540845052 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb0000039b0 0x7fb000003dd0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fb00c009ac0 tx=0x7fb000005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eeadb7c2e343ae54 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:29.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.876+0000 7fb01b400640 1 -- 172.21.15.6:0/1540845052 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb00c039070 con 0x7fb0000039b0 2024-03-04T09:25:29.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.876+0000 7fb01b400640 1 -- 172.21.15.6:0/1540845052 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb00c02c710 con 0x7fb0000039b0 2024-03-04T09:25:29.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.876+0000 7fb01b400640 1 -- 172.21.15.6:0/1540845052 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb00c009690 con 0x7fb0000039b0 2024-03-04T09:25:29.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.877+0000 7fb021600640 1 -- 172.21.15.6:0/1540845052 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb0000039b0 msgr2=0x7fb000003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:29.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.877+0000 7fb021600640 1 --2- 172.21.15.6:0/1540845052 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb0000039b0 0x7fb000003dd0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fb00c009ac0 tx=0x7fb000005830 comp rx=0 tx=0).stop 2024-03-04T09:25:29.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.877+0000 7fb020c00640 1 -- 172.21.15.6:0/1540845052 reap_dead start 2024-03-04T09:25:29.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.877+0000 7fb021600640 1 -- 172.21.15.6:0/1540845052 shutdown_connections 2024-03-04T09:25:29.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.877+0000 7fb021600640 1 -- 172.21.15.6:0/1540845052 >> 172.21.15.6:0/1540845052 conn(0x7fb01c06b3d0 msgr2=0x7fb01c06b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:29.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.877+0000 7fb021600640 1 -- 172.21.15.6:0/1540845052 shutdown_connections 2024-03-04T09:25:29.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.877+0000 7fb021600640 1 -- 172.21.15.6:0/1540845052 wait complete. 2024-03-04T09:25:29.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.879+0000 7fb021600640 1 Processor -- start 2024-03-04T09:25:29.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.879+0000 7fb021600640 1 -- start start 2024-03-04T09:25:29.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.880+0000 7fb021600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb0000039b0 0x7fb01c15b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:29.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.880+0000 7fb021600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb01c1430f0 0x7fb01c143550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:29.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.880+0000 7fb021600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb01c0aadc0 0x7fb01c0ab240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:29.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.880+0000 7fb021600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb01c15d580 con 0x7fb01c1430f0 2024-03-04T09:25:29.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.880+0000 7fb021600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb01c1456e0 con 0x7fb01c0aadc0 2024-03-04T09:25:29.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.880+0000 7fb021600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb01c15d700 con 0x7fb0000039b0 2024-03-04T09:25:29.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.880+0000 7fb013e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb01c1430f0 0x7fb01c143550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:29.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.880+0000 7fb013e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb01c1430f0 0x7fb01c143550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45702/0 (socket says 172.21.15.6:45702) 2024-03-04T09:25:29.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.880+0000 7fb013e00640 1 -- 172.21.15.6:0/410289887 learned_addr learned my addr 172.21.15.6:0/410289887 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:25:29.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.880+0000 7fb01be00640 1 --2- 172.21.15.6:0/410289887 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb0000039b0 0x7fb01c15b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:29.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.880+0000 7fb020c00640 1 --2- 172.21.15.6:0/410289887 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb01c0aadc0 0x7fb01c0ab240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:29.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.880+0000 7fb013e00640 1 -- 172.21.15.6:0/410289887 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb0000039b0 msgr2=0x7fb01c15b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:29.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.880+0000 7fb013e00640 1 --2- 172.21.15.6:0/410289887 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb0000039b0 0x7fb01c15b430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:29.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.880+0000 7fb013e00640 1 -- 172.21.15.6:0/410289887 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb01c0aadc0 msgr2=0x7fb01c0ab240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:29.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.880+0000 7fb013e00640 1 --2- 172.21.15.6:0/410289887 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb01c0aadc0 0x7fb01c0ab240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:29.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.880+0000 7fb013e00640 1 -- 172.21.15.6:0/410289887 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb01c0aba80 con 0x7fb01c1430f0 2024-03-04T09:25:29.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.881+0000 7fb013e00640 1 --2- 172.21.15.6:0/410289887 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb01c1430f0 0x7fb01c143550 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fb00400e7e0 tx=0x7fb00400eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:29.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.881+0000 7fb018c00640 1 -- 172.21.15.6:0/410289887 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb00400f040 con 0x7fb01c1430f0 2024-03-04T09:25:29.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.881+0000 7fb018c00640 1 -- 172.21.15.6:0/410289887 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb004004230 con 0x7fb01c1430f0 2024-03-04T09:25:29.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.881+0000 7fb018c00640 1 -- 172.21.15.6:0/410289887 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb00400c6d0 con 0x7fb01c1430f0 2024-03-04T09:25:29.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.881+0000 7fb021600640 1 -- 172.21.15.6:0/410289887 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb01c0abd60 con 0x7fb01c1430f0 2024-03-04T09:25:29.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.881+0000 7fb021600640 1 -- 172.21.15.6:0/410289887 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb01c0ac320 con 0x7fb01c1430f0 2024-03-04T09:25:29.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.883+0000 7fb021600640 1 -- 172.21.15.6:0/410289887 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb01c04f330 con 0x7fb01c1430f0 2024-03-04T09:25:29.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.883+0000 7fb018c00640 1 -- 172.21.15.6:0/410289887 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb00400c870 con 0x7fb01c1430f0 2024-03-04T09:25:29.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.885+0000 7fb018c00640 1 --2- 172.21.15.6:0/410289887 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7faff806b520 0x7faff806d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:29.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.885+0000 7fb018c00640 1 -- 172.21.15.6:0/410289887 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb004089fe0 con 0x7fb01c1430f0 2024-03-04T09:25:29.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.885+0000 7fb01be00640 1 --2- 172.21.15.6:0/410289887 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7faff806b520 0x7faff806d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:29.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.886+0000 7fb01be00640 1 --2- 172.21.15.6:0/410289887 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7faff806b520 0x7faff806d9e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fb00c036000 tx=0x7fb00c030000 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:29.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:29.889+0000 7fb018c00640 1 -- 172.21.15.6:0/410289887 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb004017020 con 0x7fb01c1430f0 2024-03-04T09:25:30.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.080+0000 7fb021600640 1 -- 172.21.15.6:0/410289887 --> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb01c15ec70 con 0x7faff806b520 2024-03-04T09:25:30.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.109+0000 7fb018c00640 1 -- 172.21.15.6:0/410289887 <== mgr.4113 v2:172.21.15.146:6832/1788613334 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251837 (secure 0 0 0) 0x7fb00c0827d0 con 0x7faff806b520 2024-03-04T09:25:30.110 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:25:30.111 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-03-04T09:25:30.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.114+0000 7fb021600640 1 -- 172.21.15.6:0/410289887 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7faff806b520 msgr2=0x7faff806d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:30.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.115+0000 7fb021600640 1 --2- 172.21.15.6:0/410289887 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7faff806b520 0x7faff806d9e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fb00c036000 tx=0x7fb00c030000 comp rx=0 tx=0).stop 2024-03-04T09:25:30.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.115+0000 7fb021600640 1 -- 172.21.15.6:0/410289887 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb01c1430f0 msgr2=0x7fb01c143550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:30.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.115+0000 7fb021600640 1 --2- 172.21.15.6:0/410289887 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb01c1430f0 0x7fb01c143550 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fb00400e7e0 tx=0x7fb00400eca0 comp rx=0 tx=0).stop 2024-03-04T09:25:30.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.115+0000 7fb021600640 1 -- 172.21.15.6:0/410289887 shutdown_connections 2024-03-04T09:25:30.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.115+0000 7fb021600640 1 --2- 172.21.15.6:0/410289887 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb01c0aadc0 0x7fb01c0ab240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:30.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.115+0000 7fb021600640 1 --2- 172.21.15.6:0/410289887 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb01c1430f0 0x7fb01c143550 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:30.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.115+0000 7fb021600640 1 --2- 172.21.15.6:0/410289887 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7faff806b520 0x7faff806d9e0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:30.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.115+0000 7fb021600640 1 --2- 172.21.15.6:0/410289887 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb0000039b0 0x7fb01c15b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:30.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.115+0000 7fb021600640 1 -- 172.21.15.6:0/410289887 >> 172.21.15.6:0/410289887 conn(0x7fb01c06b3d0 msgr2=0x7fb01c15dbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:30.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.115+0000 7fb021600640 1 -- 172.21.15.6:0/410289887 shutdown_connections 2024-03-04T09:25:30.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.115+0000 7fb021600640 1 -- 172.21.15.6:0/410289887 wait complete. 2024-03-04T09:25:30.131 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-03-04T09:25:29.862596+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218988,"kb_used_data":2824,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755732,"statfs":{"total":773094113280,"available":772869869568,"internally_reserved":0,"allocated":2891776,"data_stored":1753028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.004378"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.152203+0000","last_change":"2024-03-04T09:25:19.480195+0000","last_active":"2024-03-04T09:25:21.152203+0000","last_peered":"2024-03-04T09:25:21.152203+0000","last_clean":"2024-03-04T09:25:21.152203+0000","last_became_active":"2024-03-04T09:25:19.479496+0000","last_became_peered":"2024-03-04T09:25:19.479496+0000","last_unstale":"2024-03-04T09:25:21.152203+0000","last_undegraded":"2024-03-04T09:25:21.152203+0000","last_fullsized":"2024-03-04T09:25:21.152203+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:36:55.714408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.457360+0000","last_change":"2024-03-04T09:25:19.457568+0000","last_active":"2024-03-04T09:25:20.457360+0000","last_peered":"2024-03-04T09:25:20.457360+0000","last_clean":"2024-03-04T09:25:20.457360+0000","last_became_active":"2024-03-04T09:25:19.457288+0000","last_became_peered":"2024-03-04T09:25:19.457288+0000","last_unstale":"2024-03-04T09:25:20.457360+0000","last_undegraded":"2024-03-04T09:25:20.457360+0000","last_fullsized":"2024-03-04T09:25:20.457360+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:33:16.368258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.149774+0000","last_change":"2024-03-04T09:25:19.465855+0000","last_active":"2024-03-04T09:25:21.149774+0000","last_peered":"2024-03-04T09:25:21.149774+0000","last_clean":"2024-03-04T09:25:21.149774+0000","last_became_active":"2024-03-04T09:25:19.465489+0000","last_became_peered":"2024-03-04T09:25:19.465489+0000","last_unstale":"2024-03-04T09:25:21.149774+0000","last_undegraded":"2024-03-04T09:25:21.149774+0000","last_fullsized":"2024-03-04T09:25:21.149774+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:05:57.624897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.497998+0000","last_change":"2024-03-04T09:25:19.472849+0000","last_active":"2024-03-04T09:25:20.497998+0000","last_peered":"2024-03-04T09:25:20.497998+0000","last_clean":"2024-03-04T09:25:20.497998+0000","last_became_active":"2024-03-04T09:25:19.468252+0000","last_became_peered":"2024-03-04T09:25:19.468252+0000","last_unstale":"2024-03-04T09:25:20.497998+0000","last_undegraded":"2024-03-04T09:25:20.497998+0000","last_fullsized":"2024-03-04T09:25:20.497998+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:20:25.135538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.151031+0000","last_change":"2024-03-04T09:25:19.471747+0000","last_active":"2024-03-04T09:25:21.151031+0000","last_peered":"2024-03-04T09:25:21.151031+0000","last_clean":"2024-03-04T09:25:21.151031+0000","last_became_active":"2024-03-04T09:25:19.469589+0000","last_became_peered":"2024-03-04T09:25:19.469589+0000","last_unstale":"2024-03-04T09:25:21.151031+0000","last_undegraded":"2024-03-04T09:25:21.151031+0000","last_fullsized":"2024-03-04T09:25:21.151031+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:36:58.654226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.497539+0000","last_change":"2024-03-04T09:25:19.458354+0000","last_active":"2024-03-04T09:25:20.497539+0000","last_peered":"2024-03-04T09:25:20.497539+0000","last_clean":"2024-03-04T09:25:20.497539+0000","last_became_active":"2024-03-04T09:25:19.456693+0000","last_became_peered":"2024-03-04T09:25:19.456693+0000","last_unstale":"2024-03-04T09:25:20.497539+0000","last_undegraded":"2024-03-04T09:25:20.497539+0000","last_fullsized":"2024-03-04T09:25:20.497539+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:11:11.501102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.150488+0000","last_change":"2024-03-04T09:25:19.476465+0000","last_active":"2024-03-04T09:25:21.150488+0000","last_peered":"2024-03-04T09:25:21.150488+0000","last_clean":"2024-03-04T09:25:21.150488+0000","last_became_active":"2024-03-04T09:25:19.471519+0000","last_became_peered":"2024-03-04T09:25:19.471519+0000","last_unstale":"2024-03-04T09:25:21.150488+0000","last_undegraded":"2024-03-04T09:25:21.150488+0000","last_fullsized":"2024-03-04T09:25:21.150488+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:01:33.498760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.455119+0000","last_change":"2024-03-04T09:25:19.471301+0000","last_active":"2024-03-04T09:25:20.455119+0000","last_peered":"2024-03-04T09:25:20.455119+0000","last_clean":"2024-03-04T09:25:20.455119+0000","last_became_active":"2024-03-04T09:25:19.470983+0000","last_became_peered":"2024-03-04T09:25:19.470983+0000","last_unstale":"2024-03-04T09:25:20.455119+0000","last_undegraded":"2024-03-04T09:25:20.455119+0000","last_fullsized":"2024-03-04T09:25:20.455119+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:50:18.265156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.152495+0000","last_change":"2024-03-04T09:25:19.485751+0000","last_active":"2024-03-04T09:25:21.152495+0000","last_peered":"2024-03-04T09:25:21.152495+0000","last_clean":"2024-03-04T09:25:21.152495+0000","last_became_active":"2024-03-04T09:25:19.485488+0000","last_became_peered":"2024-03-04T09:25:19.485488+0000","last_unstale":"2024-03-04T09:25:21.152495+0000","last_undegraded":"2024-03-04T09:25:21.152495+0000","last_fullsized":"2024-03-04T09:25:21.152495+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:45:23.780438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.464582+0000","last_change":"2024-03-04T09:25:19.466385+0000","last_active":"2024-03-04T09:25:20.464582+0000","last_peered":"2024-03-04T09:25:20.464582+0000","last_clean":"2024-03-04T09:25:20.464582+0000","last_became_active":"2024-03-04T09:25:19.466078+0000","last_became_peered":"2024-03-04T09:25:19.466078+0000","last_unstale":"2024-03-04T09:25:20.464582+0000","last_undegraded":"2024-03-04T09:25:20.464582+0000","last_fullsized":"2024-03-04T09:25:20.464582+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:56:18.833810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.464502+0000","last_change":"2024-03-04T09:25:19.458668+0000","last_active":"2024-03-04T09:25:20.464502+0000","last_peered":"2024-03-04T09:25:20.464502+0000","last_clean":"2024-03-04T09:25:20.464502+0000","last_became_active":"2024-03-04T09:25:19.458080+0000","last_became_peered":"2024-03-04T09:25:19.458080+0000","last_unstale":"2024-03-04T09:25:20.464502+0000","last_undegraded":"2024-03-04T09:25:20.464502+0000","last_fullsized":"2024-03-04T09:25:20.464502+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:44:37.465937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.152136+0000","last_change":"2024-03-04T09:25:19.470591+0000","last_active":"2024-03-04T09:25:21.152136+0000","last_peered":"2024-03-04T09:25:21.152136+0000","last_clean":"2024-03-04T09:25:21.152136+0000","last_became_active":"2024-03-04T09:25:19.463604+0000","last_became_peered":"2024-03-04T09:25:19.463604+0000","last_unstale":"2024-03-04T09:25:21.152136+0000","last_undegraded":"2024-03-04T09:25:21.152136+0000","last_fullsized":"2024-03-04T09:25:21.152136+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:02:45.778608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.456030+0000","last_change":"2024-03-04T09:25:19.462024+0000","last_active":"2024-03-04T09:25:20.456030+0000","last_peered":"2024-03-04T09:25:20.456030+0000","last_clean":"2024-03-04T09:25:20.456030+0000","last_became_active":"2024-03-04T09:25:19.461767+0000","last_became_peered":"2024-03-04T09:25:19.461767+0000","last_unstale":"2024-03-04T09:25:20.456030+0000","last_undegraded":"2024-03-04T09:25:20.456030+0000","last_fullsized":"2024-03-04T09:25:20.456030+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:47:26.445286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.149510+0000","last_change":"2024-03-04T09:25:19.468514+0000","last_active":"2024-03-04T09:25:21.149510+0000","last_peered":"2024-03-04T09:25:21.149510+0000","last_clean":"2024-03-04T09:25:21.149510+0000","last_became_active":"2024-03-04T09:25:19.468029+0000","last_became_peered":"2024-03-04T09:25:19.468029+0000","last_unstale":"2024-03-04T09:25:21.149510+0000","last_undegraded":"2024-03-04T09:25:21.149510+0000","last_fullsized":"2024-03-04T09:25:21.149510+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:45:15.160192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.455915+0000","last_change":"2024-03-04T09:25:19.448212+0000","last_active":"2024-03-04T09:25:20.455915+0000","last_peered":"2024-03-04T09:25:20.455915+0000","last_clean":"2024-03-04T09:25:20.455915+0000","last_became_active":"2024-03-04T09:25:19.447824+0000","last_became_peered":"2024-03-04T09:25:19.447824+0000","last_unstale":"2024-03-04T09:25:20.455915+0000","last_undegraded":"2024-03-04T09:25:20.455915+0000","last_fullsized":"2024-03-04T09:25:20.455915+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:44:37.465937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.497265+0000","last_change":"2024-03-04T09:25:19.471682+0000","last_active":"2024-03-04T09:25:20.497265+0000","last_peered":"2024-03-04T09:25:20.497265+0000","last_clean":"2024-03-04T09:25:20.497265+0000","last_became_active":"2024-03-04T09:25:19.471416+0000","last_became_peered":"2024-03-04T09:25:19.471416+0000","last_unstale":"2024-03-04T09:25:20.497265+0000","last_undegraded":"2024-03-04T09:25:20.497265+0000","last_fullsized":"2024-03-04T09:25:20.497265+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:01:33.498760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.499060+0000","last_change":"2024-03-04T09:25:19.484010+0000","last_active":"2024-03-04T09:25:20.499060+0000","last_peered":"2024-03-04T09:25:20.499060+0000","last_clean":"2024-03-04T09:25:20.499060+0000","last_became_active":"2024-03-04T09:25:19.477995+0000","last_became_peered":"2024-03-04T09:25:19.477995+0000","last_unstale":"2024-03-04T09:25:20.499060+0000","last_undegraded":"2024-03-04T09:25:20.499060+0000","last_fullsized":"2024-03-04T09:25:20.499060+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:36:31.167806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.458074+0000","last_change":"2024-03-04T09:25:19.456238+0000","last_active":"2024-03-04T09:25:20.458074+0000","last_peered":"2024-03-04T09:25:20.458074+0000","last_clean":"2024-03-04T09:25:20.458074+0000","last_became_active":"2024-03-04T09:25:19.455908+0000","last_became_peered":"2024-03-04T09:25:19.455908+0000","last_unstale":"2024-03-04T09:25:20.458074+0000","last_undegraded":"2024-03-04T09:25:20.458074+0000","last_fullsized":"2024-03-04T09:25:20.458074+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:32:59.918382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.151921+0000","last_change":"2024-03-04T09:25:19.479250+0000","last_active":"2024-03-04T09:25:21.151921+0000","last_peered":"2024-03-04T09:25:21.151921+0000","last_clean":"2024-03-04T09:25:21.151921+0000","last_became_active":"2024-03-04T09:25:19.478971+0000","last_became_peered":"2024-03-04T09:25:19.478971+0000","last_unstale":"2024-03-04T09:25:21.151921+0000","last_undegraded":"2024-03-04T09:25:21.151921+0000","last_fullsized":"2024-03-04T09:25:21.151921+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:12:45.452669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.464032+0000","last_change":"2024-03-04T09:25:19.471817+0000","last_active":"2024-03-04T09:25:20.464032+0000","last_peered":"2024-03-04T09:25:20.464032+0000","last_clean":"2024-03-04T09:25:20.464032+0000","last_became_active":"2024-03-04T09:25:19.471551+0000","last_became_peered":"2024-03-04T09:25:19.471551+0000","last_unstale":"2024-03-04T09:25:20.464032+0000","last_undegraded":"2024-03-04T09:25:20.464032+0000","last_fullsized":"2024-03-04T09:25:20.464032+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:45:59.096795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.149784+0000","last_change":"2024-03-04T09:25:19.464870+0000","last_active":"2024-03-04T09:25:21.149784+0000","last_peered":"2024-03-04T09:25:21.149784+0000","last_clean":"2024-03-04T09:25:21.149784+0000","last_became_active":"2024-03-04T09:25:19.464544+0000","last_became_peered":"2024-03-04T09:25:19.464544+0000","last_unstale":"2024-03-04T09:25:21.149784+0000","last_undegraded":"2024-03-04T09:25:21.149784+0000","last_fullsized":"2024-03-04T09:25:21.149784+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:45:59.096795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.497519+0000","last_change":"2024-03-04T09:25:19.459890+0000","last_active":"2024-03-04T09:25:20.497519+0000","last_peered":"2024-03-04T09:25:20.497519+0000","last_clean":"2024-03-04T09:25:20.497519+0000","last_became_active":"2024-03-04T09:25:19.459528+0000","last_became_peered":"2024-03-04T09:25:19.459528+0000","last_unstale":"2024-03-04T09:25:20.497519+0000","last_undegraded":"2024-03-04T09:25:20.497519+0000","last_fullsized":"2024-03-04T09:25:20.497519+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:36:58.654226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.456364+0000","last_change":"2024-03-04T09:25:19.462338+0000","last_active":"2024-03-04T09:25:20.456364+0000","last_peered":"2024-03-04T09:25:20.456364+0000","last_clean":"2024-03-04T09:25:20.456364+0000","last_became_active":"2024-03-04T09:25:19.461998+0000","last_became_peered":"2024-03-04T09:25:19.461998+0000","last_unstale":"2024-03-04T09:25:20.456364+0000","last_undegraded":"2024-03-04T09:25:20.456364+0000","last_fullsized":"2024-03-04T09:25:20.456364+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:56:18.833810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.497149+0000","last_change":"2024-03-04T09:25:19.473080+0000","last_active":"2024-03-04T09:25:20.497149+0000","last_peered":"2024-03-04T09:25:20.497149+0000","last_clean":"2024-03-04T09:25:20.497149+0000","last_became_active":"2024-03-04T09:25:19.467315+0000","last_became_peered":"2024-03-04T09:25:19.467315+0000","last_unstale":"2024-03-04T09:25:20.497149+0000","last_undegraded":"2024-03-04T09:25:20.497149+0000","last_fullsized":"2024-03-04T09:25:20.497149+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:33:16.368258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.458010+0000","last_change":"2024-03-04T09:25:19.472788+0000","last_active":"2024-03-04T09:25:20.458010+0000","last_peered":"2024-03-04T09:25:20.458010+0000","last_clean":"2024-03-04T09:25:20.458010+0000","last_became_active":"2024-03-04T09:25:19.472496+0000","last_became_peered":"2024-03-04T09:25:19.472496+0000","last_unstale":"2024-03-04T09:25:20.458010+0000","last_undegraded":"2024-03-04T09:25:20.458010+0000","last_fullsized":"2024-03-04T09:25:20.458010+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:01:33.498760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.464735+0000","last_change":"2024-03-04T09:25:19.461265+0000","last_active":"2024-03-04T09:25:20.464735+0000","last_peered":"2024-03-04T09:25:20.464735+0000","last_clean":"2024-03-04T09:25:20.464735+0000","last_became_active":"2024-03-04T09:25:19.458576+0000","last_became_peered":"2024-03-04T09:25:19.458576+0000","last_unstale":"2024-03-04T09:25:20.464735+0000","last_undegraded":"2024-03-04T09:25:20.464735+0000","last_fullsized":"2024-03-04T09:25:20.464735+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:33:58.076780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.456949+0000","last_change":"2024-03-04T09:25:19.467165+0000","last_active":"2024-03-04T09:25:20.456949+0000","last_peered":"2024-03-04T09:25:20.456949+0000","last_clean":"2024-03-04T09:25:20.456949+0000","last_became_active":"2024-03-04T09:25:19.466793+0000","last_became_peered":"2024-03-04T09:25:19.466793+0000","last_unstale":"2024-03-04T09:25:20.456949+0000","last_undegraded":"2024-03-04T09:25:20.456949+0000","last_fullsized":"2024-03-04T09:25:20.456949+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:33:58.076780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.457770+0000","last_change":"2024-03-04T09:25:19.474283+0000","last_active":"2024-03-04T09:25:20.457770+0000","last_peered":"2024-03-04T09:25:20.457770+0000","last_clean":"2024-03-04T09:25:20.457770+0000","last_became_active":"2024-03-04T09:25:19.474124+0000","last_became_peered":"2024-03-04T09:25:19.474124+0000","last_unstale":"2024-03-04T09:25:20.457770+0000","last_undegraded":"2024-03-04T09:25:20.457770+0000","last_fullsized":"2024-03-04T09:25:20.457770+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:33:16.368258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.150029+0000","last_change":"2024-03-04T09:25:19.461996+0000","last_active":"2024-03-04T09:25:21.150029+0000","last_peered":"2024-03-04T09:25:21.150029+0000","last_clean":"2024-03-04T09:25:21.150029+0000","last_became_active":"2024-03-04T09:25:19.461683+0000","last_became_peered":"2024-03-04T09:25:19.461683+0000","last_unstale":"2024-03-04T09:25:21.150029+0000","last_undegraded":"2024-03-04T09:25:21.150029+0000","last_fullsized":"2024-03-04T09:25:21.150029+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:47:26.445286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.497702+0000","last_change":"2024-03-04T09:25:19.466189+0000","last_active":"2024-03-04T09:25:20.497702+0000","last_peered":"2024-03-04T09:25:20.497702+0000","last_clean":"2024-03-04T09:25:20.497702+0000","last_became_active":"2024-03-04T09:25:19.465903+0000","last_became_peered":"2024-03-04T09:25:19.465903+0000","last_unstale":"2024-03-04T09:25:20.497702+0000","last_undegraded":"2024-03-04T09:25:20.497702+0000","last_fullsized":"2024-03-04T09:25:20.497702+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:04:48.675219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.457411+0000","last_change":"2024-03-04T09:25:19.458477+0000","last_active":"2024-03-04T09:25:20.457411+0000","last_peered":"2024-03-04T09:25:20.457411+0000","last_clean":"2024-03-04T09:25:20.457411+0000","last_became_active":"2024-03-04T09:25:19.455994+0000","last_became_peered":"2024-03-04T09:25:19.455994+0000","last_unstale":"2024-03-04T09:25:20.457411+0000","last_undegraded":"2024-03-04T09:25:20.457411+0000","last_fullsized":"2024-03-04T09:25:20.457411+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:11:52.240719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.463964+0000","last_change":"2024-03-04T09:25:19.464315+0000","last_active":"2024-03-04T09:25:20.463964+0000","last_peered":"2024-03-04T09:25:20.463964+0000","last_clean":"2024-03-04T09:25:20.463964+0000","last_became_active":"2024-03-04T09:25:19.463942+0000","last_became_peered":"2024-03-04T09:25:19.463942+0000","last_unstale":"2024-03-04T09:25:20.463964+0000","last_undegraded":"2024-03-04T09:25:20.463964+0000","last_fullsized":"2024-03-04T09:25:20.463964+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:05:57.624897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.149802+0000","last_change":"2024-03-04T09:25:19.448874+0000","last_active":"2024-03-04T09:25:21.149802+0000","last_peered":"2024-03-04T09:25:21.149802+0000","last_clean":"2024-03-04T09:25:21.149802+0000","last_became_active":"2024-03-04T09:25:19.445047+0000","last_became_peered":"2024-03-04T09:25:19.445047+0000","last_unstale":"2024-03-04T09:25:21.149802+0000","last_undegraded":"2024-03-04T09:25:21.149802+0000","last_fullsized":"2024-03-04T09:25:21.149802+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:23:35.947152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.456151+0000","last_change":"2024-03-04T09:25:18.469771+0000","last_active":"2024-03-04T09:25:20.456151+0000","last_peered":"2024-03-04T09:25:20.456151+0000","last_clean":"2024-03-04T09:25:20.456151+0000","last_became_active":"2024-03-04T09:25:18.469445+0000","last_became_peered":"2024-03-04T09:25:18.469445+0000","last_unstale":"2024-03-04T09:25:20.456151+0000","last_undegraded":"2024-03-04T09:25:20.456151+0000","last_fullsized":"2024-03-04T09:25:20.456151+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:39:20.774577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.150269+0000","last_change":"2024-03-04T09:25:19.463486+0000","last_active":"2024-03-04T09:25:21.150269+0000","last_peered":"2024-03-04T09:25:21.150269+0000","last_clean":"2024-03-04T09:25:21.150269+0000","last_became_active":"2024-03-04T09:25:19.463204+0000","last_became_peered":"2024-03-04T09:25:19.463204+0000","last_unstale":"2024-03-04T09:25:21.150269+0000","last_undegraded":"2024-03-04T09:25:21.150269+0000","last_fullsized":"2024-03-04T09:25:21.150269+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:33:58.076780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.463976+0000","last_change":"2024-03-04T09:25:18.472794+0000","last_active":"2024-03-04T09:25:20.463976+0000","last_peered":"2024-03-04T09:25:20.463976+0000","last_clean":"2024-03-04T09:25:20.463976+0000","last_became_active":"2024-03-04T09:25:18.472449+0000","last_became_peered":"2024-03-04T09:25:18.472449+0000","last_unstale":"2024-03-04T09:25:20.463976+0000","last_undegraded":"2024-03-04T09:25:20.463976+0000","last_fullsized":"2024-03-04T09:25:20.463976+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:23:34.932850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.456455+0000","last_change":"2024-03-04T09:25:18.474050+0000","last_active":"2024-03-04T09:25:20.456455+0000","last_peered":"2024-03-04T09:25:20.456455+0000","last_clean":"2024-03-04T09:25:20.456455+0000","last_became_active":"2024-03-04T09:25:18.473760+0000","last_became_peered":"2024-03-04T09:25:18.473760+0000","last_unstale":"2024-03-04T09:25:20.456455+0000","last_undegraded":"2024-03-04T09:25:20.456455+0000","last_fullsized":"2024-03-04T09:25:20.456455+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:31:15.318756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.455119+0000","last_change":"2024-03-04T09:25:19.464447+0000","last_active":"2024-03-04T09:25:20.455119+0000","last_peered":"2024-03-04T09:25:20.455119+0000","last_clean":"2024-03-04T09:25:20.455119+0000","last_became_active":"2024-03-04T09:25:19.463201+0000","last_became_peered":"2024-03-04T09:25:19.463201+0000","last_unstale":"2024-03-04T09:25:20.455119+0000","last_undegraded":"2024-03-04T09:25:20.455119+0000","last_fullsized":"2024-03-04T09:25:20.455119+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:05:57.624897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.497223+0000","last_change":"2024-03-04T09:25:18.476149+0000","last_active":"2024-03-04T09:25:20.497223+0000","last_peered":"2024-03-04T09:25:20.497223+0000","last_clean":"2024-03-04T09:25:20.497223+0000","last_became_active":"2024-03-04T09:25:18.475664+0000","last_became_peered":"2024-03-04T09:25:18.475664+0000","last_unstale":"2024-03-04T09:25:20.497223+0000","last_undegraded":"2024-03-04T09:25:20.497223+0000","last_fullsized":"2024-03-04T09:25:20.497223+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:33:35.151096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.455640+0000","last_change":"2024-03-04T09:25:19.467051+0000","last_active":"2024-03-04T09:25:20.455640+0000","last_peered":"2024-03-04T09:25:20.455640+0000","last_clean":"2024-03-04T09:25:20.455640+0000","last_became_active":"2024-03-04T09:25:19.466755+0000","last_became_peered":"2024-03-04T09:25:19.466755+0000","last_unstale":"2024-03-04T09:25:20.455640+0000","last_undegraded":"2024-03-04T09:25:20.455640+0000","last_fullsized":"2024-03-04T09:25:20.455640+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:47:26.445286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.464438+0000","last_change":"2024-03-04T09:25:18.479294+0000","last_active":"2024-03-04T09:25:20.464438+0000","last_peered":"2024-03-04T09:25:20.464438+0000","last_clean":"2024-03-04T09:25:20.464438+0000","last_became_active":"2024-03-04T09:25:18.478561+0000","last_became_peered":"2024-03-04T09:25:18.478561+0000","last_unstale":"2024-03-04T09:25:20.464438+0000","last_undegraded":"2024-03-04T09:25:20.464438+0000","last_fullsized":"2024-03-04T09:25:20.464438+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:33:35.151096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.498563+0000","last_change":"2024-03-04T09:25:19.459933+0000","last_active":"2024-03-04T09:25:20.498563+0000","last_peered":"2024-03-04T09:25:20.498563+0000","last_clean":"2024-03-04T09:25:20.498563+0000","last_became_active":"2024-03-04T09:25:19.459622+0000","last_became_peered":"2024-03-04T09:25:19.459622+0000","last_unstale":"2024-03-04T09:25:20.498563+0000","last_undegraded":"2024-03-04T09:25:20.498563+0000","last_fullsized":"2024-03-04T09:25:20.498563+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:45:23.780438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.498673+0000","last_change":"2024-03-04T09:25:18.476726+0000","last_active":"2024-03-04T09:25:20.498673+0000","last_peered":"2024-03-04T09:25:20.498673+0000","last_clean":"2024-03-04T09:25:20.498673+0000","last_became_active":"2024-03-04T09:25:18.475636+0000","last_became_peered":"2024-03-04T09:25:18.475636+0000","last_unstale":"2024-03-04T09:25:20.498673+0000","last_undegraded":"2024-03-04T09:25:20.498673+0000","last_fullsized":"2024-03-04T09:25:20.498673+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:54:01.985207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.150507+0000","last_change":"2024-03-04T09:25:19.473920+0000","last_active":"2024-03-04T09:25:21.150507+0000","last_peered":"2024-03-04T09:25:21.150507+0000","last_clean":"2024-03-04T09:25:21.150507+0000","last_became_active":"2024-03-04T09:25:19.473457+0000","last_became_peered":"2024-03-04T09:25:19.473457+0000","last_unstale":"2024-03-04T09:25:21.150507+0000","last_undegraded":"2024-03-04T09:25:21.150507+0000","last_fullsized":"2024-03-04T09:25:21.150507+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:33:16.368258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.457014+0000","last_change":"2024-03-04T09:25:18.471507+0000","last_active":"2024-03-04T09:25:20.457014+0000","last_peered":"2024-03-04T09:25:20.457014+0000","last_clean":"2024-03-04T09:25:20.457014+0000","last_became_active":"2024-03-04T09:25:18.470138+0000","last_became_peered":"2024-03-04T09:25:18.470138+0000","last_unstale":"2024-03-04T09:25:20.457014+0000","last_undegraded":"2024-03-04T09:25:20.457014+0000","last_fullsized":"2024-03-04T09:25:20.457014+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:54:01.985207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.151586+0000","last_change":"2024-03-04T09:25:19.471146+0000","last_active":"2024-03-04T09:25:21.151586+0000","last_peered":"2024-03-04T09:25:21.151586+0000","last_clean":"2024-03-04T09:25:21.151586+0000","last_became_active":"2024-03-04T09:25:19.470904+0000","last_became_peered":"2024-03-04T09:25:19.470904+0000","last_unstale":"2024-03-04T09:25:21.151586+0000","last_undegraded":"2024-03-04T09:25:21.151586+0000","last_fullsized":"2024-03-04T09:25:21.151586+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:32:59.918382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"19'6","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.463922+0000","last_change":"2024-03-04T09:25:18.474956+0000","last_active":"2024-03-04T09:25:20.463922+0000","last_peered":"2024-03-04T09:25:20.463922+0000","last_clean":"2024-03-04T09:25:20.463922+0000","last_became_active":"2024-03-04T09:25:18.474011+0000","last_became_peered":"2024-03-04T09:25:18.474011+0000","last_unstale":"2024-03-04T09:25:20.463922+0000","last_undegraded":"2024-03-04T09:25:20.463922+0000","last_fullsized":"2024-03-04T09:25:20.463922+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:54:01.985207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.151254+0000","last_change":"2024-03-04T09:25:19.477899+0000","last_active":"2024-03-04T09:25:21.151254+0000","last_peered":"2024-03-04T09:25:21.151254+0000","last_clean":"2024-03-04T09:25:21.151254+0000","last_became_active":"2024-03-04T09:25:19.477117+0000","last_became_peered":"2024-03-04T09:25:19.477117+0000","last_unstale":"2024-03-04T09:25:21.151254+0000","last_undegraded":"2024-03-04T09:25:21.151254+0000","last_fullsized":"2024-03-04T09:25:21.151254+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:04:48.675219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.150191+0000","last_change":"2024-03-04T09:25:18.469651+0000","last_active":"2024-03-04T09:25:21.150191+0000","last_peered":"2024-03-04T09:25:21.150191+0000","last_clean":"2024-03-04T09:25:21.150191+0000","last_became_active":"2024-03-04T09:25:18.469259+0000","last_became_peered":"2024-03-04T09:25:18.469259+0000","last_unstale":"2024-03-04T09:25:21.150191+0000","last_undegraded":"2024-03-04T09:25:21.150191+0000","last_fullsized":"2024-03-04T09:25:21.150191+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:57:52.768999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.149691+0000","last_change":"2024-03-04T09:25:19.465339+0000","last_active":"2024-03-04T09:25:21.149691+0000","last_peered":"2024-03-04T09:25:21.149691+0000","last_clean":"2024-03-04T09:25:21.149691+0000","last_became_active":"2024-03-04T09:25:19.464957+0000","last_became_peered":"2024-03-04T09:25:19.464957+0000","last_unstale":"2024-03-04T09:25:21.149691+0000","last_undegraded":"2024-03-04T09:25:21.149691+0000","last_fullsized":"2024-03-04T09:25:21.149691+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:50:18.265156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.497632+0000","last_change":"2024-03-04T09:25:18.470800+0000","last_active":"2024-03-04T09:25:20.497632+0000","last_peered":"2024-03-04T09:25:20.497632+0000","last_clean":"2024-03-04T09:25:20.497632+0000","last_became_active":"2024-03-04T09:25:18.470431+0000","last_became_peered":"2024-03-04T09:25:18.470431+0000","last_unstale":"2024-03-04T09:25:20.497632+0000","last_undegraded":"2024-03-04T09:25:20.497632+0000","last_fullsized":"2024-03-04T09:25:20.497632+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:39:20.774577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.151318+0000","last_change":"2024-03-04T09:25:19.478795+0000","last_active":"2024-03-04T09:25:21.151318+0000","last_peered":"2024-03-04T09:25:21.151318+0000","last_clean":"2024-03-04T09:25:21.151318+0000","last_became_active":"2024-03-04T09:25:19.478635+0000","last_became_peered":"2024-03-04T09:25:19.478635+0000","last_unstale":"2024-03-04T09:25:21.151318+0000","last_undegraded":"2024-03-04T09:25:21.151318+0000","last_fullsized":"2024-03-04T09:25:21.151318+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:04:20.349240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.463677+0000","last_change":"2024-03-04T09:25:18.474750+0000","last_active":"2024-03-04T09:25:20.463677+0000","last_peered":"2024-03-04T09:25:20.463677+0000","last_clean":"2024-03-04T09:25:20.463677+0000","last_became_active":"2024-03-04T09:25:18.473463+0000","last_became_peered":"2024-03-04T09:25:18.473463+0000","last_unstale":"2024-03-04T09:25:20.463677+0000","last_undegraded":"2024-03-04T09:25:20.463677+0000","last_fullsized":"2024-03-04T09:25:20.463677+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:38:16.906069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.463789+0000","last_change":"2024-03-04T09:25:19.472274+0000","last_active":"2024-03-04T09:25:20.463789+0000","last_peered":"2024-03-04T09:25:20.463789+0000","last_clean":"2024-03-04T09:25:20.463789+0000","last_became_active":"2024-03-04T09:25:19.470669+0000","last_became_peered":"2024-03-04T09:25:19.470669+0000","last_unstale":"2024-03-04T09:25:20.463789+0000","last_undegraded":"2024-03-04T09:25:20.463789+0000","last_fullsized":"2024-03-04T09:25:20.463789+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:50:18.265156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":22,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.456829+0000","last_change":"2024-03-04T09:25:18.462460+0000","last_active":"2024-03-04T09:25:20.456829+0000","last_peered":"2024-03-04T09:25:20.456829+0000","last_clean":"2024-03-04T09:25:20.456829+0000","last_became_active":"2024-03-04T09:25:18.460692+0000","last_became_peered":"2024-03-04T09:25:18.460692+0000","last_unstale":"2024-03-04T09:25:20.456829+0000","last_undegraded":"2024-03-04T09:25:20.456829+0000","last_fullsized":"2024-03-04T09:25:20.456829+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:38:16.906069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.150513+0000","last_change":"2024-03-04T09:25:19.478013+0000","last_active":"2024-03-04T09:25:21.150513+0000","last_peered":"2024-03-04T09:25:21.150513+0000","last_clean":"2024-03-04T09:25:21.150513+0000","last_became_active":"2024-03-04T09:25:19.477150+0000","last_became_peered":"2024-03-04T09:25:19.477150+0000","last_unstale":"2024-03-04T09:25:21.150513+0000","last_undegraded":"2024-03-04T09:25:21.150513+0000","last_fullsized":"2024-03-04T09:25:21.150513+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:11:11.501102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.152408+0000","last_change":"2024-03-04T09:25:18.468798+0000","last_active":"2024-03-04T09:25:21.152408+0000","last_peered":"2024-03-04T09:25:21.152408+0000","last_clean":"2024-03-04T09:25:21.152408+0000","last_became_active":"2024-03-04T09:25:18.468412+0000","last_became_peered":"2024-03-04T09:25:18.468412+0000","last_unstale":"2024-03-04T09:25:21.152408+0000","last_undegraded":"2024-03-04T09:25:21.152408+0000","last_fullsized":"2024-03-04T09:25:21.152408+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:17:37.720962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.151303+0000","last_change":"2024-03-04T09:25:19.478586+0000","last_active":"2024-03-04T09:25:21.151303+0000","last_peered":"2024-03-04T09:25:21.151303+0000","last_clean":"2024-03-04T09:25:21.151303+0000","last_became_active":"2024-03-04T09:25:19.478339+0000","last_became_peered":"2024-03-04T09:25:19.478339+0000","last_unstale":"2024-03-04T09:25:21.151303+0000","last_undegraded":"2024-03-04T09:25:21.151303+0000","last_fullsized":"2024-03-04T09:25:21.151303+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:20:25.135538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.458139+0000","last_change":"2024-03-04T09:25:18.459079+0000","last_active":"2024-03-04T09:25:20.458139+0000","last_peered":"2024-03-04T09:25:20.458139+0000","last_clean":"2024-03-04T09:25:20.458139+0000","last_became_active":"2024-03-04T09:25:18.458767+0000","last_became_peered":"2024-03-04T09:25:18.458767+0000","last_unstale":"2024-03-04T09:25:20.458139+0000","last_undegraded":"2024-03-04T09:25:20.458139+0000","last_fullsized":"2024-03-04T09:25:20.458139+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:33:35.151096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.152708+0000","last_change":"2024-03-04T09:25:19.481294+0000","last_active":"2024-03-04T09:25:21.152708+0000","last_peered":"2024-03-04T09:25:21.152708+0000","last_clean":"2024-03-04T09:25:21.152708+0000","last_became_active":"2024-03-04T09:25:19.479163+0000","last_became_peered":"2024-03-04T09:25:19.479163+0000","last_unstale":"2024-03-04T09:25:21.152708+0000","last_undegraded":"2024-03-04T09:25:21.152708+0000","last_fullsized":"2024-03-04T09:25:21.152708+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:36:31.167806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.457070+0000","last_change":"2024-03-04T09:25:18.464660+0000","last_active":"2024-03-04T09:25:20.457070+0000","last_peered":"2024-03-04T09:25:20.457070+0000","last_clean":"2024-03-04T09:25:20.457070+0000","last_became_active":"2024-03-04T09:25:18.464412+0000","last_became_peered":"2024-03-04T09:25:18.464412+0000","last_unstale":"2024-03-04T09:25:20.457070+0000","last_undegraded":"2024-03-04T09:25:20.457070+0000","last_fullsized":"2024-03-04T09:25:20.457070+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:32:58.904081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.498417+0000","last_change":"2024-03-04T09:25:19.480725+0000","last_active":"2024-03-04T09:25:20.498417+0000","last_peered":"2024-03-04T09:25:20.498417+0000","last_clean":"2024-03-04T09:25:20.498417+0000","last_became_active":"2024-03-04T09:25:19.468626+0000","last_became_peered":"2024-03-04T09:25:19.468626+0000","last_unstale":"2024-03-04T09:25:20.498417+0000","last_undegraded":"2024-03-04T09:25:20.498417+0000","last_fullsized":"2024-03-04T09:25:20.498417+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:36:55.714408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.457915+0000","last_change":"2024-03-04T09:25:18.479022+0000","last_active":"2024-03-04T09:25:20.457915+0000","last_peered":"2024-03-04T09:25:20.457915+0000","last_clean":"2024-03-04T09:25:20.457915+0000","last_became_active":"2024-03-04T09:25:18.478697+0000","last_became_peered":"2024-03-04T09:25:18.478697+0000","last_unstale":"2024-03-04T09:25:20.457915+0000","last_undegraded":"2024-03-04T09:25:20.457915+0000","last_fullsized":"2024-03-04T09:25:20.457915+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:39:20.774577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.151536+0000","last_change":"2024-03-04T09:25:19.485083+0000","last_active":"2024-03-04T09:25:21.151536+0000","last_peered":"2024-03-04T09:25:21.151536+0000","last_clean":"2024-03-04T09:25:21.151536+0000","last_became_active":"2024-03-04T09:25:19.484619+0000","last_became_peered":"2024-03-04T09:25:19.484619+0000","last_unstale":"2024-03-04T09:25:21.151536+0000","last_undegraded":"2024-03-04T09:25:21.151536+0000","last_fullsized":"2024-03-04T09:25:21.151536+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:39:25.865493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":170,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.150247+0000","last_change":"2024-03-04T09:25:09.316159+0000","last_active":"2024-03-04T09:25:21.150247+0000","last_peered":"2024-03-04T09:25:21.150247+0000","last_clean":"2024-03-04T09:25:21.150247+0000","last_became_active":"2024-03-04T09:25:09.313767+0000","last_became_peered":"2024-03-04T09:25:09.313767+0000","last_unstale":"2024-03-04T09:25:21.150247+0000","last_undegraded":"2024-03-04T09:25:21.150247+0000","last_fullsized":"2024-03-04T09:25:21.150247+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:08.110953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:08.110953+0000","last_clean_scrub_stamp":"2024-03-04T09:25:08.110953+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:34:21.745390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.497328+0000","last_change":"2024-03-04T09:25:18.464859+0000","last_active":"2024-03-04T09:25:20.497328+0000","last_peered":"2024-03-04T09:25:20.497328+0000","last_clean":"2024-03-04T09:25:20.497328+0000","last_became_active":"2024-03-04T09:25:18.463515+0000","last_became_peered":"2024-03-04T09:25:18.463515+0000","last_unstale":"2024-03-04T09:25:20.497328+0000","last_undegraded":"2024-03-04T09:25:20.497328+0000","last_fullsized":"2024-03-04T09:25:20.497328+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:38:16.906069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.456773+0000","last_change":"2024-03-04T09:25:19.460896+0000","last_active":"2024-03-04T09:25:20.456773+0000","last_peered":"2024-03-04T09:25:20.456773+0000","last_clean":"2024-03-04T09:25:20.456773+0000","last_became_active":"2024-03-04T09:25:19.460598+0000","last_became_peered":"2024-03-04T09:25:19.460598+0000","last_unstale":"2024-03-04T09:25:20.456773+0000","last_undegraded":"2024-03-04T09:25:20.456773+0000","last_fullsized":"2024-03-04T09:25:20.456773+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:33:58.076780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.149994+0000","last_change":"2024-03-04T09:25:18.480278+0000","last_active":"2024-03-04T09:25:21.149994+0000","last_peered":"2024-03-04T09:25:21.149994+0000","last_clean":"2024-03-04T09:25:21.149994+0000","last_became_active":"2024-03-04T09:25:18.478848+0000","last_became_peered":"2024-03-04T09:25:18.478848+0000","last_unstale":"2024-03-04T09:25:21.149994+0000","last_undegraded":"2024-03-04T09:25:21.149994+0000","last_fullsized":"2024-03-04T09:25:21.149994+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:54:01.985207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.149481+0000","last_change":"2024-03-04T09:25:19.473908+0000","last_active":"2024-03-04T09:25:21.149481+0000","last_peered":"2024-03-04T09:25:21.149481+0000","last_clean":"2024-03-04T09:25:21.149481+0000","last_became_active":"2024-03-04T09:25:19.472415+0000","last_became_peered":"2024-03-04T09:25:19.472415+0000","last_unstale":"2024-03-04T09:25:21.149481+0000","last_undegraded":"2024-03-04T09:25:21.149481+0000","last_fullsized":"2024-03-04T09:25:21.149481+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:46:44.342046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.457503+0000","last_change":"2024-03-04T09:25:18.462877+0000","last_active":"2024-03-04T09:25:20.457503+0000","last_peered":"2024-03-04T09:25:20.457503+0000","last_clean":"2024-03-04T09:25:20.457503+0000","last_became_active":"2024-03-04T09:25:18.459553+0000","last_became_peered":"2024-03-04T09:25:18.459553+0000","last_unstale":"2024-03-04T09:25:20.457503+0000","last_undegraded":"2024-03-04T09:25:20.457503+0000","last_fullsized":"2024-03-04T09:25:20.457503+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:49:03.118927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.456676+0000","last_change":"2024-03-04T09:25:19.452298+0000","last_active":"2024-03-04T09:25:20.456676+0000","last_peered":"2024-03-04T09:25:20.456676+0000","last_clean":"2024-03-04T09:25:20.456676+0000","last_became_active":"2024-03-04T09:25:19.451945+0000","last_became_peered":"2024-03-04T09:25:19.451945+0000","last_unstale":"2024-03-04T09:25:20.456676+0000","last_undegraded":"2024-03-04T09:25:20.456676+0000","last_fullsized":"2024-03-04T09:25:20.456676+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:56:18.833810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.497812+0000","last_change":"2024-03-04T09:25:18.473872+0000","last_active":"2024-03-04T09:25:20.497812+0000","last_peered":"2024-03-04T09:25:20.497812+0000","last_clean":"2024-03-04T09:25:20.497812+0000","last_became_active":"2024-03-04T09:25:18.473390+0000","last_became_peered":"2024-03-04T09:25:18.473390+0000","last_unstale":"2024-03-04T09:25:20.497812+0000","last_undegraded":"2024-03-04T09:25:20.497812+0000","last_fullsized":"2024-03-04T09:25:20.497812+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:23:34.932850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.150539+0000","last_change":"2024-03-04T09:25:19.467593+0000","last_active":"2024-03-04T09:25:21.150539+0000","last_peered":"2024-03-04T09:25:21.150539+0000","last_clean":"2024-03-04T09:25:21.150539+0000","last_became_active":"2024-03-04T09:25:19.467079+0000","last_became_peered":"2024-03-04T09:25:19.467079+0000","last_unstale":"2024-03-04T09:25:21.150539+0000","last_undegraded":"2024-03-04T09:25:21.150539+0000","last_fullsized":"2024-03-04T09:25:21.150539+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:11:52.240719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.456743+0000","last_change":"2024-03-04T09:25:18.465721+0000","last_active":"2024-03-04T09:25:20.456743+0000","last_peered":"2024-03-04T09:25:20.456743+0000","last_clean":"2024-03-04T09:25:20.456743+0000","last_became_active":"2024-03-04T09:25:18.465438+0000","last_became_peered":"2024-03-04T09:25:18.465438+0000","last_unstale":"2024-03-04T09:25:20.456743+0000","last_undegraded":"2024-03-04T09:25:20.456743+0000","last_fullsized":"2024-03-04T09:25:20.456743+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:39:20.774577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.498362+0000","last_change":"2024-03-04T09:25:19.472271+0000","last_active":"2024-03-04T09:25:20.498362+0000","last_peered":"2024-03-04T09:25:20.498362+0000","last_clean":"2024-03-04T09:25:20.498362+0000","last_became_active":"2024-03-04T09:25:19.467596+0000","last_became_peered":"2024-03-04T09:25:19.467596+0000","last_unstale":"2024-03-04T09:25:20.498362+0000","last_undegraded":"2024-03-04T09:25:20.498362+0000","last_fullsized":"2024-03-04T09:25:20.498362+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:12:45.452669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.151714+0000","last_change":"2024-03-04T09:25:18.481607+0000","last_active":"2024-03-04T09:25:21.151714+0000","last_peered":"2024-03-04T09:25:21.151714+0000","last_clean":"2024-03-04T09:25:21.151714+0000","last_became_active":"2024-03-04T09:25:18.481270+0000","last_became_peered":"2024-03-04T09:25:18.481270+0000","last_unstale":"2024-03-04T09:25:21.151714+0000","last_undegraded":"2024-03-04T09:25:21.151714+0000","last_fullsized":"2024-03-04T09:25:21.151714+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:52:36.632272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.455410+0000","last_change":"2024-03-04T09:25:19.450231+0000","last_active":"2024-03-04T09:25:20.455410+0000","last_peered":"2024-03-04T09:25:20.455410+0000","last_clean":"2024-03-04T09:25:20.455410+0000","last_became_active":"2024-03-04T09:25:19.449920+0000","last_became_peered":"2024-03-04T09:25:19.449920+0000","last_unstale":"2024-03-04T09:25:20.455410+0000","last_undegraded":"2024-03-04T09:25:20.455410+0000","last_fullsized":"2024-03-04T09:25:20.455410+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:45:59.096795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.457374+0000","last_change":"2024-03-04T09:25:18.472718+0000","last_active":"2024-03-04T09:25:20.457374+0000","last_peered":"2024-03-04T09:25:20.457374+0000","last_clean":"2024-03-04T09:25:20.457374+0000","last_became_active":"2024-03-04T09:25:18.472426+0000","last_became_peered":"2024-03-04T09:25:18.472426+0000","last_unstale":"2024-03-04T09:25:20.457374+0000","last_undegraded":"2024-03-04T09:25:20.457374+0000","last_fullsized":"2024-03-04T09:25:20.457374+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:23:34.932850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.458202+0000","last_change":"2024-03-04T09:25:19.470531+0000","last_active":"2024-03-04T09:25:20.458202+0000","last_peered":"2024-03-04T09:25:20.458202+0000","last_clean":"2024-03-04T09:25:20.458202+0000","last_became_active":"2024-03-04T09:25:19.469607+0000","last_became_peered":"2024-03-04T09:25:19.469607+0000","last_unstale":"2024-03-04T09:25:20.458202+0000","last_undegraded":"2024-03-04T09:25:20.458202+0000","last_fullsized":"2024-03-04T09:25:20.458202+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:11:11.501102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.464561+0000","last_change":"2024-03-04T09:25:18.473250+0000","last_active":"2024-03-04T09:25:20.464561+0000","last_peered":"2024-03-04T09:25:20.464561+0000","last_clean":"2024-03-04T09:25:20.464561+0000","last_became_active":"2024-03-04T09:25:18.472968+0000","last_became_peered":"2024-03-04T09:25:18.472968+0000","last_unstale":"2024-03-04T09:25:20.464561+0000","last_undegraded":"2024-03-04T09:25:20.464561+0000","last_fullsized":"2024-03-04T09:25:20.464561+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:39:20.774577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.150075+0000","last_change":"2024-03-04T09:25:19.467410+0000","last_active":"2024-03-04T09:25:21.150075+0000","last_peered":"2024-03-04T09:25:21.150075+0000","last_clean":"2024-03-04T09:25:21.150075+0000","last_became_active":"2024-03-04T09:25:19.467101+0000","last_became_peered":"2024-03-04T09:25:19.467101+0000","last_unstale":"2024-03-04T09:25:21.150075+0000","last_undegraded":"2024-03-04T09:25:21.150075+0000","last_fullsized":"2024-03-04T09:25:21.150075+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:44:37.465937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.149988+0000","last_change":"2024-03-04T09:25:18.475507+0000","last_active":"2024-03-04T09:25:21.149988+0000","last_peered":"2024-03-04T09:25:21.149988+0000","last_clean":"2024-03-04T09:25:21.149988+0000","last_became_active":"2024-03-04T09:25:18.475159+0000","last_became_peered":"2024-03-04T09:25:18.475159+0000","last_unstale":"2024-03-04T09:25:21.149988+0000","last_undegraded":"2024-03-04T09:25:21.149988+0000","last_fullsized":"2024-03-04T09:25:21.149988+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:23:34.932850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.149845+0000","last_change":"2024-03-04T09:25:19.453119+0000","last_active":"2024-03-04T09:25:21.149845+0000","last_peered":"2024-03-04T09:25:21.149845+0000","last_clean":"2024-03-04T09:25:21.149845+0000","last_became_active":"2024-03-04T09:25:19.452764+0000","last_became_peered":"2024-03-04T09:25:19.452764+0000","last_unstale":"2024-03-04T09:25:21.149845+0000","last_undegraded":"2024-03-04T09:25:21.149845+0000","last_fullsized":"2024-03-04T09:25:21.149845+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:54:02.999509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.456033+0000","last_change":"2024-03-04T09:25:18.473872+0000","last_active":"2024-03-04T09:25:20.456033+0000","last_peered":"2024-03-04T09:25:20.456033+0000","last_clean":"2024-03-04T09:25:20.456033+0000","last_became_active":"2024-03-04T09:25:18.473554+0000","last_became_peered":"2024-03-04T09:25:18.473554+0000","last_unstale":"2024-03-04T09:25:20.456033+0000","last_undegraded":"2024-03-04T09:25:20.456033+0000","last_fullsized":"2024-03-04T09:25:20.456033+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:11:51.226417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.455488+0000","last_change":"2024-03-04T09:25:19.458923+0000","last_active":"2024-03-04T09:25:20.455488+0000","last_peered":"2024-03-04T09:25:20.455488+0000","last_clean":"2024-03-04T09:25:20.455488+0000","last_became_active":"2024-03-04T09:25:19.458562+0000","last_became_peered":"2024-03-04T09:25:19.458562+0000","last_unstale":"2024-03-04T09:25:20.455488+0000","last_undegraded":"2024-03-04T09:25:20.455488+0000","last_fullsized":"2024-03-04T09:25:20.455488+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:45:59.096795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.456036+0000","last_change":"2024-03-04T09:25:18.458201+0000","last_active":"2024-03-04T09:25:20.456036+0000","last_peered":"2024-03-04T09:25:20.456036+0000","last_clean":"2024-03-04T09:25:20.456036+0000","last_became_active":"2024-03-04T09:25:18.455667+0000","last_became_peered":"2024-03-04T09:25:18.455667+0000","last_unstale":"2024-03-04T09:25:20.456036+0000","last_undegraded":"2024-03-04T09:25:20.456036+0000","last_fullsized":"2024-03-04T09:25:20.456036+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:38:16.906069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.456043+0000","last_change":"2024-03-04T09:25:19.464390+0000","last_active":"2024-03-04T09:25:20.456043+0000","last_peered":"2024-03-04T09:25:20.456043+0000","last_clean":"2024-03-04T09:25:20.456043+0000","last_became_active":"2024-03-04T09:25:19.463511+0000","last_became_peered":"2024-03-04T09:25:19.463511+0000","last_unstale":"2024-03-04T09:25:20.456043+0000","last_undegraded":"2024-03-04T09:25:20.456043+0000","last_fullsized":"2024-03-04T09:25:20.456043+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:44:37.465937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.151481+0000","last_change":"2024-03-04T09:25:18.479732+0000","last_active":"2024-03-04T09:25:21.151481+0000","last_peered":"2024-03-04T09:25:21.151481+0000","last_clean":"2024-03-04T09:25:21.151481+0000","last_became_active":"2024-03-04T09:25:18.478014+0000","last_became_peered":"2024-03-04T09:25:18.478014+0000","last_unstale":"2024-03-04T09:25:21.151481+0000","last_undegraded":"2024-03-04T09:25:21.151481+0000","last_fullsized":"2024-03-04T09:25:21.151481+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:32:58.904081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.464302+0000","last_change":"2024-03-04T09:25:19.455001+0000","last_active":"2024-03-04T09:25:20.464302+0000","last_peered":"2024-03-04T09:25:20.464302+0000","last_clean":"2024-03-04T09:25:20.464302+0000","last_became_active":"2024-03-04T09:25:19.454695+0000","last_became_peered":"2024-03-04T09:25:19.454695+0000","last_unstale":"2024-03-04T09:25:20.464302+0000","last_undegraded":"2024-03-04T09:25:20.464302+0000","last_fullsized":"2024-03-04T09:25:20.464302+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:47:26.445286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.498167+0000","last_change":"2024-03-04T09:25:19.452288+0000","last_active":"2024-03-04T09:25:20.498167+0000","last_peered":"2024-03-04T09:25:20.498167+0000","last_clean":"2024-03-04T09:25:20.498167+0000","last_became_active":"2024-03-04T09:25:19.451982+0000","last_became_peered":"2024-03-04T09:25:19.451982+0000","last_unstale":"2024-03-04T09:25:20.498167+0000","last_undegraded":"2024-03-04T09:25:20.498167+0000","last_fullsized":"2024-03-04T09:25:20.498167+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:39:25.865493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.151027+0000","last_change":"2024-03-04T09:25:18.481511+0000","last_active":"2024-03-04T09:25:21.151027+0000","last_peered":"2024-03-04T09:25:21.151027+0000","last_clean":"2024-03-04T09:25:21.151027+0000","last_became_active":"2024-03-04T09:25:18.481207+0000","last_became_peered":"2024-03-04T09:25:18.481207+0000","last_unstale":"2024-03-04T09:25:21.151027+0000","last_undegraded":"2024-03-04T09:25:21.151027+0000","last_fullsized":"2024-03-04T09:25:21.151027+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:11:51.226417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.498103+0000","last_change":"2024-03-04T09:25:19.453592+0000","last_active":"2024-03-04T09:25:20.498103+0000","last_peered":"2024-03-04T09:25:20.498103+0000","last_clean":"2024-03-04T09:25:20.498103+0000","last_became_active":"2024-03-04T09:25:19.453318+0000","last_became_peered":"2024-03-04T09:25:19.453318+0000","last_unstale":"2024-03-04T09:25:20.498103+0000","last_undegraded":"2024-03-04T09:25:20.498103+0000","last_fullsized":"2024-03-04T09:25:20.498103+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:04:20.349240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.151412+0000","last_change":"2024-03-04T09:25:18.483252+0000","last_active":"2024-03-04T09:25:21.151412+0000","last_peered":"2024-03-04T09:25:21.151412+0000","last_clean":"2024-03-04T09:25:21.151412+0000","last_became_active":"2024-03-04T09:25:18.482987+0000","last_became_peered":"2024-03-04T09:25:18.482987+0000","last_unstale":"2024-03-04T09:25:21.151412+0000","last_undegraded":"2024-03-04T09:25:21.151412+0000","last_fullsized":"2024-03-04T09:25:21.151412+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:02:44.764307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.150110+0000","last_change":"2024-03-04T09:25:19.473423+0000","last_active":"2024-03-04T09:25:21.150110+0000","last_peered":"2024-03-04T09:25:21.150110+0000","last_clean":"2024-03-04T09:25:21.150110+0000","last_became_active":"2024-03-04T09:25:19.473030+0000","last_became_peered":"2024-03-04T09:25:19.473030+0000","last_unstale":"2024-03-04T09:25:21.150110+0000","last_undegraded":"2024-03-04T09:25:21.150110+0000","last_fullsized":"2024-03-04T09:25:21.150110+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:56:18.833810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.458390+0000","last_change":"2024-03-04T09:25:18.466373+0000","last_active":"2024-03-04T09:25:20.458390+0000","last_peered":"2024-03-04T09:25:20.458390+0000","last_clean":"2024-03-04T09:25:20.458390+0000","last_became_active":"2024-03-04T09:25:18.466069+0000","last_became_peered":"2024-03-04T09:25:18.466069+0000","last_unstale":"2024-03-04T09:25:20.458390+0000","last_undegraded":"2024-03-04T09:25:20.458390+0000","last_fullsized":"2024-03-04T09:25:20.458390+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:11:51.226417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:20.458472+0000","last_change":"2024-03-04T09:25:19.459312+0000","last_active":"2024-03-04T09:25:20.458472+0000","last_peered":"2024-03-04T09:25:20.458472+0000","last_clean":"2024-03-04T09:25:20.458472+0000","last_became_active":"2024-03-04T09:25:19.458993+0000","last_became_peered":"2024-03-04T09:25:19.458993+0000","last_unstale":"2024-03-04T09:25:20.458472+0000","last_undegraded":"2024-03-04T09:25:20.458472+0000","last_fullsized":"2024-03-04T09:25:20.458472+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:18.418112+0000","last_clean_scrub_stamp":"2024-03-04T09:25:18.418112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:36:58.654226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"19'4","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-04T09:25:21.150271+0000","last_change":"2024-03-04T09:25:18.473489+0000","last_active":"2024-03-04T09:25:21.150271+0000","last_peered":"2024-03-04T09:25:21.150271+0000","last_clean":"2024-03-04T09:25:21.150271+0000","last_became_active":"2024-03-04T09:25:18.473154+0000","last_became_peered":"2024-03-04T09:25:18.473154+0000","last_unstale":"2024-03-04T09:25:21.150271+0000","last_undegraded":"2024-03-04T09:25:21.150271+0000","last_fullsized":"2024-03-04T09:25:21.150271+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:17.403811+0000","last_clean_scrub_stamp":"2024-03-04T09:25:17.403811+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:31:15.318756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574855,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27804,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344036,"statfs":{"total":96636764160,"available":96608292864,"internally_reserved":0,"allocated":802816,"data_stored":661884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574855,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":72546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574857,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":208896,"data_stored":71024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574854,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":71060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574856,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":72546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574855,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":70934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574856,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":212992,"data_stored":71094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574855,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27816,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344024,"statfs":{"total":96636764160,"available":96608280576,"internally_reserved":0,"allocated":811008,"data_stored":661940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-04T09:25:30.135 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-03-04T09:25:30.135 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-04T09:25:30.135 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-03-04T09:25:30.135 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-03-04T09:25:30.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.377+0000 7f11f3200640 1 Processor -- start 2024-03-04T09:25:30.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.377+0000 7f11f3200640 1 -- start start 2024-03-04T09:25:30.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.379+0000 7f11f3200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f11f41455e0 0x7f11f415c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:30.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.379+0000 7f11f3200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f11f406daf0 0x7f11f406df20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:30.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.379+0000 7f11f3200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11f4070cd0 con 0x7f11f40cc4a0 2024-03-04T09:25:30.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.379+0000 7f11f3200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11f40cf690 con 0x7f11f41455e0 2024-03-04T09:25:30.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.379+0000 7f11f3200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11f4070a80 con 0x7f11f406daf0 2024-03-04T09:25:30.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.379+0000 7f11f2800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f11f406daf0 0x7f11f406df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:30.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.379+0000 7f11f2800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f11f406daf0 0x7f11f406df20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.6:51018/0 (socket says 172.21.15.6:51018) 2024-03-04T09:25:30.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.379+0000 7f11f2800640 1 -- 172.21.15.6:0/3073510784 learned_addr learned my addr 172.21.15.6:0/3073510784 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:25:30.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.379+0000 7f11f1e00640 1 --2- 172.21.15.6:0/3073510784 >> v2:172.21.15.146:3300/0 conn(0x7f11f41455e0 0x7f11f415c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:30.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.380+0000 7f11f2800640 1 -- 172.21.15.6:0/3073510784 >> v2:172.21.15.146:3300/0 conn(0x7f11f41455e0 msgr2=0x7f11f415c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:30.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.380+0000 7f11f2800640 1 --2- 172.21.15.6:0/3073510784 >> v2:172.21.15.146:3300/0 conn(0x7f11f41455e0 0x7f11f415c2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:30.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.380+0000 7f11f2800640 1 -- 172.21.15.6:0/3073510784 >> v1:172.21.15.146:6789/0 conn(0x7f11f40cc4a0 legacy=0x7f11f40cef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:30.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.380+0000 7f11f2800640 1 -- 172.21.15.6:0/3073510784 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11f4070fb0 con 0x7f11f406daf0 2024-03-04T09:25:30.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.381+0000 7f11f2800640 1 --2- 172.21.15.6:0/3073510784 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f11f406daf0 0x7f11f406df20 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f11ec009e50 tx=0x7f11ec02bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=89f51fe8d0318bc2 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:30.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.382+0000 7f11f1400640 1 -- 172.21.15.6:0/3073510784 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11ec02ca70 con 0x7f11f406daf0 2024-03-04T09:25:30.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.382+0000 7f11f1400640 1 -- 172.21.15.6:0/3073510784 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11ec02cc10 con 0x7f11f406daf0 2024-03-04T09:25:30.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.382+0000 7f11f1400640 1 -- 172.21.15.6:0/3073510784 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11ec0343f0 con 0x7f11f406daf0 2024-03-04T09:25:30.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.382+0000 7f11f3200640 1 -- 172.21.15.6:0/3073510784 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f11f406daf0 msgr2=0x7f11f406df20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:30.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.382+0000 7f11f3200640 1 --2- 172.21.15.6:0/3073510784 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f11f406daf0 0x7f11f406df20 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f11ec009e50 tx=0x7f11ec02bed0 comp rx=0 tx=0).stop 2024-03-04T09:25:30.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.383+0000 7f11f3200640 1 -- 172.21.15.6:0/3073510784 shutdown_connections 2024-03-04T09:25:30.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.383+0000 7f11f3200640 1 --2- 172.21.15.6:0/3073510784 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f11f406daf0 0x7f11f406df20 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:30.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.383+0000 7f11f3200640 1 --2- 172.21.15.6:0/3073510784 >> v2:172.21.15.146:3300/0 conn(0x7f11f41455e0 0x7f11f415c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:30.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.383+0000 7f11f3200640 1 -- 172.21.15.6:0/3073510784 >> 172.21.15.6:0/3073510784 conn(0x7f11f40b17c0 msgr2=0x7f11f40b1ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:30.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.383+0000 7f11f3200640 1 -- 172.21.15.6:0/3073510784 shutdown_connections 2024-03-04T09:25:30.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.383+0000 7f11f3200640 1 -- 172.21.15.6:0/3073510784 wait complete. 2024-03-04T09:25:30.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.384+0000 7f11f3200640 1 Processor -- start 2024-03-04T09:25:30.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.384+0000 7f11f3200640 1 -- start start 2024-03-04T09:25:30.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.385+0000 7f11f3200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f11f406daf0 0x7f11f40b4550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:30.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.385+0000 7f11f3200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f11f40cc4a0 0x7f11f415d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:30.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.385+0000 7f11f3200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f11f41455e0 0x7f11f4137390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:30.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.385+0000 7f11f3200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11f4071570 con 0x7f11f40cc4a0 2024-03-04T09:25:30.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.385+0000 7f11f3200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11f406e460 con 0x7f11f406daf0 2024-03-04T09:25:30.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.385+0000 7f11f3200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11f41459c0 con 0x7f11f41455e0 2024-03-04T09:25:30.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.386+0000 7f11e9400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f11f40cc4a0 0x7f11f415d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:30.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.386+0000 7f11e9400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f11f40cc4a0 0x7f11f415d500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45704/0 (socket says 172.21.15.6:45704) 2024-03-04T09:25:30.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.386+0000 7f11e9400640 1 -- 172.21.15.6:0/4085739332 learned_addr learned my addr 172.21.15.6:0/4085739332 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:25:30.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.386+0000 7f11f1e00640 1 --2- 172.21.15.6:0/4085739332 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f11f406daf0 0x7f11f40b4550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:30.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.386+0000 7f11f2800640 1 --2- 172.21.15.6:0/4085739332 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f11f41455e0 0x7f11f4137390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:30.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.386+0000 7f11e9400640 1 -- 172.21.15.6:0/4085739332 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f11f41455e0 msgr2=0x7f11f4137390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:30.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.386+0000 7f11e9400640 1 --2- 172.21.15.6:0/4085739332 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f11f41455e0 0x7f11f4137390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:30.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.386+0000 7f11e9400640 1 -- 172.21.15.6:0/4085739332 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f11f406daf0 msgr2=0x7f11f40b4550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:30.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.386+0000 7f11e9400640 1 --2- 172.21.15.6:0/4085739332 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f11f406daf0 0x7f11f40b4550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:30.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.386+0000 7f11e9400640 1 -- 172.21.15.6:0/4085739332 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11f40c4d30 con 0x7f11f40cc4a0 2024-03-04T09:25:30.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.386+0000 7f11f1e00640 1 --2- 172.21.15.6:0/4085739332 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f11f406daf0 0x7f11f40b4550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T09:25:30.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.386+0000 7f11e9400640 1 --2- 172.21.15.6:0/4085739332 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f11f40cc4a0 0x7f11f415d500 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f11dc00c430 tx=0x7f11dc00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:30.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.387+0000 7f11eaa00640 1 -- 172.21.15.6:0/4085739332 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11dc0052d0 con 0x7f11f40cc4a0 2024-03-04T09:25:30.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.387+0000 7f11eaa00640 1 -- 172.21.15.6:0/4085739332 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11dc005490 con 0x7f11f40cc4a0 2024-03-04T09:25:30.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.387+0000 7f11eaa00640 1 -- 172.21.15.6:0/4085739332 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11dc007260 con 0x7f11f40cc4a0 2024-03-04T09:25:30.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.387+0000 7f11f3200640 1 -- 172.21.15.6:0/4085739332 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11f40ae290 con 0x7f11f40cc4a0 2024-03-04T09:25:30.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.387+0000 7f11f3200640 1 -- 172.21.15.6:0/4085739332 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11f40ae850 con 0x7f11f40cc4a0 2024-03-04T09:25:30.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.388+0000 7f11f3200640 1 -- 172.21.15.6:0/4085739332 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11f404f330 con 0x7f11f40cc4a0 2024-03-04T09:25:30.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.389+0000 7f11eaa00640 1 -- 172.21.15.6:0/4085739332 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f11dc007400 con 0x7f11f40cc4a0 2024-03-04T09:25:30.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.389+0000 7f11eaa00640 1 --2- 172.21.15.6:0/4085739332 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f11d006b4d0 0x7f11d006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:30.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.390+0000 7f11eaa00640 1 -- 172.21.15.6:0/4085739332 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f11dc0896f0 con 0x7f11f40cc4a0 2024-03-04T09:25:30.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.390+0000 7f11f1e00640 1 --2- 172.21.15.6:0/4085739332 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f11d006b4d0 0x7f11d006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:30.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.391+0000 7f11f1e00640 1 --2- 172.21.15.6:0/4085739332 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f11d006b4d0 0x7f11d006d990 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f11f41f72d0 tx=0x7f11e4008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:30.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.394+0000 7f11eaa00640 1 -- 172.21.15.6:0/4085739332 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f11dc058570 con 0x7f11f40cc4a0 2024-03-04T09:25:30.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.624+0000 7f11f3200640 1 -- 172.21.15.6:0/4085739332 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f11f40ce050 con 0x7f11f40cc4a0 2024-03-04T09:25:30.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.625+0000 7f11eaa00640 1 -- 172.21.15.6:0/4085739332 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f11dc057e70 con 0x7f11f40cc4a0 2024-03-04T09:25:30.621 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:25:30.622 INFO:teuthology.orchestra.run.smithi006.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-04T09:25:30.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.627+0000 7f11f3200640 1 -- 172.21.15.6:0/4085739332 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f11d006b4d0 msgr2=0x7f11d006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:30.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.627+0000 7f11f3200640 1 --2- 172.21.15.6:0/4085739332 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f11d006b4d0 0x7f11d006d990 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f11f41f72d0 tx=0x7f11e4008040 comp rx=0 tx=0).stop 2024-03-04T09:25:30.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.627+0000 7f11f3200640 1 -- 172.21.15.6:0/4085739332 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f11f40cc4a0 msgr2=0x7f11f415d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:30.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.627+0000 7f11f3200640 1 --2- 172.21.15.6:0/4085739332 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f11f40cc4a0 0x7f11f415d500 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f11dc00c430 tx=0x7f11dc00c8f0 comp rx=0 tx=0).stop 2024-03-04T09:25:30.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.627+0000 7f11f3200640 1 -- 172.21.15.6:0/4085739332 shutdown_connections 2024-03-04T09:25:30.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.627+0000 7f11f3200640 1 --2- 172.21.15.6:0/4085739332 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f11d006b4d0 0x7f11d006d990 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:30.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.627+0000 7f11f3200640 1 --2- 172.21.15.6:0/4085739332 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f11f41455e0 0x7f11f4137390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:30.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.627+0000 7f11f3200640 1 --2- 172.21.15.6:0/4085739332 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f11f40cc4a0 0x7f11f415d500 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:30.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.627+0000 7f11f3200640 1 --2- 172.21.15.6:0/4085739332 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f11f406daf0 0x7f11f40b4550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:30.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.627+0000 7f11f3200640 1 -- 172.21.15.6:0/4085739332 >> 172.21.15.6:0/4085739332 conn(0x7f11f40b17c0 msgr2=0x7f11f40b0ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:30.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.627+0000 7f11f3200640 1 -- 172.21.15.6:0/4085739332 shutdown_connections 2024-03-04T09:25:30.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.627+0000 7f11f3200640 1 -- 172.21.15.6:0/4085739332 wait complete. 2024-03-04T09:25:30.642 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-03-04T09:25:30.643 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T09:25:30.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.855+0000 7ff5eac00640 1 Processor -- start 2024-03-04T09:25:30.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.855+0000 7ff5eac00640 1 -- start start 2024-03-04T09:25:30.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.857+0000 7ff5eac00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff5e40af1a0 0x7ff5e415c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:30.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.857+0000 7ff5eac00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ff5e40cdc20 0x7ff5e40b2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:30.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.857+0000 7ff5eac00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5e415cc40 con 0x7ff5e40af580 2024-03-04T09:25:30.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.857+0000 7ff5eac00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5e415cf20 con 0x7ff5e40cdc20 2024-03-04T09:25:30.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.857+0000 7ff5eac00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5e415d200 con 0x7ff5e40af1a0 2024-03-04T09:25:30.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.857+0000 7ff5e3400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7ff5e40af580 0x7ff5e40b4c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:36806/0 (socket says 172.21.15.6:36806) 2024-03-04T09:25:30.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.857+0000 7ff5e3400640 1 -- 172.21.15.6:0/3589740264 learned_addr learned my addr 172.21.15.6:0/3589740264 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:25:30.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.857+0000 7ff5e3e00640 1 --2- 172.21.15.6:0/3589740264 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff5e40af1a0 0x7ff5e415c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:30.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.857+0000 7ff5e8c00640 1 --2- 172.21.15.6:0/3589740264 >> v2:172.21.15.146:3300/0 conn(0x7ff5e40cdc20 0x7ff5e40b2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:30.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.858+0000 7ff5e2a00640 1 -- 172.21.15.6:0/3589740264 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1750226331 0 0) 0x7ff5e415cc40 con 0x7ff5e40af580 2024-03-04T09:25:30.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.858+0000 7ff5e2a00640 1 -- 172.21.15.6:0/3589740264 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5c8003520 con 0x7ff5e40af580 2024-03-04T09:25:30.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.858+0000 7ff5e2a00640 1 -- 172.21.15.6:0/3589740264 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2193787583 0 0) 0x7ff5c8003520 con 0x7ff5e40af580 2024-03-04T09:25:30.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.858+0000 7ff5e2a00640 1 -- 172.21.15.6:0/3589740264 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff5e40af1a0 msgr2=0x7ff5e415c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:30.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.858+0000 7ff5e2a00640 1 --2- 172.21.15.6:0/3589740264 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff5e40af1a0 0x7ff5e415c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:30.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.858+0000 7ff5e2a00640 1 -- 172.21.15.6:0/3589740264 >> v2:172.21.15.146:3300/0 conn(0x7ff5e40cdc20 msgr2=0x7ff5e40b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:30.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.858+0000 7ff5e2a00640 1 --2- 172.21.15.6:0/3589740264 >> v2:172.21.15.146:3300/0 conn(0x7ff5e40cdc20 0x7ff5e40b2dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:30.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.858+0000 7ff5e2a00640 1 -- 172.21.15.6:0/3589740264 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5e415d4e0 con 0x7ff5e40af580 2024-03-04T09:25:30.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.858+0000 7ff5e2a00640 1 -- 172.21.15.6:0/3589740264 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7ff5d0001670 con 0x7ff5e40af580 2024-03-04T09:25:30.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.858+0000 7ff5e2a00640 1 -- 172.21.15.6:0/3589740264 >> v1:172.21.15.6:6789/0 conn(0x7ff5e40af580 legacy=0x7ff5e40b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:30.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.858+0000 7ff5e2a00640 1 --2- 172.21.15.6:0/3589740264 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff5c8003a20 0x7ff5c8003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:30.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.858+0000 7ff5e2a00640 1 --2- 172.21.15.6:0/3589740264 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff5e40af1a0 0x7ff5c8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:30.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.858+0000 7ff5e2a00640 1 --2- 172.21.15.6:0/3589740264 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff5c8004d40 0x7ff5c80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:30.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.858+0000 7ff5e2a00640 1 -- 172.21.15.6:0/3589740264 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5e415cf20 con 0x7ff5c8003a20 2024-03-04T09:25:30.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.858+0000 7ff5e2a00640 1 -- 172.21.15.6:0/3589740264 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5e415d200 con 0x7ff5c8004d40 2024-03-04T09:25:30.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.858+0000 7ff5e2a00640 1 -- 172.21.15.6:0/3589740264 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5c800ba10 con 0x7ff5e40af1a0 2024-03-04T09:25:30.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.858+0000 7ff5e3e00640 1 --2- 172.21.15.6:0/3589740264 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff5c8003a20 0x7ff5c8003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:30.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.859+0000 7ff5e8c00640 1 --2- 172.21.15.6:0/3589740264 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff5c8004d40 0x7ff5c80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:30.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.859+0000 7ff5e3e00640 1 -- 172.21.15.6:0/3589740264 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff5e40af1a0 msgr2=0x7ff5c8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:30.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.859+0000 7ff5e3e00640 1 --2- 172.21.15.6:0/3589740264 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff5e40af1a0 0x7ff5c8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:30.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.859+0000 7ff5e3e00640 1 -- 172.21.15.6:0/3589740264 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff5c8004d40 msgr2=0x7ff5c80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:30.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.859+0000 7ff5e3e00640 1 --2- 172.21.15.6:0/3589740264 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff5c8004d40 0x7ff5c80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:30.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.859+0000 7ff5e3e00640 1 -- 172.21.15.6:0/3589740264 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5c8003520 con 0x7ff5c8003a20 2024-03-04T09:25:30.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.859+0000 7ff5e8c00640 1 --2- 172.21.15.6:0/3589740264 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff5c8004d40 0x7ff5c80051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:25:30.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.860+0000 7ff5e3e00640 1 --2- 172.21.15.6:0/3589740264 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff5c8003a20 0x7ff5c8003e40 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7ff5d4009ac0 tx=0x7ff5c8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eac681f871e77e4a server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:30.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.860+0000 7ff5e2a00640 1 -- 172.21.15.6:0/3589740264 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5d4039070 con 0x7ff5c8003a20 2024-03-04T09:25:30.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.860+0000 7ff5e2a00640 1 -- 172.21.15.6:0/3589740264 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5d402c710 con 0x7ff5c8003a20 2024-03-04T09:25:30.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.860+0000 7ff5e2a00640 1 -- 172.21.15.6:0/3589740264 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5d4009690 con 0x7ff5c8003a20 2024-03-04T09:25:30.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.861+0000 7ff5eac00640 1 -- 172.21.15.6:0/3589740264 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff5c8003a20 msgr2=0x7ff5c8003e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:30.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.861+0000 7ff5eac00640 1 --2- 172.21.15.6:0/3589740264 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff5c8003a20 0x7ff5c8003e40 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7ff5d4009ac0 tx=0x7ff5c8005830 comp rx=0 tx=0).stop 2024-03-04T09:25:30.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.861+0000 7ff5e8c00640 1 -- 172.21.15.6:0/3589740264 reap_dead start 2024-03-04T09:25:30.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.861+0000 7ff5eac00640 1 -- 172.21.15.6:0/3589740264 shutdown_connections 2024-03-04T09:25:30.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.861+0000 7ff5eac00640 1 -- 172.21.15.6:0/3589740264 >> 172.21.15.6:0/3589740264 conn(0x7ff5e406afb0 msgr2=0x7ff5e406b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:30.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.861+0000 7ff5eac00640 1 -- 172.21.15.6:0/3589740264 shutdown_connections 2024-03-04T09:25:30.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.861+0000 7ff5eac00640 1 -- 172.21.15.6:0/3589740264 wait complete. 2024-03-04T09:25:30.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.863+0000 7ff5eac00640 1 Processor -- start 2024-03-04T09:25:30.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.863+0000 7ff5eac00640 1 -- start start 2024-03-04T09:25:30.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.864+0000 7ff5eac00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff5c8003a20 0x7ff5e414c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:30.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.864+0000 7ff5eac00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff5e41f3450 0x7ff5e40b1140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:30.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.864+0000 7ff5eac00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff5e40cc8c0 0x7ff5e40ccd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:30.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.864+0000 7ff5eac00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5e415c990 con 0x7ff5c8003a20 2024-03-04T09:25:30.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.864+0000 7ff5eac00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5e415c810 con 0x7ff5e41f3450 2024-03-04T09:25:30.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.864+0000 7ff5eac00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5e40b3310 con 0x7ff5e40cc8c0 2024-03-04T09:25:30.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.864+0000 7ff5e3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff5c8003a20 0x7ff5e414c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:30.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.864+0000 7ff5e3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff5c8003a20 0x7ff5e414c450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45718/0 (socket says 172.21.15.6:45718) 2024-03-04T09:25:30.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.864+0000 7ff5e3e00640 1 -- 172.21.15.6:0/4037718738 learned_addr learned my addr 172.21.15.6:0/4037718738 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:25:30.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.865+0000 7ff5e3e00640 1 -- 172.21.15.6:0/4037718738 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff5e40cc8c0 msgr2=0x7ff5e40ccd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:30.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.865+0000 7ff5e8c00640 1 --2- 172.21.15.6:0/4037718738 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff5e40cc8c0 0x7ff5e40ccd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:30.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.865+0000 7ff5e3e00640 1 --2- 172.21.15.6:0/4037718738 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff5e40cc8c0 0x7ff5e40ccd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:30.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.865+0000 7ff5e3e00640 1 -- 172.21.15.6:0/4037718738 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff5e41f3450 msgr2=0x7ff5e40b1140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:30.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.865+0000 7ff5e3e00640 1 --2- 172.21.15.6:0/4037718738 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff5e41f3450 0x7ff5e40b1140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:30.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.865+0000 7ff5e3e00640 1 -- 172.21.15.6:0/4037718738 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5e406ccb0 con 0x7ff5c8003a20 2024-03-04T09:25:30.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.865+0000 7ff5e3e00640 1 --2- 172.21.15.6:0/4037718738 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff5c8003a20 0x7ff5e414c450 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7ff5d4037040 tx=0x7ff5d4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:30.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.865+0000 7ff5dbe00640 1 -- 172.21.15.6:0/4037718738 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5d4039050 con 0x7ff5c8003a20 2024-03-04T09:25:30.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.865+0000 7ff5dbe00640 1 -- 172.21.15.6:0/4037718738 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5d4045070 con 0x7ff5c8003a20 2024-03-04T09:25:30.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.865+0000 7ff5eac00640 1 -- 172.21.15.6:0/4037718738 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5e406cf90 con 0x7ff5c8003a20 2024-03-04T09:25:30.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.866+0000 7ff5dbe00640 1 -- 172.21.15.6:0/4037718738 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5d4003a30 con 0x7ff5c8003a20 2024-03-04T09:25:30.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.866+0000 7ff5eac00640 1 -- 172.21.15.6:0/4037718738 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5e406d550 con 0x7ff5c8003a20 2024-03-04T09:25:30.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.867+0000 7ff5eac00640 1 -- 172.21.15.6:0/4037718738 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5e4069510 con 0x7ff5c8003a20 2024-03-04T09:25:30.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.868+0000 7ff5dbe00640 1 -- 172.21.15.6:0/4037718738 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff5d4003bd0 con 0x7ff5c8003a20 2024-03-04T09:25:30.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.869+0000 7ff5dbe00640 1 --2- 172.21.15.6:0/4037718738 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff5c406b4d0 0x7ff5c406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:30.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.869+0000 7ff5dbe00640 1 -- 172.21.15.6:0/4037718738 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff5d40ac500 con 0x7ff5c8003a20 2024-03-04T09:25:30.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.869+0000 7ff5e3400640 1 --2- 172.21.15.6:0/4037718738 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff5c406b4d0 0x7ff5c406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:30.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.871+0000 7ff5e3400640 1 --2- 172.21.15.6:0/4037718738 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff5c406b4d0 0x7ff5c406d990 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7ff5d0000fd0 tx=0x7ff5d0001b50 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:30.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:30.873+0000 7ff5dbe00640 1 -- 172.21.15.6:0/4037718738 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff5d407b430 con 0x7ff5c8003a20 2024-03-04T09:25:31.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.071+0000 7ff5eac00640 1 -- 172.21.15.6:0/4037718738 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff5e404f3a0 con 0x7ff5c8003a20 2024-03-04T09:25:31.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.073+0000 7ff5dbe00640 1 -- 172.21.15.6:0/4037718738 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16557 (secure 0 0 0) 0x7ff5d407ad30 con 0x7ff5c8003a20 2024-03-04T09:25:31.070 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:25:31.070 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":20,"fsid":"edce0578-5ae4-4402-b45a-552bdef84d7c","created":"2024-03-04T09:24:54.523780+0000","modified":"2024-03-04T09:25:20.446333+0000","last_up_change":"2024-03-04T09:25:07.089049+0000","last_in_change":"2024-03-04T09:24:57.982077+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T09:25:07.935198+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-04T09:25:17.118573+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-04T09:25:17.930536+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a96574f-81f1-4bc6-8463-757520b84a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6825","nonce":1364449401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6827","nonce":1364449401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6831","nonce":1364449401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6829","nonce":1364449401}]},"public_addr":"172.21.15.6:6825/1364449401","cluster_addr":"172.21.15.6:6827/1364449401","heartbeat_back_addr":"172.21.15.6:6831/1364449401","heartbeat_front_addr":"172.21.15.6:6829/1364449401","state":["exists","up"]},{"osd":1,"uuid":"7d79b9c1-6b87-476a-bc36-7db5ee64cd90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6809","nonce":2494103078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6811","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6813","nonce":2494103078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6821","nonce":2494103078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6817","nonce":2494103078}]},"public_addr":"172.21.15.6:6809/2494103078","cluster_addr":"172.21.15.6:6813/2494103078","heartbeat_back_addr":"172.21.15.6:6821/2494103078","heartbeat_front_addr":"172.21.15.6:6817/2494103078","state":["exists","up"]},{"osd":2,"uuid":"492331b9-4ee9-40b6-bacd-85d44cd6c45a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6812","nonce":1500066681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6816","nonce":1500066681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6824","nonce":1500066681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6820","nonce":1500066681}]},"public_addr":"172.21.15.6:6812/1500066681","cluster_addr":"172.21.15.6:6816/1500066681","heartbeat_back_addr":"172.21.15.6:6824/1500066681","heartbeat_front_addr":"172.21.15.6:6820/1500066681","state":["exists","up"]},{"osd":3,"uuid":"21bb2efe-9ba7-4bdc-bc31-180cd1102e21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6801","nonce":1558806123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6803","nonce":1558806123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6807","nonce":1558806123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6805","nonce":1558806123}]},"public_addr":"172.21.15.6:6801/1558806123","cluster_addr":"172.21.15.6:6803/1558806123","heartbeat_back_addr":"172.21.15.6:6807/1558806123","heartbeat_front_addr":"172.21.15.6:6805/1558806123","state":["exists","up"]},{"osd":4,"uuid":"6e2c4b71-19d9-4737-83d9-558705d8644a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6801","nonce":1526135780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6803","nonce":1526135780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6807","nonce":1526135780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6805","nonce":1526135780}]},"public_addr":"172.21.15.146:6801/1526135780","cluster_addr":"172.21.15.146:6803/1526135780","heartbeat_back_addr":"172.21.15.146:6807/1526135780","heartbeat_front_addr":"172.21.15.146:6805/1526135780","state":["exists","up"]},{"osd":5,"uuid":"fb69c525-f1be-4b7d-ab32-6c29b0d6fddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6809","nonce":460262507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6811","nonce":460262507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6815","nonce":460262507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6813","nonce":460262507}]},"public_addr":"172.21.15.146:6809/460262507","cluster_addr":"172.21.15.146:6811/460262507","heartbeat_back_addr":"172.21.15.146:6815/460262507","heartbeat_front_addr":"172.21.15.146:6813/460262507","state":["exists","up"]},{"osd":6,"uuid":"c32e7fed-370d-4401-8ce0-6a4335b0d168","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6823","nonce":1392897493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6827","nonce":1392897493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6831","nonce":1392897493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6829","nonce":1392897493}]},"public_addr":"172.21.15.146:6823/1392897493","cluster_addr":"172.21.15.146:6827/1392897493","heartbeat_back_addr":"172.21.15.146:6831/1392897493","heartbeat_front_addr":"172.21.15.146:6829/1392897493","state":["exists","up"]},{"osd":7,"uuid":"75a79b44-fa5f-426d-a7d3-71f21f78835c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6817","nonce":3268323156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6819","nonce":3268323156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6825","nonce":3268323156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6822","nonce":3268323156}]},"public_addr":"172.21.15.146:6817/3268323156","cluster_addr":"172.21.15.146:6819/3268323156","heartbeat_back_addr":"172.21.15.146:6825/3268323156","heartbeat_front_addr":"172.21.15.146:6822/3268323156","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.310437+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.295850+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.294674+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.526855+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.300023+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.595160+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.346046+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.558741+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T09:25:31.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.076+0000 7ff5eac00640 1 -- 172.21.15.6:0/4037718738 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff5c406b4d0 msgr2=0x7ff5c406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:31.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.076+0000 7ff5eac00640 1 --2- 172.21.15.6:0/4037718738 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff5c406b4d0 0x7ff5c406d990 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7ff5d0000fd0 tx=0x7ff5d0001b50 comp rx=0 tx=0).stop 2024-03-04T09:25:31.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.076+0000 7ff5eac00640 1 -- 172.21.15.6:0/4037718738 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff5c8003a20 msgr2=0x7ff5e414c450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:31.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.076+0000 7ff5eac00640 1 --2- 172.21.15.6:0/4037718738 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff5c8003a20 0x7ff5e414c450 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7ff5d4037040 tx=0x7ff5d4004730 comp rx=0 tx=0).stop 2024-03-04T09:25:31.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.076+0000 7ff5eac00640 1 -- 172.21.15.6:0/4037718738 shutdown_connections 2024-03-04T09:25:31.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.076+0000 7ff5eac00640 1 --2- 172.21.15.6:0/4037718738 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff5c406b4d0 0x7ff5c406d990 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:31.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.076+0000 7ff5eac00640 1 --2- 172.21.15.6:0/4037718738 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff5e40cc8c0 0x7ff5e40ccd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:31.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.076+0000 7ff5eac00640 1 --2- 172.21.15.6:0/4037718738 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff5e41f3450 0x7ff5e40b1140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:31.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.076+0000 7ff5eac00640 1 --2- 172.21.15.6:0/4037718738 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff5c8003a20 0x7ff5e414c450 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:31.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.076+0000 7ff5eac00640 1 -- 172.21.15.6:0/4037718738 >> 172.21.15.6:0/4037718738 conn(0x7ff5e406afb0 msgr2=0x7ff5e40ad970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:31.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.076+0000 7ff5eac00640 1 -- 172.21.15.6:0/4037718738 shutdown_connections 2024-03-04T09:25:31.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.076+0000 7ff5eac00640 1 -- 172.21.15.6:0/4037718738 wait complete. 2024-03-04T09:25:31.093 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T09:25:07.935198+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-04T09:25:17.118573+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-04T09:25:17.930536+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-04T09:25:31.094 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-04T09:25:31.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.320+0000 7f6f01a00640 1 Processor -- start 2024-03-04T09:25:31.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.320+0000 7f6f01a00640 1 -- start start 2024-03-04T09:25:31.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.322+0000 7f6f01a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6efc0cf3a0 0x7f6efc070ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:31.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.322+0000 7f6f01a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6efc071c60 con 0x7f6efc15afa0 2024-03-04T09:25:31.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.322+0000 7f6f01a00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6efc071f40 con 0x7f6efc0cd1e0 2024-03-04T09:25:31.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.322+0000 7f6f01a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6efc0cfa10 con 0x7f6efc0cf3a0 2024-03-04T09:25:31.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.323+0000 7f6efaa00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f6efc15afa0 0x7f6efc0ccad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:36820/0 (socket says 172.21.15.6:36820) 2024-03-04T09:25:31.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.323+0000 7f6efaa00640 1 -- 172.21.15.6:0/3192909468 learned_addr learned my addr 172.21.15.6:0/3192909468 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:25:31.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.323+0000 7f6efb400640 1 --2- 172.21.15.6:0/3192909468 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6efc0cf3a0 0x7f6efc070ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:31.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.323+0000 7f6efa000640 1 -- 172.21.15.6:0/3192909468 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2558450176 0 0) 0x7f6efc071c60 con 0x7f6efc15afa0 2024-03-04T09:25:31.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.323+0000 7f6efa000640 1 -- 172.21.15.6:0/3192909468 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ee0003520 con 0x7f6efc15afa0 2024-03-04T09:25:31.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.323+0000 7f6efa000640 1 -- 172.21.15.6:0/3192909468 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4180445007 0 0) 0x7f6ee0003520 con 0x7f6efc15afa0 2024-03-04T09:25:31.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.323+0000 7f6efa000640 1 -- 172.21.15.6:0/3192909468 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6efc0cf3a0 msgr2=0x7f6efc070ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:31.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.323+0000 7f6efa000640 1 --2- 172.21.15.6:0/3192909468 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6efc0cf3a0 0x7f6efc070ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:31.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.323+0000 7f6efa000640 1 -- 172.21.15.6:0/3192909468 >> v1:172.21.15.146:6789/0 conn(0x7f6efc0cd1e0 legacy=0x7f6efc0b3ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:31.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.323+0000 7f6efa000640 1 -- 172.21.15.6:0/3192909468 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6efc072220 con 0x7f6efc15afa0 2024-03-04T09:25:31.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.323+0000 7f6efa000640 1 -- 172.21.15.6:0/3192909468 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7f6eec001670 con 0x7f6efc15afa0 2024-03-04T09:25:31.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.323+0000 7f6efa000640 1 -- 172.21.15.6:0/3192909468 >> v1:172.21.15.6:6789/0 conn(0x7f6efc15afa0 legacy=0x7f6efc0ccad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:31.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.323+0000 7f6efa000640 1 --2- 172.21.15.6:0/3192909468 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6ee00039e0 0x7f6ee0003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:31.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.323+0000 7f6efa000640 1 --2- 172.21.15.6:0/3192909468 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6efc0cf3a0 0x7f6ee0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:31.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.324+0000 7f6efa000640 1 --2- 172.21.15.6:0/3192909468 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6ee0004d40 0x7f6ee00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:31.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.324+0000 7f6efa000640 1 -- 172.21.15.6:0/3192909468 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6efc0cfa10 con 0x7f6ee0004d40 2024-03-04T09:25:31.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.324+0000 7f6efa000640 1 -- 172.21.15.6:0/3192909468 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ee000baa0 con 0x7f6ee00039e0 2024-03-04T09:25:31.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.324+0000 7f6efa000640 1 -- 172.21.15.6:0/3192909468 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ee000bca0 con 0x7f6efc0cf3a0 2024-03-04T09:25:31.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.324+0000 7f6efb400640 1 --2- 172.21.15.6:0/3192909468 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6ee0004d40 0x7f6ee00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:31.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.324+0000 7f6ef2000640 1 --2- 172.21.15.6:0/3192909468 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6efc0cf3a0 0x7f6ee0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:31.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.324+0000 7f6efaa00640 1 --2- 172.21.15.6:0/3192909468 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6ee00039e0 0x7f6ee0003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:31.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.324+0000 7f6efb400640 1 -- 172.21.15.6:0/3192909468 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6efc0cf3a0 msgr2=0x7f6ee0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:31.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.324+0000 7f6efb400640 1 --2- 172.21.15.6:0/3192909468 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6efc0cf3a0 0x7f6ee0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:31.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.324+0000 7f6efb400640 1 -- 172.21.15.6:0/3192909468 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6ee00039e0 msgr2=0x7f6ee0003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:31.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.324+0000 7f6efb400640 1 --2- 172.21.15.6:0/3192909468 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6ee00039e0 0x7f6ee0003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:31.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.324+0000 7f6efb400640 1 -- 172.21.15.6:0/3192909468 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ee0003520 con 0x7f6ee0004d40 2024-03-04T09:25:31.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.324+0000 7f6efaa00640 1 --2- 172.21.15.6:0/3192909468 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6ee00039e0 0x7f6ee0003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:25:31.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.325+0000 7f6efb400640 1 --2- 172.21.15.6:0/3192909468 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6ee0004d40 0x7f6ee00051c0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f6ef4009f90 tx=0x7f6ee0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f469e85cd8888eba server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:31.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.326+0000 7f6efa000640 1 -- 172.21.15.6:0/3192909468 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ef4039070 con 0x7f6ee0004d40 2024-03-04T09:25:31.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.326+0000 7f6efa000640 1 -- 172.21.15.6:0/3192909468 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ef402cbd0 con 0x7f6ee0004d40 2024-03-04T09:25:31.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.326+0000 7f6efa000640 1 -- 172.21.15.6:0/3192909468 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ef4009c10 con 0x7f6ee0004d40 2024-03-04T09:25:31.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.326+0000 7f6f01a00640 1 -- 172.21.15.6:0/3192909468 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6ee0004d40 msgr2=0x7f6ee00051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:31.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.326+0000 7f6f01a00640 1 --2- 172.21.15.6:0/3192909468 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6ee0004d40 0x7f6ee00051c0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f6ef4009f90 tx=0x7f6ee0005830 comp rx=0 tx=0).stop 2024-03-04T09:25:31.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.326+0000 7f6efb400640 1 -- 172.21.15.6:0/3192909468 reap_dead start 2024-03-04T09:25:31.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.326+0000 7f6f01a00640 1 -- 172.21.15.6:0/3192909468 shutdown_connections 2024-03-04T09:25:31.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.326+0000 7f6f01a00640 1 -- 172.21.15.6:0/3192909468 >> 172.21.15.6:0/3192909468 conn(0x7f6efc06b3d0 msgr2=0x7f6efc06b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:31.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.326+0000 7f6f01a00640 1 -- 172.21.15.6:0/3192909468 shutdown_connections 2024-03-04T09:25:31.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.326+0000 7f6f01a00640 1 -- 172.21.15.6:0/3192909468 wait complete. 2024-03-04T09:25:31.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.328+0000 7f6f01a00640 1 Processor -- start 2024-03-04T09:25:31.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.328+0000 7f6f01a00640 1 -- start start 2024-03-04T09:25:31.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.329+0000 7f6f01a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6efc152040 0x7f6efc141850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:31.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.329+0000 7f6f01a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6efc143740 0x7f6efc1f5ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:31.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.329+0000 7f6f01a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6efc152420 0x7f6efc140bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:31.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.329+0000 7f6f01a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6efc070e60 con 0x7f6efc143740 2024-03-04T09:25:31.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.329+0000 7f6f01a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6efc070ce0 con 0x7f6efc152420 2024-03-04T09:25:31.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.329+0000 7f6f01a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6efc070fe0 con 0x7f6efc152040 2024-03-04T09:25:31.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.330+0000 7f6ef2000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6efc143740 0x7f6efc1f5ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:31.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.330+0000 7f6ef2000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6efc143740 0x7f6efc1f5ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45736/0 (socket says 172.21.15.6:45736) 2024-03-04T09:25:31.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.330+0000 7f6ef2000640 1 -- 172.21.15.6:0/2235307499 learned_addr learned my addr 172.21.15.6:0/2235307499 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:25:31.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.330+0000 7f6efaa00640 1 --2- 172.21.15.6:0/2235307499 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6efc152040 0x7f6efc141850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:31.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.330+0000 7f6efb400640 1 --2- 172.21.15.6:0/2235307499 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6efc152420 0x7f6efc140bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:31.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.330+0000 7f6ef2000640 1 -- 172.21.15.6:0/2235307499 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6efc152040 msgr2=0x7f6efc141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:31.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.330+0000 7f6ef2000640 1 --2- 172.21.15.6:0/2235307499 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6efc152040 0x7f6efc141850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:31.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.330+0000 7f6ef2000640 1 -- 172.21.15.6:0/2235307499 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6efc152420 msgr2=0x7f6efc140bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:31.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.330+0000 7f6ef2000640 1 --2- 172.21.15.6:0/2235307499 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6efc152420 0x7f6efc140bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:31.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.330+0000 7f6ef2000640 1 -- 172.21.15.6:0/2235307499 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6efc143d70 con 0x7f6efc143740 2024-03-04T09:25:31.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.330+0000 7f6efb400640 1 --2- 172.21.15.6:0/2235307499 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6efc152420 0x7f6efc140bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:25:31.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.330+0000 7f6ef2000640 1 --2- 172.21.15.6:0/2235307499 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6efc143740 0x7f6efc1f5ea0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f6ee400e550 tx=0x7f6ee400ea10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:31.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.331+0000 7f6ef3400640 1 -- 172.21.15.6:0/2235307499 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ee40052e0 con 0x7f6efc143740 2024-03-04T09:25:31.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.331+0000 7f6ef3400640 1 -- 172.21.15.6:0/2235307499 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ee4018070 con 0x7f6efc143740 2024-03-04T09:25:31.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.331+0000 7f6f01a00640 1 -- 172.21.15.6:0/2235307499 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6efc144050 con 0x7f6efc143740 2024-03-04T09:25:31.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.331+0000 7f6ef3400640 1 -- 172.21.15.6:0/2235307499 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ee4003810 con 0x7f6efc143740 2024-03-04T09:25:31.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.331+0000 7f6f01a00640 1 -- 172.21.15.6:0/2235307499 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6efc144610 con 0x7f6efc143740 2024-03-04T09:25:31.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.332+0000 7f6f01a00640 1 -- 172.21.15.6:0/2235307499 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6efc04f330 con 0x7f6efc143740 2024-03-04T09:25:31.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.333+0000 7f6ef3400640 1 -- 172.21.15.6:0/2235307499 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6ee40039b0 con 0x7f6efc143740 2024-03-04T09:25:31.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.334+0000 7f6ef3400640 1 --2- 172.21.15.6:0/2235307499 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f6ed806b4d0 0x7f6ed806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:31.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.334+0000 7f6ef3400640 1 -- 172.21.15.6:0/2235307499 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6ee4089f40 con 0x7f6efc143740 2024-03-04T09:25:31.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.334+0000 7f6efaa00640 1 --2- 172.21.15.6:0/2235307499 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f6ed806b4d0 0x7f6ed806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:31.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.335+0000 7f6efaa00640 1 --2- 172.21.15.6:0/2235307499 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f6ed806b4d0 0x7f6ed806d990 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f6eec001eb0 tx=0x7f6eec009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:31.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.338+0000 7f6ef3400640 1 -- 172.21.15.6:0/2235307499 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6ee4058d20 con 0x7f6efc143740 2024-03-04T09:25:31.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.537+0000 7f6f01a00640 1 -- 172.21.15.6:0/2235307499 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f6efc0b4b70 con 0x7f6efc143740 2024-03-04T09:25:31.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.538+0000 7f6ef3400640 1 -- 172.21.15.6:0/2235307499 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6ee400ce10 con 0x7f6efc143740 2024-03-04T09:25:31.535 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-04T09:25:31.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.541+0000 7f6f01a00640 1 -- 172.21.15.6:0/2235307499 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f6ed806b4d0 msgr2=0x7f6ed806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:31.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.541+0000 7f6f01a00640 1 --2- 172.21.15.6:0/2235307499 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f6ed806b4d0 0x7f6ed806d990 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f6eec001eb0 tx=0x7f6eec009040 comp rx=0 tx=0).stop 2024-03-04T09:25:31.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.541+0000 7f6f01a00640 1 -- 172.21.15.6:0/2235307499 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6efc143740 msgr2=0x7f6efc1f5ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:31.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.541+0000 7f6f01a00640 1 --2- 172.21.15.6:0/2235307499 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6efc143740 0x7f6efc1f5ea0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f6ee400e550 tx=0x7f6ee400ea10 comp rx=0 tx=0).stop 2024-03-04T09:25:31.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.541+0000 7f6f01a00640 1 -- 172.21.15.6:0/2235307499 shutdown_connections 2024-03-04T09:25:31.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.541+0000 7f6f01a00640 1 --2- 172.21.15.6:0/2235307499 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6efc152420 0x7f6efc140bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:31.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.541+0000 7f6f01a00640 1 --2- 172.21.15.6:0/2235307499 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6efc143740 0x7f6efc1f5ea0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:31.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.541+0000 7f6f01a00640 1 --2- 172.21.15.6:0/2235307499 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f6ed806b4d0 0x7f6ed806d990 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:31.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.541+0000 7f6f01a00640 1 --2- 172.21.15.6:0/2235307499 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6efc152040 0x7f6efc141850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:31.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.541+0000 7f6f01a00640 1 -- 172.21.15.6:0/2235307499 >> 172.21.15.6:0/2235307499 conn(0x7f6efc06b3d0 msgr2=0x7f6efc156f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:31.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.541+0000 7f6f01a00640 1 -- 172.21.15.6:0/2235307499 shutdown_connections 2024-03-04T09:25:31.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.541+0000 7f6f01a00640 1 -- 172.21.15.6:0/2235307499 wait complete. 2024-03-04T09:25:31.558 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-04T09:25:31.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.778+0000 7f5470400640 1 Processor -- start 2024-03-04T09:25:31.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.779+0000 7f5470400640 1 -- start start 2024-03-04T09:25:31.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.780+0000 7f5470400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f5468072520 0x7f54680cdfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:31.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.780+0000 7f5470400640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5468072df0 con 0x7f5468071e20 2024-03-04T09:25:31.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.780+0000 7f5470400640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54680cdab0 con 0x7f546815bf20 2024-03-04T09:25:31.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.780+0000 7f5470400640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54680b4d70 con 0x7f5468072520 2024-03-04T09:25:31.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.781+0000 7f546e400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f5468071e20 0x7f54680ccad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:36824/0 (socket says 172.21.15.6:36824) 2024-03-04T09:25:31.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.781+0000 7f546e400640 1 -- 172.21.15.6:0/25231071 learned_addr learned my addr 172.21.15.6:0/25231071 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:25:31.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.781+0000 7f546d000640 1 --2- 172.21.15.6:0/25231071 >> v2:172.21.15.146:3300/0 conn(0x7f5468072520 0x7f54680cdfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:31.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.781+0000 7f5467e00640 1 -- 172.21.15.6:0/25231071 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3064435475 0 0) 0x7f5468072df0 con 0x7f5468071e20 2024-03-04T09:25:31.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.781+0000 7f5467e00640 1 -- 172.21.15.6:0/25231071 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5450003520 con 0x7f5468071e20 2024-03-04T09:25:31.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.781+0000 7f5467e00640 1 -- 172.21.15.6:0/25231071 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 243307250 0 0) 0x7f5450003520 con 0x7f5468071e20 2024-03-04T09:25:31.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.781+0000 7f5467e00640 1 -- 172.21.15.6:0/25231071 >> v2:172.21.15.146:3300/0 conn(0x7f5468072520 msgr2=0x7f54680cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:31.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.782+0000 7f5467e00640 1 --2- 172.21.15.6:0/25231071 >> v2:172.21.15.146:3300/0 conn(0x7f5468072520 0x7f54680cdfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:31.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.782+0000 7f5467e00640 1 -- 172.21.15.6:0/25231071 >> v1:172.21.15.146:6789/0 conn(0x7f546815bf20 legacy=0x7f54680acd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:31.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.782+0000 7f5467e00640 1 -- 172.21.15.6:0/25231071 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54680b5050 con 0x7f5468071e20 2024-03-04T09:25:31.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.782+0000 7f5467e00640 1 -- 172.21.15.6:0/25231071 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7f5460001670 con 0x7f5468071e20 2024-03-04T09:25:31.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.782+0000 7f5467e00640 1 -- 172.21.15.6:0/25231071 >> v1:172.21.15.6:6789/0 conn(0x7f5468071e20 legacy=0x7f54680ccad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:31.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.782+0000 7f5467e00640 1 --2- 172.21.15.6:0/25231071 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54500039b0 0x7f5450003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:31.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.782+0000 7f5467e00640 1 --2- 172.21.15.6:0/25231071 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5450004800 0x7f5450004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:31.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.782+0000 7f5467e00640 1 --2- 172.21.15.6:0/25231071 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f54500051c0 0x7f545000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:31.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.782+0000 7f5467e00640 1 -- 172.21.15.6:0/25231071 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54680b4d70 con 0x7f54500039b0 2024-03-04T09:25:31.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.782+0000 7f5467e00640 1 -- 172.21.15.6:0/25231071 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f545000c110 con 0x7f5450004800 2024-03-04T09:25:31.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.782+0000 7f5467e00640 1 -- 172.21.15.6:0/25231071 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f545000c310 con 0x7f54500051c0 2024-03-04T09:25:31.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.782+0000 7f546da00640 1 --2- 172.21.15.6:0/25231071 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54500039b0 0x7f5450003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:31.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.782+0000 7f546d000640 1 --2- 172.21.15.6:0/25231071 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5450004800 0x7f5450004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:31.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.782+0000 7f546e400640 1 --2- 172.21.15.6:0/25231071 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f54500051c0 0x7f545000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:31.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.782+0000 7f546da00640 1 -- 172.21.15.6:0/25231071 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f54500051c0 msgr2=0x7f545000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:31.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.782+0000 7f546da00640 1 --2- 172.21.15.6:0/25231071 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f54500051c0 0x7f545000b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:31.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.782+0000 7f546da00640 1 -- 172.21.15.6:0/25231071 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5450004800 msgr2=0x7f5450004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:31.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.782+0000 7f546da00640 1 --2- 172.21.15.6:0/25231071 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5450004800 0x7f5450004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:31.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.782+0000 7f546da00640 1 -- 172.21.15.6:0/25231071 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5450003520 con 0x7f54500039b0 2024-03-04T09:25:31.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.783+0000 7f546e400640 1 -- 172.21.15.6:0/25231071 reap_dead start 2024-03-04T09:25:31.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.784+0000 7f546da00640 1 --2- 172.21.15.6:0/25231071 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54500039b0 0x7f5450003dd0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f545c001e50 tx=0x7f5450005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=509f0450ad2dca10 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:31.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.784+0000 7f5467e00640 1 -- 172.21.15.6:0/25231071 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f545c001a70 con 0x7f54500039b0 2024-03-04T09:25:31.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.784+0000 7f5467e00640 1 -- 172.21.15.6:0/25231071 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f545c003b70 con 0x7f54500039b0 2024-03-04T09:25:31.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.784+0000 7f5467e00640 1 -- 172.21.15.6:0/25231071 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f545c0050d0 con 0x7f54500039b0 2024-03-04T09:25:31.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.784+0000 7f5470400640 1 -- 172.21.15.6:0/25231071 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54500039b0 msgr2=0x7f5450003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:31.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.784+0000 7f5470400640 1 --2- 172.21.15.6:0/25231071 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54500039b0 0x7f5450003dd0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f545c001e50 tx=0x7f5450005830 comp rx=0 tx=0).stop 2024-03-04T09:25:31.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.784+0000 7f5470400640 1 -- 172.21.15.6:0/25231071 shutdown_connections 2024-03-04T09:25:31.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.785+0000 7f5470400640 1 --2- 172.21.15.6:0/25231071 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54500039b0 0x7f5450003dd0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:31.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.785+0000 7f5470400640 1 -- 172.21.15.6:0/25231071 >> 172.21.15.6:0/25231071 conn(0x7f546806b3d0 msgr2=0x7f546806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:31.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.785+0000 7f5470400640 1 -- 172.21.15.6:0/25231071 shutdown_connections 2024-03-04T09:25:31.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.785+0000 7f5470400640 1 -- 172.21.15.6:0/25231071 wait complete. 2024-03-04T09:25:31.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.786+0000 7f5470400640 1 Processor -- start 2024-03-04T09:25:31.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.786+0000 7f5470400640 1 -- start start 2024-03-04T09:25:31.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.787+0000 7f5470400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f54500039b0 0x7f54681f4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:31.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.787+0000 7f5470400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f546815d240 0x7f54680ae250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:31.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.787+0000 7f5470400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f546815d6a0 0x7f54680ae790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:31.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.787+0000 7f5470400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54680b4b10 con 0x7f546815d6a0 2024-03-04T09:25:31.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.787+0000 7f5470400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54680b4990 con 0x7f546815d240 2024-03-04T09:25:31.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.787+0000 7f5470400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54680acd40 con 0x7f54500039b0 2024-03-04T09:25:31.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.788+0000 7f546e400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f546815d6a0 0x7f54680ae790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:31.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.788+0000 7f546e400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f546815d6a0 0x7f54680ae790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45750/0 (socket says 172.21.15.6:45750) 2024-03-04T09:25:31.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.788+0000 7f546e400640 1 -- 172.21.15.6:0/113267561 learned_addr learned my addr 172.21.15.6:0/113267561 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:25:31.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.788+0000 7f546da00640 1 --2- 172.21.15.6:0/113267561 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f54500039b0 0x7f54681f4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:31.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.788+0000 7f546d000640 1 --2- 172.21.15.6:0/113267561 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f546815d240 0x7f54680ae250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:31.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.788+0000 7f546e400640 1 -- 172.21.15.6:0/113267561 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f54500039b0 msgr2=0x7f54681f4eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:31.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.788+0000 7f546e400640 1 --2- 172.21.15.6:0/113267561 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f54500039b0 0x7f54681f4eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:31.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.788+0000 7f546e400640 1 -- 172.21.15.6:0/113267561 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f546815d240 msgr2=0x7f54680ae250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:31.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.788+0000 7f546e400640 1 --2- 172.21.15.6:0/113267561 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f546815d240 0x7f54680ae250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:31.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.788+0000 7f546e400640 1 -- 172.21.15.6:0/113267561 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f546815c480 con 0x7f546815d6a0 2024-03-04T09:25:31.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.788+0000 7f546e400640 1 --2- 172.21.15.6:0/113267561 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f546815d6a0 0x7f54680ae790 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f546806c030 tx=0x7f546000d950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:31.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.789+0000 7f5465600640 1 -- 172.21.15.6:0/113267561 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5460007440 con 0x7f546815d6a0 2024-03-04T09:25:31.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.789+0000 7f5465600640 1 -- 172.21.15.6:0/113267561 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5460007600 con 0x7f546815d6a0 2024-03-04T09:25:31.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.789+0000 7f5465600640 1 -- 172.21.15.6:0/113267561 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5460004a20 con 0x7f546815d6a0 2024-03-04T09:25:31.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.789+0000 7f5470400640 1 -- 172.21.15.6:0/113267561 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f546815c760 con 0x7f546815d6a0 2024-03-04T09:25:31.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.789+0000 7f5470400640 1 -- 172.21.15.6:0/113267561 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f546815cd20 con 0x7f546815d6a0 2024-03-04T09:25:31.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.790+0000 7f5470400640 1 -- 172.21.15.6:0/113267561 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f546804f330 con 0x7f546815d6a0 2024-03-04T09:25:31.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.791+0000 7f5465600640 1 -- 172.21.15.6:0/113267561 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5460004bc0 con 0x7f546815d6a0 2024-03-04T09:25:31.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.792+0000 7f5465600640 1 --2- 172.21.15.6:0/113267561 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f544806b500 0x7f544806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:31.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.793+0000 7f5465600640 1 -- 172.21.15.6:0/113267561 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5460089840 con 0x7f546815d6a0 2024-03-04T09:25:31.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.793+0000 7f546da00640 1 --2- 172.21.15.6:0/113267561 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f544806b500 0x7f544806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:31.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.794+0000 7f546da00640 1 --2- 172.21.15.6:0/113267561 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f544806b500 0x7f544806d9c0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f545c034000 tx=0x7f545c009030 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:31.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.797+0000 7f5465600640 1 -- 172.21.15.6:0/113267561 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f54600586c0 con 0x7f546815d6a0 2024-03-04T09:25:31.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.998+0000 7f5470400640 1 -- 172.21.15.6:0/113267561 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f546815cef0 con 0x7f546815d6a0 2024-03-04T09:25:31.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:31.999+0000 7f5465600640 1 -- 172.21.15.6:0/113267561 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7f5460057fc0 con 0x7f546815d6a0 2024-03-04T09:25:31.996 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-03-04T09:25:31.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.001+0000 7f5470400640 1 -- 172.21.15.6:0/113267561 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f544806b500 msgr2=0x7f544806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:31.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.001+0000 7f5470400640 1 --2- 172.21.15.6:0/113267561 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f544806b500 0x7f544806d9c0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f545c034000 tx=0x7f545c009030 comp rx=0 tx=0).stop 2024-03-04T09:25:31.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.001+0000 7f5470400640 1 -- 172.21.15.6:0/113267561 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f546815d6a0 msgr2=0x7f54680ae790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:31.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.001+0000 7f5470400640 1 --2- 172.21.15.6:0/113267561 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f546815d6a0 0x7f54680ae790 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f546806c030 tx=0x7f546000d950 comp rx=0 tx=0).stop 2024-03-04T09:25:31.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.002+0000 7f5470400640 1 -- 172.21.15.6:0/113267561 shutdown_connections 2024-03-04T09:25:31.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.002+0000 7f5470400640 1 --2- 172.21.15.6:0/113267561 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f546815d6a0 0x7f54680ae790 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:31.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.002+0000 7f5470400640 1 --2- 172.21.15.6:0/113267561 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f546815d240 0x7f54680ae250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:31.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.002+0000 7f5470400640 1 --2- 172.21.15.6:0/113267561 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f544806b500 0x7f544806d9c0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:31.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.002+0000 7f5470400640 1 --2- 172.21.15.6:0/113267561 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f54500039b0 0x7f54681f4eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:31.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.002+0000 7f5470400640 1 -- 172.21.15.6:0/113267561 >> 172.21.15.6:0/113267561 conn(0x7f546806b3d0 msgr2=0x7f546809ca20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:31.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.002+0000 7f5470400640 1 -- 172.21.15.6:0/113267561 shutdown_connections 2024-03-04T09:25:31.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.002+0000 7f5470400640 1 -- 172.21.15.6:0/113267561 wait complete. 2024-03-04T09:25:32.018 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-04T09:25:32.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.249+0000 7f237be00640 1 Processor -- start 2024-03-04T09:25:32.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.249+0000 7f237be00640 1 -- start start 2024-03-04T09:25:32.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.251+0000 7f237be00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f237c145430 0x7f237c15f810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:32.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.251+0000 7f237be00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f237c15cc90 0x7f237c0b1b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:32.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.251+0000 7f237be00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f237c0b3fe0 con 0x7f237c0b2080 2024-03-04T09:25:32.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.251+0000 7f237be00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f237c0b24d0 con 0x7f237c145430 2024-03-04T09:25:32.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.251+0000 7f237be00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f237c15fe40 con 0x7f237c15cc90 2024-03-04T09:25:32.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.251+0000 7f237aa00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f237c145430 0x7f237c15f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:32.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.251+0000 7f237aa00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f237c145430 0x7f237c15f810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45766/0 (socket says 172.21.15.6:45766) 2024-03-04T09:25:32.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.251+0000 7f237aa00640 1 -- 172.21.15.6:0/3273674410 learned_addr learned my addr 172.21.15.6:0/3273674410 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:25:32.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.252+0000 7f237a000640 1 --2- 172.21.15.6:0/3273674410 >> v2:172.21.15.146:3300/0 conn(0x7f237c15cc90 0x7f237c0b1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:32.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.252+0000 7f237aa00640 1 -- 172.21.15.6:0/3273674410 >> v2:172.21.15.146:3300/0 conn(0x7f237c15cc90 msgr2=0x7f237c0b1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:32.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.252+0000 7f237aa00640 1 --2- 172.21.15.6:0/3273674410 >> v2:172.21.15.146:3300/0 conn(0x7f237c15cc90 0x7f237c0b1b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:32.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.252+0000 7f237aa00640 1 -- 172.21.15.6:0/3273674410 >> v1:172.21.15.146:6789/0 conn(0x7f237c0b2080 legacy=0x7f237c06c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:32.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.252+0000 7f237aa00640 1 -- 172.21.15.6:0/3273674410 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f237c160120 con 0x7f237c145430 2024-03-04T09:25:32.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.253+0000 7f237aa00640 1 --2- 172.21.15.6:0/3273674410 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f237c145430 0x7f237c15f810 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f237c06d8c0 tx=0x7f236c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3398b91c90dfcff6 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:32.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.253+0000 7f2379600640 1 -- 172.21.15.6:0/3273674410 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f236c035020 con 0x7f237c145430 2024-03-04T09:25:32.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.253+0000 7f2379600640 1 -- 172.21.15.6:0/3273674410 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f236c031870 con 0x7f237c145430 2024-03-04T09:25:32.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.253+0000 7f2379600640 1 -- 172.21.15.6:0/3273674410 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f236c031c40 con 0x7f237c145430 2024-03-04T09:25:32.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.254+0000 7f237be00640 1 -- 172.21.15.6:0/3273674410 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f237c145430 msgr2=0x7f237c15f810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:32.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.254+0000 7f237be00640 1 --2- 172.21.15.6:0/3273674410 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f237c145430 0x7f237c15f810 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f237c06d8c0 tx=0x7f236c02b910 comp rx=0 tx=0).stop 2024-03-04T09:25:32.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.254+0000 7f237be00640 1 -- 172.21.15.6:0/3273674410 shutdown_connections 2024-03-04T09:25:32.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.254+0000 7f237be00640 1 --2- 172.21.15.6:0/3273674410 >> v2:172.21.15.146:3300/0 conn(0x7f237c15cc90 0x7f237c0b1b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:32.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.254+0000 7f237be00640 1 --2- 172.21.15.6:0/3273674410 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f237c145430 0x7f237c15f810 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:32.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.254+0000 7f237be00640 1 -- 172.21.15.6:0/3273674410 >> 172.21.15.6:0/3273674410 conn(0x7f237c06b3a0 msgr2=0x7f237c06b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:32.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.254+0000 7f237be00640 1 -- 172.21.15.6:0/3273674410 shutdown_connections 2024-03-04T09:25:32.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.254+0000 7f237be00640 1 -- 172.21.15.6:0/3273674410 wait complete. 2024-03-04T09:25:32.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.256+0000 7f237be00640 1 Processor -- start 2024-03-04T09:25:32.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.256+0000 7f237be00640 1 -- start start 2024-03-04T09:25:32.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.257+0000 7f237be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f237c0b2080 0x7f237c15b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:32.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.257+0000 7f237be00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f237c145430 0x7f237c071cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:32.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.257+0000 7f237be00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f237c15cc90 0x7f237c13ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:32.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.257+0000 7f237be00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f237c15d0d0 con 0x7f237c0b2080 2024-03-04T09:25:32.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.257+0000 7f237be00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f237c1606e0 con 0x7f237c145430 2024-03-04T09:25:32.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.257+0000 7f237be00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f237c145810 con 0x7f237c15cc90 2024-03-04T09:25:32.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.257+0000 7f237aa00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f237c0b2080 0x7f237c15b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:32.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.257+0000 7f237aa00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f237c0b2080 0x7f237c15b970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45778/0 (socket says 172.21.15.6:45778) 2024-03-04T09:25:32.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.257+0000 7f237aa00640 1 -- 172.21.15.6:0/2186195836 learned_addr learned my addr 172.21.15.6:0/2186195836 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:25:32.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.258+0000 7f237a000640 1 --2- 172.21.15.6:0/2186195836 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f237c145430 0x7f237c071cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:32.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.258+0000 7f237b400640 1 --2- 172.21.15.6:0/2186195836 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f237c15cc90 0x7f237c13ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:32.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.258+0000 7f237aa00640 1 -- 172.21.15.6:0/2186195836 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f237c15cc90 msgr2=0x7f237c13ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:32.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.258+0000 7f237aa00640 1 --2- 172.21.15.6:0/2186195836 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f237c15cc90 0x7f237c13ca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:32.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.258+0000 7f237aa00640 1 -- 172.21.15.6:0/2186195836 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f237c145430 msgr2=0x7f237c071cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:32.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.258+0000 7f237aa00640 1 --2- 172.21.15.6:0/2186195836 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f237c145430 0x7f237c071cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:32.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.258+0000 7f237aa00640 1 -- 172.21.15.6:0/2186195836 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f237c136b80 con 0x7f237c0b2080 2024-03-04T09:25:32.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.258+0000 7f237a000640 1 --2- 172.21.15.6:0/2186195836 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f237c145430 0x7f237c071cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:25:32.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.258+0000 7f237aa00640 1 --2- 172.21.15.6:0/2186195836 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f237c0b2080 0x7f237c15b970 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f236c000c00 tx=0x7f236c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:32.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.258+0000 7f2373400640 1 -- 172.21.15.6:0/2186195836 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f236c040070 con 0x7f237c0b2080 2024-03-04T09:25:32.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.258+0000 7f2373400640 1 -- 172.21.15.6:0/2186195836 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f236c030d00 con 0x7f237c0b2080 2024-03-04T09:25:32.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.258+0000 7f2373400640 1 -- 172.21.15.6:0/2186195836 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f236c035040 con 0x7f237c0b2080 2024-03-04T09:25:32.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.258+0000 7f237be00640 1 -- 172.21.15.6:0/2186195836 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f237c0716f0 con 0x7f237c0b2080 2024-03-04T09:25:32.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.259+0000 7f237be00640 1 -- 172.21.15.6:0/2186195836 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f237c136e40 con 0x7f237c0b2080 2024-03-04T09:25:32.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.260+0000 7f237be00640 1 -- 172.21.15.6:0/2186195836 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f237c04f330 con 0x7f237c0b2080 2024-03-04T09:25:32.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.261+0000 7f2373400640 1 -- 172.21.15.6:0/2186195836 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f236c04b060 con 0x7f237c0b2080 2024-03-04T09:25:32.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.262+0000 7f2373400640 1 --2- 172.21.15.6:0/2186195836 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f235c06b4d0 0x7f235c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:32.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.262+0000 7f2373400640 1 -- 172.21.15.6:0/2186195836 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f236c0abf30 con 0x7f237c0b2080 2024-03-04T09:25:32.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.263+0000 7f237a000640 1 --2- 172.21.15.6:0/2186195836 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f235c06b4d0 0x7f235c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:32.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.264+0000 7f237a000640 1 --2- 172.21.15.6:0/2186195836 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f235c06b4d0 0x7f235c06d990 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f2368006fd0 tx=0x7f2368008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:32.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.267+0000 7f2373400640 1 -- 172.21.15.6:0/2186195836 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f236c07ae40 con 0x7f237c0b2080 2024-03-04T09:25:32.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.465+0000 7f237be00640 1 -- 172.21.15.6:0/2186195836 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f237c06e240 con 0x7f237c0b2080 2024-03-04T09:25:32.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.466+0000 7f2373400640 1 -- 172.21.15.6:0/2186195836 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f236c07a740 con 0x7f237c0b2080 2024-03-04T09:25:32.463 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 64 2024-03-04T09:25:32.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.469+0000 7f237be00640 1 -- 172.21.15.6:0/2186195836 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f235c06b4d0 msgr2=0x7f235c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:32.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.469+0000 7f237be00640 1 --2- 172.21.15.6:0/2186195836 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f235c06b4d0 0x7f235c06d990 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f2368006fd0 tx=0x7f2368008040 comp rx=0 tx=0).stop 2024-03-04T09:25:32.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.469+0000 7f237be00640 1 -- 172.21.15.6:0/2186195836 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f237c0b2080 msgr2=0x7f237c15b970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:32.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.469+0000 7f237be00640 1 --2- 172.21.15.6:0/2186195836 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f237c0b2080 0x7f237c15b970 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f236c000c00 tx=0x7f236c0046c0 comp rx=0 tx=0).stop 2024-03-04T09:25:32.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.469+0000 7f237be00640 1 -- 172.21.15.6:0/2186195836 shutdown_connections 2024-03-04T09:25:32.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.469+0000 7f237be00640 1 --2- 172.21.15.6:0/2186195836 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f235c06b4d0 0x7f235c06d990 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:32.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.469+0000 7f237be00640 1 --2- 172.21.15.6:0/2186195836 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f237c15cc90 0x7f237c13ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:32.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.469+0000 7f237be00640 1 --2- 172.21.15.6:0/2186195836 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f237c145430 0x7f237c071cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:32.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.469+0000 7f237be00640 1 --2- 172.21.15.6:0/2186195836 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f237c0b2080 0x7f237c15b970 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:32.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.469+0000 7f237be00640 1 -- 172.21.15.6:0/2186195836 >> 172.21.15.6:0/2186195836 conn(0x7f237c06b3a0 msgr2=0x7f237c13fad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:32.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.469+0000 7f237be00640 1 -- 172.21.15.6:0/2186195836 shutdown_connections 2024-03-04T09:25:32.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.469+0000 7f237be00640 1 -- 172.21.15.6:0/2186195836 wait complete. 2024-03-04T09:25:32.486 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:25:32.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.707+0000 7faad1a00640 1 Processor -- start 2024-03-04T09:25:32.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.707+0000 7faad1a00640 1 -- start start 2024-03-04T09:25:32.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.709+0000 7faad1a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faacc0af1a0 0x7faacc15c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:32.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.709+0000 7faad1a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7faacc0af580 0x7faacc0b4c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:32.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.709+0000 7faad1a00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7faacc0cdc20 0x7faacc0b2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:32.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.709+0000 7faad1a00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faacc15cc40 con 0x7faacc0af580 2024-03-04T09:25:32.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.709+0000 7faad1a00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faacc15cf20 con 0x7faacc0cdc20 2024-03-04T09:25:32.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.709+0000 7faad1a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faacc15d200 con 0x7faacc0af1a0 2024-03-04T09:25:32.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.710+0000 7faaca000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7faacc0af580 0x7faacc0b4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:32.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.710+0000 7faaca000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7faacc0af580 0x7faacc0b4c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45784/0 (socket says 172.21.15.6:45784) 2024-03-04T09:25:32.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.710+0000 7faaca000640 1 -- 172.21.15.6:0/4005919146 learned_addr learned my addr 172.21.15.6:0/4005919146 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:25:32.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.710+0000 7faacb400640 1 --2- 172.21.15.6:0/4005919146 >> v2:172.21.15.146:3300/0 conn(0x7faacc0cdc20 0x7faacc0b2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:32.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.710+0000 7faacaa00640 1 --2- 172.21.15.6:0/4005919146 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faacc0af1a0 0x7faacc15c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:32.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.710+0000 7faaca000640 1 -- 172.21.15.6:0/4005919146 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faacc0af1a0 msgr2=0x7faacc15c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:32.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.710+0000 7faaca000640 1 --2- 172.21.15.6:0/4005919146 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faacc0af1a0 0x7faacc15c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:32.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.710+0000 7faaca000640 1 -- 172.21.15.6:0/4005919146 >> v2:172.21.15.146:3300/0 conn(0x7faacc0cdc20 msgr2=0x7faacc0b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:32.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.710+0000 7faaca000640 1 --2- 172.21.15.6:0/4005919146 >> v2:172.21.15.146:3300/0 conn(0x7faacc0cdc20 0x7faacc0b2dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:32.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.710+0000 7faaca000640 1 -- 172.21.15.6:0/4005919146 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faacc15d4e0 con 0x7faacc0af580 2024-03-04T09:25:32.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.710+0000 7faacb400640 1 --2- 172.21.15.6:0/4005919146 >> v2:172.21.15.146:3300/0 conn(0x7faacc0cdc20 0x7faacc0b2dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T09:25:32.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.711+0000 7faaca000640 1 --2- 172.21.15.6:0/4005919146 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faacc0af580 0x7faacc0b4c20 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7faacc0cda30 tx=0x7faab802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f660772a9501d3b server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:32.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.712+0000 7faac9600640 1 -- 172.21.15.6:0/4005919146 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faab802e070 con 0x7faacc0af580 2024-03-04T09:25:32.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.712+0000 7faac9600640 1 -- 172.21.15.6:0/4005919146 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faab8037070 con 0x7faacc0af580 2024-03-04T09:25:32.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.712+0000 7faac9600640 1 -- 172.21.15.6:0/4005919146 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faab8031a00 con 0x7faacc0af580 2024-03-04T09:25:32.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.713+0000 7faad1a00640 1 -- 172.21.15.6:0/4005919146 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faacc0af580 msgr2=0x7faacc0b4c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:32.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.713+0000 7faad1a00640 1 --2- 172.21.15.6:0/4005919146 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faacc0af580 0x7faacc0b4c20 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7faacc0cda30 tx=0x7faab802b880 comp rx=0 tx=0).stop 2024-03-04T09:25:32.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.713+0000 7faad1a00640 1 -- 172.21.15.6:0/4005919146 shutdown_connections 2024-03-04T09:25:32.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.713+0000 7faad1a00640 1 --2- 172.21.15.6:0/4005919146 >> v2:172.21.15.146:3300/0 conn(0x7faacc0cdc20 0x7faacc0b2dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:32.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.713+0000 7faad1a00640 1 --2- 172.21.15.6:0/4005919146 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faacc0af580 0x7faacc0b4c20 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:32.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.713+0000 7faad1a00640 1 --2- 172.21.15.6:0/4005919146 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faacc0af1a0 0x7faacc15c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:32.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.713+0000 7faad1a00640 1 -- 172.21.15.6:0/4005919146 >> 172.21.15.6:0/4005919146 conn(0x7faacc06afb0 msgr2=0x7faacc06b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:32.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.713+0000 7faad1a00640 1 -- 172.21.15.6:0/4005919146 shutdown_connections 2024-03-04T09:25:32.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.713+0000 7faad1a00640 1 -- 172.21.15.6:0/4005919146 wait complete. 2024-03-04T09:25:32.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.715+0000 7faad1a00640 1 Processor -- start 2024-03-04T09:25:32.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.715+0000 7faad1a00640 1 -- start start 2024-03-04T09:25:32.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.716+0000 7faad1a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faacc0af1a0 0x7faacc154140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:32.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.716+0000 7faad1a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faacc0af580 0x7faacc15a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:32.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.716+0000 7faad1a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faacc0cdc20 0x7faacc0b09f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:32.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.716+0000 7faad1a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faacc0b3490 con 0x7faacc0af1a0 2024-03-04T09:25:32.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.716+0000 7faad1a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faacc0b3310 con 0x7faacc0cdc20 2024-03-04T09:25:32.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.716+0000 7faad1a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faacc0b3610 con 0x7faacc0af580 2024-03-04T09:25:32.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.716+0000 7faacaa00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faacc0af1a0 0x7faacc154140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:32.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.716+0000 7faacaa00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faacc0af1a0 0x7faacc154140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45786/0 (socket says 172.21.15.6:45786) 2024-03-04T09:25:32.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.716+0000 7faacaa00640 1 -- 172.21.15.6:0/4248068548 learned_addr learned my addr 172.21.15.6:0/4248068548 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:25:32.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.717+0000 7faaca000640 1 --2- 172.21.15.6:0/4248068548 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faacc0af580 0x7faacc15a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:32.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.717+0000 7faacb400640 1 --2- 172.21.15.6:0/4248068548 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faacc0cdc20 0x7faacc0b09f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:32.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.717+0000 7faacaa00640 1 -- 172.21.15.6:0/4248068548 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faacc0af580 msgr2=0x7faacc15a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:32.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.717+0000 7faacaa00640 1 --2- 172.21.15.6:0/4248068548 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faacc0af580 0x7faacc15a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:32.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.717+0000 7faacaa00640 1 -- 172.21.15.6:0/4248068548 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faacc0cdc20 msgr2=0x7faacc0b09f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:32.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.717+0000 7faacaa00640 1 --2- 172.21.15.6:0/4248068548 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faacc0cdc20 0x7faacc0b09f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:32.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.717+0000 7faacaa00640 1 -- 172.21.15.6:0/4248068548 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faacc15af00 con 0x7faacc0af1a0 2024-03-04T09:25:32.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.717+0000 7faacb400640 1 --2- 172.21.15.6:0/4248068548 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faacc0cdc20 0x7faacc0b09f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:25:32.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.717+0000 7faacaa00640 1 --2- 172.21.15.6:0/4248068548 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faacc0af1a0 0x7faacc154140 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7faabc00d5f0 tx=0x7faabc00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:32.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.717+0000 7faac2a00640 1 -- 172.21.15.6:0/4248068548 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faabc018020 con 0x7faacc0af1a0 2024-03-04T09:25:32.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.718+0000 7faac2a00640 1 -- 172.21.15.6:0/4248068548 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faabc014620 con 0x7faacc0af1a0 2024-03-04T09:25:32.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.718+0000 7faac2a00640 1 -- 172.21.15.6:0/4248068548 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faabc014a10 con 0x7faacc0af1a0 2024-03-04T09:25:32.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.718+0000 7faad1a00640 1 -- 172.21.15.6:0/4248068548 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faacc09ac60 con 0x7faacc0af1a0 2024-03-04T09:25:32.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.718+0000 7faad1a00640 1 -- 172.21.15.6:0/4248068548 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faacc09b220 con 0x7faacc0af1a0 2024-03-04T09:25:32.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.719+0000 7faad1a00640 1 -- 172.21.15.6:0/4248068548 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faacc069510 con 0x7faacc0af1a0 2024-03-04T09:25:32.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.720+0000 7faac2a00640 1 -- 172.21.15.6:0/4248068548 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7faabc01b020 con 0x7faacc0af1a0 2024-03-04T09:25:32.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.721+0000 7faac2a00640 1 --2- 172.21.15.6:0/4248068548 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7faaac06b4d0 0x7faaac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:32.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.721+0000 7faac2a00640 1 -- 172.21.15.6:0/4248068548 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7faabc08a300 con 0x7faacc0af1a0 2024-03-04T09:25:32.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.721+0000 7faaca000640 1 --2- 172.21.15.6:0/4248068548 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7faaac06b4d0 0x7faaac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:32.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.722+0000 7faaca000640 1 --2- 172.21.15.6:0/4248068548 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7faaac06b4d0 0x7faaac06d990 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7faab8036000 tx=0x7faab803d040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:32.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.725+0000 7faac2a00640 1 -- 172.21.15.6:0/4248068548 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7faabc022030 con 0x7faacc0af1a0 2024-03-04T09:25:32.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.964+0000 7faad1a00640 1 -- 172.21.15.6:0/4248068548 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faacc0cd590 con 0x7faacc0af1a0 2024-03-04T09:25:32.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.965+0000 7faac2a00640 1 -- 172.21.15.6:0/4248068548 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4971 (secure 0 0 0) 0x7faabc059210 con 0x7faacc0af1a0 2024-03-04T09:25:32.966 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:25:32.966 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4248,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/2702058134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2702058134},{"type":"v1","addr":"172.21.15.6:6833","nonce":2702058134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4263,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/3866998852","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":3866998852},{"type":"v1","addr":"172.21.15.6:6835","nonce":3866998852}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4285,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/3523455455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":3523455455},{"type":"v1","addr":"172.21.15.146:6837","nonce":3523455455}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:25:18.912808+0000","modified":"2024-03-04T09:25:22.016493+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4288},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4288":{"gid":4288,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.146:6836/1946537763","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":1946537763},{"type":"v1","addr":"172.21.15.146:6836","nonce":1946537763}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4288,"qdb_cluster":[4288]},"id":1}]} 2024-03-04T09:25:32.967 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 7 2024-03-04T09:25:32.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.968+0000 7faad1a00640 1 -- 172.21.15.6:0/4248068548 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7faaac06b4d0 msgr2=0x7faaac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:32.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.968+0000 7faad1a00640 1 --2- 172.21.15.6:0/4248068548 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7faaac06b4d0 0x7faaac06d990 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7faab8036000 tx=0x7faab803d040 comp rx=0 tx=0).stop 2024-03-04T09:25:32.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.968+0000 7faad1a00640 1 -- 172.21.15.6:0/4248068548 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faacc0af1a0 msgr2=0x7faacc154140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:32.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.968+0000 7faad1a00640 1 --2- 172.21.15.6:0/4248068548 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faacc0af1a0 0x7faacc154140 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7faabc00d5f0 tx=0x7faabc00dab0 comp rx=0 tx=0).stop 2024-03-04T09:25:32.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.969+0000 7faad1a00640 1 -- 172.21.15.6:0/4248068548 shutdown_connections 2024-03-04T09:25:32.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.969+0000 7faad1a00640 1 --2- 172.21.15.6:0/4248068548 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faacc0cdc20 0x7faacc0b09f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:32.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.969+0000 7faad1a00640 1 --2- 172.21.15.6:0/4248068548 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7faaac06b4d0 0x7faaac06d990 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:32.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.969+0000 7faad1a00640 1 --2- 172.21.15.6:0/4248068548 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faacc0af580 0x7faacc15a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:32.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.969+0000 7faad1a00640 1 --2- 172.21.15.6:0/4248068548 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faacc0af1a0 0x7faacc154140 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:32.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.969+0000 7faad1a00640 1 -- 172.21.15.6:0/4248068548 >> 172.21.15.6:0/4248068548 conn(0x7faacc06afb0 msgr2=0x7faacc06b6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:32.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.969+0000 7faad1a00640 1 -- 172.21.15.6:0/4248068548 shutdown_connections 2024-03-04T09:25:32.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:32.969+0000 7faad1a00640 1 -- 172.21.15.6:0/4248068548 wait complete. 2024-03-04T09:25:32.985 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T09:25:33.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.209+0000 7f1ea1200640 1 Processor -- start 2024-03-04T09:25:33.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.209+0000 7f1ea1200640 1 -- start start 2024-03-04T09:25:33.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.211+0000 7f1ea1200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1e9c0cd060 0x7f1e9c1667e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:33.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.211+0000 7f1ea1200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e9c0714d0 con 0x7f1e9c145300 2024-03-04T09:25:33.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.211+0000 7f1ea1200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e9c0717b0 con 0x7f1e9c070f10 2024-03-04T09:25:33.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.211+0000 7f1ea1200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e9c071a90 con 0x7f1e9c0cd060 2024-03-04T09:25:33.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.212+0000 7f1e9b400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f1e9c145300 0x7f1e9c0b49a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:36834/0 (socket says 172.21.15.6:36834) 2024-03-04T09:25:33.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.212+0000 7f1e9b400640 1 -- 172.21.15.6:0/1092155721 learned_addr learned my addr 172.21.15.6:0/1092155721 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:25:33.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.212+0000 7f1e9be00640 1 --2- 172.21.15.6:0/1092155721 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1e9c0cd060 0x7f1e9c1667e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:33.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.212+0000 7f1e9aa00640 1 -- 172.21.15.6:0/1092155721 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 577985108 0 0) 0x7f1e9c0714d0 con 0x7f1e9c145300 2024-03-04T09:25:33.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.212+0000 7f1e9aa00640 1 -- 172.21.15.6:0/1092155721 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e80003520 con 0x7f1e9c145300 2024-03-04T09:25:33.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.212+0000 7f1e9aa00640 1 -- 172.21.15.6:0/1092155721 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 392815012 0 0) 0x7f1e80003520 con 0x7f1e9c145300 2024-03-04T09:25:33.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.212+0000 7f1e9aa00640 1 -- 172.21.15.6:0/1092155721 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1e9c0cd060 msgr2=0x7f1e9c1667e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:33.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.212+0000 7f1e9aa00640 1 --2- 172.21.15.6:0/1092155721 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1e9c0cd060 0x7f1e9c1667e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:33.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.212+0000 7f1e9aa00640 1 -- 172.21.15.6:0/1092155721 >> v1:172.21.15.146:6789/0 conn(0x7f1e9c070f10 legacy=0x7f1e9c0cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:33.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.213+0000 7f1e9aa00640 1 -- 172.21.15.6:0/1092155721 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e9c071d70 con 0x7f1e9c145300 2024-03-04T09:25:33.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.213+0000 7f1e9aa00640 1 -- 172.21.15.6:0/1092155721 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7f1e8c001670 con 0x7f1e9c145300 2024-03-04T09:25:33.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.213+0000 7f1e9aa00640 1 -- 172.21.15.6:0/1092155721 >> v1:172.21.15.6:6789/0 conn(0x7f1e9c145300 legacy=0x7f1e9c0b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:33.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.213+0000 7f1e9aa00640 1 --2- 172.21.15.6:0/1092155721 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1e800039e0 0x7f1e80003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:33.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.213+0000 7f1e9aa00640 1 --2- 172.21.15.6:0/1092155721 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1e9c0cd060 0x7f1e80004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:33.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.213+0000 7f1e9aa00640 1 --2- 172.21.15.6:0/1092155721 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1e80004d40 0x7f1e800051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:33.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.213+0000 7f1e9aa00640 1 -- 172.21.15.6:0/1092155721 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e9c071a90 con 0x7f1e800039e0 2024-03-04T09:25:33.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.213+0000 7f1e9aa00640 1 -- 172.21.15.6:0/1092155721 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e8000baa0 con 0x7f1e80004d40 2024-03-04T09:25:33.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.213+0000 7f1e9aa00640 1 -- 172.21.15.6:0/1092155721 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e8000bca0 con 0x7f1e9c0cd060 2024-03-04T09:25:33.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.213+0000 7f1e9b400640 1 --2- 172.21.15.6:0/1092155721 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1e800039e0 0x7f1e80003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:33.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.213+0000 7f1e93e00640 1 --2- 172.21.15.6:0/1092155721 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1e9c0cd060 0x7f1e80004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:33.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.213+0000 7f1e9be00640 1 --2- 172.21.15.6:0/1092155721 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1e80004d40 0x7f1e800051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:33.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.214+0000 7f1e9b400640 1 -- 172.21.15.6:0/1092155721 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1e9c0cd060 msgr2=0x7f1e80004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:33.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.214+0000 7f1e9b400640 1 --2- 172.21.15.6:0/1092155721 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1e9c0cd060 0x7f1e80004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:33.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.214+0000 7f1e9b400640 1 -- 172.21.15.6:0/1092155721 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1e80004d40 msgr2=0x7f1e800051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:33.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.214+0000 7f1e9b400640 1 --2- 172.21.15.6:0/1092155721 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1e80004d40 0x7f1e800051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:33.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.214+0000 7f1e9b400640 1 -- 172.21.15.6:0/1092155721 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e80003520 con 0x7f1e800039e0 2024-03-04T09:25:33.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.215+0000 7f1e9b400640 1 --2- 172.21.15.6:0/1092155721 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1e800039e0 0x7f1e80003dc0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f1e8c00c240 tx=0x7f1e80005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=735380b5110191dc server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:33.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.215+0000 7f1e9aa00640 1 -- 172.21.15.6:0/1092155721 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e8c007d30 con 0x7f1e800039e0 2024-03-04T09:25:33.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.215+0000 7f1e9aa00640 1 -- 172.21.15.6:0/1092155721 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e8c030070 con 0x7f1e800039e0 2024-03-04T09:25:33.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.215+0000 7f1e9aa00640 1 -- 172.21.15.6:0/1092155721 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e8c02cd60 con 0x7f1e800039e0 2024-03-04T09:25:33.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.216+0000 7f1ea1200640 1 -- 172.21.15.6:0/1092155721 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1e800039e0 msgr2=0x7f1e80003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:33.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.216+0000 7f1ea1200640 1 --2- 172.21.15.6:0/1092155721 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1e800039e0 0x7f1e80003dc0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f1e8c00c240 tx=0x7f1e80005830 comp rx=0 tx=0).stop 2024-03-04T09:25:33.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.216+0000 7f1e9be00640 1 -- 172.21.15.6:0/1092155721 reap_dead start 2024-03-04T09:25:33.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.216+0000 7f1ea1200640 1 -- 172.21.15.6:0/1092155721 shutdown_connections 2024-03-04T09:25:33.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.216+0000 7f1ea1200640 1 -- 172.21.15.6:0/1092155721 >> 172.21.15.6:0/1092155721 conn(0x7f1e9c06b3d0 msgr2=0x7f1e9c06b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:33.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.216+0000 7f1ea1200640 1 -- 172.21.15.6:0/1092155721 shutdown_connections 2024-03-04T09:25:33.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.216+0000 7f1ea1200640 1 -- 172.21.15.6:0/1092155721 wait complete. 2024-03-04T09:25:33.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.218+0000 7f1ea1200640 1 Processor -- start 2024-03-04T09:25:33.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.218+0000 7f1ea1200640 1 -- start start 2024-03-04T09:25:33.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.219+0000 7f1ea1200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1e800039e0 0x7f1e9c13ca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:33.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.219+0000 7f1ea1200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1e9c06ddb0 0x7f1e9c06e230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:33.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.219+0000 7f1ea1200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1e9c141770 0x7f1e9c141bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:33.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.219+0000 7f1ea1200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e9c15d700 con 0x7f1e9c06ddb0 2024-03-04T09:25:33.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.219+0000 7f1ea1200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e9c15d580 con 0x7f1e9c141770 2024-03-04T09:25:33.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.219+0000 7f1ea1200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e9c15d880 con 0x7f1e800039e0 2024-03-04T09:25:33.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.219+0000 7f1e93e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1e9c06ddb0 0x7f1e9c06e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:33.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.219+0000 7f1e93e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1e9c06ddb0 0x7f1e9c06e230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45804/0 (socket says 172.21.15.6:45804) 2024-03-04T09:25:33.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.219+0000 7f1e93e00640 1 -- 172.21.15.6:0/4026272660 learned_addr learned my addr 172.21.15.6:0/4026272660 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:25:33.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.220+0000 7f1e9be00640 1 --2- 172.21.15.6:0/4026272660 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1e9c141770 0x7f1e9c141bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:33.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.220+0000 7f1e9b400640 1 --2- 172.21.15.6:0/4026272660 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1e800039e0 0x7f1e9c13ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:33.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.220+0000 7f1e93e00640 1 -- 172.21.15.6:0/4026272660 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1e800039e0 msgr2=0x7f1e9c13ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:33.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.220+0000 7f1e93e00640 1 --2- 172.21.15.6:0/4026272660 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1e800039e0 0x7f1e9c13ca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:33.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.220+0000 7f1e93e00640 1 -- 172.21.15.6:0/4026272660 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1e9c141770 msgr2=0x7f1e9c141bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:33.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.220+0000 7f1e93e00640 1 --2- 172.21.15.6:0/4026272660 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1e9c141770 0x7f1e9c141bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:33.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.220+0000 7f1e93e00640 1 -- 172.21.15.6:0/4026272660 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e9c13cf40 con 0x7f1e9c06ddb0 2024-03-04T09:25:33.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.220+0000 7f1e93e00640 1 --2- 172.21.15.6:0/4026272660 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1e9c06ddb0 0x7f1e9c06e230 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f1e8400e800 tx=0x7f1e8400ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:33.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.221+0000 7f1e93400640 1 -- 172.21.15.6:0/4026272660 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e840050d0 con 0x7f1e9c06ddb0 2024-03-04T09:25:33.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.221+0000 7f1e93400640 1 -- 172.21.15.6:0/4026272660 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e84018070 con 0x7f1e9c06ddb0 2024-03-04T09:25:33.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.221+0000 7f1e93400640 1 -- 172.21.15.6:0/4026272660 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e840054b0 con 0x7f1e9c06ddb0 2024-03-04T09:25:33.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.221+0000 7f1ea1200640 1 -- 172.21.15.6:0/4026272660 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e9c1409b0 con 0x7f1e9c06ddb0 2024-03-04T09:25:33.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.221+0000 7f1ea1200640 1 -- 172.21.15.6:0/4026272660 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e9c140f70 con 0x7f1e9c06ddb0 2024-03-04T09:25:33.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.222+0000 7f1ea1200640 1 -- 172.21.15.6:0/4026272660 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e9c04f330 con 0x7f1e9c06ddb0 2024-03-04T09:25:33.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.223+0000 7f1e93400640 1 -- 172.21.15.6:0/4026272660 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1e8400c6c0 con 0x7f1e9c06ddb0 2024-03-04T09:25:33.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.224+0000 7f1e93400640 1 --2- 172.21.15.6:0/4026272660 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f1e7806b500 0x7f1e7806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:33.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.224+0000 7f1e93400640 1 -- 172.21.15.6:0/4026272660 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1e84089fb0 con 0x7f1e9c06ddb0 2024-03-04T09:25:33.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.225+0000 7f1e9b400640 1 --2- 172.21.15.6:0/4026272660 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f1e7806b500 0x7f1e7806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:33.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.226+0000 7f1e9b400640 1 --2- 172.21.15.6:0/4026272660 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f1e7806b500 0x7f1e7806d9c0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f1e8c034000 tx=0x7f1e8c009030 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:33.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.229+0000 7f1e93400640 1 -- 172.21.15.6:0/4026272660 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1e84058d90 con 0x7f1e9c06ddb0 2024-03-04T09:25:33.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.432+0000 7f1ea1200640 1 -- 172.21.15.6:0/4026272660 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1e9c06ba90 con 0x7f1e9c06ddb0 2024-03-04T09:25:33.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.434+0000 7f1e93400640 1 -- 172.21.15.6:0/4026272660 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16557 (secure 0 0 0) 0x7f1e84058690 con 0x7f1e9c06ddb0 2024-03-04T09:25:33.431 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:25:33.431 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":20,"fsid":"edce0578-5ae4-4402-b45a-552bdef84d7c","created":"2024-03-04T09:24:54.523780+0000","modified":"2024-03-04T09:25:20.446333+0000","last_up_change":"2024-03-04T09:25:07.089049+0000","last_in_change":"2024-03-04T09:24:57.982077+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T09:25:07.935198+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-04T09:25:17.118573+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-04T09:25:17.930536+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a96574f-81f1-4bc6-8463-757520b84a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6825","nonce":1364449401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6827","nonce":1364449401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6831","nonce":1364449401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6829","nonce":1364449401}]},"public_addr":"172.21.15.6:6825/1364449401","cluster_addr":"172.21.15.6:6827/1364449401","heartbeat_back_addr":"172.21.15.6:6831/1364449401","heartbeat_front_addr":"172.21.15.6:6829/1364449401","state":["exists","up"]},{"osd":1,"uuid":"7d79b9c1-6b87-476a-bc36-7db5ee64cd90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6809","nonce":2494103078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6811","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6813","nonce":2494103078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6821","nonce":2494103078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6817","nonce":2494103078}]},"public_addr":"172.21.15.6:6809/2494103078","cluster_addr":"172.21.15.6:6813/2494103078","heartbeat_back_addr":"172.21.15.6:6821/2494103078","heartbeat_front_addr":"172.21.15.6:6817/2494103078","state":["exists","up"]},{"osd":2,"uuid":"492331b9-4ee9-40b6-bacd-85d44cd6c45a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6812","nonce":1500066681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6816","nonce":1500066681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6824","nonce":1500066681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6820","nonce":1500066681}]},"public_addr":"172.21.15.6:6812/1500066681","cluster_addr":"172.21.15.6:6816/1500066681","heartbeat_back_addr":"172.21.15.6:6824/1500066681","heartbeat_front_addr":"172.21.15.6:6820/1500066681","state":["exists","up"]},{"osd":3,"uuid":"21bb2efe-9ba7-4bdc-bc31-180cd1102e21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6801","nonce":1558806123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6803","nonce":1558806123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6807","nonce":1558806123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6805","nonce":1558806123}]},"public_addr":"172.21.15.6:6801/1558806123","cluster_addr":"172.21.15.6:6803/1558806123","heartbeat_back_addr":"172.21.15.6:6807/1558806123","heartbeat_front_addr":"172.21.15.6:6805/1558806123","state":["exists","up"]},{"osd":4,"uuid":"6e2c4b71-19d9-4737-83d9-558705d8644a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6801","nonce":1526135780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6803","nonce":1526135780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6807","nonce":1526135780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6805","nonce":1526135780}]},"public_addr":"172.21.15.146:6801/1526135780","cluster_addr":"172.21.15.146:6803/1526135780","heartbeat_back_addr":"172.21.15.146:6807/1526135780","heartbeat_front_addr":"172.21.15.146:6805/1526135780","state":["exists","up"]},{"osd":5,"uuid":"fb69c525-f1be-4b7d-ab32-6c29b0d6fddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6809","nonce":460262507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6811","nonce":460262507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6815","nonce":460262507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6813","nonce":460262507}]},"public_addr":"172.21.15.146:6809/460262507","cluster_addr":"172.21.15.146:6811/460262507","heartbeat_back_addr":"172.21.15.146:6815/460262507","heartbeat_front_addr":"172.21.15.146:6813/460262507","state":["exists","up"]},{"osd":6,"uuid":"c32e7fed-370d-4401-8ce0-6a4335b0d168","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6823","nonce":1392897493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6827","nonce":1392897493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6831","nonce":1392897493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6829","nonce":1392897493}]},"public_addr":"172.21.15.146:6823/1392897493","cluster_addr":"172.21.15.146:6827/1392897493","heartbeat_back_addr":"172.21.15.146:6831/1392897493","heartbeat_front_addr":"172.21.15.146:6829/1392897493","state":["exists","up"]},{"osd":7,"uuid":"75a79b44-fa5f-426d-a7d3-71f21f78835c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6817","nonce":3268323156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6819","nonce":3268323156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6825","nonce":3268323156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6822","nonce":3268323156}]},"public_addr":"172.21.15.146:6817/3268323156","cluster_addr":"172.21.15.146:6819/3268323156","heartbeat_back_addr":"172.21.15.146:6825/3268323156","heartbeat_front_addr":"172.21.15.146:6822/3268323156","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.310437+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.295850+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.294674+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.526855+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.300023+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.595160+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.346046+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.558741+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T09:25:33.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.436+0000 7f1ea1200640 1 -- 172.21.15.6:0/4026272660 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f1e7806b500 msgr2=0x7f1e7806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:33.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.436+0000 7f1ea1200640 1 --2- 172.21.15.6:0/4026272660 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f1e7806b500 0x7f1e7806d9c0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f1e8c034000 tx=0x7f1e8c009030 comp rx=0 tx=0).stop 2024-03-04T09:25:33.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.436+0000 7f1ea1200640 1 -- 172.21.15.6:0/4026272660 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1e9c06ddb0 msgr2=0x7f1e9c06e230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:33.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.436+0000 7f1ea1200640 1 --2- 172.21.15.6:0/4026272660 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1e9c06ddb0 0x7f1e9c06e230 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f1e8400e800 tx=0x7f1e8400ecc0 comp rx=0 tx=0).stop 2024-03-04T09:25:33.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.437+0000 7f1ea1200640 1 -- 172.21.15.6:0/4026272660 shutdown_connections 2024-03-04T09:25:33.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.437+0000 7f1ea1200640 1 --2- 172.21.15.6:0/4026272660 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1e9c141770 0x7f1e9c141bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:33.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.437+0000 7f1ea1200640 1 --2- 172.21.15.6:0/4026272660 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1e9c06ddb0 0x7f1e9c06e230 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:33.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.437+0000 7f1ea1200640 1 --2- 172.21.15.6:0/4026272660 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f1e7806b500 0x7f1e7806d9c0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:33.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.437+0000 7f1ea1200640 1 --2- 172.21.15.6:0/4026272660 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1e800039e0 0x7f1e9c13ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:33.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.437+0000 7f1ea1200640 1 -- 172.21.15.6:0/4026272660 >> 172.21.15.6:0/4026272660 conn(0x7f1e9c06b3d0 msgr2=0x7f1e9c15e290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:33.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.437+0000 7f1ea1200640 1 -- 172.21.15.6:0/4026272660 shutdown_connections 2024-03-04T09:25:33.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.437+0000 7f1ea1200640 1 -- 172.21.15.6:0/4026272660 wait complete. 2024-03-04T09:25:33.454 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T09:25:07.935198+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-04T09:25:17.118573+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-04T09:25:17.930536+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-04T09:25:33.454 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-04T09:25:33.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.687+0000 7f305be00640 1 Processor -- start 2024-03-04T09:25:33.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.687+0000 7f305be00640 1 -- start start 2024-03-04T09:25:33.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.689+0000 7f305be00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f305c145430 0x7f305c15f810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:33.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.689+0000 7f305be00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f305c0b2250 0x7f305c0b2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:33.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.689+0000 7f305be00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f305c15fe40 con 0x7f305c15cc90 2024-03-04T09:25:33.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.689+0000 7f305be00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f305c160120 con 0x7f305c0b2250 2024-03-04T09:25:33.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.689+0000 7f305be00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f305c160400 con 0x7f305c145430 2024-03-04T09:25:33.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.689+0000 7f305b400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f305c0b2250 0x7f305c0b2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:33.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.689+0000 7f305a000640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f305c15cc90 0x7f305c0b1b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:36850/0 (socket says 172.21.15.6:36850) 2024-03-04T09:25:33.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.689+0000 7f305a000640 1 -- 172.21.15.6:0/2915898508 learned_addr learned my addr 172.21.15.6:0/2915898508 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:25:33.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.689+0000 7f305b400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f305c0b2250 0x7f305c0b2fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45820/0 (socket says 172.21.15.6:45820) 2024-03-04T09:25:33.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.689+0000 7f305aa00640 1 --2- 172.21.15.6:0/2915898508 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f305c145430 0x7f305c15f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:33.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.690+0000 7f305b400640 1 -- 172.21.15.6:0/2915898508 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f305c145430 msgr2=0x7f305c15f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:33.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.690+0000 7f305b400640 1 --2- 172.21.15.6:0/2915898508 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f305c145430 0x7f305c15f810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:33.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.690+0000 7f305b400640 1 -- 172.21.15.6:0/2915898508 >> v1:172.21.15.6:6789/0 conn(0x7f305c15cc90 legacy=0x7f305c0b1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:33.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.690+0000 7f305b400640 1 -- 172.21.15.6:0/2915898508 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f305c1606e0 con 0x7f305c0b2250 2024-03-04T09:25:33.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.690+0000 7f3059600640 1 -- 172.21.15.6:0/2915898508 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2270789899 0 0) 0x7f305c15fe40 con 0x7f305c15cc90 2024-03-04T09:25:33.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.691+0000 7f305b400640 1 --2- 172.21.15.6:0/2915898508 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f305c0b2250 0x7f305c0b2fd0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f3054009d90 tx=0x7f305402bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec907d0f51c6d9c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:33.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.691+0000 7f3059600640 1 -- 172.21.15.6:0/2915898508 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3054039070 con 0x7f305c0b2250 2024-03-04T09:25:33.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.691+0000 7f3059600640 1 -- 172.21.15.6:0/2915898508 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f305402cc50 con 0x7f305c0b2250 2024-03-04T09:25:33.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.691+0000 7f3059600640 1 -- 172.21.15.6:0/2915898508 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3054034440 con 0x7f305c0b2250 2024-03-04T09:25:33.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.692+0000 7f305be00640 1 -- 172.21.15.6:0/2915898508 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f305c0b2250 msgr2=0x7f305c0b2fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:33.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.692+0000 7f305be00640 1 --2- 172.21.15.6:0/2915898508 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f305c0b2250 0x7f305c0b2fd0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f3054009d90 tx=0x7f305402bea0 comp rx=0 tx=0).stop 2024-03-04T09:25:33.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.692+0000 7f305be00640 1 -- 172.21.15.6:0/2915898508 shutdown_connections 2024-03-04T09:25:33.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.692+0000 7f305be00640 1 --2- 172.21.15.6:0/2915898508 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f305c0b2250 0x7f305c0b2fd0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:33.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.692+0000 7f305be00640 1 --2- 172.21.15.6:0/2915898508 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f305c145430 0x7f305c15f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:33.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.692+0000 7f305be00640 1 -- 172.21.15.6:0/2915898508 >> 172.21.15.6:0/2915898508 conn(0x7f305c06b3a0 msgr2=0x7f305c06b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:33.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.692+0000 7f305be00640 1 -- 172.21.15.6:0/2915898508 shutdown_connections 2024-03-04T09:25:33.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.692+0000 7f305be00640 1 -- 172.21.15.6:0/2915898508 wait complete. 2024-03-04T09:25:33.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.694+0000 7f305be00640 1 Processor -- start 2024-03-04T09:25:33.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.694+0000 7f305be00640 1 -- start start 2024-03-04T09:25:33.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.695+0000 7f305be00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f305c0b2250 0x7f305c0ae3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:33.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.695+0000 7f305be00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f305c145430 0x7f305c15b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:33.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.695+0000 7f305be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f305c15cc90 0x7f305c0abe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:33.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.695+0000 7f305be00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f305c145990 con 0x7f305c15cc90 2024-03-04T09:25:33.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.695+0000 7f305be00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f305c145810 con 0x7f305c0b2250 2024-03-04T09:25:33.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.695+0000 7f305be00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f305c0cde00 con 0x7f305c145430 2024-03-04T09:25:33.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.695+0000 7f305b400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f305c15cc90 0x7f305c0abe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:33.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.695+0000 7f305b400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f305c15cc90 0x7f305c0abe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45830/0 (socket says 172.21.15.6:45830) 2024-03-04T09:25:33.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.695+0000 7f305b400640 1 -- 172.21.15.6:0/3470100658 learned_addr learned my addr 172.21.15.6:0/3470100658 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:25:33.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.695+0000 7f305aa00640 1 --2- 172.21.15.6:0/3470100658 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f305c0b2250 0x7f305c0ae3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:33.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.695+0000 7f305a000640 1 --2- 172.21.15.6:0/3470100658 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f305c145430 0x7f305c15b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:33.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.695+0000 7f305b400640 1 -- 172.21.15.6:0/3470100658 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f305c145430 msgr2=0x7f305c15b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:33.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.696+0000 7f305b400640 1 --2- 172.21.15.6:0/3470100658 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f305c145430 0x7f305c15b7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:33.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.696+0000 7f305b400640 1 -- 172.21.15.6:0/3470100658 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f305c0b2250 msgr2=0x7f305c0ae3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:33.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.696+0000 7f305b400640 1 --2- 172.21.15.6:0/3470100658 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f305c0b2250 0x7f305c0ae3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:33.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.696+0000 7f305b400640 1 -- 172.21.15.6:0/3470100658 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f305c0aea80 con 0x7f305c15cc90 2024-03-04T09:25:33.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.696+0000 7f305aa00640 1 --2- 172.21.15.6:0/3470100658 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f305c0b2250 0x7f305c0ae3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:25:33.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.696+0000 7f305a000640 1 --2- 172.21.15.6:0/3470100658 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f305c145430 0x7f305c15b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:25:33.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.696+0000 7f305b400640 1 --2- 172.21.15.6:0/3470100658 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f305c15cc90 0x7f305c0abe80 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f3054009eb0 tx=0x7f3054004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:33.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.696+0000 7f3052a00640 1 -- 172.21.15.6:0/3470100658 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3054039050 con 0x7f305c15cc90 2024-03-04T09:25:33.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.696+0000 7f3052a00640 1 -- 172.21.15.6:0/3470100658 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3054045070 con 0x7f305c15cc90 2024-03-04T09:25:33.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.696+0000 7f305be00640 1 -- 172.21.15.6:0/3470100658 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f305c0aed60 con 0x7f305c15cc90 2024-03-04T09:25:33.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.696+0000 7f3052a00640 1 -- 172.21.15.6:0/3470100658 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f305403e610 con 0x7f305c15cc90 2024-03-04T09:25:33.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.696+0000 7f305be00640 1 -- 172.21.15.6:0/3470100658 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f305c0af320 con 0x7f305c15cc90 2024-03-04T09:25:33.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.697+0000 7f305be00640 1 -- 172.21.15.6:0/3470100658 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f305c04f330 con 0x7f305c15cc90 2024-03-04T09:25:33.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.698+0000 7f3052a00640 1 -- 172.21.15.6:0/3470100658 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f305403e7b0 con 0x7f305c15cc90 2024-03-04T09:25:33.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.699+0000 7f3052a00640 1 --2- 172.21.15.6:0/3470100658 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f303c06b4d0 0x7f303c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:33.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.699+0000 7f3052a00640 1 -- 172.21.15.6:0/3470100658 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f30540acdb0 con 0x7f305c15cc90 2024-03-04T09:25:33.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.700+0000 7f305aa00640 1 --2- 172.21.15.6:0/3470100658 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f303c06b4d0 0x7f303c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:33.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.701+0000 7f305aa00640 1 --2- 172.21.15.6:0/3470100658 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f303c06b4d0 0x7f303c06d990 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f304c006fd0 tx=0x7f304c008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:33.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.704+0000 7f3052a00640 1 -- 172.21.15.6:0/3470100658 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f30540b1070 con 0x7f305c15cc90 2024-03-04T09:25:33.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.910+0000 7f305be00640 1 -- 172.21.15.6:0/3470100658 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f305c06e240 con 0x7f305c15cc90 2024-03-04T09:25:34.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.911+0000 7f3052a00640 1 -- 172.21.15.6:0/3470100658 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f305407bc20 con 0x7f305c15cc90 2024-03-04T09:25:34.629 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-04T09:25:34.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.913+0000 7f305be00640 1 -- 172.21.15.6:0/3470100658 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f303c06b4d0 msgr2=0x7f303c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:34.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.913+0000 7f305be00640 1 --2- 172.21.15.6:0/3470100658 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f303c06b4d0 0x7f303c06d990 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f304c006fd0 tx=0x7f304c008040 comp rx=0 tx=0).stop 2024-03-04T09:25:34.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.913+0000 7f305be00640 1 -- 172.21.15.6:0/3470100658 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f305c15cc90 msgr2=0x7f305c0abe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:34.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.913+0000 7f305be00640 1 --2- 172.21.15.6:0/3470100658 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f305c15cc90 0x7f305c0abe80 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f3054009eb0 tx=0x7f3054004a40 comp rx=0 tx=0).stop 2024-03-04T09:25:34.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.913+0000 7f305be00640 1 -- 172.21.15.6:0/3470100658 shutdown_connections 2024-03-04T09:25:34.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.913+0000 7f305be00640 1 --2- 172.21.15.6:0/3470100658 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f305c15cc90 0x7f305c0abe80 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:34.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.913+0000 7f305be00640 1 --2- 172.21.15.6:0/3470100658 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f303c06b4d0 0x7f303c06d990 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:34.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.913+0000 7f305be00640 1 --2- 172.21.15.6:0/3470100658 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f305c145430 0x7f305c15b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:34.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.913+0000 7f305be00640 1 --2- 172.21.15.6:0/3470100658 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f305c0b2250 0x7f305c0ae3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:34.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.913+0000 7f305be00640 1 -- 172.21.15.6:0/3470100658 >> 172.21.15.6:0/3470100658 conn(0x7f305c06b3a0 msgr2=0x7f305c15e900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:34.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.913+0000 7f305be00640 1 -- 172.21.15.6:0/3470100658 shutdown_connections 2024-03-04T09:25:34.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:33.913+0000 7f305be00640 1 -- 172.21.15.6:0/3470100658 wait complete. 2024-03-04T09:25:34.636 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-04T09:25:34.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:34.873+0000 7f0430400640 1 Processor -- start 2024-03-04T09:25:34.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:34.874+0000 7f0430400640 1 -- start start 2024-03-04T09:25:34.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:34.876+0000 7f0430400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0428072520 0x7f04280cdfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:34.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:34.876+0000 7f0430400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f0428071e20 0x7f04280ce4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:34.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:34.876+0000 7f0430400640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0428072df0 con 0x7f042815bf20 2024-03-04T09:25:34.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:34.876+0000 7f0430400640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04280cdab0 con 0x7f0428071e20 2024-03-04T09:25:34.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:34.876+0000 7f0430400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04280b4d70 con 0x7f0428072520 2024-03-04T09:25:34.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:34.876+0000 7f042da00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f042815bf20 0x7f04280acd40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:36862/0 (socket says 172.21.15.6:36862) 2024-03-04T09:25:34.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:34.876+0000 7f042da00640 1 -- 172.21.15.6:0/3453012730 learned_addr learned my addr 172.21.15.6:0/3453012730 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:25:34.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:34.877+0000 7f042e400640 1 --2- 172.21.15.6:0/3453012730 >> v2:172.21.15.146:3300/0 conn(0x7f0428071e20 0x7f04280ce4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:34.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:34.877+0000 7f042d000640 1 --2- 172.21.15.6:0/3453012730 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0428072520 0x7f04280cdfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:34.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:34.877+0000 7f0427e00640 1 -- 172.21.15.6:0/3453012730 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 791845058 0 0) 0x7f0428072df0 con 0x7f042815bf20 2024-03-04T09:25:34.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:34.877+0000 7f0427e00640 1 -- 172.21.15.6:0/3453012730 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0410003520 con 0x7f042815bf20 2024-03-04T09:25:34.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:34.877+0000 7f0427e00640 1 -- 172.21.15.6:0/3453012730 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2861279762 0 0) 0x7f0410003520 con 0x7f042815bf20 2024-03-04T09:25:34.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:34.877+0000 7f0427e00640 1 -- 172.21.15.6:0/3453012730 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0428072520 msgr2=0x7f04280cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:34.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:34.877+0000 7f0427e00640 1 --2- 172.21.15.6:0/3453012730 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0428072520 0x7f04280cdfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:34.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:34.877+0000 7f0427e00640 1 -- 172.21.15.6:0/3453012730 >> v2:172.21.15.146:3300/0 conn(0x7f0428071e20 msgr2=0x7f04280ce4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:34.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:34.877+0000 7f0427e00640 1 --2- 172.21.15.6:0/3453012730 >> v2:172.21.15.146:3300/0 conn(0x7f0428071e20 0x7f04280ce4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:34.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:34.877+0000 7f0427e00640 1 -- 172.21.15.6:0/3453012730 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04280b5050 con 0x7f042815bf20 2024-03-04T09:25:34.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:34.877+0000 7f0427e00640 1 -- 172.21.15.6:0/3453012730 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7f041c0019f0 con 0x7f042815bf20 2024-03-04T09:25:34.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:34.877+0000 7f0427e00640 1 -- 172.21.15.6:0/3453012730 >> v1:172.21.15.6:6789/0 conn(0x7f042815bf20 legacy=0x7f04280acd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:34.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:34.877+0000 7f0427e00640 1 --2- 172.21.15.6:0/3453012730 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0428072520 0x7f04100039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:34.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:34.877+0000 7f0427e00640 1 --2- 172.21.15.6:0/3453012730 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0410003f30 0x7f0410004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:34.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:34.877+0000 7f0427e00640 1 --2- 172.21.15.6:0/3453012730 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0410004d40 0x7f04100051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:34.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:34.877+0000 7f0427e00640 1 -- 172.21.15.6:0/3453012730 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04280cdab0 con 0x7f0410004d40 2024-03-04T09:25:34.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:34.877+0000 7f0427e00640 1 -- 172.21.15.6:0/3453012730 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04280b4d70 con 0x7f0410003f30 2024-03-04T09:25:34.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:34.877+0000 7f0427e00640 1 -- 172.21.15.6:0/3453012730 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f041000b9c0 con 0x7f0428072520 2024-03-04T09:25:34.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:34.878+0000 7f042e400640 1 --2- 172.21.15.6:0/3453012730 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0410004d40 0x7f04100051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:34.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:34.878+0000 7f042d000640 1 --2- 172.21.15.6:0/3453012730 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0410003f30 0x7f0410004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:34.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:34.878+0000 7f042da00640 1 --2- 172.21.15.6:0/3453012730 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0428072520 0x7f04100039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:34.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:34.878+0000 7f042e400640 1 -- 172.21.15.6:0/3453012730 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0428072520 msgr2=0x7f04100039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:34.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:34.878+0000 7f042e400640 1 --2- 172.21.15.6:0/3453012730 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0428072520 0x7f04100039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:34.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:34.878+0000 7f042e400640 1 -- 172.21.15.6:0/3453012730 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0410003f30 msgr2=0x7f0410004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:34.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:34.878+0000 7f042e400640 1 --2- 172.21.15.6:0/3453012730 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0410003f30 0x7f0410004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:34.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:34.878+0000 7f042e400640 1 -- 172.21.15.6:0/3453012730 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0410003520 con 0x7f0410004d40 2024-03-04T09:25:34.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:34.879+0000 7f042e400640 1 --2- 172.21.15.6:0/3453012730 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0410004d40 0x7f04100051c0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f0420009a90 tx=0x7f0410005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18af40c1d0cc590d server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:34.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:34.879+0000 7f0427e00640 1 -- 172.21.15.6:0/3453012730 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0420039070 con 0x7f0410004d40 2024-03-04T09:25:34.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:34.879+0000 7f0427e00640 1 -- 172.21.15.6:0/3453012730 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f042002c6b0 con 0x7f0410004d40 2024-03-04T09:25:34.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:34.879+0000 7f0427e00640 1 -- 172.21.15.6:0/3453012730 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0420009690 con 0x7f0410004d40 2024-03-04T09:25:34.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:34.880+0000 7f0430400640 1 -- 172.21.15.6:0/3453012730 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0410004d40 msgr2=0x7f04100051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:34.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:34.880+0000 7f0430400640 1 --2- 172.21.15.6:0/3453012730 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0410004d40 0x7f04100051c0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f0420009a90 tx=0x7f0410005830 comp rx=0 tx=0).stop 2024-03-04T09:25:34.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:34.880+0000 7f042e400640 1 -- 172.21.15.6:0/3453012730 reap_dead start 2024-03-04T09:25:34.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:34.880+0000 7f0430400640 1 -- 172.21.15.6:0/3453012730 shutdown_connections 2024-03-04T09:25:34.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:34.880+0000 7f0430400640 1 -- 172.21.15.6:0/3453012730 >> 172.21.15.6:0/3453012730 conn(0x7f042806b3d0 msgr2=0x7f042806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:34.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:34.880+0000 7f0430400640 1 -- 172.21.15.6:0/3453012730 shutdown_connections 2024-03-04T09:25:34.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:34.880+0000 7f0430400640 1 -- 172.21.15.6:0/3453012730 wait complete. 2024-03-04T09:25:34.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:34.882+0000 7f0430400640 1 Processor -- start 2024-03-04T09:25:34.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:34.882+0000 7f0430400640 1 -- start start 2024-03-04T09:25:34.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:34.883+0000 7f0430400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0410004d40 0x7f04281f7c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:34.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:34.883+0000 7f0430400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f042809ba70 0x7f042814d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:34.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:34.883+0000 7f0430400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f04280c9c40 0x7f04280ca0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:34.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:34.883+0000 7f0430400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04280ad450 con 0x7f042809ba70 2024-03-04T09:25:34.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:34.883+0000 7f0430400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04280b4990 con 0x7f0410004d40 2024-03-04T09:25:34.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:34.883+0000 7f0430400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f042809cf90 con 0x7f04280c9c40 2024-03-04T09:25:34.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:34.884+0000 7f042d000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f042809ba70 0x7f042814d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:34.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:34.884+0000 7f042d000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f042809ba70 0x7f042814d5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45838/0 (socket says 172.21.15.6:45838) 2024-03-04T09:25:34.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:34.884+0000 7f042d000640 1 -- 172.21.15.6:0/2948022589 learned_addr learned my addr 172.21.15.6:0/2948022589 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:25:34.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:34.884+0000 7f042e400640 1 --2- 172.21.15.6:0/2948022589 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f04280c9c40 0x7f04280ca0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:34.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:34.884+0000 7f042da00640 1 --2- 172.21.15.6:0/2948022589 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0410004d40 0x7f04281f7c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:34.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:34.884+0000 7f042d000640 1 -- 172.21.15.6:0/2948022589 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f04280c9c40 msgr2=0x7f04280ca0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:34.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:34.884+0000 7f042d000640 1 --2- 172.21.15.6:0/2948022589 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f04280c9c40 0x7f04280ca0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:34.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:34.884+0000 7f042d000640 1 -- 172.21.15.6:0/2948022589 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0410004d40 msgr2=0x7f04281f7c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:34.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:34.884+0000 7f042d000640 1 --2- 172.21.15.6:0/2948022589 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0410004d40 0x7f04281f7c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:34.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:34.884+0000 7f042d000640 1 -- 172.21.15.6:0/2948022589 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f042809c0a0 con 0x7f042809ba70 2024-03-04T09:25:34.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:34.884+0000 7f042da00640 1 --2- 172.21.15.6:0/2948022589 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0410004d40 0x7f04281f7c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T09:25:34.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:34.885+0000 7f042d000640 1 --2- 172.21.15.6:0/2948022589 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f042809ba70 0x7f042814d5d0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f0414011540 tx=0x7f0414011a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:34.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:34.885+0000 7f0425600640 1 -- 172.21.15.6:0/2948022589 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0414015930 con 0x7f042809ba70 2024-03-04T09:25:34.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:34.885+0000 7f0425600640 1 -- 172.21.15.6:0/2948022589 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0414015ad0 con 0x7f042809ba70 2024-03-04T09:25:34.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:34.885+0000 7f0425600640 1 -- 172.21.15.6:0/2948022589 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f041400f460 con 0x7f042809ba70 2024-03-04T09:25:34.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:34.885+0000 7f0430400640 1 -- 172.21.15.6:0/2948022589 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f042809c380 con 0x7f042809ba70 2024-03-04T09:25:34.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:34.885+0000 7f0430400640 1 -- 172.21.15.6:0/2948022589 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f042809c940 con 0x7f042809ba70 2024-03-04T09:25:34.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:34.886+0000 7f0430400640 1 -- 172.21.15.6:0/2948022589 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f042804f330 con 0x7f042809ba70 2024-03-04T09:25:34.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:34.887+0000 7f0425600640 1 -- 172.21.15.6:0/2948022589 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f041400f600 con 0x7f042809ba70 2024-03-04T09:25:34.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:34.888+0000 7f0425600640 1 --2- 172.21.15.6:0/2948022589 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f040806b4d0 0x7f040806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:34.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:34.888+0000 7f0425600640 1 -- 172.21.15.6:0/2948022589 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f041408aa50 con 0x7f042809ba70 2024-03-04T09:25:34.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:34.889+0000 7f042da00640 1 --2- 172.21.15.6:0/2948022589 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f040806b4d0 0x7f040806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:34.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:34.890+0000 7f042da00640 1 --2- 172.21.15.6:0/2948022589 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f040806b4d0 0x7f040806d990 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f041c005fb0 tx=0x7f041c009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:34.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:34.893+0000 7f0425600640 1 -- 172.21.15.6:0/2948022589 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0414019030 con 0x7f042809ba70 2024-03-04T09:25:35.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.092+0000 7f0430400640 1 -- 172.21.15.6:0/2948022589 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f042809be80 con 0x7f042809ba70 2024-03-04T09:25:35.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.093+0000 7f0425600640 1 -- 172.21.15.6:0/2948022589 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7f04140598c0 con 0x7f042809ba70 2024-03-04T09:25:35.090 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-03-04T09:25:35.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.096+0000 7f0430400640 1 -- 172.21.15.6:0/2948022589 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f040806b4d0 msgr2=0x7f040806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:35.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.096+0000 7f0430400640 1 --2- 172.21.15.6:0/2948022589 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f040806b4d0 0x7f040806d990 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f041c005fb0 tx=0x7f041c009040 comp rx=0 tx=0).stop 2024-03-04T09:25:35.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.096+0000 7f0430400640 1 -- 172.21.15.6:0/2948022589 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f042809ba70 msgr2=0x7f042814d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:35.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.096+0000 7f0430400640 1 --2- 172.21.15.6:0/2948022589 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f042809ba70 0x7f042814d5d0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f0414011540 tx=0x7f0414011a00 comp rx=0 tx=0).stop 2024-03-04T09:25:35.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.096+0000 7f0430400640 1 -- 172.21.15.6:0/2948022589 shutdown_connections 2024-03-04T09:25:35.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.096+0000 7f0430400640 1 --2- 172.21.15.6:0/2948022589 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f040806b4d0 0x7f040806d990 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:35.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.096+0000 7f0430400640 1 --2- 172.21.15.6:0/2948022589 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f04280c9c40 0x7f04280ca0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:35.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.096+0000 7f0430400640 1 --2- 172.21.15.6:0/2948022589 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f042809ba70 0x7f042814d5d0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:35.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.096+0000 7f0430400640 1 --2- 172.21.15.6:0/2948022589 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0410004d40 0x7f04281f7c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:35.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.096+0000 7f0430400640 1 -- 172.21.15.6:0/2948022589 >> 172.21.15.6:0/2948022589 conn(0x7f042806b3d0 msgr2=0x7f04280ccaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:35.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.096+0000 7f0430400640 1 -- 172.21.15.6:0/2948022589 shutdown_connections 2024-03-04T09:25:35.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.096+0000 7f0430400640 1 -- 172.21.15.6:0/2948022589 wait complete. 2024-03-04T09:25:35.115 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-04T09:25:35.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.345+0000 7fac57000640 1 Processor -- start 2024-03-04T09:25:35.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.345+0000 7fac57000640 1 -- start start 2024-03-04T09:25:35.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.347+0000 7fac57000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fac500b3c20 0x7fac500729d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:35.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.347+0000 7fac57000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fac500b4000 0x7fac5015b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:35.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.347+0000 7fac57000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac50071930 con 0x7fac500b1b40 2024-03-04T09:25:35.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.347+0000 7fac57000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac500b4460 con 0x7fac500b4000 2024-03-04T09:25:35.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.347+0000 7fac57000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac50071c10 con 0x7fac500b3c20 2024-03-04T09:25:35.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.347+0000 7fac56600640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fac500b1b40 0x7fac5015c220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:36872/0 (socket says 172.21.15.6:36872) 2024-03-04T09:25:35.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.347+0000 7fac56600640 1 -- 172.21.15.6:0/3328356208 learned_addr learned my addr 172.21.15.6:0/3328356208 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:25:35.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.347+0000 7fac55c00640 1 --2- 172.21.15.6:0/3328356208 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fac500b3c20 0x7fac500729d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:35.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.347+0000 7fac55200640 1 --2- 172.21.15.6:0/3328356208 >> v2:172.21.15.146:3300/0 conn(0x7fac500b4000 0x7fac5015b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:35.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.348+0000 7fac4fe00640 1 -- 172.21.15.6:0/3328356208 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 977667797 0 0) 0x7fac50071930 con 0x7fac500b1b40 2024-03-04T09:25:35.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.348+0000 7fac4fe00640 1 -- 172.21.15.6:0/3328356208 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac38003520 con 0x7fac500b1b40 2024-03-04T09:25:35.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.348+0000 7fac4fe00640 1 -- 172.21.15.6:0/3328356208 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1764420459 0 0) 0x7fac38003520 con 0x7fac500b1b40 2024-03-04T09:25:35.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.348+0000 7fac4fe00640 1 -- 172.21.15.6:0/3328356208 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fac500b3c20 msgr2=0x7fac500729d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:35.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.348+0000 7fac4fe00640 1 --2- 172.21.15.6:0/3328356208 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fac500b3c20 0x7fac500729d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:35.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.348+0000 7fac4fe00640 1 -- 172.21.15.6:0/3328356208 >> v2:172.21.15.146:3300/0 conn(0x7fac500b4000 msgr2=0x7fac5015b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:35.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.348+0000 7fac4fe00640 1 --2- 172.21.15.6:0/3328356208 >> v2:172.21.15.146:3300/0 conn(0x7fac500b4000 0x7fac5015b8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:35.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.348+0000 7fac4fe00640 1 -- 172.21.15.6:0/3328356208 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac50071ef0 con 0x7fac500b1b40 2024-03-04T09:25:35.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.348+0000 7fac4fe00640 1 -- 172.21.15.6:0/3328356208 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7fac480019f0 con 0x7fac500b1b40 2024-03-04T09:25:35.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.348+0000 7fac4fe00640 1 -- 172.21.15.6:0/3328356208 >> v1:172.21.15.6:6789/0 conn(0x7fac500b1b40 legacy=0x7fac5015c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:35.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.348+0000 7fac4fe00640 1 --2- 172.21.15.6:0/3328356208 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fac380039b0 0x7fac38003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:35.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.348+0000 7fac4fe00640 1 --2- 172.21.15.6:0/3328356208 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fac38004800 0x7fac38004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:35.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.348+0000 7fac4fe00640 1 --2- 172.21.15.6:0/3328356208 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fac500b3c20 0x7fac380051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:35.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.348+0000 7fac4fe00640 1 -- 172.21.15.6:0/3328356208 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac500b4460 con 0x7fac38004800 2024-03-04T09:25:35.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.348+0000 7fac4fe00640 1 -- 172.21.15.6:0/3328356208 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac50071c10 con 0x7fac380039b0 2024-03-04T09:25:35.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.348+0000 7fac4fe00640 1 -- 172.21.15.6:0/3328356208 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac3800dbd0 con 0x7fac500b3c20 2024-03-04T09:25:35.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.348+0000 7fac55200640 1 --2- 172.21.15.6:0/3328356208 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fac38004800 0x7fac38004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:35.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.349+0000 7fac55c00640 1 --2- 172.21.15.6:0/3328356208 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fac380039b0 0x7fac38003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:35.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.349+0000 7fac55200640 1 -- 172.21.15.6:0/3328356208 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fac500b3c20 msgr2=0x7fac380051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:35.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.349+0000 7fac55200640 1 --2- 172.21.15.6:0/3328356208 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fac500b3c20 0x7fac380051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:35.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.349+0000 7fac55200640 1 -- 172.21.15.6:0/3328356208 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fac380039b0 msgr2=0x7fac38003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:35.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.349+0000 7fac55200640 1 --2- 172.21.15.6:0/3328356208 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fac380039b0 0x7fac38003dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:35.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.349+0000 7fac55200640 1 -- 172.21.15.6:0/3328356208 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac38003520 con 0x7fac38004800 2024-03-04T09:25:35.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.350+0000 7fac55200640 1 --2- 172.21.15.6:0/3328356208 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fac38004800 0x7fac38004c80 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fac40009ac0 tx=0x7fac38005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81510ac059efabb0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:35.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.350+0000 7fac4fe00640 1 -- 172.21.15.6:0/3328356208 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac40039070 con 0x7fac38004800 2024-03-04T09:25:35.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.350+0000 7fac4fe00640 1 -- 172.21.15.6:0/3328356208 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac4002c6d0 con 0x7fac38004800 2024-03-04T09:25:35.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.350+0000 7fac4fe00640 1 -- 172.21.15.6:0/3328356208 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac40009690 con 0x7fac38004800 2024-03-04T09:25:35.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.351+0000 7fac57000640 1 -- 172.21.15.6:0/3328356208 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fac38004800 msgr2=0x7fac38004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:35.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.351+0000 7fac57000640 1 --2- 172.21.15.6:0/3328356208 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fac38004800 0x7fac38004c80 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fac40009ac0 tx=0x7fac38005830 comp rx=0 tx=0).stop 2024-03-04T09:25:35.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.351+0000 7fac56600640 1 -- 172.21.15.6:0/3328356208 reap_dead start 2024-03-04T09:25:35.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.351+0000 7fac57000640 1 -- 172.21.15.6:0/3328356208 shutdown_connections 2024-03-04T09:25:35.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.351+0000 7fac57000640 1 -- 172.21.15.6:0/3328356208 >> 172.21.15.6:0/3328356208 conn(0x7fac5006b3a0 msgr2=0x7fac5006b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:35.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.351+0000 7fac57000640 1 -- 172.21.15.6:0/3328356208 shutdown_connections 2024-03-04T09:25:35.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.351+0000 7fac57000640 1 -- 172.21.15.6:0/3328356208 wait complete. 2024-03-04T09:25:35.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.353+0000 7fac57000640 1 Processor -- start 2024-03-04T09:25:35.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.353+0000 7fac57000640 1 -- start start 2024-03-04T09:25:35.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.354+0000 7fac57000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fac38004800 0x7fac500c6550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:35.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.354+0000 7fac57000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fac501f48d0 0x7fac5015c200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:35.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.354+0000 7fac57000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fac500cdba0 0x7fac500ce050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:35.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.354+0000 7fac57000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac5015bde0 con 0x7fac501f48d0 2024-03-04T09:25:35.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.354+0000 7fac57000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac500724b0 con 0x7fac500cdba0 2024-03-04T09:25:35.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.354+0000 7fac57000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac5015bf60 con 0x7fac38004800 2024-03-04T09:25:35.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.354+0000 7fac55200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fac501f48d0 0x7fac5015c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:35.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.354+0000 7fac55200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fac501f48d0 0x7fac5015c200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45856/0 (socket says 172.21.15.6:45856) 2024-03-04T09:25:35.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.354+0000 7fac55200640 1 -- 172.21.15.6:0/745616030 learned_addr learned my addr 172.21.15.6:0/745616030 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:25:35.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.355+0000 7fac55c00640 1 --2- 172.21.15.6:0/745616030 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fac38004800 0x7fac500c6550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:35.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.355+0000 7fac55200640 1 -- 172.21.15.6:0/745616030 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fac38004800 msgr2=0x7fac500c6550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:35.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.355+0000 7fac55200640 1 --2- 172.21.15.6:0/745616030 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fac38004800 0x7fac500c6550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:35.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.355+0000 7fac55200640 1 -- 172.21.15.6:0/745616030 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fac500cdba0 msgr2=0x7fac500ce050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:35.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.355+0000 7fac55200640 1 --2- 172.21.15.6:0/745616030 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fac500cdba0 0x7fac500ce050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:35.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.355+0000 7fac55200640 1 -- 172.21.15.6:0/745616030 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac500c6a90 con 0x7fac501f48d0 2024-03-04T09:25:35.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.355+0000 7fac55200640 1 --2- 172.21.15.6:0/745616030 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fac501f48d0 0x7fac5015c200 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fac40000c00 tx=0x7fac40004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:35.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.355+0000 7fac4d600640 1 -- 172.21.15.6:0/745616030 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac40039050 con 0x7fac501f48d0 2024-03-04T09:25:35.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.355+0000 7fac4d600640 1 -- 172.21.15.6:0/745616030 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac40045070 con 0x7fac501f48d0 2024-03-04T09:25:35.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.355+0000 7fac4d600640 1 -- 172.21.15.6:0/745616030 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac400044e0 con 0x7fac501f48d0 2024-03-04T09:25:35.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.355+0000 7fac57000640 1 -- 172.21.15.6:0/745616030 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac5015cb00 con 0x7fac501f48d0 2024-03-04T09:25:35.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.355+0000 7fac57000640 1 -- 172.21.15.6:0/745616030 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac5015d0c0 con 0x7fac501f48d0 2024-03-04T09:25:35.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.357+0000 7fac57000640 1 -- 172.21.15.6:0/745616030 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac5004f330 con 0x7fac501f48d0 2024-03-04T09:25:35.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.357+0000 7fac4d600640 1 -- 172.21.15.6:0/745616030 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fac40003a50 con 0x7fac501f48d0 2024-03-04T09:25:35.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.358+0000 7fac4d600640 1 --2- 172.21.15.6:0/745616030 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fac3406b500 0x7fac3406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:35.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.358+0000 7fac4d600640 1 -- 172.21.15.6:0/745616030 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fac400ac420 con 0x7fac501f48d0 2024-03-04T09:25:35.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.359+0000 7fac55c00640 1 --2- 172.21.15.6:0/745616030 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fac3406b500 0x7fac3406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:35.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.360+0000 7fac55c00640 1 --2- 172.21.15.6:0/745616030 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fac3406b500 0x7fac3406d9c0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fac44009830 tx=0x7fac44008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:35.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.363+0000 7fac4d600640 1 -- 172.21.15.6:0/745616030 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fac4007b2a0 con 0x7fac501f48d0 2024-03-04T09:25:35.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.569+0000 7fac57000640 1 -- 172.21.15.6:0/745616030 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fac500728d0 con 0x7fac501f48d0 2024-03-04T09:25:35.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.570+0000 7fac4d600640 1 -- 172.21.15.6:0/745616030 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7fac4007aba0 con 0x7fac501f48d0 2024-03-04T09:25:35.567 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 64 2024-03-04T09:25:35.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.572+0000 7fac57000640 1 -- 172.21.15.6:0/745616030 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fac3406b500 msgr2=0x7fac3406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:35.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.572+0000 7fac57000640 1 --2- 172.21.15.6:0/745616030 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fac3406b500 0x7fac3406d9c0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fac44009830 tx=0x7fac44008040 comp rx=0 tx=0).stop 2024-03-04T09:25:35.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.572+0000 7fac57000640 1 -- 172.21.15.6:0/745616030 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fac501f48d0 msgr2=0x7fac5015c200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:35.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.572+0000 7fac57000640 1 --2- 172.21.15.6:0/745616030 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fac501f48d0 0x7fac5015c200 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fac40000c00 tx=0x7fac40004730 comp rx=0 tx=0).stop 2024-03-04T09:25:35.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.573+0000 7fac57000640 1 -- 172.21.15.6:0/745616030 shutdown_connections 2024-03-04T09:25:35.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.573+0000 7fac57000640 1 --2- 172.21.15.6:0/745616030 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fac500cdba0 0x7fac500ce050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:35.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.573+0000 7fac57000640 1 --2- 172.21.15.6:0/745616030 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fac501f48d0 0x7fac5015c200 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:35.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.573+0000 7fac57000640 1 --2- 172.21.15.6:0/745616030 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fac3406b500 0x7fac3406d9c0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:35.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.573+0000 7fac57000640 1 --2- 172.21.15.6:0/745616030 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fac38004800 0x7fac500c6550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:35.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.573+0000 7fac57000640 1 -- 172.21.15.6:0/745616030 >> 172.21.15.6:0/745616030 conn(0x7fac5006b3a0 msgr2=0x7fac5006c8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:35.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.573+0000 7fac57000640 1 -- 172.21.15.6:0/745616030 shutdown_connections 2024-03-04T09:25:35.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.573+0000 7fac57000640 1 -- 172.21.15.6:0/745616030 wait complete. 2024-03-04T09:25:35.591 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:25:35.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.812+0000 7f2fbac00640 1 Processor -- start 2024-03-04T09:25:35.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.813+0000 7f2fbac00640 1 -- start start 2024-03-04T09:25:35.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.814+0000 7f2fbac00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f2fb40af1a0 0x7f2fb415c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:35.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.814+0000 7f2fbac00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2fb40cdc20 0x7f2fb40b2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:35.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.814+0000 7f2fbac00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fb415cc40 con 0x7f2fb40af580 2024-03-04T09:25:35.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.815+0000 7f2fbac00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fb415cf20 con 0x7f2fb40af1a0 2024-03-04T09:25:35.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.815+0000 7f2fbac00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fb415d200 con 0x7f2fb40cdc20 2024-03-04T09:25:35.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.815+0000 7f2fb8c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2fb40cdc20 0x7f2fb40b2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:35.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.815+0000 7f2fb8c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2fb40cdc20 0x7f2fb40b2dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.6:51228/0 (socket says 172.21.15.6:51228) 2024-03-04T09:25:35.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.815+0000 7f2fb8c00640 1 -- 172.21.15.6:0/2769918481 learned_addr learned my addr 172.21.15.6:0/2769918481 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:25:35.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.815+0000 7f2fb3e00640 1 --2- 172.21.15.6:0/2769918481 >> v2:172.21.15.146:3300/0 conn(0x7f2fb40af1a0 0x7f2fb415c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:35.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.816+0000 7f2fb3e00640 1 -- 172.21.15.6:0/2769918481 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2fb40cdc20 msgr2=0x7f2fb40b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:35.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.816+0000 7f2fb3e00640 1 --2- 172.21.15.6:0/2769918481 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2fb40cdc20 0x7f2fb40b2dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:35.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.816+0000 7f2fb3e00640 1 -- 172.21.15.6:0/2769918481 >> v1:172.21.15.146:6789/0 conn(0x7f2fb40af580 legacy=0x7f2fb40b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:35.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.816+0000 7f2fb3e00640 1 -- 172.21.15.6:0/2769918481 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fb415d4e0 con 0x7f2fb40af1a0 2024-03-04T09:25:35.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.816+0000 7f2fb8c00640 1 --2- 172.21.15.6:0/2769918481 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2fb40cdc20 0x7f2fb40b2dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T09:25:35.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.817+0000 7f2fb3e00640 1 --2- 172.21.15.6:0/2769918481 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2fb40af1a0 0x7f2fb415c600 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f2f9c0097a0 tx=0x7f2f9c02b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d84db1d1c8774b11 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:35.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.818+0000 7f2fb2a00640 1 -- 172.21.15.6:0/2769918481 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f9c0315a0 con 0x7f2fb40af1a0 2024-03-04T09:25:35.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.818+0000 7f2fb2a00640 1 -- 172.21.15.6:0/2769918481 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f9c037070 con 0x7f2fb40af1a0 2024-03-04T09:25:35.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.818+0000 7f2fb2a00640 1 -- 172.21.15.6:0/2769918481 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f9c031960 con 0x7f2fb40af1a0 2024-03-04T09:25:35.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.819+0000 7f2fbac00640 1 -- 172.21.15.6:0/2769918481 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2fb40af1a0 msgr2=0x7f2fb415c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:35.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.819+0000 7f2fbac00640 1 --2- 172.21.15.6:0/2769918481 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2fb40af1a0 0x7f2fb415c600 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f2f9c0097a0 tx=0x7f2f9c02b840 comp rx=0 tx=0).stop 2024-03-04T09:25:35.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.819+0000 7f2fbac00640 1 -- 172.21.15.6:0/2769918481 shutdown_connections 2024-03-04T09:25:35.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.819+0000 7f2fbac00640 1 --2- 172.21.15.6:0/2769918481 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2fb40cdc20 0x7f2fb40b2dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:35.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.819+0000 7f2fbac00640 1 --2- 172.21.15.6:0/2769918481 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2fb40af1a0 0x7f2fb415c600 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:35.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.820+0000 7f2fbac00640 1 -- 172.21.15.6:0/2769918481 >> 172.21.15.6:0/2769918481 conn(0x7f2fb406afb0 msgr2=0x7f2fb406b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:35.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.820+0000 7f2fbac00640 1 -- 172.21.15.6:0/2769918481 shutdown_connections 2024-03-04T09:25:35.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.820+0000 7f2fbac00640 1 -- 172.21.15.6:0/2769918481 wait complete. 2024-03-04T09:25:35.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.822+0000 7f2fbac00640 1 Processor -- start 2024-03-04T09:25:35.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.822+0000 7f2fbac00640 1 -- start start 2024-03-04T09:25:35.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.823+0000 7f2fbac00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2fb40af1a0 0x7f2fb41456c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:35.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.823+0000 7f2fbac00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2fb40af580 0x7f2fb409aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:35.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.823+0000 7f2fbac00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2fb40cdc20 0x7f2fb406e310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:35.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.824+0000 7f2fbac00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fb40b3720 con 0x7f2fb40af580 2024-03-04T09:25:35.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.824+0000 7f2fbac00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fb40b35a0 con 0x7f2fb40af1a0 2024-03-04T09:25:35.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.824+0000 7f2fbac00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fb4155e70 con 0x7f2fb40cdc20 2024-03-04T09:25:35.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.824+0000 7f2fb3400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2fb40af580 0x7f2fb409aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:35.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.824+0000 7f2fb3400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2fb40af580 0x7f2fb409aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45868/0 (socket says 172.21.15.6:45868) 2024-03-04T09:25:35.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.824+0000 7f2fb3400640 1 -- 172.21.15.6:0/3771733106 learned_addr learned my addr 172.21.15.6:0/3771733106 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:25:35.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.824+0000 7f2fb3e00640 1 --2- 172.21.15.6:0/3771733106 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2fb40af1a0 0x7f2fb41456c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:35.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.824+0000 7f2fb8c00640 1 --2- 172.21.15.6:0/3771733106 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2fb40cdc20 0x7f2fb406e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:35.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.824+0000 7f2fb3400640 1 -- 172.21.15.6:0/3771733106 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2fb40cdc20 msgr2=0x7f2fb406e310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:35.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.824+0000 7f2fb3400640 1 --2- 172.21.15.6:0/3771733106 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2fb40cdc20 0x7f2fb406e310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:35.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.824+0000 7f2fb3400640 1 -- 172.21.15.6:0/3771733106 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2fb40af1a0 msgr2=0x7f2fb41456c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:35.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.824+0000 7f2fb3400640 1 --2- 172.21.15.6:0/3771733106 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2fb40af1a0 0x7f2fb41456c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:35.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.824+0000 7f2fb3400640 1 -- 172.21.15.6:0/3771733106 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fb4150d90 con 0x7f2fb40af580 2024-03-04T09:25:35.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.824+0000 7f2fb3e00640 1 --2- 172.21.15.6:0/3771733106 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2fb40af1a0 0x7f2fb41456c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T09:25:35.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.825+0000 7f2fb3400640 1 --2- 172.21.15.6:0/3771733106 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2fb40af580 0x7f2fb409aa10 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f2fa400c7d0 tx=0x7f2fa400cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:35.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.825+0000 7f2fabe00640 1 -- 172.21.15.6:0/3771733106 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fa4016020 con 0x7f2fb40af580 2024-03-04T09:25:35.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.825+0000 7f2fabe00640 1 -- 172.21.15.6:0/3771733106 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fa4006d50 con 0x7f2fb40af580 2024-03-04T09:25:35.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.825+0000 7f2fbac00640 1 -- 172.21.15.6:0/3771733106 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fb406d270 con 0x7f2fb40af580 2024-03-04T09:25:35.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.825+0000 7f2fabe00640 1 -- 172.21.15.6:0/3771733106 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fa4003bb0 con 0x7f2fb40af580 2024-03-04T09:25:35.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.825+0000 7f2fbac00640 1 -- 172.21.15.6:0/3771733106 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fb406d830 con 0x7f2fb40af580 2024-03-04T09:25:35.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.826+0000 7f2fbac00640 1 -- 172.21.15.6:0/3771733106 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fb4069510 con 0x7f2fb40af580 2024-03-04T09:25:35.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.827+0000 7f2fabe00640 1 -- 172.21.15.6:0/3771733106 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2fa400a850 con 0x7f2fb40af580 2024-03-04T09:25:35.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.829+0000 7f2fabe00640 1 --2- 172.21.15.6:0/3771733106 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2f9006b4d0 0x7f2f9006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:35.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.829+0000 7f2fabe00640 1 -- 172.21.15.6:0/3771733106 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2fa4089ef0 con 0x7f2fb40af580 2024-03-04T09:25:35.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.829+0000 7f2fb3e00640 1 --2- 172.21.15.6:0/3771733106 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2f9006b4d0 0x7f2f9006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:35.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.830+0000 7f2fb3e00640 1 --2- 172.21.15.6:0/3771733106 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2f9006b4d0 0x7f2f9006d990 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f2f9c036000 tx=0x7f2f9c03d040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:35.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:35.833+0000 7f2fabe00640 1 -- 172.21.15.6:0/3771733106 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2fa4020020 con 0x7f2fb40af580 2024-03-04T09:25:36.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.074+0000 7f2fbac00640 1 -- 172.21.15.6:0/3771733106 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2fb415b850 con 0x7f2fb40af580 2024-03-04T09:25:36.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.075+0000 7f2fabe00640 1 -- 172.21.15.6:0/3771733106 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4971 (secure 0 0 0) 0x7f2fa4019310 con 0x7f2fb40af580 2024-03-04T09:25:36.073 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:25:36.073 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4248,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/2702058134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2702058134},{"type":"v1","addr":"172.21.15.6:6833","nonce":2702058134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4263,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/3866998852","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":3866998852},{"type":"v1","addr":"172.21.15.6:6835","nonce":3866998852}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4285,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/3523455455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":3523455455},{"type":"v1","addr":"172.21.15.146:6837","nonce":3523455455}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:25:18.912808+0000","modified":"2024-03-04T09:25:22.016493+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4288},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4288":{"gid":4288,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.146:6836/1946537763","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":1946537763},{"type":"v1","addr":"172.21.15.146:6836","nonce":1946537763}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4288,"qdb_cluster":[4288]},"id":1}]} 2024-03-04T09:25:36.073 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 7 2024-03-04T09:25:36.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.078+0000 7f2fbac00640 1 -- 172.21.15.6:0/3771733106 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2f9006b4d0 msgr2=0x7f2f9006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:36.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.078+0000 7f2fbac00640 1 --2- 172.21.15.6:0/3771733106 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2f9006b4d0 0x7f2f9006d990 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f2f9c036000 tx=0x7f2f9c03d040 comp rx=0 tx=0).stop 2024-03-04T09:25:36.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.078+0000 7f2fbac00640 1 -- 172.21.15.6:0/3771733106 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2fb40af580 msgr2=0x7f2fb409aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:36.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.078+0000 7f2fbac00640 1 --2- 172.21.15.6:0/3771733106 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2fb40af580 0x7f2fb409aa10 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f2fa400c7d0 tx=0x7f2fa400cc90 comp rx=0 tx=0).stop 2024-03-04T09:25:36.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.078+0000 7f2fbac00640 1 -- 172.21.15.6:0/3771733106 shutdown_connections 2024-03-04T09:25:36.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.078+0000 7f2fbac00640 1 --2- 172.21.15.6:0/3771733106 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2f9006b4d0 0x7f2f9006d990 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:36.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.078+0000 7f2fbac00640 1 --2- 172.21.15.6:0/3771733106 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2fb40cdc20 0x7f2fb406e310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:36.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.078+0000 7f2fbac00640 1 --2- 172.21.15.6:0/3771733106 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2fb40af580 0x7f2fb409aa10 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:36.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.078+0000 7f2fbac00640 1 --2- 172.21.15.6:0/3771733106 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2fb40af1a0 0x7f2fb41456c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:36.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.078+0000 7f2fbac00640 1 -- 172.21.15.6:0/3771733106 >> 172.21.15.6:0/3771733106 conn(0x7f2fb406afb0 msgr2=0x7f2fb406b670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:36.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.078+0000 7f2fbac00640 1 -- 172.21.15.6:0/3771733106 shutdown_connections 2024-03-04T09:25:36.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.079+0000 7f2fbac00640 1 -- 172.21.15.6:0/3771733106 wait complete. 2024-03-04T09:25:36.093 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T09:25:36.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.320+0000 7ff8e9e00640 1 Processor -- start 2024-03-04T09:25:36.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.320+0000 7ff8e9e00640 1 -- start start 2024-03-04T09:25:36.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.322+0000 7ff8e9e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff8e40af1a0 0x7ff8e415c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:36.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.322+0000 7ff8e9e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ff8e40af580 0x7ff8e40b4c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:36.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.322+0000 7ff8e9e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff8e40cdc20 0x7ff8e40b2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:36.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.322+0000 7ff8e9e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8e415cc40 con 0x7ff8e40af1a0 2024-03-04T09:25:36.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.322+0000 7ff8e9e00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8e415cf20 con 0x7ff8e40af580 2024-03-04T09:25:36.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.322+0000 7ff8e9e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8e415d200 con 0x7ff8e40cdc20 2024-03-04T09:25:36.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.322+0000 7ff8e2a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff8e40af1a0 0x7ff8e415c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:36.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.322+0000 7ff8e2a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff8e40af1a0 0x7ff8e415c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45876/0 (socket says 172.21.15.6:45876) 2024-03-04T09:25:36.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.322+0000 7ff8e2a00640 1 -- 172.21.15.6:0/11201704 learned_addr learned my addr 172.21.15.6:0/11201704 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:25:36.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.323+0000 7ff8e3400640 1 --2- 172.21.15.6:0/11201704 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff8e40cdc20 0x7ff8e40b2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:36.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.323+0000 7ff8da000640 1 --2- 172.21.15.6:0/11201704 >> v2:172.21.15.146:3300/0 conn(0x7ff8e40af580 0x7ff8e40b4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:36.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.323+0000 7ff8e2a00640 1 -- 172.21.15.6:0/11201704 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff8e40cdc20 msgr2=0x7ff8e40b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:36.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.323+0000 7ff8e2a00640 1 --2- 172.21.15.6:0/11201704 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff8e40cdc20 0x7ff8e40b2dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:36.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.323+0000 7ff8e2a00640 1 -- 172.21.15.6:0/11201704 >> v2:172.21.15.146:3300/0 conn(0x7ff8e40af580 msgr2=0x7ff8e40b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:36.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.323+0000 7ff8e2a00640 1 --2- 172.21.15.6:0/11201704 >> v2:172.21.15.146:3300/0 conn(0x7ff8e40af580 0x7ff8e40b4c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:36.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.323+0000 7ff8e2a00640 1 -- 172.21.15.6:0/11201704 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8e415d4e0 con 0x7ff8e40af1a0 2024-03-04T09:25:36.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.323+0000 7ff8da000640 1 --2- 172.21.15.6:0/11201704 >> v2:172.21.15.146:3300/0 conn(0x7ff8e40af580 0x7ff8e40b4c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T09:25:36.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.324+0000 7ff8e2a00640 1 --2- 172.21.15.6:0/11201704 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff8e40af1a0 0x7ff8e415c600 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7ff8e40b54c0 tx=0x7ff8d402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf76e6db5084733f server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:36.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.324+0000 7ff8e2000640 1 -- 172.21.15.6:0/11201704 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8d402c9e0 con 0x7ff8e40af1a0 2024-03-04T09:25:36.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.324+0000 7ff8e2000640 1 -- 172.21.15.6:0/11201704 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8d402cb80 con 0x7ff8e40af1a0 2024-03-04T09:25:36.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.324+0000 7ff8e2000640 1 -- 172.21.15.6:0/11201704 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8d40343f0 con 0x7ff8e40af1a0 2024-03-04T09:25:36.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.325+0000 7ff8e9e00640 1 -- 172.21.15.6:0/11201704 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff8e40af1a0 msgr2=0x7ff8e415c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:36.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.325+0000 7ff8e9e00640 1 --2- 172.21.15.6:0/11201704 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff8e40af1a0 0x7ff8e415c600 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7ff8e40b54c0 tx=0x7ff8d402be40 comp rx=0 tx=0).stop 2024-03-04T09:25:36.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.325+0000 7ff8e9e00640 1 -- 172.21.15.6:0/11201704 shutdown_connections 2024-03-04T09:25:36.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.325+0000 7ff8e9e00640 1 --2- 172.21.15.6:0/11201704 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff8e40cdc20 0x7ff8e40b2dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:36.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.325+0000 7ff8e9e00640 1 --2- 172.21.15.6:0/11201704 >> v2:172.21.15.146:3300/0 conn(0x7ff8e40af580 0x7ff8e40b4c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:36.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.325+0000 7ff8e9e00640 1 --2- 172.21.15.6:0/11201704 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff8e40af1a0 0x7ff8e415c600 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:36.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.325+0000 7ff8e9e00640 1 -- 172.21.15.6:0/11201704 >> 172.21.15.6:0/11201704 conn(0x7ff8e406afb0 msgr2=0x7ff8e406b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:36.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.325+0000 7ff8e9e00640 1 -- 172.21.15.6:0/11201704 shutdown_connections 2024-03-04T09:25:36.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.325+0000 7ff8e9e00640 1 -- 172.21.15.6:0/11201704 wait complete. 2024-03-04T09:25:36.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.327+0000 7ff8e9e00640 1 Processor -- start 2024-03-04T09:25:36.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.327+0000 7ff8e9e00640 1 -- start start 2024-03-04T09:25:36.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.328+0000 7ff8e9e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff8e40af1a0 0x7ff8e415ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:36.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.328+0000 7ff8e9e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff8e40af580 0x7ff8e409c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:36.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.328+0000 7ff8e9e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff8e40cdc20 0x7ff8e406cb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:36.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.328+0000 7ff8e9e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8e40b3490 con 0x7ff8e40af1a0 2024-03-04T09:25:36.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.328+0000 7ff8e9e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8e40b3310 con 0x7ff8e40af580 2024-03-04T09:25:36.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.328+0000 7ff8e9e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8e40b3610 con 0x7ff8e40cdc20 2024-03-04T09:25:36.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.328+0000 7ff8e2a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff8e40af1a0 0x7ff8e415ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:36.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.328+0000 7ff8e2a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff8e40af1a0 0x7ff8e415ddd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45880/0 (socket says 172.21.15.6:45880) 2024-03-04T09:25:36.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.328+0000 7ff8e2a00640 1 -- 172.21.15.6:0/3083029587 learned_addr learned my addr 172.21.15.6:0/3083029587 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:25:36.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.329+0000 7ff8da000640 1 --2- 172.21.15.6:0/3083029587 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff8e40af580 0x7ff8e409c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:36.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.329+0000 7ff8e3400640 1 --2- 172.21.15.6:0/3083029587 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff8e40cdc20 0x7ff8e406cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:36.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.329+0000 7ff8e2a00640 1 -- 172.21.15.6:0/3083029587 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff8e40cdc20 msgr2=0x7ff8e406cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:36.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.329+0000 7ff8e2a00640 1 --2- 172.21.15.6:0/3083029587 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff8e40cdc20 0x7ff8e406cb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:36.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.329+0000 7ff8e2a00640 1 -- 172.21.15.6:0/3083029587 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff8e40af580 msgr2=0x7ff8e409c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:36.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.329+0000 7ff8e2a00640 1 --2- 172.21.15.6:0/3083029587 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff8e40af580 0x7ff8e409c6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:36.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.329+0000 7ff8e2a00640 1 -- 172.21.15.6:0/3083029587 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8e4151ca0 con 0x7ff8e40af1a0 2024-03-04T09:25:36.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.329+0000 7ff8e2a00640 1 --2- 172.21.15.6:0/3083029587 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff8e40af1a0 0x7ff8e415ddd0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7ff8d4000f80 tx=0x7ff8d4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:36.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.329+0000 7ff8dbe00640 1 -- 172.21.15.6:0/3083029587 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8d4043020 con 0x7ff8e40af1a0 2024-03-04T09:25:36.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.329+0000 7ff8dbe00640 1 -- 172.21.15.6:0/3083029587 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8d403e450 con 0x7ff8e40af1a0 2024-03-04T09:25:36.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.329+0000 7ff8dbe00640 1 -- 172.21.15.6:0/3083029587 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8d4009510 con 0x7ff8e40af1a0 2024-03-04T09:25:36.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.329+0000 7ff8e9e00640 1 -- 172.21.15.6:0/3083029587 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8e4151f80 con 0x7ff8e40af1a0 2024-03-04T09:25:36.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.330+0000 7ff8e9e00640 1 -- 172.21.15.6:0/3083029587 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8e4152540 con 0x7ff8e40af1a0 2024-03-04T09:25:36.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.331+0000 7ff8e9e00640 1 -- 172.21.15.6:0/3083029587 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8e4152740 con 0x7ff8e40af1a0 2024-03-04T09:25:36.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.332+0000 7ff8dbe00640 1 -- 172.21.15.6:0/3083029587 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff8d4046020 con 0x7ff8e40af1a0 2024-03-04T09:25:36.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.332+0000 7ff8dbe00640 1 --2- 172.21.15.6:0/3083029587 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff8c006b4d0 0x7ff8c006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:36.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.332+0000 7ff8dbe00640 1 -- 172.21.15.6:0/3083029587 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff8d40aca90 con 0x7ff8e40af1a0 2024-03-04T09:25:36.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.333+0000 7ff8da000640 1 --2- 172.21.15.6:0/3083029587 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff8c006b4d0 0x7ff8c006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:36.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.334+0000 7ff8da000640 1 --2- 172.21.15.6:0/3083029587 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff8c006b4d0 0x7ff8c006d990 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7ff8cc006fd0 tx=0x7ff8cc008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:36.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.337+0000 7ff8dbe00640 1 -- 172.21.15.6:0/3083029587 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff8e4152740 con 0x7ff8e40af1a0 2024-03-04T09:25:36.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.540+0000 7ff8e9e00640 1 -- 172.21.15.6:0/3083029587 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff8e40adac0 con 0x7ff8e40af1a0 2024-03-04T09:25:36.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.543+0000 7ff8dbe00640 1 -- 172.21.15.6:0/3083029587 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16557 (secure 0 0 0) 0x7ff8d404d030 con 0x7ff8e40af1a0 2024-03-04T09:25:36.540 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:25:36.540 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":20,"fsid":"edce0578-5ae4-4402-b45a-552bdef84d7c","created":"2024-03-04T09:24:54.523780+0000","modified":"2024-03-04T09:25:20.446333+0000","last_up_change":"2024-03-04T09:25:07.089049+0000","last_in_change":"2024-03-04T09:24:57.982077+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T09:25:07.935198+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-04T09:25:17.118573+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-04T09:25:17.930536+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a96574f-81f1-4bc6-8463-757520b84a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6825","nonce":1364449401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6827","nonce":1364449401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6831","nonce":1364449401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6829","nonce":1364449401}]},"public_addr":"172.21.15.6:6825/1364449401","cluster_addr":"172.21.15.6:6827/1364449401","heartbeat_back_addr":"172.21.15.6:6831/1364449401","heartbeat_front_addr":"172.21.15.6:6829/1364449401","state":["exists","up"]},{"osd":1,"uuid":"7d79b9c1-6b87-476a-bc36-7db5ee64cd90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6809","nonce":2494103078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6811","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6813","nonce":2494103078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6821","nonce":2494103078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6817","nonce":2494103078}]},"public_addr":"172.21.15.6:6809/2494103078","cluster_addr":"172.21.15.6:6813/2494103078","heartbeat_back_addr":"172.21.15.6:6821/2494103078","heartbeat_front_addr":"172.21.15.6:6817/2494103078","state":["exists","up"]},{"osd":2,"uuid":"492331b9-4ee9-40b6-bacd-85d44cd6c45a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6812","nonce":1500066681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6816","nonce":1500066681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6824","nonce":1500066681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6820","nonce":1500066681}]},"public_addr":"172.21.15.6:6812/1500066681","cluster_addr":"172.21.15.6:6816/1500066681","heartbeat_back_addr":"172.21.15.6:6824/1500066681","heartbeat_front_addr":"172.21.15.6:6820/1500066681","state":["exists","up"]},{"osd":3,"uuid":"21bb2efe-9ba7-4bdc-bc31-180cd1102e21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6801","nonce":1558806123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6803","nonce":1558806123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6807","nonce":1558806123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6805","nonce":1558806123}]},"public_addr":"172.21.15.6:6801/1558806123","cluster_addr":"172.21.15.6:6803/1558806123","heartbeat_back_addr":"172.21.15.6:6807/1558806123","heartbeat_front_addr":"172.21.15.6:6805/1558806123","state":["exists","up"]},{"osd":4,"uuid":"6e2c4b71-19d9-4737-83d9-558705d8644a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6801","nonce":1526135780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6803","nonce":1526135780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6807","nonce":1526135780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6805","nonce":1526135780}]},"public_addr":"172.21.15.146:6801/1526135780","cluster_addr":"172.21.15.146:6803/1526135780","heartbeat_back_addr":"172.21.15.146:6807/1526135780","heartbeat_front_addr":"172.21.15.146:6805/1526135780","state":["exists","up"]},{"osd":5,"uuid":"fb69c525-f1be-4b7d-ab32-6c29b0d6fddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6809","nonce":460262507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6811","nonce":460262507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6815","nonce":460262507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6813","nonce":460262507}]},"public_addr":"172.21.15.146:6809/460262507","cluster_addr":"172.21.15.146:6811/460262507","heartbeat_back_addr":"172.21.15.146:6815/460262507","heartbeat_front_addr":"172.21.15.146:6813/460262507","state":["exists","up"]},{"osd":6,"uuid":"c32e7fed-370d-4401-8ce0-6a4335b0d168","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6823","nonce":1392897493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6827","nonce":1392897493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6831","nonce":1392897493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6829","nonce":1392897493}]},"public_addr":"172.21.15.146:6823/1392897493","cluster_addr":"172.21.15.146:6827/1392897493","heartbeat_back_addr":"172.21.15.146:6831/1392897493","heartbeat_front_addr":"172.21.15.146:6829/1392897493","state":["exists","up"]},{"osd":7,"uuid":"75a79b44-fa5f-426d-a7d3-71f21f78835c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6817","nonce":3268323156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6819","nonce":3268323156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6825","nonce":3268323156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6822","nonce":3268323156}]},"public_addr":"172.21.15.146:6817/3268323156","cluster_addr":"172.21.15.146:6819/3268323156","heartbeat_back_addr":"172.21.15.146:6825/3268323156","heartbeat_front_addr":"172.21.15.146:6822/3268323156","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.310437+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.295850+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.294674+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.526855+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.300023+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.595160+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.346046+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.558741+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T09:25:36.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.545+0000 7ff8e9e00640 1 -- 172.21.15.6:0/3083029587 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff8c006b4d0 msgr2=0x7ff8c006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:36.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.545+0000 7ff8e9e00640 1 --2- 172.21.15.6:0/3083029587 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff8c006b4d0 0x7ff8c006d990 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7ff8cc006fd0 tx=0x7ff8cc008040 comp rx=0 tx=0).stop 2024-03-04T09:25:36.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.545+0000 7ff8e9e00640 1 -- 172.21.15.6:0/3083029587 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff8e40af1a0 msgr2=0x7ff8e415ddd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:36.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.545+0000 7ff8e9e00640 1 --2- 172.21.15.6:0/3083029587 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff8e40af1a0 0x7ff8e415ddd0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7ff8d4000f80 tx=0x7ff8d4004a40 comp rx=0 tx=0).stop 2024-03-04T09:25:36.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.546+0000 7ff8e9e00640 1 -- 172.21.15.6:0/3083029587 shutdown_connections 2024-03-04T09:25:36.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.546+0000 7ff8e9e00640 1 --2- 172.21.15.6:0/3083029587 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff8c006b4d0 0x7ff8c006d990 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:36.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.546+0000 7ff8e9e00640 1 --2- 172.21.15.6:0/3083029587 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff8e40cdc20 0x7ff8e406cb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:36.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.546+0000 7ff8e9e00640 1 --2- 172.21.15.6:0/3083029587 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff8e40af580 0x7ff8e409c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:36.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.546+0000 7ff8e9e00640 1 --2- 172.21.15.6:0/3083029587 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff8e40af1a0 0x7ff8e415ddd0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:36.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.546+0000 7ff8e9e00640 1 -- 172.21.15.6:0/3083029587 >> 172.21.15.6:0/3083029587 conn(0x7ff8e406afb0 msgr2=0x7ff8e4071620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:36.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.546+0000 7ff8e9e00640 1 -- 172.21.15.6:0/3083029587 shutdown_connections 2024-03-04T09:25:36.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.546+0000 7ff8e9e00640 1 -- 172.21.15.6:0/3083029587 wait complete. 2024-03-04T09:25:36.563 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:25:36.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.788+0000 7fabee200640 1 Processor -- start 2024-03-04T09:25:36.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.789+0000 7fabee200640 1 -- start start 2024-03-04T09:25:36.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.790+0000 7fabee200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fabe8153170 0x7fabe80b4b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:36.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.790+0000 7fabee200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabe80af340 con 0x7fabe815d300 2024-03-04T09:25:36.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.790+0000 7fabee200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabe80af5c0 con 0x7fabe80b5d80 2024-03-04T09:25:36.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.790+0000 7fabee200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabe80af8a0 con 0x7fabe8153170 2024-03-04T09:25:36.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.791+0000 7fabe6a00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fabe815d300 0x7fabe80b5670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:36876/0 (socket says 172.21.15.6:36876) 2024-03-04T09:25:36.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.791+0000 7fabe6a00640 1 -- 172.21.15.6:0/286139586 learned_addr learned my addr 172.21.15.6:0/286139586 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:25:36.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.791+0000 7fabe7400640 1 --2- 172.21.15.6:0/286139586 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fabe8153170 0x7fabe80b4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:36.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.791+0000 7fabe6000640 1 -- 172.21.15.6:0/286139586 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1731203616 0 0) 0x7fabd0001110 con 0x7fabe815d300 2024-03-04T09:25:36.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.791+0000 7fabe6000640 1 -- 172.21.15.6:0/286139586 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabcc003520 con 0x7fabe815d300 2024-03-04T09:25:36.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.792+0000 7fabe6000640 1 -- 172.21.15.6:0/286139586 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 462920027 0 0) 0x7fabcc003520 con 0x7fabe815d300 2024-03-04T09:25:36.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.792+0000 7fabe6000640 1 -- 172.21.15.6:0/286139586 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fabe8153170 msgr2=0x7fabe80b4b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:36.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.792+0000 7fabe6000640 1 --2- 172.21.15.6:0/286139586 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fabe8153170 0x7fabe80b4b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:36.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.792+0000 7fabe6000640 1 -- 172.21.15.6:0/286139586 >> v1:172.21.15.146:6789/0 conn(0x7fabe80b5d80 legacy=0x7fabe80b0660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:36.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.792+0000 7fabe6000640 1 -- 172.21.15.6:0/286139586 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabe80afb80 con 0x7fabe815d300 2024-03-04T09:25:36.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.792+0000 7fabe6000640 1 -- 172.21.15.6:0/286139586 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7fabd0001840 con 0x7fabe815d300 2024-03-04T09:25:36.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.792+0000 7fabe6000640 1 -- 172.21.15.6:0/286139586 >> v1:172.21.15.6:6789/0 conn(0x7fabe815d300 legacy=0x7fabe80b5670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:36.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.792+0000 7fabe6000640 1 --2- 172.21.15.6:0/286139586 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fabcc003a50 0x7fabcc003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:36.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.792+0000 7fabe6000640 1 --2- 172.21.15.6:0/286139586 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fabcc004800 0x7fabcc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:36.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.792+0000 7fabe6000640 1 --2- 172.21.15.6:0/286139586 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fabe8153170 0x7fabcc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:36.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.792+0000 7fabe6000640 1 -- 172.21.15.6:0/286139586 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabe80af8a0 con 0x7fabcc003a50 2024-03-04T09:25:36.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.792+0000 7fabe6000640 1 -- 172.21.15.6:0/286139586 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabcc00bac0 con 0x7fabcc004800 2024-03-04T09:25:36.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.792+0000 7fabe6000640 1 -- 172.21.15.6:0/286139586 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabcc00bcc0 con 0x7fabe8153170 2024-03-04T09:25:36.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.792+0000 7fabe7400640 1 --2- 172.21.15.6:0/286139586 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fabcc003a50 0x7fabcc003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:36.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.792+0000 7fabe6a00640 1 --2- 172.21.15.6:0/286139586 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fabcc004800 0x7fabcc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:36.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.792+0000 7fabe7e00640 1 --2- 172.21.15.6:0/286139586 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fabe8153170 0x7fabcc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:36.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.792+0000 7fabe7400640 1 -- 172.21.15.6:0/286139586 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fabe8153170 msgr2=0x7fabcc0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:36.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.792+0000 7fabe7400640 1 --2- 172.21.15.6:0/286139586 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fabe8153170 0x7fabcc0051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:36.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.792+0000 7fabe7400640 1 -- 172.21.15.6:0/286139586 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fabcc004800 msgr2=0x7fabcc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:36.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.792+0000 7fabe7400640 1 --2- 172.21.15.6:0/286139586 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fabcc004800 0x7fabcc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:36.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.792+0000 7fabe7400640 1 -- 172.21.15.6:0/286139586 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabcc003520 con 0x7fabcc003a50 2024-03-04T09:25:36.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.792+0000 7fabe6a00640 1 --2- 172.21.15.6:0/286139586 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fabcc004800 0x7fabcc004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:25:36.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.793+0000 7fabe7400640 1 --2- 172.21.15.6:0/286139586 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fabcc003a50 0x7fabcc003e70 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fabd8009ac0 tx=0x7fabcc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1da679f4c289483 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:36.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.794+0000 7fabe6000640 1 -- 172.21.15.6:0/286139586 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabd8039070 con 0x7fabcc003a50 2024-03-04T09:25:36.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.794+0000 7fabe6000640 1 -- 172.21.15.6:0/286139586 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabd802c6b0 con 0x7fabcc003a50 2024-03-04T09:25:36.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.794+0000 7fabe6000640 1 -- 172.21.15.6:0/286139586 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabd8009690 con 0x7fabcc003a50 2024-03-04T09:25:36.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.794+0000 7fabee200640 1 -- 172.21.15.6:0/286139586 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fabcc003a50 msgr2=0x7fabcc003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:36.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.794+0000 7fabee200640 1 --2- 172.21.15.6:0/286139586 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fabcc003a50 0x7fabcc003e70 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fabd8009ac0 tx=0x7fabcc005830 comp rx=0 tx=0).stop 2024-03-04T09:25:36.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.794+0000 7fabe7e00640 1 -- 172.21.15.6:0/286139586 reap_dead start 2024-03-04T09:25:36.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.795+0000 7fabee200640 1 -- 172.21.15.6:0/286139586 shutdown_connections 2024-03-04T09:25:36.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.795+0000 7fabee200640 1 -- 172.21.15.6:0/286139586 >> 172.21.15.6:0/286139586 conn(0x7fabe809aa10 msgr2=0x7fabe809ae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:36.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.795+0000 7fabee200640 1 -- 172.21.15.6:0/286139586 shutdown_connections 2024-03-04T09:25:36.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.795+0000 7fabee200640 1 -- 172.21.15.6:0/286139586 wait complete. 2024-03-04T09:25:36.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.796+0000 7fabee200640 1 Processor -- start 2024-03-04T09:25:36.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.797+0000 7fabee200640 1 -- start start 2024-03-04T09:25:36.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.797+0000 7fabee200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fabcc003a50 0x7fabe8160460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:36.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.797+0000 7fabee200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fabe8153550 0x7fabe8154a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:36.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.797+0000 7fabee200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fabe8155910 0x7fabe81442d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:36.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.797+0000 7fabee200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabe806ee90 con 0x7fabe8153550 2024-03-04T09:25:36.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.797+0000 7fabee200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabe80b0140 con 0x7fabcc003a50 2024-03-04T09:25:36.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.797+0000 7fabee200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabe806f010 con 0x7fabe8155910 2024-03-04T09:25:36.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.798+0000 7fabe6a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fabe8153550 0x7fabe8154a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:36.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.798+0000 7fabe6a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fabe8153550 0x7fabe8154a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45892/0 (socket says 172.21.15.6:45892) 2024-03-04T09:25:36.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.798+0000 7fabe6a00640 1 -- 172.21.15.6:0/1585280139 learned_addr learned my addr 172.21.15.6:0/1585280139 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:25:36.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.798+0000 7fabe7e00640 1 --2- 172.21.15.6:0/1585280139 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fabe8155910 0x7fabe81442d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:36.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.798+0000 7fabe7400640 1 --2- 172.21.15.6:0/1585280139 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fabcc003a50 0x7fabe8160460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:36.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.798+0000 7fabe6a00640 1 -- 172.21.15.6:0/1585280139 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fabe8155910 msgr2=0x7fabe81442d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:36.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.798+0000 7fabe6a00640 1 --2- 172.21.15.6:0/1585280139 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fabe8155910 0x7fabe81442d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:36.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.798+0000 7fabe6a00640 1 -- 172.21.15.6:0/1585280139 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fabcc003a50 msgr2=0x7fabe8160460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:36.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.798+0000 7fabe6a00640 1 --2- 172.21.15.6:0/1585280139 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fabcc003a50 0x7fabe8160460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:36.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.798+0000 7fabe6a00640 1 -- 172.21.15.6:0/1585280139 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabe81539e0 con 0x7fabe8153550 2024-03-04T09:25:36.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.798+0000 7fabe6a00640 1 --2- 172.21.15.6:0/1585280139 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fabe8153550 0x7fabe8154a80 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fabd000d8e0 tx=0x7fabd000dda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:36.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.799+0000 7fabdf400640 1 -- 172.21.15.6:0/1585280139 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabd0019070 con 0x7fabe8153550 2024-03-04T09:25:36.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.799+0000 7fabdf400640 1 -- 172.21.15.6:0/1585280139 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabd00074e0 con 0x7fabe8153550 2024-03-04T09:25:36.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.799+0000 7fabdf400640 1 -- 172.21.15.6:0/1585280139 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabd00049e0 con 0x7fabe8153550 2024-03-04T09:25:36.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.799+0000 7fabee200640 1 -- 172.21.15.6:0/1585280139 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabe8153cc0 con 0x7fabe8153550 2024-03-04T09:25:36.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.799+0000 7fabee200640 1 -- 172.21.15.6:0/1585280139 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabe8154280 con 0x7fabe8153550 2024-03-04T09:25:36.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.800+0000 7fabee200640 1 -- 172.21.15.6:0/1585280139 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabe804f3a0 con 0x7fabe8153550 2024-03-04T09:25:36.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.801+0000 7fabdf400640 1 -- 172.21.15.6:0/1585280139 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fabd001c070 con 0x7fabe8153550 2024-03-04T09:25:36.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.802+0000 7fabdf400640 1 --2- 172.21.15.6:0/1585280139 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fabc406b520 0x7fabc406d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:36.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.802+0000 7fabdf400640 1 -- 172.21.15.6:0/1585280139 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fabd0089d40 con 0x7fabe8153550 2024-03-04T09:25:36.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.802+0000 7fabe7400640 1 --2- 172.21.15.6:0/1585280139 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fabc406b520 0x7fabc406d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:36.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.804+0000 7fabe7400640 1 --2- 172.21.15.6:0/1585280139 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fabc406b520 0x7fabc406d9e0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fabd8036000 tx=0x7fabd8030000 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:36.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:36.806+0000 7fabdf400640 1 -- 172.21.15.6:0/1585280139 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fabe80af340 con 0x7fabe8153550 2024-03-04T09:25:37.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.045+0000 7fabee200640 1 -- 172.21.15.6:0/1585280139 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fabe8074b20 con 0x7fabe8153550 2024-03-04T09:25:37.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.046+0000 7fabdf400640 1 -- 172.21.15.6:0/1585280139 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4971 (secure 0 0 0) 0x7fabd00584c0 con 0x7fabe8153550 2024-03-04T09:25:37.043 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:25:37.043 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4248,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/2702058134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2702058134},{"type":"v1","addr":"172.21.15.6:6833","nonce":2702058134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4263,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/3866998852","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":3866998852},{"type":"v1","addr":"172.21.15.6:6835","nonce":3866998852}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4285,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/3523455455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":3523455455},{"type":"v1","addr":"172.21.15.146:6837","nonce":3523455455}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:25:18.912808+0000","modified":"2024-03-04T09:25:22.016493+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4288},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4288":{"gid":4288,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.146:6836/1946537763","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":1946537763},{"type":"v1","addr":"172.21.15.146:6836","nonce":1946537763}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4288,"qdb_cluster":[4288]},"id":1}]} 2024-03-04T09:25:37.043 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 7 2024-03-04T09:25:37.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.048+0000 7fabee200640 1 -- 172.21.15.6:0/1585280139 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fabc406b520 msgr2=0x7fabc406d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:37.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.048+0000 7fabee200640 1 --2- 172.21.15.6:0/1585280139 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fabc406b520 0x7fabc406d9e0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fabd8036000 tx=0x7fabd8030000 comp rx=0 tx=0).stop 2024-03-04T09:25:37.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.048+0000 7fabee200640 1 -- 172.21.15.6:0/1585280139 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fabe8153550 msgr2=0x7fabe8154a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:37.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.048+0000 7fabee200640 1 --2- 172.21.15.6:0/1585280139 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fabe8153550 0x7fabe8154a80 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fabd000d8e0 tx=0x7fabd000dda0 comp rx=0 tx=0).stop 2024-03-04T09:25:37.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.048+0000 7fabee200640 1 -- 172.21.15.6:0/1585280139 shutdown_connections 2024-03-04T09:25:37.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.048+0000 7fabee200640 1 --2- 172.21.15.6:0/1585280139 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fabc406b520 0x7fabc406d9e0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:37.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.048+0000 7fabee200640 1 --2- 172.21.15.6:0/1585280139 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fabe8155910 0x7fabe81442d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:37.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.048+0000 7fabee200640 1 --2- 172.21.15.6:0/1585280139 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fabe8153550 0x7fabe8154a80 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:37.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.048+0000 7fabee200640 1 --2- 172.21.15.6:0/1585280139 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fabcc003a50 0x7fabe8160460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:37.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.048+0000 7fabee200640 1 -- 172.21.15.6:0/1585280139 >> 172.21.15.6:0/1585280139 conn(0x7fabe809aa10 msgr2=0x7fabe80b5080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:37.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.049+0000 7fabee200640 1 -- 172.21.15.6:0/1585280139 shutdown_connections 2024-03-04T09:25:37.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.049+0000 7fabee200640 1 -- 172.21.15.6:0/1585280139 wait complete. 2024-03-04T09:25:37.467 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 7, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:25:18.912808+0000', 'modified': '2024-03-04T09:25:22.016493+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4288}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4288': {'gid': 4288, 'name': 'd', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.146:6836/1946537763', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 1946537763}, {'type': 'v1', 'addr': '172.21.15.146:6836', 'nonce': 1946537763}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4288, 'qdb_cluster': [4288]} 2024-03-04T09:25:37.467 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-04T09:25:37.467 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-03-04T09:25:37.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.698+0000 7fb0abe00640 1 Processor -- start 2024-03-04T09:25:37.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.698+0000 7fb0abe00640 1 -- start start 2024-03-04T09:25:37.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.699+0000 7fb0abe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fb0ac15e2a0 0x7fb0ac0cef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:37.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.699+0000 7fb0abe00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fb0ac0cf460 0x7fb0ac0b1330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:37.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.699+0000 7fb0abe00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0ac0d1120 con 0x7fb0ac15dec0 2024-03-04T09:25:37.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.699+0000 7fb0abe00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0ac0b0ea0 con 0x7fb0ac15e2a0 2024-03-04T09:25:37.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.699+0000 7fb0abe00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0ac0d1400 con 0x7fb0ac0cf460 2024-03-04T09:25:37.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.700+0000 7fb0aaa00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fb0ac15dec0 0x7fb0ac0ce810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:36888/0 (socket says 172.21.15.6:36888) 2024-03-04T09:25:37.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.700+0000 7fb0aa000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fb0ac15e2a0 0x7fb0ac0cef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:37.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.700+0000 7fb0aaa00640 1 -- 172.21.15.6:0/3303825331 learned_addr learned my addr 172.21.15.6:0/3303825331 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:25:37.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.700+0000 7fb0ab400640 1 --2- 172.21.15.6:0/3303825331 >> v2:172.21.15.146:3300/0 conn(0x7fb0ac0cf460 0x7fb0ac0b1330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:37.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.700+0000 7fb0a9600640 1 -- 172.21.15.6:0/3303825331 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 43788352 0 0) 0x7fb0ac0d1120 con 0x7fb0ac15dec0 2024-03-04T09:25:37.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.700+0000 7fb0a9600640 1 -- 172.21.15.6:0/3303825331 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb08c003520 con 0x7fb0ac15dec0 2024-03-04T09:25:37.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.700+0000 7fb0aa000640 1 -- 172.21.15.6:0/3303825331 >> v2:172.21.15.146:3300/0 conn(0x7fb0ac0cf460 msgr2=0x7fb0ac0b1330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:37.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.700+0000 7fb0aa000640 1 --2- 172.21.15.6:0/3303825331 >> v2:172.21.15.146:3300/0 conn(0x7fb0ac0cf460 0x7fb0ac0b1330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:37.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.700+0000 7fb0aa000640 1 -- 172.21.15.6:0/3303825331 >> v1:172.21.15.6:6789/0 conn(0x7fb0ac15dec0 legacy=0x7fb0ac0ce810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:37.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.700+0000 7fb0aa000640 1 -- 172.21.15.6:0/3303825331 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0ac0b0b30 con 0x7fb0ac15e2a0 2024-03-04T09:25:37.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.702+0000 7fb0aa000640 1 --2- 172.21.15.6:0/3303825331 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb0ac15e2a0 0x7fb0ac0cef20 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fb0ac08c570 tx=0x7fb09402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bdaf4767865a8c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:37.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.702+0000 7fb0a9600640 1 -- 172.21.15.6:0/3303825331 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb094039070 con 0x7fb0ac15e2a0 2024-03-04T09:25:37.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.702+0000 7fb0a9600640 1 -- 172.21.15.6:0/3303825331 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb09402cc80 con 0x7fb0ac15e2a0 2024-03-04T09:25:37.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.702+0000 7fb0a9600640 1 -- 172.21.15.6:0/3303825331 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb094034640 con 0x7fb0ac15e2a0 2024-03-04T09:25:37.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.703+0000 7fb0abe00640 1 -- 172.21.15.6:0/3303825331 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb0ac15e2a0 msgr2=0x7fb0ac0cef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:37.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.703+0000 7fb0abe00640 1 --2- 172.21.15.6:0/3303825331 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb0ac15e2a0 0x7fb0ac0cef20 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fb0ac08c570 tx=0x7fb09402bed0 comp rx=0 tx=0).stop 2024-03-04T09:25:37.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.703+0000 7fb0abe00640 1 -- 172.21.15.6:0/3303825331 shutdown_connections 2024-03-04T09:25:37.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.703+0000 7fb0abe00640 1 --2- 172.21.15.6:0/3303825331 >> v2:172.21.15.146:3300/0 conn(0x7fb0ac0cf460 0x7fb0ac0b1330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:37.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.703+0000 7fb0abe00640 1 --2- 172.21.15.6:0/3303825331 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb0ac15e2a0 0x7fb0ac0cef20 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:37.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.703+0000 7fb0abe00640 1 -- 172.21.15.6:0/3303825331 >> 172.21.15.6:0/3303825331 conn(0x7fb0ac0d3d40 msgr2=0x7fb0ac0d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:37.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.703+0000 7fb0abe00640 1 -- 172.21.15.6:0/3303825331 shutdown_connections 2024-03-04T09:25:37.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.703+0000 7fb0abe00640 1 -- 172.21.15.6:0/3303825331 wait complete. 2024-03-04T09:25:37.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.705+0000 7fb0abe00640 1 Processor -- start 2024-03-04T09:25:37.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.705+0000 7fb0abe00640 1 -- start start 2024-03-04T09:25:37.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.706+0000 7fb0abe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb0ac0cf460 0x7fb0ac1f2f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:37.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.706+0000 7fb0abe00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb0ac15dec0 0x7fb0ac06b960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:37.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.706+0000 7fb0abe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb0ac15e2a0 0x7fb0ac06bea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:37.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.706+0000 7fb0abe00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0ac0d0ec0 con 0x7fb0ac0cf460 2024-03-04T09:25:37.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.706+0000 7fb0abe00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0ac0d0d40 con 0x7fb0ac15e2a0 2024-03-04T09:25:37.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.706+0000 7fb0abe00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0ac0b1870 con 0x7fb0ac15dec0 2024-03-04T09:25:37.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.706+0000 7fb0aaa00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb0ac0cf460 0x7fb0ac1f2f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:37.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.706+0000 7fb0aaa00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb0ac0cf460 0x7fb0ac1f2f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45910/0 (socket says 172.21.15.6:45910) 2024-03-04T09:25:37.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.706+0000 7fb0aaa00640 1 -- 172.21.15.6:0/3616246525 learned_addr learned my addr 172.21.15.6:0/3616246525 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:25:37.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.706+0000 7fb0aa000640 1 --2- 172.21.15.6:0/3616246525 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb0ac15dec0 0x7fb0ac06b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:37.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.706+0000 7fb0ab400640 1 --2- 172.21.15.6:0/3616246525 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb0ac15e2a0 0x7fb0ac06bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:37.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.707+0000 7fb0aaa00640 1 -- 172.21.15.6:0/3616246525 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb0ac15dec0 msgr2=0x7fb0ac06b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:37.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.707+0000 7fb0aaa00640 1 --2- 172.21.15.6:0/3616246525 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb0ac15dec0 0x7fb0ac06b960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:37.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.707+0000 7fb0aaa00640 1 -- 172.21.15.6:0/3616246525 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb0ac15e2a0 msgr2=0x7fb0ac06bea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:37.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.707+0000 7fb0aaa00640 1 --2- 172.21.15.6:0/3616246525 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb0ac15e2a0 0x7fb0ac06bea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:37.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.707+0000 7fb0aaa00640 1 -- 172.21.15.6:0/3616246525 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0ac0c8570 con 0x7fb0ac0cf460 2024-03-04T09:25:37.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.707+0000 7fb0aaa00640 1 --2- 172.21.15.6:0/3616246525 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb0ac0cf460 0x7fb0ac1f2f90 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fb09c00c460 tx=0x7fb09c00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:37.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.707+0000 7fb0a3400640 1 -- 172.21.15.6:0/3616246525 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb09c017070 con 0x7fb0ac0cf460 2024-03-04T09:25:37.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.707+0000 7fb0a3400640 1 -- 172.21.15.6:0/3616246525 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb09c005ce0 con 0x7fb0ac0cf460 2024-03-04T09:25:37.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.707+0000 7fb0a3400640 1 -- 172.21.15.6:0/3616246525 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb09c007290 con 0x7fb0ac0cf460 2024-03-04T09:25:37.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.707+0000 7fb0abe00640 1 -- 172.21.15.6:0/3616246525 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0ac0c79f0 con 0x7fb0ac0cf460 2024-03-04T09:25:37.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.708+0000 7fb0abe00640 1 -- 172.21.15.6:0/3616246525 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0ac0c7fb0 con 0x7fb0ac0cf460 2024-03-04T09:25:37.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.709+0000 7fb0a3400640 1 -- 172.21.15.6:0/3616246525 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb09c01a070 con 0x7fb0ac0cf460 2024-03-04T09:25:37.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.710+0000 7fb0a3400640 1 --2- 172.21.15.6:0/3616246525 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb08806b4d0 0x7fb08806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:37.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.710+0000 7fb0a3400640 1 -- 172.21.15.6:0/3616246525 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb09c0894f0 con 0x7fb0ac0cf460 2024-03-04T09:25:37.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.711+0000 7fb0aa000640 1 --2- 172.21.15.6:0/3616246525 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb08806b4d0 0x7fb08806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:37.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.711+0000 7fb0b4a83740 1 Processor -- start 2024-03-04T09:25:37.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.711+0000 7fb0b4a83740 1 -- start start 2024-03-04T09:25:37.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.712+0000 7fb0aa000640 1 --2- 172.21.15.6:0/3616246525 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb08806b4d0 0x7fb08806d990 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fb094036000 tx=0x7fb094030000 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:37.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.712+0000 7fb0b4a83740 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x563d9be92990 0x563d9be65930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:37.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.712+0000 7fb0b4a83740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x563d9be92d70 0x563d9be95720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:37.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.712+0000 7fb0b4a83740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563d9be65e70 0x563d9be95c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:37.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.712+0000 7fb0b4a83740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d9bdcf600 con 0x563d9be65e70 2024-03-04T09:25:37.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.712+0000 7fb0b4a83740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d9bd78930 con 0x563d9be92d70 2024-03-04T09:25:37.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.712+0000 7fb0b4a83740 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x563d9bdcb6a0 con 0x563d9be92990 2024-03-04T09:25:37.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.713+0000 7fb0ab400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563d9be65e70 0x563d9be95c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:37.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.713+0000 7fb0aa000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x563d9be92d70 0x563d9be95720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:37.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.713+0000 7fb0ab400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563d9be65e70 0x563d9be95c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45920/0 (socket says 172.21.15.6:45920) 2024-03-04T09:25:37.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.713+0000 7fb0ab400640 1 -- 172.21.15.6:0/4078390335 learned_addr learned my addr 172.21.15.6:0/4078390335 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:25:37.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.713+0000 7fb0aaa00640 1 --2- 172.21.15.6:0/4078390335 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x563d9be92990 0x563d9be65930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:37.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.713+0000 7fb0ab400640 1 -- 172.21.15.6:0/4078390335 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x563d9be92990 msgr2=0x563d9be65930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:37.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.713+0000 7fb0ab400640 1 --2- 172.21.15.6:0/4078390335 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x563d9be92990 0x563d9be65930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:37.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.713+0000 7fb0ab400640 1 -- 172.21.15.6:0/4078390335 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x563d9be92d70 msgr2=0x563d9be95720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:37.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.713+0000 7fb0ab400640 1 --2- 172.21.15.6:0/4078390335 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x563d9be92d70 0x563d9be95720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:37.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.713+0000 7fb0ab400640 1 -- 172.21.15.6:0/4078390335 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563d9bae42d0 con 0x563d9be65e70 2024-03-04T09:25:37.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.713+0000 7fb0ab400640 1 --2- 172.21.15.6:0/4078390335 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563d9be65e70 0x563d9be95c60 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fb0a4009b00 tx=0x7fb0a4009190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7f0afecd59cf1f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:37.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.714+0000 7fb09b400640 1 -- 172.21.15.6:0/4078390335 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0a4011c20 con 0x563d9be65e70 2024-03-04T09:25:37.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.714+0000 7fb09b400640 1 -- 172.21.15.6:0/4078390335 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0a4011dc0 con 0x563d9be65e70 2024-03-04T09:25:37.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.714+0000 7fb09b400640 1 -- 172.21.15.6:0/4078390335 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0a4004270 con 0x563d9be65e70 2024-03-04T09:25:37.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.714+0000 7fb0b4a83740 1 -- 172.21.15.6:0/4078390335 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563d9be65e70 msgr2=0x563d9be95c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:37.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.714+0000 7fb0b4a83740 1 --2- 172.21.15.6:0/4078390335 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563d9be65e70 0x563d9be95c60 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fb0a4009b00 tx=0x7fb0a4009190 comp rx=0 tx=0).stop 2024-03-04T09:25:37.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.714+0000 7fb0b4a83740 1 -- 172.21.15.6:0/4078390335 shutdown_connections 2024-03-04T09:25:37.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.714+0000 7fb0b4a83740 1 --2- 172.21.15.6:0/4078390335 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563d9be65e70 0x563d9be95c60 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:37.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.714+0000 7fb0b4a83740 1 --2- 172.21.15.6:0/4078390335 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x563d9be92d70 0x563d9be95720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:37.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.714+0000 7fb0b4a83740 1 --2- 172.21.15.6:0/4078390335 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x563d9be92990 0x563d9be65930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:37.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.714+0000 7fb0b4a83740 1 -- 172.21.15.6:0/4078390335 >> 172.21.15.6:0/4078390335 conn(0x563d9be67ba0 msgr2=0x563d9beb0720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:37.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.714+0000 7fb0b4a83740 1 -- 172.21.15.6:0/4078390335 shutdown_connections 2024-03-04T09:25:37.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.715+0000 7fb0b4a83740 1 -- 172.21.15.6:0/4078390335 wait complete. 2024-03-04T09:25:37.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.715+0000 7fb0b4a83740 1 -- start start 2024-03-04T09:25:37.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.715+0000 7fb0b4a83740 1 Processor -- start 2024-03-04T09:25:37.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.717+0000 7fb0b4a83740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x563d9be92990 0x563d9be97a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:37.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.717+0000 7fb0b4a83740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563d9be92d70 0x563d9be8ece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:37.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.717+0000 7fb0b4a83740 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x563d9bec3c30 0x563d9be8f220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:37.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.717+0000 7fb0b4a83740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d9be6a140 con 0x563d9be92d70 2024-03-04T09:25:37.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.717+0000 7fb0b4a83740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d9bd8bd90 con 0x563d9be92990 2024-03-04T09:25:37.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.717+0000 7fb0b4a83740 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x563d9be69530 con 0x563d9bec3c30 2024-03-04T09:25:37.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.717+0000 7fb0b4a83740 10 client.0 _ll_register_callbacks cb 0x563d9bdc7350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T09:25:37.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.717+0000 7fb0aa000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563d9be92d70 0x563d9be8ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:37.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.717+0000 7fb0aa000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563d9be92d70 0x563d9be8ece0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45922/0 (socket says 172.21.15.6:45922) 2024-03-04T09:25:37.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.717+0000 7fb0aa000640 1 -- 172.21.15.6:0/3920332855 learned_addr learned my addr 172.21.15.6:0/3920332855 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:25:37.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.717+0000 7fb0aaa00640 1 --2- 172.21.15.6:0/3920332855 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x563d9be92990 0x563d9be97a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:37.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.717+0000 7fb0ab400640 1 --2- 172.21.15.6:0/3920332855 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x563d9bec3c30 0x563d9be8f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:37.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.717+0000 7fb0aa000640 1 -- 172.21.15.6:0/3920332855 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x563d9bec3c30 msgr2=0x563d9be8f220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:37.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.717+0000 7fb0aa000640 1 --2- 172.21.15.6:0/3920332855 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x563d9bec3c30 0x563d9be8f220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:37.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.717+0000 7fb0aa000640 1 -- 172.21.15.6:0/3920332855 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x563d9be92990 msgr2=0x563d9be97a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:37.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.717+0000 7fb0aa000640 1 --2- 172.21.15.6:0/3920332855 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x563d9be92990 0x563d9be97a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:37.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.717+0000 7fb0aa000640 1 -- 172.21.15.6:0/3920332855 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563d9bbae6b0 con 0x563d9be92d70 2024-03-04T09:25:37.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.718+0000 7fb0aa000640 1 --2- 172.21.15.6:0/3920332855 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563d9be92d70 0x563d9be8ece0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fb09402c4e0 tx=0x7fb094004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37ff337ec71f9c73 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:37.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.718+0000 7fb093e00640 10 client.0 ms_handle_connect on v2:172.21.15.6:3300/0 2024-03-04T09:25:37.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.718+0000 7fb093e00640 1 -- 172.21.15.6:0/3920332855 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb094050020 con 0x563d9be92d70 2024-03-04T09:25:37.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.718+0000 7fb093e00640 1 -- 172.21.15.6:0/3920332855 <== mon.0 v2:172.21.15.6:3300/0 2 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb09404c660 con 0x563d9be92d70 2024-03-04T09:25:37.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.718+0000 7fb0b4a83740 1 -- 172.21.15.6:0/3920332855 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb0a4011dc0 con 0x563d9be92d70 2024-03-04T09:25:37.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.718+0000 7fb093e00640 1 -- 172.21.15.6:0/3920332855 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_get_version_reply(handle=1 version=7) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb09405a050 con 0x563d9be92d70 2024-03-04T09:25:37.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.718+0000 7fb0b4a83740 10 client.4608 fetch_fsmap learned FSMap version 7 2024-03-04T09:25:37.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.718+0000 7fb0b4a83740 1 -- 172.21.15.6:0/3920332855 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,fsmap=7,monmap=2+}) v3 -- 0x563d9be98fa0 con 0x563d9be92d70 2024-03-04T09:25:37.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.718+0000 7fb093e00640 1 -- 172.21.15.6:0/3920332855 <== mon.0 v2:172.21.15.6:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0a4011dc0 con 0x563d9be92d70 2024-03-04T09:25:37.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.719+0000 7fb093e00640 1 -- 172.21.15.6:0/3920332855 <== mon.0 v2:172.21.15.6:3300/0 5 ==== fsmap(e 7) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fb09405d020 con 0x563d9be92d70 2024-03-04T09:25:37.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.719+0000 7fb0b4a83740 10 client.4608 fetch_fsmap finished waiting for FSMap version 7 2024-03-04T09:25:37.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.719+0000 7fb0b4a83740 10 client.4608 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-04T09:25:37.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.719+0000 7fb0b4a83740 20 client.4608 populate_metadata read hostname 'smithi006' 2024-03-04T09:25:37.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.719+0000 7fb0b4a83740 1 --2- 172.21.15.6:0/3920332855 >> [v2:172.21.15.146:6835/1946537763,v1:172.21.15.146:6836/1946537763] conn(0x563d9be916b0 0x563d9bec2670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:37.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.719+0000 7fb0b4a83740 4 client.4608 mds_command: new command op to 4288 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T09:25:37.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.719+0000 7fb0b4a83740 1 -- 172.21.15.6:0/3920332855 --> [v2:172.21.15.146:6835/1946537763,v1:172.21.15.146:6836/1946537763] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563d9bddcb40 con 0x563d9be916b0 2024-03-04T09:25:37.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.719+0000 7fb0aaa00640 1 --2- 172.21.15.6:0/3920332855 >> [v2:172.21.15.146:6835/1946537763,v1:172.21.15.146:6836/1946537763] conn(0x563d9be916b0 0x563d9bec2670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:37.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.720+0000 7fb0aaa00640 1 --2- 172.21.15.6:0/3920332855 >> [v2:172.21.15.146:6835/1946537763,v1:172.21.15.146:6836/1946537763] conn(0x563d9be916b0 0x563d9bec2670 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=131d4393000a16c0 server_cookie=b8c20a3733542540 in_seq=0 out_seq=0 2024-03-04T09:25:37.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.720+0000 7fb093e00640 10 client.4608 ms_handle_connect on v2:172.21.15.146:6835/1946537763 2024-03-04T09:25:37.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.722+0000 7fb093e00640 1 -- 172.21.15.6:0/3920332855 <== mds.0 v2:172.21.15.146:6835/1946537763 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x563d9bddcb40 con 0x563d9be916b0 2024-03-04T09:25:37.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.722+0000 7fb093e00640 10 client.4608 handle_command_reply: tid=0 2024-03-04T09:25:37.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.722+0000 7fb0b4a83740 1 client.4608 shutdown 2024-03-04T09:25:37.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.723+0000 7fb0b4a83740 1 -- 172.21.15.6:0/3920332855 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563d9be92d70 msgr2=0x563d9be8ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:37.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.723+0000 7fb0b4a83740 1 --2- 172.21.15.6:0/3920332855 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563d9be92d70 0x563d9be8ece0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fb09402c4e0 tx=0x7fb094004ab0 comp rx=0 tx=0).stop 2024-03-04T09:25:37.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.723+0000 7fb0b4a83740 1 -- 172.21.15.6:0/3920332855 shutdown_connections 2024-03-04T09:25:37.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.723+0000 7fb0b4a83740 1 --2- 172.21.15.6:0/3920332855 >> [v2:172.21.15.146:6835/1946537763,v1:172.21.15.146:6836/1946537763] conn(0x563d9be916b0 0x563d9bec2670 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:37.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.723+0000 7fb0b4a83740 1 --2- 172.21.15.6:0/3920332855 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x563d9bec3c30 0x563d9be8f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:37.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.723+0000 7fb0b4a83740 1 --2- 172.21.15.6:0/3920332855 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563d9be92d70 0x563d9be8ece0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:37.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.723+0000 7fb0b4a83740 1 --2- 172.21.15.6:0/3920332855 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x563d9be92990 0x563d9be97a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:37.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.723+0000 7fb093e00640 0 client.4608 ms_handle_reset on v2:172.21.15.146:6835/1946537763 2024-03-04T09:25:37.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.723+0000 7fb0b4a83740 1 -- 172.21.15.6:0/3920332855 >> 172.21.15.6:0/3920332855 conn(0x563d9be65e70 msgr2=0x563d9be65050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:37.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.723+0000 7fb0b4a83740 1 -- 172.21.15.6:0/3920332855 shutdown_connections 2024-03-04T09:25:37.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.723+0000 7fb0b4a83740 1 -- 172.21.15.6:0/3920332855 wait complete. 2024-03-04T09:25:37.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.723+0000 7fb0b4a83740 20 client.4608 trim_cache size 0 max 16384 2024-03-04T09:25:37.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.741+0000 7fb0b4a83740 1 Processor -- start 2024-03-04T09:25:37.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.741+0000 7fb0b4a83740 1 -- start start 2024-03-04T09:25:37.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.741+0000 7fb0b4a83740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x563d9be92990 0x563d9be8eb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:37.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.741+0000 7fb0b4a83740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563d9be92d70 0x563d9be8f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:37.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.741+0000 7fb0b4a83740 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x563d9bec3c30 0x563d9be647c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:37.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.741+0000 7fb0b4a83740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0ac1fef50 con 0x563d9be92d70 2024-03-04T09:25:37.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.741+0000 7fb0b4a83740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0ac14c900 con 0x563d9be92990 2024-03-04T09:25:37.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.741+0000 7fb0b4a83740 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0ac14cf00 con 0x563d9bec3c30 2024-03-04T09:25:37.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.742+0000 7fb0aa000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563d9be92d70 0x563d9be8f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:37.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.742+0000 7fb0aa000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563d9be92d70 0x563d9be8f0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45930/0 (socket says 172.21.15.6:45930) 2024-03-04T09:25:37.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.742+0000 7fb0aa000640 1 -- 172.21.15.6:0/636107132 learned_addr learned my addr 172.21.15.6:0/636107132 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:25:37.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.742+0000 7fb0aaa00640 1 --2- 172.21.15.6:0/636107132 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x563d9be92990 0x563d9be8eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:37.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.742+0000 7fb0aa000640 1 -- 172.21.15.6:0/636107132 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x563d9bec3c30 msgr2=0x563d9be647c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:37.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.742+0000 7fb0aa000640 1 --2- 172.21.15.6:0/636107132 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x563d9bec3c30 0x563d9be647c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:37.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.742+0000 7fb0aa000640 1 -- 172.21.15.6:0/636107132 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x563d9be92990 msgr2=0x563d9be8eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:37.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.742+0000 7fb0aa000640 1 --2- 172.21.15.6:0/636107132 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x563d9be92990 0x563d9be8eb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:37.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.742+0000 7fb0aa000640 1 -- 172.21.15.6:0/636107132 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563d9be91da0 con 0x563d9be92d70 2024-03-04T09:25:37.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.742+0000 7fb0aaa00640 1 --2- 172.21.15.6:0/636107132 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x563d9be92990 0x563d9be8eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:25:37.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.743+0000 7fb0aa000640 1 --2- 172.21.15.6:0/636107132 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563d9be92d70 0x563d9be8f0c0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fb0940070f0 tx=0x7fb09404a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=527e0eedda157603 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:37.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.743+0000 7fb09aa00640 1 -- 172.21.15.6:0/636107132 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb094050070 con 0x563d9be92d70 2024-03-04T09:25:37.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.743+0000 7fb09aa00640 1 -- 172.21.15.6:0/636107132 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb09404cb40 con 0x563d9be92d70 2024-03-04T09:25:37.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.743+0000 7fb09aa00640 1 -- 172.21.15.6:0/636107132 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb094039040 con 0x563d9be92d70 2024-03-04T09:25:37.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.743+0000 7fb0b4a83740 1 -- 172.21.15.6:0/636107132 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563d9be92d70 msgr2=0x563d9be8f0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:37.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.743+0000 7fb0b4a83740 1 --2- 172.21.15.6:0/636107132 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563d9be92d70 0x563d9be8f0c0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fb0940070f0 tx=0x7fb09404a000 comp rx=0 tx=0).stop 2024-03-04T09:25:37.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.743+0000 7fb0b4a83740 1 -- 172.21.15.6:0/636107132 shutdown_connections 2024-03-04T09:25:37.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.743+0000 7fb0b4a83740 1 --2- 172.21.15.6:0/636107132 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x563d9bec3c30 0x563d9be647c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:37.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.743+0000 7fb0b4a83740 1 --2- 172.21.15.6:0/636107132 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563d9be92d70 0x563d9be8f0c0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:37.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.743+0000 7fb0b4a83740 1 --2- 172.21.15.6:0/636107132 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x563d9be92990 0x563d9be8eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:37.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.743+0000 7fb0b4a83740 1 -- 172.21.15.6:0/636107132 >> 172.21.15.6:0/636107132 conn(0x563d9be65e70 msgr2=0x563d9bec1050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:37.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.744+0000 7fb0b4a83740 1 -- 172.21.15.6:0/636107132 shutdown_connections 2024-03-04T09:25:37.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.744+0000 7fb0b4a83740 1 -- 172.21.15.6:0/636107132 wait complete. 2024-03-04T09:25:37.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.744+0000 7fb0b4a83740 1 -- start start 2024-03-04T09:25:37.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.744+0000 7fb0b4a83740 1 Processor -- start 2024-03-04T09:25:37.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.745+0000 7fb0b4a83740 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x563d9be92990 0x563d9be6aa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:37.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.745+0000 7fb0b4a83740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563d9bec3c30 0x563d9beb41d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:37.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.745+0000 7fb0b4a83740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x563d9be97960 0x563d9bf2a560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:37.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.745+0000 7fb0b4a83740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d9be91530 con 0x563d9bec3c30 2024-03-04T09:25:37.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.745+0000 7fb0b4a83740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d9bec2f00 con 0x563d9be97960 2024-03-04T09:25:37.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.745+0000 7fb0b4a83740 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x563d9be91b00 con 0x563d9be92990 2024-03-04T09:25:37.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.745+0000 7fb0b4a83740 10 client.0 _ll_register_callbacks cb 0x7fb08c0036f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T09:25:37.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.746+0000 7fb0aa000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563d9bec3c30 0x563d9beb41d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:37.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.746+0000 7fb0aa000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563d9bec3c30 0x563d9beb41d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45932/0 (socket says 172.21.15.6:45932) 2024-03-04T09:25:37.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.746+0000 7fb0aa000640 1 -- 172.21.15.6:0/3441448517 learned_addr learned my addr 172.21.15.6:0/3441448517 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:25:37.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.746+0000 7fb0aaa00640 1 --2- 172.21.15.6:0/3441448517 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x563d9be92990 0x563d9be6aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:37.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.746+0000 7fb0ab400640 1 --2- 172.21.15.6:0/3441448517 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x563d9be97960 0x563d9bf2a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:37.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.746+0000 7fb0aa000640 1 -- 172.21.15.6:0/3441448517 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x563d9be92990 msgr2=0x563d9be6aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:37.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.746+0000 7fb0aa000640 1 --2- 172.21.15.6:0/3441448517 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x563d9be92990 0x563d9be6aa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:37.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.746+0000 7fb0aa000640 1 -- 172.21.15.6:0/3441448517 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x563d9be97960 msgr2=0x563d9bf2a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:37.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.746+0000 7fb0aa000640 1 --2- 172.21.15.6:0/3441448517 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x563d9be97960 0x563d9bf2a560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:37.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.746+0000 7fb0aa000640 1 -- 172.21.15.6:0/3441448517 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563d9be6af70 con 0x563d9bec3c30 2024-03-04T09:25:37.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.746+0000 7fb0ab400640 1 --2- 172.21.15.6:0/3441448517 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x563d9be97960 0x563d9bf2a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:25:37.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.746+0000 7fb0aa000640 1 --2- 172.21.15.6:0/3441448517 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563d9bec3c30 0x563d9beb41d0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fb09402c4e0 tx=0x7fb094009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3701245afbefcf06 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:37.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.746+0000 7fb093e00640 10 client.0 ms_handle_connect on v2:172.21.15.6:3300/0 2024-03-04T09:25:37.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.747+0000 7fb093e00640 1 -- 172.21.15.6:0/3441448517 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb09406d060 con 0x563d9bec3c30 2024-03-04T09:25:37.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.747+0000 7fb093e00640 1 -- 172.21.15.6:0/3441448517 <== mon.0 v2:172.21.15.6:3300/0 2 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb094034850 con 0x563d9bec3c30 2024-03-04T09:25:37.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.747+0000 7fb0b4a83740 1 -- 172.21.15.6:0/3441448517 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb09404cb40 con 0x563d9bec3c30 2024-03-04T09:25:37.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.747+0000 7fb093e00640 1 -- 172.21.15.6:0/3441448517 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_get_version_reply(handle=1 version=7) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb094073050 con 0x563d9bec3c30 2024-03-04T09:25:37.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.747+0000 7fb0b4a83740 10 client.4614 fetch_fsmap learned FSMap version 7 2024-03-04T09:25:37.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.747+0000 7fb0b4a83740 1 -- 172.21.15.6:0/3441448517 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,fsmap=7,monmap=2+}) v3 -- 0x563d9bed3a80 con 0x563d9bec3c30 2024-03-04T09:25:37.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.747+0000 7fb093e00640 1 -- 172.21.15.6:0/3441448517 <== mon.0 v2:172.21.15.6:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb09404cb40 con 0x563d9bec3c30 2024-03-04T09:25:37.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.747+0000 7fb093e00640 1 -- 172.21.15.6:0/3441448517 <== mon.0 v2:172.21.15.6:3300/0 5 ==== fsmap(e 7) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fb094076020 con 0x563d9bec3c30 2024-03-04T09:25:37.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.747+0000 7fb0b4a83740 10 client.4614 fetch_fsmap finished waiting for FSMap version 7 2024-03-04T09:25:37.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.747+0000 7fb0b4a83740 10 client.4614 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-04T09:25:37.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.747+0000 7fb0b4a83740 20 client.4614 populate_metadata read hostname 'smithi006' 2024-03-04T09:25:37.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.747+0000 7fb0b4a83740 1 --2- 172.21.15.6:0/3441448517 >> [v2:172.21.15.146:6835/1946537763,v1:172.21.15.146:6836/1946537763] conn(0x563d9bf291c0 0x563d9bf295a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:37.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.747+0000 7fb0b4a83740 4 client.4614 mds_command: new command op to 4288 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-04T09:25:37.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.747+0000 7fb0b4a83740 1 -- 172.21.15.6:0/3441448517 --> [v2:172.21.15.146:6835/1946537763,v1:172.21.15.146:6836/1946537763] -- command(tid 0: {"prefix": "status"}) v1 -- 0x563d9bdd6e90 con 0x563d9bf291c0 2024-03-04T09:25:37.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.748+0000 7fb0aaa00640 1 --2- 172.21.15.6:0/3441448517 >> [v2:172.21.15.146:6835/1946537763,v1:172.21.15.146:6836/1946537763] conn(0x563d9bf291c0 0x563d9bf295a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:37.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.749+0000 7fb0aaa00640 1 --2- 172.21.15.6:0/3441448517 >> [v2:172.21.15.146:6835/1946537763,v1:172.21.15.146:6836/1946537763] conn(0x563d9bf291c0 0x563d9bf295a0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4a17425a51ae8e9c server_cookie=277dcd04ca473c9b in_seq=0 out_seq=0 2024-03-04T09:25:37.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.749+0000 7fb093e00640 10 client.4614 ms_handle_connect on v2:172.21.15.146:6835/1946537763 2024-03-04T09:25:37.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.749+0000 7fb093e00640 1 -- 172.21.15.6:0/3441448517 <== mds.0 v2:172.21.15.146:6835/1946537763 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+330 (crc 0 0 0) 0x563d9bdd6e90 con 0x563d9bf291c0 2024-03-04T09:25:37.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.749+0000 7fb093e00640 10 client.4614 handle_command_reply: tid=0 2024-03-04T09:25:37.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.749+0000 7fb0b4a83740 1 client.4614 shutdown 2024-03-04T09:25:37.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.750+0000 7fb0b4a83740 1 -- 172.21.15.6:0/3441448517 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563d9bec3c30 msgr2=0x563d9beb41d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:37.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.750+0000 7fb0b4a83740 1 --2- 172.21.15.6:0/3441448517 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563d9bec3c30 0x563d9beb41d0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fb09402c4e0 tx=0x7fb094009750 comp rx=0 tx=0).stop 2024-03-04T09:25:37.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.750+0000 7fb0b4a83740 1 -- 172.21.15.6:0/3441448517 shutdown_connections 2024-03-04T09:25:37.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.750+0000 7fb0b4a83740 1 --2- 172.21.15.6:0/3441448517 >> [v2:172.21.15.146:6835/1946537763,v1:172.21.15.146:6836/1946537763] conn(0x563d9bf291c0 0x563d9bf295a0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:37.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.750+0000 7fb0b4a83740 1 --2- 172.21.15.6:0/3441448517 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x563d9be97960 0x563d9bf2a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:37.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.750+0000 7fb0b4a83740 1 --2- 172.21.15.6:0/3441448517 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563d9bec3c30 0x563d9beb41d0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:37.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.750+0000 7fb0b4a83740 1 --2- 172.21.15.6:0/3441448517 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x563d9be92990 0x563d9be6aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:37.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.750+0000 7fb093e00640 0 client.4614 ms_handle_reset on v2:172.21.15.146:6835/1946537763 2024-03-04T09:25:37.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.750+0000 7fb0b4a83740 1 -- 172.21.15.6:0/3441448517 >> 172.21.15.6:0/3441448517 conn(0x563d9be65e70 msgr2=0x563d9be8eb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:37.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.750+0000 7fb0b4a83740 1 -- 172.21.15.6:0/3441448517 shutdown_connections 2024-03-04T09:25:37.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.750+0000 7fb0b4a83740 1 -- 172.21.15.6:0/3441448517 wait complete. 2024-03-04T09:25:37.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.750+0000 7fb0b4a83740 20 client.4614 trim_cache size 0 max 16384 2024-03-04T09:25:37.751 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-04T09:25:37.751 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_fsid": "edce0578-5ae4-4402-b45a-552bdef84d7c", 2024-03-04T09:25:37.751 INFO:teuthology.orchestra.run.smithi006.stdout: "whoami": 0, 2024-03-04T09:25:37.751 INFO:teuthology.orchestra.run.smithi006.stdout: "id": 4288, 2024-03-04T09:25:37.751 INFO:teuthology.orchestra.run.smithi006.stdout: "want_state": "up:active", 2024-03-04T09:25:37.751 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "up:active", 2024-03-04T09:25:37.751 INFO:teuthology.orchestra.run.smithi006.stdout: "fs_name": "cephfs", 2024-03-04T09:25:37.751 INFO:teuthology.orchestra.run.smithi006.stdout: "rank_uptime": 18.299737732000001, 2024-03-04T09:25:37.752 INFO:teuthology.orchestra.run.smithi006.stdout: "mdsmap_epoch": 7, 2024-03-04T09:25:37.752 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap_epoch": 20, 2024-03-04T09:25:37.752 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap_epoch_barrier": 20, 2024-03-04T09:25:37.752 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 23.691081270000002 2024-03-04T09:25:37.752 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-04T09:25:37.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.751+0000 7fb0abe00640 1 -- 172.21.15.6:0/3616246525 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb08806b4d0 msgr2=0x7fb08806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:37.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.751+0000 7fb0abe00640 1 --2- 172.21.15.6:0/3616246525 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb08806b4d0 0x7fb08806d990 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fb094036000 tx=0x7fb094030000 comp rx=0 tx=0).stop 2024-03-04T09:25:37.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.751+0000 7fb0abe00640 1 -- 172.21.15.6:0/3616246525 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb0ac0cf460 msgr2=0x7fb0ac1f2f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:37.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.751+0000 7fb0abe00640 1 --2- 172.21.15.6:0/3616246525 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb0ac0cf460 0x7fb0ac1f2f90 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fb09c00c460 tx=0x7fb09c00c920 comp rx=0 tx=0).stop 2024-03-04T09:25:37.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.751+0000 7fb0abe00640 1 -- 172.21.15.6:0/3616246525 shutdown_connections 2024-03-04T09:25:37.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.751+0000 7fb0abe00640 1 --2- 172.21.15.6:0/3616246525 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb0ac15e2a0 0x7fb0ac06bea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:37.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.751+0000 7fb0abe00640 1 --2- 172.21.15.6:0/3616246525 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb08806b4d0 0x7fb08806d990 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:37.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.751+0000 7fb0abe00640 1 --2- 172.21.15.6:0/3616246525 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb0ac15dec0 0x7fb0ac06b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:37.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.751+0000 7fb0abe00640 1 --2- 172.21.15.6:0/3616246525 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb0ac0cf460 0x7fb0ac1f2f90 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:37.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.751+0000 7fb0abe00640 1 -- 172.21.15.6:0/3616246525 >> 172.21.15.6:0/3616246525 conn(0x7fb0ac0d3d40 msgr2=0x7fb0ac160e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:37.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.751+0000 7fb0abe00640 1 -- 172.21.15.6:0/3616246525 shutdown_connections 2024-03-04T09:25:37.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:37.751+0000 7fb0abe00640 1 -- 172.21.15.6:0/3616246525 wait complete. 2024-03-04T09:25:37.764 INFO:teuthology.run_tasks:Running task kclient... 2024-03-04T09:25:37.789 INFO:tasks.kclient:Mounting kernel clients... 2024-03-04T09:25:37.789 INFO:tasks.kclient:config is {'client.0': {}, 'client.2': {}, 'client.1': {}, 'client.3': {}} 2024-03-04T09:25:37.789 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:25:37.790 DEBUG:teuthology.orchestra.run.smithi006:> ip netns list 2024-03-04T09:25:37.845 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:25:37.845 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip link delete ceph-brx 2024-03-04T09:25:37.951 INFO:teuthology.orchestra.run.smithi006.stderr:Cannot find device "ceph-brx" 2024-03-04T09:25:37.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T09:25:37.956 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:25:37.956 DEBUG:teuthology.orchestra.run.smithi006:> ip netns list 2024-03-04T09:25:38.013 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:25:38.013 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip link delete ceph-brx 2024-03-04T09:25:38.114 INFO:teuthology.orchestra.run.smithi006.stderr:Cannot find device "ceph-brx" 2024-03-04T09:25:38.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T09:25:38.118 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:25:38.118 DEBUG:teuthology.orchestra.run.smithi146:> ip netns list 2024-03-04T09:25:38.175 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:25:38.175 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip link delete ceph-brx 2024-03-04T09:25:38.277 INFO:teuthology.orchestra.run.smithi146.stderr:Cannot find device "ceph-brx" 2024-03-04T09:25:38.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T09:25:38.282 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:25:38.282 DEBUG:teuthology.orchestra.run.smithi146:> ip netns list 2024-03-04T09:25:38.332 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:25:38.332 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip link delete ceph-brx 2024-03-04T09:25:38.425 INFO:teuthology.orchestra.run.smithi146.stderr:Cannot find device "ceph-brx" 2024-03-04T09:25:38.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T09:25:38.428 INFO:tasks.kclient:client.0 config is {'syntax': 'v2'} 2024-03-04T09:25:38.428 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-04T09:25:38.428 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-04T09:25:38.428 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-04T09:25:38.609 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:25:38.617 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:25:38.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:38.656+0000 7f7598600640 1 Processor -- start 2024-03-04T09:25:38.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:38.656+0000 7f7598600640 1 -- start start 2024-03-04T09:25:38.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:38.659+0000 7f7598600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f75900cdc20 0x7f7590071760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:38.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:38.659+0000 7f7598600640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75900b4860 con 0x7f75900b3c90 2024-03-04T09:25:38.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:38.659+0000 7f7598600640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75900b4b40 con 0x7f75900cf5c0 2024-03-04T09:25:38.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:38.659+0000 7f7598600640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75900b4e20 con 0x7f75900cdc20 2024-03-04T09:25:38.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:38.659+0000 7f7595c00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f75900b3c90 0x7f75900b4070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:36892/0 (socket says 172.21.15.6:36892) 2024-03-04T09:25:38.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:38.659+0000 7f7595c00640 1 -- 172.21.15.6:0/3607246757 learned_addr learned my addr 172.21.15.6:0/3607246757 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:25:38.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:38.659+0000 7f7596600640 1 --2- 172.21.15.6:0/3607246757 >> v2:172.21.15.6:3300/0 conn(0x7f75900cdc20 0x7f7590071760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:38.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:38.659+0000 7f758fe00640 1 -- 172.21.15.6:0/3607246757 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 172618520 0 0) 0x7f75900b4860 con 0x7f75900b3c90 2024-03-04T09:25:38.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:38.660+0000 7f758fe00640 1 -- 172.21.15.6:0/3607246757 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7578003520 con 0x7f75900b3c90 2024-03-04T09:25:38.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:38.660+0000 7f758fe00640 1 -- 172.21.15.6:0/3607246757 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1946900859 0 0) 0x7f75900b4b40 con 0x7f75900cf5c0 2024-03-04T09:25:38.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:38.660+0000 7f7596600640 1 -- 172.21.15.6:0/3607246757 >> v1:172.21.15.146:6789/0 conn(0x7f75900cf5c0 legacy=0x7f7590071050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:38.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:38.660+0000 7f7596600640 1 -- 172.21.15.6:0/3607246757 >> v1:172.21.15.6:6789/0 conn(0x7f75900b3c90 legacy=0x7f75900b4070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:38.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:38.660+0000 7f7596600640 1 -- 172.21.15.6:0/3607246757 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75900b5100 con 0x7f75900cdc20 2024-03-04T09:25:38.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:38.661+0000 7f7596600640 1 --2- 172.21.15.6:0/3607246757 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f75900cdc20 0x7f7590071760 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f7588009d90 tx=0x7f758802bfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6a1f396aa7a7dc5 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:38.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:38.662+0000 7f758fe00640 1 -- 172.21.15.6:0/3607246757 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7588039070 con 0x7f75900cdc20 2024-03-04T09:25:38.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:38.662+0000 7f758fe00640 1 -- 172.21.15.6:0/3607246757 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f758802cd80 con 0x7f75900cdc20 2024-03-04T09:25:38.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:38.662+0000 7f758fe00640 1 -- 172.21.15.6:0/3607246757 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7588034640 con 0x7f75900cdc20 2024-03-04T09:25:38.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:38.662+0000 7f7598600640 1 -- 172.21.15.6:0/3607246757 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f75900cdc20 msgr2=0x7f7590071760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:38.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:38.662+0000 7f7598600640 1 --2- 172.21.15.6:0/3607246757 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f75900cdc20 0x7f7590071760 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f7588009d90 tx=0x7f758802bfd0 comp rx=0 tx=0).stop 2024-03-04T09:25:38.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:38.662+0000 7f7598600640 1 -- 172.21.15.6:0/3607246757 shutdown_connections 2024-03-04T09:25:38.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:38.662+0000 7f7598600640 1 --2- 172.21.15.6:0/3607246757 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f75900cdc20 0x7f7590071760 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:38.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:38.662+0000 7f7598600640 1 -- 172.21.15.6:0/3607246757 >> 172.21.15.6:0/3607246757 conn(0x7f759006afb0 msgr2=0x7f759006b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:38.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:38.663+0000 7f7598600640 1 -- 172.21.15.6:0/3607246757 shutdown_connections 2024-03-04T09:25:38.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:38.663+0000 7f7598600640 1 -- 172.21.15.6:0/3607246757 wait complete. 2024-03-04T09:25:38.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:38.664+0000 7f7598600640 1 Processor -- start 2024-03-04T09:25:38.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:38.664+0000 7f7598600640 1 -- start start 2024-03-04T09:25:38.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:38.665+0000 7f7598600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f75900b3c90 0x7f759013d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:38.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:38.665+0000 7f7598600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f75900cdc20 0x7f7590141140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:38.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:38.665+0000 7f7598600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f75900cf5c0 0x7f75900adbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:38.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:38.665+0000 7f7598600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7590071ca0 con 0x7f75900cf5c0 2024-03-04T09:25:38.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:38.665+0000 7f7598600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75900ce060 con 0x7f75900b3c90 2024-03-04T09:25:38.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:38.665+0000 7f7598600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7590071e20 con 0x7f75900cdc20 2024-03-04T09:25:38.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:38.665+0000 7f7596600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f75900cf5c0 0x7f75900adbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:38.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:38.666+0000 7f7596600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f75900cf5c0 0x7f75900adbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45940/0 (socket says 172.21.15.6:45940) 2024-03-04T09:25:38.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:38.666+0000 7f7596600640 1 -- 172.21.15.6:0/3802627630 learned_addr learned my addr 172.21.15.6:0/3802627630 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:25:38.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:38.666+0000 7f7595200640 1 --2- 172.21.15.6:0/3802627630 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f75900cdc20 0x7f7590141140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:38.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:38.666+0000 7f7595c00640 1 --2- 172.21.15.6:0/3802627630 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f75900b3c90 0x7f759013d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:38.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:38.666+0000 7f7596600640 1 -- 172.21.15.6:0/3802627630 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f75900cdc20 msgr2=0x7f7590141140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:38.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:38.666+0000 7f7596600640 1 --2- 172.21.15.6:0/3802627630 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f75900cdc20 0x7f7590141140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:38.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:38.666+0000 7f7596600640 1 -- 172.21.15.6:0/3802627630 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f75900b3c90 msgr2=0x7f759013d6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:38.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:38.666+0000 7f7596600640 1 --2- 172.21.15.6:0/3802627630 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f75900b3c90 0x7f759013d6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:38.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:38.666+0000 7f7596600640 1 -- 172.21.15.6:0/3802627630 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7590158d70 con 0x7f75900cf5c0 2024-03-04T09:25:38.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:38.666+0000 7f7595200640 1 --2- 172.21.15.6:0/3802627630 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f75900cdc20 0x7f7590141140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T09:25:38.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:38.666+0000 7f7595c00640 1 --2- 172.21.15.6:0/3802627630 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f75900b3c90 0x7f759013d6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T09:25:38.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:38.666+0000 7f7596600640 1 --2- 172.21.15.6:0/3802627630 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f75900cf5c0 0x7f75900adbf0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f7588009eb0 tx=0x7f7588004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:38.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:38.667+0000 7f758d600640 1 -- 172.21.15.6:0/3802627630 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7588039050 con 0x7f75900cf5c0 2024-03-04T09:25:38.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:38.667+0000 7f758d600640 1 -- 172.21.15.6:0/3802627630 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7588045070 con 0x7f75900cf5c0 2024-03-04T09:25:38.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:38.667+0000 7f758d600640 1 -- 172.21.15.6:0/3802627630 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7588009450 con 0x7f75900cf5c0 2024-03-04T09:25:38.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:38.667+0000 7f7598600640 1 -- 172.21.15.6:0/3802627630 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7590159050 con 0x7f75900cf5c0 2024-03-04T09:25:38.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:38.667+0000 7f7598600640 1 -- 172.21.15.6:0/3802627630 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7590157600 con 0x7f75900cf5c0 2024-03-04T09:25:38.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:38.668+0000 7f7598600640 1 -- 172.21.15.6:0/3802627630 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7590069510 con 0x7f75900cf5c0 2024-03-04T09:25:38.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:38.669+0000 7f758d600640 1 -- 172.21.15.6:0/3802627630 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f758803e830 con 0x7f75900cf5c0 2024-03-04T09:25:38.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:38.670+0000 7f758d600640 1 --2- 172.21.15.6:0/3802627630 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f757006b4d0 0x7f757006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:38.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:38.670+0000 7f758d600640 1 -- 172.21.15.6:0/3802627630 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f75880acc00 con 0x7f75900cf5c0 2024-03-04T09:25:38.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:38.670+0000 7f7595c00640 1 --2- 172.21.15.6:0/3802627630 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f757006b4d0 0x7f757006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:38.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:38.671+0000 7f7595c00640 1 --2- 172.21.15.6:0/3802627630 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f757006b4d0 0x7f757006d990 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f7584007fd0 tx=0x7f7584009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:38.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:38.674+0000 7f758d600640 1 -- 172.21.15.6:0/3802627630 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f75880b3080 con 0x7f75900cf5c0 2024-03-04T09:25:38.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:38.916+0000 7f7598600640 1 -- 172.21.15.6:0/3802627630 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f759004f3a0 con 0x7f75900cf5c0 2024-03-04T09:25:38.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:38.917+0000 7f758d600640 1 -- 172.21.15.6:0/3802627630 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v7) v1 ==== 53+0+73 (secure 0 0 0) 0x7f758804f020 con 0x7f75900cf5c0 2024-03-04T09:25:38.914 INFO:teuthology.orchestra.run.smithi006.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-04T09:25:38.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:38.920+0000 7f7598600640 1 -- 172.21.15.6:0/3802627630 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f757006b4d0 msgr2=0x7f757006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:38.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:38.920+0000 7f7598600640 1 --2- 172.21.15.6:0/3802627630 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f757006b4d0 0x7f757006d990 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f7584007fd0 tx=0x7f7584009040 comp rx=0 tx=0).stop 2024-03-04T09:25:38.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:38.920+0000 7f7598600640 1 -- 172.21.15.6:0/3802627630 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f75900cf5c0 msgr2=0x7f75900adbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:38.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:38.920+0000 7f7598600640 1 --2- 172.21.15.6:0/3802627630 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f75900cf5c0 0x7f75900adbf0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f7588009eb0 tx=0x7f7588004a40 comp rx=0 tx=0).stop 2024-03-04T09:25:38.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:38.920+0000 7f7598600640 1 -- 172.21.15.6:0/3802627630 shutdown_connections 2024-03-04T09:25:38.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:38.920+0000 7f7598600640 1 --2- 172.21.15.6:0/3802627630 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f75900cf5c0 0x7f75900adbf0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:38.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:38.920+0000 7f7598600640 1 --2- 172.21.15.6:0/3802627630 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f757006b4d0 0x7f757006d990 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:38.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:38.920+0000 7f7598600640 1 --2- 172.21.15.6:0/3802627630 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f75900cdc20 0x7f7590141140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:38.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:38.920+0000 7f7598600640 1 --2- 172.21.15.6:0/3802627630 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f75900b3c90 0x7f759013d6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:38.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:38.920+0000 7f7598600640 1 -- 172.21.15.6:0/3802627630 >> 172.21.15.6:0/3802627630 conn(0x7f759006afb0 msgr2=0x7f7590070f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:38.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:38.920+0000 7f7598600640 1 -- 172.21.15.6:0/3802627630 shutdown_connections 2024-03-04T09:25:38.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:38.920+0000 7f7598600640 1 -- 172.21.15.6:0/3802627630 wait complete. 2024-03-04T09:25:38.937 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-04T09:25:38.937 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-04T09:25:38.938 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi006.front.sepia.ceph.com 2024-03-04T09:25:38.938 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-04T09:25:38.938 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-04T09:25:38.938 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-04T09:25:38.938 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-04T09:25:38.938 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-04T09:25:38.938 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-04T09:25:38.938 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:25:38.938 DEBUG:teuthology.orchestra.run.smithi006:> ip addr 2024-03-04T09:25:38.992 INFO:teuthology.orchestra.run.smithi006.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-04T09:25:38.993 INFO:teuthology.orchestra.run.smithi006.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-04T09:25:38.993 INFO:teuthology.orchestra.run.smithi006.stdout: inet 127.0.0.1/8 scope host lo 2024-03-04T09:25:38.993 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-04T09:25:38.993 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 ::1/128 scope host 2024-03-04T09:25:38.993 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-04T09:25:38.993 INFO:teuthology.orchestra.run.smithi006.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-04T09:25:38.993 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:6b:fe:e6 brd ff:ff:ff:ff:ff:ff 2024-03-04T09:25:38.993 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp7s0f0 2024-03-04T09:25:38.993 INFO:teuthology.orchestra.run.smithi006.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-04T09:25:38.993 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:6b:fe:e7 brd ff:ff:ff:ff:ff:ff 2024-03-04T09:25:38.993 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp7s0f1 2024-03-04T09:25:38.993 INFO:teuthology.orchestra.run.smithi006.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-04T09:25:38.993 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:bd:15:e4 brd ff:ff:ff:ff:ff:ff 2024-03-04T09:25:38.994 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp1s0f0 2024-03-04T09:25:38.994 INFO:teuthology.orchestra.run.smithi006.stdout: inet 172.21.15.6/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-04T09:25:38.994 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft 42595sec preferred_lft 42595sec 2024-03-04T09:25:38.994 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::ec4:7aff:febd:15e4/64 scope link noprefixroute 2024-03-04T09:25:38.994 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-04T09:25:38.994 INFO:teuthology.orchestra.run.smithi006.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-04T09:25:38.994 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:bd:15:e5 brd ff:ff:ff:ff:ff:ff 2024-03-04T09:25:38.994 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp1s0f1 2024-03-04T09:25:38.994 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-04T09:25:38.995 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:25:38.995 DEBUG:teuthology.orchestra.run.smithi006:> (cd / && exec bash -c ' 2024-03-04T09:25:38.995 DEBUG:teuthology.orchestra.run.smithi006:> set -e 2024-03-04T09:25:38.995 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip link add name ceph-brx type bridge 2024-03-04T09:25:38.995 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip addr flush dev ceph-brx 2024-03-04T09:25:38.995 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip link set ceph-brx up 2024-03-04T09:25:38.995 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-04T09:25:38.995 DEBUG:teuthology.orchestra.run.smithi006:> ') 2024-03-04T09:25:39.470 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:25:39.470 DEBUG:teuthology.orchestra.run.smithi006:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-04T09:25:39.552 INFO:teuthology.orchestra.run.smithi006.stdout:1 2024-03-04T09:25:39.556 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:25:39.557 DEBUG:teuthology.orchestra.run.smithi006:> route 2024-03-04T09:25:39.620 INFO:teuthology.orchestra.run.smithi006.stdout:Kernel IP routing table 2024-03-04T09:25:39.621 INFO:teuthology.orchestra.run.smithi006.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-04T09:25:39.621 INFO:teuthology.orchestra.run.smithi006.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-04T09:25:39.621 INFO:teuthology.orchestra.run.smithi006.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-04T09:25:39.621 INFO:teuthology.orchestra.run.smithi006.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-04T09:25:39.622 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:25:39.622 DEBUG:teuthology.orchestra.run.smithi006:> (cd / && exec bash -c ' 2024-03-04T09:25:39.622 DEBUG:teuthology.orchestra.run.smithi006:> set -e 2024-03-04T09:25:39.622 DEBUG:teuthology.orchestra.run.smithi006:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-04T09:25:39.622 DEBUG:teuthology.orchestra.run.smithi006:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-04T09:25:39.622 DEBUG:teuthology.orchestra.run.smithi006:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-04T09:25:39.622 DEBUG:teuthology.orchestra.run.smithi006:> ') 2024-03-04T09:25:40.361 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:25:40.361 DEBUG:teuthology.orchestra.run.smithi006:> ip netns list 2024-03-04T09:25:40.421 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:25:40.421 DEBUG:teuthology.orchestra.run.smithi006:> ip netns list-id 2024-03-04T09:25:40.478 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:25:40.478 DEBUG:teuthology.orchestra.run.smithi006:> (cd / && exec bash -c ' 2024-03-04T09:25:40.478 DEBUG:teuthology.orchestra.run.smithi006:> set -e 2024-03-04T09:25:40.478 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-04T09:25:40.478 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-03-04T09:25:40.478 DEBUG:teuthology.orchestra.run.smithi006:> ') 2024-03-04T09:25:40.629 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-03-04T09:25:40.629 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:25:40.629 DEBUG:teuthology.orchestra.run.smithi006:> (cd / && exec bash -c ' 2024-03-04T09:25:40.629 DEBUG:teuthology.orchestra.run.smithi006:> set -e 2024-03-04T09:25:40.629 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-03-04T09:25:40.629 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-04T09:25:40.629 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-03-04T09:25:40.629 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-03-04T09:25:40.629 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-03-04T09:25:40.630 DEBUG:teuthology.orchestra.run.smithi006:> ') 2024-03-04T09:25:40.943 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:25:40.944 DEBUG:teuthology.orchestra.run.smithi006:> (cd / && exec bash -c ' 2024-03-04T09:25:40.944 DEBUG:teuthology.orchestra.run.smithi006:> set -e 2024-03-04T09:25:40.944 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip link set brx.0 up 2024-03-04T09:25:40.944 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip link set dev brx.0 master ceph-brx 2024-03-04T09:25:40.944 DEBUG:teuthology.orchestra.run.smithi006:> ') 2024-03-04T09:25:41.120 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-04T09:25:41.120 DEBUG:teuthology.orchestra.run.smithi006:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-04T09:25:41.169 INFO:teuthology.orchestra.run.smithi006.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-04T09:25:41.170 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-04T09:25:41.170 DEBUG:teuthology.orchestra.run.smithi006:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-04T09:25:41.272 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-03-04T09:25:41.272 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:25:41.272 DEBUG:teuthology.orchestra.run.smithi006:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-04T09:25:41.414 INFO:teuthology.orchestra.run.smithi006.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-04T09:25:41.414 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-04T09:25:42.072 INFO:teuthology.orchestra.run.smithi006.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-04T09:25:42.072 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-04T09:25:42.072 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: resolved to: "172.21.15.6:3300,172.21.15.146:3300,172.21.15.146:3301" 2024-03-04T09:25:42.072 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: trying mount with new device syntax: 0@edce0578-5ae4-4402-b45a-552bdef84d7c.cephfs=/ 2024-03-04T09:25:42.073 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.6:3300/172.21.15.146:3300/172.21.15.146:3301" will pass to kernel 2024-03-04T09:25:42.075 INFO:teuthology.orchestra.run.smithi006.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-04T09:25:42.075 INFO:teuthology.orchestra.run.smithi006.stdout: You just mounted a file system that supports labels which does not 2024-03-04T09:25:42.076 INFO:teuthology.orchestra.run.smithi006.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-04T09:25:42.076 INFO:teuthology.orchestra.run.smithi006.stdout: applications will generate AVC messages and not be allowed access to 2024-03-04T09:25:42.076 INFO:teuthology.orchestra.run.smithi006.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-04T09:25:42.081 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-04T09:25:42.082 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:25:42.082 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-04T09:25:42.184 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:25:42.184 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-04T09:25:42.271 INFO:teuthology.orchestra.run.smithi006.stdout:client4490 2024-03-04T09:25:42.271 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-04T09:25:42.271 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T09:25:42.331 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:25:42.331 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-04T09:25:42.409 INFO:teuthology.orchestra.run.smithi006.stdout:client4490 2024-03-04T09:25:42.410 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:25:42.410 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-04T09:25:42.491 INFO:teuthology.orchestra.run.smithi006.stdout:client4490 2024-03-04T09:25:42.491 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:25:42.491 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/edce0578-5ae4-4402-b45a-552bdef84d7c.client4490/status') 2024-03-04T09:25:42.608 INFO:teuthology.orchestra.run.smithi006.stdout:instance: client.4490 (3)172.21.15.6:0/415171648 2024-03-04T09:25:42.608 INFO:teuthology.orchestra.run.smithi006.stdout:blocklisted: false 2024-03-04T09:25:42.609 INFO:teuthology.orchestra.run.smithi006.stderr:0+1 records in 2024-03-04T09:25:42.609 INFO:teuthology.orchestra.run.smithi006.stderr:0+1 records out 2024-03-04T09:25:42.609 INFO:teuthology.orchestra.run.smithi006.stderr:68 bytes copied, 7.1045e-05 s, 957 kB/s 2024-03-04T09:25:42.614 INFO:tasks.kclient:client.2 config is {'syntax': 'v2'} 2024-03-04T09:25:42.614 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-04T09:25:42.614 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-03-04T09:25:42.614 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-04T09:25:42.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:42.845+0000 7f54cd400640 1 Processor -- start 2024-03-04T09:25:42.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:42.845+0000 7f54cd400640 1 -- start start 2024-03-04T09:25:42.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:42.847+0000 7f54cd400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f54c81552b0 0x7f54c815b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:42.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:42.847+0000 7f54cd400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f54c815e810 0x7f54c80cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:42.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:42.847+0000 7f54cd400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f54c80cce90 0x7f54c806c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:42.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:42.847+0000 7f54cd400640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54c815ed10 con 0x7f54c815e810 2024-03-04T09:25:42.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:42.847+0000 7f54cd400640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54c815eff0 con 0x7f54c80cce90 2024-03-04T09:25:42.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:42.847+0000 7f54cd400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54c815f2d0 con 0x7f54c81552b0 2024-03-04T09:25:42.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:42.847+0000 7f54c7400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f54c815e810 0x7f54c80cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:42.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:42.848+0000 7f54c7400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f54c815e810 0x7f54c80cc950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45074/0 (socket says 172.21.15.6:45074) 2024-03-04T09:25:42.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:42.848+0000 7f54c7400640 1 -- 172.21.15.6:0/1229073978 learned_addr learned my addr 172.21.15.6:0/1229073978 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:25:42.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:42.848+0000 7f54cca00640 1 --2- 172.21.15.6:0/1229073978 >> v2:172.21.15.146:3300/0 conn(0x7f54c80cce90 0x7f54c806c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:42.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:42.848+0000 7f54c7e00640 1 --2- 172.21.15.6:0/1229073978 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f54c81552b0 0x7f54c815b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:42.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:42.848+0000 7f54c7400640 1 -- 172.21.15.6:0/1229073978 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f54c81552b0 msgr2=0x7f54c815b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:42.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:42.848+0000 7f54c7400640 1 --2- 172.21.15.6:0/1229073978 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f54c81552b0 0x7f54c815b1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:42.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:42.848+0000 7f54c7400640 1 -- 172.21.15.6:0/1229073978 >> v2:172.21.15.146:3300/0 conn(0x7f54c80cce90 msgr2=0x7f54c806c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:42.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:42.848+0000 7f54c7400640 1 --2- 172.21.15.6:0/1229073978 >> v2:172.21.15.146:3300/0 conn(0x7f54c80cce90 0x7f54c806c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:42.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:42.848+0000 7f54c7400640 1 -- 172.21.15.6:0/1229073978 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54c815f5b0 con 0x7f54c815e810 2024-03-04T09:25:42.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:42.848+0000 7f54c7e00640 1 --2- 172.21.15.6:0/1229073978 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f54c81552b0 0x7f54c815b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:25:42.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:42.849+0000 7f54c7400640 1 --2- 172.21.15.6:0/1229073978 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54c815e810 0x7f54c80cc950 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f54c806dc10 tx=0x7f54b402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddf70ac093ed52f6 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:42.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:42.850+0000 7f54c6a00640 1 -- 172.21.15.6:0/1229073978 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54b402e070 con 0x7f54c815e810 2024-03-04T09:25:42.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:42.850+0000 7f54c6a00640 1 -- 172.21.15.6:0/1229073978 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54b4037070 con 0x7f54c815e810 2024-03-04T09:25:42.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:42.850+0000 7f54c6a00640 1 -- 172.21.15.6:0/1229073978 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54b4031a00 con 0x7f54c815e810 2024-03-04T09:25:42.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:42.850+0000 7f54cd400640 1 -- 172.21.15.6:0/1229073978 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54c815e810 msgr2=0x7f54c80cc950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:42.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:42.850+0000 7f54cd400640 1 --2- 172.21.15.6:0/1229073978 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54c815e810 0x7f54c80cc950 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f54c806dc10 tx=0x7f54b402b880 comp rx=0 tx=0).stop 2024-03-04T09:25:42.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:42.850+0000 7f54cd400640 1 -- 172.21.15.6:0/1229073978 shutdown_connections 2024-03-04T09:25:42.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:42.850+0000 7f54cd400640 1 --2- 172.21.15.6:0/1229073978 >> v2:172.21.15.146:3300/0 conn(0x7f54c80cce90 0x7f54c806c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:42.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:42.850+0000 7f54cd400640 1 --2- 172.21.15.6:0/1229073978 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54c815e810 0x7f54c80cc950 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:42.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:42.850+0000 7f54cd400640 1 --2- 172.21.15.6:0/1229073978 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f54c81552b0 0x7f54c815b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:42.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:42.850+0000 7f54cd400640 1 -- 172.21.15.6:0/1229073978 >> 172.21.15.6:0/1229073978 conn(0x7f54c806b3d0 msgr2=0x7f54c806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:42.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:42.850+0000 7f54cd400640 1 -- 172.21.15.6:0/1229073978 shutdown_connections 2024-03-04T09:25:42.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:42.850+0000 7f54cd400640 1 -- 172.21.15.6:0/1229073978 wait complete. 2024-03-04T09:25:42.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:42.852+0000 7f54cd400640 1 Processor -- start 2024-03-04T09:25:42.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:42.852+0000 7f54cd400640 1 -- start start 2024-03-04T09:25:42.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:42.853+0000 7f54cd400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f54c80cce90 0x7f54c813a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:42.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:42.853+0000 7f54cd400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f54c81552b0 0x7f54c814e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:42.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:42.853+0000 7f54cd400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54c815e810 0x7f54c8152580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:42.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:42.853+0000 7f54cd400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54c806dd30 con 0x7f54c815e810 2024-03-04T09:25:42.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:42.853+0000 7f54cd400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54c80b3760 con 0x7f54c80cce90 2024-03-04T09:25:42.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:42.853+0000 7f54cd400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54c806deb0 con 0x7f54c81552b0 2024-03-04T09:25:42.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:42.853+0000 7f54cca00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54c815e810 0x7f54c8152580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:42.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:42.854+0000 7f54cca00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54c815e810 0x7f54c8152580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45084/0 (socket says 172.21.15.6:45084) 2024-03-04T09:25:42.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:42.854+0000 7f54cca00640 1 -- 172.21.15.6:0/3798277468 learned_addr learned my addr 172.21.15.6:0/3798277468 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:25:42.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:42.854+0000 7f54c7400640 1 --2- 172.21.15.6:0/3798277468 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f54c81552b0 0x7f54c814e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:42.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:42.854+0000 7f54c7e00640 1 --2- 172.21.15.6:0/3798277468 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f54c80cce90 0x7f54c813a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:42.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:42.854+0000 7f54cca00640 1 -- 172.21.15.6:0/3798277468 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f54c81552b0 msgr2=0x7f54c814e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:42.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:42.854+0000 7f54cca00640 1 --2- 172.21.15.6:0/3798277468 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f54c81552b0 0x7f54c814e5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:42.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:42.854+0000 7f54cca00640 1 -- 172.21.15.6:0/3798277468 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f54c80cce90 msgr2=0x7f54c813a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:42.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:42.854+0000 7f54cca00640 1 --2- 172.21.15.6:0/3798277468 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f54c80cce90 0x7f54c813a250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:42.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:42.854+0000 7f54cca00640 1 -- 172.21.15.6:0/3798277468 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54c813a790 con 0x7f54c815e810 2024-03-04T09:25:42.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:42.854+0000 7f54cca00640 1 --2- 172.21.15.6:0/3798277468 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54c815e810 0x7f54c8152580 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f54c0007e60 tx=0x7f54c000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:42.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:42.855+0000 7f54bfe00640 1 -- 172.21.15.6:0/3798277468 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54c0014620 con 0x7f54c815e810 2024-03-04T09:25:42.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:42.855+0000 7f54bfe00640 1 -- 172.21.15.6:0/3798277468 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54c0019070 con 0x7f54c815e810 2024-03-04T09:25:42.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:42.855+0000 7f54cd400640 1 -- 172.21.15.6:0/3798277468 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54c814cc10 con 0x7f54c815e810 2024-03-04T09:25:42.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:42.855+0000 7f54bfe00640 1 -- 172.21.15.6:0/3798277468 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54c0014a20 con 0x7f54c815e810 2024-03-04T09:25:42.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:42.855+0000 7f54cd400640 1 -- 172.21.15.6:0/3798277468 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54c814d1d0 con 0x7f54c815e810 2024-03-04T09:25:42.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:42.856+0000 7f54cd400640 1 -- 172.21.15.6:0/3798277468 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54c804f330 con 0x7f54c815e810 2024-03-04T09:25:42.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:42.857+0000 7f54bfe00640 1 -- 172.21.15.6:0/3798277468 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f54c0014bc0 con 0x7f54c815e810 2024-03-04T09:25:42.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:42.858+0000 7f54bfe00640 1 --2- 172.21.15.6:0/3798277468 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f54a806b4d0 0x7f54a806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:42.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:42.858+0000 7f54bfe00640 1 -- 172.21.15.6:0/3798277468 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f54c008aad0 con 0x7f54c815e810 2024-03-04T09:25:42.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:42.859+0000 7f54c7e00640 1 --2- 172.21.15.6:0/3798277468 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f54a806b4d0 0x7f54a806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:42.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:42.860+0000 7f54c7e00640 1 --2- 172.21.15.6:0/3798277468 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f54a806b4d0 0x7f54a806d990 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f54b800a990 tx=0x7f54b8008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:42.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:42.863+0000 7f54bfe00640 1 -- 172.21.15.6:0/3798277468 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f54c0059940 con 0x7f54c815e810 2024-03-04T09:25:43.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:43.103+0000 7f54cd400640 1 -- 172.21.15.6:0/3798277468 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f54c80cf4a0 con 0x7f54c815e810 2024-03-04T09:25:43.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:43.104+0000 7f54bfe00640 1 -- 172.21.15.6:0/3798277468 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v7) v1 ==== 53+0+73 (secure 0 0 0) 0x7f54c0059240 con 0x7f54c815e810 2024-03-04T09:25:43.101 INFO:teuthology.orchestra.run.smithi006.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-04T09:25:43.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:43.107+0000 7f54cd400640 1 -- 172.21.15.6:0/3798277468 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f54a806b4d0 msgr2=0x7f54a806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:43.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:43.107+0000 7f54cd400640 1 --2- 172.21.15.6:0/3798277468 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f54a806b4d0 0x7f54a806d990 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f54b800a990 tx=0x7f54b8008040 comp rx=0 tx=0).stop 2024-03-04T09:25:43.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:43.107+0000 7f54cd400640 1 -- 172.21.15.6:0/3798277468 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54c815e810 msgr2=0x7f54c8152580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:43.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:43.107+0000 7f54cd400640 1 --2- 172.21.15.6:0/3798277468 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54c815e810 0x7f54c8152580 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f54c0007e60 tx=0x7f54c000e4b0 comp rx=0 tx=0).stop 2024-03-04T09:25:43.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:43.107+0000 7f54cd400640 1 -- 172.21.15.6:0/3798277468 shutdown_connections 2024-03-04T09:25:43.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:43.107+0000 7f54cd400640 1 --2- 172.21.15.6:0/3798277468 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54c815e810 0x7f54c8152580 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:43.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:43.107+0000 7f54cd400640 1 --2- 172.21.15.6:0/3798277468 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f54a806b4d0 0x7f54a806d990 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:43.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:43.107+0000 7f54cd400640 1 --2- 172.21.15.6:0/3798277468 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f54c81552b0 0x7f54c814e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:43.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:43.107+0000 7f54cd400640 1 --2- 172.21.15.6:0/3798277468 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f54c80cce90 0x7f54c813a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:43.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:43.107+0000 7f54cd400640 1 -- 172.21.15.6:0/3798277468 >> 172.21.15.6:0/3798277468 conn(0x7f54c806b3d0 msgr2=0x7f54c80b17c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:43.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:43.107+0000 7f54cd400640 1 -- 172.21.15.6:0/3798277468 shutdown_connections 2024-03-04T09:25:43.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:43.107+0000 7f54cd400640 1 -- 172.21.15.6:0/3798277468 wait complete. 2024-03-04T09:25:43.124 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-04T09:25:43.124 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-04T09:25:43.124 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi006.front.sepia.ceph.com 2024-03-04T09:25:43.124 INFO:tasks.cephfs.mount:self.client.name = client.2 2024-03-04T09:25:43.124 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-03-04T09:25:43.124 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-04T09:25:43.124 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-04T09:25:43.124 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-04T09:25:43.124 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.2' 2024-03-04T09:25:43.124 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:25:43.125 DEBUG:teuthology.orchestra.run.smithi006:> ip addr 2024-03-04T09:25:43.181 INFO:teuthology.orchestra.run.smithi006.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-04T09:25:43.182 INFO:teuthology.orchestra.run.smithi006.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-04T09:25:43.182 INFO:teuthology.orchestra.run.smithi006.stdout: inet 127.0.0.1/8 scope host lo 2024-03-04T09:25:43.182 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-04T09:25:43.182 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 ::1/128 scope host 2024-03-04T09:25:43.182 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-04T09:25:43.182 INFO:teuthology.orchestra.run.smithi006.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-04T09:25:43.182 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:6b:fe:e6 brd ff:ff:ff:ff:ff:ff 2024-03-04T09:25:43.182 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp7s0f0 2024-03-04T09:25:43.182 INFO:teuthology.orchestra.run.smithi006.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-04T09:25:43.182 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:6b:fe:e7 brd ff:ff:ff:ff:ff:ff 2024-03-04T09:25:43.182 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp7s0f1 2024-03-04T09:25:43.182 INFO:teuthology.orchestra.run.smithi006.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-04T09:25:43.182 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:bd:15:e4 brd ff:ff:ff:ff:ff:ff 2024-03-04T09:25:43.182 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp1s0f0 2024-03-04T09:25:43.182 INFO:teuthology.orchestra.run.smithi006.stdout: inet 172.21.15.6/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-04T09:25:43.182 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft 42591sec preferred_lft 42591sec 2024-03-04T09:25:43.183 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::ec4:7aff:febd:15e4/64 scope link noprefixroute 2024-03-04T09:25:43.183 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-04T09:25:43.183 INFO:teuthology.orchestra.run.smithi006.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-04T09:25:43.183 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:bd:15:e5 brd ff:ff:ff:ff:ff:ff 2024-03-04T09:25:43.183 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp1s0f1 2024-03-04T09:25:43.183 INFO:teuthology.orchestra.run.smithi006.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-04T09:25:43.183 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 7e:67:6d:ea:f9:ff brd ff:ff:ff:ff:ff:ff 2024-03-04T09:25:43.183 INFO:teuthology.orchestra.run.smithi006.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-04T09:25:43.183 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-04T09:25:43.183 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::c6f:adff:fefe:e00d/64 scope link proto kernel_ll 2024-03-04T09:25:43.183 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-04T09:25:43.183 INFO:teuthology.orchestra.run.smithi006.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-04T09:25:43.183 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 7e:67:6d:ea:f9:ff brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-04T09:25:43.183 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::7c67:6dff:feea:f9ff/64 scope link proto kernel_ll 2024-03-04T09:25:43.183 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-04T09:25:43.185 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:25:43.185 DEBUG:teuthology.orchestra.run.smithi006:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-04T09:25:43.276 INFO:teuthology.orchestra.run.smithi006.stdout:1 2024-03-04T09:25:43.281 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:25:43.281 DEBUG:teuthology.orchestra.run.smithi006:> route 2024-03-04T09:25:43.337 INFO:teuthology.orchestra.run.smithi006.stdout:Kernel IP routing table 2024-03-04T09:25:43.337 INFO:teuthology.orchestra.run.smithi006.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-04T09:25:43.337 INFO:teuthology.orchestra.run.smithi006.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-04T09:25:43.337 INFO:teuthology.orchestra.run.smithi006.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-04T09:25:43.337 INFO:teuthology.orchestra.run.smithi006.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-04T09:25:43.338 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:25:43.338 DEBUG:teuthology.orchestra.run.smithi006:> (cd / && exec bash -c ' 2024-03-04T09:25:43.339 DEBUG:teuthology.orchestra.run.smithi006:> set -e 2024-03-04T09:25:43.339 DEBUG:teuthology.orchestra.run.smithi006:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-04T09:25:43.339 DEBUG:teuthology.orchestra.run.smithi006:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-04T09:25:43.339 DEBUG:teuthology.orchestra.run.smithi006:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-04T09:25:43.339 DEBUG:teuthology.orchestra.run.smithi006:> ') 2024-03-04T09:25:43.538 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:25:43.538 DEBUG:teuthology.orchestra.run.smithi006:> ip netns list 2024-03-04T09:25:43.590 INFO:teuthology.orchestra.run.smithi006.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-04T09:25:43.591 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:25:43.591 DEBUG:teuthology.orchestra.run.smithi006:> ip netns list-id 2024-03-04T09:25:43.639 INFO:teuthology.orchestra.run.smithi006.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.0) 2024-03-04T09:25:43.640 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:25:43.640 DEBUG:teuthology.orchestra.run.smithi006:> (cd / && exec bash -c ' 2024-03-04T09:25:43.640 DEBUG:teuthology.orchestra.run.smithi006:> set -e 2024-03-04T09:25:43.641 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-04T09:25:43.641 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.2 1 2024-03-04T09:25:43.641 DEBUG:teuthology.orchestra.run.smithi006:> ') 2024-03-04T09:25:43.779 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:25:43.779 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-03-04T09:25:43.875 INFO:teuthology.orchestra.run.smithi006.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-04T09:25:43.875 INFO:teuthology.orchestra.run.smithi006.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-04T09:25:43.875 INFO:teuthology.orchestra.run.smithi006.stdout: inet 127.0.0.1/8 scope host lo 2024-03-04T09:25:43.875 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-04T09:25:43.875 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 ::1/128 scope host proto kernel_lo 2024-03-04T09:25:43.875 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-04T09:25:43.875 INFO:teuthology.orchestra.run.smithi006.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-04T09:25:43.875 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 16:de:d8:f7:4b:06 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-03-04T09:25:43.875 INFO:teuthology.orchestra.run.smithi006.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-03-04T09:25:43.875 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-04T09:25:43.876 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::14de:d8ff:fef7:4b06/64 scope link proto kernel_ll 2024-03-04T09:25:43.876 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-04T09:25:43.880 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:25:43.880 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-03-04T09:25:43.980 INFO:teuthology.orchestra.run.smithi006.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-04T09:25:43.980 INFO:teuthology.orchestra.run.smithi006.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-04T09:25:43.980 INFO:teuthology.orchestra.run.smithi006.stdout: inet 127.0.0.1/8 scope host lo 2024-03-04T09:25:43.980 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-04T09:25:43.980 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 ::1/128 scope host proto kernel_lo 2024-03-04T09:25:43.980 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-04T09:25:43.980 INFO:teuthology.orchestra.run.smithi006.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-04T09:25:43.980 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 16:de:d8:f7:4b:06 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-03-04T09:25:43.980 INFO:teuthology.orchestra.run.smithi006.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-03-04T09:25:43.981 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-04T09:25:43.981 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::14de:d8ff:fef7:4b06/64 scope link proto kernel_ll 2024-03-04T09:25:43.981 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-04T09:25:43.985 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.2' with 192.168.0.2/16 2024-03-04T09:25:43.985 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:25:43.985 DEBUG:teuthology.orchestra.run.smithi006:> (cd / && exec bash -c ' 2024-03-04T09:25:43.986 DEBUG:teuthology.orchestra.run.smithi006:> set -e 2024-03-04T09:25:43.986 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.2 type veth peer name brx.1 2024-03-04T09:25:43.986 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-03-04T09:25:43.986 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set veth0 up 2024-03-04T09:25:43.986 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set lo up 2024-03-04T09:25:43.986 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip route add default via 192.168.255.254 2024-03-04T09:25:43.987 DEBUG:teuthology.orchestra.run.smithi006:> ') 2024-03-04T09:25:44.282 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:25:44.283 DEBUG:teuthology.orchestra.run.smithi006:> (cd / && exec bash -c ' 2024-03-04T09:25:44.283 DEBUG:teuthology.orchestra.run.smithi006:> set -e 2024-03-04T09:25:44.283 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip link set brx.1 up 2024-03-04T09:25:44.283 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip link set dev brx.1 master ceph-brx 2024-03-04T09:25:44.283 DEBUG:teuthology.orchestra.run.smithi006:> ') 2024-03-04T09:25:44.463 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-04T09:25:44.463 DEBUG:teuthology.orchestra.run.smithi006:> mkdir -p -v /home/ubuntu/cephtest/mnt.2 2024-03-04T09:25:44.511 INFO:teuthology.orchestra.run.smithi006.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.2' 2024-03-04T09:25:44.511 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-04T09:25:44.511 DEBUG:teuthology.orchestra.run.smithi006:> chmod 0000 /home/ubuntu/cephtest/mnt.2 2024-03-04T09:25:44.557 INFO:tasks.cephfs.kernel_mount:mounting using device: 2@.cephfs=/ 2024-03-04T09:25:44.558 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:25:44.558 DEBUG:teuthology.orchestra.run.smithi006:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 2@.cephfs=/ /home/ubuntu/cephtest/mnt.2 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-04T09:25:44.750 INFO:teuthology.orchestra.run.smithi006.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-04T09:25:44.750 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-04T09:25:44.785 INFO:teuthology.orchestra.run.smithi006.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-04T09:25:44.786 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-04T09:25:44.786 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: resolved to: "172.21.15.6:3300,172.21.15.146:3300,172.21.15.146:3301" 2024-03-04T09:25:44.786 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: trying mount with new device syntax: 2@edce0578-5ae4-4402-b45a-552bdef84d7c.cephfs=/ 2024-03-04T09:25:44.786 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,ms_mode=prefer-crc,key=2,mon_addr=172.21.15.6:3300/172.21.15.146:3300/172.21.15.146:3301" will pass to kernel 2024-03-04T09:25:44.788 INFO:teuthology.orchestra.run.smithi006.stdout:mount: /home/ubuntu/cephtest/mnt.2 does not contain SELinux labels. 2024-03-04T09:25:44.788 INFO:teuthology.orchestra.run.smithi006.stdout: You just mounted a file system that supports labels which does not 2024-03-04T09:25:44.788 INFO:teuthology.orchestra.run.smithi006.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-04T09:25:44.788 INFO:teuthology.orchestra.run.smithi006.stdout: applications will generate AVC messages and not be allowed access to 2024-03-04T09:25:44.788 INFO:teuthology.orchestra.run.smithi006.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-04T09:25:44.792 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-04T09:25:44.792 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:25:44.792 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.2 2024-03-04T09:25:44.892 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:25:44.892 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest/mnt.2 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-04T09:25:44.954 INFO:teuthology.orchestra.run.smithi006.stdout:client4632 2024-03-04T09:25:44.954 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-04T09:25:44.954 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T09:25:45.008 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:25:45.008 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest/mnt.2 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-04T09:25:45.067 INFO:teuthology.orchestra.run.smithi006.stdout:client4632 2024-03-04T09:25:45.067 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:25:45.068 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest/mnt.2 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-04T09:25:45.128 INFO:teuthology.orchestra.run.smithi006.stdout:client4632 2024-03-04T09:25:45.129 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:25:45.129 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest/mnt.2 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/edce0578-5ae4-4402-b45a-552bdef84d7c.client4632/status') 2024-03-04T09:25:45.224 INFO:teuthology.orchestra.run.smithi006.stdout:instance: client.4632 (3)192.168.0.2:0/2621468437 2024-03-04T09:25:45.225 INFO:teuthology.orchestra.run.smithi006.stdout:blocklisted: false 2024-03-04T09:25:45.225 INFO:teuthology.orchestra.run.smithi006.stderr:0+1 records in 2024-03-04T09:25:45.225 INFO:teuthology.orchestra.run.smithi006.stderr:0+1 records out 2024-03-04T09:25:45.225 INFO:teuthology.orchestra.run.smithi006.stderr:69 bytes copied, 7.794e-05 s, 885 kB/s 2024-03-04T09:25:45.229 INFO:tasks.kclient:client.1 config is {'syntax': 'v2'} 2024-03-04T09:25:45.229 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-04T09:25:45.229 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-03-04T09:25:45.229 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-04T09:25:45.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:45.454+0000 7fc4f1200640 1 Processor -- start 2024-03-04T09:25:45.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:45.454+0000 7fc4f1200640 1 -- start start 2024-03-04T09:25:45.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:45.455+0000 7fc4f1200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc4ec071380 0x7fc4ec0ad090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:45.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:45.455+0000 7fc4f1200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc4ec06e240 0x7fc4ec0ad5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:45.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:45.455+0000 7fc4f1200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4ec0ce9a0 con 0x7fc4ec161670 2024-03-04T09:25:45.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:45.456+0000 7fc4f1200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4ec0730d0 con 0x7fc4ec06e240 2024-03-04T09:25:45.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:45.456+0000 7fc4f1200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4ec0cec80 con 0x7fc4ec071380 2024-03-04T09:25:45.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:45.456+0000 7fc4ea800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc4ec06e240 0x7fc4ec0ad5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:45.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:45.456+0000 7fc4ea800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc4ec06e240 0x7fc4ec0ad5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45096/0 (socket says 172.21.15.6:45096) 2024-03-04T09:25:45.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:45.456+0000 7fc4ea800640 1 -- 172.21.15.6:0/3194018242 learned_addr learned my addr 172.21.15.6:0/3194018242 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:25:45.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:45.456+0000 7fc4e9400640 1 --2- 172.21.15.6:0/3194018242 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc4ec071380 0x7fc4ec0ad090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:45.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:45.457+0000 7fc4ea800640 1 -- 172.21.15.6:0/3194018242 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc4ec071380 msgr2=0x7fc4ec0ad090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:45.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:45.457+0000 7fc4ea800640 1 --2- 172.21.15.6:0/3194018242 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc4ec071380 0x7fc4ec0ad090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:45.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:45.457+0000 7fc4ea800640 1 -- 172.21.15.6:0/3194018242 >> v1:172.21.15.146:6789/0 conn(0x7fc4ec161670 legacy=0x7fc4ec070c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:45.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:45.457+0000 7fc4ea800640 1 -- 172.21.15.6:0/3194018242 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4ec0cef60 con 0x7fc4ec06e240 2024-03-04T09:25:45.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:45.458+0000 7fc4ea800640 1 --2- 172.21.15.6:0/3194018242 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc4ec06e240 0x7fc4ec0ad5d0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fc4e4009870 tx=0x7fc4e402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c9cf7763802a1c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:45.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:45.458+0000 7fc4e8a00640 1 -- 172.21.15.6:0/3194018242 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4e4034070 con 0x7fc4ec06e240 2024-03-04T09:25:45.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:45.458+0000 7fc4e8a00640 1 -- 172.21.15.6:0/3194018242 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4e4030840 con 0x7fc4ec06e240 2024-03-04T09:25:45.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:45.458+0000 7fc4e8a00640 1 -- 172.21.15.6:0/3194018242 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4e4030c10 con 0x7fc4ec06e240 2024-03-04T09:25:45.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:45.459+0000 7fc4f1200640 1 -- 172.21.15.6:0/3194018242 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc4ec06e240 msgr2=0x7fc4ec0ad5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:45.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:45.459+0000 7fc4f1200640 1 --2- 172.21.15.6:0/3194018242 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc4ec06e240 0x7fc4ec0ad5d0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fc4e4009870 tx=0x7fc4e402b980 comp rx=0 tx=0).stop 2024-03-04T09:25:45.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:45.459+0000 7fc4f1200640 1 -- 172.21.15.6:0/3194018242 shutdown_connections 2024-03-04T09:25:45.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:45.459+0000 7fc4f1200640 1 --2- 172.21.15.6:0/3194018242 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc4ec06e240 0x7fc4ec0ad5d0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:45.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:45.459+0000 7fc4f1200640 1 --2- 172.21.15.6:0/3194018242 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc4ec071380 0x7fc4ec0ad090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:45.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:45.459+0000 7fc4f1200640 1 -- 172.21.15.6:0/3194018242 >> 172.21.15.6:0/3194018242 conn(0x7fc4ec06afb0 msgr2=0x7fc4ec06b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:45.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:45.459+0000 7fc4f1200640 1 -- 172.21.15.6:0/3194018242 shutdown_connections 2024-03-04T09:25:45.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:45.459+0000 7fc4f1200640 1 -- 172.21.15.6:0/3194018242 wait complete. 2024-03-04T09:25:45.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:45.461+0000 7fc4f1200640 1 Processor -- start 2024-03-04T09:25:45.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:45.461+0000 7fc4f1200640 1 -- start start 2024-03-04T09:25:45.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:45.462+0000 7fc4f1200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc4ec06e240 0x7fc4ec09aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:45.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:45.462+0000 7fc4f1200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc4ec071380 0x7fc4ec0b03e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:45.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:45.462+0000 7fc4f1200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc4ec161670 0x7fc4ec0cb860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:45.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:45.462+0000 7fc4f1200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4ec0ce600 con 0x7fc4ec071380 2024-03-04T09:25:45.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:45.462+0000 7fc4f1200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4ec0cf520 con 0x7fc4ec161670 2024-03-04T09:25:45.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:45.462+0000 7fc4f1200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4ec0ce780 con 0x7fc4ec06e240 2024-03-04T09:25:45.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:45.462+0000 7fc4e9400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc4ec071380 0x7fc4ec0b03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:45.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:45.462+0000 7fc4e9e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc4ec06e240 0x7fc4ec09aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:45.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:45.463+0000 7fc4e9400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc4ec071380 0x7fc4ec0b03e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45110/0 (socket says 172.21.15.6:45110) 2024-03-04T09:25:45.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:45.463+0000 7fc4e9400640 1 -- 172.21.15.6:0/712992418 learned_addr learned my addr 172.21.15.6:0/712992418 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:25:45.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:45.463+0000 7fc4ea800640 1 --2- 172.21.15.6:0/712992418 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc4ec161670 0x7fc4ec0cb860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:45.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:45.463+0000 7fc4e9400640 1 -- 172.21.15.6:0/712992418 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc4ec06e240 msgr2=0x7fc4ec09aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:45.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:45.463+0000 7fc4e9400640 1 --2- 172.21.15.6:0/712992418 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc4ec06e240 0x7fc4ec09aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:45.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:45.463+0000 7fc4e9400640 1 -- 172.21.15.6:0/712992418 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc4ec161670 msgr2=0x7fc4ec0cb860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:45.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:45.463+0000 7fc4e9400640 1 --2- 172.21.15.6:0/712992418 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc4ec161670 0x7fc4ec0cb860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:45.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:45.463+0000 7fc4e9400640 1 -- 172.21.15.6:0/712992418 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4ec0ca7c0 con 0x7fc4ec071380 2024-03-04T09:25:45.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:45.463+0000 7fc4e9400640 1 --2- 172.21.15.6:0/712992418 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc4ec071380 0x7fc4ec0b03e0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fc4d400e9b0 tx=0x7fc4d400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:45.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:45.464+0000 7fc4e2000640 1 -- 172.21.15.6:0/712992418 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4d4017070 con 0x7fc4ec071380 2024-03-04T09:25:45.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:45.464+0000 7fc4e2000640 1 -- 172.21.15.6:0/712992418 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4d400c9a0 con 0x7fc4ec071380 2024-03-04T09:25:45.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:45.464+0000 7fc4e2000640 1 -- 172.21.15.6:0/712992418 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4d400cd90 con 0x7fc4ec071380 2024-03-04T09:25:45.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:45.464+0000 7fc4f1200640 1 -- 172.21.15.6:0/712992418 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4ec0caaa0 con 0x7fc4ec071380 2024-03-04T09:25:45.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:45.464+0000 7fc4f1200640 1 -- 172.21.15.6:0/712992418 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4ec0cb060 con 0x7fc4ec071380 2024-03-04T09:25:45.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:45.465+0000 7fc4f1200640 1 -- 172.21.15.6:0/712992418 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4ec069510 con 0x7fc4ec071380 2024-03-04T09:25:45.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:45.466+0000 7fc4e2000640 1 -- 172.21.15.6:0/712992418 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc4d401a070 con 0x7fc4ec071380 2024-03-04T09:25:45.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:45.467+0000 7fc4e2000640 1 --2- 172.21.15.6:0/712992418 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc4c806b4d0 0x7fc4c806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:45.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:45.467+0000 7fc4e2000640 1 -- 172.21.15.6:0/712992418 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc4d408a5d0 con 0x7fc4ec071380 2024-03-04T09:25:45.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:45.467+0000 7fc4e9e00640 1 --2- 172.21.15.6:0/712992418 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc4c806b4d0 0x7fc4c806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:45.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:45.468+0000 7fc4e9e00640 1 --2- 172.21.15.6:0/712992418 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc4c806b4d0 0x7fc4c806d990 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fc4dc001e70 tx=0x7fc4dc009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:45.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:45.471+0000 7fc4e2000640 1 -- 172.21.15.6:0/712992418 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc4d4058d60 con 0x7fc4ec071380 2024-03-04T09:25:45.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:45.719+0000 7fc4f1200640 1 -- 172.21.15.6:0/712992418 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fc4ec071010 con 0x7fc4ec071380 2024-03-04T09:25:45.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:45.720+0000 7fc4e2000640 1 -- 172.21.15.6:0/712992418 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v7) v1 ==== 53+0+73 (secure 0 0 0) 0x7fc4d405ce00 con 0x7fc4ec071380 2024-03-04T09:25:45.720 INFO:teuthology.orchestra.run.smithi006.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-04T09:25:45.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:45.723+0000 7fc4f1200640 1 -- 172.21.15.6:0/712992418 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc4c806b4d0 msgr2=0x7fc4c806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:45.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:45.723+0000 7fc4f1200640 1 --2- 172.21.15.6:0/712992418 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc4c806b4d0 0x7fc4c806d990 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fc4dc001e70 tx=0x7fc4dc009040 comp rx=0 tx=0).stop 2024-03-04T09:25:45.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:45.723+0000 7fc4f1200640 1 -- 172.21.15.6:0/712992418 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc4ec071380 msgr2=0x7fc4ec0b03e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:45.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:45.723+0000 7fc4f1200640 1 --2- 172.21.15.6:0/712992418 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc4ec071380 0x7fc4ec0b03e0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fc4d400e9b0 tx=0x7fc4d400ee70 comp rx=0 tx=0).stop 2024-03-04T09:25:45.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:45.724+0000 7fc4f1200640 1 -- 172.21.15.6:0/712992418 shutdown_connections 2024-03-04T09:25:45.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:45.724+0000 7fc4f1200640 1 --2- 172.21.15.6:0/712992418 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc4ec161670 0x7fc4ec0cb860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:45.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:45.724+0000 7fc4f1200640 1 --2- 172.21.15.6:0/712992418 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc4ec071380 0x7fc4ec0b03e0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:45.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:45.724+0000 7fc4f1200640 1 --2- 172.21.15.6:0/712992418 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc4c806b4d0 0x7fc4c806d990 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:45.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:45.724+0000 7fc4f1200640 1 --2- 172.21.15.6:0/712992418 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc4ec06e240 0x7fc4ec09aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:45.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:45.724+0000 7fc4f1200640 1 -- 172.21.15.6:0/712992418 >> 172.21.15.6:0/712992418 conn(0x7fc4ec06afb0 msgr2=0x7fc4ec0cc7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:45.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:45.724+0000 7fc4f1200640 1 -- 172.21.15.6:0/712992418 shutdown_connections 2024-03-04T09:25:45.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:45.724+0000 7fc4f1200640 1 -- 172.21.15.6:0/712992418 wait complete. 2024-03-04T09:25:45.741 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-04T09:25:45.741 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-04T09:25:45.741 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi146.front.sepia.ceph.com 2024-03-04T09:25:45.741 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-03-04T09:25:45.741 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-03-04T09:25:45.741 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-04T09:25:45.741 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-04T09:25:45.741 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-04T09:25:45.741 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-03-04T09:25:45.742 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:25:45.742 DEBUG:teuthology.orchestra.run.smithi146:> ip addr 2024-03-04T09:25:45.796 INFO:teuthology.orchestra.run.smithi146.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-04T09:25:45.797 INFO:teuthology.orchestra.run.smithi146.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-04T09:25:45.797 INFO:teuthology.orchestra.run.smithi146.stdout: inet 127.0.0.1/8 scope host lo 2024-03-04T09:25:45.797 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-03-04T09:25:45.797 INFO:teuthology.orchestra.run.smithi146.stdout: inet6 ::1/128 scope host 2024-03-04T09:25:45.797 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-03-04T09:25:45.797 INFO:teuthology.orchestra.run.smithi146.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-04T09:25:45.797 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 0c:c4:7a:d9:42:6e brd ff:ff:ff:ff:ff:ff 2024-03-04T09:25:45.797 INFO:teuthology.orchestra.run.smithi146.stdout: altname enp7s0f0 2024-03-04T09:25:45.797 INFO:teuthology.orchestra.run.smithi146.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-04T09:25:45.797 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 0c:c4:7a:d9:42:6f brd ff:ff:ff:ff:ff:ff 2024-03-04T09:25:45.797 INFO:teuthology.orchestra.run.smithi146.stdout: altname enp7s0f1 2024-03-04T09:25:45.797 INFO:teuthology.orchestra.run.smithi146.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-04T09:25:45.797 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 0c:c4:7a:88:76:fc brd ff:ff:ff:ff:ff:ff 2024-03-04T09:25:45.798 INFO:teuthology.orchestra.run.smithi146.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-04T09:25:45.798 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 0c:c4:7a:88:76:fd brd ff:ff:ff:ff:ff:ff 2024-03-04T09:25:45.798 INFO:teuthology.orchestra.run.smithi146.stdout: inet 172.21.15.146/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-04T09:25:45.798 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft 42590sec preferred_lft 42590sec 2024-03-04T09:25:45.798 INFO:teuthology.orchestra.run.smithi146.stdout: inet6 fe80::ec4:7aff:fe88:76fd/64 scope link proto kernel_ll 2024-03-04T09:25:45.798 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-03-04T09:25:45.798 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-04T09:25:45.798 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:25:45.798 DEBUG:teuthology.orchestra.run.smithi146:> (cd / && exec bash -c ' 2024-03-04T09:25:45.798 DEBUG:teuthology.orchestra.run.smithi146:> set -e 2024-03-04T09:25:45.799 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip link add name ceph-brx type bridge 2024-03-04T09:25:45.799 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip addr flush dev ceph-brx 2024-03-04T09:25:45.799 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip link set ceph-brx up 2024-03-04T09:25:45.799 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-04T09:25:45.799 DEBUG:teuthology.orchestra.run.smithi146:> ') 2024-03-04T09:25:46.204 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:25:46.204 DEBUG:teuthology.orchestra.run.smithi146:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-04T09:25:46.279 INFO:teuthology.orchestra.run.smithi146.stdout:1 2024-03-04T09:25:46.283 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:25:46.284 DEBUG:teuthology.orchestra.run.smithi146:> route 2024-03-04T09:25:46.332 INFO:teuthology.orchestra.run.smithi146.stdout:Kernel IP routing table 2024-03-04T09:25:46.332 INFO:teuthology.orchestra.run.smithi146.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-04T09:25:46.332 INFO:teuthology.orchestra.run.smithi146.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-04T09:25:46.332 INFO:teuthology.orchestra.run.smithi146.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-04T09:25:46.332 INFO:teuthology.orchestra.run.smithi146.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-04T09:25:46.332 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:25:46.333 DEBUG:teuthology.orchestra.run.smithi146:> (cd / && exec bash -c ' 2024-03-04T09:25:46.333 DEBUG:teuthology.orchestra.run.smithi146:> set -e 2024-03-04T09:25:46.333 DEBUG:teuthology.orchestra.run.smithi146:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-04T09:25:46.333 DEBUG:teuthology.orchestra.run.smithi146:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-04T09:25:46.333 DEBUG:teuthology.orchestra.run.smithi146:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-04T09:25:46.333 DEBUG:teuthology.orchestra.run.smithi146:> ') 2024-03-04T09:25:47.022 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:25:47.023 DEBUG:teuthology.orchestra.run.smithi146:> ip netns list 2024-03-04T09:25:47.073 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:25:47.073 DEBUG:teuthology.orchestra.run.smithi146:> ip netns list-id 2024-03-04T09:25:47.117 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:25:47.117 DEBUG:teuthology.orchestra.run.smithi146:> (cd / && exec bash -c ' 2024-03-04T09:25:47.117 DEBUG:teuthology.orchestra.run.smithi146:> set -e 2024-03-04T09:25:47.117 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-04T09:25:47.118 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-03-04T09:25:47.118 DEBUG:teuthology.orchestra.run.smithi146:> ') 2024-03-04T09:25:47.294 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-03-04T09:25:47.294 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:25:47.295 DEBUG:teuthology.orchestra.run.smithi146:> (cd / && exec bash -c ' 2024-03-04T09:25:47.295 DEBUG:teuthology.orchestra.run.smithi146:> set -e 2024-03-04T09:25:47.295 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-03-04T09:25:47.295 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-04T09:25:47.295 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-03-04T09:25:47.295 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-03-04T09:25:47.295 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-03-04T09:25:47.295 DEBUG:teuthology.orchestra.run.smithi146:> ') 2024-03-04T09:25:47.558 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:25:47.558 DEBUG:teuthology.orchestra.run.smithi146:> (cd / && exec bash -c ' 2024-03-04T09:25:47.558 DEBUG:teuthology.orchestra.run.smithi146:> set -e 2024-03-04T09:25:47.558 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip link set brx.0 up 2024-03-04T09:25:47.558 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip link set dev brx.0 master ceph-brx 2024-03-04T09:25:47.558 DEBUG:teuthology.orchestra.run.smithi146:> ') 2024-03-04T09:25:47.725 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-04T09:25:47.725 DEBUG:teuthology.orchestra.run.smithi146:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-03-04T09:25:47.772 INFO:teuthology.orchestra.run.smithi146.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-03-04T09:25:47.773 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-04T09:25:47.773 DEBUG:teuthology.orchestra.run.smithi146:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-03-04T09:25:47.864 INFO:tasks.cephfs.kernel_mount:mounting using device: 1@.cephfs=/ 2024-03-04T09:25:47.865 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:25:47.865 DEBUG:teuthology.orchestra.run.smithi146:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 1@.cephfs=/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-04T09:25:48.008 INFO:teuthology.orchestra.run.smithi146.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-04T09:25:48.008 INFO:teuthology.orchestra.run.smithi146.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-04T09:25:48.681 INFO:teuthology.orchestra.run.smithi146.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-04T09:25:48.682 INFO:teuthology.orchestra.run.smithi146.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-04T09:25:48.682 INFO:teuthology.orchestra.run.smithi146.stdout:mount.ceph: resolved to: "172.21.15.6:3300,172.21.15.146:3300,172.21.15.146:3301" 2024-03-04T09:25:48.682 INFO:teuthology.orchestra.run.smithi146.stdout:mount.ceph: trying mount with new device syntax: 1@edce0578-5ae4-4402-b45a-552bdef84d7c.cephfs=/ 2024-03-04T09:25:48.682 INFO:teuthology.orchestra.run.smithi146.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,ms_mode=prefer-crc,key=1,mon_addr=172.21.15.6:3300/172.21.15.146:3300/172.21.15.146:3301" will pass to kernel 2024-03-04T09:25:48.685 INFO:teuthology.orchestra.run.smithi146.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2024-03-04T09:25:48.685 INFO:teuthology.orchestra.run.smithi146.stdout: You just mounted a file system that supports labels which does not 2024-03-04T09:25:48.685 INFO:teuthology.orchestra.run.smithi146.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-04T09:25:48.685 INFO:teuthology.orchestra.run.smithi146.stdout: applications will generate AVC messages and not be allowed access to 2024-03-04T09:25:48.685 INFO:teuthology.orchestra.run.smithi146.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-04T09:25:48.690 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-04T09:25:48.690 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:25:48.690 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-03-04T09:25:48.773 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:25:48.774 DEBUG:teuthology.orchestra.run.smithi146:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-04T09:25:48.828 INFO:teuthology.orchestra.run.smithi146.stdout:client4502 2024-03-04T09:25:48.828 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-04T09:25:48.828 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T09:25:48.871 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:25:48.871 DEBUG:teuthology.orchestra.run.smithi146:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-04T09:25:48.966 INFO:teuthology.orchestra.run.smithi146.stdout:client4502 2024-03-04T09:25:48.967 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:25:48.967 DEBUG:teuthology.orchestra.run.smithi146:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-04T09:25:49.013 INFO:teuthology.orchestra.run.smithi146.stdout:client4502 2024-03-04T09:25:49.013 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:25:49.013 DEBUG:teuthology.orchestra.run.smithi146:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/edce0578-5ae4-4402-b45a-552bdef84d7c.client4502/status') 2024-03-04T09:25:49.147 INFO:teuthology.orchestra.run.smithi146.stdout:instance: client.4502 (3)192.168.0.1:0/3306367326 2024-03-04T09:25:49.147 INFO:teuthology.orchestra.run.smithi146.stdout:blocklisted: false 2024-03-04T09:25:49.147 INFO:teuthology.orchestra.run.smithi146.stderr:0+1 records in 2024-03-04T09:25:49.148 INFO:teuthology.orchestra.run.smithi146.stderr:0+1 records out 2024-03-04T09:25:49.148 INFO:teuthology.orchestra.run.smithi146.stderr:69 bytes copied, 6.622e-05 s, 1.0 MB/s 2024-03-04T09:25:49.152 INFO:tasks.kclient:client.3 config is {'syntax': 'v2'} 2024-03-04T09:25:49.152 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-04T09:25:49.152 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-03-04T09:25:49.153 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-04T09:25:49.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:49.388+0000 7f8853600640 1 Processor -- start 2024-03-04T09:25:49.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:49.388+0000 7f8853600640 1 -- start start 2024-03-04T09:25:49.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:49.390+0000 7f8853600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f884c0cf5c0 0x7f884c0cdc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:49.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:49.390+0000 7f8853600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f884c071050 0x7f884c071490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:49.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:49.390+0000 7f8853600640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f884c0b4860 con 0x7f884c0b3c90 2024-03-04T09:25:49.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:49.390+0000 7f8853600640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f884c0ce160 con 0x7f884c0cf5c0 2024-03-04T09:25:49.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:49.390+0000 7f8853600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f884c0b4b40 con 0x7f884c071050 2024-03-04T09:25:49.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:49.390+0000 7f8850c00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f884c0b3c90 0x7f884c0b4070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:58642/0 (socket says 172.21.15.6:58642) 2024-03-04T09:25:49.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:49.390+0000 7f8850c00640 1 -- 172.21.15.6:0/1294377959 learned_addr learned my addr 172.21.15.6:0/1294377959 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:25:49.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:49.390+0000 7f884be00640 1 --2- 172.21.15.6:0/1294377959 >> v2:172.21.15.6:3300/0 conn(0x7f884c0cf5c0 0x7f884c0cdc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:49.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:49.391+0000 7f8851600640 1 --2- 172.21.15.6:0/1294377959 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f884c071050 0x7f884c071490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:49.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:49.391+0000 7f884be00640 1 -- 172.21.15.6:0/1294377959 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f884c071050 msgr2=0x7f884c071490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:49.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:49.391+0000 7f884be00640 1 --2- 172.21.15.6:0/1294377959 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f884c071050 0x7f884c071490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:49.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:49.391+0000 7f884be00640 1 -- 172.21.15.6:0/1294377959 >> v1:172.21.15.6:6789/0 conn(0x7f884c0b3c90 legacy=0x7f884c0b4070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:49.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:49.391+0000 7f884be00640 1 -- 172.21.15.6:0/1294377959 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f884c0b4e20 con 0x7f884c0cf5c0 2024-03-04T09:25:49.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:49.391+0000 7f884b400640 1 -- 172.21.15.6:0/1294377959 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2336839724 0 0) 0x7f884c0b4860 con 0x7f884c0b3c90 2024-03-04T09:25:49.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:49.391+0000 7f8851600640 1 --2- 172.21.15.6:0/1294377959 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f884c071050 0x7f884c071490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:25:49.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:49.392+0000 7f884be00640 1 --2- 172.21.15.6:0/1294377959 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f884c0cf5c0 0x7f884c0cdc20 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f88380097a0 tx=0x7f883802b940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e21e1e9ec663af64 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:49.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:49.392+0000 7f884b400640 1 -- 172.21.15.6:0/1294377959 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8838034070 con 0x7f884c0cf5c0 2024-03-04T09:25:49.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:49.392+0000 7f884b400640 1 -- 172.21.15.6:0/1294377959 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8838030800 con 0x7f884c0cf5c0 2024-03-04T09:25:49.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:49.392+0000 7f884b400640 1 -- 172.21.15.6:0/1294377959 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8838030bd0 con 0x7f884c0cf5c0 2024-03-04T09:25:49.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:49.393+0000 7f8853600640 1 -- 172.21.15.6:0/1294377959 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f884c0cf5c0 msgr2=0x7f884c0cdc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:49.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:49.393+0000 7f8853600640 1 --2- 172.21.15.6:0/1294377959 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f884c0cf5c0 0x7f884c0cdc20 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f88380097a0 tx=0x7f883802b940 comp rx=0 tx=0).stop 2024-03-04T09:25:49.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:49.393+0000 7f8853600640 1 -- 172.21.15.6:0/1294377959 shutdown_connections 2024-03-04T09:25:49.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:49.393+0000 7f8853600640 1 --2- 172.21.15.6:0/1294377959 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f884c071050 0x7f884c071490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:49.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:49.393+0000 7f8853600640 1 --2- 172.21.15.6:0/1294377959 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f884c0cf5c0 0x7f884c0cdc20 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:49.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:49.393+0000 7f8853600640 1 -- 172.21.15.6:0/1294377959 >> 172.21.15.6:0/1294377959 conn(0x7f884c06afb0 msgr2=0x7f884c06b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:49.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:49.393+0000 7f8853600640 1 -- 172.21.15.6:0/1294377959 shutdown_connections 2024-03-04T09:25:49.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:49.393+0000 7f8853600640 1 -- 172.21.15.6:0/1294377959 wait complete. 2024-03-04T09:25:49.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:49.395+0000 7f8853600640 1 Processor -- start 2024-03-04T09:25:49.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:49.395+0000 7f8853600640 1 -- start start 2024-03-04T09:25:49.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:49.396+0000 7f8853600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f884c071050 0x7f884c146ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:49.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:49.396+0000 7f8853600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f884c0b3c90 0x7f884c153140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:49.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:49.396+0000 7f8853600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f884c0cf5c0 0x7f884c06ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:49.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:49.396+0000 7f8853600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f884c0719d0 con 0x7f884c0b3c90 2024-03-04T09:25:49.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:49.396+0000 7f8853600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f884c15b340 con 0x7f884c071050 2024-03-04T09:25:49.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:49.396+0000 7f8853600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f884c071b50 con 0x7f884c0cf5c0 2024-03-04T09:25:49.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:49.396+0000 7f884be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f884c0b3c90 0x7f884c153140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:49.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:49.396+0000 7f884be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f884c0b3c90 0x7f884c153140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51836/0 (socket says 172.21.15.6:51836) 2024-03-04T09:25:49.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:49.396+0000 7f884be00640 1 -- 172.21.15.6:0/469681689 learned_addr learned my addr 172.21.15.6:0/469681689 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:25:49.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:49.397+0000 7f8850c00640 1 --2- 172.21.15.6:0/469681689 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f884c071050 0x7f884c146ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:49.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:49.397+0000 7f8851600640 1 --2- 172.21.15.6:0/469681689 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f884c0cf5c0 0x7f884c06ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:49.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:49.397+0000 7f884be00640 1 -- 172.21.15.6:0/469681689 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f884c0cf5c0 msgr2=0x7f884c06ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:49.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:49.397+0000 7f884be00640 1 --2- 172.21.15.6:0/469681689 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f884c0cf5c0 0x7f884c06ccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:49.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:49.397+0000 7f884be00640 1 -- 172.21.15.6:0/469681689 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f884c071050 msgr2=0x7f884c146ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:49.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:49.397+0000 7f884be00640 1 --2- 172.21.15.6:0/469681689 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f884c071050 0x7f884c146ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:49.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:49.397+0000 7f884be00640 1 -- 172.21.15.6:0/469681689 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f884c153680 con 0x7f884c0b3c90 2024-03-04T09:25:49.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:49.397+0000 7f884be00640 1 --2- 172.21.15.6:0/469681689 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f884c0b3c90 0x7f884c153140 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f8838000c00 tx=0x7f88380046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:49.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:49.398+0000 7f8848c00640 1 -- 172.21.15.6:0/469681689 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8838047070 con 0x7f884c0b3c90 2024-03-04T09:25:49.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:49.398+0000 7f8848c00640 1 -- 172.21.15.6:0/469681689 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8838030600 con 0x7f884c0b3c90 2024-03-04T09:25:49.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:49.398+0000 7f8853600640 1 -- 172.21.15.6:0/469681689 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f884c147970 con 0x7f884c0b3c90 2024-03-04T09:25:49.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:49.398+0000 7f8848c00640 1 -- 172.21.15.6:0/469681689 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8838034040 con 0x7f884c0b3c90 2024-03-04T09:25:49.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:49.398+0000 7f8853600640 1 -- 172.21.15.6:0/469681689 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f884c147ef0 con 0x7f884c0b3c90 2024-03-04T09:25:49.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:49.399+0000 7f8853600640 1 -- 172.21.15.6:0/469681689 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f884c069510 con 0x7f884c0b3c90 2024-03-04T09:25:49.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:49.400+0000 7f8848c00640 1 -- 172.21.15.6:0/469681689 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f883804a070 con 0x7f884c0b3c90 2024-03-04T09:25:49.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:49.401+0000 7f8848c00640 1 --2- 172.21.15.6:0/469681689 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f882c06b4d0 0x7f882c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:49.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:49.401+0000 7f8848c00640 1 -- 172.21.15.6:0/469681689 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f88380abc80 con 0x7f884c0b3c90 2024-03-04T09:25:49.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:49.401+0000 7f8850c00640 1 --2- 172.21.15.6:0/469681689 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f882c06b4d0 0x7f882c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:49.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:49.402+0000 7f8850c00640 1 --2- 172.21.15.6:0/469681689 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f882c06b4d0 0x7f882c06d990 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f8840000f60 tx=0x7f8840007b70 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:49.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:49.405+0000 7f8848c00640 1 -- 172.21.15.6:0/469681689 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f883807ab80 con 0x7f884c0b3c90 2024-03-04T09:25:49.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:49.648+0000 7f8853600640 1 -- 172.21.15.6:0/469681689 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f884c09de90 con 0x7f884c0b3c90 2024-03-04T09:25:49.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:49.649+0000 7f8848c00640 1 -- 172.21.15.6:0/469681689 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v7) v1 ==== 53+0+73 (secure 0 0 0) 0x7f883807a480 con 0x7f884c0b3c90 2024-03-04T09:25:49.645 INFO:teuthology.orchestra.run.smithi006.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-04T09:25:49.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:49.652+0000 7f8853600640 1 -- 172.21.15.6:0/469681689 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f882c06b4d0 msgr2=0x7f882c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:49.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:49.652+0000 7f8853600640 1 --2- 172.21.15.6:0/469681689 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f882c06b4d0 0x7f882c06d990 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f8840000f60 tx=0x7f8840007b70 comp rx=0 tx=0).stop 2024-03-04T09:25:49.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:49.652+0000 7f8853600640 1 -- 172.21.15.6:0/469681689 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f884c0b3c90 msgr2=0x7f884c153140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:49.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:49.652+0000 7f8853600640 1 --2- 172.21.15.6:0/469681689 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f884c0b3c90 0x7f884c153140 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f8838000c00 tx=0x7f88380046c0 comp rx=0 tx=0).stop 2024-03-04T09:25:49.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:49.652+0000 7f8853600640 1 -- 172.21.15.6:0/469681689 shutdown_connections 2024-03-04T09:25:49.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:49.652+0000 7f8853600640 1 --2- 172.21.15.6:0/469681689 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f882c06b4d0 0x7f882c06d990 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:49.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:49.652+0000 7f8853600640 1 --2- 172.21.15.6:0/469681689 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f884c0cf5c0 0x7f884c06ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:49.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:49.652+0000 7f8853600640 1 --2- 172.21.15.6:0/469681689 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f884c0b3c90 0x7f884c153140 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:49.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:49.652+0000 7f8853600640 1 --2- 172.21.15.6:0/469681689 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f884c071050 0x7f884c146ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:49.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:49.652+0000 7f8853600640 1 -- 172.21.15.6:0/469681689 >> 172.21.15.6:0/469681689 conn(0x7f884c06afb0 msgr2=0x7f884c0ce750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:49.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:49.652+0000 7f8853600640 1 -- 172.21.15.6:0/469681689 shutdown_connections 2024-03-04T09:25:49.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:49.652+0000 7f8853600640 1 -- 172.21.15.6:0/469681689 wait complete. 2024-03-04T09:25:49.670 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-04T09:25:49.670 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-04T09:25:49.670 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi146.front.sepia.ceph.com 2024-03-04T09:25:49.670 INFO:tasks.cephfs.mount:self.client.name = client.3 2024-03-04T09:25:49.670 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-03-04T09:25:49.670 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-04T09:25:49.671 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-04T09:25:49.671 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-04T09:25:49.671 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.3' 2024-03-04T09:25:49.671 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:25:49.671 DEBUG:teuthology.orchestra.run.smithi146:> ip addr 2024-03-04T09:25:49.725 INFO:teuthology.orchestra.run.smithi146.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-04T09:25:49.725 INFO:teuthology.orchestra.run.smithi146.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-04T09:25:49.725 INFO:teuthology.orchestra.run.smithi146.stdout: inet 127.0.0.1/8 scope host lo 2024-03-04T09:25:49.725 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-03-04T09:25:49.725 INFO:teuthology.orchestra.run.smithi146.stdout: inet6 ::1/128 scope host 2024-03-04T09:25:49.726 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-03-04T09:25:49.726 INFO:teuthology.orchestra.run.smithi146.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-04T09:25:49.726 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 0c:c4:7a:d9:42:6e brd ff:ff:ff:ff:ff:ff 2024-03-04T09:25:49.726 INFO:teuthology.orchestra.run.smithi146.stdout: altname enp7s0f0 2024-03-04T09:25:49.726 INFO:teuthology.orchestra.run.smithi146.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-04T09:25:49.726 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 0c:c4:7a:d9:42:6f brd ff:ff:ff:ff:ff:ff 2024-03-04T09:25:49.726 INFO:teuthology.orchestra.run.smithi146.stdout: altname enp7s0f1 2024-03-04T09:25:49.726 INFO:teuthology.orchestra.run.smithi146.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-04T09:25:49.726 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 0c:c4:7a:88:76:fc brd ff:ff:ff:ff:ff:ff 2024-03-04T09:25:49.726 INFO:teuthology.orchestra.run.smithi146.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-04T09:25:49.726 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 0c:c4:7a:88:76:fd brd ff:ff:ff:ff:ff:ff 2024-03-04T09:25:49.726 INFO:teuthology.orchestra.run.smithi146.stdout: inet 172.21.15.146/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-04T09:25:49.727 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft 42586sec preferred_lft 42586sec 2024-03-04T09:25:49.727 INFO:teuthology.orchestra.run.smithi146.stdout: inet6 fe80::ec4:7aff:fe88:76fd/64 scope link proto kernel_ll 2024-03-04T09:25:49.727 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-03-04T09:25:49.727 INFO:teuthology.orchestra.run.smithi146.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-04T09:25:49.727 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 7a:22:73:17:ea:5a brd ff:ff:ff:ff:ff:ff 2024-03-04T09:25:49.727 INFO:teuthology.orchestra.run.smithi146.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-04T09:25:49.727 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-03-04T09:25:49.727 INFO:teuthology.orchestra.run.smithi146.stdout: inet6 fe80::80fa:26ff:fe06:97d/64 scope link proto kernel_ll 2024-03-04T09:25:49.727 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-03-04T09:25:49.727 INFO:teuthology.orchestra.run.smithi146.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-04T09:25:49.727 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 7a:22:73:17:ea:5a brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-04T09:25:49.727 INFO:teuthology.orchestra.run.smithi146.stdout: inet6 fe80::7822:73ff:fe17:ea5a/64 scope link proto kernel_ll 2024-03-04T09:25:49.727 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-03-04T09:25:49.729 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:25:49.729 DEBUG:teuthology.orchestra.run.smithi146:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-04T09:25:49.812 INFO:teuthology.orchestra.run.smithi146.stdout:1 2024-03-04T09:25:49.816 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:25:49.817 DEBUG:teuthology.orchestra.run.smithi146:> route 2024-03-04T09:25:49.871 INFO:teuthology.orchestra.run.smithi146.stdout:Kernel IP routing table 2024-03-04T09:25:49.871 INFO:teuthology.orchestra.run.smithi146.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-04T09:25:49.871 INFO:teuthology.orchestra.run.smithi146.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-04T09:25:49.871 INFO:teuthology.orchestra.run.smithi146.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-04T09:25:49.871 INFO:teuthology.orchestra.run.smithi146.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-04T09:25:49.872 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:25:49.872 DEBUG:teuthology.orchestra.run.smithi146:> (cd / && exec bash -c ' 2024-03-04T09:25:49.872 DEBUG:teuthology.orchestra.run.smithi146:> set -e 2024-03-04T09:25:49.873 DEBUG:teuthology.orchestra.run.smithi146:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-04T09:25:49.873 DEBUG:teuthology.orchestra.run.smithi146:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-04T09:25:49.873 DEBUG:teuthology.orchestra.run.smithi146:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-04T09:25:49.873 DEBUG:teuthology.orchestra.run.smithi146:> ') 2024-03-04T09:25:50.044 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:25:50.044 DEBUG:teuthology.orchestra.run.smithi146:> ip netns list 2024-03-04T09:25:50.094 INFO:teuthology.orchestra.run.smithi146.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-03-04T09:25:50.095 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:25:50.096 DEBUG:teuthology.orchestra.run.smithi146:> ip netns list-id 2024-03-04T09:25:50.144 INFO:teuthology.orchestra.run.smithi146.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.1) 2024-03-04T09:25:50.145 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:25:50.145 DEBUG:teuthology.orchestra.run.smithi146:> (cd / && exec bash -c ' 2024-03-04T09:25:50.145 DEBUG:teuthology.orchestra.run.smithi146:> set -e 2024-03-04T09:25:50.145 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-04T09:25:50.145 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.3 1 2024-03-04T09:25:50.145 DEBUG:teuthology.orchestra.run.smithi146:> ') 2024-03-04T09:25:50.329 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:25:50.329 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-03-04T09:25:50.418 INFO:teuthology.orchestra.run.smithi146.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-04T09:25:50.418 INFO:teuthology.orchestra.run.smithi146.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-04T09:25:50.419 INFO:teuthology.orchestra.run.smithi146.stdout: inet 127.0.0.1/8 scope host lo 2024-03-04T09:25:50.419 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-03-04T09:25:50.419 INFO:teuthology.orchestra.run.smithi146.stdout: inet6 ::1/128 scope host proto kernel_lo 2024-03-04T09:25:50.419 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-03-04T09:25:50.419 INFO:teuthology.orchestra.run.smithi146.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-04T09:25:50.419 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 12:ec:0d:29:9e:5d brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-03-04T09:25:50.419 INFO:teuthology.orchestra.run.smithi146.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-03-04T09:25:50.419 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-03-04T09:25:50.419 INFO:teuthology.orchestra.run.smithi146.stdout: inet6 fe80::10ec:dff:fe29:9e5d/64 scope link proto kernel_ll 2024-03-04T09:25:50.419 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-03-04T09:25:50.424 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:25:50.424 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-03-04T09:25:50.521 INFO:teuthology.orchestra.run.smithi146.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-04T09:25:50.522 INFO:teuthology.orchestra.run.smithi146.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-04T09:25:50.522 INFO:teuthology.orchestra.run.smithi146.stdout: inet 127.0.0.1/8 scope host lo 2024-03-04T09:25:50.522 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-03-04T09:25:50.522 INFO:teuthology.orchestra.run.smithi146.stdout: inet6 ::1/128 scope host proto kernel_lo 2024-03-04T09:25:50.522 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-03-04T09:25:50.522 INFO:teuthology.orchestra.run.smithi146.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-04T09:25:50.522 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 12:ec:0d:29:9e:5d brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-03-04T09:25:50.522 INFO:teuthology.orchestra.run.smithi146.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-03-04T09:25:50.522 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-03-04T09:25:50.522 INFO:teuthology.orchestra.run.smithi146.stdout: inet6 fe80::10ec:dff:fe29:9e5d/64 scope link proto kernel_ll 2024-03-04T09:25:50.522 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-03-04T09:25:50.526 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.3' with 192.168.0.2/16 2024-03-04T09:25:50.526 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:25:50.527 DEBUG:teuthology.orchestra.run.smithi146:> (cd / && exec bash -c ' 2024-03-04T09:25:50.527 DEBUG:teuthology.orchestra.run.smithi146:> set -e 2024-03-04T09:25:50.527 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.3 type veth peer name brx.1 2024-03-04T09:25:50.527 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-03-04T09:25:50.527 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set veth0 up 2024-03-04T09:25:50.527 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set lo up 2024-03-04T09:25:50.527 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip route add default via 192.168.255.254 2024-03-04T09:25:50.527 DEBUG:teuthology.orchestra.run.smithi146:> ') 2024-03-04T09:25:50.797 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:25:50.798 DEBUG:teuthology.orchestra.run.smithi146:> (cd / && exec bash -c ' 2024-03-04T09:25:50.798 DEBUG:teuthology.orchestra.run.smithi146:> set -e 2024-03-04T09:25:50.798 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip link set brx.1 up 2024-03-04T09:25:50.798 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip link set dev brx.1 master ceph-brx 2024-03-04T09:25:50.798 DEBUG:teuthology.orchestra.run.smithi146:> ') 2024-03-04T09:25:50.955 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-04T09:25:50.955 DEBUG:teuthology.orchestra.run.smithi146:> mkdir -p -v /home/ubuntu/cephtest/mnt.3 2024-03-04T09:25:51.000 INFO:teuthology.orchestra.run.smithi146.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.3' 2024-03-04T09:25:51.000 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-04T09:25:51.001 DEBUG:teuthology.orchestra.run.smithi146:> chmod 0000 /home/ubuntu/cephtest/mnt.3 2024-03-04T09:25:51.092 INFO:tasks.cephfs.kernel_mount:mounting using device: 3@.cephfs=/ 2024-03-04T09:25:51.092 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:25:51.092 DEBUG:teuthology.orchestra.run.smithi146:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 3@.cephfs=/ /home/ubuntu/cephtest/mnt.3 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-04T09:25:51.218 INFO:teuthology.orchestra.run.smithi146.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-04T09:25:51.218 INFO:teuthology.orchestra.run.smithi146.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-04T09:25:51.252 INFO:teuthology.orchestra.run.smithi146.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-04T09:25:51.252 INFO:teuthology.orchestra.run.smithi146.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-04T09:25:51.253 INFO:teuthology.orchestra.run.smithi146.stdout:mount.ceph: resolved to: "172.21.15.6:3300,172.21.15.146:3300,172.21.15.146:3301" 2024-03-04T09:25:51.253 INFO:teuthology.orchestra.run.smithi146.stdout:mount.ceph: trying mount with new device syntax: 3@edce0578-5ae4-4402-b45a-552bdef84d7c.cephfs=/ 2024-03-04T09:25:51.253 INFO:teuthology.orchestra.run.smithi146.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,ms_mode=prefer-crc,key=3,mon_addr=172.21.15.6:3300/172.21.15.146:3300/172.21.15.146:3301" will pass to kernel 2024-03-04T09:25:51.254 INFO:teuthology.orchestra.run.smithi146.stdout:mount: /home/ubuntu/cephtest/mnt.3 does not contain SELinux labels. 2024-03-04T09:25:51.254 INFO:teuthology.orchestra.run.smithi146.stdout: You just mounted a file system that supports labels which does not 2024-03-04T09:25:51.254 INFO:teuthology.orchestra.run.smithi146.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-04T09:25:51.255 INFO:teuthology.orchestra.run.smithi146.stdout: applications will generate AVC messages and not be allowed access to 2024-03-04T09:25:51.255 INFO:teuthology.orchestra.run.smithi146.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-04T09:25:51.260 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-04T09:25:51.260 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:25:51.260 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.3 2024-03-04T09:25:51.341 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:25:51.341 DEBUG:teuthology.orchestra.run.smithi146:> (cd /home/ubuntu/cephtest/mnt.3 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-04T09:25:51.389 INFO:teuthology.orchestra.run.smithi146.stdout:client4508 2024-03-04T09:25:51.389 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-04T09:25:51.389 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T09:25:51.487 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:25:51.487 DEBUG:teuthology.orchestra.run.smithi146:> (cd /home/ubuntu/cephtest/mnt.3 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-04T09:25:51.536 INFO:teuthology.orchestra.run.smithi146.stdout:client4508 2024-03-04T09:25:51.536 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:25:51.537 DEBUG:teuthology.orchestra.run.smithi146:> (cd /home/ubuntu/cephtest/mnt.3 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-04T09:25:51.584 INFO:teuthology.orchestra.run.smithi146.stdout:client4508 2024-03-04T09:25:51.584 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:25:51.584 DEBUG:teuthology.orchestra.run.smithi146:> (cd /home/ubuntu/cephtest/mnt.3 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/edce0578-5ae4-4402-b45a-552bdef84d7c.client4508/status') 2024-03-04T09:25:51.709 INFO:teuthology.orchestra.run.smithi146.stdout:instance: client.4508 (3)192.168.0.2:0/1554886236 2024-03-04T09:25:51.710 INFO:teuthology.orchestra.run.smithi146.stdout:blocklisted: false 2024-03-04T09:25:51.710 INFO:teuthology.orchestra.run.smithi146.stderr:0+1 records in 2024-03-04T09:25:51.710 INFO:teuthology.orchestra.run.smithi146.stderr:0+1 records out 2024-03-04T09:25:51.710 INFO:teuthology.orchestra.run.smithi146.stderr:69 bytes copied, 8.2101e-05 s, 840 kB/s 2024-03-04T09:25:51.713 INFO:teuthology.run_tasks:Running task cephfs_test_runner... 2024-03-04T09:25:51.731 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T09:25:51.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:51.966+0000 7fe203e00640 1 Processor -- start 2024-03-04T09:25:51.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:51.966+0000 7fe203e00640 1 -- start start 2024-03-04T09:25:51.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:51.967+0000 7fe203e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fe2040cf080 0x7fe2040d00a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:51.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:51.967+0000 7fe203e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2040b1370 con 0x7fe2040d05e0 2024-03-04T09:25:51.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:51.967+0000 7fe203e00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2040ccf80 con 0x7fe2040b0f90 2024-03-04T09:25:51.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:51.967+0000 7fe203e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2040d0a10 con 0x7fe2040cf080 2024-03-04T09:25:51.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:51.968+0000 7fe202000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fe2040cf080 0x7fe2040d00a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:51.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:51.968+0000 7fe203400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fe2040d05e0 0x7fe20415d750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:58648/0 (socket says 172.21.15.6:58648) 2024-03-04T09:25:51.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:51.968+0000 7fe202000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fe2040cf080 0x7fe2040d00a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51852/0 (socket says 172.21.15.6:51852) 2024-03-04T09:25:51.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:51.968+0000 7fe203400640 1 -- 172.21.15.6:0/367974756 learned_addr learned my addr 172.21.15.6:0/367974756 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:25:51.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:51.968+0000 7fe202000640 1 -- 172.21.15.6:0/367974756 >> v1:172.21.15.146:6789/0 conn(0x7fe2040b0f90 legacy=0x7fe2040cc730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:51.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:51.968+0000 7fe202000640 1 -- 172.21.15.6:0/367974756 >> v1:172.21.15.6:6789/0 conn(0x7fe2040d05e0 legacy=0x7fe20415d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:51.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:51.968+0000 7fe202000640 1 -- 172.21.15.6:0/367974756 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2040cd260 con 0x7fe2040cf080 2024-03-04T09:25:51.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:51.968+0000 7fe201600640 1 -- 172.21.15.6:0/367974756 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 252545596 0 0) 0x7fe2040b1370 con 0x7fe2040d05e0 2024-03-04T09:25:51.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:51.970+0000 7fe202000640 1 --2- 172.21.15.6:0/367974756 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe2040cf080 0x7fe2040d00a0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fe1f0009d90 tx=0x7fe1f002be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=746bd522d53e44c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:51.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:51.970+0000 7fe201600640 1 -- 172.21.15.6:0/367974756 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1f0039070 con 0x7fe2040cf080 2024-03-04T09:25:51.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:51.970+0000 7fe201600640 1 -- 172.21.15.6:0/367974756 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1f002cbd0 con 0x7fe2040cf080 2024-03-04T09:25:51.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:51.970+0000 7fe201600640 1 -- 172.21.15.6:0/367974756 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1f0034440 con 0x7fe2040cf080 2024-03-04T09:25:51.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:51.971+0000 7fe203e00640 1 -- 172.21.15.6:0/367974756 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe2040cf080 msgr2=0x7fe2040d00a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:51.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:51.971+0000 7fe203e00640 1 --2- 172.21.15.6:0/367974756 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe2040cf080 0x7fe2040d00a0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fe1f0009d90 tx=0x7fe1f002be20 comp rx=0 tx=0).stop 2024-03-04T09:25:51.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:51.971+0000 7fe203e00640 1 -- 172.21.15.6:0/367974756 shutdown_connections 2024-03-04T09:25:51.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:51.971+0000 7fe203e00640 1 --2- 172.21.15.6:0/367974756 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe2040cf080 0x7fe2040d00a0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:51.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:51.971+0000 7fe203e00640 1 -- 172.21.15.6:0/367974756 >> 172.21.15.6:0/367974756 conn(0x7fe2040d1ae0 msgr2=0x7fe2040d1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:51.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:51.971+0000 7fe203e00640 1 -- 172.21.15.6:0/367974756 shutdown_connections 2024-03-04T09:25:51.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:51.971+0000 7fe203e00640 1 -- 172.21.15.6:0/367974756 wait complete. 2024-03-04T09:25:51.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:51.973+0000 7fe203e00640 1 Processor -- start 2024-03-04T09:25:51.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:51.973+0000 7fe203e00640 1 -- start start 2024-03-04T09:25:51.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:51.974+0000 7fe203e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe2040b0f90 0x7fe2040ca990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:51.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:51.974+0000 7fe203e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe2040cf080 0x7fe2040caed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:51.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:51.974+0000 7fe203e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe2040d05e0 0x7fe2041f3f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:51.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:51.974+0000 7fe203e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe20406ffe0 con 0x7fe2040cf080 2024-03-04T09:25:51.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:51.974+0000 7fe203e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2040cd820 con 0x7fe2040d05e0 2024-03-04T09:25:51.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:51.974+0000 7fe203e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe204070160 con 0x7fe2040b0f90 2024-03-04T09:25:51.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:51.974+0000 7fe202000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe2040cf080 0x7fe2040caed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:51.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:51.974+0000 7fe202000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe2040cf080 0x7fe2040caed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51868/0 (socket says 172.21.15.6:51868) 2024-03-04T09:25:51.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:51.974+0000 7fe202000640 1 -- 172.21.15.6:0/3196624384 learned_addr learned my addr 172.21.15.6:0/3196624384 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:25:51.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:51.974+0000 7fe202a00640 1 --2- 172.21.15.6:0/3196624384 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe2040b0f90 0x7fe2040ca990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:51.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:51.974+0000 7fe203400640 1 --2- 172.21.15.6:0/3196624384 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe2040d05e0 0x7fe2041f3f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:51.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:51.975+0000 7fe202000640 1 -- 172.21.15.6:0/3196624384 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe2040b0f90 msgr2=0x7fe2040ca990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:51.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:51.975+0000 7fe202000640 1 --2- 172.21.15.6:0/3196624384 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe2040b0f90 0x7fe2040ca990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:51.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:51.975+0000 7fe202000640 1 -- 172.21.15.6:0/3196624384 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe2040d05e0 msgr2=0x7fe2041f3f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:51.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:51.975+0000 7fe202000640 1 --2- 172.21.15.6:0/3196624384 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe2040d05e0 0x7fe2041f3f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:51.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:51.975+0000 7fe202000640 1 -- 172.21.15.6:0/3196624384 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2040c9070 con 0x7fe2040cf080 2024-03-04T09:25:51.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:51.975+0000 7fe202a00640 1 --2- 172.21.15.6:0/3196624384 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe2040b0f90 0x7fe2040ca990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:25:51.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:51.975+0000 7fe202000640 1 --2- 172.21.15.6:0/3196624384 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe2040cf080 0x7fe2040caed0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fe1f0009d90 tx=0x7fe1f0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:51.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:51.975+0000 7fe1faa00640 1 -- 172.21.15.6:0/3196624384 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1f0044020 con 0x7fe2040cf080 2024-03-04T09:25:51.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:51.975+0000 7fe1faa00640 1 -- 172.21.15.6:0/3196624384 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1f003f670 con 0x7fe2040cf080 2024-03-04T09:25:51.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:51.975+0000 7fe1faa00640 1 -- 172.21.15.6:0/3196624384 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1f0039040 con 0x7fe2040cf080 2024-03-04T09:25:51.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:51.975+0000 7fe203e00640 1 -- 172.21.15.6:0/3196624384 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2040c9350 con 0x7fe2040cf080 2024-03-04T09:25:51.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:51.976+0000 7fe203e00640 1 -- 172.21.15.6:0/3196624384 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2040c9910 con 0x7fe2040cf080 2024-03-04T09:25:51.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:51.977+0000 7fe203e00640 1 -- 172.21.15.6:0/3196624384 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe20404f330 con 0x7fe2040cf080 2024-03-04T09:25:51.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:51.978+0000 7fe1faa00640 1 -- 172.21.15.6:0/3196624384 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe1f0047020 con 0x7fe2040cf080 2024-03-04T09:25:51.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:51.979+0000 7fe1faa00640 1 --2- 172.21.15.6:0/3196624384 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fe1e406b4d0 0x7fe1e406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:51.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:51.979+0000 7fe1faa00640 1 -- 172.21.15.6:0/3196624384 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe1f00acf70 con 0x7fe2040cf080 2024-03-04T09:25:51.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:51.979+0000 7fe202a00640 1 --2- 172.21.15.6:0/3196624384 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fe1e406b4d0 0x7fe1e406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:51.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:51.981+0000 7fe202a00640 1 --2- 172.21.15.6:0/3196624384 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fe1e406b4d0 0x7fe1e406d990 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fe204160bf0 tx=0x7fe1f4008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:51.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:51.983+0000 7fe1faa00640 1 -- 172.21.15.6:0/3196624384 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe1f004e030 con 0x7fe2040cf080 2024-03-04T09:25:52.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.186+0000 7fe203e00640 1 -- 172.21.15.6:0/3196624384 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe2040d2240 con 0x7fe2040cf080 2024-03-04T09:25:52.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.188+0000 7fe1faa00640 1 -- 172.21.15.6:0/3196624384 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16557 (secure 0 0 0) 0x7fe1f007bd50 con 0x7fe2040cf080 2024-03-04T09:25:52.185 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:25:52.185 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":20,"fsid":"edce0578-5ae4-4402-b45a-552bdef84d7c","created":"2024-03-04T09:24:54.523780+0000","modified":"2024-03-04T09:25:20.446333+0000","last_up_change":"2024-03-04T09:25:07.089049+0000","last_in_change":"2024-03-04T09:24:57.982077+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T09:25:07.935198+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-04T09:25:17.118573+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-04T09:25:17.930536+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a96574f-81f1-4bc6-8463-757520b84a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6825","nonce":1364449401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6827","nonce":1364449401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6831","nonce":1364449401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6829","nonce":1364449401}]},"public_addr":"172.21.15.6:6825/1364449401","cluster_addr":"172.21.15.6:6827/1364449401","heartbeat_back_addr":"172.21.15.6:6831/1364449401","heartbeat_front_addr":"172.21.15.6:6829/1364449401","state":["exists","up"]},{"osd":1,"uuid":"7d79b9c1-6b87-476a-bc36-7db5ee64cd90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6809","nonce":2494103078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6811","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6813","nonce":2494103078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6821","nonce":2494103078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6817","nonce":2494103078}]},"public_addr":"172.21.15.6:6809/2494103078","cluster_addr":"172.21.15.6:6813/2494103078","heartbeat_back_addr":"172.21.15.6:6821/2494103078","heartbeat_front_addr":"172.21.15.6:6817/2494103078","state":["exists","up"]},{"osd":2,"uuid":"492331b9-4ee9-40b6-bacd-85d44cd6c45a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6812","nonce":1500066681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6816","nonce":1500066681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6824","nonce":1500066681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6820","nonce":1500066681}]},"public_addr":"172.21.15.6:6812/1500066681","cluster_addr":"172.21.15.6:6816/1500066681","heartbeat_back_addr":"172.21.15.6:6824/1500066681","heartbeat_front_addr":"172.21.15.6:6820/1500066681","state":["exists","up"]},{"osd":3,"uuid":"21bb2efe-9ba7-4bdc-bc31-180cd1102e21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6801","nonce":1558806123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6803","nonce":1558806123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6807","nonce":1558806123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6805","nonce":1558806123}]},"public_addr":"172.21.15.6:6801/1558806123","cluster_addr":"172.21.15.6:6803/1558806123","heartbeat_back_addr":"172.21.15.6:6807/1558806123","heartbeat_front_addr":"172.21.15.6:6805/1558806123","state":["exists","up"]},{"osd":4,"uuid":"6e2c4b71-19d9-4737-83d9-558705d8644a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6801","nonce":1526135780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6803","nonce":1526135780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6807","nonce":1526135780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6805","nonce":1526135780}]},"public_addr":"172.21.15.146:6801/1526135780","cluster_addr":"172.21.15.146:6803/1526135780","heartbeat_back_addr":"172.21.15.146:6807/1526135780","heartbeat_front_addr":"172.21.15.146:6805/1526135780","state":["exists","up"]},{"osd":5,"uuid":"fb69c525-f1be-4b7d-ab32-6c29b0d6fddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6809","nonce":460262507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6811","nonce":460262507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6815","nonce":460262507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6813","nonce":460262507}]},"public_addr":"172.21.15.146:6809/460262507","cluster_addr":"172.21.15.146:6811/460262507","heartbeat_back_addr":"172.21.15.146:6815/460262507","heartbeat_front_addr":"172.21.15.146:6813/460262507","state":["exists","up"]},{"osd":6,"uuid":"c32e7fed-370d-4401-8ce0-6a4335b0d168","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6823","nonce":1392897493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6827","nonce":1392897493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6831","nonce":1392897493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6829","nonce":1392897493}]},"public_addr":"172.21.15.146:6823/1392897493","cluster_addr":"172.21.15.146:6827/1392897493","heartbeat_back_addr":"172.21.15.146:6831/1392897493","heartbeat_front_addr":"172.21.15.146:6829/1392897493","state":["exists","up"]},{"osd":7,"uuid":"75a79b44-fa5f-426d-a7d3-71f21f78835c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6817","nonce":3268323156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6819","nonce":3268323156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6825","nonce":3268323156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6822","nonce":3268323156}]},"public_addr":"172.21.15.146:6817/3268323156","cluster_addr":"172.21.15.146:6819/3268323156","heartbeat_back_addr":"172.21.15.146:6825/3268323156","heartbeat_front_addr":"172.21.15.146:6822/3268323156","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.310437+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.295850+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.294674+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.526855+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.300023+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.595160+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.346046+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.558741+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T09:25:52.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.191+0000 7fe203e00640 1 -- 172.21.15.6:0/3196624384 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fe1e406b4d0 msgr2=0x7fe1e406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:52.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.191+0000 7fe203e00640 1 --2- 172.21.15.6:0/3196624384 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fe1e406b4d0 0x7fe1e406d990 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fe204160bf0 tx=0x7fe1f4008040 comp rx=0 tx=0).stop 2024-03-04T09:25:52.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.191+0000 7fe203e00640 1 -- 172.21.15.6:0/3196624384 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe2040cf080 msgr2=0x7fe2040caed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:52.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.191+0000 7fe203e00640 1 --2- 172.21.15.6:0/3196624384 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe2040cf080 0x7fe2040caed0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fe1f0009d90 tx=0x7fe1f0004a40 comp rx=0 tx=0).stop 2024-03-04T09:25:52.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.191+0000 7fe203e00640 1 -- 172.21.15.6:0/3196624384 shutdown_connections 2024-03-04T09:25:52.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.191+0000 7fe203e00640 1 --2- 172.21.15.6:0/3196624384 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe2040d05e0 0x7fe2041f3f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:52.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.191+0000 7fe203e00640 1 --2- 172.21.15.6:0/3196624384 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe2040cf080 0x7fe2040caed0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:52.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.191+0000 7fe203e00640 1 --2- 172.21.15.6:0/3196624384 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fe1e406b4d0 0x7fe1e406d990 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:52.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.191+0000 7fe203e00640 1 --2- 172.21.15.6:0/3196624384 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe2040b0f90 0x7fe2040ca990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:52.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.191+0000 7fe203e00640 1 -- 172.21.15.6:0/3196624384 >> 172.21.15.6:0/3196624384 conn(0x7fe2040d1ae0 msgr2=0x7fe20406be90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:52.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.191+0000 7fe203e00640 1 -- 172.21.15.6:0/3196624384 shutdown_connections 2024-03-04T09:25:52.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.191+0000 7fe203e00640 1 -- 172.21.15.6:0/3196624384 wait complete. 2024-03-04T09:25:52.206 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T09:25:07.935198+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-04T09:25:17.118573+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-04T09:25:17.930536+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-04T09:25:52.207 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-04T09:25:52.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.433+0000 7ffbe7e00640 1 Processor -- start 2024-03-04T09:25:52.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.434+0000 7ffbe7e00640 1 -- start start 2024-03-04T09:25:52.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.435+0000 7ffbe7e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ffbe80d0d40 0x7ffbe80b3510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:52.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.435+0000 7ffbe7e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbe80ceca0 con 0x7ffbe815dee0 2024-03-04T09:25:52.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.435+0000 7ffbe7e00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbe80cef80 con 0x7ffbe80d4860 2024-03-04T09:25:52.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.435+0000 7ffbe7e00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffbe80cf5d0 con 0x7ffbe80d0d40 2024-03-04T09:25:52.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.436+0000 7ffbe6a00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7ffbe815dee0 0x7ffbe80d18a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:58660/0 (socket says 172.21.15.6:58660) 2024-03-04T09:25:52.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.436+0000 7ffbe6a00640 1 -- 172.21.15.6:0/999251345 learned_addr learned my addr 172.21.15.6:0/999251345 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:25:52.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.436+0000 7ffbe7400640 1 --2- 172.21.15.6:0/999251345 >> v2:172.21.15.146:3300/0 conn(0x7ffbe80d0d40 0x7ffbe80b3510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:52.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.436+0000 7ffbe5600640 1 -- 172.21.15.6:0/999251345 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3979282487 0 0) 0x7ffbe80ceca0 con 0x7ffbe815dee0 2024-03-04T09:25:52.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.436+0000 7ffbe5600640 1 -- 172.21.15.6:0/999251345 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffbcc003520 con 0x7ffbe815dee0 2024-03-04T09:25:52.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.436+0000 7ffbe5600640 1 -- 172.21.15.6:0/999251345 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1470622326 0 0) 0x7ffbcc003520 con 0x7ffbe815dee0 2024-03-04T09:25:52.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.437+0000 7ffbe5600640 1 -- 172.21.15.6:0/999251345 >> v2:172.21.15.146:3300/0 conn(0x7ffbe80d0d40 msgr2=0x7ffbe80b3510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:52.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.437+0000 7ffbe5600640 1 --2- 172.21.15.6:0/999251345 >> v2:172.21.15.146:3300/0 conn(0x7ffbe80d0d40 0x7ffbe80b3510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:52.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.437+0000 7ffbe5600640 1 -- 172.21.15.6:0/999251345 >> v1:172.21.15.146:6789/0 conn(0x7ffbe80d4860 legacy=0x7ffbe80d0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:52.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.437+0000 7ffbe5600640 1 -- 172.21.15.6:0/999251345 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbe80cf260 con 0x7ffbe815dee0 2024-03-04T09:25:52.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.437+0000 7ffbe5600640 1 -- 172.21.15.6:0/999251345 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7ffbd80019f0 con 0x7ffbe815dee0 2024-03-04T09:25:52.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.437+0000 7ffbe5600640 1 -- 172.21.15.6:0/999251345 >> v1:172.21.15.6:6789/0 conn(0x7ffbe815dee0 legacy=0x7ffbe80d18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:52.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.437+0000 7ffbe5600640 1 --2- 172.21.15.6:0/999251345 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffbcc0039b0 0x7ffbcc003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:52.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.437+0000 7ffbe5600640 1 --2- 172.21.15.6:0/999251345 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffbcc0043b0 0x7ffbcc004880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:52.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.437+0000 7ffbe5600640 1 --2- 172.21.15.6:0/999251345 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ffbcc005080 0x7ffbcc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:52.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.437+0000 7ffbe5600640 1 -- 172.21.15.6:0/999251345 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbe80cf5d0 con 0x7ffbcc0039b0 2024-03-04T09:25:52.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.437+0000 7ffbe5600640 1 -- 172.21.15.6:0/999251345 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbcc00bfe0 con 0x7ffbcc0043b0 2024-03-04T09:25:52.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.437+0000 7ffbe5600640 1 -- 172.21.15.6:0/999251345 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbcc00c1e0 con 0x7ffbcc005080 2024-03-04T09:25:52.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.437+0000 7ffbe6a00640 1 --2- 172.21.15.6:0/999251345 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffbcc0039b0 0x7ffbcc003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:52.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.437+0000 7ffbe6000640 1 --2- 172.21.15.6:0/999251345 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffbcc0043b0 0x7ffbcc004880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:52.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.438+0000 7ffbe6a00640 1 -- 172.21.15.6:0/999251345 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ffbcc005080 msgr2=0x7ffbcc00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:52.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.438+0000 7ffbe6a00640 1 --2- 172.21.15.6:0/999251345 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ffbcc005080 0x7ffbcc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:52.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.438+0000 7ffbe6a00640 1 -- 172.21.15.6:0/999251345 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffbcc0043b0 msgr2=0x7ffbcc004880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:52.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.438+0000 7ffbe6a00640 1 --2- 172.21.15.6:0/999251345 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffbcc0043b0 0x7ffbcc004880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:52.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.438+0000 7ffbe6a00640 1 -- 172.21.15.6:0/999251345 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbcc003520 con 0x7ffbcc0039b0 2024-03-04T09:25:52.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.438+0000 7ffbe6000640 1 --2- 172.21.15.6:0/999251345 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffbcc0043b0 0x7ffbcc004880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:25:52.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.438+0000 7ffbe7400640 1 -- 172.21.15.6:0/999251345 reap_dead start 2024-03-04T09:25:52.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.439+0000 7ffbe6a00640 1 --2- 172.21.15.6:0/999251345 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffbcc0039b0 0x7ffbcc003d90 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7ffbd800a880 tx=0x7ffbcc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d82c239e92ffd58b server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:52.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.439+0000 7ffbe5600640 1 -- 172.21.15.6:0/999251345 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbd800a4a0 con 0x7ffbcc0039b0 2024-03-04T09:25:52.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.439+0000 7ffbe5600640 1 -- 172.21.15.6:0/999251345 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbd8034650 con 0x7ffbcc0039b0 2024-03-04T09:25:52.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.439+0000 7ffbe5600640 1 -- 172.21.15.6:0/999251345 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbd8034a80 con 0x7ffbcc0039b0 2024-03-04T09:25:52.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.439+0000 7ffbe7e00640 1 -- 172.21.15.6:0/999251345 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffbcc0039b0 msgr2=0x7ffbcc003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:52.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.439+0000 7ffbe7e00640 1 --2- 172.21.15.6:0/999251345 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffbcc0039b0 0x7ffbcc003d90 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7ffbd800a880 tx=0x7ffbcc005830 comp rx=0 tx=0).stop 2024-03-04T09:25:52.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.440+0000 7ffbe7e00640 1 -- 172.21.15.6:0/999251345 shutdown_connections 2024-03-04T09:25:52.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.440+0000 7ffbe7e00640 1 --2- 172.21.15.6:0/999251345 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffbcc0039b0 0x7ffbcc003d90 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:52.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.440+0000 7ffbe7e00640 1 -- 172.21.15.6:0/999251345 >> 172.21.15.6:0/999251345 conn(0x7ffbe80d3cd0 msgr2=0x7ffbe80d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:52.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.440+0000 7ffbe7e00640 1 -- 172.21.15.6:0/999251345 shutdown_connections 2024-03-04T09:25:52.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.440+0000 7ffbe7e00640 1 -- 172.21.15.6:0/999251345 wait complete. 2024-03-04T09:25:52.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.442+0000 7ffbe7e00640 1 Processor -- start 2024-03-04T09:25:52.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.442+0000 7ffbe7e00640 1 -- start start 2024-03-04T09:25:52.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.443+0000 7ffbe7e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ffbcc0039b0 0x7ffbe80b30b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:52.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.443+0000 7ffbe7e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffbe81546c0 0x7ffbe80b7ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:52.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.443+0000 7ffbe7e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffbe80b35f0 0x7ffbe80b8530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:52.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.443+0000 7ffbe7e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbe80cfa60 con 0x7ffbe81546c0 2024-03-04T09:25:52.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.443+0000 7ffbe7e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbe815e2c0 con 0x7ffbe80b35f0 2024-03-04T09:25:52.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.443+0000 7ffbe7e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbe80cfbe0 con 0x7ffbcc0039b0 2024-03-04T09:25:52.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.443+0000 7ffbe6000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffbe81546c0 0x7ffbe80b7ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:52.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.443+0000 7ffbe6000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffbe81546c0 0x7ffbe80b7ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51894/0 (socket says 172.21.15.6:51894) 2024-03-04T09:25:52.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.443+0000 7ffbe6000640 1 -- 172.21.15.6:0/2011101440 learned_addr learned my addr 172.21.15.6:0/2011101440 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:25:52.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.443+0000 7ffbe6a00640 1 --2- 172.21.15.6:0/2011101440 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ffbcc0039b0 0x7ffbe80b30b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:52.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.443+0000 7ffbe7400640 1 --2- 172.21.15.6:0/2011101440 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffbe80b35f0 0x7ffbe80b8530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:52.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.444+0000 7ffbe6000640 1 -- 172.21.15.6:0/2011101440 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ffbcc0039b0 msgr2=0x7ffbe80b30b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:52.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.444+0000 7ffbe6000640 1 --2- 172.21.15.6:0/2011101440 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ffbcc0039b0 0x7ffbe80b30b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:52.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.444+0000 7ffbe6000640 1 -- 172.21.15.6:0/2011101440 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffbe80b35f0 msgr2=0x7ffbe80b8530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:52.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.444+0000 7ffbe6000640 1 --2- 172.21.15.6:0/2011101440 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffbe80b35f0 0x7ffbe80b8530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:52.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.444+0000 7ffbe6000640 1 -- 172.21.15.6:0/2011101440 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbe80b3b90 con 0x7ffbe81546c0 2024-03-04T09:25:52.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.444+0000 7ffbe7400640 1 --2- 172.21.15.6:0/2011101440 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffbe80b35f0 0x7ffbe80b8530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:25:52.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.444+0000 7ffbe6000640 1 --2- 172.21.15.6:0/2011101440 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffbe81546c0 0x7ffbe80b7ff0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7ffbd000d590 tx=0x7ffbd000da50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:52.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.444+0000 7ffbdea00640 1 -- 172.21.15.6:0/2011101440 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbd0005280 con 0x7ffbe81546c0 2024-03-04T09:25:52.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.444+0000 7ffbdea00640 1 -- 172.21.15.6:0/2011101440 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbd0005420 con 0x7ffbe81546c0 2024-03-04T09:25:52.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.444+0000 7ffbdea00640 1 -- 172.21.15.6:0/2011101440 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbd0007270 con 0x7ffbe81546c0 2024-03-04T09:25:52.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.444+0000 7ffbe7e00640 1 -- 172.21.15.6:0/2011101440 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbe80b3e70 con 0x7ffbe81546c0 2024-03-04T09:25:52.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.444+0000 7ffbe7e00640 1 -- 172.21.15.6:0/2011101440 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbe80b4430 con 0x7ffbe81546c0 2024-03-04T09:25:52.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.445+0000 7ffbe7e00640 1 -- 172.21.15.6:0/2011101440 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbe804f330 con 0x7ffbe81546c0 2024-03-04T09:25:52.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.446+0000 7ffbdea00640 1 -- 172.21.15.6:0/2011101440 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ffbd001a020 con 0x7ffbe81546c0 2024-03-04T09:25:52.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.447+0000 7ffbdea00640 1 --2- 172.21.15.6:0/2011101440 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ffbc406b500 0x7ffbc406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:52.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.447+0000 7ffbdea00640 1 -- 172.21.15.6:0/2011101440 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ffbd0089a10 con 0x7ffbe81546c0 2024-03-04T09:25:52.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.448+0000 7ffbe6a00640 1 --2- 172.21.15.6:0/2011101440 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ffbc406b500 0x7ffbc406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:52.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.449+0000 7ffbe6a00640 1 --2- 172.21.15.6:0/2011101440 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ffbc406b500 0x7ffbc406d9c0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7ffbd8034000 tx=0x7ffbd802f000 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:52.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.452+0000 7ffbdea00640 1 -- 172.21.15.6:0/2011101440 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ffbd0058820 con 0x7ffbe81546c0 2024-03-04T09:25:52.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.649+0000 7ffbe7e00640 1 -- 172.21.15.6:0/2011101440 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ffbe80ce4b0 con 0x7ffbe81546c0 2024-03-04T09:25:52.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.650+0000 7ffbdea00640 1 -- 172.21.15.6:0/2011101440 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7ffbd0058120 con 0x7ffbe81546c0 2024-03-04T09:25:52.647 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-04T09:25:52.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.652+0000 7ffbe7e00640 1 -- 172.21.15.6:0/2011101440 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ffbc406b500 msgr2=0x7ffbc406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:52.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.652+0000 7ffbe7e00640 1 --2- 172.21.15.6:0/2011101440 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ffbc406b500 0x7ffbc406d9c0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7ffbd8034000 tx=0x7ffbd802f000 comp rx=0 tx=0).stop 2024-03-04T09:25:52.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.652+0000 7ffbe7e00640 1 -- 172.21.15.6:0/2011101440 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffbe81546c0 msgr2=0x7ffbe80b7ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:52.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.652+0000 7ffbe7e00640 1 --2- 172.21.15.6:0/2011101440 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffbe81546c0 0x7ffbe80b7ff0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7ffbd000d590 tx=0x7ffbd000da50 comp rx=0 tx=0).stop 2024-03-04T09:25:52.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.652+0000 7ffbe7e00640 1 -- 172.21.15.6:0/2011101440 shutdown_connections 2024-03-04T09:25:52.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.652+0000 7ffbe7e00640 1 --2- 172.21.15.6:0/2011101440 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffbe80b35f0 0x7ffbe80b8530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:52.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.652+0000 7ffbe7e00640 1 --2- 172.21.15.6:0/2011101440 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffbe81546c0 0x7ffbe80b7ff0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:52.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.652+0000 7ffbe7e00640 1 --2- 172.21.15.6:0/2011101440 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ffbc406b500 0x7ffbc406d9c0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:52.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.653+0000 7ffbe7e00640 1 --2- 172.21.15.6:0/2011101440 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ffbcc0039b0 0x7ffbe80b30b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:52.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.653+0000 7ffbe7e00640 1 -- 172.21.15.6:0/2011101440 >> 172.21.15.6:0/2011101440 conn(0x7ffbe80d3cd0 msgr2=0x7ffbe806bf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:52.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.653+0000 7ffbe7e00640 1 -- 172.21.15.6:0/2011101440 shutdown_connections 2024-03-04T09:25:52.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.653+0000 7ffbe7e00640 1 -- 172.21.15.6:0/2011101440 wait complete. 2024-03-04T09:25:52.670 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-04T09:25:52.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.890+0000 7fbfd9a00640 1 Processor -- start 2024-03-04T09:25:52.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.891+0000 7fbfd9a00640 1 -- start start 2024-03-04T09:25:52.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.892+0000 7fbfd9a00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fbfd40b3c20 0x7fbfd40729d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:52.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.892+0000 7fbfd9a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfd4071930 con 0x7fbfd40b4000 2024-03-04T09:25:52.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.892+0000 7fbfd9a00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfd4071c10 con 0x7fbfd40b1b40 2024-03-04T09:25:52.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.892+0000 7fbfd9a00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfd40b4490 con 0x7fbfd40b3c20 2024-03-04T09:25:52.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.893+0000 7fbfcbe00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fbfd40b4000 0x7fbfd415b8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:58674/0 (socket says 172.21.15.6:58674) 2024-03-04T09:25:52.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.893+0000 7fbfcbe00640 1 -- 172.21.15.6:0/3204564478 learned_addr learned my addr 172.21.15.6:0/3204564478 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:25:52.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.893+0000 7fbfd3e00640 1 --2- 172.21.15.6:0/3204564478 >> v2:172.21.15.146:3300/0 conn(0x7fbfd40b3c20 0x7fbfd40729d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:52.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.893+0000 7fbfd3400640 1 -- 172.21.15.6:0/3204564478 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 846328097 0 0) 0x7fbfd4071930 con 0x7fbfd40b4000 2024-03-04T09:25:52.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.893+0000 7fbfd3400640 1 -- 172.21.15.6:0/3204564478 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbfb8003520 con 0x7fbfd40b4000 2024-03-04T09:25:52.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.894+0000 7fbfd3400640 1 -- 172.21.15.6:0/3204564478 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3016610935 0 0) 0x7fbfb8003520 con 0x7fbfd40b4000 2024-03-04T09:25:52.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.894+0000 7fbfd3400640 1 -- 172.21.15.6:0/3204564478 >> v2:172.21.15.146:3300/0 conn(0x7fbfd40b3c20 msgr2=0x7fbfd40729d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:52.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.894+0000 7fbfd3400640 1 --2- 172.21.15.6:0/3204564478 >> v2:172.21.15.146:3300/0 conn(0x7fbfd40b3c20 0x7fbfd40729d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:52.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.894+0000 7fbfd3400640 1 -- 172.21.15.6:0/3204564478 >> v1:172.21.15.146:6789/0 conn(0x7fbfd40b1b40 legacy=0x7fbfd415c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:52.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.894+0000 7fbfd3400640 1 -- 172.21.15.6:0/3204564478 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfd4071ef0 con 0x7fbfd40b4000 2024-03-04T09:25:52.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.894+0000 7fbfd3400640 1 -- 172.21.15.6:0/3204564478 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7fbfbc001670 con 0x7fbfd40b4000 2024-03-04T09:25:52.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.894+0000 7fbfd3400640 1 -- 172.21.15.6:0/3204564478 >> v1:172.21.15.6:6789/0 conn(0x7fbfd40b4000 legacy=0x7fbfd415b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:52.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.894+0000 7fbfd3400640 1 --2- 172.21.15.6:0/3204564478 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbfb80039b0 0x7fbfb8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:52.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.894+0000 7fbfd3400640 1 --2- 172.21.15.6:0/3204564478 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbfb80043b0 0x7fbfb8004880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:52.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.894+0000 7fbfd3400640 1 --2- 172.21.15.6:0/3204564478 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbfb8005080 0x7fbfb800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:52.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.894+0000 7fbfd3400640 1 -- 172.21.15.6:0/3204564478 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfb800bfb0 con 0x7fbfb8005080 2024-03-04T09:25:52.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.894+0000 7fbfd3400640 1 -- 172.21.15.6:0/3204564478 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfb800c160 con 0x7fbfb80039b0 2024-03-04T09:25:52.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.894+0000 7fbfd3400640 1 -- 172.21.15.6:0/3204564478 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfb800c360 con 0x7fbfb80043b0 2024-03-04T09:25:52.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.894+0000 7fbfd9000640 1 --2- 172.21.15.6:0/3204564478 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbfb8005080 0x7fbfb800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:52.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.895+0000 7fbfcbe00640 1 --2- 172.21.15.6:0/3204564478 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbfb80043b0 0x7fbfb8004880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:52.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.895+0000 7fbfd3e00640 1 --2- 172.21.15.6:0/3204564478 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbfb80039b0 0x7fbfb8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:52.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.895+0000 7fbfd9000640 1 -- 172.21.15.6:0/3204564478 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbfb80043b0 msgr2=0x7fbfb8004880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:52.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.895+0000 7fbfd9000640 1 --2- 172.21.15.6:0/3204564478 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbfb80043b0 0x7fbfb8004880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:52.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.895+0000 7fbfd9000640 1 -- 172.21.15.6:0/3204564478 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbfb80039b0 msgr2=0x7fbfb8003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:52.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.895+0000 7fbfd9000640 1 --2- 172.21.15.6:0/3204564478 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbfb80039b0 0x7fbfb8003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:52.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.895+0000 7fbfd9000640 1 -- 172.21.15.6:0/3204564478 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfb8003520 con 0x7fbfb8005080 2024-03-04T09:25:52.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.896+0000 7fbfd9000640 1 -- 172.21.15.6:0/3204564478 reap_dead start 2024-03-04T09:25:52.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.896+0000 7fbfd9000640 1 --2- 172.21.15.6:0/3204564478 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbfb8005080 0x7fbfb800b9c0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fbfcc001af0 tx=0x7fbfb8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fe314ef4b5cfade server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:52.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.896+0000 7fbfd3400640 1 -- 172.21.15.6:0/3204564478 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfcc001b30 con 0x7fbfb8005080 2024-03-04T09:25:52.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.896+0000 7fbfd3400640 1 -- 172.21.15.6:0/3204564478 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfcc003b90 con 0x7fbfb8005080 2024-03-04T09:25:52.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.897+0000 7fbfd3400640 1 -- 172.21.15.6:0/3204564478 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfcc005110 con 0x7fbfb8005080 2024-03-04T09:25:52.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.897+0000 7fbfd9a00640 1 -- 172.21.15.6:0/3204564478 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbfb8005080 msgr2=0x7fbfb800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:52.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.897+0000 7fbfd9a00640 1 --2- 172.21.15.6:0/3204564478 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbfb8005080 0x7fbfb800b9c0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fbfcc001af0 tx=0x7fbfb8005830 comp rx=0 tx=0).stop 2024-03-04T09:25:52.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.897+0000 7fbfd9a00640 1 -- 172.21.15.6:0/3204564478 shutdown_connections 2024-03-04T09:25:52.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.897+0000 7fbfd9a00640 1 --2- 172.21.15.6:0/3204564478 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbfb8005080 0x7fbfb800b9c0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:52.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.897+0000 7fbfd9a00640 1 -- 172.21.15.6:0/3204564478 >> 172.21.15.6:0/3204564478 conn(0x7fbfd406b3a0 msgr2=0x7fbfd406b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:52.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.897+0000 7fbfd9a00640 1 -- 172.21.15.6:0/3204564478 shutdown_connections 2024-03-04T09:25:52.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.897+0000 7fbfd9a00640 1 -- 172.21.15.6:0/3204564478 wait complete. 2024-03-04T09:25:52.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.899+0000 7fbfd9a00640 1 Processor -- start 2024-03-04T09:25:52.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.899+0000 7fbfd9a00640 1 -- start start 2024-03-04T09:25:52.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.900+0000 7fbfd9a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbfb8005080 0x7fbfd41582d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:52.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.900+0000 7fbfd9a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbfd4155720 0x7fbfd414be90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:52.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.900+0000 7fbfd9a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbfd4156fa0 0x7fbfd4157420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:52.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.900+0000 7fbfd9a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfd40729d0 con 0x7fbfd4156fa0 2024-03-04T09:25:52.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.900+0000 7fbfd9a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfd40724b0 con 0x7fbfd4155720 2024-03-04T09:25:52.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.900+0000 7fbfd9a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfd4072b50 con 0x7fbfb8005080 2024-03-04T09:25:52.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.900+0000 7fbfd9000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbfd4156fa0 0x7fbfd4157420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:52.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.900+0000 7fbfd9000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbfd4156fa0 0x7fbfd4157420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51912/0 (socket says 172.21.15.6:51912) 2024-03-04T09:25:52.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.900+0000 7fbfd9000640 1 -- 172.21.15.6:0/677475177 learned_addr learned my addr 172.21.15.6:0/677475177 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:25:52.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.901+0000 7fbfd3e00640 1 --2- 172.21.15.6:0/677475177 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbfb8005080 0x7fbfd41582d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:52.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.901+0000 7fbfcbe00640 1 --2- 172.21.15.6:0/677475177 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbfd4155720 0x7fbfd414be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:52.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.901+0000 7fbfd9000640 1 -- 172.21.15.6:0/677475177 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbfb8005080 msgr2=0x7fbfd41582d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:52.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.901+0000 7fbfd9000640 1 --2- 172.21.15.6:0/677475177 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbfb8005080 0x7fbfd41582d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:52.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.901+0000 7fbfd9000640 1 -- 172.21.15.6:0/677475177 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbfd4155720 msgr2=0x7fbfd414be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:52.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.901+0000 7fbfd9000640 1 --2- 172.21.15.6:0/677475177 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbfd4155720 0x7fbfd414be90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:52.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.901+0000 7fbfd9000640 1 -- 172.21.15.6:0/677475177 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfd4155f00 con 0x7fbfd4156fa0 2024-03-04T09:25:52.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.901+0000 7fbfd9000640 1 --2- 172.21.15.6:0/677475177 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbfd4156fa0 0x7fbfd4157420 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fbfcc002050 tx=0x7fbfcc0015b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:52.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.901+0000 7fbfd0c00640 1 -- 172.21.15.6:0/677475177 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfcc001b30 con 0x7fbfd4156fa0 2024-03-04T09:25:52.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.901+0000 7fbfd0c00640 1 -- 172.21.15.6:0/677475177 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfcc005030 con 0x7fbfd4156fa0 2024-03-04T09:25:52.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.901+0000 7fbfd0c00640 1 -- 172.21.15.6:0/677475177 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfcc005420 con 0x7fbfd4156fa0 2024-03-04T09:25:52.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.901+0000 7fbfd9a00640 1 -- 172.21.15.6:0/677475177 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfd41561e0 con 0x7fbfd4156fa0 2024-03-04T09:25:52.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.902+0000 7fbfd9a00640 1 -- 172.21.15.6:0/677475177 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfd41567a0 con 0x7fbfd4156fa0 2024-03-04T09:25:52.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.903+0000 7fbfd9a00640 1 -- 172.21.15.6:0/677475177 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfd404f330 con 0x7fbfd4156fa0 2024-03-04T09:25:52.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.904+0000 7fbfd0c00640 1 -- 172.21.15.6:0/677475177 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbfcc037450 con 0x7fbfd4156fa0 2024-03-04T09:25:52.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.904+0000 7fbfd0c00640 1 --2- 172.21.15.6:0/677475177 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fbfb006b500 0x7fbfb006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:52.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.905+0000 7fbfd0c00640 1 -- 172.21.15.6:0/677475177 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbfcc0ab810 con 0x7fbfd4156fa0 2024-03-04T09:25:52.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.905+0000 7fbfd3e00640 1 --2- 172.21.15.6:0/677475177 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fbfb006b500 0x7fbfb006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:52.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.906+0000 7fbfd3e00640 1 --2- 172.21.15.6:0/677475177 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fbfb006b500 0x7fbfb006d9c0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fbfc40098d0 tx=0x7fbfc4008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:52.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:52.909+0000 7fbfd0c00640 1 -- 172.21.15.6:0/677475177 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbfcc048030 con 0x7fbfd4156fa0 2024-03-04T09:25:53.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.108+0000 7fbfd9a00640 1 -- 172.21.15.6:0/677475177 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fbfd40addb0 con 0x7fbfd4156fa0 2024-03-04T09:25:53.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.109+0000 7fbfd0c00640 1 -- 172.21.15.6:0/677475177 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7fbfcc07a5f0 con 0x7fbfd4156fa0 2024-03-04T09:25:53.106 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-03-04T09:25:53.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.112+0000 7fbfd9a00640 1 -- 172.21.15.6:0/677475177 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fbfb006b500 msgr2=0x7fbfb006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:53.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.112+0000 7fbfd9a00640 1 --2- 172.21.15.6:0/677475177 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fbfb006b500 0x7fbfb006d9c0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fbfc40098d0 tx=0x7fbfc4008040 comp rx=0 tx=0).stop 2024-03-04T09:25:53.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.112+0000 7fbfd9a00640 1 -- 172.21.15.6:0/677475177 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbfd4156fa0 msgr2=0x7fbfd4157420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:53.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.112+0000 7fbfd9a00640 1 --2- 172.21.15.6:0/677475177 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbfd4156fa0 0x7fbfd4157420 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fbfcc002050 tx=0x7fbfcc0015b0 comp rx=0 tx=0).stop 2024-03-04T09:25:53.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.112+0000 7fbfd9a00640 1 -- 172.21.15.6:0/677475177 shutdown_connections 2024-03-04T09:25:53.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.112+0000 7fbfd9a00640 1 --2- 172.21.15.6:0/677475177 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbfd4156fa0 0x7fbfd4157420 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:53.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.112+0000 7fbfd9a00640 1 --2- 172.21.15.6:0/677475177 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbfd4155720 0x7fbfd414be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:53.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.112+0000 7fbfd9a00640 1 --2- 172.21.15.6:0/677475177 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fbfb006b500 0x7fbfb006d9c0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:53.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.112+0000 7fbfd9a00640 1 --2- 172.21.15.6:0/677475177 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbfb8005080 0x7fbfd41582d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:53.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.112+0000 7fbfd9a00640 1 -- 172.21.15.6:0/677475177 >> 172.21.15.6:0/677475177 conn(0x7fbfd406b3a0 msgr2=0x7fbfd415ce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:53.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.112+0000 7fbfd9a00640 1 -- 172.21.15.6:0/677475177 shutdown_connections 2024-03-04T09:25:53.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.112+0000 7fbfd9a00640 1 -- 172.21.15.6:0/677475177 wait complete. 2024-03-04T09:25:53.129 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-04T09:25:53.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.352+0000 7f78f8c00640 1 Processor -- start 2024-03-04T09:25:53.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.352+0000 7f78f8c00640 1 -- start start 2024-03-04T09:25:53.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.354+0000 7f78f8c00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f78f40b3c20 0x7f78f40729d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:53.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.354+0000 7f78f8c00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78f4071930 con 0x7f78f40b4000 2024-03-04T09:25:53.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.354+0000 7f78f8c00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78f4071c10 con 0x7f78f40b1b40 2024-03-04T09:25:53.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.354+0000 7f78f8c00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78f40b4490 con 0x7f78f40b3c20 2024-03-04T09:25:53.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.354+0000 7f78ebe00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f78f40b4000 0x7f78f415b8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:58688/0 (socket says 172.21.15.6:58688) 2024-03-04T09:25:53.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.354+0000 7f78ebe00640 1 -- 172.21.15.6:0/1835448250 learned_addr learned my addr 172.21.15.6:0/1835448250 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:25:53.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.354+0000 7f78f3400640 1 --2- 172.21.15.6:0/1835448250 >> v2:172.21.15.146:3300/0 conn(0x7f78f40b3c20 0x7f78f40729d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:53.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.355+0000 7f78f2a00640 1 -- 172.21.15.6:0/1835448250 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2592764186 0 0) 0x7f78f4071930 con 0x7f78f40b4000 2024-03-04T09:25:53.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.355+0000 7f78f2a00640 1 -- 172.21.15.6:0/1835448250 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78d8003520 con 0x7f78f40b4000 2024-03-04T09:25:53.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.355+0000 7f78f2a00640 1 -- 172.21.15.6:0/1835448250 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3349798039 0 0) 0x7f78d8003520 con 0x7f78f40b4000 2024-03-04T09:25:53.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.355+0000 7f78f2a00640 1 -- 172.21.15.6:0/1835448250 >> v2:172.21.15.146:3300/0 conn(0x7f78f40b3c20 msgr2=0x7f78f40729d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:53.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.355+0000 7f78f2a00640 1 --2- 172.21.15.6:0/1835448250 >> v2:172.21.15.146:3300/0 conn(0x7f78f40b3c20 0x7f78f40729d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:53.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.355+0000 7f78f2a00640 1 -- 172.21.15.6:0/1835448250 >> v1:172.21.15.146:6789/0 conn(0x7f78f40b1b40 legacy=0x7f78f415c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:53.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.355+0000 7f78f2a00640 1 -- 172.21.15.6:0/1835448250 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78f4071ef0 con 0x7f78f40b4000 2024-03-04T09:25:53.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.355+0000 7f78f2a00640 1 -- 172.21.15.6:0/1835448250 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7f78dc001670 con 0x7f78f40b4000 2024-03-04T09:25:53.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.355+0000 7f78f2a00640 1 -- 172.21.15.6:0/1835448250 >> v1:172.21.15.6:6789/0 conn(0x7f78f40b4000 legacy=0x7f78f415b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:53.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.355+0000 7f78f2a00640 1 --2- 172.21.15.6:0/1835448250 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f78d80039b0 0x7f78d8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:53.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.355+0000 7f78f2a00640 1 --2- 172.21.15.6:0/1835448250 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f78d8004800 0x7f78d8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:53.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.355+0000 7f78f2a00640 1 --2- 172.21.15.6:0/1835448250 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f78d80051c0 0x7f78d800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:53.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.355+0000 7f78f2a00640 1 -- 172.21.15.6:0/1835448250 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78d800c030 con 0x7f78d80039b0 2024-03-04T09:25:53.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.355+0000 7f78f2a00640 1 -- 172.21.15.6:0/1835448250 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78d800c1e0 con 0x7f78d8004800 2024-03-04T09:25:53.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.355+0000 7f78f2a00640 1 -- 172.21.15.6:0/1835448250 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78d800c3e0 con 0x7f78d80051c0 2024-03-04T09:25:53.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.356+0000 7f78f3400640 1 --2- 172.21.15.6:0/1835448250 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f78d80039b0 0x7f78d8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:53.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.356+0000 7f78ebe00640 1 --2- 172.21.15.6:0/1835448250 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f78d8004800 0x7f78d8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:53.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.356+0000 7f78f3e00640 1 --2- 172.21.15.6:0/1835448250 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f78d80051c0 0x7f78d800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:53.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.356+0000 7f78f3400640 1 -- 172.21.15.6:0/1835448250 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f78d80051c0 msgr2=0x7f78d800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:53.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.356+0000 7f78f3400640 1 --2- 172.21.15.6:0/1835448250 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f78d80051c0 0x7f78d800b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:53.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.356+0000 7f78f3400640 1 -- 172.21.15.6:0/1835448250 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f78d8004800 msgr2=0x7f78d8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:53.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.356+0000 7f78f3400640 1 --2- 172.21.15.6:0/1835448250 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f78d8004800 0x7f78d8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:53.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.356+0000 7f78f3400640 1 -- 172.21.15.6:0/1835448250 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78d8003520 con 0x7f78d80039b0 2024-03-04T09:25:53.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.356+0000 7f78f3e00640 1 -- 172.21.15.6:0/1835448250 reap_dead start 2024-03-04T09:25:53.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.356+0000 7f78ebe00640 1 --2- 172.21.15.6:0/1835448250 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f78d8004800 0x7f78d8004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:25:53.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.357+0000 7f78f3400640 1 --2- 172.21.15.6:0/1835448250 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f78d80039b0 0x7f78d8003dd0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f78e4009650 tx=0x7f78d8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d25a741d17ca2e server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:53.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.357+0000 7f78f2a00640 1 -- 172.21.15.6:0/1835448250 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78e403b070 con 0x7f78d80039b0 2024-03-04T09:25:53.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.357+0000 7f78f2a00640 1 -- 172.21.15.6:0/1835448250 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78e402e8c0 con 0x7f78d80039b0 2024-03-04T09:25:53.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.357+0000 7f78f2a00640 1 -- 172.21.15.6:0/1835448250 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78e4009690 con 0x7f78d80039b0 2024-03-04T09:25:53.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.358+0000 7f78f8c00640 1 -- 172.21.15.6:0/1835448250 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f78d80039b0 msgr2=0x7f78d8003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:53.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.358+0000 7f78f8c00640 1 --2- 172.21.15.6:0/1835448250 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f78d80039b0 0x7f78d8003dd0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f78e4009650 tx=0x7f78d8005830 comp rx=0 tx=0).stop 2024-03-04T09:25:53.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.358+0000 7f78f8c00640 1 -- 172.21.15.6:0/1835448250 shutdown_connections 2024-03-04T09:25:53.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.358+0000 7f78f8c00640 1 --2- 172.21.15.6:0/1835448250 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f78d80039b0 0x7f78d8003dd0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:53.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.358+0000 7f78f8c00640 1 -- 172.21.15.6:0/1835448250 >> 172.21.15.6:0/1835448250 conn(0x7f78f406b3a0 msgr2=0x7f78f406b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:53.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.358+0000 7f78f8c00640 1 -- 172.21.15.6:0/1835448250 shutdown_connections 2024-03-04T09:25:53.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.358+0000 7f78f8c00640 1 -- 172.21.15.6:0/1835448250 wait complete. 2024-03-04T09:25:53.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.360+0000 7f78f8c00640 1 Processor -- start 2024-03-04T09:25:53.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.360+0000 7f78f8c00640 1 -- start start 2024-03-04T09:25:53.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.361+0000 7f78f8c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f78d80039b0 0x7f78f41f37b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:53.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.361+0000 7f78f8c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f78f4156980 0x7f78f41ef5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:53.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.361+0000 7f78f8c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f78f4144710 0x7f78f4155720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:53.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.361+0000 7f78f8c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78f40729d0 con 0x7f78d80039b0 2024-03-04T09:25:53.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.361+0000 7f78f8c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78f40724b0 con 0x7f78f4156980 2024-03-04T09:25:53.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.361+0000 7f78f8c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78f4072b50 con 0x7f78f4144710 2024-03-04T09:25:53.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.362+0000 7f78f3400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f78d80039b0 0x7f78f41f37b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:53.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.362+0000 7f78f3400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f78d80039b0 0x7f78f41f37b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51932/0 (socket says 172.21.15.6:51932) 2024-03-04T09:25:53.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.362+0000 7f78f3400640 1 -- 172.21.15.6:0/792200504 learned_addr learned my addr 172.21.15.6:0/792200504 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:25:53.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.362+0000 7f78ebe00640 1 --2- 172.21.15.6:0/792200504 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f78f4156980 0x7f78f41ef5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:53.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.362+0000 7f78f3e00640 1 --2- 172.21.15.6:0/792200504 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f78f4144710 0x7f78f4155720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:53.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.362+0000 7f78f3400640 1 -- 172.21.15.6:0/792200504 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f78f4144710 msgr2=0x7f78f4155720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:53.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.362+0000 7f78f3400640 1 --2- 172.21.15.6:0/792200504 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f78f4144710 0x7f78f4155720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:53.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.362+0000 7f78f3400640 1 -- 172.21.15.6:0/792200504 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f78f4156980 msgr2=0x7f78f41ef5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:53.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.362+0000 7f78f3400640 1 --2- 172.21.15.6:0/792200504 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f78f4156980 0x7f78f41ef5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:53.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.362+0000 7f78f3400640 1 -- 172.21.15.6:0/792200504 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78f41efea0 con 0x7f78d80039b0 2024-03-04T09:25:53.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.362+0000 7f78f3e00640 1 --2- 172.21.15.6:0/792200504 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f78f4144710 0x7f78f4155720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:25:53.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.363+0000 7f78f3400640 1 --2- 172.21.15.6:0/792200504 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f78d80039b0 0x7f78f41f37b0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f78e4039040 tx=0x7f78e4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:53.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.363+0000 7f78eb400640 1 -- 172.21.15.6:0/792200504 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78e403b050 con 0x7f78d80039b0 2024-03-04T09:25:53.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.363+0000 7f78eb400640 1 -- 172.21.15.6:0/792200504 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78e4047070 con 0x7f78d80039b0 2024-03-04T09:25:53.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.363+0000 7f78eb400640 1 -- 172.21.15.6:0/792200504 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78e4009690 con 0x7f78d80039b0 2024-03-04T09:25:53.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.363+0000 7f78f8c00640 1 -- 172.21.15.6:0/792200504 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78f41f0180 con 0x7f78d80039b0 2024-03-04T09:25:53.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.363+0000 7f78f8c00640 1 -- 172.21.15.6:0/792200504 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78f4156de0 con 0x7f78d80039b0 2024-03-04T09:25:53.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.364+0000 7f78f8c00640 1 -- 172.21.15.6:0/792200504 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78f404f330 con 0x7f78d80039b0 2024-03-04T09:25:53.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.365+0000 7f78eb400640 1 -- 172.21.15.6:0/792200504 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f78e4003a10 con 0x7f78d80039b0 2024-03-04T09:25:53.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.366+0000 7f78eb400640 1 --2- 172.21.15.6:0/792200504 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f78d006b500 0x7f78d006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:53.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.366+0000 7f78eb400640 1 -- 172.21.15.6:0/792200504 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f78e40ada20 con 0x7f78d80039b0 2024-03-04T09:25:53.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.367+0000 7f78ebe00640 1 --2- 172.21.15.6:0/792200504 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f78d006b500 0x7f78d006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:53.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.368+0000 7f78ebe00640 1 --2- 172.21.15.6:0/792200504 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f78d006b500 0x7f78d006d9c0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f78f406e8f0 tx=0x7f78dc009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:53.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.371+0000 7f78eb400640 1 -- 172.21.15.6:0/792200504 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f78e407c780 con 0x7f78d80039b0 2024-03-04T09:25:53.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.573+0000 7f78f8c00640 1 -- 172.21.15.6:0/792200504 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f78f40cbef0 con 0x7f78d80039b0 2024-03-04T09:25:53.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.574+0000 7f78eb400640 1 -- 172.21.15.6:0/792200504 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f78e407c080 con 0x7f78d80039b0 2024-03-04T09:25:53.571 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 64 2024-03-04T09:25:53.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.576+0000 7f78f8c00640 1 -- 172.21.15.6:0/792200504 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f78d006b500 msgr2=0x7f78d006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:53.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.576+0000 7f78f8c00640 1 --2- 172.21.15.6:0/792200504 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f78d006b500 0x7f78d006d9c0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f78f406e8f0 tx=0x7f78dc009040 comp rx=0 tx=0).stop 2024-03-04T09:25:53.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.576+0000 7f78f8c00640 1 -- 172.21.15.6:0/792200504 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f78d80039b0 msgr2=0x7f78f41f37b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:53.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.576+0000 7f78f8c00640 1 --2- 172.21.15.6:0/792200504 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f78d80039b0 0x7f78f41f37b0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f78e4039040 tx=0x7f78e4004730 comp rx=0 tx=0).stop 2024-03-04T09:25:53.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.576+0000 7f78f8c00640 1 -- 172.21.15.6:0/792200504 shutdown_connections 2024-03-04T09:25:53.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.576+0000 7f78f8c00640 1 --2- 172.21.15.6:0/792200504 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f78d006b500 0x7f78d006d9c0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:53.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.576+0000 7f78f8c00640 1 --2- 172.21.15.6:0/792200504 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f78f4144710 0x7f78f4155720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:53.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.576+0000 7f78f8c00640 1 --2- 172.21.15.6:0/792200504 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f78f4156980 0x7f78f41ef5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:53.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.576+0000 7f78f8c00640 1 --2- 172.21.15.6:0/792200504 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f78d80039b0 0x7f78f41f37b0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:53.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.576+0000 7f78f8c00640 1 -- 172.21.15.6:0/792200504 >> 172.21.15.6:0/792200504 conn(0x7f78f406b3a0 msgr2=0x7f78f406b780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:53.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.577+0000 7f78f8c00640 1 -- 172.21.15.6:0/792200504 shutdown_connections 2024-03-04T09:25:53.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.577+0000 7f78f8c00640 1 -- 172.21.15.6:0/792200504 wait complete. 2024-03-04T09:25:53.593 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T09:25:53.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.817+0000 7f56eac00640 1 Processor -- start 2024-03-04T09:25:53.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.817+0000 7f56eac00640 1 -- start start 2024-03-04T09:25:53.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.819+0000 7f56eac00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f56e40af580 0x7f56e415c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:53.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.819+0000 7f56eac00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f56e40cdc20 0x7f56e40b2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:53.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.819+0000 7f56eac00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56e415cc40 con 0x7f56e40af1a0 2024-03-04T09:25:53.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.819+0000 7f56eac00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56e415cf20 con 0x7f56e40cdc20 2024-03-04T09:25:53.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.819+0000 7f56eac00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56e415d200 con 0x7f56e40af580 2024-03-04T09:25:53.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.819+0000 7f56e8c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f56e40cdc20 0x7f56e40b2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:53.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.819+0000 7f56e8c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f56e40cdc20 0x7f56e40b2dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51940/0 (socket says 172.21.15.6:51940) 2024-03-04T09:25:53.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.819+0000 7f56e8c00640 1 -- 172.21.15.6:0/2056361264 learned_addr learned my addr 172.21.15.6:0/2056361264 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:25:53.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.820+0000 7f56e3400640 1 --2- 172.21.15.6:0/2056361264 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f56e40af580 0x7f56e415c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:53.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.820+0000 7f56e8c00640 1 -- 172.21.15.6:0/2056361264 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f56e40af580 msgr2=0x7f56e415c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:53.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.820+0000 7f56e8c00640 1 --2- 172.21.15.6:0/2056361264 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f56e40af580 0x7f56e415c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:53.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.820+0000 7f56e8c00640 1 -- 172.21.15.6:0/2056361264 >> v1:172.21.15.6:6789/0 conn(0x7f56e40af1a0 legacy=0x7f56e40b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:53.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.820+0000 7f56e8c00640 1 -- 172.21.15.6:0/2056361264 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56e415d4e0 con 0x7f56e40cdc20 2024-03-04T09:25:53.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.821+0000 7f56e8c00640 1 --2- 172.21.15.6:0/2056361264 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f56e40cdc20 0x7f56e40b2dd0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f56dc009e50 tx=0x7f56dc02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6aa56e817b61f8d server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:53.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.821+0000 7f56e2a00640 1 -- 172.21.15.6:0/2056361264 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56dc02ca80 con 0x7f56e40cdc20 2024-03-04T09:25:53.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.821+0000 7f56e2a00640 1 -- 172.21.15.6:0/2056361264 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56dc02cc20 con 0x7f56e40cdc20 2024-03-04T09:25:53.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.821+0000 7f56e2a00640 1 -- 172.21.15.6:0/2056361264 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56dc0343f0 con 0x7f56e40cdc20 2024-03-04T09:25:53.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.822+0000 7f56eac00640 1 -- 172.21.15.6:0/2056361264 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f56e40cdc20 msgr2=0x7f56e40b2dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:53.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.822+0000 7f56eac00640 1 --2- 172.21.15.6:0/2056361264 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f56e40cdc20 0x7f56e40b2dd0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f56dc009e50 tx=0x7f56dc02bee0 comp rx=0 tx=0).stop 2024-03-04T09:25:53.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.822+0000 7f56eac00640 1 -- 172.21.15.6:0/2056361264 shutdown_connections 2024-03-04T09:25:53.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.822+0000 7f56eac00640 1 --2- 172.21.15.6:0/2056361264 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f56e40cdc20 0x7f56e40b2dd0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:53.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.822+0000 7f56eac00640 1 --2- 172.21.15.6:0/2056361264 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f56e40af580 0x7f56e415c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:53.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.822+0000 7f56eac00640 1 -- 172.21.15.6:0/2056361264 >> 172.21.15.6:0/2056361264 conn(0x7f56e406afb0 msgr2=0x7f56e406b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:53.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.822+0000 7f56eac00640 1 -- 172.21.15.6:0/2056361264 shutdown_connections 2024-03-04T09:25:53.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.822+0000 7f56eac00640 1 -- 172.21.15.6:0/2056361264 wait complete. 2024-03-04T09:25:53.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.824+0000 7f56eac00640 1 Processor -- start 2024-03-04T09:25:53.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.824+0000 7f56eac00640 1 -- start start 2024-03-04T09:25:53.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.825+0000 7f56eac00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f56e40af1a0 0x7f56e415d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:53.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.825+0000 7f56eac00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f56e40af580 0x7f56e41491f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:53.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.825+0000 7f56eac00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f56e40cdc20 0x7f56e4149730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:53.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.825+0000 7f56eac00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56e40b36a0 con 0x7f56e40af1a0 2024-03-04T09:25:53.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.825+0000 7f56eac00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56e40b3520 con 0x7f56e40cdc20 2024-03-04T09:25:53.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.825+0000 7f56eac00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56e40b3820 con 0x7f56e40af580 2024-03-04T09:25:53.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.825+0000 7f56e3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f56e40af1a0 0x7f56e415d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:53.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.825+0000 7f56e3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f56e40af1a0 0x7f56e415d970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51956/0 (socket says 172.21.15.6:51956) 2024-03-04T09:25:53.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.825+0000 7f56e3e00640 1 -- 172.21.15.6:0/2458586453 learned_addr learned my addr 172.21.15.6:0/2458586453 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:25:53.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.825+0000 7f56e3400640 1 --2- 172.21.15.6:0/2458586453 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f56e40af580 0x7f56e41491f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:53.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.825+0000 7f56e3e00640 1 -- 172.21.15.6:0/2458586453 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f56e40af580 msgr2=0x7f56e41491f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:53.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.825+0000 7f56e3e00640 1 --2- 172.21.15.6:0/2458586453 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f56e40af580 0x7f56e41491f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:53.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.825+0000 7f56e3e00640 1 -- 172.21.15.6:0/2458586453 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f56e40cdc20 msgr2=0x7f56e4149730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:53.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.825+0000 7f56e3e00640 1 --2- 172.21.15.6:0/2458586453 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f56e40cdc20 0x7f56e4149730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:53.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.825+0000 7f56e3e00640 1 -- 172.21.15.6:0/2458586453 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56e406cbb0 con 0x7f56e40af1a0 2024-03-04T09:25:53.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.826+0000 7f56e3e00640 1 --2- 172.21.15.6:0/2458586453 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f56e40af1a0 0x7f56e415d970 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f56d400c400 tx=0x7f56d400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:53.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.826+0000 7f56dbe00640 1 -- 172.21.15.6:0/2458586453 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56d40052d0 con 0x7f56e40af1a0 2024-03-04T09:25:53.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.826+0000 7f56dbe00640 1 -- 172.21.15.6:0/2458586453 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56d4005490 con 0x7f56e40af1a0 2024-03-04T09:25:53.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.826+0000 7f56dbe00640 1 -- 172.21.15.6:0/2458586453 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56d4007260 con 0x7f56e40af1a0 2024-03-04T09:25:53.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.826+0000 7f56eac00640 1 -- 172.21.15.6:0/2458586453 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56e406ce20 con 0x7f56e40af1a0 2024-03-04T09:25:53.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.826+0000 7f56eac00640 1 -- 172.21.15.6:0/2458586453 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56e406d3e0 con 0x7f56e40af1a0 2024-03-04T09:25:53.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.827+0000 7f56eac00640 1 -- 172.21.15.6:0/2458586453 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56e4069510 con 0x7f56e40af1a0 2024-03-04T09:25:53.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.828+0000 7f56dbe00640 1 -- 172.21.15.6:0/2458586453 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f56d4007400 con 0x7f56e40af1a0 2024-03-04T09:25:53.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.829+0000 7f56dbe00640 1 --2- 172.21.15.6:0/2458586453 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f56c006b520 0x7f56c006d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:53.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.829+0000 7f56dbe00640 1 -- 172.21.15.6:0/2458586453 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f56d40896a0 con 0x7f56e40af1a0 2024-03-04T09:25:53.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.830+0000 7f56e3400640 1 --2- 172.21.15.6:0/2458586453 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f56c006b520 0x7f56c006d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:53.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.831+0000 7f56e3400640 1 --2- 172.21.15.6:0/2458586453 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f56c006b520 0x7f56c006d9e0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f56e415bec0 tx=0x7f56cc008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:53.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:53.834+0000 7f56dbe00640 1 -- 172.21.15.6:0/2458586453 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f56d4058600 con 0x7f56e40af1a0 2024-03-04T09:25:54.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.039+0000 7f56eac00640 1 -- 172.21.15.6:0/2458586453 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f56e415bef0 con 0x7f56e40af1a0 2024-03-04T09:25:54.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.042+0000 7f56dbe00640 1 -- 172.21.15.6:0/2458586453 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16557 (secure 0 0 0) 0x7f56d4057f00 con 0x7f56e40af1a0 2024-03-04T09:25:54.039 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:25:54.039 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":20,"fsid":"edce0578-5ae4-4402-b45a-552bdef84d7c","created":"2024-03-04T09:24:54.523780+0000","modified":"2024-03-04T09:25:20.446333+0000","last_up_change":"2024-03-04T09:25:07.089049+0000","last_in_change":"2024-03-04T09:24:57.982077+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T09:25:07.935198+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-04T09:25:17.118573+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-04T09:25:17.930536+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a96574f-81f1-4bc6-8463-757520b84a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6825","nonce":1364449401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6827","nonce":1364449401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6831","nonce":1364449401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6829","nonce":1364449401}]},"public_addr":"172.21.15.6:6825/1364449401","cluster_addr":"172.21.15.6:6827/1364449401","heartbeat_back_addr":"172.21.15.6:6831/1364449401","heartbeat_front_addr":"172.21.15.6:6829/1364449401","state":["exists","up"]},{"osd":1,"uuid":"7d79b9c1-6b87-476a-bc36-7db5ee64cd90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6809","nonce":2494103078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6811","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6813","nonce":2494103078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6821","nonce":2494103078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6817","nonce":2494103078}]},"public_addr":"172.21.15.6:6809/2494103078","cluster_addr":"172.21.15.6:6813/2494103078","heartbeat_back_addr":"172.21.15.6:6821/2494103078","heartbeat_front_addr":"172.21.15.6:6817/2494103078","state":["exists","up"]},{"osd":2,"uuid":"492331b9-4ee9-40b6-bacd-85d44cd6c45a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6812","nonce":1500066681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6816","nonce":1500066681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6824","nonce":1500066681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6820","nonce":1500066681}]},"public_addr":"172.21.15.6:6812/1500066681","cluster_addr":"172.21.15.6:6816/1500066681","heartbeat_back_addr":"172.21.15.6:6824/1500066681","heartbeat_front_addr":"172.21.15.6:6820/1500066681","state":["exists","up"]},{"osd":3,"uuid":"21bb2efe-9ba7-4bdc-bc31-180cd1102e21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6801","nonce":1558806123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6803","nonce":1558806123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6807","nonce":1558806123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6805","nonce":1558806123}]},"public_addr":"172.21.15.6:6801/1558806123","cluster_addr":"172.21.15.6:6803/1558806123","heartbeat_back_addr":"172.21.15.6:6807/1558806123","heartbeat_front_addr":"172.21.15.6:6805/1558806123","state":["exists","up"]},{"osd":4,"uuid":"6e2c4b71-19d9-4737-83d9-558705d8644a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6801","nonce":1526135780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6803","nonce":1526135780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6807","nonce":1526135780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6805","nonce":1526135780}]},"public_addr":"172.21.15.146:6801/1526135780","cluster_addr":"172.21.15.146:6803/1526135780","heartbeat_back_addr":"172.21.15.146:6807/1526135780","heartbeat_front_addr":"172.21.15.146:6805/1526135780","state":["exists","up"]},{"osd":5,"uuid":"fb69c525-f1be-4b7d-ab32-6c29b0d6fddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6809","nonce":460262507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6811","nonce":460262507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6815","nonce":460262507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6813","nonce":460262507}]},"public_addr":"172.21.15.146:6809/460262507","cluster_addr":"172.21.15.146:6811/460262507","heartbeat_back_addr":"172.21.15.146:6815/460262507","heartbeat_front_addr":"172.21.15.146:6813/460262507","state":["exists","up"]},{"osd":6,"uuid":"c32e7fed-370d-4401-8ce0-6a4335b0d168","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6823","nonce":1392897493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6827","nonce":1392897493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6831","nonce":1392897493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6829","nonce":1392897493}]},"public_addr":"172.21.15.146:6823/1392897493","cluster_addr":"172.21.15.146:6827/1392897493","heartbeat_back_addr":"172.21.15.146:6831/1392897493","heartbeat_front_addr":"172.21.15.146:6829/1392897493","state":["exists","up"]},{"osd":7,"uuid":"75a79b44-fa5f-426d-a7d3-71f21f78835c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6817","nonce":3268323156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6819","nonce":3268323156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6825","nonce":3268323156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6822","nonce":3268323156}]},"public_addr":"172.21.15.146:6817/3268323156","cluster_addr":"172.21.15.146:6819/3268323156","heartbeat_back_addr":"172.21.15.146:6825/3268323156","heartbeat_front_addr":"172.21.15.146:6822/3268323156","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.310437+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.295850+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.294674+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.526855+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.300023+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.595160+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.346046+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.558741+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T09:25:54.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.044+0000 7f56eac00640 1 -- 172.21.15.6:0/2458586453 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f56c006b520 msgr2=0x7f56c006d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:54.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.044+0000 7f56eac00640 1 --2- 172.21.15.6:0/2458586453 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f56c006b520 0x7f56c006d9e0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f56e415bec0 tx=0x7f56cc008040 comp rx=0 tx=0).stop 2024-03-04T09:25:54.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.044+0000 7f56eac00640 1 -- 172.21.15.6:0/2458586453 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f56e40af1a0 msgr2=0x7f56e415d970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:54.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.044+0000 7f56eac00640 1 --2- 172.21.15.6:0/2458586453 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f56e40af1a0 0x7f56e415d970 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f56d400c400 tx=0x7f56d400c8c0 comp rx=0 tx=0).stop 2024-03-04T09:25:54.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.044+0000 7f56eac00640 1 -- 172.21.15.6:0/2458586453 shutdown_connections 2024-03-04T09:25:54.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.044+0000 7f56eac00640 1 --2- 172.21.15.6:0/2458586453 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f56e40cdc20 0x7f56e4149730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:54.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.044+0000 7f56eac00640 1 --2- 172.21.15.6:0/2458586453 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f56c006b520 0x7f56c006d9e0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:54.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.044+0000 7f56eac00640 1 --2- 172.21.15.6:0/2458586453 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f56e40af580 0x7f56e41491f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:54.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.045+0000 7f56eac00640 1 --2- 172.21.15.6:0/2458586453 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f56e40af1a0 0x7f56e415d970 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:54.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.045+0000 7f56eac00640 1 -- 172.21.15.6:0/2458586453 >> 172.21.15.6:0/2458586453 conn(0x7f56e406afb0 msgr2=0x7f56e406b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:54.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.045+0000 7f56eac00640 1 -- 172.21.15.6:0/2458586453 shutdown_connections 2024-03-04T09:25:54.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.045+0000 7f56eac00640 1 -- 172.21.15.6:0/2458586453 wait complete. 2024-03-04T09:25:54.060 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T09:25:07.935198+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-04T09:25:17.118573+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-04T09:25:17.930536+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-04T09:25:54.060 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-04T09:25:54.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.283+0000 7f5a96e00640 1 Processor -- start 2024-03-04T09:25:54.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.283+0000 7f5a96e00640 1 -- start start 2024-03-04T09:25:54.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.285+0000 7f5a96e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f5a9006da40 0x7f5a9006e340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:54.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.285+0000 7f5a96e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a9006d090 con 0x7f5a9015e680 2024-03-04T09:25:54.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.285+0000 7f5a96e00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a9015eb80 con 0x7f5a900ccad0 2024-03-04T09:25:54.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.285+0000 7f5a96e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a90161c70 con 0x7f5a9006da40 2024-03-04T09:25:54.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.285+0000 7f5a8fe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f5a9006da40 0x7f5a9006e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:54.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.285+0000 7f5a8fe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f5a9006da40 0x7f5a9006e340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51966/0 (socket says 172.21.15.6:51966) 2024-03-04T09:25:54.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.286+0000 7f5a8fe00640 1 -- 172.21.15.6:0/3533660131 learned_addr learned my addr 172.21.15.6:0/3533660131 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:25:54.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.286+0000 7f5a8fe00640 1 -- 172.21.15.6:0/3533660131 >> v1:172.21.15.146:6789/0 conn(0x7f5a900ccad0 legacy=0x7f5a90162100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:54.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.286+0000 7f5a8fe00640 1 -- 172.21.15.6:0/3533660131 >> v1:172.21.15.6:6789/0 conn(0x7f5a9015e680 legacy=0x7f5a9006c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:54.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.286+0000 7f5a8fe00640 1 -- 172.21.15.6:0/3533660131 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a9015ee60 con 0x7f5a9006da40 2024-03-04T09:25:54.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.287+0000 7f5a8fe00640 1 --2- 172.21.15.6:0/3533660131 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5a9006da40 0x7f5a9006e340 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f5a80009e50 tx=0x7f5a8002bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a84b5ba7b5b1eebc server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:54.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.287+0000 7f5a8ea00640 1 -- 172.21.15.6:0/3533660131 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a8002cb10 con 0x7f5a9006da40 2024-03-04T09:25:54.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.287+0000 7f5a8ea00640 1 -- 172.21.15.6:0/3533660131 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a8002ccb0 con 0x7f5a9006da40 2024-03-04T09:25:54.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.287+0000 7f5a8ea00640 1 -- 172.21.15.6:0/3533660131 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a8002ce60 con 0x7f5a9006da40 2024-03-04T09:25:54.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.288+0000 7f5a96e00640 1 -- 172.21.15.6:0/3533660131 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5a9006da40 msgr2=0x7f5a9006e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:54.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.288+0000 7f5a96e00640 1 --2- 172.21.15.6:0/3533660131 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5a9006da40 0x7f5a9006e340 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f5a80009e50 tx=0x7f5a8002bf70 comp rx=0 tx=0).stop 2024-03-04T09:25:54.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.288+0000 7f5a96e00640 1 -- 172.21.15.6:0/3533660131 shutdown_connections 2024-03-04T09:25:54.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.288+0000 7f5a96e00640 1 --2- 172.21.15.6:0/3533660131 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5a9006da40 0x7f5a9006e340 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:54.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.288+0000 7f5a96e00640 1 -- 172.21.15.6:0/3533660131 >> 172.21.15.6:0/3533660131 conn(0x7f5a9006b3d0 msgr2=0x7f5a9006b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:54.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.288+0000 7f5a96e00640 1 -- 172.21.15.6:0/3533660131 shutdown_connections 2024-03-04T09:25:54.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.288+0000 7f5a96e00640 1 -- 172.21.15.6:0/3533660131 wait complete. 2024-03-04T09:25:54.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.290+0000 7f5a96e00640 1 Processor -- start 2024-03-04T09:25:54.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.290+0000 7f5a96e00640 1 -- start start 2024-03-04T09:25:54.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.291+0000 7f5a96e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5a9006da40 0x7f5a9014d060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:54.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.291+0000 7f5a96e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5a900ccad0 0x7f5a9014a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:54.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.291+0000 7f5a96e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5a9015e680 0x7f5a9009aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:54.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.291+0000 7f5a96e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a9006e880 con 0x7f5a900ccad0 2024-03-04T09:25:54.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.291+0000 7f5a96e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a9015f420 con 0x7f5a9006da40 2024-03-04T09:25:54.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.291+0000 7f5a96e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a9006ea00 con 0x7f5a9015e680 2024-03-04T09:25:54.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.291+0000 7f5a8f400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5a900ccad0 0x7f5a9014a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:54.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.291+0000 7f5a8f400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5a900ccad0 0x7f5a9014a8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51980/0 (socket says 172.21.15.6:51980) 2024-03-04T09:25:54.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.291+0000 7f5a8f400640 1 -- 172.21.15.6:0/1074471498 learned_addr learned my addr 172.21.15.6:0/1074471498 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:25:54.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.292+0000 7f5a8fe00640 1 --2- 172.21.15.6:0/1074471498 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5a9006da40 0x7f5a9014d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:54.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.292+0000 7f5a8f400640 1 -- 172.21.15.6:0/1074471498 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5a9015e680 msgr2=0x7f5a9009aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:54.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.292+0000 7f5a94e00640 1 --2- 172.21.15.6:0/1074471498 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5a9015e680 0x7f5a9009aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:54.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.292+0000 7f5a8f400640 1 --2- 172.21.15.6:0/1074471498 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5a9015e680 0x7f5a9009aeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:54.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.292+0000 7f5a8f400640 1 -- 172.21.15.6:0/1074471498 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5a9006da40 msgr2=0x7f5a9014d060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:54.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.292+0000 7f5a8f400640 1 --2- 172.21.15.6:0/1074471498 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5a9006da40 0x7f5a9014d060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:54.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.292+0000 7f5a8f400640 1 -- 172.21.15.6:0/1074471498 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a9014d660 con 0x7f5a900ccad0 2024-03-04T09:25:54.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.292+0000 7f5a8fe00640 1 --2- 172.21.15.6:0/1074471498 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5a9006da40 0x7f5a9014d060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:25:54.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.292+0000 7f5a94e00640 1 --2- 172.21.15.6:0/1074471498 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5a9015e680 0x7f5a9009aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:25:54.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.292+0000 7f5a8f400640 1 --2- 172.21.15.6:0/1074471498 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5a900ccad0 0x7f5a9014a8e0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f5a7800c3c0 tx=0x7f5a7800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:54.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.292+0000 7f5a87e00640 1 -- 172.21.15.6:0/1074471498 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a780052d0 con 0x7f5a900ccad0 2024-03-04T09:25:54.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.292+0000 7f5a87e00640 1 -- 172.21.15.6:0/1074471498 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a78005490 con 0x7f5a900ccad0 2024-03-04T09:25:54.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.292+0000 7f5a87e00640 1 -- 172.21.15.6:0/1074471498 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a78007260 con 0x7f5a900ccad0 2024-03-04T09:25:54.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.292+0000 7f5a96e00640 1 -- 172.21.15.6:0/1074471498 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a9014d940 con 0x7f5a900ccad0 2024-03-04T09:25:54.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.293+0000 7f5a96e00640 1 -- 172.21.15.6:0/1074471498 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a9014df00 con 0x7f5a900ccad0 2024-03-04T09:25:54.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.294+0000 7f5a96e00640 1 -- 172.21.15.6:0/1074471498 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a9004f330 con 0x7f5a900ccad0 2024-03-04T09:25:54.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.295+0000 7f5a87e00640 1 -- 172.21.15.6:0/1074471498 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5a78007400 con 0x7f5a900ccad0 2024-03-04T09:25:54.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.296+0000 7f5a87e00640 1 --2- 172.21.15.6:0/1074471498 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f5a6c06b4d0 0x7f5a6c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:54.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.296+0000 7f5a87e00640 1 -- 172.21.15.6:0/1074471498 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5a780896f0 con 0x7f5a900ccad0 2024-03-04T09:25:54.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.297+0000 7f5a8fe00640 1 --2- 172.21.15.6:0/1074471498 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f5a6c06b4d0 0x7f5a6c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:54.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.298+0000 7f5a8fe00640 1 --2- 172.21.15.6:0/1074471498 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f5a6c06b4d0 0x7f5a6c06d990 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f5a80035000 tx=0x7f5a80030000 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:54.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.300+0000 7f5a87e00640 1 -- 172.21.15.6:0/1074471498 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5a78058560 con 0x7f5a900ccad0 2024-03-04T09:25:54.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.497+0000 7f5a96e00640 1 -- 172.21.15.6:0/1074471498 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f5a900cd2c0 con 0x7f5a900ccad0 2024-03-04T09:25:54.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.499+0000 7f5a87e00640 1 -- 172.21.15.6:0/1074471498 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5a78057e60 con 0x7f5a900ccad0 2024-03-04T09:25:54.495 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-04T09:25:54.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.501+0000 7f5a96e00640 1 -- 172.21.15.6:0/1074471498 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f5a6c06b4d0 msgr2=0x7f5a6c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:54.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.501+0000 7f5a96e00640 1 --2- 172.21.15.6:0/1074471498 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f5a6c06b4d0 0x7f5a6c06d990 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f5a80035000 tx=0x7f5a80030000 comp rx=0 tx=0).stop 2024-03-04T09:25:54.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.501+0000 7f5a96e00640 1 -- 172.21.15.6:0/1074471498 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5a900ccad0 msgr2=0x7f5a9014a8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:54.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.501+0000 7f5a96e00640 1 --2- 172.21.15.6:0/1074471498 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5a900ccad0 0x7f5a9014a8e0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f5a7800c3c0 tx=0x7f5a7800c880 comp rx=0 tx=0).stop 2024-03-04T09:25:54.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.501+0000 7f5a96e00640 1 -- 172.21.15.6:0/1074471498 shutdown_connections 2024-03-04T09:25:54.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.501+0000 7f5a96e00640 1 --2- 172.21.15.6:0/1074471498 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f5a6c06b4d0 0x7f5a6c06d990 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:54.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.501+0000 7f5a96e00640 1 --2- 172.21.15.6:0/1074471498 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5a9015e680 0x7f5a9009aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:54.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.501+0000 7f5a96e00640 1 --2- 172.21.15.6:0/1074471498 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5a900ccad0 0x7f5a9014a8e0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:54.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.501+0000 7f5a96e00640 1 --2- 172.21.15.6:0/1074471498 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5a9006da40 0x7f5a9014d060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:54.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.501+0000 7f5a96e00640 1 -- 172.21.15.6:0/1074471498 >> 172.21.15.6:0/1074471498 conn(0x7f5a9006b3d0 msgr2=0x7f5a9006b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:54.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.501+0000 7f5a96e00640 1 -- 172.21.15.6:0/1074471498 shutdown_connections 2024-03-04T09:25:54.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.501+0000 7f5a96e00640 1 -- 172.21.15.6:0/1074471498 wait complete. 2024-03-04T09:25:54.516 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-04T09:25:54.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.746+0000 7f4c9c600640 1 Processor -- start 2024-03-04T09:25:54.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.746+0000 7f4c9c600640 1 -- start start 2024-03-04T09:25:54.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.748+0000 7f4c9c600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f4c940ccad0 0x7f4c940cceb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:54.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.748+0000 7f4c9c600640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f4c94162bb0 0x7f4c94163040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:54.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.748+0000 7f4c9c600640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c940aeb10 con 0x7f4c940cdab0 2024-03-04T09:25:54.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.748+0000 7f4c9c600640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c940cd3f0 con 0x7f4c940ccad0 2024-03-04T09:25:54.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.748+0000 7f4c9c600640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c94163950 con 0x7f4c94162bb0 2024-03-04T09:25:54.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.748+0000 7f4c99c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f4c940ccad0 0x7f4c940cceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:54.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.748+0000 7f4c99c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f4c940ccad0 0x7f4c940cceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51990/0 (socket says 172.21.15.6:51990) 2024-03-04T09:25:54.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.748+0000 7f4c99c00640 1 -- 172.21.15.6:0/1134633031 learned_addr learned my addr 172.21.15.6:0/1134633031 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:25:54.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.749+0000 7f4c99200640 1 --2- 172.21.15.6:0/1134633031 >> v2:172.21.15.146:3300/0 conn(0x7f4c94162bb0 0x7f4c94163040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:54.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.749+0000 7f4c99c00640 1 -- 172.21.15.6:0/1134633031 >> v2:172.21.15.146:3300/0 conn(0x7f4c94162bb0 msgr2=0x7f4c94163040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:54.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.749+0000 7f4c99c00640 1 --2- 172.21.15.6:0/1134633031 >> v2:172.21.15.146:3300/0 conn(0x7f4c94162bb0 0x7f4c94163040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:54.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.749+0000 7f4c99c00640 1 -- 172.21.15.6:0/1134633031 >> v1:172.21.15.146:6789/0 conn(0x7f4c940cdab0 legacy=0x7f4c940cdec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:54.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.749+0000 7f4c99c00640 1 -- 172.21.15.6:0/1134633031 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c940acd40 con 0x7f4c940ccad0 2024-03-04T09:25:54.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.749+0000 7f4c99200640 1 --2- 172.21.15.6:0/1134633031 >> v2:172.21.15.146:3300/0 conn(0x7f4c94162bb0 0x7f4c94163040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:25:54.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.750+0000 7f4c99c00640 1 --2- 172.21.15.6:0/1134633031 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4c940ccad0 0x7f4c940cceb0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f4c94070b90 tx=0x7f4c8802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f43a3dc666763b88 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:54.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.751+0000 7f4c93e00640 1 -- 172.21.15.6:0/1134633031 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c88035020 con 0x7f4c940ccad0 2024-03-04T09:25:54.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.751+0000 7f4c93e00640 1 -- 172.21.15.6:0/1134633031 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c88031870 con 0x7f4c940ccad0 2024-03-04T09:25:54.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.751+0000 7f4c93e00640 1 -- 172.21.15.6:0/1134633031 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c88031c40 con 0x7f4c940ccad0 2024-03-04T09:25:54.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.751+0000 7f4c9c600640 1 -- 172.21.15.6:0/1134633031 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4c940ccad0 msgr2=0x7f4c940cceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:54.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.751+0000 7f4c9c600640 1 --2- 172.21.15.6:0/1134633031 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4c940ccad0 0x7f4c940cceb0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f4c94070b90 tx=0x7f4c8802b910 comp rx=0 tx=0).stop 2024-03-04T09:25:54.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.752+0000 7f4c9c600640 1 -- 172.21.15.6:0/1134633031 shutdown_connections 2024-03-04T09:25:54.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.752+0000 7f4c9c600640 1 --2- 172.21.15.6:0/1134633031 >> v2:172.21.15.146:3300/0 conn(0x7f4c94162bb0 0x7f4c94163040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:54.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.752+0000 7f4c9c600640 1 --2- 172.21.15.6:0/1134633031 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4c940ccad0 0x7f4c940cceb0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:54.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.752+0000 7f4c9c600640 1 -- 172.21.15.6:0/1134633031 >> 172.21.15.6:0/1134633031 conn(0x7f4c9406b3d0 msgr2=0x7f4c9406b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:54.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.752+0000 7f4c9c600640 1 -- 172.21.15.6:0/1134633031 shutdown_connections 2024-03-04T09:25:54.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.752+0000 7f4c9c600640 1 -- 172.21.15.6:0/1134633031 wait complete. 2024-03-04T09:25:54.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.753+0000 7f4c9c600640 1 Processor -- start 2024-03-04T09:25:54.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.754+0000 7f4c9c600640 1 -- start start 2024-03-04T09:25:54.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.754+0000 7f4c9c600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4c940ccad0 0x7f4c9415f290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:54.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.754+0000 7f4c9c600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4c940cdab0 0x7f4c9409cde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:54.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.755+0000 7f4c9c600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4c94162bb0 0x7f4c9409d320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:54.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.755+0000 7f4c9c600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c940ad5e0 con 0x7f4c94162bb0 2024-03-04T09:25:54.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.755+0000 7f4c9c600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c940ad300 con 0x7f4c940ccad0 2024-03-04T09:25:54.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.755+0000 7f4c9c600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c9406e7d0 con 0x7f4c940cdab0 2024-03-04T09:25:54.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.755+0000 7f4c9a600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4c94162bb0 0x7f4c9409d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:54.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.755+0000 7f4c9a600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4c94162bb0 0x7f4c9409d320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51996/0 (socket says 172.21.15.6:51996) 2024-03-04T09:25:54.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.755+0000 7f4c9a600640 1 -- 172.21.15.6:0/138024344 learned_addr learned my addr 172.21.15.6:0/138024344 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:25:54.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.755+0000 7f4c99200640 1 --2- 172.21.15.6:0/138024344 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4c940cdab0 0x7f4c9409cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:54.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.755+0000 7f4c99c00640 1 --2- 172.21.15.6:0/138024344 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4c940ccad0 0x7f4c9415f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:54.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.755+0000 7f4c9a600640 1 -- 172.21.15.6:0/138024344 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4c940cdab0 msgr2=0x7f4c9409cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:54.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.755+0000 7f4c9a600640 1 --2- 172.21.15.6:0/138024344 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4c940cdab0 0x7f4c9409cde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:54.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.755+0000 7f4c9a600640 1 -- 172.21.15.6:0/138024344 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4c940ccad0 msgr2=0x7f4c9415f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:54.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.755+0000 7f4c9a600640 1 --2- 172.21.15.6:0/138024344 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4c940ccad0 0x7f4c9415f290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:54.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.755+0000 7f4c9a600640 1 -- 172.21.15.6:0/138024344 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c9409b5f0 con 0x7f4c94162bb0 2024-03-04T09:25:54.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.755+0000 7f4c99200640 1 --2- 172.21.15.6:0/138024344 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4c940cdab0 0x7f4c9409cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:25:54.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.755+0000 7f4c99c00640 1 --2- 172.21.15.6:0/138024344 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4c940ccad0 0x7f4c9415f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:25:54.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.756+0000 7f4c9a600640 1 --2- 172.21.15.6:0/138024344 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4c94162bb0 0x7f4c9409d320 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f4c8c00b970 tx=0x7f4c8c00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:54.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.756+0000 7f4c91600640 1 -- 172.21.15.6:0/138024344 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c8c019070 con 0x7f4c94162bb0 2024-03-04T09:25:54.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.756+0000 7f4c91600640 1 -- 172.21.15.6:0/138024344 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c8c00cb40 con 0x7f4c94162bb0 2024-03-04T09:25:54.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.756+0000 7f4c91600640 1 -- 172.21.15.6:0/138024344 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c8c014440 con 0x7f4c94162bb0 2024-03-04T09:25:54.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.756+0000 7f4c9c600640 1 -- 172.21.15.6:0/138024344 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c9409b8d0 con 0x7f4c94162bb0 2024-03-04T09:25:54.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.756+0000 7f4c9c600640 1 -- 172.21.15.6:0/138024344 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c9409be90 con 0x7f4c94162bb0 2024-03-04T09:25:54.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.757+0000 7f4c9c600640 1 -- 172.21.15.6:0/138024344 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c9404f330 con 0x7f4c94162bb0 2024-03-04T09:25:54.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.758+0000 7f4c91600640 1 -- 172.21.15.6:0/138024344 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4c8c01c070 con 0x7f4c94162bb0 2024-03-04T09:25:54.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.759+0000 7f4c91600640 1 --2- 172.21.15.6:0/138024344 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f4c7406b4d0 0x7f4c7406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:54.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.759+0000 7f4c91600640 1 -- 172.21.15.6:0/138024344 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4c8c089ec0 con 0x7f4c94162bb0 2024-03-04T09:25:54.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.760+0000 7f4c99c00640 1 --2- 172.21.15.6:0/138024344 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f4c7406b4d0 0x7f4c7406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:54.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.761+0000 7f4c99c00640 1 --2- 172.21.15.6:0/138024344 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f4c7406b4d0 0x7f4c7406d990 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f4c88036000 tx=0x7f4c88008030 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:54.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.764+0000 7f4c91600640 1 -- 172.21.15.6:0/138024344 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4c8c058d30 con 0x7f4c94162bb0 2024-03-04T09:25:54.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.963+0000 7f4c9c600640 1 -- 172.21.15.6:0/138024344 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f4c9406ba90 con 0x7f4c94162bb0 2024-03-04T09:25:54.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.964+0000 7f4c91600640 1 -- 172.21.15.6:0/138024344 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7f4c8c058630 con 0x7f4c94162bb0 2024-03-04T09:25:54.961 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-03-04T09:25:54.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.966+0000 7f4c9c600640 1 -- 172.21.15.6:0/138024344 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f4c7406b4d0 msgr2=0x7f4c7406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:54.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.966+0000 7f4c9c600640 1 --2- 172.21.15.6:0/138024344 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f4c7406b4d0 0x7f4c7406d990 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f4c88036000 tx=0x7f4c88008030 comp rx=0 tx=0).stop 2024-03-04T09:25:54.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.966+0000 7f4c9c600640 1 -- 172.21.15.6:0/138024344 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4c94162bb0 msgr2=0x7f4c9409d320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:54.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.966+0000 7f4c9c600640 1 --2- 172.21.15.6:0/138024344 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4c94162bb0 0x7f4c9409d320 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f4c8c00b970 tx=0x7f4c8c00be30 comp rx=0 tx=0).stop 2024-03-04T09:25:54.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.967+0000 7f4c9c600640 1 -- 172.21.15.6:0/138024344 shutdown_connections 2024-03-04T09:25:54.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.967+0000 7f4c9c600640 1 --2- 172.21.15.6:0/138024344 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4c94162bb0 0x7f4c9409d320 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:54.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.967+0000 7f4c9c600640 1 --2- 172.21.15.6:0/138024344 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f4c7406b4d0 0x7f4c7406d990 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:54.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.967+0000 7f4c9c600640 1 --2- 172.21.15.6:0/138024344 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4c940cdab0 0x7f4c9409cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:54.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.967+0000 7f4c9c600640 1 --2- 172.21.15.6:0/138024344 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4c940ccad0 0x7f4c9415f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:54.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.967+0000 7f4c9c600640 1 -- 172.21.15.6:0/138024344 >> 172.21.15.6:0/138024344 conn(0x7f4c9406b3d0 msgr2=0x7f4c94161c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:54.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.967+0000 7f4c9c600640 1 -- 172.21.15.6:0/138024344 shutdown_connections 2024-03-04T09:25:54.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:54.967+0000 7f4c9c600640 1 -- 172.21.15.6:0/138024344 wait complete. 2024-03-04T09:25:54.982 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-04T09:25:55.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.210+0000 7f5709600640 1 Processor -- start 2024-03-04T09:25:55.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.211+0000 7f5709600640 1 -- start start 2024-03-04T09:25:55.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.212+0000 7f5709600640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f5704145430 0x7f570415f810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:55.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.212+0000 7f5709600640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f570415fe40 con 0x7f57040b2250 2024-03-04T09:25:55.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.212+0000 7f5709600640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5704160120 con 0x7f570415cc90 2024-03-04T09:25:55.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.212+0000 7f5709600640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5704160400 con 0x7f5704145430 2024-03-04T09:25:55.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.213+0000 7f5708c00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f57040b2250 0x7f57040b2fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:58710/0 (socket says 172.21.15.6:58710) 2024-03-04T09:25:55.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.213+0000 7f5708c00640 1 -- 172.21.15.6:0/2745761997 learned_addr learned my addr 172.21.15.6:0/2745761997 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:25:55.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.213+0000 7f5703e00640 1 --2- 172.21.15.6:0/2745761997 >> v2:172.21.15.146:3300/0 conn(0x7f5704145430 0x7f570415f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:55.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.213+0000 7f5702a00640 1 -- 172.21.15.6:0/2745761997 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2433993371 0 0) 0x7f570415fe40 con 0x7f57040b2250 2024-03-04T09:25:55.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.213+0000 7f5702a00640 1 -- 172.21.15.6:0/2745761997 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56e4003520 con 0x7f57040b2250 2024-03-04T09:25:55.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.213+0000 7f5702a00640 1 -- 172.21.15.6:0/2745761997 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 672913515 0 0) 0x7f56e4003520 con 0x7f57040b2250 2024-03-04T09:25:55.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.213+0000 7f5702a00640 1 -- 172.21.15.6:0/2745761997 >> v2:172.21.15.146:3300/0 conn(0x7f5704145430 msgr2=0x7f570415f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:55.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.213+0000 7f5702a00640 1 --2- 172.21.15.6:0/2745761997 >> v2:172.21.15.146:3300/0 conn(0x7f5704145430 0x7f570415f810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:55.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.213+0000 7f5702a00640 1 -- 172.21.15.6:0/2745761997 >> v1:172.21.15.146:6789/0 conn(0x7f570415cc90 legacy=0x7f57040b1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:55.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.213+0000 7f5702a00640 1 -- 172.21.15.6:0/2745761997 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57041606e0 con 0x7f57040b2250 2024-03-04T09:25:55.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.214+0000 7f5702a00640 1 -- 172.21.15.6:0/2745761997 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7f56fc001670 con 0x7f57040b2250 2024-03-04T09:25:55.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.214+0000 7f5702a00640 1 -- 172.21.15.6:0/2745761997 >> v1:172.21.15.6:6789/0 conn(0x7f57040b2250 legacy=0x7f57040b2fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:55.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.214+0000 7f5702a00640 1 --2- 172.21.15.6:0/2745761997 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f56e40039b0 0x7f56e4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:55.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.214+0000 7f5702a00640 1 --2- 172.21.15.6:0/2745761997 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f56e4004800 0x7f56e4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:55.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.214+0000 7f5702a00640 1 --2- 172.21.15.6:0/2745761997 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f56e40051c0 0x7f56e400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:55.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.214+0000 7f5702a00640 1 -- 172.21.15.6:0/2745761997 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5704160400 con 0x7f56e4004800 2024-03-04T09:25:55.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.214+0000 7f5702a00640 1 -- 172.21.15.6:0/2745761997 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56e400c060 con 0x7f56e40051c0 2024-03-04T09:25:55.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.214+0000 7f5702a00640 1 -- 172.21.15.6:0/2745761997 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56e400c260 con 0x7f56e40039b0 2024-03-04T09:25:55.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.214+0000 7f5703400640 1 --2- 172.21.15.6:0/2745761997 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f56e4004800 0x7f56e4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:55.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.214+0000 7f5708c00640 1 --2- 172.21.15.6:0/2745761997 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f56e40051c0 0x7f56e400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:55.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.214+0000 7f5703e00640 1 --2- 172.21.15.6:0/2745761997 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f56e40039b0 0x7f56e4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:55.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.214+0000 7f5703400640 1 -- 172.21.15.6:0/2745761997 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f56e40039b0 msgr2=0x7f56e4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:55.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.214+0000 7f5703400640 1 --2- 172.21.15.6:0/2745761997 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f56e40039b0 0x7f56e4003dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:55.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.214+0000 7f5703400640 1 -- 172.21.15.6:0/2745761997 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f56e40051c0 msgr2=0x7f56e400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:55.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.214+0000 7f5703400640 1 --2- 172.21.15.6:0/2745761997 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f56e40051c0 0x7f56e400b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:55.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.214+0000 7f5703400640 1 -- 172.21.15.6:0/2745761997 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56e4003520 con 0x7f56e4004800 2024-03-04T09:25:55.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.214+0000 7f5708c00640 1 -- 172.21.15.6:0/2745761997 reap_dead start 2024-03-04T09:25:55.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.215+0000 7f5703400640 1 --2- 172.21.15.6:0/2745761997 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f56e4004800 0x7f56e4004c80 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f56ec001e50 tx=0x7f56e4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c40f83b09cfa957 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:55.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.216+0000 7f5702a00640 1 -- 172.21.15.6:0/2745761997 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56ec001a70 con 0x7f56e4004800 2024-03-04T09:25:55.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.216+0000 7f5702a00640 1 -- 172.21.15.6:0/2745761997 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56ec003b70 con 0x7f56e4004800 2024-03-04T09:25:55.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.216+0000 7f5702a00640 1 -- 172.21.15.6:0/2745761997 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56ec0050d0 con 0x7f56e4004800 2024-03-04T09:25:55.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.216+0000 7f5709600640 1 -- 172.21.15.6:0/2745761997 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f56e4004800 msgr2=0x7f56e4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:55.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.216+0000 7f5709600640 1 --2- 172.21.15.6:0/2745761997 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f56e4004800 0x7f56e4004c80 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f56ec001e50 tx=0x7f56e4005830 comp rx=0 tx=0).stop 2024-03-04T09:25:55.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.216+0000 7f5709600640 1 -- 172.21.15.6:0/2745761997 shutdown_connections 2024-03-04T09:25:55.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.216+0000 7f5709600640 1 --2- 172.21.15.6:0/2745761997 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f56e4004800 0x7f56e4004c80 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:55.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.216+0000 7f5709600640 1 -- 172.21.15.6:0/2745761997 >> 172.21.15.6:0/2745761997 conn(0x7f570406b3a0 msgr2=0x7f570406b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:55.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.216+0000 7f5709600640 1 -- 172.21.15.6:0/2745761997 shutdown_connections 2024-03-04T09:25:55.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.217+0000 7f5709600640 1 -- 172.21.15.6:0/2745761997 wait complete. 2024-03-04T09:25:55.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.218+0000 7f5709600640 1 Processor -- start 2024-03-04T09:25:55.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.218+0000 7f5709600640 1 -- start start 2024-03-04T09:25:55.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.219+0000 7f5709600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f56e4004800 0x7f5704072430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:55.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.219+0000 7f5709600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5704070f00 0x7f5704142040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:55.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.219+0000 7f5709600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f570415c5b0 0x7f57041540a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:55.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.219+0000 7f5709600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5704145990 con 0x7f570415c5b0 2024-03-04T09:25:55.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.219+0000 7f5709600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5704145810 con 0x7f56e4004800 2024-03-04T09:25:55.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.219+0000 7f5709600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57040cde00 con 0x7f5704070f00 2024-03-04T09:25:55.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.220+0000 7f5708c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f570415c5b0 0x7f57041540a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:55.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.220+0000 7f5708c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f570415c5b0 0x7f57041540a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:52020/0 (socket says 172.21.15.6:52020) 2024-03-04T09:25:55.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.220+0000 7f5708c00640 1 -- 172.21.15.6:0/2367395250 learned_addr learned my addr 172.21.15.6:0/2367395250 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:25:55.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.220+0000 7f5703e00640 1 --2- 172.21.15.6:0/2367395250 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f56e4004800 0x7f5704072430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:55.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.220+0000 7f5703400640 1 --2- 172.21.15.6:0/2367395250 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5704070f00 0x7f5704142040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:55.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.220+0000 7f5708c00640 1 -- 172.21.15.6:0/2367395250 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5704070f00 msgr2=0x7f5704142040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:55.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.220+0000 7f5708c00640 1 --2- 172.21.15.6:0/2367395250 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5704070f00 0x7f5704142040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:55.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.220+0000 7f5708c00640 1 -- 172.21.15.6:0/2367395250 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f56e4004800 msgr2=0x7f5704072430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:55.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.220+0000 7f5708c00640 1 --2- 172.21.15.6:0/2367395250 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f56e4004800 0x7f5704072430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:55.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.220+0000 7f5708c00640 1 -- 172.21.15.6:0/2367395250 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5704071390 con 0x7f570415c5b0 2024-03-04T09:25:55.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.221+0000 7f5708c00640 1 --2- 172.21.15.6:0/2367395250 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f570415c5b0 0x7f57041540a0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f57040b2af0 tx=0x7f56fc00d900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:55.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.221+0000 7f56fbe00640 1 -- 172.21.15.6:0/2367395250 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56fc0074e0 con 0x7f570415c5b0 2024-03-04T09:25:55.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.221+0000 7f56fbe00640 1 -- 172.21.15.6:0/2367395250 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56fc0047e0 con 0x7f570415c5b0 2024-03-04T09:25:55.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.221+0000 7f56fbe00640 1 -- 172.21.15.6:0/2367395250 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56fc004bd0 con 0x7f570415c5b0 2024-03-04T09:25:55.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.221+0000 7f5709600640 1 -- 172.21.15.6:0/2367395250 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5704071670 con 0x7f570415c5b0 2024-03-04T09:25:55.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.221+0000 7f5709600640 1 -- 172.21.15.6:0/2367395250 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5704071c30 con 0x7f570415c5b0 2024-03-04T09:25:55.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.222+0000 7f5709600640 1 -- 172.21.15.6:0/2367395250 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f570404f330 con 0x7f570415c5b0 2024-03-04T09:25:55.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.223+0000 7f56fbe00640 1 -- 172.21.15.6:0/2367395250 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f56fc0133d0 con 0x7f570415c5b0 2024-03-04T09:25:55.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.224+0000 7f56fbe00640 1 --2- 172.21.15.6:0/2367395250 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f56e006b4d0 0x7f56e006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:55.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.224+0000 7f56fbe00640 1 -- 172.21.15.6:0/2367395250 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f56fc089f20 con 0x7f570415c5b0 2024-03-04T09:25:55.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.225+0000 7f5703e00640 1 --2- 172.21.15.6:0/2367395250 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f56e006b4d0 0x7f56e006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:55.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.226+0000 7f5703e00640 1 --2- 172.21.15.6:0/2367395250 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f56e006b4d0 0x7f56e006d990 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f5704069ad0 tx=0x7f56f4008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:55.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.228+0000 7f56fbe00640 1 -- 172.21.15.6:0/2367395250 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f56fc004d70 con 0x7f570415c5b0 2024-03-04T09:25:55.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.418+0000 7f5709600640 1 -- 172.21.15.6:0/2367395250 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f570415ecd0 con 0x7f570415c5b0 2024-03-04T09:25:55.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.419+0000 7f56fbe00640 1 -- 172.21.15.6:0/2367395250 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f56fc058d90 con 0x7f570415c5b0 2024-03-04T09:25:55.416 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 64 2024-03-04T09:25:55.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.421+0000 7f5709600640 1 -- 172.21.15.6:0/2367395250 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f56e006b4d0 msgr2=0x7f56e006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:55.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.421+0000 7f5709600640 1 --2- 172.21.15.6:0/2367395250 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f56e006b4d0 0x7f56e006d990 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f5704069ad0 tx=0x7f56f4008040 comp rx=0 tx=0).stop 2024-03-04T09:25:55.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.421+0000 7f5709600640 1 -- 172.21.15.6:0/2367395250 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f570415c5b0 msgr2=0x7f57041540a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:55.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.421+0000 7f5709600640 1 --2- 172.21.15.6:0/2367395250 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f570415c5b0 0x7f57041540a0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f57040b2af0 tx=0x7f56fc00d900 comp rx=0 tx=0).stop 2024-03-04T09:25:55.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.421+0000 7f5709600640 1 -- 172.21.15.6:0/2367395250 shutdown_connections 2024-03-04T09:25:55.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.421+0000 7f5709600640 1 --2- 172.21.15.6:0/2367395250 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f570415c5b0 0x7f57041540a0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:55.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.421+0000 7f5709600640 1 --2- 172.21.15.6:0/2367395250 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f56e006b4d0 0x7f56e006d990 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:55.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.421+0000 7f5709600640 1 --2- 172.21.15.6:0/2367395250 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5704070f00 0x7f5704142040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:55.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.421+0000 7f5709600640 1 --2- 172.21.15.6:0/2367395250 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f56e4004800 0x7f5704072430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:55.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.421+0000 7f5709600640 1 -- 172.21.15.6:0/2367395250 >> 172.21.15.6:0/2367395250 conn(0x7f570406b3a0 msgr2=0x7f570415d2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:55.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.421+0000 7f5709600640 1 -- 172.21.15.6:0/2367395250 shutdown_connections 2024-03-04T09:25:55.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.421+0000 7f5709600640 1 -- 172.21.15.6:0/2367395250 wait complete. 2024-03-04T09:25:55.435 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T09:25:55.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.656+0000 7ff3d7e00640 1 Processor -- start 2024-03-04T09:25:55.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.656+0000 7ff3d7e00640 1 -- start start 2024-03-04T09:25:55.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.658+0000 7ff3d7e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ff3d815efc0 0x7ff3d80cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:55.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.658+0000 7ff3d7e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff3d81621d0 0x7ff3d8070c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:55.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.658+0000 7ff3d7e00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3d81627f0 con 0x7ff3d80cce90 2024-03-04T09:25:55.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.658+0000 7ff3d7e00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3d815f3a0 con 0x7ff3d815efc0 2024-03-04T09:25:55.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.658+0000 7ff3d7e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3d806d190 con 0x7ff3d81621d0 2024-03-04T09:25:55.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.659+0000 7ff3d7400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff3d81621d0 0x7ff3d8070c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:55.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.659+0000 7ff3d6000640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7ff3d80cce90 0x7ff3d8161ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.6:50206/0 (socket says 172.21.15.6:50206) 2024-03-04T09:25:55.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.659+0000 7ff3d6000640 1 -- 172.21.15.6:0/4293102389 learned_addr learned my addr 172.21.15.6:0/4293102389 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:25:55.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.659+0000 7ff3d6a00640 1 --2- 172.21.15.6:0/4293102389 >> v2:172.21.15.146:3300/0 conn(0x7ff3d815efc0 0x7ff3d80cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:55.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.659+0000 7ff3d5600640 1 -- 172.21.15.6:0/4293102389 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 15339120 0 0) 0x7ff3d81627f0 con 0x7ff3d80cce90 2024-03-04T09:25:55.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.660+0000 7ff3d5600640 1 -- 172.21.15.6:0/4293102389 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3bc003520 con 0x7ff3d80cce90 2024-03-04T09:25:55.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.660+0000 7ff3d6a00640 1 -- 172.21.15.6:0/4293102389 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff3d81621d0 msgr2=0x7ff3d8070c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:55.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.660+0000 7ff3d6a00640 1 --2- 172.21.15.6:0/4293102389 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff3d81621d0 0x7ff3d8070c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:55.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.660+0000 7ff3d6a00640 1 -- 172.21.15.6:0/4293102389 >> v1:172.21.15.146:6789/0 conn(0x7ff3d80cce90 legacy=0x7ff3d8161ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:55.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.660+0000 7ff3d6a00640 1 -- 172.21.15.6:0/4293102389 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3d806c860 con 0x7ff3d815efc0 2024-03-04T09:25:55.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.661+0000 7ff3d6a00640 1 --2- 172.21.15.6:0/4293102389 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff3d815efc0 0x7ff3d80cc950 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7ff3c80097a0 tx=0x7ff3c802b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4cb588c091755e6d server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:55.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.661+0000 7ff3d5600640 1 -- 172.21.15.6:0/4293102389 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3c8035020 con 0x7ff3d815efc0 2024-03-04T09:25:55.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.661+0000 7ff3d5600640 1 -- 172.21.15.6:0/4293102389 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3c8031810 con 0x7ff3d815efc0 2024-03-04T09:25:55.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.662+0000 7ff3d5600640 1 -- 172.21.15.6:0/4293102389 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3c8031be0 con 0x7ff3d815efc0 2024-03-04T09:25:55.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.662+0000 7ff3d7e00640 1 -- 172.21.15.6:0/4293102389 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff3d815efc0 msgr2=0x7ff3d80cc950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:55.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.662+0000 7ff3d7e00640 1 --2- 172.21.15.6:0/4293102389 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff3d815efc0 0x7ff3d80cc950 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7ff3c80097a0 tx=0x7ff3c802b8b0 comp rx=0 tx=0).stop 2024-03-04T09:25:55.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.662+0000 7ff3d7e00640 1 -- 172.21.15.6:0/4293102389 shutdown_connections 2024-03-04T09:25:55.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.662+0000 7ff3d7e00640 1 --2- 172.21.15.6:0/4293102389 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff3d81621d0 0x7ff3d8070c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:55.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.662+0000 7ff3d7e00640 1 --2- 172.21.15.6:0/4293102389 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff3d815efc0 0x7ff3d80cc950 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:55.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.662+0000 7ff3d7e00640 1 -- 172.21.15.6:0/4293102389 >> 172.21.15.6:0/4293102389 conn(0x7ff3d806b3d0 msgr2=0x7ff3d806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:55.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.662+0000 7ff3d7e00640 1 -- 172.21.15.6:0/4293102389 shutdown_connections 2024-03-04T09:25:55.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.662+0000 7ff3d7e00640 1 -- 172.21.15.6:0/4293102389 wait complete. 2024-03-04T09:25:55.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.665+0000 7ff3d7e00640 1 Processor -- start 2024-03-04T09:25:55.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.665+0000 7ff3d7e00640 1 -- start start 2024-03-04T09:25:55.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.666+0000 7ff3d7e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff3d80cce90 0x7ff3d8157c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:55.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.666+0000 7ff3d7e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff3d815efc0 0x7ff3d81596a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:55.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.666+0000 7ff3d7e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff3d81621d0 0x7ff3d8159be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:55.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.666+0000 7ff3d7e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3d806d620 con 0x7ff3d815efc0 2024-03-04T09:25:55.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.666+0000 7ff3d7e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3d806ce20 con 0x7ff3d80cce90 2024-03-04T09:25:55.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.666+0000 7ff3d7e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3d8137b90 con 0x7ff3d81621d0 2024-03-04T09:25:55.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.666+0000 7ff3d6000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff3d815efc0 0x7ff3d81596a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:55.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.666+0000 7ff3d6000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff3d815efc0 0x7ff3d81596a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:52026/0 (socket says 172.21.15.6:52026) 2024-03-04T09:25:55.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.666+0000 7ff3d6000640 1 -- 172.21.15.6:0/3269612666 learned_addr learned my addr 172.21.15.6:0/3269612666 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:25:55.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.667+0000 7ff3d6a00640 1 --2- 172.21.15.6:0/3269612666 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff3d80cce90 0x7ff3d8157c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:55.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.667+0000 7ff3d7400640 1 --2- 172.21.15.6:0/3269612666 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff3d81621d0 0x7ff3d8159be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:55.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.667+0000 7ff3d6000640 1 -- 172.21.15.6:0/3269612666 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff3d81621d0 msgr2=0x7ff3d8159be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:55.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.667+0000 7ff3d6000640 1 --2- 172.21.15.6:0/3269612666 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff3d81621d0 0x7ff3d8159be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:55.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.667+0000 7ff3d6000640 1 -- 172.21.15.6:0/3269612666 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff3d80cce90 msgr2=0x7ff3d8157c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:55.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.667+0000 7ff3d6000640 1 --2- 172.21.15.6:0/3269612666 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff3d80cce90 0x7ff3d8157c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:55.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.667+0000 7ff3d6000640 1 -- 172.21.15.6:0/3269612666 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3d8158bc0 con 0x7ff3d815efc0 2024-03-04T09:25:55.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.667+0000 7ff3d6000640 1 --2- 172.21.15.6:0/3269612666 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff3d815efc0 0x7ff3d81596a0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7ff3c400c430 tx=0x7ff3c400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:55.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.668+0000 7ff3cea00640 1 -- 172.21.15.6:0/3269612666 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3c4017070 con 0x7ff3d815efc0 2024-03-04T09:25:55.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.668+0000 7ff3cea00640 1 -- 172.21.15.6:0/3269612666 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3c4005ce0 con 0x7ff3d815efc0 2024-03-04T09:25:55.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.668+0000 7ff3cea00640 1 -- 172.21.15.6:0/3269612666 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3c40072f0 con 0x7ff3d815efc0 2024-03-04T09:25:55.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.668+0000 7ff3d7e00640 1 -- 172.21.15.6:0/3269612666 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3d8158ea0 con 0x7ff3d815efc0 2024-03-04T09:25:55.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.668+0000 7ff3d7e00640 1 -- 172.21.15.6:0/3269612666 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3d81581d0 con 0x7ff3d815efc0 2024-03-04T09:25:55.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.669+0000 7ff3d7e00640 1 -- 172.21.15.6:0/3269612666 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3d804f330 con 0x7ff3d815efc0 2024-03-04T09:25:55.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.670+0000 7ff3cea00640 1 -- 172.21.15.6:0/3269612666 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff3c401a070 con 0x7ff3d815efc0 2024-03-04T09:25:55.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.671+0000 7ff3cea00640 1 --2- 172.21.15.6:0/3269612666 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff3b806b4d0 0x7ff3b806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:55.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.671+0000 7ff3cea00640 1 -- 172.21.15.6:0/3269612666 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff3c4089750 con 0x7ff3d815efc0 2024-03-04T09:25:55.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.672+0000 7ff3d6a00640 1 --2- 172.21.15.6:0/3269612666 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff3b806b4d0 0x7ff3b806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:55.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.672+0000 7ff3d6a00640 1 --2- 172.21.15.6:0/3269612666 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff3b806b4d0 0x7ff3b806d990 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7ff3c8036000 tx=0x7ff3c8008030 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:55.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.675+0000 7ff3cea00640 1 -- 172.21.15.6:0/3269612666 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff3c4058530 con 0x7ff3d815efc0 2024-03-04T09:25:55.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.880+0000 7ff3d7e00640 1 -- 172.21.15.6:0/3269612666 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff3d80cf420 con 0x7ff3d815efc0 2024-03-04T09:25:55.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.883+0000 7ff3cea00640 1 -- 172.21.15.6:0/3269612666 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16557 (secure 0 0 0) 0x7ff3c4057e30 con 0x7ff3d815efc0 2024-03-04T09:25:55.880 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:25:55.880 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":20,"fsid":"edce0578-5ae4-4402-b45a-552bdef84d7c","created":"2024-03-04T09:24:54.523780+0000","modified":"2024-03-04T09:25:20.446333+0000","last_up_change":"2024-03-04T09:25:07.089049+0000","last_in_change":"2024-03-04T09:24:57.982077+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T09:25:07.935198+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-04T09:25:17.118573+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-04T09:25:17.930536+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a96574f-81f1-4bc6-8463-757520b84a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6825","nonce":1364449401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6827","nonce":1364449401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6831","nonce":1364449401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6829","nonce":1364449401}]},"public_addr":"172.21.15.6:6825/1364449401","cluster_addr":"172.21.15.6:6827/1364449401","heartbeat_back_addr":"172.21.15.6:6831/1364449401","heartbeat_front_addr":"172.21.15.6:6829/1364449401","state":["exists","up"]},{"osd":1,"uuid":"7d79b9c1-6b87-476a-bc36-7db5ee64cd90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6809","nonce":2494103078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6811","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6813","nonce":2494103078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6821","nonce":2494103078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6817","nonce":2494103078}]},"public_addr":"172.21.15.6:6809/2494103078","cluster_addr":"172.21.15.6:6813/2494103078","heartbeat_back_addr":"172.21.15.6:6821/2494103078","heartbeat_front_addr":"172.21.15.6:6817/2494103078","state":["exists","up"]},{"osd":2,"uuid":"492331b9-4ee9-40b6-bacd-85d44cd6c45a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6812","nonce":1500066681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6816","nonce":1500066681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6824","nonce":1500066681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6820","nonce":1500066681}]},"public_addr":"172.21.15.6:6812/1500066681","cluster_addr":"172.21.15.6:6816/1500066681","heartbeat_back_addr":"172.21.15.6:6824/1500066681","heartbeat_front_addr":"172.21.15.6:6820/1500066681","state":["exists","up"]},{"osd":3,"uuid":"21bb2efe-9ba7-4bdc-bc31-180cd1102e21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6801","nonce":1558806123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6803","nonce":1558806123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6807","nonce":1558806123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6805","nonce":1558806123}]},"public_addr":"172.21.15.6:6801/1558806123","cluster_addr":"172.21.15.6:6803/1558806123","heartbeat_back_addr":"172.21.15.6:6807/1558806123","heartbeat_front_addr":"172.21.15.6:6805/1558806123","state":["exists","up"]},{"osd":4,"uuid":"6e2c4b71-19d9-4737-83d9-558705d8644a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6801","nonce":1526135780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6803","nonce":1526135780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6807","nonce":1526135780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6805","nonce":1526135780}]},"public_addr":"172.21.15.146:6801/1526135780","cluster_addr":"172.21.15.146:6803/1526135780","heartbeat_back_addr":"172.21.15.146:6807/1526135780","heartbeat_front_addr":"172.21.15.146:6805/1526135780","state":["exists","up"]},{"osd":5,"uuid":"fb69c525-f1be-4b7d-ab32-6c29b0d6fddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6809","nonce":460262507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6811","nonce":460262507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6815","nonce":460262507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6813","nonce":460262507}]},"public_addr":"172.21.15.146:6809/460262507","cluster_addr":"172.21.15.146:6811/460262507","heartbeat_back_addr":"172.21.15.146:6815/460262507","heartbeat_front_addr":"172.21.15.146:6813/460262507","state":["exists","up"]},{"osd":6,"uuid":"c32e7fed-370d-4401-8ce0-6a4335b0d168","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6823","nonce":1392897493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6827","nonce":1392897493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6831","nonce":1392897493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6829","nonce":1392897493}]},"public_addr":"172.21.15.146:6823/1392897493","cluster_addr":"172.21.15.146:6827/1392897493","heartbeat_back_addr":"172.21.15.146:6831/1392897493","heartbeat_front_addr":"172.21.15.146:6829/1392897493","state":["exists","up"]},{"osd":7,"uuid":"75a79b44-fa5f-426d-a7d3-71f21f78835c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6817","nonce":3268323156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6819","nonce":3268323156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6825","nonce":3268323156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6822","nonce":3268323156}]},"public_addr":"172.21.15.146:6817/3268323156","cluster_addr":"172.21.15.146:6819/3268323156","heartbeat_back_addr":"172.21.15.146:6825/3268323156","heartbeat_front_addr":"172.21.15.146:6822/3268323156","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.310437+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.295850+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.294674+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.526855+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.300023+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.595160+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.346046+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.558741+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T09:25:55.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.885+0000 7ff3d7e00640 1 -- 172.21.15.6:0/3269612666 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff3b806b4d0 msgr2=0x7ff3b806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:55.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.885+0000 7ff3d7e00640 1 --2- 172.21.15.6:0/3269612666 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff3b806b4d0 0x7ff3b806d990 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7ff3c8036000 tx=0x7ff3c8008030 comp rx=0 tx=0).stop 2024-03-04T09:25:55.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.885+0000 7ff3d7e00640 1 -- 172.21.15.6:0/3269612666 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff3d815efc0 msgr2=0x7ff3d81596a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:55.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.885+0000 7ff3d7e00640 1 --2- 172.21.15.6:0/3269612666 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff3d815efc0 0x7ff3d81596a0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7ff3c400c430 tx=0x7ff3c400c8f0 comp rx=0 tx=0).stop 2024-03-04T09:25:55.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.885+0000 7ff3d7e00640 1 -- 172.21.15.6:0/3269612666 shutdown_connections 2024-03-04T09:25:55.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.885+0000 7ff3d7e00640 1 --2- 172.21.15.6:0/3269612666 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff3b806b4d0 0x7ff3b806d990 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:55.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.885+0000 7ff3d7e00640 1 --2- 172.21.15.6:0/3269612666 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff3d81621d0 0x7ff3d8159be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:55.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.885+0000 7ff3d7e00640 1 --2- 172.21.15.6:0/3269612666 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff3d815efc0 0x7ff3d81596a0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:55.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.885+0000 7ff3d7e00640 1 --2- 172.21.15.6:0/3269612666 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff3d80cce90 0x7ff3d8157c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:55.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.885+0000 7ff3d7e00640 1 -- 172.21.15.6:0/3269612666 >> 172.21.15.6:0/3269612666 conn(0x7ff3d806b3d0 msgr2=0x7ff3d80ad880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:55.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.885+0000 7ff3d7e00640 1 -- 172.21.15.6:0/3269612666 shutdown_connections 2024-03-04T09:25:55.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:55.886+0000 7ff3d7e00640 1 -- 172.21.15.6:0/3269612666 wait complete. 2024-03-04T09:25:55.901 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T09:25:07.935198+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-04T09:25:17.118573+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-04T09:25:17.930536+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-04T09:25:55.901 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-04T09:25:56.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.124+0000 7f1b56e00640 1 Processor -- start 2024-03-04T09:25:56.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.124+0000 7f1b56e00640 1 -- start start 2024-03-04T09:25:56.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.126+0000 7f1b56e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f1b500b3c20 0x7f1b500729d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:56.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.126+0000 7f1b56e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1b500b1b40 0x7f1b500b1fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:56.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.126+0000 7f1b56e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b50071930 con 0x7f1b500b4000 2024-03-04T09:25:56.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.126+0000 7f1b56e00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b500b4490 con 0x7f1b500b3c20 2024-03-04T09:25:56.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.126+0000 7f1b56e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b500b24e0 con 0x7f1b500b1b40 2024-03-04T09:25:56.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.126+0000 7f1b55000640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f1b500b4000 0x7f1b5015b8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:58716/0 (socket says 172.21.15.6:58716) 2024-03-04T09:25:56.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.126+0000 7f1b55000640 1 -- 172.21.15.6:0/1531424836 learned_addr learned my addr 172.21.15.6:0/1531424836 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:25:56.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.127+0000 7f1b55a00640 1 --2- 172.21.15.6:0/1531424836 >> v2:172.21.15.146:3300/0 conn(0x7f1b500b3c20 0x7f1b500729d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:56.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.127+0000 7f1b56400640 1 --2- 172.21.15.6:0/1531424836 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1b500b1b40 0x7f1b500b1fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:56.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.127+0000 7f1b4fe00640 1 -- 172.21.15.6:0/1531424836 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 139897958 0 0) 0x7f1b50071930 con 0x7f1b500b4000 2024-03-04T09:25:56.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.127+0000 7f1b4fe00640 1 -- 172.21.15.6:0/1531424836 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b38003520 con 0x7f1b500b4000 2024-03-04T09:25:56.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.127+0000 7f1b4fe00640 1 -- 172.21.15.6:0/1531424836 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 887565007 0 0) 0x7f1b38003520 con 0x7f1b500b4000 2024-03-04T09:25:56.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.127+0000 7f1b4fe00640 1 -- 172.21.15.6:0/1531424836 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1b500b1b40 msgr2=0x7f1b500b1fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:56.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.127+0000 7f1b4fe00640 1 --2- 172.21.15.6:0/1531424836 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1b500b1b40 0x7f1b500b1fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:56.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.127+0000 7f1b4fe00640 1 -- 172.21.15.6:0/1531424836 >> v2:172.21.15.146:3300/0 conn(0x7f1b500b3c20 msgr2=0x7f1b500729d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:56.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.127+0000 7f1b4fe00640 1 --2- 172.21.15.6:0/1531424836 >> v2:172.21.15.146:3300/0 conn(0x7f1b500b3c20 0x7f1b500729d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:56.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.127+0000 7f1b4fe00640 1 -- 172.21.15.6:0/1531424836 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b50071c10 con 0x7f1b500b4000 2024-03-04T09:25:56.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.127+0000 7f1b4fe00640 1 -- 172.21.15.6:0/1531424836 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7f1b3c001690 con 0x7f1b500b4000 2024-03-04T09:25:56.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.127+0000 7f1b4fe00640 1 -- 172.21.15.6:0/1531424836 >> v1:172.21.15.6:6789/0 conn(0x7f1b500b4000 legacy=0x7f1b5015b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:56.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.128+0000 7f1b4fe00640 1 --2- 172.21.15.6:0/1531424836 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1b500b1b40 0x7f1b380039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:56.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.128+0000 7f1b4fe00640 1 --2- 172.21.15.6:0/1531424836 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1b38003ef0 0x7f1b38004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:56.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.128+0000 7f1b4fe00640 1 --2- 172.21.15.6:0/1531424836 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1b38004370 0x7f1b38004d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:56.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.128+0000 7f1b4fe00640 1 -- 172.21.15.6:0/1531424836 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b38005500 con 0x7f1b38003ef0 2024-03-04T09:25:56.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.128+0000 7f1b4fe00640 1 -- 172.21.15.6:0/1531424836 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b3800ffd0 con 0x7f1b38004370 2024-03-04T09:25:56.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.128+0000 7f1b4fe00640 1 -- 172.21.15.6:0/1531424836 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b38010150 con 0x7f1b500b1b40 2024-03-04T09:25:56.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.128+0000 7f1b55000640 1 --2- 172.21.15.6:0/1531424836 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1b38003ef0 0x7f1b38004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:56.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.128+0000 7f1b55a00640 1 --2- 172.21.15.6:0/1531424836 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1b500b1b40 0x7f1b380039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:56.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.128+0000 7f1b55000640 1 -- 172.21.15.6:0/1531424836 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1b500b1b40 msgr2=0x7f1b380039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:56.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.128+0000 7f1b55000640 1 --2- 172.21.15.6:0/1531424836 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1b500b1b40 0x7f1b380039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:56.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.128+0000 7f1b56400640 1 --2- 172.21.15.6:0/1531424836 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1b38004370 0x7f1b38004d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:56.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.128+0000 7f1b55000640 1 -- 172.21.15.6:0/1531424836 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1b38004370 msgr2=0x7f1b38004d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:56.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.128+0000 7f1b55000640 1 --2- 172.21.15.6:0/1531424836 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1b38004370 0x7f1b38004d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:56.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.128+0000 7f1b55000640 1 -- 172.21.15.6:0/1531424836 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b38003520 con 0x7f1b38003ef0 2024-03-04T09:25:56.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.128+0000 7f1b55a00640 1 --2- 172.21.15.6:0/1531424836 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1b500b1b40 0x7f1b380039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:25:56.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.128+0000 7f1b56400640 1 --2- 172.21.15.6:0/1531424836 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1b38004370 0x7f1b38004d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:25:56.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.130+0000 7f1b55000640 1 --2- 172.21.15.6:0/1531424836 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1b38003ef0 0x7f1b38004800 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f1b3c007fd0 tx=0x7f1b38005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eff81bfb540d917c server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:56.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.130+0000 7f1b4fe00640 1 -- 172.21.15.6:0/1531424836 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b3c007d50 con 0x7f1b38003ef0 2024-03-04T09:25:56.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.130+0000 7f1b4fe00640 1 -- 172.21.15.6:0/1531424836 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b3c030070 con 0x7f1b38003ef0 2024-03-04T09:25:56.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.130+0000 7f1b4fe00640 1 -- 172.21.15.6:0/1531424836 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b3c005d10 con 0x7f1b38003ef0 2024-03-04T09:25:56.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.130+0000 7f1b56e00640 1 -- 172.21.15.6:0/1531424836 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1b38003ef0 msgr2=0x7f1b38004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:56.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.130+0000 7f1b56e00640 1 --2- 172.21.15.6:0/1531424836 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1b38003ef0 0x7f1b38004800 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f1b3c007fd0 tx=0x7f1b38005830 comp rx=0 tx=0).stop 2024-03-04T09:25:56.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.130+0000 7f1b56400640 1 -- 172.21.15.6:0/1531424836 reap_dead start 2024-03-04T09:25:56.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.131+0000 7f1b56e00640 1 -- 172.21.15.6:0/1531424836 shutdown_connections 2024-03-04T09:25:56.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.131+0000 7f1b56e00640 1 -- 172.21.15.6:0/1531424836 >> 172.21.15.6:0/1531424836 conn(0x7f1b5006b3a0 msgr2=0x7f1b5006b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:56.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.131+0000 7f1b56e00640 1 -- 172.21.15.6:0/1531424836 shutdown_connections 2024-03-04T09:25:56.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.131+0000 7f1b56e00640 1 -- 172.21.15.6:0/1531424836 wait complete. 2024-03-04T09:25:56.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.132+0000 7f1b56e00640 1 Processor -- start 2024-03-04T09:25:56.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.133+0000 7f1b56e00640 1 -- start start 2024-03-04T09:25:56.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.133+0000 7f1b56e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1b38003ef0 0x7f1b50157690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:56.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.133+0000 7f1b56e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1b500ccd60 0x7f1b5015a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:56.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.134+0000 7f1b56e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1b50143f30 0x7f1b501443b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:56.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.134+0000 7f1b56e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b500721d0 con 0x7f1b38003ef0 2024-03-04T09:25:56.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.134+0000 7f1b56e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b5015bfb0 con 0x7f1b500ccd60 2024-03-04T09:25:56.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.134+0000 7f1b56e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b500724b0 con 0x7f1b50143f30 2024-03-04T09:25:56.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.134+0000 7f1b55a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1b38003ef0 0x7f1b50157690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:56.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.134+0000 7f1b55a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1b38003ef0 0x7f1b50157690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:52040/0 (socket says 172.21.15.6:52040) 2024-03-04T09:25:56.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.134+0000 7f1b55a00640 1 -- 172.21.15.6:0/669150283 learned_addr learned my addr 172.21.15.6:0/669150283 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:25:56.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.134+0000 7f1b55000640 1 --2- 172.21.15.6:0/669150283 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1b500ccd60 0x7f1b5015a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:56.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.134+0000 7f1b55a00640 1 -- 172.21.15.6:0/669150283 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1b50143f30 msgr2=0x7f1b501443b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:56.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.134+0000 7f1b55a00640 1 --2- 172.21.15.6:0/669150283 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1b50143f30 0x7f1b501443b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:56.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.134+0000 7f1b55a00640 1 -- 172.21.15.6:0/669150283 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1b500ccd60 msgr2=0x7f1b5015a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:56.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.134+0000 7f1b55a00640 1 --2- 172.21.15.6:0/669150283 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1b500ccd60 0x7f1b5015a1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:56.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.134+0000 7f1b55a00640 1 -- 172.21.15.6:0/669150283 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b50157bd0 con 0x7f1b38003ef0 2024-03-04T09:25:56.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.135+0000 7f1b55a00640 1 --2- 172.21.15.6:0/669150283 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1b38003ef0 0x7f1b50157690 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f1b44010a00 tx=0x7f1b44010ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:56.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.135+0000 7f1b4d600640 1 -- 172.21.15.6:0/669150283 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b4400f040 con 0x7f1b38003ef0 2024-03-04T09:25:56.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.135+0000 7f1b4d600640 1 -- 172.21.15.6:0/669150283 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b4400eba0 con 0x7f1b38003ef0 2024-03-04T09:25:56.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.135+0000 7f1b4d600640 1 -- 172.21.15.6:0/669150283 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b4401c400 con 0x7f1b38003ef0 2024-03-04T09:25:56.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.135+0000 7f1b56e00640 1 -- 172.21.15.6:0/669150283 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b501448f0 con 0x7f1b38003ef0 2024-03-04T09:25:56.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.135+0000 7f1b56e00640 1 -- 172.21.15.6:0/669150283 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b500cd5b0 con 0x7f1b38003ef0 2024-03-04T09:25:56.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.136+0000 7f1b56e00640 1 -- 172.21.15.6:0/669150283 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b500cdf50 con 0x7f1b38003ef0 2024-03-04T09:25:56.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.137+0000 7f1b4d600640 1 -- 172.21.15.6:0/669150283 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1b4401c5a0 con 0x7f1b38003ef0 2024-03-04T09:25:56.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.138+0000 7f1b4d600640 1 --2- 172.21.15.6:0/669150283 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f1b3006b500 0x7f1b3006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:56.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.139+0000 7f1b4d600640 1 -- 172.21.15.6:0/669150283 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1b4408a8f0 con 0x7f1b38003ef0 2024-03-04T09:25:56.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.139+0000 7f1b55000640 1 --2- 172.21.15.6:0/669150283 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f1b3006b500 0x7f1b3006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:56.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.141+0000 7f1b55000640 1 --2- 172.21.15.6:0/669150283 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f1b3006b500 0x7f1b3006d9c0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f1b3c034000 tx=0x7f1b3c007000 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:56.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.143+0000 7f1b4d600640 1 -- 172.21.15.6:0/669150283 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1b500cdf50 con 0x7f1b38003ef0 2024-03-04T09:25:56.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.339+0000 7f1b56e00640 1 -- 172.21.15.6:0/669150283 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1b5004f330 con 0x7f1b38003ef0 2024-03-04T09:25:56.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.340+0000 7f1b4d600640 1 -- 172.21.15.6:0/669150283 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1b50157bd0 con 0x7f1b38003ef0 2024-03-04T09:25:56.337 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-04T09:25:56.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.342+0000 7f1b56e00640 1 -- 172.21.15.6:0/669150283 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f1b3006b500 msgr2=0x7f1b3006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:56.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.342+0000 7f1b56e00640 1 --2- 172.21.15.6:0/669150283 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f1b3006b500 0x7f1b3006d9c0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f1b3c034000 tx=0x7f1b3c007000 comp rx=0 tx=0).stop 2024-03-04T09:25:56.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.342+0000 7f1b56e00640 1 -- 172.21.15.6:0/669150283 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1b38003ef0 msgr2=0x7f1b50157690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:56.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.342+0000 7f1b56e00640 1 --2- 172.21.15.6:0/669150283 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1b38003ef0 0x7f1b50157690 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f1b44010a00 tx=0x7f1b44010ec0 comp rx=0 tx=0).stop 2024-03-04T09:25:56.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.342+0000 7f1b56e00640 1 -- 172.21.15.6:0/669150283 shutdown_connections 2024-03-04T09:25:56.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.342+0000 7f1b56e00640 1 --2- 172.21.15.6:0/669150283 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f1b3006b500 0x7f1b3006d9c0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:56.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.342+0000 7f1b56e00640 1 --2- 172.21.15.6:0/669150283 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1b50143f30 0x7f1b501443b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:56.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.342+0000 7f1b56e00640 1 --2- 172.21.15.6:0/669150283 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1b500ccd60 0x7f1b5015a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:56.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.342+0000 7f1b56e00640 1 --2- 172.21.15.6:0/669150283 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1b38003ef0 0x7f1b50157690 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:56.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.342+0000 7f1b56e00640 1 -- 172.21.15.6:0/669150283 >> 172.21.15.6:0/669150283 conn(0x7f1b5006b3a0 msgr2=0x7f1b500b0860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:56.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.343+0000 7f1b56e00640 1 -- 172.21.15.6:0/669150283 shutdown_connections 2024-03-04T09:25:56.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.343+0000 7f1b56e00640 1 -- 172.21.15.6:0/669150283 wait complete. 2024-03-04T09:25:56.357 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-04T09:25:56.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.585+0000 7f735f200640 1 Processor -- start 2024-03-04T09:25:56.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.585+0000 7f735f200640 1 -- start start 2024-03-04T09:25:56.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.587+0000 7f735f200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f736015bb50 0x7f73600b4a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:56.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.587+0000 7f735f200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f73600b2670 0x7f73600b59e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:56.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.587+0000 7f735f200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73600b0090 con 0x7f73600b5550 2024-03-04T09:25:56.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.587+0000 7f735f200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73600b0370 con 0x7f73600b2670 2024-03-04T09:25:56.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.587+0000 7f735f200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73600b0650 con 0x7f736015bb50 2024-03-04T09:25:56.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.588+0000 7f735e800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f73600b2670 0x7f73600b59e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:56.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.588+0000 7f735e800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f73600b2670 0x7f73600b59e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:52046/0 (socket says 172.21.15.6:52046) 2024-03-04T09:25:56.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.588+0000 7f735e800640 1 -- 172.21.15.6:0/1918909124 learned_addr learned my addr 172.21.15.6:0/1918909124 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:25:56.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.588+0000 7f735de00640 1 --2- 172.21.15.6:0/1918909124 >> v2:172.21.15.146:3300/0 conn(0x7f736015bb50 0x7f73600b4a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:56.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.588+0000 7f735e800640 1 -- 172.21.15.6:0/1918909124 >> v2:172.21.15.146:3300/0 conn(0x7f736015bb50 msgr2=0x7f73600b4a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:56.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.588+0000 7f735e800640 1 --2- 172.21.15.6:0/1918909124 >> v2:172.21.15.146:3300/0 conn(0x7f736015bb50 0x7f73600b4a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:56.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.588+0000 7f735e800640 1 -- 172.21.15.6:0/1918909124 >> v1:172.21.15.146:6789/0 conn(0x7f73600b5550 legacy=0x7f736015f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:56.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.588+0000 7f735e800640 1 -- 172.21.15.6:0/1918909124 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73600b0930 con 0x7f73600b2670 2024-03-04T09:25:56.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.589+0000 7f735e800640 1 --2- 172.21.15.6:0/1918909124 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f73600b2670 0x7f73600b59e0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f73600b4790 tx=0x7f735802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=467217523eeb028f server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:56.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.590+0000 7f735d400640 1 -- 172.21.15.6:0/1918909124 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f735802ca80 con 0x7f73600b2670 2024-03-04T09:25:56.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.590+0000 7f735d400640 1 -- 172.21.15.6:0/1918909124 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f735802cc20 con 0x7f73600b2670 2024-03-04T09:25:56.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.590+0000 7f735d400640 1 -- 172.21.15.6:0/1918909124 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73580343f0 con 0x7f73600b2670 2024-03-04T09:25:56.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.590+0000 7f735f200640 1 -- 172.21.15.6:0/1918909124 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f73600b2670 msgr2=0x7f73600b59e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:56.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.590+0000 7f735f200640 1 --2- 172.21.15.6:0/1918909124 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f73600b2670 0x7f73600b59e0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f73600b4790 tx=0x7f735802bee0 comp rx=0 tx=0).stop 2024-03-04T09:25:56.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.591+0000 7f735f200640 1 -- 172.21.15.6:0/1918909124 shutdown_connections 2024-03-04T09:25:56.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.591+0000 7f735f200640 1 --2- 172.21.15.6:0/1918909124 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f73600b2670 0x7f73600b59e0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:56.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.591+0000 7f735f200640 1 --2- 172.21.15.6:0/1918909124 >> v2:172.21.15.146:3300/0 conn(0x7f736015bb50 0x7f73600b4a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:56.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.591+0000 7f735f200640 1 -- 172.21.15.6:0/1918909124 >> 172.21.15.6:0/1918909124 conn(0x7f736009aa10 msgr2=0x7f736009ae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:56.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.591+0000 7f735f200640 1 -- 172.21.15.6:0/1918909124 shutdown_connections 2024-03-04T09:25:56.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.591+0000 7f735f200640 1 -- 172.21.15.6:0/1918909124 wait complete. 2024-03-04T09:25:56.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.592+0000 7f735f200640 1 Processor -- start 2024-03-04T09:25:56.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.593+0000 7f735f200640 1 -- start start 2024-03-04T09:25:56.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.594+0000 7f735f200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f73600b2670 0x7f736009f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:56.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.594+0000 7f735f200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f73600b5550 0x7f73600ce230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:56.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.594+0000 7f735f200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f736015bb50 0x7f7360148560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:56.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.594+0000 7f735f200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73600afde0 con 0x7f736015bb50 2024-03-04T09:25:56.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.594+0000 7f735f200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73600afc60 con 0x7f73600b2670 2024-03-04T09:25:56.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.594+0000 7f735f200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73600b4f80 con 0x7f73600b5550 2024-03-04T09:25:56.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.594+0000 7f735e800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f736015bb50 0x7f7360148560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:56.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.594+0000 7f735e800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f736015bb50 0x7f7360148560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:52052/0 (socket says 172.21.15.6:52052) 2024-03-04T09:25:56.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.594+0000 7f735e800640 1 -- 172.21.15.6:0/1366214070 learned_addr learned my addr 172.21.15.6:0/1366214070 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:25:56.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.594+0000 7f7355400640 1 --2- 172.21.15.6:0/1366214070 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f73600b5550 0x7f73600ce230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:56.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.594+0000 7f735de00640 1 --2- 172.21.15.6:0/1366214070 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f73600b2670 0x7f736009f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:56.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.594+0000 7f735e800640 1 -- 172.21.15.6:0/1366214070 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f73600b5550 msgr2=0x7f73600ce230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:56.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.594+0000 7f735e800640 1 --2- 172.21.15.6:0/1366214070 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f73600b5550 0x7f73600ce230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:56.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.594+0000 7f735e800640 1 -- 172.21.15.6:0/1366214070 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f73600b2670 msgr2=0x7f736009f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:56.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.594+0000 7f735e800640 1 --2- 172.21.15.6:0/1366214070 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f73600b2670 0x7f736009f730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:56.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.594+0000 7f735e800640 1 -- 172.21.15.6:0/1366214070 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73600ce930 con 0x7f736015bb50 2024-03-04T09:25:56.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.595+0000 7f735e800640 1 --2- 172.21.15.6:0/1366214070 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f736015bb50 0x7f7360148560 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f7358009f40 tx=0x7f7358004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:56.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.595+0000 7f7356a00640 1 -- 172.21.15.6:0/1366214070 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7358043020 con 0x7f736015bb50 2024-03-04T09:25:56.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.595+0000 7f7356a00640 1 -- 172.21.15.6:0/1366214070 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f735803e450 con 0x7f736015bb50 2024-03-04T09:25:56.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.595+0000 7f7356a00640 1 -- 172.21.15.6:0/1366214070 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7358009510 con 0x7f736015bb50 2024-03-04T09:25:56.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.595+0000 7f735f200640 1 -- 172.21.15.6:0/1366214070 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f736009e310 con 0x7f736015bb50 2024-03-04T09:25:56.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.595+0000 7f735f200640 1 -- 172.21.15.6:0/1366214070 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f736009d270 con 0x7f736015bb50 2024-03-04T09:25:56.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.596+0000 7f735f200640 1 -- 172.21.15.6:0/1366214070 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f736009e4e0 con 0x7f736015bb50 2024-03-04T09:25:56.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.598+0000 7f7356a00640 1 -- 172.21.15.6:0/1366214070 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7358046020 con 0x7f736015bb50 2024-03-04T09:25:56.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.599+0000 7f7356a00640 1 --2- 172.21.15.6:0/1366214070 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f733c06b500 0x7f733c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:56.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.599+0000 7f7356a00640 1 -- 172.21.15.6:0/1366214070 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f73580ac7e0 con 0x7f736015bb50 2024-03-04T09:25:56.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.600+0000 7f735de00640 1 --2- 172.21.15.6:0/1366214070 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f733c06b500 0x7f733c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:56.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.601+0000 7f735de00640 1 --2- 172.21.15.6:0/1366214070 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f733c06b500 0x7f733c06d9c0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f7350006fd0 tx=0x7f7350008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:56.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.604+0000 7f7356a00640 1 -- 172.21.15.6:0/1366214070 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f735804d070 con 0x7f736015bb50 2024-03-04T09:25:56.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.806+0000 7f735f200640 1 -- 172.21.15.6:0/1366214070 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f736004f330 con 0x7f736015bb50 2024-03-04T09:25:56.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.807+0000 7f7356a00640 1 -- 172.21.15.6:0/1366214070 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7f73600ce930 con 0x7f736015bb50 2024-03-04T09:25:56.804 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-03-04T09:25:56.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.810+0000 7f735f200640 1 -- 172.21.15.6:0/1366214070 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f733c06b500 msgr2=0x7f733c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:56.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.810+0000 7f735f200640 1 --2- 172.21.15.6:0/1366214070 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f733c06b500 0x7f733c06d9c0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f7350006fd0 tx=0x7f7350008040 comp rx=0 tx=0).stop 2024-03-04T09:25:56.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.810+0000 7f735f200640 1 -- 172.21.15.6:0/1366214070 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f736015bb50 msgr2=0x7f7360148560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:56.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.810+0000 7f735f200640 1 --2- 172.21.15.6:0/1366214070 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f736015bb50 0x7f7360148560 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f7358009f40 tx=0x7f7358004a40 comp rx=0 tx=0).stop 2024-03-04T09:25:56.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.810+0000 7f735f200640 1 -- 172.21.15.6:0/1366214070 shutdown_connections 2024-03-04T09:25:56.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.810+0000 7f735f200640 1 --2- 172.21.15.6:0/1366214070 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f736015bb50 0x7f7360148560 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:56.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.810+0000 7f735f200640 1 --2- 172.21.15.6:0/1366214070 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f733c06b500 0x7f733c06d9c0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:56.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.810+0000 7f735f200640 1 --2- 172.21.15.6:0/1366214070 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f73600b5550 0x7f73600ce230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:56.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.810+0000 7f735f200640 1 --2- 172.21.15.6:0/1366214070 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f73600b2670 0x7f736009f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:56.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.810+0000 7f735f200640 1 -- 172.21.15.6:0/1366214070 >> 172.21.15.6:0/1366214070 conn(0x7f736009aa10 msgr2=0x7f73600b1560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:56.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.810+0000 7f735f200640 1 -- 172.21.15.6:0/1366214070 shutdown_connections 2024-03-04T09:25:56.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:56.810+0000 7f735f200640 1 -- 172.21.15.6:0/1366214070 wait complete. 2024-03-04T09:25:56.826 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-04T09:25:57.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.051+0000 7f148ca00640 1 Processor -- start 2024-03-04T09:25:57.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.051+0000 7f148ca00640 1 -- start start 2024-03-04T09:25:57.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.052+0000 7f148ca00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f148815afa0 0x7f14880cf3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:57.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.052+0000 7f148ca00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1488071c60 con 0x7f1488070ce0 2024-03-04T09:25:57.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.052+0000 7f148ca00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1488071f40 con 0x7f14880cd1e0 2024-03-04T09:25:57.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.052+0000 7f148ca00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14880cfa10 con 0x7f148815afa0 2024-03-04T09:25:57.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.053+0000 7f1485400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f1488070ce0 0x7f14880ccad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:58726/0 (socket says 172.21.15.6:58726) 2024-03-04T09:25:57.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.053+0000 7f1485400640 1 -- 172.21.15.6:0/3454141770 learned_addr learned my addr 172.21.15.6:0/3454141770 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:25:57.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.053+0000 7f1485e00640 1 --2- 172.21.15.6:0/3454141770 >> v2:172.21.15.146:3300/0 conn(0x7f148815afa0 0x7f14880cf3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:57.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.053+0000 7f1484a00640 1 -- 172.21.15.6:0/3454141770 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3179939954 0 0) 0x7f1488071c60 con 0x7f1488070ce0 2024-03-04T09:25:57.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.053+0000 7f1484a00640 1 -- 172.21.15.6:0/3454141770 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f146c003520 con 0x7f1488070ce0 2024-03-04T09:25:57.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.054+0000 7f1484a00640 1 -- 172.21.15.6:0/3454141770 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2738999954 0 0) 0x7f146c003520 con 0x7f1488070ce0 2024-03-04T09:25:57.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.054+0000 7f1484a00640 1 -- 172.21.15.6:0/3454141770 >> v2:172.21.15.146:3300/0 conn(0x7f148815afa0 msgr2=0x7f14880cf3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:57.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.054+0000 7f1484a00640 1 --2- 172.21.15.6:0/3454141770 >> v2:172.21.15.146:3300/0 conn(0x7f148815afa0 0x7f14880cf3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:57.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.054+0000 7f1484a00640 1 -- 172.21.15.6:0/3454141770 >> v1:172.21.15.146:6789/0 conn(0x7f14880cd1e0 legacy=0x7f14880b3ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:57.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.054+0000 7f1484a00640 1 -- 172.21.15.6:0/3454141770 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1488072220 con 0x7f1488070ce0 2024-03-04T09:25:57.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.054+0000 7f1484a00640 1 -- 172.21.15.6:0/3454141770 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7f1474001670 con 0x7f1488070ce0 2024-03-04T09:25:57.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.054+0000 7f1484a00640 1 -- 172.21.15.6:0/3454141770 >> v1:172.21.15.6:6789/0 conn(0x7f1488070ce0 legacy=0x7f14880ccad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:57.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.054+0000 7f1484a00640 1 --2- 172.21.15.6:0/3454141770 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f146c003a00 0x7f146c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:57.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.054+0000 7f1484a00640 1 --2- 172.21.15.6:0/3454141770 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f146c004800 0x7f146c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:57.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.054+0000 7f1484a00640 1 --2- 172.21.15.6:0/3454141770 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f146c005210 0x7f146c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:57.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.054+0000 7f1484a00640 1 -- 172.21.15.6:0/3454141770 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14880cfa10 con 0x7f146c005210 2024-03-04T09:25:57.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.054+0000 7f1484a00640 1 -- 172.21.15.6:0/3454141770 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f146c00c160 con 0x7f146c003a00 2024-03-04T09:25:57.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.054+0000 7f1484a00640 1 -- 172.21.15.6:0/3454141770 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f146c00c360 con 0x7f146c004800 2024-03-04T09:25:57.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.054+0000 7f1486800640 1 --2- 172.21.15.6:0/3454141770 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f146c005210 0x7f146c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:57.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.055+0000 7f1485e00640 1 --2- 172.21.15.6:0/3454141770 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f146c003a00 0x7f146c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:57.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.055+0000 7f1485400640 1 --2- 172.21.15.6:0/3454141770 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f146c004800 0x7f146c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:57.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.055+0000 7f1486800640 1 -- 172.21.15.6:0/3454141770 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f146c004800 msgr2=0x7f146c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:57.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.055+0000 7f1486800640 1 --2- 172.21.15.6:0/3454141770 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f146c004800 0x7f146c004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:57.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.055+0000 7f1486800640 1 -- 172.21.15.6:0/3454141770 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f146c003a00 msgr2=0x7f146c003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:57.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.055+0000 7f1486800640 1 --2- 172.21.15.6:0/3454141770 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f146c003a00 0x7f146c003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:57.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.055+0000 7f1486800640 1 -- 172.21.15.6:0/3454141770 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f146c003520 con 0x7f146c005210 2024-03-04T09:25:57.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.056+0000 7f1486800640 1 -- 172.21.15.6:0/3454141770 reap_dead start 2024-03-04T09:25:57.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.056+0000 7f1486800640 1 --2- 172.21.15.6:0/3454141770 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f146c005210 0x7f146c00b9c0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f148806de00 tx=0x7f146c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77267e61a3ce70cb server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:57.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.056+0000 7f1484a00640 1 -- 172.21.15.6:0/3454141770 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1480034070 con 0x7f146c005210 2024-03-04T09:25:57.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.056+0000 7f1484a00640 1 -- 172.21.15.6:0/3454141770 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1480003b70 con 0x7f146c005210 2024-03-04T09:25:57.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.056+0000 7f1484a00640 1 -- 172.21.15.6:0/3454141770 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1480001a70 con 0x7f146c005210 2024-03-04T09:25:57.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.056+0000 7f148ca00640 1 -- 172.21.15.6:0/3454141770 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f146c005210 msgr2=0x7f146c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:57.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.056+0000 7f148ca00640 1 --2- 172.21.15.6:0/3454141770 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f146c005210 0x7f146c00b9c0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f148806de00 tx=0x7f146c005830 comp rx=0 tx=0).stop 2024-03-04T09:25:57.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.057+0000 7f148ca00640 1 -- 172.21.15.6:0/3454141770 shutdown_connections 2024-03-04T09:25:57.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.057+0000 7f148ca00640 1 --2- 172.21.15.6:0/3454141770 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f146c005210 0x7f146c00b9c0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:57.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.057+0000 7f148ca00640 1 -- 172.21.15.6:0/3454141770 >> 172.21.15.6:0/3454141770 conn(0x7f148806b3d0 msgr2=0x7f148806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:57.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.057+0000 7f148ca00640 1 -- 172.21.15.6:0/3454141770 shutdown_connections 2024-03-04T09:25:57.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.057+0000 7f148ca00640 1 -- 172.21.15.6:0/3454141770 wait complete. 2024-03-04T09:25:57.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.058+0000 7f148ca00640 1 Processor -- start 2024-03-04T09:25:57.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.058+0000 7f148ca00640 1 -- start start 2024-03-04T09:25:57.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.059+0000 7f148ca00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f146c005210 0x7f148809c300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:57.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.059+0000 7f148ca00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f14880c8fe0 0x7f14880c9460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:57.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.059+0000 7f148ca00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f14880c99a0 0x7f148815cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:57.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.059+0000 7f148ca00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14880cf3a0 con 0x7f14880c8fe0 2024-03-04T09:25:57.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.059+0000 7f148ca00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1488071170 con 0x7f14880c99a0 2024-03-04T09:25:57.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.059+0000 7f148ca00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14880cf520 con 0x7f146c005210 2024-03-04T09:25:57.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.060+0000 7f1485400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f14880c8fe0 0x7f14880c9460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:57.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.060+0000 7f1485400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f14880c8fe0 0x7f14880c9460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:52070/0 (socket says 172.21.15.6:52070) 2024-03-04T09:25:57.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.060+0000 7f1485400640 1 -- 172.21.15.6:0/3376790909 learned_addr learned my addr 172.21.15.6:0/3376790909 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:25:57.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.060+0000 7f1485e00640 1 --2- 172.21.15.6:0/3376790909 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f146c005210 0x7f148809c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:57.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.060+0000 7f1486800640 1 --2- 172.21.15.6:0/3376790909 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f14880c99a0 0x7f148815cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:57.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.060+0000 7f1485400640 1 -- 172.21.15.6:0/3376790909 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f146c005210 msgr2=0x7f148809c300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:57.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.060+0000 7f1485400640 1 --2- 172.21.15.6:0/3376790909 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f146c005210 0x7f148809c300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:57.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.060+0000 7f1485400640 1 -- 172.21.15.6:0/3376790909 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f14880c99a0 msgr2=0x7f148815cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:57.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.060+0000 7f1485400640 1 --2- 172.21.15.6:0/3376790909 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f14880c99a0 0x7f148815cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:57.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.060+0000 7f1485400640 1 -- 172.21.15.6:0/3376790909 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14880cbbc0 con 0x7f14880c8fe0 2024-03-04T09:25:57.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.061+0000 7f1485400640 1 --2- 172.21.15.6:0/3376790909 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f14880c8fe0 0x7f14880c9460 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f147400d7d0 tx=0x7f147400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:57.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.061+0000 7f147e000640 1 -- 172.21.15.6:0/3376790909 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1474019070 con 0x7f14880c8fe0 2024-03-04T09:25:57.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.061+0000 7f147e000640 1 -- 172.21.15.6:0/3376790909 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14740074e0 con 0x7f14880c8fe0 2024-03-04T09:25:57.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.061+0000 7f147e000640 1 -- 172.21.15.6:0/3376790909 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14740049e0 con 0x7f14880c8fe0 2024-03-04T09:25:57.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.061+0000 7f148ca00640 1 -- 172.21.15.6:0/3376790909 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f148809b820 con 0x7f14880c8fe0 2024-03-04T09:25:57.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.061+0000 7f148ca00640 1 -- 172.21.15.6:0/3376790909 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14880cb850 con 0x7f14880c8fe0 2024-03-04T09:25:57.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.062+0000 7f148ca00640 1 -- 172.21.15.6:0/3376790909 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f148804f330 con 0x7f14880c8fe0 2024-03-04T09:25:57.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.063+0000 7f147e000640 1 -- 172.21.15.6:0/3376790909 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1474004b80 con 0x7f14880c8fe0 2024-03-04T09:25:57.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.064+0000 7f147e000640 1 --2- 172.21.15.6:0/3376790909 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f146806b4d0 0x7f146806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:57.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.064+0000 7f147e000640 1 -- 172.21.15.6:0/3376790909 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1474089c00 con 0x7f14880c8fe0 2024-03-04T09:25:57.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.065+0000 7f1485e00640 1 --2- 172.21.15.6:0/3376790909 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f146806b4d0 0x7f146806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:57.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.066+0000 7f1485e00640 1 --2- 172.21.15.6:0/3376790909 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f146806b4d0 0x7f146806d990 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f1478009990 tx=0x7f1478008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:57.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.069+0000 7f147e000640 1 -- 172.21.15.6:0/3376790909 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f14740589e0 con 0x7f14880c8fe0 2024-03-04T09:25:57.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.265+0000 7f148ca00640 1 -- 172.21.15.6:0/3376790909 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f14880b47c0 con 0x7f14880c8fe0 2024-03-04T09:25:57.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.265+0000 7f147e000640 1 -- 172.21.15.6:0/3376790909 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f14740582e0 con 0x7f14880c8fe0 2024-03-04T09:25:57.262 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 64 2024-03-04T09:25:57.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.268+0000 7f148ca00640 1 -- 172.21.15.6:0/3376790909 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f146806b4d0 msgr2=0x7f146806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:57.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.268+0000 7f148ca00640 1 --2- 172.21.15.6:0/3376790909 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f146806b4d0 0x7f146806d990 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f1478009990 tx=0x7f1478008040 comp rx=0 tx=0).stop 2024-03-04T09:25:57.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.268+0000 7f148ca00640 1 -- 172.21.15.6:0/3376790909 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f14880c8fe0 msgr2=0x7f14880c9460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:57.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.268+0000 7f148ca00640 1 --2- 172.21.15.6:0/3376790909 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f14880c8fe0 0x7f14880c9460 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f147400d7d0 tx=0x7f147400dc90 comp rx=0 tx=0).stop 2024-03-04T09:25:57.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.268+0000 7f148ca00640 1 -- 172.21.15.6:0/3376790909 shutdown_connections 2024-03-04T09:25:57.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.268+0000 7f148ca00640 1 --2- 172.21.15.6:0/3376790909 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f14880c99a0 0x7f148815cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:57.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.268+0000 7f148ca00640 1 --2- 172.21.15.6:0/3376790909 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f14880c8fe0 0x7f14880c9460 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:57.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.268+0000 7f148ca00640 1 --2- 172.21.15.6:0/3376790909 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f146806b4d0 0x7f146806d990 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:57.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.268+0000 7f148ca00640 1 --2- 172.21.15.6:0/3376790909 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f146c005210 0x7f148809c300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:57.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.268+0000 7f148ca00640 1 -- 172.21.15.6:0/3376790909 >> 172.21.15.6:0/3376790909 conn(0x7f148806b3d0 msgr2=0x7f14880b39a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:57.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.268+0000 7f148ca00640 1 -- 172.21.15.6:0/3376790909 shutdown_connections 2024-03-04T09:25:57.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.269+0000 7f148ca00640 1 -- 172.21.15.6:0/3376790909 wait complete. 2024-03-04T09:25:57.283 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T09:25:57.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.498+0000 7fed2e400640 1 Processor -- start 2024-03-04T09:25:57.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.498+0000 7fed2e400640 1 -- start start 2024-03-04T09:25:57.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.499+0000 7fed2e400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fed280af1a0 0x7fed2815c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:57.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.499+0000 7fed2e400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fed280af580 0x7fed280b4c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:57.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.499+0000 7fed2e400640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed2815cc40 con 0x7fed280cdc20 2024-03-04T09:25:57.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.500+0000 7fed2e400640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed2815cf20 con 0x7fed280af580 2024-03-04T09:25:57.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.500+0000 7fed2e400640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed2815d200 con 0x7fed280af1a0 2024-03-04T09:25:57.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.500+0000 7fed26a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fed280af580 0x7fed280b4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:57.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.500+0000 7fed26a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fed280af580 0x7fed280b4c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:52084/0 (socket says 172.21.15.6:52084) 2024-03-04T09:25:57.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.500+0000 7fed26a00640 1 -- 172.21.15.6:0/3451053977 learned_addr learned my addr 172.21.15.6:0/3451053977 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:25:57.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.500+0000 7fed27400640 1 --2- 172.21.15.6:0/3451053977 >> v2:172.21.15.146:3300/0 conn(0x7fed280af1a0 0x7fed2815c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:57.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.500+0000 7fed26a00640 1 -- 172.21.15.6:0/3451053977 >> v2:172.21.15.146:3300/0 conn(0x7fed280af1a0 msgr2=0x7fed2815c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:57.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.500+0000 7fed26a00640 1 --2- 172.21.15.6:0/3451053977 >> v2:172.21.15.146:3300/0 conn(0x7fed280af1a0 0x7fed2815c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:57.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.500+0000 7fed26a00640 1 -- 172.21.15.6:0/3451053977 >> v1:172.21.15.146:6789/0 conn(0x7fed280cdc20 legacy=0x7fed280b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:57.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.500+0000 7fed26a00640 1 -- 172.21.15.6:0/3451053977 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed2815d4e0 con 0x7fed280af580 2024-03-04T09:25:57.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.500+0000 7fed27400640 1 --2- 172.21.15.6:0/3451053977 >> v2:172.21.15.146:3300/0 conn(0x7fed280af1a0 0x7fed2815c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:25:57.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.501+0000 7fed26a00640 1 --2- 172.21.15.6:0/3451053977 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fed280af580 0x7fed280b4c20 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fed28074960 tx=0x7fed1402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65c4a4790b05758 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:57.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.502+0000 7fed26000640 1 -- 172.21.15.6:0/3451053977 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed1402e070 con 0x7fed280af580 2024-03-04T09:25:57.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.502+0000 7fed26000640 1 -- 172.21.15.6:0/3451053977 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed14037070 con 0x7fed280af580 2024-03-04T09:25:57.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.502+0000 7fed26000640 1 -- 172.21.15.6:0/3451053977 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed14031a00 con 0x7fed280af580 2024-03-04T09:25:57.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.503+0000 7fed2e400640 1 -- 172.21.15.6:0/3451053977 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fed280af580 msgr2=0x7fed280b4c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:57.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.503+0000 7fed2e400640 1 --2- 172.21.15.6:0/3451053977 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fed280af580 0x7fed280b4c20 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fed28074960 tx=0x7fed1402b880 comp rx=0 tx=0).stop 2024-03-04T09:25:57.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.503+0000 7fed2e400640 1 -- 172.21.15.6:0/3451053977 shutdown_connections 2024-03-04T09:25:57.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.503+0000 7fed2e400640 1 --2- 172.21.15.6:0/3451053977 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fed280af580 0x7fed280b4c20 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:57.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.503+0000 7fed2e400640 1 --2- 172.21.15.6:0/3451053977 >> v2:172.21.15.146:3300/0 conn(0x7fed280af1a0 0x7fed2815c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:57.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.503+0000 7fed2e400640 1 -- 172.21.15.6:0/3451053977 >> 172.21.15.6:0/3451053977 conn(0x7fed2806afb0 msgr2=0x7fed2806b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:57.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.503+0000 7fed2e400640 1 -- 172.21.15.6:0/3451053977 shutdown_connections 2024-03-04T09:25:57.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.503+0000 7fed2e400640 1 -- 172.21.15.6:0/3451053977 wait complete. 2024-03-04T09:25:57.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.504+0000 7fed2e400640 1 Processor -- start 2024-03-04T09:25:57.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.505+0000 7fed2e400640 1 -- start start 2024-03-04T09:25:57.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.506+0000 7fed2e400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fed280af1a0 0x7fed281511a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:57.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.506+0000 7fed2e400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fed280af580 0x7fed2815e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:57.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.506+0000 7fed2e400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fed280cdc20 0x7fed2815ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:57.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.506+0000 7fed2e400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed280b3660 con 0x7fed280af1a0 2024-03-04T09:25:57.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.506+0000 7fed2e400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed280b34e0 con 0x7fed280af580 2024-03-04T09:25:57.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.506+0000 7fed2e400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed280b37e0 con 0x7fed280cdc20 2024-03-04T09:25:57.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.506+0000 7fed27400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fed280af1a0 0x7fed281511a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:57.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.506+0000 7fed27400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fed280af1a0 0x7fed281511a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:52094/0 (socket says 172.21.15.6:52094) 2024-03-04T09:25:57.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.506+0000 7fed27400640 1 -- 172.21.15.6:0/388940440 learned_addr learned my addr 172.21.15.6:0/388940440 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:25:57.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.506+0000 7fed27e00640 1 --2- 172.21.15.6:0/388940440 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fed280cdc20 0x7fed2815ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:57.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.506+0000 7fed26a00640 1 --2- 172.21.15.6:0/388940440 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fed280af580 0x7fed2815e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:57.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.506+0000 7fed27400640 1 -- 172.21.15.6:0/388940440 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fed280cdc20 msgr2=0x7fed2815ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:57.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.506+0000 7fed27400640 1 --2- 172.21.15.6:0/388940440 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fed280cdc20 0x7fed2815ee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:57.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.506+0000 7fed27400640 1 -- 172.21.15.6:0/388940440 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fed280af580 msgr2=0x7fed2815e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:57.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.506+0000 7fed27400640 1 --2- 172.21.15.6:0/388940440 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fed280af580 0x7fed2815e940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:57.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.506+0000 7fed27400640 1 -- 172.21.15.6:0/388940440 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed28151cd0 con 0x7fed280af1a0 2024-03-04T09:25:57.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.507+0000 7fed27400640 1 --2- 172.21.15.6:0/388940440 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fed280af1a0 0x7fed281511a0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fed1800c820 tx=0x7fed1800cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:57.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.507+0000 7fed1f400640 1 -- 172.21.15.6:0/388940440 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed18015070 con 0x7fed280af1a0 2024-03-04T09:25:57.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.507+0000 7fed1f400640 1 -- 172.21.15.6:0/388940440 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed18007a30 con 0x7fed280af1a0 2024-03-04T09:25:57.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.507+0000 7fed1f400640 1 -- 172.21.15.6:0/388940440 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed18007e00 con 0x7fed280af1a0 2024-03-04T09:25:57.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.507+0000 7fed2e400640 1 -- 172.21.15.6:0/388940440 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed28155650 con 0x7fed280af1a0 2024-03-04T09:25:57.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.507+0000 7fed2e400640 1 -- 172.21.15.6:0/388940440 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed28155c10 con 0x7fed280af1a0 2024-03-04T09:25:57.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.508+0000 7fed2e400640 1 -- 172.21.15.6:0/388940440 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed28069510 con 0x7fed280af1a0 2024-03-04T09:25:57.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.509+0000 7fed1f400640 1 -- 172.21.15.6:0/388940440 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fed18018070 con 0x7fed280af1a0 2024-03-04T09:25:57.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.510+0000 7fed1f400640 1 --2- 172.21.15.6:0/388940440 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fed0806b4d0 0x7fed0806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:57.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.510+0000 7fed1f400640 1 -- 172.21.15.6:0/388940440 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fed1808a1f0 con 0x7fed280af1a0 2024-03-04T09:25:57.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.511+0000 7fed26a00640 1 --2- 172.21.15.6:0/388940440 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fed0806b4d0 0x7fed0806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:57.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.512+0000 7fed26a00640 1 --2- 172.21.15.6:0/388940440 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fed0806b4d0 0x7fed0806d990 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fed14036000 tx=0x7fed14008030 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:57.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.515+0000 7fed1f400640 1 -- 172.21.15.6:0/388940440 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fed18059060 con 0x7fed280af1a0 2024-03-04T09:25:57.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.714+0000 7fed2e400640 1 -- 172.21.15.6:0/388940440 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fed280b2e30 con 0x7fed280af1a0 2024-03-04T09:25:57.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.718+0000 7fed1f400640 1 -- 172.21.15.6:0/388940440 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16557 (secure 0 0 0) 0x7fed18058960 con 0x7fed280af1a0 2024-03-04T09:25:57.715 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:25:57.715 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":20,"fsid":"edce0578-5ae4-4402-b45a-552bdef84d7c","created":"2024-03-04T09:24:54.523780+0000","modified":"2024-03-04T09:25:20.446333+0000","last_up_change":"2024-03-04T09:25:07.089049+0000","last_in_change":"2024-03-04T09:24:57.982077+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T09:25:07.935198+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-04T09:25:17.118573+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-04T09:25:17.930536+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a96574f-81f1-4bc6-8463-757520b84a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6825","nonce":1364449401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6827","nonce":1364449401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6831","nonce":1364449401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6829","nonce":1364449401}]},"public_addr":"172.21.15.6:6825/1364449401","cluster_addr":"172.21.15.6:6827/1364449401","heartbeat_back_addr":"172.21.15.6:6831/1364449401","heartbeat_front_addr":"172.21.15.6:6829/1364449401","state":["exists","up"]},{"osd":1,"uuid":"7d79b9c1-6b87-476a-bc36-7db5ee64cd90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6809","nonce":2494103078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6811","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6813","nonce":2494103078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6821","nonce":2494103078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6817","nonce":2494103078}]},"public_addr":"172.21.15.6:6809/2494103078","cluster_addr":"172.21.15.6:6813/2494103078","heartbeat_back_addr":"172.21.15.6:6821/2494103078","heartbeat_front_addr":"172.21.15.6:6817/2494103078","state":["exists","up"]},{"osd":2,"uuid":"492331b9-4ee9-40b6-bacd-85d44cd6c45a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6812","nonce":1500066681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6816","nonce":1500066681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6824","nonce":1500066681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6820","nonce":1500066681}]},"public_addr":"172.21.15.6:6812/1500066681","cluster_addr":"172.21.15.6:6816/1500066681","heartbeat_back_addr":"172.21.15.6:6824/1500066681","heartbeat_front_addr":"172.21.15.6:6820/1500066681","state":["exists","up"]},{"osd":3,"uuid":"21bb2efe-9ba7-4bdc-bc31-180cd1102e21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6801","nonce":1558806123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6803","nonce":1558806123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6807","nonce":1558806123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6805","nonce":1558806123}]},"public_addr":"172.21.15.6:6801/1558806123","cluster_addr":"172.21.15.6:6803/1558806123","heartbeat_back_addr":"172.21.15.6:6807/1558806123","heartbeat_front_addr":"172.21.15.6:6805/1558806123","state":["exists","up"]},{"osd":4,"uuid":"6e2c4b71-19d9-4737-83d9-558705d8644a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6801","nonce":1526135780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6803","nonce":1526135780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6807","nonce":1526135780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6805","nonce":1526135780}]},"public_addr":"172.21.15.146:6801/1526135780","cluster_addr":"172.21.15.146:6803/1526135780","heartbeat_back_addr":"172.21.15.146:6807/1526135780","heartbeat_front_addr":"172.21.15.146:6805/1526135780","state":["exists","up"]},{"osd":5,"uuid":"fb69c525-f1be-4b7d-ab32-6c29b0d6fddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6809","nonce":460262507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6811","nonce":460262507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6815","nonce":460262507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6813","nonce":460262507}]},"public_addr":"172.21.15.146:6809/460262507","cluster_addr":"172.21.15.146:6811/460262507","heartbeat_back_addr":"172.21.15.146:6815/460262507","heartbeat_front_addr":"172.21.15.146:6813/460262507","state":["exists","up"]},{"osd":6,"uuid":"c32e7fed-370d-4401-8ce0-6a4335b0d168","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6823","nonce":1392897493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6827","nonce":1392897493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6831","nonce":1392897493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6829","nonce":1392897493}]},"public_addr":"172.21.15.146:6823/1392897493","cluster_addr":"172.21.15.146:6827/1392897493","heartbeat_back_addr":"172.21.15.146:6831/1392897493","heartbeat_front_addr":"172.21.15.146:6829/1392897493","state":["exists","up"]},{"osd":7,"uuid":"75a79b44-fa5f-426d-a7d3-71f21f78835c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6817","nonce":3268323156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6819","nonce":3268323156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6825","nonce":3268323156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6822","nonce":3268323156}]},"public_addr":"172.21.15.146:6817/3268323156","cluster_addr":"172.21.15.146:6819/3268323156","heartbeat_back_addr":"172.21.15.146:6825/3268323156","heartbeat_front_addr":"172.21.15.146:6822/3268323156","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.310437+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.295850+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.294674+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.526855+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.300023+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.595160+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.346046+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.558741+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T09:25:57.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.720+0000 7fed2e400640 1 -- 172.21.15.6:0/388940440 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fed0806b4d0 msgr2=0x7fed0806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:57.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.720+0000 7fed2e400640 1 --2- 172.21.15.6:0/388940440 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fed0806b4d0 0x7fed0806d990 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fed14036000 tx=0x7fed14008030 comp rx=0 tx=0).stop 2024-03-04T09:25:57.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.720+0000 7fed2e400640 1 -- 172.21.15.6:0/388940440 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fed280af1a0 msgr2=0x7fed281511a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:57.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.720+0000 7fed2e400640 1 --2- 172.21.15.6:0/388940440 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fed280af1a0 0x7fed281511a0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fed1800c820 tx=0x7fed1800cc20 comp rx=0 tx=0).stop 2024-03-04T09:25:57.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.720+0000 7fed2e400640 1 -- 172.21.15.6:0/388940440 shutdown_connections 2024-03-04T09:25:57.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.720+0000 7fed2e400640 1 --2- 172.21.15.6:0/388940440 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fed0806b4d0 0x7fed0806d990 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:57.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.720+0000 7fed2e400640 1 --2- 172.21.15.6:0/388940440 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fed280cdc20 0x7fed2815ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:57.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.720+0000 7fed2e400640 1 --2- 172.21.15.6:0/388940440 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fed280af580 0x7fed2815e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:57.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.720+0000 7fed2e400640 1 --2- 172.21.15.6:0/388940440 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fed280af1a0 0x7fed281511a0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:57.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.720+0000 7fed2e400640 1 -- 172.21.15.6:0/388940440 >> 172.21.15.6:0/388940440 conn(0x7fed2806afb0 msgr2=0x7fed280ccdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:57.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.721+0000 7fed2e400640 1 -- 172.21.15.6:0/388940440 shutdown_connections 2024-03-04T09:25:57.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.721+0000 7fed2e400640 1 -- 172.21.15.6:0/388940440 wait complete. 2024-03-04T09:25:57.738 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T09:25:07.935198+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-04T09:25:17.118573+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-04T09:25:17.930536+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-04T09:25:57.738 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-04T09:25:57.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.961+0000 7ff719e00640 1 Processor -- start 2024-03-04T09:25:57.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.961+0000 7ff719e00640 1 -- start start 2024-03-04T09:25:57.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.963+0000 7ff719e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ff71415bf20 0x7ff7140acd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:57.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.963+0000 7ff719e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff714072df0 con 0x7ff714071e20 2024-03-04T09:25:57.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.963+0000 7ff719e00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7140cdab0 con 0x7ff714072520 2024-03-04T09:25:57.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.963+0000 7ff719e00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7140b4d70 con 0x7ff71415bf20 2024-03-04T09:25:57.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.963+0000 7ff713400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7ff714071e20 0x7ff7140ccad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:58730/0 (socket says 172.21.15.6:58730) 2024-03-04T09:25:57.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.963+0000 7ff713400640 1 -- 172.21.15.6:0/3230915178 learned_addr learned my addr 172.21.15.6:0/3230915178 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:25:57.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.964+0000 7ff712a00640 1 --2- 172.21.15.6:0/3230915178 >> v2:172.21.15.146:3300/0 conn(0x7ff71415bf20 0x7ff7140acd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:57.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.964+0000 7ff711600640 1 -- 172.21.15.6:0/3230915178 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 615288037 0 0) 0x7ff714072df0 con 0x7ff714071e20 2024-03-04T09:25:57.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.964+0000 7ff711600640 1 -- 172.21.15.6:0/3230915178 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6f8003520 con 0x7ff714071e20 2024-03-04T09:25:57.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.964+0000 7ff711600640 1 -- 172.21.15.6:0/3230915178 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 415554388 0 0) 0x7ff6f8003520 con 0x7ff714071e20 2024-03-04T09:25:57.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.964+0000 7ff711600640 1 -- 172.21.15.6:0/3230915178 >> v2:172.21.15.146:3300/0 conn(0x7ff71415bf20 msgr2=0x7ff7140acd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:57.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.964+0000 7ff711600640 1 --2- 172.21.15.6:0/3230915178 >> v2:172.21.15.146:3300/0 conn(0x7ff71415bf20 0x7ff7140acd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:57.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.964+0000 7ff711600640 1 -- 172.21.15.6:0/3230915178 >> v1:172.21.15.146:6789/0 conn(0x7ff714072520 legacy=0x7ff7140cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:57.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.964+0000 7ff711600640 1 -- 172.21.15.6:0/3230915178 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7140b5050 con 0x7ff714071e20 2024-03-04T09:25:57.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.964+0000 7ff711600640 1 -- 172.21.15.6:0/3230915178 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7ff70c0019f0 con 0x7ff714071e20 2024-03-04T09:25:57.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.964+0000 7ff711600640 1 -- 172.21.15.6:0/3230915178 >> v1:172.21.15.6:6789/0 conn(0x7ff714071e20 legacy=0x7ff7140ccad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:57.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.964+0000 7ff711600640 1 --2- 172.21.15.6:0/3230915178 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff6f8003a00 0x7ff6f8003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:57.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.964+0000 7ff711600640 1 --2- 172.21.15.6:0/3230915178 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff6f8004800 0x7ff6f8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:57.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.964+0000 7ff711600640 1 --2- 172.21.15.6:0/3230915178 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff6f80051c0 0x7ff6f800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:57.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.964+0000 7ff711600640 1 -- 172.21.15.6:0/3230915178 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7140b4d70 con 0x7ff6f8004800 2024-03-04T09:25:57.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.964+0000 7ff711600640 1 -- 172.21.15.6:0/3230915178 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f800c0c0 con 0x7ff6f8003a00 2024-03-04T09:25:57.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.964+0000 7ff711600640 1 -- 172.21.15.6:0/3230915178 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f800c2c0 con 0x7ff6f80051c0 2024-03-04T09:25:57.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.965+0000 7ff712000640 1 --2- 172.21.15.6:0/3230915178 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff6f8004800 0x7ff6f8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:57.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.965+0000 7ff712a00640 1 --2- 172.21.15.6:0/3230915178 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff6f8003a00 0x7ff6f8003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:57.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.965+0000 7ff713400640 1 --2- 172.21.15.6:0/3230915178 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff6f80051c0 0x7ff6f800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:57.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.965+0000 7ff712000640 1 -- 172.21.15.6:0/3230915178 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff6f80051c0 msgr2=0x7ff6f800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:57.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.965+0000 7ff712000640 1 --2- 172.21.15.6:0/3230915178 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff6f80051c0 0x7ff6f800b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:57.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.965+0000 7ff712000640 1 -- 172.21.15.6:0/3230915178 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff6f8003a00 msgr2=0x7ff6f8003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:57.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.965+0000 7ff712000640 1 --2- 172.21.15.6:0/3230915178 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff6f8003a00 0x7ff6f8003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:57.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.965+0000 7ff712000640 1 -- 172.21.15.6:0/3230915178 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6f8003520 con 0x7ff6f8004800 2024-03-04T09:25:57.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.965+0000 7ff713400640 1 -- 172.21.15.6:0/3230915178 reap_dead start 2024-03-04T09:25:57.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.966+0000 7ff712000640 1 --2- 172.21.15.6:0/3230915178 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff6f8004800 0x7ff6f8004c80 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7ff700001750 tx=0x7ff6f8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b02563480c9f4caf server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:57.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.966+0000 7ff711600640 1 -- 172.21.15.6:0/3230915178 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff700001370 con 0x7ff6f8004800 2024-03-04T09:25:57.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.966+0000 7ff711600640 1 -- 172.21.15.6:0/3230915178 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7000052d0 con 0x7ff6f8004800 2024-03-04T09:25:57.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.966+0000 7ff711600640 1 -- 172.21.15.6:0/3230915178 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff700003830 con 0x7ff6f8004800 2024-03-04T09:25:57.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.967+0000 7ff719e00640 1 -- 172.21.15.6:0/3230915178 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff6f8004800 msgr2=0x7ff6f8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:57.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.967+0000 7ff719e00640 1 --2- 172.21.15.6:0/3230915178 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff6f8004800 0x7ff6f8004c80 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7ff700001750 tx=0x7ff6f8005830 comp rx=0 tx=0).stop 2024-03-04T09:25:57.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.967+0000 7ff719e00640 1 -- 172.21.15.6:0/3230915178 shutdown_connections 2024-03-04T09:25:57.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.967+0000 7ff719e00640 1 --2- 172.21.15.6:0/3230915178 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff6f8004800 0x7ff6f8004c80 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:57.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.967+0000 7ff719e00640 1 -- 172.21.15.6:0/3230915178 >> 172.21.15.6:0/3230915178 conn(0x7ff71406b3d0 msgr2=0x7ff71406b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:57.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.967+0000 7ff719e00640 1 -- 172.21.15.6:0/3230915178 shutdown_connections 2024-03-04T09:25:57.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.967+0000 7ff719e00640 1 -- 172.21.15.6:0/3230915178 wait complete. 2024-03-04T09:25:57.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.969+0000 7ff719e00640 1 Processor -- start 2024-03-04T09:25:57.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.969+0000 7ff719e00640 1 -- start start 2024-03-04T09:25:57.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.970+0000 7ff719e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff6f8004800 0x7ff7140b1b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:57.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.970+0000 7ff719e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff7140c9f10 0x7ff71415d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:57.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.970+0000 7ff719e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff71415d560 0x7ff7140b0730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:57.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.970+0000 7ff719e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7140b4b10 con 0x7ff6f8004800 2024-03-04T09:25:57.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.970+0000 7ff719e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7140b4990 con 0x7ff7140c9f10 2024-03-04T09:25:57.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.970+0000 7ff719e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7140acd40 con 0x7ff71415d560 2024-03-04T09:25:57.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.971+0000 7ff712a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff6f8004800 0x7ff7140b1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:57.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.971+0000 7ff712a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff6f8004800 0x7ff7140b1b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:52112/0 (socket says 172.21.15.6:52112) 2024-03-04T09:25:57.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.971+0000 7ff712a00640 1 -- 172.21.15.6:0/4004839861 learned_addr learned my addr 172.21.15.6:0/4004839861 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:25:57.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.971+0000 7ff713400640 1 --2- 172.21.15.6:0/4004839861 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff71415d560 0x7ff7140b0730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:57.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.971+0000 7ff712000640 1 --2- 172.21.15.6:0/4004839861 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff7140c9f10 0x7ff71415d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:57.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.971+0000 7ff712a00640 1 -- 172.21.15.6:0/4004839861 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff71415d560 msgr2=0x7ff7140b0730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:57.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.971+0000 7ff712a00640 1 --2- 172.21.15.6:0/4004839861 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff71415d560 0x7ff7140b0730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:57.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.971+0000 7ff712a00640 1 -- 172.21.15.6:0/4004839861 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff7140c9f10 msgr2=0x7ff71415d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:57.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.971+0000 7ff712a00640 1 --2- 172.21.15.6:0/4004839861 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff7140c9f10 0x7ff71415d020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:57.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.971+0000 7ff712a00640 1 -- 172.21.15.6:0/4004839861 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7140af970 con 0x7ff6f8004800 2024-03-04T09:25:57.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.972+0000 7ff712a00640 1 --2- 172.21.15.6:0/4004839861 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff6f8004800 0x7ff7140b1b40 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7ff704011510 tx=0x7ff7040119d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:57.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.972+0000 7ff70b400640 1 -- 172.21.15.6:0/4004839861 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff704015870 con 0x7ff6f8004800 2024-03-04T09:25:57.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.972+0000 7ff70b400640 1 -- 172.21.15.6:0/4004839861 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff704015a10 con 0x7ff6f8004800 2024-03-04T09:25:57.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.972+0000 7ff70b400640 1 -- 172.21.15.6:0/4004839861 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff704015dc0 con 0x7ff6f8004800 2024-03-04T09:25:57.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.972+0000 7ff719e00640 1 -- 172.21.15.6:0/4004839861 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7140afc50 con 0x7ff6f8004800 2024-03-04T09:25:57.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.972+0000 7ff719e00640 1 -- 172.21.15.6:0/4004839861 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7140b0210 con 0x7ff6f8004800 2024-03-04T09:25:57.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.973+0000 7ff719e00640 1 -- 172.21.15.6:0/4004839861 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff71406ede0 con 0x7ff6f8004800 2024-03-04T09:25:57.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.974+0000 7ff70b400640 1 -- 172.21.15.6:0/4004839861 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff704013400 con 0x7ff6f8004800 2024-03-04T09:25:57.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.975+0000 7ff70b400640 1 --2- 172.21.15.6:0/4004839861 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff6f406b4d0 0x7ff6f406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:57.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.975+0000 7ff70b400640 1 -- 172.21.15.6:0/4004839861 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff70408aa60 con 0x7ff6f8004800 2024-03-04T09:25:57.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.976+0000 7ff712000640 1 --2- 172.21.15.6:0/4004839861 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff6f406b4d0 0x7ff6f406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:57.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.977+0000 7ff712000640 1 --2- 172.21.15.6:0/4004839861 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff6f406b4d0 0x7ff6f406d990 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7ff700033000 tx=0x7ff700009030 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:57.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:57.979+0000 7ff70b400640 1 -- 172.21.15.6:0/4004839861 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff71406ede0 con 0x7ff6f8004800 2024-03-04T09:25:58.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.178+0000 7ff719e00640 1 -- 172.21.15.6:0/4004839861 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff71404f330 con 0x7ff6f8004800 2024-03-04T09:25:58.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.179+0000 7ff70b400640 1 -- 172.21.15.6:0/4004839861 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff704019030 con 0x7ff6f8004800 2024-03-04T09:25:58.176 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-04T09:25:58.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.181+0000 7ff719e00640 1 -- 172.21.15.6:0/4004839861 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff6f406b4d0 msgr2=0x7ff6f406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:58.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.181+0000 7ff719e00640 1 --2- 172.21.15.6:0/4004839861 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff6f406b4d0 0x7ff6f406d990 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7ff700033000 tx=0x7ff700009030 comp rx=0 tx=0).stop 2024-03-04T09:25:58.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.181+0000 7ff719e00640 1 -- 172.21.15.6:0/4004839861 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff6f8004800 msgr2=0x7ff7140b1b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:58.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.181+0000 7ff719e00640 1 --2- 172.21.15.6:0/4004839861 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff6f8004800 0x7ff7140b1b40 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7ff704011510 tx=0x7ff7040119d0 comp rx=0 tx=0).stop 2024-03-04T09:25:58.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.181+0000 7ff719e00640 1 -- 172.21.15.6:0/4004839861 shutdown_connections 2024-03-04T09:25:58.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.181+0000 7ff719e00640 1 --2- 172.21.15.6:0/4004839861 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff6f406b4d0 0x7ff6f406d990 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:58.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.181+0000 7ff719e00640 1 --2- 172.21.15.6:0/4004839861 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff71415d560 0x7ff7140b0730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:58.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.181+0000 7ff719e00640 1 --2- 172.21.15.6:0/4004839861 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff7140c9f10 0x7ff71415d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:58.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.181+0000 7ff719e00640 1 --2- 172.21.15.6:0/4004839861 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff6f8004800 0x7ff7140b1b40 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:58.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.181+0000 7ff719e00640 1 -- 172.21.15.6:0/4004839861 >> 172.21.15.6:0/4004839861 conn(0x7ff71406b3d0 msgr2=0x7ff7140ccaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:58.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.182+0000 7ff719e00640 1 -- 172.21.15.6:0/4004839861 shutdown_connections 2024-03-04T09:25:58.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.182+0000 7ff719e00640 1 -- 172.21.15.6:0/4004839861 wait complete. 2024-03-04T09:25:58.199 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-04T09:25:58.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.420+0000 7f38c2a00640 1 Processor -- start 2024-03-04T09:25:58.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.420+0000 7f38c2a00640 1 -- start start 2024-03-04T09:25:58.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.421+0000 7f38c2a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f38bc15afa0 0x7f38bc0cf3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:58.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.421+0000 7f38c2a00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f38bc070ce0 0x7f38bc0ccad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:58.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.422+0000 7f38c2a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38bc071c60 con 0x7f38bc0cd010 2024-03-04T09:25:58.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.422+0000 7f38c2a00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38bc0cd480 con 0x7f38bc15afa0 2024-03-04T09:25:58.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.422+0000 7f38c2a00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38bc0cfa10 con 0x7f38bc070ce0 2024-03-04T09:25:58.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.422+0000 7f38bbe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f38bc15afa0 0x7f38bc0cf3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:58.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.422+0000 7f38c0a00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f38bc0cd010 0x7f38bc06c980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:58744/0 (socket says 172.21.15.6:58744) 2024-03-04T09:25:58.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.422+0000 7f38bbe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f38bc15afa0 0x7f38bc0cf3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:52126/0 (socket says 172.21.15.6:52126) 2024-03-04T09:25:58.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.422+0000 7f38c0a00640 1 -- 172.21.15.6:0/2479044542 learned_addr learned my addr 172.21.15.6:0/2479044542 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:25:58.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.422+0000 7f38b3e00640 1 --2- 172.21.15.6:0/2479044542 >> v2:172.21.15.146:3300/0 conn(0x7f38bc070ce0 0x7f38bc0ccad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:58.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.422+0000 7f38bbe00640 1 -- 172.21.15.6:0/2479044542 >> v2:172.21.15.146:3300/0 conn(0x7f38bc070ce0 msgr2=0x7f38bc0ccad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:58.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.422+0000 7f38bbe00640 1 --2- 172.21.15.6:0/2479044542 >> v2:172.21.15.146:3300/0 conn(0x7f38bc070ce0 0x7f38bc0ccad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:58.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.422+0000 7f38bbe00640 1 -- 172.21.15.6:0/2479044542 >> v1:172.21.15.6:6789/0 conn(0x7f38bc0cd010 legacy=0x7f38bc06c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:58.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.422+0000 7f38bbe00640 1 -- 172.21.15.6:0/2479044542 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38bc071f40 con 0x7f38bc15afa0 2024-03-04T09:25:58.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.424+0000 7f38bbe00640 1 --2- 172.21.15.6:0/2479044542 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38bc15afa0 0x7f38bc0cf3a0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f38bc0b3650 tx=0x7f38ac02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68e391d8456b6445 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:58.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.424+0000 7f38bb400640 1 -- 172.21.15.6:0/2479044542 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38ac0315a0 con 0x7f38bc15afa0 2024-03-04T09:25:58.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.424+0000 7f38bb400640 1 -- 172.21.15.6:0/2479044542 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38ac037070 con 0x7f38bc15afa0 2024-03-04T09:25:58.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.424+0000 7f38bb400640 1 -- 172.21.15.6:0/2479044542 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38ac031960 con 0x7f38bc15afa0 2024-03-04T09:25:58.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.424+0000 7f38c2a00640 1 -- 172.21.15.6:0/2479044542 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38bc15afa0 msgr2=0x7f38bc0cf3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:58.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.424+0000 7f38c2a00640 1 --2- 172.21.15.6:0/2479044542 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38bc15afa0 0x7f38bc0cf3a0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f38bc0b3650 tx=0x7f38ac02b840 comp rx=0 tx=0).stop 2024-03-04T09:25:58.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.425+0000 7f38c2a00640 1 -- 172.21.15.6:0/2479044542 shutdown_connections 2024-03-04T09:25:58.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.425+0000 7f38c2a00640 1 --2- 172.21.15.6:0/2479044542 >> v2:172.21.15.146:3300/0 conn(0x7f38bc070ce0 0x7f38bc0ccad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:58.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.425+0000 7f38c2a00640 1 --2- 172.21.15.6:0/2479044542 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38bc15afa0 0x7f38bc0cf3a0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:58.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.425+0000 7f38c2a00640 1 -- 172.21.15.6:0/2479044542 >> 172.21.15.6:0/2479044542 conn(0x7f38bc06b3d0 msgr2=0x7f38bc06b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:58.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.425+0000 7f38c2a00640 1 -- 172.21.15.6:0/2479044542 shutdown_connections 2024-03-04T09:25:58.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.425+0000 7f38c2a00640 1 -- 172.21.15.6:0/2479044542 wait complete. 2024-03-04T09:25:58.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.426+0000 7f38c2a00640 1 Processor -- start 2024-03-04T09:25:58.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.427+0000 7f38c2a00640 1 -- start start 2024-03-04T09:25:58.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.428+0000 7f38c2a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f38bc070ce0 0x7f38bc1e4400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:58.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.428+0000 7f38c2a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f38bc0cd010 0x7f38bc156860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:58.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.428+0000 7f38c2a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38bc15afa0 0x7f38bc1e51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:58.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.428+0000 7f38c2a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38bc071170 con 0x7f38bc15afa0 2024-03-04T09:25:58.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.428+0000 7f38c2a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38bc072500 con 0x7f38bc0cd010 2024-03-04T09:25:58.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.428+0000 7f38c2a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38bc0b4ad0 con 0x7f38bc070ce0 2024-03-04T09:25:58.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.428+0000 7f38c0a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38bc15afa0 0x7f38bc1e51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:58.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.428+0000 7f38c0a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38bc15afa0 0x7f38bc1e51f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:52138/0 (socket says 172.21.15.6:52138) 2024-03-04T09:25:58.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.428+0000 7f38c0a00640 1 -- 172.21.15.6:0/3445661846 learned_addr learned my addr 172.21.15.6:0/3445661846 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:25:58.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.428+0000 7f38b3e00640 1 --2- 172.21.15.6:0/3445661846 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f38bc0cd010 0x7f38bc156860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:58.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.428+0000 7f38bbe00640 1 --2- 172.21.15.6:0/3445661846 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f38bc070ce0 0x7f38bc1e4400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:58.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.428+0000 7f38c0a00640 1 -- 172.21.15.6:0/3445661846 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f38bc070ce0 msgr2=0x7f38bc1e4400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:58.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.429+0000 7f38c0a00640 1 --2- 172.21.15.6:0/3445661846 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f38bc070ce0 0x7f38bc1e4400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:58.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.429+0000 7f38c0a00640 1 -- 172.21.15.6:0/3445661846 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f38bc0cd010 msgr2=0x7f38bc156860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:58.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.429+0000 7f38c0a00640 1 --2- 172.21.15.6:0/3445661846 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f38bc0cd010 0x7f38bc156860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:58.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.429+0000 7f38c0a00640 1 -- 172.21.15.6:0/3445661846 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38bc1e5d60 con 0x7f38bc15afa0 2024-03-04T09:25:58.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.429+0000 7f38b3e00640 1 --2- 172.21.15.6:0/3445661846 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f38bc0cd010 0x7f38bc156860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T09:25:58.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.429+0000 7f38c0a00640 1 --2- 172.21.15.6:0/3445661846 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38bc15afa0 0x7f38bc1e51f0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f38b400cbc0 tx=0x7f38b400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:58.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.429+0000 7f38b8c00640 1 -- 172.21.15.6:0/3445661846 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38b4017070 con 0x7f38bc15afa0 2024-03-04T09:25:58.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.429+0000 7f38b8c00640 1 -- 172.21.15.6:0/3445661846 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38b400ad40 con 0x7f38bc15afa0 2024-03-04T09:25:58.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.429+0000 7f38b8c00640 1 -- 172.21.15.6:0/3445661846 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38b4012600 con 0x7f38bc15afa0 2024-03-04T09:25:58.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.429+0000 7f38c2a00640 1 -- 172.21.15.6:0/3445661846 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38bc1e3640 con 0x7f38bc15afa0 2024-03-04T09:25:58.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.429+0000 7f38c2a00640 1 -- 172.21.15.6:0/3445661846 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38bc1e3ee0 con 0x7f38bc15afa0 2024-03-04T09:25:58.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.431+0000 7f38c2a00640 1 -- 172.21.15.6:0/3445661846 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38bc1e3810 con 0x7f38bc15afa0 2024-03-04T09:25:58.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.432+0000 7f38b8c00640 1 -- 172.21.15.6:0/3445661846 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f38b401a070 con 0x7f38bc15afa0 2024-03-04T09:25:58.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.433+0000 7f38b8c00640 1 --2- 172.21.15.6:0/3445661846 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f389806b4d0 0x7f389806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:58.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.433+0000 7f38b8c00640 1 -- 172.21.15.6:0/3445661846 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f38b4089f50 con 0x7f38bc15afa0 2024-03-04T09:25:58.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.433+0000 7f38bbe00640 1 --2- 172.21.15.6:0/3445661846 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f389806b4d0 0x7f389806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:58.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.435+0000 7f38bbe00640 1 --2- 172.21.15.6:0/3445661846 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f389806b4d0 0x7f389806d990 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f38ac036000 tx=0x7f38ac03d040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:58.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.437+0000 7f38b8c00640 1 -- 172.21.15.6:0/3445661846 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f38b40586e0 con 0x7f38bc15afa0 2024-03-04T09:25:58.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.641+0000 7f38c2a00640 1 -- 172.21.15.6:0/3445661846 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f38bc04f330 con 0x7f38bc15afa0 2024-03-04T09:25:58.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.642+0000 7f38b8c00640 1 -- 172.21.15.6:0/3445661846 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7f38b405c780 con 0x7f38bc15afa0 2024-03-04T09:25:58.639 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-03-04T09:25:58.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.645+0000 7f38c2a00640 1 -- 172.21.15.6:0/3445661846 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f389806b4d0 msgr2=0x7f389806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:58.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.645+0000 7f38c2a00640 1 --2- 172.21.15.6:0/3445661846 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f389806b4d0 0x7f389806d990 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f38ac036000 tx=0x7f38ac03d040 comp rx=0 tx=0).stop 2024-03-04T09:25:58.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.645+0000 7f38c2a00640 1 -- 172.21.15.6:0/3445661846 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38bc15afa0 msgr2=0x7f38bc1e51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:58.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.645+0000 7f38c2a00640 1 --2- 172.21.15.6:0/3445661846 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38bc15afa0 0x7f38bc1e51f0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f38b400cbc0 tx=0x7f38b400a430 comp rx=0 tx=0).stop 2024-03-04T09:25:58.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.645+0000 7f38c2a00640 1 -- 172.21.15.6:0/3445661846 shutdown_connections 2024-03-04T09:25:58.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.645+0000 7f38c2a00640 1 --2- 172.21.15.6:0/3445661846 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38bc15afa0 0x7f38bc1e51f0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:58.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.645+0000 7f38c2a00640 1 --2- 172.21.15.6:0/3445661846 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f38bc0cd010 0x7f38bc156860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:58.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.645+0000 7f38c2a00640 1 --2- 172.21.15.6:0/3445661846 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f389806b4d0 0x7f389806d990 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:58.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.645+0000 7f38c2a00640 1 --2- 172.21.15.6:0/3445661846 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f38bc070ce0 0x7f38bc1e4400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:58.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.645+0000 7f38c2a00640 1 -- 172.21.15.6:0/3445661846 >> 172.21.15.6:0/3445661846 conn(0x7f38bc06b3d0 msgr2=0x7f38bc0b1a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:58.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.645+0000 7f38c2a00640 1 -- 172.21.15.6:0/3445661846 shutdown_connections 2024-03-04T09:25:58.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.645+0000 7f38c2a00640 1 -- 172.21.15.6:0/3445661846 wait complete. 2024-03-04T09:25:58.660 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-04T09:25:58.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.862+0000 7f0571800640 1 Processor -- start 2024-03-04T09:25:58.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.863+0000 7f0571800640 1 -- start start 2024-03-04T09:25:58.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.864+0000 7f0571800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f056c1615b0 0x7f056c0d47f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:58.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.864+0000 7f0571800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f056c0d1a80 0x7f056c0d22d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:58.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.864+0000 7f0571800640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f056c0cf670 con 0x7f056c0d2810 2024-03-04T09:25:58.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.864+0000 7f0571800640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f056c161990 con 0x7f056c1615b0 2024-03-04T09:25:58.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.864+0000 7f0571800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f056c0d1ff0 con 0x7f056c0d1a80 2024-03-04T09:25:58.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.865+0000 7f056aa00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f056c1615b0 0x7f056c0d47f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:58.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.865+0000 7f056b400640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f056c0d2810 0x7f056c0d06f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.6:57512/0 (socket says 172.21.15.6:57512) 2024-03-04T09:25:58.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.865+0000 7f056b400640 1 -- 172.21.15.6:0/2797228408 learned_addr learned my addr 172.21.15.6:0/2797228408 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:25:58.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.865+0000 7f0563e00640 1 --2- 172.21.15.6:0/2797228408 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f056c0d1a80 0x7f056c0d22d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:58.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.866+0000 7f056aa00640 1 -- 172.21.15.6:0/2797228408 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f056c0d1a80 msgr2=0x7f056c0d22d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:58.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.866+0000 7f056aa00640 1 --2- 172.21.15.6:0/2797228408 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f056c0d1a80 0x7f056c0d22d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:58.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.866+0000 7f056aa00640 1 -- 172.21.15.6:0/2797228408 >> v1:172.21.15.146:6789/0 conn(0x7f056c0d2810 legacy=0x7f056c0d06f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:58.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.866+0000 7f056aa00640 1 -- 172.21.15.6:0/2797228408 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f056c0cf910 con 0x7f056c1615b0 2024-03-04T09:25:58.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.867+0000 7f056aa00640 1 --2- 172.21.15.6:0/2797228408 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f056c1615b0 0x7f056c0d47f0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f055c0097a0 tx=0x7f055c02b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b432c1fa0ff2ccf server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:58.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.867+0000 7f056a000640 1 -- 172.21.15.6:0/2797228408 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f055c035020 con 0x7f056c1615b0 2024-03-04T09:25:58.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.868+0000 7f056a000640 1 -- 172.21.15.6:0/2797228408 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f055c031810 con 0x7f056c1615b0 2024-03-04T09:25:58.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.868+0000 7f056a000640 1 -- 172.21.15.6:0/2797228408 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f055c031be0 con 0x7f056c1615b0 2024-03-04T09:25:58.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.868+0000 7f0571800640 1 -- 172.21.15.6:0/2797228408 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f056c1615b0 msgr2=0x7f056c0d47f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:58.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.868+0000 7f0571800640 1 --2- 172.21.15.6:0/2797228408 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f056c1615b0 0x7f056c0d47f0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f055c0097a0 tx=0x7f055c02b8b0 comp rx=0 tx=0).stop 2024-03-04T09:25:58.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.868+0000 7f0571800640 1 -- 172.21.15.6:0/2797228408 shutdown_connections 2024-03-04T09:25:58.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.868+0000 7f0571800640 1 --2- 172.21.15.6:0/2797228408 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f056c0d1a80 0x7f056c0d22d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:58.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.868+0000 7f0571800640 1 --2- 172.21.15.6:0/2797228408 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f056c1615b0 0x7f056c0d47f0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:58.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.868+0000 7f0571800640 1 -- 172.21.15.6:0/2797228408 >> 172.21.15.6:0/2797228408 conn(0x7f056c09d180 msgr2=0x7f056c0d3eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:58.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.869+0000 7f0571800640 1 -- 172.21.15.6:0/2797228408 shutdown_connections 2024-03-04T09:25:58.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.869+0000 7f0571800640 1 -- 172.21.15.6:0/2797228408 wait complete. 2024-03-04T09:25:58.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.870+0000 7f0571800640 1 Processor -- start 2024-03-04T09:25:58.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.871+0000 7f0571800640 1 -- start start 2024-03-04T09:25:58.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.872+0000 7f0571800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f056c0d1a80 0x7f056c154fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:58.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.872+0000 7f0571800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f056c0d2810 0x7f056c0c97a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:58.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.872+0000 7f0571800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f056c1615b0 0x7f056c0ca110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:58.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.872+0000 7f0571800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f056c0d2ca0 con 0x7f056c1615b0 2024-03-04T09:25:58.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.872+0000 7f0571800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f056c0d4d30 con 0x7f056c0d1a80 2024-03-04T09:25:58.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.872+0000 7f0571800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f056c0d2e20 con 0x7f056c0d2810 2024-03-04T09:25:58.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.872+0000 7f056b400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f056c1615b0 0x7f056c0ca110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:58.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.872+0000 7f056b400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f056c1615b0 0x7f056c0ca110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:46546/0 (socket says 172.21.15.6:46546) 2024-03-04T09:25:58.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.872+0000 7f056b400640 1 -- 172.21.15.6:0/409530730 learned_addr learned my addr 172.21.15.6:0/409530730 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:25:58.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.872+0000 7f056aa00640 1 --2- 172.21.15.6:0/409530730 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f056c0d1a80 0x7f056c154fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:58.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.872+0000 7f0563e00640 1 --2- 172.21.15.6:0/409530730 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f056c0d2810 0x7f056c0c97a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:58.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.873+0000 7f056b400640 1 -- 172.21.15.6:0/409530730 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f056c0d2810 msgr2=0x7f056c0c97a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:58.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.873+0000 7f056b400640 1 --2- 172.21.15.6:0/409530730 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f056c0d2810 0x7f056c0c97a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:58.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.873+0000 7f056b400640 1 -- 172.21.15.6:0/409530730 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f056c0d1a80 msgr2=0x7f056c154fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:58.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.873+0000 7f056b400640 1 --2- 172.21.15.6:0/409530730 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f056c0d1a80 0x7f056c154fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:58.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.873+0000 7f056b400640 1 -- 172.21.15.6:0/409530730 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f056c0c87d0 con 0x7f056c1615b0 2024-03-04T09:25:58.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.873+0000 7f056aa00640 1 --2- 172.21.15.6:0/409530730 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f056c0d1a80 0x7f056c154fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T09:25:58.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.873+0000 7f056b400640 1 --2- 172.21.15.6:0/409530730 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f056c1615b0 0x7f056c0ca110 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f056400cbc0 tx=0x7f056400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:58.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.873+0000 7f0562a00640 1 -- 172.21.15.6:0/409530730 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0564017070 con 0x7f056c1615b0 2024-03-04T09:25:58.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.873+0000 7f0562a00640 1 -- 172.21.15.6:0/409530730 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f056400ad40 con 0x7f056c1615b0 2024-03-04T09:25:58.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.873+0000 7f0562a00640 1 -- 172.21.15.6:0/409530730 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0564012600 con 0x7f056c1615b0 2024-03-04T09:25:58.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.873+0000 7f0571800640 1 -- 172.21.15.6:0/409530730 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f056c0c8ab0 con 0x7f056c1615b0 2024-03-04T09:25:58.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.874+0000 7f0571800640 1 -- 172.21.15.6:0/409530730 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f056c1554e0 con 0x7f056c1615b0 2024-03-04T09:25:58.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.875+0000 7f0571800640 1 -- 172.21.15.6:0/409530730 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f056c0cc210 con 0x7f056c1615b0 2024-03-04T09:25:58.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.876+0000 7f0562a00640 1 -- 172.21.15.6:0/409530730 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f056401a070 con 0x7f056c1615b0 2024-03-04T09:25:58.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.877+0000 7f0562a00640 1 --2- 172.21.15.6:0/409530730 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f054806b4d0 0x7f054806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:58.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.877+0000 7f0562a00640 1 -- 172.21.15.6:0/409530730 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0564089e30 con 0x7f056c1615b0 2024-03-04T09:25:58.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.877+0000 7f056aa00640 1 --2- 172.21.15.6:0/409530730 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f054806b4d0 0x7f054806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:58.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.879+0000 7f056aa00640 1 --2- 172.21.15.6:0/409530730 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f054806b4d0 0x7f054806d990 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f055c036000 tx=0x7f055c03c040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:58.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:58.882+0000 7f0562a00640 1 -- 172.21.15.6:0/409530730 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f056c0cc210 con 0x7f056c1615b0 2024-03-04T09:25:59.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.116+0000 7f0571800640 1 -- 172.21.15.6:0/409530730 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f056c04f330 con 0x7f056c1615b0 2024-03-04T09:25:59.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.117+0000 7f0562a00640 1 -- 172.21.15.6:0/409530730 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f0564058650 con 0x7f056c1615b0 2024-03-04T09:25:59.114 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 64 2024-03-04T09:25:59.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.120+0000 7f0571800640 1 -- 172.21.15.6:0/409530730 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f054806b4d0 msgr2=0x7f054806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:59.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.120+0000 7f0571800640 1 --2- 172.21.15.6:0/409530730 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f054806b4d0 0x7f054806d990 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f055c036000 tx=0x7f055c03c040 comp rx=0 tx=0).stop 2024-03-04T09:25:59.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.120+0000 7f0571800640 1 -- 172.21.15.6:0/409530730 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f056c1615b0 msgr2=0x7f056c0ca110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:59.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.120+0000 7f0571800640 1 --2- 172.21.15.6:0/409530730 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f056c1615b0 0x7f056c0ca110 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f056400cbc0 tx=0x7f056400a430 comp rx=0 tx=0).stop 2024-03-04T09:25:59.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.121+0000 7f0571800640 1 -- 172.21.15.6:0/409530730 shutdown_connections 2024-03-04T09:25:59.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.121+0000 7f0571800640 1 --2- 172.21.15.6:0/409530730 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f056c1615b0 0x7f056c0ca110 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:59.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.121+0000 7f0571800640 1 --2- 172.21.15.6:0/409530730 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f054806b4d0 0x7f054806d990 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:59.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.121+0000 7f0571800640 1 --2- 172.21.15.6:0/409530730 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f056c0d2810 0x7f056c0c97a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:59.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.121+0000 7f0571800640 1 --2- 172.21.15.6:0/409530730 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f056c0d1a80 0x7f056c154fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:59.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.121+0000 7f0571800640 1 -- 172.21.15.6:0/409530730 >> 172.21.15.6:0/409530730 conn(0x7f056c09d180 msgr2=0x7f056c0d04e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:59.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.121+0000 7f0571800640 1 -- 172.21.15.6:0/409530730 shutdown_connections 2024-03-04T09:25:59.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.121+0000 7f0571800640 1 -- 172.21.15.6:0/409530730 wait complete. 2024-03-04T09:25:59.177 INFO:tasks.cephfs_test_runner:Starting test: test_dir_deletion (tasks.cephfs.test_strays.TestStrays) That when deleting a bunch of dentries and the containing 2024-03-04T09:25:59.177 INFO:tasks.cephfs_test_runner:test_dir_deletion (tasks.cephfs.test_strays.TestStrays) 2024-03-04T09:25:59.178 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T09:25:59.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.379+0000 7f8fed800640 1 Processor -- start 2024-03-04T09:25:59.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.379+0000 7f8fed800640 1 -- start start 2024-03-04T09:25:59.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.380+0000 7f8fed800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f8fe80cede0 0x7f8fe80b3f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:59.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.380+0000 7f8fed800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f8fe8071b60 0x7f8fe815b320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:59.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.380+0000 7f8fed800640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fe80b31d0 con 0x7f8fe80b4a70 2024-03-04T09:25:59.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.380+0000 7f8fed800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fe80b4ed0 con 0x7f8fe8071b60 2024-03-04T09:25:59.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.380+0000 7f8fed800640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fe80b3b00 con 0x7f8fe80cede0 2024-03-04T09:25:59.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.381+0000 7f8fece00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f8fe8071b60 0x7f8fe815b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:59.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.381+0000 7f8fe7400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f8fe80b4a70 0x7f8fe8071450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:56472/0 (socket says 172.21.15.6:56472) 2024-03-04T09:25:59.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.381+0000 7f8fe7400640 1 -- 172.21.15.6:0/11922017 learned_addr learned my addr 172.21.15.6:0/11922017 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:25:59.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.381+0000 7f8fe7e00640 1 --2- 172.21.15.6:0/11922017 >> v2:172.21.15.146:3300/0 conn(0x7f8fe80cede0 0x7f8fe80b3f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:59.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.381+0000 7f8fece00640 1 -- 172.21.15.6:0/11922017 >> v2:172.21.15.146:3300/0 conn(0x7f8fe80cede0 msgr2=0x7f8fe80b3f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:59.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.381+0000 7f8fece00640 1 --2- 172.21.15.6:0/11922017 >> v2:172.21.15.146:3300/0 conn(0x7f8fe80cede0 0x7f8fe80b3f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:59.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.381+0000 7f8fece00640 1 -- 172.21.15.6:0/11922017 >> v1:172.21.15.6:6789/0 conn(0x7f8fe80b4a70 legacy=0x7f8fe8071450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:59.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.381+0000 7f8fece00640 1 -- 172.21.15.6:0/11922017 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fe80b34b0 con 0x7f8fe8071b60 2024-03-04T09:25:59.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.382+0000 7f8fece00640 1 --2- 172.21.15.6:0/11922017 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8fe8071b60 0x7f8fe815b320 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f8fe8069240 tx=0x7f8fe002e050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6eacac0cd9f61c87 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:59.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.383+0000 7f8fe6a00640 1 -- 172.21.15.6:0/11922017 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fe0034070 con 0x7f8fe8071b60 2024-03-04T09:25:59.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.383+0000 7f8fe6a00640 1 -- 172.21.15.6:0/11922017 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fe002ee00 con 0x7f8fe8071b60 2024-03-04T09:25:59.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.383+0000 7f8fe6a00640 1 -- 172.21.15.6:0/11922017 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fe0038620 con 0x7f8fe8071b60 2024-03-04T09:25:59.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.383+0000 7f8fed800640 1 -- 172.21.15.6:0/11922017 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8fe8071b60 msgr2=0x7f8fe815b320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:59.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.383+0000 7f8fed800640 1 --2- 172.21.15.6:0/11922017 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8fe8071b60 0x7f8fe815b320 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f8fe8069240 tx=0x7f8fe002e050 comp rx=0 tx=0).stop 2024-03-04T09:25:59.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.383+0000 7f8fed800640 1 -- 172.21.15.6:0/11922017 shutdown_connections 2024-03-04T09:25:59.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.383+0000 7f8fed800640 1 --2- 172.21.15.6:0/11922017 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8fe8071b60 0x7f8fe815b320 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:59.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.383+0000 7f8fed800640 1 --2- 172.21.15.6:0/11922017 >> v2:172.21.15.146:3300/0 conn(0x7f8fe80cede0 0x7f8fe80b3f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:59.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.383+0000 7f8fed800640 1 -- 172.21.15.6:0/11922017 >> 172.21.15.6:0/11922017 conn(0x7f8fe806b3d0 msgr2=0x7f8fe806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:59.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.383+0000 7f8fed800640 1 -- 172.21.15.6:0/11922017 shutdown_connections 2024-03-04T09:25:59.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.384+0000 7f8fed800640 1 -- 172.21.15.6:0/11922017 wait complete. 2024-03-04T09:25:59.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.385+0000 7f8fed800640 1 Processor -- start 2024-03-04T09:25:59.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.385+0000 7f8fed800640 1 -- start start 2024-03-04T09:25:59.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.386+0000 7f8fed800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8fe8071b60 0x7f8fe81f8340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:59.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.386+0000 7f8fed800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8fe80b4a70 0x7f8fe8143450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:59.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.386+0000 7f8fed800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8fe80cede0 0x7f8fe814f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:59.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.386+0000 7f8fed800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fe80cf1c0 con 0x7f8fe80cede0 2024-03-04T09:25:59.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.386+0000 7f8fed800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fe80b2f70 con 0x7f8fe80b4a70 2024-03-04T09:25:59.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.386+0000 7f8fed800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fe80cc950 con 0x7f8fe8071b60 2024-03-04T09:25:59.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.387+0000 7f8fece00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8fe80cede0 0x7f8fe814f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:59.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.387+0000 7f8fece00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8fe80cede0 0x7f8fe814f1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:46562/0 (socket says 172.21.15.6:46562) 2024-03-04T09:25:59.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.387+0000 7f8fece00640 1 -- 172.21.15.6:0/2067742707 learned_addr learned my addr 172.21.15.6:0/2067742707 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:25:59.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.387+0000 7f8fe7e00640 1 --2- 172.21.15.6:0/2067742707 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8fe8071b60 0x7f8fe81f8340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:59.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.387+0000 7f8fe7400640 1 --2- 172.21.15.6:0/2067742707 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8fe80b4a70 0x7f8fe8143450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:59.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.387+0000 7f8fece00640 1 -- 172.21.15.6:0/2067742707 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8fe8071b60 msgr2=0x7f8fe81f8340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:59.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.387+0000 7f8fece00640 1 --2- 172.21.15.6:0/2067742707 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8fe8071b60 0x7f8fe81f8340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:59.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.387+0000 7f8fece00640 1 -- 172.21.15.6:0/2067742707 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8fe80b4a70 msgr2=0x7f8fe8143450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:59.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.387+0000 7f8fece00640 1 --2- 172.21.15.6:0/2067742707 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8fe80b4a70 0x7f8fe8143450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:59.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.387+0000 7f8fece00640 1 -- 172.21.15.6:0/2067742707 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fe81f8880 con 0x7f8fe80cede0 2024-03-04T09:25:59.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.387+0000 7f8fece00640 1 --2- 172.21.15.6:0/2067742707 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8fe80cede0 0x7f8fe814f1a0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f8fe000d140 tx=0x7f8fe0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:59.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.388+0000 7f8fdfe00640 1 -- 172.21.15.6:0/2067742707 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fe0046030 con 0x7f8fe80cede0 2024-03-04T09:25:59.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.388+0000 7f8fdfe00640 1 -- 172.21.15.6:0/2067742707 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fe002ec00 con 0x7f8fe80cede0 2024-03-04T09:25:59.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.388+0000 7f8fdfe00640 1 -- 172.21.15.6:0/2067742707 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fe0042400 con 0x7f8fe80cede0 2024-03-04T09:25:59.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.388+0000 7f8fed800640 1 -- 172.21.15.6:0/2067742707 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fe814e100 con 0x7f8fe80cede0 2024-03-04T09:25:59.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.388+0000 7f8fed800640 1 -- 172.21.15.6:0/2067742707 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fe814e6c0 con 0x7f8fe80cede0 2024-03-04T09:25:59.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.389+0000 7f8fed800640 1 -- 172.21.15.6:0/2067742707 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fe804f330 con 0x7f8fe80cede0 2024-03-04T09:25:59.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.390+0000 7f8fdfe00640 1 -- 172.21.15.6:0/2067742707 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8fe0049070 con 0x7f8fe80cede0 2024-03-04T09:25:59.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.391+0000 7f8fdfe00640 1 --2- 172.21.15.6:0/2067742707 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f8fc806b4d0 0x7f8fc806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:59.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.391+0000 7f8fdfe00640 1 -- 172.21.15.6:0/2067742707 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8fe00acc60 con 0x7f8fe80cede0 2024-03-04T09:25:59.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.391+0000 7f8fe7e00640 1 --2- 172.21.15.6:0/2067742707 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f8fc806b4d0 0x7f8fc806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:59.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.393+0000 7f8fe7e00640 1 --2- 172.21.15.6:0/2067742707 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f8fc806b4d0 0x7f8fc806d990 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f8fd8006fd0 tx=0x7f8fd8008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:59.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.395+0000 7f8fdfe00640 1 -- 172.21.15.6:0/2067742707 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8fe0050070 con 0x7f8fe80cede0 2024-03-04T09:25:59.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.597+0000 7f8fed800640 1 -- 172.21.15.6:0/2067742707 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8fe806cad0 con 0x7f8fe80cede0 2024-03-04T09:25:59.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.600+0000 7f8fdfe00640 1 -- 172.21.15.6:0/2067742707 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16557 (secure 0 0 0) 0x7f8fe81f8880 con 0x7f8fe80cede0 2024-03-04T09:25:59.597 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:25:59.597 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":20,"fsid":"edce0578-5ae4-4402-b45a-552bdef84d7c","created":"2024-03-04T09:24:54.523780+0000","modified":"2024-03-04T09:25:20.446333+0000","last_up_change":"2024-03-04T09:25:07.089049+0000","last_in_change":"2024-03-04T09:24:57.982077+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T09:25:07.935198+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-04T09:25:17.118573+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-04T09:25:17.930536+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a96574f-81f1-4bc6-8463-757520b84a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6825","nonce":1364449401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6827","nonce":1364449401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6831","nonce":1364449401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6829","nonce":1364449401}]},"public_addr":"172.21.15.6:6825/1364449401","cluster_addr":"172.21.15.6:6827/1364449401","heartbeat_back_addr":"172.21.15.6:6831/1364449401","heartbeat_front_addr":"172.21.15.6:6829/1364449401","state":["exists","up"]},{"osd":1,"uuid":"7d79b9c1-6b87-476a-bc36-7db5ee64cd90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6809","nonce":2494103078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6811","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6813","nonce":2494103078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6821","nonce":2494103078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6817","nonce":2494103078}]},"public_addr":"172.21.15.6:6809/2494103078","cluster_addr":"172.21.15.6:6813/2494103078","heartbeat_back_addr":"172.21.15.6:6821/2494103078","heartbeat_front_addr":"172.21.15.6:6817/2494103078","state":["exists","up"]},{"osd":2,"uuid":"492331b9-4ee9-40b6-bacd-85d44cd6c45a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6812","nonce":1500066681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6816","nonce":1500066681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6824","nonce":1500066681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6820","nonce":1500066681}]},"public_addr":"172.21.15.6:6812/1500066681","cluster_addr":"172.21.15.6:6816/1500066681","heartbeat_back_addr":"172.21.15.6:6824/1500066681","heartbeat_front_addr":"172.21.15.6:6820/1500066681","state":["exists","up"]},{"osd":3,"uuid":"21bb2efe-9ba7-4bdc-bc31-180cd1102e21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6801","nonce":1558806123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6803","nonce":1558806123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6807","nonce":1558806123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6805","nonce":1558806123}]},"public_addr":"172.21.15.6:6801/1558806123","cluster_addr":"172.21.15.6:6803/1558806123","heartbeat_back_addr":"172.21.15.6:6807/1558806123","heartbeat_front_addr":"172.21.15.6:6805/1558806123","state":["exists","up"]},{"osd":4,"uuid":"6e2c4b71-19d9-4737-83d9-558705d8644a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6801","nonce":1526135780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6803","nonce":1526135780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6807","nonce":1526135780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6805","nonce":1526135780}]},"public_addr":"172.21.15.146:6801/1526135780","cluster_addr":"172.21.15.146:6803/1526135780","heartbeat_back_addr":"172.21.15.146:6807/1526135780","heartbeat_front_addr":"172.21.15.146:6805/1526135780","state":["exists","up"]},{"osd":5,"uuid":"fb69c525-f1be-4b7d-ab32-6c29b0d6fddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6809","nonce":460262507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6811","nonce":460262507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6815","nonce":460262507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6813","nonce":460262507}]},"public_addr":"172.21.15.146:6809/460262507","cluster_addr":"172.21.15.146:6811/460262507","heartbeat_back_addr":"172.21.15.146:6815/460262507","heartbeat_front_addr":"172.21.15.146:6813/460262507","state":["exists","up"]},{"osd":6,"uuid":"c32e7fed-370d-4401-8ce0-6a4335b0d168","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6823","nonce":1392897493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6827","nonce":1392897493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6831","nonce":1392897493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6829","nonce":1392897493}]},"public_addr":"172.21.15.146:6823/1392897493","cluster_addr":"172.21.15.146:6827/1392897493","heartbeat_back_addr":"172.21.15.146:6831/1392897493","heartbeat_front_addr":"172.21.15.146:6829/1392897493","state":["exists","up"]},{"osd":7,"uuid":"75a79b44-fa5f-426d-a7d3-71f21f78835c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6817","nonce":3268323156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6819","nonce":3268323156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6825","nonce":3268323156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6822","nonce":3268323156}]},"public_addr":"172.21.15.146:6817/3268323156","cluster_addr":"172.21.15.146:6819/3268323156","heartbeat_back_addr":"172.21.15.146:6825/3268323156","heartbeat_front_addr":"172.21.15.146:6822/3268323156","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.310437+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.295850+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.294674+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.526855+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.300023+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.595160+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.346046+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.558741+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T09:25:59.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.602+0000 7f8fed800640 1 -- 172.21.15.6:0/2067742707 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f8fc806b4d0 msgr2=0x7f8fc806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:59.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.602+0000 7f8fed800640 1 --2- 172.21.15.6:0/2067742707 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f8fc806b4d0 0x7f8fc806d990 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f8fd8006fd0 tx=0x7f8fd8008040 comp rx=0 tx=0).stop 2024-03-04T09:25:59.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.602+0000 7f8fed800640 1 -- 172.21.15.6:0/2067742707 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8fe80cede0 msgr2=0x7f8fe814f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:59.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.602+0000 7f8fed800640 1 --2- 172.21.15.6:0/2067742707 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8fe80cede0 0x7f8fe814f1a0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f8fe000d140 tx=0x7f8fe0004a40 comp rx=0 tx=0).stop 2024-03-04T09:25:59.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.602+0000 7f8fed800640 1 -- 172.21.15.6:0/2067742707 shutdown_connections 2024-03-04T09:25:59.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.602+0000 7f8fed800640 1 --2- 172.21.15.6:0/2067742707 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8fe80cede0 0x7f8fe814f1a0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:59.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.602+0000 7f8fed800640 1 --2- 172.21.15.6:0/2067742707 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8fe80b4a70 0x7f8fe8143450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:59.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.602+0000 7f8fed800640 1 --2- 172.21.15.6:0/2067742707 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f8fc806b4d0 0x7f8fc806d990 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:59.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.602+0000 7f8fed800640 1 --2- 172.21.15.6:0/2067742707 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8fe8071b60 0x7f8fe81f8340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:59.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.602+0000 7f8fed800640 1 -- 172.21.15.6:0/2067742707 >> 172.21.15.6:0/2067742707 conn(0x7f8fe806b3d0 msgr2=0x7f8fe8071240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:59.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.603+0000 7f8fed800640 1 -- 172.21.15.6:0/2067742707 shutdown_connections 2024-03-04T09:25:59.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.603+0000 7f8fed800640 1 -- 172.21.15.6:0/2067742707 wait complete. 2024-03-04T09:25:59.616 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T09:25:07.935198+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-04T09:25:17.118573+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-04T09:25:17.930536+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-04T09:25:59.616 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-04T09:25:59.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.822+0000 7f4f3b800640 1 Processor -- start 2024-03-04T09:25:59.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.822+0000 7f4f3b800640 1 -- start start 2024-03-04T09:25:59.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.823+0000 7f4f3b800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f4f340d0e50 0x7f4f3415fc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:59.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.823+0000 7f4f3b800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f4f340d2bb0 0x7f4f340cf190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:59.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.823+0000 7f4f3b800640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f340d4e20 con 0x7f4f340d49e0 2024-03-04T09:25:59.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.823+0000 7f4f3b800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f340ced00 con 0x7f4f340d2bb0 2024-03-04T09:25:59.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.823+0000 7f4f3b800640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f340d1fc0 con 0x7f4f340d0e50 2024-03-04T09:25:59.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.824+0000 7f4f39800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f4f340d2bb0 0x7f4f340cf190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:59.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.824+0000 7f4f33e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f4f340d49e0 0x7f4f340d24a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:56488/0 (socket says 172.21.15.6:56488) 2024-03-04T09:25:59.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.824+0000 7f4f33e00640 1 -- 172.21.15.6:0/190007295 learned_addr learned my addr 172.21.15.6:0/190007295 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:25:59.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.824+0000 7f4f38e00640 1 --2- 172.21.15.6:0/190007295 >> v2:172.21.15.146:3300/0 conn(0x7f4f340d0e50 0x7f4f3415fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:59.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.824+0000 7f4f39800640 1 -- 172.21.15.6:0/190007295 >> v2:172.21.15.146:3300/0 conn(0x7f4f340d0e50 msgr2=0x7f4f3415fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:25:59.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.824+0000 7f4f39800640 1 --2- 172.21.15.6:0/190007295 >> v2:172.21.15.146:3300/0 conn(0x7f4f340d0e50 0x7f4f3415fc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:59.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.824+0000 7f4f39800640 1 -- 172.21.15.6:0/190007295 >> v1:172.21.15.6:6789/0 conn(0x7f4f340d49e0 legacy=0x7f4f340d24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:59.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.824+0000 7f4f39800640 1 -- 172.21.15.6:0/190007295 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f340d1230 con 0x7f4f340d2bb0 2024-03-04T09:25:59.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.825+0000 7f4f39800640 1 --2- 172.21.15.6:0/190007295 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4f340d2bb0 0x7f4f340cf190 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f4f3408cf90 tx=0x7f4f2c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d47f755ca60edbc server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:59.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.826+0000 7f4f33400640 1 -- 172.21.15.6:0/190007295 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f2c039070 con 0x7f4f340d2bb0 2024-03-04T09:25:59.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.826+0000 7f4f33400640 1 -- 172.21.15.6:0/190007295 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f2c02cc80 con 0x7f4f340d2bb0 2024-03-04T09:25:59.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.826+0000 7f4f33400640 1 -- 172.21.15.6:0/190007295 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f2c034660 con 0x7f4f340d2bb0 2024-03-04T09:25:59.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.826+0000 7f4f3b800640 1 -- 172.21.15.6:0/190007295 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4f340d2bb0 msgr2=0x7f4f340cf190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:59.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.826+0000 7f4f3b800640 1 --2- 172.21.15.6:0/190007295 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4f340d2bb0 0x7f4f340cf190 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f4f3408cf90 tx=0x7f4f2c02bed0 comp rx=0 tx=0).stop 2024-03-04T09:25:59.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.826+0000 7f4f3b800640 1 -- 172.21.15.6:0/190007295 shutdown_connections 2024-03-04T09:25:59.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.826+0000 7f4f3b800640 1 --2- 172.21.15.6:0/190007295 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4f340d2bb0 0x7f4f340cf190 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:59.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.826+0000 7f4f3b800640 1 --2- 172.21.15.6:0/190007295 >> v2:172.21.15.146:3300/0 conn(0x7f4f340d0e50 0x7f4f3415fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:59.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.826+0000 7f4f3b800640 1 -- 172.21.15.6:0/190007295 >> 172.21.15.6:0/190007295 conn(0x7f4f340d3ec0 msgr2=0x7f4f340d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:25:59.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.826+0000 7f4f3b800640 1 -- 172.21.15.6:0/190007295 shutdown_connections 2024-03-04T09:25:59.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.826+0000 7f4f3b800640 1 -- 172.21.15.6:0/190007295 wait complete. 2024-03-04T09:25:59.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.829+0000 7f4f3b800640 1 Processor -- start 2024-03-04T09:25:59.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.829+0000 7f4f3b800640 1 -- start start 2024-03-04T09:25:59.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.831+0000 7f4f3b800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4f340d0e50 0x7f4f3415d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:59.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.831+0000 7f4f3b800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4f340d2bb0 0x7f4f340b90f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:59.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.831+0000 7f4f3b800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4f340d49e0 0x7f4f340b9630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:59.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.831+0000 7f4f3b800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f3415f490 con 0x7f4f340d2bb0 2024-03-04T09:25:59.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.831+0000 7f4f3b800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f3415f1b0 con 0x7f4f340d49e0 2024-03-04T09:25:59.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.831+0000 7f4f3b800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f3415f770 con 0x7f4f340d0e50 2024-03-04T09:25:59.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.831+0000 7f4f33e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4f340d2bb0 0x7f4f340b90f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:59.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.831+0000 7f4f33e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4f340d2bb0 0x7f4f340b90f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:46574/0 (socket says 172.21.15.6:46574) 2024-03-04T09:25:59.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.831+0000 7f4f33e00640 1 -- 172.21.15.6:0/3440862378 learned_addr learned my addr 172.21.15.6:0/3440862378 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:25:59.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.832+0000 7f4f39800640 1 --2- 172.21.15.6:0/3440862378 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4f340d49e0 0x7f4f340b9630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:59.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.832+0000 7f4f38e00640 1 --2- 172.21.15.6:0/3440862378 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4f340d0e50 0x7f4f3415d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:59.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.832+0000 7f4f33e00640 1 -- 172.21.15.6:0/3440862378 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4f340d0e50 msgr2=0x7f4f3415d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:59.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.832+0000 7f4f33e00640 1 --2- 172.21.15.6:0/3440862378 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4f340d0e50 0x7f4f3415d710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:59.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.832+0000 7f4f33e00640 1 -- 172.21.15.6:0/3440862378 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4f340d49e0 msgr2=0x7f4f340b9630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:25:59.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.832+0000 7f4f33e00640 1 --2- 172.21.15.6:0/3440862378 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4f340d49e0 0x7f4f340b9630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:25:59.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.832+0000 7f4f33e00640 1 -- 172.21.15.6:0/3440862378 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f340b8760 con 0x7f4f340d2bb0 2024-03-04T09:25:59.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.832+0000 7f4f39800640 1 --2- 172.21.15.6:0/3440862378 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4f340d49e0 0x7f4f340b9630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:25:59.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.832+0000 7f4f33e00640 1 --2- 172.21.15.6:0/3440862378 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4f340d2bb0 0x7f4f340b90f0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f4f2000c3c0 tx=0x7f4f2000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:59.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.832+0000 7f4f30c00640 1 -- 172.21.15.6:0/3440862378 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f20017070 con 0x7f4f340d2bb0 2024-03-04T09:25:59.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.832+0000 7f4f30c00640 1 -- 172.21.15.6:0/3440862378 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f20005ce0 con 0x7f4f340d2bb0 2024-03-04T09:25:59.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.832+0000 7f4f3b800640 1 -- 172.21.15.6:0/3440862378 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f3415dc50 con 0x7f4f340d2bb0 2024-03-04T09:25:59.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.832+0000 7f4f30c00640 1 -- 172.21.15.6:0/3440862378 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f200072f0 con 0x7f4f340d2bb0 2024-03-04T09:25:59.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.833+0000 7f4f3b800640 1 -- 172.21.15.6:0/3440862378 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f340b79a0 con 0x7f4f340d2bb0 2024-03-04T09:25:59.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.833+0000 7f4f3b800640 1 -- 172.21.15.6:0/3440862378 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f3404f330 con 0x7f4f340d2bb0 2024-03-04T09:25:59.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.835+0000 7f4f30c00640 1 -- 172.21.15.6:0/3440862378 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4f2001a070 con 0x7f4f340d2bb0 2024-03-04T09:25:59.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.835+0000 7f4f30c00640 1 --2- 172.21.15.6:0/3440862378 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f4f1406b4d0 0x7f4f1406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:25:59.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.836+0000 7f4f30c00640 1 -- 172.21.15.6:0/3440862378 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4f20089680 con 0x7f4f340d2bb0 2024-03-04T09:25:59.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.836+0000 7f4f38e00640 1 --2- 172.21.15.6:0/3440862378 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f4f1406b4d0 0x7f4f1406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:25:59.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.837+0000 7f4f38e00640 1 --2- 172.21.15.6:0/3440862378 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f4f1406b4d0 0x7f4f1406d990 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f4f28006fd0 tx=0x7f4f28008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:25:59.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:25:59.840+0000 7f4f30c00640 1 -- 172.21.15.6:0/3440862378 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4f20058460 con 0x7f4f340d2bb0 2024-03-04T09:26:00.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.034+0000 7f4f3b800640 1 -- 172.21.15.6:0/3440862378 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f4f340d2280 con 0x7f4f340d2bb0 2024-03-04T09:26:00.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.035+0000 7f4f30c00640 1 -- 172.21.15.6:0/3440862378 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4f20057d60 con 0x7f4f340d2bb0 2024-03-04T09:26:00.031 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-04T09:26:00.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.037+0000 7f4f3b800640 1 -- 172.21.15.6:0/3440862378 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f4f1406b4d0 msgr2=0x7f4f1406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:00.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.037+0000 7f4f3b800640 1 --2- 172.21.15.6:0/3440862378 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f4f1406b4d0 0x7f4f1406d990 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f4f28006fd0 tx=0x7f4f28008040 comp rx=0 tx=0).stop 2024-03-04T09:26:00.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.037+0000 7f4f3b800640 1 -- 172.21.15.6:0/3440862378 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4f340d2bb0 msgr2=0x7f4f340b90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:00.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.037+0000 7f4f3b800640 1 --2- 172.21.15.6:0/3440862378 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4f340d2bb0 0x7f4f340b90f0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f4f2000c3c0 tx=0x7f4f2000c880 comp rx=0 tx=0).stop 2024-03-04T09:26:00.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.037+0000 7f4f3b800640 1 -- 172.21.15.6:0/3440862378 shutdown_connections 2024-03-04T09:26:00.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.037+0000 7f4f3b800640 1 --2- 172.21.15.6:0/3440862378 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4f340d49e0 0x7f4f340b9630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:00.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.037+0000 7f4f3b800640 1 --2- 172.21.15.6:0/3440862378 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4f340d2bb0 0x7f4f340b90f0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:00.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.037+0000 7f4f3b800640 1 --2- 172.21.15.6:0/3440862378 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f4f1406b4d0 0x7f4f1406d990 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:00.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.037+0000 7f4f3b800640 1 --2- 172.21.15.6:0/3440862378 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4f340d0e50 0x7f4f3415d710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:00.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.037+0000 7f4f3b800640 1 -- 172.21.15.6:0/3440862378 >> 172.21.15.6:0/3440862378 conn(0x7f4f340d3ec0 msgr2=0x7f4f340cb000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:00.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.037+0000 7f4f3b800640 1 -- 172.21.15.6:0/3440862378 shutdown_connections 2024-03-04T09:26:00.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.038+0000 7f4f3b800640 1 -- 172.21.15.6:0/3440862378 wait complete. 2024-03-04T09:26:00.051 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-04T09:26:00.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.269+0000 7f63fe000640 1 Processor -- start 2024-03-04T09:26:00.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.270+0000 7f63fe000640 1 -- start start 2024-03-04T09:26:00.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.271+0000 7f63fe000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f63f8072520 0x7f63f80cdfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:00.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.271+0000 7f63fe000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f63f8071e20 0x7f63f80ce4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:00.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.271+0000 7f63fe000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63f8072df0 con 0x7f63f815bf20 2024-03-04T09:26:00.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.271+0000 7f63fe000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63f80cdab0 con 0x7f63f8072520 2024-03-04T09:26:00.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.271+0000 7f63fe000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63f80b4d70 con 0x7f63f8071e20 2024-03-04T09:26:00.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.272+0000 7f63f7e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f63f8071e20 0x7f63f80ce4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:00.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.272+0000 7f63f7e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f63f8071e20 0x7f63f80ce4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.6:44552/0 (socket says 172.21.15.6:44552) 2024-03-04T09:26:00.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.272+0000 7f63f7e00640 1 -- 172.21.15.6:0/1301482861 learned_addr learned my addr 172.21.15.6:0/1301482861 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:00.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.272+0000 7f63efe00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f63f8072520 0x7f63f80cdfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:00.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.273+0000 7f63f7e00640 1 -- 172.21.15.6:0/1301482861 >> v2:172.21.15.146:3300/0 conn(0x7f63f8072520 msgr2=0x7f63f80cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:00.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.273+0000 7f63f7e00640 1 --2- 172.21.15.6:0/1301482861 >> v2:172.21.15.146:3300/0 conn(0x7f63f8072520 0x7f63f80cdfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:00.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.273+0000 7f63f7e00640 1 -- 172.21.15.6:0/1301482861 >> v1:172.21.15.146:6789/0 conn(0x7f63f815bf20 legacy=0x7f63f80acd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:00.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.273+0000 7f63f7e00640 1 -- 172.21.15.6:0/1301482861 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63f80b5050 con 0x7f63f8071e20 2024-03-04T09:26:00.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.273+0000 7f63efe00640 1 --2- 172.21.15.6:0/1301482861 >> v2:172.21.15.146:3300/0 conn(0x7f63f8072520 0x7f63f80cdfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T09:26:00.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.274+0000 7f63f7e00640 1 --2- 172.21.15.6:0/1301482861 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f63f8071e20 0x7f63f80ce4f0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f63f0009e50 tx=0x7f63f002bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bdc0d2588564b28 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:00.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.275+0000 7f63f6a00640 1 -- 172.21.15.6:0/1301482861 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f002cb00 con 0x7f63f8071e20 2024-03-04T09:26:00.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.275+0000 7f63f6a00640 1 -- 172.21.15.6:0/1301482861 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f002cca0 con 0x7f63f8071e20 2024-03-04T09:26:00.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.275+0000 7f63f6a00640 1 -- 172.21.15.6:0/1301482861 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0034610 con 0x7f63f8071e20 2024-03-04T09:26:00.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.275+0000 7f63fe000640 1 -- 172.21.15.6:0/1301482861 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f63f8071e20 msgr2=0x7f63f80ce4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:00.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.275+0000 7f63fe000640 1 --2- 172.21.15.6:0/1301482861 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f63f8071e20 0x7f63f80ce4f0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f63f0009e50 tx=0x7f63f002bf60 comp rx=0 tx=0).stop 2024-03-04T09:26:00.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.275+0000 7f63fe000640 1 -- 172.21.15.6:0/1301482861 shutdown_connections 2024-03-04T09:26:00.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.275+0000 7f63fe000640 1 --2- 172.21.15.6:0/1301482861 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f63f8071e20 0x7f63f80ce4f0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:00.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.275+0000 7f63fe000640 1 --2- 172.21.15.6:0/1301482861 >> v2:172.21.15.146:3300/0 conn(0x7f63f8072520 0x7f63f80cdfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:00.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.275+0000 7f63fe000640 1 -- 172.21.15.6:0/1301482861 >> 172.21.15.6:0/1301482861 conn(0x7f63f806b3d0 msgr2=0x7f63f806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:00.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.276+0000 7f63fe000640 1 -- 172.21.15.6:0/1301482861 shutdown_connections 2024-03-04T09:26:00.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.276+0000 7f63fe000640 1 -- 172.21.15.6:0/1301482861 wait complete. 2024-03-04T09:26:00.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.277+0000 7f63fe000640 1 Processor -- start 2024-03-04T09:26:00.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.278+0000 7f63fe000640 1 -- start start 2024-03-04T09:26:00.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.278+0000 7f63fe000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f63f8071e20 0x7f63f815ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:00.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.278+0000 7f63fe000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f63f8072520 0x7f63f80b1ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:00.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.278+0000 7f63fe000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f63f815bf20 0x7f63f81f4400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:00.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.278+0000 7f63fe000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63f80ad450 con 0x7f63f8072520 2024-03-04T09:26:00.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.278+0000 7f63fe000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63f80b4990 con 0x7f63f815bf20 2024-03-04T09:26:00.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.278+0000 7f63fe000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63f80b0580 con 0x7f63f8071e20 2024-03-04T09:26:00.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.279+0000 7f63efe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f63f8072520 0x7f63f80b1ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:00.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.279+0000 7f63efe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f63f8072520 0x7f63f80b1ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:46580/0 (socket says 172.21.15.6:46580) 2024-03-04T09:26:00.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.279+0000 7f63efe00640 1 -- 172.21.15.6:0/1539253459 learned_addr learned my addr 172.21.15.6:0/1539253459 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:00.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.279+0000 7f63f7e00640 1 --2- 172.21.15.6:0/1539253459 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f63f815bf20 0x7f63f81f4400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:00.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.279+0000 7f63f7400640 1 --2- 172.21.15.6:0/1539253459 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f63f8071e20 0x7f63f815ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:00.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.279+0000 7f63efe00640 1 -- 172.21.15.6:0/1539253459 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f63f8071e20 msgr2=0x7f63f815ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:00.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.279+0000 7f63efe00640 1 --2- 172.21.15.6:0/1539253459 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f63f8071e20 0x7f63f815ec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:00.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.279+0000 7f63efe00640 1 -- 172.21.15.6:0/1539253459 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f63f815bf20 msgr2=0x7f63f81f4400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:00.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.279+0000 7f63efe00640 1 --2- 172.21.15.6:0/1539253459 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f63f815bf20 0x7f63f81f4400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:00.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.279+0000 7f63efe00640 1 -- 172.21.15.6:0/1539253459 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63f80af970 con 0x7f63f8072520 2024-03-04T09:26:00.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.279+0000 7f63f7e00640 1 --2- 172.21.15.6:0/1539253459 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f63f815bf20 0x7f63f81f4400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T09:26:00.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.280+0000 7f63efe00640 1 --2- 172.21.15.6:0/1539253459 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f63f8072520 0x7f63f80b1ef0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f63e000e960 tx=0x7f63e000ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:00.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.280+0000 7f63ef400640 1 -- 172.21.15.6:0/1539253459 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63e0017070 con 0x7f63f8072520 2024-03-04T09:26:00.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.280+0000 7f63ef400640 1 -- 172.21.15.6:0/1539253459 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63e000ca00 con 0x7f63f8072520 2024-03-04T09:26:00.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.280+0000 7f63ef400640 1 -- 172.21.15.6:0/1539253459 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63e000cdd0 con 0x7f63f8072520 2024-03-04T09:26:00.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.280+0000 7f63fe000640 1 -- 172.21.15.6:0/1539253459 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63f80afc50 con 0x7f63f8072520 2024-03-04T09:26:00.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.280+0000 7f63fe000640 1 -- 172.21.15.6:0/1539253459 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63f80b0210 con 0x7f63f8072520 2024-03-04T09:26:00.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.281+0000 7f63fe000640 1 -- 172.21.15.6:0/1539253459 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63f8073f70 con 0x7f63f8072520 2024-03-04T09:26:00.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.282+0000 7f63ef400640 1 -- 172.21.15.6:0/1539253459 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f63e001a070 con 0x7f63f8072520 2024-03-04T09:26:00.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.283+0000 7f63ef400640 1 --2- 172.21.15.6:0/1539253459 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f63d406b4d0 0x7f63d406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:00.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.283+0000 7f63ef400640 1 -- 172.21.15.6:0/1539253459 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f63e008a480 con 0x7f63f8072520 2024-03-04T09:26:00.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.284+0000 7f63f7400640 1 --2- 172.21.15.6:0/1539253459 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f63d406b4d0 0x7f63d406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:00.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.285+0000 7f63f7400640 1 --2- 172.21.15.6:0/1539253459 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f63d406b4d0 0x7f63d406d990 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f63e8001640 tx=0x7f63e8009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:00.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.287+0000 7f63ef400640 1 -- 172.21.15.6:0/1539253459 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f63f8073f70 con 0x7f63f8072520 2024-03-04T09:26:00.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.486+0000 7f63fe000640 1 -- 172.21.15.6:0/1539253459 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f63f804f330 con 0x7f63f8072520 2024-03-04T09:26:00.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.487+0000 7f63ef400640 1 -- 172.21.15.6:0/1539253459 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7f63e00592f0 con 0x7f63f8072520 2024-03-04T09:26:00.484 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-03-04T09:26:00.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.490+0000 7f63fe000640 1 -- 172.21.15.6:0/1539253459 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f63d406b4d0 msgr2=0x7f63d406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:00.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.490+0000 7f63fe000640 1 --2- 172.21.15.6:0/1539253459 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f63d406b4d0 0x7f63d406d990 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f63e8001640 tx=0x7f63e8009040 comp rx=0 tx=0).stop 2024-03-04T09:26:00.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.490+0000 7f63fe000640 1 -- 172.21.15.6:0/1539253459 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f63f8072520 msgr2=0x7f63f80b1ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:00.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.490+0000 7f63fe000640 1 --2- 172.21.15.6:0/1539253459 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f63f8072520 0x7f63f80b1ef0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f63e000e960 tx=0x7f63e000ee20 comp rx=0 tx=0).stop 2024-03-04T09:26:00.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.490+0000 7f63fe000640 1 -- 172.21.15.6:0/1539253459 shutdown_connections 2024-03-04T09:26:00.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.490+0000 7f63fe000640 1 --2- 172.21.15.6:0/1539253459 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f63f815bf20 0x7f63f81f4400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:00.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.490+0000 7f63fe000640 1 --2- 172.21.15.6:0/1539253459 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f63f8072520 0x7f63f80b1ef0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:00.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.490+0000 7f63fe000640 1 --2- 172.21.15.6:0/1539253459 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f63d406b4d0 0x7f63d406d990 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:00.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.490+0000 7f63fe000640 1 --2- 172.21.15.6:0/1539253459 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f63f8071e20 0x7f63f815ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:00.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.490+0000 7f63fe000640 1 -- 172.21.15.6:0/1539253459 >> 172.21.15.6:0/1539253459 conn(0x7f63f806b3d0 msgr2=0x7f63f80b2a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:00.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.490+0000 7f63fe000640 1 -- 172.21.15.6:0/1539253459 shutdown_connections 2024-03-04T09:26:00.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.490+0000 7f63fe000640 1 -- 172.21.15.6:0/1539253459 wait complete. 2024-03-04T09:26:00.506 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-04T09:26:00.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.734+0000 7f3b0d000640 1 Processor -- start 2024-03-04T09:26:00.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.734+0000 7f3b0d000640 1 -- start start 2024-03-04T09:26:00.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.736+0000 7f3b0d000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f3b08145430 0x7f3b0815f810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:00.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.736+0000 7f3b0d000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f3b0815cc90 0x7f3b080b1b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:00.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.736+0000 7f3b0d000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b080b3fe0 con 0x7f3b080b2080 2024-03-04T09:26:00.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.736+0000 7f3b0d000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b080b24d0 con 0x7f3b0815cc90 2024-03-04T09:26:00.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.736+0000 7f3b0d000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b0815fe40 con 0x7f3b08145430 2024-03-04T09:26:00.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.736+0000 7f3b06a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f3b0815cc90 0x7f3b080b1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:00.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.736+0000 7f3b06a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f3b0815cc90 0x7f3b080b1b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:46588/0 (socket says 172.21.15.6:46588) 2024-03-04T09:26:00.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.736+0000 7f3b06a00640 1 -- 172.21.15.6:0/2636560452 learned_addr learned my addr 172.21.15.6:0/2636560452 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:00.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.737+0000 7f3b06a00640 1 -- 172.21.15.6:0/2636560452 >> v2:172.21.15.146:3300/0 conn(0x7f3b08145430 msgr2=0x7f3b0815f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:00.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.737+0000 7f3b07400640 1 --2- 172.21.15.6:0/2636560452 >> v2:172.21.15.146:3300/0 conn(0x7f3b08145430 0x7f3b0815f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:00.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.737+0000 7f3b06a00640 1 --2- 172.21.15.6:0/2636560452 >> v2:172.21.15.146:3300/0 conn(0x7f3b08145430 0x7f3b0815f810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:00.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.737+0000 7f3b06a00640 1 -- 172.21.15.6:0/2636560452 >> v1:172.21.15.6:6789/0 conn(0x7f3b080b2080 legacy=0x7f3b0806c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:00.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.737+0000 7f3b06a00640 1 -- 172.21.15.6:0/2636560452 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b08160120 con 0x7f3b0815cc90 2024-03-04T09:26:00.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.737+0000 7f3b06000640 1 -- 172.21.15.6:0/2636560452 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2118526788 0 0) 0x7f3b080b3fe0 con 0x7f3b080b2080 2024-03-04T09:26:00.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.737+0000 7f3b07400640 1 --2- 172.21.15.6:0/2636560452 >> v2:172.21.15.146:3300/0 conn(0x7f3b08145430 0x7f3b0815f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:26:00.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.738+0000 7f3b06a00640 1 --2- 172.21.15.6:0/2636560452 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3b0815cc90 0x7f3b080b1b40 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f3b080b2670 tx=0x7f3af402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9455ab6a377d6ffc server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:00.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.739+0000 7f3b06000640 1 -- 172.21.15.6:0/2636560452 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3af4034070 con 0x7f3b0815cc90 2024-03-04T09:26:00.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.739+0000 7f3b06000640 1 -- 172.21.15.6:0/2636560452 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3af4030860 con 0x7f3b0815cc90 2024-03-04T09:26:00.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.739+0000 7f3b06000640 1 -- 172.21.15.6:0/2636560452 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3af4030c30 con 0x7f3b0815cc90 2024-03-04T09:26:00.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.740+0000 7f3b0d000640 1 -- 172.21.15.6:0/2636560452 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3b0815cc90 msgr2=0x7f3b080b1b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:00.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.740+0000 7f3b0d000640 1 --2- 172.21.15.6:0/2636560452 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3b0815cc90 0x7f3b080b1b40 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f3b080b2670 tx=0x7f3af402b9a0 comp rx=0 tx=0).stop 2024-03-04T09:26:00.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.741+0000 7f3b0d000640 1 -- 172.21.15.6:0/2636560452 shutdown_connections 2024-03-04T09:26:00.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.741+0000 7f3b0d000640 1 --2- 172.21.15.6:0/2636560452 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3b0815cc90 0x7f3b080b1b40 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:00.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.741+0000 7f3b0d000640 1 --2- 172.21.15.6:0/2636560452 >> v2:172.21.15.146:3300/0 conn(0x7f3b08145430 0x7f3b0815f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:00.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.741+0000 7f3b0d000640 1 -- 172.21.15.6:0/2636560452 >> 172.21.15.6:0/2636560452 conn(0x7f3b0806b3a0 msgr2=0x7f3b0806b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:00.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.741+0000 7f3b0d000640 1 -- 172.21.15.6:0/2636560452 shutdown_connections 2024-03-04T09:26:00.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.742+0000 7f3b0d000640 1 -- 172.21.15.6:0/2636560452 wait complete. 2024-03-04T09:26:00.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.743+0000 7f3b0d000640 1 Processor -- start 2024-03-04T09:26:00.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.744+0000 7f3b0d000640 1 -- start start 2024-03-04T09:26:00.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.744+0000 7f3b0d000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3b080b2080 0x7f3b0813f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:00.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.744+0000 7f3b0d000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3b08145430 0x7f3b08071ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:00.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.744+0000 7f3b0d000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3b0813c500 0x7f3b08072010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:00.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.744+0000 7f3b0d000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b0815d0d0 con 0x7f3b08145430 2024-03-04T09:26:00.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.744+0000 7f3b0d000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b081606e0 con 0x7f3b0813c500 2024-03-04T09:26:00.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.744+0000 7f3b0d000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b08145810 con 0x7f3b080b2080 2024-03-04T09:26:00.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.745+0000 7f3b06a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3b08145430 0x7f3b08071ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:00.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.745+0000 7f3b06a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3b08145430 0x7f3b08071ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:46604/0 (socket says 172.21.15.6:46604) 2024-03-04T09:26:00.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.745+0000 7f3b06a00640 1 -- 172.21.15.6:0/3191295152 learned_addr learned my addr 172.21.15.6:0/3191295152 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:00.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.745+0000 7f3b07e00640 1 --2- 172.21.15.6:0/3191295152 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3b0813c500 0x7f3b08072010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:00.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.745+0000 7f3b07400640 1 --2- 172.21.15.6:0/3191295152 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3b080b2080 0x7f3b0813f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:00.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.745+0000 7f3b06a00640 1 -- 172.21.15.6:0/3191295152 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3b080b2080 msgr2=0x7f3b0813f3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:00.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.745+0000 7f3b06a00640 1 --2- 172.21.15.6:0/3191295152 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3b080b2080 0x7f3b0813f3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:00.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.745+0000 7f3b06a00640 1 -- 172.21.15.6:0/3191295152 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3b0813c500 msgr2=0x7f3b08072010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:00.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.745+0000 7f3b06a00640 1 --2- 172.21.15.6:0/3191295152 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3b0813c500 0x7f3b08072010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:00.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.745+0000 7f3b06a00640 1 -- 172.21.15.6:0/3191295152 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b0813ca70 con 0x7f3b08145430 2024-03-04T09:26:00.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.745+0000 7f3b07400640 1 --2- 172.21.15.6:0/3191295152 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3b080b2080 0x7f3b0813f3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:26:00.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.746+0000 7f3b06a00640 1 --2- 172.21.15.6:0/3191295152 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3b08145430 0x7f3b08071ad0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f3af40090b0 tx=0x7f3af4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:00.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.746+0000 7f3aff400640 1 -- 172.21.15.6:0/3191295152 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3af4048070 con 0x7f3b08145430 2024-03-04T09:26:00.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.746+0000 7f3aff400640 1 -- 172.21.15.6:0/3191295152 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3af4030660 con 0x7f3b08145430 2024-03-04T09:26:00.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.746+0000 7f3b0d000640 1 -- 172.21.15.6:0/3191295152 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b0813cd50 con 0x7f3b08145430 2024-03-04T09:26:00.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.746+0000 7f3aff400640 1 -- 172.21.15.6:0/3191295152 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3af4034040 con 0x7f3b08145430 2024-03-04T09:26:00.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.746+0000 7f3b0d000640 1 -- 172.21.15.6:0/3191295152 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b0813d310 con 0x7f3b08145430 2024-03-04T09:26:00.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.747+0000 7f3b0d000640 1 -- 172.21.15.6:0/3191295152 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b080cd140 con 0x7f3b08145430 2024-03-04T09:26:00.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.748+0000 7f3aff400640 1 -- 172.21.15.6:0/3191295152 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3af404b070 con 0x7f3b08145430 2024-03-04T09:26:00.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.749+0000 7f3aff400640 1 --2- 172.21.15.6:0/3191295152 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f3ae806b4d0 0x7f3ae806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:00.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.749+0000 7f3aff400640 1 -- 172.21.15.6:0/3191295152 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f3af40ac3d0 con 0x7f3b08145430 2024-03-04T09:26:00.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.750+0000 7f3b07400640 1 --2- 172.21.15.6:0/3191295152 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f3ae806b4d0 0x7f3ae806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:00.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.750+0000 7f3b07400640 1 --2- 172.21.15.6:0/3191295152 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f3ae806b4d0 0x7f3ae806d990 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f3af8006fd0 tx=0x7f3af8008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:00.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.754+0000 7f3aff400640 1 -- 172.21.15.6:0/3191295152 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3b080cd140 con 0x7f3b08145430 2024-03-04T09:26:00.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.951+0000 7f3b0d000640 1 -- 172.21.15.6:0/3191295152 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f3b0804f330 con 0x7f3b08145430 2024-03-04T09:26:00.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.952+0000 7f3aff400640 1 -- 172.21.15.6:0/3191295152 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7f3af407b2d0 con 0x7f3b08145430 2024-03-04T09:26:00.949 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 64 2024-03-04T09:26:00.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.955+0000 7f3b0d000640 1 -- 172.21.15.6:0/3191295152 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f3ae806b4d0 msgr2=0x7f3ae806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:00.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.955+0000 7f3b0d000640 1 --2- 172.21.15.6:0/3191295152 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f3ae806b4d0 0x7f3ae806d990 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f3af8006fd0 tx=0x7f3af8008040 comp rx=0 tx=0).stop 2024-03-04T09:26:00.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.955+0000 7f3b0d000640 1 -- 172.21.15.6:0/3191295152 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3b08145430 msgr2=0x7f3b08071ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:00.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.955+0000 7f3b0d000640 1 --2- 172.21.15.6:0/3191295152 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3b08145430 0x7f3b08071ad0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f3af40090b0 tx=0x7f3af4004730 comp rx=0 tx=0).stop 2024-03-04T09:26:00.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.955+0000 7f3b0d000640 1 -- 172.21.15.6:0/3191295152 shutdown_connections 2024-03-04T09:26:00.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.955+0000 7f3b0d000640 1 --2- 172.21.15.6:0/3191295152 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3b0813c500 0x7f3b08072010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:00.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.955+0000 7f3b0d000640 1 --2- 172.21.15.6:0/3191295152 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3b08145430 0x7f3b08071ad0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:00.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.955+0000 7f3b0d000640 1 --2- 172.21.15.6:0/3191295152 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f3ae806b4d0 0x7f3ae806d990 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:00.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.955+0000 7f3b0d000640 1 --2- 172.21.15.6:0/3191295152 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3b080b2080 0x7f3b0813f3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:00.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.955+0000 7f3b0d000640 1 -- 172.21.15.6:0/3191295152 >> 172.21.15.6:0/3191295152 conn(0x7f3b0806b3a0 msgr2=0x7f3b080cde00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:00.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.955+0000 7f3b0d000640 1 -- 172.21.15.6:0/3191295152 shutdown_connections 2024-03-04T09:26:00.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:00.955+0000 7f3b0d000640 1 -- 172.21.15.6:0/3191295152 wait complete. 2024-03-04T09:26:00.969 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_dir_deletion' 2024-03-04T09:26:01.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.185+0000 7f50b7200640 1 Processor -- start 2024-03-04T09:26:01.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.186+0000 7f50b7200640 1 -- start start 2024-03-04T09:26:01.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.187+0000 7f50b7200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f50b0145300 0x7f50b00b49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:01.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.187+0000 7f50b7200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f50b0070f10 0x7f50b00cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:01.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.187+0000 7f50b7200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f50b00cce90 0x7f50b00cd930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:01.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.187+0000 7f50b7200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50b00cd300 con 0x7f50b0145300 2024-03-04T09:26:01.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.187+0000 7f50b7200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50b00714d0 con 0x7f50b00cce90 2024-03-04T09:26:01.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.187+0000 7f50b7200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50b00717b0 con 0x7f50b0070f10 2024-03-04T09:26:01.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.188+0000 7f50b5e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f50b0145300 0x7f50b00b49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:01.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.188+0000 7f50b5e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f50b0145300 0x7f50b00b49a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:46614/0 (socket says 172.21.15.6:46614) 2024-03-04T09:26:01.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.188+0000 7f50b5e00640 1 -- 172.21.15.6:0/1100734128 learned_addr learned my addr 172.21.15.6:0/1100734128 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:01.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.188+0000 7f50b6800640 1 --2- 172.21.15.6:0/1100734128 >> v2:172.21.15.146:3300/0 conn(0x7f50b00cce90 0x7f50b00cd930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:01.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.188+0000 7f50b5400640 1 --2- 172.21.15.6:0/1100734128 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f50b0070f10 0x7f50b00cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:01.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.188+0000 7f50b5e00640 1 -- 172.21.15.6:0/1100734128 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f50b0070f10 msgr2=0x7f50b00cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:01.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.188+0000 7f50b5e00640 1 --2- 172.21.15.6:0/1100734128 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f50b0070f10 0x7f50b00cc950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:01.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.188+0000 7f50b5e00640 1 -- 172.21.15.6:0/1100734128 >> v2:172.21.15.146:3300/0 conn(0x7f50b00cce90 msgr2=0x7f50b00cd930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:01.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.188+0000 7f50b5e00640 1 --2- 172.21.15.6:0/1100734128 >> v2:172.21.15.146:3300/0 conn(0x7f50b00cce90 0x7f50b00cd930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:01.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.188+0000 7f50b5e00640 1 -- 172.21.15.6:0/1100734128 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50b0071a90 con 0x7f50b0145300 2024-03-04T09:26:01.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.189+0000 7f50b5e00640 1 --2- 172.21.15.6:0/1100734128 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f50b0145300 0x7f50b00b49a0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f50b0072420 tx=0x7f50a402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86f5fca2e3648473 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:01.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.190+0000 7f50b4a00640 1 -- 172.21.15.6:0/1100734128 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50a4035020 con 0x7f50b0145300 2024-03-04T09:26:01.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.190+0000 7f50b4a00640 1 -- 172.21.15.6:0/1100734128 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50a4031870 con 0x7f50b0145300 2024-03-04T09:26:01.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.190+0000 7f50b4a00640 1 -- 172.21.15.6:0/1100734128 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50a4031c40 con 0x7f50b0145300 2024-03-04T09:26:01.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.190+0000 7f50b7200640 1 -- 172.21.15.6:0/1100734128 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f50b0145300 msgr2=0x7f50b00b49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:01.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.190+0000 7f50b7200640 1 --2- 172.21.15.6:0/1100734128 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f50b0145300 0x7f50b00b49a0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f50b0072420 tx=0x7f50a402b910 comp rx=0 tx=0).stop 2024-03-04T09:26:01.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.190+0000 7f50b7200640 1 -- 172.21.15.6:0/1100734128 shutdown_connections 2024-03-04T09:26:01.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.190+0000 7f50b7200640 1 --2- 172.21.15.6:0/1100734128 >> v2:172.21.15.146:3300/0 conn(0x7f50b00cce90 0x7f50b00cd930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:01.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.190+0000 7f50b7200640 1 --2- 172.21.15.6:0/1100734128 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f50b0070f10 0x7f50b00cc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:01.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.190+0000 7f50b7200640 1 --2- 172.21.15.6:0/1100734128 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f50b0145300 0x7f50b00b49a0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:01.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.190+0000 7f50b7200640 1 -- 172.21.15.6:0/1100734128 >> 172.21.15.6:0/1100734128 conn(0x7f50b006b3d0 msgr2=0x7f50b006b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:01.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.191+0000 7f50b7200640 1 -- 172.21.15.6:0/1100734128 shutdown_connections 2024-03-04T09:26:01.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.191+0000 7f50b7200640 1 -- 172.21.15.6:0/1100734128 wait complete. 2024-03-04T09:26:01.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.192+0000 7f50b7200640 1 Processor -- start 2024-03-04T09:26:01.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.193+0000 7f50b7200640 1 -- start start 2024-03-04T09:26:01.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.193+0000 7f50b7200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f50b0070f10 0x7f50b01417b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:01.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.193+0000 7f50b7200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f50b00cce90 0x7f50b013ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:01.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.194+0000 7f50b7200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f50b0145300 0x7f50b013fee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:01.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.194+0000 7f50b7200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50b01456e0 con 0x7f50b00cce90 2024-03-04T09:26:01.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.194+0000 7f50b7200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50b00b4ee0 con 0x7f50b0070f10 2024-03-04T09:26:01.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.194+0000 7f50b7200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50b00cde70 con 0x7f50b0145300 2024-03-04T09:26:01.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.194+0000 7f50b5400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f50b00cce90 0x7f50b013ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:01.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.194+0000 7f50b5400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f50b00cce90 0x7f50b013ddc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:46630/0 (socket says 172.21.15.6:46630) 2024-03-04T09:26:01.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.194+0000 7f50b5400640 1 -- 172.21.15.6:0/1895059249 learned_addr learned my addr 172.21.15.6:0/1895059249 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:01.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.194+0000 7f50b5e00640 1 --2- 172.21.15.6:0/1895059249 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f50b0070f10 0x7f50b01417b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:01.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.194+0000 7f50b6800640 1 --2- 172.21.15.6:0/1895059249 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f50b0145300 0x7f50b013fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:01.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.194+0000 7f50b5400640 1 -- 172.21.15.6:0/1895059249 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f50b0145300 msgr2=0x7f50b013fee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:01.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.194+0000 7f50b5400640 1 --2- 172.21.15.6:0/1895059249 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f50b0145300 0x7f50b013fee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:01.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.194+0000 7f50b5400640 1 -- 172.21.15.6:0/1895059249 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f50b0070f10 msgr2=0x7f50b01417b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:01.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.194+0000 7f50b5400640 1 --2- 172.21.15.6:0/1895059249 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f50b0070f10 0x7f50b01417b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:01.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.194+0000 7f50b5400640 1 -- 172.21.15.6:0/1895059249 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50b0140710 con 0x7f50b00cce90 2024-03-04T09:26:01.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.194+0000 7f50b5e00640 1 --2- 172.21.15.6:0/1895059249 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f50b0070f10 0x7f50b01417b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:26:01.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.195+0000 7f50b5400640 1 --2- 172.21.15.6:0/1895059249 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f50b00cce90 0x7f50b013ddc0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f509c00d800 tx=0x7f509c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:01.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.195+0000 7f50aa000640 1 -- 172.21.15.6:0/1895059249 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f509c00b6c0 con 0x7f50b00cce90 2024-03-04T09:26:01.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.195+0000 7f50aa000640 1 -- 172.21.15.6:0/1895059249 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f509c019070 con 0x7f50b00cce90 2024-03-04T09:26:01.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.195+0000 7f50b7200640 1 -- 172.21.15.6:0/1895059249 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50b01409f0 con 0x7f50b00cce90 2024-03-04T09:26:01.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.195+0000 7f50aa000640 1 -- 172.21.15.6:0/1895059249 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f509c00ba80 con 0x7f50b00cce90 2024-03-04T09:26:01.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.195+0000 7f50b7200640 1 -- 172.21.15.6:0/1895059249 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50b0140fb0 con 0x7f50b00cce90 2024-03-04T09:26:01.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.196+0000 7f50b7200640 1 -- 172.21.15.6:0/1895059249 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50b004f330 con 0x7f50b00cce90 2024-03-04T09:26:01.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.197+0000 7f50aa000640 1 -- 172.21.15.6:0/1895059249 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f509c00bc20 con 0x7f50b00cce90 2024-03-04T09:26:01.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.198+0000 7f50aa000640 1 --2- 172.21.15.6:0/1895059249 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f509006b4d0 0x7f509006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:01.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.198+0000 7f50aa000640 1 -- 172.21.15.6:0/1895059249 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f509c08a8a0 con 0x7f50b00cce90 2024-03-04T09:26:01.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.199+0000 7f50b5e00640 1 --2- 172.21.15.6:0/1895059249 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f509006b4d0 0x7f509006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:01.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.200+0000 7f50b5e00640 1 --2- 172.21.15.6:0/1895059249 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f509006b4d0 0x7f509006d990 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f50a4036000 tx=0x7f50a4008030 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:01.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.203+0000 7f50aa000640 1 -- 172.21.15.6:0/1895059249 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f509c059560 con 0x7f50b00cce90 2024-03-04T09:26:01.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.475+0000 7f50b7200640 1 -- 172.21.15.6:0/1895059249 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_dir_deletion"]} v 0) v1 -- 0x7f50b01413b0 con 0x7f50b00cce90 2024-03-04T09:26:01.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.746+0000 7f50aa000640 1 -- 172.21.15.6:0/1895059249 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_dir_deletion"]}]=0 v60) v1 ==== 135+0+0 (secure 0 0 0) 0x7f509c058e60 con 0x7f50b00cce90 2024-03-04T09:26:01.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.751+0000 7f50b7200640 1 -- 172.21.15.6:0/1895059249 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f509006b4d0 msgr2=0x7f509006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:01.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.751+0000 7f50b7200640 1 --2- 172.21.15.6:0/1895059249 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f509006b4d0 0x7f509006d990 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f50a4036000 tx=0x7f50a4008030 comp rx=0 tx=0).stop 2024-03-04T09:26:01.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.751+0000 7f50b7200640 1 -- 172.21.15.6:0/1895059249 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f50b00cce90 msgr2=0x7f50b013ddc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:01.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.751+0000 7f50b7200640 1 --2- 172.21.15.6:0/1895059249 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f50b00cce90 0x7f50b013ddc0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f509c00d800 tx=0x7f509c00dcc0 comp rx=0 tx=0).stop 2024-03-04T09:26:01.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.752+0000 7f50b7200640 1 -- 172.21.15.6:0/1895059249 shutdown_connections 2024-03-04T09:26:01.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.752+0000 7f50b7200640 1 --2- 172.21.15.6:0/1895059249 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f509006b4d0 0x7f509006d990 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:01.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.752+0000 7f50b7200640 1 --2- 172.21.15.6:0/1895059249 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f50b0145300 0x7f50b013fee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:01.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.752+0000 7f50b7200640 1 --2- 172.21.15.6:0/1895059249 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f50b00cce90 0x7f50b013ddc0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:01.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.752+0000 7f50b7200640 1 --2- 172.21.15.6:0/1895059249 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f50b0070f10 0x7f50b01417b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:01.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.752+0000 7f50b7200640 1 -- 172.21.15.6:0/1895059249 >> 172.21.15.6:0/1895059249 conn(0x7f50b006b3d0 msgr2=0x7f50b00b28b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:01.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.752+0000 7f50b7200640 1 -- 172.21.15.6:0/1895059249 shutdown_connections 2024-03-04T09:26:01.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.752+0000 7f50b7200640 1 -- 172.21.15.6:0/1895059249 wait complete. 2024-03-04T09:26:01.769 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-04T09:26:01.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.983+0000 7fdd14800640 1 Processor -- start 2024-03-04T09:26:01.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.983+0000 7fdd14800640 1 -- start start 2024-03-04T09:26:01.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.985+0000 7fdd14800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fdd0c0cf500 0x7fdd0c0d21c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:01.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.985+0000 7fdd14800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdd0c0b43a0 0x7fdd0c0b4830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:01.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.985+0000 7fdd14800640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd0c15e640 con 0x7fdd0c15e260 2024-03-04T09:26:01.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.985+0000 7fdd14800640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd0c0d4ca0 con 0x7fdd0c0cf500 2024-03-04T09:26:01.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.985+0000 7fdd14800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd0c0cfa70 con 0x7fdd0c0b43a0 2024-03-04T09:26:01.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.985+0000 7fdd11e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fdd0c15e260 0x7fdd0c0d1ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:56518/0 (socket says 172.21.15.6:56518) 2024-03-04T09:26:01.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.985+0000 7fdd11e00640 1 -- 172.21.15.6:0/2299291299 learned_addr learned my addr 172.21.15.6:0/2299291299 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:26:01.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.985+0000 7fdd12800640 1 --2- 172.21.15.6:0/2299291299 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdd0c0b43a0 0x7fdd0c0b4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:01.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.985+0000 7fdd11400640 1 --2- 172.21.15.6:0/2299291299 >> v2:172.21.15.146:3300/0 conn(0x7fdd0c0cf500 0x7fdd0c0d21c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:01.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.986+0000 7fdd10a00640 1 -- 172.21.15.6:0/2299291299 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 728586899 0 0) 0x7fdd0c15e640 con 0x7fdd0c15e260 2024-03-04T09:26:01.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.986+0000 7fdd10a00640 1 -- 172.21.15.6:0/2299291299 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdcf8003520 con 0x7fdd0c15e260 2024-03-04T09:26:01.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.986+0000 7fdd10a00640 1 -- 172.21.15.6:0/2299291299 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2144289422 0 0) 0x7fdcf8003520 con 0x7fdd0c15e260 2024-03-04T09:26:01.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.986+0000 7fdd10a00640 1 -- 172.21.15.6:0/2299291299 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdd0c0b43a0 msgr2=0x7fdd0c0b4830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:01.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.986+0000 7fdd10a00640 1 --2- 172.21.15.6:0/2299291299 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdd0c0b43a0 0x7fdd0c0b4830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:01.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.986+0000 7fdd10a00640 1 -- 172.21.15.6:0/2299291299 >> v2:172.21.15.146:3300/0 conn(0x7fdd0c0cf500 msgr2=0x7fdd0c0d21c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:01.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.986+0000 7fdd10a00640 1 --2- 172.21.15.6:0/2299291299 >> v2:172.21.15.146:3300/0 conn(0x7fdd0c0cf500 0x7fdd0c0d21c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:01.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.986+0000 7fdd10a00640 1 -- 172.21.15.6:0/2299291299 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd0c0cfd50 con 0x7fdd0c15e260 2024-03-04T09:26:01.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.986+0000 7fdd10a00640 1 -- 172.21.15.6:0/2299291299 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7fdd04001670 con 0x7fdd0c15e260 2024-03-04T09:26:01.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.986+0000 7fdd10a00640 1 -- 172.21.15.6:0/2299291299 >> v1:172.21.15.6:6789/0 conn(0x7fdd0c15e260 legacy=0x7fdd0c0d1ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:01.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.986+0000 7fdd10a00640 1 --2- 172.21.15.6:0/2299291299 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdd0c0b43a0 0x7fdcf80039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:01.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.986+0000 7fdd10a00640 1 --2- 172.21.15.6:0/2299291299 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdcf8003f30 0x7fdcf8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:01.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.986+0000 7fdd10a00640 1 --2- 172.21.15.6:0/2299291299 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdcf8004d40 0x7fdcf80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:01.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.986+0000 7fdd10a00640 1 -- 172.21.15.6:0/2299291299 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd0c0d4ca0 con 0x7fdcf8004d40 2024-03-04T09:26:01.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.986+0000 7fdd10a00640 1 -- 172.21.15.6:0/2299291299 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd0c0cfa70 con 0x7fdcf8003f30 2024-03-04T09:26:01.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.986+0000 7fdd10a00640 1 -- 172.21.15.6:0/2299291299 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcf800b9c0 con 0x7fdd0c0b43a0 2024-03-04T09:26:01.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.987+0000 7fdd12800640 1 --2- 172.21.15.6:0/2299291299 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdcf8004d40 0x7fdcf80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:01.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.987+0000 7fdd11e00640 1 --2- 172.21.15.6:0/2299291299 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdd0c0b43a0 0x7fdcf80039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:01.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.987+0000 7fdd11400640 1 --2- 172.21.15.6:0/2299291299 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdcf8003f30 0x7fdcf8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:01.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.987+0000 7fdd12800640 1 -- 172.21.15.6:0/2299291299 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdd0c0b43a0 msgr2=0x7fdcf80039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:01.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.987+0000 7fdd12800640 1 --2- 172.21.15.6:0/2299291299 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdd0c0b43a0 0x7fdcf80039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:01.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.987+0000 7fdd12800640 1 -- 172.21.15.6:0/2299291299 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdcf8003f30 msgr2=0x7fdcf8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:01.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.987+0000 7fdd12800640 1 --2- 172.21.15.6:0/2299291299 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdcf8003f30 0x7fdcf8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:01.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.987+0000 7fdd12800640 1 -- 172.21.15.6:0/2299291299 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcf8003520 con 0x7fdcf8004d40 2024-03-04T09:26:01.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.988+0000 7fdd12800640 1 --2- 172.21.15.6:0/2299291299 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdcf8004d40 0x7fdcf80051c0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fdd08009fc0 tx=0x7fdcf8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3c59eb1d7585f2b server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:01.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.988+0000 7fdd10a00640 1 -- 172.21.15.6:0/2299291299 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd08039070 con 0x7fdcf8004d40 2024-03-04T09:26:01.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.989+0000 7fdd10a00640 1 -- 172.21.15.6:0/2299291299 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd0802cc00 con 0x7fdcf8004d40 2024-03-04T09:26:01.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.989+0000 7fdd10a00640 1 -- 172.21.15.6:0/2299291299 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd08009c10 con 0x7fdcf8004d40 2024-03-04T09:26:01.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.989+0000 7fdd14800640 1 -- 172.21.15.6:0/2299291299 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdcf8004d40 msgr2=0x7fdcf80051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:01.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.989+0000 7fdd14800640 1 --2- 172.21.15.6:0/2299291299 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdcf8004d40 0x7fdcf80051c0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fdd08009fc0 tx=0x7fdcf8005830 comp rx=0 tx=0).stop 2024-03-04T09:26:01.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.989+0000 7fdd12800640 1 -- 172.21.15.6:0/2299291299 reap_dead start 2024-03-04T09:26:01.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.989+0000 7fdd14800640 1 -- 172.21.15.6:0/2299291299 shutdown_connections 2024-03-04T09:26:01.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.989+0000 7fdd14800640 1 -- 172.21.15.6:0/2299291299 >> 172.21.15.6:0/2299291299 conn(0x7fdd0c0d3ee0 msgr2=0x7fdd0c0d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:01.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.989+0000 7fdd14800640 1 -- 172.21.15.6:0/2299291299 shutdown_connections 2024-03-04T09:26:01.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.989+0000 7fdd14800640 1 -- 172.21.15.6:0/2299291299 wait complete. 2024-03-04T09:26:01.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.991+0000 7fdd14800640 1 Processor -- start 2024-03-04T09:26:01.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.991+0000 7fdd14800640 1 -- start start 2024-03-04T09:26:01.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.992+0000 7fdd14800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdcf8004d40 0x7fdd0c0a1550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:01.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.992+0000 7fdd14800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdd0c0cda40 0x7fdd0c157d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:01.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.992+0000 7fdd14800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdd0c160a00 0x7fdd0c160e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:01.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.992+0000 7fdd14800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd0c0b3e80 con 0x7fdd0c0cda40 2024-03-04T09:26:01.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.992+0000 7fdd14800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd0c0b3ba0 con 0x7fdd0c160a00 2024-03-04T09:26:01.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.992+0000 7fdd14800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd0c0d2700 con 0x7fdcf8004d40 2024-03-04T09:26:01.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.993+0000 7fdd11400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdd0c0cda40 0x7fdd0c157d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:01.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.993+0000 7fdd11400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdd0c0cda40 0x7fdd0c157d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:46650/0 (socket says 172.21.15.6:46650) 2024-03-04T09:26:01.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.993+0000 7fdd11400640 1 -- 172.21.15.6:0/3246768138 learned_addr learned my addr 172.21.15.6:0/3246768138 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:01.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.993+0000 7fdd11e00640 1 --2- 172.21.15.6:0/3246768138 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdcf8004d40 0x7fdd0c0a1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:01.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.993+0000 7fdd12800640 1 --2- 172.21.15.6:0/3246768138 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdd0c160a00 0x7fdd0c160e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:01.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.993+0000 7fdd11400640 1 -- 172.21.15.6:0/3246768138 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdcf8004d40 msgr2=0x7fdd0c0a1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:01.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.993+0000 7fdd11400640 1 --2- 172.21.15.6:0/3246768138 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdcf8004d40 0x7fdd0c0a1550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:01.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.993+0000 7fdd11400640 1 -- 172.21.15.6:0/3246768138 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdd0c160a00 msgr2=0x7fdd0c160e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:01.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.993+0000 7fdd11400640 1 --2- 172.21.15.6:0/3246768138 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdd0c160a00 0x7fdd0c160e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:01.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.993+0000 7fdd11400640 1 -- 172.21.15.6:0/3246768138 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd0c158660 con 0x7fdd0c0cda40 2024-03-04T09:26:01.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.993+0000 7fdd11e00640 1 --2- 172.21.15.6:0/3246768138 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdcf8004d40 0x7fdd0c0a1550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T09:26:01.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.994+0000 7fdd11400640 1 --2- 172.21.15.6:0/3246768138 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdd0c0cda40 0x7fdd0c157d60 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fdcfc011600 tx=0x7fdcfc011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:01.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.994+0000 7fdd02000640 1 -- 172.21.15.6:0/3246768138 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcfc00f920 con 0x7fdd0c0cda40 2024-03-04T09:26:01.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.994+0000 7fdd02000640 1 -- 172.21.15.6:0/3246768138 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcfc00fac0 con 0x7fdd0c0cda40 2024-03-04T09:26:01.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.994+0000 7fdd02000640 1 -- 172.21.15.6:0/3246768138 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcfc00fe70 con 0x7fdd0c0cda40 2024-03-04T09:26:01.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.994+0000 7fdd14800640 1 -- 172.21.15.6:0/3246768138 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd0c158940 con 0x7fdd0c0cda40 2024-03-04T09:26:01.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.994+0000 7fdd14800640 1 -- 172.21.15.6:0/3246768138 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd0c1593c0 con 0x7fdd0c0cda40 2024-03-04T09:26:01.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.995+0000 7fdd14800640 1 -- 172.21.15.6:0/3246768138 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd0c1613c0 con 0x7fdd0c0cda40 2024-03-04T09:26:01.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.996+0000 7fdd02000640 1 -- 172.21.15.6:0/3246768138 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdcfc015460 con 0x7fdd0c0cda40 2024-03-04T09:26:01.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.997+0000 7fdd02000640 1 --2- 172.21.15.6:0/3246768138 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fdcf006b4d0 0x7fdcf006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:01.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.997+0000 7fdd02000640 1 -- 172.21.15.6:0/3246768138 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7fdcfc08acb0 con 0x7fdd0c0cda40 2024-03-04T09:26:01.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.998+0000 7fdd11e00640 1 --2- 172.21.15.6:0/3246768138 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fdcf006b4d0 0x7fdcf006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:01.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:01.999+0000 7fdd11e00640 1 --2- 172.21.15.6:0/3246768138 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fdcf006b4d0 0x7fdcf006d990 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fdd04001eb0 tx=0x7fdd04009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:01.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:02.001+0000 7fdd02000640 1 -- 172.21.15.6:0/3246768138 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdcfc019030 con 0x7fdd0c0cda40 2024-03-04T09:26:02.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:02.203+0000 7fdd14800640 1 -- 172.21.15.6:0/3246768138 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fdd0c04f330 con 0x7fdd0c0cda40 2024-03-04T09:26:02.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:02.210+0000 7fdd02000640 1 -- 172.21.15.6:0/3246768138 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v10)=0 v10) v1 ==== 122+0+0 (secure 0 0 0) 0x7fdcfc059a00 con 0x7fdd0c0cda40 2024-03-04T09:26:02.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:02.213+0000 7fdd14800640 1 -- 172.21.15.6:0/3246768138 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fdcf006b4d0 msgr2=0x7fdcf006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:02.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:02.213+0000 7fdd14800640 1 --2- 172.21.15.6:0/3246768138 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fdcf006b4d0 0x7fdcf006d990 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fdd04001eb0 tx=0x7fdd04009040 comp rx=0 tx=0).stop 2024-03-04T09:26:02.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:02.213+0000 7fdd14800640 1 -- 172.21.15.6:0/3246768138 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdd0c0cda40 msgr2=0x7fdd0c157d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:02.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:02.213+0000 7fdd14800640 1 --2- 172.21.15.6:0/3246768138 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdd0c0cda40 0x7fdd0c157d60 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fdcfc011600 tx=0x7fdcfc011ac0 comp rx=0 tx=0).stop 2024-03-04T09:26:02.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:02.213+0000 7fdd14800640 1 -- 172.21.15.6:0/3246768138 shutdown_connections 2024-03-04T09:26:02.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:02.213+0000 7fdd14800640 1 --2- 172.21.15.6:0/3246768138 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdd0c160a00 0x7fdd0c160e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:02.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:02.213+0000 7fdd14800640 1 --2- 172.21.15.6:0/3246768138 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdd0c0cda40 0x7fdd0c157d60 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:02.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:02.213+0000 7fdd14800640 1 --2- 172.21.15.6:0/3246768138 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fdcf006b4d0 0x7fdcf006d990 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:02.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:02.213+0000 7fdd14800640 1 --2- 172.21.15.6:0/3246768138 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdcf8004d40 0x7fdd0c0a1550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:02.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:02.213+0000 7fdd14800640 1 -- 172.21.15.6:0/3246768138 >> 172.21.15.6:0/3246768138 conn(0x7fdd0c0d3ee0 msgr2=0x7fdd0c0d0110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:02.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:02.213+0000 7fdd14800640 1 -- 172.21.15.6:0/3246768138 shutdown_connections 2024-03-04T09:26:02.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:02.213+0000 7fdd14800640 1 -- 172.21.15.6:0/3246768138 wait complete. 2024-03-04T09:26:02.229 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-04T09:26:02.229 DEBUG:teuthology.orchestra.run.smithi006:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-04T09:26:02.346 DEBUG:teuthology.orchestra.run.smithi146:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-04T09:26:02.457 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-04T09:26:02.458 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-04T09:26:02.458 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T09:26:02.518 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-04T09:26:02.518 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T09:26:02.572 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-04T09:26:02.572 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T09:26:02.630 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T09:26:02.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:02.857+0000 7f4b9ea00640 1 Processor -- start 2024-03-04T09:26:02.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:02.857+0000 7f4b9ea00640 1 -- start start 2024-03-04T09:26:02.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:02.859+0000 7f4b9ea00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f4b980cdc20 0x7f4b98071760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:02.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:02.859+0000 7f4b9ea00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b980b4860 con 0x7f4b980b3c90 2024-03-04T09:26:02.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:02.859+0000 7f4b9ea00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b980b4b40 con 0x7f4b980cf5c0 2024-03-04T09:26:02.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:02.859+0000 7f4b9ea00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b980b4e20 con 0x7f4b980cdc20 2024-03-04T09:26:02.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:02.859+0000 7f4b97e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f4b980b3c90 0x7f4b980b4070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:56526/0 (socket says 172.21.15.6:56526) 2024-03-04T09:26:02.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:02.859+0000 7f4b97e00640 1 -- 172.21.15.6:0/1988398230 learned_addr learned my addr 172.21.15.6:0/1988398230 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:26:02.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:02.859+0000 7f4b9ca00640 1 --2- 172.21.15.6:0/1988398230 >> v2:172.21.15.6:3300/0 conn(0x7f4b980cdc20 0x7f4b98071760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:02.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:02.859+0000 7f4b97400640 1 -- 172.21.15.6:0/1988398230 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2218012502 0 0) 0x7f4b980b4860 con 0x7f4b980b3c90 2024-03-04T09:26:02.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:02.860+0000 7f4b97400640 1 -- 172.21.15.6:0/1988398230 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b7c005730 con 0x7f4b980b3c90 2024-03-04T09:26:02.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:02.860+0000 7f4b9ca00640 1 -- 172.21.15.6:0/1988398230 >> v1:172.21.15.146:6789/0 conn(0x7f4b980cf5c0 legacy=0x7f4b98071050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:02.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:02.860+0000 7f4b9ca00640 1 -- 172.21.15.6:0/1988398230 >> v1:172.21.15.6:6789/0 conn(0x7f4b980b3c90 legacy=0x7f4b980b4070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:02.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:02.860+0000 7f4b9ca00640 1 -- 172.21.15.6:0/1988398230 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b980b5100 con 0x7f4b980cdc20 2024-03-04T09:26:02.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:02.861+0000 7f4b9ca00640 1 --2- 172.21.15.6:0/1988398230 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4b980cdc20 0x7f4b98071760 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f4b900097a0 tx=0x7f4b9002b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25a6e8d24b636c9f server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:02.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:02.861+0000 7f4b97400640 1 -- 172.21.15.6:0/1988398230 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b9002f810 con 0x7f4b980cdc20 2024-03-04T09:26:02.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:02.861+0000 7f4b97400640 1 -- 172.21.15.6:0/1988398230 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b9002f9b0 con 0x7f4b980cdc20 2024-03-04T09:26:02.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:02.861+0000 7f4b97400640 1 -- 172.21.15.6:0/1988398230 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b9002fd80 con 0x7f4b980cdc20 2024-03-04T09:26:02.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:02.862+0000 7f4b9ea00640 1 -- 172.21.15.6:0/1988398230 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4b980cdc20 msgr2=0x7f4b98071760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:02.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:02.862+0000 7f4b9ea00640 1 --2- 172.21.15.6:0/1988398230 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4b980cdc20 0x7f4b98071760 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f4b900097a0 tx=0x7f4b9002b9f0 comp rx=0 tx=0).stop 2024-03-04T09:26:02.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:02.862+0000 7f4b9ea00640 1 -- 172.21.15.6:0/1988398230 shutdown_connections 2024-03-04T09:26:02.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:02.862+0000 7f4b9ea00640 1 --2- 172.21.15.6:0/1988398230 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4b980cdc20 0x7f4b98071760 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:02.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:02.862+0000 7f4b9ea00640 1 -- 172.21.15.6:0/1988398230 >> 172.21.15.6:0/1988398230 conn(0x7f4b9806afb0 msgr2=0x7f4b9806b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:02.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:02.862+0000 7f4b9ea00640 1 -- 172.21.15.6:0/1988398230 shutdown_connections 2024-03-04T09:26:02.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:02.862+0000 7f4b9ea00640 1 -- 172.21.15.6:0/1988398230 wait complete. 2024-03-04T09:26:02.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:02.864+0000 7f4b9ea00640 1 Processor -- start 2024-03-04T09:26:02.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:02.864+0000 7f4b9ea00640 1 -- start start 2024-03-04T09:26:02.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:02.865+0000 7f4b9ea00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4b980b3c90 0x7f4b980af940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:02.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:02.865+0000 7f4b9ea00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4b980cdc20 0x7f4b9809aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:02.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:02.865+0000 7f4b9ea00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4b980cf5c0 0x7f4b980ae070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:02.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:02.865+0000 7f4b9ea00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b98071ca0 con 0x7f4b980b3c90 2024-03-04T09:26:02.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:02.865+0000 7f4b9ea00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b980ce060 con 0x7f4b980cf5c0 2024-03-04T09:26:02.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:02.865+0000 7f4b9ea00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b98071e20 con 0x7f4b980cdc20 2024-03-04T09:26:02.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:02.865+0000 7f4b97e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4b980b3c90 0x7f4b980af940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:02.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:02.865+0000 7f4b97e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4b980b3c90 0x7f4b980af940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:46674/0 (socket says 172.21.15.6:46674) 2024-03-04T09:26:02.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:02.865+0000 7f4b97e00640 1 -- 172.21.15.6:0/1401682504 learned_addr learned my addr 172.21.15.6:0/1401682504 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:02.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:02.866+0000 7f4b8f400640 1 --2- 172.21.15.6:0/1401682504 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4b980cdc20 0x7f4b9809aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:02.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:02.866+0000 7f4b9ca00640 1 --2- 172.21.15.6:0/1401682504 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4b980cf5c0 0x7f4b980ae070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:02.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:02.866+0000 7f4b97e00640 1 -- 172.21.15.6:0/1401682504 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4b980cdc20 msgr2=0x7f4b9809aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:02.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:02.866+0000 7f4b97e00640 1 --2- 172.21.15.6:0/1401682504 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4b980cdc20 0x7f4b9809aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:02.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:02.866+0000 7f4b97e00640 1 -- 172.21.15.6:0/1401682504 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4b980cf5c0 msgr2=0x7f4b980ae070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:02.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:02.866+0000 7f4b97e00640 1 --2- 172.21.15.6:0/1401682504 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4b980cf5c0 0x7f4b980ae070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:02.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:02.866+0000 7f4b97e00640 1 -- 172.21.15.6:0/1401682504 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b9809af50 con 0x7f4b980b3c90 2024-03-04T09:26:02.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:02.866+0000 7f4b9ca00640 1 --2- 172.21.15.6:0/1401682504 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4b980cf5c0 0x7f4b980ae070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:26:02.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:02.866+0000 7f4b97e00640 1 --2- 172.21.15.6:0/1401682504 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4b980b3c90 0x7f4b980af940 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f4b8800cbc0 tx=0x7f4b8800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:02.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:02.866+0000 7f4b94c00640 1 -- 172.21.15.6:0/1401682504 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b88017070 con 0x7f4b980b3c90 2024-03-04T09:26:02.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:02.866+0000 7f4b94c00640 1 -- 172.21.15.6:0/1401682504 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b8800ad40 con 0x7f4b980b3c90 2024-03-04T09:26:02.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:02.866+0000 7f4b94c00640 1 -- 172.21.15.6:0/1401682504 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b88012600 con 0x7f4b980b3c90 2024-03-04T09:26:02.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:02.867+0000 7f4b9ea00640 1 -- 172.21.15.6:0/1401682504 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b980affd0 con 0x7f4b980b3c90 2024-03-04T09:26:02.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:02.867+0000 7f4b9ea00640 1 -- 172.21.15.6:0/1401682504 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b980b0590 con 0x7f4b980b3c90 2024-03-04T09:26:02.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:02.868+0000 7f4b9ea00640 1 -- 172.21.15.6:0/1401682504 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b9804f3a0 con 0x7f4b980b3c90 2024-03-04T09:26:02.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:02.869+0000 7f4b94c00640 1 -- 172.21.15.6:0/1401682504 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4b8801a070 con 0x7f4b980b3c90 2024-03-04T09:26:02.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:02.870+0000 7f4b94c00640 1 --2- 172.21.15.6:0/1401682504 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f4b7406b520 0x7f4b7406d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:02.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:02.870+0000 7f4b94c00640 1 -- 172.21.15.6:0/1401682504 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f4b8808a080 con 0x7f4b980b3c90 2024-03-04T09:26:02.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:02.871+0000 7f4b8f400640 1 --2- 172.21.15.6:0/1401682504 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f4b7406b520 0x7f4b7406d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:02.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:02.872+0000 7f4b8f400640 1 --2- 172.21.15.6:0/1401682504 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f4b7406b520 0x7f4b7406d9e0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f4b80007fd0 tx=0x7f4b80009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:02.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:02.875+0000 7f4b94c00640 1 -- 172.21.15.6:0/1401682504 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4b88058800 con 0x7f4b980b3c90 2024-03-04T09:26:03.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.075+0000 7f4b9ea00640 1 -- 172.21.15.6:0/1401682504 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4b9806b710 con 0x7f4b980b3c90 2024-03-04T09:26:03.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.077+0000 7f4b94c00640 1 -- 172.21.15.6:0/1401682504 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16852 (secure 0 0 0) 0x7f4b8805c8a0 con 0x7f4b980b3c90 2024-03-04T09:26:03.075 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:26:03.075 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":22,"fsid":"edce0578-5ae4-4402-b45a-552bdef84d7c","created":"2024-03-04T09:24:54.523780+0000","modified":"2024-03-04T09:26:01.727682+0000","last_up_change":"2024-03-04T09:25:07.089049+0000","last_in_change":"2024-03-04T09:24:57.982077+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T09:25:07.935198+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-04T09:25:17.118573+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-04T09:25:17.930536+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a96574f-81f1-4bc6-8463-757520b84a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6825","nonce":1364449401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6827","nonce":1364449401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6831","nonce":1364449401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6829","nonce":1364449401}]},"public_addr":"172.21.15.6:6825/1364449401","cluster_addr":"172.21.15.6:6827/1364449401","heartbeat_back_addr":"172.21.15.6:6831/1364449401","heartbeat_front_addr":"172.21.15.6:6829/1364449401","state":["exists","up"]},{"osd":1,"uuid":"7d79b9c1-6b87-476a-bc36-7db5ee64cd90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6809","nonce":2494103078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6811","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6813","nonce":2494103078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6821","nonce":2494103078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6817","nonce":2494103078}]},"public_addr":"172.21.15.6:6809/2494103078","cluster_addr":"172.21.15.6:6813/2494103078","heartbeat_back_addr":"172.21.15.6:6821/2494103078","heartbeat_front_addr":"172.21.15.6:6817/2494103078","state":["exists","up"]},{"osd":2,"uuid":"492331b9-4ee9-40b6-bacd-85d44cd6c45a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6812","nonce":1500066681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6816","nonce":1500066681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6824","nonce":1500066681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6820","nonce":1500066681}]},"public_addr":"172.21.15.6:6812/1500066681","cluster_addr":"172.21.15.6:6816/1500066681","heartbeat_back_addr":"172.21.15.6:6824/1500066681","heartbeat_front_addr":"172.21.15.6:6820/1500066681","state":["exists","up"]},{"osd":3,"uuid":"21bb2efe-9ba7-4bdc-bc31-180cd1102e21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6801","nonce":1558806123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6803","nonce":1558806123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6807","nonce":1558806123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6805","nonce":1558806123}]},"public_addr":"172.21.15.6:6801/1558806123","cluster_addr":"172.21.15.6:6803/1558806123","heartbeat_back_addr":"172.21.15.6:6807/1558806123","heartbeat_front_addr":"172.21.15.6:6805/1558806123","state":["exists","up"]},{"osd":4,"uuid":"6e2c4b71-19d9-4737-83d9-558705d8644a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6801","nonce":1526135780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6803","nonce":1526135780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6807","nonce":1526135780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6805","nonce":1526135780}]},"public_addr":"172.21.15.146:6801/1526135780","cluster_addr":"172.21.15.146:6803/1526135780","heartbeat_back_addr":"172.21.15.146:6807/1526135780","heartbeat_front_addr":"172.21.15.146:6805/1526135780","state":["exists","up"]},{"osd":5,"uuid":"fb69c525-f1be-4b7d-ab32-6c29b0d6fddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6809","nonce":460262507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6811","nonce":460262507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6815","nonce":460262507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6813","nonce":460262507}]},"public_addr":"172.21.15.146:6809/460262507","cluster_addr":"172.21.15.146:6811/460262507","heartbeat_back_addr":"172.21.15.146:6815/460262507","heartbeat_front_addr":"172.21.15.146:6813/460262507","state":["exists","up"]},{"osd":6,"uuid":"c32e7fed-370d-4401-8ce0-6a4335b0d168","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6823","nonce":1392897493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6827","nonce":1392897493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6831","nonce":1392897493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6829","nonce":1392897493}]},"public_addr":"172.21.15.146:6823/1392897493","cluster_addr":"172.21.15.146:6827/1392897493","heartbeat_back_addr":"172.21.15.146:6831/1392897493","heartbeat_front_addr":"172.21.15.146:6829/1392897493","state":["exists","up"]},{"osd":7,"uuid":"75a79b44-fa5f-426d-a7d3-71f21f78835c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6817","nonce":3268323156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6819","nonce":3268323156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6825","nonce":3268323156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6822","nonce":3268323156}]},"public_addr":"172.21.15.146:6817/3268323156","cluster_addr":"172.21.15.146:6819/3268323156","heartbeat_back_addr":"172.21.15.146:6825/3268323156","heartbeat_front_addr":"172.21.15.146:6822/3268323156","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.310437+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.295850+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.294674+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.526855+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.300023+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.595160+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.346046+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.558741+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.7","mappings":[{"from":3,"to":2}]},{"pgid":"3.f","mappings":[{"from":7,"to":2}]},{"pgid":"3.21","mappings":[{"from":5,"to":1}]},{"pgid":"3.22","mappings":[{"from":5,"to":2}]},{"pgid":"3.28","mappings":[{"from":3,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T09:26:03.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.079+0000 7f4b9ea00640 1 -- 172.21.15.6:0/1401682504 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f4b7406b520 msgr2=0x7f4b7406d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:03.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.079+0000 7f4b9ea00640 1 --2- 172.21.15.6:0/1401682504 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f4b7406b520 0x7f4b7406d9e0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f4b80007fd0 tx=0x7f4b80009040 comp rx=0 tx=0).stop 2024-03-04T09:26:03.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.079+0000 7f4b9ea00640 1 -- 172.21.15.6:0/1401682504 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4b980b3c90 msgr2=0x7f4b980af940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:03.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.079+0000 7f4b9ea00640 1 --2- 172.21.15.6:0/1401682504 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4b980b3c90 0x7f4b980af940 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f4b8800cbc0 tx=0x7f4b8800a430 comp rx=0 tx=0).stop 2024-03-04T09:26:03.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.080+0000 7f4b9ea00640 1 -- 172.21.15.6:0/1401682504 shutdown_connections 2024-03-04T09:26:03.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.080+0000 7f4b9ea00640 1 --2- 172.21.15.6:0/1401682504 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4b980cf5c0 0x7f4b980ae070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:03.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.080+0000 7f4b9ea00640 1 --2- 172.21.15.6:0/1401682504 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f4b7406b520 0x7f4b7406d9e0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:03.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.080+0000 7f4b9ea00640 1 --2- 172.21.15.6:0/1401682504 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4b980cdc20 0x7f4b9809aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:03.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.080+0000 7f4b9ea00640 1 --2- 172.21.15.6:0/1401682504 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4b980b3c90 0x7f4b980af940 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:03.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.080+0000 7f4b9ea00640 1 -- 172.21.15.6:0/1401682504 >> 172.21.15.6:0/1401682504 conn(0x7f4b9806afb0 msgr2=0x7f4b980727b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:03.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.080+0000 7f4b9ea00640 1 -- 172.21.15.6:0/1401682504 shutdown_connections 2024-03-04T09:26:03.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.080+0000 7f4b9ea00640 1 -- 172.21.15.6:0/1401682504 wait complete. 2024-03-04T09:26:03.097 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T09:25:07.935198+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-04T09:25:17.118573+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-04T09:25:17.930536+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-04T09:26:03.098 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-04T09:26:03.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.326+0000 7f3210a00640 1 Processor -- start 2024-03-04T09:26:03.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.327+0000 7f3210a00640 1 -- start start 2024-03-04T09:26:03.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.328+0000 7f3210a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f320c147590 0x7f320c071480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:03.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.328+0000 7f3210a00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f320c06d8d0 0x7f320c0b1b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:03.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.328+0000 7f3210a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f320c0b2080 0x7f320c0ad880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:03.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.328+0000 7f3210a00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f320c0b24f0 con 0x7f320c0b2080 2024-03-04T09:26:03.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.328+0000 7f3210a00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f320c147970 con 0x7f320c06d8d0 2024-03-04T09:26:03.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.328+0000 7f3210a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f320c0b31b0 con 0x7f320c147590 2024-03-04T09:26:03.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.329+0000 7f320b200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f320c0b2080 0x7f320c0ad880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:03.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.329+0000 7f320b200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f320c0b2080 0x7f320c0ad880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:46690/0 (socket says 172.21.15.6:46690) 2024-03-04T09:26:03.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.329+0000 7f320b200640 1 -- 172.21.15.6:0/2238391381 learned_addr learned my addr 172.21.15.6:0/2238391381 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:03.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.329+0000 7f3201e00640 1 --2- 172.21.15.6:0/2238391381 >> v2:172.21.15.146:3300/0 conn(0x7f320c06d8d0 0x7f320c0b1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:03.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.329+0000 7f320a800640 1 --2- 172.21.15.6:0/2238391381 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f320c147590 0x7f320c071480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:03.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.329+0000 7f320b200640 1 -- 172.21.15.6:0/2238391381 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f320c147590 msgr2=0x7f320c071480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:03.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.329+0000 7f320b200640 1 --2- 172.21.15.6:0/2238391381 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f320c147590 0x7f320c071480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:03.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.329+0000 7f320b200640 1 -- 172.21.15.6:0/2238391381 >> v2:172.21.15.146:3300/0 conn(0x7f320c06d8d0 msgr2=0x7f320c0b1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:03.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.329+0000 7f320b200640 1 --2- 172.21.15.6:0/2238391381 >> v2:172.21.15.146:3300/0 conn(0x7f320c06d8d0 0x7f320c0b1b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:03.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.329+0000 7f320b200640 1 -- 172.21.15.6:0/2238391381 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f320c0719c0 con 0x7f320c0b2080 2024-03-04T09:26:03.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.331+0000 7f320b200640 1 --2- 172.21.15.6:0/2238391381 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f320c0b2080 0x7f320c0ad880 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f320c0b26a0 tx=0x7f320402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc0a3724b52effd5 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:03.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.331+0000 7f3209e00640 1 -- 172.21.15.6:0/2238391381 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f320402ca80 con 0x7f320c0b2080 2024-03-04T09:26:03.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.331+0000 7f3209e00640 1 -- 172.21.15.6:0/2238391381 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f320402cc20 con 0x7f320c0b2080 2024-03-04T09:26:03.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.331+0000 7f3209e00640 1 -- 172.21.15.6:0/2238391381 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32040343f0 con 0x7f320c0b2080 2024-03-04T09:26:03.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.332+0000 7f3210a00640 1 -- 172.21.15.6:0/2238391381 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f320c0b2080 msgr2=0x7f320c0ad880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:03.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.332+0000 7f3210a00640 1 --2- 172.21.15.6:0/2238391381 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f320c0b2080 0x7f320c0ad880 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f320c0b26a0 tx=0x7f320402bee0 comp rx=0 tx=0).stop 2024-03-04T09:26:03.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.332+0000 7f3210a00640 1 -- 172.21.15.6:0/2238391381 shutdown_connections 2024-03-04T09:26:03.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.332+0000 7f3210a00640 1 --2- 172.21.15.6:0/2238391381 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f320c0b2080 0x7f320c0ad880 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:03.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.332+0000 7f3210a00640 1 --2- 172.21.15.6:0/2238391381 >> v2:172.21.15.146:3300/0 conn(0x7f320c06d8d0 0x7f320c0b1b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:03.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.332+0000 7f3210a00640 1 --2- 172.21.15.6:0/2238391381 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f320c147590 0x7f320c071480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:03.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.332+0000 7f3210a00640 1 -- 172.21.15.6:0/2238391381 >> 172.21.15.6:0/2238391381 conn(0x7f320c06b3a0 msgr2=0x7f320c06b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:03.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.332+0000 7f3210a00640 1 -- 172.21.15.6:0/2238391381 shutdown_connections 2024-03-04T09:26:03.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.332+0000 7f3210a00640 1 -- 172.21.15.6:0/2238391381 wait complete. 2024-03-04T09:26:03.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.334+0000 7f3210a00640 1 Processor -- start 2024-03-04T09:26:03.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.334+0000 7f3210a00640 1 -- start start 2024-03-04T09:26:03.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.335+0000 7f3210a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f320c06d8d0 0x7f320c13f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:03.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.335+0000 7f3210a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f320c0b2080 0x7f320c143f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:03.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.335+0000 7f3210a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f320c147590 0x7f320c144470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:03.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.335+0000 7f3210a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f320c0b2e40 con 0x7f320c06d8d0 2024-03-04T09:26:03.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.335+0000 7f3210a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f320c06e120 con 0x7f320c0b2080 2024-03-04T09:26:03.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.335+0000 7f3210a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f320c0b2b50 con 0x7f320c147590 2024-03-04T09:26:03.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.335+0000 7f320a800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f320c06d8d0 0x7f320c13f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:03.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.335+0000 7f320a800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f320c06d8d0 0x7f320c13f030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:46698/0 (socket says 172.21.15.6:46698) 2024-03-04T09:26:03.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.335+0000 7f320a800640 1 -- 172.21.15.6:0/668021539 learned_addr learned my addr 172.21.15.6:0/668021539 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:03.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.335+0000 7f3201e00640 1 --2- 172.21.15.6:0/668021539 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f320c0b2080 0x7f320c143f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:03.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.335+0000 7f320a800640 1 -- 172.21.15.6:0/668021539 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f320c147590 msgr2=0x7f320c144470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:03.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.335+0000 7f320a800640 1 --2- 172.21.15.6:0/668021539 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f320c147590 0x7f320c144470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:03.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.335+0000 7f320a800640 1 -- 172.21.15.6:0/668021539 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f320c0b2080 msgr2=0x7f320c143f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:03.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.335+0000 7f320a800640 1 --2- 172.21.15.6:0/668021539 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f320c0b2080 0x7f320c143f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:03.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.335+0000 7f320a800640 1 -- 172.21.15.6:0/668021539 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f320c13f9a0 con 0x7f320c06d8d0 2024-03-04T09:26:03.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.336+0000 7f3201e00640 1 --2- 172.21.15.6:0/668021539 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f320c0b2080 0x7f320c143f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:26:03.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.336+0000 7f320a800640 1 --2- 172.21.15.6:0/668021539 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f320c06d8d0 0x7f320c13f030 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f31fc00d800 tx=0x7f31fc00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:03.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.336+0000 7f3203400640 1 -- 172.21.15.6:0/668021539 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31fc00b6c0 con 0x7f320c06d8d0 2024-03-04T09:26:03.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.336+0000 7f3203400640 1 -- 172.21.15.6:0/668021539 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31fc019070 con 0x7f320c06d8d0 2024-03-04T09:26:03.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.336+0000 7f3203400640 1 -- 172.21.15.6:0/668021539 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31fc00ba80 con 0x7f320c06d8d0 2024-03-04T09:26:03.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.336+0000 7f3210a00640 1 -- 172.21.15.6:0/668021539 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f320c13fc80 con 0x7f320c06d8d0 2024-03-04T09:26:03.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.336+0000 7f3210a00640 1 -- 172.21.15.6:0/668021539 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f320c140240 con 0x7f320c06d8d0 2024-03-04T09:26:03.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.338+0000 7f3210a00640 1 -- 172.21.15.6:0/668021539 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f320c04f330 con 0x7f320c06d8d0 2024-03-04T09:26:03.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.338+0000 7f3203400640 1 -- 172.21.15.6:0/668021539 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f31fc00bc20 con 0x7f320c06d8d0 2024-03-04T09:26:03.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.339+0000 7f3203400640 1 --2- 172.21.15.6:0/668021539 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f31e806b500 0x7f31e806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:03.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.339+0000 7f3203400640 1 -- 172.21.15.6:0/668021539 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f31fc08a6f0 con 0x7f320c06d8d0 2024-03-04T09:26:03.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.340+0000 7f3201e00640 1 --2- 172.21.15.6:0/668021539 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f31e806b500 0x7f31e806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:03.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.341+0000 7f3201e00640 1 --2- 172.21.15.6:0/668021539 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f31e806b500 0x7f31e806d9c0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f31f4006fd0 tx=0x7f31f4008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:03.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.344+0000 7f3203400640 1 -- 172.21.15.6:0/668021539 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f31fc059440 con 0x7f320c06d8d0 2024-03-04T09:26:03.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.543+0000 7f3210a00640 1 -- 172.21.15.6:0/668021539 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f320c071cf0 con 0x7f320c06d8d0 2024-03-04T09:26:03.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.544+0000 7f3203400640 1 -- 172.21.15.6:0/668021539 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7f31fc058d40 con 0x7f320c06d8d0 2024-03-04T09:26:03.542 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-04T09:26:03.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.547+0000 7f3210a00640 1 -- 172.21.15.6:0/668021539 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f31e806b500 msgr2=0x7f31e806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:03.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.547+0000 7f3210a00640 1 --2- 172.21.15.6:0/668021539 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f31e806b500 0x7f31e806d9c0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f31f4006fd0 tx=0x7f31f4008040 comp rx=0 tx=0).stop 2024-03-04T09:26:03.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.547+0000 7f3210a00640 1 -- 172.21.15.6:0/668021539 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f320c06d8d0 msgr2=0x7f320c13f030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:03.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.547+0000 7f3210a00640 1 --2- 172.21.15.6:0/668021539 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f320c06d8d0 0x7f320c13f030 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f31fc00d800 tx=0x7f31fc00dcc0 comp rx=0 tx=0).stop 2024-03-04T09:26:03.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.547+0000 7f3210a00640 1 -- 172.21.15.6:0/668021539 shutdown_connections 2024-03-04T09:26:03.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.547+0000 7f3210a00640 1 --2- 172.21.15.6:0/668021539 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f31e806b500 0x7f31e806d9c0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:03.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.547+0000 7f3210a00640 1 --2- 172.21.15.6:0/668021539 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f320c147590 0x7f320c144470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:03.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.547+0000 7f3210a00640 1 --2- 172.21.15.6:0/668021539 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f320c0b2080 0x7f320c143f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:03.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.547+0000 7f3210a00640 1 --2- 172.21.15.6:0/668021539 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f320c06d8d0 0x7f320c13f030 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:03.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.547+0000 7f3210a00640 1 -- 172.21.15.6:0/668021539 >> 172.21.15.6:0/668021539 conn(0x7f320c06b3a0 msgr2=0x7f320c0ad880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:03.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.547+0000 7f3210a00640 1 -- 172.21.15.6:0/668021539 shutdown_connections 2024-03-04T09:26:03.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.547+0000 7f3210a00640 1 -- 172.21.15.6:0/668021539 wait complete. 2024-03-04T09:26:03.562 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-04T09:26:03.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.784+0000 7f02c5600640 1 Processor -- start 2024-03-04T09:26:03.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.785+0000 7f02c5600640 1 -- start start 2024-03-04T09:26:03.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.786+0000 7f02c5600640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f02c01615b0 0x7f02c00d47f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:03.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.786+0000 7f02c5600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f02c00d29e0 0x7f02c00cb130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:03.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.786+0000 7f02c5600640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02c00cf670 con 0x7f02c00d1a80 2024-03-04T09:26:03.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.786+0000 7f02c5600640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02c0161990 con 0x7f02c01615b0 2024-03-04T09:26:03.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.786+0000 7f02c5600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02c00d1ff0 con 0x7f02c00d29e0 2024-03-04T09:26:03.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.787+0000 7f02bf400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f02c00d29e0 0x7f02c00cb130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:03.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.787+0000 7f02be000640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f02c00d1a80 0x7f02c00d22d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.6:57530/0 (socket says 172.21.15.6:57530) 2024-03-04T09:26:03.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.787+0000 7f02be000640 1 -- 172.21.15.6:0/389258791 learned_addr learned my addr 172.21.15.6:0/389258791 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:26:03.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.787+0000 7f02bea00640 1 --2- 172.21.15.6:0/389258791 >> v2:172.21.15.146:3300/0 conn(0x7f02c01615b0 0x7f02c00d47f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:03.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.788+0000 7f02bea00640 1 -- 172.21.15.6:0/389258791 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f02c00d29e0 msgr2=0x7f02c00cb130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:03.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.788+0000 7f02bea00640 1 --2- 172.21.15.6:0/389258791 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f02c00d29e0 0x7f02c00cb130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:03.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.788+0000 7f02bea00640 1 -- 172.21.15.6:0/389258791 >> v1:172.21.15.146:6789/0 conn(0x7f02c00d1a80 legacy=0x7f02c00d22d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:03.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.788+0000 7f02bea00640 1 -- 172.21.15.6:0/389258791 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02c00cf910 con 0x7f02c01615b0 2024-03-04T09:26:03.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.788+0000 7f02bf400640 1 --2- 172.21.15.6:0/389258791 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f02c00d29e0 0x7f02c00cb130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T09:26:03.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.789+0000 7f02bea00640 1 --2- 172.21.15.6:0/389258791 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f02c01615b0 0x7f02c00d47f0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f02b0009d90 tx=0x7f02b002bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2732e6fae0e99f6a server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:03.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.789+0000 7f02bd600640 1 -- 172.21.15.6:0/389258791 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02b0039070 con 0x7f02c01615b0 2024-03-04T09:26:03.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.790+0000 7f02bd600640 1 -- 172.21.15.6:0/389258791 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02b002cc50 con 0x7f02c01615b0 2024-03-04T09:26:03.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.790+0000 7f02bd600640 1 -- 172.21.15.6:0/389258791 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02b0034440 con 0x7f02c01615b0 2024-03-04T09:26:03.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.790+0000 7f02c5600640 1 -- 172.21.15.6:0/389258791 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f02c01615b0 msgr2=0x7f02c00d47f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:03.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.790+0000 7f02c5600640 1 --2- 172.21.15.6:0/389258791 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f02c01615b0 0x7f02c00d47f0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f02b0009d90 tx=0x7f02b002bea0 comp rx=0 tx=0).stop 2024-03-04T09:26:03.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.790+0000 7f02c5600640 1 -- 172.21.15.6:0/389258791 shutdown_connections 2024-03-04T09:26:03.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.790+0000 7f02c5600640 1 --2- 172.21.15.6:0/389258791 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f02c00d29e0 0x7f02c00cb130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:03.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.790+0000 7f02c5600640 1 --2- 172.21.15.6:0/389258791 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f02c01615b0 0x7f02c00d47f0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:03.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.790+0000 7f02c5600640 1 -- 172.21.15.6:0/389258791 >> 172.21.15.6:0/389258791 conn(0x7f02c009d180 msgr2=0x7f02c00d3eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:03.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.791+0000 7f02c5600640 1 -- 172.21.15.6:0/389258791 shutdown_connections 2024-03-04T09:26:03.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.791+0000 7f02c5600640 1 -- 172.21.15.6:0/389258791 wait complete. 2024-03-04T09:26:03.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.793+0000 7f02c5600640 1 Processor -- start 2024-03-04T09:26:03.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.793+0000 7f02c5600640 1 -- start start 2024-03-04T09:26:03.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.794+0000 7f02c5600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f02c00d1a80 0x7f02c00b7760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:03.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.794+0000 7f02c5600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f02c00d29e0 0x7f02c0154f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:03.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.794+0000 7f02c5600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f02c01615b0 0x7f02c00b9120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:03.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.794+0000 7f02c5600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02c00cb670 con 0x7f02c01615b0 2024-03-04T09:26:03.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.794+0000 7f02c5600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02c00d4d30 con 0x7f02c00d29e0 2024-03-04T09:26:03.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.794+0000 7f02c5600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02c00cb7f0 con 0x7f02c00d1a80 2024-03-04T09:26:03.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.795+0000 7f02bf400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f02c01615b0 0x7f02c00b9120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:03.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.795+0000 7f02bf400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f02c01615b0 0x7f02c00b9120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:46702/0 (socket says 172.21.15.6:46702) 2024-03-04T09:26:03.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.795+0000 7f02bf400640 1 -- 172.21.15.6:0/3606899487 learned_addr learned my addr 172.21.15.6:0/3606899487 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:03.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.795+0000 7f02bea00640 1 --2- 172.21.15.6:0/3606899487 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f02c00d1a80 0x7f02c00b7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:03.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.795+0000 7f02be000640 1 --2- 172.21.15.6:0/3606899487 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f02c00d29e0 0x7f02c0154f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:03.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.795+0000 7f02bf400640 1 -- 172.21.15.6:0/3606899487 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f02c00d1a80 msgr2=0x7f02c00b7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:03.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.795+0000 7f02bf400640 1 --2- 172.21.15.6:0/3606899487 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f02c00d1a80 0x7f02c00b7760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:03.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.795+0000 7f02bf400640 1 -- 172.21.15.6:0/3606899487 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f02c00d29e0 msgr2=0x7f02c0154f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:03.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.795+0000 7f02bf400640 1 --2- 172.21.15.6:0/3606899487 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f02c00d29e0 0x7f02c0154f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:03.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.795+0000 7f02bf400640 1 -- 172.21.15.6:0/3606899487 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02c00b7ed0 con 0x7f02c01615b0 2024-03-04T09:26:03.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.796+0000 7f02bf400640 1 --2- 172.21.15.6:0/3606899487 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f02c01615b0 0x7f02c00b9120 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f02b800e930 tx=0x7f02b800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:03.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.796+0000 7f02b6a00640 1 -- 172.21.15.6:0/3606899487 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02b8017070 con 0x7f02c01615b0 2024-03-04T09:26:03.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.796+0000 7f02b6a00640 1 -- 172.21.15.6:0/3606899487 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02b800ca00 con 0x7f02c01615b0 2024-03-04T09:26:03.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.796+0000 7f02c5600640 1 -- 172.21.15.6:0/3606899487 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02c00b81b0 con 0x7f02c01615b0 2024-03-04T09:26:03.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.796+0000 7f02b6a00640 1 -- 172.21.15.6:0/3606899487 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02b800cdd0 con 0x7f02c01615b0 2024-03-04T09:26:03.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.796+0000 7f02c5600640 1 -- 172.21.15.6:0/3606899487 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02c00c8450 con 0x7f02c01615b0 2024-03-04T09:26:03.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.797+0000 7f02c5600640 1 -- 172.21.15.6:0/3606899487 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02c00c8650 con 0x7f02c01615b0 2024-03-04T09:26:03.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.798+0000 7f02b6a00640 1 -- 172.21.15.6:0/3606899487 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f02b8013470 con 0x7f02c01615b0 2024-03-04T09:26:03.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.799+0000 7f02b6a00640 1 --2- 172.21.15.6:0/3606899487 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f029c06b4d0 0x7f029c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:03.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.799+0000 7f02b6a00640 1 -- 172.21.15.6:0/3606899487 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f02b808a1c0 con 0x7f02c01615b0 2024-03-04T09:26:03.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.800+0000 7f02bea00640 1 --2- 172.21.15.6:0/3606899487 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f029c06b4d0 0x7f029c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:03.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.801+0000 7f02bea00640 1 --2- 172.21.15.6:0/3606899487 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f029c06b4d0 0x7f029c06d990 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f02b0036000 tx=0x7f02b0030000 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:03.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:03.806+0000 7f02b6a00640 1 -- 172.21.15.6:0/3606899487 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f02b8059070 con 0x7f02c01615b0 2024-03-04T09:26:04.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.007+0000 7f02c5600640 1 -- 172.21.15.6:0/3606899487 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f02c004f330 con 0x7f02c01615b0 2024-03-04T09:26:04.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.008+0000 7f02b6a00640 1 -- 172.21.15.6:0/3606899487 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7f02b8058970 con 0x7f02c01615b0 2024-03-04T09:26:04.006 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-03-04T09:26:04.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.011+0000 7f02c5600640 1 -- 172.21.15.6:0/3606899487 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f029c06b4d0 msgr2=0x7f029c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:04.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.011+0000 7f02c5600640 1 --2- 172.21.15.6:0/3606899487 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f029c06b4d0 0x7f029c06d990 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f02b0036000 tx=0x7f02b0030000 comp rx=0 tx=0).stop 2024-03-04T09:26:04.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.011+0000 7f02c5600640 1 -- 172.21.15.6:0/3606899487 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f02c01615b0 msgr2=0x7f02c00b9120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:04.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.011+0000 7f02c5600640 1 --2- 172.21.15.6:0/3606899487 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f02c01615b0 0x7f02c00b9120 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f02b800e930 tx=0x7f02b800edf0 comp rx=0 tx=0).stop 2024-03-04T09:26:04.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.011+0000 7f02c5600640 1 -- 172.21.15.6:0/3606899487 shutdown_connections 2024-03-04T09:26:04.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.011+0000 7f02c5600640 1 --2- 172.21.15.6:0/3606899487 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f02c01615b0 0x7f02c00b9120 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:04.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.011+0000 7f02c5600640 1 --2- 172.21.15.6:0/3606899487 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f02c00d29e0 0x7f02c0154f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:04.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.011+0000 7f02c5600640 1 --2- 172.21.15.6:0/3606899487 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f029c06b4d0 0x7f029c06d990 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:04.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.011+0000 7f02c5600640 1 --2- 172.21.15.6:0/3606899487 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f02c00d1a80 0x7f02c00b7760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:04.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.011+0000 7f02c5600640 1 -- 172.21.15.6:0/3606899487 >> 172.21.15.6:0/3606899487 conn(0x7f02c009d180 msgr2=0x7f02c00ce9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:04.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.011+0000 7f02c5600640 1 -- 172.21.15.6:0/3606899487 shutdown_connections 2024-03-04T09:26:04.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.011+0000 7f02c5600640 1 -- 172.21.15.6:0/3606899487 wait complete. 2024-03-04T09:26:04.028 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-04T09:26:04.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.248+0000 7fa861800640 1 Processor -- start 2024-03-04T09:26:04.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.248+0000 7fa861800640 1 -- start start 2024-03-04T09:26:04.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.250+0000 7fa861800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa85c0b2250 0x7fa85c168950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:04.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.250+0000 7fa861800640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa85c06de40 con 0x7fa85c06d8d0 2024-03-04T09:26:04.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.250+0000 7fa861800640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa85c06e120 con 0x7fa85c147590 2024-03-04T09:26:04.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.250+0000 7fa861800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa85c147970 con 0x7fa85c0b2250 2024-03-04T09:26:04.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.251+0000 7fa860e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa85c0b2250 0x7fa85c168950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:04.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.251+0000 7fa860e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa85c0b2250 0x7fa85c168950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:46716/0 (socket says 172.21.15.6:46716) 2024-03-04T09:26:04.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.251+0000 7fa860e00640 1 -- 172.21.15.6:0/739752867 learned_addr learned my addr 172.21.15.6:0/739752867 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:04.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.251+0000 7fa860e00640 1 -- 172.21.15.6:0/739752867 >> v1:172.21.15.146:6789/0 conn(0x7fa85c147590 legacy=0x7fa85c071480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:04.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.251+0000 7fa860e00640 1 -- 172.21.15.6:0/739752867 >> v1:172.21.15.6:6789/0 conn(0x7fa85c06d8d0 legacy=0x7fa85c0b1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:04.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.251+0000 7fa860e00640 1 -- 172.21.15.6:0/739752867 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa85c0b2e40 con 0x7fa85c0b2250 2024-03-04T09:26:04.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.252+0000 7fa860e00640 1 --2- 172.21.15.6:0/739752867 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa85c0b2250 0x7fa85c168950 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fa854009e50 tx=0x7fa85402c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1533900a839d59ef server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:04.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.253+0000 7fa85aa00640 1 -- 172.21.15.6:0/739752867 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa85402cba0 con 0x7fa85c0b2250 2024-03-04T09:26:04.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.253+0000 7fa85aa00640 1 -- 172.21.15.6:0/739752867 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa85402cd40 con 0x7fa85c0b2250 2024-03-04T09:26:04.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.253+0000 7fa85aa00640 1 -- 172.21.15.6:0/739752867 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa854034610 con 0x7fa85c0b2250 2024-03-04T09:26:04.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.253+0000 7fa861800640 1 -- 172.21.15.6:0/739752867 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa85c0b2250 msgr2=0x7fa85c168950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:04.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.253+0000 7fa861800640 1 --2- 172.21.15.6:0/739752867 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa85c0b2250 0x7fa85c168950 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fa854009e50 tx=0x7fa85402c000 comp rx=0 tx=0).stop 2024-03-04T09:26:04.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.253+0000 7fa861800640 1 -- 172.21.15.6:0/739752867 shutdown_connections 2024-03-04T09:26:04.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.253+0000 7fa861800640 1 --2- 172.21.15.6:0/739752867 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa85c0b2250 0x7fa85c168950 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:04.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.253+0000 7fa861800640 1 -- 172.21.15.6:0/739752867 >> 172.21.15.6:0/739752867 conn(0x7fa85c06b3a0 msgr2=0x7fa85c06b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:04.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.253+0000 7fa861800640 1 -- 172.21.15.6:0/739752867 shutdown_connections 2024-03-04T09:26:04.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.254+0000 7fa861800640 1 -- 172.21.15.6:0/739752867 wait complete. 2024-03-04T09:26:04.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.255+0000 7fa861800640 1 Processor -- start 2024-03-04T09:26:04.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.255+0000 7fa861800640 1 -- start start 2024-03-04T09:26:04.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.256+0000 7fa861800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa85c06d8d0 0x7fa85c0c7d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:04.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.256+0000 7fa861800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa85c0b2250 0x7fa85c132d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:04.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.256+0000 7fa861800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa85c147590 0x7fa85c0c7060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:04.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.256+0000 7fa861800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa85c0b4650 con 0x7fa85c147590 2024-03-04T09:26:04.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.256+0000 7fa861800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa85c0b31b0 con 0x7fa85c0b2250 2024-03-04T09:26:04.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.256+0000 7fa861800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa85c0b47d0 con 0x7fa85c06d8d0 2024-03-04T09:26:04.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.257+0000 7fa860e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa85c147590 0x7fa85c0c7060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:04.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.257+0000 7fa860e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa85c147590 0x7fa85c0c7060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:46724/0 (socket says 172.21.15.6:46724) 2024-03-04T09:26:04.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.257+0000 7fa860e00640 1 -- 172.21.15.6:0/1794781056 learned_addr learned my addr 172.21.15.6:0/1794781056 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:04.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.257+0000 7fa85be00640 1 --2- 172.21.15.6:0/1794781056 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa85c06d8d0 0x7fa85c0c7d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:04.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.257+0000 7fa85b400640 1 --2- 172.21.15.6:0/1794781056 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa85c0b2250 0x7fa85c132d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:04.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.257+0000 7fa860e00640 1 -- 172.21.15.6:0/1794781056 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa85c06d8d0 msgr2=0x7fa85c0c7d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:04.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.257+0000 7fa860e00640 1 --2- 172.21.15.6:0/1794781056 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa85c06d8d0 0x7fa85c0c7d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:04.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.257+0000 7fa860e00640 1 -- 172.21.15.6:0/1794781056 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa85c0b2250 msgr2=0x7fa85c132d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:04.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.257+0000 7fa860e00640 1 --2- 172.21.15.6:0/1794781056 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa85c0b2250 0x7fa85c132d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:04.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.257+0000 7fa860e00640 1 -- 172.21.15.6:0/1794781056 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa85c0c5fc0 con 0x7fa85c147590 2024-03-04T09:26:04.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.257+0000 7fa85be00640 1 --2- 172.21.15.6:0/1794781056 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa85c06d8d0 0x7fa85c0c7d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T09:26:04.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.257+0000 7fa85b400640 1 --2- 172.21.15.6:0/1794781056 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa85c0b2250 0x7fa85c132d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:26:04.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.258+0000 7fa860e00640 1 --2- 172.21.15.6:0/1794781056 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa85c147590 0x7fa85c0c7060 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fa854036040 tx=0x7fa854004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:04.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.258+0000 7fa853e00640 1 -- 172.21.15.6:0/1794781056 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa854043020 con 0x7fa85c147590 2024-03-04T09:26:04.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.258+0000 7fa853e00640 1 -- 172.21.15.6:0/1794781056 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa85403e450 con 0x7fa85c147590 2024-03-04T09:26:04.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.258+0000 7fa853e00640 1 -- 172.21.15.6:0/1794781056 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa854009510 con 0x7fa85c147590 2024-03-04T09:26:04.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.258+0000 7fa861800640 1 -- 172.21.15.6:0/1794781056 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa85c0c62a0 con 0x7fa85c147590 2024-03-04T09:26:04.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.258+0000 7fa861800640 1 -- 172.21.15.6:0/1794781056 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa85c0c6860 con 0x7fa85c147590 2024-03-04T09:26:04.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.259+0000 7fa861800640 1 -- 172.21.15.6:0/1794781056 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa85c04f330 con 0x7fa85c147590 2024-03-04T09:26:04.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.260+0000 7fa853e00640 1 -- 172.21.15.6:0/1794781056 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa854046020 con 0x7fa85c147590 2024-03-04T09:26:04.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.261+0000 7fa853e00640 1 --2- 172.21.15.6:0/1794781056 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa83c06b4d0 0x7fa83c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:04.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.261+0000 7fa853e00640 1 -- 172.21.15.6:0/1794781056 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7fa8540acc70 con 0x7fa85c147590 2024-03-04T09:26:04.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.262+0000 7fa85be00640 1 --2- 172.21.15.6:0/1794781056 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa83c06b4d0 0x7fa83c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:04.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.263+0000 7fa85be00640 1 --2- 172.21.15.6:0/1794781056 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa83c06b4d0 0x7fa83c06d990 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fa84c001660 tx=0x7fa84c009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:04.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.266+0000 7fa853e00640 1 -- 172.21.15.6:0/1794781056 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa85404d030 con 0x7fa85c147590 2024-03-04T09:26:04.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.464+0000 7fa861800640 1 -- 172.21.15.6:0/1794781056 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fa85c074ab0 con 0x7fa85c147590 2024-03-04T09:26:04.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.465+0000 7fa853e00640 1 -- 172.21.15.6:0/1794781056 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7fa85407b9c0 con 0x7fa85c147590 2024-03-04T09:26:04.463 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 64 2024-03-04T09:26:04.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.467+0000 7fa861800640 1 -- 172.21.15.6:0/1794781056 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa83c06b4d0 msgr2=0x7fa83c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:04.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.467+0000 7fa861800640 1 --2- 172.21.15.6:0/1794781056 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa83c06b4d0 0x7fa83c06d990 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fa84c001660 tx=0x7fa84c009040 comp rx=0 tx=0).stop 2024-03-04T09:26:04.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.467+0000 7fa861800640 1 -- 172.21.15.6:0/1794781056 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa85c147590 msgr2=0x7fa85c0c7060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:04.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.467+0000 7fa861800640 1 --2- 172.21.15.6:0/1794781056 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa85c147590 0x7fa85c0c7060 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fa854036040 tx=0x7fa854004a40 comp rx=0 tx=0).stop 2024-03-04T09:26:04.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.468+0000 7fa861800640 1 -- 172.21.15.6:0/1794781056 shutdown_connections 2024-03-04T09:26:04.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.468+0000 7fa861800640 1 --2- 172.21.15.6:0/1794781056 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa85c147590 0x7fa85c0c7060 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:04.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.468+0000 7fa861800640 1 --2- 172.21.15.6:0/1794781056 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa85c0b2250 0x7fa85c132d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:04.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.468+0000 7fa861800640 1 --2- 172.21.15.6:0/1794781056 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa83c06b4d0 0x7fa83c06d990 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:04.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.468+0000 7fa861800640 1 --2- 172.21.15.6:0/1794781056 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa85c06d8d0 0x7fa85c0c7d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:04.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.468+0000 7fa861800640 1 -- 172.21.15.6:0/1794781056 >> 172.21.15.6:0/1794781056 conn(0x7fa85c06b3a0 msgr2=0x7fa85c0af6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:04.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.468+0000 7fa861800640 1 -- 172.21.15.6:0/1794781056 shutdown_connections 2024-03-04T09:26:04.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.468+0000 7fa861800640 1 -- 172.21.15.6:0/1794781056 wait complete. 2024-03-04T09:26:04.485 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-04T09:26:04.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.701+0000 7f957ce00640 1 Processor -- start 2024-03-04T09:26:04.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.701+0000 7f957ce00640 1 -- start start 2024-03-04T09:26:04.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.703+0000 7f957ce00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f9578070ee0 0x7f9578162c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:04.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.703+0000 7f957ce00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f95780712c0 0x7f957806e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:04.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.703+0000 7f957ce00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95780ce090 con 0x7f9578072da0 2024-03-04T09:26:04.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.703+0000 7f957ce00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95780ceca0 con 0x7f9578070ee0 2024-03-04T09:26:04.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.703+0000 7f957ce00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95780ce370 con 0x7f95780712c0 2024-03-04T09:26:04.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.703+0000 7f9575e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f9578070ee0 0x7f9578162c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:04.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.703+0000 7f9575e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f9578070ee0 0x7f9578162c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:46728/0 (socket says 172.21.15.6:46728) 2024-03-04T09:26:04.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.703+0000 7f9575e00640 1 -- 172.21.15.6:0/4059000974 learned_addr learned my addr 172.21.15.6:0/4059000974 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:04.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.703+0000 7f9576800640 1 --2- 172.21.15.6:0/4059000974 >> v2:172.21.15.146:3300/0 conn(0x7f95780712c0 0x7f957806e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:04.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.704+0000 7f9575e00640 1 -- 172.21.15.6:0/4059000974 >> v2:172.21.15.146:3300/0 conn(0x7f95780712c0 msgr2=0x7f957806e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:04.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.704+0000 7f9575e00640 1 --2- 172.21.15.6:0/4059000974 >> v2:172.21.15.146:3300/0 conn(0x7f95780712c0 0x7f957806e7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:04.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.704+0000 7f9575e00640 1 -- 172.21.15.6:0/4059000974 >> v1:172.21.15.146:6789/0 conn(0x7f9578072da0 legacy=0x7f957806e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:04.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.704+0000 7f9575e00640 1 -- 172.21.15.6:0/4059000974 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95780ce650 con 0x7f9578070ee0 2024-03-04T09:26:04.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.705+0000 7f9575e00640 1 --2- 172.21.15.6:0/4059000974 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9578070ee0 0x7f9578162c40 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f957806f070 tx=0x7f956002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33e6b3ea0ec1782d server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:04.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.705+0000 7f9574a00640 1 -- 172.21.15.6:0/4059000974 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9560035020 con 0x7f9578070ee0 2024-03-04T09:26:04.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.706+0000 7f9574a00640 1 -- 172.21.15.6:0/4059000974 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9560031870 con 0x7f9578070ee0 2024-03-04T09:26:04.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.706+0000 7f9574a00640 1 -- 172.21.15.6:0/4059000974 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9560031c40 con 0x7f9578070ee0 2024-03-04T09:26:04.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.706+0000 7f957ce00640 1 -- 172.21.15.6:0/4059000974 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9578070ee0 msgr2=0x7f9578162c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:04.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.706+0000 7f957ce00640 1 --2- 172.21.15.6:0/4059000974 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9578070ee0 0x7f9578162c40 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f957806f070 tx=0x7f956002b910 comp rx=0 tx=0).stop 2024-03-04T09:26:04.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.706+0000 7f957ce00640 1 -- 172.21.15.6:0/4059000974 shutdown_connections 2024-03-04T09:26:04.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.706+0000 7f957ce00640 1 --2- 172.21.15.6:0/4059000974 >> v2:172.21.15.146:3300/0 conn(0x7f95780712c0 0x7f957806e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:04.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.706+0000 7f957ce00640 1 --2- 172.21.15.6:0/4059000974 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9578070ee0 0x7f9578162c40 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:04.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.706+0000 7f957ce00640 1 -- 172.21.15.6:0/4059000974 >> 172.21.15.6:0/4059000974 conn(0x7f957806afb0 msgr2=0x7f957806b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:04.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.706+0000 7f957ce00640 1 -- 172.21.15.6:0/4059000974 shutdown_connections 2024-03-04T09:26:04.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.706+0000 7f957ce00640 1 -- 172.21.15.6:0/4059000974 wait complete. 2024-03-04T09:26:04.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.708+0000 7f957ce00640 1 Processor -- start 2024-03-04T09:26:04.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.708+0000 7f957ce00640 1 -- start start 2024-03-04T09:26:04.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.709+0000 7f957ce00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9578070ee0 0x7f9578162250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:04.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.709+0000 7f957ce00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f95780712c0 0x7f95780b4230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:04.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.709+0000 7f957ce00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9578072da0 0x7f9578164f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:04.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.709+0000 7f957ce00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95780cdda0 con 0x7f95780712c0 2024-03-04T09:26:04.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.709+0000 7f957ce00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95780cdc20 con 0x7f9578072da0 2024-03-04T09:26:04.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.709+0000 7f957ce00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95780cf130 con 0x7f9578070ee0 2024-03-04T09:26:04.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.709+0000 7f9575400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f95780712c0 0x7f95780b4230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:04.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.709+0000 7f9575400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f95780712c0 0x7f95780b4230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:46740/0 (socket says 172.21.15.6:46740) 2024-03-04T09:26:04.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.709+0000 7f9575400640 1 -- 172.21.15.6:0/3087147648 learned_addr learned my addr 172.21.15.6:0/3087147648 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:04.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.710+0000 7f9575400640 1 -- 172.21.15.6:0/3087147648 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9578070ee0 msgr2=0x7f9578162250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:04.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.710+0000 7f9575400640 1 --2- 172.21.15.6:0/3087147648 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9578070ee0 0x7f9578162250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:04.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.710+0000 7f9575400640 1 -- 172.21.15.6:0/3087147648 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9578072da0 msgr2=0x7f9578164f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:04.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.710+0000 7f9575400640 1 --2- 172.21.15.6:0/3087147648 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9578072da0 0x7f9578164f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:04.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.710+0000 7f9575400640 1 -- 172.21.15.6:0/3087147648 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9578162790 con 0x7f95780712c0 2024-03-04T09:26:04.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.710+0000 7f9575400640 1 --2- 172.21.15.6:0/3087147648 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f95780712c0 0x7f95780b4230 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f956800b3f0 tx=0x7f956800b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:04.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.710+0000 7f956e000640 1 -- 172.21.15.6:0/3087147648 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9568015020 con 0x7f95780712c0 2024-03-04T09:26:04.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.710+0000 7f956e000640 1 -- 172.21.15.6:0/3087147648 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9568011830 con 0x7f95780712c0 2024-03-04T09:26:04.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.710+0000 7f957ce00640 1 -- 172.21.15.6:0/3087147648 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95780b4770 con 0x7f95780712c0 2024-03-04T09:26:04.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.710+0000 7f956e000640 1 -- 172.21.15.6:0/3087147648 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9568011c20 con 0x7f95780712c0 2024-03-04T09:26:04.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.710+0000 7f957ce00640 1 -- 172.21.15.6:0/3087147648 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9578161770 con 0x7f95780712c0 2024-03-04T09:26:04.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.711+0000 7f957ce00640 1 -- 172.21.15.6:0/3087147648 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9578069510 con 0x7f95780712c0 2024-03-04T09:26:04.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.712+0000 7f956e000640 1 -- 172.21.15.6:0/3087147648 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9568018020 con 0x7f95780712c0 2024-03-04T09:26:04.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.713+0000 7f956e000640 1 --2- 172.21.15.6:0/3087147648 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f955406b4d0 0x7f955406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:04.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.713+0000 7f956e000640 1 -- 172.21.15.6:0/3087147648 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f9568089ec0 con 0x7f95780712c0 2024-03-04T09:26:04.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.714+0000 7f9575e00640 1 --2- 172.21.15.6:0/3087147648 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f955406b4d0 0x7f955406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:04.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.716+0000 7f9575e00640 1 --2- 172.21.15.6:0/3087147648 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f955406b4d0 0x7f955406d990 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f9560036000 tx=0x7f9560030000 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:04.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.719+0000 7f956e000640 1 -- 172.21.15.6:0/3087147648 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f956801f030 con 0x7f95780712c0 2024-03-04T09:26:04.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.933+0000 7f957ce00640 1 -- 172.21.15.6:0/3087147648 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f957804f3a0 con 0x7f95780712c0 2024-03-04T09:26:04.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.934+0000 7f956e000640 1 -- 172.21.15.6:0/3087147648 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v22) v1 ==== 80+0+0 (secure 0 0 0) 0x7f95680172c0 con 0x7f95780712c0 2024-03-04T09:26:04.978 INFO:teuthology.orchestra.run.smithi006.stderr:listed 0 entries 2024-03-04T09:26:04.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.937+0000 7f957ce00640 1 -- 172.21.15.6:0/3087147648 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f955406b4d0 msgr2=0x7f955406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:04.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.937+0000 7f957ce00640 1 --2- 172.21.15.6:0/3087147648 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f955406b4d0 0x7f955406d990 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f9560036000 tx=0x7f9560030000 comp rx=0 tx=0).stop 2024-03-04T09:26:04.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.937+0000 7f957ce00640 1 -- 172.21.15.6:0/3087147648 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f95780712c0 msgr2=0x7f95780b4230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:04.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.937+0000 7f957ce00640 1 --2- 172.21.15.6:0/3087147648 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f95780712c0 0x7f95780b4230 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f956800b3f0 tx=0x7f956800b8b0 comp rx=0 tx=0).stop 2024-03-04T09:26:04.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.937+0000 7f957ce00640 1 -- 172.21.15.6:0/3087147648 shutdown_connections 2024-03-04T09:26:04.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.937+0000 7f957ce00640 1 --2- 172.21.15.6:0/3087147648 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9578072da0 0x7f9578164f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:04.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.937+0000 7f957ce00640 1 --2- 172.21.15.6:0/3087147648 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f95780712c0 0x7f95780b4230 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:04.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.937+0000 7f957ce00640 1 --2- 172.21.15.6:0/3087147648 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f955406b4d0 0x7f955406d990 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:04.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.937+0000 7f957ce00640 1 --2- 172.21.15.6:0/3087147648 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9578070ee0 0x7f9578162250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:04.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.937+0000 7f957ce00640 1 -- 172.21.15.6:0/3087147648 >> 172.21.15.6:0/3087147648 conn(0x7f957806afb0 msgr2=0x7f95780b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:04.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.937+0000 7f957ce00640 1 -- 172.21.15.6:0/3087147648 shutdown_connections 2024-03-04T09:26:04.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:04.937+0000 7f957ce00640 1 -- 172.21.15.6:0/3087147648 wait complete. 2024-03-04T09:26:04.979 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-04T09:26:04.979 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:26:04.979 DEBUG:teuthology.orchestra.run.smithi006:> sudo umount /home/ubuntu/cephtest/mnt.0 -f 2024-03-04T09:26:05.091 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi006.front.sepia.ceph.com 2024-03-04T09:26:05.092 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:26:05.092 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-04T09:26:05.151 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-04T09:26:05.151 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T09:26:05.205 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-04T09:26:05.206 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T09:26:05.252 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-04T09:26:05.253 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T09:26:05.346 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T09:26:05.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:05.579+0000 7f5f0c400640 1 Processor -- start 2024-03-04T09:26:05.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:05.579+0000 7f5f0c400640 1 -- start start 2024-03-04T09:26:05.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:05.580+0000 7f5f0c400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f5f040cfd90 0x7f5f040d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:05.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:05.580+0000 7f5f0c400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f5f040d2760 0x7f5f040b3740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:05.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:05.580+0000 7f5f0c400640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f040cea50 con 0x7f5f04161b40 2024-03-04T09:26:05.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:05.580+0000 7f5f0c400640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f040b5770 con 0x7f5f040cfd90 2024-03-04T09:26:05.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:05.580+0000 7f5f0c400640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f040cf900 con 0x7f5f040d2760 2024-03-04T09:26:05.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:05.580+0000 7f5f09000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f5f040cfd90 0x7f5f040d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:05.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:05.581+0000 7f5f09000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f5f040cfd90 0x7f5f040d2220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:46754/0 (socket says 172.21.15.6:46754) 2024-03-04T09:26:05.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:05.581+0000 7f5f09000640 1 -- 172.21.15.6:0/2193414558 learned_addr learned my addr 172.21.15.6:0/2193414558 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:05.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:05.581+0000 7f5f09000640 1 -- 172.21.15.6:0/2193414558 >> v2:172.21.15.146:3300/0 conn(0x7f5f040d2760 msgr2=0x7f5f040b3740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:05.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:05.581+0000 7f5f0a400640 1 --2- 172.21.15.6:0/2193414558 >> v2:172.21.15.146:3300/0 conn(0x7f5f040d2760 0x7f5f040b3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:05.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:05.581+0000 7f5f09000640 1 --2- 172.21.15.6:0/2193414558 >> v2:172.21.15.146:3300/0 conn(0x7f5f040d2760 0x7f5f040b3740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:05.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:05.581+0000 7f5f09000640 1 -- 172.21.15.6:0/2193414558 >> v1:172.21.15.146:6789/0 conn(0x7f5f04161b40 legacy=0x7f5f040b5060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:05.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:05.581+0000 7f5f09000640 1 -- 172.21.15.6:0/2193414558 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f040cecf0 con 0x7f5f040cfd90 2024-03-04T09:26:05.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:05.583+0000 7f5f09000640 1 --2- 172.21.15.6:0/2193414558 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5f040cfd90 0x7f5f040d2220 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f5f0406b7f0 tx=0x7f5ef002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b8556ea30078a5b server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:05.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:05.584+0000 7f5f03e00640 1 -- 172.21.15.6:0/2193414558 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ef0035020 con 0x7f5f040cfd90 2024-03-04T09:26:05.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:05.584+0000 7f5f03e00640 1 -- 172.21.15.6:0/2193414558 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ef0031870 con 0x7f5f040cfd90 2024-03-04T09:26:05.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:05.584+0000 7f5f03e00640 1 -- 172.21.15.6:0/2193414558 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ef0031c40 con 0x7f5f040cfd90 2024-03-04T09:26:05.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:05.584+0000 7f5f0c400640 1 -- 172.21.15.6:0/2193414558 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5f040cfd90 msgr2=0x7f5f040d2220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:05.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:05.584+0000 7f5f0c400640 1 --2- 172.21.15.6:0/2193414558 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5f040cfd90 0x7f5f040d2220 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f5f0406b7f0 tx=0x7f5ef002b910 comp rx=0 tx=0).stop 2024-03-04T09:26:05.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:05.584+0000 7f5f0c400640 1 -- 172.21.15.6:0/2193414558 shutdown_connections 2024-03-04T09:26:05.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:05.584+0000 7f5f0c400640 1 --2- 172.21.15.6:0/2193414558 >> v2:172.21.15.146:3300/0 conn(0x7f5f040d2760 0x7f5f040b3740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:05.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:05.584+0000 7f5f0c400640 1 --2- 172.21.15.6:0/2193414558 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5f040cfd90 0x7f5f040d2220 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:05.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:05.584+0000 7f5f0c400640 1 -- 172.21.15.6:0/2193414558 >> 172.21.15.6:0/2193414558 conn(0x7f5f0409d220 msgr2=0x7f5f040d3ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:05.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:05.584+0000 7f5f0c400640 1 -- 172.21.15.6:0/2193414558 shutdown_connections 2024-03-04T09:26:05.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:05.584+0000 7f5f0c400640 1 -- 172.21.15.6:0/2193414558 wait complete. 2024-03-04T09:26:05.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:05.585+0000 7f5f0c400640 1 Processor -- start 2024-03-04T09:26:05.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:05.585+0000 7f5f0c400640 1 -- start start 2024-03-04T09:26:05.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:05.586+0000 7f5f0c400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5f040cfd90 0x7f5f040cd0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:05.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:05.586+0000 7f5f0c400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5f040d2760 0x7f5f041f7240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:05.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:05.586+0000 7f5f0c400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5f04161b40 0x7f5f040b7190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:05.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:05.586+0000 7f5f0c400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f040cf590 con 0x7f5f040cfd90 2024-03-04T09:26:05.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:05.586+0000 7f5f0c400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f040cf2b0 con 0x7f5f04161b40 2024-03-04T09:26:05.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:05.586+0000 7f5f0c400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f040b2ff0 con 0x7f5f040d2760 2024-03-04T09:26:05.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:05.587+0000 7f5f09a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5f040cfd90 0x7f5f040cd0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:05.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:05.587+0000 7f5f09a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5f040cfd90 0x7f5f040cd0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:46758/0 (socket says 172.21.15.6:46758) 2024-03-04T09:26:05.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:05.587+0000 7f5f09a00640 1 -- 172.21.15.6:0/1612898539 learned_addr learned my addr 172.21.15.6:0/1612898539 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:05.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:05.587+0000 7f5f09000640 1 --2- 172.21.15.6:0/1612898539 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5f040d2760 0x7f5f041f7240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:05.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:05.587+0000 7f5f09a00640 1 -- 172.21.15.6:0/1612898539 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5f040d2760 msgr2=0x7f5f041f7240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:05.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:05.587+0000 7f5f09a00640 1 --2- 172.21.15.6:0/1612898539 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5f040d2760 0x7f5f041f7240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:05.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:05.587+0000 7f5f09a00640 1 -- 172.21.15.6:0/1612898539 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5f04161b40 msgr2=0x7f5f040b7190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:05.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:05.587+0000 7f5f09a00640 1 --2- 172.21.15.6:0/1612898539 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5f04161b40 0x7f5f040b7190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:05.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:05.587+0000 7f5f09a00640 1 -- 172.21.15.6:0/1612898539 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f041f7780 con 0x7f5f040cfd90 2024-03-04T09:26:05.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:05.587+0000 7f5f09a00640 1 --2- 172.21.15.6:0/1612898539 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5f040cfd90 0x7f5f040cd0e0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f5ef800c7c0 tx=0x7f5ef800cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:05.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:05.588+0000 7f5f01600640 1 -- 172.21.15.6:0/1612898539 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ef8016020 con 0x7f5f040cfd90 2024-03-04T09:26:05.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:05.588+0000 7f5f01600640 1 -- 172.21.15.6:0/1612898539 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ef800aa50 con 0x7f5f040cfd90 2024-03-04T09:26:05.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:05.588+0000 7f5f01600640 1 -- 172.21.15.6:0/1612898539 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ef8012420 con 0x7f5f040cfd90 2024-03-04T09:26:05.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:05.588+0000 7f5f0c400640 1 -- 172.21.15.6:0/1612898539 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f040b76d0 con 0x7f5f040cfd90 2024-03-04T09:26:05.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:05.588+0000 7f5f0c400640 1 -- 172.21.15.6:0/1612898539 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f041f7ce0 con 0x7f5f040cfd90 2024-03-04T09:26:05.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:05.589+0000 7f5f0c400640 1 -- 172.21.15.6:0/1612898539 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f0404f3a0 con 0x7f5f040cfd90 2024-03-04T09:26:05.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:05.590+0000 7f5f01600640 1 -- 172.21.15.6:0/1612898539 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5ef8019020 con 0x7f5f040cfd90 2024-03-04T09:26:05.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:05.592+0000 7f5f01600640 1 --2- 172.21.15.6:0/1612898539 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f5ee406b4d0 0x7f5ee406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:05.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:05.592+0000 7f5f01600640 1 -- 172.21.15.6:0/1612898539 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f5ef808a070 con 0x7f5f040cfd90 2024-03-04T09:26:05.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:05.592+0000 7f5f09000640 1 --2- 172.21.15.6:0/1612898539 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f5ee406b4d0 0x7f5ee406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:05.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:05.593+0000 7f5f09000640 1 --2- 172.21.15.6:0/1612898539 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f5ee406b4d0 0x7f5ee406d990 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f5ef0036000 tx=0x7f5ef003c040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:05.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:05.597+0000 7f5f01600640 1 -- 172.21.15.6:0/1612898539 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5ef8020070 con 0x7f5f040cfd90 2024-03-04T09:26:05.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:05.797+0000 7f5f0c400640 1 -- 172.21.15.6:0/1612898539 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5f040d1df0 con 0x7f5f040cfd90 2024-03-04T09:26:05.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:05.801+0000 7f5f01600640 1 -- 172.21.15.6:0/1612898539 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16852 (secure 0 0 0) 0x7f5f041f7780 con 0x7f5f040cfd90 2024-03-04T09:26:05.798 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:26:05.798 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":22,"fsid":"edce0578-5ae4-4402-b45a-552bdef84d7c","created":"2024-03-04T09:24:54.523780+0000","modified":"2024-03-04T09:26:01.727682+0000","last_up_change":"2024-03-04T09:25:07.089049+0000","last_in_change":"2024-03-04T09:24:57.982077+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T09:25:07.935198+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-04T09:25:17.118573+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-04T09:25:17.930536+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a96574f-81f1-4bc6-8463-757520b84a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6825","nonce":1364449401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6827","nonce":1364449401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6831","nonce":1364449401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6829","nonce":1364449401}]},"public_addr":"172.21.15.6:6825/1364449401","cluster_addr":"172.21.15.6:6827/1364449401","heartbeat_back_addr":"172.21.15.6:6831/1364449401","heartbeat_front_addr":"172.21.15.6:6829/1364449401","state":["exists","up"]},{"osd":1,"uuid":"7d79b9c1-6b87-476a-bc36-7db5ee64cd90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6809","nonce":2494103078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6811","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6813","nonce":2494103078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6821","nonce":2494103078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6817","nonce":2494103078}]},"public_addr":"172.21.15.6:6809/2494103078","cluster_addr":"172.21.15.6:6813/2494103078","heartbeat_back_addr":"172.21.15.6:6821/2494103078","heartbeat_front_addr":"172.21.15.6:6817/2494103078","state":["exists","up"]},{"osd":2,"uuid":"492331b9-4ee9-40b6-bacd-85d44cd6c45a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6812","nonce":1500066681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6816","nonce":1500066681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6824","nonce":1500066681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6820","nonce":1500066681}]},"public_addr":"172.21.15.6:6812/1500066681","cluster_addr":"172.21.15.6:6816/1500066681","heartbeat_back_addr":"172.21.15.6:6824/1500066681","heartbeat_front_addr":"172.21.15.6:6820/1500066681","state":["exists","up"]},{"osd":3,"uuid":"21bb2efe-9ba7-4bdc-bc31-180cd1102e21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6801","nonce":1558806123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6803","nonce":1558806123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6807","nonce":1558806123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6805","nonce":1558806123}]},"public_addr":"172.21.15.6:6801/1558806123","cluster_addr":"172.21.15.6:6803/1558806123","heartbeat_back_addr":"172.21.15.6:6807/1558806123","heartbeat_front_addr":"172.21.15.6:6805/1558806123","state":["exists","up"]},{"osd":4,"uuid":"6e2c4b71-19d9-4737-83d9-558705d8644a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6801","nonce":1526135780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6803","nonce":1526135780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6807","nonce":1526135780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6805","nonce":1526135780}]},"public_addr":"172.21.15.146:6801/1526135780","cluster_addr":"172.21.15.146:6803/1526135780","heartbeat_back_addr":"172.21.15.146:6807/1526135780","heartbeat_front_addr":"172.21.15.146:6805/1526135780","state":["exists","up"]},{"osd":5,"uuid":"fb69c525-f1be-4b7d-ab32-6c29b0d6fddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6809","nonce":460262507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6811","nonce":460262507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6815","nonce":460262507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6813","nonce":460262507}]},"public_addr":"172.21.15.146:6809/460262507","cluster_addr":"172.21.15.146:6811/460262507","heartbeat_back_addr":"172.21.15.146:6815/460262507","heartbeat_front_addr":"172.21.15.146:6813/460262507","state":["exists","up"]},{"osd":6,"uuid":"c32e7fed-370d-4401-8ce0-6a4335b0d168","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6823","nonce":1392897493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6827","nonce":1392897493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6831","nonce":1392897493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6829","nonce":1392897493}]},"public_addr":"172.21.15.146:6823/1392897493","cluster_addr":"172.21.15.146:6827/1392897493","heartbeat_back_addr":"172.21.15.146:6831/1392897493","heartbeat_front_addr":"172.21.15.146:6829/1392897493","state":["exists","up"]},{"osd":7,"uuid":"75a79b44-fa5f-426d-a7d3-71f21f78835c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6817","nonce":3268323156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6819","nonce":3268323156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6825","nonce":3268323156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6822","nonce":3268323156}]},"public_addr":"172.21.15.146:6817/3268323156","cluster_addr":"172.21.15.146:6819/3268323156","heartbeat_back_addr":"172.21.15.146:6825/3268323156","heartbeat_front_addr":"172.21.15.146:6822/3268323156","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.310437+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.295850+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.294674+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.526855+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.300023+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.595160+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.346046+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.558741+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.7","mappings":[{"from":3,"to":2}]},{"pgid":"3.f","mappings":[{"from":7,"to":2}]},{"pgid":"3.21","mappings":[{"from":5,"to":1}]},{"pgid":"3.22","mappings":[{"from":5,"to":2}]},{"pgid":"3.28","mappings":[{"from":3,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T09:26:05.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:05.804+0000 7f5f0c400640 1 -- 172.21.15.6:0/1612898539 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f5ee406b4d0 msgr2=0x7f5ee406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:05.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:05.804+0000 7f5f0c400640 1 --2- 172.21.15.6:0/1612898539 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f5ee406b4d0 0x7f5ee406d990 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f5ef0036000 tx=0x7f5ef003c040 comp rx=0 tx=0).stop 2024-03-04T09:26:05.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:05.804+0000 7f5f0c400640 1 -- 172.21.15.6:0/1612898539 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5f040cfd90 msgr2=0x7f5f040cd0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:05.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:05.804+0000 7f5f0c400640 1 --2- 172.21.15.6:0/1612898539 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5f040cfd90 0x7f5f040cd0e0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f5ef800c7c0 tx=0x7f5ef800cc80 comp rx=0 tx=0).stop 2024-03-04T09:26:05.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:05.804+0000 7f5f0c400640 1 -- 172.21.15.6:0/1612898539 shutdown_connections 2024-03-04T09:26:05.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:05.804+0000 7f5f0c400640 1 --2- 172.21.15.6:0/1612898539 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5f04161b40 0x7f5f040b7190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:05.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:05.804+0000 7f5f0c400640 1 --2- 172.21.15.6:0/1612898539 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f5ee406b4d0 0x7f5ee406d990 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:05.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:05.804+0000 7f5f0c400640 1 --2- 172.21.15.6:0/1612898539 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5f040d2760 0x7f5f041f7240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:05.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:05.804+0000 7f5f0c400640 1 --2- 172.21.15.6:0/1612898539 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5f040cfd90 0x7f5f040cd0e0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:05.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:05.804+0000 7f5f0c400640 1 -- 172.21.15.6:0/1612898539 >> 172.21.15.6:0/1612898539 conn(0x7f5f0409d220 msgr2=0x7f5f040b5040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:05.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:05.804+0000 7f5f0c400640 1 -- 172.21.15.6:0/1612898539 shutdown_connections 2024-03-04T09:26:05.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:05.804+0000 7f5f0c400640 1 -- 172.21.15.6:0/1612898539 wait complete. 2024-03-04T09:26:05.821 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T09:25:07.935198+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-04T09:25:17.118573+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-04T09:25:17.930536+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-04T09:26:05.822 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-04T09:26:06.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.050+0000 7fada7200640 1 Processor -- start 2024-03-04T09:26:06.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.051+0000 7fada7200640 1 -- start start 2024-03-04T09:26:06.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.052+0000 7fada7200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fada0145430 0x7fada015f810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:06.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.052+0000 7fada7200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fada015cc90 0x7fada00b1b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:06.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.052+0000 7fada7200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fada00b2080 0x7fada006c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:06.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.052+0000 7fada7200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fada015fe40 con 0x7fada015cc90 2024-03-04T09:26:06.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.052+0000 7fada7200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fada0160120 con 0x7fada00b2080 2024-03-04T09:26:06.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.052+0000 7fada7200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fada0160400 con 0x7fada0145430 2024-03-04T09:26:06.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.052+0000 7fada5400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fada015cc90 0x7fada00b1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:06.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.052+0000 7fada5400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fada015cc90 0x7fada00b1b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:46768/0 (socket says 172.21.15.6:46768) 2024-03-04T09:26:06.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.053+0000 7fada5400640 1 -- 172.21.15.6:0/2574498661 learned_addr learned my addr 172.21.15.6:0/2574498661 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:06.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.053+0000 7fada5e00640 1 --2- 172.21.15.6:0/2574498661 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fada0145430 0x7fada015f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:06.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.053+0000 7fada5400640 1 -- 172.21.15.6:0/2574498661 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fada0145430 msgr2=0x7fada015f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:06.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.053+0000 7fada5400640 1 --2- 172.21.15.6:0/2574498661 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fada0145430 0x7fada015f810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:06.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.053+0000 7fada5400640 1 -- 172.21.15.6:0/2574498661 >> v2:172.21.15.146:3300/0 conn(0x7fada00b2080 msgr2=0x7fada006c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:06.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.053+0000 7fada5400640 1 --2- 172.21.15.6:0/2574498661 >> v2:172.21.15.146:3300/0 conn(0x7fada00b2080 0x7fada006c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:06.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.053+0000 7fada5400640 1 -- 172.21.15.6:0/2574498661 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fada01606e0 con 0x7fada015cc90 2024-03-04T09:26:06.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.053+0000 7fada5e00640 1 --2- 172.21.15.6:0/2574498661 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fada0145430 0x7fada015f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:26:06.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.055+0000 7fada5400640 1 --2- 172.21.15.6:0/2574498661 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fada015cc90 0x7fada00b1b40 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fada006eaf0 tx=0x7fad9002b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb229995ec9b8837 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:06.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.055+0000 7fada4a00640 1 -- 172.21.15.6:0/2574498661 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad90035020 con 0x7fada015cc90 2024-03-04T09:26:06.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.055+0000 7fada4a00640 1 -- 172.21.15.6:0/2574498661 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad9002d040 con 0x7fada015cc90 2024-03-04T09:26:06.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.055+0000 7fada4a00640 1 -- 172.21.15.6:0/2574498661 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad90031a80 con 0x7fada015cc90 2024-03-04T09:26:06.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.056+0000 7fada7200640 1 -- 172.21.15.6:0/2574498661 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fada015cc90 msgr2=0x7fada00b1b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:06.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.056+0000 7fada7200640 1 --2- 172.21.15.6:0/2574498661 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fada015cc90 0x7fada00b1b40 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fada006eaf0 tx=0x7fad9002b900 comp rx=0 tx=0).stop 2024-03-04T09:26:06.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.056+0000 7fada7200640 1 -- 172.21.15.6:0/2574498661 shutdown_connections 2024-03-04T09:26:06.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.056+0000 7fada7200640 1 --2- 172.21.15.6:0/2574498661 >> v2:172.21.15.146:3300/0 conn(0x7fada00b2080 0x7fada006c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:06.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.056+0000 7fada7200640 1 --2- 172.21.15.6:0/2574498661 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fada015cc90 0x7fada00b1b40 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:06.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.056+0000 7fada7200640 1 --2- 172.21.15.6:0/2574498661 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fada0145430 0x7fada015f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:06.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.056+0000 7fada7200640 1 -- 172.21.15.6:0/2574498661 >> 172.21.15.6:0/2574498661 conn(0x7fada006b3a0 msgr2=0x7fada006b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:06.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.056+0000 7fada7200640 1 -- 172.21.15.6:0/2574498661 shutdown_connections 2024-03-04T09:26:06.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.056+0000 7fada7200640 1 -- 172.21.15.6:0/2574498661 wait complete. 2024-03-04T09:26:06.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.058+0000 7fada7200640 1 Processor -- start 2024-03-04T09:26:06.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.058+0000 7fada7200640 1 -- start start 2024-03-04T09:26:06.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.059+0000 7fada7200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fada00b2080 0x7fada00ab890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:06.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.059+0000 7fada7200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fada0145430 0x7fada013f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:06.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.059+0000 7fada7200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fada015cc90 0x7fada00aeb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:06.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.059+0000 7fada7200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fada0145990 con 0x7fada015cc90 2024-03-04T09:26:06.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.059+0000 7fada7200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fada0145810 con 0x7fada00b2080 2024-03-04T09:26:06.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.059+0000 7fada7200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fada00cde00 con 0x7fada0145430 2024-03-04T09:26:06.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.059+0000 7fada6800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fada015cc90 0x7fada00aeb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:06.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.059+0000 7fada6800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fada015cc90 0x7fada00aeb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:46772/0 (socket says 172.21.15.6:46772) 2024-03-04T09:26:06.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.059+0000 7fada6800640 1 -- 172.21.15.6:0/1316253085 learned_addr learned my addr 172.21.15.6:0/1316253085 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:06.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.059+0000 7fada5e00640 1 --2- 172.21.15.6:0/1316253085 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fada00b2080 0x7fada00ab890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:06.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.059+0000 7fada5400640 1 --2- 172.21.15.6:0/1316253085 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fada0145430 0x7fada013f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:06.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.060+0000 7fada6800640 1 -- 172.21.15.6:0/1316253085 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fada0145430 msgr2=0x7fada013f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:06.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.060+0000 7fada6800640 1 --2- 172.21.15.6:0/1316253085 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fada0145430 0x7fada013f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:06.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.060+0000 7fada6800640 1 -- 172.21.15.6:0/1316253085 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fada00b2080 msgr2=0x7fada00ab890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:06.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.060+0000 7fada6800640 1 --2- 172.21.15.6:0/1316253085 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fada00b2080 0x7fada00ab890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:06.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.060+0000 7fada6800640 1 -- 172.21.15.6:0/1316253085 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fada00aa510 con 0x7fada015cc90 2024-03-04T09:26:06.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.060+0000 7fada6800640 1 --2- 172.21.15.6:0/1316253085 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fada015cc90 0x7fada00aeb40 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fad9c00b440 tx=0x7fad9c00b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:06.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.060+0000 7fad96000640 1 -- 172.21.15.6:0/1316253085 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad9c015020 con 0x7fada015cc90 2024-03-04T09:26:06.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.060+0000 7fad96000640 1 -- 172.21.15.6:0/1316253085 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad9c011830 con 0x7fada015cc90 2024-03-04T09:26:06.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.060+0000 7fad96000640 1 -- 172.21.15.6:0/1316253085 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad9c011c20 con 0x7fada015cc90 2024-03-04T09:26:06.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.060+0000 7fada7200640 1 -- 172.21.15.6:0/1316253085 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fada00aa7f0 con 0x7fada015cc90 2024-03-04T09:26:06.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.061+0000 7fada7200640 1 -- 172.21.15.6:0/1316253085 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fada00aadb0 con 0x7fada015cc90 2024-03-04T09:26:06.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.062+0000 7fada7200640 1 -- 172.21.15.6:0/1316253085 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fada004f330 con 0x7fada015cc90 2024-03-04T09:26:06.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.063+0000 7fad96000640 1 -- 172.21.15.6:0/1316253085 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fad9c018020 con 0x7fada015cc90 2024-03-04T09:26:06.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.064+0000 7fad96000640 1 --2- 172.21.15.6:0/1316253085 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fad8406b4d0 0x7fad8406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:06.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.064+0000 7fad96000640 1 -- 172.21.15.6:0/1316253085 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7fad9c08a1f0 con 0x7fada015cc90 2024-03-04T09:26:06.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.064+0000 7fada5e00640 1 --2- 172.21.15.6:0/1316253085 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fad8406b4d0 0x7fad8406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:06.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.066+0000 7fada5e00640 1 --2- 172.21.15.6:0/1316253085 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fad8406b4d0 0x7fad8406d990 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fad980075b0 tx=0x7fad98007540 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:06.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.068+0000 7fad96000640 1 -- 172.21.15.6:0/1316253085 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fad9c01f020 con 0x7fada015cc90 2024-03-04T09:26:06.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.266+0000 7fada7200640 1 -- 172.21.15.6:0/1316253085 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fada00b3d60 con 0x7fada015cc90 2024-03-04T09:26:06.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.267+0000 7fad96000640 1 -- 172.21.15.6:0/1316253085 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7fad9c0172c0 con 0x7fada015cc90 2024-03-04T09:26:06.264 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-04T09:26:06.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.269+0000 7fada7200640 1 -- 172.21.15.6:0/1316253085 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fad8406b4d0 msgr2=0x7fad8406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:06.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.269+0000 7fada7200640 1 --2- 172.21.15.6:0/1316253085 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fad8406b4d0 0x7fad8406d990 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fad980075b0 tx=0x7fad98007540 comp rx=0 tx=0).stop 2024-03-04T09:26:06.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.269+0000 7fada7200640 1 -- 172.21.15.6:0/1316253085 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fada015cc90 msgr2=0x7fada00aeb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:06.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.269+0000 7fada7200640 1 --2- 172.21.15.6:0/1316253085 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fada015cc90 0x7fada00aeb40 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fad9c00b440 tx=0x7fad9c00b900 comp rx=0 tx=0).stop 2024-03-04T09:26:06.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.269+0000 7fada7200640 1 -- 172.21.15.6:0/1316253085 shutdown_connections 2024-03-04T09:26:06.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.269+0000 7fada7200640 1 --2- 172.21.15.6:0/1316253085 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fada015cc90 0x7fada00aeb40 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:06.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.269+0000 7fada7200640 1 --2- 172.21.15.6:0/1316253085 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fad8406b4d0 0x7fad8406d990 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:06.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.269+0000 7fada7200640 1 --2- 172.21.15.6:0/1316253085 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fada0145430 0x7fada013f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:06.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.269+0000 7fada7200640 1 --2- 172.21.15.6:0/1316253085 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fada00b2080 0x7fada00ab890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:06.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.269+0000 7fada7200640 1 -- 172.21.15.6:0/1316253085 >> 172.21.15.6:0/1316253085 conn(0x7fada006b3a0 msgr2=0x7fada015dab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:06.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.270+0000 7fada7200640 1 -- 172.21.15.6:0/1316253085 shutdown_connections 2024-03-04T09:26:06.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.270+0000 7fada7200640 1 -- 172.21.15.6:0/1316253085 wait complete. 2024-03-04T09:26:06.284 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-04T09:26:06.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.517+0000 7fb571600640 1 Processor -- start 2024-03-04T09:26:06.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.517+0000 7fb571600640 1 -- start start 2024-03-04T09:26:06.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.518+0000 7fb571600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb56c15bf20 0x7fb56c0acd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:06.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.519+0000 7fb571600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fb56c071e20 0x7fb56c0ccad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:06.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.519+0000 7fb571600640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb56c072df0 con 0x7fb56c072520 2024-03-04T09:26:06.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.519+0000 7fb571600640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb56c0cdab0 con 0x7fb56c071e20 2024-03-04T09:26:06.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.519+0000 7fb571600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb56c0b4d70 con 0x7fb56c15bf20 2024-03-04T09:26:06.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.519+0000 7fb56b400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fb56c071e20 0x7fb56c0ccad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:06.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.519+0000 7fb56b400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fb56c071e20 0x7fb56c0ccad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:46774/0 (socket says 172.21.15.6:46774) 2024-03-04T09:26:06.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.519+0000 7fb56b400640 1 -- 172.21.15.6:0/444660585 learned_addr learned my addr 172.21.15.6:0/444660585 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:06.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.519+0000 7fb56aa00640 1 --2- 172.21.15.6:0/444660585 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb56c15bf20 0x7fb56c0acd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:06.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.519+0000 7fb56b400640 1 -- 172.21.15.6:0/444660585 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb56c15bf20 msgr2=0x7fb56c0acd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:06.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.519+0000 7fb56b400640 1 --2- 172.21.15.6:0/444660585 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb56c15bf20 0x7fb56c0acd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:06.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.519+0000 7fb56b400640 1 -- 172.21.15.6:0/444660585 >> v1:172.21.15.6:6789/0 conn(0x7fb56c072520 legacy=0x7fb56c0cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:06.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.519+0000 7fb56b400640 1 -- 172.21.15.6:0/444660585 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb56c0b5050 con 0x7fb56c071e20 2024-03-04T09:26:06.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.521+0000 7fb56b400640 1 --2- 172.21.15.6:0/444660585 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb56c071e20 0x7fb56c0ccad0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fb564009e50 tx=0x7fb56402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cfa617680402f21 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:06.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.521+0000 7fb56a000640 1 -- 172.21.15.6:0/444660585 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb56402cb00 con 0x7fb56c071e20 2024-03-04T09:26:06.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.521+0000 7fb56a000640 1 -- 172.21.15.6:0/444660585 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb56402cca0 con 0x7fb56c071e20 2024-03-04T09:26:06.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.521+0000 7fb56a000640 1 -- 172.21.15.6:0/444660585 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb564034610 con 0x7fb56c071e20 2024-03-04T09:26:06.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.522+0000 7fb571600640 1 -- 172.21.15.6:0/444660585 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb56c071e20 msgr2=0x7fb56c0ccad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:06.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.522+0000 7fb571600640 1 --2- 172.21.15.6:0/444660585 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb56c071e20 0x7fb56c0ccad0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fb564009e50 tx=0x7fb56402bf60 comp rx=0 tx=0).stop 2024-03-04T09:26:06.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.522+0000 7fb571600640 1 -- 172.21.15.6:0/444660585 shutdown_connections 2024-03-04T09:26:06.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.522+0000 7fb571600640 1 --2- 172.21.15.6:0/444660585 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb56c071e20 0x7fb56c0ccad0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:06.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.522+0000 7fb571600640 1 --2- 172.21.15.6:0/444660585 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb56c15bf20 0x7fb56c0acd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:06.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.522+0000 7fb571600640 1 -- 172.21.15.6:0/444660585 >> 172.21.15.6:0/444660585 conn(0x7fb56c06b3d0 msgr2=0x7fb56c06b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:06.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.522+0000 7fb571600640 1 -- 172.21.15.6:0/444660585 shutdown_connections 2024-03-04T09:26:06.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.522+0000 7fb571600640 1 -- 172.21.15.6:0/444660585 wait complete. 2024-03-04T09:26:06.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.524+0000 7fb571600640 1 Processor -- start 2024-03-04T09:26:06.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.524+0000 7fb571600640 1 -- start start 2024-03-04T09:26:06.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.525+0000 7fb571600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb56c071e20 0x7fb56c1e4ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:06.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.525+0000 7fb571600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb56c072520 0x7fb56c1f7ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:06.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.525+0000 7fb571600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb56c15bf20 0x7fb56c0ca2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:06.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.525+0000 7fb571600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb56c0b4b10 con 0x7fb56c15bf20 2024-03-04T09:26:06.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.525+0000 7fb571600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb56c0b4990 con 0x7fb56c071e20 2024-03-04T09:26:06.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.525+0000 7fb571600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb56c0ce6c0 con 0x7fb56c072520 2024-03-04T09:26:06.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.525+0000 7fb56b400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb56c15bf20 0x7fb56c0ca2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:06.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.525+0000 7fb56b400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb56c15bf20 0x7fb56c0ca2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:46776/0 (socket says 172.21.15.6:46776) 2024-03-04T09:26:06.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.525+0000 7fb56b400640 1 -- 172.21.15.6:0/3952984053 learned_addr learned my addr 172.21.15.6:0/3952984053 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:06.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.525+0000 7fb562000640 1 --2- 172.21.15.6:0/3952984053 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb56c072520 0x7fb56c1f7ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:06.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.525+0000 7fb56aa00640 1 --2- 172.21.15.6:0/3952984053 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb56c071e20 0x7fb56c1e4ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:06.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.525+0000 7fb56b400640 1 -- 172.21.15.6:0/3952984053 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb56c072520 msgr2=0x7fb56c1f7ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:06.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.525+0000 7fb56b400640 1 --2- 172.21.15.6:0/3952984053 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb56c072520 0x7fb56c1f7ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:06.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.525+0000 7fb56b400640 1 -- 172.21.15.6:0/3952984053 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb56c071e20 msgr2=0x7fb56c1e4ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:06.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.525+0000 7fb56b400640 1 --2- 172.21.15.6:0/3952984053 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb56c071e20 0x7fb56c1e4ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:06.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.525+0000 7fb56b400640 1 -- 172.21.15.6:0/3952984053 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb56c1f8010 con 0x7fb56c15bf20 2024-03-04T09:26:06.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.525+0000 7fb562000640 1 --2- 172.21.15.6:0/3952984053 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb56c072520 0x7fb56c1f7ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:26:06.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.526+0000 7fb56b400640 1 --2- 172.21.15.6:0/3952984053 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb56c15bf20 0x7fb56c0ca2a0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fb564009e50 tx=0x7fb564004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:06.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.526+0000 7fb563400640 1 -- 172.21.15.6:0/3952984053 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb564043020 con 0x7fb56c15bf20 2024-03-04T09:26:06.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.526+0000 7fb563400640 1 -- 172.21.15.6:0/3952984053 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb56402ce50 con 0x7fb56c15bf20 2024-03-04T09:26:06.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.526+0000 7fb563400640 1 -- 172.21.15.6:0/3952984053 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb564009510 con 0x7fb56c15bf20 2024-03-04T09:26:06.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.526+0000 7fb571600640 1 -- 172.21.15.6:0/3952984053 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb56c1f82b0 con 0x7fb56c15bf20 2024-03-04T09:26:06.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.526+0000 7fb571600640 1 -- 172.21.15.6:0/3952984053 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb56c1f8870 con 0x7fb56c15bf20 2024-03-04T09:26:06.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.527+0000 7fb571600640 1 -- 172.21.15.6:0/3952984053 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb56c0cbc60 con 0x7fb56c15bf20 2024-03-04T09:26:06.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.528+0000 7fb563400640 1 -- 172.21.15.6:0/3952984053 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb564046020 con 0x7fb56c15bf20 2024-03-04T09:26:06.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.529+0000 7fb563400640 1 --2- 172.21.15.6:0/3952984053 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb54806b4d0 0x7fb54806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:06.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.529+0000 7fb563400640 1 -- 172.21.15.6:0/3952984053 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7fb5640ac920 con 0x7fb56c15bf20 2024-03-04T09:26:06.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.530+0000 7fb56aa00640 1 --2- 172.21.15.6:0/3952984053 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb54806b4d0 0x7fb54806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:06.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.531+0000 7fb56aa00640 1 --2- 172.21.15.6:0/3952984053 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb54806b4d0 0x7fb54806d990 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fb55c006fd0 tx=0x7fb55c008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:06.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.534+0000 7fb563400640 1 -- 172.21.15.6:0/3952984053 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb56c0cbc60 con 0x7fb56c15bf20 2024-03-04T09:26:06.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.736+0000 7fb571600640 1 -- 172.21.15.6:0/3952984053 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb56c04f330 con 0x7fb56c15bf20 2024-03-04T09:26:06.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.737+0000 7fb563400640 1 -- 172.21.15.6:0/3952984053 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7fb56404d030 con 0x7fb56c15bf20 2024-03-04T09:26:06.734 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-03-04T09:26:06.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.740+0000 7fb571600640 1 -- 172.21.15.6:0/3952984053 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb54806b4d0 msgr2=0x7fb54806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:06.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.740+0000 7fb571600640 1 --2- 172.21.15.6:0/3952984053 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb54806b4d0 0x7fb54806d990 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fb55c006fd0 tx=0x7fb55c008040 comp rx=0 tx=0).stop 2024-03-04T09:26:06.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.740+0000 7fb571600640 1 -- 172.21.15.6:0/3952984053 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb56c15bf20 msgr2=0x7fb56c0ca2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:06.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.740+0000 7fb571600640 1 --2- 172.21.15.6:0/3952984053 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb56c15bf20 0x7fb56c0ca2a0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fb564009e50 tx=0x7fb564004a40 comp rx=0 tx=0).stop 2024-03-04T09:26:06.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.740+0000 7fb571600640 1 -- 172.21.15.6:0/3952984053 shutdown_connections 2024-03-04T09:26:06.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.740+0000 7fb571600640 1 --2- 172.21.15.6:0/3952984053 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb56c15bf20 0x7fb56c0ca2a0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:06.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.740+0000 7fb571600640 1 --2- 172.21.15.6:0/3952984053 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb54806b4d0 0x7fb54806d990 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:06.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.740+0000 7fb571600640 1 --2- 172.21.15.6:0/3952984053 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb56c072520 0x7fb56c1f7ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:06.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.740+0000 7fb571600640 1 --2- 172.21.15.6:0/3952984053 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb56c071e20 0x7fb56c1e4ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:06.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.740+0000 7fb571600640 1 -- 172.21.15.6:0/3952984053 >> 172.21.15.6:0/3952984053 conn(0x7fb56c06b3d0 msgr2=0x7fb56c0cb550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:06.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.740+0000 7fb571600640 1 -- 172.21.15.6:0/3952984053 shutdown_connections 2024-03-04T09:26:06.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.741+0000 7fb571600640 1 -- 172.21.15.6:0/3952984053 wait complete. 2024-03-04T09:26:06.756 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-04T09:26:06.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.981+0000 7f9441400640 1 Processor -- start 2024-03-04T09:26:06.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.981+0000 7f9441400640 1 -- start start 2024-03-04T09:26:06.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.983+0000 7f9441400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f943c147590 0x7f943c071480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:06.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.983+0000 7f9441400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f943c06d8d0 0x7f943c0b1b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:06.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.983+0000 7f9441400640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f943c0b2490 con 0x7f943c0b2080 2024-03-04T09:26:06.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.983+0000 7f9441400640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f943c147970 con 0x7f943c06d8d0 2024-03-04T09:26:06.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.983+0000 7f9441400640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f943c0b31b0 con 0x7f943c147590 2024-03-04T09:26:06.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.983+0000 7f9433e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f943c06d8d0 0x7f943c0b1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:06.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.984+0000 7f9433e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f943c06d8d0 0x7f943c0b1b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:46790/0 (socket says 172.21.15.6:46790) 2024-03-04T09:26:06.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.984+0000 7f9433e00640 1 -- 172.21.15.6:0/2109043986 learned_addr learned my addr 172.21.15.6:0/2109043986 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:06.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.984+0000 7f9433e00640 1 -- 172.21.15.6:0/2109043986 >> v2:172.21.15.146:3300/0 conn(0x7f943c147590 msgr2=0x7f943c071480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:06.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.984+0000 7f943be00640 1 --2- 172.21.15.6:0/2109043986 >> v2:172.21.15.146:3300/0 conn(0x7f943c147590 0x7f943c071480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:06.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.984+0000 7f9433e00640 1 --2- 172.21.15.6:0/2109043986 >> v2:172.21.15.146:3300/0 conn(0x7f943c147590 0x7f943c071480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:06.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.984+0000 7f9433e00640 1 -- 172.21.15.6:0/2109043986 >> v1:172.21.15.146:6789/0 conn(0x7f943c0b2080 legacy=0x7f943c0ad880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:06.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.984+0000 7f9433e00640 1 -- 172.21.15.6:0/2109043986 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f943c0719c0 con 0x7f943c06d8d0 2024-03-04T09:26:06.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.984+0000 7f943be00640 1 --2- 172.21.15.6:0/2109043986 >> v2:172.21.15.146:3300/0 conn(0x7f943c147590 0x7f943c071480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:26:06.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.985+0000 7f9433e00640 1 --2- 172.21.15.6:0/2109043986 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f943c06d8d0 0x7f943c0b1b40 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f943c0b2790 tx=0x7f942402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55dd06e56c6ccd83 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:06.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.986+0000 7f943b400640 1 -- 172.21.15.6:0/2109043986 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9424035020 con 0x7f943c06d8d0 2024-03-04T09:26:06.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.986+0000 7f943b400640 1 -- 172.21.15.6:0/2109043986 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9424031870 con 0x7f943c06d8d0 2024-03-04T09:26:06.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.986+0000 7f943b400640 1 -- 172.21.15.6:0/2109043986 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9424031c40 con 0x7f943c06d8d0 2024-03-04T09:26:06.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.986+0000 7f9441400640 1 -- 172.21.15.6:0/2109043986 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f943c06d8d0 msgr2=0x7f943c0b1b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:06.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.986+0000 7f9441400640 1 --2- 172.21.15.6:0/2109043986 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f943c06d8d0 0x7f943c0b1b40 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f943c0b2790 tx=0x7f942402b910 comp rx=0 tx=0).stop 2024-03-04T09:26:06.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.986+0000 7f9441400640 1 -- 172.21.15.6:0/2109043986 shutdown_connections 2024-03-04T09:26:06.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.986+0000 7f9441400640 1 --2- 172.21.15.6:0/2109043986 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f943c06d8d0 0x7f943c0b1b40 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:06.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.986+0000 7f9441400640 1 --2- 172.21.15.6:0/2109043986 >> v2:172.21.15.146:3300/0 conn(0x7f943c147590 0x7f943c071480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:06.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.986+0000 7f9441400640 1 -- 172.21.15.6:0/2109043986 >> 172.21.15.6:0/2109043986 conn(0x7f943c06b3a0 msgr2=0x7f943c06b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:06.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.986+0000 7f9441400640 1 -- 172.21.15.6:0/2109043986 shutdown_connections 2024-03-04T09:26:06.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.987+0000 7f9441400640 1 -- 172.21.15.6:0/2109043986 wait complete. 2024-03-04T09:26:06.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.988+0000 7f9441400640 1 Processor -- start 2024-03-04T09:26:06.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.988+0000 7f9441400640 1 -- start start 2024-03-04T09:26:06.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.989+0000 7f9441400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f943c06d8d0 0x7f943c13f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:06.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.989+0000 7f9441400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f943c0b2080 0x7f943c132be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:06.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.989+0000 7f9441400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f943c147590 0x7f943c133120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:06.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.989+0000 7f9441400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f943c0b2e40 con 0x7f943c06d8d0 2024-03-04T09:26:06.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.989+0000 7f9441400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f943c06e120 con 0x7f943c147590 2024-03-04T09:26:06.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.989+0000 7f9441400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f943c0b2b50 con 0x7f943c0b2080 2024-03-04T09:26:06.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.989+0000 7f943be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f943c06d8d0 0x7f943c13f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:06.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.990+0000 7f943be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f943c06d8d0 0x7f943c13f8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:46796/0 (socket says 172.21.15.6:46796) 2024-03-04T09:26:06.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.990+0000 7f943be00640 1 -- 172.21.15.6:0/1641921431 learned_addr learned my addr 172.21.15.6:0/1641921431 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:06.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.990+0000 7f9440a00640 1 --2- 172.21.15.6:0/1641921431 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f943c147590 0x7f943c133120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:06.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.990+0000 7f9433e00640 1 --2- 172.21.15.6:0/1641921431 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f943c0b2080 0x7f943c132be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:06.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.990+0000 7f943be00640 1 -- 172.21.15.6:0/1641921431 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f943c0b2080 msgr2=0x7f943c132be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:06.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.990+0000 7f943be00640 1 --2- 172.21.15.6:0/1641921431 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f943c0b2080 0x7f943c132be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:06.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.990+0000 7f943be00640 1 -- 172.21.15.6:0/1641921431 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f943c147590 msgr2=0x7f943c133120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:06.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.990+0000 7f943be00640 1 --2- 172.21.15.6:0/1641921431 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f943c147590 0x7f943c133120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:06.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.990+0000 7f943be00640 1 -- 172.21.15.6:0/1641921431 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f943c0c4ec0 con 0x7f943c06d8d0 2024-03-04T09:26:06.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.990+0000 7f9440a00640 1 --2- 172.21.15.6:0/1641921431 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f943c147590 0x7f943c133120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:26:06.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.990+0000 7f943be00640 1 --2- 172.21.15.6:0/1641921431 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f943c06d8d0 0x7f943c13f8c0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f942c00c7f0 tx=0x7f942c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:06.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.991+0000 7f9438c00640 1 -- 172.21.15.6:0/1641921431 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f942c015070 con 0x7f943c06d8d0 2024-03-04T09:26:06.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.991+0000 7f9438c00640 1 -- 172.21.15.6:0/1641921431 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f942c007820 con 0x7f943c06d8d0 2024-03-04T09:26:06.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.991+0000 7f9438c00640 1 -- 172.21.15.6:0/1641921431 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f942c007bd0 con 0x7f943c06d8d0 2024-03-04T09:26:06.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.991+0000 7f9441400640 1 -- 172.21.15.6:0/1641921431 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f943c140bf0 con 0x7f943c06d8d0 2024-03-04T09:26:06.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.991+0000 7f9441400640 1 -- 172.21.15.6:0/1641921431 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f943c13e900 con 0x7f943c06d8d0 2024-03-04T09:26:06.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.992+0000 7f9441400640 1 -- 172.21.15.6:0/1641921431 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f943c04f330 con 0x7f943c06d8d0 2024-03-04T09:26:06.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.993+0000 7f9438c00640 1 -- 172.21.15.6:0/1641921431 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f942c018070 con 0x7f943c06d8d0 2024-03-04T09:26:06.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.994+0000 7f9438c00640 1 --2- 172.21.15.6:0/1641921431 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f941806b520 0x7f941806d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:06.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.994+0000 7f9438c00640 1 -- 172.21.15.6:0/1641921431 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f942c08a2b0 con 0x7f943c06d8d0 2024-03-04T09:26:06.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.994+0000 7f9433e00640 1 --2- 172.21.15.6:0/1641921431 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f941806b520 0x7f941806d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:06.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.995+0000 7f9433e00640 1 --2- 172.21.15.6:0/1641921431 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f941806b520 0x7f941806d9e0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f9424036000 tx=0x7f9424008030 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:06.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:06.998+0000 7f9438c00640 1 -- 172.21.15.6:0/1641921431 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f942c059210 con 0x7f943c06d8d0 2024-03-04T09:26:07.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:07.193+0000 7f9441400640 1 -- 172.21.15.6:0/1641921431 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f943c13fe20 con 0x7f943c06d8d0 2024-03-04T09:26:07.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:07.194+0000 7f9438c00640 1 -- 172.21.15.6:0/1641921431 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7f942c058b10 con 0x7f943c06d8d0 2024-03-04T09:26:07.191 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 64 2024-03-04T09:26:07.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:07.197+0000 7f9441400640 1 -- 172.21.15.6:0/1641921431 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f941806b520 msgr2=0x7f941806d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:07.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:07.197+0000 7f9441400640 1 --2- 172.21.15.6:0/1641921431 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f941806b520 0x7f941806d9e0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f9424036000 tx=0x7f9424008030 comp rx=0 tx=0).stop 2024-03-04T09:26:07.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:07.197+0000 7f9441400640 1 -- 172.21.15.6:0/1641921431 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f943c06d8d0 msgr2=0x7f943c13f8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:07.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:07.197+0000 7f9441400640 1 --2- 172.21.15.6:0/1641921431 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f943c06d8d0 0x7f943c13f8c0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f942c00c7f0 tx=0x7f942c00ccb0 comp rx=0 tx=0).stop 2024-03-04T09:26:07.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:07.197+0000 7f9441400640 1 -- 172.21.15.6:0/1641921431 shutdown_connections 2024-03-04T09:26:07.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:07.197+0000 7f9441400640 1 --2- 172.21.15.6:0/1641921431 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f943c147590 0x7f943c133120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:07.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:07.197+0000 7f9441400640 1 --2- 172.21.15.6:0/1641921431 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f941806b520 0x7f941806d9e0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:07.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:07.197+0000 7f9441400640 1 --2- 172.21.15.6:0/1641921431 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f943c0b2080 0x7f943c132be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:07.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:07.197+0000 7f9441400640 1 --2- 172.21.15.6:0/1641921431 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f943c06d8d0 0x7f943c13f8c0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:07.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:07.197+0000 7f9441400640 1 -- 172.21.15.6:0/1641921431 >> 172.21.15.6:0/1641921431 conn(0x7f943c06b3a0 msgr2=0x7f943c06b780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:07.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:07.197+0000 7f9441400640 1 -- 172.21.15.6:0/1641921431 shutdown_connections 2024-03-04T09:26:07.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:07.197+0000 7f9441400640 1 -- 172.21.15.6:0/1641921431 wait complete. 2024-03-04T09:26:07.214 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-04T09:26:07.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:07.426+0000 7efe2ac00640 1 Processor -- start 2024-03-04T09:26:07.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:07.426+0000 7efe2ac00640 1 -- start start 2024-03-04T09:26:07.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:07.427+0000 7efe2ac00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efe2415e0b0 0x7efe240cf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:07.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:07.427+0000 7efe2ac00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7efe240d4a10 0x7efe240d0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:07.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:07.427+0000 7efe2ac00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7efe240d0c80 0x7efe240d1900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:07.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:07.427+0000 7efe2ac00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe240b09e0 con 0x7efe240d0c80 2024-03-04T09:26:07.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:07.427+0000 7efe2ac00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe240b3710 con 0x7efe240d4a10 2024-03-04T09:26:07.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:07.427+0000 7efe2ac00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe240b39f0 con 0x7efe2415e0b0 2024-03-04T09:26:07.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:07.428+0000 7efe2a200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7efe240d0c80 0x7efe240d1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:07.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:07.428+0000 7efe2a200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7efe240d0c80 0x7efe240d1900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:46808/0 (socket says 172.21.15.6:46808) 2024-03-04T09:26:07.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:07.428+0000 7efe2a200640 1 -- 172.21.15.6:0/4130838439 learned_addr learned my addr 172.21.15.6:0/4130838439 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:07.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:07.428+0000 7efe29800640 1 --2- 172.21.15.6:0/4130838439 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efe2415e0b0 0x7efe240cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:07.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:07.428+0000 7efe28e00640 1 --2- 172.21.15.6:0/4130838439 >> v2:172.21.15.146:3300/0 conn(0x7efe240d4a10 0x7efe240d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:07.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:07.428+0000 7efe2a200640 1 -- 172.21.15.6:0/4130838439 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efe2415e0b0 msgr2=0x7efe240cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:07.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:07.428+0000 7efe2a200640 1 --2- 172.21.15.6:0/4130838439 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efe2415e0b0 0x7efe240cf310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:07.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:07.428+0000 7efe2a200640 1 -- 172.21.15.6:0/4130838439 >> v2:172.21.15.146:3300/0 conn(0x7efe240d4a10 msgr2=0x7efe240d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:07.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:07.428+0000 7efe2a200640 1 --2- 172.21.15.6:0/4130838439 >> v2:172.21.15.146:3300/0 conn(0x7efe240d4a10 0x7efe240d0740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:07.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:07.428+0000 7efe2a200640 1 -- 172.21.15.6:0/4130838439 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe240b3cd0 con 0x7efe240d0c80 2024-03-04T09:26:07.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:07.428+0000 7efe28e00640 1 --2- 172.21.15.6:0/4130838439 >> v2:172.21.15.146:3300/0 conn(0x7efe240d4a10 0x7efe240d0740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:26:07.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:07.429+0000 7efe2a200640 1 --2- 172.21.15.6:0/4130838439 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efe240d0c80 0x7efe240d1900 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7efe240b6920 tx=0x7efe1c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7640ea41e6a53e9f server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:07.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:07.430+0000 7efe23e00640 1 -- 172.21.15.6:0/4130838439 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe1c02ca70 con 0x7efe240d0c80 2024-03-04T09:26:07.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:07.430+0000 7efe23e00640 1 -- 172.21.15.6:0/4130838439 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe1c02cc10 con 0x7efe240d0c80 2024-03-04T09:26:07.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:07.430+0000 7efe23e00640 1 -- 172.21.15.6:0/4130838439 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe1c0343f0 con 0x7efe240d0c80 2024-03-04T09:26:07.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:07.430+0000 7efe2ac00640 1 -- 172.21.15.6:0/4130838439 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efe240d0c80 msgr2=0x7efe240d1900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:07.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:07.430+0000 7efe2ac00640 1 --2- 172.21.15.6:0/4130838439 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efe240d0c80 0x7efe240d1900 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7efe240b6920 tx=0x7efe1c02bed0 comp rx=0 tx=0).stop 2024-03-04T09:26:07.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:07.431+0000 7efe2ac00640 1 -- 172.21.15.6:0/4130838439 shutdown_connections 2024-03-04T09:26:07.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:07.431+0000 7efe2ac00640 1 --2- 172.21.15.6:0/4130838439 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efe240d0c80 0x7efe240d1900 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:07.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:07.431+0000 7efe2ac00640 1 --2- 172.21.15.6:0/4130838439 >> v2:172.21.15.146:3300/0 conn(0x7efe240d4a10 0x7efe240d0740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:07.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:07.431+0000 7efe2ac00640 1 --2- 172.21.15.6:0/4130838439 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efe2415e0b0 0x7efe240cf310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:07.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:07.431+0000 7efe2ac00640 1 -- 172.21.15.6:0/4130838439 >> 172.21.15.6:0/4130838439 conn(0x7efe240d3d30 msgr2=0x7efe240d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:07.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:07.431+0000 7efe2ac00640 1 -- 172.21.15.6:0/4130838439 shutdown_connections 2024-03-04T09:26:07.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:07.431+0000 7efe2ac00640 1 -- 172.21.15.6:0/4130838439 wait complete. 2024-03-04T09:26:07.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:07.432+0000 7efe2ac00640 1 Processor -- start 2024-03-04T09:26:07.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:07.432+0000 7efe2ac00640 1 -- start start 2024-03-04T09:26:07.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:07.433+0000 7efe2ac00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efe240d0c80 0x7efe2415ad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:07.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:07.433+0000 7efe2ac00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efe240d4a10 0x7efe240b8ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:07.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:07.433+0000 7efe2ac00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efe2415e0b0 0x7efe24160620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:07.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:07.433+0000 7efe2ac00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe240cfc60 con 0x7efe240d0c80 2024-03-04T09:26:07.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:07.433+0000 7efe2ac00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe2415e490 con 0x7efe240d4a10 2024-03-04T09:26:07.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:07.433+0000 7efe2ac00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe240cf850 con 0x7efe2415e0b0 2024-03-04T09:26:07.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:07.434+0000 7efe29800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efe240d0c80 0x7efe2415ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:07.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:07.434+0000 7efe29800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efe240d0c80 0x7efe2415ad60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:46824/0 (socket says 172.21.15.6:46824) 2024-03-04T09:26:07.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:07.434+0000 7efe29800640 1 -- 172.21.15.6:0/3465613022 learned_addr learned my addr 172.21.15.6:0/3465613022 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:07.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:07.434+0000 7efe2a200640 1 --2- 172.21.15.6:0/3465613022 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efe2415e0b0 0x7efe24160620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:07.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:07.434+0000 7efe29800640 1 -- 172.21.15.6:0/3465613022 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efe2415e0b0 msgr2=0x7efe24160620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:07.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:07.434+0000 7efe29800640 1 --2- 172.21.15.6:0/3465613022 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efe2415e0b0 0x7efe24160620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:07.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:07.434+0000 7efe29800640 1 -- 172.21.15.6:0/3465613022 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efe240d4a10 msgr2=0x7efe240b8ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:07.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:07.434+0000 7efe29800640 1 --2- 172.21.15.6:0/3465613022 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efe240d4a10 0x7efe240b8ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:07.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:07.434+0000 7efe29800640 1 -- 172.21.15.6:0/3465613022 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe240cb3d0 con 0x7efe240d0c80 2024-03-04T09:26:07.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:07.434+0000 7efe29800640 1 --2- 172.21.15.6:0/3465613022 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efe240d0c80 0x7efe2415ad60 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7efe1800d620 tx=0x7efe1800dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:07.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:07.434+0000 7efe21600640 1 -- 172.21.15.6:0/3465613022 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe1800b620 con 0x7efe240d0c80 2024-03-04T09:26:07.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:07.435+0000 7efe21600640 1 -- 172.21.15.6:0/3465613022 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe18019070 con 0x7efe240d0c80 2024-03-04T09:26:07.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:07.435+0000 7efe21600640 1 -- 172.21.15.6:0/3465613022 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe1800ba00 con 0x7efe240d0c80 2024-03-04T09:26:07.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:07.435+0000 7efe2ac00640 1 -- 172.21.15.6:0/3465613022 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe240cb6b0 con 0x7efe240d0c80 2024-03-04T09:26:07.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:07.435+0000 7efe2ac00640 1 -- 172.21.15.6:0/3465613022 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe240cbc70 con 0x7efe240d0c80 2024-03-04T09:26:07.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:07.436+0000 7efe2ac00640 1 -- 172.21.15.6:0/3465613022 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe2404f330 con 0x7efe240d0c80 2024-03-04T09:26:07.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:07.437+0000 7efe21600640 1 -- 172.21.15.6:0/3465613022 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efe1800bba0 con 0x7efe240d0c80 2024-03-04T09:26:07.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:07.438+0000 7efe21600640 1 --2- 172.21.15.6:0/3465613022 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7efe0406b4d0 0x7efe0406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:07.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:07.438+0000 7efe21600640 1 -- 172.21.15.6:0/3465613022 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7efe1808a610 con 0x7efe240d0c80 2024-03-04T09:26:07.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:07.438+0000 7efe28e00640 1 --2- 172.21.15.6:0/3465613022 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7efe0406b4d0 0x7efe0406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:07.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:07.439+0000 7efe28e00640 1 --2- 172.21.15.6:0/3465613022 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7efe0406b4d0 0x7efe0406d990 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7efe10006fd0 tx=0x7efe10008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:07.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:07.442+0000 7efe21600640 1 -- 172.21.15.6:0/3465613022 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7efe18059490 con 0x7efe240d0c80 2024-03-04T09:26:07.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:07.651+0000 7efe2ac00640 1 -- 172.21.15.6:0/3465613022 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7efe240b5860 con 0x7efe240d0c80 2024-03-04T09:26:07.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:07.652+0000 7efe21600640 1 -- 172.21.15.6:0/3465613022 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v22) v1 ==== 80+0+0 (secure 0 0 0) 0x7efe18058d90 con 0x7efe240d0c80 2024-03-04T09:26:07.649 INFO:teuthology.orchestra.run.smithi006.stderr:listed 0 entries 2024-03-04T09:26:07.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:07.654+0000 7efe2ac00640 1 -- 172.21.15.6:0/3465613022 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7efe0406b4d0 msgr2=0x7efe0406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:07.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:07.654+0000 7efe2ac00640 1 --2- 172.21.15.6:0/3465613022 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7efe0406b4d0 0x7efe0406d990 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7efe10006fd0 tx=0x7efe10008040 comp rx=0 tx=0).stop 2024-03-04T09:26:07.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:07.654+0000 7efe2ac00640 1 -- 172.21.15.6:0/3465613022 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efe240d0c80 msgr2=0x7efe2415ad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:07.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:07.654+0000 7efe2ac00640 1 --2- 172.21.15.6:0/3465613022 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efe240d0c80 0x7efe2415ad60 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7efe1800d620 tx=0x7efe1800dae0 comp rx=0 tx=0).stop 2024-03-04T09:26:07.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:07.655+0000 7efe2ac00640 1 -- 172.21.15.6:0/3465613022 shutdown_connections 2024-03-04T09:26:07.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:07.655+0000 7efe2ac00640 1 --2- 172.21.15.6:0/3465613022 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7efe0406b4d0 0x7efe0406d990 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:07.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:07.655+0000 7efe2ac00640 1 --2- 172.21.15.6:0/3465613022 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efe2415e0b0 0x7efe24160620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:07.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:07.655+0000 7efe2ac00640 1 --2- 172.21.15.6:0/3465613022 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efe240d4a10 0x7efe240b8ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:07.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:07.655+0000 7efe2ac00640 1 --2- 172.21.15.6:0/3465613022 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efe240d0c80 0x7efe2415ad60 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:07.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:07.655+0000 7efe2ac00640 1 -- 172.21.15.6:0/3465613022 >> 172.21.15.6:0/3465613022 conn(0x7efe240d3d30 msgr2=0x7efe240b0b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:07.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:07.655+0000 7efe2ac00640 1 -- 172.21.15.6:0/3465613022 shutdown_connections 2024-03-04T09:26:07.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:07.655+0000 7efe2ac00640 1 -- 172.21.15.6:0/3465613022 wait complete. 2024-03-04T09:26:07.668 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2024-03-04T09:26:07.668 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:26:07.668 DEBUG:teuthology.orchestra.run.smithi146:> sudo umount /home/ubuntu/cephtest/mnt.1 -f 2024-03-04T09:26:07.762 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi146.front.sepia.ceph.com 2024-03-04T09:26:07.762 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:26:07.763 DEBUG:teuthology.orchestra.run.smithi146:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-04T09:26:07.816 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-04T09:26:07.816 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T09:26:07.872 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-04T09:26:07.872 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T09:26:07.922 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-04T09:26:07.922 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T09:26:08.023 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T09:26:08.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.252+0000 7f4418200640 1 Processor -- start 2024-03-04T09:26:08.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.252+0000 7f4418200640 1 -- start start 2024-03-04T09:26:08.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.253+0000 7f4418200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f4410161670 0x7f4410070c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:08.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.254+0000 7f4418200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f441006e240 0x7f44100711b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:08.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.254+0000 7f4418200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44100ce9a0 con 0x7f441006cfe0 2024-03-04T09:26:08.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.254+0000 7f4418200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44100730d0 con 0x7f441006e240 2024-03-04T09:26:08.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.254+0000 7f4418200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44100cec80 con 0x7f4410161670 2024-03-04T09:26:08.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.254+0000 7f4414e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f441006e240 0x7f44100711b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:08.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.254+0000 7f4414e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f441006e240 0x7f44100711b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:46836/0 (socket says 172.21.15.6:46836) 2024-03-04T09:26:08.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.254+0000 7f4414e00640 1 -- 172.21.15.6:0/312143925 learned_addr learned my addr 172.21.15.6:0/312143925 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:08.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.254+0000 7f4415800640 1 --2- 172.21.15.6:0/312143925 >> v2:172.21.15.146:3300/0 conn(0x7f4410161670 0x7f4410070c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:08.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.255+0000 7f4414e00640 1 -- 172.21.15.6:0/312143925 >> v2:172.21.15.146:3300/0 conn(0x7f4410161670 msgr2=0x7f4410070c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:08.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.255+0000 7f4414e00640 1 --2- 172.21.15.6:0/312143925 >> v2:172.21.15.146:3300/0 conn(0x7f4410161670 0x7f4410070c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:08.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.255+0000 7f4414e00640 1 -- 172.21.15.6:0/312143925 >> v1:172.21.15.146:6789/0 conn(0x7f441006cfe0 legacy=0x7f441006d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:08.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.255+0000 7f4414e00640 1 -- 172.21.15.6:0/312143925 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44100cef60 con 0x7f441006e240 2024-03-04T09:26:08.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.256+0000 7f4414e00640 1 --2- 172.21.15.6:0/312143925 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f441006e240 0x7f44100711b0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f4410073610 tx=0x7f43fc02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74d1f3493430afd2 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:08.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.256+0000 7f440fe00640 1 -- 172.21.15.6:0/312143925 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43fc02e070 con 0x7f441006e240 2024-03-04T09:26:08.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.256+0000 7f440fe00640 1 -- 172.21.15.6:0/312143925 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43fc037070 con 0x7f441006e240 2024-03-04T09:26:08.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.256+0000 7f440fe00640 1 -- 172.21.15.6:0/312143925 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43fc031a00 con 0x7f441006e240 2024-03-04T09:26:08.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.257+0000 7f4418200640 1 -- 172.21.15.6:0/312143925 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f441006e240 msgr2=0x7f44100711b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:08.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.257+0000 7f4418200640 1 --2- 172.21.15.6:0/312143925 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f441006e240 0x7f44100711b0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f4410073610 tx=0x7f43fc02b880 comp rx=0 tx=0).stop 2024-03-04T09:26:08.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.257+0000 7f4418200640 1 -- 172.21.15.6:0/312143925 shutdown_connections 2024-03-04T09:26:08.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.257+0000 7f4418200640 1 --2- 172.21.15.6:0/312143925 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f441006e240 0x7f44100711b0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:08.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.257+0000 7f4418200640 1 --2- 172.21.15.6:0/312143925 >> v2:172.21.15.146:3300/0 conn(0x7f4410161670 0x7f4410070c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:08.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.257+0000 7f4418200640 1 -- 172.21.15.6:0/312143925 >> 172.21.15.6:0/312143925 conn(0x7f441006afb0 msgr2=0x7f441006b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:08.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.257+0000 7f4418200640 1 -- 172.21.15.6:0/312143925 shutdown_connections 2024-03-04T09:26:08.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.257+0000 7f4418200640 1 -- 172.21.15.6:0/312143925 wait complete. 2024-03-04T09:26:08.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.259+0000 7f4418200640 1 Processor -- start 2024-03-04T09:26:08.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.260+0000 7f4418200640 1 -- start start 2024-03-04T09:26:08.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.261+0000 7f4418200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f441006cfe0 0x7f4410145790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:08.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.261+0000 7f4418200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f441006e240 0x7f441009aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:08.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.261+0000 7f4418200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4410161670 0x7f441009af50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:08.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.261+0000 7f4418200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44100ce600 con 0x7f441006cfe0 2024-03-04T09:26:08.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.261+0000 7f4418200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44100cf520 con 0x7f4410161670 2024-03-04T09:26:08.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.261+0000 7f4418200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44100ce780 con 0x7f441006e240 2024-03-04T09:26:08.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.261+0000 7f4415800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f441006cfe0 0x7f4410145790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:08.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.261+0000 7f4415800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f441006cfe0 0x7f4410145790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:46838/0 (socket says 172.21.15.6:46838) 2024-03-04T09:26:08.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.261+0000 7f4415800640 1 -- 172.21.15.6:0/473304089 learned_addr learned my addr 172.21.15.6:0/473304089 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:08.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.261+0000 7f4414e00640 1 --2- 172.21.15.6:0/473304089 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f441006e240 0x7f441009aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:08.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.262+0000 7f4415800640 1 -- 172.21.15.6:0/473304089 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f441006e240 msgr2=0x7f441009aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:08.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.262+0000 7f4415800640 1 --2- 172.21.15.6:0/473304089 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f441006e240 0x7f441009aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:08.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.262+0000 7f4415800640 1 -- 172.21.15.6:0/473304089 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4410161670 msgr2=0x7f441009af50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:08.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.262+0000 7f4416200640 1 --2- 172.21.15.6:0/473304089 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4410161670 0x7f441009af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:08.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.262+0000 7f4415800640 1 --2- 172.21.15.6:0/473304089 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4410161670 0x7f441009af50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:08.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.262+0000 7f4415800640 1 -- 172.21.15.6:0/473304089 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44100af3a0 con 0x7f441006cfe0 2024-03-04T09:26:08.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.262+0000 7f4416200640 1 --2- 172.21.15.6:0/473304089 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4410161670 0x7f441009af50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:26:08.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.262+0000 7f4415800640 1 --2- 172.21.15.6:0/473304089 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f441006cfe0 0x7f4410145790 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f440400d5f0 tx=0x7f440400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:08.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.262+0000 7f440d600640 1 -- 172.21.15.6:0/473304089 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44040125f0 con 0x7f441006cfe0 2024-03-04T09:26:08.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.262+0000 7f440d600640 1 -- 172.21.15.6:0/473304089 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4404019070 con 0x7f441006cfe0 2024-03-04T09:26:08.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.262+0000 7f440d600640 1 -- 172.21.15.6:0/473304089 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44040129d0 con 0x7f441006cfe0 2024-03-04T09:26:08.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.262+0000 7f4418200640 1 -- 172.21.15.6:0/473304089 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44100af680 con 0x7f441006cfe0 2024-03-04T09:26:08.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.263+0000 7f4418200640 1 -- 172.21.15.6:0/473304089 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44100afc40 con 0x7f441006cfe0 2024-03-04T09:26:08.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.264+0000 7f4418200640 1 -- 172.21.15.6:0/473304089 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4410069510 con 0x7f441006cfe0 2024-03-04T09:26:08.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.265+0000 7f440d600640 1 -- 172.21.15.6:0/473304089 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4404012b70 con 0x7f441006cfe0 2024-03-04T09:26:08.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.266+0000 7f440d600640 1 --2- 172.21.15.6:0/473304089 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f43f006b500 0x7f43f006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:08.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.266+0000 7f440d600640 1 -- 172.21.15.6:0/473304089 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f440408a750 con 0x7f441006cfe0 2024-03-04T09:26:08.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.267+0000 7f4414e00640 1 --2- 172.21.15.6:0/473304089 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f43f006b500 0x7f43f006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:08.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.268+0000 7f4414e00640 1 --2- 172.21.15.6:0/473304089 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f43f006b500 0x7f43f006d9c0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f43fc036000 tx=0x7f43fc03d040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:08.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.272+0000 7f440d600640 1 -- 172.21.15.6:0/473304089 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f44040594a0 con 0x7f441006cfe0 2024-03-04T09:26:08.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.476+0000 7f4418200640 1 -- 172.21.15.6:0/473304089 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f441004f3a0 con 0x7f441006cfe0 2024-03-04T09:26:08.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.478+0000 7f440d600640 1 -- 172.21.15.6:0/473304089 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16852 (secure 0 0 0) 0x7f4404058da0 con 0x7f441006cfe0 2024-03-04T09:26:08.476 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:26:08.476 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":22,"fsid":"edce0578-5ae4-4402-b45a-552bdef84d7c","created":"2024-03-04T09:24:54.523780+0000","modified":"2024-03-04T09:26:01.727682+0000","last_up_change":"2024-03-04T09:25:07.089049+0000","last_in_change":"2024-03-04T09:24:57.982077+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T09:25:07.935198+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-04T09:25:17.118573+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-04T09:25:17.930536+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a96574f-81f1-4bc6-8463-757520b84a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6825","nonce":1364449401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6827","nonce":1364449401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6831","nonce":1364449401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6829","nonce":1364449401}]},"public_addr":"172.21.15.6:6825/1364449401","cluster_addr":"172.21.15.6:6827/1364449401","heartbeat_back_addr":"172.21.15.6:6831/1364449401","heartbeat_front_addr":"172.21.15.6:6829/1364449401","state":["exists","up"]},{"osd":1,"uuid":"7d79b9c1-6b87-476a-bc36-7db5ee64cd90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6809","nonce":2494103078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6811","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6813","nonce":2494103078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6821","nonce":2494103078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6817","nonce":2494103078}]},"public_addr":"172.21.15.6:6809/2494103078","cluster_addr":"172.21.15.6:6813/2494103078","heartbeat_back_addr":"172.21.15.6:6821/2494103078","heartbeat_front_addr":"172.21.15.6:6817/2494103078","state":["exists","up"]},{"osd":2,"uuid":"492331b9-4ee9-40b6-bacd-85d44cd6c45a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6812","nonce":1500066681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6816","nonce":1500066681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6824","nonce":1500066681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6820","nonce":1500066681}]},"public_addr":"172.21.15.6:6812/1500066681","cluster_addr":"172.21.15.6:6816/1500066681","heartbeat_back_addr":"172.21.15.6:6824/1500066681","heartbeat_front_addr":"172.21.15.6:6820/1500066681","state":["exists","up"]},{"osd":3,"uuid":"21bb2efe-9ba7-4bdc-bc31-180cd1102e21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6801","nonce":1558806123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6803","nonce":1558806123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6807","nonce":1558806123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6805","nonce":1558806123}]},"public_addr":"172.21.15.6:6801/1558806123","cluster_addr":"172.21.15.6:6803/1558806123","heartbeat_back_addr":"172.21.15.6:6807/1558806123","heartbeat_front_addr":"172.21.15.6:6805/1558806123","state":["exists","up"]},{"osd":4,"uuid":"6e2c4b71-19d9-4737-83d9-558705d8644a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6801","nonce":1526135780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6803","nonce":1526135780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6807","nonce":1526135780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6805","nonce":1526135780}]},"public_addr":"172.21.15.146:6801/1526135780","cluster_addr":"172.21.15.146:6803/1526135780","heartbeat_back_addr":"172.21.15.146:6807/1526135780","heartbeat_front_addr":"172.21.15.146:6805/1526135780","state":["exists","up"]},{"osd":5,"uuid":"fb69c525-f1be-4b7d-ab32-6c29b0d6fddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6809","nonce":460262507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6811","nonce":460262507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6815","nonce":460262507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6813","nonce":460262507}]},"public_addr":"172.21.15.146:6809/460262507","cluster_addr":"172.21.15.146:6811/460262507","heartbeat_back_addr":"172.21.15.146:6815/460262507","heartbeat_front_addr":"172.21.15.146:6813/460262507","state":["exists","up"]},{"osd":6,"uuid":"c32e7fed-370d-4401-8ce0-6a4335b0d168","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6823","nonce":1392897493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6827","nonce":1392897493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6831","nonce":1392897493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6829","nonce":1392897493}]},"public_addr":"172.21.15.146:6823/1392897493","cluster_addr":"172.21.15.146:6827/1392897493","heartbeat_back_addr":"172.21.15.146:6831/1392897493","heartbeat_front_addr":"172.21.15.146:6829/1392897493","state":["exists","up"]},{"osd":7,"uuid":"75a79b44-fa5f-426d-a7d3-71f21f78835c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6817","nonce":3268323156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6819","nonce":3268323156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6825","nonce":3268323156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6822","nonce":3268323156}]},"public_addr":"172.21.15.146:6817/3268323156","cluster_addr":"172.21.15.146:6819/3268323156","heartbeat_back_addr":"172.21.15.146:6825/3268323156","heartbeat_front_addr":"172.21.15.146:6822/3268323156","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.310437+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.295850+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.294674+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.526855+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.300023+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.595160+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.346046+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.558741+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.7","mappings":[{"from":3,"to":2}]},{"pgid":"3.f","mappings":[{"from":7,"to":2}]},{"pgid":"3.21","mappings":[{"from":5,"to":1}]},{"pgid":"3.22","mappings":[{"from":5,"to":2}]},{"pgid":"3.28","mappings":[{"from":3,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T09:26:08.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.481+0000 7f4418200640 1 -- 172.21.15.6:0/473304089 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f43f006b500 msgr2=0x7f43f006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:08.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.481+0000 7f4418200640 1 --2- 172.21.15.6:0/473304089 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f43f006b500 0x7f43f006d9c0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f43fc036000 tx=0x7f43fc03d040 comp rx=0 tx=0).stop 2024-03-04T09:26:08.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.481+0000 7f4418200640 1 -- 172.21.15.6:0/473304089 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f441006cfe0 msgr2=0x7f4410145790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:08.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.481+0000 7f4418200640 1 --2- 172.21.15.6:0/473304089 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f441006cfe0 0x7f4410145790 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f440400d5f0 tx=0x7f440400dab0 comp rx=0 tx=0).stop 2024-03-04T09:26:08.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.482+0000 7f4418200640 1 -- 172.21.15.6:0/473304089 shutdown_connections 2024-03-04T09:26:08.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.482+0000 7f4418200640 1 --2- 172.21.15.6:0/473304089 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4410161670 0x7f441009af50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:08.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.482+0000 7f4418200640 1 --2- 172.21.15.6:0/473304089 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f43f006b500 0x7f43f006d9c0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:08.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.482+0000 7f4418200640 1 --2- 172.21.15.6:0/473304089 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f441006e240 0x7f441009aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:08.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.482+0000 7f4418200640 1 --2- 172.21.15.6:0/473304089 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f441006cfe0 0x7f4410145790 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:08.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.482+0000 7f4418200640 1 -- 172.21.15.6:0/473304089 >> 172.21.15.6:0/473304089 conn(0x7f441006afb0 msgr2=0x7f44100ae0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:08.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.482+0000 7f4418200640 1 -- 172.21.15.6:0/473304089 shutdown_connections 2024-03-04T09:26:08.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.482+0000 7f4418200640 1 -- 172.21.15.6:0/473304089 wait complete. 2024-03-04T09:26:08.498 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T09:25:07.935198+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-04T09:25:17.118573+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-04T09:25:17.930536+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-04T09:26:08.498 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-04T09:26:08.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.725+0000 7f0b1fa00640 1 Processor -- start 2024-03-04T09:26:08.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.725+0000 7f0b1fa00640 1 -- start start 2024-03-04T09:26:08.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.727+0000 7f0b1fa00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f0b1815bf20 0x7f0b180acd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:08.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.727+0000 7f0b1fa00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0b18071e20 0x7f0b180ccad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:08.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.727+0000 7f0b1fa00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b18072df0 con 0x7f0b18072520 2024-03-04T09:26:08.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.727+0000 7f0b1fa00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b180cdab0 con 0x7f0b1815bf20 2024-03-04T09:26:08.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.727+0000 7f0b1fa00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b180b4d70 con 0x7f0b18071e20 2024-03-04T09:26:08.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.727+0000 7f0b1d000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f0b1815bf20 0x7f0b180acd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:08.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.727+0000 7f0b1d000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f0b1815bf20 0x7f0b180acd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:46854/0 (socket says 172.21.15.6:46854) 2024-03-04T09:26:08.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.727+0000 7f0b1d000640 1 -- 172.21.15.6:0/1325283166 learned_addr learned my addr 172.21.15.6:0/1325283166 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:08.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.728+0000 7f0b1da00640 1 --2- 172.21.15.6:0/1325283166 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0b18071e20 0x7f0b180ccad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:08.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.728+0000 7f0b1d000640 1 -- 172.21.15.6:0/1325283166 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0b18071e20 msgr2=0x7f0b180ccad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:08.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.728+0000 7f0b1d000640 1 --2- 172.21.15.6:0/1325283166 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0b18071e20 0x7f0b180ccad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:08.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.728+0000 7f0b1d000640 1 -- 172.21.15.6:0/1325283166 >> v1:172.21.15.6:6789/0 conn(0x7f0b18072520 legacy=0x7f0b180cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:08.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.728+0000 7f0b1d000640 1 -- 172.21.15.6:0/1325283166 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b180b5050 con 0x7f0b1815bf20 2024-03-04T09:26:08.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.729+0000 7f0b1d000640 1 --2- 172.21.15.6:0/1325283166 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0b1815bf20 0x7f0b180acd40 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f0b0c009870 tx=0x7f0b0c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1dc67b6c2c88930f server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:08.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.729+0000 7f0b17400640 1 -- 172.21.15.6:0/1325283166 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b0c034070 con 0x7f0b1815bf20 2024-03-04T09:26:08.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.729+0000 7f0b17400640 1 -- 172.21.15.6:0/1325283166 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b0c030840 con 0x7f0b1815bf20 2024-03-04T09:26:08.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.729+0000 7f0b17400640 1 -- 172.21.15.6:0/1325283166 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b0c030c10 con 0x7f0b1815bf20 2024-03-04T09:26:08.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.730+0000 7f0b1fa00640 1 -- 172.21.15.6:0/1325283166 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0b1815bf20 msgr2=0x7f0b180acd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:08.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.730+0000 7f0b1fa00640 1 --2- 172.21.15.6:0/1325283166 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0b1815bf20 0x7f0b180acd40 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f0b0c009870 tx=0x7f0b0c02b980 comp rx=0 tx=0).stop 2024-03-04T09:26:08.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.730+0000 7f0b1fa00640 1 -- 172.21.15.6:0/1325283166 shutdown_connections 2024-03-04T09:26:08.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.730+0000 7f0b1fa00640 1 --2- 172.21.15.6:0/1325283166 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0b18071e20 0x7f0b180ccad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:08.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.730+0000 7f0b1fa00640 1 --2- 172.21.15.6:0/1325283166 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0b1815bf20 0x7f0b180acd40 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:08.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.730+0000 7f0b1fa00640 1 -- 172.21.15.6:0/1325283166 >> 172.21.15.6:0/1325283166 conn(0x7f0b1806b3d0 msgr2=0x7f0b1806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:08.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.730+0000 7f0b1fa00640 1 -- 172.21.15.6:0/1325283166 shutdown_connections 2024-03-04T09:26:08.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.730+0000 7f0b1fa00640 1 -- 172.21.15.6:0/1325283166 wait complete. 2024-03-04T09:26:08.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.732+0000 7f0b1fa00640 1 Processor -- start 2024-03-04T09:26:08.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.732+0000 7f0b1fa00640 1 -- start start 2024-03-04T09:26:08.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.733+0000 7f0b1fa00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0b18071e20 0x7f0b1806dc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:08.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.733+0000 7f0b1fa00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0b18072520 0x7f0b1815cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:08.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.733+0000 7f0b1fa00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0b1815bf20 0x7f0b1815d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:08.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.733+0000 7f0b1fa00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b180b4b10 con 0x7f0b18072520 2024-03-04T09:26:08.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.733+0000 7f0b1fa00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b180b4990 con 0x7f0b18071e20 2024-03-04T09:26:08.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.733+0000 7f0b1fa00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b180ce6c0 con 0x7f0b1815bf20 2024-03-04T09:26:08.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.733+0000 7f0b17e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0b18072520 0x7f0b1815cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:08.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.733+0000 7f0b17e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0b18072520 0x7f0b1815cdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:46860/0 (socket says 172.21.15.6:46860) 2024-03-04T09:26:08.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.733+0000 7f0b17e00640 1 -- 172.21.15.6:0/143042160 learned_addr learned my addr 172.21.15.6:0/143042160 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:08.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.733+0000 7f0b1da00640 1 --2- 172.21.15.6:0/143042160 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0b1815bf20 0x7f0b1815d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:08.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.734+0000 7f0b17e00640 1 -- 172.21.15.6:0/143042160 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0b1815bf20 msgr2=0x7f0b1815d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:08.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.734+0000 7f0b17e00640 1 --2- 172.21.15.6:0/143042160 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0b1815bf20 0x7f0b1815d300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:08.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.734+0000 7f0b17e00640 1 -- 172.21.15.6:0/143042160 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0b18071e20 msgr2=0x7f0b1806dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:08.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.734+0000 7f0b17e00640 1 --2- 172.21.15.6:0/143042160 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0b18071e20 0x7f0b1806dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:08.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.734+0000 7f0b17e00640 1 -- 172.21.15.6:0/143042160 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b1815c300 con 0x7f0b18072520 2024-03-04T09:26:08.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.734+0000 7f0b17e00640 1 --2- 172.21.15.6:0/143042160 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0b18072520 0x7f0b1815cdc0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f0b0400c3c0 tx=0x7f0b0400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:08.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.734+0000 7f0b14c00640 1 -- 172.21.15.6:0/143042160 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b040052d0 con 0x7f0b18072520 2024-03-04T09:26:08.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.734+0000 7f0b14c00640 1 -- 172.21.15.6:0/143042160 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b04005490 con 0x7f0b18072520 2024-03-04T09:26:08.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.734+0000 7f0b14c00640 1 -- 172.21.15.6:0/143042160 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b04007260 con 0x7f0b18072520 2024-03-04T09:26:08.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.734+0000 7f0b1fa00640 1 -- 172.21.15.6:0/143042160 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b1809d3d0 con 0x7f0b18072520 2024-03-04T09:26:08.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.734+0000 7f0b1fa00640 1 -- 172.21.15.6:0/143042160 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b1815c5c0 con 0x7f0b18072520 2024-03-04T09:26:08.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.736+0000 7f0b1fa00640 1 -- 172.21.15.6:0/143042160 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b1804f330 con 0x7f0b18072520 2024-03-04T09:26:08.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.737+0000 7f0b14c00640 1 -- 172.21.15.6:0/143042160 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0b04007400 con 0x7f0b18072520 2024-03-04T09:26:08.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.738+0000 7f0b14c00640 1 --2- 172.21.15.6:0/143042160 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f0af806b4d0 0x7f0af806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:08.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.738+0000 7f0b14c00640 1 -- 172.21.15.6:0/143042160 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f0b040896a0 con 0x7f0b18072520 2024-03-04T09:26:08.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.738+0000 7f0b1d000640 1 --2- 172.21.15.6:0/143042160 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f0af806b4d0 0x7f0af806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:08.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.739+0000 7f0b1d000640 1 --2- 172.21.15.6:0/143042160 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f0af806b4d0 0x7f0af806d990 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f0b0c036000 tx=0x7f0b0c030000 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:08.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.742+0000 7f0b14c00640 1 -- 172.21.15.6:0/143042160 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0b04058520 con 0x7f0b18072520 2024-03-04T09:26:08.747 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:26:08.751 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:26:08.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.944+0000 7f0b1fa00640 1 -- 172.21.15.6:0/143042160 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0b1815f720 con 0x7f0b18072520 2024-03-04T09:26:08.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.946+0000 7f0b14c00640 1 -- 172.21.15.6:0/143042160 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0b04057e20 con 0x7f0b18072520 2024-03-04T09:26:08.943 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-04T09:26:08.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.948+0000 7f0b1fa00640 1 -- 172.21.15.6:0/143042160 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f0af806b4d0 msgr2=0x7f0af806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:08.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.948+0000 7f0b1fa00640 1 --2- 172.21.15.6:0/143042160 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f0af806b4d0 0x7f0af806d990 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f0b0c036000 tx=0x7f0b0c030000 comp rx=0 tx=0).stop 2024-03-04T09:26:08.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.948+0000 7f0b1fa00640 1 -- 172.21.15.6:0/143042160 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0b18072520 msgr2=0x7f0b1815cdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:08.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.948+0000 7f0b1fa00640 1 --2- 172.21.15.6:0/143042160 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0b18072520 0x7f0b1815cdc0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f0b0400c3c0 tx=0x7f0b0400c880 comp rx=0 tx=0).stop 2024-03-04T09:26:08.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.948+0000 7f0b1fa00640 1 -- 172.21.15.6:0/143042160 shutdown_connections 2024-03-04T09:26:08.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.948+0000 7f0b1fa00640 1 --2- 172.21.15.6:0/143042160 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f0af806b4d0 0x7f0af806d990 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:08.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.948+0000 7f0b1fa00640 1 --2- 172.21.15.6:0/143042160 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0b1815bf20 0x7f0b1815d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:08.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.948+0000 7f0b1fa00640 1 --2- 172.21.15.6:0/143042160 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0b18072520 0x7f0b1815cdc0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:08.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.948+0000 7f0b1fa00640 1 --2- 172.21.15.6:0/143042160 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0b18071e20 0x7f0b1806dc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:08.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.948+0000 7f0b1fa00640 1 -- 172.21.15.6:0/143042160 >> 172.21.15.6:0/143042160 conn(0x7f0b1806b3d0 msgr2=0x7f0b18159b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:08.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.949+0000 7f0b1fa00640 1 -- 172.21.15.6:0/143042160 shutdown_connections 2024-03-04T09:26:08.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:08.949+0000 7f0b1fa00640 1 -- 172.21.15.6:0/143042160 wait complete. 2024-03-04T09:26:08.966 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-04T09:26:09.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.194+0000 7f0f04e00640 1 Processor -- start 2024-03-04T09:26:09.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.194+0000 7f0f04e00640 1 -- start start 2024-03-04T09:26:09.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.195+0000 7f0f04e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f0f0015e1a0 0x7f0f000cf350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:09.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.195+0000 7f0f04e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f0f000cf890 0x7f0f000d22d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:09.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.196+0000 7f0f04e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f000d1d70 con 0x7f0f000d2810 2024-03-04T09:26:09.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.196+0000 7f0f04e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f000cfcb0 con 0x7f0f0015e1a0 2024-03-04T09:26:09.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.196+0000 7f0f04e00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f000b77f0 con 0x7f0f000cf890 2024-03-04T09:26:09.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.196+0000 7f0eff400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f0f0015e1a0 0x7f0f000cf350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:09.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.196+0000 7f0effe00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f0f000d2810 0x7f0f000b4900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:60506/0 (socket says 172.21.15.6:60506) 2024-03-04T09:26:09.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.196+0000 7f0effe00640 1 -- 172.21.15.6:0/3475369088 learned_addr learned my addr 172.21.15.6:0/3475369088 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:26:09.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.196+0000 7f0efea00640 1 --2- 172.21.15.6:0/3475369088 >> v2:172.21.15.146:3300/0 conn(0x7f0f000cf890 0x7f0f000d22d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:09.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.197+0000 7f0efe000640 1 -- 172.21.15.6:0/3475369088 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2131836337 0 0) 0x7f0f000d1d70 con 0x7f0f000d2810 2024-03-04T09:26:09.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.197+0000 7f0efe000640 1 -- 172.21.15.6:0/3475369088 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ee4003520 con 0x7f0f000d2810 2024-03-04T09:26:09.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.197+0000 7f0eff400640 1 -- 172.21.15.6:0/3475369088 >> v2:172.21.15.146:3300/0 conn(0x7f0f000cf890 msgr2=0x7f0f000d22d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:09.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.197+0000 7f0eff400640 1 --2- 172.21.15.6:0/3475369088 >> v2:172.21.15.146:3300/0 conn(0x7f0f000cf890 0x7f0f000d22d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:09.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.197+0000 7f0eff400640 1 -- 172.21.15.6:0/3475369088 >> v1:172.21.15.6:6789/0 conn(0x7f0f000d2810 legacy=0x7f0f000b4900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:09.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.197+0000 7f0eff400640 1 -- 172.21.15.6:0/3475369088 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f000b6be0 con 0x7f0f0015e1a0 2024-03-04T09:26:09.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.198+0000 7f0eff400640 1 --2- 172.21.15.6:0/3475369088 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0f0015e1a0 0x7f0f000cf350 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f0f0008d010 tx=0x7f0ef002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40ab7d4972afdb08 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:09.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.198+0000 7f0efe000640 1 -- 172.21.15.6:0/3475369088 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ef0039070 con 0x7f0f0015e1a0 2024-03-04T09:26:09.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.198+0000 7f0efe000640 1 -- 172.21.15.6:0/3475369088 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ef002cc80 con 0x7f0f0015e1a0 2024-03-04T09:26:09.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.198+0000 7f0efe000640 1 -- 172.21.15.6:0/3475369088 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ef0034660 con 0x7f0f0015e1a0 2024-03-04T09:26:09.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.199+0000 7f0f04e00640 1 -- 172.21.15.6:0/3475369088 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0f0015e1a0 msgr2=0x7f0f000cf350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:09.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.199+0000 7f0f04e00640 1 --2- 172.21.15.6:0/3475369088 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0f0015e1a0 0x7f0f000cf350 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f0f0008d010 tx=0x7f0ef002bed0 comp rx=0 tx=0).stop 2024-03-04T09:26:09.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.199+0000 7f0f04e00640 1 -- 172.21.15.6:0/3475369088 shutdown_connections 2024-03-04T09:26:09.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.199+0000 7f0f04e00640 1 --2- 172.21.15.6:0/3475369088 >> v2:172.21.15.146:3300/0 conn(0x7f0f000cf890 0x7f0f000d22d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:09.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.199+0000 7f0f04e00640 1 --2- 172.21.15.6:0/3475369088 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0f0015e1a0 0x7f0f000cf350 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:09.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.199+0000 7f0f04e00640 1 -- 172.21.15.6:0/3475369088 >> 172.21.15.6:0/3475369088 conn(0x7f0f0008d2a0 msgr2=0x7f0f000d3cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:09.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.199+0000 7f0f04e00640 1 -- 172.21.15.6:0/3475369088 shutdown_connections 2024-03-04T09:26:09.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.199+0000 7f0f04e00640 1 -- 172.21.15.6:0/3475369088 wait complete. 2024-03-04T09:26:09.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.201+0000 7f0f04e00640 1 Processor -- start 2024-03-04T09:26:09.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.201+0000 7f0f04e00640 1 -- start start 2024-03-04T09:26:09.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.202+0000 7f0f04e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0f000cf890 0x7f0f000cd200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:09.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.202+0000 7f0f04e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0f000d2810 0x7f0f000a1160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:09.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.202+0000 7f0f04e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0f0015e1a0 0x7f0f0015fab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:09.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.202+0000 7f0f04e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f000b71a0 con 0x7f0f000d2810 2024-03-04T09:26:09.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.202+0000 7f0f04e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f0015e580 con 0x7f0f000cf890 2024-03-04T09:26:09.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.202+0000 7f0f04e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f000b7480 con 0x7f0f0015e1a0 2024-03-04T09:26:09.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.202+0000 7f0efea00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0f000d2810 0x7f0f000a1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:09.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.202+0000 7f0efea00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0f000d2810 0x7f0f000a1160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:59660/0 (socket says 172.21.15.6:59660) 2024-03-04T09:26:09.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.202+0000 7f0efea00640 1 -- 172.21.15.6:0/3140646518 learned_addr learned my addr 172.21.15.6:0/3140646518 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:09.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.202+0000 7f0eff400640 1 --2- 172.21.15.6:0/3140646518 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0f000cf890 0x7f0f000cd200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:09.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.202+0000 7f0effe00640 1 --2- 172.21.15.6:0/3140646518 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0f0015e1a0 0x7f0f0015fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:09.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.202+0000 7f0efea00640 1 -- 172.21.15.6:0/3140646518 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0f0015e1a0 msgr2=0x7f0f0015fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:09.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.202+0000 7f0efea00640 1 --2- 172.21.15.6:0/3140646518 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0f0015e1a0 0x7f0f0015fab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:09.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.202+0000 7f0efea00640 1 -- 172.21.15.6:0/3140646518 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0f000cf890 msgr2=0x7f0f000cd200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:09.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.202+0000 7f0efea00640 1 --2- 172.21.15.6:0/3140646518 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0f000cf890 0x7f0f000cd200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:09.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.202+0000 7f0efea00640 1 -- 172.21.15.6:0/3140646518 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f000a03a0 con 0x7f0f000d2810 2024-03-04T09:26:09.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.202+0000 7f0effe00640 1 --2- 172.21.15.6:0/3140646518 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0f0015e1a0 0x7f0f0015fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:26:09.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.203+0000 7f0efea00640 1 --2- 172.21.15.6:0/3140646518 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0f000d2810 0x7f0f000a1160 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f0ee800e9b0 tx=0x7f0ee800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:09.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.203+0000 7f0ef7400640 1 -- 172.21.15.6:0/3140646518 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ee8017070 con 0x7f0f000d2810 2024-03-04T09:26:09.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.203+0000 7f0ef7400640 1 -- 172.21.15.6:0/3140646518 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ee800c9a0 con 0x7f0f000d2810 2024-03-04T09:26:09.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.203+0000 7f0f04e00640 1 -- 172.21.15.6:0/3140646518 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f000a0680 con 0x7f0f000d2810 2024-03-04T09:26:09.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.203+0000 7f0ef7400640 1 -- 172.21.15.6:0/3140646518 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ee800cd90 con 0x7f0f000d2810 2024-03-04T09:26:09.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.203+0000 7f0f04e00640 1 -- 172.21.15.6:0/3140646518 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f000cd740 con 0x7f0f000d2810 2024-03-04T09:26:09.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.204+0000 7f0f04e00640 1 -- 172.21.15.6:0/3140646518 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f0004f330 con 0x7f0f000d2810 2024-03-04T09:26:09.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.205+0000 7f0ef7400640 1 -- 172.21.15.6:0/3140646518 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0ee801a070 con 0x7f0f000d2810 2024-03-04T09:26:09.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.206+0000 7f0ef7400640 1 --2- 172.21.15.6:0/3140646518 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f0edc06b4d0 0x7f0edc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:09.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.207+0000 7f0ef7400640 1 -- 172.21.15.6:0/3140646518 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f0ee808a290 con 0x7f0f000d2810 2024-03-04T09:26:09.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.207+0000 7f0eff400640 1 --2- 172.21.15.6:0/3140646518 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f0edc06b4d0 0x7f0edc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:09.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.208+0000 7f0eff400640 1 --2- 172.21.15.6:0/3140646518 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f0edc06b4d0 0x7f0edc06d990 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f0ef0036000 tx=0x7f0ef0030000 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:09.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.211+0000 7f0ef7400640 1 -- 172.21.15.6:0/3140646518 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0ee8059110 con 0x7f0f000d2810 2024-03-04T09:26:09.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.414+0000 7f0f04e00640 1 -- 172.21.15.6:0/3140646518 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f0f000d20b0 con 0x7f0f000d2810 2024-03-04T09:26:09.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.415+0000 7f0ef7400640 1 -- 172.21.15.6:0/3140646518 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7f0ee8058a10 con 0x7f0f000d2810 2024-03-04T09:26:09.412 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-03-04T09:26:09.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.418+0000 7f0f04e00640 1 -- 172.21.15.6:0/3140646518 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f0edc06b4d0 msgr2=0x7f0edc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:09.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.418+0000 7f0f04e00640 1 --2- 172.21.15.6:0/3140646518 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f0edc06b4d0 0x7f0edc06d990 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f0ef0036000 tx=0x7f0ef0030000 comp rx=0 tx=0).stop 2024-03-04T09:26:09.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.418+0000 7f0f04e00640 1 -- 172.21.15.6:0/3140646518 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0f000d2810 msgr2=0x7f0f000a1160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:09.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.418+0000 7f0f04e00640 1 --2- 172.21.15.6:0/3140646518 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0f000d2810 0x7f0f000a1160 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f0ee800e9b0 tx=0x7f0ee800ee70 comp rx=0 tx=0).stop 2024-03-04T09:26:09.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.418+0000 7f0f04e00640 1 -- 172.21.15.6:0/3140646518 shutdown_connections 2024-03-04T09:26:09.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.418+0000 7f0f04e00640 1 --2- 172.21.15.6:0/3140646518 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f0edc06b4d0 0x7f0edc06d990 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:09.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.418+0000 7f0f04e00640 1 --2- 172.21.15.6:0/3140646518 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0f0015e1a0 0x7f0f0015fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:09.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.418+0000 7f0f04e00640 1 --2- 172.21.15.6:0/3140646518 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0f000d2810 0x7f0f000a1160 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:09.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.418+0000 7f0f04e00640 1 --2- 172.21.15.6:0/3140646518 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0f000cf890 0x7f0f000cd200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:09.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.418+0000 7f0f04e00640 1 -- 172.21.15.6:0/3140646518 >> 172.21.15.6:0/3140646518 conn(0x7f0f0008d2a0 msgr2=0x7f0f000b7c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:09.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.418+0000 7f0f04e00640 1 -- 172.21.15.6:0/3140646518 shutdown_connections 2024-03-04T09:26:09.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.418+0000 7f0f04e00640 1 -- 172.21.15.6:0/3140646518 wait complete. 2024-03-04T09:26:09.436 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-04T09:26:09.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.665+0000 7f6bb6c00640 1 Processor -- start 2024-03-04T09:26:09.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.665+0000 7f6bb6c00640 1 -- start start 2024-03-04T09:26:09.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.667+0000 7f6bb6c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f6bb00d07d0 0x7f6bb00b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:09.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.667+0000 7f6bb6c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6bb015e6c0 0x7f6bb00b3c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:09.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.667+0000 7f6bb6c00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bb00cf780 con 0x7f6bb00d4930 2024-03-04T09:26:09.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.667+0000 7f6bb6c00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6bb00b3770 con 0x7f6bb00d07d0 2024-03-04T09:26:09.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.667+0000 7f6bb6c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bb00cfa60 con 0x7f6bb015e6c0 2024-03-04T09:26:09.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.667+0000 7f6bafe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f6bb00d07d0 0x7f6bb00b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:09.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.667+0000 7f6bafe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f6bb00d07d0 0x7f6bb00b4f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:59674/0 (socket says 172.21.15.6:59674) 2024-03-04T09:26:09.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.667+0000 7f6bafe00640 1 -- 172.21.15.6:0/1170134404 learned_addr learned my addr 172.21.15.6:0/1170134404 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:09.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.667+0000 7f6baf400640 1 --1- 172.21.15.6:0/1170134404 >> v1:172.21.15.6:6789/0 conn(0x7f6bb00d4930 0x7f6bb015dfb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:60508/0 (socket says 172.21.15.6:60508) 2024-03-04T09:26:09.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.667+0000 7f6bb4c00640 1 --2- 172.21.15.6:0/1170134404 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6bb015e6c0 0x7f6bb00b3c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:09.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.668+0000 7f6bafe00640 1 -- 172.21.15.6:0/1170134404 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6bb015e6c0 msgr2=0x7f6bb00b3c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:09.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.668+0000 7f6bafe00640 1 --2- 172.21.15.6:0/1170134404 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6bb015e6c0 0x7f6bb00b3c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:09.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.668+0000 7f6bafe00640 1 -- 172.21.15.6:0/1170134404 >> v1:172.21.15.6:6789/0 conn(0x7f6bb00d4930 legacy=0x7f6bb015dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:09.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.668+0000 7f6bafe00640 1 -- 172.21.15.6:0/1170134404 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bb00cfd40 con 0x7f6bb00d07d0 2024-03-04T09:26:09.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.668+0000 7f6bb4c00640 1 --2- 172.21.15.6:0/1170134404 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6bb015e6c0 0x7f6bb00b3c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T09:26:09.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.669+0000 7f6bafe00640 1 --2- 172.21.15.6:0/1170134404 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6bb00d07d0 0x7f6bb00b4f80 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f6ba0009e50 tx=0x7f6ba002bff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=430f63a1c26fbc91 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:09.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.669+0000 7f6baea00640 1 -- 172.21.15.6:0/1170134404 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ba002cb90 con 0x7f6bb00d07d0 2024-03-04T09:26:09.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.669+0000 7f6baea00640 1 -- 172.21.15.6:0/1170134404 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ba002cd30 con 0x7f6bb00d07d0 2024-03-04T09:26:09.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.669+0000 7f6baea00640 1 -- 172.21.15.6:0/1170134404 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ba0034610 con 0x7f6bb00d07d0 2024-03-04T09:26:09.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.670+0000 7f6bb6c00640 1 -- 172.21.15.6:0/1170134404 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6bb00d07d0 msgr2=0x7f6bb00b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:09.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.670+0000 7f6bb6c00640 1 --2- 172.21.15.6:0/1170134404 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6bb00d07d0 0x7f6bb00b4f80 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f6ba0009e50 tx=0x7f6ba002bff0 comp rx=0 tx=0).stop 2024-03-04T09:26:09.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.670+0000 7f6bb6c00640 1 -- 172.21.15.6:0/1170134404 shutdown_connections 2024-03-04T09:26:09.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.670+0000 7f6bb6c00640 1 --2- 172.21.15.6:0/1170134404 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6bb015e6c0 0x7f6bb00b3c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:09.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.670+0000 7f6bb6c00640 1 --2- 172.21.15.6:0/1170134404 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6bb00d07d0 0x7f6bb00b4f80 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:09.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.670+0000 7f6bb6c00640 1 -- 172.21.15.6:0/1170134404 >> 172.21.15.6:0/1170134404 conn(0x7f6bb00d3e10 msgr2=0x7f6bb00d4220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:09.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.670+0000 7f6bb6c00640 1 -- 172.21.15.6:0/1170134404 shutdown_connections 2024-03-04T09:26:09.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.670+0000 7f6bb6c00640 1 -- 172.21.15.6:0/1170134404 wait complete. 2024-03-04T09:26:09.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.672+0000 7f6bb6c00640 1 Processor -- start 2024-03-04T09:26:09.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.672+0000 7f6bb6c00640 1 -- start start 2024-03-04T09:26:09.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.673+0000 7f6bb6c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6bb00d07d0 0x7f6bb015a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:09.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.673+0000 7f6bb6c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6bb00d4930 0x7f6bb00cc920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:09.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.673+0000 7f6bb6c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6bb015e6c0 0x7f6bb00c9ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:09.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.673+0000 7f6bb6c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bb00d0d30 con 0x7f6bb00d4930 2024-03-04T09:26:09.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.673+0000 7f6bb6c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bb00d0bb0 con 0x7f6bb00d07d0 2024-03-04T09:26:09.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.673+0000 7f6bb6c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bb00d0eb0 con 0x7f6bb015e6c0 2024-03-04T09:26:09.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.673+0000 7f6baf400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6bb00d4930 0x7f6bb00cc920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:09.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.673+0000 7f6baf400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6bb00d4930 0x7f6bb00cc920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:59682/0 (socket says 172.21.15.6:59682) 2024-03-04T09:26:09.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.673+0000 7f6baf400640 1 -- 172.21.15.6:0/3926687977 learned_addr learned my addr 172.21.15.6:0/3926687977 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:09.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.674+0000 7f6bafe00640 1 --2- 172.21.15.6:0/3926687977 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6bb00d07d0 0x7f6bb015a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:09.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.674+0000 7f6bb4c00640 1 --2- 172.21.15.6:0/3926687977 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6bb015e6c0 0x7f6bb00c9ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:09.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.674+0000 7f6baf400640 1 -- 172.21.15.6:0/3926687977 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6bb015e6c0 msgr2=0x7f6bb00c9ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:09.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.674+0000 7f6baf400640 1 --2- 172.21.15.6:0/3926687977 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6bb015e6c0 0x7f6bb00c9ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:09.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.674+0000 7f6baf400640 1 -- 172.21.15.6:0/3926687977 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6bb00d07d0 msgr2=0x7f6bb015a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:09.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.674+0000 7f6baf400640 1 --2- 172.21.15.6:0/3926687977 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6bb00d07d0 0x7f6bb015a6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:09.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.674+0000 7f6baf400640 1 -- 172.21.15.6:0/3926687977 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bb0160dc0 con 0x7f6bb00d4930 2024-03-04T09:26:09.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.674+0000 7f6baf400640 1 --2- 172.21.15.6:0/3926687977 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6bb00d4930 0x7f6bb00cc920 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f6b9800c400 tx=0x7f6b9800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:09.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.674+0000 7f6ba7e00640 1 -- 172.21.15.6:0/3926687977 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b980074e0 con 0x7f6bb00d4930 2024-03-04T09:26:09.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.674+0000 7f6ba7e00640 1 -- 172.21.15.6:0/3926687977 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b98007680 con 0x7f6bb00d4930 2024-03-04T09:26:09.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.675+0000 7f6bb6c00640 1 -- 172.21.15.6:0/3926687977 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bb01610a0 con 0x7f6bb00d4930 2024-03-04T09:26:09.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.675+0000 7f6ba7e00640 1 -- 172.21.15.6:0/3926687977 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b98004a00 con 0x7f6bb00d4930 2024-03-04T09:26:09.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.675+0000 7f6bb6c00640 1 -- 172.21.15.6:0/3926687977 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bb015ac00 con 0x7f6bb00d4930 2024-03-04T09:26:09.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.676+0000 7f6bb6c00640 1 -- 172.21.15.6:0/3926687977 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bb004f330 con 0x7f6bb00d4930 2024-03-04T09:26:09.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.677+0000 7f6ba7e00640 1 -- 172.21.15.6:0/3926687977 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6b9801a020 con 0x7f6bb00d4930 2024-03-04T09:26:09.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.678+0000 7f6ba7e00640 1 --2- 172.21.15.6:0/3926687977 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f6b8c06b4d0 0x7f6b8c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:09.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.678+0000 7f6ba7e00640 1 -- 172.21.15.6:0/3926687977 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f6b980899f0 con 0x7f6bb00d4930 2024-03-04T09:26:09.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.678+0000 7f6bafe00640 1 --2- 172.21.15.6:0/3926687977 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f6b8c06b4d0 0x7f6b8c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:09.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.679+0000 7f6bafe00640 1 --2- 172.21.15.6:0/3926687977 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f6b8c06b4d0 0x7f6b8c06d990 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f6ba0035000 tx=0x7f6ba0030000 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:09.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.683+0000 7f6ba7e00640 1 -- 172.21.15.6:0/3926687977 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6b980587e0 con 0x7f6bb00d4930 2024-03-04T09:26:09.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.886+0000 7f6bb6c00640 1 -- 172.21.15.6:0/3926687977 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f6bb00d4290 con 0x7f6bb00d4930 2024-03-04T09:26:09.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.887+0000 7f6ba7e00640 1 -- 172.21.15.6:0/3926687977 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7f6b980580e0 con 0x7f6bb00d4930 2024-03-04T09:26:09.883 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 64 2024-03-04T09:26:09.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.889+0000 7f6bb6c00640 1 -- 172.21.15.6:0/3926687977 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f6b8c06b4d0 msgr2=0x7f6b8c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:09.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.889+0000 7f6bb6c00640 1 --2- 172.21.15.6:0/3926687977 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f6b8c06b4d0 0x7f6b8c06d990 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f6ba0035000 tx=0x7f6ba0030000 comp rx=0 tx=0).stop 2024-03-04T09:26:09.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.889+0000 7f6bb6c00640 1 -- 172.21.15.6:0/3926687977 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6bb00d4930 msgr2=0x7f6bb00cc920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:09.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.889+0000 7f6bb6c00640 1 --2- 172.21.15.6:0/3926687977 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6bb00d4930 0x7f6bb00cc920 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f6b9800c400 tx=0x7f6b9800c8c0 comp rx=0 tx=0).stop 2024-03-04T09:26:09.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.889+0000 7f6bb6c00640 1 -- 172.21.15.6:0/3926687977 shutdown_connections 2024-03-04T09:26:09.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.889+0000 7f6bb6c00640 1 --2- 172.21.15.6:0/3926687977 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f6b8c06b4d0 0x7f6b8c06d990 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:09.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.889+0000 7f6bb6c00640 1 --2- 172.21.15.6:0/3926687977 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6bb015e6c0 0x7f6bb00c9ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:09.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.889+0000 7f6bb6c00640 1 --2- 172.21.15.6:0/3926687977 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6bb00d4930 0x7f6bb00cc920 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:09.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.889+0000 7f6bb6c00640 1 --2- 172.21.15.6:0/3926687977 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6bb00d07d0 0x7f6bb015a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:09.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.889+0000 7f6bb6c00640 1 -- 172.21.15.6:0/3926687977 >> 172.21.15.6:0/3926687977 conn(0x7f6bb00d3e10 msgr2=0x7f6bb006c740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:09.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.890+0000 7f6bb6c00640 1 -- 172.21.15.6:0/3926687977 shutdown_connections 2024-03-04T09:26:09.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:09.890+0000 7f6bb6c00640 1 -- 172.21.15.6:0/3926687977 wait complete. 2024-03-04T09:26:09.905 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-04T09:26:10.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.113+0000 7f68bf600640 1 Processor -- start 2024-03-04T09:26:10.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.114+0000 7f68bf600640 1 -- start start 2024-03-04T09:26:10.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.115+0000 7f68bf600640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f68b80af1a0 0x7f68b815c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:10.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.115+0000 7f68bf600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f68b80af580 0x7f68b80b4c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:10.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.115+0000 7f68bf600640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68b815cc40 con 0x7f68b80cdc20 2024-03-04T09:26:10.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.115+0000 7f68bf600640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68b815cf20 con 0x7f68b80af1a0 2024-03-04T09:26:10.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.115+0000 7f68bf600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68b815d200 con 0x7f68b80af580 2024-03-04T09:26:10.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.116+0000 7f68bd600640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f68b80cdc20 0x7f68b80b2dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.6:38266/0 (socket says 172.21.15.6:38266) 2024-03-04T09:26:10.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.116+0000 7f68b7e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f68b80af580 0x7f68b80b4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:10.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.116+0000 7f68bd600640 1 -- 172.21.15.6:0/2699846740 learned_addr learned my addr 172.21.15.6:0/2699846740 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:26:10.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.116+0000 7f68bcc00640 1 --2- 172.21.15.6:0/2699846740 >> v2:172.21.15.146:3300/0 conn(0x7f68b80af1a0 0x7f68b815c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:10.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.117+0000 7f68b7400640 1 -- 172.21.15.6:0/2699846740 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1864802858 0 0) 0x7f68b815cc40 con 0x7f68b80cdc20 2024-03-04T09:26:10.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.117+0000 7f68b7400640 1 -- 172.21.15.6:0/2699846740 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68a0003520 con 0x7f68b80cdc20 2024-03-04T09:26:10.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.117+0000 7f68bcc00640 1 -- 172.21.15.6:0/2699846740 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f68b80af580 msgr2=0x7f68b80b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:10.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.117+0000 7f68bcc00640 1 --2- 172.21.15.6:0/2699846740 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f68b80af580 0x7f68b80b4c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:10.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.117+0000 7f68bcc00640 1 -- 172.21.15.6:0/2699846740 >> v1:172.21.15.146:6789/0 conn(0x7f68b80cdc20 legacy=0x7f68b80b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:10.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.117+0000 7f68bcc00640 1 -- 172.21.15.6:0/2699846740 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68b815d4e0 con 0x7f68b80af1a0 2024-03-04T09:26:10.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.119+0000 7f68bcc00640 1 --2- 172.21.15.6:0/2699846740 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f68b80af1a0 0x7f68b815c600 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f68ac0097a0 tx=0x7f68ac02b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1eddfc708458eb55 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:10.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.119+0000 7f68b7400640 1 -- 172.21.15.6:0/2699846740 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68ac0315a0 con 0x7f68b80af1a0 2024-03-04T09:26:10.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.119+0000 7f68b7400640 1 -- 172.21.15.6:0/2699846740 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68ac037070 con 0x7f68b80af1a0 2024-03-04T09:26:10.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.119+0000 7f68b7400640 1 -- 172.21.15.6:0/2699846740 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68ac031960 con 0x7f68b80af1a0 2024-03-04T09:26:10.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.120+0000 7f68bf600640 1 -- 172.21.15.6:0/2699846740 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f68b80af1a0 msgr2=0x7f68b815c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:10.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.120+0000 7f68bf600640 1 --2- 172.21.15.6:0/2699846740 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f68b80af1a0 0x7f68b815c600 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f68ac0097a0 tx=0x7f68ac02b840 comp rx=0 tx=0).stop 2024-03-04T09:26:10.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.120+0000 7f68bf600640 1 -- 172.21.15.6:0/2699846740 shutdown_connections 2024-03-04T09:26:10.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.120+0000 7f68bf600640 1 --2- 172.21.15.6:0/2699846740 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f68b80af580 0x7f68b80b4c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:10.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.120+0000 7f68bf600640 1 --2- 172.21.15.6:0/2699846740 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f68b80af1a0 0x7f68b815c600 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:10.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.120+0000 7f68bf600640 1 -- 172.21.15.6:0/2699846740 >> 172.21.15.6:0/2699846740 conn(0x7f68b806afb0 msgr2=0x7f68b806b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:10.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.120+0000 7f68bf600640 1 -- 172.21.15.6:0/2699846740 shutdown_connections 2024-03-04T09:26:10.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.120+0000 7f68bf600640 1 -- 172.21.15.6:0/2699846740 wait complete. 2024-03-04T09:26:10.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.122+0000 7f68bf600640 1 Processor -- start 2024-03-04T09:26:10.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.122+0000 7f68bf600640 1 -- start start 2024-03-04T09:26:10.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.123+0000 7f68bf600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f68b80af1a0 0x7f68b814f460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:10.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.123+0000 7f68bf600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f68b80af580 0x7f68b809b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:10.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.123+0000 7f68bf600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f68b80cdc20 0x7f68b8071bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:10.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.123+0000 7f68bf600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68b80b3660 con 0x7f68b80cdc20 2024-03-04T09:26:10.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.123+0000 7f68bf600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68b80b34e0 con 0x7f68b80af1a0 2024-03-04T09:26:10.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.123+0000 7f68bf600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68b80b37e0 con 0x7f68b80af580 2024-03-04T09:26:10.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.123+0000 7f68bd600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f68b80cdc20 0x7f68b8071bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:10.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.123+0000 7f68bd600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f68b80cdc20 0x7f68b8071bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:59684/0 (socket says 172.21.15.6:59684) 2024-03-04T09:26:10.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.124+0000 7f68bd600640 1 -- 172.21.15.6:0/1262922550 learned_addr learned my addr 172.21.15.6:0/1262922550 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:10.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.124+0000 7f68b7e00640 1 --2- 172.21.15.6:0/1262922550 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f68b80af580 0x7f68b809b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:10.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.124+0000 7f68bcc00640 1 --2- 172.21.15.6:0/1262922550 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f68b80af1a0 0x7f68b814f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:10.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.124+0000 7f68bd600640 1 -- 172.21.15.6:0/1262922550 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f68b80af580 msgr2=0x7f68b809b3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:10.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.124+0000 7f68bd600640 1 --2- 172.21.15.6:0/1262922550 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f68b80af580 0x7f68b809b3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:10.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.124+0000 7f68bd600640 1 -- 172.21.15.6:0/1262922550 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f68b80af1a0 msgr2=0x7f68b814f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:10.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.124+0000 7f68bd600640 1 --2- 172.21.15.6:0/1262922550 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f68b80af1a0 0x7f68b814f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:10.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.124+0000 7f68bd600640 1 -- 172.21.15.6:0/1262922550 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68b809d150 con 0x7f68b80cdc20 2024-03-04T09:26:10.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.124+0000 7f68bd600640 1 --2- 172.21.15.6:0/1262922550 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f68b80cdc20 0x7f68b8071bc0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f68b000cad0 tx=0x7f68b000cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:10.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.125+0000 7f68b4c00640 1 -- 172.21.15.6:0/1262922550 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68b0017070 con 0x7f68b80cdc20 2024-03-04T09:26:10.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.125+0000 7f68b4c00640 1 -- 172.21.15.6:0/1262922550 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68b000ac30 con 0x7f68b80cdc20 2024-03-04T09:26:10.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.125+0000 7f68b4c00640 1 -- 172.21.15.6:0/1262922550 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68b0012400 con 0x7f68b80cdc20 2024-03-04T09:26:10.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.125+0000 7f68bf600640 1 -- 172.21.15.6:0/1262922550 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68b809d3c0 con 0x7f68b80cdc20 2024-03-04T09:26:10.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.125+0000 7f68bf600640 1 -- 172.21.15.6:0/1262922550 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68b806cd60 con 0x7f68b80cdc20 2024-03-04T09:26:10.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.126+0000 7f68bf600640 1 -- 172.21.15.6:0/1262922550 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68b8069510 con 0x7f68b80cdc20 2024-03-04T09:26:10.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.127+0000 7f68b4c00640 1 -- 172.21.15.6:0/1262922550 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f68b001a070 con 0x7f68b80cdc20 2024-03-04T09:26:10.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.128+0000 7f68b4c00640 1 --2- 172.21.15.6:0/1262922550 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f689806b4d0 0x7f689806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:10.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.128+0000 7f68b4c00640 1 -- 172.21.15.6:0/1262922550 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f68b0089ff0 con 0x7f68b80cdc20 2024-03-04T09:26:10.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.128+0000 7f68bcc00640 1 --2- 172.21.15.6:0/1262922550 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f689806b4d0 0x7f689806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:10.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.130+0000 7f68bcc00640 1 --2- 172.21.15.6:0/1262922550 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f689806b4d0 0x7f689806d990 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f68ac036000 tx=0x7f68ac03d040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:10.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.132+0000 7f68b4c00640 1 -- 172.21.15.6:0/1262922550 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f68b0058e10 con 0x7f68b80cdc20 2024-03-04T09:26:10.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.344+0000 7f68bf600640 1 -- 172.21.15.6:0/1262922550 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f68b809dee0 con 0x7f68b80cdc20 2024-03-04T09:26:10.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.345+0000 7f68b4c00640 1 -- 172.21.15.6:0/1262922550 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v22) v1 ==== 80+0+0 (secure 0 0 0) 0x7f68b0058710 con 0x7f68b80cdc20 2024-03-04T09:26:10.342 INFO:teuthology.orchestra.run.smithi006.stderr:listed 0 entries 2024-03-04T09:26:10.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.348+0000 7f68bf600640 1 -- 172.21.15.6:0/1262922550 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f689806b4d0 msgr2=0x7f689806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:10.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.348+0000 7f68bf600640 1 --2- 172.21.15.6:0/1262922550 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f689806b4d0 0x7f689806d990 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f68ac036000 tx=0x7f68ac03d040 comp rx=0 tx=0).stop 2024-03-04T09:26:10.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.348+0000 7f68bf600640 1 -- 172.21.15.6:0/1262922550 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f68b80cdc20 msgr2=0x7f68b8071bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:10.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.348+0000 7f68bf600640 1 --2- 172.21.15.6:0/1262922550 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f68b80cdc20 0x7f68b8071bc0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f68b000cad0 tx=0x7f68b000cf90 comp rx=0 tx=0).stop 2024-03-04T09:26:10.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.348+0000 7f68bf600640 1 -- 172.21.15.6:0/1262922550 shutdown_connections 2024-03-04T09:26:10.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.348+0000 7f68bf600640 1 --2- 172.21.15.6:0/1262922550 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f68b80cdc20 0x7f68b8071bc0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:10.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.348+0000 7f68bf600640 1 --2- 172.21.15.6:0/1262922550 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f689806b4d0 0x7f689806d990 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:10.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.348+0000 7f68bf600640 1 --2- 172.21.15.6:0/1262922550 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f68b80af580 0x7f68b809b3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:10.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.348+0000 7f68bf600640 1 --2- 172.21.15.6:0/1262922550 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f68b80af1a0 0x7f68b814f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:10.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.348+0000 7f68bf600640 1 -- 172.21.15.6:0/1262922550 >> 172.21.15.6:0/1262922550 conn(0x7f68b806afb0 msgr2=0x7f68b80cb5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:10.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.348+0000 7f68bf600640 1 -- 172.21.15.6:0/1262922550 shutdown_connections 2024-03-04T09:26:10.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.348+0000 7f68bf600640 1 -- 172.21.15.6:0/1262922550 wait complete. 2024-03-04T09:26:10.363 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.2... 2024-03-04T09:26:10.363 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:26:10.363 DEBUG:teuthology.orchestra.run.smithi006:> sudo umount /home/ubuntu/cephtest/mnt.2 -f 2024-03-04T09:26:10.462 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi006.front.sepia.ceph.com 2024-03-04T09:26:10.462 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:26:10.462 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-04T09:26:10.513 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-04T09:26:10.513 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T09:26:10.566 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-04T09:26:10.566 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T09:26:10.610 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-04T09:26:10.611 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T09:26:10.701 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T09:26:10.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.936+0000 7f4942e00640 1 Processor -- start 2024-03-04T09:26:10.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.937+0000 7f4942e00640 1 -- start start 2024-03-04T09:26:10.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.938+0000 7f4942e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f493c0cede0 0x7f493c0b3f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:10.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.938+0000 7f4942e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f493c071b60 0x7f493c15b320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:10.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.939+0000 7f4942e00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f493c0b31d0 con 0x7f493c0b4a70 2024-03-04T09:26:10.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.939+0000 7f4942e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f493c0b4ed0 con 0x7f493c071b60 2024-03-04T09:26:10.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.939+0000 7f4942e00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f493c0b3b00 con 0x7f493c0cede0 2024-03-04T09:26:10.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.939+0000 7f4942400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f493c071b60 0x7f493c15b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:10.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.939+0000 7f4941a00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f493c0cede0 0x7f493c0b3f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:10.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.939+0000 7f4941000640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f493c0b4a70 0x7f493c071450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.6:38268/0 (socket says 172.21.15.6:38268) 2024-03-04T09:26:10.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.939+0000 7f4941000640 1 -- 172.21.15.6:0/4231567231 learned_addr learned my addr 172.21.15.6:0/4231567231 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:26:10.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.940+0000 7f4942400640 1 -- 172.21.15.6:0/4231567231 >> v2:172.21.15.146:3300/0 conn(0x7f493c0cede0 msgr2=0x7f493c0b3f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:10.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.940+0000 7f4942400640 1 --2- 172.21.15.6:0/4231567231 >> v2:172.21.15.146:3300/0 conn(0x7f493c0cede0 0x7f493c0b3f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:10.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.940+0000 7f4942400640 1 -- 172.21.15.6:0/4231567231 >> v1:172.21.15.146:6789/0 conn(0x7f493c0b4a70 legacy=0x7f493c071450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:10.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.940+0000 7f4942400640 1 -- 172.21.15.6:0/4231567231 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f493c0b34b0 con 0x7f493c071b60 2024-03-04T09:26:10.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.940+0000 7f4941a00640 1 --2- 172.21.15.6:0/4231567231 >> v2:172.21.15.146:3300/0 conn(0x7f493c0cede0 0x7f493c0b3f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T09:26:10.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.940+0000 7f493be00640 1 -- 172.21.15.6:0/4231567231 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2931680405 0 0) 0x7f493c0b31d0 con 0x7f493c0b4a70 2024-03-04T09:26:10.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.941+0000 7f4942400640 1 --2- 172.21.15.6:0/4231567231 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f493c071b60 0x7f493c15b320 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f493c069240 tx=0x7f493402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fbc9785ce0e1300 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:10.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.942+0000 7f493be00640 1 -- 172.21.15.6:0/4231567231 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4934039070 con 0x7f493c071b60 2024-03-04T09:26:10.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.942+0000 7f493be00640 1 -- 172.21.15.6:0/4231567231 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f493402cbf0 con 0x7f493c071b60 2024-03-04T09:26:10.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.942+0000 7f493be00640 1 -- 172.21.15.6:0/4231567231 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4934034440 con 0x7f493c071b60 2024-03-04T09:26:10.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.942+0000 7f4942e00640 1 -- 172.21.15.6:0/4231567231 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f493c071b60 msgr2=0x7f493c15b320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:10.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.942+0000 7f4942e00640 1 --2- 172.21.15.6:0/4231567231 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f493c071b60 0x7f493c15b320 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f493c069240 tx=0x7f493402be40 comp rx=0 tx=0).stop 2024-03-04T09:26:10.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.942+0000 7f4942e00640 1 -- 172.21.15.6:0/4231567231 shutdown_connections 2024-03-04T09:26:10.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.942+0000 7f4942e00640 1 --2- 172.21.15.6:0/4231567231 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f493c071b60 0x7f493c15b320 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:10.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.942+0000 7f4942e00640 1 --2- 172.21.15.6:0/4231567231 >> v2:172.21.15.146:3300/0 conn(0x7f493c0cede0 0x7f493c0b3f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:10.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.942+0000 7f4942e00640 1 -- 172.21.15.6:0/4231567231 >> 172.21.15.6:0/4231567231 conn(0x7f493c06b3d0 msgr2=0x7f493c06b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:10.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.943+0000 7f4942e00640 1 -- 172.21.15.6:0/4231567231 shutdown_connections 2024-03-04T09:26:10.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.943+0000 7f4942e00640 1 -- 172.21.15.6:0/4231567231 wait complete. 2024-03-04T09:26:10.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.945+0000 7f4942e00640 1 Processor -- start 2024-03-04T09:26:10.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.945+0000 7f4942e00640 1 -- start start 2024-03-04T09:26:10.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.946+0000 7f4942e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f493c071b60 0x7f493c159290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:10.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.946+0000 7f4942e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f493c0b4a70 0x7f493c157470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:10.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.946+0000 7f4942e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f493c0cede0 0x7f493c1f4150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:10.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.946+0000 7f4942e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f493c0cf1c0 con 0x7f493c0cede0 2024-03-04T09:26:10.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.946+0000 7f4942e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f493c0b2f70 con 0x7f493c0b4a70 2024-03-04T09:26:10.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.946+0000 7f4942e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f493c0cc950 con 0x7f493c071b60 2024-03-04T09:26:10.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.946+0000 7f4942400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f493c0cede0 0x7f493c1f4150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:10.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.946+0000 7f4942400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f493c0cede0 0x7f493c1f4150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:59702/0 (socket says 172.21.15.6:59702) 2024-03-04T09:26:10.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.946+0000 7f4942400640 1 -- 172.21.15.6:0/1461281757 learned_addr learned my addr 172.21.15.6:0/1461281757 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:10.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.947+0000 7f4941000640 1 --2- 172.21.15.6:0/1461281757 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f493c0b4a70 0x7f493c157470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:10.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.947+0000 7f4941a00640 1 --2- 172.21.15.6:0/1461281757 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f493c071b60 0x7f493c159290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:10.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.947+0000 7f4942400640 1 -- 172.21.15.6:0/1461281757 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f493c071b60 msgr2=0x7f493c159290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:10.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.947+0000 7f4942400640 1 --2- 172.21.15.6:0/1461281757 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f493c071b60 0x7f493c159290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:10.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.947+0000 7f4942400640 1 -- 172.21.15.6:0/1461281757 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f493c0b4a70 msgr2=0x7f493c157470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:10.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.947+0000 7f4942400640 1 --2- 172.21.15.6:0/1461281757 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f493c0b4a70 0x7f493c157470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:10.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.947+0000 7f4942400640 1 -- 172.21.15.6:0/1461281757 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f493c154ed0 con 0x7f493c0cede0 2024-03-04T09:26:10.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.947+0000 7f4942400640 1 --2- 172.21.15.6:0/1461281757 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f493c0cede0 0x7f493c1f4150 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f4934037040 tx=0x7f4934004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:10.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.948+0000 7f4939600640 1 -- 172.21.15.6:0/1461281757 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4934044020 con 0x7f493c0cede0 2024-03-04T09:26:10.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.948+0000 7f4939600640 1 -- 172.21.15.6:0/1461281757 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f493403a040 con 0x7f493c0cede0 2024-03-04T09:26:10.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.948+0000 7f4942e00640 1 -- 172.21.15.6:0/1461281757 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f493c159bb0 con 0x7f493c0cede0 2024-03-04T09:26:10.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.948+0000 7f4939600640 1 -- 172.21.15.6:0/1461281757 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4934039050 con 0x7f493c0cede0 2024-03-04T09:26:10.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.948+0000 7f4942e00640 1 -- 172.21.15.6:0/1461281757 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f493c15a3f0 con 0x7f493c0cede0 2024-03-04T09:26:10.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.949+0000 7f4942e00640 1 -- 172.21.15.6:0/1461281757 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f493c04f330 con 0x7f493c0cede0 2024-03-04T09:26:10.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.950+0000 7f4939600640 1 -- 172.21.15.6:0/1461281757 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4934003b90 con 0x7f493c0cede0 2024-03-04T09:26:10.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.951+0000 7f4939600640 1 --2- 172.21.15.6:0/1461281757 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f491c06b500 0x7f491c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:10.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.951+0000 7f4939600640 1 -- 172.21.15.6:0/1461281757 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f49340ac9e0 con 0x7f493c0cede0 2024-03-04T09:26:10.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.951+0000 7f4941a00640 1 --2- 172.21.15.6:0/1461281757 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f491c06b500 0x7f491c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:10.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.953+0000 7f4941a00640 1 --2- 172.21.15.6:0/1461281757 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f491c06b500 0x7f491c06d9c0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f493000ab30 tx=0x7f4930008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:10.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:10.955+0000 7f4939600640 1 -- 172.21.15.6:0/1461281757 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f493404e020 con 0x7f493c0cede0 2024-03-04T09:26:11.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.156+0000 7f4942e00640 1 -- 172.21.15.6:0/1461281757 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f493c06cad0 con 0x7f493c0cede0 2024-03-04T09:26:11.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.158+0000 7f4939600640 1 -- 172.21.15.6:0/1461281757 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16852 (secure 0 0 0) 0x7f493c159bb0 con 0x7f493c0cede0 2024-03-04T09:26:11.156 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:26:11.156 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":22,"fsid":"edce0578-5ae4-4402-b45a-552bdef84d7c","created":"2024-03-04T09:24:54.523780+0000","modified":"2024-03-04T09:26:01.727682+0000","last_up_change":"2024-03-04T09:25:07.089049+0000","last_in_change":"2024-03-04T09:24:57.982077+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T09:25:07.935198+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-04T09:25:17.118573+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-04T09:25:17.930536+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a96574f-81f1-4bc6-8463-757520b84a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6825","nonce":1364449401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6827","nonce":1364449401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6831","nonce":1364449401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6829","nonce":1364449401}]},"public_addr":"172.21.15.6:6825/1364449401","cluster_addr":"172.21.15.6:6827/1364449401","heartbeat_back_addr":"172.21.15.6:6831/1364449401","heartbeat_front_addr":"172.21.15.6:6829/1364449401","state":["exists","up"]},{"osd":1,"uuid":"7d79b9c1-6b87-476a-bc36-7db5ee64cd90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6809","nonce":2494103078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6811","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6813","nonce":2494103078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6821","nonce":2494103078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6817","nonce":2494103078}]},"public_addr":"172.21.15.6:6809/2494103078","cluster_addr":"172.21.15.6:6813/2494103078","heartbeat_back_addr":"172.21.15.6:6821/2494103078","heartbeat_front_addr":"172.21.15.6:6817/2494103078","state":["exists","up"]},{"osd":2,"uuid":"492331b9-4ee9-40b6-bacd-85d44cd6c45a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6812","nonce":1500066681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6816","nonce":1500066681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6824","nonce":1500066681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6820","nonce":1500066681}]},"public_addr":"172.21.15.6:6812/1500066681","cluster_addr":"172.21.15.6:6816/1500066681","heartbeat_back_addr":"172.21.15.6:6824/1500066681","heartbeat_front_addr":"172.21.15.6:6820/1500066681","state":["exists","up"]},{"osd":3,"uuid":"21bb2efe-9ba7-4bdc-bc31-180cd1102e21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6801","nonce":1558806123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6803","nonce":1558806123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6807","nonce":1558806123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6805","nonce":1558806123}]},"public_addr":"172.21.15.6:6801/1558806123","cluster_addr":"172.21.15.6:6803/1558806123","heartbeat_back_addr":"172.21.15.6:6807/1558806123","heartbeat_front_addr":"172.21.15.6:6805/1558806123","state":["exists","up"]},{"osd":4,"uuid":"6e2c4b71-19d9-4737-83d9-558705d8644a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6801","nonce":1526135780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6803","nonce":1526135780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6807","nonce":1526135780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6805","nonce":1526135780}]},"public_addr":"172.21.15.146:6801/1526135780","cluster_addr":"172.21.15.146:6803/1526135780","heartbeat_back_addr":"172.21.15.146:6807/1526135780","heartbeat_front_addr":"172.21.15.146:6805/1526135780","state":["exists","up"]},{"osd":5,"uuid":"fb69c525-f1be-4b7d-ab32-6c29b0d6fddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6809","nonce":460262507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6811","nonce":460262507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6815","nonce":460262507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6813","nonce":460262507}]},"public_addr":"172.21.15.146:6809/460262507","cluster_addr":"172.21.15.146:6811/460262507","heartbeat_back_addr":"172.21.15.146:6815/460262507","heartbeat_front_addr":"172.21.15.146:6813/460262507","state":["exists","up"]},{"osd":6,"uuid":"c32e7fed-370d-4401-8ce0-6a4335b0d168","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6823","nonce":1392897493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6827","nonce":1392897493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6831","nonce":1392897493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6829","nonce":1392897493}]},"public_addr":"172.21.15.146:6823/1392897493","cluster_addr":"172.21.15.146:6827/1392897493","heartbeat_back_addr":"172.21.15.146:6831/1392897493","heartbeat_front_addr":"172.21.15.146:6829/1392897493","state":["exists","up"]},{"osd":7,"uuid":"75a79b44-fa5f-426d-a7d3-71f21f78835c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6817","nonce":3268323156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6819","nonce":3268323156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6825","nonce":3268323156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6822","nonce":3268323156}]},"public_addr":"172.21.15.146:6817/3268323156","cluster_addr":"172.21.15.146:6819/3268323156","heartbeat_back_addr":"172.21.15.146:6825/3268323156","heartbeat_front_addr":"172.21.15.146:6822/3268323156","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.310437+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.295850+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.294674+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.526855+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.300023+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.595160+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.346046+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.558741+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.7","mappings":[{"from":3,"to":2}]},{"pgid":"3.f","mappings":[{"from":7,"to":2}]},{"pgid":"3.21","mappings":[{"from":5,"to":1}]},{"pgid":"3.22","mappings":[{"from":5,"to":2}]},{"pgid":"3.28","mappings":[{"from":3,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T09:26:11.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.161+0000 7f4942e00640 1 -- 172.21.15.6:0/1461281757 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f491c06b500 msgr2=0x7f491c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:11.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.161+0000 7f4942e00640 1 --2- 172.21.15.6:0/1461281757 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f491c06b500 0x7f491c06d9c0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f493000ab30 tx=0x7f4930008040 comp rx=0 tx=0).stop 2024-03-04T09:26:11.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.161+0000 7f4942e00640 1 -- 172.21.15.6:0/1461281757 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f493c0cede0 msgr2=0x7f493c1f4150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:11.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.161+0000 7f4942e00640 1 --2- 172.21.15.6:0/1461281757 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f493c0cede0 0x7f493c1f4150 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f4934037040 tx=0x7f4934004a40 comp rx=0 tx=0).stop 2024-03-04T09:26:11.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.161+0000 7f4942e00640 1 -- 172.21.15.6:0/1461281757 shutdown_connections 2024-03-04T09:26:11.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.161+0000 7f4942e00640 1 --2- 172.21.15.6:0/1461281757 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f493c0cede0 0x7f493c1f4150 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:11.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.162+0000 7f4942e00640 1 --2- 172.21.15.6:0/1461281757 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f493c0b4a70 0x7f493c157470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:11.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.162+0000 7f4942e00640 1 --2- 172.21.15.6:0/1461281757 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f491c06b500 0x7f491c06d9c0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:11.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.162+0000 7f4942e00640 1 --2- 172.21.15.6:0/1461281757 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f493c071b60 0x7f493c159290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:11.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.162+0000 7f4942e00640 1 -- 172.21.15.6:0/1461281757 >> 172.21.15.6:0/1461281757 conn(0x7f493c06b3d0 msgr2=0x7f493c14c820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:11.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.162+0000 7f4942e00640 1 -- 172.21.15.6:0/1461281757 shutdown_connections 2024-03-04T09:26:11.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.162+0000 7f4942e00640 1 -- 172.21.15.6:0/1461281757 wait complete. 2024-03-04T09:26:11.177 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T09:25:07.935198+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-04T09:25:17.118573+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-04T09:25:17.930536+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-04T09:26:11.177 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-04T09:26:11.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.404+0000 7f4ef6400640 1 Processor -- start 2024-03-04T09:26:11.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.404+0000 7f4ef6400640 1 -- start start 2024-03-04T09:26:11.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.405+0000 7f4ef6400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f4ef015bf20 0x7f4ef00acd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:11.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.405+0000 7f4ef6400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f4ef0072520 0x7f4ef00cdfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:11.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.405+0000 7f4ef6400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4ef0071e20 0x7f4ef00ce4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:11.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.405+0000 7f4ef6400640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ef0072df0 con 0x7f4ef0072520 2024-03-04T09:26:11.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.405+0000 7f4ef6400640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ef00cdab0 con 0x7f4ef015bf20 2024-03-04T09:26:11.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.405+0000 7f4ef6400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ef00b4d70 con 0x7f4ef0071e20 2024-03-04T09:26:11.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.406+0000 7f4eeea00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f4ef0072520 0x7f4ef00cdfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:11.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.406+0000 7f4eeea00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f4ef0072520 0x7f4ef00cdfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:59710/0 (socket says 172.21.15.6:59710) 2024-03-04T09:26:11.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.406+0000 7f4eeea00640 1 -- 172.21.15.6:0/1841383662 learned_addr learned my addr 172.21.15.6:0/1841383662 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:11.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.406+0000 7f4eefe00640 1 --2- 172.21.15.6:0/1841383662 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4ef0071e20 0x7f4ef00ce4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:11.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.406+0000 7f4eef400640 1 --2- 172.21.15.6:0/1841383662 >> v2:172.21.15.146:3300/0 conn(0x7f4ef015bf20 0x7f4ef00acd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:11.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.406+0000 7f4eeea00640 1 -- 172.21.15.6:0/1841383662 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4ef0071e20 msgr2=0x7f4ef00ce4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:11.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.406+0000 7f4eeea00640 1 --2- 172.21.15.6:0/1841383662 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4ef0071e20 0x7f4ef00ce4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:11.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.406+0000 7f4eeea00640 1 -- 172.21.15.6:0/1841383662 >> v2:172.21.15.146:3300/0 conn(0x7f4ef015bf20 msgr2=0x7f4ef00acd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:11.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.406+0000 7f4eeea00640 1 --2- 172.21.15.6:0/1841383662 >> v2:172.21.15.146:3300/0 conn(0x7f4ef015bf20 0x7f4ef00acd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:11.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.406+0000 7f4eeea00640 1 -- 172.21.15.6:0/1841383662 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ef00b5050 con 0x7f4ef0072520 2024-03-04T09:26:11.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.407+0000 7f4eeea00640 1 --2- 172.21.15.6:0/1841383662 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4ef0072520 0x7f4ef00cdfb0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f4ef006ec40 tx=0x7f4edc02b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6c0365a8f62bde5 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:11.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.408+0000 7f4eee000640 1 -- 172.21.15.6:0/1841383662 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4edc035020 con 0x7f4ef0072520 2024-03-04T09:26:11.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.408+0000 7f4eee000640 1 -- 172.21.15.6:0/1841383662 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4edc02d040 con 0x7f4ef0072520 2024-03-04T09:26:11.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.408+0000 7f4eee000640 1 -- 172.21.15.6:0/1841383662 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4edc031a80 con 0x7f4ef0072520 2024-03-04T09:26:11.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.408+0000 7f4ef6400640 1 -- 172.21.15.6:0/1841383662 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4ef0072520 msgr2=0x7f4ef00cdfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:11.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.408+0000 7f4ef6400640 1 --2- 172.21.15.6:0/1841383662 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4ef0072520 0x7f4ef00cdfb0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f4ef006ec40 tx=0x7f4edc02b900 comp rx=0 tx=0).stop 2024-03-04T09:26:11.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.409+0000 7f4ef6400640 1 -- 172.21.15.6:0/1841383662 shutdown_connections 2024-03-04T09:26:11.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.409+0000 7f4ef6400640 1 --2- 172.21.15.6:0/1841383662 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4ef0071e20 0x7f4ef00ce4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:11.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.409+0000 7f4ef6400640 1 --2- 172.21.15.6:0/1841383662 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4ef0072520 0x7f4ef00cdfb0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:11.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.409+0000 7f4ef6400640 1 --2- 172.21.15.6:0/1841383662 >> v2:172.21.15.146:3300/0 conn(0x7f4ef015bf20 0x7f4ef00acd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:11.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.409+0000 7f4ef6400640 1 -- 172.21.15.6:0/1841383662 >> 172.21.15.6:0/1841383662 conn(0x7f4ef006b3d0 msgr2=0x7f4ef006b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:11.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.409+0000 7f4ef6400640 1 -- 172.21.15.6:0/1841383662 shutdown_connections 2024-03-04T09:26:11.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.409+0000 7f4ef6400640 1 -- 172.21.15.6:0/1841383662 wait complete. 2024-03-04T09:26:11.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.410+0000 7f4ef6400640 1 Processor -- start 2024-03-04T09:26:11.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.411+0000 7f4ef6400640 1 -- start start 2024-03-04T09:26:11.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.411+0000 7f4ef6400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4ef0071e20 0x7f4ef01f4160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:11.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.412+0000 7f4ef6400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4ef0072520 0x7f4ef009d110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:11.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.412+0000 7f4ef6400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4ef015bf20 0x7f4ef0142fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:11.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.412+0000 7f4ef6400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ef00b4b10 con 0x7f4ef015bf20 2024-03-04T09:26:11.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.412+0000 7f4ef6400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ef00b4990 con 0x7f4ef0072520 2024-03-04T09:26:11.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.412+0000 7f4ef6400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ef00ad280 con 0x7f4ef0071e20 2024-03-04T09:26:11.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.412+0000 7f4eefe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4ef015bf20 0x7f4ef0142fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:11.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.412+0000 7f4eefe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4ef015bf20 0x7f4ef0142fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:59718/0 (socket says 172.21.15.6:59718) 2024-03-04T09:26:11.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.412+0000 7f4eefe00640 1 -- 172.21.15.6:0/1356577558 learned_addr learned my addr 172.21.15.6:0/1356577558 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:11.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.412+0000 7f4eeea00640 1 --2- 172.21.15.6:0/1356577558 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4ef0072520 0x7f4ef009d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:11.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.412+0000 7f4eef400640 1 --2- 172.21.15.6:0/1356577558 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4ef0071e20 0x7f4ef01f4160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:11.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.412+0000 7f4eefe00640 1 -- 172.21.15.6:0/1356577558 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4ef0071e20 msgr2=0x7f4ef01f4160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:11.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.412+0000 7f4eefe00640 1 --2- 172.21.15.6:0/1356577558 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4ef0071e20 0x7f4ef01f4160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:11.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.412+0000 7f4eefe00640 1 -- 172.21.15.6:0/1356577558 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4ef0072520 msgr2=0x7f4ef009d110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:11.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.412+0000 7f4eefe00640 1 --2- 172.21.15.6:0/1356577558 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4ef0072520 0x7f4ef009d110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:11.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.412+0000 7f4eefe00640 1 -- 172.21.15.6:0/1356577558 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ef015e650 con 0x7f4ef015bf20 2024-03-04T09:26:11.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.413+0000 7f4eefe00640 1 --2- 172.21.15.6:0/1356577558 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4ef015bf20 0x7f4ef0142fb0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f4ee800d5d0 tx=0x7f4ee800da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:11.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.413+0000 7f4ee7400640 1 -- 172.21.15.6:0/1356577558 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ee80125f0 con 0x7f4ef015bf20 2024-03-04T09:26:11.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.413+0000 7f4ee7400640 1 -- 172.21.15.6:0/1356577558 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ee8019070 con 0x7f4ef015bf20 2024-03-04T09:26:11.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.413+0000 7f4ee7400640 1 -- 172.21.15.6:0/1356577558 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ee80129d0 con 0x7f4ef015bf20 2024-03-04T09:26:11.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.413+0000 7f4ef6400640 1 -- 172.21.15.6:0/1356577558 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ef015ec10 con 0x7f4ef015bf20 2024-03-04T09:26:11.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.413+0000 7f4ef6400640 1 -- 172.21.15.6:0/1356577558 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ef015f1d0 con 0x7f4ef015bf20 2024-03-04T09:26:11.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.414+0000 7f4ef6400640 1 -- 172.21.15.6:0/1356577558 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ef004f330 con 0x7f4ef015bf20 2024-03-04T09:26:11.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.415+0000 7f4ee7400640 1 -- 172.21.15.6:0/1356577558 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4ee8012b70 con 0x7f4ef015bf20 2024-03-04T09:26:11.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.417+0000 7f4ee7400640 1 --2- 172.21.15.6:0/1356577558 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f4ed006b4d0 0x7f4ed006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:11.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.417+0000 7f4ee7400640 1 -- 172.21.15.6:0/1356577558 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f4ee808a700 con 0x7f4ef015bf20 2024-03-04T09:26:11.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.418+0000 7f4eef400640 1 --2- 172.21.15.6:0/1356577558 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f4ed006b4d0 0x7f4ed006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:11.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.419+0000 7f4eef400640 1 --2- 172.21.15.6:0/1356577558 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f4ed006b4d0 0x7f4ed006d990 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f4ee0009940 tx=0x7f4ee0003cd0 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:11.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.422+0000 7f4ee7400640 1 -- 172.21.15.6:0/1356577558 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4ee80594e0 con 0x7f4ef015bf20 2024-03-04T09:26:11.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.623+0000 7f4ef6400640 1 -- 172.21.15.6:0/1356577558 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f4ef015b280 con 0x7f4ef015bf20 2024-03-04T09:26:11.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.624+0000 7f4ee7400640 1 -- 172.21.15.6:0/1356577558 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4ee8058de0 con 0x7f4ef015bf20 2024-03-04T09:26:11.621 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-04T09:26:11.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.627+0000 7f4ef6400640 1 -- 172.21.15.6:0/1356577558 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f4ed006b4d0 msgr2=0x7f4ed006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:11.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.627+0000 7f4ef6400640 1 --2- 172.21.15.6:0/1356577558 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f4ed006b4d0 0x7f4ed006d990 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f4ee0009940 tx=0x7f4ee0003cd0 comp rx=0 tx=0).stop 2024-03-04T09:26:11.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.627+0000 7f4ef6400640 1 -- 172.21.15.6:0/1356577558 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4ef015bf20 msgr2=0x7f4ef0142fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:11.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.627+0000 7f4ef6400640 1 --2- 172.21.15.6:0/1356577558 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4ef015bf20 0x7f4ef0142fb0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f4ee800d5d0 tx=0x7f4ee800da90 comp rx=0 tx=0).stop 2024-03-04T09:26:11.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.627+0000 7f4ef6400640 1 -- 172.21.15.6:0/1356577558 shutdown_connections 2024-03-04T09:26:11.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.627+0000 7f4ef6400640 1 --2- 172.21.15.6:0/1356577558 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4ef015bf20 0x7f4ef0142fb0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:11.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.627+0000 7f4ef6400640 1 --2- 172.21.15.6:0/1356577558 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4ef0072520 0x7f4ef009d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:11.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.627+0000 7f4ef6400640 1 --2- 172.21.15.6:0/1356577558 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f4ed006b4d0 0x7f4ed006d990 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:11.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.627+0000 7f4ef6400640 1 --2- 172.21.15.6:0/1356577558 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4ef0071e20 0x7f4ef01f4160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:11.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.627+0000 7f4ef6400640 1 -- 172.21.15.6:0/1356577558 >> 172.21.15.6:0/1356577558 conn(0x7f4ef006b3d0 msgr2=0x7f4ef00b3a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:11.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.628+0000 7f4ef6400640 1 -- 172.21.15.6:0/1356577558 shutdown_connections 2024-03-04T09:26:11.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.628+0000 7f4ef6400640 1 -- 172.21.15.6:0/1356577558 wait complete. 2024-03-04T09:26:11.645 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-04T09:26:11.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.864+0000 7f4fe6200640 1 Processor -- start 2024-03-04T09:26:11.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.865+0000 7f4fe6200640 1 -- start start 2024-03-04T09:26:11.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.866+0000 7f4fe6200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f4fe0147590 0x7f4fe0071480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:11.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.866+0000 7f4fe6200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4fe00b2250 0x7f4fe00b3640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:11.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.866+0000 7f4fe6200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fe00719c0 con 0x7f4fe006d8d0 2024-03-04T09:26:11.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.866+0000 7f4fe6200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4fe0147970 con 0x7f4fe0147590 2024-03-04T09:26:11.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.866+0000 7f4fe6200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fe00b31b0 con 0x7f4fe00b2250 2024-03-04T09:26:11.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.867+0000 7f4fdfe00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f4fe006d8d0 0x7f4fe00b1b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:60522/0 (socket says 172.21.15.6:60522) 2024-03-04T09:26:11.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.867+0000 7f4fdfe00640 1 -- 172.21.15.6:0/3520352742 learned_addr learned my addr 172.21.15.6:0/3520352742 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:26:11.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.867+0000 7f4fe4e00640 1 --2- 172.21.15.6:0/3520352742 >> v2:172.21.15.6:3300/0 conn(0x7f4fe0147590 0x7f4fe0071480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:11.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.867+0000 7f4fdf400640 1 -- 172.21.15.6:0/3520352742 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3822516627 0 0) 0x7f4fe00719c0 con 0x7f4fe006d8d0 2024-03-04T09:26:11.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.867+0000 7f4fdf400640 1 -- 172.21.15.6:0/3520352742 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4fc4003520 con 0x7f4fe006d8d0 2024-03-04T09:26:11.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.868+0000 7f4fe4e00640 1 -- 172.21.15.6:0/3520352742 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4fe00b2250 msgr2=0x7f4fe00b3640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:11.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.868+0000 7f4fe5800640 1 --2- 172.21.15.6:0/3520352742 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4fe00b2250 0x7f4fe00b3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:11.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.868+0000 7f4fe4e00640 1 --2- 172.21.15.6:0/3520352742 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4fe00b2250 0x7f4fe00b3640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:11.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.868+0000 7f4fe4e00640 1 -- 172.21.15.6:0/3520352742 >> v1:172.21.15.6:6789/0 conn(0x7f4fe006d8d0 legacy=0x7f4fe00b1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:11.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.868+0000 7f4fdf400640 1 -- 172.21.15.6:0/3520352742 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4004866358 0 0) 0x7f4fc4003520 con 0x7f4fe006d8d0 2024-03-04T09:26:11.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.868+0000 7f4fe4e00640 1 -- 172.21.15.6:0/3520352742 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fe006de40 con 0x7f4fe0147590 2024-03-04T09:26:11.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.868+0000 7f4fe5800640 1 --2- 172.21.15.6:0/3520352742 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4fe00b2250 0x7f4fe00b3640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:26:11.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.869+0000 7f4fe4e00640 1 --2- 172.21.15.6:0/3520352742 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4fe0147590 0x7f4fe0071480 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f4fd00097a0 tx=0x7f4fd002b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7200971d41be094 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:11.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.870+0000 7f4fdf400640 1 -- 172.21.15.6:0/3520352742 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fd0037070 con 0x7f4fe0147590 2024-03-04T09:26:11.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.870+0000 7f4fdf400640 1 -- 172.21.15.6:0/3520352742 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fd0030510 con 0x7f4fe0147590 2024-03-04T09:26:11.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.870+0000 7f4fdf400640 1 -- 172.21.15.6:0/3520352742 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fcc001690 con 0x7f4fe0147590 2024-03-04T09:26:11.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.870+0000 7f4fe6200640 1 -- 172.21.15.6:0/3520352742 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4fe0147590 msgr2=0x7f4fe0071480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:11.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.870+0000 7f4fe6200640 1 --2- 172.21.15.6:0/3520352742 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4fe0147590 0x7f4fe0071480 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f4fd00097a0 tx=0x7f4fd002b790 comp rx=0 tx=0).stop 2024-03-04T09:26:11.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.871+0000 7f4fe6200640 1 -- 172.21.15.6:0/3520352742 shutdown_connections 2024-03-04T09:26:11.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.871+0000 7f4fe6200640 1 --2- 172.21.15.6:0/3520352742 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4fe00b2250 0x7f4fe00b3640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:11.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.871+0000 7f4fe6200640 1 --2- 172.21.15.6:0/3520352742 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4fe0147590 0x7f4fe0071480 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:11.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.871+0000 7f4fe6200640 1 -- 172.21.15.6:0/3520352742 >> 172.21.15.6:0/3520352742 conn(0x7f4fe006b3a0 msgr2=0x7f4fe006b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:11.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.871+0000 7f4fe6200640 1 -- 172.21.15.6:0/3520352742 shutdown_connections 2024-03-04T09:26:11.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.871+0000 7f4fe6200640 1 -- 172.21.15.6:0/3520352742 wait complete. 2024-03-04T09:26:11.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.872+0000 7f4fe6200640 1 Processor -- start 2024-03-04T09:26:11.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.872+0000 7f4fe6200640 1 -- start start 2024-03-04T09:26:11.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.873+0000 7f4fe6200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4fe006d8d0 0x7f4fe015efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:11.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.873+0000 7f4fe6200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4fe00b2250 0x7f4fe013e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:11.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.873+0000 7f4fe6200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4fe0147590 0x7f4fe013eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:11.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.873+0000 7f4fe6200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fe00b2b50 con 0x7f4fe00b2250 2024-03-04T09:26:11.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.873+0000 7f4fe6200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fe00b2e40 con 0x7f4fe0147590 2024-03-04T09:26:11.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.873+0000 7f4fe6200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fe00b3b80 con 0x7f4fe006d8d0 2024-03-04T09:26:11.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.874+0000 7f4fdfe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4fe00b2250 0x7f4fe013e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:11.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.874+0000 7f4fdfe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4fe00b2250 0x7f4fe013e5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:59742/0 (socket says 172.21.15.6:59742) 2024-03-04T09:26:11.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.874+0000 7f4fdfe00640 1 -- 172.21.15.6:0/1879244647 learned_addr learned my addr 172.21.15.6:0/1879244647 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:11.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.874+0000 7f4fe4e00640 1 --2- 172.21.15.6:0/1879244647 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4fe006d8d0 0x7f4fe015efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:11.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.874+0000 7f4fe5800640 1 --2- 172.21.15.6:0/1879244647 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4fe0147590 0x7f4fe013eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:11.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.874+0000 7f4fdfe00640 1 -- 172.21.15.6:0/1879244647 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4fe006d8d0 msgr2=0x7f4fe015efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:11.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.874+0000 7f4fdfe00640 1 --2- 172.21.15.6:0/1879244647 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4fe006d8d0 0x7f4fe015efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:11.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.874+0000 7f4fdfe00640 1 -- 172.21.15.6:0/1879244647 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4fe0147590 msgr2=0x7f4fe013eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:11.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.874+0000 7f4fdfe00640 1 --2- 172.21.15.6:0/1879244647 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4fe0147590 0x7f4fe013eb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:11.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.874+0000 7f4fdfe00640 1 -- 172.21.15.6:0/1879244647 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fe013d820 con 0x7f4fe00b2250 2024-03-04T09:26:11.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.874+0000 7f4fe5800640 1 --2- 172.21.15.6:0/1879244647 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4fe0147590 0x7f4fe013eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:26:11.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.875+0000 7f4fdfe00640 1 --2- 172.21.15.6:0/1879244647 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4fe00b2250 0x7f4fe013e5c0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f4fcc00c750 tx=0x7f4fcc00cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:11.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.875+0000 7f4fdcc00640 1 -- 172.21.15.6:0/1879244647 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fcc018070 con 0x7f4fe00b2250 2024-03-04T09:26:11.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.875+0000 7f4fdcc00640 1 -- 172.21.15.6:0/1879244647 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fcc0072f0 con 0x7f4fe00b2250 2024-03-04T09:26:11.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.875+0000 7f4fdcc00640 1 -- 172.21.15.6:0/1879244647 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fcc0047e0 con 0x7f4fe00b2250 2024-03-04T09:26:11.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.875+0000 7f4fe6200640 1 -- 172.21.15.6:0/1879244647 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fe013c120 con 0x7f4fe00b2250 2024-03-04T09:26:11.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.875+0000 7f4fe6200640 1 -- 172.21.15.6:0/1879244647 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fe013ddc0 con 0x7f4fe00b2250 2024-03-04T09:26:11.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.877+0000 7f4fe6200640 1 -- 172.21.15.6:0/1879244647 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fe00ae9b0 con 0x7f4fe00b2250 2024-03-04T09:26:11.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.877+0000 7f4fdcc00640 1 -- 172.21.15.6:0/1879244647 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4fcc01b070 con 0x7f4fe00b2250 2024-03-04T09:26:11.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.878+0000 7f4fdcc00640 1 --2- 172.21.15.6:0/1879244647 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f4fc006b4d0 0x7f4fc006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:11.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.878+0000 7f4fdcc00640 1 -- 172.21.15.6:0/1879244647 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f4fcc0896d0 con 0x7f4fe00b2250 2024-03-04T09:26:11.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.879+0000 7f4fe4e00640 1 --2- 172.21.15.6:0/1879244647 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f4fc006b4d0 0x7f4fc006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:11.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.880+0000 7f4fe4e00640 1 --2- 172.21.15.6:0/1879244647 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f4fc006b4d0 0x7f4fc006d990 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f4fd0034000 tx=0x7f4fd002f000 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:11.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:11.883+0000 7f4fdcc00640 1 -- 172.21.15.6:0/1879244647 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4fe00ae9b0 con 0x7f4fe00b2250 2024-03-04T09:26:12.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.086+0000 7f4fe6200640 1 -- 172.21.15.6:0/1879244647 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f4fe00aee50 con 0x7f4fe00b2250 2024-03-04T09:26:12.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.086+0000 7f4fdcc00640 1 -- 172.21.15.6:0/1879244647 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7f4fcc0584c0 con 0x7f4fe00b2250 2024-03-04T09:26:12.083 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-03-04T09:26:12.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.089+0000 7f4fe6200640 1 -- 172.21.15.6:0/1879244647 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f4fc006b4d0 msgr2=0x7f4fc006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:12.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.090+0000 7f4fe6200640 1 --2- 172.21.15.6:0/1879244647 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f4fc006b4d0 0x7f4fc006d990 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f4fd0034000 tx=0x7f4fd002f000 comp rx=0 tx=0).stop 2024-03-04T09:26:12.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.090+0000 7f4fe6200640 1 -- 172.21.15.6:0/1879244647 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4fe00b2250 msgr2=0x7f4fe013e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:12.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.090+0000 7f4fe6200640 1 --2- 172.21.15.6:0/1879244647 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4fe00b2250 0x7f4fe013e5c0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f4fcc00c750 tx=0x7f4fcc00cc10 comp rx=0 tx=0).stop 2024-03-04T09:26:12.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.090+0000 7f4fe6200640 1 -- 172.21.15.6:0/1879244647 shutdown_connections 2024-03-04T09:26:12.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.090+0000 7f4fe6200640 1 --2- 172.21.15.6:0/1879244647 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4fe0147590 0x7f4fe013eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:12.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.090+0000 7f4fe6200640 1 --2- 172.21.15.6:0/1879244647 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4fe00b2250 0x7f4fe013e5c0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:12.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.090+0000 7f4fe6200640 1 --2- 172.21.15.6:0/1879244647 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f4fc006b4d0 0x7f4fc006d990 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:12.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.090+0000 7f4fe6200640 1 --2- 172.21.15.6:0/1879244647 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4fe006d8d0 0x7f4fe015efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:12.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.090+0000 7f4fe6200640 1 -- 172.21.15.6:0/1879244647 >> 172.21.15.6:0/1879244647 conn(0x7f4fe006b3a0 msgr2=0x7f4fe006b780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:12.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.090+0000 7f4fe6200640 1 -- 172.21.15.6:0/1879244647 shutdown_connections 2024-03-04T09:26:12.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.090+0000 7f4fe6200640 1 -- 172.21.15.6:0/1879244647 wait complete. 2024-03-04T09:26:12.106 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-04T09:26:12.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.331+0000 7f0c31400640 1 Processor -- start 2024-03-04T09:26:12.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.332+0000 7f0c31400640 1 -- start start 2024-03-04T09:26:12.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.333+0000 7f0c31400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f0c2c15dee0 0x7f0c2c0d18a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:12.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.333+0000 7f0c31400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0c2c0d4860 0x7f0c2c0d0630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:12.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.333+0000 7f0c31400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f0c2c0cfa60 0x7f0c2c0d0b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:12.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.333+0000 7f0c31400640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c2c0cf5d0 con 0x7f0c2c0cfa60 2024-03-04T09:26:12.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.333+0000 7f0c31400640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c2c0ceca0 con 0x7f0c2c15dee0 2024-03-04T09:26:12.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.333+0000 7f0c31400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c2c0cef80 con 0x7f0c2c0d4860 2024-03-04T09:26:12.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.334+0000 7f0c30a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f0c2c0cfa60 0x7f0c2c0d0b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:12.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.334+0000 7f0c30a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f0c2c0cfa60 0x7f0c2c0d0b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:59754/0 (socket says 172.21.15.6:59754) 2024-03-04T09:26:12.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.334+0000 7f0c30a00640 1 -- 172.21.15.6:0/2890904573 learned_addr learned my addr 172.21.15.6:0/2890904573 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:12.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.334+0000 7f0c2b400640 1 --2- 172.21.15.6:0/2890904573 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0c2c0d4860 0x7f0c2c0d0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:12.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.334+0000 7f0c2be00640 1 --2- 172.21.15.6:0/2890904573 >> v2:172.21.15.146:3300/0 conn(0x7f0c2c15dee0 0x7f0c2c0d18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:12.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.334+0000 7f0c30a00640 1 -- 172.21.15.6:0/2890904573 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0c2c0d4860 msgr2=0x7f0c2c0d0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:12.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.334+0000 7f0c30a00640 1 --2- 172.21.15.6:0/2890904573 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0c2c0d4860 0x7f0c2c0d0630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:12.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.334+0000 7f0c30a00640 1 -- 172.21.15.6:0/2890904573 >> v2:172.21.15.146:3300/0 conn(0x7f0c2c15dee0 msgr2=0x7f0c2c0d18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:12.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.334+0000 7f0c30a00640 1 --2- 172.21.15.6:0/2890904573 >> v2:172.21.15.146:3300/0 conn(0x7f0c2c15dee0 0x7f0c2c0d18a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:12.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.334+0000 7f0c30a00640 1 -- 172.21.15.6:0/2890904573 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c2c0cf260 con 0x7f0c2c0cfa60 2024-03-04T09:26:12.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.336+0000 7f0c30a00640 1 --2- 172.21.15.6:0/2890904573 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0c2c0cfa60 0x7f0c2c0d0b70 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f0c2c08c6f0 tx=0x7f0c2402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6eb421242d359ab server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:12.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.336+0000 7f0c2aa00640 1 -- 172.21.15.6:0/2890904573 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c24035020 con 0x7f0c2c0cfa60 2024-03-04T09:26:12.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.336+0000 7f0c2aa00640 1 -- 172.21.15.6:0/2890904573 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c24031870 con 0x7f0c2c0cfa60 2024-03-04T09:26:12.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.336+0000 7f0c2aa00640 1 -- 172.21.15.6:0/2890904573 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c24031c40 con 0x7f0c2c0cfa60 2024-03-04T09:26:12.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.337+0000 7f0c31400640 1 -- 172.21.15.6:0/2890904573 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0c2c0cfa60 msgr2=0x7f0c2c0d0b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:12.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.337+0000 7f0c31400640 1 --2- 172.21.15.6:0/2890904573 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0c2c0cfa60 0x7f0c2c0d0b70 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f0c2c08c6f0 tx=0x7f0c2402b910 comp rx=0 tx=0).stop 2024-03-04T09:26:12.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.337+0000 7f0c31400640 1 -- 172.21.15.6:0/2890904573 shutdown_connections 2024-03-04T09:26:12.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.337+0000 7f0c31400640 1 --2- 172.21.15.6:0/2890904573 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0c2c0cfa60 0x7f0c2c0d0b70 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:12.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.337+0000 7f0c31400640 1 --2- 172.21.15.6:0/2890904573 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0c2c0d4860 0x7f0c2c0d0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:12.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.337+0000 7f0c31400640 1 --2- 172.21.15.6:0/2890904573 >> v2:172.21.15.146:3300/0 conn(0x7f0c2c15dee0 0x7f0c2c0d18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:12.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.337+0000 7f0c31400640 1 -- 172.21.15.6:0/2890904573 >> 172.21.15.6:0/2890904573 conn(0x7f0c2c0d3cd0 msgr2=0x7f0c2c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:12.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.337+0000 7f0c31400640 1 -- 172.21.15.6:0/2890904573 shutdown_connections 2024-03-04T09:26:12.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.337+0000 7f0c31400640 1 -- 172.21.15.6:0/2890904573 wait complete. 2024-03-04T09:26:12.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.339+0000 7f0c31400640 1 Processor -- start 2024-03-04T09:26:12.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.339+0000 7f0c31400640 1 -- start start 2024-03-04T09:26:12.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.340+0000 7f0c31400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0c2c0cfa60 0x7f0c2c1523e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:12.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.340+0000 7f0c31400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0c2c0d4860 0x7f0c2c160a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:12.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.340+0000 7f0c31400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0c2c15dee0 0x7f0c2c160f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:12.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.340+0000 7f0c31400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c2c0d1de0 con 0x7f0c2c15dee0 2024-03-04T09:26:12.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.340+0000 7f0c31400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c2c15e2c0 con 0x7f0c2c0cfa60 2024-03-04T09:26:12.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.340+0000 7f0c31400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c2c0d1f60 con 0x7f0c2c0d4860 2024-03-04T09:26:12.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.340+0000 7f0c30a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0c2c15dee0 0x7f0c2c160f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:12.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.340+0000 7f0c30a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0c2c15dee0 0x7f0c2c160f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:59768/0 (socket says 172.21.15.6:59768) 2024-03-04T09:26:12.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.340+0000 7f0c30a00640 1 -- 172.21.15.6:0/2206588195 learned_addr learned my addr 172.21.15.6:0/2206588195 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:12.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.341+0000 7f0c2b400640 1 --2- 172.21.15.6:0/2206588195 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0c2c0d4860 0x7f0c2c160a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:12.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.341+0000 7f0c2be00640 1 --2- 172.21.15.6:0/2206588195 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0c2c0cfa60 0x7f0c2c1523e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:12.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.341+0000 7f0c30a00640 1 -- 172.21.15.6:0/2206588195 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0c2c0d4860 msgr2=0x7f0c2c160a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:12.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.341+0000 7f0c30a00640 1 --2- 172.21.15.6:0/2206588195 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0c2c0d4860 0x7f0c2c160a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:12.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.341+0000 7f0c30a00640 1 -- 172.21.15.6:0/2206588195 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0c2c0cfa60 msgr2=0x7f0c2c1523e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:12.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.341+0000 7f0c30a00640 1 --2- 172.21.15.6:0/2206588195 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0c2c0cfa60 0x7f0c2c1523e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:12.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.341+0000 7f0c30a00640 1 -- 172.21.15.6:0/2206588195 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c2c15f780 con 0x7f0c2c15dee0 2024-03-04T09:26:12.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.341+0000 7f0c30a00640 1 --2- 172.21.15.6:0/2206588195 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0c2c15dee0 0x7f0c2c160f40 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f0c2402b950 tx=0x7f0c240046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:12.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.341+0000 7f0c23e00640 1 -- 172.21.15.6:0/2206588195 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c2403acb0 con 0x7f0c2c15dee0 2024-03-04T09:26:12.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.341+0000 7f0c23e00640 1 -- 172.21.15.6:0/2206588195 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c2403ae50 con 0x7f0c2c15dee0 2024-03-04T09:26:12.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.341+0000 7f0c23e00640 1 -- 172.21.15.6:0/2206588195 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c24044630 con 0x7f0c2c15dee0 2024-03-04T09:26:12.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.341+0000 7f0c31400640 1 -- 172.21.15.6:0/2206588195 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c2c0c9670 con 0x7f0c2c15dee0 2024-03-04T09:26:12.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.342+0000 7f0c31400640 1 -- 172.21.15.6:0/2206588195 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c2c0c9c30 con 0x7f0c2c15dee0 2024-03-04T09:26:12.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.343+0000 7f0c31400640 1 -- 172.21.15.6:0/2206588195 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c2c04f330 con 0x7f0c2c15dee0 2024-03-04T09:26:12.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.344+0000 7f0c23e00640 1 -- 172.21.15.6:0/2206588195 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0c24038020 con 0x7f0c2c15dee0 2024-03-04T09:26:12.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.344+0000 7f0c23e00640 1 --2- 172.21.15.6:0/2206588195 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f0c0806b4d0 0x7f0c0806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:12.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.345+0000 7f0c23e00640 1 -- 172.21.15.6:0/2206588195 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f0c240ac4a0 con 0x7f0c2c15dee0 2024-03-04T09:26:12.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.345+0000 7f0c2be00640 1 --2- 172.21.15.6:0/2206588195 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f0c0806b4d0 0x7f0c0806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:12.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.346+0000 7f0c2be00640 1 --2- 172.21.15.6:0/2206588195 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f0c0806b4d0 0x7f0c0806d990 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f0c2c0b9520 tx=0x7f0c1c00b040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:12.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.349+0000 7f0c23e00640 1 -- 172.21.15.6:0/2206588195 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0c2c15f780 con 0x7f0c2c15dee0 2024-03-04T09:26:12.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.550+0000 7f0c31400640 1 -- 172.21.15.6:0/2206588195 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f0c2c08d7d0 con 0x7f0c2c15dee0 2024-03-04T09:26:12.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.551+0000 7f0c23e00640 1 -- 172.21.15.6:0/2206588195 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7f0c2403f070 con 0x7f0c2c15dee0 2024-03-04T09:26:12.548 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 64 2024-03-04T09:26:12.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.553+0000 7f0c31400640 1 -- 172.21.15.6:0/2206588195 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f0c0806b4d0 msgr2=0x7f0c0806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:12.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.553+0000 7f0c31400640 1 --2- 172.21.15.6:0/2206588195 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f0c0806b4d0 0x7f0c0806d990 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f0c2c0b9520 tx=0x7f0c1c00b040 comp rx=0 tx=0).stop 2024-03-04T09:26:12.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.553+0000 7f0c31400640 1 -- 172.21.15.6:0/2206588195 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0c2c15dee0 msgr2=0x7f0c2c160f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:12.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.553+0000 7f0c31400640 1 --2- 172.21.15.6:0/2206588195 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0c2c15dee0 0x7f0c2c160f40 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f0c2402b950 tx=0x7f0c240046c0 comp rx=0 tx=0).stop 2024-03-04T09:26:12.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.553+0000 7f0c31400640 1 -- 172.21.15.6:0/2206588195 shutdown_connections 2024-03-04T09:26:12.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.553+0000 7f0c31400640 1 --2- 172.21.15.6:0/2206588195 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0c2c15dee0 0x7f0c2c160f40 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:12.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.553+0000 7f0c31400640 1 --2- 172.21.15.6:0/2206588195 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f0c0806b4d0 0x7f0c0806d990 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:12.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.553+0000 7f0c31400640 1 --2- 172.21.15.6:0/2206588195 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0c2c0d4860 0x7f0c2c160a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:12.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.553+0000 7f0c31400640 1 --2- 172.21.15.6:0/2206588195 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0c2c0cfa60 0x7f0c2c1523e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:12.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.553+0000 7f0c31400640 1 -- 172.21.15.6:0/2206588195 >> 172.21.15.6:0/2206588195 conn(0x7f0c2c0d3cd0 msgr2=0x7f0c2c0cff60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:12.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.554+0000 7f0c31400640 1 -- 172.21.15.6:0/2206588195 shutdown_connections 2024-03-04T09:26:12.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.554+0000 7f0c31400640 1 -- 172.21.15.6:0/2206588195 wait complete. 2024-03-04T09:26:12.568 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-04T09:26:12.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.794+0000 7f5717200640 1 Processor -- start 2024-03-04T09:26:12.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.795+0000 7f5717200640 1 -- start start 2024-03-04T09:26:12.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.796+0000 7f5717200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5718070f10 0x7f57180cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:12.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.796+0000 7f5717200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57180714d0 con 0x7f57180cce90 2024-03-04T09:26:12.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.796+0000 7f5717200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57180717b0 con 0x7f5718145300 2024-03-04T09:26:12.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.796+0000 7f5717200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57180cd300 con 0x7f5718070f10 2024-03-04T09:26:12.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.797+0000 7f5716800640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f57180cce90 0x7f571815c570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:60528/0 (socket says 172.21.15.6:60528) 2024-03-04T09:26:12.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.797+0000 7f5716800640 1 -- 172.21.15.6:0/3341774936 learned_addr learned my addr 172.21.15.6:0/3341774936 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:26:12.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.797+0000 7f570fe00640 1 --2- 172.21.15.6:0/3341774936 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5718070f10 0x7f57180cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:12.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.797+0000 7f5715400640 1 -- 172.21.15.6:0/3341774936 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 255544350 0 0) 0x7f57180714d0 con 0x7f57180cce90 2024-03-04T09:26:12.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.797+0000 7f5715400640 1 -- 172.21.15.6:0/3341774936 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56fc003520 con 0x7f57180cce90 2024-03-04T09:26:12.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.798+0000 7f5715400640 1 -- 172.21.15.6:0/3341774936 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3517445729 0 0) 0x7f57180717b0 con 0x7f5718145300 2024-03-04T09:26:12.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.798+0000 7f5715400640 1 -- 172.21.15.6:0/3341774936 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57180714d0 con 0x7f5718145300 2024-03-04T09:26:12.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.798+0000 7f5715400640 1 -- 172.21.15.6:0/3341774936 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 586518980 0 0) 0x7f56fc003520 con 0x7f57180cce90 2024-03-04T09:26:12.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.798+0000 7f5715400640 1 -- 172.21.15.6:0/3341774936 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5718070f10 msgr2=0x7f57180cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:12.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.798+0000 7f5715400640 1 --2- 172.21.15.6:0/3341774936 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5718070f10 0x7f57180cc950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:12.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.798+0000 7f5715400640 1 -- 172.21.15.6:0/3341774936 >> v1:172.21.15.146:6789/0 conn(0x7f5718145300 legacy=0x7f57180b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:12.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.798+0000 7f5715400640 1 -- 172.21.15.6:0/3341774936 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5718071a90 con 0x7f57180cce90 2024-03-04T09:26:12.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.798+0000 7f5715400640 1 -- 172.21.15.6:0/3341774936 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7f57100019f0 con 0x7f57180cce90 2024-03-04T09:26:12.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.798+0000 7f5715400640 1 -- 172.21.15.6:0/3341774936 >> v1:172.21.15.6:6789/0 conn(0x7f57180cce90 legacy=0x7f571815c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:12.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.798+0000 7f5715400640 1 --2- 172.21.15.6:0/3341774936 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f56fc003b70 0x7f56fc003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:12.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.798+0000 7f5715400640 1 --2- 172.21.15.6:0/3341774936 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f56fc004800 0x7f56fc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:12.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.798+0000 7f5715400640 1 --2- 172.21.15.6:0/3341774936 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5718070f10 0x7f56fc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:12.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.798+0000 7f5715400640 1 -- 172.21.15.6:0/3341774936 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57180cd300 con 0x7f56fc004800 2024-03-04T09:26:12.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.798+0000 7f5715400640 1 -- 172.21.15.6:0/3341774936 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56fc00ba70 con 0x7f56fc003b70 2024-03-04T09:26:12.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.798+0000 7f5715400640 1 -- 172.21.15.6:0/3341774936 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56fc00bc40 con 0x7f5718070f10 2024-03-04T09:26:12.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.798+0000 7f570fe00640 1 --2- 172.21.15.6:0/3341774936 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f56fc004800 0x7f56fc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:12.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.798+0000 7f5715e00640 1 --2- 172.21.15.6:0/3341774936 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f56fc003b70 0x7f56fc003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:12.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.799+0000 7f570fe00640 1 -- 172.21.15.6:0/3341774936 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5718070f10 msgr2=0x7f56fc0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:12.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.799+0000 7f570fe00640 1 --2- 172.21.15.6:0/3341774936 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5718070f10 0x7f56fc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:12.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.799+0000 7f570fe00640 1 -- 172.21.15.6:0/3341774936 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f56fc003b70 msgr2=0x7f56fc003f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:12.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.799+0000 7f570fe00640 1 --2- 172.21.15.6:0/3341774936 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f56fc003b70 0x7f56fc003f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:12.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.799+0000 7f570fe00640 1 -- 172.21.15.6:0/3341774936 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56fc003520 con 0x7f56fc004800 2024-03-04T09:26:12.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.800+0000 7f570fe00640 1 --2- 172.21.15.6:0/3341774936 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f56fc004800 0x7f56fc004c80 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f5700009ac0 tx=0x7f56fc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab9aa168bb4ddeeb server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:12.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.800+0000 7f5715400640 1 -- 172.21.15.6:0/3341774936 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5700039070 con 0x7f56fc004800 2024-03-04T09:26:12.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.800+0000 7f5715400640 1 -- 172.21.15.6:0/3341774936 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f570002c680 con 0x7f56fc004800 2024-03-04T09:26:12.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.800+0000 7f5715400640 1 -- 172.21.15.6:0/3341774936 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5700009690 con 0x7f56fc004800 2024-03-04T09:26:12.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.801+0000 7f5717200640 1 -- 172.21.15.6:0/3341774936 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f56fc004800 msgr2=0x7f56fc004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:12.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.801+0000 7f5717200640 1 --2- 172.21.15.6:0/3341774936 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f56fc004800 0x7f56fc004c80 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f5700009ac0 tx=0x7f56fc005830 comp rx=0 tx=0).stop 2024-03-04T09:26:12.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.801+0000 7f5716800640 1 -- 172.21.15.6:0/3341774936 reap_dead start 2024-03-04T09:26:12.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.801+0000 7f5717200640 1 -- 172.21.15.6:0/3341774936 shutdown_connections 2024-03-04T09:26:12.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.801+0000 7f5717200640 1 -- 172.21.15.6:0/3341774936 >> 172.21.15.6:0/3341774936 conn(0x7f571806b3d0 msgr2=0x7f571806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:12.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.801+0000 7f5717200640 1 -- 172.21.15.6:0/3341774936 shutdown_connections 2024-03-04T09:26:12.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.801+0000 7f5717200640 1 -- 172.21.15.6:0/3341774936 wait complete. 2024-03-04T09:26:12.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.803+0000 7f5717200640 1 Processor -- start 2024-03-04T09:26:12.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.803+0000 7f5717200640 1 -- start start 2024-03-04T09:26:12.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.804+0000 7f5717200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f56fc004800 0x7f571806da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:12.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.804+0000 7f5717200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f571815b3e0 0x7f571815a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:12.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.804+0000 7f5717200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f57180adff0 0x7f5718141d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:12.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.804+0000 7f5717200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57180cc950 con 0x7f56fc004800 2024-03-04T09:26:12.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.804+0000 7f5717200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57181456e0 con 0x7f57180adff0 2024-03-04T09:26:12.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.804+0000 7f5717200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57180ccad0 con 0x7f571815b3e0 2024-03-04T09:26:12.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.804+0000 7f5715e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f56fc004800 0x7f571806da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:12.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.804+0000 7f5715e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f56fc004800 0x7f571806da70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:59782/0 (socket says 172.21.15.6:59782) 2024-03-04T09:26:12.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.804+0000 7f5715e00640 1 -- 172.21.15.6:0/4193499936 learned_addr learned my addr 172.21.15.6:0/4193499936 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:12.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.804+0000 7f5715e00640 1 -- 172.21.15.6:0/4193499936 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f571815b3e0 msgr2=0x7f571815a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:12.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.804+0000 7f5715e00640 1 --2- 172.21.15.6:0/4193499936 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f571815b3e0 0x7f571815a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:12.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.804+0000 7f5715e00640 1 -- 172.21.15.6:0/4193499936 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f57180adff0 msgr2=0x7f5718141d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:12.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.804+0000 7f5715e00640 1 --2- 172.21.15.6:0/4193499936 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f57180adff0 0x7f5718141d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:12.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.805+0000 7f5715e00640 1 -- 172.21.15.6:0/4193499936 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f571806e0c0 con 0x7f56fc004800 2024-03-04T09:26:12.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.805+0000 7f5715e00640 1 --2- 172.21.15.6:0/4193499936 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f56fc004800 0x7f571806da70 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f570800e7e0 tx=0x7f570800eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:12.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.805+0000 7f570e000640 1 -- 172.21.15.6:0/4193499936 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f570800ee90 con 0x7f56fc004800 2024-03-04T09:26:12.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.805+0000 7f570e000640 1 -- 172.21.15.6:0/4193499936 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5708018070 con 0x7f56fc004800 2024-03-04T09:26:12.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.805+0000 7f570e000640 1 -- 172.21.15.6:0/4193499936 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f570800c6c0 con 0x7f56fc004800 2024-03-04T09:26:12.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.805+0000 7f5717200640 1 -- 172.21.15.6:0/4193499936 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f571806e3a0 con 0x7f56fc004800 2024-03-04T09:26:12.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.805+0000 7f5717200640 1 -- 172.21.15.6:0/4193499936 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f571815b820 con 0x7f56fc004800 2024-03-04T09:26:12.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.806+0000 7f5717200640 1 -- 172.21.15.6:0/4193499936 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f571804f330 con 0x7f56fc004800 2024-03-04T09:26:12.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.807+0000 7f570e000640 1 -- 172.21.15.6:0/4193499936 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f570800c860 con 0x7f56fc004800 2024-03-04T09:26:12.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.808+0000 7f570e000640 1 --2- 172.21.15.6:0/4193499936 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f56f406b670 0x7f56f406db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:12.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.808+0000 7f570e000640 1 -- 172.21.15.6:0/4193499936 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f570808a100 con 0x7f56fc004800 2024-03-04T09:26:12.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.809+0000 7f570fe00640 1 --2- 172.21.15.6:0/4193499936 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f56f406b670 0x7f56f406db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:12.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.810+0000 7f570fe00640 1 --2- 172.21.15.6:0/4193499936 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f56f406b670 0x7f56f406db30 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f5700036000 tx=0x7f570002e000 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:12.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:12.813+0000 7f570e000640 1 -- 172.21.15.6:0/4193499936 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5708058ee0 con 0x7f56fc004800 2024-03-04T09:26:13.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.032+0000 7f5717200640 1 -- 172.21.15.6:0/4193499936 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f571806ba90 con 0x7f56fc004800 2024-03-04T09:26:13.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.033+0000 7f570e000640 1 -- 172.21.15.6:0/4193499936 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v22) v1 ==== 80+0+0 (secure 0 0 0) 0x7f57080587e0 con 0x7f56fc004800 2024-03-04T09:26:13.029 INFO:teuthology.orchestra.run.smithi006.stderr:listed 0 entries 2024-03-04T09:26:13.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.036+0000 7f5717200640 1 -- 172.21.15.6:0/4193499936 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f56f406b670 msgr2=0x7f56f406db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:13.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.036+0000 7f5717200640 1 --2- 172.21.15.6:0/4193499936 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f56f406b670 0x7f56f406db30 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f5700036000 tx=0x7f570002e000 comp rx=0 tx=0).stop 2024-03-04T09:26:13.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.036+0000 7f5717200640 1 -- 172.21.15.6:0/4193499936 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f56fc004800 msgr2=0x7f571806da70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:13.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.036+0000 7f5717200640 1 --2- 172.21.15.6:0/4193499936 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f56fc004800 0x7f571806da70 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f570800e7e0 tx=0x7f570800eca0 comp rx=0 tx=0).stop 2024-03-04T09:26:13.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.036+0000 7f5717200640 1 -- 172.21.15.6:0/4193499936 shutdown_connections 2024-03-04T09:26:13.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.036+0000 7f5717200640 1 --2- 172.21.15.6:0/4193499936 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f57180adff0 0x7f5718141d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:13.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.036+0000 7f5717200640 1 --2- 172.21.15.6:0/4193499936 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f56f406b670 0x7f56f406db30 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:13.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.036+0000 7f5717200640 1 --2- 172.21.15.6:0/4193499936 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f571815b3e0 0x7f571815a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:13.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.036+0000 7f5717200640 1 --2- 172.21.15.6:0/4193499936 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f56fc004800 0x7f571806da70 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:13.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.036+0000 7f5717200640 1 -- 172.21.15.6:0/4193499936 >> 172.21.15.6:0/4193499936 conn(0x7f571806b3d0 msgr2=0x7f5718143650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:13.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.036+0000 7f5717200640 1 -- 172.21.15.6:0/4193499936 shutdown_connections 2024-03-04T09:26:13.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.036+0000 7f5717200640 1 -- 172.21.15.6:0/4193499936 wait complete. 2024-03-04T09:26:13.053 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.3... 2024-03-04T09:26:13.053 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:26:13.053 DEBUG:teuthology.orchestra.run.smithi146:> sudo umount /home/ubuntu/cephtest/mnt.3 -f 2024-03-04T09:26:13.153 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi146.front.sepia.ceph.com 2024-03-04T09:26:13.154 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:26:13.154 DEBUG:teuthology.orchestra.run.smithi146:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-04T09:26:13.204 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-04T09:26:13.204 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:26:13.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.446+0000 7f5310a00640 1 Processor -- start 2024-03-04T09:26:13.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.446+0000 7f5310a00640 1 -- start start 2024-03-04T09:26:13.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.447+0000 7f5310a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f530c15dec0 0x7f530c0ce810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:13.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.447+0000 7f5310a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f530c15e2a0 0x7f530c0ced50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:13.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.447+0000 7f5310a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f530c0cf720 con 0x7f530c0cf290 2024-03-04T09:26:13.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.447+0000 7f5310a00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f530c0b0ea0 con 0x7f530c15e2a0 2024-03-04T09:26:13.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.447+0000 7f5310a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f530c0d1120 con 0x7f530c15dec0 2024-03-04T09:26:13.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.448+0000 7f530b200640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f530c0cf290 0x7f530c0d1b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:60542/0 (socket says 172.21.15.6:60542) 2024-03-04T09:26:13.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.448+0000 7f530b200640 1 -- 172.21.15.6:0/65888178 learned_addr learned my addr 172.21.15.6:0/65888178 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:26:13.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.448+0000 7f5301e00640 1 --2- 172.21.15.6:0/65888178 >> v2:172.21.15.6:3300/0 conn(0x7f530c15e2a0 0x7f530c0ced50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:13.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.448+0000 7f530a800640 1 --2- 172.21.15.6:0/65888178 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f530c15dec0 0x7f530c0ce810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:13.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.448+0000 7f5301e00640 1 -- 172.21.15.6:0/65888178 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f530c15dec0 msgr2=0x7f530c0ce810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:13.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.448+0000 7f5301e00640 1 --2- 172.21.15.6:0/65888178 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f530c15dec0 0x7f530c0ce810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:13.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.448+0000 7f5301e00640 1 -- 172.21.15.6:0/65888178 >> v1:172.21.15.6:6789/0 conn(0x7f530c0cf290 legacy=0x7f530c0d1b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:13.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.448+0000 7f5301e00640 1 -- 172.21.15.6:0/65888178 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f530c0d1400 con 0x7f530c15e2a0 2024-03-04T09:26:13.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.448+0000 7f5309e00640 1 -- 172.21.15.6:0/65888178 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2558507815 0 0) 0x7f5304001490 con 0x7f530c0cf290 2024-03-04T09:26:13.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.448+0000 7f530a800640 1 --2- 172.21.15.6:0/65888178 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f530c15dec0 0x7f530c0ce810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T09:26:13.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.450+0000 7f5301e00640 1 --2- 172.21.15.6:0/65888178 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f530c15e2a0 0x7f530c0ced50 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f52f40097a0 tx=0x7f52f402b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a044199d2ccd6ff7 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:13.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.450+0000 7f5309e00640 1 -- 172.21.15.6:0/65888178 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52f4035020 con 0x7f530c15e2a0 2024-03-04T09:26:13.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.450+0000 7f5309e00640 1 -- 172.21.15.6:0/65888178 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52f4031810 con 0x7f530c15e2a0 2024-03-04T09:26:13.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.450+0000 7f5309e00640 1 -- 172.21.15.6:0/65888178 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52f4031be0 con 0x7f530c15e2a0 2024-03-04T09:26:13.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.451+0000 7f5310a00640 1 -- 172.21.15.6:0/65888178 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f530c15e2a0 msgr2=0x7f530c0ced50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:13.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.451+0000 7f5310a00640 1 --2- 172.21.15.6:0/65888178 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f530c15e2a0 0x7f530c0ced50 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f52f40097a0 tx=0x7f52f402b8b0 comp rx=0 tx=0).stop 2024-03-04T09:26:13.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.451+0000 7f5310a00640 1 -- 172.21.15.6:0/65888178 shutdown_connections 2024-03-04T09:26:13.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.451+0000 7f5310a00640 1 --2- 172.21.15.6:0/65888178 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f530c15e2a0 0x7f530c0ced50 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:13.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.451+0000 7f5310a00640 1 --2- 172.21.15.6:0/65888178 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f530c15dec0 0x7f530c0ce810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:13.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.451+0000 7f5310a00640 1 -- 172.21.15.6:0/65888178 >> 172.21.15.6:0/65888178 conn(0x7f530c0d3d40 msgr2=0x7f530c0d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:13.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.451+0000 7f5310a00640 1 -- 172.21.15.6:0/65888178 shutdown_connections 2024-03-04T09:26:13.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.451+0000 7f5310a00640 1 -- 172.21.15.6:0/65888178 wait complete. 2024-03-04T09:26:13.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.453+0000 7f5310a00640 1 Processor -- start 2024-03-04T09:26:13.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.453+0000 7f5310a00640 1 -- start start 2024-03-04T09:26:13.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.454+0000 7f5310a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f530c0cf290 0x7f530c152d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:13.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.454+0000 7f5310a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f530c15dec0 0x7f530c1f4930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:13.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.454+0000 7f5310a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f530c15e2a0 0x7f530c1f5440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:13.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.454+0000 7f5310a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f530c0d0ec0 con 0x7f530c0cf290 2024-03-04T09:26:13.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.454+0000 7f5310a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f530c0d0d40 con 0x7f530c15e2a0 2024-03-04T09:26:13.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.454+0000 7f5310a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f530c0b2340 con 0x7f530c15dec0 2024-03-04T09:26:13.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.454+0000 7f530a800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f530c0cf290 0x7f530c152d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:13.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.454+0000 7f530a800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f530c0cf290 0x7f530c152d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:59812/0 (socket says 172.21.15.6:59812) 2024-03-04T09:26:13.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.454+0000 7f530a800640 1 -- 172.21.15.6:0/977798827 learned_addr learned my addr 172.21.15.6:0/977798827 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:13.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.455+0000 7f5301e00640 1 --2- 172.21.15.6:0/977798827 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f530c15dec0 0x7f530c1f4930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:13.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.455+0000 7f530a800640 1 -- 172.21.15.6:0/977798827 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f530c15dec0 msgr2=0x7f530c1f4930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:13.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.455+0000 7f530a800640 1 --2- 172.21.15.6:0/977798827 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f530c15dec0 0x7f530c1f4930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:13.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.455+0000 7f530b200640 1 --2- 172.21.15.6:0/977798827 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f530c15e2a0 0x7f530c1f5440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:13.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.455+0000 7f530a800640 1 -- 172.21.15.6:0/977798827 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f530c15e2a0 msgr2=0x7f530c1f5440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:13.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.455+0000 7f530a800640 1 --2- 172.21.15.6:0/977798827 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f530c15e2a0 0x7f530c1f5440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:13.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.455+0000 7f530a800640 1 -- 172.21.15.6:0/977798827 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f530c1f4e70 con 0x7f530c0cf290 2024-03-04T09:26:13.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.455+0000 7f530a800640 1 --2- 172.21.15.6:0/977798827 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f530c0cf290 0x7f530c152d80 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f52fc00e760 tx=0x7f52fc00ec20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:13.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.455+0000 7f5303400640 1 -- 172.21.15.6:0/977798827 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52fc00f040 con 0x7f530c0cf290 2024-03-04T09:26:13.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.455+0000 7f5303400640 1 -- 172.21.15.6:0/977798827 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52fc004230 con 0x7f530c0cf290 2024-03-04T09:26:13.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.455+0000 7f5303400640 1 -- 172.21.15.6:0/977798827 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52fc00ee70 con 0x7f530c0cf290 2024-03-04T09:26:13.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.455+0000 7f5310a00640 1 -- 172.21.15.6:0/977798827 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f530c0ca050 con 0x7f530c0cf290 2024-03-04T09:26:13.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.456+0000 7f5310a00640 1 -- 172.21.15.6:0/977798827 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f530c0ca610 con 0x7f530c0cf290 2024-03-04T09:26:13.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.457+0000 7f5310a00640 1 -- 172.21.15.6:0/977798827 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f530c0ca7e0 con 0x7f530c0cf290 2024-03-04T09:26:13.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.458+0000 7f5303400640 1 -- 172.21.15.6:0/977798827 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f52fc00c6d0 con 0x7f530c0cf290 2024-03-04T09:26:13.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.458+0000 7f5303400640 1 --2- 172.21.15.6:0/977798827 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f52e806b4d0 0x7f52e806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:13.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.459+0000 7f5303400640 1 -- 172.21.15.6:0/977798827 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f52fc089f80 con 0x7f530c0cf290 2024-03-04T09:26:13.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.459+0000 7f5301e00640 1 --2- 172.21.15.6:0/977798827 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f52e806b4d0 0x7f52e806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:13.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.460+0000 7f5301e00640 1 --2- 172.21.15.6:0/977798827 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f52e806b4d0 0x7f52e806d990 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f52f4036000 tx=0x7f52f4008030 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:13.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.463+0000 7f5303400640 1 -- 172.21.15.6:0/977798827 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f530c0ca7e0 con 0x7f530c0cf290 2024-03-04T09:26:13.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.701+0000 7f5310a00640 1 -- 172.21.15.6:0/977798827 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f530c04f330 con 0x7f530c0cf290 2024-03-04T09:26:13.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.702+0000 7f5303400640 1 -- 172.21.15.6:0/977798827 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4971 (secure 0 0 0) 0x7f52fc017020 con 0x7f530c0cf290 2024-03-04T09:26:13.699 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:26:13.700 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4248,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/2702058134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2702058134},{"type":"v1","addr":"172.21.15.6:6833","nonce":2702058134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4263,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/3866998852","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":3866998852},{"type":"v1","addr":"172.21.15.6:6835","nonce":3866998852}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4285,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/3523455455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":3523455455},{"type":"v1","addr":"172.21.15.146:6837","nonce":3523455455}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:25:18.912808+0000","modified":"2024-03-04T09:25:22.016493+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4288},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4288":{"gid":4288,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.146:6836/1946537763","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":1946537763},{"type":"v1","addr":"172.21.15.146:6836","nonce":1946537763}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4288,"qdb_cluster":[4288]},"id":1}]} 2024-03-04T09:26:13.700 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 7 2024-03-04T09:26:13.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.705+0000 7f5310a00640 1 -- 172.21.15.6:0/977798827 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f52e806b4d0 msgr2=0x7f52e806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:13.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.705+0000 7f5310a00640 1 --2- 172.21.15.6:0/977798827 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f52e806b4d0 0x7f52e806d990 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f52f4036000 tx=0x7f52f4008030 comp rx=0 tx=0).stop 2024-03-04T09:26:13.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.705+0000 7f5310a00640 1 -- 172.21.15.6:0/977798827 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f530c0cf290 msgr2=0x7f530c152d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:13.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.705+0000 7f5310a00640 1 --2- 172.21.15.6:0/977798827 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f530c0cf290 0x7f530c152d80 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f52fc00e760 tx=0x7f52fc00ec20 comp rx=0 tx=0).stop 2024-03-04T09:26:13.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.705+0000 7f5310a00640 1 -- 172.21.15.6:0/977798827 shutdown_connections 2024-03-04T09:26:13.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.705+0000 7f5310a00640 1 --2- 172.21.15.6:0/977798827 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f530c15e2a0 0x7f530c1f5440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:13.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.705+0000 7f5310a00640 1 --2- 172.21.15.6:0/977798827 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f52e806b4d0 0x7f52e806d990 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:13.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.705+0000 7f5310a00640 1 --2- 172.21.15.6:0/977798827 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f530c15dec0 0x7f530c1f4930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:13.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.705+0000 7f5310a00640 1 --2- 172.21.15.6:0/977798827 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f530c0cf290 0x7f530c152d80 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:13.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.705+0000 7f5310a00640 1 -- 172.21.15.6:0/977798827 >> 172.21.15.6:0/977798827 conn(0x7f530c0d3d40 msgr2=0x7f530c06c570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:13.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.705+0000 7f5310a00640 1 -- 172.21.15.6:0/977798827 shutdown_connections 2024-03-04T09:26:13.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.705+0000 7f5310a00640 1 -- 172.21.15.6:0/977798827 wait complete. 2024-03-04T09:26:13.722 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T09:26:13.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.947+0000 7f4a46800640 1 Processor -- start 2024-03-04T09:26:13.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.947+0000 7f4a46800640 1 -- start start 2024-03-04T09:26:13.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.949+0000 7f4a46800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f4a4015c920 0x7f4a40070a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:13.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.949+0000 7f4a46800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4a400ce850 0x7f4a400cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:13.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.949+0000 7f4a46800640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a400cdad0 con 0x7f4a40070f80 2024-03-04T09:26:13.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.949+0000 7f4a46800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a40071500 con 0x7f4a4015c920 2024-03-04T09:26:13.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.949+0000 7f4a46800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a400cdd70 con 0x7f4a400ce850 2024-03-04T09:26:13.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.949+0000 7f4a45e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f4a40070f80 0x7f4a40165e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:60550/0 (socket says 172.21.15.6:60550) 2024-03-04T09:26:13.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.949+0000 7f4a45e00640 1 -- 172.21.15.6:0/2056283392 learned_addr learned my addr 172.21.15.6:0/2056283392 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:26:13.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.949+0000 7f4a45400640 1 --2- 172.21.15.6:0/2056283392 >> v2:172.21.15.6:3300/0 conn(0x7f4a4015c920 0x7f4a40070a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:13.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.950+0000 7f4a44a00640 1 --2- 172.21.15.6:0/2056283392 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4a400ce850 0x7f4a400cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:13.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.950+0000 7f4a3fe00640 1 -- 172.21.15.6:0/2056283392 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1213544086 0 0) 0x7f4a400cdad0 con 0x7f4a40070f80 2024-03-04T09:26:13.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.950+0000 7f4a3fe00640 1 -- 172.21.15.6:0/2056283392 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a28003520 con 0x7f4a40070f80 2024-03-04T09:26:13.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.950+0000 7f4a45400640 1 -- 172.21.15.6:0/2056283392 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4a400ce850 msgr2=0x7f4a400cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:13.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.950+0000 7f4a45400640 1 --2- 172.21.15.6:0/2056283392 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4a400ce850 0x7f4a400cc950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:13.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.950+0000 7f4a45400640 1 -- 172.21.15.6:0/2056283392 >> v1:172.21.15.6:6789/0 conn(0x7f4a40070f80 legacy=0x7f4a40165e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:13.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.950+0000 7f4a45400640 1 -- 172.21.15.6:0/2056283392 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a400ce050 con 0x7f4a4015c920 2024-03-04T09:26:13.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.951+0000 7f4a45400640 1 --2- 172.21.15.6:0/2056283392 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4a4015c920 0x7f4a40070a40 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f4a340097a0 tx=0x7f4a3402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5bd2acc75985fef server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:13.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.951+0000 7f4a3fe00640 1 -- 172.21.15.6:0/2056283392 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a34035020 con 0x7f4a4015c920 2024-03-04T09:26:13.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.952+0000 7f4a3fe00640 1 -- 172.21.15.6:0/2056283392 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a34031830 con 0x7f4a4015c920 2024-03-04T09:26:13.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.952+0000 7f4a3fe00640 1 -- 172.21.15.6:0/2056283392 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a34031c00 con 0x7f4a4015c920 2024-03-04T09:26:13.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.952+0000 7f4a46800640 1 -- 172.21.15.6:0/2056283392 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4a4015c920 msgr2=0x7f4a40070a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:13.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.952+0000 7f4a46800640 1 --2- 172.21.15.6:0/2056283392 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4a4015c920 0x7f4a40070a40 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f4a340097a0 tx=0x7f4a3402b8d0 comp rx=0 tx=0).stop 2024-03-04T09:26:13.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.952+0000 7f4a46800640 1 -- 172.21.15.6:0/2056283392 shutdown_connections 2024-03-04T09:26:13.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.952+0000 7f4a46800640 1 --2- 172.21.15.6:0/2056283392 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4a400ce850 0x7f4a400cc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:13.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.952+0000 7f4a46800640 1 --2- 172.21.15.6:0/2056283392 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4a4015c920 0x7f4a40070a40 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:13.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.952+0000 7f4a46800640 1 -- 172.21.15.6:0/2056283392 >> 172.21.15.6:0/2056283392 conn(0x7f4a4006b3d0 msgr2=0x7f4a4006b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:13.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.952+0000 7f4a46800640 1 -- 172.21.15.6:0/2056283392 shutdown_connections 2024-03-04T09:26:13.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.952+0000 7f4a46800640 1 -- 172.21.15.6:0/2056283392 wait complete. 2024-03-04T09:26:13.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.954+0000 7f4a46800640 1 Processor -- start 2024-03-04T09:26:13.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.954+0000 7f4a46800640 1 -- start start 2024-03-04T09:26:13.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.955+0000 7f4a46800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4a40070f80 0x7f4a40150530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:13.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.955+0000 7f4a46800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4a400ce850 0x7f4a40153fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:13.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.955+0000 7f4a46800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4a4015c920 0x7f4a40154500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:13.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.955+0000 7f4a46800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a400cce90 con 0x7f4a400ce850 2024-03-04T09:26:13.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.955+0000 7f4a46800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a40163790 con 0x7f4a40070f80 2024-03-04T09:26:13.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.955+0000 7f4a46800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a400cd010 con 0x7f4a4015c920 2024-03-04T09:26:13.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.955+0000 7f4a44a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4a400ce850 0x7f4a40153fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:13.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.955+0000 7f4a44a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4a400ce850 0x7f4a40153fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:59830/0 (socket says 172.21.15.6:59830) 2024-03-04T09:26:13.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.955+0000 7f4a44a00640 1 -- 172.21.15.6:0/3441076218 learned_addr learned my addr 172.21.15.6:0/3441076218 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:13.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.956+0000 7f4a44a00640 1 -- 172.21.15.6:0/3441076218 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4a4015c920 msgr2=0x7f4a40154500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:13.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.956+0000 7f4a44a00640 1 --2- 172.21.15.6:0/3441076218 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4a4015c920 0x7f4a40154500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:13.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.956+0000 7f4a44a00640 1 -- 172.21.15.6:0/3441076218 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4a40070f80 msgr2=0x7f4a40150530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:13.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.956+0000 7f4a44a00640 1 --2- 172.21.15.6:0/3441076218 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4a40070f80 0x7f4a40150530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:13.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.956+0000 7f4a44a00640 1 -- 172.21.15.6:0/3441076218 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a400b2610 con 0x7f4a400ce850 2024-03-04T09:26:13.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.956+0000 7f4a44a00640 1 --2- 172.21.15.6:0/3441076218 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4a400ce850 0x7f4a40153fc0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f4a2c00e9b0 tx=0x7f4a2c00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:13.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.956+0000 7f4a3d600640 1 -- 172.21.15.6:0/3441076218 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a2c017070 con 0x7f4a400ce850 2024-03-04T09:26:13.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.956+0000 7f4a3d600640 1 -- 172.21.15.6:0/3441076218 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a2c00c9a0 con 0x7f4a400ce850 2024-03-04T09:26:13.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.956+0000 7f4a3d600640 1 -- 172.21.15.6:0/3441076218 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a2c00cd90 con 0x7f4a400ce850 2024-03-04T09:26:13.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.956+0000 7f4a46800640 1 -- 172.21.15.6:0/3441076218 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a401f86f0 con 0x7f4a400ce850 2024-03-04T09:26:13.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.956+0000 7f4a46800640 1 -- 172.21.15.6:0/3441076218 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a401f8cb0 con 0x7f4a400ce850 2024-03-04T09:26:13.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.958+0000 7f4a46800640 1 -- 172.21.15.6:0/3441076218 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a4004f330 con 0x7f4a400ce850 2024-03-04T09:26:13.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.958+0000 7f4a3d600640 1 -- 172.21.15.6:0/3441076218 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4a2c01a070 con 0x7f4a400ce850 2024-03-04T09:26:13.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.960+0000 7f4a3d600640 1 --2- 172.21.15.6:0/3441076218 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f4a2006b4d0 0x7f4a2006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:13.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.960+0000 7f4a3d600640 1 -- 172.21.15.6:0/3441076218 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f4a2c08a3e0 con 0x7f4a400ce850 2024-03-04T09:26:13.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.960+0000 7f4a45400640 1 --2- 172.21.15.6:0/3441076218 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f4a2006b4d0 0x7f4a2006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:13.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.961+0000 7f4a45400640 1 --2- 172.21.15.6:0/3441076218 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f4a2006b4d0 0x7f4a2006d990 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f4a34036000 tx=0x7f4a34030000 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:13.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:13.964+0000 7f4a3d600640 1 -- 172.21.15.6:0/3441076218 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4a2c0590b0 con 0x7f4a400ce850 2024-03-04T09:26:14.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.156+0000 7f4a46800640 1 -- 172.21.15.6:0/3441076218 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4a4006e820 con 0x7f4a400ce850 2024-03-04T09:26:14.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.159+0000 7f4a3d600640 1 -- 172.21.15.6:0/3441076218 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16852 (secure 0 0 0) 0x7f4a2c0589b0 con 0x7f4a400ce850 2024-03-04T09:26:14.157 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:26:14.157 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":22,"fsid":"edce0578-5ae4-4402-b45a-552bdef84d7c","created":"2024-03-04T09:24:54.523780+0000","modified":"2024-03-04T09:26:01.727682+0000","last_up_change":"2024-03-04T09:25:07.089049+0000","last_in_change":"2024-03-04T09:24:57.982077+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T09:25:07.935198+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-04T09:25:17.118573+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-04T09:25:17.930536+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a96574f-81f1-4bc6-8463-757520b84a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6825","nonce":1364449401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6827","nonce":1364449401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6831","nonce":1364449401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6829","nonce":1364449401}]},"public_addr":"172.21.15.6:6825/1364449401","cluster_addr":"172.21.15.6:6827/1364449401","heartbeat_back_addr":"172.21.15.6:6831/1364449401","heartbeat_front_addr":"172.21.15.6:6829/1364449401","state":["exists","up"]},{"osd":1,"uuid":"7d79b9c1-6b87-476a-bc36-7db5ee64cd90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6809","nonce":2494103078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6811","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6813","nonce":2494103078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6821","nonce":2494103078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6817","nonce":2494103078}]},"public_addr":"172.21.15.6:6809/2494103078","cluster_addr":"172.21.15.6:6813/2494103078","heartbeat_back_addr":"172.21.15.6:6821/2494103078","heartbeat_front_addr":"172.21.15.6:6817/2494103078","state":["exists","up"]},{"osd":2,"uuid":"492331b9-4ee9-40b6-bacd-85d44cd6c45a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6812","nonce":1500066681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6816","nonce":1500066681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6824","nonce":1500066681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6820","nonce":1500066681}]},"public_addr":"172.21.15.6:6812/1500066681","cluster_addr":"172.21.15.6:6816/1500066681","heartbeat_back_addr":"172.21.15.6:6824/1500066681","heartbeat_front_addr":"172.21.15.6:6820/1500066681","state":["exists","up"]},{"osd":3,"uuid":"21bb2efe-9ba7-4bdc-bc31-180cd1102e21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6801","nonce":1558806123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6803","nonce":1558806123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6807","nonce":1558806123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6805","nonce":1558806123}]},"public_addr":"172.21.15.6:6801/1558806123","cluster_addr":"172.21.15.6:6803/1558806123","heartbeat_back_addr":"172.21.15.6:6807/1558806123","heartbeat_front_addr":"172.21.15.6:6805/1558806123","state":["exists","up"]},{"osd":4,"uuid":"6e2c4b71-19d9-4737-83d9-558705d8644a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6801","nonce":1526135780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6803","nonce":1526135780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6807","nonce":1526135780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6805","nonce":1526135780}]},"public_addr":"172.21.15.146:6801/1526135780","cluster_addr":"172.21.15.146:6803/1526135780","heartbeat_back_addr":"172.21.15.146:6807/1526135780","heartbeat_front_addr":"172.21.15.146:6805/1526135780","state":["exists","up"]},{"osd":5,"uuid":"fb69c525-f1be-4b7d-ab32-6c29b0d6fddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6809","nonce":460262507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6811","nonce":460262507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6815","nonce":460262507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6813","nonce":460262507}]},"public_addr":"172.21.15.146:6809/460262507","cluster_addr":"172.21.15.146:6811/460262507","heartbeat_back_addr":"172.21.15.146:6815/460262507","heartbeat_front_addr":"172.21.15.146:6813/460262507","state":["exists","up"]},{"osd":6,"uuid":"c32e7fed-370d-4401-8ce0-6a4335b0d168","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6823","nonce":1392897493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6827","nonce":1392897493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6831","nonce":1392897493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6829","nonce":1392897493}]},"public_addr":"172.21.15.146:6823/1392897493","cluster_addr":"172.21.15.146:6827/1392897493","heartbeat_back_addr":"172.21.15.146:6831/1392897493","heartbeat_front_addr":"172.21.15.146:6829/1392897493","state":["exists","up"]},{"osd":7,"uuid":"75a79b44-fa5f-426d-a7d3-71f21f78835c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6817","nonce":3268323156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6819","nonce":3268323156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6825","nonce":3268323156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6822","nonce":3268323156}]},"public_addr":"172.21.15.146:6817/3268323156","cluster_addr":"172.21.15.146:6819/3268323156","heartbeat_back_addr":"172.21.15.146:6825/3268323156","heartbeat_front_addr":"172.21.15.146:6822/3268323156","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.310437+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.295850+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.294674+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.526855+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.300023+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.595160+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.346046+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.558741+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.7","mappings":[{"from":3,"to":2}]},{"pgid":"3.f","mappings":[{"from":7,"to":2}]},{"pgid":"3.21","mappings":[{"from":5,"to":1}]},{"pgid":"3.22","mappings":[{"from":5,"to":2}]},{"pgid":"3.28","mappings":[{"from":3,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T09:26:14.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.162+0000 7f4a46800640 1 -- 172.21.15.6:0/3441076218 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f4a2006b4d0 msgr2=0x7f4a2006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:14.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.162+0000 7f4a46800640 1 --2- 172.21.15.6:0/3441076218 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f4a2006b4d0 0x7f4a2006d990 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f4a34036000 tx=0x7f4a34030000 comp rx=0 tx=0).stop 2024-03-04T09:26:14.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.162+0000 7f4a46800640 1 -- 172.21.15.6:0/3441076218 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4a400ce850 msgr2=0x7f4a40153fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:14.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.162+0000 7f4a46800640 1 --2- 172.21.15.6:0/3441076218 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4a400ce850 0x7f4a40153fc0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f4a2c00e9b0 tx=0x7f4a2c00ee70 comp rx=0 tx=0).stop 2024-03-04T09:26:14.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.162+0000 7f4a46800640 1 -- 172.21.15.6:0/3441076218 shutdown_connections 2024-03-04T09:26:14.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.162+0000 7f4a46800640 1 --2- 172.21.15.6:0/3441076218 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f4a2006b4d0 0x7f4a2006d990 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:14.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.162+0000 7f4a46800640 1 --2- 172.21.15.6:0/3441076218 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4a4015c920 0x7f4a40154500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:14.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.162+0000 7f4a46800640 1 --2- 172.21.15.6:0/3441076218 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4a400ce850 0x7f4a40153fc0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:14.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.162+0000 7f4a46800640 1 --2- 172.21.15.6:0/3441076218 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4a40070f80 0x7f4a40150530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:14.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.162+0000 7f4a46800640 1 -- 172.21.15.6:0/3441076218 >> 172.21.15.6:0/3441076218 conn(0x7f4a4006b3d0 msgr2=0x7f4a40161500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:14.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.162+0000 7f4a46800640 1 -- 172.21.15.6:0/3441076218 shutdown_connections 2024-03-04T09:26:14.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.162+0000 7f4a46800640 1 -- 172.21.15.6:0/3441076218 wait complete. 2024-03-04T09:26:14.178 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T09:25:07.935198+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-04T09:25:17.118573+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-04T09:25:17.930536+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-04T09:26:14.178 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-04T09:26:14.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.403+0000 7faf65000640 1 Processor -- start 2024-03-04T09:26:14.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.404+0000 7faf65000640 1 -- start start 2024-03-04T09:26:14.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.405+0000 7faf65000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7faf600d0d40 0x7faf600b3510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:14.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.405+0000 7faf65000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf600ceca0 con 0x7faf600d4860 2024-03-04T09:26:14.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.405+0000 7faf65000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf600cef80 con 0x7faf6015dee0 2024-03-04T09:26:14.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.405+0000 7faf65000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf600cf5d0 con 0x7faf600d0d40 2024-03-04T09:26:14.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.405+0000 7faf5ea00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7faf600d4860 0x7faf600d0630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:60566/0 (socket says 172.21.15.6:60566) 2024-03-04T09:26:14.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.405+0000 7faf5ea00640 1 -- 172.21.15.6:0/1205607086 learned_addr learned my addr 172.21.15.6:0/1205607086 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:26:14.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.406+0000 7faf5e000640 1 -- 172.21.15.6:0/1205607086 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2699211337 0 0) 0x7faf600ceca0 con 0x7faf600d4860 2024-03-04T09:26:14.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.406+0000 7faf5e000640 1 -- 172.21.15.6:0/1205607086 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf44003520 con 0x7faf600d4860 2024-03-04T09:26:14.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.406+0000 7faf5fe00640 1 --2- 172.21.15.6:0/1205607086 >> v2:172.21.15.146:3300/0 conn(0x7faf600d0d40 0x7faf600b3510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:14.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.406+0000 7faf5e000640 1 -- 172.21.15.6:0/1205607086 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3277061776 0 0) 0x7faf44003520 con 0x7faf600d4860 2024-03-04T09:26:14.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.406+0000 7faf5e000640 1 -- 172.21.15.6:0/1205607086 >> v2:172.21.15.146:3300/0 conn(0x7faf600d0d40 msgr2=0x7faf600b3510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:14.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.406+0000 7faf5e000640 1 --2- 172.21.15.6:0/1205607086 >> v2:172.21.15.146:3300/0 conn(0x7faf600d0d40 0x7faf600b3510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:14.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.406+0000 7faf5e000640 1 -- 172.21.15.6:0/1205607086 >> v1:172.21.15.146:6789/0 conn(0x7faf6015dee0 legacy=0x7faf600d18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:14.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.406+0000 7faf5e000640 1 -- 172.21.15.6:0/1205607086 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf600cf260 con 0x7faf600d4860 2024-03-04T09:26:14.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.406+0000 7faf5e000640 1 -- 172.21.15.6:0/1205607086 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7faf48001670 con 0x7faf600d4860 2024-03-04T09:26:14.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.406+0000 7faf5e000640 1 -- 172.21.15.6:0/1205607086 >> v1:172.21.15.6:6789/0 conn(0x7faf600d4860 legacy=0x7faf600d0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:14.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.406+0000 7faf5e000640 1 --2- 172.21.15.6:0/1205607086 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faf440039b0 0x7faf44003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:14.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.406+0000 7faf5e000640 1 --2- 172.21.15.6:0/1205607086 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faf44004800 0x7faf44004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:14.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.406+0000 7faf5e000640 1 --2- 172.21.15.6:0/1205607086 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faf440051c0 0x7faf4400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:14.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.406+0000 7faf5e000640 1 -- 172.21.15.6:0/1205607086 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf600cf5d0 con 0x7faf440039b0 2024-03-04T09:26:14.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.406+0000 7faf5e000640 1 -- 172.21.15.6:0/1205607086 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf4400c110 con 0x7faf44004800 2024-03-04T09:26:14.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.406+0000 7faf5e000640 1 -- 172.21.15.6:0/1205607086 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf4400c310 con 0x7faf440051c0 2024-03-04T09:26:14.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.407+0000 7faf5f400640 1 --2- 172.21.15.6:0/1205607086 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faf440039b0 0x7faf44003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:14.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.407+0000 7faf5ea00640 1 --2- 172.21.15.6:0/1205607086 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faf44004800 0x7faf44004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:14.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.407+0000 7faf5f400640 1 -- 172.21.15.6:0/1205607086 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faf440051c0 msgr2=0x7faf4400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:14.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.407+0000 7faf5f400640 1 --2- 172.21.15.6:0/1205607086 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faf440051c0 0x7faf4400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:14.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.407+0000 7faf5f400640 1 -- 172.21.15.6:0/1205607086 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faf44004800 msgr2=0x7faf44004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:14.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.407+0000 7faf5f400640 1 --2- 172.21.15.6:0/1205607086 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faf44004800 0x7faf44004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:14.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.407+0000 7faf5f400640 1 -- 172.21.15.6:0/1205607086 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf44003520 con 0x7faf440039b0 2024-03-04T09:26:14.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.407+0000 7faf5fe00640 1 -- 172.21.15.6:0/1205607086 reap_dead start 2024-03-04T09:26:14.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.408+0000 7faf5f400640 1 --2- 172.21.15.6:0/1205607086 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faf440039b0 0x7faf44003dd0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7faf50001e50 tx=0x7faf44005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c600a2320f31d40b server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:14.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.408+0000 7faf5e000640 1 -- 172.21.15.6:0/1205607086 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf50001a70 con 0x7faf440039b0 2024-03-04T09:26:14.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.408+0000 7faf5e000640 1 -- 172.21.15.6:0/1205607086 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf50003b70 con 0x7faf440039b0 2024-03-04T09:26:14.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.408+0000 7faf5e000640 1 -- 172.21.15.6:0/1205607086 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf500050d0 con 0x7faf440039b0 2024-03-04T09:26:14.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.409+0000 7faf65000640 1 -- 172.21.15.6:0/1205607086 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faf440039b0 msgr2=0x7faf44003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:14.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.409+0000 7faf65000640 1 --2- 172.21.15.6:0/1205607086 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faf440039b0 0x7faf44003dd0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7faf50001e50 tx=0x7faf44005830 comp rx=0 tx=0).stop 2024-03-04T09:26:14.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.409+0000 7faf65000640 1 -- 172.21.15.6:0/1205607086 shutdown_connections 2024-03-04T09:26:14.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.409+0000 7faf65000640 1 --2- 172.21.15.6:0/1205607086 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faf440039b0 0x7faf44003dd0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:14.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.409+0000 7faf65000640 1 -- 172.21.15.6:0/1205607086 >> 172.21.15.6:0/1205607086 conn(0x7faf600d3cd0 msgr2=0x7faf600d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:14.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.409+0000 7faf65000640 1 -- 172.21.15.6:0/1205607086 shutdown_connections 2024-03-04T09:26:14.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.409+0000 7faf65000640 1 -- 172.21.15.6:0/1205607086 wait complete. 2024-03-04T09:26:14.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.411+0000 7faf65000640 1 Processor -- start 2024-03-04T09:26:14.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.411+0000 7faf65000640 1 -- start start 2024-03-04T09:26:14.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.412+0000 7faf65000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faf440039b0 0x7faf60155060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:14.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.412+0000 7faf65000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faf60160fd0 0x7faf60152cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:14.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.412+0000 7faf65000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faf6015ac00 0x7faf6015b080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:14.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.412+0000 7faf65000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf600cfa60 con 0x7faf6015ac00 2024-03-04T09:26:14.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.412+0000 7faf65000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf6015e2c0 con 0x7faf60160fd0 2024-03-04T09:26:14.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.412+0000 7faf65000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf600cfbe0 con 0x7faf440039b0 2024-03-04T09:26:14.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.412+0000 7faf5fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faf6015ac00 0x7faf6015b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:14.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.412+0000 7faf5fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faf6015ac00 0x7faf6015b080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:59858/0 (socket says 172.21.15.6:59858) 2024-03-04T09:26:14.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.412+0000 7faf5fe00640 1 -- 172.21.15.6:0/266316320 learned_addr learned my addr 172.21.15.6:0/266316320 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:14.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.412+0000 7faf5f400640 1 --2- 172.21.15.6:0/266316320 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faf440039b0 0x7faf60155060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:14.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.412+0000 7faf5fe00640 1 -- 172.21.15.6:0/266316320 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faf440039b0 msgr2=0x7faf60155060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:14.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.412+0000 7faf5fe00640 1 --2- 172.21.15.6:0/266316320 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faf440039b0 0x7faf60155060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:14.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.412+0000 7faf5fe00640 1 -- 172.21.15.6:0/266316320 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faf60160fd0 msgr2=0x7faf60152cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:14.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.412+0000 7faf5fe00640 1 --2- 172.21.15.6:0/266316320 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faf60160fd0 0x7faf60152cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:14.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.412+0000 7faf5fe00640 1 -- 172.21.15.6:0/266316320 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf601597a0 con 0x7faf6015ac00 2024-03-04T09:26:14.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.413+0000 7faf5fe00640 1 --2- 172.21.15.6:0/266316320 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faf6015ac00 0x7faf6015b080 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7faf600cad50 tx=0x7faf5800d8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:14.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.413+0000 7faf57400640 1 -- 172.21.15.6:0/266316320 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf58004020 con 0x7faf6015ac00 2024-03-04T09:26:14.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.413+0000 7faf57400640 1 -- 172.21.15.6:0/266316320 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf580041c0 con 0x7faf6015ac00 2024-03-04T09:26:14.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.413+0000 7faf57400640 1 -- 172.21.15.6:0/266316320 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf580043f0 con 0x7faf6015ac00 2024-03-04T09:26:14.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.413+0000 7faf65000640 1 -- 172.21.15.6:0/266316320 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf60159a80 con 0x7faf6015ac00 2024-03-04T09:26:14.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.413+0000 7faf65000640 1 -- 172.21.15.6:0/266316320 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf6015a040 con 0x7faf6015ac00 2024-03-04T09:26:14.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.414+0000 7faf65000640 1 -- 172.21.15.6:0/266316320 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf6004f330 con 0x7faf6015ac00 2024-03-04T09:26:14.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.415+0000 7faf57400640 1 -- 172.21.15.6:0/266316320 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7faf5801a020 con 0x7faf6015ac00 2024-03-04T09:26:14.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.416+0000 7faf57400640 1 --2- 172.21.15.6:0/266316320 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7faf3c06b500 0x7faf3c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:14.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.416+0000 7faf57400640 1 -- 172.21.15.6:0/266316320 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7faf58089820 con 0x7faf6015ac00 2024-03-04T09:26:14.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.417+0000 7faf5f400640 1 --2- 172.21.15.6:0/266316320 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7faf3c06b500 0x7faf3c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:14.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.418+0000 7faf5f400640 1 --2- 172.21.15.6:0/266316320 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7faf3c06b500 0x7faf3c06d9c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7faf50034000 tx=0x7faf50009030 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:14.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.420+0000 7faf57400640 1 -- 172.21.15.6:0/266316320 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7faf58021020 con 0x7faf6015ac00 2024-03-04T09:26:14.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.614+0000 7faf65000640 1 -- 172.21.15.6:0/266316320 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7faf6015a210 con 0x7faf6015ac00 2024-03-04T09:26:14.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.615+0000 7faf57400640 1 -- 172.21.15.6:0/266316320 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7faf58058610 con 0x7faf6015ac00 2024-03-04T09:26:14.612 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-04T09:26:14.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.617+0000 7faf65000640 1 -- 172.21.15.6:0/266316320 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7faf3c06b500 msgr2=0x7faf3c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:14.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.617+0000 7faf65000640 1 --2- 172.21.15.6:0/266316320 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7faf3c06b500 0x7faf3c06d9c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7faf50034000 tx=0x7faf50009030 comp rx=0 tx=0).stop 2024-03-04T09:26:14.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.617+0000 7faf65000640 1 -- 172.21.15.6:0/266316320 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faf6015ac00 msgr2=0x7faf6015b080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:14.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.617+0000 7faf65000640 1 --2- 172.21.15.6:0/266316320 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faf6015ac00 0x7faf6015b080 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7faf600cad50 tx=0x7faf5800d8a0 comp rx=0 tx=0).stop 2024-03-04T09:26:14.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.618+0000 7faf65000640 1 -- 172.21.15.6:0/266316320 shutdown_connections 2024-03-04T09:26:14.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.618+0000 7faf65000640 1 --2- 172.21.15.6:0/266316320 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faf6015ac00 0x7faf6015b080 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:14.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.618+0000 7faf65000640 1 --2- 172.21.15.6:0/266316320 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faf60160fd0 0x7faf60152cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:14.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.618+0000 7faf65000640 1 --2- 172.21.15.6:0/266316320 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7faf3c06b500 0x7faf3c06d9c0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:14.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.618+0000 7faf65000640 1 --2- 172.21.15.6:0/266316320 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faf440039b0 0x7faf60155060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:14.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.618+0000 7faf65000640 1 -- 172.21.15.6:0/266316320 >> 172.21.15.6:0/266316320 conn(0x7faf600d3cd0 msgr2=0x7faf60073f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:14.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.618+0000 7faf65000640 1 -- 172.21.15.6:0/266316320 shutdown_connections 2024-03-04T09:26:14.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.618+0000 7faf65000640 1 -- 172.21.15.6:0/266316320 wait complete. 2024-03-04T09:26:14.633 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-04T09:26:14.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.859+0000 7f2280a00640 1 Processor -- start 2024-03-04T09:26:14.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.859+0000 7f2280a00640 1 -- start start 2024-03-04T09:26:14.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.861+0000 7f2280a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f227c15dee0 0x7f227c0d18a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:14.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.861+0000 7f2280a00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f227c0d4860 0x7f227c0d0630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:14.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.861+0000 7f2280a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f227c0ceca0 con 0x7f227c0cfa60 2024-03-04T09:26:14.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.861+0000 7f2280a00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f227c0cf5d0 con 0x7f227c0d4860 2024-03-04T09:26:14.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.861+0000 7f2280a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f227c0cef80 con 0x7f227c15dee0 2024-03-04T09:26:14.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.861+0000 7f227b200640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f227c0cfa60 0x7f227c0b3510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:60576/0 (socket says 172.21.15.6:60576) 2024-03-04T09:26:14.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.861+0000 7f227b200640 1 -- 172.21.15.6:0/2197999063 learned_addr learned my addr 172.21.15.6:0/2197999063 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:26:14.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.861+0000 7f227a800640 1 --2- 172.21.15.6:0/2197999063 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f227c15dee0 0x7f227c0d18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:14.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.861+0000 7f2279e00640 1 --2- 172.21.15.6:0/2197999063 >> v2:172.21.15.146:3300/0 conn(0x7f227c0d4860 0x7f227c0d0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:14.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.862+0000 7f2279400640 1 -- 172.21.15.6:0/2197999063 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3413836573 0 0) 0x7f227c0ceca0 con 0x7f227c0cfa60 2024-03-04T09:26:14.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.862+0000 7f2279400640 1 -- 172.21.15.6:0/2197999063 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2260003520 con 0x7f227c0cfa60 2024-03-04T09:26:14.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.862+0000 7f227a800640 1 -- 172.21.15.6:0/2197999063 >> v2:172.21.15.146:3300/0 conn(0x7f227c0d4860 msgr2=0x7f227c0d0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:14.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.862+0000 7f227a800640 1 --2- 172.21.15.6:0/2197999063 >> v2:172.21.15.146:3300/0 conn(0x7f227c0d4860 0x7f227c0d0630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:14.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.862+0000 7f2279400640 1 -- 172.21.15.6:0/2197999063 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1977247083 0 0) 0x7f2260003520 con 0x7f227c0cfa60 2024-03-04T09:26:14.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.862+0000 7f227a800640 1 -- 172.21.15.6:0/2197999063 >> v1:172.21.15.6:6789/0 conn(0x7f227c0cfa60 legacy=0x7f227c0b3510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:14.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.862+0000 7f227a800640 1 -- 172.21.15.6:0/2197999063 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f227c0cf260 con 0x7f227c15dee0 2024-03-04T09:26:14.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.862+0000 7f2279e00640 1 --2- 172.21.15.6:0/2197999063 >> v2:172.21.15.146:3300/0 conn(0x7f227c0d4860 0x7f227c0d0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T09:26:14.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.864+0000 7f227a800640 1 --2- 172.21.15.6:0/2197999063 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f227c15dee0 0x7f227c0d18a0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f226c0097a0 tx=0x7f226c02b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=728f33bce4e06963 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:14.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.864+0000 7f2279400640 1 -- 172.21.15.6:0/2197999063 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f226c037070 con 0x7f227c15dee0 2024-03-04T09:26:14.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.864+0000 7f2279400640 1 -- 172.21.15.6:0/2197999063 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f226c030510 con 0x7f227c15dee0 2024-03-04T09:26:14.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.864+0000 7f2279400640 1 -- 172.21.15.6:0/2197999063 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22740019f0 con 0x7f227c15dee0 2024-03-04T09:26:14.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.865+0000 7f2280a00640 1 -- 172.21.15.6:0/2197999063 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f227c15dee0 msgr2=0x7f227c0d18a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:14.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.865+0000 7f2280a00640 1 --2- 172.21.15.6:0/2197999063 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f227c15dee0 0x7f227c0d18a0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f226c0097a0 tx=0x7f226c02b790 comp rx=0 tx=0).stop 2024-03-04T09:26:14.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.865+0000 7f2280a00640 1 -- 172.21.15.6:0/2197999063 shutdown_connections 2024-03-04T09:26:14.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.865+0000 7f2280a00640 1 --2- 172.21.15.6:0/2197999063 >> v2:172.21.15.146:3300/0 conn(0x7f227c0d4860 0x7f227c0d0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:14.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.865+0000 7f2280a00640 1 --2- 172.21.15.6:0/2197999063 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f227c15dee0 0x7f227c0d18a0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:14.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.865+0000 7f2280a00640 1 -- 172.21.15.6:0/2197999063 >> 172.21.15.6:0/2197999063 conn(0x7f227c0d3cd0 msgr2=0x7f227c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:14.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.865+0000 7f2280a00640 1 -- 172.21.15.6:0/2197999063 shutdown_connections 2024-03-04T09:26:14.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.865+0000 7f2280a00640 1 -- 172.21.15.6:0/2197999063 wait complete. 2024-03-04T09:26:14.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.867+0000 7f2280a00640 1 Processor -- start 2024-03-04T09:26:14.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.867+0000 7f2280a00640 1 -- start start 2024-03-04T09:26:14.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.868+0000 7f2280a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f227c0cfa60 0x7f227c09d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:14.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.868+0000 7f2280a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f227c0d4860 0x7f227c0cdc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:14.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.868+0000 7f2280a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f227c15dee0 0x7f227c1530f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:14.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.868+0000 7f2280a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f227c0d1de0 con 0x7f227c15dee0 2024-03-04T09:26:14.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.868+0000 7f2280a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f227c15e2c0 con 0x7f227c0cfa60 2024-03-04T09:26:14.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.868+0000 7f2280a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f227c0d1f60 con 0x7f227c0d4860 2024-03-04T09:26:14.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.868+0000 7f227b200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f227c15dee0 0x7f227c1530f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:14.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.868+0000 7f227b200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f227c15dee0 0x7f227c1530f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:59860/0 (socket says 172.21.15.6:59860) 2024-03-04T09:26:14.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.868+0000 7f227b200640 1 -- 172.21.15.6:0/1918258855 learned_addr learned my addr 172.21.15.6:0/1918258855 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:14.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.868+0000 7f2279e00640 1 --2- 172.21.15.6:0/1918258855 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f227c0d4860 0x7f227c0cdc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:14.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.868+0000 7f227a800640 1 --2- 172.21.15.6:0/1918258855 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f227c0cfa60 0x7f227c09d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:14.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.869+0000 7f227b200640 1 -- 172.21.15.6:0/1918258855 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f227c0d4860 msgr2=0x7f227c0cdc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:14.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.869+0000 7f227b200640 1 --2- 172.21.15.6:0/1918258855 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f227c0d4860 0x7f227c0cdc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:14.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.869+0000 7f227b200640 1 -- 172.21.15.6:0/1918258855 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f227c0cfa60 msgr2=0x7f227c09d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:14.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.869+0000 7f227b200640 1 --2- 172.21.15.6:0/1918258855 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f227c0cfa60 0x7f227c09d9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:14.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.869+0000 7f227b200640 1 -- 172.21.15.6:0/1918258855 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f227c0cc070 con 0x7f227c15dee0 2024-03-04T09:26:14.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.869+0000 7f227a800640 1 --2- 172.21.15.6:0/1918258855 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f227c0cfa60 0x7f227c09d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:26:14.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.869+0000 7f227b200640 1 --2- 172.21.15.6:0/1918258855 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f227c15dee0 0x7f227c1530f0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f2274001890 tx=0x7f227400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:14.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.869+0000 7f2272a00640 1 -- 172.21.15.6:0/1918258855 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2274012400 con 0x7f227c15dee0 2024-03-04T09:26:14.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.869+0000 7f2272a00640 1 -- 172.21.15.6:0/1918258855 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22740125c0 con 0x7f227c15dee0 2024-03-04T09:26:14.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.869+0000 7f2272a00640 1 -- 172.21.15.6:0/1918258855 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22740129b0 con 0x7f227c15dee0 2024-03-04T09:26:14.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.869+0000 7f2280a00640 1 -- 172.21.15.6:0/1918258855 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f227c0cc350 con 0x7f227c15dee0 2024-03-04T09:26:14.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.869+0000 7f2280a00640 1 -- 172.21.15.6:0/1918258855 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f227c0cc910 con 0x7f227c15dee0 2024-03-04T09:26:14.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.870+0000 7f2280a00640 1 -- 172.21.15.6:0/1918258855 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f227c04f330 con 0x7f227c15dee0 2024-03-04T09:26:14.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.871+0000 7f2272a00640 1 -- 172.21.15.6:0/1918258855 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2274012b50 con 0x7f227c15dee0 2024-03-04T09:26:14.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.872+0000 7f2272a00640 1 --2- 172.21.15.6:0/1918258855 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f225c06b4d0 0x7f225c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:14.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.872+0000 7f2272a00640 1 -- 172.21.15.6:0/1918258855 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f227408aba0 con 0x7f227c15dee0 2024-03-04T09:26:14.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.873+0000 7f227a800640 1 --2- 172.21.15.6:0/1918258855 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f225c06b4d0 0x7f225c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:14.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.874+0000 7f227a800640 1 --2- 172.21.15.6:0/1918258855 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f225c06b4d0 0x7f225c06d990 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f226c034000 tx=0x7f226c02f000 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:14.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:14.878+0000 7f2272a00640 1 -- 172.21.15.6:0/1918258855 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f22740182b0 con 0x7f227c15dee0 2024-03-04T09:26:15.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.079+0000 7f2280a00640 1 -- 172.21.15.6:0/1918258855 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f227c08d7d0 con 0x7f227c15dee0 2024-03-04T09:26:15.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.080+0000 7f2272a00640 1 -- 172.21.15.6:0/1918258855 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7f2274059990 con 0x7f227c15dee0 2024-03-04T09:26:15.077 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-03-04T09:26:15.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.083+0000 7f2280a00640 1 -- 172.21.15.6:0/1918258855 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f225c06b4d0 msgr2=0x7f225c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:15.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.083+0000 7f2280a00640 1 --2- 172.21.15.6:0/1918258855 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f225c06b4d0 0x7f225c06d990 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f226c034000 tx=0x7f226c02f000 comp rx=0 tx=0).stop 2024-03-04T09:26:15.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.083+0000 7f2280a00640 1 -- 172.21.15.6:0/1918258855 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f227c15dee0 msgr2=0x7f227c1530f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:15.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.083+0000 7f2280a00640 1 --2- 172.21.15.6:0/1918258855 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f227c15dee0 0x7f227c1530f0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f2274001890 tx=0x7f227400a430 comp rx=0 tx=0).stop 2024-03-04T09:26:15.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.083+0000 7f2280a00640 1 -- 172.21.15.6:0/1918258855 shutdown_connections 2024-03-04T09:26:15.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.083+0000 7f2280a00640 1 --2- 172.21.15.6:0/1918258855 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f227c15dee0 0x7f227c1530f0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:15.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.083+0000 7f2280a00640 1 --2- 172.21.15.6:0/1918258855 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f225c06b4d0 0x7f225c06d990 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:15.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.083+0000 7f2280a00640 1 --2- 172.21.15.6:0/1918258855 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f227c0d4860 0x7f227c0cdc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:15.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.083+0000 7f2280a00640 1 --2- 172.21.15.6:0/1918258855 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f227c0cfa60 0x7f227c09d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:15.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.083+0000 7f2280a00640 1 -- 172.21.15.6:0/1918258855 >> 172.21.15.6:0/1918258855 conn(0x7f227c0d3cd0 msgr2=0x7f227c0cff60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:15.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.083+0000 7f2280a00640 1 -- 172.21.15.6:0/1918258855 shutdown_connections 2024-03-04T09:26:15.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.083+0000 7f2280a00640 1 -- 172.21.15.6:0/1918258855 wait complete. 2024-03-04T09:26:15.098 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-04T09:26:15.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.316+0000 7f536b000640 1 Processor -- start 2024-03-04T09:26:15.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.317+0000 7f536b000640 1 -- start start 2024-03-04T09:26:15.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.318+0000 7f536b000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f53640b3c20 0x7f53640729d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:15.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.318+0000 7f536b000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f53640b1b40 0x7f53640b1fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:15.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.318+0000 7f536b000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5364071930 con 0x7f53640b4000 2024-03-04T09:26:15.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.318+0000 7f536b000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53640b4490 con 0x7f53640b3c20 2024-03-04T09:26:15.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.318+0000 7f536b000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53640b24e0 con 0x7f53640b1b40 2024-03-04T09:26:15.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.319+0000 7f5369c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f53640b3c20 0x7f53640729d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:15.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.319+0000 7f5369c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f53640b3c20 0x7f53640729d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:59876/0 (socket says 172.21.15.6:59876) 2024-03-04T09:26:15.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.319+0000 7f5369c00640 1 -- 172.21.15.6:0/2135345331 learned_addr learned my addr 172.21.15.6:0/2135345331 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:15.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.319+0000 7f5369c00640 1 -- 172.21.15.6:0/2135345331 >> v2:172.21.15.146:3300/0 conn(0x7f53640b1b40 msgr2=0x7f53640b1fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:15.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.319+0000 7f5369c00640 1 --2- 172.21.15.6:0/2135345331 >> v2:172.21.15.146:3300/0 conn(0x7f53640b1b40 0x7f53640b1fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:15.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.319+0000 7f5369c00640 1 -- 172.21.15.6:0/2135345331 >> v1:172.21.15.146:6789/0 conn(0x7f53640b4000 legacy=0x7f536415b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:15.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.319+0000 7f5369c00640 1 -- 172.21.15.6:0/2135345331 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5364071c10 con 0x7f53640b3c20 2024-03-04T09:26:15.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.320+0000 7f5369c00640 1 --2- 172.21.15.6:0/2135345331 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f53640b3c20 0x7f53640729d0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f53640af660 tx=0x7f535802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2caa126270926916 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:15.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.321+0000 7f5363e00640 1 -- 172.21.15.6:0/2135345331 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5358035020 con 0x7f53640b3c20 2024-03-04T09:26:15.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.321+0000 7f5363e00640 1 -- 172.21.15.6:0/2135345331 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5358031870 con 0x7f53640b3c20 2024-03-04T09:26:15.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.321+0000 7f5363e00640 1 -- 172.21.15.6:0/2135345331 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5358031c40 con 0x7f53640b3c20 2024-03-04T09:26:15.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.321+0000 7f536b000640 1 -- 172.21.15.6:0/2135345331 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f53640b3c20 msgr2=0x7f53640729d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:15.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.321+0000 7f536b000640 1 --2- 172.21.15.6:0/2135345331 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f53640b3c20 0x7f53640729d0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f53640af660 tx=0x7f535802b910 comp rx=0 tx=0).stop 2024-03-04T09:26:15.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.321+0000 7f536b000640 1 -- 172.21.15.6:0/2135345331 shutdown_connections 2024-03-04T09:26:15.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.321+0000 7f536b000640 1 --2- 172.21.15.6:0/2135345331 >> v2:172.21.15.146:3300/0 conn(0x7f53640b1b40 0x7f53640b1fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:15.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.321+0000 7f536b000640 1 --2- 172.21.15.6:0/2135345331 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f53640b3c20 0x7f53640729d0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:15.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.321+0000 7f536b000640 1 -- 172.21.15.6:0/2135345331 >> 172.21.15.6:0/2135345331 conn(0x7f536406b3a0 msgr2=0x7f536406b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:15.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.321+0000 7f536b000640 1 -- 172.21.15.6:0/2135345331 shutdown_connections 2024-03-04T09:26:15.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.321+0000 7f536b000640 1 -- 172.21.15.6:0/2135345331 wait complete. 2024-03-04T09:26:15.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.323+0000 7f536b000640 1 Processor -- start 2024-03-04T09:26:15.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.323+0000 7f536b000640 1 -- start start 2024-03-04T09:26:15.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.324+0000 7f536b000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f53640b1b40 0x7f536406dac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:15.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.324+0000 7f536b000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f53640b3c20 0x7f536413fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:15.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.324+0000 7f536b000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f53640b4000 0x7f5364141800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:15.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.324+0000 7f536b000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53640721d0 con 0x7f53640b3c20 2024-03-04T09:26:15.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.324+0000 7f536b000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f536415bfb0 con 0x7f53640b1b40 2024-03-04T09:26:15.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.324+0000 7f536b000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53640724b0 con 0x7f53640b4000 2024-03-04T09:26:15.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.324+0000 7f5369200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f53640b3c20 0x7f536413fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:15.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.325+0000 7f5369200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f53640b3c20 0x7f536413fc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:59880/0 (socket says 172.21.15.6:59880) 2024-03-04T09:26:15.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.325+0000 7f5369200640 1 -- 172.21.15.6:0/2731302107 learned_addr learned my addr 172.21.15.6:0/2731302107 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:15.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.325+0000 7f536a600640 1 --2- 172.21.15.6:0/2731302107 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f53640b4000 0x7f5364141800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:15.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.325+0000 7f5369c00640 1 --2- 172.21.15.6:0/2731302107 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f53640b1b40 0x7f536406dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:15.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.325+0000 7f5369200640 1 -- 172.21.15.6:0/2731302107 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f53640b4000 msgr2=0x7f5364141800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:15.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.325+0000 7f5369200640 1 --2- 172.21.15.6:0/2731302107 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f53640b4000 0x7f5364141800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:15.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.325+0000 7f5369200640 1 -- 172.21.15.6:0/2731302107 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f53640b1b40 msgr2=0x7f536406dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:15.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.325+0000 7f5369200640 1 --2- 172.21.15.6:0/2731302107 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f53640b1b40 0x7f536406dac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:15.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.325+0000 7f5369200640 1 -- 172.21.15.6:0/2731302107 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f536413f4a0 con 0x7f53640b3c20 2024-03-04T09:26:15.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.325+0000 7f5369200640 1 --2- 172.21.15.6:0/2731302107 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f53640b3c20 0x7f536413fc60 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f535000c410 tx=0x7f535000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:15.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.326+0000 7f5361600640 1 -- 172.21.15.6:0/2731302107 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5350017070 con 0x7f53640b3c20 2024-03-04T09:26:15.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.326+0000 7f5361600640 1 -- 172.21.15.6:0/2731302107 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53500035f0 con 0x7f53640b3c20 2024-03-04T09:26:15.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.326+0000 7f5361600640 1 -- 172.21.15.6:0/2731302107 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5350003a20 con 0x7f53640b3c20 2024-03-04T09:26:15.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.326+0000 7f536b000640 1 -- 172.21.15.6:0/2731302107 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5364143370 con 0x7f53640b3c20 2024-03-04T09:26:15.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.326+0000 7f536b000640 1 -- 172.21.15.6:0/2731302107 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5364143930 con 0x7f53640b3c20 2024-03-04T09:26:15.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.327+0000 7f536b000640 1 -- 172.21.15.6:0/2731302107 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f536404f330 con 0x7f53640b3c20 2024-03-04T09:26:15.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.328+0000 7f5361600640 1 -- 172.21.15.6:0/2731302107 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f535001a070 con 0x7f53640b3c20 2024-03-04T09:26:15.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.329+0000 7f5361600640 1 --2- 172.21.15.6:0/2731302107 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f534406b500 0x7f534406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:15.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.329+0000 7f5361600640 1 -- 172.21.15.6:0/2731302107 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f5350089780 con 0x7f53640b3c20 2024-03-04T09:26:15.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.329+0000 7f5369c00640 1 --2- 172.21.15.6:0/2731302107 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f534406b500 0x7f534406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:15.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.331+0000 7f5369c00640 1 --2- 172.21.15.6:0/2731302107 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f534406b500 0x7f534406d9c0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f5358036000 tx=0x7f535803c040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:15.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.333+0000 7f5361600640 1 -- 172.21.15.6:0/2731302107 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f53500584d0 con 0x7f53640b3c20 2024-03-04T09:26:15.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.541+0000 7f536b000640 1 -- 172.21.15.6:0/2731302107 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f53640ce750 con 0x7f53640b3c20 2024-03-04T09:26:15.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.542+0000 7f5361600640 1 -- 172.21.15.6:0/2731302107 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7f53500010c0 con 0x7f53640b3c20 2024-03-04T09:26:15.539 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 64 2024-03-04T09:26:15.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.545+0000 7f536b000640 1 -- 172.21.15.6:0/2731302107 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f534406b500 msgr2=0x7f534406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:15.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.545+0000 7f536b000640 1 --2- 172.21.15.6:0/2731302107 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f534406b500 0x7f534406d9c0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f5358036000 tx=0x7f535803c040 comp rx=0 tx=0).stop 2024-03-04T09:26:15.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.545+0000 7f536b000640 1 -- 172.21.15.6:0/2731302107 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f53640b3c20 msgr2=0x7f536413fc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:15.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.545+0000 7f536b000640 1 --2- 172.21.15.6:0/2731302107 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f53640b3c20 0x7f536413fc60 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f535000c410 tx=0x7f535000c8d0 comp rx=0 tx=0).stop 2024-03-04T09:26:15.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.545+0000 7f536b000640 1 -- 172.21.15.6:0/2731302107 shutdown_connections 2024-03-04T09:26:15.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.545+0000 7f536b000640 1 --2- 172.21.15.6:0/2731302107 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f534406b500 0x7f534406d9c0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:15.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.545+0000 7f536b000640 1 --2- 172.21.15.6:0/2731302107 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f53640b4000 0x7f5364141800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:15.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.545+0000 7f536b000640 1 --2- 172.21.15.6:0/2731302107 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f53640b3c20 0x7f536413fc60 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:15.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.545+0000 7f536b000640 1 --2- 172.21.15.6:0/2731302107 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f53640b1b40 0x7f536406dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:15.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.545+0000 7f536b000640 1 -- 172.21.15.6:0/2731302107 >> 172.21.15.6:0/2731302107 conn(0x7f536406b3a0 msgr2=0x7f53640cec70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:15.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.545+0000 7f536b000640 1 -- 172.21.15.6:0/2731302107 shutdown_connections 2024-03-04T09:26:15.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.545+0000 7f536b000640 1 -- 172.21.15.6:0/2731302107 wait complete. 2024-03-04T09:26:15.561 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:26:15.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.788+0000 7f2d58c00640 1 Processor -- start 2024-03-04T09:26:15.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.789+0000 7f2d58c00640 1 -- start start 2024-03-04T09:26:15.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.790+0000 7f2d58c00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f2d540cd060 0x7f2d541667e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:15.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.790+0000 7f2d58c00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d540714d0 con 0x7f2d54070f10 2024-03-04T09:26:15.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.790+0000 7f2d58c00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d540717b0 con 0x7f2d54145300 2024-03-04T09:26:15.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.790+0000 7f2d58c00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d54071a90 con 0x7f2d540cd060 2024-03-04T09:26:15.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.791+0000 7f2d4be00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f2d54070f10 0x7f2d540cc950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:60584/0 (socket says 172.21.15.6:60584) 2024-03-04T09:26:15.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.791+0000 7f2d4be00640 1 -- 172.21.15.6:0/2283451056 learned_addr learned my addr 172.21.15.6:0/2283451056 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:26:15.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.791+0000 7f2d53e00640 1 --2- 172.21.15.6:0/2283451056 >> v2:172.21.15.146:3300/0 conn(0x7f2d540cd060 0x7f2d541667e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:15.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.792+0000 7f2d52a00640 1 -- 172.21.15.6:0/2283451056 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 318813242 0 0) 0x7f2d540714d0 con 0x7f2d54070f10 2024-03-04T09:26:15.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.792+0000 7f2d52a00640 1 -- 172.21.15.6:0/2283451056 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d38003520 con 0x7f2d54070f10 2024-03-04T09:26:15.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.792+0000 7f2d53e00640 1 -- 172.21.15.6:0/2283451056 >> v1:172.21.15.146:6789/0 conn(0x7f2d54145300 legacy=0x7f2d540b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:15.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.792+0000 7f2d53e00640 1 -- 172.21.15.6:0/2283451056 >> v1:172.21.15.6:6789/0 conn(0x7f2d54070f10 legacy=0x7f2d540cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:15.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.792+0000 7f2d53e00640 1 -- 172.21.15.6:0/2283451056 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d54071d70 con 0x7f2d540cd060 2024-03-04T09:26:15.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.794+0000 7f2d53e00640 1 --2- 172.21.15.6:0/2283451056 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2d540cd060 0x7f2d541667e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f2d4c009d90 tx=0x7f2d4c02beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a11d3ba507fb5f26 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:15.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.794+0000 7f2d52a00640 1 -- 172.21.15.6:0/2283451056 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d4c039070 con 0x7f2d540cd060 2024-03-04T09:26:15.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.794+0000 7f2d52a00640 1 -- 172.21.15.6:0/2283451056 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d4c02cc60 con 0x7f2d540cd060 2024-03-04T09:26:15.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.794+0000 7f2d52a00640 1 -- 172.21.15.6:0/2283451056 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d4c034640 con 0x7f2d540cd060 2024-03-04T09:26:15.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.795+0000 7f2d58c00640 1 -- 172.21.15.6:0/2283451056 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2d540cd060 msgr2=0x7f2d541667e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:15.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.795+0000 7f2d58c00640 1 --2- 172.21.15.6:0/2283451056 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2d540cd060 0x7f2d541667e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f2d4c009d90 tx=0x7f2d4c02beb0 comp rx=0 tx=0).stop 2024-03-04T09:26:15.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.795+0000 7f2d58c00640 1 -- 172.21.15.6:0/2283451056 shutdown_connections 2024-03-04T09:26:15.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.795+0000 7f2d58c00640 1 --2- 172.21.15.6:0/2283451056 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2d540cd060 0x7f2d541667e0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:15.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.795+0000 7f2d58c00640 1 -- 172.21.15.6:0/2283451056 >> 172.21.15.6:0/2283451056 conn(0x7f2d5406b3d0 msgr2=0x7f2d5406b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:15.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.795+0000 7f2d58c00640 1 -- 172.21.15.6:0/2283451056 shutdown_connections 2024-03-04T09:26:15.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.795+0000 7f2d58c00640 1 -- 172.21.15.6:0/2283451056 wait complete. 2024-03-04T09:26:15.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.797+0000 7f2d58c00640 1 Processor -- start 2024-03-04T09:26:15.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.797+0000 7f2d58c00640 1 -- start start 2024-03-04T09:26:15.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.798+0000 7f2d58c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2d54070f10 0x7f2d5413c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:15.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.798+0000 7f2d58c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2d540cd060 0x7f2d54150e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:15.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.798+0000 7f2d58c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2d54145300 0x7f2d5413d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:15.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.798+0000 7f2d58c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d5415d700 con 0x7f2d540cd060 2024-03-04T09:26:15.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.798+0000 7f2d58c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d5415d580 con 0x7f2d54070f10 2024-03-04T09:26:15.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.798+0000 7f2d58c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d5415d880 con 0x7f2d54145300 2024-03-04T09:26:15.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.798+0000 7f2d4be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2d540cd060 0x7f2d54150e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:15.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.798+0000 7f2d4be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2d540cd060 0x7f2d54150e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:59886/0 (socket says 172.21.15.6:59886) 2024-03-04T09:26:15.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.798+0000 7f2d4be00640 1 -- 172.21.15.6:0/3427881275 learned_addr learned my addr 172.21.15.6:0/3427881275 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:15.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.798+0000 7f2d53e00640 1 --2- 172.21.15.6:0/3427881275 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2d54145300 0x7f2d5413d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:15.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.798+0000 7f2d4be00640 1 -- 172.21.15.6:0/3427881275 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2d54145300 msgr2=0x7f2d5413d6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:15.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.798+0000 7f2d4be00640 1 --2- 172.21.15.6:0/3427881275 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2d54145300 0x7f2d5413d6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:15.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.799+0000 7f2d4be00640 1 -- 172.21.15.6:0/3427881275 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2d54070f10 msgr2=0x7f2d5413c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:15.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.799+0000 7f2d4be00640 1 --2- 172.21.15.6:0/3427881275 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2d54070f10 0x7f2d5413c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:15.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.799+0000 7f2d4be00640 1 -- 172.21.15.6:0/3427881275 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d541513d0 con 0x7f2d540cd060 2024-03-04T09:26:15.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.799+0000 7f2d4be00640 1 --2- 172.21.15.6:0/3427881275 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2d540cd060 0x7f2d54150e90 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f2d3c00c500 tx=0x7f2d3c00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:15.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.799+0000 7f2d4b400640 1 -- 172.21.15.6:0/3427881275 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d3c017070 con 0x7f2d540cd060 2024-03-04T09:26:15.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.799+0000 7f2d4b400640 1 -- 172.21.15.6:0/3427881275 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d3c005ce0 con 0x7f2d540cd060 2024-03-04T09:26:15.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.799+0000 7f2d58c00640 1 -- 172.21.15.6:0/3427881275 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d5406cb30 con 0x7f2d540cd060 2024-03-04T09:26:15.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.799+0000 7f2d4b400640 1 -- 172.21.15.6:0/3427881275 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d3c007330 con 0x7f2d540cd060 2024-03-04T09:26:15.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.799+0000 7f2d58c00640 1 -- 172.21.15.6:0/3427881275 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d5406d0f0 con 0x7f2d540cd060 2024-03-04T09:26:15.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.801+0000 7f2d58c00640 1 -- 172.21.15.6:0/3427881275 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d5404f330 con 0x7f2d540cd060 2024-03-04T09:26:15.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.802+0000 7f2d4b400640 1 -- 172.21.15.6:0/3427881275 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2d3c01a070 con 0x7f2d540cd060 2024-03-04T09:26:15.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.803+0000 7f2d4b400640 1 --2- 172.21.15.6:0/3427881275 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2d3006b4d0 0x7f2d3006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:15.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.803+0000 7f2d4b400640 1 -- 172.21.15.6:0/3427881275 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f2d3c089730 con 0x7f2d540cd060 2024-03-04T09:26:15.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.804+0000 7f2d53400640 1 --2- 172.21.15.6:0/3427881275 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2d3006b4d0 0x7f2d3006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:15.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.805+0000 7f2d53400640 1 --2- 172.21.15.6:0/3427881275 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2d3006b4d0 0x7f2d3006d990 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f2d44000f60 tx=0x7f2d440079d0 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:15.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:15.808+0000 7f2d4b400640 1 -- 172.21.15.6:0/3427881275 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2d3c0585d0 con 0x7f2d540cd060 2024-03-04T09:26:16.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.052+0000 7f2d58c00640 1 -- 172.21.15.6:0/3427881275 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2d541448d0 con 0x7f2d540cd060 2024-03-04T09:26:16.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.053+0000 7f2d4b400640 1 -- 172.21.15.6:0/3427881275 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4971 (secure 0 0 0) 0x7f2d3c057ed0 con 0x7f2d540cd060 2024-03-04T09:26:16.050 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:26:16.050 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4248,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/2702058134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2702058134},{"type":"v1","addr":"172.21.15.6:6833","nonce":2702058134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4263,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/3866998852","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":3866998852},{"type":"v1","addr":"172.21.15.6:6835","nonce":3866998852}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4285,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/3523455455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":3523455455},{"type":"v1","addr":"172.21.15.146:6837","nonce":3523455455}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:25:18.912808+0000","modified":"2024-03-04T09:25:22.016493+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4288},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4288":{"gid":4288,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.146:6836/1946537763","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":1946537763},{"type":"v1","addr":"172.21.15.146:6836","nonce":1946537763}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4288,"qdb_cluster":[4288]},"id":1}]} 2024-03-04T09:26:16.051 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 7 2024-03-04T09:26:16.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.056+0000 7f2d58c00640 1 -- 172.21.15.6:0/3427881275 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2d3006b4d0 msgr2=0x7f2d3006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:16.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.056+0000 7f2d58c00640 1 --2- 172.21.15.6:0/3427881275 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2d3006b4d0 0x7f2d3006d990 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f2d44000f60 tx=0x7f2d440079d0 comp rx=0 tx=0).stop 2024-03-04T09:26:16.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.056+0000 7f2d58c00640 1 -- 172.21.15.6:0/3427881275 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2d540cd060 msgr2=0x7f2d54150e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:16.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.056+0000 7f2d58c00640 1 --2- 172.21.15.6:0/3427881275 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2d540cd060 0x7f2d54150e90 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f2d3c00c500 tx=0x7f2d3c00c9c0 comp rx=0 tx=0).stop 2024-03-04T09:26:16.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.056+0000 7f2d58c00640 1 -- 172.21.15.6:0/3427881275 shutdown_connections 2024-03-04T09:26:16.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.056+0000 7f2d58c00640 1 --2- 172.21.15.6:0/3427881275 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2d3006b4d0 0x7f2d3006d990 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:16.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.056+0000 7f2d58c00640 1 --2- 172.21.15.6:0/3427881275 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2d54145300 0x7f2d5413d6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:16.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.056+0000 7f2d58c00640 1 --2- 172.21.15.6:0/3427881275 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2d540cd060 0x7f2d54150e90 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:16.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.056+0000 7f2d58c00640 1 --2- 172.21.15.6:0/3427881275 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2d54070f10 0x7f2d5413c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:16.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.056+0000 7f2d58c00640 1 -- 172.21.15.6:0/3427881275 >> 172.21.15.6:0/3427881275 conn(0x7f2d5406b3d0 msgr2=0x7f2d54072570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:16.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.056+0000 7f2d58c00640 1 -- 172.21.15.6:0/3427881275 shutdown_connections 2024-03-04T09:26:16.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.056+0000 7f2d58c00640 1 -- 172.21.15.6:0/3427881275 wait complete. 2024-03-04T09:26:16.070 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T09:26:16.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.302+0000 7f0d25c00640 1 Processor -- start 2024-03-04T09:26:16.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.302+0000 7f0d25c00640 1 -- start start 2024-03-04T09:26:16.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.303+0000 7f0d25c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0d200af1a0 0x7f0d2015c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:16.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.304+0000 7f0d25c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f0d200cdc20 0x7f0d200b2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:16.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.304+0000 7f0d25c00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d2015cc40 con 0x7f0d200af580 2024-03-04T09:26:16.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.304+0000 7f0d25c00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d2015cf20 con 0x7f0d200cdc20 2024-03-04T09:26:16.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.304+0000 7f0d25c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d2015d200 con 0x7f0d200af1a0 2024-03-04T09:26:16.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.304+0000 7f0d1e000640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f0d200af580 0x7f0d200b4c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:60596/0 (socket says 172.21.15.6:60596) 2024-03-04T09:26:16.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.304+0000 7f0d1e000640 1 -- 172.21.15.6:0/2275680062 learned_addr learned my addr 172.21.15.6:0/2275680062 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:26:16.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.304+0000 7f0d1f400640 1 --2- 172.21.15.6:0/2275680062 >> v2:172.21.15.6:3300/0 conn(0x7f0d200cdc20 0x7f0d200b2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:16.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.304+0000 7f0d1ea00640 1 --2- 172.21.15.6:0/2275680062 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0d200af1a0 0x7f0d2015c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:16.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.305+0000 7f0d1d600640 1 -- 172.21.15.6:0/2275680062 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3849481669 0 0) 0x7f0d2015cc40 con 0x7f0d200af580 2024-03-04T09:26:16.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.305+0000 7f0d1f400640 1 -- 172.21.15.6:0/2275680062 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0d200af1a0 msgr2=0x7f0d2015c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:16.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.305+0000 7f0d1f400640 1 --2- 172.21.15.6:0/2275680062 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0d200af1a0 0x7f0d2015c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:16.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.305+0000 7f0d1f400640 1 -- 172.21.15.6:0/2275680062 >> v1:172.21.15.6:6789/0 conn(0x7f0d200af580 legacy=0x7f0d200b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:16.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.305+0000 7f0d1f400640 1 -- 172.21.15.6:0/2275680062 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d2015d4e0 con 0x7f0d200cdc20 2024-03-04T09:26:16.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.306+0000 7f0d1f400640 1 --2- 172.21.15.6:0/2275680062 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0d200cdc20 0x7f0d200b2dd0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f0d180097a0 tx=0x7f0d1802b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a020ffb92c7b41f server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:16.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.306+0000 7f0d1d600640 1 -- 172.21.15.6:0/2275680062 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d18031780 con 0x7f0d200cdc20 2024-03-04T09:26:16.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.306+0000 7f0d1d600640 1 -- 172.21.15.6:0/2275680062 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d18037070 con 0x7f0d200cdc20 2024-03-04T09:26:16.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.306+0000 7f0d1d600640 1 -- 172.21.15.6:0/2275680062 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d18031b40 con 0x7f0d200cdc20 2024-03-04T09:26:16.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.307+0000 7f0d25c00640 1 -- 172.21.15.6:0/2275680062 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0d200cdc20 msgr2=0x7f0d200b2dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:16.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.307+0000 7f0d25c00640 1 --2- 172.21.15.6:0/2275680062 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0d200cdc20 0x7f0d200b2dd0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f0d180097a0 tx=0x7f0d1802b820 comp rx=0 tx=0).stop 2024-03-04T09:26:16.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.307+0000 7f0d25c00640 1 -- 172.21.15.6:0/2275680062 shutdown_connections 2024-03-04T09:26:16.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.307+0000 7f0d25c00640 1 --2- 172.21.15.6:0/2275680062 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0d200cdc20 0x7f0d200b2dd0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:16.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.307+0000 7f0d25c00640 1 --2- 172.21.15.6:0/2275680062 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0d200af1a0 0x7f0d2015c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:16.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.307+0000 7f0d25c00640 1 -- 172.21.15.6:0/2275680062 >> 172.21.15.6:0/2275680062 conn(0x7f0d2006afb0 msgr2=0x7f0d2006b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:16.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.307+0000 7f0d25c00640 1 -- 172.21.15.6:0/2275680062 shutdown_connections 2024-03-04T09:26:16.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.307+0000 7f0d25c00640 1 -- 172.21.15.6:0/2275680062 wait complete. 2024-03-04T09:26:16.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.309+0000 7f0d25c00640 1 Processor -- start 2024-03-04T09:26:16.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.309+0000 7f0d25c00640 1 -- start start 2024-03-04T09:26:16.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.310+0000 7f0d25c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0d200af1a0 0x7f0d2015e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:16.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.310+0000 7f0d25c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0d200af580 0x7f0d20148230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:16.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.310+0000 7f0d25c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0d200cdc20 0x7f0d200cad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:16.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.310+0000 7f0d25c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d200b36a0 con 0x7f0d200cdc20 2024-03-04T09:26:16.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.310+0000 7f0d25c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d200b3520 con 0x7f0d200af580 2024-03-04T09:26:16.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.310+0000 7f0d25c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d200b3820 con 0x7f0d200af1a0 2024-03-04T09:26:16.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.310+0000 7f0d1f400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0d200cdc20 0x7f0d200cad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:16.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.310+0000 7f0d1f400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0d200cdc20 0x7f0d200cad30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:59914/0 (socket says 172.21.15.6:59914) 2024-03-04T09:26:16.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.310+0000 7f0d1f400640 1 -- 172.21.15.6:0/415553601 learned_addr learned my addr 172.21.15.6:0/415553601 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:16.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.310+0000 7f0d1ea00640 1 --2- 172.21.15.6:0/415553601 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0d200af1a0 0x7f0d2015e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:16.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.310+0000 7f0d1f400640 1 -- 172.21.15.6:0/415553601 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0d200af1a0 msgr2=0x7f0d2015e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:16.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.310+0000 7f0d1f400640 1 --2- 172.21.15.6:0/415553601 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0d200af1a0 0x7f0d2015e9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:16.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.310+0000 7f0d1e000640 1 --2- 172.21.15.6:0/415553601 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0d200af580 0x7f0d20148230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:16.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.310+0000 7f0d1f400640 1 -- 172.21.15.6:0/415553601 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0d200af580 msgr2=0x7f0d20148230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:16.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.311+0000 7f0d1f400640 1 --2- 172.21.15.6:0/415553601 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0d200af580 0x7f0d20148230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:16.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.311+0000 7f0d1f400640 1 -- 172.21.15.6:0/415553601 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d200cb270 con 0x7f0d200cdc20 2024-03-04T09:26:16.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.311+0000 7f0d1f400640 1 --2- 172.21.15.6:0/415553601 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0d200cdc20 0x7f0d200cad30 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f0d180097a0 tx=0x7f0d180046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:16.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.311+0000 7f0d16a00640 1 -- 172.21.15.6:0/415553601 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d18004230 con 0x7f0d200cdc20 2024-03-04T09:26:16.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.311+0000 7f0d16a00640 1 -- 172.21.15.6:0/415553601 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d180043f0 con 0x7f0d200cdc20 2024-03-04T09:26:16.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.311+0000 7f0d16a00640 1 -- 172.21.15.6:0/415553601 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d18003850 con 0x7f0d200cdc20 2024-03-04T09:26:16.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.311+0000 7f0d25c00640 1 -- 172.21.15.6:0/415553601 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d200cb510 con 0x7f0d200cdc20 2024-03-04T09:26:16.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.311+0000 7f0d25c00640 1 -- 172.21.15.6:0/415553601 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d200cbad0 con 0x7f0d200cdc20 2024-03-04T09:26:16.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.313+0000 7f0d25c00640 1 -- 172.21.15.6:0/415553601 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d200cbd10 con 0x7f0d200cdc20 2024-03-04T09:26:16.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.314+0000 7f0d16a00640 1 -- 172.21.15.6:0/415553601 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0d180039f0 con 0x7f0d200cdc20 2024-03-04T09:26:16.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.315+0000 7f0d16a00640 1 --2- 172.21.15.6:0/415553601 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f0cfc06b4d0 0x7f0cfc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:16.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.315+0000 7f0d16a00640 1 -- 172.21.15.6:0/415553601 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f0d180ac0e0 con 0x7f0d200cdc20 2024-03-04T09:26:16.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.315+0000 7f0d1ea00640 1 --2- 172.21.15.6:0/415553601 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f0cfc06b4d0 0x7f0cfc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:16.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.317+0000 7f0d1ea00640 1 --2- 172.21.15.6:0/415553601 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f0cfc06b4d0 0x7f0cfc06d990 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f0d10009f30 tx=0x7f0d1000b040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:16.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.319+0000 7f0d16a00640 1 -- 172.21.15.6:0/415553601 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0d200cbd10 con 0x7f0d200cdc20 2024-03-04T09:26:16.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.519+0000 7f0d25c00640 1 -- 172.21.15.6:0/415553601 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0d200ae760 con 0x7f0d200cdc20 2024-03-04T09:26:16.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.521+0000 7f0d16a00640 1 -- 172.21.15.6:0/415553601 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16852 (secure 0 0 0) 0x7f0d18041070 con 0x7f0d200cdc20 2024-03-04T09:26:16.519 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:26:16.519 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":22,"fsid":"edce0578-5ae4-4402-b45a-552bdef84d7c","created":"2024-03-04T09:24:54.523780+0000","modified":"2024-03-04T09:26:01.727682+0000","last_up_change":"2024-03-04T09:25:07.089049+0000","last_in_change":"2024-03-04T09:24:57.982077+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T09:25:07.935198+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-04T09:25:17.118573+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-04T09:25:17.930536+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a96574f-81f1-4bc6-8463-757520b84a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6825","nonce":1364449401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6827","nonce":1364449401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6831","nonce":1364449401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6829","nonce":1364449401}]},"public_addr":"172.21.15.6:6825/1364449401","cluster_addr":"172.21.15.6:6827/1364449401","heartbeat_back_addr":"172.21.15.6:6831/1364449401","heartbeat_front_addr":"172.21.15.6:6829/1364449401","state":["exists","up"]},{"osd":1,"uuid":"7d79b9c1-6b87-476a-bc36-7db5ee64cd90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6809","nonce":2494103078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6811","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6813","nonce":2494103078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6821","nonce":2494103078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6817","nonce":2494103078}]},"public_addr":"172.21.15.6:6809/2494103078","cluster_addr":"172.21.15.6:6813/2494103078","heartbeat_back_addr":"172.21.15.6:6821/2494103078","heartbeat_front_addr":"172.21.15.6:6817/2494103078","state":["exists","up"]},{"osd":2,"uuid":"492331b9-4ee9-40b6-bacd-85d44cd6c45a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6812","nonce":1500066681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6816","nonce":1500066681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6824","nonce":1500066681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6820","nonce":1500066681}]},"public_addr":"172.21.15.6:6812/1500066681","cluster_addr":"172.21.15.6:6816/1500066681","heartbeat_back_addr":"172.21.15.6:6824/1500066681","heartbeat_front_addr":"172.21.15.6:6820/1500066681","state":["exists","up"]},{"osd":3,"uuid":"21bb2efe-9ba7-4bdc-bc31-180cd1102e21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6801","nonce":1558806123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6803","nonce":1558806123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6807","nonce":1558806123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6805","nonce":1558806123}]},"public_addr":"172.21.15.6:6801/1558806123","cluster_addr":"172.21.15.6:6803/1558806123","heartbeat_back_addr":"172.21.15.6:6807/1558806123","heartbeat_front_addr":"172.21.15.6:6805/1558806123","state":["exists","up"]},{"osd":4,"uuid":"6e2c4b71-19d9-4737-83d9-558705d8644a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6801","nonce":1526135780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6803","nonce":1526135780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6807","nonce":1526135780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6805","nonce":1526135780}]},"public_addr":"172.21.15.146:6801/1526135780","cluster_addr":"172.21.15.146:6803/1526135780","heartbeat_back_addr":"172.21.15.146:6807/1526135780","heartbeat_front_addr":"172.21.15.146:6805/1526135780","state":["exists","up"]},{"osd":5,"uuid":"fb69c525-f1be-4b7d-ab32-6c29b0d6fddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6809","nonce":460262507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6811","nonce":460262507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6815","nonce":460262507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6813","nonce":460262507}]},"public_addr":"172.21.15.146:6809/460262507","cluster_addr":"172.21.15.146:6811/460262507","heartbeat_back_addr":"172.21.15.146:6815/460262507","heartbeat_front_addr":"172.21.15.146:6813/460262507","state":["exists","up"]},{"osd":6,"uuid":"c32e7fed-370d-4401-8ce0-6a4335b0d168","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6823","nonce":1392897493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6827","nonce":1392897493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6831","nonce":1392897493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6829","nonce":1392897493}]},"public_addr":"172.21.15.146:6823/1392897493","cluster_addr":"172.21.15.146:6827/1392897493","heartbeat_back_addr":"172.21.15.146:6831/1392897493","heartbeat_front_addr":"172.21.15.146:6829/1392897493","state":["exists","up"]},{"osd":7,"uuid":"75a79b44-fa5f-426d-a7d3-71f21f78835c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6817","nonce":3268323156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6819","nonce":3268323156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6825","nonce":3268323156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6822","nonce":3268323156}]},"public_addr":"172.21.15.146:6817/3268323156","cluster_addr":"172.21.15.146:6819/3268323156","heartbeat_back_addr":"172.21.15.146:6825/3268323156","heartbeat_front_addr":"172.21.15.146:6822/3268323156","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.310437+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.295850+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.294674+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.526855+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.300023+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.595160+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.346046+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.558741+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.7","mappings":[{"from":3,"to":2}]},{"pgid":"3.f","mappings":[{"from":7,"to":2}]},{"pgid":"3.21","mappings":[{"from":5,"to":1}]},{"pgid":"3.22","mappings":[{"from":5,"to":2}]},{"pgid":"3.28","mappings":[{"from":3,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T09:26:16.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.524+0000 7f0d25c00640 1 -- 172.21.15.6:0/415553601 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f0cfc06b4d0 msgr2=0x7f0cfc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:16.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.524+0000 7f0d25c00640 1 --2- 172.21.15.6:0/415553601 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f0cfc06b4d0 0x7f0cfc06d990 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f0d10009f30 tx=0x7f0d1000b040 comp rx=0 tx=0).stop 2024-03-04T09:26:16.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.524+0000 7f0d25c00640 1 -- 172.21.15.6:0/415553601 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0d200cdc20 msgr2=0x7f0d200cad30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:16.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.524+0000 7f0d25c00640 1 --2- 172.21.15.6:0/415553601 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0d200cdc20 0x7f0d200cad30 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f0d180097a0 tx=0x7f0d180046c0 comp rx=0 tx=0).stop 2024-03-04T09:26:16.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.524+0000 7f0d25c00640 1 -- 172.21.15.6:0/415553601 shutdown_connections 2024-03-04T09:26:16.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.524+0000 7f0d25c00640 1 --2- 172.21.15.6:0/415553601 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0d200cdc20 0x7f0d200cad30 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:16.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.524+0000 7f0d25c00640 1 --2- 172.21.15.6:0/415553601 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0d200af580 0x7f0d20148230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:16.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.525+0000 7f0d25c00640 1 --2- 172.21.15.6:0/415553601 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f0cfc06b4d0 0x7f0cfc06d990 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:16.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.525+0000 7f0d25c00640 1 --2- 172.21.15.6:0/415553601 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0d200af1a0 0x7f0d2015e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:16.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.525+0000 7f0d25c00640 1 -- 172.21.15.6:0/415553601 >> 172.21.15.6:0/415553601 conn(0x7f0d2006afb0 msgr2=0x7f0d2015bb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:16.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.525+0000 7f0d25c00640 1 -- 172.21.15.6:0/415553601 shutdown_connections 2024-03-04T09:26:16.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.525+0000 7f0d25c00640 1 -- 172.21.15.6:0/415553601 wait complete. 2024-03-04T09:26:16.542 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-04T09:26:16.543 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:26:16.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.762+0000 7fd2de000640 1 Processor -- start 2024-03-04T09:26:16.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.762+0000 7fd2de000640 1 -- start start 2024-03-04T09:26:16.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.763+0000 7fd2de000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd2d815e070 0x7fd2d80d2960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:16.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.764+0000 7fd2de000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2d80d18c0 con 0x7fd2d80d0cd0 2024-03-04T09:26:16.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.764+0000 7fd2de000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2d80d1ba0 con 0x7fd2d80cf410 2024-03-04T09:26:16.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.764+0000 7fd2de000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2d80d1e80 con 0x7fd2d815e070 2024-03-04T09:26:16.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.764+0000 7fd2dd600640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fd2d80d0cd0 0x7fd2d80b3630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:60606/0 (socket says 172.21.15.6:60606) 2024-03-04T09:26:16.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.764+0000 7fd2dd600640 1 -- 172.21.15.6:0/2028488952 learned_addr learned my addr 172.21.15.6:0/2028488952 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:26:16.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.764+0000 7fd2dcc00640 1 --2- 172.21.15.6:0/2028488952 >> v2:172.21.15.146:3300/0 conn(0x7fd2d815e070 0x7fd2d80d2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:16.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.764+0000 7fd2d7400640 1 -- 172.21.15.6:0/2028488952 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4009667024 0 0) 0x7fd2d80d18c0 con 0x7fd2d80d0cd0 2024-03-04T09:26:16.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.764+0000 7fd2d7400640 1 -- 172.21.15.6:0/2028488952 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2c0003520 con 0x7fd2d80d0cd0 2024-03-04T09:26:16.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.765+0000 7fd2d7400640 1 -- 172.21.15.6:0/2028488952 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3713169593 0 0) 0x7fd2c0003520 con 0x7fd2d80d0cd0 2024-03-04T09:26:16.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.765+0000 7fd2d7400640 1 -- 172.21.15.6:0/2028488952 >> v2:172.21.15.146:3300/0 conn(0x7fd2d815e070 msgr2=0x7fd2d80d2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:16.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.765+0000 7fd2d7400640 1 --2- 172.21.15.6:0/2028488952 >> v2:172.21.15.146:3300/0 conn(0x7fd2d815e070 0x7fd2d80d2960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:16.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.765+0000 7fd2d7400640 1 -- 172.21.15.6:0/2028488952 >> v1:172.21.15.146:6789/0 conn(0x7fd2d80cf410 legacy=0x7fd2d80d05c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:16.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.765+0000 7fd2d7400640 1 -- 172.21.15.6:0/2028488952 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2d80d2160 con 0x7fd2d80d0cd0 2024-03-04T09:26:16.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.765+0000 7fd2d7400640 1 -- 172.21.15.6:0/2028488952 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7fd2d00019f0 con 0x7fd2d80d0cd0 2024-03-04T09:26:16.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.765+0000 7fd2d7400640 1 -- 172.21.15.6:0/2028488952 >> v1:172.21.15.6:6789/0 conn(0x7fd2d80d0cd0 legacy=0x7fd2d80b3630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:16.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.765+0000 7fd2d7400640 1 --2- 172.21.15.6:0/2028488952 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd2c0003a00 0x7fd2c0003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:16.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.765+0000 7fd2d7400640 1 --2- 172.21.15.6:0/2028488952 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd2c0004800 0x7fd2c0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:16.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.765+0000 7fd2d7400640 1 --2- 172.21.15.6:0/2028488952 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd2c00051c0 0x7fd2c000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:16.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.765+0000 7fd2d7400640 1 -- 172.21.15.6:0/2028488952 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2d80d1e80 con 0x7fd2c0003a00 2024-03-04T09:26:16.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.765+0000 7fd2d7400640 1 -- 172.21.15.6:0/2028488952 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2c000c0c0 con 0x7fd2c0004800 2024-03-04T09:26:16.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.765+0000 7fd2d7400640 1 -- 172.21.15.6:0/2028488952 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2c000c2c0 con 0x7fd2c00051c0 2024-03-04T09:26:16.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.765+0000 7fd2dcc00640 1 --2- 172.21.15.6:0/2028488952 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd2c0003a00 0x7fd2c0003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:16.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.766+0000 7fd2dcc00640 1 -- 172.21.15.6:0/2028488952 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd2c00051c0 msgr2=0x7fd2c000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:16.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.766+0000 7fd2dd600640 1 --2- 172.21.15.6:0/2028488952 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd2c00051c0 0x7fd2c000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:16.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.766+0000 7fd2dcc00640 1 --2- 172.21.15.6:0/2028488952 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd2c00051c0 0x7fd2c000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:16.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.766+0000 7fd2d7e00640 1 --2- 172.21.15.6:0/2028488952 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd2c0004800 0x7fd2c0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:16.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.766+0000 7fd2dcc00640 1 -- 172.21.15.6:0/2028488952 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd2c0004800 msgr2=0x7fd2c0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:16.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.766+0000 7fd2dcc00640 1 --2- 172.21.15.6:0/2028488952 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd2c0004800 0x7fd2c0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:16.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.766+0000 7fd2dcc00640 1 -- 172.21.15.6:0/2028488952 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2c0003520 con 0x7fd2c0003a00 2024-03-04T09:26:16.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.766+0000 7fd2d7e00640 1 --2- 172.21.15.6:0/2028488952 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd2c0004800 0x7fd2c0004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:26:16.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.766+0000 7fd2dd600640 1 --2- 172.21.15.6:0/2028488952 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd2c00051c0 0x7fd2c000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:26:16.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.766+0000 7fd2dd600640 1 -- 172.21.15.6:0/2028488952 reap_dead start 2024-03-04T09:26:16.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.768+0000 7fd2dcc00640 1 --2- 172.21.15.6:0/2028488952 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd2c0003a00 0x7fd2c0003e20 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fd2cc009ac0 tx=0x7fd2c0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94215ef0e53d2827 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:16.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.768+0000 7fd2d7400640 1 -- 172.21.15.6:0/2028488952 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2cc039070 con 0x7fd2c0003a00 2024-03-04T09:26:16.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.768+0000 7fd2d7400640 1 -- 172.21.15.6:0/2028488952 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2cc02c6b0 con 0x7fd2c0003a00 2024-03-04T09:26:16.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.768+0000 7fd2d7400640 1 -- 172.21.15.6:0/2028488952 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2cc009690 con 0x7fd2c0003a00 2024-03-04T09:26:16.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.768+0000 7fd2de000640 1 -- 172.21.15.6:0/2028488952 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd2c0003a00 msgr2=0x7fd2c0003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:16.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.768+0000 7fd2de000640 1 --2- 172.21.15.6:0/2028488952 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd2c0003a00 0x7fd2c0003e20 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fd2cc009ac0 tx=0x7fd2c0005830 comp rx=0 tx=0).stop 2024-03-04T09:26:16.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.769+0000 7fd2de000640 1 -- 172.21.15.6:0/2028488952 shutdown_connections 2024-03-04T09:26:16.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.769+0000 7fd2de000640 1 --2- 172.21.15.6:0/2028488952 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd2c0003a00 0x7fd2c0003e20 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:16.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.769+0000 7fd2de000640 1 -- 172.21.15.6:0/2028488952 >> 172.21.15.6:0/2028488952 conn(0x7fd2d808d2a0 msgr2=0x7fd2d80d3cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:16.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.769+0000 7fd2de000640 1 -- 172.21.15.6:0/2028488952 shutdown_connections 2024-03-04T09:26:16.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.769+0000 7fd2de000640 1 -- 172.21.15.6:0/2028488952 wait complete. 2024-03-04T09:26:16.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.771+0000 7fd2de000640 1 Processor -- start 2024-03-04T09:26:16.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.771+0000 7fd2de000640 1 -- start start 2024-03-04T09:26:16.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.772+0000 7fd2de000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd2d80cdbf0 0x7fd2d8160f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:16.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.772+0000 7fd2de000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd2d80b7320 0x7fd2d80cbc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:16.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.772+0000 7fd2de000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd2d81f77a0 0x7fd2d81f7c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:16.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.772+0000 7fd2de000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2d80d4730 con 0x7fd2d80b7320 2024-03-04T09:26:16.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.772+0000 7fd2de000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2d80d45b0 con 0x7fd2d81f77a0 2024-03-04T09:26:16.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.772+0000 7fd2de000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2d80cf870 con 0x7fd2d80cdbf0 2024-03-04T09:26:16.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.772+0000 7fd2d7e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd2d80b7320 0x7fd2d80cbc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:16.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.772+0000 7fd2d7e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd2d80b7320 0x7fd2d80cbc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:59922/0 (socket says 172.21.15.6:59922) 2024-03-04T09:26:16.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.772+0000 7fd2d7e00640 1 -- 172.21.15.6:0/3888975759 learned_addr learned my addr 172.21.15.6:0/3888975759 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:16.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.773+0000 7fd2dcc00640 1 --2- 172.21.15.6:0/3888975759 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd2d80cdbf0 0x7fd2d8160f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:16.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.773+0000 7fd2dd600640 1 --2- 172.21.15.6:0/3888975759 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd2d81f77a0 0x7fd2d81f7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:16.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.773+0000 7fd2d7e00640 1 -- 172.21.15.6:0/3888975759 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd2d80cdbf0 msgr2=0x7fd2d8160f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:16.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.773+0000 7fd2d7e00640 1 --2- 172.21.15.6:0/3888975759 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd2d80cdbf0 0x7fd2d8160f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:16.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.773+0000 7fd2d7e00640 1 -- 172.21.15.6:0/3888975759 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd2d81f77a0 msgr2=0x7fd2d81f7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:16.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.773+0000 7fd2d7e00640 1 --2- 172.21.15.6:0/3888975759 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd2d81f77a0 0x7fd2d81f7c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:16.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.773+0000 7fd2d7e00640 1 -- 172.21.15.6:0/3888975759 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2d80b7a90 con 0x7fd2d80b7320 2024-03-04T09:26:16.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.773+0000 7fd2d7e00640 1 --2- 172.21.15.6:0/3888975759 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd2d80b7320 0x7fd2d80cbc20 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fd2c400d7d0 tx=0x7fd2c400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:16.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.773+0000 7fd2d4c00640 1 -- 172.21.15.6:0/3888975759 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2c4005070 con 0x7fd2d80b7320 2024-03-04T09:26:16.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.773+0000 7fd2d4c00640 1 -- 172.21.15.6:0/3888975759 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2c4005230 con 0x7fd2d80b7320 2024-03-04T09:26:16.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.773+0000 7fd2d4c00640 1 -- 172.21.15.6:0/3888975759 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2c4007030 con 0x7fd2d80b7320 2024-03-04T09:26:16.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.773+0000 7fd2de000640 1 -- 172.21.15.6:0/3888975759 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2d815e6a0 con 0x7fd2d80b7320 2024-03-04T09:26:16.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.774+0000 7fd2de000640 1 -- 172.21.15.6:0/3888975759 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2d80cdfd0 con 0x7fd2d80b7320 2024-03-04T09:26:16.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.775+0000 7fd2de000640 1 -- 172.21.15.6:0/3888975759 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2d804f330 con 0x7fd2d80b7320 2024-03-04T09:26:16.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.776+0000 7fd2d4c00640 1 -- 172.21.15.6:0/3888975759 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd2c401a030 con 0x7fd2d80b7320 2024-03-04T09:26:16.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.777+0000 7fd2d4c00640 1 --2- 172.21.15.6:0/3888975759 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd2b806b500 0x7fd2b806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:16.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.777+0000 7fd2d4c00640 1 -- 172.21.15.6:0/3888975759 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7fd2c4089970 con 0x7fd2d80b7320 2024-03-04T09:26:16.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.777+0000 7fd2dcc00640 1 --2- 172.21.15.6:0/3888975759 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd2b806b500 0x7fd2b806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:16.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.778+0000 7fd2dcc00640 1 --2- 172.21.15.6:0/3888975759 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd2b806b500 0x7fd2b806d9c0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fd2cc036000 tx=0x7fd2cc030000 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:16.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:16.781+0000 7fd2d4c00640 1 -- 172.21.15.6:0/3888975759 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd2c4021020 con 0x7fd2d80b7320 2024-03-04T09:26:17.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.023+0000 7fd2de000640 1 -- 172.21.15.6:0/3888975759 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd2d806cc90 con 0x7fd2d80b7320 2024-03-04T09:26:17.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.023+0000 7fd2d4c00640 1 -- 172.21.15.6:0/3888975759 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4971 (secure 0 0 0) 0x7fd2c40587f0 con 0x7fd2d80b7320 2024-03-04T09:26:17.020 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:26:17.021 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4248,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/2702058134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2702058134},{"type":"v1","addr":"172.21.15.6:6833","nonce":2702058134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4263,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/3866998852","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":3866998852},{"type":"v1","addr":"172.21.15.6:6835","nonce":3866998852}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4285,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/3523455455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":3523455455},{"type":"v1","addr":"172.21.15.146:6837","nonce":3523455455}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:25:18.912808+0000","modified":"2024-03-04T09:25:22.016493+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4288},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4288":{"gid":4288,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.146:6836/1946537763","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":1946537763},{"type":"v1","addr":"172.21.15.146:6836","nonce":1946537763}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4288,"qdb_cluster":[4288]},"id":1}]} 2024-03-04T09:26:17.021 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 7 2024-03-04T09:26:17.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.026+0000 7fd2de000640 1 -- 172.21.15.6:0/3888975759 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd2b806b500 msgr2=0x7fd2b806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:17.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.026+0000 7fd2de000640 1 --2- 172.21.15.6:0/3888975759 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd2b806b500 0x7fd2b806d9c0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fd2cc036000 tx=0x7fd2cc030000 comp rx=0 tx=0).stop 2024-03-04T09:26:17.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.026+0000 7fd2de000640 1 -- 172.21.15.6:0/3888975759 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd2d80b7320 msgr2=0x7fd2d80cbc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:17.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.026+0000 7fd2de000640 1 --2- 172.21.15.6:0/3888975759 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd2d80b7320 0x7fd2d80cbc20 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fd2c400d7d0 tx=0x7fd2c400dc90 comp rx=0 tx=0).stop 2024-03-04T09:26:17.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.027+0000 7fd2de000640 1 -- 172.21.15.6:0/3888975759 shutdown_connections 2024-03-04T09:26:17.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.027+0000 7fd2de000640 1 --2- 172.21.15.6:0/3888975759 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd2d81f77a0 0x7fd2d81f7c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:17.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.027+0000 7fd2de000640 1 --2- 172.21.15.6:0/3888975759 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd2d80b7320 0x7fd2d80cbc20 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:17.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.027+0000 7fd2de000640 1 --2- 172.21.15.6:0/3888975759 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd2b806b500 0x7fd2b806d9c0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:17.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.027+0000 7fd2de000640 1 --2- 172.21.15.6:0/3888975759 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd2d80cdbf0 0x7fd2d8160f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:17.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.027+0000 7fd2de000640 1 -- 172.21.15.6:0/3888975759 >> 172.21.15.6:0/3888975759 conn(0x7fd2d808d2a0 msgr2=0x7fd2d806cee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:17.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.027+0000 7fd2de000640 1 -- 172.21.15.6:0/3888975759 shutdown_connections 2024-03-04T09:26:17.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.027+0000 7fd2de000640 1 -- 172.21.15.6:0/3888975759 wait complete. 2024-03-04T09:26:17.043 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:26:17.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.269+0000 7f769ca00640 1 Processor -- start 2024-03-04T09:26:17.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.269+0000 7f769ca00640 1 -- start start 2024-03-04T09:26:17.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.271+0000 7f769ca00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f7698145300 0x7f76980b49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:17.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.271+0000 7f769ca00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f76980cd060 0x7f769815c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:17.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.271+0000 7f769ca00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76980714d0 con 0x7f7698070f10 2024-03-04T09:26:17.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.271+0000 7f769ca00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76980717b0 con 0x7f7698145300 2024-03-04T09:26:17.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.271+0000 7f769ca00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7698071a90 con 0x7f76980cd060 2024-03-04T09:26:17.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.271+0000 7f7697400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f7698145300 0x7f76980b49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:17.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.271+0000 7f7697400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f7698145300 0x7f76980b49a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:59938/0 (socket says 172.21.15.6:59938) 2024-03-04T09:26:17.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.271+0000 7f7697400640 1 -- 172.21.15.6:0/1785175006 learned_addr learned my addr 172.21.15.6:0/1785175006 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:17.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.272+0000 7f7697e00640 1 --2- 172.21.15.6:0/1785175006 >> v2:172.21.15.146:3300/0 conn(0x7f76980cd060 0x7f769815c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:17.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.272+0000 7f7697400640 1 -- 172.21.15.6:0/1785175006 >> v2:172.21.15.146:3300/0 conn(0x7f76980cd060 msgr2=0x7f769815c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:17.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.272+0000 7f7697400640 1 --2- 172.21.15.6:0/1785175006 >> v2:172.21.15.146:3300/0 conn(0x7f76980cd060 0x7f769815c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:17.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.272+0000 7f7697400640 1 -- 172.21.15.6:0/1785175006 >> v1:172.21.15.146:6789/0 conn(0x7f7698070f10 legacy=0x7f76980cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:17.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.272+0000 7f7697400640 1 -- 172.21.15.6:0/1785175006 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7698071d70 con 0x7f7698145300 2024-03-04T09:26:17.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.273+0000 7f7697400640 1 --2- 172.21.15.6:0/1785175006 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7698145300 0x7f76980b49a0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f76980cc500 tx=0x7f768802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37cf6aa64588c42d server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:17.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.274+0000 7f7696a00640 1 -- 172.21.15.6:0/1785175006 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f768802e070 con 0x7f7698145300 2024-03-04T09:26:17.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.274+0000 7f7696a00640 1 -- 172.21.15.6:0/1785175006 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7688037070 con 0x7f7698145300 2024-03-04T09:26:17.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.274+0000 7f7696a00640 1 -- 172.21.15.6:0/1785175006 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7688031a00 con 0x7f7698145300 2024-03-04T09:26:17.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.274+0000 7f769ca00640 1 -- 172.21.15.6:0/1785175006 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7698145300 msgr2=0x7f76980b49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:17.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.274+0000 7f769ca00640 1 --2- 172.21.15.6:0/1785175006 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7698145300 0x7f76980b49a0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f76980cc500 tx=0x7f768802b880 comp rx=0 tx=0).stop 2024-03-04T09:26:17.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.274+0000 7f769ca00640 1 -- 172.21.15.6:0/1785175006 shutdown_connections 2024-03-04T09:26:17.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.274+0000 7f769ca00640 1 --2- 172.21.15.6:0/1785175006 >> v2:172.21.15.146:3300/0 conn(0x7f76980cd060 0x7f769815c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:17.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.274+0000 7f769ca00640 1 --2- 172.21.15.6:0/1785175006 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7698145300 0x7f76980b49a0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:17.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.274+0000 7f769ca00640 1 -- 172.21.15.6:0/1785175006 >> 172.21.15.6:0/1785175006 conn(0x7f769806b3d0 msgr2=0x7f769806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:17.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.274+0000 7f769ca00640 1 -- 172.21.15.6:0/1785175006 shutdown_connections 2024-03-04T09:26:17.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.274+0000 7f769ca00640 1 -- 172.21.15.6:0/1785175006 wait complete. 2024-03-04T09:26:17.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.276+0000 7f769ca00640 1 Processor -- start 2024-03-04T09:26:17.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.276+0000 7f769ca00640 1 -- start start 2024-03-04T09:26:17.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.277+0000 7f769ca00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7698070f10 0x7f7698140800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:17.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.277+0000 7f769ca00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f76980cd060 0x7f76981e4d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:17.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.277+0000 7f769ca00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7698145300 0x7f76980b03a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:17.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.277+0000 7f769ca00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f769815cab0 con 0x7f76980cd060 2024-03-04T09:26:17.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.277+0000 7f769ca00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76981456e0 con 0x7f7698070f10 2024-03-04T09:26:17.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.277+0000 7f769ca00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f769815cc30 con 0x7f7698145300 2024-03-04T09:26:17.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.278+0000 7f768fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f76980cd060 0x7f76981e4d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:17.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.278+0000 7f768fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f76980cd060 0x7f76981e4d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:59950/0 (socket says 172.21.15.6:59950) 2024-03-04T09:26:17.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.278+0000 7f768fe00640 1 -- 172.21.15.6:0/2730507916 learned_addr learned my addr 172.21.15.6:0/2730507916 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:17.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.278+0000 7f7697400640 1 --2- 172.21.15.6:0/2730507916 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7698070f10 0x7f7698140800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:17.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.278+0000 7f7697e00640 1 --2- 172.21.15.6:0/2730507916 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7698145300 0x7f76980b03a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:17.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.278+0000 7f768fe00640 1 -- 172.21.15.6:0/2730507916 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7698145300 msgr2=0x7f76980b03a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:17.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.278+0000 7f768fe00640 1 --2- 172.21.15.6:0/2730507916 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7698145300 0x7f76980b03a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:17.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.278+0000 7f768fe00640 1 -- 172.21.15.6:0/2730507916 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7698070f10 msgr2=0x7f7698140800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:17.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.278+0000 7f768fe00640 1 --2- 172.21.15.6:0/2730507916 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7698070f10 0x7f7698140800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:17.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.278+0000 7f768fe00640 1 -- 172.21.15.6:0/2730507916 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76981e74d0 con 0x7f76980cd060 2024-03-04T09:26:17.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.278+0000 7f7697400640 1 --2- 172.21.15.6:0/2730507916 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7698070f10 0x7f7698140800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T09:26:17.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.279+0000 7f768fe00640 1 --2- 172.21.15.6:0/2730507916 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f76980cd060 0x7f76981e4d10 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f768000c430 tx=0x7f768000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:17.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.279+0000 7f768f400640 1 -- 172.21.15.6:0/2730507916 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7680011790 con 0x7f76980cd060 2024-03-04T09:26:17.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.279+0000 7f768f400640 1 -- 172.21.15.6:0/2730507916 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7680017070 con 0x7f76980cd060 2024-03-04T09:26:17.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.279+0000 7f769ca00640 1 -- 172.21.15.6:0/2730507916 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76981e77b0 con 0x7f76980cd060 2024-03-04T09:26:17.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.279+0000 7f768f400640 1 -- 172.21.15.6:0/2730507916 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7680011bb0 con 0x7f76980cd060 2024-03-04T09:26:17.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.279+0000 7f769ca00640 1 -- 172.21.15.6:0/2730507916 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76981e7d70 con 0x7f76980cd060 2024-03-04T09:26:17.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.280+0000 7f769ca00640 1 -- 172.21.15.6:0/2730507916 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f769804f330 con 0x7f76980cd060 2024-03-04T09:26:17.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.281+0000 7f768f400640 1 -- 172.21.15.6:0/2730507916 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f768001c460 con 0x7f76980cd060 2024-03-04T09:26:17.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.282+0000 7f768f400640 1 --2- 172.21.15.6:0/2730507916 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f767406b4d0 0x7f767406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:17.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.282+0000 7f768f400640 1 -- 172.21.15.6:0/2730507916 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f768008a0f0 con 0x7f76980cd060 2024-03-04T09:26:17.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.283+0000 7f7697400640 1 --2- 172.21.15.6:0/2730507916 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f767406b4d0 0x7f767406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:17.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.284+0000 7f7697400640 1 --2- 172.21.15.6:0/2730507916 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f767406b4d0 0x7f767406d990 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f7688036000 tx=0x7f768803d040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:17.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.286+0000 7f768f400640 1 -- 172.21.15.6:0/2730507916 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7680058f70 con 0x7f76980cd060 2024-03-04T09:26:17.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.529+0000 7f769ca00640 1 -- 172.21.15.6:0/2730507916 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7698140d40 con 0x7f76980cd060 2024-03-04T09:26:17.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.530+0000 7f768f400640 1 -- 172.21.15.6:0/2730507916 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4971 (secure 0 0 0) 0x7f7680058870 con 0x7f76980cd060 2024-03-04T09:26:17.527 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:26:17.527 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4248,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/2702058134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2702058134},{"type":"v1","addr":"172.21.15.6:6833","nonce":2702058134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4263,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/3866998852","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":3866998852},{"type":"v1","addr":"172.21.15.6:6835","nonce":3866998852}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4285,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/3523455455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":3523455455},{"type":"v1","addr":"172.21.15.146:6837","nonce":3523455455}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:25:18.912808+0000","modified":"2024-03-04T09:25:22.016493+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4288},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4288":{"gid":4288,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.146:6836/1946537763","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":1946537763},{"type":"v1","addr":"172.21.15.146:6836","nonce":1946537763}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4288,"qdb_cluster":[4288]},"id":1}]} 2024-03-04T09:26:17.528 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 7 2024-03-04T09:26:17.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.533+0000 7f769ca00640 1 -- 172.21.15.6:0/2730507916 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f767406b4d0 msgr2=0x7f767406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:17.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.533+0000 7f769ca00640 1 --2- 172.21.15.6:0/2730507916 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f767406b4d0 0x7f767406d990 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f7688036000 tx=0x7f768803d040 comp rx=0 tx=0).stop 2024-03-04T09:26:17.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.533+0000 7f769ca00640 1 -- 172.21.15.6:0/2730507916 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f76980cd060 msgr2=0x7f76981e4d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:17.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.533+0000 7f769ca00640 1 --2- 172.21.15.6:0/2730507916 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f76980cd060 0x7f76981e4d10 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f768000c430 tx=0x7f768000c8f0 comp rx=0 tx=0).stop 2024-03-04T09:26:17.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.533+0000 7f769ca00640 1 -- 172.21.15.6:0/2730507916 shutdown_connections 2024-03-04T09:26:17.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.533+0000 7f769ca00640 1 --2- 172.21.15.6:0/2730507916 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f767406b4d0 0x7f767406d990 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:17.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.533+0000 7f769ca00640 1 --2- 172.21.15.6:0/2730507916 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7698145300 0x7f76980b03a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:17.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.533+0000 7f769ca00640 1 --2- 172.21.15.6:0/2730507916 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f76980cd060 0x7f76981e4d10 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:17.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.533+0000 7f769ca00640 1 --2- 172.21.15.6:0/2730507916 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7698070f10 0x7f7698140800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:17.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.533+0000 7f769ca00640 1 -- 172.21.15.6:0/2730507916 >> 172.21.15.6:0/2730507916 conn(0x7f769806b3d0 msgr2=0x7f769815cf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:17.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.533+0000 7f769ca00640 1 -- 172.21.15.6:0/2730507916 shutdown_connections 2024-03-04T09:26:17.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.533+0000 7f769ca00640 1 -- 172.21.15.6:0/2730507916 wait complete. 2024-03-04T09:26:17.550 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T09:26:17.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.771+0000 7f57c4800640 1 Processor -- start 2024-03-04T09:26:17.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.772+0000 7f57c4800640 1 -- start start 2024-03-04T09:26:17.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.773+0000 7f57c4800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f57bc0af580 0x7f57bc15c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:17.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.773+0000 7f57c4800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f57bc0cdc20 0x7f57bc0b2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:17.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.773+0000 7f57c4800640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57bc15cc40 con 0x7f57bc0af1a0 2024-03-04T09:26:17.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.773+0000 7f57c4800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57bc15cf20 con 0x7f57bc0cdc20 2024-03-04T09:26:17.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.773+0000 7f57c4800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57bc15d200 con 0x7f57bc0af580 2024-03-04T09:26:17.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.774+0000 7f57c1e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f57bc0af1a0 0x7f57bc0b4c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:60620/0 (socket says 172.21.15.6:60620) 2024-03-04T09:26:17.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.774+0000 7f57c1e00640 1 -- 172.21.15.6:0/4186997548 learned_addr learned my addr 172.21.15.6:0/4186997548 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:26:17.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.774+0000 7f57c2800640 1 --2- 172.21.15.6:0/4186997548 >> v2:172.21.15.6:3300/0 conn(0x7f57bc0cdc20 0x7f57bc0b2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:17.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.774+0000 7f57c1400640 1 --2- 172.21.15.6:0/4186997548 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f57bc0af580 0x7f57bc15c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:17.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.774+0000 7f57c0a00640 1 -- 172.21.15.6:0/4186997548 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2507058876 0 0) 0x7f57bc15cc40 con 0x7f57bc0af1a0 2024-03-04T09:26:17.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.774+0000 7f57c0a00640 1 -- 172.21.15.6:0/4186997548 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57a8003520 con 0x7f57bc0af1a0 2024-03-04T09:26:17.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.774+0000 7f57c2800640 1 -- 172.21.15.6:0/4186997548 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f57bc0af580 msgr2=0x7f57bc15c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:17.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.774+0000 7f57c2800640 1 --2- 172.21.15.6:0/4186997548 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f57bc0af580 0x7f57bc15c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:17.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.774+0000 7f57c2800640 1 -- 172.21.15.6:0/4186997548 >> v1:172.21.15.6:6789/0 conn(0x7f57bc0af1a0 legacy=0x7f57bc0b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:17.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.774+0000 7f57c2800640 1 -- 172.21.15.6:0/4186997548 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57bc15d4e0 con 0x7f57bc0cdc20 2024-03-04T09:26:17.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.776+0000 7f57c2800640 1 --2- 172.21.15.6:0/4186997548 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f57bc0cdc20 0x7f57bc0b2dd0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f57b80097a0 tx=0x7f57b802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6c57b51c33418e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:17.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.776+0000 7f57c0a00640 1 -- 172.21.15.6:0/4186997548 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57b80315a0 con 0x7f57bc0cdc20 2024-03-04T09:26:17.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.776+0000 7f57c0a00640 1 -- 172.21.15.6:0/4186997548 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57b8037070 con 0x7f57bc0cdc20 2024-03-04T09:26:17.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.776+0000 7f57c0a00640 1 -- 172.21.15.6:0/4186997548 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57b8031940 con 0x7f57bc0cdc20 2024-03-04T09:26:17.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.777+0000 7f57c4800640 1 -- 172.21.15.6:0/4186997548 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f57bc0cdc20 msgr2=0x7f57bc0b2dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:17.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.777+0000 7f57c4800640 1 --2- 172.21.15.6:0/4186997548 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f57bc0cdc20 0x7f57bc0b2dd0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f57b80097a0 tx=0x7f57b802b840 comp rx=0 tx=0).stop 2024-03-04T09:26:17.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.777+0000 7f57c4800640 1 -- 172.21.15.6:0/4186997548 shutdown_connections 2024-03-04T09:26:17.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.777+0000 7f57c4800640 1 --2- 172.21.15.6:0/4186997548 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f57bc0cdc20 0x7f57bc0b2dd0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:17.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.777+0000 7f57c4800640 1 --2- 172.21.15.6:0/4186997548 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f57bc0af580 0x7f57bc15c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:17.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.777+0000 7f57c4800640 1 -- 172.21.15.6:0/4186997548 >> 172.21.15.6:0/4186997548 conn(0x7f57bc06afb0 msgr2=0x7f57bc06b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:17.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.777+0000 7f57c4800640 1 -- 172.21.15.6:0/4186997548 shutdown_connections 2024-03-04T09:26:17.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.777+0000 7f57c4800640 1 -- 172.21.15.6:0/4186997548 wait complete. 2024-03-04T09:26:17.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.778+0000 7f57c4800640 1 Processor -- start 2024-03-04T09:26:17.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.779+0000 7f57c4800640 1 -- start start 2024-03-04T09:26:17.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.780+0000 7f57c4800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f57bc0af1a0 0x7f57bc0c9980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:17.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.780+0000 7f57c4800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f57bc0af580 0x7f57bc156cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:17.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.780+0000 7f57c4800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f57bc0cdc20 0x7f57bc06db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:17.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.780+0000 7f57c4800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57bc0b36a0 con 0x7f57bc0af1a0 2024-03-04T09:26:17.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.780+0000 7f57c4800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57bc0b3520 con 0x7f57bc0cdc20 2024-03-04T09:26:17.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.780+0000 7f57c4800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57bc0b3820 con 0x7f57bc0af580 2024-03-04T09:26:17.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.780+0000 7f57c1e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f57bc0af1a0 0x7f57bc0c9980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:17.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.780+0000 7f57c1e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f57bc0af1a0 0x7f57bc0c9980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:59972/0 (socket says 172.21.15.6:59972) 2024-03-04T09:26:17.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.780+0000 7f57c1e00640 1 -- 172.21.15.6:0/422343345 learned_addr learned my addr 172.21.15.6:0/422343345 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:17.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.780+0000 7f57c1e00640 1 -- 172.21.15.6:0/422343345 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f57bc0af580 msgr2=0x7f57bc156cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:17.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.780+0000 7f57c1e00640 1 --2- 172.21.15.6:0/422343345 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f57bc0af580 0x7f57bc156cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:17.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.780+0000 7f57c1e00640 1 -- 172.21.15.6:0/422343345 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f57bc0cdc20 msgr2=0x7f57bc06db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:17.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.780+0000 7f57c1e00640 1 --2- 172.21.15.6:0/422343345 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f57bc0cdc20 0x7f57bc06db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:17.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.780+0000 7f57c1e00640 1 -- 172.21.15.6:0/422343345 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57bc1571f0 con 0x7f57bc0af1a0 2024-03-04T09:26:17.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.781+0000 7f57c1e00640 1 --2- 172.21.15.6:0/422343345 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f57bc0af1a0 0x7f57bc0c9980 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f57b400cb40 tx=0x7f57b400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:17.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.781+0000 7f57b2000640 1 -- 172.21.15.6:0/422343345 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57b4017070 con 0x7f57bc0af1a0 2024-03-04T09:26:17.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.781+0000 7f57b2000640 1 -- 172.21.15.6:0/422343345 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57b400aca0 con 0x7f57bc0af1a0 2024-03-04T09:26:17.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.781+0000 7f57b2000640 1 -- 172.21.15.6:0/422343345 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57b4012620 con 0x7f57bc0af1a0 2024-03-04T09:26:17.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.781+0000 7f57c4800640 1 -- 172.21.15.6:0/422343345 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57bc06e0a0 con 0x7f57bc0af1a0 2024-03-04T09:26:17.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.781+0000 7f57c4800640 1 -- 172.21.15.6:0/422343345 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57bc0ca3f0 con 0x7f57bc0af1a0 2024-03-04T09:26:17.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.782+0000 7f57c4800640 1 -- 172.21.15.6:0/422343345 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57bc069510 con 0x7f57bc0af1a0 2024-03-04T09:26:17.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.783+0000 7f57b2000640 1 -- 172.21.15.6:0/422343345 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f57b401a070 con 0x7f57bc0af1a0 2024-03-04T09:26:17.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.785+0000 7f57b2000640 1 --2- 172.21.15.6:0/422343345 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f57a006b4d0 0x7f57a006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:17.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.785+0000 7f57b2000640 1 -- 172.21.15.6:0/422343345 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f57b408a1f0 con 0x7f57bc0af1a0 2024-03-04T09:26:17.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.786+0000 7f57c1400640 1 --2- 172.21.15.6:0/422343345 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f57a006b4d0 0x7f57a006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:17.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.787+0000 7f57c1400640 1 --2- 172.21.15.6:0/422343345 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f57a006b4d0 0x7f57a006d990 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f57ac006a90 tx=0x7f57ac006a20 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:17.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.790+0000 7f57b2000640 1 -- 172.21.15.6:0/422343345 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f57bc1571f0 con 0x7f57bc0af1a0 2024-03-04T09:26:17.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.990+0000 7f57c4800640 1 -- 172.21.15.6:0/422343345 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f57bc04f3a0 con 0x7f57bc0af1a0 2024-03-04T09:26:17.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.993+0000 7f57b2000640 1 -- 172.21.15.6:0/422343345 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16852 (secure 0 0 0) 0x7f57b4058eb0 con 0x7f57bc0af1a0 2024-03-04T09:26:17.990 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:26:17.990 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":22,"fsid":"edce0578-5ae4-4402-b45a-552bdef84d7c","created":"2024-03-04T09:24:54.523780+0000","modified":"2024-03-04T09:26:01.727682+0000","last_up_change":"2024-03-04T09:25:07.089049+0000","last_in_change":"2024-03-04T09:24:57.982077+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T09:25:07.935198+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-04T09:25:17.118573+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-04T09:25:17.930536+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a96574f-81f1-4bc6-8463-757520b84a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6825","nonce":1364449401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6827","nonce":1364449401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6831","nonce":1364449401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6829","nonce":1364449401}]},"public_addr":"172.21.15.6:6825/1364449401","cluster_addr":"172.21.15.6:6827/1364449401","heartbeat_back_addr":"172.21.15.6:6831/1364449401","heartbeat_front_addr":"172.21.15.6:6829/1364449401","state":["exists","up"]},{"osd":1,"uuid":"7d79b9c1-6b87-476a-bc36-7db5ee64cd90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6809","nonce":2494103078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6811","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6813","nonce":2494103078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6821","nonce":2494103078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6817","nonce":2494103078}]},"public_addr":"172.21.15.6:6809/2494103078","cluster_addr":"172.21.15.6:6813/2494103078","heartbeat_back_addr":"172.21.15.6:6821/2494103078","heartbeat_front_addr":"172.21.15.6:6817/2494103078","state":["exists","up"]},{"osd":2,"uuid":"492331b9-4ee9-40b6-bacd-85d44cd6c45a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6812","nonce":1500066681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6816","nonce":1500066681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6824","nonce":1500066681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6820","nonce":1500066681}]},"public_addr":"172.21.15.6:6812/1500066681","cluster_addr":"172.21.15.6:6816/1500066681","heartbeat_back_addr":"172.21.15.6:6824/1500066681","heartbeat_front_addr":"172.21.15.6:6820/1500066681","state":["exists","up"]},{"osd":3,"uuid":"21bb2efe-9ba7-4bdc-bc31-180cd1102e21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6801","nonce":1558806123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6803","nonce":1558806123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6807","nonce":1558806123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6805","nonce":1558806123}]},"public_addr":"172.21.15.6:6801/1558806123","cluster_addr":"172.21.15.6:6803/1558806123","heartbeat_back_addr":"172.21.15.6:6807/1558806123","heartbeat_front_addr":"172.21.15.6:6805/1558806123","state":["exists","up"]},{"osd":4,"uuid":"6e2c4b71-19d9-4737-83d9-558705d8644a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6801","nonce":1526135780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6803","nonce":1526135780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6807","nonce":1526135780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6805","nonce":1526135780}]},"public_addr":"172.21.15.146:6801/1526135780","cluster_addr":"172.21.15.146:6803/1526135780","heartbeat_back_addr":"172.21.15.146:6807/1526135780","heartbeat_front_addr":"172.21.15.146:6805/1526135780","state":["exists","up"]},{"osd":5,"uuid":"fb69c525-f1be-4b7d-ab32-6c29b0d6fddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6809","nonce":460262507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6811","nonce":460262507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6815","nonce":460262507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6813","nonce":460262507}]},"public_addr":"172.21.15.146:6809/460262507","cluster_addr":"172.21.15.146:6811/460262507","heartbeat_back_addr":"172.21.15.146:6815/460262507","heartbeat_front_addr":"172.21.15.146:6813/460262507","state":["exists","up"]},{"osd":6,"uuid":"c32e7fed-370d-4401-8ce0-6a4335b0d168","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6823","nonce":1392897493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6827","nonce":1392897493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6831","nonce":1392897493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6829","nonce":1392897493}]},"public_addr":"172.21.15.146:6823/1392897493","cluster_addr":"172.21.15.146:6827/1392897493","heartbeat_back_addr":"172.21.15.146:6831/1392897493","heartbeat_front_addr":"172.21.15.146:6829/1392897493","state":["exists","up"]},{"osd":7,"uuid":"75a79b44-fa5f-426d-a7d3-71f21f78835c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6817","nonce":3268323156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6819","nonce":3268323156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6825","nonce":3268323156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6822","nonce":3268323156}]},"public_addr":"172.21.15.146:6817/3268323156","cluster_addr":"172.21.15.146:6819/3268323156","heartbeat_back_addr":"172.21.15.146:6825/3268323156","heartbeat_front_addr":"172.21.15.146:6822/3268323156","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.310437+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.295850+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.294674+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.526855+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.300023+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.595160+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.346046+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.558741+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.7","mappings":[{"from":3,"to":2}]},{"pgid":"3.f","mappings":[{"from":7,"to":2}]},{"pgid":"3.21","mappings":[{"from":5,"to":1}]},{"pgid":"3.22","mappings":[{"from":5,"to":2}]},{"pgid":"3.28","mappings":[{"from":3,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T09:26:17.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.996+0000 7f57c4800640 1 -- 172.21.15.6:0/422343345 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f57a006b4d0 msgr2=0x7f57a006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:17.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.996+0000 7f57c4800640 1 --2- 172.21.15.6:0/422343345 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f57a006b4d0 0x7f57a006d990 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f57ac006a90 tx=0x7f57ac006a20 comp rx=0 tx=0).stop 2024-03-04T09:26:17.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.996+0000 7f57c4800640 1 -- 172.21.15.6:0/422343345 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f57bc0af1a0 msgr2=0x7f57bc0c9980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:17.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.996+0000 7f57c4800640 1 --2- 172.21.15.6:0/422343345 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f57bc0af1a0 0x7f57bc0c9980 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f57b400cb40 tx=0x7f57b400a060 comp rx=0 tx=0).stop 2024-03-04T09:26:17.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.996+0000 7f57c4800640 1 -- 172.21.15.6:0/422343345 shutdown_connections 2024-03-04T09:26:17.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.996+0000 7f57c4800640 1 --2- 172.21.15.6:0/422343345 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f57bc0cdc20 0x7f57bc06db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:17.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.996+0000 7f57c4800640 1 --2- 172.21.15.6:0/422343345 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f57a006b4d0 0x7f57a006d990 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:17.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.996+0000 7f57c4800640 1 --2- 172.21.15.6:0/422343345 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f57bc0af580 0x7f57bc156cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:17.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.996+0000 7f57c4800640 1 --2- 172.21.15.6:0/422343345 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f57bc0af1a0 0x7f57bc0c9980 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:17.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.996+0000 7f57c4800640 1 -- 172.21.15.6:0/422343345 >> 172.21.15.6:0/422343345 conn(0x7f57bc06afb0 msgr2=0x7f57bc15ba80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:17.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.996+0000 7f57c4800640 1 -- 172.21.15.6:0/422343345 shutdown_connections 2024-03-04T09:26:17.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:17.996+0000 7f57c4800640 1 -- 172.21.15.6:0/422343345 wait complete. 2024-03-04T09:26:18.014 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-04T09:26:18.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.239+0000 7f6361000640 1 Processor -- start 2024-03-04T09:26:18.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.239+0000 7f6361000640 1 -- start start 2024-03-04T09:26:18.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.240+0000 7f6361000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f635c0af580 0x7f635c15c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:18.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.240+0000 7f6361000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f635c15cc40 con 0x7f635c0cdc20 2024-03-04T09:26:18.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.241+0000 7f6361000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f635c15cf20 con 0x7f635c0af1a0 2024-03-04T09:26:18.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.241+0000 7f6361000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f635c15d200 con 0x7f635c0af580 2024-03-04T09:26:18.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.241+0000 7f635a800640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f635c0cdc20 0x7f635c0b2dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:60632/0 (socket says 172.21.15.6:60632) 2024-03-04T09:26:18.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.241+0000 7f6359400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f635c0af580 0x7f635c15c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:18.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.241+0000 7f635a800640 1 -- 172.21.15.6:0/1274788188 learned_addr learned my addr 172.21.15.6:0/1274788188 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:26:18.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.241+0000 7f6358a00640 1 -- 172.21.15.6:0/1274788188 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1665404036 0 0) 0x7f635c15cc40 con 0x7f635c0cdc20 2024-03-04T09:26:18.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.241+0000 7f6358a00640 1 -- 172.21.15.6:0/1274788188 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6340003520 con 0x7f635c0cdc20 2024-03-04T09:26:18.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.241+0000 7f6359400640 1 -- 172.21.15.6:0/1274788188 >> v1:172.21.15.146:6789/0 conn(0x7f635c0af1a0 legacy=0x7f635c0b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:18.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.242+0000 7f6359400640 1 -- 172.21.15.6:0/1274788188 >> v1:172.21.15.6:6789/0 conn(0x7f635c0cdc20 legacy=0x7f635c0b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:18.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.242+0000 7f6359400640 1 -- 172.21.15.6:0/1274788188 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f635c15d4e0 con 0x7f635c0af580 2024-03-04T09:26:18.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.243+0000 7f6359400640 1 --2- 172.21.15.6:0/1274788188 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f635c0af580 0x7f635c15c600 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f63480097a0 tx=0x7f634802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11bdfcbed9412010 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:18.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.243+0000 7f6358a00640 1 -- 172.21.15.6:0/1274788188 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6348035020 con 0x7f635c0af580 2024-03-04T09:26:18.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.243+0000 7f6358a00640 1 -- 172.21.15.6:0/1274788188 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6348031830 con 0x7f635c0af580 2024-03-04T09:26:18.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.243+0000 7f6358a00640 1 -- 172.21.15.6:0/1274788188 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6348031be0 con 0x7f635c0af580 2024-03-04T09:26:18.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.244+0000 7f6361000640 1 -- 172.21.15.6:0/1274788188 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f635c0af580 msgr2=0x7f635c15c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:18.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.244+0000 7f6361000640 1 --2- 172.21.15.6:0/1274788188 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f635c0af580 0x7f635c15c600 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f63480097a0 tx=0x7f634802b8d0 comp rx=0 tx=0).stop 2024-03-04T09:26:18.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.244+0000 7f6361000640 1 -- 172.21.15.6:0/1274788188 shutdown_connections 2024-03-04T09:26:18.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.244+0000 7f6361000640 1 --2- 172.21.15.6:0/1274788188 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f635c0af580 0x7f635c15c600 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:18.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.244+0000 7f6361000640 1 -- 172.21.15.6:0/1274788188 >> 172.21.15.6:0/1274788188 conn(0x7f635c06afb0 msgr2=0x7f635c06b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:18.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.244+0000 7f6361000640 1 -- 172.21.15.6:0/1274788188 shutdown_connections 2024-03-04T09:26:18.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.244+0000 7f6361000640 1 -- 172.21.15.6:0/1274788188 wait complete. 2024-03-04T09:26:18.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.246+0000 7f6361000640 1 Processor -- start 2024-03-04T09:26:18.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.246+0000 7f6361000640 1 -- start start 2024-03-04T09:26:18.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.247+0000 7f6361000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f635c0af1a0 0x7f635c151050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:18.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.247+0000 7f6361000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f635c0af580 0x7f635c158550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:18.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.247+0000 7f6361000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f635c0cdc20 0x7f635c09ce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:18.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.247+0000 7f6361000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f635c155e70 con 0x7f635c0af580 2024-03-04T09:26:18.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.247+0000 7f6361000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f635c0b3730 con 0x7f635c0af1a0 2024-03-04T09:26:18.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.247+0000 7f6361000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f635c155ff0 con 0x7f635c0cdc20 2024-03-04T09:26:18.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.247+0000 7f6359400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f635c0af580 0x7f635c158550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:18.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.247+0000 7f6359400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f635c0af580 0x7f635c158550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:59990/0 (socket says 172.21.15.6:59990) 2024-03-04T09:26:18.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.247+0000 7f6359400640 1 -- 172.21.15.6:0/302560130 learned_addr learned my addr 172.21.15.6:0/302560130 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:18.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.247+0000 7f6359e00640 1 --2- 172.21.15.6:0/302560130 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f635c0af1a0 0x7f635c151050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:18.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.248+0000 7f635a800640 1 --2- 172.21.15.6:0/302560130 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f635c0cdc20 0x7f635c09ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:18.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.248+0000 7f6359400640 1 -- 172.21.15.6:0/302560130 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f635c0cdc20 msgr2=0x7f635c09ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:18.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.248+0000 7f6359400640 1 --2- 172.21.15.6:0/302560130 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f635c0cdc20 0x7f635c09ce50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:18.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.248+0000 7f6359400640 1 -- 172.21.15.6:0/302560130 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f635c0af1a0 msgr2=0x7f635c151050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:18.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.248+0000 7f6359400640 1 --2- 172.21.15.6:0/302560130 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f635c0af1a0 0x7f635c151050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:18.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.248+0000 7f6359400640 1 -- 172.21.15.6:0/302560130 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f635c1564e0 con 0x7f635c0af580 2024-03-04T09:26:18.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.248+0000 7f6359400640 1 --2- 172.21.15.6:0/302560130 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f635c0af580 0x7f635c158550 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f63480098c0 tx=0x7f63480046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:18.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.248+0000 7f6352000640 1 -- 172.21.15.6:0/302560130 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6348048070 con 0x7f635c0af580 2024-03-04T09:26:18.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.248+0000 7f6352000640 1 -- 172.21.15.6:0/302560130 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6348004450 con 0x7f635c0af580 2024-03-04T09:26:18.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.248+0000 7f6352000640 1 -- 172.21.15.6:0/302560130 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6348035070 con 0x7f635c0af580 2024-03-04T09:26:18.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.249+0000 7f6361000640 1 -- 172.21.15.6:0/302560130 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f635c1567c0 con 0x7f635c0af580 2024-03-04T09:26:18.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.249+0000 7f6361000640 1 -- 172.21.15.6:0/302560130 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f635c156d80 con 0x7f635c0af580 2024-03-04T09:26:18.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.250+0000 7f6361000640 1 -- 172.21.15.6:0/302560130 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f635c069510 con 0x7f635c0af580 2024-03-04T09:26:18.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.251+0000 7f6352000640 1 -- 172.21.15.6:0/302560130 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f634804b070 con 0x7f635c0af580 2024-03-04T09:26:18.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.252+0000 7f6352000640 1 --2- 172.21.15.6:0/302560130 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f633c06b4d0 0x7f633c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:18.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.252+0000 7f6352000640 1 -- 172.21.15.6:0/302560130 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f63480aba90 con 0x7f635c0af580 2024-03-04T09:26:18.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.252+0000 7f6359e00640 1 --2- 172.21.15.6:0/302560130 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f633c06b4d0 0x7f633c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:18.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.253+0000 7f6359e00640 1 --2- 172.21.15.6:0/302560130 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f633c06b4d0 0x7f633c06d990 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f635c09c5d0 tx=0x7f634c009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:18.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.256+0000 7f6352000640 1 -- 172.21.15.6:0/302560130 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6348040070 con 0x7f635c0af580 2024-03-04T09:26:18.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.498+0000 7f6361000640 1 -- 172.21.15.6:0/302560130 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f635c155910 con 0x7f635c0af580 2024-03-04T09:26:18.496 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-03-04T09:26:18.499+0000 7f1606a00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-04T09:26:18.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.502+0000 7f6352000640 1 -- 172.21.15.6:0/302560130 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v8) v1 ==== 161+0+0 (secure 0 0 0) 0x7f634807a220 con 0x7f635c0af580 2024-03-04T09:26:18.499 INFO:teuthology.orchestra.run.smithi006.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-04T09:26:18.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.506+0000 7f6361000640 1 -- 172.21.15.6:0/302560130 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f633c06b4d0 msgr2=0x7f633c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:18.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.506+0000 7f6361000640 1 --2- 172.21.15.6:0/302560130 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f633c06b4d0 0x7f633c06d990 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f635c09c5d0 tx=0x7f634c009040 comp rx=0 tx=0).stop 2024-03-04T09:26:18.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.506+0000 7f6361000640 1 -- 172.21.15.6:0/302560130 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f635c0af580 msgr2=0x7f635c158550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:18.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.506+0000 7f6361000640 1 --2- 172.21.15.6:0/302560130 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f635c0af580 0x7f635c158550 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f63480098c0 tx=0x7f63480046c0 comp rx=0 tx=0).stop 2024-03-04T09:26:18.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.506+0000 7f6361000640 1 -- 172.21.15.6:0/302560130 shutdown_connections 2024-03-04T09:26:18.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.506+0000 7f6361000640 1 --2- 172.21.15.6:0/302560130 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f633c06b4d0 0x7f633c06d990 secure :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f635c09c5d0 tx=0x7f634c009040 comp rx=0 tx=0).stop 2024-03-04T09:26:18.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.506+0000 7f6361000640 1 --2- 172.21.15.6:0/302560130 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f635c0cdc20 0x7f635c09ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:18.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.506+0000 7f6361000640 1 --2- 172.21.15.6:0/302560130 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f635c0af580 0x7f635c158550 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:18.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.506+0000 7f6361000640 1 --2- 172.21.15.6:0/302560130 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f635c0af1a0 0x7f635c151050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:18.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.506+0000 7f6361000640 1 -- 172.21.15.6:0/302560130 >> 172.21.15.6:0/302560130 conn(0x7f635c06afb0 msgr2=0x7f635c0cc4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:18.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.506+0000 7f6361000640 1 -- 172.21.15.6:0/302560130 shutdown_connections 2024-03-04T09:26:18.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.506+0000 7f6361000640 1 -- 172.21.15.6:0/302560130 wait complete. 2024-03-04T09:26:18.524 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-04T09:26:18.541 INFO:tasks.ceph.mds.d.smithi146.stderr: -7979> 2024-03-04T09:25:20.054+0000 7f9f6f200640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-04T09:26:18.633 INFO:tasks.ceph.mds.d.smithi146.stdout:starting mds.d at 2024-03-04T09:26:18.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.743+0000 7f26e2200640 1 Processor -- start 2024-03-04T09:26:18.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.743+0000 7f26e2200640 1 -- start start 2024-03-04T09:26:18.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.745+0000 7f26e2200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f26dc15dee0 0x7f26dc0d18a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:18.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.745+0000 7f26e2200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f26dc0d4860 0x7f26dc0d0630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:18.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.745+0000 7f26e2200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26dc0ceca0 con 0x7f26dc0cfa60 2024-03-04T09:26:18.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.745+0000 7f26e2200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26dc0cf5d0 con 0x7f26dc0d4860 2024-03-04T09:26:18.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.745+0000 7f26e2200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26dc0cef80 con 0x7f26dc15dee0 2024-03-04T09:26:18.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.746+0000 7f26e0e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f26dc15dee0 0x7f26dc0d18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:18.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.746+0000 7f26e1800640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f26dc0cfa60 0x7f26dc0b3510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.6:38342/0 (socket says 172.21.15.6:38342) 2024-03-04T09:26:18.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.746+0000 7f26e1800640 1 -- 172.21.15.6:0/1180987078 learned_addr learned my addr 172.21.15.6:0/1180987078 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:26:18.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.746+0000 7f26dbe00640 1 --2- 172.21.15.6:0/1180987078 >> v2:172.21.15.146:3300/0 conn(0x7f26dc0d4860 0x7f26dc0d0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:18.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.747+0000 7f26e0e00640 1 -- 172.21.15.6:0/1180987078 >> v2:172.21.15.146:3300/0 conn(0x7f26dc0d4860 msgr2=0x7f26dc0d0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:18.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.747+0000 7f26e0e00640 1 --2- 172.21.15.6:0/1180987078 >> v2:172.21.15.146:3300/0 conn(0x7f26dc0d4860 0x7f26dc0d0630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:18.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.747+0000 7f26e0e00640 1 -- 172.21.15.6:0/1180987078 >> v1:172.21.15.146:6789/0 conn(0x7f26dc0cfa60 legacy=0x7f26dc0b3510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:18.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.747+0000 7f26e0e00640 1 -- 172.21.15.6:0/1180987078 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26dc0cf260 con 0x7f26dc15dee0 2024-03-04T09:26:18.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.747+0000 7f26db400640 1 -- 172.21.15.6:0/1180987078 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1882905685 0 0) 0x7f26dc0ceca0 con 0x7f26dc0cfa60 2024-03-04T09:26:18.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.749+0000 7f26e0e00640 1 --2- 172.21.15.6:0/1180987078 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f26dc15dee0 0x7f26dc0d18a0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f26d00097a0 tx=0x7f26d002b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f76def25472b2147 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:18.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.749+0000 7f26db400640 1 -- 172.21.15.6:0/1180987078 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26d0035020 con 0x7f26dc15dee0 2024-03-04T09:26:18.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.749+0000 7f26db400640 1 -- 172.21.15.6:0/1180987078 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26d0031830 con 0x7f26dc15dee0 2024-03-04T09:26:18.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.749+0000 7f26db400640 1 -- 172.21.15.6:0/1180987078 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26d0031c00 con 0x7f26dc15dee0 2024-03-04T09:26:18.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.750+0000 7f26e2200640 1 -- 172.21.15.6:0/1180987078 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f26dc15dee0 msgr2=0x7f26dc0d18a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:18.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.750+0000 7f26e2200640 1 --2- 172.21.15.6:0/1180987078 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f26dc15dee0 0x7f26dc0d18a0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f26d00097a0 tx=0x7f26d002b8d0 comp rx=0 tx=0).stop 2024-03-04T09:26:18.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.750+0000 7f26e2200640 1 -- 172.21.15.6:0/1180987078 shutdown_connections 2024-03-04T09:26:18.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.750+0000 7f26e2200640 1 --2- 172.21.15.6:0/1180987078 >> v2:172.21.15.146:3300/0 conn(0x7f26dc0d4860 0x7f26dc0d0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:18.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.750+0000 7f26e2200640 1 --2- 172.21.15.6:0/1180987078 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f26dc15dee0 0x7f26dc0d18a0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:18.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.750+0000 7f26e2200640 1 -- 172.21.15.6:0/1180987078 >> 172.21.15.6:0/1180987078 conn(0x7f26dc0d3cd0 msgr2=0x7f26dc0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:18.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.750+0000 7f26e2200640 1 -- 172.21.15.6:0/1180987078 shutdown_connections 2024-03-04T09:26:18.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.750+0000 7f26e2200640 1 -- 172.21.15.6:0/1180987078 wait complete. 2024-03-04T09:26:18.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.752+0000 7f26e2200640 1 Processor -- start 2024-03-04T09:26:18.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.753+0000 7f26e2200640 1 -- start start 2024-03-04T09:26:18.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.754+0000 7f26e2200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f26dc0cfa60 0x7f26dc1e71d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:18.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.754+0000 7f26e2200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f26dc0d4860 0x7f26dc149b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:18.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.754+0000 7f26e2200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f26dc15dee0 0x7f26dc0d08c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:18.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.754+0000 7f26e2200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26dc0d1de0 con 0x7f26dc0cfa60 2024-03-04T09:26:18.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.754+0000 7f26e2200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26dc15e2c0 con 0x7f26dc0d4860 2024-03-04T09:26:18.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.754+0000 7f26e2200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26dc0d1f60 con 0x7f26dc15dee0 2024-03-04T09:26:18.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.754+0000 7f26e0e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f26dc0cfa60 0x7f26dc1e71d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:18.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.754+0000 7f26e0e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f26dc0cfa60 0x7f26dc1e71d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60006/0 (socket says 172.21.15.6:60006) 2024-03-04T09:26:18.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.754+0000 7f26e0e00640 1 -- 172.21.15.6:0/148498921 learned_addr learned my addr 172.21.15.6:0/148498921 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:18.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.755+0000 7f26e1800640 1 --2- 172.21.15.6:0/148498921 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f26dc15dee0 0x7f26dc0d08c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:18.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.755+0000 7f26dbe00640 1 --2- 172.21.15.6:0/148498921 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f26dc0d4860 0x7f26dc149b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:18.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.755+0000 7f26e0e00640 1 -- 172.21.15.6:0/148498921 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f26dc15dee0 msgr2=0x7f26dc0d08c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:18.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.755+0000 7f26e0e00640 1 --2- 172.21.15.6:0/148498921 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f26dc15dee0 0x7f26dc0d08c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:18.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.755+0000 7f26e0e00640 1 -- 172.21.15.6:0/148498921 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f26dc0d4860 msgr2=0x7f26dc149b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:18.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.755+0000 7f26e0e00640 1 --2- 172.21.15.6:0/148498921 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f26dc0d4860 0x7f26dc149b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:18.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.755+0000 7f26e0e00640 1 -- 172.21.15.6:0/148498921 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26dc0cbf50 con 0x7f26dc0cfa60 2024-03-04T09:26:18.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.755+0000 7f26dbe00640 1 --2- 172.21.15.6:0/148498921 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f26dc0d4860 0x7f26dc149b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:26:18.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.755+0000 7f26e1800640 1 --2- 172.21.15.6:0/148498921 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f26dc15dee0 0x7f26dc0d08c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:26:18.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.755+0000 7f26e0e00640 1 --2- 172.21.15.6:0/148498921 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f26dc0cfa60 0x7f26dc1e71d0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f26d0000c00 tx=0x7f26d00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:18.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.755+0000 7f26d8c00640 1 -- 172.21.15.6:0/148498921 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26d0040070 con 0x7f26dc0cfa60 2024-03-04T09:26:18.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.755+0000 7f26d8c00640 1 -- 172.21.15.6:0/148498921 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26d003a910 con 0x7f26dc0cfa60 2024-03-04T09:26:18.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.756+0000 7f26d8c00640 1 -- 172.21.15.6:0/148498921 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26d0035040 con 0x7f26dc0cfa60 2024-03-04T09:26:18.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.756+0000 7f26e2200640 1 -- 172.21.15.6:0/148498921 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26dc0cc230 con 0x7f26dc0cfa60 2024-03-04T09:26:18.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.756+0000 7f26e2200640 1 -- 172.21.15.6:0/148498921 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26dc0cc7f0 con 0x7f26dc0cfa60 2024-03-04T09:26:18.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.757+0000 7f26e2200640 1 -- 172.21.15.6:0/148498921 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26dc04f330 con 0x7f26dc0cfa60 2024-03-04T09:26:18.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.758+0000 7f26d8c00640 1 -- 172.21.15.6:0/148498921 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f26d004b060 con 0x7f26dc0cfa60 2024-03-04T09:26:18.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.758+0000 7f26d8c00640 1 --2- 172.21.15.6:0/148498921 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f26bc06b4d0 0x7f26bc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:18.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.759+0000 7f26d8c00640 1 -- 172.21.15.6:0/148498921 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5940+0+0 (secure 0 0 0) 0x7f26d00abaf0 con 0x7f26dc0cfa60 2024-03-04T09:26:18.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.759+0000 7f26dbe00640 1 --2- 172.21.15.6:0/148498921 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f26bc06b4d0 0x7f26bc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:18.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.760+0000 7f26dbe00640 1 --2- 172.21.15.6:0/148498921 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f26bc06b4d0 0x7f26bc06d990 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f26dc0c7140 tx=0x7f26c8008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:18.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:18.764+0000 7f26d8c00640 1 -- 172.21.15.6:0/148498921 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f26d007a910 con 0x7f26dc0cfa60 2024-03-04T09:26:19.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:19.005+0000 7f26e2200640 1 -- 172.21.15.6:0/148498921 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f26dc08d7d0 con 0x7f26dc0cfa60 2024-03-04T09:26:19.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:19.506+0000 7f26d8c00640 1 -- 172.21.15.6:0/148498921 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v9) v1 ==== 104+0+0 (secure 0 0 0) 0x7f26d007a210 con 0x7f26dc0cfa60 2024-03-04T09:26:19.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:19.510+0000 7f26e2200640 1 -- 172.21.15.6:0/148498921 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f26bc06b4d0 msgr2=0x7f26bc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:19.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:19.510+0000 7f26e2200640 1 --2- 172.21.15.6:0/148498921 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f26bc06b4d0 0x7f26bc06d990 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f26dc0c7140 tx=0x7f26c8008040 comp rx=0 tx=0).stop 2024-03-04T09:26:19.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:19.510+0000 7f26e2200640 1 -- 172.21.15.6:0/148498921 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f26dc0cfa60 msgr2=0x7f26dc1e71d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:19.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:19.510+0000 7f26e2200640 1 --2- 172.21.15.6:0/148498921 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f26dc0cfa60 0x7f26dc1e71d0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f26d0000c00 tx=0x7f26d00046c0 comp rx=0 tx=0).stop 2024-03-04T09:26:19.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:19.510+0000 7f26e2200640 1 -- 172.21.15.6:0/148498921 shutdown_connections 2024-03-04T09:26:19.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:19.510+0000 7f26e2200640 1 --2- 172.21.15.6:0/148498921 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f26bc06b4d0 0x7f26bc06d990 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:19.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:19.510+0000 7f26e2200640 1 --2- 172.21.15.6:0/148498921 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f26dc15dee0 0x7f26dc0d08c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:19.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:19.510+0000 7f26e2200640 1 --2- 172.21.15.6:0/148498921 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f26dc0d4860 0x7f26dc149b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:19.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:19.510+0000 7f26e2200640 1 --2- 172.21.15.6:0/148498921 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f26dc0cfa60 0x7f26dc1e71d0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:19.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:19.510+0000 7f26e2200640 1 -- 172.21.15.6:0/148498921 >> 172.21.15.6:0/148498921 conn(0x7f26dc0d3cd0 msgr2=0x7f26dc0ca630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:19.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:19.510+0000 7f26e2200640 1 -- 172.21.15.6:0/148498921 shutdown_connections 2024-03-04T09:26:19.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:19.510+0000 7f26e2200640 1 -- 172.21.15.6:0/148498921 wait complete. 2024-03-04T09:26:19.527 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-04T09:26:19.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:19.752+0000 7fc85a000640 1 Processor -- start 2024-03-04T09:26:19.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:19.752+0000 7fc85a000640 1 -- start start 2024-03-04T09:26:19.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:19.753+0000 7fc85a000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc854159730 0x7fc85415cc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:19.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:19.753+0000 7fc85a000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc854159b10 con 0x7fc8540ceea0 2024-03-04T09:26:19.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:19.753+0000 7fc85a000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc854071690 con 0x7fc8540ccc20 2024-03-04T09:26:19.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:19.753+0000 7fc85a000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc854072b60 con 0x7fc854159730 2024-03-04T09:26:19.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:19.754+0000 7fc858c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc854159730 0x7fc85415cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:19.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:19.754+0000 7fc858c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc854159730 0x7fc85415cc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45630/0 (socket says 172.21.15.6:45630) 2024-03-04T09:26:19.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:19.754+0000 7fc858c00640 1 -- 172.21.15.6:0/2391896890 learned_addr learned my addr 172.21.15.6:0/2391896890 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:19.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:19.754+0000 7fc858c00640 1 -- 172.21.15.6:0/2391896890 >> v1:172.21.15.146:6789/0 conn(0x7fc8540ccc20 legacy=0x7fc85406c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:19.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:19.754+0000 7fc858c00640 1 -- 172.21.15.6:0/2391896890 >> v1:172.21.15.6:6789/0 conn(0x7fc8540ceea0 legacy=0x7fc854072450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:19.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:19.754+0000 7fc858c00640 1 -- 172.21.15.6:0/2391896890 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc854071970 con 0x7fc854159730 2024-03-04T09:26:19.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:19.756+0000 7fc858c00640 1 --2- 172.21.15.6:0/2391896890 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc854159730 0x7fc85415cc20 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fc848009870 tx=0x7fc84802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fb79baaffbddf64 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:19.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:19.756+0000 7fc853400640 1 -- 172.21.15.6:0/2391896890 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc848034070 con 0x7fc854159730 2024-03-04T09:26:19.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:19.756+0000 7fc853400640 1 -- 172.21.15.6:0/2391896890 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc848030860 con 0x7fc854159730 2024-03-04T09:26:19.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:19.756+0000 7fc853400640 1 -- 172.21.15.6:0/2391896890 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc848030c30 con 0x7fc854159730 2024-03-04T09:26:19.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:19.756+0000 7fc85a000640 1 -- 172.21.15.6:0/2391896890 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc854159730 msgr2=0x7fc85415cc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:19.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:19.756+0000 7fc85a000640 1 --2- 172.21.15.6:0/2391896890 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc854159730 0x7fc85415cc20 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fc848009870 tx=0x7fc84802b9a0 comp rx=0 tx=0).stop 2024-03-04T09:26:19.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:19.757+0000 7fc85a000640 1 -- 172.21.15.6:0/2391896890 shutdown_connections 2024-03-04T09:26:19.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:19.757+0000 7fc85a000640 1 --2- 172.21.15.6:0/2391896890 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc854159730 0x7fc85415cc20 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:19.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:19.757+0000 7fc85a000640 1 -- 172.21.15.6:0/2391896890 >> 172.21.15.6:0/2391896890 conn(0x7fc85406afb0 msgr2=0x7fc85406b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:19.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:19.757+0000 7fc85a000640 1 -- 172.21.15.6:0/2391896890 shutdown_connections 2024-03-04T09:26:19.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:19.757+0000 7fc85a000640 1 -- 172.21.15.6:0/2391896890 wait complete. 2024-03-04T09:26:19.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:19.758+0000 7fc85a000640 1 Processor -- start 2024-03-04T09:26:19.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:19.758+0000 7fc85a000640 1 -- start start 2024-03-04T09:26:19.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:19.759+0000 7fc85a000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc8540ccc20 0x7fc854143190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:19.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:19.759+0000 7fc85a000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc8540ceea0 0x7fc854153210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:19.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:19.759+0000 7fc85a000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc854159730 0x7fc8540b0610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:19.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:19.759+0000 7fc85a000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8540cf6d0 con 0x7fc854159730 2024-03-04T09:26:19.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:19.759+0000 7fc85a000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc854071f30 con 0x7fc8540ccc20 2024-03-04T09:26:19.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:19.759+0000 7fc85a000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8540cf330 con 0x7fc8540ceea0 2024-03-04T09:26:19.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:19.760+0000 7fc859600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc854159730 0x7fc8540b0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:19.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:19.760+0000 7fc859600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc854159730 0x7fc8540b0610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45634/0 (socket says 172.21.15.6:45634) 2024-03-04T09:26:19.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:19.760+0000 7fc859600640 1 -- 172.21.15.6:0/1390009130 learned_addr learned my addr 172.21.15.6:0/1390009130 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:19.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:19.760+0000 7fc858c00640 1 --2- 172.21.15.6:0/1390009130 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc8540ccc20 0x7fc854143190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:19.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:19.760+0000 7fc853e00640 1 --2- 172.21.15.6:0/1390009130 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc8540ceea0 0x7fc854153210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:19.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:19.760+0000 7fc859600640 1 -- 172.21.15.6:0/1390009130 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc8540ceea0 msgr2=0x7fc854153210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:19.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:19.760+0000 7fc859600640 1 --2- 172.21.15.6:0/1390009130 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc8540ceea0 0x7fc854153210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:19.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:19.760+0000 7fc859600640 1 -- 172.21.15.6:0/1390009130 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc8540ccc20 msgr2=0x7fc854143190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:19.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:19.760+0000 7fc859600640 1 --2- 172.21.15.6:0/1390009130 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc8540ccc20 0x7fc854143190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:19.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:19.760+0000 7fc859600640 1 -- 172.21.15.6:0/1390009130 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc854152170 con 0x7fc854159730 2024-03-04T09:26:19.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:19.761+0000 7fc859600640 1 --2- 172.21.15.6:0/1390009130 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc854159730 0x7fc8540b0610 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fc84c00c780 tx=0x7fc84c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:19.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:19.761+0000 7fc850c00640 1 -- 172.21.15.6:0/1390009130 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc84c017070 con 0x7fc854159730 2024-03-04T09:26:19.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:19.761+0000 7fc850c00640 1 -- 172.21.15.6:0/1390009130 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc84c003da0 con 0x7fc854159730 2024-03-04T09:26:19.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:19.761+0000 7fc850c00640 1 -- 172.21.15.6:0/1390009130 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc84c0052b0 con 0x7fc854159730 2024-03-04T09:26:19.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:19.761+0000 7fc85a000640 1 -- 172.21.15.6:0/1390009130 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc854152450 con 0x7fc854159730 2024-03-04T09:26:19.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:19.761+0000 7fc85a000640 1 -- 172.21.15.6:0/1390009130 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc854152cf0 con 0x7fc854159730 2024-03-04T09:26:19.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:19.762+0000 7fc85a000640 1 -- 172.21.15.6:0/1390009130 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc854152620 con 0x7fc854159730 2024-03-04T09:26:19.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:19.763+0000 7fc850c00640 1 -- 172.21.15.6:0/1390009130 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc84c01a070 con 0x7fc854159730 2024-03-04T09:26:19.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:19.764+0000 7fc850c00640 1 --2- 172.21.15.6:0/1390009130 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc83406b4d0 0x7fc83406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:19.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:19.764+0000 7fc850c00640 1 -- 172.21.15.6:0/1390009130 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5940+0+0 (secure 0 0 0) 0x7fc84c0898d0 con 0x7fc854159730 2024-03-04T09:26:19.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:19.765+0000 7fc858c00640 1 --2- 172.21.15.6:0/1390009130 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc83406b4d0 0x7fc83406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:19.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:19.766+0000 7fc858c00640 1 --2- 172.21.15.6:0/1390009130 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc83406b4d0 0x7fc83406d990 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fc848036000 tx=0x7fc84803b040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:19.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:19.769+0000 7fc850c00640 1 -- 172.21.15.6:0/1390009130 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc84c0586f0 con 0x7fc854159730 2024-03-04T09:26:19.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:19.966+0000 7fc85a000640 1 -- 172.21.15.6:0/1390009130 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fc85404f330 con 0x7fc854159730 2024-03-04T09:26:20.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:20.879+0000 7fc850c00640 1 -- 172.21.15.6:0/1390009130 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v24) v1 ==== 181+0+0 (secure 0 0 0) 0x7fc84c057ff0 con 0x7fc854159730 2024-03-04T09:26:20.876 INFO:teuthology.orchestra.run.smithi006.stderr:pool 'cephfs_metadata' removed 2024-03-04T09:26:20.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:20.883+0000 7fc85a000640 1 -- 172.21.15.6:0/1390009130 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc83406b4d0 msgr2=0x7fc83406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:20.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:20.883+0000 7fc85a000640 1 --2- 172.21.15.6:0/1390009130 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc83406b4d0 0x7fc83406d990 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fc848036000 tx=0x7fc84803b040 comp rx=0 tx=0).stop 2024-03-04T09:26:20.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:20.883+0000 7fc85a000640 1 -- 172.21.15.6:0/1390009130 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc854159730 msgr2=0x7fc8540b0610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:20.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:20.883+0000 7fc85a000640 1 --2- 172.21.15.6:0/1390009130 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc854159730 0x7fc8540b0610 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fc84c00c780 tx=0x7fc84c00cc40 comp rx=0 tx=0).stop 2024-03-04T09:26:20.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:20.883+0000 7fc85a000640 1 -- 172.21.15.6:0/1390009130 shutdown_connections 2024-03-04T09:26:20.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:20.883+0000 7fc85a000640 1 --2- 172.21.15.6:0/1390009130 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc854159730 0x7fc8540b0610 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:20.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:20.883+0000 7fc85a000640 1 --2- 172.21.15.6:0/1390009130 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc83406b4d0 0x7fc83406d990 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:20.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:20.883+0000 7fc85a000640 1 --2- 172.21.15.6:0/1390009130 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc8540ceea0 0x7fc854153210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:20.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:20.883+0000 7fc85a000640 1 --2- 172.21.15.6:0/1390009130 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc8540ccc20 0x7fc854143190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:20.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:20.883+0000 7fc85a000640 1 -- 172.21.15.6:0/1390009130 >> 172.21.15.6:0/1390009130 conn(0x7fc85406afb0 msgr2=0x7fc85415b100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:20.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:20.883+0000 7fc85a000640 1 -- 172.21.15.6:0/1390009130 shutdown_connections 2024-03-04T09:26:20.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:20.884+0000 7fc85a000640 1 -- 172.21.15.6:0/1390009130 wait complete. 2024-03-04T09:26:20.906 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-04T09:26:21.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:21.108+0000 7f228f200640 1 Processor -- start 2024-03-04T09:26:21.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:21.109+0000 7f228f200640 1 -- start start 2024-03-04T09:26:21.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:21.110+0000 7f228f200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f2290145570 0x7f22900cf420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:21.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:21.110+0000 7f228f200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f229006e200 0x7f229015c300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:21.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:21.110+0000 7f228f200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2290071c50 con 0x7f2290071450 2024-03-04T09:26:21.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:21.110+0000 7f228f200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2290071f30 con 0x7f2290145570 2024-03-04T09:26:21.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:21.110+0000 7f228f200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2290072210 con 0x7f229006e200 2024-03-04T09:26:21.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:21.111+0000 7f228d400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f2290071450 0x7f229006daf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:54576/0 (socket says 172.21.15.6:54576) 2024-03-04T09:26:21.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:21.111+0000 7f228d400640 1 -- 172.21.15.6:0/524746385 learned_addr learned my addr 172.21.15.6:0/524746385 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:26:21.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:21.111+0000 7f228de00640 1 --2- 172.21.15.6:0/524746385 >> v2:172.21.15.146:3300/0 conn(0x7f2290145570 0x7f22900cf420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:21.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:21.111+0000 7f228e800640 1 --2- 172.21.15.6:0/524746385 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f229006e200 0x7f229015c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:21.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:21.111+0000 7f228ca00640 1 -- 172.21.15.6:0/524746385 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 397435606 0 0) 0x7f2290071c50 con 0x7f2290071450 2024-03-04T09:26:21.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:21.111+0000 7f228ca00640 1 -- 172.21.15.6:0/524746385 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2274003520 con 0x7f2290071450 2024-03-04T09:26:21.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:21.112+0000 7f228ca00640 1 -- 172.21.15.6:0/524746385 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3389580425 0 0) 0x7f2274003520 con 0x7f2290071450 2024-03-04T09:26:21.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:21.112+0000 7f228ca00640 1 -- 172.21.15.6:0/524746385 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f229006e200 msgr2=0x7f229015c300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:21.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:21.112+0000 7f228ca00640 1 --2- 172.21.15.6:0/524746385 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f229006e200 0x7f229015c300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:21.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:21.112+0000 7f228ca00640 1 -- 172.21.15.6:0/524746385 >> v2:172.21.15.146:3300/0 conn(0x7f2290145570 msgr2=0x7f22900cf420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:21.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:21.112+0000 7f228ca00640 1 --2- 172.21.15.6:0/524746385 >> v2:172.21.15.146:3300/0 conn(0x7f2290145570 0x7f22900cf420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:21.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:21.112+0000 7f228ca00640 1 -- 172.21.15.6:0/524746385 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22900724f0 con 0x7f2290071450 2024-03-04T09:26:21.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:21.112+0000 7f228ca00640 1 -- 172.21.15.6:0/524746385 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7f227c001670 con 0x7f2290071450 2024-03-04T09:26:21.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:21.112+0000 7f228ca00640 1 -- 172.21.15.6:0/524746385 >> v1:172.21.15.6:6789/0 conn(0x7f2290071450 legacy=0x7f229006daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:21.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:21.112+0000 7f228ca00640 1 --2- 172.21.15.6:0/524746385 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f229006e200 0x7f22740039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:21.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:21.112+0000 7f228ca00640 1 --2- 172.21.15.6:0/524746385 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2274003f80 0x7f2274004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:21.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:21.112+0000 7f228ca00640 1 --2- 172.21.15.6:0/524746385 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2274004d40 0x7f22740051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:21.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:21.112+0000 7f228ca00640 1 -- 172.21.15.6:0/524746385 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2290071f30 con 0x7f2274003f80 2024-03-04T09:26:21.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:21.112+0000 7f228ca00640 1 -- 172.21.15.6:0/524746385 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2290072210 con 0x7f2274004d40 2024-03-04T09:26:21.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:21.112+0000 7f228ca00640 1 -- 172.21.15.6:0/524746385 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f227400ff30 con 0x7f229006e200 2024-03-04T09:26:21.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:21.112+0000 7f228de00640 1 --2- 172.21.15.6:0/524746385 >> v2:172.21.15.146:3300/0 conn(0x7f2290145570 0x7f22900cf420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T09:26:21.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:21.112+0000 7f228d400640 1 --2- 172.21.15.6:0/524746385 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2274003f80 0x7f2274004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:21.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:21.113+0000 7f228d400640 1 -- 172.21.15.6:0/524746385 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f229006e200 msgr2=0x7f22740039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:21.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:21.113+0000 7f228d400640 1 --2- 172.21.15.6:0/524746385 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f229006e200 0x7f22740039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:21.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:21.113+0000 7f228d400640 1 -- 172.21.15.6:0/524746385 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2274004d40 msgr2=0x7f22740051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:21.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:21.113+0000 7f228d400640 1 --2- 172.21.15.6:0/524746385 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2274004d40 0x7f22740051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:21.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:21.113+0000 7f228d400640 1 -- 172.21.15.6:0/524746385 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2274003520 con 0x7f2274003f80 2024-03-04T09:26:21.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:21.114+0000 7f228d400640 1 --2- 172.21.15.6:0/524746385 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2274003f80 0x7f2274004800 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f227c00c160 tx=0x7f2274005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc4178e30bcc7c9f server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:21.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:21.114+0000 7f228ca00640 1 -- 172.21.15.6:0/524746385 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f227c007c80 con 0x7f2274003f80 2024-03-04T09:26:21.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:21.114+0000 7f228ca00640 1 -- 172.21.15.6:0/524746385 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f227c030070 con 0x7f2274003f80 2024-03-04T09:26:21.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:21.114+0000 7f228ca00640 1 -- 172.21.15.6:0/524746385 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f227c005d70 con 0x7f2274003f80 2024-03-04T09:26:21.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:21.114+0000 7f228f200640 1 -- 172.21.15.6:0/524746385 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2274003f80 msgr2=0x7f2274004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:21.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:21.114+0000 7f228f200640 1 --2- 172.21.15.6:0/524746385 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2274003f80 0x7f2274004800 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f227c00c160 tx=0x7f2274005830 comp rx=0 tx=0).stop 2024-03-04T09:26:21.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:21.114+0000 7f228e800640 1 -- 172.21.15.6:0/524746385 reap_dead start 2024-03-04T09:26:21.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:21.114+0000 7f228f200640 1 -- 172.21.15.6:0/524746385 shutdown_connections 2024-03-04T09:26:21.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:21.114+0000 7f228f200640 1 -- 172.21.15.6:0/524746385 >> 172.21.15.6:0/524746385 conn(0x7f22900b17c0 msgr2=0x7f22900b1ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:21.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:21.115+0000 7f228f200640 1 -- 172.21.15.6:0/524746385 shutdown_connections 2024-03-04T09:26:21.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:21.115+0000 7f228f200640 1 -- 172.21.15.6:0/524746385 wait complete. 2024-03-04T09:26:21.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:21.116+0000 7f228f200640 1 Processor -- start 2024-03-04T09:26:21.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:21.116+0000 7f228f200640 1 -- start start 2024-03-04T09:26:21.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:21.117+0000 7f228f200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2274003f80 0x7f22901419b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:21.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:21.117+0000 7f228f200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f22900a9470 0x7f229013ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:21.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:21.117+0000 7f228f200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f22900b28b0 0x7f22900b2d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:21.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:21.117+0000 7f228f200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2290071a30 con 0x7f2274003f80 2024-03-04T09:26:21.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:21.117+0000 7f228f200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22900718b0 con 0x7f22900a9470 2024-03-04T09:26:21.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:21.117+0000 7f228f200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f229015c300 con 0x7f22900b28b0 2024-03-04T09:26:21.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:21.118+0000 7f228de00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2274003f80 0x7f22901419b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:21.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:21.118+0000 7f228de00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2274003f80 0x7f22901419b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45660/0 (socket says 172.21.15.6:45660) 2024-03-04T09:26:21.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:21.118+0000 7f228de00640 1 -- 172.21.15.6:0/3190591109 learned_addr learned my addr 172.21.15.6:0/3190591109 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:21.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:21.118+0000 7f228d400640 1 --2- 172.21.15.6:0/3190591109 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f22900a9470 0x7f229013ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:21.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:21.118+0000 7f228de00640 1 -- 172.21.15.6:0/3190591109 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f22900b28b0 msgr2=0x7f22900b2d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:21.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:21.118+0000 7f228de00640 1 --2- 172.21.15.6:0/3190591109 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f22900b28b0 0x7f22900b2d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:21.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:21.118+0000 7f228de00640 1 -- 172.21.15.6:0/3190591109 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f22900a9470 msgr2=0x7f229013ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:21.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:21.118+0000 7f228de00640 1 --2- 172.21.15.6:0/3190591109 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f22900a9470 0x7f229013ebc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:21.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:21.118+0000 7f228de00640 1 -- 172.21.15.6:0/3190591109 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22900a9b50 con 0x7f2274003f80 2024-03-04T09:26:21.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:21.118+0000 7f228de00640 1 --2- 172.21.15.6:0/3190591109 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2274003f80 0x7f22901419b0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f228000e600 tx=0x7f228000eac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:21.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:21.119+0000 7f2286000640 1 -- 172.21.15.6:0/3190591109 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f228000f040 con 0x7f2274003f80 2024-03-04T09:26:21.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:21.119+0000 7f2286000640 1 -- 172.21.15.6:0/3190591109 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2280004230 con 0x7f2274003f80 2024-03-04T09:26:21.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:21.119+0000 7f228f200640 1 -- 172.21.15.6:0/3190591109 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22900a9e30 con 0x7f2274003f80 2024-03-04T09:26:21.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:21.119+0000 7f2286000640 1 -- 172.21.15.6:0/3190591109 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2280003610 con 0x7f2274003f80 2024-03-04T09:26:21.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:21.119+0000 7f228f200640 1 -- 172.21.15.6:0/3190591109 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22900aa3f0 con 0x7f2274003f80 2024-03-04T09:26:21.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:21.120+0000 7f228f200640 1 -- 172.21.15.6:0/3190591109 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f229004f330 con 0x7f2274003f80 2024-03-04T09:26:21.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:21.121+0000 7f2286000640 1 -- 172.21.15.6:0/3190591109 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f22800037b0 con 0x7f2274003f80 2024-03-04T09:26:21.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:21.122+0000 7f2286000640 1 --2- 172.21.15.6:0/3190591109 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f227006b4d0 0x7f227006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:21.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:21.122+0000 7f2286000640 1 -- 172.21.15.6:0/3190591109 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5502+0+0 (secure 0 0 0) 0x7f2280089e30 con 0x7f2274003f80 2024-03-04T09:26:21.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:21.123+0000 7f228d400640 1 --2- 172.21.15.6:0/3190591109 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f227006b4d0 0x7f227006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:21.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:21.124+0000 7f228d400640 1 --2- 172.21.15.6:0/3190591109 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f227006b4d0 0x7f227006d990 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f227c034000 tx=0x7f227c007000 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:21.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:21.126+0000 7f2286000640 1 -- 172.21.15.6:0/3190591109 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2280017020 con 0x7f2274003f80 2024-03-04T09:26:21.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:21.334+0000 7f228f200640 1 -- 172.21.15.6:0/3190591109 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f22900cbfc0 con 0x7f2274003f80 2024-03-04T09:26:21.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:21.882+0000 7f2286000640 1 -- 172.21.15.6:0/3190591109 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v25) v1 ==== 169+0+0 (secure 0 0 0) 0x7f2280058ce0 con 0x7f2274003f80 2024-03-04T09:26:21.880 INFO:teuthology.orchestra.run.smithi006.stderr:pool 'cephfs_data' removed 2024-03-04T09:26:21.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:21.886+0000 7f228f200640 1 -- 172.21.15.6:0/3190591109 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f227006b4d0 msgr2=0x7f227006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:21.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:21.886+0000 7f228f200640 1 --2- 172.21.15.6:0/3190591109 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f227006b4d0 0x7f227006d990 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f227c034000 tx=0x7f227c007000 comp rx=0 tx=0).stop 2024-03-04T09:26:21.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:21.886+0000 7f228f200640 1 -- 172.21.15.6:0/3190591109 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2274003f80 msgr2=0x7f22901419b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:21.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:21.886+0000 7f228f200640 1 --2- 172.21.15.6:0/3190591109 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2274003f80 0x7f22901419b0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f228000e600 tx=0x7f228000eac0 comp rx=0 tx=0).stop 2024-03-04T09:26:21.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:21.887+0000 7f228f200640 1 -- 172.21.15.6:0/3190591109 shutdown_connections 2024-03-04T09:26:21.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:21.887+0000 7f228f200640 1 --2- 172.21.15.6:0/3190591109 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f227006b4d0 0x7f227006d990 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:21.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:21.887+0000 7f228f200640 1 --2- 172.21.15.6:0/3190591109 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f22900b28b0 0x7f22900b2d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:21.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:21.887+0000 7f228f200640 1 --2- 172.21.15.6:0/3190591109 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f22900a9470 0x7f229013ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:21.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:21.887+0000 7f228f200640 1 --2- 172.21.15.6:0/3190591109 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2274003f80 0x7f22901419b0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:21.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:21.887+0000 7f228f200640 1 -- 172.21.15.6:0/3190591109 >> 172.21.15.6:0/3190591109 conn(0x7f22900b17c0 msgr2=0x7f22900b00d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:21.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:21.887+0000 7f228f200640 1 -- 172.21.15.6:0/3190591109 shutdown_connections 2024-03-04T09:26:21.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:21.887+0000 7f228f200640 1 -- 172.21.15.6:0/3190591109 wait complete. 2024-03-04T09:26:21.911 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-04T09:26:21.911 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-04T09:26:21.911 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-04T09:26:21.911 INFO:teuthology.orchestra.run:waiting for 300 2024-03-04T09:26:21.911 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-04T09:26:21.911 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-04T09:26:21.912 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-04T09:26:21.912 INFO:teuthology.orchestra.run:waiting for 300 2024-03-04T09:26:21.912 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-04T09:26:21.912 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-04T09:26:21.912 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-04T09:26:21.912 INFO:teuthology.orchestra.run:waiting for 300 2024-03-04T09:26:21.912 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-04T09:26:21.913 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-04T09:26:21.913 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-04T09:26:21.913 INFO:teuthology.orchestra.run:waiting for 300 2024-03-04T09:26:21.931 INFO:tasks.ceph.mds.c:Stopped 2024-03-04T09:26:21.931 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-04T09:26:21.932 INFO:tasks.ceph.mds.a:Stopped 2024-03-04T09:26:21.932 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-04T09:26:21.937 INFO:tasks.ceph.mds.b:Stopped 2024-03-04T09:26:21.937 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-04T09:26:21.939 INFO:tasks.ceph.mds.d:Stopped 2024-03-04T09:26:21.939 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-04T09:26:21.978 INFO:tasks.ceph.mds.c:Started 2024-03-04T09:26:21.978 DEBUG:teuthology.parallel:result is None 2024-03-04T09:26:21.983 INFO:tasks.ceph.mds.a:Started 2024-03-04T09:26:21.984 DEBUG:teuthology.parallel:result is None 2024-03-04T09:26:21.984 INFO:tasks.ceph.mds.b:Started 2024-03-04T09:26:21.985 DEBUG:teuthology.parallel:result is None 2024-03-04T09:26:21.990 INFO:tasks.ceph.mds.d:Started 2024-03-04T09:26:21.990 DEBUG:teuthology.parallel:result is None 2024-03-04T09:26:21.990 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-04T09:26:22.135 INFO:tasks.ceph.mds.c.smithi006.stdout:starting mds.c at 2024-03-04T09:26:22.141 INFO:tasks.ceph.mds.a.smithi006.stdout:starting mds.a at 2024-03-04T09:26:22.142 INFO:tasks.ceph.mds.b.smithi146.stdout:starting mds.b at 2024-03-04T09:26:22.142 INFO:tasks.ceph.mds.d.smithi146.stdout:starting mds.d at 2024-03-04T09:26:22.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:22.186+0000 7f639f600640 1 Processor -- start 2024-03-04T09:26:22.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:22.186+0000 7f639f600640 1 -- start start 2024-03-04T09:26:22.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:22.187+0000 7f639f600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f63980af1a0 0x7f639815c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:22.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:22.187+0000 7f639f600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f63980af580 0x7f63980b4c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:22.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:22.187+0000 7f639f600640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f639815cc40 con 0x7f63980cdc20 2024-03-04T09:26:22.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:22.187+0000 7f639f600640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f639815cf20 con 0x7f63980af1a0 2024-03-04T09:26:22.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:22.187+0000 7f639f600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f639815d200 con 0x7f63980af580 2024-03-04T09:26:22.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:22.187+0000 7f639cc00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f63980af1a0 0x7f639815c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:22.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:22.187+0000 7f639cc00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f63980af1a0 0x7f639815c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45690/0 (socket says 172.21.15.6:45690) 2024-03-04T09:26:22.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:22.187+0000 7f639cc00640 1 -- 172.21.15.6:0/1547244136 learned_addr learned my addr 172.21.15.6:0/1547244136 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:22.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:22.188+0000 7f6397e00640 1 --2- 172.21.15.6:0/1547244136 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f63980af580 0x7f63980b4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:22.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:22.188+0000 7f639cc00640 1 -- 172.21.15.6:0/1547244136 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f63980af580 msgr2=0x7f63980b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:22.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:22.188+0000 7f639cc00640 1 --2- 172.21.15.6:0/1547244136 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f63980af580 0x7f63980b4c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:22.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:22.188+0000 7f639cc00640 1 -- 172.21.15.6:0/1547244136 >> v1:172.21.15.6:6789/0 conn(0x7f63980cdc20 legacy=0x7f63980b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:22.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:22.188+0000 7f639cc00640 1 -- 172.21.15.6:0/1547244136 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f639815d4e0 con 0x7f63980af1a0 2024-03-04T09:26:22.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:22.189+0000 7f639cc00640 1 --2- 172.21.15.6:0/1547244136 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f63980af1a0 0x7f639815c600 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f6388009870 tx=0x7f638802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=497b6b70e5a933a3 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:22.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:22.189+0000 7f6397400640 1 -- 172.21.15.6:0/1547244136 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6388035020 con 0x7f63980af1a0 2024-03-04T09:26:22.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:22.189+0000 7f6397400640 1 -- 172.21.15.6:0/1547244136 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6388031640 con 0x7f63980af1a0 2024-03-04T09:26:22.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:22.189+0000 7f6397400640 1 -- 172.21.15.6:0/1547244136 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f638802d040 con 0x7f63980af1a0 2024-03-04T09:26:22.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:22.190+0000 7f639f600640 1 -- 172.21.15.6:0/1547244136 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f63980af1a0 msgr2=0x7f639815c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:22.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:22.190+0000 7f639f600640 1 --2- 172.21.15.6:0/1547244136 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f63980af1a0 0x7f639815c600 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f6388009870 tx=0x7f638802b8f0 comp rx=0 tx=0).stop 2024-03-04T09:26:22.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:22.190+0000 7f639f600640 1 -- 172.21.15.6:0/1547244136 shutdown_connections 2024-03-04T09:26:22.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:22.190+0000 7f639f600640 1 --2- 172.21.15.6:0/1547244136 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f63980af580 0x7f63980b4c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:22.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:22.190+0000 7f639f600640 1 --2- 172.21.15.6:0/1547244136 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f63980af1a0 0x7f639815c600 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:22.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:22.190+0000 7f639f600640 1 -- 172.21.15.6:0/1547244136 >> 172.21.15.6:0/1547244136 conn(0x7f639806afb0 msgr2=0x7f639806b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:22.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:22.190+0000 7f639f600640 1 -- 172.21.15.6:0/1547244136 shutdown_connections 2024-03-04T09:26:22.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:22.190+0000 7f639f600640 1 -- 172.21.15.6:0/1547244136 wait complete. 2024-03-04T09:26:22.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:22.192+0000 7f639f600640 1 Processor -- start 2024-03-04T09:26:22.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:22.192+0000 7f639f600640 1 -- start start 2024-03-04T09:26:22.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:22.193+0000 7f639f600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f63980af1a0 0x7f6398156870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:22.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:22.193+0000 7f639f600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f63980af580 0x7f63981f8280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:22.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:22.193+0000 7f639f600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f63980cdc20 0x7f63981f87c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:22.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:22.193+0000 7f639f600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63980b3660 con 0x7f63980af580 2024-03-04T09:26:22.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:22.193+0000 7f639f600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63980b34e0 con 0x7f63980cdc20 2024-03-04T09:26:22.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:22.193+0000 7f639f600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63980b37e0 con 0x7f63980af1a0 2024-03-04T09:26:22.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:22.193+0000 7f6397e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f63980af580 0x7f63981f8280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:22.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:22.193+0000 7f6397e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f63980af580 0x7f63981f8280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45700/0 (socket says 172.21.15.6:45700) 2024-03-04T09:26:22.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:22.193+0000 7f6397e00640 1 -- 172.21.15.6:0/4023018921 learned_addr learned my addr 172.21.15.6:0/4023018921 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:22.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:22.193+0000 7f639cc00640 1 --2- 172.21.15.6:0/4023018921 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f63980af1a0 0x7f6398156870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:22.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:22.193+0000 7f639d600640 1 --2- 172.21.15.6:0/4023018921 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f63980cdc20 0x7f63981f87c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:22.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:22.194+0000 7f6397e00640 1 -- 172.21.15.6:0/4023018921 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f63980af1a0 msgr2=0x7f6398156870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:22.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:22.194+0000 7f6397e00640 1 --2- 172.21.15.6:0/4023018921 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f63980af1a0 0x7f6398156870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:22.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:22.194+0000 7f6397e00640 1 -- 172.21.15.6:0/4023018921 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f63980cdc20 msgr2=0x7f63981f87c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:22.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:22.194+0000 7f6397e00640 1 --2- 172.21.15.6:0/4023018921 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f63980cdc20 0x7f63981f87c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:22.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:22.194+0000 7f6397e00640 1 -- 172.21.15.6:0/4023018921 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f639806d5c0 con 0x7f63980af580 2024-03-04T09:26:22.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:22.194+0000 7f6397e00640 1 --2- 172.21.15.6:0/4023018921 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f63980af580 0x7f63981f8280 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f638400d830 tx=0x7f638400dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:22.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:22.194+0000 7f6394c00640 1 -- 172.21.15.6:0/4023018921 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f638400b6c0 con 0x7f63980af580 2024-03-04T09:26:22.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:22.194+0000 7f6394c00640 1 -- 172.21.15.6:0/4023018921 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6384019070 con 0x7f63980af580 2024-03-04T09:26:22.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:22.194+0000 7f6394c00640 1 -- 172.21.15.6:0/4023018921 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f638400ba80 con 0x7f63980af580 2024-03-04T09:26:22.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:22.194+0000 7f639f600640 1 -- 172.21.15.6:0/4023018921 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f639806d860 con 0x7f63980af580 2024-03-04T09:26:22.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:22.195+0000 7f639f600640 1 -- 172.21.15.6:0/4023018921 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6398157390 con 0x7f63980af580 2024-03-04T09:26:22.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:22.196+0000 7f639f600640 1 -- 172.21.15.6:0/4023018921 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6398069510 con 0x7f63980af580 2024-03-04T09:26:22.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:22.197+0000 7f6394c00640 1 -- 172.21.15.6:0/4023018921 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f638400bc20 con 0x7f63980af580 2024-03-04T09:26:22.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:22.198+0000 7f6394c00640 1 --2- 172.21.15.6:0/4023018921 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f637806b4d0 0x7f637806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:22.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:22.198+0000 7f6394c00640 1 -- 172.21.15.6:0/4023018921 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f638408a250 con 0x7f63980af580 2024-03-04T09:26:22.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:22.199+0000 7f639cc00640 1 --2- 172.21.15.6:0/4023018921 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f637806b4d0 0x7f637806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:22.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:22.199+0000 7f639cc00640 1 --2- 172.21.15.6:0/4023018921 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f637806b4d0 0x7f637806d990 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f6388036000 tx=0x7f638803c040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:22.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:22.203+0000 7f6394c00640 1 -- 172.21.15.6:0/4023018921 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6384058d70 con 0x7f63980af580 2024-03-04T09:26:22.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:22.408+0000 7f639f600640 1 -- 172.21.15.6:0/4023018921 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f63980b4f40 con 0x7f63980af580 2024-03-04T09:26:22.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:22.915+0000 7f6394c00640 1 -- 172.21.15.6:0/4023018921 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v28) v1 ==== 97+0+0 (secure 0 0 0) 0x7f638405ce10 con 0x7f63980af580 2024-03-04T09:26:22.912 INFO:teuthology.orchestra.run.smithi006.stderr: removed all blocklist entries 2024-03-04T09:26:22.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:22.919+0000 7f639f600640 1 -- 172.21.15.6:0/4023018921 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f637806b4d0 msgr2=0x7f637806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:22.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:22.919+0000 7f639f600640 1 --2- 172.21.15.6:0/4023018921 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f637806b4d0 0x7f637806d990 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f6388036000 tx=0x7f638803c040 comp rx=0 tx=0).stop 2024-03-04T09:26:22.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:22.919+0000 7f639f600640 1 -- 172.21.15.6:0/4023018921 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f63980af580 msgr2=0x7f63981f8280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:22.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:22.919+0000 7f639f600640 1 --2- 172.21.15.6:0/4023018921 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f63980af580 0x7f63981f8280 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f638400d830 tx=0x7f638400dcf0 comp rx=0 tx=0).stop 2024-03-04T09:26:22.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:22.919+0000 7f639f600640 1 -- 172.21.15.6:0/4023018921 shutdown_connections 2024-03-04T09:26:22.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:22.919+0000 7f639f600640 1 --2- 172.21.15.6:0/4023018921 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f63980cdc20 0x7f63981f87c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:22.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:22.919+0000 7f639f600640 1 --2- 172.21.15.6:0/4023018921 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f63980af580 0x7f63981f8280 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:22.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:22.919+0000 7f639f600640 1 --2- 172.21.15.6:0/4023018921 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f637806b4d0 0x7f637806d990 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:22.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:22.919+0000 7f639f600640 1 --2- 172.21.15.6:0/4023018921 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f63980af1a0 0x7f6398156870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:22.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:22.919+0000 7f639f600640 1 -- 172.21.15.6:0/4023018921 >> 172.21.15.6:0/4023018921 conn(0x7f639806afb0 msgr2=0x7f639815c0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:22.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:22.919+0000 7f639f600640 1 -- 172.21.15.6:0/4023018921 shutdown_connections 2024-03-04T09:26:22.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:22.920+0000 7f639f600640 1 -- 172.21.15.6:0/4023018921 wait complete. 2024-03-04T09:26:22.937 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-04T09:26:23.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.145+0000 7f691f200640 1 Processor -- start 2024-03-04T09:26:23.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.145+0000 7f691f200640 1 -- start start 2024-03-04T09:26:23.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.147+0000 7f691f200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f6920145300 0x7f69200b49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:23.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.147+0000 7f691f200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6920070f10 0x7f69200cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:23.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.147+0000 7f691f200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69200714d0 con 0x7f69200cce90 2024-03-04T09:26:23.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.147+0000 7f691f200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69200cd300 con 0x7f6920145300 2024-03-04T09:26:23.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.147+0000 7f691f200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69200717b0 con 0x7f6920070f10 2024-03-04T09:26:23.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.147+0000 7f691de00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f6920145300 0x7f69200b49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:23.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.147+0000 7f691de00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f6920145300 0x7f69200b49a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45714/0 (socket says 172.21.15.6:45714) 2024-03-04T09:26:23.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.147+0000 7f691de00640 1 -- 172.21.15.6:0/12275276 learned_addr learned my addr 172.21.15.6:0/12275276 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:23.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.148+0000 7f691d400640 1 --2- 172.21.15.6:0/12275276 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6920070f10 0x7f69200cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:23.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.148+0000 7f691de00640 1 -- 172.21.15.6:0/12275276 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6920070f10 msgr2=0x7f69200cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:23.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.148+0000 7f691de00640 1 --2- 172.21.15.6:0/12275276 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6920070f10 0x7f69200cc950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:23.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.148+0000 7f691de00640 1 -- 172.21.15.6:0/12275276 >> v1:172.21.15.146:6789/0 conn(0x7f69200cce90 legacy=0x7f69200cd930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:23.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.148+0000 7f691de00640 1 -- 172.21.15.6:0/12275276 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6920071a90 con 0x7f6920145300 2024-03-04T09:26:23.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.149+0000 7f691de00640 1 --2- 172.21.15.6:0/12275276 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6920145300 0x7f69200b49a0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f6910009e50 tx=0x7f691002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87bd6a7eaa41dad8 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:23.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.149+0000 7f691ca00640 1 -- 172.21.15.6:0/12275276 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f691002ca70 con 0x7f6920145300 2024-03-04T09:26:23.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.149+0000 7f691ca00640 1 -- 172.21.15.6:0/12275276 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f691002cc10 con 0x7f6920145300 2024-03-04T09:26:23.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.149+0000 7f691ca00640 1 -- 172.21.15.6:0/12275276 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69100343f0 con 0x7f6920145300 2024-03-04T09:26:23.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.150+0000 7f691f200640 1 -- 172.21.15.6:0/12275276 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6920145300 msgr2=0x7f69200b49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:23.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.150+0000 7f691f200640 1 --2- 172.21.15.6:0/12275276 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6920145300 0x7f69200b49a0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f6910009e50 tx=0x7f691002bed0 comp rx=0 tx=0).stop 2024-03-04T09:26:23.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.150+0000 7f691f200640 1 -- 172.21.15.6:0/12275276 shutdown_connections 2024-03-04T09:26:23.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.150+0000 7f691f200640 1 --2- 172.21.15.6:0/12275276 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6920070f10 0x7f69200cc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:23.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.150+0000 7f691f200640 1 --2- 172.21.15.6:0/12275276 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6920145300 0x7f69200b49a0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:23.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.150+0000 7f691f200640 1 -- 172.21.15.6:0/12275276 >> 172.21.15.6:0/12275276 conn(0x7f692006b3d0 msgr2=0x7f692006b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:23.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.150+0000 7f691f200640 1 -- 172.21.15.6:0/12275276 shutdown_connections 2024-03-04T09:26:23.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.150+0000 7f691f200640 1 -- 172.21.15.6:0/12275276 wait complete. 2024-03-04T09:26:23.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.152+0000 7f691f200640 1 Processor -- start 2024-03-04T09:26:23.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.152+0000 7f691f200640 1 -- start start 2024-03-04T09:26:23.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.153+0000 7f691f200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6920070f10 0x7f69201f7f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:23.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.153+0000 7f691f200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f69200cce90 0x7f692012daa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:23.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.153+0000 7f691f200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6920145300 0x7f692006ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:23.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.153+0000 7f691f200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69201456e0 con 0x7f6920145300 2024-03-04T09:26:23.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.153+0000 7f691f200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69200b4ee0 con 0x7f69200cce90 2024-03-04T09:26:23.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.153+0000 7f691f200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f692015d580 con 0x7f6920070f10 2024-03-04T09:26:23.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.153+0000 7f691e800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6920145300 0x7f692006ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:23.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.153+0000 7f691e800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6920145300 0x7f692006ddd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45730/0 (socket says 172.21.15.6:45730) 2024-03-04T09:26:23.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.153+0000 7f691e800640 1 -- 172.21.15.6:0/2933038143 learned_addr learned my addr 172.21.15.6:0/2933038143 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:23.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.153+0000 7f691de00640 1 --2- 172.21.15.6:0/2933038143 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6920070f10 0x7f69201f7f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:23.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.153+0000 7f691d400640 1 --2- 172.21.15.6:0/2933038143 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f69200cce90 0x7f692012daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:23.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.154+0000 7f691e800640 1 -- 172.21.15.6:0/2933038143 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6920070f10 msgr2=0x7f69201f7f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:23.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.154+0000 7f691e800640 1 --2- 172.21.15.6:0/2933038143 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6920070f10 0x7f69201f7f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:23.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.154+0000 7f691e800640 1 -- 172.21.15.6:0/2933038143 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f69200cce90 msgr2=0x7f692012daa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:23.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.154+0000 7f691e800640 1 --2- 172.21.15.6:0/2933038143 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f69200cce90 0x7f692012daa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:23.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.154+0000 7f691e800640 1 -- 172.21.15.6:0/2933038143 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69201309d0 con 0x7f6920145300 2024-03-04T09:26:23.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.154+0000 7f691e800640 1 --2- 172.21.15.6:0/2933038143 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6920145300 0x7f692006ddd0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f691800c440 tx=0x7f691800c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:23.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.154+0000 7f6916000640 1 -- 172.21.15.6:0/2933038143 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6918011790 con 0x7f6920145300 2024-03-04T09:26:23.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.154+0000 7f6916000640 1 -- 172.21.15.6:0/2933038143 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6918017070 con 0x7f6920145300 2024-03-04T09:26:23.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.154+0000 7f6916000640 1 -- 172.21.15.6:0/2933038143 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6918011b70 con 0x7f6920145300 2024-03-04T09:26:23.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.154+0000 7f691f200640 1 -- 172.21.15.6:0/2933038143 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6920130cb0 con 0x7f6920145300 2024-03-04T09:26:23.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.155+0000 7f691f200640 1 -- 172.21.15.6:0/2933038143 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6920131270 con 0x7f6920145300 2024-03-04T09:26:23.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.155+0000 7f691f200640 1 -- 172.21.15.6:0/2933038143 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69200b3eb0 con 0x7f6920145300 2024-03-04T09:26:23.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.156+0000 7f6916000640 1 -- 172.21.15.6:0/2933038143 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6918011d10 con 0x7f6920145300 2024-03-04T09:26:23.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.157+0000 7f6916000640 1 --2- 172.21.15.6:0/2933038143 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f690006b4d0 0x7f690006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:23.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.157+0000 7f6916000640 1 -- 172.21.15.6:0/2933038143 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f6918089d60 con 0x7f6920145300 2024-03-04T09:26:23.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.158+0000 7f691de00640 1 --2- 172.21.15.6:0/2933038143 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f690006b4d0 0x7f690006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:23.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.159+0000 7f691de00640 1 --2- 172.21.15.6:0/2933038143 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f690006b4d0 0x7f690006d990 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f6910035000 tx=0x7f6910030000 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:23.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.162+0000 7f6916000640 1 -- 172.21.15.6:0/2933038143 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f69200b3eb0 con 0x7f6920145300 2024-03-04T09:26:23.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.427+0000 7f691f200640 1 -- 172.21.15.6:0/2933038143 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f692004f330 con 0x7f6920145300 2024-03-04T09:26:23.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.428+0000 7f6916000640 1 -- 172.21.15.6:0/2933038143 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v3) v1 ==== 80+0+6645 (secure 0 0 0) 0x7f6918058e90 con 0x7f6920145300 2024-03-04T09:26:23.425 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:26:23.426 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-04T09:26:23.426 INFO:teuthology.orchestra.run.smithi006.stdout: "auth_dump": [ 2024-03-04T09:26:23.426 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:26:23.426 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "mds.a", 2024-03-04T09:26:23.426 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQA8k+VlHCzFIBAAnKYlmCMDw45LqnSjxnZY5Q==", 2024-03-04T09:26:23.426 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:26:23.426 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-04T09:26:23.426 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow *", 2024-03-04T09:26:23.426 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow *", 2024-03-04T09:26:23.426 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-04T09:26:23.426 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:26:23.426 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:26:23.426 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:26:23.427 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "mds.b", 2024-03-04T09:26:23.427 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQA9k+VlNY7fCBAAX3z8X72/6LkQD6mwGYT69g==", 2024-03-04T09:26:23.427 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:26:23.427 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-04T09:26:23.427 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow *", 2024-03-04T09:26:23.427 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow *", 2024-03-04T09:26:23.427 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-04T09:26:23.427 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:26:23.427 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:26:23.427 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:26:23.427 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "mds.c", 2024-03-04T09:26:23.428 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQA8k+Vl0pjOMhAA7WcTw8awv4EAsrpP4yncNA==", 2024-03-04T09:26:23.428 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:26:23.428 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-04T09:26:23.428 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow *", 2024-03-04T09:26:23.428 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow *", 2024-03-04T09:26:23.428 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-04T09:26:23.428 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:26:23.428 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:26:23.428 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:26:23.428 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "mds.d", 2024-03-04T09:26:23.428 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQA9k+VlbladGRAABNgUcaqTVYuJfCqy5c/G0g==", 2024-03-04T09:26:23.429 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:26:23.429 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-04T09:26:23.429 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow *", 2024-03-04T09:26:23.429 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow *", 2024-03-04T09:26:23.429 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-04T09:26:23.429 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:26:23.429 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:26:23.429 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:26:23.429 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.0", 2024-03-04T09:26:23.429 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQBCk+VlyjMMExAAqb2FJ9n5PWehfx/6PB19ig==", 2024-03-04T09:26:23.429 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:26:23.429 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-04T09:26:23.430 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-04T09:26:23.430 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-04T09:26:23.430 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:26:23.430 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:26:23.430 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:26:23.430 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.1", 2024-03-04T09:26:23.430 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQBFk+VlXD3cLBAAGclSivcYWDslqN8MLlj7eQ==", 2024-03-04T09:26:23.430 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:26:23.430 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-04T09:26:23.430 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-04T09:26:23.430 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-04T09:26:23.430 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:26:23.430 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:26:23.430 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:26:23.431 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.2", 2024-03-04T09:26:23.431 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQBJk+Vl7d9nCxAAnb3W4EI3Yf3XY8ht3VnKEQ==", 2024-03-04T09:26:23.431 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:26:23.431 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-04T09:26:23.431 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-04T09:26:23.431 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-04T09:26:23.431 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:26:23.431 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:26:23.431 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:26:23.431 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.3", 2024-03-04T09:26:23.432 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQBMk+Vlv+Y1NBAAiRYSUqov2i2sj3iInfpHvA==", 2024-03-04T09:26:23.432 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:26:23.432 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-04T09:26:23.432 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-04T09:26:23.432 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-04T09:26:23.432 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:26:23.432 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:26:23.432 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:26:23.432 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.4", 2024-03-04T09:26:23.432 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQBUk+Vl1OlAARAAxMV4GUGS+RJ6XlrRhrVKcg==", 2024-03-04T09:26:23.432 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:26:23.432 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-04T09:26:23.433 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-04T09:26:23.433 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-04T09:26:23.433 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:26:23.433 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:26:23.433 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:26:23.433 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.5", 2024-03-04T09:26:23.433 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQBXk+Vl1lKLGhAAHlsiRr+1+OSRWhkB+weIJg==", 2024-03-04T09:26:23.433 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:26:23.433 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-04T09:26:23.433 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-04T09:26:23.433 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-04T09:26:23.433 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:26:23.433 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:26:23.434 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:26:23.434 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.6", 2024-03-04T09:26:23.434 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQBak+VlwB+GJBAA3f9KUz+zZlQuUR/ywQfkDA==", 2024-03-04T09:26:23.434 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:26:23.434 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-04T09:26:23.434 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-04T09:26:23.434 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-04T09:26:23.434 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:26:23.434 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:26:23.434 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:26:23.434 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.7", 2024-03-04T09:26:23.434 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQBdk+Vle4o8LhAAko+iH5VqqCsfWkNfKZjGVA==", 2024-03-04T09:26:23.434 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:26:23.435 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-04T09:26:23.435 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-04T09:26:23.435 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-04T09:26:23.435 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:26:23.435 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:26:23.435 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:26:23.435 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "client.0", 2024-03-04T09:26:23.435 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQA9k+Vl4L+rJxAACaDbQ4/Bybp5hAZAnJzlVw==", 2024-03-04T09:26:23.435 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:26:23.435 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-04T09:26:23.435 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow r", 2024-03-04T09:26:23.435 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow rw", 2024-03-04T09:26:23.436 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow rwx" 2024-03-04T09:26:23.436 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:26:23.436 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:26:23.436 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:26:23.436 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "client.1", 2024-03-04T09:26:23.436 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQA+k+VlGjkOAhAAJNzB8+tQl3mqCnTgMn9EVw==", 2024-03-04T09:26:23.436 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:26:23.436 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-04T09:26:23.436 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow r", 2024-03-04T09:26:23.437 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow rw", 2024-03-04T09:26:23.437 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow rwx" 2024-03-04T09:26:23.437 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:26:23.437 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:26:23.437 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:26:23.437 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "client.2", 2024-03-04T09:26:23.437 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQA9k+VlUc8DMxAA8pGUtIh+BO6UKrrYnCqVgQ==", 2024-03-04T09:26:23.437 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:26:23.437 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-04T09:26:23.437 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow r", 2024-03-04T09:26:23.437 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow rw", 2024-03-04T09:26:23.437 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow rwx" 2024-03-04T09:26:23.438 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:26:23.438 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:26:23.438 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:26:23.438 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "client.3", 2024-03-04T09:26:23.438 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQA+k+VlzMFKDBAAGr713I2hcmGMuHMIjNknYA==", 2024-03-04T09:26:23.438 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:26:23.438 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-04T09:26:23.438 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow r", 2024-03-04T09:26:23.438 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow rw", 2024-03-04T09:26:23.438 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow rwx" 2024-03-04T09:26:23.438 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:26:23.438 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:26:23.438 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:26:23.439 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "client.admin", 2024-03-04T09:26:23.439 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQA7k+Vllb7NFhAAmnFaNFKBcEri+iJAPnG5Gg==", 2024-03-04T09:26:23.439 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:26:23.439 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow *", 2024-03-04T09:26:23.439 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow *", 2024-03-04T09:26:23.439 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow *", 2024-03-04T09:26:23.439 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-04T09:26:23.439 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:26:23.439 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:26:23.439 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:26:23.439 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "client.bootstrap-mds", 2024-03-04T09:26:23.439 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQBmk+VlIoRCHxAAM31cbNOpGe1zScc6tWGDmg==", 2024-03-04T09:26:23.439 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:26:23.440 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile bootstrap-mds" 2024-03-04T09:26:23.440 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:26:23.440 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:26:23.440 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:26:23.440 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "client.bootstrap-mgr", 2024-03-04T09:26:23.440 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQBmk+VlQZRCHxAA9C0b/oy7EKh51JcV9SU5nA==", 2024-03-04T09:26:23.440 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:26:23.440 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile bootstrap-mgr" 2024-03-04T09:26:23.440 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:26:23.440 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:26:23.440 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:26:23.440 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "client.bootstrap-osd", 2024-03-04T09:26:23.441 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQBmk+VlBaNCHxAA8ReBGFatZelAksT0+dXzYA==", 2024-03-04T09:26:23.441 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:26:23.441 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile bootstrap-osd" 2024-03-04T09:26:23.441 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:26:23.441 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:26:23.441 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:26:23.441 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "client.bootstrap-rbd", 2024-03-04T09:26:23.441 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQBmk+VlzatCHxAAyzfb+Vv1ey+A37Hi3lhcwA==", 2024-03-04T09:26:23.441 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:26:23.441 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile bootstrap-rbd" 2024-03-04T09:26:23.442 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:26:23.442 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:26:23.442 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:26:23.442 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "client.bootstrap-rbd-mirror", 2024-03-04T09:26:23.442 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQBmk+VlabRCHxAAErApWRnJ2xC/CupUdnaW4A==", 2024-03-04T09:26:23.442 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:26:23.442 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile bootstrap-rbd-mirror" 2024-03-04T09:26:23.442 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:26:23.442 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:26:23.443 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:26:23.443 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "client.bootstrap-rgw", 2024-03-04T09:26:23.443 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQBmk+VlEL1CHxAAKG6G3oKhYNr63vjFKZrTpw==", 2024-03-04T09:26:23.443 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:26:23.443 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile bootstrap-rgw" 2024-03-04T09:26:23.443 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:26:23.443 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:26:23.443 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:26:23.443 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "mgr.x", 2024-03-04T09:26:23.443 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQA8k+VlPFatFBAA/o3Ps/uJrcg1738HOGy/FA==", 2024-03-04T09:26:23.443 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:26:23.444 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow *", 2024-03-04T09:26:23.444 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile mgr", 2024-03-04T09:26:23.444 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-04T09:26:23.444 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:26:23.444 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:26:23.444 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:26:23.444 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "mgr.y", 2024-03-04T09:26:23.444 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQA8k+VlEA2ACBAAXofDj3gmqItVTJ1pYLqZyQ==", 2024-03-04T09:26:23.444 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:26:23.444 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow *", 2024-03-04T09:26:23.444 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile mgr", 2024-03-04T09:26:23.445 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-04T09:26:23.445 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:26:23.445 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:26:23.445 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-04T09:26:23.445 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-04T09:26:23.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.431+0000 7f691f200640 1 -- 172.21.15.6:0/2933038143 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f690006b4d0 msgr2=0x7f690006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:23.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.431+0000 7f691f200640 1 --2- 172.21.15.6:0/2933038143 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f690006b4d0 0x7f690006d990 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f6910035000 tx=0x7f6910030000 comp rx=0 tx=0).stop 2024-03-04T09:26:23.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.431+0000 7f691f200640 1 -- 172.21.15.6:0/2933038143 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6920145300 msgr2=0x7f692006ddd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:23.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.431+0000 7f691f200640 1 --2- 172.21.15.6:0/2933038143 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6920145300 0x7f692006ddd0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f691800c440 tx=0x7f691800c900 comp rx=0 tx=0).stop 2024-03-04T09:26:23.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.431+0000 7f691f200640 1 -- 172.21.15.6:0/2933038143 shutdown_connections 2024-03-04T09:26:23.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.431+0000 7f691f200640 1 --2- 172.21.15.6:0/2933038143 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6920145300 0x7f692006ddd0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:23.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.431+0000 7f691f200640 1 --2- 172.21.15.6:0/2933038143 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f69200cce90 0x7f692012daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:23.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.431+0000 7f691f200640 1 --2- 172.21.15.6:0/2933038143 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f690006b4d0 0x7f690006d990 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:23.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.431+0000 7f691f200640 1 --2- 172.21.15.6:0/2933038143 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6920070f10 0x7f69201f7f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:23.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.431+0000 7f691f200640 1 -- 172.21.15.6:0/2933038143 >> 172.21.15.6:0/2933038143 conn(0x7f692006b3d0 msgr2=0x7f69200ce110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:23.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.432+0000 7f691f200640 1 -- 172.21.15.6:0/2933038143 shutdown_connections 2024-03-04T09:26:23.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.432+0000 7f691f200640 1 -- 172.21.15.6:0/2933038143 wait complete. 2024-03-04T09:26:23.448 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mds 2024-03-04T09:26:23.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.676+0000 7f3b8f200640 1 Processor -- start 2024-03-04T09:26:23.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.676+0000 7f3b8f200640 1 -- start start 2024-03-04T09:26:23.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.678+0000 7f3b8f200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f3b90070f10 0x7f3b900cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:23.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.678+0000 7f3b8f200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b900714d0 con 0x7f3b90145300 2024-03-04T09:26:23.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.678+0000 7f3b8f200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b900717b0 con 0x7f3b900cce90 2024-03-04T09:26:23.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.678+0000 7f3b8f200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b900cd300 con 0x7f3b90070f10 2024-03-04T09:26:23.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.678+0000 7f3b87e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f3b90070f10 0x7f3b900cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:23.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.678+0000 7f3b87e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f3b90070f10 0x7f3b900cc950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45734/0 (socket says 172.21.15.6:45734) 2024-03-04T09:26:23.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.678+0000 7f3b87e00640 1 -- 172.21.15.6:0/1306707524 learned_addr learned my addr 172.21.15.6:0/1306707524 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:23.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.678+0000 7f3b8de00640 1 --1- 172.21.15.6:0/1306707524 >> v1:172.21.15.6:6789/0 conn(0x7f3b90145300 0x7f3b900b49a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:54596/0 (socket says 172.21.15.6:54596) 2024-03-04T09:26:23.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.678+0000 7f3b87e00640 1 -- 172.21.15.6:0/1306707524 >> v1:172.21.15.146:6789/0 conn(0x7f3b900cce90 legacy=0x7f3b9015c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:23.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.678+0000 7f3b87e00640 1 -- 172.21.15.6:0/1306707524 >> v1:172.21.15.6:6789/0 conn(0x7f3b90145300 legacy=0x7f3b900b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:23.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.678+0000 7f3b87e00640 1 -- 172.21.15.6:0/1306707524 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b90071a90 con 0x7f3b90070f10 2024-03-04T09:26:23.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.680+0000 7f3b87e00640 1 --2- 172.21.15.6:0/1306707524 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3b90070f10 0x7f3b900cc950 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f3b78009870 tx=0x7f3b7802b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c321084ececfead5 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:23.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.680+0000 7f3b8d400640 1 -- 172.21.15.6:0/1306707524 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b78035020 con 0x7f3b90070f10 2024-03-04T09:26:23.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.680+0000 7f3b8d400640 1 -- 172.21.15.6:0/1306707524 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b7802d040 con 0x7f3b90070f10 2024-03-04T09:26:23.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.680+0000 7f3b8d400640 1 -- 172.21.15.6:0/1306707524 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b78031a80 con 0x7f3b90070f10 2024-03-04T09:26:23.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.680+0000 7f3b8f200640 1 -- 172.21.15.6:0/1306707524 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3b90070f10 msgr2=0x7f3b900cc950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:23.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.680+0000 7f3b8f200640 1 --2- 172.21.15.6:0/1306707524 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3b90070f10 0x7f3b900cc950 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f3b78009870 tx=0x7f3b7802b900 comp rx=0 tx=0).stop 2024-03-04T09:26:23.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.680+0000 7f3b8f200640 1 -- 172.21.15.6:0/1306707524 shutdown_connections 2024-03-04T09:26:23.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.681+0000 7f3b8f200640 1 --2- 172.21.15.6:0/1306707524 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3b90070f10 0x7f3b900cc950 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:23.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.681+0000 7f3b8f200640 1 -- 172.21.15.6:0/1306707524 >> 172.21.15.6:0/1306707524 conn(0x7f3b9006b3d0 msgr2=0x7f3b9006b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:23.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.681+0000 7f3b8f200640 1 -- 172.21.15.6:0/1306707524 shutdown_connections 2024-03-04T09:26:23.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.681+0000 7f3b8f200640 1 -- 172.21.15.6:0/1306707524 wait complete. 2024-03-04T09:26:23.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.682+0000 7f3b8f200640 1 Processor -- start 2024-03-04T09:26:23.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.682+0000 7f3b8f200640 1 -- start start 2024-03-04T09:26:23.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.683+0000 7f3b8f200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3b90070f10 0x7f3b901e6ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:23.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.683+0000 7f3b8f200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3b900cce90 0x7f3b900abe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:23.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.683+0000 7f3b8f200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3b90145300 0x7f3b900ac370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:23.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.683+0000 7f3b8f200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b9015cc80 con 0x7f3b90070f10 2024-03-04T09:26:23.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.683+0000 7f3b8f200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b901456e0 con 0x7f3b90145300 2024-03-04T09:26:23.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.683+0000 7f3b8f200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b9015ce00 con 0x7f3b900cce90 2024-03-04T09:26:23.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.683+0000 7f3b8de00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3b90070f10 0x7f3b901e6ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:23.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.684+0000 7f3b8de00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3b90070f10 0x7f3b901e6ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45746/0 (socket says 172.21.15.6:45746) 2024-03-04T09:26:23.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.684+0000 7f3b8de00640 1 -- 172.21.15.6:0/328698534 learned_addr learned my addr 172.21.15.6:0/328698534 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:23.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.684+0000 7f3b8de00640 1 -- 172.21.15.6:0/328698534 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3b900cce90 msgr2=0x7f3b900abe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:23.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.684+0000 7f3b8de00640 1 --2- 172.21.15.6:0/328698534 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3b900cce90 0x7f3b900abe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:23.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.684+0000 7f3b8de00640 1 -- 172.21.15.6:0/328698534 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3b90145300 msgr2=0x7f3b900ac370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:23.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.684+0000 7f3b8de00640 1 --2- 172.21.15.6:0/328698534 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3b90145300 0x7f3b900ac370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:23.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.684+0000 7f3b8de00640 1 -- 172.21.15.6:0/328698534 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b901e5820 con 0x7f3b90070f10 2024-03-04T09:26:23.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.684+0000 7f3b8de00640 1 --2- 172.21.15.6:0/328698534 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3b90070f10 0x7f3b901e6ba0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f3b8000c3f0 tx=0x7f3b8000c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:23.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.685+0000 7f3b86000640 1 -- 172.21.15.6:0/328698534 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b800074e0 con 0x7f3b90070f10 2024-03-04T09:26:23.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.685+0000 7f3b86000640 1 -- 172.21.15.6:0/328698534 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b80007680 con 0x7f3b90070f10 2024-03-04T09:26:23.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.685+0000 7f3b86000640 1 -- 172.21.15.6:0/328698534 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b80004a00 con 0x7f3b90070f10 2024-03-04T09:26:23.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.685+0000 7f3b8f200640 1 -- 172.21.15.6:0/328698534 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b901e5b00 con 0x7f3b90070f10 2024-03-04T09:26:23.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.685+0000 7f3b8f200640 1 -- 172.21.15.6:0/328698534 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b901e60c0 con 0x7f3b90070f10 2024-03-04T09:26:23.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.686+0000 7f3b8f200640 1 -- 172.21.15.6:0/328698534 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b9004f330 con 0x7f3b90070f10 2024-03-04T09:26:23.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.687+0000 7f3b86000640 1 -- 172.21.15.6:0/328698534 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3b8001a020 con 0x7f3b90070f10 2024-03-04T09:26:23.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.688+0000 7f3b86000640 1 --2- 172.21.15.6:0/328698534 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f3b6c06b4d0 0x7f3b6c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:23.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.688+0000 7f3b86000640 1 -- 172.21.15.6:0/328698534 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f3b80089770 con 0x7f3b90070f10 2024-03-04T09:26:23.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.689+0000 7f3b87e00640 1 --2- 172.21.15.6:0/328698534 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f3b6c06b4d0 0x7f3b6c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:23.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.690+0000 7f3b87e00640 1 --2- 172.21.15.6:0/328698534 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f3b6c06b4d0 0x7f3b6c06d990 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f3b78036000 tx=0x7f3b78030000 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:23.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.692+0000 7f3b86000640 1 -- 172.21.15.6:0/328698534 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3b80058810 con 0x7f3b90070f10 2024-03-04T09:26:23.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.969+0000 7f3b8f200640 1 -- 172.21.15.6:0/328698534 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mds"} v 0) v1 -- 0x7f3b9015ba50 con 0x7f3b90070f10 2024-03-04T09:26:23.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.973+0000 7f3b86000640 1 -- 172.21.15.6:0/328698534 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mds"}]=0 v4) v1 ==== 90+0+0 (secure 0 0 0) 0x7f3b80058110 con 0x7f3b90070f10 2024-03-04T09:26:23.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.976+0000 7f3b8f200640 1 -- 172.21.15.6:0/328698534 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f3b6c06b4d0 msgr2=0x7f3b6c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:23.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.976+0000 7f3b8f200640 1 --2- 172.21.15.6:0/328698534 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f3b6c06b4d0 0x7f3b6c06d990 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f3b78036000 tx=0x7f3b78030000 comp rx=0 tx=0).stop 2024-03-04T09:26:23.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.976+0000 7f3b8f200640 1 -- 172.21.15.6:0/328698534 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3b90070f10 msgr2=0x7f3b901e6ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:23.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.976+0000 7f3b8f200640 1 --2- 172.21.15.6:0/328698534 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3b90070f10 0x7f3b901e6ba0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f3b8000c3f0 tx=0x7f3b8000c8b0 comp rx=0 tx=0).stop 2024-03-04T09:26:23.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.976+0000 7f3b8f200640 1 -- 172.21.15.6:0/328698534 shutdown_connections 2024-03-04T09:26:23.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.976+0000 7f3b8f200640 1 --2- 172.21.15.6:0/328698534 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3b90145300 0x7f3b900ac370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:23.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.976+0000 7f3b8f200640 1 --2- 172.21.15.6:0/328698534 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f3b6c06b4d0 0x7f3b6c06d990 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:23.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.976+0000 7f3b8f200640 1 --2- 172.21.15.6:0/328698534 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3b900cce90 0x7f3b900abe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:23.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.976+0000 7f3b8f200640 1 --2- 172.21.15.6:0/328698534 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3b90070f10 0x7f3b901e6ba0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:23.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.976+0000 7f3b8f200640 1 -- 172.21.15.6:0/328698534 >> 172.21.15.6:0/328698534 conn(0x7f3b9006b3d0 msgr2=0x7f3b900cd930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:23.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.976+0000 7f3b8f200640 1 -- 172.21.15.6:0/328698534 shutdown_connections 2024-03-04T09:26:23.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:23.976+0000 7f3b8f200640 1 -- 172.21.15.6:0/328698534 wait complete. 2024-03-04T09:26:23.992 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mgr 2024-03-04T09:26:24.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.217+0000 7fb365000640 1 Processor -- start 2024-03-04T09:26:24.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.217+0000 7fb365000640 1 -- start start 2024-03-04T09:26:24.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.219+0000 7fb365000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb3600ce850 0x7fb3600cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:24.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.219+0000 7fb365000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fb3600cce90 0x7fb36006cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:24.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.219+0000 7fb365000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3600cdad0 con 0x7fb36015c920 2024-03-04T09:26:24.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.219+0000 7fb365000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb360071500 con 0x7fb3600cce90 2024-03-04T09:26:24.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.219+0000 7fb365000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3600cdd70 con 0x7fb3600ce850 2024-03-04T09:26:24.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.220+0000 7fb35fe00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fb3600cce90 0x7fb36006cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:24.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.220+0000 7fb35fe00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fb3600cce90 0x7fb36006cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.6:34464/0 (socket says 172.21.15.6:34464) 2024-03-04T09:26:24.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.220+0000 7fb35fe00640 1 -- 172.21.15.6:0/1826113673 learned_addr learned my addr 172.21.15.6:0/1826113673 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:24.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.220+0000 7fb356a00640 1 --2- 172.21.15.6:0/1826113673 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb3600ce850 0x7fb3600cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:24.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.220+0000 7fb35fe00640 1 -- 172.21.15.6:0/1826113673 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb3600ce850 msgr2=0x7fb3600cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:24.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.220+0000 7fb35fe00640 1 --2- 172.21.15.6:0/1826113673 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb3600ce850 0x7fb3600cc950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:24.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.220+0000 7fb35fe00640 1 -- 172.21.15.6:0/1826113673 >> v1:172.21.15.146:6789/0 conn(0x7fb36015c920 legacy=0x7fb360070a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:24.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.220+0000 7fb35fe00640 1 -- 172.21.15.6:0/1826113673 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3600ce050 con 0x7fb3600cce90 2024-03-04T09:26:24.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.222+0000 7fb35fe00640 1 --2- 172.21.15.6:0/1826113673 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb3600cce90 0x7fb36006cac0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fb358009870 tx=0x7fb35802b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1f1c40ecf3646d12 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:24.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.222+0000 7fb35ea00640 1 -- 172.21.15.6:0/1826113673 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb358034070 con 0x7fb3600cce90 2024-03-04T09:26:24.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.222+0000 7fb35ea00640 1 -- 172.21.15.6:0/1826113673 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb358030840 con 0x7fb3600cce90 2024-03-04T09:26:24.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.222+0000 7fb35ea00640 1 -- 172.21.15.6:0/1826113673 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb358030c10 con 0x7fb3600cce90 2024-03-04T09:26:24.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.223+0000 7fb365000640 1 -- 172.21.15.6:0/1826113673 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb3600cce90 msgr2=0x7fb36006cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:24.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.223+0000 7fb365000640 1 --2- 172.21.15.6:0/1826113673 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb3600cce90 0x7fb36006cac0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fb358009870 tx=0x7fb35802b980 comp rx=0 tx=0).stop 2024-03-04T09:26:24.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.223+0000 7fb365000640 1 -- 172.21.15.6:0/1826113673 shutdown_connections 2024-03-04T09:26:24.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.223+0000 7fb365000640 1 --2- 172.21.15.6:0/1826113673 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb3600cce90 0x7fb36006cac0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:24.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.223+0000 7fb365000640 1 --2- 172.21.15.6:0/1826113673 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb3600ce850 0x7fb3600cc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:24.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.223+0000 7fb365000640 1 -- 172.21.15.6:0/1826113673 >> 172.21.15.6:0/1826113673 conn(0x7fb36006b3d0 msgr2=0x7fb36006b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:24.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.223+0000 7fb365000640 1 -- 172.21.15.6:0/1826113673 shutdown_connections 2024-03-04T09:26:24.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.223+0000 7fb365000640 1 -- 172.21.15.6:0/1826113673 wait complete. 2024-03-04T09:26:24.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.225+0000 7fb365000640 1 Processor -- start 2024-03-04T09:26:24.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.225+0000 7fb365000640 1 -- start start 2024-03-04T09:26:24.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.226+0000 7fb365000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb3600cce90 0x7fb360150f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:24.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.226+0000 7fb365000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb3600ce850 0x7fb3600b0350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:24.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.226+0000 7fb365000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb36015c920 0x7fb3601519b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:24.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.226+0000 7fb365000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb360071150 con 0x7fb3600cce90 2024-03-04T09:26:24.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.226+0000 7fb365000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb360163790 con 0x7fb3600ce850 2024-03-04T09:26:24.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.226+0000 7fb365000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3600b2940 con 0x7fb36015c920 2024-03-04T09:26:24.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.226+0000 7fb35f400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb3600cce90 0x7fb360150f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:24.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.227+0000 7fb35f400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb3600cce90 0x7fb360150f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45754/0 (socket says 172.21.15.6:45754) 2024-03-04T09:26:24.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.227+0000 7fb35f400640 1 -- 172.21.15.6:0/1446214359 learned_addr learned my addr 172.21.15.6:0/1446214359 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:24.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.227+0000 7fb356a00640 1 --2- 172.21.15.6:0/1446214359 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb3600ce850 0x7fb3600b0350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:24.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.227+0000 7fb35fe00640 1 --2- 172.21.15.6:0/1446214359 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb36015c920 0x7fb3601519b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:24.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.227+0000 7fb35f400640 1 -- 172.21.15.6:0/1446214359 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb36015c920 msgr2=0x7fb3601519b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:24.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.227+0000 7fb35f400640 1 --2- 172.21.15.6:0/1446214359 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb36015c920 0x7fb3601519b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:24.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.227+0000 7fb35f400640 1 -- 172.21.15.6:0/1446214359 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb3600ce850 msgr2=0x7fb3600b0350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:24.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.227+0000 7fb35f400640 1 --2- 172.21.15.6:0/1446214359 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb3600ce850 0x7fb3600b0350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:24.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.227+0000 7fb35f400640 1 -- 172.21.15.6:0/1446214359 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3600b0c50 con 0x7fb3600cce90 2024-03-04T09:26:24.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.227+0000 7fb35fe00640 1 --2- 172.21.15.6:0/1446214359 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb36015c920 0x7fb3601519b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:26:24.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.227+0000 7fb35f400640 1 --2- 172.21.15.6:0/1446214359 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb3600cce90 0x7fb360150f60 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fb35000c730 tx=0x7fb35000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:24.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.228+0000 7fb357e00640 1 -- 172.21.15.6:0/1446214359 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb350016020 con 0x7fb3600cce90 2024-03-04T09:26:24.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.228+0000 7fb357e00640 1 -- 172.21.15.6:0/1446214359 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb350006d50 con 0x7fb3600cce90 2024-03-04T09:26:24.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.228+0000 7fb357e00640 1 -- 172.21.15.6:0/1446214359 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb350003bb0 con 0x7fb3600cce90 2024-03-04T09:26:24.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.228+0000 7fb365000640 1 -- 172.21.15.6:0/1446214359 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3600b0f30 con 0x7fb3600cce90 2024-03-04T09:26:24.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.228+0000 7fb365000640 1 -- 172.21.15.6:0/1446214359 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3600b14f0 con 0x7fb3600cce90 2024-03-04T09:26:24.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.229+0000 7fb365000640 1 -- 172.21.15.6:0/1446214359 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb36004f330 con 0x7fb3600cce90 2024-03-04T09:26:24.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.230+0000 7fb357e00640 1 -- 172.21.15.6:0/1446214359 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb35000a850 con 0x7fb3600cce90 2024-03-04T09:26:24.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.232+0000 7fb357e00640 1 --2- 172.21.15.6:0/1446214359 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb33c06b4d0 0x7fb33c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:24.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.232+0000 7fb357e00640 1 -- 172.21.15.6:0/1446214359 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fb350089ba0 con 0x7fb3600cce90 2024-03-04T09:26:24.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.232+0000 7fb356a00640 1 --2- 172.21.15.6:0/1446214359 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb33c06b4d0 0x7fb33c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:24.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.234+0000 7fb356a00640 1 --2- 172.21.15.6:0/1446214359 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb33c06b4d0 0x7fb33c06d990 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fb3601514a0 tx=0x7fb348008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:24.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.237+0000 7fb357e00640 1 -- 172.21.15.6:0/1446214359 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb350020020 con 0x7fb3600cce90 2024-03-04T09:26:24.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.497+0000 7fb365000640 1 -- 172.21.15.6:0/1446214359 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mgr"} v 0) v1 -- 0x7fb3600cf3a0 con 0x7fb3600cce90 2024-03-04T09:26:24.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.502+0000 7fb357e00640 1 -- 172.21.15.6:0/1446214359 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mgr"}]=0 v5) v1 ==== 90+0+0 (secure 0 0 0) 0x7fb350019310 con 0x7fb3600cce90 2024-03-04T09:26:24.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.505+0000 7fb365000640 1 -- 172.21.15.6:0/1446214359 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb33c06b4d0 msgr2=0x7fb33c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:24.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.505+0000 7fb365000640 1 --2- 172.21.15.6:0/1446214359 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb33c06b4d0 0x7fb33c06d990 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fb3601514a0 tx=0x7fb348008040 comp rx=0 tx=0).stop 2024-03-04T09:26:24.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.505+0000 7fb365000640 1 -- 172.21.15.6:0/1446214359 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb3600cce90 msgr2=0x7fb360150f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:24.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.505+0000 7fb365000640 1 --2- 172.21.15.6:0/1446214359 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb3600cce90 0x7fb360150f60 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fb35000c730 tx=0x7fb35000cbf0 comp rx=0 tx=0).stop 2024-03-04T09:26:24.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.505+0000 7fb365000640 1 -- 172.21.15.6:0/1446214359 shutdown_connections 2024-03-04T09:26:24.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.505+0000 7fb365000640 1 --2- 172.21.15.6:0/1446214359 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb33c06b4d0 0x7fb33c06d990 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:24.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.505+0000 7fb365000640 1 --2- 172.21.15.6:0/1446214359 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb36015c920 0x7fb3601519b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:24.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.505+0000 7fb365000640 1 --2- 172.21.15.6:0/1446214359 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb3600ce850 0x7fb3600b0350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:24.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.505+0000 7fb365000640 1 --2- 172.21.15.6:0/1446214359 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb3600cce90 0x7fb360150f60 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:24.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.505+0000 7fb365000640 1 -- 172.21.15.6:0/1446214359 >> 172.21.15.6:0/1446214359 conn(0x7fb36006b3d0 msgr2=0x7fb3601649b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:24.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.505+0000 7fb365000640 1 -- 172.21.15.6:0/1446214359 shutdown_connections 2024-03-04T09:26:24.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.505+0000 7fb365000640 1 -- 172.21.15.6:0/1446214359 wait complete. 2024-03-04T09:26:24.522 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-osd 2024-03-04T09:26:24.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.743+0000 7f477b200640 1 Processor -- start 2024-03-04T09:26:24.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.744+0000 7f477b200640 1 -- start start 2024-03-04T09:26:24.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.745+0000 7f477b200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f477c1552b0 0x7f477c15b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:24.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.745+0000 7f477b200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f477c0cd060 0x7f477c0b3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:24.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.745+0000 7f477b200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f477c15ed10 con 0x7f477c15e810 2024-03-04T09:26:24.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.745+0000 7f477b200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f477c15eff0 con 0x7f477c0cd060 2024-03-04T09:26:24.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.746+0000 7f477b200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f477c15f2d0 con 0x7f477c1552b0 2024-03-04T09:26:24.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.746+0000 7f477a800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f477c0cd060 0x7f477c0b3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:24.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.746+0000 7f477a800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f477c0cd060 0x7f477c0b3cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45760/0 (socket says 172.21.15.6:45760) 2024-03-04T09:26:24.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.746+0000 7f477a800640 1 -- 172.21.15.6:0/3347545799 learned_addr learned my addr 172.21.15.6:0/3347545799 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:24.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.746+0000 7f4779e00640 1 --2- 172.21.15.6:0/3347545799 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f477c1552b0 0x7f477c15b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:24.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.746+0000 7f477a800640 1 -- 172.21.15.6:0/3347545799 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f477c1552b0 msgr2=0x7f477c15b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:24.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.746+0000 7f477a800640 1 --2- 172.21.15.6:0/3347545799 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f477c1552b0 0x7f477c15b1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:24.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.746+0000 7f477a800640 1 -- 172.21.15.6:0/3347545799 >> v1:172.21.15.146:6789/0 conn(0x7f477c15e810 legacy=0x7f477c0cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:24.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.746+0000 7f477a800640 1 -- 172.21.15.6:0/3347545799 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f477c15f5b0 con 0x7f477c0cd060 2024-03-04T09:26:24.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.747+0000 7f477a800640 1 --2- 172.21.15.6:0/3347545799 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f477c0cd060 0x7f477c0b3cb0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f4774009e50 tx=0x7f477402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3517dc688eafe977 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:24.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.748+0000 7f4779400640 1 -- 172.21.15.6:0/3347545799 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f477402c9e0 con 0x7f477c0cd060 2024-03-04T09:26:24.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.748+0000 7f4779400640 1 -- 172.21.15.6:0/3347545799 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f477402cb80 con 0x7f477c0cd060 2024-03-04T09:26:24.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.748+0000 7f4779400640 1 -- 172.21.15.6:0/3347545799 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47740343f0 con 0x7f477c0cd060 2024-03-04T09:26:24.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.749+0000 7f477b200640 1 -- 172.21.15.6:0/3347545799 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f477c0cd060 msgr2=0x7f477c0b3cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:24.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.749+0000 7f477b200640 1 --2- 172.21.15.6:0/3347545799 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f477c0cd060 0x7f477c0b3cb0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f4774009e50 tx=0x7f477402be40 comp rx=0 tx=0).stop 2024-03-04T09:26:24.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.749+0000 7f477b200640 1 -- 172.21.15.6:0/3347545799 shutdown_connections 2024-03-04T09:26:24.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.749+0000 7f477b200640 1 --2- 172.21.15.6:0/3347545799 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f477c0cd060 0x7f477c0b3cb0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:24.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.749+0000 7f477b200640 1 --2- 172.21.15.6:0/3347545799 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f477c1552b0 0x7f477c15b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:24.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.749+0000 7f477b200640 1 -- 172.21.15.6:0/3347545799 >> 172.21.15.6:0/3347545799 conn(0x7f477c06b3d0 msgr2=0x7f477c06b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:24.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.749+0000 7f477b200640 1 -- 172.21.15.6:0/3347545799 shutdown_connections 2024-03-04T09:26:24.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.749+0000 7f477b200640 1 -- 172.21.15.6:0/3347545799 wait complete. 2024-03-04T09:26:24.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.750+0000 7f477b200640 1 Processor -- start 2024-03-04T09:26:24.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.751+0000 7f477b200640 1 -- start start 2024-03-04T09:26:24.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.751+0000 7f477b200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f477c0cd060 0x7f477c14b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:24.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.751+0000 7f477b200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f477c1552b0 0x7f477c0b1120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:24.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.751+0000 7f477b200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f477c15e810 0x7f477c151cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:24.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.751+0000 7f477b200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f477c06dd30 con 0x7f477c15e810 2024-03-04T09:26:24.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.751+0000 7f477b200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f477c0b3760 con 0x7f477c0cd060 2024-03-04T09:26:24.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.751+0000 7f477b200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f477c06deb0 con 0x7f477c1552b0 2024-03-04T09:26:24.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.752+0000 7f477a800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f477c15e810 0x7f477c151cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:24.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.752+0000 7f477a800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f477c15e810 0x7f477c151cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45774/0 (socket says 172.21.15.6:45774) 2024-03-04T09:26:24.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.752+0000 7f477a800640 1 -- 172.21.15.6:0/2037448900 learned_addr learned my addr 172.21.15.6:0/2037448900 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:24.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.752+0000 7f4779e00640 1 --2- 172.21.15.6:0/2037448900 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f477c0cd060 0x7f477c14b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:24.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.752+0000 7f4773e00640 1 --2- 172.21.15.6:0/2037448900 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f477c1552b0 0x7f477c0b1120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:24.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.752+0000 7f477a800640 1 -- 172.21.15.6:0/2037448900 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f477c1552b0 msgr2=0x7f477c0b1120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:24.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.752+0000 7f477a800640 1 --2- 172.21.15.6:0/2037448900 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f477c1552b0 0x7f477c0b1120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:24.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.752+0000 7f477a800640 1 -- 172.21.15.6:0/2037448900 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f477c0cd060 msgr2=0x7f477c14b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:24.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.752+0000 7f477a800640 1 --2- 172.21.15.6:0/2037448900 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f477c0cd060 0x7f477c14b3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:24.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.752+0000 7f477a800640 1 -- 172.21.15.6:0/2037448900 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f477c152d10 con 0x7f477c15e810 2024-03-04T09:26:24.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.752+0000 7f4779e00640 1 --2- 172.21.15.6:0/2037448900 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f477c0cd060 0x7f477c14b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:26:24.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.752+0000 7f4773e00640 1 --2- 172.21.15.6:0/2037448900 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f477c1552b0 0x7f477c0b1120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:26:24.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.753+0000 7f477a800640 1 --2- 172.21.15.6:0/2037448900 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f477c15e810 0x7f477c151cd0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f4774009f70 tx=0x7f4774004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:24.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.753+0000 7f4772000640 1 -- 172.21.15.6:0/2037448900 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4774043020 con 0x7f477c15e810 2024-03-04T09:26:24.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.753+0000 7f4772000640 1 -- 172.21.15.6:0/2037448900 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f477403e450 con 0x7f477c15e810 2024-03-04T09:26:24.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.753+0000 7f4772000640 1 -- 172.21.15.6:0/2037448900 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4774009510 con 0x7f477c15e810 2024-03-04T09:26:24.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.753+0000 7f477b200640 1 -- 172.21.15.6:0/2037448900 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f477c152750 con 0x7f477c15e810 2024-03-04T09:26:24.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.753+0000 7f477b200640 1 -- 172.21.15.6:0/2037448900 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f477c0b0920 con 0x7f477c15e810 2024-03-04T09:26:24.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.754+0000 7f477b200640 1 -- 172.21.15.6:0/2037448900 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f477c04f330 con 0x7f477c15e810 2024-03-04T09:26:24.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.756+0000 7f4772000640 1 -- 172.21.15.6:0/2037448900 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4774005d10 con 0x7f477c15e810 2024-03-04T09:26:24.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.757+0000 7f4772000640 1 --2- 172.21.15.6:0/2037448900 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f475806b500 0x7f475806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:24.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.757+0000 7f4772000640 1 -- 172.21.15.6:0/2037448900 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f47740ac960 con 0x7f477c15e810 2024-03-04T09:26:24.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.758+0000 7f4779e00640 1 --2- 172.21.15.6:0/2037448900 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f475806b500 0x7f475806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:24.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.759+0000 7f4779e00640 1 --2- 172.21.15.6:0/2037448900 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f475806b500 0x7f475806d9c0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f476c006fd0 tx=0x7f476c008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:24.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:24.762+0000 7f4772000640 1 -- 172.21.15.6:0/2037448900 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f477407bb20 con 0x7f477c15e810 2024-03-04T09:26:25.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.032+0000 7f477b200640 1 -- 172.21.15.6:0/2037448900 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-osd"} v 0) v1 -- 0x7f477c0cf4a0 con 0x7f477c15e810 2024-03-04T09:26:25.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.036+0000 7f4772000640 1 -- 172.21.15.6:0/2037448900 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-osd"}]=0 v6) v1 ==== 90+0+0 (secure 0 0 0) 0x7f477407b420 con 0x7f477c15e810 2024-03-04T09:26:25.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.038+0000 7f477b200640 1 -- 172.21.15.6:0/2037448900 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f475806b500 msgr2=0x7f475806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:25.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.038+0000 7f477b200640 1 --2- 172.21.15.6:0/2037448900 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f475806b500 0x7f475806d9c0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f476c006fd0 tx=0x7f476c008040 comp rx=0 tx=0).stop 2024-03-04T09:26:25.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.039+0000 7f477b200640 1 -- 172.21.15.6:0/2037448900 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f477c15e810 msgr2=0x7f477c151cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:25.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.039+0000 7f477b200640 1 --2- 172.21.15.6:0/2037448900 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f477c15e810 0x7f477c151cd0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f4774009f70 tx=0x7f4774004a40 comp rx=0 tx=0).stop 2024-03-04T09:26:25.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.039+0000 7f477b200640 1 -- 172.21.15.6:0/2037448900 shutdown_connections 2024-03-04T09:26:25.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.039+0000 7f477b200640 1 --2- 172.21.15.6:0/2037448900 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f477c15e810 0x7f477c151cd0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:25.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.039+0000 7f477b200640 1 --2- 172.21.15.6:0/2037448900 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f475806b500 0x7f475806d9c0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:25.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.039+0000 7f477b200640 1 --2- 172.21.15.6:0/2037448900 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f477c1552b0 0x7f477c0b1120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:25.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.039+0000 7f477b200640 1 --2- 172.21.15.6:0/2037448900 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f477c0cd060 0x7f477c14b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:25.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.039+0000 7f477b200640 1 -- 172.21.15.6:0/2037448900 >> 172.21.15.6:0/2037448900 conn(0x7f477c06b3d0 msgr2=0x7f477c0cc920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:25.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.039+0000 7f477b200640 1 -- 172.21.15.6:0/2037448900 shutdown_connections 2024-03-04T09:26:25.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.039+0000 7f477b200640 1 -- 172.21.15.6:0/2037448900 wait complete. 2024-03-04T09:26:25.054 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd 2024-03-04T09:26:25.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.283+0000 7f4edae00640 1 Processor -- start 2024-03-04T09:26:25.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.283+0000 7f4edae00640 1 -- start start 2024-03-04T09:26:25.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.285+0000 7f4edae00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f4ed40d4a10 0x7f4ed40d0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:25.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.285+0000 7f4edae00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ed40b3710 con 0x7f4ed40d0c80 2024-03-04T09:26:25.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.285+0000 7f4edae00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ed40b39f0 con 0x7f4ed415e0b0 2024-03-04T09:26:25.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.285+0000 7f4edae00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ed40d10f0 con 0x7f4ed40d4a10 2024-03-04T09:26:25.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.286+0000 7f4ed9000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f4ed40d4a10 0x7f4ed40d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:25.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.286+0000 7f4eda400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f4ed40d0c80 0x7f4ed40b44d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:54604/0 (socket says 172.21.15.6:54604) 2024-03-04T09:26:25.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.286+0000 7f4eda400640 1 -- 172.21.15.6:0/927808200 learned_addr learned my addr 172.21.15.6:0/927808200 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:26:25.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.286+0000 7f4ed9000640 1 -- 172.21.15.6:0/927808200 >> v1:172.21.15.146:6789/0 conn(0x7f4ed415e0b0 legacy=0x7f4ed40cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:25.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.286+0000 7f4ed9000640 1 -- 172.21.15.6:0/927808200 >> v1:172.21.15.6:6789/0 conn(0x7f4ed40d0c80 legacy=0x7f4ed40b44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:25.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.286+0000 7f4ed9000640 1 -- 172.21.15.6:0/927808200 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ed40b3cd0 con 0x7f4ed40d4a10 2024-03-04T09:26:25.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.286+0000 7f4ed3e00640 1 -- 172.21.15.6:0/927808200 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2779733741 0 0) 0x7f4ed40b3710 con 0x7f4ed40d0c80 2024-03-04T09:26:25.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.287+0000 7f4ed9000640 1 --2- 172.21.15.6:0/927808200 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4ed40d4a10 0x7f4ed40d0740 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f4ec00097a0 tx=0x7f4ec002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=387fb7572fbd830c server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:25.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.288+0000 7f4ed3e00640 1 -- 172.21.15.6:0/927808200 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ec0035020 con 0x7f4ed40d4a10 2024-03-04T09:26:25.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.288+0000 7f4ed3e00640 1 -- 172.21.15.6:0/927808200 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ec0031830 con 0x7f4ed40d4a10 2024-03-04T09:26:25.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.288+0000 7f4ed3e00640 1 -- 172.21.15.6:0/927808200 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ec0031c00 con 0x7f4ed40d4a10 2024-03-04T09:26:25.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.288+0000 7f4edae00640 1 -- 172.21.15.6:0/927808200 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4ed40d4a10 msgr2=0x7f4ed40d0740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:25.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.288+0000 7f4edae00640 1 --2- 172.21.15.6:0/927808200 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4ed40d4a10 0x7f4ed40d0740 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f4ec00097a0 tx=0x7f4ec002b8d0 comp rx=0 tx=0).stop 2024-03-04T09:26:25.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.288+0000 7f4edae00640 1 -- 172.21.15.6:0/927808200 shutdown_connections 2024-03-04T09:26:25.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.288+0000 7f4edae00640 1 --2- 172.21.15.6:0/927808200 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4ed40d4a10 0x7f4ed40d0740 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:25.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.288+0000 7f4edae00640 1 -- 172.21.15.6:0/927808200 >> 172.21.15.6:0/927808200 conn(0x7f4ed40d3d30 msgr2=0x7f4ed40d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:25.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.289+0000 7f4edae00640 1 -- 172.21.15.6:0/927808200 shutdown_connections 2024-03-04T09:26:25.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.289+0000 7f4edae00640 1 -- 172.21.15.6:0/927808200 wait complete. 2024-03-04T09:26:25.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.290+0000 7f4edae00640 1 Processor -- start 2024-03-04T09:26:25.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.290+0000 7f4edae00640 1 -- start start 2024-03-04T09:26:25.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.291+0000 7f4edae00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4ed40d0c80 0x7f4ed4158b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:25.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.291+0000 7f4edae00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4ed40d4a10 0x7f4ed414f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:25.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.292+0000 7f4edae00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4ed415e0b0 0x7f4ed40b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:25.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.292+0000 7f4edae00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ed40cfa20 con 0x7f4ed40d0c80 2024-03-04T09:26:25.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.292+0000 7f4edae00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ed415e490 con 0x7f4ed40d4a10 2024-03-04T09:26:25.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.292+0000 7f4edae00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ed40cfc60 con 0x7f4ed415e0b0 2024-03-04T09:26:25.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.292+0000 7f4ed9a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4ed40d0c80 0x7f4ed4158b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:25.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.292+0000 7f4ed9a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4ed40d0c80 0x7f4ed4158b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45788/0 (socket says 172.21.15.6:45788) 2024-03-04T09:26:25.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.292+0000 7f4ed9a00640 1 -- 172.21.15.6:0/730446402 learned_addr learned my addr 172.21.15.6:0/730446402 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:25.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.292+0000 7f4ed9a00640 1 -- 172.21.15.6:0/730446402 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4ed415e0b0 msgr2=0x7f4ed40b4f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:25.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.292+0000 7f4ed9a00640 1 --2- 172.21.15.6:0/730446402 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4ed415e0b0 0x7f4ed40b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:25.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.292+0000 7f4ed9000640 1 --2- 172.21.15.6:0/730446402 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4ed40d4a10 0x7f4ed414f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:25.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.292+0000 7f4ed9a00640 1 -- 172.21.15.6:0/730446402 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4ed40d4a10 msgr2=0x7f4ed414f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:25.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.292+0000 7f4ed9a00640 1 --2- 172.21.15.6:0/730446402 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4ed40d4a10 0x7f4ed414f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:25.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.292+0000 7f4ed9a00640 1 -- 172.21.15.6:0/730446402 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ed41f7480 con 0x7f4ed40d0c80 2024-03-04T09:26:25.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.292+0000 7f4ed9000640 1 --2- 172.21.15.6:0/730446402 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4ed40d4a10 0x7f4ed414f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:26:25.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.293+0000 7f4ed9a00640 1 --2- 172.21.15.6:0/730446402 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4ed40d0c80 0x7f4ed4158b50 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f4ec800c7c0 tx=0x7f4ec800cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:25.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.293+0000 7f4ed1600640 1 -- 172.21.15.6:0/730446402 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ec8016020 con 0x7f4ed40d0c80 2024-03-04T09:26:25.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.293+0000 7f4ed1600640 1 -- 172.21.15.6:0/730446402 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ec800aa50 con 0x7f4ed40d0c80 2024-03-04T09:26:25.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.293+0000 7f4ed1600640 1 -- 172.21.15.6:0/730446402 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ec8012420 con 0x7f4ed40d0c80 2024-03-04T09:26:25.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.293+0000 7f4edae00640 1 -- 172.21.15.6:0/730446402 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ed41f7760 con 0x7f4ed40d0c80 2024-03-04T09:26:25.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.293+0000 7f4edae00640 1 -- 172.21.15.6:0/730446402 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ed41f7d20 con 0x7f4ed40d0c80 2024-03-04T09:26:25.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.294+0000 7f4edae00640 1 -- 172.21.15.6:0/730446402 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ed404f330 con 0x7f4ed40d0c80 2024-03-04T09:26:25.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.295+0000 7f4ed1600640 1 -- 172.21.15.6:0/730446402 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4ec8019020 con 0x7f4ed40d0c80 2024-03-04T09:26:25.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.296+0000 7f4ed1600640 1 --2- 172.21.15.6:0/730446402 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f4eb406b4d0 0x7f4eb406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:25.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.296+0000 7f4ed1600640 1 -- 172.21.15.6:0/730446402 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f4ec8089c30 con 0x7f4ed40d0c80 2024-03-04T09:26:25.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.297+0000 7f4ed9000640 1 --2- 172.21.15.6:0/730446402 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f4eb406b4d0 0x7f4eb406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:25.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.298+0000 7f4ed9000640 1 --2- 172.21.15.6:0/730446402 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f4eb406b4d0 0x7f4eb406d990 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f4ec0036000 tx=0x7f4ec0008030 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:25.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.301+0000 7f4ed1600640 1 -- 172.21.15.6:0/730446402 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4ec8020070 con 0x7f4ed40d0c80 2024-03-04T09:26:25.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.572+0000 7f4edae00640 1 -- 172.21.15.6:0/730446402 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd"} v 0) v1 -- 0x7f4ed40b19e0 con 0x7f4ed40d0c80 2024-03-04T09:26:25.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.578+0000 7f4ed1600640 1 -- 172.21.15.6:0/730446402 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd"}]=0 v7) v1 ==== 90+0+0 (secure 0 0 0) 0x7f4ec8058e00 con 0x7f4ed40d0c80 2024-03-04T09:26:25.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.580+0000 7f4edae00640 1 -- 172.21.15.6:0/730446402 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f4eb406b4d0 msgr2=0x7f4eb406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:25.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.580+0000 7f4edae00640 1 --2- 172.21.15.6:0/730446402 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f4eb406b4d0 0x7f4eb406d990 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f4ec0036000 tx=0x7f4ec0008030 comp rx=0 tx=0).stop 2024-03-04T09:26:25.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.580+0000 7f4edae00640 1 -- 172.21.15.6:0/730446402 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4ed40d0c80 msgr2=0x7f4ed4158b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:25.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.580+0000 7f4edae00640 1 --2- 172.21.15.6:0/730446402 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4ed40d0c80 0x7f4ed4158b50 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f4ec800c7c0 tx=0x7f4ec800cc80 comp rx=0 tx=0).stop 2024-03-04T09:26:25.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.581+0000 7f4edae00640 1 -- 172.21.15.6:0/730446402 shutdown_connections 2024-03-04T09:26:25.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.581+0000 7f4edae00640 1 --2- 172.21.15.6:0/730446402 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f4eb406b4d0 0x7f4eb406d990 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:25.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.581+0000 7f4edae00640 1 --2- 172.21.15.6:0/730446402 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4ed415e0b0 0x7f4ed40b4f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:25.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.581+0000 7f4edae00640 1 --2- 172.21.15.6:0/730446402 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4ed40d4a10 0x7f4ed414f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:25.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.581+0000 7f4edae00640 1 --2- 172.21.15.6:0/730446402 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4ed40d0c80 0x7f4ed4158b50 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:25.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.581+0000 7f4edae00640 1 -- 172.21.15.6:0/730446402 >> 172.21.15.6:0/730446402 conn(0x7f4ed40d3d30 msgr2=0x7f4ed40b1cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:25.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.581+0000 7f4edae00640 1 -- 172.21.15.6:0/730446402 shutdown_connections 2024-03-04T09:26:25.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.581+0000 7f4edae00640 1 -- 172.21.15.6:0/730446402 wait complete. 2024-03-04T09:26:25.597 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd-mirror 2024-03-04T09:26:25.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.824+0000 7f03d6c00640 1 Processor -- start 2024-03-04T09:26:25.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.825+0000 7f03d6c00640 1 -- start start 2024-03-04T09:26:25.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.826+0000 7f03d6c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f03d0070f10 0x7f03d00cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:25.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.826+0000 7f03d6c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f03d00cce90 0x7f03d015c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:25.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.826+0000 7f03d6c00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03d00714d0 con 0x7f03d0145300 2024-03-04T09:26:25.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.826+0000 7f03d6c00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03d00cd300 con 0x7f03d00cce90 2024-03-04T09:26:25.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.826+0000 7f03d6c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03d00717b0 con 0x7f03d0070f10 2024-03-04T09:26:25.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.826+0000 7f03d6200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f03d00cce90 0x7f03d015c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:25.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.827+0000 7f03d5800640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f03d0145300 0x7f03d00b49a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:54608/0 (socket says 172.21.15.6:54608) 2024-03-04T09:26:25.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.827+0000 7f03d5800640 1 -- 172.21.15.6:0/688466677 learned_addr learned my addr 172.21.15.6:0/688466677 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:26:25.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.827+0000 7f03d4e00640 1 --2- 172.21.15.6:0/688466677 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f03d0070f10 0x7f03d00cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:25.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.827+0000 7f03d6200640 1 -- 172.21.15.6:0/688466677 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f03d0070f10 msgr2=0x7f03d00cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:25.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.827+0000 7f03d6200640 1 --2- 172.21.15.6:0/688466677 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f03d0070f10 0x7f03d00cc950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:25.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.827+0000 7f03d6200640 1 -- 172.21.15.6:0/688466677 >> v1:172.21.15.6:6789/0 conn(0x7f03d0145300 legacy=0x7f03d00b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:25.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.827+0000 7f03d6200640 1 -- 172.21.15.6:0/688466677 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03d0071a90 con 0x7f03d00cce90 2024-03-04T09:26:25.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.827+0000 7f03cfe00640 1 -- 172.21.15.6:0/688466677 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2027829874 0 0) 0x7f03d00714d0 con 0x7f03d0145300 2024-03-04T09:26:25.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.828+0000 7f03d6200640 1 --2- 172.21.15.6:0/688466677 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f03d00cce90 0x7f03d015c570 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f03c8009d90 tx=0x7f03c802bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c66b87246144762 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:25.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.829+0000 7f03cfe00640 1 -- 172.21.15.6:0/688466677 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03c8039070 con 0x7f03d00cce90 2024-03-04T09:26:25.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.829+0000 7f03cfe00640 1 -- 172.21.15.6:0/688466677 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03c802cc50 con 0x7f03d00cce90 2024-03-04T09:26:25.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.829+0000 7f03cfe00640 1 -- 172.21.15.6:0/688466677 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03c8034440 con 0x7f03d00cce90 2024-03-04T09:26:25.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.829+0000 7f03d6c00640 1 -- 172.21.15.6:0/688466677 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f03d00cce90 msgr2=0x7f03d015c570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:25.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.829+0000 7f03d6c00640 1 --2- 172.21.15.6:0/688466677 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f03d00cce90 0x7f03d015c570 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f03c8009d90 tx=0x7f03c802bea0 comp rx=0 tx=0).stop 2024-03-04T09:26:25.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.829+0000 7f03d6c00640 1 -- 172.21.15.6:0/688466677 shutdown_connections 2024-03-04T09:26:25.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.829+0000 7f03d6c00640 1 --2- 172.21.15.6:0/688466677 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f03d00cce90 0x7f03d015c570 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:25.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.829+0000 7f03d6c00640 1 --2- 172.21.15.6:0/688466677 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f03d0070f10 0x7f03d00cc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:25.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.829+0000 7f03d6c00640 1 -- 172.21.15.6:0/688466677 >> 172.21.15.6:0/688466677 conn(0x7f03d006b3d0 msgr2=0x7f03d006b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:25.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.829+0000 7f03d6c00640 1 -- 172.21.15.6:0/688466677 shutdown_connections 2024-03-04T09:26:25.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.829+0000 7f03d6c00640 1 -- 172.21.15.6:0/688466677 wait complete. 2024-03-04T09:26:25.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.831+0000 7f03d6c00640 1 Processor -- start 2024-03-04T09:26:25.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.831+0000 7f03d6c00640 1 -- start start 2024-03-04T09:26:25.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.832+0000 7f03d6c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f03d0070f10 0x7f03d013dd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:25.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.832+0000 7f03d6c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f03d00cce90 0x7f03d0142f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:25.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.832+0000 7f03d6c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f03d0145300 0x7f03d01434d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:25.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.832+0000 7f03d6c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03d015cab0 con 0x7f03d0145300 2024-03-04T09:26:25.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.832+0000 7f03d6c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03d01456e0 con 0x7f03d0070f10 2024-03-04T09:26:25.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.832+0000 7f03d6c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03d015cc30 con 0x7f03d00cce90 2024-03-04T09:26:25.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.833+0000 7f03d6200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f03d0145300 0x7f03d01434d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:25.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.833+0000 7f03d6200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f03d0145300 0x7f03d01434d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45800/0 (socket says 172.21.15.6:45800) 2024-03-04T09:26:25.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.833+0000 7f03d6200640 1 -- 172.21.15.6:0/3209802952 learned_addr learned my addr 172.21.15.6:0/3209802952 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:25.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.833+0000 7f03d5800640 1 --2- 172.21.15.6:0/3209802952 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f03d0070f10 0x7f03d013dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:25.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.833+0000 7f03d4e00640 1 --2- 172.21.15.6:0/3209802952 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f03d00cce90 0x7f03d0142f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:25.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.833+0000 7f03d6200640 1 -- 172.21.15.6:0/3209802952 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f03d00cce90 msgr2=0x7f03d0142f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:25.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.833+0000 7f03d6200640 1 --2- 172.21.15.6:0/3209802952 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f03d00cce90 0x7f03d0142f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:25.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.833+0000 7f03d6200640 1 -- 172.21.15.6:0/3209802952 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f03d0070f10 msgr2=0x7f03d013dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:25.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.833+0000 7f03d6200640 1 --2- 172.21.15.6:0/3209802952 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f03d0070f10 0x7f03d013dd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:25.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.833+0000 7f03d6200640 1 -- 172.21.15.6:0/3209802952 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03d0140820 con 0x7f03d0145300 2024-03-04T09:26:25.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.833+0000 7f03d5800640 1 --2- 172.21.15.6:0/3209802952 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f03d0070f10 0x7f03d013dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:26:25.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.833+0000 7f03d6200640 1 --2- 172.21.15.6:0/3209802952 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f03d0145300 0x7f03d01434d0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f03c8009d90 tx=0x7f03c8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:25.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.834+0000 7f03cd600640 1 -- 172.21.15.6:0/3209802952 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03c8039050 con 0x7f03d0145300 2024-03-04T09:26:25.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.834+0000 7f03cd600640 1 -- 172.21.15.6:0/3209802952 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03c8045070 con 0x7f03d0145300 2024-03-04T09:26:25.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.834+0000 7f03cd600640 1 -- 172.21.15.6:0/3209802952 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03c803e610 con 0x7f03d0145300 2024-03-04T09:26:25.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.834+0000 7f03d6c00640 1 -- 172.21.15.6:0/3209802952 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03d0140b00 con 0x7f03d0145300 2024-03-04T09:26:25.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.834+0000 7f03d6c00640 1 -- 172.21.15.6:0/3209802952 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03d01410c0 con 0x7f03d0145300 2024-03-04T09:26:25.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.835+0000 7f03d6c00640 1 -- 172.21.15.6:0/3209802952 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03d004f330 con 0x7f03d0145300 2024-03-04T09:26:25.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.836+0000 7f03cd600640 1 -- 172.21.15.6:0/3209802952 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f03c803e7b0 con 0x7f03d0145300 2024-03-04T09:26:25.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.837+0000 7f03cd600640 1 --2- 172.21.15.6:0/3209802952 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f03b006b4d0 0x7f03b006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:25.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.838+0000 7f03cd600640 1 -- 172.21.15.6:0/3209802952 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f03c80acbb0 con 0x7f03d0145300 2024-03-04T09:26:25.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.838+0000 7f03d5800640 1 --2- 172.21.15.6:0/3209802952 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f03b006b4d0 0x7f03b006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:25.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.839+0000 7f03d5800640 1 --2- 172.21.15.6:0/3209802952 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f03b006b4d0 0x7f03b006d990 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f03c4000f60 tx=0x7f03c4009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:25.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:25.842+0000 7f03cd600640 1 -- 172.21.15.6:0/3209802952 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f03c807bc50 con 0x7f03d0145300 2024-03-04T09:26:26.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.115+0000 7f03d6c00640 1 -- 172.21.15.6:0/3209802952 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"} v 0) v1 -- 0x7f03d00ad920 con 0x7f03d0145300 2024-03-04T09:26:26.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.120+0000 7f03cd600640 1 -- 172.21.15.6:0/3209802952 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"}]=0 v8) v1 ==== 97+0+0 (secure 0 0 0) 0x7f03c807b550 con 0x7f03d0145300 2024-03-04T09:26:26.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.122+0000 7f03d6c00640 1 -- 172.21.15.6:0/3209802952 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f03b006b4d0 msgr2=0x7f03b006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:26.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.122+0000 7f03d6c00640 1 --2- 172.21.15.6:0/3209802952 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f03b006b4d0 0x7f03b006d990 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f03c4000f60 tx=0x7f03c4009040 comp rx=0 tx=0).stop 2024-03-04T09:26:26.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.122+0000 7f03d6c00640 1 -- 172.21.15.6:0/3209802952 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f03d0145300 msgr2=0x7f03d01434d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:26.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.122+0000 7f03d6c00640 1 --2- 172.21.15.6:0/3209802952 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f03d0145300 0x7f03d01434d0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f03c8009d90 tx=0x7f03c8004a40 comp rx=0 tx=0).stop 2024-03-04T09:26:26.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.122+0000 7f03d6c00640 1 -- 172.21.15.6:0/3209802952 shutdown_connections 2024-03-04T09:26:26.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.122+0000 7f03d6c00640 1 --2- 172.21.15.6:0/3209802952 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f03d0145300 0x7f03d01434d0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:26.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.122+0000 7f03d6c00640 1 --2- 172.21.15.6:0/3209802952 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f03b006b4d0 0x7f03b006d990 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:26.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.122+0000 7f03d6c00640 1 --2- 172.21.15.6:0/3209802952 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f03d00cce90 0x7f03d0142f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:26.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.122+0000 7f03d6c00640 1 --2- 172.21.15.6:0/3209802952 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f03d0070f10 0x7f03d013dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:26.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.122+0000 7f03d6c00640 1 -- 172.21.15.6:0/3209802952 >> 172.21.15.6:0/3209802952 conn(0x7f03d006b3d0 msgr2=0x7f03d00cd930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:26.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.123+0000 7f03d6c00640 1 -- 172.21.15.6:0/3209802952 shutdown_connections 2024-03-04T09:26:26.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.123+0000 7f03d6c00640 1 -- 172.21.15.6:0/3209802952 wait complete. 2024-03-04T09:26:26.139 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rgw 2024-03-04T09:26:26.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.367+0000 7f20b8c00640 1 Processor -- start 2024-03-04T09:26:26.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.368+0000 7f20b8c00640 1 -- start start 2024-03-04T09:26:26.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.369+0000 7f20b8c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f20b40cede0 0x7f20b40b3f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:26.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.369+0000 7f20b8c00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f20b4071b60 0x7f20b415b320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:26.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.369+0000 7f20b8c00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20b40b31d0 con 0x7f20b40b4a70 2024-03-04T09:26:26.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.369+0000 7f20b8c00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20b40b4ed0 con 0x7f20b4071b60 2024-03-04T09:26:26.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.369+0000 7f20b8c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20b40b3b00 con 0x7f20b40cede0 2024-03-04T09:26:26.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.370+0000 7f20b3e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f20b4071b60 0x7f20b415b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:26.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.370+0000 7f20b3400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f20b40cede0 0x7f20b40b3f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:26.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.370+0000 7f20aaa00640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f20b40b4a70 0x7f20b4071450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.6:33564/0 (socket says 172.21.15.6:33564) 2024-03-04T09:26:26.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.370+0000 7f20aaa00640 1 -- 172.21.15.6:0/3865115605 learned_addr learned my addr 172.21.15.6:0/3865115605 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:26:26.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.370+0000 7f20b2a00640 1 -- 172.21.15.6:0/3865115605 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3996229133 0 0) 0x7f20b40b31d0 con 0x7f20b40b4a70 2024-03-04T09:26:26.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.371+0000 7f20b2a00640 1 -- 172.21.15.6:0/3865115605 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2098003440 con 0x7f20b40b4a70 2024-03-04T09:26:26.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.371+0000 7f20b3400640 1 -- 172.21.15.6:0/3865115605 >> v2:172.21.15.146:3300/0 conn(0x7f20b4071b60 msgr2=0x7f20b415b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:26.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.371+0000 7f20b3400640 1 --2- 172.21.15.6:0/3865115605 >> v2:172.21.15.146:3300/0 conn(0x7f20b4071b60 0x7f20b415b320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:26.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.371+0000 7f20b3400640 1 -- 172.21.15.6:0/3865115605 >> v1:172.21.15.146:6789/0 conn(0x7f20b40b4a70 legacy=0x7f20b4071450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:26.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.371+0000 7f20b3400640 1 -- 172.21.15.6:0/3865115605 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20b40b34b0 con 0x7f20b40cede0 2024-03-04T09:26:26.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.371+0000 7f20b3e00640 1 --2- 172.21.15.6:0/3865115605 >> v2:172.21.15.146:3300/0 conn(0x7f20b4071b60 0x7f20b415b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T09:26:26.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.372+0000 7f20b3400640 1 --2- 172.21.15.6:0/3865115605 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f20b40cede0 0x7f20b40b3f90 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f20a40097a0 tx=0x7f20a402b820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d2999844795479d0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:26.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.372+0000 7f20b2a00640 1 -- 172.21.15.6:0/3865115605 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20a4031780 con 0x7f20b40cede0 2024-03-04T09:26:26.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.372+0000 7f20b2a00640 1 -- 172.21.15.6:0/3865115605 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20a4037070 con 0x7f20b40cede0 2024-03-04T09:26:26.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.372+0000 7f20b2a00640 1 -- 172.21.15.6:0/3865115605 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20a4031b20 con 0x7f20b40cede0 2024-03-04T09:26:26.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.373+0000 7f20b8c00640 1 -- 172.21.15.6:0/3865115605 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f20b40cede0 msgr2=0x7f20b40b3f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:26.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.373+0000 7f20b8c00640 1 --2- 172.21.15.6:0/3865115605 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f20b40cede0 0x7f20b40b3f90 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f20a40097a0 tx=0x7f20a402b820 comp rx=0 tx=0).stop 2024-03-04T09:26:26.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.373+0000 7f20b8c00640 1 -- 172.21.15.6:0/3865115605 shutdown_connections 2024-03-04T09:26:26.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.373+0000 7f20b8c00640 1 --2- 172.21.15.6:0/3865115605 >> v2:172.21.15.146:3300/0 conn(0x7f20b4071b60 0x7f20b415b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:26.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.373+0000 7f20b8c00640 1 --2- 172.21.15.6:0/3865115605 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f20b40cede0 0x7f20b40b3f90 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:26.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.373+0000 7f20b8c00640 1 -- 172.21.15.6:0/3865115605 >> 172.21.15.6:0/3865115605 conn(0x7f20b406b3d0 msgr2=0x7f20b406b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:26.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.373+0000 7f20b8c00640 1 -- 172.21.15.6:0/3865115605 shutdown_connections 2024-03-04T09:26:26.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.373+0000 7f20b8c00640 1 -- 172.21.15.6:0/3865115605 wait complete. 2024-03-04T09:26:26.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.375+0000 7f20b8c00640 1 Processor -- start 2024-03-04T09:26:26.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.375+0000 7f20b8c00640 1 -- start start 2024-03-04T09:26:26.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.376+0000 7f20b8c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f20b4071b60 0x7f20b41f3760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:26.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.376+0000 7f20b8c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f20b40b4a70 0x7f20b413f910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:26.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.376+0000 7f20b8c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f20b40cede0 0x7f20b415ba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:26.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.376+0000 7f20b8c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20b40cf1c0 con 0x7f20b4071b60 2024-03-04T09:26:26.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.376+0000 7f20b8c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20b40b2f70 con 0x7f20b40b4a70 2024-03-04T09:26:26.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.376+0000 7f20b8c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20b40cc950 con 0x7f20b40cede0 2024-03-04T09:26:26.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.376+0000 7f20b3400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f20b4071b60 0x7f20b41f3760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:26.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.377+0000 7f20b3400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f20b4071b60 0x7f20b41f3760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45816/0 (socket says 172.21.15.6:45816) 2024-03-04T09:26:26.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.377+0000 7f20b3400640 1 -- 172.21.15.6:0/1032988087 learned_addr learned my addr 172.21.15.6:0/1032988087 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:26.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.377+0000 7f20b3e00640 1 --2- 172.21.15.6:0/1032988087 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f20b40cede0 0x7f20b415ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:26.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.377+0000 7f20aaa00640 1 --2- 172.21.15.6:0/1032988087 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f20b40b4a70 0x7f20b413f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:26.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.377+0000 7f20b3400640 1 -- 172.21.15.6:0/1032988087 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f20b40cede0 msgr2=0x7f20b415ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:26.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.377+0000 7f20b3400640 1 --2- 172.21.15.6:0/1032988087 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f20b40cede0 0x7f20b415ba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:26.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.377+0000 7f20b3400640 1 -- 172.21.15.6:0/1032988087 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f20b40b4a70 msgr2=0x7f20b413f910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:26.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.377+0000 7f20b3400640 1 --2- 172.21.15.6:0/1032988087 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f20b40b4a70 0x7f20b413f910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:26.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.377+0000 7f20b3400640 1 -- 172.21.15.6:0/1032988087 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20b4140240 con 0x7f20b4071b60 2024-03-04T09:26:26.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.377+0000 7f20aaa00640 1 --2- 172.21.15.6:0/1032988087 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f20b40b4a70 0x7f20b413f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:26:26.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.377+0000 7f20b3400640 1 --2- 172.21.15.6:0/1032988087 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f20b4071b60 0x7f20b41f3760 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f20a4034040 tx=0x7f20a40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:26.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.378+0000 7f20abe00640 1 -- 172.21.15.6:0/1032988087 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20a4032610 con 0x7f20b4071b60 2024-03-04T09:26:26.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.378+0000 7f20abe00640 1 -- 172.21.15.6:0/1032988087 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20a40327d0 con 0x7f20b4071b60 2024-03-04T09:26:26.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.378+0000 7f20b8c00640 1 -- 172.21.15.6:0/1032988087 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20b4140520 con 0x7f20b4071b60 2024-03-04T09:26:26.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.378+0000 7f20abe00640 1 -- 172.21.15.6:0/1032988087 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20a4032bc0 con 0x7f20b4071b60 2024-03-04T09:26:26.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.378+0000 7f20b8c00640 1 -- 172.21.15.6:0/1032988087 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20b4140ae0 con 0x7f20b4071b60 2024-03-04T09:26:26.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.379+0000 7f20b8c00640 1 -- 172.21.15.6:0/1032988087 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20b41424a0 con 0x7f20b4071b60 2024-03-04T09:26:26.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.380+0000 7f20abe00640 1 -- 172.21.15.6:0/1032988087 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f20a404c060 con 0x7f20b4071b60 2024-03-04T09:26:26.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.381+0000 7f20abe00640 1 --2- 172.21.15.6:0/1032988087 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f209006b500 0x7f209006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:26.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.381+0000 7f20abe00640 1 -- 172.21.15.6:0/1032988087 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f20a40ab3c0 con 0x7f20b4071b60 2024-03-04T09:26:26.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.381+0000 7f20aaa00640 1 --2- 172.21.15.6:0/1032988087 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f209006b500 0x7f209006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:26.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.382+0000 7f20aaa00640 1 --2- 172.21.15.6:0/1032988087 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f209006b500 0x7f209006d9c0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f20b415cd70 tx=0x7f209c009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:26.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.385+0000 7f20abe00640 1 -- 172.21.15.6:0/1032988087 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f20a4040070 con 0x7f20b4071b60 2024-03-04T09:26:26.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.652+0000 7f20b8c00640 1 -- 172.21.15.6:0/1032988087 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rgw"} v 0) v1 -- 0x7f20b404f330 con 0x7f20b4071b60 2024-03-04T09:26:26.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.656+0000 7f20abe00640 1 -- 172.21.15.6:0/1032988087 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rgw"}]=0 v9) v1 ==== 90+0+0 (secure 0 0 0) 0x7f20a402f050 con 0x7f20b4071b60 2024-03-04T09:26:26.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.658+0000 7f20b8c00640 1 -- 172.21.15.6:0/1032988087 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f209006b500 msgr2=0x7f209006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:26.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.658+0000 7f20b8c00640 1 --2- 172.21.15.6:0/1032988087 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f209006b500 0x7f209006d9c0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f20b415cd70 tx=0x7f209c009040 comp rx=0 tx=0).stop 2024-03-04T09:26:26.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.658+0000 7f20b8c00640 1 -- 172.21.15.6:0/1032988087 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f20b4071b60 msgr2=0x7f20b41f3760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:26.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.658+0000 7f20b8c00640 1 --2- 172.21.15.6:0/1032988087 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f20b4071b60 0x7f20b41f3760 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f20a4034040 tx=0x7f20a40046c0 comp rx=0 tx=0).stop 2024-03-04T09:26:26.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.659+0000 7f20b8c00640 1 -- 172.21.15.6:0/1032988087 shutdown_connections 2024-03-04T09:26:26.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.659+0000 7f20b8c00640 1 --2- 172.21.15.6:0/1032988087 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f209006b500 0x7f209006d9c0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:26.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.659+0000 7f20b8c00640 1 --2- 172.21.15.6:0/1032988087 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f20b40cede0 0x7f20b415ba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:26.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.659+0000 7f20b8c00640 1 --2- 172.21.15.6:0/1032988087 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f20b40b4a70 0x7f20b413f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:26.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.659+0000 7f20b8c00640 1 --2- 172.21.15.6:0/1032988087 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f20b4071b60 0x7f20b41f3760 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:26.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.659+0000 7f20b8c00640 1 -- 172.21.15.6:0/1032988087 >> 172.21.15.6:0/1032988087 conn(0x7f20b406b3d0 msgr2=0x7f20b406ccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:26.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.659+0000 7f20b8c00640 1 -- 172.21.15.6:0/1032988087 shutdown_connections 2024-03-04T09:26:26.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.659+0000 7f20b8c00640 1 -- 172.21.15.6:0/1032988087 wait complete. 2024-03-04T09:26:26.675 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T09:26:26.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.895+0000 7f29bfe00640 1 Processor -- start 2024-03-04T09:26:26.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.895+0000 7f29bfe00640 1 -- start start 2024-03-04T09:26:26.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.896+0000 7f29bfe00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f29c01552b0 0x7f29c015b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:26.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.896+0000 7f29bfe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f29c00cd060 0x7f29c00b3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:26.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.896+0000 7f29bfe00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29c015ed10 con 0x7f29c015e810 2024-03-04T09:26:26.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.896+0000 7f29bfe00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29c015eff0 con 0x7f29c00cd060 2024-03-04T09:26:26.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.896+0000 7f29bfe00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29c015f2d0 con 0x7f29c01552b0 2024-03-04T09:26:26.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.897+0000 7f29b6000640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f29c015e810 0x7f29c00cc950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:54616/0 (socket says 172.21.15.6:54616) 2024-03-04T09:26:26.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.897+0000 7f29b6000640 1 -- 172.21.15.6:0/3334419741 learned_addr learned my addr 172.21.15.6:0/3334419741 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:26:26.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.897+0000 7f29bf400640 1 --2- 172.21.15.6:0/3334419741 >> v2:172.21.15.6:3300/0 conn(0x7f29c00cd060 0x7f29c00b3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:26.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.897+0000 7f29bea00640 1 --2- 172.21.15.6:0/3334419741 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f29c01552b0 0x7f29c015b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:26.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.897+0000 7f29bf400640 1 -- 172.21.15.6:0/3334419741 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f29c01552b0 msgr2=0x7f29c015b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:26.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.897+0000 7f29bf400640 1 --2- 172.21.15.6:0/3334419741 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f29c01552b0 0x7f29c015b1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:26.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.897+0000 7f29bf400640 1 -- 172.21.15.6:0/3334419741 >> v1:172.21.15.6:6789/0 conn(0x7f29c015e810 legacy=0x7f29c00cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:26.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.897+0000 7f29bf400640 1 -- 172.21.15.6:0/3334419741 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29c015f5b0 con 0x7f29c00cd060 2024-03-04T09:26:26.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.898+0000 7f29bf400640 1 --2- 172.21.15.6:0/3334419741 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f29c00cd060 0x7f29c00b3cb0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f29b8009d90 tx=0x7f29b802be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c874f2ad4a000b1d server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:26.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.899+0000 7f29be000640 1 -- 172.21.15.6:0/3334419741 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29b8039070 con 0x7f29c00cd060 2024-03-04T09:26:26.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.899+0000 7f29be000640 1 -- 172.21.15.6:0/3334419741 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29b802cbd0 con 0x7f29c00cd060 2024-03-04T09:26:26.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.899+0000 7f29be000640 1 -- 172.21.15.6:0/3334419741 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29b8034440 con 0x7f29c00cd060 2024-03-04T09:26:26.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.899+0000 7f29bfe00640 1 -- 172.21.15.6:0/3334419741 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f29c00cd060 msgr2=0x7f29c00b3cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:26.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.899+0000 7f29bfe00640 1 --2- 172.21.15.6:0/3334419741 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f29c00cd060 0x7f29c00b3cb0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f29b8009d90 tx=0x7f29b802be20 comp rx=0 tx=0).stop 2024-03-04T09:26:26.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.899+0000 7f29bfe00640 1 -- 172.21.15.6:0/3334419741 shutdown_connections 2024-03-04T09:26:26.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.899+0000 7f29bfe00640 1 --2- 172.21.15.6:0/3334419741 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f29c00cd060 0x7f29c00b3cb0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:26.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.899+0000 7f29bfe00640 1 --2- 172.21.15.6:0/3334419741 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f29c01552b0 0x7f29c015b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:26.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.899+0000 7f29bfe00640 1 -- 172.21.15.6:0/3334419741 >> 172.21.15.6:0/3334419741 conn(0x7f29c006b3d0 msgr2=0x7f29c006b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:26.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.899+0000 7f29bfe00640 1 -- 172.21.15.6:0/3334419741 shutdown_connections 2024-03-04T09:26:26.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.899+0000 7f29bfe00640 1 -- 172.21.15.6:0/3334419741 wait complete. 2024-03-04T09:26:26.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.901+0000 7f29bfe00640 1 Processor -- start 2024-03-04T09:26:26.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.901+0000 7f29bfe00640 1 -- start start 2024-03-04T09:26:26.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.902+0000 7f29bfe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f29c00cd060 0x7f29c00b1ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:26.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.902+0000 7f29bfe00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f29c01552b0 0x7f29c0146230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:26.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.902+0000 7f29bfe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f29c015e810 0x7f29c0158b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:26.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.902+0000 7f29bfe00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29c006dd30 con 0x7f29c00cd060 2024-03-04T09:26:26.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.902+0000 7f29bfe00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29c00b3760 con 0x7f29c015e810 2024-03-04T09:26:26.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.902+0000 7f29bfe00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29c006deb0 con 0x7f29c01552b0 2024-03-04T09:26:26.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.902+0000 7f29bea00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f29c00cd060 0x7f29c00b1ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:26.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.903+0000 7f29bea00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f29c00cd060 0x7f29c00b1ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45834/0 (socket says 172.21.15.6:45834) 2024-03-04T09:26:26.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.903+0000 7f29bea00640 1 -- 172.21.15.6:0/1427046647 learned_addr learned my addr 172.21.15.6:0/1427046647 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:26.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.903+0000 7f29bf400640 1 --2- 172.21.15.6:0/1427046647 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f29c015e810 0x7f29c0158b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:26.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.903+0000 7f29bea00640 1 -- 172.21.15.6:0/1427046647 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f29c01552b0 msgr2=0x7f29c0146230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:26.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.903+0000 7f29bea00640 1 --2- 172.21.15.6:0/1427046647 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f29c01552b0 0x7f29c0146230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:26.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.903+0000 7f29bea00640 1 -- 172.21.15.6:0/1427046647 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f29c015e810 msgr2=0x7f29c0158b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:26.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.903+0000 7f29bea00640 1 --2- 172.21.15.6:0/1427046647 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f29c015e810 0x7f29c0158b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:26.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.903+0000 7f29bea00640 1 -- 172.21.15.6:0/1427046647 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29c0157520 con 0x7f29c00cd060 2024-03-04T09:26:26.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.903+0000 7f29bea00640 1 --2- 172.21.15.6:0/1427046647 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f29c00cd060 0x7f29c00b1ab0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f29b000d800 tx=0x7f29b000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:26.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.904+0000 7f29b7400640 1 -- 172.21.15.6:0/1427046647 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29b000b6c0 con 0x7f29c00cd060 2024-03-04T09:26:26.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.904+0000 7f29b7400640 1 -- 172.21.15.6:0/1427046647 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29b0019070 con 0x7f29c00cd060 2024-03-04T09:26:26.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.904+0000 7f29b7400640 1 -- 172.21.15.6:0/1427046647 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29b000ba80 con 0x7f29c00cd060 2024-03-04T09:26:26.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.904+0000 7f29bfe00640 1 -- 172.21.15.6:0/1427046647 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29c0157800 con 0x7f29c00cd060 2024-03-04T09:26:26.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.904+0000 7f29bfe00640 1 -- 172.21.15.6:0/1427046647 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29c0157dc0 con 0x7f29c00cd060 2024-03-04T09:26:26.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.905+0000 7f29bfe00640 1 -- 172.21.15.6:0/1427046647 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29c004f330 con 0x7f29c00cd060 2024-03-04T09:26:26.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.906+0000 7f29b7400640 1 -- 172.21.15.6:0/1427046647 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f29b000bc20 con 0x7f29c00cd060 2024-03-04T09:26:26.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.907+0000 7f29b7400640 1 --2- 172.21.15.6:0/1427046647 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f299c06b500 0x7f299c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:26.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.907+0000 7f29b7400640 1 -- 172.21.15.6:0/1427046647 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f29b008a350 con 0x7f29c00cd060 2024-03-04T09:26:26.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.907+0000 7f29b6000640 1 --2- 172.21.15.6:0/1427046647 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f299c06b500 0x7f299c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:26.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.908+0000 7f29b6000640 1 --2- 172.21.15.6:0/1427046647 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f299c06b500 0x7f299c06d9c0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f29a8000f60 tx=0x7f29a8007a80 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:26.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:26.911+0000 7f29b7400640 1 -- 172.21.15.6:0/1427046647 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f29b0059510 con 0x7f29c00cd060 2024-03-04T09:26:27.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.111+0000 7f29bfe00640 1 -- 172.21.15.6:0/1427046647 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f29c00cf4a0 con 0x7f29c00cd060 2024-03-04T09:26:27.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.113+0000 7f29b7400640 1 -- 172.21.15.6:0/1427046647 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) v1 ==== 74+0+12843 (secure 0 0 0) 0x7f29b0058e10 con 0x7f29c00cd060 2024-03-04T09:26:27.111 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:26:27.111 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":29,"fsid":"edce0578-5ae4-4402-b45a-552bdef84d7c","created":"2024-03-04T09:24:54.523780+0000","modified":"2024-03-04T09:26:22.921871+0000","last_up_change":"2024-03-04T09:25:07.089049+0000","last_in_change":"2024-03-04T09:24:57.982077+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T09:25:07.935198+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a96574f-81f1-4bc6-8463-757520b84a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6825","nonce":1364449401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6827","nonce":1364449401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6831","nonce":1364449401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6829","nonce":1364449401}]},"public_addr":"172.21.15.6:6825/1364449401","cluster_addr":"172.21.15.6:6827/1364449401","heartbeat_back_addr":"172.21.15.6:6831/1364449401","heartbeat_front_addr":"172.21.15.6:6829/1364449401","state":["exists","up"]},{"osd":1,"uuid":"7d79b9c1-6b87-476a-bc36-7db5ee64cd90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6809","nonce":2494103078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6811","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6813","nonce":2494103078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6821","nonce":2494103078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6817","nonce":2494103078}]},"public_addr":"172.21.15.6:6809/2494103078","cluster_addr":"172.21.15.6:6813/2494103078","heartbeat_back_addr":"172.21.15.6:6821/2494103078","heartbeat_front_addr":"172.21.15.6:6817/2494103078","state":["exists","up"]},{"osd":2,"uuid":"492331b9-4ee9-40b6-bacd-85d44cd6c45a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6812","nonce":1500066681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6816","nonce":1500066681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6824","nonce":1500066681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6820","nonce":1500066681}]},"public_addr":"172.21.15.6:6812/1500066681","cluster_addr":"172.21.15.6:6816/1500066681","heartbeat_back_addr":"172.21.15.6:6824/1500066681","heartbeat_front_addr":"172.21.15.6:6820/1500066681","state":["exists","up"]},{"osd":3,"uuid":"21bb2efe-9ba7-4bdc-bc31-180cd1102e21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6801","nonce":1558806123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6803","nonce":1558806123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6807","nonce":1558806123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6805","nonce":1558806123}]},"public_addr":"172.21.15.6:6801/1558806123","cluster_addr":"172.21.15.6:6803/1558806123","heartbeat_back_addr":"172.21.15.6:6807/1558806123","heartbeat_front_addr":"172.21.15.6:6805/1558806123","state":["exists","up"]},{"osd":4,"uuid":"6e2c4b71-19d9-4737-83d9-558705d8644a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6801","nonce":1526135780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6803","nonce":1526135780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6807","nonce":1526135780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6805","nonce":1526135780}]},"public_addr":"172.21.15.146:6801/1526135780","cluster_addr":"172.21.15.146:6803/1526135780","heartbeat_back_addr":"172.21.15.146:6807/1526135780","heartbeat_front_addr":"172.21.15.146:6805/1526135780","state":["exists","up"]},{"osd":5,"uuid":"fb69c525-f1be-4b7d-ab32-6c29b0d6fddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6809","nonce":460262507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6811","nonce":460262507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6815","nonce":460262507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6813","nonce":460262507}]},"public_addr":"172.21.15.146:6809/460262507","cluster_addr":"172.21.15.146:6811/460262507","heartbeat_back_addr":"172.21.15.146:6815/460262507","heartbeat_front_addr":"172.21.15.146:6813/460262507","state":["exists","up"]},{"osd":6,"uuid":"c32e7fed-370d-4401-8ce0-6a4335b0d168","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6823","nonce":1392897493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6827","nonce":1392897493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6831","nonce":1392897493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6829","nonce":1392897493}]},"public_addr":"172.21.15.146:6823/1392897493","cluster_addr":"172.21.15.146:6827/1392897493","heartbeat_back_addr":"172.21.15.146:6831/1392897493","heartbeat_front_addr":"172.21.15.146:6829/1392897493","state":["exists","up"]},{"osd":7,"uuid":"75a79b44-fa5f-426d-a7d3-71f21f78835c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6817","nonce":3268323156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6819","nonce":3268323156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6825","nonce":3268323156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6822","nonce":3268323156}]},"public_addr":"172.21.15.146:6817/3268323156","cluster_addr":"172.21.15.146:6819/3268323156","heartbeat_back_addr":"172.21.15.146:6825/3268323156","heartbeat_front_addr":"172.21.15.146:6822/3268323156","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.310437+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.295850+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.294674+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.526855+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.300023+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.595160+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.346046+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.558741+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T09:26:27.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.116+0000 7f29bfe00640 1 -- 172.21.15.6:0/1427046647 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f299c06b500 msgr2=0x7f299c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:27.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.116+0000 7f29bfe00640 1 --2- 172.21.15.6:0/1427046647 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f299c06b500 0x7f299c06d9c0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f29a8000f60 tx=0x7f29a8007a80 comp rx=0 tx=0).stop 2024-03-04T09:26:27.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.116+0000 7f29bfe00640 1 -- 172.21.15.6:0/1427046647 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f29c00cd060 msgr2=0x7f29c00b1ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:27.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.116+0000 7f29bfe00640 1 --2- 172.21.15.6:0/1427046647 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f29c00cd060 0x7f29c00b1ab0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f29b000d800 tx=0x7f29b000dcc0 comp rx=0 tx=0).stop 2024-03-04T09:26:27.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.116+0000 7f29bfe00640 1 -- 172.21.15.6:0/1427046647 shutdown_connections 2024-03-04T09:26:27.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.116+0000 7f29bfe00640 1 --2- 172.21.15.6:0/1427046647 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f29c015e810 0x7f29c0158b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:27.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.116+0000 7f29bfe00640 1 --2- 172.21.15.6:0/1427046647 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f299c06b500 0x7f299c06d9c0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:27.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.116+0000 7f29bfe00640 1 --2- 172.21.15.6:0/1427046647 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f29c01552b0 0x7f29c0146230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:27.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.116+0000 7f29bfe00640 1 --2- 172.21.15.6:0/1427046647 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f29c00cd060 0x7f29c00b1ab0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:27.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.116+0000 7f29bfe00640 1 -- 172.21.15.6:0/1427046647 >> 172.21.15.6:0/1427046647 conn(0x7f29c006b3d0 msgr2=0x7f29c00b4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:27.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.116+0000 7f29bfe00640 1 -- 172.21.15.6:0/1427046647 shutdown_connections 2024-03-04T09:26:27.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.116+0000 7f29bfe00640 1 -- 172.21.15.6:0/1427046647 wait complete. 2024-03-04T09:26:27.132 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T09:25:07.935198+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-04T09:26:27.132 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-04T09:26:27.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.357+0000 7f8fee200640 1 Processor -- start 2024-03-04T09:26:27.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.358+0000 7f8fee200640 1 -- start start 2024-03-04T09:26:27.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.359+0000 7f8fee200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8fe8161700 0x7f8fe80b4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:27.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.359+0000 7f8fee200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f8fe80d24f0 0x7f8fe80b1220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:27.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.359+0000 7f8fee200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fe80b2c00 con 0x7f8fe80d4890 2024-03-04T09:26:27.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.359+0000 7f8fee200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fe8161ae0 con 0x7f8fe80d24f0 2024-03-04T09:26:27.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.359+0000 7f8fee200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fe80b2ea0 con 0x7f8fe8161700 2024-03-04T09:26:27.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.359+0000 7f8fed800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f8fe80d24f0 0x7f8fe80b1220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:27.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.359+0000 7f8fe7e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f8fe80d4890 0x7f8fe80d1de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:54620/0 (socket says 172.21.15.6:54620) 2024-03-04T09:26:27.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.360+0000 7f8fe7e00640 1 -- 172.21.15.6:0/468812460 learned_addr learned my addr 172.21.15.6:0/468812460 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:26:27.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.360+0000 7f8fece00640 1 --2- 172.21.15.6:0/468812460 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8fe8161700 0x7f8fe80b4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:27.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.360+0000 7f8fed800640 1 -- 172.21.15.6:0/468812460 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8fe8161700 msgr2=0x7f8fe80b4fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:27.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.360+0000 7f8fed800640 1 --2- 172.21.15.6:0/468812460 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8fe8161700 0x7f8fe80b4fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:27.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.360+0000 7f8fed800640 1 -- 172.21.15.6:0/468812460 >> v1:172.21.15.6:6789/0 conn(0x7f8fe80d4890 legacy=0x7f8fe80d1de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:27.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.360+0000 7f8fed800640 1 -- 172.21.15.6:0/468812460 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fe80b3180 con 0x7f8fe80d24f0 2024-03-04T09:26:27.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.360+0000 7f8fe7400640 1 -- 172.21.15.6:0/468812460 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2941111570 0 0) 0x7f8fe80b2c00 con 0x7f8fe80d4890 2024-03-04T09:26:27.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.361+0000 7f8fed800640 1 --2- 172.21.15.6:0/468812460 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8fe80d24f0 0x7f8fe80b1220 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f8fe0009d90 tx=0x7f8fe002bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=888d019b20971f6a server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:27.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.361+0000 7f8fe7400640 1 -- 172.21.15.6:0/468812460 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fe0039070 con 0x7f8fe80d24f0 2024-03-04T09:26:27.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.362+0000 7f8fe7400640 1 -- 172.21.15.6:0/468812460 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fe002cc50 con 0x7f8fe80d24f0 2024-03-04T09:26:27.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.362+0000 7f8fe7400640 1 -- 172.21.15.6:0/468812460 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fe0034440 con 0x7f8fe80d24f0 2024-03-04T09:26:27.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.362+0000 7f8fee200640 1 -- 172.21.15.6:0/468812460 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8fe80d24f0 msgr2=0x7f8fe80b1220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:27.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.362+0000 7f8fee200640 1 --2- 172.21.15.6:0/468812460 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8fe80d24f0 0x7f8fe80b1220 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f8fe0009d90 tx=0x7f8fe002bea0 comp rx=0 tx=0).stop 2024-03-04T09:26:27.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.362+0000 7f8fee200640 1 -- 172.21.15.6:0/468812460 shutdown_connections 2024-03-04T09:26:27.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.362+0000 7f8fee200640 1 --2- 172.21.15.6:0/468812460 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8fe80d24f0 0x7f8fe80b1220 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:27.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.362+0000 7f8fee200640 1 --2- 172.21.15.6:0/468812460 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8fe8161700 0x7f8fe80b4fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:27.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.362+0000 7f8fee200640 1 -- 172.21.15.6:0/468812460 >> 172.21.15.6:0/468812460 conn(0x7f8fe80d3d70 msgr2=0x7f8fe80d4180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:27.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.362+0000 7f8fee200640 1 -- 172.21.15.6:0/468812460 shutdown_connections 2024-03-04T09:26:27.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.362+0000 7f8fee200640 1 -- 172.21.15.6:0/468812460 wait complete. 2024-03-04T09:26:27.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.364+0000 7f8fee200640 1 Processor -- start 2024-03-04T09:26:27.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.364+0000 7f8fee200640 1 -- start start 2024-03-04T09:26:27.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.365+0000 7f8fee200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8fe80d24f0 0x7f8fe815eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:27.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.365+0000 7f8fee200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8fe80d4890 0x7f8fe80cb500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:27.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.365+0000 7f8fee200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8fe8161700 0x7f8fe80cd030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:27.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.365+0000 7f8fee200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fe80b5510 con 0x7f8fe80d24f0 2024-03-04T09:26:27.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.365+0000 7f8fee200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fe80b0d00 con 0x7f8fe8161700 2024-03-04T09:26:27.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.365+0000 7f8fee200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fe80b5690 con 0x7f8fe80d4890 2024-03-04T09:26:27.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.365+0000 7f8fece00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8fe80d24f0 0x7f8fe815eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:27.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.365+0000 7f8fece00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8fe80d24f0 0x7f8fe815eeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45846/0 (socket says 172.21.15.6:45846) 2024-03-04T09:26:27.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.365+0000 7f8fece00640 1 -- 172.21.15.6:0/663296747 learned_addr learned my addr 172.21.15.6:0/663296747 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:27.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.366+0000 7f8fece00640 1 -- 172.21.15.6:0/663296747 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8fe80d4890 msgr2=0x7f8fe80cb500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:27.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.366+0000 7f8fece00640 1 --2- 172.21.15.6:0/663296747 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8fe80d4890 0x7f8fe80cb500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:27.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.366+0000 7f8fece00640 1 -- 172.21.15.6:0/663296747 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8fe8161700 msgr2=0x7f8fe80cd030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:27.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.366+0000 7f8fece00640 1 --2- 172.21.15.6:0/663296747 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8fe8161700 0x7f8fe80cd030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:27.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.366+0000 7f8fece00640 1 -- 172.21.15.6:0/663296747 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fe815f490 con 0x7f8fe80d24f0 2024-03-04T09:26:27.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.366+0000 7f8fece00640 1 --2- 172.21.15.6:0/663296747 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8fe80d24f0 0x7f8fe815eeb0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f8fd800d620 tx=0x7f8fd800dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:27.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.366+0000 7f8fe4c00640 1 -- 172.21.15.6:0/663296747 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fd800b620 con 0x7f8fe80d24f0 2024-03-04T09:26:27.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.366+0000 7f8fe4c00640 1 -- 172.21.15.6:0/663296747 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fd8019070 con 0x7f8fe80d24f0 2024-03-04T09:26:27.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.366+0000 7f8fee200640 1 -- 172.21.15.6:0/663296747 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fe815f770 con 0x7f8fe80d24f0 2024-03-04T09:26:27.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.366+0000 7f8fe4c00640 1 -- 172.21.15.6:0/663296747 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fd800ba00 con 0x7f8fe80d24f0 2024-03-04T09:26:27.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.367+0000 7f8fee200640 1 -- 172.21.15.6:0/663296747 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fe8160010 con 0x7f8fe80d24f0 2024-03-04T09:26:27.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.367+0000 7f8fee200640 1 -- 172.21.15.6:0/663296747 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fe815f940 con 0x7f8fe80d24f0 2024-03-04T09:26:27.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.369+0000 7f8fe4c00640 1 -- 172.21.15.6:0/663296747 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8fd800bba0 con 0x7f8fe80d24f0 2024-03-04T09:26:27.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.369+0000 7f8fe4c00640 1 --2- 172.21.15.6:0/663296747 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f8fc806b4d0 0x7f8fc806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:27.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.369+0000 7f8fe4c00640 1 -- 172.21.15.6:0/663296747 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f8fd808a370 con 0x7f8fe80d24f0 2024-03-04T09:26:27.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.370+0000 7f8fe7e00640 1 --2- 172.21.15.6:0/663296747 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f8fc806b4d0 0x7f8fc806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:27.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.371+0000 7f8fe7e00640 1 --2- 172.21.15.6:0/663296747 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f8fc806b4d0 0x7f8fc806d990 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f8fd4007fd0 tx=0x7f8fd4007920 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:27.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.374+0000 7f8fe4c00640 1 -- 172.21.15.6:0/663296747 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8fd80594a0 con 0x7f8fe80d24f0 2024-03-04T09:26:27.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.564+0000 7f8fee200640 1 -- 172.21.15.6:0/663296747 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8fe804f330 con 0x7f8fe80d24f0 2024-03-04T09:26:27.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.565+0000 7f8fe4c00640 1 -- 172.21.15.6:0/663296747 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v29) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8fd8058da0 con 0x7f8fe80d24f0 2024-03-04T09:26:27.562 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-04T09:26:27.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.567+0000 7f8fee200640 1 -- 172.21.15.6:0/663296747 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f8fc806b4d0 msgr2=0x7f8fc806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:27.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.567+0000 7f8fee200640 1 --2- 172.21.15.6:0/663296747 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f8fc806b4d0 0x7f8fc806d990 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f8fd4007fd0 tx=0x7f8fd4007920 comp rx=0 tx=0).stop 2024-03-04T09:26:27.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.567+0000 7f8fee200640 1 -- 172.21.15.6:0/663296747 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8fe80d24f0 msgr2=0x7f8fe815eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:27.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.567+0000 7f8fee200640 1 --2- 172.21.15.6:0/663296747 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8fe80d24f0 0x7f8fe815eeb0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f8fd800d620 tx=0x7f8fd800dae0 comp rx=0 tx=0).stop 2024-03-04T09:26:27.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.567+0000 7f8fee200640 1 -- 172.21.15.6:0/663296747 shutdown_connections 2024-03-04T09:26:27.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.567+0000 7f8fee200640 1 --2- 172.21.15.6:0/663296747 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8fe8161700 0x7f8fe80cd030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:27.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.567+0000 7f8fee200640 1 --2- 172.21.15.6:0/663296747 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f8fc806b4d0 0x7f8fc806d990 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:27.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.567+0000 7f8fee200640 1 --2- 172.21.15.6:0/663296747 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8fe80d4890 0x7f8fe80cb500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:27.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.568+0000 7f8fee200640 1 --2- 172.21.15.6:0/663296747 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8fe80d24f0 0x7f8fe815eeb0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:27.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.568+0000 7f8fee200640 1 -- 172.21.15.6:0/663296747 >> 172.21.15.6:0/663296747 conn(0x7f8fe80d3d70 msgr2=0x7f8fe806c400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:27.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.568+0000 7f8fee200640 1 -- 172.21.15.6:0/663296747 shutdown_connections 2024-03-04T09:26:27.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.568+0000 7f8fee200640 1 -- 172.21.15.6:0/663296747 wait complete. 2024-03-04T09:26:27.583 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-04T09:26:27.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.806+0000 7fd245600640 1 Processor -- start 2024-03-04T09:26:27.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.806+0000 7fd245600640 1 -- start start 2024-03-04T09:26:27.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.808+0000 7fd245600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd24015e200 0x7fd2400b3de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:27.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.808+0000 7fd245600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd24006afb0 0x7fd24006b430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:27.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.808+0000 7fd245600640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd24006bec0 con 0x7fd2400afdf0 2024-03-04T09:26:27.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.808+0000 7fd245600640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2400b0670 con 0x7fd24015e200 2024-03-04T09:26:27.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.808+0000 7fd245600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2400ae380 con 0x7fd24006afb0 2024-03-04T09:26:27.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.808+0000 7fd23ea00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd24015e200 0x7fd2400b3de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:27.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.808+0000 7fd23ea00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd24015e200 0x7fd2400b3de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45854/0 (socket says 172.21.15.6:45854) 2024-03-04T09:26:27.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.808+0000 7fd23ea00640 1 -- 172.21.15.6:0/623756987 learned_addr learned my addr 172.21.15.6:0/623756987 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:27.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.809+0000 7fd23ea00640 1 -- 172.21.15.6:0/623756987 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd24006afb0 msgr2=0x7fd24006b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:27.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.809+0000 7fd23ea00640 1 --2- 172.21.15.6:0/623756987 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd24006afb0 0x7fd24006b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:27.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.809+0000 7fd23ea00640 1 -- 172.21.15.6:0/623756987 >> v1:172.21.15.6:6789/0 conn(0x7fd2400afdf0 legacy=0x7fd2400af140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:27.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.809+0000 7fd23ea00640 1 -- 172.21.15.6:0/623756987 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2400ae660 con 0x7fd24015e200 2024-03-04T09:26:27.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.810+0000 7fd23ea00640 1 --2- 172.21.15.6:0/623756987 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd24015e200 0x7fd2400b3de0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fd230009870 tx=0x7fd23002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b5bbb99598631c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:27.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.810+0000 7fd23e000640 1 -- 172.21.15.6:0/623756987 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd230034070 con 0x7fd24015e200 2024-03-04T09:26:27.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.810+0000 7fd23e000640 1 -- 172.21.15.6:0/623756987 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd230030840 con 0x7fd24015e200 2024-03-04T09:26:27.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.810+0000 7fd23e000640 1 -- 172.21.15.6:0/623756987 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd230030c10 con 0x7fd24015e200 2024-03-04T09:26:27.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.811+0000 7fd245600640 1 -- 172.21.15.6:0/623756987 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd24015e200 msgr2=0x7fd2400b3de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:27.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.811+0000 7fd245600640 1 --2- 172.21.15.6:0/623756987 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd24015e200 0x7fd2400b3de0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fd230009870 tx=0x7fd23002b980 comp rx=0 tx=0).stop 2024-03-04T09:26:27.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.811+0000 7fd245600640 1 -- 172.21.15.6:0/623756987 shutdown_connections 2024-03-04T09:26:27.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.811+0000 7fd245600640 1 --2- 172.21.15.6:0/623756987 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd24006afb0 0x7fd24006b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:27.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.811+0000 7fd245600640 1 --2- 172.21.15.6:0/623756987 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd24015e200 0x7fd2400b3de0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:27.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.811+0000 7fd245600640 1 -- 172.21.15.6:0/623756987 >> 172.21.15.6:0/623756987 conn(0x7fd2400b2ae0 msgr2=0x7fd2400b2ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:27.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.811+0000 7fd245600640 1 -- 172.21.15.6:0/623756987 shutdown_connections 2024-03-04T09:26:27.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.811+0000 7fd245600640 1 -- 172.21.15.6:0/623756987 wait complete. 2024-03-04T09:26:27.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.813+0000 7fd245600640 1 Processor -- start 2024-03-04T09:26:27.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.813+0000 7fd245600640 1 -- start start 2024-03-04T09:26:27.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.814+0000 7fd245600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd24006afb0 0x7fd240140d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:27.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.814+0000 7fd245600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd2400afdf0 0x7fd24015f630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:27.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.814+0000 7fd245600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd24015e200 0x7fd24015fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:27.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.814+0000 7fd245600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd24006dbb0 con 0x7fd2400afdf0 2024-03-04T09:26:27.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.814+0000 7fd245600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2400aec20 con 0x7fd24006afb0 2024-03-04T09:26:27.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.814+0000 7fd245600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd24006dd30 con 0x7fd24015e200 2024-03-04T09:26:27.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.814+0000 7fd236000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd2400afdf0 0x7fd24015f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:27.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.814+0000 7fd236000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd2400afdf0 0x7fd24015f630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45858/0 (socket says 172.21.15.6:45858) 2024-03-04T09:26:27.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.814+0000 7fd236000640 1 -- 172.21.15.6:0/1336438160 learned_addr learned my addr 172.21.15.6:0/1336438160 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:27.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.814+0000 7fd23ea00640 1 --2- 172.21.15.6:0/1336438160 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd24006afb0 0x7fd240140d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:27.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.814+0000 7fd236000640 1 -- 172.21.15.6:0/1336438160 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd24015e200 msgr2=0x7fd24015fb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:27.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.814+0000 7fd236000640 1 --2- 172.21.15.6:0/1336438160 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd24015e200 0x7fd24015fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:27.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.814+0000 7fd236000640 1 -- 172.21.15.6:0/1336438160 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd24006afb0 msgr2=0x7fd240140d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:27.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.814+0000 7fd236000640 1 --2- 172.21.15.6:0/1336438160 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd24006afb0 0x7fd240140d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:27.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.814+0000 7fd236000640 1 -- 172.21.15.6:0/1336438160 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2401f3fa0 con 0x7fd2400afdf0 2024-03-04T09:26:27.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.815+0000 7fd23ea00640 1 --2- 172.21.15.6:0/1336438160 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd24006afb0 0x7fd240140d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:26:27.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.815+0000 7fd236000640 1 --2- 172.21.15.6:0/1336438160 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd2400afdf0 0x7fd24015f630 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fd22800c440 tx=0x7fd228001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:27.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.815+0000 7fd237400640 1 -- 172.21.15.6:0/1336438160 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2280052d0 con 0x7fd2400afdf0 2024-03-04T09:26:27.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.815+0000 7fd237400640 1 -- 172.21.15.6:0/1336438160 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd228005470 con 0x7fd2400afdf0 2024-03-04T09:26:27.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.815+0000 7fd245600640 1 -- 172.21.15.6:0/1336438160 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2401e86e0 con 0x7fd2400afdf0 2024-03-04T09:26:27.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.815+0000 7fd237400640 1 -- 172.21.15.6:0/1336438160 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd228007260 con 0x7fd2400afdf0 2024-03-04T09:26:27.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.815+0000 7fd245600640 1 -- 172.21.15.6:0/1336438160 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd240143b90 con 0x7fd2400afdf0 2024-03-04T09:26:27.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.816+0000 7fd245600640 1 -- 172.21.15.6:0/1336438160 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd24004f3a0 con 0x7fd2400afdf0 2024-03-04T09:26:27.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.817+0000 7fd237400640 1 -- 172.21.15.6:0/1336438160 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd228007400 con 0x7fd2400afdf0 2024-03-04T09:26:27.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.818+0000 7fd237400640 1 --2- 172.21.15.6:0/1336438160 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd21c06b4d0 0x7fd21c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:27.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.818+0000 7fd237400640 1 -- 172.21.15.6:0/1336438160 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fd228089400 con 0x7fd2400afdf0 2024-03-04T09:26:27.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.819+0000 7fd23ea00640 1 --2- 172.21.15.6:0/1336438160 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd21c06b4d0 0x7fd21c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:27.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.820+0000 7fd23ea00640 1 --2- 172.21.15.6:0/1336438160 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd21c06b4d0 0x7fd21c06d990 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fd230036000 tx=0x7fd23003b040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:27.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:27.822+0000 7fd237400640 1 -- 172.21.15.6:0/1336438160 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd2280585c0 con 0x7fd2400afdf0 2024-03-04T09:26:28.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:28.026+0000 7fd245600640 1 -- 172.21.15.6:0/1336438160 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fd240073050 con 0x7fd2400afdf0 2024-03-04T09:26:28.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:28.027+0000 7fd237400640 1 -- 172.21.15.6:0/1336438160 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v29) v1 ==== 84+0+65 (secure 0 0 0) 0x7fd228057ec0 con 0x7fd2400afdf0 2024-03-04T09:26:28.024 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:26:28.024 INFO:teuthology.orchestra.run.smithi006.stdout:[ 2024-03-04T09:26:28.024 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:26:28.024 INFO:teuthology.orchestra.run.smithi006.stdout: "poolnum": 1, 2024-03-04T09:26:28.024 INFO:teuthology.orchestra.run.smithi006.stdout: "poolname": ".mgr" 2024-03-04T09:26:28.024 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:26:28.024 INFO:teuthology.orchestra.run.smithi006.stdout:] 2024-03-04T09:26:28.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:28.030+0000 7fd245600640 1 -- 172.21.15.6:0/1336438160 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd21c06b4d0 msgr2=0x7fd21c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:28.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:28.030+0000 7fd245600640 1 --2- 172.21.15.6:0/1336438160 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd21c06b4d0 0x7fd21c06d990 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fd230036000 tx=0x7fd23003b040 comp rx=0 tx=0).stop 2024-03-04T09:26:28.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:28.030+0000 7fd245600640 1 -- 172.21.15.6:0/1336438160 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd2400afdf0 msgr2=0x7fd24015f630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:28.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:28.030+0000 7fd245600640 1 --2- 172.21.15.6:0/1336438160 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd2400afdf0 0x7fd24015f630 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fd22800c440 tx=0x7fd228001e70 comp rx=0 tx=0).stop 2024-03-04T09:26:28.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:28.030+0000 7fd245600640 1 -- 172.21.15.6:0/1336438160 shutdown_connections 2024-03-04T09:26:28.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:28.030+0000 7fd245600640 1 --2- 172.21.15.6:0/1336438160 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd21c06b4d0 0x7fd21c06d990 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:28.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:28.030+0000 7fd245600640 1 --2- 172.21.15.6:0/1336438160 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd24015e200 0x7fd24015fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:28.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:28.030+0000 7fd245600640 1 --2- 172.21.15.6:0/1336438160 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd2400afdf0 0x7fd24015f630 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:28.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:28.030+0000 7fd245600640 1 --2- 172.21.15.6:0/1336438160 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd24006afb0 0x7fd240140d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:28.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:28.030+0000 7fd245600640 1 -- 172.21.15.6:0/1336438160 >> 172.21.15.6:0/1336438160 conn(0x7fd2400b2ae0 msgr2=0x7fd240071d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:28.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:28.030+0000 7fd245600640 1 -- 172.21.15.6:0/1336438160 shutdown_connections 2024-03-04T09:26:28.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:28.030+0000 7fd245600640 1 -- 172.21.15.6:0/1336438160 wait complete. 2024-03-04T09:26:28.047 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-04T09:26:28.048 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-04T09:26:28.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:28.279+0000 7fb9e1e00640 1 Processor -- start 2024-03-04T09:26:28.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:28.279+0000 7fb9e1e00640 1 -- start start 2024-03-04T09:26:28.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:28.281+0000 7fb9e1e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb9dc15e1f0 0x7fb9dc0d0680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:28.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:28.281+0000 7fb9e1e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fb9dc0d4660 0x7fb9dc0d0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:28.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:28.281+0000 7fb9e1e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9dc0d23c0 con 0x7fb9dc0b38f0 2024-03-04T09:26:28.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:28.281+0000 7fb9e1e00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9dc0d4c30 con 0x7fb9dc0d4660 2024-03-04T09:26:28.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:28.281+0000 7fb9e1e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9dc0d26a0 con 0x7fb9dc15e1f0 2024-03-04T09:26:28.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:28.281+0000 7fb9e1400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fb9dc0b38f0 0x7fb9dc0b3d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:54646/0 (socket says 172.21.15.6:54646) 2024-03-04T09:26:28.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:28.281+0000 7fb9e1400640 1 -- 172.21.15.6:0/1743351221 learned_addr learned my addr 172.21.15.6:0/1743351221 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:26:28.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:28.282+0000 7fb9dbe00640 1 --2- 172.21.15.6:0/1743351221 >> v2:172.21.15.146:3300/0 conn(0x7fb9dc0d4660 0x7fb9dc0d0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:28.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:28.282+0000 7fb9e0a00640 1 --2- 172.21.15.6:0/1743351221 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb9dc15e1f0 0x7fb9dc0d0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:28.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:28.282+0000 7fb9db400640 1 -- 172.21.15.6:0/1743351221 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2196807819 0 0) 0x7fb9dc0d23c0 con 0x7fb9dc0b38f0 2024-03-04T09:26:28.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:28.282+0000 7fb9db400640 1 -- 172.21.15.6:0/1743351221 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9c4003520 con 0x7fb9dc0b38f0 2024-03-04T09:26:28.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:28.282+0000 7fb9db400640 1 -- 172.21.15.6:0/1743351221 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3273809946 0 0) 0x7fb9c4003520 con 0x7fb9dc0b38f0 2024-03-04T09:26:28.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:28.282+0000 7fb9db400640 1 -- 172.21.15.6:0/1743351221 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb9dc15e1f0 msgr2=0x7fb9dc0d0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:28.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:28.282+0000 7fb9db400640 1 --2- 172.21.15.6:0/1743351221 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb9dc15e1f0 0x7fb9dc0d0680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:28.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:28.282+0000 7fb9db400640 1 -- 172.21.15.6:0/1743351221 >> v2:172.21.15.146:3300/0 conn(0x7fb9dc0d4660 msgr2=0x7fb9dc0d0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:28.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:28.282+0000 7fb9db400640 1 --2- 172.21.15.6:0/1743351221 >> v2:172.21.15.146:3300/0 conn(0x7fb9dc0d4660 0x7fb9dc0d0bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:28.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:28.282+0000 7fb9db400640 1 -- 172.21.15.6:0/1743351221 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9dc0d2980 con 0x7fb9dc0b38f0 2024-03-04T09:26:28.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:28.282+0000 7fb9db400640 1 -- 172.21.15.6:0/1743351221 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7fb9d40019f0 con 0x7fb9dc0b38f0 2024-03-04T09:26:28.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:28.282+0000 7fb9db400640 1 -- 172.21.15.6:0/1743351221 >> v1:172.21.15.6:6789/0 conn(0x7fb9dc0b38f0 legacy=0x7fb9dc0b3d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:28.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:28.282+0000 7fb9db400640 1 --2- 172.21.15.6:0/1743351221 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb9c40039b0 0x7fb9c4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:28.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:28.282+0000 7fb9db400640 1 --2- 172.21.15.6:0/1743351221 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb9c4004800 0x7fb9c4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:28.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:28.283+0000 7fb9db400640 1 --2- 172.21.15.6:0/1743351221 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb9dc15e1f0 0x7fb9c40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:28.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:28.283+0000 7fb9db400640 1 -- 172.21.15.6:0/1743351221 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9dc0d4c30 con 0x7fb9c4004800 2024-03-04T09:26:28.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:28.283+0000 7fb9db400640 1 -- 172.21.15.6:0/1743351221 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9dc0d26a0 con 0x7fb9c40039b0 2024-03-04T09:26:28.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:28.283+0000 7fb9db400640 1 -- 172.21.15.6:0/1743351221 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9c400b9c0 con 0x7fb9dc15e1f0 2024-03-04T09:26:28.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:28.283+0000 7fb9dbe00640 1 --2- 172.21.15.6:0/1743351221 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb9c4004800 0x7fb9c4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:28.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:28.283+0000 7fb9e0a00640 1 --2- 172.21.15.6:0/1743351221 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb9c40039b0 0x7fb9c4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:28.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:28.283+0000 7fb9dbe00640 1 -- 172.21.15.6:0/1743351221 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb9dc15e1f0 msgr2=0x7fb9c40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:28.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:28.283+0000 7fb9dbe00640 1 --2- 172.21.15.6:0/1743351221 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb9dc15e1f0 0x7fb9c40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:28.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:28.283+0000 7fb9dbe00640 1 -- 172.21.15.6:0/1743351221 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb9c40039b0 msgr2=0x7fb9c4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:28.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:28.283+0000 7fb9dbe00640 1 --2- 172.21.15.6:0/1743351221 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb9c40039b0 0x7fb9c4003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:28.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:28.283+0000 7fb9dbe00640 1 -- 172.21.15.6:0/1743351221 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9c4003520 con 0x7fb9c4004800 2024-03-04T09:26:28.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:28.285+0000 7fb9dbe00640 1 --2- 172.21.15.6:0/1743351221 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb9c4004800 0x7fb9c4004c80 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fb9c8009af0 tx=0x7fb9c4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f7453a5fe6f7815 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:28.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:28.285+0000 7fb9db400640 1 -- 172.21.15.6:0/1743351221 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9c8039070 con 0x7fb9c4004800 2024-03-04T09:26:28.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:28.285+0000 7fb9db400640 1 -- 172.21.15.6:0/1743351221 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9c802c710 con 0x7fb9c4004800 2024-03-04T09:26:28.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:28.285+0000 7fb9db400640 1 -- 172.21.15.6:0/1743351221 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9c8009690 con 0x7fb9c4004800 2024-03-04T09:26:28.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:28.285+0000 7fb9e1e00640 1 -- 172.21.15.6:0/1743351221 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb9c4004800 msgr2=0x7fb9c4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:28.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:28.285+0000 7fb9e1e00640 1 --2- 172.21.15.6:0/1743351221 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb9c4004800 0x7fb9c4004c80 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fb9c8009af0 tx=0x7fb9c4005830 comp rx=0 tx=0).stop 2024-03-04T09:26:28.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:28.285+0000 7fb9e1400640 1 -- 172.21.15.6:0/1743351221 reap_dead start 2024-03-04T09:26:28.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:28.285+0000 7fb9e1e00640 1 -- 172.21.15.6:0/1743351221 shutdown_connections 2024-03-04T09:26:28.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:28.285+0000 7fb9e1e00640 1 -- 172.21.15.6:0/1743351221 >> 172.21.15.6:0/1743351221 conn(0x7fb9dc09d000 msgr2=0x7fb9dc0d3cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:28.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:28.286+0000 7fb9e1e00640 1 -- 172.21.15.6:0/1743351221 shutdown_connections 2024-03-04T09:26:28.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:28.286+0000 7fb9e1e00640 1 -- 172.21.15.6:0/1743351221 wait complete. 2024-03-04T09:26:28.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:28.287+0000 7fb9e1e00640 1 Processor -- start 2024-03-04T09:26:28.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:28.288+0000 7fb9e1e00640 1 -- start start 2024-03-04T09:26:28.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:28.288+0000 7fb9e1e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb9c4004800 0x7fb9dc0b68a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:28.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:28.288+0000 7fb9e1e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb9dc153050 0x7fb9dc1f30b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:28.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:28.288+0000 7fb9e1e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb9dc0b82e0 0x7fb9dc0b8760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:28.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:28.288+0000 7fb9e1e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9dc0d0680 con 0x7fb9dc153050 2024-03-04T09:26:28.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:28.288+0000 7fb9e1e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9dc0b2da0 con 0x7fb9c4004800 2024-03-04T09:26:28.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:28.288+0000 7fb9e1e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9dc0d0800 con 0x7fb9dc0b82e0 2024-03-04T09:26:28.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:28.289+0000 7fb9dbe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb9dc153050 0x7fb9dc1f30b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:28.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:28.289+0000 7fb9dbe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb9dc153050 0x7fb9dc1f30b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45884/0 (socket says 172.21.15.6:45884) 2024-03-04T09:26:28.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:28.289+0000 7fb9dbe00640 1 -- 172.21.15.6:0/1521985538 learned_addr learned my addr 172.21.15.6:0/1521985538 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:28.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:28.289+0000 7fb9e0a00640 1 --2- 172.21.15.6:0/1521985538 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb9c4004800 0x7fb9dc0b68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:28.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:28.289+0000 7fb9e1400640 1 --2- 172.21.15.6:0/1521985538 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb9dc0b82e0 0x7fb9dc0b8760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:28.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:28.289+0000 7fb9dbe00640 1 -- 172.21.15.6:0/1521985538 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb9dc0b82e0 msgr2=0x7fb9dc0b8760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:28.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:28.289+0000 7fb9dbe00640 1 --2- 172.21.15.6:0/1521985538 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb9dc0b82e0 0x7fb9dc0b8760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:28.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:28.289+0000 7fb9dbe00640 1 -- 172.21.15.6:0/1521985538 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb9c4004800 msgr2=0x7fb9dc0b68a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:28.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:28.289+0000 7fb9dbe00640 1 --2- 172.21.15.6:0/1521985538 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb9c4004800 0x7fb9dc0b68a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:28.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:28.289+0000 7fb9dbe00640 1 -- 172.21.15.6:0/1521985538 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9dc0b6de0 con 0x7fb9dc153050 2024-03-04T09:26:28.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:28.289+0000 7fb9e0a00640 1 --2- 172.21.15.6:0/1521985538 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb9c4004800 0x7fb9dc0b68a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:26:28.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:28.290+0000 7fb9dbe00640 1 --2- 172.21.15.6:0/1521985538 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb9dc153050 0x7fb9dc1f30b0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fb9c8009af0 tx=0x7fb9c8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:28.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:28.290+0000 7fb9d8c00640 1 -- 172.21.15.6:0/1521985538 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9c8042020 con 0x7fb9dc153050 2024-03-04T09:26:28.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:28.290+0000 7fb9d8c00640 1 -- 172.21.15.6:0/1521985538 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9c803a040 con 0x7fb9dc153050 2024-03-04T09:26:28.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:28.290+0000 7fb9d8c00640 1 -- 172.21.15.6:0/1521985538 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9c8039070 con 0x7fb9dc153050 2024-03-04T09:26:28.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:28.290+0000 7fb9e1e00640 1 -- 172.21.15.6:0/1521985538 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9dc0b7060 con 0x7fb9dc153050 2024-03-04T09:26:28.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:28.290+0000 7fb9e1e00640 1 -- 172.21.15.6:0/1521985538 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9dc0b7620 con 0x7fb9dc153050 2024-03-04T09:26:28.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:28.291+0000 7fb9e1e00640 1 -- 172.21.15.6:0/1521985538 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9dc04f330 con 0x7fb9dc153050 2024-03-04T09:26:28.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:28.292+0000 7fb9d8c00640 1 -- 172.21.15.6:0/1521985538 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb9c8045020 con 0x7fb9dc153050 2024-03-04T09:26:28.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:28.293+0000 7fb9d8c00640 1 --2- 172.21.15.6:0/1521985538 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb9bc06b4d0 0x7fb9bc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:28.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:28.293+0000 7fb9d8c00640 1 -- 172.21.15.6:0/1521985538 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fb9c80ab7b0 con 0x7fb9dc153050 2024-03-04T09:26:28.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:28.294+0000 7fb9e0a00640 1 --2- 172.21.15.6:0/1521985538 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb9bc06b4d0 0x7fb9bc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:28.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:28.295+0000 7fb9e0a00640 1 --2- 172.21.15.6:0/1521985538 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb9bc06b4d0 0x7fb9bc06d990 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fb9d0006ce0 tx=0x7fb9d0008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:28.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:28.297+0000 7fb9d8c00640 1 -- 172.21.15.6:0/1521985538 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb9c804c030 con 0x7fb9dc153050 2024-03-04T09:26:28.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:28.503+0000 7fb9e1e00640 1 -- 172.21.15.6:0/1521985538 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fb9dc0d0250 con 0x7fb9dc153050 2024-03-04T09:26:28.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:28.951+0000 7fb9d8c00640 1 -- 172.21.15.6:0/1521985538 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v30) v1 ==== 138+0+0 (secure 0 0 0) 0x7fb9dc0b6de0 con 0x7fb9dc153050 2024-03-04T09:26:28.948 INFO:teuthology.orchestra.run.smithi006.stderr:pool 'cephfs_metadata' created 2024-03-04T09:26:28.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:28.955+0000 7fb9e1e00640 1 -- 172.21.15.6:0/1521985538 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb9bc06b4d0 msgr2=0x7fb9bc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:28.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:28.955+0000 7fb9e1e00640 1 --2- 172.21.15.6:0/1521985538 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb9bc06b4d0 0x7fb9bc06d990 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fb9d0006ce0 tx=0x7fb9d0008040 comp rx=0 tx=0).stop 2024-03-04T09:26:28.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:28.955+0000 7fb9e1e00640 1 -- 172.21.15.6:0/1521985538 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb9dc153050 msgr2=0x7fb9dc1f30b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:28.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:28.955+0000 7fb9e1e00640 1 --2- 172.21.15.6:0/1521985538 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb9dc153050 0x7fb9dc1f30b0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fb9c8009af0 tx=0x7fb9c8004730 comp rx=0 tx=0).stop 2024-03-04T09:26:28.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:28.956+0000 7fb9e1e00640 1 -- 172.21.15.6:0/1521985538 shutdown_connections 2024-03-04T09:26:28.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:28.956+0000 7fb9e1e00640 1 --2- 172.21.15.6:0/1521985538 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb9bc06b4d0 0x7fb9bc06d990 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:28.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:28.956+0000 7fb9e1e00640 1 --2- 172.21.15.6:0/1521985538 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb9dc0b82e0 0x7fb9dc0b8760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:28.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:28.956+0000 7fb9e1e00640 1 --2- 172.21.15.6:0/1521985538 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb9dc153050 0x7fb9dc1f30b0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:28.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:28.956+0000 7fb9e1e00640 1 --2- 172.21.15.6:0/1521985538 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb9c4004800 0x7fb9dc0b68a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:28.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:28.956+0000 7fb9e1e00640 1 -- 172.21.15.6:0/1521985538 >> 172.21.15.6:0/1521985538 conn(0x7fb9dc09d000 msgr2=0x7fb9dc0ce8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:28.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:28.956+0000 7fb9e1e00640 1 -- 172.21.15.6:0/1521985538 shutdown_connections 2024-03-04T09:26:28.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:28.956+0000 7fb9e1e00640 1 -- 172.21.15.6:0/1521985538 wait complete. 2024-03-04T09:26:28.977 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-04T09:26:29.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:29.199+0000 7f4175600640 1 Processor -- start 2024-03-04T09:26:29.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:29.200+0000 7f4175600640 1 -- start start 2024-03-04T09:26:29.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:29.201+0000 7f4175600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f41700ad6f0 0x7f417015c370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:29.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:29.201+0000 7f4175600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f417006d5d0 0x7f417006da20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:29.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:29.201+0000 7f4175600640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4170156830 con 0x7f417015f960 2024-03-04T09:26:29.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:29.201+0000 7f4175600640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f417006ce60 con 0x7f41700ad6f0 2024-03-04T09:26:29.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:29.201+0000 7f4175600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4170160120 con 0x7f417006d5d0 2024-03-04T09:26:29.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:29.202+0000 7f416ea00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f41700ad6f0 0x7f417015c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:29.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:29.202+0000 7f416ea00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f41700ad6f0 0x7f417015c370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:40618/0 (socket says 172.21.15.6:40618) 2024-03-04T09:26:29.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:29.202+0000 7f416ea00640 1 -- 172.21.15.6:0/722398839 learned_addr learned my addr 172.21.15.6:0/722398839 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:29.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:29.202+0000 7f416f400640 1 --2- 172.21.15.6:0/722398839 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f417006d5d0 0x7f417006da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:29.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:29.202+0000 7f416ea00640 1 -- 172.21.15.6:0/722398839 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f417006d5d0 msgr2=0x7f417006da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:29.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:29.202+0000 7f416ea00640 1 --2- 172.21.15.6:0/722398839 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f417006d5d0 0x7f417006da20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:29.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:29.202+0000 7f416ea00640 1 -- 172.21.15.6:0/722398839 >> v1:172.21.15.146:6789/0 conn(0x7f417015f960 legacy=0x7f41700adad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:29.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:29.202+0000 7f416ea00640 1 -- 172.21.15.6:0/722398839 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4170160400 con 0x7f41700ad6f0 2024-03-04T09:26:29.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:29.204+0000 7f416ea00640 1 --2- 172.21.15.6:0/722398839 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f41700ad6f0 0x7f417015c370 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f417006cfe0 tx=0x7f415802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1609e08d6cf5ecd server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:29.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:29.205+0000 7f416e000640 1 -- 172.21.15.6:0/722398839 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4158034070 con 0x7f41700ad6f0 2024-03-04T09:26:29.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:29.205+0000 7f416e000640 1 -- 172.21.15.6:0/722398839 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4158030840 con 0x7f41700ad6f0 2024-03-04T09:26:29.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:29.205+0000 7f416e000640 1 -- 172.21.15.6:0/722398839 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4158030c10 con 0x7f41700ad6f0 2024-03-04T09:26:29.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:29.205+0000 7f4175600640 1 -- 172.21.15.6:0/722398839 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f41700ad6f0 msgr2=0x7f417015c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:29.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:29.205+0000 7f4175600640 1 --2- 172.21.15.6:0/722398839 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f41700ad6f0 0x7f417015c370 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f417006cfe0 tx=0x7f415802b980 comp rx=0 tx=0).stop 2024-03-04T09:26:29.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:29.205+0000 7f4175600640 1 -- 172.21.15.6:0/722398839 shutdown_connections 2024-03-04T09:26:29.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:29.205+0000 7f4175600640 1 --2- 172.21.15.6:0/722398839 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f417006d5d0 0x7f417006da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:29.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:29.205+0000 7f4175600640 1 --2- 172.21.15.6:0/722398839 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f41700ad6f0 0x7f417015c370 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:29.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:29.205+0000 7f4175600640 1 -- 172.21.15.6:0/722398839 >> 172.21.15.6:0/722398839 conn(0x7f417006b3d0 msgr2=0x7f417006b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:29.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:29.206+0000 7f4175600640 1 -- 172.21.15.6:0/722398839 shutdown_connections 2024-03-04T09:26:29.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:29.206+0000 7f4175600640 1 -- 172.21.15.6:0/722398839 wait complete. 2024-03-04T09:26:29.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:29.208+0000 7f4175600640 1 Processor -- start 2024-03-04T09:26:29.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:29.208+0000 7f4175600640 1 -- start start 2024-03-04T09:26:29.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:29.209+0000 7f4175600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f417006d5d0 0x7f417009c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:29.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:29.209+0000 7f4175600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f41700ad6f0 0x7f41701f5150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:29.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:29.209+0000 7f4175600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f417015f960 0x7f41701f3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:29.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:29.209+0000 7f4175600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f417015fdf0 con 0x7f41700ad6f0 2024-03-04T09:26:29.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:29.209+0000 7f4175600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f417006d0b0 con 0x7f417006d5d0 2024-03-04T09:26:29.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:29.209+0000 7f4175600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f417006df60 con 0x7f417015f960 2024-03-04T09:26:29.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:29.209+0000 7f4166000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f41700ad6f0 0x7f41701f5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:29.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:29.209+0000 7f4166000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f41700ad6f0 0x7f41701f5150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:40626/0 (socket says 172.21.15.6:40626) 2024-03-04T09:26:29.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:29.209+0000 7f4166000640 1 -- 172.21.15.6:0/2114632081 learned_addr learned my addr 172.21.15.6:0/2114632081 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:29.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:29.210+0000 7f416ea00640 1 --2- 172.21.15.6:0/2114632081 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f417006d5d0 0x7f417009c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:29.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:29.210+0000 7f416f400640 1 --2- 172.21.15.6:0/2114632081 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f417015f960 0x7f41701f3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:29.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:29.210+0000 7f4166000640 1 -- 172.21.15.6:0/2114632081 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f417015f960 msgr2=0x7f41701f3f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:29.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:29.210+0000 7f4166000640 1 --2- 172.21.15.6:0/2114632081 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f417015f960 0x7f41701f3f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:29.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:29.210+0000 7f4166000640 1 -- 172.21.15.6:0/2114632081 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f417006d5d0 msgr2=0x7f417009c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:29.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:29.210+0000 7f4166000640 1 --2- 172.21.15.6:0/2114632081 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f417006d5d0 0x7f417009c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:29.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:29.210+0000 7f4166000640 1 -- 172.21.15.6:0/2114632081 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4170072620 con 0x7f41700ad6f0 2024-03-04T09:26:29.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:29.210+0000 7f416ea00640 1 --2- 172.21.15.6:0/2114632081 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f417006d5d0 0x7f417009c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:26:29.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:29.210+0000 7f4166000640 1 --2- 172.21.15.6:0/2114632081 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f41700ad6f0 0x7f41701f5150 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f416000c3f0 tx=0x7f416000c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:29.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:29.210+0000 7f4167400640 1 -- 172.21.15.6:0/2114632081 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41600052d0 con 0x7f41700ad6f0 2024-03-04T09:26:29.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:29.210+0000 7f4167400640 1 -- 172.21.15.6:0/2114632081 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4160005490 con 0x7f41700ad6f0 2024-03-04T09:26:29.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:29.210+0000 7f4167400640 1 -- 172.21.15.6:0/2114632081 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4160007260 con 0x7f41700ad6f0 2024-03-04T09:26:29.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:29.211+0000 7f4175600640 1 -- 172.21.15.6:0/2114632081 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4170072900 con 0x7f41700ad6f0 2024-03-04T09:26:29.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:29.211+0000 7f4175600640 1 -- 172.21.15.6:0/2114632081 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41701f7cf0 con 0x7f41700ad6f0 2024-03-04T09:26:29.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:29.212+0000 7f4175600640 1 -- 172.21.15.6:0/2114632081 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41701f7ef0 con 0x7f41700ad6f0 2024-03-04T09:26:29.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:29.214+0000 7f4167400640 1 -- 172.21.15.6:0/2114632081 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4160007400 con 0x7f41700ad6f0 2024-03-04T09:26:29.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:29.214+0000 7f4167400640 1 --2- 172.21.15.6:0/2114632081 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f414c06b4d0 0x7f414c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:29.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:29.215+0000 7f4167400640 1 -- 172.21.15.6:0/2114632081 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f41600898b0 con 0x7f41700ad6f0 2024-03-04T09:26:29.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:29.215+0000 7f416ea00640 1 --2- 172.21.15.6:0/2114632081 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f414c06b4d0 0x7f414c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:29.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:29.216+0000 7f416ea00640 1 --2- 172.21.15.6:0/2114632081 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f414c06b4d0 0x7f414c06d990 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f4158036000 tx=0x7f415803b040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:29.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:29.218+0000 7f4167400640 1 -- 172.21.15.6:0/2114632081 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f41701f7ef0 con 0x7f41700ad6f0 2024-03-04T09:26:29.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:29.419+0000 7f4175600640 1 -- 172.21.15.6:0/2114632081 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f4170068010 con 0x7f41700ad6f0 2024-03-04T09:26:29.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:29.957+0000 7f4167400640 1 -- 172.21.15.6:0/2114632081 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v31) v1 ==== 170+0+0 (secure 0 0 0) 0x7f4160058a80 con 0x7f41700ad6f0 2024-03-04T09:26:29.955 INFO:teuthology.orchestra.run.smithi006.stderr:pool 'cephfs_data' created 2024-03-04T09:26:29.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:29.962+0000 7f4175600640 1 -- 172.21.15.6:0/2114632081 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f414c06b4d0 msgr2=0x7f414c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:29.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:29.962+0000 7f4175600640 1 --2- 172.21.15.6:0/2114632081 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f414c06b4d0 0x7f414c06d990 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f4158036000 tx=0x7f415803b040 comp rx=0 tx=0).stop 2024-03-04T09:26:29.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:29.962+0000 7f4175600640 1 -- 172.21.15.6:0/2114632081 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f41700ad6f0 msgr2=0x7f41701f5150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:29.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:29.962+0000 7f4175600640 1 --2- 172.21.15.6:0/2114632081 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f41700ad6f0 0x7f41701f5150 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f416000c3f0 tx=0x7f416000c8b0 comp rx=0 tx=0).stop 2024-03-04T09:26:29.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:29.963+0000 7f4175600640 1 -- 172.21.15.6:0/2114632081 shutdown_connections 2024-03-04T09:26:29.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:29.963+0000 7f4175600640 1 --2- 172.21.15.6:0/2114632081 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f414c06b4d0 0x7f414c06d990 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:29.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:29.963+0000 7f4175600640 1 --2- 172.21.15.6:0/2114632081 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f417015f960 0x7f41701f3f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:29.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:29.963+0000 7f4175600640 1 --2- 172.21.15.6:0/2114632081 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f41700ad6f0 0x7f41701f5150 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:29.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:29.963+0000 7f4175600640 1 --2- 172.21.15.6:0/2114632081 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f417006d5d0 0x7f417009c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:29.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:29.963+0000 7f4175600640 1 -- 172.21.15.6:0/2114632081 >> 172.21.15.6:0/2114632081 conn(0x7f417006b3d0 msgr2=0x7f41700b2a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:29.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:29.964+0000 7f4175600640 1 -- 172.21.15.6:0/2114632081 shutdown_connections 2024-03-04T09:26:29.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:29.965+0000 7f4175600640 1 -- 172.21.15.6:0/2114632081 wait complete. 2024-03-04T09:26:29.999 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-04T09:26:30.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:30.185+0000 7f31efe00640 1 Processor -- start 2024-03-04T09:26:30.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:30.185+0000 7f31efe00640 1 -- start start 2024-03-04T09:26:30.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:30.186+0000 7f31efe00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f31e80d07d0 0x7f31e80b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:30.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:30.186+0000 7f31efe00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f31e80d4930 0x7f31e815dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:30.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:30.187+0000 7f31efe00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31e80cf780 con 0x7f31e80d0bb0 2024-03-04T09:26:30.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:30.187+0000 7f31efe00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31e80b3770 con 0x7f31e80d07d0 2024-03-04T09:26:30.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:30.187+0000 7f31efe00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31e80cfa60 con 0x7f31e80d4930 2024-03-04T09:26:30.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:30.187+0000 7f31ed400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f31e80d07d0 0x7f31e80b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:30.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:30.187+0000 7f31eca00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f31e80d4930 0x7f31e815dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:30.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:30.187+0000 7f31ed400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f31e80d07d0 0x7f31e80b4f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.6:50366/0 (socket says 172.21.15.6:50366) 2024-03-04T09:26:30.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:30.187+0000 7f31ed400640 1 -- 172.21.15.6:0/2353790204 learned_addr learned my addr 172.21.15.6:0/2353790204 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:30.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:30.188+0000 7f31ed400640 1 -- 172.21.15.6:0/2353790204 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f31e80d4930 msgr2=0x7f31e815dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:30.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:30.188+0000 7f31ed400640 1 --2- 172.21.15.6:0/2353790204 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f31e80d4930 0x7f31e815dfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:30.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:30.188+0000 7f31ed400640 1 -- 172.21.15.6:0/2353790204 >> v1:172.21.15.146:6789/0 conn(0x7f31e80d0bb0 legacy=0x7f31e80d1bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:30.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:30.188+0000 7f31ed400640 1 -- 172.21.15.6:0/2353790204 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31e80cfd40 con 0x7f31e80d07d0 2024-03-04T09:26:30.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:30.188+0000 7f31eca00640 1 --2- 172.21.15.6:0/2353790204 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f31e80d4930 0x7f31e815dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T09:26:30.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:30.190+0000 7f31ed400640 1 --2- 172.21.15.6:0/2353790204 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f31e80d07d0 0x7f31e80b4f80 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f31d8009e50 tx=0x7f31d802bff0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7a15ca9a70fb65a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:30.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:30.190+0000 7f31e7e00640 1 -- 172.21.15.6:0/2353790204 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31d802cb90 con 0x7f31e80d07d0 2024-03-04T09:26:30.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:30.190+0000 7f31e7e00640 1 -- 172.21.15.6:0/2353790204 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31d802cd30 con 0x7f31e80d07d0 2024-03-04T09:26:30.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:30.190+0000 7f31e7e00640 1 -- 172.21.15.6:0/2353790204 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31d8034610 con 0x7f31e80d07d0 2024-03-04T09:26:30.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:30.191+0000 7f31efe00640 1 -- 172.21.15.6:0/2353790204 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f31e80d07d0 msgr2=0x7f31e80b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:30.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:30.191+0000 7f31efe00640 1 --2- 172.21.15.6:0/2353790204 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f31e80d07d0 0x7f31e80b4f80 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f31d8009e50 tx=0x7f31d802bff0 comp rx=0 tx=0).stop 2024-03-04T09:26:30.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:30.191+0000 7f31efe00640 1 -- 172.21.15.6:0/2353790204 shutdown_connections 2024-03-04T09:26:30.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:30.191+0000 7f31efe00640 1 --2- 172.21.15.6:0/2353790204 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f31e80d4930 0x7f31e815dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:30.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:30.191+0000 7f31efe00640 1 --2- 172.21.15.6:0/2353790204 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f31e80d07d0 0x7f31e80b4f80 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:30.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:30.191+0000 7f31efe00640 1 -- 172.21.15.6:0/2353790204 >> 172.21.15.6:0/2353790204 conn(0x7f31e80d3e10 msgr2=0x7f31e80d4220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:30.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:30.191+0000 7f31efe00640 1 -- 172.21.15.6:0/2353790204 shutdown_connections 2024-03-04T09:26:30.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:30.191+0000 7f31efe00640 1 -- 172.21.15.6:0/2353790204 wait complete. 2024-03-04T09:26:30.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:30.193+0000 7f31efe00640 1 Processor -- start 2024-03-04T09:26:30.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:30.193+0000 7f31efe00640 1 -- start start 2024-03-04T09:26:30.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:30.194+0000 7f31efe00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f31e80d07d0 0x7f31e814c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:30.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:30.194+0000 7f31efe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f31e80d0bb0 0x7f31e814fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:30.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:30.194+0000 7f31efe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f31e80d4930 0x7f31e80c9910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:30.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:30.194+0000 7f31efe00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31e80b4d90 con 0x7f31e80d4930 2024-03-04T09:26:30.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:30.194+0000 7f31efe00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31e80b4c10 con 0x7f31e80d0bb0 2024-03-04T09:26:30.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:30.194+0000 7f31efe00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31e815e4f0 con 0x7f31e80d07d0 2024-03-04T09:26:30.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:30.194+0000 7f31ede00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f31e80d4930 0x7f31e80c9910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:30.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:30.194+0000 7f31ede00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f31e80d4930 0x7f31e80c9910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:40636/0 (socket says 172.21.15.6:40636) 2024-03-04T09:26:30.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:30.194+0000 7f31ede00640 1 -- 172.21.15.6:0/4212766655 learned_addr learned my addr 172.21.15.6:0/4212766655 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:30.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:30.195+0000 7f31eca00640 1 --2- 172.21.15.6:0/4212766655 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f31e80d0bb0 0x7f31e814fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:30.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:30.195+0000 7f31ed400640 1 --2- 172.21.15.6:0/4212766655 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f31e80d07d0 0x7f31e814c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:30.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:30.195+0000 7f31ede00640 1 -- 172.21.15.6:0/4212766655 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f31e80d07d0 msgr2=0x7f31e814c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:30.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:30.195+0000 7f31ede00640 1 --2- 172.21.15.6:0/4212766655 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f31e80d07d0 0x7f31e814c9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:30.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:30.195+0000 7f31ede00640 1 -- 172.21.15.6:0/4212766655 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f31e80d0bb0 msgr2=0x7f31e814fbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:30.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:30.195+0000 7f31ede00640 1 --2- 172.21.15.6:0/4212766655 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f31e80d0bb0 0x7f31e814fbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:30.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:30.195+0000 7f31ede00640 1 -- 172.21.15.6:0/4212766655 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31e8150100 con 0x7f31e80d4930 2024-03-04T09:26:30.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:30.195+0000 7f31eca00640 1 --2- 172.21.15.6:0/4212766655 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f31e80d0bb0 0x7f31e814fbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:26:30.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:30.195+0000 7f31ede00640 1 --2- 172.21.15.6:0/4212766655 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f31e80d4930 0x7f31e80c9910 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f31e0001eb0 tx=0x7f31e000c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:30.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:30.195+0000 7f31e5600640 1 -- 172.21.15.6:0/4212766655 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31e0017070 con 0x7f31e80d4930 2024-03-04T09:26:30.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:30.195+0000 7f31e5600640 1 -- 172.21.15.6:0/4212766655 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31e00052d0 con 0x7f31e80d4930 2024-03-04T09:26:30.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:30.195+0000 7f31e5600640 1 -- 172.21.15.6:0/4212766655 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31e0003810 con 0x7f31e80d4930 2024-03-04T09:26:30.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:30.195+0000 7f31efe00640 1 -- 172.21.15.6:0/4212766655 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31e81f4ea0 con 0x7f31e80d4930 2024-03-04T09:26:30.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:30.196+0000 7f31efe00640 1 -- 172.21.15.6:0/4212766655 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31e81f4600 con 0x7f31e80d4930 2024-03-04T09:26:30.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:30.197+0000 7f31efe00640 1 -- 172.21.15.6:0/4212766655 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31e804f330 con 0x7f31e80d4930 2024-03-04T09:26:30.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:30.198+0000 7f31e5600640 1 -- 172.21.15.6:0/4212766655 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f31e001a070 con 0x7f31e80d4930 2024-03-04T09:26:30.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:30.199+0000 7f31e5600640 1 --2- 172.21.15.6:0/4212766655 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f31c806b4d0 0x7f31c806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:30.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:30.199+0000 7f31e5600640 1 -- 172.21.15.6:0/4212766655 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f31e00895a0 con 0x7f31e80d4930 2024-03-04T09:26:30.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:30.200+0000 7f31ed400640 1 --2- 172.21.15.6:0/4212766655 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f31c806b4d0 0x7f31c806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:30.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:30.201+0000 7f31ed400640 1 --2- 172.21.15.6:0/4212766655 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f31c806b4d0 0x7f31c806d990 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f31d8035000 tx=0x7f31d8030000 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:30.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:30.204+0000 7f31e5600640 1 -- 172.21.15.6:0/4212766655 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f31e00583e0 con 0x7f31e80d4930 2024-03-04T09:26:30.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:30.443+0000 7f31efe00640 1 -- 172.21.15.6:0/4212766655 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f31e80b61f0 con 0x7f31e80d4930 2024-03-04T09:26:30.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:30.974+0000 7f31e5600640 1 -- 172.21.15.6:0/4212766655 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 4 and data pool 5 v12) v1 ==== 172+0+0 (secure 0 0 0) 0x7f31e0057ce0 con 0x7f31e80d4930 2024-03-04T09:26:30.972 INFO:teuthology.orchestra.run.smithi006.stderr:new fs with metadata pool 4 and data pool 5 2024-03-04T09:26:30.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:30.979+0000 7f31efe00640 1 -- 172.21.15.6:0/4212766655 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f31c806b4d0 msgr2=0x7f31c806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:30.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:30.979+0000 7f31efe00640 1 --2- 172.21.15.6:0/4212766655 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f31c806b4d0 0x7f31c806d990 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f31d8035000 tx=0x7f31d8030000 comp rx=0 tx=0).stop 2024-03-04T09:26:30.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:30.979+0000 7f31efe00640 1 -- 172.21.15.6:0/4212766655 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f31e80d4930 msgr2=0x7f31e80c9910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:30.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:30.979+0000 7f31efe00640 1 --2- 172.21.15.6:0/4212766655 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f31e80d4930 0x7f31e80c9910 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f31e0001eb0 tx=0x7f31e000c890 comp rx=0 tx=0).stop 2024-03-04T09:26:30.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:30.979+0000 7f31efe00640 1 -- 172.21.15.6:0/4212766655 shutdown_connections 2024-03-04T09:26:30.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:30.979+0000 7f31efe00640 1 --2- 172.21.15.6:0/4212766655 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f31e80d4930 0x7f31e80c9910 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:30.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:30.979+0000 7f31efe00640 1 --2- 172.21.15.6:0/4212766655 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f31e80d0bb0 0x7f31e814fbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:30.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:30.979+0000 7f31efe00640 1 --2- 172.21.15.6:0/4212766655 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f31c806b4d0 0x7f31c806d990 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:30.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:30.979+0000 7f31efe00640 1 --2- 172.21.15.6:0/4212766655 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f31e80d07d0 0x7f31e814c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:30.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:30.979+0000 7f31efe00640 1 -- 172.21.15.6:0/4212766655 >> 172.21.15.6:0/4212766655 conn(0x7f31e80d3e10 msgr2=0x7f31e806d090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:30.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:30.980+0000 7f31efe00640 1 -- 172.21.15.6:0/4212766655 shutdown_connections 2024-03-04T09:26:30.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:30.980+0000 7f31efe00640 1 -- 172.21.15.6:0/4212766655 wait complete. 2024-03-04T09:26:31.018 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T09:26:31.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.203+0000 7f3410a00640 1 Processor -- start 2024-03-04T09:26:31.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.204+0000 7f3410a00640 1 -- start start 2024-03-04T09:26:31.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.205+0000 7f3410a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f340c0af1a0 0x7f340c15c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:31.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.205+0000 7f3410a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f340c0af580 0x7f340c0b4c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:31.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.205+0000 7f3410a00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f340c15cc40 con 0x7f340c0cdc20 2024-03-04T09:26:31.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.205+0000 7f3410a00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f340c15cf20 con 0x7f340c0af1a0 2024-03-04T09:26:31.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.205+0000 7f3410a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f340c15d200 con 0x7f340c0af580 2024-03-04T09:26:31.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.205+0000 7f3409e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f340c0af1a0 0x7f340c15c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:31.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.206+0000 7f3409e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f340c0af1a0 0x7f340c15c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:40644/0 (socket says 172.21.15.6:40644) 2024-03-04T09:26:31.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.206+0000 7f3409e00640 1 -- 172.21.15.6:0/4043931097 learned_addr learned my addr 172.21.15.6:0/4043931097 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:31.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.206+0000 7f3403e00640 1 --2- 172.21.15.6:0/4043931097 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f340c0af580 0x7f340c0b4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:31.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.206+0000 7f3409e00640 1 -- 172.21.15.6:0/4043931097 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f340c0af580 msgr2=0x7f340c0b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:31.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.206+0000 7f3409e00640 1 --2- 172.21.15.6:0/4043931097 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f340c0af580 0x7f340c0b4c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:31.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.206+0000 7f3409e00640 1 -- 172.21.15.6:0/4043931097 >> v1:172.21.15.146:6789/0 conn(0x7f340c0cdc20 legacy=0x7f340c0b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:31.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.206+0000 7f3409e00640 1 -- 172.21.15.6:0/4043931097 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f340c15d4e0 con 0x7f340c0af1a0 2024-03-04T09:26:31.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.208+0000 7f3409e00640 1 --2- 172.21.15.6:0/4043931097 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f340c0af1a0 0x7f340c15c600 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f33fc009870 tx=0x7f33fc02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=979917e7a97f081c server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:31.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.208+0000 7f3409400640 1 -- 172.21.15.6:0/4043931097 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33fc035020 con 0x7f340c0af1a0 2024-03-04T09:26:31.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.208+0000 7f3409400640 1 -- 172.21.15.6:0/4043931097 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33fc031640 con 0x7f340c0af1a0 2024-03-04T09:26:31.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.208+0000 7f3409400640 1 -- 172.21.15.6:0/4043931097 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33fc02d040 con 0x7f340c0af1a0 2024-03-04T09:26:31.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.209+0000 7f3410a00640 1 -- 172.21.15.6:0/4043931097 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f340c0af1a0 msgr2=0x7f340c15c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:31.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.209+0000 7f3410a00640 1 --2- 172.21.15.6:0/4043931097 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f340c0af1a0 0x7f340c15c600 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f33fc009870 tx=0x7f33fc02b8f0 comp rx=0 tx=0).stop 2024-03-04T09:26:31.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.209+0000 7f3410a00640 1 -- 172.21.15.6:0/4043931097 shutdown_connections 2024-03-04T09:26:31.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.209+0000 7f3410a00640 1 --2- 172.21.15.6:0/4043931097 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f340c0af580 0x7f340c0b4c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:31.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.209+0000 7f3410a00640 1 --2- 172.21.15.6:0/4043931097 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f340c0af1a0 0x7f340c15c600 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:31.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.209+0000 7f3410a00640 1 -- 172.21.15.6:0/4043931097 >> 172.21.15.6:0/4043931097 conn(0x7f340c06afb0 msgr2=0x7f340c06b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:31.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.209+0000 7f3410a00640 1 -- 172.21.15.6:0/4043931097 shutdown_connections 2024-03-04T09:26:31.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.209+0000 7f3410a00640 1 -- 172.21.15.6:0/4043931097 wait complete. 2024-03-04T09:26:31.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.211+0000 7f3410a00640 1 Processor -- start 2024-03-04T09:26:31.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.211+0000 7f3410a00640 1 -- start start 2024-03-04T09:26:31.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.212+0000 7f3410a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f340c0af1a0 0x7f340c1581a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:31.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.212+0000 7f3410a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f340c0af580 0x7f340c15e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:31.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.212+0000 7f3410a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f340c0cdc20 0x7f340c1492f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:31.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.212+0000 7f3410a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f340c0b3660 con 0x7f340c0af1a0 2024-03-04T09:26:31.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.212+0000 7f3410a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f340c0b34e0 con 0x7f340c0cdc20 2024-03-04T09:26:31.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.212+0000 7f3410a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f340c0b37e0 con 0x7f340c0af580 2024-03-04T09:26:31.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.212+0000 7f3409e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f340c0af1a0 0x7f340c1581a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:31.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.212+0000 7f3409e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f340c0af1a0 0x7f340c1581a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:40656/0 (socket says 172.21.15.6:40656) 2024-03-04T09:26:31.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.212+0000 7f3409e00640 1 -- 172.21.15.6:0/464679397 learned_addr learned my addr 172.21.15.6:0/464679397 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:31.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.213+0000 7f3403e00640 1 --2- 172.21.15.6:0/464679397 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f340c0af580 0x7f340c15e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:31.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.213+0000 7f3409e00640 1 -- 172.21.15.6:0/464679397 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f340c0af580 msgr2=0x7f340c15e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:31.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.213+0000 7f340a800640 1 --2- 172.21.15.6:0/464679397 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f340c0cdc20 0x7f340c1492f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:31.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.213+0000 7f3409e00640 1 --2- 172.21.15.6:0/464679397 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f340c0af580 0x7f340c15e040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:31.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.213+0000 7f3409e00640 1 -- 172.21.15.6:0/464679397 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f340c0cdc20 msgr2=0x7f340c1492f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:31.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.213+0000 7f3409e00640 1 --2- 172.21.15.6:0/464679397 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f340c0cdc20 0x7f340c1492f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:31.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.213+0000 7f3409e00640 1 -- 172.21.15.6:0/464679397 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f340c09aa10 con 0x7f340c0af1a0 2024-03-04T09:26:31.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.213+0000 7f340a800640 1 --2- 172.21.15.6:0/464679397 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f340c0cdc20 0x7f340c1492f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:26:31.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.213+0000 7f3409e00640 1 --2- 172.21.15.6:0/464679397 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f340c0af1a0 0x7f340c1581a0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f33fc000c00 tx=0x7f33fc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:31.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.213+0000 7f3402000640 1 -- 172.21.15.6:0/464679397 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33fc040070 con 0x7f340c0af1a0 2024-03-04T09:26:31.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.213+0000 7f3402000640 1 -- 172.21.15.6:0/464679397 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33fc0399f0 con 0x7f340c0af1a0 2024-03-04T09:26:31.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.213+0000 7f3410a00640 1 -- 172.21.15.6:0/464679397 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f340c09acb0 con 0x7f340c0af1a0 2024-03-04T09:26:31.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.213+0000 7f3402000640 1 -- 172.21.15.6:0/464679397 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33fc035040 con 0x7f340c0af1a0 2024-03-04T09:26:31.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.214+0000 7f3410a00640 1 -- 172.21.15.6:0/464679397 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f340c09b270 con 0x7f340c0af1a0 2024-03-04T09:26:31.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.215+0000 7f3410a00640 1 -- 172.21.15.6:0/464679397 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f340c069510 con 0x7f340c0af1a0 2024-03-04T09:26:31.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.216+0000 7f3402000640 1 -- 172.21.15.6:0/464679397 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f33fc04b060 con 0x7f340c0af1a0 2024-03-04T09:26:31.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.217+0000 7f3402000640 1 --2- 172.21.15.6:0/464679397 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f33e806b4d0 0x7f33e806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:31.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.217+0000 7f3402000640 1 -- 172.21.15.6:0/464679397 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f33fc0abcb0 con 0x7f340c0af1a0 2024-03-04T09:26:31.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.218+0000 7f3403e00640 1 --2- 172.21.15.6:0/464679397 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f33e806b4d0 0x7f33e806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:31.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.219+0000 7f3403e00640 1 --2- 172.21.15.6:0/464679397 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f33e806b4d0 0x7f33e806d990 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f340c15eca0 tx=0x7f33f4008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:31.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.222+0000 7f3402000640 1 -- 172.21.15.6:0/464679397 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f33fc07aa00 con 0x7f340c0af1a0 2024-03-04T09:26:31.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.423+0000 7f3410a00640 1 -- 172.21.15.6:0/464679397 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f340c04f3a0 con 0x7f340c0af1a0 2024-03-04T09:26:31.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.426+0000 7f3402000640 1 -- 172.21.15.6:0/464679397 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v32) v1 ==== 74+0+16634 (secure 0 0 0) 0x7f33fc07a300 con 0x7f340c0af1a0 2024-03-04T09:26:31.424 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:26:31.424 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":32,"fsid":"edce0578-5ae4-4402-b45a-552bdef84d7c","created":"2024-03-04T09:24:54.523780+0000","modified":"2024-03-04T09:26:30.444814+0000","last_up_change":"2024-03-04T09:25:07.089049+0000","last_in_change":"2024-03-04T09:24:57.982077+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T09:25:07.935198+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-04T09:26:28.556219+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-04T09:26:29.471227+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a96574f-81f1-4bc6-8463-757520b84a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6825","nonce":1364449401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6827","nonce":1364449401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6831","nonce":1364449401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6829","nonce":1364449401}]},"public_addr":"172.21.15.6:6825/1364449401","cluster_addr":"172.21.15.6:6827/1364449401","heartbeat_back_addr":"172.21.15.6:6831/1364449401","heartbeat_front_addr":"172.21.15.6:6829/1364449401","state":["exists","up"]},{"osd":1,"uuid":"7d79b9c1-6b87-476a-bc36-7db5ee64cd90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6809","nonce":2494103078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6811","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6813","nonce":2494103078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6821","nonce":2494103078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6817","nonce":2494103078}]},"public_addr":"172.21.15.6:6809/2494103078","cluster_addr":"172.21.15.6:6813/2494103078","heartbeat_back_addr":"172.21.15.6:6821/2494103078","heartbeat_front_addr":"172.21.15.6:6817/2494103078","state":["exists","up"]},{"osd":2,"uuid":"492331b9-4ee9-40b6-bacd-85d44cd6c45a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6812","nonce":1500066681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6816","nonce":1500066681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6824","nonce":1500066681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6820","nonce":1500066681}]},"public_addr":"172.21.15.6:6812/1500066681","cluster_addr":"172.21.15.6:6816/1500066681","heartbeat_back_addr":"172.21.15.6:6824/1500066681","heartbeat_front_addr":"172.21.15.6:6820/1500066681","state":["exists","up"]},{"osd":3,"uuid":"21bb2efe-9ba7-4bdc-bc31-180cd1102e21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6801","nonce":1558806123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6803","nonce":1558806123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6807","nonce":1558806123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6805","nonce":1558806123}]},"public_addr":"172.21.15.6:6801/1558806123","cluster_addr":"172.21.15.6:6803/1558806123","heartbeat_back_addr":"172.21.15.6:6807/1558806123","heartbeat_front_addr":"172.21.15.6:6805/1558806123","state":["exists","up"]},{"osd":4,"uuid":"6e2c4b71-19d9-4737-83d9-558705d8644a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6801","nonce":1526135780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6803","nonce":1526135780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6807","nonce":1526135780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6805","nonce":1526135780}]},"public_addr":"172.21.15.146:6801/1526135780","cluster_addr":"172.21.15.146:6803/1526135780","heartbeat_back_addr":"172.21.15.146:6807/1526135780","heartbeat_front_addr":"172.21.15.146:6805/1526135780","state":["exists","up"]},{"osd":5,"uuid":"fb69c525-f1be-4b7d-ab32-6c29b0d6fddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6809","nonce":460262507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6811","nonce":460262507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6815","nonce":460262507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6813","nonce":460262507}]},"public_addr":"172.21.15.146:6809/460262507","cluster_addr":"172.21.15.146:6811/460262507","heartbeat_back_addr":"172.21.15.146:6815/460262507","heartbeat_front_addr":"172.21.15.146:6813/460262507","state":["exists","up"]},{"osd":6,"uuid":"c32e7fed-370d-4401-8ce0-6a4335b0d168","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6823","nonce":1392897493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6827","nonce":1392897493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6831","nonce":1392897493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6829","nonce":1392897493}]},"public_addr":"172.21.15.146:6823/1392897493","cluster_addr":"172.21.15.146:6827/1392897493","heartbeat_back_addr":"172.21.15.146:6831/1392897493","heartbeat_front_addr":"172.21.15.146:6829/1392897493","state":["exists","up"]},{"osd":7,"uuid":"75a79b44-fa5f-426d-a7d3-71f21f78835c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6817","nonce":3268323156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6819","nonce":3268323156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6825","nonce":3268323156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6822","nonce":3268323156}]},"public_addr":"172.21.15.146:6817/3268323156","cluster_addr":"172.21.15.146:6819/3268323156","heartbeat_back_addr":"172.21.15.146:6825/3268323156","heartbeat_front_addr":"172.21.15.146:6822/3268323156","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.310437+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.295850+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.294674+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.526855+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.300023+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.595160+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.346046+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.558741+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T09:26:31.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.429+0000 7f3410a00640 1 -- 172.21.15.6:0/464679397 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f33e806b4d0 msgr2=0x7f33e806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:31.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.429+0000 7f3410a00640 1 --2- 172.21.15.6:0/464679397 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f33e806b4d0 0x7f33e806d990 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f340c15eca0 tx=0x7f33f4008040 comp rx=0 tx=0).stop 2024-03-04T09:26:31.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.429+0000 7f3410a00640 1 -- 172.21.15.6:0/464679397 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f340c0af1a0 msgr2=0x7f340c1581a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:31.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.429+0000 7f3410a00640 1 --2- 172.21.15.6:0/464679397 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f340c0af1a0 0x7f340c1581a0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f33fc000c00 tx=0x7f33fc0046c0 comp rx=0 tx=0).stop 2024-03-04T09:26:31.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.429+0000 7f3410a00640 1 -- 172.21.15.6:0/464679397 shutdown_connections 2024-03-04T09:26:31.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.429+0000 7f3410a00640 1 --2- 172.21.15.6:0/464679397 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f340c0cdc20 0x7f340c1492f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:31.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.429+0000 7f3410a00640 1 --2- 172.21.15.6:0/464679397 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f33e806b4d0 0x7f33e806d990 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:31.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.429+0000 7f3410a00640 1 --2- 172.21.15.6:0/464679397 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f340c0af580 0x7f340c15e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:31.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.429+0000 7f3410a00640 1 --2- 172.21.15.6:0/464679397 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f340c0af1a0 0x7f340c1581a0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:31.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.429+0000 7f3410a00640 1 -- 172.21.15.6:0/464679397 >> 172.21.15.6:0/464679397 conn(0x7f340c06afb0 msgr2=0x7f340c15bd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:31.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.429+0000 7f3410a00640 1 -- 172.21.15.6:0/464679397 shutdown_connections 2024-03-04T09:26:31.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.429+0000 7f3410a00640 1 -- 172.21.15.6:0/464679397 wait complete. 2024-03-04T09:26:31.445 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T09:26:31.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.662+0000 7f4b1ca00640 1 Processor -- start 2024-03-04T09:26:31.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.662+0000 7f4b1ca00640 1 -- start start 2024-03-04T09:26:31.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.664+0000 7f4b1ca00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f4b18145300 0x7f4b180b49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:31.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.664+0000 7f4b1ca00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4b18070f10 0x7f4b180cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:31.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.664+0000 7f4b1ca00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b180714d0 con 0x7f4b180cce90 2024-03-04T09:26:31.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.664+0000 7f4b1ca00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b180cd300 con 0x7f4b18145300 2024-03-04T09:26:31.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.664+0000 7f4b1ca00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b180717b0 con 0x7f4b18070f10 2024-03-04T09:26:31.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.665+0000 7f4b16800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f4b18145300 0x7f4b180b49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:31.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.665+0000 7f4b15e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4b18070f10 0x7f4b180cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:31.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.665+0000 7f4b17200640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f4b180cce90 0x7f4b180cd930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.6:44270/0 (socket says 172.21.15.6:44270) 2024-03-04T09:26:31.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.665+0000 7f4b17200640 1 -- 172.21.15.6:0/635125735 learned_addr learned my addr 172.21.15.6:0/635125735 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:26:31.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.666+0000 7f4b15e00640 1 -- 172.21.15.6:0/635125735 >> v2:172.21.15.146:3300/0 conn(0x7f4b18145300 msgr2=0x7f4b180b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:31.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.666+0000 7f4b15e00640 1 --2- 172.21.15.6:0/635125735 >> v2:172.21.15.146:3300/0 conn(0x7f4b18145300 0x7f4b180b49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:31.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.666+0000 7f4b15e00640 1 -- 172.21.15.6:0/635125735 >> v1:172.21.15.146:6789/0 conn(0x7f4b180cce90 legacy=0x7f4b180cd930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:31.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.666+0000 7f4b15e00640 1 -- 172.21.15.6:0/635125735 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b18071a90 con 0x7f4b18070f10 2024-03-04T09:26:31.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.666+0000 7f4b15400640 1 -- 172.21.15.6:0/635125735 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3969428766 0 0) 0x7f4b180714d0 con 0x7f4b180cce90 2024-03-04T09:26:31.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.667+0000 7f4b15e00640 1 --2- 172.21.15.6:0/635125735 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4b18070f10 0x7f4b180cc950 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f4b040097a0 tx=0x7f4b0402b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bf10443f8da34bad server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:31.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.667+0000 7f4b15400640 1 -- 172.21.15.6:0/635125735 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b04035020 con 0x7f4b18070f10 2024-03-04T09:26:31.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.667+0000 7f4b15400640 1 -- 172.21.15.6:0/635125735 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b04031830 con 0x7f4b18070f10 2024-03-04T09:26:31.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.667+0000 7f4b15400640 1 -- 172.21.15.6:0/635125735 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b04031c00 con 0x7f4b18070f10 2024-03-04T09:26:31.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.668+0000 7f4b1ca00640 1 -- 172.21.15.6:0/635125735 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4b18070f10 msgr2=0x7f4b180cc950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:31.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.668+0000 7f4b1ca00640 1 --2- 172.21.15.6:0/635125735 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4b18070f10 0x7f4b180cc950 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f4b040097a0 tx=0x7f4b0402b8d0 comp rx=0 tx=0).stop 2024-03-04T09:26:31.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.668+0000 7f4b1ca00640 1 -- 172.21.15.6:0/635125735 shutdown_connections 2024-03-04T09:26:31.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.668+0000 7f4b1ca00640 1 --2- 172.21.15.6:0/635125735 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4b18070f10 0x7f4b180cc950 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:31.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.668+0000 7f4b1ca00640 1 --2- 172.21.15.6:0/635125735 >> v2:172.21.15.146:3300/0 conn(0x7f4b18145300 0x7f4b180b49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:31.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.668+0000 7f4b1ca00640 1 -- 172.21.15.6:0/635125735 >> 172.21.15.6:0/635125735 conn(0x7f4b1806b3d0 msgr2=0x7f4b1806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:31.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.668+0000 7f4b1ca00640 1 -- 172.21.15.6:0/635125735 shutdown_connections 2024-03-04T09:26:31.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.668+0000 7f4b1ca00640 1 -- 172.21.15.6:0/635125735 wait complete. 2024-03-04T09:26:31.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.670+0000 7f4b1ca00640 1 Processor -- start 2024-03-04T09:26:31.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.670+0000 7f4b1ca00640 1 -- start start 2024-03-04T09:26:31.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.671+0000 7f4b1ca00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4b18070f10 0x7f4b180c51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:31.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.671+0000 7f4b1ca00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4b180cce90 0x7f4b1806dc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:31.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.671+0000 7f4b1ca00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4b18145300 0x7f4b180c7a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:31.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.671+0000 7f4b1ca00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b181456e0 con 0x7f4b180cce90 2024-03-04T09:26:31.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.671+0000 7f4b1ca00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b180b4ee0 con 0x7f4b18070f10 2024-03-04T09:26:31.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.671+0000 7f4b1ca00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b1815d580 con 0x7f4b18145300 2024-03-04T09:26:31.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.672+0000 7f4b15e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4b180cce90 0x7f4b1806dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:31.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.672+0000 7f4b15e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4b180cce90 0x7f4b1806dc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:40672/0 (socket says 172.21.15.6:40672) 2024-03-04T09:26:31.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.672+0000 7f4b15e00640 1 -- 172.21.15.6:0/1779371439 learned_addr learned my addr 172.21.15.6:0/1779371439 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:31.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.672+0000 7f4b17200640 1 --2- 172.21.15.6:0/1779371439 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4b18145300 0x7f4b180c7a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:31.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.672+0000 7f4b16800640 1 --2- 172.21.15.6:0/1779371439 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4b18070f10 0x7f4b180c51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:31.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.672+0000 7f4b15e00640 1 -- 172.21.15.6:0/1779371439 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4b18145300 msgr2=0x7f4b180c7a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:31.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.672+0000 7f4b15e00640 1 --2- 172.21.15.6:0/1779371439 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4b18145300 0x7f4b180c7a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:31.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.672+0000 7f4b15e00640 1 -- 172.21.15.6:0/1779371439 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4b18070f10 msgr2=0x7f4b180c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:31.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.672+0000 7f4b15e00640 1 --2- 172.21.15.6:0/1779371439 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4b18070f10 0x7f4b180c51f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:31.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.672+0000 7f4b15e00640 1 -- 172.21.15.6:0/1779371439 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b1806cb80 con 0x7f4b180cce90 2024-03-04T09:26:31.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.673+0000 7f4b15e00640 1 --2- 172.21.15.6:0/1779371439 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4b180cce90 0x7f4b1806dc20 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f4b04009890 tx=0x7f4b040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:31.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.673+0000 7f4b0ea00640 1 -- 172.21.15.6:0/1779371439 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b04035040 con 0x7f4b180cce90 2024-03-04T09:26:31.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.673+0000 7f4b0ea00640 1 -- 172.21.15.6:0/1779371439 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b0403b920 con 0x7f4b180cce90 2024-03-04T09:26:31.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.673+0000 7f4b0ea00640 1 -- 172.21.15.6:0/1779371439 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b0403bd10 con 0x7f4b180cce90 2024-03-04T09:26:31.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.673+0000 7f4b1ca00640 1 -- 172.21.15.6:0/1779371439 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b1806ce60 con 0x7f4b180cce90 2024-03-04T09:26:31.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.673+0000 7f4b1ca00640 1 -- 172.21.15.6:0/1779371439 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b1806d420 con 0x7f4b180cce90 2024-03-04T09:26:31.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.674+0000 7f4b1ca00640 1 -- 172.21.15.6:0/1779371439 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b1804f330 con 0x7f4b180cce90 2024-03-04T09:26:31.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.676+0000 7f4b0ea00640 1 -- 172.21.15.6:0/1779371439 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4b04038020 con 0x7f4b180cce90 2024-03-04T09:26:31.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.677+0000 7f4b0ea00640 1 --2- 172.21.15.6:0/1779371439 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f4af806b550 0x7f4af806da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:31.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.677+0000 7f4b0ea00640 1 -- 172.21.15.6:0/1779371439 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4b040ac2e0 con 0x7f4b180cce90 2024-03-04T09:26:31.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.678+0000 7f4b16800640 1 --2- 172.21.15.6:0/1779371439 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f4af806b550 0x7f4af806da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:31.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.679+0000 7f4b16800640 1 --2- 172.21.15.6:0/1779371439 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f4af806b550 0x7f4af806da10 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f4b180c7920 tx=0x7f4b08008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:31.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.681+0000 7f4b0ea00640 1 -- 172.21.15.6:0/1779371439 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4b0407b150 con 0x7f4b180cce90 2024-03-04T09:26:31.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.879+0000 7f4b1ca00640 1 -- 172.21.15.6:0/1779371439 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4b180b4ca0 con 0x7f4b180cce90 2024-03-04T09:26:31.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.882+0000 7f4b0ea00640 1 -- 172.21.15.6:0/1779371439 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v32) v1 ==== 74+0+16634 (secure 0 0 0) 0x7f4b04031800 con 0x7f4b180cce90 2024-03-04T09:26:31.880 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:26:31.880 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":32,"fsid":"edce0578-5ae4-4402-b45a-552bdef84d7c","created":"2024-03-04T09:24:54.523780+0000","modified":"2024-03-04T09:26:30.444814+0000","last_up_change":"2024-03-04T09:25:07.089049+0000","last_in_change":"2024-03-04T09:24:57.982077+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T09:25:07.935198+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-04T09:26:28.556219+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-04T09:26:29.471227+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a96574f-81f1-4bc6-8463-757520b84a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6825","nonce":1364449401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6827","nonce":1364449401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6831","nonce":1364449401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6829","nonce":1364449401}]},"public_addr":"172.21.15.6:6825/1364449401","cluster_addr":"172.21.15.6:6827/1364449401","heartbeat_back_addr":"172.21.15.6:6831/1364449401","heartbeat_front_addr":"172.21.15.6:6829/1364449401","state":["exists","up"]},{"osd":1,"uuid":"7d79b9c1-6b87-476a-bc36-7db5ee64cd90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6809","nonce":2494103078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6811","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6813","nonce":2494103078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6821","nonce":2494103078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6817","nonce":2494103078}]},"public_addr":"172.21.15.6:6809/2494103078","cluster_addr":"172.21.15.6:6813/2494103078","heartbeat_back_addr":"172.21.15.6:6821/2494103078","heartbeat_front_addr":"172.21.15.6:6817/2494103078","state":["exists","up"]},{"osd":2,"uuid":"492331b9-4ee9-40b6-bacd-85d44cd6c45a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6812","nonce":1500066681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6816","nonce":1500066681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6824","nonce":1500066681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6820","nonce":1500066681}]},"public_addr":"172.21.15.6:6812/1500066681","cluster_addr":"172.21.15.6:6816/1500066681","heartbeat_back_addr":"172.21.15.6:6824/1500066681","heartbeat_front_addr":"172.21.15.6:6820/1500066681","state":["exists","up"]},{"osd":3,"uuid":"21bb2efe-9ba7-4bdc-bc31-180cd1102e21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6801","nonce":1558806123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6803","nonce":1558806123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6807","nonce":1558806123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6805","nonce":1558806123}]},"public_addr":"172.21.15.6:6801/1558806123","cluster_addr":"172.21.15.6:6803/1558806123","heartbeat_back_addr":"172.21.15.6:6807/1558806123","heartbeat_front_addr":"172.21.15.6:6805/1558806123","state":["exists","up"]},{"osd":4,"uuid":"6e2c4b71-19d9-4737-83d9-558705d8644a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6801","nonce":1526135780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6803","nonce":1526135780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6807","nonce":1526135780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6805","nonce":1526135780}]},"public_addr":"172.21.15.146:6801/1526135780","cluster_addr":"172.21.15.146:6803/1526135780","heartbeat_back_addr":"172.21.15.146:6807/1526135780","heartbeat_front_addr":"172.21.15.146:6805/1526135780","state":["exists","up"]},{"osd":5,"uuid":"fb69c525-f1be-4b7d-ab32-6c29b0d6fddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6809","nonce":460262507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6811","nonce":460262507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6815","nonce":460262507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6813","nonce":460262507}]},"public_addr":"172.21.15.146:6809/460262507","cluster_addr":"172.21.15.146:6811/460262507","heartbeat_back_addr":"172.21.15.146:6815/460262507","heartbeat_front_addr":"172.21.15.146:6813/460262507","state":["exists","up"]},{"osd":6,"uuid":"c32e7fed-370d-4401-8ce0-6a4335b0d168","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6823","nonce":1392897493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6827","nonce":1392897493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6831","nonce":1392897493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6829","nonce":1392897493}]},"public_addr":"172.21.15.146:6823/1392897493","cluster_addr":"172.21.15.146:6827/1392897493","heartbeat_back_addr":"172.21.15.146:6831/1392897493","heartbeat_front_addr":"172.21.15.146:6829/1392897493","state":["exists","up"]},{"osd":7,"uuid":"75a79b44-fa5f-426d-a7d3-71f21f78835c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6817","nonce":3268323156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6819","nonce":3268323156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6825","nonce":3268323156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6822","nonce":3268323156}]},"public_addr":"172.21.15.146:6817/3268323156","cluster_addr":"172.21.15.146:6819/3268323156","heartbeat_back_addr":"172.21.15.146:6825/3268323156","heartbeat_front_addr":"172.21.15.146:6822/3268323156","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.310437+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.295850+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.294674+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.526855+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.300023+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.595160+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.346046+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.558741+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T09:26:31.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.884+0000 7f4b1ca00640 1 -- 172.21.15.6:0/1779371439 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f4af806b550 msgr2=0x7f4af806da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:31.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.884+0000 7f4b1ca00640 1 --2- 172.21.15.6:0/1779371439 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f4af806b550 0x7f4af806da10 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f4b180c7920 tx=0x7f4b08008040 comp rx=0 tx=0).stop 2024-03-04T09:26:31.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.884+0000 7f4b1ca00640 1 -- 172.21.15.6:0/1779371439 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4b180cce90 msgr2=0x7f4b1806dc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:31.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.884+0000 7f4b1ca00640 1 --2- 172.21.15.6:0/1779371439 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4b180cce90 0x7f4b1806dc20 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f4b04009890 tx=0x7f4b040046c0 comp rx=0 tx=0).stop 2024-03-04T09:26:31.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.884+0000 7f4b1ca00640 1 -- 172.21.15.6:0/1779371439 shutdown_connections 2024-03-04T09:26:31.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.884+0000 7f4b1ca00640 1 --2- 172.21.15.6:0/1779371439 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f4af806b550 0x7f4af806da10 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:31.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.884+0000 7f4b1ca00640 1 --2- 172.21.15.6:0/1779371439 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4b18145300 0x7f4b180c7a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:31.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.884+0000 7f4b1ca00640 1 --2- 172.21.15.6:0/1779371439 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4b180cce90 0x7f4b1806dc20 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:31.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.884+0000 7f4b1ca00640 1 --2- 172.21.15.6:0/1779371439 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4b18070f10 0x7f4b180c51f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:31.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.884+0000 7f4b1ca00640 1 -- 172.21.15.6:0/1779371439 >> 172.21.15.6:0/1779371439 conn(0x7f4b1806b3d0 msgr2=0x7f4b1815e020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:31.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.885+0000 7f4b1ca00640 1 -- 172.21.15.6:0/1779371439 shutdown_connections 2024-03-04T09:26:31.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:31.885+0000 7f4b1ca00640 1 -- 172.21.15.6:0/1779371439 wait complete. 2024-03-04T09:26:31.900 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-04T09:26:32.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:32.127+0000 7faef6e00640 1 Processor -- start 2024-03-04T09:26:32.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:32.127+0000 7faef6e00640 1 -- start start 2024-03-04T09:26:32.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:32.129+0000 7faef6e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7faef0162bb0 0x7faef0163040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:32.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:32.129+0000 7faef6e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faef00adb00 0x7faef00adf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:32.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:32.130+0000 7faef6e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faef0163950 con 0x7faef00ccad0 2024-03-04T09:26:32.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:32.130+0000 7faef6e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faef00acd40 con 0x7faef0162bb0 2024-03-04T09:26:32.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:32.130+0000 7faef6e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faef00ad020 con 0x7faef00adb00 2024-03-04T09:26:32.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:32.130+0000 7faeefe00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7faef00ccad0 0x7faef00cceb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:36994/0 (socket says 172.21.15.6:36994) 2024-03-04T09:26:32.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:32.130+0000 7faeefe00640 1 -- 172.21.15.6:0/3370885140 learned_addr learned my addr 172.21.15.6:0/3370885140 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:26:32.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:32.130+0000 7faee7e00640 1 --2- 172.21.15.6:0/3370885140 >> v2:172.21.15.6:3300/0 conn(0x7faef0162bb0 0x7faef0163040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:32.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:32.130+0000 7faef4e00640 1 --2- 172.21.15.6:0/3370885140 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faef00adb00 0x7faef00adf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:32.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:32.130+0000 7faee7e00640 1 -- 172.21.15.6:0/3370885140 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faef00adb00 msgr2=0x7faef00adf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:32.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:32.130+0000 7faee7e00640 1 --2- 172.21.15.6:0/3370885140 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faef00adb00 0x7faef00adf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:32.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:32.130+0000 7faee7e00640 1 -- 172.21.15.6:0/3370885140 >> v1:172.21.15.6:6789/0 conn(0x7faef00ccad0 legacy=0x7faef00cceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:32.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:32.130+0000 7faee7e00640 1 -- 172.21.15.6:0/3370885140 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faef00ad300 con 0x7faef0162bb0 2024-03-04T09:26:32.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:32.130+0000 7faeef400640 1 -- 172.21.15.6:0/3370885140 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 124882803 0 0) 0x7faef0163950 con 0x7faef00ccad0 2024-03-04T09:26:32.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:32.131+0000 7faee7e00640 1 --2- 172.21.15.6:0/3370885140 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faef0162bb0 0x7faef0163040 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7faed80097a0 tx=0x7faed802b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=227c1a3bf94cf004 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:32.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:32.132+0000 7faeef400640 1 -- 172.21.15.6:0/3370885140 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faed8035020 con 0x7faef0162bb0 2024-03-04T09:26:32.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:32.132+0000 7faeef400640 1 -- 172.21.15.6:0/3370885140 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faed8031810 con 0x7faef0162bb0 2024-03-04T09:26:32.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:32.132+0000 7faeef400640 1 -- 172.21.15.6:0/3370885140 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faed8031be0 con 0x7faef0162bb0 2024-03-04T09:26:32.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:32.132+0000 7faef6e00640 1 -- 172.21.15.6:0/3370885140 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faef0162bb0 msgr2=0x7faef0163040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:32.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:32.132+0000 7faef6e00640 1 --2- 172.21.15.6:0/3370885140 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faef0162bb0 0x7faef0163040 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7faed80097a0 tx=0x7faed802b8b0 comp rx=0 tx=0).stop 2024-03-04T09:26:32.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:32.133+0000 7faef6e00640 1 -- 172.21.15.6:0/3370885140 shutdown_connections 2024-03-04T09:26:32.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:32.133+0000 7faef6e00640 1 --2- 172.21.15.6:0/3370885140 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faef00adb00 0x7faef00adf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:32.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:32.133+0000 7faef6e00640 1 --2- 172.21.15.6:0/3370885140 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faef0162bb0 0x7faef0163040 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:32.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:32.133+0000 7faef6e00640 1 -- 172.21.15.6:0/3370885140 >> 172.21.15.6:0/3370885140 conn(0x7faef006b3d0 msgr2=0x7faef006b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:32.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:32.133+0000 7faef6e00640 1 -- 172.21.15.6:0/3370885140 shutdown_connections 2024-03-04T09:26:32.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:32.133+0000 7faef6e00640 1 -- 172.21.15.6:0/3370885140 wait complete. 2024-03-04T09:26:32.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:32.134+0000 7faef6e00640 1 Processor -- start 2024-03-04T09:26:32.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:32.135+0000 7faef6e00640 1 -- start start 2024-03-04T09:26:32.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:32.135+0000 7faef6e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faef00adb00 0x7faef009caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:32.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:32.135+0000 7faef6e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faef00ccad0 0x7faef009d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:32.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:32.135+0000 7faef6e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faef0162bb0 0x7faef014dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:32.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:32.135+0000 7faef6e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faef006c980 con 0x7faef00adb00 2024-03-04T09:26:32.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:32.135+0000 7faef6e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faef006e7d0 con 0x7faef0162bb0 2024-03-04T09:26:32.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:32.135+0000 7faef6e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faef0163580 con 0x7faef00ccad0 2024-03-04T09:26:32.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:32.136+0000 7faeefe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faef00adb00 0x7faef009caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:32.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:32.136+0000 7faeefe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faef00adb00 0x7faef009caa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:40690/0 (socket says 172.21.15.6:40690) 2024-03-04T09:26:32.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:32.136+0000 7faeefe00640 1 -- 172.21.15.6:0/1580755124 learned_addr learned my addr 172.21.15.6:0/1580755124 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:32.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:32.136+0000 7faee7e00640 1 --2- 172.21.15.6:0/1580755124 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faef00ccad0 0x7faef009d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:32.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:32.136+0000 7faef4e00640 1 --2- 172.21.15.6:0/1580755124 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faef0162bb0 0x7faef014dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:32.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:32.136+0000 7faeefe00640 1 -- 172.21.15.6:0/1580755124 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faef00ccad0 msgr2=0x7faef009d1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:32.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:32.136+0000 7faeefe00640 1 --2- 172.21.15.6:0/1580755124 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faef00ccad0 0x7faef009d1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:32.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:32.136+0000 7faeefe00640 1 -- 172.21.15.6:0/1580755124 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faef0162bb0 msgr2=0x7faef014dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:32.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:32.136+0000 7faeefe00640 1 --2- 172.21.15.6:0/1580755124 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faef0162bb0 0x7faef014dbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:32.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:32.136+0000 7faeefe00640 1 -- 172.21.15.6:0/1580755124 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faef009ba00 con 0x7faef00adb00 2024-03-04T09:26:32.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:32.136+0000 7faee7e00640 1 --2- 172.21.15.6:0/1580755124 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faef00ccad0 0x7faef009d1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:26:32.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:32.137+0000 7faeefe00640 1 --2- 172.21.15.6:0/1580755124 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faef00adb00 0x7faef009caa0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7faee000c460 tx=0x7faee000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:32.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:32.137+0000 7faeecc00640 1 -- 172.21.15.6:0/1580755124 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faee0018070 con 0x7faef00adb00 2024-03-04T09:26:32.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:32.137+0000 7faeecc00640 1 -- 172.21.15.6:0/1580755124 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faee0007500 con 0x7faef00adb00 2024-03-04T09:26:32.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:32.137+0000 7faeecc00640 1 -- 172.21.15.6:0/1580755124 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faee0004a20 con 0x7faef00adb00 2024-03-04T09:26:32.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:32.137+0000 7faef6e00640 1 -- 172.21.15.6:0/1580755124 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faef009bce0 con 0x7faef00adb00 2024-03-04T09:26:32.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:32.137+0000 7faef6e00640 1 -- 172.21.15.6:0/1580755124 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faef009c2a0 con 0x7faef00adb00 2024-03-04T09:26:32.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:32.138+0000 7faef6e00640 1 -- 172.21.15.6:0/1580755124 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faef004f330 con 0x7faef00adb00 2024-03-04T09:26:32.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:32.139+0000 7faeecc00640 1 -- 172.21.15.6:0/1580755124 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7faee0004bc0 con 0x7faef00adb00 2024-03-04T09:26:32.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:32.140+0000 7faeecc00640 1 --2- 172.21.15.6:0/1580755124 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7faecc06b4d0 0x7faecc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:32.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:32.141+0000 7faeecc00640 1 -- 172.21.15.6:0/1580755124 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7faee00895b0 con 0x7faef00adb00 2024-03-04T09:26:32.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:32.141+0000 7faee7e00640 1 --2- 172.21.15.6:0/1580755124 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7faecc06b4d0 0x7faecc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:32.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:32.142+0000 7faee7e00640 1 --2- 172.21.15.6:0/1580755124 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7faecc06b4d0 0x7faecc06d990 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7faed8036000 tx=0x7faed8008030 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:32.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:32.144+0000 7faeecc00640 1 -- 172.21.15.6:0/1580755124 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7faee0058390 con 0x7faef00adb00 2024-03-04T09:26:32.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:32.387+0000 7faef6e00640 1 -- 172.21.15.6:0/1580755124 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7faef0163170 con 0x7faef00adb00 2024-03-04T09:26:33.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:33.014+0000 7faeecc00640 1 -- 172.21.15.6:0/1580755124 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v15) v1 ==== 118+0+0 (secure 0 0 0) 0x7faee0057c90 con 0x7faef00adb00 2024-03-04T09:26:33.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:33.019+0000 7faef6e00640 1 -- 172.21.15.6:0/1580755124 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7faecc06b4d0 msgr2=0x7faecc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:33.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:33.019+0000 7faef6e00640 1 --2- 172.21.15.6:0/1580755124 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7faecc06b4d0 0x7faecc06d990 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7faed8036000 tx=0x7faed8008030 comp rx=0 tx=0).stop 2024-03-04T09:26:33.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:33.019+0000 7faef6e00640 1 -- 172.21.15.6:0/1580755124 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faef00adb00 msgr2=0x7faef009caa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:33.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:33.019+0000 7faef6e00640 1 --2- 172.21.15.6:0/1580755124 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faef00adb00 0x7faef009caa0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7faee000c460 tx=0x7faee000c920 comp rx=0 tx=0).stop 2024-03-04T09:26:33.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:33.019+0000 7faef6e00640 1 -- 172.21.15.6:0/1580755124 shutdown_connections 2024-03-04T09:26:33.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:33.019+0000 7faef6e00640 1 --2- 172.21.15.6:0/1580755124 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faef0162bb0 0x7faef014dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:33.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:33.019+0000 7faef6e00640 1 --2- 172.21.15.6:0/1580755124 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7faecc06b4d0 0x7faecc06d990 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:33.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:33.019+0000 7faef6e00640 1 --2- 172.21.15.6:0/1580755124 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faef00ccad0 0x7faef009d1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:33.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:33.019+0000 7faef6e00640 1 --2- 172.21.15.6:0/1580755124 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faef00adb00 0x7faef009caa0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:33.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:33.019+0000 7faef6e00640 1 -- 172.21.15.6:0/1580755124 >> 172.21.15.6:0/1580755124 conn(0x7faef006b3d0 msgr2=0x7faef0160d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:33.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:33.019+0000 7faef6e00640 1 -- 172.21.15.6:0/1580755124 shutdown_connections 2024-03-04T09:26:33.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:33.020+0000 7faef6e00640 1 -- 172.21.15.6:0/1580755124 wait complete. 2024-03-04T09:26:33.039 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-04T09:26:33.040 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-04T09:26:33.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:33.271+0000 7f2a3e000640 1 Processor -- start 2024-03-04T09:26:33.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:33.271+0000 7f2a3e000640 1 -- start start 2024-03-04T09:26:33.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:33.273+0000 7f2a3e000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f2a38145430 0x7f2a3815f810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:33.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:33.273+0000 7f2a3e000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2a3815cc90 0x7f2a380b1b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:33.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:33.273+0000 7f2a3e000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f2a380b2080 0x7f2a3806c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:33.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:33.273+0000 7f2a3e000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a3815fe40 con 0x7f2a38145430 2024-03-04T09:26:33.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:33.273+0000 7f2a3e000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a38160120 con 0x7f2a380b2080 2024-03-04T09:26:33.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:33.273+0000 7f2a3e000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a38160400 con 0x7f2a3815cc90 2024-03-04T09:26:33.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:33.273+0000 7f2a3cc00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f2a38145430 0x7f2a3815f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:33.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:33.273+0000 7f2a3cc00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f2a38145430 0x7f2a3815f810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:40702/0 (socket says 172.21.15.6:40702) 2024-03-04T09:26:33.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:33.273+0000 7f2a3cc00640 1 -- 172.21.15.6:0/214957460 learned_addr learned my addr 172.21.15.6:0/214957460 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:33.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:33.274+0000 7f2a37e00640 1 --2- 172.21.15.6:0/214957460 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2a3815cc90 0x7f2a380b1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:33.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:33.274+0000 7f2a3d600640 1 --2- 172.21.15.6:0/214957460 >> v2:172.21.15.146:3300/0 conn(0x7f2a380b2080 0x7f2a3806c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:33.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:33.274+0000 7f2a3cc00640 1 -- 172.21.15.6:0/214957460 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2a3815cc90 msgr2=0x7f2a380b1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:33.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:33.274+0000 7f2a3cc00640 1 --2- 172.21.15.6:0/214957460 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2a3815cc90 0x7f2a380b1b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:33.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:33.274+0000 7f2a3cc00640 1 -- 172.21.15.6:0/214957460 >> v2:172.21.15.146:3300/0 conn(0x7f2a380b2080 msgr2=0x7f2a3806c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:33.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:33.274+0000 7f2a3cc00640 1 --2- 172.21.15.6:0/214957460 >> v2:172.21.15.146:3300/0 conn(0x7f2a380b2080 0x7f2a3806c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:33.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:33.274+0000 7f2a3cc00640 1 -- 172.21.15.6:0/214957460 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a381606e0 con 0x7f2a38145430 2024-03-04T09:26:33.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:33.274+0000 7f2a3d600640 1 --2- 172.21.15.6:0/214957460 >> v2:172.21.15.146:3300/0 conn(0x7f2a380b2080 0x7f2a3806c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:26:33.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:33.275+0000 7f2a3cc00640 1 --2- 172.21.15.6:0/214957460 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2a38145430 0x7f2a3815f810 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f2a3806eaf0 tx=0x7f2a2802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=170086dfeb2bf5ff server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:33.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:33.275+0000 7f2a37400640 1 -- 172.21.15.6:0/214957460 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a2802ca70 con 0x7f2a38145430 2024-03-04T09:26:33.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:33.275+0000 7f2a37400640 1 -- 172.21.15.6:0/214957460 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a2802cc10 con 0x7f2a38145430 2024-03-04T09:26:33.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:33.275+0000 7f2a37400640 1 -- 172.21.15.6:0/214957460 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a280343f0 con 0x7f2a38145430 2024-03-04T09:26:33.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:33.276+0000 7f2a3e000640 1 -- 172.21.15.6:0/214957460 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2a38145430 msgr2=0x7f2a3815f810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:33.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:33.276+0000 7f2a3e000640 1 --2- 172.21.15.6:0/214957460 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2a38145430 0x7f2a3815f810 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f2a3806eaf0 tx=0x7f2a2802bed0 comp rx=0 tx=0).stop 2024-03-04T09:26:33.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:33.276+0000 7f2a3e000640 1 -- 172.21.15.6:0/214957460 shutdown_connections 2024-03-04T09:26:33.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:33.276+0000 7f2a3e000640 1 --2- 172.21.15.6:0/214957460 >> v2:172.21.15.146:3300/0 conn(0x7f2a380b2080 0x7f2a3806c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:33.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:33.276+0000 7f2a3e000640 1 --2- 172.21.15.6:0/214957460 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2a3815cc90 0x7f2a380b1b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:33.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:33.276+0000 7f2a3e000640 1 --2- 172.21.15.6:0/214957460 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2a38145430 0x7f2a3815f810 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:33.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:33.276+0000 7f2a3e000640 1 -- 172.21.15.6:0/214957460 >> 172.21.15.6:0/214957460 conn(0x7f2a3806b3a0 msgr2=0x7f2a3806b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:33.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:33.276+0000 7f2a3e000640 1 -- 172.21.15.6:0/214957460 shutdown_connections 2024-03-04T09:26:33.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:33.276+0000 7f2a3e000640 1 -- 172.21.15.6:0/214957460 wait complete. 2024-03-04T09:26:33.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:33.278+0000 7f2a3e000640 1 Processor -- start 2024-03-04T09:26:33.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:33.278+0000 7f2a3e000640 1 -- start start 2024-03-04T09:26:33.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:33.279+0000 7f2a3e000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2a380b2080 0x7f2a3815b960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:33.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:33.279+0000 7f2a3e000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2a38145430 0x7f2a38160940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:33.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:33.279+0000 7f2a3e000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2a3815cc90 0x7f2a380ce170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:33.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:33.279+0000 7f2a3e000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a38145990 con 0x7f2a3815cc90 2024-03-04T09:26:33.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:33.279+0000 7f2a3e000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a38145810 con 0x7f2a380b2080 2024-03-04T09:26:33.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:33.279+0000 7f2a3e000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a380cde00 con 0x7f2a38145430 2024-03-04T09:26:33.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:33.280+0000 7f2a3d600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2a3815cc90 0x7f2a380ce170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:33.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:33.280+0000 7f2a3d600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2a3815cc90 0x7f2a380ce170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:40718/0 (socket says 172.21.15.6:40718) 2024-03-04T09:26:33.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:33.280+0000 7f2a3d600640 1 -- 172.21.15.6:0/570098389 learned_addr learned my addr 172.21.15.6:0/570098389 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:33.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:33.280+0000 7f2a3cc00640 1 --2- 172.21.15.6:0/570098389 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2a380b2080 0x7f2a3815b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:33.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:33.280+0000 7f2a3d600640 1 -- 172.21.15.6:0/570098389 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2a38145430 msgr2=0x7f2a38160940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:33.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:33.280+0000 7f2a3d600640 1 --2- 172.21.15.6:0/570098389 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2a38145430 0x7f2a38160940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:33.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:33.280+0000 7f2a3d600640 1 -- 172.21.15.6:0/570098389 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2a380b2080 msgr2=0x7f2a3815b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:33.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:33.280+0000 7f2a3d600640 1 --2- 172.21.15.6:0/570098389 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2a380b2080 0x7f2a3815b960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:33.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:33.280+0000 7f2a3d600640 1 -- 172.21.15.6:0/570098389 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a3815bea0 con 0x7f2a3815cc90 2024-03-04T09:26:33.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:33.280+0000 7f2a3d600640 1 --2- 172.21.15.6:0/570098389 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2a3815cc90 0x7f2a380ce170 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f2a3000c7f0 tx=0x7f2a3000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:33.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:33.281+0000 7f2a34c00640 1 -- 172.21.15.6:0/570098389 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a30015020 con 0x7f2a3815cc90 2024-03-04T09:26:33.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:33.281+0000 7f2a34c00640 1 -- 172.21.15.6:0/570098389 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a30004230 con 0x7f2a3815cc90 2024-03-04T09:26:33.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:33.281+0000 7f2a34c00640 1 -- 172.21.15.6:0/570098389 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a30007620 con 0x7f2a3815cc90 2024-03-04T09:26:33.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:33.281+0000 7f2a3e000640 1 -- 172.21.15.6:0/570098389 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a38160e80 con 0x7f2a3815cc90 2024-03-04T09:26:33.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:33.281+0000 7f2a3e000640 1 -- 172.21.15.6:0/570098389 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a3813a260 con 0x7f2a3815cc90 2024-03-04T09:26:33.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:33.282+0000 7f2a3e000640 1 -- 172.21.15.6:0/570098389 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a3804f330 con 0x7f2a3815cc90 2024-03-04T09:26:33.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:33.283+0000 7f2a34c00640 1 -- 172.21.15.6:0/570098389 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2a30018020 con 0x7f2a3815cc90 2024-03-04T09:26:33.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:33.285+0000 7f2a34c00640 1 --2- 172.21.15.6:0/570098389 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2a1806b4d0 0x7f2a1806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:33.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:33.285+0000 7f2a34c00640 1 -- 172.21.15.6:0/570098389 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2a3008a470 con 0x7f2a3815cc90 2024-03-04T09:26:33.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:33.285+0000 7f2a3cc00640 1 --2- 172.21.15.6:0/570098389 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2a1806b4d0 0x7f2a1806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:33.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:33.286+0000 7f2a3cc00640 1 --2- 172.21.15.6:0/570098389 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2a1806b4d0 0x7f2a1806d990 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f2a28035000 tx=0x7f2a28030000 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:33.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:33.289+0000 7f2a34c00640 1 -- 172.21.15.6:0/570098389 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2a3001f020 con 0x7f2a3815cc90 2024-03-04T09:26:33.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:33.533+0000 7f2a3e000640 1 -- 172.21.15.6:0/570098389 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f2a3806e240 con 0x7f2a3815cc90 2024-03-04T09:26:33.923 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T09:26:33.926+0000 7f4379600640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-04T09:26:34.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.018+0000 7f2a34c00640 1 -- 172.21.15.6:0/570098389 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v16) v1 ==== 122+0+0 (secure 0 0 0) 0x7f2a300593a0 con 0x7f2a3815cc90 2024-03-04T09:26:34.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.021+0000 7f2a3e000640 1 -- 172.21.15.6:0/570098389 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2a1806b4d0 msgr2=0x7f2a1806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:34.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.021+0000 7f2a3e000640 1 --2- 172.21.15.6:0/570098389 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2a1806b4d0 0x7f2a1806d990 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f2a28035000 tx=0x7f2a28030000 comp rx=0 tx=0).stop 2024-03-04T09:26:34.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.021+0000 7f2a3e000640 1 -- 172.21.15.6:0/570098389 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2a3815cc90 msgr2=0x7f2a380ce170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:34.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.021+0000 7f2a3e000640 1 --2- 172.21.15.6:0/570098389 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2a3815cc90 0x7f2a380ce170 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f2a3000c7f0 tx=0x7f2a3000ccb0 comp rx=0 tx=0).stop 2024-03-04T09:26:34.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.021+0000 7f2a3e000640 1 -- 172.21.15.6:0/570098389 shutdown_connections 2024-03-04T09:26:34.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.021+0000 7f2a3e000640 1 --2- 172.21.15.6:0/570098389 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2a3815cc90 0x7f2a380ce170 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:34.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.021+0000 7f2a3e000640 1 --2- 172.21.15.6:0/570098389 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2a1806b4d0 0x7f2a1806d990 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:34.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.021+0000 7f2a3e000640 1 --2- 172.21.15.6:0/570098389 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2a38145430 0x7f2a38160940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:34.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.021+0000 7f2a3e000640 1 --2- 172.21.15.6:0/570098389 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2a380b2080 0x7f2a3815b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:34.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.021+0000 7f2a3e000640 1 -- 172.21.15.6:0/570098389 >> 172.21.15.6:0/570098389 conn(0x7f2a3806b3a0 msgr2=0x7f2a3815d2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:34.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.022+0000 7f2a3e000640 1 -- 172.21.15.6:0/570098389 shutdown_connections 2024-03-04T09:26:34.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.022+0000 7f2a3e000640 1 -- 172.21.15.6:0/570098389 wait complete. 2024-03-04T09:26:34.044 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-04T09:26:34.044 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:26:34.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.267+0000 7f5605800640 1 Processor -- start 2024-03-04T09:26:34.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.268+0000 7f5605800640 1 -- start start 2024-03-04T09:26:34.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.269+0000 7f5605800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f560015e0b0 0x7f56000cf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:34.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.269+0000 7f5605800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f56000d4a10 0x7f56000d0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:34.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.269+0000 7f5605800640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56000b3710 con 0x7f56000d0c80 2024-03-04T09:26:34.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.269+0000 7f5605800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56000d10f0 con 0x7f560015e0b0 2024-03-04T09:26:34.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.269+0000 7f5605800640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56000b09e0 con 0x7f56000d4a10 2024-03-04T09:26:34.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.269+0000 7f55ffe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f560015e0b0 0x7f56000cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:34.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.269+0000 7f55ffe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f560015e0b0 0x7f56000cf310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:40720/0 (socket says 172.21.15.6:40720) 2024-03-04T09:26:34.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.269+0000 7f55ffe00640 1 -- 172.21.15.6:0/3415739431 learned_addr learned my addr 172.21.15.6:0/3415739431 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:34.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.270+0000 7f55ff400640 1 --2- 172.21.15.6:0/3415739431 >> v2:172.21.15.146:3300/0 conn(0x7f56000d4a10 0x7f56000d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:34.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.270+0000 7f55ffe00640 1 -- 172.21.15.6:0/3415739431 >> v2:172.21.15.146:3300/0 conn(0x7f56000d4a10 msgr2=0x7f56000d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:34.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.270+0000 7f55ffe00640 1 --2- 172.21.15.6:0/3415739431 >> v2:172.21.15.146:3300/0 conn(0x7f56000d4a10 0x7f56000d0740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:34.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.270+0000 7f55ffe00640 1 -- 172.21.15.6:0/3415739431 >> v1:172.21.15.146:6789/0 conn(0x7f56000d0c80 legacy=0x7f56000d1900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:34.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.270+0000 7f55ffe00640 1 -- 172.21.15.6:0/3415739431 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56000b39f0 con 0x7f560015e0b0 2024-03-04T09:26:34.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.270+0000 7f55ff400640 1 --2- 172.21.15.6:0/3415739431 >> v2:172.21.15.146:3300/0 conn(0x7f56000d4a10 0x7f56000d0740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T09:26:34.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.271+0000 7f55ffe00640 1 --2- 172.21.15.6:0/3415739431 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f560015e0b0 0x7f56000cf310 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f56000b68c0 tx=0x7f55f002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a361f6945b86d89 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:34.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.272+0000 7f55fea00640 1 -- 172.21.15.6:0/3415739431 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55f0035020 con 0x7f560015e0b0 2024-03-04T09:26:34.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.272+0000 7f55fea00640 1 -- 172.21.15.6:0/3415739431 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55f0031830 con 0x7f560015e0b0 2024-03-04T09:26:34.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.272+0000 7f55fea00640 1 -- 172.21.15.6:0/3415739431 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55f0031c00 con 0x7f560015e0b0 2024-03-04T09:26:34.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.272+0000 7f5605800640 1 -- 172.21.15.6:0/3415739431 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f560015e0b0 msgr2=0x7f56000cf310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:34.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.272+0000 7f5605800640 1 --2- 172.21.15.6:0/3415739431 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f560015e0b0 0x7f56000cf310 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f56000b68c0 tx=0x7f55f002b8d0 comp rx=0 tx=0).stop 2024-03-04T09:26:34.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.272+0000 7f5605800640 1 -- 172.21.15.6:0/3415739431 shutdown_connections 2024-03-04T09:26:34.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.273+0000 7f5605800640 1 --2- 172.21.15.6:0/3415739431 >> v2:172.21.15.146:3300/0 conn(0x7f56000d4a10 0x7f56000d0740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:34.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.273+0000 7f5605800640 1 --2- 172.21.15.6:0/3415739431 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f560015e0b0 0x7f56000cf310 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:34.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.273+0000 7f5605800640 1 -- 172.21.15.6:0/3415739431 >> 172.21.15.6:0/3415739431 conn(0x7f56000d3d30 msgr2=0x7f56000d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:34.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.273+0000 7f5605800640 1 -- 172.21.15.6:0/3415739431 shutdown_connections 2024-03-04T09:26:34.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.273+0000 7f5605800640 1 -- 172.21.15.6:0/3415739431 wait complete. 2024-03-04T09:26:34.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.274+0000 7f5605800640 1 Processor -- start 2024-03-04T09:26:34.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.275+0000 7f5605800640 1 -- start start 2024-03-04T09:26:34.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.275+0000 7f5605800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f56000d0c80 0x7f56000b59d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:34.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.275+0000 7f5605800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f56000d4a10 0x7f56000b6950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:34.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.275+0000 7f5605800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f560015e0b0 0x7f56000cc450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:34.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.275+0000 7f5605800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f560015e490 con 0x7f560015e0b0 2024-03-04T09:26:34.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.275+0000 7f5605800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56000b3fb0 con 0x7f56000d4a10 2024-03-04T09:26:34.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.275+0000 7f5605800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56000cfc60 con 0x7f56000d0c80 2024-03-04T09:26:34.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.276+0000 7f5604e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f560015e0b0 0x7f56000cc450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:34.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.276+0000 7f5604e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f560015e0b0 0x7f56000cc450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:40724/0 (socket says 172.21.15.6:40724) 2024-03-04T09:26:34.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.276+0000 7f5604e00640 1 -- 172.21.15.6:0/4211118023 learned_addr learned my addr 172.21.15.6:0/4211118023 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:34.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.276+0000 7f55ff400640 1 --2- 172.21.15.6:0/4211118023 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f56000d4a10 0x7f56000b6950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:34.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.276+0000 7f55ffe00640 1 --2- 172.21.15.6:0/4211118023 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f56000d0c80 0x7f56000b59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:34.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.276+0000 7f5604e00640 1 -- 172.21.15.6:0/4211118023 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f56000d0c80 msgr2=0x7f56000b59d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:34.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.276+0000 7f5604e00640 1 --2- 172.21.15.6:0/4211118023 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f56000d0c80 0x7f56000b59d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:34.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.276+0000 7f5604e00640 1 -- 172.21.15.6:0/4211118023 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f56000d4a10 msgr2=0x7f56000b6950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:34.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.276+0000 7f5604e00640 1 --2- 172.21.15.6:0/4211118023 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f56000d4a10 0x7f56000b6950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:34.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.276+0000 7f5604e00640 1 -- 172.21.15.6:0/4211118023 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56000b7410 con 0x7f560015e0b0 2024-03-04T09:26:34.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.276+0000 7f55ffe00640 1 --2- 172.21.15.6:0/4211118023 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f56000d0c80 0x7f56000b59d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:26:34.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.277+0000 7f5604e00640 1 --2- 172.21.15.6:0/4211118023 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f560015e0b0 0x7f56000cc450 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f55f800c7d0 tx=0x7f55f800cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:34.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.277+0000 7f55f7e00640 1 -- 172.21.15.6:0/4211118023 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55f8016020 con 0x7f560015e0b0 2024-03-04T09:26:34.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.277+0000 7f55f7e00640 1 -- 172.21.15.6:0/4211118023 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55f800aa50 con 0x7f560015e0b0 2024-03-04T09:26:34.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.277+0000 7f55f7e00640 1 -- 172.21.15.6:0/4211118023 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55f800ae40 con 0x7f560015e0b0 2024-03-04T09:26:34.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.277+0000 7f5605800640 1 -- 172.21.15.6:0/4211118023 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56000cb010 con 0x7f560015e0b0 2024-03-04T09:26:34.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.277+0000 7f5605800640 1 -- 172.21.15.6:0/4211118023 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56000cb5d0 con 0x7f560015e0b0 2024-03-04T09:26:34.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.279+0000 7f5605800640 1 -- 172.21.15.6:0/4211118023 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56000cb7d0 con 0x7f560015e0b0 2024-03-04T09:26:34.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.279+0000 7f55f7e00640 1 -- 172.21.15.6:0/4211118023 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f55f8019020 con 0x7f560015e0b0 2024-03-04T09:26:34.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.280+0000 7f55f7e00640 1 --2- 172.21.15.6:0/4211118023 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f55e006b500 0x7f55e006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:34.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.280+0000 7f55f7e00640 1 -- 172.21.15.6:0/4211118023 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f55f8089fc0 con 0x7f560015e0b0 2024-03-04T09:26:34.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.281+0000 7f55ffe00640 1 --2- 172.21.15.6:0/4211118023 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f55e006b500 0x7f55e006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:34.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.282+0000 7f55ffe00640 1 --2- 172.21.15.6:0/4211118023 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f55e006b500 0x7f55e006d9c0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f55f0036000 tx=0x7f55f003c040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:34.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.285+0000 7f55f7e00640 1 -- 172.21.15.6:0/4211118023 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f56000cb7d0 con 0x7f560015e0b0 2024-03-04T09:26:34.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.525+0000 7f5605800640 1 -- 172.21.15.6:0/4211118023 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f560004f330 con 0x7f560015e0b0 2024-03-04T09:26:34.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.526+0000 7f55f7e00640 1 -- 172.21.15.6:0/4211118023 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+4977 (secure 0 0 0) 0x7f55f8020070 con 0x7f560015e0b0 2024-03-04T09:26:34.523 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:26:34.523 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4685,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/1703428647","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":1703428647},{"type":"v1","addr":"172.21.15.146:6837","nonce":1703428647}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4837,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/2617429530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2617429530},{"type":"v1","addr":"172.21.15.6:6833","nonce":2617429530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":5043,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/2744913857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":2744913857},{"type":"v1","addr":"172.21.15.6:6835","nonce":2744913857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:26:30.445199+0000","modified":"2024-03-04T09:26:33.534893+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5046},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5046":{"gid":5046,"name":"d","rank":0,"incarnation":12,"state":"up:active","state_seq":4,"addr":"172.21.15.146:6836/2285803221","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":2285803221},{"type":"v1","addr":"172.21.15.146:6836","nonce":2285803221}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5046,"qdb_cluster":[5046]},"id":2}]} 2024-03-04T09:26:34.524 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 16 2024-03-04T09:26:34.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.528+0000 7f5605800640 1 -- 172.21.15.6:0/4211118023 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f55e006b500 msgr2=0x7f55e006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:34.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.528+0000 7f5605800640 1 --2- 172.21.15.6:0/4211118023 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f55e006b500 0x7f55e006d9c0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f55f0036000 tx=0x7f55f003c040 comp rx=0 tx=0).stop 2024-03-04T09:26:34.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.529+0000 7f5605800640 1 -- 172.21.15.6:0/4211118023 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f560015e0b0 msgr2=0x7f56000cc450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:34.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.529+0000 7f5605800640 1 --2- 172.21.15.6:0/4211118023 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f560015e0b0 0x7f56000cc450 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f55f800c7d0 tx=0x7f55f800cc90 comp rx=0 tx=0).stop 2024-03-04T09:26:34.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.529+0000 7f5605800640 1 -- 172.21.15.6:0/4211118023 shutdown_connections 2024-03-04T09:26:34.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.529+0000 7f5605800640 1 --2- 172.21.15.6:0/4211118023 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f560015e0b0 0x7f56000cc450 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:34.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.529+0000 7f5605800640 1 --2- 172.21.15.6:0/4211118023 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f56000d4a10 0x7f56000b6950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:34.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.529+0000 7f5605800640 1 --2- 172.21.15.6:0/4211118023 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f55e006b500 0x7f55e006d9c0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:34.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.529+0000 7f5605800640 1 --2- 172.21.15.6:0/4211118023 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f56000d0c80 0x7f56000b59d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:34.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.529+0000 7f5605800640 1 -- 172.21.15.6:0/4211118023 >> 172.21.15.6:0/4211118023 conn(0x7f56000d3d30 msgr2=0x7f560006b9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:34.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.529+0000 7f5605800640 1 -- 172.21.15.6:0/4211118023 shutdown_connections 2024-03-04T09:26:34.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.529+0000 7f5605800640 1 -- 172.21.15.6:0/4211118023 wait complete. 2024-03-04T09:26:34.546 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T09:26:34.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.767+0000 7f2163e00640 1 Processor -- start 2024-03-04T09:26:34.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.767+0000 7f2163e00640 1 -- start start 2024-03-04T09:26:34.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.769+0000 7f2163e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f2164145300 0x7f21640b49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:34.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.769+0000 7f2163e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f2164070f10 0x7f21640cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:34.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.769+0000 7f2163e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21640714d0 con 0x7f21640cce90 2024-03-04T09:26:34.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.769+0000 7f2163e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21640cd300 con 0x7f2164070f10 2024-03-04T09:26:34.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.769+0000 7f2163e00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21640717b0 con 0x7f2164145300 2024-03-04T09:26:34.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.769+0000 7f2162000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f2164070f10 0x7f21640cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:34.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.769+0000 7f2162000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f2164070f10 0x7f21640cc950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:40732/0 (socket says 172.21.15.6:40732) 2024-03-04T09:26:34.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.769+0000 7f2162000640 1 -- 172.21.15.6:0/1769715611 learned_addr learned my addr 172.21.15.6:0/1769715611 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:34.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.769+0000 7f2162a00640 1 --2- 172.21.15.6:0/1769715611 >> v2:172.21.15.146:3300/0 conn(0x7f2164145300 0x7f21640b49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:34.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.770+0000 7f2162000640 1 -- 172.21.15.6:0/1769715611 >> v2:172.21.15.146:3300/0 conn(0x7f2164145300 msgr2=0x7f21640b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:34.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.770+0000 7f2162000640 1 --2- 172.21.15.6:0/1769715611 >> v2:172.21.15.146:3300/0 conn(0x7f2164145300 0x7f21640b49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:34.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.770+0000 7f2162000640 1 -- 172.21.15.6:0/1769715611 >> v1:172.21.15.6:6789/0 conn(0x7f21640cce90 legacy=0x7f21640cd930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:34.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.770+0000 7f2162000640 1 -- 172.21.15.6:0/1769715611 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2164071a90 con 0x7f2164070f10 2024-03-04T09:26:34.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.771+0000 7f2162000640 1 --2- 172.21.15.6:0/1769715611 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2164070f10 0x7f21640cc950 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f21640cf990 tx=0x7f215002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f44f099d10cbc3e server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:34.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.771+0000 7f2161600640 1 -- 172.21.15.6:0/1769715611 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f215002ca80 con 0x7f2164070f10 2024-03-04T09:26:34.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.771+0000 7f2161600640 1 -- 172.21.15.6:0/1769715611 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f215002cc20 con 0x7f2164070f10 2024-03-04T09:26:34.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.771+0000 7f2161600640 1 -- 172.21.15.6:0/1769715611 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21500343f0 con 0x7f2164070f10 2024-03-04T09:26:34.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.772+0000 7f2163e00640 1 -- 172.21.15.6:0/1769715611 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2164070f10 msgr2=0x7f21640cc950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:34.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.772+0000 7f2163e00640 1 --2- 172.21.15.6:0/1769715611 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2164070f10 0x7f21640cc950 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f21640cf990 tx=0x7f215002bee0 comp rx=0 tx=0).stop 2024-03-04T09:26:34.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.772+0000 7f2163e00640 1 -- 172.21.15.6:0/1769715611 shutdown_connections 2024-03-04T09:26:34.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.772+0000 7f2163e00640 1 --2- 172.21.15.6:0/1769715611 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2164070f10 0x7f21640cc950 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:34.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.772+0000 7f2163e00640 1 --2- 172.21.15.6:0/1769715611 >> v2:172.21.15.146:3300/0 conn(0x7f2164145300 0x7f21640b49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:34.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.772+0000 7f2163e00640 1 -- 172.21.15.6:0/1769715611 >> 172.21.15.6:0/1769715611 conn(0x7f216406b3d0 msgr2=0x7f216406b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:34.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.772+0000 7f2163e00640 1 -- 172.21.15.6:0/1769715611 shutdown_connections 2024-03-04T09:26:34.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.772+0000 7f2163e00640 1 -- 172.21.15.6:0/1769715611 wait complete. 2024-03-04T09:26:34.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.774+0000 7f2163e00640 1 Processor -- start 2024-03-04T09:26:34.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.774+0000 7f2163e00640 1 -- start start 2024-03-04T09:26:34.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.775+0000 7f2163e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2164070f10 0x7f216413df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:34.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.775+0000 7f2163e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f21640cce90 0x7f216413f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:34.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.775+0000 7f2163e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2164145300 0x7f21641432c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:34.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.775+0000 7f2163e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21641456e0 con 0x7f21640cce90 2024-03-04T09:26:34.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.775+0000 7f2163e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21640b4ee0 con 0x7f2164145300 2024-03-04T09:26:34.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.775+0000 7f2163e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f216415d580 con 0x7f2164070f10 2024-03-04T09:26:34.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.775+0000 7f2162000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f21640cce90 0x7f216413f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:34.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.775+0000 7f2162000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f21640cce90 0x7f216413f5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:40734/0 (socket says 172.21.15.6:40734) 2024-03-04T09:26:34.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.775+0000 7f2162000640 1 -- 172.21.15.6:0/1953357909 learned_addr learned my addr 172.21.15.6:0/1953357909 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:34.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.775+0000 7f2163400640 1 --2- 172.21.15.6:0/1953357909 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2164145300 0x7f21641432c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:34.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.776+0000 7f2162a00640 1 --2- 172.21.15.6:0/1953357909 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2164070f10 0x7f216413df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:34.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.776+0000 7f2162000640 1 -- 172.21.15.6:0/1953357909 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2164070f10 msgr2=0x7f216413df60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:34.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.776+0000 7f2162000640 1 --2- 172.21.15.6:0/1953357909 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2164070f10 0x7f216413df60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:34.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.776+0000 7f2162000640 1 -- 172.21.15.6:0/1953357909 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2164145300 msgr2=0x7f21641432c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:34.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.776+0000 7f2162000640 1 --2- 172.21.15.6:0/1953357909 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2164145300 0x7f21641432c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:34.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.776+0000 7f2162000640 1 -- 172.21.15.6:0/1953357909 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2164143800 con 0x7f21640cce90 2024-03-04T09:26:34.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.776+0000 7f2162000640 1 --2- 172.21.15.6:0/1953357909 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f21640cce90 0x7f216413f5e0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f2150036040 tx=0x7f2150004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:34.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.776+0000 7f215b400640 1 -- 172.21.15.6:0/1953357909 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2150043020 con 0x7f21640cce90 2024-03-04T09:26:34.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.776+0000 7f215b400640 1 -- 172.21.15.6:0/1953357909 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f215003e450 con 0x7f21640cce90 2024-03-04T09:26:34.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.776+0000 7f215b400640 1 -- 172.21.15.6:0/1953357909 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2150009510 con 0x7f21640cce90 2024-03-04T09:26:34.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.776+0000 7f2163e00640 1 -- 172.21.15.6:0/1953357909 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f216413e540 con 0x7f21640cce90 2024-03-04T09:26:34.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.777+0000 7f2163e00640 1 -- 172.21.15.6:0/1953357909 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f216413eb00 con 0x7f21640cce90 2024-03-04T09:26:34.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.778+0000 7f2163e00640 1 -- 172.21.15.6:0/1953357909 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f216404f330 con 0x7f21640cce90 2024-03-04T09:26:34.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.779+0000 7f215b400640 1 -- 172.21.15.6:0/1953357909 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2150046020 con 0x7f21640cce90 2024-03-04T09:26:34.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.779+0000 7f215b400640 1 --2- 172.21.15.6:0/1953357909 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f214406b4d0 0x7f214406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:34.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.780+0000 7f215b400640 1 -- 172.21.15.6:0/1953357909 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f21500ac9a0 con 0x7f21640cce90 2024-03-04T09:26:34.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.780+0000 7f2162a00640 1 --2- 172.21.15.6:0/1953357909 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f214406b4d0 0x7f214406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:34.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.781+0000 7f2162a00640 1 --2- 172.21.15.6:0/1953357909 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f214406b4d0 0x7f214406d990 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f2154006fd0 tx=0x7f2154008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:34.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.785+0000 7f215b400640 1 -- 172.21.15.6:0/1953357909 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f215004d030 con 0x7f21640cce90 2024-03-04T09:26:34.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.988+0000 7f2163e00640 1 -- 172.21.15.6:0/1953357909 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21640b30f0 con 0x7f21640cce90 2024-03-04T09:26:34.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.989+0000 7f215b400640 1 -- 172.21.15.6:0/1953357909 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v33) v1 ==== 74+0+16621 (secure 0 0 0) 0x7f215007b8b0 con 0x7f21640cce90 2024-03-04T09:26:34.987 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:26:34.987 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":33,"fsid":"edce0578-5ae4-4402-b45a-552bdef84d7c","created":"2024-03-04T09:24:54.523780+0000","modified":"2024-03-04T09:26:31.991047+0000","last_up_change":"2024-03-04T09:25:07.089049+0000","last_in_change":"2024-03-04T09:24:57.982077+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T09:25:07.935198+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-04T09:26:28.556219+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-04T09:26:29.471227+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a96574f-81f1-4bc6-8463-757520b84a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6825","nonce":1364449401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6827","nonce":1364449401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6831","nonce":1364449401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6829","nonce":1364449401}]},"public_addr":"172.21.15.6:6825/1364449401","cluster_addr":"172.21.15.6:6827/1364449401","heartbeat_back_addr":"172.21.15.6:6831/1364449401","heartbeat_front_addr":"172.21.15.6:6829/1364449401","state":["exists","up"]},{"osd":1,"uuid":"7d79b9c1-6b87-476a-bc36-7db5ee64cd90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6809","nonce":2494103078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6811","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6813","nonce":2494103078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6821","nonce":2494103078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6817","nonce":2494103078}]},"public_addr":"172.21.15.6:6809/2494103078","cluster_addr":"172.21.15.6:6813/2494103078","heartbeat_back_addr":"172.21.15.6:6821/2494103078","heartbeat_front_addr":"172.21.15.6:6817/2494103078","state":["exists","up"]},{"osd":2,"uuid":"492331b9-4ee9-40b6-bacd-85d44cd6c45a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6812","nonce":1500066681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6816","nonce":1500066681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6824","nonce":1500066681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6820","nonce":1500066681}]},"public_addr":"172.21.15.6:6812/1500066681","cluster_addr":"172.21.15.6:6816/1500066681","heartbeat_back_addr":"172.21.15.6:6824/1500066681","heartbeat_front_addr":"172.21.15.6:6820/1500066681","state":["exists","up"]},{"osd":3,"uuid":"21bb2efe-9ba7-4bdc-bc31-180cd1102e21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6801","nonce":1558806123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6803","nonce":1558806123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6807","nonce":1558806123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6805","nonce":1558806123}]},"public_addr":"172.21.15.6:6801/1558806123","cluster_addr":"172.21.15.6:6803/1558806123","heartbeat_back_addr":"172.21.15.6:6807/1558806123","heartbeat_front_addr":"172.21.15.6:6805/1558806123","state":["exists","up"]},{"osd":4,"uuid":"6e2c4b71-19d9-4737-83d9-558705d8644a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6801","nonce":1526135780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6803","nonce":1526135780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6807","nonce":1526135780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6805","nonce":1526135780}]},"public_addr":"172.21.15.146:6801/1526135780","cluster_addr":"172.21.15.146:6803/1526135780","heartbeat_back_addr":"172.21.15.146:6807/1526135780","heartbeat_front_addr":"172.21.15.146:6805/1526135780","state":["exists","up"]},{"osd":5,"uuid":"fb69c525-f1be-4b7d-ab32-6c29b0d6fddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6809","nonce":460262507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6811","nonce":460262507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6815","nonce":460262507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6813","nonce":460262507}]},"public_addr":"172.21.15.146:6809/460262507","cluster_addr":"172.21.15.146:6811/460262507","heartbeat_back_addr":"172.21.15.146:6815/460262507","heartbeat_front_addr":"172.21.15.146:6813/460262507","state":["exists","up"]},{"osd":6,"uuid":"c32e7fed-370d-4401-8ce0-6a4335b0d168","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6823","nonce":1392897493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6827","nonce":1392897493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6831","nonce":1392897493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6829","nonce":1392897493}]},"public_addr":"172.21.15.146:6823/1392897493","cluster_addr":"172.21.15.146:6827/1392897493","heartbeat_back_addr":"172.21.15.146:6831/1392897493","heartbeat_front_addr":"172.21.15.146:6829/1392897493","state":["exists","up"]},{"osd":7,"uuid":"75a79b44-fa5f-426d-a7d3-71f21f78835c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6817","nonce":3268323156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6819","nonce":3268323156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6825","nonce":3268323156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6822","nonce":3268323156}]},"public_addr":"172.21.15.146:6817/3268323156","cluster_addr":"172.21.15.146:6819/3268323156","heartbeat_back_addr":"172.21.15.146:6825/3268323156","heartbeat_front_addr":"172.21.15.146:6822/3268323156","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.310437+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.295850+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.294674+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.526855+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.300023+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.595160+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.346046+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.558741+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T09:26:34.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.992+0000 7f2163e00640 1 -- 172.21.15.6:0/1953357909 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f214406b4d0 msgr2=0x7f214406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:34.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.992+0000 7f2163e00640 1 --2- 172.21.15.6:0/1953357909 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f214406b4d0 0x7f214406d990 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f2154006fd0 tx=0x7f2154008040 comp rx=0 tx=0).stop 2024-03-04T09:26:34.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.992+0000 7f2163e00640 1 -- 172.21.15.6:0/1953357909 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f21640cce90 msgr2=0x7f216413f5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:34.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.992+0000 7f2163e00640 1 --2- 172.21.15.6:0/1953357909 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f21640cce90 0x7f216413f5e0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f2150036040 tx=0x7f2150004a40 comp rx=0 tx=0).stop 2024-03-04T09:26:34.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.992+0000 7f2163e00640 1 -- 172.21.15.6:0/1953357909 shutdown_connections 2024-03-04T09:26:34.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.992+0000 7f2163e00640 1 --2- 172.21.15.6:0/1953357909 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2164145300 0x7f21641432c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:34.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.992+0000 7f2163e00640 1 --2- 172.21.15.6:0/1953357909 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f21640cce90 0x7f216413f5e0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:34.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.992+0000 7f2163e00640 1 --2- 172.21.15.6:0/1953357909 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f214406b4d0 0x7f214406d990 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:34.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.992+0000 7f2163e00640 1 --2- 172.21.15.6:0/1953357909 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2164070f10 0x7f216413df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:34.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.992+0000 7f2163e00640 1 -- 172.21.15.6:0/1953357909 >> 172.21.15.6:0/1953357909 conn(0x7f216406b3d0 msgr2=0x7f2164072680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:34.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.992+0000 7f2163e00640 1 -- 172.21.15.6:0/1953357909 shutdown_connections 2024-03-04T09:26:34.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:34.992+0000 7f2163e00640 1 -- 172.21.15.6:0/1953357909 wait complete. 2024-03-04T09:26:35.008 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-04T09:26:35.008 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T09:26:35.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.230+0000 7fc26be00640 1 Processor -- start 2024-03-04T09:26:35.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.230+0000 7fc26be00640 1 -- start start 2024-03-04T09:26:35.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.231+0000 7fc26be00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc26c145300 0x7fc26c0b49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:35.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.232+0000 7fc26be00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc26c0cd060 0x7fc26c15c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:35.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.232+0000 7fc26be00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc26c0714d0 con 0x7fc26c070f10 2024-03-04T09:26:35.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.232+0000 7fc26be00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc26c0717b0 con 0x7fc26c145300 2024-03-04T09:26:35.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.232+0000 7fc26be00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc26c071a90 con 0x7fc26c0cd060 2024-03-04T09:26:35.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.232+0000 7fc26aa00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc26c145300 0x7fc26c0b49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:35.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.232+0000 7fc26aa00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc26c145300 0x7fc26c0b49a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:40746/0 (socket says 172.21.15.6:40746) 2024-03-04T09:26:35.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.232+0000 7fc26aa00640 1 -- 172.21.15.6:0/3750775102 learned_addr learned my addr 172.21.15.6:0/3750775102 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:35.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.232+0000 7fc26b400640 1 --2- 172.21.15.6:0/3750775102 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc26c0cd060 0x7fc26c15c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:35.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.233+0000 7fc26aa00640 1 -- 172.21.15.6:0/3750775102 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc26c0cd060 msgr2=0x7fc26c15c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:35.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.233+0000 7fc26aa00640 1 --2- 172.21.15.6:0/3750775102 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc26c0cd060 0x7fc26c15c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:35.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.233+0000 7fc26aa00640 1 -- 172.21.15.6:0/3750775102 >> v1:172.21.15.146:6789/0 conn(0x7fc26c070f10 legacy=0x7fc26c0cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:35.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.233+0000 7fc26aa00640 1 -- 172.21.15.6:0/3750775102 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc26c071d70 con 0x7fc26c145300 2024-03-04T09:26:35.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.234+0000 7fc26aa00640 1 --2- 172.21.15.6:0/3750775102 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc26c145300 0x7fc26c0b49a0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fc25c009870 tx=0x7fc25c02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c37e4f35a50e9efb server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:35.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.234+0000 7fc269600640 1 -- 172.21.15.6:0/3750775102 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc25c0317c0 con 0x7fc26c145300 2024-03-04T09:26:35.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.234+0000 7fc269600640 1 -- 172.21.15.6:0/3750775102 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc25c037070 con 0x7fc26c145300 2024-03-04T09:26:35.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.234+0000 7fc269600640 1 -- 172.21.15.6:0/3750775102 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc25c031b80 con 0x7fc26c145300 2024-03-04T09:26:35.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.234+0000 7fc26be00640 1 -- 172.21.15.6:0/3750775102 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc26c145300 msgr2=0x7fc26c0b49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:35.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.234+0000 7fc26be00640 1 --2- 172.21.15.6:0/3750775102 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc26c145300 0x7fc26c0b49a0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fc25c009870 tx=0x7fc25c02b860 comp rx=0 tx=0).stop 2024-03-04T09:26:35.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.235+0000 7fc26be00640 1 -- 172.21.15.6:0/3750775102 shutdown_connections 2024-03-04T09:26:35.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.235+0000 7fc26be00640 1 --2- 172.21.15.6:0/3750775102 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc26c0cd060 0x7fc26c15c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:35.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.235+0000 7fc26be00640 1 --2- 172.21.15.6:0/3750775102 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc26c145300 0x7fc26c0b49a0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:35.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.235+0000 7fc26be00640 1 -- 172.21.15.6:0/3750775102 >> 172.21.15.6:0/3750775102 conn(0x7fc26c06b3d0 msgr2=0x7fc26c06b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:35.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.235+0000 7fc26be00640 1 -- 172.21.15.6:0/3750775102 shutdown_connections 2024-03-04T09:26:35.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.235+0000 7fc26be00640 1 -- 172.21.15.6:0/3750775102 wait complete. 2024-03-04T09:26:35.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.236+0000 7fc26be00640 1 Processor -- start 2024-03-04T09:26:35.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.237+0000 7fc26be00640 1 -- start start 2024-03-04T09:26:35.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.238+0000 7fc26be00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc26c070f10 0x7fc26c1e5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:35.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.238+0000 7fc26be00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc26c0cd060 0x7fc26c15b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:35.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.238+0000 7fc26be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc26c145300 0x7fc26c1f44f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:35.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.238+0000 7fc26be00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc26c15cab0 con 0x7fc26c145300 2024-03-04T09:26:35.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.238+0000 7fc26be00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc26c1456e0 con 0x7fc26c070f10 2024-03-04T09:26:35.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.238+0000 7fc26be00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc26c15cc30 con 0x7fc26c0cd060 2024-03-04T09:26:35.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.238+0000 7fc26b400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc26c145300 0x7fc26c1f44f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:35.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.238+0000 7fc26b400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc26c145300 0x7fc26c1f44f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:40750/0 (socket says 172.21.15.6:40750) 2024-03-04T09:26:35.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.238+0000 7fc26b400640 1 -- 172.21.15.6:0/3699279879 learned_addr learned my addr 172.21.15.6:0/3699279879 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:35.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.238+0000 7fc26aa00640 1 --2- 172.21.15.6:0/3699279879 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc26c070f10 0x7fc26c1e5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:35.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.238+0000 7fc26a000640 1 --2- 172.21.15.6:0/3699279879 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc26c0cd060 0x7fc26c15b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:35.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.239+0000 7fc26b400640 1 -- 172.21.15.6:0/3699279879 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc26c0cd060 msgr2=0x7fc26c15b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:35.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.239+0000 7fc26b400640 1 --2- 172.21.15.6:0/3699279879 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc26c0cd060 0x7fc26c15b430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:35.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.239+0000 7fc26b400640 1 -- 172.21.15.6:0/3699279879 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc26c070f10 msgr2=0x7fc26c1e5470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:35.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.239+0000 7fc26b400640 1 --2- 172.21.15.6:0/3699279879 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc26c070f10 0x7fc26c1e5470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:35.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.239+0000 7fc26b400640 1 -- 172.21.15.6:0/3699279879 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc26c1f5040 con 0x7fc26c145300 2024-03-04T09:26:35.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.239+0000 7fc26b400640 1 --2- 172.21.15.6:0/3699279879 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc26c145300 0x7fc26c1f44f0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fc264007e70 tx=0x7fc26400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:35.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.239+0000 7fc263400640 1 -- 172.21.15.6:0/3699279879 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc264014620 con 0x7fc26c145300 2024-03-04T09:26:35.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.239+0000 7fc263400640 1 -- 172.21.15.6:0/3699279879 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc264019070 con 0x7fc26c145300 2024-03-04T09:26:35.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.239+0000 7fc26be00640 1 -- 172.21.15.6:0/3699279879 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc26c1f5320 con 0x7fc26c145300 2024-03-04T09:26:35.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.239+0000 7fc263400640 1 -- 172.21.15.6:0/3699279879 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc264014a20 con 0x7fc26c145300 2024-03-04T09:26:35.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.240+0000 7fc26be00640 1 -- 172.21.15.6:0/3699279879 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc26c1f58e0 con 0x7fc26c145300 2024-03-04T09:26:35.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.241+0000 7fc26be00640 1 -- 172.21.15.6:0/3699279879 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc26c04f330 con 0x7fc26c145300 2024-03-04T09:26:35.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.242+0000 7fc263400640 1 -- 172.21.15.6:0/3699279879 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc264014bc0 con 0x7fc26c145300 2024-03-04T09:26:35.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.243+0000 7fc263400640 1 --2- 172.21.15.6:0/3699279879 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc24c06b4d0 0x7fc24c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:35.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.243+0000 7fc263400640 1 -- 172.21.15.6:0/3699279879 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc26408ab00 con 0x7fc26c145300 2024-03-04T09:26:35.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.244+0000 7fc26aa00640 1 --2- 172.21.15.6:0/3699279879 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc24c06b4d0 0x7fc24c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:35.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.245+0000 7fc26aa00640 1 --2- 172.21.15.6:0/3699279879 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc24c06b4d0 0x7fc24c06d990 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fc25c036000 tx=0x7fc25c03d040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:35.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.248+0000 7fc263400640 1 -- 172.21.15.6:0/3699279879 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc264059980 con 0x7fc26c145300 2024-03-04T09:26:35.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.446+0000 7fc26be00640 1 -- 172.21.15.6:0/3699279879 --> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc26c0b4890 con 0x7fc24c06b4d0 2024-03-04T09:26:35.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.468+0000 7fc263400640 1 -- 172.21.15.6:0/3699279879 <== mgr.4113 v2:172.21.15.146:6832/1788613334 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251904 (secure 0 0 0) 0x7fc26c0b4890 con 0x7fc24c06b4d0 2024-03-04T09:26:35.469 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:26:35.470 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-03-04T09:26:35.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.474+0000 7fc26be00640 1 -- 172.21.15.6:0/3699279879 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc24c06b4d0 msgr2=0x7fc24c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:35.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.474+0000 7fc26be00640 1 --2- 172.21.15.6:0/3699279879 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc24c06b4d0 0x7fc24c06d990 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fc25c036000 tx=0x7fc25c03d040 comp rx=0 tx=0).stop 2024-03-04T09:26:35.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.474+0000 7fc26be00640 1 -- 172.21.15.6:0/3699279879 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc26c145300 msgr2=0x7fc26c1f44f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:35.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.474+0000 7fc26be00640 1 --2- 172.21.15.6:0/3699279879 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc26c145300 0x7fc26c1f44f0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fc264007e70 tx=0x7fc26400e4b0 comp rx=0 tx=0).stop 2024-03-04T09:26:35.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.474+0000 7fc26be00640 1 -- 172.21.15.6:0/3699279879 shutdown_connections 2024-03-04T09:26:35.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.474+0000 7fc26be00640 1 --2- 172.21.15.6:0/3699279879 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc26c145300 0x7fc26c1f44f0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:35.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.474+0000 7fc26be00640 1 --2- 172.21.15.6:0/3699279879 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc24c06b4d0 0x7fc24c06d990 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:35.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.474+0000 7fc26be00640 1 --2- 172.21.15.6:0/3699279879 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc26c0cd060 0x7fc26c15b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:35.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.474+0000 7fc26be00640 1 --2- 172.21.15.6:0/3699279879 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc26c070f10 0x7fc26c1e5470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:35.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.474+0000 7fc26be00640 1 -- 172.21.15.6:0/3699279879 >> 172.21.15.6:0/3699279879 conn(0x7fc26c06b3d0 msgr2=0x7fc26c0ae540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:35.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.474+0000 7fc26be00640 1 -- 172.21.15.6:0/3699279879 shutdown_connections 2024-03-04T09:26:35.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.474+0000 7fc26be00640 1 -- 172.21.15.6:0/3699279879 wait complete. 2024-03-04T09:26:35.491 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":69,"stamp":"2024-03-04T09:26:33.884956+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":116,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220268,"kb_used_data":4064,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754452,"statfs":{"total":773094113280,"available":772868558848,"internally_reserved":0,"allocated":4161536,"data_stored":2322604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.929079"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:31.007535+0000","last_change":"2024-03-04T09:26:31.005107+0000","last_active":"2024-03-04T09:26:31.007535+0000","last_peered":"2024-03-04T09:26:31.007535+0000","last_clean":"2024-03-04T09:26:31.007535+0000","last_became_active":"2024-03-04T09:26:31.004571+0000","last_became_peered":"2024-03-04T09:26:31.004571+0000","last_unstale":"2024-03-04T09:26:31.007535+0000","last_undegraded":"2024-03-04T09:26:31.007535+0000","last_fullsized":"2024-03-04T09:26:31.007535+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:29:29.188429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:31.020432+0000","last_change":"2024-03-04T09:26:31.015770+0000","last_active":"2024-03-04T09:26:31.020432+0000","last_peered":"2024-03-04T09:26:31.020432+0000","last_clean":"2024-03-04T09:26:31.020432+0000","last_became_active":"2024-03-04T09:26:31.015013+0000","last_became_peered":"2024-03-04T09:26:31.015013+0000","last_unstale":"2024-03-04T09:26:31.020432+0000","last_undegraded":"2024-03-04T09:26:31.020432+0000","last_fullsized":"2024-03-04T09:26:31.020432+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:21:26.223848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:31.020606+0000","last_change":"2024-03-04T09:26:31.009260+0000","last_active":"2024-03-04T09:26:31.020606+0000","last_peered":"2024-03-04T09:26:31.020606+0000","last_clean":"2024-03-04T09:26:31.020606+0000","last_became_active":"2024-03-04T09:26:31.009007+0000","last_became_peered":"2024-03-04T09:26:31.009007+0000","last_unstale":"2024-03-04T09:26:31.020606+0000","last_undegraded":"2024-03-04T09:26:31.020606+0000","last_fullsized":"2024-03-04T09:26:31.020606+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:42:09.417392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-04T09:26:32.009864+0000","last_change":"2024-03-04T09:26:31.004397+0000","last_active":"2024-03-04T09:26:32.009864+0000","last_peered":"2024-03-04T09:26:32.009864+0000","last_clean":"2024-03-04T09:26:32.009864+0000","last_became_active":"2024-03-04T09:26:31.004112+0000","last_became_peered":"2024-03-04T09:26:31.004112+0000","last_unstale":"2024-03-04T09:26:32.009864+0000","last_undegraded":"2024-03-04T09:26:32.009864+0000","last_fullsized":"2024-03-04T09:26:32.009864+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:15:35.966281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-04T09:26:32.010900+0000","last_change":"2024-03-04T09:26:31.001936+0000","last_active":"2024-03-04T09:26:32.010900+0000","last_peered":"2024-03-04T09:26:32.010900+0000","last_clean":"2024-03-04T09:26:32.010900+0000","last_became_active":"2024-03-04T09:26:31.001613+0000","last_became_peered":"2024-03-04T09:26:31.001613+0000","last_unstale":"2024-03-04T09:26:32.010900+0000","last_undegraded":"2024-03-04T09:26:32.010900+0000","last_fullsized":"2024-03-04T09:26:32.010900+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:17:16.959389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:31.027883+0000","last_change":"2024-03-04T09:26:31.022957+0000","last_active":"2024-03-04T09:26:31.027883+0000","last_peered":"2024-03-04T09:26:31.027883+0000","last_clean":"2024-03-04T09:26:31.027883+0000","last_became_active":"2024-03-04T09:26:31.022640+0000","last_became_peered":"2024-03-04T09:26:31.022640+0000","last_unstale":"2024-03-04T09:26:31.027883+0000","last_undegraded":"2024-03-04T09:26:31.027883+0000","last_fullsized":"2024-03-04T09:26:31.027883+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:02:48.660304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:31.017997+0000","last_change":"2024-03-04T09:26:31.015526+0000","last_active":"2024-03-04T09:26:31.017997+0000","last_peered":"2024-03-04T09:26:31.017997+0000","last_clean":"2024-03-04T09:26:31.017997+0000","last_became_active":"2024-03-04T09:26:31.015310+0000","last_became_peered":"2024-03-04T09:26:31.015310+0000","last_unstale":"2024-03-04T09:26:31.017997+0000","last_undegraded":"2024-03-04T09:26:31.017997+0000","last_fullsized":"2024-03-04T09:26:31.017997+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:23:15.690106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:31.014352+0000","last_change":"2024-03-04T09:26:31.010572+0000","last_active":"2024-03-04T09:26:31.014352+0000","last_peered":"2024-03-04T09:26:31.014352+0000","last_clean":"2024-03-04T09:26:31.014352+0000","last_became_active":"2024-03-04T09:26:31.010314+0000","last_became_peered":"2024-03-04T09:26:31.010314+0000","last_unstale":"2024-03-04T09:26:31.014352+0000","last_undegraded":"2024-03-04T09:26:31.014352+0000","last_fullsized":"2024-03-04T09:26:31.014352+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:33:02.297144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:31.007901+0000","last_change":"2024-03-04T09:26:31.006526+0000","last_active":"2024-03-04T09:26:31.007901+0000","last_peered":"2024-03-04T09:26:31.007901+0000","last_clean":"2024-03-04T09:26:31.007901+0000","last_became_active":"2024-03-04T09:26:31.006225+0000","last_became_peered":"2024-03-04T09:26:31.006225+0000","last_unstale":"2024-03-04T09:26:31.007901+0000","last_undegraded":"2024-03-04T09:26:31.007901+0000","last_fullsized":"2024-03-04T09:26:31.007901+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:47:09.744504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-04T09:26:32.013327+0000","last_change":"2024-03-04T09:26:31.019152+0000","last_active":"2024-03-04T09:26:32.013327+0000","last_peered":"2024-03-04T09:26:32.013327+0000","last_clean":"2024-03-04T09:26:32.013327+0000","last_became_active":"2024-03-04T09:26:31.018841+0000","last_became_peered":"2024-03-04T09:26:31.018841+0000","last_unstale":"2024-03-04T09:26:32.013327+0000","last_undegraded":"2024-03-04T09:26:32.013327+0000","last_fullsized":"2024-03-04T09:26:32.013327+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:41:30.039171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:31.014620+0000","last_change":"2024-03-04T09:26:31.012528+0000","last_active":"2024-03-04T09:26:31.014620+0000","last_peered":"2024-03-04T09:26:31.014620+0000","last_clean":"2024-03-04T09:26:31.014620+0000","last_became_active":"2024-03-04T09:26:31.011978+0000","last_became_peered":"2024-03-04T09:26:31.011978+0000","last_unstale":"2024-03-04T09:26:31.014620+0000","last_undegraded":"2024-03-04T09:26:31.014620+0000","last_fullsized":"2024-03-04T09:26:31.014620+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:47:09.744504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:31.020234+0000","last_change":"2024-03-04T09:26:31.014860+0000","last_active":"2024-03-04T09:26:31.020234+0000","last_peered":"2024-03-04T09:26:31.020234+0000","last_clean":"2024-03-04T09:26:31.020234+0000","last_became_active":"2024-03-04T09:26:31.014554+0000","last_became_peered":"2024-03-04T09:26:31.014554+0000","last_unstale":"2024-03-04T09:26:31.020234+0000","last_undegraded":"2024-03-04T09:26:31.020234+0000","last_fullsized":"2024-03-04T09:26:31.020234+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:00:21.753592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-04T09:26:32.012576+0000","last_change":"2024-03-04T09:26:31.008167+0000","last_active":"2024-03-04T09:26:32.012576+0000","last_peered":"2024-03-04T09:26:32.012576+0000","last_clean":"2024-03-04T09:26:32.012576+0000","last_became_active":"2024-03-04T09:26:31.005783+0000","last_became_peered":"2024-03-04T09:26:31.005783+0000","last_unstale":"2024-03-04T09:26:32.012576+0000","last_undegraded":"2024-03-04T09:26:32.012576+0000","last_fullsized":"2024-03-04T09:26:32.012576+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:29:29.188429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:31.015415+0000","last_change":"2024-03-04T09:26:31.007649+0000","last_active":"2024-03-04T09:26:31.015415+0000","last_peered":"2024-03-04T09:26:31.015415+0000","last_clean":"2024-03-04T09:26:31.015415+0000","last_became_active":"2024-03-04T09:26:31.007367+0000","last_became_peered":"2024-03-04T09:26:31.007367+0000","last_unstale":"2024-03-04T09:26:31.015415+0000","last_undegraded":"2024-03-04T09:26:31.015415+0000","last_fullsized":"2024-03-04T09:26:31.015415+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:40:19.971433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:31.020921+0000","last_change":"2024-03-04T09:26:31.018291+0000","last_active":"2024-03-04T09:26:31.020921+0000","last_peered":"2024-03-04T09:26:31.020921+0000","last_clean":"2024-03-04T09:26:31.020921+0000","last_became_active":"2024-03-04T09:26:31.017998+0000","last_became_peered":"2024-03-04T09:26:31.017998+0000","last_unstale":"2024-03-04T09:26:31.020921+0000","last_undegraded":"2024-03-04T09:26:31.020921+0000","last_fullsized":"2024-03-04T09:26:31.020921+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:41:30.039171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:31.024281+0000","last_change":"2024-03-04T09:26:31.022400+0000","last_active":"2024-03-04T09:26:31.024281+0000","last_peered":"2024-03-04T09:26:31.024281+0000","last_clean":"2024-03-04T09:26:31.024281+0000","last_became_active":"2024-03-04T09:26:31.021984+0000","last_became_peered":"2024-03-04T09:26:31.021984+0000","last_unstale":"2024-03-04T09:26:31.024281+0000","last_undegraded":"2024-03-04T09:26:31.024281+0000","last_fullsized":"2024-03-04T09:26:31.024281+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:48:31.810546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:31.024431+0000","last_change":"2024-03-04T09:26:31.020714+0000","last_active":"2024-03-04T09:26:31.024431+0000","last_peered":"2024-03-04T09:26:31.024431+0000","last_clean":"2024-03-04T09:26:31.024431+0000","last_became_active":"2024-03-04T09:26:31.017378+0000","last_became_peered":"2024-03-04T09:26:31.017378+0000","last_unstale":"2024-03-04T09:26:31.024431+0000","last_undegraded":"2024-03-04T09:26:31.024431+0000","last_fullsized":"2024-03-04T09:26:31.024431+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:40:19.971433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:31.020038+0000","last_change":"2024-03-04T09:26:31.012132+0000","last_active":"2024-03-04T09:26:31.020038+0000","last_peered":"2024-03-04T09:26:31.020038+0000","last_clean":"2024-03-04T09:26:31.020038+0000","last_became_active":"2024-03-04T09:26:31.011749+0000","last_became_peered":"2024-03-04T09:26:31.011749+0000","last_unstale":"2024-03-04T09:26:31.020038+0000","last_undegraded":"2024-03-04T09:26:31.020038+0000","last_fullsized":"2024-03-04T09:26:31.020038+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:48:31.810546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:31.021506+0000","last_change":"2024-03-04T09:26:31.012281+0000","last_active":"2024-03-04T09:26:31.021506+0000","last_peered":"2024-03-04T09:26:31.021506+0000","last_clean":"2024-03-04T09:26:31.021506+0000","last_became_active":"2024-03-04T09:26:31.011970+0000","last_became_peered":"2024-03-04T09:26:31.011970+0000","last_unstale":"2024-03-04T09:26:31.021506+0000","last_undegraded":"2024-03-04T09:26:31.021506+0000","last_fullsized":"2024-03-04T09:26:31.021506+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:42:09.417392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-04T09:26:32.011051+0000","last_change":"2024-03-04T09:26:31.021324+0000","last_active":"2024-03-04T09:26:32.011051+0000","last_peered":"2024-03-04T09:26:32.011051+0000","last_clean":"2024-03-04T09:26:32.011051+0000","last_became_active":"2024-03-04T09:26:31.020950+0000","last_became_peered":"2024-03-04T09:26:31.020950+0000","last_unstale":"2024-03-04T09:26:32.011051+0000","last_undegraded":"2024-03-04T09:26:32.011051+0000","last_fullsized":"2024-03-04T09:26:32.011051+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:35:08.886585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:31.021494+0000","last_change":"2024-03-04T09:26:31.012873+0000","last_active":"2024-03-04T09:26:31.021494+0000","last_peered":"2024-03-04T09:26:31.021494+0000","last_clean":"2024-03-04T09:26:31.021494+0000","last_became_active":"2024-03-04T09:26:31.012058+0000","last_became_peered":"2024-03-04T09:26:31.012058+0000","last_unstale":"2024-03-04T09:26:31.021494+0000","last_undegraded":"2024-03-04T09:26:31.021494+0000","last_fullsized":"2024-03-04T09:26:31.021494+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:12:02.940432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:31.007854+0000","last_change":"2024-03-04T09:26:31.006301+0000","last_active":"2024-03-04T09:26:31.007854+0000","last_peered":"2024-03-04T09:26:31.007854+0000","last_clean":"2024-03-04T09:26:31.007854+0000","last_became_active":"2024-03-04T09:26:31.005959+0000","last_became_peered":"2024-03-04T09:26:31.005959+0000","last_unstale":"2024-03-04T09:26:31.007854+0000","last_undegraded":"2024-03-04T09:26:31.007854+0000","last_fullsized":"2024-03-04T09:26:31.007854+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:25:51.053806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-04T09:26:32.013536+0000","last_change":"2024-03-04T09:26:31.008209+0000","last_active":"2024-03-04T09:26:32.013536+0000","last_peered":"2024-03-04T09:26:32.013536+0000","last_clean":"2024-03-04T09:26:32.013536+0000","last_became_active":"2024-03-04T09:26:31.007877+0000","last_became_peered":"2024-03-04T09:26:31.007877+0000","last_unstale":"2024-03-04T09:26:32.013536+0000","last_undegraded":"2024-03-04T09:26:32.013536+0000","last_fullsized":"2024-03-04T09:26:32.013536+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:37:28.350121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:31.019913+0000","last_change":"2024-03-04T09:26:31.000067+0000","last_active":"2024-03-04T09:26:31.019913+0000","last_peered":"2024-03-04T09:26:31.019913+0000","last_clean":"2024-03-04T09:26:31.019913+0000","last_became_active":"2024-03-04T09:26:30.997792+0000","last_became_peered":"2024-03-04T09:26:30.997792+0000","last_unstale":"2024-03-04T09:26:31.019913+0000","last_undegraded":"2024-03-04T09:26:31.019913+0000","last_fullsized":"2024-03-04T09:26:31.019913+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:23:15.690106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-04T09:26:32.013874+0000","last_change":"2024-03-04T09:26:31.006038+0000","last_active":"2024-03-04T09:26:32.013874+0000","last_peered":"2024-03-04T09:26:32.013874+0000","last_clean":"2024-03-04T09:26:32.013874+0000","last_became_active":"2024-03-04T09:26:31.005674+0000","last_became_peered":"2024-03-04T09:26:31.005674+0000","last_unstale":"2024-03-04T09:26:32.013874+0000","last_undegraded":"2024-03-04T09:26:32.013874+0000","last_fullsized":"2024-03-04T09:26:32.013874+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:21:26.223848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:31.020287+0000","last_change":"2024-03-04T09:26:31.013925+0000","last_active":"2024-03-04T09:26:31.020287+0000","last_peered":"2024-03-04T09:26:31.020287+0000","last_clean":"2024-03-04T09:26:31.020287+0000","last_became_active":"2024-03-04T09:26:31.012754+0000","last_became_peered":"2024-03-04T09:26:31.012754+0000","last_unstale":"2024-03-04T09:26:31.020287+0000","last_undegraded":"2024-03-04T09:26:31.020287+0000","last_fullsized":"2024-03-04T09:26:31.020287+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:00:21.753592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-04T09:26:32.011810+0000","last_change":"2024-03-04T09:26:31.006926+0000","last_active":"2024-03-04T09:26:32.011810+0000","last_peered":"2024-03-04T09:26:32.011810+0000","last_clean":"2024-03-04T09:26:32.011810+0000","last_became_active":"2024-03-04T09:26:31.006621+0000","last_became_peered":"2024-03-04T09:26:31.006621+0000","last_unstale":"2024-03-04T09:26:32.011810+0000","last_undegraded":"2024-03-04T09:26:32.011810+0000","last_fullsized":"2024-03-04T09:26:32.011810+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:00:21.753592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:31.020211+0000","last_change":"2024-03-04T09:26:31.001218+0000","last_active":"2024-03-04T09:26:31.020211+0000","last_peered":"2024-03-04T09:26:31.020211+0000","last_clean":"2024-03-04T09:26:31.020211+0000","last_became_active":"2024-03-04T09:26:31.000933+0000","last_became_peered":"2024-03-04T09:26:31.000933+0000","last_unstale":"2024-03-04T09:26:31.020211+0000","last_undegraded":"2024-03-04T09:26:31.020211+0000","last_fullsized":"2024-03-04T09:26:31.020211+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:57:23.393682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:31.012325+0000","last_change":"2024-03-04T09:26:31.000711+0000","last_active":"2024-03-04T09:26:31.012325+0000","last_peered":"2024-03-04T09:26:31.012325+0000","last_clean":"2024-03-04T09:26:31.012325+0000","last_became_active":"2024-03-04T09:26:31.000241+0000","last_became_peered":"2024-03-04T09:26:31.000241+0000","last_unstale":"2024-03-04T09:26:31.012325+0000","last_undegraded":"2024-03-04T09:26:31.012325+0000","last_fullsized":"2024-03-04T09:26:31.012325+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:33:02.297144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:31.023165+0000","last_change":"2024-03-04T09:26:31.016301+0000","last_active":"2024-03-04T09:26:31.023165+0000","last_peered":"2024-03-04T09:26:31.023165+0000","last_clean":"2024-03-04T09:26:31.023165+0000","last_became_active":"2024-03-04T09:26:31.015995+0000","last_became_peered":"2024-03-04T09:26:31.015995+0000","last_unstale":"2024-03-04T09:26:31.023165+0000","last_undegraded":"2024-03-04T09:26:31.023165+0000","last_fullsized":"2024-03-04T09:26:31.023165+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:57:23.393682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:31.007976+0000","last_change":"2024-03-04T09:26:31.001596+0000","last_active":"2024-03-04T09:26:31.007976+0000","last_peered":"2024-03-04T09:26:31.007976+0000","last_clean":"2024-03-04T09:26:31.007976+0000","last_became_active":"2024-03-04T09:26:31.001271+0000","last_became_peered":"2024-03-04T09:26:31.001271+0000","last_unstale":"2024-03-04T09:26:31.007976+0000","last_undegraded":"2024-03-04T09:26:31.007976+0000","last_fullsized":"2024-03-04T09:26:31.007976+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:35:08.886585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:31.014696+0000","last_change":"2024-03-04T09:26:31.012983+0000","last_active":"2024-03-04T09:26:31.014696+0000","last_peered":"2024-03-04T09:26:31.014696+0000","last_clean":"2024-03-04T09:26:31.014696+0000","last_became_active":"2024-03-04T09:26:31.012609+0000","last_became_peered":"2024-03-04T09:26:31.012609+0000","last_unstale":"2024-03-04T09:26:31.014696+0000","last_undegraded":"2024-03-04T09:26:31.014696+0000","last_fullsized":"2024-03-04T09:26:31.014696+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:37:28.350121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-04T09:26:32.015054+0000","last_change":"2024-03-04T09:26:31.008208+0000","last_active":"2024-03-04T09:26:32.015054+0000","last_peered":"2024-03-04T09:26:32.015054+0000","last_clean":"2024-03-04T09:26:32.015054+0000","last_became_active":"2024-03-04T09:26:31.007804+0000","last_became_peered":"2024-03-04T09:26:31.007804+0000","last_unstale":"2024-03-04T09:26:32.015054+0000","last_undegraded":"2024-03-04T09:26:32.015054+0000","last_fullsized":"2024-03-04T09:26:32.015054+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:57:23.393682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"32'4","reported_seq":16,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:31.023312+0000","last_change":"2024-03-04T09:26:29.991550+0000","last_active":"2024-03-04T09:26:31.023312+0000","last_peered":"2024-03-04T09:26:31.023312+0000","last_clean":"2024-03-04T09:26:31.023312+0000","last_became_active":"2024-03-04T09:26:29.991102+0000","last_became_peered":"2024-03-04T09:26:29.991102+0000","last_unstale":"2024-03-04T09:26:31.023312+0000","last_undegraded":"2024-03-04T09:26:31.023312+0000","last_fullsized":"2024-03-04T09:26:31.023312+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_clean_scrub_stamp":"2024-03-04T09:26:28.937043+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:17:15.944991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:31.007889+0000","last_change":"2024-03-04T09:26:31.005184+0000","last_active":"2024-03-04T09:26:31.007889+0000","last_peered":"2024-03-04T09:26:31.007889+0000","last_clean":"2024-03-04T09:26:31.007889+0000","last_became_active":"2024-03-04T09:26:31.004888+0000","last_became_peered":"2024-03-04T09:26:31.004888+0000","last_unstale":"2024-03-04T09:26:31.007889+0000","last_undegraded":"2024-03-04T09:26:31.007889+0000","last_fullsized":"2024-03-04T09:26:31.007889+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:46:49.122698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"32'1","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-04T09:26:32.015107+0000","last_change":"2024-03-04T09:26:29.998151+0000","last_active":"2024-03-04T09:26:32.015107+0000","last_peered":"2024-03-04T09:26:32.015107+0000","last_clean":"2024-03-04T09:26:32.015107+0000","last_became_active":"2024-03-04T09:26:29.997651+0000","last_became_peered":"2024-03-04T09:26:29.997651+0000","last_unstale":"2024-03-04T09:26:32.015107+0000","last_undegraded":"2024-03-04T09:26:32.015107+0000","last_fullsized":"2024-03-04T09:26:32.015107+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_clean_scrub_stamp":"2024-03-04T09:26:28.937043+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:46:48.108301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:31.022061+0000","last_change":"2024-03-04T09:26:31.018994+0000","last_active":"2024-03-04T09:26:31.022061+0000","last_peered":"2024-03-04T09:26:31.022061+0000","last_clean":"2024-03-04T09:26:31.022061+0000","last_became_active":"2024-03-04T09:26:31.018689+0000","last_became_peered":"2024-03-04T09:26:31.018689+0000","last_unstale":"2024-03-04T09:26:31.022061+0000","last_undegraded":"2024-03-04T09:26:31.022061+0000","last_fullsized":"2024-03-04T09:26:31.022061+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:37:28.350121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"32'2","reported_seq":14,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:31.030084+0000","last_change":"2024-03-04T09:26:29.989521+0000","last_active":"2024-03-04T09:26:31.030084+0000","last_peered":"2024-03-04T09:26:31.030084+0000","last_clean":"2024-03-04T09:26:31.030084+0000","last_became_active":"2024-03-04T09:26:29.989198+0000","last_became_peered":"2024-03-04T09:26:29.989198+0000","last_unstale":"2024-03-04T09:26:31.030084+0000","last_undegraded":"2024-03-04T09:26:31.030084+0000","last_fullsized":"2024-03-04T09:26:31.030084+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_clean_scrub_stamp":"2024-03-04T09:26:28.937043+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:07:14.092451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-04T09:26:32.014507+0000","last_change":"2024-03-04T09:26:31.006527+0000","last_active":"2024-03-04T09:26:32.014507+0000","last_peered":"2024-03-04T09:26:32.014507+0000","last_clean":"2024-03-04T09:26:32.014507+0000","last_became_active":"2024-03-04T09:26:31.004202+0000","last_became_peered":"2024-03-04T09:26:31.004202+0000","last_unstale":"2024-03-04T09:26:32.014507+0000","last_undegraded":"2024-03-04T09:26:32.014507+0000","last_fullsized":"2024-03-04T09:26:32.014507+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:39:30.925100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"32'2","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-04T09:26:32.011409+0000","last_change":"2024-03-04T09:26:29.982166+0000","last_active":"2024-03-04T09:26:32.011409+0000","last_peered":"2024-03-04T09:26:32.011409+0000","last_clean":"2024-03-04T09:26:32.011409+0000","last_became_active":"2024-03-04T09:26:29.981820+0000","last_became_peered":"2024-03-04T09:26:29.981820+0000","last_unstale":"2024-03-04T09:26:32.011409+0000","last_undegraded":"2024-03-04T09:26:32.011409+0000","last_fullsized":"2024-03-04T09:26:32.011409+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_clean_scrub_stamp":"2024-03-04T09:26:28.937043+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:05:38.108531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:31.014946+0000","last_change":"2024-03-04T09:26:31.013645+0000","last_active":"2024-03-04T09:26:31.014946+0000","last_peered":"2024-03-04T09:26:31.014946+0000","last_clean":"2024-03-04T09:26:31.014946+0000","last_became_active":"2024-03-04T09:26:31.013411+0000","last_became_peered":"2024-03-04T09:26:31.013411+0000","last_unstale":"2024-03-04T09:26:31.014946+0000","last_undegraded":"2024-03-04T09:26:31.014946+0000","last_fullsized":"2024-03-04T09:26:31.014946+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:07:15.106849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"32'1","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:31.017614+0000","last_change":"2024-03-04T09:26:29.996529+0000","last_active":"2024-03-04T09:26:31.017614+0000","last_peered":"2024-03-04T09:26:31.017614+0000","last_clean":"2024-03-04T09:26:31.017614+0000","last_became_active":"2024-03-04T09:26:29.996238+0000","last_became_peered":"2024-03-04T09:26:29.996238+0000","last_unstale":"2024-03-04T09:26:31.017614+0000","last_undegraded":"2024-03-04T09:26:31.017614+0000","last_fullsized":"2024-03-04T09:26:31.017614+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_clean_scrub_stamp":"2024-03-04T09:26:28.937043+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:53:31.025639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:31.020533+0000","last_change":"2024-03-04T09:26:31.007279+0000","last_active":"2024-03-04T09:26:31.020533+0000","last_peered":"2024-03-04T09:26:31.020533+0000","last_clean":"2024-03-04T09:26:31.020533+0000","last_became_active":"2024-03-04T09:26:31.006978+0000","last_became_peered":"2024-03-04T09:26:31.006978+0000","last_unstale":"2024-03-04T09:26:31.020533+0000","last_undegraded":"2024-03-04T09:26:31.020533+0000","last_fullsized":"2024-03-04T09:26:31.020533+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:48:31.810546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"32'1","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:31.005668+0000","last_change":"2024-03-04T09:26:29.993112+0000","last_active":"2024-03-04T09:26:31.005668+0000","last_peered":"2024-03-04T09:26:31.005668+0000","last_clean":"2024-03-04T09:26:31.005668+0000","last_became_active":"2024-03-04T09:26:29.992487+0000","last_became_peered":"2024-03-04T09:26:29.992487+0000","last_unstale":"2024-03-04T09:26:31.005668+0000","last_undegraded":"2024-03-04T09:26:31.005668+0000","last_fullsized":"2024-03-04T09:26:31.005668+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_clean_scrub_stamp":"2024-03-04T09:26:28.937043+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:17:15.944991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:31.005677+0000","last_change":"2024-03-04T09:26:31.003867+0000","last_active":"2024-03-04T09:26:31.005677+0000","last_peered":"2024-03-04T09:26:31.005677+0000","last_clean":"2024-03-04T09:26:31.005677+0000","last_became_active":"2024-03-04T09:26:31.003586+0000","last_became_peered":"2024-03-04T09:26:31.003586+0000","last_unstale":"2024-03-04T09:26:31.005677+0000","last_undegraded":"2024-03-04T09:26:31.005677+0000","last_fullsized":"2024-03-04T09:26:31.005677+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:24:35.905855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:31.001072+0000","last_change":"2024-03-04T09:26:29.989827+0000","last_active":"2024-03-04T09:26:31.001072+0000","last_peered":"2024-03-04T09:26:31.001072+0000","last_clean":"2024-03-04T09:26:31.001072+0000","last_became_active":"2024-03-04T09:26:29.988240+0000","last_became_peered":"2024-03-04T09:26:29.988240+0000","last_unstale":"2024-03-04T09:26:31.001072+0000","last_undegraded":"2024-03-04T09:26:31.001072+0000","last_fullsized":"2024-03-04T09:26:31.001072+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_clean_scrub_stamp":"2024-03-04T09:26:28.937043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:47:08.730106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-04T09:26:32.009034+0000","last_change":"2024-03-04T09:26:30.997463+0000","last_active":"2024-03-04T09:26:32.009034+0000","last_peered":"2024-03-04T09:26:32.009034+0000","last_clean":"2024-03-04T09:26:32.009034+0000","last_became_active":"2024-03-04T09:26:30.997071+0000","last_became_peered":"2024-03-04T09:26:30.997071+0000","last_unstale":"2024-03-04T09:26:32.009034+0000","last_undegraded":"2024-03-04T09:26:32.009034+0000","last_fullsized":"2024-03-04T09:26:32.009034+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:40:19.971433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:30.989499+0000","last_change":"2024-03-04T09:26:29.995006+0000","last_active":"2024-03-04T09:26:30.989499+0000","last_peered":"2024-03-04T09:26:30.989499+0000","last_clean":"2024-03-04T09:26:30.989499+0000","last_became_active":"2024-03-04T09:26:29.994675+0000","last_became_peered":"2024-03-04T09:26:29.994675+0000","last_unstale":"2024-03-04T09:26:30.989499+0000","last_undegraded":"2024-03-04T09:26:30.989499+0000","last_fullsized":"2024-03-04T09:26:30.989499+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_clean_scrub_stamp":"2024-03-04T09:26:28.937043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:34:55.432393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:31.024549+0000","last_change":"2024-03-04T09:26:31.022492+0000","last_active":"2024-03-04T09:26:31.024549+0000","last_peered":"2024-03-04T09:26:31.024549+0000","last_clean":"2024-03-04T09:26:31.024549+0000","last_became_active":"2024-03-04T09:26:31.022114+0000","last_became_peered":"2024-03-04T09:26:31.022114+0000","last_unstale":"2024-03-04T09:26:31.024549+0000","last_undegraded":"2024-03-04T09:26:31.024549+0000","last_fullsized":"2024-03-04T09:26:31.024549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:21:26.223848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:31.006712+0000","last_change":"2024-03-04T09:26:30.000770+0000","last_active":"2024-03-04T09:26:31.006712+0000","last_peered":"2024-03-04T09:26:31.006712+0000","last_clean":"2024-03-04T09:26:31.006712+0000","last_became_active":"2024-03-04T09:26:30.000511+0000","last_became_peered":"2024-03-04T09:26:30.000511+0000","last_unstale":"2024-03-04T09:26:31.006712+0000","last_undegraded":"2024-03-04T09:26:31.006712+0000","last_fullsized":"2024-03-04T09:26:31.006712+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_clean_scrub_stamp":"2024-03-04T09:26:28.937043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:57:22.379284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:31.025243+0000","last_change":"2024-03-04T09:26:31.020118+0000","last_active":"2024-03-04T09:26:31.025243+0000","last_peered":"2024-03-04T09:26:31.025243+0000","last_clean":"2024-03-04T09:26:31.025243+0000","last_became_active":"2024-03-04T09:26:31.019801+0000","last_became_peered":"2024-03-04T09:26:31.019801+0000","last_unstale":"2024-03-04T09:26:31.025243+0000","last_undegraded":"2024-03-04T09:26:31.025243+0000","last_fullsized":"2024-03-04T09:26:31.025243+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:39:30.925100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-04T09:26:32.015276+0000","last_change":"2024-03-04T09:26:29.993674+0000","last_active":"2024-03-04T09:26:32.015276+0000","last_peered":"2024-03-04T09:26:32.015276+0000","last_clean":"2024-03-04T09:26:32.015276+0000","last_became_active":"2024-03-04T09:26:29.993123+0000","last_became_peered":"2024-03-04T09:26:29.993123+0000","last_unstale":"2024-03-04T09:26:32.015276+0000","last_undegraded":"2024-03-04T09:26:32.015276+0000","last_fullsized":"2024-03-04T09:26:32.015276+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_clean_scrub_stamp":"2024-03-04T09:26:28.937043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:29:28.174031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:31.028088+0000","last_change":"2024-03-04T09:26:31.023762+0000","last_active":"2024-03-04T09:26:31.028088+0000","last_peered":"2024-03-04T09:26:31.028088+0000","last_clean":"2024-03-04T09:26:31.028088+0000","last_became_active":"2024-03-04T09:26:31.023415+0000","last_became_peered":"2024-03-04T09:26:31.023415+0000","last_unstale":"2024-03-04T09:26:31.028088+0000","last_undegraded":"2024-03-04T09:26:31.028088+0000","last_fullsized":"2024-03-04T09:26:31.028088+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:22:08.954238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"32'3","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:31.023420+0000","last_change":"2024-03-04T09:26:29.993093+0000","last_active":"2024-03-04T09:26:31.023420+0000","last_peered":"2024-03-04T09:26:31.023420+0000","last_clean":"2024-03-04T09:26:31.023420+0000","last_became_active":"2024-03-04T09:26:29.991775+0000","last_became_peered":"2024-03-04T09:26:29.991775+0000","last_unstale":"2024-03-04T09:26:31.023420+0000","last_undegraded":"2024-03-04T09:26:31.023420+0000","last_fullsized":"2024-03-04T09:26:31.023420+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_clean_scrub_stamp":"2024-03-04T09:26:28.937043+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:37:27.335724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:31.020585+0000","last_change":"2024-03-04T09:26:31.009318+0000","last_active":"2024-03-04T09:26:31.020585+0000","last_peered":"2024-03-04T09:26:31.020585+0000","last_clean":"2024-03-04T09:26:31.020585+0000","last_became_active":"2024-03-04T09:26:31.009024+0000","last_became_peered":"2024-03-04T09:26:31.009024+0000","last_unstale":"2024-03-04T09:26:31.020585+0000","last_undegraded":"2024-03-04T09:26:31.020585+0000","last_fullsized":"2024-03-04T09:26:31.020585+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:15:35.966281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"32'6","reported_seq":18,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:31.032230+0000","last_change":"2024-03-04T09:26:29.990120+0000","last_active":"2024-03-04T09:26:31.032230+0000","last_peered":"2024-03-04T09:26:31.032230+0000","last_clean":"2024-03-04T09:26:31.032230+0000","last_became_active":"2024-03-04T09:26:29.989719+0000","last_became_peered":"2024-03-04T09:26:29.989719+0000","last_unstale":"2024-03-04T09:26:31.032230+0000","last_undegraded":"2024-03-04T09:26:31.032230+0000","last_fullsized":"2024-03-04T09:26:31.032230+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_clean_scrub_stamp":"2024-03-04T09:26:28.937043+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:23:14.675708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-04T09:26:32.009038+0000","last_change":"2024-03-04T09:26:30.993692+0000","last_active":"2024-03-04T09:26:32.009038+0000","last_peered":"2024-03-04T09:26:32.009038+0000","last_clean":"2024-03-04T09:26:32.009038+0000","last_became_active":"2024-03-04T09:26:30.993346+0000","last_became_peered":"2024-03-04T09:26:30.993346+0000","last_unstale":"2024-03-04T09:26:32.009038+0000","last_undegraded":"2024-03-04T09:26:32.009038+0000","last_fullsized":"2024-03-04T09:26:32.009038+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:48:31.810546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"32'2","reported_seq":14,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:31.012675+0000","last_change":"2024-03-04T09:26:29.994702+0000","last_active":"2024-03-04T09:26:31.012675+0000","last_peered":"2024-03-04T09:26:31.012675+0000","last_clean":"2024-03-04T09:26:31.012675+0000","last_became_active":"2024-03-04T09:26:29.994384+0000","last_became_peered":"2024-03-04T09:26:29.994384+0000","last_unstale":"2024-03-04T09:26:31.012675+0000","last_undegraded":"2024-03-04T09:26:31.012675+0000","last_fullsized":"2024-03-04T09:26:31.012675+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_clean_scrub_stamp":"2024-03-04T09:26:28.937043+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:46:15.529293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:31.020518+0000","last_change":"2024-03-04T09:26:31.014894+0000","last_active":"2024-03-04T09:26:31.020518+0000","last_peered":"2024-03-04T09:26:31.020518+0000","last_clean":"2024-03-04T09:26:31.020518+0000","last_became_active":"2024-03-04T09:26:31.014592+0000","last_became_peered":"2024-03-04T09:26:31.014592+0000","last_unstale":"2024-03-04T09:26:31.020518+0000","last_undegraded":"2024-03-04T09:26:31.020518+0000","last_fullsized":"2024-03-04T09:26:31.020518+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:41:30.039171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-04T09:26:32.009169+0000","last_change":"2024-03-04T09:26:29.991792+0000","last_active":"2024-03-04T09:26:32.009169+0000","last_peered":"2024-03-04T09:26:32.009169+0000","last_clean":"2024-03-04T09:26:32.009169+0000","last_became_active":"2024-03-04T09:26:29.991495+0000","last_became_peered":"2024-03-04T09:26:29.991495+0000","last_unstale":"2024-03-04T09:26:32.009169+0000","last_undegraded":"2024-03-04T09:26:32.009169+0000","last_fullsized":"2024-03-04T09:26:32.009169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_clean_scrub_stamp":"2024-03-04T09:26:28.937043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:53:31.025639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:31.014675+0000","last_change":"2024-03-04T09:26:31.012794+0000","last_active":"2024-03-04T09:26:31.014675+0000","last_peered":"2024-03-04T09:26:31.014675+0000","last_clean":"2024-03-04T09:26:31.014675+0000","last_became_active":"2024-03-04T09:26:31.012142+0000","last_became_peered":"2024-03-04T09:26:31.012142+0000","last_unstale":"2024-03-04T09:26:31.014675+0000","last_undegraded":"2024-03-04T09:26:31.014675+0000","last_fullsized":"2024-03-04T09:26:31.014675+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:12:02.940432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:31.002286+0000","last_change":"2024-03-04T09:26:29.998129+0000","last_active":"2024-03-04T09:26:31.002286+0000","last_peered":"2024-03-04T09:26:31.002286+0000","last_clean":"2024-03-04T09:26:31.002286+0000","last_became_active":"2024-03-04T09:26:29.997795+0000","last_became_peered":"2024-03-04T09:26:29.997795+0000","last_unstale":"2024-03-04T09:26:31.002286+0000","last_undegraded":"2024-03-04T09:26:31.002286+0000","last_fullsized":"2024-03-04T09:26:31.002286+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_clean_scrub_stamp":"2024-03-04T09:26:28.937043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:37:27.335724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:31.007996+0000","last_change":"2024-03-04T09:26:31.003241+0000","last_active":"2024-03-04T09:26:31.007996+0000","last_peered":"2024-03-04T09:26:31.007996+0000","last_clean":"2024-03-04T09:26:31.007996+0000","last_became_active":"2024-03-04T09:26:31.002791+0000","last_became_peered":"2024-03-04T09:26:31.002791+0000","last_unstale":"2024-03-04T09:26:31.007996+0000","last_undegraded":"2024-03-04T09:26:31.007996+0000","last_fullsized":"2024-03-04T09:26:31.007996+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:53:32.040036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"32'8","reported_seq":20,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:31.030650+0000","last_change":"2024-03-04T09:26:29.985221+0000","last_active":"2024-03-04T09:26:31.030650+0000","last_peered":"2024-03-04T09:26:31.030650+0000","last_clean":"2024-03-04T09:26:31.030650+0000","last_became_active":"2024-03-04T09:26:29.983987+0000","last_became_peered":"2024-03-04T09:26:29.983987+0000","last_unstale":"2024-03-04T09:26:31.030650+0000","last_undegraded":"2024-03-04T09:26:31.030650+0000","last_fullsized":"2024-03-04T09:26:31.030650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_clean_scrub_stamp":"2024-03-04T09:26:28.937043+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:12:01.926034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:31.002733+0000","last_change":"2024-03-04T09:26:31.000056+0000","last_active":"2024-03-04T09:26:31.002733+0000","last_peered":"2024-03-04T09:26:31.002733+0000","last_clean":"2024-03-04T09:26:31.002733+0000","last_became_active":"2024-03-04T09:26:30.999711+0000","last_became_peered":"2024-03-04T09:26:30.999711+0000","last_unstale":"2024-03-04T09:26:31.002733+0000","last_undegraded":"2024-03-04T09:26:31.002733+0000","last_fullsized":"2024-03-04T09:26:31.002733+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:46:49.122698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"32'1","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-04T09:26:32.014055+0000","last_change":"2024-03-04T09:26:29.992377+0000","last_active":"2024-03-04T09:26:32.014055+0000","last_peered":"2024-03-04T09:26:32.014055+0000","last_clean":"2024-03-04T09:26:32.014055+0000","last_became_active":"2024-03-04T09:26:29.992090+0000","last_became_peered":"2024-03-04T09:26:29.992090+0000","last_unstale":"2024-03-04T09:26:32.014055+0000","last_undegraded":"2024-03-04T09:26:32.014055+0000","last_fullsized":"2024-03-04T09:26:32.014055+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_clean_scrub_stamp":"2024-03-04T09:26:28.937043+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:46:48.108301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:31.019814+0000","last_change":"2024-03-04T09:26:30.999018+0000","last_active":"2024-03-04T09:26:31.019814+0000","last_peered":"2024-03-04T09:26:31.019814+0000","last_clean":"2024-03-04T09:26:31.019814+0000","last_became_active":"2024-03-04T09:26:30.998702+0000","last_became_peered":"2024-03-04T09:26:30.998702+0000","last_unstale":"2024-03-04T09:26:31.019814+0000","last_undegraded":"2024-03-04T09:26:31.019814+0000","last_fullsized":"2024-03-04T09:26:31.019814+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:07:15.106849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"32'2","reported_seq":14,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:31.007350+0000","last_change":"2024-03-04T09:26:29.998254+0000","last_active":"2024-03-04T09:26:31.007350+0000","last_peered":"2024-03-04T09:26:31.007350+0000","last_clean":"2024-03-04T09:26:31.007350+0000","last_became_active":"2024-03-04T09:26:29.997760+0000","last_became_peered":"2024-03-04T09:26:29.997760+0000","last_unstale":"2024-03-04T09:26:31.007350+0000","last_undegraded":"2024-03-04T09:26:31.007350+0000","last_fullsized":"2024-03-04T09:26:31.007350+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_clean_scrub_stamp":"2024-03-04T09:26:28.937043+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:29:28.174031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-04T09:26:32.009936+0000","last_change":"2024-03-04T09:26:30.997958+0000","last_active":"2024-03-04T09:26:32.009936+0000","last_peered":"2024-03-04T09:26:32.009936+0000","last_clean":"2024-03-04T09:26:32.009936+0000","last_became_active":"2024-03-04T09:26:30.997670+0000","last_became_peered":"2024-03-04T09:26:30.997670+0000","last_unstale":"2024-03-04T09:26:32.009936+0000","last_undegraded":"2024-03-04T09:26:32.009936+0000","last_fullsized":"2024-03-04T09:26:32.009936+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:42:09.417392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"32'1","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:31.010145+0000","last_change":"2024-03-04T09:26:29.994556+0000","last_active":"2024-03-04T09:26:31.010145+0000","last_peered":"2024-03-04T09:26:31.010145+0000","last_clean":"2024-03-04T09:26:31.010145+0000","last_became_active":"2024-03-04T09:26:29.994249+0000","last_became_peered":"2024-03-04T09:26:29.994249+0000","last_unstale":"2024-03-04T09:26:31.010145+0000","last_undegraded":"2024-03-04T09:26:31.010145+0000","last_fullsized":"2024-03-04T09:26:31.010145+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_clean_scrub_stamp":"2024-03-04T09:26:28.937043+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:35:07.872187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-04T09:26:32.014099+0000","last_change":"2024-03-04T09:26:31.016683+0000","last_active":"2024-03-04T09:26:32.014099+0000","last_peered":"2024-03-04T09:26:32.014099+0000","last_clean":"2024-03-04T09:26:32.014099+0000","last_became_active":"2024-03-04T09:26:31.016460+0000","last_became_peered":"2024-03-04T09:26:31.016460+0000","last_unstale":"2024-03-04T09:26:32.014099+0000","last_undegraded":"2024-03-04T09:26:32.014099+0000","last_fullsized":"2024-03-04T09:26:32.014099+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:33:02.297144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:30.994368+0000","last_change":"2024-03-04T09:26:29.996666+0000","last_active":"2024-03-04T09:26:30.994368+0000","last_peered":"2024-03-04T09:26:30.994368+0000","last_clean":"2024-03-04T09:26:30.994368+0000","last_became_active":"2024-03-04T09:26:29.996377+0000","last_became_peered":"2024-03-04T09:26:29.996377+0000","last_unstale":"2024-03-04T09:26:30.994368+0000","last_undegraded":"2024-03-04T09:26:30.994368+0000","last_fullsized":"2024-03-04T09:26:30.994368+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_clean_scrub_stamp":"2024-03-04T09:26:28.937043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:43:35.091061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-04T09:26:32.010949+0000","last_change":"2024-03-04T09:26:31.010839+0000","last_active":"2024-03-04T09:26:32.010949+0000","last_peered":"2024-03-04T09:26:32.010949+0000","last_clean":"2024-03-04T09:26:32.010949+0000","last_became_active":"2024-03-04T09:26:31.010577+0000","last_became_peered":"2024-03-04T09:26:31.010577+0000","last_unstale":"2024-03-04T09:26:32.010949+0000","last_undegraded":"2024-03-04T09:26:32.010949+0000","last_fullsized":"2024-03-04T09:26:32.010949+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:43:36.105459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"32'2","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-04T09:26:32.014277+0000","last_change":"2024-03-04T09:26:29.989932+0000","last_active":"2024-03-04T09:26:32.014277+0000","last_peered":"2024-03-04T09:26:32.014277+0000","last_clean":"2024-03-04T09:26:32.014277+0000","last_became_active":"2024-03-04T09:26:29.988395+0000","last_became_peered":"2024-03-04T09:26:29.988395+0000","last_unstale":"2024-03-04T09:26:32.014277+0000","last_undegraded":"2024-03-04T09:26:32.014277+0000","last_fullsized":"2024-03-04T09:26:32.014277+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_clean_scrub_stamp":"2024-03-04T09:26:28.937043+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:43:35.091061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-04T09:26:32.013080+0000","last_change":"2024-03-04T09:26:31.002635+0000","last_active":"2024-03-04T09:26:32.013080+0000","last_peered":"2024-03-04T09:26:32.013080+0000","last_clean":"2024-03-04T09:26:32.013080+0000","last_became_active":"2024-03-04T09:26:31.002322+0000","last_became_peered":"2024-03-04T09:26:31.002322+0000","last_unstale":"2024-03-04T09:26:32.013080+0000","last_undegraded":"2024-03-04T09:26:32.013080+0000","last_fullsized":"2024-03-04T09:26:32.013080+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:46:49.122698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:30.993814+0000","last_change":"2024-03-04T09:26:29.996899+0000","last_active":"2024-03-04T09:26:30.993814+0000","last_peered":"2024-03-04T09:26:30.993814+0000","last_clean":"2024-03-04T09:26:30.993814+0000","last_became_active":"2024-03-04T09:26:29.996530+0000","last_became_peered":"2024-03-04T09:26:29.996530+0000","last_unstale":"2024-03-04T09:26:30.993814+0000","last_undegraded":"2024-03-04T09:26:30.993814+0000","last_fullsized":"2024-03-04T09:26:30.993814+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_clean_scrub_stamp":"2024-03-04T09:26:28.937043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:43:35.091061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-04T09:26:32.014476+0000","last_change":"2024-03-04T09:26:31.008740+0000","last_active":"2024-03-04T09:26:32.014476+0000","last_peered":"2024-03-04T09:26:32.014476+0000","last_clean":"2024-03-04T09:26:32.014476+0000","last_became_active":"2024-03-04T09:26:31.008426+0000","last_became_peered":"2024-03-04T09:26:31.008426+0000","last_unstale":"2024-03-04T09:26:32.014476+0000","last_undegraded":"2024-03-04T09:26:32.014476+0000","last_fullsized":"2024-03-04T09:26:32.014476+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:24:35.905855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:30.989129+0000","last_change":"2024-03-04T09:26:29.995289+0000","last_active":"2024-03-04T09:26:30.989129+0000","last_peered":"2024-03-04T09:26:30.989129+0000","last_clean":"2024-03-04T09:26:30.989129+0000","last_became_active":"2024-03-04T09:26:29.994796+0000","last_became_peered":"2024-03-04T09:26:29.994796+0000","last_unstale":"2024-03-04T09:26:30.989129+0000","last_undegraded":"2024-03-04T09:26:30.989129+0000","last_fullsized":"2024-03-04T09:26:30.989129+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_clean_scrub_stamp":"2024-03-04T09:26:28.937043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:07:50.874217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:31.014522+0000","last_change":"2024-03-04T09:26:31.010860+0000","last_active":"2024-03-04T09:26:31.014522+0000","last_peered":"2024-03-04T09:26:31.014522+0000","last_clean":"2024-03-04T09:26:31.014522+0000","last_became_active":"2024-03-04T09:26:31.010548+0000","last_became_peered":"2024-03-04T09:26:31.010548+0000","last_unstale":"2024-03-04T09:26:31.014522+0000","last_undegraded":"2024-03-04T09:26:31.014522+0000","last_fullsized":"2024-03-04T09:26:31.014522+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:25:51.053806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:30.983219+0000","last_change":"2024-03-04T09:26:29.998310+0000","last_active":"2024-03-04T09:26:30.983219+0000","last_peered":"2024-03-04T09:26:30.983219+0000","last_clean":"2024-03-04T09:26:30.983219+0000","last_became_active":"2024-03-04T09:26:29.998007+0000","last_became_peered":"2024-03-04T09:26:29.998007+0000","last_unstale":"2024-03-04T09:26:30.983219+0000","last_undegraded":"2024-03-04T09:26:30.983219+0000","last_fullsized":"2024-03-04T09:26:30.983219+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_clean_scrub_stamp":"2024-03-04T09:26:28.937043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:12:01.926034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-04T09:26:32.011226+0000","last_change":"2024-03-04T09:26:31.001422+0000","last_active":"2024-03-04T09:26:32.011226+0000","last_peered":"2024-03-04T09:26:32.011226+0000","last_clean":"2024-03-04T09:26:32.011226+0000","last_became_active":"2024-03-04T09:26:31.000477+0000","last_became_peered":"2024-03-04T09:26:31.000477+0000","last_unstale":"2024-03-04T09:26:32.011226+0000","last_undegraded":"2024-03-04T09:26:32.011226+0000","last_fullsized":"2024-03-04T09:26:32.011226+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:53:32.040036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"32'2","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-04T09:26:32.014194+0000","last_change":"2024-03-04T09:26:29.992590+0000","last_active":"2024-03-04T09:26:32.014194+0000","last_peered":"2024-03-04T09:26:32.014194+0000","last_clean":"2024-03-04T09:26:32.014194+0000","last_became_active":"2024-03-04T09:26:29.991132+0000","last_became_peered":"2024-03-04T09:26:29.991132+0000","last_unstale":"2024-03-04T09:26:32.014194+0000","last_undegraded":"2024-03-04T09:26:32.014194+0000","last_fullsized":"2024-03-04T09:26:32.014194+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_clean_scrub_stamp":"2024-03-04T09:26:28.937043+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:33:01.282746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:31.020231+0000","last_change":"2024-03-04T09:26:31.007703+0000","last_active":"2024-03-04T09:26:31.020231+0000","last_peered":"2024-03-04T09:26:31.020231+0000","last_clean":"2024-03-04T09:26:31.020231+0000","last_became_active":"2024-03-04T09:26:31.007126+0000","last_became_peered":"2024-03-04T09:26:31.007126+0000","last_unstale":"2024-03-04T09:26:31.020231+0000","last_undegraded":"2024-03-04T09:26:31.020231+0000","last_fullsized":"2024-03-04T09:26:31.020231+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:40:19.971433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"32'1","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-04T09:26:32.015245+0000","last_change":"2024-03-04T09:26:29.983841+0000","last_active":"2024-03-04T09:26:32.015245+0000","last_peered":"2024-03-04T09:26:32.015245+0000","last_clean":"2024-03-04T09:26:32.015245+0000","last_became_active":"2024-03-04T09:26:29.983498+0000","last_became_peered":"2024-03-04T09:26:29.983498+0000","last_unstale":"2024-03-04T09:26:32.015245+0000","last_undegraded":"2024-03-04T09:26:32.015245+0000","last_fullsized":"2024-03-04T09:26:32.015245+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_clean_scrub_stamp":"2024-03-04T09:26:28.937043+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:53:31.025639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-04T09:26:32.013885+0000","last_change":"2024-03-04T09:26:31.010663+0000","last_active":"2024-03-04T09:26:32.013885+0000","last_peered":"2024-03-04T09:26:32.013885+0000","last_clean":"2024-03-04T09:26:32.013885+0000","last_became_active":"2024-03-04T09:26:31.008387+0000","last_became_peered":"2024-03-04T09:26:31.008387+0000","last_unstale":"2024-03-04T09:26:32.013885+0000","last_undegraded":"2024-03-04T09:26:32.013885+0000","last_fullsized":"2024-03-04T09:26:32.013885+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:07:15.106849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-04T09:26:32.010098+0000","last_change":"2024-03-04T09:26:29.979071+0000","last_active":"2024-03-04T09:26:32.010098+0000","last_peered":"2024-03-04T09:26:32.010098+0000","last_clean":"2024-03-04T09:26:32.010098+0000","last_became_active":"2024-03-04T09:26:29.977521+0000","last_became_peered":"2024-03-04T09:26:29.977521+0000","last_unstale":"2024-03-04T09:26:32.010098+0000","last_undegraded":"2024-03-04T09:26:32.010098+0000","last_fullsized":"2024-03-04T09:26:32.010098+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_clean_scrub_stamp":"2024-03-04T09:26:28.937043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:29:28.174031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:31.008059+0000","last_change":"2024-03-04T09:26:31.006312+0000","last_active":"2024-03-04T09:26:31.008059+0000","last_peered":"2024-03-04T09:26:31.008059+0000","last_clean":"2024-03-04T09:26:31.008059+0000","last_became_active":"2024-03-04T09:26:31.005764+0000","last_became_peered":"2024-03-04T09:26:31.005764+0000","last_unstale":"2024-03-04T09:26:31.008059+0000","last_undegraded":"2024-03-04T09:26:31.008059+0000","last_fullsized":"2024-03-04T09:26:31.008059+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:33:02.297144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-04T09:26:32.014467+0000","last_change":"2024-03-04T09:26:29.990486+0000","last_active":"2024-03-04T09:26:32.014467+0000","last_peered":"2024-03-04T09:26:32.014467+0000","last_clean":"2024-03-04T09:26:32.014467+0000","last_became_active":"2024-03-04T09:26:29.990286+0000","last_became_peered":"2024-03-04T09:26:29.990286+0000","last_unstale":"2024-03-04T09:26:32.014467+0000","last_undegraded":"2024-03-04T09:26:32.014467+0000","last_fullsized":"2024-03-04T09:26:32.014467+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_clean_scrub_stamp":"2024-03-04T09:26:28.937043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:35:07.872187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-04T09:26:32.014152+0000","last_change":"2024-03-04T09:26:30.998907+0000","last_active":"2024-03-04T09:26:32.014152+0000","last_peered":"2024-03-04T09:26:32.014152+0000","last_clean":"2024-03-04T09:26:32.014152+0000","last_became_active":"2024-03-04T09:26:30.998587+0000","last_became_peered":"2024-03-04T09:26:30.998587+0000","last_unstale":"2024-03-04T09:26:32.014152+0000","last_undegraded":"2024-03-04T09:26:32.014152+0000","last_fullsized":"2024-03-04T09:26:32.014152+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:23:15.690106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-04T09:26:32.009297+0000","last_change":"2024-03-04T09:26:29.994388+0000","last_active":"2024-03-04T09:26:32.009297+0000","last_peered":"2024-03-04T09:26:32.009297+0000","last_clean":"2024-03-04T09:26:32.009297+0000","last_became_active":"2024-03-04T09:26:29.994122+0000","last_became_peered":"2024-03-04T09:26:29.994122+0000","last_unstale":"2024-03-04T09:26:32.009297+0000","last_undegraded":"2024-03-04T09:26:32.009297+0000","last_fullsized":"2024-03-04T09:26:32.009297+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_clean_scrub_stamp":"2024-03-04T09:26:28.937043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:35:07.872187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:31.022076+0000","last_change":"2024-03-04T09:26:31.012143+0000","last_active":"2024-03-04T09:26:31.022076+0000","last_peered":"2024-03-04T09:26:31.022076+0000","last_clean":"2024-03-04T09:26:31.022076+0000","last_became_active":"2024-03-04T09:26:31.011832+0000","last_became_peered":"2024-03-04T09:26:31.011832+0000","last_unstale":"2024-03-04T09:26:31.022076+0000","last_undegraded":"2024-03-04T09:26:31.022076+0000","last_fullsized":"2024-03-04T09:26:31.022076+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:15:35.966281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:30.989798+0000","last_change":"2024-03-04T09:26:29.993931+0000","last_active":"2024-03-04T09:26:30.989798+0000","last_peered":"2024-03-04T09:26:30.989798+0000","last_clean":"2024-03-04T09:26:30.989798+0000","last_became_active":"2024-03-04T09:26:29.993606+0000","last_became_peered":"2024-03-04T09:26:29.993606+0000","last_unstale":"2024-03-04T09:26:30.989798+0000","last_undegraded":"2024-03-04T09:26:30.989798+0000","last_fullsized":"2024-03-04T09:26:30.989798+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_clean_scrub_stamp":"2024-03-04T09:26:28.937043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:27:09.922042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:31.022368+0000","last_change":"2024-03-04T09:26:31.019418+0000","last_active":"2024-03-04T09:26:31.022368+0000","last_peered":"2024-03-04T09:26:31.022368+0000","last_clean":"2024-03-04T09:26:31.022368+0000","last_became_active":"2024-03-04T09:26:31.018863+0000","last_became_peered":"2024-03-04T09:26:31.018863+0000","last_unstale":"2024-03-04T09:26:31.022368+0000","last_undegraded":"2024-03-04T09:26:31.022368+0000","last_fullsized":"2024-03-04T09:26:31.022368+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:07:15.106849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:31.001044+0000","last_change":"2024-03-04T09:26:29.998156+0000","last_active":"2024-03-04T09:26:31.001044+0000","last_peered":"2024-03-04T09:26:31.001044+0000","last_clean":"2024-03-04T09:26:31.001044+0000","last_became_active":"2024-03-04T09:26:29.997875+0000","last_became_peered":"2024-03-04T09:26:29.997875+0000","last_unstale":"2024-03-04T09:26:31.001044+0000","last_undegraded":"2024-03-04T09:26:31.001044+0000","last_fullsized":"2024-03-04T09:26:31.001044+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_clean_scrub_stamp":"2024-03-04T09:26:28.937043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:25:50.039408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:31.014488+0000","last_change":"2024-03-04T09:26:31.010720+0000","last_active":"2024-03-04T09:26:31.014488+0000","last_peered":"2024-03-04T09:26:31.014488+0000","last_clean":"2024-03-04T09:26:31.014488+0000","last_became_active":"2024-03-04T09:26:31.010429+0000","last_became_peered":"2024-03-04T09:26:31.010429+0000","last_unstale":"2024-03-04T09:26:31.014488+0000","last_undegraded":"2024-03-04T09:26:31.014488+0000","last_fullsized":"2024-03-04T09:26:31.014488+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:24:35.905855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:30.990914+0000","last_change":"2024-03-04T09:26:30.000139+0000","last_active":"2024-03-04T09:26:30.990914+0000","last_peered":"2024-03-04T09:26:30.990914+0000","last_clean":"2024-03-04T09:26:30.990914+0000","last_became_active":"2024-03-04T09:26:29.999809+0000","last_became_peered":"2024-03-04T09:26:29.999809+0000","last_unstale":"2024-03-04T09:26:30.990914+0000","last_undegraded":"2024-03-04T09:26:30.990914+0000","last_fullsized":"2024-03-04T09:26:30.990914+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_clean_scrub_stamp":"2024-03-04T09:26:28.937043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:27:09.922042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":192,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:30.991687+0000","last_change":"2024-03-04T09:25:09.316159+0000","last_active":"2024-03-04T09:26:30.991687+0000","last_peered":"2024-03-04T09:26:30.991687+0000","last_clean":"2024-03-04T09:26:30.991687+0000","last_became_active":"2024-03-04T09:25:09.313767+0000","last_became_peered":"2024-03-04T09:25:09.313767+0000","last_unstale":"2024-03-04T09:26:30.991687+0000","last_undegraded":"2024-03-04T09:26:30.991687+0000","last_fullsized":"2024-03-04T09:26:30.991687+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:08.110953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:08.110953+0000","last_clean_scrub_stamp":"2024-03-04T09:25:08.110953+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:34:21.745390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574868,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27964,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343876,"statfs":{"total":96636764160,"available":96608129024,"internally_reserved":0,"allocated":958464,"data_stored":731014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574868,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27384,"kb_used_data":360,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344456,"statfs":{"total":96636764160,"available":96608722944,"internally_reserved":0,"allocated":368640,"data_stored":141172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574870,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27388,"kb_used_data":360,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344452,"statfs":{"total":96636764160,"available":96608718848,"internally_reserved":0,"allocated":368640,"data_stored":141228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574867,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27384,"kb_used_data":360,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344456,"statfs":{"total":96636764160,"available":96608722944,"internally_reserved":0,"allocated":368640,"data_stored":140660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574869,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27396,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344444,"statfs":{"total":96636764160,"available":96608710656,"internally_reserved":0,"allocated":376832,"data_stored":142190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574868,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27380,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344460,"statfs":{"total":96636764160,"available":96608727040,"internally_reserved":0,"allocated":364544,"data_stored":140626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574869,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27408,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344432,"statfs":{"total":96636764160,"available":96608698368,"internally_reserved":0,"allocated":389120,"data_stored":148490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574868,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27964,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343876,"statfs":{"total":96636764160,"available":96608129024,"internally_reserved":0,"allocated":966656,"data_stored":737224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-04T09:26:35.496 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T09:26:35.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.753+0000 7efc92800640 1 Processor -- start 2024-03-04T09:26:35.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.753+0000 7efc92800640 1 -- start start 2024-03-04T09:26:35.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.754+0000 7efc92800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7efc8c0af580 0x7efc8c15c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:35.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.754+0000 7efc92800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efc8c0cdc20 0x7efc8c0b2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:35.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.754+0000 7efc92800640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc8c15cc40 con 0x7efc8c0af1a0 2024-03-04T09:26:35.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.754+0000 7efc92800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc8c15cf20 con 0x7efc8c0af580 2024-03-04T09:26:35.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.754+0000 7efc92800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc8c15d200 con 0x7efc8c0cdc20 2024-03-04T09:26:35.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.755+0000 7efc8aa00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7efc8c0af580 0x7efc8c15c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:35.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.755+0000 7efc8aa00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7efc8c0af580 0x7efc8c15c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:40752/0 (socket says 172.21.15.6:40752) 2024-03-04T09:26:35.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.755+0000 7efc8aa00640 1 -- 172.21.15.6:0/3617287134 learned_addr learned my addr 172.21.15.6:0/3617287134 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:35.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.755+0000 7efc8be00640 1 --2- 172.21.15.6:0/3617287134 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efc8c0cdc20 0x7efc8c0b2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:35.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.755+0000 7efc8aa00640 1 -- 172.21.15.6:0/3617287134 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efc8c0cdc20 msgr2=0x7efc8c0b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:35.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.755+0000 7efc8aa00640 1 --2- 172.21.15.6:0/3617287134 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efc8c0cdc20 0x7efc8c0b2dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:35.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.755+0000 7efc8aa00640 1 -- 172.21.15.6:0/3617287134 >> v1:172.21.15.146:6789/0 conn(0x7efc8c0af1a0 legacy=0x7efc8c0b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:35.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.755+0000 7efc8aa00640 1 -- 172.21.15.6:0/3617287134 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc8c15d4e0 con 0x7efc8c0af580 2024-03-04T09:26:35.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.756+0000 7efc8aa00640 1 --2- 172.21.15.6:0/3617287134 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efc8c0af580 0x7efc8c15c600 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7efc74009870 tx=0x7efc7402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=acf4468f7eba9294 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:35.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.757+0000 7efc8a000640 1 -- 172.21.15.6:0/3617287134 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc740317c0 con 0x7efc8c0af580 2024-03-04T09:26:35.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.757+0000 7efc8a000640 1 -- 172.21.15.6:0/3617287134 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc74037070 con 0x7efc8c0af580 2024-03-04T09:26:35.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.757+0000 7efc8a000640 1 -- 172.21.15.6:0/3617287134 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc74031b80 con 0x7efc8c0af580 2024-03-04T09:26:35.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.757+0000 7efc92800640 1 -- 172.21.15.6:0/3617287134 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efc8c0af580 msgr2=0x7efc8c15c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:35.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.757+0000 7efc92800640 1 --2- 172.21.15.6:0/3617287134 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efc8c0af580 0x7efc8c15c600 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7efc74009870 tx=0x7efc7402b860 comp rx=0 tx=0).stop 2024-03-04T09:26:35.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.757+0000 7efc92800640 1 -- 172.21.15.6:0/3617287134 shutdown_connections 2024-03-04T09:26:35.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.757+0000 7efc92800640 1 --2- 172.21.15.6:0/3617287134 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efc8c0cdc20 0x7efc8c0b2dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:35.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.757+0000 7efc92800640 1 --2- 172.21.15.6:0/3617287134 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efc8c0af580 0x7efc8c15c600 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:35.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.757+0000 7efc92800640 1 -- 172.21.15.6:0/3617287134 >> 172.21.15.6:0/3617287134 conn(0x7efc8c06afb0 msgr2=0x7efc8c06b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:35.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.757+0000 7efc92800640 1 -- 172.21.15.6:0/3617287134 shutdown_connections 2024-03-04T09:26:35.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.757+0000 7efc92800640 1 -- 172.21.15.6:0/3617287134 wait complete. 2024-03-04T09:26:35.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.759+0000 7efc92800640 1 Processor -- start 2024-03-04T09:26:35.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.759+0000 7efc92800640 1 -- start start 2024-03-04T09:26:35.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.760+0000 7efc92800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efc8c0af1a0 0x7efc8c1f1a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:35.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.760+0000 7efc92800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efc8c0af580 0x7efc8c14e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:35.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.761+0000 7efc92800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efc8c0cdc20 0x7efc8c0cee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:35.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.761+0000 7efc92800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc8c0b3720 con 0x7efc8c0af580 2024-03-04T09:26:35.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.761+0000 7efc92800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc8c0b35a0 con 0x7efc8c0af1a0 2024-03-04T09:26:35.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.761+0000 7efc92800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc8c155e70 con 0x7efc8c0cdc20 2024-03-04T09:26:35.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.761+0000 7efc8aa00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efc8c0af580 0x7efc8c14e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:35.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.761+0000 7efc8aa00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efc8c0af580 0x7efc8c14e060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:40754/0 (socket says 172.21.15.6:40754) 2024-03-04T09:26:35.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.761+0000 7efc8aa00640 1 -- 172.21.15.6:0/147706238 learned_addr learned my addr 172.21.15.6:0/147706238 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:35.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.761+0000 7efc8b400640 1 --2- 172.21.15.6:0/147706238 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efc8c0af1a0 0x7efc8c1f1a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:35.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.761+0000 7efc8be00640 1 --2- 172.21.15.6:0/147706238 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efc8c0cdc20 0x7efc8c0cee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:35.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.761+0000 7efc8aa00640 1 -- 172.21.15.6:0/147706238 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efc8c0cdc20 msgr2=0x7efc8c0cee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:35.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.761+0000 7efc8aa00640 1 --2- 172.21.15.6:0/147706238 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efc8c0cdc20 0x7efc8c0cee30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:35.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.761+0000 7efc8aa00640 1 -- 172.21.15.6:0/147706238 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efc8c0af1a0 msgr2=0x7efc8c1f1a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:35.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.761+0000 7efc8aa00640 1 --2- 172.21.15.6:0/147706238 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efc8c0af1a0 0x7efc8c1f1a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:35.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.761+0000 7efc8aa00640 1 -- 172.21.15.6:0/147706238 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc8c1f5510 con 0x7efc8c0af580 2024-03-04T09:26:35.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.761+0000 7efc8b400640 1 --2- 172.21.15.6:0/147706238 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efc8c0af1a0 0x7efc8c1f1a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:26:35.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.762+0000 7efc8aa00640 1 --2- 172.21.15.6:0/147706238 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efc8c0af580 0x7efc8c14e060 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7efc74000c00 tx=0x7efc740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:35.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.762+0000 7efc83400640 1 -- 172.21.15.6:0/147706238 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc74032650 con 0x7efc8c0af580 2024-03-04T09:26:35.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.762+0000 7efc83400640 1 -- 172.21.15.6:0/147706238 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc740327f0 con 0x7efc8c0af580 2024-03-04T09:26:35.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.762+0000 7efc83400640 1 -- 172.21.15.6:0/147706238 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc74032be0 con 0x7efc8c0af580 2024-03-04T09:26:35.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.762+0000 7efc92800640 1 -- 172.21.15.6:0/147706238 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc8c1f57f0 con 0x7efc8c0af580 2024-03-04T09:26:35.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.762+0000 7efc92800640 1 -- 172.21.15.6:0/147706238 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc8c1f5db0 con 0x7efc8c0af580 2024-03-04T09:26:35.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.763+0000 7efc92800640 1 -- 172.21.15.6:0/147706238 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc8c069510 con 0x7efc8c0af580 2024-03-04T09:26:35.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.764+0000 7efc83400640 1 -- 172.21.15.6:0/147706238 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efc7404c060 con 0x7efc8c0af580 2024-03-04T09:26:35.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.765+0000 7efc83400640 1 --2- 172.21.15.6:0/147706238 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7efc6806b4d0 0x7efc6806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:35.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.765+0000 7efc83400640 1 -- 172.21.15.6:0/147706238 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efc740abd50 con 0x7efc8c0af580 2024-03-04T09:26:35.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.766+0000 7efc8b400640 1 --2- 172.21.15.6:0/147706238 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7efc6806b4d0 0x7efc6806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:35.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.767+0000 7efc8b400640 1 --2- 172.21.15.6:0/147706238 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7efc6806b4d0 0x7efc6806d990 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7efc7c004520 tx=0x7efc7c009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:35.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.770+0000 7efc83400640 1 -- 172.21.15.6:0/147706238 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7efc740af030 con 0x7efc8c0af580 2024-03-04T09:26:35.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.968+0000 7efc92800640 1 -- 172.21.15.6:0/147706238 --> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7efc8c0aeb90 con 0x7efc6806b4d0 2024-03-04T09:26:35.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.987+0000 7efc83400640 1 -- 172.21.15.6:0/147706238 <== mgr.4113 v2:172.21.15.146:6832/1788613334 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251904 (secure 0 0 0) 0x7efc8c0aeb90 con 0x7efc6806b4d0 2024-03-04T09:26:35.987 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:26:35.988 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-03-04T09:26:35.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.991+0000 7efc92800640 1 -- 172.21.15.6:0/147706238 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7efc6806b4d0 msgr2=0x7efc6806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:35.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.991+0000 7efc92800640 1 --2- 172.21.15.6:0/147706238 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7efc6806b4d0 0x7efc6806d990 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7efc7c004520 tx=0x7efc7c009040 comp rx=0 tx=0).stop 2024-03-04T09:26:35.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.991+0000 7efc92800640 1 -- 172.21.15.6:0/147706238 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efc8c0af580 msgr2=0x7efc8c14e060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:35.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.991+0000 7efc92800640 1 --2- 172.21.15.6:0/147706238 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efc8c0af580 0x7efc8c14e060 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7efc74000c00 tx=0x7efc740046c0 comp rx=0 tx=0).stop 2024-03-04T09:26:35.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.991+0000 7efc92800640 1 -- 172.21.15.6:0/147706238 shutdown_connections 2024-03-04T09:26:35.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.991+0000 7efc92800640 1 --2- 172.21.15.6:0/147706238 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7efc6806b4d0 0x7efc6806d990 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:35.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.991+0000 7efc92800640 1 --2- 172.21.15.6:0/147706238 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efc8c0cdc20 0x7efc8c0cee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:35.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.991+0000 7efc92800640 1 --2- 172.21.15.6:0/147706238 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efc8c0af580 0x7efc8c14e060 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:35.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.991+0000 7efc92800640 1 --2- 172.21.15.6:0/147706238 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efc8c0af1a0 0x7efc8c1f1a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:35.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.991+0000 7efc92800640 1 -- 172.21.15.6:0/147706238 >> 172.21.15.6:0/147706238 conn(0x7efc8c06afb0 msgr2=0x7efc8c0cd1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:35.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.992+0000 7efc92800640 1 -- 172.21.15.6:0/147706238 shutdown_connections 2024-03-04T09:26:35.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:35.992+0000 7efc92800640 1 -- 172.21.15.6:0/147706238 wait complete. 2024-03-04T09:26:36.008 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":70,"stamp":"2024-03-04T09:26:35.885715+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":116,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220268,"kb_used_data":4064,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754452,"statfs":{"total":773094113280,"available":772868558848,"internally_reserved":0,"allocated":4161536,"data_stored":2322604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002075"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:31.007535+0000","last_change":"2024-03-04T09:26:31.005107+0000","last_active":"2024-03-04T09:26:31.007535+0000","last_peered":"2024-03-04T09:26:31.007535+0000","last_clean":"2024-03-04T09:26:31.007535+0000","last_became_active":"2024-03-04T09:26:31.004571+0000","last_became_peered":"2024-03-04T09:26:31.004571+0000","last_unstale":"2024-03-04T09:26:31.007535+0000","last_undegraded":"2024-03-04T09:26:31.007535+0000","last_fullsized":"2024-03-04T09:26:31.007535+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:29:29.188429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:31.020432+0000","last_change":"2024-03-04T09:26:31.015770+0000","last_active":"2024-03-04T09:26:31.020432+0000","last_peered":"2024-03-04T09:26:31.020432+0000","last_clean":"2024-03-04T09:26:31.020432+0000","last_became_active":"2024-03-04T09:26:31.015013+0000","last_became_peered":"2024-03-04T09:26:31.015013+0000","last_unstale":"2024-03-04T09:26:31.020432+0000","last_undegraded":"2024-03-04T09:26:31.020432+0000","last_fullsized":"2024-03-04T09:26:31.020432+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:21:26.223848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:31.020606+0000","last_change":"2024-03-04T09:26:31.009260+0000","last_active":"2024-03-04T09:26:31.020606+0000","last_peered":"2024-03-04T09:26:31.020606+0000","last_clean":"2024-03-04T09:26:31.020606+0000","last_became_active":"2024-03-04T09:26:31.009007+0000","last_became_peered":"2024-03-04T09:26:31.009007+0000","last_unstale":"2024-03-04T09:26:31.020606+0000","last_undegraded":"2024-03-04T09:26:31.020606+0000","last_fullsized":"2024-03-04T09:26:31.020606+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:42:09.417392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-04T09:26:32.009864+0000","last_change":"2024-03-04T09:26:31.004397+0000","last_active":"2024-03-04T09:26:32.009864+0000","last_peered":"2024-03-04T09:26:32.009864+0000","last_clean":"2024-03-04T09:26:32.009864+0000","last_became_active":"2024-03-04T09:26:31.004112+0000","last_became_peered":"2024-03-04T09:26:31.004112+0000","last_unstale":"2024-03-04T09:26:32.009864+0000","last_undegraded":"2024-03-04T09:26:32.009864+0000","last_fullsized":"2024-03-04T09:26:32.009864+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:15:35.966281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-04T09:26:32.010900+0000","last_change":"2024-03-04T09:26:31.001936+0000","last_active":"2024-03-04T09:26:32.010900+0000","last_peered":"2024-03-04T09:26:32.010900+0000","last_clean":"2024-03-04T09:26:32.010900+0000","last_became_active":"2024-03-04T09:26:31.001613+0000","last_became_peered":"2024-03-04T09:26:31.001613+0000","last_unstale":"2024-03-04T09:26:32.010900+0000","last_undegraded":"2024-03-04T09:26:32.010900+0000","last_fullsized":"2024-03-04T09:26:32.010900+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:17:16.959389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:31.027883+0000","last_change":"2024-03-04T09:26:31.022957+0000","last_active":"2024-03-04T09:26:31.027883+0000","last_peered":"2024-03-04T09:26:31.027883+0000","last_clean":"2024-03-04T09:26:31.027883+0000","last_became_active":"2024-03-04T09:26:31.022640+0000","last_became_peered":"2024-03-04T09:26:31.022640+0000","last_unstale":"2024-03-04T09:26:31.027883+0000","last_undegraded":"2024-03-04T09:26:31.027883+0000","last_fullsized":"2024-03-04T09:26:31.027883+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:02:48.660304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:31.017997+0000","last_change":"2024-03-04T09:26:31.015526+0000","last_active":"2024-03-04T09:26:31.017997+0000","last_peered":"2024-03-04T09:26:31.017997+0000","last_clean":"2024-03-04T09:26:31.017997+0000","last_became_active":"2024-03-04T09:26:31.015310+0000","last_became_peered":"2024-03-04T09:26:31.015310+0000","last_unstale":"2024-03-04T09:26:31.017997+0000","last_undegraded":"2024-03-04T09:26:31.017997+0000","last_fullsized":"2024-03-04T09:26:31.017997+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:23:15.690106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:31.014352+0000","last_change":"2024-03-04T09:26:31.010572+0000","last_active":"2024-03-04T09:26:31.014352+0000","last_peered":"2024-03-04T09:26:31.014352+0000","last_clean":"2024-03-04T09:26:31.014352+0000","last_became_active":"2024-03-04T09:26:31.010314+0000","last_became_peered":"2024-03-04T09:26:31.010314+0000","last_unstale":"2024-03-04T09:26:31.014352+0000","last_undegraded":"2024-03-04T09:26:31.014352+0000","last_fullsized":"2024-03-04T09:26:31.014352+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:33:02.297144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:31.007901+0000","last_change":"2024-03-04T09:26:31.006526+0000","last_active":"2024-03-04T09:26:31.007901+0000","last_peered":"2024-03-04T09:26:31.007901+0000","last_clean":"2024-03-04T09:26:31.007901+0000","last_became_active":"2024-03-04T09:26:31.006225+0000","last_became_peered":"2024-03-04T09:26:31.006225+0000","last_unstale":"2024-03-04T09:26:31.007901+0000","last_undegraded":"2024-03-04T09:26:31.007901+0000","last_fullsized":"2024-03-04T09:26:31.007901+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:47:09.744504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-04T09:26:32.013327+0000","last_change":"2024-03-04T09:26:31.019152+0000","last_active":"2024-03-04T09:26:32.013327+0000","last_peered":"2024-03-04T09:26:32.013327+0000","last_clean":"2024-03-04T09:26:32.013327+0000","last_became_active":"2024-03-04T09:26:31.018841+0000","last_became_peered":"2024-03-04T09:26:31.018841+0000","last_unstale":"2024-03-04T09:26:32.013327+0000","last_undegraded":"2024-03-04T09:26:32.013327+0000","last_fullsized":"2024-03-04T09:26:32.013327+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:41:30.039171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:31.014620+0000","last_change":"2024-03-04T09:26:31.012528+0000","last_active":"2024-03-04T09:26:31.014620+0000","last_peered":"2024-03-04T09:26:31.014620+0000","last_clean":"2024-03-04T09:26:31.014620+0000","last_became_active":"2024-03-04T09:26:31.011978+0000","last_became_peered":"2024-03-04T09:26:31.011978+0000","last_unstale":"2024-03-04T09:26:31.014620+0000","last_undegraded":"2024-03-04T09:26:31.014620+0000","last_fullsized":"2024-03-04T09:26:31.014620+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:47:09.744504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:31.020234+0000","last_change":"2024-03-04T09:26:31.014860+0000","last_active":"2024-03-04T09:26:31.020234+0000","last_peered":"2024-03-04T09:26:31.020234+0000","last_clean":"2024-03-04T09:26:31.020234+0000","last_became_active":"2024-03-04T09:26:31.014554+0000","last_became_peered":"2024-03-04T09:26:31.014554+0000","last_unstale":"2024-03-04T09:26:31.020234+0000","last_undegraded":"2024-03-04T09:26:31.020234+0000","last_fullsized":"2024-03-04T09:26:31.020234+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:00:21.753592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-04T09:26:32.012576+0000","last_change":"2024-03-04T09:26:31.008167+0000","last_active":"2024-03-04T09:26:32.012576+0000","last_peered":"2024-03-04T09:26:32.012576+0000","last_clean":"2024-03-04T09:26:32.012576+0000","last_became_active":"2024-03-04T09:26:31.005783+0000","last_became_peered":"2024-03-04T09:26:31.005783+0000","last_unstale":"2024-03-04T09:26:32.012576+0000","last_undegraded":"2024-03-04T09:26:32.012576+0000","last_fullsized":"2024-03-04T09:26:32.012576+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:29:29.188429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:31.015415+0000","last_change":"2024-03-04T09:26:31.007649+0000","last_active":"2024-03-04T09:26:31.015415+0000","last_peered":"2024-03-04T09:26:31.015415+0000","last_clean":"2024-03-04T09:26:31.015415+0000","last_became_active":"2024-03-04T09:26:31.007367+0000","last_became_peered":"2024-03-04T09:26:31.007367+0000","last_unstale":"2024-03-04T09:26:31.015415+0000","last_undegraded":"2024-03-04T09:26:31.015415+0000","last_fullsized":"2024-03-04T09:26:31.015415+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:40:19.971433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:31.020921+0000","last_change":"2024-03-04T09:26:31.018291+0000","last_active":"2024-03-04T09:26:31.020921+0000","last_peered":"2024-03-04T09:26:31.020921+0000","last_clean":"2024-03-04T09:26:31.020921+0000","last_became_active":"2024-03-04T09:26:31.017998+0000","last_became_peered":"2024-03-04T09:26:31.017998+0000","last_unstale":"2024-03-04T09:26:31.020921+0000","last_undegraded":"2024-03-04T09:26:31.020921+0000","last_fullsized":"2024-03-04T09:26:31.020921+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:41:30.039171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:31.024281+0000","last_change":"2024-03-04T09:26:31.022400+0000","last_active":"2024-03-04T09:26:31.024281+0000","last_peered":"2024-03-04T09:26:31.024281+0000","last_clean":"2024-03-04T09:26:31.024281+0000","last_became_active":"2024-03-04T09:26:31.021984+0000","last_became_peered":"2024-03-04T09:26:31.021984+0000","last_unstale":"2024-03-04T09:26:31.024281+0000","last_undegraded":"2024-03-04T09:26:31.024281+0000","last_fullsized":"2024-03-04T09:26:31.024281+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:48:31.810546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:31.024431+0000","last_change":"2024-03-04T09:26:31.020714+0000","last_active":"2024-03-04T09:26:31.024431+0000","last_peered":"2024-03-04T09:26:31.024431+0000","last_clean":"2024-03-04T09:26:31.024431+0000","last_became_active":"2024-03-04T09:26:31.017378+0000","last_became_peered":"2024-03-04T09:26:31.017378+0000","last_unstale":"2024-03-04T09:26:31.024431+0000","last_undegraded":"2024-03-04T09:26:31.024431+0000","last_fullsized":"2024-03-04T09:26:31.024431+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:40:19.971433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:31.020038+0000","last_change":"2024-03-04T09:26:31.012132+0000","last_active":"2024-03-04T09:26:31.020038+0000","last_peered":"2024-03-04T09:26:31.020038+0000","last_clean":"2024-03-04T09:26:31.020038+0000","last_became_active":"2024-03-04T09:26:31.011749+0000","last_became_peered":"2024-03-04T09:26:31.011749+0000","last_unstale":"2024-03-04T09:26:31.020038+0000","last_undegraded":"2024-03-04T09:26:31.020038+0000","last_fullsized":"2024-03-04T09:26:31.020038+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:48:31.810546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:31.021506+0000","last_change":"2024-03-04T09:26:31.012281+0000","last_active":"2024-03-04T09:26:31.021506+0000","last_peered":"2024-03-04T09:26:31.021506+0000","last_clean":"2024-03-04T09:26:31.021506+0000","last_became_active":"2024-03-04T09:26:31.011970+0000","last_became_peered":"2024-03-04T09:26:31.011970+0000","last_unstale":"2024-03-04T09:26:31.021506+0000","last_undegraded":"2024-03-04T09:26:31.021506+0000","last_fullsized":"2024-03-04T09:26:31.021506+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:42:09.417392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-04T09:26:32.011051+0000","last_change":"2024-03-04T09:26:31.021324+0000","last_active":"2024-03-04T09:26:32.011051+0000","last_peered":"2024-03-04T09:26:32.011051+0000","last_clean":"2024-03-04T09:26:32.011051+0000","last_became_active":"2024-03-04T09:26:31.020950+0000","last_became_peered":"2024-03-04T09:26:31.020950+0000","last_unstale":"2024-03-04T09:26:32.011051+0000","last_undegraded":"2024-03-04T09:26:32.011051+0000","last_fullsized":"2024-03-04T09:26:32.011051+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:35:08.886585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:31.021494+0000","last_change":"2024-03-04T09:26:31.012873+0000","last_active":"2024-03-04T09:26:31.021494+0000","last_peered":"2024-03-04T09:26:31.021494+0000","last_clean":"2024-03-04T09:26:31.021494+0000","last_became_active":"2024-03-04T09:26:31.012058+0000","last_became_peered":"2024-03-04T09:26:31.012058+0000","last_unstale":"2024-03-04T09:26:31.021494+0000","last_undegraded":"2024-03-04T09:26:31.021494+0000","last_fullsized":"2024-03-04T09:26:31.021494+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:12:02.940432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:31.007854+0000","last_change":"2024-03-04T09:26:31.006301+0000","last_active":"2024-03-04T09:26:31.007854+0000","last_peered":"2024-03-04T09:26:31.007854+0000","last_clean":"2024-03-04T09:26:31.007854+0000","last_became_active":"2024-03-04T09:26:31.005959+0000","last_became_peered":"2024-03-04T09:26:31.005959+0000","last_unstale":"2024-03-04T09:26:31.007854+0000","last_undegraded":"2024-03-04T09:26:31.007854+0000","last_fullsized":"2024-03-04T09:26:31.007854+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:25:51.053806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-04T09:26:32.013536+0000","last_change":"2024-03-04T09:26:31.008209+0000","last_active":"2024-03-04T09:26:32.013536+0000","last_peered":"2024-03-04T09:26:32.013536+0000","last_clean":"2024-03-04T09:26:32.013536+0000","last_became_active":"2024-03-04T09:26:31.007877+0000","last_became_peered":"2024-03-04T09:26:31.007877+0000","last_unstale":"2024-03-04T09:26:32.013536+0000","last_undegraded":"2024-03-04T09:26:32.013536+0000","last_fullsized":"2024-03-04T09:26:32.013536+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:37:28.350121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:31.019913+0000","last_change":"2024-03-04T09:26:31.000067+0000","last_active":"2024-03-04T09:26:31.019913+0000","last_peered":"2024-03-04T09:26:31.019913+0000","last_clean":"2024-03-04T09:26:31.019913+0000","last_became_active":"2024-03-04T09:26:30.997792+0000","last_became_peered":"2024-03-04T09:26:30.997792+0000","last_unstale":"2024-03-04T09:26:31.019913+0000","last_undegraded":"2024-03-04T09:26:31.019913+0000","last_fullsized":"2024-03-04T09:26:31.019913+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:23:15.690106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-04T09:26:32.013874+0000","last_change":"2024-03-04T09:26:31.006038+0000","last_active":"2024-03-04T09:26:32.013874+0000","last_peered":"2024-03-04T09:26:32.013874+0000","last_clean":"2024-03-04T09:26:32.013874+0000","last_became_active":"2024-03-04T09:26:31.005674+0000","last_became_peered":"2024-03-04T09:26:31.005674+0000","last_unstale":"2024-03-04T09:26:32.013874+0000","last_undegraded":"2024-03-04T09:26:32.013874+0000","last_fullsized":"2024-03-04T09:26:32.013874+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:21:26.223848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:31.020287+0000","last_change":"2024-03-04T09:26:31.013925+0000","last_active":"2024-03-04T09:26:31.020287+0000","last_peered":"2024-03-04T09:26:31.020287+0000","last_clean":"2024-03-04T09:26:31.020287+0000","last_became_active":"2024-03-04T09:26:31.012754+0000","last_became_peered":"2024-03-04T09:26:31.012754+0000","last_unstale":"2024-03-04T09:26:31.020287+0000","last_undegraded":"2024-03-04T09:26:31.020287+0000","last_fullsized":"2024-03-04T09:26:31.020287+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:00:21.753592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-04T09:26:32.011810+0000","last_change":"2024-03-04T09:26:31.006926+0000","last_active":"2024-03-04T09:26:32.011810+0000","last_peered":"2024-03-04T09:26:32.011810+0000","last_clean":"2024-03-04T09:26:32.011810+0000","last_became_active":"2024-03-04T09:26:31.006621+0000","last_became_peered":"2024-03-04T09:26:31.006621+0000","last_unstale":"2024-03-04T09:26:32.011810+0000","last_undegraded":"2024-03-04T09:26:32.011810+0000","last_fullsized":"2024-03-04T09:26:32.011810+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:00:21.753592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:31.020211+0000","last_change":"2024-03-04T09:26:31.001218+0000","last_active":"2024-03-04T09:26:31.020211+0000","last_peered":"2024-03-04T09:26:31.020211+0000","last_clean":"2024-03-04T09:26:31.020211+0000","last_became_active":"2024-03-04T09:26:31.000933+0000","last_became_peered":"2024-03-04T09:26:31.000933+0000","last_unstale":"2024-03-04T09:26:31.020211+0000","last_undegraded":"2024-03-04T09:26:31.020211+0000","last_fullsized":"2024-03-04T09:26:31.020211+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:57:23.393682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:31.012325+0000","last_change":"2024-03-04T09:26:31.000711+0000","last_active":"2024-03-04T09:26:31.012325+0000","last_peered":"2024-03-04T09:26:31.012325+0000","last_clean":"2024-03-04T09:26:31.012325+0000","last_became_active":"2024-03-04T09:26:31.000241+0000","last_became_peered":"2024-03-04T09:26:31.000241+0000","last_unstale":"2024-03-04T09:26:31.012325+0000","last_undegraded":"2024-03-04T09:26:31.012325+0000","last_fullsized":"2024-03-04T09:26:31.012325+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:33:02.297144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:31.023165+0000","last_change":"2024-03-04T09:26:31.016301+0000","last_active":"2024-03-04T09:26:31.023165+0000","last_peered":"2024-03-04T09:26:31.023165+0000","last_clean":"2024-03-04T09:26:31.023165+0000","last_became_active":"2024-03-04T09:26:31.015995+0000","last_became_peered":"2024-03-04T09:26:31.015995+0000","last_unstale":"2024-03-04T09:26:31.023165+0000","last_undegraded":"2024-03-04T09:26:31.023165+0000","last_fullsized":"2024-03-04T09:26:31.023165+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:57:23.393682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:31.007976+0000","last_change":"2024-03-04T09:26:31.001596+0000","last_active":"2024-03-04T09:26:31.007976+0000","last_peered":"2024-03-04T09:26:31.007976+0000","last_clean":"2024-03-04T09:26:31.007976+0000","last_became_active":"2024-03-04T09:26:31.001271+0000","last_became_peered":"2024-03-04T09:26:31.001271+0000","last_unstale":"2024-03-04T09:26:31.007976+0000","last_undegraded":"2024-03-04T09:26:31.007976+0000","last_fullsized":"2024-03-04T09:26:31.007976+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:35:08.886585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:31.014696+0000","last_change":"2024-03-04T09:26:31.012983+0000","last_active":"2024-03-04T09:26:31.014696+0000","last_peered":"2024-03-04T09:26:31.014696+0000","last_clean":"2024-03-04T09:26:31.014696+0000","last_became_active":"2024-03-04T09:26:31.012609+0000","last_became_peered":"2024-03-04T09:26:31.012609+0000","last_unstale":"2024-03-04T09:26:31.014696+0000","last_undegraded":"2024-03-04T09:26:31.014696+0000","last_fullsized":"2024-03-04T09:26:31.014696+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:37:28.350121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-04T09:26:32.015054+0000","last_change":"2024-03-04T09:26:31.008208+0000","last_active":"2024-03-04T09:26:32.015054+0000","last_peered":"2024-03-04T09:26:32.015054+0000","last_clean":"2024-03-04T09:26:32.015054+0000","last_became_active":"2024-03-04T09:26:31.007804+0000","last_became_peered":"2024-03-04T09:26:31.007804+0000","last_unstale":"2024-03-04T09:26:32.015054+0000","last_undegraded":"2024-03-04T09:26:32.015054+0000","last_fullsized":"2024-03-04T09:26:32.015054+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:57:23.393682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"32'4","reported_seq":16,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:31.023312+0000","last_change":"2024-03-04T09:26:29.991550+0000","last_active":"2024-03-04T09:26:31.023312+0000","last_peered":"2024-03-04T09:26:31.023312+0000","last_clean":"2024-03-04T09:26:31.023312+0000","last_became_active":"2024-03-04T09:26:29.991102+0000","last_became_peered":"2024-03-04T09:26:29.991102+0000","last_unstale":"2024-03-04T09:26:31.023312+0000","last_undegraded":"2024-03-04T09:26:31.023312+0000","last_fullsized":"2024-03-04T09:26:31.023312+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_clean_scrub_stamp":"2024-03-04T09:26:28.937043+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:17:15.944991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:31.007889+0000","last_change":"2024-03-04T09:26:31.005184+0000","last_active":"2024-03-04T09:26:31.007889+0000","last_peered":"2024-03-04T09:26:31.007889+0000","last_clean":"2024-03-04T09:26:31.007889+0000","last_became_active":"2024-03-04T09:26:31.004888+0000","last_became_peered":"2024-03-04T09:26:31.004888+0000","last_unstale":"2024-03-04T09:26:31.007889+0000","last_undegraded":"2024-03-04T09:26:31.007889+0000","last_fullsized":"2024-03-04T09:26:31.007889+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:46:49.122698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"32'1","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-04T09:26:32.015107+0000","last_change":"2024-03-04T09:26:29.998151+0000","last_active":"2024-03-04T09:26:32.015107+0000","last_peered":"2024-03-04T09:26:32.015107+0000","last_clean":"2024-03-04T09:26:32.015107+0000","last_became_active":"2024-03-04T09:26:29.997651+0000","last_became_peered":"2024-03-04T09:26:29.997651+0000","last_unstale":"2024-03-04T09:26:32.015107+0000","last_undegraded":"2024-03-04T09:26:32.015107+0000","last_fullsized":"2024-03-04T09:26:32.015107+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_clean_scrub_stamp":"2024-03-04T09:26:28.937043+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:46:48.108301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:31.022061+0000","last_change":"2024-03-04T09:26:31.018994+0000","last_active":"2024-03-04T09:26:31.022061+0000","last_peered":"2024-03-04T09:26:31.022061+0000","last_clean":"2024-03-04T09:26:31.022061+0000","last_became_active":"2024-03-04T09:26:31.018689+0000","last_became_peered":"2024-03-04T09:26:31.018689+0000","last_unstale":"2024-03-04T09:26:31.022061+0000","last_undegraded":"2024-03-04T09:26:31.022061+0000","last_fullsized":"2024-03-04T09:26:31.022061+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:37:28.350121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"32'2","reported_seq":14,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:31.030084+0000","last_change":"2024-03-04T09:26:29.989521+0000","last_active":"2024-03-04T09:26:31.030084+0000","last_peered":"2024-03-04T09:26:31.030084+0000","last_clean":"2024-03-04T09:26:31.030084+0000","last_became_active":"2024-03-04T09:26:29.989198+0000","last_became_peered":"2024-03-04T09:26:29.989198+0000","last_unstale":"2024-03-04T09:26:31.030084+0000","last_undegraded":"2024-03-04T09:26:31.030084+0000","last_fullsized":"2024-03-04T09:26:31.030084+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_clean_scrub_stamp":"2024-03-04T09:26:28.937043+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:07:14.092451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-04T09:26:32.014507+0000","last_change":"2024-03-04T09:26:31.006527+0000","last_active":"2024-03-04T09:26:32.014507+0000","last_peered":"2024-03-04T09:26:32.014507+0000","last_clean":"2024-03-04T09:26:32.014507+0000","last_became_active":"2024-03-04T09:26:31.004202+0000","last_became_peered":"2024-03-04T09:26:31.004202+0000","last_unstale":"2024-03-04T09:26:32.014507+0000","last_undegraded":"2024-03-04T09:26:32.014507+0000","last_fullsized":"2024-03-04T09:26:32.014507+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:39:30.925100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"32'2","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-04T09:26:32.011409+0000","last_change":"2024-03-04T09:26:29.982166+0000","last_active":"2024-03-04T09:26:32.011409+0000","last_peered":"2024-03-04T09:26:32.011409+0000","last_clean":"2024-03-04T09:26:32.011409+0000","last_became_active":"2024-03-04T09:26:29.981820+0000","last_became_peered":"2024-03-04T09:26:29.981820+0000","last_unstale":"2024-03-04T09:26:32.011409+0000","last_undegraded":"2024-03-04T09:26:32.011409+0000","last_fullsized":"2024-03-04T09:26:32.011409+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_clean_scrub_stamp":"2024-03-04T09:26:28.937043+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:05:38.108531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:31.014946+0000","last_change":"2024-03-04T09:26:31.013645+0000","last_active":"2024-03-04T09:26:31.014946+0000","last_peered":"2024-03-04T09:26:31.014946+0000","last_clean":"2024-03-04T09:26:31.014946+0000","last_became_active":"2024-03-04T09:26:31.013411+0000","last_became_peered":"2024-03-04T09:26:31.013411+0000","last_unstale":"2024-03-04T09:26:31.014946+0000","last_undegraded":"2024-03-04T09:26:31.014946+0000","last_fullsized":"2024-03-04T09:26:31.014946+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:07:15.106849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"32'1","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:31.017614+0000","last_change":"2024-03-04T09:26:29.996529+0000","last_active":"2024-03-04T09:26:31.017614+0000","last_peered":"2024-03-04T09:26:31.017614+0000","last_clean":"2024-03-04T09:26:31.017614+0000","last_became_active":"2024-03-04T09:26:29.996238+0000","last_became_peered":"2024-03-04T09:26:29.996238+0000","last_unstale":"2024-03-04T09:26:31.017614+0000","last_undegraded":"2024-03-04T09:26:31.017614+0000","last_fullsized":"2024-03-04T09:26:31.017614+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_clean_scrub_stamp":"2024-03-04T09:26:28.937043+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:53:31.025639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:31.020533+0000","last_change":"2024-03-04T09:26:31.007279+0000","last_active":"2024-03-04T09:26:31.020533+0000","last_peered":"2024-03-04T09:26:31.020533+0000","last_clean":"2024-03-04T09:26:31.020533+0000","last_became_active":"2024-03-04T09:26:31.006978+0000","last_became_peered":"2024-03-04T09:26:31.006978+0000","last_unstale":"2024-03-04T09:26:31.020533+0000","last_undegraded":"2024-03-04T09:26:31.020533+0000","last_fullsized":"2024-03-04T09:26:31.020533+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:48:31.810546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"32'1","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:31.005668+0000","last_change":"2024-03-04T09:26:29.993112+0000","last_active":"2024-03-04T09:26:31.005668+0000","last_peered":"2024-03-04T09:26:31.005668+0000","last_clean":"2024-03-04T09:26:31.005668+0000","last_became_active":"2024-03-04T09:26:29.992487+0000","last_became_peered":"2024-03-04T09:26:29.992487+0000","last_unstale":"2024-03-04T09:26:31.005668+0000","last_undegraded":"2024-03-04T09:26:31.005668+0000","last_fullsized":"2024-03-04T09:26:31.005668+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_clean_scrub_stamp":"2024-03-04T09:26:28.937043+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:17:15.944991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:31.005677+0000","last_change":"2024-03-04T09:26:31.003867+0000","last_active":"2024-03-04T09:26:31.005677+0000","last_peered":"2024-03-04T09:26:31.005677+0000","last_clean":"2024-03-04T09:26:31.005677+0000","last_became_active":"2024-03-04T09:26:31.003586+0000","last_became_peered":"2024-03-04T09:26:31.003586+0000","last_unstale":"2024-03-04T09:26:31.005677+0000","last_undegraded":"2024-03-04T09:26:31.005677+0000","last_fullsized":"2024-03-04T09:26:31.005677+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:24:35.905855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:31.001072+0000","last_change":"2024-03-04T09:26:29.989827+0000","last_active":"2024-03-04T09:26:31.001072+0000","last_peered":"2024-03-04T09:26:31.001072+0000","last_clean":"2024-03-04T09:26:31.001072+0000","last_became_active":"2024-03-04T09:26:29.988240+0000","last_became_peered":"2024-03-04T09:26:29.988240+0000","last_unstale":"2024-03-04T09:26:31.001072+0000","last_undegraded":"2024-03-04T09:26:31.001072+0000","last_fullsized":"2024-03-04T09:26:31.001072+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_clean_scrub_stamp":"2024-03-04T09:26:28.937043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:47:08.730106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-04T09:26:32.009034+0000","last_change":"2024-03-04T09:26:30.997463+0000","last_active":"2024-03-04T09:26:32.009034+0000","last_peered":"2024-03-04T09:26:32.009034+0000","last_clean":"2024-03-04T09:26:32.009034+0000","last_became_active":"2024-03-04T09:26:30.997071+0000","last_became_peered":"2024-03-04T09:26:30.997071+0000","last_unstale":"2024-03-04T09:26:32.009034+0000","last_undegraded":"2024-03-04T09:26:32.009034+0000","last_fullsized":"2024-03-04T09:26:32.009034+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:40:19.971433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:30.989499+0000","last_change":"2024-03-04T09:26:29.995006+0000","last_active":"2024-03-04T09:26:30.989499+0000","last_peered":"2024-03-04T09:26:30.989499+0000","last_clean":"2024-03-04T09:26:30.989499+0000","last_became_active":"2024-03-04T09:26:29.994675+0000","last_became_peered":"2024-03-04T09:26:29.994675+0000","last_unstale":"2024-03-04T09:26:30.989499+0000","last_undegraded":"2024-03-04T09:26:30.989499+0000","last_fullsized":"2024-03-04T09:26:30.989499+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_clean_scrub_stamp":"2024-03-04T09:26:28.937043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:34:55.432393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:31.024549+0000","last_change":"2024-03-04T09:26:31.022492+0000","last_active":"2024-03-04T09:26:31.024549+0000","last_peered":"2024-03-04T09:26:31.024549+0000","last_clean":"2024-03-04T09:26:31.024549+0000","last_became_active":"2024-03-04T09:26:31.022114+0000","last_became_peered":"2024-03-04T09:26:31.022114+0000","last_unstale":"2024-03-04T09:26:31.024549+0000","last_undegraded":"2024-03-04T09:26:31.024549+0000","last_fullsized":"2024-03-04T09:26:31.024549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:21:26.223848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:31.006712+0000","last_change":"2024-03-04T09:26:30.000770+0000","last_active":"2024-03-04T09:26:31.006712+0000","last_peered":"2024-03-04T09:26:31.006712+0000","last_clean":"2024-03-04T09:26:31.006712+0000","last_became_active":"2024-03-04T09:26:30.000511+0000","last_became_peered":"2024-03-04T09:26:30.000511+0000","last_unstale":"2024-03-04T09:26:31.006712+0000","last_undegraded":"2024-03-04T09:26:31.006712+0000","last_fullsized":"2024-03-04T09:26:31.006712+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_clean_scrub_stamp":"2024-03-04T09:26:28.937043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:57:22.379284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:31.025243+0000","last_change":"2024-03-04T09:26:31.020118+0000","last_active":"2024-03-04T09:26:31.025243+0000","last_peered":"2024-03-04T09:26:31.025243+0000","last_clean":"2024-03-04T09:26:31.025243+0000","last_became_active":"2024-03-04T09:26:31.019801+0000","last_became_peered":"2024-03-04T09:26:31.019801+0000","last_unstale":"2024-03-04T09:26:31.025243+0000","last_undegraded":"2024-03-04T09:26:31.025243+0000","last_fullsized":"2024-03-04T09:26:31.025243+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:39:30.925100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-04T09:26:32.015276+0000","last_change":"2024-03-04T09:26:29.993674+0000","last_active":"2024-03-04T09:26:32.015276+0000","last_peered":"2024-03-04T09:26:32.015276+0000","last_clean":"2024-03-04T09:26:32.015276+0000","last_became_active":"2024-03-04T09:26:29.993123+0000","last_became_peered":"2024-03-04T09:26:29.993123+0000","last_unstale":"2024-03-04T09:26:32.015276+0000","last_undegraded":"2024-03-04T09:26:32.015276+0000","last_fullsized":"2024-03-04T09:26:32.015276+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_clean_scrub_stamp":"2024-03-04T09:26:28.937043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:29:28.174031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:31.028088+0000","last_change":"2024-03-04T09:26:31.023762+0000","last_active":"2024-03-04T09:26:31.028088+0000","last_peered":"2024-03-04T09:26:31.028088+0000","last_clean":"2024-03-04T09:26:31.028088+0000","last_became_active":"2024-03-04T09:26:31.023415+0000","last_became_peered":"2024-03-04T09:26:31.023415+0000","last_unstale":"2024-03-04T09:26:31.028088+0000","last_undegraded":"2024-03-04T09:26:31.028088+0000","last_fullsized":"2024-03-04T09:26:31.028088+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:22:08.954238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"32'3","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:31.023420+0000","last_change":"2024-03-04T09:26:29.993093+0000","last_active":"2024-03-04T09:26:31.023420+0000","last_peered":"2024-03-04T09:26:31.023420+0000","last_clean":"2024-03-04T09:26:31.023420+0000","last_became_active":"2024-03-04T09:26:29.991775+0000","last_became_peered":"2024-03-04T09:26:29.991775+0000","last_unstale":"2024-03-04T09:26:31.023420+0000","last_undegraded":"2024-03-04T09:26:31.023420+0000","last_fullsized":"2024-03-04T09:26:31.023420+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_clean_scrub_stamp":"2024-03-04T09:26:28.937043+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:37:27.335724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:31.020585+0000","last_change":"2024-03-04T09:26:31.009318+0000","last_active":"2024-03-04T09:26:31.020585+0000","last_peered":"2024-03-04T09:26:31.020585+0000","last_clean":"2024-03-04T09:26:31.020585+0000","last_became_active":"2024-03-04T09:26:31.009024+0000","last_became_peered":"2024-03-04T09:26:31.009024+0000","last_unstale":"2024-03-04T09:26:31.020585+0000","last_undegraded":"2024-03-04T09:26:31.020585+0000","last_fullsized":"2024-03-04T09:26:31.020585+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:15:35.966281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"32'6","reported_seq":18,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:31.032230+0000","last_change":"2024-03-04T09:26:29.990120+0000","last_active":"2024-03-04T09:26:31.032230+0000","last_peered":"2024-03-04T09:26:31.032230+0000","last_clean":"2024-03-04T09:26:31.032230+0000","last_became_active":"2024-03-04T09:26:29.989719+0000","last_became_peered":"2024-03-04T09:26:29.989719+0000","last_unstale":"2024-03-04T09:26:31.032230+0000","last_undegraded":"2024-03-04T09:26:31.032230+0000","last_fullsized":"2024-03-04T09:26:31.032230+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_clean_scrub_stamp":"2024-03-04T09:26:28.937043+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:23:14.675708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-04T09:26:32.009038+0000","last_change":"2024-03-04T09:26:30.993692+0000","last_active":"2024-03-04T09:26:32.009038+0000","last_peered":"2024-03-04T09:26:32.009038+0000","last_clean":"2024-03-04T09:26:32.009038+0000","last_became_active":"2024-03-04T09:26:30.993346+0000","last_became_peered":"2024-03-04T09:26:30.993346+0000","last_unstale":"2024-03-04T09:26:32.009038+0000","last_undegraded":"2024-03-04T09:26:32.009038+0000","last_fullsized":"2024-03-04T09:26:32.009038+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:48:31.810546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"32'2","reported_seq":14,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:31.012675+0000","last_change":"2024-03-04T09:26:29.994702+0000","last_active":"2024-03-04T09:26:31.012675+0000","last_peered":"2024-03-04T09:26:31.012675+0000","last_clean":"2024-03-04T09:26:31.012675+0000","last_became_active":"2024-03-04T09:26:29.994384+0000","last_became_peered":"2024-03-04T09:26:29.994384+0000","last_unstale":"2024-03-04T09:26:31.012675+0000","last_undegraded":"2024-03-04T09:26:31.012675+0000","last_fullsized":"2024-03-04T09:26:31.012675+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_clean_scrub_stamp":"2024-03-04T09:26:28.937043+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:46:15.529293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:31.020518+0000","last_change":"2024-03-04T09:26:31.014894+0000","last_active":"2024-03-04T09:26:31.020518+0000","last_peered":"2024-03-04T09:26:31.020518+0000","last_clean":"2024-03-04T09:26:31.020518+0000","last_became_active":"2024-03-04T09:26:31.014592+0000","last_became_peered":"2024-03-04T09:26:31.014592+0000","last_unstale":"2024-03-04T09:26:31.020518+0000","last_undegraded":"2024-03-04T09:26:31.020518+0000","last_fullsized":"2024-03-04T09:26:31.020518+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:41:30.039171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-04T09:26:32.009169+0000","last_change":"2024-03-04T09:26:29.991792+0000","last_active":"2024-03-04T09:26:32.009169+0000","last_peered":"2024-03-04T09:26:32.009169+0000","last_clean":"2024-03-04T09:26:32.009169+0000","last_became_active":"2024-03-04T09:26:29.991495+0000","last_became_peered":"2024-03-04T09:26:29.991495+0000","last_unstale":"2024-03-04T09:26:32.009169+0000","last_undegraded":"2024-03-04T09:26:32.009169+0000","last_fullsized":"2024-03-04T09:26:32.009169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_clean_scrub_stamp":"2024-03-04T09:26:28.937043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:53:31.025639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:31.014675+0000","last_change":"2024-03-04T09:26:31.012794+0000","last_active":"2024-03-04T09:26:31.014675+0000","last_peered":"2024-03-04T09:26:31.014675+0000","last_clean":"2024-03-04T09:26:31.014675+0000","last_became_active":"2024-03-04T09:26:31.012142+0000","last_became_peered":"2024-03-04T09:26:31.012142+0000","last_unstale":"2024-03-04T09:26:31.014675+0000","last_undegraded":"2024-03-04T09:26:31.014675+0000","last_fullsized":"2024-03-04T09:26:31.014675+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:12:02.940432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:31.002286+0000","last_change":"2024-03-04T09:26:29.998129+0000","last_active":"2024-03-04T09:26:31.002286+0000","last_peered":"2024-03-04T09:26:31.002286+0000","last_clean":"2024-03-04T09:26:31.002286+0000","last_became_active":"2024-03-04T09:26:29.997795+0000","last_became_peered":"2024-03-04T09:26:29.997795+0000","last_unstale":"2024-03-04T09:26:31.002286+0000","last_undegraded":"2024-03-04T09:26:31.002286+0000","last_fullsized":"2024-03-04T09:26:31.002286+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_clean_scrub_stamp":"2024-03-04T09:26:28.937043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:37:27.335724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:31.007996+0000","last_change":"2024-03-04T09:26:31.003241+0000","last_active":"2024-03-04T09:26:31.007996+0000","last_peered":"2024-03-04T09:26:31.007996+0000","last_clean":"2024-03-04T09:26:31.007996+0000","last_became_active":"2024-03-04T09:26:31.002791+0000","last_became_peered":"2024-03-04T09:26:31.002791+0000","last_unstale":"2024-03-04T09:26:31.007996+0000","last_undegraded":"2024-03-04T09:26:31.007996+0000","last_fullsized":"2024-03-04T09:26:31.007996+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:53:32.040036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"32'8","reported_seq":20,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:31.030650+0000","last_change":"2024-03-04T09:26:29.985221+0000","last_active":"2024-03-04T09:26:31.030650+0000","last_peered":"2024-03-04T09:26:31.030650+0000","last_clean":"2024-03-04T09:26:31.030650+0000","last_became_active":"2024-03-04T09:26:29.983987+0000","last_became_peered":"2024-03-04T09:26:29.983987+0000","last_unstale":"2024-03-04T09:26:31.030650+0000","last_undegraded":"2024-03-04T09:26:31.030650+0000","last_fullsized":"2024-03-04T09:26:31.030650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_clean_scrub_stamp":"2024-03-04T09:26:28.937043+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:12:01.926034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:31.002733+0000","last_change":"2024-03-04T09:26:31.000056+0000","last_active":"2024-03-04T09:26:31.002733+0000","last_peered":"2024-03-04T09:26:31.002733+0000","last_clean":"2024-03-04T09:26:31.002733+0000","last_became_active":"2024-03-04T09:26:30.999711+0000","last_became_peered":"2024-03-04T09:26:30.999711+0000","last_unstale":"2024-03-04T09:26:31.002733+0000","last_undegraded":"2024-03-04T09:26:31.002733+0000","last_fullsized":"2024-03-04T09:26:31.002733+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:46:49.122698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"32'1","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-04T09:26:32.014055+0000","last_change":"2024-03-04T09:26:29.992377+0000","last_active":"2024-03-04T09:26:32.014055+0000","last_peered":"2024-03-04T09:26:32.014055+0000","last_clean":"2024-03-04T09:26:32.014055+0000","last_became_active":"2024-03-04T09:26:29.992090+0000","last_became_peered":"2024-03-04T09:26:29.992090+0000","last_unstale":"2024-03-04T09:26:32.014055+0000","last_undegraded":"2024-03-04T09:26:32.014055+0000","last_fullsized":"2024-03-04T09:26:32.014055+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_clean_scrub_stamp":"2024-03-04T09:26:28.937043+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:46:48.108301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:31.019814+0000","last_change":"2024-03-04T09:26:30.999018+0000","last_active":"2024-03-04T09:26:31.019814+0000","last_peered":"2024-03-04T09:26:31.019814+0000","last_clean":"2024-03-04T09:26:31.019814+0000","last_became_active":"2024-03-04T09:26:30.998702+0000","last_became_peered":"2024-03-04T09:26:30.998702+0000","last_unstale":"2024-03-04T09:26:31.019814+0000","last_undegraded":"2024-03-04T09:26:31.019814+0000","last_fullsized":"2024-03-04T09:26:31.019814+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:07:15.106849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"32'2","reported_seq":14,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:31.007350+0000","last_change":"2024-03-04T09:26:29.998254+0000","last_active":"2024-03-04T09:26:31.007350+0000","last_peered":"2024-03-04T09:26:31.007350+0000","last_clean":"2024-03-04T09:26:31.007350+0000","last_became_active":"2024-03-04T09:26:29.997760+0000","last_became_peered":"2024-03-04T09:26:29.997760+0000","last_unstale":"2024-03-04T09:26:31.007350+0000","last_undegraded":"2024-03-04T09:26:31.007350+0000","last_fullsized":"2024-03-04T09:26:31.007350+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_clean_scrub_stamp":"2024-03-04T09:26:28.937043+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:29:28.174031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-04T09:26:32.009936+0000","last_change":"2024-03-04T09:26:30.997958+0000","last_active":"2024-03-04T09:26:32.009936+0000","last_peered":"2024-03-04T09:26:32.009936+0000","last_clean":"2024-03-04T09:26:32.009936+0000","last_became_active":"2024-03-04T09:26:30.997670+0000","last_became_peered":"2024-03-04T09:26:30.997670+0000","last_unstale":"2024-03-04T09:26:32.009936+0000","last_undegraded":"2024-03-04T09:26:32.009936+0000","last_fullsized":"2024-03-04T09:26:32.009936+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:42:09.417392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"32'1","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:31.010145+0000","last_change":"2024-03-04T09:26:29.994556+0000","last_active":"2024-03-04T09:26:31.010145+0000","last_peered":"2024-03-04T09:26:31.010145+0000","last_clean":"2024-03-04T09:26:31.010145+0000","last_became_active":"2024-03-04T09:26:29.994249+0000","last_became_peered":"2024-03-04T09:26:29.994249+0000","last_unstale":"2024-03-04T09:26:31.010145+0000","last_undegraded":"2024-03-04T09:26:31.010145+0000","last_fullsized":"2024-03-04T09:26:31.010145+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_clean_scrub_stamp":"2024-03-04T09:26:28.937043+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:35:07.872187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-04T09:26:32.014099+0000","last_change":"2024-03-04T09:26:31.016683+0000","last_active":"2024-03-04T09:26:32.014099+0000","last_peered":"2024-03-04T09:26:32.014099+0000","last_clean":"2024-03-04T09:26:32.014099+0000","last_became_active":"2024-03-04T09:26:31.016460+0000","last_became_peered":"2024-03-04T09:26:31.016460+0000","last_unstale":"2024-03-04T09:26:32.014099+0000","last_undegraded":"2024-03-04T09:26:32.014099+0000","last_fullsized":"2024-03-04T09:26:32.014099+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:33:02.297144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:30.994368+0000","last_change":"2024-03-04T09:26:29.996666+0000","last_active":"2024-03-04T09:26:30.994368+0000","last_peered":"2024-03-04T09:26:30.994368+0000","last_clean":"2024-03-04T09:26:30.994368+0000","last_became_active":"2024-03-04T09:26:29.996377+0000","last_became_peered":"2024-03-04T09:26:29.996377+0000","last_unstale":"2024-03-04T09:26:30.994368+0000","last_undegraded":"2024-03-04T09:26:30.994368+0000","last_fullsized":"2024-03-04T09:26:30.994368+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_clean_scrub_stamp":"2024-03-04T09:26:28.937043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:43:35.091061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-04T09:26:32.010949+0000","last_change":"2024-03-04T09:26:31.010839+0000","last_active":"2024-03-04T09:26:32.010949+0000","last_peered":"2024-03-04T09:26:32.010949+0000","last_clean":"2024-03-04T09:26:32.010949+0000","last_became_active":"2024-03-04T09:26:31.010577+0000","last_became_peered":"2024-03-04T09:26:31.010577+0000","last_unstale":"2024-03-04T09:26:32.010949+0000","last_undegraded":"2024-03-04T09:26:32.010949+0000","last_fullsized":"2024-03-04T09:26:32.010949+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:43:36.105459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"32'2","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-04T09:26:32.014277+0000","last_change":"2024-03-04T09:26:29.989932+0000","last_active":"2024-03-04T09:26:32.014277+0000","last_peered":"2024-03-04T09:26:32.014277+0000","last_clean":"2024-03-04T09:26:32.014277+0000","last_became_active":"2024-03-04T09:26:29.988395+0000","last_became_peered":"2024-03-04T09:26:29.988395+0000","last_unstale":"2024-03-04T09:26:32.014277+0000","last_undegraded":"2024-03-04T09:26:32.014277+0000","last_fullsized":"2024-03-04T09:26:32.014277+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_clean_scrub_stamp":"2024-03-04T09:26:28.937043+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:43:35.091061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-04T09:26:32.013080+0000","last_change":"2024-03-04T09:26:31.002635+0000","last_active":"2024-03-04T09:26:32.013080+0000","last_peered":"2024-03-04T09:26:32.013080+0000","last_clean":"2024-03-04T09:26:32.013080+0000","last_became_active":"2024-03-04T09:26:31.002322+0000","last_became_peered":"2024-03-04T09:26:31.002322+0000","last_unstale":"2024-03-04T09:26:32.013080+0000","last_undegraded":"2024-03-04T09:26:32.013080+0000","last_fullsized":"2024-03-04T09:26:32.013080+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:46:49.122698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:30.993814+0000","last_change":"2024-03-04T09:26:29.996899+0000","last_active":"2024-03-04T09:26:30.993814+0000","last_peered":"2024-03-04T09:26:30.993814+0000","last_clean":"2024-03-04T09:26:30.993814+0000","last_became_active":"2024-03-04T09:26:29.996530+0000","last_became_peered":"2024-03-04T09:26:29.996530+0000","last_unstale":"2024-03-04T09:26:30.993814+0000","last_undegraded":"2024-03-04T09:26:30.993814+0000","last_fullsized":"2024-03-04T09:26:30.993814+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_clean_scrub_stamp":"2024-03-04T09:26:28.937043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:43:35.091061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-04T09:26:32.014476+0000","last_change":"2024-03-04T09:26:31.008740+0000","last_active":"2024-03-04T09:26:32.014476+0000","last_peered":"2024-03-04T09:26:32.014476+0000","last_clean":"2024-03-04T09:26:32.014476+0000","last_became_active":"2024-03-04T09:26:31.008426+0000","last_became_peered":"2024-03-04T09:26:31.008426+0000","last_unstale":"2024-03-04T09:26:32.014476+0000","last_undegraded":"2024-03-04T09:26:32.014476+0000","last_fullsized":"2024-03-04T09:26:32.014476+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:24:35.905855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:30.989129+0000","last_change":"2024-03-04T09:26:29.995289+0000","last_active":"2024-03-04T09:26:30.989129+0000","last_peered":"2024-03-04T09:26:30.989129+0000","last_clean":"2024-03-04T09:26:30.989129+0000","last_became_active":"2024-03-04T09:26:29.994796+0000","last_became_peered":"2024-03-04T09:26:29.994796+0000","last_unstale":"2024-03-04T09:26:30.989129+0000","last_undegraded":"2024-03-04T09:26:30.989129+0000","last_fullsized":"2024-03-04T09:26:30.989129+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_clean_scrub_stamp":"2024-03-04T09:26:28.937043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:07:50.874217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:31.014522+0000","last_change":"2024-03-04T09:26:31.010860+0000","last_active":"2024-03-04T09:26:31.014522+0000","last_peered":"2024-03-04T09:26:31.014522+0000","last_clean":"2024-03-04T09:26:31.014522+0000","last_became_active":"2024-03-04T09:26:31.010548+0000","last_became_peered":"2024-03-04T09:26:31.010548+0000","last_unstale":"2024-03-04T09:26:31.014522+0000","last_undegraded":"2024-03-04T09:26:31.014522+0000","last_fullsized":"2024-03-04T09:26:31.014522+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:25:51.053806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:30.983219+0000","last_change":"2024-03-04T09:26:29.998310+0000","last_active":"2024-03-04T09:26:30.983219+0000","last_peered":"2024-03-04T09:26:30.983219+0000","last_clean":"2024-03-04T09:26:30.983219+0000","last_became_active":"2024-03-04T09:26:29.998007+0000","last_became_peered":"2024-03-04T09:26:29.998007+0000","last_unstale":"2024-03-04T09:26:30.983219+0000","last_undegraded":"2024-03-04T09:26:30.983219+0000","last_fullsized":"2024-03-04T09:26:30.983219+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_clean_scrub_stamp":"2024-03-04T09:26:28.937043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:12:01.926034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-04T09:26:32.011226+0000","last_change":"2024-03-04T09:26:31.001422+0000","last_active":"2024-03-04T09:26:32.011226+0000","last_peered":"2024-03-04T09:26:32.011226+0000","last_clean":"2024-03-04T09:26:32.011226+0000","last_became_active":"2024-03-04T09:26:31.000477+0000","last_became_peered":"2024-03-04T09:26:31.000477+0000","last_unstale":"2024-03-04T09:26:32.011226+0000","last_undegraded":"2024-03-04T09:26:32.011226+0000","last_fullsized":"2024-03-04T09:26:32.011226+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:53:32.040036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"32'2","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-04T09:26:32.014194+0000","last_change":"2024-03-04T09:26:29.992590+0000","last_active":"2024-03-04T09:26:32.014194+0000","last_peered":"2024-03-04T09:26:32.014194+0000","last_clean":"2024-03-04T09:26:32.014194+0000","last_became_active":"2024-03-04T09:26:29.991132+0000","last_became_peered":"2024-03-04T09:26:29.991132+0000","last_unstale":"2024-03-04T09:26:32.014194+0000","last_undegraded":"2024-03-04T09:26:32.014194+0000","last_fullsized":"2024-03-04T09:26:32.014194+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_clean_scrub_stamp":"2024-03-04T09:26:28.937043+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:33:01.282746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:31.020231+0000","last_change":"2024-03-04T09:26:31.007703+0000","last_active":"2024-03-04T09:26:31.020231+0000","last_peered":"2024-03-04T09:26:31.020231+0000","last_clean":"2024-03-04T09:26:31.020231+0000","last_became_active":"2024-03-04T09:26:31.007126+0000","last_became_peered":"2024-03-04T09:26:31.007126+0000","last_unstale":"2024-03-04T09:26:31.020231+0000","last_undegraded":"2024-03-04T09:26:31.020231+0000","last_fullsized":"2024-03-04T09:26:31.020231+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:40:19.971433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"32'1","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-04T09:26:32.015245+0000","last_change":"2024-03-04T09:26:29.983841+0000","last_active":"2024-03-04T09:26:32.015245+0000","last_peered":"2024-03-04T09:26:32.015245+0000","last_clean":"2024-03-04T09:26:32.015245+0000","last_became_active":"2024-03-04T09:26:29.983498+0000","last_became_peered":"2024-03-04T09:26:29.983498+0000","last_unstale":"2024-03-04T09:26:32.015245+0000","last_undegraded":"2024-03-04T09:26:32.015245+0000","last_fullsized":"2024-03-04T09:26:32.015245+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_clean_scrub_stamp":"2024-03-04T09:26:28.937043+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:53:31.025639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-04T09:26:32.013885+0000","last_change":"2024-03-04T09:26:31.010663+0000","last_active":"2024-03-04T09:26:32.013885+0000","last_peered":"2024-03-04T09:26:32.013885+0000","last_clean":"2024-03-04T09:26:32.013885+0000","last_became_active":"2024-03-04T09:26:31.008387+0000","last_became_peered":"2024-03-04T09:26:31.008387+0000","last_unstale":"2024-03-04T09:26:32.013885+0000","last_undegraded":"2024-03-04T09:26:32.013885+0000","last_fullsized":"2024-03-04T09:26:32.013885+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:07:15.106849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-04T09:26:32.010098+0000","last_change":"2024-03-04T09:26:29.979071+0000","last_active":"2024-03-04T09:26:32.010098+0000","last_peered":"2024-03-04T09:26:32.010098+0000","last_clean":"2024-03-04T09:26:32.010098+0000","last_became_active":"2024-03-04T09:26:29.977521+0000","last_became_peered":"2024-03-04T09:26:29.977521+0000","last_unstale":"2024-03-04T09:26:32.010098+0000","last_undegraded":"2024-03-04T09:26:32.010098+0000","last_fullsized":"2024-03-04T09:26:32.010098+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_clean_scrub_stamp":"2024-03-04T09:26:28.937043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:29:28.174031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:31.008059+0000","last_change":"2024-03-04T09:26:31.006312+0000","last_active":"2024-03-04T09:26:31.008059+0000","last_peered":"2024-03-04T09:26:31.008059+0000","last_clean":"2024-03-04T09:26:31.008059+0000","last_became_active":"2024-03-04T09:26:31.005764+0000","last_became_peered":"2024-03-04T09:26:31.005764+0000","last_unstale":"2024-03-04T09:26:31.008059+0000","last_undegraded":"2024-03-04T09:26:31.008059+0000","last_fullsized":"2024-03-04T09:26:31.008059+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:33:02.297144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-04T09:26:32.014467+0000","last_change":"2024-03-04T09:26:29.990486+0000","last_active":"2024-03-04T09:26:32.014467+0000","last_peered":"2024-03-04T09:26:32.014467+0000","last_clean":"2024-03-04T09:26:32.014467+0000","last_became_active":"2024-03-04T09:26:29.990286+0000","last_became_peered":"2024-03-04T09:26:29.990286+0000","last_unstale":"2024-03-04T09:26:32.014467+0000","last_undegraded":"2024-03-04T09:26:32.014467+0000","last_fullsized":"2024-03-04T09:26:32.014467+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_clean_scrub_stamp":"2024-03-04T09:26:28.937043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:35:07.872187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-04T09:26:32.014152+0000","last_change":"2024-03-04T09:26:30.998907+0000","last_active":"2024-03-04T09:26:32.014152+0000","last_peered":"2024-03-04T09:26:32.014152+0000","last_clean":"2024-03-04T09:26:32.014152+0000","last_became_active":"2024-03-04T09:26:30.998587+0000","last_became_peered":"2024-03-04T09:26:30.998587+0000","last_unstale":"2024-03-04T09:26:32.014152+0000","last_undegraded":"2024-03-04T09:26:32.014152+0000","last_fullsized":"2024-03-04T09:26:32.014152+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:23:15.690106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-04T09:26:32.009297+0000","last_change":"2024-03-04T09:26:29.994388+0000","last_active":"2024-03-04T09:26:32.009297+0000","last_peered":"2024-03-04T09:26:32.009297+0000","last_clean":"2024-03-04T09:26:32.009297+0000","last_became_active":"2024-03-04T09:26:29.994122+0000","last_became_peered":"2024-03-04T09:26:29.994122+0000","last_unstale":"2024-03-04T09:26:32.009297+0000","last_undegraded":"2024-03-04T09:26:32.009297+0000","last_fullsized":"2024-03-04T09:26:32.009297+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_clean_scrub_stamp":"2024-03-04T09:26:28.937043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:35:07.872187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:31.022076+0000","last_change":"2024-03-04T09:26:31.012143+0000","last_active":"2024-03-04T09:26:31.022076+0000","last_peered":"2024-03-04T09:26:31.022076+0000","last_clean":"2024-03-04T09:26:31.022076+0000","last_became_active":"2024-03-04T09:26:31.011832+0000","last_became_peered":"2024-03-04T09:26:31.011832+0000","last_unstale":"2024-03-04T09:26:31.022076+0000","last_undegraded":"2024-03-04T09:26:31.022076+0000","last_fullsized":"2024-03-04T09:26:31.022076+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:15:35.966281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:30.989798+0000","last_change":"2024-03-04T09:26:29.993931+0000","last_active":"2024-03-04T09:26:30.989798+0000","last_peered":"2024-03-04T09:26:30.989798+0000","last_clean":"2024-03-04T09:26:30.989798+0000","last_became_active":"2024-03-04T09:26:29.993606+0000","last_became_peered":"2024-03-04T09:26:29.993606+0000","last_unstale":"2024-03-04T09:26:30.989798+0000","last_undegraded":"2024-03-04T09:26:30.989798+0000","last_fullsized":"2024-03-04T09:26:30.989798+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_clean_scrub_stamp":"2024-03-04T09:26:28.937043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:27:09.922042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:31.022368+0000","last_change":"2024-03-04T09:26:31.019418+0000","last_active":"2024-03-04T09:26:31.022368+0000","last_peered":"2024-03-04T09:26:31.022368+0000","last_clean":"2024-03-04T09:26:31.022368+0000","last_became_active":"2024-03-04T09:26:31.018863+0000","last_became_peered":"2024-03-04T09:26:31.018863+0000","last_unstale":"2024-03-04T09:26:31.022368+0000","last_undegraded":"2024-03-04T09:26:31.022368+0000","last_fullsized":"2024-03-04T09:26:31.022368+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:07:15.106849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:31.001044+0000","last_change":"2024-03-04T09:26:29.998156+0000","last_active":"2024-03-04T09:26:31.001044+0000","last_peered":"2024-03-04T09:26:31.001044+0000","last_clean":"2024-03-04T09:26:31.001044+0000","last_became_active":"2024-03-04T09:26:29.997875+0000","last_became_peered":"2024-03-04T09:26:29.997875+0000","last_unstale":"2024-03-04T09:26:31.001044+0000","last_undegraded":"2024-03-04T09:26:31.001044+0000","last_fullsized":"2024-03-04T09:26:31.001044+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_clean_scrub_stamp":"2024-03-04T09:26:28.937043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:25:50.039408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:31.014488+0000","last_change":"2024-03-04T09:26:31.010720+0000","last_active":"2024-03-04T09:26:31.014488+0000","last_peered":"2024-03-04T09:26:31.014488+0000","last_clean":"2024-03-04T09:26:31.014488+0000","last_became_active":"2024-03-04T09:26:31.010429+0000","last_became_peered":"2024-03-04T09:26:31.010429+0000","last_unstale":"2024-03-04T09:26:31.014488+0000","last_undegraded":"2024-03-04T09:26:31.014488+0000","last_fullsized":"2024-03-04T09:26:31.014488+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:29.951441+0000","last_clean_scrub_stamp":"2024-03-04T09:26:29.951441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:24:35.905855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:30.990914+0000","last_change":"2024-03-04T09:26:30.000139+0000","last_active":"2024-03-04T09:26:30.990914+0000","last_peered":"2024-03-04T09:26:30.990914+0000","last_clean":"2024-03-04T09:26:30.990914+0000","last_became_active":"2024-03-04T09:26:29.999809+0000","last_became_peered":"2024-03-04T09:26:29.999809+0000","last_unstale":"2024-03-04T09:26:30.990914+0000","last_undegraded":"2024-03-04T09:26:30.990914+0000","last_fullsized":"2024-03-04T09:26:30.990914+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:26:28.937043+0000","last_clean_scrub_stamp":"2024-03-04T09:26:28.937043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:27:09.922042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":192,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T09:26:30.991687+0000","last_change":"2024-03-04T09:25:09.316159+0000","last_active":"2024-03-04T09:26:30.991687+0000","last_peered":"2024-03-04T09:26:30.991687+0000","last_clean":"2024-03-04T09:26:30.991687+0000","last_became_active":"2024-03-04T09:25:09.313767+0000","last_became_peered":"2024-03-04T09:25:09.313767+0000","last_unstale":"2024-03-04T09:26:30.991687+0000","last_undegraded":"2024-03-04T09:26:30.991687+0000","last_fullsized":"2024-03-04T09:26:30.991687+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:08.110953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:08.110953+0000","last_clean_scrub_stamp":"2024-03-04T09:25:08.110953+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:34:21.745390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574868,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27964,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343876,"statfs":{"total":96636764160,"available":96608129024,"internally_reserved":0,"allocated":958464,"data_stored":731014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574868,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27384,"kb_used_data":360,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344456,"statfs":{"total":96636764160,"available":96608722944,"internally_reserved":0,"allocated":368640,"data_stored":141172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574870,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27388,"kb_used_data":360,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344452,"statfs":{"total":96636764160,"available":96608718848,"internally_reserved":0,"allocated":368640,"data_stored":141228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574867,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27384,"kb_used_data":360,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344456,"statfs":{"total":96636764160,"available":96608722944,"internally_reserved":0,"allocated":368640,"data_stored":140660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574869,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27396,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344444,"statfs":{"total":96636764160,"available":96608710656,"internally_reserved":0,"allocated":376832,"data_stored":142190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574868,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27380,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344460,"statfs":{"total":96636764160,"available":96608727040,"internally_reserved":0,"allocated":364544,"data_stored":140626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574869,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27408,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344432,"statfs":{"total":96636764160,"available":96608698368,"internally_reserved":0,"allocated":389120,"data_stored":148490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574868,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27964,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343876,"statfs":{"total":96636764160,"available":96608129024,"internally_reserved":0,"allocated":966656,"data_stored":737224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-04T09:26:36.012 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-04T09:26:36.013 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-04T09:26:36.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.285+0000 7fbdf0a00640 1 Processor -- start 2024-03-04T09:26:36.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.285+0000 7fbdf0a00640 1 -- start start 2024-03-04T09:26:36.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.286+0000 7fbdf0a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fbdec145570 0x7fbdec0cef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:36.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.286+0000 7fbdf0a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdec0b4ae0 con 0x7fbdec0b42c0 2024-03-04T09:26:36.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.286+0000 7fbdf0a00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdec0b4dc0 con 0x7fbdec0cd430 2024-03-04T09:26:36.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.286+0000 7fbdf0a00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbdec06c9f0 con 0x7fbdec145570 2024-03-04T09:26:36.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.286+0000 7fbdeb400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fbdec145570 0x7fbdec0cef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:36.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.287+0000 7fbdeb400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fbdec145570 0x7fbdec0cef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:40760/0 (socket says 172.21.15.6:40760) 2024-03-04T09:26:36.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.287+0000 7fbdeb400640 1 -- 172.21.15.6:0/3448344254 learned_addr learned my addr 172.21.15.6:0/3448344254 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:36.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.287+0000 7fbdeaa00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fbdec0b42c0 0x7fbdec0ccd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:37008/0 (socket says 172.21.15.6:37008) 2024-03-04T09:26:36.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.287+0000 7fbdeb400640 1 -- 172.21.15.6:0/3448344254 >> v1:172.21.15.146:6789/0 conn(0x7fbdec0cd430 legacy=0x7fbdec06d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:36.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.287+0000 7fbdeb400640 1 -- 172.21.15.6:0/3448344254 >> v1:172.21.15.6:6789/0 conn(0x7fbdec0b42c0 legacy=0x7fbdec0ccd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:36.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.287+0000 7fbdeb400640 1 -- 172.21.15.6:0/3448344254 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdec06cc40 con 0x7fbdec145570 2024-03-04T09:26:36.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.287+0000 7fbdea000640 1 -- 172.21.15.6:0/3448344254 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 446988429 0 0) 0x7fbdec0b4ae0 con 0x7fbdec0b42c0 2024-03-04T09:26:36.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.288+0000 7fbdeb400640 1 --2- 172.21.15.6:0/3448344254 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbdec145570 0x7fbdec0cef80 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fbddc009e50 tx=0x7fbddc02bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=128d74373f5c533a server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:36.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.288+0000 7fbdea000640 1 -- 172.21.15.6:0/3448344254 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbddc02cb20 con 0x7fbdec145570 2024-03-04T09:26:36.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.289+0000 7fbdea000640 1 -- 172.21.15.6:0/3448344254 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbddc02ccc0 con 0x7fbdec145570 2024-03-04T09:26:36.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.289+0000 7fbdea000640 1 -- 172.21.15.6:0/3448344254 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbddc034610 con 0x7fbdec145570 2024-03-04T09:26:36.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.289+0000 7fbdf0a00640 1 -- 172.21.15.6:0/3448344254 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbdec145570 msgr2=0x7fbdec0cef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:36.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.289+0000 7fbdf0a00640 1 --2- 172.21.15.6:0/3448344254 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbdec145570 0x7fbdec0cef80 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fbddc009e50 tx=0x7fbddc02bf80 comp rx=0 tx=0).stop 2024-03-04T09:26:36.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.289+0000 7fbdf0a00640 1 -- 172.21.15.6:0/3448344254 shutdown_connections 2024-03-04T09:26:36.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.289+0000 7fbdf0a00640 1 --2- 172.21.15.6:0/3448344254 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbdec145570 0x7fbdec0cef80 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:36.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.289+0000 7fbdf0a00640 1 -- 172.21.15.6:0/3448344254 >> 172.21.15.6:0/3448344254 conn(0x7fbdec06b190 msgr2=0x7fbdec06b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:36.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.289+0000 7fbdf0a00640 1 -- 172.21.15.6:0/3448344254 shutdown_connections 2024-03-04T09:26:36.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.289+0000 7fbdf0a00640 1 -- 172.21.15.6:0/3448344254 wait complete. 2024-03-04T09:26:36.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.291+0000 7fbdf0a00640 1 Processor -- start 2024-03-04T09:26:36.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.291+0000 7fbdf0a00640 1 -- start start 2024-03-04T09:26:36.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.292+0000 7fbdf0a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbdec0b42c0 0x7fbdec0adf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:36.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.292+0000 7fbdf0a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbdec0cd430 0x7fbdec1f17a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:36.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.292+0000 7fbdf0a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbdec145570 0x7fbdec0b1060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:36.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.292+0000 7fbdf0a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdec15c110 con 0x7fbdec145570 2024-03-04T09:26:36.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.292+0000 7fbdf0a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdec06cfb0 con 0x7fbdec0b42c0 2024-03-04T09:26:36.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.292+0000 7fbdf0a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdec06db50 con 0x7fbdec0cd430 2024-03-04T09:26:36.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.292+0000 7fbdebe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbdec145570 0x7fbdec0b1060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:36.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.293+0000 7fbdebe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbdec145570 0x7fbdec0b1060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:40772/0 (socket says 172.21.15.6:40772) 2024-03-04T09:26:36.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.293+0000 7fbdebe00640 1 -- 172.21.15.6:0/3755909531 learned_addr learned my addr 172.21.15.6:0/3755909531 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:36.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.293+0000 7fbdeb400640 1 --2- 172.21.15.6:0/3755909531 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbdec0b42c0 0x7fbdec0adf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:36.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.293+0000 7fbdebe00640 1 -- 172.21.15.6:0/3755909531 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbdec0cd430 msgr2=0x7fbdec1f17a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:36.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.293+0000 7fbdebe00640 1 --2- 172.21.15.6:0/3755909531 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbdec0cd430 0x7fbdec1f17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:36.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.293+0000 7fbdebe00640 1 -- 172.21.15.6:0/3755909531 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbdec0b42c0 msgr2=0x7fbdec0adf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:36.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.293+0000 7fbdebe00640 1 --2- 172.21.15.6:0/3755909531 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbdec0b42c0 0x7fbdec0adf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:36.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.293+0000 7fbdebe00640 1 -- 172.21.15.6:0/3755909531 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdec0ae4a0 con 0x7fbdec145570 2024-03-04T09:26:36.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.293+0000 7fbdebe00640 1 --2- 172.21.15.6:0/3755909531 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbdec145570 0x7fbdec0b1060 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fbde400c410 tx=0x7fbde400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:36.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.294+0000 7fbde3400640 1 -- 172.21.15.6:0/3755909531 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbde4011790 con 0x7fbdec145570 2024-03-04T09:26:36.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.294+0000 7fbde3400640 1 -- 172.21.15.6:0/3755909531 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbde4017070 con 0x7fbdec145570 2024-03-04T09:26:36.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.294+0000 7fbdf0a00640 1 -- 172.21.15.6:0/3755909531 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdec1f5a20 con 0x7fbdec145570 2024-03-04T09:26:36.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.294+0000 7fbde3400640 1 -- 172.21.15.6:0/3755909531 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbde4011b70 con 0x7fbdec145570 2024-03-04T09:26:36.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.294+0000 7fbdf0a00640 1 -- 172.21.15.6:0/3755909531 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdec15c710 con 0x7fbdec145570 2024-03-04T09:26:36.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.295+0000 7fbdf0a00640 1 -- 172.21.15.6:0/3755909531 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdec04f330 con 0x7fbdec145570 2024-03-04T09:26:36.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.296+0000 7fbde3400640 1 -- 172.21.15.6:0/3755909531 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbde4011d10 con 0x7fbdec145570 2024-03-04T09:26:36.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.297+0000 7fbde3400640 1 --2- 172.21.15.6:0/3755909531 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fbdcc06b4d0 0x7fbdcc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:36.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.297+0000 7fbde3400640 1 -- 172.21.15.6:0/3755909531 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbde4089fe0 con 0x7fbdec145570 2024-03-04T09:26:36.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.298+0000 7fbdeb400640 1 --2- 172.21.15.6:0/3755909531 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fbdcc06b4d0 0x7fbdcc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:36.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.299+0000 7fbdeb400640 1 --2- 172.21.15.6:0/3755909531 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fbdcc06b4d0 0x7fbdcc06d990 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fbddc035000 tx=0x7fbddc030000 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:36.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.301+0000 7fbde3400640 1 -- 172.21.15.6:0/3755909531 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbde4058e00 con 0x7fbdec145570 2024-03-04T09:26:36.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.578+0000 7fbdf0a00640 1 -- 172.21.15.6:0/3755909531 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fbdec0cc590 con 0x7fbdec145570 2024-03-04T09:26:36.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.582+0000 7fbde3400640 1 -- 172.21.15.6:0/3755909531 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v10) v1 ==== 190+0+147 (secure 0 0 0) 0x7fbde4058700 con 0x7fbdec145570 2024-03-04T09:26:36.579 INFO:teuthology.orchestra.run.smithi006.stdout:[client.0] 2024-03-04T09:26:36.579 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQA9k+Vl4L+rJxAACaDbQ4/Bybp5hAZAnJzlVw== 2024-03-04T09:26:36.579 INFO:teuthology.orchestra.run.smithi006.stdout: caps mds = "allow" 2024-03-04T09:26:36.579 INFO:teuthology.orchestra.run.smithi006.stdout: caps mon = "allow r" 2024-03-04T09:26:36.579 INFO:teuthology.orchestra.run.smithi006.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-04T09:26:36.579 INFO:teuthology.orchestra.run.smithi006.stderr:updated caps for client.0 2024-03-04T09:26:36.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.584+0000 7fbdf0a00640 1 -- 172.21.15.6:0/3755909531 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fbdcc06b4d0 msgr2=0x7fbdcc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:36.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.584+0000 7fbdf0a00640 1 --2- 172.21.15.6:0/3755909531 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fbdcc06b4d0 0x7fbdcc06d990 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fbddc035000 tx=0x7fbddc030000 comp rx=0 tx=0).stop 2024-03-04T09:26:36.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.585+0000 7fbdf0a00640 1 -- 172.21.15.6:0/3755909531 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbdec145570 msgr2=0x7fbdec0b1060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:36.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.585+0000 7fbdf0a00640 1 --2- 172.21.15.6:0/3755909531 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbdec145570 0x7fbdec0b1060 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fbde400c410 tx=0x7fbde400c8d0 comp rx=0 tx=0).stop 2024-03-04T09:26:36.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.585+0000 7fbdf0a00640 1 -- 172.21.15.6:0/3755909531 shutdown_connections 2024-03-04T09:26:36.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.585+0000 7fbdf0a00640 1 --2- 172.21.15.6:0/3755909531 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbdec145570 0x7fbdec0b1060 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:36.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.585+0000 7fbdf0a00640 1 --2- 172.21.15.6:0/3755909531 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fbdcc06b4d0 0x7fbdcc06d990 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:36.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.585+0000 7fbdf0a00640 1 --2- 172.21.15.6:0/3755909531 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbdec0cd430 0x7fbdec1f17a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:36.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.585+0000 7fbdf0a00640 1 --2- 172.21.15.6:0/3755909531 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbdec0b42c0 0x7fbdec0adf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:36.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.585+0000 7fbdf0a00640 1 -- 172.21.15.6:0/3755909531 >> 172.21.15.6:0/3755909531 conn(0x7fbdec06b190 msgr2=0x7fbdec0b1d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:36.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.585+0000 7fbdf0a00640 1 -- 172.21.15.6:0/3755909531 shutdown_connections 2024-03-04T09:26:36.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.585+0000 7fbdf0a00640 1 -- 172.21.15.6:0/3755909531 wait complete. 2024-03-04T09:26:36.603 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:26:36.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.828+0000 7f19f2000640 1 Processor -- start 2024-03-04T09:26:36.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.829+0000 7f19f2000640 1 -- start start 2024-03-04T09:26:36.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.830+0000 7f19f2000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f19ec0cf5c0 0x7f19ec0cdc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:36.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.830+0000 7f19f2000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f19ec071050 0x7f19ec071490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:36.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.830+0000 7f19f2000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19ec0b4860 con 0x7f19ec0b3c90 2024-03-04T09:26:36.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.830+0000 7f19f2000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19ec0ce160 con 0x7f19ec0cf5c0 2024-03-04T09:26:36.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.830+0000 7f19f2000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19ec0b4b40 con 0x7f19ec071050 2024-03-04T09:26:36.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.831+0000 7f19eaa00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f19ec0cf5c0 0x7f19ec0cdc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:36.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.831+0000 7f19ebe00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f19ec071050 0x7f19ec071490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:36.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.831+0000 7f19ebe00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f19ec071050 0x7f19ec071490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.6:42406/0 (socket says 172.21.15.6:42406) 2024-03-04T09:26:36.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.831+0000 7f19eaa00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f19ec0cf5c0 0x7f19ec0cdc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.6:50500/0 (socket says 172.21.15.6:50500) 2024-03-04T09:26:36.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.831+0000 7f19ebe00640 1 -- 172.21.15.6:0/2141877736 learned_addr learned my addr 172.21.15.6:0/2141877736 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:36.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.831+0000 7f19ea000640 1 -- 172.21.15.6:0/2141877736 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4203150745 0 0) 0x7f19ec0b4860 con 0x7f19ec0b3c90 2024-03-04T09:26:36.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.831+0000 7f19ea000640 1 -- 172.21.15.6:0/2141877736 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19d0003520 con 0x7f19ec0b3c90 2024-03-04T09:26:36.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.831+0000 7f19eaa00640 1 -- 172.21.15.6:0/2141877736 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f19ec071050 msgr2=0x7f19ec071490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:36.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.831+0000 7f19eaa00640 1 --2- 172.21.15.6:0/2141877736 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f19ec071050 0x7f19ec071490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:36.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.831+0000 7f19eaa00640 1 -- 172.21.15.6:0/2141877736 >> v1:172.21.15.146:6789/0 conn(0x7f19ec0b3c90 legacy=0x7f19ec0b4070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:36.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.831+0000 7f19eaa00640 1 -- 172.21.15.6:0/2141877736 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19ec0b4e20 con 0x7f19ec0cf5c0 2024-03-04T09:26:36.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.833+0000 7f19eaa00640 1 --2- 172.21.15.6:0/2141877736 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f19ec0cf5c0 0x7f19ec0cdc20 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f19d40097a0 tx=0x7f19d402b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eb299352406b90c server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:36.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.833+0000 7f19ea000640 1 -- 172.21.15.6:0/2141877736 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19d4034070 con 0x7f19ec0cf5c0 2024-03-04T09:26:36.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.833+0000 7f19ea000640 1 -- 172.21.15.6:0/2141877736 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19d4030610 con 0x7f19ec0cf5c0 2024-03-04T09:26:36.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.833+0000 7f19ea000640 1 -- 172.21.15.6:0/2141877736 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19d40309c0 con 0x7f19ec0cf5c0 2024-03-04T09:26:36.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.834+0000 7f19f2000640 1 -- 172.21.15.6:0/2141877736 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f19ec0cf5c0 msgr2=0x7f19ec0cdc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:36.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.834+0000 7f19f2000640 1 --2- 172.21.15.6:0/2141877736 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f19ec0cf5c0 0x7f19ec0cdc20 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f19d40097a0 tx=0x7f19d402b960 comp rx=0 tx=0).stop 2024-03-04T09:26:36.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.834+0000 7f19f2000640 1 -- 172.21.15.6:0/2141877736 shutdown_connections 2024-03-04T09:26:36.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.834+0000 7f19f2000640 1 --2- 172.21.15.6:0/2141877736 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f19ec071050 0x7f19ec071490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:36.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.834+0000 7f19f2000640 1 --2- 172.21.15.6:0/2141877736 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f19ec0cf5c0 0x7f19ec0cdc20 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:36.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.834+0000 7f19f2000640 1 -- 172.21.15.6:0/2141877736 >> 172.21.15.6:0/2141877736 conn(0x7f19ec06afb0 msgr2=0x7f19ec06b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:36.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.834+0000 7f19f2000640 1 -- 172.21.15.6:0/2141877736 shutdown_connections 2024-03-04T09:26:36.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.834+0000 7f19f2000640 1 -- 172.21.15.6:0/2141877736 wait complete. 2024-03-04T09:26:36.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.836+0000 7f19f2000640 1 Processor -- start 2024-03-04T09:26:36.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.836+0000 7f19f2000640 1 -- start start 2024-03-04T09:26:36.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.837+0000 7f19f2000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f19ec071050 0x7f19ec14e750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:36.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.837+0000 7f19f2000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f19ec0b3c90 0x7f19ec150c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:36.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.837+0000 7f19f2000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f19ec0cf5c0 0x7f19ec0b0ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:36.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.837+0000 7f19f2000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19ec0719d0 con 0x7f19ec0cf5c0 2024-03-04T09:26:36.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.837+0000 7f19f2000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19ec15b340 con 0x7f19ec0b3c90 2024-03-04T09:26:36.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.837+0000 7f19f2000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19ec071b50 con 0x7f19ec071050 2024-03-04T09:26:36.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.837+0000 7f19ebe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f19ec0cf5c0 0x7f19ec0b0ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:36.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.838+0000 7f19ebe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f19ec0cf5c0 0x7f19ec0b0ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:40788/0 (socket says 172.21.15.6:40788) 2024-03-04T09:26:36.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.838+0000 7f19ebe00640 1 -- 172.21.15.6:0/551033790 learned_addr learned my addr 172.21.15.6:0/551033790 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:36.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.838+0000 7f19eaa00640 1 --2- 172.21.15.6:0/551033790 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f19ec0b3c90 0x7f19ec150c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:36.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.838+0000 7f19eb400640 1 --2- 172.21.15.6:0/551033790 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f19ec071050 0x7f19ec14e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:36.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.838+0000 7f19ebe00640 1 -- 172.21.15.6:0/551033790 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f19ec071050 msgr2=0x7f19ec14e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:36.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.838+0000 7f19ebe00640 1 --2- 172.21.15.6:0/551033790 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f19ec071050 0x7f19ec14e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:36.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.838+0000 7f19ebe00640 1 -- 172.21.15.6:0/551033790 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f19ec0b3c90 msgr2=0x7f19ec150c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:36.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.838+0000 7f19ebe00640 1 --2- 172.21.15.6:0/551033790 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f19ec0b3c90 0x7f19ec150c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:36.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.838+0000 7f19ebe00640 1 -- 172.21.15.6:0/551033790 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19ec1511d0 con 0x7f19ec0cf5c0 2024-03-04T09:26:36.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.838+0000 7f19ebe00640 1 --2- 172.21.15.6:0/551033790 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f19ec0cf5c0 0x7f19ec0b0ac0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f19e400efc0 tx=0x7f19e400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:36.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.839+0000 7f19e3400640 1 -- 172.21.15.6:0/551033790 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19e400ccc0 con 0x7f19ec0cf5c0 2024-03-04T09:26:36.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.839+0000 7f19e3400640 1 -- 172.21.15.6:0/551033790 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19e4003990 con 0x7f19ec0cf5c0 2024-03-04T09:26:36.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.839+0000 7f19e3400640 1 -- 172.21.15.6:0/551033790 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19e400ce60 con 0x7f19ec0cf5c0 2024-03-04T09:26:36.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.839+0000 7f19f2000640 1 -- 172.21.15.6:0/551033790 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19ec1518b0 con 0x7f19ec0cf5c0 2024-03-04T09:26:36.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.839+0000 7f19f2000640 1 -- 172.21.15.6:0/551033790 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19ec151e70 con 0x7f19ec0cf5c0 2024-03-04T09:26:36.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.840+0000 7f19f2000640 1 -- 172.21.15.6:0/551033790 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19ec069510 con 0x7f19ec0cf5c0 2024-03-04T09:26:36.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.841+0000 7f19e3400640 1 -- 172.21.15.6:0/551033790 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f19e4005d10 con 0x7f19ec0cf5c0 2024-03-04T09:26:36.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.842+0000 7f19e3400640 1 --2- 172.21.15.6:0/551033790 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f19c806b4d0 0x7f19c806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:36.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.842+0000 7f19e3400640 1 -- 172.21.15.6:0/551033790 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f19e408a6f0 con 0x7f19ec0cf5c0 2024-03-04T09:26:36.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.843+0000 7f19eb400640 1 --2- 172.21.15.6:0/551033790 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f19c806b4d0 0x7f19c806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:36.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.844+0000 7f19eb400640 1 --2- 172.21.15.6:0/551033790 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f19c806b4d0 0x7f19c806d990 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f19dc001650 tx=0x7f19dc009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:36.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:36.847+0000 7f19e3400640 1 -- 172.21.15.6:0/551033790 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f19e4022020 con 0x7f19ec0cf5c0 2024-03-04T09:26:37.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.085+0000 7f19f2000640 1 -- 172.21.15.6:0/551033790 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f19ec072fb0 con 0x7f19ec0cf5c0 2024-03-04T09:26:37.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.086+0000 7f19e3400640 1 -- 172.21.15.6:0/551033790 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+4977 (secure 0 0 0) 0x7f19e4010040 con 0x7f19ec0cf5c0 2024-03-04T09:26:37.083 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:26:37.083 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4685,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/1703428647","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":1703428647},{"type":"v1","addr":"172.21.15.146:6837","nonce":1703428647}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4837,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/2617429530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2617429530},{"type":"v1","addr":"172.21.15.6:6833","nonce":2617429530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":5043,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/2744913857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":2744913857},{"type":"v1","addr":"172.21.15.6:6835","nonce":2744913857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:26:30.445199+0000","modified":"2024-03-04T09:26:33.534893+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5046},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5046":{"gid":5046,"name":"d","rank":0,"incarnation":12,"state":"up:active","state_seq":4,"addr":"172.21.15.146:6836/2285803221","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":2285803221},{"type":"v1","addr":"172.21.15.146:6836","nonce":2285803221}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5046,"qdb_cluster":[5046]},"id":2}]} 2024-03-04T09:26:37.083 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 16 2024-03-04T09:26:37.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.088+0000 7f19f2000640 1 -- 172.21.15.6:0/551033790 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f19c806b4d0 msgr2=0x7f19c806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:37.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.088+0000 7f19f2000640 1 --2- 172.21.15.6:0/551033790 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f19c806b4d0 0x7f19c806d990 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f19dc001650 tx=0x7f19dc009040 comp rx=0 tx=0).stop 2024-03-04T09:26:37.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.088+0000 7f19f2000640 1 -- 172.21.15.6:0/551033790 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f19ec0cf5c0 msgr2=0x7f19ec0b0ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:37.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.088+0000 7f19f2000640 1 --2- 172.21.15.6:0/551033790 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f19ec0cf5c0 0x7f19ec0b0ac0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f19e400efc0 tx=0x7f19e400c490 comp rx=0 tx=0).stop 2024-03-04T09:26:37.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.088+0000 7f19f2000640 1 -- 172.21.15.6:0/551033790 shutdown_connections 2024-03-04T09:26:37.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.088+0000 7f19f2000640 1 --2- 172.21.15.6:0/551033790 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f19ec0cf5c0 0x7f19ec0b0ac0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:37.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.088+0000 7f19f2000640 1 --2- 172.21.15.6:0/551033790 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f19ec0b3c90 0x7f19ec150c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:37.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.088+0000 7f19f2000640 1 --2- 172.21.15.6:0/551033790 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f19c806b4d0 0x7f19c806d990 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:37.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.088+0000 7f19f2000640 1 --2- 172.21.15.6:0/551033790 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f19ec071050 0x7f19ec14e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:37.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.089+0000 7f19f2000640 1 -- 172.21.15.6:0/551033790 >> 172.21.15.6:0/551033790 conn(0x7f19ec06afb0 msgr2=0x7f19ec071fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:37.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.089+0000 7f19f2000640 1 -- 172.21.15.6:0/551033790 shutdown_connections 2024-03-04T09:26:37.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.089+0000 7f19f2000640 1 -- 172.21.15.6:0/551033790 wait complete. 2024-03-04T09:26:37.508 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 16, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:26:30.445199+0000', 'modified': '2024-03-04T09:26:33.534893+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5046}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5046': {'gid': 5046, 'name': 'd', 'rank': 0, 'incarnation': 12, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.146:6836/2285803221', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6834', 'nonce': 2285803221}, {'type': 'v1', 'addr': '172.21.15.146:6836', 'nonce': 2285803221}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [5], 'metadata_pool': 4, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5046, 'qdb_cluster': [5046]} 2024-03-04T09:26:37.509 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-04T09:26:37.509 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-03-04T09:26:37.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.752+0000 7f02e0c00640 1 Processor -- start 2024-03-04T09:26:37.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.753+0000 7f02e0c00640 1 -- start start 2024-03-04T09:26:37.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.754+0000 7f02e0c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f02dc0cd060 0x7f02dc1667e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:37.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.754+0000 7f02e0c00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02dc0714d0 con 0x7f02dc070f10 2024-03-04T09:26:37.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.754+0000 7f02e0c00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02dc0717b0 con 0x7f02dc145300 2024-03-04T09:26:37.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.754+0000 7f02e0c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02dc071a90 con 0x7f02dc0cd060 2024-03-04T09:26:37.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.755+0000 7f02d3e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f02dc070f10 0x7f02dc0cc950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:37010/0 (socket says 172.21.15.6:37010) 2024-03-04T09:26:37.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.755+0000 7f02d3e00640 1 -- 172.21.15.6:0/2708025237 learned_addr learned my addr 172.21.15.6:0/2708025237 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:26:37.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.755+0000 7f02dbe00640 1 --2- 172.21.15.6:0/2708025237 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f02dc0cd060 0x7f02dc1667e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:37.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.755+0000 7f02daa00640 1 -- 172.21.15.6:0/2708025237 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2431862591 0 0) 0x7f02dc0714d0 con 0x7f02dc070f10 2024-03-04T09:26:37.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.755+0000 7f02daa00640 1 -- 172.21.15.6:0/2708025237 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02c0003520 con 0x7f02dc070f10 2024-03-04T09:26:37.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.756+0000 7f02dbe00640 1 -- 172.21.15.6:0/2708025237 >> v1:172.21.15.146:6789/0 conn(0x7f02dc145300 legacy=0x7f02dc0b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:37.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.756+0000 7f02daa00640 1 -- 172.21.15.6:0/2708025237 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1624336998 0 0) 0x7f02dc0717b0 con 0x7f02dc145300 2024-03-04T09:26:37.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.756+0000 7f02dbe00640 1 -- 172.21.15.6:0/2708025237 >> v1:172.21.15.6:6789/0 conn(0x7f02dc070f10 legacy=0x7f02dc0cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:37.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.756+0000 7f02dbe00640 1 -- 172.21.15.6:0/2708025237 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02dc071d70 con 0x7f02dc0cd060 2024-03-04T09:26:37.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.757+0000 7f02dbe00640 1 --2- 172.21.15.6:0/2708025237 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f02dc0cd060 0x7f02dc1667e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f02d4009dc0 tx=0x7f02d402bc80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=301f9dd616e959e server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:37.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.758+0000 7f02daa00640 1 -- 172.21.15.6:0/2708025237 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02d4039070 con 0x7f02dc0cd060 2024-03-04T09:26:37.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.758+0000 7f02daa00640 1 -- 172.21.15.6:0/2708025237 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02d402c7b0 con 0x7f02dc0cd060 2024-03-04T09:26:37.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.758+0000 7f02daa00640 1 -- 172.21.15.6:0/2708025237 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02c4001670 con 0x7f02dc0cd060 2024-03-04T09:26:37.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.758+0000 7f02e0c00640 1 -- 172.21.15.6:0/2708025237 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f02dc0cd060 msgr2=0x7f02dc1667e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:37.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.758+0000 7f02e0c00640 1 --2- 172.21.15.6:0/2708025237 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f02dc0cd060 0x7f02dc1667e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f02d4009dc0 tx=0x7f02d402bc80 comp rx=0 tx=0).stop 2024-03-04T09:26:37.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.758+0000 7f02e0c00640 1 -- 172.21.15.6:0/2708025237 shutdown_connections 2024-03-04T09:26:37.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.758+0000 7f02e0c00640 1 --2- 172.21.15.6:0/2708025237 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f02dc0cd060 0x7f02dc1667e0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:37.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.758+0000 7f02e0c00640 1 -- 172.21.15.6:0/2708025237 >> 172.21.15.6:0/2708025237 conn(0x7f02dc06b3d0 msgr2=0x7f02dc06b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:37.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.759+0000 7f02e0c00640 1 -- 172.21.15.6:0/2708025237 shutdown_connections 2024-03-04T09:26:37.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.759+0000 7f02e0c00640 1 -- 172.21.15.6:0/2708025237 wait complete. 2024-03-04T09:26:37.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.761+0000 7f02e0c00640 1 Processor -- start 2024-03-04T09:26:37.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.761+0000 7f02e0c00640 1 -- start start 2024-03-04T09:26:37.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.762+0000 7f02e0c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f02dc070f10 0x7f02dc15b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:37.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.762+0000 7f02e0c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f02dc0cd060 0x7f02dc13af30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:37.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.762+0000 7f02e0c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f02dc145300 0x7f02dc0ae8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:37.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.762+0000 7f02e0c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02dc15d700 con 0x7f02dc070f10 2024-03-04T09:26:37.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.762+0000 7f02e0c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02dc15d580 con 0x7f02dc0cd060 2024-03-04T09:26:37.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.762+0000 7f02e0c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02dc15d880 con 0x7f02dc145300 2024-03-04T09:26:37.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.762+0000 7f02db400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f02dc070f10 0x7f02dc15b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:37.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.762+0000 7f02db400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f02dc070f10 0x7f02dc15b3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:40802/0 (socket says 172.21.15.6:40802) 2024-03-04T09:26:37.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.762+0000 7f02db400640 1 -- 172.21.15.6:0/867599869 learned_addr learned my addr 172.21.15.6:0/867599869 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:37.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.763+0000 7f02dbe00640 1 --2- 172.21.15.6:0/867599869 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f02dc145300 0x7f02dc0ae8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:37.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.763+0000 7f02d3e00640 1 --2- 172.21.15.6:0/867599869 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f02dc0cd060 0x7f02dc13af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:37.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.763+0000 7f02db400640 1 -- 172.21.15.6:0/867599869 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f02dc145300 msgr2=0x7f02dc0ae8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:37.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.763+0000 7f02db400640 1 --2- 172.21.15.6:0/867599869 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f02dc145300 0x7f02dc0ae8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:37.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.763+0000 7f02db400640 1 -- 172.21.15.6:0/867599869 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f02dc0cd060 msgr2=0x7f02dc13af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:37.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.763+0000 7f02db400640 1 --2- 172.21.15.6:0/867599869 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f02dc0cd060 0x7f02dc13af30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:37.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.763+0000 7f02db400640 1 -- 172.21.15.6:0/867599869 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02dc13eef0 con 0x7f02dc070f10 2024-03-04T09:26:37.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.763+0000 7f02db400640 1 --2- 172.21.15.6:0/867599869 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f02dc070f10 0x7f02dc15b3d0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f02cc00c4b0 tx=0x7f02cc00c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:37.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.764+0000 7f02d3400640 1 -- 172.21.15.6:0/867599869 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02cc017070 con 0x7f02dc070f10 2024-03-04T09:26:37.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.764+0000 7f02d3400640 1 -- 172.21.15.6:0/867599869 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02cc005480 con 0x7f02dc070f10 2024-03-04T09:26:37.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.764+0000 7f02e0c00640 1 -- 172.21.15.6:0/867599869 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02dc13b9f0 con 0x7f02dc070f10 2024-03-04T09:26:37.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.764+0000 7f02d3400640 1 -- 172.21.15.6:0/867599869 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02cc003830 con 0x7f02dc070f10 2024-03-04T09:26:37.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.764+0000 7f02e0c00640 1 -- 172.21.15.6:0/867599869 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02dc13bfb0 con 0x7f02dc070f10 2024-03-04T09:26:37.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.766+0000 7f02d3400640 1 -- 172.21.15.6:0/867599869 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f02cc01a070 con 0x7f02dc070f10 2024-03-04T09:26:37.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.767+0000 7f02d3400640 1 --2- 172.21.15.6:0/867599869 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f02b806b4d0 0x7f02b806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:37.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.767+0000 7f02d3400640 1 -- 172.21.15.6:0/867599869 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f02cc089620 con 0x7f02dc070f10 2024-03-04T09:26:37.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.768+0000 7f02d3e00640 1 --2- 172.21.15.6:0/867599869 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f02b806b4d0 0x7f02b806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:37.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.768+0000 7f02e4fea740 1 Processor -- start 2024-03-04T09:26:37.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.768+0000 7f02e4fea740 1 -- start start 2024-03-04T09:26:37.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.769+0000 7f02e4fea740 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55e483bbf990 0x55e483b92930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:37.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.769+0000 7f02e4fea740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e483bbfd70 0x55e483bc2720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:37.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.769+0000 7f02e4fea740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55e483b92e70 0x55e483bc2c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:37.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.769+0000 7f02e4fea740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e483afc600 con 0x55e483bbfd70 2024-03-04T09:26:37.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.769+0000 7f02e4fea740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e483aa5930 con 0x55e483b92e70 2024-03-04T09:26:37.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.769+0000 7f02e4fea740 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e483af86a0 con 0x55e483bbf990 2024-03-04T09:26:37.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.769+0000 7f02d3e00640 1 --2- 172.21.15.6:0/867599869 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f02b806b4d0 0x7f02b806d990 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f02dc143c50 tx=0x7f02c40018f0 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:37.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.769+0000 7f02d3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e483bbfd70 0x55e483bc2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:37.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.769+0000 7f02d3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e483bbfd70 0x55e483bc2720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:40810/0 (socket says 172.21.15.6:40810) 2024-03-04T09:26:37.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.769+0000 7f02d3e00640 1 -- 172.21.15.6:0/2351909135 learned_addr learned my addr 172.21.15.6:0/2351909135 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:37.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.769+0000 7f02db400640 1 --2- 172.21.15.6:0/2351909135 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55e483bbf990 0x55e483b92930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:37.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.769+0000 7f02dbe00640 1 --2- 172.21.15.6:0/2351909135 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55e483b92e70 0x55e483bc2c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:37.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.769+0000 7f02d3e00640 1 -- 172.21.15.6:0/2351909135 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55e483bbf990 msgr2=0x55e483b92930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:37.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.769+0000 7f02d3e00640 1 --2- 172.21.15.6:0/2351909135 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55e483bbf990 0x55e483b92930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:37.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.769+0000 7f02d3e00640 1 -- 172.21.15.6:0/2351909135 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55e483b92e70 msgr2=0x55e483bc2c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:37.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.769+0000 7f02d3e00640 1 --2- 172.21.15.6:0/2351909135 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55e483b92e70 0x55e483bc2c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:37.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.769+0000 7f02d3e00640 1 -- 172.21.15.6:0/2351909135 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e4838112d0 con 0x55e483bbfd70 2024-03-04T09:26:37.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.770+0000 7f02d3e00640 1 --2- 172.21.15.6:0/2351909135 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e483bbfd70 0x55e483bc2720 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f02c400eb80 tx=0x7f02c4011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=249bee0cdfe3ddf6 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:37.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.770+0000 7f02cb400640 1 -- 172.21.15.6:0/2351909135 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02c4004e60 con 0x55e483bbfd70 2024-03-04T09:26:37.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.770+0000 7f02cb400640 1 -- 172.21.15.6:0/2351909135 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02c4005260 con 0x55e483bbfd70 2024-03-04T09:26:37.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.770+0000 7f02cb400640 1 -- 172.21.15.6:0/2351909135 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02c4005610 con 0x55e483bbfd70 2024-03-04T09:26:37.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.771+0000 7f02e4fea740 1 -- 172.21.15.6:0/2351909135 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e483bbfd70 msgr2=0x55e483bc2720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:37.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.771+0000 7f02e4fea740 1 --2- 172.21.15.6:0/2351909135 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e483bbfd70 0x55e483bc2720 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f02c400eb80 tx=0x7f02c4011000 comp rx=0 tx=0).stop 2024-03-04T09:26:37.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.771+0000 7f02e4fea740 1 -- 172.21.15.6:0/2351909135 shutdown_connections 2024-03-04T09:26:37.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.771+0000 7f02e4fea740 1 --2- 172.21.15.6:0/2351909135 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55e483b92e70 0x55e483bc2c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:37.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.771+0000 7f02e4fea740 1 --2- 172.21.15.6:0/2351909135 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e483bbfd70 0x55e483bc2720 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:37.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.771+0000 7f02e4fea740 1 --2- 172.21.15.6:0/2351909135 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55e483bbf990 0x55e483b92930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:37.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.771+0000 7f02e4fea740 1 -- 172.21.15.6:0/2351909135 >> 172.21.15.6:0/2351909135 conn(0x55e483b94ba0 msgr2=0x55e483bdd720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:37.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.771+0000 7f02e4fea740 1 -- 172.21.15.6:0/2351909135 shutdown_connections 2024-03-04T09:26:37.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.771+0000 7f02e4fea740 1 -- 172.21.15.6:0/2351909135 wait complete. 2024-03-04T09:26:37.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.772+0000 7f02e4fea740 1 -- start start 2024-03-04T09:26:37.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.772+0000 7f02e4fea740 1 Processor -- start 2024-03-04T09:26:37.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.773+0000 7f02e4fea740 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55e483bbf990 0x55e483bc4b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:37.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.773+0000 7f02e4fea740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e483bbfd70 0x55e483bc5050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:37.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.773+0000 7f02e4fea740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55e483bb2de0 0x55e483bd69a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:37.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.773+0000 7f02e4fea740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e483b97140 con 0x55e483bbfd70 2024-03-04T09:26:37.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.773+0000 7f02e4fea740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e483ab8d90 con 0x55e483bb2de0 2024-03-04T09:26:37.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.773+0000 7f02e4fea740 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e483b96530 con 0x55e483bbf990 2024-03-04T09:26:37.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.773+0000 7f02e4fea740 10 client.0 _ll_register_callbacks cb 0x55e483af4350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T09:26:37.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.774+0000 7f02d3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e483bbfd70 0x55e483bc5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:37.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.774+0000 7f02db400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55e483bbf990 0x55e483bc4b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:37.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.774+0000 7f02d3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e483bbfd70 0x55e483bc5050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:40820/0 (socket says 172.21.15.6:40820) 2024-03-04T09:26:37.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.774+0000 7f02d3e00640 1 -- 172.21.15.6:0/2936309377 learned_addr learned my addr 172.21.15.6:0/2936309377 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:37.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.774+0000 7f02dbe00640 1 --2- 172.21.15.6:0/2936309377 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55e483bb2de0 0x55e483bd69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:37.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.774+0000 7f02d3e00640 1 -- 172.21.15.6:0/2936309377 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55e483bbf990 msgr2=0x55e483bc4b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:37.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.774+0000 7f02d3e00640 1 --2- 172.21.15.6:0/2936309377 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55e483bbf990 0x55e483bc4b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:37.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.774+0000 7f02d3e00640 1 -- 172.21.15.6:0/2936309377 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55e483bb2de0 msgr2=0x55e483bd69a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:37.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.774+0000 7f02d3e00640 1 --2- 172.21.15.6:0/2936309377 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55e483bb2de0 0x55e483bd69a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:37.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.774+0000 7f02d3e00640 1 -- 172.21.15.6:0/2936309377 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e4838db6b0 con 0x55e483bbfd70 2024-03-04T09:26:37.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.774+0000 7f02d3e00640 1 --2- 172.21.15.6:0/2936309377 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e483bbfd70 0x55e483bc5050 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f02c4004c70 tx=0x7f02c4013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38de8b7f51b74efd server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:37.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.774+0000 7f02bfe00640 10 client.0 ms_handle_connect on v2:172.21.15.6:3300/0 2024-03-04T09:26:37.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.775+0000 7f02bfe00640 1 -- 172.21.15.6:0/2936309377 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02c4021860 con 0x55e483bbfd70 2024-03-04T09:26:37.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.775+0000 7f02bfe00640 1 -- 172.21.15.6:0/2936309377 <== mon.0 v2:172.21.15.6:3300/0 2 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f02c4021a00 con 0x55e483bbfd70 2024-03-04T09:26:37.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.775+0000 7f02e4fea740 1 -- 172.21.15.6:0/2936309377 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f02c4005260 con 0x55e483bbfd70 2024-03-04T09:26:37.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.775+0000 7f02bfe00640 1 -- 172.21.15.6:0/2936309377 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) v2 ==== 24+0+0 (secure 0 0 0) 0x7f02c4021c00 con 0x55e483bbfd70 2024-03-04T09:26:37.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.775+0000 7f02e4fea740 10 client.5220 fetch_fsmap learned FSMap version 16 2024-03-04T09:26:37.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.775+0000 7f02e4fea740 1 -- 172.21.15.6:0/2936309377 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=2+}) v3 -- 0x55e483bd4d30 con 0x55e483bbfd70 2024-03-04T09:26:37.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.775+0000 7f02bfe00640 1 -- 172.21.15.6:0/2936309377 <== mon.0 v2:172.21.15.6:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02c4005260 con 0x55e483bbfd70 2024-03-04T09:26:37.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.775+0000 7f02bfe00640 1 -- 172.21.15.6:0/2936309377 <== mon.0 v2:172.21.15.6:3300/0 5 ==== fsmap(e 16) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f02c402c430 con 0x55e483bbfd70 2024-03-04T09:26:37.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.775+0000 7f02e4fea740 10 client.5220 fetch_fsmap finished waiting for FSMap version 16 2024-03-04T09:26:37.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.775+0000 7f02e4fea740 10 client.5220 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-04T09:26:37.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.775+0000 7f02e4fea740 20 client.5220 populate_metadata read hostname 'smithi006' 2024-03-04T09:26:37.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.775+0000 7f02e4fea740 1 --2- 172.21.15.6:0/2936309377 >> [v2:172.21.15.146:6834/2285803221,v1:172.21.15.146:6836/2285803221] conn(0x55e483bd5b80 0x55e483bee000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:37.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.775+0000 7f02e4fea740 4 client.5220 mds_command: new command op to 5046 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T09:26:37.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.775+0000 7f02e4fea740 1 -- 172.21.15.6:0/2936309377 --> [v2:172.21.15.146:6834/2285803221,v1:172.21.15.146:6836/2285803221] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e483b09b40 con 0x55e483bd5b80 2024-03-04T09:26:37.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.776+0000 7f02db400640 1 --2- 172.21.15.6:0/2936309377 >> [v2:172.21.15.146:6834/2285803221,v1:172.21.15.146:6836/2285803221] conn(0x55e483bd5b80 0x55e483bee000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:37.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.777+0000 7f02db400640 1 --2- 172.21.15.6:0/2936309377 >> [v2:172.21.15.146:6834/2285803221,v1:172.21.15.146:6836/2285803221] conn(0x55e483bd5b80 0x55e483bee000 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ffa1ab1c9f4f25bc server_cookie=bd08404e5d4d9148 in_seq=0 out_seq=0 2024-03-04T09:26:37.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.777+0000 7f02bfe00640 10 client.5220 ms_handle_connect on v2:172.21.15.146:6834/2285803221 2024-03-04T09:26:37.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.779+0000 7f02bfe00640 1 -- 172.21.15.6:0/2936309377 <== mds.0 v2:172.21.15.146:6834/2285803221 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55e483b09b40 con 0x55e483bd5b80 2024-03-04T09:26:37.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.779+0000 7f02bfe00640 10 client.5220 handle_command_reply: tid=0 2024-03-04T09:26:37.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.779+0000 7f02e4fea740 1 client.5220 shutdown 2024-03-04T09:26:37.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.779+0000 7f02e4fea740 1 -- 172.21.15.6:0/2936309377 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e483bbfd70 msgr2=0x55e483bc5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:37.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.779+0000 7f02e4fea740 1 --2- 172.21.15.6:0/2936309377 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e483bbfd70 0x55e483bc5050 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f02c4004c70 tx=0x7f02c4013000 comp rx=0 tx=0).stop 2024-03-04T09:26:37.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.779+0000 7f02e4fea740 1 -- 172.21.15.6:0/2936309377 shutdown_connections 2024-03-04T09:26:37.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.779+0000 7f02e4fea740 1 --2- 172.21.15.6:0/2936309377 >> [v2:172.21.15.146:6834/2285803221,v1:172.21.15.146:6836/2285803221] conn(0x55e483bd5b80 0x55e483bee000 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:37.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.779+0000 7f02e4fea740 1 --2- 172.21.15.6:0/2936309377 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55e483bb2de0 0x55e483bd69a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:37.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.779+0000 7f02e4fea740 1 --2- 172.21.15.6:0/2936309377 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e483bbfd70 0x55e483bc5050 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:37.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.779+0000 7f02e4fea740 1 --2- 172.21.15.6:0/2936309377 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55e483bbf990 0x55e483bc4b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:37.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.779+0000 7f02e4fea740 1 -- 172.21.15.6:0/2936309377 >> 172.21.15.6:0/2936309377 conn(0x55e483b92e70 msgr2=0x55e483bc2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:37.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.779+0000 7f02bfe00640 0 client.5220 ms_handle_reset on v2:172.21.15.146:6834/2285803221 2024-03-04T09:26:37.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.779+0000 7f02e4fea740 1 -- 172.21.15.6:0/2936309377 shutdown_connections 2024-03-04T09:26:37.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.780+0000 7f02e4fea740 1 -- 172.21.15.6:0/2936309377 wait complete. 2024-03-04T09:26:37.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.780+0000 7f02e4fea740 20 client.5220 trim_cache size 0 max 16384 2024-03-04T09:26:37.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.798+0000 7f02e4fea740 1 Processor -- start 2024-03-04T09:26:37.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.798+0000 7f02e4fea740 1 -- start start 2024-03-04T09:26:37.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.799+0000 7f02e4fea740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55e483bb2de0 0x55e483b968b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:37.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.799+0000 7f02e4fea740 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55e483bbf990 0x55e483be0e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:37.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.799+0000 7f02e4fea740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e483bbfd70 0x55e483be1370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:37.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.799+0000 7f02e4fea740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02dc1ff050 con 0x55e483bbfd70 2024-03-04T09:26:37.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.799+0000 7f02e4fea740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02dc13ffa0 con 0x55e483bb2de0 2024-03-04T09:26:37.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.799+0000 7f02e4fea740 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02dc1405a0 con 0x55e483bbf990 2024-03-04T09:26:37.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.800+0000 7f02dbe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e483bbfd70 0x55e483be1370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:37.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.800+0000 7f02dbe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e483bbfd70 0x55e483be1370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:40834/0 (socket says 172.21.15.6:40834) 2024-03-04T09:26:37.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.800+0000 7f02dbe00640 1 -- 172.21.15.6:0/679311501 learned_addr learned my addr 172.21.15.6:0/679311501 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:37.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.800+0000 7f02db400640 1 --2- 172.21.15.6:0/679311501 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55e483bb2de0 0x55e483b968b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:37.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.800+0000 7f02d3e00640 1 --2- 172.21.15.6:0/679311501 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55e483bbf990 0x55e483be0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:37.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.800+0000 7f02dbe00640 1 -- 172.21.15.6:0/679311501 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55e483bbf990 msgr2=0x55e483be0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:37.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.800+0000 7f02dbe00640 1 --2- 172.21.15.6:0/679311501 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55e483bbf990 0x55e483be0e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:37.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.800+0000 7f02dbe00640 1 -- 172.21.15.6:0/679311501 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55e483bb2de0 msgr2=0x55e483b968b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:37.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.800+0000 7f02dbe00640 1 --2- 172.21.15.6:0/679311501 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55e483bb2de0 0x55e483b968b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:37.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.800+0000 7f02dbe00640 1 -- 172.21.15.6:0/679311501 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e483be1bd0 con 0x55e483bbfd70 2024-03-04T09:26:37.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.800+0000 7f02db400640 1 --2- 172.21.15.6:0/679311501 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55e483bb2de0 0x55e483b968b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:26:37.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.800+0000 7f02d3e00640 1 --2- 172.21.15.6:0/679311501 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55e483bbf990 0x55e483be0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:26:37.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.800+0000 7f02dbe00640 1 --2- 172.21.15.6:0/679311501 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e483bbfd70 0x55e483be1370 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f02d402c290 tx=0x7f02d402e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7410893c83036049 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:37.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.801+0000 7f02caa00640 1 -- 172.21.15.6:0/679311501 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02d4046070 con 0x55e483bbfd70 2024-03-04T09:26:37.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.801+0000 7f02caa00640 1 -- 172.21.15.6:0/679311501 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02d4003c10 con 0x55e483bbfd70 2024-03-04T09:26:37.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.801+0000 7f02caa00640 1 -- 172.21.15.6:0/679311501 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02d4039040 con 0x55e483bbfd70 2024-03-04T09:26:37.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.801+0000 7f02e4fea740 1 -- 172.21.15.6:0/679311501 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e483bbfd70 msgr2=0x55e483be1370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:37.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.801+0000 7f02e4fea740 1 --2- 172.21.15.6:0/679311501 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e483bbfd70 0x55e483be1370 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f02d402c290 tx=0x7f02d402e000 comp rx=0 tx=0).stop 2024-03-04T09:26:37.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.801+0000 7f02e4fea740 1 -- 172.21.15.6:0/679311501 shutdown_connections 2024-03-04T09:26:37.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.801+0000 7f02e4fea740 1 --2- 172.21.15.6:0/679311501 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e483bbfd70 0x55e483be1370 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:37.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.801+0000 7f02e4fea740 1 --2- 172.21.15.6:0/679311501 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55e483bbf990 0x55e483be0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:37.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.801+0000 7f02e4fea740 1 --2- 172.21.15.6:0/679311501 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55e483bb2de0 0x55e483b968b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:37.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.801+0000 7f02e4fea740 1 -- 172.21.15.6:0/679311501 >> 172.21.15.6:0/679311501 conn(0x55e483b92e70 msgr2=0x55e483bc6700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:37.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.802+0000 7f02e4fea740 1 -- 172.21.15.6:0/679311501 shutdown_connections 2024-03-04T09:26:37.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.802+0000 7f02e4fea740 1 -- 172.21.15.6:0/679311501 wait complete. 2024-03-04T09:26:37.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.802+0000 7f02e4fea740 1 -- start start 2024-03-04T09:26:37.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.802+0000 7f02e4fea740 1 Processor -- start 2024-03-04T09:26:37.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.803+0000 7f02e4fea740 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55e483bb2de0 0x55e483bbe7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:37.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.803+0000 7f02e4fea740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e483bbf990 0x55e483bedeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:37.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.804+0000 7f02e4fea740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55e483bbfd70 0x55e483bee3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:37.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.804+0000 7f02e4fea740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e483bd5a00 con 0x55e483bbf990 2024-03-04T09:26:37.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.804+0000 7f02e4fea740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e483bd5010 con 0x55e483bbfd70 2024-03-04T09:26:37.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.804+0000 7f02e4fea740 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e483bd5fa0 con 0x55e483bb2de0 2024-03-04T09:26:37.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.804+0000 7f02e4fea740 10 client.0 _ll_register_callbacks cb 0x7f02c400ea80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T09:26:37.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.804+0000 7f02d3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e483bbf990 0x55e483bedeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:37.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.804+0000 7f02d3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e483bbf990 0x55e483bedeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:40844/0 (socket says 172.21.15.6:40844) 2024-03-04T09:26:37.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.804+0000 7f02d3e00640 1 -- 172.21.15.6:0/1730592704 learned_addr learned my addr 172.21.15.6:0/1730592704 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:37.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.804+0000 7f02d3e00640 1 -- 172.21.15.6:0/1730592704 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55e483bb2de0 msgr2=0x55e483bbe7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:37.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.804+0000 7f02d3e00640 1 --2- 172.21.15.6:0/1730592704 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55e483bb2de0 0x55e483bbe7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:37.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.804+0000 7f02d3e00640 1 -- 172.21.15.6:0/1730592704 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55e483bbfd70 msgr2=0x55e483bee3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:37.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.804+0000 7f02dbe00640 1 --2- 172.21.15.6:0/1730592704 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55e483bbfd70 0x55e483bee3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:37.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.804+0000 7f02d3e00640 1 --2- 172.21.15.6:0/1730592704 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55e483bbfd70 0x55e483bee3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:37.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.804+0000 7f02d3e00640 1 -- 172.21.15.6:0/1730592704 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e483bb2460 con 0x55e483bbf990 2024-03-04T09:26:37.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.805+0000 7f02d3e00640 1 --2- 172.21.15.6:0/1730592704 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e483bbf990 0x55e483bedeb0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f02c4004c70 tx=0x7f02c4001690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d01f238418303620 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:37.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.805+0000 7f02bfe00640 10 client.0 ms_handle_connect on v2:172.21.15.6:3300/0 2024-03-04T09:26:37.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.805+0000 7f02bfe00640 1 -- 172.21.15.6:0/1730592704 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02c4035640 con 0x55e483bbf990 2024-03-04T09:26:37.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.805+0000 7f02bfe00640 1 -- 172.21.15.6:0/1730592704 <== mon.0 v2:172.21.15.6:3300/0 2 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f02c40357e0 con 0x55e483bbf990 2024-03-04T09:26:37.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.805+0000 7f02e4fea740 1 -- 172.21.15.6:0/1730592704 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f02d4003c10 con 0x55e483bbf990 2024-03-04T09:26:37.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.805+0000 7f02bfe00640 1 -- 172.21.15.6:0/1730592704 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) v2 ==== 24+0+0 (secure 0 0 0) 0x7f02c4005260 con 0x55e483bbf990 2024-03-04T09:26:37.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.805+0000 7f02e4fea740 10 client.5226 fetch_fsmap learned FSMap version 16 2024-03-04T09:26:37.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.805+0000 7f02e4fea740 1 -- 172.21.15.6:0/1730592704 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=2+}) v3 -- 0x55e483c4d950 con 0x55e483bbf990 2024-03-04T09:26:37.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.805+0000 7f02bfe00640 1 -- 172.21.15.6:0/1730592704 <== mon.0 v2:172.21.15.6:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02d4003c10 con 0x55e483bbf990 2024-03-04T09:26:37.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.805+0000 7f02bfe00640 1 -- 172.21.15.6:0/1730592704 <== mon.0 v2:172.21.15.6:3300/0 5 ==== fsmap(e 16) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f02c4033020 con 0x55e483bbf990 2024-03-04T09:26:37.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.806+0000 7f02e4fea740 10 client.5226 fetch_fsmap finished waiting for FSMap version 16 2024-03-04T09:26:37.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.806+0000 7f02e4fea740 10 client.5226 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-04T09:26:37.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.806+0000 7f02e4fea740 20 client.5226 populate_metadata read hostname 'smithi006' 2024-03-04T09:26:37.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.806+0000 7f02e4fea740 1 --2- 172.21.15.6:0/1730592704 >> [v2:172.21.15.146:6834/2285803221,v1:172.21.15.146:6836/2285803221] conn(0x55e483c4c450 0x55e483c4c830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:37.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.806+0000 7f02e4fea740 4 client.5226 mds_command: new command op to 5046 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-04T09:26:37.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.806+0000 7f02e4fea740 1 -- 172.21.15.6:0/1730592704 --> [v2:172.21.15.146:6834/2285803221,v1:172.21.15.146:6836/2285803221] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55e483b03e90 con 0x55e483c4c450 2024-03-04T09:26:37.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.806+0000 7f02db400640 1 --2- 172.21.15.6:0/1730592704 >> [v2:172.21.15.146:6834/2285803221,v1:172.21.15.146:6836/2285803221] conn(0x55e483c4c450 0x55e483c4c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:37.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.807+0000 7f02db400640 1 --2- 172.21.15.6:0/1730592704 >> [v2:172.21.15.146:6834/2285803221,v1:172.21.15.146:6836/2285803221] conn(0x55e483c4c450 0x55e483c4c830 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=95d6d0ea3d02c887 server_cookie=4104e53accb87bb3 in_seq=0 out_seq=0 2024-03-04T09:26:37.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.807+0000 7f02bfe00640 10 client.5226 ms_handle_connect on v2:172.21.15.146:6834/2285803221 2024-03-04T09:26:37.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.808+0000 7f02bfe00640 1 -- 172.21.15.6:0/1730592704 <== mds.0 v2:172.21.15.146:6834/2285803221 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x55e483b03e90 con 0x55e483c4c450 2024-03-04T09:26:37.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.808+0000 7f02bfe00640 10 client.5226 handle_command_reply: tid=0 2024-03-04T09:26:37.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.808+0000 7f02e4fea740 1 client.5226 shutdown 2024-03-04T09:26:37.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.808+0000 7f02e4fea740 1 -- 172.21.15.6:0/1730592704 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e483bbf990 msgr2=0x55e483bedeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:37.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.808+0000 7f02e4fea740 1 --2- 172.21.15.6:0/1730592704 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e483bbf990 0x55e483bedeb0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f02c4004c70 tx=0x7f02c4001690 comp rx=0 tx=0).stop 2024-03-04T09:26:37.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.808+0000 7f02e4fea740 1 -- 172.21.15.6:0/1730592704 shutdown_connections 2024-03-04T09:26:37.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.808+0000 7f02e4fea740 1 --2- 172.21.15.6:0/1730592704 >> [v2:172.21.15.146:6834/2285803221,v1:172.21.15.146:6836/2285803221] conn(0x55e483c4c450 0x55e483c4c830 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:37.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.808+0000 7f02e4fea740 1 --2- 172.21.15.6:0/1730592704 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55e483bbfd70 0x55e483bee3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:37.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.808+0000 7f02e4fea740 1 --2- 172.21.15.6:0/1730592704 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e483bbf990 0x55e483bedeb0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:37.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.808+0000 7f02e4fea740 1 --2- 172.21.15.6:0/1730592704 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55e483bb2de0 0x55e483bbe7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:37.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.808+0000 7f02bfe00640 0 client.5226 ms_handle_reset on v2:172.21.15.146:6834/2285803221 2024-03-04T09:26:37.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.808+0000 7f02e4fea740 1 -- 172.21.15.6:0/1730592704 >> 172.21.15.6:0/1730592704 conn(0x55e483b92e70 msgr2=0x55e483b968b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:37.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.808+0000 7f02e4fea740 1 -- 172.21.15.6:0/1730592704 shutdown_connections 2024-03-04T09:26:37.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.808+0000 7f02e4fea740 1 -- 172.21.15.6:0/1730592704 wait complete. 2024-03-04T09:26:37.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.808+0000 7f02e4fea740 20 client.5226 trim_cache size 0 max 16384 2024-03-04T09:26:37.808 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-04T09:26:37.809 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_fsid": "edce0578-5ae4-4402-b45a-552bdef84d7c", 2024-03-04T09:26:37.809 INFO:teuthology.orchestra.run.smithi006.stdout: "whoami": 0, 2024-03-04T09:26:37.809 INFO:teuthology.orchestra.run.smithi006.stdout: "id": 5046, 2024-03-04T09:26:37.809 INFO:teuthology.orchestra.run.smithi006.stdout: "want_state": "up:active", 2024-03-04T09:26:37.809 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "up:active", 2024-03-04T09:26:37.809 INFO:teuthology.orchestra.run.smithi006.stdout: "fs_name": "cephfs", 2024-03-04T09:26:37.809 INFO:teuthology.orchestra.run.smithi006.stdout: "rank_uptime": 6.8306050950000001, 2024-03-04T09:26:37.809 INFO:teuthology.orchestra.run.smithi006.stdout: "mdsmap_epoch": 16, 2024-03-04T09:26:37.809 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap_epoch": 33, 2024-03-04T09:26:37.809 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap_epoch_barrier": 33, 2024-03-04T09:26:37.809 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 15.656523483999999 2024-03-04T09:26:37.809 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-04T09:26:37.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.809+0000 7f02e0c00640 1 -- 172.21.15.6:0/867599869 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f02b806b4d0 msgr2=0x7f02b806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:37.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.809+0000 7f02e0c00640 1 --2- 172.21.15.6:0/867599869 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f02b806b4d0 0x7f02b806d990 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f02dc143c50 tx=0x7f02c40018f0 comp rx=0 tx=0).stop 2024-03-04T09:26:37.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.809+0000 7f02e0c00640 1 -- 172.21.15.6:0/867599869 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f02dc070f10 msgr2=0x7f02dc15b3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:37.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.809+0000 7f02e0c00640 1 --2- 172.21.15.6:0/867599869 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f02dc070f10 0x7f02dc15b3d0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f02cc00c4b0 tx=0x7f02cc00c970 comp rx=0 tx=0).stop 2024-03-04T09:26:37.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.809+0000 7f02e0c00640 1 -- 172.21.15.6:0/867599869 shutdown_connections 2024-03-04T09:26:37.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.809+0000 7f02e0c00640 1 --2- 172.21.15.6:0/867599869 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f02b806b4d0 0x7f02b806d990 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:37.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.809+0000 7f02e0c00640 1 --2- 172.21.15.6:0/867599869 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f02dc145300 0x7f02dc0ae8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:37.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.809+0000 7f02e0c00640 1 --2- 172.21.15.6:0/867599869 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f02dc0cd060 0x7f02dc13af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:37.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.809+0000 7f02e0c00640 1 --2- 172.21.15.6:0/867599869 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f02dc070f10 0x7f02dc15b3d0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:37.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.809+0000 7f02e0c00640 1 -- 172.21.15.6:0/867599869 >> 172.21.15.6:0/867599869 conn(0x7f02dc06b3d0 msgr2=0x7f02dc0cc950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:37.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.809+0000 7f02e0c00640 1 -- 172.21.15.6:0/867599869 shutdown_connections 2024-03-04T09:26:37.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:37.809+0000 7f02e0c00640 1 -- 172.21.15.6:0/867599869 wait complete. 2024-03-04T09:26:37.823 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-04T09:26:38.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:38.047+0000 7ff564c00640 1 Processor -- start 2024-03-04T09:26:38.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:38.047+0000 7ff564c00640 1 -- start start 2024-03-04T09:26:38.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:38.049+0000 7ff564c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff5600cd060 0x7ff560161ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:38.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:38.049+0000 7ff564c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff560162000 0x7ff560070c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:38.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:38.049+0000 7ff564c00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5601627f0 con 0x7ff56015efc0 2024-03-04T09:26:38.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:38.049+0000 7ff564c00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff56015f3a0 con 0x7ff5600cd060 2024-03-04T09:26:38.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:38.049+0000 7ff564c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff56006d190 con 0x7ff560162000 2024-03-04T09:26:38.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:38.049+0000 7ff55ea00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff5600cd060 0x7ff560161ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:38.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:38.049+0000 7ff55ea00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff5600cd060 0x7ff560161ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:40856/0 (socket says 172.21.15.6:40856) 2024-03-04T09:26:38.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:38.049+0000 7ff55ea00640 1 -- 172.21.15.6:0/1909236115 learned_addr learned my addr 172.21.15.6:0/1909236115 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:38.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:38.050+0000 7ff55ea00640 1 -- 172.21.15.6:0/1909236115 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff560162000 msgr2=0x7ff560070c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:38.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:38.050+0000 7ff55fe00640 1 --2- 172.21.15.6:0/1909236115 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff560162000 0x7ff560070c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:38.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:38.050+0000 7ff55ea00640 1 --2- 172.21.15.6:0/1909236115 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff560162000 0x7ff560070c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:38.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:38.050+0000 7ff55ea00640 1 -- 172.21.15.6:0/1909236115 >> v1:172.21.15.146:6789/0 conn(0x7ff56015efc0 legacy=0x7ff5600cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:38.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:38.050+0000 7ff55ea00640 1 -- 172.21.15.6:0/1909236115 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff56006c860 con 0x7ff5600cd060 2024-03-04T09:26:38.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:38.051+0000 7ff55ea00640 1 --2- 172.21.15.6:0/1909236115 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff5600cd060 0x7ff560161ac0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7ff54c009870 tx=0x7ff54c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5b141b7f9e192df server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:38.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:38.052+0000 7ff55e000640 1 -- 172.21.15.6:0/1909236115 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff54c035020 con 0x7ff5600cd060 2024-03-04T09:26:38.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:38.052+0000 7ff55e000640 1 -- 172.21.15.6:0/1909236115 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff54c031640 con 0x7ff5600cd060 2024-03-04T09:26:38.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:38.052+0000 7ff55e000640 1 -- 172.21.15.6:0/1909236115 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff54c02d040 con 0x7ff5600cd060 2024-03-04T09:26:38.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:38.053+0000 7ff564c00640 1 -- 172.21.15.6:0/1909236115 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff5600cd060 msgr2=0x7ff560161ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:38.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:38.053+0000 7ff564c00640 1 --2- 172.21.15.6:0/1909236115 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff5600cd060 0x7ff560161ac0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7ff54c009870 tx=0x7ff54c02b8f0 comp rx=0 tx=0).stop 2024-03-04T09:26:38.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:38.053+0000 7ff564c00640 1 -- 172.21.15.6:0/1909236115 shutdown_connections 2024-03-04T09:26:38.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:38.053+0000 7ff564c00640 1 --2- 172.21.15.6:0/1909236115 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff560162000 0x7ff560070c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:38.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:38.053+0000 7ff564c00640 1 --2- 172.21.15.6:0/1909236115 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff5600cd060 0x7ff560161ac0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:38.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:38.053+0000 7ff564c00640 1 -- 172.21.15.6:0/1909236115 >> 172.21.15.6:0/1909236115 conn(0x7ff56006b3d0 msgr2=0x7ff56006b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:38.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:38.053+0000 7ff564c00640 1 -- 172.21.15.6:0/1909236115 shutdown_connections 2024-03-04T09:26:38.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:38.053+0000 7ff564c00640 1 -- 172.21.15.6:0/1909236115 wait complete. 2024-03-04T09:26:38.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:38.055+0000 7ff564c00640 1 Processor -- start 2024-03-04T09:26:38.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:38.056+0000 7ff564c00640 1 -- start start 2024-03-04T09:26:38.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:38.057+0000 7ff564c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff5600cd060 0x7ff5601ea510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:38.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:38.057+0000 7ff564c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff56015efc0 0x7ff56013a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:38.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:38.057+0000 7ff564c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff560162000 0x7ff56015c400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:38.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:38.057+0000 7ff564c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5601624b0 con 0x7ff5600cd060 2024-03-04T09:26:38.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:38.057+0000 7ff564c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff56006ce20 con 0x7ff560162000 2024-03-04T09:26:38.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:38.057+0000 7ff564c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff56006d620 con 0x7ff56015efc0 2024-03-04T09:26:38.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:38.057+0000 7ff55f400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff5600cd060 0x7ff5601ea510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:38.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:38.057+0000 7ff55f400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff5600cd060 0x7ff5601ea510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:40866/0 (socket says 172.21.15.6:40866) 2024-03-04T09:26:38.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:38.057+0000 7ff55f400640 1 -- 172.21.15.6:0/568901265 learned_addr learned my addr 172.21.15.6:0/568901265 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:38.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:38.058+0000 7ff55ea00640 1 --2- 172.21.15.6:0/568901265 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff56015efc0 0x7ff56013a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:38.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:38.058+0000 7ff55fe00640 1 --2- 172.21.15.6:0/568901265 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff560162000 0x7ff56015c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:38.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:38.058+0000 7ff55f400640 1 -- 172.21.15.6:0/568901265 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff56015efc0 msgr2=0x7ff56013a3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:38.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:38.058+0000 7ff55f400640 1 --2- 172.21.15.6:0/568901265 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff56015efc0 0x7ff56013a3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:38.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:38.058+0000 7ff55f400640 1 -- 172.21.15.6:0/568901265 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff560162000 msgr2=0x7ff56015c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:38.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:38.058+0000 7ff55f400640 1 --2- 172.21.15.6:0/568901265 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff560162000 0x7ff56015c400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:38.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:38.058+0000 7ff55f400640 1 -- 172.21.15.6:0/568901265 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff56015cd30 con 0x7ff5600cd060 2024-03-04T09:26:38.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:38.058+0000 7ff55f400640 1 --2- 172.21.15.6:0/568901265 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff5600cd060 0x7ff5601ea510 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7ff55000c460 tx=0x7ff55000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:38.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:38.059+0000 7ff557400640 1 -- 172.21.15.6:0/568901265 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff550016020 con 0x7ff5600cd060 2024-03-04T09:26:38.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:38.059+0000 7ff557400640 1 -- 172.21.15.6:0/568901265 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff550012870 con 0x7ff5600cd060 2024-03-04T09:26:38.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:38.059+0000 7ff557400640 1 -- 172.21.15.6:0/568901265 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff550012c60 con 0x7ff5600cd060 2024-03-04T09:26:38.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:38.059+0000 7ff564c00640 1 -- 172.21.15.6:0/568901265 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff56015d010 con 0x7ff5600cd060 2024-03-04T09:26:38.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:38.059+0000 7ff564c00640 1 -- 172.21.15.6:0/568901265 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff56015d5d0 con 0x7ff5600cd060 2024-03-04T09:26:38.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:38.060+0000 7ff564c00640 1 -- 172.21.15.6:0/568901265 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff56004f330 con 0x7ff5600cd060 2024-03-04T09:26:38.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:38.061+0000 7ff557400640 1 -- 172.21.15.6:0/568901265 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff550019020 con 0x7ff5600cd060 2024-03-04T09:26:38.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:38.062+0000 7ff557400640 1 --2- 172.21.15.6:0/568901265 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff54006b4d0 0x7ff54006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:38.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:38.062+0000 7ff557400640 1 -- 172.21.15.6:0/568901265 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff550089e90 con 0x7ff5600cd060 2024-03-04T09:26:38.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:38.063+0000 7ff55ea00640 1 --2- 172.21.15.6:0/568901265 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff54006b4d0 0x7ff54006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:38.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:38.064+0000 7ff55ea00640 1 --2- 172.21.15.6:0/568901265 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff54006b4d0 0x7ff54006d990 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7ff54c036000 tx=0x7ff54c03c040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:38.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:38.066+0000 7ff557400640 1 -- 172.21.15.6:0/568901265 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff550020030 con 0x7ff5600cd060 2024-03-04T09:26:38.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:38.307+0000 7ff564c00640 1 -- 172.21.15.6:0/568901265 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7ff56006e2a0 con 0x7ff5600cd060 2024-03-04T09:26:38.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:38.308+0000 7ff557400640 1 -- 172.21.15.6:0/568901265 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v16) v1 ==== 53+0+73 (secure 0 0 0) 0x7ff550058da0 con 0x7ff5600cd060 2024-03-04T09:26:38.305 INFO:teuthology.orchestra.run.smithi006.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-04T09:26:38.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:38.311+0000 7ff564c00640 1 -- 172.21.15.6:0/568901265 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff54006b4d0 msgr2=0x7ff54006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:38.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:38.311+0000 7ff564c00640 1 --2- 172.21.15.6:0/568901265 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff54006b4d0 0x7ff54006d990 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7ff54c036000 tx=0x7ff54c03c040 comp rx=0 tx=0).stop 2024-03-04T09:26:38.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:38.311+0000 7ff564c00640 1 -- 172.21.15.6:0/568901265 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff5600cd060 msgr2=0x7ff5601ea510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:38.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:38.311+0000 7ff564c00640 1 --2- 172.21.15.6:0/568901265 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff5600cd060 0x7ff5601ea510 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7ff55000c460 tx=0x7ff55000c920 comp rx=0 tx=0).stop 2024-03-04T09:26:38.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:38.311+0000 7ff564c00640 1 -- 172.21.15.6:0/568901265 shutdown_connections 2024-03-04T09:26:38.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:38.311+0000 7ff564c00640 1 --2- 172.21.15.6:0/568901265 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff560162000 0x7ff56015c400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:38.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:38.311+0000 7ff564c00640 1 --2- 172.21.15.6:0/568901265 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff54006b4d0 0x7ff54006d990 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:38.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:38.311+0000 7ff564c00640 1 --2- 172.21.15.6:0/568901265 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff56015efc0 0x7ff56013a3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:38.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:38.311+0000 7ff564c00640 1 --2- 172.21.15.6:0/568901265 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff5600cd060 0x7ff5601ea510 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:38.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:38.311+0000 7ff564c00640 1 -- 172.21.15.6:0/568901265 >> 172.21.15.6:0/568901265 conn(0x7ff56006b3d0 msgr2=0x7ff5600719a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:38.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:38.311+0000 7ff564c00640 1 -- 172.21.15.6:0/568901265 shutdown_connections 2024-03-04T09:26:38.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:38.311+0000 7ff564c00640 1 -- 172.21.15.6:0/568901265 wait complete. 2024-03-04T09:26:38.326 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-04T09:26:38.326 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-04T09:26:38.326 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi006.front.sepia.ceph.com 2024-03-04T09:26:38.326 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-04T09:26:38.326 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-04T09:26:38.326 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-04T09:26:38.326 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-03-04T09:26:38.326 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-04T09:26:38.326 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-04T09:26:38.327 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:26:38.327 DEBUG:teuthology.orchestra.run.smithi006:> ip addr 2024-03-04T09:26:38.379 INFO:teuthology.orchestra.run.smithi006.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-04T09:26:38.379 INFO:teuthology.orchestra.run.smithi006.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-04T09:26:38.379 INFO:teuthology.orchestra.run.smithi006.stdout: inet 127.0.0.1/8 scope host lo 2024-03-04T09:26:38.379 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-04T09:26:38.379 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 ::1/128 scope host 2024-03-04T09:26:38.379 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-04T09:26:38.379 INFO:teuthology.orchestra.run.smithi006.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-04T09:26:38.379 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:6b:fe:e6 brd ff:ff:ff:ff:ff:ff 2024-03-04T09:26:38.380 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp7s0f0 2024-03-04T09:26:38.380 INFO:teuthology.orchestra.run.smithi006.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-04T09:26:38.380 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:6b:fe:e7 brd ff:ff:ff:ff:ff:ff 2024-03-04T09:26:38.380 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp7s0f1 2024-03-04T09:26:38.380 INFO:teuthology.orchestra.run.smithi006.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-04T09:26:38.380 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:bd:15:e4 brd ff:ff:ff:ff:ff:ff 2024-03-04T09:26:38.380 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp1s0f0 2024-03-04T09:26:38.380 INFO:teuthology.orchestra.run.smithi006.stdout: inet 172.21.15.6/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-04T09:26:38.380 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft 42536sec preferred_lft 42536sec 2024-03-04T09:26:38.380 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::ec4:7aff:febd:15e4/64 scope link noprefixroute 2024-03-04T09:26:38.380 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-04T09:26:38.380 INFO:teuthology.orchestra.run.smithi006.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-04T09:26:38.380 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:bd:15:e5 brd ff:ff:ff:ff:ff:ff 2024-03-04T09:26:38.381 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp1s0f1 2024-03-04T09:26:38.381 INFO:teuthology.orchestra.run.smithi006.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-04T09:26:38.381 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 5e:b4:59:dc:42:58 brd ff:ff:ff:ff:ff:ff 2024-03-04T09:26:38.381 INFO:teuthology.orchestra.run.smithi006.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-04T09:26:38.381 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-04T09:26:38.381 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::c6f:adff:fefe:e00d/64 scope link proto kernel_ll 2024-03-04T09:26:38.381 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-04T09:26:38.381 INFO:teuthology.orchestra.run.smithi006.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-04T09:26:38.381 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 7e:67:6d:ea:f9:ff brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-04T09:26:38.381 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::7c67:6dff:feea:f9ff/64 scope link proto kernel_ll 2024-03-04T09:26:38.381 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-04T09:26:38.381 INFO:teuthology.orchestra.run.smithi006.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-04T09:26:38.382 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 5e:b4:59:dc:42:58 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-04T09:26:38.382 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::5cb4:59ff:fedc:4258/64 scope link proto kernel_ll 2024-03-04T09:26:38.382 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-04T09:26:38.383 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:26:38.383 DEBUG:teuthology.orchestra.run.smithi006:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-04T09:26:38.476 INFO:teuthology.orchestra.run.smithi006.stdout:1 2024-03-04T09:26:38.481 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:26:38.481 DEBUG:teuthology.orchestra.run.smithi006:> route 2024-03-04T09:26:38.536 INFO:teuthology.orchestra.run.smithi006.stdout:Kernel IP routing table 2024-03-04T09:26:38.536 INFO:teuthology.orchestra.run.smithi006.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-04T09:26:38.536 INFO:teuthology.orchestra.run.smithi006.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-04T09:26:38.536 INFO:teuthology.orchestra.run.smithi006.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-04T09:26:38.536 INFO:teuthology.orchestra.run.smithi006.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-04T09:26:38.537 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:26:38.537 DEBUG:teuthology.orchestra.run.smithi006:> (cd / && exec bash -c ' 2024-03-04T09:26:38.537 DEBUG:teuthology.orchestra.run.smithi006:> set -e 2024-03-04T09:26:38.537 DEBUG:teuthology.orchestra.run.smithi006:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-04T09:26:38.537 DEBUG:teuthology.orchestra.run.smithi006:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-04T09:26:38.537 DEBUG:teuthology.orchestra.run.smithi006:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-04T09:26:38.537 DEBUG:teuthology.orchestra.run.smithi006:> ') 2024-03-04T09:26:38.732 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:26:38.732 DEBUG:teuthology.orchestra.run.smithi006:> ip netns list 2024-03-04T09:26:38.785 INFO:teuthology.orchestra.run.smithi006.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-04T09:26:38.785 INFO:teuthology.orchestra.run.smithi006.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-04T09:26:38.786 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-04T09:26:38.786 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:26:38.786 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip link set brx.0 up 2024-03-04T09:26:38.864 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:26:38.872 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:26:38.875 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-04T09:26:38.875 DEBUG:teuthology.orchestra.run.smithi006:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-04T09:26:38.963 INFO:teuthology.orchestra.run.smithi006.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-04T09:26:38.963 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-04T09:26:38.963 DEBUG:teuthology.orchestra.run.smithi006:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-04T09:26:39.014 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-03-04T09:26:39.014 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:26:39.014 DEBUG:teuthology.orchestra.run.smithi006:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-04T09:26:39.203 INFO:teuthology.orchestra.run.smithi006.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-04T09:26:39.204 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-04T09:26:39.239 INFO:teuthology.orchestra.run.smithi006.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-04T09:26:39.239 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-04T09:26:39.239 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: resolved to: "172.21.15.6:3300,172.21.15.146:3300,172.21.15.146:3301" 2024-03-04T09:26:39.240 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: trying mount with new device syntax: 0@edce0578-5ae4-4402-b45a-552bdef84d7c.cephfs=/ 2024-03-04T09:26:39.240 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.6:3300/172.21.15.146:3300/172.21.15.146:3301" will pass to kernel 2024-03-04T09:26:39.243 INFO:teuthology.orchestra.run.smithi006.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-04T09:26:39.244 INFO:teuthology.orchestra.run.smithi006.stdout: You just mounted a file system that supports labels which does not 2024-03-04T09:26:39.244 INFO:teuthology.orchestra.run.smithi006.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-04T09:26:39.244 INFO:teuthology.orchestra.run.smithi006.stdout: applications will generate AVC messages and not be allowed access to 2024-03-04T09:26:39.244 INFO:teuthology.orchestra.run.smithi006.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-04T09:26:39.248 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-04T09:26:39.248 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:26:39.249 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-04T09:26:39.329 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:26:39.329 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-04T09:26:39.404 INFO:teuthology.orchestra.run.smithi006.stdout:client4787 2024-03-04T09:26:39.404 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-04T09:26:39.404 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T09:26:39.462 DEBUG:teuthology.orchestra.run.smithi006:> stdin-killer -- python3 -c ' 2024-03-04T09:26:39.462 DEBUG:teuthology.orchestra.run.smithi006:> import os 2024-03-04T09:26:39.463 DEBUG:teuthology.orchestra.run.smithi006:> 2024-03-04T09:26:39.463 DEBUG:teuthology.orchestra.run.smithi006:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-04T09:26:39.463 DEBUG:teuthology.orchestra.run.smithi006:> subdir = "delete_me" 2024-03-04T09:26:39.463 DEBUG:teuthology.orchestra.run.smithi006:> size = 1024 2024-03-04T09:26:39.463 DEBUG:teuthology.orchestra.run.smithi006:> file_count = 1000 2024-03-04T09:26:39.463 DEBUG:teuthology.orchestra.run.smithi006:> os.mkdir(os.path.join(mountpoint, subdir)) 2024-03-04T09:26:39.463 DEBUG:teuthology.orchestra.run.smithi006:> for i in range(0, file_count): 2024-03-04T09:26:39.463 DEBUG:teuthology.orchestra.run.smithi006:> filename = "{0}_{1}.bin".format(i, size) 2024-03-04T09:26:39.463 DEBUG:teuthology.orchestra.run.smithi006:> with open(os.path.join(mountpoint, subdir, filename), '"'"'w'"'"') as f: 2024-03-04T09:26:39.463 DEBUG:teuthology.orchestra.run.smithi006:> f.write(size * '"'"'x'"'"') 2024-03-04T09:26:39.463 DEBUG:teuthology.orchestra.run.smithi006:> ' 2024-03-04T09:26:40.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:40 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-04T09:26:40.658 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:26:40.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:40.924+0000 7faf6fe00640 1 Processor -- start 2024-03-04T09:26:40.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:40.924+0000 7faf6fe00640 1 -- start start 2024-03-04T09:26:40.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:40.925+0000 7faf6fe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7faf700cd060 0x7faf701667e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:40.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:40.925+0000 7faf6fe00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf700714d0 con 0x7faf70070f10 2024-03-04T09:26:40.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:40.925+0000 7faf6fe00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf700717b0 con 0x7faf70145300 2024-03-04T09:26:40.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:40.925+0000 7faf6fe00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf70071a90 con 0x7faf700cd060 2024-03-04T09:26:40.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:40.926+0000 7faf67e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7faf70070f10 0x7faf700cc950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:57314/0 (socket says 172.21.15.6:57314) 2024-03-04T09:26:40.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:40.926+0000 7faf67e00640 1 -- 172.21.15.6:0/3950552959 learned_addr learned my addr 172.21.15.6:0/3950552959 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:26:40.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:40.926+0000 7faf6f400640 1 --2- 172.21.15.6:0/3950552959 >> v2:172.21.15.6:3300/0 conn(0x7faf700cd060 0x7faf701667e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:40.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:40.926+0000 7faf6e000640 1 -- 172.21.15.6:0/3950552959 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2573566893 0 0) 0x7faf700714d0 con 0x7faf70070f10 2024-03-04T09:26:40.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:40.926+0000 7faf6e000640 1 -- 172.21.15.6:0/3950552959 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf54003520 con 0x7faf70070f10 2024-03-04T09:26:40.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:40.926+0000 7faf6f400640 1 -- 172.21.15.6:0/3950552959 >> v1:172.21.15.146:6789/0 conn(0x7faf70145300 legacy=0x7faf700b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:40.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:40.926+0000 7faf6f400640 1 -- 172.21.15.6:0/3950552959 >> v1:172.21.15.6:6789/0 conn(0x7faf70070f10 legacy=0x7faf700cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:40.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:40.926+0000 7faf6f400640 1 -- 172.21.15.6:0/3950552959 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf70071d70 con 0x7faf700cd060 2024-03-04T09:26:40.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:40.927+0000 7faf6f400640 1 --2- 172.21.15.6:0/3950552959 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faf700cd060 0x7faf701667e0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7faf68009d90 tx=0x7faf6802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b4006c6a3fa2821 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:40.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:40.928+0000 7faf6e000640 1 -- 172.21.15.6:0/3950552959 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf68039070 con 0x7faf700cd060 2024-03-04T09:26:40.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:40.928+0000 7faf6e000640 1 -- 172.21.15.6:0/3950552959 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf6802cc60 con 0x7faf700cd060 2024-03-04T09:26:40.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:40.928+0000 7faf6e000640 1 -- 172.21.15.6:0/3950552959 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf68034660 con 0x7faf700cd060 2024-03-04T09:26:40.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:40.928+0000 7faf6fe00640 1 -- 172.21.15.6:0/3950552959 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faf700cd060 msgr2=0x7faf701667e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:40.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:40.928+0000 7faf6fe00640 1 --2- 172.21.15.6:0/3950552959 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faf700cd060 0x7faf701667e0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7faf68009d90 tx=0x7faf6802beb0 comp rx=0 tx=0).stop 2024-03-04T09:26:40.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:40.929+0000 7faf6fe00640 1 -- 172.21.15.6:0/3950552959 shutdown_connections 2024-03-04T09:26:40.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:40.929+0000 7faf6fe00640 1 --2- 172.21.15.6:0/3950552959 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faf700cd060 0x7faf701667e0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:40.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:40.929+0000 7faf6fe00640 1 -- 172.21.15.6:0/3950552959 >> 172.21.15.6:0/3950552959 conn(0x7faf7006b3d0 msgr2=0x7faf7006b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:40.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:40.929+0000 7faf6fe00640 1 -- 172.21.15.6:0/3950552959 shutdown_connections 2024-03-04T09:26:40.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:40.929+0000 7faf6fe00640 1 -- 172.21.15.6:0/3950552959 wait complete. 2024-03-04T09:26:40.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:40.930+0000 7faf6fe00640 1 Processor -- start 2024-03-04T09:26:40.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:40.931+0000 7faf6fe00640 1 -- start start 2024-03-04T09:26:40.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:40.931+0000 7faf6fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faf70070f10 0x7faf700a9e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:40.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:40.931+0000 7faf6fe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faf700cd060 0x7faf7013e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:40.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:40.931+0000 7faf6fe00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faf70145300 0x7faf7013c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:40.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:40.931+0000 7faf6fe00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf7015d700 con 0x7faf70070f10 2024-03-04T09:26:40.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:40.931+0000 7faf6fe00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf7015d580 con 0x7faf700cd060 2024-03-04T09:26:40.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:40.932+0000 7faf6fe00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf7015d880 con 0x7faf70145300 2024-03-04T09:26:40.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:40.932+0000 7faf6ea00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faf70070f10 0x7faf700a9e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:40.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:40.932+0000 7faf6ea00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faf70070f10 0x7faf700a9e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:40602/0 (socket says 172.21.15.6:40602) 2024-03-04T09:26:40.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:40.932+0000 7faf6ea00640 1 -- 172.21.15.6:0/2982627840 learned_addr learned my addr 172.21.15.6:0/2982627840 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:40.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:40.932+0000 7faf6f400640 1 --2- 172.21.15.6:0/2982627840 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faf70145300 0x7faf7013c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:40.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:40.932+0000 7faf67e00640 1 --2- 172.21.15.6:0/2982627840 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faf700cd060 0x7faf7013e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:40.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:40.932+0000 7faf6ea00640 1 -- 172.21.15.6:0/2982627840 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faf70145300 msgr2=0x7faf7013c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:40.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:40.932+0000 7faf6ea00640 1 --2- 172.21.15.6:0/2982627840 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faf70145300 0x7faf7013c470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:40.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:40.932+0000 7faf6ea00640 1 -- 172.21.15.6:0/2982627840 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faf700cd060 msgr2=0x7faf7013e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:40.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:40.932+0000 7faf6ea00640 1 --2- 172.21.15.6:0/2982627840 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faf700cd060 0x7faf7013e140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:40.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:40.932+0000 7faf6ea00640 1 -- 172.21.15.6:0/2982627840 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf700ab620 con 0x7faf70070f10 2024-03-04T09:26:40.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:40.933+0000 7faf6ea00640 1 --2- 172.21.15.6:0/2982627840 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faf70070f10 0x7faf700a9e80 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7faf6000c3c0 tx=0x7faf6000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:40.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:40.933+0000 7faf66a00640 1 -- 172.21.15.6:0/2982627840 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf600052d0 con 0x7faf70070f10 2024-03-04T09:26:40.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:40.933+0000 7faf66a00640 1 -- 172.21.15.6:0/2982627840 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf60005490 con 0x7faf70070f10 2024-03-04T09:26:40.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:40.933+0000 7faf66a00640 1 -- 172.21.15.6:0/2982627840 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf60007260 con 0x7faf70070f10 2024-03-04T09:26:40.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:40.933+0000 7faf6fe00640 1 -- 172.21.15.6:0/2982627840 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf700aa580 con 0x7faf70070f10 2024-03-04T09:26:40.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:40.933+0000 7faf6fe00640 1 -- 172.21.15.6:0/2982627840 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf700aab40 con 0x7faf70070f10 2024-03-04T09:26:40.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:40.934+0000 7faf6fe00640 1 -- 172.21.15.6:0/2982627840 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf7004f330 con 0x7faf70070f10 2024-03-04T09:26:40.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:40.935+0000 7faf66a00640 1 -- 172.21.15.6:0/2982627840 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7faf60007400 con 0x7faf70070f10 2024-03-04T09:26:40.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:40.936+0000 7faf66a00640 1 --2- 172.21.15.6:0/2982627840 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7faf4c06b4d0 0x7faf4c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:40.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:40.936+0000 7faf66a00640 1 -- 172.21.15.6:0/2982627840 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7faf60089690 con 0x7faf70070f10 2024-03-04T09:26:40.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:40.936+0000 7faf67e00640 1 --2- 172.21.15.6:0/2982627840 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7faf4c06b4d0 0x7faf4c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:40.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:40.937+0000 7faf67e00640 1 --2- 172.21.15.6:0/2982627840 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7faf4c06b4d0 0x7faf4c06d990 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7faf7013e7f0 tx=0x7faf58009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:40.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:40.941+0000 7faf66a00640 1 -- 172.21.15.6:0/2982627840 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7faf60057e20 con 0x7faf70070f10 2024-03-04T09:26:41.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:41.181+0000 7faf6fe00640 1 -- 172.21.15.6:0/2982627840 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faf701448b0 con 0x7faf70070f10 2024-03-04T09:26:41.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:41.182+0000 7faf66a00640 1 -- 172.21.15.6:0/2982627840 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+4977 (secure 0 0 0) 0x7faf6005bec0 con 0x7faf70070f10 2024-03-04T09:26:41.179 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:26:41.179 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4685,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/1703428647","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":1703428647},{"type":"v1","addr":"172.21.15.146:6837","nonce":1703428647}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4837,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/2617429530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2617429530},{"type":"v1","addr":"172.21.15.6:6833","nonce":2617429530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":5043,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/2744913857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":2744913857},{"type":"v1","addr":"172.21.15.6:6835","nonce":2744913857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:26:30.445199+0000","modified":"2024-03-04T09:26:33.534893+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5046},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5046":{"gid":5046,"name":"d","rank":0,"incarnation":12,"state":"up:active","state_seq":4,"addr":"172.21.15.146:6836/2285803221","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":2285803221},{"type":"v1","addr":"172.21.15.146:6836","nonce":2285803221}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5046,"qdb_cluster":[5046]},"id":2}]} 2024-03-04T09:26:41.180 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 16 2024-03-04T09:26:41.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:41.185+0000 7faf6fe00640 1 -- 172.21.15.6:0/2982627840 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7faf4c06b4d0 msgr2=0x7faf4c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:41.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:41.185+0000 7faf6fe00640 1 --2- 172.21.15.6:0/2982627840 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7faf4c06b4d0 0x7faf4c06d990 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7faf7013e7f0 tx=0x7faf58009040 comp rx=0 tx=0).stop 2024-03-04T09:26:41.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:41.185+0000 7faf6fe00640 1 -- 172.21.15.6:0/2982627840 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faf70070f10 msgr2=0x7faf700a9e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:41.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:41.185+0000 7faf6fe00640 1 --2- 172.21.15.6:0/2982627840 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faf70070f10 0x7faf700a9e80 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7faf6000c3c0 tx=0x7faf6000c880 comp rx=0 tx=0).stop 2024-03-04T09:26:41.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:41.185+0000 7faf6fe00640 1 -- 172.21.15.6:0/2982627840 shutdown_connections 2024-03-04T09:26:41.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:41.185+0000 7faf6fe00640 1 --2- 172.21.15.6:0/2982627840 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7faf4c06b4d0 0x7faf4c06d990 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:41.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:41.185+0000 7faf6fe00640 1 --2- 172.21.15.6:0/2982627840 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faf70145300 0x7faf7013c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:41.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:41.185+0000 7faf6fe00640 1 --2- 172.21.15.6:0/2982627840 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faf700cd060 0x7faf7013e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:41.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:41.185+0000 7faf6fe00640 1 --2- 172.21.15.6:0/2982627840 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faf70070f10 0x7faf700a9e80 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:41.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:41.185+0000 7faf6fe00640 1 -- 172.21.15.6:0/2982627840 >> 172.21.15.6:0/2982627840 conn(0x7faf7006b3d0 msgr2=0x7faf7015da00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:41.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:41.185+0000 7faf6fe00640 1 -- 172.21.15.6:0/2982627840 shutdown_connections 2024-03-04T09:26:41.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:41.185+0000 7faf6fe00640 1 -- 172.21.15.6:0/2982627840 wait complete. 2024-03-04T09:26:41.201 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json flush journal 2024-03-04T09:26:41.784 INFO:teuthology.orchestra.run.smithi146.stdout:{"message":"","return_code":0} 2024-03-04T09:26:41.785 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-04T09:26:41.785 DEBUG:teuthology.orchestra.run.smithi006:> stdin-killer -- python3 -c ' 2024-03-04T09:26:41.785 DEBUG:teuthology.orchestra.run.smithi006:> import os 2024-03-04T09:26:41.785 DEBUG:teuthology.orchestra.run.smithi006:> import stat 2024-03-04T09:26:41.785 DEBUG:teuthology.orchestra.run.smithi006:> 2024-03-04T09:26:41.785 DEBUG:teuthology.orchestra.run.smithi006:> print(os.stat("/home/ubuntu/cephtest/mnt.0/delete_me").st_ino) 2024-03-04T09:26:41.786 DEBUG:teuthology.orchestra.run.smithi006:> ' 2024-03-04T09:26:41.896 INFO:teuthology.orchestra.run.smithi006.stdout:1099511627776 2024-03-04T09:26:41.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:41 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-04T09:26:41.909 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata stat 10000000000.00000000 2024-03-04T09:26:42.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:42.070+0000 7f5d13843f40 1 Processor -- start 2024-03-04T09:26:42.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:42.070+0000 7f5d13843f40 1 -- start start 2024-03-04T09:26:42.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:42.071+0000 7f5d13843f40 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55aed86da0d0 0x55aed87965d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:42.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:42.071+0000 7f5d13843f40 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55aed8727cf0 0x55aed879c170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:42.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:42.071+0000 7f5d13843f40 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55aed8728810 0x55aed879c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:42.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:42.071+0000 7f5d13843f40 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aed86dba50 con 0x55aed86da0d0 2024-03-04T09:26:42.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:42.071+0000 7f5d13843f40 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aed8682fd0 con 0x55aed8727cf0 2024-03-04T09:26:42.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:42.071+0000 7f5d13843f40 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x55aed8728c20 con 0x55aed8728810 2024-03-04T09:26:42.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:42.072+0000 7f5d10200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55aed86da0d0 0x55aed87965d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:42.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:42.072+0000 7f5d10200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55aed86da0d0 0x55aed87965d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:40620/0 (socket says 172.21.15.6:40620) 2024-03-04T09:26:42.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:42.072+0000 7f5d10200640 1 -- 172.21.15.6:0/3487130595 learned_addr learned my addr 172.21.15.6:0/3487130595 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:42.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:42.072+0000 7f5d10c00640 1 --2- 172.21.15.6:0/3487130595 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55aed8728810 0x55aed879c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:42.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:42.072+0000 7f5d07e00640 1 --2- 172.21.15.6:0/3487130595 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55aed8727cf0 0x55aed879c170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:42.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:42.072+0000 7f5d10200640 1 -- 172.21.15.6:0/3487130595 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55aed8728810 msgr2=0x55aed879c6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:42.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:42.072+0000 7f5d10200640 1 --2- 172.21.15.6:0/3487130595 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55aed8728810 0x55aed879c6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:42.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:42.072+0000 7f5d10200640 1 -- 172.21.15.6:0/3487130595 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55aed8727cf0 msgr2=0x55aed879c170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:42.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:42.072+0000 7f5d10200640 1 --2- 172.21.15.6:0/3487130595 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55aed8727cf0 0x55aed879c170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:42.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:42.072+0000 7f5d10200640 1 -- 172.21.15.6:0/3487130595 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55aed870c4d0 con 0x55aed86da0d0 2024-03-04T09:26:42.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:42.072+0000 7f5d07e00640 1 --2- 172.21.15.6:0/3487130595 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55aed8727cf0 0x55aed879c170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:26:42.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:42.072+0000 7f5d10200640 1 --2- 172.21.15.6:0/3487130595 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55aed86da0d0 0x55aed87965d0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f5d00055040 tx=0x7f5d0004c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ce3f86ec2075ed8 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:42.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:42.073+0000 7f5d0ee00640 1 -- 172.21.15.6:0/3487130595 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d00057070 con 0x55aed86da0d0 2024-03-04T09:26:42.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:42.073+0000 7f5d0ee00640 1 -- 172.21.15.6:0/3487130595 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d00061070 con 0x55aed86da0d0 2024-03-04T09:26:42.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:42.073+0000 7f5d0ee00640 1 -- 172.21.15.6:0/3487130595 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d000528a0 con 0x55aed86da0d0 2024-03-04T09:26:42.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:42.073+0000 7f5d13843f40 1 -- 172.21.15.6:0/3487130595 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55aed86da0d0 msgr2=0x55aed87965d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:42.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:42.073+0000 7f5d13843f40 1 --2- 172.21.15.6:0/3487130595 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55aed86da0d0 0x55aed87965d0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f5d00055040 tx=0x7f5d0004c000 comp rx=0 tx=0).stop 2024-03-04T09:26:42.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:42.074+0000 7f5d13843f40 1 -- 172.21.15.6:0/3487130595 shutdown_connections 2024-03-04T09:26:42.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:42.074+0000 7f5d13843f40 1 --2- 172.21.15.6:0/3487130595 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55aed8728810 0x55aed879c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:42.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:42.074+0000 7f5d13843f40 1 --2- 172.21.15.6:0/3487130595 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55aed8727cf0 0x55aed879c170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:42.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:42.074+0000 7f5d13843f40 1 --2- 172.21.15.6:0/3487130595 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55aed86da0d0 0x55aed87965d0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:42.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:42.074+0000 7f5d13843f40 1 -- 172.21.15.6:0/3487130595 >> 172.21.15.6:0/3487130595 conn(0x55aed870bc10 msgr2=0x55aed87280d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:42.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:42.074+0000 7f5d13843f40 1 -- 172.21.15.6:0/3487130595 shutdown_connections 2024-03-04T09:26:42.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:42.074+0000 7f5d13843f40 1 -- 172.21.15.6:0/3487130595 wait complete. 2024-03-04T09:26:42.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:42.074+0000 7f5d13843f40 1 Processor -- start 2024-03-04T09:26:42.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:42.075+0000 7f5d13843f40 1 -- start start 2024-03-04T09:26:42.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:42.076+0000 7f5d13843f40 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55aed86da0d0 0x55aed870e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:42.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:42.076+0000 7f5d13843f40 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55aed8727cf0 0x55aed8794c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:42.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:42.076+0000 7f5d13843f40 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55aed8728810 0x55aed8795180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:42.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:42.076+0000 7f5d13843f40 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aed8799730 con 0x55aed86da0d0 2024-03-04T09:26:42.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:42.076+0000 7f5d13843f40 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aed87b31d0 con 0x55aed8728810 2024-03-04T09:26:42.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:42.076+0000 7f5d13843f40 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x55aed870c1f0 con 0x55aed8727cf0 2024-03-04T09:26:42.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:42.076+0000 7f5d10200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55aed86da0d0 0x55aed870e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:42.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:42.076+0000 7f5d10200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55aed86da0d0 0x55aed870e190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:40628/0 (socket says 172.21.15.6:40628) 2024-03-04T09:26:42.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:42.076+0000 7f5d10200640 1 -- 172.21.15.6:0/2137395205 learned_addr learned my addr 172.21.15.6:0/2137395205 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:42.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:42.076+0000 7f5d10c00640 1 --2- 172.21.15.6:0/2137395205 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55aed8728810 0x55aed8795180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:42.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:42.076+0000 7f5d07e00640 1 --2- 172.21.15.6:0/2137395205 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55aed8727cf0 0x55aed8794c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:42.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:42.076+0000 7f5d10200640 1 -- 172.21.15.6:0/2137395205 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55aed8727cf0 msgr2=0x55aed8794c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:42.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:42.076+0000 7f5d10200640 1 --2- 172.21.15.6:0/2137395205 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55aed8727cf0 0x55aed8794c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:42.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:42.076+0000 7f5d10200640 1 -- 172.21.15.6:0/2137395205 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55aed8728810 msgr2=0x55aed8795180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:42.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:42.076+0000 7f5d10200640 1 --2- 172.21.15.6:0/2137395205 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55aed8728810 0x55aed8795180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:42.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:42.076+0000 7f5d10200640 1 -- 172.21.15.6:0/2137395205 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55aed8726230 con 0x55aed86da0d0 2024-03-04T09:26:42.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:42.077+0000 7f5d10200640 1 --2- 172.21.15.6:0/2137395205 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55aed86da0d0 0x55aed870e190 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f5d0001bf50 tx=0x7f5d00002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:42.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:42.077+0000 7f5d0da00640 1 -- 172.21.15.6:0/2137395205 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d00057050 con 0x55aed86da0d0 2024-03-04T09:26:42.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:42.077+0000 7f5d0da00640 1 -- 172.21.15.6:0/2137395205 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d00058040 con 0x55aed86da0d0 2024-03-04T09:26:42.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:42.077+0000 7f5d0da00640 1 -- 172.21.15.6:0/2137395205 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d00052de0 con 0x55aed86da0d0 2024-03-04T09:26:42.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:42.077+0000 7f5d13843f40 1 -- 172.21.15.6:0/2137395205 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55aed871f6a0 con 0x55aed86da0d0 2024-03-04T09:26:42.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:42.077+0000 7f5d13843f40 1 -- 172.21.15.6:0/2137395205 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55aed871fc60 con 0x55aed86da0d0 2024-03-04T09:26:42.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:42.079+0000 7f5d0da00640 1 -- 172.21.15.6:0/2137395205 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5d00065420 con 0x55aed86da0d0 2024-03-04T09:26:42.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:42.080+0000 7f5d0da00640 1 --2- 172.21.15.6:0/2137395205 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f5cec069eb0 0x7f5cec06c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:42.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:42.080+0000 7f5d0da00640 1 -- 172.21.15.6:0/2137395205 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5d000cb920 con 0x55aed86da0d0 2024-03-04T09:26:42.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:42.081+0000 7f5d13843f40 1 --2- 172.21.15.6:0/2137395205 >> [v2:172.21.15.6:6800/1558806123,v1:172.21.15.6:6801/1558806123] conn(0x55aed870e6d0 0x55aed87956c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:42.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:42.081+0000 7f5d13843f40 1 -- 172.21.15.6:0/2137395205 --> [v2:172.21.15.6:6800/1558806123,v1:172.21.15.6:6801/1558806123] -- osd_op(unknown.0.0:1 4.10 4:0cf6ad0f:::10000000000.00000000:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e33) v9 -- 0x55aed8720180 con 0x55aed870e6d0 2024-03-04T09:26:42.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:42.081+0000 7f5d07e00640 1 --2- 172.21.15.6:0/2137395205 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f5cec069eb0 0x7f5cec06c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:42.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:42.081+0000 7f5d10c00640 1 --2- 172.21.15.6:0/2137395205 >> [v2:172.21.15.6:6800/1558806123,v1:172.21.15.6:6801/1558806123] conn(0x55aed870e6d0 0x55aed87956c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:42.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:42.082+0000 7f5d10c00640 1 --2- 172.21.15.6:0/2137395205 >> [v2:172.21.15.6:6800/1558806123,v1:172.21.15.6:6801/1558806123] conn(0x55aed870e6d0 0x55aed87956c0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:42.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:42.082+0000 7f5d07e00640 1 --2- 172.21.15.6:0/2137395205 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f5cec069eb0 0x7f5cec06c370 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x55aed87b2f00 tx=0x7f5cf8001580 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:42.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:42.082+0000 7f5d10c00640 1 -- 172.21.15.6:0/2137395205 <== osd.3 v2:172.21.15.6:6800/1558806123 1 ==== osd_op_reply(1 10000000000.00000000 [stat out=16b] v0'0 uv10 ondisk = 0) v8 ==== 164+0+16 (crc 0 0 0) 0x7f5d08011040 con 0x55aed870e6d0 2024-03-04T09:26:42.079 INFO:teuthology.orchestra.run.smithi006.stdout:cephfs_metadata/10000000000.00000000 mtime 2024-03-04T09:26:41.000000+0000, size 0 2024-03-04T09:26:42.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:42.082+0000 7f5d13843f40 1 -- 172.21.15.6:0/2137395205 >> [v2:172.21.15.6:6800/1558806123,v1:172.21.15.6:6801/1558806123] conn(0x55aed870e6d0 msgr2=0x55aed87956c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:42.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:42.082+0000 7f5d13843f40 1 --2- 172.21.15.6:0/2137395205 >> [v2:172.21.15.6:6800/1558806123,v1:172.21.15.6:6801/1558806123] conn(0x55aed870e6d0 0x55aed87956c0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:42.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:42.082+0000 7f5d13843f40 1 -- 172.21.15.6:0/2137395205 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f5cec069eb0 msgr2=0x7f5cec06c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:42.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:42.082+0000 7f5d13843f40 1 --2- 172.21.15.6:0/2137395205 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f5cec069eb0 0x7f5cec06c370 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x55aed87b2f00 tx=0x7f5cf8001580 comp rx=0 tx=0).stop 2024-03-04T09:26:42.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:42.082+0000 7f5d13843f40 1 -- 172.21.15.6:0/2137395205 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55aed86da0d0 msgr2=0x55aed870e190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:42.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:42.082+0000 7f5d13843f40 1 --2- 172.21.15.6:0/2137395205 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55aed86da0d0 0x55aed870e190 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f5d0001bf50 tx=0x7f5d00002fb0 comp rx=0 tx=0).stop 2024-03-04T09:26:42.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:42.082+0000 7f5d10c00640 1 -- 172.21.15.6:0/2137395205 reap_dead start 2024-03-04T09:26:42.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:42.082+0000 7f5d13843f40 1 -- 172.21.15.6:0/2137395205 shutdown_connections 2024-03-04T09:26:42.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:42.082+0000 7f5d13843f40 1 -- 172.21.15.6:0/2137395205 >> 172.21.15.6:0/2137395205 conn(0x55aed870bc10 msgr2=0x55aed8797680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:42.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:42.083+0000 7f5d13843f40 1 -- 172.21.15.6:0/2137395205 shutdown_connections 2024-03-04T09:26:42.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:42.083+0000 7f5d13843f40 1 -- 172.21.15.6:0/2137395205 wait complete. 2024-03-04T09:26:42.084 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:26:42.084 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest/mnt.0 && exec rm -rf delete_me) 2024-03-04T09:26:45.982 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:26:46.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:46.252+0000 7fd273c00640 1 Processor -- start 2024-03-04T09:26:46.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:46.252+0000 7fd273c00640 1 -- start start 2024-03-04T09:26:46.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:46.255+0000 7fd273c00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd26c15e030 0x7fd26c161d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:46.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:46.255+0000 7fd273c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd26c15e410 0x7fd26c0d2ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:46.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:46.255+0000 7fd273c00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd26c0cea50 con 0x7fd26c0cfd50 2024-03-04T09:26:46.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:46.255+0000 7fd273c00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd26c0cecf0 con 0x7fd26c15e410 2024-03-04T09:26:46.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:46.255+0000 7fd273c00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd26c0cef90 con 0x7fd26c15e030 2024-03-04T09:26:46.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:46.255+0000 7fd271c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd26c15e410 0x7fd26c0d2ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:46.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:46.255+0000 7fd271c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd26c15e410 0x7fd26c0d2ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:40634/0 (socket says 172.21.15.6:40634) 2024-03-04T09:26:46.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:46.255+0000 7fd271c00640 1 -- 172.21.15.6:0/121444980 learned_addr learned my addr 172.21.15.6:0/121444980 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:46.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:46.256+0000 7fd271200640 1 --2- 172.21.15.6:0/121444980 >> v2:172.21.15.146:3300/0 conn(0x7fd26c15e030 0x7fd26c161d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:46.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:46.256+0000 7fd271c00640 1 -- 172.21.15.6:0/121444980 >> v2:172.21.15.146:3300/0 conn(0x7fd26c15e030 msgr2=0x7fd26c161d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:46.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:46.256+0000 7fd271c00640 1 --2- 172.21.15.6:0/121444980 >> v2:172.21.15.146:3300/0 conn(0x7fd26c15e030 0x7fd26c161d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:46.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:46.256+0000 7fd271c00640 1 -- 172.21.15.6:0/121444980 >> v1:172.21.15.6:6789/0 conn(0x7fd26c0cfd50 legacy=0x7fd26c0d2490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:46.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:46.257+0000 7fd271c00640 1 -- 172.21.15.6:0/121444980 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd26c0cf270 con 0x7fd26c15e410 2024-03-04T09:26:46.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:46.257+0000 7fd271200640 1 --2- 172.21.15.6:0/121444980 >> v2:172.21.15.146:3300/0 conn(0x7fd26c15e030 0x7fd26c161d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:26:46.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:46.260+0000 7fd271c00640 1 --2- 172.21.15.6:0/121444980 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd26c15e410 0x7fd26c0d2ba0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fd26c08cde0 tx=0x7fd26402bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4a1df9d4c3ba6a0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:46.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:46.262+0000 7fd26b400640 1 -- 172.21.15.6:0/121444980 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd26402cb20 con 0x7fd26c15e410 2024-03-04T09:26:46.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:46.262+0000 7fd26b400640 1 -- 172.21.15.6:0/121444980 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd26402ccc0 con 0x7fd26c15e410 2024-03-04T09:26:46.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:46.262+0000 7fd26b400640 1 -- 172.21.15.6:0/121444980 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd264034610 con 0x7fd26c15e410 2024-03-04T09:26:46.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:46.263+0000 7fd273c00640 1 -- 172.21.15.6:0/121444980 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd26c15e410 msgr2=0x7fd26c0d2ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:46.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:46.263+0000 7fd273c00640 1 --2- 172.21.15.6:0/121444980 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd26c15e410 0x7fd26c0d2ba0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fd26c08cde0 tx=0x7fd26402bf80 comp rx=0 tx=0).stop 2024-03-04T09:26:46.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:46.266+0000 7fd273c00640 1 -- 172.21.15.6:0/121444980 shutdown_connections 2024-03-04T09:26:46.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:46.266+0000 7fd273c00640 1 --2- 172.21.15.6:0/121444980 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd26c15e410 0x7fd26c0d2ba0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:46.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:46.266+0000 7fd273c00640 1 --2- 172.21.15.6:0/121444980 >> v2:172.21.15.146:3300/0 conn(0x7fd26c15e030 0x7fd26c161d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:46.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:46.266+0000 7fd273c00640 1 -- 172.21.15.6:0/121444980 >> 172.21.15.6:0/121444980 conn(0x7fd26c0d3eb0 msgr2=0x7fd26c0d4290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:46.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:46.267+0000 7fd273c00640 1 -- 172.21.15.6:0/121444980 shutdown_connections 2024-03-04T09:26:46.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:46.267+0000 7fd273c00640 1 -- 172.21.15.6:0/121444980 wait complete. 2024-03-04T09:26:46.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:46.271+0000 7fd273c00640 1 Processor -- start 2024-03-04T09:26:46.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:46.271+0000 7fd273c00640 1 -- start start 2024-03-04T09:26:46.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:46.273+0000 7fd273c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd26c15e030 0x7fd26c0b8eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:46.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:46.273+0000 7fd273c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd26c15e410 0x7fd26c160250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:46.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:46.273+0000 7fd273c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd26c160790 0x7fd26c15cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:46.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:46.273+0000 7fd273c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd26c162260 con 0x7fd26c15e410 2024-03-04T09:26:46.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:46.273+0000 7fd273c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd26c0cf830 con 0x7fd26c160790 2024-03-04T09:26:46.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:46.273+0000 7fd273c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd26c1623e0 con 0x7fd26c15e030 2024-03-04T09:26:46.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:46.273+0000 7fd271200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd26c15e030 0x7fd26c0b8eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:46.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:46.273+0000 7fd26be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd26c15e410 0x7fd26c160250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:46.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:46.273+0000 7fd271200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd26c15e030 0x7fd26c0b8eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.6:60284/0 (socket says 172.21.15.6:60284) 2024-03-04T09:26:46.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:46.273+0000 7fd271200640 1 -- 172.21.15.6:0/3716561316 learned_addr learned my addr 172.21.15.6:0/3716561316 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:46.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:46.274+0000 7fd271c00640 1 --2- 172.21.15.6:0/3716561316 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd26c160790 0x7fd26c15cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:46.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:46.275+0000 7fd271c00640 1 -- 172.21.15.6:0/3716561316 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd26c15e030 msgr2=0x7fd26c0b8eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:46.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:46.275+0000 7fd271c00640 1 --2- 172.21.15.6:0/3716561316 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd26c15e030 0x7fd26c0b8eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:46.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:46.275+0000 7fd271c00640 1 -- 172.21.15.6:0/3716561316 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd26c15e410 msgr2=0x7fd26c160250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:46.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:46.275+0000 7fd271c00640 1 --2- 172.21.15.6:0/3716561316 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd26c15e410 0x7fd26c160250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:46.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:46.275+0000 7fd271c00640 1 -- 172.21.15.6:0/3716561316 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd26c160c20 con 0x7fd26c160790 2024-03-04T09:26:46.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:46.275+0000 7fd26be00640 1 --2- 172.21.15.6:0/3716561316 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd26c15e410 0x7fd26c160250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T09:26:46.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:46.277+0000 7fd271c00640 1 --2- 172.21.15.6:0/3716561316 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd26c160790 0x7fd26c15cfd0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fd264032000 tx=0x7fd264004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:46.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:46.278+0000 7fd268c00640 1 -- 172.21.15.6:0/3716561316 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd264043020 con 0x7fd26c160790 2024-03-04T09:26:46.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:46.278+0000 7fd268c00640 1 -- 172.21.15.6:0/3716561316 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd26403e450 con 0x7fd26c160790 2024-03-04T09:26:46.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:46.278+0000 7fd268c00640 1 -- 172.21.15.6:0/3716561316 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd26403e820 con 0x7fd26c160790 2024-03-04T09:26:46.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:46.278+0000 7fd273c00640 1 -- 172.21.15.6:0/3716561316 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd26c1f6570 con 0x7fd26c160790 2024-03-04T09:26:46.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:46.278+0000 7fd273c00640 1 -- 172.21.15.6:0/3716561316 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd26c0cc790 con 0x7fd26c160790 2024-03-04T09:26:46.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:46.280+0000 7fd273c00640 1 -- 172.21.15.6:0/3716561316 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd26c04f3a0 con 0x7fd26c160790 2024-03-04T09:26:46.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:46.283+0000 7fd268c00640 1 -- 172.21.15.6:0/3716561316 <== mon.1 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd264046020 con 0x7fd26c160790 2024-03-04T09:26:46.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:46.285+0000 7fd268c00640 1 --2- 172.21.15.6:0/3716561316 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd24c06b520 0x7fd24c06d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:46.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:46.285+0000 7fd268c00640 1 -- 172.21.15.6:0/3716561316 <== mon.1 v2:172.21.15.146:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd2640acf20 con 0x7fd26c160790 2024-03-04T09:26:46.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:46.286+0000 7fd271200640 1 --2- 172.21.15.6:0/3716561316 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd24c06b520 0x7fd24c06d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:46.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:46.287+0000 7fd271200640 1 --2- 172.21.15.6:0/3716561316 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd24c06b520 0x7fd24c06d9e0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fd26c0c74e0 tx=0x7fd260008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:46.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:46.293+0000 7fd268c00640 1 -- 172.21.15.6:0/3716561316 <== mon.1 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd26404d030 con 0x7fd26c160790 2024-03-04T09:26:46.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:46.570+0000 7fd273c00640 1 -- 172.21.15.6:0/3716561316 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd26c0cc160 con 0x7fd26c160790 2024-03-04T09:26:46.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:46.571+0000 7fd268c00640 1 -- 172.21.15.6:0/3716561316 <== mon.1 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+4977 (secure 0 0 0) 0x7fd26407c030 con 0x7fd26c160790 2024-03-04T09:26:46.568 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:26:46.568 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4685,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/1703428647","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":1703428647},{"type":"v1","addr":"172.21.15.146:6837","nonce":1703428647}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4837,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/2617429530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2617429530},{"type":"v1","addr":"172.21.15.6:6833","nonce":2617429530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":5043,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/2744913857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":2744913857},{"type":"v1","addr":"172.21.15.6:6835","nonce":2744913857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:26:30.445199+0000","modified":"2024-03-04T09:26:33.534893+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5046},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5046":{"gid":5046,"name":"d","rank":0,"incarnation":12,"state":"up:active","state_seq":4,"addr":"172.21.15.146:6836/2285803221","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":2285803221},{"type":"v1","addr":"172.21.15.146:6836","nonce":2285803221}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5046,"qdb_cluster":[5046]},"id":2}]} 2024-03-04T09:26:46.568 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 16 2024-03-04T09:26:46.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:46.573+0000 7fd273c00640 1 -- 172.21.15.6:0/3716561316 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd24c06b520 msgr2=0x7fd24c06d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:46.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:46.573+0000 7fd273c00640 1 --2- 172.21.15.6:0/3716561316 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd24c06b520 0x7fd24c06d9e0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fd26c0c74e0 tx=0x7fd260008040 comp rx=0 tx=0).stop 2024-03-04T09:26:46.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:46.573+0000 7fd273c00640 1 -- 172.21.15.6:0/3716561316 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd26c160790 msgr2=0x7fd26c15cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:46.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:46.573+0000 7fd273c00640 1 --2- 172.21.15.6:0/3716561316 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd26c160790 0x7fd26c15cfd0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fd264032000 tx=0x7fd264004a40 comp rx=0 tx=0).stop 2024-03-04T09:26:46.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:46.573+0000 7fd273c00640 1 -- 172.21.15.6:0/3716561316 shutdown_connections 2024-03-04T09:26:46.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:46.573+0000 7fd273c00640 1 --2- 172.21.15.6:0/3716561316 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd26c160790 0x7fd26c15cfd0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:46.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:46.573+0000 7fd273c00640 1 --2- 172.21.15.6:0/3716561316 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd26c15e410 0x7fd26c160250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:46.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:46.573+0000 7fd273c00640 1 --2- 172.21.15.6:0/3716561316 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd24c06b520 0x7fd24c06d9e0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:46.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:46.573+0000 7fd273c00640 1 --2- 172.21.15.6:0/3716561316 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd26c15e030 0x7fd26c0b8eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:46.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:46.573+0000 7fd273c00640 1 -- 172.21.15.6:0/3716561316 >> 172.21.15.6:0/3716561316 conn(0x7fd26c0d3eb0 msgr2=0x7fd26c06bb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:46.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:46.574+0000 7fd273c00640 1 -- 172.21.15.6:0/3716561316 shutdown_connections 2024-03-04T09:26:46.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:46.574+0000 7fd273c00640 1 -- 172.21.15.6:0/3716561316 wait complete. 2024-03-04T09:26:46.589 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json flush journal 2024-03-04T09:26:46.954 INFO:teuthology.orchestra.run.smithi146.stdout:{"message":"","return_code":0} 2024-03-04T09:26:46.955 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-04T09:26:46.955 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:26:47.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.176+0000 7f001fc00640 1 Processor -- start 2024-03-04T09:26:47.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.176+0000 7f001fc00640 1 -- start start 2024-03-04T09:26:47.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.177+0000 7f001fc00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f001815e030 0x7f0018161d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:47.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.177+0000 7f001fc00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f001815e410 0x7f00180d2ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:47.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.177+0000 7f001fc00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00180cea50 con 0x7f00180cfd50 2024-03-04T09:26:47.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.177+0000 7f001fc00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00180cecf0 con 0x7f001815e410 2024-03-04T09:26:47.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.177+0000 7f001fc00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00180cef90 con 0x7f001815e030 2024-03-04T09:26:47.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.178+0000 7f0017e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f00180cfd50 0x7f00180d2490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:57352/0 (socket says 172.21.15.6:57352) 2024-03-04T09:26:47.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.178+0000 7f0017e00640 1 -- 172.21.15.6:0/4054975953 learned_addr learned my addr 172.21.15.6:0/4054975953 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:26:47.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.178+0000 7f001dc00640 1 --2- 172.21.15.6:0/4054975953 >> v2:172.21.15.146:3300/0 conn(0x7f001815e410 0x7f00180d2ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:47.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.178+0000 7f001d200640 1 --2- 172.21.15.6:0/4054975953 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f001815e030 0x7f0018161d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:47.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.179+0000 7f0017400640 1 -- 172.21.15.6:0/4054975953 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1728463366 0 0) 0x7f00180cea50 con 0x7f00180cfd50 2024-03-04T09:26:47.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.179+0000 7f0017400640 1 -- 172.21.15.6:0/4054975953 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efffc003520 con 0x7f00180cfd50 2024-03-04T09:26:47.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.179+0000 7f001dc00640 1 -- 172.21.15.6:0/4054975953 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f001815e030 msgr2=0x7f0018161d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:47.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.179+0000 7f001dc00640 1 --2- 172.21.15.6:0/4054975953 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f001815e030 0x7f0018161d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:47.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.179+0000 7f001dc00640 1 -- 172.21.15.6:0/4054975953 >> v1:172.21.15.6:6789/0 conn(0x7f00180cfd50 legacy=0x7f00180d2490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:47.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.179+0000 7f001dc00640 1 -- 172.21.15.6:0/4054975953 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00180cf270 con 0x7f001815e410 2024-03-04T09:26:47.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.180+0000 7f001dc00640 1 --2- 172.21.15.6:0/4054975953 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f001815e410 0x7f00180d2ba0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f00100097a0 tx=0x7f001002b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=21a2845ae2b53775 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:47.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.181+0000 7f0017400640 1 -- 172.21.15.6:0/4054975953 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0010034070 con 0x7f001815e410 2024-03-04T09:26:47.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.181+0000 7f0017400640 1 -- 172.21.15.6:0/4054975953 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0010030610 con 0x7f001815e410 2024-03-04T09:26:47.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.181+0000 7f0017400640 1 -- 172.21.15.6:0/4054975953 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00100309c0 con 0x7f001815e410 2024-03-04T09:26:47.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.181+0000 7f001fc00640 1 -- 172.21.15.6:0/4054975953 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f001815e410 msgr2=0x7f00180d2ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:47.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.181+0000 7f001fc00640 1 --2- 172.21.15.6:0/4054975953 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f001815e410 0x7f00180d2ba0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f00100097a0 tx=0x7f001002b960 comp rx=0 tx=0).stop 2024-03-04T09:26:47.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.182+0000 7f001fc00640 1 -- 172.21.15.6:0/4054975953 shutdown_connections 2024-03-04T09:26:47.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.182+0000 7f001fc00640 1 --2- 172.21.15.6:0/4054975953 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f001815e410 0x7f00180d2ba0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:47.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.182+0000 7f001fc00640 1 --2- 172.21.15.6:0/4054975953 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f001815e030 0x7f0018161d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:47.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.182+0000 7f001fc00640 1 -- 172.21.15.6:0/4054975953 >> 172.21.15.6:0/4054975953 conn(0x7f00180d3eb0 msgr2=0x7f00180d4290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:47.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.182+0000 7f001fc00640 1 -- 172.21.15.6:0/4054975953 shutdown_connections 2024-03-04T09:26:47.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.182+0000 7f001fc00640 1 -- 172.21.15.6:0/4054975953 wait complete. 2024-03-04T09:26:47.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.184+0000 7f001fc00640 1 Processor -- start 2024-03-04T09:26:47.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.184+0000 7f001fc00640 1 -- start start 2024-03-04T09:26:47.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.185+0000 7f001fc00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f001815e030 0x7f00181f7dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:47.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.185+0000 7f001fc00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f001815e410 0x7f00180b92a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:47.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.186+0000 7f001fc00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f00180c9ef0 0x7f00181609d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:47.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.186+0000 7f001fc00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0018162260 con 0x7f001815e030 2024-03-04T09:26:47.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.186+0000 7f001fc00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00180cf830 con 0x7f00180c9ef0 2024-03-04T09:26:47.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.186+0000 7f001fc00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00181623e0 con 0x7f001815e410 2024-03-04T09:26:47.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.186+0000 7f001d200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f001815e030 0x7f00181f7dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:47.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.186+0000 7f001d200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f001815e030 0x7f00181f7dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:40658/0 (socket says 172.21.15.6:40658) 2024-03-04T09:26:47.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.186+0000 7f001d200640 1 -- 172.21.15.6:0/2942034902 learned_addr learned my addr 172.21.15.6:0/2942034902 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:47.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.186+0000 7f001dc00640 1 --2- 172.21.15.6:0/2942034902 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f00180c9ef0 0x7f00181609d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:47.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.186+0000 7f0017e00640 1 --2- 172.21.15.6:0/2942034902 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f001815e410 0x7f00180b92a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:47.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.187+0000 7f001d200640 1 -- 172.21.15.6:0/2942034902 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f001815e410 msgr2=0x7f00180b92a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:47.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.187+0000 7f001d200640 1 --2- 172.21.15.6:0/2942034902 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f001815e410 0x7f00180b92a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:47.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.187+0000 7f001d200640 1 -- 172.21.15.6:0/2942034902 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f00180c9ef0 msgr2=0x7f00181609d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:47.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.187+0000 7f001d200640 1 --2- 172.21.15.6:0/2942034902 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f00180c9ef0 0x7f00181609d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:47.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.187+0000 7f001d200640 1 -- 172.21.15.6:0/2942034902 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00180b97e0 con 0x7f001815e030 2024-03-04T09:26:47.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.187+0000 7f001d200640 1 --2- 172.21.15.6:0/2942034902 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f001815e030 0x7f00181f7dc0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f000000e9e0 tx=0x7f000000eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:47.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.187+0000 7f0014c00640 1 -- 172.21.15.6:0/2942034902 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0000017070 con 0x7f001815e030 2024-03-04T09:26:47.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.187+0000 7f0014c00640 1 -- 172.21.15.6:0/2942034902 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f000000c9d0 con 0x7f001815e030 2024-03-04T09:26:47.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.187+0000 7f0014c00640 1 -- 172.21.15.6:0/2942034902 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f000000cdc0 con 0x7f001815e030 2024-03-04T09:26:47.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.187+0000 7f001fc00640 1 -- 172.21.15.6:0/2942034902 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0018161320 con 0x7f001815e030 2024-03-04T09:26:47.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.188+0000 7f001fc00640 1 -- 172.21.15.6:0/2942034902 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00181f72e0 con 0x7f001815e030 2024-03-04T09:26:47.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.189+0000 7f001fc00640 1 -- 172.21.15.6:0/2942034902 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f001804f3a0 con 0x7f001815e030 2024-03-04T09:26:47.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.190+0000 7f0014c00640 1 -- 172.21.15.6:0/2942034902 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f000001a070 con 0x7f001815e030 2024-03-04T09:26:47.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.191+0000 7f0014c00640 1 --2- 172.21.15.6:0/2942034902 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7efff406b500 0x7efff406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:47.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.191+0000 7f0014c00640 1 -- 172.21.15.6:0/2942034902 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f000008a1e0 con 0x7f001815e030 2024-03-04T09:26:47.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.192+0000 7f0017e00640 1 --2- 172.21.15.6:0/2942034902 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7efff406b500 0x7efff406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:47.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.192+0000 7f0017e00640 1 --2- 172.21.15.6:0/2942034902 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7efff406b500 0x7efff406d9c0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f00180ce4f0 tx=0x7f0008009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:47.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.196+0000 7f0014c00640 1 -- 172.21.15.6:0/2942034902 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0000058970 con 0x7f001815e030 2024-03-04T09:26:47.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.434+0000 7f001fc00640 1 -- 172.21.15.6:0/2942034902 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f00181599a0 con 0x7f001815e030 2024-03-04T09:26:47.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.435+0000 7f0014c00640 1 -- 172.21.15.6:0/2942034902 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+4977 (secure 0 0 0) 0x7f000005ca10 con 0x7f001815e030 2024-03-04T09:26:47.432 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:26:47.432 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4685,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/1703428647","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":1703428647},{"type":"v1","addr":"172.21.15.146:6837","nonce":1703428647}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4837,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/2617429530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2617429530},{"type":"v1","addr":"172.21.15.6:6833","nonce":2617429530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":5043,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/2744913857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":2744913857},{"type":"v1","addr":"172.21.15.6:6835","nonce":2744913857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:26:30.445199+0000","modified":"2024-03-04T09:26:33.534893+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5046},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5046":{"gid":5046,"name":"d","rank":0,"incarnation":12,"state":"up:active","state_seq":4,"addr":"172.21.15.146:6836/2285803221","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":2285803221},{"type":"v1","addr":"172.21.15.146:6836","nonce":2285803221}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5046,"qdb_cluster":[5046]},"id":2}]} 2024-03-04T09:26:47.432 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 16 2024-03-04T09:26:47.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.437+0000 7f001fc00640 1 -- 172.21.15.6:0/2942034902 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7efff406b500 msgr2=0x7efff406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:47.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.437+0000 7f001fc00640 1 --2- 172.21.15.6:0/2942034902 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7efff406b500 0x7efff406d9c0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f00180ce4f0 tx=0x7f0008009040 comp rx=0 tx=0).stop 2024-03-04T09:26:47.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.437+0000 7f001fc00640 1 -- 172.21.15.6:0/2942034902 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f001815e030 msgr2=0x7f00181f7dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:47.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.437+0000 7f001fc00640 1 --2- 172.21.15.6:0/2942034902 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f001815e030 0x7f00181f7dc0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f000000e9e0 tx=0x7f000000eea0 comp rx=0 tx=0).stop 2024-03-04T09:26:47.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.437+0000 7f001fc00640 1 -- 172.21.15.6:0/2942034902 shutdown_connections 2024-03-04T09:26:47.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.437+0000 7f001fc00640 1 --2- 172.21.15.6:0/2942034902 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f00180c9ef0 0x7f00181609d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:47.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.437+0000 7f001fc00640 1 --2- 172.21.15.6:0/2942034902 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7efff406b500 0x7efff406d9c0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:47.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.437+0000 7f001fc00640 1 --2- 172.21.15.6:0/2942034902 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f001815e410 0x7f00180b92a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:47.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.437+0000 7f001fc00640 1 --2- 172.21.15.6:0/2942034902 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f001815e030 0x7f00181f7dc0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:47.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.437+0000 7f001fc00640 1 -- 172.21.15.6:0/2942034902 >> 172.21.15.6:0/2942034902 conn(0x7f00180d3eb0 msgr2=0x7f00180b0ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:47.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.437+0000 7f001fc00640 1 -- 172.21.15.6:0/2942034902 shutdown_connections 2024-03-04T09:26:47.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.437+0000 7f001fc00640 1 -- 172.21.15.6:0/2942034902 wait complete. 2024-03-04T09:26:47.455 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json perf dump mds_cache strays_created 2024-03-04T09:26:47.677 INFO:teuthology.orchestra.run.smithi146.stdout:{"mds_cache":{"strays_created":1001}} 2024-03-04T09:26:47.678 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_created": 1001 } } 2024-03-04T09:26:47.678 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:26:47.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.910+0000 7f9152c00640 1 Processor -- start 2024-03-04T09:26:47.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.911+0000 7f9152c00640 1 -- start start 2024-03-04T09:26:47.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.912+0000 7f9152c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f914c15e2a0 0x7f914c0cef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:47.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.912+0000 7f9152c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f914c0cf460 0x7f914c0b1330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:47.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.912+0000 7f9152c00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f914c0d1120 con 0x7f914c15dec0 2024-03-04T09:26:47.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.912+0000 7f9152c00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f914c0b0ea0 con 0x7f914c15e2a0 2024-03-04T09:26:47.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.912+0000 7f9152c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f914c0d1400 con 0x7f914c0cf460 2024-03-04T09:26:47.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.913+0000 7f9150e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f914c15e2a0 0x7f914c0cef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:47.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.913+0000 7f9150e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f914c15e2a0 0x7f914c0cef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:40670/0 (socket says 172.21.15.6:40670) 2024-03-04T09:26:47.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.913+0000 7f9150e00640 1 -- 172.21.15.6:0/2776444942 learned_addr learned my addr 172.21.15.6:0/2776444942 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:47.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.913+0000 7f9152200640 1 --2- 172.21.15.6:0/2776444942 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f914c0cf460 0x7f914c0b1330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:47.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.913+0000 7f9150e00640 1 -- 172.21.15.6:0/2776444942 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f914c0cf460 msgr2=0x7f914c0b1330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:47.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.913+0000 7f9150e00640 1 --2- 172.21.15.6:0/2776444942 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f914c0cf460 0x7f914c0b1330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:47.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.913+0000 7f9150e00640 1 -- 172.21.15.6:0/2776444942 >> v1:172.21.15.146:6789/0 conn(0x7f914c15dec0 legacy=0x7f914c0ce810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:47.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.913+0000 7f9150e00640 1 -- 172.21.15.6:0/2776444942 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f914c0b0b30 con 0x7f914c15e2a0 2024-03-04T09:26:47.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.914+0000 7f9150e00640 1 --2- 172.21.15.6:0/2776444942 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f914c15e2a0 0x7f914c0cef20 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f9138009870 tx=0x7f913802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6df03092b04abadc server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:47.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.915+0000 7f914be00640 1 -- 172.21.15.6:0/2776444942 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9138035020 con 0x7f914c15e2a0 2024-03-04T09:26:47.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.915+0000 7f914be00640 1 -- 172.21.15.6:0/2776444942 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9138031640 con 0x7f914c15e2a0 2024-03-04T09:26:47.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.915+0000 7f914be00640 1 -- 172.21.15.6:0/2776444942 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f913802d040 con 0x7f914c15e2a0 2024-03-04T09:26:47.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.915+0000 7f9152c00640 1 -- 172.21.15.6:0/2776444942 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f914c15e2a0 msgr2=0x7f914c0cef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:47.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.915+0000 7f9152c00640 1 --2- 172.21.15.6:0/2776444942 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f914c15e2a0 0x7f914c0cef20 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f9138009870 tx=0x7f913802b8f0 comp rx=0 tx=0).stop 2024-03-04T09:26:47.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.915+0000 7f9152c00640 1 -- 172.21.15.6:0/2776444942 shutdown_connections 2024-03-04T09:26:47.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.915+0000 7f9152c00640 1 --2- 172.21.15.6:0/2776444942 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f914c0cf460 0x7f914c0b1330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:47.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.915+0000 7f9152c00640 1 --2- 172.21.15.6:0/2776444942 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f914c15e2a0 0x7f914c0cef20 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:47.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.915+0000 7f9152c00640 1 -- 172.21.15.6:0/2776444942 >> 172.21.15.6:0/2776444942 conn(0x7f914c0d3d40 msgr2=0x7f914c0d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:47.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.915+0000 7f9152c00640 1 -- 172.21.15.6:0/2776444942 shutdown_connections 2024-03-04T09:26:47.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.915+0000 7f9152c00640 1 -- 172.21.15.6:0/2776444942 wait complete. 2024-03-04T09:26:47.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.917+0000 7f9152c00640 1 Processor -- start 2024-03-04T09:26:47.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.917+0000 7f9152c00640 1 -- start start 2024-03-04T09:26:47.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.918+0000 7f9152c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f914c0cf460 0x7f914c0b5990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:47.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.918+0000 7f9152c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f914c15dec0 0x7f914c15c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:47.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.918+0000 7f9152c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f914c15e2a0 0x7f914c15ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:47.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.918+0000 7f9152c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f914c0d0ec0 con 0x7f914c15dec0 2024-03-04T09:26:47.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.918+0000 7f9152c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f914c0d0d40 con 0x7f914c15e2a0 2024-03-04T09:26:47.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.918+0000 7f9152c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f914c0b1870 con 0x7f914c0cf460 2024-03-04T09:26:47.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.918+0000 7f9150e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f914c15dec0 0x7f914c15c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:47.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.919+0000 7f9150e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f914c15dec0 0x7f914c15c780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:40686/0 (socket says 172.21.15.6:40686) 2024-03-04T09:26:47.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.919+0000 7f9150e00640 1 -- 172.21.15.6:0/956638891 learned_addr learned my addr 172.21.15.6:0/956638891 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:47.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.919+0000 7f9151800640 1 --2- 172.21.15.6:0/956638891 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f914c0cf460 0x7f914c0b5990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:47.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.919+0000 7f9152200640 1 --2- 172.21.15.6:0/956638891 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f914c15e2a0 0x7f914c15ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:47.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.919+0000 7f9150e00640 1 -- 172.21.15.6:0/956638891 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f914c0cf460 msgr2=0x7f914c0b5990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:47.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.919+0000 7f9150e00640 1 --2- 172.21.15.6:0/956638891 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f914c0cf460 0x7f914c0b5990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:47.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.919+0000 7f9150e00640 1 -- 172.21.15.6:0/956638891 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f914c15e2a0 msgr2=0x7f914c15ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:47.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.919+0000 7f9150e00640 1 --2- 172.21.15.6:0/956638891 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f914c15e2a0 0x7f914c15ccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:47.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.919+0000 7f9150e00640 1 -- 172.21.15.6:0/956638891 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f914c0b4bd0 con 0x7f914c15dec0 2024-03-04T09:26:47.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.919+0000 7f9150e00640 1 --2- 172.21.15.6:0/956638891 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f914c15dec0 0x7f914c15c780 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f913802b930 tx=0x7f91380046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:47.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.920+0000 7f9149600640 1 -- 172.21.15.6:0/956638891 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9138040070 con 0x7f914c15dec0 2024-03-04T09:26:47.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.920+0000 7f9149600640 1 -- 172.21.15.6:0/956638891 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91380399f0 con 0x7f914c15dec0 2024-03-04T09:26:47.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.920+0000 7f9152c00640 1 -- 172.21.15.6:0/956638891 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f914c0b4eb0 con 0x7f914c15dec0 2024-03-04T09:26:47.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.920+0000 7f9149600640 1 -- 172.21.15.6:0/956638891 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9138035040 con 0x7f914c15dec0 2024-03-04T09:26:47.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.920+0000 7f9152c00640 1 -- 172.21.15.6:0/956638891 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f914c0b5470 con 0x7f914c15dec0 2024-03-04T09:26:47.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.921+0000 7f9152c00640 1 -- 172.21.15.6:0/956638891 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f914c04f330 con 0x7f914c15dec0 2024-03-04T09:26:47.919 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.922+0000 7f9149600640 1 -- 172.21.15.6:0/956638891 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9138038020 con 0x7f914c15dec0 2024-03-04T09:26:47.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.923+0000 7f9149600640 1 --2- 172.21.15.6:0/956638891 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f912c06b4d0 0x7f912c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:47.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.924+0000 7f9149600640 1 -- 172.21.15.6:0/956638891 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f91380abae0 con 0x7f914c15dec0 2024-03-04T09:26:47.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.924+0000 7f9151800640 1 --2- 172.21.15.6:0/956638891 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f912c06b4d0 0x7f912c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:47.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.925+0000 7f9151800640 1 --2- 172.21.15.6:0/956638891 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f912c06b4d0 0x7f912c06d990 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f914c0ca870 tx=0x7f9140009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:47.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:47.928+0000 7f9149600640 1 -- 172.21.15.6:0/956638891 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f913807a9f0 con 0x7f914c15dec0 2024-03-04T09:26:48.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:48.179+0000 7f9152c00640 1 -- 172.21.15.6:0/956638891 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f914c1614e0 con 0x7f914c15dec0 2024-03-04T09:26:48.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:48.181+0000 7f9149600640 1 -- 172.21.15.6:0/956638891 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+4977 (secure 0 0 0) 0x7f913807a2f0 con 0x7f914c15dec0 2024-03-04T09:26:48.178 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:26:48.178 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4685,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/1703428647","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":1703428647},{"type":"v1","addr":"172.21.15.146:6837","nonce":1703428647}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4837,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/2617429530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2617429530},{"type":"v1","addr":"172.21.15.6:6833","nonce":2617429530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":5043,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/2744913857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":2744913857},{"type":"v1","addr":"172.21.15.6:6835","nonce":2744913857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:26:30.445199+0000","modified":"2024-03-04T09:26:33.534893+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5046},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5046":{"gid":5046,"name":"d","rank":0,"incarnation":12,"state":"up:active","state_seq":4,"addr":"172.21.15.146:6836/2285803221","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":2285803221},{"type":"v1","addr":"172.21.15.146:6836","nonce":2285803221}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5046,"qdb_cluster":[5046]},"id":2}]} 2024-03-04T09:26:48.178 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 16 2024-03-04T09:26:48.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:48.183+0000 7f9152c00640 1 -- 172.21.15.6:0/956638891 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f912c06b4d0 msgr2=0x7f912c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:48.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:48.183+0000 7f9152c00640 1 --2- 172.21.15.6:0/956638891 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f912c06b4d0 0x7f912c06d990 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f914c0ca870 tx=0x7f9140009040 comp rx=0 tx=0).stop 2024-03-04T09:26:48.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:48.183+0000 7f9152c00640 1 -- 172.21.15.6:0/956638891 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f914c15dec0 msgr2=0x7f914c15c780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:48.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:48.183+0000 7f9152c00640 1 --2- 172.21.15.6:0/956638891 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f914c15dec0 0x7f914c15c780 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f913802b930 tx=0x7f91380046c0 comp rx=0 tx=0).stop 2024-03-04T09:26:48.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:48.183+0000 7f9152c00640 1 -- 172.21.15.6:0/956638891 shutdown_connections 2024-03-04T09:26:48.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:48.183+0000 7f9152c00640 1 --2- 172.21.15.6:0/956638891 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f914c15e2a0 0x7f914c15ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:48.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:48.183+0000 7f9152c00640 1 --2- 172.21.15.6:0/956638891 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f914c15dec0 0x7f914c15c780 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:48.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:48.183+0000 7f9152c00640 1 --2- 172.21.15.6:0/956638891 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f912c06b4d0 0x7f912c06d990 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:48.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:48.183+0000 7f9152c00640 1 --2- 172.21.15.6:0/956638891 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f914c0cf460 0x7f914c0b5990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:48.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:48.184+0000 7f9152c00640 1 -- 172.21.15.6:0/956638891 >> 172.21.15.6:0/956638891 conn(0x7f914c0d3d40 msgr2=0x7f914c06b960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:48.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:48.184+0000 7f9152c00640 1 -- 172.21.15.6:0/956638891 shutdown_connections 2024-03-04T09:26:48.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:48.184+0000 7f9152c00640 1 -- 172.21.15.6:0/956638891 wait complete. 2024-03-04T09:26:48.201 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json perf dump mds_cache strays_enqueued 2024-03-04T09:26:48.433 INFO:teuthology.orchestra.run.smithi146.stdout:{"mds_cache":{"strays_enqueued":970}} 2024-03-04T09:26:48.433 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 970 } } 2024-03-04T09:26:48.433 DEBUG:tasks.ceph_test_case:wait_until_equal: 970 != 1001, waiting (timeout=600)... 2024-03-04T09:26:53.434 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:26:53.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:53.670+0000 7f22f3e00640 1 Processor -- start 2024-03-04T09:26:53.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:53.670+0000 7f22f3e00640 1 -- start start 2024-03-04T09:26:53.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:53.671+0000 7f22f3e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f22f40cf080 0x7f22f40d00a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:53.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:53.671+0000 7f22f3e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f22f40d05e0 0x7f22f415d750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:53.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:53.671+0000 7f22f3e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22f40b1370 con 0x7f22f40b0f90 2024-03-04T09:26:53.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:53.671+0000 7f22f3e00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22f40d0a50 con 0x7f22f40d05e0 2024-03-04T09:26:53.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:53.671+0000 7f22f3e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22f40ccf80 con 0x7f22f40cf080 2024-03-04T09:26:53.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:53.671+0000 7f22f2a00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f22f40b0f90 0x7f22f40cc730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:33998/0 (socket says 172.21.15.6:33998) 2024-03-04T09:26:53.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:53.671+0000 7f22f2a00640 1 -- 172.21.15.6:0/1013806413 learned_addr learned my addr 172.21.15.6:0/1013806413 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:26:53.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:53.672+0000 7f22ea000640 1 --2- 172.21.15.6:0/1013806413 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f22f40cf080 0x7f22f40d00a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:53.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:53.672+0000 7f22f3400640 1 --2- 172.21.15.6:0/1013806413 >> v2:172.21.15.146:3300/0 conn(0x7f22f40d05e0 0x7f22f415d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:53.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:53.672+0000 7f22f2000640 1 -- 172.21.15.6:0/1013806413 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4068522082 0 0) 0x7f22f40b1370 con 0x7f22f40b0f90 2024-03-04T09:26:53.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:53.672+0000 7f22f2000640 1 -- 172.21.15.6:0/1013806413 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22d8003520 con 0x7f22f40b0f90 2024-03-04T09:26:53.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:53.672+0000 7f22f2000640 1 -- 172.21.15.6:0/1013806413 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2289272419 0 0) 0x7f22d8003520 con 0x7f22f40b0f90 2024-03-04T09:26:53.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:53.672+0000 7f22f2000640 1 -- 172.21.15.6:0/1013806413 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f22f40cf080 msgr2=0x7f22f40d00a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:53.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:53.672+0000 7f22f2000640 1 --2- 172.21.15.6:0/1013806413 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f22f40cf080 0x7f22f40d00a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:53.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:53.672+0000 7f22f2000640 1 -- 172.21.15.6:0/1013806413 >> v2:172.21.15.146:3300/0 conn(0x7f22f40d05e0 msgr2=0x7f22f415d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:53.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:53.672+0000 7f22f2000640 1 --2- 172.21.15.6:0/1013806413 >> v2:172.21.15.146:3300/0 conn(0x7f22f40d05e0 0x7f22f415d750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:53.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:53.672+0000 7f22f2000640 1 -- 172.21.15.6:0/1013806413 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22f40cd260 con 0x7f22f40b0f90 2024-03-04T09:26:53.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:53.672+0000 7f22f2000640 1 -- 172.21.15.6:0/1013806413 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7f22e4001670 con 0x7f22f40b0f90 2024-03-04T09:26:53.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:53.672+0000 7f22f2000640 1 -- 172.21.15.6:0/1013806413 >> v1:172.21.15.6:6789/0 conn(0x7f22f40b0f90 legacy=0x7f22f40cc730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:53.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:53.673+0000 7f22f2000640 1 --2- 172.21.15.6:0/1013806413 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f22d80039b0 0x7f22d8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:53.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:53.673+0000 7f22f2000640 1 --2- 172.21.15.6:0/1013806413 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f22d8004800 0x7f22d8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:53.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:53.673+0000 7f22f2000640 1 --2- 172.21.15.6:0/1013806413 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f22f40cf080 0x7f22d80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:53.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:53.673+0000 7f22f2000640 1 -- 172.21.15.6:0/1013806413 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22f40d0a50 con 0x7f22d80039b0 2024-03-04T09:26:53.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:53.673+0000 7f22f2000640 1 -- 172.21.15.6:0/1013806413 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22f40ccf80 con 0x7f22d8004800 2024-03-04T09:26:53.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:53.673+0000 7f22f2000640 1 -- 172.21.15.6:0/1013806413 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22d800b9c0 con 0x7f22f40cf080 2024-03-04T09:26:53.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:53.673+0000 7f22f2a00640 1 --2- 172.21.15.6:0/1013806413 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f22d80039b0 0x7f22d8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:53.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:53.673+0000 7f22f3400640 1 --2- 172.21.15.6:0/1013806413 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f22f40cf080 0x7f22d80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:53.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:53.673+0000 7f22ea000640 1 --2- 172.21.15.6:0/1013806413 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f22d8004800 0x7f22d8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:53.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:53.673+0000 7f22f2a00640 1 -- 172.21.15.6:0/1013806413 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f22f40cf080 msgr2=0x7f22d80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:53.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:53.673+0000 7f22f2a00640 1 --2- 172.21.15.6:0/1013806413 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f22f40cf080 0x7f22d80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:53.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:53.673+0000 7f22f2a00640 1 -- 172.21.15.6:0/1013806413 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f22d8004800 msgr2=0x7f22d8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:53.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:53.673+0000 7f22f2a00640 1 --2- 172.21.15.6:0/1013806413 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f22d8004800 0x7f22d8004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:53.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:53.673+0000 7f22f2a00640 1 -- 172.21.15.6:0/1013806413 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22d8003520 con 0x7f22d80039b0 2024-03-04T09:26:53.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:53.673+0000 7f22f3400640 1 --2- 172.21.15.6:0/1013806413 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f22f40cf080 0x7f22d80051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:26:53.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:53.675+0000 7f22f2a00640 1 --2- 172.21.15.6:0/1013806413 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f22d80039b0 0x7f22d8003dd0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f22e400c240 tx=0x7f22d8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f17eff614b688089 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:53.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:53.675+0000 7f22f2000640 1 -- 172.21.15.6:0/1013806413 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22e4007d50 con 0x7f22d80039b0 2024-03-04T09:26:53.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:53.675+0000 7f22f2000640 1 -- 172.21.15.6:0/1013806413 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22e4030070 con 0x7f22d80039b0 2024-03-04T09:26:53.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:53.675+0000 7f22f2000640 1 -- 172.21.15.6:0/1013806413 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22e402cd70 con 0x7f22d80039b0 2024-03-04T09:26:53.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:53.675+0000 7f22f3e00640 1 -- 172.21.15.6:0/1013806413 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f22d80039b0 msgr2=0x7f22d8003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:53.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:53.675+0000 7f22f3e00640 1 --2- 172.21.15.6:0/1013806413 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f22d80039b0 0x7f22d8003dd0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f22e400c240 tx=0x7f22d8005830 comp rx=0 tx=0).stop 2024-03-04T09:26:53.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:53.675+0000 7f22f3400640 1 -- 172.21.15.6:0/1013806413 reap_dead start 2024-03-04T09:26:53.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:53.675+0000 7f22f3e00640 1 -- 172.21.15.6:0/1013806413 shutdown_connections 2024-03-04T09:26:53.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:53.676+0000 7f22f3e00640 1 -- 172.21.15.6:0/1013806413 >> 172.21.15.6:0/1013806413 conn(0x7f22f40d1ae0 msgr2=0x7f22f40d1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:53.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:53.676+0000 7f22f3e00640 1 -- 172.21.15.6:0/1013806413 shutdown_connections 2024-03-04T09:26:53.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:53.676+0000 7f22f3e00640 1 -- 172.21.15.6:0/1013806413 wait complete. 2024-03-04T09:26:53.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:53.677+0000 7f22f3e00640 1 Processor -- start 2024-03-04T09:26:53.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:53.677+0000 7f22f3e00640 1 -- start start 2024-03-04T09:26:53.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:53.678+0000 7f22f3e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f22d80039b0 0x7f22f40c95c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:53.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:53.678+0000 7f22f3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f22f4155260 0x7f22f40ca2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:53.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:53.678+0000 7f22f3e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f22f40ca820 0x7f22f4159760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:53.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:53.678+0000 7f22f3e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22f40d00a0 con 0x7f22f4155260 2024-03-04T09:26:53.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:53.678+0000 7f22f3e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22f40cd820 con 0x7f22f40ca820 2024-03-04T09:26:53.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:53.678+0000 7f22f3e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22f40d0220 con 0x7f22d80039b0 2024-03-04T09:26:53.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:53.679+0000 7f22ea000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f22f4155260 0x7f22f40ca2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:53.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:53.679+0000 7f22ea000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f22f4155260 0x7f22f40ca2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:40490/0 (socket says 172.21.15.6:40490) 2024-03-04T09:26:53.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:53.679+0000 7f22ea000640 1 -- 172.21.15.6:0/3923549872 learned_addr learned my addr 172.21.15.6:0/3923549872 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:53.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:53.679+0000 7f22f2a00640 1 --2- 172.21.15.6:0/3923549872 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f22d80039b0 0x7f22f40c95c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:53.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:53.679+0000 7f22f3400640 1 --2- 172.21.15.6:0/3923549872 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f22f40ca820 0x7f22f4159760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:53.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:53.679+0000 7f22ea000640 1 -- 172.21.15.6:0/3923549872 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f22d80039b0 msgr2=0x7f22f40c95c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:53.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:53.679+0000 7f22ea000640 1 --2- 172.21.15.6:0/3923549872 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f22d80039b0 0x7f22f40c95c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:53.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:53.679+0000 7f22ea000640 1 -- 172.21.15.6:0/3923549872 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f22f40ca820 msgr2=0x7f22f4159760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:53.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:53.679+0000 7f22ea000640 1 --2- 172.21.15.6:0/3923549872 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f22f40ca820 0x7f22f4159760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:53.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:53.679+0000 7f22ea000640 1 -- 172.21.15.6:0/3923549872 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22f40c9b00 con 0x7f22f4155260 2024-03-04T09:26:53.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:53.679+0000 7f22f3400640 1 --2- 172.21.15.6:0/3923549872 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f22f40ca820 0x7f22f4159760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:26:53.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:53.680+0000 7f22ea000640 1 --2- 172.21.15.6:0/3923549872 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f22f4155260 0x7f22f40ca2e0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f22dc010860 tx=0x7f22dc010d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:53.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:53.680+0000 7f22eb400640 1 -- 172.21.15.6:0/3923549872 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22dc00f040 con 0x7f22f4155260 2024-03-04T09:26:53.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:53.680+0000 7f22eb400640 1 -- 172.21.15.6:0/3923549872 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22dc00eb60 con 0x7f22f4155260 2024-03-04T09:26:53.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:53.680+0000 7f22eb400640 1 -- 172.21.15.6:0/3923549872 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22dc01c400 con 0x7f22f4155260 2024-03-04T09:26:53.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:53.680+0000 7f22f3e00640 1 -- 172.21.15.6:0/3923549872 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22f40c9d80 con 0x7f22f4155260 2024-03-04T09:26:53.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:53.680+0000 7f22f3e00640 1 -- 172.21.15.6:0/3923549872 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22f4158f60 con 0x7f22f4155260 2024-03-04T09:26:53.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:53.681+0000 7f22f3e00640 1 -- 172.21.15.6:0/3923549872 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22f4068930 con 0x7f22f4155260 2024-03-04T09:26:53.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:53.682+0000 7f22eb400640 1 -- 172.21.15.6:0/3923549872 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f22dc01c5a0 con 0x7f22f4155260 2024-03-04T09:26:53.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:53.683+0000 7f22eb400640 1 --2- 172.21.15.6:0/3923549872 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f22d006b500 0x7f22d006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:53.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:53.683+0000 7f22eb400640 1 -- 172.21.15.6:0/3923549872 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f22dc08a860 con 0x7f22f4155260 2024-03-04T09:26:53.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:53.684+0000 7f22f2a00640 1 --2- 172.21.15.6:0/3923549872 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f22d006b500 0x7f22d006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:53.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:53.685+0000 7f22f2a00640 1 --2- 172.21.15.6:0/3923549872 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f22d006b500 0x7f22d006d9c0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f22e4034000 tx=0x7f22e4007000 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:53.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:53.688+0000 7f22eb400640 1 -- 172.21.15.6:0/3923549872 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f22f4068930 con 0x7f22f4155260 2024-03-04T09:26:53.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:53.927+0000 7f22f3e00640 1 -- 172.21.15.6:0/3923549872 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f22f40aec80 con 0x7f22f4155260 2024-03-04T09:26:53.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:53.928+0000 7f22eb400640 1 -- 172.21.15.6:0/3923549872 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+4977 (secure 0 0 0) 0x7f22f40c9b00 con 0x7f22f4155260 2024-03-04T09:26:53.926 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:26:53.926 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4685,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/1703428647","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":1703428647},{"type":"v1","addr":"172.21.15.146:6837","nonce":1703428647}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4837,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/2617429530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2617429530},{"type":"v1","addr":"172.21.15.6:6833","nonce":2617429530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":5043,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/2744913857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":2744913857},{"type":"v1","addr":"172.21.15.6:6835","nonce":2744913857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:26:30.445199+0000","modified":"2024-03-04T09:26:33.534893+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5046},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5046":{"gid":5046,"name":"d","rank":0,"incarnation":12,"state":"up:active","state_seq":4,"addr":"172.21.15.146:6836/2285803221","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":2285803221},{"type":"v1","addr":"172.21.15.146:6836","nonce":2285803221}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5046,"qdb_cluster":[5046]},"id":2}]} 2024-03-04T09:26:53.926 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 16 2024-03-04T09:26:53.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:53.931+0000 7f22f3e00640 1 -- 172.21.15.6:0/3923549872 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f22d006b500 msgr2=0x7f22d006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:53.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:53.931+0000 7f22f3e00640 1 --2- 172.21.15.6:0/3923549872 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f22d006b500 0x7f22d006d9c0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f22e4034000 tx=0x7f22e4007000 comp rx=0 tx=0).stop 2024-03-04T09:26:53.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:53.931+0000 7f22f3e00640 1 -- 172.21.15.6:0/3923549872 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f22f4155260 msgr2=0x7f22f40ca2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:53.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:53.931+0000 7f22f3e00640 1 --2- 172.21.15.6:0/3923549872 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f22f4155260 0x7f22f40ca2e0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f22dc010860 tx=0x7f22dc010d20 comp rx=0 tx=0).stop 2024-03-04T09:26:53.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:53.931+0000 7f22f3e00640 1 -- 172.21.15.6:0/3923549872 shutdown_connections 2024-03-04T09:26:53.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:53.931+0000 7f22f3e00640 1 --2- 172.21.15.6:0/3923549872 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f22f40ca820 0x7f22f4159760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:53.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:53.931+0000 7f22f3e00640 1 --2- 172.21.15.6:0/3923549872 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f22f4155260 0x7f22f40ca2e0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:53.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:53.931+0000 7f22f3e00640 1 --2- 172.21.15.6:0/3923549872 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f22d006b500 0x7f22d006d9c0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:53.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:53.931+0000 7f22f3e00640 1 --2- 172.21.15.6:0/3923549872 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f22d80039b0 0x7f22f40c95c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:53.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:53.931+0000 7f22f3e00640 1 -- 172.21.15.6:0/3923549872 >> 172.21.15.6:0/3923549872 conn(0x7f22f40d1ae0 msgr2=0x7f22f40d0bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:53.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:53.932+0000 7f22f3e00640 1 -- 172.21.15.6:0/3923549872 shutdown_connections 2024-03-04T09:26:53.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:53.932+0000 7f22f3e00640 1 -- 172.21.15.6:0/3923549872 wait complete. 2024-03-04T09:26:53.948 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json perf dump mds_cache strays_enqueued 2024-03-04T09:26:54.181 INFO:teuthology.orchestra.run.smithi146.stdout:{"mds_cache":{"strays_enqueued":1001}} 2024-03-04T09:26:54.181 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 1001 } } 2024-03-04T09:26:54.181 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:26:54.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:54.412+0000 7ff723000640 1 Processor -- start 2024-03-04T09:26:54.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:54.413+0000 7ff723000640 1 -- start start 2024-03-04T09:26:54.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:54.414+0000 7ff723000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ff71c0b3c90 0x7ff71c0cdc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:54.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:54.414+0000 7ff723000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff71c071050 0x7ff71c071490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:54.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:54.414+0000 7ff723000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff71c0b4860 con 0x7ff71c0cf5c0 2024-03-04T09:26:54.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:54.414+0000 7ff723000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff71c0ce160 con 0x7ff71c0b3c90 2024-03-04T09:26:54.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:54.414+0000 7ff723000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff71c0b4b40 con 0x7ff71c071050 2024-03-04T09:26:54.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:54.415+0000 7ff71be00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ff71c0b3c90 0x7ff71c0cdc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:54.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:54.415+0000 7ff721000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff71c071050 0x7ff71c071490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:54.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:54.415+0000 7ff71be00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ff71c0b3c90 0x7ff71c0cdc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.6:44580/0 (socket says 172.21.15.6:44580) 2024-03-04T09:26:54.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:54.415+0000 7ff71be00640 1 -- 172.21.15.6:0/2490523832 learned_addr learned my addr 172.21.15.6:0/2490523832 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:54.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:54.416+0000 7ff71be00640 1 -- 172.21.15.6:0/2490523832 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff71c071050 msgr2=0x7ff71c071490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:54.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:54.416+0000 7ff71be00640 1 --2- 172.21.15.6:0/2490523832 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff71c071050 0x7ff71c071490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:54.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:54.416+0000 7ff71be00640 1 -- 172.21.15.6:0/2490523832 >> v1:172.21.15.146:6789/0 conn(0x7ff71c0cf5c0 legacy=0x7ff71c0b4070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:54.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:54.416+0000 7ff71be00640 1 -- 172.21.15.6:0/2490523832 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff71c0b4e20 con 0x7ff71c0b3c90 2024-03-04T09:26:54.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:54.418+0000 7ff71be00640 1 --2- 172.21.15.6:0/2490523832 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff71c0b3c90 0x7ff71c0cdc20 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7ff70c009870 tx=0x7ff70c02dc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fcb707d1b5ec8e10 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:54.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:54.418+0000 7ff71aa00640 1 -- 172.21.15.6:0/2490523832 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff70c03b070 con 0x7ff71c0b3c90 2024-03-04T09:26:54.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:54.418+0000 7ff71aa00640 1 -- 172.21.15.6:0/2490523832 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff70c02e9e0 con 0x7ff71c0b3c90 2024-03-04T09:26:54.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:54.418+0000 7ff71aa00640 1 -- 172.21.15.6:0/2490523832 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff70c02edb0 con 0x7ff71c0b3c90 2024-03-04T09:26:54.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:54.419+0000 7ff723000640 1 -- 172.21.15.6:0/2490523832 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff71c0b3c90 msgr2=0x7ff71c0cdc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:54.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:54.419+0000 7ff723000640 1 --2- 172.21.15.6:0/2490523832 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff71c0b3c90 0x7ff71c0cdc20 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7ff70c009870 tx=0x7ff70c02dc30 comp rx=0 tx=0).stop 2024-03-04T09:26:54.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:54.419+0000 7ff723000640 1 -- 172.21.15.6:0/2490523832 shutdown_connections 2024-03-04T09:26:54.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:54.419+0000 7ff723000640 1 --2- 172.21.15.6:0/2490523832 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff71c071050 0x7ff71c071490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:54.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:54.419+0000 7ff723000640 1 --2- 172.21.15.6:0/2490523832 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff71c0b3c90 0x7ff71c0cdc20 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:54.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:54.419+0000 7ff723000640 1 -- 172.21.15.6:0/2490523832 >> 172.21.15.6:0/2490523832 conn(0x7ff71c06afb0 msgr2=0x7ff71c06b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:54.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:54.419+0000 7ff723000640 1 -- 172.21.15.6:0/2490523832 shutdown_connections 2024-03-04T09:26:54.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:54.419+0000 7ff723000640 1 -- 172.21.15.6:0/2490523832 wait complete. 2024-03-04T09:26:54.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:54.421+0000 7ff723000640 1 Processor -- start 2024-03-04T09:26:54.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:54.421+0000 7ff723000640 1 -- start start 2024-03-04T09:26:54.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:54.422+0000 7ff723000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff71c071050 0x7ff71c154a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:54.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:54.422+0000 7ff723000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff71c0b3c90 0x7ff71c09ba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:54.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:54.422+0000 7ff723000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff71c0cf5c0 0x7ff71c1e01d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:54.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:54.422+0000 7ff723000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff71c0719d0 con 0x7ff71c0b3c90 2024-03-04T09:26:54.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:54.422+0000 7ff723000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff71c15b340 con 0x7ff71c071050 2024-03-04T09:26:54.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:54.422+0000 7ff723000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff71c071b50 con 0x7ff71c0cf5c0 2024-03-04T09:26:54.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:54.422+0000 7ff71b400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff71c0b3c90 0x7ff71c09ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:54.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:54.422+0000 7ff71b400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff71c0b3c90 0x7ff71c09ba60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:40492/0 (socket says 172.21.15.6:40492) 2024-03-04T09:26:54.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:54.422+0000 7ff71b400640 1 -- 172.21.15.6:0/2054531075 learned_addr learned my addr 172.21.15.6:0/2054531075 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:54.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:54.423+0000 7ff71be00640 1 --2- 172.21.15.6:0/2054531075 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff71c071050 0x7ff71c154a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:54.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:54.423+0000 7ff721000640 1 --2- 172.21.15.6:0/2054531075 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff71c0cf5c0 0x7ff71c1e01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:54.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:54.423+0000 7ff71b400640 1 -- 172.21.15.6:0/2054531075 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff71c0cf5c0 msgr2=0x7ff71c1e01d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:54.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:54.423+0000 7ff71b400640 1 --2- 172.21.15.6:0/2054531075 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff71c0cf5c0 0x7ff71c1e01d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:54.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:54.423+0000 7ff71b400640 1 -- 172.21.15.6:0/2054531075 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff71c071050 msgr2=0x7ff71c154a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:54.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:54.423+0000 7ff71b400640 1 --2- 172.21.15.6:0/2054531075 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff71c071050 0x7ff71c154a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:54.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:54.423+0000 7ff71b400640 1 -- 172.21.15.6:0/2054531075 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff71c09c360 con 0x7ff71c0b3c90 2024-03-04T09:26:54.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:54.423+0000 7ff71b400640 1 --2- 172.21.15.6:0/2054531075 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff71c0b3c90 0x7ff71c09ba60 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7ff70400c3c0 tx=0x7ff70400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:54.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:54.424+0000 7ff713e00640 1 -- 172.21.15.6:0/2054531075 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7040052d0 con 0x7ff71c0b3c90 2024-03-04T09:26:54.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:54.424+0000 7ff713e00640 1 -- 172.21.15.6:0/2054531075 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff704005490 con 0x7ff71c0b3c90 2024-03-04T09:26:54.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:54.424+0000 7ff723000640 1 -- 172.21.15.6:0/2054531075 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff71c09c640 con 0x7ff71c0b3c90 2024-03-04T09:26:54.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:54.424+0000 7ff713e00640 1 -- 172.21.15.6:0/2054531075 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff704007260 con 0x7ff71c0b3c90 2024-03-04T09:26:54.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:54.424+0000 7ff723000640 1 -- 172.21.15.6:0/2054531075 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff71c09cc00 con 0x7ff71c0b3c90 2024-03-04T09:26:54.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:54.425+0000 7ff723000640 1 -- 172.21.15.6:0/2054531075 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff71c04f3a0 con 0x7ff71c0b3c90 2024-03-04T09:26:54.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:54.426+0000 7ff713e00640 1 -- 172.21.15.6:0/2054531075 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff704007400 con 0x7ff71c0b3c90 2024-03-04T09:26:54.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:54.427+0000 7ff713e00640 1 --2- 172.21.15.6:0/2054531075 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff6f806b4d0 0x7ff6f806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:54.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:54.427+0000 7ff713e00640 1 -- 172.21.15.6:0/2054531075 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff704089720 con 0x7ff71c0b3c90 2024-03-04T09:26:54.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:54.428+0000 7ff71be00640 1 --2- 172.21.15.6:0/2054531075 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff6f806b4d0 0x7ff6f806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:54.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:54.429+0000 7ff71be00640 1 --2- 172.21.15.6:0/2054531075 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff6f806b4d0 0x7ff6f806d990 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7ff70c038000 tx=0x7ff70c032000 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:54.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:54.432+0000 7ff713e00640 1 -- 172.21.15.6:0/2054531075 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff704058510 con 0x7ff71c0b3c90 2024-03-04T09:26:54.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:54.671+0000 7ff723000640 1 -- 172.21.15.6:0/2054531075 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff71c09cfe0 con 0x7ff71c0b3c90 2024-03-04T09:26:54.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:54.672+0000 7ff713e00640 1 -- 172.21.15.6:0/2054531075 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+4977 (secure 0 0 0) 0x7ff704057e10 con 0x7ff71c0b3c90 2024-03-04T09:26:54.669 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:26:54.669 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4685,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/1703428647","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":1703428647},{"type":"v1","addr":"172.21.15.146:6837","nonce":1703428647}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4837,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/2617429530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2617429530},{"type":"v1","addr":"172.21.15.6:6833","nonce":2617429530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":5043,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/2744913857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":2744913857},{"type":"v1","addr":"172.21.15.6:6835","nonce":2744913857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:26:30.445199+0000","modified":"2024-03-04T09:26:33.534893+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5046},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5046":{"gid":5046,"name":"d","rank":0,"incarnation":12,"state":"up:active","state_seq":4,"addr":"172.21.15.146:6836/2285803221","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":2285803221},{"type":"v1","addr":"172.21.15.146:6836","nonce":2285803221}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5046,"qdb_cluster":[5046]},"id":2}]} 2024-03-04T09:26:54.669 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 16 2024-03-04T09:26:54.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:54.674+0000 7ff723000640 1 -- 172.21.15.6:0/2054531075 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff6f806b4d0 msgr2=0x7ff6f806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:54.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:54.674+0000 7ff723000640 1 --2- 172.21.15.6:0/2054531075 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff6f806b4d0 0x7ff6f806d990 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7ff70c038000 tx=0x7ff70c032000 comp rx=0 tx=0).stop 2024-03-04T09:26:54.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:54.674+0000 7ff723000640 1 -- 172.21.15.6:0/2054531075 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff71c0b3c90 msgr2=0x7ff71c09ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:54.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:54.674+0000 7ff723000640 1 --2- 172.21.15.6:0/2054531075 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff71c0b3c90 0x7ff71c09ba60 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7ff70400c3c0 tx=0x7ff70400c880 comp rx=0 tx=0).stop 2024-03-04T09:26:54.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:54.674+0000 7ff723000640 1 -- 172.21.15.6:0/2054531075 shutdown_connections 2024-03-04T09:26:54.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:54.674+0000 7ff723000640 1 --2- 172.21.15.6:0/2054531075 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff6f806b4d0 0x7ff6f806d990 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:54.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:54.674+0000 7ff723000640 1 --2- 172.21.15.6:0/2054531075 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff71c0cf5c0 0x7ff71c1e01d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:54.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:54.674+0000 7ff723000640 1 --2- 172.21.15.6:0/2054531075 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff71c0b3c90 0x7ff71c09ba60 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:54.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:54.674+0000 7ff723000640 1 --2- 172.21.15.6:0/2054531075 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff71c071050 0x7ff71c154a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:54.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:54.674+0000 7ff723000640 1 -- 172.21.15.6:0/2054531075 >> 172.21.15.6:0/2054531075 conn(0x7ff71c06afb0 msgr2=0x7ff71c0cc790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:54.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:54.674+0000 7ff723000640 1 -- 172.21.15.6:0/2054531075 shutdown_connections 2024-03-04T09:26:54.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:54.674+0000 7ff723000640 1 -- 172.21.15.6:0/2054531075 wait complete. 2024-03-04T09:26:54.691 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json perf dump purge_queue pq_executed 2024-03-04T09:26:54.917 INFO:teuthology.orchestra.run.smithi146.stdout:{"purge_queue":{"pq_executed":1001}} 2024-03-04T09:26:54.918 DEBUG:tasks.cephfs.filesystem:_json_asok output { "purge_queue": { "pq_executed": 1001 } } 2024-03-04T09:26:54.918 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata stat 10000000000.00000000 2024-03-04T09:26:55.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.049+0000 7f37a0b84f40 1 Processor -- start 2024-03-04T09:26:55.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.049+0000 7f37a0b84f40 1 -- start start 2024-03-04T09:26:55.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.050+0000 7f37a0b84f40 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557517513880 0x55751759e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:55.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.050+0000 7f37a0b84f40 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x557517513c60 0x55751759e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:55.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.050+0000 7f37a0b84f40 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5575175a53f0 0x5575175a5870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:55.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.050+0000 7f37a0b84f40 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x55751748cfd0 con 0x557517513c60 2024-03-04T09:26:55.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.050+0000 7f37a0b84f40 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x557517490f30 con 0x557517513880 2024-03-04T09:26:55.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.050+0000 7f37a0b84f40 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x5575174e5a50 con 0x5575175a53f0 2024-03-04T09:26:55.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.050+0000 7f3795400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x557517513c60 0x55751759e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:55.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.050+0000 7f3795400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x557517513c60 0x55751759e6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:40506/0 (socket says 172.21.15.6:40506) 2024-03-04T09:26:55.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.050+0000 7f3795400640 1 -- 172.21.15.6:0/3390352119 learned_addr learned my addr 172.21.15.6:0/3390352119 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:55.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.050+0000 7f379de00640 1 --2- 172.21.15.6:0/3390352119 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557517513880 0x55751759e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:55.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.050+0000 7f379e800640 1 --2- 172.21.15.6:0/3390352119 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5575175a53f0 0x5575175a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:55.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.051+0000 7f3795400640 1 -- 172.21.15.6:0/3390352119 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5575175a53f0 msgr2=0x5575175a5870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:55.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.051+0000 7f3795400640 1 --2- 172.21.15.6:0/3390352119 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5575175a53f0 0x5575175a5870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:55.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.051+0000 7f3795400640 1 -- 172.21.15.6:0/3390352119 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557517513880 msgr2=0x55751759e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:55.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.051+0000 7f3795400640 1 --2- 172.21.15.6:0/3390352119 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557517513880 0x55751759e1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:55.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.051+0000 7f3795400640 1 -- 172.21.15.6:0/3390352119 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5575175a4f20 con 0x557517513c60 2024-03-04T09:26:55.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.051+0000 7f3795400640 1 --2- 172.21.15.6:0/3390352119 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x557517513c60 0x55751759e6e0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f378800d290 tx=0x7f378800d750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fef646d1397184b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:55.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.051+0000 7f379ca00640 1 -- 172.21.15.6:0/3390352119 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3788003110 con 0x557517513c60 2024-03-04T09:26:55.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.051+0000 7f379ca00640 1 -- 172.21.15.6:0/3390352119 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37880032b0 con 0x557517513c60 2024-03-04T09:26:55.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.051+0000 7f379ca00640 1 -- 172.21.15.6:0/3390352119 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3788001600 con 0x557517513c60 2024-03-04T09:26:55.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.052+0000 7f37a0b84f40 1 -- 172.21.15.6:0/3390352119 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x557517513c60 msgr2=0x55751759e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:55.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.052+0000 7f37a0b84f40 1 --2- 172.21.15.6:0/3390352119 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x557517513c60 0x55751759e6e0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f378800d290 tx=0x7f378800d750 comp rx=0 tx=0).stop 2024-03-04T09:26:55.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.052+0000 7f37a0b84f40 1 -- 172.21.15.6:0/3390352119 shutdown_connections 2024-03-04T09:26:55.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.052+0000 7f37a0b84f40 1 --2- 172.21.15.6:0/3390352119 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5575175a53f0 0x5575175a5870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:55.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.052+0000 7f37a0b84f40 1 --2- 172.21.15.6:0/3390352119 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x557517513c60 0x55751759e6e0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:55.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.052+0000 7f37a0b84f40 1 --2- 172.21.15.6:0/3390352119 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557517513880 0x55751759e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:55.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.052+0000 7f37a0b84f40 1 -- 172.21.15.6:0/3390352119 >> 172.21.15.6:0/3390352119 conn(0x557517515c10 msgr2=0x557517518900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:55.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.052+0000 7f37a0b84f40 1 -- 172.21.15.6:0/3390352119 shutdown_connections 2024-03-04T09:26:55.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.052+0000 7f37a0b84f40 1 -- 172.21.15.6:0/3390352119 wait complete. 2024-03-04T09:26:55.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.053+0000 7f37a0b84f40 1 Processor -- start 2024-03-04T09:26:55.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.053+0000 7f37a0b84f40 1 -- start start 2024-03-04T09:26:55.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.054+0000 7f37a0b84f40 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557517513880 0x557517514660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:55.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.054+0000 7f37a0b84f40 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x557517513c60 0x55751759ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:55.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.054+0000 7f37a0b84f40 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5575175a53f0 0x5575175bb300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:55.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.054+0000 7f37a0b84f40 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x5575175bd450 con 0x5575175a53f0 2024-03-04T09:26:55.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.054+0000 7f37a0b84f40 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x5575175173f0 con 0x557517513880 2024-03-04T09:26:55.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.054+0000 7f37a0b84f40 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x557517530160 con 0x557517513c60 2024-03-04T09:26:55.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.054+0000 7f379e800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5575175a53f0 0x5575175bb300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:55.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.054+0000 7f379e800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5575175a53f0 0x5575175bb300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:40510/0 (socket says 172.21.15.6:40510) 2024-03-04T09:26:55.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.054+0000 7f379e800640 1 -- 172.21.15.6:0/3550078496 learned_addr learned my addr 172.21.15.6:0/3550078496 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:55.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.055+0000 7f3795400640 1 --2- 172.21.15.6:0/3550078496 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x557517513c60 0x55751759ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:55.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.055+0000 7f379de00640 1 --2- 172.21.15.6:0/3550078496 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557517513880 0x557517514660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:55.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.055+0000 7f379e800640 1 -- 172.21.15.6:0/3550078496 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x557517513c60 msgr2=0x55751759ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:55.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.055+0000 7f379e800640 1 --2- 172.21.15.6:0/3550078496 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x557517513c60 0x55751759ee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:55.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.055+0000 7f379e800640 1 -- 172.21.15.6:0/3550078496 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557517513880 msgr2=0x557517514660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:55.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.055+0000 7f379e800640 1 --2- 172.21.15.6:0/3550078496 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557517513880 0x557517514660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:55.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.055+0000 7f379e800640 1 -- 172.21.15.6:0/3550078496 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5575175bde40 con 0x5575175a53f0 2024-03-04T09:26:55.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.055+0000 7f3795400640 1 --2- 172.21.15.6:0/3550078496 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x557517513c60 0x55751759ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T09:26:55.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.055+0000 7f379de00640 1 --2- 172.21.15.6:0/3550078496 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557517513880 0x557517514660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T09:26:55.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.055+0000 7f379e800640 1 --2- 172.21.15.6:0/3550078496 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5575175a53f0 0x5575175bb300 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f3798005fd0 tx=0x7f37980106f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:55.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.056+0000 7f3797400640 1 -- 172.21.15.6:0/3550078496 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3798018070 con 0x5575175a53f0 2024-03-04T09:26:55.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.056+0000 7f3797400640 1 -- 172.21.15.6:0/3550078496 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3798010e40 con 0x5575175a53f0 2024-03-04T09:26:55.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.056+0000 7f3797400640 1 -- 172.21.15.6:0/3550078496 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3798014870 con 0x5575175a53f0 2024-03-04T09:26:55.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.056+0000 7f37a0b84f40 1 -- 172.21.15.6:0/3550078496 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x5575175be120 con 0x5575175a53f0 2024-03-04T09:26:55.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.056+0000 7f37a0b84f40 1 -- 172.21.15.6:0/3550078496 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5575175be6e0 con 0x5575175a53f0 2024-03-04T09:26:55.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.058+0000 7f3797400640 1 -- 172.21.15.6:0/3550078496 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f379801b070 con 0x5575175a53f0 2024-03-04T09:26:55.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.059+0000 7f3797400640 1 --2- 172.21.15.6:0/3550078496 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f377c069eb0 0x7f377c06c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:55.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.059+0000 7f3797400640 1 -- 172.21.15.6:0/3550078496 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3798089270 con 0x5575175a53f0 2024-03-04T09:26:55.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.059+0000 7f37a0b84f40 1 --2- 172.21.15.6:0/3550078496 >> [v2:172.21.15.6:6800/1558806123,v1:172.21.15.6:6801/1558806123] conn(0x55751759f3b0 0x55751765a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:55.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.059+0000 7f37a0b84f40 1 -- 172.21.15.6:0/3550078496 --> [v2:172.21.15.6:6800/1558806123,v1:172.21.15.6:6801/1558806123] -- osd_op(unknown.0.0:1 4.10 4:0cf6ad0f:::10000000000.00000000:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e33) v9 -- 0x55751765abe0 con 0x55751759f3b0 2024-03-04T09:26:55.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.059+0000 7f379de00640 1 --2- 172.21.15.6:0/3550078496 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f377c069eb0 0x7f377c06c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:55.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.059+0000 7f3795400640 1 --2- 172.21.15.6:0/3550078496 >> [v2:172.21.15.6:6800/1558806123,v1:172.21.15.6:6801/1558806123] conn(0x55751759f3b0 0x55751765a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:55.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.060+0000 7f3795400640 1 --2- 172.21.15.6:0/3550078496 >> [v2:172.21.15.6:6800/1558806123,v1:172.21.15.6:6801/1558806123] conn(0x55751759f3b0 0x55751765a6a0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:55.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.061+0000 7f3795400640 1 -- 172.21.15.6:0/3550078496 <== osd.3 v2:172.21.15.6:6800/1558806123 1 ==== osd_op_reply(1 10000000000.00000000 [stat] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 164+0+0 (crc 0 0 0) 0x7f378800e040 con 0x55751759f3b0 2024-03-04T09:26:55.058 INFO:teuthology.orchestra.run.smithi006.stderr: error stat-ing cephfs_metadata/10000000000.00000000: (2) No such file or directory 2024-03-04T09:26:55.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.061+0000 7f379de00640 1 --2- 172.21.15.6:0/3550078496 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f377c069eb0 0x7f377c06c370 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f37840040d0 tx=0x7f379004e000 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:55.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.061+0000 7f37a0b84f40 1 -- 172.21.15.6:0/3550078496 >> [v2:172.21.15.6:6800/1558806123,v1:172.21.15.6:6801/1558806123] conn(0x55751759f3b0 msgr2=0x55751765a6a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:55.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.061+0000 7f37a0b84f40 1 --2- 172.21.15.6:0/3550078496 >> [v2:172.21.15.6:6800/1558806123,v1:172.21.15.6:6801/1558806123] conn(0x55751759f3b0 0x55751765a6a0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:55.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.061+0000 7f37a0b84f40 1 -- 172.21.15.6:0/3550078496 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f377c069eb0 msgr2=0x7f377c06c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:55.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.061+0000 7f37a0b84f40 1 --2- 172.21.15.6:0/3550078496 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f377c069eb0 0x7f377c06c370 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f37840040d0 tx=0x7f379004e000 comp rx=0 tx=0).stop 2024-03-04T09:26:55.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.061+0000 7f37a0b84f40 1 -- 172.21.15.6:0/3550078496 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5575175a53f0 msgr2=0x5575175bb300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:55.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.061+0000 7f37a0b84f40 1 --2- 172.21.15.6:0/3550078496 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5575175a53f0 0x5575175bb300 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f3798005fd0 tx=0x7f37980106f0 comp rx=0 tx=0).stop 2024-03-04T09:26:55.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.061+0000 7f379e800640 1 -- 172.21.15.6:0/3550078496 reap_dead start 2024-03-04T09:26:55.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.061+0000 7f37a0b84f40 1 -- 172.21.15.6:0/3550078496 shutdown_connections 2024-03-04T09:26:55.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.061+0000 7f37a0b84f40 1 -- 172.21.15.6:0/3550078496 >> 172.21.15.6:0/3550078496 conn(0x557517515c10 msgr2=0x55751759d200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:55.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.061+0000 7f37a0b84f40 1 -- 172.21.15.6:0/3550078496 shutdown_connections 2024-03-04T09:26:55.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.061+0000 7f37a0b84f40 1 -- 172.21.15.6:0/3550078496 wait complete. 2024-03-04T09:26:55.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T09:26:55.063 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph df --format=json-pretty 2024-03-04T09:26:55.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.255+0000 7f24dca00640 1 Processor -- start 2024-03-04T09:26:55.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.256+0000 7f24dca00640 1 -- start start 2024-03-04T09:26:55.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.257+0000 7f24dca00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f24d806f690 0x7f24d80728b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:55.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.257+0000 7f24dca00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f24d80b4120 0x7f24d80b59d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:55.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.257+0000 7f24dca00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f24d806afb0 0x7f24d806b440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:55.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.257+0000 7f24dca00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24d8072f70 con 0x7f24d806f690 2024-03-04T09:26:55.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.257+0000 7f24dca00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24d80d1a10 con 0x7f24d80b4120 2024-03-04T09:26:55.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.257+0000 7f24dca00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24d80d1450 con 0x7f24d806afb0 2024-03-04T09:26:55.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.258+0000 7f24d6800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f24d806f690 0x7f24d80728b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:55.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.258+0000 7f24d6800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f24d806f690 0x7f24d80728b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:40514/0 (socket says 172.21.15.6:40514) 2024-03-04T09:26:55.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.258+0000 7f24d6800640 1 -- 172.21.15.6:0/2354530042 learned_addr learned my addr 172.21.15.6:0/2354530042 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:55.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.258+0000 7f24d5e00640 1 --2- 172.21.15.6:0/2354530042 >> v2:172.21.15.146:3300/0 conn(0x7f24d80b4120 0x7f24d80b59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:55.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.258+0000 7f24d7200640 1 --2- 172.21.15.6:0/2354530042 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f24d806afb0 0x7f24d806b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:55.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.258+0000 7f24d6800640 1 -- 172.21.15.6:0/2354530042 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f24d806afb0 msgr2=0x7f24d806b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:55.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.258+0000 7f24d6800640 1 --2- 172.21.15.6:0/2354530042 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f24d806afb0 0x7f24d806b440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:55.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.258+0000 7f24d6800640 1 -- 172.21.15.6:0/2354530042 >> v2:172.21.15.146:3300/0 conn(0x7f24d80b4120 msgr2=0x7f24d80b59d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:55.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.258+0000 7f24d6800640 1 --2- 172.21.15.6:0/2354530042 >> v2:172.21.15.146:3300/0 conn(0x7f24d80b4120 0x7f24d80b59d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:55.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.258+0000 7f24d6800640 1 -- 172.21.15.6:0/2354530042 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24d80d1730 con 0x7f24d806f690 2024-03-04T09:26:55.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.259+0000 7f24d6800640 1 --2- 172.21.15.6:0/2354530042 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f24d806f690 0x7f24d80728b0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f24d806c090 tx=0x7f24c802b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdcaa7319026c0eb server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:55.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.260+0000 7f24d5400640 1 -- 172.21.15.6:0/2354530042 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24c80317d0 con 0x7f24d806f690 2024-03-04T09:26:55.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.260+0000 7f24d5400640 1 -- 172.21.15.6:0/2354530042 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24c8037070 con 0x7f24d806f690 2024-03-04T09:26:55.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.260+0000 7f24d5400640 1 -- 172.21.15.6:0/2354530042 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24c8031b90 con 0x7f24d806f690 2024-03-04T09:26:55.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.260+0000 7f24dca00640 1 -- 172.21.15.6:0/2354530042 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f24d806f690 msgr2=0x7f24d80728b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:55.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.260+0000 7f24dca00640 1 --2- 172.21.15.6:0/2354530042 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f24d806f690 0x7f24d80728b0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f24d806c090 tx=0x7f24c802b870 comp rx=0 tx=0).stop 2024-03-04T09:26:55.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.260+0000 7f24dca00640 1 -- 172.21.15.6:0/2354530042 shutdown_connections 2024-03-04T09:26:55.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.260+0000 7f24dca00640 1 --2- 172.21.15.6:0/2354530042 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f24d806afb0 0x7f24d806b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:55.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.260+0000 7f24dca00640 1 --2- 172.21.15.6:0/2354530042 >> v2:172.21.15.146:3300/0 conn(0x7f24d80b4120 0x7f24d80b59d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:55.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.260+0000 7f24dca00640 1 --2- 172.21.15.6:0/2354530042 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f24d806f690 0x7f24d80728b0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:55.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.260+0000 7f24dca00640 1 -- 172.21.15.6:0/2354530042 >> 172.21.15.6:0/2354530042 conn(0x7f24d806c0c0 msgr2=0x7f24d806c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:55.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.261+0000 7f24dca00640 1 -- 172.21.15.6:0/2354530042 shutdown_connections 2024-03-04T09:26:55.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.261+0000 7f24dca00640 1 -- 172.21.15.6:0/2354530042 wait complete. 2024-03-04T09:26:55.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.263+0000 7f24dca00640 1 Processor -- start 2024-03-04T09:26:55.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.263+0000 7f24dca00640 1 -- start start 2024-03-04T09:26:55.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.264+0000 7f24dca00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f24d806afb0 0x7f24d8144620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:55.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.264+0000 7f24dca00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f24d806f690 0x7f24d80caf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:55.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.264+0000 7f24dca00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f24d80b4120 0x7f24d80c7730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:55.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.264+0000 7f24dca00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24d8161a50 con 0x7f24d806f690 2024-03-04T09:26:55.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.264+0000 7f24dca00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24d806fa70 con 0x7f24d80b4120 2024-03-04T09:26:55.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.264+0000 7f24dca00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24d8161bd0 con 0x7f24d806afb0 2024-03-04T09:26:55.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.264+0000 7f24d5e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f24d806f690 0x7f24d80caf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:55.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.264+0000 7f24d5e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f24d806f690 0x7f24d80caf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:40522/0 (socket says 172.21.15.6:40522) 2024-03-04T09:26:55.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.264+0000 7f24d5e00640 1 -- 172.21.15.6:0/3680911242 learned_addr learned my addr 172.21.15.6:0/3680911242 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:55.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.265+0000 7f24d6800640 1 --2- 172.21.15.6:0/3680911242 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f24d806afb0 0x7f24d8144620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:55.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.265+0000 7f24d7200640 1 --2- 172.21.15.6:0/3680911242 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f24d80b4120 0x7f24d80c7730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:55.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.265+0000 7f24d5e00640 1 -- 172.21.15.6:0/3680911242 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f24d806afb0 msgr2=0x7f24d8144620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:55.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.265+0000 7f24d5e00640 1 --2- 172.21.15.6:0/3680911242 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f24d806afb0 0x7f24d8144620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:55.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.265+0000 7f24d5e00640 1 -- 172.21.15.6:0/3680911242 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f24d80b4120 msgr2=0x7f24d80c7730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:55.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.265+0000 7f24d5e00640 1 --2- 172.21.15.6:0/3680911242 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f24d80b4120 0x7f24d80c7730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:55.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.265+0000 7f24d5e00640 1 -- 172.21.15.6:0/3680911242 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24d80c6690 con 0x7f24d806f690 2024-03-04T09:26:55.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.265+0000 7f24d5e00640 1 --2- 172.21.15.6:0/3680911242 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f24d806f690 0x7f24d80caf90 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f24c400d5f0 tx=0x7f24c400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:55.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.265+0000 7f24cea00640 1 -- 172.21.15.6:0/3680911242 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24c40125f0 con 0x7f24d806f690 2024-03-04T09:26:55.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.265+0000 7f24cea00640 1 -- 172.21.15.6:0/3680911242 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24c4019070 con 0x7f24d806f690 2024-03-04T09:26:55.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.265+0000 7f24dca00640 1 -- 172.21.15.6:0/3680911242 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24d80c6970 con 0x7f24d806f690 2024-03-04T09:26:55.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.265+0000 7f24cea00640 1 -- 172.21.15.6:0/3680911242 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24c40129d0 con 0x7f24d806f690 2024-03-04T09:26:55.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.266+0000 7f24dca00640 1 -- 172.21.15.6:0/3680911242 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24d80c6f30 con 0x7f24d806f690 2024-03-04T09:26:55.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.267+0000 7f24dca00640 1 -- 172.21.15.6:0/3680911242 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24d804f330 con 0x7f24d806f690 2024-03-04T09:26:55.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.267+0000 7f24cea00640 1 -- 172.21.15.6:0/3680911242 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f24c4012b70 con 0x7f24d806f690 2024-03-04T09:26:55.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.269+0000 7f24cea00640 1 --2- 172.21.15.6:0/3680911242 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f24b806b4d0 0x7f24b806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:55.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.269+0000 7f24cea00640 1 -- 172.21.15.6:0/3680911242 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f24c408a540 con 0x7f24d806f690 2024-03-04T09:26:55.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.269+0000 7f24d6800640 1 --2- 172.21.15.6:0/3680911242 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f24b806b4d0 0x7f24b806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:55.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.271+0000 7f24d6800640 1 --2- 172.21.15.6:0/3680911242 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f24b806b4d0 0x7f24b806d990 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f24c8036000 tx=0x7f24c803d040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:55.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.274+0000 7f24cea00640 1 -- 172.21.15.6:0/3680911242 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f24c4059450 con 0x7f24d806f690 2024-03-04T09:26:55.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.542+0000 7f24dca00640 1 -- 172.21.15.6:0/3680911242 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "df", "format": "json-pretty"} v 0) v1 -- 0x7f24d8072460 con 0x7f24d806f690 2024-03-04T09:26:55.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.543+0000 7f24cea00640 1 -- 172.21.15.6:0/3680911242 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "df", "format": "json-pretty"}]=0 v0) v1 ==== 75+0+1657 (secure 0 0 0) 0x7f24c4058d50 con 0x7f24d806f690 2024-03-04T09:26:55.540 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:26:55.540 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-04T09:26:55.540 INFO:teuthology.orchestra.run.smithi006.stdout: "stats": { 2024-03-04T09:26:55.540 INFO:teuthology.orchestra.run.smithi006.stdout: "total_bytes": 773094113280, 2024-03-04T09:26:55.540 INFO:teuthology.orchestra.run.smithi006.stdout: "total_avail_bytes": 772841652224, 2024-03-04T09:26:55.540 INFO:teuthology.orchestra.run.smithi006.stdout: "total_used_bytes": 252461056, 2024-03-04T09:26:55.540 INFO:teuthology.orchestra.run.smithi006.stdout: "total_used_raw_bytes": 252461056, 2024-03-04T09:26:55.540 INFO:teuthology.orchestra.run.smithi006.stdout: "total_used_raw_ratio": 0.00032655926770530641, 2024-03-04T09:26:55.540 INFO:teuthology.orchestra.run.smithi006.stdout: "num_osds": 8, 2024-03-04T09:26:55.541 INFO:teuthology.orchestra.run.smithi006.stdout: "num_per_pool_osds": 8, 2024-03-04T09:26:55.541 INFO:teuthology.orchestra.run.smithi006.stdout: "num_per_pool_omap_osds": 8 2024-03-04T09:26:55.541 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:26:55.541 INFO:teuthology.orchestra.run.smithi006.stdout: "stats_by_class": { 2024-03-04T09:26:55.541 INFO:teuthology.orchestra.run.smithi006.stdout: "ssd": { 2024-03-04T09:26:55.541 INFO:teuthology.orchestra.run.smithi006.stdout: "total_bytes": 773094113280, 2024-03-04T09:26:55.541 INFO:teuthology.orchestra.run.smithi006.stdout: "total_avail_bytes": 772841652224, 2024-03-04T09:26:55.541 INFO:teuthology.orchestra.run.smithi006.stdout: "total_used_bytes": 252461056, 2024-03-04T09:26:55.541 INFO:teuthology.orchestra.run.smithi006.stdout: "total_used_raw_bytes": 252461056, 2024-03-04T09:26:55.541 INFO:teuthology.orchestra.run.smithi006.stdout: "total_used_raw_ratio": 0.00032655926770530641 2024-03-04T09:26:55.541 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:26:55.541 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:26:55.541 INFO:teuthology.orchestra.run.smithi006.stdout: "pools": [ 2024-03-04T09:26:55.541 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:26:55.541 INFO:teuthology.orchestra.run.smithi006.stdout: "name": ".mgr", 2024-03-04T09:26:55.542 INFO:teuthology.orchestra.run.smithi006.stdout: "id": 1, 2024-03-04T09:26:55.542 INFO:teuthology.orchestra.run.smithi006.stdout: "stats": { 2024-03-04T09:26:55.542 INFO:teuthology.orchestra.run.smithi006.stdout: "stored": 590368, 2024-03-04T09:26:55.542 INFO:teuthology.orchestra.run.smithi006.stdout: "objects": 2, 2024-03-04T09:26:55.542 INFO:teuthology.orchestra.run.smithi006.stdout: "kb_used": 1160, 2024-03-04T09:26:55.542 INFO:teuthology.orchestra.run.smithi006.stdout: "bytes_used": 1187840, 2024-03-04T09:26:55.542 INFO:teuthology.orchestra.run.smithi006.stdout: "percent_used": 1.6179296835616697e-06, 2024-03-04T09:26:55.542 INFO:teuthology.orchestra.run.smithi006.stdout: "max_avail": 367085813760 2024-03-04T09:26:55.542 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:26:55.542 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:26:55.542 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:26:55.542 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "cephfs_metadata", 2024-03-04T09:26:55.542 INFO:teuthology.orchestra.run.smithi006.stdout: "id": 4, 2024-03-04T09:26:55.542 INFO:teuthology.orchestra.run.smithi006.stdout: "stats": { 2024-03-04T09:26:55.543 INFO:teuthology.orchestra.run.smithi006.stdout: "stored": 676270, 2024-03-04T09:26:55.543 INFO:teuthology.orchestra.run.smithi006.stdout: "objects": 23, 2024-03-04T09:26:55.543 INFO:teuthology.orchestra.run.smithi006.stdout: "kb_used": 1384, 2024-03-04T09:26:55.543 INFO:teuthology.orchestra.run.smithi006.stdout: "bytes_used": 1417216, 2024-03-04T09:26:55.543 INFO:teuthology.orchestra.run.smithi006.stdout: "percent_used": 1.9303568024042761e-06, 2024-03-04T09:26:55.543 INFO:teuthology.orchestra.run.smithi006.stdout: "max_avail": 367085813760 2024-03-04T09:26:55.543 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:26:55.543 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:26:55.543 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:26:55.543 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "cephfs_data", 2024-03-04T09:26:55.543 INFO:teuthology.orchestra.run.smithi006.stdout: "id": 5, 2024-03-04T09:26:55.543 INFO:teuthology.orchestra.run.smithi006.stdout: "stats": { 2024-03-04T09:26:55.543 INFO:teuthology.orchestra.run.smithi006.stdout: "stored": 0, 2024-03-04T09:26:55.544 INFO:teuthology.orchestra.run.smithi006.stdout: "objects": 0, 2024-03-04T09:26:55.544 INFO:teuthology.orchestra.run.smithi006.stdout: "kb_used": 0, 2024-03-04T09:26:55.544 INFO:teuthology.orchestra.run.smithi006.stdout: "bytes_used": 0, 2024-03-04T09:26:55.544 INFO:teuthology.orchestra.run.smithi006.stdout: "percent_used": 0, 2024-03-04T09:26:55.544 INFO:teuthology.orchestra.run.smithi006.stdout: "max_avail": 367085813760 2024-03-04T09:26:55.544 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:26:55.544 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:26:55.544 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-04T09:26:55.544 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-04T09:26:55.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.546+0000 7f24dca00640 1 -- 172.21.15.6:0/3680911242 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f24b806b4d0 msgr2=0x7f24b806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:55.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.546+0000 7f24dca00640 1 --2- 172.21.15.6:0/3680911242 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f24b806b4d0 0x7f24b806d990 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f24c8036000 tx=0x7f24c803d040 comp rx=0 tx=0).stop 2024-03-04T09:26:55.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.546+0000 7f24dca00640 1 -- 172.21.15.6:0/3680911242 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f24d806f690 msgr2=0x7f24d80caf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:55.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.546+0000 7f24dca00640 1 --2- 172.21.15.6:0/3680911242 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f24d806f690 0x7f24d80caf90 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f24c400d5f0 tx=0x7f24c400dab0 comp rx=0 tx=0).stop 2024-03-04T09:26:55.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.546+0000 7f24dca00640 1 -- 172.21.15.6:0/3680911242 shutdown_connections 2024-03-04T09:26:55.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.546+0000 7f24dca00640 1 --2- 172.21.15.6:0/3680911242 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f24d80b4120 0x7f24d80c7730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:55.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.546+0000 7f24dca00640 1 --2- 172.21.15.6:0/3680911242 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f24d806f690 0x7f24d80caf90 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:55.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.546+0000 7f24dca00640 1 --2- 172.21.15.6:0/3680911242 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f24b806b4d0 0x7f24b806d990 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:55.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.546+0000 7f24dca00640 1 --2- 172.21.15.6:0/3680911242 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f24d806afb0 0x7f24d8144620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:55.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.546+0000 7f24dca00640 1 -- 172.21.15.6:0/3680911242 >> 172.21.15.6:0/3680911242 conn(0x7f24d806c0c0 msgr2=0x7f24d8161f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:55.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.546+0000 7f24dca00640 1 -- 172.21.15.6:0/3680911242 shutdown_connections 2024-03-04T09:26:55.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:55.546+0000 7f24dca00640 1 -- 172.21.15.6:0/3680911242 wait complete. 2024-03-04T09:26:55.561 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-03-04T09:26:55.561 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-04T09:26:55.562 DEBUG:teuthology.orchestra.run.smithi006:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-04T09:26:55.672 DEBUG:teuthology.orchestra.run.smithi146:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-04T09:26:55.783 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-04T09:26:55.784 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-04T09:26:55.784 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T09:26:55.845 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-04T09:26:55.846 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T09:26:55.904 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T09:26:56.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.140+0000 7fd140200640 1 Processor -- start 2024-03-04T09:26:56.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.140+0000 7fd140200640 1 -- start start 2024-03-04T09:26:56.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.141+0000 7fd140200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd1380af1a0 0x7fd13815c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:56.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.142+0000 7fd140200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd13815cc40 con 0x7fd1380af580 2024-03-04T09:26:56.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.142+0000 7fd140200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd13815cf20 con 0x7fd1380cdc20 2024-03-04T09:26:56.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.142+0000 7fd140200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd13815d200 con 0x7fd1380af1a0 2024-03-04T09:26:56.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.142+0000 7fd13ce00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fd1380af580 0x7fd1380b4c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:34014/0 (socket says 172.21.15.6:34014) 2024-03-04T09:26:56.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.142+0000 7fd13ce00640 1 -- 172.21.15.6:0/4061042595 learned_addr learned my addr 172.21.15.6:0/4061042595 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:26:56.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.142+0000 7fd13d800640 1 --2- 172.21.15.6:0/4061042595 >> v2:172.21.15.146:3300/0 conn(0x7fd1380af1a0 0x7fd13815c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:56.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.142+0000 7fd137e00640 1 -- 172.21.15.6:0/4061042595 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3645500198 0 0) 0x7fd13815cc40 con 0x7fd1380af580 2024-03-04T09:26:56.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.142+0000 7fd137e00640 1 -- 172.21.15.6:0/4061042595 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd120003520 con 0x7fd1380af580 2024-03-04T09:26:56.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.143+0000 7fd137e00640 1 -- 172.21.15.6:0/4061042595 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 920558480 0 0) 0x7fd120003520 con 0x7fd1380af580 2024-03-04T09:26:56.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.143+0000 7fd137e00640 1 -- 172.21.15.6:0/4061042595 >> v2:172.21.15.146:3300/0 conn(0x7fd1380af1a0 msgr2=0x7fd13815c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:56.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.143+0000 7fd137e00640 1 --2- 172.21.15.6:0/4061042595 >> v2:172.21.15.146:3300/0 conn(0x7fd1380af1a0 0x7fd13815c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:56.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.143+0000 7fd137e00640 1 -- 172.21.15.6:0/4061042595 >> v1:172.21.15.146:6789/0 conn(0x7fd1380cdc20 legacy=0x7fd1380b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:56.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.143+0000 7fd137e00640 1 -- 172.21.15.6:0/4061042595 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd13815d4e0 con 0x7fd1380af580 2024-03-04T09:26:56.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.143+0000 7fd137e00640 1 -- 172.21.15.6:0/4061042595 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7fd124001670 con 0x7fd1380af580 2024-03-04T09:26:56.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.143+0000 7fd137e00640 1 -- 172.21.15.6:0/4061042595 >> v1:172.21.15.6:6789/0 conn(0x7fd1380af580 legacy=0x7fd1380b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:56.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.143+0000 7fd137e00640 1 --2- 172.21.15.6:0/4061042595 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd120003a70 0x7fd120003e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:56.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.143+0000 7fd137e00640 1 --2- 172.21.15.6:0/4061042595 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd120004800 0x7fd120004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:56.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.143+0000 7fd137e00640 1 --2- 172.21.15.6:0/4061042595 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd1200052e0 0x7fd12000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:56.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.143+0000 7fd137e00640 1 -- 172.21.15.6:0/4061042595 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd13815d200 con 0x7fd120003a70 2024-03-04T09:26:56.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.143+0000 7fd137e00640 1 -- 172.21.15.6:0/4061042595 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd12000c1b0 con 0x7fd120004800 2024-03-04T09:26:56.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.143+0000 7fd137e00640 1 -- 172.21.15.6:0/4061042595 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd12000c3b0 con 0x7fd1200052e0 2024-03-04T09:26:56.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.143+0000 7fd13d800640 1 --2- 172.21.15.6:0/4061042595 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd120003a70 0x7fd120003e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:56.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.144+0000 7fd13d800640 1 -- 172.21.15.6:0/4061042595 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd1200052e0 msgr2=0x7fd12000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:56.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.144+0000 7fd13d800640 1 --2- 172.21.15.6:0/4061042595 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd1200052e0 0x7fd12000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:56.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.144+0000 7fd13d800640 1 -- 172.21.15.6:0/4061042595 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd120004800 msgr2=0x7fd120004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:56.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.144+0000 7fd13d800640 1 --2- 172.21.15.6:0/4061042595 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd120004800 0x7fd120004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:56.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.144+0000 7fd13d800640 1 -- 172.21.15.6:0/4061042595 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd120003520 con 0x7fd120003a70 2024-03-04T09:26:56.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.144+0000 7fd13e200640 1 -- 172.21.15.6:0/4061042595 reap_dead start 2024-03-04T09:26:56.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.145+0000 7fd13d800640 1 --2- 172.21.15.6:0/4061042595 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd120003a70 0x7fd120003e90 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fd12c009ac0 tx=0x7fd120005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab7dd511f66a8a4d server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:56.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.145+0000 7fd137e00640 1 -- 172.21.15.6:0/4061042595 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd12c039070 con 0x7fd120003a70 2024-03-04T09:26:56.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.145+0000 7fd137e00640 1 -- 172.21.15.6:0/4061042595 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd12c02c6b0 con 0x7fd120003a70 2024-03-04T09:26:56.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.145+0000 7fd137e00640 1 -- 172.21.15.6:0/4061042595 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd12c009690 con 0x7fd120003a70 2024-03-04T09:26:56.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.146+0000 7fd140200640 1 -- 172.21.15.6:0/4061042595 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd120003a70 msgr2=0x7fd120003e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:56.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.146+0000 7fd140200640 1 --2- 172.21.15.6:0/4061042595 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd120003a70 0x7fd120003e90 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fd12c009ac0 tx=0x7fd120005830 comp rx=0 tx=0).stop 2024-03-04T09:26:56.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.146+0000 7fd140200640 1 -- 172.21.15.6:0/4061042595 shutdown_connections 2024-03-04T09:26:56.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.146+0000 7fd140200640 1 --2- 172.21.15.6:0/4061042595 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd120003a70 0x7fd120003e90 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:56.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.146+0000 7fd140200640 1 -- 172.21.15.6:0/4061042595 >> 172.21.15.6:0/4061042595 conn(0x7fd13806afb0 msgr2=0x7fd13806b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:56.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.146+0000 7fd140200640 1 -- 172.21.15.6:0/4061042595 shutdown_connections 2024-03-04T09:26:56.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.146+0000 7fd140200640 1 -- 172.21.15.6:0/4061042595 wait complete. 2024-03-04T09:26:56.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.148+0000 7fd140200640 1 Processor -- start 2024-03-04T09:26:56.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.148+0000 7fd140200640 1 -- start start 2024-03-04T09:26:56.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.149+0000 7fd140200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd120003a70 0x7fd138150870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:56.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.149+0000 7fd140200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd13806e100 0x7fd13806e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:56.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.149+0000 7fd140200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd138159890 0x7fd138159d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:56.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.149+0000 7fd140200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd13815c990 con 0x7fd13806e100 2024-03-04T09:26:56.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.149+0000 7fd140200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd13815c810 con 0x7fd120003a70 2024-03-04T09:26:56.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.149+0000 7fd140200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1380b4c20 con 0x7fd138159890 2024-03-04T09:26:56.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.149+0000 7fd13ce00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd13806e100 0x7fd13806e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:56.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.149+0000 7fd13ce00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd13806e100 0x7fd13806e580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:40530/0 (socket says 172.21.15.6:40530) 2024-03-04T09:26:56.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.149+0000 7fd13ce00640 1 -- 172.21.15.6:0/2953788538 learned_addr learned my addr 172.21.15.6:0/2953788538 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:56.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.150+0000 7fd13ce00640 1 -- 172.21.15.6:0/2953788538 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd138159890 msgr2=0x7fd138159d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:56.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.150+0000 7fd13ce00640 1 --2- 172.21.15.6:0/2953788538 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd138159890 0x7fd138159d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:56.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.150+0000 7fd13ce00640 1 -- 172.21.15.6:0/2953788538 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd120003a70 msgr2=0x7fd138150870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:56.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.150+0000 7fd13d800640 1 --2- 172.21.15.6:0/2953788538 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd120003a70 0x7fd138150870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:56.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.150+0000 7fd13ce00640 1 --2- 172.21.15.6:0/2953788538 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd120003a70 0x7fd138150870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:56.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.150+0000 7fd13ce00640 1 -- 172.21.15.6:0/2953788538 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd13815a5c0 con 0x7fd13806e100 2024-03-04T09:26:56.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.150+0000 7fd13ce00640 1 --2- 172.21.15.6:0/2953788538 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd13806e100 0x7fd13806e580 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fd12400c700 tx=0x7fd12400cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:56.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.150+0000 7fd135600640 1 -- 172.21.15.6:0/2953788538 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd124018070 con 0x7fd13806e100 2024-03-04T09:26:56.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.150+0000 7fd135600640 1 -- 172.21.15.6:0/2953788538 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1240072d0 con 0x7fd13806e100 2024-03-04T09:26:56.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.150+0000 7fd135600640 1 -- 172.21.15.6:0/2953788538 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1240047e0 con 0x7fd13806e100 2024-03-04T09:26:56.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.150+0000 7fd140200640 1 -- 172.21.15.6:0/2953788538 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd138158330 con 0x7fd13806e100 2024-03-04T09:26:56.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.150+0000 7fd140200640 1 -- 172.21.15.6:0/2953788538 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1381588f0 con 0x7fd13806e100 2024-03-04T09:26:56.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.151+0000 7fd140200640 1 -- 172.21.15.6:0/2953788538 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd138069510 con 0x7fd13806e100 2024-03-04T09:26:56.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.152+0000 7fd135600640 1 -- 172.21.15.6:0/2953788538 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd12401b070 con 0x7fd13806e100 2024-03-04T09:26:56.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.153+0000 7fd135600640 1 --2- 172.21.15.6:0/2953788538 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd11806b500 0x7fd11806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:56.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.153+0000 7fd135600640 1 -- 172.21.15.6:0/2953788538 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd1240895f0 con 0x7fd13806e100 2024-03-04T09:26:56.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.154+0000 7fd13d800640 1 --2- 172.21.15.6:0/2953788538 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd11806b500 0x7fd11806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:56.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.155+0000 7fd13d800640 1 --2- 172.21.15.6:0/2953788538 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd11806b500 0x7fd11806d9c0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fd12c036000 tx=0x7fd12c030000 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:56.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.158+0000 7fd135600640 1 -- 172.21.15.6:0/2953788538 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd124058460 con 0x7fd13806e100 2024-03-04T09:26:56.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.357+0000 7fd140200640 1 -- 172.21.15.6:0/2953788538 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd13804f3a0 con 0x7fd13806e100 2024-03-04T09:26:56.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.359+0000 7fd135600640 1 -- 172.21.15.6:0/2953788538 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v33) v1 ==== 74+0+16621 (secure 0 0 0) 0x7fd124057d60 con 0x7fd13806e100 2024-03-04T09:26:56.358 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:26:56.358 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":33,"fsid":"edce0578-5ae4-4402-b45a-552bdef84d7c","created":"2024-03-04T09:24:54.523780+0000","modified":"2024-03-04T09:26:31.991047+0000","last_up_change":"2024-03-04T09:25:07.089049+0000","last_in_change":"2024-03-04T09:24:57.982077+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T09:25:07.935198+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-04T09:26:28.556219+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-04T09:26:29.471227+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a96574f-81f1-4bc6-8463-757520b84a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6825","nonce":1364449401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6827","nonce":1364449401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6831","nonce":1364449401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6829","nonce":1364449401}]},"public_addr":"172.21.15.6:6825/1364449401","cluster_addr":"172.21.15.6:6827/1364449401","heartbeat_back_addr":"172.21.15.6:6831/1364449401","heartbeat_front_addr":"172.21.15.6:6829/1364449401","state":["exists","up"]},{"osd":1,"uuid":"7d79b9c1-6b87-476a-bc36-7db5ee64cd90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6809","nonce":2494103078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6811","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6813","nonce":2494103078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6821","nonce":2494103078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6817","nonce":2494103078}]},"public_addr":"172.21.15.6:6809/2494103078","cluster_addr":"172.21.15.6:6813/2494103078","heartbeat_back_addr":"172.21.15.6:6821/2494103078","heartbeat_front_addr":"172.21.15.6:6817/2494103078","state":["exists","up"]},{"osd":2,"uuid":"492331b9-4ee9-40b6-bacd-85d44cd6c45a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6812","nonce":1500066681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6816","nonce":1500066681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6824","nonce":1500066681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6820","nonce":1500066681}]},"public_addr":"172.21.15.6:6812/1500066681","cluster_addr":"172.21.15.6:6816/1500066681","heartbeat_back_addr":"172.21.15.6:6824/1500066681","heartbeat_front_addr":"172.21.15.6:6820/1500066681","state":["exists","up"]},{"osd":3,"uuid":"21bb2efe-9ba7-4bdc-bc31-180cd1102e21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6801","nonce":1558806123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6803","nonce":1558806123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6807","nonce":1558806123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6805","nonce":1558806123}]},"public_addr":"172.21.15.6:6801/1558806123","cluster_addr":"172.21.15.6:6803/1558806123","heartbeat_back_addr":"172.21.15.6:6807/1558806123","heartbeat_front_addr":"172.21.15.6:6805/1558806123","state":["exists","up"]},{"osd":4,"uuid":"6e2c4b71-19d9-4737-83d9-558705d8644a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6801","nonce":1526135780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6803","nonce":1526135780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6807","nonce":1526135780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6805","nonce":1526135780}]},"public_addr":"172.21.15.146:6801/1526135780","cluster_addr":"172.21.15.146:6803/1526135780","heartbeat_back_addr":"172.21.15.146:6807/1526135780","heartbeat_front_addr":"172.21.15.146:6805/1526135780","state":["exists","up"]},{"osd":5,"uuid":"fb69c525-f1be-4b7d-ab32-6c29b0d6fddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6809","nonce":460262507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6811","nonce":460262507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6815","nonce":460262507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6813","nonce":460262507}]},"public_addr":"172.21.15.146:6809/460262507","cluster_addr":"172.21.15.146:6811/460262507","heartbeat_back_addr":"172.21.15.146:6815/460262507","heartbeat_front_addr":"172.21.15.146:6813/460262507","state":["exists","up"]},{"osd":6,"uuid":"c32e7fed-370d-4401-8ce0-6a4335b0d168","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6823","nonce":1392897493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6827","nonce":1392897493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6831","nonce":1392897493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6829","nonce":1392897493}]},"public_addr":"172.21.15.146:6823/1392897493","cluster_addr":"172.21.15.146:6827/1392897493","heartbeat_back_addr":"172.21.15.146:6831/1392897493","heartbeat_front_addr":"172.21.15.146:6829/1392897493","state":["exists","up"]},{"osd":7,"uuid":"75a79b44-fa5f-426d-a7d3-71f21f78835c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6817","nonce":3268323156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6819","nonce":3268323156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6825","nonce":3268323156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6822","nonce":3268323156}]},"public_addr":"172.21.15.146:6817/3268323156","cluster_addr":"172.21.15.146:6819/3268323156","heartbeat_back_addr":"172.21.15.146:6825/3268323156","heartbeat_front_addr":"172.21.15.146:6822/3268323156","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.310437+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.295850+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.294674+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.526855+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.300023+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.595160+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.346046+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.558741+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T09:26:56.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.362+0000 7fd140200640 1 -- 172.21.15.6:0/2953788538 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd11806b500 msgr2=0x7fd11806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:56.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.362+0000 7fd140200640 1 --2- 172.21.15.6:0/2953788538 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd11806b500 0x7fd11806d9c0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fd12c036000 tx=0x7fd12c030000 comp rx=0 tx=0).stop 2024-03-04T09:26:56.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.362+0000 7fd140200640 1 -- 172.21.15.6:0/2953788538 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd13806e100 msgr2=0x7fd13806e580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:56.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.362+0000 7fd140200640 1 --2- 172.21.15.6:0/2953788538 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd13806e100 0x7fd13806e580 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fd12400c700 tx=0x7fd12400cbc0 comp rx=0 tx=0).stop 2024-03-04T09:26:56.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.362+0000 7fd140200640 1 -- 172.21.15.6:0/2953788538 shutdown_connections 2024-03-04T09:26:56.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.362+0000 7fd140200640 1 --2- 172.21.15.6:0/2953788538 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd11806b500 0x7fd11806d9c0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:56.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.362+0000 7fd140200640 1 --2- 172.21.15.6:0/2953788538 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd138159890 0x7fd138159d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:56.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.362+0000 7fd140200640 1 --2- 172.21.15.6:0/2953788538 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd13806e100 0x7fd13806e580 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:56.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.362+0000 7fd140200640 1 --2- 172.21.15.6:0/2953788538 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd120003a70 0x7fd138150870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:56.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.362+0000 7fd140200640 1 -- 172.21.15.6:0/2953788538 >> 172.21.15.6:0/2953788538 conn(0x7fd13806afb0 msgr2=0x7fd1380ccff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:56.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.362+0000 7fd140200640 1 -- 172.21.15.6:0/2953788538 shutdown_connections 2024-03-04T09:26:56.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.363+0000 7fd140200640 1 -- 172.21.15.6:0/2953788538 wait complete. 2024-03-04T09:26:56.378 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T09:25:07.935198+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-04T09:26:28.556219+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '32', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2024-03-04T09:26:29.471227+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '33', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-04T09:26:56.379 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-04T09:26:56.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.600+0000 7f4634e00640 1 Processor -- start 2024-03-04T09:26:56.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.600+0000 7f4634e00640 1 -- start start 2024-03-04T09:26:56.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.602+0000 7f4634e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f463015bf20 0x7f46300acd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:56.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.602+0000 7f4634e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f4630072520 0x7f46300cdfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:56.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.602+0000 7f4634e00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4630072df0 con 0x7f4630071e20 2024-03-04T09:26:56.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.602+0000 7f4634e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46300cdab0 con 0x7f4630072520 2024-03-04T09:26:56.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.602+0000 7f4634e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46300b4d70 con 0x7f463015bf20 2024-03-04T09:26:56.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.602+0000 7f462d400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f4630072520 0x7f46300cdfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:56.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.602+0000 7f462d400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f4630072520 0x7f46300cdfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:40546/0 (socket says 172.21.15.6:40546) 2024-03-04T09:26:56.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.602+0000 7f462d400640 1 -- 172.21.15.6:0/232980727 learned_addr learned my addr 172.21.15.6:0/232980727 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:56.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.603+0000 7f462de00640 1 --2- 172.21.15.6:0/232980727 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f463015bf20 0x7f46300acd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:56.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.603+0000 7f462d400640 1 -- 172.21.15.6:0/232980727 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f463015bf20 msgr2=0x7f46300acd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:56.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.603+0000 7f462d400640 1 --2- 172.21.15.6:0/232980727 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f463015bf20 0x7f46300acd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:56.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.603+0000 7f462d400640 1 -- 172.21.15.6:0/232980727 >> v1:172.21.15.146:6789/0 conn(0x7f4630071e20 legacy=0x7f46300ccad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:56.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.603+0000 7f462d400640 1 -- 172.21.15.6:0/232980727 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46300b5050 con 0x7f4630072520 2024-03-04T09:26:56.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.604+0000 7f462d400640 1 --2- 172.21.15.6:0/232980727 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4630072520 0x7f46300cdfb0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f461c009870 tx=0x7f461c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2447205ee258051 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:56.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.604+0000 7f462ca00640 1 -- 172.21.15.6:0/232980727 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f461c035020 con 0x7f4630072520 2024-03-04T09:26:56.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.605+0000 7f462ca00640 1 -- 172.21.15.6:0/232980727 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f461c031640 con 0x7f4630072520 2024-03-04T09:26:56.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.605+0000 7f462ca00640 1 -- 172.21.15.6:0/232980727 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f461c02d040 con 0x7f4630072520 2024-03-04T09:26:56.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.605+0000 7f4634e00640 1 -- 172.21.15.6:0/232980727 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4630072520 msgr2=0x7f46300cdfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:56.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.605+0000 7f4634e00640 1 --2- 172.21.15.6:0/232980727 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4630072520 0x7f46300cdfb0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f461c009870 tx=0x7f461c02b8f0 comp rx=0 tx=0).stop 2024-03-04T09:26:56.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.605+0000 7f4634e00640 1 -- 172.21.15.6:0/232980727 shutdown_connections 2024-03-04T09:26:56.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.605+0000 7f4634e00640 1 --2- 172.21.15.6:0/232980727 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4630072520 0x7f46300cdfb0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:56.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.605+0000 7f4634e00640 1 --2- 172.21.15.6:0/232980727 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f463015bf20 0x7f46300acd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:56.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.605+0000 7f4634e00640 1 -- 172.21.15.6:0/232980727 >> 172.21.15.6:0/232980727 conn(0x7f463006b3d0 msgr2=0x7f463006b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:56.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.605+0000 7f4634e00640 1 -- 172.21.15.6:0/232980727 shutdown_connections 2024-03-04T09:26:56.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.605+0000 7f4634e00640 1 -- 172.21.15.6:0/232980727 wait complete. 2024-03-04T09:26:56.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.607+0000 7f4634e00640 1 Processor -- start 2024-03-04T09:26:56.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.607+0000 7f4634e00640 1 -- start start 2024-03-04T09:26:56.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.608+0000 7f4634e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4630071e20 0x7f46300afb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:56.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.608+0000 7f4634e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4630072520 0x7f46300ca9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:56.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.608+0000 7f4634e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f463015bf20 0x7f46301504e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:56.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.608+0000 7f4634e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46300b4b10 con 0x7f4630071e20 2024-03-04T09:26:56.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.608+0000 7f4634e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46300b4990 con 0x7f4630072520 2024-03-04T09:26:56.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.608+0000 7f4634e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46300cd1e0 con 0x7f463015bf20 2024-03-04T09:26:56.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.609+0000 7f462de00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4630071e20 0x7f46300afb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:56.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.609+0000 7f462de00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4630071e20 0x7f46300afb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:40562/0 (socket says 172.21.15.6:40562) 2024-03-04T09:26:56.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.609+0000 7f462de00640 1 -- 172.21.15.6:0/3198934982 learned_addr learned my addr 172.21.15.6:0/3198934982 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:56.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.609+0000 7f462d400640 1 --2- 172.21.15.6:0/3198934982 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4630072520 0x7f46300ca9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:56.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.609+0000 7f462e800640 1 --2- 172.21.15.6:0/3198934982 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f463015bf20 0x7f46301504e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:56.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.609+0000 7f462de00640 1 -- 172.21.15.6:0/3198934982 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f463015bf20 msgr2=0x7f46301504e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:56.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.609+0000 7f462de00640 1 --2- 172.21.15.6:0/3198934982 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f463015bf20 0x7f46301504e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:56.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.609+0000 7f462de00640 1 -- 172.21.15.6:0/3198934982 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4630072520 msgr2=0x7f46300ca9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:56.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.609+0000 7f462de00640 1 --2- 172.21.15.6:0/3198934982 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4630072520 0x7f46300ca9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:56.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.609+0000 7f462de00640 1 -- 172.21.15.6:0/3198934982 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46300b00a0 con 0x7f4630071e20 2024-03-04T09:26:56.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.609+0000 7f462de00640 1 --2- 172.21.15.6:0/3198934982 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4630071e20 0x7f46300afb60 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f462000d620 tx=0x7f462000dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:56.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.610+0000 7f4626000640 1 -- 172.21.15.6:0/3198934982 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f462000b620 con 0x7f4630071e20 2024-03-04T09:26:56.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.610+0000 7f4626000640 1 -- 172.21.15.6:0/3198934982 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4620019070 con 0x7f4630071e20 2024-03-04T09:26:56.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.610+0000 7f4626000640 1 -- 172.21.15.6:0/3198934982 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f462000ba00 con 0x7f4630071e20 2024-03-04T09:26:56.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.610+0000 7f4634e00640 1 -- 172.21.15.6:0/3198934982 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46300c9550 con 0x7f4630071e20 2024-03-04T09:26:56.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.610+0000 7f4634e00640 1 -- 172.21.15.6:0/3198934982 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46300c9b10 con 0x7f4630071e20 2024-03-04T09:26:56.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.611+0000 7f4634e00640 1 -- 172.21.15.6:0/3198934982 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f463004f330 con 0x7f4630071e20 2024-03-04T09:26:56.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.612+0000 7f4626000640 1 -- 172.21.15.6:0/3198934982 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f462000bba0 con 0x7f4630071e20 2024-03-04T09:26:56.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.613+0000 7f4626000640 1 --2- 172.21.15.6:0/3198934982 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f461006b4d0 0x7f461006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:56.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.613+0000 7f4626000640 1 -- 172.21.15.6:0/3198934982 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f462008a640 con 0x7f4630071e20 2024-03-04T09:26:56.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.614+0000 7f462d400640 1 --2- 172.21.15.6:0/3198934982 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f461006b4d0 0x7f461006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:56.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.615+0000 7f462d400640 1 --2- 172.21.15.6:0/3198934982 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f461006b4d0 0x7f461006d990 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f461c036000 tx=0x7f461c03c040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:56.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.617+0000 7f4626000640 1 -- 172.21.15.6:0/3198934982 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f46300b00a0 con 0x7f4630071e20 2024-03-04T09:26:56.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.814+0000 7f4634e00640 1 -- 172.21.15.6:0/3198934982 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f46300ccad0 con 0x7f4630071e20 2024-03-04T09:26:56.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.815+0000 7f4626000640 1 -- 172.21.15.6:0/3198934982 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v33) v1 ==== 93+0+10 (secure 0 0 0) 0x7f46200594c0 con 0x7f4630071e20 2024-03-04T09:26:56.812 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-04T09:26:56.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.818+0000 7f4634e00640 1 -- 172.21.15.6:0/3198934982 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f461006b4d0 msgr2=0x7f461006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:56.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.818+0000 7f4634e00640 1 --2- 172.21.15.6:0/3198934982 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f461006b4d0 0x7f461006d990 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f461c036000 tx=0x7f461c03c040 comp rx=0 tx=0).stop 2024-03-04T09:26:56.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.818+0000 7f4634e00640 1 -- 172.21.15.6:0/3198934982 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4630071e20 msgr2=0x7f46300afb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:56.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.818+0000 7f4634e00640 1 --2- 172.21.15.6:0/3198934982 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4630071e20 0x7f46300afb60 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f462000d620 tx=0x7f462000dae0 comp rx=0 tx=0).stop 2024-03-04T09:26:56.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.818+0000 7f4634e00640 1 -- 172.21.15.6:0/3198934982 shutdown_connections 2024-03-04T09:26:56.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.818+0000 7f4634e00640 1 --2- 172.21.15.6:0/3198934982 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f461006b4d0 0x7f461006d990 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:56.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.818+0000 7f4634e00640 1 --2- 172.21.15.6:0/3198934982 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f463015bf20 0x7f46301504e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:56.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.818+0000 7f4634e00640 1 --2- 172.21.15.6:0/3198934982 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4630072520 0x7f46300ca9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:56.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.818+0000 7f4634e00640 1 --2- 172.21.15.6:0/3198934982 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4630071e20 0x7f46300afb60 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:56.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.818+0000 7f4634e00640 1 -- 172.21.15.6:0/3198934982 >> 172.21.15.6:0/3198934982 conn(0x7f463006b3d0 msgr2=0x7f46300ae800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:56.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.818+0000 7f4634e00640 1 -- 172.21.15.6:0/3198934982 shutdown_connections 2024-03-04T09:26:56.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:56.818+0000 7f4634e00640 1 -- 172.21.15.6:0/3198934982 wait complete. 2024-03-04T09:26:56.833 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-04T09:26:57.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.058+0000 7f7e43000640 1 Processor -- start 2024-03-04T09:26:57.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.059+0000 7f7e43000640 1 -- start start 2024-03-04T09:26:57.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.060+0000 7f7e43000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f7e3c0ce550 0x7f7e3c15f230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:57.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.060+0000 7f7e43000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f7e3c15d940 0x7f7e3c15ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:57.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.060+0000 7f7e43000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e3c0d0780 con 0x7f7e3c0cffa0 2024-03-04T09:26:57.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.060+0000 7f7e43000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e3c0d0a60 con 0x7f7e3c15d940 2024-03-04T09:26:57.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.060+0000 7f7e43000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e3c0d0d40 con 0x7f7e3c0ce550 2024-03-04T09:26:57.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.061+0000 7f7e3b400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f7e3c0cffa0 0x7f7e3c0ce930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:34022/0 (socket says 172.21.15.6:34022) 2024-03-04T09:26:57.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.061+0000 7f7e3b400640 1 -- 172.21.15.6:0/1246261442 learned_addr learned my addr 172.21.15.6:0/1246261442 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:26:57.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.061+0000 7f7e41000640 1 --2- 172.21.15.6:0/1246261442 >> v2:172.21.15.6:3300/0 conn(0x7f7e3c15d940 0x7f7e3c15ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:57.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.061+0000 7f7e3be00640 1 --2- 172.21.15.6:0/1246261442 >> v2:172.21.15.146:3300/0 conn(0x7f7e3c0ce550 0x7f7e3c15f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:57.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.061+0000 7f7e3aa00640 1 -- 172.21.15.6:0/1246261442 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2461003676 0 0) 0x7f7e3c0d0780 con 0x7f7e3c0cffa0 2024-03-04T09:26:57.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.061+0000 7f7e3aa00640 1 -- 172.21.15.6:0/1246261442 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e20003520 con 0x7f7e3c0cffa0 2024-03-04T09:26:57.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.061+0000 7f7e41000640 1 -- 172.21.15.6:0/1246261442 >> v2:172.21.15.146:3300/0 conn(0x7f7e3c0ce550 msgr2=0x7f7e3c15f230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:57.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.061+0000 7f7e41000640 1 --2- 172.21.15.6:0/1246261442 >> v2:172.21.15.146:3300/0 conn(0x7f7e3c0ce550 0x7f7e3c15f230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:57.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.061+0000 7f7e41000640 1 -- 172.21.15.6:0/1246261442 >> v1:172.21.15.6:6789/0 conn(0x7f7e3c0cffa0 legacy=0x7f7e3c0ce930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:57.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.061+0000 7f7e41000640 1 -- 172.21.15.6:0/1246261442 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e3c0d1020 con 0x7f7e3c15d940 2024-03-04T09:26:57.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.063+0000 7f7e41000640 1 --2- 172.21.15.6:0/1246261442 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7e3c15d940 0x7f7e3c15ddd0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f7e3c06e3c0 tx=0x7f7e3402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=203be5fb78008fea server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:57.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.063+0000 7f7e3aa00640 1 -- 172.21.15.6:0/1246261442 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e34035020 con 0x7f7e3c15d940 2024-03-04T09:26:57.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.063+0000 7f7e3aa00640 1 -- 172.21.15.6:0/1246261442 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e34031830 con 0x7f7e3c15d940 2024-03-04T09:26:57.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.063+0000 7f7e3aa00640 1 -- 172.21.15.6:0/1246261442 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e34031be0 con 0x7f7e3c15d940 2024-03-04T09:26:57.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.063+0000 7f7e43000640 1 -- 172.21.15.6:0/1246261442 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7e3c15d940 msgr2=0x7f7e3c15ddd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:57.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.063+0000 7f7e43000640 1 --2- 172.21.15.6:0/1246261442 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7e3c15d940 0x7f7e3c15ddd0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f7e3c06e3c0 tx=0x7f7e3402b8d0 comp rx=0 tx=0).stop 2024-03-04T09:26:57.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.064+0000 7f7e43000640 1 -- 172.21.15.6:0/1246261442 shutdown_connections 2024-03-04T09:26:57.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.064+0000 7f7e43000640 1 --2- 172.21.15.6:0/1246261442 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7e3c15d940 0x7f7e3c15ddd0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:57.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.064+0000 7f7e43000640 1 --2- 172.21.15.6:0/1246261442 >> v2:172.21.15.146:3300/0 conn(0x7f7e3c0ce550 0x7f7e3c15f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:57.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.064+0000 7f7e43000640 1 -- 172.21.15.6:0/1246261442 >> 172.21.15.6:0/1246261442 conn(0x7f7e3c0b3820 msgr2=0x7f7e3c0b3c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:57.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.064+0000 7f7e43000640 1 -- 172.21.15.6:0/1246261442 shutdown_connections 2024-03-04T09:26:57.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.064+0000 7f7e43000640 1 -- 172.21.15.6:0/1246261442 wait complete. 2024-03-04T09:26:57.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.065+0000 7f7e43000640 1 Processor -- start 2024-03-04T09:26:57.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.065+0000 7f7e43000640 1 -- start start 2024-03-04T09:26:57.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.066+0000 7f7e43000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7e3c0ce550 0x7f7e3c154b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:57.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.066+0000 7f7e43000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7e3c0cffa0 0x7f7e3c159980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:57.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.066+0000 7f7e43000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7e3c15d940 0x7f7e3c06ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:57.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.066+0000 7f7e43000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e3c0cdfb0 con 0x7f7e3c15d940 2024-03-04T09:26:57.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.066+0000 7f7e43000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e3c0d0400 con 0x7f7e3c0ce550 2024-03-04T09:26:57.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.066+0000 7f7e43000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e3c0ce130 con 0x7f7e3c0cffa0 2024-03-04T09:26:57.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.067+0000 7f7e3be00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7e3c0ce550 0x7f7e3c154b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:57.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.067+0000 7f7e3b400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7e3c0cffa0 0x7f7e3c159980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:57.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.067+0000 7f7e41000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7e3c15d940 0x7f7e3c06ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:57.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.067+0000 7f7e3be00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7e3c0ce550 0x7f7e3c154b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.6:44682/0 (socket says 172.21.15.6:44682) 2024-03-04T09:26:57.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.067+0000 7f7e3be00640 1 -- 172.21.15.6:0/236219284 learned_addr learned my addr 172.21.15.6:0/236219284 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:57.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.067+0000 7f7e41000640 1 -- 172.21.15.6:0/236219284 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7e3c0cffa0 msgr2=0x7f7e3c159980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:57.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.067+0000 7f7e41000640 1 --2- 172.21.15.6:0/236219284 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7e3c0cffa0 0x7f7e3c159980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:57.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.068+0000 7f7e41000640 1 -- 172.21.15.6:0/236219284 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7e3c0ce550 msgr2=0x7f7e3c154b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:57.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.068+0000 7f7e41000640 1 --2- 172.21.15.6:0/236219284 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7e3c0ce550 0x7f7e3c154b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:57.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.068+0000 7f7e41000640 1 -- 172.21.15.6:0/236219284 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e3c1f4200 con 0x7f7e3c15d940 2024-03-04T09:26:57.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.068+0000 7f7e41000640 1 --2- 172.21.15.6:0/236219284 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7e3c15d940 0x7f7e3c06ef60 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f7e34033000 tx=0x7f7e340046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:57.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.068+0000 7f7e33e00640 1 -- 172.21.15.6:0/236219284 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e34035040 con 0x7f7e3c15d940 2024-03-04T09:26:57.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.068+0000 7f7e33e00640 1 -- 172.21.15.6:0/236219284 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e3403b840 con 0x7f7e3c15d940 2024-03-04T09:26:57.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.068+0000 7f7e33e00640 1 -- 172.21.15.6:0/236219284 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e3403bbf0 con 0x7f7e3c15d940 2024-03-04T09:26:57.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.068+0000 7f7e43000640 1 -- 172.21.15.6:0/236219284 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e3c0cab50 con 0x7f7e3c15d940 2024-03-04T09:26:57.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.068+0000 7f7e43000640 1 -- 172.21.15.6:0/236219284 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e3c159ec0 con 0x7f7e3c15d940 2024-03-04T09:26:57.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.070+0000 7f7e43000640 1 -- 172.21.15.6:0/236219284 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e3c06ed80 con 0x7f7e3c15d940 2024-03-04T09:26:57.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.071+0000 7f7e33e00640 1 -- 172.21.15.6:0/236219284 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7e34038020 con 0x7f7e3c15d940 2024-03-04T09:26:57.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.072+0000 7f7e33e00640 1 --2- 172.21.15.6:0/236219284 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f7e1806b4d0 0x7f7e1806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:57.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.072+0000 7f7e33e00640 1 -- 172.21.15.6:0/236219284 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7e340ac160 con 0x7f7e3c15d940 2024-03-04T09:26:57.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.073+0000 7f7e3be00640 1 --2- 172.21.15.6:0/236219284 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f7e1806b4d0 0x7f7e1806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:57.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.074+0000 7f7e3be00640 1 --2- 172.21.15.6:0/236219284 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f7e1806b4d0 0x7f7e1806d990 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f7e2c0094c0 tx=0x7f7e2c00b040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:57.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.076+0000 7f7e33e00640 1 -- 172.21.15.6:0/236219284 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7e3c06ed80 con 0x7f7e3c15d940 2024-03-04T09:26:57.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.269+0000 7f7e43000640 1 -- 172.21.15.6:0/236219284 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f7e3c1f4010 con 0x7f7e3c15d940 2024-03-04T09:26:57.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.270+0000 7f7e33e00640 1 -- 172.21.15.6:0/236219284 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v33) v1 ==== 104+0+11 (secure 0 0 0) 0x7f7e3407afe0 con 0x7f7e3c15d940 2024-03-04T09:26:57.267 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-03-04T09:26:57.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.272+0000 7f7e43000640 1 -- 172.21.15.6:0/236219284 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f7e1806b4d0 msgr2=0x7f7e1806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:57.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.272+0000 7f7e43000640 1 --2- 172.21.15.6:0/236219284 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f7e1806b4d0 0x7f7e1806d990 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f7e2c0094c0 tx=0x7f7e2c00b040 comp rx=0 tx=0).stop 2024-03-04T09:26:57.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.272+0000 7f7e43000640 1 -- 172.21.15.6:0/236219284 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7e3c15d940 msgr2=0x7f7e3c06ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:57.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.272+0000 7f7e43000640 1 --2- 172.21.15.6:0/236219284 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7e3c15d940 0x7f7e3c06ef60 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f7e34033000 tx=0x7f7e340046c0 comp rx=0 tx=0).stop 2024-03-04T09:26:57.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.272+0000 7f7e43000640 1 -- 172.21.15.6:0/236219284 shutdown_connections 2024-03-04T09:26:57.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.272+0000 7f7e43000640 1 --2- 172.21.15.6:0/236219284 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7e3c15d940 0x7f7e3c06ef60 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:57.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.272+0000 7f7e43000640 1 --2- 172.21.15.6:0/236219284 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f7e1806b4d0 0x7f7e1806d990 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:57.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.272+0000 7f7e43000640 1 --2- 172.21.15.6:0/236219284 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7e3c0cffa0 0x7f7e3c159980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:57.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.272+0000 7f7e43000640 1 --2- 172.21.15.6:0/236219284 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7e3c0ce550 0x7f7e3c154b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:57.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.272+0000 7f7e43000640 1 -- 172.21.15.6:0/236219284 >> 172.21.15.6:0/236219284 conn(0x7f7e3c0b3820 msgr2=0x7f7e3c070380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:57.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.272+0000 7f7e43000640 1 -- 172.21.15.6:0/236219284 shutdown_connections 2024-03-04T09:26:57.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.272+0000 7f7e43000640 1 -- 172.21.15.6:0/236219284 wait complete. 2024-03-04T09:26:57.287 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-04T09:26:57.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.497+0000 7fbb0cc00640 1 Processor -- start 2024-03-04T09:26:57.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.497+0000 7fbb0cc00640 1 -- start start 2024-03-04T09:26:57.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.499+0000 7fbb0cc00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fbb0815dee0 0x7fbb080d18a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:57.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.499+0000 7fbb0cc00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fbb080d0d40 0x7fbb080b3510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:57.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.499+0000 7fbb0cc00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb080ceca0 con 0x7fbb080d4860 2024-03-04T09:26:57.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.499+0000 7fbb0cc00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb080cf5d0 con 0x7fbb0815dee0 2024-03-04T09:26:57.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.499+0000 7fbb0cc00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb080cef80 con 0x7fbb080d0d40 2024-03-04T09:26:57.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.499+0000 7fbb07400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fbb0815dee0 0x7fbb080d18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:57.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.499+0000 7fbb07400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fbb0815dee0 0x7fbb080d18a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:40582/0 (socket says 172.21.15.6:40582) 2024-03-04T09:26:57.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.499+0000 7fbb07400640 1 -- 172.21.15.6:0/924443728 learned_addr learned my addr 172.21.15.6:0/924443728 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:57.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.500+0000 7fbb07400640 1 -- 172.21.15.6:0/924443728 >> v2:172.21.15.146:3300/0 conn(0x7fbb080d0d40 msgr2=0x7fbb080b3510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:57.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.500+0000 7fbb07e00640 1 --2- 172.21.15.6:0/924443728 >> v2:172.21.15.146:3300/0 conn(0x7fbb080d0d40 0x7fbb080b3510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:57.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.500+0000 7fbb07400640 1 --2- 172.21.15.6:0/924443728 >> v2:172.21.15.146:3300/0 conn(0x7fbb080d0d40 0x7fbb080b3510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:57.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.500+0000 7fbb07400640 1 -- 172.21.15.6:0/924443728 >> v1:172.21.15.146:6789/0 conn(0x7fbb080d4860 legacy=0x7fbb080d0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:57.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.500+0000 7fbb07400640 1 -- 172.21.15.6:0/924443728 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb080cf260 con 0x7fbb0815dee0 2024-03-04T09:26:57.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.500+0000 7fbb07e00640 1 --2- 172.21.15.6:0/924443728 >> v2:172.21.15.146:3300/0 conn(0x7fbb080d0d40 0x7fbb080b3510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:26:57.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.501+0000 7fbb07400640 1 --2- 172.21.15.6:0/924443728 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbb0815dee0 0x7fbb080d18a0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fbb0806b2d0 tx=0x7fbaf802b930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f06f6783822fff46 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:57.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.501+0000 7fbb06a00640 1 -- 172.21.15.6:0/924443728 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaf8035020 con 0x7fbb0815dee0 2024-03-04T09:26:57.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.501+0000 7fbb06a00640 1 -- 172.21.15.6:0/924443728 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbaf8031890 con 0x7fbb0815dee0 2024-03-04T09:26:57.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.501+0000 7fbb06a00640 1 -- 172.21.15.6:0/924443728 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaf8031c60 con 0x7fbb0815dee0 2024-03-04T09:26:57.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.502+0000 7fbb0cc00640 1 -- 172.21.15.6:0/924443728 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbb0815dee0 msgr2=0x7fbb080d18a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:57.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.502+0000 7fbb0cc00640 1 --2- 172.21.15.6:0/924443728 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbb0815dee0 0x7fbb080d18a0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fbb0806b2d0 tx=0x7fbaf802b930 comp rx=0 tx=0).stop 2024-03-04T09:26:57.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.502+0000 7fbb0cc00640 1 -- 172.21.15.6:0/924443728 shutdown_connections 2024-03-04T09:26:57.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.502+0000 7fbb0cc00640 1 --2- 172.21.15.6:0/924443728 >> v2:172.21.15.146:3300/0 conn(0x7fbb080d0d40 0x7fbb080b3510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:57.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.502+0000 7fbb0cc00640 1 --2- 172.21.15.6:0/924443728 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbb0815dee0 0x7fbb080d18a0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:57.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.502+0000 7fbb0cc00640 1 -- 172.21.15.6:0/924443728 >> 172.21.15.6:0/924443728 conn(0x7fbb080d3cd0 msgr2=0x7fbb080d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:57.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.502+0000 7fbb0cc00640 1 -- 172.21.15.6:0/924443728 shutdown_connections 2024-03-04T09:26:57.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.502+0000 7fbb0cc00640 1 -- 172.21.15.6:0/924443728 wait complete. 2024-03-04T09:26:57.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.503+0000 7fbb0cc00640 1 Processor -- start 2024-03-04T09:26:57.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.504+0000 7fbb0cc00640 1 -- start start 2024-03-04T09:26:57.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.504+0000 7fbb0cc00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbb080d0d40 0x7fbb081f4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:57.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.505+0000 7fbb0cc00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbb080d4860 0x7fbb081f6dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:57.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.505+0000 7fbb0cc00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbb0815dee0 0x7fbb080cc0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:57.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.505+0000 7fbb0cc00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb080d1de0 con 0x7fbb080d0d40 2024-03-04T09:26:57.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.505+0000 7fbb0cc00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb0815e2c0 con 0x7fbb080d4860 2024-03-04T09:26:57.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.505+0000 7fbb0cc00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb080d1f60 con 0x7fbb0815dee0 2024-03-04T09:26:57.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.505+0000 7fbb07400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbb080d0d40 0x7fbb081f4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:57.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.505+0000 7fbb07400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbb080d0d40 0x7fbb081f4060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:40596/0 (socket says 172.21.15.6:40596) 2024-03-04T09:26:57.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.505+0000 7fbb07400640 1 -- 172.21.15.6:0/3462898624 learned_addr learned my addr 172.21.15.6:0/3462898624 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:57.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.505+0000 7fbb07400640 1 -- 172.21.15.6:0/3462898624 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbb0815dee0 msgr2=0x7fbb080cc0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:57.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.505+0000 7fbb07400640 1 --2- 172.21.15.6:0/3462898624 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbb0815dee0 0x7fbb080cc0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:57.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.505+0000 7fbb07400640 1 -- 172.21.15.6:0/3462898624 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbb080d4860 msgr2=0x7fbb081f6dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:57.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.505+0000 7fbb07400640 1 --2- 172.21.15.6:0/3462898624 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbb080d4860 0x7fbb081f6dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:57.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.505+0000 7fbb07400640 1 -- 172.21.15.6:0/3462898624 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb081e7a50 con 0x7fbb080d0d40 2024-03-04T09:26:57.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.506+0000 7fbb07400640 1 --2- 172.21.15.6:0/3462898624 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbb080d0d40 0x7fbb081f4060 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fbaf8000c00 tx=0x7fbaf80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:57.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.506+0000 7fbaff400640 1 -- 172.21.15.6:0/3462898624 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaf8040070 con 0x7fbb080d0d40 2024-03-04T09:26:57.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.506+0000 7fbaff400640 1 -- 172.21.15.6:0/3462898624 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbaf803a420 con 0x7fbb080d0d40 2024-03-04T09:26:57.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.506+0000 7fbaff400640 1 -- 172.21.15.6:0/3462898624 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaf8035040 con 0x7fbb080d0d40 2024-03-04T09:26:57.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.506+0000 7fbb0cc00640 1 -- 172.21.15.6:0/3462898624 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb081e7d30 con 0x7fbb080d0d40 2024-03-04T09:26:57.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.506+0000 7fbb0cc00640 1 -- 172.21.15.6:0/3462898624 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb081e82f0 con 0x7fbb080d0d40 2024-03-04T09:26:57.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.507+0000 7fbb0cc00640 1 -- 172.21.15.6:0/3462898624 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb0804f330 con 0x7fbb080d0d40 2024-03-04T09:26:57.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.508+0000 7fbaff400640 1 -- 172.21.15.6:0/3462898624 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbaf804b060 con 0x7fbb080d0d40 2024-03-04T09:26:57.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.509+0000 7fbaff400640 1 --2- 172.21.15.6:0/3462898624 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fbae406b4d0 0x7fbae406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:57.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.509+0000 7fbaff400640 1 -- 172.21.15.6:0/3462898624 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbaf80ac270 con 0x7fbb080d0d40 2024-03-04T09:26:57.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.509+0000 7fbaffe00640 1 --2- 172.21.15.6:0/3462898624 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fbae406b4d0 0x7fbae406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:57.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.510+0000 7fbaffe00640 1 --2- 172.21.15.6:0/3462898624 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fbae406b4d0 0x7fbae406d990 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fbaf0003cc0 tx=0x7fbaf00079b0 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:57.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.514+0000 7fbaff400640 1 -- 172.21.15.6:0/3462898624 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbaf802d040 con 0x7fbb080d0d40 2024-03-04T09:26:57.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.711+0000 7fbb0cc00640 1 -- 172.21.15.6:0/3462898624 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fbb0808d7d0 con 0x7fbb080d0d40 2024-03-04T09:26:57.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.712+0000 7fbaff400640 1 -- 172.21.15.6:0/3462898624 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v33) v1 ==== 100+0+11 (secure 0 0 0) 0x7fbaf807b0e0 con 0x7fbb080d0d40 2024-03-04T09:26:57.709 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 64 2024-03-04T09:26:57.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.715+0000 7fbb0cc00640 1 -- 172.21.15.6:0/3462898624 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fbae406b4d0 msgr2=0x7fbae406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:57.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.715+0000 7fbb0cc00640 1 --2- 172.21.15.6:0/3462898624 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fbae406b4d0 0x7fbae406d990 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fbaf0003cc0 tx=0x7fbaf00079b0 comp rx=0 tx=0).stop 2024-03-04T09:26:57.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.715+0000 7fbb0cc00640 1 -- 172.21.15.6:0/3462898624 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbb080d0d40 msgr2=0x7fbb081f4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:57.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.715+0000 7fbb0cc00640 1 --2- 172.21.15.6:0/3462898624 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbb080d0d40 0x7fbb081f4060 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fbaf8000c00 tx=0x7fbaf80046c0 comp rx=0 tx=0).stop 2024-03-04T09:26:57.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.715+0000 7fbb0cc00640 1 -- 172.21.15.6:0/3462898624 shutdown_connections 2024-03-04T09:26:57.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.715+0000 7fbb0cc00640 1 --2- 172.21.15.6:0/3462898624 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fbae406b4d0 0x7fbae406d990 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:57.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.715+0000 7fbb0cc00640 1 --2- 172.21.15.6:0/3462898624 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbb0815dee0 0x7fbb080cc0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:57.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.715+0000 7fbb0cc00640 1 --2- 172.21.15.6:0/3462898624 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbb080d4860 0x7fbb081f6dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:57.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.715+0000 7fbb0cc00640 1 --2- 172.21.15.6:0/3462898624 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbb080d0d40 0x7fbb081f4060 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:57.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.715+0000 7fbb0cc00640 1 -- 172.21.15.6:0/3462898624 >> 172.21.15.6:0/3462898624 conn(0x7fbb080d3cd0 msgr2=0x7fbb080b7c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:57.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.715+0000 7fbb0cc00640 1 -- 172.21.15.6:0/3462898624 shutdown_connections 2024-03-04T09:26:57.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.715+0000 7fbb0cc00640 1 -- 172.21.15.6:0/3462898624 wait complete. 2024-03-04T09:26:57.730 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-04T09:26:57.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.956+0000 7f38cf800640 1 Processor -- start 2024-03-04T09:26:57.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.956+0000 7f38cf800640 1 -- start start 2024-03-04T09:26:57.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.958+0000 7f38cf800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f38c8145790 0x7f38c80d1630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:57.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.958+0000 7f38cf800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f38c806afb0 0x7f38c806b440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:57.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.958+0000 7f38cf800640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38c80b5de0 con 0x7f38c80add50 2024-03-04T09:26:57.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.958+0000 7f38cf800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38c80d1b70 con 0x7f38c806afb0 2024-03-04T09:26:57.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.958+0000 7f38cf800640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38c809ef10 con 0x7f38c8145790 2024-03-04T09:26:57.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.959+0000 7f38c7e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f38c80add50 0x7f38c81603f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:34038/0 (socket says 172.21.15.6:34038) 2024-03-04T09:26:57.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.959+0000 7f38c7e00640 1 -- 172.21.15.6:0/3002317067 learned_addr learned my addr 172.21.15.6:0/3002317067 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:26:57.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.959+0000 7f38cd800640 1 --2- 172.21.15.6:0/3002317067 >> v2:172.21.15.6:3300/0 conn(0x7f38c806afb0 0x7f38c806b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:57.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.959+0000 7f38cce00640 1 --2- 172.21.15.6:0/3002317067 >> v2:172.21.15.146:3300/0 conn(0x7f38c8145790 0x7f38c80d1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:57.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.959+0000 7f38c7400640 1 -- 172.21.15.6:0/3002317067 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 680622964 0 0) 0x7f38c80b5de0 con 0x7f38c80add50 2024-03-04T09:26:57.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.959+0000 7f38c7400640 1 -- 172.21.15.6:0/3002317067 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38b0003520 con 0x7f38c80add50 2024-03-04T09:26:57.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.959+0000 7f38cd800640 1 -- 172.21.15.6:0/3002317067 >> v2:172.21.15.146:3300/0 conn(0x7f38c8145790 msgr2=0x7f38c80d1630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:57.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.959+0000 7f38cd800640 1 --2- 172.21.15.6:0/3002317067 >> v2:172.21.15.146:3300/0 conn(0x7f38c8145790 0x7f38c80d1630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:57.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.959+0000 7f38cd800640 1 -- 172.21.15.6:0/3002317067 >> v1:172.21.15.6:6789/0 conn(0x7f38c80add50 legacy=0x7f38c81603f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:57.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.959+0000 7f38cd800640 1 -- 172.21.15.6:0/3002317067 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38c80b6080 con 0x7f38c806afb0 2024-03-04T09:26:57.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.961+0000 7f38cd800640 1 --2- 172.21.15.6:0/3002317067 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38c806afb0 0x7f38c806b440 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f38c8071f50 tx=0x7f38c002bec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffdd5f7fbddff397 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:57.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.961+0000 7f38c7400640 1 -- 172.21.15.6:0/3002317067 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38c0039070 con 0x7f38c806afb0 2024-03-04T09:26:57.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.961+0000 7f38c7400640 1 -- 172.21.15.6:0/3002317067 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38c002cc70 con 0x7f38c806afb0 2024-03-04T09:26:57.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.961+0000 7f38c7400640 1 -- 172.21.15.6:0/3002317067 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38c0034640 con 0x7f38c806afb0 2024-03-04T09:26:57.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.961+0000 7f38cf800640 1 -- 172.21.15.6:0/3002317067 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38c806afb0 msgr2=0x7f38c806b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:57.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.961+0000 7f38cf800640 1 --2- 172.21.15.6:0/3002317067 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38c806afb0 0x7f38c806b440 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f38c8071f50 tx=0x7f38c002bec0 comp rx=0 tx=0).stop 2024-03-04T09:26:57.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.962+0000 7f38cf800640 1 -- 172.21.15.6:0/3002317067 shutdown_connections 2024-03-04T09:26:57.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.962+0000 7f38cf800640 1 --2- 172.21.15.6:0/3002317067 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38c806afb0 0x7f38c806b440 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:57.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.962+0000 7f38cf800640 1 --2- 172.21.15.6:0/3002317067 >> v2:172.21.15.146:3300/0 conn(0x7f38c8145790 0x7f38c80d1630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:57.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.962+0000 7f38cf800640 1 -- 172.21.15.6:0/3002317067 >> 172.21.15.6:0/3002317067 conn(0x7f38c806c2a0 msgr2=0x7f38c806c6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:57.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.962+0000 7f38cf800640 1 -- 172.21.15.6:0/3002317067 shutdown_connections 2024-03-04T09:26:57.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.962+0000 7f38cf800640 1 -- 172.21.15.6:0/3002317067 wait complete. 2024-03-04T09:26:57.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.963+0000 7f38cf800640 1 Processor -- start 2024-03-04T09:26:57.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.964+0000 7f38cf800640 1 -- start start 2024-03-04T09:26:57.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.964+0000 7f38cf800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f38c806afb0 0x7f38c813d250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:57.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.965+0000 7f38cf800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38c80add50 0x7f38c80b3290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:57.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.965+0000 7f38cf800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f38c8145790 0x7f38c815ace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:57.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.965+0000 7f38cf800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38c8145b70 con 0x7f38c80add50 2024-03-04T09:26:57.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.965+0000 7f38cf800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38c815fed0 con 0x7f38c806afb0 2024-03-04T09:26:57.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.965+0000 7f38cf800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38c80af720 con 0x7f38c8145790 2024-03-04T09:26:57.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.965+0000 7f38c7e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38c80add50 0x7f38c80b3290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:57.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.965+0000 7f38c7e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38c80add50 0x7f38c80b3290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:40624/0 (socket says 172.21.15.6:40624) 2024-03-04T09:26:57.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.965+0000 7f38c7e00640 1 -- 172.21.15.6:0/924542440 learned_addr learned my addr 172.21.15.6:0/924542440 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:57.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.965+0000 7f38c7e00640 1 -- 172.21.15.6:0/924542440 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f38c8145790 msgr2=0x7f38c815ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:57.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.965+0000 7f38c7e00640 1 --2- 172.21.15.6:0/924542440 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f38c8145790 0x7f38c815ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:57.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.965+0000 7f38c7e00640 1 -- 172.21.15.6:0/924542440 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f38c806afb0 msgr2=0x7f38c813d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:57.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.965+0000 7f38cce00640 1 --2- 172.21.15.6:0/924542440 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f38c806afb0 0x7f38c813d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:57.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.965+0000 7f38c7e00640 1 --2- 172.21.15.6:0/924542440 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f38c806afb0 0x7f38c813d250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:57.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.965+0000 7f38c7e00640 1 -- 172.21.15.6:0/924542440 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38c80b0a70 con 0x7f38c80add50 2024-03-04T09:26:57.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.966+0000 7f38c7e00640 1 --2- 172.21.15.6:0/924542440 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38c80add50 0x7f38c80b3290 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f38b400c400 tx=0x7f38b400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:57.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.966+0000 7f38c4c00640 1 -- 172.21.15.6:0/924542440 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38b4017070 con 0x7f38c80add50 2024-03-04T09:26:57.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.966+0000 7f38c4c00640 1 -- 172.21.15.6:0/924542440 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38b4005d00 con 0x7f38c80add50 2024-03-04T09:26:57.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.966+0000 7f38c4c00640 1 -- 172.21.15.6:0/924542440 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38b4007310 con 0x7f38c80add50 2024-03-04T09:26:57.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.966+0000 7f38cf800640 1 -- 172.21.15.6:0/924542440 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38c80b21f0 con 0x7f38c80add50 2024-03-04T09:26:57.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.966+0000 7f38cf800640 1 -- 172.21.15.6:0/924542440 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38c80b27b0 con 0x7f38c80add50 2024-03-04T09:26:57.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.967+0000 7f38cf800640 1 -- 172.21.15.6:0/924542440 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38c804f3a0 con 0x7f38c80add50 2024-03-04T09:26:57.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.968+0000 7f38c4c00640 1 -- 172.21.15.6:0/924542440 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f38b401a070 con 0x7f38c80add50 2024-03-04T09:26:57.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.969+0000 7f38c4c00640 1 --2- 172.21.15.6:0/924542440 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f38a806b4d0 0x7f38a806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:57.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.969+0000 7f38c4c00640 1 -- 172.21.15.6:0/924542440 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f38b40896a0 con 0x7f38c80add50 2024-03-04T09:26:57.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.970+0000 7f38cce00640 1 --2- 172.21.15.6:0/924542440 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f38a806b4d0 0x7f38a806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:57.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.971+0000 7f38cce00640 1 --2- 172.21.15.6:0/924542440 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f38a806b4d0 0x7f38a806d990 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f38c815ba80 tx=0x7f38bc008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:57.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:57.974+0000 7f38c4c00640 1 -- 172.21.15.6:0/924542440 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f38b4058510 con 0x7f38c80add50 2024-03-04T09:26:58.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:58.187+0000 7f38cf800640 1 -- 172.21.15.6:0/924542440 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f38c8068b00 con 0x7f38c80add50 2024-03-04T09:26:58.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:58.188+0000 7f38c4c00640 1 -- 172.21.15.6:0/924542440 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v33) v1 ==== 80+0+0 (secure 0 0 0) 0x7f38b4057e10 con 0x7f38c80add50 2024-03-04T09:26:58.184 INFO:teuthology.orchestra.run.smithi006.stderr:listed 0 entries 2024-03-04T09:26:58.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:58.190+0000 7f38cf800640 1 -- 172.21.15.6:0/924542440 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f38a806b4d0 msgr2=0x7f38a806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:58.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:58.190+0000 7f38cf800640 1 --2- 172.21.15.6:0/924542440 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f38a806b4d0 0x7f38a806d990 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f38c815ba80 tx=0x7f38bc008040 comp rx=0 tx=0).stop 2024-03-04T09:26:58.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:58.190+0000 7f38cf800640 1 -- 172.21.15.6:0/924542440 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38c80add50 msgr2=0x7f38c80b3290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:58.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:58.190+0000 7f38cf800640 1 --2- 172.21.15.6:0/924542440 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38c80add50 0x7f38c80b3290 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f38b400c400 tx=0x7f38b400c8c0 comp rx=0 tx=0).stop 2024-03-04T09:26:58.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:58.191+0000 7f38cf800640 1 -- 172.21.15.6:0/924542440 shutdown_connections 2024-03-04T09:26:58.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:58.191+0000 7f38cf800640 1 --2- 172.21.15.6:0/924542440 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f38a806b4d0 0x7f38a806d990 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:58.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:58.191+0000 7f38cf800640 1 --2- 172.21.15.6:0/924542440 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f38c8145790 0x7f38c815ace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:58.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:58.191+0000 7f38cf800640 1 --2- 172.21.15.6:0/924542440 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38c80add50 0x7f38c80b3290 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:58.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:58.191+0000 7f38cf800640 1 --2- 172.21.15.6:0/924542440 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f38c806afb0 0x7f38c813d250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:58.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:58.191+0000 7f38cf800640 1 -- 172.21.15.6:0/924542440 >> 172.21.15.6:0/924542440 conn(0x7f38c806c2a0 msgr2=0x7f38c80afae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:58.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:58.191+0000 7f38cf800640 1 -- 172.21.15.6:0/924542440 shutdown_connections 2024-03-04T09:26:58.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:58.191+0000 7f38cf800640 1 -- 172.21.15.6:0/924542440 wait complete. 2024-03-04T09:26:58.208 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-04T09:26:58.208 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:26:58.208 DEBUG:teuthology.orchestra.run.smithi006:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-04T09:26:58.345 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi006.front.sepia.ceph.com 2024-03-04T09:26:58.345 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:26:58.346 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-04T09:26:58.386 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-04T09:26:58.386 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T09:26:58.424 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053429,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249180k,nr_inodes=812295,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-04T09:26:58.424 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-04T09:26:58.424 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T09:26:58.467 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054202,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249800k,nr_inodes=812450,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-04T09:26:58.468 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-04T09:26:58.468 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T09:26:58.511 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053429,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249180k,nr_inodes=812295,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-04T09:26:58.511 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:26:58.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:58.741+0000 7f15efe00640 1 Processor -- start 2024-03-04T09:26:58.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:58.741+0000 7f15efe00640 1 -- start start 2024-03-04T09:26:58.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:58.742+0000 7f15efe00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f15f015e0b0 0x7f15f00cf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:58.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:58.742+0000 7f15efe00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f15f00d4a10 0x7f15f00d0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:58.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:58.742+0000 7f15efe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f15f00d0c80 0x7f15f00d1900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:58.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:58.742+0000 7f15efe00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15f00b09e0 con 0x7f15f00d0c80 2024-03-04T09:26:58.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:58.742+0000 7f15efe00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15f00b3710 con 0x7f15f015e0b0 2024-03-04T09:26:58.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:58.742+0000 7f15efe00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15f00b39f0 con 0x7f15f00d4a10 2024-03-04T09:26:58.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:58.743+0000 7f15ef400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f15f00d0c80 0x7f15f00d1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:58.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:58.743+0000 7f15ef400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f15f00d0c80 0x7f15f00d1900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:40630/0 (socket says 172.21.15.6:40630) 2024-03-04T09:26:58.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:58.743+0000 7f15ef400640 1 -- 172.21.15.6:0/1149307883 learned_addr learned my addr 172.21.15.6:0/1149307883 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:58.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:58.743+0000 7f15ee000640 1 --2- 172.21.15.6:0/1149307883 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f15f00d4a10 0x7f15f00d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:58.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:58.743+0000 7f15eea00640 1 --2- 172.21.15.6:0/1149307883 >> v2:172.21.15.146:3300/0 conn(0x7f15f015e0b0 0x7f15f00cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:58.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:58.743+0000 7f15ef400640 1 -- 172.21.15.6:0/1149307883 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f15f00d4a10 msgr2=0x7f15f00d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:58.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:58.743+0000 7f15ef400640 1 --2- 172.21.15.6:0/1149307883 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f15f00d4a10 0x7f15f00d0740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:58.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:58.743+0000 7f15ef400640 1 -- 172.21.15.6:0/1149307883 >> v2:172.21.15.146:3300/0 conn(0x7f15f015e0b0 msgr2=0x7f15f00cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:58.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:58.743+0000 7f15ef400640 1 --2- 172.21.15.6:0/1149307883 >> v2:172.21.15.146:3300/0 conn(0x7f15f015e0b0 0x7f15f00cf310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:58.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:58.743+0000 7f15ef400640 1 -- 172.21.15.6:0/1149307883 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15f00b3cd0 con 0x7f15f00d0c80 2024-03-04T09:26:58.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:58.744+0000 7f15ef400640 1 --2- 172.21.15.6:0/1149307883 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f15f00d0c80 0x7f15f00d1900 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f15f00b68c0 tx=0x7f15e802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8fb1150b272e379 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:58.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:58.745+0000 7f15ed600640 1 -- 172.21.15.6:0/1149307883 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15e802ca70 con 0x7f15f00d0c80 2024-03-04T09:26:58.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:58.745+0000 7f15ed600640 1 -- 172.21.15.6:0/1149307883 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15e802cc10 con 0x7f15f00d0c80 2024-03-04T09:26:58.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:58.745+0000 7f15ed600640 1 -- 172.21.15.6:0/1149307883 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15e80343f0 con 0x7f15f00d0c80 2024-03-04T09:26:58.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:58.745+0000 7f15efe00640 1 -- 172.21.15.6:0/1149307883 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f15f00d0c80 msgr2=0x7f15f00d1900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:58.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:58.745+0000 7f15efe00640 1 --2- 172.21.15.6:0/1149307883 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f15f00d0c80 0x7f15f00d1900 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f15f00b68c0 tx=0x7f15e802bed0 comp rx=0 tx=0).stop 2024-03-04T09:26:58.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:58.745+0000 7f15efe00640 1 -- 172.21.15.6:0/1149307883 shutdown_connections 2024-03-04T09:26:58.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:58.745+0000 7f15efe00640 1 --2- 172.21.15.6:0/1149307883 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f15f00d0c80 0x7f15f00d1900 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:58.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:58.745+0000 7f15efe00640 1 --2- 172.21.15.6:0/1149307883 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f15f00d4a10 0x7f15f00d0740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:58.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:58.745+0000 7f15efe00640 1 --2- 172.21.15.6:0/1149307883 >> v2:172.21.15.146:3300/0 conn(0x7f15f015e0b0 0x7f15f00cf310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:58.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:58.745+0000 7f15efe00640 1 -- 172.21.15.6:0/1149307883 >> 172.21.15.6:0/1149307883 conn(0x7f15f00d3d30 msgr2=0x7f15f00d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:58.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:58.745+0000 7f15efe00640 1 -- 172.21.15.6:0/1149307883 shutdown_connections 2024-03-04T09:26:58.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:58.745+0000 7f15efe00640 1 -- 172.21.15.6:0/1149307883 wait complete. 2024-03-04T09:26:58.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:58.747+0000 7f15efe00640 1 Processor -- start 2024-03-04T09:26:58.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:58.747+0000 7f15efe00640 1 -- start start 2024-03-04T09:26:58.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:58.748+0000 7f15efe00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f15f00d0c80 0x7f15f01f3180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:58.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:58.748+0000 7f15efe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f15f00d4a10 0x7f15f00cae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:58.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:58.748+0000 7f15efe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f15f015e0b0 0x7f15f00cc9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:58.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:58.748+0000 7f15efe00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15f00cfc60 con 0x7f15f00d4a10 2024-03-04T09:26:58.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:58.748+0000 7f15efe00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15f015e490 con 0x7f15f015e0b0 2024-03-04T09:26:58.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:58.748+0000 7f15efe00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15f00cf850 con 0x7f15f00d0c80 2024-03-04T09:26:58.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:58.748+0000 7f15ee000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f15f00d4a10 0x7f15f00cae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:58.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:58.748+0000 7f15ee000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f15f00d4a10 0x7f15f00cae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:40632/0 (socket says 172.21.15.6:40632) 2024-03-04T09:26:58.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:58.748+0000 7f15ee000640 1 -- 172.21.15.6:0/3129914350 learned_addr learned my addr 172.21.15.6:0/3129914350 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:58.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:58.749+0000 7f15ef400640 1 --2- 172.21.15.6:0/3129914350 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f15f015e0b0 0x7f15f00cc9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:58.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:58.749+0000 7f15eea00640 1 --2- 172.21.15.6:0/3129914350 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f15f00d0c80 0x7f15f01f3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:58.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:58.749+0000 7f15ee000640 1 -- 172.21.15.6:0/3129914350 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f15f00d0c80 msgr2=0x7f15f01f3180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:58.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:58.749+0000 7f15ee000640 1 --2- 172.21.15.6:0/3129914350 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f15f00d0c80 0x7f15f01f3180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:58.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:58.749+0000 7f15ee000640 1 -- 172.21.15.6:0/3129914350 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f15f015e0b0 msgr2=0x7f15f00cc9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:58.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:58.749+0000 7f15ee000640 1 --2- 172.21.15.6:0/3129914350 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f15f015e0b0 0x7f15f00cc9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:58.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:58.749+0000 7f15ee000640 1 -- 172.21.15.6:0/3129914350 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15f01f36c0 con 0x7f15f00d4a10 2024-03-04T09:26:58.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:58.749+0000 7f15ee000640 1 --2- 172.21.15.6:0/3129914350 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f15f00d4a10 0x7f15f00cae00 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f15dc00d800 tx=0x7f15dc00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:58.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:58.749+0000 7f15e6a00640 1 -- 172.21.15.6:0/3129914350 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15dc00b6c0 con 0x7f15f00d4a10 2024-03-04T09:26:58.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:58.749+0000 7f15e6a00640 1 -- 172.21.15.6:0/3129914350 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15dc019070 con 0x7f15f00d4a10 2024-03-04T09:26:58.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:58.749+0000 7f15efe00640 1 -- 172.21.15.6:0/3129914350 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15f00cb700 con 0x7f15f00d4a10 2024-03-04T09:26:58.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:58.750+0000 7f15e6a00640 1 -- 172.21.15.6:0/3129914350 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15dc00ba80 con 0x7f15f00d4a10 2024-03-04T09:26:58.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:58.750+0000 7f15efe00640 1 -- 172.21.15.6:0/3129914350 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15f00cbcc0 con 0x7f15f00d4a10 2024-03-04T09:26:58.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:58.751+0000 7f15efe00640 1 -- 172.21.15.6:0/3129914350 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15f00cabb0 con 0x7f15f00d4a10 2024-03-04T09:26:58.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:58.752+0000 7f15e6a00640 1 -- 172.21.15.6:0/3129914350 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f15dc00bc20 con 0x7f15f00d4a10 2024-03-04T09:26:58.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:58.753+0000 7f15e6a00640 1 --2- 172.21.15.6:0/3129914350 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f15d006b4d0 0x7f15d006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:58.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:58.753+0000 7f15e6a00640 1 -- 172.21.15.6:0/3129914350 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f15dc08a700 con 0x7f15f00d4a10 2024-03-04T09:26:58.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:58.753+0000 7f15eea00640 1 --2- 172.21.15.6:0/3129914350 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f15d006b4d0 0x7f15d006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:58.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:58.754+0000 7f15eea00640 1 --2- 172.21.15.6:0/3129914350 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f15d006b4d0 0x7f15d006d990 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f15e0006fd0 tx=0x7f15e0008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:58.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:58.757+0000 7f15e6a00640 1 -- 172.21.15.6:0/3129914350 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f15f00cabb0 con 0x7f15f00d4a10 2024-03-04T09:26:58.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:58.995+0000 7f15efe00640 1 -- 172.21.15.6:0/3129914350 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f15f015e610 con 0x7f15f00d4a10 2024-03-04T09:26:58.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:58.996+0000 7f15e6a00640 1 -- 172.21.15.6:0/3129914350 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+4977 (secure 0 0 0) 0x7f15dc01c360 con 0x7f15f00d4a10 2024-03-04T09:26:58.993 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:26:58.993 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4685,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/1703428647","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":1703428647},{"type":"v1","addr":"172.21.15.146:6837","nonce":1703428647}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4837,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/2617429530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2617429530},{"type":"v1","addr":"172.21.15.6:6833","nonce":2617429530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":5043,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/2744913857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":2744913857},{"type":"v1","addr":"172.21.15.6:6835","nonce":2744913857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:26:30.445199+0000","modified":"2024-03-04T09:26:33.534893+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5046},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5046":{"gid":5046,"name":"d","rank":0,"incarnation":12,"state":"up:active","state_seq":4,"addr":"172.21.15.146:6836/2285803221","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":2285803221},{"type":"v1","addr":"172.21.15.146:6836","nonce":2285803221}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5046,"qdb_cluster":[5046]},"id":2}]} 2024-03-04T09:26:58.993 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 16 2024-03-04T09:26:58.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:58.998+0000 7f15efe00640 1 -- 172.21.15.6:0/3129914350 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f15d006b4d0 msgr2=0x7f15d006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:58.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:58.998+0000 7f15efe00640 1 --2- 172.21.15.6:0/3129914350 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f15d006b4d0 0x7f15d006d990 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f15e0006fd0 tx=0x7f15e0008040 comp rx=0 tx=0).stop 2024-03-04T09:26:58.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:58.998+0000 7f15efe00640 1 -- 172.21.15.6:0/3129914350 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f15f00d4a10 msgr2=0x7f15f00cae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:58.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:58.998+0000 7f15efe00640 1 --2- 172.21.15.6:0/3129914350 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f15f00d4a10 0x7f15f00cae00 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f15dc00d800 tx=0x7f15dc00dcc0 comp rx=0 tx=0).stop 2024-03-04T09:26:58.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:58.998+0000 7f15efe00640 1 -- 172.21.15.6:0/3129914350 shutdown_connections 2024-03-04T09:26:58.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:58.998+0000 7f15efe00640 1 --2- 172.21.15.6:0/3129914350 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f15f015e0b0 0x7f15f00cc9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:58.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:58.998+0000 7f15efe00640 1 --2- 172.21.15.6:0/3129914350 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f15f00d4a10 0x7f15f00cae00 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:58.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:58.998+0000 7f15efe00640 1 --2- 172.21.15.6:0/3129914350 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f15d006b4d0 0x7f15d006d990 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:58.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:58.998+0000 7f15efe00640 1 --2- 172.21.15.6:0/3129914350 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f15f00d0c80 0x7f15f01f3180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:58.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:58.998+0000 7f15efe00640 1 -- 172.21.15.6:0/3129914350 >> 172.21.15.6:0/3129914350 conn(0x7f15f00d3d30 msgr2=0x7f15f00b0b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:58.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:58.998+0000 7f15efe00640 1 -- 172.21.15.6:0/3129914350 shutdown_connections 2024-03-04T09:26:58.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:58.998+0000 7f15efe00640 1 -- 172.21.15.6:0/3129914350 wait complete. 2024-03-04T09:26:59.014 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T09:26:59.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.234+0000 7fda58200640 1 Processor -- start 2024-03-04T09:26:59.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.234+0000 7fda58200640 1 -- start start 2024-03-04T09:26:59.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.236+0000 7fda58200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fda500b3c90 0x7fda500cdc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:59.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.236+0000 7fda58200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fda500cf5c0 0x7fda500b4070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:59.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.236+0000 7fda58200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda500b45b0 con 0x7fda5006cb10 2024-03-04T09:26:59.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.236+0000 7fda58200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda500ce160 con 0x7fda500b3c90 2024-03-04T09:26:59.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.236+0000 7fda58200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda500b4860 con 0x7fda500cf5c0 2024-03-04T09:26:59.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.236+0000 7fda55800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fda500b3c90 0x7fda500cdc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:59.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.237+0000 7fda54e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fda500cf5c0 0x7fda500b4070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:59.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.237+0000 7fda56200640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fda5006cb10 0x7fda5006cfa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.6:33208/0 (socket says 172.21.15.6:33208) 2024-03-04T09:26:59.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.237+0000 7fda56200640 1 -- 172.21.15.6:0/3108301932 learned_addr learned my addr 172.21.15.6:0/3108301932 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:26:59.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.237+0000 7fda55800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fda500b3c90 0x7fda500cdc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.6:55044/0 (socket says 172.21.15.6:55044) 2024-03-04T09:26:59.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.237+0000 7fda4fe00640 1 -- 172.21.15.6:0/3108301932 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 440082686 0 0) 0x7fda500b45b0 con 0x7fda5006cb10 2024-03-04T09:26:59.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.237+0000 7fda54e00640 1 -- 172.21.15.6:0/3108301932 >> v2:172.21.15.146:3300/0 conn(0x7fda500b3c90 msgr2=0x7fda500cdc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:59.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.237+0000 7fda54e00640 1 --2- 172.21.15.6:0/3108301932 >> v2:172.21.15.146:3300/0 conn(0x7fda500b3c90 0x7fda500cdc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:59.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.237+0000 7fda54e00640 1 -- 172.21.15.6:0/3108301932 >> v1:172.21.15.146:6789/0 conn(0x7fda5006cb10 legacy=0x7fda5006cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:59.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.237+0000 7fda54e00640 1 -- 172.21.15.6:0/3108301932 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda500b4b40 con 0x7fda500cf5c0 2024-03-04T09:26:59.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.239+0000 7fda54e00640 1 --2- 172.21.15.6:0/3108301932 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fda500cf5c0 0x7fda500b4070 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fda3c0097a0 tx=0x7fda3c02b940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4ba4c04b2c074ade server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:59.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.239+0000 7fda4fe00640 1 -- 172.21.15.6:0/3108301932 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda3c034070 con 0x7fda500cf5c0 2024-03-04T09:26:59.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.239+0000 7fda4fe00640 1 -- 172.21.15.6:0/3108301932 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda3c030800 con 0x7fda500cf5c0 2024-03-04T09:26:59.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.239+0000 7fda4fe00640 1 -- 172.21.15.6:0/3108301932 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda3c030bd0 con 0x7fda500cf5c0 2024-03-04T09:26:59.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.240+0000 7fda58200640 1 -- 172.21.15.6:0/3108301932 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fda500cf5c0 msgr2=0x7fda500b4070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:59.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.240+0000 7fda58200640 1 --2- 172.21.15.6:0/3108301932 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fda500cf5c0 0x7fda500b4070 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fda3c0097a0 tx=0x7fda3c02b940 comp rx=0 tx=0).stop 2024-03-04T09:26:59.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.240+0000 7fda58200640 1 -- 172.21.15.6:0/3108301932 shutdown_connections 2024-03-04T09:26:59.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.240+0000 7fda58200640 1 --2- 172.21.15.6:0/3108301932 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fda500cf5c0 0x7fda500b4070 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:59.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.240+0000 7fda58200640 1 --2- 172.21.15.6:0/3108301932 >> v2:172.21.15.146:3300/0 conn(0x7fda500b3c90 0x7fda500cdc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:59.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.240+0000 7fda58200640 1 -- 172.21.15.6:0/3108301932 >> 172.21.15.6:0/3108301932 conn(0x7fda5006afb0 msgr2=0x7fda5006b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:59.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.240+0000 7fda58200640 1 -- 172.21.15.6:0/3108301932 shutdown_connections 2024-03-04T09:26:59.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.240+0000 7fda58200640 1 -- 172.21.15.6:0/3108301932 wait complete. 2024-03-04T09:26:59.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.242+0000 7fda58200640 1 Processor -- start 2024-03-04T09:26:59.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.242+0000 7fda58200640 1 -- start start 2024-03-04T09:26:59.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.243+0000 7fda58200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fda5006cb10 0x7fda50156240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:59.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.243+0000 7fda58200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fda500b3c90 0x7fda50155ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:59.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.243+0000 7fda58200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fda500cf5c0 0x7fda50154330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:59.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.243+0000 7fda58200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda5015b340 con 0x7fda500cf5c0 2024-03-04T09:26:59.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.243+0000 7fda58200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda500b5100 con 0x7fda500b3c90 2024-03-04T09:26:59.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.243+0000 7fda58200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda5006d6b0 con 0x7fda5006cb10 2024-03-04T09:26:59.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.243+0000 7fda56200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fda500cf5c0 0x7fda50154330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:59.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.243+0000 7fda56200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fda500cf5c0 0x7fda50154330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49550/0 (socket says 172.21.15.6:49550) 2024-03-04T09:26:59.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.243+0000 7fda56200640 1 -- 172.21.15.6:0/638003688 learned_addr learned my addr 172.21.15.6:0/638003688 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:59.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.244+0000 7fda55800640 1 --2- 172.21.15.6:0/638003688 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fda5006cb10 0x7fda50156240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:59.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.244+0000 7fda54e00640 1 --2- 172.21.15.6:0/638003688 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fda500b3c90 0x7fda50155ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:59.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.244+0000 7fda56200640 1 -- 172.21.15.6:0/638003688 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fda5006cb10 msgr2=0x7fda50156240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:59.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.244+0000 7fda56200640 1 --2- 172.21.15.6:0/638003688 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fda5006cb10 0x7fda50156240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:59.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.244+0000 7fda56200640 1 -- 172.21.15.6:0/638003688 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fda500b3c90 msgr2=0x7fda50155ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:59.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.244+0000 7fda56200640 1 --2- 172.21.15.6:0/638003688 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fda500b3c90 0x7fda50155ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:59.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.244+0000 7fda56200640 1 -- 172.21.15.6:0/638003688 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda50154a10 con 0x7fda500cf5c0 2024-03-04T09:26:59.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.244+0000 7fda56200640 1 --2- 172.21.15.6:0/638003688 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fda500cf5c0 0x7fda50154330 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fda4800c7a0 tx=0x7fda4800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:59.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.244+0000 7fda4d600640 1 -- 172.21.15.6:0/638003688 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda48018070 con 0x7fda500cf5c0 2024-03-04T09:26:59.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.245+0000 7fda4d600640 1 -- 172.21.15.6:0/638003688 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda4800a640 con 0x7fda500cf5c0 2024-03-04T09:26:59.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.245+0000 7fda4d600640 1 -- 172.21.15.6:0/638003688 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda4800aab0 con 0x7fda500cf5c0 2024-03-04T09:26:59.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.245+0000 7fda58200640 1 -- 172.21.15.6:0/638003688 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda50154cf0 con 0x7fda500cf5c0 2024-03-04T09:26:59.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.245+0000 7fda58200640 1 -- 172.21.15.6:0/638003688 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda501552b0 con 0x7fda500cf5c0 2024-03-04T09:26:59.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.246+0000 7fda58200640 1 -- 172.21.15.6:0/638003688 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda5006bf00 con 0x7fda500cf5c0 2024-03-04T09:26:59.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.247+0000 7fda4d600640 1 -- 172.21.15.6:0/638003688 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fda4800ac50 con 0x7fda500cf5c0 2024-03-04T09:26:59.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.248+0000 7fda4d600640 1 --2- 172.21.15.6:0/638003688 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fda3006b4d0 0x7fda3006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:59.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.248+0000 7fda4d600640 1 -- 172.21.15.6:0/638003688 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fda48089890 con 0x7fda500cf5c0 2024-03-04T09:26:59.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.248+0000 7fda55800640 1 --2- 172.21.15.6:0/638003688 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fda3006b4d0 0x7fda3006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:59.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.250+0000 7fda55800640 1 --2- 172.21.15.6:0/638003688 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fda3006b4d0 0x7fda3006d990 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fda44009ab0 tx=0x7fda44008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:59.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.252+0000 7fda4d600640 1 -- 172.21.15.6:0/638003688 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fda48058710 con 0x7fda500cf5c0 2024-03-04T09:26:59.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.457+0000 7fda58200640 1 -- 172.21.15.6:0/638003688 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fda500ad270 con 0x7fda500cf5c0 2024-03-04T09:26:59.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.459+0000 7fda4d600640 1 -- 172.21.15.6:0/638003688 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v33) v1 ==== 74+0+16621 (secure 0 0 0) 0x7fda48058010 con 0x7fda500cf5c0 2024-03-04T09:26:59.457 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:26:59.457 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":33,"fsid":"edce0578-5ae4-4402-b45a-552bdef84d7c","created":"2024-03-04T09:24:54.523780+0000","modified":"2024-03-04T09:26:31.991047+0000","last_up_change":"2024-03-04T09:25:07.089049+0000","last_in_change":"2024-03-04T09:24:57.982077+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T09:25:07.935198+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-04T09:26:28.556219+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-04T09:26:29.471227+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a96574f-81f1-4bc6-8463-757520b84a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6825","nonce":1364449401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6827","nonce":1364449401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6831","nonce":1364449401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6829","nonce":1364449401}]},"public_addr":"172.21.15.6:6825/1364449401","cluster_addr":"172.21.15.6:6827/1364449401","heartbeat_back_addr":"172.21.15.6:6831/1364449401","heartbeat_front_addr":"172.21.15.6:6829/1364449401","state":["exists","up"]},{"osd":1,"uuid":"7d79b9c1-6b87-476a-bc36-7db5ee64cd90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6809","nonce":2494103078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6811","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6813","nonce":2494103078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6821","nonce":2494103078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6817","nonce":2494103078}]},"public_addr":"172.21.15.6:6809/2494103078","cluster_addr":"172.21.15.6:6813/2494103078","heartbeat_back_addr":"172.21.15.6:6821/2494103078","heartbeat_front_addr":"172.21.15.6:6817/2494103078","state":["exists","up"]},{"osd":2,"uuid":"492331b9-4ee9-40b6-bacd-85d44cd6c45a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6812","nonce":1500066681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6816","nonce":1500066681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6824","nonce":1500066681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6820","nonce":1500066681}]},"public_addr":"172.21.15.6:6812/1500066681","cluster_addr":"172.21.15.6:6816/1500066681","heartbeat_back_addr":"172.21.15.6:6824/1500066681","heartbeat_front_addr":"172.21.15.6:6820/1500066681","state":["exists","up"]},{"osd":3,"uuid":"21bb2efe-9ba7-4bdc-bc31-180cd1102e21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6801","nonce":1558806123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6803","nonce":1558806123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6807","nonce":1558806123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6805","nonce":1558806123}]},"public_addr":"172.21.15.6:6801/1558806123","cluster_addr":"172.21.15.6:6803/1558806123","heartbeat_back_addr":"172.21.15.6:6807/1558806123","heartbeat_front_addr":"172.21.15.6:6805/1558806123","state":["exists","up"]},{"osd":4,"uuid":"6e2c4b71-19d9-4737-83d9-558705d8644a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6801","nonce":1526135780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6803","nonce":1526135780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6807","nonce":1526135780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6805","nonce":1526135780}]},"public_addr":"172.21.15.146:6801/1526135780","cluster_addr":"172.21.15.146:6803/1526135780","heartbeat_back_addr":"172.21.15.146:6807/1526135780","heartbeat_front_addr":"172.21.15.146:6805/1526135780","state":["exists","up"]},{"osd":5,"uuid":"fb69c525-f1be-4b7d-ab32-6c29b0d6fddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6809","nonce":460262507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6811","nonce":460262507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6815","nonce":460262507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6813","nonce":460262507}]},"public_addr":"172.21.15.146:6809/460262507","cluster_addr":"172.21.15.146:6811/460262507","heartbeat_back_addr":"172.21.15.146:6815/460262507","heartbeat_front_addr":"172.21.15.146:6813/460262507","state":["exists","up"]},{"osd":6,"uuid":"c32e7fed-370d-4401-8ce0-6a4335b0d168","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6823","nonce":1392897493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6827","nonce":1392897493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6831","nonce":1392897493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6829","nonce":1392897493}]},"public_addr":"172.21.15.146:6823/1392897493","cluster_addr":"172.21.15.146:6827/1392897493","heartbeat_back_addr":"172.21.15.146:6831/1392897493","heartbeat_front_addr":"172.21.15.146:6829/1392897493","state":["exists","up"]},{"osd":7,"uuid":"75a79b44-fa5f-426d-a7d3-71f21f78835c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6817","nonce":3268323156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6819","nonce":3268323156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6825","nonce":3268323156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6822","nonce":3268323156}]},"public_addr":"172.21.15.146:6817/3268323156","cluster_addr":"172.21.15.146:6819/3268323156","heartbeat_back_addr":"172.21.15.146:6825/3268323156","heartbeat_front_addr":"172.21.15.146:6822/3268323156","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.310437+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.295850+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.294674+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.526855+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.300023+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.595160+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.346046+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.558741+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T09:26:59.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.462+0000 7fda58200640 1 -- 172.21.15.6:0/638003688 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fda3006b4d0 msgr2=0x7fda3006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:59.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.462+0000 7fda58200640 1 --2- 172.21.15.6:0/638003688 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fda3006b4d0 0x7fda3006d990 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fda44009ab0 tx=0x7fda44008040 comp rx=0 tx=0).stop 2024-03-04T09:26:59.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.462+0000 7fda58200640 1 -- 172.21.15.6:0/638003688 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fda500cf5c0 msgr2=0x7fda50154330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:59.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.462+0000 7fda58200640 1 --2- 172.21.15.6:0/638003688 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fda500cf5c0 0x7fda50154330 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fda4800c7a0 tx=0x7fda4800cc60 comp rx=0 tx=0).stop 2024-03-04T09:26:59.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.462+0000 7fda58200640 1 -- 172.21.15.6:0/638003688 shutdown_connections 2024-03-04T09:26:59.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.462+0000 7fda58200640 1 --2- 172.21.15.6:0/638003688 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fda500cf5c0 0x7fda50154330 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:59.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.462+0000 7fda58200640 1 --2- 172.21.15.6:0/638003688 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fda500b3c90 0x7fda50155ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:59.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.462+0000 7fda58200640 1 --2- 172.21.15.6:0/638003688 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fda3006b4d0 0x7fda3006d990 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:59.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.462+0000 7fda58200640 1 --2- 172.21.15.6:0/638003688 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fda5006cb10 0x7fda50156240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:59.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.462+0000 7fda58200640 1 -- 172.21.15.6:0/638003688 >> 172.21.15.6:0/638003688 conn(0x7fda5006afb0 msgr2=0x7fda5006cfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:59.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.463+0000 7fda58200640 1 -- 172.21.15.6:0/638003688 shutdown_connections 2024-03-04T09:26:59.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.463+0000 7fda58200640 1 -- 172.21.15.6:0/638003688 wait complete. 2024-03-04T09:26:59.479 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T09:25:07.935198+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-04T09:26:28.556219+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '32', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2024-03-04T09:26:29.471227+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '33', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-04T09:26:59.480 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-04T09:26:59.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.704+0000 7f7a26400640 1 Processor -- start 2024-03-04T09:26:59.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.704+0000 7f7a26400640 1 -- start start 2024-03-04T09:26:59.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.706+0000 7f7a26400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f7a20145430 0x7f7a2015f810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:59.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.706+0000 7f7a26400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f7a200b2250 0x7f7a200b2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:59.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.706+0000 7f7a26400640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a2015fe40 con 0x7f7a2015cc90 2024-03-04T09:26:59.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.706+0000 7f7a26400640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a20160120 con 0x7f7a20145430 2024-03-04T09:26:59.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.706+0000 7f7a26400640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a20160400 con 0x7f7a200b2250 2024-03-04T09:26:59.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.706+0000 7f7a25000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f7a20145430 0x7f7a2015f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:59.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.706+0000 7f7a25000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f7a20145430 0x7f7a2015f810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49562/0 (socket says 172.21.15.6:49562) 2024-03-04T09:26:59.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.706+0000 7f7a25000640 1 -- 172.21.15.6:0/20244458 learned_addr learned my addr 172.21.15.6:0/20244458 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:59.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.706+0000 7f7a1fe00640 1 --1- 172.21.15.6:0/20244458 >> v1:172.21.15.6:6789/0 conn(0x7f7a2015cc90 0x7f7a200b1b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:38966/0 (socket says 172.21.15.6:38966) 2024-03-04T09:26:59.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.706+0000 7f7a25a00640 1 --2- 172.21.15.6:0/20244458 >> v2:172.21.15.146:3300/0 conn(0x7f7a200b2250 0x7f7a200b2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:59.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.707+0000 7f7a25000640 1 -- 172.21.15.6:0/20244458 >> v2:172.21.15.146:3300/0 conn(0x7f7a200b2250 msgr2=0x7f7a200b2fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:26:59.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.707+0000 7f7a25000640 1 --2- 172.21.15.6:0/20244458 >> v2:172.21.15.146:3300/0 conn(0x7f7a200b2250 0x7f7a200b2fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:59.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.707+0000 7f7a25000640 1 -- 172.21.15.6:0/20244458 >> v1:172.21.15.6:6789/0 conn(0x7f7a2015cc90 legacy=0x7f7a200b1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:59.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.707+0000 7f7a25000640 1 -- 172.21.15.6:0/20244458 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a201606e0 con 0x7f7a20145430 2024-03-04T09:26:59.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.708+0000 7f7a25000640 1 --2- 172.21.15.6:0/20244458 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7a20145430 0x7f7a2015f810 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f7a200b4440 tx=0x7f7a1402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30c12538ede4b6a server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:59.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.708+0000 7f7a1f400640 1 -- 172.21.15.6:0/20244458 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a14034070 con 0x7f7a20145430 2024-03-04T09:26:59.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.708+0000 7f7a1f400640 1 -- 172.21.15.6:0/20244458 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a14030860 con 0x7f7a20145430 2024-03-04T09:26:59.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.708+0000 7f7a1f400640 1 -- 172.21.15.6:0/20244458 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a14030c30 con 0x7f7a20145430 2024-03-04T09:26:59.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.709+0000 7f7a26400640 1 -- 172.21.15.6:0/20244458 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7a20145430 msgr2=0x7f7a2015f810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:59.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.709+0000 7f7a26400640 1 --2- 172.21.15.6:0/20244458 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7a20145430 0x7f7a2015f810 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f7a200b4440 tx=0x7f7a1402b9a0 comp rx=0 tx=0).stop 2024-03-04T09:26:59.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.709+0000 7f7a26400640 1 -- 172.21.15.6:0/20244458 shutdown_connections 2024-03-04T09:26:59.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.709+0000 7f7a26400640 1 --2- 172.21.15.6:0/20244458 >> v2:172.21.15.146:3300/0 conn(0x7f7a200b2250 0x7f7a200b2fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:59.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.709+0000 7f7a26400640 1 --2- 172.21.15.6:0/20244458 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7a20145430 0x7f7a2015f810 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:59.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.709+0000 7f7a26400640 1 -- 172.21.15.6:0/20244458 >> 172.21.15.6:0/20244458 conn(0x7f7a2006b3a0 msgr2=0x7f7a2006b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:59.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.709+0000 7f7a26400640 1 -- 172.21.15.6:0/20244458 shutdown_connections 2024-03-04T09:26:59.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.709+0000 7f7a26400640 1 -- 172.21.15.6:0/20244458 wait complete. 2024-03-04T09:26:59.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.711+0000 7f7a26400640 1 Processor -- start 2024-03-04T09:26:59.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.712+0000 7f7a26400640 1 -- start start 2024-03-04T09:26:59.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.713+0000 7f7a26400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7a200b2250 0x7f7a2013fe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:59.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.713+0000 7f7a26400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7a20145430 0x7f7a201f3bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:59.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.713+0000 7f7a26400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7a2015cc90 0x7f7a201f40f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:59.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.713+0000 7f7a26400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a20145990 con 0x7f7a20145430 2024-03-04T09:26:59.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.713+0000 7f7a26400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a20145810 con 0x7f7a2015cc90 2024-03-04T09:26:59.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.713+0000 7f7a26400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a200cde00 con 0x7f7a200b2250 2024-03-04T09:26:59.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.713+0000 7f7a1fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7a20145430 0x7f7a201f3bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:59.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.713+0000 7f7a1fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7a20145430 0x7f7a201f3bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49576/0 (socket says 172.21.15.6:49576) 2024-03-04T09:26:59.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.713+0000 7f7a1fe00640 1 -- 172.21.15.6:0/878977780 learned_addr learned my addr 172.21.15.6:0/878977780 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:26:59.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.713+0000 7f7a25000640 1 --2- 172.21.15.6:0/878977780 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7a200b2250 0x7f7a2013fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:59.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.714+0000 7f7a25a00640 1 --2- 172.21.15.6:0/878977780 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7a2015cc90 0x7f7a201f40f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:59.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.714+0000 7f7a1fe00640 1 -- 172.21.15.6:0/878977780 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7a200b2250 msgr2=0x7f7a2013fe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:59.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.714+0000 7f7a1fe00640 1 --2- 172.21.15.6:0/878977780 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7a200b2250 0x7f7a2013fe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:59.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.714+0000 7f7a1fe00640 1 -- 172.21.15.6:0/878977780 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7a2015cc90 msgr2=0x7f7a201f40f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:59.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.714+0000 7f7a1fe00640 1 --2- 172.21.15.6:0/878977780 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7a2015cc90 0x7f7a201f40f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:59.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.714+0000 7f7a1fe00640 1 -- 172.21.15.6:0/878977780 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a20142d60 con 0x7f7a20145430 2024-03-04T09:26:59.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.714+0000 7f7a1fe00640 1 --2- 172.21.15.6:0/878977780 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7a20145430 0x7f7a201f3bb0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f7a0c00c3c0 tx=0x7f7a0c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:59.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.714+0000 7f7a1cc00640 1 -- 172.21.15.6:0/878977780 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a0c0074e0 con 0x7f7a20145430 2024-03-04T09:26:59.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.714+0000 7f7a1cc00640 1 -- 172.21.15.6:0/878977780 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a0c007680 con 0x7f7a20145430 2024-03-04T09:26:59.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.714+0000 7f7a1cc00640 1 -- 172.21.15.6:0/878977780 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a0c004a00 con 0x7f7a20145430 2024-03-04T09:26:59.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.714+0000 7f7a26400640 1 -- 172.21.15.6:0/878977780 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a20143040 con 0x7f7a20145430 2024-03-04T09:26:59.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.715+0000 7f7a26400640 1 -- 172.21.15.6:0/878977780 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a20143600 con 0x7f7a20145430 2024-03-04T09:26:59.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.716+0000 7f7a26400640 1 -- 172.21.15.6:0/878977780 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a2004f330 con 0x7f7a20145430 2024-03-04T09:26:59.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.717+0000 7f7a1cc00640 1 -- 172.21.15.6:0/878977780 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7a0c01a020 con 0x7f7a20145430 2024-03-04T09:26:59.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.718+0000 7f7a1cc00640 1 --2- 172.21.15.6:0/878977780 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f7a0006b4d0 0x7f7a0006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:26:59.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.718+0000 7f7a1cc00640 1 -- 172.21.15.6:0/878977780 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7a0c089970 con 0x7f7a20145430 2024-03-04T09:26:59.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.719+0000 7f7a25000640 1 --2- 172.21.15.6:0/878977780 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f7a0006b4d0 0x7f7a0006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:26:59.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.720+0000 7f7a25000640 1 --2- 172.21.15.6:0/878977780 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f7a0006b4d0 0x7f7a0006d990 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f7a14036000 tx=0x7f7a1403b040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:26:59.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.725+0000 7f7a1cc00640 1 -- 172.21.15.6:0/878977780 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7a0c0587f0 con 0x7f7a20145430 2024-03-04T09:26:59.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.925+0000 7f7a26400640 1 -- 172.21.15.6:0/878977780 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7a200cf310 con 0x7f7a20145430 2024-03-04T09:26:59.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.926+0000 7f7a1cc00640 1 -- 172.21.15.6:0/878977780 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v33) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7a0c0580f0 con 0x7f7a20145430 2024-03-04T09:26:59.923 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-04T09:26:59.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.929+0000 7f7a26400640 1 -- 172.21.15.6:0/878977780 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f7a0006b4d0 msgr2=0x7f7a0006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:59.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.929+0000 7f7a26400640 1 --2- 172.21.15.6:0/878977780 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f7a0006b4d0 0x7f7a0006d990 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f7a14036000 tx=0x7f7a1403b040 comp rx=0 tx=0).stop 2024-03-04T09:26:59.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.929+0000 7f7a26400640 1 -- 172.21.15.6:0/878977780 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7a20145430 msgr2=0x7f7a201f3bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:26:59.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.929+0000 7f7a26400640 1 --2- 172.21.15.6:0/878977780 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7a20145430 0x7f7a201f3bb0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f7a0c00c3c0 tx=0x7f7a0c00c880 comp rx=0 tx=0).stop 2024-03-04T09:26:59.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.929+0000 7f7a26400640 1 -- 172.21.15.6:0/878977780 shutdown_connections 2024-03-04T09:26:59.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.929+0000 7f7a26400640 1 --2- 172.21.15.6:0/878977780 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7a2015cc90 0x7f7a201f40f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:59.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.929+0000 7f7a26400640 1 --2- 172.21.15.6:0/878977780 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7a20145430 0x7f7a201f3bb0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:59.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.929+0000 7f7a26400640 1 --2- 172.21.15.6:0/878977780 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f7a0006b4d0 0x7f7a0006d990 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:59.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.929+0000 7f7a26400640 1 --2- 172.21.15.6:0/878977780 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7a200b2250 0x7f7a2013fe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:26:59.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.929+0000 7f7a26400640 1 -- 172.21.15.6:0/878977780 >> 172.21.15.6:0/878977780 conn(0x7f7a2006b3a0 msgr2=0x7f7a200ade90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:26:59.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.930+0000 7f7a26400640 1 -- 172.21.15.6:0/878977780 shutdown_connections 2024-03-04T09:26:59.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:26:59.930+0000 7f7a26400640 1 -- 172.21.15.6:0/878977780 wait complete. 2024-03-04T09:26:59.946 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-04T09:27:00.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.167+0000 7f95c5e00640 1 Processor -- start 2024-03-04T09:27:00.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.168+0000 7f95c5e00640 1 -- start start 2024-03-04T09:27:00.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.169+0000 7f95c5e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f95c016eb90 0x7f95c016efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:00.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.169+0000 7f95c5e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95c016b700 con 0x7f95c008bb10 2024-03-04T09:27:00.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.169+0000 7f95c5e00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95c016b9e0 con 0x7f95c00b4870 2024-03-04T09:27:00.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.169+0000 7f95c5e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95c016bcc0 con 0x7f95c016eb90 2024-03-04T09:27:00.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.169+0000 7f95c5400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f95c016eb90 0x7f95c016efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:00.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.169+0000 7f95c5400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f95c016eb90 0x7f95c016efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49582/0 (socket says 172.21.15.6:49582) 2024-03-04T09:27:00.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.169+0000 7f95c5400640 1 -- 172.21.15.6:0/219629343 learned_addr learned my addr 172.21.15.6:0/219629343 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:27:00.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.170+0000 7f95bf400640 1 -- 172.21.15.6:0/219629343 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1818364845 0 0) 0x7f95c016b700 con 0x7f95c008bb10 2024-03-04T09:27:00.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.172+0000 7f95bf400640 1 -- 172.21.15.6:0/219629343 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95a8003500 con 0x7f95c008bb10 2024-03-04T09:27:00.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.172+0000 7f95bf400640 1 -- 172.21.15.6:0/219629343 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2884975164 0 0) 0x7f95c016b9e0 con 0x7f95c00b4870 2024-03-04T09:27:00.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.172+0000 7f95bf400640 1 -- 172.21.15.6:0/219629343 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95c016b700 con 0x7f95c00b4870 2024-03-04T09:27:00.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.172+0000 7f95c5400640 1 -- 172.21.15.6:0/219629343 >> v1:172.21.15.146:6789/0 conn(0x7f95c00b4870 legacy=0x7f95c0070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:00.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.172+0000 7f95c5400640 1 -- 172.21.15.6:0/219629343 >> v1:172.21.15.6:6789/0 conn(0x7f95c008bb10 legacy=0x7f95c016aee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:00.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.172+0000 7f95bf400640 1 -- 172.21.15.6:0/219629343 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2284153817 0 0) 0x7f95a8003500 con 0x7f95c008bb10 2024-03-04T09:27:00.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.172+0000 7f95c5400640 1 -- 172.21.15.6:0/219629343 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95c0071660 con 0x7f95c016eb90 2024-03-04T09:27:00.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.173+0000 7f95c5400640 1 --2- 172.21.15.6:0/219629343 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f95c016eb90 0x7f95c016efe0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f95b8007730 tx=0x7f95b802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40a14085b2a23ffb server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:00.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.174+0000 7f95bf400640 1 -- 172.21.15.6:0/219629343 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95b4001670 con 0x7f95c016eb90 2024-03-04T09:27:00.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.174+0000 7f95bf400640 1 -- 172.21.15.6:0/219629343 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95b80092e0 con 0x7f95c016eb90 2024-03-04T09:27:00.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.174+0000 7f95bf400640 1 -- 172.21.15.6:0/219629343 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95b8009690 con 0x7f95c016eb90 2024-03-04T09:27:00.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.174+0000 7f95c5e00640 1 -- 172.21.15.6:0/219629343 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f95c016eb90 msgr2=0x7f95c016efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:00.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.174+0000 7f95c5e00640 1 --2- 172.21.15.6:0/219629343 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f95c016eb90 0x7f95c016efe0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f95b8007730 tx=0x7f95b802bed0 comp rx=0 tx=0).stop 2024-03-04T09:27:00.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.175+0000 7f95c5e00640 1 -- 172.21.15.6:0/219629343 shutdown_connections 2024-03-04T09:27:00.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.175+0000 7f95c5e00640 1 --2- 172.21.15.6:0/219629343 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f95c016eb90 0x7f95c016efe0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:00.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.175+0000 7f95c5e00640 1 -- 172.21.15.6:0/219629343 >> 172.21.15.6:0/219629343 conn(0x7f95c006afb0 msgr2=0x7f95c006b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:00.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.175+0000 7f95c5e00640 1 -- 172.21.15.6:0/219629343 shutdown_connections 2024-03-04T09:27:00.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.175+0000 7f95c5e00640 1 -- 172.21.15.6:0/219629343 wait complete. 2024-03-04T09:27:00.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.176+0000 7f95c5e00640 1 Processor -- start 2024-03-04T09:27:00.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.177+0000 7f95c5e00640 1 -- start start 2024-03-04T09:27:00.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.177+0000 7f95c5e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f95c008bb10 0x7f95c020ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:00.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.177+0000 7f95c5e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f95c00b4870 0x7f95c00cd200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:00.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.177+0000 7f95c5e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f95c016eb90 0x7f95c00cd740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:00.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.177+0000 7f95c5e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95c00b1af0 con 0x7f95c016eb90 2024-03-04T09:27:00.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.177+0000 7f95c5e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95c00b1970 con 0x7f95c00b4870 2024-03-04T09:27:00.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.177+0000 7f95c5e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95c00b1c70 con 0x7f95c008bb10 2024-03-04T09:27:00.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.178+0000 7f95c5400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f95c016eb90 0x7f95c00cd740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:00.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.178+0000 7f95c5400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f95c016eb90 0x7f95c00cd740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49588/0 (socket says 172.21.15.6:49588) 2024-03-04T09:27:00.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.178+0000 7f95c5400640 1 -- 172.21.15.6:0/4085639097 learned_addr learned my addr 172.21.15.6:0/4085639097 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:27:00.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.178+0000 7f95bfe00640 1 --2- 172.21.15.6:0/4085639097 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f95c00b4870 0x7f95c00cd200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:00.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.178+0000 7f95c4a00640 1 --2- 172.21.15.6:0/4085639097 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f95c008bb10 0x7f95c020ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:00.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.178+0000 7f95c5400640 1 -- 172.21.15.6:0/4085639097 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f95c008bb10 msgr2=0x7f95c020ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:00.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.178+0000 7f95c5400640 1 --2- 172.21.15.6:0/4085639097 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f95c008bb10 0x7f95c020ced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:00.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.178+0000 7f95c5400640 1 -- 172.21.15.6:0/4085639097 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f95c00b4870 msgr2=0x7f95c00cd200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:00.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.178+0000 7f95c5400640 1 --2- 172.21.15.6:0/4085639097 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f95c00b4870 0x7f95c00cd200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:00.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.178+0000 7f95c5400640 1 -- 172.21.15.6:0/4085639097 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95c01f4800 con 0x7f95c016eb90 2024-03-04T09:27:00.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.179+0000 7f95c5400640 1 --2- 172.21.15.6:0/4085639097 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f95c016eb90 0x7f95c00cd740 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f95b8009f50 tx=0x7f95b8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:00.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.179+0000 7f95bcc00640 1 -- 172.21.15.6:0/4085639097 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95b8039040 con 0x7f95c016eb90 2024-03-04T09:27:00.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.179+0000 7f95bcc00640 1 -- 172.21.15.6:0/4085639097 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95b8007a40 con 0x7f95c016eb90 2024-03-04T09:27:00.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.179+0000 7f95bcc00640 1 -- 172.21.15.6:0/4085639097 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95b8007df0 con 0x7f95c016eb90 2024-03-04T09:27:00.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.179+0000 7f95c5e00640 1 -- 172.21.15.6:0/4085639097 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95c01f4ac0 con 0x7f95c016eb90 2024-03-04T09:27:00.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.179+0000 7f95c5e00640 1 -- 172.21.15.6:0/4085639097 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95c01f5080 con 0x7f95c016eb90 2024-03-04T09:27:00.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.180+0000 7f95c5e00640 1 -- 172.21.15.6:0/4085639097 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95c004f330 con 0x7f95c016eb90 2024-03-04T09:27:00.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.181+0000 7f95bcc00640 1 -- 172.21.15.6:0/4085639097 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f95b802c7f0 con 0x7f95c016eb90 2024-03-04T09:27:00.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.182+0000 7f95bcc00640 1 --2- 172.21.15.6:0/4085639097 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f95a006b4d0 0x7f95a006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:00.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.182+0000 7f95bcc00640 1 -- 172.21.15.6:0/4085639097 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f95b80ac210 con 0x7f95c016eb90 2024-03-04T09:27:00.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.183+0000 7f95c4a00640 1 --2- 172.21.15.6:0/4085639097 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f95a006b4d0 0x7f95a006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:00.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.184+0000 7f95c4a00640 1 --2- 172.21.15.6:0/4085639097 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f95a006b4d0 0x7f95a006d990 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f95b4007980 tx=0x7f95b4009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:00.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.189+0000 7f95bcc00640 1 -- 172.21.15.6:0/4085639097 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f95b807b080 con 0x7f95c016eb90 2024-03-04T09:27:00.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.397+0000 7f95c5e00640 1 -- 172.21.15.6:0/4085639097 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f95c00b3de0 con 0x7f95c016eb90 2024-03-04T09:27:00.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.398+0000 7f95bcc00640 1 -- 172.21.15.6:0/4085639097 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v33) v1 ==== 104+0+11 (secure 0 0 0) 0x7f95b807a980 con 0x7f95c016eb90 2024-03-04T09:27:00.395 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-03-04T09:27:00.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.401+0000 7f95c5e00640 1 -- 172.21.15.6:0/4085639097 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f95a006b4d0 msgr2=0x7f95a006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:00.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.401+0000 7f95c5e00640 1 --2- 172.21.15.6:0/4085639097 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f95a006b4d0 0x7f95a006d990 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f95b4007980 tx=0x7f95b4009040 comp rx=0 tx=0).stop 2024-03-04T09:27:00.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.401+0000 7f95c5e00640 1 -- 172.21.15.6:0/4085639097 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f95c016eb90 msgr2=0x7f95c00cd740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:00.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.401+0000 7f95c5e00640 1 --2- 172.21.15.6:0/4085639097 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f95c016eb90 0x7f95c00cd740 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f95b8009f50 tx=0x7f95b8004a40 comp rx=0 tx=0).stop 2024-03-04T09:27:00.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.401+0000 7f95c5e00640 1 -- 172.21.15.6:0/4085639097 shutdown_connections 2024-03-04T09:27:00.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.401+0000 7f95c5e00640 1 --2- 172.21.15.6:0/4085639097 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f95c016eb90 0x7f95c00cd740 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:00.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.401+0000 7f95c5e00640 1 --2- 172.21.15.6:0/4085639097 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f95c00b4870 0x7f95c00cd200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:00.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.401+0000 7f95c5e00640 1 --2- 172.21.15.6:0/4085639097 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f95a006b4d0 0x7f95a006d990 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:00.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.401+0000 7f95c5e00640 1 --2- 172.21.15.6:0/4085639097 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f95c008bb10 0x7f95c020ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:00.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.401+0000 7f95c5e00640 1 -- 172.21.15.6:0/4085639097 >> 172.21.15.6:0/4085639097 conn(0x7f95c006afb0 msgr2=0x7f95c00b07b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:00.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.401+0000 7f95c5e00640 1 -- 172.21.15.6:0/4085639097 shutdown_connections 2024-03-04T09:27:00.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.402+0000 7f95c5e00640 1 -- 172.21.15.6:0/4085639097 wait complete. 2024-03-04T09:27:00.418 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-04T09:27:00.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.651+0000 7fbc3d400640 1 Processor -- start 2024-03-04T09:27:00.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.651+0000 7fbc3d400640 1 -- start start 2024-03-04T09:27:00.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.653+0000 7fbc3d400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fbc380ccad0 0x7fbc380cceb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:00.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.653+0000 7fbc3d400640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc380cd3f0 con 0x7fbc380adb00 2024-03-04T09:27:00.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.653+0000 7fbc3d400640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc38163950 con 0x7fbc38162bb0 2024-03-04T09:27:00.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.653+0000 7fbc3d400640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc380acd40 con 0x7fbc380ccad0 2024-03-04T09:27:00.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.653+0000 7fbc36800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fbc380ccad0 0x7fbc380cceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:00.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.653+0000 7fbc36800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fbc380ccad0 0x7fbc380cceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49602/0 (socket says 172.21.15.6:49602) 2024-03-04T09:27:00.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.653+0000 7fbc36800640 1 -- 172.21.15.6:0/146486541 learned_addr learned my addr 172.21.15.6:0/146486541 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:27:00.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.653+0000 7fbc37200640 1 --1- 172.21.15.6:0/146486541 >> v1:172.21.15.6:6789/0 conn(0x7fbc380adb00 0x7fbc380adf10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:38986/0 (socket says 172.21.15.6:38986) 2024-03-04T09:27:00.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.654+0000 7fbc36800640 1 -- 172.21.15.6:0/146486541 >> v1:172.21.15.146:6789/0 conn(0x7fbc38162bb0 legacy=0x7fbc38163040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:27:00.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.654+0000 7fbc36800640 1 -- 172.21.15.6:0/146486541 >> v1:172.21.15.6:6789/0 conn(0x7fbc380adb00 legacy=0x7fbc380adf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:00.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.654+0000 7fbc36800640 1 -- 172.21.15.6:0/146486541 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc380ad020 con 0x7fbc380ccad0 2024-03-04T09:27:00.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.654+0000 7fbc35400640 1 -- 172.21.15.6:0/146486541 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 744413668 0 0) 0x7fbc380cd3f0 con 0x7fbc380adb00 2024-03-04T09:27:00.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.655+0000 7fbc36800640 1 --2- 172.21.15.6:0/146486541 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbc380ccad0 0x7fbc380cceb0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fbc20009870 tx=0x7fbc2002b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6af5293cdef9d897 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:00.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.655+0000 7fbc35400640 1 -- 172.21.15.6:0/146486541 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc20034070 con 0x7fbc380ccad0 2024-03-04T09:27:00.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.655+0000 7fbc35400640 1 -- 172.21.15.6:0/146486541 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc20030850 con 0x7fbc380ccad0 2024-03-04T09:27:00.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.655+0000 7fbc35400640 1 -- 172.21.15.6:0/146486541 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc20030c20 con 0x7fbc380ccad0 2024-03-04T09:27:00.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.656+0000 7fbc3d400640 1 -- 172.21.15.6:0/146486541 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbc380ccad0 msgr2=0x7fbc380cceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:00.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.656+0000 7fbc3d400640 1 --2- 172.21.15.6:0/146486541 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbc380ccad0 0x7fbc380cceb0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fbc20009870 tx=0x7fbc2002b990 comp rx=0 tx=0).stop 2024-03-04T09:27:00.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.656+0000 7fbc3d400640 1 -- 172.21.15.6:0/146486541 shutdown_connections 2024-03-04T09:27:00.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.656+0000 7fbc3d400640 1 --2- 172.21.15.6:0/146486541 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbc380ccad0 0x7fbc380cceb0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:00.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.656+0000 7fbc3d400640 1 -- 172.21.15.6:0/146486541 >> 172.21.15.6:0/146486541 conn(0x7fbc3806b3d0 msgr2=0x7fbc3806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:00.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.656+0000 7fbc3d400640 1 -- 172.21.15.6:0/146486541 shutdown_connections 2024-03-04T09:27:00.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.656+0000 7fbc3d400640 1 -- 172.21.15.6:0/146486541 wait complete. 2024-03-04T09:27:00.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.658+0000 7fbc3d400640 1 Processor -- start 2024-03-04T09:27:00.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.658+0000 7fbc3d400640 1 -- start start 2024-03-04T09:27:00.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.659+0000 7fbc3d400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbc380adb00 0x7fbc380b42f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:00.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.659+0000 7fbc3d400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbc380ccad0 0x7fbc380c4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:00.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.659+0000 7fbc3d400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbc38162bb0 0x7fbc380c5490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:00.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.659+0000 7fbc3d400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc3806e7d0 con 0x7fbc380ccad0 2024-03-04T09:27:00.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.659+0000 7fbc3d400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc380ad5e0 con 0x7fbc38162bb0 2024-03-04T09:27:00.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.659+0000 7fbc3d400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc3806c980 con 0x7fbc380adb00 2024-03-04T09:27:00.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.659+0000 7fbc35e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbc380ccad0 0x7fbc380c4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:00.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.659+0000 7fbc35e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbc380ccad0 0x7fbc380c4f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49612/0 (socket says 172.21.15.6:49612) 2024-03-04T09:27:00.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.659+0000 7fbc35e00640 1 -- 172.21.15.6:0/545322607 learned_addr learned my addr 172.21.15.6:0/545322607 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:27:00.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.660+0000 7fbc37200640 1 --2- 172.21.15.6:0/545322607 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbc38162bb0 0x7fbc380c5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:00.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.660+0000 7fbc36800640 1 --2- 172.21.15.6:0/545322607 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbc380adb00 0x7fbc380b42f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:00.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.660+0000 7fbc35e00640 1 -- 172.21.15.6:0/545322607 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbc380adb00 msgr2=0x7fbc380b42f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:00.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.660+0000 7fbc35e00640 1 --2- 172.21.15.6:0/545322607 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbc380adb00 0x7fbc380b42f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:00.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.660+0000 7fbc35e00640 1 -- 172.21.15.6:0/545322607 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbc38162bb0 msgr2=0x7fbc380c5490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:00.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.660+0000 7fbc35e00640 1 --2- 172.21.15.6:0/545322607 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbc38162bb0 0x7fbc380c5490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:00.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.660+0000 7fbc35e00640 1 -- 172.21.15.6:0/545322607 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc380b3250 con 0x7fbc380ccad0 2024-03-04T09:27:00.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.660+0000 7fbc37200640 1 --2- 172.21.15.6:0/545322607 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbc38162bb0 0x7fbc380c5490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:27:00.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.660+0000 7fbc35e00640 1 --2- 172.21.15.6:0/545322607 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbc380ccad0 0x7fbc380c4f50 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fbc2800c7c0 tx=0x7fbc2800cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:00.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.660+0000 7fbc2ea00640 1 -- 172.21.15.6:0/545322607 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc28016020 con 0x7fbc380ccad0 2024-03-04T09:27:00.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.660+0000 7fbc2ea00640 1 -- 172.21.15.6:0/545322607 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc2800aa50 con 0x7fbc380ccad0 2024-03-04T09:27:00.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.660+0000 7fbc2ea00640 1 -- 172.21.15.6:0/545322607 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc2800ae40 con 0x7fbc380ccad0 2024-03-04T09:27:00.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.660+0000 7fbc3d400640 1 -- 172.21.15.6:0/545322607 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc380b3530 con 0x7fbc380ccad0 2024-03-04T09:27:00.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.661+0000 7fbc3d400640 1 -- 172.21.15.6:0/545322607 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc380b3af0 con 0x7fbc380ccad0 2024-03-04T09:27:00.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.662+0000 7fbc3d400640 1 -- 172.21.15.6:0/545322607 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc3804f330 con 0x7fbc380ccad0 2024-03-04T09:27:00.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.663+0000 7fbc2ea00640 1 -- 172.21.15.6:0/545322607 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbc28019020 con 0x7fbc380ccad0 2024-03-04T09:27:00.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.663+0000 7fbc2ea00640 1 --2- 172.21.15.6:0/545322607 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fbc1406b4d0 0x7fbc1406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:00.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.664+0000 7fbc2ea00640 1 -- 172.21.15.6:0/545322607 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbc28089f80 con 0x7fbc380ccad0 2024-03-04T09:27:00.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.664+0000 7fbc36800640 1 --2- 172.21.15.6:0/545322607 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fbc1406b4d0 0x7fbc1406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:00.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.665+0000 7fbc36800640 1 --2- 172.21.15.6:0/545322607 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fbc1406b4d0 0x7fbc1406d990 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fbc20036000 tx=0x7fbc20008030 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:00.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.668+0000 7fbc2ea00640 1 -- 172.21.15.6:0/545322607 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbc28020070 con 0x7fbc380ccad0 2024-03-04T09:27:00.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.871+0000 7fbc3d400640 1 -- 172.21.15.6:0/545322607 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fbc380b16d0 con 0x7fbc380ccad0 2024-03-04T09:27:00.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.872+0000 7fbc2ea00640 1 -- 172.21.15.6:0/545322607 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v33) v1 ==== 100+0+11 (secure 0 0 0) 0x7fbc28058e00 con 0x7fbc380ccad0 2024-03-04T09:27:00.869 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 64 2024-03-04T09:27:00.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.874+0000 7fbc3d400640 1 -- 172.21.15.6:0/545322607 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fbc1406b4d0 msgr2=0x7fbc1406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:00.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.874+0000 7fbc3d400640 1 --2- 172.21.15.6:0/545322607 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fbc1406b4d0 0x7fbc1406d990 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fbc20036000 tx=0x7fbc20008030 comp rx=0 tx=0).stop 2024-03-04T09:27:00.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.874+0000 7fbc3d400640 1 -- 172.21.15.6:0/545322607 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbc380ccad0 msgr2=0x7fbc380c4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:00.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.874+0000 7fbc3d400640 1 --2- 172.21.15.6:0/545322607 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbc380ccad0 0x7fbc380c4f50 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fbc2800c7c0 tx=0x7fbc2800cc80 comp rx=0 tx=0).stop 2024-03-04T09:27:00.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.875+0000 7fbc3d400640 1 -- 172.21.15.6:0/545322607 shutdown_connections 2024-03-04T09:27:00.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.875+0000 7fbc3d400640 1 --2- 172.21.15.6:0/545322607 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbc38162bb0 0x7fbc380c5490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:00.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.875+0000 7fbc3d400640 1 --2- 172.21.15.6:0/545322607 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbc380ccad0 0x7fbc380c4f50 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:00.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.875+0000 7fbc3d400640 1 --2- 172.21.15.6:0/545322607 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fbc1406b4d0 0x7fbc1406d990 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:00.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.875+0000 7fbc3d400640 1 --2- 172.21.15.6:0/545322607 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbc380adb00 0x7fbc380b42f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:00.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.875+0000 7fbc3d400640 1 -- 172.21.15.6:0/545322607 >> 172.21.15.6:0/545322607 conn(0x7fbc3806b3d0 msgr2=0x7fbc380cdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:00.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.875+0000 7fbc3d400640 1 -- 172.21.15.6:0/545322607 shutdown_connections 2024-03-04T09:27:00.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:00.875+0000 7fbc3d400640 1 -- 172.21.15.6:0/545322607 wait complete. 2024-03-04T09:27:00.892 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:27:01.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.117+0000 7f2d1d000640 1 Processor -- start 2024-03-04T09:27:01.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.117+0000 7f2d1d000640 1 -- start start 2024-03-04T09:27:01.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.119+0000 7f2d1d000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f2d180af1a0 0x7f2d1815c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:01.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.119+0000 7f2d1d000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f2d180af580 0x7f2d180b4c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:01.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.119+0000 7f2d1d000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d1815cc40 con 0x7f2d180cdc20 2024-03-04T09:27:01.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.119+0000 7f2d1d000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d1815cf20 con 0x7f2d180af580 2024-03-04T09:27:01.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.119+0000 7f2d1d000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d1815d200 con 0x7f2d180af1a0 2024-03-04T09:27:01.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.120+0000 7f2d16800640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f2d180cdc20 0x7f2d180b2dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:38994/0 (socket says 172.21.15.6:38994) 2024-03-04T09:27:01.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.120+0000 7f2d16800640 1 -- 172.21.15.6:0/63904672 learned_addr learned my addr 172.21.15.6:0/63904672 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:27:01.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.120+0000 7f2d0fe00640 1 --2- 172.21.15.6:0/63904672 >> v2:172.21.15.6:3300/0 conn(0x7f2d180af580 0x7f2d180b4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:01.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.120+0000 7f2d15e00640 1 --2- 172.21.15.6:0/63904672 >> v2:172.21.15.146:3300/0 conn(0x7f2d180af1a0 0x7f2d1815c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:01.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.120+0000 7f2d0fe00640 1 -- 172.21.15.6:0/63904672 >> v2:172.21.15.146:3300/0 conn(0x7f2d180af1a0 msgr2=0x7f2d1815c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:27:01.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.120+0000 7f2d0fe00640 1 --2- 172.21.15.6:0/63904672 >> v2:172.21.15.146:3300/0 conn(0x7f2d180af1a0 0x7f2d1815c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:01.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.120+0000 7f2d0fe00640 1 -- 172.21.15.6:0/63904672 >> v1:172.21.15.6:6789/0 conn(0x7f2d180cdc20 legacy=0x7f2d180b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:01.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.120+0000 7f2d15400640 1 -- 172.21.15.6:0/63904672 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1826089520 0 0) 0x7f2d1815cc40 con 0x7f2d180cdc20 2024-03-04T09:27:01.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.120+0000 7f2d0fe00640 1 -- 172.21.15.6:0/63904672 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d1815d4e0 con 0x7f2d180af580 2024-03-04T09:27:01.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.120+0000 7f2d15e00640 1 --2- 172.21.15.6:0/63904672 >> v2:172.21.15.146:3300/0 conn(0x7f2d180af1a0 0x7f2d1815c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T09:27:01.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.121+0000 7f2d0fe00640 1 --2- 172.21.15.6:0/63904672 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2d180af580 0x7f2d180b4c20 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f2d18074960 tx=0x7f2d0002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b07d1a340894f141 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:01.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.122+0000 7f2d15400640 1 -- 172.21.15.6:0/63904672 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d000315a0 con 0x7f2d180af580 2024-03-04T09:27:01.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.122+0000 7f2d15400640 1 -- 172.21.15.6:0/63904672 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d00037070 con 0x7f2d180af580 2024-03-04T09:27:01.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.122+0000 7f2d15400640 1 -- 172.21.15.6:0/63904672 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d00031960 con 0x7f2d180af580 2024-03-04T09:27:01.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.122+0000 7f2d1d000640 1 -- 172.21.15.6:0/63904672 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2d180af580 msgr2=0x7f2d180b4c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:01.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.122+0000 7f2d1d000640 1 --2- 172.21.15.6:0/63904672 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2d180af580 0x7f2d180b4c20 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f2d18074960 tx=0x7f2d0002b840 comp rx=0 tx=0).stop 2024-03-04T09:27:01.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.123+0000 7f2d1d000640 1 -- 172.21.15.6:0/63904672 shutdown_connections 2024-03-04T09:27:01.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.123+0000 7f2d1d000640 1 --2- 172.21.15.6:0/63904672 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2d180af580 0x7f2d180b4c20 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:01.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.123+0000 7f2d1d000640 1 --2- 172.21.15.6:0/63904672 >> v2:172.21.15.146:3300/0 conn(0x7f2d180af1a0 0x7f2d1815c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:01.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.123+0000 7f2d1d000640 1 -- 172.21.15.6:0/63904672 >> 172.21.15.6:0/63904672 conn(0x7f2d1806afb0 msgr2=0x7f2d1806b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:01.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.123+0000 7f2d1d000640 1 -- 172.21.15.6:0/63904672 shutdown_connections 2024-03-04T09:27:01.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.123+0000 7f2d1d000640 1 -- 172.21.15.6:0/63904672 wait complete. 2024-03-04T09:27:01.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.125+0000 7f2d1d000640 1 Processor -- start 2024-03-04T09:27:01.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.125+0000 7f2d1d000640 1 -- start start 2024-03-04T09:27:01.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.126+0000 7f2d1d000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2d180af1a0 0x7f2d1814e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:01.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.126+0000 7f2d1d000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2d180af580 0x7f2d180af9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:01.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.126+0000 7f2d1d000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2d180cdc20 0x7f2d1815e270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:01.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.126+0000 7f2d1d000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d180b3660 con 0x7f2d180cdc20 2024-03-04T09:27:01.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.126+0000 7f2d1d000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d180b34e0 con 0x7f2d180af1a0 2024-03-04T09:27:01.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.126+0000 7f2d1d000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d180b37e0 con 0x7f2d180af580 2024-03-04T09:27:01.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.126+0000 7f2d16800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2d180cdc20 0x7f2d1815e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:01.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.126+0000 7f2d16800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2d180cdc20 0x7f2d1815e270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49636/0 (socket says 172.21.15.6:49636) 2024-03-04T09:27:01.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.126+0000 7f2d16800640 1 -- 172.21.15.6:0/1021944565 learned_addr learned my addr 172.21.15.6:0/1021944565 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:27:01.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.126+0000 7f2d15e00640 1 --2- 172.21.15.6:0/1021944565 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2d180af1a0 0x7f2d1814e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:01.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.126+0000 7f2d0fe00640 1 --2- 172.21.15.6:0/1021944565 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2d180af580 0x7f2d180af9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:01.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.127+0000 7f2d16800640 1 -- 172.21.15.6:0/1021944565 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2d180af580 msgr2=0x7f2d180af9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:01.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.127+0000 7f2d16800640 1 --2- 172.21.15.6:0/1021944565 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2d180af580 0x7f2d180af9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:01.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.127+0000 7f2d16800640 1 -- 172.21.15.6:0/1021944565 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2d180af1a0 msgr2=0x7f2d1814e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:01.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.127+0000 7f2d16800640 1 --2- 172.21.15.6:0/1021944565 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2d180af1a0 0x7f2d1814e5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:01.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.127+0000 7f2d16800640 1 -- 172.21.15.6:0/1021944565 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d180aff10 con 0x7f2d180cdc20 2024-03-04T09:27:01.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.127+0000 7f2d16800640 1 --2- 172.21.15.6:0/1021944565 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2d180cdc20 0x7f2d1815e270 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f2d1000c780 tx=0x7f2d1000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:01.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.127+0000 7f2d0e000640 1 -- 172.21.15.6:0/1021944565 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d10018070 con 0x7f2d180cdc20 2024-03-04T09:27:01.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.127+0000 7f2d0e000640 1 -- 172.21.15.6:0/1021944565 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d1000a640 con 0x7f2d180cdc20 2024-03-04T09:27:01.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.127+0000 7f2d0e000640 1 -- 172.21.15.6:0/1021944565 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d1000aa30 con 0x7f2d180cdc20 2024-03-04T09:27:01.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.127+0000 7f2d1d000640 1 -- 172.21.15.6:0/1021944565 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d1806cd30 con 0x7f2d180cdc20 2024-03-04T09:27:01.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.128+0000 7f2d1d000640 1 -- 172.21.15.6:0/1021944565 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d180ca0e0 con 0x7f2d180cdc20 2024-03-04T09:27:01.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.129+0000 7f2d1d000640 1 -- 172.21.15.6:0/1021944565 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d1804f3a0 con 0x7f2d180cdc20 2024-03-04T09:27:01.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.130+0000 7f2d0e000640 1 -- 172.21.15.6:0/1021944565 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2d1000abd0 con 0x7f2d180cdc20 2024-03-04T09:27:01.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.131+0000 7f2d0e000640 1 --2- 172.21.15.6:0/1021944565 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2cf406b4d0 0x7f2cf406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:01.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.131+0000 7f2d0e000640 1 -- 172.21.15.6:0/1021944565 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2d10089750 con 0x7f2d180cdc20 2024-03-04T09:27:01.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.132+0000 7f2d15e00640 1 --2- 172.21.15.6:0/1021944565 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2cf406b4d0 0x7f2cf406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:01.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.133+0000 7f2d15e00640 1 --2- 172.21.15.6:0/1021944565 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2cf406b4d0 0x7f2cf406d990 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f2d08006fd0 tx=0x7f2d08008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:01.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.136+0000 7f2d0e000640 1 -- 172.21.15.6:0/1021944565 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2d100585d0 con 0x7f2d180cdc20 2024-03-04T09:27:01.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.381+0000 7f2d1d000640 1 -- 172.21.15.6:0/1021944565 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2d1815e7b0 con 0x7f2d180cdc20 2024-03-04T09:27:01.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.382+0000 7f2d0e000640 1 -- 172.21.15.6:0/1021944565 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+4977 (secure 0 0 0) 0x7f2d10057ed0 con 0x7f2d180cdc20 2024-03-04T09:27:01.379 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:27:01.379 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4685,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/1703428647","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":1703428647},{"type":"v1","addr":"172.21.15.146:6837","nonce":1703428647}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4837,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/2617429530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2617429530},{"type":"v1","addr":"172.21.15.6:6833","nonce":2617429530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":5043,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/2744913857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":2744913857},{"type":"v1","addr":"172.21.15.6:6835","nonce":2744913857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:26:30.445199+0000","modified":"2024-03-04T09:26:33.534893+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5046},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5046":{"gid":5046,"name":"d","rank":0,"incarnation":12,"state":"up:active","state_seq":4,"addr":"172.21.15.146:6836/2285803221","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":2285803221},{"type":"v1","addr":"172.21.15.146:6836","nonce":2285803221}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5046,"qdb_cluster":[5046]},"id":2}]} 2024-03-04T09:27:01.380 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 16 2024-03-04T09:27:01.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.385+0000 7f2d1d000640 1 -- 172.21.15.6:0/1021944565 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2cf406b4d0 msgr2=0x7f2cf406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:01.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.385+0000 7f2d1d000640 1 --2- 172.21.15.6:0/1021944565 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2cf406b4d0 0x7f2cf406d990 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f2d08006fd0 tx=0x7f2d08008040 comp rx=0 tx=0).stop 2024-03-04T09:27:01.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.385+0000 7f2d1d000640 1 -- 172.21.15.6:0/1021944565 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2d180cdc20 msgr2=0x7f2d1815e270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:01.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.385+0000 7f2d1d000640 1 --2- 172.21.15.6:0/1021944565 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2d180cdc20 0x7f2d1815e270 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f2d1000c780 tx=0x7f2d1000cc40 comp rx=0 tx=0).stop 2024-03-04T09:27:01.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.386+0000 7f2d1d000640 1 -- 172.21.15.6:0/1021944565 shutdown_connections 2024-03-04T09:27:01.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.386+0000 7f2d1d000640 1 --2- 172.21.15.6:0/1021944565 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2d180cdc20 0x7f2d1815e270 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:01.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.386+0000 7f2d1d000640 1 --2- 172.21.15.6:0/1021944565 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2cf406b4d0 0x7f2cf406d990 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:01.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.386+0000 7f2d1d000640 1 --2- 172.21.15.6:0/1021944565 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2d180af580 0x7f2d180af9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:01.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.386+0000 7f2d1d000640 1 --2- 172.21.15.6:0/1021944565 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2d180af1a0 0x7f2d1814e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:01.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.386+0000 7f2d1d000640 1 -- 172.21.15.6:0/1021944565 >> 172.21.15.6:0/1021944565 conn(0x7f2d1806afb0 msgr2=0x7f2d180ae5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:01.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.386+0000 7f2d1d000640 1 -- 172.21.15.6:0/1021944565 shutdown_connections 2024-03-04T09:27:01.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.386+0000 7f2d1d000640 1 -- 172.21.15.6:0/1021944565 wait complete. 2024-03-04T09:27:01.403 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T09:27:01.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.630+0000 7fb4f6200640 1 Processor -- start 2024-03-04T09:27:01.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.630+0000 7fb4f6200640 1 -- start start 2024-03-04T09:27:01.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.632+0000 7fb4f6200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fb4f01552b0 0x7fb4f015b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:01.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.632+0000 7fb4f6200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb4f00cd060 0x7fb4f00b3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:01.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.632+0000 7fb4f6200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4f015ed10 con 0x7fb4f015e810 2024-03-04T09:27:01.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.632+0000 7fb4f6200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4f015eff0 con 0x7fb4f01552b0 2024-03-04T09:27:01.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.632+0000 7fb4f6200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4f015f2d0 con 0x7fb4f00cd060 2024-03-04T09:27:01.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.632+0000 7fb4f4e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fb4f01552b0 0x7fb4f015b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:01.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.632+0000 7fb4f4e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fb4f01552b0 0x7fb4f015b1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49646/0 (socket says 172.21.15.6:49646) 2024-03-04T09:27:01.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.632+0000 7fb4f4e00640 1 -- 172.21.15.6:0/1891166328 learned_addr learned my addr 172.21.15.6:0/1891166328 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:27:01.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.633+0000 7fb4f4e00640 1 -- 172.21.15.6:0/1891166328 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb4f00cd060 msgr2=0x7fb4f00b3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:27:01.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.633+0000 7fb4f4e00640 1 --2- 172.21.15.6:0/1891166328 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb4f00cd060 0x7fb4f00b3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:01.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.633+0000 7fb4f4e00640 1 -- 172.21.15.6:0/1891166328 >> v1:172.21.15.6:6789/0 conn(0x7fb4f015e810 legacy=0x7fb4f00cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:01.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.633+0000 7fb4f4e00640 1 -- 172.21.15.6:0/1891166328 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4f015f5b0 con 0x7fb4f01552b0 2024-03-04T09:27:01.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.634+0000 7fb4f4e00640 1 --2- 172.21.15.6:0/1891166328 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb4f01552b0 0x7fb4f015b1d0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fb4e0009870 tx=0x7fb4e002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fb8b512300f3e3a server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:01.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.634+0000 7fb4efe00640 1 -- 172.21.15.6:0/1891166328 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4e0035020 con 0x7fb4f01552b0 2024-03-04T09:27:01.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.634+0000 7fb4efe00640 1 -- 172.21.15.6:0/1891166328 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4e0031870 con 0x7fb4f01552b0 2024-03-04T09:27:01.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.634+0000 7fb4efe00640 1 -- 172.21.15.6:0/1891166328 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4e0031c40 con 0x7fb4f01552b0 2024-03-04T09:27:01.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.635+0000 7fb4f6200640 1 -- 172.21.15.6:0/1891166328 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb4f01552b0 msgr2=0x7fb4f015b1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:01.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.635+0000 7fb4f6200640 1 --2- 172.21.15.6:0/1891166328 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb4f01552b0 0x7fb4f015b1d0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fb4e0009870 tx=0x7fb4e002b910 comp rx=0 tx=0).stop 2024-03-04T09:27:01.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.635+0000 7fb4f6200640 1 -- 172.21.15.6:0/1891166328 shutdown_connections 2024-03-04T09:27:01.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.635+0000 7fb4f6200640 1 --2- 172.21.15.6:0/1891166328 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb4f00cd060 0x7fb4f00b3cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:01.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.635+0000 7fb4f6200640 1 --2- 172.21.15.6:0/1891166328 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb4f01552b0 0x7fb4f015b1d0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:01.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.635+0000 7fb4f6200640 1 -- 172.21.15.6:0/1891166328 >> 172.21.15.6:0/1891166328 conn(0x7fb4f006b3d0 msgr2=0x7fb4f006b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:01.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.635+0000 7fb4f6200640 1 -- 172.21.15.6:0/1891166328 shutdown_connections 2024-03-04T09:27:01.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.635+0000 7fb4f6200640 1 -- 172.21.15.6:0/1891166328 wait complete. 2024-03-04T09:27:01.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.637+0000 7fb4f6200640 1 Processor -- start 2024-03-04T09:27:01.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.637+0000 7fb4f6200640 1 -- start start 2024-03-04T09:27:01.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.638+0000 7fb4f6200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb4f00cd060 0x7fb4f0147720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:01.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.638+0000 7fb4f6200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb4f01552b0 0x7fb4f014d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:01.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.638+0000 7fb4f6200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb4f015e810 0x7fb4f014dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:01.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.638+0000 7fb4f6200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4f006dd30 con 0x7fb4f00cd060 2024-03-04T09:27:01.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.638+0000 7fb4f6200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4f00b3760 con 0x7fb4f01552b0 2024-03-04T09:27:01.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.638+0000 7fb4f6200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4f006deb0 con 0x7fb4f015e810 2024-03-04T09:27:01.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.638+0000 7fb4f4e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb4f00cd060 0x7fb4f0147720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:01.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.638+0000 7fb4f4e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb4f00cd060 0x7fb4f0147720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49650/0 (socket says 172.21.15.6:49650) 2024-03-04T09:27:01.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.638+0000 7fb4f4e00640 1 -- 172.21.15.6:0/2799461043 learned_addr learned my addr 172.21.15.6:0/2799461043 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:27:01.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.638+0000 7fb4e7e00640 1 --2- 172.21.15.6:0/2799461043 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb4f01552b0 0x7fb4f014d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:01.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.638+0000 7fb4f4e00640 1 -- 172.21.15.6:0/2799461043 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb4f015e810 msgr2=0x7fb4f014dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:01.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.638+0000 7fb4f4e00640 1 --2- 172.21.15.6:0/2799461043 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb4f015e810 0x7fb4f014dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:01.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.639+0000 7fb4f4e00640 1 -- 172.21.15.6:0/2799461043 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb4f01552b0 msgr2=0x7fb4f014d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:01.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.639+0000 7fb4f4e00640 1 --2- 172.21.15.6:0/2799461043 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb4f01552b0 0x7fb4f014d690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:01.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.639+0000 7fb4f4e00640 1 -- 172.21.15.6:0/2799461043 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4f014e3b0 con 0x7fb4f00cd060 2024-03-04T09:27:01.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.639+0000 7fb4f4e00640 1 --2- 172.21.15.6:0/2799461043 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb4f00cd060 0x7fb4f0147720 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fb4e0009870 tx=0x7fb4e00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:01.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.639+0000 7fb4ed600640 1 -- 172.21.15.6:0/2799461043 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4e00443f0 con 0x7fb4f00cd060 2024-03-04T09:27:01.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.639+0000 7fb4ed600640 1 -- 172.21.15.6:0/2799461043 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4e0044590 con 0x7fb4f00cd060 2024-03-04T09:27:01.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.639+0000 7fb4ed600640 1 -- 172.21.15.6:0/2799461043 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4e0044960 con 0x7fb4f00cd060 2024-03-04T09:27:01.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.639+0000 7fb4f6200640 1 -- 172.21.15.6:0/2799461043 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4f014e650 con 0x7fb4f00cd060 2024-03-04T09:27:01.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.639+0000 7fb4f6200640 1 -- 172.21.15.6:0/2799461043 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4f014ec10 con 0x7fb4f00cd060 2024-03-04T09:27:01.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.641+0000 7fb4f6200640 1 -- 172.21.15.6:0/2799461043 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4f004f330 con 0x7fb4f00cd060 2024-03-04T09:27:01.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.641+0000 7fb4ed600640 1 -- 172.21.15.6:0/2799461043 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb4e004b060 con 0x7fb4f00cd060 2024-03-04T09:27:01.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.642+0000 7fb4ed600640 1 --2- 172.21.15.6:0/2799461043 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb4cc06b4d0 0x7fb4cc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:01.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.642+0000 7fb4ed600640 1 -- 172.21.15.6:0/2799461043 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb4e00ac480 con 0x7fb4f00cd060 2024-03-04T09:27:01.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.643+0000 7fb4e7e00640 1 --2- 172.21.15.6:0/2799461043 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb4cc06b4d0 0x7fb4cc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:01.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.644+0000 7fb4e7e00640 1 --2- 172.21.15.6:0/2799461043 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb4cc06b4d0 0x7fb4cc06d990 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fb4d8007fd0 tx=0x7fb4d8009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:01.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.647+0000 7fb4ed600640 1 -- 172.21.15.6:0/2799461043 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb4e003f070 con 0x7fb4f00cd060 2024-03-04T09:27:01.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.847+0000 7fb4f6200640 1 -- 172.21.15.6:0/2799461043 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb4f00cf4a0 con 0x7fb4f00cd060 2024-03-04T09:27:01.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.850+0000 7fb4ed600640 1 -- 172.21.15.6:0/2799461043 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v33) v1 ==== 74+0+16621 (secure 0 0 0) 0x7fb4e007b2f0 con 0x7fb4f00cd060 2024-03-04T09:27:01.848 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:27:01.848 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":33,"fsid":"edce0578-5ae4-4402-b45a-552bdef84d7c","created":"2024-03-04T09:24:54.523780+0000","modified":"2024-03-04T09:26:31.991047+0000","last_up_change":"2024-03-04T09:25:07.089049+0000","last_in_change":"2024-03-04T09:24:57.982077+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T09:25:07.935198+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-04T09:26:28.556219+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-04T09:26:29.471227+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a96574f-81f1-4bc6-8463-757520b84a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6825","nonce":1364449401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6827","nonce":1364449401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6831","nonce":1364449401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6829","nonce":1364449401}]},"public_addr":"172.21.15.6:6825/1364449401","cluster_addr":"172.21.15.6:6827/1364449401","heartbeat_back_addr":"172.21.15.6:6831/1364449401","heartbeat_front_addr":"172.21.15.6:6829/1364449401","state":["exists","up"]},{"osd":1,"uuid":"7d79b9c1-6b87-476a-bc36-7db5ee64cd90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6809","nonce":2494103078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6811","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6813","nonce":2494103078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6821","nonce":2494103078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6817","nonce":2494103078}]},"public_addr":"172.21.15.6:6809/2494103078","cluster_addr":"172.21.15.6:6813/2494103078","heartbeat_back_addr":"172.21.15.6:6821/2494103078","heartbeat_front_addr":"172.21.15.6:6817/2494103078","state":["exists","up"]},{"osd":2,"uuid":"492331b9-4ee9-40b6-bacd-85d44cd6c45a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6812","nonce":1500066681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6816","nonce":1500066681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6824","nonce":1500066681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6820","nonce":1500066681}]},"public_addr":"172.21.15.6:6812/1500066681","cluster_addr":"172.21.15.6:6816/1500066681","heartbeat_back_addr":"172.21.15.6:6824/1500066681","heartbeat_front_addr":"172.21.15.6:6820/1500066681","state":["exists","up"]},{"osd":3,"uuid":"21bb2efe-9ba7-4bdc-bc31-180cd1102e21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6801","nonce":1558806123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6803","nonce":1558806123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6807","nonce":1558806123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6805","nonce":1558806123}]},"public_addr":"172.21.15.6:6801/1558806123","cluster_addr":"172.21.15.6:6803/1558806123","heartbeat_back_addr":"172.21.15.6:6807/1558806123","heartbeat_front_addr":"172.21.15.6:6805/1558806123","state":["exists","up"]},{"osd":4,"uuid":"6e2c4b71-19d9-4737-83d9-558705d8644a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6801","nonce":1526135780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6803","nonce":1526135780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6807","nonce":1526135780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6805","nonce":1526135780}]},"public_addr":"172.21.15.146:6801/1526135780","cluster_addr":"172.21.15.146:6803/1526135780","heartbeat_back_addr":"172.21.15.146:6807/1526135780","heartbeat_front_addr":"172.21.15.146:6805/1526135780","state":["exists","up"]},{"osd":5,"uuid":"fb69c525-f1be-4b7d-ab32-6c29b0d6fddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6809","nonce":460262507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6811","nonce":460262507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6815","nonce":460262507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6813","nonce":460262507}]},"public_addr":"172.21.15.146:6809/460262507","cluster_addr":"172.21.15.146:6811/460262507","heartbeat_back_addr":"172.21.15.146:6815/460262507","heartbeat_front_addr":"172.21.15.146:6813/460262507","state":["exists","up"]},{"osd":6,"uuid":"c32e7fed-370d-4401-8ce0-6a4335b0d168","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6823","nonce":1392897493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6827","nonce":1392897493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6831","nonce":1392897493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6829","nonce":1392897493}]},"public_addr":"172.21.15.146:6823/1392897493","cluster_addr":"172.21.15.146:6827/1392897493","heartbeat_back_addr":"172.21.15.146:6831/1392897493","heartbeat_front_addr":"172.21.15.146:6829/1392897493","state":["exists","up"]},{"osd":7,"uuid":"75a79b44-fa5f-426d-a7d3-71f21f78835c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6817","nonce":3268323156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6819","nonce":3268323156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6825","nonce":3268323156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6822","nonce":3268323156}]},"public_addr":"172.21.15.146:6817/3268323156","cluster_addr":"172.21.15.146:6819/3268323156","heartbeat_back_addr":"172.21.15.146:6825/3268323156","heartbeat_front_addr":"172.21.15.146:6822/3268323156","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.310437+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.295850+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.294674+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.526855+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.300023+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.595160+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.346046+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.558741+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T09:27:01.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.853+0000 7fb4f6200640 1 -- 172.21.15.6:0/2799461043 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb4cc06b4d0 msgr2=0x7fb4cc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:01.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.853+0000 7fb4f6200640 1 --2- 172.21.15.6:0/2799461043 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb4cc06b4d0 0x7fb4cc06d990 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fb4d8007fd0 tx=0x7fb4d8009040 comp rx=0 tx=0).stop 2024-03-04T09:27:01.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.853+0000 7fb4f6200640 1 -- 172.21.15.6:0/2799461043 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb4f00cd060 msgr2=0x7fb4f0147720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:01.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.853+0000 7fb4f6200640 1 --2- 172.21.15.6:0/2799461043 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb4f00cd060 0x7fb4f0147720 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fb4e0009870 tx=0x7fb4e00046c0 comp rx=0 tx=0).stop 2024-03-04T09:27:01.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.853+0000 7fb4f6200640 1 -- 172.21.15.6:0/2799461043 shutdown_connections 2024-03-04T09:27:01.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.853+0000 7fb4f6200640 1 --2- 172.21.15.6:0/2799461043 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb4cc06b4d0 0x7fb4cc06d990 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:01.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.853+0000 7fb4f6200640 1 --2- 172.21.15.6:0/2799461043 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb4f015e810 0x7fb4f014dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:01.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.853+0000 7fb4f6200640 1 --2- 172.21.15.6:0/2799461043 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb4f01552b0 0x7fb4f014d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:01.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.853+0000 7fb4f6200640 1 --2- 172.21.15.6:0/2799461043 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb4f00cd060 0x7fb4f0147720 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:01.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.853+0000 7fb4f6200640 1 -- 172.21.15.6:0/2799461043 >> 172.21.15.6:0/2799461043 conn(0x7fb4f006b3d0 msgr2=0x7fb4f00cde70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:01.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.853+0000 7fb4f6200640 1 -- 172.21.15.6:0/2799461043 shutdown_connections 2024-03-04T09:27:01.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:01.853+0000 7fb4f6200640 1 -- 172.21.15.6:0/2799461043 wait complete. 2024-03-04T09:27:01.869 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-04T09:27:01.869 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:27:02.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.084+0000 7fa1ffe00640 1 Processor -- start 2024-03-04T09:27:02.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.085+0000 7fa1ffe00640 1 -- start start 2024-03-04T09:27:02.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.086+0000 7fa1ffe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa20015efc0 0x7fa2000cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:02.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.087+0000 7fa1ffe00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2001627f0 con 0x7fa2000cce90 2024-03-04T09:27:02.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.087+0000 7fa1ffe00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa20006c860 con 0x7fa2001621d0 2024-03-04T09:27:02.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.087+0000 7fa1ffe00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa20015f3a0 con 0x7fa20015efc0 2024-03-04T09:27:02.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.087+0000 7fa1fea00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa20015efc0 0x7fa2000cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:02.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.087+0000 7fa1f7e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fa2000cce90 0x7fa200161ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:39004/0 (socket says 172.21.15.6:39004) 2024-03-04T09:27:02.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.087+0000 7fa1f7e00640 1 -- 172.21.15.6:0/1957249010 learned_addr learned my addr 172.21.15.6:0/1957249010 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:27:02.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.087+0000 7fa1fea00640 1 -- 172.21.15.6:0/1957249010 >> v1:172.21.15.146:6789/0 conn(0x7fa2001621d0 legacy=0x7fa200070c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:27:02.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.087+0000 7fa1fea00640 1 -- 172.21.15.6:0/1957249010 >> v1:172.21.15.6:6789/0 conn(0x7fa2000cce90 legacy=0x7fa200161ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:02.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.087+0000 7fa1fea00640 1 -- 172.21.15.6:0/1957249010 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa20006cb40 con 0x7fa20015efc0 2024-03-04T09:27:02.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.089+0000 7fa1fea00640 1 --2- 172.21.15.6:0/1957249010 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa20015efc0 0x7fa2000cc950 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fa1f00097a0 tx=0x7fa1f002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bdd24123761ecea9 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:02.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.089+0000 7fa1fe000640 1 -- 172.21.15.6:0/1957249010 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1f0035020 con 0x7fa20015efc0 2024-03-04T09:27:02.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.089+0000 7fa1fe000640 1 -- 172.21.15.6:0/1957249010 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1f0031830 con 0x7fa20015efc0 2024-03-04T09:27:02.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.089+0000 7fa1fe000640 1 -- 172.21.15.6:0/1957249010 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1f0031c00 con 0x7fa20015efc0 2024-03-04T09:27:02.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.089+0000 7fa1ffe00640 1 -- 172.21.15.6:0/1957249010 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa20015efc0 msgr2=0x7fa2000cc950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:02.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.089+0000 7fa1ffe00640 1 --2- 172.21.15.6:0/1957249010 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa20015efc0 0x7fa2000cc950 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fa1f00097a0 tx=0x7fa1f002b8d0 comp rx=0 tx=0).stop 2024-03-04T09:27:02.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.090+0000 7fa1ffe00640 1 -- 172.21.15.6:0/1957249010 shutdown_connections 2024-03-04T09:27:02.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.090+0000 7fa1ffe00640 1 --2- 172.21.15.6:0/1957249010 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa20015efc0 0x7fa2000cc950 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:02.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.090+0000 7fa1ffe00640 1 -- 172.21.15.6:0/1957249010 >> 172.21.15.6:0/1957249010 conn(0x7fa20006b3d0 msgr2=0x7fa20006b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:02.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.090+0000 7fa1ffe00640 1 -- 172.21.15.6:0/1957249010 shutdown_connections 2024-03-04T09:27:02.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.090+0000 7fa1ffe00640 1 -- 172.21.15.6:0/1957249010 wait complete. 2024-03-04T09:27:02.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.091+0000 7fa1ffe00640 1 Processor -- start 2024-03-04T09:27:02.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.092+0000 7fa1ffe00640 1 -- start start 2024-03-04T09:27:02.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.092+0000 7fa1ffe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa2000cce90 0x7fa2000b4670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:02.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.092+0000 7fa1ffe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa20015efc0 0x7fa2000b3af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:02.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.092+0000 7fa1ffe00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa2001621d0 0x7fa20013a330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:02.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.093+0000 7fa1ffe00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa20006d620 con 0x7fa20015efc0 2024-03-04T09:27:02.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.093+0000 7fa1ffe00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa20006d190 con 0x7fa2000cce90 2024-03-04T09:27:02.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.093+0000 7fa1ffe00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2000b28b0 con 0x7fa2001621d0 2024-03-04T09:27:02.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.093+0000 7fa1f7e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa20015efc0 0x7fa2000b3af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:02.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.093+0000 7fa1f7e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa20015efc0 0x7fa2000b3af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49672/0 (socket says 172.21.15.6:49672) 2024-03-04T09:27:02.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.093+0000 7fa1f7e00640 1 -- 172.21.15.6:0/3328908567 learned_addr learned my addr 172.21.15.6:0/3328908567 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:27:02.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.093+0000 7fa1fea00640 1 --2- 172.21.15.6:0/3328908567 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa2000cce90 0x7fa2000b4670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:02.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.093+0000 7fa1f7e00640 1 -- 172.21.15.6:0/3328908567 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa2001621d0 msgr2=0x7fa20013a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:02.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.093+0000 7fa1ff400640 1 --2- 172.21.15.6:0/3328908567 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa2001621d0 0x7fa20013a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:02.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.093+0000 7fa1f7e00640 1 --2- 172.21.15.6:0/3328908567 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa2001621d0 0x7fa20013a330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:02.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.093+0000 7fa1f7e00640 1 -- 172.21.15.6:0/3328908567 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa2000cce90 msgr2=0x7fa2000b4670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:02.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.093+0000 7fa1f7e00640 1 --2- 172.21.15.6:0/3328908567 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa2000cce90 0x7fa2000b4670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:02.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.093+0000 7fa1f7e00640 1 -- 172.21.15.6:0/3328908567 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2000b2b00 con 0x7fa20015efc0 2024-03-04T09:27:02.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.093+0000 7fa1fea00640 1 --2- 172.21.15.6:0/3328908567 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa2000cce90 0x7fa2000b4670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:27:02.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.093+0000 7fa1ff400640 1 --2- 172.21.15.6:0/3328908567 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa2001621d0 0x7fa20013a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:27:02.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.094+0000 7fa1f7e00640 1 --2- 172.21.15.6:0/3328908567 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa20015efc0 0x7fa2000b3af0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fa1e800c400 tx=0x7fa1e800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:02.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.094+0000 7fa1f6a00640 1 -- 172.21.15.6:0/3328908567 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1e8017070 con 0x7fa20015efc0 2024-03-04T09:27:02.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.094+0000 7fa1f6a00640 1 -- 172.21.15.6:0/3328908567 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1e8005ce0 con 0x7fa20015efc0 2024-03-04T09:27:02.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.094+0000 7fa1f6a00640 1 -- 172.21.15.6:0/3328908567 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1e80072f0 con 0x7fa20015efc0 2024-03-04T09:27:02.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.094+0000 7fa1ffe00640 1 -- 172.21.15.6:0/3328908567 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2000b2de0 con 0x7fa20015efc0 2024-03-04T09:27:02.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.094+0000 7fa1ffe00640 1 -- 172.21.15.6:0/3328908567 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2000b33a0 con 0x7fa20015efc0 2024-03-04T09:27:02.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.095+0000 7fa1ffe00640 1 -- 172.21.15.6:0/3328908567 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa20015c3b0 con 0x7fa20015efc0 2024-03-04T09:27:02.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.096+0000 7fa1f6a00640 1 -- 172.21.15.6:0/3328908567 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa1e801a070 con 0x7fa20015efc0 2024-03-04T09:27:02.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.097+0000 7fa1f6a00640 1 --2- 172.21.15.6:0/3328908567 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa1dc06b4d0 0x7fa1dc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:02.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.097+0000 7fa1f6a00640 1 -- 172.21.15.6:0/3328908567 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa1e80896e0 con 0x7fa20015efc0 2024-03-04T09:27:02.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.098+0000 7fa1fea00640 1 --2- 172.21.15.6:0/3328908567 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa1dc06b4d0 0x7fa1dc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:02.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.099+0000 7fa1fea00640 1 --2- 172.21.15.6:0/3328908567 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa1dc06b4d0 0x7fa1dc06d990 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fa1f0036000 tx=0x7fa1f0008030 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:02.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.102+0000 7fa1f6a00640 1 -- 172.21.15.6:0/3328908567 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa1e80584c0 con 0x7fa20015efc0 2024-03-04T09:27:02.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.344+0000 7fa1ffe00640 1 -- 172.21.15.6:0/3328908567 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa20004f330 con 0x7fa20015efc0 2024-03-04T09:27:02.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.345+0000 7fa1f6a00640 1 -- 172.21.15.6:0/3328908567 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+4977 (secure 0 0 0) 0x7fa1e8057dc0 con 0x7fa20015efc0 2024-03-04T09:27:02.342 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:27:02.342 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4685,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/1703428647","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":1703428647},{"type":"v1","addr":"172.21.15.146:6837","nonce":1703428647}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4837,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/2617429530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2617429530},{"type":"v1","addr":"172.21.15.6:6833","nonce":2617429530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":5043,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/2744913857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":2744913857},{"type":"v1","addr":"172.21.15.6:6835","nonce":2744913857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:26:30.445199+0000","modified":"2024-03-04T09:26:33.534893+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5046},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5046":{"gid":5046,"name":"d","rank":0,"incarnation":12,"state":"up:active","state_seq":4,"addr":"172.21.15.146:6836/2285803221","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":2285803221},{"type":"v1","addr":"172.21.15.146:6836","nonce":2285803221}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5046,"qdb_cluster":[5046]},"id":2}]} 2024-03-04T09:27:02.342 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 16 2024-03-04T09:27:02.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.348+0000 7fa1ffe00640 1 -- 172.21.15.6:0/3328908567 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa1dc06b4d0 msgr2=0x7fa1dc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:02.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.348+0000 7fa1ffe00640 1 --2- 172.21.15.6:0/3328908567 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa1dc06b4d0 0x7fa1dc06d990 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fa1f0036000 tx=0x7fa1f0008030 comp rx=0 tx=0).stop 2024-03-04T09:27:02.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.348+0000 7fa1ffe00640 1 -- 172.21.15.6:0/3328908567 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa20015efc0 msgr2=0x7fa2000b3af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:02.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.348+0000 7fa1ffe00640 1 --2- 172.21.15.6:0/3328908567 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa20015efc0 0x7fa2000b3af0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fa1e800c400 tx=0x7fa1e800c8c0 comp rx=0 tx=0).stop 2024-03-04T09:27:02.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.348+0000 7fa1ffe00640 1 -- 172.21.15.6:0/3328908567 shutdown_connections 2024-03-04T09:27:02.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.348+0000 7fa1ffe00640 1 --2- 172.21.15.6:0/3328908567 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa1dc06b4d0 0x7fa1dc06d990 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:02.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.348+0000 7fa1ffe00640 1 --2- 172.21.15.6:0/3328908567 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa2001621d0 0x7fa20013a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:02.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.348+0000 7fa1ffe00640 1 --2- 172.21.15.6:0/3328908567 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa20015efc0 0x7fa2000b3af0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:02.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.348+0000 7fa1ffe00640 1 --2- 172.21.15.6:0/3328908567 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa2000cce90 0x7fa2000b4670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:02.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.348+0000 7fa1ffe00640 1 -- 172.21.15.6:0/3328908567 >> 172.21.15.6:0/3328908567 conn(0x7fa20006b3d0 msgr2=0x7fa2000b0c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:02.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.348+0000 7fa1ffe00640 1 -- 172.21.15.6:0/3328908567 shutdown_connections 2024-03-04T09:27:02.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.348+0000 7fa1ffe00640 1 -- 172.21.15.6:0/3328908567 wait complete. 2024-03-04T09:27:02.364 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:27:02.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.591+0000 7fac99e00640 1 Processor -- start 2024-03-04T09:27:02.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.591+0000 7fac99e00640 1 -- start start 2024-03-04T09:27:02.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.592+0000 7fac99e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fac940cd060 0x7fac941667e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:02.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.592+0000 7fac99e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac940714d0 con 0x7fac94145300 2024-03-04T09:27:02.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.592+0000 7fac99e00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac940717b0 con 0x7fac94070f10 2024-03-04T09:27:02.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.592+0000 7fac99e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac94071a90 con 0x7fac940cd060 2024-03-04T09:27:02.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.593+0000 7fac99400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fac940cd060 0x7fac941667e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:02.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.593+0000 7fac99400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fac940cd060 0x7fac941667e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49684/0 (socket says 172.21.15.6:49684) 2024-03-04T09:27:02.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.593+0000 7fac98a00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fac94145300 0x7fac940b49a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:39006/0 (socket says 172.21.15.6:39006) 2024-03-04T09:27:02.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.593+0000 7fac99400640 1 -- 172.21.15.6:0/207096196 learned_addr learned my addr 172.21.15.6:0/207096196 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:27:02.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.593+0000 7fac99400640 1 -- 172.21.15.6:0/207096196 >> v1:172.21.15.146:6789/0 conn(0x7fac94070f10 legacy=0x7fac940cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:27:02.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.593+0000 7fac99400640 1 -- 172.21.15.6:0/207096196 >> v1:172.21.15.6:6789/0 conn(0x7fac94145300 legacy=0x7fac940b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:02.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.593+0000 7fac93400640 1 -- 172.21.15.6:0/207096196 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1056070130 0 0) 0x7fac940714d0 con 0x7fac94145300 2024-03-04T09:27:02.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.593+0000 7fac99400640 1 -- 172.21.15.6:0/207096196 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac94071d70 con 0x7fac940cd060 2024-03-04T09:27:02.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.594+0000 7fac99400640 1 --2- 172.21.15.6:0/207096196 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fac940cd060 0x7fac941667e0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fac8c009e50 tx=0x7fac8c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8e59c8b717e7444 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:02.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.595+0000 7fac93400640 1 -- 172.21.15.6:0/207096196 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac8c02ca80 con 0x7fac940cd060 2024-03-04T09:27:02.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.595+0000 7fac93400640 1 -- 172.21.15.6:0/207096196 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac8c02cc20 con 0x7fac940cd060 2024-03-04T09:27:02.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.595+0000 7fac93400640 1 -- 172.21.15.6:0/207096196 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac8c0343f0 con 0x7fac940cd060 2024-03-04T09:27:02.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.595+0000 7fac99e00640 1 -- 172.21.15.6:0/207096196 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fac940cd060 msgr2=0x7fac941667e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:02.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.595+0000 7fac99e00640 1 --2- 172.21.15.6:0/207096196 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fac940cd060 0x7fac941667e0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fac8c009e50 tx=0x7fac8c02bee0 comp rx=0 tx=0).stop 2024-03-04T09:27:02.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.595+0000 7fac99e00640 1 -- 172.21.15.6:0/207096196 shutdown_connections 2024-03-04T09:27:02.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.595+0000 7fac99e00640 1 --2- 172.21.15.6:0/207096196 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fac940cd060 0x7fac941667e0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:02.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.595+0000 7fac99e00640 1 -- 172.21.15.6:0/207096196 >> 172.21.15.6:0/207096196 conn(0x7fac9406b3d0 msgr2=0x7fac9406b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:02.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.595+0000 7fac99e00640 1 -- 172.21.15.6:0/207096196 shutdown_connections 2024-03-04T09:27:02.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.595+0000 7fac99e00640 1 -- 172.21.15.6:0/207096196 wait complete. 2024-03-04T09:27:02.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.597+0000 7fac99e00640 1 Processor -- start 2024-03-04T09:27:02.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.597+0000 7fac99e00640 1 -- start start 2024-03-04T09:27:02.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.598+0000 7fac99e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fac94070f10 0x7fac94141540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:02.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.598+0000 7fac99e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fac940cd060 0x7fac9413ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:02.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.598+0000 7fac99e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fac94145300 0x7fac941404e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:02.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.598+0000 7fac99e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac9415d700 con 0x7fac94070f10 2024-03-04T09:27:02.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.598+0000 7fac99e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac9415d580 con 0x7fac940cd060 2024-03-04T09:27:02.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.598+0000 7fac99e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac9415d880 con 0x7fac94145300 2024-03-04T09:27:02.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.598+0000 7fac98a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fac94070f10 0x7fac94141540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:02.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.599+0000 7fac98a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fac94070f10 0x7fac94141540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49688/0 (socket says 172.21.15.6:49688) 2024-03-04T09:27:02.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.599+0000 7fac98a00640 1 -- 172.21.15.6:0/2369743607 learned_addr learned my addr 172.21.15.6:0/2369743607 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:27:02.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.599+0000 7fac93e00640 1 --2- 172.21.15.6:0/2369743607 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fac940cd060 0x7fac9413ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:02.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.599+0000 7fac98a00640 1 -- 172.21.15.6:0/2369743607 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fac94145300 msgr2=0x7fac941404e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:02.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.599+0000 7fac98a00640 1 --2- 172.21.15.6:0/2369743607 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fac94145300 0x7fac941404e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:02.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.599+0000 7fac98a00640 1 -- 172.21.15.6:0/2369743607 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fac940cd060 msgr2=0x7fac9413ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:02.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.599+0000 7fac98a00640 1 --2- 172.21.15.6:0/2369743607 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fac940cd060 0x7fac9413ffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:02.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.599+0000 7fac98a00640 1 -- 172.21.15.6:0/2369743607 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac9413d4b0 con 0x7fac94070f10 2024-03-04T09:27:02.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.599+0000 7fac98a00640 1 --2- 172.21.15.6:0/2369743607 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fac94070f10 0x7fac94141540 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fac8800c430 tx=0x7fac8800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:02.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.600+0000 7fac90c00640 1 -- 172.21.15.6:0/2369743607 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac880074e0 con 0x7fac94070f10 2024-03-04T09:27:02.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.600+0000 7fac90c00640 1 -- 172.21.15.6:0/2369743607 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac880047e0 con 0x7fac94070f10 2024-03-04T09:27:02.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.600+0000 7fac90c00640 1 -- 172.21.15.6:0/2369743607 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac88004bd0 con 0x7fac94070f10 2024-03-04T09:27:02.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.600+0000 7fac99e00640 1 -- 172.21.15.6:0/2369743607 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac940aeb70 con 0x7fac94070f10 2024-03-04T09:27:02.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.600+0000 7fac99e00640 1 -- 172.21.15.6:0/2369743607 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac9413da50 con 0x7fac94070f10 2024-03-04T09:27:02.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.601+0000 7fac99e00640 1 -- 172.21.15.6:0/2369743607 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac9404f330 con 0x7fac94070f10 2024-03-04T09:27:02.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.602+0000 7fac90c00640 1 -- 172.21.15.6:0/2369743607 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fac88011070 con 0x7fac94070f10 2024-03-04T09:27:02.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.603+0000 7fac90c00640 1 --2- 172.21.15.6:0/2369743607 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fac7406b4d0 0x7fac7406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:02.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.603+0000 7fac90c00640 1 -- 172.21.15.6:0/2369743607 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fac88089ed0 con 0x7fac94070f10 2024-03-04T09:27:02.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.604+0000 7fac93e00640 1 --2- 172.21.15.6:0/2369743607 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fac7406b4d0 0x7fac7406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:02.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.605+0000 7fac93e00640 1 --2- 172.21.15.6:0/2369743607 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fac7406b4d0 0x7fac7406d990 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fac80007fd0 tx=0x7fac80009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:02.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.608+0000 7fac90c00640 1 -- 172.21.15.6:0/2369743607 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fac88058cb0 con 0x7fac94070f10 2024-03-04T09:27:02.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.847+0000 7fac99e00640 1 -- 172.21.15.6:0/2369743607 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fac9413d2d0 con 0x7fac94070f10 2024-03-04T09:27:02.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.848+0000 7fac90c00640 1 -- 172.21.15.6:0/2369743607 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+4977 (secure 0 0 0) 0x7fac9413d2d0 con 0x7fac94070f10 2024-03-04T09:27:02.845 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:27:02.845 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4685,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/1703428647","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":1703428647},{"type":"v1","addr":"172.21.15.146:6837","nonce":1703428647}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4837,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/2617429530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2617429530},{"type":"v1","addr":"172.21.15.6:6833","nonce":2617429530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":5043,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/2744913857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":2744913857},{"type":"v1","addr":"172.21.15.6:6835","nonce":2744913857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:26:30.445199+0000","modified":"2024-03-04T09:26:33.534893+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5046},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5046":{"gid":5046,"name":"d","rank":0,"incarnation":12,"state":"up:active","state_seq":4,"addr":"172.21.15.146:6836/2285803221","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":2285803221},{"type":"v1","addr":"172.21.15.146:6836","nonce":2285803221}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5046,"qdb_cluster":[5046]},"id":2}]} 2024-03-04T09:27:02.845 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 16 2024-03-04T09:27:02.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.851+0000 7fac99e00640 1 -- 172.21.15.6:0/2369743607 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fac7406b4d0 msgr2=0x7fac7406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:02.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.851+0000 7fac99e00640 1 --2- 172.21.15.6:0/2369743607 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fac7406b4d0 0x7fac7406d990 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fac80007fd0 tx=0x7fac80009040 comp rx=0 tx=0).stop 2024-03-04T09:27:02.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.851+0000 7fac99e00640 1 -- 172.21.15.6:0/2369743607 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fac94070f10 msgr2=0x7fac94141540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:02.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.851+0000 7fac99e00640 1 --2- 172.21.15.6:0/2369743607 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fac94070f10 0x7fac94141540 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fac8800c430 tx=0x7fac8800c8f0 comp rx=0 tx=0).stop 2024-03-04T09:27:02.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.851+0000 7fac99e00640 1 -- 172.21.15.6:0/2369743607 shutdown_connections 2024-03-04T09:27:02.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.851+0000 7fac99e00640 1 --2- 172.21.15.6:0/2369743607 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fac7406b4d0 0x7fac7406d990 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:02.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.851+0000 7fac99e00640 1 --2- 172.21.15.6:0/2369743607 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fac94145300 0x7fac941404e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:02.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.851+0000 7fac99e00640 1 --2- 172.21.15.6:0/2369743607 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fac940cd060 0x7fac9413ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:02.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.851+0000 7fac99e00640 1 --2- 172.21.15.6:0/2369743607 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fac94070f10 0x7fac94141540 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:02.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.851+0000 7fac99e00640 1 -- 172.21.15.6:0/2369743607 >> 172.21.15.6:0/2369743607 conn(0x7fac9406b3d0 msgr2=0x7fac9415da00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:02.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.851+0000 7fac99e00640 1 -- 172.21.15.6:0/2369743607 shutdown_connections 2024-03-04T09:27:02.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:02.851+0000 7fac99e00640 1 -- 172.21.15.6:0/2369743607 wait complete. 2024-03-04T09:27:02.869 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T09:27:03.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.095+0000 7faeb8a00640 1 Processor -- start 2024-03-04T09:27:03.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.095+0000 7faeb8a00640 1 -- start start 2024-03-04T09:27:03.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.097+0000 7faeb8a00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7faeb40af1a0 0x7faeb415c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:03.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.097+0000 7faeb8a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7faeb40cdc20 0x7faeb40b2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:03.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.097+0000 7faeb8a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faeb415cc40 con 0x7faeb40af580 2024-03-04T09:27:03.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.097+0000 7faeb8a00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faeb415cf20 con 0x7faeb40cdc20 2024-03-04T09:27:03.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.097+0000 7faeb8a00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faeb415d200 con 0x7faeb40af1a0 2024-03-04T09:27:03.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.097+0000 7faeb2800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7faeb40cdc20 0x7faeb40b2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:03.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.097+0000 7faeb2800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7faeb40cdc20 0x7faeb40b2dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49690/0 (socket says 172.21.15.6:49690) 2024-03-04T09:27:03.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.097+0000 7faea9400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7faeb40af580 0x7faeb40b4c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:39010/0 (socket says 172.21.15.6:39010) 2024-03-04T09:27:03.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.098+0000 7faeb2800640 1 -- 172.21.15.6:0/2203832122 learned_addr learned my addr 172.21.15.6:0/2203832122 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:27:03.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.098+0000 7faeb1e00640 1 --2- 172.21.15.6:0/2203832122 >> v2:172.21.15.146:3300/0 conn(0x7faeb40af1a0 0x7faeb415c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:03.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.098+0000 7faeb2800640 1 -- 172.21.15.6:0/2203832122 >> v2:172.21.15.146:3300/0 conn(0x7faeb40af1a0 msgr2=0x7faeb415c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:27:03.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.098+0000 7faeb2800640 1 --2- 172.21.15.6:0/2203832122 >> v2:172.21.15.146:3300/0 conn(0x7faeb40af1a0 0x7faeb415c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:03.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.098+0000 7faeb2800640 1 -- 172.21.15.6:0/2203832122 >> v1:172.21.15.6:6789/0 conn(0x7faeb40af580 legacy=0x7faeb40b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:03.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.098+0000 7faeb2800640 1 -- 172.21.15.6:0/2203832122 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faeb415d4e0 con 0x7faeb40cdc20 2024-03-04T09:27:03.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.099+0000 7faeb2800640 1 --2- 172.21.15.6:0/2203832122 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faeb40cdc20 0x7faeb40b2dd0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7faeb4074960 tx=0x7faeac02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=373bf144a3cac8e7 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:03.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.099+0000 7faeb1400640 1 -- 172.21.15.6:0/2203832122 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faeac02ca80 con 0x7faeb40cdc20 2024-03-04T09:27:03.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.099+0000 7faeb1400640 1 -- 172.21.15.6:0/2203832122 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faeac02cc20 con 0x7faeb40cdc20 2024-03-04T09:27:03.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.099+0000 7faeb1400640 1 -- 172.21.15.6:0/2203832122 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faeac0343f0 con 0x7faeb40cdc20 2024-03-04T09:27:03.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.100+0000 7faeb8a00640 1 -- 172.21.15.6:0/2203832122 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faeb40cdc20 msgr2=0x7faeb40b2dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:03.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.100+0000 7faeb8a00640 1 --2- 172.21.15.6:0/2203832122 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faeb40cdc20 0x7faeb40b2dd0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7faeb4074960 tx=0x7faeac02bee0 comp rx=0 tx=0).stop 2024-03-04T09:27:03.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.100+0000 7faeb8a00640 1 -- 172.21.15.6:0/2203832122 shutdown_connections 2024-03-04T09:27:03.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.100+0000 7faeb8a00640 1 --2- 172.21.15.6:0/2203832122 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faeb40cdc20 0x7faeb40b2dd0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:03.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.100+0000 7faeb8a00640 1 --2- 172.21.15.6:0/2203832122 >> v2:172.21.15.146:3300/0 conn(0x7faeb40af1a0 0x7faeb415c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:03.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.100+0000 7faeb8a00640 1 -- 172.21.15.6:0/2203832122 >> 172.21.15.6:0/2203832122 conn(0x7faeb406afb0 msgr2=0x7faeb406b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:03.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.100+0000 7faeb8a00640 1 -- 172.21.15.6:0/2203832122 shutdown_connections 2024-03-04T09:27:03.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.100+0000 7faeb8a00640 1 -- 172.21.15.6:0/2203832122 wait complete. 2024-03-04T09:27:03.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.102+0000 7faeb8a00640 1 Processor -- start 2024-03-04T09:27:03.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.102+0000 7faeb8a00640 1 -- start start 2024-03-04T09:27:03.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.103+0000 7faeb8a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faeb40af1a0 0x7faeb40b0610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:03.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.103+0000 7faeb8a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faeb40af580 0x7faeb41f4510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:03.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.103+0000 7faeb8a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faeb40cdc20 0x7faeb406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:03.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.103+0000 7faeb8a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faeb40b36a0 con 0x7faeb40cdc20 2024-03-04T09:27:03.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.103+0000 7faeb8a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faeb40b3520 con 0x7faeb40af580 2024-03-04T09:27:03.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.103+0000 7faeb8a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faeb40b3820 con 0x7faeb40af1a0 2024-03-04T09:27:03.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.104+0000 7faeb2800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faeb40cdc20 0x7faeb406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:03.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.104+0000 7faeb2800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faeb40cdc20 0x7faeb406e2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49702/0 (socket says 172.21.15.6:49702) 2024-03-04T09:27:03.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.104+0000 7faeb2800640 1 -- 172.21.15.6:0/2258123195 learned_addr learned my addr 172.21.15.6:0/2258123195 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:27:03.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.104+0000 7faea9400640 1 --2- 172.21.15.6:0/2258123195 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faeb40af580 0x7faeb41f4510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:03.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.104+0000 7faeb1e00640 1 --2- 172.21.15.6:0/2258123195 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faeb40af1a0 0x7faeb40b0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:03.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.104+0000 7faeb2800640 1 -- 172.21.15.6:0/2258123195 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faeb40af1a0 msgr2=0x7faeb40b0610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:03.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.104+0000 7faeb2800640 1 --2- 172.21.15.6:0/2258123195 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faeb40af1a0 0x7faeb40b0610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:03.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.104+0000 7faeb2800640 1 -- 172.21.15.6:0/2258123195 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faeb40af580 msgr2=0x7faeb41f4510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:03.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.104+0000 7faeb2800640 1 --2- 172.21.15.6:0/2258123195 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faeb40af580 0x7faeb41f4510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:03.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.104+0000 7faeb2800640 1 -- 172.21.15.6:0/2258123195 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faeb40b1700 con 0x7faeb40cdc20 2024-03-04T09:27:03.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.104+0000 7faeb2800640 1 --2- 172.21.15.6:0/2258123195 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faeb40cdc20 0x7faeb406e2f0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7faeac036040 tx=0x7faeac004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:03.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.105+0000 7faeaaa00640 1 -- 172.21.15.6:0/2258123195 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faeac043020 con 0x7faeb40cdc20 2024-03-04T09:27:03.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.105+0000 7faeaaa00640 1 -- 172.21.15.6:0/2258123195 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faeac03e450 con 0x7faeb40cdc20 2024-03-04T09:27:03.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.105+0000 7faeaaa00640 1 -- 172.21.15.6:0/2258123195 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faeac009510 con 0x7faeb40cdc20 2024-03-04T09:27:03.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.105+0000 7faeb8a00640 1 -- 172.21.15.6:0/2258123195 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faeb40afb70 con 0x7faeb40cdc20 2024-03-04T09:27:03.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.105+0000 7faeb8a00640 1 -- 172.21.15.6:0/2258123195 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faeb40b1420 con 0x7faeb40cdc20 2024-03-04T09:27:03.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.106+0000 7faeb8a00640 1 -- 172.21.15.6:0/2258123195 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faeb404f3a0 con 0x7faeb40cdc20 2024-03-04T09:27:03.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.107+0000 7faeaaa00640 1 -- 172.21.15.6:0/2258123195 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7faeac046020 con 0x7faeb40cdc20 2024-03-04T09:27:03.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.109+0000 7faeaaa00640 1 --2- 172.21.15.6:0/2258123195 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fae9006b4d0 0x7fae9006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:03.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.109+0000 7faeaaa00640 1 -- 172.21.15.6:0/2258123195 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7faeac0aca60 con 0x7faeb40cdc20 2024-03-04T09:27:03.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.109+0000 7faeb1e00640 1 --2- 172.21.15.6:0/2258123195 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fae9006b4d0 0x7fae9006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:03.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.110+0000 7faeb1e00640 1 --2- 172.21.15.6:0/2258123195 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fae9006b4d0 0x7fae9006d990 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7faea4006fd0 tx=0x7faea4008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:03.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.114+0000 7faeaaa00640 1 -- 172.21.15.6:0/2258123195 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7faeac04d030 con 0x7faeb40cdc20 2024-03-04T09:27:03.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.316+0000 7faeb8a00640 1 -- 172.21.15.6:0/2258123195 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faeb406b530 con 0x7faeb40cdc20 2024-03-04T09:27:03.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.318+0000 7faeaaa00640 1 -- 172.21.15.6:0/2258123195 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v33) v1 ==== 74+0+16621 (secure 0 0 0) 0x7faeac07b8d0 con 0x7faeb40cdc20 2024-03-04T09:27:03.316 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:27:03.316 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":33,"fsid":"edce0578-5ae4-4402-b45a-552bdef84d7c","created":"2024-03-04T09:24:54.523780+0000","modified":"2024-03-04T09:26:31.991047+0000","last_up_change":"2024-03-04T09:25:07.089049+0000","last_in_change":"2024-03-04T09:24:57.982077+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T09:25:07.935198+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-04T09:26:28.556219+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-04T09:26:29.471227+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a96574f-81f1-4bc6-8463-757520b84a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6825","nonce":1364449401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6827","nonce":1364449401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6831","nonce":1364449401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6829","nonce":1364449401}]},"public_addr":"172.21.15.6:6825/1364449401","cluster_addr":"172.21.15.6:6827/1364449401","heartbeat_back_addr":"172.21.15.6:6831/1364449401","heartbeat_front_addr":"172.21.15.6:6829/1364449401","state":["exists","up"]},{"osd":1,"uuid":"7d79b9c1-6b87-476a-bc36-7db5ee64cd90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6809","nonce":2494103078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6811","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6813","nonce":2494103078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6821","nonce":2494103078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6817","nonce":2494103078}]},"public_addr":"172.21.15.6:6809/2494103078","cluster_addr":"172.21.15.6:6813/2494103078","heartbeat_back_addr":"172.21.15.6:6821/2494103078","heartbeat_front_addr":"172.21.15.6:6817/2494103078","state":["exists","up"]},{"osd":2,"uuid":"492331b9-4ee9-40b6-bacd-85d44cd6c45a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6812","nonce":1500066681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6816","nonce":1500066681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6824","nonce":1500066681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6820","nonce":1500066681}]},"public_addr":"172.21.15.6:6812/1500066681","cluster_addr":"172.21.15.6:6816/1500066681","heartbeat_back_addr":"172.21.15.6:6824/1500066681","heartbeat_front_addr":"172.21.15.6:6820/1500066681","state":["exists","up"]},{"osd":3,"uuid":"21bb2efe-9ba7-4bdc-bc31-180cd1102e21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6801","nonce":1558806123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6803","nonce":1558806123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6807","nonce":1558806123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6805","nonce":1558806123}]},"public_addr":"172.21.15.6:6801/1558806123","cluster_addr":"172.21.15.6:6803/1558806123","heartbeat_back_addr":"172.21.15.6:6807/1558806123","heartbeat_front_addr":"172.21.15.6:6805/1558806123","state":["exists","up"]},{"osd":4,"uuid":"6e2c4b71-19d9-4737-83d9-558705d8644a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6801","nonce":1526135780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6803","nonce":1526135780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6807","nonce":1526135780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6805","nonce":1526135780}]},"public_addr":"172.21.15.146:6801/1526135780","cluster_addr":"172.21.15.146:6803/1526135780","heartbeat_back_addr":"172.21.15.146:6807/1526135780","heartbeat_front_addr":"172.21.15.146:6805/1526135780","state":["exists","up"]},{"osd":5,"uuid":"fb69c525-f1be-4b7d-ab32-6c29b0d6fddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6809","nonce":460262507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6811","nonce":460262507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6815","nonce":460262507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6813","nonce":460262507}]},"public_addr":"172.21.15.146:6809/460262507","cluster_addr":"172.21.15.146:6811/460262507","heartbeat_back_addr":"172.21.15.146:6815/460262507","heartbeat_front_addr":"172.21.15.146:6813/460262507","state":["exists","up"]},{"osd":6,"uuid":"c32e7fed-370d-4401-8ce0-6a4335b0d168","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6823","nonce":1392897493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6827","nonce":1392897493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6831","nonce":1392897493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6829","nonce":1392897493}]},"public_addr":"172.21.15.146:6823/1392897493","cluster_addr":"172.21.15.146:6827/1392897493","heartbeat_back_addr":"172.21.15.146:6831/1392897493","heartbeat_front_addr":"172.21.15.146:6829/1392897493","state":["exists","up"]},{"osd":7,"uuid":"75a79b44-fa5f-426d-a7d3-71f21f78835c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6817","nonce":3268323156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6819","nonce":3268323156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6825","nonce":3268323156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6822","nonce":3268323156}]},"public_addr":"172.21.15.146:6817/3268323156","cluster_addr":"172.21.15.146:6819/3268323156","heartbeat_back_addr":"172.21.15.146:6825/3268323156","heartbeat_front_addr":"172.21.15.146:6822/3268323156","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.310437+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.295850+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.294674+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.526855+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.300023+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.595160+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.346046+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.558741+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T09:27:03.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.321+0000 7faeb8a00640 1 -- 172.21.15.6:0/2258123195 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fae9006b4d0 msgr2=0x7fae9006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:03.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.321+0000 7faeb8a00640 1 --2- 172.21.15.6:0/2258123195 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fae9006b4d0 0x7fae9006d990 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7faea4006fd0 tx=0x7faea4008040 comp rx=0 tx=0).stop 2024-03-04T09:27:03.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.321+0000 7faeb8a00640 1 -- 172.21.15.6:0/2258123195 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faeb40cdc20 msgr2=0x7faeb406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:03.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.321+0000 7faeb8a00640 1 --2- 172.21.15.6:0/2258123195 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faeb40cdc20 0x7faeb406e2f0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7faeac036040 tx=0x7faeac004a40 comp rx=0 tx=0).stop 2024-03-04T09:27:03.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.321+0000 7faeb8a00640 1 -- 172.21.15.6:0/2258123195 shutdown_connections 2024-03-04T09:27:03.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.321+0000 7faeb8a00640 1 --2- 172.21.15.6:0/2258123195 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faeb40cdc20 0x7faeb406e2f0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:03.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.321+0000 7faeb8a00640 1 --2- 172.21.15.6:0/2258123195 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faeb40af580 0x7faeb41f4510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:03.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.321+0000 7faeb8a00640 1 --2- 172.21.15.6:0/2258123195 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fae9006b4d0 0x7fae9006d990 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:03.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.321+0000 7faeb8a00640 1 --2- 172.21.15.6:0/2258123195 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faeb40af1a0 0x7faeb40b0610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:03.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.321+0000 7faeb8a00640 1 -- 172.21.15.6:0/2258123195 >> 172.21.15.6:0/2258123195 conn(0x7faeb406afb0 msgr2=0x7faeb40cecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:03.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.322+0000 7faeb8a00640 1 -- 172.21.15.6:0/2258123195 shutdown_connections 2024-03-04T09:27:03.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.322+0000 7faeb8a00640 1 -- 172.21.15.6:0/2258123195 wait complete. 2024-03-04T09:27:03.339 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-04T09:27:03.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.562+0000 7f3873200640 1 Processor -- start 2024-03-04T09:27:03.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.562+0000 7f3873200640 1 -- start start 2024-03-04T09:27:03.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.564+0000 7f3873200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f387415efc0 0x7f38740cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:03.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.564+0000 7f3873200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f38740cce90 0x7f3874161ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:03.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.564+0000 7f3873200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38741627f0 con 0x7f3874162000 2024-03-04T09:27:03.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.564+0000 7f3873200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f387415f3a0 con 0x7f387415efc0 2024-03-04T09:27:03.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.564+0000 7f3873200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f387406d190 con 0x7f38740cce90 2024-03-04T09:27:03.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.564+0000 7f3871e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f387415efc0 0x7f38740cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:03.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.564+0000 7f3871e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f387415efc0 0x7f38740cc950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49716/0 (socket says 172.21.15.6:49716) 2024-03-04T09:27:03.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.564+0000 7f3871e00640 1 -- 172.21.15.6:0/911062363 learned_addr learned my addr 172.21.15.6:0/911062363 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:27:03.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.564+0000 7f3872800640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f3874162000 0x7f38740ad880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:39022/0 (socket says 172.21.15.6:39022) 2024-03-04T09:27:03.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.564+0000 7f3871400640 1 --2- 172.21.15.6:0/911062363 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f38740cce90 0x7f3874161ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:03.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.564+0000 7f3871e00640 1 -- 172.21.15.6:0/911062363 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f38740cce90 msgr2=0x7f3874161ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:27:03.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.564+0000 7f3871e00640 1 --2- 172.21.15.6:0/911062363 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f38740cce90 0x7f3874161ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:03.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.564+0000 7f3871e00640 1 -- 172.21.15.6:0/911062363 >> v1:172.21.15.6:6789/0 conn(0x7f3874162000 legacy=0x7f38740ad880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:03.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.564+0000 7f3871e00640 1 -- 172.21.15.6:0/911062363 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f387406c860 con 0x7f387415efc0 2024-03-04T09:27:03.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.564+0000 7f3871400640 1 --2- 172.21.15.6:0/911062363 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f38740cce90 0x7f3874161ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:27:03.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.566+0000 7f3871e00640 1 --2- 172.21.15.6:0/911062363 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f387415efc0 0x7f38740cc950 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f3864009870 tx=0x7f386402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56f28f47417060f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:03.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.566+0000 7f3870a00640 1 -- 172.21.15.6:0/911062363 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3864034070 con 0x7f387415efc0 2024-03-04T09:27:03.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.566+0000 7f3870a00640 1 -- 172.21.15.6:0/911062363 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3864030840 con 0x7f387415efc0 2024-03-04T09:27:03.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.566+0000 7f3870a00640 1 -- 172.21.15.6:0/911062363 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3864030c10 con 0x7f387415efc0 2024-03-04T09:27:03.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.566+0000 7f3873200640 1 -- 172.21.15.6:0/911062363 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f387415efc0 msgr2=0x7f38740cc950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:03.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.566+0000 7f3873200640 1 --2- 172.21.15.6:0/911062363 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f387415efc0 0x7f38740cc950 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f3864009870 tx=0x7f386402b980 comp rx=0 tx=0).stop 2024-03-04T09:27:03.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.567+0000 7f3873200640 1 -- 172.21.15.6:0/911062363 shutdown_connections 2024-03-04T09:27:03.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.567+0000 7f3873200640 1 --2- 172.21.15.6:0/911062363 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f38740cce90 0x7f3874161ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:03.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.567+0000 7f3873200640 1 --2- 172.21.15.6:0/911062363 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f387415efc0 0x7f38740cc950 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:03.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.567+0000 7f3873200640 1 -- 172.21.15.6:0/911062363 >> 172.21.15.6:0/911062363 conn(0x7f387406b3d0 msgr2=0x7f387406b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:03.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.567+0000 7f3873200640 1 -- 172.21.15.6:0/911062363 shutdown_connections 2024-03-04T09:27:03.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.567+0000 7f3873200640 1 -- 172.21.15.6:0/911062363 wait complete. 2024-03-04T09:27:03.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.568+0000 7f3873200640 1 Processor -- start 2024-03-04T09:27:03.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.569+0000 7f3873200640 1 -- start start 2024-03-04T09:27:03.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.570+0000 7f3873200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38740cce90 0x7f3874140ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:03.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.570+0000 7f3873200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f387415efc0 0x7f3874143b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:03.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.570+0000 7f3873200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3874162000 0x7f38740b3d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:03.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.570+0000 7f3873200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3874071c30 con 0x7f38740cce90 2024-03-04T09:27:03.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.570+0000 7f3873200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f387406ce20 con 0x7f3874162000 2024-03-04T09:27:03.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.570+0000 7f3873200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3874162490 con 0x7f387415efc0 2024-03-04T09:27:03.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.570+0000 7f3871e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38740cce90 0x7f3874140ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:03.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.570+0000 7f3871e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38740cce90 0x7f3874140ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49726/0 (socket says 172.21.15.6:49726) 2024-03-04T09:27:03.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.570+0000 7f3871e00640 1 -- 172.21.15.6:0/4118960661 learned_addr learned my addr 172.21.15.6:0/4118960661 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:27:03.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.570+0000 7f3871400640 1 --2- 172.21.15.6:0/4118960661 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f387415efc0 0x7f3874143b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:03.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.570+0000 7f3871e00640 1 -- 172.21.15.6:0/4118960661 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f387415efc0 msgr2=0x7f3874143b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:03.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.570+0000 7f3871e00640 1 --2- 172.21.15.6:0/4118960661 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f387415efc0 0x7f3874143b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:03.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.570+0000 7f3871e00640 1 -- 172.21.15.6:0/4118960661 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3874162000 msgr2=0x7f38740b3d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:03.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.570+0000 7f3872800640 1 --2- 172.21.15.6:0/4118960661 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3874162000 0x7f38740b3d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:03.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.570+0000 7f3871e00640 1 --2- 172.21.15.6:0/4118960661 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3874162000 0x7f38740b3d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:03.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.570+0000 7f3871e00640 1 -- 172.21.15.6:0/4118960661 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38741445d0 con 0x7f38740cce90 2024-03-04T09:27:03.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.570+0000 7f3871400640 1 --2- 172.21.15.6:0/4118960661 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f387415efc0 0x7f3874143b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:27:03.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.571+0000 7f3871e00640 1 --2- 172.21.15.6:0/4118960661 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38740cce90 0x7f3874140ec0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f3864009870 tx=0x7f3864004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:03.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.571+0000 7f386a000640 1 -- 172.21.15.6:0/4118960661 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3864048070 con 0x7f38740cce90 2024-03-04T09:27:03.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.571+0000 7f386a000640 1 -- 172.21.15.6:0/4118960661 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3864030640 con 0x7f38740cce90 2024-03-04T09:27:03.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.571+0000 7f386a000640 1 -- 172.21.15.6:0/4118960661 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3864034040 con 0x7f38740cce90 2024-03-04T09:27:03.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.571+0000 7f3873200640 1 -- 172.21.15.6:0/4118960661 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38741f3c20 con 0x7f38740cce90 2024-03-04T09:27:03.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.571+0000 7f3873200640 1 -- 172.21.15.6:0/4118960661 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38741f41e0 con 0x7f38740cce90 2024-03-04T09:27:03.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.572+0000 7f3873200640 1 -- 172.21.15.6:0/4118960661 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f387404f330 con 0x7f38740cce90 2024-03-04T09:27:03.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.573+0000 7f386a000640 1 -- 172.21.15.6:0/4118960661 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3864037070 con 0x7f38740cce90 2024-03-04T09:27:03.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.575+0000 7f386a000640 1 --2- 172.21.15.6:0/4118960661 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f385406b4d0 0x7f385406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:03.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.575+0000 7f386a000640 1 -- 172.21.15.6:0/4118960661 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f38640abfe0 con 0x7f38740cce90 2024-03-04T09:27:03.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.575+0000 7f3871400640 1 --2- 172.21.15.6:0/4118960661 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f385406b4d0 0x7f385406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:03.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.576+0000 7f3871400640 1 --2- 172.21.15.6:0/4118960661 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f385406b4d0 0x7f385406d990 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f3860006fd0 tx=0x7f3860008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:03.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.579+0000 7f386a000640 1 -- 172.21.15.6:0/4118960661 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f386407af50 con 0x7f38740cce90 2024-03-04T09:27:03.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.820+0000 7f3873200640 1 -- 172.21.15.6:0/4118960661 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f38740af2f0 con 0x7f38740cce90 2024-03-04T09:27:03.817 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-03-04T09:27:03.820+0000 7f1606a00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-04T09:27:03.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.824+0000 7f386a000640 1 -- 172.21.15.6:0/4118960661 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v17) v1 ==== 161+0+0 (secure 0 0 0) 0x7f386407a850 con 0x7f38740cce90 2024-03-04T09:27:03.821 INFO:teuthology.orchestra.run.smithi006.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-04T09:27:03.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.826+0000 7f3873200640 1 -- 172.21.15.6:0/4118960661 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f385406b4d0 msgr2=0x7f385406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:03.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.826+0000 7f3873200640 1 --2- 172.21.15.6:0/4118960661 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f385406b4d0 0x7f385406d990 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f3860006fd0 tx=0x7f3860008040 comp rx=0 tx=0).stop 2024-03-04T09:27:03.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.826+0000 7f3873200640 1 -- 172.21.15.6:0/4118960661 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38740cce90 msgr2=0x7f3874140ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:03.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.826+0000 7f3873200640 1 --2- 172.21.15.6:0/4118960661 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38740cce90 0x7f3874140ec0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f3864009870 tx=0x7f3864004730 comp rx=0 tx=0).stop 2024-03-04T09:27:03.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.827+0000 7f3873200640 1 -- 172.21.15.6:0/4118960661 shutdown_connections 2024-03-04T09:27:03.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.827+0000 7f3873200640 1 --2- 172.21.15.6:0/4118960661 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3874162000 0x7f38740b3d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:03.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.827+0000 7f3873200640 1 --2- 172.21.15.6:0/4118960661 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f385406b4d0 0x7f385406d990 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:03.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.827+0000 7f3873200640 1 --2- 172.21.15.6:0/4118960661 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f387415efc0 0x7f3874143b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:03.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.827+0000 7f3873200640 1 --2- 172.21.15.6:0/4118960661 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38740cce90 0x7f3874140ec0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:03.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.827+0000 7f3873200640 1 -- 172.21.15.6:0/4118960661 >> 172.21.15.6:0/4118960661 conn(0x7f387406b3d0 msgr2=0x7f38740b0800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:03.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.827+0000 7f3873200640 1 -- 172.21.15.6:0/4118960661 shutdown_connections 2024-03-04T09:27:03.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:03.827+0000 7f3873200640 1 -- 172.21.15.6:0/4118960661 wait complete. 2024-03-04T09:27:03.843 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-04T09:27:03.945 INFO:tasks.ceph.mds.d.smithi146.stdout:starting mds.d at 2024-03-04T09:27:04.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:04.065+0000 7ff88b200640 1 Processor -- start 2024-03-04T09:27:04.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:04.066+0000 7ff88b200640 1 -- start start 2024-03-04T09:27:04.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:04.067+0000 7ff88b200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff88c15e0a0 0x7ff88c0d17a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:04.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:04.067+0000 7ff88b200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff88c0d2100 con 0x7ff88c0b2630 2024-03-04T09:27:04.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:04.067+0000 7ff88b200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff88c0b61b0 con 0x7ff88c0d1ce0 2024-03-04T09:27:04.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:04.067+0000 7ff88b200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff88c0b5f60 con 0x7ff88c15e0a0 2024-03-04T09:27:04.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:04.068+0000 7ff88a800640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7ff88c0b2630 0x7ff88c167ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:39028/0 (socket says 172.21.15.6:39028) 2024-03-04T09:27:04.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:04.068+0000 7ff88a800640 1 -- 172.21.15.6:0/104231260 learned_addr learned my addr 172.21.15.6:0/104231260 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:27:04.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:04.068+0000 7ff889e00640 1 --2- 172.21.15.6:0/104231260 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff88c15e0a0 0x7ff88c0d17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:04.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:04.068+0000 7ff888a00640 1 -- 172.21.15.6:0/104231260 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 669525464 0 0) 0x7ff88c0d2100 con 0x7ff88c0b2630 2024-03-04T09:27:04.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:04.068+0000 7ff888a00640 1 -- 172.21.15.6:0/104231260 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff870003520 con 0x7ff88c0b2630 2024-03-04T09:27:04.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:04.069+0000 7ff888a00640 1 -- 172.21.15.6:0/104231260 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4292968607 0 0) 0x7ff870003520 con 0x7ff88c0b2630 2024-03-04T09:27:04.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:04.069+0000 7ff888a00640 1 -- 172.21.15.6:0/104231260 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff88c15e0a0 msgr2=0x7ff88c0d17a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:27:04.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:04.069+0000 7ff888a00640 1 --2- 172.21.15.6:0/104231260 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff88c15e0a0 0x7ff88c0d17a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:04.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:04.069+0000 7ff888a00640 1 -- 172.21.15.6:0/104231260 >> v1:172.21.15.146:6789/0 conn(0x7ff88c0d1ce0 legacy=0x7ff88c06cb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:04.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:04.069+0000 7ff888a00640 1 -- 172.21.15.6:0/104231260 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff88c0b6490 con 0x7ff88c0b2630 2024-03-04T09:27:04.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:04.069+0000 7ff888a00640 1 -- 172.21.15.6:0/104231260 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7ff884001670 con 0x7ff88c0b2630 2024-03-04T09:27:04.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:04.069+0000 7ff888a00640 1 -- 172.21.15.6:0/104231260 >> v1:172.21.15.6:6789/0 conn(0x7ff88c0b2630 legacy=0x7ff88c167ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:04.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:04.069+0000 7ff888a00640 1 --2- 172.21.15.6:0/104231260 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff88c15e0a0 0x7ff8700039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:04.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:04.069+0000 7ff888a00640 1 --2- 172.21.15.6:0/104231260 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff870003f20 0x7ff870004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:04.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:04.069+0000 7ff888a00640 1 --2- 172.21.15.6:0/104231260 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff870004d40 0x7ff8700051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:04.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:04.069+0000 7ff888a00640 1 -- 172.21.15.6:0/104231260 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff88c0b5f60 con 0x7ff870004d40 2024-03-04T09:27:04.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:04.069+0000 7ff888a00640 1 -- 172.21.15.6:0/104231260 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff87000ba70 con 0x7ff870003f20 2024-03-04T09:27:04.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:04.069+0000 7ff888a00640 1 -- 172.21.15.6:0/104231260 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff87000bc70 con 0x7ff88c15e0a0 2024-03-04T09:27:04.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:04.069+0000 7ff88a800640 1 --2- 172.21.15.6:0/104231260 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff870004d40 0x7ff8700051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:04.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:04.070+0000 7ff889400640 1 --2- 172.21.15.6:0/104231260 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff870003f20 0x7ff870004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:04.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:04.070+0000 7ff889e00640 1 --2- 172.21.15.6:0/104231260 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff88c15e0a0 0x7ff8700039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:04.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:04.070+0000 7ff88a800640 1 -- 172.21.15.6:0/104231260 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff88c15e0a0 msgr2=0x7ff8700039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:27:04.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:04.070+0000 7ff88a800640 1 --2- 172.21.15.6:0/104231260 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff88c15e0a0 0x7ff8700039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:04.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:04.070+0000 7ff88a800640 1 -- 172.21.15.6:0/104231260 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff870003f20 msgr2=0x7ff870004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:27:04.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:04.070+0000 7ff88a800640 1 --2- 172.21.15.6:0/104231260 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff870003f20 0x7ff870004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:04.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:04.070+0000 7ff88a800640 1 -- 172.21.15.6:0/104231260 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff870003520 con 0x7ff870004d40 2024-03-04T09:27:04.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:04.071+0000 7ff88a800640 1 --2- 172.21.15.6:0/104231260 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff870004d40 0x7ff8700051c0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7ff88c0b2a80 tx=0x7ff870005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=801c282d70b4f2f5 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:04.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:04.071+0000 7ff888a00640 1 -- 172.21.15.6:0/104231260 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff884007c80 con 0x7ff870004d40 2024-03-04T09:27:04.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:04.071+0000 7ff888a00640 1 -- 172.21.15.6:0/104231260 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff884030070 con 0x7ff870004d40 2024-03-04T09:27:04.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:04.071+0000 7ff888a00640 1 -- 172.21.15.6:0/104231260 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff884005d30 con 0x7ff870004d40 2024-03-04T09:27:04.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:04.072+0000 7ff88b200640 1 -- 172.21.15.6:0/104231260 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff870004d40 msgr2=0x7ff8700051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:04.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:04.072+0000 7ff88b200640 1 --2- 172.21.15.6:0/104231260 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff870004d40 0x7ff8700051c0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7ff88c0b2a80 tx=0x7ff870005830 comp rx=0 tx=0).stop 2024-03-04T09:27:04.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:04.072+0000 7ff88a800640 1 -- 172.21.15.6:0/104231260 reap_dead start 2024-03-04T09:27:04.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:04.072+0000 7ff88b200640 1 -- 172.21.15.6:0/104231260 shutdown_connections 2024-03-04T09:27:04.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:04.072+0000 7ff88b200640 1 -- 172.21.15.6:0/104231260 >> 172.21.15.6:0/104231260 conn(0x7ff88c09aa10 msgr2=0x7ff88c09ae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:04.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:04.072+0000 7ff88b200640 1 -- 172.21.15.6:0/104231260 shutdown_connections 2024-03-04T09:27:04.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:04.072+0000 7ff88b200640 1 -- 172.21.15.6:0/104231260 wait complete. 2024-03-04T09:27:04.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:04.074+0000 7ff88b200640 1 Processor -- start 2024-03-04T09:27:04.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:04.074+0000 7ff88b200640 1 -- start start 2024-03-04T09:27:04.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:04.075+0000 7ff88b200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff870004d40 0x7ff88c1e6bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:04.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:04.075+0000 7ff88b200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff88c15bce0 0x7ff88c150510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:04.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:04.075+0000 7ff88b200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff88c1e5610 0x7ff88c1e78a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:04.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:04.075+0000 7ff88b200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff88c0d17a0 con 0x7ff88c1e5610 2024-03-04T09:27:04.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:04.075+0000 7ff88b200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff88c0b6ae0 con 0x7ff870004d40 2024-03-04T09:27:04.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:04.075+0000 7ff88b200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff88c0d1920 con 0x7ff88c15bce0 2024-03-04T09:27:04.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:04.075+0000 7ff88a800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff88c1e5610 0x7ff88c1e78a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:04.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:04.075+0000 7ff88a800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff88c1e5610 0x7ff88c1e78a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49736/0 (socket says 172.21.15.6:49736) 2024-03-04T09:27:04.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:04.075+0000 7ff88a800640 1 -- 172.21.15.6:0/2135986841 learned_addr learned my addr 172.21.15.6:0/2135986841 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:27:04.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:04.076+0000 7ff889e00640 1 --2- 172.21.15.6:0/2135986841 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff870004d40 0x7ff88c1e6bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:04.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:04.076+0000 7ff889400640 1 --2- 172.21.15.6:0/2135986841 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff88c15bce0 0x7ff88c150510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:04.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:04.076+0000 7ff88a800640 1 -- 172.21.15.6:0/2135986841 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff88c15bce0 msgr2=0x7ff88c150510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:04.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:04.076+0000 7ff88a800640 1 --2- 172.21.15.6:0/2135986841 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff88c15bce0 0x7ff88c150510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:04.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:04.076+0000 7ff88a800640 1 -- 172.21.15.6:0/2135986841 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff870004d40 msgr2=0x7ff88c1e6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:04.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:04.076+0000 7ff88a800640 1 --2- 172.21.15.6:0/2135986841 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff870004d40 0x7ff88c1e6bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:04.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:04.076+0000 7ff88a800640 1 -- 172.21.15.6:0/2135986841 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff88c1e70f0 con 0x7ff88c1e5610 2024-03-04T09:27:04.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:04.076+0000 7ff889400640 1 --2- 172.21.15.6:0/2135986841 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff88c15bce0 0x7ff88c150510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:27:04.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:04.076+0000 7ff88a800640 1 --2- 172.21.15.6:0/2135986841 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff88c1e5610 0x7ff88c1e78a0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7ff88402c480 tx=0x7ff88402e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:04.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:04.076+0000 7ff882000640 1 -- 172.21.15.6:0/2135986841 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff88402de60 con 0x7ff88c1e5610 2024-03-04T09:27:04.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:04.077+0000 7ff882000640 1 -- 172.21.15.6:0/2135986841 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff884004bf0 con 0x7ff88c1e5610 2024-03-04T09:27:04.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:04.077+0000 7ff88b200640 1 -- 172.21.15.6:0/2135986841 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff88c1e5b10 con 0x7ff88c1e5610 2024-03-04T09:27:04.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:04.077+0000 7ff882000640 1 -- 172.21.15.6:0/2135986841 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff884000e40 con 0x7ff88c1e5610 2024-03-04T09:27:04.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:04.077+0000 7ff88b200640 1 -- 172.21.15.6:0/2135986841 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff88c1e60d0 con 0x7ff88c1e5610 2024-03-04T09:27:04.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:04.078+0000 7ff88b200640 1 -- 172.21.15.6:0/2135986841 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff88c04f330 con 0x7ff88c1e5610 2024-03-04T09:27:04.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:04.079+0000 7ff882000640 1 -- 172.21.15.6:0/2135986841 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff884007070 con 0x7ff88c1e5610 2024-03-04T09:27:04.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:04.080+0000 7ff882000640 1 --2- 172.21.15.6:0/2135986841 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff86806b4d0 0x7ff86806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:04.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:04.080+0000 7ff882000640 1 -- 172.21.15.6:0/2135986841 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5795+0+0 (secure 0 0 0) 0x7ff8840ab430 con 0x7ff88c1e5610 2024-03-04T09:27:04.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:04.080+0000 7ff889e00640 1 --2- 172.21.15.6:0/2135986841 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff86806b4d0 0x7ff86806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:04.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:04.081+0000 7ff889e00640 1 --2- 172.21.15.6:0/2135986841 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff86806b4d0 0x7ff86806d990 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7ff87c007520 tx=0x7ff87c00b040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:04.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:04.084+0000 7ff882000640 1 -- 172.21.15.6:0/2135986841 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff88407a1b0 con 0x7ff88c1e5610 2024-03-04T09:27:04.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:04.333+0000 7ff88b200640 1 -- 172.21.15.6:0/2135986841 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7ff88c06d340 con 0x7ff88c1e5610 2024-03-04T09:27:04.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:04.836+0000 7ff882000640 1 -- 172.21.15.6:0/2135986841 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v18) v1 ==== 104+0+0 (secure 0 0 0) 0x7ff884079ab0 con 0x7ff88c1e5610 2024-03-04T09:27:04.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:04.839+0000 7ff88b200640 1 -- 172.21.15.6:0/2135986841 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff86806b4d0 msgr2=0x7ff86806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:04.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:04.839+0000 7ff88b200640 1 --2- 172.21.15.6:0/2135986841 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff86806b4d0 0x7ff86806d990 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7ff87c007520 tx=0x7ff87c00b040 comp rx=0 tx=0).stop 2024-03-04T09:27:04.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:04.839+0000 7ff88b200640 1 -- 172.21.15.6:0/2135986841 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff88c1e5610 msgr2=0x7ff88c1e78a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:04.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:04.839+0000 7ff88b200640 1 --2- 172.21.15.6:0/2135986841 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff88c1e5610 0x7ff88c1e78a0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7ff88402c480 tx=0x7ff88402e070 comp rx=0 tx=0).stop 2024-03-04T09:27:04.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:04.840+0000 7ff88b200640 1 -- 172.21.15.6:0/2135986841 shutdown_connections 2024-03-04T09:27:04.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:04.840+0000 7ff88b200640 1 --2- 172.21.15.6:0/2135986841 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff88c1e5610 0x7ff88c1e78a0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:04.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:04.840+0000 7ff88b200640 1 --2- 172.21.15.6:0/2135986841 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff86806b4d0 0x7ff86806d990 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:04.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:04.840+0000 7ff88b200640 1 --2- 172.21.15.6:0/2135986841 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff88c15bce0 0x7ff88c150510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:04.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:04.840+0000 7ff88b200640 1 --2- 172.21.15.6:0/2135986841 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff870004d40 0x7ff88c1e6bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:04.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:04.840+0000 7ff88b200640 1 -- 172.21.15.6:0/2135986841 >> 172.21.15.6:0/2135986841 conn(0x7ff88c09aa10 msgr2=0x7ff88c0b37b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:04.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:04.840+0000 7ff88b200640 1 -- 172.21.15.6:0/2135986841 shutdown_connections 2024-03-04T09:27:04.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:04.840+0000 7ff88b200640 1 -- 172.21.15.6:0/2135986841 wait complete. 2024-03-04T09:27:04.858 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-04T09:27:05.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:05.096+0000 7f844ee00640 1 Processor -- start 2024-03-04T09:27:05.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:05.096+0000 7f844ee00640 1 -- start start 2024-03-04T09:27:05.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:05.098+0000 7f844ee00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f844806d230 0x7f8448071b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:05.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:05.098+0000 7f844ee00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8448072f40 con 0x7f8448162400 2024-03-04T09:27:05.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:05.098+0000 7f844ee00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84480b4820 con 0x7f84480cff20 2024-03-04T09:27:05.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:05.098+0000 7f844ee00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f844806cda0 con 0x7f844806d230 2024-03-04T09:27:05.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:05.098+0000 7f8447e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f8448162400 0x7f84480cf810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:39032/0 (socket says 172.21.15.6:39032) 2024-03-04T09:27:05.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:05.098+0000 7f844ce00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f844806d230 0x7f8448071b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:05.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:05.098+0000 7f8447e00640 1 -- 172.21.15.6:0/1463834462 learned_addr learned my addr 172.21.15.6:0/1463834462 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:27:05.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:05.099+0000 7f844ce00640 1 -- 172.21.15.6:0/1463834462 >> v1:172.21.15.146:6789/0 conn(0x7f84480cff20 legacy=0x7f84481627e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:27:05.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:05.099+0000 7f844ce00640 1 -- 172.21.15.6:0/1463834462 >> v1:172.21.15.6:6789/0 conn(0x7f8448162400 legacy=0x7f84480cf810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:05.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:05.099+0000 7f844ce00640 1 -- 172.21.15.6:0/1463834462 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84480b4b00 con 0x7f844806d230 2024-03-04T09:27:05.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:05.099+0000 7f8446a00640 1 -- 172.21.15.6:0/1463834462 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3845482029 0 0) 0x7f8448072f40 con 0x7f8448162400 2024-03-04T09:27:05.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:05.100+0000 7f844ce00640 1 --2- 172.21.15.6:0/1463834462 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f844806d230 0x7f8448071b50 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f84400097a0 tx=0x7f844002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f705354811c90e54 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:05.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:05.100+0000 7f8446a00640 1 -- 172.21.15.6:0/1463834462 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8440035020 con 0x7f844806d230 2024-03-04T09:27:05.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:05.100+0000 7f8446a00640 1 -- 172.21.15.6:0/1463834462 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8440031830 con 0x7f844806d230 2024-03-04T09:27:05.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:05.100+0000 7f8446a00640 1 -- 172.21.15.6:0/1463834462 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8440031c00 con 0x7f844806d230 2024-03-04T09:27:05.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:05.101+0000 7f844ee00640 1 -- 172.21.15.6:0/1463834462 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f844806d230 msgr2=0x7f8448071b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:05.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:05.101+0000 7f844ee00640 1 --2- 172.21.15.6:0/1463834462 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f844806d230 0x7f8448071b50 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f84400097a0 tx=0x7f844002b8d0 comp rx=0 tx=0).stop 2024-03-04T09:27:05.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:05.101+0000 7f844ee00640 1 -- 172.21.15.6:0/1463834462 shutdown_connections 2024-03-04T09:27:05.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:05.101+0000 7f844ee00640 1 --2- 172.21.15.6:0/1463834462 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f844806d230 0x7f8448071b50 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:05.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:05.101+0000 7f844ee00640 1 -- 172.21.15.6:0/1463834462 >> 172.21.15.6:0/1463834462 conn(0x7f844806afb0 msgr2=0x7f844806b3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:05.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:05.101+0000 7f844ee00640 1 -- 172.21.15.6:0/1463834462 shutdown_connections 2024-03-04T09:27:05.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:05.101+0000 7f844ee00640 1 -- 172.21.15.6:0/1463834462 wait complete. 2024-03-04T09:27:05.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:05.103+0000 7f844ee00640 1 Processor -- start 2024-03-04T09:27:05.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:05.103+0000 7f844ee00640 1 -- start start 2024-03-04T09:27:05.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:05.104+0000 7f844ee00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f844806d230 0x7f8448163f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:05.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:05.104+0000 7f844ee00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f84480cff20 0x7f8448165600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:05.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:05.104+0000 7f844ee00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8448162400 0x7f84481608a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:05.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:05.104+0000 7f844ee00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8448072b00 con 0x7f84480cff20 2024-03-04T09:27:05.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:05.104+0000 7f844ee00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8448072980 con 0x7f844806d230 2024-03-04T09:27:05.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:05.104+0000 7f844ee00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8448072c80 con 0x7f8448162400 2024-03-04T09:27:05.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:05.104+0000 7f8447400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f84480cff20 0x7f8448165600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:05.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:05.104+0000 7f8447400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f84480cff20 0x7f8448165600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49752/0 (socket says 172.21.15.6:49752) 2024-03-04T09:27:05.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:05.104+0000 7f8447400640 1 -- 172.21.15.6:0/70624644 learned_addr learned my addr 172.21.15.6:0/70624644 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:27:05.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:05.105+0000 7f8447e00640 1 --2- 172.21.15.6:0/70624644 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f844806d230 0x7f8448163f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:05.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:05.105+0000 7f844ce00640 1 --2- 172.21.15.6:0/70624644 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8448162400 0x7f84481608a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:05.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:05.105+0000 7f8447400640 1 -- 172.21.15.6:0/70624644 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8448162400 msgr2=0x7f84481608a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:05.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:05.105+0000 7f8447400640 1 --2- 172.21.15.6:0/70624644 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8448162400 0x7f84481608a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:05.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:05.105+0000 7f8447400640 1 -- 172.21.15.6:0/70624644 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f844806d230 msgr2=0x7f8448163f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:05.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:05.105+0000 7f8447400640 1 --2- 172.21.15.6:0/70624644 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f844806d230 0x7f8448163f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:05.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:05.105+0000 7f8447400640 1 -- 172.21.15.6:0/70624644 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8448164560 con 0x7f84480cff20 2024-03-04T09:27:05.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:05.105+0000 7f8447400640 1 --2- 172.21.15.6:0/70624644 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f84480cff20 0x7f8448165600 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f843000c450 tx=0x7f843000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:05.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:05.105+0000 7f843fe00640 1 -- 172.21.15.6:0/70624644 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84300052d0 con 0x7f84480cff20 2024-03-04T09:27:05.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:05.106+0000 7f843fe00640 1 -- 172.21.15.6:0/70624644 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8430005470 con 0x7f84480cff20 2024-03-04T09:27:05.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:05.106+0000 7f843fe00640 1 -- 172.21.15.6:0/70624644 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8430012870 con 0x7f84480cff20 2024-03-04T09:27:05.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:05.106+0000 7f844ee00640 1 -- 172.21.15.6:0/70624644 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8448164840 con 0x7f84480cff20 2024-03-04T09:27:05.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:05.106+0000 7f844ee00640 1 -- 172.21.15.6:0/70624644 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8448164e00 con 0x7f84480cff20 2024-03-04T09:27:05.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:05.107+0000 7f844ee00640 1 -- 172.21.15.6:0/70624644 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8448068010 con 0x7f84480cff20 2024-03-04T09:27:05.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:05.108+0000 7f843fe00640 1 -- 172.21.15.6:0/70624644 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8430012a10 con 0x7f84480cff20 2024-03-04T09:27:05.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:05.109+0000 7f843fe00640 1 --2- 172.21.15.6:0/70624644 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f842406b4d0 0x7f842406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:05.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:05.109+0000 7f843fe00640 1 -- 172.21.15.6:0/70624644 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f8430089f10 con 0x7f84480cff20 2024-03-04T09:27:05.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:05.110+0000 7f8447e00640 1 --2- 172.21.15.6:0/70624644 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f842406b4d0 0x7f842406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:05.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:05.111+0000 7f8447e00640 1 --2- 172.21.15.6:0/70624644 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f842406b4d0 0x7f842406d990 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f8438007fd0 tx=0x7f8438009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:05.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:05.114+0000 7f843fe00640 1 -- 172.21.15.6:0/70624644 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8430058c90 con 0x7f84480cff20 2024-03-04T09:27:05.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:05.323+0000 7f844ee00640 1 -- 172.21.15.6:0/70624644 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f84480ae240 con 0x7f84480cff20 2024-03-04T09:27:06.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:06.187+0000 7f843fe00640 1 -- 172.21.15.6:0/70624644 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v35) v1 ==== 181+0+0 (secure 0 0 0) 0x7f8430058590 con 0x7f84480cff20 2024-03-04T09:27:06.185 INFO:teuthology.orchestra.run.smithi006.stderr:pool 'cephfs_metadata' removed 2024-03-04T09:27:06.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:06.191+0000 7f844ee00640 1 -- 172.21.15.6:0/70624644 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f842406b4d0 msgr2=0x7f842406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:06.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:06.191+0000 7f844ee00640 1 --2- 172.21.15.6:0/70624644 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f842406b4d0 0x7f842406d990 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f8438007fd0 tx=0x7f8438009040 comp rx=0 tx=0).stop 2024-03-04T09:27:06.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:06.192+0000 7f844ee00640 1 -- 172.21.15.6:0/70624644 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f84480cff20 msgr2=0x7f8448165600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:06.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:06.192+0000 7f844ee00640 1 --2- 172.21.15.6:0/70624644 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f84480cff20 0x7f8448165600 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f843000c450 tx=0x7f843000c910 comp rx=0 tx=0).stop 2024-03-04T09:27:06.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:06.192+0000 7f844ee00640 1 -- 172.21.15.6:0/70624644 shutdown_connections 2024-03-04T09:27:06.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:06.192+0000 7f844ee00640 1 --2- 172.21.15.6:0/70624644 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f842406b4d0 0x7f842406d990 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:06.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:06.192+0000 7f844ee00640 1 --2- 172.21.15.6:0/70624644 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8448162400 0x7f84481608a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:06.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:06.192+0000 7f844ee00640 1 --2- 172.21.15.6:0/70624644 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f84480cff20 0x7f8448165600 secure :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f843000c450 tx=0x7f843000c910 comp rx=0 tx=0).stop 2024-03-04T09:27:06.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:06.192+0000 7f844ee00640 1 --2- 172.21.15.6:0/70624644 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f844806d230 0x7f8448163f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:06.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:06.192+0000 7f844ee00640 1 -- 172.21.15.6:0/70624644 >> 172.21.15.6:0/70624644 conn(0x7f844806afb0 msgr2=0x7f84480b01a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:06.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:06.193+0000 7f844ee00640 1 -- 172.21.15.6:0/70624644 shutdown_connections 2024-03-04T09:27:06.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:06.195+0000 7f844ee00640 1 -- 172.21.15.6:0/70624644 wait complete. 2024-03-04T09:27:06.218 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-04T09:27:06.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:06.441+0000 7f81b1c00640 1 Processor -- start 2024-03-04T09:27:06.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:06.441+0000 7f81b1c00640 1 -- start start 2024-03-04T09:27:06.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:06.443+0000 7f81b1c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f81ac15de00 0x7f81ac072910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:06.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:06.443+0000 7f81b1c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f81ac15e1e0 0x7f81ac0b33b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:06.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:06.443+0000 7f81b1c00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81ac0b2c20 con 0x7f81ac0b4af0 2024-03-04T09:27:06.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:06.443+0000 7f81b1c00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81ac0b2f20 con 0x7f81ac15de00 2024-03-04T09:27:06.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:06.443+0000 7f81b1c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81ac06cda0 con 0x7f81ac15e1e0 2024-03-04T09:27:06.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:06.443+0000 7f81aaa00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f81ac15de00 0x7f81ac072910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:06.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:06.443+0000 7f81aaa00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f81ac15de00 0x7f81ac072910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49766/0 (socket says 172.21.15.6:49766) 2024-03-04T09:27:06.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:06.443+0000 7f81aaa00640 1 -- 172.21.15.6:0/2451482267 learned_addr learned my addr 172.21.15.6:0/2451482267 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:27:06.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:06.444+0000 7f81aa000640 1 --2- 172.21.15.6:0/2451482267 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f81ac15e1e0 0x7f81ac0b33b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:06.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:06.444+0000 7f81aaa00640 1 -- 172.21.15.6:0/2451482267 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f81ac15e1e0 msgr2=0x7f81ac0b33b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:27:06.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:06.444+0000 7f81aaa00640 1 --2- 172.21.15.6:0/2451482267 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f81ac15e1e0 0x7f81ac0b33b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:06.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:06.444+0000 7f81aaa00640 1 -- 172.21.15.6:0/2451482267 >> v1:172.21.15.146:6789/0 conn(0x7f81ac0b4af0 legacy=0x7f81ac0cf810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:27:06.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:06.444+0000 7f81aaa00640 1 -- 172.21.15.6:0/2451482267 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81ac06d080 con 0x7f81ac15de00 2024-03-04T09:27:06.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:06.445+0000 7f81aaa00640 1 --2- 172.21.15.6:0/2451482267 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f81ac15de00 0x7f81ac072910 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f819c009e50 tx=0x7f819c02bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b84fa72b09941dad server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:06.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:06.446+0000 7f81a9600640 1 -- 172.21.15.6:0/2451482267 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f819c02c950 con 0x7f81ac15de00 2024-03-04T09:27:06.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:06.446+0000 7f81a9600640 1 -- 172.21.15.6:0/2451482267 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f819c02caf0 con 0x7f81ac15de00 2024-03-04T09:27:06.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:06.446+0000 7f81a9600640 1 -- 172.21.15.6:0/2451482267 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f819c0343f0 con 0x7f81ac15de00 2024-03-04T09:27:06.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:06.446+0000 7f81b1c00640 1 -- 172.21.15.6:0/2451482267 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f81ac15de00 msgr2=0x7f81ac072910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:06.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:06.446+0000 7f81b1c00640 1 --2- 172.21.15.6:0/2451482267 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f81ac15de00 0x7f81ac072910 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f819c009e50 tx=0x7f819c02bdb0 comp rx=0 tx=0).stop 2024-03-04T09:27:06.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:06.447+0000 7f81b1c00640 1 -- 172.21.15.6:0/2451482267 shutdown_connections 2024-03-04T09:27:06.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:06.447+0000 7f81b1c00640 1 --2- 172.21.15.6:0/2451482267 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f81ac15e1e0 0x7f81ac0b33b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:06.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:06.447+0000 7f81b1c00640 1 --2- 172.21.15.6:0/2451482267 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f81ac15de00 0x7f81ac072910 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:06.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:06.447+0000 7f81b1c00640 1 -- 172.21.15.6:0/2451482267 >> 172.21.15.6:0/2451482267 conn(0x7f81ac06afb0 msgr2=0x7f81ac06b3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:06.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:06.447+0000 7f81b1c00640 1 -- 172.21.15.6:0/2451482267 shutdown_connections 2024-03-04T09:27:06.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:06.447+0000 7f81b1c00640 1 -- 172.21.15.6:0/2451482267 wait complete. 2024-03-04T09:27:06.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:06.448+0000 7f81b1c00640 1 Processor -- start 2024-03-04T09:27:06.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:06.449+0000 7f81b1c00640 1 -- start start 2024-03-04T09:27:06.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:06.449+0000 7f81b1c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f81ac15de00 0x7f81ac1f4280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:06.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:06.450+0000 7f81b1c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f81ac0b0310 0x7f81ac09b460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:06.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:06.450+0000 7f81b1c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f81ac09d260 0x7f81ac161810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:06.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:06.450+0000 7f81b1c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81ac06d4e0 con 0x7f81ac09d260 2024-03-04T09:27:06.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:06.450+0000 7f81b1c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81ac06d360 con 0x7f81ac0b0310 2024-03-04T09:27:06.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:06.450+0000 7f81b1c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81ac0b38f0 con 0x7f81ac15de00 2024-03-04T09:27:06.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:06.450+0000 7f81ab400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f81ac09d260 0x7f81ac161810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:06.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:06.450+0000 7f81ab400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f81ac09d260 0x7f81ac161810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49778/0 (socket says 172.21.15.6:49778) 2024-03-04T09:27:06.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:06.450+0000 7f81ab400640 1 -- 172.21.15.6:0/2630970602 learned_addr learned my addr 172.21.15.6:0/2630970602 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:27:06.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:06.450+0000 7f81aa000640 1 --2- 172.21.15.6:0/2630970602 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f81ac0b0310 0x7f81ac09b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:06.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:06.450+0000 7f81aaa00640 1 --2- 172.21.15.6:0/2630970602 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f81ac15de00 0x7f81ac1f4280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:06.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:06.451+0000 7f81ab400640 1 -- 172.21.15.6:0/2630970602 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f81ac15de00 msgr2=0x7f81ac1f4280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:06.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:06.451+0000 7f81ab400640 1 --2- 172.21.15.6:0/2630970602 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f81ac15de00 0x7f81ac1f4280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:06.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:06.451+0000 7f81ab400640 1 -- 172.21.15.6:0/2630970602 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f81ac0b0310 msgr2=0x7f81ac09b460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:06.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:06.451+0000 7f81ab400640 1 --2- 172.21.15.6:0/2630970602 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f81ac0b0310 0x7f81ac09b460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:06.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:06.451+0000 7f81ab400640 1 -- 172.21.15.6:0/2630970602 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81ac0b0790 con 0x7f81ac09d260 2024-03-04T09:27:06.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:06.451+0000 7f81aaa00640 1 --2- 172.21.15.6:0/2630970602 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f81ac15de00 0x7f81ac1f4280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:27:06.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:06.451+0000 7f81ab400640 1 --2- 172.21.15.6:0/2630970602 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f81ac09d260 0x7f81ac161810 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f81a400c410 tx=0x7f81a400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:06.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:06.451+0000 7f81a2a00640 1 -- 172.21.15.6:0/2630970602 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81a4017070 con 0x7f81ac09d260 2024-03-04T09:27:06.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:06.451+0000 7f81a2a00640 1 -- 172.21.15.6:0/2630970602 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81a4005230 con 0x7f81ac09d260 2024-03-04T09:27:06.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:06.451+0000 7f81a2a00640 1 -- 172.21.15.6:0/2630970602 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81a4011590 con 0x7f81ac09d260 2024-03-04T09:27:06.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:06.451+0000 7f81b1c00640 1 -- 172.21.15.6:0/2630970602 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81ac0b0a20 con 0x7f81ac09d260 2024-03-04T09:27:06.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:06.451+0000 7f81b1c00640 1 -- 172.21.15.6:0/2630970602 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81ac09ac60 con 0x7f81ac09d260 2024-03-04T09:27:06.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:06.452+0000 7f81b1c00640 1 -- 172.21.15.6:0/2630970602 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81ac068010 con 0x7f81ac09d260 2024-03-04T09:27:06.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:06.454+0000 7f81a2a00640 1 -- 172.21.15.6:0/2630970602 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f81a401a070 con 0x7f81ac09d260 2024-03-04T09:27:06.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:06.454+0000 7f81a2a00640 1 --2- 172.21.15.6:0/2630970602 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f818806b4d0 0x7f818806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:06.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:06.455+0000 7f81a2a00640 1 -- 172.21.15.6:0/2630970602 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5357+0+0 (secure 0 0 0) 0x7f81a40893f0 con 0x7f81ac09d260 2024-03-04T09:27:06.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:06.455+0000 7f81aaa00640 1 --2- 172.21.15.6:0/2630970602 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f818806b4d0 0x7f818806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:06.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:06.456+0000 7f81aaa00640 1 --2- 172.21.15.6:0/2630970602 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f818806b4d0 0x7f818806d990 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f819c035000 tx=0x7f819c030000 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:06.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:06.459+0000 7f81a2a00640 1 -- 172.21.15.6:0/2630970602 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f81a4058330 con 0x7f81ac09d260 2024-03-04T09:27:06.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:06.659+0000 7f81b1c00640 1 -- 172.21.15.6:0/2630970602 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f81ac09aa10 con 0x7f81ac09d260 2024-03-04T09:27:07.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.191+0000 7f81a2a00640 1 -- 172.21.15.6:0/2630970602 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v36) v1 ==== 169+0+0 (secure 0 0 0) 0x7f81a4057c30 con 0x7f81ac09d260 2024-03-04T09:27:07.188 INFO:teuthology.orchestra.run.smithi006.stderr:pool 'cephfs_data' removed 2024-03-04T09:27:07.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.195+0000 7f81b1c00640 1 -- 172.21.15.6:0/2630970602 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f818806b4d0 msgr2=0x7f818806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:07.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.195+0000 7f81b1c00640 1 --2- 172.21.15.6:0/2630970602 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f818806b4d0 0x7f818806d990 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f819c035000 tx=0x7f819c030000 comp rx=0 tx=0).stop 2024-03-04T09:27:07.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.195+0000 7f81b1c00640 1 -- 172.21.15.6:0/2630970602 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f81ac09d260 msgr2=0x7f81ac161810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:07.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.195+0000 7f81b1c00640 1 --2- 172.21.15.6:0/2630970602 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f81ac09d260 0x7f81ac161810 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f81a400c410 tx=0x7f81a400c8d0 comp rx=0 tx=0).stop 2024-03-04T09:27:07.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.195+0000 7f81b1c00640 1 -- 172.21.15.6:0/2630970602 shutdown_connections 2024-03-04T09:27:07.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.195+0000 7f81b1c00640 1 --2- 172.21.15.6:0/2630970602 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f81ac09d260 0x7f81ac161810 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:07.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.195+0000 7f81b1c00640 1 --2- 172.21.15.6:0/2630970602 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f81ac0b0310 0x7f81ac09b460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:07.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.195+0000 7f81b1c00640 1 --2- 172.21.15.6:0/2630970602 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f818806b4d0 0x7f818806d990 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:07.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.195+0000 7f81b1c00640 1 --2- 172.21.15.6:0/2630970602 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f81ac15de00 0x7f81ac1f4280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:07.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.195+0000 7f81b1c00640 1 -- 172.21.15.6:0/2630970602 >> 172.21.15.6:0/2630970602 conn(0x7f81ac06afb0 msgr2=0x7f81ac0723c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:07.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.196+0000 7f81b1c00640 1 -- 172.21.15.6:0/2630970602 shutdown_connections 2024-03-04T09:27:07.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.196+0000 7f81b1c00640 1 -- 172.21.15.6:0/2630970602 wait complete. 2024-03-04T09:27:07.221 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-04T09:27:07.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.425+0000 7f610ec00640 1 Processor -- start 2024-03-04T09:27:07.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.425+0000 7f610ec00640 1 -- start start 2024-03-04T09:27:07.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.427+0000 7f610ec00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f61080d2bd0 0x7f61080cea80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:07.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.427+0000 7f610ec00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f61080c8d20 0x7f61080cefc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:07.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.427+0000 7f610ec00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61080d20d0 con 0x7f6108161930 2024-03-04T09:27:07.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.427+0000 7f610ec00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61080d4df0 con 0x7f61080d2bd0 2024-03-04T09:27:07.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.427+0000 7f610ec00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61080d1df0 con 0x7f61080c8d20 2024-03-04T09:27:07.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.428+0000 7f6107e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f6108161930 0x7f61080d24c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:39048/0 (socket says 172.21.15.6:39048) 2024-03-04T09:27:07.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.428+0000 7f6107e00640 1 -- 172.21.15.6:0/980812133 learned_addr learned my addr 172.21.15.6:0/980812133 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:27:07.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.428+0000 7f60ffe00640 1 --2- 172.21.15.6:0/980812133 >> v2:172.21.15.6:3300/0 conn(0x7f61080d2bd0 0x7f61080cea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:07.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.428+0000 7f610cc00640 1 --2- 172.21.15.6:0/980812133 >> v2:172.21.15.146:3300/0 conn(0x7f61080c8d20 0x7f61080cefc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:07.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.429+0000 7f60ffe00640 1 -- 172.21.15.6:0/980812133 >> v2:172.21.15.146:3300/0 conn(0x7f61080c8d20 msgr2=0x7f61080cefc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:27:07.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.429+0000 7f60ffe00640 1 --2- 172.21.15.6:0/980812133 >> v2:172.21.15.146:3300/0 conn(0x7f61080c8d20 0x7f61080cefc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:07.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.429+0000 7f60ffe00640 1 -- 172.21.15.6:0/980812133 >> v1:172.21.15.6:6789/0 conn(0x7f6108161930 legacy=0x7f61080d24c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:07.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.429+0000 7f60ffe00640 1 -- 172.21.15.6:0/980812133 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61080d4b10 con 0x7f61080d2bd0 2024-03-04T09:27:07.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.429+0000 7f6107400640 1 -- 172.21.15.6:0/980812133 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1216217575 0 0) 0x7f61080d20d0 con 0x7f6108161930 2024-03-04T09:27:07.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.430+0000 7f60ffe00640 1 --2- 172.21.15.6:0/980812133 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f61080d2bd0 0x7f61080cea80 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f610808e1c0 tx=0x7f60f002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aeaf7b264ac3df3 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:07.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.431+0000 7f6107400640 1 -- 172.21.15.6:0/980812133 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60f00315a0 con 0x7f61080d2bd0 2024-03-04T09:27:07.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.431+0000 7f6107400640 1 -- 172.21.15.6:0/980812133 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60f0037070 con 0x7f61080d2bd0 2024-03-04T09:27:07.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.431+0000 7f6107400640 1 -- 172.21.15.6:0/980812133 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60f0031960 con 0x7f61080d2bd0 2024-03-04T09:27:07.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.431+0000 7f610ec00640 1 -- 172.21.15.6:0/980812133 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f61080d2bd0 msgr2=0x7f61080cea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:07.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.431+0000 7f610ec00640 1 --2- 172.21.15.6:0/980812133 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f61080d2bd0 0x7f61080cea80 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f610808e1c0 tx=0x7f60f002b840 comp rx=0 tx=0).stop 2024-03-04T09:27:07.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.432+0000 7f610ec00640 1 -- 172.21.15.6:0/980812133 shutdown_connections 2024-03-04T09:27:07.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.432+0000 7f610ec00640 1 --2- 172.21.15.6:0/980812133 >> v2:172.21.15.146:3300/0 conn(0x7f61080c8d20 0x7f61080cefc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:07.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.432+0000 7f610ec00640 1 --2- 172.21.15.6:0/980812133 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f61080d2bd0 0x7f61080cea80 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:07.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.432+0000 7f610ec00640 1 -- 172.21.15.6:0/980812133 >> 172.21.15.6:0/980812133 conn(0x7f61080d3ee0 msgr2=0x7f61080d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:07.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.432+0000 7f610ec00640 1 -- 172.21.15.6:0/980812133 shutdown_connections 2024-03-04T09:27:07.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.432+0000 7f610ec00640 1 -- 172.21.15.6:0/980812133 wait complete. 2024-03-04T09:27:07.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.434+0000 7f610ec00640 1 Processor -- start 2024-03-04T09:27:07.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.434+0000 7f610ec00640 1 -- start start 2024-03-04T09:27:07.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.435+0000 7f610ec00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f61080c8d20 0x7f61081584a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:07.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.435+0000 7f610ec00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6108161930 0x7f61080cd850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:07.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.435+0000 7f610ec00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f610815aec0 0x7f61080cf5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:07.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.435+0000 7f610ec00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61080d0f80 con 0x7f610815aec0 2024-03-04T09:27:07.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.435+0000 7f610ec00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61080b4f30 con 0x7f6108161930 2024-03-04T09:27:07.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.435+0000 7f610ec00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61080d1100 con 0x7f61080c8d20 2024-03-04T09:27:07.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.435+0000 7f610cc00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f610815aec0 0x7f61080cf5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:07.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.435+0000 7f610cc00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f610815aec0 0x7f61080cf5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49792/0 (socket says 172.21.15.6:49792) 2024-03-04T09:27:07.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.435+0000 7f610cc00640 1 -- 172.21.15.6:0/455716568 learned_addr learned my addr 172.21.15.6:0/455716568 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:27:07.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.436+0000 7f6107e00640 1 --2- 172.21.15.6:0/455716568 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f61080c8d20 0x7f61081584a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:07.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.436+0000 7f610cc00640 1 -- 172.21.15.6:0/455716568 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f61080c8d20 msgr2=0x7f61081584a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:07.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.436+0000 7f610cc00640 1 --2- 172.21.15.6:0/455716568 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f61080c8d20 0x7f61081584a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:07.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.436+0000 7f610cc00640 1 -- 172.21.15.6:0/455716568 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6108161930 msgr2=0x7f61080cd850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:07.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.436+0000 7f610cc00640 1 --2- 172.21.15.6:0/455716568 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6108161930 0x7f61080cd850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:07.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.436+0000 7f610cc00640 1 -- 172.21.15.6:0/455716568 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f610815b3d0 con 0x7f610815aec0 2024-03-04T09:27:07.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.436+0000 7f610cc00640 1 --2- 172.21.15.6:0/455716568 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f610815aec0 0x7f61080cf5d0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f610000efc0 tx=0x7f610000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:07.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.436+0000 7f6104c00640 1 -- 172.21.15.6:0/455716568 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6100018020 con 0x7f610815aec0 2024-03-04T09:27:07.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.437+0000 7f6104c00640 1 -- 172.21.15.6:0/455716568 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6100014480 con 0x7f610815aec0 2024-03-04T09:27:07.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.437+0000 7f610ec00640 1 -- 172.21.15.6:0/455716568 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f610815b6b0 con 0x7f610815aec0 2024-03-04T09:27:07.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.437+0000 7f6104c00640 1 -- 172.21.15.6:0/455716568 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6100014870 con 0x7f610815aec0 2024-03-04T09:27:07.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.437+0000 7f610ec00640 1 -- 172.21.15.6:0/455716568 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f610815bc70 con 0x7f610815aec0 2024-03-04T09:27:07.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.438+0000 7f610ec00640 1 -- 172.21.15.6:0/455716568 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61080cd430 con 0x7f610815aec0 2024-03-04T09:27:07.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.439+0000 7f6104c00640 1 -- 172.21.15.6:0/455716568 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6100005d10 con 0x7f610815aec0 2024-03-04T09:27:07.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.440+0000 7f6104c00640 1 --2- 172.21.15.6:0/455716568 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f60e406b4d0 0x7f60e406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:07.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.440+0000 7f6104c00640 1 -- 172.21.15.6:0/455716568 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f610008adb0 con 0x7f610815aec0 2024-03-04T09:27:07.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.441+0000 7f6107e00640 1 --2- 172.21.15.6:0/455716568 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f60e406b4d0 0x7f60e406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:07.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.442+0000 7f6107e00640 1 --2- 172.21.15.6:0/455716568 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f60e406b4d0 0x7f60e406d990 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f60f8007fd0 tx=0x7f60f8009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:07.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.444+0000 7f6104c00640 1 -- 172.21.15.6:0/455716568 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f610008f070 con 0x7f610815aec0 2024-03-04T09:27:07.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.643+0000 7f610ec00640 1 -- 172.21.15.6:0/455716568 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f610804f330 con 0x7f610815aec0 2024-03-04T09:27:07.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.648+0000 7f6104c00640 1 -- 172.21.15.6:0/455716568 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v11) v1 ==== 104+0+0 (secure 0 0 0) 0x7f6100022030 con 0x7f610815aec0 2024-03-04T09:27:07.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.650+0000 7f610ec00640 1 -- 172.21.15.6:0/455716568 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f60e406b4d0 msgr2=0x7f60e406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:07.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.650+0000 7f610ec00640 1 --2- 172.21.15.6:0/455716568 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f60e406b4d0 0x7f60e406d990 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f60f8007fd0 tx=0x7f60f8009040 comp rx=0 tx=0).stop 2024-03-04T09:27:07.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.650+0000 7f610ec00640 1 -- 172.21.15.6:0/455716568 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f610815aec0 msgr2=0x7f61080cf5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:07.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.651+0000 7f610ec00640 1 --2- 172.21.15.6:0/455716568 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f610815aec0 0x7f61080cf5d0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f610000efc0 tx=0x7f610000c490 comp rx=0 tx=0).stop 2024-03-04T09:27:07.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.651+0000 7f610ec00640 1 -- 172.21.15.6:0/455716568 shutdown_connections 2024-03-04T09:27:07.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.651+0000 7f610ec00640 1 --2- 172.21.15.6:0/455716568 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f610815aec0 0x7f61080cf5d0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:07.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.651+0000 7f610ec00640 1 --2- 172.21.15.6:0/455716568 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6108161930 0x7f61080cd850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:07.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.651+0000 7f610ec00640 1 --2- 172.21.15.6:0/455716568 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f60e406b4d0 0x7f60e406d990 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:07.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.651+0000 7f610ec00640 1 --2- 172.21.15.6:0/455716568 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f61080c8d20 0x7f61081584a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:07.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.651+0000 7f610ec00640 1 -- 172.21.15.6:0/455716568 >> 172.21.15.6:0/455716568 conn(0x7f61080d3ee0 msgr2=0x7f610815f750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:07.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.651+0000 7f610ec00640 1 -- 172.21.15.6:0/455716568 shutdown_connections 2024-03-04T09:27:07.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.651+0000 7f610ec00640 1 -- 172.21.15.6:0/455716568 wait complete. 2024-03-04T09:27:07.665 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_dir_deletion' 2024-03-04T09:27:07.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.882+0000 7fd63a800640 1 Processor -- start 2024-03-04T09:27:07.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.883+0000 7fd63a800640 1 -- start start 2024-03-04T09:27:07.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.884+0000 7fd63a800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd6340cede0 0x7fd6340b3f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:07.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.884+0000 7fd63a800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd6340b4a70 0x7fd63415b320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:07.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.884+0000 7fd63a800640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6340b31d0 con 0x7fd634071450 2024-03-04T09:27:07.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.884+0000 7fd63a800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6340b4ed0 con 0x7fd6340b4a70 2024-03-04T09:27:07.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.884+0000 7fd63a800640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6340b3b00 con 0x7fd6340cede0 2024-03-04T09:27:07.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.884+0000 7fd638a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd6340b4a70 0x7fd63415b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:07.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.884+0000 7fd638a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd6340b4a70 0x7fd63415b320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49798/0 (socket says 172.21.15.6:49798) 2024-03-04T09:27:07.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.884+0000 7fd638a00640 1 -- 172.21.15.6:0/1500499065 learned_addr learned my addr 172.21.15.6:0/1500499065 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:27:07.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.885+0000 7fd639400640 1 --2- 172.21.15.6:0/1500499065 >> v2:172.21.15.146:3300/0 conn(0x7fd6340cede0 0x7fd6340b3f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:07.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.885+0000 7fd638a00640 1 -- 172.21.15.6:0/1500499065 >> v2:172.21.15.146:3300/0 conn(0x7fd6340cede0 msgr2=0x7fd6340b3f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:27:07.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.885+0000 7fd638a00640 1 --2- 172.21.15.6:0/1500499065 >> v2:172.21.15.146:3300/0 conn(0x7fd6340cede0 0x7fd6340b3f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:07.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.885+0000 7fd638a00640 1 -- 172.21.15.6:0/1500499065 >> v1:172.21.15.6:6789/0 conn(0x7fd634071450 legacy=0x7fd634150b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:07.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.885+0000 7fd638a00640 1 -- 172.21.15.6:0/1500499065 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6340b34b0 con 0x7fd6340b4a70 2024-03-04T09:27:07.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.885+0000 7fd639400640 1 --2- 172.21.15.6:0/1500499065 >> v2:172.21.15.146:3300/0 conn(0x7fd6340cede0 0x7fd6340b3f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:27:07.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.886+0000 7fd638a00640 1 --2- 172.21.15.6:0/1500499065 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd6340b4a70 0x7fd63415b320 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fd634069240 tx=0x7fd62002e0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de9254673780d136 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:07.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.886+0000 7fd633e00640 1 -- 172.21.15.6:0/1500499065 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd62003b070 con 0x7fd6340b4a70 2024-03-04T09:27:07.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.886+0000 7fd633e00640 1 -- 172.21.15.6:0/1500499065 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd620036440 con 0x7fd6340b4a70 2024-03-04T09:27:07.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.886+0000 7fd633e00640 1 -- 172.21.15.6:0/1500499065 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd620036810 con 0x7fd6340b4a70 2024-03-04T09:27:07.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.887+0000 7fd63a800640 1 -- 172.21.15.6:0/1500499065 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd6340b4a70 msgr2=0x7fd63415b320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:07.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.887+0000 7fd63a800640 1 --2- 172.21.15.6:0/1500499065 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd6340b4a70 0x7fd63415b320 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fd634069240 tx=0x7fd62002e0f0 comp rx=0 tx=0).stop 2024-03-04T09:27:07.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.887+0000 7fd63a800640 1 -- 172.21.15.6:0/1500499065 shutdown_connections 2024-03-04T09:27:07.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.887+0000 7fd63a800640 1 --2- 172.21.15.6:0/1500499065 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd6340b4a70 0x7fd63415b320 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:07.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.887+0000 7fd63a800640 1 --2- 172.21.15.6:0/1500499065 >> v2:172.21.15.146:3300/0 conn(0x7fd6340cede0 0x7fd6340b3f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:07.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.887+0000 7fd63a800640 1 -- 172.21.15.6:0/1500499065 >> 172.21.15.6:0/1500499065 conn(0x7fd63406b3d0 msgr2=0x7fd63406b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:07.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.887+0000 7fd63a800640 1 -- 172.21.15.6:0/1500499065 shutdown_connections 2024-03-04T09:27:07.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.887+0000 7fd63a800640 1 -- 172.21.15.6:0/1500499065 wait complete. 2024-03-04T09:27:07.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.889+0000 7fd63a800640 1 Processor -- start 2024-03-04T09:27:07.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.889+0000 7fd63a800640 1 -- start start 2024-03-04T09:27:07.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.890+0000 7fd63a800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd634071450 0x7fd6341495d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:07.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.890+0000 7fd63a800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd6340b4a70 0x7fd6340b0b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:07.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.890+0000 7fd63a800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd6340cede0 0x7fd6341efd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:07.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.890+0000 7fd63a800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6340cf1c0 con 0x7fd6340b4a70 2024-03-04T09:27:07.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.890+0000 7fd63a800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6340b2f70 con 0x7fd634071450 2024-03-04T09:27:07.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.890+0000 7fd63a800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6340718e0 con 0x7fd6340cede0 2024-03-04T09:27:07.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.890+0000 7fd638a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd6340b4a70 0x7fd6340b0b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:07.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.891+0000 7fd638a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd6340b4a70 0x7fd6340b0b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49806/0 (socket says 172.21.15.6:49806) 2024-03-04T09:27:07.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.891+0000 7fd638a00640 1 -- 172.21.15.6:0/4190729434 learned_addr learned my addr 172.21.15.6:0/4190729434 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:27:07.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.891+0000 7fd639400640 1 --2- 172.21.15.6:0/4190729434 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd634071450 0x7fd6341495d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:07.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.891+0000 7fd639e00640 1 --2- 172.21.15.6:0/4190729434 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd6340cede0 0x7fd6341efd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:07.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.891+0000 7fd638a00640 1 -- 172.21.15.6:0/4190729434 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd6340cede0 msgr2=0x7fd6341efd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:07.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.891+0000 7fd638a00640 1 --2- 172.21.15.6:0/4190729434 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd6340cede0 0x7fd6341efd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:07.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.891+0000 7fd638a00640 1 -- 172.21.15.6:0/4190729434 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd634071450 msgr2=0x7fd6341495d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:07.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.891+0000 7fd638a00640 1 --2- 172.21.15.6:0/4190729434 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd634071450 0x7fd6341495d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:07.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.891+0000 7fd638a00640 1 -- 172.21.15.6:0/4190729434 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6341f0500 con 0x7fd6340b4a70 2024-03-04T09:27:07.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.891+0000 7fd639400640 1 --2- 172.21.15.6:0/4190729434 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd634071450 0x7fd6341495d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T09:27:07.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.891+0000 7fd638a00640 1 --2- 172.21.15.6:0/4190729434 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd6340b4a70 0x7fd6340b0b10 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fd62002e130 tx=0x7fd6200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:07.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.892+0000 7fd631600640 1 -- 172.21.15.6:0/4190729434 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd62003b040 con 0x7fd6340b4a70 2024-03-04T09:27:07.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.892+0000 7fd631600640 1 -- 172.21.15.6:0/4190729434 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd620046070 con 0x7fd6340b4a70 2024-03-04T09:27:07.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.892+0000 7fd631600640 1 -- 172.21.15.6:0/4190729434 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd620034bb0 con 0x7fd6340b4a70 2024-03-04T09:27:07.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.892+0000 7fd63a800640 1 -- 172.21.15.6:0/4190729434 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6341f07e0 con 0x7fd6340b4a70 2024-03-04T09:27:07.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.892+0000 7fd63a800640 1 -- 172.21.15.6:0/4190729434 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd63415c260 con 0x7fd6340b4a70 2024-03-04T09:27:07.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.893+0000 7fd63a800640 1 -- 172.21.15.6:0/4190729434 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6341f0d00 con 0x7fd6340b4a70 2024-03-04T09:27:07.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.894+0000 7fd631600640 1 -- 172.21.15.6:0/4190729434 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd62004b460 con 0x7fd6340b4a70 2024-03-04T09:27:07.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.895+0000 7fd631600640 1 --2- 172.21.15.6:0/4190729434 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd61406b4d0 0x7fd61406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:07.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.895+0000 7fd631600640 1 -- 172.21.15.6:0/4190729434 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fd6200aee00 con 0x7fd6340b4a70 2024-03-04T09:27:07.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.896+0000 7fd639400640 1 --2- 172.21.15.6:0/4190729434 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd61406b4d0 0x7fd61406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:07.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.897+0000 7fd639400640 1 --2- 172.21.15.6:0/4190729434 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd61406b4d0 0x7fd61406d990 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fd6340b1910 tx=0x7fd628008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:07.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:07.899+0000 7fd631600640 1 -- 172.21.15.6:0/4190729434 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd6341f0d00 con 0x7fd6340b4a70 2024-03-04T09:27:08.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.173+0000 7fd63a800640 1 -- 172.21.15.6:0/4190729434 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_dir_deletion"]} v 0) v1 -- 0x7fd63415bb40 con 0x7fd6340b4a70 2024-03-04T09:27:08.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.206+0000 7fd631600640 1 -- 172.21.15.6:0/4190729434 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_dir_deletion"]}]=0 v121) v1 ==== 132+0+0 (secure 0 0 0) 0x7fd620049360 con 0x7fd6340b4a70 2024-03-04T09:27:08.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.209+0000 7fd63a800640 1 -- 172.21.15.6:0/4190729434 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd61406b4d0 msgr2=0x7fd61406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:08.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.209+0000 7fd63a800640 1 --2- 172.21.15.6:0/4190729434 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd61406b4d0 0x7fd61406d990 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fd6340b1910 tx=0x7fd628008040 comp rx=0 tx=0).stop 2024-03-04T09:27:08.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.209+0000 7fd63a800640 1 -- 172.21.15.6:0/4190729434 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd6340b4a70 msgr2=0x7fd6340b0b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:08.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.209+0000 7fd63a800640 1 --2- 172.21.15.6:0/4190729434 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd6340b4a70 0x7fd6340b0b10 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fd62002e130 tx=0x7fd6200046c0 comp rx=0 tx=0).stop 2024-03-04T09:27:08.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.209+0000 7fd63a800640 1 -- 172.21.15.6:0/4190729434 shutdown_connections 2024-03-04T09:27:08.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.209+0000 7fd63a800640 1 --2- 172.21.15.6:0/4190729434 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd61406b4d0 0x7fd61406d990 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:08.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.209+0000 7fd63a800640 1 --2- 172.21.15.6:0/4190729434 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd6340cede0 0x7fd6341efd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:08.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.209+0000 7fd63a800640 1 --2- 172.21.15.6:0/4190729434 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd6340b4a70 0x7fd6340b0b10 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:08.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.209+0000 7fd63a800640 1 --2- 172.21.15.6:0/4190729434 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd634071450 0x7fd6341495d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:08.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.209+0000 7fd63a800640 1 -- 172.21.15.6:0/4190729434 >> 172.21.15.6:0/4190729434 conn(0x7fd63406b3d0 msgr2=0x7fd6340cd930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:08.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.209+0000 7fd63a800640 1 -- 172.21.15.6:0/4190729434 shutdown_connections 2024-03-04T09:27:08.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.209+0000 7fd63a800640 1 -- 172.21.15.6:0/4190729434 wait complete. 2024-03-04T09:27:08.227 INFO:tasks.cephfs_test_runner:That when deleting a bunch of dentries and the containing ... ok 2024-03-04T09:27:08.227 INFO:tasks.cephfs_test_runner:Starting test: test_dirfrag_limit (tasks.cephfs.test_strays.TestStrays) That the directory fragment size cannot exceed mds_bal_fragment_size_max (using a limit of 50 in all configurations). 2024-03-04T09:27:08.227 INFO:tasks.cephfs_test_runner:test_dirfrag_limit (tasks.cephfs.test_strays.TestStrays) 2024-03-04T09:27:08.228 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T09:27:08.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.457+0000 7f0e8fe00640 1 Processor -- start 2024-03-04T09:27:08.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.457+0000 7f0e8fe00640 1 -- start start 2024-03-04T09:27:08.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.459+0000 7f0e8fe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f0e88161b40 0x7f0e880cfd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:08.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.459+0000 7f0e8fe00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0e880d2930 0x7f0e880b3740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:08.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.459+0000 7f0e8fe00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e880cea50 con 0x7f0e880b5060 2024-03-04T09:27:08.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.459+0000 7f0e8fe00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e880cf900 con 0x7f0e88161b40 2024-03-04T09:27:08.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.459+0000 7f0e8fe00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e880cecf0 con 0x7f0e880d2930 2024-03-04T09:27:08.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.459+0000 7f0e8d400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f0e88161b40 0x7f0e880cfd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:08.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.459+0000 7f0e8ca00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f0e880b5060 0x7f0e880d2220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:39066/0 (socket says 172.21.15.6:39066) 2024-03-04T09:27:08.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.459+0000 7f0e8ca00640 1 -- 172.21.15.6:0/284069573 learned_addr learned my addr 172.21.15.6:0/284069573 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:27:08.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.459+0000 7f0e8de00640 1 --2- 172.21.15.6:0/284069573 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0e880d2930 0x7f0e880b3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:08.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.460+0000 7f0e8d400640 1 -- 172.21.15.6:0/284069573 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0e880d2930 msgr2=0x7f0e880b3740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:27:08.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.460+0000 7f0e8d400640 1 --2- 172.21.15.6:0/284069573 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0e880d2930 0x7f0e880b3740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:08.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.460+0000 7f0e8d400640 1 -- 172.21.15.6:0/284069573 >> v1:172.21.15.6:6789/0 conn(0x7f0e880b5060 legacy=0x7f0e880d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:08.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.460+0000 7f0e8d400640 1 -- 172.21.15.6:0/284069573 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e880cefd0 con 0x7f0e88161b40 2024-03-04T09:27:08.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.460+0000 7f0e87e00640 1 -- 172.21.15.6:0/284069573 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1682780719 0 0) 0x7f0e880cea50 con 0x7f0e880b5060 2024-03-04T09:27:08.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.461+0000 7f0e8d400640 1 --2- 172.21.15.6:0/284069573 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0e88161b40 0x7f0e880cfd90 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f0e7c009d90 tx=0x7f0e7c02bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64002ea0ffeb8c5c server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:08.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.461+0000 7f0e87e00640 1 -- 172.21.15.6:0/284069573 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e7c039070 con 0x7f0e88161b40 2024-03-04T09:27:08.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.461+0000 7f0e87e00640 1 -- 172.21.15.6:0/284069573 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e7c02cc50 con 0x7f0e88161b40 2024-03-04T09:27:08.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.461+0000 7f0e87e00640 1 -- 172.21.15.6:0/284069573 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e7c034440 con 0x7f0e88161b40 2024-03-04T09:27:08.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.462+0000 7f0e8fe00640 1 -- 172.21.15.6:0/284069573 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0e88161b40 msgr2=0x7f0e880cfd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:08.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.462+0000 7f0e8fe00640 1 --2- 172.21.15.6:0/284069573 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0e88161b40 0x7f0e880cfd90 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f0e7c009d90 tx=0x7f0e7c02bea0 comp rx=0 tx=0).stop 2024-03-04T09:27:08.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.462+0000 7f0e8fe00640 1 -- 172.21.15.6:0/284069573 shutdown_connections 2024-03-04T09:27:08.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.462+0000 7f0e8fe00640 1 --2- 172.21.15.6:0/284069573 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0e880d2930 0x7f0e880b3740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:08.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.462+0000 7f0e8fe00640 1 --2- 172.21.15.6:0/284069573 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0e88161b40 0x7f0e880cfd90 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:08.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.462+0000 7f0e8fe00640 1 -- 172.21.15.6:0/284069573 >> 172.21.15.6:0/284069573 conn(0x7f0e8809d220 msgr2=0x7f0e880d3ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:08.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.462+0000 7f0e8fe00640 1 -- 172.21.15.6:0/284069573 shutdown_connections 2024-03-04T09:27:08.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.462+0000 7f0e8fe00640 1 -- 172.21.15.6:0/284069573 wait complete. 2024-03-04T09:27:08.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.464+0000 7f0e8fe00640 1 Processor -- start 2024-03-04T09:27:08.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.465+0000 7f0e8fe00640 1 -- start start 2024-03-04T09:27:08.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.466+0000 7f0e8fe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0e880b5060 0x7f0e8814a150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:08.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.466+0000 7f0e8fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0e880d2930 0x7f0e880ca190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:08.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.466+0000 7f0e8fe00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0e88161b40 0x7f0e880b3590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:08.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.466+0000 7f0e8fe00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e880b2ff0 con 0x7f0e880d2930 2024-03-04T09:27:08.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.466+0000 7f0e8fe00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e880cf590 con 0x7f0e880b5060 2024-03-04T09:27:08.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.466+0000 7f0e8fe00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e880b3170 con 0x7f0e88161b40 2024-03-04T09:27:08.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.466+0000 7f0e8ca00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0e880d2930 0x7f0e880ca190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:08.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.466+0000 7f0e8ca00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0e880d2930 0x7f0e880ca190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49812/0 (socket says 172.21.15.6:49812) 2024-03-04T09:27:08.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.466+0000 7f0e8ca00640 1 -- 172.21.15.6:0/1024060210 learned_addr learned my addr 172.21.15.6:0/1024060210 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:27:08.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.467+0000 7f0e8d400640 1 --2- 172.21.15.6:0/1024060210 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0e880b5060 0x7f0e8814a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:08.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.467+0000 7f0e8de00640 1 --2- 172.21.15.6:0/1024060210 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0e88161b40 0x7f0e880b3590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:08.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.467+0000 7f0e8ca00640 1 -- 172.21.15.6:0/1024060210 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0e88161b40 msgr2=0x7f0e880b3590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:08.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.467+0000 7f0e8ca00640 1 --2- 172.21.15.6:0/1024060210 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0e88161b40 0x7f0e880b3590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:08.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.467+0000 7f0e8ca00640 1 -- 172.21.15.6:0/1024060210 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0e880b5060 msgr2=0x7f0e8814a150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:08.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.467+0000 7f0e8ca00640 1 --2- 172.21.15.6:0/1024060210 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0e880b5060 0x7f0e8814a150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:08.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.467+0000 7f0e8ca00640 1 -- 172.21.15.6:0/1024060210 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e8814a980 con 0x7f0e880d2930 2024-03-04T09:27:08.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.467+0000 7f0e8ca00640 1 --2- 172.21.15.6:0/1024060210 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0e880d2930 0x7f0e880ca190 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f0e7400c4d0 tx=0x7f0e7400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:08.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.467+0000 7f0e85600640 1 -- 172.21.15.6:0/1024060210 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e74018070 con 0x7f0e880d2930 2024-03-04T09:27:08.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.468+0000 7f0e85600640 1 -- 172.21.15.6:0/1024060210 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e74007500 con 0x7f0e880d2930 2024-03-04T09:27:08.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.468+0000 7f0e85600640 1 -- 172.21.15.6:0/1024060210 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e74004a20 con 0x7f0e880d2930 2024-03-04T09:27:08.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.468+0000 7f0e8fe00640 1 -- 172.21.15.6:0/1024060210 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e8814ac60 con 0x7f0e880d2930 2024-03-04T09:27:08.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.468+0000 7f0e8fe00640 1 -- 172.21.15.6:0/1024060210 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e88148320 con 0x7f0e880d2930 2024-03-04T09:27:08.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.469+0000 7f0e8fe00640 1 -- 172.21.15.6:0/1024060210 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e8804f3a0 con 0x7f0e880d2930 2024-03-04T09:27:08.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.470+0000 7f0e85600640 1 -- 172.21.15.6:0/1024060210 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0e7401b070 con 0x7f0e880d2930 2024-03-04T09:27:08.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.471+0000 7f0e85600640 1 --2- 172.21.15.6:0/1024060210 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f0e6806b4d0 0x7f0e6806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:08.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.471+0000 7f0e85600640 1 -- 172.21.15.6:0/1024060210 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f0e74089390 con 0x7f0e880d2930 2024-03-04T09:27:08.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.471+0000 7f0e8d400640 1 --2- 172.21.15.6:0/1024060210 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f0e6806b4d0 0x7f0e6806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:08.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.473+0000 7f0e8d400640 1 --2- 172.21.15.6:0/1024060210 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f0e6806b4d0 0x7f0e6806d990 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f0e7c036000 tx=0x7f0e7c030000 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:08.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.475+0000 7f0e85600640 1 -- 172.21.15.6:0/1024060210 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0e740583e0 con 0x7f0e880d2930 2024-03-04T09:27:08.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.678+0000 7f0e8fe00640 1 -- 172.21.15.6:0/1024060210 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0e880d1df0 con 0x7f0e880d2930 2024-03-04T09:27:08.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.680+0000 7f0e85600640 1 -- 172.21.15.6:0/1024060210 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v36) v1 ==== 74+0+12974 (secure 0 0 0) 0x7f0e74057ce0 con 0x7f0e880d2930 2024-03-04T09:27:08.677 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:27:08.677 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":36,"fsid":"edce0578-5ae4-4402-b45a-552bdef84d7c","created":"2024-03-04T09:24:54.523780+0000","modified":"2024-03-04T09:27:07.188466+0000","last_up_change":"2024-03-04T09:25:07.089049+0000","last_in_change":"2024-03-04T09:24:57.982077+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T09:25:07.935198+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a96574f-81f1-4bc6-8463-757520b84a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6825","nonce":1364449401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6827","nonce":1364449401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6831","nonce":1364449401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6829","nonce":1364449401}]},"public_addr":"172.21.15.6:6825/1364449401","cluster_addr":"172.21.15.6:6827/1364449401","heartbeat_back_addr":"172.21.15.6:6831/1364449401","heartbeat_front_addr":"172.21.15.6:6829/1364449401","state":["exists","up"]},{"osd":1,"uuid":"7d79b9c1-6b87-476a-bc36-7db5ee64cd90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6809","nonce":2494103078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6811","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6813","nonce":2494103078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6821","nonce":2494103078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6817","nonce":2494103078}]},"public_addr":"172.21.15.6:6809/2494103078","cluster_addr":"172.21.15.6:6813/2494103078","heartbeat_back_addr":"172.21.15.6:6821/2494103078","heartbeat_front_addr":"172.21.15.6:6817/2494103078","state":["exists","up"]},{"osd":2,"uuid":"492331b9-4ee9-40b6-bacd-85d44cd6c45a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6812","nonce":1500066681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6816","nonce":1500066681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6824","nonce":1500066681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6820","nonce":1500066681}]},"public_addr":"172.21.15.6:6812/1500066681","cluster_addr":"172.21.15.6:6816/1500066681","heartbeat_back_addr":"172.21.15.6:6824/1500066681","heartbeat_front_addr":"172.21.15.6:6820/1500066681","state":["exists","up"]},{"osd":3,"uuid":"21bb2efe-9ba7-4bdc-bc31-180cd1102e21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6801","nonce":1558806123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6803","nonce":1558806123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6807","nonce":1558806123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6805","nonce":1558806123}]},"public_addr":"172.21.15.6:6801/1558806123","cluster_addr":"172.21.15.6:6803/1558806123","heartbeat_back_addr":"172.21.15.6:6807/1558806123","heartbeat_front_addr":"172.21.15.6:6805/1558806123","state":["exists","up"]},{"osd":4,"uuid":"6e2c4b71-19d9-4737-83d9-558705d8644a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6801","nonce":1526135780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6803","nonce":1526135780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6807","nonce":1526135780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6805","nonce":1526135780}]},"public_addr":"172.21.15.146:6801/1526135780","cluster_addr":"172.21.15.146:6803/1526135780","heartbeat_back_addr":"172.21.15.146:6807/1526135780","heartbeat_front_addr":"172.21.15.146:6805/1526135780","state":["exists","up"]},{"osd":5,"uuid":"fb69c525-f1be-4b7d-ab32-6c29b0d6fddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6809","nonce":460262507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6811","nonce":460262507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6815","nonce":460262507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6813","nonce":460262507}]},"public_addr":"172.21.15.146:6809/460262507","cluster_addr":"172.21.15.146:6811/460262507","heartbeat_back_addr":"172.21.15.146:6815/460262507","heartbeat_front_addr":"172.21.15.146:6813/460262507","state":["exists","up"]},{"osd":6,"uuid":"c32e7fed-370d-4401-8ce0-6a4335b0d168","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6823","nonce":1392897493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6827","nonce":1392897493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6831","nonce":1392897493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6829","nonce":1392897493}]},"public_addr":"172.21.15.146:6823/1392897493","cluster_addr":"172.21.15.146:6827/1392897493","heartbeat_back_addr":"172.21.15.146:6831/1392897493","heartbeat_front_addr":"172.21.15.146:6829/1392897493","state":["exists","up"]},{"osd":7,"uuid":"75a79b44-fa5f-426d-a7d3-71f21f78835c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6817","nonce":3268323156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6819","nonce":3268323156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6825","nonce":3268323156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6822","nonce":3268323156}]},"public_addr":"172.21.15.146:6817/3268323156","cluster_addr":"172.21.15.146:6819/3268323156","heartbeat_back_addr":"172.21.15.146:6825/3268323156","heartbeat_front_addr":"172.21.15.146:6822/3268323156","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.310437+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.295850+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.294674+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.526855+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.300023+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.595160+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.346046+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.558741+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.146:6836/2285803221":"2024-03-05T09:27:03.821661+0000","172.21.15.146:6834/2285803221":"2024-03-05T09:27:03.821661+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T09:27:08.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.682+0000 7f0e8fe00640 1 -- 172.21.15.6:0/1024060210 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f0e6806b4d0 msgr2=0x7f0e6806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:08.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.682+0000 7f0e8fe00640 1 --2- 172.21.15.6:0/1024060210 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f0e6806b4d0 0x7f0e6806d990 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f0e7c036000 tx=0x7f0e7c030000 comp rx=0 tx=0).stop 2024-03-04T09:27:08.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.682+0000 7f0e8fe00640 1 -- 172.21.15.6:0/1024060210 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0e880d2930 msgr2=0x7f0e880ca190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:08.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.682+0000 7f0e8fe00640 1 --2- 172.21.15.6:0/1024060210 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0e880d2930 0x7f0e880ca190 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f0e7400c4d0 tx=0x7f0e7400c990 comp rx=0 tx=0).stop 2024-03-04T09:27:08.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.683+0000 7f0e8fe00640 1 -- 172.21.15.6:0/1024060210 shutdown_connections 2024-03-04T09:27:08.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.683+0000 7f0e8fe00640 1 --2- 172.21.15.6:0/1024060210 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f0e6806b4d0 0x7f0e6806d990 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:08.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.683+0000 7f0e8fe00640 1 --2- 172.21.15.6:0/1024060210 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0e88161b40 0x7f0e880b3590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:08.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.683+0000 7f0e8fe00640 1 --2- 172.21.15.6:0/1024060210 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0e880d2930 0x7f0e880ca190 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:08.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.683+0000 7f0e8fe00640 1 --2- 172.21.15.6:0/1024060210 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0e880b5060 0x7f0e8814a150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:08.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.683+0000 7f0e8fe00640 1 -- 172.21.15.6:0/1024060210 >> 172.21.15.6:0/1024060210 conn(0x7f0e8809d220 msgr2=0x7f0e8815d4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:08.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.683+0000 7f0e8fe00640 1 -- 172.21.15.6:0/1024060210 shutdown_connections 2024-03-04T09:27:08.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.683+0000 7f0e8fe00640 1 -- 172.21.15.6:0/1024060210 wait complete. 2024-03-04T09:27:08.698 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T09:25:07.935198+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-04T09:27:08.699 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-04T09:27:08.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.925+0000 7f0552a00640 1 Processor -- start 2024-03-04T09:27:08.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.925+0000 7f0552a00640 1 -- start start 2024-03-04T09:27:08.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.927+0000 7f0552a00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f054c15afa0 0x7f054c0cf3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:08.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.927+0000 7f0552a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f054c0cd1e0 0x7f054c0b3ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:08.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.927+0000 7f0552a00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f054c071c60 con 0x7f054c070ce0 2024-03-04T09:27:08.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.927+0000 7f0552a00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f054c0cfa10 con 0x7f054c15afa0 2024-03-04T09:27:08.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.927+0000 7f0552a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f054c071f40 con 0x7f054c0cd1e0 2024-03-04T09:27:08.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.928+0000 7f054be00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f054c15afa0 0x7f054c0cf3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:08.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.928+0000 7f054b400640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f054c070ce0 0x7f054c0ccad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.6:41520/0 (socket says 172.21.15.6:41520) 2024-03-04T09:27:08.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.928+0000 7f054b400640 1 -- 172.21.15.6:0/2337067003 learned_addr learned my addr 172.21.15.6:0/2337067003 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:27:08.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.928+0000 7f0550a00640 1 --2- 172.21.15.6:0/2337067003 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f054c0cd1e0 0x7f054c0b3ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:08.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.929+0000 7f054be00640 1 -- 172.21.15.6:0/2337067003 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f054c0cd1e0 msgr2=0x7f054c0b3ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:27:08.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.929+0000 7f054be00640 1 --2- 172.21.15.6:0/2337067003 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f054c0cd1e0 0x7f054c0b3ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:08.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.929+0000 7f054be00640 1 -- 172.21.15.6:0/2337067003 >> v1:172.21.15.146:6789/0 conn(0x7f054c070ce0 legacy=0x7f054c0ccad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:08.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.929+0000 7f054be00640 1 -- 172.21.15.6:0/2337067003 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f054c072220 con 0x7f054c15afa0 2024-03-04T09:27:08.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.930+0000 7f054be00640 1 --2- 172.21.15.6:0/2337067003 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f054c15afa0 0x7f054c0cf3a0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f053c009d90 tx=0x7f053c02e030 comp rx=0 tx=0).ready entity=mon.1 client_cookie=59c1aea48e5c0bb7 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:08.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.931+0000 7f054aa00640 1 -- 172.21.15.6:0/2337067003 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f053c03b070 con 0x7f054c15afa0 2024-03-04T09:27:08.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.931+0000 7f054aa00640 1 -- 172.21.15.6:0/2337067003 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f053c02ede0 con 0x7f054c15afa0 2024-03-04T09:27:08.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.931+0000 7f054aa00640 1 -- 172.21.15.6:0/2337067003 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f053c036660 con 0x7f054c15afa0 2024-03-04T09:27:08.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.931+0000 7f0552a00640 1 -- 172.21.15.6:0/2337067003 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f054c15afa0 msgr2=0x7f054c0cf3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:08.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.931+0000 7f0552a00640 1 --2- 172.21.15.6:0/2337067003 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f054c15afa0 0x7f054c0cf3a0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f053c009d90 tx=0x7f053c02e030 comp rx=0 tx=0).stop 2024-03-04T09:27:08.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.931+0000 7f0552a00640 1 -- 172.21.15.6:0/2337067003 shutdown_connections 2024-03-04T09:27:08.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.931+0000 7f0552a00640 1 --2- 172.21.15.6:0/2337067003 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f054c0cd1e0 0x7f054c0b3ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:08.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.931+0000 7f0552a00640 1 --2- 172.21.15.6:0/2337067003 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f054c15afa0 0x7f054c0cf3a0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:08.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.932+0000 7f0552a00640 1 -- 172.21.15.6:0/2337067003 >> 172.21.15.6:0/2337067003 conn(0x7f054c06b3d0 msgr2=0x7f054c06b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:08.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.932+0000 7f0552a00640 1 -- 172.21.15.6:0/2337067003 shutdown_connections 2024-03-04T09:27:08.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.932+0000 7f0552a00640 1 -- 172.21.15.6:0/2337067003 wait complete. 2024-03-04T09:27:08.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.934+0000 7f0552a00640 1 Processor -- start 2024-03-04T09:27:08.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.934+0000 7f0552a00640 1 -- start start 2024-03-04T09:27:08.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.935+0000 7f0552a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f054c070ce0 0x7f054c152900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:08.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.936+0000 7f0552a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f054c0cd1e0 0x7f054c150970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:08.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.936+0000 7f0552a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f054c15afa0 0x7f054c13f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:08.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.936+0000 7f0552a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f054c0b4000 con 0x7f054c070ce0 2024-03-04T09:27:08.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.936+0000 7f0552a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f054c071170 con 0x7f054c15afa0 2024-03-04T09:27:08.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.936+0000 7f0552a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f054c0b4180 con 0x7f054c0cd1e0 2024-03-04T09:27:08.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.936+0000 7f054be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f054c070ce0 0x7f054c152900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:08.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.936+0000 7f054be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f054c070ce0 0x7f054c152900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37978/0 (socket says 172.21.15.6:37978) 2024-03-04T09:27:08.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.936+0000 7f054be00640 1 -- 172.21.15.6:0/1082242491 learned_addr learned my addr 172.21.15.6:0/1082242491 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:27:08.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.936+0000 7f054b400640 1 --2- 172.21.15.6:0/1082242491 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f054c0cd1e0 0x7f054c150970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:08.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.936+0000 7f0550a00640 1 --2- 172.21.15.6:0/1082242491 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f054c15afa0 0x7f054c13f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:08.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.936+0000 7f054be00640 1 -- 172.21.15.6:0/1082242491 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f054c0cd1e0 msgr2=0x7f054c150970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:08.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.936+0000 7f054be00640 1 --2- 172.21.15.6:0/1082242491 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f054c0cd1e0 0x7f054c150970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:08.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.936+0000 7f054be00640 1 -- 172.21.15.6:0/1082242491 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f054c15afa0 msgr2=0x7f054c13f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:08.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.936+0000 7f054be00640 1 --2- 172.21.15.6:0/1082242491 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f054c15afa0 0x7f054c13f310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:08.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.937+0000 7f054be00640 1 -- 172.21.15.6:0/1082242491 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f054c152e40 con 0x7f054c070ce0 2024-03-04T09:27:08.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.937+0000 7f054be00640 1 --2- 172.21.15.6:0/1082242491 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f054c070ce0 0x7f054c152900 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f053c039040 tx=0x7f053c004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:08.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.937+0000 7f0543e00640 1 -- 172.21.15.6:0/1082242491 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f053c03b040 con 0x7f054c070ce0 2024-03-04T09:27:08.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.937+0000 7f0543e00640 1 -- 172.21.15.6:0/1082242491 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f053c044070 con 0x7f054c070ce0 2024-03-04T09:27:08.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.937+0000 7f0543e00640 1 -- 172.21.15.6:0/1082242491 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f053c03f6a0 con 0x7f054c070ce0 2024-03-04T09:27:08.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.937+0000 7f0552a00640 1 -- 172.21.15.6:0/1082242491 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f054c1f79a0 con 0x7f054c070ce0 2024-03-04T09:27:08.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.937+0000 7f0552a00640 1 -- 172.21.15.6:0/1082242491 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f054c1f7f60 con 0x7f054c070ce0 2024-03-04T09:27:08.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.939+0000 7f0552a00640 1 -- 172.21.15.6:0/1082242491 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f054c04f330 con 0x7f054c070ce0 2024-03-04T09:27:08.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.940+0000 7f0543e00640 1 -- 172.21.15.6:0/1082242491 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f053c02eca0 con 0x7f054c070ce0 2024-03-04T09:27:08.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.940+0000 7f0543e00640 1 --2- 172.21.15.6:0/1082242491 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f052806b4d0 0x7f052806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:08.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.941+0000 7f0543e00640 1 -- 172.21.15.6:0/1082242491 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f053c0abff0 con 0x7f054c070ce0 2024-03-04T09:27:08.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.941+0000 7f054b400640 1 --2- 172.21.15.6:0/1082242491 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f052806b4d0 0x7f052806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:08.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.942+0000 7f054b400640 1 --2- 172.21.15.6:0/1082242491 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f052806b4d0 0x7f052806d990 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f054c15baa0 tx=0x7f0534009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:08.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:08.945+0000 7f0543e00640 1 -- 172.21.15.6:0/1082242491 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f053c07b1a0 con 0x7f054c070ce0 2024-03-04T09:27:09.004 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:27:09.011 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:27:09.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:09.148+0000 7f0552a00640 1 -- 172.21.15.6:0/1082242491 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f054c0ccb40 con 0x7f054c070ce0 2024-03-04T09:27:09.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:09.149+0000 7f0543e00640 1 -- 172.21.15.6:0/1082242491 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v36) v1 ==== 93+0+10 (secure 0 0 0) 0x7f053c07aaa0 con 0x7f054c070ce0 2024-03-04T09:27:09.146 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-04T09:27:09.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:09.152+0000 7f0552a00640 1 -- 172.21.15.6:0/1082242491 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f052806b4d0 msgr2=0x7f052806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:09.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:09.152+0000 7f0552a00640 1 --2- 172.21.15.6:0/1082242491 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f052806b4d0 0x7f052806d990 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f054c15baa0 tx=0x7f0534009040 comp rx=0 tx=0).stop 2024-03-04T09:27:09.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:09.152+0000 7f0552a00640 1 -- 172.21.15.6:0/1082242491 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f054c070ce0 msgr2=0x7f054c152900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:09.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:09.152+0000 7f0552a00640 1 --2- 172.21.15.6:0/1082242491 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f054c070ce0 0x7f054c152900 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f053c039040 tx=0x7f053c004ab0 comp rx=0 tx=0).stop 2024-03-04T09:27:09.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:09.152+0000 7f0552a00640 1 -- 172.21.15.6:0/1082242491 shutdown_connections 2024-03-04T09:27:09.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:09.152+0000 7f0552a00640 1 --2- 172.21.15.6:0/1082242491 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f054c15afa0 0x7f054c13f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:09.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:09.152+0000 7f0552a00640 1 --2- 172.21.15.6:0/1082242491 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f052806b4d0 0x7f052806d990 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:09.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:09.152+0000 7f0552a00640 1 --2- 172.21.15.6:0/1082242491 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f054c0cd1e0 0x7f054c150970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:09.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:09.152+0000 7f0552a00640 1 --2- 172.21.15.6:0/1082242491 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f054c070ce0 0x7f054c152900 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:09.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:09.152+0000 7f0552a00640 1 -- 172.21.15.6:0/1082242491 >> 172.21.15.6:0/1082242491 conn(0x7f054c06b3d0 msgr2=0x7f054c0cdab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:09.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:09.152+0000 7f0552a00640 1 -- 172.21.15.6:0/1082242491 shutdown_connections 2024-03-04T09:27:09.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:09.152+0000 7f0552a00640 1 -- 172.21.15.6:0/1082242491 wait complete. 2024-03-04T09:27:09.167 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit' 2024-03-04T09:27:09.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:09.439+0000 7f924f000640 1 Processor -- start 2024-03-04T09:27:09.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:09.440+0000 7f924f000640 1 -- start start 2024-03-04T09:27:09.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:09.441+0000 7f924f000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f92480af1a0 0x7f924815c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:09.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:09.441+0000 7f924f000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f92480af580 0x7f92480b4c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:09.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:09.441+0000 7f924f000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f92480cdc20 0x7f92480b2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:09.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:09.441+0000 7f924f000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f924815cc40 con 0x7f92480cdc20 2024-03-04T09:27:09.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:09.441+0000 7f924f000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f924815cf20 con 0x7f92480af580 2024-03-04T09:27:09.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:09.441+0000 7f924f000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f924815d200 con 0x7f92480af1a0 2024-03-04T09:27:09.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:09.442+0000 7f924d000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f92480cdc20 0x7f92480b2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:09.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:09.442+0000 7f924d000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f92480cdc20 0x7f92480b2dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37982/0 (socket says 172.21.15.6:37982) 2024-03-04T09:27:09.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:09.442+0000 7f924d000640 1 -- 172.21.15.6:0/3483043399 learned_addr learned my addr 172.21.15.6:0/3483043399 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:27:09.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:09.442+0000 7f9247400640 1 --2- 172.21.15.6:0/3483043399 >> v2:172.21.15.146:3300/0 conn(0x7f92480af580 0x7f92480b4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:09.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:09.442+0000 7f9247e00640 1 --2- 172.21.15.6:0/3483043399 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f92480af1a0 0x7f924815c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:09.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:09.442+0000 7f924d000640 1 -- 172.21.15.6:0/3483043399 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f92480af1a0 msgr2=0x7f924815c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:27:09.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:09.442+0000 7f924d000640 1 --2- 172.21.15.6:0/3483043399 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f92480af1a0 0x7f924815c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:09.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:09.442+0000 7f924d000640 1 -- 172.21.15.6:0/3483043399 >> v2:172.21.15.146:3300/0 conn(0x7f92480af580 msgr2=0x7f92480b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:27:09.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:09.442+0000 7f924d000640 1 --2- 172.21.15.6:0/3483043399 >> v2:172.21.15.146:3300/0 conn(0x7f92480af580 0x7f92480b4c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:09.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:09.443+0000 7f924d000640 1 -- 172.21.15.6:0/3483043399 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f924815d4e0 con 0x7f92480cdc20 2024-03-04T09:27:09.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:09.444+0000 7f924d000640 1 --2- 172.21.15.6:0/3483043399 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f92480cdc20 0x7f92480b2dd0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f92480748c0 tx=0x7f924002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba7d0d7b660ee4ae server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:09.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:09.444+0000 7f9246a00640 1 -- 172.21.15.6:0/3483043399 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f924002e070 con 0x7f92480cdc20 2024-03-04T09:27:09.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:09.444+0000 7f9246a00640 1 -- 172.21.15.6:0/3483043399 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9240037070 con 0x7f92480cdc20 2024-03-04T09:27:09.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:09.444+0000 7f9246a00640 1 -- 172.21.15.6:0/3483043399 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9240031a00 con 0x7f92480cdc20 2024-03-04T09:27:09.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:09.445+0000 7f924f000640 1 -- 172.21.15.6:0/3483043399 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f92480cdc20 msgr2=0x7f92480b2dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:09.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:09.445+0000 7f924f000640 1 --2- 172.21.15.6:0/3483043399 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f92480cdc20 0x7f92480b2dd0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f92480748c0 tx=0x7f924002b880 comp rx=0 tx=0).stop 2024-03-04T09:27:09.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:09.445+0000 7f924f000640 1 -- 172.21.15.6:0/3483043399 shutdown_connections 2024-03-04T09:27:09.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:09.445+0000 7f924f000640 1 --2- 172.21.15.6:0/3483043399 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f92480cdc20 0x7f92480b2dd0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:09.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:09.445+0000 7f924f000640 1 --2- 172.21.15.6:0/3483043399 >> v2:172.21.15.146:3300/0 conn(0x7f92480af580 0x7f92480b4c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:09.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:09.445+0000 7f924f000640 1 --2- 172.21.15.6:0/3483043399 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f92480af1a0 0x7f924815c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:09.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:09.445+0000 7f924f000640 1 -- 172.21.15.6:0/3483043399 >> 172.21.15.6:0/3483043399 conn(0x7f924806afb0 msgr2=0x7f924806b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:09.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:09.445+0000 7f924f000640 1 -- 172.21.15.6:0/3483043399 shutdown_connections 2024-03-04T09:27:09.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:09.445+0000 7f924f000640 1 -- 172.21.15.6:0/3483043399 wait complete. 2024-03-04T09:27:09.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:09.447+0000 7f924f000640 1 Processor -- start 2024-03-04T09:27:09.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:09.447+0000 7f924f000640 1 -- start start 2024-03-04T09:27:09.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:09.448+0000 7f924f000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f92480af1a0 0x7f92481548d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:09.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:09.448+0000 7f924f000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f92480af580 0x7f92480cb140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:09.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:09.448+0000 7f924f000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f92480cdc20 0x7f92480cb680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:09.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:09.448+0000 7f924f000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92480b3490 con 0x7f92480af580 2024-03-04T09:27:09.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:09.448+0000 7f924f000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92480b3310 con 0x7f92480cdc20 2024-03-04T09:27:09.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:09.448+0000 7f924f000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92480b3610 con 0x7f92480af1a0 2024-03-04T09:27:09.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:09.448+0000 7f9247400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f92480af580 0x7f92480cb140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:09.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:09.448+0000 7f9247400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f92480af580 0x7f92480cb140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37990/0 (socket says 172.21.15.6:37990) 2024-03-04T09:27:09.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:09.449+0000 7f9247400640 1 -- 172.21.15.6:0/926513401 learned_addr learned my addr 172.21.15.6:0/926513401 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:27:09.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:09.449+0000 7f9247e00640 1 --2- 172.21.15.6:0/926513401 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f92480af1a0 0x7f92481548d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:09.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:09.449+0000 7f924d000640 1 --2- 172.21.15.6:0/926513401 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f92480cdc20 0x7f92480cb680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:09.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:09.449+0000 7f9247400640 1 -- 172.21.15.6:0/926513401 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f92480af1a0 msgr2=0x7f92481548d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:09.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:09.449+0000 7f9247400640 1 --2- 172.21.15.6:0/926513401 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f92480af1a0 0x7f92481548d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:09.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:09.449+0000 7f9247400640 1 -- 172.21.15.6:0/926513401 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f92480cdc20 msgr2=0x7f92480cb680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:09.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:09.449+0000 7f9247400640 1 --2- 172.21.15.6:0/926513401 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f92480cdc20 0x7f92480cb680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:09.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:09.449+0000 7f9247400640 1 -- 172.21.15.6:0/926513401 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92481f3d30 con 0x7f92480af580 2024-03-04T09:27:09.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:09.449+0000 7f9247400640 1 --2- 172.21.15.6:0/926513401 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f92480af580 0x7f92480cb140 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f9230007f20 tx=0x7f923000e570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:09.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:09.450+0000 7f923fe00640 1 -- 172.21.15.6:0/926513401 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9230014620 con 0x7f92480af580 2024-03-04T09:27:09.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:09.450+0000 7f923fe00640 1 -- 172.21.15.6:0/926513401 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9230019070 con 0x7f92480af580 2024-03-04T09:27:09.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:09.450+0000 7f923fe00640 1 -- 172.21.15.6:0/926513401 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9230014a00 con 0x7f92480af580 2024-03-04T09:27:09.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:09.450+0000 7f924f000640 1 -- 172.21.15.6:0/926513401 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92481f3fd0 con 0x7f92480af580 2024-03-04T09:27:09.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:09.450+0000 7f924f000640 1 -- 172.21.15.6:0/926513401 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92481f4590 con 0x7f92480af580 2024-03-04T09:27:09.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:09.451+0000 7f924f000640 1 -- 172.21.15.6:0/926513401 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9248069510 con 0x7f92480af580 2024-03-04T09:27:09.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:09.452+0000 7f923fe00640 1 -- 172.21.15.6:0/926513401 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9230014ba0 con 0x7f92480af580 2024-03-04T09:27:09.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:09.453+0000 7f923fe00640 1 --2- 172.21.15.6:0/926513401 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f922406b4d0 0x7f922406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:09.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:09.453+0000 7f923fe00640 1 -- 172.21.15.6:0/926513401 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f923008a8f0 con 0x7f92480af580 2024-03-04T09:27:09.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:09.454+0000 7f9247e00640 1 --2- 172.21.15.6:0/926513401 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f922406b4d0 0x7f922406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:09.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:09.455+0000 7f9247e00640 1 --2- 172.21.15.6:0/926513401 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f922406b4d0 0x7f922406d990 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f9238006fd0 tx=0x7f9238008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:09.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:09.458+0000 7f923fe00640 1 -- 172.21.15.6:0/926513401 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f92300599d0 con 0x7f92480af580 2024-03-04T09:27:09.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:09.728+0000 7f924f000640 1 -- 172.21.15.6:0/926513401 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit"]} v 0) v1 -- 0x7f924815c6a0 con 0x7f92480af580 2024-03-04T09:27:10.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:10.218+0000 7f923fe00640 1 -- 172.21.15.6:0/926513401 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit"]}]=0 v123) v1 ==== 136+0+0 (secure 0 0 0) 0x7f92300592d0 con 0x7f92480af580 2024-03-04T09:27:10.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:10.221+0000 7f924f000640 1 -- 172.21.15.6:0/926513401 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f922406b4d0 msgr2=0x7f922406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:10.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:10.221+0000 7f924f000640 1 --2- 172.21.15.6:0/926513401 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f922406b4d0 0x7f922406d990 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f9238006fd0 tx=0x7f9238008040 comp rx=0 tx=0).stop 2024-03-04T09:27:10.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:10.221+0000 7f924f000640 1 -- 172.21.15.6:0/926513401 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f92480af580 msgr2=0x7f92480cb140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:10.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:10.221+0000 7f924f000640 1 --2- 172.21.15.6:0/926513401 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f92480af580 0x7f92480cb140 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f9230007f20 tx=0x7f923000e570 comp rx=0 tx=0).stop 2024-03-04T09:27:10.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:10.222+0000 7f924f000640 1 -- 172.21.15.6:0/926513401 shutdown_connections 2024-03-04T09:27:10.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:10.222+0000 7f924f000640 1 --2- 172.21.15.6:0/926513401 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f92480cdc20 0x7f92480cb680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:10.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:10.222+0000 7f924f000640 1 --2- 172.21.15.6:0/926513401 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f92480af580 0x7f92480cb140 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:10.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:10.222+0000 7f924f000640 1 --2- 172.21.15.6:0/926513401 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f922406b4d0 0x7f922406d990 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:10.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:10.222+0000 7f924f000640 1 --2- 172.21.15.6:0/926513401 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f92480af1a0 0x7f92481548d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:10.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:10.222+0000 7f924f000640 1 -- 172.21.15.6:0/926513401 >> 172.21.15.6:0/926513401 conn(0x7f924806afb0 msgr2=0x7f92480ccf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:10.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:10.222+0000 7f924f000640 1 -- 172.21.15.6:0/926513401 shutdown_connections 2024-03-04T09:27:10.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:10.222+0000 7f924f000640 1 -- 172.21.15.6:0/926513401 wait complete. 2024-03-04T09:27:10.240 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-04T09:27:10.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:10.471+0000 7f7367200640 1 Processor -- start 2024-03-04T09:27:10.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:10.471+0000 7f7367200640 1 -- start start 2024-03-04T09:27:10.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:10.473+0000 7f7367200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f73600b2250 0x7f73600b4a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:10.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:10.473+0000 7f7367200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7360166cb0 con 0x7f7360165d50 2024-03-04T09:27:10.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:10.473+0000 7f7367200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7360166410 con 0x7f7360162280 2024-03-04T09:27:10.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:10.473+0000 7f7367200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73600b4fd0 con 0x7f73600b2250 2024-03-04T09:27:10.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:10.473+0000 7f7365400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f7360165d50 0x7f73600b1b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:45412/0 (socket says 172.21.15.6:45412) 2024-03-04T09:27:10.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:10.473+0000 7f7365400640 1 -- 172.21.15.6:0/2590831708 learned_addr learned my addr 172.21.15.6:0/2590831708 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:27:10.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:10.474+0000 7f7366800640 1 --2- 172.21.15.6:0/2590831708 >> v2:172.21.15.6:3300/0 conn(0x7f73600b2250 0x7f73600b4a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:10.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:10.474+0000 7f7364a00640 1 -- 172.21.15.6:0/2590831708 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 385817294 0 0) 0x7f7360166cb0 con 0x7f7360165d50 2024-03-04T09:27:10.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:10.474+0000 7f7364a00640 1 -- 172.21.15.6:0/2590831708 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7348003520 con 0x7f7360165d50 2024-03-04T09:27:10.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:10.474+0000 7f7366800640 1 -- 172.21.15.6:0/2590831708 >> v1:172.21.15.146:6789/0 conn(0x7f7360162280 legacy=0x7f73600cc430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:10.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:10.474+0000 7f7366800640 1 -- 172.21.15.6:0/2590831708 >> v1:172.21.15.6:6789/0 conn(0x7f7360165d50 legacy=0x7f73600b1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:10.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:10.474+0000 7f7366800640 1 -- 172.21.15.6:0/2590831708 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73601666f0 con 0x7f73600b2250 2024-03-04T09:27:10.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:10.475+0000 7f7366800640 1 --2- 172.21.15.6:0/2590831708 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f73600b2250 0x7f73600b4a90 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f735c009d90 tx=0x7f735c02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68c4899f5b3dd8ab server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:10.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:10.476+0000 7f7364a00640 1 -- 172.21.15.6:0/2590831708 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f735c039070 con 0x7f73600b2250 2024-03-04T09:27:10.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:10.476+0000 7f7364a00640 1 -- 172.21.15.6:0/2590831708 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f735c02cc60 con 0x7f73600b2250 2024-03-04T09:27:10.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:10.476+0000 7f7364a00640 1 -- 172.21.15.6:0/2590831708 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f735c034660 con 0x7f73600b2250 2024-03-04T09:27:10.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:10.476+0000 7f7367200640 1 -- 172.21.15.6:0/2590831708 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f73600b2250 msgr2=0x7f73600b4a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:10.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:10.476+0000 7f7367200640 1 --2- 172.21.15.6:0/2590831708 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f73600b2250 0x7f73600b4a90 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f735c009d90 tx=0x7f735c02beb0 comp rx=0 tx=0).stop 2024-03-04T09:27:10.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:10.476+0000 7f7367200640 1 -- 172.21.15.6:0/2590831708 shutdown_connections 2024-03-04T09:27:10.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:10.476+0000 7f7367200640 1 --2- 172.21.15.6:0/2590831708 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f73600b2250 0x7f73600b4a90 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:10.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:10.476+0000 7f7367200640 1 -- 172.21.15.6:0/2590831708 >> 172.21.15.6:0/2590831708 conn(0x7f736006b3a0 msgr2=0x7f736006b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:10.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:10.477+0000 7f7367200640 1 -- 172.21.15.6:0/2590831708 shutdown_connections 2024-03-04T09:27:10.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:10.477+0000 7f7367200640 1 -- 172.21.15.6:0/2590831708 wait complete. 2024-03-04T09:27:10.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:10.478+0000 7f7367200640 1 Processor -- start 2024-03-04T09:27:10.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:10.478+0000 7f7367200640 1 -- start start 2024-03-04T09:27:10.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:10.479+0000 7f7367200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f73600b2250 0x7f7360149a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:10.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:10.479+0000 7f7367200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7360162280 0x7f7360154600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:10.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:10.479+0000 7f7367200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7360165d50 0x7f7360158e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:10.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:10.479+0000 7f7367200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7360071560 con 0x7f73600b2250 2024-03-04T09:27:10.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:10.479+0000 7f7367200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73601661b0 con 0x7f7360165d50 2024-03-04T09:27:10.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:10.479+0000 7f7367200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73600716e0 con 0x7f7360162280 2024-03-04T09:27:10.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:10.480+0000 7f7365e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f73600b2250 0x7f7360149a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:10.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:10.480+0000 7f7365e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f73600b2250 0x7f7360149a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38002/0 (socket says 172.21.15.6:38002) 2024-03-04T09:27:10.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:10.480+0000 7f7365e00640 1 -- 172.21.15.6:0/4220315618 learned_addr learned my addr 172.21.15.6:0/4220315618 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:27:10.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:10.480+0000 7f7365400640 1 --2- 172.21.15.6:0/4220315618 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7360162280 0x7f7360154600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:10.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:10.480+0000 7f7366800640 1 --2- 172.21.15.6:0/4220315618 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7360165d50 0x7f7360158e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:10.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:10.480+0000 7f7365e00640 1 -- 172.21.15.6:0/4220315618 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7360162280 msgr2=0x7f7360154600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:10.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:10.480+0000 7f7365e00640 1 --2- 172.21.15.6:0/4220315618 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7360162280 0x7f7360154600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:10.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:10.480+0000 7f7365e00640 1 -- 172.21.15.6:0/4220315618 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7360165d50 msgr2=0x7f7360158e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:10.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:10.480+0000 7f7365e00640 1 --2- 172.21.15.6:0/4220315618 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7360165d50 0x7f7360158e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:10.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:10.480+0000 7f7365e00640 1 -- 172.21.15.6:0/4220315618 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7360155060 con 0x7f73600b2250 2024-03-04T09:27:10.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:10.480+0000 7f7365400640 1 --2- 172.21.15.6:0/4220315618 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7360162280 0x7f7360154600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:27:10.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:10.480+0000 7f7365e00640 1 --2- 172.21.15.6:0/4220315618 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f73600b2250 0x7f7360149a90 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f735400c400 tx=0x7f735400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:10.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:10.481+0000 7f735a000640 1 -- 172.21.15.6:0/4220315618 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73540052d0 con 0x7f73600b2250 2024-03-04T09:27:10.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:10.481+0000 7f735a000640 1 -- 172.21.15.6:0/4220315618 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7354005490 con 0x7f73600b2250 2024-03-04T09:27:10.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:10.481+0000 7f7367200640 1 -- 172.21.15.6:0/4220315618 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7360155340 con 0x7f73600b2250 2024-03-04T09:27:10.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:10.481+0000 7f735a000640 1 -- 172.21.15.6:0/4220315618 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7354007260 con 0x7f73600b2250 2024-03-04T09:27:10.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:10.481+0000 7f7367200640 1 -- 172.21.15.6:0/4220315618 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7360165060 con 0x7f73600b2250 2024-03-04T09:27:10.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:10.482+0000 7f7367200640 1 -- 172.21.15.6:0/4220315618 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f736004f330 con 0x7f73600b2250 2024-03-04T09:27:10.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:10.483+0000 7f735a000640 1 -- 172.21.15.6:0/4220315618 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7354007400 con 0x7f73600b2250 2024-03-04T09:27:10.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:10.484+0000 7f735a000640 1 --2- 172.21.15.6:0/4220315618 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f734406b4d0 0x7f734406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:10.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:10.484+0000 7f735a000640 1 -- 172.21.15.6:0/4220315618 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f7354089460 con 0x7f73600b2250 2024-03-04T09:27:10.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:10.484+0000 7f7365400640 1 --2- 172.21.15.6:0/4220315618 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f734406b4d0 0x7f734406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:10.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:10.485+0000 7f7365400640 1 --2- 172.21.15.6:0/4220315618 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f734406b4d0 0x7f734406d990 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f7360159bc0 tx=0x7f7350009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:10.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:10.489+0000 7f735a000640 1 -- 172.21.15.6:0/4220315618 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f73540585d0 con 0x7f73600b2250 2024-03-04T09:27:10.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:10.690+0000 7f7367200640 1 -- 172.21.15.6:0/4220315618 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f736006e490 con 0x7f73600b2250 2024-03-04T09:27:10.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:10.695+0000 7f735a000640 1 -- 172.21.15.6:0/4220315618 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v12)=0 v12) v1 ==== 122+0+0 (secure 0 0 0) 0x7f7354057ed0 con 0x7f73600b2250 2024-03-04T09:27:10.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:10.697+0000 7f7367200640 1 -- 172.21.15.6:0/4220315618 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f734406b4d0 msgr2=0x7f734406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:10.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:10.698+0000 7f7367200640 1 --2- 172.21.15.6:0/4220315618 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f734406b4d0 0x7f734406d990 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f7360159bc0 tx=0x7f7350009040 comp rx=0 tx=0).stop 2024-03-04T09:27:10.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:10.698+0000 7f7367200640 1 -- 172.21.15.6:0/4220315618 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f73600b2250 msgr2=0x7f7360149a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:10.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:10.698+0000 7f7367200640 1 --2- 172.21.15.6:0/4220315618 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f73600b2250 0x7f7360149a90 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f735400c400 tx=0x7f735400c8c0 comp rx=0 tx=0).stop 2024-03-04T09:27:10.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:10.698+0000 7f7367200640 1 -- 172.21.15.6:0/4220315618 shutdown_connections 2024-03-04T09:27:10.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:10.698+0000 7f7367200640 1 --2- 172.21.15.6:0/4220315618 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7360165d50 0x7f7360158e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:10.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:10.698+0000 7f7367200640 1 --2- 172.21.15.6:0/4220315618 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f734406b4d0 0x7f734406d990 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:10.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:10.698+0000 7f7367200640 1 --2- 172.21.15.6:0/4220315618 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7360162280 0x7f7360154600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:10.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:10.698+0000 7f7367200640 1 --2- 172.21.15.6:0/4220315618 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f73600b2250 0x7f7360149a90 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:10.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:10.698+0000 7f7367200640 1 -- 172.21.15.6:0/4220315618 >> 172.21.15.6:0/4220315618 conn(0x7f736006b3a0 msgr2=0x7f7360162660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:10.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:10.698+0000 7f7367200640 1 -- 172.21.15.6:0/4220315618 shutdown_connections 2024-03-04T09:27:10.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:10.698+0000 7f7367200640 1 -- 172.21.15.6:0/4220315618 wait complete. 2024-03-04T09:27:10.712 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-04T09:27:10.713 DEBUG:teuthology.orchestra.run.smithi006:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-04T09:27:10.830 DEBUG:teuthology.orchestra.run.smithi146:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-04T09:27:10.951 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-04T09:27:10.952 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-04T09:27:10.952 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T09:27:11.013 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054202,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249800k,nr_inodes=812450,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-04T09:27:11.013 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-04T09:27:11.013 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T09:27:11.069 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053429,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249180k,nr_inodes=812295,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-04T09:27:11.069 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-04T09:27:11.069 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T09:27:11.129 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054202,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249800k,nr_inodes=812450,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-04T09:27:11.129 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-04T09:27:11.130 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T09:27:11.185 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053429,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249180k,nr_inodes=812295,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-04T09:27:11.185 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-04T09:27:11.186 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:27:11.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.416+0000 7f87fba00640 1 Processor -- start 2024-03-04T09:27:11.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.417+0000 7f87fba00640 1 -- start start 2024-03-04T09:27:11.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.418+0000 7f87fba00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f87f40af580 0x7f87f415c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:11.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.418+0000 7f87fba00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f87f40cdc20 0x7f87f40b2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:11.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.418+0000 7f87fba00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87f415cc40 con 0x7f87f40af1a0 2024-03-04T09:27:11.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.418+0000 7f87fba00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87f415cf20 con 0x7f87f40cdc20 2024-03-04T09:27:11.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.418+0000 7f87fba00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87f415d200 con 0x7f87f40af580 2024-03-04T09:27:11.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.418+0000 7f87f9000640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f87f40af1a0 0x7f87f40b4c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:45424/0 (socket says 172.21.15.6:45424) 2024-03-04T09:27:11.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.419+0000 7f87f9000640 1 -- 172.21.15.6:0/762027350 learned_addr learned my addr 172.21.15.6:0/762027350 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:27:11.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.419+0000 7f87f9a00640 1 --2- 172.21.15.6:0/762027350 >> v2:172.21.15.6:3300/0 conn(0x7f87f40cdc20 0x7f87f40b2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:11.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.419+0000 7f87f3e00640 1 --2- 172.21.15.6:0/762027350 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f87f40af580 0x7f87f415c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:11.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.419+0000 7f87f3400640 1 -- 172.21.15.6:0/762027350 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 270763541 0 0) 0x7f87f415cc40 con 0x7f87f40af1a0 2024-03-04T09:27:11.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.419+0000 7f87f3400640 1 -- 172.21.15.6:0/762027350 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87d8003520 con 0x7f87f40af1a0 2024-03-04T09:27:11.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.419+0000 7f87f9a00640 1 -- 172.21.15.6:0/762027350 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f87f40af580 msgr2=0x7f87f415c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:27:11.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.419+0000 7f87f9a00640 1 --2- 172.21.15.6:0/762027350 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f87f40af580 0x7f87f415c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:11.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.419+0000 7f87f9a00640 1 -- 172.21.15.6:0/762027350 >> v1:172.21.15.6:6789/0 conn(0x7f87f40af1a0 legacy=0x7f87f40b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:11.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.419+0000 7f87f9a00640 1 -- 172.21.15.6:0/762027350 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87f415d4e0 con 0x7f87f40cdc20 2024-03-04T09:27:11.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.421+0000 7f87f9a00640 1 --2- 172.21.15.6:0/762027350 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f87f40cdc20 0x7f87f40b2dd0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f87ec009d90 tx=0x7f87ec02be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d10464cb2adc6a91 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:11.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.421+0000 7f87f3400640 1 -- 172.21.15.6:0/762027350 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87ec039070 con 0x7f87f40cdc20 2024-03-04T09:27:11.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.421+0000 7f87f3400640 1 -- 172.21.15.6:0/762027350 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87ec02cbc0 con 0x7f87f40cdc20 2024-03-04T09:27:11.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.421+0000 7f87f3400640 1 -- 172.21.15.6:0/762027350 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87ec034440 con 0x7f87f40cdc20 2024-03-04T09:27:11.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.422+0000 7f87fba00640 1 -- 172.21.15.6:0/762027350 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f87f40cdc20 msgr2=0x7f87f40b2dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:11.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.422+0000 7f87fba00640 1 --2- 172.21.15.6:0/762027350 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f87f40cdc20 0x7f87f40b2dd0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f87ec009d90 tx=0x7f87ec02be10 comp rx=0 tx=0).stop 2024-03-04T09:27:11.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.422+0000 7f87fba00640 1 -- 172.21.15.6:0/762027350 shutdown_connections 2024-03-04T09:27:11.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.422+0000 7f87fba00640 1 --2- 172.21.15.6:0/762027350 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f87f40cdc20 0x7f87f40b2dd0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:11.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.422+0000 7f87fba00640 1 --2- 172.21.15.6:0/762027350 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f87f40af580 0x7f87f415c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:11.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.422+0000 7f87fba00640 1 -- 172.21.15.6:0/762027350 >> 172.21.15.6:0/762027350 conn(0x7f87f406afb0 msgr2=0x7f87f406b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:11.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.422+0000 7f87fba00640 1 -- 172.21.15.6:0/762027350 shutdown_connections 2024-03-04T09:27:11.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.422+0000 7f87fba00640 1 -- 172.21.15.6:0/762027350 wait complete. 2024-03-04T09:27:11.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.424+0000 7f87fba00640 1 Processor -- start 2024-03-04T09:27:11.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.424+0000 7f87fba00640 1 -- start start 2024-03-04T09:27:11.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.425+0000 7f87fba00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f87f40af1a0 0x7f87f415e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:11.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.425+0000 7f87fba00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f87f40af580 0x7f87f4150690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:11.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.425+0000 7f87fba00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f87f40cdc20 0x7f87f41f4e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:11.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.425+0000 7f87fba00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87f40b36a0 con 0x7f87f40cdc20 2024-03-04T09:27:11.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.425+0000 7f87fba00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87f40b3520 con 0x7f87f40af580 2024-03-04T09:27:11.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.425+0000 7f87fba00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87f40b3820 con 0x7f87f40af1a0 2024-03-04T09:27:11.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.426+0000 7f87f9a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f87f40cdc20 0x7f87f41f4e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:11.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.426+0000 7f87f9a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f87f40cdc20 0x7f87f41f4e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38018/0 (socket says 172.21.15.6:38018) 2024-03-04T09:27:11.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.426+0000 7f87f9a00640 1 -- 172.21.15.6:0/2204352658 learned_addr learned my addr 172.21.15.6:0/2204352658 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:27:11.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.426+0000 7f87f9000640 1 --2- 172.21.15.6:0/2204352658 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f87f40af1a0 0x7f87f415e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:11.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.426+0000 7f87f3e00640 1 --2- 172.21.15.6:0/2204352658 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f87f40af580 0x7f87f4150690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:11.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.426+0000 7f87f9a00640 1 -- 172.21.15.6:0/2204352658 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f87f40af1a0 msgr2=0x7f87f415e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:11.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.426+0000 7f87f9a00640 1 --2- 172.21.15.6:0/2204352658 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f87f40af1a0 0x7f87f415e8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:11.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.426+0000 7f87f9a00640 1 -- 172.21.15.6:0/2204352658 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f87f40af580 msgr2=0x7f87f4150690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:11.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.426+0000 7f87f9a00640 1 --2- 172.21.15.6:0/2204352658 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f87f40af580 0x7f87f4150690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:11.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.426+0000 7f87f9a00640 1 -- 172.21.15.6:0/2204352658 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87f415ede0 con 0x7f87f40cdc20 2024-03-04T09:27:11.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.427+0000 7f87f9a00640 1 --2- 172.21.15.6:0/2204352658 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f87f40cdc20 0x7f87f41f4e00 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f87ec009eb0 tx=0x7f87ec004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:11.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.427+0000 7f87f0c00640 1 -- 172.21.15.6:0/2204352658 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87ec039050 con 0x7f87f40cdc20 2024-03-04T09:27:11.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.427+0000 7f87f0c00640 1 -- 172.21.15.6:0/2204352658 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87ec045070 con 0x7f87f40cdc20 2024-03-04T09:27:11.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.427+0000 7f87fba00640 1 -- 172.21.15.6:0/2204352658 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87f40c9150 con 0x7f87f40cdc20 2024-03-04T09:27:11.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.427+0000 7f87f0c00640 1 -- 172.21.15.6:0/2204352658 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87ec009450 con 0x7f87f40cdc20 2024-03-04T09:27:11.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.427+0000 7f87fba00640 1 -- 172.21.15.6:0/2204352658 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87f40c9710 con 0x7f87f40cdc20 2024-03-04T09:27:11.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.428+0000 7f87fba00640 1 -- 172.21.15.6:0/2204352658 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87f404f3a0 con 0x7f87f40cdc20 2024-03-04T09:27:11.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.429+0000 7f87f0c00640 1 -- 172.21.15.6:0/2204352658 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f87ec03e830 con 0x7f87f40cdc20 2024-03-04T09:27:11.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.431+0000 7f87f0c00640 1 --2- 172.21.15.6:0/2204352658 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f87d006b500 0x7f87d006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:11.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.431+0000 7f87f0c00640 1 -- 172.21.15.6:0/2204352658 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f87ec0acb60 con 0x7f87f40cdc20 2024-03-04T09:27:11.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.431+0000 7f87f9000640 1 --2- 172.21.15.6:0/2204352658 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f87d006b500 0x7f87d006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:11.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.432+0000 7f87f9000640 1 --2- 172.21.15.6:0/2204352658 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f87d006b500 0x7f87d006d9c0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f87e40011e0 tx=0x7f87e4009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:11.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.435+0000 7f87f0c00640 1 -- 172.21.15.6:0/2204352658 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f87ec032e20 con 0x7f87f40cdc20 2024-03-04T09:27:11.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.671+0000 7f87fba00640 1 -- 172.21.15.6:0/2204352658 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f87f40cede0 con 0x7f87f40cdc20 2024-03-04T09:27:11.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.672+0000 7f87f0c00640 1 -- 172.21.15.6:0/2204352658 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) v1 ==== 94+0+3699 (secure 0 0 0) 0x7f87ec07be20 con 0x7f87f40cdc20 2024-03-04T09:27:11.669 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:27:11.669 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":18,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4685,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/1703428647","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":1703428647},{"type":"v1","addr":"172.21.15.146:6837","nonce":1703428647}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4837,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/2617429530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2617429530},{"type":"v1","addr":"172.21.15.6:6833","nonce":2617429530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":5043,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/2744913857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":2744913857},{"type":"v1","addr":"172.21.15.6:6835","nonce":2744913857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":5047,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6836/1858873699","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":1858873699},{"type":"v1","addr":"172.21.15.146:6836","nonce":1858873699}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":18}],"filesystems":[]} 2024-03-04T09:27:11.670 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 18 2024-03-04T09:27:11.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.675+0000 7f87fba00640 1 -- 172.21.15.6:0/2204352658 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f87d006b500 msgr2=0x7f87d006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:11.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.675+0000 7f87fba00640 1 --2- 172.21.15.6:0/2204352658 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f87d006b500 0x7f87d006d9c0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f87e40011e0 tx=0x7f87e4009040 comp rx=0 tx=0).stop 2024-03-04T09:27:11.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.675+0000 7f87fba00640 1 -- 172.21.15.6:0/2204352658 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f87f40cdc20 msgr2=0x7f87f41f4e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:11.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.675+0000 7f87fba00640 1 --2- 172.21.15.6:0/2204352658 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f87f40cdc20 0x7f87f41f4e00 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f87ec009eb0 tx=0x7f87ec004a40 comp rx=0 tx=0).stop 2024-03-04T09:27:11.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.675+0000 7f87fba00640 1 -- 172.21.15.6:0/2204352658 shutdown_connections 2024-03-04T09:27:11.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.675+0000 7f87fba00640 1 --2- 172.21.15.6:0/2204352658 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f87f40cdc20 0x7f87f41f4e00 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:11.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.675+0000 7f87fba00640 1 --2- 172.21.15.6:0/2204352658 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f87f40af580 0x7f87f4150690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:11.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.675+0000 7f87fba00640 1 --2- 172.21.15.6:0/2204352658 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f87d006b500 0x7f87d006d9c0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:11.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.675+0000 7f87fba00640 1 --2- 172.21.15.6:0/2204352658 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f87f40af1a0 0x7f87f415e8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:11.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.675+0000 7f87fba00640 1 -- 172.21.15.6:0/2204352658 >> 172.21.15.6:0/2204352658 conn(0x7f87f406afb0 msgr2=0x7f87f40cd1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:11.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.675+0000 7f87fba00640 1 -- 172.21.15.6:0/2204352658 shutdown_connections 2024-03-04T09:27:11.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.675+0000 7f87fba00640 1 -- 172.21.15.6:0/2204352658 wait complete. 2024-03-04T09:27:11.692 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-04T09:27:11.692 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-04T09:27:11.692 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-04T09:27:11.692 INFO:teuthology.orchestra.run:waiting for 300 2024-03-04T09:27:11.692 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-04T09:27:11.692 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-04T09:27:11.693 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-04T09:27:11.693 INFO:teuthology.orchestra.run:waiting for 300 2024-03-04T09:27:11.693 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-04T09:27:11.693 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-04T09:27:11.693 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-04T09:27:11.693 INFO:teuthology.orchestra.run:waiting for 300 2024-03-04T09:27:11.693 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-04T09:27:11.694 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-04T09:27:11.694 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-04T09:27:11.694 INFO:teuthology.orchestra.run:waiting for 300 2024-03-04T09:27:11.709 INFO:tasks.ceph.mds.c:Stopped 2024-03-04T09:27:11.709 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-04T09:27:11.711 INFO:tasks.ceph.mds.a:Stopped 2024-03-04T09:27:11.711 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-04T09:27:11.716 INFO:tasks.ceph.mds.d:Stopped 2024-03-04T09:27:11.716 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-04T09:27:11.721 INFO:tasks.ceph.mds.b:Stopped 2024-03-04T09:27:11.721 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-04T09:27:11.757 INFO:tasks.ceph.mds.c:Started 2024-03-04T09:27:11.757 DEBUG:teuthology.parallel:result is None 2024-03-04T09:27:11.763 INFO:tasks.ceph.mds.a:Started 2024-03-04T09:27:11.764 DEBUG:teuthology.parallel:result is None 2024-03-04T09:27:11.767 INFO:tasks.ceph.mds.d:Started 2024-03-04T09:27:11.767 DEBUG:teuthology.parallel:result is None 2024-03-04T09:27:11.773 INFO:tasks.ceph.mds.b:Started 2024-03-04T09:27:11.773 DEBUG:teuthology.parallel:result is None 2024-03-04T09:27:11.774 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-04T09:27:11.914 INFO:tasks.ceph.mds.c.smithi006.stdout:starting mds.c at 2024-03-04T09:27:11.919 INFO:tasks.ceph.mds.a.smithi006.stdout:starting mds.a at 2024-03-04T09:27:11.929 INFO:tasks.ceph.mds.d.smithi146.stdout:starting mds.d at 2024-03-04T09:27:11.934 INFO:tasks.ceph.mds.b.smithi146.stdout:starting mds.b at 2024-03-04T09:27:11.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.966+0000 7f0d92c00640 1 Processor -- start 2024-03-04T09:27:11.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.966+0000 7f0d92c00640 1 -- start start 2024-03-04T09:27:11.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.968+0000 7f0d92c00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f0d8c15e0c0 0x7f0d8c0d1cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:11.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.968+0000 7f0d92c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f0d8c0d2bc0 0x7f0d8c0c8220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:11.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.968+0000 7f0d92c00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d8c0d1160 con 0x7f0d8c15e4a0 2024-03-04T09:27:11.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.968+0000 7f0d92c00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d8c0d0f10 con 0x7f0d8c0d2bc0 2024-03-04T09:27:11.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.968+0000 7f0d92c00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d8c0d21f0 con 0x7f0d8c15e0c0 2024-03-04T09:27:11.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.968+0000 7f0d90c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f0d8c0d2bc0 0x7f0d8c0c8220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:11.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.968+0000 7f0d90c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f0d8c0d2bc0 0x7f0d8c0c8220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38040/0 (socket says 172.21.15.6:38040) 2024-03-04T09:27:11.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.968+0000 7f0d90c00640 1 -- 172.21.15.6:0/3762180538 learned_addr learned my addr 172.21.15.6:0/3762180538 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:27:11.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.968+0000 7f0d8be00640 1 --2- 172.21.15.6:0/3762180538 >> v2:172.21.15.146:3300/0 conn(0x7f0d8c15e0c0 0x7f0d8c0d1cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:11.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.968+0000 7f0d90c00640 1 -- 172.21.15.6:0/3762180538 >> v2:172.21.15.146:3300/0 conn(0x7f0d8c15e0c0 msgr2=0x7f0d8c0d1cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:27:11.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.968+0000 7f0d90c00640 1 --2- 172.21.15.6:0/3762180538 >> v2:172.21.15.146:3300/0 conn(0x7f0d8c15e0c0 0x7f0d8c0d1cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:11.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.968+0000 7f0d90c00640 1 -- 172.21.15.6:0/3762180538 >> v1:172.21.15.146:6789/0 conn(0x7f0d8c15e4a0 legacy=0x7f0d8c0d24b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:11.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.968+0000 7f0d90c00640 1 -- 172.21.15.6:0/3762180538 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d8c0d1440 con 0x7f0d8c0d2bc0 2024-03-04T09:27:11.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.970+0000 7f0d90c00640 1 --2- 172.21.15.6:0/3762180538 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0d8c0d2bc0 0x7f0d8c0c8220 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f0d8c08c940 tx=0x7f0d8402ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cb10efae5db31d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:11.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.970+0000 7f0d8b400640 1 -- 172.21.15.6:0/3762180538 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d8402f850 con 0x7f0d8c0d2bc0 2024-03-04T09:27:11.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.970+0000 7f0d8b400640 1 -- 172.21.15.6:0/3762180538 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d8402f9f0 con 0x7f0d8c0d2bc0 2024-03-04T09:27:11.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.970+0000 7f0d8b400640 1 -- 172.21.15.6:0/3762180538 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d8402fdc0 con 0x7f0d8c0d2bc0 2024-03-04T09:27:11.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.970+0000 7f0d92c00640 1 -- 172.21.15.6:0/3762180538 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0d8c0d2bc0 msgr2=0x7f0d8c0c8220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:11.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.970+0000 7f0d92c00640 1 --2- 172.21.15.6:0/3762180538 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0d8c0d2bc0 0x7f0d8c0c8220 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f0d8c08c940 tx=0x7f0d8402ba30 comp rx=0 tx=0).stop 2024-03-04T09:27:11.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.970+0000 7f0d92c00640 1 -- 172.21.15.6:0/3762180538 shutdown_connections 2024-03-04T09:27:11.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.970+0000 7f0d92c00640 1 --2- 172.21.15.6:0/3762180538 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0d8c0d2bc0 0x7f0d8c0c8220 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:11.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.970+0000 7f0d92c00640 1 --2- 172.21.15.6:0/3762180538 >> v2:172.21.15.146:3300/0 conn(0x7f0d8c15e0c0 0x7f0d8c0d1cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:11.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.970+0000 7f0d92c00640 1 -- 172.21.15.6:0/3762180538 >> 172.21.15.6:0/3762180538 conn(0x7f0d8c0d3ed0 msgr2=0x7f0d8c0d42e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:11.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.971+0000 7f0d92c00640 1 -- 172.21.15.6:0/3762180538 shutdown_connections 2024-03-04T09:27:11.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.971+0000 7f0d92c00640 1 -- 172.21.15.6:0/3762180538 wait complete. 2024-03-04T09:27:11.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.972+0000 7f0d92c00640 1 Processor -- start 2024-03-04T09:27:11.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.972+0000 7f0d92c00640 1 -- start start 2024-03-04T09:27:11.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.973+0000 7f0d92c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0d8c0d2bc0 0x7f0d8c0cca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:11.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.973+0000 7f0d92c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0d8c15e0c0 0x7f0d8c160430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:11.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.973+0000 7f0d92c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0d8c15e4a0 0x7f0d8c0cafa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:11.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.973+0000 7f0d92c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d8c0c7d00 con 0x7f0d8c0d2bc0 2024-03-04T09:27:11.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.973+0000 7f0d92c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d8c0c7a20 con 0x7f0d8c15e0c0 2024-03-04T09:27:11.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.973+0000 7f0d92c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d8c0d4b40 con 0x7f0d8c15e4a0 2024-03-04T09:27:11.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.973+0000 7f0d8be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0d8c0d2bc0 0x7f0d8c0cca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:11.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.974+0000 7f0d8be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0d8c0d2bc0 0x7f0d8c0cca90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38050/0 (socket says 172.21.15.6:38050) 2024-03-04T09:27:11.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.974+0000 7f0d8be00640 1 -- 172.21.15.6:0/654032217 learned_addr learned my addr 172.21.15.6:0/654032217 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:27:11.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.974+0000 7f0d90c00640 1 --2- 172.21.15.6:0/654032217 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0d8c15e4a0 0x7f0d8c0cafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:11.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.974+0000 7f0d8be00640 1 -- 172.21.15.6:0/654032217 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0d8c15e4a0 msgr2=0x7f0d8c0cafa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:11.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.974+0000 7f0d8be00640 1 --2- 172.21.15.6:0/654032217 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0d8c15e4a0 0x7f0d8c0cafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:11.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.974+0000 7f0d8be00640 1 -- 172.21.15.6:0/654032217 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0d8c15e0c0 msgr2=0x7f0d8c160430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:11.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.974+0000 7f0d83400640 1 --2- 172.21.15.6:0/654032217 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0d8c15e0c0 0x7f0d8c160430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:11.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.974+0000 7f0d8be00640 1 --2- 172.21.15.6:0/654032217 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0d8c15e0c0 0x7f0d8c160430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:11.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.974+0000 7f0d8be00640 1 -- 172.21.15.6:0/654032217 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d8c0cb9f0 con 0x7f0d8c0d2bc0 2024-03-04T09:27:11.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.974+0000 7f0d83400640 1 --2- 172.21.15.6:0/654032217 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0d8c15e0c0 0x7f0d8c160430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:27:11.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.974+0000 7f0d8be00640 1 --2- 172.21.15.6:0/654032217 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0d8c0d2bc0 0x7f0d8c0cca90 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f0d7c009db0 tx=0x7f0d7c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:11.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.974+0000 7f0d88c00640 1 -- 172.21.15.6:0/654032217 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d7c018020 con 0x7f0d8c0d2bc0 2024-03-04T09:27:11.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.975+0000 7f0d88c00640 1 -- 172.21.15.6:0/654032217 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d7c014480 con 0x7f0d8c0d2bc0 2024-03-04T09:27:11.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.975+0000 7f0d88c00640 1 -- 172.21.15.6:0/654032217 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d7c014850 con 0x7f0d8c0d2bc0 2024-03-04T09:27:11.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.975+0000 7f0d92c00640 1 -- 172.21.15.6:0/654032217 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d8c0cbcd0 con 0x7f0d8c0d2bc0 2024-03-04T09:27:11.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.975+0000 7f0d92c00640 1 -- 172.21.15.6:0/654032217 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d8c0cc290 con 0x7f0d8c0d2bc0 2024-03-04T09:27:11.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.976+0000 7f0d92c00640 1 -- 172.21.15.6:0/654032217 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d8c04f3a0 con 0x7f0d8c0d2bc0 2024-03-04T09:27:11.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.977+0000 7f0d88c00640 1 -- 172.21.15.6:0/654032217 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0d7c01b020 con 0x7f0d8c0d2bc0 2024-03-04T09:27:11.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.978+0000 7f0d88c00640 1 --2- 172.21.15.6:0/654032217 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f0d6806b4d0 0x7f0d6806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:11.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.978+0000 7f0d88c00640 1 -- 172.21.15.6:0/654032217 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f0d7c08aae0 con 0x7f0d8c0d2bc0 2024-03-04T09:27:11.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.978+0000 7f0d83400640 1 --2- 172.21.15.6:0/654032217 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f0d6806b4d0 0x7f0d6806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:11.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.979+0000 7f0d83400640 1 --2- 172.21.15.6:0/654032217 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f0d6806b4d0 0x7f0d6806d990 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f0d8c161000 tx=0x7f0d74009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:11.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:11.982+0000 7f0d88c00640 1 -- 172.21.15.6:0/654032217 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0d7c022030 con 0x7f0d8c0d2bc0 2024-03-04T09:27:12.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:12.196+0000 7f0d92c00640 1 -- 172.21.15.6:0/654032217 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f0d8c08da10 con 0x7f0d8c0d2bc0 2024-03-04T09:27:12.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:12.244+0000 7f0d88c00640 1 -- 172.21.15.6:0/654032217 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v38) v1 ==== 97+0+0 (secure 0 0 0) 0x7f0d7c010040 con 0x7f0d8c0d2bc0 2024-03-04T09:27:12.241 INFO:teuthology.orchestra.run.smithi006.stderr: removed all blocklist entries 2024-03-04T09:27:12.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:12.247+0000 7f0d92c00640 1 -- 172.21.15.6:0/654032217 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f0d6806b4d0 msgr2=0x7f0d6806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:12.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:12.247+0000 7f0d92c00640 1 --2- 172.21.15.6:0/654032217 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f0d6806b4d0 0x7f0d6806d990 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f0d8c161000 tx=0x7f0d74009040 comp rx=0 tx=0).stop 2024-03-04T09:27:12.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:12.247+0000 7f0d92c00640 1 -- 172.21.15.6:0/654032217 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0d8c0d2bc0 msgr2=0x7f0d8c0cca90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:12.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:12.247+0000 7f0d92c00640 1 --2- 172.21.15.6:0/654032217 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0d8c0d2bc0 0x7f0d8c0cca90 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f0d7c009db0 tx=0x7f0d7c00c490 comp rx=0 tx=0).stop 2024-03-04T09:27:12.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:12.247+0000 7f0d92c00640 1 -- 172.21.15.6:0/654032217 shutdown_connections 2024-03-04T09:27:12.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:12.247+0000 7f0d92c00640 1 --2- 172.21.15.6:0/654032217 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f0d6806b4d0 0x7f0d6806d990 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:12.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:12.247+0000 7f0d92c00640 1 --2- 172.21.15.6:0/654032217 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0d8c15e4a0 0x7f0d8c0cafa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:12.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:12.247+0000 7f0d92c00640 1 --2- 172.21.15.6:0/654032217 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0d8c15e0c0 0x7f0d8c160430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:12.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:12.247+0000 7f0d92c00640 1 --2- 172.21.15.6:0/654032217 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0d8c0d2bc0 0x7f0d8c0cca90 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:12.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:12.247+0000 7f0d92c00640 1 -- 172.21.15.6:0/654032217 >> 172.21.15.6:0/654032217 conn(0x7f0d8c0d3ed0 msgr2=0x7f0d8c0c8840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:12.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:12.247+0000 7f0d92c00640 1 -- 172.21.15.6:0/654032217 shutdown_connections 2024-03-04T09:27:12.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:12.247+0000 7f0d92c00640 1 -- 172.21.15.6:0/654032217 wait complete. 2024-03-04T09:27:12.267 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-04T09:27:12.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:12.481+0000 7f38db200640 1 Processor -- start 2024-03-04T09:27:12.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:12.481+0000 7f38db200640 1 -- start start 2024-03-04T09:27:12.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:12.482+0000 7f38db200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f38dc0ce850 0x7f38dc0cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:12.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:12.482+0000 7f38db200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38dc0cdad0 con 0x7f38dc0cce90 2024-03-04T09:27:12.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:12.482+0000 7f38db200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38dc071500 con 0x7f38dc15c920 2024-03-04T09:27:12.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:12.482+0000 7f38db200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38dc0cdd70 con 0x7f38dc0ce850 2024-03-04T09:27:12.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:12.483+0000 7f38da800640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f38dc0cce90 0x7f38dc06cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:45448/0 (socket says 172.21.15.6:45448) 2024-03-04T09:27:12.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:12.483+0000 7f38da800640 1 -- 172.21.15.6:0/435188348 learned_addr learned my addr 172.21.15.6:0/435188348 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:27:12.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:12.483+0000 7f38d1400640 1 --2- 172.21.15.6:0/435188348 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f38dc0ce850 0x7f38dc0cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:12.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:12.483+0000 7f38d9400640 1 -- 172.21.15.6:0/435188348 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2560261593 0 0) 0x7f38dc0cdad0 con 0x7f38dc0cce90 2024-03-04T09:27:12.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:12.483+0000 7f38d9400640 1 -- 172.21.15.6:0/435188348 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38c0003520 con 0x7f38dc0cce90 2024-03-04T09:27:12.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:12.484+0000 7f38d9400640 1 -- 172.21.15.6:0/435188348 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1214189608 0 0) 0x7f38c0003520 con 0x7f38dc0cce90 2024-03-04T09:27:12.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:12.484+0000 7f38d9400640 1 -- 172.21.15.6:0/435188348 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f38dc0ce850 msgr2=0x7f38dc0cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:27:12.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:12.484+0000 7f38d9400640 1 --2- 172.21.15.6:0/435188348 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f38dc0ce850 0x7f38dc0cc950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:12.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:12.484+0000 7f38d9400640 1 -- 172.21.15.6:0/435188348 >> v1:172.21.15.146:6789/0 conn(0x7f38dc15c920 legacy=0x7f38dc070a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:12.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:12.484+0000 7f38d9400640 1 -- 172.21.15.6:0/435188348 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38dc0ce050 con 0x7f38dc0cce90 2024-03-04T09:27:12.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:12.484+0000 7f38d9400640 1 -- 172.21.15.6:0/435188348 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7f38d4001670 con 0x7f38dc0cce90 2024-03-04T09:27:12.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:12.484+0000 7f38d9400640 1 -- 172.21.15.6:0/435188348 >> v1:172.21.15.6:6789/0 conn(0x7f38dc0cce90 legacy=0x7f38dc06cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:12.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:12.484+0000 7f38d9400640 1 --2- 172.21.15.6:0/435188348 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f38dc0ce850 0x7f38c00039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:12.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:12.484+0000 7f38d9400640 1 --2- 172.21.15.6:0/435188348 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38c0003f60 0x7f38c0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:12.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:12.484+0000 7f38d9400640 1 --2- 172.21.15.6:0/435188348 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f38c0004d40 0x7f38c00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:12.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:12.484+0000 7f38d9400640 1 -- 172.21.15.6:0/435188348 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38dc0cdd70 con 0x7f38c0003f60 2024-03-04T09:27:12.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:12.484+0000 7f38d9400640 1 -- 172.21.15.6:0/435188348 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38c000baa0 con 0x7f38c0004d40 2024-03-04T09:27:12.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:12.484+0000 7f38d9400640 1 -- 172.21.15.6:0/435188348 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38c000bca0 con 0x7f38dc0ce850 2024-03-04T09:27:12.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:12.484+0000 7f38d1400640 1 --2- 172.21.15.6:0/435188348 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38c0003f60 0x7f38c0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:12.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:12.484+0000 7f38da800640 1 --2- 172.21.15.6:0/435188348 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f38c0004d40 0x7f38c00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:12.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:12.484+0000 7f38d9e00640 1 --2- 172.21.15.6:0/435188348 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f38dc0ce850 0x7f38c00039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:12.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:12.485+0000 7f38d1400640 1 -- 172.21.15.6:0/435188348 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f38dc0ce850 msgr2=0x7f38c00039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:27:12.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:12.485+0000 7f38d1400640 1 --2- 172.21.15.6:0/435188348 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f38dc0ce850 0x7f38c00039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:12.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:12.485+0000 7f38d1400640 1 -- 172.21.15.6:0/435188348 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f38c0004d40 msgr2=0x7f38c00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:27:12.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:12.485+0000 7f38d1400640 1 --2- 172.21.15.6:0/435188348 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f38c0004d40 0x7f38c00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:12.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:12.485+0000 7f38d1400640 1 -- 172.21.15.6:0/435188348 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38c0003520 con 0x7f38c0003f60 2024-03-04T09:27:12.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:12.486+0000 7f38d1400640 1 --2- 172.21.15.6:0/435188348 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38c0003f60 0x7f38c0004800 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f38c4009fc0 tx=0x7f38c0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d95cefe963a60a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:12.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:12.486+0000 7f38d9400640 1 -- 172.21.15.6:0/435188348 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38c4039070 con 0x7f38c0003f60 2024-03-04T09:27:12.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:12.486+0000 7f38d9400640 1 -- 172.21.15.6:0/435188348 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38c402cc20 con 0x7f38c0003f60 2024-03-04T09:27:12.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:12.486+0000 7f38d9400640 1 -- 172.21.15.6:0/435188348 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38c4009c10 con 0x7f38c0003f60 2024-03-04T09:27:12.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:12.487+0000 7f38db200640 1 -- 172.21.15.6:0/435188348 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38c0003f60 msgr2=0x7f38c0004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:12.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:12.487+0000 7f38db200640 1 --2- 172.21.15.6:0/435188348 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38c0003f60 0x7f38c0004800 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f38c4009fc0 tx=0x7f38c0005830 comp rx=0 tx=0).stop 2024-03-04T09:27:12.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:12.487+0000 7f38da800640 1 -- 172.21.15.6:0/435188348 reap_dead start 2024-03-04T09:27:12.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:12.487+0000 7f38db200640 1 -- 172.21.15.6:0/435188348 shutdown_connections 2024-03-04T09:27:12.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:12.487+0000 7f38db200640 1 -- 172.21.15.6:0/435188348 >> 172.21.15.6:0/435188348 conn(0x7f38dc06b3d0 msgr2=0x7f38dc06b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:12.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:12.487+0000 7f38db200640 1 -- 172.21.15.6:0/435188348 shutdown_connections 2024-03-04T09:27:12.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:12.487+0000 7f38db200640 1 -- 172.21.15.6:0/435188348 wait complete. 2024-03-04T09:27:12.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:12.489+0000 7f38db200640 1 Processor -- start 2024-03-04T09:27:12.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:12.489+0000 7f38db200640 1 -- start start 2024-03-04T09:27:12.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:12.490+0000 7f38db200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f38c0003f60 0x7f38dc150530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:12.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:12.490+0000 7f38db200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f38dc161640 0x7f38dc161ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:12.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:12.490+0000 7f38db200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38dc15d0b0 0x7f38dc15d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:12.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:12.490+0000 7f38db200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38dc071150 con 0x7f38dc15d0b0 2024-03-04T09:27:12.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:12.490+0000 7f38db200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38dc163790 con 0x7f38c0003f60 2024-03-04T09:27:12.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:12.490+0000 7f38db200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38dc15da70 con 0x7f38dc161640 2024-03-04T09:27:12.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:12.490+0000 7f38da800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38dc15d0b0 0x7f38dc15d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:12.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:12.490+0000 7f38da800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38dc15d0b0 0x7f38dc15d530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38066/0 (socket says 172.21.15.6:38066) 2024-03-04T09:27:12.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:12.490+0000 7f38da800640 1 -- 172.21.15.6:0/2974605200 learned_addr learned my addr 172.21.15.6:0/2974605200 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:27:12.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:12.491+0000 7f38d9e00640 1 --2- 172.21.15.6:0/2974605200 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f38c0003f60 0x7f38dc150530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:12.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:12.491+0000 7f38d1400640 1 --2- 172.21.15.6:0/2974605200 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f38dc161640 0x7f38dc161ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:12.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:12.491+0000 7f38da800640 1 -- 172.21.15.6:0/2974605200 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f38dc161640 msgr2=0x7f38dc161ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:12.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:12.491+0000 7f38da800640 1 --2- 172.21.15.6:0/2974605200 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f38dc161640 0x7f38dc161ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:12.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:12.491+0000 7f38da800640 1 -- 172.21.15.6:0/2974605200 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f38c0003f60 msgr2=0x7f38dc150530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:12.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:12.491+0000 7f38da800640 1 --2- 172.21.15.6:0/2974605200 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f38c0003f60 0x7f38dc150530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:12.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:12.491+0000 7f38da800640 1 -- 172.21.15.6:0/2974605200 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38dc15f3f0 con 0x7f38dc15d0b0 2024-03-04T09:27:12.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:12.491+0000 7f38d9e00640 1 --2- 172.21.15.6:0/2974605200 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f38c0003f60 0x7f38dc150530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:27:12.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:12.491+0000 7f38da800640 1 --2- 172.21.15.6:0/2974605200 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38dc15d0b0 0x7f38dc15d530 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f38dc166f30 tx=0x7f38d400e960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:12.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:12.491+0000 7f38d2a00640 1 -- 172.21.15.6:0/2974605200 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38d4019070 con 0x7f38dc15d0b0 2024-03-04T09:27:12.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:12.491+0000 7f38d2a00640 1 -- 172.21.15.6:0/2974605200 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38d40105d0 con 0x7f38dc15d0b0 2024-03-04T09:27:12.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:12.491+0000 7f38d2a00640 1 -- 172.21.15.6:0/2974605200 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38d40109a0 con 0x7f38dc15d0b0 2024-03-04T09:27:12.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:12.491+0000 7f38db200640 1 -- 172.21.15.6:0/2974605200 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38dc15f6d0 con 0x7f38dc15d0b0 2024-03-04T09:27:12.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:12.491+0000 7f38db200640 1 -- 172.21.15.6:0/2974605200 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38dc15fc90 con 0x7f38dc15d0b0 2024-03-04T09:27:12.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:12.493+0000 7f38db200640 1 -- 172.21.15.6:0/2974605200 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38dc15fe90 con 0x7f38dc15d0b0 2024-03-04T09:27:12.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:12.493+0000 7f38d2a00640 1 -- 172.21.15.6:0/2974605200 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f38d4010b40 con 0x7f38dc15d0b0 2024-03-04T09:27:12.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:12.494+0000 7f38d2a00640 1 --2- 172.21.15.6:0/2974605200 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f38b806b500 0x7f38b806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:12.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:12.495+0000 7f38d2a00640 1 -- 172.21.15.6:0/2974605200 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f38d4089310 con 0x7f38dc15d0b0 2024-03-04T09:27:12.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:12.495+0000 7f38d9e00640 1 --2- 172.21.15.6:0/2974605200 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f38b806b500 0x7f38b806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:12.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:12.496+0000 7f38d9e00640 1 --2- 172.21.15.6:0/2974605200 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f38b806b500 0x7f38b806d9c0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f38cc0016b0 tx=0x7f38cc009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:12.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:12.499+0000 7f38d2a00640 1 -- 172.21.15.6:0/2974605200 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f38dc15fe90 con 0x7f38dc15d0b0 2024-03-04T09:27:12.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:12.770+0000 7f38db200640 1 -- 172.21.15.6:0/2974605200 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f38dc162000 con 0x7f38dc15d0b0 2024-03-04T09:27:12.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:12.772+0000 7f38d2a00640 1 -- 172.21.15.6:0/2974605200 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v10) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f38d40584d0 con 0x7f38dc15d0b0 2024-03-04T09:27:12.769 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:27:12.769 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-04T09:27:12.769 INFO:teuthology.orchestra.run.smithi006.stdout: "auth_dump": [ 2024-03-04T09:27:12.769 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:27:12.769 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "mds.a", 2024-03-04T09:27:12.769 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQA8k+VlHCzFIBAAnKYlmCMDw45LqnSjxnZY5Q==", 2024-03-04T09:27:12.769 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:27:12.769 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-04T09:27:12.770 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow *", 2024-03-04T09:27:12.770 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow *", 2024-03-04T09:27:12.770 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-04T09:27:12.770 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:27:12.770 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:27:12.770 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:27:12.770 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "mds.b", 2024-03-04T09:27:12.770 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQA9k+VlNY7fCBAAX3z8X72/6LkQD6mwGYT69g==", 2024-03-04T09:27:12.770 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:27:12.770 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-04T09:27:12.770 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow *", 2024-03-04T09:27:12.770 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow *", 2024-03-04T09:27:12.770 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-04T09:27:12.770 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:27:12.770 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:27:12.771 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:27:12.771 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "mds.c", 2024-03-04T09:27:12.771 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQA8k+Vl0pjOMhAA7WcTw8awv4EAsrpP4yncNA==", 2024-03-04T09:27:12.771 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:27:12.771 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-04T09:27:12.771 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow *", 2024-03-04T09:27:12.771 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow *", 2024-03-04T09:27:12.771 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-04T09:27:12.771 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:27:12.771 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:27:12.771 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:27:12.771 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "mds.d", 2024-03-04T09:27:12.771 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQA9k+VlbladGRAABNgUcaqTVYuJfCqy5c/G0g==", 2024-03-04T09:27:12.771 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:27:12.772 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-04T09:27:12.772 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow *", 2024-03-04T09:27:12.772 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow *", 2024-03-04T09:27:12.772 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-04T09:27:12.772 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:27:12.772 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:27:12.772 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:27:12.772 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.0", 2024-03-04T09:27:12.772 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQBCk+VlyjMMExAAqb2FJ9n5PWehfx/6PB19ig==", 2024-03-04T09:27:12.772 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:27:12.772 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-04T09:27:12.772 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-04T09:27:12.772 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-04T09:27:12.773 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:27:12.773 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:27:12.773 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:27:12.773 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.1", 2024-03-04T09:27:12.773 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQBFk+VlXD3cLBAAGclSivcYWDslqN8MLlj7eQ==", 2024-03-04T09:27:12.773 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:27:12.773 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-04T09:27:12.773 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-04T09:27:12.773 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-04T09:27:12.773 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:27:12.773 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:27:12.773 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:27:12.773 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.2", 2024-03-04T09:27:12.773 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQBJk+Vl7d9nCxAAnb3W4EI3Yf3XY8ht3VnKEQ==", 2024-03-04T09:27:12.774 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:27:12.774 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-04T09:27:12.774 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-04T09:27:12.774 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-04T09:27:12.774 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:27:12.774 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:27:12.774 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:27:12.774 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.3", 2024-03-04T09:27:12.774 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQBMk+Vlv+Y1NBAAiRYSUqov2i2sj3iInfpHvA==", 2024-03-04T09:27:12.774 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:27:12.774 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-04T09:27:12.774 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-04T09:27:12.774 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-04T09:27:12.774 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:27:12.775 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:27:12.775 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:27:12.775 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.4", 2024-03-04T09:27:12.775 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQBUk+Vl1OlAARAAxMV4GUGS+RJ6XlrRhrVKcg==", 2024-03-04T09:27:12.775 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:27:12.775 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-04T09:27:12.775 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-04T09:27:12.775 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-04T09:27:12.775 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:27:12.775 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:27:12.775 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:27:12.775 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.5", 2024-03-04T09:27:12.775 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQBXk+Vl1lKLGhAAHlsiRr+1+OSRWhkB+weIJg==", 2024-03-04T09:27:12.776 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:27:12.776 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-04T09:27:12.776 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-04T09:27:12.776 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-04T09:27:12.776 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:27:12.776 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:27:12.776 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:27:12.776 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.6", 2024-03-04T09:27:12.776 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQBak+VlwB+GJBAA3f9KUz+zZlQuUR/ywQfkDA==", 2024-03-04T09:27:12.776 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:27:12.776 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-04T09:27:12.776 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-04T09:27:12.776 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-04T09:27:12.777 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:27:12.777 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:27:12.777 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:27:12.777 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.7", 2024-03-04T09:27:12.777 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQBdk+Vle4o8LhAAko+iH5VqqCsfWkNfKZjGVA==", 2024-03-04T09:27:12.777 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:27:12.777 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-04T09:27:12.777 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-04T09:27:12.777 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-04T09:27:12.777 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:27:12.777 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:27:12.777 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:27:12.777 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "client.0", 2024-03-04T09:27:12.777 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQA9k+Vl4L+rJxAACaDbQ4/Bybp5hAZAnJzlVw==", 2024-03-04T09:27:12.778 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:27:12.778 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-04T09:27:12.778 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow r", 2024-03-04T09:27:12.778 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-04T09:27:12.778 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:27:12.778 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:27:12.778 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:27:12.778 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "client.1", 2024-03-04T09:27:12.778 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQA+k+VlGjkOAhAAJNzB8+tQl3mqCnTgMn9EVw==", 2024-03-04T09:27:12.778 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:27:12.778 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-04T09:27:12.778 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow r", 2024-03-04T09:27:12.778 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow rw", 2024-03-04T09:27:12.778 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow rwx" 2024-03-04T09:27:12.779 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:27:12.779 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:27:12.779 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:27:12.779 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "client.2", 2024-03-04T09:27:12.779 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQA9k+VlUc8DMxAA8pGUtIh+BO6UKrrYnCqVgQ==", 2024-03-04T09:27:12.779 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:27:12.779 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-04T09:27:12.779 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow r", 2024-03-04T09:27:12.779 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow rw", 2024-03-04T09:27:12.779 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow rwx" 2024-03-04T09:27:12.779 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:27:12.779 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:27:12.779 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:27:12.779 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "client.3", 2024-03-04T09:27:12.779 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQA+k+VlzMFKDBAAGr713I2hcmGMuHMIjNknYA==", 2024-03-04T09:27:12.780 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:27:12.780 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-04T09:27:12.780 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow r", 2024-03-04T09:27:12.780 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow rw", 2024-03-04T09:27:12.780 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow rwx" 2024-03-04T09:27:12.780 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:27:12.780 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:27:12.780 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:27:12.780 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "client.admin", 2024-03-04T09:27:12.780 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQA7k+Vllb7NFhAAmnFaNFKBcEri+iJAPnG5Gg==", 2024-03-04T09:27:12.780 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:27:12.780 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow *", 2024-03-04T09:27:12.780 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow *", 2024-03-04T09:27:12.781 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow *", 2024-03-04T09:27:12.781 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-04T09:27:12.781 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:27:12.781 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:27:12.781 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:27:12.781 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "mgr.x", 2024-03-04T09:27:12.781 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQA8k+VlPFatFBAA/o3Ps/uJrcg1738HOGy/FA==", 2024-03-04T09:27:12.781 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:27:12.781 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow *", 2024-03-04T09:27:12.781 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile mgr", 2024-03-04T09:27:12.781 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-04T09:27:12.781 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:27:12.781 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:27:12.782 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:27:12.782 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "mgr.y", 2024-03-04T09:27:12.782 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQA8k+VlEA2ACBAAXofDj3gmqItVTJ1pYLqZyQ==", 2024-03-04T09:27:12.782 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:27:12.782 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow *", 2024-03-04T09:27:12.782 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile mgr", 2024-03-04T09:27:12.782 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-04T09:27:12.782 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:27:12.782 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:27:12.782 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-04T09:27:12.782 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-04T09:27:12.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:12.775+0000 7f38db200640 1 -- 172.21.15.6:0/2974605200 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f38b806b500 msgr2=0x7f38b806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:12.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:12.775+0000 7f38db200640 1 --2- 172.21.15.6:0/2974605200 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f38b806b500 0x7f38b806d9c0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f38cc0016b0 tx=0x7f38cc009040 comp rx=0 tx=0).stop 2024-03-04T09:27:12.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:12.775+0000 7f38db200640 1 -- 172.21.15.6:0/2974605200 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38dc15d0b0 msgr2=0x7f38dc15d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:12.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:12.775+0000 7f38db200640 1 --2- 172.21.15.6:0/2974605200 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38dc15d0b0 0x7f38dc15d530 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f38dc166f30 tx=0x7f38d400e960 comp rx=0 tx=0).stop 2024-03-04T09:27:12.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:12.775+0000 7f38db200640 1 -- 172.21.15.6:0/2974605200 shutdown_connections 2024-03-04T09:27:12.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:12.775+0000 7f38db200640 1 --2- 172.21.15.6:0/2974605200 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38dc15d0b0 0x7f38dc15d530 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:12.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:12.775+0000 7f38db200640 1 --2- 172.21.15.6:0/2974605200 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f38b806b500 0x7f38b806d9c0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:12.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:12.775+0000 7f38db200640 1 --2- 172.21.15.6:0/2974605200 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f38dc161640 0x7f38dc161ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:12.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:12.775+0000 7f38db200640 1 --2- 172.21.15.6:0/2974605200 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f38c0003f60 0x7f38dc150530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:12.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:12.775+0000 7f38db200640 1 -- 172.21.15.6:0/2974605200 >> 172.21.15.6:0/2974605200 conn(0x7f38dc06b3d0 msgr2=0x7f38dc163980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:12.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:12.775+0000 7f38db200640 1 -- 172.21.15.6:0/2974605200 shutdown_connections 2024-03-04T09:27:12.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:12.775+0000 7f38db200640 1 -- 172.21.15.6:0/2974605200 wait complete. 2024-03-04T09:27:12.790 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T09:27:13.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.010+0000 7f2cfb200640 1 Processor -- start 2024-03-04T09:27:13.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.010+0000 7f2cfb200640 1 -- start start 2024-03-04T09:27:13.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.011+0000 7f2cfb200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f2cfc0cd060 0x7f2cfc161ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:13.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.011+0000 7f2cfb200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f2cfc162000 0x7f2cfc070c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:13.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.011+0000 7f2cfb200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cfc1627f0 con 0x7f2cfc15efc0 2024-03-04T09:27:13.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.011+0000 7f2cfb200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cfc15f3a0 con 0x7f2cfc0cd060 2024-03-04T09:27:13.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.011+0000 7f2cfb200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cfc06d190 con 0x7f2cfc162000 2024-03-04T09:27:13.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.012+0000 7f2cf9400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f2cfc0cd060 0x7f2cfc161ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:13.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.012+0000 7f2cf9e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f2cfc15efc0 0x7f2cfc0cc950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:45464/0 (socket says 172.21.15.6:45464) 2024-03-04T09:27:13.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.012+0000 7f2cf9e00640 1 -- 172.21.15.6:0/446098906 learned_addr learned my addr 172.21.15.6:0/446098906 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:27:13.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.012+0000 7f2cfa800640 1 --2- 172.21.15.6:0/446098906 >> v2:172.21.15.146:3300/0 conn(0x7f2cfc162000 0x7f2cfc070c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:13.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.012+0000 7f2cf9400640 1 -- 172.21.15.6:0/446098906 >> v2:172.21.15.146:3300/0 conn(0x7f2cfc162000 msgr2=0x7f2cfc070c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:27:13.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.012+0000 7f2cf9400640 1 --2- 172.21.15.6:0/446098906 >> v2:172.21.15.146:3300/0 conn(0x7f2cfc162000 0x7f2cfc070c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:13.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.012+0000 7f2cf9400640 1 -- 172.21.15.6:0/446098906 >> v1:172.21.15.6:6789/0 conn(0x7f2cfc15efc0 legacy=0x7f2cfc0cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:13.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.012+0000 7f2cf9400640 1 -- 172.21.15.6:0/446098906 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cfc06c860 con 0x7f2cfc0cd060 2024-03-04T09:27:13.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.012+0000 7f2cf8a00640 1 -- 172.21.15.6:0/446098906 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1985848286 0 0) 0x7f2cfc1627f0 con 0x7f2cfc15efc0 2024-03-04T09:27:13.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.014+0000 7f2cf9400640 1 --2- 172.21.15.6:0/446098906 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2cfc0cd060 0x7f2cfc161ac0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f2cfc0cf9e0 tx=0x7f2ce402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f1068360012752f server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:13.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.014+0000 7f2cf8a00640 1 -- 172.21.15.6:0/446098906 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ce4035020 con 0x7f2cfc0cd060 2024-03-04T09:27:13.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.014+0000 7f2cf8a00640 1 -- 172.21.15.6:0/446098906 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ce4031830 con 0x7f2cfc0cd060 2024-03-04T09:27:13.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.014+0000 7f2cf8a00640 1 -- 172.21.15.6:0/446098906 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ce4031c00 con 0x7f2cfc0cd060 2024-03-04T09:27:13.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.014+0000 7f2cfb200640 1 -- 172.21.15.6:0/446098906 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2cfc0cd060 msgr2=0x7f2cfc161ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:13.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.014+0000 7f2cfb200640 1 --2- 172.21.15.6:0/446098906 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2cfc0cd060 0x7f2cfc161ac0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f2cfc0cf9e0 tx=0x7f2ce402b8d0 comp rx=0 tx=0).stop 2024-03-04T09:27:13.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.015+0000 7f2cfb200640 1 -- 172.21.15.6:0/446098906 shutdown_connections 2024-03-04T09:27:13.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.015+0000 7f2cfb200640 1 --2- 172.21.15.6:0/446098906 >> v2:172.21.15.146:3300/0 conn(0x7f2cfc162000 0x7f2cfc070c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:13.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.015+0000 7f2cfb200640 1 --2- 172.21.15.6:0/446098906 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2cfc0cd060 0x7f2cfc161ac0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:13.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.015+0000 7f2cfb200640 1 -- 172.21.15.6:0/446098906 >> 172.21.15.6:0/446098906 conn(0x7f2cfc06b3d0 msgr2=0x7f2cfc06b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:13.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.015+0000 7f2cfb200640 1 -- 172.21.15.6:0/446098906 shutdown_connections 2024-03-04T09:27:13.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.015+0000 7f2cfb200640 1 -- 172.21.15.6:0/446098906 wait complete. 2024-03-04T09:27:13.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.016+0000 7f2cfb200640 1 Processor -- start 2024-03-04T09:27:13.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.016+0000 7f2cfb200640 1 -- start start 2024-03-04T09:27:13.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.017+0000 7f2cfb200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2cfc0cd060 0x7f2cfc147260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:13.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.017+0000 7f2cfb200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2cfc15efc0 0x7f2cfc0caf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:13.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.017+0000 7f2cfb200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2cfc162000 0x7f2cfc0cb490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:13.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.017+0000 7f2cfb200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cfc162470 con 0x7f2cfc15efc0 2024-03-04T09:27:13.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.017+0000 7f2cfb200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cfc06ce20 con 0x7f2cfc0cd060 2024-03-04T09:27:13.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.017+0000 7f2cfb200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cfc06d620 con 0x7f2cfc162000 2024-03-04T09:27:13.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.018+0000 7f2cf9400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2cfc15efc0 0x7f2cfc0caf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:13.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.018+0000 7f2cf9400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2cfc15efc0 0x7f2cfc0caf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38080/0 (socket says 172.21.15.6:38080) 2024-03-04T09:27:13.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.018+0000 7f2cf9400640 1 -- 172.21.15.6:0/1714838588 learned_addr learned my addr 172.21.15.6:0/1714838588 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:27:13.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.018+0000 7f2cfa800640 1 --2- 172.21.15.6:0/1714838588 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2cfc162000 0x7f2cfc0cb490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:13.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.018+0000 7f2cf9e00640 1 --2- 172.21.15.6:0/1714838588 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2cfc0cd060 0x7f2cfc147260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:13.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.018+0000 7f2cf9400640 1 -- 172.21.15.6:0/1714838588 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2cfc162000 msgr2=0x7f2cfc0cb490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:13.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.018+0000 7f2cf9400640 1 --2- 172.21.15.6:0/1714838588 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2cfc162000 0x7f2cfc0cb490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:13.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.018+0000 7f2cf9400640 1 -- 172.21.15.6:0/1714838588 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2cfc0cd060 msgr2=0x7f2cfc147260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:13.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.018+0000 7f2cf9400640 1 --2- 172.21.15.6:0/1714838588 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2cfc0cd060 0x7f2cfc147260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:13.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.018+0000 7f2cf9400640 1 -- 172.21.15.6:0/1714838588 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cfc1e8380 con 0x7f2cfc15efc0 2024-03-04T09:27:13.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.019+0000 7f2cf9400640 1 --2- 172.21.15.6:0/1714838588 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2cfc15efc0 0x7f2cfc0caf50 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f2ce4009890 tx=0x7f2ce40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:13.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.019+0000 7f2cf2000640 1 -- 172.21.15.6:0/1714838588 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ce4041070 con 0x7f2cfc15efc0 2024-03-04T09:27:13.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.019+0000 7f2cf2000640 1 -- 172.21.15.6:0/1714838588 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ce403aa30 con 0x7f2cfc15efc0 2024-03-04T09:27:13.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.019+0000 7f2cf2000640 1 -- 172.21.15.6:0/1714838588 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ce4035040 con 0x7f2cfc15efc0 2024-03-04T09:27:13.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.019+0000 7f2cfb200640 1 -- 172.21.15.6:0/1714838588 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cfc1e8660 con 0x7f2cfc15efc0 2024-03-04T09:27:13.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.019+0000 7f2cfb200640 1 -- 172.21.15.6:0/1714838588 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cfc1e8c20 con 0x7f2cfc15efc0 2024-03-04T09:27:13.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.020+0000 7f2cfb200640 1 -- 172.21.15.6:0/1714838588 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cfc04f330 con 0x7f2cfc15efc0 2024-03-04T09:27:13.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.021+0000 7f2cf2000640 1 -- 172.21.15.6:0/1714838588 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2ce4038020 con 0x7f2cfc15efc0 2024-03-04T09:27:13.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.023+0000 7f2cf2000640 1 --2- 172.21.15.6:0/1714838588 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2cd806b550 0x7f2cd806da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:13.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.023+0000 7f2cf2000640 1 -- 172.21.15.6:0/1714838588 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f2ce40ab950 con 0x7f2cfc15efc0 2024-03-04T09:27:13.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.023+0000 7f2cf9e00640 1 --2- 172.21.15.6:0/1714838588 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2cd806b550 0x7f2cd806da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:13.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.024+0000 7f2cf9e00640 1 --2- 172.21.15.6:0/1714838588 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2cd806b550 0x7f2cd806da10 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f2cec000f60 tx=0x7f2cec007b50 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:13.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.028+0000 7f2cf2000640 1 -- 172.21.15.6:0/1714838588 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2ce402f2e0 con 0x7f2cfc15efc0 2024-03-04T09:27:13.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.232+0000 7f2cfb200640 1 -- 172.21.15.6:0/1714838588 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2cfc0722e0 con 0x7f2cfc15efc0 2024-03-04T09:27:13.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.234+0000 7f2cf2000640 1 -- 172.21.15.6:0/1714838588 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+12843 (secure 0 0 0) 0x7f2ce407ab00 con 0x7f2cfc15efc0 2024-03-04T09:27:13.231 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:27:13.231 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":40,"fsid":"edce0578-5ae4-4402-b45a-552bdef84d7c","created":"2024-03-04T09:24:54.523780+0000","modified":"2024-03-04T09:27:12.256581+0000","last_up_change":"2024-03-04T09:25:07.089049+0000","last_in_change":"2024-03-04T09:24:57.982077+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T09:25:07.935198+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a96574f-81f1-4bc6-8463-757520b84a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6825","nonce":1364449401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6827","nonce":1364449401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6831","nonce":1364449401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6829","nonce":1364449401}]},"public_addr":"172.21.15.6:6825/1364449401","cluster_addr":"172.21.15.6:6827/1364449401","heartbeat_back_addr":"172.21.15.6:6831/1364449401","heartbeat_front_addr":"172.21.15.6:6829/1364449401","state":["exists","up"]},{"osd":1,"uuid":"7d79b9c1-6b87-476a-bc36-7db5ee64cd90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6809","nonce":2494103078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6811","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6813","nonce":2494103078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6821","nonce":2494103078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6817","nonce":2494103078}]},"public_addr":"172.21.15.6:6809/2494103078","cluster_addr":"172.21.15.6:6813/2494103078","heartbeat_back_addr":"172.21.15.6:6821/2494103078","heartbeat_front_addr":"172.21.15.6:6817/2494103078","state":["exists","up"]},{"osd":2,"uuid":"492331b9-4ee9-40b6-bacd-85d44cd6c45a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6812","nonce":1500066681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6816","nonce":1500066681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6824","nonce":1500066681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6820","nonce":1500066681}]},"public_addr":"172.21.15.6:6812/1500066681","cluster_addr":"172.21.15.6:6816/1500066681","heartbeat_back_addr":"172.21.15.6:6824/1500066681","heartbeat_front_addr":"172.21.15.6:6820/1500066681","state":["exists","up"]},{"osd":3,"uuid":"21bb2efe-9ba7-4bdc-bc31-180cd1102e21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6801","nonce":1558806123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6803","nonce":1558806123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6807","nonce":1558806123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6805","nonce":1558806123}]},"public_addr":"172.21.15.6:6801/1558806123","cluster_addr":"172.21.15.6:6803/1558806123","heartbeat_back_addr":"172.21.15.6:6807/1558806123","heartbeat_front_addr":"172.21.15.6:6805/1558806123","state":["exists","up"]},{"osd":4,"uuid":"6e2c4b71-19d9-4737-83d9-558705d8644a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6801","nonce":1526135780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6803","nonce":1526135780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6807","nonce":1526135780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6805","nonce":1526135780}]},"public_addr":"172.21.15.146:6801/1526135780","cluster_addr":"172.21.15.146:6803/1526135780","heartbeat_back_addr":"172.21.15.146:6807/1526135780","heartbeat_front_addr":"172.21.15.146:6805/1526135780","state":["exists","up"]},{"osd":5,"uuid":"fb69c525-f1be-4b7d-ab32-6c29b0d6fddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6809","nonce":460262507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6811","nonce":460262507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6815","nonce":460262507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6813","nonce":460262507}]},"public_addr":"172.21.15.146:6809/460262507","cluster_addr":"172.21.15.146:6811/460262507","heartbeat_back_addr":"172.21.15.146:6815/460262507","heartbeat_front_addr":"172.21.15.146:6813/460262507","state":["exists","up"]},{"osd":6,"uuid":"c32e7fed-370d-4401-8ce0-6a4335b0d168","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6823","nonce":1392897493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6827","nonce":1392897493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6831","nonce":1392897493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6829","nonce":1392897493}]},"public_addr":"172.21.15.146:6823/1392897493","cluster_addr":"172.21.15.146:6827/1392897493","heartbeat_back_addr":"172.21.15.146:6831/1392897493","heartbeat_front_addr":"172.21.15.146:6829/1392897493","state":["exists","up"]},{"osd":7,"uuid":"75a79b44-fa5f-426d-a7d3-71f21f78835c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6817","nonce":3268323156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6819","nonce":3268323156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6825","nonce":3268323156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6822","nonce":3268323156}]},"public_addr":"172.21.15.146:6817/3268323156","cluster_addr":"172.21.15.146:6819/3268323156","heartbeat_back_addr":"172.21.15.146:6825/3268323156","heartbeat_front_addr":"172.21.15.146:6822/3268323156","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.310437+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.295850+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.294674+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.526855+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.300023+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.595160+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.346046+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.558741+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T09:27:13.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.237+0000 7f2cfb200640 1 -- 172.21.15.6:0/1714838588 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2cd806b550 msgr2=0x7f2cd806da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:13.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.237+0000 7f2cfb200640 1 --2- 172.21.15.6:0/1714838588 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2cd806b550 0x7f2cd806da10 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f2cec000f60 tx=0x7f2cec007b50 comp rx=0 tx=0).stop 2024-03-04T09:27:13.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.237+0000 7f2cfb200640 1 -- 172.21.15.6:0/1714838588 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2cfc15efc0 msgr2=0x7f2cfc0caf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:13.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.237+0000 7f2cfb200640 1 --2- 172.21.15.6:0/1714838588 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2cfc15efc0 0x7f2cfc0caf50 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f2ce4009890 tx=0x7f2ce40046c0 comp rx=0 tx=0).stop 2024-03-04T09:27:13.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.237+0000 7f2cfb200640 1 -- 172.21.15.6:0/1714838588 shutdown_connections 2024-03-04T09:27:13.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.237+0000 7f2cfb200640 1 --2- 172.21.15.6:0/1714838588 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2cd806b550 0x7f2cd806da10 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:13.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.237+0000 7f2cfb200640 1 --2- 172.21.15.6:0/1714838588 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2cfc162000 0x7f2cfc0cb490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:13.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.237+0000 7f2cfb200640 1 --2- 172.21.15.6:0/1714838588 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2cfc15efc0 0x7f2cfc0caf50 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:13.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.237+0000 7f2cfb200640 1 --2- 172.21.15.6:0/1714838588 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2cfc0cd060 0x7f2cfc147260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:13.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.237+0000 7f2cfb200640 1 -- 172.21.15.6:0/1714838588 >> 172.21.15.6:0/1714838588 conn(0x7f2cfc06b3d0 msgr2=0x7f2cfc0ad880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:13.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.237+0000 7f2cfb200640 1 -- 172.21.15.6:0/1714838588 shutdown_connections 2024-03-04T09:27:13.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.237+0000 7f2cfb200640 1 -- 172.21.15.6:0/1714838588 wait complete. 2024-03-04T09:27:13.253 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T09:25:07.935198+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-04T09:27:13.253 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-04T09:27:13.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.469+0000 7ff7a9800640 1 Processor -- start 2024-03-04T09:27:13.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.469+0000 7ff7a9800640 1 -- start start 2024-03-04T09:27:13.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.471+0000 7ff7a9800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff7a4072520 0x7ff7a40cdfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:13.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.471+0000 7ff7a9800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ff7a4071e20 0x7ff7a40ce4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:13.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.471+0000 7ff7a9800640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7a4072df0 con 0x7ff7a415bf20 2024-03-04T09:27:13.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.471+0000 7ff7a9800640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7a40cdab0 con 0x7ff7a4071e20 2024-03-04T09:27:13.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.471+0000 7ff7a9800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7a40b4d70 con 0x7ff7a4072520 2024-03-04T09:27:13.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.472+0000 7ff7a2000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff7a4072520 0x7ff7a40cdfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:13.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.472+0000 7ff7a3400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ff7a4071e20 0x7ff7a40ce4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:13.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.472+0000 7ff7a3400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ff7a4071e20 0x7ff7a40ce4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.6:36242/0 (socket says 172.21.15.6:36242) 2024-03-04T09:27:13.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.472+0000 7ff7a3400640 1 -- 172.21.15.6:0/2859704810 learned_addr learned my addr 172.21.15.6:0/2859704810 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:27:13.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.472+0000 7ff7a2a00640 1 --1- 172.21.15.6:0/2859704810 >> v1:172.21.15.146:6789/0 conn(0x7ff7a415bf20 0x7ff7a40acd40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.6:41564/0 (socket says 172.21.15.6:41564) 2024-03-04T09:27:13.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.473+0000 7ff7a2000640 1 -- 172.21.15.6:0/2859704810 >> v2:172.21.15.146:3300/0 conn(0x7ff7a4071e20 msgr2=0x7ff7a40ce4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:27:13.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.473+0000 7ff7a2000640 1 --2- 172.21.15.6:0/2859704810 >> v2:172.21.15.146:3300/0 conn(0x7ff7a4071e20 0x7ff7a40ce4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:13.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.473+0000 7ff7a2000640 1 -- 172.21.15.6:0/2859704810 >> v1:172.21.15.146:6789/0 conn(0x7ff7a415bf20 legacy=0x7ff7a40acd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:13.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.473+0000 7ff7a2000640 1 -- 172.21.15.6:0/2859704810 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7a40b5050 con 0x7ff7a4072520 2024-03-04T09:27:13.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.474+0000 7ff7a2000640 1 --2- 172.21.15.6:0/2859704810 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff7a4072520 0x7ff7a40cdfb0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff7900097a0 tx=0x7ff79002b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=60c6a2c79eb8cfae server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:13.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.475+0000 7ff7a1600640 1 -- 172.21.15.6:0/2859704810 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff790035020 con 0x7ff7a4072520 2024-03-04T09:27:13.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.475+0000 7ff7a1600640 1 -- 172.21.15.6:0/2859704810 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff790031810 con 0x7ff7a4072520 2024-03-04T09:27:13.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.475+0000 7ff7a1600640 1 -- 172.21.15.6:0/2859704810 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff790031be0 con 0x7ff7a4072520 2024-03-04T09:27:13.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.475+0000 7ff7a9800640 1 -- 172.21.15.6:0/2859704810 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff7a4072520 msgr2=0x7ff7a40cdfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:13.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.475+0000 7ff7a9800640 1 --2- 172.21.15.6:0/2859704810 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff7a4072520 0x7ff7a40cdfb0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff7900097a0 tx=0x7ff79002b8b0 comp rx=0 tx=0).stop 2024-03-04T09:27:13.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.475+0000 7ff7a9800640 1 -- 172.21.15.6:0/2859704810 shutdown_connections 2024-03-04T09:27:13.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.475+0000 7ff7a9800640 1 --2- 172.21.15.6:0/2859704810 >> v2:172.21.15.146:3300/0 conn(0x7ff7a4071e20 0x7ff7a40ce4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:13.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.475+0000 7ff7a9800640 1 --2- 172.21.15.6:0/2859704810 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff7a4072520 0x7ff7a40cdfb0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:13.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.476+0000 7ff7a9800640 1 -- 172.21.15.6:0/2859704810 >> 172.21.15.6:0/2859704810 conn(0x7ff7a406b3d0 msgr2=0x7ff7a406b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:13.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.476+0000 7ff7a9800640 1 -- 172.21.15.6:0/2859704810 shutdown_connections 2024-03-04T09:27:13.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.476+0000 7ff7a9800640 1 -- 172.21.15.6:0/2859704810 wait complete. 2024-03-04T09:27:13.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.477+0000 7ff7a9800640 1 Processor -- start 2024-03-04T09:27:13.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.478+0000 7ff7a9800640 1 -- start start 2024-03-04T09:27:13.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.478+0000 7ff7a9800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff7a4071e20 0x7ff7a41402d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:13.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.478+0000 7ff7a9800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff7a4072520 0x7ff7a40b1390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:13.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.479+0000 7ff7a9800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff7a415bf20 0x7ff7a413cde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:13.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.479+0000 7ff7a9800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7a40ad450 con 0x7ff7a4071e20 2024-03-04T09:27:13.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.479+0000 7ff7a9800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7a40b4990 con 0x7ff7a4072520 2024-03-04T09:27:13.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.479+0000 7ff7a9800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7a40b1bc0 con 0x7ff7a415bf20 2024-03-04T09:27:13.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.479+0000 7ff7a2a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff7a4071e20 0x7ff7a41402d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:13.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.479+0000 7ff7a2a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff7a4071e20 0x7ff7a41402d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38096/0 (socket says 172.21.15.6:38096) 2024-03-04T09:27:13.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.479+0000 7ff7a2a00640 1 -- 172.21.15.6:0/591572505 learned_addr learned my addr 172.21.15.6:0/591572505 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:27:13.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.479+0000 7ff7a2000640 1 --2- 172.21.15.6:0/591572505 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff7a4072520 0x7ff7a40b1390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:13.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.479+0000 7ff7a3400640 1 --2- 172.21.15.6:0/591572505 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff7a415bf20 0x7ff7a413cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:13.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.479+0000 7ff7a2a00640 1 -- 172.21.15.6:0/591572505 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff7a415bf20 msgr2=0x7ff7a413cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:13.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.479+0000 7ff7a2a00640 1 --2- 172.21.15.6:0/591572505 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff7a415bf20 0x7ff7a413cde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:13.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.479+0000 7ff7a2a00640 1 -- 172.21.15.6:0/591572505 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff7a4072520 msgr2=0x7ff7a40b1390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:13.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.479+0000 7ff7a2a00640 1 --2- 172.21.15.6:0/591572505 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff7a4072520 0x7ff7a40b1390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:13.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.480+0000 7ff7a2a00640 1 -- 172.21.15.6:0/591572505 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7a40b1ea0 con 0x7ff7a4071e20 2024-03-04T09:27:13.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.480+0000 7ff7a2a00640 1 --2- 172.21.15.6:0/591572505 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff7a4071e20 0x7ff7a41402d0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7ff79400c3c0 tx=0x7ff79400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:13.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.480+0000 7ff79aa00640 1 -- 172.21.15.6:0/591572505 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7940074e0 con 0x7ff7a4071e20 2024-03-04T09:27:13.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.480+0000 7ff79aa00640 1 -- 172.21.15.6:0/591572505 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff794007680 con 0x7ff7a4071e20 2024-03-04T09:27:13.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.480+0000 7ff79aa00640 1 -- 172.21.15.6:0/591572505 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff794004a00 con 0x7ff7a4071e20 2024-03-04T09:27:13.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.480+0000 7ff7a9800640 1 -- 172.21.15.6:0/591572505 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7a40b2180 con 0x7ff7a4071e20 2024-03-04T09:27:13.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.480+0000 7ff7a9800640 1 -- 172.21.15.6:0/591572505 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7a40b18d0 con 0x7ff7a4071e20 2024-03-04T09:27:13.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.482+0000 7ff7a9800640 1 -- 172.21.15.6:0/591572505 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7a404f330 con 0x7ff7a4071e20 2024-03-04T09:27:13.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.482+0000 7ff79aa00640 1 -- 172.21.15.6:0/591572505 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff79401a020 con 0x7ff7a4071e20 2024-03-04T09:27:13.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.483+0000 7ff79aa00640 1 --2- 172.21.15.6:0/591572505 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff78406b4d0 0x7ff78406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:13.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.483+0000 7ff79aa00640 1 -- 172.21.15.6:0/591572505 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4857+0+0 (secure 0 0 0) 0x7ff794089630 con 0x7ff7a4071e20 2024-03-04T09:27:13.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.484+0000 7ff7a2000640 1 --2- 172.21.15.6:0/591572505 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff78406b4d0 0x7ff78406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:13.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.485+0000 7ff7a2000640 1 --2- 172.21.15.6:0/591572505 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff78406b4d0 0x7ff78406d990 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7ff790036000 tx=0x7ff79003c040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:13.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.488+0000 7ff79aa00640 1 -- 172.21.15.6:0/591572505 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff7940587f0 con 0x7ff7a4071e20 2024-03-04T09:27:13.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.688+0000 7ff7a9800640 1 -- 172.21.15.6:0/591572505 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff7a40cf750 con 0x7ff7a4071e20 2024-03-04T09:27:13.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.689+0000 7ff79aa00640 1 -- 172.21.15.6:0/591572505 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v40) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff7940580f0 con 0x7ff7a4071e20 2024-03-04T09:27:13.686 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-04T09:27:13.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.692+0000 7ff7a9800640 1 -- 172.21.15.6:0/591572505 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff78406b4d0 msgr2=0x7ff78406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:13.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.692+0000 7ff7a9800640 1 --2- 172.21.15.6:0/591572505 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff78406b4d0 0x7ff78406d990 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7ff790036000 tx=0x7ff79003c040 comp rx=0 tx=0).stop 2024-03-04T09:27:13.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.692+0000 7ff7a9800640 1 -- 172.21.15.6:0/591572505 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff7a4071e20 msgr2=0x7ff7a41402d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:13.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.692+0000 7ff7a9800640 1 --2- 172.21.15.6:0/591572505 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff7a4071e20 0x7ff7a41402d0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7ff79400c3c0 tx=0x7ff79400c880 comp rx=0 tx=0).stop 2024-03-04T09:27:13.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.692+0000 7ff7a9800640 1 -- 172.21.15.6:0/591572505 shutdown_connections 2024-03-04T09:27:13.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.692+0000 7ff7a9800640 1 --2- 172.21.15.6:0/591572505 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff78406b4d0 0x7ff78406d990 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:13.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.692+0000 7ff7a9800640 1 --2- 172.21.15.6:0/591572505 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff7a415bf20 0x7ff7a413cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:13.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.692+0000 7ff7a9800640 1 --2- 172.21.15.6:0/591572505 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff7a4072520 0x7ff7a40b1390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:13.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.692+0000 7ff7a9800640 1 --2- 172.21.15.6:0/591572505 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff7a4071e20 0x7ff7a41402d0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:13.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.692+0000 7ff7a9800640 1 -- 172.21.15.6:0/591572505 >> 172.21.15.6:0/591572505 conn(0x7ff7a406b3d0 msgr2=0x7ff7a40ccaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:13.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.692+0000 7ff7a9800640 1 -- 172.21.15.6:0/591572505 shutdown_connections 2024-03-04T09:27:13.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.692+0000 7ff7a9800640 1 -- 172.21.15.6:0/591572505 wait complete. 2024-03-04T09:27:13.709 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-04T09:27:13.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.928+0000 7f46ae800640 1 Processor -- start 2024-03-04T09:27:13.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.929+0000 7f46ae800640 1 -- start start 2024-03-04T09:27:13.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.930+0000 7f46ae800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f46a815efc0 0x7f46a80cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:13.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.930+0000 7f46ae800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f46a80cce90 0x7f46a8161ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:13.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.930+0000 7f46ae800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f46a8162000 0x7f46a80ad880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:13.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.930+0000 7f46ae800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46a815f3a0 con 0x7f46a80cce90 2024-03-04T09:27:13.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.930+0000 7f46ae800640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46a806d190 con 0x7f46a8162000 2024-03-04T09:27:13.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.930+0000 7f46ae800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46a81627f0 con 0x7f46a815efc0 2024-03-04T09:27:13.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.931+0000 7f46aca00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f46a80cce90 0x7f46a8161ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:13.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.931+0000 7f46aca00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f46a80cce90 0x7f46a8161ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38102/0 (socket says 172.21.15.6:38102) 2024-03-04T09:27:13.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.931+0000 7f46aca00640 1 -- 172.21.15.6:0/2982826622 learned_addr learned my addr 172.21.15.6:0/2982826622 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:27:13.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.931+0000 7f46aca00640 1 -- 172.21.15.6:0/2982826622 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f46a815efc0 msgr2=0x7f46a80cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:27:13.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.931+0000 7f46ad400640 1 --2- 172.21.15.6:0/2982826622 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f46a815efc0 0x7f46a80cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:13.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.931+0000 7f46aca00640 1 --2- 172.21.15.6:0/2982826622 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f46a815efc0 0x7f46a80cc950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:13.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.931+0000 7f46ade00640 1 --2- 172.21.15.6:0/2982826622 >> v2:172.21.15.146:3300/0 conn(0x7f46a8162000 0x7f46a80ad880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:13.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.931+0000 7f46aca00640 1 -- 172.21.15.6:0/2982826622 >> v2:172.21.15.146:3300/0 conn(0x7f46a8162000 msgr2=0x7f46a80ad880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:27:13.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.931+0000 7f46aca00640 1 --2- 172.21.15.6:0/2982826622 >> v2:172.21.15.146:3300/0 conn(0x7f46a8162000 0x7f46a80ad880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:13.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.931+0000 7f46aca00640 1 -- 172.21.15.6:0/2982826622 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46a806c860 con 0x7f46a80cce90 2024-03-04T09:27:13.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.932+0000 7f46aca00640 1 --2- 172.21.15.6:0/2982826622 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f46a80cce90 0x7f46a8161ac0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f46a80b4bb0 tx=0x7f469402b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cde6d6d8a4ae47c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:13.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.933+0000 7f46a7e00640 1 -- 172.21.15.6:0/2982826622 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4694035020 con 0x7f46a80cce90 2024-03-04T09:27:13.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.933+0000 7f46a7e00640 1 -- 172.21.15.6:0/2982826622 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f469402d040 con 0x7f46a80cce90 2024-03-04T09:27:13.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.933+0000 7f46a7e00640 1 -- 172.21.15.6:0/2982826622 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4694031a80 con 0x7f46a80cce90 2024-03-04T09:27:13.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.933+0000 7f46ae800640 1 -- 172.21.15.6:0/2982826622 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f46a80cce90 msgr2=0x7f46a8161ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:13.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.933+0000 7f46ae800640 1 --2- 172.21.15.6:0/2982826622 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f46a80cce90 0x7f46a8161ac0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f46a80b4bb0 tx=0x7f469402b900 comp rx=0 tx=0).stop 2024-03-04T09:27:13.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.934+0000 7f46ae800640 1 -- 172.21.15.6:0/2982826622 shutdown_connections 2024-03-04T09:27:13.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.934+0000 7f46ae800640 1 --2- 172.21.15.6:0/2982826622 >> v2:172.21.15.146:3300/0 conn(0x7f46a8162000 0x7f46a80ad880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:13.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.934+0000 7f46ae800640 1 --2- 172.21.15.6:0/2982826622 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f46a80cce90 0x7f46a8161ac0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:13.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.934+0000 7f46ae800640 1 --2- 172.21.15.6:0/2982826622 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f46a815efc0 0x7f46a80cc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:13.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.934+0000 7f46ae800640 1 -- 172.21.15.6:0/2982826622 >> 172.21.15.6:0/2982826622 conn(0x7f46a806b3d0 msgr2=0x7f46a806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:13.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.934+0000 7f46ae800640 1 -- 172.21.15.6:0/2982826622 shutdown_connections 2024-03-04T09:27:13.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.934+0000 7f46ae800640 1 -- 172.21.15.6:0/2982826622 wait complete. 2024-03-04T09:27:13.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.935+0000 7f46ae800640 1 Processor -- start 2024-03-04T09:27:13.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.936+0000 7f46ae800640 1 -- start start 2024-03-04T09:27:13.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.936+0000 7f46ae800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f46a80cce90 0x7f46a81e9690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:13.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.936+0000 7f46ae800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f46a815efc0 0x7f46a80b3ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:13.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.937+0000 7f46ae800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f46a8162000 0x7f46a80b40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:13.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.937+0000 7f46ae800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46a8162490 con 0x7f46a8162000 2024-03-04T09:27:13.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.937+0000 7f46ae800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46a806ce20 con 0x7f46a80cce90 2024-03-04T09:27:13.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.937+0000 7f46ae800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46a806d620 con 0x7f46a815efc0 2024-03-04T09:27:13.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.937+0000 7f46ade00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f46a8162000 0x7f46a80b40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:13.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.937+0000 7f46ade00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f46a8162000 0x7f46a80b40e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38118/0 (socket says 172.21.15.6:38118) 2024-03-04T09:27:13.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.937+0000 7f46ade00640 1 -- 172.21.15.6:0/503801076 learned_addr learned my addr 172.21.15.6:0/503801076 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:27:13.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.937+0000 7f46aca00640 1 --2- 172.21.15.6:0/503801076 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f46a815efc0 0x7f46a80b3ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:13.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.937+0000 7f46ade00640 1 -- 172.21.15.6:0/503801076 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f46a815efc0 msgr2=0x7f46a80b3ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:13.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.937+0000 7f46ade00640 1 --2- 172.21.15.6:0/503801076 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f46a815efc0 0x7f46a80b3ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:13.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.937+0000 7f46ad400640 1 --2- 172.21.15.6:0/503801076 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f46a80cce90 0x7f46a81e9690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:13.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.937+0000 7f46ade00640 1 -- 172.21.15.6:0/503801076 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f46a80cce90 msgr2=0x7f46a81e9690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:13.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.937+0000 7f46ade00640 1 --2- 172.21.15.6:0/503801076 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f46a80cce90 0x7f46a81e9690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:13.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.937+0000 7f46ade00640 1 -- 172.21.15.6:0/503801076 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46a81440e0 con 0x7f46a8162000 2024-03-04T09:27:13.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.938+0000 7f46ade00640 1 --2- 172.21.15.6:0/503801076 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f46a8162000 0x7f46a80b40e0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f46a000c550 tx=0x7f46a000ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:13.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.938+0000 7f46a5600640 1 -- 172.21.15.6:0/503801076 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46a0015070 con 0x7f46a8162000 2024-03-04T09:27:13.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.938+0000 7f46a5600640 1 -- 172.21.15.6:0/503801076 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46a0004220 con 0x7f46a8162000 2024-03-04T09:27:13.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.938+0000 7f46a5600640 1 -- 172.21.15.6:0/503801076 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46a0003610 con 0x7f46a8162000 2024-03-04T09:27:13.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.938+0000 7f46ae800640 1 -- 172.21.15.6:0/503801076 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46a81443c0 con 0x7f46a8162000 2024-03-04T09:27:13.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.938+0000 7f46ae800640 1 -- 172.21.15.6:0/503801076 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46a8144980 con 0x7f46a8162000 2024-03-04T09:27:13.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.939+0000 7f46ae800640 1 -- 172.21.15.6:0/503801076 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46a804f330 con 0x7f46a8162000 2024-03-04T09:27:13.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.940+0000 7f46a5600640 1 -- 172.21.15.6:0/503801076 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f46a00037b0 con 0x7f46a8162000 2024-03-04T09:27:13.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.941+0000 7f46a5600640 1 --2- 172.21.15.6:0/503801076 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f468806b520 0x7f468806d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:13.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.941+0000 7f46a5600640 1 -- 172.21.15.6:0/503801076 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f46a0089c20 con 0x7f46a8162000 2024-03-04T09:27:13.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.942+0000 7f46ad400640 1 --2- 172.21.15.6:0/503801076 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f468806b520 0x7f468806d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:13.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.943+0000 7f46ad400640 1 --2- 172.21.15.6:0/503801076 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f468806b520 0x7f468806d9e0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f4698008ef0 tx=0x7f4698008440 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:13.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:13.945+0000 7f46a5600640 1 -- 172.21.15.6:0/503801076 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f46a0058dd0 con 0x7f46a8162000 2024-03-04T09:27:14.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:14.147+0000 7f46ae800640 1 -- 172.21.15.6:0/503801076 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f46a8157c70 con 0x7f46a8162000 2024-03-04T09:27:14.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:14.148+0000 7f46a5600640 1 -- 172.21.15.6:0/503801076 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v40) v1 ==== 84+0+65 (secure 0 0 0) 0x7f46a00586d0 con 0x7f46a8162000 2024-03-04T09:27:14.145 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:27:14.145 INFO:teuthology.orchestra.run.smithi006.stdout:[ 2024-03-04T09:27:14.145 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:27:14.145 INFO:teuthology.orchestra.run.smithi006.stdout: "poolnum": 1, 2024-03-04T09:27:14.145 INFO:teuthology.orchestra.run.smithi006.stdout: "poolname": ".mgr" 2024-03-04T09:27:14.145 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:27:14.145 INFO:teuthology.orchestra.run.smithi006.stdout:] 2024-03-04T09:27:14.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:14.150+0000 7f46ae800640 1 -- 172.21.15.6:0/503801076 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f468806b520 msgr2=0x7f468806d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:14.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:14.150+0000 7f46ae800640 1 --2- 172.21.15.6:0/503801076 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f468806b520 0x7f468806d9e0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f4698008ef0 tx=0x7f4698008440 comp rx=0 tx=0).stop 2024-03-04T09:27:14.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:14.150+0000 7f46ae800640 1 -- 172.21.15.6:0/503801076 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f46a8162000 msgr2=0x7f46a80b40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:14.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:14.150+0000 7f46ae800640 1 --2- 172.21.15.6:0/503801076 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f46a8162000 0x7f46a80b40e0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f46a000c550 tx=0x7f46a000ca10 comp rx=0 tx=0).stop 2024-03-04T09:27:14.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:14.151+0000 7f46ae800640 1 -- 172.21.15.6:0/503801076 shutdown_connections 2024-03-04T09:27:14.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:14.151+0000 7f46ae800640 1 --2- 172.21.15.6:0/503801076 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f46a8162000 0x7f46a80b40e0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:14.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:14.151+0000 7f46ae800640 1 --2- 172.21.15.6:0/503801076 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f468806b520 0x7f468806d9e0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:14.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:14.151+0000 7f46ae800640 1 --2- 172.21.15.6:0/503801076 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f46a815efc0 0x7f46a80b3ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:14.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:14.151+0000 7f46ae800640 1 --2- 172.21.15.6:0/503801076 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f46a80cce90 0x7f46a81e9690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:14.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:14.151+0000 7f46ae800640 1 -- 172.21.15.6:0/503801076 >> 172.21.15.6:0/503801076 conn(0x7f46a806b3d0 msgr2=0x7f46a80cdae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:14.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:14.151+0000 7f46ae800640 1 -- 172.21.15.6:0/503801076 shutdown_connections 2024-03-04T09:27:14.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:14.151+0000 7f46ae800640 1 -- 172.21.15.6:0/503801076 wait complete. 2024-03-04T09:27:14.167 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-04T09:27:14.168 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-04T09:27:14.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:14.388+0000 7fa203200640 1 Processor -- start 2024-03-04T09:27:14.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:14.388+0000 7fa203200640 1 -- start start 2024-03-04T09:27:14.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:14.390+0000 7fa203200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa1fc0ae7b0 0x7fa1fc0aced0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:14.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:14.390+0000 7fa203200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa1fc15cdc0 0x7fa1fc06cc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:14.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:14.390+0000 7fa203200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1fc1531f0 con 0x7fa1fc0b3640 2024-03-04T09:27:14.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:14.390+0000 7fa203200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1fc0d04d0 con 0x7fa1fc0ae7b0 2024-03-04T09:27:14.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:14.390+0000 7fa203200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1fc0b2b80 con 0x7fa1fc15cdc0 2024-03-04T09:27:14.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:14.390+0000 7fa1fbe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa1fc0ae7b0 0x7fa1fc0aced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:14.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:14.390+0000 7fa1f3400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fa1fc0b3640 0x7fa1fc0aeb90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:45472/0 (socket says 172.21.15.6:45472) 2024-03-04T09:27:14.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:14.390+0000 7fa1f3400640 1 -- 172.21.15.6:0/2011696679 learned_addr learned my addr 172.21.15.6:0/2011696679 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:27:14.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:14.391+0000 7fa201200640 1 --2- 172.21.15.6:0/2011696679 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa1fc15cdc0 0x7fa1fc06cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:14.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:14.391+0000 7fa1fbe00640 1 -- 172.21.15.6:0/2011696679 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa1fc15cdc0 msgr2=0x7fa1fc06cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:27:14.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:14.391+0000 7fa1fbe00640 1 --2- 172.21.15.6:0/2011696679 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa1fc15cdc0 0x7fa1fc06cc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:14.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:14.391+0000 7fa1fbe00640 1 -- 172.21.15.6:0/2011696679 >> v1:172.21.15.6:6789/0 conn(0x7fa1fc0b3640 legacy=0x7fa1fc0aeb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:14.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:14.391+0000 7fa1fbe00640 1 -- 172.21.15.6:0/2011696679 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1fc06ebd0 con 0x7fa1fc0ae7b0 2024-03-04T09:27:14.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:14.391+0000 7fa1fb400640 1 -- 172.21.15.6:0/2011696679 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2367323152 0 0) 0x7fa1fc1531f0 con 0x7fa1fc0b3640 2024-03-04T09:27:14.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:14.392+0000 7fa1fbe00640 1 --2- 172.21.15.6:0/2011696679 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa1fc0ae7b0 0x7fa1fc0aced0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fa1ec009d90 tx=0x7fa1ec02bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=278d09009a63dd0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:14.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:14.392+0000 7fa1fb400640 1 -- 172.21.15.6:0/2011696679 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1ec039070 con 0x7fa1fc0ae7b0 2024-03-04T09:27:14.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:14.392+0000 7fa1fb400640 1 -- 172.21.15.6:0/2011696679 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1ec02cc50 con 0x7fa1fc0ae7b0 2024-03-04T09:27:14.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:14.392+0000 7fa1fb400640 1 -- 172.21.15.6:0/2011696679 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1ec034440 con 0x7fa1fc0ae7b0 2024-03-04T09:27:14.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:14.393+0000 7fa203200640 1 -- 172.21.15.6:0/2011696679 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa1fc0ae7b0 msgr2=0x7fa1fc0aced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:14.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:14.393+0000 7fa203200640 1 --2- 172.21.15.6:0/2011696679 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa1fc0ae7b0 0x7fa1fc0aced0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fa1ec009d90 tx=0x7fa1ec02bea0 comp rx=0 tx=0).stop 2024-03-04T09:27:14.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:14.393+0000 7fa203200640 1 -- 172.21.15.6:0/2011696679 shutdown_connections 2024-03-04T09:27:14.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:14.393+0000 7fa203200640 1 --2- 172.21.15.6:0/2011696679 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa1fc15cdc0 0x7fa1fc06cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:14.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:14.393+0000 7fa203200640 1 --2- 172.21.15.6:0/2011696679 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa1fc0ae7b0 0x7fa1fc0aced0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:14.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:14.393+0000 7fa203200640 1 -- 172.21.15.6:0/2011696679 >> 172.21.15.6:0/2011696679 conn(0x7fa1fc06afb0 msgr2=0x7fa1fc06b3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:14.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:14.393+0000 7fa203200640 1 -- 172.21.15.6:0/2011696679 shutdown_connections 2024-03-04T09:27:14.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:14.393+0000 7fa203200640 1 -- 172.21.15.6:0/2011696679 wait complete. 2024-03-04T09:27:14.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:14.395+0000 7fa203200640 1 Processor -- start 2024-03-04T09:27:14.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:14.395+0000 7fa203200640 1 -- start start 2024-03-04T09:27:14.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:14.396+0000 7fa203200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa1fc0ae7b0 0x7fa1fc14a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:14.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:14.396+0000 7fa203200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa1fc0b3640 0x7fa1fc153fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:14.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:14.396+0000 7fa203200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa1fc15cdc0 0x7fa1fc154520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:14.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:14.396+0000 7fa203200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1fc0b3120 con 0x7fa1fc15cdc0 2024-03-04T09:27:14.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:14.396+0000 7fa203200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1fc0b2e40 con 0x7fa1fc0ae7b0 2024-03-04T09:27:14.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:14.396+0000 7fa203200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1fc15d250 con 0x7fa1fc0b3640 2024-03-04T09:27:14.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:14.396+0000 7fa201200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa1fc15cdc0 0x7fa1fc154520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:14.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:14.397+0000 7fa201200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa1fc15cdc0 0x7fa1fc154520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38134/0 (socket says 172.21.15.6:38134) 2024-03-04T09:27:14.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:14.397+0000 7fa201200640 1 -- 172.21.15.6:0/3781378124 learned_addr learned my addr 172.21.15.6:0/3781378124 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:27:14.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:14.397+0000 7fa1f3400640 1 --2- 172.21.15.6:0/3781378124 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa1fc0b3640 0x7fa1fc153fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:14.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:14.397+0000 7fa1fbe00640 1 --2- 172.21.15.6:0/3781378124 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa1fc0ae7b0 0x7fa1fc14a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:14.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:14.397+0000 7fa201200640 1 -- 172.21.15.6:0/3781378124 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa1fc0b3640 msgr2=0x7fa1fc153fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:14.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:14.397+0000 7fa201200640 1 --2- 172.21.15.6:0/3781378124 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa1fc0b3640 0x7fa1fc153fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:14.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:14.397+0000 7fa201200640 1 -- 172.21.15.6:0/3781378124 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa1fc0ae7b0 msgr2=0x7fa1fc14a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:14.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:14.397+0000 7fa201200640 1 --2- 172.21.15.6:0/3781378124 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa1fc0ae7b0 0x7fa1fc14a590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:14.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:14.397+0000 7fa201200640 1 -- 172.21.15.6:0/3781378124 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1fc14aad0 con 0x7fa1fc15cdc0 2024-03-04T09:27:14.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:14.397+0000 7fa1fbe00640 1 --2- 172.21.15.6:0/3781378124 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa1fc0ae7b0 0x7fa1fc14a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:27:14.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:14.397+0000 7fa201200640 1 --2- 172.21.15.6:0/3781378124 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa1fc15cdc0 0x7fa1fc154520 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fa1f400d490 tx=0x7fa1f400d950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:14.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:14.398+0000 7fa1f8c00640 1 -- 172.21.15.6:0/3781378124 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1f40125f0 con 0x7fa1fc15cdc0 2024-03-04T09:27:14.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:14.398+0000 7fa1f8c00640 1 -- 172.21.15.6:0/3781378124 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1f4019070 con 0x7fa1fc15cdc0 2024-03-04T09:27:14.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:14.398+0000 7fa1f8c00640 1 -- 172.21.15.6:0/3781378124 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1f40129f0 con 0x7fa1fc15cdc0 2024-03-04T09:27:14.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:14.398+0000 7fa203200640 1 -- 172.21.15.6:0/3781378124 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1fc154d30 con 0x7fa1fc15cdc0 2024-03-04T09:27:14.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:14.398+0000 7fa203200640 1 -- 172.21.15.6:0/3781378124 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1fc1552f0 con 0x7fa1fc15cdc0 2024-03-04T09:27:14.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:14.399+0000 7fa203200640 1 -- 172.21.15.6:0/3781378124 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1fc068010 con 0x7fa1fc15cdc0 2024-03-04T09:27:14.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:14.400+0000 7fa1f8c00640 1 -- 172.21.15.6:0/3781378124 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa1f4012b90 con 0x7fa1fc15cdc0 2024-03-04T09:27:14.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:14.400+0000 7fa1f8c00640 1 --2- 172.21.15.6:0/3781378124 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa1d806b4d0 0x7fa1d806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:14.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:14.401+0000 7fa1f8c00640 1 -- 172.21.15.6:0/3781378124 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fa1f408a140 con 0x7fa1fc15cdc0 2024-03-04T09:27:14.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:14.401+0000 7fa1fbe00640 1 --2- 172.21.15.6:0/3781378124 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa1d806b4d0 0x7fa1d806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:14.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:14.402+0000 7fa1fbe00640 1 --2- 172.21.15.6:0/3781378124 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa1d806b4d0 0x7fa1d806d990 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fa1ec036000 tx=0x7fa1ec030000 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:14.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:14.405+0000 7fa1f8c00640 1 -- 172.21.15.6:0/3781378124 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa1f4059380 con 0x7fa1fc15cdc0 2024-03-04T09:27:14.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:14.604+0000 7fa203200640 1 -- 172.21.15.6:0/3781378124 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fa1fc0b4670 con 0x7fa1fc15cdc0 2024-03-04T09:27:15.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:15.273+0000 7fa1f8c00640 1 -- 172.21.15.6:0/3781378124 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v41) v1 ==== 138+0+0 (secure 0 0 0) 0x7fa1f4058c80 con 0x7fa1fc15cdc0 2024-03-04T09:27:15.270 INFO:teuthology.orchestra.run.smithi006.stderr:pool 'cephfs_metadata' created 2024-03-04T09:27:15.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:15.277+0000 7fa203200640 1 -- 172.21.15.6:0/3781378124 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa1d806b4d0 msgr2=0x7fa1d806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:15.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:15.277+0000 7fa203200640 1 --2- 172.21.15.6:0/3781378124 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa1d806b4d0 0x7fa1d806d990 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fa1ec036000 tx=0x7fa1ec030000 comp rx=0 tx=0).stop 2024-03-04T09:27:15.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:15.277+0000 7fa203200640 1 -- 172.21.15.6:0/3781378124 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa1fc15cdc0 msgr2=0x7fa1fc154520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:15.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:15.277+0000 7fa203200640 1 --2- 172.21.15.6:0/3781378124 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa1fc15cdc0 0x7fa1fc154520 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fa1f400d490 tx=0x7fa1f400d950 comp rx=0 tx=0).stop 2024-03-04T09:27:15.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:15.277+0000 7fa203200640 1 -- 172.21.15.6:0/3781378124 shutdown_connections 2024-03-04T09:27:15.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:15.277+0000 7fa203200640 1 --2- 172.21.15.6:0/3781378124 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa1fc15cdc0 0x7fa1fc154520 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:15.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:15.277+0000 7fa203200640 1 --2- 172.21.15.6:0/3781378124 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa1d806b4d0 0x7fa1d806d990 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:15.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:15.277+0000 7fa203200640 1 --2- 172.21.15.6:0/3781378124 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa1fc0b3640 0x7fa1fc153fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:15.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:15.277+0000 7fa203200640 1 --2- 172.21.15.6:0/3781378124 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa1fc0ae7b0 0x7fa1fc14a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:15.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:15.277+0000 7fa203200640 1 -- 172.21.15.6:0/3781378124 >> 172.21.15.6:0/3781378124 conn(0x7fa1fc06afb0 msgr2=0x7fa1fc0aeb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:15.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:15.278+0000 7fa203200640 1 -- 172.21.15.6:0/3781378124 shutdown_connections 2024-03-04T09:27:15.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:15.278+0000 7fa203200640 1 -- 172.21.15.6:0/3781378124 wait complete. 2024-03-04T09:27:15.306 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-04T09:27:15.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:15.529+0000 7f86f8c00640 1 Processor -- start 2024-03-04T09:27:15.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:15.529+0000 7f86f8c00640 1 -- start start 2024-03-04T09:27:15.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:15.530+0000 7f86f8c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f86f4145570 0x7f86f40cef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:15.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:15.531+0000 7f86f8c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f86f40b42c0 0x7f86f40ccd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:15.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:15.531+0000 7f86f8c00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86f40b4ae0 con 0x7f86f40cd260 2024-03-04T09:27:15.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:15.531+0000 7f86f8c00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86f406c9f0 con 0x7f86f40b42c0 2024-03-04T09:27:15.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:15.531+0000 7f86f8c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86f406cfb0 con 0x7f86f4145570 2024-03-04T09:27:15.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:15.531+0000 7f86ebe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f86f40b42c0 0x7f86f40ccd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:15.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:15.531+0000 7f86ebe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f86f40b42c0 0x7f86f40ccd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38136/0 (socket says 172.21.15.6:38136) 2024-03-04T09:27:15.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:15.531+0000 7f86ebe00640 1 -- 172.21.15.6:0/1861295635 learned_addr learned my addr 172.21.15.6:0/1861295635 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:27:15.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:15.532+0000 7f86f3400640 1 --2- 172.21.15.6:0/1861295635 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f86f4145570 0x7f86f40cef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:15.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:15.532+0000 7f86ebe00640 1 -- 172.21.15.6:0/1861295635 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f86f4145570 msgr2=0x7f86f40cef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:27:15.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:15.532+0000 7f86ebe00640 1 --2- 172.21.15.6:0/1861295635 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f86f4145570 0x7f86f40cef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:15.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:15.532+0000 7f86ebe00640 1 -- 172.21.15.6:0/1861295635 >> v1:172.21.15.146:6789/0 conn(0x7f86f40cd260 legacy=0x7f86f415f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:27:15.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:15.532+0000 7f86ebe00640 1 -- 172.21.15.6:0/1861295635 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86f40b4dc0 con 0x7f86f40b42c0 2024-03-04T09:27:15.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:15.533+0000 7f86ebe00640 1 --2- 172.21.15.6:0/1861295635 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f86f40b42c0 0x7f86f40ccd20 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f86dc009870 tx=0x7f86dc02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a47a48a41e9d2bd server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:15.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:15.533+0000 7f86f2a00640 1 -- 172.21.15.6:0/1861295635 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86dc035020 con 0x7f86f40b42c0 2024-03-04T09:27:15.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:15.533+0000 7f86f2a00640 1 -- 172.21.15.6:0/1861295635 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86dc031640 con 0x7f86f40b42c0 2024-03-04T09:27:15.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:15.533+0000 7f86f2a00640 1 -- 172.21.15.6:0/1861295635 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86dc02d040 con 0x7f86f40b42c0 2024-03-04T09:27:15.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:15.534+0000 7f86f8c00640 1 -- 172.21.15.6:0/1861295635 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f86f40b42c0 msgr2=0x7f86f40ccd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:15.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:15.534+0000 7f86f8c00640 1 --2- 172.21.15.6:0/1861295635 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f86f40b42c0 0x7f86f40ccd20 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f86dc009870 tx=0x7f86dc02b8f0 comp rx=0 tx=0).stop 2024-03-04T09:27:15.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:15.534+0000 7f86f8c00640 1 -- 172.21.15.6:0/1861295635 shutdown_connections 2024-03-04T09:27:15.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:15.534+0000 7f86f8c00640 1 --2- 172.21.15.6:0/1861295635 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f86f40b42c0 0x7f86f40ccd20 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:15.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:15.534+0000 7f86f8c00640 1 --2- 172.21.15.6:0/1861295635 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f86f4145570 0x7f86f40cef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:15.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:15.534+0000 7f86f8c00640 1 -- 172.21.15.6:0/1861295635 >> 172.21.15.6:0/1861295635 conn(0x7f86f406b190 msgr2=0x7f86f406b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:15.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:15.534+0000 7f86f8c00640 1 -- 172.21.15.6:0/1861295635 shutdown_connections 2024-03-04T09:27:15.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:15.534+0000 7f86f8c00640 1 -- 172.21.15.6:0/1861295635 wait complete. 2024-03-04T09:27:15.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:15.536+0000 7f86f8c00640 1 Processor -- start 2024-03-04T09:27:15.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:15.536+0000 7f86f8c00640 1 -- start start 2024-03-04T09:27:15.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:15.537+0000 7f86f8c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f86f40b42c0 0x7f86f4137470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:15.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:15.537+0000 7f86f8c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f86f40cd260 0x7f86f4139940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:15.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:15.537+0000 7f86f8c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f86f4145570 0x7f86f4137fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:15.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:15.537+0000 7f86f8c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86f415c110 con 0x7f86f40cd260 2024-03-04T09:27:15.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:15.537+0000 7f86f8c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86f406e450 con 0x7f86f4145570 2024-03-04T09:27:15.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:15.537+0000 7f86f8c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86f40b4740 con 0x7f86f40b42c0 2024-03-04T09:27:15.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:15.537+0000 7f86ebe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f86f40cd260 0x7f86f4139940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:15.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:15.538+0000 7f86ebe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f86f40cd260 0x7f86f4139940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38142/0 (socket says 172.21.15.6:38142) 2024-03-04T09:27:15.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:15.538+0000 7f86ebe00640 1 -- 172.21.15.6:0/1396517502 learned_addr learned my addr 172.21.15.6:0/1396517502 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:27:15.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:15.538+0000 7f86f3e00640 1 --2- 172.21.15.6:0/1396517502 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f86f4145570 0x7f86f4137fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:15.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:15.538+0000 7f86f3400640 1 --2- 172.21.15.6:0/1396517502 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f86f40b42c0 0x7f86f4137470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:15.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:15.538+0000 7f86ebe00640 1 -- 172.21.15.6:0/1396517502 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f86f40b42c0 msgr2=0x7f86f4137470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:15.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:15.538+0000 7f86ebe00640 1 --2- 172.21.15.6:0/1396517502 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f86f40b42c0 0x7f86f4137470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:15.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:15.538+0000 7f86ebe00640 1 -- 172.21.15.6:0/1396517502 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f86f4145570 msgr2=0x7f86f4137fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:15.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:15.538+0000 7f86ebe00640 1 --2- 172.21.15.6:0/1396517502 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f86f4145570 0x7f86f4137fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:15.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:15.538+0000 7f86ebe00640 1 -- 172.21.15.6:0/1396517502 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86f413a960 con 0x7f86f40cd260 2024-03-04T09:27:15.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:15.538+0000 7f86ebe00640 1 --2- 172.21.15.6:0/1396517502 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f86f40cd260 0x7f86f4139940 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f86dc000c00 tx=0x7f86dc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:15.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:15.539+0000 7f86eb400640 1 -- 172.21.15.6:0/1396517502 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86dc040070 con 0x7f86f40cd260 2024-03-04T09:27:15.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:15.539+0000 7f86eb400640 1 -- 172.21.15.6:0/1396517502 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86dc039bf0 con 0x7f86f40cd260 2024-03-04T09:27:15.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:15.539+0000 7f86f8c00640 1 -- 172.21.15.6:0/1396517502 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86f413a0c0 con 0x7f86f40cd260 2024-03-04T09:27:15.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:15.539+0000 7f86eb400640 1 -- 172.21.15.6:0/1396517502 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86dc035040 con 0x7f86f40cd260 2024-03-04T09:27:15.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:15.539+0000 7f86f8c00640 1 -- 172.21.15.6:0/1396517502 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86f413a680 con 0x7f86f40cd260 2024-03-04T09:27:15.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:15.540+0000 7f86f8c00640 1 -- 172.21.15.6:0/1396517502 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86f404f330 con 0x7f86f40cd260 2024-03-04T09:27:15.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:15.542+0000 7f86eb400640 1 -- 172.21.15.6:0/1396517502 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f86dc04b060 con 0x7f86f40cd260 2024-03-04T09:27:15.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:15.542+0000 7f86eb400640 1 --2- 172.21.15.6:0/1396517502 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f86d006b500 0x7f86d006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:15.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:15.543+0000 7f86eb400640 1 -- 172.21.15.6:0/1396517502 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f86dc0abb00 con 0x7f86f40cd260 2024-03-04T09:27:15.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:15.543+0000 7f86f3400640 1 --2- 172.21.15.6:0/1396517502 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f86d006b500 0x7f86d006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:15.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:15.544+0000 7f86f3400640 1 --2- 172.21.15.6:0/1396517502 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f86d006b500 0x7f86d006d9c0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f86e4006fd0 tx=0x7f86e4008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:15.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:15.548+0000 7f86eb400640 1 -- 172.21.15.6:0/1396517502 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f86dc031610 con 0x7f86f40cd260 2024-03-04T09:27:15.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:15.750+0000 7f86f8c00640 1 -- 172.21.15.6:0/1396517502 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f86f40b3310 con 0x7f86f40cd260 2024-03-04T09:27:16.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:16.290+0000 7f86eb400640 1 -- 172.21.15.6:0/1396517502 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v42) v1 ==== 170+0+0 (secure 0 0 0) 0x7f86dc07ab50 con 0x7f86f40cd260 2024-03-04T09:27:16.288 INFO:teuthology.orchestra.run.smithi006.stderr:pool 'cephfs_data' created 2024-03-04T09:27:16.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:16.297+0000 7f86f8c00640 1 -- 172.21.15.6:0/1396517502 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f86d006b500 msgr2=0x7f86d006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:16.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:16.297+0000 7f86f8c00640 1 --2- 172.21.15.6:0/1396517502 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f86d006b500 0x7f86d006d9c0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f86e4006fd0 tx=0x7f86e4008040 comp rx=0 tx=0).stop 2024-03-04T09:27:16.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:16.297+0000 7f86f8c00640 1 -- 172.21.15.6:0/1396517502 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f86f40cd260 msgr2=0x7f86f4139940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:16.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:16.297+0000 7f86f8c00640 1 --2- 172.21.15.6:0/1396517502 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f86f40cd260 0x7f86f4139940 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f86dc000c00 tx=0x7f86dc0046c0 comp rx=0 tx=0).stop 2024-03-04T09:27:16.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:16.298+0000 7f86f8c00640 1 -- 172.21.15.6:0/1396517502 shutdown_connections 2024-03-04T09:27:16.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:16.298+0000 7f86f8c00640 1 --2- 172.21.15.6:0/1396517502 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f86f4145570 0x7f86f4137fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:16.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:16.298+0000 7f86f8c00640 1 --2- 172.21.15.6:0/1396517502 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f86f40cd260 0x7f86f4139940 secure :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f86dc000c00 tx=0x7f86dc0046c0 comp rx=0 tx=0).stop 2024-03-04T09:27:16.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:16.298+0000 7f86f8c00640 1 --2- 172.21.15.6:0/1396517502 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f86d006b500 0x7f86d006d9c0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:16.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:16.298+0000 7f86f8c00640 1 --2- 172.21.15.6:0/1396517502 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f86f40b42c0 0x7f86f4137470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:16.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:16.298+0000 7f86f8c00640 1 -- 172.21.15.6:0/1396517502 >> 172.21.15.6:0/1396517502 conn(0x7f86f406b190 msgr2=0x7f86f40b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:16.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:16.305+0000 7f86f8c00640 1 -- 172.21.15.6:0/1396517502 shutdown_connections 2024-03-04T09:27:16.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:16.307+0000 7f86f8c00640 1 -- 172.21.15.6:0/1396517502 wait complete. 2024-03-04T09:27:16.334 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-04T09:27:16.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:16.523+0000 7f0858600640 1 Processor -- start 2024-03-04T09:27:16.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:16.523+0000 7f0858600640 1 -- start start 2024-03-04T09:27:16.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:16.525+0000 7f0858600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f08500ccad0 0x7f08500cceb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:16.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:16.525+0000 7f0858600640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08500cd3f0 con 0x7f08500adb00 2024-03-04T09:27:16.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:16.525+0000 7f0858600640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0850163950 con 0x7f0850162bb0 2024-03-04T09:27:16.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:16.525+0000 7f0858600640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08500acd40 con 0x7f08500ccad0 2024-03-04T09:27:16.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:16.525+0000 7f0856600640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f08500adb00 0x7f08500adf10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:45486/0 (socket says 172.21.15.6:45486) 2024-03-04T09:27:16.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:16.525+0000 7f0856600640 1 -- 172.21.15.6:0/2476556938 learned_addr learned my addr 172.21.15.6:0/2476556938 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:27:16.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:16.525+0000 7f0855c00640 1 --2- 172.21.15.6:0/2476556938 >> v2:172.21.15.6:3300/0 conn(0x7f08500ccad0 0x7f08500cceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:16.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:16.526+0000 7f084fe00640 1 -- 172.21.15.6:0/2476556938 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3842218927 0 0) 0x7f08500cd3f0 con 0x7f08500adb00 2024-03-04T09:27:16.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:16.526+0000 7f084fe00640 1 -- 172.21.15.6:0/2476556938 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0838003520 con 0x7f08500adb00 2024-03-04T09:27:16.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:16.526+0000 7f0855c00640 1 -- 172.21.15.6:0/2476556938 >> v1:172.21.15.146:6789/0 conn(0x7f0850162bb0 legacy=0x7f0850163040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:27:16.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:16.526+0000 7f0855c00640 1 -- 172.21.15.6:0/2476556938 >> v1:172.21.15.6:6789/0 conn(0x7f08500adb00 legacy=0x7f08500adf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:16.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:16.526+0000 7f0855c00640 1 -- 172.21.15.6:0/2476556938 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08500ad020 con 0x7f08500ccad0 2024-03-04T09:27:16.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:16.527+0000 7f0855c00640 1 --2- 172.21.15.6:0/2476556938 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f08500ccad0 0x7f08500cceb0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f08440097a0 tx=0x7f084402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98c6be005caf95e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:16.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:16.528+0000 7f084fe00640 1 -- 172.21.15.6:0/2476556938 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0844035020 con 0x7f08500ccad0 2024-03-04T09:27:16.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:16.528+0000 7f084fe00640 1 -- 172.21.15.6:0/2476556938 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0844031830 con 0x7f08500ccad0 2024-03-04T09:27:16.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:16.528+0000 7f084fe00640 1 -- 172.21.15.6:0/2476556938 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0844031c00 con 0x7f08500ccad0 2024-03-04T09:27:16.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:16.528+0000 7f0858600640 1 -- 172.21.15.6:0/2476556938 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f08500ccad0 msgr2=0x7f08500cceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:16.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:16.528+0000 7f0858600640 1 --2- 172.21.15.6:0/2476556938 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f08500ccad0 0x7f08500cceb0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f08440097a0 tx=0x7f084402b8d0 comp rx=0 tx=0).stop 2024-03-04T09:27:16.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:16.528+0000 7f0858600640 1 -- 172.21.15.6:0/2476556938 shutdown_connections 2024-03-04T09:27:16.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:16.528+0000 7f0858600640 1 --2- 172.21.15.6:0/2476556938 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f08500ccad0 0x7f08500cceb0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:16.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:16.528+0000 7f0858600640 1 -- 172.21.15.6:0/2476556938 >> 172.21.15.6:0/2476556938 conn(0x7f085006b3d0 msgr2=0x7f085006b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:16.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:16.528+0000 7f0858600640 1 -- 172.21.15.6:0/2476556938 shutdown_connections 2024-03-04T09:27:16.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:16.528+0000 7f0858600640 1 -- 172.21.15.6:0/2476556938 wait complete. 2024-03-04T09:27:16.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:16.530+0000 7f0858600640 1 Processor -- start 2024-03-04T09:27:16.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:16.530+0000 7f0858600640 1 -- start start 2024-03-04T09:27:16.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:16.531+0000 7f0858600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f08500adb00 0x7f085009aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:16.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:16.531+0000 7f0858600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f08500ccad0 0x7f0850153e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:16.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:16.531+0000 7f0858600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0850162bb0 0x7f08500b48a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:16.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:16.531+0000 7f0858600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f085006e7d0 con 0x7f08500adb00 2024-03-04T09:27:16.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:16.531+0000 7f0858600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08500ad5e0 con 0x7f08500ccad0 2024-03-04T09:27:16.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:16.531+0000 7f0858600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f085006c980 con 0x7f0850162bb0 2024-03-04T09:27:16.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:16.531+0000 7f0855c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f08500adb00 0x7f085009aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:16.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:16.531+0000 7f0855c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f08500adb00 0x7f085009aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38168/0 (socket says 172.21.15.6:38168) 2024-03-04T09:27:16.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:16.531+0000 7f0855c00640 1 -- 172.21.15.6:0/106114996 learned_addr learned my addr 172.21.15.6:0/106114996 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:27:16.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:16.532+0000 7f0855200640 1 --2- 172.21.15.6:0/106114996 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f08500ccad0 0x7f0850153e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:16.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:16.532+0000 7f0855c00640 1 -- 172.21.15.6:0/106114996 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0850162bb0 msgr2=0x7f08500b48a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:16.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:16.532+0000 7f0856600640 1 --2- 172.21.15.6:0/106114996 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0850162bb0 0x7f08500b48a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:16.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:16.532+0000 7f0855c00640 1 --2- 172.21.15.6:0/106114996 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0850162bb0 0x7f08500b48a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:16.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:16.532+0000 7f0855c00640 1 -- 172.21.15.6:0/106114996 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f08500ccad0 msgr2=0x7f0850153e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:16.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:16.532+0000 7f0855c00640 1 --2- 172.21.15.6:0/106114996 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f08500ccad0 0x7f0850153e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:16.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:16.532+0000 7f0855c00640 1 -- 172.21.15.6:0/106114996 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08500b3ae0 con 0x7f08500adb00 2024-03-04T09:27:16.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:16.532+0000 7f0855c00640 1 --2- 172.21.15.6:0/106114996 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f08500adb00 0x7f085009aa10 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f0844000c00 tx=0x7f08440046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:16.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:16.532+0000 7f084d600640 1 -- 172.21.15.6:0/106114996 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0844040070 con 0x7f08500adb00 2024-03-04T09:27:16.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:16.533+0000 7f084d600640 1 -- 172.21.15.6:0/106114996 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f084403a930 con 0x7f08500adb00 2024-03-04T09:27:16.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:16.533+0000 7f084d600640 1 -- 172.21.15.6:0/106114996 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0844035040 con 0x7f08500adb00 2024-03-04T09:27:16.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:16.533+0000 7f0858600640 1 -- 172.21.15.6:0/106114996 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08500b3dc0 con 0x7f08500adb00 2024-03-04T09:27:16.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:16.533+0000 7f0858600640 1 -- 172.21.15.6:0/106114996 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08500b4380 con 0x7f08500adb00 2024-03-04T09:27:16.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:16.534+0000 7f0858600640 1 -- 172.21.15.6:0/106114996 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f085004f330 con 0x7f08500adb00 2024-03-04T09:27:16.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:16.535+0000 7f084d600640 1 -- 172.21.15.6:0/106114996 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f084404b060 con 0x7f08500adb00 2024-03-04T09:27:16.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:16.536+0000 7f084d600640 1 --2- 172.21.15.6:0/106114996 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f083006b500 0x7f083006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:16.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:16.536+0000 7f084d600640 1 -- 172.21.15.6:0/106114996 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f08440ab9f0 con 0x7f08500adb00 2024-03-04T09:27:16.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:16.537+0000 7f0855200640 1 --2- 172.21.15.6:0/106114996 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f083006b500 0x7f083006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:16.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:16.538+0000 7f0855200640 1 --2- 172.21.15.6:0/106114996 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f083006b500 0x7f083006d9c0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f085015cc70 tx=0x7f083c009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:16.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:16.541+0000 7f084d600640 1 -- 172.21.15.6:0/106114996 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f084402f2e0 con 0x7f08500adb00 2024-03-04T09:27:16.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:16.782+0000 7f0858600640 1 -- 172.21.15.6:0/106114996 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f08500b45c0 con 0x7f08500adb00 2024-03-04T09:27:17.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.306+0000 7f084d600640 1 -- 172.21.15.6:0/106114996 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 6 and data pool 7 v21) v1 ==== 172+0+0 (secure 0 0 0) 0x7f0844031800 con 0x7f08500adb00 2024-03-04T09:27:17.307 INFO:teuthology.orchestra.run.smithi006.stderr:new fs with metadata pool 6 and data pool 7 2024-03-04T09:27:17.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.318+0000 7f0858600640 1 -- 172.21.15.6:0/106114996 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f083006b500 msgr2=0x7f083006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:17.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.318+0000 7f0858600640 1 --2- 172.21.15.6:0/106114996 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f083006b500 0x7f083006d9c0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f085015cc70 tx=0x7f083c009040 comp rx=0 tx=0).stop 2024-03-04T09:27:17.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.318+0000 7f0858600640 1 -- 172.21.15.6:0/106114996 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f08500adb00 msgr2=0x7f085009aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:17.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.318+0000 7f0858600640 1 --2- 172.21.15.6:0/106114996 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f08500adb00 0x7f085009aa10 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f0844000c00 tx=0x7f08440046c0 comp rx=0 tx=0).stop 2024-03-04T09:27:17.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.319+0000 7f0858600640 1 -- 172.21.15.6:0/106114996 shutdown_connections 2024-03-04T09:27:17.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.319+0000 7f0858600640 1 --2- 172.21.15.6:0/106114996 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f083006b500 0x7f083006d9c0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:17.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.319+0000 7f0858600640 1 --2- 172.21.15.6:0/106114996 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0850162bb0 0x7f08500b48a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:17.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.319+0000 7f0858600640 1 --2- 172.21.15.6:0/106114996 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f08500ccad0 0x7f0850153e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:17.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.319+0000 7f0858600640 1 --2- 172.21.15.6:0/106114996 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f08500adb00 0x7f085009aa10 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:17.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.319+0000 7f0858600640 1 -- 172.21.15.6:0/106114996 >> 172.21.15.6:0/106114996 conn(0x7f085006b3d0 msgr2=0x7f0850161c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:17.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.320+0000 7f0858600640 1 -- 172.21.15.6:0/106114996 shutdown_connections 2024-03-04T09:27:17.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.321+0000 7f0858600640 1 -- 172.21.15.6:0/106114996 wait complete. 2024-03-04T09:27:17.351 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T09:27:17.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.535+0000 7f15db400640 1 Processor -- start 2024-03-04T09:27:17.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.535+0000 7f15db400640 1 -- start start 2024-03-04T09:27:17.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.537+0000 7f15db400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f15d40712c0 0x7f15d4162c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:17.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.537+0000 7f15db400640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15d40ce090 con 0x7f15d4072da0 2024-03-04T09:27:17.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.537+0000 7f15db400640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15d40ce370 con 0x7f15d4070ee0 2024-03-04T09:27:17.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.537+0000 7f15db400640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15d40ceca0 con 0x7f15d40712c0 2024-03-04T09:27:17.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.537+0000 7f15d3e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f15d4072da0 0x7f15d406cfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:45488/0 (socket says 172.21.15.6:45488) 2024-03-04T09:27:17.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.537+0000 7f15d3e00640 1 -- 172.21.15.6:0/1502219243 learned_addr learned my addr 172.21.15.6:0/1502219243 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:27:17.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.537+0000 7f15d9400640 1 --2- 172.21.15.6:0/1502219243 >> v2:172.21.15.146:3300/0 conn(0x7f15d40712c0 0x7f15d4162c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:17.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.538+0000 7f15d3400640 1 -- 172.21.15.6:0/1502219243 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1676417960 0 0) 0x7f15d40ce090 con 0x7f15d4072da0 2024-03-04T09:27:17.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.538+0000 7f15d3400640 1 -- 172.21.15.6:0/1502219243 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15b8003520 con 0x7f15d4072da0 2024-03-04T09:27:17.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.538+0000 7f15d3400640 1 -- 172.21.15.6:0/1502219243 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3202369650 0 0) 0x7f15b8003520 con 0x7f15d4072da0 2024-03-04T09:27:17.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.538+0000 7f15d3400640 1 -- 172.21.15.6:0/1502219243 >> v2:172.21.15.146:3300/0 conn(0x7f15d40712c0 msgr2=0x7f15d4162c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:27:17.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.538+0000 7f15d3400640 1 --2- 172.21.15.6:0/1502219243 >> v2:172.21.15.146:3300/0 conn(0x7f15d40712c0 0x7f15d4162c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:17.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.538+0000 7f15d3400640 1 -- 172.21.15.6:0/1502219243 >> v1:172.21.15.146:6789/0 conn(0x7f15d4070ee0 legacy=0x7f15d406e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:17.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.538+0000 7f15d3400640 1 -- 172.21.15.6:0/1502219243 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15d40ce650 con 0x7f15d4072da0 2024-03-04T09:27:17.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.538+0000 7f15d3400640 1 -- 172.21.15.6:0/1502219243 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7f15c0001670 con 0x7f15d4072da0 2024-03-04T09:27:17.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.538+0000 7f15d3400640 1 -- 172.21.15.6:0/1502219243 >> v1:172.21.15.6:6789/0 conn(0x7f15d4072da0 legacy=0x7f15d406cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:17.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.538+0000 7f15d3400640 1 --2- 172.21.15.6:0/1502219243 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f15b8003a20 0x7f15b8003e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:17.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.538+0000 7f15d3400640 1 --2- 172.21.15.6:0/1502219243 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f15b8004800 0x7f15b8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:17.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.538+0000 7f15d3400640 1 --2- 172.21.15.6:0/1502219243 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f15b80051c0 0x7f15b800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:17.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.538+0000 7f15d3400640 1 -- 172.21.15.6:0/1502219243 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15d40ceca0 con 0x7f15b8003a20 2024-03-04T09:27:17.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.538+0000 7f15d3400640 1 -- 172.21.15.6:0/1502219243 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15b800c0f0 con 0x7f15b8004800 2024-03-04T09:27:17.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.538+0000 7f15d3400640 1 -- 172.21.15.6:0/1502219243 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15b800c2f0 con 0x7f15b80051c0 2024-03-04T09:27:17.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.538+0000 7f15d8a00640 1 --2- 172.21.15.6:0/1502219243 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f15b8003a20 0x7f15b8003e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:17.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.539+0000 7f15d3e00640 1 --2- 172.21.15.6:0/1502219243 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f15b8004800 0x7f15b8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:17.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.539+0000 7f15d9400640 1 --2- 172.21.15.6:0/1502219243 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f15b80051c0 0x7f15b800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:17.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.539+0000 7f15d8a00640 1 -- 172.21.15.6:0/1502219243 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f15b80051c0 msgr2=0x7f15b800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:27:17.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.539+0000 7f15d8a00640 1 --2- 172.21.15.6:0/1502219243 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f15b80051c0 0x7f15b800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:17.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.539+0000 7f15d8a00640 1 -- 172.21.15.6:0/1502219243 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f15b8004800 msgr2=0x7f15b8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:27:17.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.539+0000 7f15d8a00640 1 --2- 172.21.15.6:0/1502219243 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f15b8004800 0x7f15b8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:17.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.539+0000 7f15d8a00640 1 -- 172.21.15.6:0/1502219243 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15b8003520 con 0x7f15b8003a20 2024-03-04T09:27:17.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.539+0000 7f15d9400640 1 --2- 172.21.15.6:0/1502219243 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f15b80051c0 0x7f15b800b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:27:17.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.539+0000 7f15d9400640 1 -- 172.21.15.6:0/1502219243 reap_dead start 2024-03-04T09:27:17.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.540+0000 7f15d8a00640 1 --2- 172.21.15.6:0/1502219243 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f15b8003a20 0x7f15b8003e60 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f15c4001e20 tx=0x7f15b8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bbb8e1ffdf2d9f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:17.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.540+0000 7f15d3400640 1 -- 172.21.15.6:0/1502219243 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15c4001a70 con 0x7f15b8003a20 2024-03-04T09:27:17.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.540+0000 7f15d3400640 1 -- 172.21.15.6:0/1502219243 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15c4003b70 con 0x7f15b8003a20 2024-03-04T09:27:17.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.540+0000 7f15d3400640 1 -- 172.21.15.6:0/1502219243 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15c40050d0 con 0x7f15b8003a20 2024-03-04T09:27:17.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.541+0000 7f15db400640 1 -- 172.21.15.6:0/1502219243 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f15b8003a20 msgr2=0x7f15b8003e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:17.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.541+0000 7f15db400640 1 --2- 172.21.15.6:0/1502219243 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f15b8003a20 0x7f15b8003e60 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f15c4001e20 tx=0x7f15b8005830 comp rx=0 tx=0).stop 2024-03-04T09:27:17.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.541+0000 7f15db400640 1 -- 172.21.15.6:0/1502219243 shutdown_connections 2024-03-04T09:27:17.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.541+0000 7f15db400640 1 --2- 172.21.15.6:0/1502219243 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f15b8003a20 0x7f15b8003e60 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:17.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.541+0000 7f15db400640 1 -- 172.21.15.6:0/1502219243 >> 172.21.15.6:0/1502219243 conn(0x7f15d406afb0 msgr2=0x7f15d406b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:17.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.541+0000 7f15db400640 1 -- 172.21.15.6:0/1502219243 shutdown_connections 2024-03-04T09:27:17.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.541+0000 7f15db400640 1 -- 172.21.15.6:0/1502219243 wait complete. 2024-03-04T09:27:17.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.543+0000 7f15db400640 1 Processor -- start 2024-03-04T09:27:17.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.543+0000 7f15db400640 1 -- start start 2024-03-04T09:27:17.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.544+0000 7f15db400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f15b8003a20 0x7f15d40ae170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:17.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.544+0000 7f15db400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f15d41fcb50 0x7f15d41fc220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:17.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.544+0000 7f15db400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f15d4147510 0x7f15d4147950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:17.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.544+0000 7f15db400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15d40cdda0 con 0x7f15d41fcb50 2024-03-04T09:27:17.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.544+0000 7f15db400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15d40cdc20 con 0x7f15b8003a20 2024-03-04T09:27:17.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.544+0000 7f15db400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15d4162c40 con 0x7f15d4147510 2024-03-04T09:27:17.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.544+0000 7f15d3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f15d41fcb50 0x7f15d41fc220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:17.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.544+0000 7f15d3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f15d41fcb50 0x7f15d41fc220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38194/0 (socket says 172.21.15.6:38194) 2024-03-04T09:27:17.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.544+0000 7f15d3e00640 1 -- 172.21.15.6:0/1820589278 learned_addr learned my addr 172.21.15.6:0/1820589278 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:27:17.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.545+0000 7f15d8a00640 1 --2- 172.21.15.6:0/1820589278 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f15b8003a20 0x7f15d40ae170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:17.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.545+0000 7f15d9400640 1 --2- 172.21.15.6:0/1820589278 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f15d4147510 0x7f15d4147950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:17.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.545+0000 7f15d3e00640 1 -- 172.21.15.6:0/1820589278 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f15d4147510 msgr2=0x7f15d4147950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:17.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.545+0000 7f15d3e00640 1 --2- 172.21.15.6:0/1820589278 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f15d4147510 0x7f15d4147950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:17.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.545+0000 7f15d3e00640 1 -- 172.21.15.6:0/1820589278 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f15b8003a20 msgr2=0x7f15d40ae170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:17.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.545+0000 7f15d3e00640 1 --2- 172.21.15.6:0/1820589278 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f15b8003a20 0x7f15d40ae170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:17.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.545+0000 7f15d3e00640 1 -- 172.21.15.6:0/1820589278 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15d40ae9a0 con 0x7f15d41fcb50 2024-03-04T09:27:17.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.545+0000 7f15d3e00640 1 --2- 172.21.15.6:0/1820589278 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f15d41fcb50 0x7f15d41fc220 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f15c000e730 tx=0x7f15c000ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:17.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.545+0000 7f15d0c00640 1 -- 172.21.15.6:0/1820589278 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15c000c6c0 con 0x7f15d41fcb50 2024-03-04T09:27:17.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.545+0000 7f15d0c00640 1 -- 172.21.15.6:0/1820589278 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15c0018070 con 0x7f15d41fcb50 2024-03-04T09:27:17.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.545+0000 7f15d0c00640 1 -- 172.21.15.6:0/1820589278 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15c000cac0 con 0x7f15d41fcb50 2024-03-04T09:27:17.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.545+0000 7f15db400640 1 -- 172.21.15.6:0/1820589278 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15d40aec80 con 0x7f15d41fcb50 2024-03-04T09:27:17.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.546+0000 7f15db400640 1 -- 172.21.15.6:0/1820589278 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15d406d460 con 0x7f15d41fcb50 2024-03-04T09:27:17.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.547+0000 7f15db400640 1 -- 172.21.15.6:0/1820589278 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15d4069510 con 0x7f15d41fcb50 2024-03-04T09:27:17.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.548+0000 7f15d0c00640 1 -- 172.21.15.6:0/1820589278 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f15c000cc60 con 0x7f15d41fcb50 2024-03-04T09:27:17.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.548+0000 7f15d0c00640 1 --2- 172.21.15.6:0/1820589278 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f15b406b500 0x7f15b406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:17.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.549+0000 7f15d0c00640 1 -- 172.21.15.6:0/1820589278 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f15c0089fe0 con 0x7f15d41fcb50 2024-03-04T09:27:17.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.549+0000 7f15d8a00640 1 --2- 172.21.15.6:0/1820589278 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f15b406b500 0x7f15b406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:17.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.551+0000 7f15d8a00640 1 --2- 172.21.15.6:0/1820589278 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f15b406b500 0x7f15b406d9c0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f15c4034000 tx=0x7f15c4009030 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:17.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.553+0000 7f15d0c00640 1 -- 172.21.15.6:0/1820589278 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f15c0058e20 con 0x7f15d41fcb50 2024-03-04T09:27:17.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.748+0000 7f15db400640 1 -- 172.21.15.6:0/1820589278 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f15d406d630 con 0x7f15d41fcb50 2024-03-04T09:27:17.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.751+0000 7f15d0c00640 1 -- 172.21.15.6:0/1820589278 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v43) v1 ==== 74+0+16578 (secure 0 0 0) 0x7f15c0058720 con 0x7f15d41fcb50 2024-03-04T09:27:17.749 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:27:17.749 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":43,"fsid":"edce0578-5ae4-4402-b45a-552bdef84d7c","created":"2024-03-04T09:24:54.523780+0000","modified":"2024-03-04T09:27:16.784497+0000","last_up_change":"2024-03-04T09:25:07.089049+0000","last_in_change":"2024-03-04T09:24:57.982077+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T09:25:07.935198+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-03-04T09:27:14.657490+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"43","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-03-04T09:27:15.806886+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"43","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a96574f-81f1-4bc6-8463-757520b84a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6825","nonce":1364449401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6827","nonce":1364449401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6831","nonce":1364449401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6829","nonce":1364449401}]},"public_addr":"172.21.15.6:6825/1364449401","cluster_addr":"172.21.15.6:6827/1364449401","heartbeat_back_addr":"172.21.15.6:6831/1364449401","heartbeat_front_addr":"172.21.15.6:6829/1364449401","state":["exists","up"]},{"osd":1,"uuid":"7d79b9c1-6b87-476a-bc36-7db5ee64cd90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6809","nonce":2494103078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6811","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6813","nonce":2494103078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6821","nonce":2494103078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6817","nonce":2494103078}]},"public_addr":"172.21.15.6:6809/2494103078","cluster_addr":"172.21.15.6:6813/2494103078","heartbeat_back_addr":"172.21.15.6:6821/2494103078","heartbeat_front_addr":"172.21.15.6:6817/2494103078","state":["exists","up"]},{"osd":2,"uuid":"492331b9-4ee9-40b6-bacd-85d44cd6c45a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6812","nonce":1500066681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6816","nonce":1500066681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6824","nonce":1500066681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6820","nonce":1500066681}]},"public_addr":"172.21.15.6:6812/1500066681","cluster_addr":"172.21.15.6:6816/1500066681","heartbeat_back_addr":"172.21.15.6:6824/1500066681","heartbeat_front_addr":"172.21.15.6:6820/1500066681","state":["exists","up"]},{"osd":3,"uuid":"21bb2efe-9ba7-4bdc-bc31-180cd1102e21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6801","nonce":1558806123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6803","nonce":1558806123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6807","nonce":1558806123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6805","nonce":1558806123}]},"public_addr":"172.21.15.6:6801/1558806123","cluster_addr":"172.21.15.6:6803/1558806123","heartbeat_back_addr":"172.21.15.6:6807/1558806123","heartbeat_front_addr":"172.21.15.6:6805/1558806123","state":["exists","up"]},{"osd":4,"uuid":"6e2c4b71-19d9-4737-83d9-558705d8644a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6801","nonce":1526135780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6803","nonce":1526135780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6807","nonce":1526135780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6805","nonce":1526135780}]},"public_addr":"172.21.15.146:6801/1526135780","cluster_addr":"172.21.15.146:6803/1526135780","heartbeat_back_addr":"172.21.15.146:6807/1526135780","heartbeat_front_addr":"172.21.15.146:6805/1526135780","state":["exists","up"]},{"osd":5,"uuid":"fb69c525-f1be-4b7d-ab32-6c29b0d6fddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6809","nonce":460262507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6811","nonce":460262507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6815","nonce":460262507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6813","nonce":460262507}]},"public_addr":"172.21.15.146:6809/460262507","cluster_addr":"172.21.15.146:6811/460262507","heartbeat_back_addr":"172.21.15.146:6815/460262507","heartbeat_front_addr":"172.21.15.146:6813/460262507","state":["exists","up"]},{"osd":6,"uuid":"c32e7fed-370d-4401-8ce0-6a4335b0d168","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6823","nonce":1392897493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6827","nonce":1392897493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6831","nonce":1392897493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6829","nonce":1392897493}]},"public_addr":"172.21.15.146:6823/1392897493","cluster_addr":"172.21.15.146:6827/1392897493","heartbeat_back_addr":"172.21.15.146:6831/1392897493","heartbeat_front_addr":"172.21.15.146:6829/1392897493","state":["exists","up"]},{"osd":7,"uuid":"75a79b44-fa5f-426d-a7d3-71f21f78835c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6817","nonce":3268323156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6819","nonce":3268323156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6825","nonce":3268323156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6822","nonce":3268323156}]},"public_addr":"172.21.15.146:6817/3268323156","cluster_addr":"172.21.15.146:6819/3268323156","heartbeat_back_addr":"172.21.15.146:6825/3268323156","heartbeat_front_addr":"172.21.15.146:6822/3268323156","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.310437+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.295850+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.294674+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.526855+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.300023+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.595160+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.346046+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.558741+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T09:27:17.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.754+0000 7f15db400640 1 -- 172.21.15.6:0/1820589278 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f15b406b500 msgr2=0x7f15b406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:17.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.754+0000 7f15db400640 1 --2- 172.21.15.6:0/1820589278 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f15b406b500 0x7f15b406d9c0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f15c4034000 tx=0x7f15c4009030 comp rx=0 tx=0).stop 2024-03-04T09:27:17.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.754+0000 7f15db400640 1 -- 172.21.15.6:0/1820589278 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f15d41fcb50 msgr2=0x7f15d41fc220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:17.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.754+0000 7f15db400640 1 --2- 172.21.15.6:0/1820589278 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f15d41fcb50 0x7f15d41fc220 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f15c000e730 tx=0x7f15c000ebf0 comp rx=0 tx=0).stop 2024-03-04T09:27:17.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.754+0000 7f15db400640 1 -- 172.21.15.6:0/1820589278 shutdown_connections 2024-03-04T09:27:17.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.754+0000 7f15db400640 1 --2- 172.21.15.6:0/1820589278 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f15b406b500 0x7f15b406d9c0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:17.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.754+0000 7f15db400640 1 --2- 172.21.15.6:0/1820589278 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f15d4147510 0x7f15d4147950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:17.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.754+0000 7f15db400640 1 --2- 172.21.15.6:0/1820589278 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f15d41fcb50 0x7f15d41fc220 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:17.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.754+0000 7f15db400640 1 --2- 172.21.15.6:0/1820589278 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f15b8003a20 0x7f15d40ae170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:17.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.754+0000 7f15db400640 1 -- 172.21.15.6:0/1820589278 >> 172.21.15.6:0/1820589278 conn(0x7f15d406afb0 msgr2=0x7f15d40afce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:17.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.754+0000 7f15db400640 1 -- 172.21.15.6:0/1820589278 shutdown_connections 2024-03-04T09:27:17.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.754+0000 7f15db400640 1 -- 172.21.15.6:0/1820589278 wait complete. 2024-03-04T09:27:17.768 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T09:27:17.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.969+0000 7f41e9c00640 1 Processor -- start 2024-03-04T09:27:17.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.969+0000 7f41e9c00640 1 -- start start 2024-03-04T09:27:17.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.971+0000 7f41e9c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f41e40b34e0 0x7f41e415c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:17.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.971+0000 7f41e9c00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41e415cc40 con 0x7f41e40af1a0 2024-03-04T09:27:17.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.971+0000 7f41e9c00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41e415cf20 con 0x7f41e40af580 2024-03-04T09:27:17.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.971+0000 7f41e9c00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41e415d200 con 0x7f41e40b34e0 2024-03-04T09:27:17.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.971+0000 7f41e3400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f41e40b34e0 0x7f41e415c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:17.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.971+0000 7f41e2a00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f41e40af1a0 0x7f41e40b4c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:45496/0 (socket says 172.21.15.6:45496) 2024-03-04T09:27:17.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.971+0000 7f41e2a00640 1 -- 172.21.15.6:0/2103545529 learned_addr learned my addr 172.21.15.6:0/2103545529 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:27:17.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.971+0000 7f41e1600640 1 -- 172.21.15.6:0/2103545529 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2892886972 0 0) 0x7f41e415cc40 con 0x7f41e40af1a0 2024-03-04T09:27:17.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.971+0000 7f41e1600640 1 -- 172.21.15.6:0/2103545529 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41c8003520 con 0x7f41e40af1a0 2024-03-04T09:27:17.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.972+0000 7f41e3400640 1 -- 172.21.15.6:0/2103545529 >> v1:172.21.15.146:6789/0 conn(0x7f41e40af580 legacy=0x7f41e40b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:27:17.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.972+0000 7f41e3400640 1 -- 172.21.15.6:0/2103545529 >> v1:172.21.15.6:6789/0 conn(0x7f41e40af1a0 legacy=0x7f41e40b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:17.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.972+0000 7f41e3400640 1 -- 172.21.15.6:0/2103545529 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41e415d4e0 con 0x7f41e40b34e0 2024-03-04T09:27:17.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.973+0000 7f41e3400640 1 --2- 172.21.15.6:0/2103545529 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f41e40b34e0 0x7f41e415c600 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f41dc009d90 tx=0x7f41dc02be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=256534739918ed8b server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:17.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.973+0000 7f41e1600640 1 -- 172.21.15.6:0/2103545529 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41dc039070 con 0x7f41e40b34e0 2024-03-04T09:27:17.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.973+0000 7f41e1600640 1 -- 172.21.15.6:0/2103545529 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41dc02cbd0 con 0x7f41e40b34e0 2024-03-04T09:27:17.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.973+0000 7f41e1600640 1 -- 172.21.15.6:0/2103545529 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41dc034440 con 0x7f41e40b34e0 2024-03-04T09:27:17.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.974+0000 7f41e9c00640 1 -- 172.21.15.6:0/2103545529 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f41e40b34e0 msgr2=0x7f41e415c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:17.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.974+0000 7f41e9c00640 1 --2- 172.21.15.6:0/2103545529 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f41e40b34e0 0x7f41e415c600 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f41dc009d90 tx=0x7f41dc02be20 comp rx=0 tx=0).stop 2024-03-04T09:27:17.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.974+0000 7f41e9c00640 1 -- 172.21.15.6:0/2103545529 shutdown_connections 2024-03-04T09:27:17.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.974+0000 7f41e9c00640 1 --2- 172.21.15.6:0/2103545529 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f41e40b34e0 0x7f41e415c600 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:17.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.974+0000 7f41e9c00640 1 -- 172.21.15.6:0/2103545529 >> 172.21.15.6:0/2103545529 conn(0x7f41e406afb0 msgr2=0x7f41e406b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:17.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.974+0000 7f41e9c00640 1 -- 172.21.15.6:0/2103545529 shutdown_connections 2024-03-04T09:27:17.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.974+0000 7f41e9c00640 1 -- 172.21.15.6:0/2103545529 wait complete. 2024-03-04T09:27:17.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.975+0000 7f41e9c00640 1 Processor -- start 2024-03-04T09:27:17.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.976+0000 7f41e9c00640 1 -- start start 2024-03-04T09:27:17.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.977+0000 7f41e9c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f41e40af1a0 0x7f41e406e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:17.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.977+0000 7f41e9c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f41e40af580 0x7f41e4148230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:17.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.977+0000 7f41e9c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f41e40b34e0 0x7f41e41f48e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:17.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.977+0000 7f41e9c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41e4155e70 con 0x7f41e40af1a0 2024-03-04T09:27:17.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.977+0000 7f41e9c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41e40cde30 con 0x7f41e40b34e0 2024-03-04T09:27:17.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.977+0000 7f41e9c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41e4155ff0 con 0x7f41e40af580 2024-03-04T09:27:17.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.977+0000 7f41e2a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f41e40af1a0 0x7f41e406e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:17.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.977+0000 7f41e2a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f41e40af1a0 0x7f41e406e4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38210/0 (socket says 172.21.15.6:38210) 2024-03-04T09:27:17.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.977+0000 7f41e2a00640 1 -- 172.21.15.6:0/3805750946 learned_addr learned my addr 172.21.15.6:0/3805750946 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:27:17.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.977+0000 7f41e3400640 1 --2- 172.21.15.6:0/3805750946 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f41e40b34e0 0x7f41e41f48e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:17.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.977+0000 7f41e2000640 1 --2- 172.21.15.6:0/3805750946 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f41e40af580 0x7f41e4148230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:17.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.977+0000 7f41e2a00640 1 -- 172.21.15.6:0/3805750946 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f41e40af580 msgr2=0x7f41e4148230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:17.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.977+0000 7f41e2a00640 1 --2- 172.21.15.6:0/3805750946 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f41e40af580 0x7f41e4148230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:17.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.977+0000 7f41e2a00640 1 -- 172.21.15.6:0/3805750946 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f41e40b34e0 msgr2=0x7f41e41f48e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:17.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.977+0000 7f41e2a00640 1 --2- 172.21.15.6:0/3805750946 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f41e40b34e0 0x7f41e41f48e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:17.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.977+0000 7f41e2a00640 1 -- 172.21.15.6:0/3805750946 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41e41f4150 con 0x7f41e40af1a0 2024-03-04T09:27:17.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.978+0000 7f41e2000640 1 --2- 172.21.15.6:0/3805750946 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f41e40af580 0x7f41e4148230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:27:17.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.978+0000 7f41e3400640 1 --2- 172.21.15.6:0/3805750946 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f41e40b34e0 0x7f41e41f48e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:27:17.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.978+0000 7f41e2a00640 1 --2- 172.21.15.6:0/3805750946 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f41e40af1a0 0x7f41e406e4e0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f41d400c430 tx=0x7f41d400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:17.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.978+0000 7f41daa00640 1 -- 172.21.15.6:0/3805750946 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41d4017070 con 0x7f41e40af1a0 2024-03-04T09:27:17.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.978+0000 7f41daa00640 1 -- 172.21.15.6:0/3805750946 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41d4005ce0 con 0x7f41e40af1a0 2024-03-04T09:27:17.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.978+0000 7f41daa00640 1 -- 172.21.15.6:0/3805750946 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41d40072f0 con 0x7f41e40af1a0 2024-03-04T09:27:17.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.978+0000 7f41e9c00640 1 -- 172.21.15.6:0/3805750946 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41e415e880 con 0x7f41e40af1a0 2024-03-04T09:27:17.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.978+0000 7f41e9c00640 1 -- 172.21.15.6:0/3805750946 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41e415ee40 con 0x7f41e40af1a0 2024-03-04T09:27:17.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.979+0000 7f41e9c00640 1 -- 172.21.15.6:0/3805750946 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41e4069510 con 0x7f41e40af1a0 2024-03-04T09:27:17.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.980+0000 7f41daa00640 1 -- 172.21.15.6:0/3805750946 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f41d401a070 con 0x7f41e40af1a0 2024-03-04T09:27:17.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.982+0000 7f41daa00640 1 --2- 172.21.15.6:0/3805750946 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f41c406b500 0x7f41c406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:17.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.982+0000 7f41daa00640 1 -- 172.21.15.6:0/3805750946 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f41d4089670 con 0x7f41e40af1a0 2024-03-04T09:27:17.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.982+0000 7f41e2000640 1 --2- 172.21.15.6:0/3805750946 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f41c406b500 0x7f41c406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:17.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.983+0000 7f41e2000640 1 --2- 172.21.15.6:0/3805750946 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f41c406b500 0x7f41c406d9c0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f41e414a570 tx=0x7f41d0009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:17.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:17.986+0000 7f41daa00640 1 -- 172.21.15.6:0/3805750946 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f41d4058450 con 0x7f41e40af1a0 2024-03-04T09:27:18.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:18.184+0000 7f41e9c00640 1 -- 172.21.15.6:0/3805750946 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f41e404f3a0 con 0x7f41e40af1a0 2024-03-04T09:27:18.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:18.187+0000 7f41daa00640 1 -- 172.21.15.6:0/3805750946 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v43) v1 ==== 74+0+16578 (secure 0 0 0) 0x7f41d4057d50 con 0x7f41e40af1a0 2024-03-04T09:27:18.184 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:27:18.185 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":43,"fsid":"edce0578-5ae4-4402-b45a-552bdef84d7c","created":"2024-03-04T09:24:54.523780+0000","modified":"2024-03-04T09:27:16.784497+0000","last_up_change":"2024-03-04T09:25:07.089049+0000","last_in_change":"2024-03-04T09:24:57.982077+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T09:25:07.935198+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-03-04T09:27:14.657490+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"43","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-03-04T09:27:15.806886+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"43","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a96574f-81f1-4bc6-8463-757520b84a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6825","nonce":1364449401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6827","nonce":1364449401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6831","nonce":1364449401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6829","nonce":1364449401}]},"public_addr":"172.21.15.6:6825/1364449401","cluster_addr":"172.21.15.6:6827/1364449401","heartbeat_back_addr":"172.21.15.6:6831/1364449401","heartbeat_front_addr":"172.21.15.6:6829/1364449401","state":["exists","up"]},{"osd":1,"uuid":"7d79b9c1-6b87-476a-bc36-7db5ee64cd90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6809","nonce":2494103078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6811","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6813","nonce":2494103078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6821","nonce":2494103078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6817","nonce":2494103078}]},"public_addr":"172.21.15.6:6809/2494103078","cluster_addr":"172.21.15.6:6813/2494103078","heartbeat_back_addr":"172.21.15.6:6821/2494103078","heartbeat_front_addr":"172.21.15.6:6817/2494103078","state":["exists","up"]},{"osd":2,"uuid":"492331b9-4ee9-40b6-bacd-85d44cd6c45a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6812","nonce":1500066681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6816","nonce":1500066681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6824","nonce":1500066681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6820","nonce":1500066681}]},"public_addr":"172.21.15.6:6812/1500066681","cluster_addr":"172.21.15.6:6816/1500066681","heartbeat_back_addr":"172.21.15.6:6824/1500066681","heartbeat_front_addr":"172.21.15.6:6820/1500066681","state":["exists","up"]},{"osd":3,"uuid":"21bb2efe-9ba7-4bdc-bc31-180cd1102e21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6801","nonce":1558806123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6803","nonce":1558806123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6807","nonce":1558806123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6805","nonce":1558806123}]},"public_addr":"172.21.15.6:6801/1558806123","cluster_addr":"172.21.15.6:6803/1558806123","heartbeat_back_addr":"172.21.15.6:6807/1558806123","heartbeat_front_addr":"172.21.15.6:6805/1558806123","state":["exists","up"]},{"osd":4,"uuid":"6e2c4b71-19d9-4737-83d9-558705d8644a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6801","nonce":1526135780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6803","nonce":1526135780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6807","nonce":1526135780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6805","nonce":1526135780}]},"public_addr":"172.21.15.146:6801/1526135780","cluster_addr":"172.21.15.146:6803/1526135780","heartbeat_back_addr":"172.21.15.146:6807/1526135780","heartbeat_front_addr":"172.21.15.146:6805/1526135780","state":["exists","up"]},{"osd":5,"uuid":"fb69c525-f1be-4b7d-ab32-6c29b0d6fddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6809","nonce":460262507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6811","nonce":460262507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6815","nonce":460262507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6813","nonce":460262507}]},"public_addr":"172.21.15.146:6809/460262507","cluster_addr":"172.21.15.146:6811/460262507","heartbeat_back_addr":"172.21.15.146:6815/460262507","heartbeat_front_addr":"172.21.15.146:6813/460262507","state":["exists","up"]},{"osd":6,"uuid":"c32e7fed-370d-4401-8ce0-6a4335b0d168","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6823","nonce":1392897493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6827","nonce":1392897493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6831","nonce":1392897493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6829","nonce":1392897493}]},"public_addr":"172.21.15.146:6823/1392897493","cluster_addr":"172.21.15.146:6827/1392897493","heartbeat_back_addr":"172.21.15.146:6831/1392897493","heartbeat_front_addr":"172.21.15.146:6829/1392897493","state":["exists","up"]},{"osd":7,"uuid":"75a79b44-fa5f-426d-a7d3-71f21f78835c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6817","nonce":3268323156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6819","nonce":3268323156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6825","nonce":3268323156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6822","nonce":3268323156}]},"public_addr":"172.21.15.146:6817/3268323156","cluster_addr":"172.21.15.146:6819/3268323156","heartbeat_back_addr":"172.21.15.146:6825/3268323156","heartbeat_front_addr":"172.21.15.146:6822/3268323156","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.310437+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.295850+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.294674+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.526855+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.300023+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.595160+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.346046+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.558741+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T09:27:18.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:18.190+0000 7f41e9c00640 1 -- 172.21.15.6:0/3805750946 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f41c406b500 msgr2=0x7f41c406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:18.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:18.190+0000 7f41e9c00640 1 --2- 172.21.15.6:0/3805750946 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f41c406b500 0x7f41c406d9c0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f41e414a570 tx=0x7f41d0009040 comp rx=0 tx=0).stop 2024-03-04T09:27:18.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:18.190+0000 7f41e9c00640 1 -- 172.21.15.6:0/3805750946 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f41e40af1a0 msgr2=0x7f41e406e4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:18.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:18.190+0000 7f41e9c00640 1 --2- 172.21.15.6:0/3805750946 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f41e40af1a0 0x7f41e406e4e0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f41d400c430 tx=0x7f41d400c8f0 comp rx=0 tx=0).stop 2024-03-04T09:27:18.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:18.190+0000 7f41e9c00640 1 -- 172.21.15.6:0/3805750946 shutdown_connections 2024-03-04T09:27:18.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:18.190+0000 7f41e9c00640 1 --2- 172.21.15.6:0/3805750946 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f41e40b34e0 0x7f41e41f48e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:18.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:18.190+0000 7f41e9c00640 1 --2- 172.21.15.6:0/3805750946 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f41c406b500 0x7f41c406d9c0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:18.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:18.190+0000 7f41e9c00640 1 --2- 172.21.15.6:0/3805750946 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f41e40af580 0x7f41e4148230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:18.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:18.190+0000 7f41e9c00640 1 --2- 172.21.15.6:0/3805750946 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f41e40af1a0 0x7f41e406e4e0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:18.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:18.190+0000 7f41e9c00640 1 -- 172.21.15.6:0/3805750946 >> 172.21.15.6:0/3805750946 conn(0x7f41e406afb0 msgr2=0x7f41e40ae7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:18.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:18.190+0000 7f41e9c00640 1 -- 172.21.15.6:0/3805750946 shutdown_connections 2024-03-04T09:27:18.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:18.190+0000 7f41e9c00640 1 -- 172.21.15.6:0/3805750946 wait complete. 2024-03-04T09:27:18.206 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-04T09:27:18.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:18.428+0000 7f1ea0e00640 1 Processor -- start 2024-03-04T09:27:18.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:18.428+0000 7f1ea0e00640 1 -- start start 2024-03-04T09:27:18.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:18.430+0000 7f1ea0e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1e9c06f9c0 0x7f1e9c06fda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:18.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:18.430+0000 7f1ea0e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f1e9c06edc0 0x7f1e9c15f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:18.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:18.431+0000 7f1ea0e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f1e9c0d0c20 0x7f1e9c15fab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:18.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:18.431+0000 7f1ea0e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e9c0cff10 con 0x7f1e9c06edc0 2024-03-04T09:27:18.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:18.431+0000 7f1ea0e00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e9c15f700 con 0x7f1e9c0d0c20 2024-03-04T09:27:18.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:18.431+0000 7f1ea0e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e9c06f5a0 con 0x7f1e9c06f9c0 2024-03-04T09:27:18.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:18.431+0000 7f1e99400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f1e9c06edc0 0x7f1e9c15f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:18.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:18.431+0000 7f1e99400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f1e9c06edc0 0x7f1e9c15f1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38226/0 (socket says 172.21.15.6:38226) 2024-03-04T09:27:18.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:18.431+0000 7f1e99400640 1 -- 172.21.15.6:0/242298044 learned_addr learned my addr 172.21.15.6:0/242298044 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:27:18.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:18.432+0000 7f1e9a800640 1 --2- 172.21.15.6:0/242298044 >> v2:172.21.15.146:3300/0 conn(0x7f1e9c0d0c20 0x7f1e9c15fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:18.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:18.432+0000 7f1e99e00640 1 --2- 172.21.15.6:0/242298044 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1e9c06f9c0 0x7f1e9c06fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:18.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:18.432+0000 7f1e99400640 1 -- 172.21.15.6:0/242298044 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1e9c06f9c0 msgr2=0x7f1e9c06fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:27:18.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:18.432+0000 7f1e99400640 1 --2- 172.21.15.6:0/242298044 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1e9c06f9c0 0x7f1e9c06fda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:18.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:18.432+0000 7f1e99400640 1 -- 172.21.15.6:0/242298044 >> v2:172.21.15.146:3300/0 conn(0x7f1e9c0d0c20 msgr2=0x7f1e9c15fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:27:18.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:18.432+0000 7f1e99400640 1 --2- 172.21.15.6:0/242298044 >> v2:172.21.15.146:3300/0 conn(0x7f1e9c0d0c20 0x7f1e9c15fab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:18.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:18.432+0000 7f1e99400640 1 -- 172.21.15.6:0/242298044 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e9c0b1530 con 0x7f1e9c06edc0 2024-03-04T09:27:18.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:18.433+0000 7f1e99400640 1 --2- 172.21.15.6:0/242298044 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1e9c06edc0 0x7f1e9c15f1c0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f1e9c0719a0 tx=0x7f1e8802b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=638cbcebcb5cccee server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:18.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:18.433+0000 7f1e98a00640 1 -- 172.21.15.6:0/242298044 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e88035020 con 0x7f1e9c06edc0 2024-03-04T09:27:18.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:18.433+0000 7f1e98a00640 1 -- 172.21.15.6:0/242298044 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e8802d040 con 0x7f1e9c06edc0 2024-03-04T09:27:18.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:18.434+0000 7f1e98a00640 1 -- 172.21.15.6:0/242298044 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e88031a80 con 0x7f1e9c06edc0 2024-03-04T09:27:18.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:18.434+0000 7f1ea0e00640 1 -- 172.21.15.6:0/242298044 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1e9c06edc0 msgr2=0x7f1e9c15f1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:18.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:18.434+0000 7f1ea0e00640 1 --2- 172.21.15.6:0/242298044 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1e9c06edc0 0x7f1e9c15f1c0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f1e9c0719a0 tx=0x7f1e8802b900 comp rx=0 tx=0).stop 2024-03-04T09:27:18.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:18.434+0000 7f1ea0e00640 1 -- 172.21.15.6:0/242298044 shutdown_connections 2024-03-04T09:27:18.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:18.434+0000 7f1ea0e00640 1 --2- 172.21.15.6:0/242298044 >> v2:172.21.15.146:3300/0 conn(0x7f1e9c0d0c20 0x7f1e9c15fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:18.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:18.434+0000 7f1ea0e00640 1 --2- 172.21.15.6:0/242298044 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1e9c06edc0 0x7f1e9c15f1c0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:18.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:18.434+0000 7f1ea0e00640 1 --2- 172.21.15.6:0/242298044 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1e9c06f9c0 0x7f1e9c06fda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:18.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:18.434+0000 7f1ea0e00640 1 -- 172.21.15.6:0/242298044 >> 172.21.15.6:0/242298044 conn(0x7f1e9c0b3820 msgr2=0x7f1e9c0b3c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:18.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:18.434+0000 7f1ea0e00640 1 -- 172.21.15.6:0/242298044 shutdown_connections 2024-03-04T09:27:18.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:18.434+0000 7f1ea0e00640 1 -- 172.21.15.6:0/242298044 wait complete. 2024-03-04T09:27:18.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:18.436+0000 7f1ea0e00640 1 Processor -- start 2024-03-04T09:27:18.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:18.436+0000 7f1ea0e00640 1 -- start start 2024-03-04T09:27:18.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:18.437+0000 7f1ea0e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1e9c06edc0 0x7f1e9c152270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:18.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:18.437+0000 7f1ea0e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1e9c06f9c0 0x7f1e9c0afd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:18.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:18.437+0000 7f1ea0e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1e9c0d0c20 0x7f1e9c0c7980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:18.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:18.437+0000 7f1ea0e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e9c0d0140 con 0x7f1e9c0d0c20 2024-03-04T09:27:18.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:18.437+0000 7f1ea0e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e9c0d0790 con 0x7f1e9c06f9c0 2024-03-04T09:27:18.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:18.437+0000 7f1ea0e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e9c0d0420 con 0x7f1e9c06edc0 2024-03-04T09:27:18.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:18.438+0000 7f1e9a800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1e9c0d0c20 0x7f1e9c0c7980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:18.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:18.438+0000 7f1e9a800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1e9c0d0c20 0x7f1e9c0c7980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38240/0 (socket says 172.21.15.6:38240) 2024-03-04T09:27:18.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:18.438+0000 7f1e9a800640 1 -- 172.21.15.6:0/2346417298 learned_addr learned my addr 172.21.15.6:0/2346417298 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:27:18.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:18.438+0000 7f1e99e00640 1 --2- 172.21.15.6:0/2346417298 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1e9c06edc0 0x7f1e9c152270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:18.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:18.438+0000 7f1e99400640 1 --2- 172.21.15.6:0/2346417298 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1e9c06f9c0 0x7f1e9c0afd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:18.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:18.438+0000 7f1e9a800640 1 -- 172.21.15.6:0/2346417298 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1e9c06edc0 msgr2=0x7f1e9c152270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:18.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:18.438+0000 7f1e9a800640 1 --2- 172.21.15.6:0/2346417298 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1e9c06edc0 0x7f1e9c152270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:18.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:18.438+0000 7f1e9a800640 1 -- 172.21.15.6:0/2346417298 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1e9c06f9c0 msgr2=0x7f1e9c0afd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:18.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:18.438+0000 7f1e9a800640 1 --2- 172.21.15.6:0/2346417298 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1e9c06f9c0 0x7f1e9c0afd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:18.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:18.438+0000 7f1e9a800640 1 -- 172.21.15.6:0/2346417298 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e9c0cc300 con 0x7f1e9c0d0c20 2024-03-04T09:27:18.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:18.439+0000 7f1e9a800640 1 --2- 172.21.15.6:0/2346417298 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1e9c0d0c20 0x7f1e9c0c7980 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f1e9400cc60 tx=0x7f1e9400a5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:18.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:18.439+0000 7f1e92000640 1 -- 172.21.15.6:0/2346417298 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e94003990 con 0x7f1e9c0d0c20 2024-03-04T09:27:18.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:18.439+0000 7f1e92000640 1 -- 172.21.15.6:0/2346417298 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e94003b30 con 0x7f1e9c0d0c20 2024-03-04T09:27:18.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:18.439+0000 7f1e92000640 1 -- 172.21.15.6:0/2346417298 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e94004020 con 0x7f1e9c0d0c20 2024-03-04T09:27:18.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:18.439+0000 7f1ea0e00640 1 -- 172.21.15.6:0/2346417298 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e9c0b5200 con 0x7f1e9c0d0c20 2024-03-04T09:27:18.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:18.439+0000 7f1ea0e00640 1 -- 172.21.15.6:0/2346417298 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e9c0b57c0 con 0x7f1e9c0d0c20 2024-03-04T09:27:18.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:18.440+0000 7f1ea0e00640 1 -- 172.21.15.6:0/2346417298 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e9c1577f0 con 0x7f1e9c0d0c20 2024-03-04T09:27:18.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:18.441+0000 7f1e92000640 1 -- 172.21.15.6:0/2346417298 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1e940041c0 con 0x7f1e9c0d0c20 2024-03-04T09:27:18.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:18.442+0000 7f1e92000640 1 --2- 172.21.15.6:0/2346417298 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f1e7c06b4d0 0x7f1e7c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:18.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:18.442+0000 7f1e92000640 1 -- 172.21.15.6:0/2346417298 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1e9408a730 con 0x7f1e9c0d0c20 2024-03-04T09:27:18.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:18.443+0000 7f1e99e00640 1 --2- 172.21.15.6:0/2346417298 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f1e7c06b4d0 0x7f1e7c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:18.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:18.444+0000 7f1e99e00640 1 --2- 172.21.15.6:0/2346417298 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f1e7c06b4d0 0x7f1e7c06d990 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f1e8c006fd0 tx=0x7f1e8c008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:18.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:18.447+0000 7f1e92000640 1 -- 172.21.15.6:0/2346417298 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1e9c1577f0 con 0x7f1e9c0d0c20 2024-03-04T09:27:18.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:18.693+0000 7f1ea0e00640 1 -- 172.21.15.6:0/2346417298 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f1e9c04f330 con 0x7f1e9c0d0c20 2024-03-04T09:27:19.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:19.320+0000 7f1e92000640 1 -- 172.21.15.6:0/2346417298 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v24) v1 ==== 118+0+0 (secure 0 0 0) 0x7f1e94014030 con 0x7f1e9c0d0c20 2024-03-04T09:27:19.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:19.323+0000 7f1ea0e00640 1 -- 172.21.15.6:0/2346417298 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f1e7c06b4d0 msgr2=0x7f1e7c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:19.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:19.324+0000 7f1ea0e00640 1 --2- 172.21.15.6:0/2346417298 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f1e7c06b4d0 0x7f1e7c06d990 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f1e8c006fd0 tx=0x7f1e8c008040 comp rx=0 tx=0).stop 2024-03-04T09:27:19.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:19.324+0000 7f1ea0e00640 1 -- 172.21.15.6:0/2346417298 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1e9c0d0c20 msgr2=0x7f1e9c0c7980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:19.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:19.324+0000 7f1ea0e00640 1 --2- 172.21.15.6:0/2346417298 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1e9c0d0c20 0x7f1e9c0c7980 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f1e9400cc60 tx=0x7f1e9400a5c0 comp rx=0 tx=0).stop 2024-03-04T09:27:19.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:19.324+0000 7f1ea0e00640 1 -- 172.21.15.6:0/2346417298 shutdown_connections 2024-03-04T09:27:19.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:19.324+0000 7f1ea0e00640 1 --2- 172.21.15.6:0/2346417298 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1e9c0d0c20 0x7f1e9c0c7980 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:19.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:19.324+0000 7f1ea0e00640 1 --2- 172.21.15.6:0/2346417298 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1e9c06f9c0 0x7f1e9c0afd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:19.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:19.324+0000 7f1ea0e00640 1 --2- 172.21.15.6:0/2346417298 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f1e7c06b4d0 0x7f1e7c06d990 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:19.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:19.324+0000 7f1ea0e00640 1 --2- 172.21.15.6:0/2346417298 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1e9c06edc0 0x7f1e9c152270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:19.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:19.324+0000 7f1ea0e00640 1 -- 172.21.15.6:0/2346417298 >> 172.21.15.6:0/2346417298 conn(0x7f1e9c0b3820 msgr2=0x7f1e9c15f880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:19.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:19.324+0000 7f1ea0e00640 1 -- 172.21.15.6:0/2346417298 shutdown_connections 2024-03-04T09:27:19.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:19.324+0000 7f1ea0e00640 1 -- 172.21.15.6:0/2346417298 wait complete. 2024-03-04T09:27:19.342 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-04T09:27:19.342 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-04T09:27:19.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:19.562+0000 7fc3e7e00640 1 Processor -- start 2024-03-04T09:27:19.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:19.562+0000 7fc3e7e00640 1 -- start start 2024-03-04T09:27:19.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:19.564+0000 7fc3e7e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc3e8147590 0x7fc3e8071480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:19.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:19.564+0000 7fc3e7e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fc3e80b2250 0x7fc3e80b3640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:19.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:19.564+0000 7fc3e7e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3e80719c0 con 0x7fc3e806d8d0 2024-03-04T09:27:19.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:19.564+0000 7fc3e7e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3e8147970 con 0x7fc3e8147590 2024-03-04T09:27:19.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:19.564+0000 7fc3e7e00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3e80b31b0 con 0x7fc3e80b2250 2024-03-04T09:27:19.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:19.564+0000 7fc3e6a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc3e8147590 0x7fc3e8071480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:19.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:19.564+0000 7fc3e6a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc3e8147590 0x7fc3e8071480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:50404/0 (socket says 172.21.15.6:50404) 2024-03-04T09:27:19.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:19.564+0000 7fc3e6a00640 1 -- 172.21.15.6:0/1677793284 learned_addr learned my addr 172.21.15.6:0/1677793284 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:27:19.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:19.565+0000 7fc3e7400640 1 --2- 172.21.15.6:0/1677793284 >> v2:172.21.15.146:3300/0 conn(0x7fc3e80b2250 0x7fc3e80b3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:19.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:19.565+0000 7fc3e6a00640 1 -- 172.21.15.6:0/1677793284 >> v2:172.21.15.146:3300/0 conn(0x7fc3e80b2250 msgr2=0x7fc3e80b3640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:27:19.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:19.565+0000 7fc3e6a00640 1 --2- 172.21.15.6:0/1677793284 >> v2:172.21.15.146:3300/0 conn(0x7fc3e80b2250 0x7fc3e80b3640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:19.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:19.565+0000 7fc3e6a00640 1 -- 172.21.15.6:0/1677793284 >> v1:172.21.15.6:6789/0 conn(0x7fc3e806d8d0 legacy=0x7fc3e80b1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:19.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:19.565+0000 7fc3e6a00640 1 -- 172.21.15.6:0/1677793284 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3e806de40 con 0x7fc3e8147590 2024-03-04T09:27:19.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:19.565+0000 7fc3e7400640 1 --2- 172.21.15.6:0/1677793284 >> v2:172.21.15.146:3300/0 conn(0x7fc3e80b2250 0x7fc3e80b3640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T09:27:19.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:19.566+0000 7fc3e6a00640 1 --2- 172.21.15.6:0/1677793284 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc3e8147590 0x7fc3e8071480 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fc3e8161850 tx=0x7fc3d802b9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a19357a39a72b2b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:19.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:19.567+0000 7fc3e5600640 1 -- 172.21.15.6:0/1677793284 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3d8034070 con 0x7fc3e8147590 2024-03-04T09:27:19.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:19.567+0000 7fc3e5600640 1 -- 172.21.15.6:0/1677793284 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3d8030870 con 0x7fc3e8147590 2024-03-04T09:27:19.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:19.567+0000 7fc3e5600640 1 -- 172.21.15.6:0/1677793284 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3d8030c40 con 0x7fc3e8147590 2024-03-04T09:27:19.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:19.568+0000 7fc3e7e00640 1 -- 172.21.15.6:0/1677793284 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc3e8147590 msgr2=0x7fc3e8071480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:19.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:19.568+0000 7fc3e7e00640 1 --2- 172.21.15.6:0/1677793284 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc3e8147590 0x7fc3e8071480 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fc3e8161850 tx=0x7fc3d802b9b0 comp rx=0 tx=0).stop 2024-03-04T09:27:19.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:19.568+0000 7fc3e7e00640 1 -- 172.21.15.6:0/1677793284 shutdown_connections 2024-03-04T09:27:19.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:19.568+0000 7fc3e7e00640 1 --2- 172.21.15.6:0/1677793284 >> v2:172.21.15.146:3300/0 conn(0x7fc3e80b2250 0x7fc3e80b3640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:19.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:19.568+0000 7fc3e7e00640 1 --2- 172.21.15.6:0/1677793284 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc3e8147590 0x7fc3e8071480 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:19.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:19.568+0000 7fc3e7e00640 1 -- 172.21.15.6:0/1677793284 >> 172.21.15.6:0/1677793284 conn(0x7fc3e806b3a0 msgr2=0x7fc3e806b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:19.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:19.568+0000 7fc3e7e00640 1 -- 172.21.15.6:0/1677793284 shutdown_connections 2024-03-04T09:27:19.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:19.568+0000 7fc3e7e00640 1 -- 172.21.15.6:0/1677793284 wait complete. 2024-03-04T09:27:19.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:19.570+0000 7fc3e7e00640 1 Processor -- start 2024-03-04T09:27:19.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:19.571+0000 7fc3e7e00640 1 -- start start 2024-03-04T09:27:19.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:19.572+0000 7fc3e7e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc3e806d8d0 0x7fc3e815d610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:19.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:19.572+0000 7fc3e7e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc3e80b2250 0x7fc3e815e130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:19.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:19.572+0000 7fc3e7e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc3e8147590 0x7fc3e80c6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:19.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:19.572+0000 7fc3e7e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3e80b2b50 con 0x7fc3e806d8d0 2024-03-04T09:27:19.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:19.572+0000 7fc3e7e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3e80b2e40 con 0x7fc3e8147590 2024-03-04T09:27:19.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:19.572+0000 7fc3e7e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3e80b3b80 con 0x7fc3e80b2250 2024-03-04T09:27:19.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:19.572+0000 7fc3e6a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc3e806d8d0 0x7fc3e815d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:19.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:19.572+0000 7fc3e6a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc3e806d8d0 0x7fc3e815d610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:50420/0 (socket says 172.21.15.6:50420) 2024-03-04T09:27:19.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:19.572+0000 7fc3e6a00640 1 -- 172.21.15.6:0/292058640 learned_addr learned my addr 172.21.15.6:0/292058640 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:27:19.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:19.573+0000 7fc3e7400640 1 --2- 172.21.15.6:0/292058640 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc3e8147590 0x7fc3e80c6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:19.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:19.573+0000 7fc3e6a00640 1 -- 172.21.15.6:0/292058640 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc3e80b2250 msgr2=0x7fc3e815e130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:19.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:19.573+0000 7fc3e6a00640 1 --2- 172.21.15.6:0/292058640 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc3e80b2250 0x7fc3e815e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:19.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:19.573+0000 7fc3e6a00640 1 -- 172.21.15.6:0/292058640 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc3e8147590 msgr2=0x7fc3e80c6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:19.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:19.573+0000 7fc3e6a00640 1 --2- 172.21.15.6:0/292058640 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc3e8147590 0x7fc3e80c6250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:19.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:19.573+0000 7fc3e6a00640 1 -- 172.21.15.6:0/292058640 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3e815fbb0 con 0x7fc3e806d8d0 2024-03-04T09:27:19.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:19.573+0000 7fc3e6a00640 1 --2- 172.21.15.6:0/292058640 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc3e806d8d0 0x7fc3e815d610 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fc3d802b9f0 tx=0x7fc3d80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:19.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:19.573+0000 7fc3dea00640 1 -- 172.21.15.6:0/292058640 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3d8034040 con 0x7fc3e806d8d0 2024-03-04T09:27:19.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:19.573+0000 7fc3dea00640 1 -- 172.21.15.6:0/292058640 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3d8039d90 con 0x7fc3e806d8d0 2024-03-04T09:27:19.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:19.573+0000 7fc3dea00640 1 -- 172.21.15.6:0/292058640 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3d8043610 con 0x7fc3e806d8d0 2024-03-04T09:27:19.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:19.574+0000 7fc3e7e00640 1 -- 172.21.15.6:0/292058640 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3e815eb10 con 0x7fc3e806d8d0 2024-03-04T09:27:19.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:19.574+0000 7fc3e7e00640 1 -- 172.21.15.6:0/292058640 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3e815f0d0 con 0x7fc3e806d8d0 2024-03-04T09:27:19.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:19.575+0000 7fc3e7e00640 1 -- 172.21.15.6:0/292058640 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3e80b4b40 con 0x7fc3e806d8d0 2024-03-04T09:27:19.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:19.576+0000 7fc3dea00640 1 -- 172.21.15.6:0/292058640 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc3d8037070 con 0x7fc3e806d8d0 2024-03-04T09:27:19.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:19.577+0000 7fc3dea00640 1 --2- 172.21.15.6:0/292058640 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc3c806b500 0x7fc3c806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:19.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:19.577+0000 7fc3dea00640 1 -- 172.21.15.6:0/292058640 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc3d80ac3e0 con 0x7fc3e806d8d0 2024-03-04T09:27:19.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:19.578+0000 7fc3e6000640 1 --2- 172.21.15.6:0/292058640 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc3c806b500 0x7fc3c806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:19.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:19.579+0000 7fc3e6000640 1 --2- 172.21.15.6:0/292058640 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc3c806b500 0x7fc3c806d9c0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fc3d4000f60 tx=0x7fc3d40079d0 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:19.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:19.582+0000 7fc3dea00640 1 -- 172.21.15.6:0/292058640 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc3e80b4b40 con 0x7fc3e806d8d0 2024-03-04T09:27:19.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:19.823+0000 7fc3e7e00640 1 -- 172.21.15.6:0/292058640 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fc3e804f330 con 0x7fc3e806d8d0 2024-03-04T09:27:20.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:20.323+0000 7fc3dea00640 1 -- 172.21.15.6:0/292058640 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v25) v1 ==== 122+0+0 (secure 0 0 0) 0x7fc3d803e070 con 0x7fc3e806d8d0 2024-03-04T09:27:20.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:20.326+0000 7fc3e7e00640 1 -- 172.21.15.6:0/292058640 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc3c806b500 msgr2=0x7fc3c806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:20.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:20.327+0000 7fc3e7e00640 1 --2- 172.21.15.6:0/292058640 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc3c806b500 0x7fc3c806d9c0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fc3d4000f60 tx=0x7fc3d40079d0 comp rx=0 tx=0).stop 2024-03-04T09:27:20.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:20.327+0000 7fc3e7e00640 1 -- 172.21.15.6:0/292058640 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc3e806d8d0 msgr2=0x7fc3e815d610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:20.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:20.327+0000 7fc3e7e00640 1 --2- 172.21.15.6:0/292058640 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc3e806d8d0 0x7fc3e815d610 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fc3d802b9f0 tx=0x7fc3d80046c0 comp rx=0 tx=0).stop 2024-03-04T09:27:20.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:20.327+0000 7fc3e7e00640 1 -- 172.21.15.6:0/292058640 shutdown_connections 2024-03-04T09:27:20.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:20.327+0000 7fc3e7e00640 1 --2- 172.21.15.6:0/292058640 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc3e8147590 0x7fc3e80c6250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:20.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:20.327+0000 7fc3e7e00640 1 --2- 172.21.15.6:0/292058640 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc3c806b500 0x7fc3c806d9c0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:20.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:20.327+0000 7fc3e7e00640 1 --2- 172.21.15.6:0/292058640 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc3e80b2250 0x7fc3e815e130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:20.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:20.327+0000 7fc3e7e00640 1 --2- 172.21.15.6:0/292058640 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc3e806d8d0 0x7fc3e815d610 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:20.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:20.327+0000 7fc3e7e00640 1 -- 172.21.15.6:0/292058640 >> 172.21.15.6:0/292058640 conn(0x7fc3e806b3a0 msgr2=0x7fc3e806b780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:20.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:20.327+0000 7fc3e7e00640 1 -- 172.21.15.6:0/292058640 shutdown_connections 2024-03-04T09:27:20.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:20.327+0000 7fc3e7e00640 1 -- 172.21.15.6:0/292058640 wait complete. 2024-03-04T09:27:20.345 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-04T09:27:20.345 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:27:20.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:20.572+0000 7f63e5800640 1 Processor -- start 2024-03-04T09:27:20.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:20.573+0000 7f63e5800640 1 -- start start 2024-03-04T09:27:20.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:20.574+0000 7f63e5800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f63e015efc0 0x7f63e00cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:20.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:20.574+0000 7f63e5800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f63e01621d0 0x7f63e0070c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:20.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:20.574+0000 7f63e5800640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63e01627f0 con 0x7f63e00cce90 2024-03-04T09:27:20.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:20.574+0000 7f63e5800640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63e015f3a0 con 0x7f63e01621d0 2024-03-04T09:27:20.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:20.574+0000 7f63e5800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006d190 con 0x7f63e015efc0 2024-03-04T09:27:20.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:20.575+0000 7f63d7e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f63e00cce90 0x7f63e0161ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:45734/0 (socket says 172.21.15.6:45734) 2024-03-04T09:27:20.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:20.575+0000 7f63d7e00640 1 -- 172.21.15.6:0/2738884622 learned_addr learned my addr 172.21.15.6:0/2738884622 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:27:20.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:20.575+0000 7f63e4e00640 1 --2- 172.21.15.6:0/2738884622 >> v2:172.21.15.146:3300/0 conn(0x7f63e01621d0 0x7f63e0070c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:20.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:20.575+0000 7f63dfe00640 1 --2- 172.21.15.6:0/2738884622 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f63e015efc0 0x7f63e00cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:20.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:20.575+0000 7f63df400640 1 -- 172.21.15.6:0/2738884622 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 498573929 0 0) 0x7f63e01627f0 con 0x7f63e00cce90 2024-03-04T09:27:20.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:20.575+0000 7f63df400640 1 -- 172.21.15.6:0/2738884622 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63c4003520 con 0x7f63e00cce90 2024-03-04T09:27:20.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:20.576+0000 7f63df400640 1 -- 172.21.15.6:0/2738884622 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2358182046 0 0) 0x7f63c4003520 con 0x7f63e00cce90 2024-03-04T09:27:20.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:20.576+0000 7f63df400640 1 -- 172.21.15.6:0/2738884622 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f63e015efc0 msgr2=0x7f63e00cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:27:20.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:20.576+0000 7f63df400640 1 --2- 172.21.15.6:0/2738884622 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f63e015efc0 0x7f63e00cc950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:20.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:20.576+0000 7f63df400640 1 -- 172.21.15.6:0/2738884622 >> v2:172.21.15.146:3300/0 conn(0x7f63e01621d0 msgr2=0x7f63e0070c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:27:20.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:20.576+0000 7f63df400640 1 --2- 172.21.15.6:0/2738884622 >> v2:172.21.15.146:3300/0 conn(0x7f63e01621d0 0x7f63e0070c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:20.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:20.576+0000 7f63df400640 1 -- 172.21.15.6:0/2738884622 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63e006c860 con 0x7f63e00cce90 2024-03-04T09:27:20.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:20.576+0000 7f63df400640 1 -- 172.21.15.6:0/2738884622 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7f63c8001670 con 0x7f63e00cce90 2024-03-04T09:27:20.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:20.576+0000 7f63df400640 1 -- 172.21.15.6:0/2738884622 >> v1:172.21.15.6:6789/0 conn(0x7f63e00cce90 legacy=0x7f63e0161ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:20.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:20.576+0000 7f63df400640 1 --2- 172.21.15.6:0/2738884622 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f63e015efc0 0x7f63c40039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:20.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:20.576+0000 7f63df400640 1 --2- 172.21.15.6:0/2738884622 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f63c4003f30 0x7f63c4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:20.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:20.576+0000 7f63df400640 1 --2- 172.21.15.6:0/2738884622 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f63c4004d40 0x7f63c40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:20.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:20.576+0000 7f63df400640 1 -- 172.21.15.6:0/2738884622 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e015f3a0 con 0x7f63c4003f30 2024-03-04T09:27:20.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:20.576+0000 7f63df400640 1 -- 172.21.15.6:0/2738884622 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006d190 con 0x7f63c4004d40 2024-03-04T09:27:20.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:20.576+0000 7f63df400640 1 -- 172.21.15.6:0/2738884622 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63c400b9c0 con 0x7f63e015efc0 2024-03-04T09:27:20.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:20.576+0000 7f63d7e00640 1 --2- 172.21.15.6:0/2738884622 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f63c4003f30 0x7f63c4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:20.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:20.576+0000 7f63e4e00640 1 --2- 172.21.15.6:0/2738884622 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f63c4004d40 0x7f63c40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:20.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:20.577+0000 7f63dfe00640 1 --2- 172.21.15.6:0/2738884622 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f63e015efc0 0x7f63c40039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:20.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:20.577+0000 7f63d7e00640 1 -- 172.21.15.6:0/2738884622 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f63e015efc0 msgr2=0x7f63c40039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:27:20.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:20.577+0000 7f63d7e00640 1 --2- 172.21.15.6:0/2738884622 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f63e015efc0 0x7f63c40039b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:20.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:20.577+0000 7f63d7e00640 1 -- 172.21.15.6:0/2738884622 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f63c4004d40 msgr2=0x7f63c40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:27:20.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:20.577+0000 7f63d7e00640 1 --2- 172.21.15.6:0/2738884622 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f63c4004d40 0x7f63c40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:20.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:20.577+0000 7f63d7e00640 1 -- 172.21.15.6:0/2738884622 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63c4003520 con 0x7f63c4003f30 2024-03-04T09:27:20.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:20.578+0000 7f63d7e00640 1 --2- 172.21.15.6:0/2738884622 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f63c4003f30 0x7f63c4004800 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f63c800c210 tx=0x7f63c4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b21c679005376383 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:20.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:20.578+0000 7f63df400640 1 -- 172.21.15.6:0/2738884622 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63c8007d50 con 0x7f63c4003f30 2024-03-04T09:27:20.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:20.578+0000 7f63df400640 1 -- 172.21.15.6:0/2738884622 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63c8030070 con 0x7f63c4003f30 2024-03-04T09:27:20.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:20.578+0000 7f63df400640 1 -- 172.21.15.6:0/2738884622 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63c802cdb0 con 0x7f63c4003f30 2024-03-04T09:27:20.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:20.579+0000 7f63e5800640 1 -- 172.21.15.6:0/2738884622 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f63c4003f30 msgr2=0x7f63c4004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:20.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:20.579+0000 7f63e5800640 1 --2- 172.21.15.6:0/2738884622 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f63c4003f30 0x7f63c4004800 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f63c800c210 tx=0x7f63c4005830 comp rx=0 tx=0).stop 2024-03-04T09:27:20.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:20.579+0000 7f63e4e00640 1 -- 172.21.15.6:0/2738884622 reap_dead start 2024-03-04T09:27:20.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:20.579+0000 7f63e5800640 1 -- 172.21.15.6:0/2738884622 shutdown_connections 2024-03-04T09:27:20.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:20.579+0000 7f63e5800640 1 -- 172.21.15.6:0/2738884622 >> 172.21.15.6:0/2738884622 conn(0x7f63e006b3d0 msgr2=0x7f63e006b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:20.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:20.579+0000 7f63e5800640 1 -- 172.21.15.6:0/2738884622 shutdown_connections 2024-03-04T09:27:20.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:20.579+0000 7f63e5800640 1 -- 172.21.15.6:0/2738884622 wait complete. 2024-03-04T09:27:20.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:20.581+0000 7f63e5800640 1 Processor -- start 2024-03-04T09:27:20.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:20.581+0000 7f63e5800640 1 -- start start 2024-03-04T09:27:20.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:20.582+0000 7f63e5800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f63c4003f30 0x7f63e01f3ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:20.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:20.582+0000 7f63e5800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f63e015dc00 0x7f63e01e83e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:20.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:20.582+0000 7f63e5800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f63e0160a80 0x7f63e0160f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:20.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:20.582+0000 7f63e5800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006d620 con 0x7f63e0160a80 2024-03-04T09:27:20.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:20.582+0000 7f63e5800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006ce20 con 0x7f63e015dc00 2024-03-04T09:27:20.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:20.582+0000 7f63e5800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e01e7830 con 0x7f63c4003f30 2024-03-04T09:27:20.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:20.582+0000 7f63e4e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f63e0160a80 0x7f63e0160f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:20.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:20.582+0000 7f63e4e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f63e0160a80 0x7f63e0160f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:50436/0 (socket says 172.21.15.6:50436) 2024-03-04T09:27:20.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:20.582+0000 7f63e4e00640 1 -- 172.21.15.6:0/2039407781 learned_addr learned my addr 172.21.15.6:0/2039407781 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:27:20.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:20.582+0000 7f63d7e00640 1 --2- 172.21.15.6:0/2039407781 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f63e015dc00 0x7f63e01e83e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:20.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:20.582+0000 7f63dfe00640 1 --2- 172.21.15.6:0/2039407781 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f63c4003f30 0x7f63e01f3ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:20.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:20.582+0000 7f63e4e00640 1 -- 172.21.15.6:0/2039407781 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f63c4003f30 msgr2=0x7f63e01f3ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:20.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:20.582+0000 7f63e4e00640 1 --2- 172.21.15.6:0/2039407781 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f63c4003f30 0x7f63e01f3ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:20.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:20.582+0000 7f63e4e00640 1 -- 172.21.15.6:0/2039407781 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f63e015dc00 msgr2=0x7f63e01e83e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:20.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:20.582+0000 7f63e4e00640 1 --2- 172.21.15.6:0/2039407781 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f63e015dc00 0x7f63e01e83e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:20.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:20.582+0000 7f63e4e00640 1 -- 172.21.15.6:0/2039407781 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63e0144ed0 con 0x7f63e0160a80 2024-03-04T09:27:20.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:20.583+0000 7f63e4e00640 1 --2- 172.21.15.6:0/2039407781 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f63e0160a80 0x7f63e0160f30 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f63e00cf9e0 tx=0x7f63d80115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:20.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:20.583+0000 7f63dcc00640 1 -- 172.21.15.6:0/2039407781 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63d8011db0 con 0x7f63e0160a80 2024-03-04T09:27:20.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:20.583+0000 7f63dcc00640 1 -- 172.21.15.6:0/2039407781 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63d801b070 con 0x7f63e0160a80 2024-03-04T09:27:20.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:20.583+0000 7f63dcc00640 1 -- 172.21.15.6:0/2039407781 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63d8003990 con 0x7f63e0160a80 2024-03-04T09:27:20.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:20.583+0000 7f63e5800640 1 -- 172.21.15.6:0/2039407781 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63e01e8920 con 0x7f63e0160a80 2024-03-04T09:27:20.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:20.583+0000 7f63e5800640 1 -- 172.21.15.6:0/2039407781 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63e01f45d0 con 0x7f63e0160a80 2024-03-04T09:27:20.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:20.584+0000 7f63e5800640 1 -- 172.21.15.6:0/2039407781 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63e004f330 con 0x7f63e0160a80 2024-03-04T09:27:20.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:20.585+0000 7f63dcc00640 1 -- 172.21.15.6:0/2039407781 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f63d8005d10 con 0x7f63e0160a80 2024-03-04T09:27:20.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:20.586+0000 7f63dcc00640 1 --2- 172.21.15.6:0/2039407781 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f63bc06b500 0x7f63bc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:20.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:20.586+0000 7f63dcc00640 1 -- 172.21.15.6:0/2039407781 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f63d808a870 con 0x7f63e0160a80 2024-03-04T09:27:20.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:20.587+0000 7f63dfe00640 1 --2- 172.21.15.6:0/2039407781 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f63bc06b500 0x7f63bc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:20.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:20.588+0000 7f63dfe00640 1 --2- 172.21.15.6:0/2039407781 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f63bc06b500 0x7f63bc06d9c0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f63e0146250 tx=0x7f63d0008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:20.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:20.594+0000 7f63dcc00640 1 -- 172.21.15.6:0/2039407781 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f63d8025020 con 0x7f63e0160a80 2024-03-04T09:27:20.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:20.836+0000 7f63e5800640 1 -- 172.21.15.6:0/2039407781 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f63e00af2f0 con 0x7f63e0160a80 2024-03-04T09:27:20.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:20.837+0000 7f63dcc00640 1 -- 172.21.15.6:0/2039407781 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+4974 (secure 0 0 0) 0x7f63e0144ed0 con 0x7f63e0160a80 2024-03-04T09:27:20.834 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:27:20.835 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4913,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/1049631914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":1049631914},{"type":"v1","addr":"172.21.15.146:6835","nonce":1049631914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":5092,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/424062551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":424062551},{"type":"v1","addr":"172.21.15.6:6833","nonce":424062551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":19},{"gid":5095,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/3300873502","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":3300873502},{"type":"v1","addr":"172.21.15.6:6835","nonce":3300873502}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:27:16.785309+0000","modified":"2024-03-04T09:27:19.824667+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5110},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5110":{"gid":5110,"name":"b","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.146:6837/3982958749","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":3982958749},{"type":"v1","addr":"172.21.15.146:6837","nonce":3982958749}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5110,"qdb_cluster":[5110]},"id":3}]} 2024-03-04T09:27:20.835 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 25 2024-03-04T09:27:20.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:20.840+0000 7f63e5800640 1 -- 172.21.15.6:0/2039407781 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f63bc06b500 msgr2=0x7f63bc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:20.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:20.840+0000 7f63e5800640 1 --2- 172.21.15.6:0/2039407781 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f63bc06b500 0x7f63bc06d9c0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f63e0146250 tx=0x7f63d0008040 comp rx=0 tx=0).stop 2024-03-04T09:27:20.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:20.840+0000 7f63e5800640 1 -- 172.21.15.6:0/2039407781 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f63e0160a80 msgr2=0x7f63e0160f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:20.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:20.840+0000 7f63e5800640 1 --2- 172.21.15.6:0/2039407781 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f63e0160a80 0x7f63e0160f30 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f63e00cf9e0 tx=0x7f63d80115a0 comp rx=0 tx=0).stop 2024-03-04T09:27:20.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:20.840+0000 7f63e5800640 1 -- 172.21.15.6:0/2039407781 shutdown_connections 2024-03-04T09:27:20.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:20.840+0000 7f63e5800640 1 --2- 172.21.15.6:0/2039407781 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f63e0160a80 0x7f63e0160f30 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:20.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:20.840+0000 7f63e5800640 1 --2- 172.21.15.6:0/2039407781 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f63e015dc00 0x7f63e01e83e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:20.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:20.840+0000 7f63e5800640 1 --2- 172.21.15.6:0/2039407781 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f63bc06b500 0x7f63bc06d9c0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:20.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:20.840+0000 7f63e5800640 1 --2- 172.21.15.6:0/2039407781 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f63c4003f30 0x7f63e01f3ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:20.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:20.841+0000 7f63e5800640 1 -- 172.21.15.6:0/2039407781 >> 172.21.15.6:0/2039407781 conn(0x7f63e006b3d0 msgr2=0x7f63e00af6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:20.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:20.841+0000 7f63e5800640 1 -- 172.21.15.6:0/2039407781 shutdown_connections 2024-03-04T09:27:20.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:20.841+0000 7f63e5800640 1 -- 172.21.15.6:0/2039407781 wait complete. 2024-03-04T09:27:20.856 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T09:27:21.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.081+0000 7f0dc8c00640 1 Processor -- start 2024-03-04T09:27:21.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.081+0000 7f0dc8c00640 1 -- start start 2024-03-04T09:27:21.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.083+0000 7f0dc8c00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f0dc40cd060 0x7f0dc41667e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:21.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.083+0000 7f0dc8c00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dc40714d0 con 0x7f0dc4070f10 2024-03-04T09:27:21.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.083+0000 7f0dc8c00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dc40717b0 con 0x7f0dc4145300 2024-03-04T09:27:21.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.083+0000 7f0dc8c00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0dc4071a90 con 0x7f0dc40cd060 2024-03-04T09:27:21.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.084+0000 7f0dc2a00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f0dc4070f10 0x7f0dc40cc950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:45742/0 (socket says 172.21.15.6:45742) 2024-03-04T09:27:21.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.084+0000 7f0dc2a00640 1 -- 172.21.15.6:0/952050409 learned_addr learned my addr 172.21.15.6:0/952050409 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:27:21.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.084+0000 7f0dc3e00640 1 --2- 172.21.15.6:0/952050409 >> v2:172.21.15.146:3300/0 conn(0x7f0dc40cd060 0x7f0dc41667e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:21.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.084+0000 7f0dc2000640 1 -- 172.21.15.6:0/952050409 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3300813469 0 0) 0x7f0dc40714d0 con 0x7f0dc4070f10 2024-03-04T09:27:21.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.084+0000 7f0dc2000640 1 -- 172.21.15.6:0/952050409 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0da8003520 con 0x7f0dc4070f10 2024-03-04T09:27:21.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.084+0000 7f0dc3e00640 1 -- 172.21.15.6:0/952050409 >> v1:172.21.15.146:6789/0 conn(0x7f0dc4145300 legacy=0x7f0dc40b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:21.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.084+0000 7f0dc3e00640 1 -- 172.21.15.6:0/952050409 >> v1:172.21.15.6:6789/0 conn(0x7f0dc4070f10 legacy=0x7f0dc40cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:21.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.085+0000 7f0dc3e00640 1 -- 172.21.15.6:0/952050409 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dc4071d70 con 0x7f0dc40cd060 2024-03-04T09:27:21.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.086+0000 7f0dc3e00640 1 --2- 172.21.15.6:0/952050409 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0dc40cd060 0x7f0dc41667e0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f0dbc009d90 tx=0x7f0dbc02beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4f66d3bb92137ebe server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:21.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.086+0000 7f0dc2000640 1 -- 172.21.15.6:0/952050409 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dbc039070 con 0x7f0dc40cd060 2024-03-04T09:27:21.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.086+0000 7f0dc2000640 1 -- 172.21.15.6:0/952050409 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0dbc02cc60 con 0x7f0dc40cd060 2024-03-04T09:27:21.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.086+0000 7f0dc2000640 1 -- 172.21.15.6:0/952050409 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dbc034660 con 0x7f0dc40cd060 2024-03-04T09:27:21.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.087+0000 7f0dc8c00640 1 -- 172.21.15.6:0/952050409 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0dc40cd060 msgr2=0x7f0dc41667e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:21.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.087+0000 7f0dc8c00640 1 --2- 172.21.15.6:0/952050409 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0dc40cd060 0x7f0dc41667e0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f0dbc009d90 tx=0x7f0dbc02beb0 comp rx=0 tx=0).stop 2024-03-04T09:27:21.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.087+0000 7f0dc8c00640 1 -- 172.21.15.6:0/952050409 shutdown_connections 2024-03-04T09:27:21.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.087+0000 7f0dc8c00640 1 --2- 172.21.15.6:0/952050409 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0dc40cd060 0x7f0dc41667e0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:21.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.087+0000 7f0dc8c00640 1 -- 172.21.15.6:0/952050409 >> 172.21.15.6:0/952050409 conn(0x7f0dc406b3d0 msgr2=0x7f0dc406b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:21.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.087+0000 7f0dc8c00640 1 -- 172.21.15.6:0/952050409 shutdown_connections 2024-03-04T09:27:21.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.087+0000 7f0dc8c00640 1 -- 172.21.15.6:0/952050409 wait complete. 2024-03-04T09:27:21.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.089+0000 7f0dc8c00640 1 Processor -- start 2024-03-04T09:27:21.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.089+0000 7f0dc8c00640 1 -- start start 2024-03-04T09:27:21.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.090+0000 7f0dc8c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0dc4070f10 0x7f0dc4131550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:21.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.090+0000 7f0dc8c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0dc40cd060 0x7f0dc406d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:21.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.090+0000 7f0dc8c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0dc4145300 0x7f0dc415b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:21.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.090+0000 7f0dc8c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dc415d700 con 0x7f0dc4070f10 2024-03-04T09:27:21.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.090+0000 7f0dc8c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dc415d580 con 0x7f0dc4145300 2024-03-04T09:27:21.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.090+0000 7f0dc8c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0dc415d880 con 0x7f0dc40cd060 2024-03-04T09:27:21.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.090+0000 7f0dc3400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0dc4070f10 0x7f0dc4131550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:21.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.090+0000 7f0dc3400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0dc4070f10 0x7f0dc4131550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:50440/0 (socket says 172.21.15.6:50440) 2024-03-04T09:27:21.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.090+0000 7f0dc3400640 1 -- 172.21.15.6:0/760384710 learned_addr learned my addr 172.21.15.6:0/760384710 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:27:21.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.091+0000 7f0dc3e00640 1 --2- 172.21.15.6:0/760384710 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0dc4145300 0x7f0dc415b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:21.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.091+0000 7f0dc3400640 1 -- 172.21.15.6:0/760384710 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0dc40cd060 msgr2=0x7f0dc406d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:21.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.091+0000 7f0dc3400640 1 --2- 172.21.15.6:0/760384710 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0dc40cd060 0x7f0dc406d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:21.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.091+0000 7f0dc3400640 1 -- 172.21.15.6:0/760384710 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0dc4145300 msgr2=0x7f0dc415b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:21.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.091+0000 7f0dc3400640 1 --2- 172.21.15.6:0/760384710 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0dc4145300 0x7f0dc415b4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:21.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.091+0000 7f0dc3400640 1 -- 172.21.15.6:0/760384710 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dc415ba10 con 0x7f0dc4070f10 2024-03-04T09:27:21.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.091+0000 7f0dc3400640 1 --2- 172.21.15.6:0/760384710 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0dc4070f10 0x7f0dc4131550 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f0db400c3c0 tx=0x7f0db400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:21.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.091+0000 7f0dbb400640 1 -- 172.21.15.6:0/760384710 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0db40052d0 con 0x7f0dc4070f10 2024-03-04T09:27:21.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.091+0000 7f0dbb400640 1 -- 172.21.15.6:0/760384710 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0db4005490 con 0x7f0dc4070f10 2024-03-04T09:27:21.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.091+0000 7f0dbb400640 1 -- 172.21.15.6:0/760384710 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0db4007260 con 0x7f0dc4070f10 2024-03-04T09:27:21.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.091+0000 7f0dc8c00640 1 -- 172.21.15.6:0/760384710 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0dc41e6f40 con 0x7f0dc4070f10 2024-03-04T09:27:21.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.092+0000 7f0dc8c00640 1 -- 172.21.15.6:0/760384710 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0dc406cbd0 con 0x7f0dc4070f10 2024-03-04T09:27:21.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.093+0000 7f0dc8c00640 1 -- 172.21.15.6:0/760384710 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0dc404f330 con 0x7f0dc4070f10 2024-03-04T09:27:21.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.094+0000 7f0dbb400640 1 -- 172.21.15.6:0/760384710 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0db4007400 con 0x7f0dc4070f10 2024-03-04T09:27:21.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.094+0000 7f0dbb400640 1 --2- 172.21.15.6:0/760384710 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f0da406b4d0 0x7f0da406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:21.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.094+0000 7f0dbb400640 1 -- 172.21.15.6:0/760384710 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0db4089650 con 0x7f0dc4070f10 2024-03-04T09:27:21.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.095+0000 7f0dc2a00640 1 --2- 172.21.15.6:0/760384710 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f0da406b4d0 0x7f0da406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:21.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.096+0000 7f0dc2a00640 1 --2- 172.21.15.6:0/760384710 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f0da406b4d0 0x7f0da406d990 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f0dc4072320 tx=0x7f0db0007930 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:21.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.099+0000 7f0dbb400640 1 -- 172.21.15.6:0/760384710 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0dc415ba10 con 0x7f0dc4070f10 2024-03-04T09:27:21.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.299+0000 7f0dc8c00640 1 -- 172.21.15.6:0/760384710 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0dc40ccd20 con 0x7f0dc4070f10 2024-03-04T09:27:21.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.301+0000 7f0dbb400640 1 -- 172.21.15.6:0/760384710 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v44) v1 ==== 74+0+16565 (secure 0 0 0) 0x7f0db4058560 con 0x7f0dc4070f10 2024-03-04T09:27:21.298 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:27:21.298 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":44,"fsid":"edce0578-5ae4-4402-b45a-552bdef84d7c","created":"2024-03-04T09:24:54.523780+0000","modified":"2024-03-04T09:27:18.307032+0000","last_up_change":"2024-03-04T09:25:07.089049+0000","last_in_change":"2024-03-04T09:24:57.982077+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T09:25:07.935198+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-03-04T09:27:14.657490+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"43","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-03-04T09:27:15.806886+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a96574f-81f1-4bc6-8463-757520b84a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6825","nonce":1364449401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6827","nonce":1364449401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6831","nonce":1364449401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6829","nonce":1364449401}]},"public_addr":"172.21.15.6:6825/1364449401","cluster_addr":"172.21.15.6:6827/1364449401","heartbeat_back_addr":"172.21.15.6:6831/1364449401","heartbeat_front_addr":"172.21.15.6:6829/1364449401","state":["exists","up"]},{"osd":1,"uuid":"7d79b9c1-6b87-476a-bc36-7db5ee64cd90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6809","nonce":2494103078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6811","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6813","nonce":2494103078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6821","nonce":2494103078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6817","nonce":2494103078}]},"public_addr":"172.21.15.6:6809/2494103078","cluster_addr":"172.21.15.6:6813/2494103078","heartbeat_back_addr":"172.21.15.6:6821/2494103078","heartbeat_front_addr":"172.21.15.6:6817/2494103078","state":["exists","up"]},{"osd":2,"uuid":"492331b9-4ee9-40b6-bacd-85d44cd6c45a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6812","nonce":1500066681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6816","nonce":1500066681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6824","nonce":1500066681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6820","nonce":1500066681}]},"public_addr":"172.21.15.6:6812/1500066681","cluster_addr":"172.21.15.6:6816/1500066681","heartbeat_back_addr":"172.21.15.6:6824/1500066681","heartbeat_front_addr":"172.21.15.6:6820/1500066681","state":["exists","up"]},{"osd":3,"uuid":"21bb2efe-9ba7-4bdc-bc31-180cd1102e21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6801","nonce":1558806123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6803","nonce":1558806123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6807","nonce":1558806123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6805","nonce":1558806123}]},"public_addr":"172.21.15.6:6801/1558806123","cluster_addr":"172.21.15.6:6803/1558806123","heartbeat_back_addr":"172.21.15.6:6807/1558806123","heartbeat_front_addr":"172.21.15.6:6805/1558806123","state":["exists","up"]},{"osd":4,"uuid":"6e2c4b71-19d9-4737-83d9-558705d8644a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6801","nonce":1526135780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6803","nonce":1526135780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6807","nonce":1526135780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6805","nonce":1526135780}]},"public_addr":"172.21.15.146:6801/1526135780","cluster_addr":"172.21.15.146:6803/1526135780","heartbeat_back_addr":"172.21.15.146:6807/1526135780","heartbeat_front_addr":"172.21.15.146:6805/1526135780","state":["exists","up"]},{"osd":5,"uuid":"fb69c525-f1be-4b7d-ab32-6c29b0d6fddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6809","nonce":460262507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6811","nonce":460262507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6815","nonce":460262507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6813","nonce":460262507}]},"public_addr":"172.21.15.146:6809/460262507","cluster_addr":"172.21.15.146:6811/460262507","heartbeat_back_addr":"172.21.15.146:6815/460262507","heartbeat_front_addr":"172.21.15.146:6813/460262507","state":["exists","up"]},{"osd":6,"uuid":"c32e7fed-370d-4401-8ce0-6a4335b0d168","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6823","nonce":1392897493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6827","nonce":1392897493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6831","nonce":1392897493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6829","nonce":1392897493}]},"public_addr":"172.21.15.146:6823/1392897493","cluster_addr":"172.21.15.146:6827/1392897493","heartbeat_back_addr":"172.21.15.146:6831/1392897493","heartbeat_front_addr":"172.21.15.146:6829/1392897493","state":["exists","up"]},{"osd":7,"uuid":"75a79b44-fa5f-426d-a7d3-71f21f78835c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6817","nonce":3268323156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6819","nonce":3268323156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6825","nonce":3268323156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6822","nonce":3268323156}]},"public_addr":"172.21.15.146:6817/3268323156","cluster_addr":"172.21.15.146:6819/3268323156","heartbeat_back_addr":"172.21.15.146:6825/3268323156","heartbeat_front_addr":"172.21.15.146:6822/3268323156","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.310437+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.295850+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.294674+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.526855+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.300023+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.595160+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.346046+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.558741+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T09:27:21.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.304+0000 7f0dc8c00640 1 -- 172.21.15.6:0/760384710 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f0da406b4d0 msgr2=0x7f0da406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:21.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.304+0000 7f0dc8c00640 1 --2- 172.21.15.6:0/760384710 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f0da406b4d0 0x7f0da406d990 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f0dc4072320 tx=0x7f0db0007930 comp rx=0 tx=0).stop 2024-03-04T09:27:21.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.304+0000 7f0dc8c00640 1 -- 172.21.15.6:0/760384710 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0dc4070f10 msgr2=0x7f0dc4131550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:21.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.304+0000 7f0dc8c00640 1 --2- 172.21.15.6:0/760384710 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0dc4070f10 0x7f0dc4131550 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f0db400c3c0 tx=0x7f0db400c880 comp rx=0 tx=0).stop 2024-03-04T09:27:21.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.304+0000 7f0dc8c00640 1 -- 172.21.15.6:0/760384710 shutdown_connections 2024-03-04T09:27:21.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.304+0000 7f0dc8c00640 1 --2- 172.21.15.6:0/760384710 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0dc4145300 0x7f0dc415b4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:21.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.304+0000 7f0dc8c00640 1 --2- 172.21.15.6:0/760384710 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f0da406b4d0 0x7f0da406d990 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:21.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.304+0000 7f0dc8c00640 1 --2- 172.21.15.6:0/760384710 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0dc40cd060 0x7f0dc406d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:21.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.304+0000 7f0dc8c00640 1 --2- 172.21.15.6:0/760384710 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0dc4070f10 0x7f0dc4131550 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:21.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.304+0000 7f0dc8c00640 1 -- 172.21.15.6:0/760384710 >> 172.21.15.6:0/760384710 conn(0x7f0dc406b3d0 msgr2=0x7f0dc40726b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:21.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.304+0000 7f0dc8c00640 1 -- 172.21.15.6:0/760384710 shutdown_connections 2024-03-04T09:27:21.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.304+0000 7f0dc8c00640 1 -- 172.21.15.6:0/760384710 wait complete. 2024-03-04T09:27:21.320 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-04T09:27:21.320 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T09:27:21.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.535+0000 7fd378e00640 1 Processor -- start 2024-03-04T09:27:21.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.535+0000 7fd378e00640 1 -- start start 2024-03-04T09:27:21.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.537+0000 7fd378e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd374145300 0x7fd3740b49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:21.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.537+0000 7fd378e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd374070f10 0x7fd3740cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:21.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.537+0000 7fd378e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3740714d0 con 0x7fd3740cce90 2024-03-04T09:27:21.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.537+0000 7fd378e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3740cd300 con 0x7fd374145300 2024-03-04T09:27:21.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.537+0000 7fd378e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3740717b0 con 0x7fd374070f10 2024-03-04T09:27:21.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.537+0000 7fd373400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd374145300 0x7fd3740b49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:21.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.537+0000 7fd373e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fd3740cce90 0x7fd3740cd930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:45746/0 (socket says 172.21.15.6:45746) 2024-03-04T09:27:21.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.537+0000 7fd373e00640 1 -- 172.21.15.6:0/1242881125 learned_addr learned my addr 172.21.15.6:0/1242881125 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:27:21.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.538+0000 7fd372a00640 1 --2- 172.21.15.6:0/1242881125 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd374070f10 0x7fd3740cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:21.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.538+0000 7fd372000640 1 -- 172.21.15.6:0/1242881125 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2208584335 0 0) 0x7fd3740714d0 con 0x7fd3740cce90 2024-03-04T09:27:21.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.538+0000 7fd372000640 1 -- 172.21.15.6:0/1242881125 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd358003520 con 0x7fd3740cce90 2024-03-04T09:27:21.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.538+0000 7fd373400640 1 -- 172.21.15.6:0/1242881125 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd374070f10 msgr2=0x7fd3740cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:27:21.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.538+0000 7fd373400640 1 --2- 172.21.15.6:0/1242881125 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd374070f10 0x7fd3740cc950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:21.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.538+0000 7fd373400640 1 -- 172.21.15.6:0/1242881125 >> v1:172.21.15.6:6789/0 conn(0x7fd3740cce90 legacy=0x7fd3740cd930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:21.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.538+0000 7fd373400640 1 -- 172.21.15.6:0/1242881125 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd374071a90 con 0x7fd374145300 2024-03-04T09:27:21.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.538+0000 7fd372a00640 1 --2- 172.21.15.6:0/1242881125 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd374070f10 0x7fd3740cc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T09:27:21.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.539+0000 7fd373400640 1 --2- 172.21.15.6:0/1242881125 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd374145300 0x7fd3740b49a0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fd3640097a0 tx=0x7fd36402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c49f599d60ee988e server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:21.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.540+0000 7fd372000640 1 -- 172.21.15.6:0/1242881125 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd364035020 con 0x7fd374145300 2024-03-04T09:27:21.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.540+0000 7fd372000640 1 -- 172.21.15.6:0/1242881125 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd364031830 con 0x7fd374145300 2024-03-04T09:27:21.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.540+0000 7fd372000640 1 -- 172.21.15.6:0/1242881125 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd364031c00 con 0x7fd374145300 2024-03-04T09:27:21.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.540+0000 7fd378e00640 1 -- 172.21.15.6:0/1242881125 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd374145300 msgr2=0x7fd3740b49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:21.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.540+0000 7fd378e00640 1 --2- 172.21.15.6:0/1242881125 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd374145300 0x7fd3740b49a0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fd3640097a0 tx=0x7fd36402b8d0 comp rx=0 tx=0).stop 2024-03-04T09:27:21.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.540+0000 7fd378e00640 1 -- 172.21.15.6:0/1242881125 shutdown_connections 2024-03-04T09:27:21.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.540+0000 7fd378e00640 1 --2- 172.21.15.6:0/1242881125 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd374070f10 0x7fd3740cc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:21.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.540+0000 7fd378e00640 1 --2- 172.21.15.6:0/1242881125 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd374145300 0x7fd3740b49a0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:21.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.540+0000 7fd378e00640 1 -- 172.21.15.6:0/1242881125 >> 172.21.15.6:0/1242881125 conn(0x7fd37406b3d0 msgr2=0x7fd37406b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:21.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.540+0000 7fd378e00640 1 -- 172.21.15.6:0/1242881125 shutdown_connections 2024-03-04T09:27:21.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.541+0000 7fd378e00640 1 -- 172.21.15.6:0/1242881125 wait complete. 2024-03-04T09:27:21.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.542+0000 7fd378e00640 1 Processor -- start 2024-03-04T09:27:21.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.542+0000 7fd378e00640 1 -- start start 2024-03-04T09:27:21.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.543+0000 7fd378e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd374070f10 0x7fd37415a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:21.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.543+0000 7fd378e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd3740cce90 0x7fd37406e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:21.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.543+0000 7fd378e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd374145300 0x7fd37413c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:21.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.543+0000 7fd378e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3741456e0 con 0x7fd3740cce90 2024-03-04T09:27:21.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.543+0000 7fd378e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3740b4ee0 con 0x7fd374145300 2024-03-04T09:27:21.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.543+0000 7fd378e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd37415d580 con 0x7fd374070f10 2024-03-04T09:27:21.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.543+0000 7fd372a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd3740cce90 0x7fd37406e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:21.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.543+0000 7fd372a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd3740cce90 0x7fd37406e190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:50466/0 (socket says 172.21.15.6:50466) 2024-03-04T09:27:21.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.544+0000 7fd372a00640 1 -- 172.21.15.6:0/750968913 learned_addr learned my addr 172.21.15.6:0/750968913 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:27:21.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.544+0000 7fd373400640 1 --2- 172.21.15.6:0/750968913 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd374070f10 0x7fd37415a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:21.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.544+0000 7fd373e00640 1 --2- 172.21.15.6:0/750968913 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd374145300 0x7fd37413c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:21.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.544+0000 7fd372a00640 1 -- 172.21.15.6:0/750968913 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd374070f10 msgr2=0x7fd37415a870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:21.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.544+0000 7fd372a00640 1 --2- 172.21.15.6:0/750968913 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd374070f10 0x7fd37415a870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:21.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.544+0000 7fd372a00640 1 -- 172.21.15.6:0/750968913 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd374145300 msgr2=0x7fd37413c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:21.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.544+0000 7fd372a00640 1 --2- 172.21.15.6:0/750968913 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd374145300 0x7fd37413c900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:21.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.544+0000 7fd372a00640 1 -- 172.21.15.6:0/750968913 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd37406d3d0 con 0x7fd3740cce90 2024-03-04T09:27:21.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.544+0000 7fd373e00640 1 --2- 172.21.15.6:0/750968913 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd374145300 0x7fd37413c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:27:21.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.544+0000 7fd372a00640 1 --2- 172.21.15.6:0/750968913 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd3740cce90 0x7fd37406e190 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fd36000e730 tx=0x7fd36000ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:21.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.545+0000 7fd36b400640 1 -- 172.21.15.6:0/750968913 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd36000f040 con 0x7fd3740cce90 2024-03-04T09:27:21.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.545+0000 7fd36b400640 1 -- 172.21.15.6:0/750968913 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd360004230 con 0x7fd3740cce90 2024-03-04T09:27:21.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.545+0000 7fd36b400640 1 -- 172.21.15.6:0/750968913 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd36000ee40 con 0x7fd3740cce90 2024-03-04T09:27:21.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.545+0000 7fd378e00640 1 -- 172.21.15.6:0/750968913 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd37406d6b0 con 0x7fd3740cce90 2024-03-04T09:27:21.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.545+0000 7fd378e00640 1 -- 172.21.15.6:0/750968913 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd37413d2c0 con 0x7fd3740cce90 2024-03-04T09:27:21.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.546+0000 7fd378e00640 1 -- 172.21.15.6:0/750968913 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd37404f330 con 0x7fd3740cce90 2024-03-04T09:27:21.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.547+0000 7fd36b400640 1 -- 172.21.15.6:0/750968913 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd36000c6d0 con 0x7fd3740cce90 2024-03-04T09:27:21.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.548+0000 7fd36b400640 1 --2- 172.21.15.6:0/750968913 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd35406b4d0 0x7fd35406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:21.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.548+0000 7fd36b400640 1 -- 172.21.15.6:0/750968913 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd36008a0a0 con 0x7fd3740cce90 2024-03-04T09:27:21.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.549+0000 7fd373400640 1 --2- 172.21.15.6:0/750968913 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd35406b4d0 0x7fd35406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:21.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.550+0000 7fd373400640 1 --2- 172.21.15.6:0/750968913 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd35406b4d0 0x7fd35406d990 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fd364036000 tx=0x7fd36403c040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:21.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.553+0000 7fd36b400640 1 -- 172.21.15.6:0/750968913 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd360017020 con 0x7fd3740cce90 2024-03-04T09:27:21.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.742+0000 7fd378e00640 1 -- 172.21.15.6:0/750968913 --> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd37415ae70 con 0x7fd35406b4d0 2024-03-04T09:27:21.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.762+0000 7fd36b400640 1 -- 172.21.15.6:0/750968913 <== mgr.4113 v2:172.21.15.146:6832/1788613334 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251894 (secure 0 0 0) 0x7fd37415ae70 con 0x7fd35406b4d0 2024-03-04T09:27:21.763 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:27:21.764 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-03-04T09:27:21.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.768+0000 7fd378e00640 1 -- 172.21.15.6:0/750968913 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd35406b4d0 msgr2=0x7fd35406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:21.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.768+0000 7fd378e00640 1 --2- 172.21.15.6:0/750968913 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd35406b4d0 0x7fd35406d990 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fd364036000 tx=0x7fd36403c040 comp rx=0 tx=0).stop 2024-03-04T09:27:21.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.768+0000 7fd378e00640 1 -- 172.21.15.6:0/750968913 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd3740cce90 msgr2=0x7fd37406e190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:21.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.768+0000 7fd378e00640 1 --2- 172.21.15.6:0/750968913 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd3740cce90 0x7fd37406e190 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fd36000e730 tx=0x7fd36000ebf0 comp rx=0 tx=0).stop 2024-03-04T09:27:21.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.768+0000 7fd378e00640 1 -- 172.21.15.6:0/750968913 shutdown_connections 2024-03-04T09:27:21.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.768+0000 7fd378e00640 1 --2- 172.21.15.6:0/750968913 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd374145300 0x7fd37413c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:21.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.768+0000 7fd378e00640 1 --2- 172.21.15.6:0/750968913 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd3740cce90 0x7fd37406e190 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:21.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.768+0000 7fd378e00640 1 --2- 172.21.15.6:0/750968913 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd35406b4d0 0x7fd35406d990 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:21.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.768+0000 7fd378e00640 1 --2- 172.21.15.6:0/750968913 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd374070f10 0x7fd37415a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:21.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.768+0000 7fd378e00640 1 -- 172.21.15.6:0/750968913 >> 172.21.15.6:0/750968913 conn(0x7fd37406b3d0 msgr2=0x7fd3740b28b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:21.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.768+0000 7fd378e00640 1 -- 172.21.15.6:0/750968913 shutdown_connections 2024-03-04T09:27:21.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:21.768+0000 7fd378e00640 1 -- 172.21.15.6:0/750968913 wait complete. 2024-03-04T09:27:21.783 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":103,"stamp":"2024-03-04T09:27:19.901532+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":157,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":254528,"kb_used_data":5064,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754720192,"statfs":{"total":773094113280,"available":772833476608,"internally_reserved":0,"allocated":5185536,"data_stored":2783884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":41,"apply_latency_ms":41,"commit_latency_ns":41000000,"apply_latency_ns":41000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.625137"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.334022+0000","last_change":"2024-03-04T09:27:17.331304+0000","last_active":"2024-03-04T09:27:17.334022+0000","last_peered":"2024-03-04T09:27:17.334022+0000","last_clean":"2024-03-04T09:27:17.334022+0000","last_became_active":"2024-03-04T09:27:17.330678+0000","last_became_peered":"2024-03-04T09:27:17.330678+0000","last_unstale":"2024-03-04T09:27:17.334022+0000","last_undegraded":"2024-03-04T09:27:17.334022+0000","last_fullsized":"2024-03-04T09:27:17.334022+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:59:43.771250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.341123+0000","last_change":"2024-03-04T09:27:17.324608+0000","last_active":"2024-03-04T09:27:17.341123+0000","last_peered":"2024-03-04T09:27:17.341123+0000","last_clean":"2024-03-04T09:27:17.341123+0000","last_became_active":"2024-03-04T09:27:17.324328+0000","last_became_peered":"2024-03-04T09:27:17.324328+0000","last_unstale":"2024-03-04T09:27:17.341123+0000","last_undegraded":"2024-03-04T09:27:17.341123+0000","last_fullsized":"2024-03-04T09:27:17.341123+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:22:01.846260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.335614+0000","last_change":"2024-03-04T09:27:17.328076+0000","last_active":"2024-03-04T09:27:17.335614+0000","last_peered":"2024-03-04T09:27:17.335614+0000","last_clean":"2024-03-04T09:27:17.335614+0000","last_became_active":"2024-03-04T09:27:17.327757+0000","last_became_peered":"2024-03-04T09:27:17.327757+0000","last_unstale":"2024-03-04T09:27:17.335614+0000","last_undegraded":"2024-03-04T09:27:17.335614+0000","last_fullsized":"2024-03-04T09:27:17.335614+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:20:51.879579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.337025+0000","last_change":"2024-03-04T09:27:17.331343+0000","last_active":"2024-03-04T09:27:17.337025+0000","last_peered":"2024-03-04T09:27:17.337025+0000","last_clean":"2024-03-04T09:27:17.337025+0000","last_became_active":"2024-03-04T09:27:17.331052+0000","last_became_peered":"2024-03-04T09:27:17.331052+0000","last_unstale":"2024-03-04T09:27:17.337025+0000","last_undegraded":"2024-03-04T09:27:17.337025+0000","last_fullsized":"2024-03-04T09:27:17.337025+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:04:15.264485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.318991+0000","last_change":"2024-03-04T09:27:17.315570+0000","last_active":"2024-03-04T09:27:17.318991+0000","last_peered":"2024-03-04T09:27:17.318991+0000","last_clean":"2024-03-04T09:27:17.318991+0000","last_became_active":"2024-03-04T09:27:17.314078+0000","last_became_peered":"2024-03-04T09:27:17.314078+0000","last_unstale":"2024-03-04T09:27:17.318991+0000","last_undegraded":"2024-03-04T09:27:17.318991+0000","last_fullsized":"2024-03-04T09:27:17.318991+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:45:00.906289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.337294+0000","last_change":"2024-03-04T09:27:17.329811+0000","last_active":"2024-03-04T09:27:17.337294+0000","last_peered":"2024-03-04T09:27:17.337294+0000","last_clean":"2024-03-04T09:27:17.337294+0000","last_became_active":"2024-03-04T09:27:17.328287+0000","last_became_peered":"2024-03-04T09:27:17.328287+0000","last_unstale":"2024-03-04T09:27:17.337294+0000","last_undegraded":"2024-03-04T09:27:17.337294+0000","last_fullsized":"2024-03-04T09:27:17.337294+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:17:17.721926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.330123+0000","last_change":"2024-03-04T09:27:17.324495+0000","last_active":"2024-03-04T09:27:17.330123+0000","last_peered":"2024-03-04T09:27:17.330123+0000","last_clean":"2024-03-04T09:27:17.330123+0000","last_became_active":"2024-03-04T09:27:17.324227+0000","last_became_peered":"2024-03-04T09:27:17.324227+0000","last_unstale":"2024-03-04T09:27:17.330123+0000","last_undegraded":"2024-03-04T09:27:17.330123+0000","last_fullsized":"2024-03-04T09:27:17.330123+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:22:01.846260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.336902+0000","last_change":"2024-03-04T09:27:17.329788+0000","last_active":"2024-03-04T09:27:17.336902+0000","last_peered":"2024-03-04T09:27:17.336902+0000","last_clean":"2024-03-04T09:27:17.336902+0000","last_became_active":"2024-03-04T09:27:17.329059+0000","last_became_peered":"2024-03-04T09:27:17.329059+0000","last_unstale":"2024-03-04T09:27:17.336902+0000","last_undegraded":"2024-03-04T09:27:17.336902+0000","last_fullsized":"2024-03-04T09:27:17.336902+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:38:46.165153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.332607+0000","last_change":"2024-03-04T09:27:17.323939+0000","last_active":"2024-03-04T09:27:17.332607+0000","last_peered":"2024-03-04T09:27:17.332607+0000","last_clean":"2024-03-04T09:27:17.332607+0000","last_became_active":"2024-03-04T09:27:17.323616+0000","last_became_peered":"2024-03-04T09:27:17.323616+0000","last_unstale":"2024-03-04T09:27:17.332607+0000","last_undegraded":"2024-03-04T09:27:17.332607+0000","last_fullsized":"2024-03-04T09:27:17.332607+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:02:04.281829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.339165+0000","last_change":"2024-03-04T09:27:17.336480+0000","last_active":"2024-03-04T09:27:17.339165+0000","last_peered":"2024-03-04T09:27:17.339165+0000","last_clean":"2024-03-04T09:27:17.339165+0000","last_became_active":"2024-03-04T09:27:17.336082+0000","last_became_peered":"2024-03-04T09:27:17.336082+0000","last_unstale":"2024-03-04T09:27:17.339165+0000","last_undegraded":"2024-03-04T09:27:17.339165+0000","last_fullsized":"2024-03-04T09:27:17.339165+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:54:43.681711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.334524+0000","last_change":"2024-03-04T09:27:17.328087+0000","last_active":"2024-03-04T09:27:17.334524+0000","last_peered":"2024-03-04T09:27:17.334524+0000","last_clean":"2024-03-04T09:27:17.334524+0000","last_became_active":"2024-03-04T09:27:17.321983+0000","last_became_peered":"2024-03-04T09:27:17.321983+0000","last_unstale":"2024-03-04T09:27:17.334524+0000","last_undegraded":"2024-03-04T09:27:17.334524+0000","last_fullsized":"2024-03-04T09:27:17.334524+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:58:50.020494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.336724+0000","last_change":"2024-03-04T09:27:17.319871+0000","last_active":"2024-03-04T09:27:17.336724+0000","last_peered":"2024-03-04T09:27:17.336724+0000","last_clean":"2024-03-04T09:27:17.336724+0000","last_became_active":"2024-03-04T09:27:17.316618+0000","last_became_peered":"2024-03-04T09:27:17.316618+0000","last_unstale":"2024-03-04T09:27:17.336724+0000","last_undegraded":"2024-03-04T09:27:17.336724+0000","last_fullsized":"2024-03-04T09:27:17.336724+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:18:11.238140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.328659+0000","last_change":"2024-03-04T09:27:17.323615+0000","last_active":"2024-03-04T09:27:17.328659+0000","last_peered":"2024-03-04T09:27:17.328659+0000","last_clean":"2024-03-04T09:27:17.328659+0000","last_became_active":"2024-03-04T09:27:17.323322+0000","last_became_peered":"2024-03-04T09:27:17.323322+0000","last_unstale":"2024-03-04T09:27:17.328659+0000","last_undegraded":"2024-03-04T09:27:17.328659+0000","last_fullsized":"2024-03-04T09:27:17.328659+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:58:19.947417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.330768+0000","last_change":"2024-03-04T09:27:17.322437+0000","last_active":"2024-03-04T09:27:17.330768+0000","last_peered":"2024-03-04T09:27:17.330768+0000","last_clean":"2024-03-04T09:27:17.330768+0000","last_became_active":"2024-03-04T09:27:17.322114+0000","last_became_peered":"2024-03-04T09:27:17.322114+0000","last_unstale":"2024-03-04T09:27:17.330768+0000","last_undegraded":"2024-03-04T09:27:17.330768+0000","last_fullsized":"2024-03-04T09:27:17.330768+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:33:17.408723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.335333+0000","last_change":"2024-03-04T09:27:17.324834+0000","last_active":"2024-03-04T09:27:17.335333+0000","last_peered":"2024-03-04T09:27:17.335333+0000","last_clean":"2024-03-04T09:27:17.335333+0000","last_became_active":"2024-03-04T09:27:17.322804+0000","last_became_peered":"2024-03-04T09:27:17.322804+0000","last_unstale":"2024-03-04T09:27:17.335333+0000","last_undegraded":"2024-03-04T09:27:17.335333+0000","last_fullsized":"2024-03-04T09:27:17.335333+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:17:48.818289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.325199+0000","last_change":"2024-03-04T09:27:17.317783+0000","last_active":"2024-03-04T09:27:17.325199+0000","last_peered":"2024-03-04T09:27:17.325199+0000","last_clean":"2024-03-04T09:27:17.325199+0000","last_became_active":"2024-03-04T09:27:17.317470+0000","last_became_peered":"2024-03-04T09:27:17.317470+0000","last_unstale":"2024-03-04T09:27:17.325199+0000","last_undegraded":"2024-03-04T09:27:17.325199+0000","last_fullsized":"2024-03-04T09:27:17.325199+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:22:55.277803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.333783+0000","last_change":"2024-03-04T09:27:17.325378+0000","last_active":"2024-03-04T09:27:17.333783+0000","last_peered":"2024-03-04T09:27:17.333783+0000","last_clean":"2024-03-04T09:27:17.333783+0000","last_became_active":"2024-03-04T09:27:17.325022+0000","last_became_peered":"2024-03-04T09:27:17.325022+0000","last_unstale":"2024-03-04T09:27:17.333783+0000","last_undegraded":"2024-03-04T09:27:17.333783+0000","last_fullsized":"2024-03-04T09:27:17.333783+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:58:50.020494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.335060+0000","last_change":"2024-03-04T09:27:17.327654+0000","last_active":"2024-03-04T09:27:17.335060+0000","last_peered":"2024-03-04T09:27:17.335060+0000","last_clean":"2024-03-04T09:27:17.335060+0000","last_became_active":"2024-03-04T09:27:17.327302+0000","last_became_peered":"2024-03-04T09:27:17.327302+0000","last_unstale":"2024-03-04T09:27:17.335060+0000","last_undegraded":"2024-03-04T09:27:17.335060+0000","last_fullsized":"2024-03-04T09:27:17.335060+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:38:46.165153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.326969+0000","last_change":"2024-03-04T09:27:17.322031+0000","last_active":"2024-03-04T09:27:17.326969+0000","last_peered":"2024-03-04T09:27:17.326969+0000","last_clean":"2024-03-04T09:27:17.326969+0000","last_became_active":"2024-03-04T09:27:17.321604+0000","last_became_peered":"2024-03-04T09:27:17.321604+0000","last_unstale":"2024-03-04T09:27:17.326969+0000","last_undegraded":"2024-03-04T09:27:17.326969+0000","last_fullsized":"2024-03-04T09:27:17.326969+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:04:17.214998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.325182+0000","last_change":"2024-03-04T09:27:17.319436+0000","last_active":"2024-03-04T09:27:17.325182+0000","last_peered":"2024-03-04T09:27:17.325182+0000","last_clean":"2024-03-04T09:27:17.325182+0000","last_became_active":"2024-03-04T09:27:17.319117+0000","last_became_peered":"2024-03-04T09:27:17.319117+0000","last_unstale":"2024-03-04T09:27:17.325182+0000","last_undegraded":"2024-03-04T09:27:17.325182+0000","last_fullsized":"2024-03-04T09:27:17.325182+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:04:17.214998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.331101+0000","last_change":"2024-03-04T09:27:17.324487+0000","last_active":"2024-03-04T09:27:17.331101+0000","last_peered":"2024-03-04T09:27:17.331101+0000","last_clean":"2024-03-04T09:27:17.331101+0000","last_became_active":"2024-03-04T09:27:17.324222+0000","last_became_peered":"2024-03-04T09:27:17.324222+0000","last_unstale":"2024-03-04T09:27:17.331101+0000","last_undegraded":"2024-03-04T09:27:17.331101+0000","last_fullsized":"2024-03-04T09:27:17.331101+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:00:23.007140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.337200+0000","last_change":"2024-03-04T09:27:17.332852+0000","last_active":"2024-03-04T09:27:17.337200+0000","last_peered":"2024-03-04T09:27:17.337200+0000","last_clean":"2024-03-04T09:27:17.337200+0000","last_became_active":"2024-03-04T09:27:17.330153+0000","last_became_peered":"2024-03-04T09:27:17.330153+0000","last_unstale":"2024-03-04T09:27:17.337200+0000","last_undegraded":"2024-03-04T09:27:17.337200+0000","last_fullsized":"2024-03-04T09:27:17.337200+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:54:25.631084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.328420+0000","last_change":"2024-03-04T09:27:17.322335+0000","last_active":"2024-03-04T09:27:17.328420+0000","last_peered":"2024-03-04T09:27:17.328420+0000","last_clean":"2024-03-04T09:27:17.328420+0000","last_became_active":"2024-03-04T09:27:17.319989+0000","last_became_peered":"2024-03-04T09:27:17.319989+0000","last_unstale":"2024-03-04T09:27:17.328420+0000","last_undegraded":"2024-03-04T09:27:17.328420+0000","last_fullsized":"2024-03-04T09:27:17.328420+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:59:09.754483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.325227+0000","last_change":"2024-03-04T09:27:17.319504+0000","last_active":"2024-03-04T09:27:17.325227+0000","last_peered":"2024-03-04T09:27:17.325227+0000","last_clean":"2024-03-04T09:27:17.325227+0000","last_became_active":"2024-03-04T09:27:17.319205+0000","last_became_peered":"2024-03-04T09:27:17.319205+0000","last_unstale":"2024-03-04T09:27:17.325227+0000","last_undegraded":"2024-03-04T09:27:17.325227+0000","last_fullsized":"2024-03-04T09:27:17.325227+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:39:23.162213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.333476+0000","last_change":"2024-03-04T09:27:17.326645+0000","last_active":"2024-03-04T09:27:17.333476+0000","last_peered":"2024-03-04T09:27:17.333476+0000","last_clean":"2024-03-04T09:27:17.333476+0000","last_became_active":"2024-03-04T09:27:17.326034+0000","last_became_peered":"2024-03-04T09:27:17.326034+0000","last_unstale":"2024-03-04T09:27:17.333476+0000","last_undegraded":"2024-03-04T09:27:17.333476+0000","last_fullsized":"2024-03-04T09:27:17.333476+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:15:26.101456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.330553+0000","last_change":"2024-03-04T09:27:17.325690+0000","last_active":"2024-03-04T09:27:17.330553+0000","last_peered":"2024-03-04T09:27:17.330553+0000","last_clean":"2024-03-04T09:27:17.330553+0000","last_became_active":"2024-03-04T09:27:17.323884+0000","last_became_peered":"2024-03-04T09:27:17.323884+0000","last_unstale":"2024-03-04T09:27:17.330553+0000","last_undegraded":"2024-03-04T09:27:17.330553+0000","last_fullsized":"2024-03-04T09:27:17.330553+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:04:58.199976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.335637+0000","last_change":"2024-03-04T09:27:17.330314+0000","last_active":"2024-03-04T09:27:17.335637+0000","last_peered":"2024-03-04T09:27:17.335637+0000","last_clean":"2024-03-04T09:27:17.335637+0000","last_became_active":"2024-03-04T09:27:17.330046+0000","last_became_peered":"2024-03-04T09:27:17.330046+0000","last_unstale":"2024-03-04T09:27:17.335637+0000","last_undegraded":"2024-03-04T09:27:17.335637+0000","last_fullsized":"2024-03-04T09:27:17.335637+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:54:25.631084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.322627+0000","last_change":"2024-03-04T09:27:17.318990+0000","last_active":"2024-03-04T09:27:17.322627+0000","last_peered":"2024-03-04T09:27:17.322627+0000","last_clean":"2024-03-04T09:27:17.322627+0000","last_became_active":"2024-03-04T09:27:17.315720+0000","last_became_peered":"2024-03-04T09:27:17.315720+0000","last_unstale":"2024-03-04T09:27:17.322627+0000","last_undegraded":"2024-03-04T09:27:17.322627+0000","last_fullsized":"2024-03-04T09:27:17.322627+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:19:51.542597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.328540+0000","last_change":"2024-03-04T09:27:17.321670+0000","last_active":"2024-03-04T09:27:17.328540+0000","last_peered":"2024-03-04T09:27:17.328540+0000","last_clean":"2024-03-04T09:27:17.328540+0000","last_became_active":"2024-03-04T09:27:17.321268+0000","last_became_peered":"2024-03-04T09:27:17.321268+0000","last_unstale":"2024-03-04T09:27:17.328540+0000","last_undegraded":"2024-03-04T09:27:17.328540+0000","last_fullsized":"2024-03-04T09:27:17.328540+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:49:53.452087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.344528+0000","last_change":"2024-03-04T09:27:17.334676+0000","last_active":"2024-03-04T09:27:17.344528+0000","last_peered":"2024-03-04T09:27:17.344528+0000","last_clean":"2024-03-04T09:27:17.344528+0000","last_became_active":"2024-03-04T09:27:17.334344+0000","last_became_peered":"2024-03-04T09:27:17.334344+0000","last_unstale":"2024-03-04T09:27:17.344528+0000","last_undegraded":"2024-03-04T09:27:17.344528+0000","last_fullsized":"2024-03-04T09:27:17.344528+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:19:15.352216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.337140+0000","last_change":"2024-03-04T09:27:17.326254+0000","last_active":"2024-03-04T09:27:17.337140+0000","last_peered":"2024-03-04T09:27:17.337140+0000","last_clean":"2024-03-04T09:27:17.337140+0000","last_became_active":"2024-03-04T09:27:17.325888+0000","last_became_peered":"2024-03-04T09:27:17.325888+0000","last_unstale":"2024-03-04T09:27:17.337140+0000","last_undegraded":"2024-03-04T09:27:17.337140+0000","last_fullsized":"2024-03-04T09:27:17.337140+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:58:45.796432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.314615+0000","last_change":"2024-03-04T09:27:17.307921+0000","last_active":"2024-03-04T09:27:17.314615+0000","last_peered":"2024-03-04T09:27:17.314615+0000","last_clean":"2024-03-04T09:27:17.314615+0000","last_became_active":"2024-03-04T09:27:17.304606+0000","last_became_peered":"2024-03-04T09:27:17.304606+0000","last_unstale":"2024-03-04T09:27:17.314615+0000","last_undegraded":"2024-03-04T09:27:17.314615+0000","last_fullsized":"2024-03-04T09:27:17.314615+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:39:23.162213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.320887+0000","last_change":"2024-03-04T09:27:17.317313+0000","last_active":"2024-03-04T09:27:17.320887+0000","last_peered":"2024-03-04T09:27:17.320887+0000","last_clean":"2024-03-04T09:27:17.320887+0000","last_became_active":"2024-03-04T09:27:17.316897+0000","last_became_peered":"2024-03-04T09:27:17.316897+0000","last_unstale":"2024-03-04T09:27:17.320887+0000","last_undegraded":"2024-03-04T09:27:17.320887+0000","last_fullsized":"2024-03-04T09:27:17.320887+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:58:50.020494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"43'4","reported_seq":16,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.349524+0000","last_change":"2024-03-04T09:27:16.332982+0000","last_active":"2024-03-04T09:27:17.349524+0000","last_peered":"2024-03-04T09:27:17.349524+0000","last_clean":"2024-03-04T09:27:17.349524+0000","last_became_active":"2024-03-04T09:27:16.332723+0000","last_became_peered":"2024-03-04T09:27:16.332723+0000","last_unstale":"2024-03-04T09:27:17.349524+0000","last_undegraded":"2024-03-04T09:27:17.349524+0000","last_fullsized":"2024-03-04T09:27:17.349524+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_clean_scrub_stamp":"2024-03-04T09:27:15.259708+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:14:24.494158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.335082+0000","last_change":"2024-03-04T09:27:17.328695+0000","last_active":"2024-03-04T09:27:17.335082+0000","last_peered":"2024-03-04T09:27:17.335082+0000","last_clean":"2024-03-04T09:27:17.335082+0000","last_became_active":"2024-03-04T09:27:17.326336+0000","last_became_peered":"2024-03-04T09:27:17.326336+0000","last_unstale":"2024-03-04T09:27:17.335082+0000","last_undegraded":"2024-03-04T09:27:17.335082+0000","last_fullsized":"2024-03-04T09:27:17.335082+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:13:37.135100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"43'1","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.347966+0000","last_change":"2024-03-04T09:27:16.331657+0000","last_active":"2024-03-04T09:27:17.347966+0000","last_peered":"2024-03-04T09:27:17.347966+0000","last_clean":"2024-03-04T09:27:17.347966+0000","last_became_active":"2024-03-04T09:27:16.330639+0000","last_became_peered":"2024-03-04T09:27:16.330639+0000","last_unstale":"2024-03-04T09:27:17.347966+0000","last_undegraded":"2024-03-04T09:27:17.347966+0000","last_fullsized":"2024-03-04T09:27:17.347966+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_clean_scrub_stamp":"2024-03-04T09:27:15.259708+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:33:16.393424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.341419+0000","last_change":"2024-03-04T09:27:17.327599+0000","last_active":"2024-03-04T09:27:17.341419+0000","last_peered":"2024-03-04T09:27:17.341419+0000","last_clean":"2024-03-04T09:27:17.341419+0000","last_became_active":"2024-03-04T09:27:17.327218+0000","last_became_peered":"2024-03-04T09:27:17.327218+0000","last_unstale":"2024-03-04T09:27:17.341419+0000","last_undegraded":"2024-03-04T09:27:17.341419+0000","last_fullsized":"2024-03-04T09:27:17.341419+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:51:45.022053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"43'2","reported_seq":14,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.345758+0000","last_change":"2024-03-04T09:27:16.315657+0000","last_active":"2024-03-04T09:27:17.345758+0000","last_peered":"2024-03-04T09:27:17.345758+0000","last_clean":"2024-03-04T09:27:17.345758+0000","last_became_active":"2024-03-04T09:27:16.315371+0000","last_became_peered":"2024-03-04T09:27:16.315371+0000","last_unstale":"2024-03-04T09:27:17.345758+0000","last_undegraded":"2024-03-04T09:27:17.345758+0000","last_fullsized":"2024-03-04T09:27:17.345758+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_clean_scrub_stamp":"2024-03-04T09:27:15.259708+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:02:03.266530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.341928+0000","last_change":"2024-03-04T09:27:17.333368+0000","last_active":"2024-03-04T09:27:17.341928+0000","last_peered":"2024-03-04T09:27:17.341928+0000","last_clean":"2024-03-04T09:27:17.341928+0000","last_became_active":"2024-03-04T09:27:17.332764+0000","last_became_peered":"2024-03-04T09:27:17.332764+0000","last_unstale":"2024-03-04T09:27:17.341928+0000","last_undegraded":"2024-03-04T09:27:17.341928+0000","last_fullsized":"2024-03-04T09:27:17.341928+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:15:09.249645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"43'2","reported_seq":14,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.345055+0000","last_change":"2024-03-04T09:27:16.325795+0000","last_active":"2024-03-04T09:27:17.345055+0000","last_peered":"2024-03-04T09:27:17.345055+0000","last_clean":"2024-03-04T09:27:17.345055+0000","last_became_active":"2024-03-04T09:27:16.324784+0000","last_became_peered":"2024-03-04T09:27:16.324784+0000","last_unstale":"2024-03-04T09:27:17.345055+0000","last_undegraded":"2024-03-04T09:27:17.345055+0000","last_fullsized":"2024-03-04T09:27:17.345055+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_clean_scrub_stamp":"2024-03-04T09:27:15.259708+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:54:42.666412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.340221+0000","last_change":"2024-03-04T09:27:17.333165+0000","last_active":"2024-03-04T09:27:17.340221+0000","last_peered":"2024-03-04T09:27:17.340221+0000","last_clean":"2024-03-04T09:27:17.340221+0000","last_became_active":"2024-03-04T09:27:17.332852+0000","last_became_peered":"2024-03-04T09:27:17.332852+0000","last_unstale":"2024-03-04T09:27:17.340221+0000","last_undegraded":"2024-03-04T09:27:17.340221+0000","last_fullsized":"2024-03-04T09:27:17.340221+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:54:43.681711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"43'1","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.349510+0000","last_change":"2024-03-04T09:27:16.327989+0000","last_active":"2024-03-04T09:27:17.349510+0000","last_peered":"2024-03-04T09:27:17.349510+0000","last_clean":"2024-03-04T09:27:17.349510+0000","last_became_active":"2024-03-04T09:27:16.327643+0000","last_became_peered":"2024-03-04T09:27:16.327643+0000","last_unstale":"2024-03-04T09:27:17.349510+0000","last_undegraded":"2024-03-04T09:27:17.349510+0000","last_fullsized":"2024-03-04T09:27:17.349510+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_clean_scrub_stamp":"2024-03-04T09:27:15.259708+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:18:10.222841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.326350+0000","last_change":"2024-03-04T09:27:17.321248+0000","last_active":"2024-03-04T09:27:17.326350+0000","last_peered":"2024-03-04T09:27:17.326350+0000","last_clean":"2024-03-04T09:27:17.326350+0000","last_became_active":"2024-03-04T09:27:17.320948+0000","last_became_peered":"2024-03-04T09:27:17.320948+0000","last_unstale":"2024-03-04T09:27:17.326350+0000","last_undegraded":"2024-03-04T09:27:17.326350+0000","last_fullsized":"2024-03-04T09:27:17.326350+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:16:31.292858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"43'1","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.347203+0000","last_change":"2024-03-04T09:27:16.321759+0000","last_active":"2024-03-04T09:27:17.347203+0000","last_peered":"2024-03-04T09:27:17.347203+0000","last_clean":"2024-03-04T09:27:17.347203+0000","last_became_active":"2024-03-04T09:27:16.321404+0000","last_became_peered":"2024-03-04T09:27:16.321404+0000","last_unstale":"2024-03-04T09:27:17.347203+0000","last_undegraded":"2024-03-04T09:27:17.347203+0000","last_fullsized":"2024-03-04T09:27:17.347203+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_clean_scrub_stamp":"2024-03-04T09:27:15.259708+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:02:03.266530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.336928+0000","last_change":"2024-03-04T09:27:17.329840+0000","last_active":"2024-03-04T09:27:17.336928+0000","last_peered":"2024-03-04T09:27:17.336928+0000","last_clean":"2024-03-04T09:27:17.336928+0000","last_became_active":"2024-03-04T09:27:17.329534+0000","last_became_peered":"2024-03-04T09:27:17.329534+0000","last_unstale":"2024-03-04T09:27:17.336928+0000","last_undegraded":"2024-03-04T09:27:17.336928+0000","last_fullsized":"2024-03-04T09:27:17.336928+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:20:51.879579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.299455+0000","last_change":"2024-03-04T09:27:16.312143+0000","last_active":"2024-03-04T09:27:17.299455+0000","last_peered":"2024-03-04T09:27:17.299455+0000","last_clean":"2024-03-04T09:27:17.299455+0000","last_became_active":"2024-03-04T09:27:16.310572+0000","last_became_peered":"2024-03-04T09:27:16.310572+0000","last_unstale":"2024-03-04T09:27:17.299455+0000","last_undegraded":"2024-03-04T09:27:17.299455+0000","last_fullsized":"2024-03-04T09:27:17.299455+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_clean_scrub_stamp":"2024-03-04T09:27:15.259708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:19:14.336917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.330406+0000","last_change":"2024-03-04T09:27:17.324625+0000","last_active":"2024-03-04T09:27:17.330406+0000","last_peered":"2024-03-04T09:27:17.330406+0000","last_clean":"2024-03-04T09:27:17.330406+0000","last_became_active":"2024-03-04T09:27:17.323600+0000","last_became_peered":"2024-03-04T09:27:17.323600+0000","last_unstale":"2024-03-04T09:27:17.330406+0000","last_undegraded":"2024-03-04T09:27:17.330406+0000","last_fullsized":"2024-03-04T09:27:17.330406+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:04:58.199976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.315287+0000","last_change":"2024-03-04T09:27:16.321082+0000","last_active":"2024-03-04T09:27:17.315287+0000","last_peered":"2024-03-04T09:27:17.315287+0000","last_clean":"2024-03-04T09:27:17.315287+0000","last_became_active":"2024-03-04T09:27:16.320792+0000","last_became_peered":"2024-03-04T09:27:16.320792+0000","last_unstale":"2024-03-04T09:27:17.315287+0000","last_undegraded":"2024-03-04T09:27:17.315287+0000","last_fullsized":"2024-03-04T09:27:17.315287+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_clean_scrub_stamp":"2024-03-04T09:27:15.259708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:15:25.086157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.335039+0000","last_change":"2024-03-04T09:27:17.327337+0000","last_active":"2024-03-04T09:27:17.335039+0000","last_peered":"2024-03-04T09:27:17.335039+0000","last_clean":"2024-03-04T09:27:17.335039+0000","last_became_active":"2024-03-04T09:27:17.326769+0000","last_became_peered":"2024-03-04T09:27:17.326769+0000","last_unstale":"2024-03-04T09:27:17.335039+0000","last_undegraded":"2024-03-04T09:27:17.335039+0000","last_fullsized":"2024-03-04T09:27:17.335039+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:15:09.249645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.317989+0000","last_change":"2024-03-04T09:27:16.323682+0000","last_active":"2024-03-04T09:27:17.317989+0000","last_peered":"2024-03-04T09:27:17.317989+0000","last_clean":"2024-03-04T09:27:17.317989+0000","last_became_active":"2024-03-04T09:27:16.323065+0000","last_became_peered":"2024-03-04T09:27:16.323065+0000","last_unstale":"2024-03-04T09:27:17.317989+0000","last_undegraded":"2024-03-04T09:27:17.317989+0000","last_fullsized":"2024-03-04T09:27:17.317989+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_clean_scrub_stamp":"2024-03-04T09:27:15.259708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:24:55.129882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.323880+0000","last_change":"2024-03-04T09:27:17.319590+0000","last_active":"2024-03-04T09:27:17.323880+0000","last_peered":"2024-03-04T09:27:17.323880+0000","last_clean":"2024-03-04T09:27:17.323880+0000","last_became_active":"2024-03-04T09:27:17.319285+0000","last_became_peered":"2024-03-04T09:27:17.319285+0000","last_unstale":"2024-03-04T09:27:17.323880+0000","last_undegraded":"2024-03-04T09:27:17.323880+0000","last_fullsized":"2024-03-04T09:27:17.323880+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:19:51.542597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.316355+0000","last_change":"2024-03-04T09:27:16.326483+0000","last_active":"2024-03-04T09:27:17.316355+0000","last_peered":"2024-03-04T09:27:17.316355+0000","last_clean":"2024-03-04T09:27:17.316355+0000","last_became_active":"2024-03-04T09:27:16.326194+0000","last_became_peered":"2024-03-04T09:27:16.326194+0000","last_unstale":"2024-03-04T09:27:17.316355+0000","last_undegraded":"2024-03-04T09:27:17.316355+0000","last_fullsized":"2024-03-04T09:27:17.316355+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_clean_scrub_stamp":"2024-03-04T09:27:15.259708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:18:24.323145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.326894+0000","last_change":"2024-03-04T09:27:17.322049+0000","last_active":"2024-03-04T09:27:17.326894+0000","last_peered":"2024-03-04T09:27:17.326894+0000","last_clean":"2024-03-04T09:27:17.326894+0000","last_became_active":"2024-03-04T09:27:17.321744+0000","last_became_peered":"2024-03-04T09:27:17.321744+0000","last_unstale":"2024-03-04T09:27:17.326894+0000","last_undegraded":"2024-03-04T09:27:17.326894+0000","last_fullsized":"2024-03-04T09:27:17.326894+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:04:08.085457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"43'3","reported_seq":15,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.347219+0000","last_change":"2024-03-04T09:27:16.333306+0000","last_active":"2024-03-04T09:27:17.347219+0000","last_peered":"2024-03-04T09:27:17.347219+0000","last_clean":"2024-03-04T09:27:17.347219+0000","last_became_active":"2024-03-04T09:27:16.332789+0000","last_became_peered":"2024-03-04T09:27:16.332789+0000","last_unstale":"2024-03-04T09:27:17.347219+0000","last_undegraded":"2024-03-04T09:27:17.347219+0000","last_fullsized":"2024-03-04T09:27:17.347219+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_clean_scrub_stamp":"2024-03-04T09:27:15.259708+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:19:14.336917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.332566+0000","last_change":"2024-03-04T09:27:17.328040+0000","last_active":"2024-03-04T09:27:17.332566+0000","last_peered":"2024-03-04T09:27:17.332566+0000","last_clean":"2024-03-04T09:27:17.332566+0000","last_became_active":"2024-03-04T09:27:17.327725+0000","last_became_peered":"2024-03-04T09:27:17.327725+0000","last_unstale":"2024-03-04T09:27:17.332566+0000","last_undegraded":"2024-03-04T09:27:17.332566+0000","last_fullsized":"2024-03-04T09:27:17.332566+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:51:45.022053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"43'6","reported_seq":18,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.351449+0000","last_change":"2024-03-04T09:27:16.321906+0000","last_active":"2024-03-04T09:27:17.351449+0000","last_peered":"2024-03-04T09:27:17.351449+0000","last_clean":"2024-03-04T09:27:17.351449+0000","last_became_active":"2024-03-04T09:27:16.321626+0000","last_became_peered":"2024-03-04T09:27:16.321626+0000","last_unstale":"2024-03-04T09:27:17.351449+0000","last_undegraded":"2024-03-04T09:27:17.351449+0000","last_fullsized":"2024-03-04T09:27:17.351449+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_clean_scrub_stamp":"2024-03-04T09:27:15.259708+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:03:33.968571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.335657+0000","last_change":"2024-03-04T09:27:17.332145+0000","last_active":"2024-03-04T09:27:17.335657+0000","last_peered":"2024-03-04T09:27:17.335657+0000","last_clean":"2024-03-04T09:27:17.335657+0000","last_became_active":"2024-03-04T09:27:17.331839+0000","last_became_peered":"2024-03-04T09:27:17.331839+0000","last_unstale":"2024-03-04T09:27:17.335657+0000","last_undegraded":"2024-03-04T09:27:17.335657+0000","last_fullsized":"2024-03-04T09:27:17.335657+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:16:58.851007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"43'2","reported_seq":14,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.350348+0000","last_change":"2024-03-04T09:27:16.324507+0000","last_active":"2024-03-04T09:27:17.350348+0000","last_peered":"2024-03-04T09:27:17.350348+0000","last_clean":"2024-03-04T09:27:17.350348+0000","last_became_active":"2024-03-04T09:27:16.324223+0000","last_became_peered":"2024-03-04T09:27:16.324223+0000","last_unstale":"2024-03-04T09:27:17.350348+0000","last_undegraded":"2024-03-04T09:27:17.350348+0000","last_fullsized":"2024-03-04T09:27:17.350348+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_clean_scrub_stamp":"2024-03-04T09:27:15.259708+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:22:00.830961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.325673+0000","last_change":"2024-03-04T09:27:17.318724+0000","last_active":"2024-03-04T09:27:17.325673+0000","last_peered":"2024-03-04T09:27:17.325673+0000","last_clean":"2024-03-04T09:27:17.325673+0000","last_became_active":"2024-03-04T09:27:17.318449+0000","last_became_peered":"2024-03-04T09:27:17.318449+0000","last_unstale":"2024-03-04T09:27:17.325673+0000","last_undegraded":"2024-03-04T09:27:17.325673+0000","last_fullsized":"2024-03-04T09:27:17.325673+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:23:31.196709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.308435+0000","last_change":"2024-03-04T09:27:16.325760+0000","last_active":"2024-03-04T09:27:17.308435+0000","last_peered":"2024-03-04T09:27:17.308435+0000","last_clean":"2024-03-04T09:27:17.308435+0000","last_became_active":"2024-03-04T09:27:16.324504+0000","last_became_peered":"2024-03-04T09:27:16.324504+0000","last_unstale":"2024-03-04T09:27:17.308435+0000","last_undegraded":"2024-03-04T09:27:17.308435+0000","last_fullsized":"2024-03-04T09:27:17.308435+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_clean_scrub_stamp":"2024-03-04T09:27:15.259708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:49:20.888193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.322578+0000","last_change":"2024-03-04T09:27:17.316352+0000","last_active":"2024-03-04T09:27:17.322578+0000","last_peered":"2024-03-04T09:27:17.322578+0000","last_clean":"2024-03-04T09:27:17.322578+0000","last_became_active":"2024-03-04T09:27:17.316057+0000","last_became_peered":"2024-03-04T09:27:17.316057+0000","last_unstale":"2024-03-04T09:27:17.322578+0000","last_undegraded":"2024-03-04T09:27:17.322578+0000","last_fullsized":"2024-03-04T09:27:17.322578+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:37:24.737020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.302740+0000","last_change":"2024-03-04T09:27:16.318271+0000","last_active":"2024-03-04T09:27:17.302740+0000","last_peered":"2024-03-04T09:27:17.302740+0000","last_clean":"2024-03-04T09:27:17.302740+0000","last_became_active":"2024-03-04T09:27:16.317497+0000","last_became_peered":"2024-03-04T09:27:16.317497+0000","last_unstale":"2024-03-04T09:27:17.302740+0000","last_undegraded":"2024-03-04T09:27:17.302740+0000","last_fullsized":"2024-03-04T09:27:17.302740+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_clean_scrub_stamp":"2024-03-04T09:27:15.259708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:49:13.895272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.332487+0000","last_change":"2024-03-04T09:27:17.328279+0000","last_active":"2024-03-04T09:27:17.332487+0000","last_peered":"2024-03-04T09:27:17.332487+0000","last_clean":"2024-03-04T09:27:17.332487+0000","last_became_active":"2024-03-04T09:27:17.327970+0000","last_became_peered":"2024-03-04T09:27:17.327970+0000","last_unstale":"2024-03-04T09:27:17.332487+0000","last_undegraded":"2024-03-04T09:27:17.332487+0000","last_fullsized":"2024-03-04T09:27:17.332487+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:49:14.910571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"43'8","reported_seq":20,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.350229+0000","last_change":"2024-03-04T09:27:16.324555+0000","last_active":"2024-03-04T09:27:17.350229+0000","last_peered":"2024-03-04T09:27:17.350229+0000","last_clean":"2024-03-04T09:27:17.350229+0000","last_became_active":"2024-03-04T09:27:16.323807+0000","last_became_peered":"2024-03-04T09:27:16.323807+0000","last_unstale":"2024-03-04T09:27:17.350229+0000","last_undegraded":"2024-03-04T09:27:17.350229+0000","last_fullsized":"2024-03-04T09:27:17.350229+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_clean_scrub_stamp":"2024-03-04T09:27:15.259708+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:13:36.119801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.324313+0000","last_change":"2024-03-04T09:27:17.319575+0000","last_active":"2024-03-04T09:27:17.324313+0000","last_peered":"2024-03-04T09:27:17.324313+0000","last_clean":"2024-03-04T09:27:17.324313+0000","last_became_active":"2024-03-04T09:27:17.319261+0000","last_became_peered":"2024-03-04T09:27:17.319261+0000","last_unstale":"2024-03-04T09:27:17.324313+0000","last_undegraded":"2024-03-04T09:27:17.324313+0000","last_fullsized":"2024-03-04T09:27:17.324313+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:59:09.754483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"43'1","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.341679+0000","last_change":"2024-03-04T09:27:16.326539+0000","last_active":"2024-03-04T09:27:17.341679+0000","last_peered":"2024-03-04T09:27:17.341679+0000","last_clean":"2024-03-04T09:27:17.341679+0000","last_became_active":"2024-03-04T09:27:16.326306+0000","last_became_peered":"2024-03-04T09:27:16.326306+0000","last_unstale":"2024-03-04T09:27:17.341679+0000","last_undegraded":"2024-03-04T09:27:17.341679+0000","last_fullsized":"2024-03-04T09:27:17.341679+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_clean_scrub_stamp":"2024-03-04T09:27:15.259708+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:22:54.262504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.336855+0000","last_change":"2024-03-04T09:27:17.329575+0000","last_active":"2024-03-04T09:27:17.336855+0000","last_peered":"2024-03-04T09:27:17.336855+0000","last_clean":"2024-03-04T09:27:17.336855+0000","last_became_active":"2024-03-04T09:27:17.329299+0000","last_became_peered":"2024-03-04T09:27:17.329299+0000","last_unstale":"2024-03-04T09:27:17.336855+0000","last_undegraded":"2024-03-04T09:27:17.336855+0000","last_fullsized":"2024-03-04T09:27:17.336855+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:15:09.249645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"43'2","reported_seq":14,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.347383+0000","last_change":"2024-03-04T09:27:16.320968+0000","last_active":"2024-03-04T09:27:17.347383+0000","last_peered":"2024-03-04T09:27:17.347383+0000","last_clean":"2024-03-04T09:27:17.347383+0000","last_became_active":"2024-03-04T09:27:16.319795+0000","last_became_peered":"2024-03-04T09:27:16.319795+0000","last_unstale":"2024-03-04T09:27:17.347383+0000","last_undegraded":"2024-03-04T09:27:17.347383+0000","last_fullsized":"2024-03-04T09:27:17.347383+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_clean_scrub_stamp":"2024-03-04T09:27:15.259708+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:44:09.974493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.330527+0000","last_change":"2024-03-04T09:27:17.325848+0000","last_active":"2024-03-04T09:27:17.330527+0000","last_peered":"2024-03-04T09:27:17.330527+0000","last_clean":"2024-03-04T09:27:17.330527+0000","last_became_active":"2024-03-04T09:27:17.325531+0000","last_became_peered":"2024-03-04T09:27:17.325531+0000","last_unstale":"2024-03-04T09:27:17.330527+0000","last_undegraded":"2024-03-04T09:27:17.330527+0000","last_fullsized":"2024-03-04T09:27:17.330527+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:49:56.762411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"43'1","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.344706+0000","last_change":"2024-03-04T09:27:16.320677+0000","last_active":"2024-03-04T09:27:17.344706+0000","last_peered":"2024-03-04T09:27:17.344706+0000","last_clean":"2024-03-04T09:27:17.344706+0000","last_became_active":"2024-03-04T09:27:16.320322+0000","last_became_peered":"2024-03-04T09:27:16.320322+0000","last_unstale":"2024-03-04T09:27:17.344706+0000","last_undegraded":"2024-03-04T09:27:17.344706+0000","last_fullsized":"2024-03-04T09:27:17.344706+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_clean_scrub_stamp":"2024-03-04T09:27:15.259708+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:07:43.640071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.326990+0000","last_change":"2024-03-04T09:27:17.322747+0000","last_active":"2024-03-04T09:27:17.326990+0000","last_peered":"2024-03-04T09:27:17.326990+0000","last_clean":"2024-03-04T09:27:17.326990+0000","last_became_active":"2024-03-04T09:27:17.322303+0000","last_became_peered":"2024-03-04T09:27:17.322303+0000","last_unstale":"2024-03-04T09:27:17.326990+0000","last_undegraded":"2024-03-04T09:27:17.326990+0000","last_fullsized":"2024-03-04T09:27:17.326990+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:58:19.947417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.305333+0000","last_change":"2024-03-04T09:27:16.329803+0000","last_active":"2024-03-04T09:27:17.305333+0000","last_peered":"2024-03-04T09:27:17.305333+0000","last_clean":"2024-03-04T09:27:17.305333+0000","last_became_active":"2024-03-04T09:27:16.329488+0000","last_became_peered":"2024-03-04T09:27:16.329488+0000","last_unstale":"2024-03-04T09:27:17.305333+0000","last_undegraded":"2024-03-04T09:27:17.305333+0000","last_fullsized":"2024-03-04T09:27:17.305333+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_clean_scrub_stamp":"2024-03-04T09:27:15.259708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:59:42.755951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.325938+0000","last_change":"2024-03-04T09:27:17.321454+0000","last_active":"2024-03-04T09:27:17.325938+0000","last_peered":"2024-03-04T09:27:17.325938+0000","last_clean":"2024-03-04T09:27:17.325938+0000","last_became_active":"2024-03-04T09:27:17.321180+0000","last_became_peered":"2024-03-04T09:27:17.321180+0000","last_unstale":"2024-03-04T09:27:17.325938+0000","last_undegraded":"2024-03-04T09:27:17.325938+0000","last_fullsized":"2024-03-04T09:27:17.325938+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:46:04.631344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"43'2","reported_seq":14,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.349611+0000","last_change":"2024-03-04T09:27:16.327564+0000","last_active":"2024-03-04T09:27:17.349611+0000","last_peered":"2024-03-04T09:27:17.349611+0000","last_clean":"2024-03-04T09:27:17.349611+0000","last_became_active":"2024-03-04T09:27:16.326311+0000","last_became_peered":"2024-03-04T09:27:16.326311+0000","last_unstale":"2024-03-04T09:27:17.349611+0000","last_undegraded":"2024-03-04T09:27:17.349611+0000","last_fullsized":"2024-03-04T09:27:17.349611+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_clean_scrub_stamp":"2024-03-04T09:27:15.259708+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:51:44.006754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.321351+0000","last_change":"2024-03-04T09:27:17.317110+0000","last_active":"2024-03-04T09:27:17.321351+0000","last_peered":"2024-03-04T09:27:17.321351+0000","last_clean":"2024-03-04T09:27:17.321351+0000","last_became_active":"2024-03-04T09:27:17.316782+0000","last_became_peered":"2024-03-04T09:27:17.316782+0000","last_unstale":"2024-03-04T09:27:17.321351+0000","last_undegraded":"2024-03-04T09:27:17.321351+0000","last_fullsized":"2024-03-04T09:27:17.321351+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:24:56.145181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.299135+0000","last_change":"2024-03-04T09:27:16.329888+0000","last_active":"2024-03-04T09:27:17.299135+0000","last_peered":"2024-03-04T09:27:17.299135+0000","last_clean":"2024-03-04T09:27:17.299135+0000","last_became_active":"2024-03-04T09:27:16.329577+0000","last_became_peered":"2024-03-04T09:27:16.329577+0000","last_unstale":"2024-03-04T09:27:17.299135+0000","last_undegraded":"2024-03-04T09:27:17.299135+0000","last_fullsized":"2024-03-04T09:27:17.299135+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_clean_scrub_stamp":"2024-03-04T09:27:15.259708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:54:42.666412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.321278+0000","last_change":"2024-03-04T09:27:17.318851+0000","last_active":"2024-03-04T09:27:17.321278+0000","last_peered":"2024-03-04T09:27:17.321278+0000","last_clean":"2024-03-04T09:27:17.321278+0000","last_became_active":"2024-03-04T09:27:17.318468+0000","last_became_peered":"2024-03-04T09:27:17.318468+0000","last_unstale":"2024-03-04T09:27:17.321278+0000","last_undegraded":"2024-03-04T09:27:17.321278+0000","last_fullsized":"2024-03-04T09:27:17.321278+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:49:53.452087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.316376+0000","last_change":"2024-03-04T09:27:16.328226+0000","last_active":"2024-03-04T09:27:17.316376+0000","last_peered":"2024-03-04T09:27:17.316376+0000","last_clean":"2024-03-04T09:27:17.316376+0000","last_became_active":"2024-03-04T09:27:16.327149+0000","last_became_peered":"2024-03-04T09:27:16.327149+0000","last_unstale":"2024-03-04T09:27:17.316376+0000","last_undegraded":"2024-03-04T09:27:17.316376+0000","last_fullsized":"2024-03-04T09:27:17.316376+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_clean_scrub_stamp":"2024-03-04T09:27:15.259708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:33:55.291079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.332805+0000","last_change":"2024-03-04T09:27:17.323402+0000","last_active":"2024-03-04T09:27:17.332805+0000","last_peered":"2024-03-04T09:27:17.332805+0000","last_clean":"2024-03-04T09:27:17.332805+0000","last_became_active":"2024-03-04T09:27:17.322227+0000","last_became_peered":"2024-03-04T09:27:17.322227+0000","last_unstale":"2024-03-04T09:27:17.332805+0000","last_undegraded":"2024-03-04T09:27:17.332805+0000","last_fullsized":"2024-03-04T09:27:17.332805+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:44:10.989793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.308709+0000","last_change":"2024-03-04T09:27:16.326171+0000","last_active":"2024-03-04T09:27:17.308709+0000","last_peered":"2024-03-04T09:27:17.308709+0000","last_clean":"2024-03-04T09:27:17.308709+0000","last_became_active":"2024-03-04T09:27:16.325873+0000","last_became_peered":"2024-03-04T09:27:16.325873+0000","last_unstale":"2024-03-04T09:27:17.308709+0000","last_undegraded":"2024-03-04T09:27:17.308709+0000","last_fullsized":"2024-03-04T09:27:17.308709+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_clean_scrub_stamp":"2024-03-04T09:27:15.259708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:07:43.640071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.338553+0000","last_change":"2024-03-04T09:27:17.332391+0000","last_active":"2024-03-04T09:27:17.338553+0000","last_peered":"2024-03-04T09:27:17.338553+0000","last_clean":"2024-03-04T09:27:17.338553+0000","last_became_active":"2024-03-04T09:27:17.331771+0000","last_became_peered":"2024-03-04T09:27:17.331771+0000","last_unstale":"2024-03-04T09:27:17.338553+0000","last_undegraded":"2024-03-04T09:27:17.338553+0000","last_fullsized":"2024-03-04T09:27:17.338553+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:18:25.338444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.300727+0000","last_change":"2024-03-04T09:27:16.324697+0000","last_active":"2024-03-04T09:27:17.300727+0000","last_peered":"2024-03-04T09:27:17.300727+0000","last_clean":"2024-03-04T09:27:17.300727+0000","last_became_active":"2024-03-04T09:27:16.323098+0000","last_became_peered":"2024-03-04T09:27:16.323098+0000","last_unstale":"2024-03-04T09:27:17.300727+0000","last_undegraded":"2024-03-04T09:27:17.300727+0000","last_fullsized":"2024-03-04T09:27:17.300727+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_clean_scrub_stamp":"2024-03-04T09:27:15.259708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:19:50.527298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.332372+0000","last_change":"2024-03-04T09:27:17.326199+0000","last_active":"2024-03-04T09:27:17.332372+0000","last_peered":"2024-03-04T09:27:17.332372+0000","last_clean":"2024-03-04T09:27:17.332372+0000","last_became_active":"2024-03-04T09:27:17.326025+0000","last_became_peered":"2024-03-04T09:27:17.326025+0000","last_unstale":"2024-03-04T09:27:17.332372+0000","last_undegraded":"2024-03-04T09:27:17.332372+0000","last_fullsized":"2024-03-04T09:27:17.332372+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:24:56.145181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.309012+0000","last_change":"2024-03-04T09:27:16.333063+0000","last_active":"2024-03-04T09:27:17.309012+0000","last_peered":"2024-03-04T09:27:17.309012+0000","last_clean":"2024-03-04T09:27:17.309012+0000","last_became_active":"2024-03-04T09:27:16.332718+0000","last_became_peered":"2024-03-04T09:27:16.332718+0000","last_unstale":"2024-03-04T09:27:17.309012+0000","last_undegraded":"2024-03-04T09:27:17.309012+0000","last_fullsized":"2024-03-04T09:27:17.309012+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_clean_scrub_stamp":"2024-03-04T09:27:15.259708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:55:33.045780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.326682+0000","last_change":"2024-03-04T09:27:17.320429+0000","last_active":"2024-03-04T09:27:17.326682+0000","last_peered":"2024-03-04T09:27:17.326682+0000","last_clean":"2024-03-04T09:27:17.326682+0000","last_became_active":"2024-03-04T09:27:17.320141+0000","last_became_peered":"2024-03-04T09:27:17.320141+0000","last_unstale":"2024-03-04T09:27:17.326682+0000","last_undegraded":"2024-03-04T09:27:17.326682+0000","last_fullsized":"2024-03-04T09:27:17.326682+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:18:11.238140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.289289+0000","last_change":"2024-03-04T09:27:16.318313+0000","last_active":"2024-03-04T09:27:17.289289+0000","last_peered":"2024-03-04T09:27:17.289289+0000","last_clean":"2024-03-04T09:27:17.289289+0000","last_became_active":"2024-03-04T09:27:16.318008+0000","last_became_peered":"2024-03-04T09:27:16.318008+0000","last_unstale":"2024-03-04T09:27:17.289289+0000","last_undegraded":"2024-03-04T09:27:17.289289+0000","last_fullsized":"2024-03-04T09:27:17.289289+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_clean_scrub_stamp":"2024-03-04T09:27:15.259708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:44:09.974493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.329432+0000","last_change":"2024-03-04T09:27:17.324202+0000","last_active":"2024-03-04T09:27:17.329432+0000","last_peered":"2024-03-04T09:27:17.329432+0000","last_clean":"2024-03-04T09:27:17.329432+0000","last_became_active":"2024-03-04T09:27:17.323912+0000","last_became_peered":"2024-03-04T09:27:17.323912+0000","last_unstale":"2024-03-04T09:27:17.329432+0000","last_undegraded":"2024-03-04T09:27:17.329432+0000","last_fullsized":"2024-03-04T09:27:17.329432+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:52:44.662696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.311175+0000","last_change":"2024-03-04T09:27:16.330197+0000","last_active":"2024-03-04T09:27:17.311175+0000","last_peered":"2024-03-04T09:27:17.311175+0000","last_clean":"2024-03-04T09:27:17.311175+0000","last_became_active":"2024-03-04T09:27:16.329916+0000","last_became_peered":"2024-03-04T09:27:16.329916+0000","last_unstale":"2024-03-04T09:27:17.311175+0000","last_undegraded":"2024-03-04T09:27:17.311175+0000","last_fullsized":"2024-03-04T09:27:17.311175+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_clean_scrub_stamp":"2024-03-04T09:27:15.259708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:44:59.890990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.331080+0000","last_change":"2024-03-04T09:27:17.323907+0000","last_active":"2024-03-04T09:27:17.331080+0000","last_peered":"2024-03-04T09:27:17.331080+0000","last_clean":"2024-03-04T09:27:17.331080+0000","last_became_active":"2024-03-04T09:27:17.322745+0000","last_became_peered":"2024-03-04T09:27:17.322745+0000","last_unstale":"2024-03-04T09:27:17.331080+0000","last_undegraded":"2024-03-04T09:27:17.331080+0000","last_fullsized":"2024-03-04T09:27:17.331080+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:28:33.800438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.311885+0000","last_change":"2024-03-04T09:27:16.324235+0000","last_active":"2024-03-04T09:27:17.311885+0000","last_peered":"2024-03-04T09:27:17.311885+0000","last_clean":"2024-03-04T09:27:17.311885+0000","last_became_active":"2024-03-04T09:27:16.323926+0000","last_became_peered":"2024-03-04T09:27:16.323926+0000","last_unstale":"2024-03-04T09:27:17.311885+0000","last_undegraded":"2024-03-04T09:27:17.311885+0000","last_fullsized":"2024-03-04T09:27:17.311885+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_clean_scrub_stamp":"2024-03-04T09:27:15.259708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:44:14.792105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.319794+0000","last_change":"2024-03-04T09:27:17.313083+0000","last_active":"2024-03-04T09:27:17.319794+0000","last_peered":"2024-03-04T09:27:17.319794+0000","last_clean":"2024-03-04T09:27:17.319794+0000","last_became_active":"2024-03-04T09:27:17.312811+0000","last_became_peered":"2024-03-04T09:27:17.312811+0000","last_unstale":"2024-03-04T09:27:17.319794+0000","last_undegraded":"2024-03-04T09:27:17.319794+0000","last_fullsized":"2024-03-04T09:27:17.319794+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:55:34.061079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.293418+0000","last_change":"2024-03-04T09:27:16.330272+0000","last_active":"2024-03-04T09:27:17.293418+0000","last_peered":"2024-03-04T09:27:17.293418+0000","last_clean":"2024-03-04T09:27:17.293418+0000","last_became_active":"2024-03-04T09:27:16.329925+0000","last_became_peered":"2024-03-04T09:27:16.329925+0000","last_unstale":"2024-03-04T09:27:17.293418+0000","last_undegraded":"2024-03-04T09:27:17.293418+0000","last_fullsized":"2024-03-04T09:27:17.293418+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_clean_scrub_stamp":"2024-03-04T09:27:15.259708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:44:14.792105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.334506+0000","last_change":"2024-03-04T09:27:17.330022+0000","last_active":"2024-03-04T09:27:17.334506+0000","last_peered":"2024-03-04T09:27:17.334506+0000","last_clean":"2024-03-04T09:27:17.334506+0000","last_became_active":"2024-03-04T09:27:17.329631+0000","last_became_peered":"2024-03-04T09:27:17.329631+0000","last_unstale":"2024-03-04T09:27:17.334506+0000","last_undegraded":"2024-03-04T09:27:17.334506+0000","last_fullsized":"2024-03-04T09:27:17.334506+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:04:15.264485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.315636+0000","last_change":"2024-03-04T09:27:16.330827+0000","last_active":"2024-03-04T09:27:17.315636+0000","last_peered":"2024-03-04T09:27:17.315636+0000","last_clean":"2024-03-04T09:27:17.315636+0000","last_became_active":"2024-03-04T09:27:16.330459+0000","last_became_peered":"2024-03-04T09:27:16.330459+0000","last_unstale":"2024-03-04T09:27:17.315636+0000","last_undegraded":"2024-03-04T09:27:17.315636+0000","last_fullsized":"2024-03-04T09:27:17.315636+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_clean_scrub_stamp":"2024-03-04T09:27:15.259708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:16:57.835708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.332227+0000","last_change":"2024-03-04T09:27:17.326000+0000","last_active":"2024-03-04T09:27:17.332227+0000","last_peered":"2024-03-04T09:27:17.332227+0000","last_clean":"2024-03-04T09:27:17.332227+0000","last_became_active":"2024-03-04T09:27:17.325521+0000","last_became_peered":"2024-03-04T09:27:17.325521+0000","last_unstale":"2024-03-04T09:27:17.332227+0000","last_undegraded":"2024-03-04T09:27:17.332227+0000","last_fullsized":"2024-03-04T09:27:17.332227+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:44:15.807404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"43'2","reported_seq":14,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.344120+0000","last_change":"2024-03-04T09:27:16.317358+0000","last_active":"2024-03-04T09:27:17.344120+0000","last_peered":"2024-03-04T09:27:17.344120+0000","last_clean":"2024-03-04T09:27:17.344120+0000","last_became_active":"2024-03-04T09:27:16.316970+0000","last_became_peered":"2024-03-04T09:27:16.316970+0000","last_unstale":"2024-03-04T09:27:17.344120+0000","last_undegraded":"2024-03-04T09:27:17.344120+0000","last_fullsized":"2024-03-04T09:27:17.344120+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_clean_scrub_stamp":"2024-03-04T09:27:15.259708+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:49:20.888193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":213,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.308841+0000","last_change":"2024-03-04T09:25:09.316159+0000","last_active":"2024-03-04T09:27:17.308841+0000","last_peered":"2024-03-04T09:27:17.308841+0000","last_clean":"2024-03-04T09:27:17.308841+0000","last_became_active":"2024-03-04T09:25:09.313767+0000","last_became_peered":"2024-03-04T09:25:09.313767+0000","last_unstale":"2024-03-04T09:27:17.308841+0000","last_undegraded":"2024-03-04T09:27:17.308841+0000","last_fullsized":"2024-03-04T09:27:17.308841+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:08.110953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:08.110953+0000","last_clean_scrub_stamp":"2024-03-04T09:25:08.110953+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:34:21.745390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574877,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32256,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339584,"statfs":{"total":96636764160,"available":96603734016,"internally_reserved":0,"allocated":1101824,"data_stored":794206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574877,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31664,"kb_used_data":480,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340176,"statfs":{"total":96636764160,"available":96604340224,"internally_reserved":0,"allocated":491520,"data_stored":196973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574879,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31676,"kb_used_data":496,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340164,"statfs":{"total":96636764160,"available":96604327936,"internally_reserved":0,"allocated":507904,"data_stored":203818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574876,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31664,"kb_used_data":480,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340176,"statfs":{"total":96636764160,"available":96604340224,"internally_reserved":0,"allocated":491520,"data_stored":196973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574878,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31664,"kb_used_data":480,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340176,"statfs":{"total":96636764160,"available":96604340224,"internally_reserved":0,"allocated":491520,"data_stored":196973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574877,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31664,"kb_used_data":480,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340176,"statfs":{"total":96636764160,"available":96604340224,"internally_reserved":0,"allocated":491520,"data_stored":196973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574878,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31684,"kb_used_data":496,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340156,"statfs":{"total":96636764160,"available":96604319744,"internally_reserved":0,"allocated":507904,"data_stored":203818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574877,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32256,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339584,"statfs":{"total":96636764160,"available":96603734016,"internally_reserved":0,"allocated":1101824,"data_stored":794150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-04T09:27:21.787 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T09:27:22.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.050+0000 7f52c5800640 1 Processor -- start 2024-03-04T09:27:22.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.050+0000 7f52c5800640 1 -- start start 2024-03-04T09:27:22.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.052+0000 7f52c5800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f52c00cd060 0x7f52c015b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:22.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.052+0000 7f52c5800640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52c015ed10 con 0x7f52c015e810 2024-03-04T09:27:22.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.052+0000 7f52c5800640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52c015eff0 con 0x7f52c01552b0 2024-03-04T09:27:22.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.052+0000 7f52c5800640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52c015f2d0 con 0x7f52c00cd060 2024-03-04T09:27:22.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.052+0000 7f52c4e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f52c015e810 0x7f52c00b3cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:45762/0 (socket says 172.21.15.6:45762) 2024-03-04T09:27:22.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.052+0000 7f52c4e00640 1 -- 172.21.15.6:0/3144338631 learned_addr learned my addr 172.21.15.6:0/3144338631 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:27:22.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.052+0000 7f52bf400640 1 --2- 172.21.15.6:0/3144338631 >> v2:172.21.15.146:3300/0 conn(0x7f52c00cd060 0x7f52c015b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:22.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.052+0000 7f52bea00640 1 -- 172.21.15.6:0/3144338631 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1935566940 0 0) 0x7f52c015ed10 con 0x7f52c015e810 2024-03-04T09:27:22.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.053+0000 7f52bea00640 1 -- 172.21.15.6:0/3144338631 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52a4003520 con 0x7f52c015e810 2024-03-04T09:27:22.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.053+0000 7f52bea00640 1 -- 172.21.15.6:0/3144338631 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4177436531 0 0) 0x7f52a4003520 con 0x7f52c015e810 2024-03-04T09:27:22.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.053+0000 7f52bea00640 1 -- 172.21.15.6:0/3144338631 >> v2:172.21.15.146:3300/0 conn(0x7f52c00cd060 msgr2=0x7f52c015b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:27:22.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.053+0000 7f52bea00640 1 --2- 172.21.15.6:0/3144338631 >> v2:172.21.15.146:3300/0 conn(0x7f52c00cd060 0x7f52c015b1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:22.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.053+0000 7f52bea00640 1 -- 172.21.15.6:0/3144338631 >> v1:172.21.15.146:6789/0 conn(0x7f52c01552b0 legacy=0x7f52c00cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:22.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.053+0000 7f52bea00640 1 -- 172.21.15.6:0/3144338631 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52c015f5b0 con 0x7f52c015e810 2024-03-04T09:27:22.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.053+0000 7f52bea00640 1 -- 172.21.15.6:0/3144338631 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7f52b80019f0 con 0x7f52c015e810 2024-03-04T09:27:22.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.053+0000 7f52bea00640 1 -- 172.21.15.6:0/3144338631 >> v1:172.21.15.6:6789/0 conn(0x7f52c015e810 legacy=0x7f52c00b3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:22.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.053+0000 7f52bea00640 1 --2- 172.21.15.6:0/3144338631 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f52a40039b0 0x7f52a4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:22.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.053+0000 7f52bea00640 1 --2- 172.21.15.6:0/3144338631 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f52a4004800 0x7f52a4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:22.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.053+0000 7f52bea00640 1 --2- 172.21.15.6:0/3144338631 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f52a40052e0 0x7f52a400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:22.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.053+0000 7f52bea00640 1 -- 172.21.15.6:0/3144338631 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52c015f2d0 con 0x7f52a40039b0 2024-03-04T09:27:22.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.053+0000 7f52bea00640 1 -- 172.21.15.6:0/3144338631 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52a400c1b0 con 0x7f52a40052e0 2024-03-04T09:27:22.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.053+0000 7f52bea00640 1 -- 172.21.15.6:0/3144338631 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52a400c3b0 con 0x7f52a4004800 2024-03-04T09:27:22.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.053+0000 7f52bfe00640 1 --2- 172.21.15.6:0/3144338631 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f52a40039b0 0x7f52a4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:22.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.054+0000 7f52c4e00640 1 --2- 172.21.15.6:0/3144338631 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f52a40052e0 0x7f52a400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:22.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.054+0000 7f52bf400640 1 --2- 172.21.15.6:0/3144338631 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f52a4004800 0x7f52a4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:22.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.054+0000 7f52bfe00640 1 -- 172.21.15.6:0/3144338631 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f52a4004800 msgr2=0x7f52a4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:27:22.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.054+0000 7f52bfe00640 1 --2- 172.21.15.6:0/3144338631 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f52a4004800 0x7f52a4004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:22.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.054+0000 7f52bfe00640 1 -- 172.21.15.6:0/3144338631 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f52a40052e0 msgr2=0x7f52a400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:27:22.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.054+0000 7f52bfe00640 1 --2- 172.21.15.6:0/3144338631 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f52a40052e0 0x7f52a400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:22.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.054+0000 7f52bfe00640 1 -- 172.21.15.6:0/3144338631 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52a4003520 con 0x7f52a40039b0 2024-03-04T09:27:22.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.054+0000 7f52c4e00640 1 --2- 172.21.15.6:0/3144338631 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f52a40052e0 0x7f52a400b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:27:22.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.054+0000 7f52c4e00640 1 -- 172.21.15.6:0/3144338631 reap_dead start 2024-03-04T09:27:22.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.055+0000 7f52bfe00640 1 --2- 172.21.15.6:0/3144338631 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f52a40039b0 0x7f52a4003dd0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f52b0001720 tx=0x7f52a4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1898dc3f54e483bd server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:22.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.055+0000 7f52bea00640 1 -- 172.21.15.6:0/3144338631 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52b0001370 con 0x7f52a40039b0 2024-03-04T09:27:22.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.056+0000 7f52bea00640 1 -- 172.21.15.6:0/3144338631 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52b00052d0 con 0x7f52a40039b0 2024-03-04T09:27:22.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.056+0000 7f52bea00640 1 -- 172.21.15.6:0/3144338631 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52b0003830 con 0x7f52a40039b0 2024-03-04T09:27:22.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.056+0000 7f52c5800640 1 -- 172.21.15.6:0/3144338631 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f52a40039b0 msgr2=0x7f52a4003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:22.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.056+0000 7f52c5800640 1 --2- 172.21.15.6:0/3144338631 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f52a40039b0 0x7f52a4003dd0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f52b0001720 tx=0x7f52a4005830 comp rx=0 tx=0).stop 2024-03-04T09:27:22.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.056+0000 7f52c5800640 1 -- 172.21.15.6:0/3144338631 shutdown_connections 2024-03-04T09:27:22.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.056+0000 7f52c5800640 1 --2- 172.21.15.6:0/3144338631 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f52a40039b0 0x7f52a4003dd0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:22.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.056+0000 7f52c5800640 1 -- 172.21.15.6:0/3144338631 >> 172.21.15.6:0/3144338631 conn(0x7f52c006b3d0 msgr2=0x7f52c006b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:22.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.056+0000 7f52c5800640 1 -- 172.21.15.6:0/3144338631 shutdown_connections 2024-03-04T09:27:22.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.056+0000 7f52c5800640 1 -- 172.21.15.6:0/3144338631 wait complete. 2024-03-04T09:27:22.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.058+0000 7f52c5800640 1 Processor -- start 2024-03-04T09:27:22.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.058+0000 7f52c5800640 1 -- start start 2024-03-04T09:27:22.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.059+0000 7f52c5800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f52a40039b0 0x7f52c0152970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:22.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.059+0000 7f52c5800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f52c015a4d0 0x7f52c014d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:22.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.059+0000 7f52c5800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f52c015d410 0x7f52c015d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:22.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.059+0000 7f52c5800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52c006dd30 con 0x7f52c015a4d0 2024-03-04T09:27:22.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.059+0000 7f52c5800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52c00b3760 con 0x7f52a40039b0 2024-03-04T09:27:22.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.059+0000 7f52c5800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52c006deb0 con 0x7f52c015d410 2024-03-04T09:27:22.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.059+0000 7f52bf400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f52c015a4d0 0x7f52c014d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:22.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.059+0000 7f52bf400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f52c015a4d0 0x7f52c014d780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:50480/0 (socket says 172.21.15.6:50480) 2024-03-04T09:27:22.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.060+0000 7f52bf400640 1 -- 172.21.15.6:0/3840925229 learned_addr learned my addr 172.21.15.6:0/3840925229 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:27:22.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.060+0000 7f52bfe00640 1 --2- 172.21.15.6:0/3840925229 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f52a40039b0 0x7f52c0152970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:22.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.060+0000 7f52c4e00640 1 --2- 172.21.15.6:0/3840925229 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f52c015d410 0x7f52c015d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:22.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.060+0000 7f52bf400640 1 -- 172.21.15.6:0/3840925229 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f52c015d410 msgr2=0x7f52c015d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:22.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.060+0000 7f52bf400640 1 --2- 172.21.15.6:0/3840925229 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f52c015d410 0x7f52c015d8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:22.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.060+0000 7f52bf400640 1 -- 172.21.15.6:0/3840925229 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f52a40039b0 msgr2=0x7f52c0152970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:22.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.060+0000 7f52bf400640 1 --2- 172.21.15.6:0/3840925229 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f52a40039b0 0x7f52c0152970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:22.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.060+0000 7f52bf400640 1 -- 172.21.15.6:0/3840925229 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52c014dcc0 con 0x7f52c015a4d0 2024-03-04T09:27:22.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.060+0000 7f52c4e00640 1 --2- 172.21.15.6:0/3840925229 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f52c015d410 0x7f52c015d8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:27:22.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.060+0000 7f52bfe00640 1 --2- 172.21.15.6:0/3840925229 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f52a40039b0 0x7f52c0152970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:27:22.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.060+0000 7f52bf400640 1 --2- 172.21.15.6:0/3840925229 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f52c015a4d0 0x7f52c014d780 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f52ac00e570 tx=0x7f52ac00ea30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:22.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.060+0000 7f52b7e00640 1 -- 172.21.15.6:0/3840925229 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52ac017070 con 0x7f52c015a4d0 2024-03-04T09:27:22.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.061+0000 7f52b7e00640 1 -- 172.21.15.6:0/3840925229 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52ac013610 con 0x7f52c015a4d0 2024-03-04T09:27:22.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.061+0000 7f52b7e00640 1 -- 172.21.15.6:0/3840925229 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52ac00f040 con 0x7f52c015a4d0 2024-03-04T09:27:22.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.061+0000 7f52c5800640 1 -- 172.21.15.6:0/3840925229 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52c015a9d0 con 0x7f52c015a4d0 2024-03-04T09:27:22.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.061+0000 7f52c5800640 1 -- 172.21.15.6:0/3840925229 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52c01f79a0 con 0x7f52c015a4d0 2024-03-04T09:27:22.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.062+0000 7f52c5800640 1 -- 172.21.15.6:0/3840925229 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52c004f330 con 0x7f52c015a4d0 2024-03-04T09:27:22.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.063+0000 7f52b7e00640 1 -- 172.21.15.6:0/3840925229 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f52ac01a070 con 0x7f52c015a4d0 2024-03-04T09:27:22.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.064+0000 7f52b7e00640 1 --2- 172.21.15.6:0/3840925229 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f52a006b4d0 0x7f52a006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:22.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.064+0000 7f52b7e00640 1 -- 172.21.15.6:0/3840925229 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f52ac089c60 con 0x7f52c015a4d0 2024-03-04T09:27:22.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.064+0000 7f52bfe00640 1 --2- 172.21.15.6:0/3840925229 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f52a006b4d0 0x7f52a006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:22.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.065+0000 7f52bfe00640 1 --2- 172.21.15.6:0/3840925229 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f52a006b4d0 0x7f52a006d990 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f52b0033000 tx=0x7f52b0009030 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:22.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.068+0000 7f52b7e00640 1 -- 172.21.15.6:0/3840925229 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f52ac058a40 con 0x7f52c015a4d0 2024-03-04T09:27:22.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.262+0000 7f52c5800640 1 -- 172.21.15.6:0/3840925229 --> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f52c00cf4a0 con 0x7f52a006b4d0 2024-03-04T09:27:22.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.281+0000 7f52b7e00640 1 -- 172.21.15.6:0/3840925229 <== mgr.4113 v2:172.21.15.146:6832/1788613334 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251894 (secure 0 0 0) 0x7f52c00cf4a0 con 0x7f52a006b4d0 2024-03-04T09:27:22.281 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:27:22.283 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-03-04T09:27:22.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.287+0000 7f52c5800640 1 -- 172.21.15.6:0/3840925229 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f52a006b4d0 msgr2=0x7f52a006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:22.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.287+0000 7f52c5800640 1 --2- 172.21.15.6:0/3840925229 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f52a006b4d0 0x7f52a006d990 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f52b0033000 tx=0x7f52b0009030 comp rx=0 tx=0).stop 2024-03-04T09:27:22.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.287+0000 7f52c5800640 1 -- 172.21.15.6:0/3840925229 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f52c015a4d0 msgr2=0x7f52c014d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:22.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.287+0000 7f52c5800640 1 --2- 172.21.15.6:0/3840925229 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f52c015a4d0 0x7f52c014d780 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f52ac00e570 tx=0x7f52ac00ea30 comp rx=0 tx=0).stop 2024-03-04T09:27:22.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.287+0000 7f52c5800640 1 -- 172.21.15.6:0/3840925229 shutdown_connections 2024-03-04T09:27:22.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.287+0000 7f52c5800640 1 --2- 172.21.15.6:0/3840925229 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f52a006b4d0 0x7f52a006d990 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:22.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.287+0000 7f52c5800640 1 --2- 172.21.15.6:0/3840925229 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f52c015d410 0x7f52c015d8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:22.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.287+0000 7f52c5800640 1 --2- 172.21.15.6:0/3840925229 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f52c015a4d0 0x7f52c014d780 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:22.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.287+0000 7f52c5800640 1 --2- 172.21.15.6:0/3840925229 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f52a40039b0 0x7f52c0152970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:22.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.287+0000 7f52c5800640 1 -- 172.21.15.6:0/3840925229 >> 172.21.15.6:0/3840925229 conn(0x7f52c006b3d0 msgr2=0x7f52c0071ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:22.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.287+0000 7f52c5800640 1 -- 172.21.15.6:0/3840925229 shutdown_connections 2024-03-04T09:27:22.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.287+0000 7f52c5800640 1 -- 172.21.15.6:0/3840925229 wait complete. 2024-03-04T09:27:22.304 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":104,"stamp":"2024-03-04T09:27:21.902109+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":157,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":254528,"kb_used_data":5064,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754720192,"statfs":{"total":773094113280,"available":772833476608,"internally_reserved":0,"allocated":5185536,"data_stored":2783884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":41,"apply_latency_ms":41,"commit_latency_ns":41000000,"apply_latency_ns":41000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001787"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.334022+0000","last_change":"2024-03-04T09:27:17.331304+0000","last_active":"2024-03-04T09:27:17.334022+0000","last_peered":"2024-03-04T09:27:17.334022+0000","last_clean":"2024-03-04T09:27:17.334022+0000","last_became_active":"2024-03-04T09:27:17.330678+0000","last_became_peered":"2024-03-04T09:27:17.330678+0000","last_unstale":"2024-03-04T09:27:17.334022+0000","last_undegraded":"2024-03-04T09:27:17.334022+0000","last_fullsized":"2024-03-04T09:27:17.334022+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:59:43.771250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.341123+0000","last_change":"2024-03-04T09:27:17.324608+0000","last_active":"2024-03-04T09:27:17.341123+0000","last_peered":"2024-03-04T09:27:17.341123+0000","last_clean":"2024-03-04T09:27:17.341123+0000","last_became_active":"2024-03-04T09:27:17.324328+0000","last_became_peered":"2024-03-04T09:27:17.324328+0000","last_unstale":"2024-03-04T09:27:17.341123+0000","last_undegraded":"2024-03-04T09:27:17.341123+0000","last_fullsized":"2024-03-04T09:27:17.341123+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:22:01.846260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.335614+0000","last_change":"2024-03-04T09:27:17.328076+0000","last_active":"2024-03-04T09:27:17.335614+0000","last_peered":"2024-03-04T09:27:17.335614+0000","last_clean":"2024-03-04T09:27:17.335614+0000","last_became_active":"2024-03-04T09:27:17.327757+0000","last_became_peered":"2024-03-04T09:27:17.327757+0000","last_unstale":"2024-03-04T09:27:17.335614+0000","last_undegraded":"2024-03-04T09:27:17.335614+0000","last_fullsized":"2024-03-04T09:27:17.335614+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:20:51.879579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.337025+0000","last_change":"2024-03-04T09:27:17.331343+0000","last_active":"2024-03-04T09:27:17.337025+0000","last_peered":"2024-03-04T09:27:17.337025+0000","last_clean":"2024-03-04T09:27:17.337025+0000","last_became_active":"2024-03-04T09:27:17.331052+0000","last_became_peered":"2024-03-04T09:27:17.331052+0000","last_unstale":"2024-03-04T09:27:17.337025+0000","last_undegraded":"2024-03-04T09:27:17.337025+0000","last_fullsized":"2024-03-04T09:27:17.337025+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:04:15.264485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.318991+0000","last_change":"2024-03-04T09:27:17.315570+0000","last_active":"2024-03-04T09:27:17.318991+0000","last_peered":"2024-03-04T09:27:17.318991+0000","last_clean":"2024-03-04T09:27:17.318991+0000","last_became_active":"2024-03-04T09:27:17.314078+0000","last_became_peered":"2024-03-04T09:27:17.314078+0000","last_unstale":"2024-03-04T09:27:17.318991+0000","last_undegraded":"2024-03-04T09:27:17.318991+0000","last_fullsized":"2024-03-04T09:27:17.318991+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:45:00.906289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.337294+0000","last_change":"2024-03-04T09:27:17.329811+0000","last_active":"2024-03-04T09:27:17.337294+0000","last_peered":"2024-03-04T09:27:17.337294+0000","last_clean":"2024-03-04T09:27:17.337294+0000","last_became_active":"2024-03-04T09:27:17.328287+0000","last_became_peered":"2024-03-04T09:27:17.328287+0000","last_unstale":"2024-03-04T09:27:17.337294+0000","last_undegraded":"2024-03-04T09:27:17.337294+0000","last_fullsized":"2024-03-04T09:27:17.337294+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:17:17.721926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.330123+0000","last_change":"2024-03-04T09:27:17.324495+0000","last_active":"2024-03-04T09:27:17.330123+0000","last_peered":"2024-03-04T09:27:17.330123+0000","last_clean":"2024-03-04T09:27:17.330123+0000","last_became_active":"2024-03-04T09:27:17.324227+0000","last_became_peered":"2024-03-04T09:27:17.324227+0000","last_unstale":"2024-03-04T09:27:17.330123+0000","last_undegraded":"2024-03-04T09:27:17.330123+0000","last_fullsized":"2024-03-04T09:27:17.330123+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:22:01.846260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.336902+0000","last_change":"2024-03-04T09:27:17.329788+0000","last_active":"2024-03-04T09:27:17.336902+0000","last_peered":"2024-03-04T09:27:17.336902+0000","last_clean":"2024-03-04T09:27:17.336902+0000","last_became_active":"2024-03-04T09:27:17.329059+0000","last_became_peered":"2024-03-04T09:27:17.329059+0000","last_unstale":"2024-03-04T09:27:17.336902+0000","last_undegraded":"2024-03-04T09:27:17.336902+0000","last_fullsized":"2024-03-04T09:27:17.336902+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:38:46.165153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.332607+0000","last_change":"2024-03-04T09:27:17.323939+0000","last_active":"2024-03-04T09:27:17.332607+0000","last_peered":"2024-03-04T09:27:17.332607+0000","last_clean":"2024-03-04T09:27:17.332607+0000","last_became_active":"2024-03-04T09:27:17.323616+0000","last_became_peered":"2024-03-04T09:27:17.323616+0000","last_unstale":"2024-03-04T09:27:17.332607+0000","last_undegraded":"2024-03-04T09:27:17.332607+0000","last_fullsized":"2024-03-04T09:27:17.332607+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:02:04.281829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.339165+0000","last_change":"2024-03-04T09:27:17.336480+0000","last_active":"2024-03-04T09:27:17.339165+0000","last_peered":"2024-03-04T09:27:17.339165+0000","last_clean":"2024-03-04T09:27:17.339165+0000","last_became_active":"2024-03-04T09:27:17.336082+0000","last_became_peered":"2024-03-04T09:27:17.336082+0000","last_unstale":"2024-03-04T09:27:17.339165+0000","last_undegraded":"2024-03-04T09:27:17.339165+0000","last_fullsized":"2024-03-04T09:27:17.339165+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:54:43.681711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.334524+0000","last_change":"2024-03-04T09:27:17.328087+0000","last_active":"2024-03-04T09:27:17.334524+0000","last_peered":"2024-03-04T09:27:17.334524+0000","last_clean":"2024-03-04T09:27:17.334524+0000","last_became_active":"2024-03-04T09:27:17.321983+0000","last_became_peered":"2024-03-04T09:27:17.321983+0000","last_unstale":"2024-03-04T09:27:17.334524+0000","last_undegraded":"2024-03-04T09:27:17.334524+0000","last_fullsized":"2024-03-04T09:27:17.334524+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:58:50.020494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.336724+0000","last_change":"2024-03-04T09:27:17.319871+0000","last_active":"2024-03-04T09:27:17.336724+0000","last_peered":"2024-03-04T09:27:17.336724+0000","last_clean":"2024-03-04T09:27:17.336724+0000","last_became_active":"2024-03-04T09:27:17.316618+0000","last_became_peered":"2024-03-04T09:27:17.316618+0000","last_unstale":"2024-03-04T09:27:17.336724+0000","last_undegraded":"2024-03-04T09:27:17.336724+0000","last_fullsized":"2024-03-04T09:27:17.336724+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:18:11.238140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.328659+0000","last_change":"2024-03-04T09:27:17.323615+0000","last_active":"2024-03-04T09:27:17.328659+0000","last_peered":"2024-03-04T09:27:17.328659+0000","last_clean":"2024-03-04T09:27:17.328659+0000","last_became_active":"2024-03-04T09:27:17.323322+0000","last_became_peered":"2024-03-04T09:27:17.323322+0000","last_unstale":"2024-03-04T09:27:17.328659+0000","last_undegraded":"2024-03-04T09:27:17.328659+0000","last_fullsized":"2024-03-04T09:27:17.328659+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:58:19.947417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.330768+0000","last_change":"2024-03-04T09:27:17.322437+0000","last_active":"2024-03-04T09:27:17.330768+0000","last_peered":"2024-03-04T09:27:17.330768+0000","last_clean":"2024-03-04T09:27:17.330768+0000","last_became_active":"2024-03-04T09:27:17.322114+0000","last_became_peered":"2024-03-04T09:27:17.322114+0000","last_unstale":"2024-03-04T09:27:17.330768+0000","last_undegraded":"2024-03-04T09:27:17.330768+0000","last_fullsized":"2024-03-04T09:27:17.330768+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:33:17.408723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.335333+0000","last_change":"2024-03-04T09:27:17.324834+0000","last_active":"2024-03-04T09:27:17.335333+0000","last_peered":"2024-03-04T09:27:17.335333+0000","last_clean":"2024-03-04T09:27:17.335333+0000","last_became_active":"2024-03-04T09:27:17.322804+0000","last_became_peered":"2024-03-04T09:27:17.322804+0000","last_unstale":"2024-03-04T09:27:17.335333+0000","last_undegraded":"2024-03-04T09:27:17.335333+0000","last_fullsized":"2024-03-04T09:27:17.335333+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:17:48.818289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.325199+0000","last_change":"2024-03-04T09:27:17.317783+0000","last_active":"2024-03-04T09:27:17.325199+0000","last_peered":"2024-03-04T09:27:17.325199+0000","last_clean":"2024-03-04T09:27:17.325199+0000","last_became_active":"2024-03-04T09:27:17.317470+0000","last_became_peered":"2024-03-04T09:27:17.317470+0000","last_unstale":"2024-03-04T09:27:17.325199+0000","last_undegraded":"2024-03-04T09:27:17.325199+0000","last_fullsized":"2024-03-04T09:27:17.325199+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:22:55.277803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.333783+0000","last_change":"2024-03-04T09:27:17.325378+0000","last_active":"2024-03-04T09:27:17.333783+0000","last_peered":"2024-03-04T09:27:17.333783+0000","last_clean":"2024-03-04T09:27:17.333783+0000","last_became_active":"2024-03-04T09:27:17.325022+0000","last_became_peered":"2024-03-04T09:27:17.325022+0000","last_unstale":"2024-03-04T09:27:17.333783+0000","last_undegraded":"2024-03-04T09:27:17.333783+0000","last_fullsized":"2024-03-04T09:27:17.333783+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:58:50.020494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.335060+0000","last_change":"2024-03-04T09:27:17.327654+0000","last_active":"2024-03-04T09:27:17.335060+0000","last_peered":"2024-03-04T09:27:17.335060+0000","last_clean":"2024-03-04T09:27:17.335060+0000","last_became_active":"2024-03-04T09:27:17.327302+0000","last_became_peered":"2024-03-04T09:27:17.327302+0000","last_unstale":"2024-03-04T09:27:17.335060+0000","last_undegraded":"2024-03-04T09:27:17.335060+0000","last_fullsized":"2024-03-04T09:27:17.335060+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:38:46.165153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.326969+0000","last_change":"2024-03-04T09:27:17.322031+0000","last_active":"2024-03-04T09:27:17.326969+0000","last_peered":"2024-03-04T09:27:17.326969+0000","last_clean":"2024-03-04T09:27:17.326969+0000","last_became_active":"2024-03-04T09:27:17.321604+0000","last_became_peered":"2024-03-04T09:27:17.321604+0000","last_unstale":"2024-03-04T09:27:17.326969+0000","last_undegraded":"2024-03-04T09:27:17.326969+0000","last_fullsized":"2024-03-04T09:27:17.326969+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:04:17.214998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.325182+0000","last_change":"2024-03-04T09:27:17.319436+0000","last_active":"2024-03-04T09:27:17.325182+0000","last_peered":"2024-03-04T09:27:17.325182+0000","last_clean":"2024-03-04T09:27:17.325182+0000","last_became_active":"2024-03-04T09:27:17.319117+0000","last_became_peered":"2024-03-04T09:27:17.319117+0000","last_unstale":"2024-03-04T09:27:17.325182+0000","last_undegraded":"2024-03-04T09:27:17.325182+0000","last_fullsized":"2024-03-04T09:27:17.325182+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:04:17.214998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.331101+0000","last_change":"2024-03-04T09:27:17.324487+0000","last_active":"2024-03-04T09:27:17.331101+0000","last_peered":"2024-03-04T09:27:17.331101+0000","last_clean":"2024-03-04T09:27:17.331101+0000","last_became_active":"2024-03-04T09:27:17.324222+0000","last_became_peered":"2024-03-04T09:27:17.324222+0000","last_unstale":"2024-03-04T09:27:17.331101+0000","last_undegraded":"2024-03-04T09:27:17.331101+0000","last_fullsized":"2024-03-04T09:27:17.331101+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:00:23.007140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.337200+0000","last_change":"2024-03-04T09:27:17.332852+0000","last_active":"2024-03-04T09:27:17.337200+0000","last_peered":"2024-03-04T09:27:17.337200+0000","last_clean":"2024-03-04T09:27:17.337200+0000","last_became_active":"2024-03-04T09:27:17.330153+0000","last_became_peered":"2024-03-04T09:27:17.330153+0000","last_unstale":"2024-03-04T09:27:17.337200+0000","last_undegraded":"2024-03-04T09:27:17.337200+0000","last_fullsized":"2024-03-04T09:27:17.337200+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:54:25.631084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.328420+0000","last_change":"2024-03-04T09:27:17.322335+0000","last_active":"2024-03-04T09:27:17.328420+0000","last_peered":"2024-03-04T09:27:17.328420+0000","last_clean":"2024-03-04T09:27:17.328420+0000","last_became_active":"2024-03-04T09:27:17.319989+0000","last_became_peered":"2024-03-04T09:27:17.319989+0000","last_unstale":"2024-03-04T09:27:17.328420+0000","last_undegraded":"2024-03-04T09:27:17.328420+0000","last_fullsized":"2024-03-04T09:27:17.328420+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:59:09.754483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.325227+0000","last_change":"2024-03-04T09:27:17.319504+0000","last_active":"2024-03-04T09:27:17.325227+0000","last_peered":"2024-03-04T09:27:17.325227+0000","last_clean":"2024-03-04T09:27:17.325227+0000","last_became_active":"2024-03-04T09:27:17.319205+0000","last_became_peered":"2024-03-04T09:27:17.319205+0000","last_unstale":"2024-03-04T09:27:17.325227+0000","last_undegraded":"2024-03-04T09:27:17.325227+0000","last_fullsized":"2024-03-04T09:27:17.325227+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:39:23.162213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.333476+0000","last_change":"2024-03-04T09:27:17.326645+0000","last_active":"2024-03-04T09:27:17.333476+0000","last_peered":"2024-03-04T09:27:17.333476+0000","last_clean":"2024-03-04T09:27:17.333476+0000","last_became_active":"2024-03-04T09:27:17.326034+0000","last_became_peered":"2024-03-04T09:27:17.326034+0000","last_unstale":"2024-03-04T09:27:17.333476+0000","last_undegraded":"2024-03-04T09:27:17.333476+0000","last_fullsized":"2024-03-04T09:27:17.333476+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:15:26.101456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.330553+0000","last_change":"2024-03-04T09:27:17.325690+0000","last_active":"2024-03-04T09:27:17.330553+0000","last_peered":"2024-03-04T09:27:17.330553+0000","last_clean":"2024-03-04T09:27:17.330553+0000","last_became_active":"2024-03-04T09:27:17.323884+0000","last_became_peered":"2024-03-04T09:27:17.323884+0000","last_unstale":"2024-03-04T09:27:17.330553+0000","last_undegraded":"2024-03-04T09:27:17.330553+0000","last_fullsized":"2024-03-04T09:27:17.330553+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:04:58.199976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.335637+0000","last_change":"2024-03-04T09:27:17.330314+0000","last_active":"2024-03-04T09:27:17.335637+0000","last_peered":"2024-03-04T09:27:17.335637+0000","last_clean":"2024-03-04T09:27:17.335637+0000","last_became_active":"2024-03-04T09:27:17.330046+0000","last_became_peered":"2024-03-04T09:27:17.330046+0000","last_unstale":"2024-03-04T09:27:17.335637+0000","last_undegraded":"2024-03-04T09:27:17.335637+0000","last_fullsized":"2024-03-04T09:27:17.335637+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:54:25.631084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.322627+0000","last_change":"2024-03-04T09:27:17.318990+0000","last_active":"2024-03-04T09:27:17.322627+0000","last_peered":"2024-03-04T09:27:17.322627+0000","last_clean":"2024-03-04T09:27:17.322627+0000","last_became_active":"2024-03-04T09:27:17.315720+0000","last_became_peered":"2024-03-04T09:27:17.315720+0000","last_unstale":"2024-03-04T09:27:17.322627+0000","last_undegraded":"2024-03-04T09:27:17.322627+0000","last_fullsized":"2024-03-04T09:27:17.322627+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:19:51.542597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.328540+0000","last_change":"2024-03-04T09:27:17.321670+0000","last_active":"2024-03-04T09:27:17.328540+0000","last_peered":"2024-03-04T09:27:17.328540+0000","last_clean":"2024-03-04T09:27:17.328540+0000","last_became_active":"2024-03-04T09:27:17.321268+0000","last_became_peered":"2024-03-04T09:27:17.321268+0000","last_unstale":"2024-03-04T09:27:17.328540+0000","last_undegraded":"2024-03-04T09:27:17.328540+0000","last_fullsized":"2024-03-04T09:27:17.328540+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:49:53.452087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.344528+0000","last_change":"2024-03-04T09:27:17.334676+0000","last_active":"2024-03-04T09:27:17.344528+0000","last_peered":"2024-03-04T09:27:17.344528+0000","last_clean":"2024-03-04T09:27:17.344528+0000","last_became_active":"2024-03-04T09:27:17.334344+0000","last_became_peered":"2024-03-04T09:27:17.334344+0000","last_unstale":"2024-03-04T09:27:17.344528+0000","last_undegraded":"2024-03-04T09:27:17.344528+0000","last_fullsized":"2024-03-04T09:27:17.344528+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:19:15.352216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.337140+0000","last_change":"2024-03-04T09:27:17.326254+0000","last_active":"2024-03-04T09:27:17.337140+0000","last_peered":"2024-03-04T09:27:17.337140+0000","last_clean":"2024-03-04T09:27:17.337140+0000","last_became_active":"2024-03-04T09:27:17.325888+0000","last_became_peered":"2024-03-04T09:27:17.325888+0000","last_unstale":"2024-03-04T09:27:17.337140+0000","last_undegraded":"2024-03-04T09:27:17.337140+0000","last_fullsized":"2024-03-04T09:27:17.337140+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:58:45.796432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.314615+0000","last_change":"2024-03-04T09:27:17.307921+0000","last_active":"2024-03-04T09:27:17.314615+0000","last_peered":"2024-03-04T09:27:17.314615+0000","last_clean":"2024-03-04T09:27:17.314615+0000","last_became_active":"2024-03-04T09:27:17.304606+0000","last_became_peered":"2024-03-04T09:27:17.304606+0000","last_unstale":"2024-03-04T09:27:17.314615+0000","last_undegraded":"2024-03-04T09:27:17.314615+0000","last_fullsized":"2024-03-04T09:27:17.314615+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:39:23.162213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.320887+0000","last_change":"2024-03-04T09:27:17.317313+0000","last_active":"2024-03-04T09:27:17.320887+0000","last_peered":"2024-03-04T09:27:17.320887+0000","last_clean":"2024-03-04T09:27:17.320887+0000","last_became_active":"2024-03-04T09:27:17.316897+0000","last_became_peered":"2024-03-04T09:27:17.316897+0000","last_unstale":"2024-03-04T09:27:17.320887+0000","last_undegraded":"2024-03-04T09:27:17.320887+0000","last_fullsized":"2024-03-04T09:27:17.320887+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:58:50.020494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"43'4","reported_seq":16,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.349524+0000","last_change":"2024-03-04T09:27:16.332982+0000","last_active":"2024-03-04T09:27:17.349524+0000","last_peered":"2024-03-04T09:27:17.349524+0000","last_clean":"2024-03-04T09:27:17.349524+0000","last_became_active":"2024-03-04T09:27:16.332723+0000","last_became_peered":"2024-03-04T09:27:16.332723+0000","last_unstale":"2024-03-04T09:27:17.349524+0000","last_undegraded":"2024-03-04T09:27:17.349524+0000","last_fullsized":"2024-03-04T09:27:17.349524+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_clean_scrub_stamp":"2024-03-04T09:27:15.259708+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:14:24.494158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.335082+0000","last_change":"2024-03-04T09:27:17.328695+0000","last_active":"2024-03-04T09:27:17.335082+0000","last_peered":"2024-03-04T09:27:17.335082+0000","last_clean":"2024-03-04T09:27:17.335082+0000","last_became_active":"2024-03-04T09:27:17.326336+0000","last_became_peered":"2024-03-04T09:27:17.326336+0000","last_unstale":"2024-03-04T09:27:17.335082+0000","last_undegraded":"2024-03-04T09:27:17.335082+0000","last_fullsized":"2024-03-04T09:27:17.335082+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:13:37.135100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"43'1","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.347966+0000","last_change":"2024-03-04T09:27:16.331657+0000","last_active":"2024-03-04T09:27:17.347966+0000","last_peered":"2024-03-04T09:27:17.347966+0000","last_clean":"2024-03-04T09:27:17.347966+0000","last_became_active":"2024-03-04T09:27:16.330639+0000","last_became_peered":"2024-03-04T09:27:16.330639+0000","last_unstale":"2024-03-04T09:27:17.347966+0000","last_undegraded":"2024-03-04T09:27:17.347966+0000","last_fullsized":"2024-03-04T09:27:17.347966+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_clean_scrub_stamp":"2024-03-04T09:27:15.259708+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:33:16.393424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.341419+0000","last_change":"2024-03-04T09:27:17.327599+0000","last_active":"2024-03-04T09:27:17.341419+0000","last_peered":"2024-03-04T09:27:17.341419+0000","last_clean":"2024-03-04T09:27:17.341419+0000","last_became_active":"2024-03-04T09:27:17.327218+0000","last_became_peered":"2024-03-04T09:27:17.327218+0000","last_unstale":"2024-03-04T09:27:17.341419+0000","last_undegraded":"2024-03-04T09:27:17.341419+0000","last_fullsized":"2024-03-04T09:27:17.341419+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:51:45.022053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"43'2","reported_seq":14,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.345758+0000","last_change":"2024-03-04T09:27:16.315657+0000","last_active":"2024-03-04T09:27:17.345758+0000","last_peered":"2024-03-04T09:27:17.345758+0000","last_clean":"2024-03-04T09:27:17.345758+0000","last_became_active":"2024-03-04T09:27:16.315371+0000","last_became_peered":"2024-03-04T09:27:16.315371+0000","last_unstale":"2024-03-04T09:27:17.345758+0000","last_undegraded":"2024-03-04T09:27:17.345758+0000","last_fullsized":"2024-03-04T09:27:17.345758+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_clean_scrub_stamp":"2024-03-04T09:27:15.259708+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:02:03.266530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.341928+0000","last_change":"2024-03-04T09:27:17.333368+0000","last_active":"2024-03-04T09:27:17.341928+0000","last_peered":"2024-03-04T09:27:17.341928+0000","last_clean":"2024-03-04T09:27:17.341928+0000","last_became_active":"2024-03-04T09:27:17.332764+0000","last_became_peered":"2024-03-04T09:27:17.332764+0000","last_unstale":"2024-03-04T09:27:17.341928+0000","last_undegraded":"2024-03-04T09:27:17.341928+0000","last_fullsized":"2024-03-04T09:27:17.341928+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:15:09.249645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"43'2","reported_seq":14,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.345055+0000","last_change":"2024-03-04T09:27:16.325795+0000","last_active":"2024-03-04T09:27:17.345055+0000","last_peered":"2024-03-04T09:27:17.345055+0000","last_clean":"2024-03-04T09:27:17.345055+0000","last_became_active":"2024-03-04T09:27:16.324784+0000","last_became_peered":"2024-03-04T09:27:16.324784+0000","last_unstale":"2024-03-04T09:27:17.345055+0000","last_undegraded":"2024-03-04T09:27:17.345055+0000","last_fullsized":"2024-03-04T09:27:17.345055+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_clean_scrub_stamp":"2024-03-04T09:27:15.259708+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:54:42.666412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.340221+0000","last_change":"2024-03-04T09:27:17.333165+0000","last_active":"2024-03-04T09:27:17.340221+0000","last_peered":"2024-03-04T09:27:17.340221+0000","last_clean":"2024-03-04T09:27:17.340221+0000","last_became_active":"2024-03-04T09:27:17.332852+0000","last_became_peered":"2024-03-04T09:27:17.332852+0000","last_unstale":"2024-03-04T09:27:17.340221+0000","last_undegraded":"2024-03-04T09:27:17.340221+0000","last_fullsized":"2024-03-04T09:27:17.340221+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:54:43.681711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"43'1","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.349510+0000","last_change":"2024-03-04T09:27:16.327989+0000","last_active":"2024-03-04T09:27:17.349510+0000","last_peered":"2024-03-04T09:27:17.349510+0000","last_clean":"2024-03-04T09:27:17.349510+0000","last_became_active":"2024-03-04T09:27:16.327643+0000","last_became_peered":"2024-03-04T09:27:16.327643+0000","last_unstale":"2024-03-04T09:27:17.349510+0000","last_undegraded":"2024-03-04T09:27:17.349510+0000","last_fullsized":"2024-03-04T09:27:17.349510+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_clean_scrub_stamp":"2024-03-04T09:27:15.259708+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:18:10.222841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.326350+0000","last_change":"2024-03-04T09:27:17.321248+0000","last_active":"2024-03-04T09:27:17.326350+0000","last_peered":"2024-03-04T09:27:17.326350+0000","last_clean":"2024-03-04T09:27:17.326350+0000","last_became_active":"2024-03-04T09:27:17.320948+0000","last_became_peered":"2024-03-04T09:27:17.320948+0000","last_unstale":"2024-03-04T09:27:17.326350+0000","last_undegraded":"2024-03-04T09:27:17.326350+0000","last_fullsized":"2024-03-04T09:27:17.326350+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:16:31.292858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"43'1","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.347203+0000","last_change":"2024-03-04T09:27:16.321759+0000","last_active":"2024-03-04T09:27:17.347203+0000","last_peered":"2024-03-04T09:27:17.347203+0000","last_clean":"2024-03-04T09:27:17.347203+0000","last_became_active":"2024-03-04T09:27:16.321404+0000","last_became_peered":"2024-03-04T09:27:16.321404+0000","last_unstale":"2024-03-04T09:27:17.347203+0000","last_undegraded":"2024-03-04T09:27:17.347203+0000","last_fullsized":"2024-03-04T09:27:17.347203+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_clean_scrub_stamp":"2024-03-04T09:27:15.259708+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:02:03.266530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.336928+0000","last_change":"2024-03-04T09:27:17.329840+0000","last_active":"2024-03-04T09:27:17.336928+0000","last_peered":"2024-03-04T09:27:17.336928+0000","last_clean":"2024-03-04T09:27:17.336928+0000","last_became_active":"2024-03-04T09:27:17.329534+0000","last_became_peered":"2024-03-04T09:27:17.329534+0000","last_unstale":"2024-03-04T09:27:17.336928+0000","last_undegraded":"2024-03-04T09:27:17.336928+0000","last_fullsized":"2024-03-04T09:27:17.336928+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:20:51.879579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.299455+0000","last_change":"2024-03-04T09:27:16.312143+0000","last_active":"2024-03-04T09:27:17.299455+0000","last_peered":"2024-03-04T09:27:17.299455+0000","last_clean":"2024-03-04T09:27:17.299455+0000","last_became_active":"2024-03-04T09:27:16.310572+0000","last_became_peered":"2024-03-04T09:27:16.310572+0000","last_unstale":"2024-03-04T09:27:17.299455+0000","last_undegraded":"2024-03-04T09:27:17.299455+0000","last_fullsized":"2024-03-04T09:27:17.299455+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_clean_scrub_stamp":"2024-03-04T09:27:15.259708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:19:14.336917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.330406+0000","last_change":"2024-03-04T09:27:17.324625+0000","last_active":"2024-03-04T09:27:17.330406+0000","last_peered":"2024-03-04T09:27:17.330406+0000","last_clean":"2024-03-04T09:27:17.330406+0000","last_became_active":"2024-03-04T09:27:17.323600+0000","last_became_peered":"2024-03-04T09:27:17.323600+0000","last_unstale":"2024-03-04T09:27:17.330406+0000","last_undegraded":"2024-03-04T09:27:17.330406+0000","last_fullsized":"2024-03-04T09:27:17.330406+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:04:58.199976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.315287+0000","last_change":"2024-03-04T09:27:16.321082+0000","last_active":"2024-03-04T09:27:17.315287+0000","last_peered":"2024-03-04T09:27:17.315287+0000","last_clean":"2024-03-04T09:27:17.315287+0000","last_became_active":"2024-03-04T09:27:16.320792+0000","last_became_peered":"2024-03-04T09:27:16.320792+0000","last_unstale":"2024-03-04T09:27:17.315287+0000","last_undegraded":"2024-03-04T09:27:17.315287+0000","last_fullsized":"2024-03-04T09:27:17.315287+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_clean_scrub_stamp":"2024-03-04T09:27:15.259708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:15:25.086157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.335039+0000","last_change":"2024-03-04T09:27:17.327337+0000","last_active":"2024-03-04T09:27:17.335039+0000","last_peered":"2024-03-04T09:27:17.335039+0000","last_clean":"2024-03-04T09:27:17.335039+0000","last_became_active":"2024-03-04T09:27:17.326769+0000","last_became_peered":"2024-03-04T09:27:17.326769+0000","last_unstale":"2024-03-04T09:27:17.335039+0000","last_undegraded":"2024-03-04T09:27:17.335039+0000","last_fullsized":"2024-03-04T09:27:17.335039+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:15:09.249645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.317989+0000","last_change":"2024-03-04T09:27:16.323682+0000","last_active":"2024-03-04T09:27:17.317989+0000","last_peered":"2024-03-04T09:27:17.317989+0000","last_clean":"2024-03-04T09:27:17.317989+0000","last_became_active":"2024-03-04T09:27:16.323065+0000","last_became_peered":"2024-03-04T09:27:16.323065+0000","last_unstale":"2024-03-04T09:27:17.317989+0000","last_undegraded":"2024-03-04T09:27:17.317989+0000","last_fullsized":"2024-03-04T09:27:17.317989+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_clean_scrub_stamp":"2024-03-04T09:27:15.259708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:24:55.129882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.323880+0000","last_change":"2024-03-04T09:27:17.319590+0000","last_active":"2024-03-04T09:27:17.323880+0000","last_peered":"2024-03-04T09:27:17.323880+0000","last_clean":"2024-03-04T09:27:17.323880+0000","last_became_active":"2024-03-04T09:27:17.319285+0000","last_became_peered":"2024-03-04T09:27:17.319285+0000","last_unstale":"2024-03-04T09:27:17.323880+0000","last_undegraded":"2024-03-04T09:27:17.323880+0000","last_fullsized":"2024-03-04T09:27:17.323880+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:19:51.542597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.316355+0000","last_change":"2024-03-04T09:27:16.326483+0000","last_active":"2024-03-04T09:27:17.316355+0000","last_peered":"2024-03-04T09:27:17.316355+0000","last_clean":"2024-03-04T09:27:17.316355+0000","last_became_active":"2024-03-04T09:27:16.326194+0000","last_became_peered":"2024-03-04T09:27:16.326194+0000","last_unstale":"2024-03-04T09:27:17.316355+0000","last_undegraded":"2024-03-04T09:27:17.316355+0000","last_fullsized":"2024-03-04T09:27:17.316355+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_clean_scrub_stamp":"2024-03-04T09:27:15.259708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:18:24.323145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.326894+0000","last_change":"2024-03-04T09:27:17.322049+0000","last_active":"2024-03-04T09:27:17.326894+0000","last_peered":"2024-03-04T09:27:17.326894+0000","last_clean":"2024-03-04T09:27:17.326894+0000","last_became_active":"2024-03-04T09:27:17.321744+0000","last_became_peered":"2024-03-04T09:27:17.321744+0000","last_unstale":"2024-03-04T09:27:17.326894+0000","last_undegraded":"2024-03-04T09:27:17.326894+0000","last_fullsized":"2024-03-04T09:27:17.326894+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:04:08.085457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"43'3","reported_seq":15,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.347219+0000","last_change":"2024-03-04T09:27:16.333306+0000","last_active":"2024-03-04T09:27:17.347219+0000","last_peered":"2024-03-04T09:27:17.347219+0000","last_clean":"2024-03-04T09:27:17.347219+0000","last_became_active":"2024-03-04T09:27:16.332789+0000","last_became_peered":"2024-03-04T09:27:16.332789+0000","last_unstale":"2024-03-04T09:27:17.347219+0000","last_undegraded":"2024-03-04T09:27:17.347219+0000","last_fullsized":"2024-03-04T09:27:17.347219+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_clean_scrub_stamp":"2024-03-04T09:27:15.259708+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:19:14.336917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.332566+0000","last_change":"2024-03-04T09:27:17.328040+0000","last_active":"2024-03-04T09:27:17.332566+0000","last_peered":"2024-03-04T09:27:17.332566+0000","last_clean":"2024-03-04T09:27:17.332566+0000","last_became_active":"2024-03-04T09:27:17.327725+0000","last_became_peered":"2024-03-04T09:27:17.327725+0000","last_unstale":"2024-03-04T09:27:17.332566+0000","last_undegraded":"2024-03-04T09:27:17.332566+0000","last_fullsized":"2024-03-04T09:27:17.332566+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:51:45.022053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"43'6","reported_seq":18,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.351449+0000","last_change":"2024-03-04T09:27:16.321906+0000","last_active":"2024-03-04T09:27:17.351449+0000","last_peered":"2024-03-04T09:27:17.351449+0000","last_clean":"2024-03-04T09:27:17.351449+0000","last_became_active":"2024-03-04T09:27:16.321626+0000","last_became_peered":"2024-03-04T09:27:16.321626+0000","last_unstale":"2024-03-04T09:27:17.351449+0000","last_undegraded":"2024-03-04T09:27:17.351449+0000","last_fullsized":"2024-03-04T09:27:17.351449+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_clean_scrub_stamp":"2024-03-04T09:27:15.259708+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:03:33.968571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.335657+0000","last_change":"2024-03-04T09:27:17.332145+0000","last_active":"2024-03-04T09:27:17.335657+0000","last_peered":"2024-03-04T09:27:17.335657+0000","last_clean":"2024-03-04T09:27:17.335657+0000","last_became_active":"2024-03-04T09:27:17.331839+0000","last_became_peered":"2024-03-04T09:27:17.331839+0000","last_unstale":"2024-03-04T09:27:17.335657+0000","last_undegraded":"2024-03-04T09:27:17.335657+0000","last_fullsized":"2024-03-04T09:27:17.335657+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:16:58.851007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"43'2","reported_seq":14,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.350348+0000","last_change":"2024-03-04T09:27:16.324507+0000","last_active":"2024-03-04T09:27:17.350348+0000","last_peered":"2024-03-04T09:27:17.350348+0000","last_clean":"2024-03-04T09:27:17.350348+0000","last_became_active":"2024-03-04T09:27:16.324223+0000","last_became_peered":"2024-03-04T09:27:16.324223+0000","last_unstale":"2024-03-04T09:27:17.350348+0000","last_undegraded":"2024-03-04T09:27:17.350348+0000","last_fullsized":"2024-03-04T09:27:17.350348+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_clean_scrub_stamp":"2024-03-04T09:27:15.259708+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:22:00.830961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.325673+0000","last_change":"2024-03-04T09:27:17.318724+0000","last_active":"2024-03-04T09:27:17.325673+0000","last_peered":"2024-03-04T09:27:17.325673+0000","last_clean":"2024-03-04T09:27:17.325673+0000","last_became_active":"2024-03-04T09:27:17.318449+0000","last_became_peered":"2024-03-04T09:27:17.318449+0000","last_unstale":"2024-03-04T09:27:17.325673+0000","last_undegraded":"2024-03-04T09:27:17.325673+0000","last_fullsized":"2024-03-04T09:27:17.325673+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:23:31.196709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.308435+0000","last_change":"2024-03-04T09:27:16.325760+0000","last_active":"2024-03-04T09:27:17.308435+0000","last_peered":"2024-03-04T09:27:17.308435+0000","last_clean":"2024-03-04T09:27:17.308435+0000","last_became_active":"2024-03-04T09:27:16.324504+0000","last_became_peered":"2024-03-04T09:27:16.324504+0000","last_unstale":"2024-03-04T09:27:17.308435+0000","last_undegraded":"2024-03-04T09:27:17.308435+0000","last_fullsized":"2024-03-04T09:27:17.308435+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_clean_scrub_stamp":"2024-03-04T09:27:15.259708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:49:20.888193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.322578+0000","last_change":"2024-03-04T09:27:17.316352+0000","last_active":"2024-03-04T09:27:17.322578+0000","last_peered":"2024-03-04T09:27:17.322578+0000","last_clean":"2024-03-04T09:27:17.322578+0000","last_became_active":"2024-03-04T09:27:17.316057+0000","last_became_peered":"2024-03-04T09:27:17.316057+0000","last_unstale":"2024-03-04T09:27:17.322578+0000","last_undegraded":"2024-03-04T09:27:17.322578+0000","last_fullsized":"2024-03-04T09:27:17.322578+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:37:24.737020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.302740+0000","last_change":"2024-03-04T09:27:16.318271+0000","last_active":"2024-03-04T09:27:17.302740+0000","last_peered":"2024-03-04T09:27:17.302740+0000","last_clean":"2024-03-04T09:27:17.302740+0000","last_became_active":"2024-03-04T09:27:16.317497+0000","last_became_peered":"2024-03-04T09:27:16.317497+0000","last_unstale":"2024-03-04T09:27:17.302740+0000","last_undegraded":"2024-03-04T09:27:17.302740+0000","last_fullsized":"2024-03-04T09:27:17.302740+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_clean_scrub_stamp":"2024-03-04T09:27:15.259708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:49:13.895272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.332487+0000","last_change":"2024-03-04T09:27:17.328279+0000","last_active":"2024-03-04T09:27:17.332487+0000","last_peered":"2024-03-04T09:27:17.332487+0000","last_clean":"2024-03-04T09:27:17.332487+0000","last_became_active":"2024-03-04T09:27:17.327970+0000","last_became_peered":"2024-03-04T09:27:17.327970+0000","last_unstale":"2024-03-04T09:27:17.332487+0000","last_undegraded":"2024-03-04T09:27:17.332487+0000","last_fullsized":"2024-03-04T09:27:17.332487+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:49:14.910571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"43'8","reported_seq":20,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.350229+0000","last_change":"2024-03-04T09:27:16.324555+0000","last_active":"2024-03-04T09:27:17.350229+0000","last_peered":"2024-03-04T09:27:17.350229+0000","last_clean":"2024-03-04T09:27:17.350229+0000","last_became_active":"2024-03-04T09:27:16.323807+0000","last_became_peered":"2024-03-04T09:27:16.323807+0000","last_unstale":"2024-03-04T09:27:17.350229+0000","last_undegraded":"2024-03-04T09:27:17.350229+0000","last_fullsized":"2024-03-04T09:27:17.350229+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_clean_scrub_stamp":"2024-03-04T09:27:15.259708+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:13:36.119801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.324313+0000","last_change":"2024-03-04T09:27:17.319575+0000","last_active":"2024-03-04T09:27:17.324313+0000","last_peered":"2024-03-04T09:27:17.324313+0000","last_clean":"2024-03-04T09:27:17.324313+0000","last_became_active":"2024-03-04T09:27:17.319261+0000","last_became_peered":"2024-03-04T09:27:17.319261+0000","last_unstale":"2024-03-04T09:27:17.324313+0000","last_undegraded":"2024-03-04T09:27:17.324313+0000","last_fullsized":"2024-03-04T09:27:17.324313+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:59:09.754483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"43'1","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.341679+0000","last_change":"2024-03-04T09:27:16.326539+0000","last_active":"2024-03-04T09:27:17.341679+0000","last_peered":"2024-03-04T09:27:17.341679+0000","last_clean":"2024-03-04T09:27:17.341679+0000","last_became_active":"2024-03-04T09:27:16.326306+0000","last_became_peered":"2024-03-04T09:27:16.326306+0000","last_unstale":"2024-03-04T09:27:17.341679+0000","last_undegraded":"2024-03-04T09:27:17.341679+0000","last_fullsized":"2024-03-04T09:27:17.341679+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_clean_scrub_stamp":"2024-03-04T09:27:15.259708+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:22:54.262504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.336855+0000","last_change":"2024-03-04T09:27:17.329575+0000","last_active":"2024-03-04T09:27:17.336855+0000","last_peered":"2024-03-04T09:27:17.336855+0000","last_clean":"2024-03-04T09:27:17.336855+0000","last_became_active":"2024-03-04T09:27:17.329299+0000","last_became_peered":"2024-03-04T09:27:17.329299+0000","last_unstale":"2024-03-04T09:27:17.336855+0000","last_undegraded":"2024-03-04T09:27:17.336855+0000","last_fullsized":"2024-03-04T09:27:17.336855+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:15:09.249645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"43'2","reported_seq":14,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.347383+0000","last_change":"2024-03-04T09:27:16.320968+0000","last_active":"2024-03-04T09:27:17.347383+0000","last_peered":"2024-03-04T09:27:17.347383+0000","last_clean":"2024-03-04T09:27:17.347383+0000","last_became_active":"2024-03-04T09:27:16.319795+0000","last_became_peered":"2024-03-04T09:27:16.319795+0000","last_unstale":"2024-03-04T09:27:17.347383+0000","last_undegraded":"2024-03-04T09:27:17.347383+0000","last_fullsized":"2024-03-04T09:27:17.347383+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_clean_scrub_stamp":"2024-03-04T09:27:15.259708+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:44:09.974493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.330527+0000","last_change":"2024-03-04T09:27:17.325848+0000","last_active":"2024-03-04T09:27:17.330527+0000","last_peered":"2024-03-04T09:27:17.330527+0000","last_clean":"2024-03-04T09:27:17.330527+0000","last_became_active":"2024-03-04T09:27:17.325531+0000","last_became_peered":"2024-03-04T09:27:17.325531+0000","last_unstale":"2024-03-04T09:27:17.330527+0000","last_undegraded":"2024-03-04T09:27:17.330527+0000","last_fullsized":"2024-03-04T09:27:17.330527+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:49:56.762411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"43'1","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.344706+0000","last_change":"2024-03-04T09:27:16.320677+0000","last_active":"2024-03-04T09:27:17.344706+0000","last_peered":"2024-03-04T09:27:17.344706+0000","last_clean":"2024-03-04T09:27:17.344706+0000","last_became_active":"2024-03-04T09:27:16.320322+0000","last_became_peered":"2024-03-04T09:27:16.320322+0000","last_unstale":"2024-03-04T09:27:17.344706+0000","last_undegraded":"2024-03-04T09:27:17.344706+0000","last_fullsized":"2024-03-04T09:27:17.344706+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_clean_scrub_stamp":"2024-03-04T09:27:15.259708+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:07:43.640071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.326990+0000","last_change":"2024-03-04T09:27:17.322747+0000","last_active":"2024-03-04T09:27:17.326990+0000","last_peered":"2024-03-04T09:27:17.326990+0000","last_clean":"2024-03-04T09:27:17.326990+0000","last_became_active":"2024-03-04T09:27:17.322303+0000","last_became_peered":"2024-03-04T09:27:17.322303+0000","last_unstale":"2024-03-04T09:27:17.326990+0000","last_undegraded":"2024-03-04T09:27:17.326990+0000","last_fullsized":"2024-03-04T09:27:17.326990+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:58:19.947417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.305333+0000","last_change":"2024-03-04T09:27:16.329803+0000","last_active":"2024-03-04T09:27:17.305333+0000","last_peered":"2024-03-04T09:27:17.305333+0000","last_clean":"2024-03-04T09:27:17.305333+0000","last_became_active":"2024-03-04T09:27:16.329488+0000","last_became_peered":"2024-03-04T09:27:16.329488+0000","last_unstale":"2024-03-04T09:27:17.305333+0000","last_undegraded":"2024-03-04T09:27:17.305333+0000","last_fullsized":"2024-03-04T09:27:17.305333+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_clean_scrub_stamp":"2024-03-04T09:27:15.259708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:59:42.755951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.325938+0000","last_change":"2024-03-04T09:27:17.321454+0000","last_active":"2024-03-04T09:27:17.325938+0000","last_peered":"2024-03-04T09:27:17.325938+0000","last_clean":"2024-03-04T09:27:17.325938+0000","last_became_active":"2024-03-04T09:27:17.321180+0000","last_became_peered":"2024-03-04T09:27:17.321180+0000","last_unstale":"2024-03-04T09:27:17.325938+0000","last_undegraded":"2024-03-04T09:27:17.325938+0000","last_fullsized":"2024-03-04T09:27:17.325938+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:46:04.631344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"43'2","reported_seq":14,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.349611+0000","last_change":"2024-03-04T09:27:16.327564+0000","last_active":"2024-03-04T09:27:17.349611+0000","last_peered":"2024-03-04T09:27:17.349611+0000","last_clean":"2024-03-04T09:27:17.349611+0000","last_became_active":"2024-03-04T09:27:16.326311+0000","last_became_peered":"2024-03-04T09:27:16.326311+0000","last_unstale":"2024-03-04T09:27:17.349611+0000","last_undegraded":"2024-03-04T09:27:17.349611+0000","last_fullsized":"2024-03-04T09:27:17.349611+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_clean_scrub_stamp":"2024-03-04T09:27:15.259708+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:51:44.006754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.321351+0000","last_change":"2024-03-04T09:27:17.317110+0000","last_active":"2024-03-04T09:27:17.321351+0000","last_peered":"2024-03-04T09:27:17.321351+0000","last_clean":"2024-03-04T09:27:17.321351+0000","last_became_active":"2024-03-04T09:27:17.316782+0000","last_became_peered":"2024-03-04T09:27:17.316782+0000","last_unstale":"2024-03-04T09:27:17.321351+0000","last_undegraded":"2024-03-04T09:27:17.321351+0000","last_fullsized":"2024-03-04T09:27:17.321351+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:24:56.145181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.299135+0000","last_change":"2024-03-04T09:27:16.329888+0000","last_active":"2024-03-04T09:27:17.299135+0000","last_peered":"2024-03-04T09:27:17.299135+0000","last_clean":"2024-03-04T09:27:17.299135+0000","last_became_active":"2024-03-04T09:27:16.329577+0000","last_became_peered":"2024-03-04T09:27:16.329577+0000","last_unstale":"2024-03-04T09:27:17.299135+0000","last_undegraded":"2024-03-04T09:27:17.299135+0000","last_fullsized":"2024-03-04T09:27:17.299135+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_clean_scrub_stamp":"2024-03-04T09:27:15.259708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:54:42.666412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.321278+0000","last_change":"2024-03-04T09:27:17.318851+0000","last_active":"2024-03-04T09:27:17.321278+0000","last_peered":"2024-03-04T09:27:17.321278+0000","last_clean":"2024-03-04T09:27:17.321278+0000","last_became_active":"2024-03-04T09:27:17.318468+0000","last_became_peered":"2024-03-04T09:27:17.318468+0000","last_unstale":"2024-03-04T09:27:17.321278+0000","last_undegraded":"2024-03-04T09:27:17.321278+0000","last_fullsized":"2024-03-04T09:27:17.321278+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:49:53.452087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.316376+0000","last_change":"2024-03-04T09:27:16.328226+0000","last_active":"2024-03-04T09:27:17.316376+0000","last_peered":"2024-03-04T09:27:17.316376+0000","last_clean":"2024-03-04T09:27:17.316376+0000","last_became_active":"2024-03-04T09:27:16.327149+0000","last_became_peered":"2024-03-04T09:27:16.327149+0000","last_unstale":"2024-03-04T09:27:17.316376+0000","last_undegraded":"2024-03-04T09:27:17.316376+0000","last_fullsized":"2024-03-04T09:27:17.316376+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_clean_scrub_stamp":"2024-03-04T09:27:15.259708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:33:55.291079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.332805+0000","last_change":"2024-03-04T09:27:17.323402+0000","last_active":"2024-03-04T09:27:17.332805+0000","last_peered":"2024-03-04T09:27:17.332805+0000","last_clean":"2024-03-04T09:27:17.332805+0000","last_became_active":"2024-03-04T09:27:17.322227+0000","last_became_peered":"2024-03-04T09:27:17.322227+0000","last_unstale":"2024-03-04T09:27:17.332805+0000","last_undegraded":"2024-03-04T09:27:17.332805+0000","last_fullsized":"2024-03-04T09:27:17.332805+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:44:10.989793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.308709+0000","last_change":"2024-03-04T09:27:16.326171+0000","last_active":"2024-03-04T09:27:17.308709+0000","last_peered":"2024-03-04T09:27:17.308709+0000","last_clean":"2024-03-04T09:27:17.308709+0000","last_became_active":"2024-03-04T09:27:16.325873+0000","last_became_peered":"2024-03-04T09:27:16.325873+0000","last_unstale":"2024-03-04T09:27:17.308709+0000","last_undegraded":"2024-03-04T09:27:17.308709+0000","last_fullsized":"2024-03-04T09:27:17.308709+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_clean_scrub_stamp":"2024-03-04T09:27:15.259708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:07:43.640071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.338553+0000","last_change":"2024-03-04T09:27:17.332391+0000","last_active":"2024-03-04T09:27:17.338553+0000","last_peered":"2024-03-04T09:27:17.338553+0000","last_clean":"2024-03-04T09:27:17.338553+0000","last_became_active":"2024-03-04T09:27:17.331771+0000","last_became_peered":"2024-03-04T09:27:17.331771+0000","last_unstale":"2024-03-04T09:27:17.338553+0000","last_undegraded":"2024-03-04T09:27:17.338553+0000","last_fullsized":"2024-03-04T09:27:17.338553+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:18:25.338444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.300727+0000","last_change":"2024-03-04T09:27:16.324697+0000","last_active":"2024-03-04T09:27:17.300727+0000","last_peered":"2024-03-04T09:27:17.300727+0000","last_clean":"2024-03-04T09:27:17.300727+0000","last_became_active":"2024-03-04T09:27:16.323098+0000","last_became_peered":"2024-03-04T09:27:16.323098+0000","last_unstale":"2024-03-04T09:27:17.300727+0000","last_undegraded":"2024-03-04T09:27:17.300727+0000","last_fullsized":"2024-03-04T09:27:17.300727+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_clean_scrub_stamp":"2024-03-04T09:27:15.259708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:19:50.527298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.332372+0000","last_change":"2024-03-04T09:27:17.326199+0000","last_active":"2024-03-04T09:27:17.332372+0000","last_peered":"2024-03-04T09:27:17.332372+0000","last_clean":"2024-03-04T09:27:17.332372+0000","last_became_active":"2024-03-04T09:27:17.326025+0000","last_became_peered":"2024-03-04T09:27:17.326025+0000","last_unstale":"2024-03-04T09:27:17.332372+0000","last_undegraded":"2024-03-04T09:27:17.332372+0000","last_fullsized":"2024-03-04T09:27:17.332372+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:24:56.145181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.309012+0000","last_change":"2024-03-04T09:27:16.333063+0000","last_active":"2024-03-04T09:27:17.309012+0000","last_peered":"2024-03-04T09:27:17.309012+0000","last_clean":"2024-03-04T09:27:17.309012+0000","last_became_active":"2024-03-04T09:27:16.332718+0000","last_became_peered":"2024-03-04T09:27:16.332718+0000","last_unstale":"2024-03-04T09:27:17.309012+0000","last_undegraded":"2024-03-04T09:27:17.309012+0000","last_fullsized":"2024-03-04T09:27:17.309012+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_clean_scrub_stamp":"2024-03-04T09:27:15.259708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:55:33.045780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.326682+0000","last_change":"2024-03-04T09:27:17.320429+0000","last_active":"2024-03-04T09:27:17.326682+0000","last_peered":"2024-03-04T09:27:17.326682+0000","last_clean":"2024-03-04T09:27:17.326682+0000","last_became_active":"2024-03-04T09:27:17.320141+0000","last_became_peered":"2024-03-04T09:27:17.320141+0000","last_unstale":"2024-03-04T09:27:17.326682+0000","last_undegraded":"2024-03-04T09:27:17.326682+0000","last_fullsized":"2024-03-04T09:27:17.326682+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:18:11.238140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.289289+0000","last_change":"2024-03-04T09:27:16.318313+0000","last_active":"2024-03-04T09:27:17.289289+0000","last_peered":"2024-03-04T09:27:17.289289+0000","last_clean":"2024-03-04T09:27:17.289289+0000","last_became_active":"2024-03-04T09:27:16.318008+0000","last_became_peered":"2024-03-04T09:27:16.318008+0000","last_unstale":"2024-03-04T09:27:17.289289+0000","last_undegraded":"2024-03-04T09:27:17.289289+0000","last_fullsized":"2024-03-04T09:27:17.289289+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_clean_scrub_stamp":"2024-03-04T09:27:15.259708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:44:09.974493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.329432+0000","last_change":"2024-03-04T09:27:17.324202+0000","last_active":"2024-03-04T09:27:17.329432+0000","last_peered":"2024-03-04T09:27:17.329432+0000","last_clean":"2024-03-04T09:27:17.329432+0000","last_became_active":"2024-03-04T09:27:17.323912+0000","last_became_peered":"2024-03-04T09:27:17.323912+0000","last_unstale":"2024-03-04T09:27:17.329432+0000","last_undegraded":"2024-03-04T09:27:17.329432+0000","last_fullsized":"2024-03-04T09:27:17.329432+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:52:44.662696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.311175+0000","last_change":"2024-03-04T09:27:16.330197+0000","last_active":"2024-03-04T09:27:17.311175+0000","last_peered":"2024-03-04T09:27:17.311175+0000","last_clean":"2024-03-04T09:27:17.311175+0000","last_became_active":"2024-03-04T09:27:16.329916+0000","last_became_peered":"2024-03-04T09:27:16.329916+0000","last_unstale":"2024-03-04T09:27:17.311175+0000","last_undegraded":"2024-03-04T09:27:17.311175+0000","last_fullsized":"2024-03-04T09:27:17.311175+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_clean_scrub_stamp":"2024-03-04T09:27:15.259708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:44:59.890990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.331080+0000","last_change":"2024-03-04T09:27:17.323907+0000","last_active":"2024-03-04T09:27:17.331080+0000","last_peered":"2024-03-04T09:27:17.331080+0000","last_clean":"2024-03-04T09:27:17.331080+0000","last_became_active":"2024-03-04T09:27:17.322745+0000","last_became_peered":"2024-03-04T09:27:17.322745+0000","last_unstale":"2024-03-04T09:27:17.331080+0000","last_undegraded":"2024-03-04T09:27:17.331080+0000","last_fullsized":"2024-03-04T09:27:17.331080+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:28:33.800438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.311885+0000","last_change":"2024-03-04T09:27:16.324235+0000","last_active":"2024-03-04T09:27:17.311885+0000","last_peered":"2024-03-04T09:27:17.311885+0000","last_clean":"2024-03-04T09:27:17.311885+0000","last_became_active":"2024-03-04T09:27:16.323926+0000","last_became_peered":"2024-03-04T09:27:16.323926+0000","last_unstale":"2024-03-04T09:27:17.311885+0000","last_undegraded":"2024-03-04T09:27:17.311885+0000","last_fullsized":"2024-03-04T09:27:17.311885+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_clean_scrub_stamp":"2024-03-04T09:27:15.259708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:44:14.792105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.319794+0000","last_change":"2024-03-04T09:27:17.313083+0000","last_active":"2024-03-04T09:27:17.319794+0000","last_peered":"2024-03-04T09:27:17.319794+0000","last_clean":"2024-03-04T09:27:17.319794+0000","last_became_active":"2024-03-04T09:27:17.312811+0000","last_became_peered":"2024-03-04T09:27:17.312811+0000","last_unstale":"2024-03-04T09:27:17.319794+0000","last_undegraded":"2024-03-04T09:27:17.319794+0000","last_fullsized":"2024-03-04T09:27:17.319794+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:55:34.061079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.293418+0000","last_change":"2024-03-04T09:27:16.330272+0000","last_active":"2024-03-04T09:27:17.293418+0000","last_peered":"2024-03-04T09:27:17.293418+0000","last_clean":"2024-03-04T09:27:17.293418+0000","last_became_active":"2024-03-04T09:27:16.329925+0000","last_became_peered":"2024-03-04T09:27:16.329925+0000","last_unstale":"2024-03-04T09:27:17.293418+0000","last_undegraded":"2024-03-04T09:27:17.293418+0000","last_fullsized":"2024-03-04T09:27:17.293418+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_clean_scrub_stamp":"2024-03-04T09:27:15.259708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:44:14.792105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.334506+0000","last_change":"2024-03-04T09:27:17.330022+0000","last_active":"2024-03-04T09:27:17.334506+0000","last_peered":"2024-03-04T09:27:17.334506+0000","last_clean":"2024-03-04T09:27:17.334506+0000","last_became_active":"2024-03-04T09:27:17.329631+0000","last_became_peered":"2024-03-04T09:27:17.329631+0000","last_unstale":"2024-03-04T09:27:17.334506+0000","last_undegraded":"2024-03-04T09:27:17.334506+0000","last_fullsized":"2024-03-04T09:27:17.334506+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:04:15.264485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.315636+0000","last_change":"2024-03-04T09:27:16.330827+0000","last_active":"2024-03-04T09:27:17.315636+0000","last_peered":"2024-03-04T09:27:17.315636+0000","last_clean":"2024-03-04T09:27:17.315636+0000","last_became_active":"2024-03-04T09:27:16.330459+0000","last_became_peered":"2024-03-04T09:27:16.330459+0000","last_unstale":"2024-03-04T09:27:17.315636+0000","last_undegraded":"2024-03-04T09:27:17.315636+0000","last_fullsized":"2024-03-04T09:27:17.315636+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_clean_scrub_stamp":"2024-03-04T09:27:15.259708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:16:57.835708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.332227+0000","last_change":"2024-03-04T09:27:17.326000+0000","last_active":"2024-03-04T09:27:17.332227+0000","last_peered":"2024-03-04T09:27:17.332227+0000","last_clean":"2024-03-04T09:27:17.332227+0000","last_became_active":"2024-03-04T09:27:17.325521+0000","last_became_peered":"2024-03-04T09:27:17.325521+0000","last_unstale":"2024-03-04T09:27:17.332227+0000","last_undegraded":"2024-03-04T09:27:17.332227+0000","last_fullsized":"2024-03-04T09:27:17.332227+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:16.275007+0000","last_clean_scrub_stamp":"2024-03-04T09:27:16.275007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:44:15.807404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"43'2","reported_seq":14,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.344120+0000","last_change":"2024-03-04T09:27:16.317358+0000","last_active":"2024-03-04T09:27:17.344120+0000","last_peered":"2024-03-04T09:27:17.344120+0000","last_clean":"2024-03-04T09:27:17.344120+0000","last_became_active":"2024-03-04T09:27:16.316970+0000","last_became_peered":"2024-03-04T09:27:16.316970+0000","last_unstale":"2024-03-04T09:27:17.344120+0000","last_undegraded":"2024-03-04T09:27:17.344120+0000","last_fullsized":"2024-03-04T09:27:17.344120+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:15.259708+0000","last_clean_scrub_stamp":"2024-03-04T09:27:15.259708+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:49:20.888193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":213,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T09:27:17.308841+0000","last_change":"2024-03-04T09:25:09.316159+0000","last_active":"2024-03-04T09:27:17.308841+0000","last_peered":"2024-03-04T09:27:17.308841+0000","last_clean":"2024-03-04T09:27:17.308841+0000","last_became_active":"2024-03-04T09:25:09.313767+0000","last_became_peered":"2024-03-04T09:25:09.313767+0000","last_unstale":"2024-03-04T09:27:17.308841+0000","last_undegraded":"2024-03-04T09:27:17.308841+0000","last_fullsized":"2024-03-04T09:27:17.308841+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:08.110953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:08.110953+0000","last_clean_scrub_stamp":"2024-03-04T09:25:08.110953+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:34:21.745390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574877,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32256,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339584,"statfs":{"total":96636764160,"available":96603734016,"internally_reserved":0,"allocated":1101824,"data_stored":794206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574877,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31664,"kb_used_data":480,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340176,"statfs":{"total":96636764160,"available":96604340224,"internally_reserved":0,"allocated":491520,"data_stored":196973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574879,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31676,"kb_used_data":496,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340164,"statfs":{"total":96636764160,"available":96604327936,"internally_reserved":0,"allocated":507904,"data_stored":203818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574876,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31664,"kb_used_data":480,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340176,"statfs":{"total":96636764160,"available":96604340224,"internally_reserved":0,"allocated":491520,"data_stored":196973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574878,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31664,"kb_used_data":480,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340176,"statfs":{"total":96636764160,"available":96604340224,"internally_reserved":0,"allocated":491520,"data_stored":196973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574877,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31664,"kb_used_data":480,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340176,"statfs":{"total":96636764160,"available":96604340224,"internally_reserved":0,"allocated":491520,"data_stored":196973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574878,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31684,"kb_used_data":496,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340156,"statfs":{"total":96636764160,"available":96604319744,"internally_reserved":0,"allocated":507904,"data_stored":203818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574877,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32256,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339584,"statfs":{"total":96636764160,"available":96603734016,"internally_reserved":0,"allocated":1101824,"data_stored":794150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-04T09:27:22.308 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-04T09:27:22.308 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-04T09:27:22.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.589+0000 7f58a2a00640 1 Processor -- start 2024-03-04T09:27:22.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.589+0000 7f58a2a00640 1 -- start start 2024-03-04T09:27:22.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.591+0000 7f58a2a00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f589c15e680 0x7f589c06f930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:22.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.591+0000 7f58a2a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f589c06fe70 0x7f589c168000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:22.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.591+0000 7f58a2a00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f589c06e0d0 con 0x7f589c1551c0 2024-03-04T09:27:22.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.591+0000 7f58a2a00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f589c0702c0 con 0x7f589c15e680 2024-03-04T09:27:22.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.591+0000 7f58a2a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f589c0b5da0 con 0x7f589c06fe70 2024-03-04T09:27:22.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.592+0000 7f5893400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f589c15e680 0x7f589c06f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:22.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.592+0000 7f58a0a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f589c06fe70 0x7f589c168000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:22.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.592+0000 7f5893400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f589c15e680 0x7f589c06f930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.6:35600/0 (socket says 172.21.15.6:35600) 2024-03-04T09:27:22.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.592+0000 7f58a0a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f589c06fe70 0x7f589c168000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.6:34556/0 (socket says 172.21.15.6:34556) 2024-03-04T09:27:22.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.592+0000 7f5893400640 1 -- 172.21.15.6:0/3108796197 learned_addr learned my addr 172.21.15.6:0/3108796197 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:27:22.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.593+0000 7f58a0a00640 1 -- 172.21.15.6:0/3108796197 >> v2:172.21.15.146:3300/0 conn(0x7f589c15e680 msgr2=0x7f589c06f930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:27:22.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.593+0000 7f58a0a00640 1 --2- 172.21.15.6:0/3108796197 >> v2:172.21.15.146:3300/0 conn(0x7f589c15e680 0x7f589c06f930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:22.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.593+0000 7f58a0a00640 1 -- 172.21.15.6:0/3108796197 >> v1:172.21.15.146:6789/0 conn(0x7f589c1551c0 legacy=0x7f589c0b5540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:22.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.593+0000 7f58a0a00640 1 -- 172.21.15.6:0/3108796197 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f589c0b6e20 con 0x7f589c06fe70 2024-03-04T09:27:22.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.594+0000 7f58a0a00640 1 --2- 172.21.15.6:0/3108796197 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f589c06fe70 0x7f589c168000 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f5894009d90 tx=0x7f589402beb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=afe68cf452c89dcc server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:22.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.595+0000 7f589b400640 1 -- 172.21.15.6:0/3108796197 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5894039070 con 0x7f589c06fe70 2024-03-04T09:27:22.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.595+0000 7f589b400640 1 -- 172.21.15.6:0/3108796197 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f589402cc60 con 0x7f589c06fe70 2024-03-04T09:27:22.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.595+0000 7f589b400640 1 -- 172.21.15.6:0/3108796197 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5894034660 con 0x7f589c06fe70 2024-03-04T09:27:22.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.596+0000 7f58a2a00640 1 -- 172.21.15.6:0/3108796197 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f589c06fe70 msgr2=0x7f589c168000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:22.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.596+0000 7f58a2a00640 1 --2- 172.21.15.6:0/3108796197 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f589c06fe70 0x7f589c168000 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f5894009d90 tx=0x7f589402beb0 comp rx=0 tx=0).stop 2024-03-04T09:27:22.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.596+0000 7f58a2a00640 1 -- 172.21.15.6:0/3108796197 shutdown_connections 2024-03-04T09:27:22.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.596+0000 7f58a2a00640 1 --2- 172.21.15.6:0/3108796197 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f589c06fe70 0x7f589c168000 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:22.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.596+0000 7f58a2a00640 1 --2- 172.21.15.6:0/3108796197 >> v2:172.21.15.146:3300/0 conn(0x7f589c15e680 0x7f589c06f930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:22.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.596+0000 7f58a2a00640 1 -- 172.21.15.6:0/3108796197 >> 172.21.15.6:0/3108796197 conn(0x7f589c09aa10 msgr2=0x7f589c09ae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:22.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.596+0000 7f58a2a00640 1 -- 172.21.15.6:0/3108796197 shutdown_connections 2024-03-04T09:27:22.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.596+0000 7f58a2a00640 1 -- 172.21.15.6:0/3108796197 wait complete. 2024-03-04T09:27:22.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.598+0000 7f58a2a00640 1 Processor -- start 2024-03-04T09:27:22.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.599+0000 7f58a2a00640 1 -- start start 2024-03-04T09:27:22.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.600+0000 7f58a2a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f589c06fe70 0x7f589c148ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:22.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.600+0000 7f58a2a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f589c1551c0 0x7f589c09bd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:22.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.600+0000 7f58a2a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f589c15e680 0x7f589c09c240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:22.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.600+0000 7f58a2a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f589c0b6620 con 0x7f589c15e680 2024-03-04T09:27:22.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.600+0000 7f58a2a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f589c0b6340 con 0x7f589c1551c0 2024-03-04T09:27:22.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.600+0000 7f58a2a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f589c0b6900 con 0x7f589c06fe70 2024-03-04T09:27:22.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.600+0000 7f58a0a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f589c15e680 0x7f589c09c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:22.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.600+0000 7f58a0a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f589c15e680 0x7f589c09c240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:50482/0 (socket says 172.21.15.6:50482) 2024-03-04T09:27:22.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.600+0000 7f58a0a00640 1 -- 172.21.15.6:0/1586559116 learned_addr learned my addr 172.21.15.6:0/1586559116 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:27:22.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.601+0000 7f589be00640 1 --2- 172.21.15.6:0/1586559116 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f589c06fe70 0x7f589c148ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:22.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.601+0000 7f5893400640 1 --2- 172.21.15.6:0/1586559116 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f589c1551c0 0x7f589c09bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:22.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.601+0000 7f58a0a00640 1 -- 172.21.15.6:0/1586559116 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f589c06fe70 msgr2=0x7f589c148ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:22.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.601+0000 7f58a0a00640 1 --2- 172.21.15.6:0/1586559116 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f589c06fe70 0x7f589c148ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:22.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.601+0000 7f58a0a00640 1 -- 172.21.15.6:0/1586559116 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f589c1551c0 msgr2=0x7f589c09bd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:22.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.601+0000 7f58a0a00640 1 --2- 172.21.15.6:0/1586559116 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f589c1551c0 0x7f589c09bd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:22.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.601+0000 7f58a0a00640 1 -- 172.21.15.6:0/1586559116 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f589c09eda0 con 0x7f589c15e680 2024-03-04T09:27:22.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.601+0000 7f58a0a00640 1 --2- 172.21.15.6:0/1586559116 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f589c15e680 0x7f589c09c240 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f5894009d90 tx=0x7f5894004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:22.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.601+0000 7f5898c00640 1 -- 172.21.15.6:0/1586559116 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5894039050 con 0x7f589c15e680 2024-03-04T09:27:22.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.602+0000 7f5898c00640 1 -- 172.21.15.6:0/1586559116 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5894045070 con 0x7f589c15e680 2024-03-04T09:27:22.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.602+0000 7f5898c00640 1 -- 172.21.15.6:0/1586559116 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5894009450 con 0x7f589c15e680 2024-03-04T09:27:22.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.602+0000 7f58a2a00640 1 -- 172.21.15.6:0/1586559116 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f589c09f080 con 0x7f589c15e680 2024-03-04T09:27:22.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.602+0000 7f58a2a00640 1 -- 172.21.15.6:0/1586559116 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f589c09f640 con 0x7f589c15e680 2024-03-04T09:27:22.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.603+0000 7f58a2a00640 1 -- 172.21.15.6:0/1586559116 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f589c145fd0 con 0x7f589c15e680 2024-03-04T09:27:22.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.604+0000 7f5898c00640 1 -- 172.21.15.6:0/1586559116 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f589403e850 con 0x7f589c15e680 2024-03-04T09:27:22.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.605+0000 7f5898c00640 1 --2- 172.21.15.6:0/1586559116 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f587806b4d0 0x7f587806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:22.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.605+0000 7f5898c00640 1 -- 172.21.15.6:0/1586559116 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f58940ad0b0 con 0x7f589c15e680 2024-03-04T09:27:22.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.606+0000 7f589be00640 1 --2- 172.21.15.6:0/1586559116 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f587806b4d0 0x7f587806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:22.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.607+0000 7f589be00640 1 --2- 172.21.15.6:0/1586559116 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f587806b4d0 0x7f587806d990 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f588c007fd0 tx=0x7f588c009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:22.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.610+0000 7f5898c00640 1 -- 172.21.15.6:0/1586559116 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f589c145fd0 con 0x7f589c15e680 2024-03-04T09:27:22.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.884+0000 7f58a2a00640 1 -- 172.21.15.6:0/1586559116 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f589c068010 con 0x7f589c15e680 2024-03-04T09:27:22.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.888+0000 7f5898c00640 1 -- 172.21.15.6:0/1586559116 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v11) v1 ==== 190+0+147 (secure 0 0 0) 0x7f589407c040 con 0x7f589c15e680 2024-03-04T09:27:22.885 INFO:teuthology.orchestra.run.smithi006.stdout:[client.0] 2024-03-04T09:27:22.886 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQA9k+Vl4L+rJxAACaDbQ4/Bybp5hAZAnJzlVw== 2024-03-04T09:27:22.886 INFO:teuthology.orchestra.run.smithi006.stdout: caps mds = "allow" 2024-03-04T09:27:22.886 INFO:teuthology.orchestra.run.smithi006.stdout: caps mon = "allow r" 2024-03-04T09:27:22.886 INFO:teuthology.orchestra.run.smithi006.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-04T09:27:22.886 INFO:teuthology.orchestra.run.smithi006.stderr:updated caps for client.0 2024-03-04T09:27:22.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.891+0000 7f58a2a00640 1 -- 172.21.15.6:0/1586559116 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f587806b4d0 msgr2=0x7f587806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:22.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.891+0000 7f58a2a00640 1 --2- 172.21.15.6:0/1586559116 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f587806b4d0 0x7f587806d990 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f588c007fd0 tx=0x7f588c009040 comp rx=0 tx=0).stop 2024-03-04T09:27:22.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.891+0000 7f58a2a00640 1 -- 172.21.15.6:0/1586559116 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f589c15e680 msgr2=0x7f589c09c240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:22.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.891+0000 7f58a2a00640 1 --2- 172.21.15.6:0/1586559116 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f589c15e680 0x7f589c09c240 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f5894009d90 tx=0x7f5894004a40 comp rx=0 tx=0).stop 2024-03-04T09:27:22.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.891+0000 7f58a2a00640 1 -- 172.21.15.6:0/1586559116 shutdown_connections 2024-03-04T09:27:22.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.891+0000 7f58a2a00640 1 --2- 172.21.15.6:0/1586559116 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f589c15e680 0x7f589c09c240 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:22.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.891+0000 7f58a2a00640 1 --2- 172.21.15.6:0/1586559116 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f589c1551c0 0x7f589c09bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:22.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.891+0000 7f58a2a00640 1 --2- 172.21.15.6:0/1586559116 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f587806b4d0 0x7f587806d990 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:22.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.891+0000 7f58a2a00640 1 --2- 172.21.15.6:0/1586559116 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f589c06fe70 0x7f589c148ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:22.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.891+0000 7f58a2a00640 1 -- 172.21.15.6:0/1586559116 >> 172.21.15.6:0/1586559116 conn(0x7f589c09aa10 msgr2=0x7f589c09adf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:22.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.892+0000 7f58a2a00640 1 -- 172.21.15.6:0/1586559116 shutdown_connections 2024-03-04T09:27:22.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:22.892+0000 7f58a2a00640 1 -- 172.21.15.6:0/1586559116 wait complete. 2024-03-04T09:27:22.909 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:27:23.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:23.134+0000 7ff851600640 1 Processor -- start 2024-03-04T09:27:23.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:23.134+0000 7ff851600640 1 -- start start 2024-03-04T09:27:23.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:23.136+0000 7ff851600640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ff84c070f10 0x7ff84c0cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:23.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:23.136+0000 7ff851600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff84c0cce90 0x7ff84c15c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:23.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:23.136+0000 7ff851600640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff84c0714d0 con 0x7ff84c145300 2024-03-04T09:27:23.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:23.136+0000 7ff851600640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff84c0cd300 con 0x7ff84c0cce90 2024-03-04T09:27:23.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:23.136+0000 7ff851600640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff84c0717b0 con 0x7ff84c070f10 2024-03-04T09:27:23.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:23.136+0000 7ff850c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff84c0cce90 0x7ff84c15c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:23.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:23.136+0000 7ff850c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff84c0cce90 0x7ff84c15c570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:50488/0 (socket says 172.21.15.6:50488) 2024-03-04T09:27:23.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:23.136+0000 7ff850c00640 1 -- 172.21.15.6:0/439766314 learned_addr learned my addr 172.21.15.6:0/439766314 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:27:23.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:23.137+0000 7ff84b400640 1 --2- 172.21.15.6:0/439766314 >> v2:172.21.15.146:3300/0 conn(0x7ff84c070f10 0x7ff84c0cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:23.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:23.137+0000 7ff850c00640 1 -- 172.21.15.6:0/439766314 >> v2:172.21.15.146:3300/0 conn(0x7ff84c070f10 msgr2=0x7ff84c0cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:27:23.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:23.137+0000 7ff850c00640 1 --2- 172.21.15.6:0/439766314 >> v2:172.21.15.146:3300/0 conn(0x7ff84c070f10 0x7ff84c0cc950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:23.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:23.137+0000 7ff850c00640 1 -- 172.21.15.6:0/439766314 >> v1:172.21.15.146:6789/0 conn(0x7ff84c145300 legacy=0x7ff84c0b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:27:23.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:23.137+0000 7ff850c00640 1 -- 172.21.15.6:0/439766314 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff84c071a90 con 0x7ff84c0cce90 2024-03-04T09:27:23.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:23.138+0000 7ff850c00640 1 --2- 172.21.15.6:0/439766314 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff84c0cce90 0x7ff84c15c570 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7ff84c0b5150 tx=0x7ff84402be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a6437b8497c9617 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:23.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:23.138+0000 7ff84aa00640 1 -- 172.21.15.6:0/439766314 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff84402c9f0 con 0x7ff84c0cce90 2024-03-04T09:27:23.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:23.138+0000 7ff84aa00640 1 -- 172.21.15.6:0/439766314 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff84402cb90 con 0x7ff84c0cce90 2024-03-04T09:27:23.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:23.138+0000 7ff84aa00640 1 -- 172.21.15.6:0/439766314 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8440343f0 con 0x7ff84c0cce90 2024-03-04T09:27:23.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:23.139+0000 7ff851600640 1 -- 172.21.15.6:0/439766314 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff84c0cce90 msgr2=0x7ff84c15c570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:23.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:23.139+0000 7ff851600640 1 --2- 172.21.15.6:0/439766314 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff84c0cce90 0x7ff84c15c570 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7ff84c0b5150 tx=0x7ff84402be50 comp rx=0 tx=0).stop 2024-03-04T09:27:23.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:23.140+0000 7ff851600640 1 -- 172.21.15.6:0/439766314 shutdown_connections 2024-03-04T09:27:23.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:23.140+0000 7ff851600640 1 --2- 172.21.15.6:0/439766314 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff84c0cce90 0x7ff84c15c570 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:23.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:23.140+0000 7ff851600640 1 --2- 172.21.15.6:0/439766314 >> v2:172.21.15.146:3300/0 conn(0x7ff84c070f10 0x7ff84c0cc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:23.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:23.140+0000 7ff851600640 1 -- 172.21.15.6:0/439766314 >> 172.21.15.6:0/439766314 conn(0x7ff84c06b3d0 msgr2=0x7ff84c06b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:23.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:23.140+0000 7ff851600640 1 -- 172.21.15.6:0/439766314 shutdown_connections 2024-03-04T09:27:23.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:23.140+0000 7ff851600640 1 -- 172.21.15.6:0/439766314 wait complete. 2024-03-04T09:27:23.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:23.142+0000 7ff851600640 1 Processor -- start 2024-03-04T09:27:23.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:23.142+0000 7ff851600640 1 -- start start 2024-03-04T09:27:23.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:23.143+0000 7ff851600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff84c070f10 0x7ff84c0abda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:23.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:23.143+0000 7ff851600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff84c0cce90 0x7ff84c15a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:23.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:23.143+0000 7ff851600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff84c145300 0x7ff84c15aef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:23.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:23.143+0000 7ff851600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff84c15cab0 con 0x7ff84c070f10 2024-03-04T09:27:23.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:23.143+0000 7ff851600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff84c1456e0 con 0x7ff84c0cce90 2024-03-04T09:27:23.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:23.143+0000 7ff851600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff84c15cc30 con 0x7ff84c145300 2024-03-04T09:27:23.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:23.144+0000 7ff84be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff84c070f10 0x7ff84c0abda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:23.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:23.144+0000 7ff84be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff84c070f10 0x7ff84c0abda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:50496/0 (socket says 172.21.15.6:50496) 2024-03-04T09:27:23.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:23.144+0000 7ff84be00640 1 -- 172.21.15.6:0/3441268547 learned_addr learned my addr 172.21.15.6:0/3441268547 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:27:23.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:23.144+0000 7ff84b400640 1 --2- 172.21.15.6:0/3441268547 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff84c0cce90 0x7ff84c15a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:23.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:23.144+0000 7ff850c00640 1 --2- 172.21.15.6:0/3441268547 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff84c145300 0x7ff84c15aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:23.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:23.144+0000 7ff84be00640 1 -- 172.21.15.6:0/3441268547 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff84c145300 msgr2=0x7ff84c15aef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:23.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:23.144+0000 7ff84be00640 1 --2- 172.21.15.6:0/3441268547 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff84c145300 0x7ff84c15aef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:23.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:23.144+0000 7ff84be00640 1 -- 172.21.15.6:0/3441268547 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff84c0cce90 msgr2=0x7ff84c15a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:23.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:23.144+0000 7ff84be00640 1 --2- 172.21.15.6:0/3441268547 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff84c0cce90 0x7ff84c15a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:23.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:23.144+0000 7ff84be00640 1 -- 172.21.15.6:0/3441268547 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff84c1e6da0 con 0x7ff84c070f10 2024-03-04T09:27:23.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:23.145+0000 7ff84be00640 1 --2- 172.21.15.6:0/3441268547 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff84c070f10 0x7ff84c0abda0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7ff83c00b3f0 tx=0x7ff83c00b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:23.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:23.145+0000 7ff843e00640 1 -- 172.21.15.6:0/3441268547 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff83c015020 con 0x7ff84c070f10 2024-03-04T09:27:23.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:23.145+0000 7ff843e00640 1 -- 172.21.15.6:0/3441268547 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff83c011830 con 0x7ff84c070f10 2024-03-04T09:27:23.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:23.145+0000 7ff851600640 1 -- 172.21.15.6:0/3441268547 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff84c1e7080 con 0x7ff84c070f10 2024-03-04T09:27:23.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:23.145+0000 7ff843e00640 1 -- 172.21.15.6:0/3441268547 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff83c011c20 con 0x7ff84c070f10 2024-03-04T09:27:23.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:23.145+0000 7ff851600640 1 -- 172.21.15.6:0/3441268547 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff84c0a97f0 con 0x7ff84c070f10 2024-03-04T09:27:23.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:23.146+0000 7ff851600640 1 -- 172.21.15.6:0/3441268547 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff84c067830 con 0x7ff84c070f10 2024-03-04T09:27:23.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:23.147+0000 7ff843e00640 1 -- 172.21.15.6:0/3441268547 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff83c018020 con 0x7ff84c070f10 2024-03-04T09:27:23.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:23.149+0000 7ff843e00640 1 --2- 172.21.15.6:0/3441268547 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff82c06b4d0 0x7ff82c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:23.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:23.149+0000 7ff843e00640 1 -- 172.21.15.6:0/3441268547 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff83c089f80 con 0x7ff84c070f10 2024-03-04T09:27:23.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:23.149+0000 7ff84b400640 1 --2- 172.21.15.6:0/3441268547 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff82c06b4d0 0x7ff82c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:23.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:23.150+0000 7ff84b400640 1 --2- 172.21.15.6:0/3441268547 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff82c06b4d0 0x7ff82c06d990 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7ff84c15bf00 tx=0x7ff838008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:23.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:23.153+0000 7ff843e00640 1 -- 172.21.15.6:0/3441268547 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff84c067830 con 0x7ff84c070f10 2024-03-04T09:27:23.255 INFO:tasks.ceph.mds.b.smithi146.stderr:2024-03-04T09:27:23.258+0000 7fc7ef600640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-04T09:27:23.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:23.397+0000 7ff851600640 1 -- 172.21.15.6:0/3441268547 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff84c04f330 con 0x7ff84c070f10 2024-03-04T09:27:23.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:23.398+0000 7ff843e00640 1 -- 172.21.15.6:0/3441268547 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+4974 (secure 0 0 0) 0x7ff83c01f030 con 0x7ff84c070f10 2024-03-04T09:27:23.395 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:27:23.395 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4913,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/1049631914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":1049631914},{"type":"v1","addr":"172.21.15.146:6835","nonce":1049631914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":5092,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/424062551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":424062551},{"type":"v1","addr":"172.21.15.6:6833","nonce":424062551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":19},{"gid":5095,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/3300873502","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":3300873502},{"type":"v1","addr":"172.21.15.6:6835","nonce":3300873502}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:27:16.785309+0000","modified":"2024-03-04T09:27:19.824667+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5110},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5110":{"gid":5110,"name":"b","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.146:6837/3982958749","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":3982958749},{"type":"v1","addr":"172.21.15.146:6837","nonce":3982958749}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5110,"qdb_cluster":[5110]},"id":3}]} 2024-03-04T09:27:23.395 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 25 2024-03-04T09:27:23.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:23.400+0000 7ff851600640 1 -- 172.21.15.6:0/3441268547 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff82c06b4d0 msgr2=0x7ff82c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:23.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:23.400+0000 7ff851600640 1 --2- 172.21.15.6:0/3441268547 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff82c06b4d0 0x7ff82c06d990 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7ff84c15bf00 tx=0x7ff838008040 comp rx=0 tx=0).stop 2024-03-04T09:27:23.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:23.400+0000 7ff851600640 1 -- 172.21.15.6:0/3441268547 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff84c070f10 msgr2=0x7ff84c0abda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:23.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:23.400+0000 7ff851600640 1 --2- 172.21.15.6:0/3441268547 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff84c070f10 0x7ff84c0abda0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7ff83c00b3f0 tx=0x7ff83c00b8b0 comp rx=0 tx=0).stop 2024-03-04T09:27:23.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:23.400+0000 7ff851600640 1 -- 172.21.15.6:0/3441268547 shutdown_connections 2024-03-04T09:27:23.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:23.400+0000 7ff851600640 1 --2- 172.21.15.6:0/3441268547 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff82c06b4d0 0x7ff82c06d990 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:23.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:23.400+0000 7ff851600640 1 --2- 172.21.15.6:0/3441268547 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff84c145300 0x7ff84c15aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:23.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:23.400+0000 7ff851600640 1 --2- 172.21.15.6:0/3441268547 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff84c0cce90 0x7ff84c15a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:23.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:23.400+0000 7ff851600640 1 --2- 172.21.15.6:0/3441268547 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff84c070f10 0x7ff84c0abda0 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:23.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:23.401+0000 7ff851600640 1 -- 172.21.15.6:0/3441268547 >> 172.21.15.6:0/3441268547 conn(0x7ff84c06b3d0 msgr2=0x7ff84c15d040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:23.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:23.401+0000 7ff851600640 1 -- 172.21.15.6:0/3441268547 shutdown_connections 2024-03-04T09:27:23.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:23.401+0000 7ff851600640 1 -- 172.21.15.6:0/3441268547 wait complete. 2024-03-04T09:27:23.821 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:27:16.785309+0000', 'modified': '2024-03-04T09:27:19.824667+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5110}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5110': {'gid': 5110, 'name': 'b', 'rank': 0, 'incarnation': 21, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.146:6837/3982958749', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6836', 'nonce': 3982958749}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 3982958749}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [7], 'metadata_pool': 6, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5110, 'qdb_cluster': [5110]} 2024-03-04T09:27:23.821 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-04T09:27:23.822 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-04T09:27:24.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.058+0000 7f8b0ec00640 1 Processor -- start 2024-03-04T09:27:24.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.058+0000 7f8b0ec00640 1 -- start start 2024-03-04T09:27:24.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.060+0000 7f8b0ec00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8b081552b0 0x7f8b0815b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:24.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.060+0000 7f8b0ec00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f8b080cd060 0x7f8b080b3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:24.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.060+0000 7f8b0ec00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b0815ed10 con 0x7f8b0815e810 2024-03-04T09:27:24.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.060+0000 7f8b0ec00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b0815eff0 con 0x7f8b080cd060 2024-03-04T09:27:24.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.060+0000 7f8b0ec00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b0815f2d0 con 0x7f8b081552b0 2024-03-04T09:27:24.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.061+0000 7f8b0d800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8b081552b0 0x7f8b0815b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:24.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.061+0000 7f8b0ce00640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f8b0815e810 0x7f8b080cc950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.6:43266/0 (socket says 172.21.15.6:43266) 2024-03-04T09:27:24.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.061+0000 7f8b0ce00640 1 -- 172.21.15.6:0/383243200 learned_addr learned my addr 172.21.15.6:0/383243200 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:27:24.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.061+0000 7f8b0e200640 1 --2- 172.21.15.6:0/383243200 >> v2:172.21.15.146:3300/0 conn(0x7f8b080cd060 0x7f8b080b3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:24.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.062+0000 7f8b07e00640 1 -- 172.21.15.6:0/383243200 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2400370420 0 0) 0x7f8b0815ed10 con 0x7f8b0815e810 2024-03-04T09:27:24.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.062+0000 7f8b07e00640 1 -- 172.21.15.6:0/383243200 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8af0005730 con 0x7f8b0815e810 2024-03-04T09:27:24.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.062+0000 7f8b0d800640 1 -- 172.21.15.6:0/383243200 >> v2:172.21.15.146:3300/0 conn(0x7f8b080cd060 msgr2=0x7f8b080b3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:27:24.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.062+0000 7f8b0d800640 1 --2- 172.21.15.6:0/383243200 >> v2:172.21.15.146:3300/0 conn(0x7f8b080cd060 0x7f8b080b3cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:24.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.062+0000 7f8b0d800640 1 -- 172.21.15.6:0/383243200 >> v1:172.21.15.146:6789/0 conn(0x7f8b0815e810 legacy=0x7f8b080cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:24.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.062+0000 7f8b0d800640 1 -- 172.21.15.6:0/383243200 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b0815f5b0 con 0x7f8b081552b0 2024-03-04T09:27:24.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.063+0000 7f8b0d800640 1 --2- 172.21.15.6:0/383243200 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8b081552b0 0x7f8b0815b1d0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f8afc0097a0 tx=0x7f8afc02b820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=223243979b1eb6ab server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:24.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.064+0000 7f8b07e00640 1 -- 172.21.15.6:0/383243200 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8afc031780 con 0x7f8b081552b0 2024-03-04T09:27:24.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.064+0000 7f8b07e00640 1 -- 172.21.15.6:0/383243200 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8afc037070 con 0x7f8b081552b0 2024-03-04T09:27:24.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.064+0000 7f8b07e00640 1 -- 172.21.15.6:0/383243200 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8afc031b40 con 0x7f8b081552b0 2024-03-04T09:27:24.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.065+0000 7f8b0ec00640 1 -- 172.21.15.6:0/383243200 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8b081552b0 msgr2=0x7f8b0815b1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:24.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.065+0000 7f8b0ec00640 1 --2- 172.21.15.6:0/383243200 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8b081552b0 0x7f8b0815b1d0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f8afc0097a0 tx=0x7f8afc02b820 comp rx=0 tx=0).stop 2024-03-04T09:27:24.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.065+0000 7f8b0ec00640 1 -- 172.21.15.6:0/383243200 shutdown_connections 2024-03-04T09:27:24.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.065+0000 7f8b0ec00640 1 --2- 172.21.15.6:0/383243200 >> v2:172.21.15.146:3300/0 conn(0x7f8b080cd060 0x7f8b080b3cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:24.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.065+0000 7f8b0ec00640 1 --2- 172.21.15.6:0/383243200 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8b081552b0 0x7f8b0815b1d0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:24.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.065+0000 7f8b0ec00640 1 -- 172.21.15.6:0/383243200 >> 172.21.15.6:0/383243200 conn(0x7f8b0806b3d0 msgr2=0x7f8b0806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:24.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.065+0000 7f8b0ec00640 1 -- 172.21.15.6:0/383243200 shutdown_connections 2024-03-04T09:27:24.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.065+0000 7f8b0ec00640 1 -- 172.21.15.6:0/383243200 wait complete. 2024-03-04T09:27:24.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.067+0000 7f8b0ec00640 1 Processor -- start 2024-03-04T09:27:24.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.067+0000 7f8b0ec00640 1 -- start start 2024-03-04T09:27:24.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.068+0000 7f8b0ec00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8b080cd060 0x7f8b080b07a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:24.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.068+0000 7f8b0ec00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8b081552b0 0x7f8b08159510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:24.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.068+0000 7f8b0ec00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8b0815e810 0x7f8b08145e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:24.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.068+0000 7f8b0ec00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b0806dd30 con 0x7f8b080cd060 2024-03-04T09:27:24.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.068+0000 7f8b0ec00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b080b3760 con 0x7f8b0815e810 2024-03-04T09:27:24.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.068+0000 7f8b0ec00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b0806deb0 con 0x7f8b081552b0 2024-03-04T09:27:24.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.068+0000 7f8b0d800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8b080cd060 0x7f8b080b07a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:24.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.068+0000 7f8b0d800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8b080cd060 0x7f8b080b07a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:50508/0 (socket says 172.21.15.6:50508) 2024-03-04T09:27:24.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.068+0000 7f8b0d800640 1 -- 172.21.15.6:0/223005259 learned_addr learned my addr 172.21.15.6:0/223005259 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:27:24.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.069+0000 7f8b0e200640 1 --2- 172.21.15.6:0/223005259 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8b0815e810 0x7f8b08145e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:24.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.069+0000 7f8b0ce00640 1 --2- 172.21.15.6:0/223005259 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8b081552b0 0x7f8b08159510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:24.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.069+0000 7f8b0d800640 1 -- 172.21.15.6:0/223005259 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8b081552b0 msgr2=0x7f8b08159510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:24.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.069+0000 7f8b0d800640 1 --2- 172.21.15.6:0/223005259 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8b081552b0 0x7f8b08159510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:24.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.069+0000 7f8b0d800640 1 -- 172.21.15.6:0/223005259 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8b0815e810 msgr2=0x7f8b08145e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:24.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.069+0000 7f8b0d800640 1 --2- 172.21.15.6:0/223005259 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8b0815e810 0x7f8b08145e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:24.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.069+0000 7f8b0d800640 1 -- 172.21.15.6:0/223005259 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b08157810 con 0x7f8b080cd060 2024-03-04T09:27:24.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.069+0000 7f8b0d800640 1 --2- 172.21.15.6:0/223005259 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8b080cd060 0x7f8b080b07a0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f8afc034040 tx=0x7f8afc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:24.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.070+0000 7f8b05600640 1 -- 172.21.15.6:0/223005259 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8afc032610 con 0x7f8b080cd060 2024-03-04T09:27:24.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.070+0000 7f8b05600640 1 -- 172.21.15.6:0/223005259 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8afc0327d0 con 0x7f8b080cd060 2024-03-04T09:27:24.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.070+0000 7f8b0ec00640 1 -- 172.21.15.6:0/223005259 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b08157af0 con 0x7f8b080cd060 2024-03-04T09:27:24.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.070+0000 7f8b05600640 1 -- 172.21.15.6:0/223005259 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8afc032bc0 con 0x7f8b080cd060 2024-03-04T09:27:24.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.070+0000 7f8b0ec00640 1 -- 172.21.15.6:0/223005259 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b081580b0 con 0x7f8b080cd060 2024-03-04T09:27:24.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.072+0000 7f8b05600640 1 -- 172.21.15.6:0/223005259 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8afc04e440 con 0x7f8b080cd060 2024-03-04T09:27:24.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.073+0000 7f8b05600640 1 --2- 172.21.15.6:0/223005259 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f8ae806b4d0 0x7f8ae806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:24.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.073+0000 7f8b05600640 1 -- 172.21.15.6:0/223005259 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8afc0abe50 con 0x7f8b080cd060 2024-03-04T09:27:24.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.074+0000 7f8b0ce00640 1 --2- 172.21.15.6:0/223005259 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f8ae806b4d0 0x7f8ae806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:24.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.074+0000 7f8b12e1a740 1 Processor -- start 2024-03-04T09:27:24.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.074+0000 7f8b12e1a740 1 -- start start 2024-03-04T09:27:24.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.075+0000 7f8b12e1a740 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55f2b0d7b510 0x55f2b0d790d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:24.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.075+0000 7f8b12e1a740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55f2b0d5ad50 0x55f2b0d32a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:24.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.075+0000 7f8b12e1a740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f2b0d32f80 0x55f2b0d5d720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:24.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.075+0000 7f8b12e1a740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2b0c97600 con 0x55f2b0d5ad50 2024-03-04T09:27:24.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.075+0000 7f8b12e1a740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2b0c40930 con 0x55f2b0d32f80 2024-03-04T09:27:24.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.075+0000 7f8b12e1a740 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f2b0c936a0 con 0x55f2b0d7b510 2024-03-04T09:27:24.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.075+0000 7f8b0ce00640 1 --2- 172.21.15.6:0/223005259 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f8ae806b4d0 0x7f8ae806d990 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f8b0806e9f0 tx=0x7f8af4009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:24.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.075+0000 7f8b0ce00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55f2b0d5ad50 0x55f2b0d32a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:24.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.075+0000 7f8b0ce00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55f2b0d5ad50 0x55f2b0d32a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:50512/0 (socket says 172.21.15.6:50512) 2024-03-04T09:27:24.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.075+0000 7f8b0ce00640 1 -- 172.21.15.6:0/2114204694 learned_addr learned my addr 172.21.15.6:0/2114204694 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:27:24.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.075+0000 7f8b0e200640 1 --2- 172.21.15.6:0/2114204694 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f2b0d32f80 0x55f2b0d5d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:24.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.076+0000 7f8b0d800640 1 --2- 172.21.15.6:0/2114204694 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55f2b0d7b510 0x55f2b0d790d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:24.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.076+0000 7f8b0ce00640 1 -- 172.21.15.6:0/2114204694 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55f2b0d7b510 msgr2=0x55f2b0d790d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:27:24.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.076+0000 7f8b0ce00640 1 --2- 172.21.15.6:0/2114204694 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55f2b0d7b510 0x55f2b0d790d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:24.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.076+0000 7f8b0ce00640 1 -- 172.21.15.6:0/2114204694 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f2b0d32f80 msgr2=0x55f2b0d5d720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:27:24.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.076+0000 7f8b0ce00640 1 --2- 172.21.15.6:0/2114204694 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f2b0d32f80 0x55f2b0d5d720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:24.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.076+0000 7f8b0ce00640 1 -- 172.21.15.6:0/2114204694 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f2b09ac2d0 con 0x55f2b0d5ad50 2024-03-04T09:27:24.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.076+0000 7f8b0ce00640 1 --2- 172.21.15.6:0/2114204694 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55f2b0d5ad50 0x55f2b0d32a40 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f8af4005910 tx=0x7f8af4011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5878b0ead56cc647 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:24.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.076+0000 7f8afa000640 1 -- 172.21.15.6:0/2114204694 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8af400e7b0 con 0x55f2b0d5ad50 2024-03-04T09:27:24.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.076+0000 7f8afa000640 1 -- 172.21.15.6:0/2114204694 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8af400e950 con 0x55f2b0d5ad50 2024-03-04T09:27:24.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.076+0000 7f8afa000640 1 -- 172.21.15.6:0/2114204694 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8af400ed00 con 0x55f2b0d5ad50 2024-03-04T09:27:24.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.077+0000 7f8b12e1a740 1 -- 172.21.15.6:0/2114204694 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55f2b0d5ad50 msgr2=0x55f2b0d32a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:24.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.077+0000 7f8b12e1a740 1 --2- 172.21.15.6:0/2114204694 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55f2b0d5ad50 0x55f2b0d32a40 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f8af4005910 tx=0x7f8af4011000 comp rx=0 tx=0).stop 2024-03-04T09:27:24.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.077+0000 7f8b12e1a740 1 -- 172.21.15.6:0/2114204694 shutdown_connections 2024-03-04T09:27:24.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.077+0000 7f8b12e1a740 1 --2- 172.21.15.6:0/2114204694 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f2b0d32f80 0x55f2b0d5d720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:24.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.077+0000 7f8b12e1a740 1 --2- 172.21.15.6:0/2114204694 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55f2b0d5ad50 0x55f2b0d32a40 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:24.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.077+0000 7f8b12e1a740 1 --2- 172.21.15.6:0/2114204694 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55f2b0d7b510 0x55f2b0d790d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:24.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.077+0000 7f8b12e1a740 1 -- 172.21.15.6:0/2114204694 >> 172.21.15.6:0/2114204694 conn(0x55f2b0d2fba0 msgr2=0x55f2b0d78720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:24.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.077+0000 7f8b12e1a740 1 -- 172.21.15.6:0/2114204694 shutdown_connections 2024-03-04T09:27:24.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.077+0000 7f8b12e1a740 1 -- 172.21.15.6:0/2114204694 wait complete. 2024-03-04T09:27:24.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.078+0000 7f8b12e1a740 1 -- start start 2024-03-04T09:27:24.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.078+0000 7f8b12e1a740 1 Processor -- start 2024-03-04T09:27:24.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.079+0000 7f8b12e1a740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55f2b0d5ad50 0x55f2b0d57020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:24.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.079+0000 7f8b12e1a740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f2b0d7b510 0x55f2b0d2c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:24.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.079+0000 7f8b12e1a740 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55f2b0d722d0 0x55f2b0d2cd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:24.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.079+0000 7f8b12e1a740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2b0d31ea0 con 0x55f2b0d5ad50 2024-03-04T09:27:24.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.079+0000 7f8b12e1a740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2b0c53d90 con 0x55f2b0d7b510 2024-03-04T09:27:24.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.079+0000 7f8b12e1a740 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f2b0d2e1d0 con 0x55f2b0d722d0 2024-03-04T09:27:24.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.079+0000 7f8b12e1a740 10 client.0 _ll_register_callbacks cb 0x55f2b0c8f350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T09:27:24.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.080+0000 7f8b0d800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55f2b0d5ad50 0x55f2b0d57020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:24.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.080+0000 7f8b0d800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55f2b0d5ad50 0x55f2b0d57020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:50514/0 (socket says 172.21.15.6:50514) 2024-03-04T09:27:24.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.080+0000 7f8b0d800640 1 -- 172.21.15.6:0/3713655650 learned_addr learned my addr 172.21.15.6:0/3713655650 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:27:24.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.080+0000 7f8b0ce00640 1 --2- 172.21.15.6:0/3713655650 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f2b0d7b510 0x55f2b0d2c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:24.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.080+0000 7f8b0d800640 1 -- 172.21.15.6:0/3713655650 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55f2b0d722d0 msgr2=0x55f2b0d2cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:27:24.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.080+0000 7f8b0d800640 1 --2- 172.21.15.6:0/3713655650 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55f2b0d722d0 0x55f2b0d2cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:24.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.080+0000 7f8b0d800640 1 -- 172.21.15.6:0/3713655650 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f2b0d7b510 msgr2=0x55f2b0d2c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:27:24.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.080+0000 7f8b0d800640 1 --2- 172.21.15.6:0/3713655650 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f2b0d7b510 0x55f2b0d2c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:24.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.080+0000 7f8b0d800640 1 -- 172.21.15.6:0/3713655650 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f2b0a766b0 con 0x55f2b0d5ad50 2024-03-04T09:27:24.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.080+0000 7f8b0ce00640 1 --2- 172.21.15.6:0/3713655650 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f2b0d7b510 0x55f2b0d2c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:27:24.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.080+0000 7f8b0d800640 1 --2- 172.21.15.6:0/3713655650 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55f2b0d5ad50 0x55f2b0d57020 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f8afc0aa430 tx=0x7f8afc079fc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2b2f5cebb86bf54 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:24.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.080+0000 7f8aeea00640 10 client.0 ms_handle_connect on v2:172.21.15.6:3300/0 2024-03-04T09:27:24.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.081+0000 7f8aeea00640 1 -- 172.21.15.6:0/3713655650 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8afc07e7b0 con 0x55f2b0d5ad50 2024-03-04T09:27:24.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.081+0000 7f8aeea00640 1 -- 172.21.15.6:0/3713655650 <== mon.0 v2:172.21.15.6:3300/0 2 ==== osd_map(44..44 src has 1..44) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8afc0920d0 con 0x55f2b0d5ad50 2024-03-04T09:27:24.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.081+0000 7f8b12e1a740 1 -- 172.21.15.6:0/3713655650 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8af400e950 con 0x55f2b0d5ad50 2024-03-04T09:27:24.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.081+0000 7f8aeea00640 1 -- 172.21.15.6:0/3713655650 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8afc082d70 con 0x55f2b0d5ad50 2024-03-04T09:27:24.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.081+0000 7f8b12e1a740 10 client.5649 fetch_fsmap learned FSMap version 25 2024-03-04T09:27:24.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.081+0000 7f8b12e1a740 1 -- 172.21.15.6:0/3713655650 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=2+}) v3 -- 0x55f2b0d57560 con 0x55f2b0d5ad50 2024-03-04T09:27:24.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.081+0000 7f8aeea00640 1 -- 172.21.15.6:0/3713655650 <== mon.0 v2:172.21.15.6:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8af400e950 con 0x55f2b0d5ad50 2024-03-04T09:27:24.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.081+0000 7f8aeea00640 1 -- 172.21.15.6:0/3713655650 <== mon.0 v2:172.21.15.6:3300/0 5 ==== fsmap(e 25) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f8afc07a8c0 con 0x55f2b0d5ad50 2024-03-04T09:27:24.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.081+0000 7f8b12e1a740 10 client.5649 fetch_fsmap finished waiting for FSMap version 25 2024-03-04T09:27:24.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.081+0000 7f8b12e1a740 10 client.5649 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-04T09:27:24.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.081+0000 7f8b12e1a740 20 client.5649 populate_metadata read hostname 'smithi006' 2024-03-04T09:27:24.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.081+0000 7f8b12e1a740 1 --2- 172.21.15.6:0/3713655650 >> [v2:172.21.15.146:6836/3982958749,v1:172.21.15.146:6837/3982958749] conn(0x55f2b0d908a0 0x55f2b0d90c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:24.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.081+0000 7f8b12e1a740 4 client.5649 mds_command: new command op to 5110 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T09:27:24.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.081+0000 7f8b12e1a740 1 -- 172.21.15.6:0/3713655650 --> [v2:172.21.15.146:6836/3982958749,v1:172.21.15.146:6837/3982958749] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f2b0ca4b40 con 0x55f2b0d908a0 2024-03-04T09:27:24.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.082+0000 7f8b0ce00640 1 --2- 172.21.15.6:0/3713655650 >> [v2:172.21.15.146:6836/3982958749,v1:172.21.15.146:6837/3982958749] conn(0x55f2b0d908a0 0x55f2b0d90c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:24.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.083+0000 7f8b0ce00640 1 --2- 172.21.15.6:0/3713655650 >> [v2:172.21.15.146:6836/3982958749,v1:172.21.15.146:6837/3982958749] conn(0x55f2b0d908a0 0x55f2b0d90c80 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b171ad96f6b335da server_cookie=11443d7f693258b8 in_seq=0 out_seq=0 2024-03-04T09:27:24.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.083+0000 7f8aeea00640 10 client.5649 ms_handle_connect on v2:172.21.15.146:6836/3982958749 2024-03-04T09:27:24.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.086+0000 7f8aeea00640 1 -- 172.21.15.6:0/3713655650 <== mds.0 v2:172.21.15.146:6836/3982958749 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55f2b0ca4b40 con 0x55f2b0d908a0 2024-03-04T09:27:24.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.086+0000 7f8aeea00640 10 client.5649 handle_command_reply: tid=0 2024-03-04T09:27:24.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.086+0000 7f8b12e1a740 1 client.5649 shutdown 2024-03-04T09:27:24.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.086+0000 7f8b12e1a740 1 -- 172.21.15.6:0/3713655650 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55f2b0d5ad50 msgr2=0x55f2b0d57020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:24.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.086+0000 7f8b12e1a740 1 --2- 172.21.15.6:0/3713655650 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55f2b0d5ad50 0x55f2b0d57020 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f8afc0aa430 tx=0x7f8afc079fc0 comp rx=0 tx=0).stop 2024-03-04T09:27:24.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.086+0000 7f8b12e1a740 1 -- 172.21.15.6:0/3713655650 shutdown_connections 2024-03-04T09:27:24.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.086+0000 7f8b12e1a740 1 --2- 172.21.15.6:0/3713655650 >> [v2:172.21.15.146:6836/3982958749,v1:172.21.15.146:6837/3982958749] conn(0x55f2b0d908a0 0x55f2b0d90c80 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:24.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.086+0000 7f8b12e1a740 1 --2- 172.21.15.6:0/3713655650 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55f2b0d722d0 0x55f2b0d2cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:24.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.086+0000 7f8b12e1a740 1 --2- 172.21.15.6:0/3713655650 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f2b0d7b510 0x55f2b0d2c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:24.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.086+0000 7f8b12e1a740 1 --2- 172.21.15.6:0/3713655650 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55f2b0d5ad50 0x55f2b0d57020 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:24.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.086+0000 7f8b12e1a740 1 -- 172.21.15.6:0/3713655650 >> 172.21.15.6:0/3713655650 conn(0x55f2b0d32f80 msgr2=0x55f2b0d7b8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:24.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.086+0000 7f8aeea00640 0 client.5649 ms_handle_reset on v2:172.21.15.146:6836/3982958749 2024-03-04T09:27:24.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.086+0000 7f8b12e1a740 1 -- 172.21.15.6:0/3713655650 shutdown_connections 2024-03-04T09:27:24.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.086+0000 7f8b12e1a740 1 -- 172.21.15.6:0/3713655650 wait complete. 2024-03-04T09:27:24.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.086+0000 7f8b12e1a740 20 client.5649 trim_cache size 0 max 16384 2024-03-04T09:27:24.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.105+0000 7f8b12e1a740 1 Processor -- start 2024-03-04T09:27:24.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.105+0000 7f8b12e1a740 1 -- start start 2024-03-04T09:27:24.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.106+0000 7f8b12e1a740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f2b0d5ad50 0x55f2b0d751c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:24.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.106+0000 7f8b12e1a740 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55f2b0d722d0 0x55f2b0d32020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:24.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.106+0000 7f8b12e1a740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55f2b0d7b510 0x55f2b0d32560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:24.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.106+0000 7f8b12e1a740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b081f08e0 con 0x55f2b0d7b510 2024-03-04T09:27:24.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.106+0000 7f8b12e1a740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b081efce0 con 0x55f2b0d5ad50 2024-03-04T09:27:24.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.106+0000 7f8b12e1a740 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b081f02e0 con 0x55f2b0d722d0 2024-03-04T09:27:24.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.106+0000 7f8b0e200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55f2b0d7b510 0x55f2b0d32560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:24.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.106+0000 7f8b0e200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55f2b0d7b510 0x55f2b0d32560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:50522/0 (socket says 172.21.15.6:50522) 2024-03-04T09:27:24.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.106+0000 7f8b0e200640 1 -- 172.21.15.6:0/4020930524 learned_addr learned my addr 172.21.15.6:0/4020930524 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:27:24.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.107+0000 7f8b0d800640 1 --2- 172.21.15.6:0/4020930524 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f2b0d5ad50 0x55f2b0d751c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:24.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.107+0000 7f8b0e200640 1 -- 172.21.15.6:0/4020930524 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55f2b0d722d0 msgr2=0x55f2b0d32020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:27:24.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.107+0000 7f8b0e200640 1 --2- 172.21.15.6:0/4020930524 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55f2b0d722d0 0x55f2b0d32020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:24.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.107+0000 7f8b0e200640 1 -- 172.21.15.6:0/4020930524 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f2b0d5ad50 msgr2=0x55f2b0d751c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:27:24.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.107+0000 7f8b0e200640 1 --2- 172.21.15.6:0/4020930524 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f2b0d5ad50 0x55f2b0d751c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:24.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.107+0000 7f8b0e200640 1 -- 172.21.15.6:0/4020930524 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f2b0d77c20 con 0x55f2b0d7b510 2024-03-04T09:27:24.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.107+0000 7f8b0d800640 1 --2- 172.21.15.6:0/4020930524 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f2b0d5ad50 0x55f2b0d751c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:27:24.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.107+0000 7f8b0e200640 1 --2- 172.21.15.6:0/4020930524 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55f2b0d7b510 0x55f2b0d32560 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f8b00007360 tx=0x7f8b00007820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d35336742aee82c9 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:24.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.107+0000 7f8af9600640 1 -- 172.21.15.6:0/4020930524 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b0001a070 con 0x55f2b0d7b510 2024-03-04T09:27:24.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.107+0000 7f8af9600640 1 -- 172.21.15.6:0/4020930524 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b00016830 con 0x55f2b0d7b510 2024-03-04T09:27:24.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.107+0000 7f8af9600640 1 -- 172.21.15.6:0/4020930524 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b00016c20 con 0x55f2b0d7b510 2024-03-04T09:27:24.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.108+0000 7f8b12e1a740 1 -- 172.21.15.6:0/4020930524 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55f2b0d7b510 msgr2=0x55f2b0d32560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:24.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.108+0000 7f8b12e1a740 1 --2- 172.21.15.6:0/4020930524 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55f2b0d7b510 0x55f2b0d32560 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f8b00007360 tx=0x7f8b00007820 comp rx=0 tx=0).stop 2024-03-04T09:27:24.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.108+0000 7f8b12e1a740 1 -- 172.21.15.6:0/4020930524 shutdown_connections 2024-03-04T09:27:24.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.108+0000 7f8b12e1a740 1 --2- 172.21.15.6:0/4020930524 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55f2b0d7b510 0x55f2b0d32560 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:24.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.108+0000 7f8b12e1a740 1 --2- 172.21.15.6:0/4020930524 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55f2b0d722d0 0x55f2b0d32020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:24.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.108+0000 7f8b12e1a740 1 --2- 172.21.15.6:0/4020930524 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f2b0d5ad50 0x55f2b0d751c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:24.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.108+0000 7f8b12e1a740 1 -- 172.21.15.6:0/4020930524 >> 172.21.15.6:0/4020930524 conn(0x55f2b0d32f80 msgr2=0x55f2b0d61610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:24.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.108+0000 7f8b12e1a740 1 -- 172.21.15.6:0/4020930524 shutdown_connections 2024-03-04T09:27:24.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.108+0000 7f8b12e1a740 1 -- 172.21.15.6:0/4020930524 wait complete. 2024-03-04T09:27:24.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.109+0000 7f8b12e1a740 1 -- start start 2024-03-04T09:27:24.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.109+0000 7f8b12e1a740 1 Processor -- start 2024-03-04T09:27:24.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.110+0000 7f8b12e1a740 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55f2b0d5ad50 0x55f2b0d5a280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:24.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.110+0000 7f8b12e1a740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f2b0d722d0 0x55f2b0df1230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:24.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.110+0000 7f8b12e1a740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55f2b0d7b510 0x55f2b0d733f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:24.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.110+0000 7f8b12e1a740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2b0d31ea0 con 0x55f2b0d7b510 2024-03-04T09:27:24.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.110+0000 7f8b12e1a740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2b0c53d90 con 0x55f2b0d722d0 2024-03-04T09:27:24.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.110+0000 7f8b12e1a740 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f2b0d71920 con 0x55f2b0d5ad50 2024-03-04T09:27:24.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.110+0000 7f8b12e1a740 10 client.0 _ll_register_callbacks cb 0x7f8afc006c30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T09:27:24.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.110+0000 7f8b0e200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55f2b0d7b510 0x55f2b0d733f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:24.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.110+0000 7f8b0e200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55f2b0d7b510 0x55f2b0d733f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:50526/0 (socket says 172.21.15.6:50526) 2024-03-04T09:27:24.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.110+0000 7f8b0e200640 1 -- 172.21.15.6:0/1716280185 learned_addr learned my addr 172.21.15.6:0/1716280185 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:27:24.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.110+0000 7f8b0ce00640 1 --2- 172.21.15.6:0/1716280185 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f2b0d722d0 0x55f2b0df1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:24.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.111+0000 7f8b0d800640 1 --2- 172.21.15.6:0/1716280185 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55f2b0d5ad50 0x55f2b0d5a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:24.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.111+0000 7f8b0e200640 1 -- 172.21.15.6:0/1716280185 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55f2b0d5ad50 msgr2=0x55f2b0d5a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:27:24.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.111+0000 7f8b0e200640 1 --2- 172.21.15.6:0/1716280185 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55f2b0d5ad50 0x55f2b0d5a280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:24.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.111+0000 7f8b0e200640 1 -- 172.21.15.6:0/1716280185 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f2b0d722d0 msgr2=0x55f2b0df1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:27:24.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.111+0000 7f8b0e200640 1 --2- 172.21.15.6:0/1716280185 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f2b0d722d0 0x55f2b0df1230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:24.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.111+0000 7f8b0e200640 1 -- 172.21.15.6:0/1716280185 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f2b0d73bb0 con 0x55f2b0d7b510 2024-03-04T09:27:24.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.111+0000 7f8b0e200640 1 --2- 172.21.15.6:0/1716280185 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55f2b0d7b510 0x55f2b0d733f0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f8b0000b010 tx=0x7f8b0000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66811d8a3371d947 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:24.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.111+0000 7f8aeea00640 10 client.0 ms_handle_connect on v2:172.21.15.6:3300/0 2024-03-04T09:27:24.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.111+0000 7f8aeea00640 1 -- 172.21.15.6:0/1716280185 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b0001a040 con 0x55f2b0d7b510 2024-03-04T09:27:24.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.111+0000 7f8aeea00640 1 -- 172.21.15.6:0/1716280185 <== mon.0 v2:172.21.15.6:3300/0 2 ==== osd_map(44..44 src has 1..44) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8b0001f660 con 0x55f2b0d7b510 2024-03-04T09:27:24.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.111+0000 7f8b12e1a740 1 -- 172.21.15.6:0/1716280185 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8b00016830 con 0x55f2b0d7b510 2024-03-04T09:27:24.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.112+0000 7f8aeea00640 1 -- 172.21.15.6:0/1716280185 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8b00033050 con 0x55f2b0d7b510 2024-03-04T09:27:24.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.112+0000 7f8b12e1a740 10 client.5655 fetch_fsmap learned FSMap version 25 2024-03-04T09:27:24.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.112+0000 7f8b12e1a740 1 -- 172.21.15.6:0/1716280185 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=2+}) v3 -- 0x55f2b0d9fbe0 con 0x55f2b0d7b510 2024-03-04T09:27:24.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.112+0000 7f8aeea00640 1 -- 172.21.15.6:0/1716280185 <== mon.0 v2:172.21.15.6:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b00016830 con 0x55f2b0d7b510 2024-03-04T09:27:24.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.112+0000 7f8aeea00640 1 -- 172.21.15.6:0/1716280185 <== mon.0 v2:172.21.15.6:3300/0 5 ==== fsmap(e 25) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f8b00036020 con 0x55f2b0d7b510 2024-03-04T09:27:24.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.112+0000 7f8b12e1a740 10 client.5655 fetch_fsmap finished waiting for FSMap version 25 2024-03-04T09:27:24.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.112+0000 7f8b12e1a740 10 client.5655 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-04T09:27:24.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.112+0000 7f8b12e1a740 20 client.5655 populate_metadata read hostname 'smithi006' 2024-03-04T09:27:24.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.112+0000 7f8b12e1a740 1 --2- 172.21.15.6:0/1716280185 >> [v2:172.21.15.146:6836/3982958749,v1:172.21.15.146:6837/3982958749] conn(0x55f2b0d324a0 0x55f2b0d2d630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:24.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.112+0000 7f8b12e1a740 4 client.5655 mds_command: new command op to 5110 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-04T09:27:24.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.112+0000 7f8b12e1a740 1 -- 172.21.15.6:0/1716280185 --> [v2:172.21.15.146:6836/3982958749,v1:172.21.15.146:6837/3982958749] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55f2b0d76b70 con 0x55f2b0d324a0 2024-03-04T09:27:24.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.113+0000 7f8b0d800640 1 --2- 172.21.15.6:0/1716280185 >> [v2:172.21.15.146:6836/3982958749,v1:172.21.15.146:6837/3982958749] conn(0x55f2b0d324a0 0x55f2b0d2d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:24.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.114+0000 7f8b0d800640 1 --2- 172.21.15.6:0/1716280185 >> [v2:172.21.15.146:6836/3982958749,v1:172.21.15.146:6837/3982958749] conn(0x55f2b0d324a0 0x55f2b0d2d630 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=17b7c964672efe88 server_cookie=bfb6167b1826840a in_seq=0 out_seq=0 2024-03-04T09:27:24.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.114+0000 7f8aeea00640 10 client.5655 ms_handle_connect on v2:172.21.15.146:6836/3982958749 2024-03-04T09:27:24.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.114+0000 7f8aeea00640 1 -- 172.21.15.6:0/1716280185 <== mds.0 v2:172.21.15.146:6836/3982958749 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+324 (crc 0 0 0) 0x55f2b0d76b70 con 0x55f2b0d324a0 2024-03-04T09:27:24.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.114+0000 7f8aeea00640 10 client.5655 handle_command_reply: tid=0 2024-03-04T09:27:24.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.114+0000 7f8b12e1a740 1 client.5655 shutdown 2024-03-04T09:27:24.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.115+0000 7f8b12e1a740 1 -- 172.21.15.6:0/1716280185 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55f2b0d7b510 msgr2=0x55f2b0d733f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:24.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.115+0000 7f8b12e1a740 1 --2- 172.21.15.6:0/1716280185 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55f2b0d7b510 0x55f2b0d733f0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f8b0000b010 tx=0x7f8b0000b040 comp rx=0 tx=0).stop 2024-03-04T09:27:24.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.115+0000 7f8b12e1a740 1 -- 172.21.15.6:0/1716280185 shutdown_connections 2024-03-04T09:27:24.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.115+0000 7f8b12e1a740 1 --2- 172.21.15.6:0/1716280185 >> [v2:172.21.15.146:6836/3982958749,v1:172.21.15.146:6837/3982958749] conn(0x55f2b0d324a0 0x55f2b0d2d630 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:24.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.115+0000 7f8b12e1a740 1 --2- 172.21.15.6:0/1716280185 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55f2b0d7b510 0x55f2b0d733f0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:24.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.115+0000 7f8b12e1a740 1 --2- 172.21.15.6:0/1716280185 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f2b0d722d0 0x55f2b0df1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:24.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.115+0000 7f8b12e1a740 1 --2- 172.21.15.6:0/1716280185 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55f2b0d5ad50 0x55f2b0d5a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:24.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.115+0000 7f8aeea00640 0 client.5655 ms_handle_reset on v2:172.21.15.146:6836/3982958749 2024-03-04T09:27:24.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.115+0000 7f8b12e1a740 1 -- 172.21.15.6:0/1716280185 >> 172.21.15.6:0/1716280185 conn(0x55f2b0d32f80 msgr2=0x55f2b0d78430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:24.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.115+0000 7f8b12e1a740 1 -- 172.21.15.6:0/1716280185 shutdown_connections 2024-03-04T09:27:24.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.115+0000 7f8b12e1a740 1 -- 172.21.15.6:0/1716280185 wait complete. 2024-03-04T09:27:24.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.115+0000 7f8b12e1a740 20 client.5655 trim_cache size 0 max 16384 2024-03-04T09:27:24.115 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-04T09:27:24.115 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_fsid": "edce0578-5ae4-4402-b45a-552bdef84d7c", 2024-03-04T09:27:24.115 INFO:teuthology.orchestra.run.smithi006.stdout: "whoami": 0, 2024-03-04T09:27:24.115 INFO:teuthology.orchestra.run.smithi006.stdout: "id": 5110, 2024-03-04T09:27:24.115 INFO:teuthology.orchestra.run.smithi006.stdout: "want_state": "up:active", 2024-03-04T09:27:24.115 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "up:active", 2024-03-04T09:27:24.115 INFO:teuthology.orchestra.run.smithi006.stdout: "fs_name": "cephfs", 2024-03-04T09:27:24.115 INFO:teuthology.orchestra.run.smithi006.stdout: "rank_uptime": 6.802420315, 2024-03-04T09:27:24.115 INFO:teuthology.orchestra.run.smithi006.stdout: "mdsmap_epoch": 25, 2024-03-04T09:27:24.115 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap_epoch": 44, 2024-03-04T09:27:24.115 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap_epoch_barrier": 44, 2024-03-04T09:27:24.115 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 12.171823981999999 2024-03-04T09:27:24.116 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-04T09:27:24.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.116+0000 7f8b0ec00640 1 -- 172.21.15.6:0/223005259 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f8ae806b4d0 msgr2=0x7f8ae806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:24.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.116+0000 7f8b0ec00640 1 --2- 172.21.15.6:0/223005259 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f8ae806b4d0 0x7f8ae806d990 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f8b0806e9f0 tx=0x7f8af4009040 comp rx=0 tx=0).stop 2024-03-04T09:27:24.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.116+0000 7f8b0ec00640 1 -- 172.21.15.6:0/223005259 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8b080cd060 msgr2=0x7f8b080b07a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:24.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.116+0000 7f8b0ec00640 1 --2- 172.21.15.6:0/223005259 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8b080cd060 0x7f8b080b07a0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f8afc034040 tx=0x7f8afc0046c0 comp rx=0 tx=0).stop 2024-03-04T09:27:24.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.116+0000 7f8b0ec00640 1 -- 172.21.15.6:0/223005259 shutdown_connections 2024-03-04T09:27:24.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.116+0000 7f8b0ec00640 1 --2- 172.21.15.6:0/223005259 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8b0815e810 0x7f8b08145e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:24.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.116+0000 7f8b0ec00640 1 --2- 172.21.15.6:0/223005259 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f8ae806b4d0 0x7f8ae806d990 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:24.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.116+0000 7f8b0ec00640 1 --2- 172.21.15.6:0/223005259 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8b081552b0 0x7f8b08159510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:24.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.116+0000 7f8b0ec00640 1 --2- 172.21.15.6:0/223005259 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8b080cd060 0x7f8b080b07a0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:24.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.116+0000 7f8b0ec00640 1 -- 172.21.15.6:0/223005259 >> 172.21.15.6:0/223005259 conn(0x7f8b0806b3d0 msgr2=0x7f8b08071980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:24.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.116+0000 7f8b0ec00640 1 -- 172.21.15.6:0/223005259 shutdown_connections 2024-03-04T09:27:24.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.116+0000 7f8b0ec00640 1 -- 172.21.15.6:0/223005259 wait complete. 2024-03-04T09:27:24.130 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-04T09:27:24.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.338+0000 7fd55f200640 1 Processor -- start 2024-03-04T09:27:24.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.339+0000 7fd55f200640 1 -- start start 2024-03-04T09:27:24.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.340+0000 7fd55f200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd560145300 0x7fd5600b49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:24.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.340+0000 7fd55f200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd560070f10 0x7fd5600cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:24.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.340+0000 7fd55f200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5600714d0 con 0x7fd5600cce90 2024-03-04T09:27:24.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.340+0000 7fd55f200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5600cd300 con 0x7fd560145300 2024-03-04T09:27:24.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.340+0000 7fd55f200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5600717b0 con 0x7fd560070f10 2024-03-04T09:27:24.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.340+0000 7fd55de00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd560145300 0x7fd5600b49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:24.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.340+0000 7fd55de00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd560145300 0x7fd5600b49a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:50536/0 (socket says 172.21.15.6:50536) 2024-03-04T09:27:24.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.340+0000 7fd55de00640 1 -- 172.21.15.6:0/4133718463 learned_addr learned my addr 172.21.15.6:0/4133718463 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:27:24.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.340+0000 7fd555400640 1 --2- 172.21.15.6:0/4133718463 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd560070f10 0x7fd5600cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:24.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.341+0000 7fd55de00640 1 -- 172.21.15.6:0/4133718463 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd560070f10 msgr2=0x7fd5600cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:27:24.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.341+0000 7fd55de00640 1 --2- 172.21.15.6:0/4133718463 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd560070f10 0x7fd5600cc950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:24.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.341+0000 7fd55de00640 1 -- 172.21.15.6:0/4133718463 >> v1:172.21.15.146:6789/0 conn(0x7fd5600cce90 legacy=0x7fd5600cd930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:27:24.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.341+0000 7fd55de00640 1 -- 172.21.15.6:0/4133718463 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd560071a90 con 0x7fd560145300 2024-03-04T09:27:24.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.342+0000 7fd55de00640 1 --2- 172.21.15.6:0/4133718463 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd560145300 0x7fd5600b49a0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fd550009e50 tx=0x7fd55002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2074d9852d179ab server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:24.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.342+0000 7fd55d400640 1 -- 172.21.15.6:0/4133718463 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd55002cb00 con 0x7fd560145300 2024-03-04T09:27:24.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.342+0000 7fd55d400640 1 -- 172.21.15.6:0/4133718463 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd55002cca0 con 0x7fd560145300 2024-03-04T09:27:24.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.342+0000 7fd55d400640 1 -- 172.21.15.6:0/4133718463 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd550034610 con 0x7fd560145300 2024-03-04T09:27:24.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.343+0000 7fd55f200640 1 -- 172.21.15.6:0/4133718463 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd560145300 msgr2=0x7fd5600b49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:24.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.343+0000 7fd55f200640 1 --2- 172.21.15.6:0/4133718463 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd560145300 0x7fd5600b49a0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fd550009e50 tx=0x7fd55002bf60 comp rx=0 tx=0).stop 2024-03-04T09:27:24.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.343+0000 7fd55f200640 1 -- 172.21.15.6:0/4133718463 shutdown_connections 2024-03-04T09:27:24.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.343+0000 7fd55f200640 1 --2- 172.21.15.6:0/4133718463 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd560070f10 0x7fd5600cc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:24.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.343+0000 7fd55f200640 1 --2- 172.21.15.6:0/4133718463 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd560145300 0x7fd5600b49a0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:24.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.343+0000 7fd55f200640 1 -- 172.21.15.6:0/4133718463 >> 172.21.15.6:0/4133718463 conn(0x7fd56006b3d0 msgr2=0x7fd56006b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:24.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.343+0000 7fd55f200640 1 -- 172.21.15.6:0/4133718463 shutdown_connections 2024-03-04T09:27:24.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.343+0000 7fd55f200640 1 -- 172.21.15.6:0/4133718463 wait complete. 2024-03-04T09:27:24.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.345+0000 7fd55f200640 1 Processor -- start 2024-03-04T09:27:24.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.345+0000 7fd55f200640 1 -- start start 2024-03-04T09:27:24.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.346+0000 7fd55f200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd560070f10 0x7fd56015b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:24.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.346+0000 7fd55f200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd5600cce90 0x7fd5601e95a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:24.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.346+0000 7fd55f200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd560145300 0x7fd5601e9ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:24.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.346+0000 7fd55f200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5601456e0 con 0x7fd560070f10 2024-03-04T09:27:24.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.346+0000 7fd55f200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5600b4ee0 con 0x7fd5600cce90 2024-03-04T09:27:24.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.346+0000 7fd55f200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd56015d580 con 0x7fd560145300 2024-03-04T09:27:24.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.346+0000 7fd55de00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd560070f10 0x7fd56015b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:24.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.346+0000 7fd55de00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd560070f10 0x7fd56015b430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:50546/0 (socket says 172.21.15.6:50546) 2024-03-04T09:27:24.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.346+0000 7fd55de00640 1 -- 172.21.15.6:0/163748324 learned_addr learned my addr 172.21.15.6:0/163748324 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:27:24.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.347+0000 7fd555400640 1 --2- 172.21.15.6:0/163748324 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd5600cce90 0x7fd5601e95a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:24.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.347+0000 7fd55e800640 1 --2- 172.21.15.6:0/163748324 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd560145300 0x7fd5601e9ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:24.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.347+0000 7fd55de00640 1 -- 172.21.15.6:0/163748324 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd560145300 msgr2=0x7fd5601e9ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:24.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.347+0000 7fd55de00640 1 --2- 172.21.15.6:0/163748324 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd560145300 0x7fd5601e9ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:24.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.347+0000 7fd55de00640 1 -- 172.21.15.6:0/163748324 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd5600cce90 msgr2=0x7fd5601e95a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:24.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.347+0000 7fd55de00640 1 --2- 172.21.15.6:0/163748324 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd5600cce90 0x7fd5601e95a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:24.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.347+0000 7fd55de00640 1 -- 172.21.15.6:0/163748324 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd56015b970 con 0x7fd560070f10 2024-03-04T09:27:24.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.347+0000 7fd55e800640 1 --2- 172.21.15.6:0/163748324 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd560145300 0x7fd5601e9ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:27:24.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.347+0000 7fd55de00640 1 --2- 172.21.15.6:0/163748324 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd560070f10 0x7fd56015b430 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fd550036040 tx=0x7fd550004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:24.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.348+0000 7fd556a00640 1 -- 172.21.15.6:0/163748324 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd550043020 con 0x7fd560070f10 2024-03-04T09:27:24.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.348+0000 7fd556a00640 1 -- 172.21.15.6:0/163748324 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd55002ce50 con 0x7fd560070f10 2024-03-04T09:27:24.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.348+0000 7fd55f200640 1 -- 172.21.15.6:0/163748324 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5601e6d40 con 0x7fd560070f10 2024-03-04T09:27:24.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.348+0000 7fd556a00640 1 -- 172.21.15.6:0/163748324 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd550009510 con 0x7fd560070f10 2024-03-04T09:27:24.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.348+0000 7fd55f200640 1 -- 172.21.15.6:0/163748324 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5601e7300 con 0x7fd560070f10 2024-03-04T09:27:24.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.349+0000 7fd55f200640 1 -- 172.21.15.6:0/163748324 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5600b3eb0 con 0x7fd560070f10 2024-03-04T09:27:24.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.350+0000 7fd556a00640 1 -- 172.21.15.6:0/163748324 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd550046020 con 0x7fd560070f10 2024-03-04T09:27:24.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.352+0000 7fd556a00640 1 --2- 172.21.15.6:0/163748324 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd53c06b4d0 0x7fd53c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:24.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.352+0000 7fd556a00640 1 -- 172.21.15.6:0/163748324 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd5500ac990 con 0x7fd560070f10 2024-03-04T09:27:24.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.352+0000 7fd555400640 1 --2- 172.21.15.6:0/163748324 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd53c06b4d0 0x7fd53c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:24.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.353+0000 7fd555400640 1 --2- 172.21.15.6:0/163748324 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd53c06b4d0 0x7fd53c06d990 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fd5600b0e90 tx=0x7fd548008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:24.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.356+0000 7fd556a00640 1 -- 172.21.15.6:0/163748324 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd5600b3eb0 con 0x7fd560070f10 2024-03-04T09:27:24.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.598+0000 7fd55f200640 1 -- 172.21.15.6:0/163748324 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fd56004f330 con 0x7fd560070f10 2024-03-04T09:27:24.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.599+0000 7fd556a00640 1 -- 172.21.15.6:0/163748324 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v25) v1 ==== 53+0+73 (secure 0 0 0) 0x7fd55004d030 con 0x7fd560070f10 2024-03-04T09:27:24.596 INFO:teuthology.orchestra.run.smithi006.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-04T09:27:24.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.602+0000 7fd55f200640 1 -- 172.21.15.6:0/163748324 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd53c06b4d0 msgr2=0x7fd53c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:24.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.602+0000 7fd55f200640 1 --2- 172.21.15.6:0/163748324 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd53c06b4d0 0x7fd53c06d990 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fd5600b0e90 tx=0x7fd548008040 comp rx=0 tx=0).stop 2024-03-04T09:27:24.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.602+0000 7fd55f200640 1 -- 172.21.15.6:0/163748324 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd560070f10 msgr2=0x7fd56015b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:24.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.602+0000 7fd55f200640 1 --2- 172.21.15.6:0/163748324 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd560070f10 0x7fd56015b430 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fd550036040 tx=0x7fd550004a40 comp rx=0 tx=0).stop 2024-03-04T09:27:24.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.602+0000 7fd55f200640 1 -- 172.21.15.6:0/163748324 shutdown_connections 2024-03-04T09:27:24.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.602+0000 7fd55f200640 1 --2- 172.21.15.6:0/163748324 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd53c06b4d0 0x7fd53c06d990 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:24.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.602+0000 7fd55f200640 1 --2- 172.21.15.6:0/163748324 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd560145300 0x7fd5601e9ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:24.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.602+0000 7fd55f200640 1 --2- 172.21.15.6:0/163748324 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd5600cce90 0x7fd5601e95a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:24.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.602+0000 7fd55f200640 1 --2- 172.21.15.6:0/163748324 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd560070f10 0x7fd56015b430 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:24.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.602+0000 7fd55f200640 1 -- 172.21.15.6:0/163748324 >> 172.21.15.6:0/163748324 conn(0x7fd56006b3d0 msgr2=0x7fd56015eae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:24.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.603+0000 7fd55f200640 1 -- 172.21.15.6:0/163748324 shutdown_connections 2024-03-04T09:27:24.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:24.603+0000 7fd55f200640 1 -- 172.21.15.6:0/163748324 wait complete. 2024-03-04T09:27:24.619 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-04T09:27:24.619 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-04T09:27:24.619 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi006.front.sepia.ceph.com 2024-03-04T09:27:24.619 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-04T09:27:24.619 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-04T09:27:24.619 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-04T09:27:24.619 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-03-04T09:27:24.619 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-04T09:27:24.619 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-04T09:27:24.619 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:27:24.619 DEBUG:teuthology.orchestra.run.smithi006:> ip addr 2024-03-04T09:27:24.674 INFO:teuthology.orchestra.run.smithi006.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-04T09:27:24.674 INFO:teuthology.orchestra.run.smithi006.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-04T09:27:24.674 INFO:teuthology.orchestra.run.smithi006.stdout: inet 127.0.0.1/8 scope host lo 2024-03-04T09:27:24.674 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-04T09:27:24.674 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 ::1/128 scope host 2024-03-04T09:27:24.674 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-04T09:27:24.674 INFO:teuthology.orchestra.run.smithi006.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-04T09:27:24.674 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:6b:fe:e6 brd ff:ff:ff:ff:ff:ff 2024-03-04T09:27:24.674 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp7s0f0 2024-03-04T09:27:24.674 INFO:teuthology.orchestra.run.smithi006.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-04T09:27:24.675 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:6b:fe:e7 brd ff:ff:ff:ff:ff:ff 2024-03-04T09:27:24.675 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp7s0f1 2024-03-04T09:27:24.675 INFO:teuthology.orchestra.run.smithi006.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-04T09:27:24.675 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:bd:15:e4 brd ff:ff:ff:ff:ff:ff 2024-03-04T09:27:24.675 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp1s0f0 2024-03-04T09:27:24.675 INFO:teuthology.orchestra.run.smithi006.stdout: inet 172.21.15.6/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-04T09:27:24.675 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft 42489sec preferred_lft 42489sec 2024-03-04T09:27:24.675 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::ec4:7aff:febd:15e4/64 scope link noprefixroute 2024-03-04T09:27:24.675 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-04T09:27:24.675 INFO:teuthology.orchestra.run.smithi006.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-04T09:27:24.675 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:bd:15:e5 brd ff:ff:ff:ff:ff:ff 2024-03-04T09:27:24.675 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp1s0f1 2024-03-04T09:27:24.675 INFO:teuthology.orchestra.run.smithi006.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-04T09:27:24.675 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 5e:b4:59:dc:42:58 brd ff:ff:ff:ff:ff:ff 2024-03-04T09:27:24.675 INFO:teuthology.orchestra.run.smithi006.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-04T09:27:24.676 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-04T09:27:24.676 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::c6f:adff:fefe:e00d/64 scope link proto kernel_ll 2024-03-04T09:27:24.676 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-04T09:27:24.676 INFO:teuthology.orchestra.run.smithi006.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-04T09:27:24.676 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 7e:67:6d:ea:f9:ff brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-04T09:27:24.676 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::7c67:6dff:feea:f9ff/64 scope link proto kernel_ll 2024-03-04T09:27:24.676 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-04T09:27:24.676 INFO:teuthology.orchestra.run.smithi006.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-04T09:27:24.676 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 5e:b4:59:dc:42:58 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-04T09:27:24.676 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::5cb4:59ff:fedc:4258/64 scope link proto kernel_ll 2024-03-04T09:27:24.676 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-04T09:27:24.678 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:27:24.678 DEBUG:teuthology.orchestra.run.smithi006:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-04T09:27:24.771 INFO:teuthology.orchestra.run.smithi006.stdout:1 2024-03-04T09:27:24.776 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:27:24.776 DEBUG:teuthology.orchestra.run.smithi006:> route 2024-03-04T09:27:24.831 INFO:teuthology.orchestra.run.smithi006.stdout:Kernel IP routing table 2024-03-04T09:27:24.831 INFO:teuthology.orchestra.run.smithi006.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-04T09:27:24.831 INFO:teuthology.orchestra.run.smithi006.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-04T09:27:24.831 INFO:teuthology.orchestra.run.smithi006.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-04T09:27:24.831 INFO:teuthology.orchestra.run.smithi006.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-04T09:27:24.832 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:27:24.832 DEBUG:teuthology.orchestra.run.smithi006:> (cd / && exec bash -c ' 2024-03-04T09:27:24.833 DEBUG:teuthology.orchestra.run.smithi006:> set -e 2024-03-04T09:27:24.833 DEBUG:teuthology.orchestra.run.smithi006:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-04T09:27:24.833 DEBUG:teuthology.orchestra.run.smithi006:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-04T09:27:24.833 DEBUG:teuthology.orchestra.run.smithi006:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-04T09:27:24.833 DEBUG:teuthology.orchestra.run.smithi006:> ') 2024-03-04T09:27:25.029 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:27:25.029 DEBUG:teuthology.orchestra.run.smithi006:> ip netns list 2024-03-04T09:27:25.083 INFO:teuthology.orchestra.run.smithi006.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-04T09:27:25.084 INFO:teuthology.orchestra.run.smithi006.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-04T09:27:25.084 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-04T09:27:25.085 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:27:25.085 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip link set brx.0 up 2024-03-04T09:27:25.185 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-04T09:27:25.185 DEBUG:teuthology.orchestra.run.smithi006:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-04T09:27:25.242 INFO:teuthology.orchestra.run.smithi006.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-04T09:27:25.242 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-04T09:27:25.242 DEBUG:teuthology.orchestra.run.smithi006:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-04T09:27:25.294 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-03-04T09:27:25.294 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:27:25.295 DEBUG:teuthology.orchestra.run.smithi006:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-04T09:27:25.429 INFO:teuthology.orchestra.run.smithi006.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-04T09:27:25.429 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-04T09:27:25.467 INFO:teuthology.orchestra.run.smithi006.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-04T09:27:25.467 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-04T09:27:25.467 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: resolved to: "172.21.15.6:3300,172.21.15.146:3300,172.21.15.146:3301" 2024-03-04T09:27:25.467 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: trying mount with new device syntax: 0@edce0578-5ae4-4402-b45a-552bdef84d7c.cephfs=/ 2024-03-04T09:27:25.467 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.6:3300/172.21.15.146:3300/172.21.15.146:3301" will pass to kernel 2024-03-04T09:27:25.471 INFO:teuthology.orchestra.run.smithi006.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-04T09:27:25.471 INFO:teuthology.orchestra.run.smithi006.stdout: You just mounted a file system that supports labels which does not 2024-03-04T09:27:25.471 INFO:teuthology.orchestra.run.smithi006.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-04T09:27:25.471 INFO:teuthology.orchestra.run.smithi006.stdout: applications will generate AVC messages and not be allowed access to 2024-03-04T09:27:25.472 INFO:teuthology.orchestra.run.smithi006.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-04T09:27:25.476 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-04T09:27:25.476 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:27:25.476 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-04T09:27:25.569 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:27:25.569 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-04T09:27:25.647 INFO:teuthology.orchestra.run.smithi006.stdout:client4970 2024-03-04T09:27:25.647 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-04T09:27:25.647 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T09:27:25.705 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds mds_bal_fragment_size_max 50 2024-03-04T09:27:25.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:25.936+0000 7fd7afc00640 1 Processor -- start 2024-03-04T09:27:25.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:25.937+0000 7fd7afc00640 1 -- start start 2024-03-04T09:27:25.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:25.938+0000 7fd7afc00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd7a815bf20 0x7fd7a80acd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:25.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:25.938+0000 7fd7afc00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7a8072df0 con 0x7fd7a8072520 2024-03-04T09:27:25.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:25.938+0000 7fd7afc00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7a80cdab0 con 0x7fd7a8071e20 2024-03-04T09:27:25.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:25.938+0000 7fd7afc00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7a80b4d70 con 0x7fd7a815bf20 2024-03-04T09:27:25.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:25.939+0000 7fd7a7e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fd7a8072520 0x7fd7a80cdfb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:45764/0 (socket says 172.21.15.6:45764) 2024-03-04T09:27:25.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:25.939+0000 7fd7a7e00640 1 -- 172.21.15.6:0/4218936916 learned_addr learned my addr 172.21.15.6:0/4218936916 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:27:25.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:25.939+0000 7fd7ad200640 1 --2- 172.21.15.6:0/4218936916 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd7a815bf20 0x7fd7a80acd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:25.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:25.939+0000 7fd7a7400640 1 -- 172.21.15.6:0/4218936916 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1919847363 0 0) 0x7fd7a8072df0 con 0x7fd7a8072520 2024-03-04T09:27:25.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:25.939+0000 7fd7a7400640 1 -- 172.21.15.6:0/4218936916 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd78c003520 con 0x7fd7a8072520 2024-03-04T09:27:25.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:25.940+0000 7fd7a7400640 1 -- 172.21.15.6:0/4218936916 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2447162265 0 0) 0x7fd78c003520 con 0x7fd7a8072520 2024-03-04T09:27:25.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:25.940+0000 7fd7a7400640 1 -- 172.21.15.6:0/4218936916 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd7a815bf20 msgr2=0x7fd7a80acd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:27:25.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:25.940+0000 7fd7a7400640 1 --2- 172.21.15.6:0/4218936916 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd7a815bf20 0x7fd7a80acd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:25.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:25.940+0000 7fd7a7400640 1 -- 172.21.15.6:0/4218936916 >> v1:172.21.15.146:6789/0 conn(0x7fd7a8071e20 legacy=0x7fd7a80ccad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:25.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:25.940+0000 7fd7a7400640 1 -- 172.21.15.6:0/4218936916 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7a80b5050 con 0x7fd7a8072520 2024-03-04T09:27:25.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:25.940+0000 7fd7a7400640 1 -- 172.21.15.6:0/4218936916 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7fd794001670 con 0x7fd7a8072520 2024-03-04T09:27:25.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:25.940+0000 7fd7ad200640 1 --2- 172.21.15.6:0/4218936916 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd7a815bf20 0x7fd7a80acd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T09:27:25.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:25.940+0000 7fd7a7400640 1 -- 172.21.15.6:0/4218936916 >> v1:172.21.15.6:6789/0 conn(0x7fd7a8072520 legacy=0x7fd7a80cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:25.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:25.940+0000 7fd7a7400640 1 --2- 172.21.15.6:0/4218936916 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd78c0039e0 0x7fd78c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:25.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:25.940+0000 7fd7a7400640 1 --2- 172.21.15.6:0/4218936916 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd78c004800 0x7fd78c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:25.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:25.940+0000 7fd7a7400640 1 --2- 172.21.15.6:0/4218936916 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd7a815bf20 0x7fd78c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:25.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:25.940+0000 7fd7a7400640 1 -- 172.21.15.6:0/4218936916 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7a80b4d70 con 0x7fd78c0039e0 2024-03-04T09:27:25.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:25.940+0000 7fd7a7400640 1 -- 172.21.15.6:0/4218936916 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd78c00ba70 con 0x7fd78c004800 2024-03-04T09:27:25.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:25.940+0000 7fd7a7400640 1 -- 172.21.15.6:0/4218936916 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd78c00bc70 con 0x7fd7a815bf20 2024-03-04T09:27:25.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:25.940+0000 7fd7ad200640 1 --2- 172.21.15.6:0/4218936916 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd78c0039e0 0x7fd78c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:25.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:25.940+0000 7fd7a7e00640 1 --2- 172.21.15.6:0/4218936916 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd78c004800 0x7fd78c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:25.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:25.940+0000 7fd7ad200640 1 -- 172.21.15.6:0/4218936916 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd7a815bf20 msgr2=0x7fd78c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:27:25.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:25.941+0000 7fd7ad200640 1 --2- 172.21.15.6:0/4218936916 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd7a815bf20 0x7fd78c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:25.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:25.941+0000 7fd7ad200640 1 -- 172.21.15.6:0/4218936916 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd78c004800 msgr2=0x7fd78c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:27:25.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:25.941+0000 7fd7ad200640 1 --2- 172.21.15.6:0/4218936916 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd78c004800 0x7fd78c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:25.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:25.941+0000 7fd7ad200640 1 -- 172.21.15.6:0/4218936916 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd78c003520 con 0x7fd78c0039e0 2024-03-04T09:27:25.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:25.941+0000 7fd7a7e00640 1 --2- 172.21.15.6:0/4218936916 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd78c004800 0x7fd78c004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:27:25.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:25.942+0000 7fd7ad200640 1 --2- 172.21.15.6:0/4218936916 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd78c0039e0 0x7fd78c003e00 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fd798009c60 tx=0x7fd78c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b8bc7aebd980c8f server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:25.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:25.942+0000 7fd7a7400640 1 -- 172.21.15.6:0/4218936916 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd798039070 con 0x7fd78c0039e0 2024-03-04T09:27:25.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:25.942+0000 7fd7a7400640 1 -- 172.21.15.6:0/4218936916 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd79802c8f0 con 0x7fd78c0039e0 2024-03-04T09:27:25.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:25.942+0000 7fd7a7400640 1 -- 172.21.15.6:0/4218936916 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd798009860 con 0x7fd78c0039e0 2024-03-04T09:27:25.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:25.943+0000 7fd7afc00640 1 -- 172.21.15.6:0/4218936916 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd78c0039e0 msgr2=0x7fd78c003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:25.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:25.943+0000 7fd7afc00640 1 --2- 172.21.15.6:0/4218936916 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd78c0039e0 0x7fd78c003e00 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fd798009c60 tx=0x7fd78c005830 comp rx=0 tx=0).stop 2024-03-04T09:27:25.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:25.943+0000 7fd7adc00640 1 -- 172.21.15.6:0/4218936916 reap_dead start 2024-03-04T09:27:25.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:25.943+0000 7fd7afc00640 1 -- 172.21.15.6:0/4218936916 shutdown_connections 2024-03-04T09:27:25.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:25.943+0000 7fd7afc00640 1 -- 172.21.15.6:0/4218936916 >> 172.21.15.6:0/4218936916 conn(0x7fd7a806b3d0 msgr2=0x7fd7a806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:25.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:25.943+0000 7fd7afc00640 1 -- 172.21.15.6:0/4218936916 shutdown_connections 2024-03-04T09:27:25.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:25.943+0000 7fd7afc00640 1 -- 172.21.15.6:0/4218936916 wait complete. 2024-03-04T09:27:25.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:25.945+0000 7fd7afc00640 1 Processor -- start 2024-03-04T09:27:25.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:25.945+0000 7fd7afc00640 1 -- start start 2024-03-04T09:27:25.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:25.946+0000 7fd7afc00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd78c0039e0 0x7fd7a8150520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:25.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:25.946+0000 7fd7afc00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd7a815d590 0x7fd7a81f77c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:25.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:25.946+0000 7fd7afc00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd7a815dd30 0x7fd7a815e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:25.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:25.946+0000 7fd7afc00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7a80b4b10 con 0x7fd7a815dd30 2024-03-04T09:27:25.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:25.946+0000 7fd7afc00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7a80b4990 con 0x7fd7a815d590 2024-03-04T09:27:25.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:25.946+0000 7fd7afc00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7a80ce6c0 con 0x7fd78c0039e0 2024-03-04T09:27:25.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:25.946+0000 7fd7adc00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd7a815dd30 0x7fd7a815e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:25.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:25.946+0000 7fd7adc00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd7a815dd30 0x7fd7a815e1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:50570/0 (socket says 172.21.15.6:50570) 2024-03-04T09:27:25.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:25.946+0000 7fd7adc00640 1 -- 172.21.15.6:0/3104107265 learned_addr learned my addr 172.21.15.6:0/3104107265 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:27:25.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:25.946+0000 7fd7ad200640 1 --2- 172.21.15.6:0/3104107265 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd78c0039e0 0x7fd7a8150520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:25.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:25.946+0000 7fd7a7e00640 1 --2- 172.21.15.6:0/3104107265 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd7a815d590 0x7fd7a81f77c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:25.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:25.946+0000 7fd7adc00640 1 -- 172.21.15.6:0/3104107265 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd78c0039e0 msgr2=0x7fd7a8150520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:25.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:25.946+0000 7fd7adc00640 1 --2- 172.21.15.6:0/3104107265 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd78c0039e0 0x7fd7a8150520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:25.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:25.946+0000 7fd7adc00640 1 -- 172.21.15.6:0/3104107265 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd7a815d590 msgr2=0x7fd7a81f77c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:25.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:25.946+0000 7fd7adc00640 1 --2- 172.21.15.6:0/3104107265 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd7a815d590 0x7fd7a81f77c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:25.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:25.946+0000 7fd7adc00640 1 -- 172.21.15.6:0/3104107265 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7a815e6f0 con 0x7fd7a815dd30 2024-03-04T09:27:25.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:25.947+0000 7fd7adc00640 1 --2- 172.21.15.6:0/3104107265 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd7a815dd30 0x7fd7a815e1b0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fd7a806ec70 tx=0x7fd7a000c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:25.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:25.947+0000 7fd7a4c00640 1 -- 172.21.15.6:0/3104107265 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7a0017070 con 0x7fd7a815dd30 2024-03-04T09:27:25.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:25.947+0000 7fd7a4c00640 1 -- 172.21.15.6:0/3104107265 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7a0007070 con 0x7fd7a815dd30 2024-03-04T09:27:25.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:25.947+0000 7fd7afc00640 1 -- 172.21.15.6:0/3104107265 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7a81f8680 con 0x7fd7a815dd30 2024-03-04T09:27:25.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:25.947+0000 7fd7a4c00640 1 -- 172.21.15.6:0/3104107265 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7a0007460 con 0x7fd7a815dd30 2024-03-04T09:27:25.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:25.947+0000 7fd7afc00640 1 -- 172.21.15.6:0/3104107265 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7a81f83a0 con 0x7fd7a815dd30 2024-03-04T09:27:25.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:25.948+0000 7fd7afc00640 1 -- 172.21.15.6:0/3104107265 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7a804f330 con 0x7fd7a815dd30 2024-03-04T09:27:25.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:25.949+0000 7fd7a4c00640 1 -- 172.21.15.6:0/3104107265 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd7a001a070 con 0x7fd7a815dd30 2024-03-04T09:27:25.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:25.950+0000 7fd7a4c00640 1 --2- 172.21.15.6:0/3104107265 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd78806b500 0x7fd78806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:25.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:25.950+0000 7fd7a4c00640 1 -- 172.21.15.6:0/3104107265 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd7a0089330 con 0x7fd7a815dd30 2024-03-04T09:27:25.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:25.951+0000 7fd7ad200640 1 --2- 172.21.15.6:0/3104107265 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd78806b500 0x7fd78806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:25.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:25.952+0000 7fd7ad200640 1 --2- 172.21.15.6:0/3104107265 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd78806b500 0x7fd78806d9c0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fd798009c60 tx=0x7fd798030000 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:25.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:25.955+0000 7fd7a4c00640 1 -- 172.21.15.6:0/3104107265 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd7a0057b50 con 0x7fd7a815dd30 2024-03-04T09:27:26.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:26.155+0000 7fd7afc00640 1 -- 172.21.15.6:0/3104107265 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command([{prefix=config set, name=mds_bal_fragment_size_max}] v 0) v1 -- 0x7fd7a8150b00 con 0x7fd7a815dd30 2024-03-04T09:27:26.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:26.161+0000 7fd7a4c00640 1 -- 172.21.15.6:0/3104107265 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_fragment_size_max}]=0 v13)=0 v13) v1 ==== 124+0+0 (secure 0 0 0) 0x7fd7a005bbf0 con 0x7fd7a815dd30 2024-03-04T09:27:26.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:26.164+0000 7fd7afc00640 1 -- 172.21.15.6:0/3104107265 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd78806b500 msgr2=0x7fd78806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:26.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:26.164+0000 7fd7afc00640 1 --2- 172.21.15.6:0/3104107265 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd78806b500 0x7fd78806d9c0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fd798009c60 tx=0x7fd798030000 comp rx=0 tx=0).stop 2024-03-04T09:27:26.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:26.164+0000 7fd7afc00640 1 -- 172.21.15.6:0/3104107265 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd7a815dd30 msgr2=0x7fd7a815e1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:26.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:26.164+0000 7fd7afc00640 1 --2- 172.21.15.6:0/3104107265 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd7a815dd30 0x7fd7a815e1b0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fd7a806ec70 tx=0x7fd7a000c960 comp rx=0 tx=0).stop 2024-03-04T09:27:26.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:26.164+0000 7fd7afc00640 1 -- 172.21.15.6:0/3104107265 shutdown_connections 2024-03-04T09:27:26.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:26.164+0000 7fd7afc00640 1 --2- 172.21.15.6:0/3104107265 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd7a815dd30 0x7fd7a815e1b0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:26.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:26.164+0000 7fd7afc00640 1 --2- 172.21.15.6:0/3104107265 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd7a815d590 0x7fd7a81f77c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:26.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:26.164+0000 7fd7afc00640 1 --2- 172.21.15.6:0/3104107265 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd78806b500 0x7fd78806d9c0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:26.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:26.164+0000 7fd7afc00640 1 --2- 172.21.15.6:0/3104107265 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd78c0039e0 0x7fd7a8150520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:26.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:26.164+0000 7fd7afc00640 1 -- 172.21.15.6:0/3104107265 >> 172.21.15.6:0/3104107265 conn(0x7fd7a806b3d0 msgr2=0x7fd7a80cb120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:26.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:26.164+0000 7fd7afc00640 1 -- 172.21.15.6:0/3104107265 shutdown_connections 2024-03-04T09:27:26.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:26.164+0000 7fd7afc00640 1 -- 172.21.15.6:0/3104107265 wait complete. 2024-03-04T09:27:36.179 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-04T09:27:36.179 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T09:27:36.239 DEBUG:teuthology.orchestra.run.smithi006:> stdin-killer -- python3 -c ' 2024-03-04T09:27:36.239 DEBUG:teuthology.orchestra.run.smithi006:> import os 2024-03-04T09:27:36.240 DEBUG:teuthology.orchestra.run.smithi006:> import uuid 2024-03-04T09:27:36.240 DEBUG:teuthology.orchestra.run.smithi006:> 2024-03-04T09:27:36.240 DEBUG:teuthology.orchestra.run.smithi006:> n = 51 2024-03-04T09:27:36.240 DEBUG:teuthology.orchestra.run.smithi006:> create_hard_links = False 2024-03-04T09:27:36.240 DEBUG:teuthology.orchestra.run.smithi006:> if 0 > 0: 2024-03-04T09:27:36.240 DEBUG:teuthology.orchestra.run.smithi006:> create_hard_links = True 2024-03-04T09:27:36.240 DEBUG:teuthology.orchestra.run.smithi006:> path = "/home/ubuntu/cephtest/mnt.0/subdir/file" 2024-03-04T09:27:36.240 DEBUG:teuthology.orchestra.run.smithi006:> 2024-03-04T09:27:36.240 DEBUG:teuthology.orchestra.run.smithi006:> dpath = os.path.dirname(path) 2024-03-04T09:27:36.240 DEBUG:teuthology.orchestra.run.smithi006:> fnameprefix = os.path.basename(path) 2024-03-04T09:27:36.240 DEBUG:teuthology.orchestra.run.smithi006:> os.makedirs(dpath, exist_ok=True) 2024-03-04T09:27:36.240 DEBUG:teuthology.orchestra.run.smithi006:> 2024-03-04T09:27:36.240 DEBUG:teuthology.orchestra.run.smithi006:> try: 2024-03-04T09:27:36.240 DEBUG:teuthology.orchestra.run.smithi006:> dirfd = os.open(dpath, os.O_DIRECTORY) 2024-03-04T09:27:36.240 DEBUG:teuthology.orchestra.run.smithi006:> 2024-03-04T09:27:36.241 DEBUG:teuthology.orchestra.run.smithi006:> for i in range(n): 2024-03-04T09:27:36.241 DEBUG:teuthology.orchestra.run.smithi006:> fpath = os.path.join(dpath, f"{fnameprefix}_{i}") 2024-03-04T09:27:36.241 DEBUG:teuthology.orchestra.run.smithi006:> with open(fpath, '"'"'w'"'"') as f: 2024-03-04T09:27:36.241 DEBUG:teuthology.orchestra.run.smithi006:> f.write(f"{i}") 2024-03-04T09:27:36.241 DEBUG:teuthology.orchestra.run.smithi006:> if False: 2024-03-04T09:27:36.241 DEBUG:teuthology.orchestra.run.smithi006:> f.flush() 2024-03-04T09:27:36.241 DEBUG:teuthology.orchestra.run.smithi006:> os.fsync(f.fileno()) 2024-03-04T09:27:36.241 DEBUG:teuthology.orchestra.run.smithi006:> if False: 2024-03-04T09:27:36.241 DEBUG:teuthology.orchestra.run.smithi006:> os.unlink(fpath) 2024-03-04T09:27:36.241 DEBUG:teuthology.orchestra.run.smithi006:> if False: 2024-03-04T09:27:36.241 DEBUG:teuthology.orchestra.run.smithi006:> os.fsync(dirfd) 2024-03-04T09:27:36.241 DEBUG:teuthology.orchestra.run.smithi006:> if create_hard_links: 2024-03-04T09:27:36.241 DEBUG:teuthology.orchestra.run.smithi006:> for j in range(0): 2024-03-04T09:27:36.241 DEBUG:teuthology.orchestra.run.smithi006:> os.system(f"ln {fpath} {dpath}/{fnameprefix}_{i}_{uuid.uuid4()}") 2024-03-04T09:27:36.241 DEBUG:teuthology.orchestra.run.smithi006:> if True: 2024-03-04T09:27:36.241 DEBUG:teuthology.orchestra.run.smithi006:> os.fsync(dirfd) 2024-03-04T09:27:36.242 DEBUG:teuthology.orchestra.run.smithi006:> finally: 2024-03-04T09:27:36.242 DEBUG:teuthology.orchestra.run.smithi006:> os.close(dirfd) 2024-03-04T09:27:36.242 DEBUG:teuthology.orchestra.run.smithi006:> ' 2024-03-04T09:27:36.485 INFO:teuthology.orchestra.run.smithi006.stderr:Traceback (most recent call last): 2024-03-04T09:27:36.486 INFO:teuthology.orchestra.run.smithi006.stderr: File "", line 33, in 2024-03-04T09:27:36.486 INFO:teuthology.orchestra.run.smithi006.stderr:OSError: [Errno 28] No space left on device 2024-03-04T09:27:36.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:36 stdin-killer INFO: command exited with status 1: exiting normally with same code! 2024-03-04T09:27:36.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T09:27:36.504 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-04T09:27:36.504 DEBUG:teuthology.orchestra.run.smithi006:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-04T09:27:36.670 DEBUG:teuthology.orchestra.run.smithi146:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-04T09:27:36.797 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-04T09:27:36.797 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-04T09:27:36.797 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T09:27:36.855 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-04T09:27:36.855 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T09:27:36.910 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T09:27:37.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.133+0000 7fe274a00640 1 Processor -- start 2024-03-04T09:27:37.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.133+0000 7fe274a00640 1 -- start start 2024-03-04T09:27:37.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.135+0000 7fe274a00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fe270070f10 0x7fe2700cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:37.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.135+0000 7fe274a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fe2700cce90 0x7fe27015c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:37.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.135+0000 7fe274a00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2700714d0 con 0x7fe270145300 2024-03-04T09:27:37.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.135+0000 7fe274a00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2700cd300 con 0x7fe2700cce90 2024-03-04T09:27:37.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.135+0000 7fe274a00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2700717b0 con 0x7fe270070f10 2024-03-04T09:27:37.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.135+0000 7fe26f200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fe2700cce90 0x7fe27015c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:37.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.135+0000 7fe26f200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fe2700cce90 0x7fe27015c570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:47612/0 (socket says 172.21.15.6:47612) 2024-03-04T09:27:37.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.135+0000 7fe26f200640 1 -- 172.21.15.6:0/2144653340 learned_addr learned my addr 172.21.15.6:0/2144653340 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:27:37.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.135+0000 7fe26de00640 1 --2- 172.21.15.6:0/2144653340 >> v2:172.21.15.146:3300/0 conn(0x7fe270070f10 0x7fe2700cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:37.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.136+0000 7fe26f200640 1 -- 172.21.15.6:0/2144653340 >> v2:172.21.15.146:3300/0 conn(0x7fe270070f10 msgr2=0x7fe2700cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:27:37.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.136+0000 7fe26f200640 1 --2- 172.21.15.6:0/2144653340 >> v2:172.21.15.146:3300/0 conn(0x7fe270070f10 0x7fe2700cc950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:37.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.136+0000 7fe26f200640 1 -- 172.21.15.6:0/2144653340 >> v1:172.21.15.146:6789/0 conn(0x7fe270145300 legacy=0x7fe2700b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:27:37.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.136+0000 7fe26f200640 1 -- 172.21.15.6:0/2144653340 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe270071a90 con 0x7fe2700cce90 2024-03-04T09:27:37.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.137+0000 7fe26f200640 1 --2- 172.21.15.6:0/2144653340 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe2700cce90 0x7fe27015c570 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fe2700b5150 tx=0x7fe26802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc08f06eeda11b27 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:37.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.137+0000 7fe26d400640 1 -- 172.21.15.6:0/2144653340 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe26802e070 con 0x7fe2700cce90 2024-03-04T09:27:37.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.137+0000 7fe26d400640 1 -- 172.21.15.6:0/2144653340 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe268037070 con 0x7fe2700cce90 2024-03-04T09:27:37.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.137+0000 7fe26d400640 1 -- 172.21.15.6:0/2144653340 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe268031a00 con 0x7fe2700cce90 2024-03-04T09:27:37.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.138+0000 7fe274a00640 1 -- 172.21.15.6:0/2144653340 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe2700cce90 msgr2=0x7fe27015c570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:37.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.138+0000 7fe274a00640 1 --2- 172.21.15.6:0/2144653340 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe2700cce90 0x7fe27015c570 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fe2700b5150 tx=0x7fe26802b880 comp rx=0 tx=0).stop 2024-03-04T09:27:37.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.138+0000 7fe274a00640 1 -- 172.21.15.6:0/2144653340 shutdown_connections 2024-03-04T09:27:37.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.138+0000 7fe274a00640 1 --2- 172.21.15.6:0/2144653340 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe2700cce90 0x7fe27015c570 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:37.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.138+0000 7fe274a00640 1 --2- 172.21.15.6:0/2144653340 >> v2:172.21.15.146:3300/0 conn(0x7fe270070f10 0x7fe2700cc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:37.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.138+0000 7fe274a00640 1 -- 172.21.15.6:0/2144653340 >> 172.21.15.6:0/2144653340 conn(0x7fe27006b3d0 msgr2=0x7fe27006b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:37.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.138+0000 7fe274a00640 1 -- 172.21.15.6:0/2144653340 shutdown_connections 2024-03-04T09:27:37.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.138+0000 7fe274a00640 1 -- 172.21.15.6:0/2144653340 wait complete. 2024-03-04T09:27:37.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.140+0000 7fe274a00640 1 Processor -- start 2024-03-04T09:27:37.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.140+0000 7fe274a00640 1 -- start start 2024-03-04T09:27:37.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.141+0000 7fe274a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe270070f10 0x7fe2700abda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:37.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.141+0000 7fe274a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe2700cce90 0x7fe27015a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:37.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.141+0000 7fe274a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe270145300 0x7fe27015aef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:37.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.141+0000 7fe274a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe27015cab0 con 0x7fe270070f10 2024-03-04T09:27:37.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.141+0000 7fe274a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2701456e0 con 0x7fe2700cce90 2024-03-04T09:27:37.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.141+0000 7fe274a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe27015cc30 con 0x7fe270145300 2024-03-04T09:27:37.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.141+0000 7fe26e800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe270070f10 0x7fe2700abda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:37.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.141+0000 7fe26e800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe270070f10 0x7fe2700abda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:47628/0 (socket says 172.21.15.6:47628) 2024-03-04T09:27:37.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.141+0000 7fe26e800640 1 -- 172.21.15.6:0/3069207292 learned_addr learned my addr 172.21.15.6:0/3069207292 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:27:37.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.142+0000 7fe26de00640 1 --2- 172.21.15.6:0/3069207292 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe2700cce90 0x7fe27015a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:37.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.142+0000 7fe26f200640 1 --2- 172.21.15.6:0/3069207292 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe270145300 0x7fe27015aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:37.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.142+0000 7fe26e800640 1 -- 172.21.15.6:0/3069207292 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe270145300 msgr2=0x7fe27015aef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:37.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.142+0000 7fe26e800640 1 --2- 172.21.15.6:0/3069207292 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe270145300 0x7fe27015aef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:37.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.142+0000 7fe26e800640 1 -- 172.21.15.6:0/3069207292 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe2700cce90 msgr2=0x7fe27015a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:37.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.142+0000 7fe26e800640 1 --2- 172.21.15.6:0/3069207292 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe2700cce90 0x7fe27015a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:37.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.142+0000 7fe26e800640 1 -- 172.21.15.6:0/3069207292 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2701e6da0 con 0x7fe270070f10 2024-03-04T09:27:37.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.142+0000 7fe26e800640 1 --2- 172.21.15.6:0/3069207292 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe270070f10 0x7fe2700abda0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fe26000ba50 tx=0x7fe26000bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:37.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.142+0000 7fe266a00640 1 -- 172.21.15.6:0/3069207292 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe260019070 con 0x7fe270070f10 2024-03-04T09:27:37.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.142+0000 7fe266a00640 1 -- 172.21.15.6:0/3069207292 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe26000cc20 con 0x7fe270070f10 2024-03-04T09:27:37.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.142+0000 7fe266a00640 1 -- 172.21.15.6:0/3069207292 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe260014680 con 0x7fe270070f10 2024-03-04T09:27:37.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.142+0000 7fe274a00640 1 -- 172.21.15.6:0/3069207292 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2701e7080 con 0x7fe270070f10 2024-03-04T09:27:37.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.143+0000 7fe274a00640 1 -- 172.21.15.6:0/3069207292 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2700a97f0 con 0x7fe270070f10 2024-03-04T09:27:37.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.144+0000 7fe274a00640 1 -- 172.21.15.6:0/3069207292 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe270067830 con 0x7fe270070f10 2024-03-04T09:27:37.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.145+0000 7fe266a00640 1 -- 172.21.15.6:0/3069207292 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe26001c070 con 0x7fe270070f10 2024-03-04T09:27:37.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.146+0000 7fe266a00640 1 --2- 172.21.15.6:0/3069207292 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fe24c06b4d0 0x7fe24c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:37.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.146+0000 7fe266a00640 1 -- 172.21.15.6:0/3069207292 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe260089ed0 con 0x7fe270070f10 2024-03-04T09:27:37.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.146+0000 7fe26de00640 1 --2- 172.21.15.6:0/3069207292 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fe24c06b4d0 0x7fe24c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:37.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.147+0000 7fe26de00640 1 --2- 172.21.15.6:0/3069207292 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fe24c06b4d0 0x7fe24c06d990 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fe27015bf00 tx=0x7fe258008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:37.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.151+0000 7fe266a00640 1 -- 172.21.15.6:0/3069207292 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe270067830 con 0x7fe270070f10 2024-03-04T09:27:37.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.354+0000 7fe274a00640 1 -- 172.21.15.6:0/3069207292 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe27004f330 con 0x7fe270070f10 2024-03-04T09:27:37.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.356+0000 7fe266a00640 1 -- 172.21.15.6:0/3069207292 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v44) v1 ==== 74+0+16565 (secure 0 0 0) 0x7fe260058700 con 0x7fe270070f10 2024-03-04T09:27:37.353 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:27:37.353 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":44,"fsid":"edce0578-5ae4-4402-b45a-552bdef84d7c","created":"2024-03-04T09:24:54.523780+0000","modified":"2024-03-04T09:27:18.307032+0000","last_up_change":"2024-03-04T09:25:07.089049+0000","last_in_change":"2024-03-04T09:24:57.982077+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T09:25:07.935198+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-03-04T09:27:14.657490+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"43","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-03-04T09:27:15.806886+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a96574f-81f1-4bc6-8463-757520b84a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6825","nonce":1364449401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6827","nonce":1364449401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6831","nonce":1364449401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6829","nonce":1364449401}]},"public_addr":"172.21.15.6:6825/1364449401","cluster_addr":"172.21.15.6:6827/1364449401","heartbeat_back_addr":"172.21.15.6:6831/1364449401","heartbeat_front_addr":"172.21.15.6:6829/1364449401","state":["exists","up"]},{"osd":1,"uuid":"7d79b9c1-6b87-476a-bc36-7db5ee64cd90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6809","nonce":2494103078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6811","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6813","nonce":2494103078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6821","nonce":2494103078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6817","nonce":2494103078}]},"public_addr":"172.21.15.6:6809/2494103078","cluster_addr":"172.21.15.6:6813/2494103078","heartbeat_back_addr":"172.21.15.6:6821/2494103078","heartbeat_front_addr":"172.21.15.6:6817/2494103078","state":["exists","up"]},{"osd":2,"uuid":"492331b9-4ee9-40b6-bacd-85d44cd6c45a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6812","nonce":1500066681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6816","nonce":1500066681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6824","nonce":1500066681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6820","nonce":1500066681}]},"public_addr":"172.21.15.6:6812/1500066681","cluster_addr":"172.21.15.6:6816/1500066681","heartbeat_back_addr":"172.21.15.6:6824/1500066681","heartbeat_front_addr":"172.21.15.6:6820/1500066681","state":["exists","up"]},{"osd":3,"uuid":"21bb2efe-9ba7-4bdc-bc31-180cd1102e21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6801","nonce":1558806123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6803","nonce":1558806123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6807","nonce":1558806123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6805","nonce":1558806123}]},"public_addr":"172.21.15.6:6801/1558806123","cluster_addr":"172.21.15.6:6803/1558806123","heartbeat_back_addr":"172.21.15.6:6807/1558806123","heartbeat_front_addr":"172.21.15.6:6805/1558806123","state":["exists","up"]},{"osd":4,"uuid":"6e2c4b71-19d9-4737-83d9-558705d8644a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6801","nonce":1526135780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6803","nonce":1526135780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6807","nonce":1526135780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6805","nonce":1526135780}]},"public_addr":"172.21.15.146:6801/1526135780","cluster_addr":"172.21.15.146:6803/1526135780","heartbeat_back_addr":"172.21.15.146:6807/1526135780","heartbeat_front_addr":"172.21.15.146:6805/1526135780","state":["exists","up"]},{"osd":5,"uuid":"fb69c525-f1be-4b7d-ab32-6c29b0d6fddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6809","nonce":460262507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6811","nonce":460262507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6815","nonce":460262507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6813","nonce":460262507}]},"public_addr":"172.21.15.146:6809/460262507","cluster_addr":"172.21.15.146:6811/460262507","heartbeat_back_addr":"172.21.15.146:6815/460262507","heartbeat_front_addr":"172.21.15.146:6813/460262507","state":["exists","up"]},{"osd":6,"uuid":"c32e7fed-370d-4401-8ce0-6a4335b0d168","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6823","nonce":1392897493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6827","nonce":1392897493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6831","nonce":1392897493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6829","nonce":1392897493}]},"public_addr":"172.21.15.146:6823/1392897493","cluster_addr":"172.21.15.146:6827/1392897493","heartbeat_back_addr":"172.21.15.146:6831/1392897493","heartbeat_front_addr":"172.21.15.146:6829/1392897493","state":["exists","up"]},{"osd":7,"uuid":"75a79b44-fa5f-426d-a7d3-71f21f78835c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6817","nonce":3268323156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6819","nonce":3268323156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6825","nonce":3268323156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6822","nonce":3268323156}]},"public_addr":"172.21.15.146:6817/3268323156","cluster_addr":"172.21.15.146:6819/3268323156","heartbeat_back_addr":"172.21.15.146:6825/3268323156","heartbeat_front_addr":"172.21.15.146:6822/3268323156","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.310437+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.295850+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.294674+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.526855+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.300023+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.595160+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.346046+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.558741+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T09:27:37.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.358+0000 7fe274a00640 1 -- 172.21.15.6:0/3069207292 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fe24c06b4d0 msgr2=0x7fe24c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:37.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.358+0000 7fe274a00640 1 --2- 172.21.15.6:0/3069207292 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fe24c06b4d0 0x7fe24c06d990 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fe27015bf00 tx=0x7fe258008040 comp rx=0 tx=0).stop 2024-03-04T09:27:37.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.358+0000 7fe274a00640 1 -- 172.21.15.6:0/3069207292 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe270070f10 msgr2=0x7fe2700abda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:37.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.358+0000 7fe274a00640 1 --2- 172.21.15.6:0/3069207292 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe270070f10 0x7fe2700abda0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fe26000ba50 tx=0x7fe26000bf10 comp rx=0 tx=0).stop 2024-03-04T09:27:37.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.358+0000 7fe274a00640 1 -- 172.21.15.6:0/3069207292 shutdown_connections 2024-03-04T09:27:37.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.358+0000 7fe274a00640 1 --2- 172.21.15.6:0/3069207292 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fe24c06b4d0 0x7fe24c06d990 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:37.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.358+0000 7fe274a00640 1 --2- 172.21.15.6:0/3069207292 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe270145300 0x7fe27015aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:37.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.358+0000 7fe274a00640 1 --2- 172.21.15.6:0/3069207292 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe2700cce90 0x7fe27015a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:37.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.358+0000 7fe274a00640 1 --2- 172.21.15.6:0/3069207292 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe270070f10 0x7fe2700abda0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:37.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.359+0000 7fe274a00640 1 -- 172.21.15.6:0/3069207292 >> 172.21.15.6:0/3069207292 conn(0x7fe27006b3d0 msgr2=0x7fe27015d040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:37.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.359+0000 7fe274a00640 1 -- 172.21.15.6:0/3069207292 shutdown_connections 2024-03-04T09:27:37.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.359+0000 7fe274a00640 1 -- 172.21.15.6:0/3069207292 wait complete. 2024-03-04T09:27:37.374 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T09:25:07.935198+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-04T09:27:14.657490+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '43', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_data', 'create_time': '2024-03-04T09:27:15.806886+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '44', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-04T09:27:37.374 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-04T09:27:37.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.601+0000 7fdb41c00640 1 Processor -- start 2024-03-04T09:27:37.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.602+0000 7fdb41c00640 1 -- start start 2024-03-04T09:27:37.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.603+0000 7fdb41c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdb3c162280 0x7fdb3c165d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:37.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.603+0000 7fdb41c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fdb3c0b1b40 0x7fdb3c0b1fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:37.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.603+0000 7fdb41c00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb3c166cb0 con 0x7fdb3c0b4a90 2024-03-04T09:27:37.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.603+0000 7fdb41c00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb3c166410 con 0x7fdb3c0b1b40 2024-03-04T09:27:37.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.603+0000 7fdb41c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb3c1666f0 con 0x7fdb3c162280 2024-03-04T09:27:37.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.604+0000 7fdb41200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fdb3c0b1b40 0x7fdb3c0b1fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:37.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.604+0000 7fdb41200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fdb3c0b1b40 0x7fdb3c0b1fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:47644/0 (socket says 172.21.15.6:47644) 2024-03-04T09:27:37.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.604+0000 7fdb41200640 1 -- 172.21.15.6:0/3581455183 learned_addr learned my addr 172.21.15.6:0/3581455183 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:27:37.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.604+0000 7fdb3be00640 1 --2- 172.21.15.6:0/3581455183 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdb3c162280 0x7fdb3c165d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:37.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.604+0000 7fdb41200640 1 -- 172.21.15.6:0/3581455183 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdb3c162280 msgr2=0x7fdb3c165d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:27:37.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.604+0000 7fdb41200640 1 --2- 172.21.15.6:0/3581455183 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdb3c162280 0x7fdb3c165d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:37.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.604+0000 7fdb41200640 1 -- 172.21.15.6:0/3581455183 >> v1:172.21.15.146:6789/0 conn(0x7fdb3c0b4a90 legacy=0x7fdb3c0cc430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:27:37.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.604+0000 7fdb41200640 1 -- 172.21.15.6:0/3581455183 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb3c1669d0 con 0x7fdb3c0b1b40 2024-03-04T09:27:37.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.605+0000 7fdb41200640 1 --2- 172.21.15.6:0/3581455183 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdb3c0b1b40 0x7fdb3c0b1fd0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fdb34009e50 tx=0x7fdb3402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c930bab217b980b server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:37.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.606+0000 7fdb3aa00640 1 -- 172.21.15.6:0/3581455183 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb3402c9e0 con 0x7fdb3c0b1b40 2024-03-04T09:27:37.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.606+0000 7fdb3aa00640 1 -- 172.21.15.6:0/3581455183 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb3402cb80 con 0x7fdb3c0b1b40 2024-03-04T09:27:37.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.606+0000 7fdb3aa00640 1 -- 172.21.15.6:0/3581455183 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb340343f0 con 0x7fdb3c0b1b40 2024-03-04T09:27:37.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.606+0000 7fdb41c00640 1 -- 172.21.15.6:0/3581455183 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdb3c0b1b40 msgr2=0x7fdb3c0b1fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:37.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.606+0000 7fdb41c00640 1 --2- 172.21.15.6:0/3581455183 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdb3c0b1b40 0x7fdb3c0b1fd0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fdb34009e50 tx=0x7fdb3402be40 comp rx=0 tx=0).stop 2024-03-04T09:27:37.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.607+0000 7fdb41c00640 1 -- 172.21.15.6:0/3581455183 shutdown_connections 2024-03-04T09:27:37.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.607+0000 7fdb41c00640 1 --2- 172.21.15.6:0/3581455183 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdb3c0b1b40 0x7fdb3c0b1fd0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:37.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.607+0000 7fdb41c00640 1 --2- 172.21.15.6:0/3581455183 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdb3c162280 0x7fdb3c165d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:37.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.607+0000 7fdb41c00640 1 -- 172.21.15.6:0/3581455183 >> 172.21.15.6:0/3581455183 conn(0x7fdb3c06b3a0 msgr2=0x7fdb3c06b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:37.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.607+0000 7fdb41c00640 1 -- 172.21.15.6:0/3581455183 shutdown_connections 2024-03-04T09:27:37.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.607+0000 7fdb41c00640 1 -- 172.21.15.6:0/3581455183 wait complete. 2024-03-04T09:27:37.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.608+0000 7fdb41c00640 1 Processor -- start 2024-03-04T09:27:37.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.609+0000 7fdb41c00640 1 -- start start 2024-03-04T09:27:37.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.610+0000 7fdb41c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdb3c0b1b40 0x7fdb3c159b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:37.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.610+0000 7fdb41c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdb3c0b4a90 0x7fdb3c15fba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:37.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.610+0000 7fdb41c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdb3c162280 0x7fdb3c1ebbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:37.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.610+0000 7fdb41c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb3c0716e0 con 0x7fdb3c162280 2024-03-04T09:27:37.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.610+0000 7fdb41c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb3c071560 con 0x7fdb3c0b1b40 2024-03-04T09:27:37.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.610+0000 7fdb41c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb3c071860 con 0x7fdb3c0b4a90 2024-03-04T09:27:37.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.610+0000 7fdb41200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdb3c162280 0x7fdb3c1ebbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:37.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.610+0000 7fdb41200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdb3c162280 0x7fdb3c1ebbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:47658/0 (socket says 172.21.15.6:47658) 2024-03-04T09:27:37.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.610+0000 7fdb41200640 1 -- 172.21.15.6:0/977889756 learned_addr learned my addr 172.21.15.6:0/977889756 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:27:37.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.610+0000 7fdb3b400640 1 --2- 172.21.15.6:0/977889756 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdb3c0b4a90 0x7fdb3c15fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:37.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.610+0000 7fdb3be00640 1 --2- 172.21.15.6:0/977889756 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdb3c0b1b40 0x7fdb3c159b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:37.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.610+0000 7fdb41200640 1 -- 172.21.15.6:0/977889756 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdb3c0b4a90 msgr2=0x7fdb3c15fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:37.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.611+0000 7fdb41200640 1 --2- 172.21.15.6:0/977889756 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdb3c0b4a90 0x7fdb3c15fba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:37.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.611+0000 7fdb41200640 1 -- 172.21.15.6:0/977889756 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdb3c0b1b40 msgr2=0x7fdb3c159b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:37.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.611+0000 7fdb41200640 1 --2- 172.21.15.6:0/977889756 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdb3c0b1b40 0x7fdb3c159b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:37.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.611+0000 7fdb41200640 1 -- 172.21.15.6:0/977889756 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb3c15a050 con 0x7fdb3c162280 2024-03-04T09:27:37.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.611+0000 7fdb3be00640 1 --2- 172.21.15.6:0/977889756 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdb3c0b1b40 0x7fdb3c159b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:27:37.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.611+0000 7fdb41200640 1 --2- 172.21.15.6:0/977889756 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdb3c162280 0x7fdb3c1ebbb0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fdb34036040 tx=0x7fdb34004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:37.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.611+0000 7fdb33e00640 1 -- 172.21.15.6:0/977889756 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb34043020 con 0x7fdb3c162280 2024-03-04T09:27:37.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.611+0000 7fdb33e00640 1 -- 172.21.15.6:0/977889756 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb3403e450 con 0x7fdb3c162280 2024-03-04T09:27:37.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.611+0000 7fdb33e00640 1 -- 172.21.15.6:0/977889756 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb34009510 con 0x7fdb3c162280 2024-03-04T09:27:37.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.611+0000 7fdb41c00640 1 -- 172.21.15.6:0/977889756 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb3c1600e0 con 0x7fdb3c162280 2024-03-04T09:27:37.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.612+0000 7fdb41c00640 1 -- 172.21.15.6:0/977889756 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb3c160380 con 0x7fdb3c162280 2024-03-04T09:27:37.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.613+0000 7fdb41c00640 1 -- 172.21.15.6:0/977889756 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb3c04f330 con 0x7fdb3c162280 2024-03-04T09:27:37.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.614+0000 7fdb33e00640 1 -- 172.21.15.6:0/977889756 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdb34046020 con 0x7fdb3c162280 2024-03-04T09:27:37.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.614+0000 7fdb33e00640 1 --2- 172.21.15.6:0/977889756 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fdb1c06b520 0x7fdb1c06d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:37.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.615+0000 7fdb33e00640 1 -- 172.21.15.6:0/977889756 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdb340acaa0 con 0x7fdb3c162280 2024-03-04T09:27:37.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.615+0000 7fdb3be00640 1 --2- 172.21.15.6:0/977889756 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fdb1c06b520 0x7fdb1c06d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:37.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.616+0000 7fdb3be00640 1 --2- 172.21.15.6:0/977889756 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fdb1c06b520 0x7fdb1c06d9e0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fdb2c006fd0 tx=0x7fdb2c008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:37.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.619+0000 7fdb33e00640 1 -- 172.21.15.6:0/977889756 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdb3404d030 con 0x7fdb3c162280 2024-03-04T09:27:37.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.821+0000 7fdb41c00640 1 -- 172.21.15.6:0/977889756 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fdb3c06e490 con 0x7fdb3c162280 2024-03-04T09:27:37.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.822+0000 7fdb33e00640 1 -- 172.21.15.6:0/977889756 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v44) v1 ==== 93+0+10 (secure 0 0 0) 0x7fdb3407b900 con 0x7fdb3c162280 2024-03-04T09:27:37.819 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-04T09:27:37.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.824+0000 7fdb41c00640 1 -- 172.21.15.6:0/977889756 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fdb1c06b520 msgr2=0x7fdb1c06d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:37.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.824+0000 7fdb41c00640 1 --2- 172.21.15.6:0/977889756 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fdb1c06b520 0x7fdb1c06d9e0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fdb2c006fd0 tx=0x7fdb2c008040 comp rx=0 tx=0).stop 2024-03-04T09:27:37.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.825+0000 7fdb41c00640 1 -- 172.21.15.6:0/977889756 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdb3c162280 msgr2=0x7fdb3c1ebbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:37.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.825+0000 7fdb41c00640 1 --2- 172.21.15.6:0/977889756 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdb3c162280 0x7fdb3c1ebbb0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fdb34036040 tx=0x7fdb34004a40 comp rx=0 tx=0).stop 2024-03-04T09:27:37.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.825+0000 7fdb41c00640 1 -- 172.21.15.6:0/977889756 shutdown_connections 2024-03-04T09:27:37.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.825+0000 7fdb41c00640 1 --2- 172.21.15.6:0/977889756 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdb3c162280 0x7fdb3c1ebbb0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:37.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.825+0000 7fdb41c00640 1 --2- 172.21.15.6:0/977889756 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fdb1c06b520 0x7fdb1c06d9e0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:37.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.825+0000 7fdb41c00640 1 --2- 172.21.15.6:0/977889756 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdb3c0b4a90 0x7fdb3c15fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:37.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.825+0000 7fdb41c00640 1 --2- 172.21.15.6:0/977889756 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdb3c0b1b40 0x7fdb3c159b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:37.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.825+0000 7fdb41c00640 1 -- 172.21.15.6:0/977889756 >> 172.21.15.6:0/977889756 conn(0x7fdb3c06b3a0 msgr2=0x7fdb3c15d170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:37.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.825+0000 7fdb41c00640 1 -- 172.21.15.6:0/977889756 shutdown_connections 2024-03-04T09:27:37.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:37.825+0000 7fdb41c00640 1 -- 172.21.15.6:0/977889756 wait complete. 2024-03-04T09:27:37.842 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-04T09:27:38.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.073+0000 7ff737e00640 1 Processor -- start 2024-03-04T09:27:38.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.073+0000 7ff737e00640 1 -- start start 2024-03-04T09:27:38.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.075+0000 7ff737e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ff73815dee0 0x7ff7380d18a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:38.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.075+0000 7ff737e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff7380d0d40 0x7ff7380b3510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:38.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.075+0000 7ff737e00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7380ceca0 con 0x7ff7380d4860 2024-03-04T09:27:38.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.075+0000 7ff737e00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7380cf5d0 con 0x7ff73815dee0 2024-03-04T09:27:38.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.075+0000 7ff737e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7380cef80 con 0x7ff7380d0d40 2024-03-04T09:27:38.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.076+0000 7ff737400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff7380d0d40 0x7ff7380b3510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:38.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.076+0000 7ff737400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff7380d0d40 0x7ff7380b3510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.6:60854/0 (socket says 172.21.15.6:60854) 2024-03-04T09:27:38.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.076+0000 7ff737400640 1 -- 172.21.15.6:0/3450367068 learned_addr learned my addr 172.21.15.6:0/3450367068 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:27:38.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.076+0000 7ff736a00640 1 --2- 172.21.15.6:0/3450367068 >> v2:172.21.15.146:3300/0 conn(0x7ff73815dee0 0x7ff7380d18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:38.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.077+0000 7ff736a00640 1 -- 172.21.15.6:0/3450367068 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff7380d0d40 msgr2=0x7ff7380b3510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:27:38.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.077+0000 7ff736a00640 1 --2- 172.21.15.6:0/3450367068 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff7380d0d40 0x7ff7380b3510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:38.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.077+0000 7ff736a00640 1 -- 172.21.15.6:0/3450367068 >> v1:172.21.15.146:6789/0 conn(0x7ff7380d4860 legacy=0x7ff7380d0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:38.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.077+0000 7ff736a00640 1 -- 172.21.15.6:0/3450367068 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7380cf260 con 0x7ff73815dee0 2024-03-04T09:27:38.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.077+0000 7ff737400640 1 --2- 172.21.15.6:0/3450367068 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff7380d0d40 0x7ff7380b3510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T09:27:38.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.079+0000 7ff736a00640 1 --2- 172.21.15.6:0/3450367068 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff73815dee0 0x7ff7380d18a0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7ff7280097a0 tx=0x7ff72802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=895358b7587fa24d server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:38.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.079+0000 7ff735600640 1 -- 172.21.15.6:0/3450367068 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff728035020 con 0x7ff73815dee0 2024-03-04T09:27:38.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.079+0000 7ff735600640 1 -- 172.21.15.6:0/3450367068 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff728031830 con 0x7ff73815dee0 2024-03-04T09:27:38.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.079+0000 7ff735600640 1 -- 172.21.15.6:0/3450367068 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff728031c00 con 0x7ff73815dee0 2024-03-04T09:27:38.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.080+0000 7ff737e00640 1 -- 172.21.15.6:0/3450367068 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff73815dee0 msgr2=0x7ff7380d18a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:38.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.080+0000 7ff737e00640 1 --2- 172.21.15.6:0/3450367068 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff73815dee0 0x7ff7380d18a0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7ff7280097a0 tx=0x7ff72802b8d0 comp rx=0 tx=0).stop 2024-03-04T09:27:38.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.080+0000 7ff737e00640 1 -- 172.21.15.6:0/3450367068 shutdown_connections 2024-03-04T09:27:38.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.080+0000 7ff737e00640 1 --2- 172.21.15.6:0/3450367068 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff7380d0d40 0x7ff7380b3510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:38.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.080+0000 7ff737e00640 1 --2- 172.21.15.6:0/3450367068 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff73815dee0 0x7ff7380d18a0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:38.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.080+0000 7ff737e00640 1 -- 172.21.15.6:0/3450367068 >> 172.21.15.6:0/3450367068 conn(0x7ff7380d3cd0 msgr2=0x7ff7380d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:38.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.080+0000 7ff737e00640 1 -- 172.21.15.6:0/3450367068 shutdown_connections 2024-03-04T09:27:38.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.081+0000 7ff737e00640 1 -- 172.21.15.6:0/3450367068 wait complete. 2024-03-04T09:27:38.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.083+0000 7ff737e00640 1 Processor -- start 2024-03-04T09:27:38.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.083+0000 7ff737e00640 1 -- start start 2024-03-04T09:27:38.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.084+0000 7ff737e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff7380d0d40 0x7ff7380c8c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:38.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.084+0000 7ff737e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff7380d4860 0x7ff738158d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:38.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.084+0000 7ff737e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff73815dee0 0x7ff7381604a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:38.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.084+0000 7ff737e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7380d1de0 con 0x7ff7380d4860 2024-03-04T09:27:38.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.084+0000 7ff737e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff73815e2c0 con 0x7ff7380d0d40 2024-03-04T09:27:38.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.084+0000 7ff737e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7380d1f60 con 0x7ff73815dee0 2024-03-04T09:27:38.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.085+0000 7ff736000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff7380d4860 0x7ff738158d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:38.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.085+0000 7ff736000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff7380d4860 0x7ff738158d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:47666/0 (socket says 172.21.15.6:47666) 2024-03-04T09:27:38.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.085+0000 7ff736000640 1 -- 172.21.15.6:0/37767729 learned_addr learned my addr 172.21.15.6:0/37767729 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:27:38.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.085+0000 7ff737400640 1 --2- 172.21.15.6:0/37767729 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff73815dee0 0x7ff7381604a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:38.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.085+0000 7ff736a00640 1 --2- 172.21.15.6:0/37767729 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff7380d0d40 0x7ff7380c8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:38.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.085+0000 7ff736000640 1 -- 172.21.15.6:0/37767729 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff73815dee0 msgr2=0x7ff7381604a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:38.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.085+0000 7ff736000640 1 --2- 172.21.15.6:0/37767729 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff73815dee0 0x7ff7381604a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:38.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.085+0000 7ff736000640 1 -- 172.21.15.6:0/37767729 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff7380d0d40 msgr2=0x7ff7380c8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:38.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.085+0000 7ff736000640 1 --2- 172.21.15.6:0/37767729 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff7380d0d40 0x7ff7380c8c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:38.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.085+0000 7ff736000640 1 -- 172.21.15.6:0/37767729 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7381592b0 con 0x7ff7380d4860 2024-03-04T09:27:38.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.086+0000 7ff736000640 1 --2- 172.21.15.6:0/37767729 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff7380d4860 0x7ff738158d70 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7ff72400c340 tx=0x7ff72400c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:38.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.086+0000 7ff72ea00640 1 -- 172.21.15.6:0/37767729 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff724017070 con 0x7ff7380d4860 2024-03-04T09:27:38.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.086+0000 7ff72ea00640 1 -- 172.21.15.6:0/37767729 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7240052d0 con 0x7ff7380d4860 2024-03-04T09:27:38.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.086+0000 7ff72ea00640 1 -- 172.21.15.6:0/37767729 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff724003850 con 0x7ff7380d4860 2024-03-04T09:27:38.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.086+0000 7ff737e00640 1 -- 172.21.15.6:0/37767729 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7380c7bc0 con 0x7ff7380d4860 2024-03-04T09:27:38.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.086+0000 7ff737e00640 1 -- 172.21.15.6:0/37767729 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7380c8180 con 0x7ff7380d4860 2024-03-04T09:27:38.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.087+0000 7ff737e00640 1 -- 172.21.15.6:0/37767729 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7380c8790 con 0x7ff7380d4860 2024-03-04T09:27:38.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.088+0000 7ff72ea00640 1 -- 172.21.15.6:0/37767729 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff72401a070 con 0x7ff7380d4860 2024-03-04T09:27:38.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.089+0000 7ff72ea00640 1 --2- 172.21.15.6:0/37767729 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff71806b4d0 0x7ff71806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:38.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.089+0000 7ff72ea00640 1 -- 172.21.15.6:0/37767729 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff7240894d0 con 0x7ff7380d4860 2024-03-04T09:27:38.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.090+0000 7ff736a00640 1 --2- 172.21.15.6:0/37767729 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff71806b4d0 0x7ff71806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:38.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.091+0000 7ff736a00640 1 --2- 172.21.15.6:0/37767729 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff71806b4d0 0x7ff71806d990 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7ff728036000 tx=0x7ff728008030 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:38.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.096+0000 7ff72ea00640 1 -- 172.21.15.6:0/37767729 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff724058340 con 0x7ff7380d4860 2024-03-04T09:27:38.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.297+0000 7ff737e00640 1 -- 172.21.15.6:0/37767729 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ff73804f330 con 0x7ff7380d4860 2024-03-04T09:27:38.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.298+0000 7ff72ea00640 1 -- 172.21.15.6:0/37767729 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v44) v1 ==== 104+0+11 (secure 0 0 0) 0x7ff724057c40 con 0x7ff7380d4860 2024-03-04T09:27:38.295 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-03-04T09:27:38.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.301+0000 7ff737e00640 1 -- 172.21.15.6:0/37767729 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff71806b4d0 msgr2=0x7ff71806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:38.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.301+0000 7ff737e00640 1 --2- 172.21.15.6:0/37767729 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff71806b4d0 0x7ff71806d990 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7ff728036000 tx=0x7ff728008030 comp rx=0 tx=0).stop 2024-03-04T09:27:38.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.301+0000 7ff737e00640 1 -- 172.21.15.6:0/37767729 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff7380d4860 msgr2=0x7ff738158d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:38.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.301+0000 7ff737e00640 1 --2- 172.21.15.6:0/37767729 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff7380d4860 0x7ff738158d70 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7ff72400c340 tx=0x7ff72400c800 comp rx=0 tx=0).stop 2024-03-04T09:27:38.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.301+0000 7ff737e00640 1 -- 172.21.15.6:0/37767729 shutdown_connections 2024-03-04T09:27:38.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.301+0000 7ff737e00640 1 --2- 172.21.15.6:0/37767729 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff71806b4d0 0x7ff71806d990 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:38.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.301+0000 7ff737e00640 1 --2- 172.21.15.6:0/37767729 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff73815dee0 0x7ff7381604a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:38.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.301+0000 7ff737e00640 1 --2- 172.21.15.6:0/37767729 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff7380d4860 0x7ff738158d70 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:38.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.301+0000 7ff737e00640 1 --2- 172.21.15.6:0/37767729 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff7380d0d40 0x7ff7380c8c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:38.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.301+0000 7ff737e00640 1 -- 172.21.15.6:0/37767729 >> 172.21.15.6:0/37767729 conn(0x7ff7380d3cd0 msgr2=0x7ff7380b7290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:38.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.301+0000 7ff737e00640 1 -- 172.21.15.6:0/37767729 shutdown_connections 2024-03-04T09:27:38.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.301+0000 7ff737e00640 1 -- 172.21.15.6:0/37767729 wait complete. 2024-03-04T09:27:38.316 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-04T09:27:38.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.551+0000 7f16cf200640 1 Processor -- start 2024-03-04T09:27:38.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.552+0000 7f16cf200640 1 -- start start 2024-03-04T09:27:38.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.553+0000 7f16cf200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f16d006cab0 0x7f16d006ce90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:38.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.553+0000 7f16cf200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f16d015e2d0 0x7f16d00b5450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:38.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.553+0000 7f16cf200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16d006d3d0 con 0x7f16d00b1620 2024-03-04T09:27:38.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.554+0000 7f16cf200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16d00b5da0 con 0x7f16d006cab0 2024-03-04T09:27:38.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.554+0000 7f16cf200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16d00b6080 con 0x7f16d015e2d0 2024-03-04T09:27:38.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.554+0000 7f16ce800640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f16d00b1620 0x7f16d00b1ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:39824/0 (socket says 172.21.15.6:39824) 2024-03-04T09:27:38.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.554+0000 7f16ce800640 1 -- 172.21.15.6:0/1265686874 learned_addr learned my addr 172.21.15.6:0/1265686874 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:27:38.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.554+0000 7f16cde00640 1 --2- 172.21.15.6:0/1265686874 >> v2:172.21.15.146:3300/0 conn(0x7f16d006cab0 0x7f16d006ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:38.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.554+0000 7f16c7e00640 1 --2- 172.21.15.6:0/1265686874 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f16d015e2d0 0x7f16d00b5450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:38.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.554+0000 7f16cd400640 1 -- 172.21.15.6:0/1265686874 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1645403712 0 0) 0x7f16d006d3d0 con 0x7f16d00b1620 2024-03-04T09:27:38.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.554+0000 7f16cd400640 1 -- 172.21.15.6:0/1265686874 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16b4003520 con 0x7f16d00b1620 2024-03-04T09:27:38.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.555+0000 7f16cd400640 1 -- 172.21.15.6:0/1265686874 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1510214620 0 0) 0x7f16b4003520 con 0x7f16d00b1620 2024-03-04T09:27:38.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.555+0000 7f16cd400640 1 -- 172.21.15.6:0/1265686874 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f16d015e2d0 msgr2=0x7f16d00b5450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:27:38.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.555+0000 7f16cd400640 1 --2- 172.21.15.6:0/1265686874 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f16d015e2d0 0x7f16d00b5450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:38.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.555+0000 7f16cd400640 1 -- 172.21.15.6:0/1265686874 >> v2:172.21.15.146:3300/0 conn(0x7f16d006cab0 msgr2=0x7f16d006ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:27:38.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.555+0000 7f16cd400640 1 --2- 172.21.15.6:0/1265686874 >> v2:172.21.15.146:3300/0 conn(0x7f16d006cab0 0x7f16d006ce90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:38.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.555+0000 7f16cd400640 1 -- 172.21.15.6:0/1265686874 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16d00b6360 con 0x7f16d00b1620 2024-03-04T09:27:38.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.555+0000 7f16cd400640 1 -- 172.21.15.6:0/1265686874 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7f16c80019f0 con 0x7f16d00b1620 2024-03-04T09:27:38.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.555+0000 7f16c7e00640 1 --2- 172.21.15.6:0/1265686874 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f16d015e2d0 0x7f16d00b5450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T09:27:38.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.555+0000 7f16cd400640 1 -- 172.21.15.6:0/1265686874 >> v1:172.21.15.6:6789/0 conn(0x7f16d00b1620 legacy=0x7f16d00b1ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:38.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.555+0000 7f16cd400640 1 --2- 172.21.15.6:0/1265686874 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f16b40039b0 0x7f16b4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:38.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.555+0000 7f16cd400640 1 --2- 172.21.15.6:0/1265686874 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f16b4004800 0x7f16b4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:38.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.555+0000 7f16cd400640 1 --2- 172.21.15.6:0/1265686874 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f16d015e2d0 0x7f16b40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:38.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.555+0000 7f16cd400640 1 -- 172.21.15.6:0/1265686874 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16d00b5da0 con 0x7f16b4004800 2024-03-04T09:27:38.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.555+0000 7f16cd400640 1 -- 172.21.15.6:0/1265686874 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16d00b6080 con 0x7f16b40039b0 2024-03-04T09:27:38.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.555+0000 7f16cd400640 1 -- 172.21.15.6:0/1265686874 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16b400b9f0 con 0x7f16d015e2d0 2024-03-04T09:27:38.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.555+0000 7f16c7e00640 1 --2- 172.21.15.6:0/1265686874 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f16b4004800 0x7f16b4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:38.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.556+0000 7f16ce800640 1 --2- 172.21.15.6:0/1265686874 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f16d015e2d0 0x7f16b40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:38.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.556+0000 7f16cde00640 1 --2- 172.21.15.6:0/1265686874 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f16b40039b0 0x7f16b4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:38.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.556+0000 7f16c7e00640 1 -- 172.21.15.6:0/1265686874 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f16d015e2d0 msgr2=0x7f16b40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:27:38.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.556+0000 7f16c7e00640 1 --2- 172.21.15.6:0/1265686874 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f16d015e2d0 0x7f16b40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:38.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.556+0000 7f16c7e00640 1 -- 172.21.15.6:0/1265686874 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f16b40039b0 msgr2=0x7f16b4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:27:38.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.556+0000 7f16c7e00640 1 --2- 172.21.15.6:0/1265686874 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f16b40039b0 0x7f16b4003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:38.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.556+0000 7f16c7e00640 1 -- 172.21.15.6:0/1265686874 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16b4003520 con 0x7f16b4004800 2024-03-04T09:27:38.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.556+0000 7f16cde00640 1 --2- 172.21.15.6:0/1265686874 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f16b40039b0 0x7f16b4003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:27:38.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.557+0000 7f16c7e00640 1 --2- 172.21.15.6:0/1265686874 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f16b4004800 0x7f16b4004c80 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f16b80098a0 tx=0x7f16b4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9df21020863bb873 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:38.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.557+0000 7f16cd400640 1 -- 172.21.15.6:0/1265686874 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16b800f040 con 0x7f16b4004800 2024-03-04T09:27:38.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.557+0000 7f16cd400640 1 -- 172.21.15.6:0/1265686874 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16b8004230 con 0x7f16b4004800 2024-03-04T09:27:38.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.557+0000 7f16cd400640 1 -- 172.21.15.6:0/1265686874 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16b8003610 con 0x7f16b4004800 2024-03-04T09:27:38.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.558+0000 7f16cf200640 1 -- 172.21.15.6:0/1265686874 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f16b4004800 msgr2=0x7f16b4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:38.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.558+0000 7f16cf200640 1 --2- 172.21.15.6:0/1265686874 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f16b4004800 0x7f16b4004c80 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f16b80098a0 tx=0x7f16b4005830 comp rx=0 tx=0).stop 2024-03-04T09:27:38.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.558+0000 7f16ce800640 1 -- 172.21.15.6:0/1265686874 reap_dead start 2024-03-04T09:27:38.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.558+0000 7f16cf200640 1 -- 172.21.15.6:0/1265686874 shutdown_connections 2024-03-04T09:27:38.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.558+0000 7f16cf200640 1 -- 172.21.15.6:0/1265686874 >> 172.21.15.6:0/1265686874 conn(0x7f16d009aa10 msgr2=0x7f16d009ae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:38.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.558+0000 7f16cf200640 1 -- 172.21.15.6:0/1265686874 shutdown_connections 2024-03-04T09:27:38.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.558+0000 7f16cf200640 1 -- 172.21.15.6:0/1265686874 wait complete. 2024-03-04T09:27:38.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.560+0000 7f16cf200640 1 Processor -- start 2024-03-04T09:27:38.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.560+0000 7f16cf200640 1 -- start start 2024-03-04T09:27:38.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.561+0000 7f16cf200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f16b4004800 0x7f16d014bc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:38.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.561+0000 7f16cf200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f16d0157f20 0x7f16d01583a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:38.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.561+0000 7f16cf200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f16d00af510 0x7f16d00af990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:38.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.561+0000 7f16cf200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16d015e730 con 0x7f16b4004800 2024-03-04T09:27:38.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.561+0000 7f16cf200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16d00b6920 con 0x7f16d00af510 2024-03-04T09:27:38.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.561+0000 7f16cf200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16d015e8b0 con 0x7f16d0157f20 2024-03-04T09:27:38.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.561+0000 7f16cde00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f16b4004800 0x7f16d014bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:38.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.561+0000 7f16cde00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f16b4004800 0x7f16d014bc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:47684/0 (socket says 172.21.15.6:47684) 2024-03-04T09:27:38.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.561+0000 7f16cde00640 1 -- 172.21.15.6:0/3587889423 learned_addr learned my addr 172.21.15.6:0/3587889423 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:27:38.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.562+0000 7f16c7e00640 1 --2- 172.21.15.6:0/3587889423 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f16d0157f20 0x7f16d01583a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:38.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.562+0000 7f16ce800640 1 --2- 172.21.15.6:0/3587889423 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f16d00af510 0x7f16d00af990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:38.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.562+0000 7f16cde00640 1 -- 172.21.15.6:0/3587889423 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f16d0157f20 msgr2=0x7f16d01583a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:38.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.562+0000 7f16cde00640 1 --2- 172.21.15.6:0/3587889423 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f16d0157f20 0x7f16d01583a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:38.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.562+0000 7f16cde00640 1 -- 172.21.15.6:0/3587889423 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f16d00af510 msgr2=0x7f16d00af990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:38.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.562+0000 7f16cde00640 1 --2- 172.21.15.6:0/3587889423 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f16d00af510 0x7f16d00af990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:38.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.562+0000 7f16cde00640 1 -- 172.21.15.6:0/3587889423 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16d0149210 con 0x7f16b4004800 2024-03-04T09:27:38.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.562+0000 7f16cde00640 1 --2- 172.21.15.6:0/3587889423 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f16b4004800 0x7f16d014bc10 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f16c00109d0 tx=0x7f16c0010e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:38.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.562+0000 7f16c6000640 1 -- 172.21.15.6:0/3587889423 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16c000f040 con 0x7f16b4004800 2024-03-04T09:27:38.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.562+0000 7f16c6000640 1 -- 172.21.15.6:0/3587889423 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16c000ec20 con 0x7f16b4004800 2024-03-04T09:27:38.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.562+0000 7f16c6000640 1 -- 172.21.15.6:0/3587889423 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16c001c400 con 0x7f16b4004800 2024-03-04T09:27:38.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.562+0000 7f16cf200640 1 -- 172.21.15.6:0/3587889423 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16d00b0260 con 0x7f16b4004800 2024-03-04T09:27:38.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.563+0000 7f16cf200640 1 -- 172.21.15.6:0/3587889423 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16d0156400 con 0x7f16b4004800 2024-03-04T09:27:38.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.564+0000 7f16cf200640 1 -- 172.21.15.6:0/3587889423 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16d004f330 con 0x7f16b4004800 2024-03-04T09:27:38.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.565+0000 7f16c6000640 1 -- 172.21.15.6:0/3587889423 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f16c001c5a0 con 0x7f16b4004800 2024-03-04T09:27:38.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.566+0000 7f16c6000640 1 --2- 172.21.15.6:0/3587889423 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f16ac06b4d0 0x7f16ac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:38.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.566+0000 7f16c6000640 1 -- 172.21.15.6:0/3587889423 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f16c008aa30 con 0x7f16b4004800 2024-03-04T09:27:38.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.566+0000 7f16c7e00640 1 --2- 172.21.15.6:0/3587889423 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f16ac06b4d0 0x7f16ac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:38.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.568+0000 7f16c7e00640 1 --2- 172.21.15.6:0/3587889423 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f16ac06b4d0 0x7f16ac06d990 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f16b8035000 tx=0x7f16b803a040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:38.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.570+0000 7f16c6000640 1 -- 172.21.15.6:0/3587889423 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f16c0018030 con 0x7f16b4004800 2024-03-04T09:27:38.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.770+0000 7f16cf200640 1 -- 172.21.15.6:0/3587889423 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f16d0155ed0 con 0x7f16b4004800 2024-03-04T09:27:38.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.770+0000 7f16c6000640 1 -- 172.21.15.6:0/3587889423 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v44) v1 ==== 100+0+11 (secure 0 0 0) 0x7f16c0059250 con 0x7f16b4004800 2024-03-04T09:27:38.767 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 64 2024-03-04T09:27:38.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.773+0000 7f16cf200640 1 -- 172.21.15.6:0/3587889423 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f16ac06b4d0 msgr2=0x7f16ac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:38.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.773+0000 7f16cf200640 1 --2- 172.21.15.6:0/3587889423 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f16ac06b4d0 0x7f16ac06d990 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f16b8035000 tx=0x7f16b803a040 comp rx=0 tx=0).stop 2024-03-04T09:27:38.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.773+0000 7f16cf200640 1 -- 172.21.15.6:0/3587889423 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f16b4004800 msgr2=0x7f16d014bc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:38.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.773+0000 7f16cf200640 1 --2- 172.21.15.6:0/3587889423 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f16b4004800 0x7f16d014bc10 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f16c00109d0 tx=0x7f16c0010e90 comp rx=0 tx=0).stop 2024-03-04T09:27:38.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.773+0000 7f16cf200640 1 -- 172.21.15.6:0/3587889423 shutdown_connections 2024-03-04T09:27:38.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.773+0000 7f16cf200640 1 --2- 172.21.15.6:0/3587889423 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f16d00af510 0x7f16d00af990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:38.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.773+0000 7f16cf200640 1 --2- 172.21.15.6:0/3587889423 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f16ac06b4d0 0x7f16ac06d990 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:38.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.773+0000 7f16cf200640 1 --2- 172.21.15.6:0/3587889423 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f16d0157f20 0x7f16d01583a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:38.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.773+0000 7f16cf200640 1 --2- 172.21.15.6:0/3587889423 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f16b4004800 0x7f16d014bc10 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:38.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.773+0000 7f16cf200640 1 -- 172.21.15.6:0/3587889423 >> 172.21.15.6:0/3587889423 conn(0x7f16d009aa10 msgr2=0x7f16d015ea60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:38.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.773+0000 7f16cf200640 1 -- 172.21.15.6:0/3587889423 shutdown_connections 2024-03-04T09:27:38.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:38.774+0000 7f16cf200640 1 -- 172.21.15.6:0/3587889423 wait complete. 2024-03-04T09:27:38.790 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-04T09:27:39.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.018+0000 7f608b400640 1 Processor -- start 2024-03-04T09:27:39.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.018+0000 7f608b400640 1 -- start start 2024-03-04T09:27:39.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.020+0000 7f608b400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f60840712c0 0x7f6084162c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:39.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.020+0000 7f608b400640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60840ce090 con 0x7f6084072da0 2024-03-04T09:27:39.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.020+0000 7f608b400640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60840ce370 con 0x7f6084070ee0 2024-03-04T09:27:39.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.020+0000 7f608b400640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60840ceca0 con 0x7f60840712c0 2024-03-04T09:27:39.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.020+0000 7f607be00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f6084072da0 0x7f608406cfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:45244/0 (socket says 172.21.15.6:45244) 2024-03-04T09:27:39.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.020+0000 7f607be00640 1 -- 172.21.15.6:0/1501498822 learned_addr learned my addr 172.21.15.6:0/1501498822 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:27:39.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.020+0000 7f6089400640 1 --2- 172.21.15.6:0/1501498822 >> v2:172.21.15.6:3300/0 conn(0x7f60840712c0 0x7f6084162c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:39.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.020+0000 7f6083e00640 1 -- 172.21.15.6:0/1501498822 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 562137413 0 0) 0x7f60840ce090 con 0x7f6084072da0 2024-03-04T09:27:39.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.021+0000 7f6083e00640 1 -- 172.21.15.6:0/1501498822 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6064003520 con 0x7f6084072da0 2024-03-04T09:27:39.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.021+0000 7f6089400640 1 -- 172.21.15.6:0/1501498822 >> v1:172.21.15.146:6789/0 conn(0x7f6084070ee0 legacy=0x7f608406e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:39.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.021+0000 7f6089400640 1 -- 172.21.15.6:0/1501498822 >> v1:172.21.15.6:6789/0 conn(0x7f6084072da0 legacy=0x7f608406cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:39.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.021+0000 7f6089400640 1 -- 172.21.15.6:0/1501498822 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60840ce650 con 0x7f60840712c0 2024-03-04T09:27:39.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.022+0000 7f6089400640 1 --2- 172.21.15.6:0/1501498822 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f60840712c0 0x7f6084162c40 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f60840cee20 tx=0x7f607c02bf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ee3f335d35aba03 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:39.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.022+0000 7f6083e00640 1 -- 172.21.15.6:0/1501498822 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f607c039070 con 0x7f60840712c0 2024-03-04T09:27:39.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.022+0000 7f6083e00640 1 -- 172.21.15.6:0/1501498822 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f607c02cce0 con 0x7f60840712c0 2024-03-04T09:27:39.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.022+0000 7f6083e00640 1 -- 172.21.15.6:0/1501498822 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f607c034660 con 0x7f60840712c0 2024-03-04T09:27:39.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.023+0000 7f608b400640 1 -- 172.21.15.6:0/1501498822 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f60840712c0 msgr2=0x7f6084162c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:39.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.023+0000 7f608b400640 1 --2- 172.21.15.6:0/1501498822 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f60840712c0 0x7f6084162c40 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f60840cee20 tx=0x7f607c02bf30 comp rx=0 tx=0).stop 2024-03-04T09:27:39.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.023+0000 7f608b400640 1 -- 172.21.15.6:0/1501498822 shutdown_connections 2024-03-04T09:27:39.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.023+0000 7f608b400640 1 --2- 172.21.15.6:0/1501498822 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f60840712c0 0x7f6084162c40 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:39.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.023+0000 7f608b400640 1 -- 172.21.15.6:0/1501498822 >> 172.21.15.6:0/1501498822 conn(0x7f608406afb0 msgr2=0x7f608406b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:39.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.023+0000 7f608b400640 1 -- 172.21.15.6:0/1501498822 shutdown_connections 2024-03-04T09:27:39.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.023+0000 7f608b400640 1 -- 172.21.15.6:0/1501498822 wait complete. 2024-03-04T09:27:39.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.025+0000 7f608b400640 1 Processor -- start 2024-03-04T09:27:39.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.025+0000 7f608b400640 1 -- start start 2024-03-04T09:27:39.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.026+0000 7f608b400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6084070ee0 0x7f6084160860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:39.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.026+0000 7f608b400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f60840712c0 0x7f608414fed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:39.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.026+0000 7f608b400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6084072da0 0x7f6084162770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:39.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.026+0000 7f608b400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60840cdda0 con 0x7f60840712c0 2024-03-04T09:27:39.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.026+0000 7f608b400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60840cdc20 con 0x7f6084070ee0 2024-03-04T09:27:39.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.026+0000 7f608b400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f608406e7d0 con 0x7f6084072da0 2024-03-04T09:27:39.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.026+0000 7f607be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f60840712c0 0x7f608414fed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:39.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.026+0000 7f607be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f60840712c0 0x7f608414fed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36620/0 (socket says 172.21.15.6:36620) 2024-03-04T09:27:39.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.026+0000 7f607be00640 1 -- 172.21.15.6:0/3568177669 learned_addr learned my addr 172.21.15.6:0/3568177669 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:27:39.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.026+0000 7f6088a00640 1 --2- 172.21.15.6:0/3568177669 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6084070ee0 0x7f6084160860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:39.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.026+0000 7f607be00640 1 -- 172.21.15.6:0/3568177669 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6084072da0 msgr2=0x7f6084162770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:39.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.026+0000 7f607be00640 1 --2- 172.21.15.6:0/3568177669 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6084072da0 0x7f6084162770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:39.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.026+0000 7f607be00640 1 -- 172.21.15.6:0/3568177669 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6084070ee0 msgr2=0x7f6084160860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:39.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.026+0000 7f607be00640 1 --2- 172.21.15.6:0/3568177669 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6084070ee0 0x7f6084160860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:39.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.026+0000 7f607be00640 1 -- 172.21.15.6:0/3568177669 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6084162cb0 con 0x7f60840712c0 2024-03-04T09:27:39.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.026+0000 7f6088a00640 1 --2- 172.21.15.6:0/3568177669 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6084070ee0 0x7f6084160860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:27:39.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.027+0000 7f607be00640 1 --2- 172.21.15.6:0/3568177669 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f60840712c0 0x7f608414fed0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f607400c440 tx=0x7f607400c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:39.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.027+0000 7f6081600640 1 -- 172.21.15.6:0/3568177669 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6074017070 con 0x7f60840712c0 2024-03-04T09:27:39.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.027+0000 7f6081600640 1 -- 172.21.15.6:0/3568177669 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6074005ce0 con 0x7f60840712c0 2024-03-04T09:27:39.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.027+0000 7f6081600640 1 -- 172.21.15.6:0/3568177669 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60740072b0 con 0x7f60840712c0 2024-03-04T09:27:39.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.027+0000 7f608b400640 1 -- 172.21.15.6:0/3568177669 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6084160e40 con 0x7f60840712c0 2024-03-04T09:27:39.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.027+0000 7f608b400640 1 -- 172.21.15.6:0/3568177669 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6084161400 con 0x7f60840712c0 2024-03-04T09:27:39.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.028+0000 7f608b400640 1 -- 172.21.15.6:0/3568177669 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6084069510 con 0x7f60840712c0 2024-03-04T09:27:39.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.029+0000 7f6081600640 1 -- 172.21.15.6:0/3568177669 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f607401a070 con 0x7f60840712c0 2024-03-04T09:27:39.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.030+0000 7f6081600640 1 --2- 172.21.15.6:0/3568177669 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f606006b500 0x7f606006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:39.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.030+0000 7f6081600640 1 -- 172.21.15.6:0/3568177669 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f60740894f0 con 0x7f60840712c0 2024-03-04T09:27:39.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.031+0000 7f6088a00640 1 --2- 172.21.15.6:0/3568177669 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f606006b500 0x7f606006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:39.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.032+0000 7f6088a00640 1 --2- 172.21.15.6:0/3568177669 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f606006b500 0x7f606006d9c0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f606c000f60 tx=0x7f606c009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:39.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.035+0000 7f6081600640 1 -- 172.21.15.6:0/3568177669 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6074058430 con 0x7f60840712c0 2024-03-04T09:27:39.155 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:27:39.162 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:27:39.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.244+0000 7f608b400640 1 -- 172.21.15.6:0/3568177669 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f608404f3a0 con 0x7f60840712c0 2024-03-04T09:27:39.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.244+0000 7f6081600640 1 -- 172.21.15.6:0/3568177669 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v44) v1 ==== 80+0+0 (secure 0 0 0) 0x7f6074057d30 con 0x7f60840712c0 2024-03-04T09:27:39.241 INFO:teuthology.orchestra.run.smithi006.stderr:listed 0 entries 2024-03-04T09:27:39.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.247+0000 7f608b400640 1 -- 172.21.15.6:0/3568177669 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f606006b500 msgr2=0x7f606006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:39.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.247+0000 7f608b400640 1 --2- 172.21.15.6:0/3568177669 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f606006b500 0x7f606006d9c0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f606c000f60 tx=0x7f606c009040 comp rx=0 tx=0).stop 2024-03-04T09:27:39.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.247+0000 7f608b400640 1 -- 172.21.15.6:0/3568177669 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f60840712c0 msgr2=0x7f608414fed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:39.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.247+0000 7f608b400640 1 --2- 172.21.15.6:0/3568177669 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f60840712c0 0x7f608414fed0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f607400c440 tx=0x7f607400c900 comp rx=0 tx=0).stop 2024-03-04T09:27:39.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.247+0000 7f608b400640 1 -- 172.21.15.6:0/3568177669 shutdown_connections 2024-03-04T09:27:39.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.247+0000 7f608b400640 1 --2- 172.21.15.6:0/3568177669 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f606006b500 0x7f606006d9c0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:39.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.247+0000 7f608b400640 1 --2- 172.21.15.6:0/3568177669 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6084072da0 0x7f6084162770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:39.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.247+0000 7f608b400640 1 --2- 172.21.15.6:0/3568177669 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f60840712c0 0x7f608414fed0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:39.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.247+0000 7f608b400640 1 --2- 172.21.15.6:0/3568177669 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6084070ee0 0x7f6084160860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:39.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.247+0000 7f608b400640 1 -- 172.21.15.6:0/3568177669 >> 172.21.15.6:0/3568177669 conn(0x7f608406afb0 msgr2=0x7f6084165510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:39.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.247+0000 7f608b400640 1 -- 172.21.15.6:0/3568177669 shutdown_connections 2024-03-04T09:27:39.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.247+0000 7f608b400640 1 -- 172.21.15.6:0/3568177669 wait complete. 2024-03-04T09:27:39.260 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-04T09:27:39.261 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:27:39.261 DEBUG:teuthology.orchestra.run.smithi006:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-04T09:27:39.429 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi006.front.sepia.ceph.com 2024-03-04T09:27:39.429 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:27:39.429 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-04T09:27:39.466 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-04T09:27:39.466 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T09:27:39.503 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053429,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249180k,nr_inodes=812295,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-04T09:27:39.503 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-04T09:27:39.503 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T09:27:39.548 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054202,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249800k,nr_inodes=812450,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-04T09:27:39.548 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-04T09:27:39.548 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T09:27:39.590 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053429,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249180k,nr_inodes=812295,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-04T09:27:39.591 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:27:39.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.811+0000 7f3cb3200640 1 Processor -- start 2024-03-04T09:27:39.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.811+0000 7f3cb3200640 1 -- start start 2024-03-04T09:27:39.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.812+0000 7f3cb3200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f3cb415e070 0x7f3cb40d2960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:39.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.812+0000 7f3cb3200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f3cb40d0cd0 0x7f3cb40cf870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:39.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.812+0000 7f3cb3200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cb40d18c0 con 0x7f3cb40cf410 2024-03-04T09:27:39.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.812+0000 7f3cb3200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3cb40d1ba0 con 0x7f3cb40d0cd0 2024-03-04T09:27:39.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.812+0000 7f3cb3200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3cb40d1e80 con 0x7f3cb415e070 2024-03-04T09:27:39.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.812+0000 7f3cb2800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f3cb40d0cd0 0x7f3cb40cf870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:39.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.812+0000 7f3cb2800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f3cb40d0cd0 0x7f3cb40cf870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36624/0 (socket says 172.21.15.6:36624) 2024-03-04T09:27:39.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.812+0000 7f3cb2800640 1 -- 172.21.15.6:0/2677509952 learned_addr learned my addr 172.21.15.6:0/2677509952 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:27:39.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.813+0000 7f3cb1e00640 1 --2- 172.21.15.6:0/2677509952 >> v2:172.21.15.146:3300/0 conn(0x7f3cb415e070 0x7f3cb40d2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:39.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.813+0000 7f3cb2800640 1 -- 172.21.15.6:0/2677509952 >> v2:172.21.15.146:3300/0 conn(0x7f3cb415e070 msgr2=0x7f3cb40d2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:27:39.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.813+0000 7f3cb2800640 1 --2- 172.21.15.6:0/2677509952 >> v2:172.21.15.146:3300/0 conn(0x7f3cb415e070 0x7f3cb40d2960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:39.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.813+0000 7f3cb2800640 1 -- 172.21.15.6:0/2677509952 >> v1:172.21.15.146:6789/0 conn(0x7f3cb40cf410 legacy=0x7f3cb40d05c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:27:39.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.813+0000 7f3cb2800640 1 -- 172.21.15.6:0/2677509952 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cb40d2160 con 0x7f3cb40d0cd0 2024-03-04T09:27:39.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.814+0000 7f3cb2800640 1 --2- 172.21.15.6:0/2677509952 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3cb40d0cd0 0x7f3cb40cf870 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f3cb408d270 tx=0x7f3cac02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db2deda3dbe50eb8 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:39.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.814+0000 7f3cb1400640 1 -- 172.21.15.6:0/2677509952 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cac02ca80 con 0x7f3cb40d0cd0 2024-03-04T09:27:39.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.814+0000 7f3cb1400640 1 -- 172.21.15.6:0/2677509952 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cac02cc20 con 0x7f3cb40d0cd0 2024-03-04T09:27:39.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.814+0000 7f3cb1400640 1 -- 172.21.15.6:0/2677509952 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cac0343f0 con 0x7f3cb40d0cd0 2024-03-04T09:27:39.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.815+0000 7f3cb3200640 1 -- 172.21.15.6:0/2677509952 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3cb40d0cd0 msgr2=0x7f3cb40cf870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:39.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.815+0000 7f3cb3200640 1 --2- 172.21.15.6:0/2677509952 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3cb40d0cd0 0x7f3cb40cf870 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f3cb408d270 tx=0x7f3cac02bee0 comp rx=0 tx=0).stop 2024-03-04T09:27:39.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.815+0000 7f3cb3200640 1 -- 172.21.15.6:0/2677509952 shutdown_connections 2024-03-04T09:27:39.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.815+0000 7f3cb3200640 1 --2- 172.21.15.6:0/2677509952 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3cb40d0cd0 0x7f3cb40cf870 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:39.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.815+0000 7f3cb3200640 1 --2- 172.21.15.6:0/2677509952 >> v2:172.21.15.146:3300/0 conn(0x7f3cb415e070 0x7f3cb40d2960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:39.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.815+0000 7f3cb3200640 1 -- 172.21.15.6:0/2677509952 >> 172.21.15.6:0/2677509952 conn(0x7f3cb408d2a0 msgr2=0x7f3cb40d3cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:39.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.815+0000 7f3cb3200640 1 -- 172.21.15.6:0/2677509952 shutdown_connections 2024-03-04T09:27:39.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.815+0000 7f3cb3200640 1 -- 172.21.15.6:0/2677509952 wait complete. 2024-03-04T09:27:39.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.816+0000 7f3cb3200640 1 Processor -- start 2024-03-04T09:27:39.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.817+0000 7f3cb3200640 1 -- start start 2024-03-04T09:27:39.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.817+0000 7f3cb3200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3cb40cf410 0x7f3cb41f7d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:39.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.817+0000 7f3cb3200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3cb40d0cd0 0x7f3cb4157890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:39.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.817+0000 7f3cb3200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3cb415e070 0x7f3cb415e450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:39.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.817+0000 7f3cb3200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cb40d4730 con 0x7f3cb40cf410 2024-03-04T09:27:39.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.817+0000 7f3cb3200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cb40d45b0 con 0x7f3cb40d0cd0 2024-03-04T09:27:39.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.817+0000 7f3cb3200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cb40b3630 con 0x7f3cb415e070 2024-03-04T09:27:39.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.818+0000 7f3cb1e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3cb40cf410 0x7f3cb41f7d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:39.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.818+0000 7f3cb1e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3cb40cf410 0x7f3cb41f7d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36626/0 (socket says 172.21.15.6:36626) 2024-03-04T09:27:39.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.818+0000 7f3cb1e00640 1 -- 172.21.15.6:0/3639022678 learned_addr learned my addr 172.21.15.6:0/3639022678 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:27:39.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.818+0000 7f3cabe00640 1 --2- 172.21.15.6:0/3639022678 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3cb40d0cd0 0x7f3cb4157890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:39.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.818+0000 7f3cb2800640 1 --2- 172.21.15.6:0/3639022678 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3cb415e070 0x7f3cb415e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:39.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.818+0000 7f3cb1e00640 1 -- 172.21.15.6:0/3639022678 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3cb415e070 msgr2=0x7f3cb415e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:39.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.818+0000 7f3cb1e00640 1 --2- 172.21.15.6:0/3639022678 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3cb415e070 0x7f3cb415e450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:39.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.818+0000 7f3cb1e00640 1 -- 172.21.15.6:0/3639022678 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3cb40d0cd0 msgr2=0x7f3cb4157890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:39.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.818+0000 7f3cb1e00640 1 --2- 172.21.15.6:0/3639022678 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3cb40d0cd0 0x7f3cb4157890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:39.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.818+0000 7f3cb1e00640 1 -- 172.21.15.6:0/3639022678 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cb41f8240 con 0x7f3cb40cf410 2024-03-04T09:27:39.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.819+0000 7f3cb1e00640 1 --2- 172.21.15.6:0/3639022678 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3cb40cf410 0x7f3cb41f7d00 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f3ca400d5f0 tx=0x7f3ca400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:39.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.819+0000 7f3caa000640 1 -- 172.21.15.6:0/3639022678 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ca40125f0 con 0x7f3cb40cf410 2024-03-04T09:27:39.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.819+0000 7f3caa000640 1 -- 172.21.15.6:0/3639022678 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ca4019070 con 0x7f3cb40cf410 2024-03-04T09:27:39.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.819+0000 7f3cb3200640 1 -- 172.21.15.6:0/3639022678 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cb4160f40 con 0x7f3cb40cf410 2024-03-04T09:27:39.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.819+0000 7f3caa000640 1 -- 172.21.15.6:0/3639022678 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ca40129d0 con 0x7f3cb40cf410 2024-03-04T09:27:39.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.819+0000 7f3cb3200640 1 -- 172.21.15.6:0/3639022678 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cb4156db0 con 0x7f3cb40cf410 2024-03-04T09:27:39.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.820+0000 7f3cb3200640 1 -- 172.21.15.6:0/3639022678 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cb404f330 con 0x7f3cb40cf410 2024-03-04T09:27:39.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.821+0000 7f3caa000640 1 -- 172.21.15.6:0/3639022678 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3ca4012b70 con 0x7f3cb40cf410 2024-03-04T09:27:39.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.822+0000 7f3caa000640 1 --2- 172.21.15.6:0/3639022678 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f3c9006b4d0 0x7f3c9006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:39.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.822+0000 7f3caa000640 1 -- 172.21.15.6:0/3639022678 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3ca408a600 con 0x7f3cb40cf410 2024-03-04T09:27:39.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.823+0000 7f3cabe00640 1 --2- 172.21.15.6:0/3639022678 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f3c9006b4d0 0x7f3c9006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:39.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.824+0000 7f3cabe00640 1 --2- 172.21.15.6:0/3639022678 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f3c9006b4d0 0x7f3c9006d990 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f3cb40d4df0 tx=0x7f3c9c009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:39.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:39.827+0000 7f3caa000640 1 -- 172.21.15.6:0/3639022678 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3ca4059470 con 0x7f3cb40cf410 2024-03-04T09:27:40.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.058+0000 7f3cb3200640 1 -- 172.21.15.6:0/3639022678 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3cb406ce40 con 0x7f3cb40cf410 2024-03-04T09:27:40.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.059+0000 7f3caa000640 1 -- 172.21.15.6:0/3639022678 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+4974 (secure 0 0 0) 0x7f3ca4058d70 con 0x7f3cb40cf410 2024-03-04T09:27:40.056 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:27:40.057 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4913,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/1049631914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":1049631914},{"type":"v1","addr":"172.21.15.146:6835","nonce":1049631914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":5092,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/424062551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":424062551},{"type":"v1","addr":"172.21.15.6:6833","nonce":424062551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":19},{"gid":5095,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/3300873502","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":3300873502},{"type":"v1","addr":"172.21.15.6:6835","nonce":3300873502}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:27:16.785309+0000","modified":"2024-03-04T09:27:19.824667+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5110},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5110":{"gid":5110,"name":"b","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.146:6837/3982958749","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":3982958749},{"type":"v1","addr":"172.21.15.146:6837","nonce":3982958749}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5110,"qdb_cluster":[5110]},"id":3}]} 2024-03-04T09:27:40.057 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 25 2024-03-04T09:27:40.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.062+0000 7f3cb3200640 1 -- 172.21.15.6:0/3639022678 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f3c9006b4d0 msgr2=0x7f3c9006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:40.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.062+0000 7f3cb3200640 1 --2- 172.21.15.6:0/3639022678 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f3c9006b4d0 0x7f3c9006d990 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f3cb40d4df0 tx=0x7f3c9c009040 comp rx=0 tx=0).stop 2024-03-04T09:27:40.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.062+0000 7f3cb3200640 1 -- 172.21.15.6:0/3639022678 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3cb40cf410 msgr2=0x7f3cb41f7d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:40.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.062+0000 7f3cb3200640 1 --2- 172.21.15.6:0/3639022678 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3cb40cf410 0x7f3cb41f7d00 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f3ca400d5f0 tx=0x7f3ca400dab0 comp rx=0 tx=0).stop 2024-03-04T09:27:40.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.062+0000 7f3cb3200640 1 -- 172.21.15.6:0/3639022678 shutdown_connections 2024-03-04T09:27:40.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.062+0000 7f3cb3200640 1 --2- 172.21.15.6:0/3639022678 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f3c9006b4d0 0x7f3c9006d990 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:40.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.062+0000 7f3cb3200640 1 --2- 172.21.15.6:0/3639022678 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3cb415e070 0x7f3cb415e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:40.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.062+0000 7f3cb3200640 1 --2- 172.21.15.6:0/3639022678 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3cb40d0cd0 0x7f3cb4157890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:40.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.062+0000 7f3cb3200640 1 --2- 172.21.15.6:0/3639022678 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3cb40cf410 0x7f3cb41f7d00 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:40.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.062+0000 7f3cb3200640 1 -- 172.21.15.6:0/3639022678 >> 172.21.15.6:0/3639022678 conn(0x7f3cb408d2a0 msgr2=0x7f3cb406d060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:40.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.062+0000 7f3cb3200640 1 -- 172.21.15.6:0/3639022678 shutdown_connections 2024-03-04T09:27:40.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.062+0000 7f3cb3200640 1 -- 172.21.15.6:0/3639022678 wait complete. 2024-03-04T09:27:40.076 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T09:27:40.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.287+0000 7fa671800640 1 Processor -- start 2024-03-04T09:27:40.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.287+0000 7fa671800640 1 -- start start 2024-03-04T09:27:40.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.289+0000 7fa671800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa66c070ee0 0x7fa66c162c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:40.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.289+0000 7fa671800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fa66c072da0 0x7fa66c0cf130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:40.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.289+0000 7fa671800640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa66c0ce090 con 0x7fa66c0712c0 2024-03-04T09:27:40.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.289+0000 7fa671800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa66c0ceca0 con 0x7fa66c070ee0 2024-03-04T09:27:40.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.289+0000 7fa671800640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa66c0ce370 con 0x7fa66c072da0 2024-03-04T09:27:40.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.290+0000 7fa66aa00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa66c070ee0 0x7fa66c162c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:40.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.290+0000 7fa66b400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fa66c0712c0 0x7fa66c06e0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:45248/0 (socket says 172.21.15.6:45248) 2024-03-04T09:27:40.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.290+0000 7fa66b400640 1 -- 172.21.15.6:0/2697636189 learned_addr learned my addr 172.21.15.6:0/2697636189 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:27:40.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.290+0000 7fa66a000640 1 --2- 172.21.15.6:0/2697636189 >> v2:172.21.15.146:3300/0 conn(0x7fa66c072da0 0x7fa66c0cf130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:40.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.290+0000 7fa66aa00640 1 -- 172.21.15.6:0/2697636189 >> v2:172.21.15.146:3300/0 conn(0x7fa66c072da0 msgr2=0x7fa66c0cf130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:27:40.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.290+0000 7fa66aa00640 1 --2- 172.21.15.6:0/2697636189 >> v2:172.21.15.146:3300/0 conn(0x7fa66c072da0 0x7fa66c0cf130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:40.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.290+0000 7fa66aa00640 1 -- 172.21.15.6:0/2697636189 >> v1:172.21.15.6:6789/0 conn(0x7fa66c0712c0 legacy=0x7fa66c06e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:40.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.290+0000 7fa66aa00640 1 -- 172.21.15.6:0/2697636189 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa66c0ce650 con 0x7fa66c070ee0 2024-03-04T09:27:40.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.290+0000 7fa66a000640 1 --2- 172.21.15.6:0/2697636189 >> v2:172.21.15.146:3300/0 conn(0x7fa66c072da0 0x7fa66c0cf130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:27:40.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.291+0000 7fa66aa00640 1 --2- 172.21.15.6:0/2697636189 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa66c070ee0 0x7fa66c162c40 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fa66c06c570 tx=0x7fa65c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c62fe4dfcbf9e2e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:40.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.292+0000 7fa669600640 1 -- 172.21.15.6:0/2697636189 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa65c035020 con 0x7fa66c070ee0 2024-03-04T09:27:40.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.292+0000 7fa669600640 1 -- 172.21.15.6:0/2697636189 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa65c031830 con 0x7fa66c070ee0 2024-03-04T09:27:40.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.292+0000 7fa669600640 1 -- 172.21.15.6:0/2697636189 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa65c031c00 con 0x7fa66c070ee0 2024-03-04T09:27:40.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.292+0000 7fa671800640 1 -- 172.21.15.6:0/2697636189 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa66c070ee0 msgr2=0x7fa66c162c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:40.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.292+0000 7fa671800640 1 --2- 172.21.15.6:0/2697636189 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa66c070ee0 0x7fa66c162c40 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fa66c06c570 tx=0x7fa65c02b8d0 comp rx=0 tx=0).stop 2024-03-04T09:27:40.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.292+0000 7fa671800640 1 -- 172.21.15.6:0/2697636189 shutdown_connections 2024-03-04T09:27:40.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.292+0000 7fa671800640 1 --2- 172.21.15.6:0/2697636189 >> v2:172.21.15.146:3300/0 conn(0x7fa66c072da0 0x7fa66c0cf130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:40.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.292+0000 7fa671800640 1 --2- 172.21.15.6:0/2697636189 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa66c070ee0 0x7fa66c162c40 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:40.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.292+0000 7fa671800640 1 -- 172.21.15.6:0/2697636189 >> 172.21.15.6:0/2697636189 conn(0x7fa66c06afb0 msgr2=0x7fa66c06b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:40.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.292+0000 7fa671800640 1 -- 172.21.15.6:0/2697636189 shutdown_connections 2024-03-04T09:27:40.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.292+0000 7fa671800640 1 -- 172.21.15.6:0/2697636189 wait complete. 2024-03-04T09:27:40.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.294+0000 7fa671800640 1 Processor -- start 2024-03-04T09:27:40.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.294+0000 7fa671800640 1 -- start start 2024-03-04T09:27:40.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.295+0000 7fa671800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa66c070ee0 0x7fa66c0aec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:40.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.295+0000 7fa671800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa66c0712c0 0x7fa66c14cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:40.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.295+0000 7fa671800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa66c072da0 0x7fa66c15f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:40.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.295+0000 7fa671800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa66c0cdda0 con 0x7fa66c0712c0 2024-03-04T09:27:40.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.295+0000 7fa671800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa66c0cdc20 con 0x7fa66c072da0 2024-03-04T09:27:40.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.295+0000 7fa671800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa66c06e7d0 con 0x7fa66c070ee0 2024-03-04T09:27:40.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.295+0000 7fa66a000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa66c0712c0 0x7fa66c14cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:40.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.295+0000 7fa66a000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa66c0712c0 0x7fa66c14cf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36644/0 (socket says 172.21.15.6:36644) 2024-03-04T09:27:40.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.295+0000 7fa66a000640 1 -- 172.21.15.6:0/2525123036 learned_addr learned my addr 172.21.15.6:0/2525123036 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:27:40.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.296+0000 7fa66aa00640 1 --2- 172.21.15.6:0/2525123036 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa66c070ee0 0x7fa66c0aec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:40.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.296+0000 7fa66b400640 1 --2- 172.21.15.6:0/2525123036 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa66c072da0 0x7fa66c15f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:40.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.296+0000 7fa66a000640 1 -- 172.21.15.6:0/2525123036 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa66c070ee0 msgr2=0x7fa66c0aec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:40.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.296+0000 7fa66a000640 1 --2- 172.21.15.6:0/2525123036 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa66c070ee0 0x7fa66c0aec50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:40.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.296+0000 7fa66a000640 1 -- 172.21.15.6:0/2525123036 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa66c072da0 msgr2=0x7fa66c15f8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:40.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.296+0000 7fa66a000640 1 --2- 172.21.15.6:0/2525123036 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa66c072da0 0x7fa66c15f8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:40.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.296+0000 7fa66a000640 1 -- 172.21.15.6:0/2525123036 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa66c0acf30 con 0x7fa66c0712c0 2024-03-04T09:27:40.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.296+0000 7fa66a000640 1 --2- 172.21.15.6:0/2525123036 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa66c0712c0 0x7fa66c14cf50 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fa65800c820 tx=0x7fa65800cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:40.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.296+0000 7fa662a00640 1 -- 172.21.15.6:0/2525123036 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa658015070 con 0x7fa66c0712c0 2024-03-04T09:27:40.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.296+0000 7fa662a00640 1 -- 172.21.15.6:0/2525123036 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa658007a30 con 0x7fa66c0712c0 2024-03-04T09:27:40.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.296+0000 7fa671800640 1 -- 172.21.15.6:0/2525123036 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa66c0ad210 con 0x7fa66c0712c0 2024-03-04T09:27:40.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.296+0000 7fa662a00640 1 -- 172.21.15.6:0/2525123036 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa658007de0 con 0x7fa66c0712c0 2024-03-04T09:27:40.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.297+0000 7fa671800640 1 -- 172.21.15.6:0/2525123036 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa66c15fdf0 con 0x7fa66c0712c0 2024-03-04T09:27:40.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.298+0000 7fa671800640 1 -- 172.21.15.6:0/2525123036 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa66c069510 con 0x7fa66c0712c0 2024-03-04T09:27:40.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.299+0000 7fa662a00640 1 -- 172.21.15.6:0/2525123036 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa658018070 con 0x7fa66c0712c0 2024-03-04T09:27:40.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.299+0000 7fa662a00640 1 --2- 172.21.15.6:0/2525123036 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa64c06b4d0 0x7fa64c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:40.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.299+0000 7fa662a00640 1 -- 172.21.15.6:0/2525123036 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa65808a1c0 con 0x7fa66c0712c0 2024-03-04T09:27:40.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.300+0000 7fa66aa00640 1 --2- 172.21.15.6:0/2525123036 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa64c06b4d0 0x7fa64c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:40.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.301+0000 7fa66aa00640 1 --2- 172.21.15.6:0/2525123036 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa64c06b4d0 0x7fa64c06d990 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fa65c036000 tx=0x7fa65c008030 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:40.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.304+0000 7fa662a00640 1 -- 172.21.15.6:0/2525123036 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa658058fa0 con 0x7fa66c0712c0 2024-03-04T09:27:40.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.508+0000 7fa671800640 1 -- 172.21.15.6:0/2525123036 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa66c0b52d0 con 0x7fa66c0712c0 2024-03-04T09:27:40.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.511+0000 7fa662a00640 1 -- 172.21.15.6:0/2525123036 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v44) v1 ==== 74+0+16565 (secure 0 0 0) 0x7fa6580588a0 con 0x7fa66c0712c0 2024-03-04T09:27:40.509 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:27:40.509 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":44,"fsid":"edce0578-5ae4-4402-b45a-552bdef84d7c","created":"2024-03-04T09:24:54.523780+0000","modified":"2024-03-04T09:27:18.307032+0000","last_up_change":"2024-03-04T09:25:07.089049+0000","last_in_change":"2024-03-04T09:24:57.982077+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T09:25:07.935198+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-03-04T09:27:14.657490+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"43","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-03-04T09:27:15.806886+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a96574f-81f1-4bc6-8463-757520b84a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6825","nonce":1364449401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6827","nonce":1364449401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6831","nonce":1364449401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6829","nonce":1364449401}]},"public_addr":"172.21.15.6:6825/1364449401","cluster_addr":"172.21.15.6:6827/1364449401","heartbeat_back_addr":"172.21.15.6:6831/1364449401","heartbeat_front_addr":"172.21.15.6:6829/1364449401","state":["exists","up"]},{"osd":1,"uuid":"7d79b9c1-6b87-476a-bc36-7db5ee64cd90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6809","nonce":2494103078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6811","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6813","nonce":2494103078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6821","nonce":2494103078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6817","nonce":2494103078}]},"public_addr":"172.21.15.6:6809/2494103078","cluster_addr":"172.21.15.6:6813/2494103078","heartbeat_back_addr":"172.21.15.6:6821/2494103078","heartbeat_front_addr":"172.21.15.6:6817/2494103078","state":["exists","up"]},{"osd":2,"uuid":"492331b9-4ee9-40b6-bacd-85d44cd6c45a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6812","nonce":1500066681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6816","nonce":1500066681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6824","nonce":1500066681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6820","nonce":1500066681}]},"public_addr":"172.21.15.6:6812/1500066681","cluster_addr":"172.21.15.6:6816/1500066681","heartbeat_back_addr":"172.21.15.6:6824/1500066681","heartbeat_front_addr":"172.21.15.6:6820/1500066681","state":["exists","up"]},{"osd":3,"uuid":"21bb2efe-9ba7-4bdc-bc31-180cd1102e21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6801","nonce":1558806123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6803","nonce":1558806123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6807","nonce":1558806123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6805","nonce":1558806123}]},"public_addr":"172.21.15.6:6801/1558806123","cluster_addr":"172.21.15.6:6803/1558806123","heartbeat_back_addr":"172.21.15.6:6807/1558806123","heartbeat_front_addr":"172.21.15.6:6805/1558806123","state":["exists","up"]},{"osd":4,"uuid":"6e2c4b71-19d9-4737-83d9-558705d8644a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6801","nonce":1526135780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6803","nonce":1526135780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6807","nonce":1526135780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6805","nonce":1526135780}]},"public_addr":"172.21.15.146:6801/1526135780","cluster_addr":"172.21.15.146:6803/1526135780","heartbeat_back_addr":"172.21.15.146:6807/1526135780","heartbeat_front_addr":"172.21.15.146:6805/1526135780","state":["exists","up"]},{"osd":5,"uuid":"fb69c525-f1be-4b7d-ab32-6c29b0d6fddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6809","nonce":460262507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6811","nonce":460262507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6815","nonce":460262507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6813","nonce":460262507}]},"public_addr":"172.21.15.146:6809/460262507","cluster_addr":"172.21.15.146:6811/460262507","heartbeat_back_addr":"172.21.15.146:6815/460262507","heartbeat_front_addr":"172.21.15.146:6813/460262507","state":["exists","up"]},{"osd":6,"uuid":"c32e7fed-370d-4401-8ce0-6a4335b0d168","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6823","nonce":1392897493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6827","nonce":1392897493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6831","nonce":1392897493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6829","nonce":1392897493}]},"public_addr":"172.21.15.146:6823/1392897493","cluster_addr":"172.21.15.146:6827/1392897493","heartbeat_back_addr":"172.21.15.146:6831/1392897493","heartbeat_front_addr":"172.21.15.146:6829/1392897493","state":["exists","up"]},{"osd":7,"uuid":"75a79b44-fa5f-426d-a7d3-71f21f78835c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6817","nonce":3268323156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6819","nonce":3268323156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6825","nonce":3268323156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6822","nonce":3268323156}]},"public_addr":"172.21.15.146:6817/3268323156","cluster_addr":"172.21.15.146:6819/3268323156","heartbeat_back_addr":"172.21.15.146:6825/3268323156","heartbeat_front_addr":"172.21.15.146:6822/3268323156","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.310437+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.295850+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.294674+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.526855+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.300023+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.595160+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.346046+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.558741+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T09:27:40.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.514+0000 7fa671800640 1 -- 172.21.15.6:0/2525123036 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa64c06b4d0 msgr2=0x7fa64c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:40.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.514+0000 7fa671800640 1 --2- 172.21.15.6:0/2525123036 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa64c06b4d0 0x7fa64c06d990 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fa65c036000 tx=0x7fa65c008030 comp rx=0 tx=0).stop 2024-03-04T09:27:40.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.514+0000 7fa671800640 1 -- 172.21.15.6:0/2525123036 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa66c0712c0 msgr2=0x7fa66c14cf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:40.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.514+0000 7fa671800640 1 --2- 172.21.15.6:0/2525123036 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa66c0712c0 0x7fa66c14cf50 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fa65800c820 tx=0x7fa65800cce0 comp rx=0 tx=0).stop 2024-03-04T09:27:40.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.514+0000 7fa671800640 1 -- 172.21.15.6:0/2525123036 shutdown_connections 2024-03-04T09:27:40.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.514+0000 7fa671800640 1 --2- 172.21.15.6:0/2525123036 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa66c072da0 0x7fa66c15f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:40.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.514+0000 7fa671800640 1 --2- 172.21.15.6:0/2525123036 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa66c0712c0 0x7fa66c14cf50 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:40.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.514+0000 7fa671800640 1 --2- 172.21.15.6:0/2525123036 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa64c06b4d0 0x7fa64c06d990 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:40.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.514+0000 7fa671800640 1 --2- 172.21.15.6:0/2525123036 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa66c070ee0 0x7fa66c0aec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:40.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.514+0000 7fa671800640 1 -- 172.21.15.6:0/2525123036 >> 172.21.15.6:0/2525123036 conn(0x7fa66c06afb0 msgr2=0x7fa66c0b0c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:40.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.514+0000 7fa671800640 1 -- 172.21.15.6:0/2525123036 shutdown_connections 2024-03-04T09:27:40.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.515+0000 7fa671800640 1 -- 172.21.15.6:0/2525123036 wait complete. 2024-03-04T09:27:40.531 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T09:25:07.935198+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-04T09:27:14.657490+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '43', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_data', 'create_time': '2024-03-04T09:27:15.806886+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '44', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-04T09:27:40.532 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-04T09:27:40.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.754+0000 7f701e600640 1 Processor -- start 2024-03-04T09:27:40.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.754+0000 7f701e600640 1 -- start start 2024-03-04T09:27:40.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.756+0000 7f701e600640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f7018145430 0x7f701815f810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:40.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.756+0000 7f701e600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f70180b2250 0x7f70180b2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:40.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.756+0000 7f701e600640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f701815fe40 con 0x7f701815cc90 2024-03-04T09:27:40.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.756+0000 7f701e600640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7018160120 con 0x7f70180b2250 2024-03-04T09:27:40.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.756+0000 7f701e600640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7018160400 con 0x7f7018145430 2024-03-04T09:27:40.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.756+0000 7f7017e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f701815cc90 0x7f70180b1b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:45260/0 (socket says 172.21.15.6:45260) 2024-03-04T09:27:40.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.757+0000 7f7017e00640 1 -- 172.21.15.6:0/459029889 learned_addr learned my addr 172.21.15.6:0/459029889 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:27:40.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.757+0000 7f701dc00640 1 --2- 172.21.15.6:0/459029889 >> v2:172.21.15.6:3300/0 conn(0x7f70180b2250 0x7f70180b2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:40.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.757+0000 7f701d200640 1 --2- 172.21.15.6:0/459029889 >> v2:172.21.15.146:3300/0 conn(0x7f7018145430 0x7f701815f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:40.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.757+0000 7f7017400640 1 -- 172.21.15.6:0/459029889 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 150377702 0 0) 0x7f701815fe40 con 0x7f701815cc90 2024-03-04T09:27:40.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.757+0000 7f7017400640 1 -- 172.21.15.6:0/459029889 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7000003520 con 0x7f701815cc90 2024-03-04T09:27:40.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.757+0000 7f701dc00640 1 -- 172.21.15.6:0/459029889 >> v2:172.21.15.146:3300/0 conn(0x7f7018145430 msgr2=0x7f701815f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:27:40.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.757+0000 7f701dc00640 1 --2- 172.21.15.6:0/459029889 >> v2:172.21.15.146:3300/0 conn(0x7f7018145430 0x7f701815f810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:40.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.757+0000 7f701dc00640 1 -- 172.21.15.6:0/459029889 >> v1:172.21.15.6:6789/0 conn(0x7f701815cc90 legacy=0x7f70180b1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:40.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.757+0000 7f701dc00640 1 -- 172.21.15.6:0/459029889 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70181606e0 con 0x7f70180b2250 2024-03-04T09:27:40.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.757+0000 7f701d200640 1 --2- 172.21.15.6:0/459029889 >> v2:172.21.15.146:3300/0 conn(0x7f7018145430 0x7f701815f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T09:27:40.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.758+0000 7f701dc00640 1 --2- 172.21.15.6:0/459029889 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f70180b2250 0x7f70180b2fd0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f70180b2670 tx=0x7f701002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=284a01b48444884d server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:40.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.759+0000 7f7017400640 1 -- 172.21.15.6:0/459029889 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7010039070 con 0x7f70180b2250 2024-03-04T09:27:40.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.759+0000 7f7017400640 1 -- 172.21.15.6:0/459029889 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f701002cc80 con 0x7f70180b2250 2024-03-04T09:27:40.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.759+0000 7f7017400640 1 -- 172.21.15.6:0/459029889 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7010034660 con 0x7f70180b2250 2024-03-04T09:27:40.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.759+0000 7f701e600640 1 -- 172.21.15.6:0/459029889 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f70180b2250 msgr2=0x7f70180b2fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:40.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.759+0000 7f701e600640 1 --2- 172.21.15.6:0/459029889 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f70180b2250 0x7f70180b2fd0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f70180b2670 tx=0x7f701002bed0 comp rx=0 tx=0).stop 2024-03-04T09:27:40.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.759+0000 7f701e600640 1 -- 172.21.15.6:0/459029889 shutdown_connections 2024-03-04T09:27:40.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.759+0000 7f701e600640 1 --2- 172.21.15.6:0/459029889 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f70180b2250 0x7f70180b2fd0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:40.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.759+0000 7f701e600640 1 --2- 172.21.15.6:0/459029889 >> v2:172.21.15.146:3300/0 conn(0x7f7018145430 0x7f701815f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:40.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.759+0000 7f701e600640 1 -- 172.21.15.6:0/459029889 >> 172.21.15.6:0/459029889 conn(0x7f701806b3a0 msgr2=0x7f701806b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:40.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.760+0000 7f701e600640 1 -- 172.21.15.6:0/459029889 shutdown_connections 2024-03-04T09:27:40.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.760+0000 7f701e600640 1 -- 172.21.15.6:0/459029889 wait complete. 2024-03-04T09:27:40.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.761+0000 7f701e600640 1 Processor -- start 2024-03-04T09:27:40.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.761+0000 7f701e600640 1 -- start start 2024-03-04T09:27:40.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.762+0000 7f701e600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f70180b2250 0x7f7018152ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:40.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.762+0000 7f701e600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7018145430 0x7f70180b2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:40.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.762+0000 7f701e600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f701815cc90 0x7f70180b3510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:40.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.762+0000 7f701e600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7018145990 con 0x7f70180b2250 2024-03-04T09:27:40.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.762+0000 7f701e600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7018145810 con 0x7f701815cc90 2024-03-04T09:27:40.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.762+0000 7f701e600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70180cde00 con 0x7f7018145430 2024-03-04T09:27:40.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.763+0000 7f701d200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f70180b2250 0x7f7018152ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:40.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.763+0000 7f701d200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f70180b2250 0x7f7018152ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36662/0 (socket says 172.21.15.6:36662) 2024-03-04T09:27:40.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.763+0000 7f701d200640 1 -- 172.21.15.6:0/1854001662 learned_addr learned my addr 172.21.15.6:0/1854001662 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:27:40.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.763+0000 7f7017e00640 1 --2- 172.21.15.6:0/1854001662 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7018145430 0x7f70180b2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:40.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.763+0000 7f701dc00640 1 --2- 172.21.15.6:0/1854001662 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f701815cc90 0x7f70180b3510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:40.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.763+0000 7f701d200640 1 -- 172.21.15.6:0/1854001662 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7018145430 msgr2=0x7f70180b2fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:40.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.763+0000 7f701d200640 1 --2- 172.21.15.6:0/1854001662 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7018145430 0x7f70180b2fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:40.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.763+0000 7f701d200640 1 -- 172.21.15.6:0/1854001662 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f701815cc90 msgr2=0x7f70180b3510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:40.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.763+0000 7f701d200640 1 --2- 172.21.15.6:0/1854001662 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f701815cc90 0x7f70180b3510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:40.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.763+0000 7f701d200640 1 -- 172.21.15.6:0/1854001662 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7018154bb0 con 0x7f70180b2250 2024-03-04T09:27:40.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.763+0000 7f701dc00640 1 --2- 172.21.15.6:0/1854001662 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f701815cc90 0x7f70180b3510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:27:40.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.764+0000 7f701d200640 1 --2- 172.21.15.6:0/1854001662 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f70180b2250 0x7f7018152ca0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f700c00e730 tx=0x7f700c00ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:40.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.764+0000 7f7014c00640 1 -- 172.21.15.6:0/1854001662 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f700c00f040 con 0x7f70180b2250 2024-03-04T09:27:40.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.764+0000 7f7014c00640 1 -- 172.21.15.6:0/1854001662 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f700c004230 con 0x7f70180b2250 2024-03-04T09:27:40.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.764+0000 7f7014c00640 1 -- 172.21.15.6:0/1854001662 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f700c00ee40 con 0x7f70180b2250 2024-03-04T09:27:40.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.764+0000 7f701e600640 1 -- 172.21.15.6:0/1854001662 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7018153760 con 0x7f70180b2250 2024-03-04T09:27:40.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.764+0000 7f701e600640 1 -- 172.21.15.6:0/1854001662 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7018155130 con 0x7f70180b2250 2024-03-04T09:27:40.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.765+0000 7f701e600640 1 -- 172.21.15.6:0/1854001662 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70180ccec0 con 0x7f70180b2250 2024-03-04T09:27:40.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.766+0000 7f7014c00640 1 -- 172.21.15.6:0/1854001662 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f700c00c6d0 con 0x7f70180b2250 2024-03-04T09:27:40.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.767+0000 7f7014c00640 1 --2- 172.21.15.6:0/1854001662 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f6ff806b500 0x7f6ff806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:40.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.767+0000 7f7014c00640 1 -- 172.21.15.6:0/1854001662 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f700c089f50 con 0x7f70180b2250 2024-03-04T09:27:40.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.768+0000 7f7017e00640 1 --2- 172.21.15.6:0/1854001662 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f6ff806b500 0x7f6ff806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:40.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.769+0000 7f7017e00640 1 --2- 172.21.15.6:0/1854001662 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f6ff806b500 0x7f6ff806d9c0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f701813b3f0 tx=0x7f7004009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:40.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.771+0000 7f7014c00640 1 -- 172.21.15.6:0/1854001662 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f70180ccec0 con 0x7f70180b2250 2024-03-04T09:27:40.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.973+0000 7f701e600640 1 -- 172.21.15.6:0/1854001662 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f701806e240 con 0x7f70180b2250 2024-03-04T09:27:40.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.974+0000 7f7014c00640 1 -- 172.21.15.6:0/1854001662 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v44) v1 ==== 93+0+10 (secure 0 0 0) 0x7f700c017020 con 0x7f70180b2250 2024-03-04T09:27:40.971 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-04T09:27:40.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.977+0000 7f701e600640 1 -- 172.21.15.6:0/1854001662 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f6ff806b500 msgr2=0x7f6ff806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:40.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.977+0000 7f701e600640 1 --2- 172.21.15.6:0/1854001662 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f6ff806b500 0x7f6ff806d9c0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f701813b3f0 tx=0x7f7004009040 comp rx=0 tx=0).stop 2024-03-04T09:27:40.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.977+0000 7f701e600640 1 -- 172.21.15.6:0/1854001662 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f70180b2250 msgr2=0x7f7018152ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:40.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.977+0000 7f701e600640 1 --2- 172.21.15.6:0/1854001662 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f70180b2250 0x7f7018152ca0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f700c00e730 tx=0x7f700c00ebf0 comp rx=0 tx=0).stop 2024-03-04T09:27:40.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.977+0000 7f701e600640 1 -- 172.21.15.6:0/1854001662 shutdown_connections 2024-03-04T09:27:40.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.977+0000 7f701e600640 1 --2- 172.21.15.6:0/1854001662 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f701815cc90 0x7f70180b3510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:40.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.977+0000 7f701e600640 1 --2- 172.21.15.6:0/1854001662 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f6ff806b500 0x7f6ff806d9c0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:40.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.977+0000 7f701e600640 1 --2- 172.21.15.6:0/1854001662 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7018145430 0x7f70180b2fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:40.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.977+0000 7f701e600640 1 --2- 172.21.15.6:0/1854001662 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f70180b2250 0x7f7018152ca0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:40.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.978+0000 7f701e600640 1 -- 172.21.15.6:0/1854001662 >> 172.21.15.6:0/1854001662 conn(0x7f701806b3a0 msgr2=0x7f70180b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:40.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.978+0000 7f701e600640 1 -- 172.21.15.6:0/1854001662 shutdown_connections 2024-03-04T09:27:40.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:40.978+0000 7f701e600640 1 -- 172.21.15.6:0/1854001662 wait complete. 2024-03-04T09:27:40.994 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-04T09:27:41.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.227+0000 7f0855a00640 1 Processor -- start 2024-03-04T09:27:41.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.227+0000 7f0855a00640 1 -- start start 2024-03-04T09:27:41.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.229+0000 7f0855a00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f085015c010 0x7f085006ee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:41.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.229+0000 7f0855a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f085006f370 0x7f085015c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:41.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.229+0000 7f0855a00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08500af250 con 0x7f085015c930 2024-03-04T09:27:41.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.229+0000 7f0855a00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f085015cf50 con 0x7f085015c010 2024-03-04T09:27:41.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.229+0000 7f0855a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f085015d230 con 0x7f085006f370 2024-03-04T09:27:41.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.230+0000 7f0847e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f085006f370 0x7f085015c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:41.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.230+0000 7f084ea00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f085015c010 0x7f085006ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:41.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.230+0000 7f0847e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f085006f370 0x7f085015c3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.6:56178/0 (socket says 172.21.15.6:56178) 2024-03-04T09:27:41.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.230+0000 7f0847e00640 1 -- 172.21.15.6:0/1026345397 learned_addr learned my addr 172.21.15.6:0/1026345397 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:27:41.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.231+0000 7f084ea00640 1 -- 172.21.15.6:0/1026345397 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f085006f370 msgr2=0x7f085015c3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:27:41.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.231+0000 7f084ea00640 1 --2- 172.21.15.6:0/1026345397 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f085006f370 0x7f085015c3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:41.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.231+0000 7f084ea00640 1 -- 172.21.15.6:0/1026345397 >> v1:172.21.15.146:6789/0 conn(0x7f085015c930 legacy=0x7f08500afcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:41.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.231+0000 7f084ea00640 1 -- 172.21.15.6:0/1026345397 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08500af4f0 con 0x7f085015c010 2024-03-04T09:27:41.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.231+0000 7f0847e00640 1 --2- 172.21.15.6:0/1026345397 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f085006f370 0x7f085015c3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T09:27:41.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.232+0000 7f084ea00640 1 --2- 172.21.15.6:0/1026345397 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f085015c010 0x7f085006ee30 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f08400097a0 tx=0x7f084002b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4c285927294a0efd server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:41.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.233+0000 7f084e000640 1 -- 172.21.15.6:0/1026345397 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0840035020 con 0x7f085015c010 2024-03-04T09:27:41.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.233+0000 7f084e000640 1 -- 172.21.15.6:0/1026345397 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0840031810 con 0x7f085015c010 2024-03-04T09:27:41.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.233+0000 7f084e000640 1 -- 172.21.15.6:0/1026345397 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0840031be0 con 0x7f085015c010 2024-03-04T09:27:41.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.233+0000 7f0855a00640 1 -- 172.21.15.6:0/1026345397 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f085015c010 msgr2=0x7f085006ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:41.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.233+0000 7f0855a00640 1 --2- 172.21.15.6:0/1026345397 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f085015c010 0x7f085006ee30 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f08400097a0 tx=0x7f084002b8b0 comp rx=0 tx=0).stop 2024-03-04T09:27:41.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.234+0000 7f0855a00640 1 -- 172.21.15.6:0/1026345397 shutdown_connections 2024-03-04T09:27:41.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.234+0000 7f0855a00640 1 --2- 172.21.15.6:0/1026345397 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f085006f370 0x7f085015c3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:41.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.234+0000 7f0855a00640 1 --2- 172.21.15.6:0/1026345397 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f085015c010 0x7f085006ee30 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:41.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.234+0000 7f0855a00640 1 -- 172.21.15.6:0/1026345397 >> 172.21.15.6:0/1026345397 conn(0x7f08500732d0 msgr2=0x7f085009aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:41.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.234+0000 7f0855a00640 1 -- 172.21.15.6:0/1026345397 shutdown_connections 2024-03-04T09:27:41.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.234+0000 7f0855a00640 1 -- 172.21.15.6:0/1026345397 wait complete. 2024-03-04T09:27:41.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.236+0000 7f0855a00640 1 Processor -- start 2024-03-04T09:27:41.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.236+0000 7f0855a00640 1 -- start start 2024-03-04T09:27:41.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.237+0000 7f0855a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f085006f370 0x7f08500d0ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:41.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.237+0000 7f0855a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f085015c010 0x7f08500cc230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:41.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.237+0000 7f0855a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f085015c930 0x7f08500b1a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:41.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.237+0000 7f0855a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08500b0400 con 0x7f085015c010 2024-03-04T09:27:41.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.237+0000 7f0855a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08500d1ca0 con 0x7f085006f370 2024-03-04T09:27:41.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.237+0000 7f0855a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08500b0580 con 0x7f085015c930 2024-03-04T09:27:41.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.238+0000 7f0847e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f085015c010 0x7f08500cc230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:41.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.238+0000 7f0847e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f085015c010 0x7f08500cc230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36668/0 (socket says 172.21.15.6:36668) 2024-03-04T09:27:41.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.238+0000 7f0847e00640 1 -- 172.21.15.6:0/1777087966 learned_addr learned my addr 172.21.15.6:0/1777087966 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:27:41.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.238+0000 7f084ea00640 1 --2- 172.21.15.6:0/1777087966 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f085006f370 0x7f08500d0ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:41.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.238+0000 7f084f400640 1 --2- 172.21.15.6:0/1777087966 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f085015c930 0x7f08500b1a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:41.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.238+0000 7f0847e00640 1 -- 172.21.15.6:0/1777087966 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f085015c930 msgr2=0x7f08500b1a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:41.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.238+0000 7f0847e00640 1 --2- 172.21.15.6:0/1777087966 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f085015c930 0x7f08500b1a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:41.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.238+0000 7f0847e00640 1 -- 172.21.15.6:0/1777087966 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f085006f370 msgr2=0x7f08500d0ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:41.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.238+0000 7f0847e00640 1 --2- 172.21.15.6:0/1777087966 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f085006f370 0x7f08500d0ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:41.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.238+0000 7f0847e00640 1 -- 172.21.15.6:0/1777087966 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08500d0080 con 0x7f085015c010 2024-03-04T09:27:41.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.238+0000 7f084f400640 1 --2- 172.21.15.6:0/1777087966 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f085015c930 0x7f08500b1a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:27:41.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.239+0000 7f0847e00640 1 --2- 172.21.15.6:0/1777087966 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f085015c010 0x7f08500cc230 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f083800e9a0 tx=0x7f083800ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:41.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.239+0000 7f0846a00640 1 -- 172.21.15.6:0/1777087966 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0838017070 con 0x7f085015c010 2024-03-04T09:27:41.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.239+0000 7f0846a00640 1 -- 172.21.15.6:0/1777087966 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f083800ca00 con 0x7f085015c010 2024-03-04T09:27:41.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.239+0000 7f0846a00640 1 -- 172.21.15.6:0/1777087966 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f083800cdb0 con 0x7f085015c010 2024-03-04T09:27:41.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.239+0000 7f0855a00640 1 -- 172.21.15.6:0/1777087966 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08500d0360 con 0x7f085015c010 2024-03-04T09:27:41.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.239+0000 7f0855a00640 1 -- 172.21.15.6:0/1777087966 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08500cc770 con 0x7f085015c010 2024-03-04T09:27:41.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.241+0000 7f0855a00640 1 -- 172.21.15.6:0/1777087966 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f085004f330 con 0x7f085015c010 2024-03-04T09:27:41.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.241+0000 7f0846a00640 1 -- 172.21.15.6:0/1777087966 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f083801a070 con 0x7f085015c010 2024-03-04T09:27:41.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.242+0000 7f0846a00640 1 --2- 172.21.15.6:0/1777087966 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f082c06b4d0 0x7f082c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:41.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.242+0000 7f0846a00640 1 -- 172.21.15.6:0/1777087966 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f083808a570 con 0x7f085015c010 2024-03-04T09:27:41.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.243+0000 7f084ea00640 1 --2- 172.21.15.6:0/1777087966 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f082c06b4d0 0x7f082c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:41.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.244+0000 7f084ea00640 1 --2- 172.21.15.6:0/1777087966 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f082c06b4d0 0x7f082c06d990 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f0840036000 tx=0x7f0840008030 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:41.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.249+0000 7f0846a00640 1 -- 172.21.15.6:0/1777087966 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0838059360 con 0x7f085015c010 2024-03-04T09:27:41.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.462+0000 7f0855a00640 1 -- 172.21.15.6:0/1777087966 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f085006b380 con 0x7f085015c010 2024-03-04T09:27:41.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.463+0000 7f0846a00640 1 -- 172.21.15.6:0/1777087966 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v44) v1 ==== 104+0+11 (secure 0 0 0) 0x7f0838058c60 con 0x7f085015c010 2024-03-04T09:27:41.460 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-03-04T09:27:41.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.466+0000 7f0855a00640 1 -- 172.21.15.6:0/1777087966 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f082c06b4d0 msgr2=0x7f082c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:41.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.466+0000 7f0855a00640 1 --2- 172.21.15.6:0/1777087966 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f082c06b4d0 0x7f082c06d990 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f0840036000 tx=0x7f0840008030 comp rx=0 tx=0).stop 2024-03-04T09:27:41.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.466+0000 7f0855a00640 1 -- 172.21.15.6:0/1777087966 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f085015c010 msgr2=0x7f08500cc230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:41.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.466+0000 7f0855a00640 1 --2- 172.21.15.6:0/1777087966 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f085015c010 0x7f08500cc230 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f083800e9a0 tx=0x7f083800ee60 comp rx=0 tx=0).stop 2024-03-04T09:27:41.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.466+0000 7f0855a00640 1 -- 172.21.15.6:0/1777087966 shutdown_connections 2024-03-04T09:27:41.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.466+0000 7f0855a00640 1 --2- 172.21.15.6:0/1777087966 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f082c06b4d0 0x7f082c06d990 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:41.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.466+0000 7f0855a00640 1 --2- 172.21.15.6:0/1777087966 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f085015c930 0x7f08500b1a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:41.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.466+0000 7f0855a00640 1 --2- 172.21.15.6:0/1777087966 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f085015c010 0x7f08500cc230 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:41.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.466+0000 7f0855a00640 1 --2- 172.21.15.6:0/1777087966 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f085006f370 0x7f08500d0ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:41.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.466+0000 7f0855a00640 1 -- 172.21.15.6:0/1777087966 >> 172.21.15.6:0/1777087966 conn(0x7f08500732d0 msgr2=0x7f08500d1e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:41.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.467+0000 7f0855a00640 1 -- 172.21.15.6:0/1777087966 shutdown_connections 2024-03-04T09:27:41.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.467+0000 7f0855a00640 1 -- 172.21.15.6:0/1777087966 wait complete. 2024-03-04T09:27:41.485 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-04T09:27:41.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.700+0000 7f987fe00640 1 Processor -- start 2024-03-04T09:27:41.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.701+0000 7f987fe00640 1 -- start start 2024-03-04T09:27:41.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.702+0000 7f987fe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f988006d8d0 0x7f98800b1b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:41.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.702+0000 7f987fe00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f988006de40 con 0x7f9880147590 2024-03-04T09:27:41.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.702+0000 7f987fe00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f988006e120 con 0x7f98800b2080 2024-03-04T09:27:41.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.702+0000 7f987fe00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9880147970 con 0x7f988006d8d0 2024-03-04T09:27:41.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.702+0000 7f987e000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f988006d8d0 0x7f98800b1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:41.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.702+0000 7f987e000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f988006d8d0 0x7f98800b1b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36670/0 (socket says 172.21.15.6:36670) 2024-03-04T09:27:41.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.702+0000 7f987e000640 1 -- 172.21.15.6:0/1092334063 learned_addr learned my addr 172.21.15.6:0/1092334063 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:27:41.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.703+0000 7f987e000640 1 -- 172.21.15.6:0/1092334063 >> v1:172.21.15.146:6789/0 conn(0x7f98800b2080 legacy=0x7f98800b3640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:27:41.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.703+0000 7f987e000640 1 -- 172.21.15.6:0/1092334063 >> v1:172.21.15.6:6789/0 conn(0x7f9880147590 legacy=0x7f9880071480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:41.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.703+0000 7f987e000640 1 -- 172.21.15.6:0/1092334063 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98800b2e40 con 0x7f988006d8d0 2024-03-04T09:27:41.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.704+0000 7f987e000640 1 --2- 172.21.15.6:0/1092334063 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f988006d8d0 0x7f98800b1b40 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f9880147af0 tx=0x7f986c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=181e1acfe06c8a84 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:41.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.704+0000 7f987d600640 1 -- 172.21.15.6:0/1092334063 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f986c034070 con 0x7f988006d8d0 2024-03-04T09:27:41.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.704+0000 7f987d600640 1 -- 172.21.15.6:0/1092334063 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f986c030840 con 0x7f988006d8d0 2024-03-04T09:27:41.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.704+0000 7f987d600640 1 -- 172.21.15.6:0/1092334063 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f986c030c10 con 0x7f988006d8d0 2024-03-04T09:27:41.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.705+0000 7f987fe00640 1 -- 172.21.15.6:0/1092334063 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f988006d8d0 msgr2=0x7f98800b1b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:41.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.705+0000 7f987fe00640 1 --2- 172.21.15.6:0/1092334063 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f988006d8d0 0x7f98800b1b40 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f9880147af0 tx=0x7f986c02b980 comp rx=0 tx=0).stop 2024-03-04T09:27:41.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.705+0000 7f987fe00640 1 -- 172.21.15.6:0/1092334063 shutdown_connections 2024-03-04T09:27:41.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.705+0000 7f987fe00640 1 --2- 172.21.15.6:0/1092334063 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f988006d8d0 0x7f98800b1b40 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:41.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.705+0000 7f987fe00640 1 -- 172.21.15.6:0/1092334063 >> 172.21.15.6:0/1092334063 conn(0x7f988006b3a0 msgr2=0x7f988006b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:41.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.705+0000 7f987fe00640 1 -- 172.21.15.6:0/1092334063 shutdown_connections 2024-03-04T09:27:41.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.705+0000 7f987fe00640 1 -- 172.21.15.6:0/1092334063 wait complete. 2024-03-04T09:27:41.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.707+0000 7f987fe00640 1 Processor -- start 2024-03-04T09:27:41.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.707+0000 7f987fe00640 1 -- start start 2024-03-04T09:27:41.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.708+0000 7f987fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f988006d8d0 0x7f98801f7500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:41.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.708+0000 7f987fe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f98800b2080 0x7f988013cb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:41.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.708+0000 7f987fe00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9880147590 0x7f988015d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:41.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.708+0000 7f987fe00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98800b24b0 con 0x7f988006d8d0 2024-03-04T09:27:41.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.708+0000 7f987fe00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98800b31b0 con 0x7f98800b2080 2024-03-04T09:27:41.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.708+0000 7f987fe00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98800b3d50 con 0x7f9880147590 2024-03-04T09:27:41.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.708+0000 7f987ea00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f988006d8d0 0x7f98801f7500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:41.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.708+0000 7f987ea00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f988006d8d0 0x7f98801f7500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36676/0 (socket says 172.21.15.6:36676) 2024-03-04T09:27:41.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.708+0000 7f987ea00640 1 -- 172.21.15.6:0/3031998283 learned_addr learned my addr 172.21.15.6:0/3031998283 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:27:41.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.709+0000 7f987ea00640 1 -- 172.21.15.6:0/3031998283 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9880147590 msgr2=0x7f988015d630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:41.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.709+0000 7f987ea00640 1 --2- 172.21.15.6:0/3031998283 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9880147590 0x7f988015d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:41.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.709+0000 7f987ea00640 1 -- 172.21.15.6:0/3031998283 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f98800b2080 msgr2=0x7f988013cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:41.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.709+0000 7f987ea00640 1 --2- 172.21.15.6:0/3031998283 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f98800b2080 0x7f988013cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:41.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.709+0000 7f987ea00640 1 -- 172.21.15.6:0/3031998283 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98801f7c00 con 0x7f988006d8d0 2024-03-04T09:27:41.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.709+0000 7f987ea00640 1 --2- 172.21.15.6:0/3031998283 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f988006d8d0 0x7f98801f7500 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f987000c730 tx=0x7f987000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:41.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.709+0000 7f9876a00640 1 -- 172.21.15.6:0/3031998283 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9870016020 con 0x7f988006d8d0 2024-03-04T09:27:41.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.709+0000 7f9876a00640 1 -- 172.21.15.6:0/3031998283 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9870006d50 con 0x7f988006d8d0 2024-03-04T09:27:41.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.709+0000 7f9876a00640 1 -- 172.21.15.6:0/3031998283 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9870003bb0 con 0x7f988006d8d0 2024-03-04T09:27:41.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.709+0000 7f987fe00640 1 -- 172.21.15.6:0/3031998283 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98801f7ee0 con 0x7f988006d8d0 2024-03-04T09:27:41.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.709+0000 7f987fe00640 1 -- 172.21.15.6:0/3031998283 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98801f4640 con 0x7f988006d8d0 2024-03-04T09:27:41.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.711+0000 7f987fe00640 1 -- 172.21.15.6:0/3031998283 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98801612b0 con 0x7f988006d8d0 2024-03-04T09:27:41.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.712+0000 7f9876a00640 1 -- 172.21.15.6:0/3031998283 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f987000a850 con 0x7f988006d8d0 2024-03-04T09:27:41.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.713+0000 7f9876a00640 1 --2- 172.21.15.6:0/3031998283 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f986006b4d0 0x7f986006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:41.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.713+0000 7f9876a00640 1 -- 172.21.15.6:0/3031998283 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9870089ef0 con 0x7f988006d8d0 2024-03-04T09:27:41.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.713+0000 7f987e000640 1 --2- 172.21.15.6:0/3031998283 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f986006b4d0 0x7f986006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:41.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.714+0000 7f987e000640 1 --2- 172.21.15.6:0/3031998283 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f986006b4d0 0x7f986006d990 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f986c036000 tx=0x7f986c03b040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:41.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.717+0000 7f9876a00640 1 -- 172.21.15.6:0/3031998283 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f98801612b0 con 0x7f988006d8d0 2024-03-04T09:27:41.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.921+0000 7f987fe00640 1 -- 172.21.15.6:0/3031998283 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f988004f330 con 0x7f988006d8d0 2024-03-04T09:27:41.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.922+0000 7f9876a00640 1 -- 172.21.15.6:0/3031998283 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v44) v1 ==== 100+0+11 (secure 0 0 0) 0x7f9870020020 con 0x7f988006d8d0 2024-03-04T09:27:41.919 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 64 2024-03-04T09:27:41.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.925+0000 7f987fe00640 1 -- 172.21.15.6:0/3031998283 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f986006b4d0 msgr2=0x7f986006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:41.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.925+0000 7f987fe00640 1 --2- 172.21.15.6:0/3031998283 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f986006b4d0 0x7f986006d990 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f986c036000 tx=0x7f986c03b040 comp rx=0 tx=0).stop 2024-03-04T09:27:41.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.925+0000 7f987fe00640 1 -- 172.21.15.6:0/3031998283 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f988006d8d0 msgr2=0x7f98801f7500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:41.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.925+0000 7f987fe00640 1 --2- 172.21.15.6:0/3031998283 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f988006d8d0 0x7f98801f7500 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f987000c730 tx=0x7f987000cbf0 comp rx=0 tx=0).stop 2024-03-04T09:27:41.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.925+0000 7f987fe00640 1 -- 172.21.15.6:0/3031998283 shutdown_connections 2024-03-04T09:27:41.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.925+0000 7f987fe00640 1 --2- 172.21.15.6:0/3031998283 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f986006b4d0 0x7f986006d990 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:41.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.925+0000 7f987fe00640 1 --2- 172.21.15.6:0/3031998283 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9880147590 0x7f988015d630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:41.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.925+0000 7f987fe00640 1 --2- 172.21.15.6:0/3031998283 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f98800b2080 0x7f988013cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:41.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.925+0000 7f987fe00640 1 --2- 172.21.15.6:0/3031998283 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f988006d8d0 0x7f98801f7500 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:41.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.925+0000 7f987fe00640 1 -- 172.21.15.6:0/3031998283 >> 172.21.15.6:0/3031998283 conn(0x7f988006b3a0 msgr2=0x7f98800ad880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:41.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.925+0000 7f987fe00640 1 -- 172.21.15.6:0/3031998283 shutdown_connections 2024-03-04T09:27:41.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:41.925+0000 7f987fe00640 1 -- 172.21.15.6:0/3031998283 wait complete. 2024-03-04T09:27:41.940 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:27:42.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.165+0000 7ff857200640 1 Processor -- start 2024-03-04T09:27:42.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.165+0000 7ff857200640 1 -- start start 2024-03-04T09:27:42.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.166+0000 7ff857200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ff85015dec0 0x7ff8500ce810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:42.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.166+0000 7ff857200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff85015e2a0 0x7ff8500ced50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:42.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.166+0000 7ff857200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8500cf720 con 0x7ff8500cf290 2024-03-04T09:27:42.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.166+0000 7ff857200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8500b0ea0 con 0x7ff85015dec0 2024-03-04T09:27:42.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.166+0000 7ff857200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8500d1120 con 0x7ff85015e2a0 2024-03-04T09:27:42.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.167+0000 7ff856800640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7ff8500cf290 0x7ff8500d1b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:45278/0 (socket says 172.21.15.6:45278) 2024-03-04T09:27:42.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.167+0000 7ff856800640 1 -- 172.21.15.6:0/2818303044 learned_addr learned my addr 172.21.15.6:0/2818303044 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:27:42.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.167+0000 7ff855e00640 1 --2- 172.21.15.6:0/2818303044 >> v2:172.21.15.146:3300/0 conn(0x7ff85015dec0 0x7ff8500ce810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:42.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.167+0000 7ff855400640 1 --2- 172.21.15.6:0/2818303044 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff85015e2a0 0x7ff8500ced50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:42.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.167+0000 7ff854a00640 1 -- 172.21.15.6:0/2818303044 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3046963105 0 0) 0x7ff84c001490 con 0x7ff8500cf290 2024-03-04T09:27:42.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.167+0000 7ff854a00640 1 -- 172.21.15.6:0/2818303044 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff838003520 con 0x7ff8500cf290 2024-03-04T09:27:42.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.168+0000 7ff854a00640 1 -- 172.21.15.6:0/2818303044 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3761004204 0 0) 0x7ff838003520 con 0x7ff8500cf290 2024-03-04T09:27:42.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.168+0000 7ff855e00640 1 -- 172.21.15.6:0/2818303044 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff85015e2a0 msgr2=0x7ff8500ced50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:27:42.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.168+0000 7ff855e00640 1 --2- 172.21.15.6:0/2818303044 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff85015e2a0 0x7ff8500ced50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:42.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.168+0000 7ff855e00640 1 -- 172.21.15.6:0/2818303044 >> v1:172.21.15.6:6789/0 conn(0x7ff8500cf290 legacy=0x7ff8500d1b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:42.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.168+0000 7ff855e00640 1 -- 172.21.15.6:0/2818303044 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8500d1400 con 0x7ff85015dec0 2024-03-04T09:27:42.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.169+0000 7ff855e00640 1 --2- 172.21.15.6:0/2818303044 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff85015dec0 0x7ff8500ce810 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7ff8440097a0 tx=0x7ff84402b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d07864ddde6e2880 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:42.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.170+0000 7ff854a00640 1 -- 172.21.15.6:0/2818303044 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff844037070 con 0x7ff85015dec0 2024-03-04T09:27:42.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.170+0000 7ff854a00640 1 -- 172.21.15.6:0/2818303044 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff844030510 con 0x7ff85015dec0 2024-03-04T09:27:42.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.170+0000 7ff854a00640 1 -- 172.21.15.6:0/2818303044 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff84c001bc0 con 0x7ff85015dec0 2024-03-04T09:27:42.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.170+0000 7ff857200640 1 -- 172.21.15.6:0/2818303044 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff85015dec0 msgr2=0x7ff8500ce810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:42.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.170+0000 7ff857200640 1 --2- 172.21.15.6:0/2818303044 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff85015dec0 0x7ff8500ce810 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7ff8440097a0 tx=0x7ff84402b790 comp rx=0 tx=0).stop 2024-03-04T09:27:42.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.171+0000 7ff857200640 1 -- 172.21.15.6:0/2818303044 shutdown_connections 2024-03-04T09:27:42.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.171+0000 7ff857200640 1 --2- 172.21.15.6:0/2818303044 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff85015e2a0 0x7ff8500ced50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:42.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.171+0000 7ff857200640 1 --2- 172.21.15.6:0/2818303044 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff85015dec0 0x7ff8500ce810 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:42.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.171+0000 7ff857200640 1 -- 172.21.15.6:0/2818303044 >> 172.21.15.6:0/2818303044 conn(0x7ff8500d3d40 msgr2=0x7ff8500d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:42.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.171+0000 7ff857200640 1 -- 172.21.15.6:0/2818303044 shutdown_connections 2024-03-04T09:27:42.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.171+0000 7ff857200640 1 -- 172.21.15.6:0/2818303044 wait complete. 2024-03-04T09:27:42.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.172+0000 7ff857200640 1 Processor -- start 2024-03-04T09:27:42.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.173+0000 7ff857200640 1 -- start start 2024-03-04T09:27:42.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.173+0000 7ff857200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff8500cf290 0x7ff850161350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:42.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.173+0000 7ff857200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff85015dec0 0x7ff8500ca320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:42.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.173+0000 7ff857200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff85015e2a0 0x7ff8501572a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:42.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.173+0000 7ff857200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8500d0ec0 con 0x7ff85015dec0 2024-03-04T09:27:42.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.173+0000 7ff857200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8500d0d40 con 0x7ff85015e2a0 2024-03-04T09:27:42.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.173+0000 7ff857200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8500b2340 con 0x7ff8500cf290 2024-03-04T09:27:42.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.174+0000 7ff855400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff85015dec0 0x7ff8500ca320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:42.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.174+0000 7ff855400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff85015dec0 0x7ff8500ca320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36690/0 (socket says 172.21.15.6:36690) 2024-03-04T09:27:42.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.174+0000 7ff855400640 1 -- 172.21.15.6:0/1202634507 learned_addr learned my addr 172.21.15.6:0/1202634507 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:27:42.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.174+0000 7ff855e00640 1 --2- 172.21.15.6:0/1202634507 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff8500cf290 0x7ff850161350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:42.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.174+0000 7ff856800640 1 --2- 172.21.15.6:0/1202634507 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff85015e2a0 0x7ff8501572a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:42.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.174+0000 7ff855400640 1 -- 172.21.15.6:0/1202634507 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff8500cf290 msgr2=0x7ff850161350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:42.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.174+0000 7ff855400640 1 --2- 172.21.15.6:0/1202634507 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff8500cf290 0x7ff850161350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:42.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.174+0000 7ff855400640 1 -- 172.21.15.6:0/1202634507 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff85015e2a0 msgr2=0x7ff8501572a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:42.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.174+0000 7ff855400640 1 --2- 172.21.15.6:0/1202634507 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff85015e2a0 0x7ff8501572a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:42.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.174+0000 7ff855400640 1 -- 172.21.15.6:0/1202634507 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8500cdd00 con 0x7ff85015dec0 2024-03-04T09:27:42.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.174+0000 7ff855e00640 1 --2- 172.21.15.6:0/1202634507 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff8500cf290 0x7ff850161350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:27:42.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.174+0000 7ff856800640 1 --2- 172.21.15.6:0/1202634507 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff85015e2a0 0x7ff8501572a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:27:42.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.175+0000 7ff855400640 1 --2- 172.21.15.6:0/1202634507 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff85015dec0 0x7ff8500ca320 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7ff84000e870 tx=0x7ff84000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:42.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.175+0000 7ff84a000640 1 -- 172.21.15.6:0/1202634507 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff840017070 con 0x7ff85015dec0 2024-03-04T09:27:42.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.175+0000 7ff84a000640 1 -- 172.21.15.6:0/1202634507 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff84000c940 con 0x7ff85015dec0 2024-03-04T09:27:42.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.175+0000 7ff857200640 1 -- 172.21.15.6:0/1202634507 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8500cdfe0 con 0x7ff85015dec0 2024-03-04T09:27:42.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.175+0000 7ff84a000640 1 -- 172.21.15.6:0/1202634507 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff84000cd30 con 0x7ff85015dec0 2024-03-04T09:27:42.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.175+0000 7ff857200640 1 -- 172.21.15.6:0/1202634507 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8500ca860 con 0x7ff85015dec0 2024-03-04T09:27:42.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.176+0000 7ff857200640 1 -- 172.21.15.6:0/1202634507 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff85004f330 con 0x7ff85015dec0 2024-03-04T09:27:42.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.177+0000 7ff84a000640 1 -- 172.21.15.6:0/1202634507 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff84001a070 con 0x7ff85015dec0 2024-03-04T09:27:42.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.179+0000 7ff84a000640 1 --2- 172.21.15.6:0/1202634507 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff83406b4d0 0x7ff83406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:42.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.179+0000 7ff84a000640 1 -- 172.21.15.6:0/1202634507 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff840089ff0 con 0x7ff85015dec0 2024-03-04T09:27:42.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.179+0000 7ff855e00640 1 --2- 172.21.15.6:0/1202634507 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff83406b4d0 0x7ff83406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:42.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.180+0000 7ff855e00640 1 --2- 172.21.15.6:0/1202634507 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff83406b4d0 0x7ff83406d990 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7ff844034000 tx=0x7ff84402f000 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:42.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.183+0000 7ff84a000640 1 -- 172.21.15.6:0/1202634507 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff840058f00 con 0x7ff85015dec0 2024-03-04T09:27:42.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.422+0000 7ff857200640 1 -- 172.21.15.6:0/1202634507 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff8500d2bb0 con 0x7ff85015dec0 2024-03-04T09:27:42.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.423+0000 7ff84a000640 1 -- 172.21.15.6:0/1202634507 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+4974 (secure 0 0 0) 0x7ff840058800 con 0x7ff85015dec0 2024-03-04T09:27:42.420 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:27:42.420 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4913,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/1049631914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":1049631914},{"type":"v1","addr":"172.21.15.146:6835","nonce":1049631914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":5092,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/424062551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":424062551},{"type":"v1","addr":"172.21.15.6:6833","nonce":424062551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":19},{"gid":5095,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/3300873502","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":3300873502},{"type":"v1","addr":"172.21.15.6:6835","nonce":3300873502}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:27:16.785309+0000","modified":"2024-03-04T09:27:19.824667+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5110},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5110":{"gid":5110,"name":"b","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.146:6837/3982958749","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":3982958749},{"type":"v1","addr":"172.21.15.146:6837","nonce":3982958749}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5110,"qdb_cluster":[5110]},"id":3}]} 2024-03-04T09:27:42.420 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 25 2024-03-04T09:27:42.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.426+0000 7ff857200640 1 -- 172.21.15.6:0/1202634507 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff83406b4d0 msgr2=0x7ff83406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:42.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.426+0000 7ff857200640 1 --2- 172.21.15.6:0/1202634507 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff83406b4d0 0x7ff83406d990 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7ff844034000 tx=0x7ff84402f000 comp rx=0 tx=0).stop 2024-03-04T09:27:42.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.426+0000 7ff857200640 1 -- 172.21.15.6:0/1202634507 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff85015dec0 msgr2=0x7ff8500ca320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:42.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.426+0000 7ff857200640 1 --2- 172.21.15.6:0/1202634507 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff85015dec0 0x7ff8500ca320 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7ff84000e870 tx=0x7ff84000ed30 comp rx=0 tx=0).stop 2024-03-04T09:27:42.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.426+0000 7ff857200640 1 -- 172.21.15.6:0/1202634507 shutdown_connections 2024-03-04T09:27:42.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.426+0000 7ff857200640 1 --2- 172.21.15.6:0/1202634507 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff85015e2a0 0x7ff8501572a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:42.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.426+0000 7ff857200640 1 --2- 172.21.15.6:0/1202634507 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff85015dec0 0x7ff8500ca320 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:42.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.426+0000 7ff857200640 1 --2- 172.21.15.6:0/1202634507 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff83406b4d0 0x7ff83406d990 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:42.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.426+0000 7ff857200640 1 --2- 172.21.15.6:0/1202634507 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff8500cf290 0x7ff850161350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:42.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.426+0000 7ff857200640 1 -- 172.21.15.6:0/1202634507 >> 172.21.15.6:0/1202634507 conn(0x7ff8500d3d40 msgr2=0x7ff85015d180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:42.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.427+0000 7ff857200640 1 -- 172.21.15.6:0/1202634507 shutdown_connections 2024-03-04T09:27:42.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.427+0000 7ff857200640 1 -- 172.21.15.6:0/1202634507 wait complete. 2024-03-04T09:27:42.443 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T09:27:42.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.657+0000 7f560fe00640 1 Processor -- start 2024-03-04T09:27:42.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.657+0000 7f560fe00640 1 -- start start 2024-03-04T09:27:42.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.659+0000 7f560fe00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f561015efc0 0x7f56100cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:42.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.659+0000 7f560fe00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56101627f0 con 0x7f56100cce90 2024-03-04T09:27:42.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.659+0000 7f560fe00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f561006c860 con 0x7f56101621d0 2024-03-04T09:27:42.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.659+0000 7f560fe00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f561015f3a0 con 0x7f561015efc0 2024-03-04T09:27:42.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.659+0000 7f5607e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f56100cce90 0x7f5610161ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:45280/0 (socket says 172.21.15.6:45280) 2024-03-04T09:27:42.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.659+0000 7f5607e00640 1 -- 172.21.15.6:0/924379962 learned_addr learned my addr 172.21.15.6:0/924379962 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:27:42.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.660+0000 7f560ea00640 1 --2- 172.21.15.6:0/924379962 >> v2:172.21.15.146:3300/0 conn(0x7f561015efc0 0x7f56100cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:42.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.660+0000 7f560e000640 1 -- 172.21.15.6:0/924379962 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1524320182 0 0) 0x7f56101627f0 con 0x7f56100cce90 2024-03-04T09:27:42.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.660+0000 7f560e000640 1 -- 172.21.15.6:0/924379962 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55f4003520 con 0x7f56100cce90 2024-03-04T09:27:42.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.660+0000 7f560e000640 1 -- 172.21.15.6:0/924379962 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3066786829 0 0) 0x7f55f4003520 con 0x7f56100cce90 2024-03-04T09:27:42.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.660+0000 7f560e000640 1 -- 172.21.15.6:0/924379962 >> v2:172.21.15.146:3300/0 conn(0x7f561015efc0 msgr2=0x7f56100cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:27:42.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.660+0000 7f560e000640 1 --2- 172.21.15.6:0/924379962 >> v2:172.21.15.146:3300/0 conn(0x7f561015efc0 0x7f56100cc950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:42.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.660+0000 7f560e000640 1 -- 172.21.15.6:0/924379962 >> v1:172.21.15.146:6789/0 conn(0x7f56101621d0 legacy=0x7f5610070c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:42.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.660+0000 7f560e000640 1 -- 172.21.15.6:0/924379962 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f561006cb40 con 0x7f56100cce90 2024-03-04T09:27:42.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.661+0000 7f560e000640 1 -- 172.21.15.6:0/924379962 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7f55f8001670 con 0x7f56100cce90 2024-03-04T09:27:42.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.661+0000 7f560e000640 1 -- 172.21.15.6:0/924379962 >> v1:172.21.15.6:6789/0 conn(0x7f56100cce90 legacy=0x7f5610161ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:42.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.661+0000 7f560e000640 1 --2- 172.21.15.6:0/924379962 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f55f4003a00 0x7f55f4003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:42.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.661+0000 7f560e000640 1 --2- 172.21.15.6:0/924379962 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f55f4004800 0x7f55f4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:42.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.661+0000 7f560e000640 1 --2- 172.21.15.6:0/924379962 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f55f40051c0 0x7f55f400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:42.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.661+0000 7f560e000640 1 -- 172.21.15.6:0/924379962 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f561015f3a0 con 0x7f55f4004800 2024-03-04T09:27:42.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.661+0000 7f560e000640 1 -- 172.21.15.6:0/924379962 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55f400c0c0 con 0x7f55f40051c0 2024-03-04T09:27:42.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.661+0000 7f560e000640 1 -- 172.21.15.6:0/924379962 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55f400c2c0 con 0x7f55f4003a00 2024-03-04T09:27:42.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.661+0000 7f5607e00640 1 --2- 172.21.15.6:0/924379962 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f55f4004800 0x7f55f4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:42.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.661+0000 7f560ea00640 1 --2- 172.21.15.6:0/924379962 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f55f4003a00 0x7f55f4003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:42.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.661+0000 7f560f400640 1 --2- 172.21.15.6:0/924379962 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f55f40051c0 0x7f55f400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:42.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.661+0000 7f5607e00640 1 -- 172.21.15.6:0/924379962 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f55f4003a00 msgr2=0x7f55f4003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:27:42.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.661+0000 7f5607e00640 1 --2- 172.21.15.6:0/924379962 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f55f4003a00 0x7f55f4003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:42.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.661+0000 7f5607e00640 1 -- 172.21.15.6:0/924379962 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f55f40051c0 msgr2=0x7f55f400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:27:42.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.661+0000 7f5607e00640 1 --2- 172.21.15.6:0/924379962 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f55f40051c0 0x7f55f400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:42.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.661+0000 7f5607e00640 1 -- 172.21.15.6:0/924379962 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55f4003520 con 0x7f55f4004800 2024-03-04T09:27:42.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.661+0000 7f560f400640 1 --2- 172.21.15.6:0/924379962 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f55f40051c0 0x7f55f400b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:27:42.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.661+0000 7f560f400640 1 -- 172.21.15.6:0/924379962 reap_dead start 2024-03-04T09:27:42.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.663+0000 7f5607e00640 1 --2- 172.21.15.6:0/924379962 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f55f4004800 0x7f55f4004c80 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f55f800c190 tx=0x7f55f4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a7cdc4f2cd5d14f server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:42.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.663+0000 7f560e000640 1 -- 172.21.15.6:0/924379962 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55f8007c60 con 0x7f55f4004800 2024-03-04T09:27:42.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.663+0000 7f560e000640 1 -- 172.21.15.6:0/924379962 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55f8030070 con 0x7f55f4004800 2024-03-04T09:27:42.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.663+0000 7f560e000640 1 -- 172.21.15.6:0/924379962 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55f8005d90 con 0x7f55f4004800 2024-03-04T09:27:42.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.663+0000 7f560fe00640 1 -- 172.21.15.6:0/924379962 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f55f4004800 msgr2=0x7f55f4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:42.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.663+0000 7f560fe00640 1 --2- 172.21.15.6:0/924379962 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f55f4004800 0x7f55f4004c80 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f55f800c190 tx=0x7f55f4005830 comp rx=0 tx=0).stop 2024-03-04T09:27:42.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.663+0000 7f560fe00640 1 -- 172.21.15.6:0/924379962 shutdown_connections 2024-03-04T09:27:42.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.663+0000 7f560fe00640 1 --2- 172.21.15.6:0/924379962 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f55f4004800 0x7f55f4004c80 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:42.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.663+0000 7f560fe00640 1 -- 172.21.15.6:0/924379962 >> 172.21.15.6:0/924379962 conn(0x7f561006b3d0 msgr2=0x7f561006b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:42.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.664+0000 7f560fe00640 1 -- 172.21.15.6:0/924379962 shutdown_connections 2024-03-04T09:27:42.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.664+0000 7f560fe00640 1 -- 172.21.15.6:0/924379962 wait complete. 2024-03-04T09:27:42.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.665+0000 7f560fe00640 1 Processor -- start 2024-03-04T09:27:42.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.665+0000 7f560fe00640 1 -- start start 2024-03-04T09:27:42.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.666+0000 7f560fe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f55f4004800 0x7f5610145750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:42.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.666+0000 7f560fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5610145c90 0x7f5610130d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:42.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.666+0000 7f560fe00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f56101312b0 0x7f5610131730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:42.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.666+0000 7f560fe00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f561006d620 con 0x7f5610145c90 2024-03-04T09:27:42.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.666+0000 7f560fe00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f561006d190 con 0x7f55f4004800 2024-03-04T09:27:42.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.666+0000 7f560fe00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5610146db0 con 0x7f56101312b0 2024-03-04T09:27:42.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.667+0000 7f5607e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5610145c90 0x7f5610130d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:42.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.667+0000 7f5607e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5610145c90 0x7f5610130d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36704/0 (socket says 172.21.15.6:36704) 2024-03-04T09:27:42.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.667+0000 7f5607e00640 1 -- 172.21.15.6:0/2662055018 learned_addr learned my addr 172.21.15.6:0/2662055018 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:27:42.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.667+0000 7f560ea00640 1 --2- 172.21.15.6:0/2662055018 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f55f4004800 0x7f5610145750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:42.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.667+0000 7f560f400640 1 --2- 172.21.15.6:0/2662055018 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f56101312b0 0x7f5610131730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:42.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.667+0000 7f5607e00640 1 -- 172.21.15.6:0/2662055018 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f56101312b0 msgr2=0x7f5610131730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:42.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.667+0000 7f5607e00640 1 --2- 172.21.15.6:0/2662055018 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f56101312b0 0x7f5610131730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:42.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.667+0000 7f5607e00640 1 -- 172.21.15.6:0/2662055018 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f55f4004800 msgr2=0x7f5610145750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:42.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.667+0000 7f5607e00640 1 --2- 172.21.15.6:0/2662055018 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f55f4004800 0x7f5610145750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:42.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.667+0000 7f5607e00640 1 -- 172.21.15.6:0/2662055018 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5610146510 con 0x7f5610145c90 2024-03-04T09:27:42.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.668+0000 7f5607e00640 1 --2- 172.21.15.6:0/2662055018 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5610145c90 0x7f5610130d70 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f55f800c190 tx=0x7f55f8001970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:42.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.668+0000 7f5606a00640 1 -- 172.21.15.6:0/2662055018 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55f8045070 con 0x7f5610145c90 2024-03-04T09:27:42.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.668+0000 7f5606a00640 1 -- 172.21.15.6:0/2662055018 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55f80049f0 con 0x7f5610145c90 2024-03-04T09:27:42.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.668+0000 7f5606a00640 1 -- 172.21.15.6:0/2662055018 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55f8007c60 con 0x7f5610145c90 2024-03-04T09:27:42.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.668+0000 7f560fe00640 1 -- 172.21.15.6:0/2662055018 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56101467f0 con 0x7f5610145c90 2024-03-04T09:27:42.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.668+0000 7f560fe00640 1 -- 172.21.15.6:0/2662055018 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5610146110 con 0x7f5610145c90 2024-03-04T09:27:42.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.669+0000 7f560fe00640 1 -- 172.21.15.6:0/2662055018 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f561004f330 con 0x7f5610145c90 2024-03-04T09:27:42.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.670+0000 7f5606a00640 1 -- 172.21.15.6:0/2662055018 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f55f8007070 con 0x7f5610145c90 2024-03-04T09:27:42.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.671+0000 7f5606a00640 1 --2- 172.21.15.6:0/2662055018 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f55ec06b500 0x7f55ec06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:42.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.671+0000 7f5606a00640 1 -- 172.21.15.6:0/2662055018 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f55f80aae80 con 0x7f5610145c90 2024-03-04T09:27:42.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.672+0000 7f560ea00640 1 --2- 172.21.15.6:0/2662055018 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f55ec06b500 0x7f55ec06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:42.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.673+0000 7f560ea00640 1 --2- 172.21.15.6:0/2662055018 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f55ec06b500 0x7f55ec06d9c0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f56101470b0 tx=0x7f5600008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:42.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.675+0000 7f5606a00640 1 -- 172.21.15.6:0/2662055018 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f55f8079c60 con 0x7f5610145c90 2024-03-04T09:27:42.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.878+0000 7f560fe00640 1 -- 172.21.15.6:0/2662055018 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5610143680 con 0x7f5610145c90 2024-03-04T09:27:42.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.882+0000 7f5606a00640 1 -- 172.21.15.6:0/2662055018 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v44) v1 ==== 74+0+16565 (secure 0 0 0) 0x7f55f8079560 con 0x7f5610145c90 2024-03-04T09:27:42.879 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:27:42.879 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":44,"fsid":"edce0578-5ae4-4402-b45a-552bdef84d7c","created":"2024-03-04T09:24:54.523780+0000","modified":"2024-03-04T09:27:18.307032+0000","last_up_change":"2024-03-04T09:25:07.089049+0000","last_in_change":"2024-03-04T09:24:57.982077+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T09:25:07.935198+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-03-04T09:27:14.657490+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"43","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-03-04T09:27:15.806886+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a96574f-81f1-4bc6-8463-757520b84a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6825","nonce":1364449401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6827","nonce":1364449401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6831","nonce":1364449401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6829","nonce":1364449401}]},"public_addr":"172.21.15.6:6825/1364449401","cluster_addr":"172.21.15.6:6827/1364449401","heartbeat_back_addr":"172.21.15.6:6831/1364449401","heartbeat_front_addr":"172.21.15.6:6829/1364449401","state":["exists","up"]},{"osd":1,"uuid":"7d79b9c1-6b87-476a-bc36-7db5ee64cd90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6809","nonce":2494103078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6811","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6813","nonce":2494103078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6821","nonce":2494103078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6817","nonce":2494103078}]},"public_addr":"172.21.15.6:6809/2494103078","cluster_addr":"172.21.15.6:6813/2494103078","heartbeat_back_addr":"172.21.15.6:6821/2494103078","heartbeat_front_addr":"172.21.15.6:6817/2494103078","state":["exists","up"]},{"osd":2,"uuid":"492331b9-4ee9-40b6-bacd-85d44cd6c45a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6812","nonce":1500066681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6816","nonce":1500066681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6824","nonce":1500066681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6820","nonce":1500066681}]},"public_addr":"172.21.15.6:6812/1500066681","cluster_addr":"172.21.15.6:6816/1500066681","heartbeat_back_addr":"172.21.15.6:6824/1500066681","heartbeat_front_addr":"172.21.15.6:6820/1500066681","state":["exists","up"]},{"osd":3,"uuid":"21bb2efe-9ba7-4bdc-bc31-180cd1102e21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6801","nonce":1558806123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6803","nonce":1558806123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6807","nonce":1558806123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6805","nonce":1558806123}]},"public_addr":"172.21.15.6:6801/1558806123","cluster_addr":"172.21.15.6:6803/1558806123","heartbeat_back_addr":"172.21.15.6:6807/1558806123","heartbeat_front_addr":"172.21.15.6:6805/1558806123","state":["exists","up"]},{"osd":4,"uuid":"6e2c4b71-19d9-4737-83d9-558705d8644a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6801","nonce":1526135780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6803","nonce":1526135780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6807","nonce":1526135780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6805","nonce":1526135780}]},"public_addr":"172.21.15.146:6801/1526135780","cluster_addr":"172.21.15.146:6803/1526135780","heartbeat_back_addr":"172.21.15.146:6807/1526135780","heartbeat_front_addr":"172.21.15.146:6805/1526135780","state":["exists","up"]},{"osd":5,"uuid":"fb69c525-f1be-4b7d-ab32-6c29b0d6fddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6809","nonce":460262507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6811","nonce":460262507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6815","nonce":460262507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6813","nonce":460262507}]},"public_addr":"172.21.15.146:6809/460262507","cluster_addr":"172.21.15.146:6811/460262507","heartbeat_back_addr":"172.21.15.146:6815/460262507","heartbeat_front_addr":"172.21.15.146:6813/460262507","state":["exists","up"]},{"osd":6,"uuid":"c32e7fed-370d-4401-8ce0-6a4335b0d168","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6823","nonce":1392897493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6827","nonce":1392897493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6831","nonce":1392897493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6829","nonce":1392897493}]},"public_addr":"172.21.15.146:6823/1392897493","cluster_addr":"172.21.15.146:6827/1392897493","heartbeat_back_addr":"172.21.15.146:6831/1392897493","heartbeat_front_addr":"172.21.15.146:6829/1392897493","state":["exists","up"]},{"osd":7,"uuid":"75a79b44-fa5f-426d-a7d3-71f21f78835c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6817","nonce":3268323156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6819","nonce":3268323156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6825","nonce":3268323156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6822","nonce":3268323156}]},"public_addr":"172.21.15.146:6817/3268323156","cluster_addr":"172.21.15.146:6819/3268323156","heartbeat_back_addr":"172.21.15.146:6825/3268323156","heartbeat_front_addr":"172.21.15.146:6822/3268323156","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.310437+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.295850+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.294674+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.526855+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.300023+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.595160+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.346046+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.558741+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T09:27:42.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.884+0000 7f560fe00640 1 -- 172.21.15.6:0/2662055018 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f55ec06b500 msgr2=0x7f55ec06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:42.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.884+0000 7f560fe00640 1 --2- 172.21.15.6:0/2662055018 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f55ec06b500 0x7f55ec06d9c0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f56101470b0 tx=0x7f5600008040 comp rx=0 tx=0).stop 2024-03-04T09:27:42.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.885+0000 7f560fe00640 1 -- 172.21.15.6:0/2662055018 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5610145c90 msgr2=0x7f5610130d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:42.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.885+0000 7f560fe00640 1 --2- 172.21.15.6:0/2662055018 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5610145c90 0x7f5610130d70 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f55f800c190 tx=0x7f55f8001970 comp rx=0 tx=0).stop 2024-03-04T09:27:42.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.885+0000 7f560fe00640 1 -- 172.21.15.6:0/2662055018 shutdown_connections 2024-03-04T09:27:42.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.885+0000 7f560fe00640 1 --2- 172.21.15.6:0/2662055018 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f55ec06b500 0x7f55ec06d9c0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:42.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.885+0000 7f560fe00640 1 --2- 172.21.15.6:0/2662055018 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f56101312b0 0x7f5610131730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:42.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.885+0000 7f560fe00640 1 --2- 172.21.15.6:0/2662055018 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5610145c90 0x7f5610130d70 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:42.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.885+0000 7f560fe00640 1 --2- 172.21.15.6:0/2662055018 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f55f4004800 0x7f5610145750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:42.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.885+0000 7f560fe00640 1 -- 172.21.15.6:0/2662055018 >> 172.21.15.6:0/2662055018 conn(0x7f561006b3d0 msgr2=0x7f56100b0930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:42.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.885+0000 7f560fe00640 1 -- 172.21.15.6:0/2662055018 shutdown_connections 2024-03-04T09:27:42.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:42.885+0000 7f560fe00640 1 -- 172.21.15.6:0/2662055018 wait complete. 2024-03-04T09:27:42.901 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-04T09:27:42.902 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:27:43.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.117+0000 7f102de00640 1 Processor -- start 2024-03-04T09:27:43.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.118+0000 7f102de00640 1 -- start start 2024-03-04T09:27:43.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.119+0000 7f102de00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f1028070f10 0x7f10280cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:43.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.119+0000 7f102de00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f10280cce90 0x7f102815c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:43.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.119+0000 7f102de00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10280714d0 con 0x7f1028145300 2024-03-04T09:27:43.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.119+0000 7f102de00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10280cd300 con 0x7f1028070f10 2024-03-04T09:27:43.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.119+0000 7f102de00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10280717b0 con 0x7f10280cce90 2024-03-04T09:27:43.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.120+0000 7f1027e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f1028070f10 0x7f10280cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:43.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.120+0000 7f1027e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f1028070f10 0x7f10280cc950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36712/0 (socket says 172.21.15.6:36712) 2024-03-04T09:27:43.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.120+0000 7f1027e00640 1 -- 172.21.15.6:0/3028444872 learned_addr learned my addr 172.21.15.6:0/3028444872 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:27:43.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.120+0000 7f102d400640 1 --2- 172.21.15.6:0/3028444872 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f10280cce90 0x7f102815c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:43.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.120+0000 7f1027e00640 1 -- 172.21.15.6:0/3028444872 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f10280cce90 msgr2=0x7f102815c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:27:43.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.120+0000 7f1027e00640 1 --2- 172.21.15.6:0/3028444872 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f10280cce90 0x7f102815c570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:43.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.120+0000 7f1027e00640 1 -- 172.21.15.6:0/3028444872 >> v1:172.21.15.146:6789/0 conn(0x7f1028145300 legacy=0x7f10280b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:27:43.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.120+0000 7f1027e00640 1 -- 172.21.15.6:0/3028444872 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1028071a90 con 0x7f1028070f10 2024-03-04T09:27:43.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.122+0000 7f1027e00640 1 --2- 172.21.15.6:0/3028444872 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1028070f10 0x7f10280cc950 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f1018009870 tx=0x7f101802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=823495a18bc73649 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:43.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.122+0000 7f1027400640 1 -- 172.21.15.6:0/3028444872 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1018035020 con 0x7f1028070f10 2024-03-04T09:27:43.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.122+0000 7f1027400640 1 -- 172.21.15.6:0/3028444872 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1018031640 con 0x7f1028070f10 2024-03-04T09:27:43.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.122+0000 7f1027400640 1 -- 172.21.15.6:0/3028444872 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f101802d040 con 0x7f1028070f10 2024-03-04T09:27:43.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.123+0000 7f102de00640 1 -- 172.21.15.6:0/3028444872 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1028070f10 msgr2=0x7f10280cc950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:43.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.123+0000 7f102de00640 1 --2- 172.21.15.6:0/3028444872 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1028070f10 0x7f10280cc950 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f1018009870 tx=0x7f101802b8f0 comp rx=0 tx=0).stop 2024-03-04T09:27:43.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.123+0000 7f102de00640 1 -- 172.21.15.6:0/3028444872 shutdown_connections 2024-03-04T09:27:43.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.123+0000 7f102de00640 1 --2- 172.21.15.6:0/3028444872 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f10280cce90 0x7f102815c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:43.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.123+0000 7f102de00640 1 --2- 172.21.15.6:0/3028444872 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1028070f10 0x7f10280cc950 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:43.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.123+0000 7f102de00640 1 -- 172.21.15.6:0/3028444872 >> 172.21.15.6:0/3028444872 conn(0x7f102806b3d0 msgr2=0x7f102806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:43.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.123+0000 7f102de00640 1 -- 172.21.15.6:0/3028444872 shutdown_connections 2024-03-04T09:27:43.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.123+0000 7f102de00640 1 -- 172.21.15.6:0/3028444872 wait complete. 2024-03-04T09:27:43.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.125+0000 7f102de00640 1 Processor -- start 2024-03-04T09:27:43.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.125+0000 7f102de00640 1 -- start start 2024-03-04T09:27:43.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.127+0000 7f102de00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1028070f10 0x7f10281f3a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:43.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.127+0000 7f102de00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f10280cce90 0x7f1028140dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:43.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.127+0000 7f102de00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1028145300 0x7f10281e5e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:43.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.127+0000 7f102de00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f102815cab0 con 0x7f10280cce90 2024-03-04T09:27:43.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.127+0000 7f102de00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10281456e0 con 0x7f1028145300 2024-03-04T09:27:43.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.127+0000 7f102de00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f102815cc30 con 0x7f1028070f10 2024-03-04T09:27:43.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.127+0000 7f1027e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f10280cce90 0x7f1028140dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:43.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.127+0000 7f1027e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f10280cce90 0x7f1028140dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36728/0 (socket says 172.21.15.6:36728) 2024-03-04T09:27:43.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.127+0000 7f1027e00640 1 -- 172.21.15.6:0/2079234268 learned_addr learned my addr 172.21.15.6:0/2079234268 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:27:43.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.127+0000 7f102d400640 1 --2- 172.21.15.6:0/2079234268 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1028145300 0x7f10281e5e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:43.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.128+0000 7f102ca00640 1 --2- 172.21.15.6:0/2079234268 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1028070f10 0x7f10281f3a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:43.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.128+0000 7f1027e00640 1 -- 172.21.15.6:0/2079234268 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1028070f10 msgr2=0x7f10281f3a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:43.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.128+0000 7f1027e00640 1 --2- 172.21.15.6:0/2079234268 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1028070f10 0x7f10281f3a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:43.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.128+0000 7f1027e00640 1 -- 172.21.15.6:0/2079234268 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1028145300 msgr2=0x7f10281e5e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:43.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.128+0000 7f1027e00640 1 --2- 172.21.15.6:0/2079234268 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1028145300 0x7f10281e5e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:43.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.128+0000 7f1027e00640 1 -- 172.21.15.6:0/2079234268 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10280ad920 con 0x7f10280cce90 2024-03-04T09:27:43.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.128+0000 7f102ca00640 1 --2- 172.21.15.6:0/2079234268 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1028070f10 0x7f10281f3a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:27:43.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.128+0000 7f1027e00640 1 --2- 172.21.15.6:0/2079234268 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f10280cce90 0x7f1028140dd0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f1018009990 tx=0x7f10180046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:43.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.128+0000 7f1024c00640 1 -- 172.21.15.6:0/2079234268 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1018035040 con 0x7f10280cce90 2024-03-04T09:27:43.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.128+0000 7f1024c00640 1 -- 172.21.15.6:0/2079234268 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1018039bf0 con 0x7f10280cce90 2024-03-04T09:27:43.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.128+0000 7f1024c00640 1 -- 172.21.15.6:0/2079234268 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1018045480 con 0x7f10280cce90 2024-03-04T09:27:43.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.128+0000 7f102de00640 1 -- 172.21.15.6:0/2079234268 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10281e65d0 con 0x7f10280cce90 2024-03-04T09:27:43.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.129+0000 7f102de00640 1 -- 172.21.15.6:0/2079234268 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10281e68b0 con 0x7f10280cce90 2024-03-04T09:27:43.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.130+0000 7f102de00640 1 -- 172.21.15.6:0/2079234268 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f102804f330 con 0x7f10280cce90 2024-03-04T09:27:43.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.131+0000 7f1024c00640 1 -- 172.21.15.6:0/2079234268 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1018038020 con 0x7f10280cce90 2024-03-04T09:27:43.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.132+0000 7f1024c00640 1 --2- 172.21.15.6:0/2079234268 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f100c06b4d0 0x7f100c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:43.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.132+0000 7f1024c00640 1 -- 172.21.15.6:0/2079234268 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f10180ac090 con 0x7f10280cce90 2024-03-04T09:27:43.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.133+0000 7f102ca00640 1 --2- 172.21.15.6:0/2079234268 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f100c06b4d0 0x7f100c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:43.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.134+0000 7f102ca00640 1 --2- 172.21.15.6:0/2079234268 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f100c06b4d0 0x7f100c06d990 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f10280af490 tx=0x7f101c009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:43.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.136+0000 7f1024c00640 1 -- 172.21.15.6:0/2079234268 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1018042330 con 0x7f10280cce90 2024-03-04T09:27:43.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.367+0000 7f102de00640 1 -- 172.21.15.6:0/2079234268 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f10280b4bb0 con 0x7f10280cce90 2024-03-04T09:27:43.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.368+0000 7f1024c00640 1 -- 172.21.15.6:0/2079234268 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+4974 (secure 0 0 0) 0x7f101807afa0 con 0x7f10280cce90 2024-03-04T09:27:43.366 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:27:43.366 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4913,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/1049631914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":1049631914},{"type":"v1","addr":"172.21.15.146:6835","nonce":1049631914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":5092,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/424062551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":424062551},{"type":"v1","addr":"172.21.15.6:6833","nonce":424062551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":19},{"gid":5095,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/3300873502","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":3300873502},{"type":"v1","addr":"172.21.15.6:6835","nonce":3300873502}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:27:16.785309+0000","modified":"2024-03-04T09:27:19.824667+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5110},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5110":{"gid":5110,"name":"b","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.146:6837/3982958749","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":3982958749},{"type":"v1","addr":"172.21.15.146:6837","nonce":3982958749}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5110,"qdb_cluster":[5110]},"id":3}]} 2024-03-04T09:27:43.366 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 25 2024-03-04T09:27:43.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.371+0000 7f102de00640 1 -- 172.21.15.6:0/2079234268 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f100c06b4d0 msgr2=0x7f100c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:43.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.371+0000 7f102de00640 1 --2- 172.21.15.6:0/2079234268 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f100c06b4d0 0x7f100c06d990 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f10280af490 tx=0x7f101c009040 comp rx=0 tx=0).stop 2024-03-04T09:27:43.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.371+0000 7f102de00640 1 -- 172.21.15.6:0/2079234268 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f10280cce90 msgr2=0x7f1028140dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:43.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.371+0000 7f102de00640 1 --2- 172.21.15.6:0/2079234268 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f10280cce90 0x7f1028140dd0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f1018009990 tx=0x7f10180046c0 comp rx=0 tx=0).stop 2024-03-04T09:27:43.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.371+0000 7f102de00640 1 -- 172.21.15.6:0/2079234268 shutdown_connections 2024-03-04T09:27:43.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.371+0000 7f102de00640 1 --2- 172.21.15.6:0/2079234268 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1028145300 0x7f10281e5e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:43.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.371+0000 7f102de00640 1 --2- 172.21.15.6:0/2079234268 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f10280cce90 0x7f1028140dd0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:43.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.371+0000 7f102de00640 1 --2- 172.21.15.6:0/2079234268 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f100c06b4d0 0x7f100c06d990 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:43.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.371+0000 7f102de00640 1 --2- 172.21.15.6:0/2079234268 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1028070f10 0x7f10281f3a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:43.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.371+0000 7f102de00640 1 -- 172.21.15.6:0/2079234268 >> 172.21.15.6:0/2079234268 conn(0x7f102806b3d0 msgr2=0x7f10280cd930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:43.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.371+0000 7f102de00640 1 -- 172.21.15.6:0/2079234268 shutdown_connections 2024-03-04T09:27:43.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.371+0000 7f102de00640 1 -- 172.21.15.6:0/2079234268 wait complete. 2024-03-04T09:27:43.386 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:27:43.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.604+0000 7f3c77200640 1 Processor -- start 2024-03-04T09:27:43.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.604+0000 7f3c77200640 1 -- start start 2024-03-04T09:27:43.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.606+0000 7f3c77200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f3c78145300 0x7f3c780b49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:43.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.606+0000 7f3c77200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f3c780cd060 0x7f3c7815c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:43.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.606+0000 7f3c77200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c780714d0 con 0x7f3c78070f10 2024-03-04T09:27:43.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.606+0000 7f3c77200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c780717b0 con 0x7f3c780cd060 2024-03-04T09:27:43.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.606+0000 7f3c77200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c78071a90 con 0x7f3c78145300 2024-03-04T09:27:43.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.606+0000 7f3c76800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f3c780cd060 0x7f3c7815c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:43.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.606+0000 7f3c76800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f3c780cd060 0x7f3c7815c570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36744/0 (socket says 172.21.15.6:36744) 2024-03-04T09:27:43.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.606+0000 7f3c76800640 1 -- 172.21.15.6:0/2335203177 learned_addr learned my addr 172.21.15.6:0/2335203177 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:27:43.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.607+0000 7f3c75e00640 1 --2- 172.21.15.6:0/2335203177 >> v2:172.21.15.146:3300/0 conn(0x7f3c78145300 0x7f3c780b49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:43.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.607+0000 7f3c76800640 1 -- 172.21.15.6:0/2335203177 >> v2:172.21.15.146:3300/0 conn(0x7f3c78145300 msgr2=0x7f3c780b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:27:43.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.607+0000 7f3c76800640 1 --2- 172.21.15.6:0/2335203177 >> v2:172.21.15.146:3300/0 conn(0x7f3c78145300 0x7f3c780b49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:43.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.607+0000 7f3c76800640 1 -- 172.21.15.6:0/2335203177 >> v1:172.21.15.146:6789/0 conn(0x7f3c78070f10 legacy=0x7f3c780cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:27:43.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.607+0000 7f3c76800640 1 -- 172.21.15.6:0/2335203177 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c78071d70 con 0x7f3c780cd060 2024-03-04T09:27:43.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.607+0000 7f3c75e00640 1 --2- 172.21.15.6:0/2335203177 >> v2:172.21.15.146:3300/0 conn(0x7f3c78145300 0x7f3c780b49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T09:27:43.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.608+0000 7f3c76800640 1 --2- 172.21.15.6:0/2335203177 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3c780cd060 0x7f3c7815c570 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f3c780cf990 tx=0x7f3c7002be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce54bc6100e881d server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:43.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.609+0000 7f3c74a00640 1 -- 172.21.15.6:0/2335203177 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c7002c9f0 con 0x7f3c780cd060 2024-03-04T09:27:43.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.609+0000 7f3c74a00640 1 -- 172.21.15.6:0/2335203177 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c7002cb90 con 0x7f3c780cd060 2024-03-04T09:27:43.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.609+0000 7f3c74a00640 1 -- 172.21.15.6:0/2335203177 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c700343f0 con 0x7f3c780cd060 2024-03-04T09:27:43.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.609+0000 7f3c77200640 1 -- 172.21.15.6:0/2335203177 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3c780cd060 msgr2=0x7f3c7815c570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:43.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.609+0000 7f3c77200640 1 --2- 172.21.15.6:0/2335203177 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3c780cd060 0x7f3c7815c570 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f3c780cf990 tx=0x7f3c7002be50 comp rx=0 tx=0).stop 2024-03-04T09:27:43.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.610+0000 7f3c77200640 1 -- 172.21.15.6:0/2335203177 shutdown_connections 2024-03-04T09:27:43.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.610+0000 7f3c77200640 1 --2- 172.21.15.6:0/2335203177 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3c780cd060 0x7f3c7815c570 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:43.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.610+0000 7f3c77200640 1 --2- 172.21.15.6:0/2335203177 >> v2:172.21.15.146:3300/0 conn(0x7f3c78145300 0x7f3c780b49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:43.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.610+0000 7f3c77200640 1 -- 172.21.15.6:0/2335203177 >> 172.21.15.6:0/2335203177 conn(0x7f3c7806b3d0 msgr2=0x7f3c7806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:43.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.610+0000 7f3c77200640 1 -- 172.21.15.6:0/2335203177 shutdown_connections 2024-03-04T09:27:43.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.610+0000 7f3c77200640 1 -- 172.21.15.6:0/2335203177 wait complete. 2024-03-04T09:27:43.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.611+0000 7f3c77200640 1 Processor -- start 2024-03-04T09:27:43.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.612+0000 7f3c77200640 1 -- start start 2024-03-04T09:27:43.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.612+0000 7f3c77200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3c78070f10 0x7f3c780aec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:43.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.612+0000 7f3c77200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3c780cd060 0x7f3c7815b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:43.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.612+0000 7f3c77200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3c78145300 0x7f3c7815b800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:43.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.613+0000 7f3c77200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c7815cab0 con 0x7f3c78070f10 2024-03-04T09:27:43.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.613+0000 7f3c77200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c781456e0 con 0x7f3c780cd060 2024-03-04T09:27:43.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.613+0000 7f3c77200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c7815cc30 con 0x7f3c78145300 2024-03-04T09:27:43.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.613+0000 7f3c75e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3c78070f10 0x7f3c780aec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:43.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.613+0000 7f3c75e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3c78070f10 0x7f3c780aec70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36756/0 (socket says 172.21.15.6:36756) 2024-03-04T09:27:43.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.613+0000 7f3c75e00640 1 -- 172.21.15.6:0/1318276598 learned_addr learned my addr 172.21.15.6:0/1318276598 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:27:43.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.613+0000 7f3c75400640 1 --2- 172.21.15.6:0/1318276598 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3c780cd060 0x7f3c7815b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:43.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.613+0000 7f3c76800640 1 --2- 172.21.15.6:0/1318276598 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3c78145300 0x7f3c7815b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:43.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.613+0000 7f3c75e00640 1 -- 172.21.15.6:0/1318276598 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3c78145300 msgr2=0x7f3c7815b800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:43.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.613+0000 7f3c75e00640 1 --2- 172.21.15.6:0/1318276598 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3c78145300 0x7f3c7815b800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:43.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.613+0000 7f3c75e00640 1 -- 172.21.15.6:0/1318276598 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3c780cd060 msgr2=0x7f3c7815b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:43.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.613+0000 7f3c75e00640 1 --2- 172.21.15.6:0/1318276598 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3c780cd060 0x7f3c7815b2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:43.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.613+0000 7f3c75e00640 1 -- 172.21.15.6:0/1318276598 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c78131fa0 con 0x7f3c78070f10 2024-03-04T09:27:43.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.613+0000 7f3c75400640 1 --2- 172.21.15.6:0/1318276598 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3c780cd060 0x7f3c7815b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:27:43.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.613+0000 7f3c76800640 1 --2- 172.21.15.6:0/1318276598 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3c78145300 0x7f3c7815b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:27:43.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.614+0000 7f3c75e00640 1 --2- 172.21.15.6:0/1318276598 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3c78070f10 0x7f3c780aec70 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f3c6800d800 tx=0x7f3c6800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:43.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.614+0000 7f3c6e000640 1 -- 172.21.15.6:0/1318276598 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c68004030 con 0x7f3c78070f10 2024-03-04T09:27:43.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.614+0000 7f3c6e000640 1 -- 172.21.15.6:0/1318276598 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c680041d0 con 0x7f3c78070f10 2024-03-04T09:27:43.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.614+0000 7f3c77200640 1 -- 172.21.15.6:0/1318276598 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c78132280 con 0x7f3c78070f10 2024-03-04T09:27:43.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.614+0000 7f3c6e000640 1 -- 172.21.15.6:0/1318276598 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c6800b6c0 con 0x7f3c78070f10 2024-03-04T09:27:43.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.614+0000 7f3c77200640 1 -- 172.21.15.6:0/1318276598 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c78132840 con 0x7f3c78070f10 2024-03-04T09:27:43.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.615+0000 7f3c77200640 1 -- 172.21.15.6:0/1318276598 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c7804f330 con 0x7f3c78070f10 2024-03-04T09:27:43.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.616+0000 7f3c6e000640 1 -- 172.21.15.6:0/1318276598 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3c6800b860 con 0x7f3c78070f10 2024-03-04T09:27:43.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.617+0000 7f3c6e000640 1 --2- 172.21.15.6:0/1318276598 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f3c5806b4d0 0x7f3c5806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:43.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.617+0000 7f3c6e000640 1 -- 172.21.15.6:0/1318276598 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3c6808a4b0 con 0x7f3c78070f10 2024-03-04T09:27:43.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.618+0000 7f3c75400640 1 --2- 172.21.15.6:0/1318276598 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f3c5806b4d0 0x7f3c5806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:43.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.619+0000 7f3c75400640 1 --2- 172.21.15.6:0/1318276598 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f3c5806b4d0 0x7f3c5806d990 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f3c78071fd0 tx=0x7f3c64009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:43.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.621+0000 7f3c6e000640 1 -- 172.21.15.6:0/1318276598 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3c68059320 con 0x7f3c78070f10 2024-03-04T09:27:43.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.861+0000 7f3c77200640 1 -- 172.21.15.6:0/1318276598 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3c780b28b0 con 0x7f3c78070f10 2024-03-04T09:27:43.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.863+0000 7f3c6e000640 1 -- 172.21.15.6:0/1318276598 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+4974 (secure 0 0 0) 0x7f3c68058c20 con 0x7f3c78070f10 2024-03-04T09:27:43.860 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:27:43.860 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4913,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/1049631914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":1049631914},{"type":"v1","addr":"172.21.15.146:6835","nonce":1049631914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":5092,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/424062551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":424062551},{"type":"v1","addr":"172.21.15.6:6833","nonce":424062551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":19},{"gid":5095,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/3300873502","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":3300873502},{"type":"v1","addr":"172.21.15.6:6835","nonce":3300873502}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:27:16.785309+0000","modified":"2024-03-04T09:27:19.824667+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5110},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5110":{"gid":5110,"name":"b","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.146:6837/3982958749","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":3982958749},{"type":"v1","addr":"172.21.15.146:6837","nonce":3982958749}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5110,"qdb_cluster":[5110]},"id":3}]} 2024-03-04T09:27:43.860 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 25 2024-03-04T09:27:43.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.865+0000 7f3c77200640 1 -- 172.21.15.6:0/1318276598 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f3c5806b4d0 msgr2=0x7f3c5806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:43.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.865+0000 7f3c77200640 1 --2- 172.21.15.6:0/1318276598 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f3c5806b4d0 0x7f3c5806d990 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f3c78071fd0 tx=0x7f3c64009040 comp rx=0 tx=0).stop 2024-03-04T09:27:43.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.865+0000 7f3c77200640 1 -- 172.21.15.6:0/1318276598 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3c78070f10 msgr2=0x7f3c780aec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:43.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.865+0000 7f3c77200640 1 --2- 172.21.15.6:0/1318276598 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3c78070f10 0x7f3c780aec70 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f3c6800d800 tx=0x7f3c6800dcc0 comp rx=0 tx=0).stop 2024-03-04T09:27:43.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.865+0000 7f3c77200640 1 -- 172.21.15.6:0/1318276598 shutdown_connections 2024-03-04T09:27:43.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.865+0000 7f3c77200640 1 --2- 172.21.15.6:0/1318276598 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f3c5806b4d0 0x7f3c5806d990 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:43.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.865+0000 7f3c77200640 1 --2- 172.21.15.6:0/1318276598 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3c78145300 0x7f3c7815b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:43.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.865+0000 7f3c77200640 1 --2- 172.21.15.6:0/1318276598 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3c780cd060 0x7f3c7815b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:43.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.865+0000 7f3c77200640 1 --2- 172.21.15.6:0/1318276598 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3c78070f10 0x7f3c780aec70 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:43.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.865+0000 7f3c77200640 1 -- 172.21.15.6:0/1318276598 >> 172.21.15.6:0/1318276598 conn(0x7f3c7806b3d0 msgr2=0x7f3c780af7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:43.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.866+0000 7f3c77200640 1 -- 172.21.15.6:0/1318276598 shutdown_connections 2024-03-04T09:27:43.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:43.866+0000 7f3c77200640 1 -- 172.21.15.6:0/1318276598 wait complete. 2024-03-04T09:27:43.882 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T09:27:44.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.104+0000 7f3646600640 1 Processor -- start 2024-03-04T09:27:44.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.105+0000 7f3646600640 1 -- start start 2024-03-04T09:27:44.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.106+0000 7f3646600640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f364015e2a0 0x7f36400cef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:44.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.106+0000 7f3646600640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36400d1120 con 0x7f36400cf460 2024-03-04T09:27:44.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.106+0000 7f3646600640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36400d1400 con 0x7f364015dec0 2024-03-04T09:27:44.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.106+0000 7f3646600640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36400b0ea0 con 0x7f364015e2a0 2024-03-04T09:27:44.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.107+0000 7f3645c00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f36400cf460 0x7f36400b1330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:45288/0 (socket says 172.21.15.6:45288) 2024-03-04T09:27:44.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.107+0000 7f3645c00640 1 -- 172.21.15.6:0/568406387 learned_addr learned my addr 172.21.15.6:0/568406387 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:27:44.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.107+0000 7f3637e00640 1 --2- 172.21.15.6:0/568406387 >> v2:172.21.15.146:3300/0 conn(0x7f364015e2a0 0x7f36400cef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:44.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.107+0000 7f363fe00640 1 -- 172.21.15.6:0/568406387 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2045385276 0 0) 0x7f36400d1120 con 0x7f36400cf460 2024-03-04T09:27:44.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.107+0000 7f363fe00640 1 -- 172.21.15.6:0/568406387 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3620003520 con 0x7f36400cf460 2024-03-04T09:27:44.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.107+0000 7f363fe00640 1 -- 172.21.15.6:0/568406387 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1898267209 0 0) 0x7f3620003520 con 0x7f36400cf460 2024-03-04T09:27:44.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.107+0000 7f363fe00640 1 -- 172.21.15.6:0/568406387 >> v2:172.21.15.146:3300/0 conn(0x7f364015e2a0 msgr2=0x7f36400cef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:27:44.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.107+0000 7f363fe00640 1 --2- 172.21.15.6:0/568406387 >> v2:172.21.15.146:3300/0 conn(0x7f364015e2a0 0x7f36400cef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:44.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.107+0000 7f363fe00640 1 -- 172.21.15.6:0/568406387 >> v1:172.21.15.146:6789/0 conn(0x7f364015dec0 legacy=0x7f36400ce810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:44.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.107+0000 7f363fe00640 1 -- 172.21.15.6:0/568406387 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36400b0b30 con 0x7f36400cf460 2024-03-04T09:27:44.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.107+0000 7f363fe00640 1 -- 172.21.15.6:0/568406387 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7f36380019f0 con 0x7f36400cf460 2024-03-04T09:27:44.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.108+0000 7f363fe00640 1 -- 172.21.15.6:0/568406387 >> v1:172.21.15.6:6789/0 conn(0x7f36400cf460 legacy=0x7f36400b1330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:44.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.108+0000 7f363fe00640 1 --2- 172.21.15.6:0/568406387 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3620003a00 0x7f3620003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:44.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.108+0000 7f363fe00640 1 --2- 172.21.15.6:0/568406387 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3620004800 0x7f3620004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:44.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.108+0000 7f363fe00640 1 --2- 172.21.15.6:0/568406387 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f36200051c0 0x7f362000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:44.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.108+0000 7f363fe00640 1 -- 172.21.15.6:0/568406387 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36400b0ea0 con 0x7f3620003a00 2024-03-04T09:27:44.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.108+0000 7f363fe00640 1 -- 172.21.15.6:0/568406387 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f362000c0c0 con 0x7f36200051c0 2024-03-04T09:27:44.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.108+0000 7f363fe00640 1 -- 172.21.15.6:0/568406387 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f362000c2c0 con 0x7f3620004800 2024-03-04T09:27:44.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.108+0000 7f3645200640 1 --2- 172.21.15.6:0/568406387 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3620003a00 0x7f3620003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:44.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.108+0000 7f3645200640 1 -- 172.21.15.6:0/568406387 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3620004800 msgr2=0x7f3620004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:27:44.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.108+0000 7f3637e00640 1 --2- 172.21.15.6:0/568406387 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3620004800 0x7f3620004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:44.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.108+0000 7f3645200640 1 --2- 172.21.15.6:0/568406387 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3620004800 0x7f3620004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:44.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.108+0000 7f3645200640 1 -- 172.21.15.6:0/568406387 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f36200051c0 msgr2=0x7f362000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:27:44.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.108+0000 7f3645200640 1 --2- 172.21.15.6:0/568406387 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f36200051c0 0x7f362000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:44.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.108+0000 7f3645200640 1 -- 172.21.15.6:0/568406387 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3620003520 con 0x7f3620003a00 2024-03-04T09:27:44.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.108+0000 7f3645c00640 1 -- 172.21.15.6:0/568406387 reap_dead start 2024-03-04T09:27:44.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.110+0000 7f3645200640 1 --2- 172.21.15.6:0/568406387 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3620003a00 0x7f3620003e20 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f3628001750 tx=0x7f3620005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cb160b70fc647d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:44.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.110+0000 7f363fe00640 1 -- 172.21.15.6:0/568406387 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3628001370 con 0x7f3620003a00 2024-03-04T09:27:44.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.110+0000 7f363fe00640 1 -- 172.21.15.6:0/568406387 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36280052d0 con 0x7f3620003a00 2024-03-04T09:27:44.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.110+0000 7f363fe00640 1 -- 172.21.15.6:0/568406387 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3628003830 con 0x7f3620003a00 2024-03-04T09:27:44.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.110+0000 7f3646600640 1 -- 172.21.15.6:0/568406387 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3620003a00 msgr2=0x7f3620003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:44.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.110+0000 7f3646600640 1 --2- 172.21.15.6:0/568406387 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3620003a00 0x7f3620003e20 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f3628001750 tx=0x7f3620005830 comp rx=0 tx=0).stop 2024-03-04T09:27:44.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.110+0000 7f3646600640 1 -- 172.21.15.6:0/568406387 shutdown_connections 2024-03-04T09:27:44.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.110+0000 7f3646600640 1 --2- 172.21.15.6:0/568406387 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3620003a00 0x7f3620003e20 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:44.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.110+0000 7f3646600640 1 -- 172.21.15.6:0/568406387 >> 172.21.15.6:0/568406387 conn(0x7f36400d3d40 msgr2=0x7f36400d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:44.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.111+0000 7f3646600640 1 -- 172.21.15.6:0/568406387 shutdown_connections 2024-03-04T09:27:44.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.111+0000 7f3646600640 1 -- 172.21.15.6:0/568406387 wait complete. 2024-03-04T09:27:44.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.112+0000 7f3646600640 1 Processor -- start 2024-03-04T09:27:44.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.112+0000 7f3646600640 1 -- start start 2024-03-04T09:27:44.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.113+0000 7f3646600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3620003a00 0x7f36400c90f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:44.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.113+0000 7f3646600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f36400ccc70 0x7f36400cd0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:44.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.113+0000 7f3646600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f36400c9d30 0x7f36400ca1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:44.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.113+0000 7f3646600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36400ce990 con 0x7f36400ccc70 2024-03-04T09:27:44.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.113+0000 7f3646600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36400ce810 con 0x7f36400c9d30 2024-03-04T09:27:44.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.113+0000 7f3646600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36400ceb10 con 0x7f3620003a00 2024-03-04T09:27:44.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.114+0000 7f3637e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f36400ccc70 0x7f36400cd0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:44.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.114+0000 7f3637e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f36400ccc70 0x7f36400cd0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36776/0 (socket says 172.21.15.6:36776) 2024-03-04T09:27:44.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.114+0000 7f3637e00640 1 -- 172.21.15.6:0/3073573999 learned_addr learned my addr 172.21.15.6:0/3073573999 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:27:44.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.114+0000 7f3645200640 1 --2- 172.21.15.6:0/3073573999 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3620003a00 0x7f36400c90f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:44.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.114+0000 7f3637e00640 1 -- 172.21.15.6:0/3073573999 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3620003a00 msgr2=0x7f36400c90f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:44.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.114+0000 7f3637e00640 1 --2- 172.21.15.6:0/3073573999 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3620003a00 0x7f36400c90f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:44.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.114+0000 7f3645c00640 1 --2- 172.21.15.6:0/3073573999 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f36400c9d30 0x7f36400ca1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:44.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.114+0000 7f3637e00640 1 -- 172.21.15.6:0/3073573999 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f36400c9d30 msgr2=0x7f36400ca1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:44.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.114+0000 7f3637e00640 1 --2- 172.21.15.6:0/3073573999 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f36400c9d30 0x7f36400ca1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:44.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.114+0000 7f3637e00640 1 -- 172.21.15.6:0/3073573999 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36400c9630 con 0x7f36400ccc70 2024-03-04T09:27:44.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.115+0000 7f3637e00640 1 --2- 172.21.15.6:0/3073573999 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f36400ccc70 0x7f36400cd0f0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f36300108e0 tx=0x7f3630010da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:44.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.115+0000 7f363d600640 1 -- 172.21.15.6:0/3073573999 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f363000f040 con 0x7f36400ccc70 2024-03-04T09:27:44.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.115+0000 7f363d600640 1 -- 172.21.15.6:0/3073573999 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f363000eb60 con 0x7f36400ccc70 2024-03-04T09:27:44.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.115+0000 7f363d600640 1 -- 172.21.15.6:0/3073573999 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f363001c400 con 0x7f36400ccc70 2024-03-04T09:27:44.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.115+0000 7f3646600640 1 -- 172.21.15.6:0/3073573999 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36400caf40 con 0x7f36400ccc70 2024-03-04T09:27:44.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.115+0000 7f3646600640 1 -- 172.21.15.6:0/3073573999 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36400cb500 con 0x7f36400ccc70 2024-03-04T09:27:44.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.116+0000 7f3646600640 1 -- 172.21.15.6:0/3073573999 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f364004f330 con 0x7f36400ccc70 2024-03-04T09:27:44.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.117+0000 7f363d600640 1 -- 172.21.15.6:0/3073573999 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f363001c5a0 con 0x7f36400ccc70 2024-03-04T09:27:44.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.118+0000 7f363d600640 1 --2- 172.21.15.6:0/3073573999 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f361c06b500 0x7f361c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:44.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.118+0000 7f363d600640 1 -- 172.21.15.6:0/3073573999 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f363008a860 con 0x7f36400ccc70 2024-03-04T09:27:44.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.119+0000 7f3645200640 1 --2- 172.21.15.6:0/3073573999 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f361c06b500 0x7f361c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:44.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.120+0000 7f3645200640 1 --2- 172.21.15.6:0/3073573999 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f361c06b500 0x7f361c06d9c0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f3628033000 tx=0x7f3628009030 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:44.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.122+0000 7f363d600640 1 -- 172.21.15.6:0/3073573999 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3630018030 con 0x7f36400ccc70 2024-03-04T09:27:44.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.327+0000 7f3646600640 1 -- 172.21.15.6:0/3073573999 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f36400d1d70 con 0x7f36400ccc70 2024-03-04T09:27:44.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.330+0000 7f363d600640 1 -- 172.21.15.6:0/3073573999 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v44) v1 ==== 74+0+16565 (secure 0 0 0) 0x7f36300596d0 con 0x7f36400ccc70 2024-03-04T09:27:44.327 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:27:44.327 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":44,"fsid":"edce0578-5ae4-4402-b45a-552bdef84d7c","created":"2024-03-04T09:24:54.523780+0000","modified":"2024-03-04T09:27:18.307032+0000","last_up_change":"2024-03-04T09:25:07.089049+0000","last_in_change":"2024-03-04T09:24:57.982077+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T09:25:07.935198+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-03-04T09:27:14.657490+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"43","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-03-04T09:27:15.806886+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a96574f-81f1-4bc6-8463-757520b84a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6825","nonce":1364449401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6827","nonce":1364449401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6831","nonce":1364449401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6829","nonce":1364449401}]},"public_addr":"172.21.15.6:6825/1364449401","cluster_addr":"172.21.15.6:6827/1364449401","heartbeat_back_addr":"172.21.15.6:6831/1364449401","heartbeat_front_addr":"172.21.15.6:6829/1364449401","state":["exists","up"]},{"osd":1,"uuid":"7d79b9c1-6b87-476a-bc36-7db5ee64cd90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6809","nonce":2494103078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6811","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6813","nonce":2494103078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6821","nonce":2494103078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6817","nonce":2494103078}]},"public_addr":"172.21.15.6:6809/2494103078","cluster_addr":"172.21.15.6:6813/2494103078","heartbeat_back_addr":"172.21.15.6:6821/2494103078","heartbeat_front_addr":"172.21.15.6:6817/2494103078","state":["exists","up"]},{"osd":2,"uuid":"492331b9-4ee9-40b6-bacd-85d44cd6c45a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6812","nonce":1500066681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6816","nonce":1500066681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6824","nonce":1500066681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6820","nonce":1500066681}]},"public_addr":"172.21.15.6:6812/1500066681","cluster_addr":"172.21.15.6:6816/1500066681","heartbeat_back_addr":"172.21.15.6:6824/1500066681","heartbeat_front_addr":"172.21.15.6:6820/1500066681","state":["exists","up"]},{"osd":3,"uuid":"21bb2efe-9ba7-4bdc-bc31-180cd1102e21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6801","nonce":1558806123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6803","nonce":1558806123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6807","nonce":1558806123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6805","nonce":1558806123}]},"public_addr":"172.21.15.6:6801/1558806123","cluster_addr":"172.21.15.6:6803/1558806123","heartbeat_back_addr":"172.21.15.6:6807/1558806123","heartbeat_front_addr":"172.21.15.6:6805/1558806123","state":["exists","up"]},{"osd":4,"uuid":"6e2c4b71-19d9-4737-83d9-558705d8644a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6801","nonce":1526135780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6803","nonce":1526135780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6807","nonce":1526135780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6805","nonce":1526135780}]},"public_addr":"172.21.15.146:6801/1526135780","cluster_addr":"172.21.15.146:6803/1526135780","heartbeat_back_addr":"172.21.15.146:6807/1526135780","heartbeat_front_addr":"172.21.15.146:6805/1526135780","state":["exists","up"]},{"osd":5,"uuid":"fb69c525-f1be-4b7d-ab32-6c29b0d6fddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6809","nonce":460262507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6811","nonce":460262507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6815","nonce":460262507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6813","nonce":460262507}]},"public_addr":"172.21.15.146:6809/460262507","cluster_addr":"172.21.15.146:6811/460262507","heartbeat_back_addr":"172.21.15.146:6815/460262507","heartbeat_front_addr":"172.21.15.146:6813/460262507","state":["exists","up"]},{"osd":6,"uuid":"c32e7fed-370d-4401-8ce0-6a4335b0d168","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6823","nonce":1392897493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6827","nonce":1392897493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6831","nonce":1392897493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6829","nonce":1392897493}]},"public_addr":"172.21.15.146:6823/1392897493","cluster_addr":"172.21.15.146:6827/1392897493","heartbeat_back_addr":"172.21.15.146:6831/1392897493","heartbeat_front_addr":"172.21.15.146:6829/1392897493","state":["exists","up"]},{"osd":7,"uuid":"75a79b44-fa5f-426d-a7d3-71f21f78835c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6817","nonce":3268323156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6819","nonce":3268323156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6825","nonce":3268323156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6822","nonce":3268323156}]},"public_addr":"172.21.15.146:6817/3268323156","cluster_addr":"172.21.15.146:6819/3268323156","heartbeat_back_addr":"172.21.15.146:6825/3268323156","heartbeat_front_addr":"172.21.15.146:6822/3268323156","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.310437+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.295850+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.294674+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.526855+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.300023+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.595160+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.346046+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.558741+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T09:27:44.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.332+0000 7f3646600640 1 -- 172.21.15.6:0/3073573999 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f361c06b500 msgr2=0x7f361c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:44.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.332+0000 7f3646600640 1 --2- 172.21.15.6:0/3073573999 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f361c06b500 0x7f361c06d9c0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f3628033000 tx=0x7f3628009030 comp rx=0 tx=0).stop 2024-03-04T09:27:44.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.332+0000 7f3646600640 1 -- 172.21.15.6:0/3073573999 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f36400ccc70 msgr2=0x7f36400cd0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:44.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.332+0000 7f3646600640 1 --2- 172.21.15.6:0/3073573999 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f36400ccc70 0x7f36400cd0f0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f36300108e0 tx=0x7f3630010da0 comp rx=0 tx=0).stop 2024-03-04T09:27:44.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.332+0000 7f3646600640 1 -- 172.21.15.6:0/3073573999 shutdown_connections 2024-03-04T09:27:44.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.332+0000 7f3646600640 1 --2- 172.21.15.6:0/3073573999 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f36400c9d30 0x7f36400ca1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:44.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.332+0000 7f3646600640 1 --2- 172.21.15.6:0/3073573999 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f36400ccc70 0x7f36400cd0f0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:44.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.332+0000 7f3646600640 1 --2- 172.21.15.6:0/3073573999 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f361c06b500 0x7f361c06d9c0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:44.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.332+0000 7f3646600640 1 --2- 172.21.15.6:0/3073573999 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3620003a00 0x7f36400c90f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:44.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.332+0000 7f3646600640 1 -- 172.21.15.6:0/3073573999 >> 172.21.15.6:0/3073573999 conn(0x7f36400d3d40 msgr2=0x7f36400d2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:44.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.332+0000 7f3646600640 1 -- 172.21.15.6:0/3073573999 shutdown_connections 2024-03-04T09:27:44.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.332+0000 7f3646600640 1 -- 172.21.15.6:0/3073573999 wait complete. 2024-03-04T09:27:44.349 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-04T09:27:44.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.572+0000 7f5633200640 1 Processor -- start 2024-03-04T09:27:44.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.572+0000 7f5633200640 1 -- start start 2024-03-04T09:27:44.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.573+0000 7f5633200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f5634070f10 0x7f56340cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:44.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.573+0000 7f5633200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f56340cce90 0x7f563415c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:44.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.573+0000 7f5633200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56340714d0 con 0x7f5634145300 2024-03-04T09:27:44.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.573+0000 7f5633200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56340cd300 con 0x7f5634070f10 2024-03-04T09:27:44.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.573+0000 7f5633200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56340717b0 con 0x7f56340cce90 2024-03-04T09:27:44.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.574+0000 7f5629400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f5634070f10 0x7f56340cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:44.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.574+0000 7f5629400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f5634070f10 0x7f56340cc950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36792/0 (socket says 172.21.15.6:36792) 2024-03-04T09:27:44.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.574+0000 7f5629400640 1 -- 172.21.15.6:0/2852691002 learned_addr learned my addr 172.21.15.6:0/2852691002 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:27:44.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.574+0000 7f5632800640 1 --2- 172.21.15.6:0/2852691002 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f56340cce90 0x7f563415c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:44.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.574+0000 7f5629400640 1 -- 172.21.15.6:0/2852691002 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f56340cce90 msgr2=0x7f563415c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:27:44.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.574+0000 7f5629400640 1 --2- 172.21.15.6:0/2852691002 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f56340cce90 0x7f563415c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:44.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.574+0000 7f5629400640 1 -- 172.21.15.6:0/2852691002 >> v1:172.21.15.146:6789/0 conn(0x7f5634145300 legacy=0x7f56340b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:27:44.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.574+0000 7f5629400640 1 -- 172.21.15.6:0/2852691002 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5634071a90 con 0x7f5634070f10 2024-03-04T09:27:44.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.575+0000 7f5629400640 1 --2- 172.21.15.6:0/2852691002 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5634070f10 0x7f56340cc950 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f561c009870 tx=0x7f561c02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f787f82a21701ee server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:44.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.576+0000 7f5631400640 1 -- 172.21.15.6:0/2852691002 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f561c0317c0 con 0x7f5634070f10 2024-03-04T09:27:44.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.576+0000 7f5631400640 1 -- 172.21.15.6:0/2852691002 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f561c037070 con 0x7f5634070f10 2024-03-04T09:27:44.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.576+0000 7f5631400640 1 -- 172.21.15.6:0/2852691002 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f561c031b80 con 0x7f5634070f10 2024-03-04T09:27:44.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.576+0000 7f5633200640 1 -- 172.21.15.6:0/2852691002 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5634070f10 msgr2=0x7f56340cc950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:44.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.576+0000 7f5633200640 1 --2- 172.21.15.6:0/2852691002 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5634070f10 0x7f56340cc950 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f561c009870 tx=0x7f561c02b860 comp rx=0 tx=0).stop 2024-03-04T09:27:44.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.576+0000 7f5633200640 1 -- 172.21.15.6:0/2852691002 shutdown_connections 2024-03-04T09:27:44.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.576+0000 7f5633200640 1 --2- 172.21.15.6:0/2852691002 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f56340cce90 0x7f563415c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:44.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.576+0000 7f5633200640 1 --2- 172.21.15.6:0/2852691002 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5634070f10 0x7f56340cc950 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:44.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.576+0000 7f5633200640 1 -- 172.21.15.6:0/2852691002 >> 172.21.15.6:0/2852691002 conn(0x7f563406b3d0 msgr2=0x7f563406b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:44.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.577+0000 7f5633200640 1 -- 172.21.15.6:0/2852691002 shutdown_connections 2024-03-04T09:27:44.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.577+0000 7f5633200640 1 -- 172.21.15.6:0/2852691002 wait complete. 2024-03-04T09:27:44.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.578+0000 7f5633200640 1 Processor -- start 2024-03-04T09:27:44.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.578+0000 7f5633200640 1 -- start start 2024-03-04T09:27:44.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.579+0000 7f5633200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5634070f10 0x7f56340aa5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:44.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.579+0000 7f5633200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f56340cce90 0x7f56340a9470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:44.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.579+0000 7f5633200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5634145300 0x7f56340a99b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:44.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.579+0000 7f5633200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f563415cab0 con 0x7f5634070f10 2024-03-04T09:27:44.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.579+0000 7f5633200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56341456e0 con 0x7f56340cce90 2024-03-04T09:27:44.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.579+0000 7f5633200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f563415cc30 con 0x7f5634145300 2024-03-04T09:27:44.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.580+0000 7f5631e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5634070f10 0x7f56340aa5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:44.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.580+0000 7f5631e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5634070f10 0x7f56340aa5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36796/0 (socket says 172.21.15.6:36796) 2024-03-04T09:27:44.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.580+0000 7f5631e00640 1 -- 172.21.15.6:0/2909845390 learned_addr learned my addr 172.21.15.6:0/2909845390 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:27:44.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.580+0000 7f5629400640 1 --2- 172.21.15.6:0/2909845390 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f56340cce90 0x7f56340a9470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:44.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.580+0000 7f5632800640 1 --2- 172.21.15.6:0/2909845390 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5634145300 0x7f56340a99b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:44.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.580+0000 7f5631e00640 1 -- 172.21.15.6:0/2909845390 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5634145300 msgr2=0x7f56340a99b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:44.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.580+0000 7f5631e00640 1 --2- 172.21.15.6:0/2909845390 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5634145300 0x7f56340a99b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:44.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.580+0000 7f5631e00640 1 -- 172.21.15.6:0/2909845390 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f56340cce90 msgr2=0x7f56340a9470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:44.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.580+0000 7f5631e00640 1 --2- 172.21.15.6:0/2909845390 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f56340cce90 0x7f56340a9470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:44.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.580+0000 7f5631e00640 1 -- 172.21.15.6:0/2909845390 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56340aaf40 con 0x7f5634070f10 2024-03-04T09:27:44.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.581+0000 7f5631e00640 1 --2- 172.21.15.6:0/2909845390 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5634070f10 0x7f56340aa5d0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f562400b3f0 tx=0x7f562400b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:44.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.581+0000 7f562aa00640 1 -- 172.21.15.6:0/2909845390 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5624015020 con 0x7f5634070f10 2024-03-04T09:27:44.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.581+0000 7f562aa00640 1 -- 172.21.15.6:0/2909845390 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5624011830 con 0x7f5634070f10 2024-03-04T09:27:44.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.581+0000 7f562aa00640 1 -- 172.21.15.6:0/2909845390 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5624011c20 con 0x7f5634070f10 2024-03-04T09:27:44.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.581+0000 7f5633200640 1 -- 172.21.15.6:0/2909845390 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56340ab220 con 0x7f5634070f10 2024-03-04T09:27:44.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.581+0000 7f5633200640 1 -- 172.21.15.6:0/2909845390 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56340ab7e0 con 0x7f5634070f10 2024-03-04T09:27:44.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.582+0000 7f5633200640 1 -- 172.21.15.6:0/2909845390 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f563404f330 con 0x7f5634070f10 2024-03-04T09:27:44.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.583+0000 7f562aa00640 1 -- 172.21.15.6:0/2909845390 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5624018020 con 0x7f5634070f10 2024-03-04T09:27:44.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.584+0000 7f562aa00640 1 --2- 172.21.15.6:0/2909845390 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f561006b4d0 0x7f561006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:44.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.584+0000 7f562aa00640 1 -- 172.21.15.6:0/2909845390 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5624089e00 con 0x7f5634070f10 2024-03-04T09:27:44.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.585+0000 7f5629400640 1 --2- 172.21.15.6:0/2909845390 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f561006b4d0 0x7f561006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:44.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.586+0000 7f5629400640 1 --2- 172.21.15.6:0/2909845390 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f561006b4d0 0x7f561006d990 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f561c036000 tx=0x7f561c03d040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:44.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.589+0000 7f562aa00640 1 -- 172.21.15.6:0/2909845390 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f562401f030 con 0x7f5634070f10 2024-03-04T09:27:44.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.830+0000 7f5633200640 1 -- 172.21.15.6:0/2909845390 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f5634074ab0 con 0x7f5634070f10 2024-03-04T09:27:44.828 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-03-04T09:27:44.831+0000 7f1606a00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-04T09:27:44.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.834+0000 7f562aa00640 1 -- 172.21.15.6:0/2909845390 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v26) v1 ==== 161+0+0 (secure 0 0 0) 0x7f56240172c0 con 0x7f5634070f10 2024-03-04T09:27:44.831 INFO:teuthology.orchestra.run.smithi006.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-04T09:27:44.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.837+0000 7f5633200640 1 -- 172.21.15.6:0/2909845390 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f561006b4d0 msgr2=0x7f561006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:44.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.837+0000 7f5633200640 1 --2- 172.21.15.6:0/2909845390 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f561006b4d0 0x7f561006d990 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f561c036000 tx=0x7f561c03d040 comp rx=0 tx=0).stop 2024-03-04T09:27:44.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.837+0000 7f5633200640 1 -- 172.21.15.6:0/2909845390 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5634070f10 msgr2=0x7f56340aa5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:44.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.837+0000 7f5633200640 1 --2- 172.21.15.6:0/2909845390 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5634070f10 0x7f56340aa5d0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f562400b3f0 tx=0x7f562400b8b0 comp rx=0 tx=0).stop 2024-03-04T09:27:44.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.838+0000 7f5633200640 1 -- 172.21.15.6:0/2909845390 shutdown_connections 2024-03-04T09:27:44.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.838+0000 7f5633200640 1 --2- 172.21.15.6:0/2909845390 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f561006b4d0 0x7f561006d990 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:44.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.838+0000 7f5633200640 1 --2- 172.21.15.6:0/2909845390 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5634145300 0x7f56340a99b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:44.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.838+0000 7f5633200640 1 --2- 172.21.15.6:0/2909845390 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f56340cce90 0x7f56340a9470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:44.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.838+0000 7f5633200640 1 --2- 172.21.15.6:0/2909845390 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5634070f10 0x7f56340aa5d0 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:44.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.838+0000 7f5633200640 1 -- 172.21.15.6:0/2909845390 >> 172.21.15.6:0/2909845390 conn(0x7f563406b3d0 msgr2=0x7f563415d1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:44.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.838+0000 7f5633200640 1 -- 172.21.15.6:0/2909845390 shutdown_connections 2024-03-04T09:27:44.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:44.838+0000 7f5633200640 1 -- 172.21.15.6:0/2909845390 wait complete. 2024-03-04T09:27:44.855 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-04T09:27:44.974 INFO:tasks.ceph.mds.b.smithi146.stdout:starting mds.b at 2024-03-04T09:27:45.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:45.063+0000 7faaa7200640 1 Processor -- start 2024-03-04T09:27:45.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:45.063+0000 7faaa7200640 1 -- start start 2024-03-04T09:27:45.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:45.065+0000 7faaa7200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7faaa80729d0 0x7faaa80b4000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:45.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:45.065+0000 7faaa7200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faaa8071930 con 0x7faaa80b1b40 2024-03-04T09:27:45.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:45.065+0000 7faaa7200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faaa8071c10 con 0x7faaa80b3c20 2024-03-04T09:27:45.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:45.065+0000 7faaa7200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faaa815bfb0 con 0x7faaa80729d0 2024-03-04T09:27:45.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:45.065+0000 7faaa6800640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7faaa80b1b40 0x7faaa815c220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:45296/0 (socket says 172.21.15.6:45296) 2024-03-04T09:27:45.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:45.065+0000 7faaa6800640 1 -- 172.21.15.6:0/4098657176 learned_addr learned my addr 172.21.15.6:0/4098657176 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:27:45.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:45.065+0000 7faa9fe00640 1 --2- 172.21.15.6:0/4098657176 >> v2:172.21.15.146:3300/0 conn(0x7faaa80729d0 0x7faaa80b4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:45.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:45.066+0000 7faaa5400640 1 -- 172.21.15.6:0/4098657176 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4101866139 0 0) 0x7faaa8071930 con 0x7faaa80b1b40 2024-03-04T09:27:45.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:45.066+0000 7faaa5400640 1 -- 172.21.15.6:0/4098657176 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa8c003520 con 0x7faaa80b1b40 2024-03-04T09:27:45.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:45.066+0000 7faaa5400640 1 -- 172.21.15.6:0/4098657176 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1136119249 0 0) 0x7faa8c003520 con 0x7faaa80b1b40 2024-03-04T09:27:45.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:45.066+0000 7faaa5400640 1 -- 172.21.15.6:0/4098657176 >> v2:172.21.15.146:3300/0 conn(0x7faaa80729d0 msgr2=0x7faaa80b4000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:27:45.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:45.066+0000 7faaa5400640 1 --2- 172.21.15.6:0/4098657176 >> v2:172.21.15.146:3300/0 conn(0x7faaa80729d0 0x7faaa80b4000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:45.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:45.066+0000 7faaa5400640 1 -- 172.21.15.6:0/4098657176 >> v1:172.21.15.146:6789/0 conn(0x7faaa80b3c20 legacy=0x7faaa815b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:45.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:45.066+0000 7faaa5400640 1 -- 172.21.15.6:0/4098657176 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faaa8071ef0 con 0x7faaa80b1b40 2024-03-04T09:27:45.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:45.066+0000 7faaa5400640 1 -- 172.21.15.6:0/4098657176 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7faaa00019f0 con 0x7faaa80b1b40 2024-03-04T09:27:45.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:45.066+0000 7faaa5400640 1 -- 172.21.15.6:0/4098657176 >> v1:172.21.15.6:6789/0 conn(0x7faaa80b1b40 legacy=0x7faaa815c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:45.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:45.066+0000 7faaa5400640 1 --2- 172.21.15.6:0/4098657176 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faa8c0039b0 0x7faa8c003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:45.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:45.066+0000 7faaa5400640 1 --2- 172.21.15.6:0/4098657176 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faa8c0043d0 0x7faa8c0048a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:45.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:45.066+0000 7faaa5400640 1 --2- 172.21.15.6:0/4098657176 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faa8c0050a0 0x7faa8c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:45.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:45.066+0000 7faaa5400640 1 -- 172.21.15.6:0/4098657176 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaa815bfb0 con 0x7faa8c0039b0 2024-03-04T09:27:45.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:45.066+0000 7faaa5400640 1 -- 172.21.15.6:0/4098657176 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa8c00bfe0 con 0x7faa8c0050a0 2024-03-04T09:27:45.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:45.066+0000 7faaa5400640 1 -- 172.21.15.6:0/4098657176 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa8c00c1e0 con 0x7faa8c0043d0 2024-03-04T09:27:45.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:45.066+0000 7faaa5e00640 1 --2- 172.21.15.6:0/4098657176 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faa8c0039b0 0x7faa8c003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:45.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:45.067+0000 7faaa6800640 1 --2- 172.21.15.6:0/4098657176 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faa8c0050a0 0x7faa8c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:45.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:45.067+0000 7faaa5e00640 1 -- 172.21.15.6:0/4098657176 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faa8c0043d0 msgr2=0x7faa8c0048a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:27:45.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:45.067+0000 7faa9fe00640 1 --2- 172.21.15.6:0/4098657176 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faa8c0043d0 0x7faa8c0048a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:45.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:45.067+0000 7faaa5e00640 1 --2- 172.21.15.6:0/4098657176 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faa8c0043d0 0x7faa8c0048a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:45.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:45.067+0000 7faaa5e00640 1 -- 172.21.15.6:0/4098657176 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faa8c0050a0 msgr2=0x7faa8c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:27:45.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:45.067+0000 7faaa5e00640 1 --2- 172.21.15.6:0/4098657176 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faa8c0050a0 0x7faa8c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:45.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:45.067+0000 7faaa5e00640 1 -- 172.21.15.6:0/4098657176 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa8c003520 con 0x7faa8c0039b0 2024-03-04T09:27:45.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:45.067+0000 7faaa6800640 1 --2- 172.21.15.6:0/4098657176 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faa8c0050a0 0x7faa8c00b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:27:45.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:45.067+0000 7faaa6800640 1 -- 172.21.15.6:0/4098657176 reap_dead start 2024-03-04T09:27:45.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:45.068+0000 7faaa5e00640 1 --2- 172.21.15.6:0/4098657176 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faa8c0039b0 0x7faa8c003db0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7faa98001860 tx=0x7faa8c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef118d11964682eb server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:45.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:45.068+0000 7faaa5400640 1 -- 172.21.15.6:0/4098657176 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa98038070 con 0x7faa8c0039b0 2024-03-04T09:27:45.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:45.068+0000 7faaa5400640 1 -- 172.21.15.6:0/4098657176 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa98033410 con 0x7faa8c0039b0 2024-03-04T09:27:45.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:45.068+0000 7faaa5400640 1 -- 172.21.15.6:0/4098657176 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa98001430 con 0x7faa8c0039b0 2024-03-04T09:27:45.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:45.069+0000 7faaa7200640 1 -- 172.21.15.6:0/4098657176 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faa8c0039b0 msgr2=0x7faa8c003db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:45.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:45.069+0000 7faaa7200640 1 --2- 172.21.15.6:0/4098657176 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faa8c0039b0 0x7faa8c003db0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7faa98001860 tx=0x7faa8c005830 comp rx=0 tx=0).stop 2024-03-04T09:27:45.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:45.069+0000 7faaa7200640 1 -- 172.21.15.6:0/4098657176 shutdown_connections 2024-03-04T09:27:45.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:45.069+0000 7faaa7200640 1 --2- 172.21.15.6:0/4098657176 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faa8c0039b0 0x7faa8c003db0 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:45.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:45.069+0000 7faaa7200640 1 -- 172.21.15.6:0/4098657176 >> 172.21.15.6:0/4098657176 conn(0x7faaa806b3a0 msgr2=0x7faaa806b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:45.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:45.069+0000 7faaa7200640 1 -- 172.21.15.6:0/4098657176 shutdown_connections 2024-03-04T09:27:45.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:45.069+0000 7faaa7200640 1 -- 172.21.15.6:0/4098657176 wait complete. 2024-03-04T09:27:45.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:45.070+0000 7faaa7200640 1 Processor -- start 2024-03-04T09:27:45.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:45.071+0000 7faaa7200640 1 -- start start 2024-03-04T09:27:45.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:45.071+0000 7faaa7200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faa8c0039b0 0x7faaa8152fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:45.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:45.071+0000 7faaa7200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faaa80af170 0x7faaa81557d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:45.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:45.072+0000 7faaa7200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faaa80abd40 0x7faaa80ac1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:45.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:45.072+0000 7faaa7200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaa80b4000 con 0x7faaa80af170 2024-03-04T09:27:45.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:45.072+0000 7faaa7200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaa80724b0 con 0x7faaa80abd40 2024-03-04T09:27:45.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:45.072+0000 7faaa7200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faaa80b4180 con 0x7faa8c0039b0 2024-03-04T09:27:45.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:45.072+0000 7faa9fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faaa80af170 0x7faaa81557d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:45.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:45.072+0000 7faa9fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faaa80af170 0x7faaa81557d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36826/0 (socket says 172.21.15.6:36826) 2024-03-04T09:27:45.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:45.072+0000 7faa9fe00640 1 -- 172.21.15.6:0/3804587357 learned_addr learned my addr 172.21.15.6:0/3804587357 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:27:45.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:45.072+0000 7faaa6800640 1 --2- 172.21.15.6:0/3804587357 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faaa80abd40 0x7faaa80ac1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:45.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:45.072+0000 7faaa5e00640 1 --2- 172.21.15.6:0/3804587357 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faa8c0039b0 0x7faaa8152fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:45.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:45.072+0000 7faa9fe00640 1 -- 172.21.15.6:0/3804587357 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faa8c0039b0 msgr2=0x7faaa8152fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:45.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:45.072+0000 7faa9fe00640 1 --2- 172.21.15.6:0/3804587357 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faa8c0039b0 0x7faaa8152fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:45.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:45.072+0000 7faa9fe00640 1 -- 172.21.15.6:0/3804587357 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faaa80abd40 msgr2=0x7faaa80ac1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:45.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:45.072+0000 7faa9fe00640 1 --2- 172.21.15.6:0/3804587357 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faaa80abd40 0x7faaa80ac1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:45.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:45.072+0000 7faa9fe00640 1 -- 172.21.15.6:0/3804587357 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faaa8156020 con 0x7faaa80af170 2024-03-04T09:27:45.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:45.072+0000 7faaa6800640 1 --2- 172.21.15.6:0/3804587357 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faaa80abd40 0x7faaa80ac1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:27:45.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:45.073+0000 7faa9fe00640 1 --2- 172.21.15.6:0/3804587357 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faaa80af170 0x7faaa81557d0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7faa900107c0 tx=0x7faa90010c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:45.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:45.073+0000 7faa9e000640 1 -- 172.21.15.6:0/3804587357 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa9000f040 con 0x7faaa80af170 2024-03-04T09:27:45.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:45.073+0000 7faa9e000640 1 -- 172.21.15.6:0/3804587357 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa9000eb00 con 0x7faaa80af170 2024-03-04T09:27:45.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:45.073+0000 7faa9e000640 1 -- 172.21.15.6:0/3804587357 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa9001c400 con 0x7faaa80af170 2024-03-04T09:27:45.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:45.073+0000 7faaa7200640 1 -- 172.21.15.6:0/3804587357 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faaa8156300 con 0x7faaa80af170 2024-03-04T09:27:45.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:45.073+0000 7faaa7200640 1 -- 172.21.15.6:0/3804587357 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faaa80a97c0 con 0x7faaa80af170 2024-03-04T09:27:45.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:45.074+0000 7faaa7200640 1 -- 172.21.15.6:0/3804587357 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faaa80a95b0 con 0x7faaa80af170 2024-03-04T09:27:45.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:45.075+0000 7faa9e000640 1 -- 172.21.15.6:0/3804587357 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7faa9001c5a0 con 0x7faaa80af170 2024-03-04T09:27:45.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:45.076+0000 7faa9e000640 1 --2- 172.21.15.6:0/3804587357 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7faa8406b500 0x7faa8406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:45.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:45.076+0000 7faa9e000640 1 -- 172.21.15.6:0/3804587357 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5795+0+0 (secure 0 0 0) 0x7faa9008a6e0 con 0x7faaa80af170 2024-03-04T09:27:45.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:45.077+0000 7faaa5e00640 1 --2- 172.21.15.6:0/3804587357 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7faa8406b500 0x7faa8406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:45.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:45.078+0000 7faaa5e00640 1 --2- 172.21.15.6:0/3804587357 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7faa8406b500 0x7faa8406d9c0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7faa98035000 tx=0x7faa98009030 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:45.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:45.081+0000 7faa9e000640 1 -- 172.21.15.6:0/3804587357 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7faa90018030 con 0x7faaa80af170 2024-03-04T09:27:45.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:45.312+0000 7faaa7200640 1 -- 172.21.15.6:0/3804587357 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7faaa804f330 con 0x7faaa80af170 2024-03-04T09:27:45.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:45.837+0000 7faa9e000640 1 -- 172.21.15.6:0/3804587357 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v27) v1 ==== 104+0+0 (secure 0 0 0) 0x7faa90059460 con 0x7faaa80af170 2024-03-04T09:27:45.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:45.841+0000 7faaa7200640 1 -- 172.21.15.6:0/3804587357 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7faa8406b500 msgr2=0x7faa8406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:45.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:45.841+0000 7faaa7200640 1 --2- 172.21.15.6:0/3804587357 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7faa8406b500 0x7faa8406d9c0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7faa98035000 tx=0x7faa98009030 comp rx=0 tx=0).stop 2024-03-04T09:27:45.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:45.841+0000 7faaa7200640 1 -- 172.21.15.6:0/3804587357 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faaa80af170 msgr2=0x7faaa81557d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:45.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:45.841+0000 7faaa7200640 1 --2- 172.21.15.6:0/3804587357 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faaa80af170 0x7faaa81557d0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7faa900107c0 tx=0x7faa90010c80 comp rx=0 tx=0).stop 2024-03-04T09:27:45.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:45.841+0000 7faaa7200640 1 -- 172.21.15.6:0/3804587357 shutdown_connections 2024-03-04T09:27:45.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:45.841+0000 7faaa7200640 1 --2- 172.21.15.6:0/3804587357 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faaa80abd40 0x7faaa80ac1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:45.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:45.841+0000 7faaa7200640 1 --2- 172.21.15.6:0/3804587357 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faaa80af170 0x7faaa81557d0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:45.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:45.841+0000 7faaa7200640 1 --2- 172.21.15.6:0/3804587357 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7faa8406b500 0x7faa8406d9c0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:45.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:45.841+0000 7faaa7200640 1 --2- 172.21.15.6:0/3804587357 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faa8c0039b0 0x7faaa8152fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:45.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:45.841+0000 7faaa7200640 1 -- 172.21.15.6:0/3804587357 >> 172.21.15.6:0/3804587357 conn(0x7faaa806b3a0 msgr2=0x7faaa806b780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:45.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:45.841+0000 7faaa7200640 1 -- 172.21.15.6:0/3804587357 shutdown_connections 2024-03-04T09:27:45.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:45.841+0000 7faaa7200640 1 -- 172.21.15.6:0/3804587357 wait complete. 2024-03-04T09:27:45.859 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-04T09:27:46.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:46.089+0000 7f58bb200640 1 Processor -- start 2024-03-04T09:27:46.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:46.090+0000 7f58bb200640 1 -- start start 2024-03-04T09:27:46.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:46.091+0000 7f58bb200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f58bc1618c0 0x7f58bc0d06f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:46.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:46.091+0000 7f58bb200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58bc0b3630 con 0x7f58bc0d4410 2024-03-04T09:27:46.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:46.091+0000 7f58bb200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58bc0d49b0 con 0x7f58bc0b08f0 2024-03-04T09:27:46.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:46.091+0000 7f58bb200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58bc0b38f0 con 0x7f58bc1618c0 2024-03-04T09:27:46.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:46.091+0000 7f58b9e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f58bc1618c0 0x7f58bc0d06f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:46.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:46.091+0000 7f58ba800640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f58bc0d4410 0x7f58bc0b4800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:45306/0 (socket says 172.21.15.6:45306) 2024-03-04T09:27:46.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:46.091+0000 7f58b9e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f58bc1618c0 0x7f58bc0d06f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36838/0 (socket says 172.21.15.6:36838) 2024-03-04T09:27:46.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:46.091+0000 7f58ba800640 1 -- 172.21.15.6:0/731320225 learned_addr learned my addr 172.21.15.6:0/731320225 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:27:46.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:46.092+0000 7f58b9e00640 1 -- 172.21.15.6:0/731320225 >> v1:172.21.15.146:6789/0 conn(0x7f58bc0b08f0 legacy=0x7f58bc0b40f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:27:46.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:46.092+0000 7f58b9e00640 1 -- 172.21.15.6:0/731320225 >> v1:172.21.15.6:6789/0 conn(0x7f58bc0d4410 legacy=0x7f58bc0b4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:46.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:46.092+0000 7f58b9e00640 1 -- 172.21.15.6:0/731320225 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58bc0b3bd0 con 0x7f58bc1618c0 2024-03-04T09:27:46.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:46.092+0000 7f58b8a00640 1 -- 172.21.15.6:0/731320225 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2053810059 0 0) 0x7f58bc0b3630 con 0x7f58bc0d4410 2024-03-04T09:27:46.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:46.093+0000 7f58b9e00640 1 --2- 172.21.15.6:0/731320225 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f58bc1618c0 0x7f58bc0d06f0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f58ac009d90 tx=0x7f58ac02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2f34d7b9afd4c89 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:46.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:46.093+0000 7f58b8a00640 1 -- 172.21.15.6:0/731320225 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58ac039070 con 0x7f58bc1618c0 2024-03-04T09:27:46.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:46.093+0000 7f58b8a00640 1 -- 172.21.15.6:0/731320225 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58ac02cc60 con 0x7f58bc1618c0 2024-03-04T09:27:46.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:46.093+0000 7f58b8a00640 1 -- 172.21.15.6:0/731320225 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58ac034660 con 0x7f58bc1618c0 2024-03-04T09:27:46.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:46.094+0000 7f58bb200640 1 -- 172.21.15.6:0/731320225 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f58bc1618c0 msgr2=0x7f58bc0d06f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:46.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:46.094+0000 7f58bb200640 1 --2- 172.21.15.6:0/731320225 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f58bc1618c0 0x7f58bc0d06f0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f58ac009d90 tx=0x7f58ac02beb0 comp rx=0 tx=0).stop 2024-03-04T09:27:46.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:46.094+0000 7f58bb200640 1 -- 172.21.15.6:0/731320225 shutdown_connections 2024-03-04T09:27:46.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:46.094+0000 7f58bb200640 1 --2- 172.21.15.6:0/731320225 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f58bc1618c0 0x7f58bc0d06f0 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:46.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:46.094+0000 7f58bb200640 1 -- 172.21.15.6:0/731320225 >> 172.21.15.6:0/731320225 conn(0x7f58bc09d000 msgr2=0x7f58bc0d3d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:46.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:46.094+0000 7f58bb200640 1 -- 172.21.15.6:0/731320225 shutdown_connections 2024-03-04T09:27:46.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:46.094+0000 7f58bb200640 1 -- 172.21.15.6:0/731320225 wait complete. 2024-03-04T09:27:46.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:46.096+0000 7f58bb200640 1 Processor -- start 2024-03-04T09:27:46.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:46.096+0000 7f58bb200640 1 -- start start 2024-03-04T09:27:46.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:46.097+0000 7f58bb200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f58bc0b08f0 0x7f58bc14eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:46.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:46.097+0000 7f58bb200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f58bc0d4410 0x7f58bc0c9260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:46.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:46.097+0000 7f58bb200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f58bc1618c0 0x7f58bc14b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:46.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:46.097+0000 7f58bb200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58bc0cf900 con 0x7f58bc0d4410 2024-03-04T09:27:46.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:46.097+0000 7f58bb200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58bc0b5440 con 0x7f58bc1618c0 2024-03-04T09:27:46.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:46.097+0000 7f58bb200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58bc0cfa80 con 0x7f58bc0b08f0 2024-03-04T09:27:46.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:46.097+0000 7f58b9400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f58bc0d4410 0x7f58bc0c9260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:46.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:46.097+0000 7f58b9400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f58bc0d4410 0x7f58bc0c9260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36846/0 (socket says 172.21.15.6:36846) 2024-03-04T09:27:46.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:46.097+0000 7f58b9400640 1 -- 172.21.15.6:0/3661969134 learned_addr learned my addr 172.21.15.6:0/3661969134 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:27:46.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:46.098+0000 7f58ba800640 1 --2- 172.21.15.6:0/3661969134 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f58bc1618c0 0x7f58bc14b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:46.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:46.098+0000 7f58b9e00640 1 --2- 172.21.15.6:0/3661969134 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f58bc0b08f0 0x7f58bc14eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:46.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:46.098+0000 7f58b9400640 1 -- 172.21.15.6:0/3661969134 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f58bc0b08f0 msgr2=0x7f58bc14eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:46.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:46.098+0000 7f58b9400640 1 --2- 172.21.15.6:0/3661969134 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f58bc0b08f0 0x7f58bc14eb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:46.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:46.098+0000 7f58b9400640 1 -- 172.21.15.6:0/3661969134 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f58bc1618c0 msgr2=0x7f58bc14b6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:46.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:46.098+0000 7f58b9400640 1 --2- 172.21.15.6:0/3661969134 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f58bc1618c0 0x7f58bc14b6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:46.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:46.098+0000 7f58b9400640 1 -- 172.21.15.6:0/3661969134 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58bc0c97a0 con 0x7f58bc0d4410 2024-03-04T09:27:46.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:46.098+0000 7f58b9400640 1 --2- 172.21.15.6:0/3661969134 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f58bc0d4410 0x7f58bc0c9260 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f58a400c400 tx=0x7f58a400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:46.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:46.098+0000 7f58b2000640 1 -- 172.21.15.6:0/3661969134 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58a4011790 con 0x7f58bc0d4410 2024-03-04T09:27:46.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:46.098+0000 7f58b2000640 1 -- 172.21.15.6:0/3661969134 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58a4017070 con 0x7f58bc0d4410 2024-03-04T09:27:46.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:46.098+0000 7f58b2000640 1 -- 172.21.15.6:0/3661969134 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58a4011bb0 con 0x7f58bc0d4410 2024-03-04T09:27:46.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:46.098+0000 7f58bb200640 1 -- 172.21.15.6:0/3661969134 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58bc14bc30 con 0x7f58bc0d4410 2024-03-04T09:27:46.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:46.099+0000 7f58bb200640 1 -- 172.21.15.6:0/3661969134 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58bc0c9d20 con 0x7f58bc0d4410 2024-03-04T09:27:46.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:46.100+0000 7f58bb200640 1 -- 172.21.15.6:0/3661969134 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58bc04f330 con 0x7f58bc0d4410 2024-03-04T09:27:46.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:46.101+0000 7f58b2000640 1 -- 172.21.15.6:0/3661969134 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f58a401c460 con 0x7f58bc0d4410 2024-03-04T09:27:46.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:46.101+0000 7f58b2000640 1 --2- 172.21.15.6:0/3661969134 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f589806b4d0 0x7f589806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:46.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:46.102+0000 7f58b2000640 1 -- 172.21.15.6:0/3661969134 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f58a408a150 con 0x7f58bc0d4410 2024-03-04T09:27:46.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:46.102+0000 7f58b9e00640 1 --2- 172.21.15.6:0/3661969134 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f589806b4d0 0x7f589806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:46.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:46.103+0000 7f58b9e00640 1 --2- 172.21.15.6:0/3661969134 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f589806b4d0 0x7f589806d990 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f58ac036000 tx=0x7f58ac030000 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:46.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:46.106+0000 7f58b2000640 1 -- 172.21.15.6:0/3661969134 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f58a4058ed0 con 0x7f58bc0d4410 2024-03-04T09:27:46.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:46.311+0000 7f58bb200640 1 -- 172.21.15.6:0/3661969134 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f58bc0b5f90 con 0x7f58bc0d4410 2024-03-04T09:27:47.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:47.266+0000 7f58b2000640 1 -- 172.21.15.6:0/3661969134 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v46) v1 ==== 181+0+0 (secure 0 0 0) 0x7f58a40587d0 con 0x7f58bc0d4410 2024-03-04T09:27:47.263 INFO:teuthology.orchestra.run.smithi006.stderr:pool 'cephfs_metadata' removed 2024-03-04T09:27:47.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:47.271+0000 7f58bb200640 1 -- 172.21.15.6:0/3661969134 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f589806b4d0 msgr2=0x7f589806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:47.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:47.271+0000 7f58bb200640 1 --2- 172.21.15.6:0/3661969134 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f589806b4d0 0x7f589806d990 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f58ac036000 tx=0x7f58ac030000 comp rx=0 tx=0).stop 2024-03-04T09:27:47.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:47.271+0000 7f58bb200640 1 -- 172.21.15.6:0/3661969134 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f58bc0d4410 msgr2=0x7f58bc0c9260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:47.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:47.271+0000 7f58bb200640 1 --2- 172.21.15.6:0/3661969134 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f58bc0d4410 0x7f58bc0c9260 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f58a400c400 tx=0x7f58a400c8c0 comp rx=0 tx=0).stop 2024-03-04T09:27:47.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:47.272+0000 7f58bb200640 1 -- 172.21.15.6:0/3661969134 shutdown_connections 2024-03-04T09:27:47.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:47.272+0000 7f58bb200640 1 --2- 172.21.15.6:0/3661969134 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f58bc1618c0 0x7f58bc14b6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:47.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:47.272+0000 7f58bb200640 1 --2- 172.21.15.6:0/3661969134 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f58bc0d4410 0x7f58bc0c9260 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:47.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:47.272+0000 7f58bb200640 1 --2- 172.21.15.6:0/3661969134 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f589806b4d0 0x7f589806d990 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:47.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:47.272+0000 7f58bb200640 1 --2- 172.21.15.6:0/3661969134 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f58bc0b08f0 0x7f58bc14eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:47.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:47.272+0000 7f58bb200640 1 -- 172.21.15.6:0/3661969134 >> 172.21.15.6:0/3661969134 conn(0x7f58bc09d000 msgr2=0x7f58bc06cca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:47.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:47.272+0000 7f58bb200640 1 -- 172.21.15.6:0/3661969134 shutdown_connections 2024-03-04T09:27:47.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:47.272+0000 7f58bb200640 1 -- 172.21.15.6:0/3661969134 wait complete. 2024-03-04T09:27:47.292 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-04T09:27:47.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:47.506+0000 7ff51ea00640 1 Processor -- start 2024-03-04T09:27:47.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:47.506+0000 7ff51ea00640 1 -- start start 2024-03-04T09:27:47.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:47.507+0000 7ff51ea00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff5180cddc0 0x7ff5181669b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:47.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:47.508+0000 7ff51ea00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff518070a80 con 0x7ff51806e200 2024-03-04T09:27:47.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:47.508+0000 7ff51ea00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff518070d60 con 0x7ff5181622f0 2024-03-04T09:27:47.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:47.508+0000 7ff51ea00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff518071040 con 0x7ff5180cddc0 2024-03-04T09:27:47.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:47.508+0000 7ff51e000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff5180cddc0 0x7ff5181669b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:47.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:47.508+0000 7ff51e000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff5180cddc0 0x7ff5181669b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36852/0 (socket says 172.21.15.6:36852) 2024-03-04T09:27:47.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:47.508+0000 7ff51e000640 1 -- 172.21.15.6:0/3550447899 learned_addr learned my addr 172.21.15.6:0/3550447899 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:27:47.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:47.508+0000 7ff51cc00640 1 --1- 172.21.15.6:0/3550447899 >> v1:172.21.15.6:6789/0 conn(0x7ff51806e200 0x7ff5181662a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:45312/0 (socket says 172.21.15.6:45312) 2024-03-04T09:27:47.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:47.509+0000 7ff51e000640 1 -- 172.21.15.6:0/3550447899 >> v1:172.21.15.146:6789/0 conn(0x7ff5181622f0 legacy=0x7ff51806daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:27:47.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:47.509+0000 7ff51e000640 1 -- 172.21.15.6:0/3550447899 >> v1:172.21.15.6:6789/0 conn(0x7ff51806e200 legacy=0x7ff5181662a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:47.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:47.509+0000 7ff51e000640 1 -- 172.21.15.6:0/3550447899 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff518071320 con 0x7ff5180cddc0 2024-03-04T09:27:47.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:47.510+0000 7ff51e000640 1 --2- 172.21.15.6:0/3550447899 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff5180cddc0 0x7ff5181669b0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7ff510009e50 tx=0x7ff51002bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a686d48921c0d32 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:47.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:47.510+0000 7ff517e00640 1 -- 172.21.15.6:0/3550447899 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff51002cb10 con 0x7ff5180cddc0 2024-03-04T09:27:47.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:47.510+0000 7ff517e00640 1 -- 172.21.15.6:0/3550447899 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff51002ccb0 con 0x7ff5180cddc0 2024-03-04T09:27:47.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:47.510+0000 7ff517e00640 1 -- 172.21.15.6:0/3550447899 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff51002ce60 con 0x7ff5180cddc0 2024-03-04T09:27:47.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:47.511+0000 7ff51ea00640 1 -- 172.21.15.6:0/3550447899 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff5180cddc0 msgr2=0x7ff5181669b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:47.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:47.511+0000 7ff51ea00640 1 --2- 172.21.15.6:0/3550447899 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff5180cddc0 0x7ff5181669b0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7ff510009e50 tx=0x7ff51002bf70 comp rx=0 tx=0).stop 2024-03-04T09:27:47.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:47.511+0000 7ff51ea00640 1 -- 172.21.15.6:0/3550447899 shutdown_connections 2024-03-04T09:27:47.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:47.511+0000 7ff51ea00640 1 --2- 172.21.15.6:0/3550447899 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff5180cddc0 0x7ff5181669b0 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:47.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:47.511+0000 7ff51ea00640 1 -- 172.21.15.6:0/3550447899 >> 172.21.15.6:0/3550447899 conn(0x7ff5180b17c0 msgr2=0x7ff5180b1ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:47.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:47.511+0000 7ff51ea00640 1 -- 172.21.15.6:0/3550447899 shutdown_connections 2024-03-04T09:27:47.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:47.511+0000 7ff51ea00640 1 -- 172.21.15.6:0/3550447899 wait complete. 2024-03-04T09:27:47.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:47.513+0000 7ff51ea00640 1 Processor -- start 2024-03-04T09:27:47.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:47.513+0000 7ff51ea00640 1 -- start start 2024-03-04T09:27:47.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:47.514+0000 7ff51ea00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff51806e200 0x7ff51814aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:47.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:47.514+0000 7ff51ea00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff5180cddc0 0x7ff5181581d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:47.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:47.514+0000 7ff51ea00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff5181622f0 0x7ff518158710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:47.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:47.515+0000 7ff51ea00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff518162850 con 0x7ff51806e200 2024-03-04T09:27:47.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:47.515+0000 7ff51ea00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5181626d0 con 0x7ff5180cddc0 2024-03-04T09:27:47.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:47.515+0000 7ff51ea00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff51816d2a0 con 0x7ff5181622f0 2024-03-04T09:27:47.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:47.515+0000 7ff51d600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff51806e200 0x7ff51814aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:47.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:47.515+0000 7ff51d600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff51806e200 0x7ff51814aa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36856/0 (socket says 172.21.15.6:36856) 2024-03-04T09:27:47.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:47.515+0000 7ff51d600640 1 -- 172.21.15.6:0/845588500 learned_addr learned my addr 172.21.15.6:0/845588500 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:27:47.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:47.515+0000 7ff51cc00640 1 --2- 172.21.15.6:0/845588500 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff5180cddc0 0x7ff5181581d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:47.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:47.515+0000 7ff51d600640 1 -- 172.21.15.6:0/845588500 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff5181622f0 msgr2=0x7ff518158710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:47.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:47.515+0000 7ff51d600640 1 --2- 172.21.15.6:0/845588500 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff5181622f0 0x7ff518158710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:47.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:47.515+0000 7ff51d600640 1 -- 172.21.15.6:0/845588500 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff5180cddc0 msgr2=0x7ff5181581d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:47.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:47.515+0000 7ff51d600640 1 --2- 172.21.15.6:0/845588500 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff5180cddc0 0x7ff5181581d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:47.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:47.515+0000 7ff51d600640 1 -- 172.21.15.6:0/845588500 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff518158d40 con 0x7ff51806e200 2024-03-04T09:27:47.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:47.515+0000 7ff51cc00640 1 --2- 172.21.15.6:0/845588500 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff5180cddc0 0x7ff5181581d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:27:47.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:47.516+0000 7ff51d600640 1 --2- 172.21.15.6:0/845588500 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff51806e200 0x7ff51814aa30 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7ff50c00b420 tx=0x7ff50c00b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:47.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:47.516+0000 7ff515600640 1 -- 172.21.15.6:0/845588500 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff50c011830 con 0x7ff51806e200 2024-03-04T09:27:47.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:47.516+0000 7ff515600640 1 -- 172.21.15.6:0/845588500 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff50c016070 con 0x7ff51806e200 2024-03-04T09:27:47.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:47.516+0000 7ff51ea00640 1 -- 172.21.15.6:0/845588500 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff518159020 con 0x7ff51806e200 2024-03-04T09:27:47.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:47.516+0000 7ff515600640 1 -- 172.21.15.6:0/845588500 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff50c011c10 con 0x7ff51806e200 2024-03-04T09:27:47.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:47.516+0000 7ff51ea00640 1 -- 172.21.15.6:0/845588500 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5181595e0 con 0x7ff51806e200 2024-03-04T09:27:47.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:47.517+0000 7ff51ea00640 1 -- 172.21.15.6:0/845588500 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff51804f330 con 0x7ff51806e200 2024-03-04T09:27:47.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:47.518+0000 7ff515600640 1 -- 172.21.15.6:0/845588500 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff50c01b460 con 0x7ff51806e200 2024-03-04T09:27:47.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:47.519+0000 7ff515600640 1 --2- 172.21.15.6:0/845588500 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff4f806b4d0 0x7ff4f806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:47.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:47.519+0000 7ff515600640 1 -- 172.21.15.6:0/845588500 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5357+0+0 (secure 0 0 0) 0x7ff50c089b80 con 0x7ff51806e200 2024-03-04T09:27:47.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:47.520+0000 7ff51cc00640 1 --2- 172.21.15.6:0/845588500 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff4f806b4d0 0x7ff4f806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:47.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:47.521+0000 7ff51cc00640 1 --2- 172.21.15.6:0/845588500 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff4f806b4d0 0x7ff4f806d990 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7ff51815f5e0 tx=0x7ff504009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:47.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:47.524+0000 7ff515600640 1 -- 172.21.15.6:0/845588500 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff50c020020 con 0x7ff51806e200 2024-03-04T09:27:47.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:47.722+0000 7ff51ea00640 1 -- 172.21.15.6:0/845588500 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7ff5180b0e50 con 0x7ff51806e200 2024-03-04T09:27:48.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.276+0000 7ff515600640 1 -- 172.21.15.6:0/845588500 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v47) v1 ==== 169+0+0 (secure 0 0 0) 0x7ff50c058b50 con 0x7ff51806e200 2024-03-04T09:27:48.273 INFO:teuthology.orchestra.run.smithi006.stderr:pool 'cephfs_data' removed 2024-03-04T09:27:48.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.280+0000 7ff51ea00640 1 -- 172.21.15.6:0/845588500 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff4f806b4d0 msgr2=0x7ff4f806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:48.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.280+0000 7ff51ea00640 1 --2- 172.21.15.6:0/845588500 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff4f806b4d0 0x7ff4f806d990 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7ff51815f5e0 tx=0x7ff504009040 comp rx=0 tx=0).stop 2024-03-04T09:27:48.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.280+0000 7ff51ea00640 1 -- 172.21.15.6:0/845588500 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff51806e200 msgr2=0x7ff51814aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:48.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.280+0000 7ff51ea00640 1 --2- 172.21.15.6:0/845588500 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff51806e200 0x7ff51814aa30 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7ff50c00b420 tx=0x7ff50c00b8e0 comp rx=0 tx=0).stop 2024-03-04T09:27:48.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.280+0000 7ff51ea00640 1 -- 172.21.15.6:0/845588500 shutdown_connections 2024-03-04T09:27:48.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.280+0000 7ff51ea00640 1 --2- 172.21.15.6:0/845588500 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff4f806b4d0 0x7ff4f806d990 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:48.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.280+0000 7ff51ea00640 1 --2- 172.21.15.6:0/845588500 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff5181622f0 0x7ff518158710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:48.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.280+0000 7ff51ea00640 1 --2- 172.21.15.6:0/845588500 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff5180cddc0 0x7ff5181581d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:48.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.280+0000 7ff51ea00640 1 --2- 172.21.15.6:0/845588500 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff51806e200 0x7ff51814aa30 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:48.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.280+0000 7ff51ea00640 1 -- 172.21.15.6:0/845588500 >> 172.21.15.6:0/845588500 conn(0x7ff5180b17c0 msgr2=0x7ff5180ce210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:48.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.281+0000 7ff51ea00640 1 -- 172.21.15.6:0/845588500 shutdown_connections 2024-03-04T09:27:48.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.281+0000 7ff51ea00640 1 -- 172.21.15.6:0/845588500 wait complete. 2024-03-04T09:27:48.303 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds mds_bal_fragment_size_max 2024-03-04T09:27:48.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.515+0000 7fab47200640 1 Processor -- start 2024-03-04T09:27:48.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.516+0000 7fab47200640 1 -- start start 2024-03-04T09:27:48.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.517+0000 7fab47200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fab40145430 0x7fab4015f810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:48.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.517+0000 7fab47200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab4015fe40 con 0x7fab400b2250 2024-03-04T09:27:48.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.517+0000 7fab47200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab40160120 con 0x7fab4015cc90 2024-03-04T09:27:48.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.517+0000 7fab47200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab40160400 con 0x7fab40145430 2024-03-04T09:27:48.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.517+0000 7fab45e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fab40145430 0x7fab4015f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:48.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.517+0000 7fab46800640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fab400b2250 0x7fab400b2fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:45318/0 (socket says 172.21.15.6:45318) 2024-03-04T09:27:48.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.517+0000 7fab46800640 1 -- 172.21.15.6:0/399917489 learned_addr learned my addr 172.21.15.6:0/399917489 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:27:48.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.518+0000 7fab45e00640 1 -- 172.21.15.6:0/399917489 >> v1:172.21.15.146:6789/0 conn(0x7fab4015cc90 legacy=0x7fab400b1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:27:48.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.518+0000 7fab45e00640 1 -- 172.21.15.6:0/399917489 >> v1:172.21.15.6:6789/0 conn(0x7fab400b2250 legacy=0x7fab400b2fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:48.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.518+0000 7fab45e00640 1 -- 172.21.15.6:0/399917489 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab401606e0 con 0x7fab40145430 2024-03-04T09:27:48.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.519+0000 7fab45e00640 1 --2- 172.21.15.6:0/399917489 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fab40145430 0x7fab4015f810 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fab38009d90 tx=0x7fab3802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d782d3a9be249ab1 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:48.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.519+0000 7fab44a00640 1 -- 172.21.15.6:0/399917489 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab38039070 con 0x7fab40145430 2024-03-04T09:27:48.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.519+0000 7fab44a00640 1 -- 172.21.15.6:0/399917489 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab3802cc60 con 0x7fab40145430 2024-03-04T09:27:48.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.519+0000 7fab44a00640 1 -- 172.21.15.6:0/399917489 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab38034660 con 0x7fab40145430 2024-03-04T09:27:48.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.520+0000 7fab47200640 1 -- 172.21.15.6:0/399917489 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fab40145430 msgr2=0x7fab4015f810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:48.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.520+0000 7fab47200640 1 --2- 172.21.15.6:0/399917489 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fab40145430 0x7fab4015f810 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fab38009d90 tx=0x7fab3802beb0 comp rx=0 tx=0).stop 2024-03-04T09:27:48.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.520+0000 7fab47200640 1 -- 172.21.15.6:0/399917489 shutdown_connections 2024-03-04T09:27:48.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.520+0000 7fab47200640 1 --2- 172.21.15.6:0/399917489 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fab40145430 0x7fab4015f810 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:48.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.520+0000 7fab47200640 1 -- 172.21.15.6:0/399917489 >> 172.21.15.6:0/399917489 conn(0x7fab4006b3a0 msgr2=0x7fab4006b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:48.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.520+0000 7fab47200640 1 -- 172.21.15.6:0/399917489 shutdown_connections 2024-03-04T09:27:48.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.520+0000 7fab47200640 1 -- 172.21.15.6:0/399917489 wait complete. 2024-03-04T09:27:48.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.522+0000 7fab47200640 1 Processor -- start 2024-03-04T09:27:48.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.522+0000 7fab47200640 1 -- start start 2024-03-04T09:27:48.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.523+0000 7fab47200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fab400b2250 0x7fab401f1bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:48.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.523+0000 7fab47200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fab40145430 0x7fab401595a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:48.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.523+0000 7fab47200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fab4015cc90 0x7fab40137f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:48.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.523+0000 7fab47200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab40145990 con 0x7fab4015cc90 2024-03-04T09:27:48.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.523+0000 7fab47200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab40145810 con 0x7fab40145430 2024-03-04T09:27:48.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.523+0000 7fab47200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab400cde00 con 0x7fab400b2250 2024-03-04T09:27:48.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.523+0000 7fab46800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fab4015cc90 0x7fab40137f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:48.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.523+0000 7fab46800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fab4015cc90 0x7fab40137f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36884/0 (socket says 172.21.15.6:36884) 2024-03-04T09:27:48.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.523+0000 7fab46800640 1 -- 172.21.15.6:0/1274835895 learned_addr learned my addr 172.21.15.6:0/1274835895 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:27:48.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.523+0000 7fab45e00640 1 --2- 172.21.15.6:0/1274835895 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fab400b2250 0x7fab401f1bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:48.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.523+0000 7fab46800640 1 -- 172.21.15.6:0/1274835895 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fab400b2250 msgr2=0x7fab401f1bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:48.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.523+0000 7fab46800640 1 --2- 172.21.15.6:0/1274835895 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fab400b2250 0x7fab401f1bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:48.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.523+0000 7fab46800640 1 -- 172.21.15.6:0/1274835895 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fab40145430 msgr2=0x7fab401595a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:48.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.523+0000 7fab46800640 1 --2- 172.21.15.6:0/1274835895 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fab40145430 0x7fab401595a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:48.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.524+0000 7fab46800640 1 -- 172.21.15.6:0/1274835895 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab40136e70 con 0x7fab4015cc90 2024-03-04T09:27:48.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.524+0000 7fab45e00640 1 --2- 172.21.15.6:0/1274835895 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fab400b2250 0x7fab401f1bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:27:48.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.524+0000 7fab46800640 1 --2- 172.21.15.6:0/1274835895 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fab4015cc90 0x7fab40137f10 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fab3c00c430 tx=0x7fab3c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:48.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.524+0000 7fab36000640 1 -- 172.21.15.6:0/1274835895 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab3c017070 con 0x7fab4015cc90 2024-03-04T09:27:48.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.524+0000 7fab36000640 1 -- 172.21.15.6:0/1274835895 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab3c005ce0 con 0x7fab4015cc90 2024-03-04T09:27:48.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.524+0000 7fab36000640 1 -- 172.21.15.6:0/1274835895 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab3c0072d0 con 0x7fab4015cc90 2024-03-04T09:27:48.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.524+0000 7fab47200640 1 -- 172.21.15.6:0/1274835895 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab40137150 con 0x7fab4015cc90 2024-03-04T09:27:48.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.524+0000 7fab47200640 1 -- 172.21.15.6:0/1274835895 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab40137710 con 0x7fab4015cc90 2024-03-04T09:27:48.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.525+0000 7fab47200640 1 -- 172.21.15.6:0/1274835895 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab4004f330 con 0x7fab4015cc90 2024-03-04T09:27:48.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.526+0000 7fab36000640 1 -- 172.21.15.6:0/1274835895 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fab3c01a070 con 0x7fab4015cc90 2024-03-04T09:27:48.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.527+0000 7fab36000640 1 --2- 172.21.15.6:0/1274835895 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fab2406b4d0 0x7fab2406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:48.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.527+0000 7fab36000640 1 -- 172.21.15.6:0/1274835895 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fab3c089200 con 0x7fab4015cc90 2024-03-04T09:27:48.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.528+0000 7fab45e00640 1 --2- 172.21.15.6:0/1274835895 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fab2406b4d0 0x7fab2406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:48.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.529+0000 7fab45e00640 1 --2- 172.21.15.6:0/1274835895 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fab2406b4d0 0x7fab2406d990 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fab38036000 tx=0x7fab38030000 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:48.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.532+0000 7fab36000640 1 -- 172.21.15.6:0/1274835895 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fab3c057c90 con 0x7fab4015cc90 2024-03-04T09:27:48.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.736+0000 7fab47200640 1 -- 172.21.15.6:0/1274835895 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "config rm", "who": "mds", "name": "mds_bal_fragment_size_max"} v 0) v1 -- 0x7fab4006e240 con 0x7fab4015cc90 2024-03-04T09:27:48.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.742+0000 7fab36000640 1 -- 172.21.15.6:0/1274835895 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds", "name": "mds_bal_fragment_size_max"}]=0 v14) v1 ==== 108+0+0 (secure 0 0 0) 0x7fab3c05bd30 con 0x7fab4015cc90 2024-03-04T09:27:48.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.745+0000 7fab47200640 1 -- 172.21.15.6:0/1274835895 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fab2406b4d0 msgr2=0x7fab2406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:48.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.745+0000 7fab47200640 1 --2- 172.21.15.6:0/1274835895 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fab2406b4d0 0x7fab2406d990 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fab38036000 tx=0x7fab38030000 comp rx=0 tx=0).stop 2024-03-04T09:27:48.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.745+0000 7fab47200640 1 -- 172.21.15.6:0/1274835895 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fab4015cc90 msgr2=0x7fab40137f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:48.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.745+0000 7fab47200640 1 --2- 172.21.15.6:0/1274835895 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fab4015cc90 0x7fab40137f10 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fab3c00c430 tx=0x7fab3c00c8f0 comp rx=0 tx=0).stop 2024-03-04T09:27:48.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.745+0000 7fab47200640 1 -- 172.21.15.6:0/1274835895 shutdown_connections 2024-03-04T09:27:48.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.745+0000 7fab47200640 1 --2- 172.21.15.6:0/1274835895 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fab4015cc90 0x7fab40137f10 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:48.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.745+0000 7fab47200640 1 --2- 172.21.15.6:0/1274835895 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fab40145430 0x7fab401595a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:48.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.745+0000 7fab47200640 1 --2- 172.21.15.6:0/1274835895 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fab2406b4d0 0x7fab2406d990 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:48.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.745+0000 7fab47200640 1 --2- 172.21.15.6:0/1274835895 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fab400b2250 0x7fab401f1bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:48.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.745+0000 7fab47200640 1 -- 172.21.15.6:0/1274835895 >> 172.21.15.6:0/1274835895 conn(0x7fab4006b3a0 msgr2=0x7fab400ce100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:48.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.745+0000 7fab47200640 1 -- 172.21.15.6:0/1274835895 shutdown_connections 2024-03-04T09:27:48.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.745+0000 7fab47200640 1 -- 172.21.15.6:0/1274835895 wait complete. 2024-03-04T09:27:48.762 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-04T09:27:48.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.986+0000 7ff23f600640 1 Processor -- start 2024-03-04T09:27:48.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.987+0000 7ff23f600640 1 -- start start 2024-03-04T09:27:48.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.988+0000 7ff23f600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff238161930 0x7ff2380d24c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:48.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.988+0000 7ff23f600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff2380cfa90 0x7ff2380d2a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:48.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.988+0000 7ff23f600640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2380d20d0 con 0x7ff2380d0f80 2024-03-04T09:27:48.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.988+0000 7ff23f600640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2380d4df0 con 0x7ff238161930 2024-03-04T09:27:48.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.988+0000 7ff23f600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2380d1df0 con 0x7ff2380cfa90 2024-03-04T09:27:48.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.988+0000 7ff23cc00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff238161930 0x7ff2380d24c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:48.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.989+0000 7ff23cc00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff238161930 0x7ff2380d24c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42236/0 (socket says 172.21.15.6:42236) 2024-03-04T09:27:48.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.989+0000 7ff23cc00640 1 -- 172.21.15.6:0/2379181320 learned_addr learned my addr 172.21.15.6:0/2379181320 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:27:48.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.989+0000 7ff23d600640 1 --2- 172.21.15.6:0/2379181320 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff2380cfa90 0x7ff2380d2a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:48.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.989+0000 7ff23cc00640 1 -- 172.21.15.6:0/2379181320 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff2380cfa90 msgr2=0x7ff2380d2a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:27:48.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.989+0000 7ff23cc00640 1 --2- 172.21.15.6:0/2379181320 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff2380cfa90 0x7ff2380d2a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:48.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.989+0000 7ff23cc00640 1 -- 172.21.15.6:0/2379181320 >> v1:172.21.15.146:6789/0 conn(0x7ff2380d0f80 legacy=0x7ff2380c7d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:27:48.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.989+0000 7ff23cc00640 1 -- 172.21.15.6:0/2379181320 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2380d4b10 con 0x7ff238161930 2024-03-04T09:27:48.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.991+0000 7ff23cc00640 1 --2- 172.21.15.6:0/2379181320 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff238161930 0x7ff2380d24c0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7ff220009870 tx=0x7ff22002b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba01a5889c31c097 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:48.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.991+0000 7ff237400640 1 -- 172.21.15.6:0/2379181320 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2200317c0 con 0x7ff238161930 2024-03-04T09:27:48.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.991+0000 7ff237400640 1 -- 172.21.15.6:0/2379181320 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff220037070 con 0x7ff238161930 2024-03-04T09:27:48.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.991+0000 7ff237400640 1 -- 172.21.15.6:0/2379181320 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff220031b80 con 0x7ff238161930 2024-03-04T09:27:48.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.992+0000 7ff23f600640 1 -- 172.21.15.6:0/2379181320 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff238161930 msgr2=0x7ff2380d24c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:48.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.992+0000 7ff23f600640 1 --2- 172.21.15.6:0/2379181320 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff238161930 0x7ff2380d24c0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7ff220009870 tx=0x7ff22002b860 comp rx=0 tx=0).stop 2024-03-04T09:27:48.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.992+0000 7ff23f600640 1 -- 172.21.15.6:0/2379181320 shutdown_connections 2024-03-04T09:27:48.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.992+0000 7ff23f600640 1 --2- 172.21.15.6:0/2379181320 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff2380cfa90 0x7ff2380d2a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:48.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.992+0000 7ff23f600640 1 --2- 172.21.15.6:0/2379181320 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff238161930 0x7ff2380d24c0 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:48.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.992+0000 7ff23f600640 1 -- 172.21.15.6:0/2379181320 >> 172.21.15.6:0/2379181320 conn(0x7ff2380d3ee0 msgr2=0x7ff2380d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:48.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.992+0000 7ff23f600640 1 -- 172.21.15.6:0/2379181320 shutdown_connections 2024-03-04T09:27:48.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.992+0000 7ff23f600640 1 -- 172.21.15.6:0/2379181320 wait complete. 2024-03-04T09:27:48.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.994+0000 7ff23f600640 1 Processor -- start 2024-03-04T09:27:48.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.994+0000 7ff23f600640 1 -- start start 2024-03-04T09:27:48.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.995+0000 7ff23f600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff2380cfa90 0x7ff2381503d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:48.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.995+0000 7ff23f600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff2380d0f80 0x7ff23809dc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:48.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.995+0000 7ff23f600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff238161930 0x7ff23809e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:48.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.995+0000 7ff23f600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2380c8420 con 0x7ff238161930 2024-03-04T09:27:48.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.995+0000 7ff23f600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2380b4f30 con 0x7ff2380d0f80 2024-03-04T09:27:48.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.995+0000 7ff23f600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2380c85a0 con 0x7ff2380cfa90 2024-03-04T09:27:48.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.996+0000 7ff23d600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff238161930 0x7ff23809e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:48.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.996+0000 7ff23d600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff238161930 0x7ff23809e160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42244/0 (socket says 172.21.15.6:42244) 2024-03-04T09:27:48.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.996+0000 7ff23d600640 1 -- 172.21.15.6:0/1271715097 learned_addr learned my addr 172.21.15.6:0/1271715097 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:27:48.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.996+0000 7ff23cc00640 1 --2- 172.21.15.6:0/1271715097 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff2380cfa90 0x7ff2381503d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:48.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.996+0000 7ff237e00640 1 --2- 172.21.15.6:0/1271715097 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff2380d0f80 0x7ff23809dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:48.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.996+0000 7ff23d600640 1 -- 172.21.15.6:0/1271715097 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff2380cfa90 msgr2=0x7ff2381503d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:48.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.996+0000 7ff23d600640 1 --2- 172.21.15.6:0/1271715097 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff2380cfa90 0x7ff2381503d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:48.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.996+0000 7ff23d600640 1 -- 172.21.15.6:0/1271715097 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff2380d0f80 msgr2=0x7ff23809dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:48.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.996+0000 7ff23d600640 1 --2- 172.21.15.6:0/1271715097 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff2380d0f80 0x7ff23809dc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:48.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.996+0000 7ff23d600640 1 -- 172.21.15.6:0/1271715097 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2380c9690 con 0x7ff238161930 2024-03-04T09:27:48.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.996+0000 7ff23d600640 1 --2- 172.21.15.6:0/1271715097 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff238161930 0x7ff23809e160 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7ff23000d830 tx=0x7ff23000dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:48.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.997+0000 7ff234c00640 1 -- 172.21.15.6:0/1271715097 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff23000b6c0 con 0x7ff238161930 2024-03-04T09:27:48.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.997+0000 7ff234c00640 1 -- 172.21.15.6:0/1271715097 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff230019070 con 0x7ff238161930 2024-03-04T09:27:48.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.997+0000 7ff234c00640 1 -- 172.21.15.6:0/1271715097 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff23000bac0 con 0x7ff238161930 2024-03-04T09:27:48.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.997+0000 7ff23f600640 1 -- 172.21.15.6:0/1271715097 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2380c9970 con 0x7ff238161930 2024-03-04T09:27:48.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.997+0000 7ff23f600640 1 -- 172.21.15.6:0/1271715097 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2380c9f30 con 0x7ff238161930 2024-03-04T09:27:48.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.998+0000 7ff23f600640 1 -- 172.21.15.6:0/1271715097 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff23804f330 con 0x7ff238161930 2024-03-04T09:27:48.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:48.999+0000 7ff234c00640 1 -- 172.21.15.6:0/1271715097 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff23001e460 con 0x7ff238161930 2024-03-04T09:27:48.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:49.000+0000 7ff234c00640 1 --2- 172.21.15.6:0/1271715097 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff21406b500 0x7ff21406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:48.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:49.000+0000 7ff234c00640 1 -- 172.21.15.6:0/1271715097 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 4943+0+0 (secure 0 0 0) 0x7ff23008a4a0 con 0x7ff238161930 2024-03-04T09:27:48.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:49.001+0000 7ff23cc00640 1 --2- 172.21.15.6:0/1271715097 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff21406b500 0x7ff21406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:48.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:49.002+0000 7ff23cc00640 1 --2- 172.21.15.6:0/1271715097 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff21406b500 0x7ff21406d9c0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7ff220036000 tx=0x7ff22003d040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:49.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:49.004+0000 7ff234c00640 1 -- 172.21.15.6:0/1271715097 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff2300596b0 con 0x7ff238161930 2024-03-04T09:27:49.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:49.206+0000 7ff23f600640 1 -- 172.21.15.6:0/1271715097 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7ff2380d2a60 con 0x7ff238161930 2024-03-04T09:27:49.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:49.212+0000 7ff234c00640 1 -- 172.21.15.6:0/1271715097 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v15) v1 ==== 104+0+0 (secure 0 0 0) 0x7ff230058fb0 con 0x7ff238161930 2024-03-04T09:27:49.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:49.215+0000 7ff23f600640 1 -- 172.21.15.6:0/1271715097 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff21406b500 msgr2=0x7ff21406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:49.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:49.215+0000 7ff23f600640 1 --2- 172.21.15.6:0/1271715097 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff21406b500 0x7ff21406d9c0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7ff220036000 tx=0x7ff22003d040 comp rx=0 tx=0).stop 2024-03-04T09:27:49.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:49.215+0000 7ff23f600640 1 -- 172.21.15.6:0/1271715097 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff238161930 msgr2=0x7ff23809e160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:49.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:49.215+0000 7ff23f600640 1 --2- 172.21.15.6:0/1271715097 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff238161930 0x7ff23809e160 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7ff23000d830 tx=0x7ff23000dcf0 comp rx=0 tx=0).stop 2024-03-04T09:27:49.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:49.215+0000 7ff23f600640 1 -- 172.21.15.6:0/1271715097 shutdown_connections 2024-03-04T09:27:49.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:49.215+0000 7ff23f600640 1 --2- 172.21.15.6:0/1271715097 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff238161930 0x7ff23809e160 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:49.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:49.215+0000 7ff23f600640 1 --2- 172.21.15.6:0/1271715097 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff2380d0f80 0x7ff23809dc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:49.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:49.215+0000 7ff23f600640 1 --2- 172.21.15.6:0/1271715097 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff21406b500 0x7ff21406d9c0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:49.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:49.215+0000 7ff23f600640 1 --2- 172.21.15.6:0/1271715097 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff2380cfa90 0x7ff2381503d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:49.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:49.215+0000 7ff23f600640 1 -- 172.21.15.6:0/1271715097 >> 172.21.15.6:0/1271715097 conn(0x7ff2380d3ee0 msgr2=0x7ff238160470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:49.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:49.216+0000 7ff23f600640 1 -- 172.21.15.6:0/1271715097 shutdown_connections 2024-03-04T09:27:49.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:49.216+0000 7ff23f600640 1 -- 172.21.15.6:0/1271715097 wait complete. 2024-03-04T09:27:49.233 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit' 2024-03-04T09:27:49.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:49.450+0000 7f1ee6600640 1 Processor -- start 2024-03-04T09:27:49.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:49.451+0000 7f1ee6600640 1 -- start start 2024-03-04T09:27:49.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:49.452+0000 7f1ee6600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1ee00af580 0x7f1ee015c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:49.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:49.452+0000 7f1ee6600640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ee015cc40 con 0x7f1ee00cdc20 2024-03-04T09:27:49.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:49.452+0000 7f1ee6600640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ee015cf20 con 0x7f1ee00af1a0 2024-03-04T09:27:49.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:49.452+0000 7f1ee6600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ee015d200 con 0x7f1ee00af580 2024-03-04T09:27:49.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:49.452+0000 7f1edfe00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f1ee00cdc20 0x7f1ee00b2dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:58876/0 (socket says 172.21.15.6:58876) 2024-03-04T09:27:49.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:49.452+0000 7f1edfe00640 1 -- 172.21.15.6:0/323954896 learned_addr learned my addr 172.21.15.6:0/323954896 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:27:49.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:49.453+0000 7f1ed6a00640 1 --2- 172.21.15.6:0/323954896 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1ee00af580 0x7f1ee015c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:49.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:49.453+0000 7f1edea00640 1 -- 172.21.15.6:0/323954896 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1236498611 0 0) 0x7f1ee015cc40 con 0x7f1ee00cdc20 2024-03-04T09:27:49.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:49.453+0000 7f1edea00640 1 -- 172.21.15.6:0/323954896 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ec4003520 con 0x7f1ee00cdc20 2024-03-04T09:27:49.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:49.453+0000 7f1edea00640 1 -- 172.21.15.6:0/323954896 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3935894177 0 0) 0x7f1ec4003520 con 0x7f1ee00cdc20 2024-03-04T09:27:49.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:49.453+0000 7f1edea00640 1 -- 172.21.15.6:0/323954896 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1ee00af580 msgr2=0x7f1ee015c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:27:49.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:49.453+0000 7f1edea00640 1 --2- 172.21.15.6:0/323954896 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1ee00af580 0x7f1ee015c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:49.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:49.453+0000 7f1edea00640 1 -- 172.21.15.6:0/323954896 >> v1:172.21.15.146:6789/0 conn(0x7f1ee00af1a0 legacy=0x7f1ee00b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:49.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:49.453+0000 7f1edea00640 1 -- 172.21.15.6:0/323954896 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ee015d4e0 con 0x7f1ee00cdc20 2024-03-04T09:27:49.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:49.453+0000 7f1edea00640 1 -- 172.21.15.6:0/323954896 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7f1ed8001670 con 0x7f1ee00cdc20 2024-03-04T09:27:49.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:49.453+0000 7f1edea00640 1 -- 172.21.15.6:0/323954896 >> v1:172.21.15.6:6789/0 conn(0x7f1ee00cdc20 legacy=0x7f1ee00b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:49.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:49.453+0000 7f1edea00640 1 --2- 172.21.15.6:0/323954896 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1ec4003a50 0x7f1ec4003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:49.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:49.453+0000 7f1edea00640 1 --2- 172.21.15.6:0/323954896 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1ec4004800 0x7f1ec4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:49.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:49.453+0000 7f1edea00640 1 --2- 172.21.15.6:0/323954896 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1ee00af580 0x7f1ec400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:49.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:49.453+0000 7f1edea00640 1 -- 172.21.15.6:0/323954896 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ee015d200 con 0x7f1ec4003a50 2024-03-04T09:27:49.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:49.453+0000 7f1edea00640 1 -- 172.21.15.6:0/323954896 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ec4005550 con 0x7f1ec4004800 2024-03-04T09:27:49.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:49.453+0000 7f1edea00640 1 -- 172.21.15.6:0/323954896 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ec400bf00 con 0x7f1ee00af580 2024-03-04T09:27:49.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:49.454+0000 7f1edf400640 1 --2- 172.21.15.6:0/323954896 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1ec4003a50 0x7f1ec4003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:49.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:49.454+0000 7f1edfe00640 1 --2- 172.21.15.6:0/323954896 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1ee00af580 0x7f1ec400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:49.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:49.454+0000 7f1edf400640 1 -- 172.21.15.6:0/323954896 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1ee00af580 msgr2=0x7f1ec400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:27:49.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:49.454+0000 7f1edf400640 1 --2- 172.21.15.6:0/323954896 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1ee00af580 0x7f1ec400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:49.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:49.454+0000 7f1edf400640 1 -- 172.21.15.6:0/323954896 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1ec4004800 msgr2=0x7f1ec4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:27:49.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:49.454+0000 7f1edf400640 1 --2- 172.21.15.6:0/323954896 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1ec4004800 0x7f1ec4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:49.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:49.454+0000 7f1edf400640 1 -- 172.21.15.6:0/323954896 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ec4003520 con 0x7f1ec4003a50 2024-03-04T09:27:49.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:49.455+0000 7f1edf400640 1 --2- 172.21.15.6:0/323954896 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1ec4003a50 0x7f1ec4003e70 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f1ed0001e50 tx=0x7f1ec4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1c4158919b607b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:49.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:49.456+0000 7f1edea00640 1 -- 172.21.15.6:0/323954896 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ed0001a70 con 0x7f1ec4003a50 2024-03-04T09:27:49.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:49.456+0000 7f1edea00640 1 -- 172.21.15.6:0/323954896 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ed0003b70 con 0x7f1ec4003a50 2024-03-04T09:27:49.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:49.456+0000 7f1edea00640 1 -- 172.21.15.6:0/323954896 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ed00050d0 con 0x7f1ec4003a50 2024-03-04T09:27:49.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:49.456+0000 7f1ee6600640 1 -- 172.21.15.6:0/323954896 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1ec4003a50 msgr2=0x7f1ec4003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:49.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:49.456+0000 7f1ee6600640 1 --2- 172.21.15.6:0/323954896 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1ec4003a50 0x7f1ec4003e70 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f1ed0001e50 tx=0x7f1ec4005830 comp rx=0 tx=0).stop 2024-03-04T09:27:49.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:49.456+0000 7f1edfe00640 1 -- 172.21.15.6:0/323954896 reap_dead start 2024-03-04T09:27:49.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:49.456+0000 7f1ee6600640 1 -- 172.21.15.6:0/323954896 shutdown_connections 2024-03-04T09:27:49.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:49.456+0000 7f1ee6600640 1 -- 172.21.15.6:0/323954896 >> 172.21.15.6:0/323954896 conn(0x7f1ee006afb0 msgr2=0x7f1ee006b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:49.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:49.456+0000 7f1ee6600640 1 -- 172.21.15.6:0/323954896 shutdown_connections 2024-03-04T09:27:49.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:49.456+0000 7f1ee6600640 1 -- 172.21.15.6:0/323954896 wait complete. 2024-03-04T09:27:49.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:49.458+0000 7f1ee6600640 1 Processor -- start 2024-03-04T09:27:49.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:49.458+0000 7f1ee6600640 1 -- start start 2024-03-04T09:27:49.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:49.459+0000 7f1ee6600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1ec4003a50 0x7f1ee006e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:49.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:49.459+0000 7f1ee6600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1ee015aed0 0x7f1ee00af960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:49.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:49.459+0000 7f1ee6600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1ee015b330 0x7f1ee00afea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:49.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:49.459+0000 7f1ee6600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ee015c600 con 0x7f1ee015b330 2024-03-04T09:27:49.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:49.459+0000 7f1ee6600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ee00b36f0 con 0x7f1ee015aed0 2024-03-04T09:27:49.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:49.459+0000 7f1ee6600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ee015c780 con 0x7f1ec4003a50 2024-03-04T09:27:49.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:49.460+0000 7f1edfe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1ee015b330 0x7f1ee00afea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:49.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:49.460+0000 7f1edfe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1ee015b330 0x7f1ee00afea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42250/0 (socket says 172.21.15.6:42250) 2024-03-04T09:27:49.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:49.460+0000 7f1edfe00640 1 -- 172.21.15.6:0/2747707969 learned_addr learned my addr 172.21.15.6:0/2747707969 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:27:49.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:49.460+0000 7f1edfe00640 1 -- 172.21.15.6:0/2747707969 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1ec4003a50 msgr2=0x7f1ee006e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:49.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:49.460+0000 7f1edfe00640 1 --2- 172.21.15.6:0/2747707969 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1ec4003a50 0x7f1ee006e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:49.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:49.460+0000 7f1ed6a00640 1 --2- 172.21.15.6:0/2747707969 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1ee015aed0 0x7f1ee00af960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:49.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:49.460+0000 7f1edfe00640 1 -- 172.21.15.6:0/2747707969 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1ee015aed0 msgr2=0x7f1ee00af960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:49.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:49.460+0000 7f1edfe00640 1 --2- 172.21.15.6:0/2747707969 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1ee015aed0 0x7f1ee00af960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:49.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:49.460+0000 7f1edfe00640 1 -- 172.21.15.6:0/2747707969 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ee0156f10 con 0x7f1ee015b330 2024-03-04T09:27:49.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:49.460+0000 7f1edfe00640 1 --2- 172.21.15.6:0/2747707969 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1ee015b330 0x7f1ee00afea0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f1ee009e100 tx=0x7f1ed800e8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:49.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:49.461+0000 7f1ed7e00640 1 -- 172.21.15.6:0/2747707969 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ed8010510 con 0x7f1ee015b330 2024-03-04T09:27:49.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:49.461+0000 7f1ed7e00640 1 -- 172.21.15.6:0/2747707969 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ed80106d0 con 0x7f1ee015b330 2024-03-04T09:27:49.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:49.461+0000 7f1ee6600640 1 -- 172.21.15.6:0/2747707969 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ee0155790 con 0x7f1ee015b330 2024-03-04T09:27:49.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:49.461+0000 7f1ed7e00640 1 -- 172.21.15.6:0/2747707969 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ed8010ac0 con 0x7f1ee015b330 2024-03-04T09:27:49.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:49.461+0000 7f1ee6600640 1 -- 172.21.15.6:0/2747707969 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ee0157460 con 0x7f1ee015b330 2024-03-04T09:27:49.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:49.462+0000 7f1ee6600640 1 -- 172.21.15.6:0/2747707969 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ee0069510 con 0x7f1ee015b330 2024-03-04T09:27:49.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:49.463+0000 7f1ed7e00640 1 -- 172.21.15.6:0/2747707969 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1ed8010c60 con 0x7f1ee015b330 2024-03-04T09:27:49.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:49.464+0000 7f1ed7e00640 1 --2- 172.21.15.6:0/2747707969 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f1ebc06b4d0 0x7f1ebc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:49.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:49.464+0000 7f1ed7e00640 1 -- 172.21.15.6:0/2747707969 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f1ed8089630 con 0x7f1ee015b330 2024-03-04T09:27:49.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:49.465+0000 7f1edf400640 1 --2- 172.21.15.6:0/2747707969 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f1ebc06b4d0 0x7f1ebc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:49.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:49.466+0000 7f1edf400640 1 --2- 172.21.15.6:0/2747707969 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f1ebc06b4d0 0x7f1ebc06d990 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f1ed0034000 tx=0x7f1ed0030000 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:49.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:49.468+0000 7f1ed7e00640 1 -- 172.21.15.6:0/2747707969 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1ed8058710 con 0x7f1ee015b330 2024-03-04T09:27:49.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:49.742+0000 7f1ee6600640 1 -- 172.21.15.6:0/2747707969 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit"]} v 0) v1 -- 0x7f1ee004f3a0 con 0x7f1ee015b330 2024-03-04T09:27:50.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:50.289+0000 7f1ed7e00640 1 -- 172.21.15.6:0/2747707969 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit"]}]=0 v156) v1 ==== 133+0+0 (secure 0 0 0) 0x7f1ed8058010 con 0x7f1ee015b330 2024-03-04T09:27:50.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:50.293+0000 7f1ee6600640 1 -- 172.21.15.6:0/2747707969 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f1ebc06b4d0 msgr2=0x7f1ebc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:50.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:50.293+0000 7f1ee6600640 1 --2- 172.21.15.6:0/2747707969 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f1ebc06b4d0 0x7f1ebc06d990 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f1ed0034000 tx=0x7f1ed0030000 comp rx=0 tx=0).stop 2024-03-04T09:27:50.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:50.293+0000 7f1ee6600640 1 -- 172.21.15.6:0/2747707969 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1ee015b330 msgr2=0x7f1ee00afea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:50.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:50.293+0000 7f1ee6600640 1 --2- 172.21.15.6:0/2747707969 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1ee015b330 0x7f1ee00afea0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f1ee009e100 tx=0x7f1ed800e8a0 comp rx=0 tx=0).stop 2024-03-04T09:27:50.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:50.293+0000 7f1ee6600640 1 -- 172.21.15.6:0/2747707969 shutdown_connections 2024-03-04T09:27:50.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:50.293+0000 7f1ee6600640 1 --2- 172.21.15.6:0/2747707969 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1ee015b330 0x7f1ee00afea0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:50.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:50.293+0000 7f1ee6600640 1 --2- 172.21.15.6:0/2747707969 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1ee015aed0 0x7f1ee00af960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:50.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:50.293+0000 7f1ee6600640 1 --2- 172.21.15.6:0/2747707969 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f1ebc06b4d0 0x7f1ebc06d990 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:50.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:50.293+0000 7f1ee6600640 1 --2- 172.21.15.6:0/2747707969 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1ec4003a50 0x7f1ee006e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:50.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:50.293+0000 7f1ee6600640 1 -- 172.21.15.6:0/2747707969 >> 172.21.15.6:0/2747707969 conn(0x7f1ee006afb0 msgr2=0x7f1ee00cc750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:50.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:50.294+0000 7f1ee6600640 1 -- 172.21.15.6:0/2747707969 shutdown_connections 2024-03-04T09:27:50.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:50.294+0000 7f1ee6600640 1 -- 172.21.15.6:0/2747707969 wait complete. 2024-03-04T09:27:50.314 INFO:tasks.cephfs_test_runner:That the directory fragment size cannot exceed mds_bal_fragment_size_max (using a limit of 50 in all configurations). ... ok 2024-03-04T09:27:50.314 INFO:tasks.cephfs_test_runner:Starting test: test_dirfrag_limit_fragmented (tasks.cephfs.test_strays.TestStrays) That fragmentation (forced) will allow more entries to be created. 2024-03-04T09:27:50.314 INFO:tasks.cephfs_test_runner:test_dirfrag_limit_fragmented (tasks.cephfs.test_strays.TestStrays) 2024-03-04T09:27:50.315 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T09:27:50.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:50.538+0000 7fa682e00640 1 Processor -- start 2024-03-04T09:27:50.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:50.539+0000 7fa682e00640 1 -- start start 2024-03-04T09:27:50.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:50.540+0000 7fa682e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fa67c070f10 0x7fa67c0cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:50.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:50.540+0000 7fa682e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa67c0714d0 con 0x7fa67c145300 2024-03-04T09:27:50.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:50.540+0000 7fa682e00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa67c0717b0 con 0x7fa67c0cce90 2024-03-04T09:27:50.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:50.540+0000 7fa682e00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa67c0cd300 con 0x7fa67c070f10 2024-03-04T09:27:50.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:50.541+0000 7fa681a00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fa67c145300 0x7fa67c0b49a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:58890/0 (socket says 172.21.15.6:58890) 2024-03-04T09:27:50.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:50.541+0000 7fa681a00640 1 -- 172.21.15.6:0/1203122509 learned_addr learned my addr 172.21.15.6:0/1203122509 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:27:50.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:50.541+0000 7fa681000640 1 --2- 172.21.15.6:0/1203122509 >> v2:172.21.15.146:3300/0 conn(0x7fa67c070f10 0x7fa67c0cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:50.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:50.541+0000 7fa67be00640 1 -- 172.21.15.6:0/1203122509 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2011798070 0 0) 0x7fa67c0714d0 con 0x7fa67c145300 2024-03-04T09:27:50.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:50.541+0000 7fa67be00640 1 -- 172.21.15.6:0/1203122509 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa660003520 con 0x7fa67c145300 2024-03-04T09:27:50.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:50.541+0000 7fa67be00640 1 -- 172.21.15.6:0/1203122509 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 679970582 0 0) 0x7fa660003520 con 0x7fa67c145300 2024-03-04T09:27:50.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:50.541+0000 7fa67be00640 1 -- 172.21.15.6:0/1203122509 >> v2:172.21.15.146:3300/0 conn(0x7fa67c070f10 msgr2=0x7fa67c0cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:27:50.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:50.541+0000 7fa67be00640 1 --2- 172.21.15.6:0/1203122509 >> v2:172.21.15.146:3300/0 conn(0x7fa67c070f10 0x7fa67c0cc950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:50.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:50.541+0000 7fa67be00640 1 -- 172.21.15.6:0/1203122509 >> v1:172.21.15.146:6789/0 conn(0x7fa67c0cce90 legacy=0x7fa67c15c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:50.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:50.541+0000 7fa67be00640 1 -- 172.21.15.6:0/1203122509 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa67c071a90 con 0x7fa67c145300 2024-03-04T09:27:50.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:50.541+0000 7fa67be00640 1 -- 172.21.15.6:0/1203122509 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7fa66c0019f0 con 0x7fa67c145300 2024-03-04T09:27:50.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:50.542+0000 7fa67be00640 1 -- 172.21.15.6:0/1203122509 >> v1:172.21.15.6:6789/0 conn(0x7fa67c145300 legacy=0x7fa67c0b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:50.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:50.542+0000 7fa67be00640 1 --2- 172.21.15.6:0/1203122509 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa6600039b0 0x7fa660003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:50.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:50.542+0000 7fa67be00640 1 --2- 172.21.15.6:0/1203122509 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa660004800 0x7fa660004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:50.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:50.542+0000 7fa67be00640 1 --2- 172.21.15.6:0/1203122509 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa6600052e0 0x7fa66000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:50.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:50.542+0000 7fa67be00640 1 -- 172.21.15.6:0/1203122509 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa67c0cd300 con 0x7fa6600039b0 2024-03-04T09:27:50.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:50.542+0000 7fa67be00640 1 -- 172.21.15.6:0/1203122509 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa66000c1b0 con 0x7fa6600052e0 2024-03-04T09:27:50.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:50.542+0000 7fa67be00640 1 -- 172.21.15.6:0/1203122509 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa66000c3b0 con 0x7fa660004800 2024-03-04T09:27:50.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:50.542+0000 7fa681a00640 1 --2- 172.21.15.6:0/1203122509 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa6600039b0 0x7fa660003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:50.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:50.542+0000 7fa681000640 1 --2- 172.21.15.6:0/1203122509 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa660004800 0x7fa660004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:50.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:50.542+0000 7fa682400640 1 --2- 172.21.15.6:0/1203122509 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa6600052e0 0x7fa66000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:50.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:50.542+0000 7fa681a00640 1 -- 172.21.15.6:0/1203122509 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa660004800 msgr2=0x7fa660004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:27:50.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:50.542+0000 7fa681a00640 1 --2- 172.21.15.6:0/1203122509 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa660004800 0x7fa660004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:50.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:50.542+0000 7fa681a00640 1 -- 172.21.15.6:0/1203122509 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa6600052e0 msgr2=0x7fa66000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:27:50.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:50.542+0000 7fa681a00640 1 --2- 172.21.15.6:0/1203122509 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa6600052e0 0x7fa66000b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:50.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:50.542+0000 7fa681a00640 1 -- 172.21.15.6:0/1203122509 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa660003520 con 0x7fa6600039b0 2024-03-04T09:27:50.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:50.543+0000 7fa682400640 1 -- 172.21.15.6:0/1203122509 reap_dead start 2024-03-04T09:27:50.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:50.543+0000 7fa681000640 1 --2- 172.21.15.6:0/1203122509 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa660004800 0x7fa660004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:27:50.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:50.544+0000 7fa681a00640 1 --2- 172.21.15.6:0/1203122509 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa6600039b0 0x7fa660003dd0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fa66c00a830 tx=0x7fa660005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8540cd69addab139 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:50.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:50.544+0000 7fa67be00640 1 -- 172.21.15.6:0/1203122509 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa66c00a480 con 0x7fa6600039b0 2024-03-04T09:27:50.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:50.544+0000 7fa67be00640 1 -- 172.21.15.6:0/1203122509 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa66c034630 con 0x7fa6600039b0 2024-03-04T09:27:50.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:50.544+0000 7fa67be00640 1 -- 172.21.15.6:0/1203122509 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa66c034a80 con 0x7fa6600039b0 2024-03-04T09:27:50.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:50.545+0000 7fa682e00640 1 -- 172.21.15.6:0/1203122509 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa6600039b0 msgr2=0x7fa660003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:50.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:50.545+0000 7fa682e00640 1 --2- 172.21.15.6:0/1203122509 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa6600039b0 0x7fa660003dd0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fa66c00a830 tx=0x7fa660005830 comp rx=0 tx=0).stop 2024-03-04T09:27:50.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:50.545+0000 7fa682e00640 1 -- 172.21.15.6:0/1203122509 shutdown_connections 2024-03-04T09:27:50.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:50.545+0000 7fa682e00640 1 --2- 172.21.15.6:0/1203122509 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa6600039b0 0x7fa660003dd0 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:50.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:50.545+0000 7fa682e00640 1 -- 172.21.15.6:0/1203122509 >> 172.21.15.6:0/1203122509 conn(0x7fa67c06b3d0 msgr2=0x7fa67c06b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:50.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:50.545+0000 7fa682e00640 1 -- 172.21.15.6:0/1203122509 shutdown_connections 2024-03-04T09:27:50.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:50.545+0000 7fa682e00640 1 -- 172.21.15.6:0/1203122509 wait complete. 2024-03-04T09:27:50.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:50.548+0000 7fa682e00640 1 Processor -- start 2024-03-04T09:27:50.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:50.548+0000 7fa682e00640 1 -- start start 2024-03-04T09:27:50.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:50.549+0000 7fa682e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa6600039b0 0x7fa67c1f3d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:50.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:50.549+0000 7fa682e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa67c1e6810 0x7fa67c15b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:50.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:50.549+0000 7fa682e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa67c1e5870 0x7fa67c1e9170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:50.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:50.549+0000 7fa682e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa67c0cc950 con 0x7fa6600039b0 2024-03-04T09:27:50.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:50.549+0000 7fa682e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa67c1456e0 con 0x7fa67c1e6810 2024-03-04T09:27:50.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:50.549+0000 7fa682e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa67c0ccad0 con 0x7fa67c1e5870 2024-03-04T09:27:50.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:50.549+0000 7fa681a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa6600039b0 0x7fa67c1f3d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:50.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:50.550+0000 7fa681a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa6600039b0 0x7fa67c1f3d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42260/0 (socket says 172.21.15.6:42260) 2024-03-04T09:27:50.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:50.550+0000 7fa681a00640 1 -- 172.21.15.6:0/3787741345 learned_addr learned my addr 172.21.15.6:0/3787741345 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:27:50.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:50.550+0000 7fa682400640 1 --2- 172.21.15.6:0/3787741345 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa67c1e5870 0x7fa67c1e9170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:50.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:50.550+0000 7fa681000640 1 --2- 172.21.15.6:0/3787741345 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa67c1e6810 0x7fa67c15b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:50.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:50.550+0000 7fa681a00640 1 -- 172.21.15.6:0/3787741345 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa67c1e5870 msgr2=0x7fa67c1e9170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:50.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:50.550+0000 7fa681a00640 1 --2- 172.21.15.6:0/3787741345 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa67c1e5870 0x7fa67c1e9170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:50.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:50.550+0000 7fa681a00640 1 -- 172.21.15.6:0/3787741345 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa67c1e6810 msgr2=0x7fa67c15b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:50.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:50.550+0000 7fa681a00640 1 --2- 172.21.15.6:0/3787741345 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa67c1e6810 0x7fa67c15b430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:50.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:50.550+0000 7fa681a00640 1 -- 172.21.15.6:0/3787741345 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa67c1f42b0 con 0x7fa6600039b0 2024-03-04T09:27:50.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:50.550+0000 7fa682400640 1 --2- 172.21.15.6:0/3787741345 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa67c1e5870 0x7fa67c1e9170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:27:50.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:50.550+0000 7fa681a00640 1 --2- 172.21.15.6:0/3787741345 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa6600039b0 0x7fa67c1f3d70 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fa66c00a950 tx=0x7fa67c071980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:50.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:50.551+0000 7fa679600640 1 -- 172.21.15.6:0/3787741345 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa66c039040 con 0x7fa6600039b0 2024-03-04T09:27:50.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:50.551+0000 7fa679600640 1 -- 172.21.15.6:0/3787741345 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa66c0017f0 con 0x7fa6600039b0 2024-03-04T09:27:50.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:50.551+0000 7fa682e00640 1 -- 172.21.15.6:0/3787741345 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa67c15b970 con 0x7fa6600039b0 2024-03-04T09:27:50.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:50.551+0000 7fa679600640 1 -- 172.21.15.6:0/3787741345 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa66c00a480 con 0x7fa6600039b0 2024-03-04T09:27:50.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:50.551+0000 7fa682e00640 1 -- 172.21.15.6:0/3787741345 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa67c1e6ff0 con 0x7fa6600039b0 2024-03-04T09:27:50.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:50.552+0000 7fa682e00640 1 -- 172.21.15.6:0/3787741345 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa67c04f330 con 0x7fa6600039b0 2024-03-04T09:27:50.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:50.553+0000 7fa679600640 1 -- 172.21.15.6:0/3787741345 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa66c034630 con 0x7fa6600039b0 2024-03-04T09:27:50.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:50.554+0000 7fa679600640 1 --2- 172.21.15.6:0/3787741345 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa65c06b4d0 0x7fa65c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:50.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:50.554+0000 7fa679600640 1 -- 172.21.15.6:0/3787741345 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fa66c0abe30 con 0x7fa6600039b0 2024-03-04T09:27:50.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:50.555+0000 7fa681000640 1 --2- 172.21.15.6:0/3787741345 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa65c06b4d0 0x7fa65c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:50.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:50.556+0000 7fa681000640 1 --2- 172.21.15.6:0/3787741345 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa65c06b4d0 0x7fa65c06d990 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fa67c1f72f0 tx=0x7fa668008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:50.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:50.559+0000 7fa679600640 1 -- 172.21.15.6:0/3787741345 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa66c07af30 con 0x7fa6600039b0 2024-03-04T09:27:50.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:50.764+0000 7fa682e00640 1 -- 172.21.15.6:0/3787741345 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa67c0b1400 con 0x7fa6600039b0 2024-03-04T09:27:50.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:50.766+0000 7fa679600640 1 -- 172.21.15.6:0/3787741345 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v47) v1 ==== 74+0+12974 (secure 0 0 0) 0x7fa66c07a830 con 0x7fa6600039b0 2024-03-04T09:27:50.763 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:27:50.764 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":47,"fsid":"edce0578-5ae4-4402-b45a-552bdef84d7c","created":"2024-03-04T09:24:54.523780+0000","modified":"2024-03-04T09:27:48.267516+0000","last_up_change":"2024-03-04T09:25:07.089049+0000","last_in_change":"2024-03-04T09:24:57.982077+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T09:25:07.935198+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a96574f-81f1-4bc6-8463-757520b84a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6825","nonce":1364449401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6827","nonce":1364449401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6831","nonce":1364449401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6829","nonce":1364449401}]},"public_addr":"172.21.15.6:6825/1364449401","cluster_addr":"172.21.15.6:6827/1364449401","heartbeat_back_addr":"172.21.15.6:6831/1364449401","heartbeat_front_addr":"172.21.15.6:6829/1364449401","state":["exists","up"]},{"osd":1,"uuid":"7d79b9c1-6b87-476a-bc36-7db5ee64cd90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6809","nonce":2494103078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6811","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6813","nonce":2494103078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6821","nonce":2494103078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6817","nonce":2494103078}]},"public_addr":"172.21.15.6:6809/2494103078","cluster_addr":"172.21.15.6:6813/2494103078","heartbeat_back_addr":"172.21.15.6:6821/2494103078","heartbeat_front_addr":"172.21.15.6:6817/2494103078","state":["exists","up"]},{"osd":2,"uuid":"492331b9-4ee9-40b6-bacd-85d44cd6c45a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6812","nonce":1500066681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6816","nonce":1500066681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6824","nonce":1500066681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6820","nonce":1500066681}]},"public_addr":"172.21.15.6:6812/1500066681","cluster_addr":"172.21.15.6:6816/1500066681","heartbeat_back_addr":"172.21.15.6:6824/1500066681","heartbeat_front_addr":"172.21.15.6:6820/1500066681","state":["exists","up"]},{"osd":3,"uuid":"21bb2efe-9ba7-4bdc-bc31-180cd1102e21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6801","nonce":1558806123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6803","nonce":1558806123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6807","nonce":1558806123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6805","nonce":1558806123}]},"public_addr":"172.21.15.6:6801/1558806123","cluster_addr":"172.21.15.6:6803/1558806123","heartbeat_back_addr":"172.21.15.6:6807/1558806123","heartbeat_front_addr":"172.21.15.6:6805/1558806123","state":["exists","up"]},{"osd":4,"uuid":"6e2c4b71-19d9-4737-83d9-558705d8644a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6801","nonce":1526135780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6803","nonce":1526135780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6807","nonce":1526135780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6805","nonce":1526135780}]},"public_addr":"172.21.15.146:6801/1526135780","cluster_addr":"172.21.15.146:6803/1526135780","heartbeat_back_addr":"172.21.15.146:6807/1526135780","heartbeat_front_addr":"172.21.15.146:6805/1526135780","state":["exists","up"]},{"osd":5,"uuid":"fb69c525-f1be-4b7d-ab32-6c29b0d6fddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6809","nonce":460262507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6811","nonce":460262507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6815","nonce":460262507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6813","nonce":460262507}]},"public_addr":"172.21.15.146:6809/460262507","cluster_addr":"172.21.15.146:6811/460262507","heartbeat_back_addr":"172.21.15.146:6815/460262507","heartbeat_front_addr":"172.21.15.146:6813/460262507","state":["exists","up"]},{"osd":6,"uuid":"c32e7fed-370d-4401-8ce0-6a4335b0d168","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6823","nonce":1392897493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6827","nonce":1392897493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6831","nonce":1392897493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6829","nonce":1392897493}]},"public_addr":"172.21.15.146:6823/1392897493","cluster_addr":"172.21.15.146:6827/1392897493","heartbeat_back_addr":"172.21.15.146:6831/1392897493","heartbeat_front_addr":"172.21.15.146:6829/1392897493","state":["exists","up"]},{"osd":7,"uuid":"75a79b44-fa5f-426d-a7d3-71f21f78835c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6817","nonce":3268323156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6819","nonce":3268323156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6825","nonce":3268323156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6822","nonce":3268323156}]},"public_addr":"172.21.15.146:6817/3268323156","cluster_addr":"172.21.15.146:6819/3268323156","heartbeat_back_addr":"172.21.15.146:6825/3268323156","heartbeat_front_addr":"172.21.15.146:6822/3268323156","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.310437+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.295850+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.294674+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.526855+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.300023+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.595160+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.346046+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.558741+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.146:6837/3982958749":"2024-03-05T09:27:44.832002+0000","172.21.15.146:6836/3982958749":"2024-03-05T09:27:44.832002+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T09:27:50.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:50.769+0000 7fa682e00640 1 -- 172.21.15.6:0/3787741345 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa65c06b4d0 msgr2=0x7fa65c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:50.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:50.769+0000 7fa682e00640 1 --2- 172.21.15.6:0/3787741345 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa65c06b4d0 0x7fa65c06d990 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fa67c1f72f0 tx=0x7fa668008040 comp rx=0 tx=0).stop 2024-03-04T09:27:50.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:50.769+0000 7fa682e00640 1 -- 172.21.15.6:0/3787741345 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa6600039b0 msgr2=0x7fa67c1f3d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:50.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:50.769+0000 7fa682e00640 1 --2- 172.21.15.6:0/3787741345 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa6600039b0 0x7fa67c1f3d70 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fa66c00a950 tx=0x7fa67c071980 comp rx=0 tx=0).stop 2024-03-04T09:27:50.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:50.769+0000 7fa682e00640 1 -- 172.21.15.6:0/3787741345 shutdown_connections 2024-03-04T09:27:50.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:50.769+0000 7fa682e00640 1 --2- 172.21.15.6:0/3787741345 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa65c06b4d0 0x7fa65c06d990 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:50.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:50.769+0000 7fa682e00640 1 --2- 172.21.15.6:0/3787741345 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa67c1e5870 0x7fa67c1e9170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:50.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:50.769+0000 7fa682e00640 1 --2- 172.21.15.6:0/3787741345 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa67c1e6810 0x7fa67c15b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:50.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:50.769+0000 7fa682e00640 1 --2- 172.21.15.6:0/3787741345 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa6600039b0 0x7fa67c1f3d70 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:50.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:50.769+0000 7fa682e00640 1 -- 172.21.15.6:0/3787741345 >> 172.21.15.6:0/3787741345 conn(0x7fa67c06b3d0 msgr2=0x7fa67c0b28b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:50.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:50.770+0000 7fa682e00640 1 -- 172.21.15.6:0/3787741345 shutdown_connections 2024-03-04T09:27:50.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:50.770+0000 7fa682e00640 1 -- 172.21.15.6:0/3787741345 wait complete. 2024-03-04T09:27:50.786 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T09:25:07.935198+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-04T09:27:50.787 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-04T09:27:51.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.016+0000 7fbe04a00640 1 Processor -- start 2024-03-04T09:27:51.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.016+0000 7fbe04a00640 1 -- start start 2024-03-04T09:27:51.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.018+0000 7fbe04a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fbe0015f810 0x7fbe000b1b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:51.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.018+0000 7fbe04a00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fbe000b2080 0x7fbe000b2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:51.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.018+0000 7fbe04a00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe0015fe40 con 0x7fbe00145430 2024-03-04T09:27:51.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.018+0000 7fbe04a00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe000b24d0 con 0x7fbe0015f810 2024-03-04T09:27:51.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.018+0000 7fbe04a00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe00160120 con 0x7fbe000b2080 2024-03-04T09:27:51.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.018+0000 7fbdfea00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fbe0015f810 0x7fbe000b1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:51.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.018+0000 7fbdfea00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fbe0015f810 0x7fbe000b1b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42272/0 (socket says 172.21.15.6:42272) 2024-03-04T09:27:51.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.018+0000 7fbdfea00640 1 -- 172.21.15.6:0/53147114 learned_addr learned my addr 172.21.15.6:0/53147114 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:27:51.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.019+0000 7fbdfea00640 1 -- 172.21.15.6:0/53147114 >> v2:172.21.15.146:3300/0 conn(0x7fbe000b2080 msgr2=0x7fbe000b2fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:27:51.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.019+0000 7fbdfea00640 1 --2- 172.21.15.6:0/53147114 >> v2:172.21.15.146:3300/0 conn(0x7fbe000b2080 0x7fbe000b2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:51.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.019+0000 7fbdfea00640 1 -- 172.21.15.6:0/53147114 >> v1:172.21.15.146:6789/0 conn(0x7fbe00145430 legacy=0x7fbe0015cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:27:51.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.019+0000 7fbdfea00640 1 -- 172.21.15.6:0/53147114 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe00160400 con 0x7fbe0015f810 2024-03-04T09:27:51.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.020+0000 7fbdfea00640 1 --2- 172.21.15.6:0/53147114 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbe0015f810 0x7fbe000b1b40 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fbe000b4440 tx=0x7fbdec02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d289877e3c1a4e84 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:51.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.020+0000 7fbdfe000640 1 -- 172.21.15.6:0/53147114 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdec035020 con 0x7fbe0015f810 2024-03-04T09:27:51.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.020+0000 7fbdfe000640 1 -- 172.21.15.6:0/53147114 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdec031870 con 0x7fbe0015f810 2024-03-04T09:27:51.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.020+0000 7fbdfe000640 1 -- 172.21.15.6:0/53147114 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdec031c40 con 0x7fbe0015f810 2024-03-04T09:27:51.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.021+0000 7fbe04a00640 1 -- 172.21.15.6:0/53147114 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbe0015f810 msgr2=0x7fbe000b1b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:51.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.021+0000 7fbe04a00640 1 --2- 172.21.15.6:0/53147114 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbe0015f810 0x7fbe000b1b40 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fbe000b4440 tx=0x7fbdec02b910 comp rx=0 tx=0).stop 2024-03-04T09:27:51.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.021+0000 7fbe04a00640 1 -- 172.21.15.6:0/53147114 shutdown_connections 2024-03-04T09:27:51.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.021+0000 7fbe04a00640 1 --2- 172.21.15.6:0/53147114 >> v2:172.21.15.146:3300/0 conn(0x7fbe000b2080 0x7fbe000b2fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:51.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.021+0000 7fbe04a00640 1 --2- 172.21.15.6:0/53147114 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbe0015f810 0x7fbe000b1b40 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:51.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.021+0000 7fbe04a00640 1 -- 172.21.15.6:0/53147114 >> 172.21.15.6:0/53147114 conn(0x7fbe0006b3a0 msgr2=0x7fbe0006b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:51.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.021+0000 7fbe04a00640 1 -- 172.21.15.6:0/53147114 shutdown_connections 2024-03-04T09:27:51.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.021+0000 7fbe04a00640 1 -- 172.21.15.6:0/53147114 wait complete. 2024-03-04T09:27:51.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.023+0000 7fbe04a00640 1 Processor -- start 2024-03-04T09:27:51.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.023+0000 7fbe04a00640 1 -- start start 2024-03-04T09:27:51.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.024+0000 7fbe04a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbe000b2080 0x7fbe000ae290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:51.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.024+0000 7fbe04a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbe00145430 0x7fbe000cc3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:51.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.024+0000 7fbe04a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbe0015f810 0x7fbe00142170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:51.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.024+0000 7fbe04a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe00145990 con 0x7fbe00145430 2024-03-04T09:27:51.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.024+0000 7fbe04a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe00145810 con 0x7fbe0015f810 2024-03-04T09:27:51.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.024+0000 7fbe04a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe000cde00 con 0x7fbe000b2080 2024-03-04T09:27:51.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.024+0000 7fbdfea00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbe00145430 0x7fbe000cc3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:51.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.024+0000 7fbdfea00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbe00145430 0x7fbe000cc3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42276/0 (socket says 172.21.15.6:42276) 2024-03-04T09:27:51.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.024+0000 7fbdfea00640 1 -- 172.21.15.6:0/3096887973 learned_addr learned my addr 172.21.15.6:0/3096887973 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:27:51.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.025+0000 7fbdfea00640 1 -- 172.21.15.6:0/3096887973 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbe000b2080 msgr2=0x7fbe000ae290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:51.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.025+0000 7fbdfea00640 1 --2- 172.21.15.6:0/3096887973 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbe000b2080 0x7fbe000ae290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:51.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.025+0000 7fbdfea00640 1 -- 172.21.15.6:0/3096887973 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbe0015f810 msgr2=0x7fbe00142170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:51.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.025+0000 7fbdfea00640 1 --2- 172.21.15.6:0/3096887973 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbe0015f810 0x7fbe00142170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:51.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.025+0000 7fbdfea00640 1 -- 172.21.15.6:0/3096887973 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe000711a0 con 0x7fbe00145430 2024-03-04T09:27:51.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.025+0000 7fbdfea00640 1 --2- 172.21.15.6:0/3096887973 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbe00145430 0x7fbe000cc3f0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fbdec0090b0 tx=0x7fbdec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:51.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.025+0000 7fbdf7400640 1 -- 172.21.15.6:0/3096887973 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdec0443f0 con 0x7fbe00145430 2024-03-04T09:27:51.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.025+0000 7fbdf7400640 1 -- 172.21.15.6:0/3096887973 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdec0445b0 con 0x7fbe00145430 2024-03-04T09:27:51.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.025+0000 7fbdf7400640 1 -- 172.21.15.6:0/3096887973 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdec0449a0 con 0x7fbe00145430 2024-03-04T09:27:51.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.025+0000 7fbe04a00640 1 -- 172.21.15.6:0/3096887973 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe001426b0 con 0x7fbe00145430 2024-03-04T09:27:51.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.026+0000 7fbe04a00640 1 -- 172.21.15.6:0/3096887973 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe000aeb50 con 0x7fbe00145430 2024-03-04T09:27:51.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.027+0000 7fbe04a00640 1 -- 172.21.15.6:0/3096887973 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe0004f330 con 0x7fbe00145430 2024-03-04T09:27:51.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.028+0000 7fbdf7400640 1 -- 172.21.15.6:0/3096887973 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbdec038020 con 0x7fbe00145430 2024-03-04T09:27:51.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.029+0000 7fbdf7400640 1 --2- 172.21.15.6:0/3096887973 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fbde006b4d0 0x7fbde006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:51.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.029+0000 7fbdf7400640 1 -- 172.21.15.6:0/3096887973 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fbdec0ac290 con 0x7fbe00145430 2024-03-04T09:27:51.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.029+0000 7fbdff400640 1 --2- 172.21.15.6:0/3096887973 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fbde006b4d0 0x7fbde006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:51.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.030+0000 7fbdff400640 1 --2- 172.21.15.6:0/3096887973 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fbde006b4d0 0x7fbde006d990 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fbdf0001560 tx=0x7fbdf0007990 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:51.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.033+0000 7fbdf7400640 1 -- 172.21.15.6:0/3096887973 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbdec03f070 con 0x7fbe00145430 2024-03-04T09:27:51.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.230+0000 7fbe04a00640 1 -- 172.21.15.6:0/3096887973 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fbe00140990 con 0x7fbe00145430 2024-03-04T09:27:51.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.232+0000 7fbdf7400640 1 -- 172.21.15.6:0/3096887973 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v47) v1 ==== 93+0+10 (secure 0 0 0) 0x7fbdec07b370 con 0x7fbe00145430 2024-03-04T09:27:51.228 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-04T09:27:51.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.234+0000 7fbe04a00640 1 -- 172.21.15.6:0/3096887973 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fbde006b4d0 msgr2=0x7fbde006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:51.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.234+0000 7fbe04a00640 1 --2- 172.21.15.6:0/3096887973 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fbde006b4d0 0x7fbde006d990 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fbdf0001560 tx=0x7fbdf0007990 comp rx=0 tx=0).stop 2024-03-04T09:27:51.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.234+0000 7fbe04a00640 1 -- 172.21.15.6:0/3096887973 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbe00145430 msgr2=0x7fbe000cc3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:51.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.234+0000 7fbe04a00640 1 --2- 172.21.15.6:0/3096887973 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbe00145430 0x7fbe000cc3f0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fbdec0090b0 tx=0x7fbdec0046c0 comp rx=0 tx=0).stop 2024-03-04T09:27:51.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.234+0000 7fbe04a00640 1 -- 172.21.15.6:0/3096887973 shutdown_connections 2024-03-04T09:27:51.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.234+0000 7fbe04a00640 1 --2- 172.21.15.6:0/3096887973 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbe0015f810 0x7fbe00142170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:51.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.234+0000 7fbe04a00640 1 --2- 172.21.15.6:0/3096887973 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbe00145430 0x7fbe000cc3f0 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:51.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.234+0000 7fbe04a00640 1 --2- 172.21.15.6:0/3096887973 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fbde006b4d0 0x7fbde006d990 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:51.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.234+0000 7fbe04a00640 1 --2- 172.21.15.6:0/3096887973 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbe000b2080 0x7fbe000ae290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:51.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.234+0000 7fbe04a00640 1 -- 172.21.15.6:0/3096887973 >> 172.21.15.6:0/3096887973 conn(0x7fbe0006b3a0 msgr2=0x7fbe0015e780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:51.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.235+0000 7fbe04a00640 1 -- 172.21.15.6:0/3096887973 shutdown_connections 2024-03-04T09:27:51.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.235+0000 7fbe04a00640 1 -- 172.21.15.6:0/3096887973 wait complete. 2024-03-04T09:27:51.252 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_fragmented' 2024-03-04T09:27:51.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.470+0000 7f97e3200640 1 Processor -- start 2024-03-04T09:27:51.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.470+0000 7f97e3200640 1 -- start start 2024-03-04T09:27:51.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.472+0000 7f97e3200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f97e4145300 0x7f97e40b49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:51.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.472+0000 7f97e3200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f97e4070f10 0x7f97e40cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:51.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.472+0000 7f97e3200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97e40714d0 con 0x7f97e40cce90 2024-03-04T09:27:51.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.472+0000 7f97e3200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97e40cd300 con 0x7f97e4070f10 2024-03-04T09:27:51.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.472+0000 7f97e3200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97e40717b0 con 0x7f97e4145300 2024-03-04T09:27:51.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.472+0000 7f97e1400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f97e4070f10 0x7f97e40cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:51.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.472+0000 7f97e1400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f97e4070f10 0x7f97e40cc950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42290/0 (socket says 172.21.15.6:42290) 2024-03-04T09:27:51.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.472+0000 7f97e1400640 1 -- 172.21.15.6:0/236265319 learned_addr learned my addr 172.21.15.6:0/236265319 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:27:51.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.472+0000 7f97e2800640 1 --1- 172.21.15.6:0/236265319 >> v1:172.21.15.6:6789/0 conn(0x7f97e40cce90 0x7f97e40cd930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:58894/0 (socket says 172.21.15.6:58894) 2024-03-04T09:27:51.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.473+0000 7f97e1e00640 1 --2- 172.21.15.6:0/236265319 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f97e4145300 0x7f97e40b49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:51.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.473+0000 7f97e1400640 1 -- 172.21.15.6:0/236265319 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f97e4145300 msgr2=0x7f97e40b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:27:51.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.473+0000 7f97e1400640 1 --2- 172.21.15.6:0/236265319 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f97e4145300 0x7f97e40b49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:51.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.473+0000 7f97e1400640 1 -- 172.21.15.6:0/236265319 >> v1:172.21.15.6:6789/0 conn(0x7f97e40cce90 legacy=0x7f97e40cd930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:51.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.473+0000 7f97e0a00640 1 -- 172.21.15.6:0/236265319 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2954220444 0 0) 0x7f97e40714d0 con 0x7f97e40cce90 2024-03-04T09:27:51.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.473+0000 7f97e1400640 1 -- 172.21.15.6:0/236265319 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97e4071a90 con 0x7f97e4070f10 2024-03-04T09:27:51.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.474+0000 7f97e1400640 1 --2- 172.21.15.6:0/236265319 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f97e4070f10 0x7f97e40cc950 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f97d0009870 tx=0x7f97d002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fd4d1a77acea967 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:51.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.474+0000 7f97e0a00640 1 -- 172.21.15.6:0/236265319 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97d0034070 con 0x7f97e4070f10 2024-03-04T09:27:51.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.474+0000 7f97e0a00640 1 -- 172.21.15.6:0/236265319 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97d0030840 con 0x7f97e4070f10 2024-03-04T09:27:51.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.475+0000 7f97e0a00640 1 -- 172.21.15.6:0/236265319 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97d0030c10 con 0x7f97e4070f10 2024-03-04T09:27:51.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.475+0000 7f97e3200640 1 -- 172.21.15.6:0/236265319 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f97e4070f10 msgr2=0x7f97e40cc950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:51.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.475+0000 7f97e3200640 1 --2- 172.21.15.6:0/236265319 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f97e4070f10 0x7f97e40cc950 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f97d0009870 tx=0x7f97d002b980 comp rx=0 tx=0).stop 2024-03-04T09:27:51.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.475+0000 7f97e3200640 1 -- 172.21.15.6:0/236265319 shutdown_connections 2024-03-04T09:27:51.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.475+0000 7f97e3200640 1 --2- 172.21.15.6:0/236265319 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f97e4070f10 0x7f97e40cc950 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:51.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.475+0000 7f97e3200640 1 --2- 172.21.15.6:0/236265319 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f97e4145300 0x7f97e40b49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:51.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.475+0000 7f97e3200640 1 -- 172.21.15.6:0/236265319 >> 172.21.15.6:0/236265319 conn(0x7f97e406b3d0 msgr2=0x7f97e406b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:51.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.475+0000 7f97e3200640 1 -- 172.21.15.6:0/236265319 shutdown_connections 2024-03-04T09:27:51.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.475+0000 7f97e3200640 1 -- 172.21.15.6:0/236265319 wait complete. 2024-03-04T09:27:51.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.477+0000 7f97e3200640 1 Processor -- start 2024-03-04T09:27:51.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.477+0000 7f97e3200640 1 -- start start 2024-03-04T09:27:51.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.478+0000 7f97e3200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f97e4070f10 0x7f97e41e8030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:51.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.478+0000 7f97e3200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f97e40cce90 0x7f97e415b330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:51.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.478+0000 7f97e3200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f97e4145300 0x7f97e41e6690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:51.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.478+0000 7f97e3200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97e41456e0 con 0x7f97e4145300 2024-03-04T09:27:51.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.478+0000 7f97e3200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97e40b4ee0 con 0x7f97e40cce90 2024-03-04T09:27:51.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.478+0000 7f97e3200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97e415d580 con 0x7f97e4070f10 2024-03-04T09:27:51.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.478+0000 7f97e2800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f97e4145300 0x7f97e41e6690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:51.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.478+0000 7f97e2800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f97e4145300 0x7f97e41e6690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42294/0 (socket says 172.21.15.6:42294) 2024-03-04T09:27:51.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.478+0000 7f97e2800640 1 -- 172.21.15.6:0/2522968505 learned_addr learned my addr 172.21.15.6:0/2522968505 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:27:51.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.479+0000 7f97e1e00640 1 --2- 172.21.15.6:0/2522968505 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f97e4070f10 0x7f97e41e8030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:51.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.479+0000 7f97e2800640 1 -- 172.21.15.6:0/2522968505 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f97e4070f10 msgr2=0x7f97e41e8030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:51.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.479+0000 7f97e1400640 1 --2- 172.21.15.6:0/2522968505 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f97e40cce90 0x7f97e415b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:51.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.479+0000 7f97e2800640 1 --2- 172.21.15.6:0/2522968505 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f97e4070f10 0x7f97e41e8030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:51.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.479+0000 7f97e2800640 1 -- 172.21.15.6:0/2522968505 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f97e40cce90 msgr2=0x7f97e415b330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:51.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.479+0000 7f97e2800640 1 --2- 172.21.15.6:0/2522968505 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f97e40cce90 0x7f97e415b330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:51.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.479+0000 7f97e2800640 1 -- 172.21.15.6:0/2522968505 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97e41e6f90 con 0x7f97e4145300 2024-03-04T09:27:51.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.479+0000 7f97e2800640 1 --2- 172.21.15.6:0/2522968505 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f97e4145300 0x7f97e41e6690 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f97dc00c7a0 tx=0x7f97dc00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:51.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.479+0000 7f97da000640 1 -- 172.21.15.6:0/2522968505 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97dc018070 con 0x7f97e4145300 2024-03-04T09:27:51.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.479+0000 7f97da000640 1 -- 172.21.15.6:0/2522968505 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97dc00a640 con 0x7f97e4145300 2024-03-04T09:27:51.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.479+0000 7f97da000640 1 -- 172.21.15.6:0/2522968505 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97dc00aab0 con 0x7f97e4145300 2024-03-04T09:27:51.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.480+0000 7f97e3200640 1 -- 172.21.15.6:0/2522968505 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97e41e7270 con 0x7f97e4145300 2024-03-04T09:27:51.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.480+0000 7f97e3200640 1 -- 172.21.15.6:0/2522968505 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97e41e7830 con 0x7f97e4145300 2024-03-04T09:27:51.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.481+0000 7f97e3200640 1 -- 172.21.15.6:0/2522968505 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97e404f330 con 0x7f97e4145300 2024-03-04T09:27:51.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.482+0000 7f97da000640 1 -- 172.21.15.6:0/2522968505 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f97dc00ac50 con 0x7f97e4145300 2024-03-04T09:27:51.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.483+0000 7f97da000640 1 --2- 172.21.15.6:0/2522968505 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f97c406b500 0x7f97c406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:51.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.483+0000 7f97da000640 1 -- 172.21.15.6:0/2522968505 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f97dc089490 con 0x7f97e4145300 2024-03-04T09:27:51.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.483+0000 7f97e1e00640 1 --2- 172.21.15.6:0/2522968505 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f97c406b500 0x7f97c406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:51.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.485+0000 7f97e1e00640 1 --2- 172.21.15.6:0/2522968505 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f97c406b500 0x7f97c406d9c0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f97d4006fd0 tx=0x7f97d4008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:51.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.487+0000 7f97da000640 1 -- 172.21.15.6:0/2522968505 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f97dc058600 con 0x7f97e4145300 2024-03-04T09:27:51.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:51.759+0000 7f97e3200640 1 -- 172.21.15.6:0/2522968505 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_fragmented"]} v 0) v1 -- 0x7f97e4074ab0 con 0x7f97e4145300 2024-03-04T09:27:52.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:52.301+0000 7f97da000640 1 -- 172.21.15.6:0/2522968505 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_fragmented"]}]=0 v158) v1 ==== 147+0+0 (secure 0 0 0) 0x7f97dc057f00 con 0x7f97e4145300 2024-03-04T09:27:52.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:52.305+0000 7f97e3200640 1 -- 172.21.15.6:0/2522968505 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f97c406b500 msgr2=0x7f97c406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:52.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:52.305+0000 7f97e3200640 1 --2- 172.21.15.6:0/2522968505 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f97c406b500 0x7f97c406d9c0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f97d4006fd0 tx=0x7f97d4008040 comp rx=0 tx=0).stop 2024-03-04T09:27:52.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:52.305+0000 7f97e3200640 1 -- 172.21.15.6:0/2522968505 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f97e4145300 msgr2=0x7f97e41e6690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:52.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:52.305+0000 7f97e3200640 1 --2- 172.21.15.6:0/2522968505 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f97e4145300 0x7f97e41e6690 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f97dc00c7a0 tx=0x7f97dc00cc60 comp rx=0 tx=0).stop 2024-03-04T09:27:52.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:52.305+0000 7f97e3200640 1 -- 172.21.15.6:0/2522968505 shutdown_connections 2024-03-04T09:27:52.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:52.305+0000 7f97e3200640 1 --2- 172.21.15.6:0/2522968505 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f97e4145300 0x7f97e41e6690 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:52.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:52.305+0000 7f97e3200640 1 --2- 172.21.15.6:0/2522968505 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f97e40cce90 0x7f97e415b330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:52.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:52.305+0000 7f97e3200640 1 --2- 172.21.15.6:0/2522968505 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f97c406b500 0x7f97c406d9c0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:52.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:52.305+0000 7f97e3200640 1 --2- 172.21.15.6:0/2522968505 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f97e4070f10 0x7f97e41e8030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:52.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:52.305+0000 7f97e3200640 1 -- 172.21.15.6:0/2522968505 >> 172.21.15.6:0/2522968505 conn(0x7f97e406b3d0 msgr2=0x7f97e40b2860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:52.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:52.305+0000 7f97e3200640 1 -- 172.21.15.6:0/2522968505 shutdown_connections 2024-03-04T09:27:52.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:52.305+0000 7f97e3200640 1 -- 172.21.15.6:0/2522968505 wait complete. 2024-03-04T09:27:52.324 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-04T09:27:52.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:52.556+0000 7f4f0ae00640 1 Processor -- start 2024-03-04T09:27:52.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:52.556+0000 7f4f0ae00640 1 -- start start 2024-03-04T09:27:52.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:52.557+0000 7f4f0ae00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f4f0406da40 0x7f4f0406e340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:52.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:52.557+0000 7f4f0ae00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4f0415e680 0x7f4f0406c980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:52.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:52.557+0000 7f4f0ae00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f4f040ccad0 0x7f4f040ccf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:52.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:52.557+0000 7f4f0ae00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f04161c70 con 0x7f4f0406da40 2024-03-04T09:27:52.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:52.557+0000 7f4f0ae00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f0415eb80 con 0x7f4f040ccad0 2024-03-04T09:27:52.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:52.557+0000 7f4f0ae00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f0415ee60 con 0x7f4f0415e680 2024-03-04T09:27:52.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:52.558+0000 7f4f03e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f4f0406da40 0x7f4f0406e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:52.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:52.558+0000 7f4f03e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f4f0406da40 0x7f4f0406e340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42310/0 (socket says 172.21.15.6:42310) 2024-03-04T09:27:52.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:52.558+0000 7f4f03e00640 1 -- 172.21.15.6:0/170989557 learned_addr learned my addr 172.21.15.6:0/170989557 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:27:52.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:52.558+0000 7f4f08e00640 1 --2- 172.21.15.6:0/170989557 >> v2:172.21.15.146:3300/0 conn(0x7f4f040ccad0 0x7f4f040ccf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:52.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:52.558+0000 7f4f03400640 1 --2- 172.21.15.6:0/170989557 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4f0415e680 0x7f4f0406c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:52.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:52.558+0000 7f4f03e00640 1 -- 172.21.15.6:0/170989557 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4f0415e680 msgr2=0x7f4f0406c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:27:52.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:52.558+0000 7f4f03e00640 1 --2- 172.21.15.6:0/170989557 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4f0415e680 0x7f4f0406c980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:52.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:52.558+0000 7f4f03e00640 1 -- 172.21.15.6:0/170989557 >> v2:172.21.15.146:3300/0 conn(0x7f4f040ccad0 msgr2=0x7f4f040ccf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:27:52.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:52.558+0000 7f4f03e00640 1 --2- 172.21.15.6:0/170989557 >> v2:172.21.15.146:3300/0 conn(0x7f4f040ccad0 0x7f4f040ccf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:52.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:52.558+0000 7f4f03e00640 1 -- 172.21.15.6:0/170989557 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f0415f140 con 0x7f4f0406da40 2024-03-04T09:27:52.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:52.560+0000 7f4f03e00640 1 --2- 172.21.15.6:0/170989557 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4f0406da40 0x7f4f0406e340 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f4f0406c020 tx=0x7f4ef402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26d4719eef8de017 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:52.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:52.560+0000 7f4f02a00640 1 -- 172.21.15.6:0/170989557 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ef4035020 con 0x7f4f0406da40 2024-03-04T09:27:52.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:52.560+0000 7f4f02a00640 1 -- 172.21.15.6:0/170989557 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ef4031870 con 0x7f4f0406da40 2024-03-04T09:27:52.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:52.560+0000 7f4f02a00640 1 -- 172.21.15.6:0/170989557 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ef4031c40 con 0x7f4f0406da40 2024-03-04T09:27:52.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:52.561+0000 7f4f0ae00640 1 -- 172.21.15.6:0/170989557 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4f0406da40 msgr2=0x7f4f0406e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:52.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:52.561+0000 7f4f0ae00640 1 --2- 172.21.15.6:0/170989557 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4f0406da40 0x7f4f0406e340 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f4f0406c020 tx=0x7f4ef402b910 comp rx=0 tx=0).stop 2024-03-04T09:27:52.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:52.561+0000 7f4f0ae00640 1 -- 172.21.15.6:0/170989557 shutdown_connections 2024-03-04T09:27:52.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:52.561+0000 7f4f0ae00640 1 --2- 172.21.15.6:0/170989557 >> v2:172.21.15.146:3300/0 conn(0x7f4f040ccad0 0x7f4f040ccf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:52.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:52.561+0000 7f4f0ae00640 1 --2- 172.21.15.6:0/170989557 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4f0415e680 0x7f4f0406c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:52.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:52.561+0000 7f4f0ae00640 1 --2- 172.21.15.6:0/170989557 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4f0406da40 0x7f4f0406e340 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:52.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:52.561+0000 7f4f0ae00640 1 -- 172.21.15.6:0/170989557 >> 172.21.15.6:0/170989557 conn(0x7f4f0406b3d0 msgr2=0x7f4f0406b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:52.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:52.561+0000 7f4f0ae00640 1 -- 172.21.15.6:0/170989557 shutdown_connections 2024-03-04T09:27:52.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:52.561+0000 7f4f0ae00640 1 -- 172.21.15.6:0/170989557 wait complete. 2024-03-04T09:27:52.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:52.563+0000 7f4f0ae00640 1 Processor -- start 2024-03-04T09:27:52.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:52.563+0000 7f4f0ae00640 1 -- start start 2024-03-04T09:27:52.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:52.564+0000 7f4f0ae00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4f0406da40 0x7f4f0409b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:52.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:52.564+0000 7f4f0ae00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4f040ccad0 0x7f4f040ade20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:52.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:52.564+0000 7f4f0ae00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4f0415e680 0x7f4f040ae360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:52.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:52.564+0000 7f4f0ae00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f0406ea00 con 0x7f4f040ccad0 2024-03-04T09:27:52.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:52.564+0000 7f4f0ae00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f0406e880 con 0x7f4f0415e680 2024-03-04T09:27:52.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:52.564+0000 7f4f0ae00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f0406cec0 con 0x7f4f0406da40 2024-03-04T09:27:52.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:52.564+0000 7f4f03400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4f040ccad0 0x7f4f040ade20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:52.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:52.564+0000 7f4f03400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4f040ccad0 0x7f4f040ade20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42326/0 (socket says 172.21.15.6:42326) 2024-03-04T09:27:52.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:52.564+0000 7f4f03400640 1 -- 172.21.15.6:0/2238010896 learned_addr learned my addr 172.21.15.6:0/2238010896 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:27:52.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:52.564+0000 7f4f08e00640 1 --2- 172.21.15.6:0/2238010896 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4f0415e680 0x7f4f040ae360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:52.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:52.564+0000 7f4f03e00640 1 --2- 172.21.15.6:0/2238010896 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4f0406da40 0x7f4f0409b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:52.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:52.565+0000 7f4f03400640 1 -- 172.21.15.6:0/2238010896 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4f0406da40 msgr2=0x7f4f0409b9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:52.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:52.565+0000 7f4f03400640 1 --2- 172.21.15.6:0/2238010896 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4f0406da40 0x7f4f0409b9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:52.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:52.565+0000 7f4f03400640 1 -- 172.21.15.6:0/2238010896 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4f0415e680 msgr2=0x7f4f040ae360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:52.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:52.565+0000 7f4f03400640 1 --2- 172.21.15.6:0/2238010896 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4f0415e680 0x7f4f040ae360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:52.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:52.565+0000 7f4f03400640 1 -- 172.21.15.6:0/2238010896 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f040c8fe0 con 0x7f4f040ccad0 2024-03-04T09:27:52.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:52.565+0000 7f4f03e00640 1 --2- 172.21.15.6:0/2238010896 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4f0406da40 0x7f4f0409b9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:27:52.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:52.565+0000 7f4f03400640 1 --2- 172.21.15.6:0/2238010896 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4f040ccad0 0x7f4f040ade20 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f4ef000d5f0 tx=0x7f4ef000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:52.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:52.565+0000 7f4efbe00640 1 -- 172.21.15.6:0/2238010896 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ef00125f0 con 0x7f4f040ccad0 2024-03-04T09:27:52.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:52.565+0000 7f4efbe00640 1 -- 172.21.15.6:0/2238010896 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ef0019070 con 0x7f4f040ccad0 2024-03-04T09:27:52.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:52.565+0000 7f4efbe00640 1 -- 172.21.15.6:0/2238010896 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ef00129d0 con 0x7f4f040ccad0 2024-03-04T09:27:52.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:52.565+0000 7f4f0ae00640 1 -- 172.21.15.6:0/2238010896 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f0414dd40 con 0x7f4f040ccad0 2024-03-04T09:27:52.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:52.566+0000 7f4f0ae00640 1 -- 172.21.15.6:0/2238010896 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f040c9580 con 0x7f4f040ccad0 2024-03-04T09:27:52.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:52.567+0000 7f4f0ae00640 1 -- 172.21.15.6:0/2238010896 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f0404f330 con 0x7f4f040ccad0 2024-03-04T09:27:52.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:52.568+0000 7f4efbe00640 1 -- 172.21.15.6:0/2238010896 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4ef0012b70 con 0x7f4f040ccad0 2024-03-04T09:27:52.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:52.568+0000 7f4efbe00640 1 --2- 172.21.15.6:0/2238010896 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f4ee406b4d0 0x7f4ee406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:52.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:52.568+0000 7f4efbe00640 1 -- 172.21.15.6:0/2238010896 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f4ef008a3a0 con 0x7f4f040ccad0 2024-03-04T09:27:52.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:52.569+0000 7f4f03e00640 1 --2- 172.21.15.6:0/2238010896 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f4ee406b4d0 0x7f4ee406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:52.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:52.570+0000 7f4f03e00640 1 --2- 172.21.15.6:0/2238010896 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f4ee406b4d0 0x7f4ee406d990 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f4ef4036000 tx=0x7f4ef4008030 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:52.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:52.573+0000 7f4efbe00640 1 -- 172.21.15.6:0/2238010896 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4ef0059480 con 0x7f4f040ccad0 2024-03-04T09:27:52.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:52.775+0000 7f4f0ae00640 1 -- 172.21.15.6:0/2238010896 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f4f04074c00 con 0x7f4f040ccad0 2024-03-04T09:27:52.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:52.782+0000 7f4efbe00640 1 -- 172.21.15.6:0/2238010896 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v16)=0 v16) v1 ==== 122+0+0 (secure 0 0 0) 0x7f4ef0058d80 con 0x7f4f040ccad0 2024-03-04T09:27:52.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:52.785+0000 7f4f0ae00640 1 -- 172.21.15.6:0/2238010896 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f4ee406b4d0 msgr2=0x7f4ee406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:52.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:52.785+0000 7f4f0ae00640 1 --2- 172.21.15.6:0/2238010896 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f4ee406b4d0 0x7f4ee406d990 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f4ef4036000 tx=0x7f4ef4008030 comp rx=0 tx=0).stop 2024-03-04T09:27:52.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:52.785+0000 7f4f0ae00640 1 -- 172.21.15.6:0/2238010896 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4f040ccad0 msgr2=0x7f4f040ade20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:52.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:52.785+0000 7f4f0ae00640 1 --2- 172.21.15.6:0/2238010896 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4f040ccad0 0x7f4f040ade20 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f4ef000d5f0 tx=0x7f4ef000dab0 comp rx=0 tx=0).stop 2024-03-04T09:27:52.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:52.785+0000 7f4f0ae00640 1 -- 172.21.15.6:0/2238010896 shutdown_connections 2024-03-04T09:27:52.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:52.785+0000 7f4f0ae00640 1 --2- 172.21.15.6:0/2238010896 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4f0415e680 0x7f4f040ae360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:52.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:52.785+0000 7f4f0ae00640 1 --2- 172.21.15.6:0/2238010896 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4f040ccad0 0x7f4f040ade20 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:52.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:52.785+0000 7f4f0ae00640 1 --2- 172.21.15.6:0/2238010896 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f4ee406b4d0 0x7f4ee406d990 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:52.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:52.785+0000 7f4f0ae00640 1 --2- 172.21.15.6:0/2238010896 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4f0406da40 0x7f4f0409b9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:52.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:52.785+0000 7f4f0ae00640 1 -- 172.21.15.6:0/2238010896 >> 172.21.15.6:0/2238010896 conn(0x7f4f0406b3d0 msgr2=0x7f4f0415ca10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:52.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:52.786+0000 7f4f0ae00640 1 -- 172.21.15.6:0/2238010896 shutdown_connections 2024-03-04T09:27:52.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:52.786+0000 7f4f0ae00640 1 -- 172.21.15.6:0/2238010896 wait complete. 2024-03-04T09:27:52.802 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-04T09:27:52.802 DEBUG:teuthology.orchestra.run.smithi006:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-04T09:27:52.927 DEBUG:teuthology.orchestra.run.smithi146:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-04T09:27:53.056 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-04T09:27:53.057 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-04T09:27:53.057 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T09:27:53.121 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054202,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249800k,nr_inodes=812450,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-04T09:27:53.121 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-04T09:27:53.121 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T09:27:53.179 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053429,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249180k,nr_inodes=812295,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-04T09:27:53.179 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-04T09:27:53.179 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T09:27:53.242 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054202,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249800k,nr_inodes=812450,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-04T09:27:53.242 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-04T09:27:53.242 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T09:27:53.300 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053429,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249180k,nr_inodes=812295,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-04T09:27:53.301 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-04T09:27:53.301 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:27:53.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:53.539+0000 7f067f200640 1 Processor -- start 2024-03-04T09:27:53.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:53.540+0000 7f067f200640 1 -- start start 2024-03-04T09:27:53.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:53.541+0000 7f067f200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f06800b1280 0x7f068006cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:53.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:53.541+0000 7f067f200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06800b6270 con 0x7f068015df00 2024-03-04T09:27:53.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:53.541+0000 7f067f200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06800b6510 con 0x7f06800b0d30 2024-03-04T09:27:53.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:53.541+0000 7f067f200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06800aeb00 con 0x7f06800b1280 2024-03-04T09:27:53.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:53.542+0000 7f067de00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f068015df00 0x7f068006c880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:58896/0 (socket says 172.21.15.6:58896) 2024-03-04T09:27:53.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:53.542+0000 7f067de00640 1 -- 172.21.15.6:0/3322754194 learned_addr learned my addr 172.21.15.6:0/3322754194 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:27:53.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:53.542+0000 7f0675400640 1 --2- 172.21.15.6:0/3322754194 >> v2:172.21.15.6:3300/0 conn(0x7f06800b1280 0x7f068006cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:53.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:53.543+0000 7f0675400640 1 -- 172.21.15.6:0/3322754194 >> v1:172.21.15.146:6789/0 conn(0x7f06800b0d30 legacy=0x7f06800af270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:53.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:53.543+0000 7f0675400640 1 -- 172.21.15.6:0/3322754194 >> v1:172.21.15.6:6789/0 conn(0x7f068015df00 legacy=0x7f068006c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:53.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:53.543+0000 7f0675400640 1 -- 172.21.15.6:0/3322754194 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06800b67f0 con 0x7f06800b1280 2024-03-04T09:27:53.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:53.543+0000 7f067d400640 1 -- 172.21.15.6:0/3322754194 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3949514480 0 0) 0x7f06800b6270 con 0x7f068015df00 2024-03-04T09:27:53.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:53.544+0000 7f0675400640 1 --2- 172.21.15.6:0/3322754194 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f06800b1280 0x7f068006cf90 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f06680097a0 tx=0x7f066802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e333826708b55da server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:53.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:53.544+0000 7f067d400640 1 -- 172.21.15.6:0/3322754194 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0668035020 con 0x7f06800b1280 2024-03-04T09:27:53.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:53.544+0000 7f067d400640 1 -- 172.21.15.6:0/3322754194 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0668031830 con 0x7f06800b1280 2024-03-04T09:27:53.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:53.544+0000 7f067d400640 1 -- 172.21.15.6:0/3322754194 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0668031c00 con 0x7f06800b1280 2024-03-04T09:27:53.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:53.545+0000 7f067f200640 1 -- 172.21.15.6:0/3322754194 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f06800b1280 msgr2=0x7f068006cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:53.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:53.545+0000 7f067f200640 1 --2- 172.21.15.6:0/3322754194 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f06800b1280 0x7f068006cf90 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f06680097a0 tx=0x7f066802b8d0 comp rx=0 tx=0).stop 2024-03-04T09:27:53.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:53.545+0000 7f067f200640 1 -- 172.21.15.6:0/3322754194 shutdown_connections 2024-03-04T09:27:53.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:53.545+0000 7f067f200640 1 --2- 172.21.15.6:0/3322754194 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f06800b1280 0x7f068006cf90 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:53.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:53.545+0000 7f067f200640 1 -- 172.21.15.6:0/3322754194 >> 172.21.15.6:0/3322754194 conn(0x7f068009aa10 msgr2=0x7f068009ae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:53.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:53.545+0000 7f067f200640 1 -- 172.21.15.6:0/3322754194 shutdown_connections 2024-03-04T09:27:53.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:53.545+0000 7f067f200640 1 -- 172.21.15.6:0/3322754194 wait complete. 2024-03-04T09:27:53.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:53.547+0000 7f067f200640 1 Processor -- start 2024-03-04T09:27:53.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:53.547+0000 7f067f200640 1 -- start start 2024-03-04T09:27:53.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:53.548+0000 7f067f200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f06800b0d30 0x7f06801f3dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:53.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:53.548+0000 7f067f200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f06800b1280 0x7f0680143430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:53.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:53.548+0000 7f067f200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f068015df00 0x7f06800b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:53.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:53.548+0000 7f067f200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f068015e460 con 0x7f06800b0d30 2024-03-04T09:27:53.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:53.548+0000 7f067f200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f068015e2e0 con 0x7f06800b1280 2024-03-04T09:27:53.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:53.548+0000 7f067f200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06800af980 con 0x7f068015df00 2024-03-04T09:27:53.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:53.548+0000 7f067de00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f06800b0d30 0x7f06801f3dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:53.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:53.548+0000 7f067de00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f06800b0d30 0x7f06801f3dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42344/0 (socket says 172.21.15.6:42344) 2024-03-04T09:27:53.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:53.548+0000 7f067de00640 1 -- 172.21.15.6:0/4291076298 learned_addr learned my addr 172.21.15.6:0/4291076298 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:27:53.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:53.549+0000 7f067e800640 1 --2- 172.21.15.6:0/4291076298 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f068015df00 0x7f06800b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:53.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:53.549+0000 7f067de00640 1 -- 172.21.15.6:0/4291076298 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f068015df00 msgr2=0x7f06800b2090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:53.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:53.549+0000 7f0675400640 1 --2- 172.21.15.6:0/4291076298 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f06800b1280 0x7f0680143430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:53.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:53.549+0000 7f067de00640 1 --2- 172.21.15.6:0/4291076298 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f068015df00 0x7f06800b2090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:53.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:53.549+0000 7f067de00640 1 -- 172.21.15.6:0/4291076298 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f06800b1280 msgr2=0x7f0680143430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:53.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:53.549+0000 7f067de00640 1 --2- 172.21.15.6:0/4291076298 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f06800b1280 0x7f0680143430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:53.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:53.549+0000 7f067de00640 1 -- 172.21.15.6:0/4291076298 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06801f4c60 con 0x7f06800b0d30 2024-03-04T09:27:53.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:53.549+0000 7f0675400640 1 --2- 172.21.15.6:0/4291076298 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f06800b1280 0x7f0680143430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:27:53.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:53.549+0000 7f067e800640 1 --2- 172.21.15.6:0/4291076298 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f068015df00 0x7f06800b2090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:27:53.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:53.549+0000 7f067de00640 1 --2- 172.21.15.6:0/4291076298 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f06800b0d30 0x7f06801f3dc0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f067000c3f0 tx=0x7f067000c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:53.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:53.549+0000 7f0676a00640 1 -- 172.21.15.6:0/4291076298 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06700074e0 con 0x7f06800b0d30 2024-03-04T09:27:53.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:53.549+0000 7f0676a00640 1 -- 172.21.15.6:0/4291076298 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06700047e0 con 0x7f06800b0d30 2024-03-04T09:27:53.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:53.549+0000 7f067f200640 1 -- 172.21.15.6:0/4291076298 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06801f46a0 con 0x7f06800b0d30 2024-03-04T09:27:53.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:53.549+0000 7f0676a00640 1 -- 172.21.15.6:0/4291076298 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0670004c10 con 0x7f06800b0d30 2024-03-04T09:27:53.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:53.550+0000 7f067f200640 1 -- 172.21.15.6:0/4291076298 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06800b1890 con 0x7f06800b0d30 2024-03-04T09:27:53.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:53.551+0000 7f067f200640 1 -- 172.21.15.6:0/4291076298 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f068004f330 con 0x7f06800b0d30 2024-03-04T09:27:53.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:53.552+0000 7f0676a00640 1 -- 172.21.15.6:0/4291076298 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0670011070 con 0x7f06800b0d30 2024-03-04T09:27:53.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:53.553+0000 7f0676a00640 1 --2- 172.21.15.6:0/4291076298 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f065c06b4d0 0x7f065c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:53.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:53.553+0000 7f0676a00640 1 -- 172.21.15.6:0/4291076298 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f0670089ab0 con 0x7f06800b0d30 2024-03-04T09:27:53.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:53.553+0000 7f0675400640 1 --2- 172.21.15.6:0/4291076298 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f065c06b4d0 0x7f065c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:53.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:53.555+0000 7f0675400640 1 --2- 172.21.15.6:0/4291076298 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f065c06b4d0 0x7f065c06d990 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f0668036000 tx=0x7f0668008030 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:53.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:53.558+0000 7f0676a00640 1 -- 172.21.15.6:0/4291076298 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f067001a2b0 con 0x7f06800b0d30 2024-03-04T09:27:53.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:53.800+0000 7f067f200640 1 -- 172.21.15.6:0/4291076298 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0680157bb0 con 0x7f06800b0d30 2024-03-04T09:27:53.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:53.801+0000 7f0676a00640 1 -- 172.21.15.6:0/4291076298 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+3693 (secure 0 0 0) 0x7f0670058c20 con 0x7f06800b0d30 2024-03-04T09:27:53.799 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:27:53.799 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":27,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4913,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/1049631914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":1049631914},{"type":"v1","addr":"172.21.15.146:6835","nonce":1049631914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":5009,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/897363429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":897363429},{"type":"v1","addr":"172.21.15.146:6837","nonce":897363429}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":27},{"gid":5092,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/424062551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":424062551},{"type":"v1","addr":"172.21.15.6:6833","nonce":424062551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":19},{"gid":5095,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/3300873502","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":3300873502},{"type":"v1","addr":"172.21.15.6:6835","nonce":3300873502}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20}],"filesystems":[]} 2024-03-04T09:27:53.799 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 27 2024-03-04T09:27:53.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:53.804+0000 7f067f200640 1 -- 172.21.15.6:0/4291076298 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f065c06b4d0 msgr2=0x7f065c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:53.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:53.804+0000 7f067f200640 1 --2- 172.21.15.6:0/4291076298 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f065c06b4d0 0x7f065c06d990 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f0668036000 tx=0x7f0668008030 comp rx=0 tx=0).stop 2024-03-04T09:27:53.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:53.804+0000 7f067f200640 1 -- 172.21.15.6:0/4291076298 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f06800b0d30 msgr2=0x7f06801f3dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:53.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:53.804+0000 7f067f200640 1 --2- 172.21.15.6:0/4291076298 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f06800b0d30 0x7f06801f3dc0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f067000c3f0 tx=0x7f067000c8b0 comp rx=0 tx=0).stop 2024-03-04T09:27:53.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:53.804+0000 7f067f200640 1 -- 172.21.15.6:0/4291076298 shutdown_connections 2024-03-04T09:27:53.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:53.804+0000 7f067f200640 1 --2- 172.21.15.6:0/4291076298 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f065c06b4d0 0x7f065c06d990 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:53.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:53.804+0000 7f067f200640 1 --2- 172.21.15.6:0/4291076298 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f068015df00 0x7f06800b2090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:53.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:53.804+0000 7f067f200640 1 --2- 172.21.15.6:0/4291076298 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f06800b1280 0x7f0680143430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:53.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:53.804+0000 7f067f200640 1 --2- 172.21.15.6:0/4291076298 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f06800b0d30 0x7f06801f3dc0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:53.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:53.804+0000 7f067f200640 1 -- 172.21.15.6:0/4291076298 >> 172.21.15.6:0/4291076298 conn(0x7f068009aa10 msgr2=0x7f0680070140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:53.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:53.805+0000 7f067f200640 1 -- 172.21.15.6:0/4291076298 shutdown_connections 2024-03-04T09:27:53.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:53.805+0000 7f067f200640 1 -- 172.21.15.6:0/4291076298 wait complete. 2024-03-04T09:27:53.820 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-04T09:27:53.820 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-04T09:27:53.821 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-04T09:27:53.821 INFO:teuthology.orchestra.run:waiting for 300 2024-03-04T09:27:53.821 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-04T09:27:53.822 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-04T09:27:53.822 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-04T09:27:53.822 INFO:teuthology.orchestra.run:waiting for 300 2024-03-04T09:27:53.822 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-04T09:27:53.822 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-04T09:27:53.823 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-04T09:27:53.823 INFO:teuthology.orchestra.run:waiting for 300 2024-03-04T09:27:53.823 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-04T09:27:53.823 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-04T09:27:53.824 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-04T09:27:53.824 INFO:teuthology.orchestra.run:waiting for 300 2024-03-04T09:27:53.846 INFO:tasks.ceph.mds.b:Stopped 2024-03-04T09:27:53.846 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-04T09:27:53.850 INFO:tasks.ceph.mds.d:Stopped 2024-03-04T09:27:53.850 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-04T09:27:53.852 INFO:tasks.ceph.mds.a:Stopped 2024-03-04T09:27:53.852 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-04T09:27:53.853 INFO:tasks.ceph.mds.c:Stopped 2024-03-04T09:27:53.853 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-04T09:27:53.899 INFO:tasks.ceph.mds.b:Started 2024-03-04T09:27:53.900 DEBUG:teuthology.parallel:result is None 2024-03-04T09:27:53.902 INFO:tasks.ceph.mds.a:Started 2024-03-04T09:27:53.902 DEBUG:teuthology.parallel:result is None 2024-03-04T09:27:53.906 INFO:tasks.ceph.mds.d:Started 2024-03-04T09:27:53.906 DEBUG:teuthology.parallel:result is None 2024-03-04T09:27:53.909 INFO:tasks.ceph.mds.c:Started 2024-03-04T09:27:53.909 DEBUG:teuthology.parallel:result is None 2024-03-04T09:27:53.910 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-04T09:27:54.062 INFO:tasks.ceph.mds.c.smithi006.stdout:starting mds.c at 2024-03-04T09:27:54.062 INFO:tasks.ceph.mds.a.smithi006.stdout:starting mds.a at 2024-03-04T09:27:54.066 INFO:tasks.ceph.mds.b.smithi146.stdout:starting mds.b at 2024-03-04T09:27:54.067 INFO:tasks.ceph.mds.d.smithi146.stdout:starting mds.d at 2024-03-04T09:27:54.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:54.102+0000 7f7833c00640 1 Processor -- start 2024-03-04T09:27:54.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:54.102+0000 7f7833c00640 1 -- start start 2024-03-04T09:27:54.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:54.104+0000 7f7833c00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f782c070ee0 0x7f782c162c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:54.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:54.104+0000 7f7833c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f782c072da0 0x7f782c0cf130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:54.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:54.104+0000 7f7833c00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f782c0ce090 con 0x7f782c0712c0 2024-03-04T09:27:54.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:54.104+0000 7f7833c00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f782c0ceca0 con 0x7f782c070ee0 2024-03-04T09:27:54.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:54.104+0000 7f7833c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f782c0ce370 con 0x7f782c072da0 2024-03-04T09:27:54.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:54.104+0000 7f7831c00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f782c0712c0 0x7f782c06e0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:58924/0 (socket says 172.21.15.6:58924) 2024-03-04T09:27:54.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:54.104+0000 7f7831c00640 1 -- 172.21.15.6:0/2003256588 learned_addr learned my addr 172.21.15.6:0/2003256588 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:27:54.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:54.104+0000 7f7831200640 1 --2- 172.21.15.6:0/2003256588 >> v2:172.21.15.146:3300/0 conn(0x7f782c070ee0 0x7f782c162c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:54.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:54.104+0000 7f7823e00640 1 --2- 172.21.15.6:0/2003256588 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f782c072da0 0x7f782c0cf130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:54.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:54.105+0000 7f782be00640 1 -- 172.21.15.6:0/2003256588 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2771765819 0 0) 0x7f782c0ce090 con 0x7f782c0712c0 2024-03-04T09:27:54.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:54.105+0000 7f782be00640 1 -- 172.21.15.6:0/2003256588 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7810003520 con 0x7f782c0712c0 2024-03-04T09:27:54.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:54.105+0000 7f7823e00640 1 -- 172.21.15.6:0/2003256588 >> v2:172.21.15.146:3300/0 conn(0x7f782c070ee0 msgr2=0x7f782c162c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:27:54.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:54.105+0000 7f7823e00640 1 --2- 172.21.15.6:0/2003256588 >> v2:172.21.15.146:3300/0 conn(0x7f782c070ee0 0x7f782c162c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:54.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:54.105+0000 7f7823e00640 1 -- 172.21.15.6:0/2003256588 >> v1:172.21.15.6:6789/0 conn(0x7f782c0712c0 legacy=0x7f782c06e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:54.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:54.105+0000 7f7823e00640 1 -- 172.21.15.6:0/2003256588 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f782c0ce650 con 0x7f782c072da0 2024-03-04T09:27:54.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:54.106+0000 7f7823e00640 1 --2- 172.21.15.6:0/2003256588 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f782c072da0 0x7f782c0cf130 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f78140097a0 tx=0x7f781402b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=426d61b92ca3c5c server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:54.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:54.106+0000 7f782be00640 1 -- 172.21.15.6:0/2003256588 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7814035020 con 0x7f782c072da0 2024-03-04T09:27:54.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:54.106+0000 7f782be00640 1 -- 172.21.15.6:0/2003256588 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7814031830 con 0x7f782c072da0 2024-03-04T09:27:54.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:54.106+0000 7f782be00640 1 -- 172.21.15.6:0/2003256588 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7814031c00 con 0x7f782c072da0 2024-03-04T09:27:54.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:54.107+0000 7f7833c00640 1 -- 172.21.15.6:0/2003256588 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f782c072da0 msgr2=0x7f782c0cf130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:54.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:54.107+0000 7f7833c00640 1 --2- 172.21.15.6:0/2003256588 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f782c072da0 0x7f782c0cf130 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f78140097a0 tx=0x7f781402b8d0 comp rx=0 tx=0).stop 2024-03-04T09:27:54.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:54.107+0000 7f7833c00640 1 -- 172.21.15.6:0/2003256588 shutdown_connections 2024-03-04T09:27:54.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:54.107+0000 7f7833c00640 1 --2- 172.21.15.6:0/2003256588 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f782c072da0 0x7f782c0cf130 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:54.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:54.107+0000 7f7833c00640 1 --2- 172.21.15.6:0/2003256588 >> v2:172.21.15.146:3300/0 conn(0x7f782c070ee0 0x7f782c162c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:54.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:54.107+0000 7f7833c00640 1 -- 172.21.15.6:0/2003256588 >> 172.21.15.6:0/2003256588 conn(0x7f782c06afb0 msgr2=0x7f782c06b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:54.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:54.107+0000 7f7833c00640 1 -- 172.21.15.6:0/2003256588 shutdown_connections 2024-03-04T09:27:54.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:54.107+0000 7f7833c00640 1 -- 172.21.15.6:0/2003256588 wait complete. 2024-03-04T09:27:54.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:54.109+0000 7f7833c00640 1 Processor -- start 2024-03-04T09:27:54.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:54.109+0000 7f7833c00640 1 -- start start 2024-03-04T09:27:54.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:54.110+0000 7f7833c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f782c070ee0 0x7f782c153400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:54.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:54.110+0000 7f7833c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f782c0712c0 0x7f782c15a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:54.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:54.110+0000 7f7833c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f782c072da0 0x7f782c15f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:54.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:54.110+0000 7f7833c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f782c0cdda0 con 0x7f782c070ee0 2024-03-04T09:27:54.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:54.110+0000 7f7833c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f782c0cdc20 con 0x7f782c072da0 2024-03-04T09:27:54.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:54.110+0000 7f7833c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f782c06e7d0 con 0x7f782c0712c0 2024-03-04T09:27:54.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:54.110+0000 7f7831200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f782c070ee0 0x7f782c153400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:54.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:54.110+0000 7f7831200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f782c070ee0 0x7f782c153400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42402/0 (socket says 172.21.15.6:42402) 2024-03-04T09:27:54.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:54.110+0000 7f7831200640 1 -- 172.21.15.6:0/2226056700 learned_addr learned my addr 172.21.15.6:0/2226056700 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:27:54.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:54.110+0000 7f7823e00640 1 --2- 172.21.15.6:0/2226056700 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f782c0712c0 0x7f782c15a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:54.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:54.110+0000 7f7831c00640 1 --2- 172.21.15.6:0/2226056700 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f782c072da0 0x7f782c15f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:54.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:54.111+0000 7f7831200640 1 -- 172.21.15.6:0/2226056700 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f782c0712c0 msgr2=0x7f782c15a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:54.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:54.111+0000 7f7831200640 1 --2- 172.21.15.6:0/2226056700 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f782c0712c0 0x7f782c15a920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:54.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:54.111+0000 7f7831200640 1 -- 172.21.15.6:0/2226056700 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f782c072da0 msgr2=0x7f782c15f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:54.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:54.111+0000 7f7831200640 1 --2- 172.21.15.6:0/2226056700 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f782c072da0 0x7f782c15f9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:54.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:54.111+0000 7f7831200640 1 -- 172.21.15.6:0/2226056700 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f782c161410 con 0x7f782c070ee0 2024-03-04T09:27:54.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:54.111+0000 7f7831200640 1 --2- 172.21.15.6:0/2226056700 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f782c070ee0 0x7f782c153400 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f781c00e9b0 tx=0x7f781c00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:54.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:54.111+0000 7f7829600640 1 -- 172.21.15.6:0/2226056700 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f781c017070 con 0x7f782c070ee0 2024-03-04T09:27:54.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:54.111+0000 7f7829600640 1 -- 172.21.15.6:0/2226056700 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f781c00c9a0 con 0x7f782c070ee0 2024-03-04T09:27:54.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:54.111+0000 7f7833c00640 1 -- 172.21.15.6:0/2226056700 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f782c1616f0 con 0x7f782c070ee0 2024-03-04T09:27:54.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:54.111+0000 7f7829600640 1 -- 172.21.15.6:0/2226056700 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f781c00cd90 con 0x7f782c070ee0 2024-03-04T09:27:54.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:54.111+0000 7f7833c00640 1 -- 172.21.15.6:0/2226056700 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f782c0ae060 con 0x7f782c070ee0 2024-03-04T09:27:54.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:54.112+0000 7f7833c00640 1 -- 172.21.15.6:0/2226056700 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f782c069510 con 0x7f782c070ee0 2024-03-04T09:27:54.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:54.113+0000 7f7829600640 1 -- 172.21.15.6:0/2226056700 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f781c01a070 con 0x7f782c070ee0 2024-03-04T09:27:54.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:54.115+0000 7f7829600640 1 --2- 172.21.15.6:0/2226056700 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f780806b4d0 0x7f780806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:54.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:54.115+0000 7f7829600640 1 -- 172.21.15.6:0/2226056700 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f781c08a280 con 0x7f782c070ee0 2024-03-04T09:27:54.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:54.115+0000 7f7823e00640 1 --2- 172.21.15.6:0/2226056700 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f780806b4d0 0x7f780806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:54.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:54.116+0000 7f7823e00640 1 --2- 172.21.15.6:0/2226056700 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f780806b4d0 0x7f780806d990 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f7814036000 tx=0x7f781403c040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:54.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:54.120+0000 7f7829600640 1 -- 172.21.15.6:0/2226056700 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f781c058d10 con 0x7f782c070ee0 2024-03-04T09:27:54.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:54.334+0000 7f7833c00640 1 -- 172.21.15.6:0/2226056700 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f782c04f3a0 con 0x7f782c070ee0 2024-03-04T09:27:55.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:55.346+0000 7f7829600640 1 -- 172.21.15.6:0/2226056700 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v52) v1 ==== 97+0+0 (secure 0 0 0) 0x7f781c05cdb0 con 0x7f782c070ee0 2024-03-04T09:27:55.343 INFO:teuthology.orchestra.run.smithi006.stderr: removed all blocklist entries 2024-03-04T09:27:55.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:55.350+0000 7f7833c00640 1 -- 172.21.15.6:0/2226056700 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f780806b4d0 msgr2=0x7f780806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:55.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:55.350+0000 7f7833c00640 1 --2- 172.21.15.6:0/2226056700 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f780806b4d0 0x7f780806d990 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f7814036000 tx=0x7f781403c040 comp rx=0 tx=0).stop 2024-03-04T09:27:55.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:55.350+0000 7f7833c00640 1 -- 172.21.15.6:0/2226056700 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f782c070ee0 msgr2=0x7f782c153400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:55.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:55.350+0000 7f7833c00640 1 --2- 172.21.15.6:0/2226056700 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f782c070ee0 0x7f782c153400 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f781c00e9b0 tx=0x7f781c00ee70 comp rx=0 tx=0).stop 2024-03-04T09:27:55.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:55.350+0000 7f7833c00640 1 -- 172.21.15.6:0/2226056700 shutdown_connections 2024-03-04T09:27:55.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:55.350+0000 7f7833c00640 1 --2- 172.21.15.6:0/2226056700 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f782c072da0 0x7f782c15f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:55.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:55.350+0000 7f7833c00640 1 --2- 172.21.15.6:0/2226056700 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f780806b4d0 0x7f780806d990 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:55.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:55.350+0000 7f7833c00640 1 --2- 172.21.15.6:0/2226056700 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f782c0712c0 0x7f782c15a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:55.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:55.350+0000 7f7833c00640 1 --2- 172.21.15.6:0/2226056700 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f782c070ee0 0x7f782c153400 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:55.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:55.350+0000 7f7833c00640 1 -- 172.21.15.6:0/2226056700 >> 172.21.15.6:0/2226056700 conn(0x7f782c06afb0 msgr2=0x7f782c164ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:55.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:55.351+0000 7f7833c00640 1 -- 172.21.15.6:0/2226056700 shutdown_connections 2024-03-04T09:27:55.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:55.351+0000 7f7833c00640 1 -- 172.21.15.6:0/2226056700 wait complete. 2024-03-04T09:27:55.370 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-04T09:27:55.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:55.587+0000 7fad3c800640 1 Processor -- start 2024-03-04T09:27:55.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:55.587+0000 7fad3c800640 1 -- start start 2024-03-04T09:27:55.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:55.589+0000 7fad3c800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fad340af1a0 0x7fad3415c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:55.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:55.589+0000 7fad3c800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fad340cdc20 0x7fad340b2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:55.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:55.589+0000 7fad3c800640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad3415cc40 con 0x7fad340af580 2024-03-04T09:27:55.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:55.589+0000 7fad3c800640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad3415cf20 con 0x7fad340af1a0 2024-03-04T09:27:55.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:55.589+0000 7fad3c800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad3415d200 con 0x7fad340cdc20 2024-03-04T09:27:55.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:55.590+0000 7fad3a800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fad340cdc20 0x7fad340b2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:55.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:55.590+0000 7fad3a800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fad340cdc20 0x7fad340b2dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.6:47344/0 (socket says 172.21.15.6:47344) 2024-03-04T09:27:55.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:55.590+0000 7fad3a800640 1 -- 172.21.15.6:0/3421690646 learned_addr learned my addr 172.21.15.6:0/3421690646 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:27:55.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:55.590+0000 7fad39e00640 1 --2- 172.21.15.6:0/3421690646 >> v2:172.21.15.146:3300/0 conn(0x7fad340af1a0 0x7fad3415c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:55.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:55.591+0000 7fad3a800640 1 -- 172.21.15.6:0/3421690646 >> v2:172.21.15.146:3300/0 conn(0x7fad340af1a0 msgr2=0x7fad3415c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:27:55.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:55.591+0000 7fad3a800640 1 --2- 172.21.15.6:0/3421690646 >> v2:172.21.15.146:3300/0 conn(0x7fad340af1a0 0x7fad3415c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:55.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:55.591+0000 7fad3a800640 1 -- 172.21.15.6:0/3421690646 >> v1:172.21.15.146:6789/0 conn(0x7fad340af580 legacy=0x7fad340b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:55.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:55.591+0000 7fad3a800640 1 -- 172.21.15.6:0/3421690646 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad3415d4e0 con 0x7fad340cdc20 2024-03-04T09:27:55.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:55.592+0000 7fad3a800640 1 --2- 172.21.15.6:0/3421690646 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fad340cdc20 0x7fad340b2dd0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fad30009e50 tx=0x7fad3002bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4b14fd15f194bdf9 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:55.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:55.592+0000 7fad38a00640 1 -- 172.21.15.6:0/3421690646 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad3002ca70 con 0x7fad340cdc20 2024-03-04T09:27:55.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:55.592+0000 7fad38a00640 1 -- 172.21.15.6:0/3421690646 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad3002cc10 con 0x7fad340cdc20 2024-03-04T09:27:55.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:55.593+0000 7fad38a00640 1 -- 172.21.15.6:0/3421690646 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad300343f0 con 0x7fad340cdc20 2024-03-04T09:27:55.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:55.593+0000 7fad3c800640 1 -- 172.21.15.6:0/3421690646 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fad340cdc20 msgr2=0x7fad340b2dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:55.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:55.593+0000 7fad3c800640 1 --2- 172.21.15.6:0/3421690646 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fad340cdc20 0x7fad340b2dd0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fad30009e50 tx=0x7fad3002bed0 comp rx=0 tx=0).stop 2024-03-04T09:27:55.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:55.593+0000 7fad3c800640 1 -- 172.21.15.6:0/3421690646 shutdown_connections 2024-03-04T09:27:55.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:55.593+0000 7fad3c800640 1 --2- 172.21.15.6:0/3421690646 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fad340cdc20 0x7fad340b2dd0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:55.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:55.593+0000 7fad3c800640 1 --2- 172.21.15.6:0/3421690646 >> v2:172.21.15.146:3300/0 conn(0x7fad340af1a0 0x7fad3415c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:55.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:55.593+0000 7fad3c800640 1 -- 172.21.15.6:0/3421690646 >> 172.21.15.6:0/3421690646 conn(0x7fad3406afb0 msgr2=0x7fad3406b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:55.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:55.593+0000 7fad3c800640 1 -- 172.21.15.6:0/3421690646 shutdown_connections 2024-03-04T09:27:55.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:55.593+0000 7fad3c800640 1 -- 172.21.15.6:0/3421690646 wait complete. 2024-03-04T09:27:55.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:55.595+0000 7fad3c800640 1 Processor -- start 2024-03-04T09:27:55.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:55.595+0000 7fad3c800640 1 -- start start 2024-03-04T09:27:55.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:55.596+0000 7fad3c800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fad340af1a0 0x7fad3414ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:55.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:55.596+0000 7fad3c800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fad340af580 0x7fad34156cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:55.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:55.596+0000 7fad3c800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fad340cdc20 0x7fad341561a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:55.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:55.596+0000 7fad3c800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad340b3720 con 0x7fad340af580 2024-03-04T09:27:55.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:55.596+0000 7fad3c800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad340b35a0 con 0x7fad340cdc20 2024-03-04T09:27:55.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:55.596+0000 7fad3c800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad34155e70 con 0x7fad340af1a0 2024-03-04T09:27:55.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:55.596+0000 7fad39400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fad340af580 0x7fad34156cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:55.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:55.596+0000 7fad39400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fad340af580 0x7fad34156cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42418/0 (socket says 172.21.15.6:42418) 2024-03-04T09:27:55.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:55.596+0000 7fad39400640 1 -- 172.21.15.6:0/3722285231 learned_addr learned my addr 172.21.15.6:0/3722285231 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:27:55.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:55.597+0000 7fad39e00640 1 --2- 172.21.15.6:0/3722285231 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fad340af1a0 0x7fad3414ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:55.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:55.597+0000 7fad3a800640 1 --2- 172.21.15.6:0/3722285231 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fad340cdc20 0x7fad341561a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:55.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:55.597+0000 7fad39400640 1 -- 172.21.15.6:0/3722285231 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fad340af1a0 msgr2=0x7fad3414ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:55.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:55.597+0000 7fad39400640 1 --2- 172.21.15.6:0/3722285231 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fad340af1a0 0x7fad3414ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:55.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:55.597+0000 7fad39400640 1 -- 172.21.15.6:0/3722285231 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fad340cdc20 msgr2=0x7fad341561a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:55.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:55.597+0000 7fad39400640 1 --2- 172.21.15.6:0/3722285231 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fad340cdc20 0x7fad341561a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:55.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:55.597+0000 7fad39400640 1 -- 172.21.15.6:0/3722285231 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad341566e0 con 0x7fad340af580 2024-03-04T09:27:55.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:55.597+0000 7fad39e00640 1 --2- 172.21.15.6:0/3722285231 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fad340af1a0 0x7fad3414ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:27:55.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:55.597+0000 7fad3a800640 1 --2- 172.21.15.6:0/3722285231 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fad340cdc20 0x7fad341561a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:27:55.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:55.597+0000 7fad39400640 1 --2- 172.21.15.6:0/3722285231 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fad340af580 0x7fad34156cf0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fad2400c440 tx=0x7fad24001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:55.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:55.597+0000 7fad2e000640 1 -- 172.21.15.6:0/3722285231 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad240052d0 con 0x7fad340af580 2024-03-04T09:27:55.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:55.597+0000 7fad2e000640 1 -- 172.21.15.6:0/3722285231 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad24005470 con 0x7fad340af580 2024-03-04T09:27:55.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:55.598+0000 7fad2e000640 1 -- 172.21.15.6:0/3722285231 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad24007260 con 0x7fad340af580 2024-03-04T09:27:55.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:55.598+0000 7fad3c800640 1 -- 172.21.15.6:0/3722285231 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad3414cff0 con 0x7fad340af580 2024-03-04T09:27:55.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:55.598+0000 7fad3c800640 1 -- 172.21.15.6:0/3722285231 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad3414d5b0 con 0x7fad340af580 2024-03-04T09:27:55.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:55.599+0000 7fad3c800640 1 -- 172.21.15.6:0/3722285231 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad34069510 con 0x7fad340af580 2024-03-04T09:27:55.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:55.600+0000 7fad2e000640 1 -- 172.21.15.6:0/3722285231 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fad24007400 con 0x7fad340af580 2024-03-04T09:27:55.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:55.601+0000 7fad2e000640 1 --2- 172.21.15.6:0/3722285231 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fad1806b500 0x7fad1806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:55.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:55.602+0000 7fad2e000640 1 -- 172.21.15.6:0/3722285231 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fad240893e0 con 0x7fad340af580 2024-03-04T09:27:55.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:55.602+0000 7fad39e00640 1 --2- 172.21.15.6:0/3722285231 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fad1806b500 0x7fad1806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:55.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:55.603+0000 7fad39e00640 1 --2- 172.21.15.6:0/3722285231 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fad1806b500 0x7fad1806d9c0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fad28006fd0 tx=0x7fad28008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:55.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:55.607+0000 7fad2e000640 1 -- 172.21.15.6:0/3722285231 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fad341566e0 con 0x7fad340af580 2024-03-04T09:27:55.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:55.875+0000 7fad3c800640 1 -- 172.21.15.6:0/3722285231 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fad3414e7a0 con 0x7fad340af580 2024-03-04T09:27:55.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:55.876+0000 7fad2e000640 1 -- 172.21.15.6:0/3722285231 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v11) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fad24058510 con 0x7fad340af580 2024-03-04T09:27:55.873 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:27:55.873 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-04T09:27:55.873 INFO:teuthology.orchestra.run.smithi006.stdout: "auth_dump": [ 2024-03-04T09:27:55.874 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:27:55.874 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "mds.a", 2024-03-04T09:27:55.874 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQA8k+VlHCzFIBAAnKYlmCMDw45LqnSjxnZY5Q==", 2024-03-04T09:27:55.874 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:27:55.874 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-04T09:27:55.874 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow *", 2024-03-04T09:27:55.874 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow *", 2024-03-04T09:27:55.874 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-04T09:27:55.874 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:27:55.874 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:27:55.875 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:27:55.875 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "mds.b", 2024-03-04T09:27:55.875 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQA9k+VlNY7fCBAAX3z8X72/6LkQD6mwGYT69g==", 2024-03-04T09:27:55.875 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:27:55.875 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-04T09:27:55.875 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow *", 2024-03-04T09:27:55.875 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow *", 2024-03-04T09:27:55.875 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-04T09:27:55.875 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:27:55.875 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:27:55.875 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:27:55.875 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "mds.c", 2024-03-04T09:27:55.875 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQA8k+Vl0pjOMhAA7WcTw8awv4EAsrpP4yncNA==", 2024-03-04T09:27:55.876 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:27:55.876 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-04T09:27:55.876 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow *", 2024-03-04T09:27:55.876 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow *", 2024-03-04T09:27:55.876 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-04T09:27:55.876 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:27:55.876 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:27:55.876 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:27:55.876 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "mds.d", 2024-03-04T09:27:55.876 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQA9k+VlbladGRAABNgUcaqTVYuJfCqy5c/G0g==", 2024-03-04T09:27:55.876 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:27:55.876 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-04T09:27:55.876 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow *", 2024-03-04T09:27:55.876 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow *", 2024-03-04T09:27:55.877 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-04T09:27:55.877 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:27:55.877 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:27:55.877 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:27:55.877 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.0", 2024-03-04T09:27:55.877 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQBCk+VlyjMMExAAqb2FJ9n5PWehfx/6PB19ig==", 2024-03-04T09:27:55.877 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:27:55.877 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-04T09:27:55.877 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-04T09:27:55.877 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-04T09:27:55.877 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:27:55.877 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:27:55.877 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:27:55.877 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.1", 2024-03-04T09:27:55.877 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQBFk+VlXD3cLBAAGclSivcYWDslqN8MLlj7eQ==", 2024-03-04T09:27:55.878 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:27:55.878 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-04T09:27:55.878 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-04T09:27:55.878 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-04T09:27:55.878 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:27:55.878 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:27:55.878 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:27:55.878 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.2", 2024-03-04T09:27:55.878 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQBJk+Vl7d9nCxAAnb3W4EI3Yf3XY8ht3VnKEQ==", 2024-03-04T09:27:55.878 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:27:55.878 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-04T09:27:55.878 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-04T09:27:55.878 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-04T09:27:55.878 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:27:55.879 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:27:55.879 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:27:55.879 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.3", 2024-03-04T09:27:55.879 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQBMk+Vlv+Y1NBAAiRYSUqov2i2sj3iInfpHvA==", 2024-03-04T09:27:55.879 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:27:55.880 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-04T09:27:55.880 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-04T09:27:55.880 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-04T09:27:55.880 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:27:55.880 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:27:55.880 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:27:55.880 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.4", 2024-03-04T09:27:55.880 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQBUk+Vl1OlAARAAxMV4GUGS+RJ6XlrRhrVKcg==", 2024-03-04T09:27:55.880 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:27:55.880 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-04T09:27:55.880 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-04T09:27:55.881 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-04T09:27:55.881 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:27:55.881 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:27:55.881 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:27:55.881 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.5", 2024-03-04T09:27:55.881 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQBXk+Vl1lKLGhAAHlsiRr+1+OSRWhkB+weIJg==", 2024-03-04T09:27:55.881 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:27:55.881 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-04T09:27:55.881 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-04T09:27:55.881 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-04T09:27:55.881 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:27:55.881 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:27:55.881 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:27:55.881 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.6", 2024-03-04T09:27:55.881 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQBak+VlwB+GJBAA3f9KUz+zZlQuUR/ywQfkDA==", 2024-03-04T09:27:55.882 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:27:55.882 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-04T09:27:55.882 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-04T09:27:55.882 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-04T09:27:55.882 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:27:55.882 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:27:55.882 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:27:55.882 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.7", 2024-03-04T09:27:55.882 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQBdk+Vle4o8LhAAko+iH5VqqCsfWkNfKZjGVA==", 2024-03-04T09:27:55.882 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:27:55.882 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-04T09:27:55.882 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-04T09:27:55.882 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-04T09:27:55.882 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:27:55.883 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:27:55.883 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:27:55.883 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "client.0", 2024-03-04T09:27:55.883 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQA9k+Vl4L+rJxAACaDbQ4/Bybp5hAZAnJzlVw==", 2024-03-04T09:27:55.883 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:27:55.883 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-04T09:27:55.883 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow r", 2024-03-04T09:27:55.883 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-04T09:27:55.883 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:27:55.883 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:27:55.883 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:27:55.883 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "client.1", 2024-03-04T09:27:55.883 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQA+k+VlGjkOAhAAJNzB8+tQl3mqCnTgMn9EVw==", 2024-03-04T09:27:55.883 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:27:55.883 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-04T09:27:55.884 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow r", 2024-03-04T09:27:55.884 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow rw", 2024-03-04T09:27:55.884 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow rwx" 2024-03-04T09:27:55.884 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:27:55.884 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:27:55.884 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:27:55.884 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "client.2", 2024-03-04T09:27:55.884 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQA9k+VlUc8DMxAA8pGUtIh+BO6UKrrYnCqVgQ==", 2024-03-04T09:27:55.884 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:27:55.885 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-04T09:27:55.885 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow r", 2024-03-04T09:27:55.885 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow rw", 2024-03-04T09:27:55.885 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow rwx" 2024-03-04T09:27:55.885 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:27:55.885 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:27:55.885 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:27:55.885 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "client.3", 2024-03-04T09:27:55.885 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQA+k+VlzMFKDBAAGr713I2hcmGMuHMIjNknYA==", 2024-03-04T09:27:55.885 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:27:55.885 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-04T09:27:55.885 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow r", 2024-03-04T09:27:55.885 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow rw", 2024-03-04T09:27:55.885 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow rwx" 2024-03-04T09:27:55.886 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:27:55.886 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:27:55.886 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:27:55.886 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "client.admin", 2024-03-04T09:27:55.886 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQA7k+Vllb7NFhAAmnFaNFKBcEri+iJAPnG5Gg==", 2024-03-04T09:27:55.886 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:27:55.886 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow *", 2024-03-04T09:27:55.886 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow *", 2024-03-04T09:27:55.886 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow *", 2024-03-04T09:27:55.886 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-04T09:27:55.886 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:27:55.886 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:27:55.886 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:27:55.886 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "mgr.x", 2024-03-04T09:27:55.887 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQA8k+VlPFatFBAA/o3Ps/uJrcg1738HOGy/FA==", 2024-03-04T09:27:55.887 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:27:55.887 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow *", 2024-03-04T09:27:55.887 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile mgr", 2024-03-04T09:27:55.887 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-04T09:27:55.887 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:27:55.887 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:27:55.887 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:27:55.887 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "mgr.y", 2024-03-04T09:27:55.887 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQA8k+VlEA2ACBAAXofDj3gmqItVTJ1pYLqZyQ==", 2024-03-04T09:27:55.887 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:27:55.888 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow *", 2024-03-04T09:27:55.888 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile mgr", 2024-03-04T09:27:55.888 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-04T09:27:55.888 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:27:55.888 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:27:55.888 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-04T09:27:55.889 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-04T09:27:55.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:55.879+0000 7fad3c800640 1 -- 172.21.15.6:0/3722285231 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fad1806b500 msgr2=0x7fad1806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:55.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:55.879+0000 7fad3c800640 1 --2- 172.21.15.6:0/3722285231 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fad1806b500 0x7fad1806d9c0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fad28006fd0 tx=0x7fad28008040 comp rx=0 tx=0).stop 2024-03-04T09:27:55.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:55.879+0000 7fad3c800640 1 -- 172.21.15.6:0/3722285231 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fad340af580 msgr2=0x7fad34156cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:55.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:55.879+0000 7fad3c800640 1 --2- 172.21.15.6:0/3722285231 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fad340af580 0x7fad34156cf0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fad2400c440 tx=0x7fad24001e70 comp rx=0 tx=0).stop 2024-03-04T09:27:55.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:55.879+0000 7fad3c800640 1 -- 172.21.15.6:0/3722285231 shutdown_connections 2024-03-04T09:27:55.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:55.879+0000 7fad3c800640 1 --2- 172.21.15.6:0/3722285231 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fad340cdc20 0x7fad341561a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:55.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:55.879+0000 7fad3c800640 1 --2- 172.21.15.6:0/3722285231 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fad340af580 0x7fad34156cf0 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:55.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:55.879+0000 7fad3c800640 1 --2- 172.21.15.6:0/3722285231 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fad1806b500 0x7fad1806d9c0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:55.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:55.879+0000 7fad3c800640 1 --2- 172.21.15.6:0/3722285231 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fad340af1a0 0x7fad3414ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:55.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:55.879+0000 7fad3c800640 1 -- 172.21.15.6:0/3722285231 >> 172.21.15.6:0/3722285231 conn(0x7fad3406afb0 msgr2=0x7fad340acdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:55.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:55.879+0000 7fad3c800640 1 -- 172.21.15.6:0/3722285231 shutdown_connections 2024-03-04T09:27:55.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:55.879+0000 7fad3c800640 1 -- 172.21.15.6:0/3722285231 wait complete. 2024-03-04T09:27:55.895 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T09:27:56.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.111+0000 7fb247a00640 1 Processor -- start 2024-03-04T09:27:56.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.111+0000 7fb247a00640 1 -- start start 2024-03-04T09:27:56.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.113+0000 7fb247a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fb240161670 0x7fb240070c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:56.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.113+0000 7fb247a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb24006de00 con 0x7fb2400ae0a0 2024-03-04T09:27:56.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.113+0000 7fb247a00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2400ce9a0 con 0x7fb24006e240 2024-03-04T09:27:56.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.113+0000 7fb247a00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2400730d0 con 0x7fb240161670 2024-03-04T09:27:56.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.114+0000 7fb245000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fb240161670 0x7fb240070c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:56.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.114+0000 7fb245000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fb240161670 0x7fb240070c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42428/0 (socket says 172.21.15.6:42428) 2024-03-04T09:27:56.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.114+0000 7fb245000640 1 -- 172.21.15.6:0/3086167921 learned_addr learned my addr 172.21.15.6:0/3086167921 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:27:56.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.114+0000 7fb245000640 1 -- 172.21.15.6:0/3086167921 >> v1:172.21.15.146:6789/0 conn(0x7fb24006e240 legacy=0x7fb24006cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:27:56.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.114+0000 7fb245000640 1 -- 172.21.15.6:0/3086167921 >> v1:172.21.15.6:6789/0 conn(0x7fb2400ae0a0 legacy=0x7fb24006d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:56.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.114+0000 7fb245000640 1 -- 172.21.15.6:0/3086167921 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2400cec80 con 0x7fb240161670 2024-03-04T09:27:56.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.115+0000 7fb245000640 1 --2- 172.21.15.6:0/3086167921 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb240161670 0x7fb240070c70 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fb234009870 tx=0x7fb23402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28255dc739ca9d00 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:56.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.116+0000 7fb23f400640 1 -- 172.21.15.6:0/3086167921 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb234034070 con 0x7fb240161670 2024-03-04T09:27:56.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.116+0000 7fb23f400640 1 -- 172.21.15.6:0/3086167921 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb234030860 con 0x7fb240161670 2024-03-04T09:27:56.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.116+0000 7fb23f400640 1 -- 172.21.15.6:0/3086167921 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb234030c30 con 0x7fb240161670 2024-03-04T09:27:56.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.116+0000 7fb247a00640 1 -- 172.21.15.6:0/3086167921 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb240161670 msgr2=0x7fb240070c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:56.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.116+0000 7fb247a00640 1 --2- 172.21.15.6:0/3086167921 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb240161670 0x7fb240070c70 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fb234009870 tx=0x7fb23402b9a0 comp rx=0 tx=0).stop 2024-03-04T09:27:56.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.117+0000 7fb247a00640 1 -- 172.21.15.6:0/3086167921 shutdown_connections 2024-03-04T09:27:56.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.117+0000 7fb247a00640 1 --2- 172.21.15.6:0/3086167921 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb240161670 0x7fb240070c70 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:56.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.117+0000 7fb247a00640 1 -- 172.21.15.6:0/3086167921 >> 172.21.15.6:0/3086167921 conn(0x7fb24006afb0 msgr2=0x7fb24006b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:56.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.117+0000 7fb247a00640 1 -- 172.21.15.6:0/3086167921 shutdown_connections 2024-03-04T09:27:56.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.117+0000 7fb247a00640 1 -- 172.21.15.6:0/3086167921 wait complete. 2024-03-04T09:27:56.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.119+0000 7fb247a00640 1 Processor -- start 2024-03-04T09:27:56.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.119+0000 7fb247a00640 1 -- start start 2024-03-04T09:27:56.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.120+0000 7fb247a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb24006e240 0x7fb24015a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:56.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.120+0000 7fb247a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb2400ae0a0 0x7fb24014b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:56.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.120+0000 7fb247a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb240161670 0x7fb2401f4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:56.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.120+0000 7fb247a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2400cf520 con 0x7fb240161670 2024-03-04T09:27:56.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.120+0000 7fb247a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2400cf240 con 0x7fb2400ae0a0 2024-03-04T09:27:56.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.120+0000 7fb247a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2400ce600 con 0x7fb24006e240 2024-03-04T09:27:56.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.120+0000 7fb245a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb240161670 0x7fb2401f4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:56.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.120+0000 7fb245a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb240161670 0x7fb2401f4b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42440/0 (socket says 172.21.15.6:42440) 2024-03-04T09:27:56.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.120+0000 7fb245a00640 1 -- 172.21.15.6:0/2588782301 learned_addr learned my addr 172.21.15.6:0/2588782301 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:27:56.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.121+0000 7fb245000640 1 --2- 172.21.15.6:0/2588782301 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb24006e240 0x7fb24015a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:56.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.121+0000 7fb245a00640 1 -- 172.21.15.6:0/2588782301 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb24006e240 msgr2=0x7fb24015a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:56.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.121+0000 7fb245a00640 1 --2- 172.21.15.6:0/2588782301 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb24006e240 0x7fb24015a3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:56.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.121+0000 7fb245a00640 1 -- 172.21.15.6:0/2588782301 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb2400ae0a0 msgr2=0x7fb24014b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:56.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.121+0000 7fb245a00640 1 --2- 172.21.15.6:0/2588782301 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb2400ae0a0 0x7fb24014b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:56.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.121+0000 7fb245a00640 1 -- 172.21.15.6:0/2588782301 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb24015a9b0 con 0x7fb240161670 2024-03-04T09:27:56.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.121+0000 7fb245a00640 1 --2- 172.21.15.6:0/2588782301 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb240161670 0x7fb2401f4b70 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fb23800c7d0 tx=0x7fb23800cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:56.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.121+0000 7fb23cc00640 1 -- 172.21.15.6:0/2588782301 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb238003b70 con 0x7fb240161670 2024-03-04T09:27:56.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.121+0000 7fb23cc00640 1 -- 172.21.15.6:0/2588782301 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb238003d30 con 0x7fb240161670 2024-03-04T09:27:56.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.121+0000 7fb23cc00640 1 -- 172.21.15.6:0/2588782301 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb23800aa90 con 0x7fb240161670 2024-03-04T09:27:56.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.121+0000 7fb247a00640 1 -- 172.21.15.6:0/2588782301 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb24015ac90 con 0x7fb240161670 2024-03-04T09:27:56.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.121+0000 7fb247a00640 1 -- 172.21.15.6:0/2588782301 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb24015b250 con 0x7fb240161670 2024-03-04T09:27:56.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.123+0000 7fb247a00640 1 -- 172.21.15.6:0/2588782301 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb240069510 con 0x7fb240161670 2024-03-04T09:27:56.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.124+0000 7fb23cc00640 1 -- 172.21.15.6:0/2588782301 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb23800ac30 con 0x7fb240161670 2024-03-04T09:27:56.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.124+0000 7fb23cc00640 1 --2- 172.21.15.6:0/2588782301 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb22006b550 0x7fb22006da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:56.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.125+0000 7fb23cc00640 1 -- 172.21.15.6:0/2588782301 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fb238089e40 con 0x7fb240161670 2024-03-04T09:27:56.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.125+0000 7fb245000640 1 --2- 172.21.15.6:0/2588782301 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb22006b550 0x7fb22006da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:56.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.126+0000 7fb245000640 1 --2- 172.21.15.6:0/2588782301 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb22006b550 0x7fb22006da10 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fb234036000 tx=0x7fb23403b040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:56.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.129+0000 7fb23cc00640 1 -- 172.21.15.6:0/2588782301 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb238058ff0 con 0x7fb240161670 2024-03-04T09:27:56.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.329+0000 7fb247a00640 1 -- 172.21.15.6:0/2588782301 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb2400b52d0 con 0x7fb240161670 2024-03-04T09:27:56.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.330+0000 7fb23cc00640 1 -- 172.21.15.6:0/2588782301 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v52) v1 ==== 74+0+12843 (secure 0 0 0) 0x7fb2380588f0 con 0x7fb240161670 2024-03-04T09:27:56.327 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:27:56.327 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":52,"fsid":"edce0578-5ae4-4402-b45a-552bdef84d7c","created":"2024-03-04T09:24:54.523780+0000","modified":"2024-03-04T09:27:55.343795+0000","last_up_change":"2024-03-04T09:25:07.089049+0000","last_in_change":"2024-03-04T09:24:57.982077+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T09:25:07.935198+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a96574f-81f1-4bc6-8463-757520b84a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6825","nonce":1364449401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6827","nonce":1364449401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6831","nonce":1364449401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6829","nonce":1364449401}]},"public_addr":"172.21.15.6:6825/1364449401","cluster_addr":"172.21.15.6:6827/1364449401","heartbeat_back_addr":"172.21.15.6:6831/1364449401","heartbeat_front_addr":"172.21.15.6:6829/1364449401","state":["exists","up"]},{"osd":1,"uuid":"7d79b9c1-6b87-476a-bc36-7db5ee64cd90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6809","nonce":2494103078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6811","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6813","nonce":2494103078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6821","nonce":2494103078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6817","nonce":2494103078}]},"public_addr":"172.21.15.6:6809/2494103078","cluster_addr":"172.21.15.6:6813/2494103078","heartbeat_back_addr":"172.21.15.6:6821/2494103078","heartbeat_front_addr":"172.21.15.6:6817/2494103078","state":["exists","up"]},{"osd":2,"uuid":"492331b9-4ee9-40b6-bacd-85d44cd6c45a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6812","nonce":1500066681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6816","nonce":1500066681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6824","nonce":1500066681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6820","nonce":1500066681}]},"public_addr":"172.21.15.6:6812/1500066681","cluster_addr":"172.21.15.6:6816/1500066681","heartbeat_back_addr":"172.21.15.6:6824/1500066681","heartbeat_front_addr":"172.21.15.6:6820/1500066681","state":["exists","up"]},{"osd":3,"uuid":"21bb2efe-9ba7-4bdc-bc31-180cd1102e21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6801","nonce":1558806123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6803","nonce":1558806123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6807","nonce":1558806123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6805","nonce":1558806123}]},"public_addr":"172.21.15.6:6801/1558806123","cluster_addr":"172.21.15.6:6803/1558806123","heartbeat_back_addr":"172.21.15.6:6807/1558806123","heartbeat_front_addr":"172.21.15.6:6805/1558806123","state":["exists","up"]},{"osd":4,"uuid":"6e2c4b71-19d9-4737-83d9-558705d8644a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6801","nonce":1526135780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6803","nonce":1526135780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6807","nonce":1526135780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6805","nonce":1526135780}]},"public_addr":"172.21.15.146:6801/1526135780","cluster_addr":"172.21.15.146:6803/1526135780","heartbeat_back_addr":"172.21.15.146:6807/1526135780","heartbeat_front_addr":"172.21.15.146:6805/1526135780","state":["exists","up"]},{"osd":5,"uuid":"fb69c525-f1be-4b7d-ab32-6c29b0d6fddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6809","nonce":460262507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6811","nonce":460262507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6815","nonce":460262507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6813","nonce":460262507}]},"public_addr":"172.21.15.146:6809/460262507","cluster_addr":"172.21.15.146:6811/460262507","heartbeat_back_addr":"172.21.15.146:6815/460262507","heartbeat_front_addr":"172.21.15.146:6813/460262507","state":["exists","up"]},{"osd":6,"uuid":"c32e7fed-370d-4401-8ce0-6a4335b0d168","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6823","nonce":1392897493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6827","nonce":1392897493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6831","nonce":1392897493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6829","nonce":1392897493}]},"public_addr":"172.21.15.146:6823/1392897493","cluster_addr":"172.21.15.146:6827/1392897493","heartbeat_back_addr":"172.21.15.146:6831/1392897493","heartbeat_front_addr":"172.21.15.146:6829/1392897493","state":["exists","up"]},{"osd":7,"uuid":"75a79b44-fa5f-426d-a7d3-71f21f78835c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6817","nonce":3268323156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6819","nonce":3268323156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6825","nonce":3268323156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6822","nonce":3268323156}]},"public_addr":"172.21.15.146:6817/3268323156","cluster_addr":"172.21.15.146:6819/3268323156","heartbeat_back_addr":"172.21.15.146:6825/3268323156","heartbeat_front_addr":"172.21.15.146:6822/3268323156","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.310437+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.295850+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.294674+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.526855+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.300023+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.595160+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.346046+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.558741+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T09:27:56.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.333+0000 7fb247a00640 1 -- 172.21.15.6:0/2588782301 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb22006b550 msgr2=0x7fb22006da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:56.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.333+0000 7fb247a00640 1 --2- 172.21.15.6:0/2588782301 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb22006b550 0x7fb22006da10 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fb234036000 tx=0x7fb23403b040 comp rx=0 tx=0).stop 2024-03-04T09:27:56.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.333+0000 7fb247a00640 1 -- 172.21.15.6:0/2588782301 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb240161670 msgr2=0x7fb2401f4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:56.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.333+0000 7fb247a00640 1 --2- 172.21.15.6:0/2588782301 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb240161670 0x7fb2401f4b70 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fb23800c7d0 tx=0x7fb23800cc90 comp rx=0 tx=0).stop 2024-03-04T09:27:56.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.333+0000 7fb247a00640 1 -- 172.21.15.6:0/2588782301 shutdown_connections 2024-03-04T09:27:56.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.333+0000 7fb247a00640 1 --2- 172.21.15.6:0/2588782301 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb240161670 0x7fb2401f4b70 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:56.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.333+0000 7fb247a00640 1 --2- 172.21.15.6:0/2588782301 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb2400ae0a0 0x7fb24014b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:56.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.333+0000 7fb247a00640 1 --2- 172.21.15.6:0/2588782301 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb22006b550 0x7fb22006da10 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:56.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.333+0000 7fb247a00640 1 --2- 172.21.15.6:0/2588782301 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb24006e240 0x7fb24015a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:56.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.333+0000 7fb247a00640 1 -- 172.21.15.6:0/2588782301 >> 172.21.15.6:0/2588782301 conn(0x7fb24006afb0 msgr2=0x7fb2400cd120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:56.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.333+0000 7fb247a00640 1 -- 172.21.15.6:0/2588782301 shutdown_connections 2024-03-04T09:27:56.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.334+0000 7fb247a00640 1 -- 172.21.15.6:0/2588782301 wait complete. 2024-03-04T09:27:56.351 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T09:25:07.935198+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-04T09:27:56.352 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-04T09:27:56.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.582+0000 7fb327e00640 1 Processor -- start 2024-03-04T09:27:56.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.583+0000 7fb327e00640 1 -- start start 2024-03-04T09:27:56.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.584+0000 7fb327e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fb32815f810 0x7fb3280b1b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:56.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.584+0000 7fb327e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb32815fe40 con 0x7fb328145430 2024-03-04T09:27:56.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.584+0000 7fb327e00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb328160120 con 0x7fb3280b2080 2024-03-04T09:27:56.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.584+0000 7fb327e00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3280b24d0 con 0x7fb32815f810 2024-03-04T09:27:56.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.585+0000 7fb326a00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fb328145430 0x7fb32815cc90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:58932/0 (socket says 172.21.15.6:58932) 2024-03-04T09:27:56.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.585+0000 7fb326a00640 1 -- 172.21.15.6:0/3451056452 learned_addr learned my addr 172.21.15.6:0/3451056452 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:27:56.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.585+0000 7fb326000640 1 -- 172.21.15.6:0/3451056452 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4271519977 0 0) 0x7fb32815fe40 con 0x7fb328145430 2024-03-04T09:27:56.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.585+0000 7fb326000640 1 -- 172.21.15.6:0/3451056452 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb30c003520 con 0x7fb328145430 2024-03-04T09:27:56.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.585+0000 7fb31fe00640 1 --2- 172.21.15.6:0/3451056452 >> v2:172.21.15.146:3300/0 conn(0x7fb32815f810 0x7fb3280b1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:56.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.585+0000 7fb326000640 1 -- 172.21.15.6:0/3451056452 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1416894230 0 0) 0x7fb30c003520 con 0x7fb328145430 2024-03-04T09:27:56.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.585+0000 7fb326000640 1 -- 172.21.15.6:0/3451056452 >> v2:172.21.15.146:3300/0 conn(0x7fb32815f810 msgr2=0x7fb3280b1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:27:56.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.585+0000 7fb326000640 1 --2- 172.21.15.6:0/3451056452 >> v2:172.21.15.146:3300/0 conn(0x7fb32815f810 0x7fb3280b1b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:56.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.585+0000 7fb326000640 1 -- 172.21.15.6:0/3451056452 >> v1:172.21.15.146:6789/0 conn(0x7fb3280b2080 legacy=0x7fb3280b2fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:27:56.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.586+0000 7fb326000640 1 -- 172.21.15.6:0/3451056452 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb328160400 con 0x7fb328145430 2024-03-04T09:27:56.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.586+0000 7fb326000640 1 -- 172.21.15.6:0/3451056452 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7fb318001670 con 0x7fb328145430 2024-03-04T09:27:56.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.586+0000 7fb326000640 1 -- 172.21.15.6:0/3451056452 >> v1:172.21.15.6:6789/0 conn(0x7fb328145430 legacy=0x7fb32815cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:56.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.586+0000 7fb326000640 1 --2- 172.21.15.6:0/3451056452 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb30c003a00 0x7fb30c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:56.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.586+0000 7fb326000640 1 --2- 172.21.15.6:0/3451056452 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb30c004800 0x7fb30c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:56.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.586+0000 7fb326000640 1 --2- 172.21.15.6:0/3451056452 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb30c005210 0x7fb30c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:56.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.586+0000 7fb326000640 1 -- 172.21.15.6:0/3451056452 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3280b24d0 con 0x7fb30c003a00 2024-03-04T09:27:56.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.586+0000 7fb326000640 1 -- 172.21.15.6:0/3451056452 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb30c00c140 con 0x7fb30c004800 2024-03-04T09:27:56.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.586+0000 7fb326000640 1 -- 172.21.15.6:0/3451056452 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb30c00c340 con 0x7fb30c005210 2024-03-04T09:27:56.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.586+0000 7fb326a00640 1 --2- 172.21.15.6:0/3451056452 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb30c003a00 0x7fb30c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:56.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.586+0000 7fb327400640 1 --2- 172.21.15.6:0/3451056452 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb30c005210 0x7fb30c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:56.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.586+0000 7fb326a00640 1 -- 172.21.15.6:0/3451056452 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb30c005210 msgr2=0x7fb30c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:27:56.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.586+0000 7fb326a00640 1 --2- 172.21.15.6:0/3451056452 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb30c005210 0x7fb30c00b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:56.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.586+0000 7fb326a00640 1 -- 172.21.15.6:0/3451056452 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb30c004800 msgr2=0x7fb30c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:27:56.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.586+0000 7fb326a00640 1 --2- 172.21.15.6:0/3451056452 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb30c004800 0x7fb30c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:56.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.586+0000 7fb326a00640 1 -- 172.21.15.6:0/3451056452 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb30c003520 con 0x7fb30c003a00 2024-03-04T09:27:56.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.586+0000 7fb327400640 1 -- 172.21.15.6:0/3451056452 reap_dead start 2024-03-04T09:27:56.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.587+0000 7fb326a00640 1 --2- 172.21.15.6:0/3451056452 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb30c003a00 0x7fb30c003e00 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fb31800c0e0 tx=0x7fb30c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8df8473116350455 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:56.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.588+0000 7fb326000640 1 -- 172.21.15.6:0/3451056452 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb318007c40 con 0x7fb30c003a00 2024-03-04T09:27:56.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.588+0000 7fb326000640 1 -- 172.21.15.6:0/3451056452 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb318030070 con 0x7fb30c003a00 2024-03-04T09:27:56.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.588+0000 7fb326000640 1 -- 172.21.15.6:0/3451056452 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb318005d70 con 0x7fb30c003a00 2024-03-04T09:27:56.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.588+0000 7fb327e00640 1 -- 172.21.15.6:0/3451056452 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb30c003a00 msgr2=0x7fb30c003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:56.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.588+0000 7fb327e00640 1 --2- 172.21.15.6:0/3451056452 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb30c003a00 0x7fb30c003e00 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fb31800c0e0 tx=0x7fb30c005830 comp rx=0 tx=0).stop 2024-03-04T09:27:56.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.588+0000 7fb327e00640 1 -- 172.21.15.6:0/3451056452 shutdown_connections 2024-03-04T09:27:56.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.588+0000 7fb327e00640 1 --2- 172.21.15.6:0/3451056452 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb30c003a00 0x7fb30c003e00 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:56.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.588+0000 7fb327e00640 1 -- 172.21.15.6:0/3451056452 >> 172.21.15.6:0/3451056452 conn(0x7fb32806b3a0 msgr2=0x7fb32806b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:56.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.589+0000 7fb327e00640 1 -- 172.21.15.6:0/3451056452 shutdown_connections 2024-03-04T09:27:56.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.589+0000 7fb327e00640 1 -- 172.21.15.6:0/3451056452 wait complete. 2024-03-04T09:27:56.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.590+0000 7fb327e00640 1 Processor -- start 2024-03-04T09:27:56.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.590+0000 7fb327e00640 1 -- start start 2024-03-04T09:27:56.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.591+0000 7fb327e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb30c003a00 0x7fb328156f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:56.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.591+0000 7fb327e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb32813dcf0 0x7fb328155070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:56.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.591+0000 7fb327e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb3280719f0 0x7fb328071ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:56.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.591+0000 7fb327e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb328145990 con 0x7fb32813dcf0 2024-03-04T09:27:56.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.591+0000 7fb327e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb328145810 con 0x7fb30c003a00 2024-03-04T09:27:56.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.591+0000 7fb327e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3280cde00 con 0x7fb3280719f0 2024-03-04T09:27:56.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.592+0000 7fb31fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb32813dcf0 0x7fb328155070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:56.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.592+0000 7fb31fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb32813dcf0 0x7fb328155070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42456/0 (socket says 172.21.15.6:42456) 2024-03-04T09:27:56.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.592+0000 7fb31fe00640 1 -- 172.21.15.6:0/2227951944 learned_addr learned my addr 172.21.15.6:0/2227951944 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:27:56.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.592+0000 7fb326a00640 1 --2- 172.21.15.6:0/2227951944 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb30c003a00 0x7fb328156f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:56.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.592+0000 7fb327400640 1 --2- 172.21.15.6:0/2227951944 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb3280719f0 0x7fb328071ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:56.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.592+0000 7fb31fe00640 1 -- 172.21.15.6:0/2227951944 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb3280719f0 msgr2=0x7fb328071ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:56.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.592+0000 7fb31fe00640 1 --2- 172.21.15.6:0/2227951944 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb3280719f0 0x7fb328071ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:56.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.592+0000 7fb31fe00640 1 -- 172.21.15.6:0/2227951944 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb30c003a00 msgr2=0x7fb328156f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:56.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.592+0000 7fb31fe00640 1 --2- 172.21.15.6:0/2227951944 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb30c003a00 0x7fb328156f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:56.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.592+0000 7fb31fe00640 1 -- 172.21.15.6:0/2227951944 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb32813e170 con 0x7fb32813dcf0 2024-03-04T09:27:56.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.593+0000 7fb31fe00640 1 --2- 172.21.15.6:0/2227951944 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb32813dcf0 0x7fb328155070 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fb31000d800 tx=0x7fb31000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:56.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.593+0000 7fb31f400640 1 -- 172.21.15.6:0/2227951944 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb31000b660 con 0x7fb32813dcf0 2024-03-04T09:27:56.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.593+0000 7fb31f400640 1 -- 172.21.15.6:0/2227951944 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb310019070 con 0x7fb32813dcf0 2024-03-04T09:27:56.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.593+0000 7fb31f400640 1 -- 172.21.15.6:0/2227951944 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb31000ba40 con 0x7fb32813dcf0 2024-03-04T09:27:56.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.593+0000 7fb327e00640 1 -- 172.21.15.6:0/2227951944 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3281557e0 con 0x7fb32813dcf0 2024-03-04T09:27:56.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.593+0000 7fb327e00640 1 -- 172.21.15.6:0/2227951944 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3280711f0 con 0x7fb32813dcf0 2024-03-04T09:27:56.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.594+0000 7fb327e00640 1 -- 172.21.15.6:0/2227951944 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb32804f330 con 0x7fb32813dcf0 2024-03-04T09:27:56.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.595+0000 7fb31f400640 1 -- 172.21.15.6:0/2227951944 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb31000bbe0 con 0x7fb32813dcf0 2024-03-04T09:27:56.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.596+0000 7fb31f400640 1 --2- 172.21.15.6:0/2227951944 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb30406b500 0x7fb30406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:56.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.596+0000 7fb31f400640 1 -- 172.21.15.6:0/2227951944 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fb31008a270 con 0x7fb32813dcf0 2024-03-04T09:27:56.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.597+0000 7fb326a00640 1 --2- 172.21.15.6:0/2227951944 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb30406b500 0x7fb30406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:56.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.598+0000 7fb326a00640 1 --2- 172.21.15.6:0/2227951944 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb30406b500 0x7fb30406d9c0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fb318034000 tx=0x7fb318007000 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:56.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.601+0000 7fb31f400640 1 -- 172.21.15.6:0/2227951944 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb3100593a0 con 0x7fb32813dcf0 2024-03-04T09:27:56.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.803+0000 7fb327e00640 1 -- 172.21.15.6:0/2227951944 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb32806e240 con 0x7fb32813dcf0 2024-03-04T09:27:56.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.804+0000 7fb31f400640 1 -- 172.21.15.6:0/2227951944 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v52) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb310058ca0 con 0x7fb32813dcf0 2024-03-04T09:27:56.801 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-04T09:27:56.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.806+0000 7fb327e00640 1 -- 172.21.15.6:0/2227951944 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb30406b500 msgr2=0x7fb30406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:56.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.806+0000 7fb327e00640 1 --2- 172.21.15.6:0/2227951944 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb30406b500 0x7fb30406d9c0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fb318034000 tx=0x7fb318007000 comp rx=0 tx=0).stop 2024-03-04T09:27:56.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.807+0000 7fb327e00640 1 -- 172.21.15.6:0/2227951944 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb32813dcf0 msgr2=0x7fb328155070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:56.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.807+0000 7fb327e00640 1 --2- 172.21.15.6:0/2227951944 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb32813dcf0 0x7fb328155070 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fb31000d800 tx=0x7fb31000dcc0 comp rx=0 tx=0).stop 2024-03-04T09:27:56.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.807+0000 7fb327e00640 1 -- 172.21.15.6:0/2227951944 shutdown_connections 2024-03-04T09:27:56.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.807+0000 7fb327e00640 1 --2- 172.21.15.6:0/2227951944 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb30406b500 0x7fb30406d9c0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:56.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.807+0000 7fb327e00640 1 --2- 172.21.15.6:0/2227951944 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb3280719f0 0x7fb328071ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:56.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.807+0000 7fb327e00640 1 --2- 172.21.15.6:0/2227951944 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb32813dcf0 0x7fb328155070 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:56.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.807+0000 7fb327e00640 1 --2- 172.21.15.6:0/2227951944 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb30c003a00 0x7fb328156f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:56.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.807+0000 7fb327e00640 1 -- 172.21.15.6:0/2227951944 >> 172.21.15.6:0/2227951944 conn(0x7fb32806b3a0 msgr2=0x7fb3280cf930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:56.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.807+0000 7fb327e00640 1 -- 172.21.15.6:0/2227951944 shutdown_connections 2024-03-04T09:27:56.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:56.807+0000 7fb327e00640 1 -- 172.21.15.6:0/2227951944 wait complete. 2024-03-04T09:27:56.823 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-04T09:27:57.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.046+0000 7f2b71a00640 1 Processor -- start 2024-03-04T09:27:57.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.046+0000 7f2b71a00640 1 -- start start 2024-03-04T09:27:57.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.048+0000 7f2b71a00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f2b6c15c920 0x7f2b6c070a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:57.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.048+0000 7f2b71a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2b6c0ce850 0x7f2b6c0cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:57.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.048+0000 7f2b71a00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b6c0cdad0 con 0x7f2b6c070f80 2024-03-04T09:27:57.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.048+0000 7f2b71a00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b6c071500 con 0x7f2b6c15c920 2024-03-04T09:27:57.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.048+0000 7f2b71a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b6c0cdd70 con 0x7f2b6c0ce850 2024-03-04T09:27:57.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.049+0000 7f2b6b400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2b6c0ce850 0x7f2b6c0cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:57.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.049+0000 7f2b6b400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2b6c0ce850 0x7f2b6c0cc950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.6:47366/0 (socket says 172.21.15.6:47366) 2024-03-04T09:27:57.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.049+0000 7f2b6b400640 1 -- 172.21.15.6:0/803803008 learned_addr learned my addr 172.21.15.6:0/803803008 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:27:57.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.049+0000 7f2b71000640 1 --1- 172.21.15.6:0/803803008 >> v1:172.21.15.146:6789/0 conn(0x7f2b6c070f80 0x7f2b6c165e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.6:34808/0 (socket says 172.21.15.6:34808) 2024-03-04T09:27:57.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.049+0000 7f2b6be00640 1 --2- 172.21.15.6:0/803803008 >> v2:172.21.15.146:3300/0 conn(0x7f2b6c15c920 0x7f2b6c070a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:57.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.050+0000 7f2b6b400640 1 -- 172.21.15.6:0/803803008 >> v2:172.21.15.146:3300/0 conn(0x7f2b6c15c920 msgr2=0x7f2b6c070a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:27:57.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.050+0000 7f2b6b400640 1 --2- 172.21.15.6:0/803803008 >> v2:172.21.15.146:3300/0 conn(0x7f2b6c15c920 0x7f2b6c070a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:57.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.050+0000 7f2b6b400640 1 -- 172.21.15.6:0/803803008 >> v1:172.21.15.146:6789/0 conn(0x7f2b6c070f80 legacy=0x7f2b6c165e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:57.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.050+0000 7f2b6b400640 1 -- 172.21.15.6:0/803803008 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b6c0ce050 con 0x7f2b6c0ce850 2024-03-04T09:27:57.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.050+0000 7f2b6be00640 1 --2- 172.21.15.6:0/803803008 >> v2:172.21.15.146:3300/0 conn(0x7f2b6c15c920 0x7f2b6c070a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T09:27:57.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.051+0000 7f2b6b400640 1 --2- 172.21.15.6:0/803803008 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2b6c0ce850 0x7f2b6c0cc950 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f2b58009870 tx=0x7f2b5802b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fc080013f25e1670 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:57.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.052+0000 7f2b6aa00640 1 -- 172.21.15.6:0/803803008 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b58034070 con 0x7f2b6c0ce850 2024-03-04T09:27:57.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.052+0000 7f2b6aa00640 1 -- 172.21.15.6:0/803803008 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b58030840 con 0x7f2b6c0ce850 2024-03-04T09:27:57.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.052+0000 7f2b6aa00640 1 -- 172.21.15.6:0/803803008 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b58030c10 con 0x7f2b6c0ce850 2024-03-04T09:27:57.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.052+0000 7f2b71a00640 1 -- 172.21.15.6:0/803803008 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2b6c0ce850 msgr2=0x7f2b6c0cc950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:57.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.052+0000 7f2b71a00640 1 --2- 172.21.15.6:0/803803008 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2b6c0ce850 0x7f2b6c0cc950 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f2b58009870 tx=0x7f2b5802b980 comp rx=0 tx=0).stop 2024-03-04T09:27:57.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.053+0000 7f2b71a00640 1 -- 172.21.15.6:0/803803008 shutdown_connections 2024-03-04T09:27:57.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.053+0000 7f2b71a00640 1 --2- 172.21.15.6:0/803803008 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2b6c0ce850 0x7f2b6c0cc950 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:57.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.053+0000 7f2b71a00640 1 --2- 172.21.15.6:0/803803008 >> v2:172.21.15.146:3300/0 conn(0x7f2b6c15c920 0x7f2b6c070a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:57.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.053+0000 7f2b71a00640 1 -- 172.21.15.6:0/803803008 >> 172.21.15.6:0/803803008 conn(0x7f2b6c06b3d0 msgr2=0x7f2b6c06b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:57.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.053+0000 7f2b71a00640 1 -- 172.21.15.6:0/803803008 shutdown_connections 2024-03-04T09:27:57.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.053+0000 7f2b71a00640 1 -- 172.21.15.6:0/803803008 wait complete. 2024-03-04T09:27:57.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.054+0000 7f2b71a00640 1 Processor -- start 2024-03-04T09:27:57.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.055+0000 7f2b71a00640 1 -- start start 2024-03-04T09:27:57.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.056+0000 7f2b71a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2b6c070f80 0x7f2b6c15f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:57.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.056+0000 7f2b71a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2b6c0ce850 0x7f2b6c150210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:57.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.056+0000 7f2b71a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2b6c15c920 0x7f2b6c14fa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:57.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.056+0000 7f2b71a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b6c0cce90 con 0x7f2b6c070f80 2024-03-04T09:27:57.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.056+0000 7f2b71a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b6c163790 con 0x7f2b6c0ce850 2024-03-04T09:27:57.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.056+0000 7f2b71a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b6c0cd010 con 0x7f2b6c15c920 2024-03-04T09:27:57.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.056+0000 7f2b6be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2b6c070f80 0x7f2b6c15f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:57.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.056+0000 7f2b6be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2b6c070f80 0x7f2b6c15f9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42470/0 (socket says 172.21.15.6:42470) 2024-03-04T09:27:57.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.056+0000 7f2b6be00640 1 -- 172.21.15.6:0/1233430654 learned_addr learned my addr 172.21.15.6:0/1233430654 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:27:57.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.056+0000 7f2b6b400640 1 --2- 172.21.15.6:0/1233430654 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2b6c0ce850 0x7f2b6c150210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:57.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.056+0000 7f2b6be00640 1 -- 172.21.15.6:0/1233430654 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2b6c15c920 msgr2=0x7f2b6c14fa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:57.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.056+0000 7f2b71000640 1 --2- 172.21.15.6:0/1233430654 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2b6c15c920 0x7f2b6c14fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:57.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.056+0000 7f2b6be00640 1 --2- 172.21.15.6:0/1233430654 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2b6c15c920 0x7f2b6c14fa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:57.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.056+0000 7f2b6be00640 1 -- 172.21.15.6:0/1233430654 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2b6c0ce850 msgr2=0x7f2b6c150210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:57.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.056+0000 7f2b6be00640 1 --2- 172.21.15.6:0/1233430654 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2b6c0ce850 0x7f2b6c150210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:57.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.056+0000 7f2b6be00640 1 -- 172.21.15.6:0/1233430654 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b6c150750 con 0x7f2b6c070f80 2024-03-04T09:27:57.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.056+0000 7f2b6b400640 1 --2- 172.21.15.6:0/1233430654 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2b6c0ce850 0x7f2b6c150210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:27:57.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.057+0000 7f2b6be00640 1 --2- 172.21.15.6:0/1233430654 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2b6c070f80 0x7f2b6c15f9c0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f2b5c00e990 tx=0x7f2b5c00ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:57.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.057+0000 7f2b63e00640 1 -- 172.21.15.6:0/1233430654 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b5c017070 con 0x7f2b6c070f80 2024-03-04T09:27:57.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.057+0000 7f2b63e00640 1 -- 172.21.15.6:0/1233430654 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b5c00c9e0 con 0x7f2b6c070f80 2024-03-04T09:27:57.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.057+0000 7f2b63e00640 1 -- 172.21.15.6:0/1233430654 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b5c00cdb0 con 0x7f2b6c070f80 2024-03-04T09:27:57.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.057+0000 7f2b71a00640 1 -- 172.21.15.6:0/1233430654 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b6c1509d0 con 0x7f2b6c070f80 2024-03-04T09:27:57.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.058+0000 7f2b71a00640 1 -- 172.21.15.6:0/1233430654 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b6c150f90 con 0x7f2b6c070f80 2024-03-04T09:27:57.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.059+0000 7f2b71a00640 1 -- 172.21.15.6:0/1233430654 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b6c04f330 con 0x7f2b6c070f80 2024-03-04T09:27:57.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.060+0000 7f2b63e00640 1 -- 172.21.15.6:0/1233430654 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2b5c01a070 con 0x7f2b6c070f80 2024-03-04T09:27:57.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.060+0000 7f2b63e00640 1 --2- 172.21.15.6:0/1233430654 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2b4c06b4d0 0x7f2b4c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:57.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.061+0000 7f2b63e00640 1 -- 172.21.15.6:0/1233430654 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f2b5c08a1b0 con 0x7f2b6c070f80 2024-03-04T09:27:57.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.061+0000 7f2b6b400640 1 --2- 172.21.15.6:0/1233430654 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2b4c06b4d0 0x7f2b4c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:57.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.062+0000 7f2b6b400640 1 --2- 172.21.15.6:0/1233430654 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2b4c06b4d0 0x7f2b4c06d990 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f2b58036000 tx=0x7f2b5803b040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:57.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.065+0000 7f2b63e00640 1 -- 172.21.15.6:0/1233430654 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2b6c150750 con 0x7f2b6c070f80 2024-03-04T09:27:57.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.266+0000 7f2b71a00640 1 -- 172.21.15.6:0/1233430654 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f2b6c0cf3a0 con 0x7f2b6c070f80 2024-03-04T09:27:57.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.266+0000 7f2b63e00640 1 -- 172.21.15.6:0/1233430654 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v52) v1 ==== 84+0+65 (secure 0 0 0) 0x7f2b5c0592e0 con 0x7f2b6c070f80 2024-03-04T09:27:57.264 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:27:57.264 INFO:teuthology.orchestra.run.smithi006.stdout:[ 2024-03-04T09:27:57.264 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:27:57.264 INFO:teuthology.orchestra.run.smithi006.stdout: "poolnum": 1, 2024-03-04T09:27:57.265 INFO:teuthology.orchestra.run.smithi006.stdout: "poolname": ".mgr" 2024-03-04T09:27:57.265 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:27:57.265 INFO:teuthology.orchestra.run.smithi006.stdout:] 2024-03-04T09:27:57.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.269+0000 7f2b71a00640 1 -- 172.21.15.6:0/1233430654 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2b4c06b4d0 msgr2=0x7f2b4c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:57.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.269+0000 7f2b71a00640 1 --2- 172.21.15.6:0/1233430654 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2b4c06b4d0 0x7f2b4c06d990 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f2b58036000 tx=0x7f2b5803b040 comp rx=0 tx=0).stop 2024-03-04T09:27:57.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.269+0000 7f2b71a00640 1 -- 172.21.15.6:0/1233430654 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2b6c070f80 msgr2=0x7f2b6c15f9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:57.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.269+0000 7f2b71a00640 1 --2- 172.21.15.6:0/1233430654 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2b6c070f80 0x7f2b6c15f9c0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f2b5c00e990 tx=0x7f2b5c00ee50 comp rx=0 tx=0).stop 2024-03-04T09:27:57.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.269+0000 7f2b71a00640 1 -- 172.21.15.6:0/1233430654 shutdown_connections 2024-03-04T09:27:57.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.269+0000 7f2b71a00640 1 --2- 172.21.15.6:0/1233430654 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2b4c06b4d0 0x7f2b4c06d990 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:57.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.269+0000 7f2b71a00640 1 --2- 172.21.15.6:0/1233430654 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2b6c15c920 0x7f2b6c14fa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:57.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.269+0000 7f2b71a00640 1 --2- 172.21.15.6:0/1233430654 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2b6c0ce850 0x7f2b6c150210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:57.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.269+0000 7f2b71a00640 1 --2- 172.21.15.6:0/1233430654 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2b6c070f80 0x7f2b6c15f9c0 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:57.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.269+0000 7f2b71a00640 1 -- 172.21.15.6:0/1233430654 >> 172.21.15.6:0/1233430654 conn(0x7f2b6c06b3d0 msgr2=0x7f2b6c165310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:57.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.270+0000 7f2b71a00640 1 -- 172.21.15.6:0/1233430654 shutdown_connections 2024-03-04T09:27:57.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.270+0000 7f2b71a00640 1 -- 172.21.15.6:0/1233430654 wait complete. 2024-03-04T09:27:57.284 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-04T09:27:57.285 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-04T09:27:57.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.506+0000 7f5d06000640 1 Processor -- start 2024-03-04T09:27:57.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.506+0000 7f5d06000640 1 -- start start 2024-03-04T09:27:57.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.507+0000 7f5d06000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f5d000d4610 0x7f5d000d4a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:57.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.507+0000 7f5d06000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d000d1e40 con 0x7f5d000d23e0 2024-03-04T09:27:57.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.507+0000 7f5d06000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d000d2100 con 0x7f5d0015e180 2024-03-04T09:27:57.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.507+0000 7f5d06000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d000d1be0 con 0x7f5d000d4610 2024-03-04T09:27:57.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.507+0000 7f5cfea00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f5d000d4610 0x7f5d000d4a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:57.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.508+0000 7f5cfea00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f5d000d4610 0x7f5d000d4a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42474/0 (socket says 172.21.15.6:42474) 2024-03-04T09:27:57.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.508+0000 7f5cfea00640 1 -- 172.21.15.6:0/1526477253 learned_addr learned my addr 172.21.15.6:0/1526477253 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:27:57.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.508+0000 7f5cfea00640 1 -- 172.21.15.6:0/1526477253 >> v1:172.21.15.146:6789/0 conn(0x7f5d0015e180 legacy=0x7f5d000b35d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:27:57.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.508+0000 7f5cfea00640 1 -- 172.21.15.6:0/1526477253 >> v1:172.21.15.6:6789/0 conn(0x7f5d000d23e0 legacy=0x7f5d000d2850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:57.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.508+0000 7f5cfe000640 1 -- 172.21.15.6:0/1526477253 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 107656857 0 0) 0x7f5d000d1e40 con 0x7f5d000d23e0 2024-03-04T09:27:57.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.508+0000 7f5cfea00640 1 -- 172.21.15.6:0/1526477253 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d000ceed0 con 0x7f5d000d4610 2024-03-04T09:27:57.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.509+0000 7f5cfea00640 1 --2- 172.21.15.6:0/1526477253 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5d000d4610 0x7f5d000d4a30 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f5cec009870 tx=0x7f5cec02ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cc1fdfd950cc493 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:57.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.510+0000 7f5cfe000640 1 -- 172.21.15.6:0/1526477253 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cec02f840 con 0x7f5d000d4610 2024-03-04T09:27:57.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.510+0000 7f5cfe000640 1 -- 172.21.15.6:0/1526477253 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cec02f9e0 con 0x7f5d000d4610 2024-03-04T09:27:57.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.510+0000 7f5cfe000640 1 -- 172.21.15.6:0/1526477253 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cec02fdb0 con 0x7f5d000d4610 2024-03-04T09:27:57.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.510+0000 7f5d06000640 1 -- 172.21.15.6:0/1526477253 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5d000d4610 msgr2=0x7f5d000d4a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:57.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.510+0000 7f5d06000640 1 --2- 172.21.15.6:0/1526477253 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5d000d4610 0x7f5d000d4a30 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f5cec009870 tx=0x7f5cec02ba20 comp rx=0 tx=0).stop 2024-03-04T09:27:57.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.510+0000 7f5d06000640 1 -- 172.21.15.6:0/1526477253 shutdown_connections 2024-03-04T09:27:57.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.510+0000 7f5d06000640 1 --2- 172.21.15.6:0/1526477253 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5d000d4610 0x7f5d000d4a30 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:57.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.510+0000 7f5d06000640 1 -- 172.21.15.6:0/1526477253 >> 172.21.15.6:0/1526477253 conn(0x7f5d0009d2b0 msgr2=0x7f5d000d3f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:57.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.510+0000 7f5d06000640 1 -- 172.21.15.6:0/1526477253 shutdown_connections 2024-03-04T09:27:57.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.510+0000 7f5d06000640 1 -- 172.21.15.6:0/1526477253 wait complete. 2024-03-04T09:27:57.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.512+0000 7f5d06000640 1 Processor -- start 2024-03-04T09:27:57.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.512+0000 7f5d06000640 1 -- start start 2024-03-04T09:27:57.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.513+0000 7f5d06000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5d000d23e0 0x7f5d001e7ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:57.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.513+0000 7f5d06000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5d000d4610 0x7f5d0014a420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:57.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.513+0000 7f5d06000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5d0015e180 0x7f5d0014a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:57.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.513+0000 7f5d06000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d000b32a0 con 0x7f5d000d23e0 2024-03-04T09:27:57.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.513+0000 7f5d06000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d000cf3f0 con 0x7f5d0015e180 2024-03-04T09:27:57.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.513+0000 7f5d06000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d00148390 con 0x7f5d000d4610 2024-03-04T09:27:57.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.513+0000 7f5cff400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5d000d23e0 0x7f5d001e7ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:57.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.513+0000 7f5cff400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5d000d23e0 0x7f5d001e7ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42482/0 (socket says 172.21.15.6:42482) 2024-03-04T09:27:57.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.513+0000 7f5cff400640 1 -- 172.21.15.6:0/91960237 learned_addr learned my addr 172.21.15.6:0/91960237 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:27:57.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.514+0000 7f5cfea00640 1 --2- 172.21.15.6:0/91960237 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5d000d4610 0x7f5d0014a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:57.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.514+0000 7f5cffe00640 1 --2- 172.21.15.6:0/91960237 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5d0015e180 0x7f5d0014a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:57.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.514+0000 7f5cff400640 1 -- 172.21.15.6:0/91960237 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5d000d4610 msgr2=0x7f5d0014a420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:57.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.514+0000 7f5cff400640 1 --2- 172.21.15.6:0/91960237 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5d000d4610 0x7f5d0014a420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:57.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.514+0000 7f5cff400640 1 -- 172.21.15.6:0/91960237 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5d0015e180 msgr2=0x7f5d0014a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:57.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.514+0000 7f5cff400640 1 --2- 172.21.15.6:0/91960237 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5d0015e180 0x7f5d0014a960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:57.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.514+0000 7f5cff400640 1 -- 172.21.15.6:0/91960237 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d001485e0 con 0x7f5d000d23e0 2024-03-04T09:27:57.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.514+0000 7f5cfea00640 1 --2- 172.21.15.6:0/91960237 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5d000d4610 0x7f5d0014a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:27:57.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.514+0000 7f5cffe00640 1 --2- 172.21.15.6:0/91960237 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5d0015e180 0x7f5d0014a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:27:57.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.514+0000 7f5cff400640 1 --2- 172.21.15.6:0/91960237 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5d000d23e0 0x7f5d001e7ce0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f5cf000c400 tx=0x7f5cf000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:57.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.514+0000 7f5cf7400640 1 -- 172.21.15.6:0/91960237 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cf0017070 con 0x7f5d000d23e0 2024-03-04T09:27:57.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.515+0000 7f5cf7400640 1 -- 172.21.15.6:0/91960237 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cf00052d0 con 0x7f5d000d23e0 2024-03-04T09:27:57.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.515+0000 7f5d06000640 1 -- 172.21.15.6:0/91960237 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d001488c0 con 0x7f5d000d23e0 2024-03-04T09:27:57.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.515+0000 7f5cf7400640 1 -- 172.21.15.6:0/91960237 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cf0003830 con 0x7f5d000d23e0 2024-03-04T09:27:57.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.515+0000 7f5d06000640 1 -- 172.21.15.6:0/91960237 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d0015bc30 con 0x7f5d000d23e0 2024-03-04T09:27:57.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.516+0000 7f5d06000640 1 -- 172.21.15.6:0/91960237 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d00068010 con 0x7f5d000d23e0 2024-03-04T09:27:57.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.517+0000 7f5cf7400640 1 -- 172.21.15.6:0/91960237 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5cf001a070 con 0x7f5d000d23e0 2024-03-04T09:27:57.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.518+0000 7f5cf7400640 1 --2- 172.21.15.6:0/91960237 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f5ce006b500 0x7f5ce006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:57.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.518+0000 7f5cf7400640 1 -- 172.21.15.6:0/91960237 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f5cf00892a0 con 0x7f5d000d23e0 2024-03-04T09:27:57.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.519+0000 7f5cfea00640 1 --2- 172.21.15.6:0/91960237 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f5ce006b500 0x7f5ce006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:57.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.520+0000 7f5cfea00640 1 --2- 172.21.15.6:0/91960237 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f5ce006b500 0x7f5ce006d9c0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f5cec036000 tx=0x7f5cec039040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:57.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.523+0000 7f5cf7400640 1 -- 172.21.15.6:0/91960237 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5cf00583d0 con 0x7f5d000d23e0 2024-03-04T09:27:57.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:57.720+0000 7f5d06000640 1 -- 172.21.15.6:0/91960237 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f5d000d1580 con 0x7f5d000d23e0 2024-03-04T09:27:58.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:58.366+0000 7f5cf7400640 1 -- 172.21.15.6:0/91960237 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v53) v1 ==== 138+0+0 (secure 0 0 0) 0x7f5cf0057cd0 con 0x7f5d000d23e0 2024-03-04T09:27:58.363 INFO:teuthology.orchestra.run.smithi006.stderr:pool 'cephfs_metadata' created 2024-03-04T09:27:58.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:58.369+0000 7f5d06000640 1 -- 172.21.15.6:0/91960237 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f5ce006b500 msgr2=0x7f5ce006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:58.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:58.369+0000 7f5d06000640 1 --2- 172.21.15.6:0/91960237 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f5ce006b500 0x7f5ce006d9c0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f5cec036000 tx=0x7f5cec039040 comp rx=0 tx=0).stop 2024-03-04T09:27:58.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:58.369+0000 7f5d06000640 1 -- 172.21.15.6:0/91960237 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5d000d23e0 msgr2=0x7f5d001e7ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:58.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:58.369+0000 7f5d06000640 1 --2- 172.21.15.6:0/91960237 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5d000d23e0 0x7f5d001e7ce0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f5cf000c400 tx=0x7f5cf000c8c0 comp rx=0 tx=0).stop 2024-03-04T09:27:58.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:58.370+0000 7f5d06000640 1 -- 172.21.15.6:0/91960237 shutdown_connections 2024-03-04T09:27:58.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:58.370+0000 7f5d06000640 1 --2- 172.21.15.6:0/91960237 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5d0015e180 0x7f5d0014a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:58.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:58.370+0000 7f5d06000640 1 --2- 172.21.15.6:0/91960237 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f5ce006b500 0x7f5ce006d9c0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:58.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:58.370+0000 7f5d06000640 1 --2- 172.21.15.6:0/91960237 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5d000d4610 0x7f5d0014a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:58.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:58.370+0000 7f5d06000640 1 --2- 172.21.15.6:0/91960237 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5d000d23e0 0x7f5d001e7ce0 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:58.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:58.370+0000 7f5d06000640 1 -- 172.21.15.6:0/91960237 >> 172.21.15.6:0/91960237 conn(0x7f5d0009d2b0 msgr2=0x7f5d000cf880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:58.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:58.370+0000 7f5d06000640 1 -- 172.21.15.6:0/91960237 shutdown_connections 2024-03-04T09:27:58.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:58.370+0000 7f5d06000640 1 -- 172.21.15.6:0/91960237 wait complete. 2024-03-04T09:27:58.393 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-04T09:27:58.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:58.609+0000 7fabfee00640 1 Processor -- start 2024-03-04T09:27:58.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:58.609+0000 7fabfee00640 1 -- start start 2024-03-04T09:27:58.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:58.610+0000 7fabfee00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fabf815e170 0x7fabf80d2870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:58.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:58.610+0000 7fabfee00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fabf80b6280 0x7fabf8161eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:58.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:58.610+0000 7fabfee00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabf80b3d90 con 0x7fabf815e550 2024-03-04T09:27:58.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:58.611+0000 7fabfee00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fabf80cfee0 con 0x7fabf80b6280 2024-03-04T09:27:58.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:58.611+0000 7fabfee00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabf80cf640 con 0x7fabf815e170 2024-03-04T09:27:58.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:58.611+0000 7fabf7400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fabf80b6280 0x7fabf8161eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:58.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:58.611+0000 7fabf7e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fabf815e170 0x7fabf80d2870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:58.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:58.611+0000 7fabf7400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fabf80b6280 0x7fabf8161eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.6:57394/0 (socket says 172.21.15.6:57394) 2024-03-04T09:27:58.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:58.611+0000 7fabf7400640 1 -- 172.21.15.6:0/2354481975 learned_addr learned my addr 172.21.15.6:0/2354481975 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:27:58.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:58.611+0000 7fabfce00640 1 --1- 172.21.15.6:0/2354481975 >> v1:172.21.15.146:6789/0 conn(0x7fabf815e550 0x7fabf80d05c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.6:34836/0 (socket says 172.21.15.6:34836) 2024-03-04T09:27:58.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:58.612+0000 7fabf7400640 1 -- 172.21.15.6:0/2354481975 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fabf815e170 msgr2=0x7fabf80d2870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:27:58.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:58.612+0000 7fabf7400640 1 --2- 172.21.15.6:0/2354481975 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fabf815e170 0x7fabf80d2870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:58.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:58.612+0000 7fabf7400640 1 -- 172.21.15.6:0/2354481975 >> v1:172.21.15.146:6789/0 conn(0x7fabf815e550 legacy=0x7fabf80d05c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:58.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:58.612+0000 7fabf7400640 1 -- 172.21.15.6:0/2354481975 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabf80cf920 con 0x7fabf80b6280 2024-03-04T09:27:58.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:58.614+0000 7fabf7400640 1 --2- 172.21.15.6:0/2354481975 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fabf80b6280 0x7fabf8161eb0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fabe0009e50 tx=0x7fabe002bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4aede3c5e4537cb1 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:58.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:58.614+0000 7fabf6a00640 1 -- 172.21.15.6:0/2354481975 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabe002cb00 con 0x7fabf80b6280 2024-03-04T09:27:58.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:58.615+0000 7fabf6a00640 1 -- 172.21.15.6:0/2354481975 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabe002cca0 con 0x7fabf80b6280 2024-03-04T09:27:58.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:58.615+0000 7fabf6a00640 1 -- 172.21.15.6:0/2354481975 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabe0034610 con 0x7fabf80b6280 2024-03-04T09:27:58.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:58.615+0000 7fabfee00640 1 -- 172.21.15.6:0/2354481975 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fabf80b6280 msgr2=0x7fabf8161eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:58.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:58.615+0000 7fabfee00640 1 --2- 172.21.15.6:0/2354481975 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fabf80b6280 0x7fabf8161eb0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fabe0009e50 tx=0x7fabe002bf60 comp rx=0 tx=0).stop 2024-03-04T09:27:58.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:58.615+0000 7fabfee00640 1 -- 172.21.15.6:0/2354481975 shutdown_connections 2024-03-04T09:27:58.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:58.615+0000 7fabfee00640 1 --2- 172.21.15.6:0/2354481975 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fabf80b6280 0x7fabf8161eb0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:58.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:58.615+0000 7fabfee00640 1 --2- 172.21.15.6:0/2354481975 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fabf815e170 0x7fabf80d2870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:58.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:58.615+0000 7fabfee00640 1 -- 172.21.15.6:0/2354481975 >> 172.21.15.6:0/2354481975 conn(0x7fabf809d290 msgr2=0x7fabf808d4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:58.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:58.615+0000 7fabfee00640 1 -- 172.21.15.6:0/2354481975 shutdown_connections 2024-03-04T09:27:58.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:58.615+0000 7fabfee00640 1 -- 172.21.15.6:0/2354481975 wait complete. 2024-03-04T09:27:58.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:58.618+0000 7fabfee00640 1 Processor -- start 2024-03-04T09:27:58.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:58.618+0000 7fabfee00640 1 -- start start 2024-03-04T09:27:58.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:58.619+0000 7fabfee00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fabf80b6280 0x7fabf806cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:58.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:58.619+0000 7fabfee00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fabf815e170 0x7fabf806d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:58.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:58.619+0000 7fabfee00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fabf815e550 0x7fabf80c9aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:58.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:58.619+0000 7fabfee00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabf80d2f30 con 0x7fabf815e550 2024-03-04T09:27:58.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:58.619+0000 7fabfee00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabf80d2db0 con 0x7fabf80b6280 2024-03-04T09:27:58.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:58.619+0000 7fabfee00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabf80d2320 con 0x7fabf815e170 2024-03-04T09:27:58.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:58.620+0000 7fabfce00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fabf815e550 0x7fabf80c9aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:58.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:58.620+0000 7fabfce00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fabf815e550 0x7fabf80c9aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42490/0 (socket says 172.21.15.6:42490) 2024-03-04T09:27:58.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:58.620+0000 7fabfce00640 1 -- 172.21.15.6:0/1645444805 learned_addr learned my addr 172.21.15.6:0/1645444805 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:27:58.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:58.620+0000 7fabf7e00640 1 --2- 172.21.15.6:0/1645444805 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fabf80b6280 0x7fabf806cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:58.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:58.620+0000 7fabf7400640 1 --2- 172.21.15.6:0/1645444805 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fabf815e170 0x7fabf806d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:58.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:58.620+0000 7fabfce00640 1 -- 172.21.15.6:0/1645444805 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fabf815e170 msgr2=0x7fabf806d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:58.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:58.620+0000 7fabfce00640 1 --2- 172.21.15.6:0/1645444805 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fabf815e170 0x7fabf806d240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:58.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:58.620+0000 7fabfce00640 1 -- 172.21.15.6:0/1645444805 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fabf80b6280 msgr2=0x7fabf806cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:58.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:58.620+0000 7fabfce00640 1 --2- 172.21.15.6:0/1645444805 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fabf80b6280 0x7fabf806cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:58.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:58.620+0000 7fabfce00640 1 -- 172.21.15.6:0/1645444805 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabf81f7970 con 0x7fabf815e550 2024-03-04T09:27:58.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:58.620+0000 7fabfce00640 1 --2- 172.21.15.6:0/1645444805 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fabf815e550 0x7fabf80c9aa0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fabf000c3f0 tx=0x7fabf000c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:58.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:58.621+0000 7fabefe00640 1 -- 172.21.15.6:0/1645444805 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabf00074e0 con 0x7fabf815e550 2024-03-04T09:27:58.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:58.621+0000 7fabefe00640 1 -- 172.21.15.6:0/1645444805 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabf0007680 con 0x7fabf815e550 2024-03-04T09:27:58.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:58.621+0000 7fabefe00640 1 -- 172.21.15.6:0/1645444805 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabf0004a20 con 0x7fabf815e550 2024-03-04T09:27:58.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:58.621+0000 7fabfee00640 1 -- 172.21.15.6:0/1645444805 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabf81f7c50 con 0x7fabf815e550 2024-03-04T09:27:58.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:58.621+0000 7fabfee00640 1 -- 172.21.15.6:0/1645444805 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabf81f8210 con 0x7fabf815e550 2024-03-04T09:27:58.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:58.622+0000 7fabfee00640 1 -- 172.21.15.6:0/1645444805 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabf8068010 con 0x7fabf815e550 2024-03-04T09:27:58.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:58.623+0000 7fabefe00640 1 -- 172.21.15.6:0/1645444805 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fabf001a020 con 0x7fabf815e550 2024-03-04T09:27:58.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:58.624+0000 7fabefe00640 1 --2- 172.21.15.6:0/1645444805 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fabd406b4d0 0x7fabd406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:58.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:58.624+0000 7fabefe00640 1 -- 172.21.15.6:0/1645444805 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 5227+0+0 (secure 0 0 0) 0x7fabf0089650 con 0x7fabf815e550 2024-03-04T09:27:58.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:58.625+0000 7fabf7e00640 1 --2- 172.21.15.6:0/1645444805 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fabd406b4d0 0x7fabd406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:58.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:58.626+0000 7fabf7e00640 1 --2- 172.21.15.6:0/1645444805 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fabd406b4d0 0x7fabd406d990 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fabe8009ad0 tx=0x7fabe8008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:58.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:58.632+0000 7fabefe00640 1 -- 172.21.15.6:0/1645444805 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fabf0010300 con 0x7fabf815e550 2024-03-04T09:27:58.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:58.832+0000 7fabfee00640 1 -- 172.21.15.6:0/1645444805 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fabf815bba0 con 0x7fabf815e550 2024-03-04T09:27:59.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:59.373+0000 7fabefe00640 1 -- 172.21.15.6:0/1645444805 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v54) v1 ==== 170+0+0 (secure 0 0 0) 0x7fabf0058800 con 0x7fabf815e550 2024-03-04T09:27:59.371 INFO:teuthology.orchestra.run.smithi006.stderr:pool 'cephfs_data' created 2024-03-04T09:27:59.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:59.376+0000 7fabfee00640 1 -- 172.21.15.6:0/1645444805 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fabd406b4d0 msgr2=0x7fabd406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:59.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:59.376+0000 7fabfee00640 1 --2- 172.21.15.6:0/1645444805 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fabd406b4d0 0x7fabd406d990 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fabe8009ad0 tx=0x7fabe8008040 comp rx=0 tx=0).stop 2024-03-04T09:27:59.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:59.377+0000 7fabfee00640 1 -- 172.21.15.6:0/1645444805 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fabf815e550 msgr2=0x7fabf80c9aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:59.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:59.377+0000 7fabfee00640 1 --2- 172.21.15.6:0/1645444805 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fabf815e550 0x7fabf80c9aa0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fabf000c3f0 tx=0x7fabf000c8b0 comp rx=0 tx=0).stop 2024-03-04T09:27:59.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:59.377+0000 7fabfee00640 1 -- 172.21.15.6:0/1645444805 shutdown_connections 2024-03-04T09:27:59.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:59.377+0000 7fabfee00640 1 --2- 172.21.15.6:0/1645444805 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fabf815e550 0x7fabf80c9aa0 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:59.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:59.377+0000 7fabfee00640 1 --2- 172.21.15.6:0/1645444805 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fabd406b4d0 0x7fabd406d990 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:59.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:59.377+0000 7fabfee00640 1 --2- 172.21.15.6:0/1645444805 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fabf815e170 0x7fabf806d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:59.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:59.377+0000 7fabfee00640 1 --2- 172.21.15.6:0/1645444805 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fabf80b6280 0x7fabf806cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:59.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:59.377+0000 7fabfee00640 1 -- 172.21.15.6:0/1645444805 >> 172.21.15.6:0/1645444805 conn(0x7fabf809d290 msgr2=0x7fabf8073f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:59.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:59.377+0000 7fabfee00640 1 -- 172.21.15.6:0/1645444805 shutdown_connections 2024-03-04T09:27:59.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:59.377+0000 7fabfee00640 1 -- 172.21.15.6:0/1645444805 wait complete. 2024-03-04T09:27:59.405 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-04T09:27:59.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:59.595+0000 7f1ae3200640 1 Processor -- start 2024-03-04T09:27:59.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:59.595+0000 7f1ae3200640 1 -- start start 2024-03-04T09:27:59.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:59.596+0000 7f1ae3200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f1ae406f540 0x7f1ae406f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:59.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:59.597+0000 7f1ae3200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1ae40af930 0x7f1ae40cfa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:59.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:59.597+0000 7f1ae3200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ae406fe60 con 0x7f1ae40b4530 2024-03-04T09:27:59.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:59.597+0000 7f1ae3200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ae40b3520 con 0x7f1ae406f540 2024-03-04T09:27:59.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:59.597+0000 7f1ae3200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ae40b3770 con 0x7f1ae40af930 2024-03-04T09:27:59.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:59.597+0000 7f1ae1e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f1ae406f540 0x7f1ae406f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:59.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:59.597+0000 7f1ae2800640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f1ae40b4530 0x7f1ae40b49c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:39780/0 (socket says 172.21.15.6:39780) 2024-03-04T09:27:59.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:59.597+0000 7f1ae2800640 1 -- 172.21.15.6:0/988924558 learned_addr learned my addr 172.21.15.6:0/988924558 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:27:59.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:59.597+0000 7f1ae1400640 1 --2- 172.21.15.6:0/988924558 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1ae40af930 0x7f1ae40cfa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:59.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:59.597+0000 7f1ae0a00640 1 -- 172.21.15.6:0/988924558 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2034353142 0 0) 0x7f1ae406fe60 con 0x7f1ae40b4530 2024-03-04T09:27:59.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:59.597+0000 7f1ae1e00640 1 -- 172.21.15.6:0/988924558 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1ae40af930 msgr2=0x7f1ae40cfa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:27:59.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:59.597+0000 7f1ae1e00640 1 --2- 172.21.15.6:0/988924558 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1ae40af930 0x7f1ae40cfa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:59.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:59.597+0000 7f1ae1e00640 1 -- 172.21.15.6:0/988924558 >> v1:172.21.15.6:6789/0 conn(0x7f1ae40b4530 legacy=0x7f1ae40b49c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:59.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:59.597+0000 7f1ae1e00640 1 -- 172.21.15.6:0/988924558 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ae40b3a50 con 0x7f1ae406f540 2024-03-04T09:27:59.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:59.599+0000 7f1ae1e00640 1 --2- 172.21.15.6:0/988924558 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1ae406f540 0x7f1ae406f920 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f1ad40097a0 tx=0x7f1ad402b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d28e7d8dc7735d84 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:59.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:59.599+0000 7f1ae0a00640 1 -- 172.21.15.6:0/988924558 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ad4035020 con 0x7f1ae406f540 2024-03-04T09:27:59.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:59.599+0000 7f1ae0a00640 1 -- 172.21.15.6:0/988924558 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ad4031810 con 0x7f1ae406f540 2024-03-04T09:27:59.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:59.599+0000 7f1ae0a00640 1 -- 172.21.15.6:0/988924558 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ad4031be0 con 0x7f1ae406f540 2024-03-04T09:27:59.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:59.599+0000 7f1ae3200640 1 -- 172.21.15.6:0/988924558 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1ae406f540 msgr2=0x7f1ae406f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:59.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:59.599+0000 7f1ae3200640 1 --2- 172.21.15.6:0/988924558 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1ae406f540 0x7f1ae406f920 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f1ad40097a0 tx=0x7f1ad402b8b0 comp rx=0 tx=0).stop 2024-03-04T09:27:59.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:59.599+0000 7f1ae3200640 1 -- 172.21.15.6:0/988924558 shutdown_connections 2024-03-04T09:27:59.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:59.599+0000 7f1ae3200640 1 --2- 172.21.15.6:0/988924558 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1ae40af930 0x7f1ae40cfa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:59.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:59.599+0000 7f1ae3200640 1 --2- 172.21.15.6:0/988924558 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1ae406f540 0x7f1ae406f920 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:59.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:59.599+0000 7f1ae3200640 1 -- 172.21.15.6:0/988924558 >> 172.21.15.6:0/988924558 conn(0x7f1ae40723d0 msgr2=0x7f1ae406afb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:27:59.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:59.600+0000 7f1ae3200640 1 -- 172.21.15.6:0/988924558 shutdown_connections 2024-03-04T09:27:59.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:59.600+0000 7f1ae3200640 1 -- 172.21.15.6:0/988924558 wait complete. 2024-03-04T09:27:59.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:59.601+0000 7f1ae3200640 1 Processor -- start 2024-03-04T09:27:59.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:59.601+0000 7f1ae3200640 1 -- start start 2024-03-04T09:27:59.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:59.602+0000 7f1ae3200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1ae406f540 0x7f1ae4146dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:59.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:59.602+0000 7f1ae3200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1ae40af930 0x7f1ae414bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:59.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:59.602+0000 7f1ae3200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1ae40b4530 0x7f1ae41509c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:59.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:59.602+0000 7f1ae3200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ae40cffd0 con 0x7f1ae40af930 2024-03-04T09:27:59.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:59.602+0000 7f1ae3200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ae40b4010 con 0x7f1ae406f540 2024-03-04T09:27:59.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:59.602+0000 7f1ae3200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ae415cf00 con 0x7f1ae40b4530 2024-03-04T09:27:59.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:59.602+0000 7f1ae1400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1ae40af930 0x7f1ae414bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:59.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:59.603+0000 7f1ae1400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1ae40af930 0x7f1ae414bb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45200/0 (socket says 172.21.15.6:45200) 2024-03-04T09:27:59.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:59.603+0000 7f1ae1400640 1 -- 172.21.15.6:0/1493996466 learned_addr learned my addr 172.21.15.6:0/1493996466 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:27:59.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:59.603+0000 7f1ae1e00640 1 --2- 172.21.15.6:0/1493996466 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1ae406f540 0x7f1ae4146dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:59.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:59.603+0000 7f1ae1400640 1 -- 172.21.15.6:0/1493996466 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1ae40b4530 msgr2=0x7f1ae41509c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:59.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:59.603+0000 7f1ae1400640 1 --2- 172.21.15.6:0/1493996466 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1ae40b4530 0x7f1ae41509c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:59.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:59.603+0000 7f1ae1400640 1 -- 172.21.15.6:0/1493996466 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1ae406f540 msgr2=0x7f1ae4146dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:27:59.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:59.603+0000 7f1ae1400640 1 --2- 172.21.15.6:0/1493996466 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1ae406f540 0x7f1ae4146dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:27:59.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:59.603+0000 7f1ae1400640 1 -- 172.21.15.6:0/1493996466 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ae414c530 con 0x7f1ae40af930 2024-03-04T09:27:59.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:59.603+0000 7f1ae1e00640 1 --2- 172.21.15.6:0/1493996466 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1ae406f540 0x7f1ae4146dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:27:59.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:59.603+0000 7f1ae1400640 1 --2- 172.21.15.6:0/1493996466 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1ae40af930 0x7f1ae414bb50 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f1ad000d800 tx=0x7f1ad000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:59.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:59.603+0000 7f1ada000640 1 -- 172.21.15.6:0/1493996466 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ad000b6c0 con 0x7f1ae40af930 2024-03-04T09:27:59.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:59.604+0000 7f1ada000640 1 -- 172.21.15.6:0/1493996466 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ad0019070 con 0x7f1ae40af930 2024-03-04T09:27:59.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:59.604+0000 7f1ada000640 1 -- 172.21.15.6:0/1493996466 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ad000ba80 con 0x7f1ae40af930 2024-03-04T09:27:59.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:59.604+0000 7f1ae3200640 1 -- 172.21.15.6:0/1493996466 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ae414c810 con 0x7f1ae40af930 2024-03-04T09:27:59.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:59.604+0000 7f1ae3200640 1 -- 172.21.15.6:0/1493996466 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ae41504a0 con 0x7f1ae40af930 2024-03-04T09:27:59.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:59.605+0000 7f1ae3200640 1 -- 172.21.15.6:0/1493996466 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ae404f330 con 0x7f1ae40af930 2024-03-04T09:27:59.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:59.607+0000 7f1ada000640 1 -- 172.21.15.6:0/1493996466 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1ad000bc20 con 0x7f1ae40af930 2024-03-04T09:27:59.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:59.607+0000 7f1ada000640 1 --2- 172.21.15.6:0/1493996466 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f1ac406b4d0 0x7f1ac406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:27:59.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:59.607+0000 7f1ada000640 1 -- 172.21.15.6:0/1493996466 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f1ad008b110 con 0x7f1ae40af930 2024-03-04T09:27:59.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:59.608+0000 7f1ae1e00640 1 --2- 172.21.15.6:0/1493996466 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f1ac406b4d0 0x7f1ac406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:27:59.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:59.609+0000 7f1ae1e00640 1 --2- 172.21.15.6:0/1493996466 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f1ac406b4d0 0x7f1ac406d990 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f1ad4036000 tx=0x7f1ad4008030 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:27:59.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:59.612+0000 7f1ada000640 1 -- 172.21.15.6:0/1493996466 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1ad005a060 con 0x7f1ae40af930 2024-03-04T09:27:59.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:27:59.852+0000 7f1ae3200640 1 -- 172.21.15.6:0/1493996466 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f1ae406c980 con 0x7f1ae40af930 2024-03-04T09:28:00.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:00.412+0000 7f1ada000640 1 -- 172.21.15.6:0/1493996466 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 8 and data pool 9 v30) v1 ==== 172+0+0 (secure 0 0 0) 0x7f1ad0059960 con 0x7f1ae40af930 2024-03-04T09:28:00.411 INFO:teuthology.orchestra.run.smithi006.stderr:new fs with metadata pool 8 and data pool 9 2024-03-04T09:28:00.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:00.420+0000 7f1ae3200640 1 -- 172.21.15.6:0/1493996466 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f1ac406b4d0 msgr2=0x7f1ac406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:00.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:00.420+0000 7f1ae3200640 1 --2- 172.21.15.6:0/1493996466 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f1ac406b4d0 0x7f1ac406d990 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f1ad4036000 tx=0x7f1ad4008030 comp rx=0 tx=0).stop 2024-03-04T09:28:00.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:00.420+0000 7f1ae3200640 1 -- 172.21.15.6:0/1493996466 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1ae40af930 msgr2=0x7f1ae414bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:00.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:00.420+0000 7f1ae3200640 1 --2- 172.21.15.6:0/1493996466 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1ae40af930 0x7f1ae414bb50 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f1ad000d800 tx=0x7f1ad000dcc0 comp rx=0 tx=0).stop 2024-03-04T09:28:00.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:00.421+0000 7f1ae3200640 1 -- 172.21.15.6:0/1493996466 shutdown_connections 2024-03-04T09:28:00.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:00.421+0000 7f1ae3200640 1 --2- 172.21.15.6:0/1493996466 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f1ac406b4d0 0x7f1ac406d990 secure :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f1ad4036000 tx=0x7f1ad4008030 comp rx=0 tx=0).stop 2024-03-04T09:28:00.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:00.421+0000 7f1ae3200640 1 --2- 172.21.15.6:0/1493996466 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1ae40b4530 0x7f1ae41509c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:00.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:00.421+0000 7f1ae3200640 1 --2- 172.21.15.6:0/1493996466 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1ae40af930 0x7f1ae414bb50 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:00.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:00.421+0000 7f1ae3200640 1 --2- 172.21.15.6:0/1493996466 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1ae406f540 0x7f1ae4146dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:00.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:00.421+0000 7f1ae3200640 1 -- 172.21.15.6:0/1493996466 >> 172.21.15.6:0/1493996466 conn(0x7f1ae40723d0 msgr2=0x7f1ae40ad880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:00.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:00.422+0000 7f1ae3200640 1 -- 172.21.15.6:0/1493996466 shutdown_connections 2024-03-04T09:28:00.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:00.422+0000 7f1ae3200640 1 -- 172.21.15.6:0/1493996466 wait complete. 2024-03-04T09:28:00.456 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T09:28:00.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:00.641+0000 7fa84e800640 1 Processor -- start 2024-03-04T09:28:00.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:00.642+0000 7fa84e800640 1 -- start start 2024-03-04T09:28:00.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:00.643+0000 7fa84e800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa84815bf30 0x7fa84806f930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:00.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:00.643+0000 7fa84e800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa8480b5160 0x7fa8480b57a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:00.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:00.643+0000 7fa84e800640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8480af420 con 0x7fa8480b5ce0 2024-03-04T09:28:00.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:00.643+0000 7fa84e800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8480b6170 con 0x7fa8480b5160 2024-03-04T09:28:00.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:00.643+0000 7fa84e800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa848070230 con 0x7fa84815bf30 2024-03-04T09:28:00.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:00.644+0000 7fa846a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa8480b5160 0x7fa8480b57a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:00.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:00.644+0000 7fa846a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa8480b5160 0x7fa8480b57a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45214/0 (socket says 172.21.15.6:45214) 2024-03-04T09:28:00.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:00.644+0000 7fa846a00640 1 -- 172.21.15.6:0/2633973771 learned_addr learned my addr 172.21.15.6:0/2633973771 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:28:00.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:00.644+0000 7fa847400640 1 --2- 172.21.15.6:0/2633973771 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa84815bf30 0x7fa84806f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:00.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:00.644+0000 7fa846a00640 1 -- 172.21.15.6:0/2633973771 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa84815bf30 msgr2=0x7fa84806f930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:00.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:00.644+0000 7fa846a00640 1 --2- 172.21.15.6:0/2633973771 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa84815bf30 0x7fa84806f930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:00.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:00.644+0000 7fa846a00640 1 -- 172.21.15.6:0/2633973771 >> v1:172.21.15.146:6789/0 conn(0x7fa8480b5ce0 legacy=0x7fa84815fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:00.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:00.644+0000 7fa846a00640 1 -- 172.21.15.6:0/2633973771 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa848070510 con 0x7fa8480b5160 2024-03-04T09:28:00.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:00.645+0000 7fa846a00640 1 --2- 172.21.15.6:0/2633973771 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa8480b5160 0x7fa8480b57a0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fa830009870 tx=0x7fa83002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc66013ea8236e3f server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:00.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:00.646+0000 7fa846000640 1 -- 172.21.15.6:0/2633973771 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa830035020 con 0x7fa8480b5160 2024-03-04T09:28:00.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:00.646+0000 7fa846000640 1 -- 172.21.15.6:0/2633973771 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa830031640 con 0x7fa8480b5160 2024-03-04T09:28:00.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:00.646+0000 7fa846000640 1 -- 172.21.15.6:0/2633973771 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa83002d040 con 0x7fa8480b5160 2024-03-04T09:28:00.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:00.646+0000 7fa84e800640 1 -- 172.21.15.6:0/2633973771 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa8480b5160 msgr2=0x7fa8480b57a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:00.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:00.646+0000 7fa84e800640 1 --2- 172.21.15.6:0/2633973771 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa8480b5160 0x7fa8480b57a0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fa830009870 tx=0x7fa83002b8f0 comp rx=0 tx=0).stop 2024-03-04T09:28:00.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:00.646+0000 7fa84e800640 1 -- 172.21.15.6:0/2633973771 shutdown_connections 2024-03-04T09:28:00.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:00.646+0000 7fa84e800640 1 --2- 172.21.15.6:0/2633973771 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa8480b5160 0x7fa8480b57a0 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:00.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:00.646+0000 7fa84e800640 1 --2- 172.21.15.6:0/2633973771 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa84815bf30 0x7fa84806f930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:00.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:00.646+0000 7fa84e800640 1 -- 172.21.15.6:0/2633973771 >> 172.21.15.6:0/2633973771 conn(0x7fa84809aa10 msgr2=0x7fa84809ae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:00.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:00.647+0000 7fa84e800640 1 -- 172.21.15.6:0/2633973771 shutdown_connections 2024-03-04T09:28:00.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:00.647+0000 7fa84e800640 1 -- 172.21.15.6:0/2633973771 wait complete. 2024-03-04T09:28:00.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:00.648+0000 7fa84e800640 1 Processor -- start 2024-03-04T09:28:00.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:00.649+0000 7fa84e800640 1 -- start start 2024-03-04T09:28:00.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:00.649+0000 7fa84e800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa8480b5160 0x7fa8481e8e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:00.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:00.649+0000 7fa84e800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa8480b5ce0 0x7fa848143660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:00.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:00.650+0000 7fa84e800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa84815bf30 0x7fa8481ea2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:00.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:00.650+0000 7fa84e800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa84815c310 con 0x7fa8480b5160 2024-03-04T09:28:00.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:00.650+0000 7fa84e800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa848070ad0 con 0x7fa8480b5ce0 2024-03-04T09:28:00.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:00.650+0000 7fa84e800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa84806fe70 con 0x7fa84815bf30 2024-03-04T09:28:00.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:00.650+0000 7fa847400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa8480b5160 0x7fa8481e8e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:00.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:00.650+0000 7fa847400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa8480b5160 0x7fa8481e8e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45230/0 (socket says 172.21.15.6:45230) 2024-03-04T09:28:00.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:00.650+0000 7fa847400640 1 -- 172.21.15.6:0/2960329892 learned_addr learned my addr 172.21.15.6:0/2960329892 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:28:00.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:00.650+0000 7fa847e00640 1 --2- 172.21.15.6:0/2960329892 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa84815bf30 0x7fa8481ea2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:00.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:00.650+0000 7fa846a00640 1 --2- 172.21.15.6:0/2960329892 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa8480b5ce0 0x7fa848143660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:00.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:00.650+0000 7fa847400640 1 -- 172.21.15.6:0/2960329892 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa84815bf30 msgr2=0x7fa8481ea2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:00.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:00.650+0000 7fa847400640 1 --2- 172.21.15.6:0/2960329892 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa84815bf30 0x7fa8481ea2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:00.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:00.650+0000 7fa847400640 1 -- 172.21.15.6:0/2960329892 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa8480b5ce0 msgr2=0x7fa848143660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:00.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:00.650+0000 7fa847400640 1 --2- 172.21.15.6:0/2960329892 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa8480b5ce0 0x7fa848143660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:00.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:00.650+0000 7fa847400640 1 -- 172.21.15.6:0/2960329892 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8481e9360 con 0x7fa8480b5160 2024-03-04T09:28:00.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:00.650+0000 7fa847e00640 1 --2- 172.21.15.6:0/2960329892 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa84815bf30 0x7fa8481ea2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:28:00.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:00.651+0000 7fa847400640 1 --2- 172.21.15.6:0/2960329892 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa8480b5160 0x7fa8481e8e20 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fa83800d5f0 tx=0x7fa83800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:00.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:00.651+0000 7fa83f400640 1 -- 172.21.15.6:0/2960329892 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8380125f0 con 0x7fa8480b5160 2024-03-04T09:28:00.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:00.651+0000 7fa83f400640 1 -- 172.21.15.6:0/2960329892 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa838019070 con 0x7fa8480b5160 2024-03-04T09:28:00.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:00.651+0000 7fa84e800640 1 -- 172.21.15.6:0/2960329892 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8481e7d80 con 0x7fa8480b5160 2024-03-04T09:28:00.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:00.651+0000 7fa83f400640 1 -- 172.21.15.6:0/2960329892 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8380129d0 con 0x7fa8480b5160 2024-03-04T09:28:00.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:00.651+0000 7fa84e800640 1 -- 172.21.15.6:0/2960329892 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8481e8620 con 0x7fa8480b5160 2024-03-04T09:28:00.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:00.652+0000 7fa84e800640 1 -- 172.21.15.6:0/2960329892 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa84804f3a0 con 0x7fa8480b5160 2024-03-04T09:28:00.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:00.653+0000 7fa83f400640 1 -- 172.21.15.6:0/2960329892 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa838012b70 con 0x7fa8480b5160 2024-03-04T09:28:00.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:00.654+0000 7fa83f400640 1 --2- 172.21.15.6:0/2960329892 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa82406b4d0 0x7fa82406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:00.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:00.654+0000 7fa83f400640 1 -- 172.21.15.6:0/2960329892 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa83808a6b0 con 0x7fa8480b5160 2024-03-04T09:28:00.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:00.655+0000 7fa846a00640 1 --2- 172.21.15.6:0/2960329892 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa82406b4d0 0x7fa82406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:00.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:00.656+0000 7fa846a00640 1 --2- 172.21.15.6:0/2960329892 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa82406b4d0 0x7fa82406d990 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fa830036000 tx=0x7fa83003c040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:00.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:00.659+0000 7fa83f400640 1 -- 172.21.15.6:0/2960329892 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa838059490 con 0x7fa8480b5160 2024-03-04T09:28:00.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:00.856+0000 7fa84e800640 1 -- 172.21.15.6:0/2960329892 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa8480ce240 con 0x7fa8480b5160 2024-03-04T09:28:00.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:00.859+0000 7fa83f400640 1 -- 172.21.15.6:0/2960329892 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v55) v1 ==== 74+0+16630 (secure 0 0 0) 0x7fa838058d90 con 0x7fa8480b5160 2024-03-04T09:28:00.856 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:28:00.856 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":55,"fsid":"edce0578-5ae4-4402-b45a-552bdef84d7c","created":"2024-03-04T09:24:54.523780+0000","modified":"2024-03-04T09:27:59.853585+0000","last_up_change":"2024-03-04T09:25:07.089049+0000","last_in_change":"2024-03-04T09:24:57.982077+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T09:25:07.935198+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-04T09:27:57.774529+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-04T09:27:58.886504+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a96574f-81f1-4bc6-8463-757520b84a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6825","nonce":1364449401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6827","nonce":1364449401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6831","nonce":1364449401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6829","nonce":1364449401}]},"public_addr":"172.21.15.6:6825/1364449401","cluster_addr":"172.21.15.6:6827/1364449401","heartbeat_back_addr":"172.21.15.6:6831/1364449401","heartbeat_front_addr":"172.21.15.6:6829/1364449401","state":["exists","up"]},{"osd":1,"uuid":"7d79b9c1-6b87-476a-bc36-7db5ee64cd90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6809","nonce":2494103078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6811","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6813","nonce":2494103078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6821","nonce":2494103078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6817","nonce":2494103078}]},"public_addr":"172.21.15.6:6809/2494103078","cluster_addr":"172.21.15.6:6813/2494103078","heartbeat_back_addr":"172.21.15.6:6821/2494103078","heartbeat_front_addr":"172.21.15.6:6817/2494103078","state":["exists","up"]},{"osd":2,"uuid":"492331b9-4ee9-40b6-bacd-85d44cd6c45a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6812","nonce":1500066681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6816","nonce":1500066681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6824","nonce":1500066681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6820","nonce":1500066681}]},"public_addr":"172.21.15.6:6812/1500066681","cluster_addr":"172.21.15.6:6816/1500066681","heartbeat_back_addr":"172.21.15.6:6824/1500066681","heartbeat_front_addr":"172.21.15.6:6820/1500066681","state":["exists","up"]},{"osd":3,"uuid":"21bb2efe-9ba7-4bdc-bc31-180cd1102e21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6801","nonce":1558806123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6803","nonce":1558806123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6807","nonce":1558806123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6805","nonce":1558806123}]},"public_addr":"172.21.15.6:6801/1558806123","cluster_addr":"172.21.15.6:6803/1558806123","heartbeat_back_addr":"172.21.15.6:6807/1558806123","heartbeat_front_addr":"172.21.15.6:6805/1558806123","state":["exists","up"]},{"osd":4,"uuid":"6e2c4b71-19d9-4737-83d9-558705d8644a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6801","nonce":1526135780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6803","nonce":1526135780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6807","nonce":1526135780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6805","nonce":1526135780}]},"public_addr":"172.21.15.146:6801/1526135780","cluster_addr":"172.21.15.146:6803/1526135780","heartbeat_back_addr":"172.21.15.146:6807/1526135780","heartbeat_front_addr":"172.21.15.146:6805/1526135780","state":["exists","up"]},{"osd":5,"uuid":"fb69c525-f1be-4b7d-ab32-6c29b0d6fddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6809","nonce":460262507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6811","nonce":460262507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6815","nonce":460262507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6813","nonce":460262507}]},"public_addr":"172.21.15.146:6809/460262507","cluster_addr":"172.21.15.146:6811/460262507","heartbeat_back_addr":"172.21.15.146:6815/460262507","heartbeat_front_addr":"172.21.15.146:6813/460262507","state":["exists","up"]},{"osd":6,"uuid":"c32e7fed-370d-4401-8ce0-6a4335b0d168","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6823","nonce":1392897493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6827","nonce":1392897493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6831","nonce":1392897493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6829","nonce":1392897493}]},"public_addr":"172.21.15.146:6823/1392897493","cluster_addr":"172.21.15.146:6827/1392897493","heartbeat_back_addr":"172.21.15.146:6831/1392897493","heartbeat_front_addr":"172.21.15.146:6829/1392897493","state":["exists","up"]},{"osd":7,"uuid":"75a79b44-fa5f-426d-a7d3-71f21f78835c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6817","nonce":3268323156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6819","nonce":3268323156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6825","nonce":3268323156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6822","nonce":3268323156}]},"public_addr":"172.21.15.146:6817/3268323156","cluster_addr":"172.21.15.146:6819/3268323156","heartbeat_back_addr":"172.21.15.146:6825/3268323156","heartbeat_front_addr":"172.21.15.146:6822/3268323156","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.310437+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.295850+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.294674+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.526855+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.300023+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.595160+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.346046+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.558741+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T09:28:00.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:00.861+0000 7fa84e800640 1 -- 172.21.15.6:0/2960329892 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa82406b4d0 msgr2=0x7fa82406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:00.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:00.861+0000 7fa84e800640 1 --2- 172.21.15.6:0/2960329892 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa82406b4d0 0x7fa82406d990 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fa830036000 tx=0x7fa83003c040 comp rx=0 tx=0).stop 2024-03-04T09:28:00.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:00.861+0000 7fa84e800640 1 -- 172.21.15.6:0/2960329892 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa8480b5160 msgr2=0x7fa8481e8e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:00.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:00.861+0000 7fa84e800640 1 --2- 172.21.15.6:0/2960329892 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa8480b5160 0x7fa8481e8e20 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fa83800d5f0 tx=0x7fa83800dab0 comp rx=0 tx=0).stop 2024-03-04T09:28:00.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:00.861+0000 7fa84e800640 1 -- 172.21.15.6:0/2960329892 shutdown_connections 2024-03-04T09:28:00.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:00.861+0000 7fa84e800640 1 --2- 172.21.15.6:0/2960329892 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa82406b4d0 0x7fa82406d990 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:00.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:00.861+0000 7fa84e800640 1 --2- 172.21.15.6:0/2960329892 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa84815bf30 0x7fa8481ea2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:00.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:00.861+0000 7fa84e800640 1 --2- 172.21.15.6:0/2960329892 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa8480b5ce0 0x7fa848143660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:00.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:00.861+0000 7fa84e800640 1 --2- 172.21.15.6:0/2960329892 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa8480b5160 0x7fa8481e8e20 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:00.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:00.861+0000 7fa84e800640 1 -- 172.21.15.6:0/2960329892 >> 172.21.15.6:0/2960329892 conn(0x7fa84809aa10 msgr2=0x7fa84806d710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:00.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:00.862+0000 7fa84e800640 1 -- 172.21.15.6:0/2960329892 shutdown_connections 2024-03-04T09:28:00.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:00.862+0000 7fa84e800640 1 -- 172.21.15.6:0/2960329892 wait complete. 2024-03-04T09:28:00.876 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T09:28:01.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.096+0000 7fcbc1a00640 1 Processor -- start 2024-03-04T09:28:01.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.097+0000 7fcbc1a00640 1 -- start start 2024-03-04T09:28:01.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.098+0000 7fcbc1a00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fcbbc15e070 0x7fcbbc0d2960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:01.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.098+0000 7fcbc1a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbbc0d18c0 con 0x7fcbbc0cf410 2024-03-04T09:28:01.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.098+0000 7fcbc1a00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbbc0d1ba0 con 0x7fcbbc0d0cd0 2024-03-04T09:28:01.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.098+0000 7fcbc1a00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcbbc0d1e80 con 0x7fcbbc15e070 2024-03-04T09:28:01.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.098+0000 7fcbbb400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fcbbc0cf410 0x7fcbbc0d05c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:39794/0 (socket says 172.21.15.6:39794) 2024-03-04T09:28:01.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.099+0000 7fcbbb400640 1 -- 172.21.15.6:0/2852154875 learned_addr learned my addr 172.21.15.6:0/2852154875 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:28:01.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.099+0000 7fcbbbe00640 1 --2- 172.21.15.6:0/2852154875 >> v2:172.21.15.146:3300/0 conn(0x7fcbbc15e070 0x7fcbbc0d2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:01.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.099+0000 7fcbbaa00640 1 -- 172.21.15.6:0/2852154875 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3063513359 0 0) 0x7fcbbc0d18c0 con 0x7fcbbc0cf410 2024-03-04T09:28:01.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.099+0000 7fcbbaa00640 1 -- 172.21.15.6:0/2852154875 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcba0003520 con 0x7fcbbc0cf410 2024-03-04T09:28:01.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.099+0000 7fcbbaa00640 1 -- 172.21.15.6:0/2852154875 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 186858912 0 0) 0x7fcba0003520 con 0x7fcbbc0cf410 2024-03-04T09:28:01.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.099+0000 7fcbbaa00640 1 -- 172.21.15.6:0/2852154875 >> v2:172.21.15.146:3300/0 conn(0x7fcbbc15e070 msgr2=0x7fcbbc0d2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:01.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.099+0000 7fcbbaa00640 1 --2- 172.21.15.6:0/2852154875 >> v2:172.21.15.146:3300/0 conn(0x7fcbbc15e070 0x7fcbbc0d2960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:01.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.099+0000 7fcbbaa00640 1 -- 172.21.15.6:0/2852154875 >> v1:172.21.15.146:6789/0 conn(0x7fcbbc0d0cd0 legacy=0x7fcbbc0b3630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:01.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.100+0000 7fcbbaa00640 1 -- 172.21.15.6:0/2852154875 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbbc0d2160 con 0x7fcbbc0cf410 2024-03-04T09:28:01.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.100+0000 7fcbbaa00640 1 -- 172.21.15.6:0/2852154875 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7fcba8001670 con 0x7fcbbc0cf410 2024-03-04T09:28:01.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.100+0000 7fcbbaa00640 1 -- 172.21.15.6:0/2852154875 >> v1:172.21.15.6:6789/0 conn(0x7fcbbc0cf410 legacy=0x7fcbbc0d05c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:01.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.100+0000 7fcbbaa00640 1 --2- 172.21.15.6:0/2852154875 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcba0003a00 0x7fcba0003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:01.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.100+0000 7fcbbaa00640 1 --2- 172.21.15.6:0/2852154875 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcba0004800 0x7fcba0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:01.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.100+0000 7fcbbaa00640 1 --2- 172.21.15.6:0/2852154875 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcba00051c0 0x7fcba000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:01.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.100+0000 7fcbbaa00640 1 -- 172.21.15.6:0/2852154875 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbbc0d1e80 con 0x7fcba0004800 2024-03-04T09:28:01.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.100+0000 7fcbbaa00640 1 -- 172.21.15.6:0/2852154875 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcba000c0c0 con 0x7fcba00051c0 2024-03-04T09:28:01.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.100+0000 7fcbbaa00640 1 -- 172.21.15.6:0/2852154875 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcba000c2c0 con 0x7fcba0003a00 2024-03-04T09:28:01.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.100+0000 7fcbbb400640 1 --2- 172.21.15.6:0/2852154875 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcba0004800 0x7fcba0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:01.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.100+0000 7fcbc1000640 1 --2- 172.21.15.6:0/2852154875 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcba00051c0 0x7fcba000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:01.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.101+0000 7fcbbb400640 1 -- 172.21.15.6:0/2852154875 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcba0003a00 msgr2=0x7fcba0003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:01.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.101+0000 7fcbbbe00640 1 --2- 172.21.15.6:0/2852154875 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcba0003a00 0x7fcba0003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:01.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.101+0000 7fcbbb400640 1 --2- 172.21.15.6:0/2852154875 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcba0003a00 0x7fcba0003e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:01.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.101+0000 7fcbbb400640 1 -- 172.21.15.6:0/2852154875 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcba00051c0 msgr2=0x7fcba000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:01.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.101+0000 7fcbbb400640 1 --2- 172.21.15.6:0/2852154875 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcba00051c0 0x7fcba000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:01.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.101+0000 7fcbbb400640 1 -- 172.21.15.6:0/2852154875 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcba0003520 con 0x7fcba0004800 2024-03-04T09:28:01.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.101+0000 7fcbc1000640 1 --2- 172.21.15.6:0/2852154875 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcba00051c0 0x7fcba000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:28:01.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.101+0000 7fcbc1000640 1 -- 172.21.15.6:0/2852154875 reap_dead start 2024-03-04T09:28:01.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.102+0000 7fcbbb400640 1 --2- 172.21.15.6:0/2852154875 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcba0004800 0x7fcba0004c80 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fcba800c0e0 tx=0x7fcba0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=311fa79e65ed4501 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:01.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.102+0000 7fcbbaa00640 1 -- 172.21.15.6:0/2852154875 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcba8007c20 con 0x7fcba0004800 2024-03-04T09:28:01.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.102+0000 7fcbbaa00640 1 -- 172.21.15.6:0/2852154875 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcba8030070 con 0x7fcba0004800 2024-03-04T09:28:01.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.102+0000 7fcbbaa00640 1 -- 172.21.15.6:0/2852154875 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcba8005d70 con 0x7fcba0004800 2024-03-04T09:28:01.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.102+0000 7fcbc1a00640 1 -- 172.21.15.6:0/2852154875 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcba0004800 msgr2=0x7fcba0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:01.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.102+0000 7fcbc1a00640 1 --2- 172.21.15.6:0/2852154875 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcba0004800 0x7fcba0004c80 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fcba800c0e0 tx=0x7fcba0005830 comp rx=0 tx=0).stop 2024-03-04T09:28:01.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.103+0000 7fcbc1a00640 1 -- 172.21.15.6:0/2852154875 shutdown_connections 2024-03-04T09:28:01.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.103+0000 7fcbc1a00640 1 --2- 172.21.15.6:0/2852154875 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcba0004800 0x7fcba0004c80 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:01.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.103+0000 7fcbc1a00640 1 -- 172.21.15.6:0/2852154875 >> 172.21.15.6:0/2852154875 conn(0x7fcbbc08d2a0 msgr2=0x7fcbbc0d3cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:01.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.103+0000 7fcbc1a00640 1 -- 172.21.15.6:0/2852154875 shutdown_connections 2024-03-04T09:28:01.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.103+0000 7fcbc1a00640 1 -- 172.21.15.6:0/2852154875 wait complete. 2024-03-04T09:28:01.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.105+0000 7fcbc1a00640 1 Processor -- start 2024-03-04T09:28:01.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.105+0000 7fcbc1a00640 1 -- start start 2024-03-04T09:28:01.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.106+0000 7fcbc1a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcba0004800 0x7fcbbc153310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:01.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.106+0000 7fcbc1a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcbbc15f310 0x7fcbbc0b8b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:01.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.106+0000 7fcbc1a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcbbc15ff00 0x7fcbbc160380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:01.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.106+0000 7fcbc1a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbbc0d4730 con 0x7fcbbc15f310 2024-03-04T09:28:01.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.106+0000 7fcbc1a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbbc0d45b0 con 0x7fcbbc15ff00 2024-03-04T09:28:01.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.106+0000 7fcbc1a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbbc0cf870 con 0x7fcba0004800 2024-03-04T09:28:01.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.106+0000 7fcbbb400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcbbc15f310 0x7fcbbc0b8b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:01.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.106+0000 7fcbbb400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcbbc15f310 0x7fcbbc0b8b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45254/0 (socket says 172.21.15.6:45254) 2024-03-04T09:28:01.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.106+0000 7fcbbb400640 1 -- 172.21.15.6:0/4167041466 learned_addr learned my addr 172.21.15.6:0/4167041466 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:28:01.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.106+0000 7fcbbbe00640 1 --2- 172.21.15.6:0/4167041466 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcba0004800 0x7fcbbc153310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:01.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.106+0000 7fcbc1000640 1 --2- 172.21.15.6:0/4167041466 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcbbc15ff00 0x7fcbbc160380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:01.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.107+0000 7fcbbb400640 1 -- 172.21.15.6:0/4167041466 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcba0004800 msgr2=0x7fcbbc153310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:01.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.107+0000 7fcbbb400640 1 --2- 172.21.15.6:0/4167041466 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcba0004800 0x7fcbbc153310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:01.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.107+0000 7fcbbb400640 1 -- 172.21.15.6:0/4167041466 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcbbc15ff00 msgr2=0x7fcbbc160380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:01.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.107+0000 7fcbbb400640 1 --2- 172.21.15.6:0/4167041466 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcbbc15ff00 0x7fcbbc160380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:01.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.107+0000 7fcbbb400640 1 -- 172.21.15.6:0/4167041466 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbbc0b92d0 con 0x7fcbbc15f310 2024-03-04T09:28:01.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.107+0000 7fcbc1000640 1 --2- 172.21.15.6:0/4167041466 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcbbc15ff00 0x7fcbbc160380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:28:01.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.107+0000 7fcbbb400640 1 --2- 172.21.15.6:0/4167041466 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcbbc15f310 0x7fcbbc0b8b60 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fcba800c1d0 tx=0x7fcbbc0d1a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:01.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.107+0000 7fcbb3e00640 1 -- 172.21.15.6:0/4167041466 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcba8004bf0 con 0x7fcbbc15f310 2024-03-04T09:28:01.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.107+0000 7fcbb3e00640 1 -- 172.21.15.6:0/4167041466 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcba8000ed0 con 0x7fcbbc15f310 2024-03-04T09:28:01.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.107+0000 7fcbb3e00640 1 -- 172.21.15.6:0/4167041466 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcba8007070 con 0x7fcbbc15f310 2024-03-04T09:28:01.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.107+0000 7fcbc1a00640 1 -- 172.21.15.6:0/4167041466 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbbc0b95b0 con 0x7fcbbc15f310 2024-03-04T09:28:01.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.108+0000 7fcbc1a00640 1 -- 172.21.15.6:0/4167041466 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbbc0b6d20 con 0x7fcbbc15f310 2024-03-04T09:28:01.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.109+0000 7fcbc1a00640 1 -- 172.21.15.6:0/4167041466 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbbc04f330 con 0x7fcbbc15f310 2024-03-04T09:28:01.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.110+0000 7fcbb3e00640 1 -- 172.21.15.6:0/4167041466 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcba8007210 con 0x7fcbbc15f310 2024-03-04T09:28:01.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.110+0000 7fcbb3e00640 1 --2- 172.21.15.6:0/4167041466 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fcb9c06b4d0 0x7fcb9c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:01.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.111+0000 7fcbb3e00640 1 -- 172.21.15.6:0/4167041466 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcba80ab730 con 0x7fcbbc15f310 2024-03-04T09:28:01.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.111+0000 7fcbbbe00640 1 --2- 172.21.15.6:0/4167041466 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fcb9c06b4d0 0x7fcb9c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:01.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.112+0000 7fcbbbe00640 1 --2- 172.21.15.6:0/4167041466 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fcb9c06b4d0 0x7fcb9c06d990 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fcbac009a90 tx=0x7fcbac008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:01.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.115+0000 7fcbb3e00640 1 -- 172.21.15.6:0/4167041466 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcba807a620 con 0x7fcbbc15f310 2024-03-04T09:28:01.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.315+0000 7fcbc1a00640 1 -- 172.21.15.6:0/4167041466 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcbbc0c75c0 con 0x7fcbbc15f310 2024-03-04T09:28:01.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.318+0000 7fcbb3e00640 1 -- 172.21.15.6:0/4167041466 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v55) v1 ==== 74+0+16630 (secure 0 0 0) 0x7fcba8079f20 con 0x7fcbbc15f310 2024-03-04T09:28:01.315 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:28:01.316 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":55,"fsid":"edce0578-5ae4-4402-b45a-552bdef84d7c","created":"2024-03-04T09:24:54.523780+0000","modified":"2024-03-04T09:27:59.853585+0000","last_up_change":"2024-03-04T09:25:07.089049+0000","last_in_change":"2024-03-04T09:24:57.982077+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T09:25:07.935198+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-04T09:27:57.774529+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-04T09:27:58.886504+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a96574f-81f1-4bc6-8463-757520b84a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6825","nonce":1364449401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6827","nonce":1364449401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6831","nonce":1364449401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6829","nonce":1364449401}]},"public_addr":"172.21.15.6:6825/1364449401","cluster_addr":"172.21.15.6:6827/1364449401","heartbeat_back_addr":"172.21.15.6:6831/1364449401","heartbeat_front_addr":"172.21.15.6:6829/1364449401","state":["exists","up"]},{"osd":1,"uuid":"7d79b9c1-6b87-476a-bc36-7db5ee64cd90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6809","nonce":2494103078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6811","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6813","nonce":2494103078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6821","nonce":2494103078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6817","nonce":2494103078}]},"public_addr":"172.21.15.6:6809/2494103078","cluster_addr":"172.21.15.6:6813/2494103078","heartbeat_back_addr":"172.21.15.6:6821/2494103078","heartbeat_front_addr":"172.21.15.6:6817/2494103078","state":["exists","up"]},{"osd":2,"uuid":"492331b9-4ee9-40b6-bacd-85d44cd6c45a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6812","nonce":1500066681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6816","nonce":1500066681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6824","nonce":1500066681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6820","nonce":1500066681}]},"public_addr":"172.21.15.6:6812/1500066681","cluster_addr":"172.21.15.6:6816/1500066681","heartbeat_back_addr":"172.21.15.6:6824/1500066681","heartbeat_front_addr":"172.21.15.6:6820/1500066681","state":["exists","up"]},{"osd":3,"uuid":"21bb2efe-9ba7-4bdc-bc31-180cd1102e21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6801","nonce":1558806123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6803","nonce":1558806123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6807","nonce":1558806123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6805","nonce":1558806123}]},"public_addr":"172.21.15.6:6801/1558806123","cluster_addr":"172.21.15.6:6803/1558806123","heartbeat_back_addr":"172.21.15.6:6807/1558806123","heartbeat_front_addr":"172.21.15.6:6805/1558806123","state":["exists","up"]},{"osd":4,"uuid":"6e2c4b71-19d9-4737-83d9-558705d8644a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6801","nonce":1526135780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6803","nonce":1526135780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6807","nonce":1526135780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6805","nonce":1526135780}]},"public_addr":"172.21.15.146:6801/1526135780","cluster_addr":"172.21.15.146:6803/1526135780","heartbeat_back_addr":"172.21.15.146:6807/1526135780","heartbeat_front_addr":"172.21.15.146:6805/1526135780","state":["exists","up"]},{"osd":5,"uuid":"fb69c525-f1be-4b7d-ab32-6c29b0d6fddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6809","nonce":460262507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6811","nonce":460262507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6815","nonce":460262507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6813","nonce":460262507}]},"public_addr":"172.21.15.146:6809/460262507","cluster_addr":"172.21.15.146:6811/460262507","heartbeat_back_addr":"172.21.15.146:6815/460262507","heartbeat_front_addr":"172.21.15.146:6813/460262507","state":["exists","up"]},{"osd":6,"uuid":"c32e7fed-370d-4401-8ce0-6a4335b0d168","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6823","nonce":1392897493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6827","nonce":1392897493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6831","nonce":1392897493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6829","nonce":1392897493}]},"public_addr":"172.21.15.146:6823/1392897493","cluster_addr":"172.21.15.146:6827/1392897493","heartbeat_back_addr":"172.21.15.146:6831/1392897493","heartbeat_front_addr":"172.21.15.146:6829/1392897493","state":["exists","up"]},{"osd":7,"uuid":"75a79b44-fa5f-426d-a7d3-71f21f78835c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6817","nonce":3268323156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6819","nonce":3268323156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6825","nonce":3268323156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6822","nonce":3268323156}]},"public_addr":"172.21.15.146:6817/3268323156","cluster_addr":"172.21.15.146:6819/3268323156","heartbeat_back_addr":"172.21.15.146:6825/3268323156","heartbeat_front_addr":"172.21.15.146:6822/3268323156","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.310437+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.295850+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.294674+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.526855+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.300023+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.595160+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.346046+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.558741+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T09:28:01.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.320+0000 7fcbc1a00640 1 -- 172.21.15.6:0/4167041466 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fcb9c06b4d0 msgr2=0x7fcb9c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:01.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.321+0000 7fcbc1a00640 1 --2- 172.21.15.6:0/4167041466 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fcb9c06b4d0 0x7fcb9c06d990 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fcbac009a90 tx=0x7fcbac008040 comp rx=0 tx=0).stop 2024-03-04T09:28:01.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.321+0000 7fcbc1a00640 1 -- 172.21.15.6:0/4167041466 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcbbc15f310 msgr2=0x7fcbbc0b8b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:01.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.321+0000 7fcbc1a00640 1 --2- 172.21.15.6:0/4167041466 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcbbc15f310 0x7fcbbc0b8b60 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fcba800c1d0 tx=0x7fcbbc0d1a90 comp rx=0 tx=0).stop 2024-03-04T09:28:01.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.321+0000 7fcbc1a00640 1 -- 172.21.15.6:0/4167041466 shutdown_connections 2024-03-04T09:28:01.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.321+0000 7fcbc1a00640 1 --2- 172.21.15.6:0/4167041466 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcbbc15ff00 0x7fcbbc160380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:01.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.321+0000 7fcbc1a00640 1 --2- 172.21.15.6:0/4167041466 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcbbc15f310 0x7fcbbc0b8b60 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:01.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.321+0000 7fcbc1a00640 1 --2- 172.21.15.6:0/4167041466 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fcb9c06b4d0 0x7fcb9c06d990 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:01.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.321+0000 7fcbc1a00640 1 --2- 172.21.15.6:0/4167041466 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcba0004800 0x7fcbbc153310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:01.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.321+0000 7fcbc1a00640 1 -- 172.21.15.6:0/4167041466 >> 172.21.15.6:0/4167041466 conn(0x7fcbbc08d2a0 msgr2=0x7fcbbc15d340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:01.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.321+0000 7fcbc1a00640 1 -- 172.21.15.6:0/4167041466 shutdown_connections 2024-03-04T09:28:01.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.321+0000 7fcbc1a00640 1 -- 172.21.15.6:0/4167041466 wait complete. 2024-03-04T09:28:01.335 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-04T09:28:01.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.556+0000 7f2677200640 1 Processor -- start 2024-03-04T09:28:01.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.556+0000 7f2677200640 1 -- start start 2024-03-04T09:28:01.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.558+0000 7f2677200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f2678162280 0x7f2678165d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:01.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.558+0000 7f2677200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2678166cb0 con 0x7f26780b4a90 2024-03-04T09:28:01.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.558+0000 7f2677200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2678166410 con 0x7f26780b1b40 2024-03-04T09:28:01.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.558+0000 7f2677200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26781666f0 con 0x7f2678162280 2024-03-04T09:28:01.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.558+0000 7f2675e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f2678162280 0x7f2678165d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:01.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.558+0000 7f2675e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f2678162280 0x7f2678165d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45270/0 (socket says 172.21.15.6:45270) 2024-03-04T09:28:01.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.558+0000 7f2675e00640 1 -- 172.21.15.6:0/3645496014 learned_addr learned my addr 172.21.15.6:0/3645496014 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:28:01.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.559+0000 7f2675e00640 1 -- 172.21.15.6:0/3645496014 >> v1:172.21.15.146:6789/0 conn(0x7f26780b1b40 legacy=0x7f2678071560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:01.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.559+0000 7f2675e00640 1 -- 172.21.15.6:0/3645496014 >> v1:172.21.15.6:6789/0 conn(0x7f26780b4a90 legacy=0x7f26780cc430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:01.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.559+0000 7f2675e00640 1 -- 172.21.15.6:0/3645496014 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26781669d0 con 0x7f2678162280 2024-03-04T09:28:01.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.559+0000 7f2675400640 1 -- 172.21.15.6:0/3645496014 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1358916719 0 0) 0x7f2678166cb0 con 0x7f26780b4a90 2024-03-04T09:28:01.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.560+0000 7f2675e00640 1 --2- 172.21.15.6:0/3645496014 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2678162280 0x7f2678165d50 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f2668009e50 tx=0x7f266802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58b719e5d72f9015 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:01.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.560+0000 7f2675400640 1 -- 172.21.15.6:0/3645496014 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f266802ca80 con 0x7f2678162280 2024-03-04T09:28:01.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.560+0000 7f2675400640 1 -- 172.21.15.6:0/3645496014 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f266802cc20 con 0x7f2678162280 2024-03-04T09:28:01.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.560+0000 7f2675400640 1 -- 172.21.15.6:0/3645496014 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26680343f0 con 0x7f2678162280 2024-03-04T09:28:01.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.561+0000 7f2677200640 1 -- 172.21.15.6:0/3645496014 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2678162280 msgr2=0x7f2678165d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:01.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.561+0000 7f2677200640 1 --2- 172.21.15.6:0/3645496014 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2678162280 0x7f2678165d50 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f2668009e50 tx=0x7f266802bee0 comp rx=0 tx=0).stop 2024-03-04T09:28:01.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.561+0000 7f2677200640 1 -- 172.21.15.6:0/3645496014 shutdown_connections 2024-03-04T09:28:01.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.561+0000 7f2677200640 1 --2- 172.21.15.6:0/3645496014 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2678162280 0x7f2678165d50 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:01.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.561+0000 7f2677200640 1 -- 172.21.15.6:0/3645496014 >> 172.21.15.6:0/3645496014 conn(0x7f267806b3a0 msgr2=0x7f267806b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:01.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.561+0000 7f2677200640 1 -- 172.21.15.6:0/3645496014 shutdown_connections 2024-03-04T09:28:01.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.561+0000 7f2677200640 1 -- 172.21.15.6:0/3645496014 wait complete. 2024-03-04T09:28:01.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.563+0000 7f2677200640 1 Processor -- start 2024-03-04T09:28:01.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.563+0000 7f2677200640 1 -- start start 2024-03-04T09:28:01.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.564+0000 7f2677200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f26780b1b40 0x7f2678165250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:01.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.564+0000 7f2677200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f26780b4a90 0x7f26781628e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:01.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.564+0000 7f2677200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2678162280 0x7f2678162e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:01.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.564+0000 7f2677200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2678071df0 con 0x7f26780b1b40 2024-03-04T09:28:01.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.564+0000 7f2677200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2678071c70 con 0x7f26780b4a90 2024-03-04T09:28:01.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.564+0000 7f2677200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2678071f70 con 0x7f2678162280 2024-03-04T09:28:01.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.564+0000 7f2675e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f26780b1b40 0x7f2678165250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:01.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.564+0000 7f2675e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f26780b1b40 0x7f2678165250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45280/0 (socket says 172.21.15.6:45280) 2024-03-04T09:28:01.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.564+0000 7f2675e00640 1 -- 172.21.15.6:0/4096369767 learned_addr learned my addr 172.21.15.6:0/4096369767 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:28:01.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.565+0000 7f2675e00640 1 -- 172.21.15.6:0/4096369767 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2678162280 msgr2=0x7f2678162e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:01.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.565+0000 7f2675e00640 1 --2- 172.21.15.6:0/4096369767 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2678162280 0x7f2678162e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:01.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.565+0000 7f2675e00640 1 -- 172.21.15.6:0/4096369767 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f26780b4a90 msgr2=0x7f26781628e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:01.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.565+0000 7f266d400640 1 --2- 172.21.15.6:0/4096369767 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f26780b4a90 0x7f26781628e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:01.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.565+0000 7f2675e00640 1 --2- 172.21.15.6:0/4096369767 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f26780b4a90 0x7f26781628e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:01.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.565+0000 7f2675e00640 1 -- 172.21.15.6:0/4096369767 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26781585e0 con 0x7f26780b1b40 2024-03-04T09:28:01.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.565+0000 7f2675e00640 1 --2- 172.21.15.6:0/4096369767 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f26780b1b40 0x7f2678165250 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f2668009e50 tx=0x7f2668004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:01.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.565+0000 7f266ea00640 1 -- 172.21.15.6:0/4096369767 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2668043020 con 0x7f26780b1b40 2024-03-04T09:28:01.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.565+0000 7f266ea00640 1 -- 172.21.15.6:0/4096369767 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f266803e450 con 0x7f26780b1b40 2024-03-04T09:28:01.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.565+0000 7f266ea00640 1 -- 172.21.15.6:0/4096369767 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f266803e840 con 0x7f26780b1b40 2024-03-04T09:28:01.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.565+0000 7f2677200640 1 -- 172.21.15.6:0/4096369767 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26781588c0 con 0x7f26780b1b40 2024-03-04T09:28:01.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.565+0000 7f2677200640 1 -- 172.21.15.6:0/4096369767 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2678163360 con 0x7f26780b1b40 2024-03-04T09:28:01.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.566+0000 7f2677200640 1 -- 172.21.15.6:0/4096369767 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f267804f330 con 0x7f26780b1b40 2024-03-04T09:28:01.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.567+0000 7f266ea00640 1 -- 172.21.15.6:0/4096369767 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2668046020 con 0x7f26780b1b40 2024-03-04T09:28:01.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.568+0000 7f266ea00640 1 --2- 172.21.15.6:0/4096369767 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f265406b4d0 0x7f265406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:01.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.568+0000 7f266ea00640 1 -- 172.21.15.6:0/4096369767 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f26680acb90 con 0x7f26780b1b40 2024-03-04T09:28:01.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.569+0000 7f266d400640 1 --2- 172.21.15.6:0/4096369767 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f265406b4d0 0x7f265406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:01.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.570+0000 7f266d400640 1 --2- 172.21.15.6:0/4096369767 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f265406b4d0 0x7f265406d990 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f2678155ba0 tx=0x7f2660009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:01.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.573+0000 7f266ea00640 1 -- 172.21.15.6:0/4096369767 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f266804d030 con 0x7f26780b1b40 2024-03-04T09:28:01.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:01.813+0000 7f2677200640 1 -- 172.21.15.6:0/4096369767 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f267806e490 con 0x7f26780b1b40 2024-03-04T09:28:02.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:02.434+0000 7f266ea00640 1 -- 172.21.15.6:0/4096369767 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v33) v1 ==== 118+0+0 (secure 0 0 0) 0x7f266807ba00 con 0x7f26780b1b40 2024-03-04T09:28:02.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:02.437+0000 7f2677200640 1 -- 172.21.15.6:0/4096369767 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f265406b4d0 msgr2=0x7f265406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:02.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:02.437+0000 7f2677200640 1 --2- 172.21.15.6:0/4096369767 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f265406b4d0 0x7f265406d990 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f2678155ba0 tx=0x7f2660009040 comp rx=0 tx=0).stop 2024-03-04T09:28:02.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:02.437+0000 7f2677200640 1 -- 172.21.15.6:0/4096369767 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f26780b1b40 msgr2=0x7f2678165250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:02.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:02.437+0000 7f2677200640 1 --2- 172.21.15.6:0/4096369767 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f26780b1b40 0x7f2678165250 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f2668009e50 tx=0x7f2668004a40 comp rx=0 tx=0).stop 2024-03-04T09:28:02.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:02.438+0000 7f2677200640 1 -- 172.21.15.6:0/4096369767 shutdown_connections 2024-03-04T09:28:02.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:02.438+0000 7f2677200640 1 --2- 172.21.15.6:0/4096369767 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f265406b4d0 0x7f265406d990 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:02.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:02.438+0000 7f2677200640 1 --2- 172.21.15.6:0/4096369767 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2678162280 0x7f2678162e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:02.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:02.438+0000 7f2677200640 1 --2- 172.21.15.6:0/4096369767 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f26780b4a90 0x7f26781628e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:02.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:02.438+0000 7f2677200640 1 --2- 172.21.15.6:0/4096369767 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f26780b1b40 0x7f2678165250 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:02.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:02.438+0000 7f2677200640 1 -- 172.21.15.6:0/4096369767 >> 172.21.15.6:0/4096369767 conn(0x7f267806b3a0 msgr2=0x7f267815ca40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:02.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:02.438+0000 7f2677200640 1 -- 172.21.15.6:0/4096369767 shutdown_connections 2024-03-04T09:28:02.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:02.438+0000 7f2677200640 1 -- 172.21.15.6:0/4096369767 wait complete. 2024-03-04T09:28:02.455 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-04T09:28:02.455 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-04T09:28:02.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:02.681+0000 7f6041e00640 1 Processor -- start 2024-03-04T09:28:02.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:02.682+0000 7f6041e00640 1 -- start start 2024-03-04T09:28:02.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:02.683+0000 7f6041e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f603c0b2250 0x7f603c166ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:02.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:02.683+0000 7f6041e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f603c0b3fe0 con 0x7f603c145430 2024-03-04T09:28:02.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:02.683+0000 7f6041e00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f603c15fe40 con 0x7f603c15f810 2024-03-04T09:28:02.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:02.683+0000 7f6041e00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f603c160120 con 0x7f603c0b2250 2024-03-04T09:28:02.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:02.684+0000 7f6040a00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f603c145430 0x7f603c15cc90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:39804/0 (socket says 172.21.15.6:39804) 2024-03-04T09:28:02.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:02.684+0000 7f6040a00640 1 -- 172.21.15.6:0/532727920 learned_addr learned my addr 172.21.15.6:0/532727920 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:28:02.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:02.684+0000 7f6041400640 1 --2- 172.21.15.6:0/532727920 >> v2:172.21.15.146:3300/0 conn(0x7f603c0b2250 0x7f603c166ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:02.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:02.684+0000 7f603b400640 1 -- 172.21.15.6:0/532727920 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3400406661 0 0) 0x7f603c0b3fe0 con 0x7f603c145430 2024-03-04T09:28:02.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:02.684+0000 7f603b400640 1 -- 172.21.15.6:0/532727920 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6024003520 con 0x7f603c145430 2024-03-04T09:28:02.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:02.685+0000 7f603b400640 1 -- 172.21.15.6:0/532727920 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 998218506 0 0) 0x7f6024003520 con 0x7f603c145430 2024-03-04T09:28:02.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:02.685+0000 7f603b400640 1 -- 172.21.15.6:0/532727920 >> v2:172.21.15.146:3300/0 conn(0x7f603c0b2250 msgr2=0x7f603c166ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:02.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:02.685+0000 7f603b400640 1 --2- 172.21.15.6:0/532727920 >> v2:172.21.15.146:3300/0 conn(0x7f603c0b2250 0x7f603c166ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:02.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:02.685+0000 7f603b400640 1 -- 172.21.15.6:0/532727920 >> v1:172.21.15.146:6789/0 conn(0x7f603c15f810 legacy=0x7f603c0b1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:02.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:02.685+0000 7f603b400640 1 -- 172.21.15.6:0/532727920 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f603c160400 con 0x7f603c145430 2024-03-04T09:28:02.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:02.685+0000 7f603b400640 1 -- 172.21.15.6:0/532727920 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7f6030001670 con 0x7f603c145430 2024-03-04T09:28:02.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:02.685+0000 7f603b400640 1 -- 172.21.15.6:0/532727920 >> v1:172.21.15.6:6789/0 conn(0x7f603c145430 legacy=0x7f603c15cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:02.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:02.685+0000 7f603b400640 1 --2- 172.21.15.6:0/532727920 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f60240039b0 0x7f6024003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:02.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:02.685+0000 7f603b400640 1 --2- 172.21.15.6:0/532727920 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6024004800 0x7f6024004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:02.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:02.685+0000 7f603b400640 1 --2- 172.21.15.6:0/532727920 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f60240051c0 0x7f602400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:02.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:02.685+0000 7f603b400640 1 -- 172.21.15.6:0/532727920 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f603c160120 con 0x7f60240051c0 2024-03-04T09:28:02.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:02.685+0000 7f603b400640 1 -- 172.21.15.6:0/532727920 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f602400c060 con 0x7f60240039b0 2024-03-04T09:28:02.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:02.685+0000 7f603b400640 1 -- 172.21.15.6:0/532727920 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f602400c260 con 0x7f6024004800 2024-03-04T09:28:02.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:02.685+0000 7f6041400640 1 --2- 172.21.15.6:0/532727920 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f60240051c0 0x7f602400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:02.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:02.685+0000 7f603be00640 1 --2- 172.21.15.6:0/532727920 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6024004800 0x7f6024004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:02.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:02.685+0000 7f6041400640 1 -- 172.21.15.6:0/532727920 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6024004800 msgr2=0x7f6024004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:02.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:02.686+0000 7f6040a00640 1 --2- 172.21.15.6:0/532727920 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f60240039b0 0x7f6024003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:02.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:02.686+0000 7f6041400640 1 --2- 172.21.15.6:0/532727920 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6024004800 0x7f6024004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:02.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:02.686+0000 7f6041400640 1 -- 172.21.15.6:0/532727920 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f60240039b0 msgr2=0x7f6024003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:02.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:02.686+0000 7f6041400640 1 --2- 172.21.15.6:0/532727920 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f60240039b0 0x7f6024003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:02.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:02.686+0000 7f6041400640 1 -- 172.21.15.6:0/532727920 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6024003520 con 0x7f60240051c0 2024-03-04T09:28:02.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:02.686+0000 7f603be00640 1 --2- 172.21.15.6:0/532727920 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6024004800 0x7f6024004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:28:02.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:02.686+0000 7f6040a00640 1 --2- 172.21.15.6:0/532727920 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f60240039b0 0x7f6024003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:28:02.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:02.687+0000 7f6041400640 1 -- 172.21.15.6:0/532727920 reap_dead start 2024-03-04T09:28:02.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:02.687+0000 7f6041400640 1 --2- 172.21.15.6:0/532727920 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f60240051c0 0x7f602400b9c0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f6034009f90 tx=0x7f6024005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=863cdb4bf1d19d35 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:02.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:02.687+0000 7f603b400640 1 -- 172.21.15.6:0/532727920 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6034009c10 con 0x7f60240051c0 2024-03-04T09:28:02.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:02.687+0000 7f603b400640 1 -- 172.21.15.6:0/532727920 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f603402cc20 con 0x7f60240051c0 2024-03-04T09:28:02.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:02.687+0000 7f603b400640 1 -- 172.21.15.6:0/532727920 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6034003990 con 0x7f60240051c0 2024-03-04T09:28:02.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:02.687+0000 7f6041e00640 1 -- 172.21.15.6:0/532727920 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f60240051c0 msgr2=0x7f602400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:02.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:02.687+0000 7f6041e00640 1 --2- 172.21.15.6:0/532727920 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f60240051c0 0x7f602400b9c0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f6034009f90 tx=0x7f6024005830 comp rx=0 tx=0).stop 2024-03-04T09:28:02.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:02.688+0000 7f6041e00640 1 -- 172.21.15.6:0/532727920 shutdown_connections 2024-03-04T09:28:02.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:02.688+0000 7f6041e00640 1 --2- 172.21.15.6:0/532727920 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f60240051c0 0x7f602400b9c0 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:02.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:02.688+0000 7f6041e00640 1 -- 172.21.15.6:0/532727920 >> 172.21.15.6:0/532727920 conn(0x7f603c06b3a0 msgr2=0x7f603c06b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:02.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:02.688+0000 7f6041e00640 1 -- 172.21.15.6:0/532727920 shutdown_connections 2024-03-04T09:28:02.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:02.688+0000 7f6041e00640 1 -- 172.21.15.6:0/532727920 wait complete. 2024-03-04T09:28:02.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:02.689+0000 7f6041e00640 1 Processor -- start 2024-03-04T09:28:02.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:02.690+0000 7f6041e00640 1 -- start start 2024-03-04T09:28:02.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:02.690+0000 7f6041e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f60240051c0 0x7f603c1f0d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:02.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:02.690+0000 7f6041e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f603c0a9e80 0x7f603c0aa300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:02.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:02.690+0000 7f6041e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f603c0ae820 0x7f603c0aeca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:02.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:02.691+0000 7f6041e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f603c145990 con 0x7f60240051c0 2024-03-04T09:28:02.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:02.691+0000 7f6041e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f603c145810 con 0x7f603c0a9e80 2024-03-04T09:28:02.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:02.691+0000 7f6041e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f603c0cde00 con 0x7f603c0ae820 2024-03-04T09:28:02.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:02.691+0000 7f6040a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f60240051c0 0x7f603c1f0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:02.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:02.691+0000 7f6040a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f60240051c0 0x7f603c1f0d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45286/0 (socket says 172.21.15.6:45286) 2024-03-04T09:28:02.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:02.691+0000 7f6040a00640 1 -- 172.21.15.6:0/2266487269 learned_addr learned my addr 172.21.15.6:0/2266487269 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:28:02.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:02.691+0000 7f6041400640 1 --2- 172.21.15.6:0/2266487269 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f603c0ae820 0x7f603c0aeca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:02.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:02.691+0000 7f603be00640 1 --2- 172.21.15.6:0/2266487269 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f603c0a9e80 0x7f603c0aa300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:02.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:02.691+0000 7f6040a00640 1 -- 172.21.15.6:0/2266487269 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f603c0ae820 msgr2=0x7f603c0aeca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:02.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:02.691+0000 7f6040a00640 1 --2- 172.21.15.6:0/2266487269 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f603c0ae820 0x7f603c0aeca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:02.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:02.691+0000 7f6040a00640 1 -- 172.21.15.6:0/2266487269 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f603c0a9e80 msgr2=0x7f603c0aa300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:02.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:02.691+0000 7f6040a00640 1 --2- 172.21.15.6:0/2266487269 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f603c0a9e80 0x7f603c0aa300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:02.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:02.691+0000 7f6040a00640 1 -- 172.21.15.6:0/2266487269 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f603c0aa9f0 con 0x7f60240051c0 2024-03-04T09:28:02.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:02.691+0000 7f6041400640 1 --2- 172.21.15.6:0/2266487269 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f603c0ae820 0x7f603c0aeca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:28:02.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:02.692+0000 7f6040a00640 1 --2- 172.21.15.6:0/2266487269 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f60240051c0 0x7f603c1f0d00 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f6030000ff0 tx=0x7f603000dc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:02.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:02.692+0000 7f6038c00640 1 -- 172.21.15.6:0/2266487269 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6030019070 con 0x7f60240051c0 2024-03-04T09:28:02.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:02.692+0000 7f6038c00640 1 -- 172.21.15.6:0/2266487269 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60300074e0 con 0x7f60240051c0 2024-03-04T09:28:02.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:02.692+0000 7f6038c00640 1 -- 172.21.15.6:0/2266487269 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f603000de60 con 0x7f60240051c0 2024-03-04T09:28:02.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:02.692+0000 7f6041e00640 1 -- 172.21.15.6:0/2266487269 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f603c0aacd0 con 0x7f60240051c0 2024-03-04T09:28:02.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:02.692+0000 7f6041e00640 1 -- 172.21.15.6:0/2266487269 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f603c1539e0 con 0x7f60240051c0 2024-03-04T09:28:02.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:02.693+0000 7f6041e00640 1 -- 172.21.15.6:0/2266487269 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f603c04f330 con 0x7f60240051c0 2024-03-04T09:28:02.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:02.694+0000 7f6038c00640 1 -- 172.21.15.6:0/2266487269 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f603001c070 con 0x7f60240051c0 2024-03-04T09:28:02.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:02.695+0000 7f6038c00640 1 --2- 172.21.15.6:0/2266487269 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f601c06b4d0 0x7f601c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:02.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:02.695+0000 7f6038c00640 1 -- 172.21.15.6:0/2266487269 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6030089c90 con 0x7f60240051c0 2024-03-04T09:28:02.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:02.696+0000 7f603be00640 1 --2- 172.21.15.6:0/2266487269 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f601c06b4d0 0x7f601c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:02.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:02.697+0000 7f603be00640 1 --2- 172.21.15.6:0/2266487269 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f601c06b4d0 0x7f601c06d990 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f6028000f60 tx=0x7f6028009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:02.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:02.699+0000 7f6038c00640 1 -- 172.21.15.6:0/2266487269 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6030058a70 con 0x7f60240051c0 2024-03-04T09:28:02.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:02.943+0000 7f6041e00640 1 -- 172.21.15.6:0/2266487269 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f603c06e240 con 0x7f60240051c0 2024-03-04T09:28:03.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:03.437+0000 7f6038c00640 1 -- 172.21.15.6:0/2266487269 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v34) v1 ==== 122+0+0 (secure 0 0 0) 0x7f6030058370 con 0x7f60240051c0 2024-03-04T09:28:03.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:03.440+0000 7f6041e00640 1 -- 172.21.15.6:0/2266487269 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f601c06b4d0 msgr2=0x7f601c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:03.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:03.440+0000 7f6041e00640 1 --2- 172.21.15.6:0/2266487269 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f601c06b4d0 0x7f601c06d990 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f6028000f60 tx=0x7f6028009040 comp rx=0 tx=0).stop 2024-03-04T09:28:03.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:03.440+0000 7f6041e00640 1 -- 172.21.15.6:0/2266487269 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f60240051c0 msgr2=0x7f603c1f0d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:03.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:03.440+0000 7f6041e00640 1 --2- 172.21.15.6:0/2266487269 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f60240051c0 0x7f603c1f0d00 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f6030000ff0 tx=0x7f603000dc70 comp rx=0 tx=0).stop 2024-03-04T09:28:03.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:03.440+0000 7f6041e00640 1 -- 172.21.15.6:0/2266487269 shutdown_connections 2024-03-04T09:28:03.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:03.440+0000 7f6041e00640 1 --2- 172.21.15.6:0/2266487269 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f601c06b4d0 0x7f601c06d990 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:03.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:03.440+0000 7f6041e00640 1 --2- 172.21.15.6:0/2266487269 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f603c0ae820 0x7f603c0aeca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:03.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:03.440+0000 7f6041e00640 1 --2- 172.21.15.6:0/2266487269 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f603c0a9e80 0x7f603c0aa300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:03.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:03.440+0000 7f6041e00640 1 --2- 172.21.15.6:0/2266487269 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f60240051c0 0x7f603c1f0d00 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:03.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:03.440+0000 7f6041e00640 1 -- 172.21.15.6:0/2266487269 >> 172.21.15.6:0/2266487269 conn(0x7f603c06b3a0 msgr2=0x7f603c0ce130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:03.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:03.441+0000 7f6041e00640 1 -- 172.21.15.6:0/2266487269 shutdown_connections 2024-03-04T09:28:03.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:03.441+0000 7f6041e00640 1 -- 172.21.15.6:0/2266487269 wait complete. 2024-03-04T09:28:03.458 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-04T09:28:03.459 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:28:03.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:03.675+0000 7f6f95800640 1 Processor -- start 2024-03-04T09:28:03.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:03.675+0000 7f6f95800640 1 -- start start 2024-03-04T09:28:03.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:03.676+0000 7f6f95800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f6f901552b0 0x7f6f9015b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:03.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:03.676+0000 7f6f95800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f6f9015e810 0x7f6f900cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:03.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:03.677+0000 7f6f95800640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f9015ed10 con 0x7f6f900cce90 2024-03-04T09:28:03.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:03.677+0000 7f6f95800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f900cd300 con 0x7f6f901552b0 2024-03-04T09:28:03.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:03.677+0000 7f6f95800640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f9015eff0 con 0x7f6f9015e810 2024-03-04T09:28:03.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:03.677+0000 7f6f8fe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f6f901552b0 0x7f6f9015b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:03.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:03.677+0000 7f6f8fe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f6f901552b0 0x7f6f9015b1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45296/0 (socket says 172.21.15.6:45296) 2024-03-04T09:28:03.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:03.677+0000 7f6f8fe00640 1 -- 172.21.15.6:0/712591537 learned_addr learned my addr 172.21.15.6:0/712591537 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:28:03.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:03.677+0000 7f6f8f400640 1 --2- 172.21.15.6:0/712591537 >> v2:172.21.15.146:3300/0 conn(0x7f6f9015e810 0x7f6f900cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:03.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:03.677+0000 7f6f8fe00640 1 -- 172.21.15.6:0/712591537 >> v2:172.21.15.146:3300/0 conn(0x7f6f9015e810 msgr2=0x7f6f900cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:03.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:03.677+0000 7f6f8fe00640 1 --2- 172.21.15.6:0/712591537 >> v2:172.21.15.146:3300/0 conn(0x7f6f9015e810 0x7f6f900cc950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:03.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:03.678+0000 7f6f8fe00640 1 -- 172.21.15.6:0/712591537 >> v1:172.21.15.146:6789/0 conn(0x7f6f900cce90 legacy=0x7f6f9006c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:03.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:03.678+0000 7f6f8fe00640 1 -- 172.21.15.6:0/712591537 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f9015f2d0 con 0x7f6f901552b0 2024-03-04T09:28:03.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:03.679+0000 7f6f8fe00640 1 --2- 172.21.15.6:0/712591537 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6f901552b0 0x7f6f9015b1d0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f6f900cf1d0 tx=0x7f6f8002da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad261fa88f3b1f01 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:03.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:03.679+0000 7f6f8ea00640 1 -- 172.21.15.6:0/712591537 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f8002f850 con 0x7f6f901552b0 2024-03-04T09:28:03.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:03.679+0000 7f6f8ea00640 1 -- 172.21.15.6:0/712591537 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f8002f9f0 con 0x7f6f901552b0 2024-03-04T09:28:03.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:03.679+0000 7f6f8ea00640 1 -- 172.21.15.6:0/712591537 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f8002fdc0 con 0x7f6f901552b0 2024-03-04T09:28:03.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:03.680+0000 7f6f95800640 1 -- 172.21.15.6:0/712591537 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6f901552b0 msgr2=0x7f6f9015b1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:03.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:03.680+0000 7f6f95800640 1 --2- 172.21.15.6:0/712591537 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6f901552b0 0x7f6f9015b1d0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f6f900cf1d0 tx=0x7f6f8002da90 comp rx=0 tx=0).stop 2024-03-04T09:28:03.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:03.680+0000 7f6f95800640 1 -- 172.21.15.6:0/712591537 shutdown_connections 2024-03-04T09:28:03.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:03.680+0000 7f6f95800640 1 --2- 172.21.15.6:0/712591537 >> v2:172.21.15.146:3300/0 conn(0x7f6f9015e810 0x7f6f900cc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:03.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:03.680+0000 7f6f95800640 1 --2- 172.21.15.6:0/712591537 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6f901552b0 0x7f6f9015b1d0 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:03.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:03.680+0000 7f6f95800640 1 -- 172.21.15.6:0/712591537 >> 172.21.15.6:0/712591537 conn(0x7f6f9006b3d0 msgr2=0x7f6f9006b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:03.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:03.680+0000 7f6f95800640 1 -- 172.21.15.6:0/712591537 shutdown_connections 2024-03-04T09:28:03.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:03.680+0000 7f6f95800640 1 -- 172.21.15.6:0/712591537 wait complete. 2024-03-04T09:28:03.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:03.682+0000 7f6f95800640 1 Processor -- start 2024-03-04T09:28:03.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:03.682+0000 7f6f95800640 1 -- start start 2024-03-04T09:28:03.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:03.683+0000 7f6f95800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6f900cce90 0x7f6f901f3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:03.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:03.683+0000 7f6f95800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6f901552b0 0x7f6f9014c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:03.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:03.683+0000 7f6f95800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6f9015e810 0x7f6f90159c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:03.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:03.683+0000 7f6f95800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f900b3760 con 0x7f6f900cce90 2024-03-04T09:28:03.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:03.683+0000 7f6f95800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f90155690 con 0x7f6f901552b0 2024-03-04T09:28:03.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:03.683+0000 7f6f95800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f9006dd30 con 0x7f6f9015e810 2024-03-04T09:28:03.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:03.683+0000 7f6f8fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6f900cce90 0x7f6f901f3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:03.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:03.683+0000 7f6f8fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6f900cce90 0x7f6f901f3ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45298/0 (socket says 172.21.15.6:45298) 2024-03-04T09:28:03.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:03.683+0000 7f6f8fe00640 1 -- 172.21.15.6:0/1666181962 learned_addr learned my addr 172.21.15.6:0/1666181962 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:28:03.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:03.683+0000 7f6f8f400640 1 --2- 172.21.15.6:0/1666181962 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6f901552b0 0x7f6f9014c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:03.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:03.683+0000 7f6f8fe00640 1 -- 172.21.15.6:0/1666181962 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6f9015e810 msgr2=0x7f6f90159c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:03.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:03.683+0000 7f6f8fe00640 1 --2- 172.21.15.6:0/1666181962 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6f9015e810 0x7f6f90159c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:03.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:03.683+0000 7f6f8fe00640 1 -- 172.21.15.6:0/1666181962 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6f901552b0 msgr2=0x7f6f9014c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:03.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:03.683+0000 7f6f8fe00640 1 --2- 172.21.15.6:0/1666181962 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6f901552b0 0x7f6f9014c500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:03.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:03.683+0000 7f6f8fe00640 1 -- 172.21.15.6:0/1666181962 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f9014cee0 con 0x7f6f900cce90 2024-03-04T09:28:03.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:03.684+0000 7f6f8fe00640 1 --2- 172.21.15.6:0/1666181962 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6f900cce90 0x7f6f901f3ec0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f6f80009960 tx=0x7f6f800046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:03.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:03.684+0000 7f6f87e00640 1 -- 172.21.15.6:0/1666181962 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f80047070 con 0x7f6f900cce90 2024-03-04T09:28:03.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:03.684+0000 7f6f87e00640 1 -- 172.21.15.6:0/1666181962 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f80039750 con 0x7f6f900cce90 2024-03-04T09:28:03.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:03.684+0000 7f6f95800640 1 -- 172.21.15.6:0/1666181962 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f9014d1c0 con 0x7f6f900cce90 2024-03-04T09:28:03.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:03.684+0000 7f6f87e00640 1 -- 172.21.15.6:0/1666181962 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f80039b40 con 0x7f6f900cce90 2024-03-04T09:28:03.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:03.684+0000 7f6f95800640 1 -- 172.21.15.6:0/1666181962 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f9014d780 con 0x7f6f900cce90 2024-03-04T09:28:03.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:03.685+0000 7f6f95800640 1 -- 172.21.15.6:0/1666181962 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f9004f330 con 0x7f6f900cce90 2024-03-04T09:28:03.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:03.686+0000 7f6f87e00640 1 -- 172.21.15.6:0/1666181962 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6f80039ce0 con 0x7f6f900cce90 2024-03-04T09:28:03.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:03.687+0000 7f6f87e00640 1 --2- 172.21.15.6:0/1666181962 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f6f6c06b4d0 0x7f6f6c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:03.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:03.687+0000 7f6f87e00640 1 -- 172.21.15.6:0/1666181962 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6f800ab800 con 0x7f6f900cce90 2024-03-04T09:28:03.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:03.688+0000 7f6f8f400640 1 --2- 172.21.15.6:0/1666181962 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f6f6c06b4d0 0x7f6f6c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:03.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:03.689+0000 7f6f8f400640 1 --2- 172.21.15.6:0/1666181962 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f6f6c06b4d0 0x7f6f6c06d990 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f6f901f7960 tx=0x7f6f78008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:03.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:03.692+0000 7f6f87e00640 1 -- 172.21.15.6:0/1666181962 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6f8007a5e0 con 0x7f6f900cce90 2024-03-04T09:28:03.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:03.932+0000 7f6f95800640 1 -- 172.21.15.6:0/1666181962 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6f900cf4a0 con 0x7f6f900cce90 2024-03-04T09:28:03.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:03.933+0000 7f6f87e00640 1 -- 172.21.15.6:0/1666181962 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+4971 (secure 0 0 0) 0x7f6f80079ee0 con 0x7f6f900cce90 2024-03-04T09:28:03.930 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:28:03.930 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":34,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5030,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/1411942502","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1411942502},{"type":"v1","addr":"172.21.15.6:6834","nonce":1411942502}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28},{"gid":5039,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/1628166413","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":1628166413},{"type":"v1","addr":"172.21.15.146:6837","nonce":1628166413}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":5323,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/131491768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":131491768},{"type":"v1","addr":"172.21.15.6:6835","nonce":131491768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:27:59.854470+0000","modified":"2024-03-04T09:28:02.945053+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5326},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5326":{"gid":5326,"name":"b","rank":0,"incarnation":30,"state":"up:active","state_seq":3,"addr":"172.21.15.146:6836/527855083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":527855083},{"type":"v1","addr":"172.21.15.146:6836","nonce":527855083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5326,"qdb_cluster":[5326]},"id":4}]} 2024-03-04T09:28:03.931 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 34 2024-03-04T09:28:03.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:03.936+0000 7f6f95800640 1 -- 172.21.15.6:0/1666181962 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f6f6c06b4d0 msgr2=0x7f6f6c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:03.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:03.936+0000 7f6f95800640 1 --2- 172.21.15.6:0/1666181962 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f6f6c06b4d0 0x7f6f6c06d990 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f6f901f7960 tx=0x7f6f78008040 comp rx=0 tx=0).stop 2024-03-04T09:28:03.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:03.936+0000 7f6f95800640 1 -- 172.21.15.6:0/1666181962 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6f900cce90 msgr2=0x7f6f901f3ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:03.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:03.936+0000 7f6f95800640 1 --2- 172.21.15.6:0/1666181962 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6f900cce90 0x7f6f901f3ec0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f6f80009960 tx=0x7f6f800046c0 comp rx=0 tx=0).stop 2024-03-04T09:28:03.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:03.936+0000 7f6f95800640 1 -- 172.21.15.6:0/1666181962 shutdown_connections 2024-03-04T09:28:03.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:03.936+0000 7f6f95800640 1 --2- 172.21.15.6:0/1666181962 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f6f6c06b4d0 0x7f6f6c06d990 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:03.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:03.936+0000 7f6f95800640 1 --2- 172.21.15.6:0/1666181962 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6f9015e810 0x7f6f90159c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:03.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:03.936+0000 7f6f95800640 1 --2- 172.21.15.6:0/1666181962 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6f901552b0 0x7f6f9014c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:03.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:03.936+0000 7f6f95800640 1 --2- 172.21.15.6:0/1666181962 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6f900cce90 0x7f6f901f3ec0 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:03.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:03.936+0000 7f6f95800640 1 -- 172.21.15.6:0/1666181962 >> 172.21.15.6:0/1666181962 conn(0x7f6f9006b3d0 msgr2=0x7f6f90152280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:03.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:03.937+0000 7f6f95800640 1 -- 172.21.15.6:0/1666181962 shutdown_connections 2024-03-04T09:28:03.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:03.937+0000 7f6f95800640 1 -- 172.21.15.6:0/1666181962 wait complete. 2024-03-04T09:28:03.954 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T09:28:04.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.169+0000 7fbac6e00640 1 Processor -- start 2024-03-04T09:28:04.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.170+0000 7fbac6e00640 1 -- start start 2024-03-04T09:28:04.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.171+0000 7fbac6e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fbac015dd00 0x7fbac0162520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:04.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.171+0000 7fbac6e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fbac006bfc0 0x7fbac006c450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:04.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.171+0000 7fbac6e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbac0161a80 con 0x7fbac015e0e0 2024-03-04T09:28:04.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.171+0000 7fbac6e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbac015efa0 con 0x7fbac015dd00 2024-03-04T09:28:04.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.172+0000 7fbac6e00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbac015f280 con 0x7fbac006bfc0 2024-03-04T09:28:04.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.172+0000 7fbabfe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fbac015dd00 0x7fbac0162520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:04.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.172+0000 7fbabfe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fbac015dd00 0x7fbac0162520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45310/0 (socket says 172.21.15.6:45310) 2024-03-04T09:28:04.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.172+0000 7fbabfe00640 1 -- 172.21.15.6:0/4143276468 learned_addr learned my addr 172.21.15.6:0/4143276468 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:28:04.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.172+0000 7fbabf400640 1 --2- 172.21.15.6:0/4143276468 >> v2:172.21.15.146:3300/0 conn(0x7fbac006bfc0 0x7fbac006c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:04.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.172+0000 7fbabfe00640 1 -- 172.21.15.6:0/4143276468 >> v2:172.21.15.146:3300/0 conn(0x7fbac006bfc0 msgr2=0x7fbac006c450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:04.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.172+0000 7fbabfe00640 1 --2- 172.21.15.6:0/4143276468 >> v2:172.21.15.146:3300/0 conn(0x7fbac006bfc0 0x7fbac006c450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:04.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.172+0000 7fbabfe00640 1 -- 172.21.15.6:0/4143276468 >> v1:172.21.15.6:6789/0 conn(0x7fbac015e0e0 legacy=0x7fbac015e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:04.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.172+0000 7fbabfe00640 1 -- 172.21.15.6:0/4143276468 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbac0161d20 con 0x7fbac015dd00 2024-03-04T09:28:04.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.172+0000 7fbabea00640 1 -- 172.21.15.6:0/4143276468 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2213422495 0 0) 0x7fbac0161a80 con 0x7fbac015e0e0 2024-03-04T09:28:04.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.174+0000 7fbabfe00640 1 --2- 172.21.15.6:0/4143276468 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbac015dd00 0x7fbac0162520 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fbac00721f0 tx=0x7fbab002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80c9c424f9b3afb2 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:04.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.174+0000 7fbabea00640 1 -- 172.21.15.6:0/4143276468 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbab002ca80 con 0x7fbac015dd00 2024-03-04T09:28:04.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.174+0000 7fbabea00640 1 -- 172.21.15.6:0/4143276468 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbab002cc20 con 0x7fbac015dd00 2024-03-04T09:28:04.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.174+0000 7fbabea00640 1 -- 172.21.15.6:0/4143276468 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbab00343f0 con 0x7fbac015dd00 2024-03-04T09:28:04.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.174+0000 7fbac6e00640 1 -- 172.21.15.6:0/4143276468 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbac015dd00 msgr2=0x7fbac0162520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:04.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.174+0000 7fbac6e00640 1 --2- 172.21.15.6:0/4143276468 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbac015dd00 0x7fbac0162520 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fbac00721f0 tx=0x7fbab002bee0 comp rx=0 tx=0).stop 2024-03-04T09:28:04.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.175+0000 7fbac6e00640 1 -- 172.21.15.6:0/4143276468 shutdown_connections 2024-03-04T09:28:04.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.175+0000 7fbac6e00640 1 --2- 172.21.15.6:0/4143276468 >> v2:172.21.15.146:3300/0 conn(0x7fbac006bfc0 0x7fbac006c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:04.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.175+0000 7fbac6e00640 1 --2- 172.21.15.6:0/4143276468 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbac015dd00 0x7fbac0162520 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:04.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.175+0000 7fbac6e00640 1 -- 172.21.15.6:0/4143276468 >> 172.21.15.6:0/4143276468 conn(0x7fbac00714d0 msgr2=0x7fbac00718b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:04.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.175+0000 7fbac6e00640 1 -- 172.21.15.6:0/4143276468 shutdown_connections 2024-03-04T09:28:04.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.175+0000 7fbac6e00640 1 -- 172.21.15.6:0/4143276468 wait complete. 2024-03-04T09:28:04.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.176+0000 7fbac6e00640 1 Processor -- start 2024-03-04T09:28:04.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.176+0000 7fbac6e00640 1 -- start start 2024-03-04T09:28:04.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.177+0000 7fbac6e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbac006bfc0 0x7fbac00c6fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:04.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.177+0000 7fbac6e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbac015dd00 0x7fbac0158150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:04.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.177+0000 7fbac6e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbac015e0e0 0x7fbac0148ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:04.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.177+0000 7fbac6e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbac015ec10 con 0x7fbac015dd00 2024-03-04T09:28:04.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.177+0000 7fbac6e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbac006f280 con 0x7fbac006bfc0 2024-03-04T09:28:04.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.177+0000 7fbac6e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbac006d250 con 0x7fbac015e0e0 2024-03-04T09:28:04.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.178+0000 7fbabf400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbac015dd00 0x7fbac0158150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:04.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.178+0000 7fbabf400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbac015dd00 0x7fbac0158150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45312/0 (socket says 172.21.15.6:45312) 2024-03-04T09:28:04.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.178+0000 7fbabf400640 1 -- 172.21.15.6:0/455608950 learned_addr learned my addr 172.21.15.6:0/455608950 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:28:04.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.178+0000 7fbabfe00640 1 --2- 172.21.15.6:0/455608950 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbac006bfc0 0x7fbac00c6fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:04.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.178+0000 7fbac4e00640 1 --2- 172.21.15.6:0/455608950 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbac015e0e0 0x7fbac0148ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:04.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.178+0000 7fbabf400640 1 -- 172.21.15.6:0/455608950 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbac015e0e0 msgr2=0x7fbac0148ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:04.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.178+0000 7fbabf400640 1 --2- 172.21.15.6:0/455608950 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbac015e0e0 0x7fbac0148ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:04.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.178+0000 7fbabf400640 1 -- 172.21.15.6:0/455608950 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbac006bfc0 msgr2=0x7fbac00c6fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:04.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.178+0000 7fbabf400640 1 --2- 172.21.15.6:0/455608950 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbac006bfc0 0x7fbac00c6fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:04.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.178+0000 7fbabf400640 1 -- 172.21.15.6:0/455608950 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbac0158a10 con 0x7fbac015dd00 2024-03-04T09:28:04.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.179+0000 7fbabf400640 1 --2- 172.21.15.6:0/455608950 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbac015dd00 0x7fbac0158150 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fbaac00d800 tx=0x7fbaac00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:04.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.179+0000 7fbab7e00640 1 -- 172.21.15.6:0/455608950 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaac00b6c0 con 0x7fbac015dd00 2024-03-04T09:28:04.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.179+0000 7fbab7e00640 1 -- 172.21.15.6:0/455608950 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbaac019070 con 0x7fbac015dd00 2024-03-04T09:28:04.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.179+0000 7fbab7e00640 1 -- 172.21.15.6:0/455608950 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaac00ba80 con 0x7fbac015dd00 2024-03-04T09:28:04.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.179+0000 7fbac6e00640 1 -- 172.21.15.6:0/455608950 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbac0158cf0 con 0x7fbac015dd00 2024-03-04T09:28:04.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.179+0000 7fbac6e00640 1 -- 172.21.15.6:0/455608950 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbac01592b0 con 0x7fbac015dd00 2024-03-04T09:28:04.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.180+0000 7fbac6e00640 1 -- 172.21.15.6:0/455608950 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbac004f3a0 con 0x7fbac015dd00 2024-03-04T09:28:04.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.182+0000 7fbab7e00640 1 -- 172.21.15.6:0/455608950 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbaac00bc20 con 0x7fbac015dd00 2024-03-04T09:28:04.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.182+0000 7fbab7e00640 1 --2- 172.21.15.6:0/455608950 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fbaa006b500 0x7fbaa006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:04.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.183+0000 7fbab7e00640 1 -- 172.21.15.6:0/455608950 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbaac08a6b0 con 0x7fbac015dd00 2024-03-04T09:28:04.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.183+0000 7fbabfe00640 1 --2- 172.21.15.6:0/455608950 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fbaa006b500 0x7fbaa006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:04.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.184+0000 7fbabfe00640 1 --2- 172.21.15.6:0/455608950 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fbaa006b500 0x7fbaa006d9c0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fbab0035000 tx=0x7fbab0030000 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:04.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.187+0000 7fbab7e00640 1 -- 172.21.15.6:0/455608950 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbaac059490 con 0x7fbac015dd00 2024-03-04T09:28:04.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.393+0000 7fbac6e00640 1 -- 172.21.15.6:0/455608950 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbac00724c0 con 0x7fbac015dd00 2024-03-04T09:28:04.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.396+0000 7fbab7e00640 1 -- 172.21.15.6:0/455608950 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v56) v1 ==== 74+0+16617 (secure 0 0 0) 0x7fbaac058d90 con 0x7fbac015dd00 2024-03-04T09:28:04.393 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:28:04.393 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":56,"fsid":"edce0578-5ae4-4402-b45a-552bdef84d7c","created":"2024-03-04T09:24:54.523780+0000","modified":"2024-03-04T09:28:01.413693+0000","last_up_change":"2024-03-04T09:25:07.089049+0000","last_in_change":"2024-03-04T09:24:57.982077+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T09:25:07.935198+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-04T09:27:57.774529+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-04T09:27:58.886504+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a96574f-81f1-4bc6-8463-757520b84a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6825","nonce":1364449401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6827","nonce":1364449401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6831","nonce":1364449401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6829","nonce":1364449401}]},"public_addr":"172.21.15.6:6825/1364449401","cluster_addr":"172.21.15.6:6827/1364449401","heartbeat_back_addr":"172.21.15.6:6831/1364449401","heartbeat_front_addr":"172.21.15.6:6829/1364449401","state":["exists","up"]},{"osd":1,"uuid":"7d79b9c1-6b87-476a-bc36-7db5ee64cd90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6809","nonce":2494103078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6811","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6813","nonce":2494103078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6821","nonce":2494103078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6817","nonce":2494103078}]},"public_addr":"172.21.15.6:6809/2494103078","cluster_addr":"172.21.15.6:6813/2494103078","heartbeat_back_addr":"172.21.15.6:6821/2494103078","heartbeat_front_addr":"172.21.15.6:6817/2494103078","state":["exists","up"]},{"osd":2,"uuid":"492331b9-4ee9-40b6-bacd-85d44cd6c45a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6812","nonce":1500066681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6816","nonce":1500066681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6824","nonce":1500066681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6820","nonce":1500066681}]},"public_addr":"172.21.15.6:6812/1500066681","cluster_addr":"172.21.15.6:6816/1500066681","heartbeat_back_addr":"172.21.15.6:6824/1500066681","heartbeat_front_addr":"172.21.15.6:6820/1500066681","state":["exists","up"]},{"osd":3,"uuid":"21bb2efe-9ba7-4bdc-bc31-180cd1102e21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6801","nonce":1558806123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6803","nonce":1558806123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6807","nonce":1558806123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6805","nonce":1558806123}]},"public_addr":"172.21.15.6:6801/1558806123","cluster_addr":"172.21.15.6:6803/1558806123","heartbeat_back_addr":"172.21.15.6:6807/1558806123","heartbeat_front_addr":"172.21.15.6:6805/1558806123","state":["exists","up"]},{"osd":4,"uuid":"6e2c4b71-19d9-4737-83d9-558705d8644a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6801","nonce":1526135780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6803","nonce":1526135780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6807","nonce":1526135780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6805","nonce":1526135780}]},"public_addr":"172.21.15.146:6801/1526135780","cluster_addr":"172.21.15.146:6803/1526135780","heartbeat_back_addr":"172.21.15.146:6807/1526135780","heartbeat_front_addr":"172.21.15.146:6805/1526135780","state":["exists","up"]},{"osd":5,"uuid":"fb69c525-f1be-4b7d-ab32-6c29b0d6fddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6809","nonce":460262507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6811","nonce":460262507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6815","nonce":460262507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6813","nonce":460262507}]},"public_addr":"172.21.15.146:6809/460262507","cluster_addr":"172.21.15.146:6811/460262507","heartbeat_back_addr":"172.21.15.146:6815/460262507","heartbeat_front_addr":"172.21.15.146:6813/460262507","state":["exists","up"]},{"osd":6,"uuid":"c32e7fed-370d-4401-8ce0-6a4335b0d168","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6823","nonce":1392897493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6827","nonce":1392897493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6831","nonce":1392897493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6829","nonce":1392897493}]},"public_addr":"172.21.15.146:6823/1392897493","cluster_addr":"172.21.15.146:6827/1392897493","heartbeat_back_addr":"172.21.15.146:6831/1392897493","heartbeat_front_addr":"172.21.15.146:6829/1392897493","state":["exists","up"]},{"osd":7,"uuid":"75a79b44-fa5f-426d-a7d3-71f21f78835c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6817","nonce":3268323156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6819","nonce":3268323156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6825","nonce":3268323156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6822","nonce":3268323156}]},"public_addr":"172.21.15.146:6817/3268323156","cluster_addr":"172.21.15.146:6819/3268323156","heartbeat_back_addr":"172.21.15.146:6825/3268323156","heartbeat_front_addr":"172.21.15.146:6822/3268323156","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.310437+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.295850+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.294674+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.526855+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.300023+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.595160+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.346046+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.558741+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T09:28:04.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.399+0000 7fbac6e00640 1 -- 172.21.15.6:0/455608950 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fbaa006b500 msgr2=0x7fbaa006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:04.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.399+0000 7fbac6e00640 1 --2- 172.21.15.6:0/455608950 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fbaa006b500 0x7fbaa006d9c0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fbab0035000 tx=0x7fbab0030000 comp rx=0 tx=0).stop 2024-03-04T09:28:04.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.399+0000 7fbac6e00640 1 -- 172.21.15.6:0/455608950 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbac015dd00 msgr2=0x7fbac0158150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:04.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.399+0000 7fbac6e00640 1 --2- 172.21.15.6:0/455608950 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbac015dd00 0x7fbac0158150 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fbaac00d800 tx=0x7fbaac00dcc0 comp rx=0 tx=0).stop 2024-03-04T09:28:04.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.399+0000 7fbac6e00640 1 -- 172.21.15.6:0/455608950 shutdown_connections 2024-03-04T09:28:04.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.399+0000 7fbac6e00640 1 --2- 172.21.15.6:0/455608950 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fbaa006b500 0x7fbaa006d9c0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:04.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.399+0000 7fbac6e00640 1 --2- 172.21.15.6:0/455608950 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbac015e0e0 0x7fbac0148ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:04.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.399+0000 7fbac6e00640 1 --2- 172.21.15.6:0/455608950 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbac015dd00 0x7fbac0158150 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:04.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.399+0000 7fbac6e00640 1 --2- 172.21.15.6:0/455608950 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbac006bfc0 0x7fbac00c6fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:04.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.399+0000 7fbac6e00640 1 -- 172.21.15.6:0/455608950 >> 172.21.15.6:0/455608950 conn(0x7fbac00714d0 msgr2=0x7fbac00b11e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:04.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.399+0000 7fbac6e00640 1 -- 172.21.15.6:0/455608950 shutdown_connections 2024-03-04T09:28:04.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.399+0000 7fbac6e00640 1 -- 172.21.15.6:0/455608950 wait complete. 2024-03-04T09:28:04.415 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-04T09:28:04.415 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T09:28:04.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.640+0000 7f6433200640 1 Processor -- start 2024-03-04T09:28:04.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.640+0000 7f6433200640 1 -- start start 2024-03-04T09:28:04.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.641+0000 7f6433200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f642c15e0b0 0x7f642c0cf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:04.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.641+0000 7f6433200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f642c0d0e50 0x7f642c0b44d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:04.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.642+0000 7f6433200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f642c0b3710 con 0x7f642c0d4a10 2024-03-04T09:28:04.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.642+0000 7f6433200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f642c0b09e0 con 0x7f642c0d0e50 2024-03-04T09:28:04.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.642+0000 7f6433200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f642c0b39f0 con 0x7f642c15e0b0 2024-03-04T09:28:04.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.642+0000 7f6432800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f642c0d0e50 0x7f642c0b44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:04.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.642+0000 7f6432800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f642c0d0e50 0x7f642c0b44d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45328/0 (socket says 172.21.15.6:45328) 2024-03-04T09:28:04.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.642+0000 7f6432800640 1 -- 172.21.15.6:0/2967574321 learned_addr learned my addr 172.21.15.6:0/2967574321 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:28:04.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.642+0000 7f6431e00640 1 --2- 172.21.15.6:0/2967574321 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f642c15e0b0 0x7f642c0cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:04.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.642+0000 7f6432800640 1 -- 172.21.15.6:0/2967574321 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f642c15e0b0 msgr2=0x7f642c0cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:04.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.642+0000 7f6432800640 1 --2- 172.21.15.6:0/2967574321 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f642c15e0b0 0x7f642c0cf310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:04.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.642+0000 7f6432800640 1 -- 172.21.15.6:0/2967574321 >> v1:172.21.15.146:6789/0 conn(0x7f642c0d4a10 legacy=0x7f642c0d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:04.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.642+0000 7f6432800640 1 -- 172.21.15.6:0/2967574321 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f642c0b3cd0 con 0x7f642c0d0e50 2024-03-04T09:28:04.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.644+0000 7f6432800640 1 --2- 172.21.15.6:0/2967574321 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f642c0d0e50 0x7f642c0b44d0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f6428009e50 tx=0x7f642802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b8cab16723655ea server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:04.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.644+0000 7f6430a00640 1 -- 172.21.15.6:0/2967574321 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f642802ca80 con 0x7f642c0d0e50 2024-03-04T09:28:04.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.644+0000 7f6430a00640 1 -- 172.21.15.6:0/2967574321 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f642802cc20 con 0x7f642c0d0e50 2024-03-04T09:28:04.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.644+0000 7f6430a00640 1 -- 172.21.15.6:0/2967574321 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64280343f0 con 0x7f642c0d0e50 2024-03-04T09:28:04.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.644+0000 7f6433200640 1 -- 172.21.15.6:0/2967574321 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f642c0d0e50 msgr2=0x7f642c0b44d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:04.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.644+0000 7f6433200640 1 --2- 172.21.15.6:0/2967574321 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f642c0d0e50 0x7f642c0b44d0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f6428009e50 tx=0x7f642802bee0 comp rx=0 tx=0).stop 2024-03-04T09:28:04.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.645+0000 7f6433200640 1 -- 172.21.15.6:0/2967574321 shutdown_connections 2024-03-04T09:28:04.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.645+0000 7f6433200640 1 --2- 172.21.15.6:0/2967574321 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f642c0d0e50 0x7f642c0b44d0 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:04.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.645+0000 7f6433200640 1 --2- 172.21.15.6:0/2967574321 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f642c15e0b0 0x7f642c0cf310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:04.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.645+0000 7f6433200640 1 -- 172.21.15.6:0/2967574321 >> 172.21.15.6:0/2967574321 conn(0x7f642c0d3d30 msgr2=0x7f642c0d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:04.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.645+0000 7f6433200640 1 -- 172.21.15.6:0/2967574321 shutdown_connections 2024-03-04T09:28:04.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.645+0000 7f6433200640 1 -- 172.21.15.6:0/2967574321 wait complete. 2024-03-04T09:28:04.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.646+0000 7f6433200640 1 Processor -- start 2024-03-04T09:28:04.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.646+0000 7f6433200640 1 -- start start 2024-03-04T09:28:04.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.647+0000 7f6433200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f642c0d0e50 0x7f642c15cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:04.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.647+0000 7f6433200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f642c0d4a10 0x7f642c0b7360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:04.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.647+0000 7f6433200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f642c15e0b0 0x7f642c0b91e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:04.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.647+0000 7f6433200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f642c0cfc60 con 0x7f642c0d0e50 2024-03-04T09:28:04.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.647+0000 7f6433200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f642c15e490 con 0x7f642c0d4a10 2024-03-04T09:28:04.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.647+0000 7f6433200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f642c0cf850 con 0x7f642c15e0b0 2024-03-04T09:28:04.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.648+0000 7f6431e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f642c0d0e50 0x7f642c15cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:04.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.648+0000 7f6431e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f642c0d0e50 0x7f642c15cdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45332/0 (socket says 172.21.15.6:45332) 2024-03-04T09:28:04.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.648+0000 7f6431e00640 1 -- 172.21.15.6:0/4196235149 learned_addr learned my addr 172.21.15.6:0/4196235149 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:28:04.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.648+0000 7f6432800640 1 --2- 172.21.15.6:0/4196235149 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f642c15e0b0 0x7f642c0b91e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:04.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.648+0000 7f6431400640 1 --2- 172.21.15.6:0/4196235149 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f642c0d4a10 0x7f642c0b7360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:04.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.648+0000 7f6431e00640 1 -- 172.21.15.6:0/4196235149 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f642c15e0b0 msgr2=0x7f642c0b91e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:04.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.648+0000 7f6431e00640 1 --2- 172.21.15.6:0/4196235149 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f642c15e0b0 0x7f642c0b91e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:04.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.648+0000 7f6431e00640 1 -- 172.21.15.6:0/4196235149 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f642c0d4a10 msgr2=0x7f642c0b7360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:04.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.648+0000 7f6431e00640 1 --2- 172.21.15.6:0/4196235149 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f642c0d4a10 0x7f642c0b7360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:04.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.648+0000 7f6431e00640 1 -- 172.21.15.6:0/4196235149 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f642c15d330 con 0x7f642c0d0e50 2024-03-04T09:28:04.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.649+0000 7f6431e00640 1 --2- 172.21.15.6:0/4196235149 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f642c0d0e50 0x7f642c15cdf0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f642000c820 tx=0x7f642000cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:04.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.649+0000 7f6426000640 1 -- 172.21.15.6:0/4196235149 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6420015070 con 0x7f642c0d0e50 2024-03-04T09:28:04.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.649+0000 7f6426000640 1 -- 172.21.15.6:0/4196235149 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6420004020 con 0x7f642c0d0e50 2024-03-04T09:28:04.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.649+0000 7f6426000640 1 -- 172.21.15.6:0/4196235149 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64200043f0 con 0x7f642c0d0e50 2024-03-04T09:28:04.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.649+0000 7f6433200640 1 -- 172.21.15.6:0/4196235149 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f642c0b7bb0 con 0x7f642c0d0e50 2024-03-04T09:28:04.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.649+0000 7f6433200640 1 -- 172.21.15.6:0/4196235149 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f642c0b8170 con 0x7f642c0d0e50 2024-03-04T09:28:04.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.651+0000 7f6433200640 1 -- 172.21.15.6:0/4196235149 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f642c04f330 con 0x7f642c0d0e50 2024-03-04T09:28:04.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.651+0000 7f6426000640 1 -- 172.21.15.6:0/4196235149 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6420007830 con 0x7f642c0d0e50 2024-03-04T09:28:04.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.652+0000 7f6426000640 1 --2- 172.21.15.6:0/4196235149 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f641006b500 0x7f641006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:04.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.652+0000 7f6426000640 1 -- 172.21.15.6:0/4196235149 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f642008a000 con 0x7f642c0d0e50 2024-03-04T09:28:04.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.653+0000 7f6431400640 1 --2- 172.21.15.6:0/4196235149 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f641006b500 0x7f641006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:04.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.654+0000 7f6431400640 1 --2- 172.21.15.6:0/4196235149 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f641006b500 0x7f641006d9c0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f642c0c98f0 tx=0x7f641c009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:04.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.657+0000 7f6426000640 1 -- 172.21.15.6:0/4196235149 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6420058df0 con 0x7f642c0d0e50 2024-03-04T09:28:04.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.860+0000 7f6433200640 1 -- 172.21.15.6:0/4196235149 --> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f642c0d39a0 con 0x7f641006b500 2024-03-04T09:28:04.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.880+0000 7f6426000640 1 -- 172.21.15.6:0/4196235149 <== mgr.4113 v2:172.21.15.146:6832/1788613334 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251954 (secure 0 0 0) 0x7f642c0d39a0 con 0x7f641006b500 2024-03-04T09:28:04.881 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:28:04.882 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-03-04T09:28:04.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.886+0000 7f6433200640 1 -- 172.21.15.6:0/4196235149 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f641006b500 msgr2=0x7f641006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:04.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.886+0000 7f6433200640 1 --2- 172.21.15.6:0/4196235149 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f641006b500 0x7f641006d9c0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f642c0c98f0 tx=0x7f641c009040 comp rx=0 tx=0).stop 2024-03-04T09:28:04.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.886+0000 7f6433200640 1 -- 172.21.15.6:0/4196235149 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f642c0d0e50 msgr2=0x7f642c15cdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:04.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.886+0000 7f6433200640 1 --2- 172.21.15.6:0/4196235149 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f642c0d0e50 0x7f642c15cdf0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f642000c820 tx=0x7f642000cce0 comp rx=0 tx=0).stop 2024-03-04T09:28:04.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.887+0000 7f6433200640 1 -- 172.21.15.6:0/4196235149 shutdown_connections 2024-03-04T09:28:04.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.887+0000 7f6433200640 1 --2- 172.21.15.6:0/4196235149 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f641006b500 0x7f641006d9c0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:04.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.887+0000 7f6433200640 1 --2- 172.21.15.6:0/4196235149 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f642c15e0b0 0x7f642c0b91e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:04.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.887+0000 7f6433200640 1 --2- 172.21.15.6:0/4196235149 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f642c0d4a10 0x7f642c0b7360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:04.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.887+0000 7f6433200640 1 --2- 172.21.15.6:0/4196235149 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f642c0d0e50 0x7f642c15cdf0 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:04.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.887+0000 7f6433200640 1 -- 172.21.15.6:0/4196235149 >> 172.21.15.6:0/4196235149 conn(0x7f642c0d3d30 msgr2=0x7f642c160620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:04.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.887+0000 7f6433200640 1 -- 172.21.15.6:0/4196235149 shutdown_connections 2024-03-04T09:28:04.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:04.887+0000 7f6433200640 1 -- 172.21.15.6:0/4196235149 wait complete. 2024-03-04T09:28:04.904 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":137,"stamp":"2024-03-04T09:28:03.916879+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":163,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":255960,"kb_used_data":6472,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754718760,"statfs":{"total":773094113280,"available":772832010240,"internally_reserved":0,"allocated":6627328,"data_stored":3413996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":42,"apply_latency_ms":42,"commit_latency_ns":42000000,"apply_latency_ns":42000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.547349"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.690192+0000","last_change":"2024-03-04T09:28:00.434177+0000","last_active":"2024-03-04T09:28:01.690192+0000","last_peered":"2024-03-04T09:28:01.690192+0000","last_clean":"2024-03-04T09:28:01.690192+0000","last_became_active":"2024-03-04T09:28:00.433834+0000","last_became_peered":"2024-03-04T09:28:00.433834+0000","last_unstale":"2024-03-04T09:28:01.690192+0000","last_undegraded":"2024-03-04T09:28:01.690192+0000","last_fullsized":"2024-03-04T09:28:01.690192+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:00:26.863147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.691526+0000","last_change":"2024-03-04T09:28:00.436530+0000","last_active":"2024-03-04T09:28:01.691526+0000","last_peered":"2024-03-04T09:28:01.691526+0000","last_clean":"2024-03-04T09:28:01.691526+0000","last_became_active":"2024-03-04T09:28:00.436242+0000","last_became_peered":"2024-03-04T09:28:00.436242+0000","last_unstale":"2024-03-04T09:28:01.691526+0000","last_undegraded":"2024-03-04T09:28:01.691526+0000","last_fullsized":"2024-03-04T09:28:01.691526+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:05:00.926651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.692194+0000","last_change":"2024-03-04T09:28:00.430122+0000","last_active":"2024-03-04T09:28:01.692194+0000","last_peered":"2024-03-04T09:28:01.692194+0000","last_clean":"2024-03-04T09:28:01.692194+0000","last_became_active":"2024-03-04T09:28:00.429188+0000","last_became_peered":"2024-03-04T09:28:00.429188+0000","last_unstale":"2024-03-04T09:28:01.692194+0000","last_undegraded":"2024-03-04T09:28:01.692194+0000","last_fullsized":"2024-03-04T09:28:01.692194+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:05:38.576682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.692257+0000","last_change":"2024-03-04T09:28:00.439691+0000","last_active":"2024-03-04T09:28:01.692257+0000","last_peered":"2024-03-04T09:28:01.692257+0000","last_clean":"2024-03-04T09:28:01.692257+0000","last_became_active":"2024-03-04T09:28:00.439348+0000","last_became_peered":"2024-03-04T09:28:00.439348+0000","last_unstale":"2024-03-04T09:28:01.692257+0000","last_undegraded":"2024-03-04T09:28:01.692257+0000","last_fullsized":"2024-03-04T09:28:01.692257+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:18:19.609722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.689501+0000","last_change":"2024-03-04T09:28:00.429107+0000","last_active":"2024-03-04T09:28:01.689501+0000","last_peered":"2024-03-04T09:28:01.689501+0000","last_clean":"2024-03-04T09:28:01.689501+0000","last_became_active":"2024-03-04T09:28:00.428800+0000","last_became_peered":"2024-03-04T09:28:00.428800+0000","last_unstale":"2024-03-04T09:28:01.689501+0000","last_undegraded":"2024-03-04T09:28:01.689501+0000","last_fullsized":"2024-03-04T09:28:01.689501+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:52:28.113950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.428937+0000","last_change":"2024-03-04T09:28:00.439396+0000","last_active":"2024-03-04T09:28:01.428937+0000","last_peered":"2024-03-04T09:28:01.428937+0000","last_clean":"2024-03-04T09:28:01.428937+0000","last_became_active":"2024-03-04T09:28:00.439129+0000","last_became_peered":"2024-03-04T09:28:00.439129+0000","last_unstale":"2024-03-04T09:28:01.428937+0000","last_undegraded":"2024-03-04T09:28:01.428937+0000","last_fullsized":"2024-03-04T09:28:01.428937+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:42:00.605123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.692436+0000","last_change":"2024-03-04T09:28:00.440692+0000","last_active":"2024-03-04T09:28:01.692436+0000","last_peered":"2024-03-04T09:28:01.692436+0000","last_clean":"2024-03-04T09:28:01.692436+0000","last_became_active":"2024-03-04T09:28:00.438138+0000","last_became_peered":"2024-03-04T09:28:00.438138+0000","last_unstale":"2024-03-04T09:28:01.692436+0000","last_undegraded":"2024-03-04T09:28:01.692436+0000","last_fullsized":"2024-03-04T09:28:01.692436+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:24:39.308837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.690247+0000","last_change":"2024-03-04T09:28:00.438344+0000","last_active":"2024-03-04T09:28:01.690247+0000","last_peered":"2024-03-04T09:28:01.690247+0000","last_clean":"2024-03-04T09:28:01.690247+0000","last_became_active":"2024-03-04T09:28:00.438076+0000","last_became_peered":"2024-03-04T09:28:00.438076+0000","last_unstale":"2024-03-04T09:28:01.690247+0000","last_undegraded":"2024-03-04T09:28:01.690247+0000","last_fullsized":"2024-03-04T09:28:01.690247+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:39:24.789268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.690928+0000","last_change":"2024-03-04T09:28:00.436693+0000","last_active":"2024-03-04T09:28:01.690928+0000","last_peered":"2024-03-04T09:28:01.690928+0000","last_clean":"2024-03-04T09:28:01.690928+0000","last_became_active":"2024-03-04T09:28:00.436422+0000","last_became_peered":"2024-03-04T09:28:00.436422+0000","last_unstale":"2024-03-04T09:28:01.690928+0000","last_undegraded":"2024-03-04T09:28:01.690928+0000","last_fullsized":"2024-03-04T09:28:01.690928+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:14:30.846824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.425773+0000","last_change":"2024-03-04T09:28:00.429162+0000","last_active":"2024-03-04T09:28:01.425773+0000","last_peered":"2024-03-04T09:28:01.425773+0000","last_clean":"2024-03-04T09:28:01.425773+0000","last_became_active":"2024-03-04T09:28:00.428868+0000","last_became_peered":"2024-03-04T09:28:00.428868+0000","last_unstale":"2024-03-04T09:28:01.425773+0000","last_undegraded":"2024-03-04T09:28:01.425773+0000","last_fullsized":"2024-03-04T09:28:01.425773+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:45:56.673753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.425632+0000","last_change":"2024-03-04T09:28:00.423881+0000","last_active":"2024-03-04T09:28:01.425632+0000","last_peered":"2024-03-04T09:28:01.425632+0000","last_clean":"2024-03-04T09:28:01.425632+0000","last_became_active":"2024-03-04T09:28:00.423336+0000","last_became_peered":"2024-03-04T09:28:00.423336+0000","last_unstale":"2024-03-04T09:28:01.425632+0000","last_undegraded":"2024-03-04T09:28:01.425632+0000","last_fullsized":"2024-03-04T09:28:01.425632+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:41:20.384844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.692437+0000","last_change":"2024-03-04T09:28:00.433918+0000","last_active":"2024-03-04T09:28:01.692437+0000","last_peered":"2024-03-04T09:28:01.692437+0000","last_clean":"2024-03-04T09:28:01.692437+0000","last_became_active":"2024-03-04T09:28:00.433625+0000","last_became_peered":"2024-03-04T09:28:00.433625+0000","last_unstale":"2024-03-04T09:28:01.692437+0000","last_undegraded":"2024-03-04T09:28:01.692437+0000","last_fullsized":"2024-03-04T09:28:01.692437+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:38:44.223188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.428236+0000","last_change":"2024-03-04T09:28:00.435007+0000","last_active":"2024-03-04T09:28:01.428236+0000","last_peered":"2024-03-04T09:28:01.428236+0000","last_clean":"2024-03-04T09:28:01.428236+0000","last_became_active":"2024-03-04T09:28:00.434657+0000","last_became_peered":"2024-03-04T09:28:00.434657+0000","last_unstale":"2024-03-04T09:28:01.428236+0000","last_undegraded":"2024-03-04T09:28:01.428236+0000","last_fullsized":"2024-03-04T09:28:01.428236+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:24:53.662456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.690218+0000","last_change":"2024-03-04T09:28:00.439224+0000","last_active":"2024-03-04T09:28:01.690218+0000","last_peered":"2024-03-04T09:28:01.690218+0000","last_clean":"2024-03-04T09:28:01.690218+0000","last_became_active":"2024-03-04T09:28:00.438890+0000","last_became_peered":"2024-03-04T09:28:00.438890+0000","last_unstale":"2024-03-04T09:28:01.690218+0000","last_undegraded":"2024-03-04T09:28:01.690218+0000","last_fullsized":"2024-03-04T09:28:01.690218+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:43:29.068713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.692178+0000","last_change":"2024-03-04T09:28:00.432426+0000","last_active":"2024-03-04T09:28:01.692178+0000","last_peered":"2024-03-04T09:28:01.692178+0000","last_clean":"2024-03-04T09:28:01.692178+0000","last_became_active":"2024-03-04T09:28:00.432126+0000","last_became_peered":"2024-03-04T09:28:00.432126+0000","last_unstale":"2024-03-04T09:28:01.692178+0000","last_undegraded":"2024-03-04T09:28:01.692178+0000","last_fullsized":"2024-03-04T09:28:01.692178+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:30:38.303164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.428530+0000","last_change":"2024-03-04T09:28:00.433183+0000","last_active":"2024-03-04T09:28:01.428530+0000","last_peered":"2024-03-04T09:28:01.428530+0000","last_clean":"2024-03-04T09:28:01.428530+0000","last_became_active":"2024-03-04T09:28:00.431904+0000","last_became_peered":"2024-03-04T09:28:00.431904+0000","last_unstale":"2024-03-04T09:28:01.428530+0000","last_undegraded":"2024-03-04T09:28:01.428530+0000","last_fullsized":"2024-03-04T09:28:01.428530+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:05:38.576682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.692188+0000","last_change":"2024-03-04T09:28:00.436128+0000","last_active":"2024-03-04T09:28:01.692188+0000","last_peered":"2024-03-04T09:28:01.692188+0000","last_clean":"2024-03-04T09:28:01.692188+0000","last_became_active":"2024-03-04T09:28:00.434844+0000","last_became_peered":"2024-03-04T09:28:00.434844+0000","last_unstale":"2024-03-04T09:28:01.692188+0000","last_undegraded":"2024-03-04T09:28:01.692188+0000","last_fullsized":"2024-03-04T09:28:01.692188+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:28:31.343554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.691842+0000","last_change":"2024-03-04T09:28:00.433628+0000","last_active":"2024-03-04T09:28:01.691842+0000","last_peered":"2024-03-04T09:28:01.691842+0000","last_clean":"2024-03-04T09:28:01.691842+0000","last_became_active":"2024-03-04T09:28:00.433343+0000","last_became_peered":"2024-03-04T09:28:00.433343+0000","last_unstale":"2024-03-04T09:28:01.691842+0000","last_undegraded":"2024-03-04T09:28:01.691842+0000","last_fullsized":"2024-03-04T09:28:01.691842+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:17:09.522879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.689494+0000","last_change":"2024-03-04T09:28:00.424933+0000","last_active":"2024-03-04T09:28:01.689494+0000","last_peered":"2024-03-04T09:28:01.689494+0000","last_clean":"2024-03-04T09:28:01.689494+0000","last_became_active":"2024-03-04T09:28:00.424644+0000","last_became_peered":"2024-03-04T09:28:00.424644+0000","last_unstale":"2024-03-04T09:28:01.689494+0000","last_undegraded":"2024-03-04T09:28:01.689494+0000","last_fullsized":"2024-03-04T09:28:01.689494+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:49:58.002468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.691417+0000","last_change":"2024-03-04T09:28:00.439954+0000","last_active":"2024-03-04T09:28:01.691417+0000","last_peered":"2024-03-04T09:28:01.691417+0000","last_clean":"2024-03-04T09:28:01.691417+0000","last_became_active":"2024-03-04T09:28:00.439672+0000","last_became_peered":"2024-03-04T09:28:00.439672+0000","last_unstale":"2024-03-04T09:28:01.691417+0000","last_undegraded":"2024-03-04T09:28:01.691417+0000","last_fullsized":"2024-03-04T09:28:01.691417+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:42:03.865562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.693565+0000","last_change":"2024-03-04T09:28:00.430020+0000","last_active":"2024-03-04T09:28:01.693565+0000","last_peered":"2024-03-04T09:28:01.693565+0000","last_clean":"2024-03-04T09:28:01.693565+0000","last_became_active":"2024-03-04T09:28:00.429673+0000","last_became_peered":"2024-03-04T09:28:00.429673+0000","last_unstale":"2024-03-04T09:28:01.693565+0000","last_undegraded":"2024-03-04T09:28:01.693565+0000","last_fullsized":"2024-03-04T09:28:01.693565+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:47:11.430142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.691015+0000","last_change":"2024-03-04T09:28:00.431742+0000","last_active":"2024-03-04T09:28:01.691015+0000","last_peered":"2024-03-04T09:28:01.691015+0000","last_clean":"2024-03-04T09:28:01.691015+0000","last_became_active":"2024-03-04T09:28:00.431486+0000","last_became_peered":"2024-03-04T09:28:00.431486+0000","last_unstale":"2024-03-04T09:28:01.691015+0000","last_undegraded":"2024-03-04T09:28:01.691015+0000","last_fullsized":"2024-03-04T09:28:01.691015+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:12:48.679791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.693264+0000","last_change":"2024-03-04T09:28:00.432686+0000","last_active":"2024-03-04T09:28:01.693264+0000","last_peered":"2024-03-04T09:28:01.693264+0000","last_clean":"2024-03-04T09:28:01.693264+0000","last_became_active":"2024-03-04T09:28:00.430974+0000","last_became_peered":"2024-03-04T09:28:00.430974+0000","last_unstale":"2024-03-04T09:28:01.693264+0000","last_undegraded":"2024-03-04T09:28:01.693264+0000","last_fullsized":"2024-03-04T09:28:01.693264+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:55:21.623042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.427793+0000","last_change":"2024-03-04T09:28:00.436106+0000","last_active":"2024-03-04T09:28:01.427793+0000","last_peered":"2024-03-04T09:28:01.427793+0000","last_clean":"2024-03-04T09:28:01.427793+0000","last_became_active":"2024-03-04T09:28:00.435781+0000","last_became_peered":"2024-03-04T09:28:00.435781+0000","last_unstale":"2024-03-04T09:28:01.427793+0000","last_undegraded":"2024-03-04T09:28:01.427793+0000","last_fullsized":"2024-03-04T09:28:01.427793+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:16:28.420706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.690312+0000","last_change":"2024-03-04T09:28:00.414866+0000","last_active":"2024-03-04T09:28:01.690312+0000","last_peered":"2024-03-04T09:28:01.690312+0000","last_clean":"2024-03-04T09:28:01.690312+0000","last_became_active":"2024-03-04T09:28:00.414050+0000","last_became_peered":"2024-03-04T09:28:00.414050+0000","last_unstale":"2024-03-04T09:28:01.690312+0000","last_undegraded":"2024-03-04T09:28:01.690312+0000","last_fullsized":"2024-03-04T09:28:01.690312+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:15:08.601355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.692170+0000","last_change":"2024-03-04T09:28:00.433459+0000","last_active":"2024-03-04T09:28:01.692170+0000","last_peered":"2024-03-04T09:28:01.692170+0000","last_clean":"2024-03-04T09:28:01.692170+0000","last_became_active":"2024-03-04T09:28:00.433111+0000","last_became_peered":"2024-03-04T09:28:00.433111+0000","last_unstale":"2024-03-04T09:28:01.692170+0000","last_undegraded":"2024-03-04T09:28:01.692170+0000","last_fullsized":"2024-03-04T09:28:01.692170+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:05:38.576682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.429400+0000","last_change":"2024-03-04T09:28:00.437328+0000","last_active":"2024-03-04T09:28:01.429400+0000","last_peered":"2024-03-04T09:28:01.429400+0000","last_clean":"2024-03-04T09:28:01.429400+0000","last_became_active":"2024-03-04T09:28:00.437044+0000","last_became_peered":"2024-03-04T09:28:00.437044+0000","last_unstale":"2024-03-04T09:28:01.429400+0000","last_undegraded":"2024-03-04T09:28:01.429400+0000","last_fullsized":"2024-03-04T09:28:01.429400+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:29:14.123293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.689030+0000","last_change":"2024-03-04T09:28:00.440592+0000","last_active":"2024-03-04T09:28:01.689030+0000","last_peered":"2024-03-04T09:28:01.689030+0000","last_clean":"2024-03-04T09:28:01.689030+0000","last_became_active":"2024-03-04T09:28:00.440186+0000","last_became_peered":"2024-03-04T09:28:00.440186+0000","last_unstale":"2024-03-04T09:28:01.689030+0000","last_undegraded":"2024-03-04T09:28:01.689030+0000","last_fullsized":"2024-03-04T09:28:01.689030+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:22:44.938157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.691829+0000","last_change":"2024-03-04T09:28:00.445380+0000","last_active":"2024-03-04T09:28:01.691829+0000","last_peered":"2024-03-04T09:28:01.691829+0000","last_clean":"2024-03-04T09:28:01.691829+0000","last_became_active":"2024-03-04T09:28:00.443638+0000","last_became_peered":"2024-03-04T09:28:00.443638+0000","last_unstale":"2024-03-04T09:28:01.691829+0000","last_undegraded":"2024-03-04T09:28:01.691829+0000","last_fullsized":"2024-03-04T09:28:01.691829+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:40:01.332927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.692922+0000","last_change":"2024-03-04T09:28:00.433320+0000","last_active":"2024-03-04T09:28:01.692922+0000","last_peered":"2024-03-04T09:28:01.692922+0000","last_clean":"2024-03-04T09:28:01.692922+0000","last_became_active":"2024-03-04T09:28:00.432271+0000","last_became_peered":"2024-03-04T09:28:00.432271+0000","last_unstale":"2024-03-04T09:28:01.692922+0000","last_undegraded":"2024-03-04T09:28:01.692922+0000","last_fullsized":"2024-03-04T09:28:01.692922+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:17:30.816921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.688694+0000","last_change":"2024-03-04T09:28:00.417556+0000","last_active":"2024-03-04T09:28:01.688694+0000","last_peered":"2024-03-04T09:28:01.688694+0000","last_clean":"2024-03-04T09:28:01.688694+0000","last_became_active":"2024-03-04T09:28:00.417265+0000","last_became_peered":"2024-03-04T09:28:00.417265+0000","last_unstale":"2024-03-04T09:28:01.688694+0000","last_undegraded":"2024-03-04T09:28:01.688694+0000","last_fullsized":"2024-03-04T09:28:01.688694+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:18:54.330037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.428623+0000","last_change":"2024-03-04T09:28:00.447034+0000","last_active":"2024-03-04T09:28:01.428623+0000","last_peered":"2024-03-04T09:28:01.428623+0000","last_clean":"2024-03-04T09:28:01.428623+0000","last_became_active":"2024-03-04T09:28:00.446685+0000","last_became_peered":"2024-03-04T09:28:00.446685+0000","last_unstale":"2024-03-04T09:28:01.428623+0000","last_undegraded":"2024-03-04T09:28:01.428623+0000","last_fullsized":"2024-03-04T09:28:01.428623+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:18:19.609722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.430062+0000","last_change":"2024-03-04T09:28:00.436807+0000","last_active":"2024-03-04T09:28:01.430062+0000","last_peered":"2024-03-04T09:28:01.430062+0000","last_clean":"2024-03-04T09:28:01.430062+0000","last_became_active":"2024-03-04T09:28:00.436491+0000","last_became_peered":"2024-03-04T09:28:00.436491+0000","last_unstale":"2024-03-04T09:28:01.430062+0000","last_undegraded":"2024-03-04T09:28:01.430062+0000","last_fullsized":"2024-03-04T09:28:01.430062+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:47:11.430142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"55'4","reported_seq":18,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.429371+0000","last_change":"2024-03-04T09:27:59.407621+0000","last_active":"2024-03-04T09:28:01.429371+0000","last_peered":"2024-03-04T09:28:01.429371+0000","last_clean":"2024-03-04T09:28:01.429371+0000","last_became_active":"2024-03-04T09:27:59.407362+0000","last_became_peered":"2024-03-04T09:27:59.407362+0000","last_unstale":"2024-03-04T09:28:01.429371+0000","last_undegraded":"2024-03-04T09:28:01.429371+0000","last_fullsized":"2024-03-04T09:28:01.429371+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_clean_scrub_stamp":"2024-03-04T09:27:58.353037+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:18:59.307799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.426183+0000","last_change":"2024-03-04T09:28:00.430027+0000","last_active":"2024-03-04T09:28:01.426183+0000","last_peered":"2024-03-04T09:28:01.426183+0000","last_clean":"2024-03-04T09:28:01.426183+0000","last_became_active":"2024-03-04T09:28:00.429778+0000","last_became_peered":"2024-03-04T09:28:00.429778+0000","last_unstale":"2024-03-04T09:28:01.426183+0000","last_undegraded":"2024-03-04T09:28:01.426183+0000","last_fullsized":"2024-03-04T09:28:01.426183+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:33:19.462053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"55'1","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.691793+0000","last_change":"2024-03-04T09:27:59.412265+0000","last_active":"2024-03-04T09:28:01.691793+0000","last_peered":"2024-03-04T09:28:01.691793+0000","last_clean":"2024-03-04T09:28:01.691793+0000","last_became_active":"2024-03-04T09:27:59.411885+0000","last_became_peered":"2024-03-04T09:27:59.411885+0000","last_unstale":"2024-03-04T09:28:01.691793+0000","last_undegraded":"2024-03-04T09:28:01.691793+0000","last_fullsized":"2024-03-04T09:28:01.691793+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_clean_scrub_stamp":"2024-03-04T09:27:58.353037+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:05:37.562815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.429168+0000","last_change":"2024-03-04T09:28:00.438663+0000","last_active":"2024-03-04T09:28:01.429168+0000","last_peered":"2024-03-04T09:28:01.429168+0000","last_clean":"2024-03-04T09:28:01.429168+0000","last_became_active":"2024-03-04T09:28:00.438349+0000","last_became_peered":"2024-03-04T09:28:00.438349+0000","last_unstale":"2024-03-04T09:28:01.429168+0000","last_undegraded":"2024-03-04T09:28:01.429168+0000","last_fullsized":"2024-03-04T09:28:01.429168+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:17:30.816921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"55'2","reported_seq":16,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.430160+0000","last_change":"2024-03-04T09:27:59.409928+0000","last_active":"2024-03-04T09:28:01.430160+0000","last_peered":"2024-03-04T09:28:01.430160+0000","last_clean":"2024-03-04T09:28:01.430160+0000","last_became_active":"2024-03-04T09:27:59.409602+0000","last_became_peered":"2024-03-04T09:27:59.409602+0000","last_unstale":"2024-03-04T09:28:01.430160+0000","last_undegraded":"2024-03-04T09:28:01.430160+0000","last_fullsized":"2024-03-04T09:28:01.430160+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_clean_scrub_stamp":"2024-03-04T09:27:58.353037+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:43:28.054846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.429859+0000","last_change":"2024-03-04T09:28:00.442481+0000","last_active":"2024-03-04T09:28:01.429859+0000","last_peered":"2024-03-04T09:28:01.429859+0000","last_clean":"2024-03-04T09:28:01.429859+0000","last_became_active":"2024-03-04T09:28:00.442160+0000","last_became_peered":"2024-03-04T09:28:00.442160+0000","last_unstale":"2024-03-04T09:28:01.429859+0000","last_undegraded":"2024-03-04T09:28:01.429859+0000","last_fullsized":"2024-03-04T09:28:01.429859+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:43:51.380199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"55'2","reported_seq":16,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.692555+0000","last_change":"2024-03-04T09:27:59.410484+0000","last_active":"2024-03-04T09:28:01.692555+0000","last_peered":"2024-03-04T09:28:01.692555+0000","last_clean":"2024-03-04T09:28:01.692555+0000","last_became_active":"2024-03-04T09:27:59.409118+0000","last_became_peered":"2024-03-04T09:27:59.409118+0000","last_unstale":"2024-03-04T09:28:01.692555+0000","last_undegraded":"2024-03-04T09:28:01.692555+0000","last_fullsized":"2024-03-04T09:28:01.692555+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_clean_scrub_stamp":"2024-03-04T09:27:58.353037+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:43:28.054846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.692105+0000","last_change":"2024-03-04T09:28:00.443500+0000","last_active":"2024-03-04T09:28:01.692105+0000","last_peered":"2024-03-04T09:28:01.692105+0000","last_clean":"2024-03-04T09:28:01.692105+0000","last_became_active":"2024-03-04T09:28:00.443138+0000","last_became_peered":"2024-03-04T09:28:00.443138+0000","last_unstale":"2024-03-04T09:28:01.692105+0000","last_undegraded":"2024-03-04T09:28:01.692105+0000","last_fullsized":"2024-03-04T09:28:01.692105+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:46:54.935661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"55'1","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.691213+0000","last_change":"2024-03-04T09:27:59.408748+0000","last_active":"2024-03-04T09:28:01.691213+0000","last_peered":"2024-03-04T09:28:01.691213+0000","last_clean":"2024-03-04T09:28:01.691213+0000","last_became_active":"2024-03-04T09:27:59.407429+0000","last_became_peered":"2024-03-04T09:27:59.407429+0000","last_unstale":"2024-03-04T09:28:01.691213+0000","last_undegraded":"2024-03-04T09:28:01.691213+0000","last_fullsized":"2024-03-04T09:28:01.691213+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_clean_scrub_stamp":"2024-03-04T09:27:58.353037+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:33:18.448186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.691377+0000","last_change":"2024-03-04T09:28:00.443410+0000","last_active":"2024-03-04T09:28:01.691377+0000","last_peered":"2024-03-04T09:28:01.691377+0000","last_clean":"2024-03-04T09:28:01.691377+0000","last_became_active":"2024-03-04T09:28:00.440808+0000","last_became_peered":"2024-03-04T09:28:00.440808+0000","last_unstale":"2024-03-04T09:28:01.691377+0000","last_undegraded":"2024-03-04T09:28:01.691377+0000","last_fullsized":"2024-03-04T09:28:01.691377+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:54:46.710146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"55'1","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.428891+0000","last_change":"2024-03-04T09:27:59.415921+0000","last_active":"2024-03-04T09:28:01.428891+0000","last_peered":"2024-03-04T09:28:01.428891+0000","last_clean":"2024-03-04T09:28:01.428891+0000","last_became_active":"2024-03-04T09:27:59.415586+0000","last_became_peered":"2024-03-04T09:27:59.415586+0000","last_unstale":"2024-03-04T09:28:01.428891+0000","last_undegraded":"2024-03-04T09:28:01.428891+0000","last_fullsized":"2024-03-04T09:28:01.428891+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_clean_scrub_stamp":"2024-03-04T09:27:58.353037+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:52:31.197392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.427866+0000","last_change":"2024-03-04T09:28:00.430007+0000","last_active":"2024-03-04T09:28:01.427866+0000","last_peered":"2024-03-04T09:28:01.427866+0000","last_clean":"2024-03-04T09:28:01.427866+0000","last_became_active":"2024-03-04T09:28:00.429411+0000","last_became_peered":"2024-03-04T09:28:00.429411+0000","last_unstale":"2024-03-04T09:28:01.427866+0000","last_undegraded":"2024-03-04T09:28:01.427866+0000","last_fullsized":"2024-03-04T09:28:01.427866+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:15:40.683978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.690180+0000","last_change":"2024-03-04T09:27:59.405815+0000","last_active":"2024-03-04T09:28:01.690180+0000","last_peered":"2024-03-04T09:28:01.690180+0000","last_clean":"2024-03-04T09:28:01.690180+0000","last_became_active":"2024-03-04T09:27:59.404028+0000","last_became_peered":"2024-03-04T09:27:59.404028+0000","last_unstale":"2024-03-04T09:28:01.690180+0000","last_undegraded":"2024-03-04T09:28:01.690180+0000","last_fullsized":"2024-03-04T09:28:01.690180+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_clean_scrub_stamp":"2024-03-04T09:27:58.353037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:52:31.197392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.692299+0000","last_change":"2024-03-04T09:28:00.435342+0000","last_active":"2024-03-04T09:28:01.692299+0000","last_peered":"2024-03-04T09:28:01.692299+0000","last_clean":"2024-03-04T09:28:01.692299+0000","last_became_active":"2024-03-04T09:28:00.435039+0000","last_became_peered":"2024-03-04T09:28:00.435039+0000","last_unstale":"2024-03-04T09:28:01.692299+0000","last_undegraded":"2024-03-04T09:28:01.692299+0000","last_fullsized":"2024-03-04T09:28:01.692299+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:11:51.717176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.690109+0000","last_change":"2024-03-04T09:27:59.420694+0000","last_active":"2024-03-04T09:28:01.690109+0000","last_peered":"2024-03-04T09:28:01.690109+0000","last_clean":"2024-03-04T09:28:01.690109+0000","last_became_active":"2024-03-04T09:27:59.420432+0000","last_became_peered":"2024-03-04T09:27:59.420432+0000","last_unstale":"2024-03-04T09:28:01.690109+0000","last_undegraded":"2024-03-04T09:28:01.690109+0000","last_fullsized":"2024-03-04T09:28:01.690109+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_clean_scrub_stamp":"2024-03-04T09:27:58.353037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:41:19.370977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.691324+0000","last_change":"2024-03-04T09:28:00.433093+0000","last_active":"2024-03-04T09:28:01.691324+0000","last_peered":"2024-03-04T09:28:01.691324+0000","last_clean":"2024-03-04T09:28:01.691324+0000","last_became_active":"2024-03-04T09:28:00.432809+0000","last_became_peered":"2024-03-04T09:28:00.432809+0000","last_unstale":"2024-03-04T09:28:01.691324+0000","last_undegraded":"2024-03-04T09:28:01.691324+0000","last_fullsized":"2024-03-04T09:28:01.691324+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:01:23.257447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.692857+0000","last_change":"2024-03-04T09:27:59.412972+0000","last_active":"2024-03-04T09:28:01.692857+0000","last_peered":"2024-03-04T09:28:01.692857+0000","last_clean":"2024-03-04T09:28:01.692857+0000","last_became_active":"2024-03-04T09:27:59.411051+0000","last_became_peered":"2024-03-04T09:27:59.411051+0000","last_unstale":"2024-03-04T09:28:01.692857+0000","last_undegraded":"2024-03-04T09:28:01.692857+0000","last_fullsized":"2024-03-04T09:28:01.692857+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_clean_scrub_stamp":"2024-03-04T09:27:58.353037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:18:59.307799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.692665+0000","last_change":"2024-03-04T09:28:00.425075+0000","last_active":"2024-03-04T09:28:01.692665+0000","last_peered":"2024-03-04T09:28:01.692665+0000","last_clean":"2024-03-04T09:28:01.692665+0000","last_became_active":"2024-03-04T09:28:00.422969+0000","last_became_peered":"2024-03-04T09:28:00.422969+0000","last_unstale":"2024-03-04T09:28:01.692665+0000","last_undegraded":"2024-03-04T09:28:01.692665+0000","last_fullsized":"2024-03-04T09:28:01.692665+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:42:00.605123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.692065+0000","last_change":"2024-03-04T09:27:59.417865+0000","last_active":"2024-03-04T09:28:01.692065+0000","last_peered":"2024-03-04T09:28:01.692065+0000","last_clean":"2024-03-04T09:28:01.692065+0000","last_became_active":"2024-03-04T09:27:59.417551+0000","last_became_peered":"2024-03-04T09:27:59.417551+0000","last_unstale":"2024-03-04T09:28:01.692065+0000","last_undegraded":"2024-03-04T09:28:01.692065+0000","last_fullsized":"2024-03-04T09:28:01.692065+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_clean_scrub_stamp":"2024-03-04T09:27:58.353037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:18:18.595855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.425244+0000","last_change":"2024-03-04T09:28:00.432532+0000","last_active":"2024-03-04T09:28:01.425244+0000","last_peered":"2024-03-04T09:28:01.425244+0000","last_clean":"2024-03-04T09:28:01.425244+0000","last_became_active":"2024-03-04T09:28:00.432146+0000","last_became_peered":"2024-03-04T09:28:00.432146+0000","last_unstale":"2024-03-04T09:28:01.425244+0000","last_undegraded":"2024-03-04T09:28:01.425244+0000","last_fullsized":"2024-03-04T09:28:01.425244+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:27:10.574104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"55'3","reported_seq":17,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.693092+0000","last_change":"2024-03-04T09:27:59.412981+0000","last_active":"2024-03-04T09:28:01.693092+0000","last_peered":"2024-03-04T09:28:01.693092+0000","last_clean":"2024-03-04T09:28:01.693092+0000","last_became_active":"2024-03-04T09:27:59.412560+0000","last_became_peered":"2024-03-04T09:27:59.412560+0000","last_unstale":"2024-03-04T09:28:01.693092+0000","last_undegraded":"2024-03-04T09:28:01.693092+0000","last_fullsized":"2024-03-04T09:28:01.693092+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_clean_scrub_stamp":"2024-03-04T09:27:58.353037+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:52:31.197392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.692594+0000","last_change":"2024-03-04T09:28:00.425119+0000","last_active":"2024-03-04T09:28:01.692594+0000","last_peered":"2024-03-04T09:28:01.692594+0000","last_clean":"2024-03-04T09:28:01.692594+0000","last_became_active":"2024-03-04T09:28:00.421652+0000","last_became_peered":"2024-03-04T09:28:00.421652+0000","last_unstale":"2024-03-04T09:28:01.692594+0000","last_undegraded":"2024-03-04T09:28:01.692594+0000","last_fullsized":"2024-03-04T09:28:01.692594+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:38:44.223188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"55'6","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.690885+0000","last_change":"2024-03-04T09:27:59.410959+0000","last_active":"2024-03-04T09:28:01.690885+0000","last_peered":"2024-03-04T09:28:01.690885+0000","last_clean":"2024-03-04T09:28:01.690885+0000","last_became_active":"2024-03-04T09:27:59.410593+0000","last_became_peered":"2024-03-04T09:27:59.410593+0000","last_unstale":"2024-03-04T09:28:01.690885+0000","last_undegraded":"2024-03-04T09:28:01.690885+0000","last_fullsized":"2024-03-04T09:28:01.690885+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_clean_scrub_stamp":"2024-03-04T09:27:58.353037+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:54:18.916314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.689942+0000","last_change":"2024-03-04T09:28:00.442151+0000","last_active":"2024-03-04T09:28:01.689942+0000","last_peered":"2024-03-04T09:28:01.689942+0000","last_clean":"2024-03-04T09:28:01.689942+0000","last_became_active":"2024-03-04T09:28:00.434929+0000","last_became_peered":"2024-03-04T09:28:00.434929+0000","last_unstale":"2024-03-04T09:28:01.689942+0000","last_undegraded":"2024-03-04T09:28:01.689942+0000","last_fullsized":"2024-03-04T09:28:01.689942+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:30:38.292668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"55'2","reported_seq":16,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.691380+0000","last_change":"2024-03-04T09:27:59.421713+0000","last_active":"2024-03-04T09:28:01.691380+0000","last_peered":"2024-03-04T09:28:01.691380+0000","last_clean":"2024-03-04T09:28:01.691380+0000","last_became_active":"2024-03-04T09:27:59.421381+0000","last_became_peered":"2024-03-04T09:27:59.421381+0000","last_unstale":"2024-03-04T09:28:01.691380+0000","last_undegraded":"2024-03-04T09:28:01.691380+0000","last_fullsized":"2024-03-04T09:28:01.691380+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_clean_scrub_stamp":"2024-03-04T09:27:58.353037+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:30:37.278801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.692163+0000","last_change":"2024-03-04T09:28:00.424233+0000","last_active":"2024-03-04T09:28:01.692163+0000","last_peered":"2024-03-04T09:28:01.692163+0000","last_clean":"2024-03-04T09:28:01.692163+0000","last_became_active":"2024-03-04T09:28:00.423612+0000","last_became_peered":"2024-03-04T09:28:00.423612+0000","last_unstale":"2024-03-04T09:28:01.692163+0000","last_undegraded":"2024-03-04T09:28:01.692163+0000","last_fullsized":"2024-03-04T09:28:01.692163+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:30:38.292668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.427495+0000","last_change":"2024-03-04T09:27:59.410074+0000","last_active":"2024-03-04T09:28:01.427495+0000","last_peered":"2024-03-04T09:28:01.427495+0000","last_clean":"2024-03-04T09:28:01.427495+0000","last_became_active":"2024-03-04T09:27:59.409636+0000","last_became_peered":"2024-03-04T09:27:59.409636+0000","last_unstale":"2024-03-04T09:28:01.427495+0000","last_undegraded":"2024-03-04T09:28:01.427495+0000","last_fullsized":"2024-03-04T09:28:01.427495+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_clean_scrub_stamp":"2024-03-04T09:27:58.353037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:53:26.740726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.689879+0000","last_change":"2024-03-04T09:28:00.436300+0000","last_active":"2024-03-04T09:28:01.689879+0000","last_peered":"2024-03-04T09:28:01.689879+0000","last_clean":"2024-03-04T09:28:01.689879+0000","last_became_active":"2024-03-04T09:28:00.436005+0000","last_became_peered":"2024-03-04T09:28:00.436005+0000","last_unstale":"2024-03-04T09:28:01.689879+0000","last_undegraded":"2024-03-04T09:28:01.689879+0000","last_fullsized":"2024-03-04T09:28:01.689879+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:28:55.761203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.690386+0000","last_change":"2024-03-04T09:27:59.409146+0000","last_active":"2024-03-04T09:28:01.690386+0000","last_peered":"2024-03-04T09:28:01.690386+0000","last_clean":"2024-03-04T09:28:01.690386+0000","last_became_active":"2024-03-04T09:27:59.408651+0000","last_became_peered":"2024-03-04T09:27:59.408651+0000","last_unstale":"2024-03-04T09:28:01.690386+0000","last_undegraded":"2024-03-04T09:28:01.690386+0000","last_fullsized":"2024-03-04T09:28:01.690386+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_clean_scrub_stamp":"2024-03-04T09:27:58.353037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:28:54.747336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.693468+0000","last_change":"2024-03-04T09:28:00.432457+0000","last_active":"2024-03-04T09:28:01.693468+0000","last_peered":"2024-03-04T09:28:01.693468+0000","last_clean":"2024-03-04T09:28:01.693468+0000","last_became_active":"2024-03-04T09:28:00.430610+0000","last_became_peered":"2024-03-04T09:28:00.430610+0000","last_unstale":"2024-03-04T09:28:01.693468+0000","last_undegraded":"2024-03-04T09:28:01.693468+0000","last_fullsized":"2024-03-04T09:28:01.693468+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:43:51.380199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"55'8","reported_seq":22,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.691398+0000","last_change":"2024-03-04T09:27:59.416172+0000","last_active":"2024-03-04T09:28:01.691398+0000","last_peered":"2024-03-04T09:28:01.691398+0000","last_clean":"2024-03-04T09:28:01.691398+0000","last_became_active":"2024-03-04T09:27:59.415946+0000","last_became_peered":"2024-03-04T09:27:59.415946+0000","last_unstale":"2024-03-04T09:28:01.691398+0000","last_undegraded":"2024-03-04T09:28:01.691398+0000","last_fullsized":"2024-03-04T09:28:01.691398+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_clean_scrub_stamp":"2024-03-04T09:27:58.353037+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:34:09.176083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.692166+0000","last_change":"2024-03-04T09:28:00.430963+0000","last_active":"2024-03-04T09:28:01.692166+0000","last_peered":"2024-03-04T09:28:01.692166+0000","last_clean":"2024-03-04T09:28:01.692166+0000","last_became_active":"2024-03-04T09:28:00.430468+0000","last_became_peered":"2024-03-04T09:28:00.430468+0000","last_unstale":"2024-03-04T09:28:01.692166+0000","last_undegraded":"2024-03-04T09:28:01.692166+0000","last_fullsized":"2024-03-04T09:28:01.692166+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:41:49.927616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"55'1","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.692446+0000","last_change":"2024-03-04T09:27:59.421957+0000","last_active":"2024-03-04T09:28:01.692446+0000","last_peered":"2024-03-04T09:28:01.692446+0000","last_clean":"2024-03-04T09:28:01.692446+0000","last_became_active":"2024-03-04T09:27:59.421523+0000","last_became_peered":"2024-03-04T09:27:59.421523+0000","last_unstale":"2024-03-04T09:28:01.692446+0000","last_undegraded":"2024-03-04T09:28:01.692446+0000","last_fullsized":"2024-03-04T09:28:01.692446+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_clean_scrub_stamp":"2024-03-04T09:27:58.353037+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:38:43.209321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.693292+0000","last_change":"2024-03-04T09:28:00.444341+0000","last_active":"2024-03-04T09:28:01.693292+0000","last_peered":"2024-03-04T09:28:01.693292+0000","last_clean":"2024-03-04T09:28:01.693292+0000","last_became_active":"2024-03-04T09:28:00.443965+0000","last_became_peered":"2024-03-04T09:28:00.443965+0000","last_unstale":"2024-03-04T09:28:01.693292+0000","last_undegraded":"2024-03-04T09:28:01.693292+0000","last_fullsized":"2024-03-04T09:28:01.693292+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:17:30.816921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"55'2","reported_seq":16,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.430112+0000","last_change":"2024-03-04T09:27:59.407941+0000","last_active":"2024-03-04T09:28:01.430112+0000","last_peered":"2024-03-04T09:28:01.430112+0000","last_clean":"2024-03-04T09:28:01.430112+0000","last_became_active":"2024-03-04T09:27:59.406809+0000","last_became_peered":"2024-03-04T09:27:59.406809+0000","last_unstale":"2024-03-04T09:28:01.430112+0000","last_undegraded":"2024-03-04T09:28:01.430112+0000","last_fullsized":"2024-03-04T09:28:01.430112+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_clean_scrub_stamp":"2024-03-04T09:27:58.353037+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:14:29.832957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.691859+0000","last_change":"2024-03-04T09:28:00.436859+0000","last_active":"2024-03-04T09:28:01.691859+0000","last_peered":"2024-03-04T09:28:01.691859+0000","last_clean":"2024-03-04T09:28:01.691859+0000","last_became_active":"2024-03-04T09:28:00.436021+0000","last_became_peered":"2024-03-04T09:28:00.436021+0000","last_unstale":"2024-03-04T09:28:01.691859+0000","last_undegraded":"2024-03-04T09:28:01.691859+0000","last_fullsized":"2024-03-04T09:28:01.691859+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:22:41.561201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"55'1","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.691838+0000","last_change":"2024-03-04T09:27:59.409078+0000","last_active":"2024-03-04T09:28:01.691838+0000","last_peered":"2024-03-04T09:28:01.691838+0000","last_clean":"2024-03-04T09:28:01.691838+0000","last_became_active":"2024-03-04T09:27:59.408701+0000","last_became_peered":"2024-03-04T09:27:59.408701+0000","last_unstale":"2024-03-04T09:28:01.691838+0000","last_undegraded":"2024-03-04T09:28:01.691838+0000","last_fullsized":"2024-03-04T09:28:01.691838+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_clean_scrub_stamp":"2024-03-04T09:27:58.353037+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:01:22.243580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.693019+0000","last_change":"2024-03-04T09:28:00.420941+0000","last_active":"2024-03-04T09:28:01.693019+0000","last_peered":"2024-03-04T09:28:01.693019+0000","last_clean":"2024-03-04T09:28:01.693019+0000","last_became_active":"2024-03-04T09:28:00.420549+0000","last_became_peered":"2024-03-04T09:28:00.420549+0000","last_unstale":"2024-03-04T09:28:01.693019+0000","last_undegraded":"2024-03-04T09:28:01.693019+0000","last_fullsized":"2024-03-04T09:28:01.693019+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:29:14.123293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.429361+0000","last_change":"2024-03-04T09:27:59.424879+0000","last_active":"2024-03-04T09:28:01.429361+0000","last_peered":"2024-03-04T09:28:01.429361+0000","last_clean":"2024-03-04T09:28:01.429361+0000","last_became_active":"2024-03-04T09:27:59.424617+0000","last_became_peered":"2024-03-04T09:27:59.424617+0000","last_unstale":"2024-03-04T09:28:01.429361+0000","last_undegraded":"2024-03-04T09:28:01.429361+0000","last_fullsized":"2024-03-04T09:28:01.429361+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_clean_scrub_stamp":"2024-03-04T09:27:58.353037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:28:54.747336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.690255+0000","last_change":"2024-03-04T09:28:00.444454+0000","last_active":"2024-03-04T09:28:01.690255+0000","last_peered":"2024-03-04T09:28:01.690255+0000","last_clean":"2024-03-04T09:28:01.690255+0000","last_became_active":"2024-03-04T09:28:00.438200+0000","last_became_peered":"2024-03-04T09:28:00.438200+0000","last_unstale":"2024-03-04T09:28:01.690255+0000","last_undegraded":"2024-03-04T09:28:01.690255+0000","last_fullsized":"2024-03-04T09:28:01.690255+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:41:49.927616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"55'2","reported_seq":16,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.691690+0000","last_change":"2024-03-04T09:27:59.411348+0000","last_active":"2024-03-04T09:28:01.691690+0000","last_peered":"2024-03-04T09:28:01.691690+0000","last_clean":"2024-03-04T09:28:01.691690+0000","last_became_active":"2024-03-04T09:27:59.410984+0000","last_became_peered":"2024-03-04T09:27:59.410984+0000","last_unstale":"2024-03-04T09:28:01.691690+0000","last_undegraded":"2024-03-04T09:28:01.691690+0000","last_fullsized":"2024-03-04T09:28:01.691690+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_clean_scrub_stamp":"2024-03-04T09:27:58.353037+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:41:48.913749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.429741+0000","last_change":"2024-03-04T09:28:00.434771+0000","last_active":"2024-03-04T09:28:01.429741+0000","last_peered":"2024-03-04T09:28:01.429741+0000","last_clean":"2024-03-04T09:28:01.429741+0000","last_became_active":"2024-03-04T09:28:00.434457+0000","last_became_peered":"2024-03-04T09:28:00.434457+0000","last_unstale":"2024-03-04T09:28:01.429741+0000","last_undegraded":"2024-03-04T09:28:01.429741+0000","last_fullsized":"2024-03-04T09:28:01.429741+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:55:21.623042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.425928+0000","last_change":"2024-03-04T09:27:59.402170+0000","last_active":"2024-03-04T09:28:01.425928+0000","last_peered":"2024-03-04T09:28:01.425928+0000","last_clean":"2024-03-04T09:28:01.425928+0000","last_became_active":"2024-03-04T09:27:59.401808+0000","last_became_peered":"2024-03-04T09:27:59.401808+0000","last_unstale":"2024-03-04T09:28:01.425928+0000","last_undegraded":"2024-03-04T09:28:01.425928+0000","last_fullsized":"2024-03-04T09:28:01.425928+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_clean_scrub_stamp":"2024-03-04T09:27:58.353037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:18:30.896319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.428632+0000","last_change":"2024-03-04T09:28:00.435603+0000","last_active":"2024-03-04T09:28:01.428632+0000","last_peered":"2024-03-04T09:28:01.428632+0000","last_clean":"2024-03-04T09:28:01.428632+0000","last_became_active":"2024-03-04T09:28:00.435296+0000","last_became_peered":"2024-03-04T09:28:00.435296+0000","last_unstale":"2024-03-04T09:28:01.428632+0000","last_undegraded":"2024-03-04T09:28:01.428632+0000","last_fullsized":"2024-03-04T09:28:01.428632+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:38:44.223188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.689405+0000","last_change":"2024-03-04T09:27:59.408243+0000","last_active":"2024-03-04T09:28:01.689405+0000","last_peered":"2024-03-04T09:28:01.689405+0000","last_clean":"2024-03-04T09:28:01.689405+0000","last_became_active":"2024-03-04T09:27:59.405122+0000","last_became_peered":"2024-03-04T09:27:59.405122+0000","last_unstale":"2024-03-04T09:28:01.689405+0000","last_undegraded":"2024-03-04T09:28:01.689405+0000","last_fullsized":"2024-03-04T09:28:01.689405+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_clean_scrub_stamp":"2024-03-04T09:27:58.353037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:18:59.307799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.692453+0000","last_change":"2024-03-04T09:28:00.439522+0000","last_active":"2024-03-04T09:28:01.692453+0000","last_peered":"2024-03-04T09:28:01.692453+0000","last_clean":"2024-03-04T09:28:01.692453+0000","last_became_active":"2024-03-04T09:28:00.439003+0000","last_became_peered":"2024-03-04T09:28:00.439003+0000","last_unstale":"2024-03-04T09:28:01.692453+0000","last_undegraded":"2024-03-04T09:28:01.692453+0000","last_fullsized":"2024-03-04T09:28:01.692453+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:42:00.605123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"55'2","reported_seq":16,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.426020+0000","last_change":"2024-03-04T09:27:59.411808+0000","last_active":"2024-03-04T09:28:01.426020+0000","last_peered":"2024-03-04T09:28:01.426020+0000","last_clean":"2024-03-04T09:28:01.426020+0000","last_became_active":"2024-03-04T09:27:59.411311+0000","last_became_peered":"2024-03-04T09:27:59.411311+0000","last_unstale":"2024-03-04T09:28:01.426020+0000","last_undegraded":"2024-03-04T09:28:01.426020+0000","last_fullsized":"2024-03-04T09:28:01.426020+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_clean_scrub_stamp":"2024-03-04T09:27:58.353037+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:59:27.874462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.693581+0000","last_change":"2024-03-04T09:28:00.445347+0000","last_active":"2024-03-04T09:28:01.693581+0000","last_peered":"2024-03-04T09:28:01.693581+0000","last_clean":"2024-03-04T09:28:01.693581+0000","last_became_active":"2024-03-04T09:28:00.444521+0000","last_became_peered":"2024-03-04T09:28:00.444521+0000","last_unstale":"2024-03-04T09:28:01.693581+0000","last_undegraded":"2024-03-04T09:28:01.693581+0000","last_fullsized":"2024-03-04T09:28:01.693581+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:29:14.123293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"56'1","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.442419+0000","last_change":"2024-03-04T09:27:59.409598+0000","last_active":"2024-03-04T09:28:01.442419+0000","last_peered":"2024-03-04T09:28:01.442419+0000","last_clean":"2024-03-04T09:28:01.442419+0000","last_became_active":"2024-03-04T09:27:59.409330+0000","last_became_peered":"2024-03-04T09:27:59.409330+0000","last_unstale":"2024-03-04T09:28:01.442419+0000","last_undegraded":"2024-03-04T09:28:01.442419+0000","last_fullsized":"2024-03-04T09:28:01.442419+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_clean_scrub_stamp":"2024-03-04T09:27:58.353037+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:01:22.243580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.425623+0000","last_change":"2024-03-04T09:28:00.435631+0000","last_active":"2024-03-04T09:28:01.425623+0000","last_peered":"2024-03-04T09:28:01.425623+0000","last_clean":"2024-03-04T09:28:01.425623+0000","last_became_active":"2024-03-04T09:28:00.435201+0000","last_became_peered":"2024-03-04T09:28:00.435201+0000","last_unstale":"2024-03-04T09:28:01.425623+0000","last_undegraded":"2024-03-04T09:28:01.425623+0000","last_fullsized":"2024-03-04T09:28:01.425623+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:18:29.267029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.427659+0000","last_change":"2024-03-04T09:27:59.412187+0000","last_active":"2024-03-04T09:28:01.427659+0000","last_peered":"2024-03-04T09:28:01.427659+0000","last_clean":"2024-03-04T09:28:01.427659+0000","last_became_active":"2024-03-04T09:27:59.411905+0000","last_became_peered":"2024-03-04T09:27:59.411905+0000","last_unstale":"2024-03-04T09:28:01.427659+0000","last_undegraded":"2024-03-04T09:28:01.427659+0000","last_fullsized":"2024-03-04T09:28:01.427659+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_clean_scrub_stamp":"2024-03-04T09:27:58.353037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:01:05.085170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.430224+0000","last_change":"2024-03-04T09:28:00.448429+0000","last_active":"2024-03-04T09:28:01.430224+0000","last_peered":"2024-03-04T09:28:01.430224+0000","last_clean":"2024-03-04T09:28:01.430224+0000","last_became_active":"2024-03-04T09:28:00.446151+0000","last_became_peered":"2024-03-04T09:28:00.446151+0000","last_unstale":"2024-03-04T09:28:01.430224+0000","last_undegraded":"2024-03-04T09:28:01.430224+0000","last_fullsized":"2024-03-04T09:28:01.430224+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:00:41.718192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.692652+0000","last_change":"2024-03-04T09:27:59.410066+0000","last_active":"2024-03-04T09:28:01.692652+0000","last_peered":"2024-03-04T09:28:01.692652+0000","last_clean":"2024-03-04T09:28:01.692652+0000","last_became_active":"2024-03-04T09:27:59.409164+0000","last_became_peered":"2024-03-04T09:27:59.409164+0000","last_unstale":"2024-03-04T09:28:01.692652+0000","last_undegraded":"2024-03-04T09:28:01.692652+0000","last_fullsized":"2024-03-04T09:28:01.692652+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_clean_scrub_stamp":"2024-03-04T09:27:58.353037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:14:29.832957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.691555+0000","last_change":"2024-03-04T09:28:00.445983+0000","last_active":"2024-03-04T09:28:01.691555+0000","last_peered":"2024-03-04T09:28:01.691555+0000","last_clean":"2024-03-04T09:28:01.691555+0000","last_became_active":"2024-03-04T09:28:00.442214+0000","last_became_peered":"2024-03-04T09:28:00.442214+0000","last_unstale":"2024-03-04T09:28:01.691555+0000","last_undegraded":"2024-03-04T09:28:01.691555+0000","last_fullsized":"2024-03-04T09:28:01.691555+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:34:17.456837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.425285+0000","last_change":"2024-03-04T09:27:59.394255+0000","last_active":"2024-03-04T09:28:01.425285+0000","last_peered":"2024-03-04T09:28:01.425285+0000","last_clean":"2024-03-04T09:28:01.425285+0000","last_became_active":"2024-03-04T09:27:59.393926+0000","last_became_peered":"2024-03-04T09:27:59.393926+0000","last_unstale":"2024-03-04T09:28:01.425285+0000","last_undegraded":"2024-03-04T09:28:01.425285+0000","last_fullsized":"2024-03-04T09:28:01.425285+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_clean_scrub_stamp":"2024-03-04T09:27:58.353037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:01:05.085170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.692554+0000","last_change":"2024-03-04T09:28:00.435846+0000","last_active":"2024-03-04T09:28:01.692554+0000","last_peered":"2024-03-04T09:28:01.692554+0000","last_clean":"2024-03-04T09:28:01.692554+0000","last_became_active":"2024-03-04T09:28:00.435579+0000","last_became_peered":"2024-03-04T09:28:00.435579+0000","last_unstale":"2024-03-04T09:28:01.692554+0000","last_undegraded":"2024-03-04T09:28:01.692554+0000","last_fullsized":"2024-03-04T09:28:01.692554+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:18:19.609722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.691023+0000","last_change":"2024-03-04T09:27:59.410882+0000","last_active":"2024-03-04T09:28:01.691023+0000","last_peered":"2024-03-04T09:28:01.691023+0000","last_clean":"2024-03-04T09:28:01.691023+0000","last_became_active":"2024-03-04T09:27:59.409671+0000","last_became_peered":"2024-03-04T09:27:59.409671+0000","last_unstale":"2024-03-04T09:28:01.691023+0000","last_undegraded":"2024-03-04T09:28:01.691023+0000","last_fullsized":"2024-03-04T09:28:01.691023+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_clean_scrub_stamp":"2024-03-04T09:27:58.353037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:45:55.659886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.425720+0000","last_change":"2024-03-04T09:28:00.429447+0000","last_active":"2024-03-04T09:28:01.425720+0000","last_peered":"2024-03-04T09:28:01.425720+0000","last_clean":"2024-03-04T09:28:01.425720+0000","last_became_active":"2024-03-04T09:28:00.429183+0000","last_became_peered":"2024-03-04T09:28:00.429183+0000","last_unstale":"2024-03-04T09:28:01.425720+0000","last_undegraded":"2024-03-04T09:28:01.425720+0000","last_fullsized":"2024-03-04T09:28:01.425720+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:54:19.930181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.425125+0000","last_change":"2024-03-04T09:27:59.409340+0000","last_active":"2024-03-04T09:28:01.425125+0000","last_peered":"2024-03-04T09:28:01.425125+0000","last_clean":"2024-03-04T09:28:01.425125+0000","last_became_active":"2024-03-04T09:27:59.408288+0000","last_became_peered":"2024-03-04T09:27:59.408288+0000","last_unstale":"2024-03-04T09:28:01.425125+0000","last_undegraded":"2024-03-04T09:28:01.425125+0000","last_fullsized":"2024-03-04T09:28:01.425125+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_clean_scrub_stamp":"2024-03-04T09:27:58.353037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:53:26.740726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.690947+0000","last_change":"2024-03-04T09:28:00.429777+0000","last_active":"2024-03-04T09:28:01.690947+0000","last_peered":"2024-03-04T09:28:01.690947+0000","last_clean":"2024-03-04T09:28:01.690947+0000","last_became_active":"2024-03-04T09:28:00.429506+0000","last_became_peered":"2024-03-04T09:28:00.429506+0000","last_unstale":"2024-03-04T09:28:01.690947+0000","last_undegraded":"2024-03-04T09:28:01.690947+0000","last_fullsized":"2024-03-04T09:28:01.690947+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:45:23.923649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.690555+0000","last_change":"2024-03-04T09:27:59.404368+0000","last_active":"2024-03-04T09:28:01.690555+0000","last_peered":"2024-03-04T09:28:01.690555+0000","last_clean":"2024-03-04T09:28:01.690555+0000","last_became_active":"2024-03-04T09:27:59.403755+0000","last_became_peered":"2024-03-04T09:27:59.403755+0000","last_unstale":"2024-03-04T09:28:01.690555+0000","last_undegraded":"2024-03-04T09:28:01.690555+0000","last_fullsized":"2024-03-04T09:28:01.690555+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_clean_scrub_stamp":"2024-03-04T09:27:58.353037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:17:08.509012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.690312+0000","last_change":"2024-03-04T09:28:00.442817+0000","last_active":"2024-03-04T09:28:01.690312+0000","last_peered":"2024-03-04T09:28:01.690312+0000","last_clean":"2024-03-04T09:28:01.690312+0000","last_became_active":"2024-03-04T09:28:00.442265+0000","last_became_peered":"2024-03-04T09:28:00.442265+0000","last_unstale":"2024-03-04T09:28:01.690312+0000","last_undegraded":"2024-03-04T09:28:01.690312+0000","last_fullsized":"2024-03-04T09:28:01.690312+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:52:07.773906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.427319+0000","last_change":"2024-03-04T09:27:59.414003+0000","last_active":"2024-03-04T09:28:01.427319+0000","last_peered":"2024-03-04T09:28:01.427319+0000","last_clean":"2024-03-04T09:28:01.427319+0000","last_became_active":"2024-03-04T09:27:59.412706+0000","last_became_peered":"2024-03-04T09:27:59.412706+0000","last_unstale":"2024-03-04T09:28:01.427319+0000","last_undegraded":"2024-03-04T09:28:01.427319+0000","last_fullsized":"2024-03-04T09:28:01.427319+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_clean_scrub_stamp":"2024-03-04T09:27:58.353037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:29:15.878468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":237,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.691303+0000","last_change":"2024-03-04T09:25:09.316159+0000","last_active":"2024-03-04T09:28:01.691303+0000","last_peered":"2024-03-04T09:28:01.691303+0000","last_clean":"2024-03-04T09:28:01.691303+0000","last_became_active":"2024-03-04T09:25:09.313767+0000","last_became_peered":"2024-03-04T09:25:09.313767+0000","last_unstale":"2024-03-04T09:28:01.691303+0000","last_undegraded":"2024-03-04T09:28:01.691303+0000","last_fullsized":"2024-03-04T09:28:01.691303+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:08.110953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:08.110953+0000","last_clean_scrub_stamp":"2024-03-04T09:25:08.110953+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:34:21.745390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574886,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32432,"kb_used_data":1244,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339408,"statfs":{"total":96636764160,"available":96603553792,"internally_reserved":0,"allocated":1273856,"data_stored":869083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574886,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31840,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340000,"statfs":{"total":96636764160,"available":96604160000,"internally_reserved":0,"allocated":671744,"data_stored":278555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574888,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31852,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339988,"statfs":{"total":96636764160,"available":96604147712,"internally_reserved":0,"allocated":679936,"data_stored":279207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574885,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31852,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339988,"statfs":{"total":96636764160,"available":96604147712,"internally_reserved":0,"allocated":684032,"data_stored":280167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574887,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31852,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339988,"statfs":{"total":96636764160,"available":96604147712,"internally_reserved":0,"allocated":679936,"data_stored":278611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574886,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31848,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339992,"statfs":{"total":96636764160,"available":96604151808,"internally_reserved":0,"allocated":679936,"data_stored":278611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574887,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31844,"kb_used_data":660,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339996,"statfs":{"total":96636764160,"available":96604155904,"internally_reserved":0,"allocated":675840,"data_stored":278645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574886,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32440,"kb_used_data":1252,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339400,"statfs":{"total":96636764160,"available":96603545600,"internally_reserved":0,"allocated":1282048,"data_stored":871117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-04T09:28:04.908 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T09:28:05.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.172+0000 7f0955c00640 1 Processor -- start 2024-03-04T09:28:05.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.172+0000 7f0955c00640 1 -- start start 2024-03-04T09:28:05.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.174+0000 7f0955c00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f09500af580 0x7f095015c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:05.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.174+0000 7f0955c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f09500cdc20 0x7f09500b2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:05.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.174+0000 7f0955c00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f095015cc40 con 0x7f09500af1a0 2024-03-04T09:28:05.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.174+0000 7f0955c00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f095015cf20 con 0x7f09500cdc20 2024-03-04T09:28:05.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.174+0000 7f0955c00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f095015d200 con 0x7f09500af580 2024-03-04T09:28:05.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.175+0000 7f094f400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f09500cdc20 0x7f09500b2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:05.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.175+0000 7f094f400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f09500cdc20 0x7f09500b2dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45348/0 (socket says 172.21.15.6:45348) 2024-03-04T09:28:05.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.175+0000 7f094f400640 1 -- 172.21.15.6:0/2273510298 learned_addr learned my addr 172.21.15.6:0/2273510298 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:28:05.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.175+0000 7f0947e00640 1 --2- 172.21.15.6:0/2273510298 >> v2:172.21.15.146:3300/0 conn(0x7f09500af580 0x7f095015c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:05.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.175+0000 7f094f400640 1 -- 172.21.15.6:0/2273510298 >> v2:172.21.15.146:3300/0 conn(0x7f09500af580 msgr2=0x7f095015c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:05.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.175+0000 7f094f400640 1 --2- 172.21.15.6:0/2273510298 >> v2:172.21.15.146:3300/0 conn(0x7f09500af580 0x7f095015c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:05.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.175+0000 7f094f400640 1 -- 172.21.15.6:0/2273510298 >> v1:172.21.15.146:6789/0 conn(0x7f09500af1a0 legacy=0x7f09500b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:05.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.175+0000 7f094f400640 1 -- 172.21.15.6:0/2273510298 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f095015d4e0 con 0x7f09500cdc20 2024-03-04T09:28:05.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.176+0000 7f094f400640 1 --2- 172.21.15.6:0/2273510298 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f09500cdc20 0x7f09500b2dd0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f09500b54c0 tx=0x7f094802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dda959503bab76a1 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:05.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.177+0000 7f094e000640 1 -- 172.21.15.6:0/2273510298 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f094802ca80 con 0x7f09500cdc20 2024-03-04T09:28:05.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.177+0000 7f094e000640 1 -- 172.21.15.6:0/2273510298 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f094802cc20 con 0x7f09500cdc20 2024-03-04T09:28:05.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.177+0000 7f094e000640 1 -- 172.21.15.6:0/2273510298 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09480343f0 con 0x7f09500cdc20 2024-03-04T09:28:05.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.177+0000 7f0955c00640 1 -- 172.21.15.6:0/2273510298 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f09500cdc20 msgr2=0x7f09500b2dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:05.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.177+0000 7f0955c00640 1 --2- 172.21.15.6:0/2273510298 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f09500cdc20 0x7f09500b2dd0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f09500b54c0 tx=0x7f094802bee0 comp rx=0 tx=0).stop 2024-03-04T09:28:05.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.177+0000 7f0955c00640 1 -- 172.21.15.6:0/2273510298 shutdown_connections 2024-03-04T09:28:05.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.177+0000 7f0955c00640 1 --2- 172.21.15.6:0/2273510298 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f09500cdc20 0x7f09500b2dd0 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:05.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.177+0000 7f0955c00640 1 --2- 172.21.15.6:0/2273510298 >> v2:172.21.15.146:3300/0 conn(0x7f09500af580 0x7f095015c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:05.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.178+0000 7f0955c00640 1 -- 172.21.15.6:0/2273510298 >> 172.21.15.6:0/2273510298 conn(0x7f095006afb0 msgr2=0x7f095006b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:05.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.178+0000 7f0955c00640 1 -- 172.21.15.6:0/2273510298 shutdown_connections 2024-03-04T09:28:05.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.178+0000 7f0955c00640 1 -- 172.21.15.6:0/2273510298 wait complete. 2024-03-04T09:28:05.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.179+0000 7f0955c00640 1 Processor -- start 2024-03-04T09:28:05.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.180+0000 7f0955c00640 1 -- start start 2024-03-04T09:28:05.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.180+0000 7f0955c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f09500af1a0 0x7f095015d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:05.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.180+0000 7f0955c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f09500af580 0x7f09501573b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:05.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.180+0000 7f0955c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f09500cdc20 0x7f09501578f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:05.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.181+0000 7f0955c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09500b36a0 con 0x7f09500af1a0 2024-03-04T09:28:05.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.181+0000 7f0955c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09500b3520 con 0x7f09500af580 2024-03-04T09:28:05.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.181+0000 7f0955c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09500b3820 con 0x7f09500cdc20 2024-03-04T09:28:05.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.181+0000 7f094ea00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f09500af1a0 0x7f095015d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:05.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.181+0000 7f094ea00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f09500af1a0 0x7f095015d970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45354/0 (socket says 172.21.15.6:45354) 2024-03-04T09:28:05.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.181+0000 7f094ea00640 1 -- 172.21.15.6:0/1980747570 learned_addr learned my addr 172.21.15.6:0/1980747570 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:28:05.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.181+0000 7f0947e00640 1 --2- 172.21.15.6:0/1980747570 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f09500af580 0x7f09501573b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:05.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.181+0000 7f094f400640 1 --2- 172.21.15.6:0/1980747570 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f09500cdc20 0x7f09501578f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:05.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.181+0000 7f094ea00640 1 -- 172.21.15.6:0/1980747570 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f09500cdc20 msgr2=0x7f09501578f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:05.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.181+0000 7f094ea00640 1 --2- 172.21.15.6:0/1980747570 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f09500cdc20 0x7f09501578f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:05.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.181+0000 7f094ea00640 1 -- 172.21.15.6:0/1980747570 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f09500af580 msgr2=0x7f09501573b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:05.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.181+0000 7f094ea00640 1 --2- 172.21.15.6:0/1980747570 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f09500af580 0x7f09501573b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:05.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.181+0000 7f094ea00640 1 -- 172.21.15.6:0/1980747570 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0950158120 con 0x7f09500af1a0 2024-03-04T09:28:05.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.182+0000 7f094ea00640 1 --2- 172.21.15.6:0/1980747570 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f09500af1a0 0x7f095015d970 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f094000c400 tx=0x7f094000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:05.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.182+0000 7f0946a00640 1 -- 172.21.15.6:0/1980747570 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09400052d0 con 0x7f09500af1a0 2024-03-04T09:28:05.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.182+0000 7f0946a00640 1 -- 172.21.15.6:0/1980747570 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0940005490 con 0x7f09500af1a0 2024-03-04T09:28:05.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.182+0000 7f0946a00640 1 -- 172.21.15.6:0/1980747570 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0940007260 con 0x7f09500af1a0 2024-03-04T09:28:05.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.182+0000 7f0955c00640 1 -- 172.21.15.6:0/1980747570 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f095006cb10 con 0x7f09500af1a0 2024-03-04T09:28:05.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.182+0000 7f0955c00640 1 -- 172.21.15.6:0/1980747570 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f095006cdd0 con 0x7f09500af1a0 2024-03-04T09:28:05.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.184+0000 7f0955c00640 1 -- 172.21.15.6:0/1980747570 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f095004f3a0 con 0x7f09500af1a0 2024-03-04T09:28:05.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.185+0000 7f0946a00640 1 -- 172.21.15.6:0/1980747570 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0940007400 con 0x7f09500af1a0 2024-03-04T09:28:05.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.185+0000 7f0946a00640 1 --2- 172.21.15.6:0/1980747570 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f092c06b4d0 0x7f092c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:05.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.186+0000 7f0946a00640 1 -- 172.21.15.6:0/1980747570 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f09400897b0 con 0x7f09500af1a0 2024-03-04T09:28:05.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.186+0000 7f0947e00640 1 --2- 172.21.15.6:0/1980747570 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f092c06b4d0 0x7f092c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:05.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.187+0000 7f0947e00640 1 --2- 172.21.15.6:0/1980747570 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f092c06b4d0 0x7f092c06d990 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f095015e550 tx=0x7f0938008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:05.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.190+0000 7f0946a00640 1 -- 172.21.15.6:0/1980747570 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0950158120 con 0x7f09500af1a0 2024-03-04T09:28:05.336 INFO:tasks.ceph.mds.b.smithi146.stderr:2024-03-04T09:28:05.339+0000 7f779fa00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-04T09:28:05.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.385+0000 7f0955c00640 1 -- 172.21.15.6:0/1980747570 --> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f095006b5a0 con 0x7f092c06b4d0 2024-03-04T09:28:05.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.418+0000 7f0946a00640 1 -- 172.21.15.6:0/1980747570 <== mgr.4113 v2:172.21.15.146:6832/1788613334 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251954 (secure 0 0 0) 0x7f095006b5a0 con 0x7f092c06b4d0 2024-03-04T09:28:05.419 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:28:05.420 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-03-04T09:28:05.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.424+0000 7f0955c00640 1 -- 172.21.15.6:0/1980747570 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f092c06b4d0 msgr2=0x7f092c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:05.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.424+0000 7f0955c00640 1 --2- 172.21.15.6:0/1980747570 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f092c06b4d0 0x7f092c06d990 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f095015e550 tx=0x7f0938008040 comp rx=0 tx=0).stop 2024-03-04T09:28:05.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.424+0000 7f0955c00640 1 -- 172.21.15.6:0/1980747570 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f09500af1a0 msgr2=0x7f095015d970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:05.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.424+0000 7f0955c00640 1 --2- 172.21.15.6:0/1980747570 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f09500af1a0 0x7f095015d970 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f094000c400 tx=0x7f094000c8c0 comp rx=0 tx=0).stop 2024-03-04T09:28:05.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.424+0000 7f0955c00640 1 -- 172.21.15.6:0/1980747570 shutdown_connections 2024-03-04T09:28:05.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.424+0000 7f0955c00640 1 --2- 172.21.15.6:0/1980747570 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f092c06b4d0 0x7f092c06d990 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:05.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.424+0000 7f0955c00640 1 --2- 172.21.15.6:0/1980747570 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f09500cdc20 0x7f09501578f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:05.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.424+0000 7f0955c00640 1 --2- 172.21.15.6:0/1980747570 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f09500af580 0x7f09501573b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:05.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.424+0000 7f0955c00640 1 --2- 172.21.15.6:0/1980747570 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f09500af1a0 0x7f095015d970 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:05.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.424+0000 7f0955c00640 1 -- 172.21.15.6:0/1980747570 >> 172.21.15.6:0/1980747570 conn(0x7f095006afb0 msgr2=0x7f095015ab30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:05.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.424+0000 7f0955c00640 1 -- 172.21.15.6:0/1980747570 shutdown_connections 2024-03-04T09:28:05.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.425+0000 7f0955c00640 1 -- 172.21.15.6:0/1980747570 wait complete. 2024-03-04T09:28:05.440 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":137,"stamp":"2024-03-04T09:28:03.916879+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":163,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":255960,"kb_used_data":6472,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754718760,"statfs":{"total":773094113280,"available":772832010240,"internally_reserved":0,"allocated":6627328,"data_stored":3413996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":42,"apply_latency_ms":42,"commit_latency_ns":42000000,"apply_latency_ns":42000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.547349"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.690192+0000","last_change":"2024-03-04T09:28:00.434177+0000","last_active":"2024-03-04T09:28:01.690192+0000","last_peered":"2024-03-04T09:28:01.690192+0000","last_clean":"2024-03-04T09:28:01.690192+0000","last_became_active":"2024-03-04T09:28:00.433834+0000","last_became_peered":"2024-03-04T09:28:00.433834+0000","last_unstale":"2024-03-04T09:28:01.690192+0000","last_undegraded":"2024-03-04T09:28:01.690192+0000","last_fullsized":"2024-03-04T09:28:01.690192+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:00:26.863147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.691526+0000","last_change":"2024-03-04T09:28:00.436530+0000","last_active":"2024-03-04T09:28:01.691526+0000","last_peered":"2024-03-04T09:28:01.691526+0000","last_clean":"2024-03-04T09:28:01.691526+0000","last_became_active":"2024-03-04T09:28:00.436242+0000","last_became_peered":"2024-03-04T09:28:00.436242+0000","last_unstale":"2024-03-04T09:28:01.691526+0000","last_undegraded":"2024-03-04T09:28:01.691526+0000","last_fullsized":"2024-03-04T09:28:01.691526+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:05:00.926651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.692194+0000","last_change":"2024-03-04T09:28:00.430122+0000","last_active":"2024-03-04T09:28:01.692194+0000","last_peered":"2024-03-04T09:28:01.692194+0000","last_clean":"2024-03-04T09:28:01.692194+0000","last_became_active":"2024-03-04T09:28:00.429188+0000","last_became_peered":"2024-03-04T09:28:00.429188+0000","last_unstale":"2024-03-04T09:28:01.692194+0000","last_undegraded":"2024-03-04T09:28:01.692194+0000","last_fullsized":"2024-03-04T09:28:01.692194+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:05:38.576682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.692257+0000","last_change":"2024-03-04T09:28:00.439691+0000","last_active":"2024-03-04T09:28:01.692257+0000","last_peered":"2024-03-04T09:28:01.692257+0000","last_clean":"2024-03-04T09:28:01.692257+0000","last_became_active":"2024-03-04T09:28:00.439348+0000","last_became_peered":"2024-03-04T09:28:00.439348+0000","last_unstale":"2024-03-04T09:28:01.692257+0000","last_undegraded":"2024-03-04T09:28:01.692257+0000","last_fullsized":"2024-03-04T09:28:01.692257+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:18:19.609722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.689501+0000","last_change":"2024-03-04T09:28:00.429107+0000","last_active":"2024-03-04T09:28:01.689501+0000","last_peered":"2024-03-04T09:28:01.689501+0000","last_clean":"2024-03-04T09:28:01.689501+0000","last_became_active":"2024-03-04T09:28:00.428800+0000","last_became_peered":"2024-03-04T09:28:00.428800+0000","last_unstale":"2024-03-04T09:28:01.689501+0000","last_undegraded":"2024-03-04T09:28:01.689501+0000","last_fullsized":"2024-03-04T09:28:01.689501+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:52:28.113950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.428937+0000","last_change":"2024-03-04T09:28:00.439396+0000","last_active":"2024-03-04T09:28:01.428937+0000","last_peered":"2024-03-04T09:28:01.428937+0000","last_clean":"2024-03-04T09:28:01.428937+0000","last_became_active":"2024-03-04T09:28:00.439129+0000","last_became_peered":"2024-03-04T09:28:00.439129+0000","last_unstale":"2024-03-04T09:28:01.428937+0000","last_undegraded":"2024-03-04T09:28:01.428937+0000","last_fullsized":"2024-03-04T09:28:01.428937+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:42:00.605123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.692436+0000","last_change":"2024-03-04T09:28:00.440692+0000","last_active":"2024-03-04T09:28:01.692436+0000","last_peered":"2024-03-04T09:28:01.692436+0000","last_clean":"2024-03-04T09:28:01.692436+0000","last_became_active":"2024-03-04T09:28:00.438138+0000","last_became_peered":"2024-03-04T09:28:00.438138+0000","last_unstale":"2024-03-04T09:28:01.692436+0000","last_undegraded":"2024-03-04T09:28:01.692436+0000","last_fullsized":"2024-03-04T09:28:01.692436+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:24:39.308837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.690247+0000","last_change":"2024-03-04T09:28:00.438344+0000","last_active":"2024-03-04T09:28:01.690247+0000","last_peered":"2024-03-04T09:28:01.690247+0000","last_clean":"2024-03-04T09:28:01.690247+0000","last_became_active":"2024-03-04T09:28:00.438076+0000","last_became_peered":"2024-03-04T09:28:00.438076+0000","last_unstale":"2024-03-04T09:28:01.690247+0000","last_undegraded":"2024-03-04T09:28:01.690247+0000","last_fullsized":"2024-03-04T09:28:01.690247+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:39:24.789268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.690928+0000","last_change":"2024-03-04T09:28:00.436693+0000","last_active":"2024-03-04T09:28:01.690928+0000","last_peered":"2024-03-04T09:28:01.690928+0000","last_clean":"2024-03-04T09:28:01.690928+0000","last_became_active":"2024-03-04T09:28:00.436422+0000","last_became_peered":"2024-03-04T09:28:00.436422+0000","last_unstale":"2024-03-04T09:28:01.690928+0000","last_undegraded":"2024-03-04T09:28:01.690928+0000","last_fullsized":"2024-03-04T09:28:01.690928+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:14:30.846824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.425773+0000","last_change":"2024-03-04T09:28:00.429162+0000","last_active":"2024-03-04T09:28:01.425773+0000","last_peered":"2024-03-04T09:28:01.425773+0000","last_clean":"2024-03-04T09:28:01.425773+0000","last_became_active":"2024-03-04T09:28:00.428868+0000","last_became_peered":"2024-03-04T09:28:00.428868+0000","last_unstale":"2024-03-04T09:28:01.425773+0000","last_undegraded":"2024-03-04T09:28:01.425773+0000","last_fullsized":"2024-03-04T09:28:01.425773+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:45:56.673753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.425632+0000","last_change":"2024-03-04T09:28:00.423881+0000","last_active":"2024-03-04T09:28:01.425632+0000","last_peered":"2024-03-04T09:28:01.425632+0000","last_clean":"2024-03-04T09:28:01.425632+0000","last_became_active":"2024-03-04T09:28:00.423336+0000","last_became_peered":"2024-03-04T09:28:00.423336+0000","last_unstale":"2024-03-04T09:28:01.425632+0000","last_undegraded":"2024-03-04T09:28:01.425632+0000","last_fullsized":"2024-03-04T09:28:01.425632+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:41:20.384844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.692437+0000","last_change":"2024-03-04T09:28:00.433918+0000","last_active":"2024-03-04T09:28:01.692437+0000","last_peered":"2024-03-04T09:28:01.692437+0000","last_clean":"2024-03-04T09:28:01.692437+0000","last_became_active":"2024-03-04T09:28:00.433625+0000","last_became_peered":"2024-03-04T09:28:00.433625+0000","last_unstale":"2024-03-04T09:28:01.692437+0000","last_undegraded":"2024-03-04T09:28:01.692437+0000","last_fullsized":"2024-03-04T09:28:01.692437+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:38:44.223188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.428236+0000","last_change":"2024-03-04T09:28:00.435007+0000","last_active":"2024-03-04T09:28:01.428236+0000","last_peered":"2024-03-04T09:28:01.428236+0000","last_clean":"2024-03-04T09:28:01.428236+0000","last_became_active":"2024-03-04T09:28:00.434657+0000","last_became_peered":"2024-03-04T09:28:00.434657+0000","last_unstale":"2024-03-04T09:28:01.428236+0000","last_undegraded":"2024-03-04T09:28:01.428236+0000","last_fullsized":"2024-03-04T09:28:01.428236+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:24:53.662456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.690218+0000","last_change":"2024-03-04T09:28:00.439224+0000","last_active":"2024-03-04T09:28:01.690218+0000","last_peered":"2024-03-04T09:28:01.690218+0000","last_clean":"2024-03-04T09:28:01.690218+0000","last_became_active":"2024-03-04T09:28:00.438890+0000","last_became_peered":"2024-03-04T09:28:00.438890+0000","last_unstale":"2024-03-04T09:28:01.690218+0000","last_undegraded":"2024-03-04T09:28:01.690218+0000","last_fullsized":"2024-03-04T09:28:01.690218+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:43:29.068713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.692178+0000","last_change":"2024-03-04T09:28:00.432426+0000","last_active":"2024-03-04T09:28:01.692178+0000","last_peered":"2024-03-04T09:28:01.692178+0000","last_clean":"2024-03-04T09:28:01.692178+0000","last_became_active":"2024-03-04T09:28:00.432126+0000","last_became_peered":"2024-03-04T09:28:00.432126+0000","last_unstale":"2024-03-04T09:28:01.692178+0000","last_undegraded":"2024-03-04T09:28:01.692178+0000","last_fullsized":"2024-03-04T09:28:01.692178+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:30:38.303164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.428530+0000","last_change":"2024-03-04T09:28:00.433183+0000","last_active":"2024-03-04T09:28:01.428530+0000","last_peered":"2024-03-04T09:28:01.428530+0000","last_clean":"2024-03-04T09:28:01.428530+0000","last_became_active":"2024-03-04T09:28:00.431904+0000","last_became_peered":"2024-03-04T09:28:00.431904+0000","last_unstale":"2024-03-04T09:28:01.428530+0000","last_undegraded":"2024-03-04T09:28:01.428530+0000","last_fullsized":"2024-03-04T09:28:01.428530+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:05:38.576682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.692188+0000","last_change":"2024-03-04T09:28:00.436128+0000","last_active":"2024-03-04T09:28:01.692188+0000","last_peered":"2024-03-04T09:28:01.692188+0000","last_clean":"2024-03-04T09:28:01.692188+0000","last_became_active":"2024-03-04T09:28:00.434844+0000","last_became_peered":"2024-03-04T09:28:00.434844+0000","last_unstale":"2024-03-04T09:28:01.692188+0000","last_undegraded":"2024-03-04T09:28:01.692188+0000","last_fullsized":"2024-03-04T09:28:01.692188+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:28:31.343554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.691842+0000","last_change":"2024-03-04T09:28:00.433628+0000","last_active":"2024-03-04T09:28:01.691842+0000","last_peered":"2024-03-04T09:28:01.691842+0000","last_clean":"2024-03-04T09:28:01.691842+0000","last_became_active":"2024-03-04T09:28:00.433343+0000","last_became_peered":"2024-03-04T09:28:00.433343+0000","last_unstale":"2024-03-04T09:28:01.691842+0000","last_undegraded":"2024-03-04T09:28:01.691842+0000","last_fullsized":"2024-03-04T09:28:01.691842+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:17:09.522879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.689494+0000","last_change":"2024-03-04T09:28:00.424933+0000","last_active":"2024-03-04T09:28:01.689494+0000","last_peered":"2024-03-04T09:28:01.689494+0000","last_clean":"2024-03-04T09:28:01.689494+0000","last_became_active":"2024-03-04T09:28:00.424644+0000","last_became_peered":"2024-03-04T09:28:00.424644+0000","last_unstale":"2024-03-04T09:28:01.689494+0000","last_undegraded":"2024-03-04T09:28:01.689494+0000","last_fullsized":"2024-03-04T09:28:01.689494+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:49:58.002468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.691417+0000","last_change":"2024-03-04T09:28:00.439954+0000","last_active":"2024-03-04T09:28:01.691417+0000","last_peered":"2024-03-04T09:28:01.691417+0000","last_clean":"2024-03-04T09:28:01.691417+0000","last_became_active":"2024-03-04T09:28:00.439672+0000","last_became_peered":"2024-03-04T09:28:00.439672+0000","last_unstale":"2024-03-04T09:28:01.691417+0000","last_undegraded":"2024-03-04T09:28:01.691417+0000","last_fullsized":"2024-03-04T09:28:01.691417+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:42:03.865562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.693565+0000","last_change":"2024-03-04T09:28:00.430020+0000","last_active":"2024-03-04T09:28:01.693565+0000","last_peered":"2024-03-04T09:28:01.693565+0000","last_clean":"2024-03-04T09:28:01.693565+0000","last_became_active":"2024-03-04T09:28:00.429673+0000","last_became_peered":"2024-03-04T09:28:00.429673+0000","last_unstale":"2024-03-04T09:28:01.693565+0000","last_undegraded":"2024-03-04T09:28:01.693565+0000","last_fullsized":"2024-03-04T09:28:01.693565+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:47:11.430142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.691015+0000","last_change":"2024-03-04T09:28:00.431742+0000","last_active":"2024-03-04T09:28:01.691015+0000","last_peered":"2024-03-04T09:28:01.691015+0000","last_clean":"2024-03-04T09:28:01.691015+0000","last_became_active":"2024-03-04T09:28:00.431486+0000","last_became_peered":"2024-03-04T09:28:00.431486+0000","last_unstale":"2024-03-04T09:28:01.691015+0000","last_undegraded":"2024-03-04T09:28:01.691015+0000","last_fullsized":"2024-03-04T09:28:01.691015+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:12:48.679791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.693264+0000","last_change":"2024-03-04T09:28:00.432686+0000","last_active":"2024-03-04T09:28:01.693264+0000","last_peered":"2024-03-04T09:28:01.693264+0000","last_clean":"2024-03-04T09:28:01.693264+0000","last_became_active":"2024-03-04T09:28:00.430974+0000","last_became_peered":"2024-03-04T09:28:00.430974+0000","last_unstale":"2024-03-04T09:28:01.693264+0000","last_undegraded":"2024-03-04T09:28:01.693264+0000","last_fullsized":"2024-03-04T09:28:01.693264+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:55:21.623042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.427793+0000","last_change":"2024-03-04T09:28:00.436106+0000","last_active":"2024-03-04T09:28:01.427793+0000","last_peered":"2024-03-04T09:28:01.427793+0000","last_clean":"2024-03-04T09:28:01.427793+0000","last_became_active":"2024-03-04T09:28:00.435781+0000","last_became_peered":"2024-03-04T09:28:00.435781+0000","last_unstale":"2024-03-04T09:28:01.427793+0000","last_undegraded":"2024-03-04T09:28:01.427793+0000","last_fullsized":"2024-03-04T09:28:01.427793+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:16:28.420706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.690312+0000","last_change":"2024-03-04T09:28:00.414866+0000","last_active":"2024-03-04T09:28:01.690312+0000","last_peered":"2024-03-04T09:28:01.690312+0000","last_clean":"2024-03-04T09:28:01.690312+0000","last_became_active":"2024-03-04T09:28:00.414050+0000","last_became_peered":"2024-03-04T09:28:00.414050+0000","last_unstale":"2024-03-04T09:28:01.690312+0000","last_undegraded":"2024-03-04T09:28:01.690312+0000","last_fullsized":"2024-03-04T09:28:01.690312+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:15:08.601355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.692170+0000","last_change":"2024-03-04T09:28:00.433459+0000","last_active":"2024-03-04T09:28:01.692170+0000","last_peered":"2024-03-04T09:28:01.692170+0000","last_clean":"2024-03-04T09:28:01.692170+0000","last_became_active":"2024-03-04T09:28:00.433111+0000","last_became_peered":"2024-03-04T09:28:00.433111+0000","last_unstale":"2024-03-04T09:28:01.692170+0000","last_undegraded":"2024-03-04T09:28:01.692170+0000","last_fullsized":"2024-03-04T09:28:01.692170+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:05:38.576682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.429400+0000","last_change":"2024-03-04T09:28:00.437328+0000","last_active":"2024-03-04T09:28:01.429400+0000","last_peered":"2024-03-04T09:28:01.429400+0000","last_clean":"2024-03-04T09:28:01.429400+0000","last_became_active":"2024-03-04T09:28:00.437044+0000","last_became_peered":"2024-03-04T09:28:00.437044+0000","last_unstale":"2024-03-04T09:28:01.429400+0000","last_undegraded":"2024-03-04T09:28:01.429400+0000","last_fullsized":"2024-03-04T09:28:01.429400+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:29:14.123293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.689030+0000","last_change":"2024-03-04T09:28:00.440592+0000","last_active":"2024-03-04T09:28:01.689030+0000","last_peered":"2024-03-04T09:28:01.689030+0000","last_clean":"2024-03-04T09:28:01.689030+0000","last_became_active":"2024-03-04T09:28:00.440186+0000","last_became_peered":"2024-03-04T09:28:00.440186+0000","last_unstale":"2024-03-04T09:28:01.689030+0000","last_undegraded":"2024-03-04T09:28:01.689030+0000","last_fullsized":"2024-03-04T09:28:01.689030+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:22:44.938157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.691829+0000","last_change":"2024-03-04T09:28:00.445380+0000","last_active":"2024-03-04T09:28:01.691829+0000","last_peered":"2024-03-04T09:28:01.691829+0000","last_clean":"2024-03-04T09:28:01.691829+0000","last_became_active":"2024-03-04T09:28:00.443638+0000","last_became_peered":"2024-03-04T09:28:00.443638+0000","last_unstale":"2024-03-04T09:28:01.691829+0000","last_undegraded":"2024-03-04T09:28:01.691829+0000","last_fullsized":"2024-03-04T09:28:01.691829+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:40:01.332927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.692922+0000","last_change":"2024-03-04T09:28:00.433320+0000","last_active":"2024-03-04T09:28:01.692922+0000","last_peered":"2024-03-04T09:28:01.692922+0000","last_clean":"2024-03-04T09:28:01.692922+0000","last_became_active":"2024-03-04T09:28:00.432271+0000","last_became_peered":"2024-03-04T09:28:00.432271+0000","last_unstale":"2024-03-04T09:28:01.692922+0000","last_undegraded":"2024-03-04T09:28:01.692922+0000","last_fullsized":"2024-03-04T09:28:01.692922+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:17:30.816921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.688694+0000","last_change":"2024-03-04T09:28:00.417556+0000","last_active":"2024-03-04T09:28:01.688694+0000","last_peered":"2024-03-04T09:28:01.688694+0000","last_clean":"2024-03-04T09:28:01.688694+0000","last_became_active":"2024-03-04T09:28:00.417265+0000","last_became_peered":"2024-03-04T09:28:00.417265+0000","last_unstale":"2024-03-04T09:28:01.688694+0000","last_undegraded":"2024-03-04T09:28:01.688694+0000","last_fullsized":"2024-03-04T09:28:01.688694+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:18:54.330037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.428623+0000","last_change":"2024-03-04T09:28:00.447034+0000","last_active":"2024-03-04T09:28:01.428623+0000","last_peered":"2024-03-04T09:28:01.428623+0000","last_clean":"2024-03-04T09:28:01.428623+0000","last_became_active":"2024-03-04T09:28:00.446685+0000","last_became_peered":"2024-03-04T09:28:00.446685+0000","last_unstale":"2024-03-04T09:28:01.428623+0000","last_undegraded":"2024-03-04T09:28:01.428623+0000","last_fullsized":"2024-03-04T09:28:01.428623+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:18:19.609722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.430062+0000","last_change":"2024-03-04T09:28:00.436807+0000","last_active":"2024-03-04T09:28:01.430062+0000","last_peered":"2024-03-04T09:28:01.430062+0000","last_clean":"2024-03-04T09:28:01.430062+0000","last_became_active":"2024-03-04T09:28:00.436491+0000","last_became_peered":"2024-03-04T09:28:00.436491+0000","last_unstale":"2024-03-04T09:28:01.430062+0000","last_undegraded":"2024-03-04T09:28:01.430062+0000","last_fullsized":"2024-03-04T09:28:01.430062+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:47:11.430142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"55'4","reported_seq":18,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.429371+0000","last_change":"2024-03-04T09:27:59.407621+0000","last_active":"2024-03-04T09:28:01.429371+0000","last_peered":"2024-03-04T09:28:01.429371+0000","last_clean":"2024-03-04T09:28:01.429371+0000","last_became_active":"2024-03-04T09:27:59.407362+0000","last_became_peered":"2024-03-04T09:27:59.407362+0000","last_unstale":"2024-03-04T09:28:01.429371+0000","last_undegraded":"2024-03-04T09:28:01.429371+0000","last_fullsized":"2024-03-04T09:28:01.429371+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_clean_scrub_stamp":"2024-03-04T09:27:58.353037+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:18:59.307799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.426183+0000","last_change":"2024-03-04T09:28:00.430027+0000","last_active":"2024-03-04T09:28:01.426183+0000","last_peered":"2024-03-04T09:28:01.426183+0000","last_clean":"2024-03-04T09:28:01.426183+0000","last_became_active":"2024-03-04T09:28:00.429778+0000","last_became_peered":"2024-03-04T09:28:00.429778+0000","last_unstale":"2024-03-04T09:28:01.426183+0000","last_undegraded":"2024-03-04T09:28:01.426183+0000","last_fullsized":"2024-03-04T09:28:01.426183+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:33:19.462053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"55'1","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.691793+0000","last_change":"2024-03-04T09:27:59.412265+0000","last_active":"2024-03-04T09:28:01.691793+0000","last_peered":"2024-03-04T09:28:01.691793+0000","last_clean":"2024-03-04T09:28:01.691793+0000","last_became_active":"2024-03-04T09:27:59.411885+0000","last_became_peered":"2024-03-04T09:27:59.411885+0000","last_unstale":"2024-03-04T09:28:01.691793+0000","last_undegraded":"2024-03-04T09:28:01.691793+0000","last_fullsized":"2024-03-04T09:28:01.691793+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_clean_scrub_stamp":"2024-03-04T09:27:58.353037+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:05:37.562815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.429168+0000","last_change":"2024-03-04T09:28:00.438663+0000","last_active":"2024-03-04T09:28:01.429168+0000","last_peered":"2024-03-04T09:28:01.429168+0000","last_clean":"2024-03-04T09:28:01.429168+0000","last_became_active":"2024-03-04T09:28:00.438349+0000","last_became_peered":"2024-03-04T09:28:00.438349+0000","last_unstale":"2024-03-04T09:28:01.429168+0000","last_undegraded":"2024-03-04T09:28:01.429168+0000","last_fullsized":"2024-03-04T09:28:01.429168+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:17:30.816921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"55'2","reported_seq":16,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.430160+0000","last_change":"2024-03-04T09:27:59.409928+0000","last_active":"2024-03-04T09:28:01.430160+0000","last_peered":"2024-03-04T09:28:01.430160+0000","last_clean":"2024-03-04T09:28:01.430160+0000","last_became_active":"2024-03-04T09:27:59.409602+0000","last_became_peered":"2024-03-04T09:27:59.409602+0000","last_unstale":"2024-03-04T09:28:01.430160+0000","last_undegraded":"2024-03-04T09:28:01.430160+0000","last_fullsized":"2024-03-04T09:28:01.430160+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_clean_scrub_stamp":"2024-03-04T09:27:58.353037+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:43:28.054846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.429859+0000","last_change":"2024-03-04T09:28:00.442481+0000","last_active":"2024-03-04T09:28:01.429859+0000","last_peered":"2024-03-04T09:28:01.429859+0000","last_clean":"2024-03-04T09:28:01.429859+0000","last_became_active":"2024-03-04T09:28:00.442160+0000","last_became_peered":"2024-03-04T09:28:00.442160+0000","last_unstale":"2024-03-04T09:28:01.429859+0000","last_undegraded":"2024-03-04T09:28:01.429859+0000","last_fullsized":"2024-03-04T09:28:01.429859+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:43:51.380199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"55'2","reported_seq":16,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.692555+0000","last_change":"2024-03-04T09:27:59.410484+0000","last_active":"2024-03-04T09:28:01.692555+0000","last_peered":"2024-03-04T09:28:01.692555+0000","last_clean":"2024-03-04T09:28:01.692555+0000","last_became_active":"2024-03-04T09:27:59.409118+0000","last_became_peered":"2024-03-04T09:27:59.409118+0000","last_unstale":"2024-03-04T09:28:01.692555+0000","last_undegraded":"2024-03-04T09:28:01.692555+0000","last_fullsized":"2024-03-04T09:28:01.692555+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_clean_scrub_stamp":"2024-03-04T09:27:58.353037+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:43:28.054846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.692105+0000","last_change":"2024-03-04T09:28:00.443500+0000","last_active":"2024-03-04T09:28:01.692105+0000","last_peered":"2024-03-04T09:28:01.692105+0000","last_clean":"2024-03-04T09:28:01.692105+0000","last_became_active":"2024-03-04T09:28:00.443138+0000","last_became_peered":"2024-03-04T09:28:00.443138+0000","last_unstale":"2024-03-04T09:28:01.692105+0000","last_undegraded":"2024-03-04T09:28:01.692105+0000","last_fullsized":"2024-03-04T09:28:01.692105+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:46:54.935661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"55'1","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.691213+0000","last_change":"2024-03-04T09:27:59.408748+0000","last_active":"2024-03-04T09:28:01.691213+0000","last_peered":"2024-03-04T09:28:01.691213+0000","last_clean":"2024-03-04T09:28:01.691213+0000","last_became_active":"2024-03-04T09:27:59.407429+0000","last_became_peered":"2024-03-04T09:27:59.407429+0000","last_unstale":"2024-03-04T09:28:01.691213+0000","last_undegraded":"2024-03-04T09:28:01.691213+0000","last_fullsized":"2024-03-04T09:28:01.691213+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_clean_scrub_stamp":"2024-03-04T09:27:58.353037+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:33:18.448186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.691377+0000","last_change":"2024-03-04T09:28:00.443410+0000","last_active":"2024-03-04T09:28:01.691377+0000","last_peered":"2024-03-04T09:28:01.691377+0000","last_clean":"2024-03-04T09:28:01.691377+0000","last_became_active":"2024-03-04T09:28:00.440808+0000","last_became_peered":"2024-03-04T09:28:00.440808+0000","last_unstale":"2024-03-04T09:28:01.691377+0000","last_undegraded":"2024-03-04T09:28:01.691377+0000","last_fullsized":"2024-03-04T09:28:01.691377+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:54:46.710146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"55'1","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.428891+0000","last_change":"2024-03-04T09:27:59.415921+0000","last_active":"2024-03-04T09:28:01.428891+0000","last_peered":"2024-03-04T09:28:01.428891+0000","last_clean":"2024-03-04T09:28:01.428891+0000","last_became_active":"2024-03-04T09:27:59.415586+0000","last_became_peered":"2024-03-04T09:27:59.415586+0000","last_unstale":"2024-03-04T09:28:01.428891+0000","last_undegraded":"2024-03-04T09:28:01.428891+0000","last_fullsized":"2024-03-04T09:28:01.428891+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_clean_scrub_stamp":"2024-03-04T09:27:58.353037+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:52:31.197392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.427866+0000","last_change":"2024-03-04T09:28:00.430007+0000","last_active":"2024-03-04T09:28:01.427866+0000","last_peered":"2024-03-04T09:28:01.427866+0000","last_clean":"2024-03-04T09:28:01.427866+0000","last_became_active":"2024-03-04T09:28:00.429411+0000","last_became_peered":"2024-03-04T09:28:00.429411+0000","last_unstale":"2024-03-04T09:28:01.427866+0000","last_undegraded":"2024-03-04T09:28:01.427866+0000","last_fullsized":"2024-03-04T09:28:01.427866+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:15:40.683978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.690180+0000","last_change":"2024-03-04T09:27:59.405815+0000","last_active":"2024-03-04T09:28:01.690180+0000","last_peered":"2024-03-04T09:28:01.690180+0000","last_clean":"2024-03-04T09:28:01.690180+0000","last_became_active":"2024-03-04T09:27:59.404028+0000","last_became_peered":"2024-03-04T09:27:59.404028+0000","last_unstale":"2024-03-04T09:28:01.690180+0000","last_undegraded":"2024-03-04T09:28:01.690180+0000","last_fullsized":"2024-03-04T09:28:01.690180+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_clean_scrub_stamp":"2024-03-04T09:27:58.353037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:52:31.197392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.692299+0000","last_change":"2024-03-04T09:28:00.435342+0000","last_active":"2024-03-04T09:28:01.692299+0000","last_peered":"2024-03-04T09:28:01.692299+0000","last_clean":"2024-03-04T09:28:01.692299+0000","last_became_active":"2024-03-04T09:28:00.435039+0000","last_became_peered":"2024-03-04T09:28:00.435039+0000","last_unstale":"2024-03-04T09:28:01.692299+0000","last_undegraded":"2024-03-04T09:28:01.692299+0000","last_fullsized":"2024-03-04T09:28:01.692299+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:11:51.717176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.690109+0000","last_change":"2024-03-04T09:27:59.420694+0000","last_active":"2024-03-04T09:28:01.690109+0000","last_peered":"2024-03-04T09:28:01.690109+0000","last_clean":"2024-03-04T09:28:01.690109+0000","last_became_active":"2024-03-04T09:27:59.420432+0000","last_became_peered":"2024-03-04T09:27:59.420432+0000","last_unstale":"2024-03-04T09:28:01.690109+0000","last_undegraded":"2024-03-04T09:28:01.690109+0000","last_fullsized":"2024-03-04T09:28:01.690109+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_clean_scrub_stamp":"2024-03-04T09:27:58.353037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:41:19.370977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.691324+0000","last_change":"2024-03-04T09:28:00.433093+0000","last_active":"2024-03-04T09:28:01.691324+0000","last_peered":"2024-03-04T09:28:01.691324+0000","last_clean":"2024-03-04T09:28:01.691324+0000","last_became_active":"2024-03-04T09:28:00.432809+0000","last_became_peered":"2024-03-04T09:28:00.432809+0000","last_unstale":"2024-03-04T09:28:01.691324+0000","last_undegraded":"2024-03-04T09:28:01.691324+0000","last_fullsized":"2024-03-04T09:28:01.691324+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:01:23.257447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.692857+0000","last_change":"2024-03-04T09:27:59.412972+0000","last_active":"2024-03-04T09:28:01.692857+0000","last_peered":"2024-03-04T09:28:01.692857+0000","last_clean":"2024-03-04T09:28:01.692857+0000","last_became_active":"2024-03-04T09:27:59.411051+0000","last_became_peered":"2024-03-04T09:27:59.411051+0000","last_unstale":"2024-03-04T09:28:01.692857+0000","last_undegraded":"2024-03-04T09:28:01.692857+0000","last_fullsized":"2024-03-04T09:28:01.692857+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_clean_scrub_stamp":"2024-03-04T09:27:58.353037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:18:59.307799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.692665+0000","last_change":"2024-03-04T09:28:00.425075+0000","last_active":"2024-03-04T09:28:01.692665+0000","last_peered":"2024-03-04T09:28:01.692665+0000","last_clean":"2024-03-04T09:28:01.692665+0000","last_became_active":"2024-03-04T09:28:00.422969+0000","last_became_peered":"2024-03-04T09:28:00.422969+0000","last_unstale":"2024-03-04T09:28:01.692665+0000","last_undegraded":"2024-03-04T09:28:01.692665+0000","last_fullsized":"2024-03-04T09:28:01.692665+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:42:00.605123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.692065+0000","last_change":"2024-03-04T09:27:59.417865+0000","last_active":"2024-03-04T09:28:01.692065+0000","last_peered":"2024-03-04T09:28:01.692065+0000","last_clean":"2024-03-04T09:28:01.692065+0000","last_became_active":"2024-03-04T09:27:59.417551+0000","last_became_peered":"2024-03-04T09:27:59.417551+0000","last_unstale":"2024-03-04T09:28:01.692065+0000","last_undegraded":"2024-03-04T09:28:01.692065+0000","last_fullsized":"2024-03-04T09:28:01.692065+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_clean_scrub_stamp":"2024-03-04T09:27:58.353037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:18:18.595855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.425244+0000","last_change":"2024-03-04T09:28:00.432532+0000","last_active":"2024-03-04T09:28:01.425244+0000","last_peered":"2024-03-04T09:28:01.425244+0000","last_clean":"2024-03-04T09:28:01.425244+0000","last_became_active":"2024-03-04T09:28:00.432146+0000","last_became_peered":"2024-03-04T09:28:00.432146+0000","last_unstale":"2024-03-04T09:28:01.425244+0000","last_undegraded":"2024-03-04T09:28:01.425244+0000","last_fullsized":"2024-03-04T09:28:01.425244+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:27:10.574104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"55'3","reported_seq":17,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.693092+0000","last_change":"2024-03-04T09:27:59.412981+0000","last_active":"2024-03-04T09:28:01.693092+0000","last_peered":"2024-03-04T09:28:01.693092+0000","last_clean":"2024-03-04T09:28:01.693092+0000","last_became_active":"2024-03-04T09:27:59.412560+0000","last_became_peered":"2024-03-04T09:27:59.412560+0000","last_unstale":"2024-03-04T09:28:01.693092+0000","last_undegraded":"2024-03-04T09:28:01.693092+0000","last_fullsized":"2024-03-04T09:28:01.693092+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_clean_scrub_stamp":"2024-03-04T09:27:58.353037+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:52:31.197392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.692594+0000","last_change":"2024-03-04T09:28:00.425119+0000","last_active":"2024-03-04T09:28:01.692594+0000","last_peered":"2024-03-04T09:28:01.692594+0000","last_clean":"2024-03-04T09:28:01.692594+0000","last_became_active":"2024-03-04T09:28:00.421652+0000","last_became_peered":"2024-03-04T09:28:00.421652+0000","last_unstale":"2024-03-04T09:28:01.692594+0000","last_undegraded":"2024-03-04T09:28:01.692594+0000","last_fullsized":"2024-03-04T09:28:01.692594+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:38:44.223188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"55'6","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.690885+0000","last_change":"2024-03-04T09:27:59.410959+0000","last_active":"2024-03-04T09:28:01.690885+0000","last_peered":"2024-03-04T09:28:01.690885+0000","last_clean":"2024-03-04T09:28:01.690885+0000","last_became_active":"2024-03-04T09:27:59.410593+0000","last_became_peered":"2024-03-04T09:27:59.410593+0000","last_unstale":"2024-03-04T09:28:01.690885+0000","last_undegraded":"2024-03-04T09:28:01.690885+0000","last_fullsized":"2024-03-04T09:28:01.690885+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_clean_scrub_stamp":"2024-03-04T09:27:58.353037+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:54:18.916314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.689942+0000","last_change":"2024-03-04T09:28:00.442151+0000","last_active":"2024-03-04T09:28:01.689942+0000","last_peered":"2024-03-04T09:28:01.689942+0000","last_clean":"2024-03-04T09:28:01.689942+0000","last_became_active":"2024-03-04T09:28:00.434929+0000","last_became_peered":"2024-03-04T09:28:00.434929+0000","last_unstale":"2024-03-04T09:28:01.689942+0000","last_undegraded":"2024-03-04T09:28:01.689942+0000","last_fullsized":"2024-03-04T09:28:01.689942+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:30:38.292668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"55'2","reported_seq":16,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.691380+0000","last_change":"2024-03-04T09:27:59.421713+0000","last_active":"2024-03-04T09:28:01.691380+0000","last_peered":"2024-03-04T09:28:01.691380+0000","last_clean":"2024-03-04T09:28:01.691380+0000","last_became_active":"2024-03-04T09:27:59.421381+0000","last_became_peered":"2024-03-04T09:27:59.421381+0000","last_unstale":"2024-03-04T09:28:01.691380+0000","last_undegraded":"2024-03-04T09:28:01.691380+0000","last_fullsized":"2024-03-04T09:28:01.691380+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_clean_scrub_stamp":"2024-03-04T09:27:58.353037+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:30:37.278801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.692163+0000","last_change":"2024-03-04T09:28:00.424233+0000","last_active":"2024-03-04T09:28:01.692163+0000","last_peered":"2024-03-04T09:28:01.692163+0000","last_clean":"2024-03-04T09:28:01.692163+0000","last_became_active":"2024-03-04T09:28:00.423612+0000","last_became_peered":"2024-03-04T09:28:00.423612+0000","last_unstale":"2024-03-04T09:28:01.692163+0000","last_undegraded":"2024-03-04T09:28:01.692163+0000","last_fullsized":"2024-03-04T09:28:01.692163+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:30:38.292668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.427495+0000","last_change":"2024-03-04T09:27:59.410074+0000","last_active":"2024-03-04T09:28:01.427495+0000","last_peered":"2024-03-04T09:28:01.427495+0000","last_clean":"2024-03-04T09:28:01.427495+0000","last_became_active":"2024-03-04T09:27:59.409636+0000","last_became_peered":"2024-03-04T09:27:59.409636+0000","last_unstale":"2024-03-04T09:28:01.427495+0000","last_undegraded":"2024-03-04T09:28:01.427495+0000","last_fullsized":"2024-03-04T09:28:01.427495+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_clean_scrub_stamp":"2024-03-04T09:27:58.353037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:53:26.740726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.689879+0000","last_change":"2024-03-04T09:28:00.436300+0000","last_active":"2024-03-04T09:28:01.689879+0000","last_peered":"2024-03-04T09:28:01.689879+0000","last_clean":"2024-03-04T09:28:01.689879+0000","last_became_active":"2024-03-04T09:28:00.436005+0000","last_became_peered":"2024-03-04T09:28:00.436005+0000","last_unstale":"2024-03-04T09:28:01.689879+0000","last_undegraded":"2024-03-04T09:28:01.689879+0000","last_fullsized":"2024-03-04T09:28:01.689879+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:28:55.761203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.690386+0000","last_change":"2024-03-04T09:27:59.409146+0000","last_active":"2024-03-04T09:28:01.690386+0000","last_peered":"2024-03-04T09:28:01.690386+0000","last_clean":"2024-03-04T09:28:01.690386+0000","last_became_active":"2024-03-04T09:27:59.408651+0000","last_became_peered":"2024-03-04T09:27:59.408651+0000","last_unstale":"2024-03-04T09:28:01.690386+0000","last_undegraded":"2024-03-04T09:28:01.690386+0000","last_fullsized":"2024-03-04T09:28:01.690386+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_clean_scrub_stamp":"2024-03-04T09:27:58.353037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:28:54.747336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.693468+0000","last_change":"2024-03-04T09:28:00.432457+0000","last_active":"2024-03-04T09:28:01.693468+0000","last_peered":"2024-03-04T09:28:01.693468+0000","last_clean":"2024-03-04T09:28:01.693468+0000","last_became_active":"2024-03-04T09:28:00.430610+0000","last_became_peered":"2024-03-04T09:28:00.430610+0000","last_unstale":"2024-03-04T09:28:01.693468+0000","last_undegraded":"2024-03-04T09:28:01.693468+0000","last_fullsized":"2024-03-04T09:28:01.693468+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:43:51.380199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"55'8","reported_seq":22,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.691398+0000","last_change":"2024-03-04T09:27:59.416172+0000","last_active":"2024-03-04T09:28:01.691398+0000","last_peered":"2024-03-04T09:28:01.691398+0000","last_clean":"2024-03-04T09:28:01.691398+0000","last_became_active":"2024-03-04T09:27:59.415946+0000","last_became_peered":"2024-03-04T09:27:59.415946+0000","last_unstale":"2024-03-04T09:28:01.691398+0000","last_undegraded":"2024-03-04T09:28:01.691398+0000","last_fullsized":"2024-03-04T09:28:01.691398+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_clean_scrub_stamp":"2024-03-04T09:27:58.353037+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:34:09.176083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.692166+0000","last_change":"2024-03-04T09:28:00.430963+0000","last_active":"2024-03-04T09:28:01.692166+0000","last_peered":"2024-03-04T09:28:01.692166+0000","last_clean":"2024-03-04T09:28:01.692166+0000","last_became_active":"2024-03-04T09:28:00.430468+0000","last_became_peered":"2024-03-04T09:28:00.430468+0000","last_unstale":"2024-03-04T09:28:01.692166+0000","last_undegraded":"2024-03-04T09:28:01.692166+0000","last_fullsized":"2024-03-04T09:28:01.692166+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:41:49.927616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"55'1","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.692446+0000","last_change":"2024-03-04T09:27:59.421957+0000","last_active":"2024-03-04T09:28:01.692446+0000","last_peered":"2024-03-04T09:28:01.692446+0000","last_clean":"2024-03-04T09:28:01.692446+0000","last_became_active":"2024-03-04T09:27:59.421523+0000","last_became_peered":"2024-03-04T09:27:59.421523+0000","last_unstale":"2024-03-04T09:28:01.692446+0000","last_undegraded":"2024-03-04T09:28:01.692446+0000","last_fullsized":"2024-03-04T09:28:01.692446+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_clean_scrub_stamp":"2024-03-04T09:27:58.353037+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:38:43.209321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.693292+0000","last_change":"2024-03-04T09:28:00.444341+0000","last_active":"2024-03-04T09:28:01.693292+0000","last_peered":"2024-03-04T09:28:01.693292+0000","last_clean":"2024-03-04T09:28:01.693292+0000","last_became_active":"2024-03-04T09:28:00.443965+0000","last_became_peered":"2024-03-04T09:28:00.443965+0000","last_unstale":"2024-03-04T09:28:01.693292+0000","last_undegraded":"2024-03-04T09:28:01.693292+0000","last_fullsized":"2024-03-04T09:28:01.693292+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:17:30.816921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"55'2","reported_seq":16,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.430112+0000","last_change":"2024-03-04T09:27:59.407941+0000","last_active":"2024-03-04T09:28:01.430112+0000","last_peered":"2024-03-04T09:28:01.430112+0000","last_clean":"2024-03-04T09:28:01.430112+0000","last_became_active":"2024-03-04T09:27:59.406809+0000","last_became_peered":"2024-03-04T09:27:59.406809+0000","last_unstale":"2024-03-04T09:28:01.430112+0000","last_undegraded":"2024-03-04T09:28:01.430112+0000","last_fullsized":"2024-03-04T09:28:01.430112+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_clean_scrub_stamp":"2024-03-04T09:27:58.353037+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:14:29.832957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.691859+0000","last_change":"2024-03-04T09:28:00.436859+0000","last_active":"2024-03-04T09:28:01.691859+0000","last_peered":"2024-03-04T09:28:01.691859+0000","last_clean":"2024-03-04T09:28:01.691859+0000","last_became_active":"2024-03-04T09:28:00.436021+0000","last_became_peered":"2024-03-04T09:28:00.436021+0000","last_unstale":"2024-03-04T09:28:01.691859+0000","last_undegraded":"2024-03-04T09:28:01.691859+0000","last_fullsized":"2024-03-04T09:28:01.691859+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:22:41.561201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"55'1","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.691838+0000","last_change":"2024-03-04T09:27:59.409078+0000","last_active":"2024-03-04T09:28:01.691838+0000","last_peered":"2024-03-04T09:28:01.691838+0000","last_clean":"2024-03-04T09:28:01.691838+0000","last_became_active":"2024-03-04T09:27:59.408701+0000","last_became_peered":"2024-03-04T09:27:59.408701+0000","last_unstale":"2024-03-04T09:28:01.691838+0000","last_undegraded":"2024-03-04T09:28:01.691838+0000","last_fullsized":"2024-03-04T09:28:01.691838+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_clean_scrub_stamp":"2024-03-04T09:27:58.353037+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:01:22.243580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.693019+0000","last_change":"2024-03-04T09:28:00.420941+0000","last_active":"2024-03-04T09:28:01.693019+0000","last_peered":"2024-03-04T09:28:01.693019+0000","last_clean":"2024-03-04T09:28:01.693019+0000","last_became_active":"2024-03-04T09:28:00.420549+0000","last_became_peered":"2024-03-04T09:28:00.420549+0000","last_unstale":"2024-03-04T09:28:01.693019+0000","last_undegraded":"2024-03-04T09:28:01.693019+0000","last_fullsized":"2024-03-04T09:28:01.693019+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:29:14.123293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.429361+0000","last_change":"2024-03-04T09:27:59.424879+0000","last_active":"2024-03-04T09:28:01.429361+0000","last_peered":"2024-03-04T09:28:01.429361+0000","last_clean":"2024-03-04T09:28:01.429361+0000","last_became_active":"2024-03-04T09:27:59.424617+0000","last_became_peered":"2024-03-04T09:27:59.424617+0000","last_unstale":"2024-03-04T09:28:01.429361+0000","last_undegraded":"2024-03-04T09:28:01.429361+0000","last_fullsized":"2024-03-04T09:28:01.429361+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_clean_scrub_stamp":"2024-03-04T09:27:58.353037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:28:54.747336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.690255+0000","last_change":"2024-03-04T09:28:00.444454+0000","last_active":"2024-03-04T09:28:01.690255+0000","last_peered":"2024-03-04T09:28:01.690255+0000","last_clean":"2024-03-04T09:28:01.690255+0000","last_became_active":"2024-03-04T09:28:00.438200+0000","last_became_peered":"2024-03-04T09:28:00.438200+0000","last_unstale":"2024-03-04T09:28:01.690255+0000","last_undegraded":"2024-03-04T09:28:01.690255+0000","last_fullsized":"2024-03-04T09:28:01.690255+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:41:49.927616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"55'2","reported_seq":16,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.691690+0000","last_change":"2024-03-04T09:27:59.411348+0000","last_active":"2024-03-04T09:28:01.691690+0000","last_peered":"2024-03-04T09:28:01.691690+0000","last_clean":"2024-03-04T09:28:01.691690+0000","last_became_active":"2024-03-04T09:27:59.410984+0000","last_became_peered":"2024-03-04T09:27:59.410984+0000","last_unstale":"2024-03-04T09:28:01.691690+0000","last_undegraded":"2024-03-04T09:28:01.691690+0000","last_fullsized":"2024-03-04T09:28:01.691690+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_clean_scrub_stamp":"2024-03-04T09:27:58.353037+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:41:48.913749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.429741+0000","last_change":"2024-03-04T09:28:00.434771+0000","last_active":"2024-03-04T09:28:01.429741+0000","last_peered":"2024-03-04T09:28:01.429741+0000","last_clean":"2024-03-04T09:28:01.429741+0000","last_became_active":"2024-03-04T09:28:00.434457+0000","last_became_peered":"2024-03-04T09:28:00.434457+0000","last_unstale":"2024-03-04T09:28:01.429741+0000","last_undegraded":"2024-03-04T09:28:01.429741+0000","last_fullsized":"2024-03-04T09:28:01.429741+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:55:21.623042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.425928+0000","last_change":"2024-03-04T09:27:59.402170+0000","last_active":"2024-03-04T09:28:01.425928+0000","last_peered":"2024-03-04T09:28:01.425928+0000","last_clean":"2024-03-04T09:28:01.425928+0000","last_became_active":"2024-03-04T09:27:59.401808+0000","last_became_peered":"2024-03-04T09:27:59.401808+0000","last_unstale":"2024-03-04T09:28:01.425928+0000","last_undegraded":"2024-03-04T09:28:01.425928+0000","last_fullsized":"2024-03-04T09:28:01.425928+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_clean_scrub_stamp":"2024-03-04T09:27:58.353037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:18:30.896319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.428632+0000","last_change":"2024-03-04T09:28:00.435603+0000","last_active":"2024-03-04T09:28:01.428632+0000","last_peered":"2024-03-04T09:28:01.428632+0000","last_clean":"2024-03-04T09:28:01.428632+0000","last_became_active":"2024-03-04T09:28:00.435296+0000","last_became_peered":"2024-03-04T09:28:00.435296+0000","last_unstale":"2024-03-04T09:28:01.428632+0000","last_undegraded":"2024-03-04T09:28:01.428632+0000","last_fullsized":"2024-03-04T09:28:01.428632+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:38:44.223188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.689405+0000","last_change":"2024-03-04T09:27:59.408243+0000","last_active":"2024-03-04T09:28:01.689405+0000","last_peered":"2024-03-04T09:28:01.689405+0000","last_clean":"2024-03-04T09:28:01.689405+0000","last_became_active":"2024-03-04T09:27:59.405122+0000","last_became_peered":"2024-03-04T09:27:59.405122+0000","last_unstale":"2024-03-04T09:28:01.689405+0000","last_undegraded":"2024-03-04T09:28:01.689405+0000","last_fullsized":"2024-03-04T09:28:01.689405+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_clean_scrub_stamp":"2024-03-04T09:27:58.353037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:18:59.307799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.692453+0000","last_change":"2024-03-04T09:28:00.439522+0000","last_active":"2024-03-04T09:28:01.692453+0000","last_peered":"2024-03-04T09:28:01.692453+0000","last_clean":"2024-03-04T09:28:01.692453+0000","last_became_active":"2024-03-04T09:28:00.439003+0000","last_became_peered":"2024-03-04T09:28:00.439003+0000","last_unstale":"2024-03-04T09:28:01.692453+0000","last_undegraded":"2024-03-04T09:28:01.692453+0000","last_fullsized":"2024-03-04T09:28:01.692453+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:42:00.605123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"55'2","reported_seq":16,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.426020+0000","last_change":"2024-03-04T09:27:59.411808+0000","last_active":"2024-03-04T09:28:01.426020+0000","last_peered":"2024-03-04T09:28:01.426020+0000","last_clean":"2024-03-04T09:28:01.426020+0000","last_became_active":"2024-03-04T09:27:59.411311+0000","last_became_peered":"2024-03-04T09:27:59.411311+0000","last_unstale":"2024-03-04T09:28:01.426020+0000","last_undegraded":"2024-03-04T09:28:01.426020+0000","last_fullsized":"2024-03-04T09:28:01.426020+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_clean_scrub_stamp":"2024-03-04T09:27:58.353037+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:59:27.874462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.693581+0000","last_change":"2024-03-04T09:28:00.445347+0000","last_active":"2024-03-04T09:28:01.693581+0000","last_peered":"2024-03-04T09:28:01.693581+0000","last_clean":"2024-03-04T09:28:01.693581+0000","last_became_active":"2024-03-04T09:28:00.444521+0000","last_became_peered":"2024-03-04T09:28:00.444521+0000","last_unstale":"2024-03-04T09:28:01.693581+0000","last_undegraded":"2024-03-04T09:28:01.693581+0000","last_fullsized":"2024-03-04T09:28:01.693581+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:29:14.123293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"56'1","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.442419+0000","last_change":"2024-03-04T09:27:59.409598+0000","last_active":"2024-03-04T09:28:01.442419+0000","last_peered":"2024-03-04T09:28:01.442419+0000","last_clean":"2024-03-04T09:28:01.442419+0000","last_became_active":"2024-03-04T09:27:59.409330+0000","last_became_peered":"2024-03-04T09:27:59.409330+0000","last_unstale":"2024-03-04T09:28:01.442419+0000","last_undegraded":"2024-03-04T09:28:01.442419+0000","last_fullsized":"2024-03-04T09:28:01.442419+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_clean_scrub_stamp":"2024-03-04T09:27:58.353037+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:01:22.243580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.425623+0000","last_change":"2024-03-04T09:28:00.435631+0000","last_active":"2024-03-04T09:28:01.425623+0000","last_peered":"2024-03-04T09:28:01.425623+0000","last_clean":"2024-03-04T09:28:01.425623+0000","last_became_active":"2024-03-04T09:28:00.435201+0000","last_became_peered":"2024-03-04T09:28:00.435201+0000","last_unstale":"2024-03-04T09:28:01.425623+0000","last_undegraded":"2024-03-04T09:28:01.425623+0000","last_fullsized":"2024-03-04T09:28:01.425623+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:18:29.267029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.427659+0000","last_change":"2024-03-04T09:27:59.412187+0000","last_active":"2024-03-04T09:28:01.427659+0000","last_peered":"2024-03-04T09:28:01.427659+0000","last_clean":"2024-03-04T09:28:01.427659+0000","last_became_active":"2024-03-04T09:27:59.411905+0000","last_became_peered":"2024-03-04T09:27:59.411905+0000","last_unstale":"2024-03-04T09:28:01.427659+0000","last_undegraded":"2024-03-04T09:28:01.427659+0000","last_fullsized":"2024-03-04T09:28:01.427659+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_clean_scrub_stamp":"2024-03-04T09:27:58.353037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:01:05.085170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.430224+0000","last_change":"2024-03-04T09:28:00.448429+0000","last_active":"2024-03-04T09:28:01.430224+0000","last_peered":"2024-03-04T09:28:01.430224+0000","last_clean":"2024-03-04T09:28:01.430224+0000","last_became_active":"2024-03-04T09:28:00.446151+0000","last_became_peered":"2024-03-04T09:28:00.446151+0000","last_unstale":"2024-03-04T09:28:01.430224+0000","last_undegraded":"2024-03-04T09:28:01.430224+0000","last_fullsized":"2024-03-04T09:28:01.430224+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:00:41.718192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.692652+0000","last_change":"2024-03-04T09:27:59.410066+0000","last_active":"2024-03-04T09:28:01.692652+0000","last_peered":"2024-03-04T09:28:01.692652+0000","last_clean":"2024-03-04T09:28:01.692652+0000","last_became_active":"2024-03-04T09:27:59.409164+0000","last_became_peered":"2024-03-04T09:27:59.409164+0000","last_unstale":"2024-03-04T09:28:01.692652+0000","last_undegraded":"2024-03-04T09:28:01.692652+0000","last_fullsized":"2024-03-04T09:28:01.692652+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_clean_scrub_stamp":"2024-03-04T09:27:58.353037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:14:29.832957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.691555+0000","last_change":"2024-03-04T09:28:00.445983+0000","last_active":"2024-03-04T09:28:01.691555+0000","last_peered":"2024-03-04T09:28:01.691555+0000","last_clean":"2024-03-04T09:28:01.691555+0000","last_became_active":"2024-03-04T09:28:00.442214+0000","last_became_peered":"2024-03-04T09:28:00.442214+0000","last_unstale":"2024-03-04T09:28:01.691555+0000","last_undegraded":"2024-03-04T09:28:01.691555+0000","last_fullsized":"2024-03-04T09:28:01.691555+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:34:17.456837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.425285+0000","last_change":"2024-03-04T09:27:59.394255+0000","last_active":"2024-03-04T09:28:01.425285+0000","last_peered":"2024-03-04T09:28:01.425285+0000","last_clean":"2024-03-04T09:28:01.425285+0000","last_became_active":"2024-03-04T09:27:59.393926+0000","last_became_peered":"2024-03-04T09:27:59.393926+0000","last_unstale":"2024-03-04T09:28:01.425285+0000","last_undegraded":"2024-03-04T09:28:01.425285+0000","last_fullsized":"2024-03-04T09:28:01.425285+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_clean_scrub_stamp":"2024-03-04T09:27:58.353037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:01:05.085170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.692554+0000","last_change":"2024-03-04T09:28:00.435846+0000","last_active":"2024-03-04T09:28:01.692554+0000","last_peered":"2024-03-04T09:28:01.692554+0000","last_clean":"2024-03-04T09:28:01.692554+0000","last_became_active":"2024-03-04T09:28:00.435579+0000","last_became_peered":"2024-03-04T09:28:00.435579+0000","last_unstale":"2024-03-04T09:28:01.692554+0000","last_undegraded":"2024-03-04T09:28:01.692554+0000","last_fullsized":"2024-03-04T09:28:01.692554+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:18:19.609722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.691023+0000","last_change":"2024-03-04T09:27:59.410882+0000","last_active":"2024-03-04T09:28:01.691023+0000","last_peered":"2024-03-04T09:28:01.691023+0000","last_clean":"2024-03-04T09:28:01.691023+0000","last_became_active":"2024-03-04T09:27:59.409671+0000","last_became_peered":"2024-03-04T09:27:59.409671+0000","last_unstale":"2024-03-04T09:28:01.691023+0000","last_undegraded":"2024-03-04T09:28:01.691023+0000","last_fullsized":"2024-03-04T09:28:01.691023+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_clean_scrub_stamp":"2024-03-04T09:27:58.353037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:45:55.659886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.425720+0000","last_change":"2024-03-04T09:28:00.429447+0000","last_active":"2024-03-04T09:28:01.425720+0000","last_peered":"2024-03-04T09:28:01.425720+0000","last_clean":"2024-03-04T09:28:01.425720+0000","last_became_active":"2024-03-04T09:28:00.429183+0000","last_became_peered":"2024-03-04T09:28:00.429183+0000","last_unstale":"2024-03-04T09:28:01.425720+0000","last_undegraded":"2024-03-04T09:28:01.425720+0000","last_fullsized":"2024-03-04T09:28:01.425720+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:54:19.930181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.425125+0000","last_change":"2024-03-04T09:27:59.409340+0000","last_active":"2024-03-04T09:28:01.425125+0000","last_peered":"2024-03-04T09:28:01.425125+0000","last_clean":"2024-03-04T09:28:01.425125+0000","last_became_active":"2024-03-04T09:27:59.408288+0000","last_became_peered":"2024-03-04T09:27:59.408288+0000","last_unstale":"2024-03-04T09:28:01.425125+0000","last_undegraded":"2024-03-04T09:28:01.425125+0000","last_fullsized":"2024-03-04T09:28:01.425125+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_clean_scrub_stamp":"2024-03-04T09:27:58.353037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:53:26.740726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.690947+0000","last_change":"2024-03-04T09:28:00.429777+0000","last_active":"2024-03-04T09:28:01.690947+0000","last_peered":"2024-03-04T09:28:01.690947+0000","last_clean":"2024-03-04T09:28:01.690947+0000","last_became_active":"2024-03-04T09:28:00.429506+0000","last_became_peered":"2024-03-04T09:28:00.429506+0000","last_unstale":"2024-03-04T09:28:01.690947+0000","last_undegraded":"2024-03-04T09:28:01.690947+0000","last_fullsized":"2024-03-04T09:28:01.690947+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:45:23.923649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.690555+0000","last_change":"2024-03-04T09:27:59.404368+0000","last_active":"2024-03-04T09:28:01.690555+0000","last_peered":"2024-03-04T09:28:01.690555+0000","last_clean":"2024-03-04T09:28:01.690555+0000","last_became_active":"2024-03-04T09:27:59.403755+0000","last_became_peered":"2024-03-04T09:27:59.403755+0000","last_unstale":"2024-03-04T09:28:01.690555+0000","last_undegraded":"2024-03-04T09:28:01.690555+0000","last_fullsized":"2024-03-04T09:28:01.690555+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_clean_scrub_stamp":"2024-03-04T09:27:58.353037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:17:08.509012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.690312+0000","last_change":"2024-03-04T09:28:00.442817+0000","last_active":"2024-03-04T09:28:01.690312+0000","last_peered":"2024-03-04T09:28:01.690312+0000","last_clean":"2024-03-04T09:28:01.690312+0000","last_became_active":"2024-03-04T09:28:00.442265+0000","last_became_peered":"2024-03-04T09:28:00.442265+0000","last_unstale":"2024-03-04T09:28:01.690312+0000","last_undegraded":"2024-03-04T09:28:01.690312+0000","last_fullsized":"2024-03-04T09:28:01.690312+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:59.366904+0000","last_clean_scrub_stamp":"2024-03-04T09:27:59.366904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:52:07.773906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.427319+0000","last_change":"2024-03-04T09:27:59.414003+0000","last_active":"2024-03-04T09:28:01.427319+0000","last_peered":"2024-03-04T09:28:01.427319+0000","last_clean":"2024-03-04T09:28:01.427319+0000","last_became_active":"2024-03-04T09:27:59.412706+0000","last_became_peered":"2024-03-04T09:27:59.412706+0000","last_unstale":"2024-03-04T09:28:01.427319+0000","last_undegraded":"2024-03-04T09:28:01.427319+0000","last_fullsized":"2024-03-04T09:28:01.427319+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:27:58.353037+0000","last_clean_scrub_stamp":"2024-03-04T09:27:58.353037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:29:15.878468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":237,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-04T09:28:01.691303+0000","last_change":"2024-03-04T09:25:09.316159+0000","last_active":"2024-03-04T09:28:01.691303+0000","last_peered":"2024-03-04T09:28:01.691303+0000","last_clean":"2024-03-04T09:28:01.691303+0000","last_became_active":"2024-03-04T09:25:09.313767+0000","last_became_peered":"2024-03-04T09:25:09.313767+0000","last_unstale":"2024-03-04T09:28:01.691303+0000","last_undegraded":"2024-03-04T09:28:01.691303+0000","last_fullsized":"2024-03-04T09:28:01.691303+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:08.110953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:08.110953+0000","last_clean_scrub_stamp":"2024-03-04T09:25:08.110953+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:34:21.745390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574886,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32432,"kb_used_data":1244,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339408,"statfs":{"total":96636764160,"available":96603553792,"internally_reserved":0,"allocated":1273856,"data_stored":869083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574886,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31840,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340000,"statfs":{"total":96636764160,"available":96604160000,"internally_reserved":0,"allocated":671744,"data_stored":278555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574888,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31852,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339988,"statfs":{"total":96636764160,"available":96604147712,"internally_reserved":0,"allocated":679936,"data_stored":279207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574885,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31852,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339988,"statfs":{"total":96636764160,"available":96604147712,"internally_reserved":0,"allocated":684032,"data_stored":280167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574887,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31852,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339988,"statfs":{"total":96636764160,"available":96604147712,"internally_reserved":0,"allocated":679936,"data_stored":278611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574886,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31848,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339992,"statfs":{"total":96636764160,"available":96604151808,"internally_reserved":0,"allocated":679936,"data_stored":278611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574887,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31844,"kb_used_data":660,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339996,"statfs":{"total":96636764160,"available":96604155904,"internally_reserved":0,"allocated":675840,"data_stored":278645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574886,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32440,"kb_used_data":1252,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339400,"statfs":{"total":96636764160,"available":96603545600,"internally_reserved":0,"allocated":1282048,"data_stored":871117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-04T09:28:05.444 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-04T09:28:05.444 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-04T09:28:05.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.719+0000 7ff67d800640 1 Processor -- start 2024-03-04T09:28:05.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.719+0000 7ff67d800640 1 -- start start 2024-03-04T09:28:05.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.721+0000 7ff67d800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff6780ad6f0 0x7ff67815c370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:05.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.721+0000 7ff67d800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff67815f960 0x7ff6780adad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:05.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.721+0000 7ff67d800640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff678156830 con 0x7ff6780b3c30 2024-03-04T09:28:05.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.721+0000 7ff67d800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff67806ce60 con 0x7ff6780ad6f0 2024-03-04T09:28:05.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.721+0000 7ff67d800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6780ae010 con 0x7ff67815f960 2024-03-04T09:28:05.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.722+0000 7ff676a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff6780ad6f0 0x7ff67815c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:05.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.722+0000 7ff677400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7ff6780b3c30 0x7ff6780b4080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:39828/0 (socket says 172.21.15.6:39828) 2024-03-04T09:28:05.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.722+0000 7ff677400640 1 -- 172.21.15.6:0/1692008971 learned_addr learned my addr 172.21.15.6:0/1692008971 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:28:05.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.722+0000 7ff676a00640 1 --2- 172.21.15.6:0/1692008971 >> v2:172.21.15.6:3300/0 conn(0x7ff6780ad6f0 0x7ff67815c370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45366/0 (socket says 172.21.15.6:45366) 2024-03-04T09:28:05.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.722+0000 7ff676000640 1 --2- 172.21.15.6:0/1692008971 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff67815f960 0x7ff6780adad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:05.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.723+0000 7ff675600640 1 -- 172.21.15.6:0/1692008971 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3817008836 0 0) 0x7ff678156830 con 0x7ff6780b3c30 2024-03-04T09:28:05.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.723+0000 7ff675600640 1 -- 172.21.15.6:0/1692008971 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff65c003520 con 0x7ff6780b3c30 2024-03-04T09:28:05.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.723+0000 7ff676a00640 1 -- 172.21.15.6:0/1692008971 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff67815f960 msgr2=0x7ff6780adad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:05.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.723+0000 7ff676a00640 1 --2- 172.21.15.6:0/1692008971 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff67815f960 0x7ff6780adad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:05.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.723+0000 7ff676a00640 1 -- 172.21.15.6:0/1692008971 >> v1:172.21.15.6:6789/0 conn(0x7ff6780b3c30 legacy=0x7ff6780b4080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:05.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.723+0000 7ff676a00640 1 -- 172.21.15.6:0/1692008971 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff678160120 con 0x7ff6780ad6f0 2024-03-04T09:28:05.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.724+0000 7ff676a00640 1 --2- 172.21.15.6:0/1692008971 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff6780ad6f0 0x7ff67815c370 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7ff6680097a0 tx=0x7ff66802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ec58cd95c2f5ac5 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:05.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.725+0000 7ff675600640 1 -- 172.21.15.6:0/1692008971 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff668035020 con 0x7ff6780ad6f0 2024-03-04T09:28:05.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.725+0000 7ff675600640 1 -- 172.21.15.6:0/1692008971 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff668031830 con 0x7ff6780ad6f0 2024-03-04T09:28:05.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.725+0000 7ff675600640 1 -- 172.21.15.6:0/1692008971 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff668031c00 con 0x7ff6780ad6f0 2024-03-04T09:28:05.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.725+0000 7ff67d800640 1 -- 172.21.15.6:0/1692008971 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff6780ad6f0 msgr2=0x7ff67815c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:05.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.725+0000 7ff67d800640 1 --2- 172.21.15.6:0/1692008971 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff6780ad6f0 0x7ff67815c370 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7ff6680097a0 tx=0x7ff66802b8d0 comp rx=0 tx=0).stop 2024-03-04T09:28:05.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.726+0000 7ff67d800640 1 -- 172.21.15.6:0/1692008971 shutdown_connections 2024-03-04T09:28:05.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.726+0000 7ff67d800640 1 --2- 172.21.15.6:0/1692008971 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff67815f960 0x7ff6780adad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:05.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.726+0000 7ff67d800640 1 --2- 172.21.15.6:0/1692008971 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff6780ad6f0 0x7ff67815c370 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:05.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.726+0000 7ff67d800640 1 -- 172.21.15.6:0/1692008971 >> 172.21.15.6:0/1692008971 conn(0x7ff67806b3d0 msgr2=0x7ff67806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:05.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.726+0000 7ff67d800640 1 -- 172.21.15.6:0/1692008971 shutdown_connections 2024-03-04T09:28:05.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.726+0000 7ff67d800640 1 -- 172.21.15.6:0/1692008971 wait complete. 2024-03-04T09:28:05.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.727+0000 7ff67d800640 1 Processor -- start 2024-03-04T09:28:05.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.728+0000 7ff67d800640 1 -- start start 2024-03-04T09:28:05.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.729+0000 7ff67d800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff6780ad6f0 0x7ff67814a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:05.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.729+0000 7ff67d800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff6780b3c30 0x7ff6780cd6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:05.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.729+0000 7ff67d800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff67815f960 0x7ff6780cdbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:05.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.729+0000 7ff67d800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff67806d0b0 con 0x7ff67815f960 2024-03-04T09:28:05.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.729+0000 7ff67d800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6781606e0 con 0x7ff6780b3c30 2024-03-04T09:28:05.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.729+0000 7ff67d800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff67815fdc0 con 0x7ff6780ad6f0 2024-03-04T09:28:05.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.729+0000 7ff677400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff67815f960 0x7ff6780cdbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:05.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.729+0000 7ff677400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff67815f960 0x7ff6780cdbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45378/0 (socket says 172.21.15.6:45378) 2024-03-04T09:28:05.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.729+0000 7ff677400640 1 -- 172.21.15.6:0/238085332 learned_addr learned my addr 172.21.15.6:0/238085332 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:28:05.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.729+0000 7ff676000640 1 --2- 172.21.15.6:0/238085332 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff6780b3c30 0x7ff6780cd6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:05.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.729+0000 7ff676a00640 1 --2- 172.21.15.6:0/238085332 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff6780ad6f0 0x7ff67814a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:05.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.730+0000 7ff677400640 1 -- 172.21.15.6:0/238085332 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff6780ad6f0 msgr2=0x7ff67814a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:05.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.730+0000 7ff677400640 1 --2- 172.21.15.6:0/238085332 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff6780ad6f0 0x7ff67814a6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:05.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.730+0000 7ff677400640 1 -- 172.21.15.6:0/238085332 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff6780b3c30 msgr2=0x7ff6780cd6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:05.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.730+0000 7ff677400640 1 --2- 172.21.15.6:0/238085332 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff6780b3c30 0x7ff6780cd6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:05.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.730+0000 7ff677400640 1 -- 172.21.15.6:0/238085332 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6781f3c80 con 0x7ff67815f960 2024-03-04T09:28:05.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.730+0000 7ff677400640 1 --2- 172.21.15.6:0/238085332 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff67815f960 0x7ff6780cdbf0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7ff67000cb10 tx=0x7ff67000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:05.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.730+0000 7ff66ea00640 1 -- 172.21.15.6:0/238085332 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff670017070 con 0x7ff67815f960 2024-03-04T09:28:05.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.730+0000 7ff66ea00640 1 -- 172.21.15.6:0/238085332 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff67000ac70 con 0x7ff67815f960 2024-03-04T09:28:05.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.730+0000 7ff66ea00640 1 -- 172.21.15.6:0/238085332 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff67000ae60 con 0x7ff67815f960 2024-03-04T09:28:05.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.730+0000 7ff67d800640 1 -- 172.21.15.6:0/238085332 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff67809ddb0 con 0x7ff67815f960 2024-03-04T09:28:05.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.730+0000 7ff67d800640 1 -- 172.21.15.6:0/238085332 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff67809b500 con 0x7ff67815f960 2024-03-04T09:28:05.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.732+0000 7ff67d800640 1 -- 172.21.15.6:0/238085332 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff678068010 con 0x7ff67815f960 2024-03-04T09:28:05.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.733+0000 7ff66ea00640 1 -- 172.21.15.6:0/238085332 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff67001a070 con 0x7ff67815f960 2024-03-04T09:28:05.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.734+0000 7ff66ea00640 1 --2- 172.21.15.6:0/238085332 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff65806b4d0 0x7ff65806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:05.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.734+0000 7ff66ea00640 1 -- 172.21.15.6:0/238085332 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff670089fd0 con 0x7ff67815f960 2024-03-04T09:28:05.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.735+0000 7ff676a00640 1 --2- 172.21.15.6:0/238085332 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff65806b4d0 0x7ff65806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:05.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.736+0000 7ff676a00640 1 --2- 172.21.15.6:0/238085332 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff65806b4d0 0x7ff65806d990 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7ff668036000 tx=0x7ff66803c040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:05.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:05.739+0000 7ff66ea00640 1 -- 172.21.15.6:0/238085332 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff670058e50 con 0x7ff67815f960 2024-03-04T09:28:06.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:06.006+0000 7ff67d800640 1 -- 172.21.15.6:0/238085332 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7ff6780b35b0 con 0x7ff67815f960 2024-03-04T09:28:06.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:06.010+0000 7ff66ea00640 1 -- 172.21.15.6:0/238085332 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v12) v1 ==== 190+0+147 (secure 0 0 0) 0x7ff670058750 con 0x7ff67815f960 2024-03-04T09:28:06.007 INFO:teuthology.orchestra.run.smithi006.stdout:[client.0] 2024-03-04T09:28:06.007 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQA9k+Vl4L+rJxAACaDbQ4/Bybp5hAZAnJzlVw== 2024-03-04T09:28:06.007 INFO:teuthology.orchestra.run.smithi006.stdout: caps mds = "allow" 2024-03-04T09:28:06.007 INFO:teuthology.orchestra.run.smithi006.stdout: caps mon = "allow r" 2024-03-04T09:28:06.007 INFO:teuthology.orchestra.run.smithi006.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-04T09:28:06.008 INFO:teuthology.orchestra.run.smithi006.stderr:updated caps for client.0 2024-03-04T09:28:06.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:06.013+0000 7ff67d800640 1 -- 172.21.15.6:0/238085332 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff65806b4d0 msgr2=0x7ff65806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:06.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:06.013+0000 7ff67d800640 1 --2- 172.21.15.6:0/238085332 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff65806b4d0 0x7ff65806d990 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7ff668036000 tx=0x7ff66803c040 comp rx=0 tx=0).stop 2024-03-04T09:28:06.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:06.013+0000 7ff67d800640 1 -- 172.21.15.6:0/238085332 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff67815f960 msgr2=0x7ff6780cdbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:06.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:06.013+0000 7ff67d800640 1 --2- 172.21.15.6:0/238085332 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff67815f960 0x7ff6780cdbf0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7ff67000cb10 tx=0x7ff67000a060 comp rx=0 tx=0).stop 2024-03-04T09:28:06.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:06.013+0000 7ff67d800640 1 -- 172.21.15.6:0/238085332 shutdown_connections 2024-03-04T09:28:06.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:06.013+0000 7ff67d800640 1 --2- 172.21.15.6:0/238085332 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff67815f960 0x7ff6780cdbf0 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:06.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:06.013+0000 7ff67d800640 1 --2- 172.21.15.6:0/238085332 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff6780b3c30 0x7ff6780cd6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:06.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:06.013+0000 7ff67d800640 1 --2- 172.21.15.6:0/238085332 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff65806b4d0 0x7ff65806d990 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:06.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:06.013+0000 7ff67d800640 1 --2- 172.21.15.6:0/238085332 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff6780ad6f0 0x7ff67814a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:06.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:06.013+0000 7ff67d800640 1 -- 172.21.15.6:0/238085332 >> 172.21.15.6:0/238085332 conn(0x7ff67806b3d0 msgr2=0x7ff67815cb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:06.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:06.013+0000 7ff67d800640 1 -- 172.21.15.6:0/238085332 shutdown_connections 2024-03-04T09:28:06.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:06.013+0000 7ff67d800640 1 -- 172.21.15.6:0/238085332 wait complete. 2024-03-04T09:28:06.030 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:28:06.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:06.232+0000 7f76d0e00640 1 Processor -- start 2024-03-04T09:28:06.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:06.232+0000 7f76d0e00640 1 -- start start 2024-03-04T09:28:06.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:06.234+0000 7f76d0e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f76cc070f10 0x7f76cc0cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:06.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:06.234+0000 7f76d0e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f76cc0cce90 0x7f76cc15c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:06.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:06.234+0000 7f76d0e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76cc0714d0 con 0x7f76cc145300 2024-03-04T09:28:06.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:06.234+0000 7f76d0e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76cc0cd300 con 0x7f76cc070f10 2024-03-04T09:28:06.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:06.234+0000 7f76d0e00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76cc0717b0 con 0x7f76cc0cce90 2024-03-04T09:28:06.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:06.234+0000 7f76caa00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f76cc070f10 0x7f76cc0cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:06.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:06.234+0000 7f76cb400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f76cc145300 0x7f76cc0b49a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:39832/0 (socket says 172.21.15.6:39832) 2024-03-04T09:28:06.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:06.234+0000 7f76cb400640 1 -- 172.21.15.6:0/3955344408 learned_addr learned my addr 172.21.15.6:0/3955344408 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:28:06.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:06.235+0000 7f76cbe00640 1 --2- 172.21.15.6:0/3955344408 >> v2:172.21.15.146:3300/0 conn(0x7f76cc0cce90 0x7f76cc15c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:06.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:06.235+0000 7f76caa00640 1 -- 172.21.15.6:0/3955344408 >> v2:172.21.15.146:3300/0 conn(0x7f76cc0cce90 msgr2=0x7f76cc15c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:06.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:06.235+0000 7f76caa00640 1 --2- 172.21.15.6:0/3955344408 >> v2:172.21.15.146:3300/0 conn(0x7f76cc0cce90 0x7f76cc15c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:06.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:06.235+0000 7f76caa00640 1 -- 172.21.15.6:0/3955344408 >> v1:172.21.15.6:6789/0 conn(0x7f76cc145300 legacy=0x7f76cc0b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:06.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:06.235+0000 7f76caa00640 1 -- 172.21.15.6:0/3955344408 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76cc071a90 con 0x7f76cc070f10 2024-03-04T09:28:06.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:06.236+0000 7f76caa00640 1 --2- 172.21.15.6:0/3955344408 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f76cc070f10 0x7f76cc0cc950 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f76cc0cc500 tx=0x7f76b802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4783d42821ad1b84 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:06.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:06.236+0000 7f76ca000640 1 -- 172.21.15.6:0/3955344408 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76b80315a0 con 0x7f76cc070f10 2024-03-04T09:28:06.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:06.236+0000 7f76ca000640 1 -- 172.21.15.6:0/3955344408 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76b8037070 con 0x7f76cc070f10 2024-03-04T09:28:06.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:06.236+0000 7f76ca000640 1 -- 172.21.15.6:0/3955344408 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76b8031960 con 0x7f76cc070f10 2024-03-04T09:28:06.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:06.237+0000 7f76d0e00640 1 -- 172.21.15.6:0/3955344408 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f76cc070f10 msgr2=0x7f76cc0cc950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:06.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:06.237+0000 7f76d0e00640 1 --2- 172.21.15.6:0/3955344408 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f76cc070f10 0x7f76cc0cc950 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f76cc0cc500 tx=0x7f76b802b840 comp rx=0 tx=0).stop 2024-03-04T09:28:06.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:06.237+0000 7f76d0e00640 1 -- 172.21.15.6:0/3955344408 shutdown_connections 2024-03-04T09:28:06.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:06.237+0000 7f76d0e00640 1 --2- 172.21.15.6:0/3955344408 >> v2:172.21.15.146:3300/0 conn(0x7f76cc0cce90 0x7f76cc15c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:06.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:06.237+0000 7f76d0e00640 1 --2- 172.21.15.6:0/3955344408 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f76cc070f10 0x7f76cc0cc950 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:06.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:06.237+0000 7f76d0e00640 1 -- 172.21.15.6:0/3955344408 >> 172.21.15.6:0/3955344408 conn(0x7f76cc06b3d0 msgr2=0x7f76cc06b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:06.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:06.237+0000 7f76d0e00640 1 -- 172.21.15.6:0/3955344408 shutdown_connections 2024-03-04T09:28:06.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:06.237+0000 7f76d0e00640 1 -- 172.21.15.6:0/3955344408 wait complete. 2024-03-04T09:28:06.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:06.238+0000 7f76d0e00640 1 Processor -- start 2024-03-04T09:28:06.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:06.239+0000 7f76d0e00640 1 -- start start 2024-03-04T09:28:06.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:06.239+0000 7f76d0e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f76cc070f10 0x7f76cc15b230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:06.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:06.240+0000 7f76d0e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f76cc0cce90 0x7f76cc0ae290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:06.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:06.240+0000 7f76d0e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f76cc145300 0x7f76cc131d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:06.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:06.240+0000 7f76d0e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76cc15cab0 con 0x7f76cc070f10 2024-03-04T09:28:06.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:06.240+0000 7f76d0e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76cc1456e0 con 0x7f76cc0cce90 2024-03-04T09:28:06.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:06.240+0000 7f76d0e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76cc15cc30 con 0x7f76cc145300 2024-03-04T09:28:06.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:06.240+0000 7f76cb400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f76cc070f10 0x7f76cc15b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:06.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:06.240+0000 7f76cb400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f76cc070f10 0x7f76cc15b230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45390/0 (socket says 172.21.15.6:45390) 2024-03-04T09:28:06.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:06.240+0000 7f76cb400640 1 -- 172.21.15.6:0/3146242060 learned_addr learned my addr 172.21.15.6:0/3146242060 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:28:06.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:06.240+0000 7f76caa00640 1 --2- 172.21.15.6:0/3146242060 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f76cc0cce90 0x7f76cc0ae290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:06.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:06.240+0000 7f76cb400640 1 -- 172.21.15.6:0/3146242060 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f76cc145300 msgr2=0x7f76cc131d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:06.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:06.240+0000 7f76cb400640 1 --2- 172.21.15.6:0/3146242060 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f76cc145300 0x7f76cc131d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:06.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:06.240+0000 7f76cb400640 1 -- 172.21.15.6:0/3146242060 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f76cc0cce90 msgr2=0x7f76cc0ae290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:06.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:06.240+0000 7f76cb400640 1 --2- 172.21.15.6:0/3146242060 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f76cc0cce90 0x7f76cc0ae290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:06.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:06.240+0000 7f76cb400640 1 -- 172.21.15.6:0/3146242060 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76cc0aec00 con 0x7f76cc070f10 2024-03-04T09:28:06.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:06.240+0000 7f76cb400640 1 --2- 172.21.15.6:0/3146242060 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f76cc070f10 0x7f76cc15b230 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f76bc00c3c0 tx=0x7f76bc00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:06.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:06.241+0000 7f76c3400640 1 -- 172.21.15.6:0/3146242060 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76bc017070 con 0x7f76cc070f10 2024-03-04T09:28:06.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:06.241+0000 7f76c3400640 1 -- 172.21.15.6:0/3146242060 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76bc005ce0 con 0x7f76cc070f10 2024-03-04T09:28:06.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:06.241+0000 7f76d0e00640 1 -- 172.21.15.6:0/3146242060 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76cc0aeee0 con 0x7f76cc070f10 2024-03-04T09:28:06.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:06.241+0000 7f76c3400640 1 -- 172.21.15.6:0/3146242060 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76bc0072f0 con 0x7f76cc070f10 2024-03-04T09:28:06.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:06.241+0000 7f76d0e00640 1 -- 172.21.15.6:0/3146242060 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76cc15b770 con 0x7f76cc070f10 2024-03-04T09:28:06.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:06.242+0000 7f76d0e00640 1 -- 172.21.15.6:0/3146242060 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76cc04f330 con 0x7f76cc070f10 2024-03-04T09:28:06.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:06.243+0000 7f76c3400640 1 -- 172.21.15.6:0/3146242060 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f76bc01a070 con 0x7f76cc070f10 2024-03-04T09:28:06.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:06.244+0000 7f76c3400640 1 --2- 172.21.15.6:0/3146242060 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f76ac06b500 0x7f76ac06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:06.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:06.244+0000 7f76c3400640 1 -- 172.21.15.6:0/3146242060 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f76bc089570 con 0x7f76cc070f10 2024-03-04T09:28:06.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:06.245+0000 7f76caa00640 1 --2- 172.21.15.6:0/3146242060 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f76ac06b500 0x7f76ac06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:06.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:06.246+0000 7f76caa00640 1 --2- 172.21.15.6:0/3146242060 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f76ac06b500 0x7f76ac06d9c0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f76b8036000 tx=0x7f76b8008030 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:06.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:06.248+0000 7f76c3400640 1 -- 172.21.15.6:0/3146242060 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f76bc0583e0 con 0x7f76cc070f10 2024-03-04T09:28:06.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:06.487+0000 7f76d0e00640 1 -- 172.21.15.6:0/3146242060 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f76cc1e5f20 con 0x7f76cc070f10 2024-03-04T09:28:06.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:06.488+0000 7f76c3400640 1 -- 172.21.15.6:0/3146242060 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+4971 (secure 0 0 0) 0x7f76bc087e20 con 0x7f76cc070f10 2024-03-04T09:28:06.485 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:28:06.485 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":34,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5030,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/1411942502","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1411942502},{"type":"v1","addr":"172.21.15.6:6834","nonce":1411942502}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28},{"gid":5039,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/1628166413","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":1628166413},{"type":"v1","addr":"172.21.15.146:6837","nonce":1628166413}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":5323,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/131491768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":131491768},{"type":"v1","addr":"172.21.15.6:6835","nonce":131491768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:27:59.854470+0000","modified":"2024-03-04T09:28:02.945053+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5326},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5326":{"gid":5326,"name":"b","rank":0,"incarnation":30,"state":"up:active","state_seq":3,"addr":"172.21.15.146:6836/527855083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":527855083},{"type":"v1","addr":"172.21.15.146:6836","nonce":527855083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5326,"qdb_cluster":[5326]},"id":4}]} 2024-03-04T09:28:06.485 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 34 2024-03-04T09:28:06.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:06.490+0000 7f76d0e00640 1 -- 172.21.15.6:0/3146242060 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f76ac06b500 msgr2=0x7f76ac06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:06.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:06.490+0000 7f76d0e00640 1 --2- 172.21.15.6:0/3146242060 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f76ac06b500 0x7f76ac06d9c0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f76b8036000 tx=0x7f76b8008030 comp rx=0 tx=0).stop 2024-03-04T09:28:06.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:06.490+0000 7f76d0e00640 1 -- 172.21.15.6:0/3146242060 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f76cc070f10 msgr2=0x7f76cc15b230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:06.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:06.490+0000 7f76d0e00640 1 --2- 172.21.15.6:0/3146242060 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f76cc070f10 0x7f76cc15b230 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f76bc00c3c0 tx=0x7f76bc00c880 comp rx=0 tx=0).stop 2024-03-04T09:28:06.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:06.490+0000 7f76d0e00640 1 -- 172.21.15.6:0/3146242060 shutdown_connections 2024-03-04T09:28:06.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:06.490+0000 7f76d0e00640 1 --2- 172.21.15.6:0/3146242060 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f76ac06b500 0x7f76ac06d9c0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:06.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:06.490+0000 7f76d0e00640 1 --2- 172.21.15.6:0/3146242060 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f76cc145300 0x7f76cc131d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:06.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:06.490+0000 7f76d0e00640 1 --2- 172.21.15.6:0/3146242060 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f76cc0cce90 0x7f76cc0ae290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:06.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:06.490+0000 7f76d0e00640 1 --2- 172.21.15.6:0/3146242060 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f76cc070f10 0x7f76cc15b230 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:06.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:06.490+0000 7f76d0e00640 1 -- 172.21.15.6:0/3146242060 >> 172.21.15.6:0/3146242060 conn(0x7f76cc06b3d0 msgr2=0x7f76cc0cd930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:06.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:06.491+0000 7f76d0e00640 1 -- 172.21.15.6:0/3146242060 shutdown_connections 2024-03-04T09:28:06.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:06.491+0000 7f76d0e00640 1 -- 172.21.15.6:0/3146242060 wait complete. 2024-03-04T09:28:06.909 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 34, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:27:59.854470+0000', 'modified': '2024-03-04T09:28:02.945053+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5326}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5326': {'gid': 5326, 'name': 'b', 'rank': 0, 'incarnation': 30, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.146:6836/527855083', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6834', 'nonce': 527855083}, {'type': 'v1', 'addr': '172.21.15.146:6836', 'nonce': 527855083}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [9], 'metadata_pool': 8, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5326, 'qdb_cluster': [5326]} 2024-03-04T09:28:06.909 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-04T09:28:06.910 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-04T09:28:07.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.141+0000 7f2bbe400640 1 Processor -- start 2024-03-04T09:28:07.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.141+0000 7f2bbe400640 1 -- start start 2024-03-04T09:28:07.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.143+0000 7f2bbe400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f2bb81552b0 0x7f2bb815b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:07.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.143+0000 7f2bbe400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2bb815e810 0x7f2bb80cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:07.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.143+0000 7f2bbe400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f2bb80cce90 0x7f2bb806c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:07.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.143+0000 7f2bbe400640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2bb815ed10 con 0x7f2bb80cce90 2024-03-04T09:28:07.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.143+0000 7f2bbe400640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2bb815eff0 con 0x7f2bb81552b0 2024-03-04T09:28:07.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.143+0000 7f2bbe400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bb815f2d0 con 0x7f2bb815e810 2024-03-04T09:28:07.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.143+0000 7f2bbda00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f2bb80cce90 0x7f2bb806c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:07.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.143+0000 7f2bbda00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f2bb80cce90 0x7f2bb806c7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45396/0 (socket says 172.21.15.6:45396) 2024-03-04T09:28:07.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.143+0000 7f2bbda00640 1 -- 172.21.15.6:0/3035831695 learned_addr learned my addr 172.21.15.6:0/3035831695 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:28:07.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.144+0000 7f2bbd000640 1 --2- 172.21.15.6:0/3035831695 >> v2:172.21.15.146:3300/0 conn(0x7f2bb81552b0 0x7f2bb815b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:07.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.144+0000 7f2bb7e00640 1 --2- 172.21.15.6:0/3035831695 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2bb815e810 0x7f2bb80cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:07.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.144+0000 7f2bbda00640 1 -- 172.21.15.6:0/3035831695 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2bb815e810 msgr2=0x7f2bb80cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:07.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.144+0000 7f2bbda00640 1 --2- 172.21.15.6:0/3035831695 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2bb815e810 0x7f2bb80cc950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:07.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.144+0000 7f2bbda00640 1 -- 172.21.15.6:0/3035831695 >> v2:172.21.15.146:3300/0 conn(0x7f2bb81552b0 msgr2=0x7f2bb815b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:07.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.144+0000 7f2bbda00640 1 --2- 172.21.15.6:0/3035831695 >> v2:172.21.15.146:3300/0 conn(0x7f2bb81552b0 0x7f2bb815b1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:07.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.144+0000 7f2bbda00640 1 -- 172.21.15.6:0/3035831695 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bb815f5b0 con 0x7f2bb80cce90 2024-03-04T09:28:07.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.145+0000 7f2bbda00640 1 --2- 172.21.15.6:0/3035831695 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2bb80cce90 0x7f2bb806c7c0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f2bb80cd540 tx=0x7f2bb002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47a55c67d416fc81 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:07.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.146+0000 7f2bb7400640 1 -- 172.21.15.6:0/3035831695 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bb002c9e0 con 0x7f2bb80cce90 2024-03-04T09:28:07.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.146+0000 7f2bb7400640 1 -- 172.21.15.6:0/3035831695 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bb002cb80 con 0x7f2bb80cce90 2024-03-04T09:28:07.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.146+0000 7f2bb7400640 1 -- 172.21.15.6:0/3035831695 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bb00343f0 con 0x7f2bb80cce90 2024-03-04T09:28:07.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.146+0000 7f2bbe400640 1 -- 172.21.15.6:0/3035831695 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2bb80cce90 msgr2=0x7f2bb806c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:07.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.146+0000 7f2bbe400640 1 --2- 172.21.15.6:0/3035831695 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2bb80cce90 0x7f2bb806c7c0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f2bb80cd540 tx=0x7f2bb002be40 comp rx=0 tx=0).stop 2024-03-04T09:28:07.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.147+0000 7f2bbe400640 1 -- 172.21.15.6:0/3035831695 shutdown_connections 2024-03-04T09:28:07.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.147+0000 7f2bbe400640 1 --2- 172.21.15.6:0/3035831695 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2bb80cce90 0x7f2bb806c7c0 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:07.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.147+0000 7f2bbe400640 1 --2- 172.21.15.6:0/3035831695 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2bb815e810 0x7f2bb80cc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:07.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.147+0000 7f2bbe400640 1 --2- 172.21.15.6:0/3035831695 >> v2:172.21.15.146:3300/0 conn(0x7f2bb81552b0 0x7f2bb815b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:07.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.147+0000 7f2bbe400640 1 -- 172.21.15.6:0/3035831695 >> 172.21.15.6:0/3035831695 conn(0x7f2bb806b3d0 msgr2=0x7f2bb806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:07.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.147+0000 7f2bbe400640 1 -- 172.21.15.6:0/3035831695 shutdown_connections 2024-03-04T09:28:07.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.147+0000 7f2bbe400640 1 -- 172.21.15.6:0/3035831695 wait complete. 2024-03-04T09:28:07.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.149+0000 7f2bbe400640 1 Processor -- start 2024-03-04T09:28:07.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.150+0000 7f2bbe400640 1 -- start start 2024-03-04T09:28:07.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.151+0000 7f2bbe400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2bb80cce90 0x7f2bb80b1a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:07.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.151+0000 7f2bbe400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2bb81552b0 0x7f2bb81e4630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:07.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.151+0000 7f2bbe400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2bb815e810 0x7f2bb8143f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:07.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.151+0000 7f2bbe400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bb806dd30 con 0x7f2bb81552b0 2024-03-04T09:28:07.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.151+0000 7f2bbe400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bb80b3760 con 0x7f2bb815e810 2024-03-04T09:28:07.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.151+0000 7f2bbe400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bb806deb0 con 0x7f2bb80cce90 2024-03-04T09:28:07.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.151+0000 7f2bb7e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2bb81552b0 0x7f2bb81e4630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:07.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.151+0000 7f2bb7e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2bb81552b0 0x7f2bb81e4630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45402/0 (socket says 172.21.15.6:45402) 2024-03-04T09:28:07.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.151+0000 7f2bb7e00640 1 -- 172.21.15.6:0/3855058261 learned_addr learned my addr 172.21.15.6:0/3855058261 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:28:07.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.152+0000 7f2bbd000640 1 --2- 172.21.15.6:0/3855058261 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2bb80cce90 0x7f2bb80b1a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:07.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.152+0000 7f2bbda00640 1 --2- 172.21.15.6:0/3855058261 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2bb815e810 0x7f2bb8143f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:07.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.152+0000 7f2bb7e00640 1 -- 172.21.15.6:0/3855058261 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2bb80cce90 msgr2=0x7f2bb80b1a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:07.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.152+0000 7f2bb7e00640 1 --2- 172.21.15.6:0/3855058261 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2bb80cce90 0x7f2bb80b1a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:07.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.152+0000 7f2bb7e00640 1 -- 172.21.15.6:0/3855058261 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2bb815e810 msgr2=0x7f2bb8143f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:07.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.152+0000 7f2bb7e00640 1 --2- 172.21.15.6:0/3855058261 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2bb815e810 0x7f2bb8143f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:07.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.152+0000 7f2bb7e00640 1 -- 172.21.15.6:0/3855058261 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bb81f3e10 con 0x7f2bb81552b0 2024-03-04T09:28:07.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.152+0000 7f2bb7e00640 1 --2- 172.21.15.6:0/3855058261 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2bb81552b0 0x7f2bb81e4630 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f2ba400c820 tx=0x7f2ba400cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:07.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.152+0000 7f2bb4c00640 1 -- 172.21.15.6:0/3855058261 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ba4015070 con 0x7f2bb81552b0 2024-03-04T09:28:07.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.153+0000 7f2bb4c00640 1 -- 172.21.15.6:0/3855058261 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ba4004240 con 0x7f2bb81552b0 2024-03-04T09:28:07.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.153+0000 7f2bb4c00640 1 -- 172.21.15.6:0/3855058261 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ba400ce10 con 0x7f2bb81552b0 2024-03-04T09:28:07.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.153+0000 7f2bbe400640 1 -- 172.21.15.6:0/3855058261 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bb81e50f0 con 0x7f2bb81552b0 2024-03-04T09:28:07.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.153+0000 7f2bbe400640 1 -- 172.21.15.6:0/3855058261 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bb81e5990 con 0x7f2bb81552b0 2024-03-04T09:28:07.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.156+0000 7f2bb4c00640 1 -- 172.21.15.6:0/3855058261 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2ba40077d0 con 0x7f2bb81552b0 2024-03-04T09:28:07.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.157+0000 7f2bb4c00640 1 --2- 172.21.15.6:0/3855058261 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2b9806b4d0 0x7f2b9806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:07.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.157+0000 7f2bb4c00640 1 -- 172.21.15.6:0/3855058261 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2ba408a1e0 con 0x7f2bb81552b0 2024-03-04T09:28:07.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.157+0000 7f2bc2637740 1 Processor -- start 2024-03-04T09:28:07.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.157+0000 7f2bc2637740 1 -- start start 2024-03-04T09:28:07.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.158+0000 7f2bbd000640 1 --2- 172.21.15.6:0/3855058261 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2b9806b4d0 0x7f2b9806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:07.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.158+0000 7f2bc2637740 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55cd39443990 0x55cd39416930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:07.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.158+0000 7f2bc2637740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55cd39443d70 0x55cd39446720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:07.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.158+0000 7f2bc2637740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55cd39416e70 0x55cd39446c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:07.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.158+0000 7f2bc2637740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd39380600 con 0x55cd39443d70 2024-03-04T09:28:07.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.158+0000 7f2bc2637740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd39329930 con 0x55cd39416e70 2024-03-04T09:28:07.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.158+0000 7f2bc2637740 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x55cd3937c6a0 con 0x55cd39443990 2024-03-04T09:28:07.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.159+0000 7f2bb7e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55cd39443d70 0x55cd39446720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:07.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.159+0000 7f2bb7e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55cd39443d70 0x55cd39446720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45406/0 (socket says 172.21.15.6:45406) 2024-03-04T09:28:07.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.159+0000 7f2bb7e00640 1 -- 172.21.15.6:0/2913077380 learned_addr learned my addr 172.21.15.6:0/2913077380 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:28:07.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.159+0000 7f2bbd000640 1 --2- 172.21.15.6:0/2913077380 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55cd39443990 0x55cd39416930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:07.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.159+0000 7f2bbda00640 1 --2- 172.21.15.6:0/2913077380 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55cd39416e70 0x55cd39446c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:07.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.159+0000 7f2bb7e00640 1 -- 172.21.15.6:0/2913077380 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55cd39443990 msgr2=0x55cd39416930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:07.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.159+0000 7f2bb7e00640 1 --2- 172.21.15.6:0/2913077380 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55cd39443990 0x55cd39416930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:07.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.159+0000 7f2bb7e00640 1 -- 172.21.15.6:0/2913077380 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55cd39416e70 msgr2=0x55cd39446c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:07.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.159+0000 7f2bb7e00640 1 --2- 172.21.15.6:0/2913077380 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55cd39416e70 0x55cd39446c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:07.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.159+0000 7f2bb7e00640 1 -- 172.21.15.6:0/2913077380 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cd390952d0 con 0x55cd39443d70 2024-03-04T09:28:07.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.159+0000 7f2bbd000640 1 --2- 172.21.15.6:0/2913077380 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55cd39443990 0x55cd39416930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:28:07.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.159+0000 7f2bbda00640 1 --2- 172.21.15.6:0/2913077380 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55cd39416e70 0x55cd39446c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:28:07.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.159+0000 7f2bbd000640 1 --2- 172.21.15.6:0/3855058261 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2b9806b4d0 0x7f2b9806d990 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f2bac006fd0 tx=0x7f2bac008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:07.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.159+0000 7f2bb7e00640 1 --2- 172.21.15.6:0/2913077380 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55cd39443d70 0x55cd39446720 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f2ba4060aa0 tx=0x7f2ba4059070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fc5d2bc8a06144a server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:07.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.160+0000 7f2ba9600640 1 -- 172.21.15.6:0/2913077380 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ba401e040 con 0x55cd39443d70 2024-03-04T09:28:07.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.160+0000 7f2ba9600640 1 -- 172.21.15.6:0/2913077380 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ba4060fd0 con 0x55cd39443d70 2024-03-04T09:28:07.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.160+0000 7f2ba9600640 1 -- 172.21.15.6:0/2913077380 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ba4054ea0 con 0x55cd39443d70 2024-03-04T09:28:07.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.160+0000 7f2bc2637740 1 -- 172.21.15.6:0/2913077380 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55cd39443d70 msgr2=0x55cd39446720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:07.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.160+0000 7f2bc2637740 1 --2- 172.21.15.6:0/2913077380 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55cd39443d70 0x55cd39446720 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f2ba4060aa0 tx=0x7f2ba4059070 comp rx=0 tx=0).stop 2024-03-04T09:28:07.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.160+0000 7f2bc2637740 1 -- 172.21.15.6:0/2913077380 shutdown_connections 2024-03-04T09:28:07.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.160+0000 7f2bc2637740 1 --2- 172.21.15.6:0/2913077380 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55cd39416e70 0x55cd39446c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:07.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.160+0000 7f2bc2637740 1 --2- 172.21.15.6:0/2913077380 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55cd39443d70 0x55cd39446720 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:07.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.160+0000 7f2bc2637740 1 --2- 172.21.15.6:0/2913077380 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55cd39443990 0x55cd39416930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:07.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.160+0000 7f2bc2637740 1 -- 172.21.15.6:0/2913077380 >> 172.21.15.6:0/2913077380 conn(0x55cd39418ba0 msgr2=0x55cd39461720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:07.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.160+0000 7f2bc2637740 1 -- 172.21.15.6:0/2913077380 shutdown_connections 2024-03-04T09:28:07.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.161+0000 7f2bc2637740 1 -- 172.21.15.6:0/2913077380 wait complete. 2024-03-04T09:28:07.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.161+0000 7f2bc2637740 1 -- start start 2024-03-04T09:28:07.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.161+0000 7f2bc2637740 1 Processor -- start 2024-03-04T09:28:07.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.162+0000 7f2bc2637740 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55cd39443990 0x55cd39434ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:07.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.162+0000 7f2bc2637740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55cd39443d70 0x55cd394497b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:07.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.163+0000 7f2bc2637740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55cd39449cf0 0x55cd3945bf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:07.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.163+0000 7f2bc2637740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd3941b140 con 0x55cd39449cf0 2024-03-04T09:28:07.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.163+0000 7f2bc2637740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd3933cd90 con 0x55cd39443d70 2024-03-04T09:28:07.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.163+0000 7f2bc2637740 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x55cd3941a530 con 0x55cd39443990 2024-03-04T09:28:07.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.163+0000 7f2bc2637740 10 client.0 _ll_register_callbacks cb 0x55cd39378350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T09:28:07.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.163+0000 7f2bbda00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55cd39449cf0 0x55cd3945bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:07.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.163+0000 7f2bbda00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55cd39449cf0 0x55cd3945bf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45420/0 (socket says 172.21.15.6:45420) 2024-03-04T09:28:07.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.163+0000 7f2bbda00640 1 -- 172.21.15.6:0/4026004852 learned_addr learned my addr 172.21.15.6:0/4026004852 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:28:07.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.163+0000 7f2bb7e00640 1 --2- 172.21.15.6:0/4026004852 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55cd39443d70 0x55cd394497b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:07.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.163+0000 7f2bbd000640 1 --2- 172.21.15.6:0/4026004852 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55cd39443990 0x55cd39434ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:07.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.163+0000 7f2bbda00640 1 -- 172.21.15.6:0/4026004852 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55cd39443990 msgr2=0x55cd39434ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:07.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.163+0000 7f2bbda00640 1 --2- 172.21.15.6:0/4026004852 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55cd39443990 0x55cd39434ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:07.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.163+0000 7f2bbda00640 1 -- 172.21.15.6:0/4026004852 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55cd39443d70 msgr2=0x55cd394497b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:07.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.163+0000 7f2bbda00640 1 --2- 172.21.15.6:0/4026004852 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55cd39443d70 0x55cd394497b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:07.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.163+0000 7f2bbda00640 1 -- 172.21.15.6:0/4026004852 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55cd3915f6b0 con 0x55cd39449cf0 2024-03-04T09:28:07.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.164+0000 7f2bbda00640 1 --2- 172.21.15.6:0/4026004852 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55cd39449cf0 0x55cd3945bf80 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f2bb0036040 tx=0x7f2bb0004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b05ef1ed9145955 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:07.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.164+0000 7f2b9e000640 10 client.0 ms_handle_connect on v2:172.21.15.6:3300/0 2024-03-04T09:28:07.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.164+0000 7f2b9e000640 1 -- 172.21.15.6:0/4026004852 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bb0009510 con 0x55cd39449cf0 2024-03-04T09:28:07.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.164+0000 7f2b9e000640 1 -- 172.21.15.6:0/4026004852 <== mon.0 v2:172.21.15.6:3300/0 2 ==== osd_map(56..56 src has 1..56) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2bb0034b30 con 0x55cd39449cf0 2024-03-04T09:28:07.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.164+0000 7f2bc2637740 1 -- 172.21.15.6:0/4026004852 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2ba4060fd0 con 0x55cd39449cf0 2024-03-04T09:28:07.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.164+0000 7f2b9e000640 1 -- 172.21.15.6:0/4026004852 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2bb004f050 con 0x55cd39449cf0 2024-03-04T09:28:07.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.164+0000 7f2bc2637740 10 client.6015 fetch_fsmap learned FSMap version 34 2024-03-04T09:28:07.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.164+0000 7f2bc2637740 1 -- 172.21.15.6:0/4026004852 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=2+}) v3 -- 0x55cd3945a040 con 0x55cd39449cf0 2024-03-04T09:28:07.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.165+0000 7f2b9e000640 1 -- 172.21.15.6:0/4026004852 <== mon.0 v2:172.21.15.6:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ba4060fd0 con 0x55cd39449cf0 2024-03-04T09:28:07.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.165+0000 7f2b9e000640 1 -- 172.21.15.6:0/4026004852 <== mon.0 v2:172.21.15.6:3300/0 5 ==== fsmap(e 34) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f2bb0052020 con 0x55cd39449cf0 2024-03-04T09:28:07.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.165+0000 7f2bc2637740 10 client.6015 fetch_fsmap finished waiting for FSMap version 34 2024-03-04T09:28:07.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.165+0000 7f2bc2637740 10 client.6015 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-04T09:28:07.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.165+0000 7f2bc2637740 20 client.6015 populate_metadata read hostname 'smithi006' 2024-03-04T09:28:07.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.165+0000 7f2bc2637740 1 --2- 172.21.15.6:0/4026004852 >> [v2:172.21.15.146:6834/527855083,v1:172.21.15.146:6836/527855083] conn(0x55cd394756a0 0x55cd3945aec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:07.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.165+0000 7f2bc2637740 4 client.6015 mds_command: new command op to 5326 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T09:28:07.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.165+0000 7f2bc2637740 1 -- 172.21.15.6:0/4026004852 --> [v2:172.21.15.146:6834/527855083,v1:172.21.15.146:6836/527855083] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55cd3938db40 con 0x55cd394756a0 2024-03-04T09:28:07.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.165+0000 7f2bbd000640 1 --2- 172.21.15.6:0/4026004852 >> [v2:172.21.15.146:6834/527855083,v1:172.21.15.146:6836/527855083] conn(0x55cd394756a0 0x55cd3945aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:07.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.166+0000 7f2bbd000640 1 --2- 172.21.15.6:0/4026004852 >> [v2:172.21.15.146:6834/527855083,v1:172.21.15.146:6836/527855083] conn(0x55cd394756a0 0x55cd3945aec0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c53efb68bede8085 server_cookie=a8e37ab1398cc497 in_seq=0 out_seq=0 2024-03-04T09:28:07.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.166+0000 7f2b9e000640 10 client.6015 ms_handle_connect on v2:172.21.15.146:6834/527855083 2024-03-04T09:28:07.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.168+0000 7f2b9e000640 1 -- 172.21.15.6:0/4026004852 <== mds.0 v2:172.21.15.146:6834/527855083 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55cd3938db40 con 0x55cd394756a0 2024-03-04T09:28:07.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.168+0000 7f2b9e000640 10 client.6015 handle_command_reply: tid=0 2024-03-04T09:28:07.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.168+0000 7f2bc2637740 1 client.6015 shutdown 2024-03-04T09:28:07.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.168+0000 7f2bc2637740 1 -- 172.21.15.6:0/4026004852 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55cd39449cf0 msgr2=0x55cd3945bf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:07.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.168+0000 7f2bc2637740 1 --2- 172.21.15.6:0/4026004852 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55cd39449cf0 0x55cd3945bf80 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f2bb0036040 tx=0x7f2bb0004ab0 comp rx=0 tx=0).stop 2024-03-04T09:28:07.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.168+0000 7f2bc2637740 1 -- 172.21.15.6:0/4026004852 shutdown_connections 2024-03-04T09:28:07.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.168+0000 7f2bc2637740 1 --2- 172.21.15.6:0/4026004852 >> [v2:172.21.15.146:6834/527855083,v1:172.21.15.146:6836/527855083] conn(0x55cd394756a0 0x55cd3945aec0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:07.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.168+0000 7f2bc2637740 1 --2- 172.21.15.6:0/4026004852 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55cd39449cf0 0x55cd3945bf80 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:07.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.168+0000 7f2bc2637740 1 --2- 172.21.15.6:0/4026004852 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55cd39443d70 0x55cd394497b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:07.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.168+0000 7f2bc2637740 1 --2- 172.21.15.6:0/4026004852 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55cd39443990 0x55cd39434ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:07.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.168+0000 7f2b9e000640 0 client.6015 ms_handle_reset on v2:172.21.15.146:6834/527855083 2024-03-04T09:28:07.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.168+0000 7f2bc2637740 1 -- 172.21.15.6:0/4026004852 >> 172.21.15.6:0/4026004852 conn(0x55cd39416e70 msgr2=0x55cd39446220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:07.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.168+0000 7f2bc2637740 1 -- 172.21.15.6:0/4026004852 shutdown_connections 2024-03-04T09:28:07.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.168+0000 7f2bc2637740 1 -- 172.21.15.6:0/4026004852 wait complete. 2024-03-04T09:28:07.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.168+0000 7f2bc2637740 20 client.6015 trim_cache size 0 max 16384 2024-03-04T09:28:07.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.187+0000 7f2bc2637740 1 Processor -- start 2024-03-04T09:28:07.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.187+0000 7f2bc2637740 1 -- start start 2024-03-04T09:28:07.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.188+0000 7f2bc2637740 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55cd39443990 0x55cd39411b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:07.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.188+0000 7f2bc2637740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55cd39443d70 0x55cd394159b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:07.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.188+0000 7f2bc2637740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55cd39449cf0 0x55cd394463b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:07.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.188+0000 7f2bc2637740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bb81f54e0 con 0x55cd39443d70 2024-03-04T09:28:07.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.188+0000 7f2bc2637740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bb8151cd0 con 0x55cd39449cf0 2024-03-04T09:28:07.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.188+0000 7f2bc2637740 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bb81f4ee0 con 0x55cd39443990 2024-03-04T09:28:07.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.188+0000 7f2bb7e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55cd39443d70 0x55cd394159b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:07.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.188+0000 7f2bb7e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55cd39443d70 0x55cd394159b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45428/0 (socket says 172.21.15.6:45428) 2024-03-04T09:28:07.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.188+0000 7f2bb7e00640 1 -- 172.21.15.6:0/2287048199 learned_addr learned my addr 172.21.15.6:0/2287048199 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:28:07.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.189+0000 7f2bb7e00640 1 -- 172.21.15.6:0/2287048199 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55cd39443990 msgr2=0x55cd39411b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:07.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.189+0000 7f2bb7e00640 1 --2- 172.21.15.6:0/2287048199 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55cd39443990 0x55cd39411b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:07.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.189+0000 7f2bb7e00640 1 -- 172.21.15.6:0/2287048199 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55cd39449cf0 msgr2=0x55cd394463b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:07.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.189+0000 7f2bb7e00640 1 --2- 172.21.15.6:0/2287048199 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55cd39449cf0 0x55cd394463b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:07.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.189+0000 7f2bb7e00640 1 -- 172.21.15.6:0/2287048199 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cd3941b120 con 0x55cd39443d70 2024-03-04T09:28:07.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.189+0000 7f2bb7e00640 1 --2- 172.21.15.6:0/2287048199 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55cd39443d70 0x55cd394159b0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f2ba4058530 tx=0x7f2ba406e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fce8d80b41e02b62 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:07.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.189+0000 7f2ba8c00640 1 -- 172.21.15.6:0/2287048199 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ba4087070 con 0x55cd39443d70 2024-03-04T09:28:07.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.189+0000 7f2ba8c00640 1 -- 172.21.15.6:0/2287048199 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ba4056b30 con 0x55cd39443d70 2024-03-04T09:28:07.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.189+0000 7f2ba8c00640 1 -- 172.21.15.6:0/2287048199 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ba401e040 con 0x55cd39443d70 2024-03-04T09:28:07.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.190+0000 7f2bc2637740 1 -- 172.21.15.6:0/2287048199 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55cd39443d70 msgr2=0x55cd394159b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:07.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.190+0000 7f2bc2637740 1 --2- 172.21.15.6:0/2287048199 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55cd39443d70 0x55cd394159b0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f2ba4058530 tx=0x7f2ba406e000 comp rx=0 tx=0).stop 2024-03-04T09:28:07.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.190+0000 7f2bc2637740 1 -- 172.21.15.6:0/2287048199 shutdown_connections 2024-03-04T09:28:07.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.190+0000 7f2bc2637740 1 --2- 172.21.15.6:0/2287048199 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55cd39449cf0 0x55cd394463b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:07.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.190+0000 7f2bc2637740 1 --2- 172.21.15.6:0/2287048199 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55cd39443d70 0x55cd394159b0 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:07.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.190+0000 7f2bc2637740 1 --2- 172.21.15.6:0/2287048199 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55cd39443990 0x55cd39411b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:07.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.190+0000 7f2bc2637740 1 -- 172.21.15.6:0/2287048199 >> 172.21.15.6:0/2287048199 conn(0x55cd39416e70 msgr2=0x55cd3941a7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:07.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.190+0000 7f2bc2637740 1 -- 172.21.15.6:0/2287048199 shutdown_connections 2024-03-04T09:28:07.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.190+0000 7f2bc2637740 1 -- 172.21.15.6:0/2287048199 wait complete. 2024-03-04T09:28:07.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.191+0000 7f2bc2637740 1 -- start start 2024-03-04T09:28:07.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.191+0000 7f2bc2637740 1 Processor -- start 2024-03-04T09:28:07.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.192+0000 7f2bc2637740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55cd39443990 0x55cd394cb550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:07.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.192+0000 7f2bc2637740 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55cd39443d70 0x55cd39462c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:07.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.192+0000 7f2bc2637740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55cd39449cf0 0x55cd3941b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:07.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.192+0000 7f2bc2637740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd39475520 con 0x55cd39449cf0 2024-03-04T09:28:07.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.192+0000 7f2bc2637740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd3945a320 con 0x55cd39443990 2024-03-04T09:28:07.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.192+0000 7f2bc2637740 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x55cd39475b20 con 0x55cd39443d70 2024-03-04T09:28:07.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.192+0000 7f2bc2637740 10 client.0 _ll_register_callbacks cb 0x7f2bb0009d50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T09:28:07.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.192+0000 7f2bbda00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55cd39449cf0 0x55cd3941b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:07.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.192+0000 7f2bbda00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55cd39449cf0 0x55cd3941b2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45432/0 (socket says 172.21.15.6:45432) 2024-03-04T09:28:07.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.192+0000 7f2bbda00640 1 -- 172.21.15.6:0/2177869483 learned_addr learned my addr 172.21.15.6:0/2177869483 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:28:07.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.193+0000 7f2bb7e00640 1 --2- 172.21.15.6:0/2177869483 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55cd39443d70 0x55cd39462c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:07.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.193+0000 7f2bbda00640 1 -- 172.21.15.6:0/2177869483 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55cd39443d70 msgr2=0x55cd39462c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:07.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.193+0000 7f2bbd000640 1 --2- 172.21.15.6:0/2177869483 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55cd39443990 0x55cd394cb550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:07.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.193+0000 7f2bbda00640 1 --2- 172.21.15.6:0/2177869483 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55cd39443d70 0x55cd39462c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:07.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.193+0000 7f2bbda00640 1 -- 172.21.15.6:0/2177869483 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55cd39443990 msgr2=0x55cd394cb550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:07.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.193+0000 7f2bbda00640 1 --2- 172.21.15.6:0/2177869483 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55cd39443990 0x55cd394cb550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:07.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.193+0000 7f2bbda00640 1 -- 172.21.15.6:0/2177869483 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55cd39459510 con 0x55cd39449cf0 2024-03-04T09:28:07.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.193+0000 7f2bbda00640 1 --2- 172.21.15.6:0/2177869483 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55cd39449cf0 0x55cd3941b2f0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f2bb0035f20 tx=0x7f2bb0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fceb625a8f62d4e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:07.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.193+0000 7f2b9e000640 10 client.0 ms_handle_connect on v2:172.21.15.6:3300/0 2024-03-04T09:28:07.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.193+0000 7f2b9e000640 1 -- 172.21.15.6:0/2177869483 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bb0057030 con 0x55cd39449cf0 2024-03-04T09:28:07.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.193+0000 7f2b9e000640 1 -- 172.21.15.6:0/2177869483 <== mon.0 v2:172.21.15.6:3300/0 2 ==== osd_map(56..56 src has 1..56) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2bb0003990 con 0x55cd39449cf0 2024-03-04T09:28:07.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.193+0000 7f2bc2637740 1 -- 172.21.15.6:0/2177869483 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2ba4056b30 con 0x55cd39449cf0 2024-03-04T09:28:07.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.194+0000 7f2b9e000640 1 -- 172.21.15.6:0/2177869483 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2bb0009510 con 0x55cd39449cf0 2024-03-04T09:28:07.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.194+0000 7f2bc2637740 10 client.6021 fetch_fsmap learned FSMap version 34 2024-03-04T09:28:07.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.194+0000 7f2bc2637740 1 -- 172.21.15.6:0/2177869483 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=2+}) v3 -- 0x55cd39463150 con 0x55cd39449cf0 2024-03-04T09:28:07.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.194+0000 7f2b9e000640 1 -- 172.21.15.6:0/2177869483 <== mon.0 v2:172.21.15.6:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ba4056b30 con 0x55cd39449cf0 2024-03-04T09:28:07.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.194+0000 7f2b9e000640 1 -- 172.21.15.6:0/2177869483 <== mon.0 v2:172.21.15.6:3300/0 5 ==== fsmap(e 34) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f2bb0008050 con 0x55cd39449cf0 2024-03-04T09:28:07.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.194+0000 7f2bc2637740 10 client.6021 fetch_fsmap finished waiting for FSMap version 34 2024-03-04T09:28:07.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.194+0000 7f2bc2637740 10 client.6021 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-04T09:28:07.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.194+0000 7f2bc2637740 20 client.6021 populate_metadata read hostname 'smithi006' 2024-03-04T09:28:07.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.194+0000 7f2bc2637740 1 --2- 172.21.15.6:0/2177869483 >> [v2:172.21.15.146:6834/527855083,v1:172.21.15.146:6836/527855083] conn(0x55cd39448cb0 0x55cd3946fe40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:07.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.194+0000 7f2bc2637740 4 client.6021 mds_command: new command op to 5326 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-04T09:28:07.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.194+0000 7f2bc2637740 1 -- 172.21.15.6:0/2177869483 --> [v2:172.21.15.146:6834/527855083,v1:172.21.15.146:6836/527855083] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55cd39387e90 con 0x55cd39448cb0 2024-03-04T09:28:07.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.195+0000 7f2bbd000640 1 --2- 172.21.15.6:0/2177869483 >> [v2:172.21.15.146:6834/527855083,v1:172.21.15.146:6836/527855083] conn(0x55cd39448cb0 0x55cd3946fe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:07.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.196+0000 7f2bbd000640 1 --2- 172.21.15.6:0/2177869483 >> [v2:172.21.15.146:6834/527855083,v1:172.21.15.146:6836/527855083] conn(0x55cd39448cb0 0x55cd3946fe40 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=63a178c86d7cf040 server_cookie=164494be87f0d30f in_seq=0 out_seq=0 2024-03-04T09:28:07.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.196+0000 7f2b9e000640 10 client.6021 ms_handle_connect on v2:172.21.15.146:6834/527855083 2024-03-04T09:28:07.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.196+0000 7f2b9e000640 1 -- 172.21.15.6:0/2177869483 <== mds.0 v2:172.21.15.146:6834/527855083 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x55cd39387e90 con 0x55cd39448cb0 2024-03-04T09:28:07.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.196+0000 7f2b9e000640 10 client.6021 handle_command_reply: tid=0 2024-03-04T09:28:07.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.196+0000 7f2bc2637740 1 client.6021 shutdown 2024-03-04T09:28:07.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.196+0000 7f2bc2637740 1 -- 172.21.15.6:0/2177869483 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55cd39449cf0 msgr2=0x55cd3941b2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:07.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.196+0000 7f2bc2637740 1 --2- 172.21.15.6:0/2177869483 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55cd39449cf0 0x55cd3941b2f0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f2bb0035f20 tx=0x7f2bb0004a40 comp rx=0 tx=0).stop 2024-03-04T09:28:07.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.197+0000 7f2bc2637740 1 -- 172.21.15.6:0/2177869483 shutdown_connections 2024-03-04T09:28:07.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.197+0000 7f2bc2637740 1 --2- 172.21.15.6:0/2177869483 >> [v2:172.21.15.146:6834/527855083,v1:172.21.15.146:6836/527855083] conn(0x55cd39448cb0 0x55cd3946fe40 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:07.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.197+0000 7f2bc2637740 1 --2- 172.21.15.6:0/2177869483 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55cd39449cf0 0x55cd3941b2f0 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:07.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.197+0000 7f2bc2637740 1 --2- 172.21.15.6:0/2177869483 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55cd39443d70 0x55cd39462c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:07.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.197+0000 7f2bc2637740 1 --2- 172.21.15.6:0/2177869483 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55cd39443990 0x55cd394cb550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:07.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.197+0000 7f2b9e000640 0 client.6021 ms_handle_reset on v2:172.21.15.146:6834/527855083 2024-03-04T09:28:07.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.197+0000 7f2bc2637740 1 -- 172.21.15.6:0/2177869483 >> 172.21.15.6:0/2177869483 conn(0x55cd39416e70 msgr2=0x55cd3945d060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:07.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.197+0000 7f2bc2637740 1 -- 172.21.15.6:0/2177869483 shutdown_connections 2024-03-04T09:28:07.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.197+0000 7f2bc2637740 1 -- 172.21.15.6:0/2177869483 wait complete. 2024-03-04T09:28:07.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.197+0000 7f2bc2637740 20 client.6021 trim_cache size 0 max 16384 2024-03-04T09:28:07.197 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-04T09:28:07.197 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_fsid": "edce0578-5ae4-4402-b45a-552bdef84d7c", 2024-03-04T09:28:07.197 INFO:teuthology.orchestra.run.smithi006.stdout: "whoami": 0, 2024-03-04T09:28:07.197 INFO:teuthology.orchestra.run.smithi006.stdout: "id": 5326, 2024-03-04T09:28:07.197 INFO:teuthology.orchestra.run.smithi006.stdout: "want_state": "up:active", 2024-03-04T09:28:07.197 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "up:active", 2024-03-04T09:28:07.197 INFO:teuthology.orchestra.run.smithi006.stdout: "fs_name": "cephfs", 2024-03-04T09:28:07.197 INFO:teuthology.orchestra.run.smithi006.stdout: "rank_uptime": 6.7708082789999997, 2024-03-04T09:28:07.198 INFO:teuthology.orchestra.run.smithi006.stdout: "mdsmap_epoch": 34, 2024-03-04T09:28:07.198 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap_epoch": 56, 2024-03-04T09:28:07.198 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap_epoch_barrier": 56, 2024-03-04T09:28:07.198 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 13.122312604999999 2024-03-04T09:28:07.198 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-04T09:28:07.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.197+0000 7f2bbe400640 1 -- 172.21.15.6:0/3855058261 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2b9806b4d0 msgr2=0x7f2b9806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:07.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.197+0000 7f2bbe400640 1 --2- 172.21.15.6:0/3855058261 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2b9806b4d0 0x7f2b9806d990 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f2bac006fd0 tx=0x7f2bac008040 comp rx=0 tx=0).stop 2024-03-04T09:28:07.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.197+0000 7f2bbe400640 1 -- 172.21.15.6:0/3855058261 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2bb81552b0 msgr2=0x7f2bb81e4630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:07.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.197+0000 7f2bbe400640 1 --2- 172.21.15.6:0/3855058261 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2bb81552b0 0x7f2bb81e4630 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f2ba400c820 tx=0x7f2ba400cce0 comp rx=0 tx=0).stop 2024-03-04T09:28:07.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.198+0000 7f2bbe400640 1 -- 172.21.15.6:0/3855058261 shutdown_connections 2024-03-04T09:28:07.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.198+0000 7f2bbe400640 1 --2- 172.21.15.6:0/3855058261 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2bb815e810 0x7f2bb8143f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:07.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.198+0000 7f2bbe400640 1 --2- 172.21.15.6:0/3855058261 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2bb81552b0 0x7f2bb81e4630 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:07.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.198+0000 7f2bbe400640 1 --2- 172.21.15.6:0/3855058261 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2b9806b4d0 0x7f2b9806d990 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:07.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.198+0000 7f2bbe400640 1 --2- 172.21.15.6:0/3855058261 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2bb80cce90 0x7f2bb80b1a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:07.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.198+0000 7f2bbe400640 1 -- 172.21.15.6:0/3855058261 >> 172.21.15.6:0/3855058261 conn(0x7f2bb806b3d0 msgr2=0x7f2bb80ce6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:07.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.198+0000 7f2bbe400640 1 -- 172.21.15.6:0/3855058261 shutdown_connections 2024-03-04T09:28:07.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.198+0000 7f2bbe400640 1 -- 172.21.15.6:0/3855058261 wait complete. 2024-03-04T09:28:07.213 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-04T09:28:07.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.427+0000 7f501b000640 1 Processor -- start 2024-03-04T09:28:07.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.427+0000 7f501b000640 1 -- start start 2024-03-04T09:28:07.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.428+0000 7f501b000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5014145300 0x7f50140b49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:07.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.428+0000 7f501b000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f50140cd060 0x7f501415c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:07.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.428+0000 7f501b000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50140714d0 con 0x7f5014070f10 2024-03-04T09:28:07.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.428+0000 7f501b000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50140717b0 con 0x7f50140cd060 2024-03-04T09:28:07.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.428+0000 7f501b000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5014071a90 con 0x7f5014145300 2024-03-04T09:28:07.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.429+0000 7f5019200640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f5014070f10 0x7f50140cc950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:39836/0 (socket says 172.21.15.6:39836) 2024-03-04T09:28:07.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.429+0000 7f5019200640 1 -- 172.21.15.6:0/1935034822 learned_addr learned my addr 172.21.15.6:0/1935034822 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:28:07.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.429+0000 7f5019c00640 1 --2- 172.21.15.6:0/1935034822 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5014145300 0x7f50140b49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:07.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.429+0000 7f501a600640 1 --2- 172.21.15.6:0/1935034822 >> v2:172.21.15.146:3300/0 conn(0x7f50140cd060 0x7f501415c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:07.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.430+0000 7f5013e00640 1 -- 172.21.15.6:0/1935034822 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 281235805 0 0) 0x7f50140714d0 con 0x7f5014070f10 2024-03-04T09:28:07.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.430+0000 7f5013e00640 1 -- 172.21.15.6:0/1935034822 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ffc003520 con 0x7f5014070f10 2024-03-04T09:28:07.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.430+0000 7f501a600640 1 -- 172.21.15.6:0/1935034822 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5014145300 msgr2=0x7f50140b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:07.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.430+0000 7f501a600640 1 --2- 172.21.15.6:0/1935034822 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5014145300 0x7f50140b49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:07.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.430+0000 7f501a600640 1 -- 172.21.15.6:0/1935034822 >> v1:172.21.15.6:6789/0 conn(0x7f5014070f10 legacy=0x7f50140cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:07.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.430+0000 7f501a600640 1 -- 172.21.15.6:0/1935034822 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5014071d70 con 0x7f50140cd060 2024-03-04T09:28:07.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.430+0000 7f5019c00640 1 --2- 172.21.15.6:0/1935034822 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5014145300 0x7f50140b49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T09:28:07.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.431+0000 7f501a600640 1 --2- 172.21.15.6:0/1935034822 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f50140cd060 0x7f501415c570 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f500c009d90 tx=0x7f500c02beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ac538d101158b494 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:07.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.432+0000 7f5013e00640 1 -- 172.21.15.6:0/1935034822 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f500c039070 con 0x7f50140cd060 2024-03-04T09:28:07.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.432+0000 7f5013e00640 1 -- 172.21.15.6:0/1935034822 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f500c02cc60 con 0x7f50140cd060 2024-03-04T09:28:07.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.432+0000 7f5013e00640 1 -- 172.21.15.6:0/1935034822 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f500c034660 con 0x7f50140cd060 2024-03-04T09:28:07.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.432+0000 7f501b000640 1 -- 172.21.15.6:0/1935034822 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f50140cd060 msgr2=0x7f501415c570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:07.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.432+0000 7f501b000640 1 --2- 172.21.15.6:0/1935034822 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f50140cd060 0x7f501415c570 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f500c009d90 tx=0x7f500c02beb0 comp rx=0 tx=0).stop 2024-03-04T09:28:07.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.433+0000 7f501b000640 1 -- 172.21.15.6:0/1935034822 shutdown_connections 2024-03-04T09:28:07.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.433+0000 7f501b000640 1 --2- 172.21.15.6:0/1935034822 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f50140cd060 0x7f501415c570 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:07.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.433+0000 7f501b000640 1 --2- 172.21.15.6:0/1935034822 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5014145300 0x7f50140b49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:07.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.433+0000 7f501b000640 1 -- 172.21.15.6:0/1935034822 >> 172.21.15.6:0/1935034822 conn(0x7f501406b3d0 msgr2=0x7f501406b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:07.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.433+0000 7f501b000640 1 -- 172.21.15.6:0/1935034822 shutdown_connections 2024-03-04T09:28:07.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.433+0000 7f501b000640 1 -- 172.21.15.6:0/1935034822 wait complete. 2024-03-04T09:28:07.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.434+0000 7f501b000640 1 Processor -- start 2024-03-04T09:28:07.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.435+0000 7f501b000640 1 -- start start 2024-03-04T09:28:07.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.436+0000 7f501b000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5014070f10 0x7f50140c6640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:07.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.436+0000 7f501b000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f50140cd060 0x7f50140c9600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:07.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.436+0000 7f501b000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5014145300 0x7f5014137bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:07.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.436+0000 7f501b000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f501415cab0 con 0x7f5014145300 2024-03-04T09:28:07.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.436+0000 7f501b000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50141456e0 con 0x7f50140cd060 2024-03-04T09:28:07.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.436+0000 7f501b000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f501415cc30 con 0x7f5014070f10 2024-03-04T09:28:07.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.436+0000 7f501a600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5014145300 0x7f5014137bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:07.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.436+0000 7f501a600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5014145300 0x7f5014137bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45446/0 (socket says 172.21.15.6:45446) 2024-03-04T09:28:07.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.436+0000 7f501a600640 1 -- 172.21.15.6:0/1866384072 learned_addr learned my addr 172.21.15.6:0/1866384072 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:28:07.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.436+0000 7f5019c00640 1 --2- 172.21.15.6:0/1866384072 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5014070f10 0x7f50140c6640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:07.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.436+0000 7f5019200640 1 --2- 172.21.15.6:0/1866384072 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f50140cd060 0x7f50140c9600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:07.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.436+0000 7f501a600640 1 -- 172.21.15.6:0/1866384072 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5014070f10 msgr2=0x7f50140c6640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:07.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.436+0000 7f501a600640 1 --2- 172.21.15.6:0/1866384072 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5014070f10 0x7f50140c6640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:07.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.436+0000 7f501a600640 1 -- 172.21.15.6:0/1866384072 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f50140cd060 msgr2=0x7f50140c9600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:07.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.436+0000 7f501a600640 1 --2- 172.21.15.6:0/1866384072 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f50140cd060 0x7f50140c9600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:07.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.436+0000 7f501a600640 1 -- 172.21.15.6:0/1866384072 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50140c5410 con 0x7f5014145300 2024-03-04T09:28:07.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.437+0000 7f501a600640 1 --2- 172.21.15.6:0/1866384072 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5014145300 0x7f5014137bb0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f500c009d90 tx=0x7f500c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:07.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.437+0000 7f5011600640 1 -- 172.21.15.6:0/1866384072 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f500c039050 con 0x7f5014145300 2024-03-04T09:28:07.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.437+0000 7f5011600640 1 -- 172.21.15.6:0/1866384072 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f500c045070 con 0x7f5014145300 2024-03-04T09:28:07.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.437+0000 7f5011600640 1 -- 172.21.15.6:0/1866384072 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f500c009450 con 0x7f5014145300 2024-03-04T09:28:07.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.437+0000 7f501b000640 1 -- 172.21.15.6:0/1866384072 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50140c6be0 con 0x7f5014145300 2024-03-04T09:28:07.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.437+0000 7f501b000640 1 -- 172.21.15.6:0/1866384072 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50140c5c70 con 0x7f5014145300 2024-03-04T09:28:07.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.438+0000 7f501b000640 1 -- 172.21.15.6:0/1866384072 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f501404f330 con 0x7f5014145300 2024-03-04T09:28:07.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.439+0000 7f5011600640 1 -- 172.21.15.6:0/1866384072 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f500c03e850 con 0x7f5014145300 2024-03-04T09:28:07.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.440+0000 7f5011600640 1 --2- 172.21.15.6:0/1866384072 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f4ff406b4d0 0x7f4ff406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:07.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.440+0000 7f5011600640 1 -- 172.21.15.6:0/1866384072 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f500c0acfe0 con 0x7f5014145300 2024-03-04T09:28:07.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.441+0000 7f5019c00640 1 --2- 172.21.15.6:0/1866384072 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f4ff406b4d0 0x7f4ff406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:07.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.442+0000 7f5019c00640 1 --2- 172.21.15.6:0/1866384072 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f4ff406b4d0 0x7f4ff406d990 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f5008009a30 tx=0x7f5008008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:07.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.445+0000 7f5011600640 1 -- 172.21.15.6:0/1866384072 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f500c07bf20 con 0x7f5014145300 2024-03-04T09:28:07.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.684+0000 7f501b000640 1 -- 172.21.15.6:0/1866384072 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f50140ae930 con 0x7f5014145300 2024-03-04T09:28:07.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.684+0000 7f5011600640 1 -- 172.21.15.6:0/1866384072 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v34) v1 ==== 53+0+73 (secure 0 0 0) 0x7f500c07b820 con 0x7f5014145300 2024-03-04T09:28:07.681 INFO:teuthology.orchestra.run.smithi006.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-04T09:28:07.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.687+0000 7f501b000640 1 -- 172.21.15.6:0/1866384072 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f4ff406b4d0 msgr2=0x7f4ff406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:07.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.687+0000 7f501b000640 1 --2- 172.21.15.6:0/1866384072 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f4ff406b4d0 0x7f4ff406d990 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f5008009a30 tx=0x7f5008008040 comp rx=0 tx=0).stop 2024-03-04T09:28:07.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.687+0000 7f501b000640 1 -- 172.21.15.6:0/1866384072 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5014145300 msgr2=0x7f5014137bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:07.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.687+0000 7f501b000640 1 --2- 172.21.15.6:0/1866384072 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5014145300 0x7f5014137bb0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f500c009d90 tx=0x7f500c004a40 comp rx=0 tx=0).stop 2024-03-04T09:28:07.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.687+0000 7f501b000640 1 -- 172.21.15.6:0/1866384072 shutdown_connections 2024-03-04T09:28:07.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.687+0000 7f501b000640 1 --2- 172.21.15.6:0/1866384072 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5014145300 0x7f5014137bb0 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:07.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.687+0000 7f501b000640 1 --2- 172.21.15.6:0/1866384072 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f50140cd060 0x7f50140c9600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:07.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.687+0000 7f501b000640 1 --2- 172.21.15.6:0/1866384072 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f4ff406b4d0 0x7f4ff406d990 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:07.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.687+0000 7f501b000640 1 --2- 172.21.15.6:0/1866384072 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5014070f10 0x7f50140c6640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:07.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.687+0000 7f501b000640 1 -- 172.21.15.6:0/1866384072 >> 172.21.15.6:0/1866384072 conn(0x7f501406b3d0 msgr2=0x7f5014073f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:07.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.687+0000 7f501b000640 1 -- 172.21.15.6:0/1866384072 shutdown_connections 2024-03-04T09:28:07.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:07.687+0000 7f501b000640 1 -- 172.21.15.6:0/1866384072 wait complete. 2024-03-04T09:28:07.704 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-04T09:28:07.704 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-04T09:28:07.704 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi006.front.sepia.ceph.com 2024-03-04T09:28:07.704 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-04T09:28:07.704 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-04T09:28:07.704 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-04T09:28:07.704 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-03-04T09:28:07.704 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-04T09:28:07.704 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-04T09:28:07.704 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:28:07.705 DEBUG:teuthology.orchestra.run.smithi006:> ip addr 2024-03-04T09:28:07.756 INFO:teuthology.orchestra.run.smithi006.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-04T09:28:07.757 INFO:teuthology.orchestra.run.smithi006.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-04T09:28:07.757 INFO:teuthology.orchestra.run.smithi006.stdout: inet 127.0.0.1/8 scope host lo 2024-03-04T09:28:07.757 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-04T09:28:07.757 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 ::1/128 scope host 2024-03-04T09:28:07.757 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-04T09:28:07.757 INFO:teuthology.orchestra.run.smithi006.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-04T09:28:07.757 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:6b:fe:e6 brd ff:ff:ff:ff:ff:ff 2024-03-04T09:28:07.757 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp7s0f0 2024-03-04T09:28:07.757 INFO:teuthology.orchestra.run.smithi006.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-04T09:28:07.757 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:6b:fe:e7 brd ff:ff:ff:ff:ff:ff 2024-03-04T09:28:07.757 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp7s0f1 2024-03-04T09:28:07.757 INFO:teuthology.orchestra.run.smithi006.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-04T09:28:07.757 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:bd:15:e4 brd ff:ff:ff:ff:ff:ff 2024-03-04T09:28:07.758 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp1s0f0 2024-03-04T09:28:07.758 INFO:teuthology.orchestra.run.smithi006.stdout: inet 172.21.15.6/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-04T09:28:07.758 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft 42446sec preferred_lft 42446sec 2024-03-04T09:28:07.758 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::ec4:7aff:febd:15e4/64 scope link noprefixroute 2024-03-04T09:28:07.758 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-04T09:28:07.758 INFO:teuthology.orchestra.run.smithi006.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-04T09:28:07.758 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:bd:15:e5 brd ff:ff:ff:ff:ff:ff 2024-03-04T09:28:07.758 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp1s0f1 2024-03-04T09:28:07.758 INFO:teuthology.orchestra.run.smithi006.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-04T09:28:07.758 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 5e:b4:59:dc:42:58 brd ff:ff:ff:ff:ff:ff 2024-03-04T09:28:07.758 INFO:teuthology.orchestra.run.smithi006.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-04T09:28:07.758 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-04T09:28:07.758 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::c6f:adff:fefe:e00d/64 scope link proto kernel_ll 2024-03-04T09:28:07.758 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-04T09:28:07.759 INFO:teuthology.orchestra.run.smithi006.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-04T09:28:07.759 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 7e:67:6d:ea:f9:ff brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-04T09:28:07.759 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::7c67:6dff:feea:f9ff/64 scope link proto kernel_ll 2024-03-04T09:28:07.759 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-04T09:28:07.759 INFO:teuthology.orchestra.run.smithi006.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-04T09:28:07.759 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 5e:b4:59:dc:42:58 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-04T09:28:07.759 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::5cb4:59ff:fedc:4258/64 scope link proto kernel_ll 2024-03-04T09:28:07.759 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-04T09:28:07.759 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:28:07.759 DEBUG:teuthology.orchestra.run.smithi006:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-04T09:28:07.850 INFO:teuthology.orchestra.run.smithi006.stdout:1 2024-03-04T09:28:07.855 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:28:07.855 DEBUG:teuthology.orchestra.run.smithi006:> route 2024-03-04T09:28:07.908 INFO:teuthology.orchestra.run.smithi006.stdout:Kernel IP routing table 2024-03-04T09:28:07.908 INFO:teuthology.orchestra.run.smithi006.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-04T09:28:07.908 INFO:teuthology.orchestra.run.smithi006.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-04T09:28:07.908 INFO:teuthology.orchestra.run.smithi006.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-04T09:28:07.908 INFO:teuthology.orchestra.run.smithi006.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-04T09:28:07.909 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:28:07.909 DEBUG:teuthology.orchestra.run.smithi006:> (cd / && exec bash -c ' 2024-03-04T09:28:07.909 DEBUG:teuthology.orchestra.run.smithi006:> set -e 2024-03-04T09:28:07.909 DEBUG:teuthology.orchestra.run.smithi006:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-04T09:28:07.909 DEBUG:teuthology.orchestra.run.smithi006:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-04T09:28:07.909 DEBUG:teuthology.orchestra.run.smithi006:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-04T09:28:07.909 DEBUG:teuthology.orchestra.run.smithi006:> ') 2024-03-04T09:28:08.106 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:28:08.106 DEBUG:teuthology.orchestra.run.smithi006:> ip netns list 2024-03-04T09:28:08.160 INFO:teuthology.orchestra.run.smithi006.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-04T09:28:08.161 INFO:teuthology.orchestra.run.smithi006.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-04T09:28:08.161 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-04T09:28:08.162 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:28:08.162 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip link set brx.0 up 2024-03-04T09:28:08.260 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-04T09:28:08.261 DEBUG:teuthology.orchestra.run.smithi006:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-04T09:28:08.317 INFO:teuthology.orchestra.run.smithi006.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-04T09:28:08.317 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-04T09:28:08.317 DEBUG:teuthology.orchestra.run.smithi006:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-04T09:28:08.373 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-03-04T09:28:08.373 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:28:08.373 DEBUG:teuthology.orchestra.run.smithi006:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-04T09:28:08.525 INFO:teuthology.orchestra.run.smithi006.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-04T09:28:08.680 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-04T09:28:08.681 INFO:teuthology.orchestra.run.smithi006.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-04T09:28:08.682 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-04T09:28:08.682 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: resolved to: "172.21.15.6:3300,172.21.15.146:3300,172.21.15.146:3301" 2024-03-04T09:28:08.682 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: trying mount with new device syntax: 0@edce0578-5ae4-4402-b45a-552bdef84d7c.cephfs=/ 2024-03-04T09:28:08.682 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.6:3300/172.21.15.146:3300/172.21.15.146:3301" will pass to kernel 2024-03-04T09:28:08.682 INFO:teuthology.orchestra.run.smithi006.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-04T09:28:08.682 INFO:teuthology.orchestra.run.smithi006.stdout: You just mounted a file system that supports labels which does not 2024-03-04T09:28:08.682 INFO:teuthology.orchestra.run.smithi006.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-04T09:28:08.682 INFO:teuthology.orchestra.run.smithi006.stdout: applications will generate AVC messages and not be allowed access to 2024-03-04T09:28:08.682 INFO:teuthology.orchestra.run.smithi006.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-04T09:28:08.682 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-04T09:28:08.682 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:28:08.683 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-04T09:28:08.784 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:28:08.784 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-04T09:28:08.867 INFO:teuthology.orchestra.run.smithi006.stdout:client5419 2024-03-04T09:28:08.867 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-04T09:28:08.867 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T09:28:08.927 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds mds_bal_fragment_size_max 50 2024-03-04T09:28:09.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.172+0000 7f49c2600640 1 Processor -- start 2024-03-04T09:28:09.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.172+0000 7f49c2600640 1 -- start start 2024-03-04T09:28:09.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.173+0000 7f49c2600640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f49bc15e280 0x7f49bc0d1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:09.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.173+0000 7f49c2600640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49bc0d0080 con 0x7f49bc0d2990 2024-03-04T09:28:09.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.173+0000 7f49c2600640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49bc0cf500 con 0x7f49bc0d4a00 2024-03-04T09:28:09.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.173+0000 7f49c2600640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49bc0cf7e0 con 0x7f49bc15e280 2024-03-04T09:28:09.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.174+0000 7f49bbe00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f49bc0d2990 0x7f49bc0cb600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:42884/0 (socket says 172.21.15.6:42884) 2024-03-04T09:28:09.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.174+0000 7f49bbe00640 1 -- 172.21.15.6:0/2748489866 learned_addr learned my addr 172.21.15.6:0/2748489866 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:28:09.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.174+0000 7f49bb400640 1 --2- 172.21.15.6:0/2748489866 >> v2:172.21.15.146:3300/0 conn(0x7f49bc15e280 0x7f49bc0d1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:09.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.174+0000 7f49ba000640 1 -- 172.21.15.6:0/2748489866 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 640145776 0 0) 0x7f49bc0d0080 con 0x7f49bc0d2990 2024-03-04T09:28:09.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.174+0000 7f49ba000640 1 -- 172.21.15.6:0/2748489866 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49a0003520 con 0x7f49bc0d2990 2024-03-04T09:28:09.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.175+0000 7f49ba000640 1 -- 172.21.15.6:0/2748489866 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2189008402 0 0) 0x7f49a0003520 con 0x7f49bc0d2990 2024-03-04T09:28:09.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.175+0000 7f49ba000640 1 -- 172.21.15.6:0/2748489866 >> v2:172.21.15.146:3300/0 conn(0x7f49bc15e280 msgr2=0x7f49bc0d1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:09.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.175+0000 7f49ba000640 1 --2- 172.21.15.6:0/2748489866 >> v2:172.21.15.146:3300/0 conn(0x7f49bc15e280 0x7f49bc0d1cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:09.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.175+0000 7f49ba000640 1 -- 172.21.15.6:0/2748489866 >> v1:172.21.15.146:6789/0 conn(0x7f49bc0d4a00 legacy=0x7f49bc0d2280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:09.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.175+0000 7f49ba000640 1 -- 172.21.15.6:0/2748489866 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49bc0cfac0 con 0x7f49bc0d2990 2024-03-04T09:28:09.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.175+0000 7f49ba000640 1 -- 172.21.15.6:0/2748489866 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7f49b4001670 con 0x7f49bc0d2990 2024-03-04T09:28:09.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.175+0000 7f49ba000640 1 -- 172.21.15.6:0/2748489866 >> v1:172.21.15.6:6789/0 conn(0x7f49bc0d2990 legacy=0x7f49bc0cb600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:09.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.175+0000 7f49ba000640 1 --2- 172.21.15.6:0/2748489866 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f49a00039b0 0x7f49a0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:09.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.175+0000 7f49ba000640 1 --2- 172.21.15.6:0/2748489866 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f49a0004800 0x7f49a0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:09.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.175+0000 7f49ba000640 1 --2- 172.21.15.6:0/2748489866 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f49a00051c0 0x7f49a000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:09.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.175+0000 7f49ba000640 1 -- 172.21.15.6:0/2748489866 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49bc0cf7e0 con 0x7f49a00039b0 2024-03-04T09:28:09.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.175+0000 7f49ba000640 1 -- 172.21.15.6:0/2748489866 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49a000c160 con 0x7f49a0004800 2024-03-04T09:28:09.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.175+0000 7f49ba000640 1 -- 172.21.15.6:0/2748489866 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49a000c360 con 0x7f49a00051c0 2024-03-04T09:28:09.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.175+0000 7f49bb400640 1 --2- 172.21.15.6:0/2748489866 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f49a00039b0 0x7f49a0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:09.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.176+0000 7f49bb400640 1 -- 172.21.15.6:0/2748489866 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f49a00051c0 msgr2=0x7f49a000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:09.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.176+0000 7f49bb400640 1 --2- 172.21.15.6:0/2748489866 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f49a00051c0 0x7f49a000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:09.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.176+0000 7f49baa00640 1 --2- 172.21.15.6:0/2748489866 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f49a0004800 0x7f49a0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:09.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.176+0000 7f49bb400640 1 -- 172.21.15.6:0/2748489866 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f49a0004800 msgr2=0x7f49a0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:09.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.176+0000 7f49bb400640 1 --2- 172.21.15.6:0/2748489866 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f49a0004800 0x7f49a0004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:09.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.176+0000 7f49bb400640 1 -- 172.21.15.6:0/2748489866 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49a0003520 con 0x7f49a00039b0 2024-03-04T09:28:09.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.176+0000 7f49bbe00640 1 -- 172.21.15.6:0/2748489866 reap_dead start 2024-03-04T09:28:09.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.177+0000 7f49bb400640 1 --2- 172.21.15.6:0/2748489866 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f49a00039b0 0x7f49a0003dd0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f49ac009fc0 tx=0x7f49a0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cab5b967966aeed6 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:09.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.177+0000 7f49ba000640 1 -- 172.21.15.6:0/2748489866 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49ac039070 con 0x7f49a00039b0 2024-03-04T09:28:09.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.177+0000 7f49ba000640 1 -- 172.21.15.6:0/2748489866 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49ac02cc20 con 0x7f49a00039b0 2024-03-04T09:28:09.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.177+0000 7f49ba000640 1 -- 172.21.15.6:0/2748489866 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49ac009c10 con 0x7f49a00039b0 2024-03-04T09:28:09.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.178+0000 7f49c2600640 1 -- 172.21.15.6:0/2748489866 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f49a00039b0 msgr2=0x7f49a0003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:09.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.178+0000 7f49c2600640 1 --2- 172.21.15.6:0/2748489866 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f49a00039b0 0x7f49a0003dd0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f49ac009fc0 tx=0x7f49a0005830 comp rx=0 tx=0).stop 2024-03-04T09:28:09.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.178+0000 7f49c2600640 1 -- 172.21.15.6:0/2748489866 shutdown_connections 2024-03-04T09:28:09.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.178+0000 7f49c2600640 1 --2- 172.21.15.6:0/2748489866 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f49a00039b0 0x7f49a0003dd0 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:09.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.178+0000 7f49c2600640 1 -- 172.21.15.6:0/2748489866 >> 172.21.15.6:0/2748489866 conn(0x7f49bc0d3ee0 msgr2=0x7f49bc0d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:09.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.178+0000 7f49c2600640 1 -- 172.21.15.6:0/2748489866 shutdown_connections 2024-03-04T09:28:09.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.178+0000 7f49c2600640 1 -- 172.21.15.6:0/2748489866 wait complete. 2024-03-04T09:28:09.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.180+0000 7f49c2600640 1 Processor -- start 2024-03-04T09:28:09.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.180+0000 7f49c2600640 1 -- start start 2024-03-04T09:28:09.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.181+0000 7f49c2600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f49a00039b0 0x7f49bc1582b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:09.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.181+0000 7f49c2600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f49bc156ff0 0x7f49bc1f78f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:09.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.181+0000 7f49c2600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f49bc1f7e30 0x7f49bc1601f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:09.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.181+0000 7f49c2600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49bc15e660 con 0x7f49bc156ff0 2024-03-04T09:28:09.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.181+0000 7f49c2600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49bc0cb0e0 con 0x7f49a00039b0 2024-03-04T09:28:09.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.181+0000 7f49c2600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49bc0d1cc0 con 0x7f49bc1f7e30 2024-03-04T09:28:09.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.181+0000 7f49baa00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f49bc156ff0 0x7f49bc1f78f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:09.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.182+0000 7f49baa00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f49bc156ff0 0x7f49bc1f78f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57318/0 (socket says 172.21.15.6:57318) 2024-03-04T09:28:09.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.182+0000 7f49baa00640 1 -- 172.21.15.6:0/413735236 learned_addr learned my addr 172.21.15.6:0/413735236 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:28:09.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.182+0000 7f49bb400640 1 --2- 172.21.15.6:0/413735236 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f49a00039b0 0x7f49bc1582b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:09.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.182+0000 7f49baa00640 1 -- 172.21.15.6:0/413735236 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f49bc1f7e30 msgr2=0x7f49bc1601f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:09.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.182+0000 7f49bbe00640 1 --2- 172.21.15.6:0/413735236 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f49bc1f7e30 0x7f49bc1601f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:09.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.182+0000 7f49baa00640 1 --2- 172.21.15.6:0/413735236 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f49bc1f7e30 0x7f49bc1601f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:09.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.182+0000 7f49baa00640 1 -- 172.21.15.6:0/413735236 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f49a00039b0 msgr2=0x7f49bc1582b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:09.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.182+0000 7f49baa00640 1 --2- 172.21.15.6:0/413735236 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f49a00039b0 0x7f49bc1582b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:09.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.182+0000 7f49baa00640 1 -- 172.21.15.6:0/413735236 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49bc157a80 con 0x7f49bc156ff0 2024-03-04T09:28:09.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.182+0000 7f49bbe00640 1 --2- 172.21.15.6:0/413735236 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f49bc1f7e30 0x7f49bc1601f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:28:09.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.182+0000 7f49baa00640 1 --2- 172.21.15.6:0/413735236 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f49bc156ff0 0x7f49bc1f78f0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f49a400d400 tx=0x7f49a400d8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:09.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.183+0000 7f49b3400640 1 -- 172.21.15.6:0/413735236 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49a4018070 con 0x7f49bc156ff0 2024-03-04T09:28:09.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.183+0000 7f49b3400640 1 -- 172.21.15.6:0/413735236 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49a4003610 con 0x7f49bc156ff0 2024-03-04T09:28:09.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.183+0000 7f49c2600640 1 -- 172.21.15.6:0/413735236 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49bc1f8460 con 0x7f49bc156ff0 2024-03-04T09:28:09.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.183+0000 7f49b3400640 1 -- 172.21.15.6:0/413735236 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49a4003a00 con 0x7f49bc156ff0 2024-03-04T09:28:09.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.183+0000 7f49c2600640 1 -- 172.21.15.6:0/413735236 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49bc0c7ab0 con 0x7f49bc156ff0 2024-03-04T09:28:09.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.184+0000 7f49c2600640 1 -- 172.21.15.6:0/413735236 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49bc04f330 con 0x7f49bc156ff0 2024-03-04T09:28:09.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.185+0000 7f49b3400640 1 -- 172.21.15.6:0/413735236 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f49a401b070 con 0x7f49bc156ff0 2024-03-04T09:28:09.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.186+0000 7f49b3400640 1 --2- 172.21.15.6:0/413735236 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f499806b4d0 0x7f499806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:09.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.186+0000 7f49b3400640 1 -- 172.21.15.6:0/413735236 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f49a4089920 con 0x7f49bc156ff0 2024-03-04T09:28:09.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.187+0000 7f49bb400640 1 --2- 172.21.15.6:0/413735236 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f499806b4d0 0x7f499806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:09.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.188+0000 7f49bb400640 1 --2- 172.21.15.6:0/413735236 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f499806b4d0 0x7f499806d990 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f49ac036000 tx=0x7f49ac02e000 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:09.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.190+0000 7f49b3400640 1 -- 172.21.15.6:0/413735236 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f49a4058140 con 0x7f49bc156ff0 2024-03-04T09:28:09.288 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:28:09.296 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:28:09.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.391+0000 7f49c2600640 1 -- 172.21.15.6:0/413735236 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command([{prefix=config set, name=mds_bal_fragment_size_max}] v 0) v1 -- 0x7f49bc15ffd0 con 0x7f49bc156ff0 2024-03-04T09:28:09.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.396+0000 7f49b3400640 1 -- 172.21.15.6:0/413735236 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_fragment_size_max}]=0 v17)=0 v17) v1 ==== 124+0+0 (secure 0 0 0) 0x7f49a405c1e0 con 0x7f49bc156ff0 2024-03-04T09:28:09.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.399+0000 7f49c2600640 1 -- 172.21.15.6:0/413735236 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f499806b4d0 msgr2=0x7f499806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:09.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.399+0000 7f49c2600640 1 --2- 172.21.15.6:0/413735236 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f499806b4d0 0x7f499806d990 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f49ac036000 tx=0x7f49ac02e000 comp rx=0 tx=0).stop 2024-03-04T09:28:09.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.399+0000 7f49c2600640 1 -- 172.21.15.6:0/413735236 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f49bc156ff0 msgr2=0x7f49bc1f78f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:09.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.399+0000 7f49c2600640 1 --2- 172.21.15.6:0/413735236 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f49bc156ff0 0x7f49bc1f78f0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f49a400d400 tx=0x7f49a400d8c0 comp rx=0 tx=0).stop 2024-03-04T09:28:09.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.399+0000 7f49c2600640 1 -- 172.21.15.6:0/413735236 shutdown_connections 2024-03-04T09:28:09.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.399+0000 7f49c2600640 1 --2- 172.21.15.6:0/413735236 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f499806b4d0 0x7f499806d990 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:09.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.399+0000 7f49c2600640 1 --2- 172.21.15.6:0/413735236 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f49bc1f7e30 0x7f49bc1601f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:09.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.399+0000 7f49c2600640 1 --2- 172.21.15.6:0/413735236 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f49bc156ff0 0x7f49bc1f78f0 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:09.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.399+0000 7f49c2600640 1 --2- 172.21.15.6:0/413735236 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f49a00039b0 0x7f49bc1582b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:09.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.399+0000 7f49c2600640 1 -- 172.21.15.6:0/413735236 >> 172.21.15.6:0/413735236 conn(0x7f49bc0d3ee0 msgr2=0x7f49bc0cd4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:09.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.400+0000 7f49c2600640 1 -- 172.21.15.6:0/413735236 shutdown_connections 2024-03-04T09:28:09.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.400+0000 7f49c2600640 1 -- 172.21.15.6:0/413735236 wait complete. 2024-03-04T09:28:09.416 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds mds_bal_merge_size 1 2024-03-04T09:28:09.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.680+0000 7f3706800640 1 Processor -- start 2024-03-04T09:28:09.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.681+0000 7f3706800640 1 -- start start 2024-03-04T09:28:09.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.682+0000 7f3706800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f370015dee0 0x7f37000d18a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:09.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.682+0000 7f3706800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f37000d0d40 0x7f37000b3510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:09.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.682+0000 7f3706800640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37000ceca0 con 0x7f37000d4860 2024-03-04T09:28:09.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.682+0000 7f3706800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37000cf5d0 con 0x7f37000d0d40 2024-03-04T09:28:09.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.682+0000 7f3706800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37000cef80 con 0x7f370015dee0 2024-03-04T09:28:09.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.682+0000 7f3705e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f37000d0d40 0x7f37000b3510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:09.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.682+0000 7f3705e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f37000d0d40 0x7f37000b3510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57330/0 (socket says 172.21.15.6:57330) 2024-03-04T09:28:09.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.682+0000 7f3705e00640 1 -- 172.21.15.6:0/2001253386 learned_addr learned my addr 172.21.15.6:0/2001253386 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:28:09.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.683+0000 7f3705400640 1 --2- 172.21.15.6:0/2001253386 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f370015dee0 0x7f37000d18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:09.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.683+0000 7f3705e00640 1 -- 172.21.15.6:0/2001253386 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f370015dee0 msgr2=0x7f37000d18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:09.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.683+0000 7f3705e00640 1 --2- 172.21.15.6:0/2001253386 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f370015dee0 0x7f37000d18a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:09.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.683+0000 7f3705e00640 1 -- 172.21.15.6:0/2001253386 >> v1:172.21.15.146:6789/0 conn(0x7f37000d4860 legacy=0x7f37000d0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:09.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.683+0000 7f3705e00640 1 -- 172.21.15.6:0/2001253386 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37000cf260 con 0x7f37000d0d40 2024-03-04T09:28:09.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.684+0000 7f3705e00640 1 --2- 172.21.15.6:0/2001253386 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f37000d0d40 0x7f37000b3510 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f36f8009870 tx=0x7f36f802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3241b52ca6b4db36 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:09.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.684+0000 7f36ffe00640 1 -- 172.21.15.6:0/2001253386 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36f8035020 con 0x7f37000d0d40 2024-03-04T09:28:09.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.685+0000 7f36ffe00640 1 -- 172.21.15.6:0/2001253386 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36f8031640 con 0x7f37000d0d40 2024-03-04T09:28:09.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.685+0000 7f36ffe00640 1 -- 172.21.15.6:0/2001253386 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36f802d040 con 0x7f37000d0d40 2024-03-04T09:28:09.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.685+0000 7f3706800640 1 -- 172.21.15.6:0/2001253386 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f37000d0d40 msgr2=0x7f37000b3510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:09.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.685+0000 7f3706800640 1 --2- 172.21.15.6:0/2001253386 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f37000d0d40 0x7f37000b3510 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f36f8009870 tx=0x7f36f802b8f0 comp rx=0 tx=0).stop 2024-03-04T09:28:09.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.685+0000 7f3706800640 1 -- 172.21.15.6:0/2001253386 shutdown_connections 2024-03-04T09:28:09.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.685+0000 7f3706800640 1 --2- 172.21.15.6:0/2001253386 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f37000d0d40 0x7f37000b3510 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:09.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.685+0000 7f3706800640 1 --2- 172.21.15.6:0/2001253386 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f370015dee0 0x7f37000d18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:09.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.685+0000 7f3706800640 1 -- 172.21.15.6:0/2001253386 >> 172.21.15.6:0/2001253386 conn(0x7f37000d3cd0 msgr2=0x7f37000d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:09.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.685+0000 7f3706800640 1 -- 172.21.15.6:0/2001253386 shutdown_connections 2024-03-04T09:28:09.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.685+0000 7f3706800640 1 -- 172.21.15.6:0/2001253386 wait complete. 2024-03-04T09:28:09.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.687+0000 7f3706800640 1 Processor -- start 2024-03-04T09:28:09.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.687+0000 7f3706800640 1 -- start start 2024-03-04T09:28:09.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.688+0000 7f3706800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f37000d0d40 0x7f3700159d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:09.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.688+0000 7f3706800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f37000d4860 0x7f370015a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:09.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.688+0000 7f3706800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f370015dee0 0x7f37001604b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:09.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.688+0000 7f3706800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37000d1de0 con 0x7f37000d0d40 2024-03-04T09:28:09.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.688+0000 7f3706800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f370015e2c0 con 0x7f370015dee0 2024-03-04T09:28:09.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.688+0000 7f3706800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37000d1f60 con 0x7f37000d4860 2024-03-04T09:28:09.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.688+0000 7f3705400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f37000d0d40 0x7f3700159d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:09.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.688+0000 7f3705400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f37000d0d40 0x7f3700159d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57338/0 (socket says 172.21.15.6:57338) 2024-03-04T09:28:09.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.688+0000 7f3705400640 1 -- 172.21.15.6:0/4222764552 learned_addr learned my addr 172.21.15.6:0/4222764552 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:28:09.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.689+0000 7f3705400640 1 -- 172.21.15.6:0/4222764552 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f37000d4860 msgr2=0x7f370015a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:09.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.689+0000 7f3705400640 1 --2- 172.21.15.6:0/4222764552 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f37000d4860 0x7f370015a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:09.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.689+0000 7f3705400640 1 -- 172.21.15.6:0/4222764552 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f370015dee0 msgr2=0x7f37001604b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:09.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.689+0000 7f3705e00640 1 --2- 172.21.15.6:0/4222764552 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f370015dee0 0x7f37001604b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:09.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.689+0000 7f3705400640 1 --2- 172.21.15.6:0/4222764552 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f370015dee0 0x7f37001604b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:09.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.689+0000 7f3705400640 1 -- 172.21.15.6:0/4222764552 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37000c9830 con 0x7f37000d0d40 2024-03-04T09:28:09.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.689+0000 7f3705400640 1 --2- 172.21.15.6:0/4222764552 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f37000d0d40 0x7f3700159d60 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f36f000ce10 tx=0x7f36f000a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:09.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.689+0000 7f36fd600640 1 -- 172.21.15.6:0/4222764552 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36f0003b90 con 0x7f37000d0d40 2024-03-04T09:28:09.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.689+0000 7f36fd600640 1 -- 172.21.15.6:0/4222764552 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36f0003d30 con 0x7f37000d0d40 2024-03-04T09:28:09.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.689+0000 7f36fd600640 1 -- 172.21.15.6:0/4222764552 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36f0004240 con 0x7f37000d0d40 2024-03-04T09:28:09.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.690+0000 7f3706800640 1 -- 172.21.15.6:0/4222764552 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37000c9af0 con 0x7f37000d0d40 2024-03-04T09:28:09.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.690+0000 7f3706800640 1 -- 172.21.15.6:0/4222764552 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37000ca0b0 con 0x7f37000d0d40 2024-03-04T09:28:09.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.691+0000 7f3706800640 1 -- 172.21.15.6:0/4222764552 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f370004f330 con 0x7f37000d0d40 2024-03-04T09:28:09.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.692+0000 7f36fd600640 1 -- 172.21.15.6:0/4222764552 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f36f00043e0 con 0x7f37000d0d40 2024-03-04T09:28:09.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.693+0000 7f36fd600640 1 --2- 172.21.15.6:0/4222764552 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f36e006b500 0x7f36e006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:09.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.693+0000 7f36fd600640 1 -- 172.21.15.6:0/4222764552 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f36f008ad00 con 0x7f37000d0d40 2024-03-04T09:28:09.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.694+0000 7f3704a00640 1 --2- 172.21.15.6:0/4222764552 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f36e006b500 0x7f36e006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:09.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.695+0000 7f3704a00640 1 --2- 172.21.15.6:0/4222764552 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f36e006b500 0x7f36e006d9c0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f36ec006fd0 tx=0x7f36ec006bc0 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:09.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.698+0000 7f36fd600640 1 -- 172.21.15.6:0/4222764552 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f36f001d340 con 0x7f37000d0d40 2024-03-04T09:28:09.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.897+0000 7f3706800640 1 -- 172.21.15.6:0/4222764552 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command([{prefix=config set, name=mds_bal_merge_size}] v 0) v1 -- 0x7f370008d7d0 con 0x7f37000d0d40 2024-03-04T09:28:09.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.905+0000 7f36fd600640 1 -- 172.21.15.6:0/4222764552 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_merge_size}]=0 v18)=0 v18) v1 ==== 116+0+0 (secure 0 0 0) 0x7f36f0059b80 con 0x7f37000d0d40 2024-03-04T09:28:09.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.907+0000 7f3706800640 1 -- 172.21.15.6:0/4222764552 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f36e006b500 msgr2=0x7f36e006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:09.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.907+0000 7f3706800640 1 --2- 172.21.15.6:0/4222764552 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f36e006b500 0x7f36e006d9c0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f36ec006fd0 tx=0x7f36ec006bc0 comp rx=0 tx=0).stop 2024-03-04T09:28:09.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.907+0000 7f3706800640 1 -- 172.21.15.6:0/4222764552 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f37000d0d40 msgr2=0x7f3700159d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:09.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.907+0000 7f3706800640 1 --2- 172.21.15.6:0/4222764552 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f37000d0d40 0x7f3700159d60 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f36f000ce10 tx=0x7f36f000a660 comp rx=0 tx=0).stop 2024-03-04T09:28:09.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.907+0000 7f3706800640 1 -- 172.21.15.6:0/4222764552 shutdown_connections 2024-03-04T09:28:09.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.907+0000 7f3706800640 1 --2- 172.21.15.6:0/4222764552 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f370015dee0 0x7f37001604b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:09.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.907+0000 7f3706800640 1 --2- 172.21.15.6:0/4222764552 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f36e006b500 0x7f36e006d9c0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:09.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.907+0000 7f3706800640 1 --2- 172.21.15.6:0/4222764552 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f37000d4860 0x7f370015a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:09.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.907+0000 7f3706800640 1 --2- 172.21.15.6:0/4222764552 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f37000d0d40 0x7f3700159d60 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:09.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.907+0000 7f3706800640 1 -- 172.21.15.6:0/4222764552 >> 172.21.15.6:0/4222764552 conn(0x7f37000d3cd0 msgr2=0x7f370006b8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:09.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.908+0000 7f3706800640 1 -- 172.21.15.6:0/4222764552 shutdown_connections 2024-03-04T09:28:09.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:09.908+0000 7f3706800640 1 -- 172.21.15.6:0/4222764552 wait complete. 2024-03-04T09:28:19.924 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-04T09:28:19.924 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T09:28:19.986 DEBUG:teuthology.orchestra.run.smithi006:> stdin-killer -- python3 -c ' 2024-03-04T09:28:19.986 DEBUG:teuthology.orchestra.run.smithi006:> import os 2024-03-04T09:28:19.986 DEBUG:teuthology.orchestra.run.smithi006:> import uuid 2024-03-04T09:28:19.986 DEBUG:teuthology.orchestra.run.smithi006:> 2024-03-04T09:28:19.987 DEBUG:teuthology.orchestra.run.smithi006:> n = 50 2024-03-04T09:28:19.987 DEBUG:teuthology.orchestra.run.smithi006:> create_hard_links = False 2024-03-04T09:28:19.987 DEBUG:teuthology.orchestra.run.smithi006:> if 0 > 0: 2024-03-04T09:28:19.987 DEBUG:teuthology.orchestra.run.smithi006:> create_hard_links = True 2024-03-04T09:28:19.987 DEBUG:teuthology.orchestra.run.smithi006:> path = "/home/ubuntu/cephtest/mnt.0/subdir/file" 2024-03-04T09:28:19.987 DEBUG:teuthology.orchestra.run.smithi006:> 2024-03-04T09:28:19.987 DEBUG:teuthology.orchestra.run.smithi006:> dpath = os.path.dirname(path) 2024-03-04T09:28:19.987 DEBUG:teuthology.orchestra.run.smithi006:> fnameprefix = os.path.basename(path) 2024-03-04T09:28:19.987 DEBUG:teuthology.orchestra.run.smithi006:> os.makedirs(dpath, exist_ok=True) 2024-03-04T09:28:19.987 DEBUG:teuthology.orchestra.run.smithi006:> 2024-03-04T09:28:19.987 DEBUG:teuthology.orchestra.run.smithi006:> try: 2024-03-04T09:28:19.987 DEBUG:teuthology.orchestra.run.smithi006:> dirfd = os.open(dpath, os.O_DIRECTORY) 2024-03-04T09:28:19.988 DEBUG:teuthology.orchestra.run.smithi006:> 2024-03-04T09:28:19.988 DEBUG:teuthology.orchestra.run.smithi006:> for i in range(n): 2024-03-04T09:28:19.988 DEBUG:teuthology.orchestra.run.smithi006:> fpath = os.path.join(dpath, f"{fnameprefix}_{i}") 2024-03-04T09:28:19.988 DEBUG:teuthology.orchestra.run.smithi006:> with open(fpath, '"'"'w'"'"') as f: 2024-03-04T09:28:19.988 DEBUG:teuthology.orchestra.run.smithi006:> f.write(f"{i}") 2024-03-04T09:28:19.988 DEBUG:teuthology.orchestra.run.smithi006:> if False: 2024-03-04T09:28:19.988 DEBUG:teuthology.orchestra.run.smithi006:> f.flush() 2024-03-04T09:28:19.988 DEBUG:teuthology.orchestra.run.smithi006:> os.fsync(f.fileno()) 2024-03-04T09:28:19.988 DEBUG:teuthology.orchestra.run.smithi006:> if False: 2024-03-04T09:28:19.988 DEBUG:teuthology.orchestra.run.smithi006:> os.unlink(fpath) 2024-03-04T09:28:19.988 DEBUG:teuthology.orchestra.run.smithi006:> if False: 2024-03-04T09:28:19.988 DEBUG:teuthology.orchestra.run.smithi006:> os.fsync(dirfd) 2024-03-04T09:28:19.988 DEBUG:teuthology.orchestra.run.smithi006:> if create_hard_links: 2024-03-04T09:28:19.989 DEBUG:teuthology.orchestra.run.smithi006:> for j in range(0): 2024-03-04T09:28:19.989 DEBUG:teuthology.orchestra.run.smithi006:> os.system(f"ln {fpath} {dpath}/{fnameprefix}_{i}_{uuid.uuid4()}") 2024-03-04T09:28:19.989 DEBUG:teuthology.orchestra.run.smithi006:> if True: 2024-03-04T09:28:19.989 DEBUG:teuthology.orchestra.run.smithi006:> os.fsync(dirfd) 2024-03-04T09:28:19.989 DEBUG:teuthology.orchestra.run.smithi006:> finally: 2024-03-04T09:28:19.989 DEBUG:teuthology.orchestra.run.smithi006:> os.close(dirfd) 2024-03-04T09:28:19.989 DEBUG:teuthology.orchestra.run.smithi006:> ' 2024-03-04T09:28:20.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:20 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-04T09:28:20.341 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-04T09:28:20.341 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T09:28:20.440 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-04T09:28:20.441 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T09:28:20.499 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T09:28:20.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:20.728+0000 7f7403e00640 1 Processor -- start 2024-03-04T09:28:20.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:20.728+0000 7f7403e00640 1 -- start start 2024-03-04T09:28:20.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:20.730+0000 7f7403e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f7404145300 0x7f74040b49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:20.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:20.730+0000 7f7403e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f74040cd060 0x7f740415c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:20.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:20.730+0000 7f7403e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74040714d0 con 0x7f7404070f10 2024-03-04T09:28:20.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:20.730+0000 7f7403e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74040717b0 con 0x7f7404145300 2024-03-04T09:28:20.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:20.730+0000 7f7403e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7404071a90 con 0x7f74040cd060 2024-03-04T09:28:20.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:20.730+0000 7f73fa000640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f7404070f10 0x7f74040cc950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:51786/0 (socket says 172.21.15.6:51786) 2024-03-04T09:28:20.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:20.730+0000 7f73fa000640 1 -- 172.21.15.6:0/1697403537 learned_addr learned my addr 172.21.15.6:0/1697403537 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:28:20.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:20.730+0000 7f7402a00640 1 --2- 172.21.15.6:0/1697403537 >> v2:172.21.15.6:3300/0 conn(0x7f7404145300 0x7f74040b49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:20.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:20.730+0000 7f7403400640 1 --2- 172.21.15.6:0/1697403537 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f74040cd060 0x7f740415c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:20.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:20.731+0000 7f7402000640 1 -- 172.21.15.6:0/1697403537 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3666080664 0 0) 0x7f74040714d0 con 0x7f7404070f10 2024-03-04T09:28:20.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:20.731+0000 7f7402000640 1 -- 172.21.15.6:0/1697403537 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73e8003520 con 0x7f7404070f10 2024-03-04T09:28:20.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:20.731+0000 7f7402a00640 1 -- 172.21.15.6:0/1697403537 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f74040cd060 msgr2=0x7f740415c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:20.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:20.731+0000 7f7402a00640 1 --2- 172.21.15.6:0/1697403537 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f74040cd060 0x7f740415c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:20.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:20.731+0000 7f7402a00640 1 -- 172.21.15.6:0/1697403537 >> v1:172.21.15.6:6789/0 conn(0x7f7404070f10 legacy=0x7f74040cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:20.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:20.731+0000 7f7402a00640 1 -- 172.21.15.6:0/1697403537 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7404071d70 con 0x7f7404145300 2024-03-04T09:28:20.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:20.732+0000 7f7402a00640 1 --2- 172.21.15.6:0/1697403537 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7404145300 0x7f74040b49a0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f73f40097a0 tx=0x7f73f402b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=142629de5a754c23 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:20.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:20.732+0000 7f7402000640 1 -- 172.21.15.6:0/1697403537 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73f4031780 con 0x7f7404145300 2024-03-04T09:28:20.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:20.732+0000 7f7402000640 1 -- 172.21.15.6:0/1697403537 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73f4037070 con 0x7f7404145300 2024-03-04T09:28:20.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:20.732+0000 7f7402000640 1 -- 172.21.15.6:0/1697403537 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73f4031b20 con 0x7f7404145300 2024-03-04T09:28:20.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:20.733+0000 7f7403e00640 1 -- 172.21.15.6:0/1697403537 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7404145300 msgr2=0x7f74040b49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:20.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:20.733+0000 7f7403e00640 1 --2- 172.21.15.6:0/1697403537 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7404145300 0x7f74040b49a0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f73f40097a0 tx=0x7f73f402b820 comp rx=0 tx=0).stop 2024-03-04T09:28:20.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:20.733+0000 7f7403e00640 1 -- 172.21.15.6:0/1697403537 shutdown_connections 2024-03-04T09:28:20.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:20.733+0000 7f7403e00640 1 --2- 172.21.15.6:0/1697403537 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f74040cd060 0x7f740415c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:20.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:20.733+0000 7f7403e00640 1 --2- 172.21.15.6:0/1697403537 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7404145300 0x7f74040b49a0 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:20.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:20.733+0000 7f7403e00640 1 -- 172.21.15.6:0/1697403537 >> 172.21.15.6:0/1697403537 conn(0x7f740406b3d0 msgr2=0x7f740406b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:20.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:20.733+0000 7f7403e00640 1 -- 172.21.15.6:0/1697403537 shutdown_connections 2024-03-04T09:28:20.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:20.733+0000 7f7403e00640 1 -- 172.21.15.6:0/1697403537 wait complete. 2024-03-04T09:28:20.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:20.735+0000 7f7403e00640 1 Processor -- start 2024-03-04T09:28:20.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:20.735+0000 7f7403e00640 1 -- start start 2024-03-04T09:28:20.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:20.736+0000 7f7403e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7404070f10 0x7f740413e880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:20.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:20.736+0000 7f7403e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f74040cd060 0x7f740406d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:20.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:20.736+0000 7f7403e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7404145300 0x7f740413cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:20.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:20.736+0000 7f7403e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f740415cab0 con 0x7f74040cd060 2024-03-04T09:28:20.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:20.736+0000 7f7403e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74041456e0 con 0x7f7404145300 2024-03-04T09:28:20.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:20.736+0000 7f7403e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f740415cc30 con 0x7f7404070f10 2024-03-04T09:28:20.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:20.736+0000 7f73fa000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f74040cd060 0x7f740406d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:20.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:20.736+0000 7f73fa000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f74040cd060 0x7f740406d650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:48120/0 (socket says 172.21.15.6:48120) 2024-03-04T09:28:20.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:20.736+0000 7f73fa000640 1 -- 172.21.15.6:0/2535398116 learned_addr learned my addr 172.21.15.6:0/2535398116 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:28:20.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:20.737+0000 7f7403400640 1 --2- 172.21.15.6:0/2535398116 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7404145300 0x7f740413cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:20.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:20.737+0000 7f7402a00640 1 --2- 172.21.15.6:0/2535398116 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7404070f10 0x7f740413e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:20.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:20.737+0000 7f73fa000640 1 -- 172.21.15.6:0/2535398116 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7404070f10 msgr2=0x7f740413e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:20.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:20.737+0000 7f73fa000640 1 --2- 172.21.15.6:0/2535398116 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7404070f10 0x7f740413e880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:20.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:20.737+0000 7f73fa000640 1 -- 172.21.15.6:0/2535398116 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7404145300 msgr2=0x7f740413cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:20.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:20.737+0000 7f73fa000640 1 --2- 172.21.15.6:0/2535398116 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7404145300 0x7f740413cd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:20.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:20.737+0000 7f73fa000640 1 -- 172.21.15.6:0/2535398116 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f740413d7e0 con 0x7f74040cd060 2024-03-04T09:28:20.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:20.737+0000 7f7402a00640 1 --2- 172.21.15.6:0/2535398116 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7404070f10 0x7f740413e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:28:20.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:20.737+0000 7f73fa000640 1 --2- 172.21.15.6:0/2535398116 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f74040cd060 0x7f740406d650 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f73ec00c430 tx=0x7f73ec00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:20.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:20.737+0000 7f73fbe00640 1 -- 172.21.15.6:0/2535398116 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73ec017070 con 0x7f74040cd060 2024-03-04T09:28:20.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:20.737+0000 7f73fbe00640 1 -- 172.21.15.6:0/2535398116 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73ec005ce0 con 0x7f74040cd060 2024-03-04T09:28:20.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:20.737+0000 7f7403e00640 1 -- 172.21.15.6:0/2535398116 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f740413dac0 con 0x7f74040cd060 2024-03-04T09:28:20.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:20.737+0000 7f73fbe00640 1 -- 172.21.15.6:0/2535398116 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73ec0072f0 con 0x7f74040cd060 2024-03-04T09:28:20.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:20.738+0000 7f7403e00640 1 -- 172.21.15.6:0/2535398116 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f740413e360 con 0x7f74040cd060 2024-03-04T09:28:20.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:20.739+0000 7f7403e00640 1 -- 172.21.15.6:0/2535398116 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f740415bf10 con 0x7f74040cd060 2024-03-04T09:28:20.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:20.740+0000 7f73fbe00640 1 -- 172.21.15.6:0/2535398116 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f73ec01a070 con 0x7f74040cd060 2024-03-04T09:28:20.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:20.740+0000 7f73fbe00640 1 --2- 172.21.15.6:0/2535398116 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f73e006b4d0 0x7f73e006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:20.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:20.741+0000 7f73fbe00640 1 -- 172.21.15.6:0/2535398116 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f73ec089650 con 0x7f74040cd060 2024-03-04T09:28:20.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:20.741+0000 7f7402a00640 1 --2- 172.21.15.6:0/2535398116 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f73e006b4d0 0x7f73e006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:20.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:20.742+0000 7f7402a00640 1 --2- 172.21.15.6:0/2535398116 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f73e006b4d0 0x7f73e006d990 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f73f4036000 tx=0x7f73f4008030 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:20.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:20.745+0000 7f73fbe00640 1 -- 172.21.15.6:0/2535398116 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f740415bf10 con 0x7f74040cd060 2024-03-04T09:28:20.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:20.946+0000 7f7403e00640 1 -- 172.21.15.6:0/2535398116 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f74040b4d00 con 0x7f74040cd060 2024-03-04T09:28:20.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:20.948+0000 7f73fbe00640 1 -- 172.21.15.6:0/2535398116 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v56) v1 ==== 74+0+16617 (secure 0 0 0) 0x7f74040b4d00 con 0x7f74040cd060 2024-03-04T09:28:20.946 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:28:20.946 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":56,"fsid":"edce0578-5ae4-4402-b45a-552bdef84d7c","created":"2024-03-04T09:24:54.523780+0000","modified":"2024-03-04T09:28:01.413693+0000","last_up_change":"2024-03-04T09:25:07.089049+0000","last_in_change":"2024-03-04T09:24:57.982077+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T09:25:07.935198+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-04T09:27:57.774529+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-04T09:27:58.886504+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a96574f-81f1-4bc6-8463-757520b84a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6825","nonce":1364449401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6827","nonce":1364449401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6831","nonce":1364449401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6829","nonce":1364449401}]},"public_addr":"172.21.15.6:6825/1364449401","cluster_addr":"172.21.15.6:6827/1364449401","heartbeat_back_addr":"172.21.15.6:6831/1364449401","heartbeat_front_addr":"172.21.15.6:6829/1364449401","state":["exists","up"]},{"osd":1,"uuid":"7d79b9c1-6b87-476a-bc36-7db5ee64cd90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6809","nonce":2494103078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6811","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6813","nonce":2494103078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6821","nonce":2494103078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6817","nonce":2494103078}]},"public_addr":"172.21.15.6:6809/2494103078","cluster_addr":"172.21.15.6:6813/2494103078","heartbeat_back_addr":"172.21.15.6:6821/2494103078","heartbeat_front_addr":"172.21.15.6:6817/2494103078","state":["exists","up"]},{"osd":2,"uuid":"492331b9-4ee9-40b6-bacd-85d44cd6c45a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6812","nonce":1500066681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6816","nonce":1500066681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6824","nonce":1500066681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6820","nonce":1500066681}]},"public_addr":"172.21.15.6:6812/1500066681","cluster_addr":"172.21.15.6:6816/1500066681","heartbeat_back_addr":"172.21.15.6:6824/1500066681","heartbeat_front_addr":"172.21.15.6:6820/1500066681","state":["exists","up"]},{"osd":3,"uuid":"21bb2efe-9ba7-4bdc-bc31-180cd1102e21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6801","nonce":1558806123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6803","nonce":1558806123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6807","nonce":1558806123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6805","nonce":1558806123}]},"public_addr":"172.21.15.6:6801/1558806123","cluster_addr":"172.21.15.6:6803/1558806123","heartbeat_back_addr":"172.21.15.6:6807/1558806123","heartbeat_front_addr":"172.21.15.6:6805/1558806123","state":["exists","up"]},{"osd":4,"uuid":"6e2c4b71-19d9-4737-83d9-558705d8644a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6801","nonce":1526135780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6803","nonce":1526135780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6807","nonce":1526135780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6805","nonce":1526135780}]},"public_addr":"172.21.15.146:6801/1526135780","cluster_addr":"172.21.15.146:6803/1526135780","heartbeat_back_addr":"172.21.15.146:6807/1526135780","heartbeat_front_addr":"172.21.15.146:6805/1526135780","state":["exists","up"]},{"osd":5,"uuid":"fb69c525-f1be-4b7d-ab32-6c29b0d6fddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6809","nonce":460262507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6811","nonce":460262507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6815","nonce":460262507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6813","nonce":460262507}]},"public_addr":"172.21.15.146:6809/460262507","cluster_addr":"172.21.15.146:6811/460262507","heartbeat_back_addr":"172.21.15.146:6815/460262507","heartbeat_front_addr":"172.21.15.146:6813/460262507","state":["exists","up"]},{"osd":6,"uuid":"c32e7fed-370d-4401-8ce0-6a4335b0d168","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6823","nonce":1392897493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6827","nonce":1392897493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6831","nonce":1392897493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6829","nonce":1392897493}]},"public_addr":"172.21.15.146:6823/1392897493","cluster_addr":"172.21.15.146:6827/1392897493","heartbeat_back_addr":"172.21.15.146:6831/1392897493","heartbeat_front_addr":"172.21.15.146:6829/1392897493","state":["exists","up"]},{"osd":7,"uuid":"75a79b44-fa5f-426d-a7d3-71f21f78835c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6817","nonce":3268323156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6819","nonce":3268323156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6825","nonce":3268323156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6822","nonce":3268323156}]},"public_addr":"172.21.15.146:6817/3268323156","cluster_addr":"172.21.15.146:6819/3268323156","heartbeat_back_addr":"172.21.15.146:6825/3268323156","heartbeat_front_addr":"172.21.15.146:6822/3268323156","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.310437+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.295850+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.294674+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.526855+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.300023+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.595160+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.346046+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.558741+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T09:28:20.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:20.950+0000 7f7403e00640 1 -- 172.21.15.6:0/2535398116 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f73e006b4d0 msgr2=0x7f73e006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:20.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:20.950+0000 7f7403e00640 1 --2- 172.21.15.6:0/2535398116 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f73e006b4d0 0x7f73e006d990 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f73f4036000 tx=0x7f73f4008030 comp rx=0 tx=0).stop 2024-03-04T09:28:20.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:20.950+0000 7f7403e00640 1 -- 172.21.15.6:0/2535398116 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f74040cd060 msgr2=0x7f740406d650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:20.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:20.950+0000 7f7403e00640 1 --2- 172.21.15.6:0/2535398116 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f74040cd060 0x7f740406d650 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f73ec00c430 tx=0x7f73ec00c8f0 comp rx=0 tx=0).stop 2024-03-04T09:28:20.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:20.951+0000 7f7403e00640 1 -- 172.21.15.6:0/2535398116 shutdown_connections 2024-03-04T09:28:20.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:20.951+0000 7f7403e00640 1 --2- 172.21.15.6:0/2535398116 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7404145300 0x7f740413cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:20.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:20.951+0000 7f7403e00640 1 --2- 172.21.15.6:0/2535398116 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f74040cd060 0x7f740406d650 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:20.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:20.951+0000 7f7403e00640 1 --2- 172.21.15.6:0/2535398116 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f73e006b4d0 0x7f73e006d990 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:20.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:20.951+0000 7f7403e00640 1 --2- 172.21.15.6:0/2535398116 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7404070f10 0x7f740413e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:20.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:20.951+0000 7f7403e00640 1 -- 172.21.15.6:0/2535398116 >> 172.21.15.6:0/2535398116 conn(0x7f740406b3d0 msgr2=0x7f74040afec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:20.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:20.951+0000 7f7403e00640 1 -- 172.21.15.6:0/2535398116 shutdown_connections 2024-03-04T09:28:20.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:20.951+0000 7f7403e00640 1 -- 172.21.15.6:0/2535398116 wait complete. 2024-03-04T09:28:20.966 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T09:25:07.935198+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-04T09:27:57.774529+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '55', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2024-03-04T09:27:58.886504+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '56', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-04T09:28:20.967 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-04T09:28:21.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.184+0000 7f949b000640 1 Processor -- start 2024-03-04T09:28:21.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.184+0000 7f949b000640 1 -- start start 2024-03-04T09:28:21.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.185+0000 7f949b000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f949415bf20 0x7f94940acd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:21.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.185+0000 7f949b000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f9494071e20 0x7f94940ccad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:21.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.185+0000 7f949b000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9494072df0 con 0x7f9494072520 2024-03-04T09:28:21.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.185+0000 7f949b000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94940cdab0 con 0x7f9494071e20 2024-03-04T09:28:21.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.185+0000 7f949b000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94940b4d70 con 0x7f949415bf20 2024-03-04T09:28:21.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.186+0000 7f948be00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f9494072520 0x7f94940cdfb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:51802/0 (socket says 172.21.15.6:51802) 2024-03-04T09:28:21.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.186+0000 7f948be00640 1 -- 172.21.15.6:0/681046112 learned_addr learned my addr 172.21.15.6:0/681046112 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:28:21.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.186+0000 7f9499000640 1 --2- 172.21.15.6:0/681046112 >> v2:172.21.15.6:3300/0 conn(0x7f9494071e20 0x7f94940ccad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:21.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.186+0000 7f9493e00640 1 --2- 172.21.15.6:0/681046112 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f949415bf20 0x7f94940acd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:21.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.186+0000 7f9493400640 1 -- 172.21.15.6:0/681046112 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4004176171 0 0) 0x7f9494072df0 con 0x7f9494072520 2024-03-04T09:28:21.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.186+0000 7f9493400640 1 -- 172.21.15.6:0/681046112 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9478003520 con 0x7f9494072520 2024-03-04T09:28:21.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.187+0000 7f9499000640 1 -- 172.21.15.6:0/681046112 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f949415bf20 msgr2=0x7f94940acd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:21.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.187+0000 7f9499000640 1 --2- 172.21.15.6:0/681046112 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f949415bf20 0x7f94940acd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:21.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.187+0000 7f9499000640 1 -- 172.21.15.6:0/681046112 >> v1:172.21.15.6:6789/0 conn(0x7f9494072520 legacy=0x7f94940cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:21.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.187+0000 7f9499000640 1 -- 172.21.15.6:0/681046112 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94940b5050 con 0x7f9494071e20 2024-03-04T09:28:21.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.188+0000 7f9499000640 1 --2- 172.21.15.6:0/681046112 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9494071e20 0x7f94940ccad0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f948c0097a0 tx=0x7f948c02b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9a4fc2d96bfffdc server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:21.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.188+0000 7f9493400640 1 -- 172.21.15.6:0/681046112 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f948c035020 con 0x7f9494071e20 2024-03-04T09:28:21.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.189+0000 7f9493400640 1 -- 172.21.15.6:0/681046112 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f948c031810 con 0x7f9494071e20 2024-03-04T09:28:21.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.189+0000 7f9493400640 1 -- 172.21.15.6:0/681046112 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f948c031bc0 con 0x7f9494071e20 2024-03-04T09:28:21.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.189+0000 7f949b000640 1 -- 172.21.15.6:0/681046112 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9494071e20 msgr2=0x7f94940ccad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:21.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.189+0000 7f949b000640 1 --2- 172.21.15.6:0/681046112 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9494071e20 0x7f94940ccad0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f948c0097a0 tx=0x7f948c02b8b0 comp rx=0 tx=0).stop 2024-03-04T09:28:21.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.189+0000 7f949b000640 1 -- 172.21.15.6:0/681046112 shutdown_connections 2024-03-04T09:28:21.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.189+0000 7f949b000640 1 --2- 172.21.15.6:0/681046112 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9494071e20 0x7f94940ccad0 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:21.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.189+0000 7f949b000640 1 --2- 172.21.15.6:0/681046112 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f949415bf20 0x7f94940acd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:21.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.189+0000 7f949b000640 1 -- 172.21.15.6:0/681046112 >> 172.21.15.6:0/681046112 conn(0x7f949406b3d0 msgr2=0x7f949406b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:21.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.189+0000 7f949b000640 1 -- 172.21.15.6:0/681046112 shutdown_connections 2024-03-04T09:28:21.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.189+0000 7f949b000640 1 -- 172.21.15.6:0/681046112 wait complete. 2024-03-04T09:28:21.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.191+0000 7f949b000640 1 Processor -- start 2024-03-04T09:28:21.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.191+0000 7f949b000640 1 -- start start 2024-03-04T09:28:21.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.192+0000 7f949b000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9494071e20 0x7f94940caf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:21.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.192+0000 7f949b000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9494072520 0x7f949414fe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:21.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.192+0000 7f949b000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f949415bf20 0x7f949406dc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:21.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.192+0000 7f949b000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94940b4b10 con 0x7f9494071e20 2024-03-04T09:28:21.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.192+0000 7f949b000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94940b4990 con 0x7f949415bf20 2024-03-04T09:28:21.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.192+0000 7f949b000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94940ce6c0 con 0x7f9494072520 2024-03-04T09:28:21.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.192+0000 7f9493e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9494071e20 0x7f94940caf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:21.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.193+0000 7f9493e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9494071e20 0x7f94940caf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:48140/0 (socket says 172.21.15.6:48140) 2024-03-04T09:28:21.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.193+0000 7f9493e00640 1 -- 172.21.15.6:0/2859119935 learned_addr learned my addr 172.21.15.6:0/2859119935 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:28:21.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.193+0000 7f9499000640 1 --2- 172.21.15.6:0/2859119935 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f949415bf20 0x7f949406dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:21.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.193+0000 7f948be00640 1 --2- 172.21.15.6:0/2859119935 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9494072520 0x7f949414fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:21.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.193+0000 7f9493e00640 1 -- 172.21.15.6:0/2859119935 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9494072520 msgr2=0x7f949414fe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:21.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.193+0000 7f9493e00640 1 --2- 172.21.15.6:0/2859119935 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9494072520 0x7f949414fe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:21.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.193+0000 7f9493e00640 1 -- 172.21.15.6:0/2859119935 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f949415bf20 msgr2=0x7f949406dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:21.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.193+0000 7f9493e00640 1 --2- 172.21.15.6:0/2859119935 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f949415bf20 0x7f949406dc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:21.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.193+0000 7f9493e00640 1 -- 172.21.15.6:0/2859119935 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94940c9e60 con 0x7f9494071e20 2024-03-04T09:28:21.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.193+0000 7f9499000640 1 --2- 172.21.15.6:0/2859119935 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f949415bf20 0x7f949406dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:28:21.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.193+0000 7f948be00640 1 --2- 172.21.15.6:0/2859119935 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9494072520 0x7f949414fe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:28:21.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.193+0000 7f9493e00640 1 --2- 172.21.15.6:0/2859119935 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9494071e20 0x7f94940caf00 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f9484009d10 tx=0x7f948400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:21.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.194+0000 7f9490c00640 1 -- 172.21.15.6:0/2859119935 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9484018020 con 0x7f9494071e20 2024-03-04T09:28:21.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.194+0000 7f9490c00640 1 -- 172.21.15.6:0/2859119935 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9484014480 con 0x7f9494071e20 2024-03-04T09:28:21.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.194+0000 7f9490c00640 1 -- 172.21.15.6:0/2859119935 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9484014830 con 0x7f9494071e20 2024-03-04T09:28:21.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.194+0000 7f949b000640 1 -- 172.21.15.6:0/2859119935 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94940ca140 con 0x7f9494071e20 2024-03-04T09:28:21.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.194+0000 7f949b000640 1 -- 172.21.15.6:0/2859119935 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94940ca700 con 0x7f9494071e20 2024-03-04T09:28:21.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.195+0000 7f949b000640 1 -- 172.21.15.6:0/2859119935 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f949404f330 con 0x7f9494071e20 2024-03-04T09:28:21.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.196+0000 7f9490c00640 1 -- 172.21.15.6:0/2859119935 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f948401b020 con 0x7f9494071e20 2024-03-04T09:28:21.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.197+0000 7f9490c00640 1 --2- 172.21.15.6:0/2859119935 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f947006b4d0 0x7f947006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:21.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.197+0000 7f9490c00640 1 -- 172.21.15.6:0/2859119935 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f948408ad20 con 0x7f9494071e20 2024-03-04T09:28:21.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.198+0000 7f948be00640 1 --2- 172.21.15.6:0/2859119935 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f947006b4d0 0x7f947006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:21.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.199+0000 7f948be00640 1 --2- 172.21.15.6:0/2859119935 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f947006b4d0 0x7f947006d990 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f949406d5c0 tx=0x7f947c009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:21.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.201+0000 7f9490c00640 1 -- 172.21.15.6:0/2859119935 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9484022030 con 0x7f9494071e20 2024-03-04T09:28:21.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.405+0000 7f949b000640 1 -- 172.21.15.6:0/2859119935 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f94940ce200 con 0x7f9494071e20 2024-03-04T09:28:21.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.406+0000 7f9490c00640 1 -- 172.21.15.6:0/2859119935 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v56) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9484010040 con 0x7f9494071e20 2024-03-04T09:28:21.403 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-04T09:28:21.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.409+0000 7f949b000640 1 -- 172.21.15.6:0/2859119935 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f947006b4d0 msgr2=0x7f947006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:21.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.409+0000 7f949b000640 1 --2- 172.21.15.6:0/2859119935 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f947006b4d0 0x7f947006d990 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f949406d5c0 tx=0x7f947c009040 comp rx=0 tx=0).stop 2024-03-04T09:28:21.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.409+0000 7f949b000640 1 -- 172.21.15.6:0/2859119935 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9494071e20 msgr2=0x7f94940caf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:21.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.409+0000 7f949b000640 1 --2- 172.21.15.6:0/2859119935 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9494071e20 0x7f94940caf00 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f9484009d10 tx=0x7f948400c490 comp rx=0 tx=0).stop 2024-03-04T09:28:21.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.410+0000 7f949b000640 1 -- 172.21.15.6:0/2859119935 shutdown_connections 2024-03-04T09:28:21.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.410+0000 7f949b000640 1 --2- 172.21.15.6:0/2859119935 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f949415bf20 0x7f949406dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:21.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.410+0000 7f949b000640 1 --2- 172.21.15.6:0/2859119935 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f947006b4d0 0x7f947006d990 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:21.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.410+0000 7f949b000640 1 --2- 172.21.15.6:0/2859119935 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9494072520 0x7f949414fe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:21.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.410+0000 7f949b000640 1 --2- 172.21.15.6:0/2859119935 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9494071e20 0x7f94940caf00 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:21.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.410+0000 7f949b000640 1 -- 172.21.15.6:0/2859119935 >> 172.21.15.6:0/2859119935 conn(0x7f949406b3d0 msgr2=0x7f949415af30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:21.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.410+0000 7f949b000640 1 -- 172.21.15.6:0/2859119935 shutdown_connections 2024-03-04T09:28:21.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.410+0000 7f949b000640 1 -- 172.21.15.6:0/2859119935 wait complete. 2024-03-04T09:28:21.426 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-04T09:28:21.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.656+0000 7fc60ec00640 1 Processor -- start 2024-03-04T09:28:21.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.656+0000 7fc60ec00640 1 -- start start 2024-03-04T09:28:21.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.658+0000 7fc60ec00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc608147590 0x7fc608071480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:21.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.658+0000 7fc60ec00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc60806d8d0 0x7fc6080b1b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:21.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.658+0000 7fc60ec00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6080719c0 con 0x7fc6080b2080 2024-03-04T09:28:21.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.658+0000 7fc60ec00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6080b24f0 con 0x7fc608147590 2024-03-04T09:28:21.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.658+0000 7fc60ec00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc608147970 con 0x7fc60806d8d0 2024-03-04T09:28:21.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.659+0000 7fc60d800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc608147590 0x7fc608071480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:21.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.659+0000 7fc60e200640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fc6080b2080 0x7fc6080ad880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:51808/0 (socket says 172.21.15.6:51808) 2024-03-04T09:28:21.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.659+0000 7fc60d800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc608147590 0x7fc608071480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:48148/0 (socket says 172.21.15.6:48148) 2024-03-04T09:28:21.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.659+0000 7fc60e200640 1 -- 172.21.15.6:0/2298800608 learned_addr learned my addr 172.21.15.6:0/2298800608 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:28:21.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.659+0000 7fc60ce00640 1 --2- 172.21.15.6:0/2298800608 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc60806d8d0 0x7fc6080b1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:21.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.659+0000 7fc60d800640 1 -- 172.21.15.6:0/2298800608 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc60806d8d0 msgr2=0x7fc6080b1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:21.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.659+0000 7fc60d800640 1 --2- 172.21.15.6:0/2298800608 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc60806d8d0 0x7fc6080b1b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:21.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.659+0000 7fc60d800640 1 -- 172.21.15.6:0/2298800608 >> v1:172.21.15.6:6789/0 conn(0x7fc6080b2080 legacy=0x7fc6080ad880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:21.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.659+0000 7fc60d800640 1 -- 172.21.15.6:0/2298800608 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc60806de40 con 0x7fc608147590 2024-03-04T09:28:21.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.661+0000 7fc60d800640 1 --2- 172.21.15.6:0/2298800608 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc608147590 0x7fc608071480 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fc5fc0097a0 tx=0x7fc5fc02b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e66de7f4d98e660 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:21.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.661+0000 7fc607e00640 1 -- 172.21.15.6:0/2298800608 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5fc035020 con 0x7fc608147590 2024-03-04T09:28:21.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.661+0000 7fc607e00640 1 -- 172.21.15.6:0/2298800608 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5fc031810 con 0x7fc608147590 2024-03-04T09:28:21.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.661+0000 7fc607e00640 1 -- 172.21.15.6:0/2298800608 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5fc031be0 con 0x7fc608147590 2024-03-04T09:28:21.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.661+0000 7fc60ec00640 1 -- 172.21.15.6:0/2298800608 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc608147590 msgr2=0x7fc608071480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:21.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.661+0000 7fc60ec00640 1 --2- 172.21.15.6:0/2298800608 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc608147590 0x7fc608071480 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fc5fc0097a0 tx=0x7fc5fc02b8b0 comp rx=0 tx=0).stop 2024-03-04T09:28:21.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.662+0000 7fc60ec00640 1 -- 172.21.15.6:0/2298800608 shutdown_connections 2024-03-04T09:28:21.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.662+0000 7fc60ec00640 1 --2- 172.21.15.6:0/2298800608 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc60806d8d0 0x7fc6080b1b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:21.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.662+0000 7fc60ec00640 1 --2- 172.21.15.6:0/2298800608 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc608147590 0x7fc608071480 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:21.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.662+0000 7fc60ec00640 1 -- 172.21.15.6:0/2298800608 >> 172.21.15.6:0/2298800608 conn(0x7fc60806b3a0 msgr2=0x7fc60806b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:21.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.662+0000 7fc60ec00640 1 -- 172.21.15.6:0/2298800608 shutdown_connections 2024-03-04T09:28:21.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.662+0000 7fc60ec00640 1 -- 172.21.15.6:0/2298800608 wait complete. 2024-03-04T09:28:21.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.663+0000 7fc60ec00640 1 Processor -- start 2024-03-04T09:28:21.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.664+0000 7fc60ec00640 1 -- start start 2024-03-04T09:28:21.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.664+0000 7fc60ec00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc60806d8d0 0x7fc608142dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:21.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.664+0000 7fc60ec00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc6080b2080 0x7fc608142660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:21.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.664+0000 7fc60ec00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc608147590 0x7fc608160560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:21.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.665+0000 7fc60ec00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6080b2e40 con 0x7fc608147590 2024-03-04T09:28:21.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.665+0000 7fc60ec00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6080b31b0 con 0x7fc60806d8d0 2024-03-04T09:28:21.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.665+0000 7fc60ec00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6080b2b50 con 0x7fc6080b2080 2024-03-04T09:28:21.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.665+0000 7fc60e200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc608147590 0x7fc608160560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:21.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.665+0000 7fc60e200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc608147590 0x7fc608160560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:48160/0 (socket says 172.21.15.6:48160) 2024-03-04T09:28:21.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.665+0000 7fc60e200640 1 -- 172.21.15.6:0/3872261590 learned_addr learned my addr 172.21.15.6:0/3872261590 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:28:21.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.665+0000 7fc60ce00640 1 --2- 172.21.15.6:0/3872261590 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc6080b2080 0x7fc608142660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:21.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.665+0000 7fc60d800640 1 --2- 172.21.15.6:0/3872261590 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc60806d8d0 0x7fc608142dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:21.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.665+0000 7fc60e200640 1 -- 172.21.15.6:0/3872261590 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc6080b2080 msgr2=0x7fc608142660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:21.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.665+0000 7fc60e200640 1 --2- 172.21.15.6:0/3872261590 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc6080b2080 0x7fc608142660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:21.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.665+0000 7fc60e200640 1 -- 172.21.15.6:0/3872261590 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc60806d8d0 msgr2=0x7fc608142dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:21.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.665+0000 7fc60e200640 1 --2- 172.21.15.6:0/3872261590 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc60806d8d0 0x7fc608142dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:21.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.665+0000 7fc60e200640 1 -- 172.21.15.6:0/3872261590 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6080c4b60 con 0x7fc608147590 2024-03-04T09:28:21.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.666+0000 7fc60e200640 1 --2- 172.21.15.6:0/3872261590 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc608147590 0x7fc608160560 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fc60000cbc0 tx=0x7fc60000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:21.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.666+0000 7fc605600640 1 -- 172.21.15.6:0/3872261590 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc600017070 con 0x7fc608147590 2024-03-04T09:28:21.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.666+0000 7fc605600640 1 -- 172.21.15.6:0/3872261590 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc60000ad40 con 0x7fc608147590 2024-03-04T09:28:21.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.666+0000 7fc605600640 1 -- 172.21.15.6:0/3872261590 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc600012600 con 0x7fc608147590 2024-03-04T09:28:21.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.666+0000 7fc60ec00640 1 -- 172.21.15.6:0/3872261590 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc608142ba0 con 0x7fc608147590 2024-03-04T09:28:21.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.666+0000 7fc60ec00640 1 -- 172.21.15.6:0/3872261590 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6080c5440 con 0x7fc608147590 2024-03-04T09:28:21.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.667+0000 7fc60ec00640 1 -- 172.21.15.6:0/3872261590 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6080c4d30 con 0x7fc608147590 2024-03-04T09:28:21.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.668+0000 7fc605600640 1 -- 172.21.15.6:0/3872261590 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc60001a070 con 0x7fc608147590 2024-03-04T09:28:21.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.669+0000 7fc605600640 1 --2- 172.21.15.6:0/3872261590 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc5e806b4d0 0x7fc5e806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:21.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.669+0000 7fc605600640 1 -- 172.21.15.6:0/3872261590 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc600089f10 con 0x7fc608147590 2024-03-04T09:28:21.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.670+0000 7fc60d800640 1 --2- 172.21.15.6:0/3872261590 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc5e806b4d0 0x7fc5e806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:21.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.671+0000 7fc60d800640 1 --2- 172.21.15.6:0/3872261590 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc5e806b4d0 0x7fc5e806d990 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fc5fc036000 tx=0x7fc5fc008030 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:21.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.674+0000 7fc605600640 1 -- 172.21.15.6:0/3872261590 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc6000586a0 con 0x7fc608147590 2024-03-04T09:28:21.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.875+0000 7fc60ec00640 1 -- 172.21.15.6:0/3872261590 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fc60804f330 con 0x7fc608147590 2024-03-04T09:28:21.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.876+0000 7fc605600640 1 -- 172.21.15.6:0/3872261590 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v56) v1 ==== 104+0+11 (secure 0 0 0) 0x7fc60005c740 con 0x7fc608147590 2024-03-04T09:28:21.873 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-03-04T09:28:21.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.878+0000 7fc60ec00640 1 -- 172.21.15.6:0/3872261590 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc5e806b4d0 msgr2=0x7fc5e806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:21.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.878+0000 7fc60ec00640 1 --2- 172.21.15.6:0/3872261590 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc5e806b4d0 0x7fc5e806d990 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fc5fc036000 tx=0x7fc5fc008030 comp rx=0 tx=0).stop 2024-03-04T09:28:21.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.878+0000 7fc60ec00640 1 -- 172.21.15.6:0/3872261590 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc608147590 msgr2=0x7fc608160560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:21.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.878+0000 7fc60ec00640 1 --2- 172.21.15.6:0/3872261590 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc608147590 0x7fc608160560 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fc60000cbc0 tx=0x7fc60000a430 comp rx=0 tx=0).stop 2024-03-04T09:28:21.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.878+0000 7fc60ec00640 1 -- 172.21.15.6:0/3872261590 shutdown_connections 2024-03-04T09:28:21.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.878+0000 7fc60ec00640 1 --2- 172.21.15.6:0/3872261590 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc608147590 0x7fc608160560 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:21.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.878+0000 7fc60ec00640 1 --2- 172.21.15.6:0/3872261590 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc5e806b4d0 0x7fc5e806d990 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:21.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.878+0000 7fc60ec00640 1 --2- 172.21.15.6:0/3872261590 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc6080b2080 0x7fc608142660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:21.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.878+0000 7fc60ec00640 1 --2- 172.21.15.6:0/3872261590 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc60806d8d0 0x7fc608142dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:21.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.879+0000 7fc60ec00640 1 -- 172.21.15.6:0/3872261590 >> 172.21.15.6:0/3872261590 conn(0x7fc60806b3a0 msgr2=0x7fc6080b3640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:21.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.879+0000 7fc60ec00640 1 -- 172.21.15.6:0/3872261590 shutdown_connections 2024-03-04T09:28:21.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:21.879+0000 7fc60ec00640 1 -- 172.21.15.6:0/3872261590 wait complete. 2024-03-04T09:28:21.894 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-04T09:28:22.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.129+0000 7f8efce00640 1 Processor -- start 2024-03-04T09:28:22.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.129+0000 7f8efce00640 1 -- start start 2024-03-04T09:28:22.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.131+0000 7f8efce00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f8ef815dee0 0x7f8ef80d18a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:22.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.131+0000 7f8efce00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f8ef80d0d40 0x7f8ef80b3510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:22.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.131+0000 7f8efce00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ef80ceca0 con 0x7f8ef80d4860 2024-03-04T09:28:22.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.131+0000 7f8efce00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ef80cf5d0 con 0x7f8ef815dee0 2024-03-04T09:28:22.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.131+0000 7f8efce00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ef80cef80 con 0x7f8ef80d0d40 2024-03-04T09:28:22.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.131+0000 7f8ef7400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f8ef815dee0 0x7f8ef80d18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:22.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.131+0000 7f8eefe00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f8ef80d4860 0x7f8ef80d0630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:51818/0 (socket says 172.21.15.6:51818) 2024-03-04T09:28:22.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.131+0000 7f8eefe00640 1 -- 172.21.15.6:0/2686265177 learned_addr learned my addr 172.21.15.6:0/2686265177 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:28:22.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.132+0000 7f8ef7e00640 1 --2- 172.21.15.6:0/2686265177 >> v2:172.21.15.146:3300/0 conn(0x7f8ef80d0d40 0x7f8ef80b3510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:22.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.132+0000 7f8ef6a00640 1 -- 172.21.15.6:0/2686265177 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2166678640 0 0) 0x7f8ef80ceca0 con 0x7f8ef80d4860 2024-03-04T09:28:22.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.132+0000 7f8ef6a00640 1 -- 172.21.15.6:0/2686265177 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8edc003520 con 0x7f8ef80d4860 2024-03-04T09:28:22.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.132+0000 7f8ef7400640 1 -- 172.21.15.6:0/2686265177 >> v2:172.21.15.146:3300/0 conn(0x7f8ef80d0d40 msgr2=0x7f8ef80b3510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:22.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.132+0000 7f8ef7400640 1 --2- 172.21.15.6:0/2686265177 >> v2:172.21.15.146:3300/0 conn(0x7f8ef80d0d40 0x7f8ef80b3510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:22.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.132+0000 7f8ef7400640 1 -- 172.21.15.6:0/2686265177 >> v1:172.21.15.6:6789/0 conn(0x7f8ef80d4860 legacy=0x7f8ef80d0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:22.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.132+0000 7f8ef7400640 1 -- 172.21.15.6:0/2686265177 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ef80cf260 con 0x7f8ef815dee0 2024-03-04T09:28:22.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.132+0000 7f8ef7e00640 1 --2- 172.21.15.6:0/2686265177 >> v2:172.21.15.146:3300/0 conn(0x7f8ef80d0d40 0x7f8ef80b3510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T09:28:22.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.133+0000 7f8ef7400640 1 --2- 172.21.15.6:0/2686265177 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8ef815dee0 0x7f8ef80d18a0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f8ef806b2d0 tx=0x7f8ee802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b079046ff0435a95 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:22.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.133+0000 7f8ef6a00640 1 -- 172.21.15.6:0/2686265177 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ee8035020 con 0x7f8ef815dee0 2024-03-04T09:28:22.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.134+0000 7f8ef6a00640 1 -- 172.21.15.6:0/2686265177 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ee8031830 con 0x7f8ef815dee0 2024-03-04T09:28:22.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.134+0000 7f8ef6a00640 1 -- 172.21.15.6:0/2686265177 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ee8031c00 con 0x7f8ef815dee0 2024-03-04T09:28:22.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.134+0000 7f8efce00640 1 -- 172.21.15.6:0/2686265177 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8ef815dee0 msgr2=0x7f8ef80d18a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:22.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.134+0000 7f8efce00640 1 --2- 172.21.15.6:0/2686265177 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8ef815dee0 0x7f8ef80d18a0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f8ef806b2d0 tx=0x7f8ee802b8d0 comp rx=0 tx=0).stop 2024-03-04T09:28:22.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.134+0000 7f8efce00640 1 -- 172.21.15.6:0/2686265177 shutdown_connections 2024-03-04T09:28:22.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.134+0000 7f8efce00640 1 --2- 172.21.15.6:0/2686265177 >> v2:172.21.15.146:3300/0 conn(0x7f8ef80d0d40 0x7f8ef80b3510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:22.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.134+0000 7f8efce00640 1 --2- 172.21.15.6:0/2686265177 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8ef815dee0 0x7f8ef80d18a0 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:22.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.134+0000 7f8efce00640 1 -- 172.21.15.6:0/2686265177 >> 172.21.15.6:0/2686265177 conn(0x7f8ef80d3cd0 msgr2=0x7f8ef80d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:22.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.134+0000 7f8efce00640 1 -- 172.21.15.6:0/2686265177 shutdown_connections 2024-03-04T09:28:22.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.134+0000 7f8efce00640 1 -- 172.21.15.6:0/2686265177 wait complete. 2024-03-04T09:28:22.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.136+0000 7f8efce00640 1 Processor -- start 2024-03-04T09:28:22.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.136+0000 7f8efce00640 1 -- start start 2024-03-04T09:28:22.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.137+0000 7f8efce00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8ef80d0d40 0x7f8ef815aab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:22.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.137+0000 7f8efce00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8ef80d4860 0x7f8ef809f600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:22.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.137+0000 7f8efce00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8ef815dee0 0x7f8ef809fb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:22.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.137+0000 7f8efce00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ef80d1de0 con 0x7f8ef80d0d40 2024-03-04T09:28:22.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.137+0000 7f8efce00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ef815e2c0 con 0x7f8ef815dee0 2024-03-04T09:28:22.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.137+0000 7f8efce00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ef80d1f60 con 0x7f8ef80d4860 2024-03-04T09:28:22.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.137+0000 7f8ef7400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8ef80d0d40 0x7f8ef815aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:22.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.137+0000 7f8ef7400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8ef80d0d40 0x7f8ef815aab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:48186/0 (socket says 172.21.15.6:48186) 2024-03-04T09:28:22.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.137+0000 7f8ef7400640 1 -- 172.21.15.6:0/647610202 learned_addr learned my addr 172.21.15.6:0/647610202 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:28:22.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.138+0000 7f8eefe00640 1 --2- 172.21.15.6:0/647610202 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8ef80d4860 0x7f8ef809f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:22.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.138+0000 7f8ef7e00640 1 --2- 172.21.15.6:0/647610202 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8ef815dee0 0x7f8ef809fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:22.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.138+0000 7f8ef7400640 1 -- 172.21.15.6:0/647610202 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8ef80d4860 msgr2=0x7f8ef809f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:22.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.138+0000 7f8ef7400640 1 --2- 172.21.15.6:0/647610202 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8ef80d4860 0x7f8ef809f600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:22.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.138+0000 7f8ef7400640 1 -- 172.21.15.6:0/647610202 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8ef815dee0 msgr2=0x7f8ef809fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:22.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.138+0000 7f8ef7400640 1 --2- 172.21.15.6:0/647610202 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8ef815dee0 0x7f8ef809fb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:22.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.138+0000 7f8ef7400640 1 -- 172.21.15.6:0/647610202 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ef80c82b0 con 0x7f8ef80d0d40 2024-03-04T09:28:22.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.138+0000 7f8ef7e00640 1 --2- 172.21.15.6:0/647610202 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8ef815dee0 0x7f8ef809fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:28:22.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.138+0000 7f8ef7400640 1 --2- 172.21.15.6:0/647610202 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8ef80d0d40 0x7f8ef815aab0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f8ee8009890 tx=0x7f8ee80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:22.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.139+0000 7f8eef400640 1 -- 172.21.15.6:0/647610202 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ee8041070 con 0x7f8ef80d0d40 2024-03-04T09:28:22.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.139+0000 7f8eef400640 1 -- 172.21.15.6:0/647610202 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ee803b920 con 0x7f8ef80d0d40 2024-03-04T09:28:22.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.139+0000 7f8eef400640 1 -- 172.21.15.6:0/647610202 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ee8035040 con 0x7f8ef80d0d40 2024-03-04T09:28:22.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.139+0000 7f8efce00640 1 -- 172.21.15.6:0/647610202 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ef80c8590 con 0x7f8ef80d0d40 2024-03-04T09:28:22.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.139+0000 7f8efce00640 1 -- 172.21.15.6:0/647610202 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ef80c8b50 con 0x7f8ef80d0d40 2024-03-04T09:28:22.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.140+0000 7f8efce00640 1 -- 172.21.15.6:0/647610202 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ef804f330 con 0x7f8ef80d0d40 2024-03-04T09:28:22.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.141+0000 7f8eef400640 1 -- 172.21.15.6:0/647610202 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8ee8038020 con 0x7f8ef80d0d40 2024-03-04T09:28:22.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.142+0000 7f8eef400640 1 --2- 172.21.15.6:0/647610202 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f8ed406b4d0 0x7f8ed406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:22.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.142+0000 7f8eef400640 1 -- 172.21.15.6:0/647610202 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8ee80abc30 con 0x7f8ef80d0d40 2024-03-04T09:28:22.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.143+0000 7f8eefe00640 1 --2- 172.21.15.6:0/647610202 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f8ed406b4d0 0x7f8ed406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:22.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.144+0000 7f8eefe00640 1 --2- 172.21.15.6:0/647610202 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f8ed406b4d0 0x7f8ed406d990 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f8ef815aff0 tx=0x7f8ee0009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:22.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.146+0000 7f8eef400640 1 -- 172.21.15.6:0/647610202 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8ee802f2e0 con 0x7f8ef80d0d40 2024-03-04T09:28:22.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.345+0000 7f8efce00640 1 -- 172.21.15.6:0/647610202 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f8ef808d7d0 con 0x7f8ef80d0d40 2024-03-04T09:28:22.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.346+0000 7f8eef400640 1 -- 172.21.15.6:0/647610202 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v56) v1 ==== 100+0+11 (secure 0 0 0) 0x7f8ee8031800 con 0x7f8ef80d0d40 2024-03-04T09:28:22.343 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 64 2024-03-04T09:28:22.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.348+0000 7f8efce00640 1 -- 172.21.15.6:0/647610202 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f8ed406b4d0 msgr2=0x7f8ed406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:22.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.348+0000 7f8efce00640 1 --2- 172.21.15.6:0/647610202 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f8ed406b4d0 0x7f8ed406d990 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f8ef815aff0 tx=0x7f8ee0009040 comp rx=0 tx=0).stop 2024-03-04T09:28:22.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.348+0000 7f8efce00640 1 -- 172.21.15.6:0/647610202 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8ef80d0d40 msgr2=0x7f8ef815aab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:22.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.348+0000 7f8efce00640 1 --2- 172.21.15.6:0/647610202 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8ef80d0d40 0x7f8ef815aab0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f8ee8009890 tx=0x7f8ee80046c0 comp rx=0 tx=0).stop 2024-03-04T09:28:22.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.348+0000 7f8efce00640 1 -- 172.21.15.6:0/647610202 shutdown_connections 2024-03-04T09:28:22.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.348+0000 7f8efce00640 1 --2- 172.21.15.6:0/647610202 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8ef815dee0 0x7f8ef809fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:22.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.348+0000 7f8efce00640 1 --2- 172.21.15.6:0/647610202 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f8ed406b4d0 0x7f8ed406d990 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:22.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.348+0000 7f8efce00640 1 --2- 172.21.15.6:0/647610202 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8ef80d4860 0x7f8ef809f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:22.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.348+0000 7f8efce00640 1 --2- 172.21.15.6:0/647610202 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8ef80d0d40 0x7f8ef815aab0 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:22.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.348+0000 7f8efce00640 1 -- 172.21.15.6:0/647610202 >> 172.21.15.6:0/647610202 conn(0x7f8ef80d3cd0 msgr2=0x7f8ef815bc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:22.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.348+0000 7f8efce00640 1 -- 172.21.15.6:0/647610202 shutdown_connections 2024-03-04T09:28:22.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.348+0000 7f8efce00640 1 -- 172.21.15.6:0/647610202 wait complete. 2024-03-04T09:28:22.364 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-04T09:28:22.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.595+0000 7fbf63800640 1 Processor -- start 2024-03-04T09:28:22.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.595+0000 7fbf63800640 1 -- start start 2024-03-04T09:28:22.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.596+0000 7fbf63800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbf5c0af1a0 0x7fbf5c15c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:22.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.596+0000 7fbf63800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fbf5c0af580 0x7fbf5c0b4c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:22.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.596+0000 7fbf63800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fbf5c0cdc20 0x7fbf5c0b2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:22.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.596+0000 7fbf63800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf5c15cc40 con 0x7fbf5c0af580 2024-03-04T09:28:22.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.596+0000 7fbf63800640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf5c15cf20 con 0x7fbf5c0cdc20 2024-03-04T09:28:22.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.596+0000 7fbf63800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf5c15d200 con 0x7fbf5c0af1a0 2024-03-04T09:28:22.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.596+0000 7fbf5be00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fbf5c0af580 0x7fbf5c0b4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:22.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.597+0000 7fbf5be00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fbf5c0af580 0x7fbf5c0b4c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:48188/0 (socket says 172.21.15.6:48188) 2024-03-04T09:28:22.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.597+0000 7fbf5be00640 1 -- 172.21.15.6:0/2351086915 learned_addr learned my addr 172.21.15.6:0/2351086915 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:28:22.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.597+0000 7fbf60e00640 1 --2- 172.21.15.6:0/2351086915 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbf5c0af1a0 0x7fbf5c15c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:22.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.597+0000 7fbf61800640 1 --2- 172.21.15.6:0/2351086915 >> v2:172.21.15.146:3300/0 conn(0x7fbf5c0cdc20 0x7fbf5c0b2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:22.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.597+0000 7fbf5be00640 1 -- 172.21.15.6:0/2351086915 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbf5c0af1a0 msgr2=0x7fbf5c15c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:22.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.597+0000 7fbf5be00640 1 --2- 172.21.15.6:0/2351086915 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbf5c0af1a0 0x7fbf5c15c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:22.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.597+0000 7fbf5be00640 1 -- 172.21.15.6:0/2351086915 >> v2:172.21.15.146:3300/0 conn(0x7fbf5c0cdc20 msgr2=0x7fbf5c0b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:22.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.597+0000 7fbf5be00640 1 --2- 172.21.15.6:0/2351086915 >> v2:172.21.15.146:3300/0 conn(0x7fbf5c0cdc20 0x7fbf5c0b2dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:22.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.597+0000 7fbf5be00640 1 -- 172.21.15.6:0/2351086915 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf5c15d4e0 con 0x7fbf5c0af580 2024-03-04T09:28:22.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.597+0000 7fbf60e00640 1 --2- 172.21.15.6:0/2351086915 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbf5c0af1a0 0x7fbf5c15c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T09:28:22.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.598+0000 7fbf5be00640 1 --2- 172.21.15.6:0/2351086915 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbf5c0af580 0x7fbf5c0b4c20 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fbf5c0cda30 tx=0x7fbf4c02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bb44be1e4f34575 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:22.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.599+0000 7fbf5b400640 1 -- 172.21.15.6:0/2351086915 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf4c02c9e0 con 0x7fbf5c0af580 2024-03-04T09:28:22.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.599+0000 7fbf5b400640 1 -- 172.21.15.6:0/2351086915 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf4c02cb80 con 0x7fbf5c0af580 2024-03-04T09:28:22.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.599+0000 7fbf5b400640 1 -- 172.21.15.6:0/2351086915 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf4c0343f0 con 0x7fbf5c0af580 2024-03-04T09:28:22.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.599+0000 7fbf63800640 1 -- 172.21.15.6:0/2351086915 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbf5c0af580 msgr2=0x7fbf5c0b4c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:22.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.599+0000 7fbf63800640 1 --2- 172.21.15.6:0/2351086915 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbf5c0af580 0x7fbf5c0b4c20 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fbf5c0cda30 tx=0x7fbf4c02be40 comp rx=0 tx=0).stop 2024-03-04T09:28:22.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.599+0000 7fbf63800640 1 -- 172.21.15.6:0/2351086915 shutdown_connections 2024-03-04T09:28:22.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.599+0000 7fbf63800640 1 --2- 172.21.15.6:0/2351086915 >> v2:172.21.15.146:3300/0 conn(0x7fbf5c0cdc20 0x7fbf5c0b2dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:22.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.599+0000 7fbf63800640 1 --2- 172.21.15.6:0/2351086915 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbf5c0af580 0x7fbf5c0b4c20 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:22.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.599+0000 7fbf63800640 1 --2- 172.21.15.6:0/2351086915 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbf5c0af1a0 0x7fbf5c15c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:22.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.599+0000 7fbf63800640 1 -- 172.21.15.6:0/2351086915 >> 172.21.15.6:0/2351086915 conn(0x7fbf5c06afb0 msgr2=0x7fbf5c06b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:22.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.600+0000 7fbf63800640 1 -- 172.21.15.6:0/2351086915 shutdown_connections 2024-03-04T09:28:22.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.600+0000 7fbf63800640 1 -- 172.21.15.6:0/2351086915 wait complete. 2024-03-04T09:28:22.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.601+0000 7fbf63800640 1 Processor -- start 2024-03-04T09:28:22.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.601+0000 7fbf63800640 1 -- start start 2024-03-04T09:28:22.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.602+0000 7fbf63800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbf5c0af1a0 0x7fbf5c0cb8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:22.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.602+0000 7fbf63800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbf5c0af580 0x7fbf5c154620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:22.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.602+0000 7fbf63800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbf5c0cdc20 0x7fbf5c0ad420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:22.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.602+0000 7fbf63800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf5c0b3490 con 0x7fbf5c0af1a0 2024-03-04T09:28:22.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.602+0000 7fbf63800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf5c0b3310 con 0x7fbf5c0af580 2024-03-04T09:28:22.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.602+0000 7fbf63800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf5c0b3610 con 0x7fbf5c0cdc20 2024-03-04T09:28:22.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.603+0000 7fbf60e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbf5c0af1a0 0x7fbf5c0cb8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:22.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.603+0000 7fbf60e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbf5c0af1a0 0x7fbf5c0cb8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:48202/0 (socket says 172.21.15.6:48202) 2024-03-04T09:28:22.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.603+0000 7fbf60e00640 1 -- 172.21.15.6:0/871644415 learned_addr learned my addr 172.21.15.6:0/871644415 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:28:22.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.603+0000 7fbf61800640 1 --2- 172.21.15.6:0/871644415 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbf5c0cdc20 0x7fbf5c0ad420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:22.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.603+0000 7fbf5be00640 1 --2- 172.21.15.6:0/871644415 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbf5c0af580 0x7fbf5c154620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:22.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.603+0000 7fbf60e00640 1 -- 172.21.15.6:0/871644415 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbf5c0cdc20 msgr2=0x7fbf5c0ad420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:22.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.603+0000 7fbf60e00640 1 --2- 172.21.15.6:0/871644415 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbf5c0cdc20 0x7fbf5c0ad420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:22.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.603+0000 7fbf60e00640 1 -- 172.21.15.6:0/871644415 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbf5c0af580 msgr2=0x7fbf5c154620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:22.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.603+0000 7fbf60e00640 1 --2- 172.21.15.6:0/871644415 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbf5c0af580 0x7fbf5c154620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:22.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.603+0000 7fbf60e00640 1 -- 172.21.15.6:0/871644415 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf5c1f4b10 con 0x7fbf5c0af1a0 2024-03-04T09:28:22.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.603+0000 7fbf5be00640 1 --2- 172.21.15.6:0/871644415 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbf5c0af580 0x7fbf5c154620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:28:22.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.603+0000 7fbf61800640 1 --2- 172.21.15.6:0/871644415 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbf5c0cdc20 0x7fbf5c0ad420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:28:22.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.603+0000 7fbf60e00640 1 --2- 172.21.15.6:0/871644415 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbf5c0af1a0 0x7fbf5c0cb8a0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fbf4400e810 tx=0x7fbf4400ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:22.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.604+0000 7fbf58c00640 1 -- 172.21.15.6:0/871644415 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf4400f040 con 0x7fbf5c0af1a0 2024-03-04T09:28:22.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.604+0000 7fbf63800640 1 -- 172.21.15.6:0/871644415 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf5c0c9400 con 0x7fbf5c0af1a0 2024-03-04T09:28:22.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.604+0000 7fbf58c00640 1 -- 172.21.15.6:0/871644415 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf44004030 con 0x7fbf5c0af1a0 2024-03-04T09:28:22.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.604+0000 7fbf58c00640 1 -- 172.21.15.6:0/871644415 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf44004400 con 0x7fbf5c0af1a0 2024-03-04T09:28:22.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.604+0000 7fbf63800640 1 -- 172.21.15.6:0/871644415 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf5c0cadc0 con 0x7fbf5c0af1a0 2024-03-04T09:28:22.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.605+0000 7fbf63800640 1 -- 172.21.15.6:0/871644415 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf5c04f3a0 con 0x7fbf5c0af1a0 2024-03-04T09:28:22.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.606+0000 7fbf58c00640 1 -- 172.21.15.6:0/871644415 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbf4400c670 con 0x7fbf5c0af1a0 2024-03-04T09:28:22.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.607+0000 7fbf58c00640 1 --2- 172.21.15.6:0/871644415 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fbf3806b4d0 0x7fbf3806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:22.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.607+0000 7fbf58c00640 1 -- 172.21.15.6:0/871644415 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbf44089ec0 con 0x7fbf5c0af1a0 2024-03-04T09:28:22.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.608+0000 7fbf5be00640 1 --2- 172.21.15.6:0/871644415 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fbf3806b4d0 0x7fbf3806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:22.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.609+0000 7fbf5be00640 1 --2- 172.21.15.6:0/871644415 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fbf3806b4d0 0x7fbf3806d990 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fbf4c035000 tx=0x7fbf4c030000 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:22.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.611+0000 7fbf58c00640 1 -- 172.21.15.6:0/871644415 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbf44017020 con 0x7fbf5c0af1a0 2024-03-04T09:28:22.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.824+0000 7fbf63800640 1 -- 172.21.15.6:0/871644415 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fbf5c15c820 con 0x7fbf5c0af1a0 2024-03-04T09:28:22.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.825+0000 7fbf58c00640 1 -- 172.21.15.6:0/871644415 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v56) v1 ==== 80+0+0 (secure 0 0 0) 0x7fbf44058cb0 con 0x7fbf5c0af1a0 2024-03-04T09:28:22.822 INFO:teuthology.orchestra.run.smithi006.stderr:listed 0 entries 2024-03-04T09:28:22.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.828+0000 7fbf63800640 1 -- 172.21.15.6:0/871644415 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fbf3806b4d0 msgr2=0x7fbf3806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:22.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.828+0000 7fbf63800640 1 --2- 172.21.15.6:0/871644415 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fbf3806b4d0 0x7fbf3806d990 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fbf4c035000 tx=0x7fbf4c030000 comp rx=0 tx=0).stop 2024-03-04T09:28:22.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.829+0000 7fbf63800640 1 -- 172.21.15.6:0/871644415 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbf5c0af1a0 msgr2=0x7fbf5c0cb8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:22.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.829+0000 7fbf63800640 1 --2- 172.21.15.6:0/871644415 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbf5c0af1a0 0x7fbf5c0cb8a0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fbf4400e810 tx=0x7fbf4400ecd0 comp rx=0 tx=0).stop 2024-03-04T09:28:22.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.829+0000 7fbf63800640 1 -- 172.21.15.6:0/871644415 shutdown_connections 2024-03-04T09:28:22.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.829+0000 7fbf63800640 1 --2- 172.21.15.6:0/871644415 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fbf3806b4d0 0x7fbf3806d990 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:22.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.829+0000 7fbf63800640 1 --2- 172.21.15.6:0/871644415 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbf5c0cdc20 0x7fbf5c0ad420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:22.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.829+0000 7fbf63800640 1 --2- 172.21.15.6:0/871644415 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbf5c0af580 0x7fbf5c154620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:22.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.829+0000 7fbf63800640 1 --2- 172.21.15.6:0/871644415 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbf5c0af1a0 0x7fbf5c0cb8a0 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:22.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.829+0000 7fbf63800640 1 -- 172.21.15.6:0/871644415 >> 172.21.15.6:0/871644415 conn(0x7fbf5c06afb0 msgr2=0x7fbf5c0cc520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:22.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.829+0000 7fbf63800640 1 -- 172.21.15.6:0/871644415 shutdown_connections 2024-03-04T09:28:22.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:22.829+0000 7fbf63800640 1 -- 172.21.15.6:0/871644415 wait complete. 2024-03-04T09:28:22.844 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-04T09:28:22.844 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:28:22.844 DEBUG:teuthology.orchestra.run.smithi006:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-04T09:28:23.037 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi006.front.sepia.ceph.com 2024-03-04T09:28:23.037 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:28:23.038 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-04T09:28:23.077 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:28:23.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:23.317+0000 7f819f800640 1 Processor -- start 2024-03-04T09:28:23.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:23.317+0000 7f819f800640 1 -- start start 2024-03-04T09:28:23.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:23.318+0000 7f819f800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8198161670 0x7f8198070c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:23.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:23.318+0000 7f819f800640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f819806de00 con 0x7f81980ae0a0 2024-03-04T09:28:23.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:23.318+0000 7f819f800640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81980ce9a0 con 0x7f819806e240 2024-03-04T09:28:23.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:23.319+0000 7f819f800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81980730d0 con 0x7f8198161670 2024-03-04T09:28:23.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:23.319+0000 7f819d800640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f81980ae0a0 0x7f819806d6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:51832/0 (socket says 172.21.15.6:51832) 2024-03-04T09:28:23.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:23.319+0000 7f819d800640 1 -- 172.21.15.6:0/1240494644 learned_addr learned my addr 172.21.15.6:0/1240494644 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:28:23.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:23.319+0000 7f819ce00640 1 --2- 172.21.15.6:0/1240494644 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8198161670 0x7f8198070c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:23.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:23.320+0000 7f8197400640 1 -- 172.21.15.6:0/1240494644 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3468389900 0 0) 0x7f819806de00 con 0x7f81980ae0a0 2024-03-04T09:28:23.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:23.320+0000 7f8197400640 1 -- 172.21.15.6:0/1240494644 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f817c003520 con 0x7f81980ae0a0 2024-03-04T09:28:23.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:23.320+0000 7f819ce00640 1 -- 172.21.15.6:0/1240494644 >> v1:172.21.15.146:6789/0 conn(0x7f819806e240 legacy=0x7f819806cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:23.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:23.320+0000 7f819ce00640 1 -- 172.21.15.6:0/1240494644 >> v1:172.21.15.6:6789/0 conn(0x7f81980ae0a0 legacy=0x7f819806d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:23.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:23.320+0000 7f819ce00640 1 -- 172.21.15.6:0/1240494644 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81980cec80 con 0x7f8198161670 2024-03-04T09:28:23.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:23.321+0000 7f819ce00640 1 --2- 172.21.15.6:0/1240494644 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8198161670 0x7f8198070c70 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f81800097d0 tx=0x7f818002b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=650c63f10cbae68f server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:23.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:23.321+0000 7f8197400640 1 -- 172.21.15.6:0/1240494644 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8180035020 con 0x7f8198161670 2024-03-04T09:28:23.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:23.322+0000 7f8197400640 1 -- 172.21.15.6:0/1240494644 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8180031890 con 0x7f8198161670 2024-03-04T09:28:23.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:23.322+0000 7f8197400640 1 -- 172.21.15.6:0/1240494644 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8180031c40 con 0x7f8198161670 2024-03-04T09:28:23.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:23.322+0000 7f819f800640 1 -- 172.21.15.6:0/1240494644 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8198161670 msgr2=0x7f8198070c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:23.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:23.322+0000 7f819f800640 1 --2- 172.21.15.6:0/1240494644 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8198161670 0x7f8198070c70 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f81800097d0 tx=0x7f818002b930 comp rx=0 tx=0).stop 2024-03-04T09:28:23.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:23.322+0000 7f819f800640 1 -- 172.21.15.6:0/1240494644 shutdown_connections 2024-03-04T09:28:23.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:23.322+0000 7f819f800640 1 --2- 172.21.15.6:0/1240494644 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8198161670 0x7f8198070c70 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:23.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:23.322+0000 7f819f800640 1 -- 172.21.15.6:0/1240494644 >> 172.21.15.6:0/1240494644 conn(0x7f819806afb0 msgr2=0x7f819806b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:23.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:23.322+0000 7f819f800640 1 -- 172.21.15.6:0/1240494644 shutdown_connections 2024-03-04T09:28:23.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:23.322+0000 7f819f800640 1 -- 172.21.15.6:0/1240494644 wait complete. 2024-03-04T09:28:23.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:23.324+0000 7f819f800640 1 Processor -- start 2024-03-04T09:28:23.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:23.324+0000 7f819f800640 1 -- start start 2024-03-04T09:28:23.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:23.325+0000 7f819f800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f819806e240 0x7f81980cbb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:23.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:23.325+0000 7f819f800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f81980ae0a0 0x7f8198149690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:23.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:23.325+0000 7f819f800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8198161670 0x7f819809aa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:23.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:23.325+0000 7f819f800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81980cf520 con 0x7f819806e240 2024-03-04T09:28:23.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:23.325+0000 7f819f800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81980cf240 con 0x7f8198161670 2024-03-04T09:28:23.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:23.325+0000 7f819f800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81980ce600 con 0x7f81980ae0a0 2024-03-04T09:28:23.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:23.326+0000 7f819ce00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f819806e240 0x7f81980cbb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:23.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:23.326+0000 7f819ce00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f819806e240 0x7f81980cbb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:48206/0 (socket says 172.21.15.6:48206) 2024-03-04T09:28:23.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:23.326+0000 7f819ce00640 1 -- 172.21.15.6:0/3446238428 learned_addr learned my addr 172.21.15.6:0/3446238428 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:28:23.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:23.326+0000 7f8197e00640 1 --2- 172.21.15.6:0/3446238428 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f81980ae0a0 0x7f8198149690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:23.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:23.326+0000 7f819ce00640 1 -- 172.21.15.6:0/3446238428 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f81980ae0a0 msgr2=0x7f8198149690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:23.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:23.326+0000 7f819ce00640 1 --2- 172.21.15.6:0/3446238428 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f81980ae0a0 0x7f8198149690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:23.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:23.326+0000 7f819ce00640 1 -- 172.21.15.6:0/3446238428 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8198161670 msgr2=0x7f819809aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:23.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:23.326+0000 7f819ce00640 1 --2- 172.21.15.6:0/3446238428 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8198161670 0x7f819809aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:23.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:23.326+0000 7f819ce00640 1 -- 172.21.15.6:0/3446238428 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8198149bd0 con 0x7f819806e240 2024-03-04T09:28:23.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:23.326+0000 7f819ce00640 1 --2- 172.21.15.6:0/3446238428 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f819806e240 0x7f81980cbb40 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f8180034040 tx=0x7f81800046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:23.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:23.327+0000 7f8194c00640 1 -- 172.21.15.6:0/3446238428 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8180035040 con 0x7f819806e240 2024-03-04T09:28:23.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:23.327+0000 7f8194c00640 1 -- 172.21.15.6:0/3446238428 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8180003820 con 0x7f819806e240 2024-03-04T09:28:23.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:23.327+0000 7f8194c00640 1 -- 172.21.15.6:0/3446238428 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8180003bf0 con 0x7f819806e240 2024-03-04T09:28:23.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:23.327+0000 7f819f800640 1 -- 172.21.15.6:0/3446238428 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f819809afc0 con 0x7f819806e240 2024-03-04T09:28:23.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:23.327+0000 7f819f800640 1 -- 172.21.15.6:0/3446238428 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81980cad80 con 0x7f819806e240 2024-03-04T09:28:23.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:23.328+0000 7f819f800640 1 -- 172.21.15.6:0/3446238428 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f819804f3a0 con 0x7f819806e240 2024-03-04T09:28:23.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:23.329+0000 7f8194c00640 1 -- 172.21.15.6:0/3446238428 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8180038020 con 0x7f819806e240 2024-03-04T09:28:23.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:23.330+0000 7f8194c00640 1 --2- 172.21.15.6:0/3446238428 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f817406b4d0 0x7f817406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:23.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:23.330+0000 7f8194c00640 1 -- 172.21.15.6:0/3446238428 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f81800ac940 con 0x7f819806e240 2024-03-04T09:28:23.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:23.331+0000 7f8197e00640 1 --2- 172.21.15.6:0/3446238428 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f817406b4d0 0x7f817406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:23.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:23.332+0000 7f8197e00640 1 --2- 172.21.15.6:0/3446238428 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f817406b4d0 0x7f817406d990 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f819814cfd0 tx=0x7f8188009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:23.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:23.334+0000 7f8194c00640 1 -- 172.21.15.6:0/3446238428 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8198149bd0 con 0x7f819806e240 2024-03-04T09:28:23.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:23.576+0000 7f819f800640 1 -- 172.21.15.6:0/3446238428 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f81980b52d0 con 0x7f819806e240 2024-03-04T09:28:23.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:23.578+0000 7f8194c00640 1 -- 172.21.15.6:0/3446238428 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+4971 (secure 0 0 0) 0x7f818002d040 con 0x7f819806e240 2024-03-04T09:28:23.575 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:28:23.575 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":34,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5030,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/1411942502","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1411942502},{"type":"v1","addr":"172.21.15.6:6834","nonce":1411942502}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28},{"gid":5039,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/1628166413","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":1628166413},{"type":"v1","addr":"172.21.15.146:6837","nonce":1628166413}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":5323,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/131491768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":131491768},{"type":"v1","addr":"172.21.15.6:6835","nonce":131491768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:27:59.854470+0000","modified":"2024-03-04T09:28:02.945053+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5326},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5326":{"gid":5326,"name":"b","rank":0,"incarnation":30,"state":"up:active","state_seq":3,"addr":"172.21.15.146:6836/527855083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":527855083},{"type":"v1","addr":"172.21.15.146:6836","nonce":527855083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5326,"qdb_cluster":[5326]},"id":4}]} 2024-03-04T09:28:23.575 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 34 2024-03-04T09:28:23.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:23.581+0000 7f819f800640 1 -- 172.21.15.6:0/3446238428 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f817406b4d0 msgr2=0x7f817406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:23.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:23.581+0000 7f819f800640 1 --2- 172.21.15.6:0/3446238428 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f817406b4d0 0x7f817406d990 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f819814cfd0 tx=0x7f8188009040 comp rx=0 tx=0).stop 2024-03-04T09:28:23.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:23.581+0000 7f819f800640 1 -- 172.21.15.6:0/3446238428 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f819806e240 msgr2=0x7f81980cbb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:23.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:23.581+0000 7f819f800640 1 --2- 172.21.15.6:0/3446238428 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f819806e240 0x7f81980cbb40 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f8180034040 tx=0x7f81800046c0 comp rx=0 tx=0).stop 2024-03-04T09:28:23.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:23.581+0000 7f819f800640 1 -- 172.21.15.6:0/3446238428 shutdown_connections 2024-03-04T09:28:23.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:23.581+0000 7f819f800640 1 --2- 172.21.15.6:0/3446238428 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8198161670 0x7f819809aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:23.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:23.581+0000 7f819f800640 1 --2- 172.21.15.6:0/3446238428 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f817406b4d0 0x7f817406d990 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:23.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:23.581+0000 7f819f800640 1 --2- 172.21.15.6:0/3446238428 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f81980ae0a0 0x7f8198149690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:23.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:23.581+0000 7f819f800640 1 --2- 172.21.15.6:0/3446238428 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f819806e240 0x7f81980cbb40 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:23.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:23.581+0000 7f819f800640 1 -- 172.21.15.6:0/3446238428 >> 172.21.15.6:0/3446238428 conn(0x7f819806afb0 msgr2=0x7f81980ad910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:23.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:23.581+0000 7f819f800640 1 -- 172.21.15.6:0/3446238428 shutdown_connections 2024-03-04T09:28:23.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:23.581+0000 7f819f800640 1 -- 172.21.15.6:0/3446238428 wait complete. 2024-03-04T09:28:23.595 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json dirfrag split /subdir 0/0 1 2024-03-04T09:28:23.821 DEBUG:tasks.cephfs.filesystem:_json_asok output empty 2024-03-04T09:28:23.821 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:28:24.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.057+0000 7f2f17a00640 1 Processor -- start 2024-03-04T09:28:24.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.058+0000 7f2f17a00640 1 -- start start 2024-03-04T09:28:24.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.059+0000 7f2f17a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f2f10161670 0x7f2f10070c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:24.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.059+0000 7f2f17a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2f100ae0a0 0x7f2f100711b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:24.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.059+0000 7f2f17a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f100ce9a0 con 0x7f2f1006e240 2024-03-04T09:28:24.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.059+0000 7f2f17a00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f100730d0 con 0x7f2f10161670 2024-03-04T09:28:24.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.059+0000 7f2f17a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f100cec80 con 0x7f2f100ae0a0 2024-03-04T09:28:24.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.060+0000 7f2f0fe00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f2f1006e240 0x7f2f1006cfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:51838/0 (socket says 172.21.15.6:51838) 2024-03-04T09:28:24.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.060+0000 7f2f0fe00640 1 -- 172.21.15.6:0/2265162007 learned_addr learned my addr 172.21.15.6:0/2265162007 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:28:24.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.060+0000 7f2f15000640 1 --2- 172.21.15.6:0/2265162007 >> v2:172.21.15.6:3300/0 conn(0x7f2f10161670 0x7f2f10070c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:24.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.060+0000 7f2f15a00640 1 --2- 172.21.15.6:0/2265162007 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2f100ae0a0 0x7f2f100711b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:24.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.060+0000 7f2f0f400640 1 -- 172.21.15.6:0/2265162007 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3750807110 0 0) 0x7f2f100ce9a0 con 0x7f2f1006e240 2024-03-04T09:28:24.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.060+0000 7f2f0f400640 1 -- 172.21.15.6:0/2265162007 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ef4003520 con 0x7f2f1006e240 2024-03-04T09:28:24.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.060+0000 7f2f15000640 1 -- 172.21.15.6:0/2265162007 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2f100ae0a0 msgr2=0x7f2f100711b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:24.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.060+0000 7f2f15000640 1 --2- 172.21.15.6:0/2265162007 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2f100ae0a0 0x7f2f100711b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:24.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.060+0000 7f2f15000640 1 -- 172.21.15.6:0/2265162007 >> v1:172.21.15.6:6789/0 conn(0x7f2f1006e240 legacy=0x7f2f1006cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:24.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.060+0000 7f2f15000640 1 -- 172.21.15.6:0/2265162007 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f100cef60 con 0x7f2f10161670 2024-03-04T09:28:24.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.060+0000 7f2f15a00640 1 --2- 172.21.15.6:0/2265162007 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2f100ae0a0 0x7f2f100711b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T09:28:24.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.062+0000 7f2f15000640 1 --2- 172.21.15.6:0/2265162007 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2f10161670 0x7f2f10070c70 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f2f000097a0 tx=0x7f2f0002b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d10dbd5e75f639 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:24.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.062+0000 7f2f0f400640 1 -- 172.21.15.6:0/2265162007 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f00035020 con 0x7f2f10161670 2024-03-04T09:28:24.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.062+0000 7f2f0f400640 1 -- 172.21.15.6:0/2265162007 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f00031810 con 0x7f2f10161670 2024-03-04T09:28:24.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.062+0000 7f2f0f400640 1 -- 172.21.15.6:0/2265162007 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f00031bc0 con 0x7f2f10161670 2024-03-04T09:28:24.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.062+0000 7f2f17a00640 1 -- 172.21.15.6:0/2265162007 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2f10161670 msgr2=0x7f2f10070c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:24.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.062+0000 7f2f17a00640 1 --2- 172.21.15.6:0/2265162007 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2f10161670 0x7f2f10070c70 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f2f000097a0 tx=0x7f2f0002b8b0 comp rx=0 tx=0).stop 2024-03-04T09:28:24.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.063+0000 7f2f17a00640 1 -- 172.21.15.6:0/2265162007 shutdown_connections 2024-03-04T09:28:24.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.063+0000 7f2f17a00640 1 --2- 172.21.15.6:0/2265162007 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2f100ae0a0 0x7f2f100711b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:24.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.063+0000 7f2f17a00640 1 --2- 172.21.15.6:0/2265162007 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2f10161670 0x7f2f10070c70 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:24.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.063+0000 7f2f17a00640 1 -- 172.21.15.6:0/2265162007 >> 172.21.15.6:0/2265162007 conn(0x7f2f1006afb0 msgr2=0x7f2f1006b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:24.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.063+0000 7f2f17a00640 1 -- 172.21.15.6:0/2265162007 shutdown_connections 2024-03-04T09:28:24.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.063+0000 7f2f17a00640 1 -- 172.21.15.6:0/2265162007 wait complete. 2024-03-04T09:28:24.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.064+0000 7f2f17a00640 1 Processor -- start 2024-03-04T09:28:24.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.065+0000 7f2f17a00640 1 -- start start 2024-03-04T09:28:24.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.065+0000 7f2f17a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2f1006e240 0x7f2f1014c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:24.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.065+0000 7f2f17a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2f100ae0a0 0x7f2f10156330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:24.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.065+0000 7f2f17a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2f10161670 0x7f2f1015de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:24.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.066+0000 7f2f17a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f100ce600 con 0x7f2f100ae0a0 2024-03-04T09:28:24.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.066+0000 7f2f17a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f100cf520 con 0x7f2f10161670 2024-03-04T09:28:24.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.066+0000 7f2f17a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f100ce780 con 0x7f2f1006e240 2024-03-04T09:28:24.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.066+0000 7f2f0fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2f100ae0a0 0x7f2f10156330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:24.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.066+0000 7f2f0fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2f100ae0a0 0x7f2f10156330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:48220/0 (socket says 172.21.15.6:48220) 2024-03-04T09:28:24.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.066+0000 7f2f0fe00640 1 -- 172.21.15.6:0/3413535167 learned_addr learned my addr 172.21.15.6:0/3413535167 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:28:24.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.066+0000 7f2f15000640 1 --2- 172.21.15.6:0/3413535167 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2f1006e240 0x7f2f1014c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:24.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.066+0000 7f2f15a00640 1 --2- 172.21.15.6:0/3413535167 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2f10161670 0x7f2f1015de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:24.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.066+0000 7f2f0fe00640 1 -- 172.21.15.6:0/3413535167 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2f1006e240 msgr2=0x7f2f1014c6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:24.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.066+0000 7f2f0fe00640 1 --2- 172.21.15.6:0/3413535167 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2f1006e240 0x7f2f1014c6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:24.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.066+0000 7f2f0fe00640 1 -- 172.21.15.6:0/3413535167 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2f10161670 msgr2=0x7f2f1015de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:24.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.066+0000 7f2f0fe00640 1 --2- 172.21.15.6:0/3413535167 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2f10161670 0x7f2f1015de80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:24.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.066+0000 7f2f0fe00640 1 -- 172.21.15.6:0/3413535167 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f101f7410 con 0x7f2f100ae0a0 2024-03-04T09:28:24.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.067+0000 7f2f0fe00640 1 --2- 172.21.15.6:0/3413535167 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2f100ae0a0 0x7f2f10156330 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f2ef80010e0 tx=0x7f2ef800c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:24.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.067+0000 7f2f0cc00640 1 -- 172.21.15.6:0/3413535167 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ef8017070 con 0x7f2f100ae0a0 2024-03-04T09:28:24.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.067+0000 7f2f0cc00640 1 -- 172.21.15.6:0/3413535167 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ef8005d00 con 0x7f2f100ae0a0 2024-03-04T09:28:24.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.067+0000 7f2f0cc00640 1 -- 172.21.15.6:0/3413535167 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ef80072b0 con 0x7f2f100ae0a0 2024-03-04T09:28:24.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.067+0000 7f2f17a00640 1 -- 172.21.15.6:0/3413535167 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f101f76f0 con 0x7f2f100ae0a0 2024-03-04T09:28:24.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.067+0000 7f2f17a00640 1 -- 172.21.15.6:0/3413535167 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f101f7cb0 con 0x7f2f100ae0a0 2024-03-04T09:28:24.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.068+0000 7f2f17a00640 1 -- 172.21.15.6:0/3413535167 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f10069510 con 0x7f2f100ae0a0 2024-03-04T09:28:24.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.069+0000 7f2f0cc00640 1 -- 172.21.15.6:0/3413535167 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2ef801a070 con 0x7f2f100ae0a0 2024-03-04T09:28:24.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.070+0000 7f2f0cc00640 1 --2- 172.21.15.6:0/3413535167 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2eec06b4d0 0x7f2eec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:24.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.070+0000 7f2f0cc00640 1 -- 172.21.15.6:0/3413535167 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2ef80896b0 con 0x7f2f100ae0a0 2024-03-04T09:28:24.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.071+0000 7f2f15000640 1 --2- 172.21.15.6:0/3413535167 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2eec06b4d0 0x7f2eec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:24.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.072+0000 7f2f15000640 1 --2- 172.21.15.6:0/3413535167 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2eec06b4d0 0x7f2eec06d990 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f2f00036000 tx=0x7f2f0003c040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:24.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.074+0000 7f2f0cc00640 1 -- 172.21.15.6:0/3413535167 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2ef8058520 con 0x7f2f100ae0a0 2024-03-04T09:28:24.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.311+0000 7f2f17a00640 1 -- 172.21.15.6:0/3413535167 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2f1004f3a0 con 0x7f2f100ae0a0 2024-03-04T09:28:24.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.312+0000 7f2f0cc00640 1 -- 172.21.15.6:0/3413535167 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+4971 (secure 0 0 0) 0x7f2ef8057e20 con 0x7f2f100ae0a0 2024-03-04T09:28:24.309 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:28:24.309 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":34,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5030,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/1411942502","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1411942502},{"type":"v1","addr":"172.21.15.6:6834","nonce":1411942502}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28},{"gid":5039,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/1628166413","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":1628166413},{"type":"v1","addr":"172.21.15.146:6837","nonce":1628166413}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":5323,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/131491768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":131491768},{"type":"v1","addr":"172.21.15.6:6835","nonce":131491768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:27:59.854470+0000","modified":"2024-03-04T09:28:02.945053+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5326},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5326":{"gid":5326,"name":"b","rank":0,"incarnation":30,"state":"up:active","state_seq":3,"addr":"172.21.15.146:6836/527855083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":527855083},{"type":"v1","addr":"172.21.15.146:6836","nonce":527855083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5326,"qdb_cluster":[5326]},"id":4}]} 2024-03-04T09:28:24.309 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 34 2024-03-04T09:28:24.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.314+0000 7f2f17a00640 1 -- 172.21.15.6:0/3413535167 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2eec06b4d0 msgr2=0x7f2eec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:24.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.315+0000 7f2f17a00640 1 --2- 172.21.15.6:0/3413535167 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2eec06b4d0 0x7f2eec06d990 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f2f00036000 tx=0x7f2f0003c040 comp rx=0 tx=0).stop 2024-03-04T09:28:24.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.315+0000 7f2f17a00640 1 -- 172.21.15.6:0/3413535167 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2f100ae0a0 msgr2=0x7f2f10156330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:24.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.315+0000 7f2f17a00640 1 --2- 172.21.15.6:0/3413535167 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2f100ae0a0 0x7f2f10156330 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f2ef80010e0 tx=0x7f2ef800c960 comp rx=0 tx=0).stop 2024-03-04T09:28:24.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.315+0000 7f2f17a00640 1 -- 172.21.15.6:0/3413535167 shutdown_connections 2024-03-04T09:28:24.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.315+0000 7f2f17a00640 1 --2- 172.21.15.6:0/3413535167 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2f10161670 0x7f2f1015de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:24.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.315+0000 7f2f17a00640 1 --2- 172.21.15.6:0/3413535167 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2f100ae0a0 0x7f2f10156330 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:24.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.315+0000 7f2f17a00640 1 --2- 172.21.15.6:0/3413535167 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2eec06b4d0 0x7f2eec06d990 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:24.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.315+0000 7f2f17a00640 1 --2- 172.21.15.6:0/3413535167 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2f1006e240 0x7f2f1014c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:24.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.315+0000 7f2f17a00640 1 -- 172.21.15.6:0/3413535167 >> 172.21.15.6:0/3413535167 conn(0x7f2f1006afb0 msgr2=0x7f2f100cd070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:24.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.315+0000 7f2f17a00640 1 -- 172.21.15.6:0/3413535167 shutdown_connections 2024-03-04T09:28:24.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.315+0000 7f2f17a00640 1 -- 172.21.15.6:0/3413535167 wait complete. 2024-03-04T09:28:24.333 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json flush journal 2024-03-04T09:28:24.599 INFO:teuthology.orchestra.run.smithi146.stdout:{"message":"","return_code":0} 2024-03-04T09:28:24.599 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-04T09:28:24.599 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-04T09:28:24.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.809+0000 7fdecf600640 1 Processor -- start 2024-03-04T09:28:24.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.809+0000 7fdecf600640 1 -- start start 2024-03-04T09:28:24.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.811+0000 7fdecf600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdec8161670 0x7fdec8070c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:24.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.811+0000 7fdecf600640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fdec806e240 0x7fdec80711b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:24.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.811+0000 7fdecf600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fdec806cfe0 0x7fdec806d420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:24.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.811+0000 7fdecf600640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdec80730d0 con 0x7fdec806cfe0 2024-03-04T09:28:24.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.811+0000 7fdecf600640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdec80ce9a0 con 0x7fdec806e240 2024-03-04T09:28:24.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.811+0000 7fdecf600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdec80cec80 con 0x7fdec8161670 2024-03-04T09:28:24.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.811+0000 7fdecd600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fdec806cfe0 0x7fdec806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:24.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.811+0000 7fdecd600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fdec806cfe0 0x7fdec806d420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:48236/0 (socket says 172.21.15.6:48236) 2024-03-04T09:28:24.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.811+0000 7fdecd600640 1 -- 172.21.15.6:0/228269520 learned_addr learned my addr 172.21.15.6:0/228269520 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:28:24.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.811+0000 7fdec7e00640 1 --2- 172.21.15.6:0/228269520 >> v2:172.21.15.146:3300/0 conn(0x7fdec806e240 0x7fdec80711b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:24.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.812+0000 7fdeccc00640 1 --2- 172.21.15.6:0/228269520 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdec8161670 0x7fdec8070c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:24.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.812+0000 7fdecd600640 1 -- 172.21.15.6:0/228269520 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdec8161670 msgr2=0x7fdec8070c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:24.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.812+0000 7fdecd600640 1 --2- 172.21.15.6:0/228269520 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdec8161670 0x7fdec8070c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:24.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.812+0000 7fdecd600640 1 -- 172.21.15.6:0/228269520 >> v2:172.21.15.146:3300/0 conn(0x7fdec806e240 msgr2=0x7fdec80711b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:24.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.812+0000 7fdecd600640 1 --2- 172.21.15.6:0/228269520 >> v2:172.21.15.146:3300/0 conn(0x7fdec806e240 0x7fdec80711b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:24.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.812+0000 7fdecd600640 1 -- 172.21.15.6:0/228269520 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdec80cef60 con 0x7fdec806cfe0 2024-03-04T09:28:24.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.813+0000 7fdecd600640 1 --2- 172.21.15.6:0/228269520 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdec806cfe0 0x7fdec806d420 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fdec80748c0 tx=0x7fdec002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cc578b59ddbced7 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:24.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.813+0000 7fdec7400640 1 -- 172.21.15.6:0/228269520 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdec002ca80 con 0x7fdec806cfe0 2024-03-04T09:28:24.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.813+0000 7fdec7400640 1 -- 172.21.15.6:0/228269520 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdec002cc20 con 0x7fdec806cfe0 2024-03-04T09:28:24.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.813+0000 7fdec7400640 1 -- 172.21.15.6:0/228269520 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdec00343f0 con 0x7fdec806cfe0 2024-03-04T09:28:24.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.814+0000 7fdecf600640 1 -- 172.21.15.6:0/228269520 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdec806cfe0 msgr2=0x7fdec806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:24.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.814+0000 7fdecf600640 1 --2- 172.21.15.6:0/228269520 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdec806cfe0 0x7fdec806d420 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fdec80748c0 tx=0x7fdec002bee0 comp rx=0 tx=0).stop 2024-03-04T09:28:24.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.814+0000 7fdecf600640 1 -- 172.21.15.6:0/228269520 shutdown_connections 2024-03-04T09:28:24.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.814+0000 7fdecf600640 1 --2- 172.21.15.6:0/228269520 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdec806cfe0 0x7fdec806d420 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:24.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.814+0000 7fdecf600640 1 --2- 172.21.15.6:0/228269520 >> v2:172.21.15.146:3300/0 conn(0x7fdec806e240 0x7fdec80711b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:24.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.814+0000 7fdecf600640 1 --2- 172.21.15.6:0/228269520 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdec8161670 0x7fdec8070c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:24.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.814+0000 7fdecf600640 1 -- 172.21.15.6:0/228269520 >> 172.21.15.6:0/228269520 conn(0x7fdec806afb0 msgr2=0x7fdec806b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:24.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.814+0000 7fdecf600640 1 -- 172.21.15.6:0/228269520 shutdown_connections 2024-03-04T09:28:24.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.814+0000 7fdecf600640 1 -- 172.21.15.6:0/228269520 wait complete. 2024-03-04T09:28:24.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.816+0000 7fdecf600640 1 Processor -- start 2024-03-04T09:28:24.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.816+0000 7fdecf600640 1 -- start start 2024-03-04T09:28:24.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.817+0000 7fdecf600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdec806cfe0 0x7fdec814d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:24.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.817+0000 7fdecf600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdec806e240 0x7fdec80cb4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:24.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.817+0000 7fdecf600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdec8161670 0x7fdec80cba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:24.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.817+0000 7fdecf600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdec80ce600 con 0x7fdec806cfe0 2024-03-04T09:28:24.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.817+0000 7fdecf600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdec80cf520 con 0x7fdec806e240 2024-03-04T09:28:24.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.817+0000 7fdecf600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdec80ce780 con 0x7fdec8161670 2024-03-04T09:28:24.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.817+0000 7fdeccc00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdec806cfe0 0x7fdec814d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:24.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.817+0000 7fdeccc00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdec806cfe0 0x7fdec814d6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:48246/0 (socket says 172.21.15.6:48246) 2024-03-04T09:28:24.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.817+0000 7fdeccc00640 1 -- 172.21.15.6:0/1349545453 learned_addr learned my addr 172.21.15.6:0/1349545453 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:28:24.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.818+0000 7fdec7e00640 1 --2- 172.21.15.6:0/1349545453 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdec806e240 0x7fdec80cb4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:24.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.818+0000 7fdeccc00640 1 -- 172.21.15.6:0/1349545453 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdec8161670 msgr2=0x7fdec80cba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:24.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.818+0000 7fdecd600640 1 --2- 172.21.15.6:0/1349545453 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdec8161670 0x7fdec80cba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:24.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.818+0000 7fdeccc00640 1 --2- 172.21.15.6:0/1349545453 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdec8161670 0x7fdec80cba30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:24.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.818+0000 7fdeccc00640 1 -- 172.21.15.6:0/1349545453 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdec806e240 msgr2=0x7fdec80cb4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:24.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.818+0000 7fdeccc00640 1 --2- 172.21.15.6:0/1349545453 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdec806e240 0x7fdec80cb4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:24.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.818+0000 7fdeccc00640 1 -- 172.21.15.6:0/1349545453 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdec809bfa0 con 0x7fdec806cfe0 2024-03-04T09:28:24.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.818+0000 7fdec7e00640 1 --2- 172.21.15.6:0/1349545453 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdec806e240 0x7fdec80cb4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:28:24.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.818+0000 7fdeccc00640 1 --2- 172.21.15.6:0/1349545453 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdec806cfe0 0x7fdec814d6d0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fdebc00d830 tx=0x7fdebc00dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:24.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.818+0000 7fdec4c00640 1 -- 172.21.15.6:0/1349545453 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdebc00b6c0 con 0x7fdec806cfe0 2024-03-04T09:28:24.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.818+0000 7fdec4c00640 1 -- 172.21.15.6:0/1349545453 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdebc019070 con 0x7fdec806cfe0 2024-03-04T09:28:24.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.818+0000 7fdec4c00640 1 -- 172.21.15.6:0/1349545453 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdebc00ba80 con 0x7fdec806cfe0 2024-03-04T09:28:24.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.818+0000 7fdecf600640 1 -- 172.21.15.6:0/1349545453 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdec809b700 con 0x7fdec806cfe0 2024-03-04T09:28:24.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.818+0000 7fdecf600640 1 -- 172.21.15.6:0/1349545453 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdec809bcc0 con 0x7fdec806cfe0 2024-03-04T09:28:24.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.819+0000 7fdecf600640 1 -- 172.21.15.6:0/1349545453 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdec8069510 con 0x7fdec806cfe0 2024-03-04T09:28:24.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.821+0000 7fdec4c00640 1 -- 172.21.15.6:0/1349545453 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdebc01e460 con 0x7fdec806cfe0 2024-03-04T09:28:24.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.822+0000 7fdec4c00640 1 --2- 172.21.15.6:0/1349545453 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fdea806b520 0x7fdea806d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:24.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.822+0000 7fdec4c00640 1 -- 172.21.15.6:0/1349545453 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdebc08a8f0 con 0x7fdec806cfe0 2024-03-04T09:28:24.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.822+0000 7fdec7e00640 1 --2- 172.21.15.6:0/1349545453 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fdea806b520 0x7fdea806d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:24.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.823+0000 7fdec7e00640 1 --2- 172.21.15.6:0/1349545453 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fdea806b520 0x7fdea806d9e0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fdec81f74e0 tx=0x7fdeb4008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:24.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:24.826+0000 7fdec4c00640 1 -- 172.21.15.6:0/1349545453 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdebc059750 con 0x7fdec806cfe0 2024-03-04T09:28:25.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:25.077+0000 7fdecf600640 1 -- 172.21.15.6:0/1349545453 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fdec80b18f0 con 0x7fdec806cfe0 2024-03-04T09:28:25.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:25.078+0000 7fdec4c00640 1 -- 172.21.15.6:0/1349545453 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v34) v1 ==== 53+0+73 (secure 0 0 0) 0x7fdec80b18f0 con 0x7fdec806cfe0 2024-03-04T09:28:25.074 INFO:teuthology.orchestra.run.smithi006.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-04T09:28:25.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:25.080+0000 7fdecf600640 1 -- 172.21.15.6:0/1349545453 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fdea806b520 msgr2=0x7fdea806d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:25.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:25.080+0000 7fdecf600640 1 --2- 172.21.15.6:0/1349545453 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fdea806b520 0x7fdea806d9e0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fdec81f74e0 tx=0x7fdeb4008040 comp rx=0 tx=0).stop 2024-03-04T09:28:25.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:25.080+0000 7fdecf600640 1 -- 172.21.15.6:0/1349545453 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdec806cfe0 msgr2=0x7fdec814d6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:25.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:25.080+0000 7fdecf600640 1 --2- 172.21.15.6:0/1349545453 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdec806cfe0 0x7fdec814d6d0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fdebc00d830 tx=0x7fdebc00dcf0 comp rx=0 tx=0).stop 2024-03-04T09:28:25.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:25.081+0000 7fdecf600640 1 -- 172.21.15.6:0/1349545453 shutdown_connections 2024-03-04T09:28:25.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:25.081+0000 7fdecf600640 1 --2- 172.21.15.6:0/1349545453 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fdea806b520 0x7fdea806d9e0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:25.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:25.081+0000 7fdecf600640 1 --2- 172.21.15.6:0/1349545453 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdec8161670 0x7fdec80cba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:25.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:25.081+0000 7fdecf600640 1 --2- 172.21.15.6:0/1349545453 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdec806e240 0x7fdec80cb4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:25.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:25.081+0000 7fdecf600640 1 --2- 172.21.15.6:0/1349545453 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdec806cfe0 0x7fdec814d6d0 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:25.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:25.081+0000 7fdecf600640 1 -- 172.21.15.6:0/1349545453 >> 172.21.15.6:0/1349545453 conn(0x7fdec806afb0 msgr2=0x7fdec80ae180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:25.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:25.081+0000 7fdecf600640 1 -- 172.21.15.6:0/1349545453 shutdown_connections 2024-03-04T09:28:25.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:25.081+0000 7fdecf600640 1 -- 172.21.15.6:0/1349545453 wait complete. 2024-03-04T09:28:25.098 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-04T09:28:25.098 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-04T09:28:25.098 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi006.front.sepia.ceph.com 2024-03-04T09:28:25.098 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-04T09:28:25.098 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-04T09:28:25.098 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-04T09:28:25.098 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-03-04T09:28:25.098 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-04T09:28:25.098 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-04T09:28:25.098 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:28:25.099 DEBUG:teuthology.orchestra.run.smithi006:> ip addr 2024-03-04T09:28:25.156 INFO:teuthology.orchestra.run.smithi006.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-04T09:28:25.156 INFO:teuthology.orchestra.run.smithi006.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-04T09:28:25.157 INFO:teuthology.orchestra.run.smithi006.stdout: inet 127.0.0.1/8 scope host lo 2024-03-04T09:28:25.157 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-04T09:28:25.157 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 ::1/128 scope host 2024-03-04T09:28:25.157 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-04T09:28:25.157 INFO:teuthology.orchestra.run.smithi006.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-04T09:28:25.157 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:6b:fe:e6 brd ff:ff:ff:ff:ff:ff 2024-03-04T09:28:25.157 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp7s0f0 2024-03-04T09:28:25.157 INFO:teuthology.orchestra.run.smithi006.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-04T09:28:25.157 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:6b:fe:e7 brd ff:ff:ff:ff:ff:ff 2024-03-04T09:28:25.157 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp7s0f1 2024-03-04T09:28:25.157 INFO:teuthology.orchestra.run.smithi006.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-04T09:28:25.157 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:bd:15:e4 brd ff:ff:ff:ff:ff:ff 2024-03-04T09:28:25.157 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp1s0f0 2024-03-04T09:28:25.157 INFO:teuthology.orchestra.run.smithi006.stdout: inet 172.21.15.6/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-04T09:28:25.158 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft 42429sec preferred_lft 42429sec 2024-03-04T09:28:25.158 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::ec4:7aff:febd:15e4/64 scope link noprefixroute 2024-03-04T09:28:25.158 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-04T09:28:25.158 INFO:teuthology.orchestra.run.smithi006.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-04T09:28:25.158 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:bd:15:e5 brd ff:ff:ff:ff:ff:ff 2024-03-04T09:28:25.158 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp1s0f1 2024-03-04T09:28:25.158 INFO:teuthology.orchestra.run.smithi006.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-04T09:28:25.158 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 5e:b4:59:dc:42:58 brd ff:ff:ff:ff:ff:ff 2024-03-04T09:28:25.158 INFO:teuthology.orchestra.run.smithi006.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-04T09:28:25.158 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-04T09:28:25.158 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::c6f:adff:fefe:e00d/64 scope link proto kernel_ll 2024-03-04T09:28:25.158 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-04T09:28:25.158 INFO:teuthology.orchestra.run.smithi006.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-04T09:28:25.158 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 7e:67:6d:ea:f9:ff brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-04T09:28:25.158 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::7c67:6dff:feea:f9ff/64 scope link proto kernel_ll 2024-03-04T09:28:25.159 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-04T09:28:25.159 INFO:teuthology.orchestra.run.smithi006.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-04T09:28:25.159 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 5e:b4:59:dc:42:58 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-04T09:28:25.159 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::5cb4:59ff:fedc:4258/64 scope link proto kernel_ll 2024-03-04T09:28:25.159 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-04T09:28:25.160 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:28:25.160 DEBUG:teuthology.orchestra.run.smithi006:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-04T09:28:25.251 INFO:teuthology.orchestra.run.smithi006.stdout:1 2024-03-04T09:28:25.256 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:28:25.256 DEBUG:teuthology.orchestra.run.smithi006:> route 2024-03-04T09:28:25.313 INFO:teuthology.orchestra.run.smithi006.stdout:Kernel IP routing table 2024-03-04T09:28:25.313 INFO:teuthology.orchestra.run.smithi006.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-04T09:28:25.313 INFO:teuthology.orchestra.run.smithi006.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-04T09:28:25.313 INFO:teuthology.orchestra.run.smithi006.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-04T09:28:25.313 INFO:teuthology.orchestra.run.smithi006.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-04T09:28:25.315 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:28:25.315 DEBUG:teuthology.orchestra.run.smithi006:> (cd / && exec bash -c ' 2024-03-04T09:28:25.315 DEBUG:teuthology.orchestra.run.smithi006:> set -e 2024-03-04T09:28:25.315 DEBUG:teuthology.orchestra.run.smithi006:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-04T09:28:25.315 DEBUG:teuthology.orchestra.run.smithi006:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-04T09:28:25.315 DEBUG:teuthology.orchestra.run.smithi006:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-04T09:28:25.315 DEBUG:teuthology.orchestra.run.smithi006:> ') 2024-03-04T09:28:25.507 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:28:25.507 DEBUG:teuthology.orchestra.run.smithi006:> ip netns list 2024-03-04T09:28:25.563 INFO:teuthology.orchestra.run.smithi006.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-04T09:28:25.563 INFO:teuthology.orchestra.run.smithi006.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-04T09:28:25.564 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-04T09:28:25.564 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:28:25.564 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip link set brx.0 up 2024-03-04T09:28:25.664 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-04T09:28:25.665 DEBUG:teuthology.orchestra.run.smithi006:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-04T09:28:25.722 INFO:teuthology.orchestra.run.smithi006.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-04T09:28:25.722 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-04T09:28:25.723 DEBUG:teuthology.orchestra.run.smithi006:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-04T09:28:25.778 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-03-04T09:28:25.778 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:28:25.778 DEBUG:teuthology.orchestra.run.smithi006:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-04T09:28:25.924 INFO:teuthology.orchestra.run.smithi006.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-04T09:28:25.924 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-04T09:28:25.959 INFO:teuthology.orchestra.run.smithi006.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-04T09:28:25.960 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-04T09:28:25.960 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: resolved to: "172.21.15.6:3300,172.21.15.146:3300,172.21.15.146:3301" 2024-03-04T09:28:25.960 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: trying mount with new device syntax: 0@edce0578-5ae4-4402-b45a-552bdef84d7c.cephfs=/ 2024-03-04T09:28:25.960 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.6:3300/172.21.15.146:3300/172.21.15.146:3301" will pass to kernel 2024-03-04T09:28:25.963 INFO:teuthology.orchestra.run.smithi006.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-04T09:28:25.963 INFO:teuthology.orchestra.run.smithi006.stdout: You just mounted a file system that supports labels which does not 2024-03-04T09:28:25.963 INFO:teuthology.orchestra.run.smithi006.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-04T09:28:25.963 INFO:teuthology.orchestra.run.smithi006.stdout: applications will generate AVC messages and not be allowed access to 2024-03-04T09:28:25.963 INFO:teuthology.orchestra.run.smithi006.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-04T09:28:25.967 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-04T09:28:25.967 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:28:25.967 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-04T09:28:26.056 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:28:26.056 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-04T09:28:26.114 INFO:teuthology.orchestra.run.smithi006.stdout:client5446 2024-03-04T09:28:26.114 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-04T09:28:26.114 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T09:28:26.166 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-04T09:28:26.166 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T09:28:26.215 DEBUG:teuthology.orchestra.run.smithi006:> stdin-killer -- python3 -c ' 2024-03-04T09:28:26.215 DEBUG:teuthology.orchestra.run.smithi006:> import os 2024-03-04T09:28:26.215 DEBUG:teuthology.orchestra.run.smithi006:> import uuid 2024-03-04T09:28:26.215 DEBUG:teuthology.orchestra.run.smithi006:> 2024-03-04T09:28:26.215 DEBUG:teuthology.orchestra.run.smithi006:> n = 75 2024-03-04T09:28:26.215 DEBUG:teuthology.orchestra.run.smithi006:> create_hard_links = False 2024-03-04T09:28:26.216 DEBUG:teuthology.orchestra.run.smithi006:> if 0 > 0: 2024-03-04T09:28:26.216 DEBUG:teuthology.orchestra.run.smithi006:> create_hard_links = True 2024-03-04T09:28:26.216 DEBUG:teuthology.orchestra.run.smithi006:> path = "/home/ubuntu/cephtest/mnt.0/subdir/file" 2024-03-04T09:28:26.216 DEBUG:teuthology.orchestra.run.smithi006:> 2024-03-04T09:28:26.216 DEBUG:teuthology.orchestra.run.smithi006:> dpath = os.path.dirname(path) 2024-03-04T09:28:26.216 DEBUG:teuthology.orchestra.run.smithi006:> fnameprefix = os.path.basename(path) 2024-03-04T09:28:26.216 DEBUG:teuthology.orchestra.run.smithi006:> os.makedirs(dpath, exist_ok=True) 2024-03-04T09:28:26.216 DEBUG:teuthology.orchestra.run.smithi006:> 2024-03-04T09:28:26.216 DEBUG:teuthology.orchestra.run.smithi006:> try: 2024-03-04T09:28:26.216 DEBUG:teuthology.orchestra.run.smithi006:> dirfd = os.open(dpath, os.O_DIRECTORY) 2024-03-04T09:28:26.216 DEBUG:teuthology.orchestra.run.smithi006:> 2024-03-04T09:28:26.216 DEBUG:teuthology.orchestra.run.smithi006:> for i in range(n): 2024-03-04T09:28:26.216 DEBUG:teuthology.orchestra.run.smithi006:> fpath = os.path.join(dpath, f"{fnameprefix}_{i}") 2024-03-04T09:28:26.216 DEBUG:teuthology.orchestra.run.smithi006:> with open(fpath, '"'"'w'"'"') as f: 2024-03-04T09:28:26.217 DEBUG:teuthology.orchestra.run.smithi006:> f.write(f"{i}") 2024-03-04T09:28:26.217 DEBUG:teuthology.orchestra.run.smithi006:> if False: 2024-03-04T09:28:26.217 DEBUG:teuthology.orchestra.run.smithi006:> f.flush() 2024-03-04T09:28:26.217 DEBUG:teuthology.orchestra.run.smithi006:> os.fsync(f.fileno()) 2024-03-04T09:28:26.217 DEBUG:teuthology.orchestra.run.smithi006:> if False: 2024-03-04T09:28:26.217 DEBUG:teuthology.orchestra.run.smithi006:> os.unlink(fpath) 2024-03-04T09:28:26.217 DEBUG:teuthology.orchestra.run.smithi006:> if False: 2024-03-04T09:28:26.217 DEBUG:teuthology.orchestra.run.smithi006:> os.fsync(dirfd) 2024-03-04T09:28:26.217 DEBUG:teuthology.orchestra.run.smithi006:> if create_hard_links: 2024-03-04T09:28:26.217 DEBUG:teuthology.orchestra.run.smithi006:> for j in range(0): 2024-03-04T09:28:26.217 DEBUG:teuthology.orchestra.run.smithi006:> os.system(f"ln {fpath} {dpath}/{fnameprefix}_{i}_{uuid.uuid4()}") 2024-03-04T09:28:26.217 DEBUG:teuthology.orchestra.run.smithi006:> if True: 2024-03-04T09:28:26.217 DEBUG:teuthology.orchestra.run.smithi006:> os.fsync(dirfd) 2024-03-04T09:28:26.217 DEBUG:teuthology.orchestra.run.smithi006:> finally: 2024-03-04T09:28:26.217 DEBUG:teuthology.orchestra.run.smithi006:> os.close(dirfd) 2024-03-04T09:28:26.218 DEBUG:teuthology.orchestra.run.smithi006:> ' 2024-03-04T09:28:27.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:27 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-04T09:28:27.335 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-04T09:28:27.336 DEBUG:teuthology.orchestra.run.smithi006:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-04T09:28:27.500 DEBUG:teuthology.orchestra.run.smithi146:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-04T09:28:27.613 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-04T09:28:27.613 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-04T09:28:27.613 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T09:28:27.675 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-04T09:28:27.675 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T09:28:27.731 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T09:28:27.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:27.960+0000 7f582e400640 1 Processor -- start 2024-03-04T09:28:27.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:27.961+0000 7f582e400640 1 -- start start 2024-03-04T09:28:27.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:27.962+0000 7f582e400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f58280d12e0 0x7f58281658b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:27.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:27.962+0000 7f582e400640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f582815b110 con 0x7f582815ad30 2024-03-04T09:28:27.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:27.962+0000 7f582e400640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58280aed60 con 0x7f58280ae5f0 2024-03-04T09:28:27.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:27.962+0000 7f582e400640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58280af040 con 0x7f58280d12e0 2024-03-04T09:28:27.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:27.963+0000 7f5827400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f582815ad30 0x7f58280b52a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:51842/0 (socket says 172.21.15.6:51842) 2024-03-04T09:28:27.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:27.963+0000 7f5827400640 1 -- 172.21.15.6:0/3682581345 learned_addr learned my addr 172.21.15.6:0/3682581345 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:28:27.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:27.963+0000 7f5827e00640 1 --2- 172.21.15.6:0/3682581345 >> v2:172.21.15.146:3300/0 conn(0x7f58280d12e0 0x7f58281658b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:27.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:27.963+0000 7f5826a00640 1 -- 172.21.15.6:0/3682581345 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3713901676 0 0) 0x7f582815b110 con 0x7f582815ad30 2024-03-04T09:28:27.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:27.964+0000 7f5826a00640 1 -- 172.21.15.6:0/3682581345 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f580c003520 con 0x7f582815ad30 2024-03-04T09:28:27.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:27.964+0000 7f5826a00640 1 -- 172.21.15.6:0/3682581345 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4007859883 0 0) 0x7f580c003520 con 0x7f582815ad30 2024-03-04T09:28:27.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:27.964+0000 7f5826a00640 1 -- 172.21.15.6:0/3682581345 >> v2:172.21.15.146:3300/0 conn(0x7f58280d12e0 msgr2=0x7f58281658b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:27.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:27.964+0000 7f5826a00640 1 --2- 172.21.15.6:0/3682581345 >> v2:172.21.15.146:3300/0 conn(0x7f58280d12e0 0x7f58281658b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:27.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:27.964+0000 7f5826a00640 1 -- 172.21.15.6:0/3682581345 >> v1:172.21.15.146:6789/0 conn(0x7f58280ae5f0 legacy=0x7f58280d0bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:27.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:27.964+0000 7f5826a00640 1 -- 172.21.15.6:0/3682581345 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58280af320 con 0x7f582815ad30 2024-03-04T09:28:27.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:27.964+0000 7f5826a00640 1 -- 172.21.15.6:0/3682581345 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7f5818001670 con 0x7f582815ad30 2024-03-04T09:28:27.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:27.964+0000 7f5826a00640 1 -- 172.21.15.6:0/3682581345 >> v1:172.21.15.6:6789/0 conn(0x7f582815ad30 legacy=0x7f58280b52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:27.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:27.964+0000 7f5826a00640 1 --2- 172.21.15.6:0/3682581345 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f580c003a70 0x7f580c003e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:27.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:27.964+0000 7f5826a00640 1 --2- 172.21.15.6:0/3682581345 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f580c004800 0x7f580c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:27.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:27.964+0000 7f5826a00640 1 --2- 172.21.15.6:0/3682581345 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f580c005210 0x7f580c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:27.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:27.964+0000 7f5826a00640 1 -- 172.21.15.6:0/3682581345 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58280af040 con 0x7f580c003a70 2024-03-04T09:28:27.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:27.964+0000 7f5826a00640 1 -- 172.21.15.6:0/3682581345 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f580c00c140 con 0x7f580c005210 2024-03-04T09:28:27.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:27.964+0000 7f5826a00640 1 -- 172.21.15.6:0/3682581345 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f580c00c340 con 0x7f580c004800 2024-03-04T09:28:27.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:27.964+0000 7f5827400640 1 --2- 172.21.15.6:0/3682581345 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f580c003a70 0x7f580c003e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:27.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:27.965+0000 7f581fe00640 1 --2- 172.21.15.6:0/3682581345 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f580c004800 0x7f580c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:27.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:27.965+0000 7f5827e00640 1 --2- 172.21.15.6:0/3682581345 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f580c005210 0x7f580c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:27.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:27.965+0000 7f5827400640 1 -- 172.21.15.6:0/3682581345 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f580c004800 msgr2=0x7f580c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:27.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:27.965+0000 7f5827400640 1 --2- 172.21.15.6:0/3682581345 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f580c004800 0x7f580c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:27.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:27.965+0000 7f5827400640 1 -- 172.21.15.6:0/3682581345 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f580c005210 msgr2=0x7f580c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:27.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:27.965+0000 7f5827400640 1 --2- 172.21.15.6:0/3682581345 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f580c005210 0x7f580c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:27.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:27.965+0000 7f5827400640 1 -- 172.21.15.6:0/3682581345 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f580c003520 con 0x7f580c003a70 2024-03-04T09:28:27.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:27.965+0000 7f581fe00640 1 --2- 172.21.15.6:0/3682581345 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f580c004800 0x7f580c004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:28:27.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:27.965+0000 7f5827e00640 1 --2- 172.21.15.6:0/3682581345 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f580c005210 0x7f580c00b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:28:27.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:27.965+0000 7f5827e00640 1 -- 172.21.15.6:0/3682581345 reap_dead start 2024-03-04T09:28:27.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:27.966+0000 7f5827400640 1 --2- 172.21.15.6:0/3682581345 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f580c003a70 0x7f580c003e90 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f581800c0e0 tx=0x7f580c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e40f366ab541f65a server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:27.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:27.966+0000 7f5826a00640 1 -- 172.21.15.6:0/3682581345 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5818007c20 con 0x7f580c003a70 2024-03-04T09:28:27.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:27.966+0000 7f5826a00640 1 -- 172.21.15.6:0/3682581345 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5818030070 con 0x7f580c003a70 2024-03-04T09:28:27.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:27.966+0000 7f5826a00640 1 -- 172.21.15.6:0/3682581345 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5818005d70 con 0x7f580c003a70 2024-03-04T09:28:27.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:27.967+0000 7f582e400640 1 -- 172.21.15.6:0/3682581345 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f580c003a70 msgr2=0x7f580c003e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:27.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:27.967+0000 7f582e400640 1 --2- 172.21.15.6:0/3682581345 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f580c003a70 0x7f580c003e90 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f581800c0e0 tx=0x7f580c005830 comp rx=0 tx=0).stop 2024-03-04T09:28:27.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:27.967+0000 7f582e400640 1 -- 172.21.15.6:0/3682581345 shutdown_connections 2024-03-04T09:28:27.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:27.967+0000 7f582e400640 1 --2- 172.21.15.6:0/3682581345 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f580c003a70 0x7f580c003e90 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:27.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:27.967+0000 7f582e400640 1 -- 172.21.15.6:0/3682581345 >> 172.21.15.6:0/3682581345 conn(0x7f582806c2a0 msgr2=0x7f582806c6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:27.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:27.967+0000 7f582e400640 1 -- 172.21.15.6:0/3682581345 shutdown_connections 2024-03-04T09:28:27.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:27.967+0000 7f582e400640 1 -- 172.21.15.6:0/3682581345 wait complete. 2024-03-04T09:28:27.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:27.969+0000 7f582e400640 1 Processor -- start 2024-03-04T09:28:27.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:27.969+0000 7f582e400640 1 -- start start 2024-03-04T09:28:27.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:27.970+0000 7f582e400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f580c003a70 0x7f58281e4130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:27.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:27.970+0000 7f582e400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f58280b03d0 0x7f58281e4670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:27.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:27.970+0000 7f582e400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f58280cc4e0 0x7f58280cc960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:27.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:27.970+0000 7f582e400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58280aea50 con 0x7f580c003a70 2024-03-04T09:28:27.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:27.970+0000 7f582e400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58280af8e0 con 0x7f58280cc4e0 2024-03-04T09:28:27.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:27.970+0000 7f582e400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58280d0bd0 con 0x7f58280b03d0 2024-03-04T09:28:27.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:27.970+0000 7f5827400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f580c003a70 0x7f58281e4130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:27.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:27.970+0000 7f5827400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f580c003a70 0x7f58281e4130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:48260/0 (socket says 172.21.15.6:48260) 2024-03-04T09:28:27.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:27.970+0000 7f5827400640 1 -- 172.21.15.6:0/3550437383 learned_addr learned my addr 172.21.15.6:0/3550437383 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:28:27.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:27.971+0000 7f5827e00640 1 --2- 172.21.15.6:0/3550437383 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f58280cc4e0 0x7f58280cc960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:27.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:27.971+0000 7f581fe00640 1 --2- 172.21.15.6:0/3550437383 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f58280b03d0 0x7f58281e4670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:27.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:27.971+0000 7f5827400640 1 -- 172.21.15.6:0/3550437383 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f58280b03d0 msgr2=0x7f58281e4670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:27.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:27.971+0000 7f5827400640 1 --2- 172.21.15.6:0/3550437383 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f58280b03d0 0x7f58281e4670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:27.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:27.971+0000 7f5827400640 1 -- 172.21.15.6:0/3550437383 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f58280cc4e0 msgr2=0x7f58280cc960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:27.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:27.971+0000 7f5827400640 1 --2- 172.21.15.6:0/3550437383 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f58280cc4e0 0x7f58280cc960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:27.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:27.971+0000 7f5827400640 1 -- 172.21.15.6:0/3550437383 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58281e4dd0 con 0x7f580c003a70 2024-03-04T09:28:27.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:27.971+0000 7f5827400640 1 --2- 172.21.15.6:0/3550437383 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f580c003a70 0x7f58281e4130 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f581800c200 tx=0x7f58280aef30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:27.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:27.972+0000 7f581f400640 1 -- 172.21.15.6:0/3550437383 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5818001b60 con 0x7f580c003a70 2024-03-04T09:28:27.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:27.972+0000 7f581f400640 1 -- 172.21.15.6:0/3550437383 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5818000ed0 con 0x7f580c003a70 2024-03-04T09:28:27.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:27.972+0000 7f581f400640 1 -- 172.21.15.6:0/3550437383 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5818004df0 con 0x7f580c003a70 2024-03-04T09:28:27.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:27.972+0000 7f582e400640 1 -- 172.21.15.6:0/3550437383 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58280cba00 con 0x7f580c003a70 2024-03-04T09:28:27.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:27.972+0000 7f582e400640 1 -- 172.21.15.6:0/3550437383 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58280b2340 con 0x7f580c003a70 2024-03-04T09:28:27.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:27.973+0000 7f582e400640 1 -- 172.21.15.6:0/3550437383 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f582804f3a0 con 0x7f580c003a70 2024-03-04T09:28:27.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:27.974+0000 7f581f400640 1 -- 172.21.15.6:0/3550437383 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5818048020 con 0x7f580c003a70 2024-03-04T09:28:27.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:27.975+0000 7f581f400640 1 --2- 172.21.15.6:0/3550437383 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f580406b4d0 0x7f580406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:27.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:27.975+0000 7f581f400640 1 -- 172.21.15.6:0/3550437383 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f58180ab3b0 con 0x7f580c003a70 2024-03-04T09:28:27.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:27.976+0000 7f581fe00640 1 --2- 172.21.15.6:0/3550437383 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f580406b4d0 0x7f580406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:27.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:27.977+0000 7f581fe00640 1 --2- 172.21.15.6:0/3550437383 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f580406b4d0 0x7f580406d990 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f58280d05e0 tx=0x7f5810009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:27.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:27.979+0000 7f581f400640 1 -- 172.21.15.6:0/3550437383 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f581807a2c0 con 0x7f580c003a70 2024-03-04T09:28:28.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.179+0000 7f582e400640 1 -- 172.21.15.6:0/3550437383 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f582806db20 con 0x7f580c003a70 2024-03-04T09:28:28.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.181+0000 7f581f400640 1 -- 172.21.15.6:0/3550437383 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v56) v1 ==== 74+0+16617 (secure 0 0 0) 0x7f5818079bc0 con 0x7f580c003a70 2024-03-04T09:28:28.178 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:28:28.178 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":56,"fsid":"edce0578-5ae4-4402-b45a-552bdef84d7c","created":"2024-03-04T09:24:54.523780+0000","modified":"2024-03-04T09:28:01.413693+0000","last_up_change":"2024-03-04T09:25:07.089049+0000","last_in_change":"2024-03-04T09:24:57.982077+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T09:25:07.935198+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-04T09:27:57.774529+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-04T09:27:58.886504+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a96574f-81f1-4bc6-8463-757520b84a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6825","nonce":1364449401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6827","nonce":1364449401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6831","nonce":1364449401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6829","nonce":1364449401}]},"public_addr":"172.21.15.6:6825/1364449401","cluster_addr":"172.21.15.6:6827/1364449401","heartbeat_back_addr":"172.21.15.6:6831/1364449401","heartbeat_front_addr":"172.21.15.6:6829/1364449401","state":["exists","up"]},{"osd":1,"uuid":"7d79b9c1-6b87-476a-bc36-7db5ee64cd90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6809","nonce":2494103078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6811","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6813","nonce":2494103078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6821","nonce":2494103078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6817","nonce":2494103078}]},"public_addr":"172.21.15.6:6809/2494103078","cluster_addr":"172.21.15.6:6813/2494103078","heartbeat_back_addr":"172.21.15.6:6821/2494103078","heartbeat_front_addr":"172.21.15.6:6817/2494103078","state":["exists","up"]},{"osd":2,"uuid":"492331b9-4ee9-40b6-bacd-85d44cd6c45a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6812","nonce":1500066681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6816","nonce":1500066681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6824","nonce":1500066681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6820","nonce":1500066681}]},"public_addr":"172.21.15.6:6812/1500066681","cluster_addr":"172.21.15.6:6816/1500066681","heartbeat_back_addr":"172.21.15.6:6824/1500066681","heartbeat_front_addr":"172.21.15.6:6820/1500066681","state":["exists","up"]},{"osd":3,"uuid":"21bb2efe-9ba7-4bdc-bc31-180cd1102e21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6801","nonce":1558806123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6803","nonce":1558806123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6807","nonce":1558806123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6805","nonce":1558806123}]},"public_addr":"172.21.15.6:6801/1558806123","cluster_addr":"172.21.15.6:6803/1558806123","heartbeat_back_addr":"172.21.15.6:6807/1558806123","heartbeat_front_addr":"172.21.15.6:6805/1558806123","state":["exists","up"]},{"osd":4,"uuid":"6e2c4b71-19d9-4737-83d9-558705d8644a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6801","nonce":1526135780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6803","nonce":1526135780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6807","nonce":1526135780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6805","nonce":1526135780}]},"public_addr":"172.21.15.146:6801/1526135780","cluster_addr":"172.21.15.146:6803/1526135780","heartbeat_back_addr":"172.21.15.146:6807/1526135780","heartbeat_front_addr":"172.21.15.146:6805/1526135780","state":["exists","up"]},{"osd":5,"uuid":"fb69c525-f1be-4b7d-ab32-6c29b0d6fddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6809","nonce":460262507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6811","nonce":460262507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6815","nonce":460262507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6813","nonce":460262507}]},"public_addr":"172.21.15.146:6809/460262507","cluster_addr":"172.21.15.146:6811/460262507","heartbeat_back_addr":"172.21.15.146:6815/460262507","heartbeat_front_addr":"172.21.15.146:6813/460262507","state":["exists","up"]},{"osd":6,"uuid":"c32e7fed-370d-4401-8ce0-6a4335b0d168","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6823","nonce":1392897493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6827","nonce":1392897493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6831","nonce":1392897493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6829","nonce":1392897493}]},"public_addr":"172.21.15.146:6823/1392897493","cluster_addr":"172.21.15.146:6827/1392897493","heartbeat_back_addr":"172.21.15.146:6831/1392897493","heartbeat_front_addr":"172.21.15.146:6829/1392897493","state":["exists","up"]},{"osd":7,"uuid":"75a79b44-fa5f-426d-a7d3-71f21f78835c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6817","nonce":3268323156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6819","nonce":3268323156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6825","nonce":3268323156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6822","nonce":3268323156}]},"public_addr":"172.21.15.146:6817/3268323156","cluster_addr":"172.21.15.146:6819/3268323156","heartbeat_back_addr":"172.21.15.146:6825/3268323156","heartbeat_front_addr":"172.21.15.146:6822/3268323156","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.310437+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.295850+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.294674+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.526855+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.300023+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.595160+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.346046+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.558741+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T09:28:28.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.184+0000 7f582e400640 1 -- 172.21.15.6:0/3550437383 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f580406b4d0 msgr2=0x7f580406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:28.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.184+0000 7f582e400640 1 --2- 172.21.15.6:0/3550437383 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f580406b4d0 0x7f580406d990 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f58280d05e0 tx=0x7f5810009040 comp rx=0 tx=0).stop 2024-03-04T09:28:28.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.184+0000 7f582e400640 1 -- 172.21.15.6:0/3550437383 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f580c003a70 msgr2=0x7f58281e4130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:28.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.184+0000 7f582e400640 1 --2- 172.21.15.6:0/3550437383 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f580c003a70 0x7f58281e4130 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f581800c200 tx=0x7f58280aef30 comp rx=0 tx=0).stop 2024-03-04T09:28:28.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.184+0000 7f582e400640 1 -- 172.21.15.6:0/3550437383 shutdown_connections 2024-03-04T09:28:28.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.184+0000 7f582e400640 1 --2- 172.21.15.6:0/3550437383 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f58280cc4e0 0x7f58280cc960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:28.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.184+0000 7f582e400640 1 --2- 172.21.15.6:0/3550437383 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f580406b4d0 0x7f580406d990 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:28.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.184+0000 7f582e400640 1 --2- 172.21.15.6:0/3550437383 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f58280b03d0 0x7f58281e4670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:28.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.184+0000 7f582e400640 1 --2- 172.21.15.6:0/3550437383 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f580c003a70 0x7f58281e4130 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:28.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.184+0000 7f582e400640 1 -- 172.21.15.6:0/3550437383 >> 172.21.15.6:0/3550437383 conn(0x7f582806c2a0 msgr2=0x7f58281597e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:28.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.184+0000 7f582e400640 1 -- 172.21.15.6:0/3550437383 shutdown_connections 2024-03-04T09:28:28.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.184+0000 7f582e400640 1 -- 172.21.15.6:0/3550437383 wait complete. 2024-03-04T09:28:28.201 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T09:25:07.935198+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-04T09:27:57.774529+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '55', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2024-03-04T09:27:58.886504+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '56', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-04T09:28:28.202 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-04T09:28:28.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.428+0000 7f31d7a00640 1 Processor -- start 2024-03-04T09:28:28.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.428+0000 7f31d7a00640 1 -- start start 2024-03-04T09:28:28.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.429+0000 7f31d7a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f31d01615b0 0x7f31d00d47f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:28.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.429+0000 7f31d7a00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f31d00d1a80 0x7f31d00d22d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:28.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.429+0000 7f31d7a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31d00cf670 con 0x7f31d00d2810 2024-03-04T09:28:28.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.429+0000 7f31d7a00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31d0161990 con 0x7f31d01615b0 2024-03-04T09:28:28.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.429+0000 7f31d7a00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31d00d1ff0 con 0x7f31d00d1a80 2024-03-04T09:28:28.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.429+0000 7f31d5000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f31d01615b0 0x7f31d00d47f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:28.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.429+0000 7f31d5000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f31d01615b0 0x7f31d00d47f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:48272/0 (socket says 172.21.15.6:48272) 2024-03-04T09:28:28.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.430+0000 7f31d5000640 1 -- 172.21.15.6:0/2209058496 learned_addr learned my addr 172.21.15.6:0/2209058496 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:28:28.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.430+0000 7f31cfe00640 1 --2- 172.21.15.6:0/2209058496 >> v2:172.21.15.146:3300/0 conn(0x7f31d00d1a80 0x7f31d00d22d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:28.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.430+0000 7f31d5000640 1 -- 172.21.15.6:0/2209058496 >> v2:172.21.15.146:3300/0 conn(0x7f31d00d1a80 msgr2=0x7f31d00d22d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:28.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.430+0000 7f31d5000640 1 --2- 172.21.15.6:0/2209058496 >> v2:172.21.15.146:3300/0 conn(0x7f31d00d1a80 0x7f31d00d22d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:28.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.430+0000 7f31d5000640 1 -- 172.21.15.6:0/2209058496 >> v1:172.21.15.6:6789/0 conn(0x7f31d00d2810 legacy=0x7f31d00d06f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:28.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.430+0000 7f31d5000640 1 -- 172.21.15.6:0/2209058496 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31d00cf910 con 0x7f31d01615b0 2024-03-04T09:28:28.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.431+0000 7f31d5000640 1 --2- 172.21.15.6:0/2209058496 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f31d01615b0 0x7f31d00d47f0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f31d008e280 tx=0x7f31c002bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ef043e2e4312a01 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:28.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.431+0000 7f31cf400640 1 -- 172.21.15.6:0/2209058496 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31c002cb10 con 0x7f31d01615b0 2024-03-04T09:28:28.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.431+0000 7f31cf400640 1 -- 172.21.15.6:0/2209058496 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31c002ccb0 con 0x7f31d01615b0 2024-03-04T09:28:28.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.431+0000 7f31cf400640 1 -- 172.21.15.6:0/2209058496 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31c002ce60 con 0x7f31d01615b0 2024-03-04T09:28:28.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.432+0000 7f31d7a00640 1 -- 172.21.15.6:0/2209058496 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f31d01615b0 msgr2=0x7f31d00d47f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:28.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.432+0000 7f31d7a00640 1 --2- 172.21.15.6:0/2209058496 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f31d01615b0 0x7f31d00d47f0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f31d008e280 tx=0x7f31c002bf70 comp rx=0 tx=0).stop 2024-03-04T09:28:28.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.432+0000 7f31d7a00640 1 -- 172.21.15.6:0/2209058496 shutdown_connections 2024-03-04T09:28:28.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.432+0000 7f31d7a00640 1 --2- 172.21.15.6:0/2209058496 >> v2:172.21.15.146:3300/0 conn(0x7f31d00d1a80 0x7f31d00d22d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:28.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.432+0000 7f31d7a00640 1 --2- 172.21.15.6:0/2209058496 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f31d01615b0 0x7f31d00d47f0 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:28.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.432+0000 7f31d7a00640 1 -- 172.21.15.6:0/2209058496 >> 172.21.15.6:0/2209058496 conn(0x7f31d009d180 msgr2=0x7f31d00d3eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:28.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.432+0000 7f31d7a00640 1 -- 172.21.15.6:0/2209058496 shutdown_connections 2024-03-04T09:28:28.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.432+0000 7f31d7a00640 1 -- 172.21.15.6:0/2209058496 wait complete. 2024-03-04T09:28:28.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.434+0000 7f31d7a00640 1 Processor -- start 2024-03-04T09:28:28.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.434+0000 7f31d7a00640 1 -- start start 2024-03-04T09:28:28.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.435+0000 7f31d7a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f31d00d1a80 0x7f31d006cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:28.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.435+0000 7f31d7a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f31d00d2810 0x7f31d015b690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:28.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.435+0000 7f31d7a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f31d01615b0 0x7f31d00b6ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:28.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.435+0000 7f31d7a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31d00d2ca0 con 0x7f31d01615b0 2024-03-04T09:28:28.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.435+0000 7f31d7a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31d00d4d30 con 0x7f31d00d2810 2024-03-04T09:28:28.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.435+0000 7f31d7a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31d00d2e20 con 0x7f31d00d1a80 2024-03-04T09:28:28.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.435+0000 7f31d5a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f31d01615b0 0x7f31d00b6ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:28.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.436+0000 7f31d5a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f31d01615b0 0x7f31d00b6ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:48274/0 (socket says 172.21.15.6:48274) 2024-03-04T09:28:28.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.436+0000 7f31d5a00640 1 -- 172.21.15.6:0/180654036 learned_addr learned my addr 172.21.15.6:0/180654036 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:28:28.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.436+0000 7f31cfe00640 1 --2- 172.21.15.6:0/180654036 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f31d00d2810 0x7f31d015b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:28.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.436+0000 7f31d5000640 1 --2- 172.21.15.6:0/180654036 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f31d00d1a80 0x7f31d006cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:28.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.436+0000 7f31d5a00640 1 -- 172.21.15.6:0/180654036 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f31d00d1a80 msgr2=0x7f31d006cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:28.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.436+0000 7f31d5a00640 1 --2- 172.21.15.6:0/180654036 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f31d00d1a80 0x7f31d006cb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:28.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.436+0000 7f31d5a00640 1 -- 172.21.15.6:0/180654036 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f31d00d2810 msgr2=0x7f31d015b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:28.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.436+0000 7f31d5a00640 1 --2- 172.21.15.6:0/180654036 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f31d00d2810 0x7f31d015b690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:28.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.436+0000 7f31d5a00640 1 -- 172.21.15.6:0/180654036 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31d0159700 con 0x7f31d01615b0 2024-03-04T09:28:28.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.436+0000 7f31d5a00640 1 --2- 172.21.15.6:0/180654036 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f31d01615b0 0x7f31d00b6ab0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f31c800c3f0 tx=0x7f31c800c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:28.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.437+0000 7f31ccc00640 1 -- 172.21.15.6:0/180654036 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31c80052d0 con 0x7f31d01615b0 2024-03-04T09:28:28.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.437+0000 7f31ccc00640 1 -- 172.21.15.6:0/180654036 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31c8005490 con 0x7f31d01615b0 2024-03-04T09:28:28.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.437+0000 7f31ccc00640 1 -- 172.21.15.6:0/180654036 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31c8007260 con 0x7f31d01615b0 2024-03-04T09:28:28.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.437+0000 7f31d7a00640 1 -- 172.21.15.6:0/180654036 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31d00a15c0 con 0x7f31d01615b0 2024-03-04T09:28:28.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.437+0000 7f31d7a00640 1 -- 172.21.15.6:0/180654036 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31d0157610 con 0x7f31d01615b0 2024-03-04T09:28:28.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.438+0000 7f31d7a00640 1 -- 172.21.15.6:0/180654036 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31d00a1ae0 con 0x7f31d01615b0 2024-03-04T09:28:28.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.439+0000 7f31ccc00640 1 -- 172.21.15.6:0/180654036 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f31c8007400 con 0x7f31d01615b0 2024-03-04T09:28:28.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.440+0000 7f31ccc00640 1 --2- 172.21.15.6:0/180654036 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f31ac06b500 0x7f31ac06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:28.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.440+0000 7f31ccc00640 1 -- 172.21.15.6:0/180654036 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f31c8089730 con 0x7f31d01615b0 2024-03-04T09:28:28.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.441+0000 7f31d5000640 1 --2- 172.21.15.6:0/180654036 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f31ac06b500 0x7f31ac06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:28.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.442+0000 7f31d5000640 1 --2- 172.21.15.6:0/180654036 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f31ac06b500 0x7f31ac06d9c0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f31c0035000 tx=0x7f31c0030000 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:28.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.444+0000 7f31ccc00640 1 -- 172.21.15.6:0/180654036 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f31d00a1ae0 con 0x7f31d01615b0 2024-03-04T09:28:28.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.644+0000 7f31d7a00640 1 -- 172.21.15.6:0/180654036 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f31d0157e80 con 0x7f31d01615b0 2024-03-04T09:28:28.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.645+0000 7f31ccc00640 1 -- 172.21.15.6:0/180654036 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v56) v1 ==== 93+0+10 (secure 0 0 0) 0x7f31c8058510 con 0x7f31d01615b0 2024-03-04T09:28:28.642 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-04T09:28:28.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.647+0000 7f31d7a00640 1 -- 172.21.15.6:0/180654036 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f31ac06b500 msgr2=0x7f31ac06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:28.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.647+0000 7f31d7a00640 1 --2- 172.21.15.6:0/180654036 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f31ac06b500 0x7f31ac06d9c0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f31c0035000 tx=0x7f31c0030000 comp rx=0 tx=0).stop 2024-03-04T09:28:28.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.647+0000 7f31d7a00640 1 -- 172.21.15.6:0/180654036 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f31d01615b0 msgr2=0x7f31d00b6ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:28.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.647+0000 7f31d7a00640 1 --2- 172.21.15.6:0/180654036 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f31d01615b0 0x7f31d00b6ab0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f31c800c3f0 tx=0x7f31c800c8b0 comp rx=0 tx=0).stop 2024-03-04T09:28:28.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.648+0000 7f31d7a00640 1 -- 172.21.15.6:0/180654036 shutdown_connections 2024-03-04T09:28:28.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.648+0000 7f31d7a00640 1 --2- 172.21.15.6:0/180654036 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f31d01615b0 0x7f31d00b6ab0 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:28.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.648+0000 7f31d7a00640 1 --2- 172.21.15.6:0/180654036 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f31d00d2810 0x7f31d015b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:28.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.648+0000 7f31d7a00640 1 --2- 172.21.15.6:0/180654036 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f31ac06b500 0x7f31ac06d9c0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:28.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.648+0000 7f31d7a00640 1 --2- 172.21.15.6:0/180654036 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f31d00d1a80 0x7f31d006cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:28.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.648+0000 7f31d7a00640 1 -- 172.21.15.6:0/180654036 >> 172.21.15.6:0/180654036 conn(0x7f31d009d180 msgr2=0x7f31d015a3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:28.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.648+0000 7f31d7a00640 1 -- 172.21.15.6:0/180654036 shutdown_connections 2024-03-04T09:28:28.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.648+0000 7f31d7a00640 1 -- 172.21.15.6:0/180654036 wait complete. 2024-03-04T09:28:28.664 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-04T09:28:28.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.894+0000 7efeda800640 1 Processor -- start 2024-03-04T09:28:28.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.895+0000 7efeda800640 1 -- start start 2024-03-04T09:28:28.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.896+0000 7efeda800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7efed40d4860 0x7efed40d0630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:28.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.896+0000 7efeda800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efed40cfa60 0x7efed40d0b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:28.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.896+0000 7efeda800640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efed40ceca0 con 0x7efed415dee0 2024-03-04T09:28:28.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.896+0000 7efeda800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efed40cf5d0 con 0x7efed40d4860 2024-03-04T09:28:28.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.896+0000 7efeda800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efed40cef80 con 0x7efed40cfa60 2024-03-04T09:28:28.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.897+0000 7efed8a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7efed40d4860 0x7efed40d0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:28.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.897+0000 7efed9400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7efed415dee0 0x7efed40d18a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:41112/0 (socket says 172.21.15.6:41112) 2024-03-04T09:28:28.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.897+0000 7efed8a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7efed40d4860 0x7efed40d0630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53582/0 (socket says 172.21.15.6:53582) 2024-03-04T09:28:28.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.897+0000 7efed9400640 1 -- 172.21.15.6:0/3524245434 learned_addr learned my addr 172.21.15.6:0/3524245434 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:28:28.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.897+0000 7efed9e00640 1 --2- 172.21.15.6:0/3524245434 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efed40cfa60 0x7efed40d0b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:28.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.897+0000 7efed8a00640 1 -- 172.21.15.6:0/3524245434 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efed40cfa60 msgr2=0x7efed40d0b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:28.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.897+0000 7efed8a00640 1 --2- 172.21.15.6:0/3524245434 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efed40cfa60 0x7efed40d0b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:28.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.897+0000 7efed8a00640 1 -- 172.21.15.6:0/3524245434 >> v1:172.21.15.6:6789/0 conn(0x7efed415dee0 legacy=0x7efed40d18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:28.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.897+0000 7efed8a00640 1 -- 172.21.15.6:0/3524245434 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efed40cf260 con 0x7efed40d4860 2024-03-04T09:28:28.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.897+0000 7efed9e00640 1 --2- 172.21.15.6:0/3524245434 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efed40cfa60 0x7efed40d0b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:28:28.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.898+0000 7efed8a00640 1 --2- 172.21.15.6:0/3524245434 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efed40d4860 0x7efed40d0630 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7efec00097a0 tx=0x7efec002b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c22ba9bf787f6784 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:28.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.898+0000 7efed3e00640 1 -- 172.21.15.6:0/3524245434 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efec0035020 con 0x7efed40d4860 2024-03-04T09:28:28.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.899+0000 7efed3e00640 1 -- 172.21.15.6:0/3524245434 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efec0031810 con 0x7efed40d4860 2024-03-04T09:28:28.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.899+0000 7efed3e00640 1 -- 172.21.15.6:0/3524245434 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efec0031be0 con 0x7efed40d4860 2024-03-04T09:28:28.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.899+0000 7efeda800640 1 -- 172.21.15.6:0/3524245434 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efed40d4860 msgr2=0x7efed40d0630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:28.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.899+0000 7efeda800640 1 --2- 172.21.15.6:0/3524245434 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efed40d4860 0x7efed40d0630 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7efec00097a0 tx=0x7efec002b8b0 comp rx=0 tx=0).stop 2024-03-04T09:28:28.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.899+0000 7efeda800640 1 -- 172.21.15.6:0/3524245434 shutdown_connections 2024-03-04T09:28:28.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.899+0000 7efeda800640 1 --2- 172.21.15.6:0/3524245434 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efed40cfa60 0x7efed40d0b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:28.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.899+0000 7efeda800640 1 --2- 172.21.15.6:0/3524245434 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efed40d4860 0x7efed40d0630 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:28.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.899+0000 7efeda800640 1 -- 172.21.15.6:0/3524245434 >> 172.21.15.6:0/3524245434 conn(0x7efed40d3cd0 msgr2=0x7efed40d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:28.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.899+0000 7efeda800640 1 -- 172.21.15.6:0/3524245434 shutdown_connections 2024-03-04T09:28:28.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.899+0000 7efeda800640 1 -- 172.21.15.6:0/3524245434 wait complete. 2024-03-04T09:28:28.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.901+0000 7efeda800640 1 Processor -- start 2024-03-04T09:28:28.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.901+0000 7efeda800640 1 -- start start 2024-03-04T09:28:28.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.902+0000 7efeda800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efed40cfa60 0x7efed41f3f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:28.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.902+0000 7efeda800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efed40d4860 0x7efed40cada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:28.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.902+0000 7efeda800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efed415dee0 0x7efed41510b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:28.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.902+0000 7efeda800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efed40b3510 con 0x7efed415dee0 2024-03-04T09:28:28.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.902+0000 7efeda800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efed415e2c0 con 0x7efed40cfa60 2024-03-04T09:28:28.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.902+0000 7efeda800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efed40b3690 con 0x7efed40d4860 2024-03-04T09:28:28.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.902+0000 7efed9e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efed415dee0 0x7efed41510b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:28.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.902+0000 7efed9e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efed415dee0 0x7efed41510b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53596/0 (socket says 172.21.15.6:53596) 2024-03-04T09:28:28.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.902+0000 7efed9e00640 1 -- 172.21.15.6:0/2401558412 learned_addr learned my addr 172.21.15.6:0/2401558412 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:28:28.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.903+0000 7efed8a00640 1 --2- 172.21.15.6:0/2401558412 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efed40d4860 0x7efed40cada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:28.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.903+0000 7efed9400640 1 --2- 172.21.15.6:0/2401558412 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efed40cfa60 0x7efed41f3f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:28.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.903+0000 7efed9e00640 1 -- 172.21.15.6:0/2401558412 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efed40d4860 msgr2=0x7efed40cada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:28.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.903+0000 7efed9e00640 1 --2- 172.21.15.6:0/2401558412 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efed40d4860 0x7efed40cada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:28.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.903+0000 7efed9e00640 1 -- 172.21.15.6:0/2401558412 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efed40cfa60 msgr2=0x7efed41f3f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:28.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.903+0000 7efed9e00640 1 --2- 172.21.15.6:0/2401558412 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efed40cfa60 0x7efed41f3f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:28.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.903+0000 7efed9e00640 1 -- 172.21.15.6:0/2401558412 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efed40cb630 con 0x7efed415dee0 2024-03-04T09:28:28.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.903+0000 7efed9e00640 1 --2- 172.21.15.6:0/2401558412 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efed415dee0 0x7efed41510b0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7efecc00c7f0 tx=0x7efecc00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:28.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.903+0000 7efed1600640 1 -- 172.21.15.6:0/2401558412 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efecc015070 con 0x7efed415dee0 2024-03-04T09:28:28.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.903+0000 7efed1600640 1 -- 172.21.15.6:0/2401558412 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efecc007820 con 0x7efed415dee0 2024-03-04T09:28:28.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.903+0000 7efeda800640 1 -- 172.21.15.6:0/2401558412 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efed40cb910 con 0x7efed415dee0 2024-03-04T09:28:28.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.903+0000 7efed1600640 1 -- 172.21.15.6:0/2401558412 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efecc007c50 con 0x7efed415dee0 2024-03-04T09:28:28.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.904+0000 7efeda800640 1 -- 172.21.15.6:0/2401558412 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efed40cbed0 con 0x7efed415dee0 2024-03-04T09:28:28.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.905+0000 7efeda800640 1 -- 172.21.15.6:0/2401558412 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efed404f330 con 0x7efed415dee0 2024-03-04T09:28:28.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.906+0000 7efed1600640 1 -- 172.21.15.6:0/2401558412 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efecc018070 con 0x7efed415dee0 2024-03-04T09:28:28.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.907+0000 7efed1600640 1 --2- 172.21.15.6:0/2401558412 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7efeb406b4d0 0x7efeb406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:28.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.907+0000 7efed1600640 1 -- 172.21.15.6:0/2401558412 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efecc08a4a0 con 0x7efed415dee0 2024-03-04T09:28:28.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.908+0000 7efed9400640 1 --2- 172.21.15.6:0/2401558412 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7efeb406b4d0 0x7efeb406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:28.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.909+0000 7efed9400640 1 --2- 172.21.15.6:0/2401558412 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7efeb406b4d0 0x7efeb406d990 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7efed41f73d0 tx=0x7efec8009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:28.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:28.912+0000 7efed1600640 1 -- 172.21.15.6:0/2401558412 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7efecc059280 con 0x7efed415dee0 2024-03-04T09:28:29.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.114+0000 7efeda800640 1 -- 172.21.15.6:0/2401558412 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7efed40d0820 con 0x7efed415dee0 2024-03-04T09:28:29.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.114+0000 7efed1600640 1 -- 172.21.15.6:0/2401558412 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v56) v1 ==== 104+0+11 (secure 0 0 0) 0x7efecc058b80 con 0x7efed415dee0 2024-03-04T09:28:29.112 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-03-04T09:28:29.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.117+0000 7efeda800640 1 -- 172.21.15.6:0/2401558412 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7efeb406b4d0 msgr2=0x7efeb406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:29.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.117+0000 7efeda800640 1 --2- 172.21.15.6:0/2401558412 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7efeb406b4d0 0x7efeb406d990 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7efed41f73d0 tx=0x7efec8009040 comp rx=0 tx=0).stop 2024-03-04T09:28:29.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.117+0000 7efeda800640 1 -- 172.21.15.6:0/2401558412 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efed415dee0 msgr2=0x7efed41510b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:29.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.117+0000 7efeda800640 1 --2- 172.21.15.6:0/2401558412 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efed415dee0 0x7efed41510b0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7efecc00c7f0 tx=0x7efecc00ccb0 comp rx=0 tx=0).stop 2024-03-04T09:28:29.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.118+0000 7efeda800640 1 -- 172.21.15.6:0/2401558412 shutdown_connections 2024-03-04T09:28:29.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.118+0000 7efeda800640 1 --2- 172.21.15.6:0/2401558412 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efed415dee0 0x7efed41510b0 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:29.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.118+0000 7efeda800640 1 --2- 172.21.15.6:0/2401558412 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7efeb406b4d0 0x7efeb406d990 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:29.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.118+0000 7efeda800640 1 --2- 172.21.15.6:0/2401558412 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efed40d4860 0x7efed40cada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:29.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.118+0000 7efeda800640 1 --2- 172.21.15.6:0/2401558412 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efed40cfa60 0x7efed41f3f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:29.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.118+0000 7efeda800640 1 -- 172.21.15.6:0/2401558412 >> 172.21.15.6:0/2401558412 conn(0x7efed40d3cd0 msgr2=0x7efed40b0c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:29.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.118+0000 7efeda800640 1 -- 172.21.15.6:0/2401558412 shutdown_connections 2024-03-04T09:28:29.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.118+0000 7efeda800640 1 -- 172.21.15.6:0/2401558412 wait complete. 2024-03-04T09:28:29.134 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-04T09:28:29.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.359+0000 7f2e8f200640 1 Processor -- start 2024-03-04T09:28:29.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.359+0000 7f2e8f200640 1 -- start start 2024-03-04T09:28:29.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.361+0000 7f2e8f200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f2e900729d0 0x7f2e900b4000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:29.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.361+0000 7f2e8f200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2e900b1b40 0x7f2e900b1fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:29.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.361+0000 7f2e8f200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e90071930 con 0x7f2e900b3c20 2024-03-04T09:28:29.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.361+0000 7f2e8f200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e900b24e0 con 0x7f2e900729d0 2024-03-04T09:28:29.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.361+0000 7f2e8f200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e9015bfb0 con 0x7f2e900b1b40 2024-03-04T09:28:29.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.361+0000 7f2e8de00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f2e900b3c20 0x7f2e9015b8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:41126/0 (socket says 172.21.15.6:41126) 2024-03-04T09:28:29.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.361+0000 7f2e8de00640 1 -- 172.21.15.6:0/1592381044 learned_addr learned my addr 172.21.15.6:0/1592381044 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:28:29.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.361+0000 7f2e85400640 1 --2- 172.21.15.6:0/1592381044 >> v2:172.21.15.6:3300/0 conn(0x7f2e900729d0 0x7f2e900b4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:29.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.362+0000 7f2e8e800640 1 --2- 172.21.15.6:0/1592381044 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2e900b1b40 0x7f2e900b1fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:29.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.362+0000 7f2e8d400640 1 -- 172.21.15.6:0/1592381044 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 837154128 0 0) 0x7f2e90071930 con 0x7f2e900b3c20 2024-03-04T09:28:29.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.362+0000 7f2e8d400640 1 -- 172.21.15.6:0/1592381044 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e74005730 con 0x7f2e900b3c20 2024-03-04T09:28:29.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.362+0000 7f2e85400640 1 -- 172.21.15.6:0/1592381044 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2e900b1b40 msgr2=0x7f2e900b1fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:29.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.362+0000 7f2e85400640 1 --2- 172.21.15.6:0/1592381044 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2e900b1b40 0x7f2e900b1fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:29.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.362+0000 7f2e85400640 1 -- 172.21.15.6:0/1592381044 >> v1:172.21.15.6:6789/0 conn(0x7f2e900b3c20 legacy=0x7f2e9015b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:29.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.362+0000 7f2e85400640 1 -- 172.21.15.6:0/1592381044 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e90071c10 con 0x7f2e900729d0 2024-03-04T09:28:29.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.363+0000 7f2e85400640 1 --2- 172.21.15.6:0/1592381044 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2e900729d0 0x7f2e900b4000 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f2e78009d90 tx=0x7f2e7802bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90c0f8de97242bd7 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:29.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.363+0000 7f2e8d400640 1 -- 172.21.15.6:0/1592381044 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e78039070 con 0x7f2e900729d0 2024-03-04T09:28:29.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.363+0000 7f2e8d400640 1 -- 172.21.15.6:0/1592381044 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e7802cc50 con 0x7f2e900729d0 2024-03-04T09:28:29.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.363+0000 7f2e8d400640 1 -- 172.21.15.6:0/1592381044 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e78034440 con 0x7f2e900729d0 2024-03-04T09:28:29.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.364+0000 7f2e8f200640 1 -- 172.21.15.6:0/1592381044 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2e900729d0 msgr2=0x7f2e900b4000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:29.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.364+0000 7f2e8f200640 1 --2- 172.21.15.6:0/1592381044 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2e900729d0 0x7f2e900b4000 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f2e78009d90 tx=0x7f2e7802bea0 comp rx=0 tx=0).stop 2024-03-04T09:28:29.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.364+0000 7f2e8f200640 1 -- 172.21.15.6:0/1592381044 shutdown_connections 2024-03-04T09:28:29.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.364+0000 7f2e8f200640 1 --2- 172.21.15.6:0/1592381044 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2e900b1b40 0x7f2e900b1fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:29.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.364+0000 7f2e8f200640 1 --2- 172.21.15.6:0/1592381044 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2e900729d0 0x7f2e900b4000 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:29.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.364+0000 7f2e8f200640 1 -- 172.21.15.6:0/1592381044 >> 172.21.15.6:0/1592381044 conn(0x7f2e9006b3a0 msgr2=0x7f2e9006b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:29.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.364+0000 7f2e8f200640 1 -- 172.21.15.6:0/1592381044 shutdown_connections 2024-03-04T09:28:29.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.364+0000 7f2e8f200640 1 -- 172.21.15.6:0/1592381044 wait complete. 2024-03-04T09:28:29.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.366+0000 7f2e8f200640 1 Processor -- start 2024-03-04T09:28:29.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.366+0000 7f2e8f200640 1 -- start start 2024-03-04T09:28:29.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.367+0000 7f2e8f200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2e900729d0 0x7f2e90150b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:29.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.367+0000 7f2e8f200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2e900b1b40 0x7f2e90158140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:29.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.367+0000 7f2e8f200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2e900b3c20 0x7f2e9014f340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:29.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.367+0000 7f2e8f200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e900724b0 con 0x7f2e900b1b40 2024-03-04T09:28:29.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.367+0000 7f2e8f200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e900721d0 con 0x7f2e900729d0 2024-03-04T09:28:29.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.367+0000 7f2e8f200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e9015c220 con 0x7f2e900b3c20 2024-03-04T09:28:29.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.367+0000 7f2e85400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2e900b1b40 0x7f2e90158140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:29.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.367+0000 7f2e85400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2e900b1b40 0x7f2e90158140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53616/0 (socket says 172.21.15.6:53616) 2024-03-04T09:28:29.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.367+0000 7f2e85400640 1 -- 172.21.15.6:0/18115062 learned_addr learned my addr 172.21.15.6:0/18115062 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:28:29.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.368+0000 7f2e8de00640 1 --2- 172.21.15.6:0/18115062 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2e900729d0 0x7f2e90150b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:29.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.368+0000 7f2e8e800640 1 --2- 172.21.15.6:0/18115062 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2e900b3c20 0x7f2e9014f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:29.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.368+0000 7f2e85400640 1 -- 172.21.15.6:0/18115062 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2e900b3c20 msgr2=0x7f2e9014f340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:29.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.368+0000 7f2e85400640 1 --2- 172.21.15.6:0/18115062 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2e900b3c20 0x7f2e9014f340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:29.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.368+0000 7f2e85400640 1 -- 172.21.15.6:0/18115062 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2e900729d0 msgr2=0x7f2e90150b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:29.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.368+0000 7f2e85400640 1 --2- 172.21.15.6:0/18115062 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2e900729d0 0x7f2e90150b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:29.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.368+0000 7f2e85400640 1 -- 172.21.15.6:0/18115062 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e9014fad0 con 0x7f2e900b1b40 2024-03-04T09:28:29.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.368+0000 7f2e85400640 1 --2- 172.21.15.6:0/18115062 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2e900b1b40 0x7f2e90158140 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f2e78037040 tx=0x7f2e78004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:29.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.368+0000 7f2e86a00640 1 -- 172.21.15.6:0/18115062 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e78039050 con 0x7f2e900b1b40 2024-03-04T09:28:29.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.369+0000 7f2e86a00640 1 -- 172.21.15.6:0/18115062 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e78045070 con 0x7f2e900b1b40 2024-03-04T09:28:29.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.369+0000 7f2e86a00640 1 -- 172.21.15.6:0/18115062 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e7803e610 con 0x7f2e900b1b40 2024-03-04T09:28:29.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.369+0000 7f2e8f200640 1 -- 172.21.15.6:0/18115062 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e9014fdb0 con 0x7f2e900b1b40 2024-03-04T09:28:29.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.369+0000 7f2e8f200640 1 -- 172.21.15.6:0/18115062 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e90150370 con 0x7f2e900b1b40 2024-03-04T09:28:29.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.370+0000 7f2e8f200640 1 -- 172.21.15.6:0/18115062 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e9004f330 con 0x7f2e900b1b40 2024-03-04T09:28:29.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.371+0000 7f2e86a00640 1 -- 172.21.15.6:0/18115062 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2e7803e7b0 con 0x7f2e900b1b40 2024-03-04T09:28:29.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.372+0000 7f2e86a00640 1 --2- 172.21.15.6:0/18115062 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2e6c06b4d0 0x7f2e6c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:29.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.372+0000 7f2e86a00640 1 -- 172.21.15.6:0/18115062 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2e780acc70 con 0x7f2e900b1b40 2024-03-04T09:28:29.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.372+0000 7f2e8de00640 1 --2- 172.21.15.6:0/18115062 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2e6c06b4d0 0x7f2e6c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:29.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.373+0000 7f2e8de00640 1 --2- 172.21.15.6:0/18115062 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2e6c06b4d0 0x7f2e6c06d990 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f2e80007fd0 tx=0x7f2e80009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:29.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.376+0000 7f2e86a00640 1 -- 172.21.15.6:0/18115062 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2e7807bca0 con 0x7f2e900b1b40 2024-03-04T09:28:29.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.570+0000 7f2e8f200640 1 -- 172.21.15.6:0/18115062 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f2e9015b7d0 con 0x7f2e900b1b40 2024-03-04T09:28:29.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.571+0000 7f2e86a00640 1 -- 172.21.15.6:0/18115062 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v56) v1 ==== 100+0+11 (secure 0 0 0) 0x7f2e7807b5a0 con 0x7f2e900b1b40 2024-03-04T09:28:29.568 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 64 2024-03-04T09:28:29.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.573+0000 7f2e8f200640 1 -- 172.21.15.6:0/18115062 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2e6c06b4d0 msgr2=0x7f2e6c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:29.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.573+0000 7f2e8f200640 1 --2- 172.21.15.6:0/18115062 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2e6c06b4d0 0x7f2e6c06d990 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f2e80007fd0 tx=0x7f2e80009040 comp rx=0 tx=0).stop 2024-03-04T09:28:29.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.573+0000 7f2e8f200640 1 -- 172.21.15.6:0/18115062 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2e900b1b40 msgr2=0x7f2e90158140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:29.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.573+0000 7f2e8f200640 1 --2- 172.21.15.6:0/18115062 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2e900b1b40 0x7f2e90158140 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f2e78037040 tx=0x7f2e78004a40 comp rx=0 tx=0).stop 2024-03-04T09:28:29.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.573+0000 7f2e8f200640 1 -- 172.21.15.6:0/18115062 shutdown_connections 2024-03-04T09:28:29.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.573+0000 7f2e8f200640 1 --2- 172.21.15.6:0/18115062 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2e6c06b4d0 0x7f2e6c06d990 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:29.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.573+0000 7f2e8f200640 1 --2- 172.21.15.6:0/18115062 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2e900b3c20 0x7f2e9014f340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:29.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.573+0000 7f2e8f200640 1 --2- 172.21.15.6:0/18115062 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2e900b1b40 0x7f2e90158140 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:29.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.573+0000 7f2e8f200640 1 --2- 172.21.15.6:0/18115062 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2e900729d0 0x7f2e90150b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:29.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.573+0000 7f2e8f200640 1 -- 172.21.15.6:0/18115062 >> 172.21.15.6:0/18115062 conn(0x7f2e9006b3a0 msgr2=0x7f2e901544e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:29.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.573+0000 7f2e8f200640 1 -- 172.21.15.6:0/18115062 shutdown_connections 2024-03-04T09:28:29.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.573+0000 7f2e8f200640 1 -- 172.21.15.6:0/18115062 wait complete. 2024-03-04T09:28:29.591 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-04T09:28:29.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.817+0000 7fd6d6200640 1 Processor -- start 2024-03-04T09:28:29.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.817+0000 7fd6d6200640 1 -- start start 2024-03-04T09:28:29.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.818+0000 7fd6d6200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd6d006e240 0x7fd6d00ad7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:29.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.819+0000 7fd6d6200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6d00ce9a0 con 0x7fd6d0161670 2024-03-04T09:28:29.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.819+0000 7fd6d6200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6d00cec80 con 0x7fd6d0071380 2024-03-04T09:28:29.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.819+0000 7fd6d6200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6d00730d0 con 0x7fd6d006e240 2024-03-04T09:28:29.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.819+0000 7fd6cf400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fd6d0161670 0x7fd6d0070c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:41128/0 (socket says 172.21.15.6:41128) 2024-03-04T09:28:29.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.819+0000 7fd6cf400640 1 -- 172.21.15.6:0/658100521 learned_addr learned my addr 172.21.15.6:0/658100521 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:28:29.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.819+0000 7fd6cfe00640 1 --2- 172.21.15.6:0/658100521 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd6d006e240 0x7fd6d00ad7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:29.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.819+0000 7fd6cea00640 1 -- 172.21.15.6:0/658100521 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 916347849 0 0) 0x7fd6d00ce9a0 con 0x7fd6d0161670 2024-03-04T09:28:29.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.819+0000 7fd6cea00640 1 -- 172.21.15.6:0/658100521 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6b0003520 con 0x7fd6d0161670 2024-03-04T09:28:29.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.820+0000 7fd6cea00640 1 -- 172.21.15.6:0/658100521 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3016551649 0 0) 0x7fd6b0003520 con 0x7fd6d0161670 2024-03-04T09:28:29.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.820+0000 7fd6cea00640 1 -- 172.21.15.6:0/658100521 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd6d006e240 msgr2=0x7fd6d00ad7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:29.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.820+0000 7fd6cea00640 1 --2- 172.21.15.6:0/658100521 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd6d006e240 0x7fd6d00ad7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:29.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.820+0000 7fd6cea00640 1 -- 172.21.15.6:0/658100521 >> v1:172.21.15.146:6789/0 conn(0x7fd6d0071380 legacy=0x7fd6d00ad090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:29.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.820+0000 7fd6cea00640 1 -- 172.21.15.6:0/658100521 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6d00cef60 con 0x7fd6d0161670 2024-03-04T09:28:29.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.820+0000 7fd6cea00640 1 -- 172.21.15.6:0/658100521 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7fd6c0001670 con 0x7fd6d0161670 2024-03-04T09:28:29.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.820+0000 7fd6cea00640 1 -- 172.21.15.6:0/658100521 >> v1:172.21.15.6:6789/0 conn(0x7fd6d0161670 legacy=0x7fd6d0070c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:29.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.820+0000 7fd6cea00640 1 --2- 172.21.15.6:0/658100521 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd6d006e240 0x7fd6b0003a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:29.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.820+0000 7fd6cea00640 1 --2- 172.21.15.6:0/658100521 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd6b0003fd0 0x7fd6b0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:29.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.820+0000 7fd6cea00640 1 --2- 172.21.15.6:0/658100521 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd6b0004d40 0x7fd6b00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:29.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.820+0000 7fd6cea00640 1 -- 172.21.15.6:0/658100521 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6d00730d0 con 0x7fd6b0004d40 2024-03-04T09:28:29.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.820+0000 7fd6cea00640 1 -- 172.21.15.6:0/658100521 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6b000bbb0 con 0x7fd6b0003fd0 2024-03-04T09:28:29.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.820+0000 7fd6cea00640 1 -- 172.21.15.6:0/658100521 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6b000bdb0 con 0x7fd6d006e240 2024-03-04T09:28:29.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.820+0000 7fd6cfe00640 1 --2- 172.21.15.6:0/658100521 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd6b0004d40 0x7fd6b00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:29.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.820+0000 7fd6c7e00640 1 --2- 172.21.15.6:0/658100521 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd6b0003fd0 0x7fd6b0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:29.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.820+0000 7fd6cf400640 1 --2- 172.21.15.6:0/658100521 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd6d006e240 0x7fd6b0003a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:29.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.821+0000 7fd6cfe00640 1 -- 172.21.15.6:0/658100521 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd6d006e240 msgr2=0x7fd6b0003a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:29.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.821+0000 7fd6cfe00640 1 --2- 172.21.15.6:0/658100521 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd6d006e240 0x7fd6b0003a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:29.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.821+0000 7fd6cfe00640 1 -- 172.21.15.6:0/658100521 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd6b0003fd0 msgr2=0x7fd6b0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:29.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.821+0000 7fd6cfe00640 1 --2- 172.21.15.6:0/658100521 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd6b0003fd0 0x7fd6b0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:29.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.821+0000 7fd6cfe00640 1 -- 172.21.15.6:0/658100521 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6b0003520 con 0x7fd6b0004d40 2024-03-04T09:28:29.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.821+0000 7fd6cf400640 1 --2- 172.21.15.6:0/658100521 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd6d006e240 0x7fd6b0003a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:28:29.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.821+0000 7fd6c7e00640 1 --2- 172.21.15.6:0/658100521 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd6b0003fd0 0x7fd6b0004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:28:29.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.822+0000 7fd6cfe00640 1 --2- 172.21.15.6:0/658100521 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd6b0004d40 0x7fd6b00051c0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fd6c8009f90 tx=0x7fd6b0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=261b8d8c1d6638db server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:29.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.822+0000 7fd6cea00640 1 -- 172.21.15.6:0/658100521 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6c8039070 con 0x7fd6b0004d40 2024-03-04T09:28:29.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.822+0000 7fd6cea00640 1 -- 172.21.15.6:0/658100521 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6c802cbf0 con 0x7fd6b0004d40 2024-03-04T09:28:29.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.822+0000 7fd6cea00640 1 -- 172.21.15.6:0/658100521 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6c8009c10 con 0x7fd6b0004d40 2024-03-04T09:28:29.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.822+0000 7fd6d6200640 1 -- 172.21.15.6:0/658100521 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd6b0004d40 msgr2=0x7fd6b00051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:29.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.822+0000 7fd6d6200640 1 --2- 172.21.15.6:0/658100521 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd6b0004d40 0x7fd6b00051c0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fd6c8009f90 tx=0x7fd6b0005830 comp rx=0 tx=0).stop 2024-03-04T09:28:29.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.822+0000 7fd6cfe00640 1 -- 172.21.15.6:0/658100521 reap_dead start 2024-03-04T09:28:29.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.822+0000 7fd6d6200640 1 -- 172.21.15.6:0/658100521 shutdown_connections 2024-03-04T09:28:29.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.822+0000 7fd6d6200640 1 -- 172.21.15.6:0/658100521 >> 172.21.15.6:0/658100521 conn(0x7fd6d006afb0 msgr2=0x7fd6d006b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:29.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.823+0000 7fd6d6200640 1 -- 172.21.15.6:0/658100521 shutdown_connections 2024-03-04T09:28:29.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.823+0000 7fd6d6200640 1 -- 172.21.15.6:0/658100521 wait complete. 2024-03-04T09:28:29.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.824+0000 7fd6d6200640 1 Processor -- start 2024-03-04T09:28:29.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.825+0000 7fd6d6200640 1 -- start start 2024-03-04T09:28:29.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.825+0000 7fd6d6200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd6b0004d40 0x7fd6d014e130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:29.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.825+0000 7fd6d6200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd6d00ca3b0 0x7fd6d00b0830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:29.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.825+0000 7fd6d6200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd6d00b0d70 0x7fd6d00c9150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:29.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.825+0000 7fd6d6200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6d00ce600 con 0x7fd6b0004d40 2024-03-04T09:28:29.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.825+0000 7fd6d6200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6d00cf520 con 0x7fd6d00ca3b0 2024-03-04T09:28:29.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.826+0000 7fd6d6200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6d00ce780 con 0x7fd6d00b0d70 2024-03-04T09:28:29.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.826+0000 7fd6cf400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd6b0004d40 0x7fd6d014e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:29.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.826+0000 7fd6cf400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd6b0004d40 0x7fd6d014e130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53632/0 (socket says 172.21.15.6:53632) 2024-03-04T09:28:29.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.826+0000 7fd6cf400640 1 -- 172.21.15.6:0/3593367852 learned_addr learned my addr 172.21.15.6:0/3593367852 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:28:29.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.826+0000 7fd6cfe00640 1 --2- 172.21.15.6:0/3593367852 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd6d00b0d70 0x7fd6d00c9150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:29.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.826+0000 7fd6c7e00640 1 --2- 172.21.15.6:0/3593367852 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd6d00ca3b0 0x7fd6d00b0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:29.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.826+0000 7fd6cf400640 1 -- 172.21.15.6:0/3593367852 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd6d00b0d70 msgr2=0x7fd6d00c9150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:29.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.826+0000 7fd6cf400640 1 --2- 172.21.15.6:0/3593367852 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd6d00b0d70 0x7fd6d00c9150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:29.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.826+0000 7fd6cf400640 1 -- 172.21.15.6:0/3593367852 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd6d00ca3b0 msgr2=0x7fd6d00b0830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:29.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.826+0000 7fd6cf400640 1 --2- 172.21.15.6:0/3593367852 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd6d00ca3b0 0x7fd6d00b0830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:29.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.826+0000 7fd6cf400640 1 -- 172.21.15.6:0/3593367852 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6d01f4180 con 0x7fd6b0004d40 2024-03-04T09:28:29.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.826+0000 7fd6cfe00640 1 --2- 172.21.15.6:0/3593367852 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd6d00b0d70 0x7fd6d00c9150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:28:29.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.826+0000 7fd6cf400640 1 --2- 172.21.15.6:0/3593367852 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd6b0004d40 0x7fd6d014e130 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fd6c000d890 tx=0x7fd6c000dd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:29.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.827+0000 7fd6c7400640 1 -- 172.21.15.6:0/3593367852 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6c0019070 con 0x7fd6b0004d40 2024-03-04T09:28:29.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.827+0000 7fd6c7400640 1 -- 172.21.15.6:0/3593367852 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6c00074e0 con 0x7fd6b0004d40 2024-03-04T09:28:29.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.827+0000 7fd6c7400640 1 -- 172.21.15.6:0/3593367852 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6c0004a20 con 0x7fd6b0004d40 2024-03-04T09:28:29.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.827+0000 7fd6d6200640 1 -- 172.21.15.6:0/3593367852 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6d01f4460 con 0x7fd6b0004d40 2024-03-04T09:28:29.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.827+0000 7fd6d6200640 1 -- 172.21.15.6:0/3593367852 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6d00ca830 con 0x7fd6b0004d40 2024-03-04T09:28:29.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.828+0000 7fd6d6200640 1 -- 172.21.15.6:0/3593367852 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6d0069510 con 0x7fd6b0004d40 2024-03-04T09:28:29.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.829+0000 7fd6c7400640 1 -- 172.21.15.6:0/3593367852 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd6c0004bc0 con 0x7fd6b0004d40 2024-03-04T09:28:29.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.830+0000 7fd6c7400640 1 --2- 172.21.15.6:0/3593367852 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd6ac06b4d0 0x7fd6ac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:29.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.830+0000 7fd6c7400640 1 -- 172.21.15.6:0/3593367852 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd6c0089da0 con 0x7fd6b0004d40 2024-03-04T09:28:29.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.831+0000 7fd6c7e00640 1 --2- 172.21.15.6:0/3593367852 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd6ac06b4d0 0x7fd6ac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:29.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.832+0000 7fd6c7e00640 1 --2- 172.21.15.6:0/3593367852 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd6ac06b4d0 0x7fd6ac06d990 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fd6b8007fd0 tx=0x7fd6b8009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:29.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:29.835+0000 7fd6c7400640 1 -- 172.21.15.6:0/3593367852 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd6c0058c40 con 0x7fd6b0004d40 2024-03-04T09:28:30.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:30.049+0000 7fd6d6200640 1 -- 172.21.15.6:0/3593367852 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fd6d004f3a0 con 0x7fd6b0004d40 2024-03-04T09:28:30.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:30.050+0000 7fd6c7400640 1 -- 172.21.15.6:0/3593367852 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v56) v1 ==== 80+0+0 (secure 0 0 0) 0x7fd6c0058540 con 0x7fd6b0004d40 2024-03-04T09:28:30.047 INFO:teuthology.orchestra.run.smithi006.stderr:listed 0 entries 2024-03-04T09:28:30.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:30.053+0000 7fd6d6200640 1 -- 172.21.15.6:0/3593367852 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd6ac06b4d0 msgr2=0x7fd6ac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:30.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:30.053+0000 7fd6d6200640 1 --2- 172.21.15.6:0/3593367852 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd6ac06b4d0 0x7fd6ac06d990 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fd6b8007fd0 tx=0x7fd6b8009040 comp rx=0 tx=0).stop 2024-03-04T09:28:30.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:30.053+0000 7fd6d6200640 1 -- 172.21.15.6:0/3593367852 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd6b0004d40 msgr2=0x7fd6d014e130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:30.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:30.053+0000 7fd6d6200640 1 --2- 172.21.15.6:0/3593367852 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd6b0004d40 0x7fd6d014e130 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fd6c000d890 tx=0x7fd6c000dd50 comp rx=0 tx=0).stop 2024-03-04T09:28:30.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:30.053+0000 7fd6d6200640 1 -- 172.21.15.6:0/3593367852 shutdown_connections 2024-03-04T09:28:30.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:30.053+0000 7fd6d6200640 1 --2- 172.21.15.6:0/3593367852 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd6ac06b4d0 0x7fd6ac06d990 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:30.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:30.053+0000 7fd6d6200640 1 --2- 172.21.15.6:0/3593367852 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd6d00b0d70 0x7fd6d00c9150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:30.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:30.053+0000 7fd6d6200640 1 --2- 172.21.15.6:0/3593367852 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd6d00ca3b0 0x7fd6d00b0830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:30.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:30.053+0000 7fd6d6200640 1 --2- 172.21.15.6:0/3593367852 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd6b0004d40 0x7fd6d014e130 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:30.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:30.053+0000 7fd6d6200640 1 -- 172.21.15.6:0/3593367852 >> 172.21.15.6:0/3593367852 conn(0x7fd6d006afb0 msgr2=0x7fd6d0070c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:30.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:30.053+0000 7fd6d6200640 1 -- 172.21.15.6:0/3593367852 shutdown_connections 2024-03-04T09:28:30.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:30.053+0000 7fd6d6200640 1 -- 172.21.15.6:0/3593367852 wait complete. 2024-03-04T09:28:30.071 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-04T09:28:30.071 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:28:30.071 DEBUG:teuthology.orchestra.run.smithi006:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-04T09:28:30.310 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi006.front.sepia.ceph.com 2024-03-04T09:28:30.310 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:28:30.310 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-04T09:28:30.349 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-04T09:28:30.350 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T09:28:30.386 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053429,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249180k,nr_inodes=812295,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-04T09:28:30.387 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-04T09:28:30.387 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T09:28:30.435 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054202,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249800k,nr_inodes=812450,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-04T09:28:30.435 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-04T09:28:30.435 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T09:28:30.478 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053429,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249180k,nr_inodes=812295,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-04T09:28:30.478 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:28:30.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:30.703+0000 7f5140e00640 1 Processor -- start 2024-03-04T09:28:30.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:30.703+0000 7f5140e00640 1 -- start start 2024-03-04T09:28:30.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:30.705+0000 7f5140e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f513c15efc0 0x7f513c0cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:30.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:30.705+0000 7f5140e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f513c0cce90 0x7f513c161ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:30.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:30.705+0000 7f5140e00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f513c1627f0 con 0x7f513c162000 2024-03-04T09:28:30.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:30.705+0000 7f5140e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f513c15f3a0 con 0x7f513c15efc0 2024-03-04T09:28:30.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:30.705+0000 7f5140e00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f513c06d190 con 0x7f513c0cce90 2024-03-04T09:28:30.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:30.705+0000 7f513b400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f513c15efc0 0x7f513c0cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:30.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:30.706+0000 7f513b400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f513c15efc0 0x7f513c0cc950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53638/0 (socket says 172.21.15.6:53638) 2024-03-04T09:28:30.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:30.706+0000 7f513b400640 1 -- 172.21.15.6:0/181819554 learned_addr learned my addr 172.21.15.6:0/181819554 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:28:30.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:30.706+0000 7f513aa00640 1 --2- 172.21.15.6:0/181819554 >> v2:172.21.15.146:3300/0 conn(0x7f513c0cce90 0x7f513c161ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:30.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:30.706+0000 7f513b400640 1 -- 172.21.15.6:0/181819554 >> v2:172.21.15.146:3300/0 conn(0x7f513c0cce90 msgr2=0x7f513c161ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:30.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:30.706+0000 7f513b400640 1 --2- 172.21.15.6:0/181819554 >> v2:172.21.15.146:3300/0 conn(0x7f513c0cce90 0x7f513c161ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:30.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:30.706+0000 7f513b400640 1 -- 172.21.15.6:0/181819554 >> v1:172.21.15.146:6789/0 conn(0x7f513c162000 legacy=0x7f513c0ad880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:30.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:30.706+0000 7f513b400640 1 -- 172.21.15.6:0/181819554 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f513c06c860 con 0x7f513c15efc0 2024-03-04T09:28:30.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:30.707+0000 7f513b400640 1 --2- 172.21.15.6:0/181819554 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f513c15efc0 0x7f513c0cc950 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f513c0b51d0 tx=0x7f512402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cbcf2ec5eef99c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:30.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:30.707+0000 7f513a000640 1 -- 172.21.15.6:0/181819554 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5124034070 con 0x7f513c15efc0 2024-03-04T09:28:30.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:30.707+0000 7f513a000640 1 -- 172.21.15.6:0/181819554 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5124030860 con 0x7f513c15efc0 2024-03-04T09:28:30.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:30.707+0000 7f513a000640 1 -- 172.21.15.6:0/181819554 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5124030c30 con 0x7f513c15efc0 2024-03-04T09:28:30.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:30.708+0000 7f5140e00640 1 -- 172.21.15.6:0/181819554 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f513c15efc0 msgr2=0x7f513c0cc950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:30.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:30.708+0000 7f5140e00640 1 --2- 172.21.15.6:0/181819554 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f513c15efc0 0x7f513c0cc950 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f513c0b51d0 tx=0x7f512402b9a0 comp rx=0 tx=0).stop 2024-03-04T09:28:30.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:30.708+0000 7f5140e00640 1 -- 172.21.15.6:0/181819554 shutdown_connections 2024-03-04T09:28:30.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:30.708+0000 7f5140e00640 1 --2- 172.21.15.6:0/181819554 >> v2:172.21.15.146:3300/0 conn(0x7f513c0cce90 0x7f513c161ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:30.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:30.708+0000 7f5140e00640 1 --2- 172.21.15.6:0/181819554 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f513c15efc0 0x7f513c0cc950 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:30.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:30.708+0000 7f5140e00640 1 -- 172.21.15.6:0/181819554 >> 172.21.15.6:0/181819554 conn(0x7f513c06b3d0 msgr2=0x7f513c06b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:30.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:30.708+0000 7f5140e00640 1 -- 172.21.15.6:0/181819554 shutdown_connections 2024-03-04T09:28:30.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:30.708+0000 7f5140e00640 1 -- 172.21.15.6:0/181819554 wait complete. 2024-03-04T09:28:30.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:30.710+0000 7f5140e00640 1 Processor -- start 2024-03-04T09:28:30.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:30.710+0000 7f5140e00640 1 -- start start 2024-03-04T09:28:30.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:30.711+0000 7f5140e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f513c0cce90 0x7f513c143060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:30.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:30.711+0000 7f5140e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f513c15efc0 0x7f513c145160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:30.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:30.711+0000 7f5140e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f513c162000 0x7f513c140f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:30.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:30.711+0000 7f5140e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f513c071c30 con 0x7f513c0cce90 2024-03-04T09:28:30.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:30.711+0000 7f5140e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f513c06ce20 con 0x7f513c162000 2024-03-04T09:28:30.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:30.711+0000 7f5140e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f513c162470 con 0x7f513c15efc0 2024-03-04T09:28:30.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:30.711+0000 7f513b400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f513c0cce90 0x7f513c143060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:30.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:30.711+0000 7f513b400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f513c0cce90 0x7f513c143060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53652/0 (socket says 172.21.15.6:53652) 2024-03-04T09:28:30.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:30.711+0000 7f513b400640 1 -- 172.21.15.6:0/634558444 learned_addr learned my addr 172.21.15.6:0/634558444 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:28:30.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:30.712+0000 7f513aa00640 1 --2- 172.21.15.6:0/634558444 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f513c15efc0 0x7f513c145160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:30.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:30.712+0000 7f513be00640 1 --2- 172.21.15.6:0/634558444 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f513c162000 0x7f513c140f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:30.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:30.712+0000 7f513b400640 1 -- 172.21.15.6:0/634558444 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f513c15efc0 msgr2=0x7f513c145160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:30.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:30.712+0000 7f513b400640 1 --2- 172.21.15.6:0/634558444 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f513c15efc0 0x7f513c145160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:30.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:30.712+0000 7f513b400640 1 -- 172.21.15.6:0/634558444 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f513c162000 msgr2=0x7f513c140f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:30.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:30.712+0000 7f513b400640 1 --2- 172.21.15.6:0/634558444 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f513c162000 0x7f513c140f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:30.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:30.712+0000 7f513b400640 1 -- 172.21.15.6:0/634558444 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f513c1458b0 con 0x7f513c0cce90 2024-03-04T09:28:30.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:30.712+0000 7f513be00640 1 --2- 172.21.15.6:0/634558444 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f513c162000 0x7f513c140f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:28:30.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:30.712+0000 7f513b400640 1 --2- 172.21.15.6:0/634558444 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f513c0cce90 0x7f513c143060 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f512402b9e0 tx=0x7f5124004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:30.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:30.713+0000 7f5133400640 1 -- 172.21.15.6:0/634558444 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5124048070 con 0x7f513c0cce90 2024-03-04T09:28:30.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:30.713+0000 7f5133400640 1 -- 172.21.15.6:0/634558444 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5124030660 con 0x7f513c0cce90 2024-03-04T09:28:30.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:30.713+0000 7f5133400640 1 -- 172.21.15.6:0/634558444 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5124034040 con 0x7f513c0cce90 2024-03-04T09:28:30.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:30.713+0000 7f5140e00640 1 -- 172.21.15.6:0/634558444 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f513c145b90 con 0x7f513c0cce90 2024-03-04T09:28:30.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:30.713+0000 7f5140e00640 1 -- 172.21.15.6:0/634558444 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f513c146150 con 0x7f513c0cce90 2024-03-04T09:28:30.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:30.714+0000 7f5140e00640 1 -- 172.21.15.6:0/634558444 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f513c04f330 con 0x7f513c0cce90 2024-03-04T09:28:30.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:30.715+0000 7f5133400640 1 -- 172.21.15.6:0/634558444 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5124037070 con 0x7f513c0cce90 2024-03-04T09:28:30.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:30.716+0000 7f5133400640 1 --2- 172.21.15.6:0/634558444 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f511806b4d0 0x7f511806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:30.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:30.716+0000 7f5133400640 1 -- 172.21.15.6:0/634558444 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f51240ac100 con 0x7f513c0cce90 2024-03-04T09:28:30.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:30.716+0000 7f513aa00640 1 --2- 172.21.15.6:0/634558444 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f511806b4d0 0x7f511806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:30.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:30.717+0000 7f513aa00640 1 --2- 172.21.15.6:0/634558444 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f511806b4d0 0x7f511806d990 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f513c0cd4a0 tx=0x7f512c008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:30.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:30.720+0000 7f5133400640 1 -- 172.21.15.6:0/634558444 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f512407b090 con 0x7f513c0cce90 2024-03-04T09:28:30.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:30.960+0000 7f5140e00640 1 -- 172.21.15.6:0/634558444 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f513c0af2f0 con 0x7f513c0cce90 2024-03-04T09:28:30.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:30.962+0000 7f5133400640 1 -- 172.21.15.6:0/634558444 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+4971 (secure 0 0 0) 0x7f512407a990 con 0x7f513c0cce90 2024-03-04T09:28:30.959 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:28:30.959 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":34,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5030,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/1411942502","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1411942502},{"type":"v1","addr":"172.21.15.6:6834","nonce":1411942502}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28},{"gid":5039,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/1628166413","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":1628166413},{"type":"v1","addr":"172.21.15.146:6837","nonce":1628166413}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":5323,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/131491768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":131491768},{"type":"v1","addr":"172.21.15.6:6835","nonce":131491768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:27:59.854470+0000","modified":"2024-03-04T09:28:02.945053+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5326},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5326":{"gid":5326,"name":"b","rank":0,"incarnation":30,"state":"up:active","state_seq":3,"addr":"172.21.15.146:6836/527855083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":527855083},{"type":"v1","addr":"172.21.15.146:6836","nonce":527855083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5326,"qdb_cluster":[5326]},"id":4}]} 2024-03-04T09:28:30.959 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 34 2024-03-04T09:28:30.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:30.964+0000 7f5140e00640 1 -- 172.21.15.6:0/634558444 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f511806b4d0 msgr2=0x7f511806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:30.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:30.964+0000 7f5140e00640 1 --2- 172.21.15.6:0/634558444 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f511806b4d0 0x7f511806d990 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f513c0cd4a0 tx=0x7f512c008040 comp rx=0 tx=0).stop 2024-03-04T09:28:30.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:30.964+0000 7f5140e00640 1 -- 172.21.15.6:0/634558444 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f513c0cce90 msgr2=0x7f513c143060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:30.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:30.964+0000 7f5140e00640 1 --2- 172.21.15.6:0/634558444 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f513c0cce90 0x7f513c143060 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f512402b9e0 tx=0x7f5124004730 comp rx=0 tx=0).stop 2024-03-04T09:28:30.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:30.965+0000 7f5140e00640 1 -- 172.21.15.6:0/634558444 shutdown_connections 2024-03-04T09:28:30.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:30.965+0000 7f5140e00640 1 --2- 172.21.15.6:0/634558444 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f513c162000 0x7f513c140f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:30.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:30.965+0000 7f5140e00640 1 --2- 172.21.15.6:0/634558444 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f511806b4d0 0x7f511806d990 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:30.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:30.965+0000 7f5140e00640 1 --2- 172.21.15.6:0/634558444 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f513c15efc0 0x7f513c145160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:30.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:30.965+0000 7f5140e00640 1 --2- 172.21.15.6:0/634558444 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f513c0cce90 0x7f513c143060 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:30.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:30.965+0000 7f5140e00640 1 -- 172.21.15.6:0/634558444 >> 172.21.15.6:0/634558444 conn(0x7f513c06b3d0 msgr2=0x7f513c0ad880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:30.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:30.965+0000 7f5140e00640 1 -- 172.21.15.6:0/634558444 shutdown_connections 2024-03-04T09:28:30.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:30.965+0000 7f5140e00640 1 -- 172.21.15.6:0/634558444 wait complete. 2024-03-04T09:28:30.982 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T09:28:31.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.211+0000 7fef35c00640 1 Processor -- start 2024-03-04T09:28:31.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.212+0000 7fef35c00640 1 -- start start 2024-03-04T09:28:31.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.213+0000 7fef35c00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fef3015e0b0 0x7fef300cf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:31.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.213+0000 7fef35c00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef300b3710 con 0x7fef300d0e50 2024-03-04T09:28:31.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.213+0000 7fef35c00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef300b39f0 con 0x7fef300d4a10 2024-03-04T09:28:31.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.213+0000 7fef35c00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef300b09e0 con 0x7fef3015e0b0 2024-03-04T09:28:31.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.213+0000 7fef35200640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fef300d0e50 0x7fef300b44d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:41142/0 (socket says 172.21.15.6:41142) 2024-03-04T09:28:31.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.213+0000 7fef35200640 1 -- 172.21.15.6:0/2529201409 learned_addr learned my addr 172.21.15.6:0/2529201409 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:28:31.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.214+0000 7fef2ea00640 1 -- 172.21.15.6:0/2529201409 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2280656147 0 0) 0x7fef300b3710 con 0x7fef300d0e50 2024-03-04T09:28:31.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.214+0000 7fef2fe00640 1 --2- 172.21.15.6:0/2529201409 >> v2:172.21.15.146:3300/0 conn(0x7fef3015e0b0 0x7fef300cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:31.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.214+0000 7fef2ea00640 1 -- 172.21.15.6:0/2529201409 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef10003520 con 0x7fef300d0e50 2024-03-04T09:28:31.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.214+0000 7fef2ea00640 1 -- 172.21.15.6:0/2529201409 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1932281370 0 0) 0x7fef10003520 con 0x7fef300d0e50 2024-03-04T09:28:31.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.214+0000 7fef2ea00640 1 -- 172.21.15.6:0/2529201409 >> v2:172.21.15.146:3300/0 conn(0x7fef3015e0b0 msgr2=0x7fef300cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:31.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.214+0000 7fef2ea00640 1 --2- 172.21.15.6:0/2529201409 >> v2:172.21.15.146:3300/0 conn(0x7fef3015e0b0 0x7fef300cf310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:31.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.214+0000 7fef2ea00640 1 -- 172.21.15.6:0/2529201409 >> v1:172.21.15.146:6789/0 conn(0x7fef300d4a10 legacy=0x7fef300d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:31.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.215+0000 7fef2ea00640 1 -- 172.21.15.6:0/2529201409 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef300b3cd0 con 0x7fef300d0e50 2024-03-04T09:28:31.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.215+0000 7fef2ea00640 1 -- 172.21.15.6:0/2529201409 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7fef28001670 con 0x7fef300d0e50 2024-03-04T09:28:31.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.215+0000 7fef2ea00640 1 -- 172.21.15.6:0/2529201409 >> v1:172.21.15.6:6789/0 conn(0x7fef300d0e50 legacy=0x7fef300b44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:31.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.215+0000 7fef2ea00640 1 --2- 172.21.15.6:0/2529201409 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fef10003a00 0x7fef10003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:31.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.215+0000 7fef2ea00640 1 --2- 172.21.15.6:0/2529201409 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fef10004800 0x7fef10004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:31.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.215+0000 7fef2ea00640 1 --2- 172.21.15.6:0/2529201409 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fef100051c0 0x7fef1000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:31.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.215+0000 7fef2ea00640 1 -- 172.21.15.6:0/2529201409 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef1000c090 con 0x7fef100051c0 2024-03-04T09:28:31.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.215+0000 7fef2ea00640 1 -- 172.21.15.6:0/2529201409 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef1000c240 con 0x7fef10003a00 2024-03-04T09:28:31.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.215+0000 7fef2ea00640 1 -- 172.21.15.6:0/2529201409 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef1000c440 con 0x7fef10004800 2024-03-04T09:28:31.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.215+0000 7fef35200640 1 --2- 172.21.15.6:0/2529201409 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fef100051c0 0x7fef1000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:31.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.215+0000 7fef2fe00640 1 --2- 172.21.15.6:0/2529201409 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fef10003a00 0x7fef10003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:31.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.215+0000 7fef2f400640 1 --2- 172.21.15.6:0/2529201409 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fef10004800 0x7fef10004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:31.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.215+0000 7fef35200640 1 -- 172.21.15.6:0/2529201409 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fef10004800 msgr2=0x7fef10004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:31.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.215+0000 7fef35200640 1 --2- 172.21.15.6:0/2529201409 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fef10004800 0x7fef10004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:31.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.215+0000 7fef35200640 1 -- 172.21.15.6:0/2529201409 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fef10003a00 msgr2=0x7fef10003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:31.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.215+0000 7fef35200640 1 --2- 172.21.15.6:0/2529201409 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fef10003a00 0x7fef10003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:31.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.216+0000 7fef35200640 1 -- 172.21.15.6:0/2529201409 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef10003520 con 0x7fef100051c0 2024-03-04T09:28:31.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.216+0000 7fef2fe00640 1 --2- 172.21.15.6:0/2529201409 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fef10003a00 0x7fef10003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:28:31.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.216+0000 7fef2f400640 1 --2- 172.21.15.6:0/2529201409 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fef10004800 0x7fef10004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:28:31.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.216+0000 7fef35200640 1 -- 172.21.15.6:0/2529201409 reap_dead start 2024-03-04T09:28:31.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.217+0000 7fef35200640 1 --2- 172.21.15.6:0/2529201409 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fef100051c0 0x7fef1000b9c0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fef2800c0e0 tx=0x7fef10005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbe5bdd8e091b9a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:31.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.217+0000 7fef2ea00640 1 -- 172.21.15.6:0/2529201409 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef28007c40 con 0x7fef100051c0 2024-03-04T09:28:31.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.217+0000 7fef2ea00640 1 -- 172.21.15.6:0/2529201409 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef28030070 con 0x7fef100051c0 2024-03-04T09:28:31.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.217+0000 7fef2ea00640 1 -- 172.21.15.6:0/2529201409 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef28005d10 con 0x7fef100051c0 2024-03-04T09:28:31.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.218+0000 7fef35c00640 1 -- 172.21.15.6:0/2529201409 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fef100051c0 msgr2=0x7fef1000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:31.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.218+0000 7fef35c00640 1 --2- 172.21.15.6:0/2529201409 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fef100051c0 0x7fef1000b9c0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fef2800c0e0 tx=0x7fef10005830 comp rx=0 tx=0).stop 2024-03-04T09:28:31.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.218+0000 7fef35c00640 1 -- 172.21.15.6:0/2529201409 shutdown_connections 2024-03-04T09:28:31.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.218+0000 7fef35c00640 1 --2- 172.21.15.6:0/2529201409 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fef100051c0 0x7fef1000b9c0 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:31.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.218+0000 7fef35c00640 1 -- 172.21.15.6:0/2529201409 >> 172.21.15.6:0/2529201409 conn(0x7fef300d3d30 msgr2=0x7fef300d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:31.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.218+0000 7fef35c00640 1 -- 172.21.15.6:0/2529201409 shutdown_connections 2024-03-04T09:28:31.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.218+0000 7fef35c00640 1 -- 172.21.15.6:0/2529201409 wait complete. 2024-03-04T09:28:31.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.220+0000 7fef35c00640 1 Processor -- start 2024-03-04T09:28:31.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.220+0000 7fef35c00640 1 -- start start 2024-03-04T09:28:31.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.221+0000 7fef35c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fef100051c0 0x7fef3015e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:31.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.221+0000 7fef35c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fef300b8b70 0x7fef301f1b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:31.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.221+0000 7fef35c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fef300b8ff0 0x7fef301f20d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:31.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.221+0000 7fef35c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef300cfc60 con 0x7fef100051c0 2024-03-04T09:28:31.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.221+0000 7fef35c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef3015e490 con 0x7fef300b8ff0 2024-03-04T09:28:31.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.221+0000 7fef35c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef300d0740 con 0x7fef300b8b70 2024-03-04T09:28:31.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.221+0000 7fef2fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fef100051c0 0x7fef3015e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:31.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.221+0000 7fef2fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fef100051c0 0x7fef3015e6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53670/0 (socket says 172.21.15.6:53670) 2024-03-04T09:28:31.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.221+0000 7fef2fe00640 1 -- 172.21.15.6:0/4220025488 learned_addr learned my addr 172.21.15.6:0/4220025488 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:28:31.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.221+0000 7fef2f400640 1 --2- 172.21.15.6:0/4220025488 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fef300b8b70 0x7fef301f1b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:31.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.222+0000 7fef35200640 1 --2- 172.21.15.6:0/4220025488 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fef300b8ff0 0x7fef301f20d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:31.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.222+0000 7fef2fe00640 1 -- 172.21.15.6:0/4220025488 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fef300b8b70 msgr2=0x7fef301f1b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:31.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.222+0000 7fef2fe00640 1 --2- 172.21.15.6:0/4220025488 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fef300b8b70 0x7fef301f1b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:31.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.222+0000 7fef2fe00640 1 -- 172.21.15.6:0/4220025488 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fef300b8ff0 msgr2=0x7fef301f20d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:31.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.222+0000 7fef2fe00640 1 --2- 172.21.15.6:0/4220025488 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fef300b8ff0 0x7fef301f20d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:31.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.222+0000 7fef2fe00640 1 -- 172.21.15.6:0/4220025488 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef30160460 con 0x7fef100051c0 2024-03-04T09:28:31.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.222+0000 7fef2fe00640 1 --2- 172.21.15.6:0/4220025488 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fef100051c0 0x7fef3015e6c0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fef20010900 tx=0x7fef20010dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:31.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.222+0000 7fef27e00640 1 -- 172.21.15.6:0/4220025488 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef2000f040 con 0x7fef100051c0 2024-03-04T09:28:31.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.222+0000 7fef27e00640 1 -- 172.21.15.6:0/4220025488 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef2000ebc0 con 0x7fef100051c0 2024-03-04T09:28:31.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.222+0000 7fef27e00640 1 -- 172.21.15.6:0/4220025488 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef2001c400 con 0x7fef100051c0 2024-03-04T09:28:31.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.222+0000 7fef35c00640 1 -- 172.21.15.6:0/4220025488 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef30160740 con 0x7fef100051c0 2024-03-04T09:28:31.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.222+0000 7fef35c00640 1 -- 172.21.15.6:0/4220025488 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef30160d00 con 0x7fef100051c0 2024-03-04T09:28:31.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.224+0000 7fef35c00640 1 -- 172.21.15.6:0/4220025488 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef3004f330 con 0x7fef100051c0 2024-03-04T09:28:31.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.225+0000 7fef27e00640 1 -- 172.21.15.6:0/4220025488 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fef2001c5a0 con 0x7fef100051c0 2024-03-04T09:28:31.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.226+0000 7fef27e00640 1 --2- 172.21.15.6:0/4220025488 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fef0c06b4d0 0x7fef0c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:31.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.226+0000 7fef27e00640 1 -- 172.21.15.6:0/4220025488 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fef2008a9b0 con 0x7fef100051c0 2024-03-04T09:28:31.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.227+0000 7fef2f400640 1 --2- 172.21.15.6:0/4220025488 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fef0c06b4d0 0x7fef0c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:31.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.228+0000 7fef2f400640 1 --2- 172.21.15.6:0/4220025488 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fef0c06b4d0 0x7fef0c06d990 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fef18007fd0 tx=0x7fef18009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:31.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.231+0000 7fef27e00640 1 -- 172.21.15.6:0/4220025488 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fef20018030 con 0x7fef100051c0 2024-03-04T09:28:31.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.434+0000 7fef35c00640 1 -- 172.21.15.6:0/4220025488 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fef300c73b0 con 0x7fef100051c0 2024-03-04T09:28:31.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.436+0000 7fef27e00640 1 -- 172.21.15.6:0/4220025488 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v56) v1 ==== 74+0+16617 (secure 0 0 0) 0x7fef20059140 con 0x7fef100051c0 2024-03-04T09:28:31.433 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:28:31.433 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":56,"fsid":"edce0578-5ae4-4402-b45a-552bdef84d7c","created":"2024-03-04T09:24:54.523780+0000","modified":"2024-03-04T09:28:01.413693+0000","last_up_change":"2024-03-04T09:25:07.089049+0000","last_in_change":"2024-03-04T09:24:57.982077+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T09:25:07.935198+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-04T09:27:57.774529+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-04T09:27:58.886504+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a96574f-81f1-4bc6-8463-757520b84a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6825","nonce":1364449401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6827","nonce":1364449401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6831","nonce":1364449401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6829","nonce":1364449401}]},"public_addr":"172.21.15.6:6825/1364449401","cluster_addr":"172.21.15.6:6827/1364449401","heartbeat_back_addr":"172.21.15.6:6831/1364449401","heartbeat_front_addr":"172.21.15.6:6829/1364449401","state":["exists","up"]},{"osd":1,"uuid":"7d79b9c1-6b87-476a-bc36-7db5ee64cd90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6809","nonce":2494103078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6811","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6813","nonce":2494103078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6821","nonce":2494103078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6817","nonce":2494103078}]},"public_addr":"172.21.15.6:6809/2494103078","cluster_addr":"172.21.15.6:6813/2494103078","heartbeat_back_addr":"172.21.15.6:6821/2494103078","heartbeat_front_addr":"172.21.15.6:6817/2494103078","state":["exists","up"]},{"osd":2,"uuid":"492331b9-4ee9-40b6-bacd-85d44cd6c45a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6812","nonce":1500066681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6816","nonce":1500066681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6824","nonce":1500066681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6820","nonce":1500066681}]},"public_addr":"172.21.15.6:6812/1500066681","cluster_addr":"172.21.15.6:6816/1500066681","heartbeat_back_addr":"172.21.15.6:6824/1500066681","heartbeat_front_addr":"172.21.15.6:6820/1500066681","state":["exists","up"]},{"osd":3,"uuid":"21bb2efe-9ba7-4bdc-bc31-180cd1102e21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6801","nonce":1558806123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6803","nonce":1558806123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6807","nonce":1558806123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6805","nonce":1558806123}]},"public_addr":"172.21.15.6:6801/1558806123","cluster_addr":"172.21.15.6:6803/1558806123","heartbeat_back_addr":"172.21.15.6:6807/1558806123","heartbeat_front_addr":"172.21.15.6:6805/1558806123","state":["exists","up"]},{"osd":4,"uuid":"6e2c4b71-19d9-4737-83d9-558705d8644a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6801","nonce":1526135780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6803","nonce":1526135780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6807","nonce":1526135780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6805","nonce":1526135780}]},"public_addr":"172.21.15.146:6801/1526135780","cluster_addr":"172.21.15.146:6803/1526135780","heartbeat_back_addr":"172.21.15.146:6807/1526135780","heartbeat_front_addr":"172.21.15.146:6805/1526135780","state":["exists","up"]},{"osd":5,"uuid":"fb69c525-f1be-4b7d-ab32-6c29b0d6fddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6809","nonce":460262507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6811","nonce":460262507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6815","nonce":460262507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6813","nonce":460262507}]},"public_addr":"172.21.15.146:6809/460262507","cluster_addr":"172.21.15.146:6811/460262507","heartbeat_back_addr":"172.21.15.146:6815/460262507","heartbeat_front_addr":"172.21.15.146:6813/460262507","state":["exists","up"]},{"osd":6,"uuid":"c32e7fed-370d-4401-8ce0-6a4335b0d168","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6823","nonce":1392897493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6827","nonce":1392897493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6831","nonce":1392897493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6829","nonce":1392897493}]},"public_addr":"172.21.15.146:6823/1392897493","cluster_addr":"172.21.15.146:6827/1392897493","heartbeat_back_addr":"172.21.15.146:6831/1392897493","heartbeat_front_addr":"172.21.15.146:6829/1392897493","state":["exists","up"]},{"osd":7,"uuid":"75a79b44-fa5f-426d-a7d3-71f21f78835c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6817","nonce":3268323156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6819","nonce":3268323156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6825","nonce":3268323156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6822","nonce":3268323156}]},"public_addr":"172.21.15.146:6817/3268323156","cluster_addr":"172.21.15.146:6819/3268323156","heartbeat_back_addr":"172.21.15.146:6825/3268323156","heartbeat_front_addr":"172.21.15.146:6822/3268323156","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.310437+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.295850+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.294674+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.526855+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.300023+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.595160+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.346046+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.558741+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T09:28:31.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.438+0000 7fef35c00640 1 -- 172.21.15.6:0/4220025488 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fef0c06b4d0 msgr2=0x7fef0c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:31.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.438+0000 7fef35c00640 1 --2- 172.21.15.6:0/4220025488 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fef0c06b4d0 0x7fef0c06d990 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fef18007fd0 tx=0x7fef18009040 comp rx=0 tx=0).stop 2024-03-04T09:28:31.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.438+0000 7fef35c00640 1 -- 172.21.15.6:0/4220025488 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fef100051c0 msgr2=0x7fef3015e6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:31.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.438+0000 7fef35c00640 1 --2- 172.21.15.6:0/4220025488 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fef100051c0 0x7fef3015e6c0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fef20010900 tx=0x7fef20010dc0 comp rx=0 tx=0).stop 2024-03-04T09:28:31.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.438+0000 7fef35c00640 1 -- 172.21.15.6:0/4220025488 shutdown_connections 2024-03-04T09:28:31.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.438+0000 7fef35c00640 1 --2- 172.21.15.6:0/4220025488 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fef300b8ff0 0x7fef301f20d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:31.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.438+0000 7fef35c00640 1 --2- 172.21.15.6:0/4220025488 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fef0c06b4d0 0x7fef0c06d990 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:31.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.438+0000 7fef35c00640 1 --2- 172.21.15.6:0/4220025488 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fef300b8b70 0x7fef301f1b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:31.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.438+0000 7fef35c00640 1 --2- 172.21.15.6:0/4220025488 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fef100051c0 0x7fef3015e6c0 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:31.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.438+0000 7fef35c00640 1 -- 172.21.15.6:0/4220025488 >> 172.21.15.6:0/4220025488 conn(0x7fef300d3d30 msgr2=0x7fef300b51c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:31.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.439+0000 7fef35c00640 1 -- 172.21.15.6:0/4220025488 shutdown_connections 2024-03-04T09:28:31.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.439+0000 7fef35c00640 1 -- 172.21.15.6:0/4220025488 wait complete. 2024-03-04T09:28:31.456 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T09:25:07.935198+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-04T09:27:57.774529+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '55', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2024-03-04T09:27:58.886504+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '56', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-04T09:28:31.457 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-04T09:28:31.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.673+0000 7f2fe8e00640 1 Processor -- start 2024-03-04T09:28:31.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.673+0000 7f2fe8e00640 1 -- start start 2024-03-04T09:28:31.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.675+0000 7f2fe8e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f2fe415f810 0x7f2fe40b1b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:31.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.675+0000 7f2fe8e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fe415fe40 con 0x7f2fe40b2080 2024-03-04T09:28:31.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.675+0000 7f2fe8e00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fe4160120 con 0x7f2fe4145430 2024-03-04T09:28:31.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.675+0000 7f2fe8e00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fe40b24d0 con 0x7f2fe415f810 2024-03-04T09:28:31.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.675+0000 7f2fe3e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f2fe40b2080 0x7f2fe40b2fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:41146/0 (socket says 172.21.15.6:41146) 2024-03-04T09:28:31.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.675+0000 7f2fe3e00640 1 -- 172.21.15.6:0/4110445298 learned_addr learned my addr 172.21.15.6:0/4110445298 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:28:31.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.676+0000 7f2fe2a00640 1 --2- 172.21.15.6:0/4110445298 >> v2:172.21.15.146:3300/0 conn(0x7f2fe415f810 0x7f2fe40b1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:31.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.676+0000 7f2fe2000640 1 -- 172.21.15.6:0/4110445298 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3435741711 0 0) 0x7f2fe415fe40 con 0x7f2fe40b2080 2024-03-04T09:28:31.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.676+0000 7f2fe2000640 1 -- 172.21.15.6:0/4110445298 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fc8003520 con 0x7f2fe40b2080 2024-03-04T09:28:31.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.676+0000 7f2fe2000640 1 -- 172.21.15.6:0/4110445298 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2703724206 0 0) 0x7f2fc8003520 con 0x7f2fe40b2080 2024-03-04T09:28:31.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.676+0000 7f2fe2000640 1 -- 172.21.15.6:0/4110445298 >> v2:172.21.15.146:3300/0 conn(0x7f2fe415f810 msgr2=0x7f2fe40b1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:31.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.676+0000 7f2fe2000640 1 --2- 172.21.15.6:0/4110445298 >> v2:172.21.15.146:3300/0 conn(0x7f2fe415f810 0x7f2fe40b1b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:31.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.676+0000 7f2fe2000640 1 -- 172.21.15.6:0/4110445298 >> v1:172.21.15.146:6789/0 conn(0x7f2fe4145430 legacy=0x7f2fe415cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:31.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.677+0000 7f2fe2000640 1 -- 172.21.15.6:0/4110445298 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fe4160400 con 0x7f2fe40b2080 2024-03-04T09:28:31.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.677+0000 7f2fe2000640 1 -- 172.21.15.6:0/4110445298 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7f2fdc0019f0 con 0x7f2fe40b2080 2024-03-04T09:28:31.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.677+0000 7f2fe2000640 1 -- 172.21.15.6:0/4110445298 >> v1:172.21.15.6:6789/0 conn(0x7f2fe40b2080 legacy=0x7f2fe40b2fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:31.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.677+0000 7f2fe2000640 1 --2- 172.21.15.6:0/4110445298 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2fc8003a00 0x7f2fc8003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:31.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.677+0000 7f2fe2000640 1 --2- 172.21.15.6:0/4110445298 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2fc8004800 0x7f2fc8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:31.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.677+0000 7f2fe2000640 1 --2- 172.21.15.6:0/4110445298 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2fc80051c0 0x7f2fc800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:31.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.677+0000 7f2fe2000640 1 -- 172.21.15.6:0/4110445298 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fe40b24d0 con 0x7f2fc80051c0 2024-03-04T09:28:31.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.677+0000 7f2fe2000640 1 -- 172.21.15.6:0/4110445298 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fc800c0c0 con 0x7f2fc8003a00 2024-03-04T09:28:31.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.677+0000 7f2fe2000640 1 -- 172.21.15.6:0/4110445298 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fc800c2c0 con 0x7f2fc8004800 2024-03-04T09:28:31.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.677+0000 7f2fe3e00640 1 --2- 172.21.15.6:0/4110445298 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2fc80051c0 0x7f2fc800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:31.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.677+0000 7f2fe2a00640 1 --2- 172.21.15.6:0/4110445298 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2fc8004800 0x7f2fc8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:31.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.678+0000 7f2fe3400640 1 --2- 172.21.15.6:0/4110445298 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2fc8003a00 0x7f2fc8003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:31.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.678+0000 7f2fe3e00640 1 -- 172.21.15.6:0/4110445298 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2fc8004800 msgr2=0x7f2fc8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:31.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.678+0000 7f2fe3e00640 1 --2- 172.21.15.6:0/4110445298 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2fc8004800 0x7f2fc8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:31.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.678+0000 7f2fe3e00640 1 -- 172.21.15.6:0/4110445298 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2fc8003a00 msgr2=0x7f2fc8003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:31.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.678+0000 7f2fe3e00640 1 --2- 172.21.15.6:0/4110445298 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2fc8003a00 0x7f2fc8003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:31.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.678+0000 7f2fe3e00640 1 -- 172.21.15.6:0/4110445298 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fc8003520 con 0x7f2fc80051c0 2024-03-04T09:28:31.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.678+0000 7f2fe3400640 1 --2- 172.21.15.6:0/4110445298 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2fc8003a00 0x7f2fc8003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:28:31.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.679+0000 7f2fe3e00640 1 -- 172.21.15.6:0/4110445298 reap_dead start 2024-03-04T09:28:31.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.679+0000 7f2fe3e00640 1 --2- 172.21.15.6:0/4110445298 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2fc80051c0 0x7f2fc800b9c0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f2fdc00a770 tx=0x7f2fc8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfa035319f8b6c54 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:31.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.679+0000 7f2fe2000640 1 -- 172.21.15.6:0/4110445298 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fdc00a390 con 0x7f2fc80051c0 2024-03-04T09:28:31.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.679+0000 7f2fe2000640 1 -- 172.21.15.6:0/4110445298 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fdc034650 con 0x7f2fc80051c0 2024-03-04T09:28:31.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.679+0000 7f2fe2000640 1 -- 172.21.15.6:0/4110445298 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fdc034a40 con 0x7f2fc80051c0 2024-03-04T09:28:31.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.680+0000 7f2fe8e00640 1 -- 172.21.15.6:0/4110445298 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2fc80051c0 msgr2=0x7f2fc800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:31.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.680+0000 7f2fe8e00640 1 --2- 172.21.15.6:0/4110445298 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2fc80051c0 0x7f2fc800b9c0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f2fdc00a770 tx=0x7f2fc8005830 comp rx=0 tx=0).stop 2024-03-04T09:28:31.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.680+0000 7f2fe8e00640 1 -- 172.21.15.6:0/4110445298 shutdown_connections 2024-03-04T09:28:31.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.680+0000 7f2fe8e00640 1 --2- 172.21.15.6:0/4110445298 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2fc80051c0 0x7f2fc800b9c0 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:31.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.680+0000 7f2fe8e00640 1 -- 172.21.15.6:0/4110445298 >> 172.21.15.6:0/4110445298 conn(0x7f2fe406b3a0 msgr2=0x7f2fe406b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:31.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.680+0000 7f2fe8e00640 1 -- 172.21.15.6:0/4110445298 shutdown_connections 2024-03-04T09:28:31.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.680+0000 7f2fe8e00640 1 -- 172.21.15.6:0/4110445298 wait complete. 2024-03-04T09:28:31.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.682+0000 7f2fe8e00640 1 Processor -- start 2024-03-04T09:28:31.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.682+0000 7f2fe8e00640 1 -- start start 2024-03-04T09:28:31.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.683+0000 7f2fe8e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2fc80051c0 0x7f2fe4143d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:31.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.683+0000 7f2fe8e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2fe41f1b60 0x7f2fe41e6a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:31.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.683+0000 7f2fe8e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2fe41436c0 0x7f2fe41584f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:31.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.683+0000 7f2fe8e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fe4145990 con 0x7f2fe41436c0 2024-03-04T09:28:31.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.683+0000 7f2fe8e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fe4145810 con 0x7f2fc80051c0 2024-03-04T09:28:31.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.683+0000 7f2fe8e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fe40cde00 con 0x7f2fe41f1b60 2024-03-04T09:28:31.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.683+0000 7f2fe3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2fe41436c0 0x7f2fe41584f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:31.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.683+0000 7f2fe3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2fe41436c0 0x7f2fe41584f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53690/0 (socket says 172.21.15.6:53690) 2024-03-04T09:28:31.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.683+0000 7f2fe3e00640 1 -- 172.21.15.6:0/3684022793 learned_addr learned my addr 172.21.15.6:0/3684022793 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:28:31.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.683+0000 7f2fe3400640 1 --2- 172.21.15.6:0/3684022793 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2fc80051c0 0x7f2fe4143d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:31.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.683+0000 7f2fe2a00640 1 --2- 172.21.15.6:0/3684022793 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2fe41f1b60 0x7f2fe41e6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:31.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.683+0000 7f2fe3e00640 1 -- 172.21.15.6:0/3684022793 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2fe41f1b60 msgr2=0x7f2fe41e6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:31.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.683+0000 7f2fe3e00640 1 --2- 172.21.15.6:0/3684022793 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2fe41f1b60 0x7f2fe41e6a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:31.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.683+0000 7f2fe3e00640 1 -- 172.21.15.6:0/3684022793 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2fc80051c0 msgr2=0x7f2fe4143d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:31.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.683+0000 7f2fe3e00640 1 --2- 172.21.15.6:0/3684022793 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2fc80051c0 0x7f2fe4143d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:31.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.684+0000 7f2fe3e00640 1 -- 172.21.15.6:0/3684022793 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fe4143b70 con 0x7f2fe41436c0 2024-03-04T09:28:31.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.684+0000 7f2fe3e00640 1 --2- 172.21.15.6:0/3684022793 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2fe41436c0 0x7f2fe41584f0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f2fdc00a890 tx=0x7f2fe41602f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:31.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.684+0000 7f2fdb400640 1 -- 172.21.15.6:0/3684022793 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fdc039040 con 0x7f2fe41436c0 2024-03-04T09:28:31.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.684+0000 7f2fdb400640 1 -- 172.21.15.6:0/3684022793 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fdc043070 con 0x7f2fe41436c0 2024-03-04T09:28:31.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.684+0000 7f2fe8e00640 1 -- 172.21.15.6:0/3684022793 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fe4155820 con 0x7f2fe41436c0 2024-03-04T09:28:31.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.684+0000 7f2fdb400640 1 -- 172.21.15.6:0/3684022793 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fdc0019b0 con 0x7f2fe41436c0 2024-03-04T09:28:31.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.684+0000 7f2fe8e00640 1 -- 172.21.15.6:0/3684022793 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fe41e7470 con 0x7f2fe41436c0 2024-03-04T09:28:31.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.685+0000 7f2fe8e00640 1 -- 172.21.15.6:0/3684022793 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fe404f330 con 0x7f2fe41436c0 2024-03-04T09:28:31.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.686+0000 7f2fdb400640 1 -- 172.21.15.6:0/3684022793 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2fdc004b60 con 0x7f2fe41436c0 2024-03-04T09:28:31.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.687+0000 7f2fdb400640 1 --2- 172.21.15.6:0/3684022793 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2fc406b4d0 0x7f2fc406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:31.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.687+0000 7f2fdb400640 1 -- 172.21.15.6:0/3684022793 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2fdc0abef0 con 0x7f2fe41436c0 2024-03-04T09:28:31.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.688+0000 7f2fe3400640 1 --2- 172.21.15.6:0/3684022793 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2fc406b4d0 0x7f2fc406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:31.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.689+0000 7f2fe3400640 1 --2- 172.21.15.6:0/3684022793 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2fc406b4d0 0x7f2fc406d990 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f2fe41f5060 tx=0x7f2fd4009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:31.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.692+0000 7f2fdb400640 1 -- 172.21.15.6:0/3684022793 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2fe4143b70 con 0x7f2fe41436c0 2024-03-04T09:28:31.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.892+0000 7f2fe8e00640 1 -- 172.21.15.6:0/3684022793 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2fe4144be0 con 0x7f2fe41436c0 2024-03-04T09:28:31.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.893+0000 7f2fdb400640 1 -- 172.21.15.6:0/3684022793 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v56) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2fdc07acd0 con 0x7f2fe41436c0 2024-03-04T09:28:31.890 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-04T09:28:31.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.896+0000 7f2fe8e00640 1 -- 172.21.15.6:0/3684022793 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2fc406b4d0 msgr2=0x7f2fc406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:31.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.896+0000 7f2fe8e00640 1 --2- 172.21.15.6:0/3684022793 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2fc406b4d0 0x7f2fc406d990 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f2fe41f5060 tx=0x7f2fd4009040 comp rx=0 tx=0).stop 2024-03-04T09:28:31.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.896+0000 7f2fe8e00640 1 -- 172.21.15.6:0/3684022793 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2fe41436c0 msgr2=0x7f2fe41584f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:31.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.896+0000 7f2fe8e00640 1 --2- 172.21.15.6:0/3684022793 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2fe41436c0 0x7f2fe41584f0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f2fdc00a890 tx=0x7f2fe41602f0 comp rx=0 tx=0).stop 2024-03-04T09:28:31.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.896+0000 7f2fe8e00640 1 -- 172.21.15.6:0/3684022793 shutdown_connections 2024-03-04T09:28:31.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.896+0000 7f2fe8e00640 1 --2- 172.21.15.6:0/3684022793 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2fe41436c0 0x7f2fe41584f0 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:31.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.896+0000 7f2fe8e00640 1 --2- 172.21.15.6:0/3684022793 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2fc406b4d0 0x7f2fc406d990 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:31.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.896+0000 7f2fe8e00640 1 --2- 172.21.15.6:0/3684022793 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2fe41f1b60 0x7f2fe41e6a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:31.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.896+0000 7f2fe8e00640 1 --2- 172.21.15.6:0/3684022793 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2fc80051c0 0x7f2fe4143d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:31.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.896+0000 7f2fe8e00640 1 -- 172.21.15.6:0/3684022793 >> 172.21.15.6:0/3684022793 conn(0x7f2fe406b3a0 msgr2=0x7f2fe406b780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:31.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.896+0000 7f2fe8e00640 1 -- 172.21.15.6:0/3684022793 shutdown_connections 2024-03-04T09:28:31.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:31.896+0000 7f2fe8e00640 1 -- 172.21.15.6:0/3684022793 wait complete. 2024-03-04T09:28:31.911 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-04T09:28:32.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.128+0000 7fa48f000640 1 Processor -- start 2024-03-04T09:28:32.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.128+0000 7fa48f000640 1 -- start start 2024-03-04T09:28:32.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.130+0000 7fa48f000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fa48815bf20 0x7fa4880acd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:32.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.130+0000 7fa48f000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa488071e20 0x7fa4880ccad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:32.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.130+0000 7fa48f000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa488072df0 con 0x7fa488072520 2024-03-04T09:28:32.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.130+0000 7fa48f000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4880cdab0 con 0x7fa48815bf20 2024-03-04T09:28:32.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.130+0000 7fa48f000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4880b4d70 con 0x7fa488071e20 2024-03-04T09:28:32.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.130+0000 7fa47f400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fa488072520 0x7fa4880cdfb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:41158/0 (socket says 172.21.15.6:41158) 2024-03-04T09:28:32.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.131+0000 7fa47f400640 1 -- 172.21.15.6:0/2876797647 learned_addr learned my addr 172.21.15.6:0/2876797647 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:28:32.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.131+0000 7fa48d000640 1 --2- 172.21.15.6:0/2876797647 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa488071e20 0x7fa4880ccad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:32.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.131+0000 7fa487e00640 1 --2- 172.21.15.6:0/2876797647 >> v2:172.21.15.146:3300/0 conn(0x7fa48815bf20 0x7fa4880acd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:32.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.131+0000 7fa487400640 1 -- 172.21.15.6:0/2876797647 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2188262809 0 0) 0x7fa488072df0 con 0x7fa488072520 2024-03-04T09:28:32.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.131+0000 7fa487400640 1 -- 172.21.15.6:0/2876797647 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa46c003520 con 0x7fa488072520 2024-03-04T09:28:32.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.131+0000 7fa487400640 1 -- 172.21.15.6:0/2876797647 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1259995462 0 0) 0x7fa46c003520 con 0x7fa488072520 2024-03-04T09:28:32.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.131+0000 7fa487400640 1 -- 172.21.15.6:0/2876797647 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa488071e20 msgr2=0x7fa4880ccad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:32.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.131+0000 7fa487400640 1 --2- 172.21.15.6:0/2876797647 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa488071e20 0x7fa4880ccad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:32.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.131+0000 7fa487400640 1 -- 172.21.15.6:0/2876797647 >> v2:172.21.15.146:3300/0 conn(0x7fa48815bf20 msgr2=0x7fa4880acd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:32.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.131+0000 7fa487400640 1 --2- 172.21.15.6:0/2876797647 >> v2:172.21.15.146:3300/0 conn(0x7fa48815bf20 0x7fa4880acd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:32.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.131+0000 7fa487400640 1 -- 172.21.15.6:0/2876797647 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4880b5050 con 0x7fa488072520 2024-03-04T09:28:32.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.132+0000 7fa487400640 1 -- 172.21.15.6:0/2876797647 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7fa470001670 con 0x7fa488072520 2024-03-04T09:28:32.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.132+0000 7fa487400640 1 -- 172.21.15.6:0/2876797647 >> v1:172.21.15.6:6789/0 conn(0x7fa488072520 legacy=0x7fa4880cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:32.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.132+0000 7fa487400640 1 --2- 172.21.15.6:0/2876797647 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa46c0039b0 0x7fa46c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:32.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.132+0000 7fa487400640 1 --2- 172.21.15.6:0/2876797647 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa46c004800 0x7fa46c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:32.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.132+0000 7fa487400640 1 --2- 172.21.15.6:0/2876797647 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa488071e20 0x7fa46c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:32.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.132+0000 7fa487400640 1 -- 172.21.15.6:0/2876797647 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4880cdab0 con 0x7fa46c004800 2024-03-04T09:28:32.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.132+0000 7fa487400640 1 -- 172.21.15.6:0/2876797647 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4880b4d70 con 0x7fa46c0039b0 2024-03-04T09:28:32.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.132+0000 7fa487400640 1 -- 172.21.15.6:0/2876797647 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa46c00b9c0 con 0x7fa488071e20 2024-03-04T09:28:32.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.132+0000 7fa47f400640 1 --2- 172.21.15.6:0/2876797647 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa46c004800 0x7fa46c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:32.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.132+0000 7fa48d000640 1 --2- 172.21.15.6:0/2876797647 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa488071e20 0x7fa46c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:32.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.132+0000 7fa487e00640 1 --2- 172.21.15.6:0/2876797647 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa46c0039b0 0x7fa46c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:32.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.132+0000 7fa47f400640 1 -- 172.21.15.6:0/2876797647 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa488071e20 msgr2=0x7fa46c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:32.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.132+0000 7fa47f400640 1 --2- 172.21.15.6:0/2876797647 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa488071e20 0x7fa46c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:32.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.132+0000 7fa47f400640 1 -- 172.21.15.6:0/2876797647 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa46c0039b0 msgr2=0x7fa46c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:32.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.132+0000 7fa47f400640 1 --2- 172.21.15.6:0/2876797647 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa46c0039b0 0x7fa46c003dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:32.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.132+0000 7fa47f400640 1 -- 172.21.15.6:0/2876797647 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa46c003520 con 0x7fa46c004800 2024-03-04T09:28:32.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.133+0000 7fa48d000640 1 --2- 172.21.15.6:0/2876797647 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa488071e20 0x7fa46c0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:28:32.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.134+0000 7fa47f400640 1 --2- 172.21.15.6:0/2876797647 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa46c004800 0x7fa46c004c80 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fa47000c160 tx=0x7fa46c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cef3eefa25d06709 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:32.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.134+0000 7fa487400640 1 -- 172.21.15.6:0/2876797647 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa470007c80 con 0x7fa46c004800 2024-03-04T09:28:32.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.134+0000 7fa487400640 1 -- 172.21.15.6:0/2876797647 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa470030070 con 0x7fa46c004800 2024-03-04T09:28:32.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.134+0000 7fa487400640 1 -- 172.21.15.6:0/2876797647 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa470005d30 con 0x7fa46c004800 2024-03-04T09:28:32.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.134+0000 7fa48f000640 1 -- 172.21.15.6:0/2876797647 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa46c004800 msgr2=0x7fa46c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:32.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.134+0000 7fa48f000640 1 --2- 172.21.15.6:0/2876797647 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa46c004800 0x7fa46c004c80 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fa47000c160 tx=0x7fa46c005830 comp rx=0 tx=0).stop 2024-03-04T09:28:32.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.134+0000 7fa48d000640 1 -- 172.21.15.6:0/2876797647 reap_dead start 2024-03-04T09:28:32.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.135+0000 7fa48f000640 1 -- 172.21.15.6:0/2876797647 shutdown_connections 2024-03-04T09:28:32.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.135+0000 7fa48f000640 1 -- 172.21.15.6:0/2876797647 >> 172.21.15.6:0/2876797647 conn(0x7fa48806b3d0 msgr2=0x7fa48806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:32.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.135+0000 7fa48f000640 1 -- 172.21.15.6:0/2876797647 shutdown_connections 2024-03-04T09:28:32.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.135+0000 7fa48f000640 1 -- 172.21.15.6:0/2876797647 wait complete. 2024-03-04T09:28:32.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.137+0000 7fa48f000640 1 Processor -- start 2024-03-04T09:28:32.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.137+0000 7fa48f000640 1 -- start start 2024-03-04T09:28:32.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.138+0000 7fa48f000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa46c004800 0x7fa488143450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:32.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.138+0000 7fa48f000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa48814f960 0x7fa4881550c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:32.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.138+0000 7fa48f000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa48809ca20 0x7fa48809ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:32.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.138+0000 7fa48f000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4880b4b10 con 0x7fa48809ca20 2024-03-04T09:28:32.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.138+0000 7fa48f000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4880b4990 con 0x7fa46c004800 2024-03-04T09:28:32.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.138+0000 7fa48f000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4880ce6c0 con 0x7fa48814f960 2024-03-04T09:28:32.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.138+0000 7fa48d000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa48809ca20 0x7fa48809ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:32.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.138+0000 7fa48d000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa48809ca20 0x7fa48809ceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53716/0 (socket says 172.21.15.6:53716) 2024-03-04T09:28:32.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.138+0000 7fa48d000640 1 -- 172.21.15.6:0/3139069162 learned_addr learned my addr 172.21.15.6:0/3139069162 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:28:32.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.138+0000 7fa47f400640 1 --2- 172.21.15.6:0/3139069162 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa48814f960 0x7fa4881550c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:32.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.138+0000 7fa487e00640 1 --2- 172.21.15.6:0/3139069162 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa46c004800 0x7fa488143450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:32.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.138+0000 7fa48d000640 1 -- 172.21.15.6:0/3139069162 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa48814f960 msgr2=0x7fa4881550c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:32.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.138+0000 7fa48d000640 1 --2- 172.21.15.6:0/3139069162 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa48814f960 0x7fa4881550c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:32.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.139+0000 7fa48d000640 1 -- 172.21.15.6:0/3139069162 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa46c004800 msgr2=0x7fa488143450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:32.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.139+0000 7fa48d000640 1 --2- 172.21.15.6:0/3139069162 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa46c004800 0x7fa488143450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:32.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.139+0000 7fa48d000640 1 -- 172.21.15.6:0/3139069162 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa488155a30 con 0x7fa48809ca20 2024-03-04T09:28:32.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.139+0000 7fa487e00640 1 --2- 172.21.15.6:0/3139069162 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa46c004800 0x7fa488143450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:28:32.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.139+0000 7fa48d000640 1 --2- 172.21.15.6:0/3139069162 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa48809ca20 0x7fa48809ceb0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7fa4880cd670 tx=0x7fa48000f4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:32.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.139+0000 7fa484c00640 1 -- 172.21.15.6:0/3139069162 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa48001c070 con 0x7fa48809ca20 2024-03-04T09:28:32.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.139+0000 7fa484c00640 1 -- 172.21.15.6:0/3139069162 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa480017400 con 0x7fa48809ca20 2024-03-04T09:28:32.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.139+0000 7fa484c00640 1 -- 172.21.15.6:0/3139069162 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4800177d0 con 0x7fa48809ca20 2024-03-04T09:28:32.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.139+0000 7fa48f000640 1 -- 172.21.15.6:0/3139069162 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa488155d10 con 0x7fa48809ca20 2024-03-04T09:28:32.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.139+0000 7fa48f000640 1 -- 172.21.15.6:0/3139069162 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4881562d0 con 0x7fa48809ca20 2024-03-04T09:28:32.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.141+0000 7fa48f000640 1 -- 172.21.15.6:0/3139069162 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa48804f330 con 0x7fa48809ca20 2024-03-04T09:28:32.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.141+0000 7fa484c00640 1 -- 172.21.15.6:0/3139069162 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa48001f070 con 0x7fa48809ca20 2024-03-04T09:28:32.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.142+0000 7fa484c00640 1 --2- 172.21.15.6:0/3139069162 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa46406b4d0 0x7fa46406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:32.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.143+0000 7fa484c00640 1 -- 172.21.15.6:0/3139069162 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa48008a330 con 0x7fa48809ca20 2024-03-04T09:28:32.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.143+0000 7fa487e00640 1 --2- 172.21.15.6:0/3139069162 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa46406b4d0 0x7fa46406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:32.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.144+0000 7fa487e00640 1 --2- 172.21.15.6:0/3139069162 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa46406b4d0 0x7fa46406d990 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fa478009a60 tx=0x7fa478008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:32.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.147+0000 7fa484c00640 1 -- 172.21.15.6:0/3139069162 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa480012070 con 0x7fa48809ca20 2024-03-04T09:28:32.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.348+0000 7fa48f000640 1 -- 172.21.15.6:0/3139069162 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fa48815f720 con 0x7fa48809ca20 2024-03-04T09:28:32.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.348+0000 7fa484c00640 1 -- 172.21.15.6:0/3139069162 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v56) v1 ==== 104+0+11 (secure 0 0 0) 0x7fa480059110 con 0x7fa48809ca20 2024-03-04T09:28:32.345 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-03-04T09:28:32.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.351+0000 7fa48f000640 1 -- 172.21.15.6:0/3139069162 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa46406b4d0 msgr2=0x7fa46406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:32.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.351+0000 7fa48f000640 1 --2- 172.21.15.6:0/3139069162 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa46406b4d0 0x7fa46406d990 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fa478009a60 tx=0x7fa478008040 comp rx=0 tx=0).stop 2024-03-04T09:28:32.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.351+0000 7fa48f000640 1 -- 172.21.15.6:0/3139069162 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa48809ca20 msgr2=0x7fa48809ceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:32.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.351+0000 7fa48f000640 1 --2- 172.21.15.6:0/3139069162 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa48809ca20 0x7fa48809ceb0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7fa4880cd670 tx=0x7fa48000f4a0 comp rx=0 tx=0).stop 2024-03-04T09:28:32.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.351+0000 7fa48f000640 1 -- 172.21.15.6:0/3139069162 shutdown_connections 2024-03-04T09:28:32.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.351+0000 7fa48f000640 1 --2- 172.21.15.6:0/3139069162 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa48809ca20 0x7fa48809ceb0 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:32.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.351+0000 7fa48f000640 1 --2- 172.21.15.6:0/3139069162 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa46406b4d0 0x7fa46406d990 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:32.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.351+0000 7fa48f000640 1 --2- 172.21.15.6:0/3139069162 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa48814f960 0x7fa4881550c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:32.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.351+0000 7fa48f000640 1 --2- 172.21.15.6:0/3139069162 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa46c004800 0x7fa488143450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:32.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.351+0000 7fa48f000640 1 -- 172.21.15.6:0/3139069162 >> 172.21.15.6:0/3139069162 conn(0x7fa48806b3d0 msgr2=0x7fa4880b2c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:32.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.351+0000 7fa48f000640 1 -- 172.21.15.6:0/3139069162 shutdown_connections 2024-03-04T09:28:32.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.351+0000 7fa48f000640 1 -- 172.21.15.6:0/3139069162 wait complete. 2024-03-04T09:28:32.368 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-04T09:28:32.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.597+0000 7f2114800640 1 Processor -- start 2024-03-04T09:28:32.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.598+0000 7f2114800640 1 -- start start 2024-03-04T09:28:32.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.599+0000 7f2114800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f210c071e20 0x7f210c0ccad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:32.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.599+0000 7f2114800640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f210c072df0 con 0x7f210c072520 2024-03-04T09:28:32.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.599+0000 7f2114800640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f210c0cdab0 con 0x7f210c15bf20 2024-03-04T09:28:32.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.599+0000 7f2114800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f210c0b4d70 con 0x7f210c071e20 2024-03-04T09:28:32.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.600+0000 7f2111400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f210c072520 0x7f210c0cdfb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:41164/0 (socket says 172.21.15.6:41164) 2024-03-04T09:28:32.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.600+0000 7f2111400640 1 -- 172.21.15.6:0/2164267565 learned_addr learned my addr 172.21.15.6:0/2164267565 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:28:32.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.600+0000 7f2112800640 1 --2- 172.21.15.6:0/2164267565 >> v2:172.21.15.6:3300/0 conn(0x7f210c071e20 0x7f210c0ccad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:32.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.600+0000 7f2110a00640 1 -- 172.21.15.6:0/2164267565 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1215657550 0 0) 0x7f210c072df0 con 0x7f210c072520 2024-03-04T09:28:32.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.600+0000 7f2110a00640 1 -- 172.21.15.6:0/2164267565 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20f8003520 con 0x7f210c072520 2024-03-04T09:28:32.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.600+0000 7f2112800640 1 -- 172.21.15.6:0/2164267565 >> v1:172.21.15.146:6789/0 conn(0x7f210c15bf20 legacy=0x7f210c0acd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:32.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.600+0000 7f2112800640 1 -- 172.21.15.6:0/2164267565 >> v1:172.21.15.6:6789/0 conn(0x7f210c072520 legacy=0x7f210c0cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:32.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.600+0000 7f2112800640 1 -- 172.21.15.6:0/2164267565 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f210c0b5050 con 0x7f210c071e20 2024-03-04T09:28:32.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.602+0000 7f2112800640 1 --2- 172.21.15.6:0/2164267565 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f210c071e20 0x7f210c0ccad0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f2108009d90 tx=0x7f210802bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3425c094e476a2dc server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:32.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.602+0000 7f2110a00640 1 -- 172.21.15.6:0/2164267565 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2108039070 con 0x7f210c071e20 2024-03-04T09:28:32.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.602+0000 7f2110a00640 1 -- 172.21.15.6:0/2164267565 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f210802cd00 con 0x7f210c071e20 2024-03-04T09:28:32.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.602+0000 7f2110a00640 1 -- 172.21.15.6:0/2164267565 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2108034660 con 0x7f210c071e20 2024-03-04T09:28:32.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.603+0000 7f2114800640 1 -- 172.21.15.6:0/2164267565 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f210c071e20 msgr2=0x7f210c0ccad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:32.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.603+0000 7f2114800640 1 --2- 172.21.15.6:0/2164267565 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f210c071e20 0x7f210c0ccad0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f2108009d90 tx=0x7f210802bf50 comp rx=0 tx=0).stop 2024-03-04T09:28:32.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.603+0000 7f2114800640 1 -- 172.21.15.6:0/2164267565 shutdown_connections 2024-03-04T09:28:32.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.603+0000 7f2114800640 1 --2- 172.21.15.6:0/2164267565 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f210c071e20 0x7f210c0ccad0 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:32.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.603+0000 7f2114800640 1 -- 172.21.15.6:0/2164267565 >> 172.21.15.6:0/2164267565 conn(0x7f210c06b3d0 msgr2=0x7f210c06b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:32.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.603+0000 7f2114800640 1 -- 172.21.15.6:0/2164267565 shutdown_connections 2024-03-04T09:28:32.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.603+0000 7f2114800640 1 -- 172.21.15.6:0/2164267565 wait complete. 2024-03-04T09:28:32.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.605+0000 7f2114800640 1 Processor -- start 2024-03-04T09:28:32.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.605+0000 7f2114800640 1 -- start start 2024-03-04T09:28:32.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.606+0000 7f2114800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f210c071e20 0x7f210c09b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:32.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.606+0000 7f2114800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f210c072520 0x7f210c09bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:32.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.606+0000 7f2114800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f210c15bf20 0x7f210c14fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:32.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.606+0000 7f2114800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f210c0ad450 con 0x7f210c071e20 2024-03-04T09:28:32.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.606+0000 7f2114800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f210c0b4990 con 0x7f210c15bf20 2024-03-04T09:28:32.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.606+0000 7f2114800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f210c14a210 con 0x7f210c072520 2024-03-04T09:28:32.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.606+0000 7f2111e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f210c071e20 0x7f210c09b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:32.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.606+0000 7f2111e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f210c071e20 0x7f210c09b420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53742/0 (socket says 172.21.15.6:53742) 2024-03-04T09:28:32.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.606+0000 7f2111e00640 1 -- 172.21.15.6:0/3175801402 learned_addr learned my addr 172.21.15.6:0/3175801402 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:28:32.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.606+0000 7f2111400640 1 --2- 172.21.15.6:0/3175801402 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f210c072520 0x7f210c09bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:32.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.607+0000 7f2112800640 1 --2- 172.21.15.6:0/3175801402 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f210c15bf20 0x7f210c14fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:32.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.607+0000 7f2111e00640 1 -- 172.21.15.6:0/3175801402 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f210c072520 msgr2=0x7f210c09bcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:32.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.607+0000 7f2111e00640 1 --2- 172.21.15.6:0/3175801402 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f210c072520 0x7f210c09bcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:32.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.607+0000 7f2111e00640 1 -- 172.21.15.6:0/3175801402 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f210c15bf20 msgr2=0x7f210c14fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:32.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.607+0000 7f2111e00640 1 --2- 172.21.15.6:0/3175801402 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f210c15bf20 0x7f210c14fbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:32.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.607+0000 7f2111e00640 1 -- 172.21.15.6:0/3175801402 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f210c14ce60 con 0x7f210c071e20 2024-03-04T09:28:32.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.607+0000 7f2111e00640 1 --2- 172.21.15.6:0/3175801402 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f210c071e20 0x7f210c09b420 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f210400c400 tx=0x7f210400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:32.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.607+0000 7f2102000640 1 -- 172.21.15.6:0/3175801402 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21040052d0 con 0x7f210c071e20 2024-03-04T09:28:32.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.607+0000 7f2102000640 1 -- 172.21.15.6:0/3175801402 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2104005490 con 0x7f210c071e20 2024-03-04T09:28:32.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.607+0000 7f2102000640 1 -- 172.21.15.6:0/3175801402 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2104007260 con 0x7f210c071e20 2024-03-04T09:28:32.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.607+0000 7f2114800640 1 -- 172.21.15.6:0/3175801402 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f210c149ee0 con 0x7f210c071e20 2024-03-04T09:28:32.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.608+0000 7f2114800640 1 -- 172.21.15.6:0/3175801402 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f210c14cb80 con 0x7f210c071e20 2024-03-04T09:28:32.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.609+0000 7f2114800640 1 -- 172.21.15.6:0/3175801402 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f210c04f330 con 0x7f210c071e20 2024-03-04T09:28:32.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.610+0000 7f2102000640 1 -- 172.21.15.6:0/3175801402 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2104007400 con 0x7f210c071e20 2024-03-04T09:28:32.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.611+0000 7f2102000640 1 --2- 172.21.15.6:0/3175801402 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f20f006b4d0 0x7f20f006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:32.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.611+0000 7f2102000640 1 -- 172.21.15.6:0/3175801402 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2104089740 con 0x7f210c071e20 2024-03-04T09:28:32.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.611+0000 7f2111400640 1 --2- 172.21.15.6:0/3175801402 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f20f006b4d0 0x7f20f006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:32.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.613+0000 7f2111400640 1 --2- 172.21.15.6:0/3175801402 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f20f006b4d0 0x7f20f006d990 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f210c15ec10 tx=0x7f20fc009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:32.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.615+0000 7f2102000640 1 -- 172.21.15.6:0/3175801402 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f21040585b0 con 0x7f210c071e20 2024-03-04T09:28:32.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.817+0000 7f2114800640 1 -- 172.21.15.6:0/3175801402 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f210c06e970 con 0x7f210c071e20 2024-03-04T09:28:32.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.818+0000 7f2102000640 1 -- 172.21.15.6:0/3175801402 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v56) v1 ==== 100+0+11 (secure 0 0 0) 0x7f2104057eb0 con 0x7f210c071e20 2024-03-04T09:28:32.817 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 64 2024-03-04T09:28:32.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.820+0000 7f2114800640 1 -- 172.21.15.6:0/3175801402 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f20f006b4d0 msgr2=0x7f20f006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:32.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.820+0000 7f2114800640 1 --2- 172.21.15.6:0/3175801402 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f20f006b4d0 0x7f20f006d990 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f210c15ec10 tx=0x7f20fc009040 comp rx=0 tx=0).stop 2024-03-04T09:28:32.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.820+0000 7f2114800640 1 -- 172.21.15.6:0/3175801402 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f210c071e20 msgr2=0x7f210c09b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:32.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.820+0000 7f2114800640 1 --2- 172.21.15.6:0/3175801402 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f210c071e20 0x7f210c09b420 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f210400c400 tx=0x7f210400c8c0 comp rx=0 tx=0).stop 2024-03-04T09:28:32.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.821+0000 7f2114800640 1 -- 172.21.15.6:0/3175801402 shutdown_connections 2024-03-04T09:28:32.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.821+0000 7f2114800640 1 --2- 172.21.15.6:0/3175801402 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f210c15bf20 0x7f210c14fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:32.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.821+0000 7f2114800640 1 --2- 172.21.15.6:0/3175801402 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f20f006b4d0 0x7f20f006d990 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:32.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.821+0000 7f2114800640 1 --2- 172.21.15.6:0/3175801402 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f210c072520 0x7f210c09bcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:32.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.821+0000 7f2114800640 1 --2- 172.21.15.6:0/3175801402 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f210c071e20 0x7f210c09b420 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:32.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.821+0000 7f2114800640 1 -- 172.21.15.6:0/3175801402 >> 172.21.15.6:0/3175801402 conn(0x7f210c06b3d0 msgr2=0x7f210c06e000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:32.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.821+0000 7f2114800640 1 -- 172.21.15.6:0/3175801402 shutdown_connections 2024-03-04T09:28:32.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:32.821+0000 7f2114800640 1 -- 172.21.15.6:0/3175801402 wait complete. 2024-03-04T09:28:32.837 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:28:33.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.063+0000 7f249f200640 1 Processor -- start 2024-03-04T09:28:33.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.063+0000 7f249f200640 1 -- start start 2024-03-04T09:28:33.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.065+0000 7f249f200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f24a01552b0 0x7f24a015b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:33.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.065+0000 7f249f200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f24a015e810 0x7f24a00cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:33.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.065+0000 7f249f200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24a015ed10 con 0x7f24a00cce90 2024-03-04T09:28:33.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.065+0000 7f249f200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24a00cd300 con 0x7f24a015e810 2024-03-04T09:28:33.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.065+0000 7f249f200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24a015eff0 con 0x7f24a01552b0 2024-03-04T09:28:33.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.065+0000 7f2497e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f24a015e810 0x7f24a00cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:33.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.065+0000 7f249e800640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f24a00cce90 0x7f24a006c7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:41172/0 (socket says 172.21.15.6:41172) 2024-03-04T09:28:33.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.065+0000 7f249e800640 1 -- 172.21.15.6:0/2825389307 learned_addr learned my addr 172.21.15.6:0/2825389307 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:28:33.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.065+0000 7f249de00640 1 --2- 172.21.15.6:0/2825389307 >> v2:172.21.15.146:3300/0 conn(0x7f24a01552b0 0x7f24a015b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:33.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.066+0000 7f2497e00640 1 -- 172.21.15.6:0/2825389307 >> v2:172.21.15.146:3300/0 conn(0x7f24a01552b0 msgr2=0x7f24a015b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:33.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.066+0000 7f2497e00640 1 --2- 172.21.15.6:0/2825389307 >> v2:172.21.15.146:3300/0 conn(0x7f24a01552b0 0x7f24a015b1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:33.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.066+0000 7f2497e00640 1 -- 172.21.15.6:0/2825389307 >> v1:172.21.15.6:6789/0 conn(0x7f24a00cce90 legacy=0x7f24a006c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:33.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.066+0000 7f2497e00640 1 -- 172.21.15.6:0/2825389307 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24a015f2d0 con 0x7f24a015e810 2024-03-04T09:28:33.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.066+0000 7f249d400640 1 -- 172.21.15.6:0/2825389307 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1675470434 0 0) 0x7f24a015ed10 con 0x7f24a00cce90 2024-03-04T09:28:33.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.067+0000 7f2497e00640 1 --2- 172.21.15.6:0/2825389307 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f24a015e810 0x7f24a00cc950 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f24a00cd540 tx=0x7f248802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71ae86f62b84a1f6 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:33.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.067+0000 7f249d400640 1 -- 172.21.15.6:0/2825389307 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24880315a0 con 0x7f24a015e810 2024-03-04T09:28:33.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.067+0000 7f249d400640 1 -- 172.21.15.6:0/2825389307 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2488037070 con 0x7f24a015e810 2024-03-04T09:28:33.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.067+0000 7f249d400640 1 -- 172.21.15.6:0/2825389307 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2488031960 con 0x7f24a015e810 2024-03-04T09:28:33.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.068+0000 7f249f200640 1 -- 172.21.15.6:0/2825389307 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f24a015e810 msgr2=0x7f24a00cc950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:33.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.068+0000 7f249f200640 1 --2- 172.21.15.6:0/2825389307 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f24a015e810 0x7f24a00cc950 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f24a00cd540 tx=0x7f248802b840 comp rx=0 tx=0).stop 2024-03-04T09:28:33.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.068+0000 7f249f200640 1 -- 172.21.15.6:0/2825389307 shutdown_connections 2024-03-04T09:28:33.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.068+0000 7f249f200640 1 --2- 172.21.15.6:0/2825389307 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f24a015e810 0x7f24a00cc950 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:33.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.068+0000 7f249f200640 1 --2- 172.21.15.6:0/2825389307 >> v2:172.21.15.146:3300/0 conn(0x7f24a01552b0 0x7f24a015b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:33.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.068+0000 7f249f200640 1 -- 172.21.15.6:0/2825389307 >> 172.21.15.6:0/2825389307 conn(0x7f24a006b3d0 msgr2=0x7f24a006b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:33.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.068+0000 7f249f200640 1 -- 172.21.15.6:0/2825389307 shutdown_connections 2024-03-04T09:28:33.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.068+0000 7f249f200640 1 -- 172.21.15.6:0/2825389307 wait complete. 2024-03-04T09:28:33.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.070+0000 7f249f200640 1 Processor -- start 2024-03-04T09:28:33.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.070+0000 7f249f200640 1 -- start start 2024-03-04T09:28:33.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.071+0000 7f249f200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f24a00cce90 0x7f24a0145500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:33.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.071+0000 7f249f200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f24a01552b0 0x7f24a0157a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:33.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.071+0000 7f249f200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f24a015e810 0x7f24a0157f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:33.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.071+0000 7f249f200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24a00b3760 con 0x7f24a00cce90 2024-03-04T09:28:33.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.071+0000 7f249f200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24a0155690 con 0x7f24a015e810 2024-03-04T09:28:33.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.071+0000 7f249f200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24a006dd30 con 0x7f24a01552b0 2024-03-04T09:28:33.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.071+0000 7f249de00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f24a00cce90 0x7f24a0145500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:33.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.071+0000 7f249de00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f24a00cce90 0x7f24a0145500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53770/0 (socket says 172.21.15.6:53770) 2024-03-04T09:28:33.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.071+0000 7f249de00640 1 -- 172.21.15.6:0/1812118980 learned_addr learned my addr 172.21.15.6:0/1812118980 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:28:33.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.072+0000 7f2497e00640 1 --2- 172.21.15.6:0/1812118980 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f24a01552b0 0x7f24a0157a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:33.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.072+0000 7f249e800640 1 --2- 172.21.15.6:0/1812118980 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f24a015e810 0x7f24a0157f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:33.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.072+0000 7f249de00640 1 -- 172.21.15.6:0/1812118980 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f24a01552b0 msgr2=0x7f24a0157a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:33.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.072+0000 7f249de00640 1 --2- 172.21.15.6:0/1812118980 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f24a01552b0 0x7f24a0157a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:33.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.072+0000 7f249de00640 1 -- 172.21.15.6:0/1812118980 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f24a015e810 msgr2=0x7f24a0157f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:33.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.072+0000 7f249de00640 1 --2- 172.21.15.6:0/1812118980 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f24a015e810 0x7f24a0157f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:33.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.072+0000 7f249de00640 1 -- 172.21.15.6:0/1812118980 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24a01597c0 con 0x7f24a00cce90 2024-03-04T09:28:33.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.072+0000 7f249e800640 1 --2- 172.21.15.6:0/1812118980 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f24a015e810 0x7f24a0157f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:28:33.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.072+0000 7f2497e00640 1 --2- 172.21.15.6:0/1812118980 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f24a01552b0 0x7f24a0157a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:28:33.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.072+0000 7f249de00640 1 --2- 172.21.15.6:0/1812118980 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f24a00cce90 0x7f24a0145500 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f249000d830 tx=0x7f249000dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:33.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.072+0000 7f2496000640 1 -- 172.21.15.6:0/1812118980 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f249000b6c0 con 0x7f24a00cce90 2024-03-04T09:28:33.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.072+0000 7f2496000640 1 -- 172.21.15.6:0/1812118980 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2490019070 con 0x7f24a00cce90 2024-03-04T09:28:33.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.072+0000 7f2496000640 1 -- 172.21.15.6:0/1812118980 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f249000ba80 con 0x7f24a00cce90 2024-03-04T09:28:33.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.072+0000 7f249f200640 1 -- 172.21.15.6:0/1812118980 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24a0155810 con 0x7f24a00cce90 2024-03-04T09:28:33.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.073+0000 7f249f200640 1 -- 172.21.15.6:0/1812118980 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24a0155dd0 con 0x7f24a00cce90 2024-03-04T09:28:33.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.074+0000 7f249f200640 1 -- 172.21.15.6:0/1812118980 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24a004f330 con 0x7f24a00cce90 2024-03-04T09:28:33.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.075+0000 7f2496000640 1 -- 172.21.15.6:0/1812118980 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f249001e460 con 0x7f24a00cce90 2024-03-04T09:28:33.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.075+0000 7f2496000640 1 --2- 172.21.15.6:0/1812118980 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f247c06b4d0 0x7f247c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:33.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.076+0000 7f2496000640 1 -- 172.21.15.6:0/1812118980 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f249008a820 con 0x7f24a00cce90 2024-03-04T09:28:33.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.076+0000 7f2497e00640 1 --2- 172.21.15.6:0/1812118980 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f247c06b4d0 0x7f247c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:33.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.077+0000 7f2497e00640 1 --2- 172.21.15.6:0/1812118980 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f247c06b4d0 0x7f247c06d990 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f2488036000 tx=0x7f2488008030 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:33.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.080+0000 7f2496000640 1 -- 172.21.15.6:0/1812118980 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f24a01597c0 con 0x7f24a00cce90 2024-03-04T09:28:33.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.317+0000 7f249f200640 1 -- 172.21.15.6:0/1812118980 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f24a00cf4a0 con 0x7f24a00cce90 2024-03-04T09:28:33.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.318+0000 7f2496000640 1 -- 172.21.15.6:0/1812118980 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+4971 (secure 0 0 0) 0x7f2490059730 con 0x7f24a00cce90 2024-03-04T09:28:33.315 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:28:33.315 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":34,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5030,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/1411942502","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1411942502},{"type":"v1","addr":"172.21.15.6:6834","nonce":1411942502}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28},{"gid":5039,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/1628166413","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":1628166413},{"type":"v1","addr":"172.21.15.146:6837","nonce":1628166413}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":5323,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/131491768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":131491768},{"type":"v1","addr":"172.21.15.6:6835","nonce":131491768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:27:59.854470+0000","modified":"2024-03-04T09:28:02.945053+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5326},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5326":{"gid":5326,"name":"b","rank":0,"incarnation":30,"state":"up:active","state_seq":3,"addr":"172.21.15.146:6836/527855083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":527855083},{"type":"v1","addr":"172.21.15.146:6836","nonce":527855083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5326,"qdb_cluster":[5326]},"id":4}]} 2024-03-04T09:28:33.315 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 34 2024-03-04T09:28:33.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.320+0000 7f249f200640 1 -- 172.21.15.6:0/1812118980 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f247c06b4d0 msgr2=0x7f247c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:33.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.320+0000 7f249f200640 1 --2- 172.21.15.6:0/1812118980 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f247c06b4d0 0x7f247c06d990 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f2488036000 tx=0x7f2488008030 comp rx=0 tx=0).stop 2024-03-04T09:28:33.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.320+0000 7f249f200640 1 -- 172.21.15.6:0/1812118980 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f24a00cce90 msgr2=0x7f24a0145500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:33.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.320+0000 7f249f200640 1 --2- 172.21.15.6:0/1812118980 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f24a00cce90 0x7f24a0145500 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f249000d830 tx=0x7f249000dcf0 comp rx=0 tx=0).stop 2024-03-04T09:28:33.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.320+0000 7f249f200640 1 -- 172.21.15.6:0/1812118980 shutdown_connections 2024-03-04T09:28:33.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.320+0000 7f249f200640 1 --2- 172.21.15.6:0/1812118980 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f24a015e810 0x7f24a0157f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:33.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.320+0000 7f249f200640 1 --2- 172.21.15.6:0/1812118980 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f247c06b4d0 0x7f247c06d990 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:33.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.320+0000 7f249f200640 1 --2- 172.21.15.6:0/1812118980 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f24a01552b0 0x7f24a0157a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:33.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.320+0000 7f249f200640 1 --2- 172.21.15.6:0/1812118980 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f24a00cce90 0x7f24a0145500 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:33.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.320+0000 7f249f200640 1 -- 172.21.15.6:0/1812118980 >> 172.21.15.6:0/1812118980 conn(0x7f24a006b3d0 msgr2=0x7f24a014b580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:33.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.320+0000 7f249f200640 1 -- 172.21.15.6:0/1812118980 shutdown_connections 2024-03-04T09:28:33.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.320+0000 7f249f200640 1 -- 172.21.15.6:0/1812118980 wait complete. 2024-03-04T09:28:33.334 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T09:28:33.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.549+0000 7f40d0200640 1 Processor -- start 2024-03-04T09:28:33.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.549+0000 7f40d0200640 1 -- start start 2024-03-04T09:28:33.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.551+0000 7f40d0200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f40c8161670 0x7f40c8070c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:33.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.551+0000 7f40d0200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f40c806e240 0x7f40c80711b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:33.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.551+0000 7f40d0200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40c80ce9a0 con 0x7f40c806cfe0 2024-03-04T09:28:33.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.551+0000 7f40d0200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40c80730d0 con 0x7f40c806e240 2024-03-04T09:28:33.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.551+0000 7f40d0200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40c80cec80 con 0x7f40c8161670 2024-03-04T09:28:33.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.551+0000 7f40cce00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f40c806e240 0x7f40c80711b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:33.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.551+0000 7f40cce00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f40c806e240 0x7f40c80711b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53776/0 (socket says 172.21.15.6:53776) 2024-03-04T09:28:33.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.551+0000 7f40cce00640 1 -- 172.21.15.6:0/756379915 learned_addr learned my addr 172.21.15.6:0/756379915 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:28:33.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.552+0000 7f40cd800640 1 --2- 172.21.15.6:0/756379915 >> v2:172.21.15.146:3300/0 conn(0x7f40c8161670 0x7f40c8070c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:33.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.552+0000 7f40cce00640 1 -- 172.21.15.6:0/756379915 >> v2:172.21.15.146:3300/0 conn(0x7f40c8161670 msgr2=0x7f40c8070c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:33.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.552+0000 7f40cce00640 1 --2- 172.21.15.6:0/756379915 >> v2:172.21.15.146:3300/0 conn(0x7f40c8161670 0x7f40c8070c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:33.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.552+0000 7f40cce00640 1 -- 172.21.15.6:0/756379915 >> v1:172.21.15.6:6789/0 conn(0x7f40c806cfe0 legacy=0x7f40c806d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:33.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.552+0000 7f40c7e00640 1 -- 172.21.15.6:0/756379915 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 440355032 0 0) 0x7f40c80ce9a0 con 0x7f40c806cfe0 2024-03-04T09:28:33.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.552+0000 7f40cce00640 1 -- 172.21.15.6:0/756379915 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40c80cef60 con 0x7f40c806e240 2024-03-04T09:28:33.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.553+0000 7f40cce00640 1 --2- 172.21.15.6:0/756379915 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f40c806e240 0x7f40c80711b0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f40c8073610 tx=0x7f40b402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa00e90f1b88c6e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:33.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.553+0000 7f40c7e00640 1 -- 172.21.15.6:0/756379915 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40b4035020 con 0x7f40c806e240 2024-03-04T09:28:33.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.553+0000 7f40c7e00640 1 -- 172.21.15.6:0/756379915 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40b4031870 con 0x7f40c806e240 2024-03-04T09:28:33.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.553+0000 7f40c7e00640 1 -- 172.21.15.6:0/756379915 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40b4031c40 con 0x7f40c806e240 2024-03-04T09:28:33.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.554+0000 7f40d0200640 1 -- 172.21.15.6:0/756379915 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f40c806e240 msgr2=0x7f40c80711b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:33.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.554+0000 7f40d0200640 1 --2- 172.21.15.6:0/756379915 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f40c806e240 0x7f40c80711b0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f40c8073610 tx=0x7f40b402b910 comp rx=0 tx=0).stop 2024-03-04T09:28:33.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.554+0000 7f40d0200640 1 -- 172.21.15.6:0/756379915 shutdown_connections 2024-03-04T09:28:33.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.554+0000 7f40d0200640 1 --2- 172.21.15.6:0/756379915 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f40c806e240 0x7f40c80711b0 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:33.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.554+0000 7f40d0200640 1 --2- 172.21.15.6:0/756379915 >> v2:172.21.15.146:3300/0 conn(0x7f40c8161670 0x7f40c8070c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:33.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.554+0000 7f40d0200640 1 -- 172.21.15.6:0/756379915 >> 172.21.15.6:0/756379915 conn(0x7f40c806afb0 msgr2=0x7f40c806b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:33.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.554+0000 7f40d0200640 1 -- 172.21.15.6:0/756379915 shutdown_connections 2024-03-04T09:28:33.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.554+0000 7f40d0200640 1 -- 172.21.15.6:0/756379915 wait complete. 2024-03-04T09:28:33.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.556+0000 7f40d0200640 1 Processor -- start 2024-03-04T09:28:33.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.556+0000 7f40d0200640 1 -- start start 2024-03-04T09:28:33.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.557+0000 7f40d0200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f40c806cfe0 0x7f40c814d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:33.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.557+0000 7f40d0200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f40c806e240 0x7f40c806d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:33.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.557+0000 7f40d0200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f40c8161670 0x7f40c806d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:33.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.557+0000 7f40d0200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40c80ce600 con 0x7f40c806e240 2024-03-04T09:28:33.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.557+0000 7f40d0200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40c80cf520 con 0x7f40c806cfe0 2024-03-04T09:28:33.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.557+0000 7f40d0200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40c80ce780 con 0x7f40c8161670 2024-03-04T09:28:33.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.557+0000 7f40cce00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f40c806e240 0x7f40c806d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:33.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.557+0000 7f40cce00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f40c806e240 0x7f40c806d440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53782/0 (socket says 172.21.15.6:53782) 2024-03-04T09:28:33.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.557+0000 7f40cce00640 1 -- 172.21.15.6:0/1331056587 learned_addr learned my addr 172.21.15.6:0/1331056587 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:28:33.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.558+0000 7f40ce200640 1 --2- 172.21.15.6:0/1331056587 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f40c8161670 0x7f40c806d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:33.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.558+0000 7f40cce00640 1 -- 172.21.15.6:0/1331056587 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f40c8161670 msgr2=0x7f40c806d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:33.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.558+0000 7f40cce00640 1 --2- 172.21.15.6:0/1331056587 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f40c8161670 0x7f40c806d980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:33.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.558+0000 7f40cce00640 1 -- 172.21.15.6:0/1331056587 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f40c806cfe0 msgr2=0x7f40c814d860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:33.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.558+0000 7f40cce00640 1 --2- 172.21.15.6:0/1331056587 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f40c806cfe0 0x7f40c814d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:33.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.558+0000 7f40cce00640 1 -- 172.21.15.6:0/1331056587 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40c809abb0 con 0x7f40c806e240 2024-03-04T09:28:33.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.558+0000 7f40cce00640 1 --2- 172.21.15.6:0/1331056587 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f40c806e240 0x7f40c806d440 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f40b4000c00 tx=0x7f40b40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:33.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.559+0000 7f40c5600640 1 -- 172.21.15.6:0/1331056587 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40b4041070 con 0x7f40c806e240 2024-03-04T09:28:33.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.559+0000 7f40c5600640 1 -- 172.21.15.6:0/1331056587 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40b403ad00 con 0x7f40c806e240 2024-03-04T09:28:33.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.559+0000 7f40c5600640 1 -- 172.21.15.6:0/1331056587 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40b4035040 con 0x7f40c806e240 2024-03-04T09:28:33.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.559+0000 7f40d0200640 1 -- 172.21.15.6:0/1331056587 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40c809ae90 con 0x7f40c806e240 2024-03-04T09:28:33.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.559+0000 7f40d0200640 1 -- 172.21.15.6:0/1331056587 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40c80cad40 con 0x7f40c806e240 2024-03-04T09:28:33.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.560+0000 7f40d0200640 1 -- 172.21.15.6:0/1331056587 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40c8069510 con 0x7f40c806e240 2024-03-04T09:28:33.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.561+0000 7f40c5600640 1 -- 172.21.15.6:0/1331056587 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f40b4038020 con 0x7f40c806e240 2024-03-04T09:28:33.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.562+0000 7f40c5600640 1 --2- 172.21.15.6:0/1331056587 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f40a806b4d0 0x7f40a806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:33.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.562+0000 7f40c5600640 1 -- 172.21.15.6:0/1331056587 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f40b40ac140 con 0x7f40c806e240 2024-03-04T09:28:33.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.562+0000 7f40cd800640 1 --2- 172.21.15.6:0/1331056587 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f40a806b4d0 0x7f40a806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:33.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.564+0000 7f40cd800640 1 --2- 172.21.15.6:0/1331056587 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f40a806b4d0 0x7f40a806d990 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f40bc0070b0 tx=0x7f40bc007040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:33.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.566+0000 7f40c5600640 1 -- 172.21.15.6:0/1331056587 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f40b407afb0 con 0x7f40c806e240 2024-03-04T09:28:33.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.765+0000 7f40d0200640 1 -- 172.21.15.6:0/1331056587 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f40c815fd00 con 0x7f40c806e240 2024-03-04T09:28:33.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.767+0000 7f40c5600640 1 -- 172.21.15.6:0/1331056587 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v56) v1 ==== 74+0+16617 (secure 0 0 0) 0x7f40c815fd00 con 0x7f40c806e240 2024-03-04T09:28:33.765 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:28:33.765 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":56,"fsid":"edce0578-5ae4-4402-b45a-552bdef84d7c","created":"2024-03-04T09:24:54.523780+0000","modified":"2024-03-04T09:28:01.413693+0000","last_up_change":"2024-03-04T09:25:07.089049+0000","last_in_change":"2024-03-04T09:24:57.982077+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T09:25:07.935198+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-04T09:27:57.774529+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-04T09:27:58.886504+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a96574f-81f1-4bc6-8463-757520b84a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6825","nonce":1364449401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6827","nonce":1364449401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6831","nonce":1364449401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6829","nonce":1364449401}]},"public_addr":"172.21.15.6:6825/1364449401","cluster_addr":"172.21.15.6:6827/1364449401","heartbeat_back_addr":"172.21.15.6:6831/1364449401","heartbeat_front_addr":"172.21.15.6:6829/1364449401","state":["exists","up"]},{"osd":1,"uuid":"7d79b9c1-6b87-476a-bc36-7db5ee64cd90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6809","nonce":2494103078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6811","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6813","nonce":2494103078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6821","nonce":2494103078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6817","nonce":2494103078}]},"public_addr":"172.21.15.6:6809/2494103078","cluster_addr":"172.21.15.6:6813/2494103078","heartbeat_back_addr":"172.21.15.6:6821/2494103078","heartbeat_front_addr":"172.21.15.6:6817/2494103078","state":["exists","up"]},{"osd":2,"uuid":"492331b9-4ee9-40b6-bacd-85d44cd6c45a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6812","nonce":1500066681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6816","nonce":1500066681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6824","nonce":1500066681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6820","nonce":1500066681}]},"public_addr":"172.21.15.6:6812/1500066681","cluster_addr":"172.21.15.6:6816/1500066681","heartbeat_back_addr":"172.21.15.6:6824/1500066681","heartbeat_front_addr":"172.21.15.6:6820/1500066681","state":["exists","up"]},{"osd":3,"uuid":"21bb2efe-9ba7-4bdc-bc31-180cd1102e21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6801","nonce":1558806123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6803","nonce":1558806123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6807","nonce":1558806123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6805","nonce":1558806123}]},"public_addr":"172.21.15.6:6801/1558806123","cluster_addr":"172.21.15.6:6803/1558806123","heartbeat_back_addr":"172.21.15.6:6807/1558806123","heartbeat_front_addr":"172.21.15.6:6805/1558806123","state":["exists","up"]},{"osd":4,"uuid":"6e2c4b71-19d9-4737-83d9-558705d8644a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6801","nonce":1526135780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6803","nonce":1526135780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6807","nonce":1526135780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6805","nonce":1526135780}]},"public_addr":"172.21.15.146:6801/1526135780","cluster_addr":"172.21.15.146:6803/1526135780","heartbeat_back_addr":"172.21.15.146:6807/1526135780","heartbeat_front_addr":"172.21.15.146:6805/1526135780","state":["exists","up"]},{"osd":5,"uuid":"fb69c525-f1be-4b7d-ab32-6c29b0d6fddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6809","nonce":460262507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6811","nonce":460262507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6815","nonce":460262507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6813","nonce":460262507}]},"public_addr":"172.21.15.146:6809/460262507","cluster_addr":"172.21.15.146:6811/460262507","heartbeat_back_addr":"172.21.15.146:6815/460262507","heartbeat_front_addr":"172.21.15.146:6813/460262507","state":["exists","up"]},{"osd":6,"uuid":"c32e7fed-370d-4401-8ce0-6a4335b0d168","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6823","nonce":1392897493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6827","nonce":1392897493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6831","nonce":1392897493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6829","nonce":1392897493}]},"public_addr":"172.21.15.146:6823/1392897493","cluster_addr":"172.21.15.146:6827/1392897493","heartbeat_back_addr":"172.21.15.146:6831/1392897493","heartbeat_front_addr":"172.21.15.146:6829/1392897493","state":["exists","up"]},{"osd":7,"uuid":"75a79b44-fa5f-426d-a7d3-71f21f78835c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6817","nonce":3268323156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6819","nonce":3268323156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6825","nonce":3268323156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6822","nonce":3268323156}]},"public_addr":"172.21.15.146:6817/3268323156","cluster_addr":"172.21.15.146:6819/3268323156","heartbeat_back_addr":"172.21.15.146:6825/3268323156","heartbeat_front_addr":"172.21.15.146:6822/3268323156","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.310437+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.295850+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.294674+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.526855+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.300023+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.595160+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.346046+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.558741+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T09:28:33.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.770+0000 7f40d0200640 1 -- 172.21.15.6:0/1331056587 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f40a806b4d0 msgr2=0x7f40a806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:33.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.770+0000 7f40d0200640 1 --2- 172.21.15.6:0/1331056587 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f40a806b4d0 0x7f40a806d990 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f40bc0070b0 tx=0x7f40bc007040 comp rx=0 tx=0).stop 2024-03-04T09:28:33.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.770+0000 7f40d0200640 1 -- 172.21.15.6:0/1331056587 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f40c806e240 msgr2=0x7f40c806d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:33.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.770+0000 7f40d0200640 1 --2- 172.21.15.6:0/1331056587 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f40c806e240 0x7f40c806d440 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f40b4000c00 tx=0x7f40b40046c0 comp rx=0 tx=0).stop 2024-03-04T09:28:33.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.770+0000 7f40d0200640 1 -- 172.21.15.6:0/1331056587 shutdown_connections 2024-03-04T09:28:33.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.770+0000 7f40d0200640 1 --2- 172.21.15.6:0/1331056587 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f40a806b4d0 0x7f40a806d990 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:33.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.770+0000 7f40d0200640 1 --2- 172.21.15.6:0/1331056587 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f40c8161670 0x7f40c806d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:33.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.770+0000 7f40d0200640 1 --2- 172.21.15.6:0/1331056587 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f40c806e240 0x7f40c806d440 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:33.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.770+0000 7f40d0200640 1 --2- 172.21.15.6:0/1331056587 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f40c806cfe0 0x7f40c814d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:33.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.770+0000 7f40d0200640 1 -- 172.21.15.6:0/1331056587 >> 172.21.15.6:0/1331056587 conn(0x7f40c806afb0 msgr2=0x7f40c80b4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:33.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.770+0000 7f40d0200640 1 -- 172.21.15.6:0/1331056587 shutdown_connections 2024-03-04T09:28:33.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:33.771+0000 7f40d0200640 1 -- 172.21.15.6:0/1331056587 wait complete. 2024-03-04T09:28:33.785 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-04T09:28:33.785 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:28:34.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.003+0000 7f3c2f200640 1 Processor -- start 2024-03-04T09:28:34.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.003+0000 7f3c2f200640 1 -- start start 2024-03-04T09:28:34.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.005+0000 7f3c2f200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3c30145300 0x7f3c300b49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:34.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.005+0000 7f3c2f200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f3c300cd060 0x7f3c3015c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:34.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.005+0000 7f3c2f200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c300714d0 con 0x7f3c30070f10 2024-03-04T09:28:34.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.005+0000 7f3c2f200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c300717b0 con 0x7f3c300cd060 2024-03-04T09:28:34.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.005+0000 7f3c2f200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c30071a90 con 0x7f3c30145300 2024-03-04T09:28:34.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.005+0000 7f3c2e800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f3c300cd060 0x7f3c3015c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:34.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.005+0000 7f3c2d400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f3c30070f10 0x7f3c300cc950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:41196/0 (socket says 172.21.15.6:41196) 2024-03-04T09:28:34.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.005+0000 7f3c2d400640 1 -- 172.21.15.6:0/4072596816 learned_addr learned my addr 172.21.15.6:0/4072596816 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:28:34.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.005+0000 7f3c2de00640 1 --2- 172.21.15.6:0/4072596816 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3c30145300 0x7f3c300b49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:34.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.006+0000 7f3c2e800640 1 -- 172.21.15.6:0/4072596816 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3c30145300 msgr2=0x7f3c300b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:34.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.006+0000 7f3c2e800640 1 --2- 172.21.15.6:0/4072596816 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3c30145300 0x7f3c300b49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:34.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.006+0000 7f3c2e800640 1 -- 172.21.15.6:0/4072596816 >> v1:172.21.15.6:6789/0 conn(0x7f3c30070f10 legacy=0x7f3c300cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:34.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.006+0000 7f3c2e800640 1 -- 172.21.15.6:0/4072596816 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c30071d70 con 0x7f3c300cd060 2024-03-04T09:28:34.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.006+0000 7f3c2ca00640 1 -- 172.21.15.6:0/4072596816 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3522499322 0 0) 0x7f3c300714d0 con 0x7f3c30070f10 2024-03-04T09:28:34.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.007+0000 7f3c2e800640 1 --2- 172.21.15.6:0/4072596816 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3c300cd060 0x7f3c3015c570 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f3c28009d90 tx=0x7f3c2802bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54f80aa97d292f40 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:34.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.007+0000 7f3c2ca00640 1 -- 172.21.15.6:0/4072596816 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c28039070 con 0x7f3c300cd060 2024-03-04T09:28:34.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.007+0000 7f3c2ca00640 1 -- 172.21.15.6:0/4072596816 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c2802cc50 con 0x7f3c300cd060 2024-03-04T09:28:34.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.007+0000 7f3c2ca00640 1 -- 172.21.15.6:0/4072596816 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c28034440 con 0x7f3c300cd060 2024-03-04T09:28:34.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.008+0000 7f3c2f200640 1 -- 172.21.15.6:0/4072596816 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3c300cd060 msgr2=0x7f3c3015c570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:34.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.008+0000 7f3c2f200640 1 --2- 172.21.15.6:0/4072596816 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3c300cd060 0x7f3c3015c570 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f3c28009d90 tx=0x7f3c2802bea0 comp rx=0 tx=0).stop 2024-03-04T09:28:34.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.008+0000 7f3c2f200640 1 -- 172.21.15.6:0/4072596816 shutdown_connections 2024-03-04T09:28:34.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.008+0000 7f3c2f200640 1 --2- 172.21.15.6:0/4072596816 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3c300cd060 0x7f3c3015c570 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:34.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.008+0000 7f3c2f200640 1 --2- 172.21.15.6:0/4072596816 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3c30145300 0x7f3c300b49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:34.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.008+0000 7f3c2f200640 1 -- 172.21.15.6:0/4072596816 >> 172.21.15.6:0/4072596816 conn(0x7f3c3006b3d0 msgr2=0x7f3c3006b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:34.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.008+0000 7f3c2f200640 1 -- 172.21.15.6:0/4072596816 shutdown_connections 2024-03-04T09:28:34.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.008+0000 7f3c2f200640 1 -- 172.21.15.6:0/4072596816 wait complete. 2024-03-04T09:28:34.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.010+0000 7f3c2f200640 1 Processor -- start 2024-03-04T09:28:34.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.010+0000 7f3c2f200640 1 -- start start 2024-03-04T09:28:34.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.011+0000 7f3c2f200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3c30070f10 0x7f3c301e9100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:34.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.011+0000 7f3c2f200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3c300cd060 0x7f3c301e4420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:34.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.011+0000 7f3c2f200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3c30145300 0x7f3c3015ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:34.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.011+0000 7f3c2f200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c3015cab0 con 0x7f3c300cd060 2024-03-04T09:28:34.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.011+0000 7f3c2f200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c301456e0 con 0x7f3c30070f10 2024-03-04T09:28:34.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.011+0000 7f3c2f200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c3015cc30 con 0x7f3c30145300 2024-03-04T09:28:34.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.011+0000 7f3c2d400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3c300cd060 0x7f3c301e4420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:34.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.011+0000 7f3c2d400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3c300cd060 0x7f3c301e4420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53804/0 (socket says 172.21.15.6:53804) 2024-03-04T09:28:34.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.011+0000 7f3c2d400640 1 -- 172.21.15.6:0/2174301236 learned_addr learned my addr 172.21.15.6:0/2174301236 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:28:34.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.012+0000 7f3c2e800640 1 --2- 172.21.15.6:0/2174301236 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3c30145300 0x7f3c3015ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:34.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.012+0000 7f3c2d400640 1 -- 172.21.15.6:0/2174301236 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3c30145300 msgr2=0x7f3c3015ae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:34.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.012+0000 7f3c2d400640 1 --2- 172.21.15.6:0/2174301236 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3c30145300 0x7f3c3015ae90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:34.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.012+0000 7f3c2d400640 1 -- 172.21.15.6:0/2174301236 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3c30070f10 msgr2=0x7f3c301e9100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:34.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.012+0000 7f3c2d400640 1 --2- 172.21.15.6:0/2174301236 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3c30070f10 0x7f3c301e9100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:34.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.012+0000 7f3c2d400640 1 -- 172.21.15.6:0/2174301236 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c301e4e70 con 0x7f3c300cd060 2024-03-04T09:28:34.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.012+0000 7f3c2e800640 1 --2- 172.21.15.6:0/2174301236 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3c30145300 0x7f3c3015ae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:28:34.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.012+0000 7f3c2d400640 1 --2- 172.21.15.6:0/2174301236 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3c300cd060 0x7f3c301e4420 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f3c1c00c500 tx=0x7f3c1c00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:34.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.012+0000 7f3c26000640 1 -- 172.21.15.6:0/2174301236 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c1c018070 con 0x7f3c300cd060 2024-03-04T09:28:34.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.012+0000 7f3c26000640 1 -- 172.21.15.6:0/2174301236 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c1c007580 con 0x7f3c300cd060 2024-03-04T09:28:34.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.012+0000 7f3c2f200640 1 -- 172.21.15.6:0/2174301236 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c301e5150 con 0x7f3c300cd060 2024-03-04T09:28:34.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.012+0000 7f3c26000640 1 -- 172.21.15.6:0/2174301236 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c1c004a20 con 0x7f3c300cd060 2024-03-04T09:28:34.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.013+0000 7f3c2f200640 1 -- 172.21.15.6:0/2174301236 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c301e5710 con 0x7f3c300cd060 2024-03-04T09:28:34.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.014+0000 7f3c2f200640 1 -- 172.21.15.6:0/2174301236 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c3004f330 con 0x7f3c300cd060 2024-03-04T09:28:34.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.015+0000 7f3c26000640 1 -- 172.21.15.6:0/2174301236 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3c1c01b070 con 0x7f3c300cd060 2024-03-04T09:28:34.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.016+0000 7f3c26000640 1 --2- 172.21.15.6:0/2174301236 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f3c1006b4d0 0x7f3c1006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:34.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.016+0000 7f3c26000640 1 -- 172.21.15.6:0/2174301236 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3c1c089540 con 0x7f3c300cd060 2024-03-04T09:28:34.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.017+0000 7f3c2de00640 1 --2- 172.21.15.6:0/2174301236 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f3c1006b4d0 0x7f3c1006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:34.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.018+0000 7f3c2de00640 1 --2- 172.21.15.6:0/2174301236 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f3c1006b4d0 0x7f3c1006d990 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f3c20006fd0 tx=0x7f3c20008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:34.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.020+0000 7f3c26000640 1 -- 172.21.15.6:0/2174301236 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3c1c0583b0 con 0x7f3c300cd060 2024-03-04T09:28:34.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.255+0000 7f3c2f200640 1 -- 172.21.15.6:0/2174301236 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3c300b2b90 con 0x7f3c300cd060 2024-03-04T09:28:34.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.256+0000 7f3c26000640 1 -- 172.21.15.6:0/2174301236 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+4971 (secure 0 0 0) 0x7f3c1c057cb0 con 0x7f3c300cd060 2024-03-04T09:28:34.253 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:28:34.253 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":34,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5030,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/1411942502","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1411942502},{"type":"v1","addr":"172.21.15.6:6834","nonce":1411942502}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28},{"gid":5039,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/1628166413","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":1628166413},{"type":"v1","addr":"172.21.15.146:6837","nonce":1628166413}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":5323,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/131491768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":131491768},{"type":"v1","addr":"172.21.15.6:6835","nonce":131491768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:27:59.854470+0000","modified":"2024-03-04T09:28:02.945053+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5326},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5326":{"gid":5326,"name":"b","rank":0,"incarnation":30,"state":"up:active","state_seq":3,"addr":"172.21.15.146:6836/527855083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":527855083},{"type":"v1","addr":"172.21.15.146:6836","nonce":527855083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5326,"qdb_cluster":[5326]},"id":4}]} 2024-03-04T09:28:34.254 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 34 2024-03-04T09:28:34.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.259+0000 7f3c2f200640 1 -- 172.21.15.6:0/2174301236 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f3c1006b4d0 msgr2=0x7f3c1006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:34.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.259+0000 7f3c2f200640 1 --2- 172.21.15.6:0/2174301236 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f3c1006b4d0 0x7f3c1006d990 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f3c20006fd0 tx=0x7f3c20008040 comp rx=0 tx=0).stop 2024-03-04T09:28:34.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.259+0000 7f3c2f200640 1 -- 172.21.15.6:0/2174301236 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3c300cd060 msgr2=0x7f3c301e4420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:34.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.259+0000 7f3c2f200640 1 --2- 172.21.15.6:0/2174301236 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3c300cd060 0x7f3c301e4420 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f3c1c00c500 tx=0x7f3c1c00c9c0 comp rx=0 tx=0).stop 2024-03-04T09:28:34.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.259+0000 7f3c2f200640 1 -- 172.21.15.6:0/2174301236 shutdown_connections 2024-03-04T09:28:34.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.259+0000 7f3c2f200640 1 --2- 172.21.15.6:0/2174301236 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f3c1006b4d0 0x7f3c1006d990 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:34.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.259+0000 7f3c2f200640 1 --2- 172.21.15.6:0/2174301236 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3c30145300 0x7f3c3015ae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:34.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.259+0000 7f3c2f200640 1 --2- 172.21.15.6:0/2174301236 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3c300cd060 0x7f3c301e4420 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:34.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.259+0000 7f3c2f200640 1 --2- 172.21.15.6:0/2174301236 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3c30070f10 0x7f3c301e9100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:34.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.259+0000 7f3c2f200640 1 -- 172.21.15.6:0/2174301236 >> 172.21.15.6:0/2174301236 conn(0x7f3c3006b3d0 msgr2=0x7f3c300ad880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:34.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.259+0000 7f3c2f200640 1 -- 172.21.15.6:0/2174301236 shutdown_connections 2024-03-04T09:28:34.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.259+0000 7f3c2f200640 1 -- 172.21.15.6:0/2174301236 wait complete. 2024-03-04T09:28:34.275 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:28:34.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.500+0000 7fe903200640 1 Processor -- start 2024-03-04T09:28:34.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.500+0000 7fe903200640 1 -- start start 2024-03-04T09:28:34.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.502+0000 7fe903200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe904070f10 0x7fe9040cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:34.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.502+0000 7fe903200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fe9040cce90 0x7fe90415c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:34.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.502+0000 7fe903200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9040714d0 con 0x7fe904145300 2024-03-04T09:28:34.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.502+0000 7fe903200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9040cd300 con 0x7fe9040cce90 2024-03-04T09:28:34.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.502+0000 7fe903200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9040717b0 con 0x7fe904070f10 2024-03-04T09:28:34.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.502+0000 7fe902800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fe9040cce90 0x7fe90415c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:34.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.502+0000 7fe902800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fe9040cce90 0x7fe90415c570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53810/0 (socket says 172.21.15.6:53810) 2024-03-04T09:28:34.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.502+0000 7fe902800640 1 -- 172.21.15.6:0/1457155867 learned_addr learned my addr 172.21.15.6:0/1457155867 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:28:34.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.502+0000 7fe8fbe00640 1 --2- 172.21.15.6:0/1457155867 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe904070f10 0x7fe9040cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:34.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.503+0000 7fe902800640 1 -- 172.21.15.6:0/1457155867 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe904070f10 msgr2=0x7fe9040cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:34.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.503+0000 7fe902800640 1 --2- 172.21.15.6:0/1457155867 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe904070f10 0x7fe9040cc950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:34.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.503+0000 7fe902800640 1 -- 172.21.15.6:0/1457155867 >> v1:172.21.15.6:6789/0 conn(0x7fe904145300 legacy=0x7fe9040b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:34.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.503+0000 7fe902800640 1 -- 172.21.15.6:0/1457155867 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe904071a90 con 0x7fe9040cce90 2024-03-04T09:28:34.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.504+0000 7fe902800640 1 --2- 172.21.15.6:0/1457155867 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe9040cce90 0x7fe90415c570 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fe8fc009e50 tx=0x7fe8fc02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49e15e28ae554396 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:34.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.504+0000 7fe901400640 1 -- 172.21.15.6:0/1457155867 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8fc02cb00 con 0x7fe9040cce90 2024-03-04T09:28:34.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.504+0000 7fe901400640 1 -- 172.21.15.6:0/1457155867 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8fc02cca0 con 0x7fe9040cce90 2024-03-04T09:28:34.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.504+0000 7fe901400640 1 -- 172.21.15.6:0/1457155867 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8fc034610 con 0x7fe9040cce90 2024-03-04T09:28:34.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.505+0000 7fe903200640 1 -- 172.21.15.6:0/1457155867 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe9040cce90 msgr2=0x7fe90415c570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:34.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.505+0000 7fe903200640 1 --2- 172.21.15.6:0/1457155867 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe9040cce90 0x7fe90415c570 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fe8fc009e50 tx=0x7fe8fc02bf60 comp rx=0 tx=0).stop 2024-03-04T09:28:34.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.505+0000 7fe903200640 1 -- 172.21.15.6:0/1457155867 shutdown_connections 2024-03-04T09:28:34.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.505+0000 7fe903200640 1 --2- 172.21.15.6:0/1457155867 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe9040cce90 0x7fe90415c570 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:34.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.505+0000 7fe903200640 1 --2- 172.21.15.6:0/1457155867 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe904070f10 0x7fe9040cc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:34.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.505+0000 7fe903200640 1 -- 172.21.15.6:0/1457155867 >> 172.21.15.6:0/1457155867 conn(0x7fe90406b3d0 msgr2=0x7fe90406b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:34.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.505+0000 7fe903200640 1 -- 172.21.15.6:0/1457155867 shutdown_connections 2024-03-04T09:28:34.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.505+0000 7fe903200640 1 -- 172.21.15.6:0/1457155867 wait complete. 2024-03-04T09:28:34.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.507+0000 7fe903200640 1 Processor -- start 2024-03-04T09:28:34.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.507+0000 7fe903200640 1 -- start start 2024-03-04T09:28:34.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.508+0000 7fe903200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe904070f10 0x7fe90406e480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:34.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.508+0000 7fe903200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe9040cce90 0x7fe90413b050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:34.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.508+0000 7fe903200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe904145300 0x7fe904153f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:34.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.508+0000 7fe903200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe90415cab0 con 0x7fe904070f10 2024-03-04T09:28:34.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.508+0000 7fe903200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9041456e0 con 0x7fe904145300 2024-03-04T09:28:34.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.508+0000 7fe903200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe90415cc30 con 0x7fe9040cce90 2024-03-04T09:28:34.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.509+0000 7fe901e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe904070f10 0x7fe90406e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:34.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.509+0000 7fe901e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe904070f10 0x7fe90406e480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53822/0 (socket says 172.21.15.6:53822) 2024-03-04T09:28:34.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.509+0000 7fe901e00640 1 -- 172.21.15.6:0/3901687374 learned_addr learned my addr 172.21.15.6:0/3901687374 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:28:34.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.509+0000 7fe8fbe00640 1 --2- 172.21.15.6:0/3901687374 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe9040cce90 0x7fe90413b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:34.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.509+0000 7fe902800640 1 --2- 172.21.15.6:0/3901687374 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe904145300 0x7fe904153f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:34.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.509+0000 7fe901e00640 1 -- 172.21.15.6:0/3901687374 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe9040cce90 msgr2=0x7fe90413b050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:34.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.509+0000 7fe901e00640 1 --2- 172.21.15.6:0/3901687374 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe9040cce90 0x7fe90413b050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:34.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.509+0000 7fe901e00640 1 -- 172.21.15.6:0/3901687374 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe904145300 msgr2=0x7fe904153f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:34.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.509+0000 7fe901e00640 1 --2- 172.21.15.6:0/3901687374 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe904145300 0x7fe904153f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:34.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.509+0000 7fe901e00640 1 -- 172.21.15.6:0/3901687374 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9041402f0 con 0x7fe904070f10 2024-03-04T09:28:34.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.510+0000 7fe901e00640 1 --2- 172.21.15.6:0/3901687374 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe904070f10 0x7fe90406e480 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fe8f400c3c0 tx=0x7fe8f400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:34.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.510+0000 7fe8fa000640 1 -- 172.21.15.6:0/3901687374 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8f40052d0 con 0x7fe904070f10 2024-03-04T09:28:34.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.510+0000 7fe8fa000640 1 -- 172.21.15.6:0/3901687374 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8f4005490 con 0x7fe904070f10 2024-03-04T09:28:34.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.510+0000 7fe8fa000640 1 -- 172.21.15.6:0/3901687374 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8f4007260 con 0x7fe904070f10 2024-03-04T09:28:34.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.510+0000 7fe903200640 1 -- 172.21.15.6:0/3901687374 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe90413b710 con 0x7fe904070f10 2024-03-04T09:28:34.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.510+0000 7fe903200640 1 -- 172.21.15.6:0/3901687374 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe90413bcd0 con 0x7fe904070f10 2024-03-04T09:28:34.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.511+0000 7fe903200640 1 -- 172.21.15.6:0/3901687374 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe90404f330 con 0x7fe904070f10 2024-03-04T09:28:34.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.512+0000 7fe8fa000640 1 -- 172.21.15.6:0/3901687374 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe8f4007400 con 0x7fe904070f10 2024-03-04T09:28:34.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.513+0000 7fe8fa000640 1 --2- 172.21.15.6:0/3901687374 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fe8e006b4d0 0x7fe8e006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:34.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.513+0000 7fe8fa000640 1 -- 172.21.15.6:0/3901687374 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe8f4089780 con 0x7fe904070f10 2024-03-04T09:28:34.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.514+0000 7fe8fbe00640 1 --2- 172.21.15.6:0/3901687374 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fe8e006b4d0 0x7fe8e006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:34.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.515+0000 7fe8fbe00640 1 --2- 172.21.15.6:0/3901687374 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fe8e006b4d0 0x7fe8e006d990 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fe9041f7130 tx=0x7fe8ec008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:34.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.517+0000 7fe8fa000640 1 -- 172.21.15.6:0/3901687374 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe8f4058560 con 0x7fe904070f10 2024-03-04T09:28:34.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.759+0000 7fe903200640 1 -- 172.21.15.6:0/3901687374 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe904144820 con 0x7fe904070f10 2024-03-04T09:28:34.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.760+0000 7fe8fa000640 1 -- 172.21.15.6:0/3901687374 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+4971 (secure 0 0 0) 0x7fe8f4057e60 con 0x7fe904070f10 2024-03-04T09:28:34.758 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:28:34.758 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":34,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5030,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/1411942502","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1411942502},{"type":"v1","addr":"172.21.15.6:6834","nonce":1411942502}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28},{"gid":5039,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/1628166413","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":1628166413},{"type":"v1","addr":"172.21.15.146:6837","nonce":1628166413}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":5323,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/131491768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":131491768},{"type":"v1","addr":"172.21.15.6:6835","nonce":131491768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:27:59.854470+0000","modified":"2024-03-04T09:28:02.945053+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5326},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5326":{"gid":5326,"name":"b","rank":0,"incarnation":30,"state":"up:active","state_seq":3,"addr":"172.21.15.146:6836/527855083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":527855083},{"type":"v1","addr":"172.21.15.146:6836","nonce":527855083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5326,"qdb_cluster":[5326]},"id":4}]} 2024-03-04T09:28:34.758 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 34 2024-03-04T09:28:34.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.762+0000 7fe903200640 1 -- 172.21.15.6:0/3901687374 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fe8e006b4d0 msgr2=0x7fe8e006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:34.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.762+0000 7fe903200640 1 --2- 172.21.15.6:0/3901687374 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fe8e006b4d0 0x7fe8e006d990 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fe9041f7130 tx=0x7fe8ec008040 comp rx=0 tx=0).stop 2024-03-04T09:28:34.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.762+0000 7fe903200640 1 -- 172.21.15.6:0/3901687374 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe904070f10 msgr2=0x7fe90406e480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:34.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.762+0000 7fe903200640 1 --2- 172.21.15.6:0/3901687374 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe904070f10 0x7fe90406e480 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fe8f400c3c0 tx=0x7fe8f400c880 comp rx=0 tx=0).stop 2024-03-04T09:28:34.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.763+0000 7fe903200640 1 -- 172.21.15.6:0/3901687374 shutdown_connections 2024-03-04T09:28:34.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.763+0000 7fe903200640 1 --2- 172.21.15.6:0/3901687374 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe904145300 0x7fe904153f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:34.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.763+0000 7fe903200640 1 --2- 172.21.15.6:0/3901687374 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fe8e006b4d0 0x7fe8e006d990 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:34.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.763+0000 7fe903200640 1 --2- 172.21.15.6:0/3901687374 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe9040cce90 0x7fe90413b050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:34.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.763+0000 7fe903200640 1 --2- 172.21.15.6:0/3901687374 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe904070f10 0x7fe90406e480 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:34.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.763+0000 7fe903200640 1 -- 172.21.15.6:0/3901687374 >> 172.21.15.6:0/3901687374 conn(0x7fe90406b3d0 msgr2=0x7fe90415dc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:34.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.763+0000 7fe903200640 1 -- 172.21.15.6:0/3901687374 shutdown_connections 2024-03-04T09:28:34.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.763+0000 7fe903200640 1 -- 172.21.15.6:0/3901687374 wait complete. 2024-03-04T09:28:34.779 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T09:28:34.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.987+0000 7fd8d1c00640 1 Processor -- start 2024-03-04T09:28:34.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.988+0000 7fd8d1c00640 1 -- start start 2024-03-04T09:28:34.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.989+0000 7fd8d1c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd8cc0af1a0 0x7fd8cc15c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:34.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.990+0000 7fd8d1c00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd8cc0cdc20 0x7fd8cc0b2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:34.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.990+0000 7fd8d1c00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8cc15cc40 con 0x7fd8cc0af580 2024-03-04T09:28:34.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.990+0000 7fd8d1c00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8cc15cf20 con 0x7fd8cc0cdc20 2024-03-04T09:28:34.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.990+0000 7fd8d1c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8cc15d200 con 0x7fd8cc0af1a0 2024-03-04T09:28:34.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.990+0000 7fd8ca000640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fd8cc0af580 0x7fd8cc0b4c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:41206/0 (socket says 172.21.15.6:41206) 2024-03-04T09:28:34.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.990+0000 7fd8ca000640 1 -- 172.21.15.6:0/3481231890 learned_addr learned my addr 172.21.15.6:0/3481231890 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:28:34.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.990+0000 7fd8caa00640 1 --2- 172.21.15.6:0/3481231890 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd8cc0af1a0 0x7fd8cc15c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:34.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.990+0000 7fd8cb400640 1 --2- 172.21.15.6:0/3481231890 >> v2:172.21.15.146:3300/0 conn(0x7fd8cc0cdc20 0x7fd8cc0b2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:34.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.990+0000 7fd8c9600640 1 -- 172.21.15.6:0/3481231890 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1083214970 0 0) 0x7fd8cc15cc40 con 0x7fd8cc0af580 2024-03-04T09:28:34.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.990+0000 7fd8c9600640 1 -- 172.21.15.6:0/3481231890 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8b0003520 con 0x7fd8cc0af580 2024-03-04T09:28:34.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.991+0000 7fd8c9600640 1 -- 172.21.15.6:0/3481231890 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 871763139 0 0) 0x7fd8b0003520 con 0x7fd8cc0af580 2024-03-04T09:28:34.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.991+0000 7fd8c9600640 1 -- 172.21.15.6:0/3481231890 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd8cc0af1a0 msgr2=0x7fd8cc15c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:34.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.991+0000 7fd8c9600640 1 --2- 172.21.15.6:0/3481231890 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd8cc0af1a0 0x7fd8cc15c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:34.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.991+0000 7fd8c9600640 1 -- 172.21.15.6:0/3481231890 >> v2:172.21.15.146:3300/0 conn(0x7fd8cc0cdc20 msgr2=0x7fd8cc0b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:34.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.991+0000 7fd8c9600640 1 --2- 172.21.15.6:0/3481231890 >> v2:172.21.15.146:3300/0 conn(0x7fd8cc0cdc20 0x7fd8cc0b2dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:34.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.991+0000 7fd8c9600640 1 -- 172.21.15.6:0/3481231890 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8cc15d4e0 con 0x7fd8cc0af580 2024-03-04T09:28:34.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.991+0000 7fd8c9600640 1 -- 172.21.15.6:0/3481231890 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7fd8b8001670 con 0x7fd8cc0af580 2024-03-04T09:28:34.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.991+0000 7fd8c9600640 1 -- 172.21.15.6:0/3481231890 >> v1:172.21.15.6:6789/0 conn(0x7fd8cc0af580 legacy=0x7fd8cc0b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:34.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.991+0000 7fd8c9600640 1 --2- 172.21.15.6:0/3481231890 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd8b0003a20 0x7fd8b0003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:34.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.991+0000 7fd8c9600640 1 --2- 172.21.15.6:0/3481231890 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd8cc0af1a0 0x7fd8b0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:34.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.991+0000 7fd8c9600640 1 --2- 172.21.15.6:0/3481231890 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd8b0004d40 0x7fd8b00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:34.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.991+0000 7fd8c9600640 1 -- 172.21.15.6:0/3481231890 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8cc15cf20 con 0x7fd8b0004d40 2024-03-04T09:28:34.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.991+0000 7fd8c9600640 1 -- 172.21.15.6:0/3481231890 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8cc15d200 con 0x7fd8b0003a20 2024-03-04T09:28:34.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.991+0000 7fd8c9600640 1 -- 172.21.15.6:0/3481231890 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b000ba10 con 0x7fd8cc0af1a0 2024-03-04T09:28:34.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.991+0000 7fd8cb400640 1 --2- 172.21.15.6:0/3481231890 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd8b0004d40 0x7fd8b00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:34.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.992+0000 7fd8ca000640 1 --2- 172.21.15.6:0/3481231890 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd8cc0af1a0 0x7fd8b0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:34.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.992+0000 7fd8caa00640 1 --2- 172.21.15.6:0/3481231890 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd8b0003a20 0x7fd8b0003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:34.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.992+0000 7fd8cb400640 1 -- 172.21.15.6:0/3481231890 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd8cc0af1a0 msgr2=0x7fd8b0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:34.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.992+0000 7fd8cb400640 1 --2- 172.21.15.6:0/3481231890 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd8cc0af1a0 0x7fd8b0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:34.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.992+0000 7fd8cb400640 1 -- 172.21.15.6:0/3481231890 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd8b0003a20 msgr2=0x7fd8b0003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:34.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.992+0000 7fd8cb400640 1 --2- 172.21.15.6:0/3481231890 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd8b0003a20 0x7fd8b0003e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:34.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.992+0000 7fd8cb400640 1 -- 172.21.15.6:0/3481231890 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8b0003520 con 0x7fd8b0004d40 2024-03-04T09:28:34.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.992+0000 7fd8caa00640 1 --2- 172.21.15.6:0/3481231890 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd8b0003a20 0x7fd8b0003e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:28:34.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.993+0000 7fd8cb400640 1 --2- 172.21.15.6:0/3481231890 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd8b0004d40 0x7fd8b00051c0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7fd8c4009fc0 tx=0x7fd8b0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ec1587dd42d7637 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:34.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.993+0000 7fd8c9600640 1 -- 172.21.15.6:0/3481231890 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8c4039070 con 0x7fd8b0004d40 2024-03-04T09:28:34.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.993+0000 7fd8c9600640 1 -- 172.21.15.6:0/3481231890 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8c402cc60 con 0x7fd8b0004d40 2024-03-04T09:28:34.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.993+0000 7fd8c9600640 1 -- 172.21.15.6:0/3481231890 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8c4009c10 con 0x7fd8b0004d40 2024-03-04T09:28:34.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.994+0000 7fd8d1c00640 1 -- 172.21.15.6:0/3481231890 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd8b0004d40 msgr2=0x7fd8b00051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:34.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.994+0000 7fd8d1c00640 1 --2- 172.21.15.6:0/3481231890 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd8b0004d40 0x7fd8b00051c0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7fd8c4009fc0 tx=0x7fd8b0005830 comp rx=0 tx=0).stop 2024-03-04T09:28:34.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.994+0000 7fd8cb400640 1 -- 172.21.15.6:0/3481231890 reap_dead start 2024-03-04T09:28:34.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.994+0000 7fd8d1c00640 1 -- 172.21.15.6:0/3481231890 shutdown_connections 2024-03-04T09:28:34.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.994+0000 7fd8d1c00640 1 -- 172.21.15.6:0/3481231890 >> 172.21.15.6:0/3481231890 conn(0x7fd8cc06afb0 msgr2=0x7fd8cc06b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:34.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.994+0000 7fd8d1c00640 1 -- 172.21.15.6:0/3481231890 shutdown_connections 2024-03-04T09:28:34.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.994+0000 7fd8d1c00640 1 -- 172.21.15.6:0/3481231890 wait complete. 2024-03-04T09:28:34.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.996+0000 7fd8d1c00640 1 Processor -- start 2024-03-04T09:28:34.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.996+0000 7fd8d1c00640 1 -- start start 2024-03-04T09:28:34.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.997+0000 7fd8d1c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd8b0004d40 0x7fd8cc15f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:34.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.997+0000 7fd8d1c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd8cc1f3cb0 0x7fd8cc14d640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:34.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.997+0000 7fd8d1c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd8cc0cc790 0x7fd8cc0ccc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:34.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.997+0000 7fd8d1c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8cc15c990 con 0x7fd8cc1f3cb0 2024-03-04T09:28:34.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.997+0000 7fd8d1c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8cc15c810 con 0x7fd8cc0cc790 2024-03-04T09:28:34.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.997+0000 7fd8d1c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8cc0b3310 con 0x7fd8b0004d40 2024-03-04T09:28:34.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.997+0000 7fd8ca000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd8cc1f3cb0 0x7fd8cc14d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:34.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.997+0000 7fd8ca000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd8cc1f3cb0 0x7fd8cc14d640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53840/0 (socket says 172.21.15.6:53840) 2024-03-04T09:28:34.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.997+0000 7fd8ca000640 1 -- 172.21.15.6:0/1938998781 learned_addr learned my addr 172.21.15.6:0/1938998781 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:28:34.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.998+0000 7fd8caa00640 1 --2- 172.21.15.6:0/1938998781 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd8b0004d40 0x7fd8cc15f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:34.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.998+0000 7fd8cb400640 1 --2- 172.21.15.6:0/1938998781 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd8cc0cc790 0x7fd8cc0ccc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:34.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.998+0000 7fd8ca000640 1 -- 172.21.15.6:0/1938998781 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd8b0004d40 msgr2=0x7fd8cc15f2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:34.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.998+0000 7fd8ca000640 1 --2- 172.21.15.6:0/1938998781 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd8b0004d40 0x7fd8cc15f2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:34.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.998+0000 7fd8ca000640 1 -- 172.21.15.6:0/1938998781 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd8cc0cc790 msgr2=0x7fd8cc0ccc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:34.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.998+0000 7fd8ca000640 1 --2- 172.21.15.6:0/1938998781 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd8cc0cc790 0x7fd8cc0ccc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:34.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.998+0000 7fd8ca000640 1 -- 172.21.15.6:0/1938998781 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8cc1f4c60 con 0x7fd8cc1f3cb0 2024-03-04T09:28:34.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.998+0000 7fd8cb400640 1 --2- 172.21.15.6:0/1938998781 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd8cc0cc790 0x7fd8cc0ccc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:28:34.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.998+0000 7fd8ca000640 1 --2- 172.21.15.6:0/1938998781 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd8cc1f3cb0 0x7fd8cc14d640 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fd8b800e7f0 tx=0x7fd8b800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:34.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.998+0000 7fd8c2a00640 1 -- 172.21.15.6:0/1938998781 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8b800c870 con 0x7fd8cc1f3cb0 2024-03-04T09:28:34.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.998+0000 7fd8c2a00640 1 -- 172.21.15.6:0/1938998781 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8b8018070 con 0x7fd8cc1f3cb0 2024-03-04T09:28:34.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.998+0000 7fd8c2a00640 1 -- 172.21.15.6:0/1938998781 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8b800cc70 con 0x7fd8cc1f3cb0 2024-03-04T09:28:34.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.998+0000 7fd8d1c00640 1 -- 172.21.15.6:0/1938998781 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8cc09ab40 con 0x7fd8cc1f3cb0 2024-03-04T09:28:34.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:34.999+0000 7fd8d1c00640 1 -- 172.21.15.6:0/1938998781 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8cc1f43c0 con 0x7fd8cc1f3cb0 2024-03-04T09:28:34.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.000+0000 7fd8d1c00640 1 -- 172.21.15.6:0/1938998781 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8cc069510 con 0x7fd8cc1f3cb0 2024-03-04T09:28:34.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.001+0000 7fd8c2a00640 1 -- 172.21.15.6:0/1938998781 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd8b8013420 con 0x7fd8cc1f3cb0 2024-03-04T09:28:34.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.001+0000 7fd8c2a00640 1 --2- 172.21.15.6:0/1938998781 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd8ac06b500 0x7fd8ac06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:34.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.002+0000 7fd8c2a00640 1 -- 172.21.15.6:0/1938998781 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd8b8089e50 con 0x7fd8cc1f3cb0 2024-03-04T09:28:34.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.002+0000 7fd8caa00640 1 --2- 172.21.15.6:0/1938998781 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd8ac06b500 0x7fd8ac06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:35.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.003+0000 7fd8caa00640 1 --2- 172.21.15.6:0/1938998781 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd8ac06b500 0x7fd8ac06d9c0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fd8bc009aa0 tx=0x7fd8bc008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:35.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.006+0000 7fd8c2a00640 1 -- 172.21.15.6:0/1938998781 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd8b8058cc0 con 0x7fd8cc1f3cb0 2024-03-04T09:28:35.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.200+0000 7fd8d1c00640 1 -- 172.21.15.6:0/1938998781 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd8cc04f3a0 con 0x7fd8cc1f3cb0 2024-03-04T09:28:35.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.203+0000 7fd8c2a00640 1 -- 172.21.15.6:0/1938998781 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v56) v1 ==== 74+0+16617 (secure 0 0 0) 0x7fd8b80585c0 con 0x7fd8cc1f3cb0 2024-03-04T09:28:35.200 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:28:35.200 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":56,"fsid":"edce0578-5ae4-4402-b45a-552bdef84d7c","created":"2024-03-04T09:24:54.523780+0000","modified":"2024-03-04T09:28:01.413693+0000","last_up_change":"2024-03-04T09:25:07.089049+0000","last_in_change":"2024-03-04T09:24:57.982077+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T09:25:07.935198+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-04T09:27:57.774529+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-04T09:27:58.886504+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a96574f-81f1-4bc6-8463-757520b84a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6825","nonce":1364449401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6827","nonce":1364449401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6831","nonce":1364449401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6829","nonce":1364449401}]},"public_addr":"172.21.15.6:6825/1364449401","cluster_addr":"172.21.15.6:6827/1364449401","heartbeat_back_addr":"172.21.15.6:6831/1364449401","heartbeat_front_addr":"172.21.15.6:6829/1364449401","state":["exists","up"]},{"osd":1,"uuid":"7d79b9c1-6b87-476a-bc36-7db5ee64cd90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6809","nonce":2494103078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6811","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6813","nonce":2494103078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6821","nonce":2494103078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6817","nonce":2494103078}]},"public_addr":"172.21.15.6:6809/2494103078","cluster_addr":"172.21.15.6:6813/2494103078","heartbeat_back_addr":"172.21.15.6:6821/2494103078","heartbeat_front_addr":"172.21.15.6:6817/2494103078","state":["exists","up"]},{"osd":2,"uuid":"492331b9-4ee9-40b6-bacd-85d44cd6c45a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6812","nonce":1500066681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6816","nonce":1500066681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6824","nonce":1500066681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6820","nonce":1500066681}]},"public_addr":"172.21.15.6:6812/1500066681","cluster_addr":"172.21.15.6:6816/1500066681","heartbeat_back_addr":"172.21.15.6:6824/1500066681","heartbeat_front_addr":"172.21.15.6:6820/1500066681","state":["exists","up"]},{"osd":3,"uuid":"21bb2efe-9ba7-4bdc-bc31-180cd1102e21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6801","nonce":1558806123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6803","nonce":1558806123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6807","nonce":1558806123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6805","nonce":1558806123}]},"public_addr":"172.21.15.6:6801/1558806123","cluster_addr":"172.21.15.6:6803/1558806123","heartbeat_back_addr":"172.21.15.6:6807/1558806123","heartbeat_front_addr":"172.21.15.6:6805/1558806123","state":["exists","up"]},{"osd":4,"uuid":"6e2c4b71-19d9-4737-83d9-558705d8644a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6801","nonce":1526135780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6803","nonce":1526135780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6807","nonce":1526135780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6805","nonce":1526135780}]},"public_addr":"172.21.15.146:6801/1526135780","cluster_addr":"172.21.15.146:6803/1526135780","heartbeat_back_addr":"172.21.15.146:6807/1526135780","heartbeat_front_addr":"172.21.15.146:6805/1526135780","state":["exists","up"]},{"osd":5,"uuid":"fb69c525-f1be-4b7d-ab32-6c29b0d6fddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6809","nonce":460262507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6811","nonce":460262507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6815","nonce":460262507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6813","nonce":460262507}]},"public_addr":"172.21.15.146:6809/460262507","cluster_addr":"172.21.15.146:6811/460262507","heartbeat_back_addr":"172.21.15.146:6815/460262507","heartbeat_front_addr":"172.21.15.146:6813/460262507","state":["exists","up"]},{"osd":6,"uuid":"c32e7fed-370d-4401-8ce0-6a4335b0d168","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6823","nonce":1392897493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6827","nonce":1392897493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6831","nonce":1392897493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6829","nonce":1392897493}]},"public_addr":"172.21.15.146:6823/1392897493","cluster_addr":"172.21.15.146:6827/1392897493","heartbeat_back_addr":"172.21.15.146:6831/1392897493","heartbeat_front_addr":"172.21.15.146:6829/1392897493","state":["exists","up"]},{"osd":7,"uuid":"75a79b44-fa5f-426d-a7d3-71f21f78835c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6817","nonce":3268323156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6819","nonce":3268323156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6825","nonce":3268323156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6822","nonce":3268323156}]},"public_addr":"172.21.15.146:6817/3268323156","cluster_addr":"172.21.15.146:6819/3268323156","heartbeat_back_addr":"172.21.15.146:6825/3268323156","heartbeat_front_addr":"172.21.15.146:6822/3268323156","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.310437+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.295850+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.294674+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.526855+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.300023+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.595160+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.346046+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.558741+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T09:28:35.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.205+0000 7fd8d1c00640 1 -- 172.21.15.6:0/1938998781 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd8ac06b500 msgr2=0x7fd8ac06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:35.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.205+0000 7fd8d1c00640 1 --2- 172.21.15.6:0/1938998781 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd8ac06b500 0x7fd8ac06d9c0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fd8bc009aa0 tx=0x7fd8bc008040 comp rx=0 tx=0).stop 2024-03-04T09:28:35.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.205+0000 7fd8d1c00640 1 -- 172.21.15.6:0/1938998781 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd8cc1f3cb0 msgr2=0x7fd8cc14d640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:35.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.205+0000 7fd8d1c00640 1 --2- 172.21.15.6:0/1938998781 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd8cc1f3cb0 0x7fd8cc14d640 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fd8b800e7f0 tx=0x7fd8b800ecb0 comp rx=0 tx=0).stop 2024-03-04T09:28:35.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.205+0000 7fd8d1c00640 1 -- 172.21.15.6:0/1938998781 shutdown_connections 2024-03-04T09:28:35.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.205+0000 7fd8d1c00640 1 --2- 172.21.15.6:0/1938998781 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd8cc0cc790 0x7fd8cc0ccc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:35.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.205+0000 7fd8d1c00640 1 --2- 172.21.15.6:0/1938998781 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd8cc1f3cb0 0x7fd8cc14d640 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:35.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.205+0000 7fd8d1c00640 1 --2- 172.21.15.6:0/1938998781 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd8ac06b500 0x7fd8ac06d9c0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:35.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.205+0000 7fd8d1c00640 1 --2- 172.21.15.6:0/1938998781 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd8b0004d40 0x7fd8cc15f2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:35.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.205+0000 7fd8d1c00640 1 -- 172.21.15.6:0/1938998781 >> 172.21.15.6:0/1938998781 conn(0x7fd8cc06afb0 msgr2=0x7fd8cc0ceda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:35.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.206+0000 7fd8d1c00640 1 -- 172.21.15.6:0/1938998781 shutdown_connections 2024-03-04T09:28:35.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.206+0000 7fd8d1c00640 1 -- 172.21.15.6:0/1938998781 wait complete. 2024-03-04T09:28:35.220 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-04T09:28:35.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.424+0000 7f3f0b200640 1 Processor -- start 2024-03-04T09:28:35.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.425+0000 7f3f0b200640 1 -- start start 2024-03-04T09:28:35.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.426+0000 7f3f0b200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f3f0c15e070 0x7f3f0c0d2960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:35.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.426+0000 7f3f0b200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3f0c0d0cd0 0x7f3f0c0cf870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:35.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.426+0000 7f3f0b200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f0c0d18c0 con 0x7f3f0c0cf410 2024-03-04T09:28:35.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.426+0000 7f3f0b200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f0c0d1ba0 con 0x7f3f0c15e070 2024-03-04T09:28:35.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.426+0000 7f3f0b200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f0c0d1e80 con 0x7f3f0c0d0cd0 2024-03-04T09:28:35.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.427+0000 7f3f09400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f3f0c0cf410 0x7f3f0c0d05c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:41208/0 (socket says 172.21.15.6:41208) 2024-03-04T09:28:35.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.427+0000 7f3f09400640 1 -- 172.21.15.6:0/3339126991 learned_addr learned my addr 172.21.15.6:0/3339126991 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:28:35.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.427+0000 7f3f0a800640 1 --2- 172.21.15.6:0/3339126991 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3f0c0d0cd0 0x7f3f0c0cf870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:35.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.427+0000 7f3f09e00640 1 --2- 172.21.15.6:0/3339126991 >> v2:172.21.15.146:3300/0 conn(0x7f3f0c15e070 0x7f3f0c0d2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:35.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.427+0000 7f3f08a00640 1 -- 172.21.15.6:0/3339126991 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2235982772 0 0) 0x7f3f0c0d18c0 con 0x7f3f0c0cf410 2024-03-04T09:28:35.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.427+0000 7f3f08a00640 1 -- 172.21.15.6:0/3339126991 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3eec003520 con 0x7f3f0c0cf410 2024-03-04T09:28:35.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.428+0000 7f3f08a00640 1 -- 172.21.15.6:0/3339126991 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3672122976 0 0) 0x7f3eec003520 con 0x7f3f0c0cf410 2024-03-04T09:28:35.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.428+0000 7f3f08a00640 1 -- 172.21.15.6:0/3339126991 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3f0c0d0cd0 msgr2=0x7f3f0c0cf870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:35.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.428+0000 7f3f08a00640 1 --2- 172.21.15.6:0/3339126991 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3f0c0d0cd0 0x7f3f0c0cf870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:35.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.428+0000 7f3f08a00640 1 -- 172.21.15.6:0/3339126991 >> v2:172.21.15.146:3300/0 conn(0x7f3f0c15e070 msgr2=0x7f3f0c0d2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:35.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.428+0000 7f3f08a00640 1 --2- 172.21.15.6:0/3339126991 >> v2:172.21.15.146:3300/0 conn(0x7f3f0c15e070 0x7f3f0c0d2960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:35.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.428+0000 7f3f08a00640 1 -- 172.21.15.6:0/3339126991 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f0c0d2160 con 0x7f3f0c0cf410 2024-03-04T09:28:35.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.428+0000 7f3f08a00640 1 -- 172.21.15.6:0/3339126991 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7f3ef4001670 con 0x7f3f0c0cf410 2024-03-04T09:28:35.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.428+0000 7f3f08a00640 1 -- 172.21.15.6:0/3339126991 >> v1:172.21.15.6:6789/0 conn(0x7f3f0c0cf410 legacy=0x7f3f0c0d05c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:35.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.428+0000 7f3f08a00640 1 --2- 172.21.15.6:0/3339126991 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3eec0039b0 0x7f3eec003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:35.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.428+0000 7f3f08a00640 1 --2- 172.21.15.6:0/3339126991 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3eec004800 0x7f3eec004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:35.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.428+0000 7f3f08a00640 1 --2- 172.21.15.6:0/3339126991 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3f0c0d0cd0 0x7f3eec0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:35.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.428+0000 7f3f08a00640 1 -- 172.21.15.6:0/3339126991 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f0c0d1ba0 con 0x7f3eec004800 2024-03-04T09:28:35.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.428+0000 7f3f08a00640 1 -- 172.21.15.6:0/3339126991 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f0c0d1e80 con 0x7f3eec0039b0 2024-03-04T09:28:35.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.428+0000 7f3f08a00640 1 -- 172.21.15.6:0/3339126991 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3eec00b9c0 con 0x7f3f0c0d0cd0 2024-03-04T09:28:35.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.428+0000 7f3f09400640 1 --2- 172.21.15.6:0/3339126991 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3eec004800 0x7f3eec004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:35.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.429+0000 7f3f09400640 1 -- 172.21.15.6:0/3339126991 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3f0c0d0cd0 msgr2=0x7f3eec0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:35.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.429+0000 7f3f09400640 1 --2- 172.21.15.6:0/3339126991 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3f0c0d0cd0 0x7f3eec0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:35.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.429+0000 7f3f09400640 1 -- 172.21.15.6:0/3339126991 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3eec0039b0 msgr2=0x7f3eec003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:35.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.429+0000 7f3f09400640 1 --2- 172.21.15.6:0/3339126991 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3eec0039b0 0x7f3eec003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:35.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.429+0000 7f3f09400640 1 -- 172.21.15.6:0/3339126991 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3eec003520 con 0x7f3eec004800 2024-03-04T09:28:35.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.430+0000 7f3f09400640 1 --2- 172.21.15.6:0/3339126991 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3eec004800 0x7f3eec004c80 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f3ef400c240 tx=0x7f3eec005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4ab48b53fba596b server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:35.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.430+0000 7f3f08a00640 1 -- 172.21.15.6:0/3339126991 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ef4007d30 con 0x7f3eec004800 2024-03-04T09:28:35.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.430+0000 7f3f08a00640 1 -- 172.21.15.6:0/3339126991 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ef4030070 con 0x7f3eec004800 2024-03-04T09:28:35.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.430+0000 7f3f08a00640 1 -- 172.21.15.6:0/3339126991 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ef402cd60 con 0x7f3eec004800 2024-03-04T09:28:35.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.430+0000 7f3f0b200640 1 -- 172.21.15.6:0/3339126991 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3eec004800 msgr2=0x7f3eec004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:35.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.430+0000 7f3f0b200640 1 --2- 172.21.15.6:0/3339126991 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3eec004800 0x7f3eec004c80 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f3ef400c240 tx=0x7f3eec005830 comp rx=0 tx=0).stop 2024-03-04T09:28:35.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.431+0000 7f3f0a800640 1 -- 172.21.15.6:0/3339126991 reap_dead start 2024-03-04T09:28:35.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.431+0000 7f3f0b200640 1 -- 172.21.15.6:0/3339126991 shutdown_connections 2024-03-04T09:28:35.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.431+0000 7f3f0b200640 1 -- 172.21.15.6:0/3339126991 >> 172.21.15.6:0/3339126991 conn(0x7f3f0c08d2a0 msgr2=0x7f3f0c0d3cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:35.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.431+0000 7f3f0b200640 1 -- 172.21.15.6:0/3339126991 shutdown_connections 2024-03-04T09:28:35.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.431+0000 7f3f0b200640 1 -- 172.21.15.6:0/3339126991 wait complete. 2024-03-04T09:28:35.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.433+0000 7f3f0b200640 1 Processor -- start 2024-03-04T09:28:35.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.433+0000 7f3f0b200640 1 -- start start 2024-03-04T09:28:35.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.434+0000 7f3f0b200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3f0c1f3be0 0x7f3f0c1f3fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:35.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.434+0000 7f3f0b200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3f0c0c9300 0x7f3f0c15ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:35.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.434+0000 7f3f0b200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3f0c0c9760 0x7f3f0c15f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:35.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.434+0000 7f3f0b200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f0c0d4730 con 0x7f3f0c0c9300 2024-03-04T09:28:35.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.434+0000 7f3f0b200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f0c0d45b0 con 0x7f3f0c1f3be0 2024-03-04T09:28:35.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.434+0000 7f3f0b200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f0c0cf870 con 0x7f3f0c0c9760 2024-03-04T09:28:35.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.434+0000 7f3f09400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3f0c0c9300 0x7f3f0c15ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:35.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.434+0000 7f3f09400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3f0c0c9300 0x7f3f0c15ee60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53862/0 (socket says 172.21.15.6:53862) 2024-03-04T09:28:35.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.434+0000 7f3f09400640 1 -- 172.21.15.6:0/753884629 learned_addr learned my addr 172.21.15.6:0/753884629 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:28:35.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.434+0000 7f3f09400640 1 -- 172.21.15.6:0/753884629 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3f0c0c9760 msgr2=0x7f3f0c15f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:35.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.434+0000 7f3f09400640 1 --2- 172.21.15.6:0/753884629 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3f0c0c9760 0x7f3f0c15f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:35.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.434+0000 7f3f09400640 1 -- 172.21.15.6:0/753884629 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3f0c1f3be0 msgr2=0x7f3f0c1f3fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:35.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.434+0000 7f3f09400640 1 --2- 172.21.15.6:0/753884629 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3f0c1f3be0 0x7f3f0c1f3fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:35.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.434+0000 7f3f09400640 1 -- 172.21.15.6:0/753884629 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f0c0cbca0 con 0x7f3f0c0c9300 2024-03-04T09:28:35.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.435+0000 7f3f09400640 1 --2- 172.21.15.6:0/753884629 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3f0c0c9300 0x7f3f0c15ee60 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f3ef400c240 tx=0x7f3f0c0d1a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:35.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.435+0000 7f3f02000640 1 -- 172.21.15.6:0/753884629 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ef4045070 con 0x7f3f0c0c9300 2024-03-04T09:28:35.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.435+0000 7f3f02000640 1 -- 172.21.15.6:0/753884629 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ef4004cb0 con 0x7f3f0c0c9300 2024-03-04T09:28:35.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.435+0000 7f3f02000640 1 -- 172.21.15.6:0/753884629 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ef4001c70 con 0x7f3f0c0c9300 2024-03-04T09:28:35.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.435+0000 7f3f0b200640 1 -- 172.21.15.6:0/753884629 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f0c0c9ce0 con 0x7f3f0c0c9300 2024-03-04T09:28:35.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.435+0000 7f3f0b200640 1 -- 172.21.15.6:0/753884629 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f0c0cbf40 con 0x7f3f0c0c9300 2024-03-04T09:28:35.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.436+0000 7f3f0b200640 1 -- 172.21.15.6:0/753884629 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f0c15d180 con 0x7f3f0c0c9300 2024-03-04T09:28:35.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.437+0000 7f3f02000640 1 -- 172.21.15.6:0/753884629 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3ef4048070 con 0x7f3f0c0c9300 2024-03-04T09:28:35.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.438+0000 7f3f02000640 1 --2- 172.21.15.6:0/753884629 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f3ee806b520 0x7f3ee806d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:35.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.438+0000 7f3f02000640 1 -- 172.21.15.6:0/753884629 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3ef40ab2d0 con 0x7f3f0c0c9300 2024-03-04T09:28:35.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.439+0000 7f3f09e00640 1 --2- 172.21.15.6:0/753884629 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f3ee806b520 0x7f3ee806d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:35.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.441+0000 7f3f09e00640 1 --2- 172.21.15.6:0/753884629 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f3ee806b520 0x7f3ee806d9e0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f3efc006a90 tx=0x7f3efc006a20 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:35.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.443+0000 7f3f02000640 1 -- 172.21.15.6:0/753884629 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3f0c15d180 con 0x7f3f0c0c9300 2024-03-04T09:28:35.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.686+0000 7f3f0b200640 1 -- 172.21.15.6:0/753884629 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f3f0c04f330 con 0x7f3f0c0c9300 2024-03-04T09:28:35.684 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-03-04T09:28:35.687+0000 7f1606a00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-04T09:28:35.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.691+0000 7f3f02000640 1 -- 172.21.15.6:0/753884629 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v35) v1 ==== 161+0+0 (secure 0 0 0) 0x7f3ef407a130 con 0x7f3f0c0c9300 2024-03-04T09:28:35.691 INFO:teuthology.orchestra.run.smithi006.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-04T09:28:35.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.694+0000 7f3f0b200640 1 -- 172.21.15.6:0/753884629 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f3ee806b520 msgr2=0x7f3ee806d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:35.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.694+0000 7f3f0b200640 1 --2- 172.21.15.6:0/753884629 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f3ee806b520 0x7f3ee806d9e0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f3efc006a90 tx=0x7f3efc006a20 comp rx=0 tx=0).stop 2024-03-04T09:28:35.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.694+0000 7f3f0b200640 1 -- 172.21.15.6:0/753884629 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3f0c0c9300 msgr2=0x7f3f0c15ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:35.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.694+0000 7f3f0b200640 1 --2- 172.21.15.6:0/753884629 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3f0c0c9300 0x7f3f0c15ee60 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f3ef400c240 tx=0x7f3f0c0d1a90 comp rx=0 tx=0).stop 2024-03-04T09:28:35.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.695+0000 7f3f0b200640 1 -- 172.21.15.6:0/753884629 shutdown_connections 2024-03-04T09:28:35.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.695+0000 7f3f0b200640 1 --2- 172.21.15.6:0/753884629 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f3ee806b520 0x7f3ee806d9e0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:35.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.695+0000 7f3f0b200640 1 --2- 172.21.15.6:0/753884629 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3f0c0c9760 0x7f3f0c15f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:35.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.695+0000 7f3f0b200640 1 --2- 172.21.15.6:0/753884629 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3f0c0c9300 0x7f3f0c15ee60 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:35.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.695+0000 7f3f0b200640 1 --2- 172.21.15.6:0/753884629 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3f0c1f3be0 0x7f3f0c1f3fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:35.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.695+0000 7f3f0b200640 1 -- 172.21.15.6:0/753884629 >> 172.21.15.6:0/753884629 conn(0x7f3f0c08d2a0 msgr2=0x7f3f0c0ce8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:35.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.696+0000 7f3f0b200640 1 -- 172.21.15.6:0/753884629 shutdown_connections 2024-03-04T09:28:35.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.696+0000 7f3f0b200640 1 -- 172.21.15.6:0/753884629 wait complete. 2024-03-04T09:28:35.712 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-04T09:28:35.817 INFO:tasks.ceph.mds.b.smithi146.stdout:starting mds.b at 2024-03-04T09:28:35.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.935+0000 7f16fda00640 1 Processor -- start 2024-03-04T09:28:35.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.935+0000 7f16fda00640 1 -- start start 2024-03-04T09:28:35.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.937+0000 7f16fda00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f16f80ccad0 0x7f16f80ccf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:35.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.937+0000 7f16fda00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16f806d090 con 0x7f16f815e680 2024-03-04T09:28:35.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.937+0000 7f16fda00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16f815eb80 con 0x7f16f806da40 2024-03-04T09:28:35.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.937+0000 7f16fda00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16f806ea50 con 0x7f16f80ccad0 2024-03-04T09:28:35.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.937+0000 7f16f6000640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f16f815e680 0x7f16f806c980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:41220/0 (socket says 172.21.15.6:41220) 2024-03-04T09:28:35.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.937+0000 7f16f6000640 1 -- 172.21.15.6:0/1399760815 learned_addr learned my addr 172.21.15.6:0/1399760815 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:28:35.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.937+0000 7f16f7400640 1 --2- 172.21.15.6:0/1399760815 >> v2:172.21.15.6:3300/0 conn(0x7f16f80ccad0 0x7f16f80ccf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:35.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.938+0000 7f16f5600640 1 -- 172.21.15.6:0/1399760815 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2874488407 0 0) 0x7f16f806d090 con 0x7f16f815e680 2024-03-04T09:28:35.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.938+0000 7f16f5600640 1 -- 172.21.15.6:0/1399760815 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16dc003520 con 0x7f16f815e680 2024-03-04T09:28:35.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.938+0000 7f16f7400640 1 -- 172.21.15.6:0/1399760815 >> v1:172.21.15.146:6789/0 conn(0x7f16f806da40 legacy=0x7f16f806e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:35.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.938+0000 7f16f7400640 1 -- 172.21.15.6:0/1399760815 >> v1:172.21.15.6:6789/0 conn(0x7f16f815e680 legacy=0x7f16f806c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:35.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.938+0000 7f16f7400640 1 -- 172.21.15.6:0/1399760815 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16f815ee60 con 0x7f16f80ccad0 2024-03-04T09:28:35.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.939+0000 7f16f7400640 1 --2- 172.21.15.6:0/1399760815 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f16f80ccad0 0x7f16f80ccf80 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f16f0009d90 tx=0x7f16f002bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d04800970d2a122 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:35.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.940+0000 7f16f5600640 1 -- 172.21.15.6:0/1399760815 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16f0039070 con 0x7f16f80ccad0 2024-03-04T09:28:35.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.940+0000 7f16f5600640 1 -- 172.21.15.6:0/1399760815 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16f002ccf0 con 0x7f16f80ccad0 2024-03-04T09:28:35.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.940+0000 7f16f5600640 1 -- 172.21.15.6:0/1399760815 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16f0034660 con 0x7f16f80ccad0 2024-03-04T09:28:35.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.940+0000 7f16fda00640 1 -- 172.21.15.6:0/1399760815 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f16f80ccad0 msgr2=0x7f16f80ccf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:35.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.940+0000 7f16fda00640 1 --2- 172.21.15.6:0/1399760815 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f16f80ccad0 0x7f16f80ccf80 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f16f0009d90 tx=0x7f16f002bf40 comp rx=0 tx=0).stop 2024-03-04T09:28:35.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.940+0000 7f16fda00640 1 -- 172.21.15.6:0/1399760815 shutdown_connections 2024-03-04T09:28:35.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.940+0000 7f16fda00640 1 --2- 172.21.15.6:0/1399760815 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f16f80ccad0 0x7f16f80ccf80 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:35.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.940+0000 7f16fda00640 1 -- 172.21.15.6:0/1399760815 >> 172.21.15.6:0/1399760815 conn(0x7f16f806b3d0 msgr2=0x7f16f806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:35.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.940+0000 7f16fda00640 1 -- 172.21.15.6:0/1399760815 shutdown_connections 2024-03-04T09:28:35.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.940+0000 7f16fda00640 1 -- 172.21.15.6:0/1399760815 wait complete. 2024-03-04T09:28:35.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.942+0000 7f16fda00640 1 Processor -- start 2024-03-04T09:28:35.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.942+0000 7f16fda00640 1 -- start start 2024-03-04T09:28:35.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.943+0000 7f16fda00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f16f806da40 0x7f16f809b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:35.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.943+0000 7f16fda00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f16f80ccad0 0x7f16f814fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:35.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.943+0000 7f16fda00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f16f815e680 0x7f16f81603f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:35.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.943+0000 7f16fda00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16f815f420 con 0x7f16f80ccad0 2024-03-04T09:28:35.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.943+0000 7f16fda00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16f8161c70 con 0x7f16f815e680 2024-03-04T09:28:35.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.943+0000 7f16fda00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16f8162100 con 0x7f16f806da40 2024-03-04T09:28:35.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.943+0000 7f16f6000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f16f80ccad0 0x7f16f814fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:35.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.944+0000 7f16f6000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f16f80ccad0 0x7f16f814fd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53878/0 (socket says 172.21.15.6:53878) 2024-03-04T09:28:35.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.944+0000 7f16f6000640 1 -- 172.21.15.6:0/649396387 learned_addr learned my addr 172.21.15.6:0/649396387 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:28:35.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.944+0000 7f16f6000640 1 -- 172.21.15.6:0/649396387 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f16f806da40 msgr2=0x7f16f809b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:35.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.944+0000 7f16f6000640 1 --2- 172.21.15.6:0/649396387 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f16f806da40 0x7f16f809b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:35.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.944+0000 7f16f6000640 1 -- 172.21.15.6:0/649396387 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f16f815e680 msgr2=0x7f16f81603f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:35.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.944+0000 7f16f6000640 1 --2- 172.21.15.6:0/649396387 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f16f815e680 0x7f16f81603f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:35.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.944+0000 7f16f6000640 1 -- 172.21.15.6:0/649396387 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16f80c9cd0 con 0x7f16f80ccad0 2024-03-04T09:28:35.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.944+0000 7f16f6000640 1 --2- 172.21.15.6:0/649396387 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f16f80ccad0 0x7f16f814fd60 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f16e000c410 tx=0x7f16e000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:35.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.944+0000 7f16eea00640 1 -- 172.21.15.6:0/649396387 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16e0017070 con 0x7f16f80ccad0 2024-03-04T09:28:35.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.945+0000 7f16eea00640 1 -- 172.21.15.6:0/649396387 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16e0005ce0 con 0x7f16f80ccad0 2024-03-04T09:28:35.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.945+0000 7f16eea00640 1 -- 172.21.15.6:0/649396387 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16e00072f0 con 0x7f16f80ccad0 2024-03-04T09:28:35.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.945+0000 7f16fda00640 1 -- 172.21.15.6:0/649396387 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16f809ab90 con 0x7f16f80ccad0 2024-03-04T09:28:35.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.945+0000 7f16fda00640 1 -- 172.21.15.6:0/649396387 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16f809b150 con 0x7f16f80ccad0 2024-03-04T09:28:35.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.946+0000 7f16fda00640 1 -- 172.21.15.6:0/649396387 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16f809b350 con 0x7f16f80ccad0 2024-03-04T09:28:35.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.947+0000 7f16eea00640 1 -- 172.21.15.6:0/649396387 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f16e001a070 con 0x7f16f80ccad0 2024-03-04T09:28:35.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.948+0000 7f16eea00640 1 --2- 172.21.15.6:0/649396387 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f16d406b4d0 0x7f16d406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:35.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.949+0000 7f16eea00640 1 -- 172.21.15.6:0/649396387 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f16e0089670 con 0x7f16f80ccad0 2024-03-04T09:28:35.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.949+0000 7f16f6a00640 1 --2- 172.21.15.6:0/649396387 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f16d406b4d0 0x7f16d406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:35.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.950+0000 7f16f6a00640 1 --2- 172.21.15.6:0/649396387 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f16d406b4d0 0x7f16d406d990 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f16e8000f60 tx=0x7f16e80079b0 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:35.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:35.953+0000 7f16eea00640 1 -- 172.21.15.6:0/649396387 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f16f809b350 con 0x7f16f80ccad0 2024-03-04T09:28:36.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:36.197+0000 7f16fda00640 1 -- 172.21.15.6:0/649396387 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f16f804f330 con 0x7f16f80ccad0 2024-03-04T09:28:36.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:36.701+0000 7f16eea00640 1 -- 172.21.15.6:0/649396387 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v36) v1 ==== 104+0+0 (secure 0 0 0) 0x7f16e0058550 con 0x7f16f80ccad0 2024-03-04T09:28:36.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:36.705+0000 7f16fda00640 1 -- 172.21.15.6:0/649396387 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f16d406b4d0 msgr2=0x7f16d406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:36.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:36.705+0000 7f16fda00640 1 --2- 172.21.15.6:0/649396387 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f16d406b4d0 0x7f16d406d990 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f16e8000f60 tx=0x7f16e80079b0 comp rx=0 tx=0).stop 2024-03-04T09:28:36.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:36.705+0000 7f16fda00640 1 -- 172.21.15.6:0/649396387 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f16f80ccad0 msgr2=0x7f16f814fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:36.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:36.705+0000 7f16fda00640 1 --2- 172.21.15.6:0/649396387 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f16f80ccad0 0x7f16f814fd60 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f16e000c410 tx=0x7f16e000c8d0 comp rx=0 tx=0).stop 2024-03-04T09:28:36.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:36.705+0000 7f16fda00640 1 -- 172.21.15.6:0/649396387 shutdown_connections 2024-03-04T09:28:36.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:36.705+0000 7f16fda00640 1 --2- 172.21.15.6:0/649396387 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f16f815e680 0x7f16f81603f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:36.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:36.705+0000 7f16fda00640 1 --2- 172.21.15.6:0/649396387 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f16f80ccad0 0x7f16f814fd60 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:36.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:36.705+0000 7f16fda00640 1 --2- 172.21.15.6:0/649396387 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f16d406b4d0 0x7f16d406d990 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:36.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:36.705+0000 7f16fda00640 1 --2- 172.21.15.6:0/649396387 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f16f806da40 0x7f16f809b670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:36.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:36.705+0000 7f16fda00640 1 -- 172.21.15.6:0/649396387 >> 172.21.15.6:0/649396387 conn(0x7f16f806b3d0 msgr2=0x7f16f806b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:36.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:36.705+0000 7f16fda00640 1 -- 172.21.15.6:0/649396387 shutdown_connections 2024-03-04T09:28:36.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:36.706+0000 7f16fda00640 1 -- 172.21.15.6:0/649396387 wait complete. 2024-03-04T09:28:36.724 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-04T09:28:36.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:36.958+0000 7f829d200640 1 Processor -- start 2024-03-04T09:28:36.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:36.958+0000 7f829d200640 1 -- start start 2024-03-04T09:28:36.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:36.960+0000 7f829d200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f8298159730 0x7f829815cc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:36.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:36.960+0000 7f829d200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f82980ceea0 0x7f8298072450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:36.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:36.960+0000 7f829d200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8298159b10 con 0x7f82980ccc20 2024-03-04T09:28:36.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:36.960+0000 7f829d200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8298071690 con 0x7f8298159730 2024-03-04T09:28:36.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:36.960+0000 7f829d200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8298071970 con 0x7f82980ceea0 2024-03-04T09:28:36.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:36.961+0000 7f8296a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f82980ceea0 0x7f8298072450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:36.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:36.961+0000 7f8296a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f82980ceea0 0x7f8298072450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.6:50622/0 (socket says 172.21.15.6:50622) 2024-03-04T09:28:36.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:36.961+0000 7f8296a00640 1 -- 172.21.15.6:0/2550829413 learned_addr learned my addr 172.21.15.6:0/2550829413 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:28:36.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:36.961+0000 7f8297400640 1 --2- 172.21.15.6:0/2550829413 >> v2:172.21.15.146:3300/0 conn(0x7f8298159730 0x7f829815cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:36.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:36.962+0000 7f8296a00640 1 -- 172.21.15.6:0/2550829413 >> v2:172.21.15.146:3300/0 conn(0x7f8298159730 msgr2=0x7f829815cc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:36.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:36.962+0000 7f8296a00640 1 --2- 172.21.15.6:0/2550829413 >> v2:172.21.15.146:3300/0 conn(0x7f8298159730 0x7f829815cc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:36.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:36.962+0000 7f8296a00640 1 -- 172.21.15.6:0/2550829413 >> v1:172.21.15.146:6789/0 conn(0x7f82980ccc20 legacy=0x7f82980cd040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:36.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:36.962+0000 7f8296a00640 1 -- 172.21.15.6:0/2550829413 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8298071c50 con 0x7f82980ceea0 2024-03-04T09:28:36.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:36.963+0000 7f8296a00640 1 --2- 172.21.15.6:0/2550829413 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f82980ceea0 0x7f8298072450 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f8284009870 tx=0x7f828402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=684948c3a56cc63a server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:36.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:36.964+0000 7f8296000640 1 -- 172.21.15.6:0/2550829413 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8284034070 con 0x7f82980ceea0 2024-03-04T09:28:36.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:36.964+0000 7f8296000640 1 -- 172.21.15.6:0/2550829413 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8284030840 con 0x7f82980ceea0 2024-03-04T09:28:36.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:36.964+0000 7f8296000640 1 -- 172.21.15.6:0/2550829413 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8284030c10 con 0x7f82980ceea0 2024-03-04T09:28:36.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:36.964+0000 7f829d200640 1 -- 172.21.15.6:0/2550829413 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f82980ceea0 msgr2=0x7f8298072450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:36.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:36.964+0000 7f829d200640 1 --2- 172.21.15.6:0/2550829413 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f82980ceea0 0x7f8298072450 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f8284009870 tx=0x7f828402b980 comp rx=0 tx=0).stop 2024-03-04T09:28:36.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:36.965+0000 7f829d200640 1 -- 172.21.15.6:0/2550829413 shutdown_connections 2024-03-04T09:28:36.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:36.965+0000 7f829d200640 1 --2- 172.21.15.6:0/2550829413 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f82980ceea0 0x7f8298072450 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:36.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:36.965+0000 7f829d200640 1 --2- 172.21.15.6:0/2550829413 >> v2:172.21.15.146:3300/0 conn(0x7f8298159730 0x7f829815cc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:36.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:36.965+0000 7f829d200640 1 -- 172.21.15.6:0/2550829413 >> 172.21.15.6:0/2550829413 conn(0x7f829806afb0 msgr2=0x7f829806b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:36.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:36.965+0000 7f829d200640 1 -- 172.21.15.6:0/2550829413 shutdown_connections 2024-03-04T09:28:36.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:36.965+0000 7f829d200640 1 -- 172.21.15.6:0/2550829413 wait complete. 2024-03-04T09:28:36.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:36.967+0000 7f829d200640 1 Processor -- start 2024-03-04T09:28:36.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:36.967+0000 7f829d200640 1 -- start start 2024-03-04T09:28:36.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:36.968+0000 7f829d200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f82980ccc20 0x7f82980adf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:36.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:36.968+0000 7f829d200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f82980ceea0 0x7f8298152320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:36.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:36.968+0000 7f829d200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8298159730 0x7f82981f1f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:36.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:36.968+0000 7f829d200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82980cf300 con 0x7f82980ceea0 2024-03-04T09:28:36.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:36.968+0000 7f829d200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82980cf6d0 con 0x7f8298159730 2024-03-04T09:28:36.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:36.968+0000 7f829d200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82980cf480 con 0x7f82980ccc20 2024-03-04T09:28:36.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:36.969+0000 7f8296a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f82980ceea0 0x7f8298152320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:36.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:36.969+0000 7f8296a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f82980ceea0 0x7f8298152320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53888/0 (socket says 172.21.15.6:53888) 2024-03-04T09:28:36.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:36.969+0000 7f8296a00640 1 -- 172.21.15.6:0/313624701 learned_addr learned my addr 172.21.15.6:0/313624701 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:28:36.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:36.969+0000 7f8297400640 1 --2- 172.21.15.6:0/313624701 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f82980ccc20 0x7f82980adf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:36.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:36.969+0000 7f8297e00640 1 --2- 172.21.15.6:0/313624701 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8298159730 0x7f82981f1f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:36.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:36.969+0000 7f8296a00640 1 -- 172.21.15.6:0/313624701 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f82980ccc20 msgr2=0x7f82980adf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:36.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:36.969+0000 7f8296a00640 1 --2- 172.21.15.6:0/313624701 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f82980ccc20 0x7f82980adf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:36.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:36.969+0000 7f8296a00640 1 -- 172.21.15.6:0/313624701 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8298159730 msgr2=0x7f82981f1f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:36.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:36.969+0000 7f8296a00640 1 --2- 172.21.15.6:0/313624701 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8298159730 0x7f82981f1f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:36.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:36.970+0000 7f8296a00640 1 -- 172.21.15.6:0/313624701 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82981f2470 con 0x7f82980ceea0 2024-03-04T09:28:36.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:36.970+0000 7f8296a00640 1 --2- 172.21.15.6:0/313624701 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f82980ceea0 0x7f8298152320 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f8284000c00 tx=0x7f8284004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:36.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:36.970+0000 7f828f400640 1 -- 172.21.15.6:0/313624701 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8284048070 con 0x7f82980ceea0 2024-03-04T09:28:36.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:36.970+0000 7f828f400640 1 -- 172.21.15.6:0/313624701 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8284030640 con 0x7f82980ceea0 2024-03-04T09:28:36.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:36.970+0000 7f829d200640 1 -- 172.21.15.6:0/313624701 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82981f2710 con 0x7f82980ceea0 2024-03-04T09:28:36.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:36.970+0000 7f828f400640 1 -- 172.21.15.6:0/313624701 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8284034040 con 0x7f82980ceea0 2024-03-04T09:28:36.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:36.971+0000 7f829d200640 1 -- 172.21.15.6:0/313624701 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82981f2cd0 con 0x7f82980ceea0 2024-03-04T09:28:36.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:36.972+0000 7f829d200640 1 -- 172.21.15.6:0/313624701 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f829806b390 con 0x7f82980ceea0 2024-03-04T09:28:36.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:36.973+0000 7f828f400640 1 -- 172.21.15.6:0/313624701 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f828404b070 con 0x7f82980ceea0 2024-03-04T09:28:36.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:36.974+0000 7f828f400640 1 --2- 172.21.15.6:0/313624701 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f827806b4d0 0x7f827806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:36.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:36.974+0000 7f828f400640 1 -- 172.21.15.6:0/313624701 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f82840ac0d0 con 0x7f82980ceea0 2024-03-04T09:28:36.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:36.975+0000 7f8297400640 1 --2- 172.21.15.6:0/313624701 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f827806b4d0 0x7f827806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:36.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:36.976+0000 7f8297400640 1 --2- 172.21.15.6:0/313624701 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f827806b4d0 0x7f827806d990 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f82880079a0 tx=0x7f8288008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:36.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:36.981+0000 7f828f400640 1 -- 172.21.15.6:0/313624701 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f829806b390 con 0x7f82980ceea0 2024-03-04T09:28:37.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:37.183+0000 7f829d200640 1 -- 172.21.15.6:0/313624701 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f829804f330 con 0x7f82980ceea0 2024-03-04T09:28:38.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:38.113+0000 7f828f400640 1 -- 172.21.15.6:0/313624701 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v58) v1 ==== 181+0+0 (secure 0 0 0) 0x7f828407a890 con 0x7f82980ceea0 2024-03-04T09:28:38.111 INFO:teuthology.orchestra.run.smithi006.stderr:pool 'cephfs_metadata' removed 2024-03-04T09:28:38.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:38.117+0000 7f829d200640 1 -- 172.21.15.6:0/313624701 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f827806b4d0 msgr2=0x7f827806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:38.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:38.117+0000 7f829d200640 1 --2- 172.21.15.6:0/313624701 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f827806b4d0 0x7f827806d990 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f82880079a0 tx=0x7f8288008040 comp rx=0 tx=0).stop 2024-03-04T09:28:38.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:38.117+0000 7f829d200640 1 -- 172.21.15.6:0/313624701 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f82980ceea0 msgr2=0x7f8298152320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:38.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:38.117+0000 7f829d200640 1 --2- 172.21.15.6:0/313624701 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f82980ceea0 0x7f8298152320 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f8284000c00 tx=0x7f8284004730 comp rx=0 tx=0).stop 2024-03-04T09:28:38.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:38.118+0000 7f829d200640 1 -- 172.21.15.6:0/313624701 shutdown_connections 2024-03-04T09:28:38.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:38.118+0000 7f829d200640 1 --2- 172.21.15.6:0/313624701 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8298159730 0x7f82981f1f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:38.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:38.118+0000 7f829d200640 1 --2- 172.21.15.6:0/313624701 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f82980ceea0 0x7f8298152320 secure :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f8284000c00 tx=0x7f8284004730 comp rx=0 tx=0).stop 2024-03-04T09:28:38.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:38.118+0000 7f829d200640 1 --2- 172.21.15.6:0/313624701 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f827806b4d0 0x7f827806d990 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:38.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:38.118+0000 7f829d200640 1 --2- 172.21.15.6:0/313624701 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f82980ccc20 0x7f82980adf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:38.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:38.118+0000 7f829d200640 1 -- 172.21.15.6:0/313624701 >> 172.21.15.6:0/313624701 conn(0x7f829806afb0 msgr2=0x7f8298159f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:38.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:38.118+0000 7f829d200640 1 -- 172.21.15.6:0/313624701 shutdown_connections 2024-03-04T09:28:38.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:38.119+0000 7f829d200640 1 -- 172.21.15.6:0/313624701 wait complete. 2024-03-04T09:28:38.139 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-04T09:28:38.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:38.364+0000 7fc6c5400640 1 Processor -- start 2024-03-04T09:28:38.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:38.365+0000 7fc6c5400640 1 -- start start 2024-03-04T09:28:38.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:38.366+0000 7fc6c5400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc6c015df80 0x7fc6c015ef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:38.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:38.366+0000 7fc6c5400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc6c00cec90 0x7fc6c00b48a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:38.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:38.366+0000 7fc6c5400640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6c00cf140 con 0x7fc6c00ce810 2024-03-04T09:28:38.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:38.366+0000 7fc6c5400640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6c00cf420 con 0x7fc6c00cec90 2024-03-04T09:28:38.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:38.366+0000 7fc6c5400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6c00cf700 con 0x7fc6c015df80 2024-03-04T09:28:38.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:38.367+0000 7fc6c4a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc6c00cec90 0x7fc6c00b48a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:38.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:38.367+0000 7fc6c4a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc6c00cec90 0x7fc6c00b48a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53892/0 (socket says 172.21.15.6:53892) 2024-03-04T09:28:38.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:38.367+0000 7fc6c4a00640 1 -- 172.21.15.6:0/1095479934 learned_addr learned my addr 172.21.15.6:0/1095479934 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:28:38.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:38.367+0000 7fc6bfe00640 1 --2- 172.21.15.6:0/1095479934 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc6c015df80 0x7fc6c015ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:38.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:38.367+0000 7fc6c4a00640 1 -- 172.21.15.6:0/1095479934 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc6c015df80 msgr2=0x7fc6c015ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:38.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:38.367+0000 7fc6c4a00640 1 --2- 172.21.15.6:0/1095479934 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc6c015df80 0x7fc6c015ef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:38.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:38.367+0000 7fc6c4a00640 1 -- 172.21.15.6:0/1095479934 >> v1:172.21.15.146:6789/0 conn(0x7fc6c00ce810 legacy=0x7fc6c00d0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:38.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:38.367+0000 7fc6c4a00640 1 -- 172.21.15.6:0/1095479934 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6c00cf9e0 con 0x7fc6c00cec90 2024-03-04T09:28:38.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:38.369+0000 7fc6c4a00640 1 --2- 172.21.15.6:0/1095479934 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc6c00cec90 0x7fc6c00b48a0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7fc6b8009870 tx=0x7fc6b802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42ca3330c6a8bad server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:38.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:38.369+0000 7fc6bea00640 1 -- 172.21.15.6:0/1095479934 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6b8035020 con 0x7fc6c00cec90 2024-03-04T09:28:38.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:38.369+0000 7fc6bea00640 1 -- 172.21.15.6:0/1095479934 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6b8031640 con 0x7fc6c00cec90 2024-03-04T09:28:38.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:38.369+0000 7fc6bea00640 1 -- 172.21.15.6:0/1095479934 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6b802d040 con 0x7fc6c00cec90 2024-03-04T09:28:38.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:38.369+0000 7fc6c5400640 1 -- 172.21.15.6:0/1095479934 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc6c00cec90 msgr2=0x7fc6c00b48a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:38.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:38.369+0000 7fc6c5400640 1 --2- 172.21.15.6:0/1095479934 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc6c00cec90 0x7fc6c00b48a0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7fc6b8009870 tx=0x7fc6b802b8f0 comp rx=0 tx=0).stop 2024-03-04T09:28:38.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:38.370+0000 7fc6c5400640 1 -- 172.21.15.6:0/1095479934 shutdown_connections 2024-03-04T09:28:38.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:38.370+0000 7fc6c5400640 1 --2- 172.21.15.6:0/1095479934 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc6c00cec90 0x7fc6c00b48a0 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:38.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:38.370+0000 7fc6c5400640 1 --2- 172.21.15.6:0/1095479934 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc6c015df80 0x7fc6c015ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:38.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:38.370+0000 7fc6c5400640 1 -- 172.21.15.6:0/1095479934 >> 172.21.15.6:0/1095479934 conn(0x7fc6c008d2a0 msgr2=0x7fc6c00d3c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:38.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:38.370+0000 7fc6c5400640 1 -- 172.21.15.6:0/1095479934 shutdown_connections 2024-03-04T09:28:38.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:38.370+0000 7fc6c5400640 1 -- 172.21.15.6:0/1095479934 wait complete. 2024-03-04T09:28:38.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:38.371+0000 7fc6c5400640 1 Processor -- start 2024-03-04T09:28:38.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:38.372+0000 7fc6c5400640 1 -- start start 2024-03-04T09:28:38.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:38.372+0000 7fc6c5400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc6c00ce810 0x7fc6c00b6bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:38.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:38.372+0000 7fc6c5400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc6c00cec90 0x7fc6c00b7100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:38.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:38.372+0000 7fc6c5400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc6c015df80 0x7fc6c00c96d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:38.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:38.372+0000 7fc6c5400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6c00d0ed0 con 0x7fc6c00ce810 2024-03-04T09:28:38.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:38.372+0000 7fc6c5400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6c00d0d50 con 0x7fc6c00cec90 2024-03-04T09:28:38.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:38.372+0000 7fc6c5400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6c00b4de0 con 0x7fc6c015df80 2024-03-04T09:28:38.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:38.373+0000 7fc6bfe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc6c00ce810 0x7fc6c00b6bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:38.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:38.373+0000 7fc6bfe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc6c00ce810 0x7fc6c00b6bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53900/0 (socket says 172.21.15.6:53900) 2024-03-04T09:28:38.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:38.373+0000 7fc6bfe00640 1 -- 172.21.15.6:0/1329086540 learned_addr learned my addr 172.21.15.6:0/1329086540 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:28:38.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:38.373+0000 7fc6bf400640 1 --2- 172.21.15.6:0/1329086540 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc6c00cec90 0x7fc6c00b7100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:38.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:38.373+0000 7fc6c4a00640 1 --2- 172.21.15.6:0/1329086540 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc6c015df80 0x7fc6c00c96d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:38.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:38.373+0000 7fc6bfe00640 1 -- 172.21.15.6:0/1329086540 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc6c015df80 msgr2=0x7fc6c00c96d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:38.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:38.373+0000 7fc6bfe00640 1 --2- 172.21.15.6:0/1329086540 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc6c015df80 0x7fc6c00c96d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:38.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:38.373+0000 7fc6bfe00640 1 -- 172.21.15.6:0/1329086540 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc6c00cec90 msgr2=0x7fc6c00b7100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:38.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:38.373+0000 7fc6bfe00640 1 --2- 172.21.15.6:0/1329086540 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc6c00cec90 0x7fc6c00b7100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:38.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:38.373+0000 7fc6bfe00640 1 -- 172.21.15.6:0/1329086540 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6c00b7960 con 0x7fc6c00ce810 2024-03-04T09:28:38.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:38.373+0000 7fc6bf400640 1 --2- 172.21.15.6:0/1329086540 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc6c00cec90 0x7fc6c00b7100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:28:38.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:38.373+0000 7fc6bfe00640 1 --2- 172.21.15.6:0/1329086540 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc6c00ce810 0x7fc6c00b6bc0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7fc6b0007e40 tx=0x7fc6b000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:38.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:38.374+0000 7fc6b7e00640 1 -- 172.21.15.6:0/1329086540 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6b0014620 con 0x7fc6c00ce810 2024-03-04T09:28:38.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:38.374+0000 7fc6b7e00640 1 -- 172.21.15.6:0/1329086540 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6b0019070 con 0x7fc6c00ce810 2024-03-04T09:28:38.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:38.374+0000 7fc6b7e00640 1 -- 172.21.15.6:0/1329086540 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6b0014a00 con 0x7fc6c00ce810 2024-03-04T09:28:38.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:38.374+0000 7fc6c5400640 1 -- 172.21.15.6:0/1329086540 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6c00b7c40 con 0x7fc6c00ce810 2024-03-04T09:28:38.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:38.374+0000 7fc6c5400640 1 -- 172.21.15.6:0/1329086540 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6c00c91b0 con 0x7fc6c00ce810 2024-03-04T09:28:38.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:38.375+0000 7fc6c5400640 1 -- 172.21.15.6:0/1329086540 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6c004f330 con 0x7fc6c00ce810 2024-03-04T09:28:38.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:38.376+0000 7fc6b7e00640 1 -- 172.21.15.6:0/1329086540 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc6b0014ba0 con 0x7fc6c00ce810 2024-03-04T09:28:38.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:38.377+0000 7fc6b7e00640 1 --2- 172.21.15.6:0/1329086540 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc6a006b4d0 0x7fc6a006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:38.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:38.377+0000 7fc6b7e00640 1 -- 172.21.15.6:0/1329086540 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5357+0+0 (secure 0 0 0) 0x7fc6b008a950 con 0x7fc6c00ce810 2024-03-04T09:28:38.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:38.378+0000 7fc6bf400640 1 --2- 172.21.15.6:0/1329086540 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc6a006b4d0 0x7fc6a006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:38.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:38.379+0000 7fc6bf400640 1 --2- 172.21.15.6:0/1329086540 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc6a006b4d0 0x7fc6a006d990 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fc6c0068520 tx=0x7fc6ac009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:38.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:38.381+0000 7fc6b7e00640 1 -- 172.21.15.6:0/1329086540 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc6b0059920 con 0x7fc6c00ce810 2024-03-04T09:28:38.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:38.593+0000 7fc6c5400640 1 -- 172.21.15.6:0/1329086540 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fc6c00b9af0 con 0x7fc6c00ce810 2024-03-04T09:28:39.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.117+0000 7fc6b7e00640 1 -- 172.21.15.6:0/1329086540 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v59) v1 ==== 169+0+0 (secure 0 0 0) 0x7fc6b0059220 con 0x7fc6c00ce810 2024-03-04T09:28:39.115 INFO:teuthology.orchestra.run.smithi006.stderr:pool 'cephfs_data' removed 2024-03-04T09:28:39.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.124+0000 7fc6c5400640 1 -- 172.21.15.6:0/1329086540 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc6a006b4d0 msgr2=0x7fc6a006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:39.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.124+0000 7fc6c5400640 1 --2- 172.21.15.6:0/1329086540 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc6a006b4d0 0x7fc6a006d990 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fc6c0068520 tx=0x7fc6ac009040 comp rx=0 tx=0).stop 2024-03-04T09:28:39.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.124+0000 7fc6c5400640 1 -- 172.21.15.6:0/1329086540 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc6c00ce810 msgr2=0x7fc6c00b6bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:39.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.124+0000 7fc6c5400640 1 --2- 172.21.15.6:0/1329086540 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc6c00ce810 0x7fc6c00b6bc0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7fc6b0007e40 tx=0x7fc6b000e4b0 comp rx=0 tx=0).stop 2024-03-04T09:28:39.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.124+0000 7fc6c5400640 1 -- 172.21.15.6:0/1329086540 shutdown_connections 2024-03-04T09:28:39.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.124+0000 7fc6c5400640 1 --2- 172.21.15.6:0/1329086540 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc6a006b4d0 0x7fc6a006d990 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:39.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.124+0000 7fc6c5400640 1 --2- 172.21.15.6:0/1329086540 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc6c015df80 0x7fc6c00c96d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:39.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.124+0000 7fc6c5400640 1 --2- 172.21.15.6:0/1329086540 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc6c00cec90 0x7fc6c00b7100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:39.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.124+0000 7fc6c5400640 1 --2- 172.21.15.6:0/1329086540 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc6c00ce810 0x7fc6c00b6bc0 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:39.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.124+0000 7fc6c5400640 1 -- 172.21.15.6:0/1329086540 >> 172.21.15.6:0/1329086540 conn(0x7fc6c008d2a0 msgr2=0x7fc6c006bd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:39.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.125+0000 7fc6c5400640 1 -- 172.21.15.6:0/1329086540 shutdown_connections 2024-03-04T09:28:39.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.126+0000 7fc6c5400640 1 -- 172.21.15.6:0/1329086540 wait complete. 2024-03-04T09:28:39.150 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds mds_bal_merge_size 2024-03-04T09:28:39.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.350+0000 7f1733200640 1 Processor -- start 2024-03-04T09:28:39.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.350+0000 7f1733200640 1 -- start start 2024-03-04T09:28:39.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.351+0000 7f1733200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f17340b2250 0x7f17340b4a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:39.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.352+0000 7f1733200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1734166cb0 con 0x7f1734165d50 2024-03-04T09:28:39.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.352+0000 7f1733200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1734166410 con 0x7f1734162280 2024-03-04T09:28:39.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.352+0000 7f1733200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17340b4fd0 con 0x7f17340b2250 2024-03-04T09:28:39.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.352+0000 7f1731400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f1734165d50 0x7f17340b1b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:60946/0 (socket says 172.21.15.6:60946) 2024-03-04T09:28:39.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.352+0000 7f1731400640 1 -- 172.21.15.6:0/759889620 learned_addr learned my addr 172.21.15.6:0/759889620 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:28:39.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.352+0000 7f1732800640 1 --2- 172.21.15.6:0/759889620 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f17340b2250 0x7f17340b4a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:39.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.352+0000 7f1730a00640 1 -- 172.21.15.6:0/759889620 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 48858720 0 0) 0x7f1734166cb0 con 0x7f1734165d50 2024-03-04T09:28:39.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.352+0000 7f1730a00640 1 -- 172.21.15.6:0/759889620 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1718003520 con 0x7f1734165d50 2024-03-04T09:28:39.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.353+0000 7f1730a00640 1 -- 172.21.15.6:0/759889620 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1455077783 0 0) 0x7f1718003520 con 0x7f1734165d50 2024-03-04T09:28:39.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.353+0000 7f1730a00640 1 -- 172.21.15.6:0/759889620 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f17340b2250 msgr2=0x7f17340b4a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:39.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.353+0000 7f1730a00640 1 --2- 172.21.15.6:0/759889620 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f17340b2250 0x7f17340b4a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:39.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.353+0000 7f1730a00640 1 -- 172.21.15.6:0/759889620 >> v1:172.21.15.146:6789/0 conn(0x7f1734162280 legacy=0x7f17340cc430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:39.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.353+0000 7f1730a00640 1 -- 172.21.15.6:0/759889620 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17341666f0 con 0x7f1734165d50 2024-03-04T09:28:39.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.353+0000 7f1730a00640 1 -- 172.21.15.6:0/759889620 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7f1720001670 con 0x7f1734165d50 2024-03-04T09:28:39.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.353+0000 7f1732800640 1 --2- 172.21.15.6:0/759889620 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f17340b2250 0x7f17340b4a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T09:28:39.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.353+0000 7f1730a00640 1 -- 172.21.15.6:0/759889620 >> v1:172.21.15.6:6789/0 conn(0x7f1734165d50 legacy=0x7f17340b1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:39.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.353+0000 7f1730a00640 1 --2- 172.21.15.6:0/759889620 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f17340b2250 0x7f17180039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:39.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.353+0000 7f1730a00640 1 --2- 172.21.15.6:0/759889620 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1718004020 0x7f1718004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:39.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.353+0000 7f1730a00640 1 --2- 172.21.15.6:0/759889620 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1718004d40 0x7f17180051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:39.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.353+0000 7f1730a00640 1 -- 172.21.15.6:0/759889620 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17340b4fd0 con 0x7f1718004d40 2024-03-04T09:28:39.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.353+0000 7f1730a00640 1 -- 172.21.15.6:0/759889620 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f171800bc00 con 0x7f1718004020 2024-03-04T09:28:39.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.353+0000 7f1730a00640 1 -- 172.21.15.6:0/759889620 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f171800be00 con 0x7f17340b2250 2024-03-04T09:28:39.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.353+0000 7f1732800640 1 --2- 172.21.15.6:0/759889620 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1718004d40 0x7f17180051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:39.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.354+0000 7f1731e00640 1 --2- 172.21.15.6:0/759889620 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f17340b2250 0x7f17180039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:39.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.354+0000 7f1732800640 1 -- 172.21.15.6:0/759889620 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f17340b2250 msgr2=0x7f17180039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:39.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.354+0000 7f1731400640 1 --2- 172.21.15.6:0/759889620 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1718004020 0x7f1718004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:39.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.354+0000 7f1732800640 1 --2- 172.21.15.6:0/759889620 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f17340b2250 0x7f17180039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:39.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.354+0000 7f1732800640 1 -- 172.21.15.6:0/759889620 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1718004020 msgr2=0x7f1718004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:39.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.354+0000 7f1732800640 1 --2- 172.21.15.6:0/759889620 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1718004020 0x7f1718004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:39.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.354+0000 7f1732800640 1 -- 172.21.15.6:0/759889620 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1718003520 con 0x7f1718004d40 2024-03-04T09:28:39.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.354+0000 7f1731400640 1 --2- 172.21.15.6:0/759889620 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1718004020 0x7f1718004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:28:39.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.355+0000 7f1732800640 1 --2- 172.21.15.6:0/759889620 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1718004d40 0x7f17180051c0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f172c0073b0 tx=0x7f1718005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d01e16cf8e7a891 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:39.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.356+0000 7f1730a00640 1 -- 172.21.15.6:0/759889620 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f172c039070 con 0x7f1718004d40 2024-03-04T09:28:39.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.356+0000 7f1730a00640 1 -- 172.21.15.6:0/759889620 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f172c02cd30 con 0x7f1718004d40 2024-03-04T09:28:39.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.356+0000 7f1730a00640 1 -- 172.21.15.6:0/759889620 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f172c009c80 con 0x7f1718004d40 2024-03-04T09:28:39.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.356+0000 7f1733200640 1 -- 172.21.15.6:0/759889620 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1718004d40 msgr2=0x7f17180051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:39.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.356+0000 7f1733200640 1 --2- 172.21.15.6:0/759889620 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1718004d40 0x7f17180051c0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f172c0073b0 tx=0x7f1718005830 comp rx=0 tx=0).stop 2024-03-04T09:28:39.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.356+0000 7f1732800640 1 -- 172.21.15.6:0/759889620 reap_dead start 2024-03-04T09:28:39.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.356+0000 7f1733200640 1 -- 172.21.15.6:0/759889620 shutdown_connections 2024-03-04T09:28:39.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.356+0000 7f1733200640 1 -- 172.21.15.6:0/759889620 >> 172.21.15.6:0/759889620 conn(0x7f173406b3a0 msgr2=0x7f173406b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:39.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.356+0000 7f1733200640 1 -- 172.21.15.6:0/759889620 shutdown_connections 2024-03-04T09:28:39.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.356+0000 7f1733200640 1 -- 172.21.15.6:0/759889620 wait complete. 2024-03-04T09:28:39.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.358+0000 7f1733200640 1 Processor -- start 2024-03-04T09:28:39.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.358+0000 7f1733200640 1 -- start start 2024-03-04T09:28:39.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.359+0000 7f1733200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1718004d40 0x7f17341f5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:39.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.359+0000 7f1733200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f17341f89c0 0x7f173415f900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:39.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.359+0000 7f1733200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f17341f8ee0 0x7f173415ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:39.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.359+0000 7f1733200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17340b4a90 con 0x7f17341f8ee0 2024-03-04T09:28:39.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.359+0000 7f1733200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17341661b0 con 0x7f1718004d40 2024-03-04T09:28:39.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.359+0000 7f1733200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17340b4c10 con 0x7f17341f89c0 2024-03-04T09:28:39.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.359+0000 7f1732800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f17341f8ee0 0x7f173415ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:39.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.359+0000 7f1732800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f17341f8ee0 0x7f173415ecd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:44114/0 (socket says 172.21.15.6:44114) 2024-03-04T09:28:39.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.359+0000 7f1732800640 1 -- 172.21.15.6:0/4001887566 learned_addr learned my addr 172.21.15.6:0/4001887566 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:28:39.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.360+0000 7f1731400640 1 --2- 172.21.15.6:0/4001887566 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f17341f89c0 0x7f173415f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:39.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.360+0000 7f1732800640 1 -- 172.21.15.6:0/4001887566 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f17341f89c0 msgr2=0x7f173415f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:39.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.360+0000 7f1731e00640 1 --2- 172.21.15.6:0/4001887566 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1718004d40 0x7f17341f5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:39.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.360+0000 7f1732800640 1 --2- 172.21.15.6:0/4001887566 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f17341f89c0 0x7f173415f900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:39.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.360+0000 7f1732800640 1 -- 172.21.15.6:0/4001887566 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1718004d40 msgr2=0x7f17341f5170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:39.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.360+0000 7f1732800640 1 --2- 172.21.15.6:0/4001887566 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1718004d40 0x7f17341f5170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:39.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.360+0000 7f1732800640 1 -- 172.21.15.6:0/4001887566 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17341601c0 con 0x7f17341f8ee0 2024-03-04T09:28:39.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.360+0000 7f1731e00640 1 --2- 172.21.15.6:0/4001887566 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1718004d40 0x7f17341f5170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:28:39.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.360+0000 7f1732800640 1 --2- 172.21.15.6:0/4001887566 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f17341f8ee0 0x7f173415ecd0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f172c0073b0 tx=0x7f172c004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:39.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.360+0000 7f172a000640 1 -- 172.21.15.6:0/4001887566 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f172c039030 con 0x7f17341f8ee0 2024-03-04T09:28:39.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.361+0000 7f172a000640 1 -- 172.21.15.6:0/4001887566 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f172c03a040 con 0x7f17341f8ee0 2024-03-04T09:28:39.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.361+0000 7f1733200640 1 -- 172.21.15.6:0/4001887566 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f173415e1f0 con 0x7f17341f8ee0 2024-03-04T09:28:39.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.361+0000 7f172a000640 1 -- 172.21.15.6:0/4001887566 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f172c009c80 con 0x7f17341f8ee0 2024-03-04T09:28:39.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.361+0000 7f1733200640 1 -- 172.21.15.6:0/4001887566 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f173415e7b0 con 0x7f17341f8ee0 2024-03-04T09:28:39.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.362+0000 7f1733200640 1 -- 172.21.15.6:0/4001887566 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f173404f330 con 0x7f17341f8ee0 2024-03-04T09:28:39.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.363+0000 7f172a000640 1 -- 172.21.15.6:0/4001887566 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f172c045020 con 0x7f17341f8ee0 2024-03-04T09:28:39.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.364+0000 7f172a000640 1 --2- 172.21.15.6:0/4001887566 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f171406b4d0 0x7f171406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:39.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.364+0000 7f172a000640 1 -- 172.21.15.6:0/4001887566 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f172c0ab7e0 con 0x7f17341f8ee0 2024-03-04T09:28:39.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.365+0000 7f1731e00640 1 --2- 172.21.15.6:0/4001887566 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f171406b4d0 0x7f171406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:39.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.366+0000 7f1731e00640 1 --2- 172.21.15.6:0/4001887566 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f171406b4d0 0x7f171406d990 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f17340d0110 tx=0x7f1724009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:39.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.368+0000 7f172a000640 1 -- 172.21.15.6:0/4001887566 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f172c07a8c0 con 0x7f17341f8ee0 2024-03-04T09:28:39.430 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:28:39.438 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:28:39.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.571+0000 7f1733200640 1 -- 172.21.15.6:0/4001887566 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "config rm", "who": "mds", "name": "mds_bal_merge_size"} v 0) v1 -- 0x7f173415e980 con 0x7f17341f8ee0 2024-03-04T09:28:39.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.575+0000 7f172a000640 1 -- 172.21.15.6:0/4001887566 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds", "name": "mds_bal_merge_size"}]=0 v19) v1 ==== 101+0+0 (secure 0 0 0) 0x7f172c07a1c0 con 0x7f17341f8ee0 2024-03-04T09:28:39.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.578+0000 7f1733200640 1 -- 172.21.15.6:0/4001887566 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f171406b4d0 msgr2=0x7f171406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:39.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.578+0000 7f1733200640 1 --2- 172.21.15.6:0/4001887566 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f171406b4d0 0x7f171406d990 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f17340d0110 tx=0x7f1724009040 comp rx=0 tx=0).stop 2024-03-04T09:28:39.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.578+0000 7f1733200640 1 -- 172.21.15.6:0/4001887566 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f17341f8ee0 msgr2=0x7f173415ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:39.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.578+0000 7f1733200640 1 --2- 172.21.15.6:0/4001887566 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f17341f8ee0 0x7f173415ecd0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f172c0073b0 tx=0x7f172c004ab0 comp rx=0 tx=0).stop 2024-03-04T09:28:39.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.578+0000 7f1733200640 1 -- 172.21.15.6:0/4001887566 shutdown_connections 2024-03-04T09:28:39.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.578+0000 7f1733200640 1 --2- 172.21.15.6:0/4001887566 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f17341f8ee0 0x7f173415ecd0 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:39.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.578+0000 7f1733200640 1 --2- 172.21.15.6:0/4001887566 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f171406b4d0 0x7f171406d990 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:39.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.578+0000 7f1733200640 1 --2- 172.21.15.6:0/4001887566 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f17341f89c0 0x7f173415f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:39.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.578+0000 7f1733200640 1 --2- 172.21.15.6:0/4001887566 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1718004d40 0x7f17341f5170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:39.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.578+0000 7f1733200640 1 -- 172.21.15.6:0/4001887566 >> 172.21.15.6:0/4001887566 conn(0x7f173406b3a0 msgr2=0x7f1734161460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:39.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.578+0000 7f1733200640 1 -- 172.21.15.6:0/4001887566 shutdown_connections 2024-03-04T09:28:39.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.578+0000 7f1733200640 1 -- 172.21.15.6:0/4001887566 wait complete. 2024-03-04T09:28:39.595 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds mds_bal_fragment_size_max 2024-03-04T09:28:39.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.864+0000 7f1850a00640 1 Processor -- start 2024-03-04T09:28:39.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.864+0000 7f1850a00640 1 -- start start 2024-03-04T09:28:39.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.866+0000 7f1850a00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f184c147590 0x7f184c071480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:39.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.866+0000 7f1850a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f184c0b2250 0x7f184c0b3640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:39.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.866+0000 7f1850a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f184c0719c0 con 0x7f184c06d8d0 2024-03-04T09:28:39.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.866+0000 7f1850a00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f184c147970 con 0x7f184c0b2250 2024-03-04T09:28:39.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.866+0000 7f1850a00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f184c0b31b0 con 0x7f184c147590 2024-03-04T09:28:39.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.866+0000 7f184b200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f184c0b2250 0x7f184c0b3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:39.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.866+0000 7f1849e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f184c06d8d0 0x7f184c0b1b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:60952/0 (socket says 172.21.15.6:60952) 2024-03-04T09:28:39.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.866+0000 7f184b200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f184c0b2250 0x7f184c0b3640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:44126/0 (socket says 172.21.15.6:44126) 2024-03-04T09:28:39.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.866+0000 7f1849e00640 1 -- 172.21.15.6:0/3661445487 learned_addr learned my addr 172.21.15.6:0/3661445487 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:28:39.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.867+0000 7f184a800640 1 --2- 172.21.15.6:0/3661445487 >> v2:172.21.15.146:3300/0 conn(0x7f184c147590 0x7f184c071480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:39.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.867+0000 7f184b200640 1 -- 172.21.15.6:0/3661445487 >> v2:172.21.15.146:3300/0 conn(0x7f184c147590 msgr2=0x7f184c071480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:39.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.867+0000 7f184b200640 1 --2- 172.21.15.6:0/3661445487 >> v2:172.21.15.146:3300/0 conn(0x7f184c147590 0x7f184c071480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:39.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.867+0000 7f184b200640 1 -- 172.21.15.6:0/3661445487 >> v1:172.21.15.6:6789/0 conn(0x7f184c06d8d0 legacy=0x7f184c0b1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:39.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.867+0000 7f184b200640 1 -- 172.21.15.6:0/3661445487 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f184c06de40 con 0x7f184c0b2250 2024-03-04T09:28:39.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.868+0000 7f184b200640 1 --2- 172.21.15.6:0/3661445487 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f184c0b2250 0x7f184c0b3640 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f184c0b2790 tx=0x7f184402bec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=401f6f5c9f13ce9c server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:39.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.869+0000 7f1849400640 1 -- 172.21.15.6:0/3661445487 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1844039070 con 0x7f184c0b2250 2024-03-04T09:28:39.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.869+0000 7f1849400640 1 -- 172.21.15.6:0/3661445487 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f184402cc70 con 0x7f184c0b2250 2024-03-04T09:28:39.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.869+0000 7f1849400640 1 -- 172.21.15.6:0/3661445487 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1844034660 con 0x7f184c0b2250 2024-03-04T09:28:39.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.869+0000 7f1850a00640 1 -- 172.21.15.6:0/3661445487 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f184c0b2250 msgr2=0x7f184c0b3640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:39.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.869+0000 7f1850a00640 1 --2- 172.21.15.6:0/3661445487 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f184c0b2250 0x7f184c0b3640 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f184c0b2790 tx=0x7f184402bec0 comp rx=0 tx=0).stop 2024-03-04T09:28:39.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.869+0000 7f1850a00640 1 -- 172.21.15.6:0/3661445487 shutdown_connections 2024-03-04T09:28:39.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.869+0000 7f1850a00640 1 --2- 172.21.15.6:0/3661445487 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f184c0b2250 0x7f184c0b3640 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:39.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.869+0000 7f1850a00640 1 --2- 172.21.15.6:0/3661445487 >> v2:172.21.15.146:3300/0 conn(0x7f184c147590 0x7f184c071480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:39.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.869+0000 7f1850a00640 1 -- 172.21.15.6:0/3661445487 >> 172.21.15.6:0/3661445487 conn(0x7f184c06b3a0 msgr2=0x7f184c06b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:39.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.869+0000 7f1850a00640 1 -- 172.21.15.6:0/3661445487 shutdown_connections 2024-03-04T09:28:39.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.870+0000 7f1850a00640 1 -- 172.21.15.6:0/3661445487 wait complete. 2024-03-04T09:28:39.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.871+0000 7f1850a00640 1 Processor -- start 2024-03-04T09:28:39.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.871+0000 7f1850a00640 1 -- start start 2024-03-04T09:28:39.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.872+0000 7f1850a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f184c06d8d0 0x7f184c1e8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:39.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.872+0000 7f1850a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f184c0b2250 0x7f184c1e7f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:39.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.872+0000 7f1850a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f184c147590 0x7f184c1e63b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:39.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.872+0000 7f1850a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f184c0b2b50 con 0x7f184c147590 2024-03-04T09:28:39.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.872+0000 7f1850a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f184c0b2e40 con 0x7f184c0b2250 2024-03-04T09:28:39.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.872+0000 7f1850a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f184c0b3b80 con 0x7f184c06d8d0 2024-03-04T09:28:39.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.873+0000 7f184b200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f184c147590 0x7f184c1e63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:39.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.873+0000 7f184b200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f184c147590 0x7f184c1e63b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:44128/0 (socket says 172.21.15.6:44128) 2024-03-04T09:28:39.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.873+0000 7f184b200640 1 -- 172.21.15.6:0/3896609494 learned_addr learned my addr 172.21.15.6:0/3896609494 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:28:39.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.873+0000 7f1849e00640 1 --2- 172.21.15.6:0/3896609494 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f184c0b2250 0x7f184c1e7f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:39.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.873+0000 7f184a800640 1 --2- 172.21.15.6:0/3896609494 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f184c06d8d0 0x7f184c1e8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:39.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.873+0000 7f184b200640 1 -- 172.21.15.6:0/3896609494 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f184c06d8d0 msgr2=0x7f184c1e8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:39.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.873+0000 7f184b200640 1 --2- 172.21.15.6:0/3896609494 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f184c06d8d0 0x7f184c1e8b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:39.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.873+0000 7f184b200640 1 -- 172.21.15.6:0/3896609494 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f184c0b2250 msgr2=0x7f184c1e7f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:39.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.873+0000 7f184b200640 1 --2- 172.21.15.6:0/3896609494 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f184c0b2250 0x7f184c1e7f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:39.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.873+0000 7f184b200640 1 -- 172.21.15.6:0/3896609494 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f184c1e6e70 con 0x7f184c147590 2024-03-04T09:28:39.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.873+0000 7f184a800640 1 --2- 172.21.15.6:0/3896609494 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f184c06d8d0 0x7f184c1e8b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:28:39.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.873+0000 7f184b200640 1 --2- 172.21.15.6:0/3896609494 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f184c147590 0x7f184c1e63b0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f184402c4d0 tx=0x7f1844004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:39.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.874+0000 7f1842a00640 1 -- 172.21.15.6:0/3896609494 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1844039050 con 0x7f184c147590 2024-03-04T09:28:39.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.874+0000 7f1842a00640 1 -- 172.21.15.6:0/3896609494 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1844045070 con 0x7f184c147590 2024-03-04T09:28:39.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.874+0000 7f1842a00640 1 -- 172.21.15.6:0/3896609494 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f184403e830 con 0x7f184c147590 2024-03-04T09:28:39.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.874+0000 7f1850a00640 1 -- 172.21.15.6:0/3896609494 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f184c1e7150 con 0x7f184c147590 2024-03-04T09:28:39.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.874+0000 7f1850a00640 1 -- 172.21.15.6:0/3896609494 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f184c1e7710 con 0x7f184c147590 2024-03-04T09:28:39.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.875+0000 7f1850a00640 1 -- 172.21.15.6:0/3896609494 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f184c071070 con 0x7f184c147590 2024-03-04T09:28:39.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.876+0000 7f1842a00640 1 -- 172.21.15.6:0/3896609494 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f184403e9d0 con 0x7f184c147590 2024-03-04T09:28:39.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.877+0000 7f1842a00640 1 --2- 172.21.15.6:0/3896609494 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f182c06b4d0 0x7f182c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:39.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.877+0000 7f1842a00640 1 -- 172.21.15.6:0/3896609494 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f18440acca0 con 0x7f184c147590 2024-03-04T09:28:39.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.878+0000 7f184a800640 1 --2- 172.21.15.6:0/3896609494 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f182c06b4d0 0x7f182c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:39.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.879+0000 7f184a800640 1 --2- 172.21.15.6:0/3896609494 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f182c06b4d0 0x7f182c06d990 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f183c006fd0 tx=0x7f183c008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:39.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:39.882+0000 7f1842a00640 1 -- 172.21.15.6:0/3896609494 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f184c071070 con 0x7f184c147590 2024-03-04T09:28:40.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.085+0000 7f1850a00640 1 -- 172.21.15.6:0/3896609494 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "config rm", "who": "mds", "name": "mds_bal_fragment_size_max"} v 0) v1 -- 0x7f184c04f330 con 0x7f184c147590 2024-03-04T09:28:40.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.092+0000 7f1842a00640 1 -- 172.21.15.6:0/3896609494 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds", "name": "mds_bal_fragment_size_max"}]=0 v20) v1 ==== 108+0+0 (secure 0 0 0) 0x7f184402ce20 con 0x7f184c147590 2024-03-04T09:28:40.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.096+0000 7f1850a00640 1 -- 172.21.15.6:0/3896609494 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f182c06b4d0 msgr2=0x7f182c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:40.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.096+0000 7f1850a00640 1 --2- 172.21.15.6:0/3896609494 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f182c06b4d0 0x7f182c06d990 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f183c006fd0 tx=0x7f183c008040 comp rx=0 tx=0).stop 2024-03-04T09:28:40.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.096+0000 7f1850a00640 1 -- 172.21.15.6:0/3896609494 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f184c147590 msgr2=0x7f184c1e63b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:40.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.096+0000 7f1850a00640 1 --2- 172.21.15.6:0/3896609494 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f184c147590 0x7f184c1e63b0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f184402c4d0 tx=0x7f1844004a40 comp rx=0 tx=0).stop 2024-03-04T09:28:40.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.096+0000 7f1850a00640 1 -- 172.21.15.6:0/3896609494 shutdown_connections 2024-03-04T09:28:40.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.096+0000 7f1850a00640 1 --2- 172.21.15.6:0/3896609494 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f184c147590 0x7f184c1e63b0 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:40.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.096+0000 7f1850a00640 1 --2- 172.21.15.6:0/3896609494 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f184c0b2250 0x7f184c1e7f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:40.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.096+0000 7f1850a00640 1 --2- 172.21.15.6:0/3896609494 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f182c06b4d0 0x7f182c06d990 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:40.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.096+0000 7f1850a00640 1 --2- 172.21.15.6:0/3896609494 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f184c06d8d0 0x7f184c1e8b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:40.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.096+0000 7f1850a00640 1 -- 172.21.15.6:0/3896609494 >> 172.21.15.6:0/3896609494 conn(0x7f184c06b3a0 msgr2=0x7f184c15fe70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:40.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.097+0000 7f1850a00640 1 -- 172.21.15.6:0/3896609494 shutdown_connections 2024-03-04T09:28:40.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.097+0000 7f1850a00640 1 -- 172.21.15.6:0/3896609494 wait complete. 2024-03-04T09:28:40.115 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-04T09:28:40.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.342+0000 7f0d00a00640 1 Processor -- start 2024-03-04T09:28:40.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.342+0000 7f0d00a00640 1 -- start start 2024-03-04T09:28:40.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.344+0000 7f0d00a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0cfc15e680 0x7f0cfc06c980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:40.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.344+0000 7f0d00a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f0cfc0ccad0 0x7f0cfc0ccf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:40.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.344+0000 7f0d00a00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cfc15eb80 con 0x7f0cfc06da40 2024-03-04T09:28:40.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.344+0000 7f0d00a00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cfc06ea50 con 0x7f0cfc0ccad0 2024-03-04T09:28:40.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.344+0000 7f0d00a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cfc161c70 con 0x7f0cfc15e680 2024-03-04T09:28:40.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.345+0000 7f0cfa800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f0cfc0ccad0 0x7f0cfc0ccf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:40.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.345+0000 7f0cfa800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f0cfc0ccad0 0x7f0cfc0ccf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:44144/0 (socket says 172.21.15.6:44144) 2024-03-04T09:28:40.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.345+0000 7f0cfa800640 1 -- 172.21.15.6:0/369825679 learned_addr learned my addr 172.21.15.6:0/369825679 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:28:40.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.345+0000 7f0cf9400640 1 --2- 172.21.15.6:0/369825679 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0cfc15e680 0x7f0cfc06c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:40.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.345+0000 7f0cfa800640 1 -- 172.21.15.6:0/369825679 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0cfc15e680 msgr2=0x7f0cfc06c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:40.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.345+0000 7f0cfa800640 1 --2- 172.21.15.6:0/369825679 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0cfc15e680 0x7f0cfc06c980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:40.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.345+0000 7f0cfa800640 1 -- 172.21.15.6:0/369825679 >> v1:172.21.15.146:6789/0 conn(0x7f0cfc06da40 legacy=0x7f0cfc06e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:40.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.345+0000 7f0cfa800640 1 -- 172.21.15.6:0/369825679 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cfc15ee60 con 0x7f0cfc0ccad0 2024-03-04T09:28:40.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.346+0000 7f0cfa800640 1 --2- 172.21.15.6:0/369825679 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0cfc0ccad0 0x7f0cfc0ccf80 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f0cf4009e50 tx=0x7f0cf402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d6e78e809d1e3b server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:40.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.347+0000 7f0cf8a00640 1 -- 172.21.15.6:0/369825679 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cf402ca70 con 0x7f0cfc0ccad0 2024-03-04T09:28:40.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.347+0000 7f0cf8a00640 1 -- 172.21.15.6:0/369825679 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cf402cc10 con 0x7f0cfc0ccad0 2024-03-04T09:28:40.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.347+0000 7f0cf8a00640 1 -- 172.21.15.6:0/369825679 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cf40343f0 con 0x7f0cfc0ccad0 2024-03-04T09:28:40.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.347+0000 7f0d00a00640 1 -- 172.21.15.6:0/369825679 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0cfc0ccad0 msgr2=0x7f0cfc0ccf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:40.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.347+0000 7f0d00a00640 1 --2- 172.21.15.6:0/369825679 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0cfc0ccad0 0x7f0cfc0ccf80 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f0cf4009e50 tx=0x7f0cf402bed0 comp rx=0 tx=0).stop 2024-03-04T09:28:40.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.348+0000 7f0d00a00640 1 -- 172.21.15.6:0/369825679 shutdown_connections 2024-03-04T09:28:40.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.348+0000 7f0d00a00640 1 --2- 172.21.15.6:0/369825679 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0cfc0ccad0 0x7f0cfc0ccf80 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:40.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.348+0000 7f0d00a00640 1 --2- 172.21.15.6:0/369825679 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0cfc15e680 0x7f0cfc06c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:40.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.348+0000 7f0d00a00640 1 -- 172.21.15.6:0/369825679 >> 172.21.15.6:0/369825679 conn(0x7f0cfc06b3d0 msgr2=0x7f0cfc06b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:40.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.348+0000 7f0d00a00640 1 -- 172.21.15.6:0/369825679 shutdown_connections 2024-03-04T09:28:40.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.348+0000 7f0d00a00640 1 -- 172.21.15.6:0/369825679 wait complete. 2024-03-04T09:28:40.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.350+0000 7f0d00a00640 1 Processor -- start 2024-03-04T09:28:40.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.350+0000 7f0d00a00640 1 -- start start 2024-03-04T09:28:40.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.351+0000 7f0d00a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0cfc06da40 0x7f0cfc15cf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:40.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.351+0000 7f0d00a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0cfc0ccad0 0x7f0cfc0c8fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:40.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.351+0000 7f0d00a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0cfc15e680 0x7f0cfc0b3cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:40.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.351+0000 7f0d00a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cfc06cec0 con 0x7f0cfc06da40 2024-03-04T09:28:40.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.351+0000 7f0d00a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cfc15f420 con 0x7f0cfc15e680 2024-03-04T09:28:40.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.351+0000 7f0d00a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cfc06d040 con 0x7f0cfc0ccad0 2024-03-04T09:28:40.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.351+0000 7f0cf9e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0cfc06da40 0x7f0cfc15cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:40.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.351+0000 7f0cf9e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0cfc06da40 0x7f0cfc15cf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:44150/0 (socket says 172.21.15.6:44150) 2024-03-04T09:28:40.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.351+0000 7f0cf9e00640 1 -- 172.21.15.6:0/1772922323 learned_addr learned my addr 172.21.15.6:0/1772922323 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:28:40.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.351+0000 7f0cf9400640 1 --2- 172.21.15.6:0/1772922323 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0cfc0ccad0 0x7f0cfc0c8fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:40.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.351+0000 7f0cfa800640 1 --2- 172.21.15.6:0/1772922323 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0cfc15e680 0x7f0cfc0b3cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:40.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.352+0000 7f0cf9e00640 1 -- 172.21.15.6:0/1772922323 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0cfc0ccad0 msgr2=0x7f0cfc0c8fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:40.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.352+0000 7f0cf9e00640 1 --2- 172.21.15.6:0/1772922323 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0cfc0ccad0 0x7f0cfc0c8fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:40.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.352+0000 7f0cf9e00640 1 -- 172.21.15.6:0/1772922323 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0cfc15e680 msgr2=0x7f0cfc0b3cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:40.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.352+0000 7f0cf9e00640 1 --2- 172.21.15.6:0/1772922323 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0cfc15e680 0x7f0cfc0b3cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:40.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.352+0000 7f0cf9e00640 1 -- 172.21.15.6:0/1772922323 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cfc153390 con 0x7f0cfc06da40 2024-03-04T09:28:40.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.352+0000 7f0cf9e00640 1 --2- 172.21.15.6:0/1772922323 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0cfc06da40 0x7f0cfc15cf30 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f0cec00c430 tx=0x7f0cec00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:40.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.352+0000 7f0cf2000640 1 -- 172.21.15.6:0/1772922323 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cec011790 con 0x7f0cfc06da40 2024-03-04T09:28:40.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.352+0000 7f0cf2000640 1 -- 172.21.15.6:0/1772922323 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cec017070 con 0x7f0cfc06da40 2024-03-04T09:28:40.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.352+0000 7f0d00a00640 1 -- 172.21.15.6:0/1772922323 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cfc1525f0 con 0x7f0cfc06da40 2024-03-04T09:28:40.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.352+0000 7f0cf2000640 1 -- 172.21.15.6:0/1772922323 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cec011bb0 con 0x7f0cfc06da40 2024-03-04T09:28:40.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.353+0000 7f0d00a00640 1 -- 172.21.15.6:0/1772922323 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cfc152bb0 con 0x7f0cfc06da40 2024-03-04T09:28:40.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.354+0000 7f0d00a00640 1 -- 172.21.15.6:0/1772922323 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cfc04f330 con 0x7f0cfc06da40 2024-03-04T09:28:40.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.354+0000 7f0cf2000640 1 -- 172.21.15.6:0/1772922323 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0cec01c460 con 0x7f0cfc06da40 2024-03-04T09:28:40.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.355+0000 7f0cf2000640 1 --2- 172.21.15.6:0/1772922323 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f0cdc06b4d0 0x7f0cdc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:40.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.355+0000 7f0cf2000640 1 -- 172.21.15.6:0/1772922323 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f0cec089d70 con 0x7f0cfc06da40 2024-03-04T09:28:40.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.356+0000 7f0cf9400640 1 --2- 172.21.15.6:0/1772922323 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f0cdc06b4d0 0x7f0cdc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:40.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.357+0000 7f0cf9400640 1 --2- 172.21.15.6:0/1772922323 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f0cdc06b4d0 0x7f0cdc06d990 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f0ce8006fd0 tx=0x7f0ce8008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:40.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.360+0000 7f0cf2000640 1 -- 172.21.15.6:0/1772922323 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0cec058f60 con 0x7f0cfc06da40 2024-03-04T09:28:40.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.565+0000 7f0d00a00640 1 -- 172.21.15.6:0/1772922323 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f0cfc0b1280 con 0x7f0cfc06da40 2024-03-04T09:28:40.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.570+0000 7f0cf2000640 1 -- 172.21.15.6:0/1772922323 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v21) v1 ==== 104+0+0 (secure 0 0 0) 0x7f0cec058860 con 0x7f0cfc06da40 2024-03-04T09:28:40.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.574+0000 7f0d00a00640 1 -- 172.21.15.6:0/1772922323 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f0cdc06b4d0 msgr2=0x7f0cdc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:40.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.574+0000 7f0d00a00640 1 --2- 172.21.15.6:0/1772922323 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f0cdc06b4d0 0x7f0cdc06d990 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f0ce8006fd0 tx=0x7f0ce8008040 comp rx=0 tx=0).stop 2024-03-04T09:28:40.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.574+0000 7f0d00a00640 1 -- 172.21.15.6:0/1772922323 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0cfc06da40 msgr2=0x7f0cfc15cf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:40.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.574+0000 7f0d00a00640 1 --2- 172.21.15.6:0/1772922323 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0cfc06da40 0x7f0cfc15cf30 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f0cec00c430 tx=0x7f0cec00c8f0 comp rx=0 tx=0).stop 2024-03-04T09:28:40.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.574+0000 7f0d00a00640 1 -- 172.21.15.6:0/1772922323 shutdown_connections 2024-03-04T09:28:40.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.574+0000 7f0d00a00640 1 --2- 172.21.15.6:0/1772922323 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0cfc15e680 0x7f0cfc0b3cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:40.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.574+0000 7f0d00a00640 1 --2- 172.21.15.6:0/1772922323 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f0cdc06b4d0 0x7f0cdc06d990 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:40.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.574+0000 7f0d00a00640 1 --2- 172.21.15.6:0/1772922323 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0cfc0ccad0 0x7f0cfc0c8fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:40.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.574+0000 7f0d00a00640 1 --2- 172.21.15.6:0/1772922323 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0cfc06da40 0x7f0cfc15cf30 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:40.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.574+0000 7f0d00a00640 1 -- 172.21.15.6:0/1772922323 >> 172.21.15.6:0/1772922323 conn(0x7f0cfc06b3d0 msgr2=0x7f0cfc1624a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:40.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.574+0000 7f0d00a00640 1 -- 172.21.15.6:0/1772922323 shutdown_connections 2024-03-04T09:28:40.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.576+0000 7f0d00a00640 1 -- 172.21.15.6:0/1772922323 wait complete. 2024-03-04T09:28:40.605 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_fragmented' 2024-03-04T09:28:40.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.824+0000 7fc94a400640 1 Processor -- start 2024-03-04T09:28:40.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.825+0000 7fc94a400640 1 -- start start 2024-03-04T09:28:40.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.826+0000 7fc94a400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc944161670 0x7fc944070c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:40.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.826+0000 7fc94a400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc9440ae0a0 0x7fc9440711b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:40.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.826+0000 7fc94a400640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9440ce9a0 con 0x7fc94406e240 2024-03-04T09:28:40.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.826+0000 7fc94a400640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9440730d0 con 0x7fc944161670 2024-03-04T09:28:40.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.826+0000 7fc94a400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9440cec80 con 0x7fc9440ae0a0 2024-03-04T09:28:40.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.827+0000 7fc943400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc944161670 0x7fc944070c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:40.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.827+0000 7fc943400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc944161670 0x7fc944070c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:44152/0 (socket says 172.21.15.6:44152) 2024-03-04T09:28:40.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.827+0000 7fc943400640 1 -- 172.21.15.6:0/2392028670 learned_addr learned my addr 172.21.15.6:0/2392028670 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:28:40.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.827+0000 7fc942a00640 1 --1- 172.21.15.6:0/2392028670 >> v1:172.21.15.6:6789/0 conn(0x7fc94406e240 0x7fc94406cfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:60968/0 (socket says 172.21.15.6:60968) 2024-03-04T09:28:40.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.827+0000 7fc943e00640 1 --2- 172.21.15.6:0/2392028670 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc9440ae0a0 0x7fc9440711b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:40.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.827+0000 7fc943400640 1 -- 172.21.15.6:0/2392028670 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc9440ae0a0 msgr2=0x7fc9440711b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:40.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.827+0000 7fc943400640 1 --2- 172.21.15.6:0/2392028670 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc9440ae0a0 0x7fc9440711b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:40.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.827+0000 7fc943400640 1 -- 172.21.15.6:0/2392028670 >> v1:172.21.15.6:6789/0 conn(0x7fc94406e240 legacy=0x7fc94406cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:40.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.827+0000 7fc943400640 1 -- 172.21.15.6:0/2392028670 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9440cef60 con 0x7fc944161670 2024-03-04T09:28:40.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.827+0000 7fc942000640 1 -- 172.21.15.6:0/2392028670 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3674685653 0 0) 0x7fc9440ce9a0 con 0x7fc94406e240 2024-03-04T09:28:40.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.829+0000 7fc943400640 1 --2- 172.21.15.6:0/2392028670 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc944161670 0x7fc944070c70 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fc92c009870 tx=0x7fc92c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=133cb561205d666c server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:40.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.829+0000 7fc942000640 1 -- 172.21.15.6:0/2392028670 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc92c034070 con 0x7fc944161670 2024-03-04T09:28:40.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.829+0000 7fc942000640 1 -- 172.21.15.6:0/2392028670 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc92c030840 con 0x7fc944161670 2024-03-04T09:28:40.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.829+0000 7fc942000640 1 -- 172.21.15.6:0/2392028670 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc92c030c10 con 0x7fc944161670 2024-03-04T09:28:40.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.829+0000 7fc94a400640 1 -- 172.21.15.6:0/2392028670 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc944161670 msgr2=0x7fc944070c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:40.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.829+0000 7fc94a400640 1 --2- 172.21.15.6:0/2392028670 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc944161670 0x7fc944070c70 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fc92c009870 tx=0x7fc92c02b980 comp rx=0 tx=0).stop 2024-03-04T09:28:40.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.830+0000 7fc94a400640 1 -- 172.21.15.6:0/2392028670 shutdown_connections 2024-03-04T09:28:40.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.830+0000 7fc94a400640 1 --2- 172.21.15.6:0/2392028670 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc9440ae0a0 0x7fc9440711b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:40.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.830+0000 7fc94a400640 1 --2- 172.21.15.6:0/2392028670 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc944161670 0x7fc944070c70 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:40.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.830+0000 7fc94a400640 1 -- 172.21.15.6:0/2392028670 >> 172.21.15.6:0/2392028670 conn(0x7fc94406afb0 msgr2=0x7fc94406b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:40.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.830+0000 7fc94a400640 1 -- 172.21.15.6:0/2392028670 shutdown_connections 2024-03-04T09:28:40.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.830+0000 7fc94a400640 1 -- 172.21.15.6:0/2392028670 wait complete. 2024-03-04T09:28:40.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.831+0000 7fc94a400640 1 Processor -- start 2024-03-04T09:28:40.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.832+0000 7fc94a400640 1 -- start start 2024-03-04T09:28:40.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.833+0000 7fc94a400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc94406e240 0x7fc9440b3ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:40.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.833+0000 7fc94a400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc9440ae0a0 0x7fc94415c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:40.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.833+0000 7fc94a400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc944161670 0x7fc94415ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:40.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.833+0000 7fc94a400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9440ce600 con 0x7fc944161670 2024-03-04T09:28:40.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.833+0000 7fc94a400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9440cf520 con 0x7fc94406e240 2024-03-04T09:28:40.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.833+0000 7fc94a400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9440ce780 con 0x7fc9440ae0a0 2024-03-04T09:28:40.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.833+0000 7fc943e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc944161670 0x7fc94415ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:40.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.833+0000 7fc943e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc944161670 0x7fc94415ce60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:44158/0 (socket says 172.21.15.6:44158) 2024-03-04T09:28:40.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.833+0000 7fc943e00640 1 -- 172.21.15.6:0/728112052 learned_addr learned my addr 172.21.15.6:0/728112052 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:28:40.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.833+0000 7fc943400640 1 --2- 172.21.15.6:0/728112052 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc94406e240 0x7fc9440b3ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:40.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.833+0000 7fc942a00640 1 --2- 172.21.15.6:0/728112052 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc9440ae0a0 0x7fc94415c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:40.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.833+0000 7fc943e00640 1 -- 172.21.15.6:0/728112052 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc9440ae0a0 msgr2=0x7fc94415c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:40.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.833+0000 7fc943e00640 1 --2- 172.21.15.6:0/728112052 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc9440ae0a0 0x7fc94415c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:40.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.833+0000 7fc943e00640 1 -- 172.21.15.6:0/728112052 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc94406e240 msgr2=0x7fc9440b3ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:40.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.833+0000 7fc943e00640 1 --2- 172.21.15.6:0/728112052 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc94406e240 0x7fc9440b3ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:40.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.833+0000 7fc943e00640 1 -- 172.21.15.6:0/728112052 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc94415d690 con 0x7fc944161670 2024-03-04T09:28:40.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.834+0000 7fc943e00640 1 --2- 172.21.15.6:0/728112052 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc944161670 0x7fc94415ce60 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fc93c00d830 tx=0x7fc93c00dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:40.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.834+0000 7fc93b400640 1 -- 172.21.15.6:0/728112052 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc93c00b6c0 con 0x7fc944161670 2024-03-04T09:28:40.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.834+0000 7fc93b400640 1 -- 172.21.15.6:0/728112052 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc93c019070 con 0x7fc944161670 2024-03-04T09:28:40.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.834+0000 7fc93b400640 1 -- 172.21.15.6:0/728112052 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc93c00bac0 con 0x7fc944161670 2024-03-04T09:28:40.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.834+0000 7fc94a400640 1 -- 172.21.15.6:0/728112052 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc94415d970 con 0x7fc944161670 2024-03-04T09:28:40.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.834+0000 7fc94a400640 1 -- 172.21.15.6:0/728112052 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc94415d3a0 con 0x7fc944161670 2024-03-04T09:28:40.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.835+0000 7fc94a400640 1 -- 172.21.15.6:0/728112052 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc944069510 con 0x7fc944161670 2024-03-04T09:28:40.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.836+0000 7fc93b400640 1 -- 172.21.15.6:0/728112052 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc93c01e460 con 0x7fc944161670 2024-03-04T09:28:40.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.837+0000 7fc93b400640 1 --2- 172.21.15.6:0/728112052 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc92006b520 0x7fc92006d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:40.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.837+0000 7fc93b400640 1 -- 172.21.15.6:0/728112052 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fc93c08a530 con 0x7fc944161670 2024-03-04T09:28:40.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.838+0000 7fc943400640 1 --2- 172.21.15.6:0/728112052 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc92006b520 0x7fc92006d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:40.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.839+0000 7fc943400640 1 --2- 172.21.15.6:0/728112052 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc92006b520 0x7fc92006d9e0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fc92c036000 tx=0x7fc92c008030 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:40.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:40.842+0000 7fc93b400640 1 -- 172.21.15.6:0/728112052 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc93c059720 con 0x7fc944161670 2024-03-04T09:28:41.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.113+0000 7fc94a400640 1 -- 172.21.15.6:0/728112052 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_fragmented"]} v 0) v1 -- 0x7fc9440ad4c0 con 0x7fc944161670 2024-03-04T09:28:41.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.134+0000 7fc93b400640 1 -- 172.21.15.6:0/728112052 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_fragmented"]}]=0 v200) v1 ==== 144+0+0 (secure 0 0 0) 0x7fc93c059020 con 0x7fc944161670 2024-03-04T09:28:41.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.137+0000 7fc94a400640 1 -- 172.21.15.6:0/728112052 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc92006b520 msgr2=0x7fc92006d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:41.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.137+0000 7fc94a400640 1 --2- 172.21.15.6:0/728112052 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc92006b520 0x7fc92006d9e0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fc92c036000 tx=0x7fc92c008030 comp rx=0 tx=0).stop 2024-03-04T09:28:41.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.137+0000 7fc94a400640 1 -- 172.21.15.6:0/728112052 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc944161670 msgr2=0x7fc94415ce60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:41.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.137+0000 7fc94a400640 1 --2- 172.21.15.6:0/728112052 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc944161670 0x7fc94415ce60 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fc93c00d830 tx=0x7fc93c00dcf0 comp rx=0 tx=0).stop 2024-03-04T09:28:41.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.138+0000 7fc94a400640 1 -- 172.21.15.6:0/728112052 shutdown_connections 2024-03-04T09:28:41.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.138+0000 7fc94a400640 1 --2- 172.21.15.6:0/728112052 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc944161670 0x7fc94415ce60 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:41.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.138+0000 7fc94a400640 1 --2- 172.21.15.6:0/728112052 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc92006b520 0x7fc92006d9e0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:41.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.138+0000 7fc94a400640 1 --2- 172.21.15.6:0/728112052 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc9440ae0a0 0x7fc94415c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:41.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.138+0000 7fc94a400640 1 --2- 172.21.15.6:0/728112052 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc94406e240 0x7fc9440b3ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:41.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.138+0000 7fc94a400640 1 -- 172.21.15.6:0/728112052 >> 172.21.15.6:0/728112052 conn(0x7fc94406afb0 msgr2=0x7fc94406b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:41.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.138+0000 7fc94a400640 1 -- 172.21.15.6:0/728112052 shutdown_connections 2024-03-04T09:28:41.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.138+0000 7fc94a400640 1 -- 172.21.15.6:0/728112052 wait complete. 2024-03-04T09:28:41.155 INFO:tasks.cephfs_test_runner:That fragmentation (forced) will allow more entries to be created. ... ok 2024-03-04T09:28:41.155 INFO:tasks.cephfs_test_runner:Starting test: test_dirfrag_limit_strays (tasks.cephfs.test_strays.TestStrays) That unlinking fails when the stray directory fragment becomes too 2024-03-04T09:28:41.155 INFO:tasks.cephfs_test_runner:test_dirfrag_limit_strays (tasks.cephfs.test_strays.TestStrays) 2024-03-04T09:28:41.155 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T09:28:41.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.382+0000 7f9111e00640 1 Processor -- start 2024-03-04T09:28:41.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.383+0000 7f9111e00640 1 -- start start 2024-03-04T09:28:41.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.384+0000 7f9111e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f910c161670 0x7f910c070c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:41.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.384+0000 7f9111e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f910c0ae0a0 0x7f910c0711b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:41.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.384+0000 7f9111e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f910c0ce9a0 con 0x7f910c06e240 2024-03-04T09:28:41.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.384+0000 7f9111e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f910c0730d0 con 0x7f910c161670 2024-03-04T09:28:41.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.384+0000 7f9111e00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f910c0cec80 con 0x7f910c0ae0a0 2024-03-04T09:28:41.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.385+0000 7f910aa00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f910c161670 0x7f910c070c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:41.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.385+0000 7f910a000640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f910c06e240 0x7f910c06cfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:60972/0 (socket says 172.21.15.6:60972) 2024-03-04T09:28:41.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.385+0000 7f910a000640 1 -- 172.21.15.6:0/2914999766 learned_addr learned my addr 172.21.15.6:0/2914999766 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:28:41.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.385+0000 7f910b400640 1 --2- 172.21.15.6:0/2914999766 >> v2:172.21.15.146:3300/0 conn(0x7f910c0ae0a0 0x7f910c0711b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:41.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.385+0000 7f910aa00640 1 -- 172.21.15.6:0/2914999766 >> v2:172.21.15.146:3300/0 conn(0x7f910c0ae0a0 msgr2=0x7f910c0711b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:41.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.385+0000 7f910aa00640 1 --2- 172.21.15.6:0/2914999766 >> v2:172.21.15.146:3300/0 conn(0x7f910c0ae0a0 0x7f910c0711b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:41.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.385+0000 7f910aa00640 1 -- 172.21.15.6:0/2914999766 >> v1:172.21.15.6:6789/0 conn(0x7f910c06e240 legacy=0x7f910c06cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:41.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.385+0000 7f910aa00640 1 -- 172.21.15.6:0/2914999766 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f910c0cef60 con 0x7f910c161670 2024-03-04T09:28:41.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.385+0000 7f9109600640 1 -- 172.21.15.6:0/2914999766 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2191693072 0 0) 0x7f910c0ce9a0 con 0x7f910c06e240 2024-03-04T09:28:41.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.386+0000 7f910aa00640 1 --2- 172.21.15.6:0/2914999766 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f910c161670 0x7f910c070c70 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f910c0cc7c0 tx=0x7f90fc02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb25f57005940f88 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:41.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.387+0000 7f9109600640 1 -- 172.21.15.6:0/2914999766 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90fc039070 con 0x7f910c161670 2024-03-04T09:28:41.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.387+0000 7f9109600640 1 -- 172.21.15.6:0/2914999766 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90fc02cbf0 con 0x7f910c161670 2024-03-04T09:28:41.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.387+0000 7f9109600640 1 -- 172.21.15.6:0/2914999766 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90fc034440 con 0x7f910c161670 2024-03-04T09:28:41.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.387+0000 7f9111e00640 1 -- 172.21.15.6:0/2914999766 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f910c161670 msgr2=0x7f910c070c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:41.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.387+0000 7f9111e00640 1 --2- 172.21.15.6:0/2914999766 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f910c161670 0x7f910c070c70 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f910c0cc7c0 tx=0x7f90fc02be40 comp rx=0 tx=0).stop 2024-03-04T09:28:41.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.387+0000 7f9111e00640 1 -- 172.21.15.6:0/2914999766 shutdown_connections 2024-03-04T09:28:41.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.387+0000 7f9111e00640 1 --2- 172.21.15.6:0/2914999766 >> v2:172.21.15.146:3300/0 conn(0x7f910c0ae0a0 0x7f910c0711b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:41.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.387+0000 7f9111e00640 1 --2- 172.21.15.6:0/2914999766 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f910c161670 0x7f910c070c70 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:41.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.387+0000 7f9111e00640 1 -- 172.21.15.6:0/2914999766 >> 172.21.15.6:0/2914999766 conn(0x7f910c06afb0 msgr2=0x7f910c06b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:41.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.388+0000 7f9111e00640 1 -- 172.21.15.6:0/2914999766 shutdown_connections 2024-03-04T09:28:41.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.388+0000 7f9111e00640 1 -- 172.21.15.6:0/2914999766 wait complete. 2024-03-04T09:28:41.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.389+0000 7f9111e00640 1 Processor -- start 2024-03-04T09:28:41.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.389+0000 7f9111e00640 1 -- start start 2024-03-04T09:28:41.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.390+0000 7f9111e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f910c06e240 0x7f910c0cba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:41.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.390+0000 7f9111e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f910c0ae0a0 0x7f910c14c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:41.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.390+0000 7f9111e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f910c161670 0x7f910c156250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:41.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.391+0000 7f9111e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f910c0ce600 con 0x7f910c0ae0a0 2024-03-04T09:28:41.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.391+0000 7f9111e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f910c0cf520 con 0x7f910c06e240 2024-03-04T09:28:41.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.391+0000 7f9111e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f910c0ce780 con 0x7f910c161670 2024-03-04T09:28:41.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.391+0000 7f910a000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f910c0ae0a0 0x7f910c14c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:41.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.391+0000 7f910a000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f910c0ae0a0 0x7f910c14c650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:44174/0 (socket says 172.21.15.6:44174) 2024-03-04T09:28:41.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.391+0000 7f910a000640 1 -- 172.21.15.6:0/1697882206 learned_addr learned my addr 172.21.15.6:0/1697882206 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:28:41.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.391+0000 7f910aa00640 1 --2- 172.21.15.6:0/1697882206 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f910c06e240 0x7f910c0cba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:41.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.391+0000 7f910b400640 1 --2- 172.21.15.6:0/1697882206 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f910c161670 0x7f910c156250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:41.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.391+0000 7f910a000640 1 -- 172.21.15.6:0/1697882206 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f910c161670 msgr2=0x7f910c156250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:41.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.391+0000 7f910a000640 1 --2- 172.21.15.6:0/1697882206 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f910c161670 0x7f910c156250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:41.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.391+0000 7f910a000640 1 -- 172.21.15.6:0/1697882206 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f910c06e240 msgr2=0x7f910c0cba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:41.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.391+0000 7f910a000640 1 --2- 172.21.15.6:0/1697882206 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f910c06e240 0x7f910c0cba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:41.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.391+0000 7f910a000640 1 -- 172.21.15.6:0/1697882206 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f910c0cbf80 con 0x7f910c0ae0a0 2024-03-04T09:28:41.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.392+0000 7f910a000640 1 --2- 172.21.15.6:0/1697882206 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f910c0ae0a0 0x7f910c14c650 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f90f400c500 tx=0x7f90f400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:41.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.392+0000 7f9103400640 1 -- 172.21.15.6:0/1697882206 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90f4018070 con 0x7f910c0ae0a0 2024-03-04T09:28:41.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.392+0000 7f9103400640 1 -- 172.21.15.6:0/1697882206 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90f4007580 con 0x7f910c0ae0a0 2024-03-04T09:28:41.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.392+0000 7f9111e00640 1 -- 172.21.15.6:0/1697882206 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f910c0ca9a0 con 0x7f910c0ae0a0 2024-03-04T09:28:41.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.392+0000 7f9103400640 1 -- 172.21.15.6:0/1697882206 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90f4004a20 con 0x7f910c0ae0a0 2024-03-04T09:28:41.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.392+0000 7f9111e00640 1 -- 172.21.15.6:0/1697882206 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f910c0caf60 con 0x7f910c0ae0a0 2024-03-04T09:28:41.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.393+0000 7f9111e00640 1 -- 172.21.15.6:0/1697882206 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f910c069510 con 0x7f910c0ae0a0 2024-03-04T09:28:41.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.394+0000 7f9103400640 1 -- 172.21.15.6:0/1697882206 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f90f401b070 con 0x7f910c0ae0a0 2024-03-04T09:28:41.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.395+0000 7f9103400640 1 --2- 172.21.15.6:0/1697882206 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f90e806b4d0 0x7f90e806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:41.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.395+0000 7f9103400640 1 -- 172.21.15.6:0/1697882206 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f90f4089220 con 0x7f910c0ae0a0 2024-03-04T09:28:41.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.396+0000 7f910aa00640 1 --2- 172.21.15.6:0/1697882206 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f90e806b4d0 0x7f90e806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:41.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.397+0000 7f910aa00640 1 --2- 172.21.15.6:0/1697882206 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f90e806b4d0 0x7f90e806d990 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f90fc036000 tx=0x7f90fc030000 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:41.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.400+0000 7f9103400640 1 -- 172.21.15.6:0/1697882206 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f90f4058390 con 0x7f910c0ae0a0 2024-03-04T09:28:41.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.599+0000 7f9111e00640 1 -- 172.21.15.6:0/1697882206 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f910c0745e0 con 0x7f910c0ae0a0 2024-03-04T09:28:41.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.601+0000 7f9103400640 1 -- 172.21.15.6:0/1697882206 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v59) v1 ==== 74+0+12972 (secure 0 0 0) 0x7f90f4057c90 con 0x7f910c0ae0a0 2024-03-04T09:28:41.607 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:28:41.607 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":59,"fsid":"edce0578-5ae4-4402-b45a-552bdef84d7c","created":"2024-03-04T09:24:54.523780+0000","modified":"2024-03-04T09:28:39.114303+0000","last_up_change":"2024-03-04T09:25:07.089049+0000","last_in_change":"2024-03-04T09:24:57.982077+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T09:25:07.935198+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a96574f-81f1-4bc6-8463-757520b84a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6825","nonce":1364449401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6827","nonce":1364449401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6831","nonce":1364449401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6829","nonce":1364449401}]},"public_addr":"172.21.15.6:6825/1364449401","cluster_addr":"172.21.15.6:6827/1364449401","heartbeat_back_addr":"172.21.15.6:6831/1364449401","heartbeat_front_addr":"172.21.15.6:6829/1364449401","state":["exists","up"]},{"osd":1,"uuid":"7d79b9c1-6b87-476a-bc36-7db5ee64cd90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6809","nonce":2494103078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6811","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6813","nonce":2494103078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6821","nonce":2494103078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6817","nonce":2494103078}]},"public_addr":"172.21.15.6:6809/2494103078","cluster_addr":"172.21.15.6:6813/2494103078","heartbeat_back_addr":"172.21.15.6:6821/2494103078","heartbeat_front_addr":"172.21.15.6:6817/2494103078","state":["exists","up"]},{"osd":2,"uuid":"492331b9-4ee9-40b6-bacd-85d44cd6c45a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6812","nonce":1500066681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6816","nonce":1500066681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6824","nonce":1500066681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6820","nonce":1500066681}]},"public_addr":"172.21.15.6:6812/1500066681","cluster_addr":"172.21.15.6:6816/1500066681","heartbeat_back_addr":"172.21.15.6:6824/1500066681","heartbeat_front_addr":"172.21.15.6:6820/1500066681","state":["exists","up"]},{"osd":3,"uuid":"21bb2efe-9ba7-4bdc-bc31-180cd1102e21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6801","nonce":1558806123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6803","nonce":1558806123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6807","nonce":1558806123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6805","nonce":1558806123}]},"public_addr":"172.21.15.6:6801/1558806123","cluster_addr":"172.21.15.6:6803/1558806123","heartbeat_back_addr":"172.21.15.6:6807/1558806123","heartbeat_front_addr":"172.21.15.6:6805/1558806123","state":["exists","up"]},{"osd":4,"uuid":"6e2c4b71-19d9-4737-83d9-558705d8644a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6801","nonce":1526135780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6803","nonce":1526135780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6807","nonce":1526135780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6805","nonce":1526135780}]},"public_addr":"172.21.15.146:6801/1526135780","cluster_addr":"172.21.15.146:6803/1526135780","heartbeat_back_addr":"172.21.15.146:6807/1526135780","heartbeat_front_addr":"172.21.15.146:6805/1526135780","state":["exists","up"]},{"osd":5,"uuid":"fb69c525-f1be-4b7d-ab32-6c29b0d6fddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6809","nonce":460262507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6811","nonce":460262507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6815","nonce":460262507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6813","nonce":460262507}]},"public_addr":"172.21.15.146:6809/460262507","cluster_addr":"172.21.15.146:6811/460262507","heartbeat_back_addr":"172.21.15.146:6815/460262507","heartbeat_front_addr":"172.21.15.146:6813/460262507","state":["exists","up"]},{"osd":6,"uuid":"c32e7fed-370d-4401-8ce0-6a4335b0d168","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6823","nonce":1392897493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6827","nonce":1392897493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6831","nonce":1392897493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6829","nonce":1392897493}]},"public_addr":"172.21.15.146:6823/1392897493","cluster_addr":"172.21.15.146:6827/1392897493","heartbeat_back_addr":"172.21.15.146:6831/1392897493","heartbeat_front_addr":"172.21.15.146:6829/1392897493","state":["exists","up"]},{"osd":7,"uuid":"75a79b44-fa5f-426d-a7d3-71f21f78835c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6817","nonce":3268323156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6819","nonce":3268323156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6825","nonce":3268323156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6822","nonce":3268323156}]},"public_addr":"172.21.15.146:6817/3268323156","cluster_addr":"172.21.15.146:6819/3268323156","heartbeat_back_addr":"172.21.15.146:6825/3268323156","heartbeat_front_addr":"172.21.15.146:6822/3268323156","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.310437+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.295850+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.294674+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.526855+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.300023+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.595160+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.346046+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.558741+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.146:6836/527855083":"2024-03-05T09:28:35.687878+0000","172.21.15.146:6834/527855083":"2024-03-05T09:28:35.687878+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T09:28:41.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.604+0000 7f9111e00640 1 -- 172.21.15.6:0/1697882206 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f90e806b4d0 msgr2=0x7f90e806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:41.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.604+0000 7f9111e00640 1 --2- 172.21.15.6:0/1697882206 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f90e806b4d0 0x7f90e806d990 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f90fc036000 tx=0x7f90fc030000 comp rx=0 tx=0).stop 2024-03-04T09:28:41.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.604+0000 7f9111e00640 1 -- 172.21.15.6:0/1697882206 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f910c0ae0a0 msgr2=0x7f910c14c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:41.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.604+0000 7f9111e00640 1 --2- 172.21.15.6:0/1697882206 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f910c0ae0a0 0x7f910c14c650 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f90f400c500 tx=0x7f90f400c9c0 comp rx=0 tx=0).stop 2024-03-04T09:28:41.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.604+0000 7f9111e00640 1 -- 172.21.15.6:0/1697882206 shutdown_connections 2024-03-04T09:28:41.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.604+0000 7f9111e00640 1 --2- 172.21.15.6:0/1697882206 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f90e806b4d0 0x7f90e806d990 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:41.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.604+0000 7f9111e00640 1 --2- 172.21.15.6:0/1697882206 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f910c161670 0x7f910c156250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:41.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.604+0000 7f9111e00640 1 --2- 172.21.15.6:0/1697882206 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f910c0ae0a0 0x7f910c14c650 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:41.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.604+0000 7f9111e00640 1 --2- 172.21.15.6:0/1697882206 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f910c06e240 0x7f910c0cba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:41.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.604+0000 7f9111e00640 1 -- 172.21.15.6:0/1697882206 >> 172.21.15.6:0/1697882206 conn(0x7f910c06afb0 msgr2=0x7f910c0b2ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:41.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.604+0000 7f9111e00640 1 -- 172.21.15.6:0/1697882206 shutdown_connections 2024-03-04T09:28:41.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.604+0000 7f9111e00640 1 -- 172.21.15.6:0/1697882206 wait complete. 2024-03-04T09:28:41.622 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T09:25:07.935198+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-04T09:28:41.622 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-04T09:28:41.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.842+0000 7f3010a00640 1 Processor -- start 2024-03-04T09:28:41.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.843+0000 7f3010a00640 1 -- start start 2024-03-04T09:28:41.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.844+0000 7f3010a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f300c071e20 0x7f300c0ccad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:41.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.844+0000 7f3010a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f300c072df0 con 0x7f300c072520 2024-03-04T09:28:41.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.844+0000 7f3010a00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f300c0cdab0 con 0x7f300c15bf20 2024-03-04T09:28:41.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.844+0000 7f3010a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f300c0b4d70 con 0x7f300c071e20 2024-03-04T09:28:41.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.845+0000 7f3009400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f300c072520 0x7f300c0cdfb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:60982/0 (socket says 172.21.15.6:60982) 2024-03-04T09:28:41.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.845+0000 7f3009400640 1 -- 172.21.15.6:0/3136211353 learned_addr learned my addr 172.21.15.6:0/3136211353 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:28:41.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.845+0000 7f300a800640 1 --2- 172.21.15.6:0/3136211353 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f300c071e20 0x7f300c0ccad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:41.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.846+0000 7f3008a00640 1 -- 172.21.15.6:0/3136211353 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3248235022 0 0) 0x7f300c072df0 con 0x7f300c072520 2024-03-04T09:28:41.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.846+0000 7f3008a00640 1 -- 172.21.15.6:0/3136211353 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ff0003520 con 0x7f300c072520 2024-03-04T09:28:41.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.846+0000 7f3008a00640 1 -- 172.21.15.6:0/3136211353 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2594600637 0 0) 0x7f300c0cdab0 con 0x7f300c15bf20 2024-03-04T09:28:41.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.846+0000 7f3008a00640 1 -- 172.21.15.6:0/3136211353 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f300c072df0 con 0x7f300c15bf20 2024-03-04T09:28:41.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.846+0000 7f3008a00640 1 -- 172.21.15.6:0/3136211353 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2845746941 0 0) 0x7f2ff0003520 con 0x7f300c072520 2024-03-04T09:28:41.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.846+0000 7f3008a00640 1 -- 172.21.15.6:0/3136211353 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f300c071e20 msgr2=0x7f300c0ccad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:41.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.846+0000 7f3008a00640 1 --2- 172.21.15.6:0/3136211353 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f300c071e20 0x7f300c0ccad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:41.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.846+0000 7f3008a00640 1 -- 172.21.15.6:0/3136211353 >> v1:172.21.15.146:6789/0 conn(0x7f300c15bf20 legacy=0x7f300c0acd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:41.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.846+0000 7f3008a00640 1 -- 172.21.15.6:0/3136211353 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f300c0b5050 con 0x7f300c072520 2024-03-04T09:28:41.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.846+0000 7f3008a00640 1 -- 172.21.15.6:0/3136211353 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7f2ff4001670 con 0x7f300c072520 2024-03-04T09:28:41.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.846+0000 7f300a800640 1 --2- 172.21.15.6:0/3136211353 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f300c071e20 0x7f300c0ccad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T09:28:41.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.846+0000 7f3008a00640 1 -- 172.21.15.6:0/3136211353 >> v1:172.21.15.6:6789/0 conn(0x7f300c072520 legacy=0x7f300c0cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:41.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.846+0000 7f3008a00640 1 --2- 172.21.15.6:0/3136211353 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2ff0003b70 0x7f2ff0003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:41.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.846+0000 7f3008a00640 1 --2- 172.21.15.6:0/3136211353 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f300c071e20 0x7f2ff0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:41.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.846+0000 7f3008a00640 1 --2- 172.21.15.6:0/3136211353 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2ff0004d40 0x7f2ff00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:41.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.846+0000 7f3008a00640 1 -- 172.21.15.6:0/3136211353 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f300c0b4d70 con 0x7f2ff0003b70 2024-03-04T09:28:41.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.846+0000 7f3008a00640 1 -- 172.21.15.6:0/3136211353 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ff000ba70 con 0x7f2ff0004d40 2024-03-04T09:28:41.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.846+0000 7f3008a00640 1 -- 172.21.15.6:0/3136211353 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ff000bbf0 con 0x7f300c071e20 2024-03-04T09:28:41.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.847+0000 7f3009e00640 1 --2- 172.21.15.6:0/3136211353 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2ff0003b70 0x7f2ff0003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:41.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.847+0000 7f3009400640 1 --2- 172.21.15.6:0/3136211353 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f300c071e20 0x7f2ff0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:41.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.847+0000 7f300a800640 1 --2- 172.21.15.6:0/3136211353 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2ff0004d40 0x7f2ff00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:41.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.847+0000 7f3009e00640 1 -- 172.21.15.6:0/3136211353 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f300c071e20 msgr2=0x7f2ff0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:41.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.847+0000 7f3009e00640 1 --2- 172.21.15.6:0/3136211353 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f300c071e20 0x7f2ff0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:41.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.847+0000 7f3009e00640 1 -- 172.21.15.6:0/3136211353 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2ff0004d40 msgr2=0x7f2ff00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:41.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.847+0000 7f3009e00640 1 --2- 172.21.15.6:0/3136211353 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2ff0004d40 0x7f2ff00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:41.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.847+0000 7f3009e00640 1 -- 172.21.15.6:0/3136211353 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ff0003520 con 0x7f2ff0003b70 2024-03-04T09:28:41.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.847+0000 7f300a800640 1 --2- 172.21.15.6:0/3136211353 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2ff0004d40 0x7f2ff00051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:28:41.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.848+0000 7f3009e00640 1 --2- 172.21.15.6:0/3136211353 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2ff0003b70 0x7f2ff0003f70 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f2ffc002000 tx=0x7f2ff0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8e139bf71ab1ecf server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:41.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.849+0000 7f3008a00640 1 -- 172.21.15.6:0/3136211353 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ffc001bf0 con 0x7f2ff0003b70 2024-03-04T09:28:41.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.849+0000 7f3008a00640 1 -- 172.21.15.6:0/3136211353 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ffc003b70 con 0x7f2ff0003b70 2024-03-04T09:28:41.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.849+0000 7f3008a00640 1 -- 172.21.15.6:0/3136211353 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ffc0050d0 con 0x7f2ff0003b70 2024-03-04T09:28:41.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.849+0000 7f3010a00640 1 -- 172.21.15.6:0/3136211353 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2ff0003b70 msgr2=0x7f2ff0003f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:41.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.849+0000 7f3010a00640 1 --2- 172.21.15.6:0/3136211353 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2ff0003b70 0x7f2ff0003f70 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f2ffc002000 tx=0x7f2ff0005830 comp rx=0 tx=0).stop 2024-03-04T09:28:41.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.849+0000 7f300a800640 1 -- 172.21.15.6:0/3136211353 reap_dead start 2024-03-04T09:28:41.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.849+0000 7f3010a00640 1 -- 172.21.15.6:0/3136211353 shutdown_connections 2024-03-04T09:28:41.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.849+0000 7f3010a00640 1 -- 172.21.15.6:0/3136211353 >> 172.21.15.6:0/3136211353 conn(0x7f300c06b3d0 msgr2=0x7f300c06b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:41.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.849+0000 7f3010a00640 1 -- 172.21.15.6:0/3136211353 shutdown_connections 2024-03-04T09:28:41.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.850+0000 7f3010a00640 1 -- 172.21.15.6:0/3136211353 wait complete. 2024-03-04T09:28:41.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.851+0000 7f3010a00640 1 Processor -- start 2024-03-04T09:28:41.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.852+0000 7f3010a00640 1 -- start start 2024-03-04T09:28:41.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.852+0000 7f3010a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2ff0003b70 0x7f300c15e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:41.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.852+0000 7f3010a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f300c147490 0x7f300c15c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:41.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.853+0000 7f3010a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f300c1574d0 0x7f300c157950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:41.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.853+0000 7f3010a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f300c0ad450 con 0x7f300c1574d0 2024-03-04T09:28:41.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.853+0000 7f3010a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f300c0b4990 con 0x7f2ff0003b70 2024-03-04T09:28:41.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.853+0000 7f3010a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f300c0afbc0 con 0x7f300c147490 2024-03-04T09:28:41.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.853+0000 7f300a800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f300c1574d0 0x7f300c157950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:41.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.853+0000 7f300a800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f300c1574d0 0x7f300c157950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:44196/0 (socket says 172.21.15.6:44196) 2024-03-04T09:28:41.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.853+0000 7f300a800640 1 -- 172.21.15.6:0/2330630940 learned_addr learned my addr 172.21.15.6:0/2330630940 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:28:41.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.853+0000 7f3009e00640 1 --2- 172.21.15.6:0/2330630940 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2ff0003b70 0x7f300c15e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:41.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.853+0000 7f3009400640 1 --2- 172.21.15.6:0/2330630940 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f300c147490 0x7f300c15c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:41.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.853+0000 7f300a800640 1 -- 172.21.15.6:0/2330630940 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f300c147490 msgr2=0x7f300c15c370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:41.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.853+0000 7f300a800640 1 --2- 172.21.15.6:0/2330630940 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f300c147490 0x7f300c15c370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:41.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.853+0000 7f300a800640 1 -- 172.21.15.6:0/2330630940 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2ff0003b70 msgr2=0x7f300c15e900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:41.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.853+0000 7f300a800640 1 --2- 172.21.15.6:0/2330630940 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2ff0003b70 0x7f300c15e900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:41.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.853+0000 7f300a800640 1 -- 172.21.15.6:0/2330630940 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f300c15ef10 con 0x7f300c1574d0 2024-03-04T09:28:41.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.853+0000 7f3009e00640 1 --2- 172.21.15.6:0/2330630940 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2ff0003b70 0x7f300c15e900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:28:41.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.854+0000 7f300a800640 1 --2- 172.21.15.6:0/2330630940 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f300c1574d0 0x7f300c157950 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f300c06c030 tx=0x7f3004010b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:41.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.854+0000 7f3002000640 1 -- 172.21.15.6:0/2330630940 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f300400f040 con 0x7f300c1574d0 2024-03-04T09:28:41.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.854+0000 7f3002000640 1 -- 172.21.15.6:0/2330630940 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f300401b070 con 0x7f300c1574d0 2024-03-04T09:28:41.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.854+0000 7f3010a00640 1 -- 172.21.15.6:0/2330630940 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f300c15f1f0 con 0x7f300c1574d0 2024-03-04T09:28:41.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.854+0000 7f3002000640 1 -- 172.21.15.6:0/2330630940 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3004010ea0 con 0x7f300c1574d0 2024-03-04T09:28:41.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.854+0000 7f3010a00640 1 -- 172.21.15.6:0/2330630940 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f300c0b0160 con 0x7f300c1574d0 2024-03-04T09:28:41.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.855+0000 7f3010a00640 1 -- 172.21.15.6:0/2330630940 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f300c0b0360 con 0x7f300c1574d0 2024-03-04T09:28:41.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.856+0000 7f3002000640 1 -- 172.21.15.6:0/2330630940 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f300400e6a0 con 0x7f300c1574d0 2024-03-04T09:28:41.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.857+0000 7f3002000640 1 --2- 172.21.15.6:0/2330630940 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2fe806b4d0 0x7f2fe806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:41.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.857+0000 7f3002000640 1 -- 172.21.15.6:0/2330630940 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f3004089af0 con 0x7f300c1574d0 2024-03-04T09:28:41.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.858+0000 7f3009e00640 1 --2- 172.21.15.6:0/2330630940 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2fe806b4d0 0x7f2fe806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:41.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.860+0000 7f3009e00640 1 --2- 172.21.15.6:0/2330630940 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2fe806b4d0 0x7f2fe806d990 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f2ffc034000 tx=0x7f2ffc009030 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:41.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:41.862+0000 7f3002000640 1 -- 172.21.15.6:0/2330630940 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f300c0b0360 con 0x7f300c1574d0 2024-03-04T09:28:42.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:42.065+0000 7f3010a00640 1 -- 172.21.15.6:0/2330630940 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f300c04f330 con 0x7f300c1574d0 2024-03-04T09:28:42.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:42.066+0000 7f3002000640 1 -- 172.21.15.6:0/2330630940 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v59) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3004025020 con 0x7f300c1574d0 2024-03-04T09:28:42.069 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-04T09:28:42.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:42.069+0000 7f3010a00640 1 -- 172.21.15.6:0/2330630940 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2fe806b4d0 msgr2=0x7f2fe806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:42.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:42.069+0000 7f3010a00640 1 --2- 172.21.15.6:0/2330630940 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2fe806b4d0 0x7f2fe806d990 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f2ffc034000 tx=0x7f2ffc009030 comp rx=0 tx=0).stop 2024-03-04T09:28:42.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:42.069+0000 7f3010a00640 1 -- 172.21.15.6:0/2330630940 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f300c1574d0 msgr2=0x7f300c157950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:42.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:42.069+0000 7f3010a00640 1 --2- 172.21.15.6:0/2330630940 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f300c1574d0 0x7f300c157950 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f300c06c030 tx=0x7f3004010b60 comp rx=0 tx=0).stop 2024-03-04T09:28:42.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:42.069+0000 7f3010a00640 1 -- 172.21.15.6:0/2330630940 shutdown_connections 2024-03-04T09:28:42.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:42.069+0000 7f3010a00640 1 --2- 172.21.15.6:0/2330630940 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f300c1574d0 0x7f300c157950 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:42.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:42.069+0000 7f3010a00640 1 --2- 172.21.15.6:0/2330630940 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2fe806b4d0 0x7f2fe806d990 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:42.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:42.069+0000 7f3010a00640 1 --2- 172.21.15.6:0/2330630940 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f300c147490 0x7f300c15c370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:42.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:42.069+0000 7f3010a00640 1 --2- 172.21.15.6:0/2330630940 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2ff0003b70 0x7f300c15e900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:42.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:42.069+0000 7f3010a00640 1 -- 172.21.15.6:0/2330630940 >> 172.21.15.6:0/2330630940 conn(0x7f300c06b3d0 msgr2=0x7f300c0ccaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:42.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:42.069+0000 7f3010a00640 1 -- 172.21.15.6:0/2330630940 shutdown_connections 2024-03-04T09:28:42.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:42.069+0000 7f3010a00640 1 -- 172.21.15.6:0/2330630940 wait complete. 2024-03-04T09:28:42.089 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_strays' 2024-03-04T09:28:42.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:42.319+0000 7fc15ee00640 1 Processor -- start 2024-03-04T09:28:42.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:42.319+0000 7fc15ee00640 1 -- start start 2024-03-04T09:28:42.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:42.321+0000 7fc15ee00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fc158145300 0x7fc1580b49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:42.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:42.321+0000 7fc15ee00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc158070f10 0x7fc1580cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:42.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:42.321+0000 7fc15ee00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1580714d0 con 0x7fc1580cce90 2024-03-04T09:28:42.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:42.321+0000 7fc15ee00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1580cd300 con 0x7fc158070f10 2024-03-04T09:28:42.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:42.321+0000 7fc15ee00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1580717b0 con 0x7fc158145300 2024-03-04T09:28:42.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:42.321+0000 7fc15d000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc158070f10 0x7fc1580cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:42.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:42.321+0000 7fc15d000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc158070f10 0x7fc1580cc950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:44210/0 (socket says 172.21.15.6:44210) 2024-03-04T09:28:42.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:42.321+0000 7fc15d000640 1 -- 172.21.15.6:0/1182735226 learned_addr learned my addr 172.21.15.6:0/1182735226 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:28:42.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:42.322+0000 7fc15d000640 1 -- 172.21.15.6:0/1182735226 >> v2:172.21.15.146:3300/0 conn(0x7fc158145300 msgr2=0x7fc1580b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:42.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:42.322+0000 7fc15d000640 1 --2- 172.21.15.6:0/1182735226 >> v2:172.21.15.146:3300/0 conn(0x7fc158145300 0x7fc1580b49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:42.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:42.322+0000 7fc15d000640 1 -- 172.21.15.6:0/1182735226 >> v1:172.21.15.146:6789/0 conn(0x7fc1580cce90 legacy=0x7fc1580cd930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:42.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:42.322+0000 7fc15d000640 1 -- 172.21.15.6:0/1182735226 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc158071a90 con 0x7fc158070f10 2024-03-04T09:28:42.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:42.323+0000 7fc15d000640 1 --2- 172.21.15.6:0/1182735226 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc158070f10 0x7fc1580cc950 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fc1580cf990 tx=0x7fc14402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=870c34b84883bb41 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:42.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:42.323+0000 7fc157e00640 1 -- 172.21.15.6:0/1182735226 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc14402e070 con 0x7fc158070f10 2024-03-04T09:28:42.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:42.323+0000 7fc157e00640 1 -- 172.21.15.6:0/1182735226 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc144037070 con 0x7fc158070f10 2024-03-04T09:28:42.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:42.323+0000 7fc157e00640 1 -- 172.21.15.6:0/1182735226 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc144031a00 con 0x7fc158070f10 2024-03-04T09:28:42.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:42.324+0000 7fc15ee00640 1 -- 172.21.15.6:0/1182735226 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc158070f10 msgr2=0x7fc1580cc950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:42.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:42.324+0000 7fc15ee00640 1 --2- 172.21.15.6:0/1182735226 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc158070f10 0x7fc1580cc950 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fc1580cf990 tx=0x7fc14402b880 comp rx=0 tx=0).stop 2024-03-04T09:28:42.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:42.324+0000 7fc15ee00640 1 -- 172.21.15.6:0/1182735226 shutdown_connections 2024-03-04T09:28:42.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:42.324+0000 7fc15ee00640 1 --2- 172.21.15.6:0/1182735226 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc158070f10 0x7fc1580cc950 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:42.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:42.324+0000 7fc15ee00640 1 --2- 172.21.15.6:0/1182735226 >> v2:172.21.15.146:3300/0 conn(0x7fc158145300 0x7fc1580b49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:42.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:42.324+0000 7fc15ee00640 1 -- 172.21.15.6:0/1182735226 >> 172.21.15.6:0/1182735226 conn(0x7fc15806b3d0 msgr2=0x7fc15806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:42.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:42.324+0000 7fc15ee00640 1 -- 172.21.15.6:0/1182735226 shutdown_connections 2024-03-04T09:28:42.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:42.324+0000 7fc15ee00640 1 -- 172.21.15.6:0/1182735226 wait complete. 2024-03-04T09:28:42.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:42.326+0000 7fc15ee00640 1 Processor -- start 2024-03-04T09:28:42.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:42.326+0000 7fc15ee00640 1 -- start start 2024-03-04T09:28:42.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:42.327+0000 7fc15ee00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc158070f10 0x7fc15815b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:42.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:42.327+0000 7fc15ee00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc1580cce90 0x7fc1581f3b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:42.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:42.327+0000 7fc15ee00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc158145300 0x7fc1581f40a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:42.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:42.327+0000 7fc15ee00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1581456e0 con 0x7fc1580cce90 2024-03-04T09:28:42.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:42.327+0000 7fc15ee00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1580b4ee0 con 0x7fc158070f10 2024-03-04T09:28:42.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:42.327+0000 7fc15ee00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc15815d580 con 0x7fc158145300 2024-03-04T09:28:42.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:42.327+0000 7fc15d000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc1580cce90 0x7fc1581f3b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:42.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:42.327+0000 7fc15d000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc1580cce90 0x7fc1581f3b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:44224/0 (socket says 172.21.15.6:44224) 2024-03-04T09:28:42.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:42.327+0000 7fc15d000640 1 -- 172.21.15.6:0/480839175 learned_addr learned my addr 172.21.15.6:0/480839175 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:28:42.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:42.327+0000 7fc15e400640 1 --2- 172.21.15.6:0/480839175 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc158145300 0x7fc1581f40a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:42.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:42.327+0000 7fc15da00640 1 --2- 172.21.15.6:0/480839175 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc158070f10 0x7fc15815b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:42.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:42.327+0000 7fc15d000640 1 -- 172.21.15.6:0/480839175 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc158145300 msgr2=0x7fc1581f40a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:42.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:42.327+0000 7fc15d000640 1 --2- 172.21.15.6:0/480839175 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc158145300 0x7fc1581f40a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:42.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:42.328+0000 7fc15d000640 1 -- 172.21.15.6:0/480839175 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc158070f10 msgr2=0x7fc15815b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:42.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:42.328+0000 7fc15d000640 1 --2- 172.21.15.6:0/480839175 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc158070f10 0x7fc15815b2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:42.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:42.328+0000 7fc15d000640 1 -- 172.21.15.6:0/480839175 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1581e5c30 con 0x7fc1580cce90 2024-03-04T09:28:42.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:42.328+0000 7fc15e400640 1 --2- 172.21.15.6:0/480839175 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc158145300 0x7fc1581f40a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:28:42.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:42.328+0000 7fc15d000640 1 --2- 172.21.15.6:0/480839175 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc1580cce90 0x7fc1581f3b60 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fc144000c00 tx=0x7fc1440046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:42.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:42.328+0000 7fc155600640 1 -- 172.21.15.6:0/480839175 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc144032880 con 0x7fc1580cce90 2024-03-04T09:28:42.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:42.328+0000 7fc155600640 1 -- 172.21.15.6:0/480839175 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc144032a40 con 0x7fc1580cce90 2024-03-04T09:28:42.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:42.328+0000 7fc15ee00640 1 -- 172.21.15.6:0/480839175 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1581f4f80 con 0x7fc1580cce90 2024-03-04T09:28:42.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:42.328+0000 7fc155600640 1 -- 172.21.15.6:0/480839175 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc144032e30 con 0x7fc1580cce90 2024-03-04T09:28:42.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:42.329+0000 7fc15ee00640 1 -- 172.21.15.6:0/480839175 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1581f5540 con 0x7fc1580cce90 2024-03-04T09:28:42.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:42.330+0000 7fc15ee00640 1 -- 172.21.15.6:0/480839175 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc15804f330 con 0x7fc1580cce90 2024-03-04T09:28:42.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:42.331+0000 7fc155600640 1 -- 172.21.15.6:0/480839175 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc14404c060 con 0x7fc1580cce90 2024-03-04T09:28:42.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:42.332+0000 7fc155600640 1 --2- 172.21.15.6:0/480839175 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc13806b4d0 0x7fc13806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:42.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:42.332+0000 7fc155600640 1 -- 172.21.15.6:0/480839175 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fc1440abc70 con 0x7fc1580cce90 2024-03-04T09:28:42.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:42.332+0000 7fc15da00640 1 --2- 172.21.15.6:0/480839175 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc13806b4d0 0x7fc13806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:42.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:42.334+0000 7fc15da00640 1 --2- 172.21.15.6:0/480839175 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc13806b4d0 0x7fc13806d990 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fc14c00a770 tx=0x7fc14c008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:42.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:42.337+0000 7fc155600640 1 -- 172.21.15.6:0/480839175 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc144040070 con 0x7fc1580cce90 2024-03-04T09:28:42.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:42.619+0000 7fc15ee00640 1 -- 172.21.15.6:0/480839175 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_strays"]} v 0) v1 -- 0x7fc1580ae330 con 0x7fc1580cce90 2024-03-04T09:28:43.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:43.146+0000 7fc155600640 1 -- 172.21.15.6:0/480839175 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_strays"]}]=0 v202) v1 ==== 143+0+0 (secure 0 0 0) 0x7fc144039310 con 0x7fc1580cce90 2024-03-04T09:28:43.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:43.150+0000 7fc15ee00640 1 -- 172.21.15.6:0/480839175 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc13806b4d0 msgr2=0x7fc13806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:43.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:43.150+0000 7fc15ee00640 1 --2- 172.21.15.6:0/480839175 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc13806b4d0 0x7fc13806d990 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fc14c00a770 tx=0x7fc14c008040 comp rx=0 tx=0).stop 2024-03-04T09:28:43.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:43.150+0000 7fc15ee00640 1 -- 172.21.15.6:0/480839175 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc1580cce90 msgr2=0x7fc1581f3b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:43.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:43.150+0000 7fc15ee00640 1 --2- 172.21.15.6:0/480839175 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc1580cce90 0x7fc1581f3b60 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fc144000c00 tx=0x7fc1440046c0 comp rx=0 tx=0).stop 2024-03-04T09:28:43.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:43.150+0000 7fc15ee00640 1 -- 172.21.15.6:0/480839175 shutdown_connections 2024-03-04T09:28:43.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:43.150+0000 7fc15ee00640 1 --2- 172.21.15.6:0/480839175 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc13806b4d0 0x7fc13806d990 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:43.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:43.150+0000 7fc15ee00640 1 --2- 172.21.15.6:0/480839175 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc158145300 0x7fc1581f40a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:43.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:43.150+0000 7fc15ee00640 1 --2- 172.21.15.6:0/480839175 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc1580cce90 0x7fc1581f3b60 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:43.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:43.150+0000 7fc15ee00640 1 --2- 172.21.15.6:0/480839175 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc158070f10 0x7fc15815b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:43.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:43.150+0000 7fc15ee00640 1 -- 172.21.15.6:0/480839175 >> 172.21.15.6:0/480839175 conn(0x7fc15806b3d0 msgr2=0x7fc1580b17c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:43.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:43.150+0000 7fc15ee00640 1 -- 172.21.15.6:0/480839175 shutdown_connections 2024-03-04T09:28:43.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:43.150+0000 7fc15ee00640 1 -- 172.21.15.6:0/480839175 wait complete. 2024-03-04T09:28:43.170 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-04T09:28:43.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:43.405+0000 7fd5aa600640 1 Processor -- start 2024-03-04T09:28:43.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:43.405+0000 7fd5aa600640 1 -- start start 2024-03-04T09:28:43.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:43.406+0000 7fd5aa600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd5a406da40 0x7fd5a406e340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:43.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:43.406+0000 7fd5aa600640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd5a415e680 0x7fd5a406c980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:43.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:43.407+0000 7fd5aa600640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5a415eb80 con 0x7fd5a40ccad0 2024-03-04T09:28:43.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:43.407+0000 7fd5aa600640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5a4161c70 con 0x7fd5a415e680 2024-03-04T09:28:43.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:43.407+0000 7fd5aa600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5a415ee60 con 0x7fd5a406da40 2024-03-04T09:28:43.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:43.407+0000 7fd5a3400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd5a406da40 0x7fd5a406e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:43.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:43.408+0000 7fd5a3400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd5a406da40 0x7fd5a406e340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.6:53632/0 (socket says 172.21.15.6:53632) 2024-03-04T09:28:43.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:43.408+0000 7fd5a3400640 1 -- 172.21.15.6:0/2643794470 learned_addr learned my addr 172.21.15.6:0/2643794470 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:28:43.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:43.408+0000 7fd5a2a00640 1 --2- 172.21.15.6:0/2643794470 >> v2:172.21.15.146:3300/0 conn(0x7fd5a415e680 0x7fd5a406c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:43.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:43.408+0000 7fd5a3400640 1 -- 172.21.15.6:0/2643794470 >> v2:172.21.15.146:3300/0 conn(0x7fd5a415e680 msgr2=0x7fd5a406c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:43.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:43.408+0000 7fd5a3400640 1 --2- 172.21.15.6:0/2643794470 >> v2:172.21.15.146:3300/0 conn(0x7fd5a415e680 0x7fd5a406c980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:43.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:43.408+0000 7fd5a3400640 1 -- 172.21.15.6:0/2643794470 >> v1:172.21.15.146:6789/0 conn(0x7fd5a40ccad0 legacy=0x7fd5a4162100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:43.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:43.409+0000 7fd5a3400640 1 -- 172.21.15.6:0/2643794470 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5a415f140 con 0x7fd5a406da40 2024-03-04T09:28:43.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:43.409+0000 7fd5a2a00640 1 --2- 172.21.15.6:0/2643794470 >> v2:172.21.15.146:3300/0 conn(0x7fd5a415e680 0x7fd5a406c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T09:28:43.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:43.410+0000 7fd5a3400640 1 --2- 172.21.15.6:0/2643794470 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd5a406da40 0x7fd5a406e340 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fd594009870 tx=0x7fd59402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=645222270739836c server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:43.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:43.411+0000 7fd5a2000640 1 -- 172.21.15.6:0/2643794470 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd594034070 con 0x7fd5a406da40 2024-03-04T09:28:43.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:43.411+0000 7fd5a2000640 1 -- 172.21.15.6:0/2643794470 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd594030840 con 0x7fd5a406da40 2024-03-04T09:28:43.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:43.411+0000 7fd5a2000640 1 -- 172.21.15.6:0/2643794470 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd594030c10 con 0x7fd5a406da40 2024-03-04T09:28:43.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:43.412+0000 7fd5aa600640 1 -- 172.21.15.6:0/2643794470 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd5a406da40 msgr2=0x7fd5a406e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:43.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:43.412+0000 7fd5aa600640 1 --2- 172.21.15.6:0/2643794470 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd5a406da40 0x7fd5a406e340 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fd594009870 tx=0x7fd59402b980 comp rx=0 tx=0).stop 2024-03-04T09:28:43.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:43.412+0000 7fd5aa600640 1 -- 172.21.15.6:0/2643794470 shutdown_connections 2024-03-04T09:28:43.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:43.412+0000 7fd5aa600640 1 --2- 172.21.15.6:0/2643794470 >> v2:172.21.15.146:3300/0 conn(0x7fd5a415e680 0x7fd5a406c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:43.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:43.412+0000 7fd5aa600640 1 --2- 172.21.15.6:0/2643794470 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd5a406da40 0x7fd5a406e340 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:43.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:43.412+0000 7fd5aa600640 1 -- 172.21.15.6:0/2643794470 >> 172.21.15.6:0/2643794470 conn(0x7fd5a406b3d0 msgr2=0x7fd5a406b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:43.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:43.412+0000 7fd5aa600640 1 -- 172.21.15.6:0/2643794470 shutdown_connections 2024-03-04T09:28:43.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:43.412+0000 7fd5aa600640 1 -- 172.21.15.6:0/2643794470 wait complete. 2024-03-04T09:28:43.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:43.415+0000 7fd5aa600640 1 Processor -- start 2024-03-04T09:28:43.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:43.415+0000 7fd5aa600640 1 -- start start 2024-03-04T09:28:43.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:43.416+0000 7fd5aa600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd5a406da40 0x7fd5a409c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:43.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:43.416+0000 7fd5aa600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd5a40ccad0 0x7fd5a4141730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:43.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:43.416+0000 7fd5aa600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd5a415e680 0x7fd5a409acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:43.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:43.416+0000 7fd5aa600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5a406ea00 con 0x7fd5a40ccad0 2024-03-04T09:28:43.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:43.416+0000 7fd5aa600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5a406e880 con 0x7fd5a415e680 2024-03-04T09:28:43.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:43.416+0000 7fd5aa600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5a406cec0 con 0x7fd5a406da40 2024-03-04T09:28:43.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:43.417+0000 7fd5a2a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd5a40ccad0 0x7fd5a4141730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:43.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:43.417+0000 7fd5a2a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd5a40ccad0 0x7fd5a4141730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:44230/0 (socket says 172.21.15.6:44230) 2024-03-04T09:28:43.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:43.417+0000 7fd5a2a00640 1 -- 172.21.15.6:0/2310609804 learned_addr learned my addr 172.21.15.6:0/2310609804 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:28:43.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:43.417+0000 7fd5a3400640 1 --2- 172.21.15.6:0/2310609804 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd5a406da40 0x7fd5a409c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:43.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:43.417+0000 7fd5a3e00640 1 --2- 172.21.15.6:0/2310609804 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd5a415e680 0x7fd5a409acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:43.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:43.417+0000 7fd5a2a00640 1 -- 172.21.15.6:0/2310609804 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd5a406da40 msgr2=0x7fd5a409c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:43.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:43.417+0000 7fd5a2a00640 1 --2- 172.21.15.6:0/2310609804 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd5a406da40 0x7fd5a409c7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:43.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:43.417+0000 7fd5a2a00640 1 -- 172.21.15.6:0/2310609804 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd5a415e680 msgr2=0x7fd5a409acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:43.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:43.417+0000 7fd5a2a00640 1 --2- 172.21.15.6:0/2310609804 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd5a415e680 0x7fd5a409acc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:43.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:43.417+0000 7fd5a2a00640 1 -- 172.21.15.6:0/2310609804 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5a409cce0 con 0x7fd5a40ccad0 2024-03-04T09:28:43.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:43.417+0000 7fd5a3400640 1 --2- 172.21.15.6:0/2310609804 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd5a406da40 0x7fd5a409c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:28:43.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:43.417+0000 7fd5a3e00640 1 --2- 172.21.15.6:0/2310609804 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd5a415e680 0x7fd5a409acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:28:43.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:43.418+0000 7fd5a2a00640 1 --2- 172.21.15.6:0/2310609804 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd5a40ccad0 0x7fd5a4141730 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fd59000e990 tx=0x7fd59000ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:43.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:43.418+0000 7fd59b400640 1 -- 172.21.15.6:0/2310609804 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd590017070 con 0x7fd5a40ccad0 2024-03-04T09:28:43.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:43.418+0000 7fd59b400640 1 -- 172.21.15.6:0/2310609804 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd59000c9e0 con 0x7fd5a40ccad0 2024-03-04T09:28:43.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:43.418+0000 7fd59b400640 1 -- 172.21.15.6:0/2310609804 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd59000cdb0 con 0x7fd5a40ccad0 2024-03-04T09:28:43.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:43.418+0000 7fd5aa600640 1 -- 172.21.15.6:0/2310609804 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5a4141c70 con 0x7fd5a40ccad0 2024-03-04T09:28:43.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:43.418+0000 7fd5aa600640 1 -- 172.21.15.6:0/2310609804 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5a41f7640 con 0x7fd5a40ccad0 2024-03-04T09:28:43.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:43.419+0000 7fd5aa600640 1 -- 172.21.15.6:0/2310609804 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5a4067830 con 0x7fd5a40ccad0 2024-03-04T09:28:43.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:43.420+0000 7fd59b400640 1 -- 172.21.15.6:0/2310609804 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd59001a070 con 0x7fd5a40ccad0 2024-03-04T09:28:43.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:43.421+0000 7fd59b400640 1 --2- 172.21.15.6:0/2310609804 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd58406b4d0 0x7fd58406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:43.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:43.421+0000 7fd59b400640 1 -- 172.21.15.6:0/2310609804 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fd590089e60 con 0x7fd5a40ccad0 2024-03-04T09:28:43.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:43.422+0000 7fd5a3400640 1 --2- 172.21.15.6:0/2310609804 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd58406b4d0 0x7fd58406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:43.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:43.423+0000 7fd5a3400640 1 --2- 172.21.15.6:0/2310609804 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd58406b4d0 0x7fd58406d990 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fd594036000 tx=0x7fd59403b040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:43.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:43.426+0000 7fd59b400640 1 -- 172.21.15.6:0/2310609804 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd5a4067830 con 0x7fd5a40ccad0 2024-03-04T09:28:43.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:43.627+0000 7fd5aa600640 1 -- 172.21.15.6:0/2310609804 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fd5a40b4a40 con 0x7fd5a40ccad0 2024-03-04T09:28:43.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:43.634+0000 7fd59b400640 1 -- 172.21.15.6:0/2310609804 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v22)=0 v22) v1 ==== 122+0+0 (secure 0 0 0) 0x7fd590058f40 con 0x7fd5a40ccad0 2024-03-04T09:28:43.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:43.637+0000 7fd5aa600640 1 -- 172.21.15.6:0/2310609804 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd58406b4d0 msgr2=0x7fd58406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:43.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:43.637+0000 7fd5aa600640 1 --2- 172.21.15.6:0/2310609804 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd58406b4d0 0x7fd58406d990 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fd594036000 tx=0x7fd59403b040 comp rx=0 tx=0).stop 2024-03-04T09:28:43.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:43.637+0000 7fd5aa600640 1 -- 172.21.15.6:0/2310609804 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd5a40ccad0 msgr2=0x7fd5a4141730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:43.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:43.637+0000 7fd5aa600640 1 --2- 172.21.15.6:0/2310609804 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd5a40ccad0 0x7fd5a4141730 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fd59000e990 tx=0x7fd59000ee50 comp rx=0 tx=0).stop 2024-03-04T09:28:43.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:43.637+0000 7fd5aa600640 1 -- 172.21.15.6:0/2310609804 shutdown_connections 2024-03-04T09:28:43.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:43.637+0000 7fd5aa600640 1 --2- 172.21.15.6:0/2310609804 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd5a415e680 0x7fd5a409acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:43.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:43.637+0000 7fd5aa600640 1 --2- 172.21.15.6:0/2310609804 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd5a40ccad0 0x7fd5a4141730 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:43.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:43.637+0000 7fd5aa600640 1 --2- 172.21.15.6:0/2310609804 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd58406b4d0 0x7fd58406d990 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:43.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:43.637+0000 7fd5aa600640 1 --2- 172.21.15.6:0/2310609804 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd5a406da40 0x7fd5a409c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:43.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:43.637+0000 7fd5aa600640 1 -- 172.21.15.6:0/2310609804 >> 172.21.15.6:0/2310609804 conn(0x7fd5a406b3d0 msgr2=0x7fd5a4073f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:43.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:43.637+0000 7fd5aa600640 1 -- 172.21.15.6:0/2310609804 shutdown_connections 2024-03-04T09:28:43.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:43.638+0000 7fd5aa600640 1 -- 172.21.15.6:0/2310609804 wait complete. 2024-03-04T09:28:43.655 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-04T09:28:43.655 DEBUG:teuthology.orchestra.run.smithi006:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-04T09:28:43.762 DEBUG:teuthology.orchestra.run.smithi146:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-04T09:28:43.880 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-04T09:28:43.881 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-04T09:28:43.881 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T09:28:43.942 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054202,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249800k,nr_inodes=812450,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-04T09:28:43.942 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-04T09:28:43.942 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T09:28:44.000 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053429,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249180k,nr_inodes=812295,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-04T09:28:44.000 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-04T09:28:44.000 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T09:28:44.061 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054202,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249800k,nr_inodes=812450,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-04T09:28:44.062 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-04T09:28:44.062 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T09:28:44.118 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053429,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249180k,nr_inodes=812295,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-04T09:28:44.119 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-04T09:28:44.119 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:28:44.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.350+0000 7f1927e00640 1 Processor -- start 2024-03-04T09:28:44.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.351+0000 7f1927e00640 1 -- start start 2024-03-04T09:28:44.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.352+0000 7f1927e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f192815e0b0 0x7f19280cf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:44.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.352+0000 7f1927e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19280b3710 con 0x7f19280d0e50 2024-03-04T09:28:44.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.352+0000 7f1927e00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19280b39f0 con 0x7f19280d4a10 2024-03-04T09:28:44.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.352+0000 7f1927e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19280b09e0 con 0x7f192815e0b0 2024-03-04T09:28:44.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.352+0000 7f1926a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f192815e0b0 0x7f19280cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:44.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.352+0000 7f1926a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f192815e0b0 0x7f19280cf310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:44244/0 (socket says 172.21.15.6:44244) 2024-03-04T09:28:44.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.352+0000 7f1926a00640 1 -- 172.21.15.6:0/2714095559 learned_addr learned my addr 172.21.15.6:0/2714095559 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:28:44.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.353+0000 7f1926a00640 1 -- 172.21.15.6:0/2714095559 >> v1:172.21.15.146:6789/0 conn(0x7f19280d4a10 legacy=0x7f19280d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:44.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.353+0000 7f1926a00640 1 -- 172.21.15.6:0/2714095559 >> v1:172.21.15.6:6789/0 conn(0x7f19280d0e50 legacy=0x7f19280b44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:44.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.353+0000 7f1926a00640 1 -- 172.21.15.6:0/2714095559 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19280b3cd0 con 0x7f192815e0b0 2024-03-04T09:28:44.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.354+0000 7f1926a00640 1 --2- 172.21.15.6:0/2714095559 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f192815e0b0 0x7f19280cf310 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f1918009870 tx=0x7f191802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16214222ea755f78 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:44.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.354+0000 7f1925600640 1 -- 172.21.15.6:0/2714095559 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1918034070 con 0x7f192815e0b0 2024-03-04T09:28:44.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.354+0000 7f1925600640 1 -- 172.21.15.6:0/2714095559 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1918030860 con 0x7f192815e0b0 2024-03-04T09:28:44.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.354+0000 7f1925600640 1 -- 172.21.15.6:0/2714095559 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1918030c30 con 0x7f192815e0b0 2024-03-04T09:28:44.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.355+0000 7f1927e00640 1 -- 172.21.15.6:0/2714095559 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f192815e0b0 msgr2=0x7f19280cf310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:44.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.355+0000 7f1927e00640 1 --2- 172.21.15.6:0/2714095559 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f192815e0b0 0x7f19280cf310 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f1918009870 tx=0x7f191802b9a0 comp rx=0 tx=0).stop 2024-03-04T09:28:44.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.355+0000 7f1927e00640 1 -- 172.21.15.6:0/2714095559 shutdown_connections 2024-03-04T09:28:44.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.355+0000 7f1927e00640 1 --2- 172.21.15.6:0/2714095559 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f192815e0b0 0x7f19280cf310 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:44.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.355+0000 7f1927e00640 1 -- 172.21.15.6:0/2714095559 >> 172.21.15.6:0/2714095559 conn(0x7f19280d3d30 msgr2=0x7f19280d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:44.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.355+0000 7f1927e00640 1 -- 172.21.15.6:0/2714095559 shutdown_connections 2024-03-04T09:28:44.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.355+0000 7f1927e00640 1 -- 172.21.15.6:0/2714095559 wait complete. 2024-03-04T09:28:44.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.357+0000 7f1927e00640 1 Processor -- start 2024-03-04T09:28:44.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.357+0000 7f1927e00640 1 -- start start 2024-03-04T09:28:44.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.358+0000 7f1927e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f19280d0e50 0x7f192806cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:44.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.358+0000 7f1927e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f19280d4a10 0x7f19281f3b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:44.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.358+0000 7f1927e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f192815e0b0 0x7f192815eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:44.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.358+0000 7f1927e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19280cfc60 con 0x7f192815e0b0 2024-03-04T09:28:44.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.358+0000 7f1927e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f192815e490 con 0x7f19280d4a10 2024-03-04T09:28:44.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.358+0000 7f1927e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19280cf850 con 0x7f19280d0e50 2024-03-04T09:28:44.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.358+0000 7f1927400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f192815e0b0 0x7f192815eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:44.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.358+0000 7f1927400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f192815e0b0 0x7f192815eeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:44254/0 (socket says 172.21.15.6:44254) 2024-03-04T09:28:44.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.358+0000 7f1927400640 1 -- 172.21.15.6:0/4278268424 learned_addr learned my addr 172.21.15.6:0/4278268424 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:28:44.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.359+0000 7f1926000640 1 --2- 172.21.15.6:0/4278268424 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f19280d4a10 0x7f19281f3b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:44.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.359+0000 7f1926a00640 1 --2- 172.21.15.6:0/4278268424 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f19280d0e50 0x7f192806cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:44.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.359+0000 7f1927400640 1 -- 172.21.15.6:0/4278268424 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f19280d0e50 msgr2=0x7f192806cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:44.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.359+0000 7f1927400640 1 --2- 172.21.15.6:0/4278268424 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f19280d0e50 0x7f192806cbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:44.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.359+0000 7f1927400640 1 -- 172.21.15.6:0/4278268424 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f19280d4a10 msgr2=0x7f19281f3b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:44.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.359+0000 7f1927400640 1 --2- 172.21.15.6:0/4278268424 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f19280d4a10 0x7f19281f3b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:44.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.359+0000 7f1927400640 1 -- 172.21.15.6:0/4278268424 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19281f47b0 con 0x7f192815e0b0 2024-03-04T09:28:44.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.359+0000 7f1927400640 1 --2- 172.21.15.6:0/4278268424 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f192815e0b0 0x7f192815eeb0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f192000c3c0 tx=0x7f192000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:44.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.359+0000 7f191ea00640 1 -- 172.21.15.6:0/4278268424 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19200052d0 con 0x7f192815e0b0 2024-03-04T09:28:44.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.359+0000 7f191ea00640 1 -- 172.21.15.6:0/4278268424 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1920005470 con 0x7f192815e0b0 2024-03-04T09:28:44.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.359+0000 7f1927e00640 1 -- 172.21.15.6:0/4278268424 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19281f4a90 con 0x7f192815e0b0 2024-03-04T09:28:44.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.359+0000 7f191ea00640 1 -- 172.21.15.6:0/4278268424 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1920007260 con 0x7f192815e0b0 2024-03-04T09:28:44.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.360+0000 7f1927e00640 1 -- 172.21.15.6:0/4278268424 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f192815e990 con 0x7f192815e0b0 2024-03-04T09:28:44.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.361+0000 7f1927e00640 1 -- 172.21.15.6:0/4278268424 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f192804f330 con 0x7f192815e0b0 2024-03-04T09:28:44.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.361+0000 7f191ea00640 1 -- 172.21.15.6:0/4278268424 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1920007400 con 0x7f192815e0b0 2024-03-04T09:28:44.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.362+0000 7f191ea00640 1 --2- 172.21.15.6:0/4278268424 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f190406b4d0 0x7f190406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:44.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.363+0000 7f191ea00640 1 -- 172.21.15.6:0/4278268424 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f1920089310 con 0x7f192815e0b0 2024-03-04T09:28:44.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.363+0000 7f1926a00640 1 --2- 172.21.15.6:0/4278268424 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f190406b4d0 0x7f190406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:44.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.364+0000 7f1926a00640 1 --2- 172.21.15.6:0/4278268424 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f190406b4d0 0x7f190406d990 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f1918036000 tx=0x7f191803b040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:44.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.367+0000 7f191ea00640 1 -- 172.21.15.6:0/4278268424 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1920058480 con 0x7f192815e0b0 2024-03-04T09:28:44.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.605+0000 7f1927e00640 1 -- 172.21.15.6:0/4278268424 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f192806d220 con 0x7f192815e0b0 2024-03-04T09:28:44.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.605+0000 7f191ea00640 1 -- 172.21.15.6:0/4278268424 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+3693 (secure 0 0 0) 0x7f1920057d80 con 0x7f192815e0b0 2024-03-04T09:28:44.602 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:28:44.602 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":36,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5030,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/1411942502","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1411942502},{"type":"v1","addr":"172.21.15.6:6834","nonce":1411942502}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28},{"gid":5039,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/1628166413","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":1628166413},{"type":"v1","addr":"172.21.15.146:6837","nonce":1628166413}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":5323,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/131491768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":131491768},{"type":"v1","addr":"172.21.15.6:6835","nonce":131491768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":5518,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6836/335797471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":335797471},{"type":"v1","addr":"172.21.15.146:6836","nonce":335797471}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36}],"filesystems":[]} 2024-03-04T09:28:44.603 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 36 2024-03-04T09:28:44.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.608+0000 7f1927e00640 1 -- 172.21.15.6:0/4278268424 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f190406b4d0 msgr2=0x7f190406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:44.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.608+0000 7f1927e00640 1 --2- 172.21.15.6:0/4278268424 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f190406b4d0 0x7f190406d990 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f1918036000 tx=0x7f191803b040 comp rx=0 tx=0).stop 2024-03-04T09:28:44.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.608+0000 7f1927e00640 1 -- 172.21.15.6:0/4278268424 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f192815e0b0 msgr2=0x7f192815eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:44.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.608+0000 7f1927e00640 1 --2- 172.21.15.6:0/4278268424 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f192815e0b0 0x7f192815eeb0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f192000c3c0 tx=0x7f192000c880 comp rx=0 tx=0).stop 2024-03-04T09:28:44.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.608+0000 7f1927e00640 1 -- 172.21.15.6:0/4278268424 shutdown_connections 2024-03-04T09:28:44.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.608+0000 7f1927e00640 1 --2- 172.21.15.6:0/4278268424 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f192815e0b0 0x7f192815eeb0 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:44.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.608+0000 7f1927e00640 1 --2- 172.21.15.6:0/4278268424 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f19280d4a10 0x7f19281f3b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:44.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.608+0000 7f1927e00640 1 --2- 172.21.15.6:0/4278268424 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f190406b4d0 0x7f190406d990 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:44.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.608+0000 7f1927e00640 1 --2- 172.21.15.6:0/4278268424 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f19280d0e50 0x7f192806cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:44.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.608+0000 7f1927e00640 1 -- 172.21.15.6:0/4278268424 >> 172.21.15.6:0/4278268424 conn(0x7f19280d3d30 msgr2=0x7f19280b52e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:44.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.608+0000 7f1927e00640 1 -- 172.21.15.6:0/4278268424 shutdown_connections 2024-03-04T09:28:44.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.608+0000 7f1927e00640 1 -- 172.21.15.6:0/4278268424 wait complete. 2024-03-04T09:28:44.626 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-04T09:28:44.626 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-04T09:28:44.626 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-04T09:28:44.626 INFO:teuthology.orchestra.run:waiting for 300 2024-03-04T09:28:44.627 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-04T09:28:44.627 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-04T09:28:44.627 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-04T09:28:44.627 INFO:teuthology.orchestra.run:waiting for 300 2024-03-04T09:28:44.627 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-04T09:28:44.627 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-04T09:28:44.627 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-04T09:28:44.627 INFO:teuthology.orchestra.run:waiting for 300 2024-03-04T09:28:44.628 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-04T09:28:44.628 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-04T09:28:44.628 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-04T09:28:44.628 INFO:teuthology.orchestra.run:waiting for 300 2024-03-04T09:28:44.649 INFO:tasks.ceph.mds.c:Stopped 2024-03-04T09:28:44.649 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-04T09:28:44.652 INFO:tasks.ceph.mds.b:Stopped 2024-03-04T09:28:44.652 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-04T09:28:44.654 INFO:tasks.ceph.mds.a:Stopped 2024-03-04T09:28:44.654 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-04T09:28:44.655 INFO:tasks.ceph.mds.d:Stopped 2024-03-04T09:28:44.655 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-04T09:28:44.700 INFO:tasks.ceph.mds.c:Started 2024-03-04T09:28:44.701 DEBUG:teuthology.parallel:result is None 2024-03-04T09:28:44.702 INFO:tasks.ceph.mds.b:Started 2024-03-04T09:28:44.703 DEBUG:teuthology.parallel:result is None 2024-03-04T09:28:44.707 INFO:tasks.ceph.mds.a:Started 2024-03-04T09:28:44.708 DEBUG:teuthology.parallel:result is None 2024-03-04T09:28:44.708 INFO:tasks.ceph.mds.d:Started 2024-03-04T09:28:44.708 DEBUG:teuthology.parallel:result is None 2024-03-04T09:28:44.709 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-04T09:28:44.862 INFO:tasks.ceph.mds.b.smithi146.stdout:starting mds.b at 2024-03-04T09:28:44.863 INFO:tasks.ceph.mds.a.smithi006.stdout:starting mds.a at 2024-03-04T09:28:44.864 INFO:tasks.ceph.mds.c.smithi006.stdout:starting mds.c at 2024-03-04T09:28:44.872 INFO:tasks.ceph.mds.d.smithi146.stdout:starting mds.d at 2024-03-04T09:28:44.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.901+0000 7f0be2e00640 1 Processor -- start 2024-03-04T09:28:44.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.901+0000 7f0be2e00640 1 -- start start 2024-03-04T09:28:44.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.902+0000 7f0be2e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0bdc15e070 0x7f0bdc0d2960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:44.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.902+0000 7f0be2e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f0bdc0cf410 0x7f0bdc0cf870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:44.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.902+0000 7f0be2e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f0bdc0d05c0 0x7f0bdc0d0a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:44.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.902+0000 7f0be2e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bdc0d18c0 con 0x7f0bdc0cf410 2024-03-04T09:28:44.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.902+0000 7f0be2e00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bdc0d1ba0 con 0x7f0bdc0d05c0 2024-03-04T09:28:44.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.902+0000 7f0be2e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bdc0d1e80 con 0x7f0bdc15e070 2024-03-04T09:28:44.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.902+0000 7f0be1000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f0bdc0cf410 0x7f0bdc0cf870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:44.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.902+0000 7f0be2400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f0bdc0d05c0 0x7f0bdc0d0a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:44.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.903+0000 7f0be1a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0bdc15e070 0x7f0bdc0d2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:44.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.903+0000 7f0be1000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f0bdc0cf410 0x7f0bdc0cf870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:44280/0 (socket says 172.21.15.6:44280) 2024-03-04T09:28:44.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.903+0000 7f0be1000640 1 -- 172.21.15.6:0/1594499397 learned_addr learned my addr 172.21.15.6:0/1594499397 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:28:44.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.903+0000 7f0be1000640 1 -- 172.21.15.6:0/1594499397 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0bdc15e070 msgr2=0x7f0bdc0d2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:44.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.903+0000 7f0be1000640 1 --2- 172.21.15.6:0/1594499397 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0bdc15e070 0x7f0bdc0d2960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:44.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.903+0000 7f0be1000640 1 -- 172.21.15.6:0/1594499397 >> v2:172.21.15.146:3300/0 conn(0x7f0bdc0d05c0 msgr2=0x7f0bdc0d0a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:44.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.903+0000 7f0be1000640 1 --2- 172.21.15.6:0/1594499397 >> v2:172.21.15.146:3300/0 conn(0x7f0bdc0d05c0 0x7f0bdc0d0a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:44.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.903+0000 7f0be1000640 1 -- 172.21.15.6:0/1594499397 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bdc0d2160 con 0x7f0bdc0cf410 2024-03-04T09:28:44.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.904+0000 7f0be1000640 1 --2- 172.21.15.6:0/1594499397 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0bdc0cf410 0x7f0bdc0cf870 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f0bdc0d3bc0 tx=0x7f0bc802b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a00271fd810c5d19 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:44.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.904+0000 7f0bdbe00640 1 -- 172.21.15.6:0/1594499397 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bc8035020 con 0x7f0bdc0cf410 2024-03-04T09:28:44.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.904+0000 7f0bdbe00640 1 -- 172.21.15.6:0/1594499397 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bc802d040 con 0x7f0bdc0cf410 2024-03-04T09:28:44.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.904+0000 7f0bdbe00640 1 -- 172.21.15.6:0/1594499397 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bc8031a80 con 0x7f0bdc0cf410 2024-03-04T09:28:44.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.905+0000 7f0be2e00640 1 -- 172.21.15.6:0/1594499397 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0bdc0cf410 msgr2=0x7f0bdc0cf870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:44.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.905+0000 7f0be2e00640 1 --2- 172.21.15.6:0/1594499397 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0bdc0cf410 0x7f0bdc0cf870 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f0bdc0d3bc0 tx=0x7f0bc802b900 comp rx=0 tx=0).stop 2024-03-04T09:28:44.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.905+0000 7f0be2e00640 1 -- 172.21.15.6:0/1594499397 shutdown_connections 2024-03-04T09:28:44.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.905+0000 7f0be2e00640 1 --2- 172.21.15.6:0/1594499397 >> v2:172.21.15.146:3300/0 conn(0x7f0bdc0d05c0 0x7f0bdc0d0a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:44.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.905+0000 7f0be2e00640 1 --2- 172.21.15.6:0/1594499397 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0bdc0cf410 0x7f0bdc0cf870 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:44.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.905+0000 7f0be2e00640 1 --2- 172.21.15.6:0/1594499397 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0bdc15e070 0x7f0bdc0d2960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:44.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.905+0000 7f0be2e00640 1 -- 172.21.15.6:0/1594499397 >> 172.21.15.6:0/1594499397 conn(0x7f0bdc08d2a0 msgr2=0x7f0bdc0d3cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:44.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.905+0000 7f0be2e00640 1 -- 172.21.15.6:0/1594499397 shutdown_connections 2024-03-04T09:28:44.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.905+0000 7f0be2e00640 1 -- 172.21.15.6:0/1594499397 wait complete. 2024-03-04T09:28:44.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.907+0000 7f0be2e00640 1 Processor -- start 2024-03-04T09:28:44.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.907+0000 7f0be2e00640 1 -- start start 2024-03-04T09:28:44.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.908+0000 7f0be2e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0bdc0cf410 0x7f0bdc0c87f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:44.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.908+0000 7f0be2e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0bdc0d05c0 0x7f0bdc155330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:44.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.908+0000 7f0be2e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0bdc15e070 0x7f0bdc0b15d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:44.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.908+0000 7f0be2e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bdc0d4730 con 0x7f0bdc0d05c0 2024-03-04T09:28:44.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.908+0000 7f0be2e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bdc0d45b0 con 0x7f0bdc15e070 2024-03-04T09:28:44.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.908+0000 7f0be2e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bdc0b3630 con 0x7f0bdc0cf410 2024-03-04T09:28:44.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.908+0000 7f0be1000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0bdc0d05c0 0x7f0bdc155330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:44.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.908+0000 7f0be1000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0bdc0d05c0 0x7f0bdc155330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:44296/0 (socket says 172.21.15.6:44296) 2024-03-04T09:28:44.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.908+0000 7f0be1000640 1 -- 172.21.15.6:0/3934071057 learned_addr learned my addr 172.21.15.6:0/3934071057 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:28:44.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.908+0000 7f0be1a00640 1 --2- 172.21.15.6:0/3934071057 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0bdc0cf410 0x7f0bdc0c87f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:44.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.908+0000 7f0be2400640 1 --2- 172.21.15.6:0/3934071057 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0bdc15e070 0x7f0bdc0b15d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:44.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.909+0000 7f0be1000640 1 -- 172.21.15.6:0/3934071057 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0bdc0cf410 msgr2=0x7f0bdc0c87f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:44.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.909+0000 7f0be1000640 1 --2- 172.21.15.6:0/3934071057 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0bdc0cf410 0x7f0bdc0c87f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:44.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.909+0000 7f0be1000640 1 -- 172.21.15.6:0/3934071057 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0bdc15e070 msgr2=0x7f0bdc0b15d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:44.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.909+0000 7f0be1000640 1 --2- 172.21.15.6:0/3934071057 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0bdc15e070 0x7f0bdc0b15d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:44.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.909+0000 7f0be1000640 1 -- 172.21.15.6:0/3934071057 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bdc0c7a30 con 0x7f0bdc0d05c0 2024-03-04T09:28:44.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.909+0000 7f0be1000640 1 --2- 172.21.15.6:0/3934071057 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0bdc0d05c0 0x7f0bdc155330 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f0bc802b940 tx=0x7f0bc80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:44.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.910+0000 7f0bd9600640 1 -- 172.21.15.6:0/3934071057 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bc803a930 con 0x7f0bdc0d05c0 2024-03-04T09:28:44.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.910+0000 7f0bd9600640 1 -- 172.21.15.6:0/3934071057 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bc803aad0 con 0x7f0bdc0d05c0 2024-03-04T09:28:44.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.910+0000 7f0bd9600640 1 -- 172.21.15.6:0/3934071057 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bc80443f0 con 0x7f0bdc0d05c0 2024-03-04T09:28:44.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.910+0000 7f0be2e00640 1 -- 172.21.15.6:0/3934071057 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bdc0c7d10 con 0x7f0bdc0d05c0 2024-03-04T09:28:44.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.910+0000 7f0be2e00640 1 -- 172.21.15.6:0/3934071057 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bdc0c82d0 con 0x7f0bdc0d05c0 2024-03-04T09:28:44.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.911+0000 7f0be2e00640 1 -- 172.21.15.6:0/3934071057 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bdc04f330 con 0x7f0bdc0d05c0 2024-03-04T09:28:44.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.912+0000 7f0bd9600640 1 -- 172.21.15.6:0/3934071057 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0bc8038020 con 0x7f0bdc0d05c0 2024-03-04T09:28:44.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.914+0000 7f0bd9600640 1 --2- 172.21.15.6:0/3934071057 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f0bbc06b4d0 0x7f0bbc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:44.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.914+0000 7f0bd9600640 1 -- 172.21.15.6:0/3934071057 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f0bc80abd00 con 0x7f0bdc0d05c0 2024-03-04T09:28:44.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.914+0000 7f0be1a00640 1 --2- 172.21.15.6:0/3934071057 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f0bbc06b4d0 0x7f0bbc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:44.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.915+0000 7f0be1a00640 1 --2- 172.21.15.6:0/3934071057 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f0bbc06b4d0 0x7f0bbc06d990 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f0bd0009aa0 tx=0x7f0bd0009190 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:44.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:44.919+0000 7f0bd9600640 1 -- 172.21.15.6:0/3934071057 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0bc803f070 con 0x7f0bdc0d05c0 2024-03-04T09:28:45.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:45.129+0000 7f0be2e00640 1 -- 172.21.15.6:0/3934071057 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f0bdc0c84a0 con 0x7f0bdc0d05c0 2024-03-04T09:28:46.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.186+0000 7f0bd9600640 1 -- 172.21.15.6:0/3934071057 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v64) v1 ==== 97+0+0 (secure 0 0 0) 0x7f0bc807a790 con 0x7f0bdc0d05c0 2024-03-04T09:28:46.183 INFO:teuthology.orchestra.run.smithi006.stderr: removed all blocklist entries 2024-03-04T09:28:46.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.190+0000 7f0be2e00640 1 -- 172.21.15.6:0/3934071057 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f0bbc06b4d0 msgr2=0x7f0bbc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:46.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.191+0000 7f0be2e00640 1 --2- 172.21.15.6:0/3934071057 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f0bbc06b4d0 0x7f0bbc06d990 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f0bd0009aa0 tx=0x7f0bd0009190 comp rx=0 tx=0).stop 2024-03-04T09:28:46.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.191+0000 7f0be2e00640 1 -- 172.21.15.6:0/3934071057 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0bdc0d05c0 msgr2=0x7f0bdc155330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:46.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.191+0000 7f0be2e00640 1 --2- 172.21.15.6:0/3934071057 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0bdc0d05c0 0x7f0bdc155330 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f0bc802b940 tx=0x7f0bc80046c0 comp rx=0 tx=0).stop 2024-03-04T09:28:46.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.191+0000 7f0be2e00640 1 -- 172.21.15.6:0/3934071057 shutdown_connections 2024-03-04T09:28:46.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.191+0000 7f0be2e00640 1 --2- 172.21.15.6:0/3934071057 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0bdc15e070 0x7f0bdc0b15d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:46.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.191+0000 7f0be2e00640 1 --2- 172.21.15.6:0/3934071057 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0bdc0d05c0 0x7f0bdc155330 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:46.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.191+0000 7f0be2e00640 1 --2- 172.21.15.6:0/3934071057 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f0bbc06b4d0 0x7f0bbc06d990 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:46.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.191+0000 7f0be2e00640 1 --2- 172.21.15.6:0/3934071057 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0bdc0cf410 0x7f0bdc0c87f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:46.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.191+0000 7f0be2e00640 1 -- 172.21.15.6:0/3934071057 >> 172.21.15.6:0/3934071057 conn(0x7f0bdc08d2a0 msgr2=0x7f0bdc0d2960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:46.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.191+0000 7f0be2e00640 1 -- 172.21.15.6:0/3934071057 shutdown_connections 2024-03-04T09:28:46.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.191+0000 7f0be2e00640 1 -- 172.21.15.6:0/3934071057 wait complete. 2024-03-04T09:28:46.208 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-04T09:28:46.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.429+0000 7f8183200640 1 Processor -- start 2024-03-04T09:28:46.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.429+0000 7f8183200640 1 -- start start 2024-03-04T09:28:46.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.430+0000 7f8183200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f817c0b3c90 0x7f817c0cdc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:46.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.430+0000 7f8183200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f817c0cf5c0 0x7f817c0b4070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:46.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.431+0000 7f8183200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f817c0b45b0 con 0x7f817c06cb10 2024-03-04T09:28:46.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.431+0000 7f8183200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f817c0ce160 con 0x7f817c0b3c90 2024-03-04T09:28:46.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.431+0000 7f8183200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f817c0b4860 con 0x7f817c0cf5c0 2024-03-04T09:28:46.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.431+0000 7f817be00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f817c0b3c90 0x7f817c0cdc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:46.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.431+0000 7f817be00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f817c0b3c90 0x7f817c0cdc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:44302/0 (socket says 172.21.15.6:44302) 2024-03-04T09:28:46.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.431+0000 7f817be00640 1 -- 172.21.15.6:0/3940980059 learned_addr learned my addr 172.21.15.6:0/3940980059 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:28:46.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.431+0000 7f817be00640 1 -- 172.21.15.6:0/3940980059 >> v2:172.21.15.146:3300/0 conn(0x7f817c0cf5c0 msgr2=0x7f817c0b4070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:46.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.431+0000 7f817be00640 1 --2- 172.21.15.6:0/3940980059 >> v2:172.21.15.146:3300/0 conn(0x7f817c0cf5c0 0x7f817c0b4070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:46.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.431+0000 7f817be00640 1 -- 172.21.15.6:0/3940980059 >> v1:172.21.15.6:6789/0 conn(0x7f817c06cb10 legacy=0x7f817c06cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:46.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.431+0000 7f817be00640 1 -- 172.21.15.6:0/3940980059 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f817c0b4b40 con 0x7f817c0b3c90 2024-03-04T09:28:46.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.432+0000 7f817be00640 1 --2- 172.21.15.6:0/3940980059 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f817c0b3c90 0x7f817c0cdc20 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f817c0b2aa0 tx=0x7f816c02ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9b05b15eb996205 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:46.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.433+0000 7f817aa00640 1 -- 172.21.15.6:0/3940980059 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f816c02f850 con 0x7f817c0b3c90 2024-03-04T09:28:46.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.433+0000 7f817aa00640 1 -- 172.21.15.6:0/3940980059 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f816c02f9f0 con 0x7f817c0b3c90 2024-03-04T09:28:46.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.433+0000 7f817aa00640 1 -- 172.21.15.6:0/3940980059 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f816c02fdc0 con 0x7f817c0b3c90 2024-03-04T09:28:46.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.433+0000 7f8183200640 1 -- 172.21.15.6:0/3940980059 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f817c0b3c90 msgr2=0x7f817c0cdc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:46.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.434+0000 7f8183200640 1 --2- 172.21.15.6:0/3940980059 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f817c0b3c90 0x7f817c0cdc20 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f817c0b2aa0 tx=0x7f816c02ba30 comp rx=0 tx=0).stop 2024-03-04T09:28:46.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.434+0000 7f8183200640 1 -- 172.21.15.6:0/3940980059 shutdown_connections 2024-03-04T09:28:46.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.434+0000 7f8183200640 1 --2- 172.21.15.6:0/3940980059 >> v2:172.21.15.146:3300/0 conn(0x7f817c0cf5c0 0x7f817c0b4070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:46.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.434+0000 7f8183200640 1 --2- 172.21.15.6:0/3940980059 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f817c0b3c90 0x7f817c0cdc20 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:46.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.434+0000 7f8183200640 1 -- 172.21.15.6:0/3940980059 >> 172.21.15.6:0/3940980059 conn(0x7f817c06afb0 msgr2=0x7f817c06b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:46.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.434+0000 7f8183200640 1 -- 172.21.15.6:0/3940980059 shutdown_connections 2024-03-04T09:28:46.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.434+0000 7f8183200640 1 -- 172.21.15.6:0/3940980059 wait complete. 2024-03-04T09:28:46.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.435+0000 7f8183200640 1 Processor -- start 2024-03-04T09:28:46.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.436+0000 7f8183200640 1 -- start start 2024-03-04T09:28:46.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.436+0000 7f8183200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f817c06cb10 0x7f817c09d580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:46.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.436+0000 7f8183200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f817c0b3c90 0x7f817c0ade00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:46.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.436+0000 7f8183200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f817c0cf5c0 0x7f817c0ae340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:46.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.436+0000 7f8183200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f817c15b340 con 0x7f817c0b3c90 2024-03-04T09:28:46.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.436+0000 7f8183200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f817c0b5100 con 0x7f817c06cb10 2024-03-04T09:28:46.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.436+0000 7f8183200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f817c06d690 con 0x7f817c0cf5c0 2024-03-04T09:28:46.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.437+0000 7f817b400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f817c0b3c90 0x7f817c0ade00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:46.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.437+0000 7f817b400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f817c0b3c90 0x7f817c0ade00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:44308/0 (socket says 172.21.15.6:44308) 2024-03-04T09:28:46.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.437+0000 7f817b400640 1 -- 172.21.15.6:0/4199745299 learned_addr learned my addr 172.21.15.6:0/4199745299 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:28:46.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.437+0000 7f817be00640 1 --2- 172.21.15.6:0/4199745299 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f817c06cb10 0x7f817c09d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:46.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.437+0000 7f817b400640 1 -- 172.21.15.6:0/4199745299 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f817c0cf5c0 msgr2=0x7f817c0ae340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:46.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.437+0000 7f8181200640 1 --2- 172.21.15.6:0/4199745299 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f817c0cf5c0 0x7f817c0ae340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:46.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.437+0000 7f817b400640 1 --2- 172.21.15.6:0/4199745299 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f817c0cf5c0 0x7f817c0ae340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:46.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.437+0000 7f817b400640 1 -- 172.21.15.6:0/4199745299 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f817c06cb10 msgr2=0x7f817c09d580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:46.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.437+0000 7f817b400640 1 --2- 172.21.15.6:0/4199745299 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f817c06cb10 0x7f817c09d580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:46.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.437+0000 7f817b400640 1 -- 172.21.15.6:0/4199745299 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f817c1504c0 con 0x7f817c0b3c90 2024-03-04T09:28:46.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.437+0000 7f817be00640 1 --2- 172.21.15.6:0/4199745299 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f817c06cb10 0x7f817c09d580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:28:46.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.437+0000 7f8181200640 1 --2- 172.21.15.6:0/4199745299 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f817c0cf5c0 0x7f817c0ae340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:28:46.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.438+0000 7f817b400640 1 --2- 172.21.15.6:0/4199745299 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f817c0b3c90 0x7f817c0ade00 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f816800b410 tx=0x7f816800b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:46.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.438+0000 7f8173e00640 1 -- 172.21.15.6:0/4199745299 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8168015020 con 0x7f817c0b3c90 2024-03-04T09:28:46.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.438+0000 7f8173e00640 1 -- 172.21.15.6:0/4199745299 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8168011850 con 0x7f817c0b3c90 2024-03-04T09:28:46.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.438+0000 7f8173e00640 1 -- 172.21.15.6:0/4199745299 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8168011c40 con 0x7f817c0b3c90 2024-03-04T09:28:46.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.438+0000 7f8183200640 1 -- 172.21.15.6:0/4199745299 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f817c1507a0 con 0x7f817c0b3c90 2024-03-04T09:28:46.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.438+0000 7f8183200640 1 -- 172.21.15.6:0/4199745299 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f817c0c93a0 con 0x7f817c0b3c90 2024-03-04T09:28:46.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.439+0000 7f8183200640 1 -- 172.21.15.6:0/4199745299 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f817c04f3a0 con 0x7f817c0b3c90 2024-03-04T09:28:46.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.440+0000 7f8173e00640 1 -- 172.21.15.6:0/4199745299 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8168018020 con 0x7f817c0b3c90 2024-03-04T09:28:46.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.442+0000 7f8173e00640 1 --2- 172.21.15.6:0/4199745299 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f815c06b4d0 0x7f815c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:46.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.442+0000 7f8173e00640 1 -- 172.21.15.6:0/4199745299 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f8168089e60 con 0x7f817c0b3c90 2024-03-04T09:28:46.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.442+0000 7f817be00640 1 --2- 172.21.15.6:0/4199745299 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f815c06b4d0 0x7f815c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:46.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.443+0000 7f817be00640 1 --2- 172.21.15.6:0/4199745299 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f815c06b4d0 0x7f815c06d990 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f816c036000 tx=0x7f816c039040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:46.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.446+0000 7f8173e00640 1 -- 172.21.15.6:0/4199745299 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f816801f020 con 0x7f817c0b3c90 2024-03-04T09:28:46.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.715+0000 7f8183200640 1 -- 172.21.15.6:0/4199745299 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f817c0c9150 con 0x7f817c0b3c90 2024-03-04T09:28:46.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.716+0000 7f8173e00640 1 -- 172.21.15.6:0/4199745299 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v12) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f81680172c0 con 0x7f817c0b3c90 2024-03-04T09:28:46.713 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:28:46.713 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-04T09:28:46.713 INFO:teuthology.orchestra.run.smithi006.stdout: "auth_dump": [ 2024-03-04T09:28:46.713 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:28:46.713 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "mds.a", 2024-03-04T09:28:46.713 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQA8k+VlHCzFIBAAnKYlmCMDw45LqnSjxnZY5Q==", 2024-03-04T09:28:46.713 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:28:46.714 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-04T09:28:46.714 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow *", 2024-03-04T09:28:46.714 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow *", 2024-03-04T09:28:46.714 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-04T09:28:46.714 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:28:46.714 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:28:46.714 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:28:46.714 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "mds.b", 2024-03-04T09:28:46.714 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQA9k+VlNY7fCBAAX3z8X72/6LkQD6mwGYT69g==", 2024-03-04T09:28:46.714 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:28:46.714 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-04T09:28:46.714 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow *", 2024-03-04T09:28:46.714 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow *", 2024-03-04T09:28:46.714 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-04T09:28:46.714 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:28:46.715 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:28:46.715 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:28:46.715 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "mds.c", 2024-03-04T09:28:46.715 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQA8k+Vl0pjOMhAA7WcTw8awv4EAsrpP4yncNA==", 2024-03-04T09:28:46.715 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:28:46.715 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-04T09:28:46.715 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow *", 2024-03-04T09:28:46.715 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow *", 2024-03-04T09:28:46.715 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-04T09:28:46.715 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:28:46.715 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:28:46.715 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:28:46.715 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "mds.d", 2024-03-04T09:28:46.715 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQA9k+VlbladGRAABNgUcaqTVYuJfCqy5c/G0g==", 2024-03-04T09:28:46.715 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:28:46.716 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-04T09:28:46.716 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow *", 2024-03-04T09:28:46.716 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow *", 2024-03-04T09:28:46.716 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-04T09:28:46.716 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:28:46.716 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:28:46.716 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:28:46.716 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.0", 2024-03-04T09:28:46.716 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQBCk+VlyjMMExAAqb2FJ9n5PWehfx/6PB19ig==", 2024-03-04T09:28:46.716 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:28:46.716 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-04T09:28:46.716 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-04T09:28:46.716 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-04T09:28:46.716 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:28:46.717 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:28:46.717 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:28:46.717 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.1", 2024-03-04T09:28:46.717 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQBFk+VlXD3cLBAAGclSivcYWDslqN8MLlj7eQ==", 2024-03-04T09:28:46.717 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:28:46.717 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-04T09:28:46.717 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-04T09:28:46.717 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-04T09:28:46.717 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:28:46.717 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:28:46.717 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:28:46.717 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.2", 2024-03-04T09:28:46.717 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQBJk+Vl7d9nCxAAnb3W4EI3Yf3XY8ht3VnKEQ==", 2024-03-04T09:28:46.717 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:28:46.717 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-04T09:28:46.717 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-04T09:28:46.718 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-04T09:28:46.718 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:28:46.718 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:28:46.718 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:28:46.718 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.3", 2024-03-04T09:28:46.718 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQBMk+Vlv+Y1NBAAiRYSUqov2i2sj3iInfpHvA==", 2024-03-04T09:28:46.718 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:28:46.718 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-04T09:28:46.718 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-04T09:28:46.718 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-04T09:28:46.718 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:28:46.718 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:28:46.718 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:28:46.718 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.4", 2024-03-04T09:28:46.718 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQBUk+Vl1OlAARAAxMV4GUGS+RJ6XlrRhrVKcg==", 2024-03-04T09:28:46.719 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:28:46.719 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-04T09:28:46.719 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-04T09:28:46.719 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-04T09:28:46.719 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:28:46.719 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:28:46.719 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:28:46.719 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.5", 2024-03-04T09:28:46.719 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQBXk+Vl1lKLGhAAHlsiRr+1+OSRWhkB+weIJg==", 2024-03-04T09:28:46.719 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:28:46.719 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-04T09:28:46.719 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-04T09:28:46.719 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-04T09:28:46.719 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:28:46.719 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:28:46.720 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:28:46.720 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.6", 2024-03-04T09:28:46.720 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQBak+VlwB+GJBAA3f9KUz+zZlQuUR/ywQfkDA==", 2024-03-04T09:28:46.720 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:28:46.720 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-04T09:28:46.720 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-04T09:28:46.720 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-04T09:28:46.720 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:28:46.720 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:28:46.720 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:28:46.720 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.7", 2024-03-04T09:28:46.720 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQBdk+Vle4o8LhAAko+iH5VqqCsfWkNfKZjGVA==", 2024-03-04T09:28:46.720 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:28:46.720 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-04T09:28:46.721 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-04T09:28:46.721 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-04T09:28:46.721 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:28:46.721 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:28:46.721 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:28:46.721 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "client.0", 2024-03-04T09:28:46.721 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQA9k+Vl4L+rJxAACaDbQ4/Bybp5hAZAnJzlVw==", 2024-03-04T09:28:46.721 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:28:46.721 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-04T09:28:46.721 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow r", 2024-03-04T09:28:46.721 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-04T09:28:46.721 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:28:46.721 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:28:46.721 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:28:46.721 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "client.1", 2024-03-04T09:28:46.721 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQA+k+VlGjkOAhAAJNzB8+tQl3mqCnTgMn9EVw==", 2024-03-04T09:28:46.722 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:28:46.722 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-04T09:28:46.722 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow r", 2024-03-04T09:28:46.722 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow rw", 2024-03-04T09:28:46.722 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow rwx" 2024-03-04T09:28:46.722 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:28:46.722 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:28:46.722 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:28:46.722 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "client.2", 2024-03-04T09:28:46.722 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQA9k+VlUc8DMxAA8pGUtIh+BO6UKrrYnCqVgQ==", 2024-03-04T09:28:46.722 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:28:46.722 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-04T09:28:46.722 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow r", 2024-03-04T09:28:46.722 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow rw", 2024-03-04T09:28:46.722 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow rwx" 2024-03-04T09:28:46.723 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:28:46.723 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:28:46.723 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:28:46.723 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "client.3", 2024-03-04T09:28:46.723 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQA+k+VlzMFKDBAAGr713I2hcmGMuHMIjNknYA==", 2024-03-04T09:28:46.723 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:28:46.723 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-04T09:28:46.723 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow r", 2024-03-04T09:28:46.723 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow rw", 2024-03-04T09:28:46.723 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow rwx" 2024-03-04T09:28:46.723 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:28:46.723 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:28:46.723 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:28:46.723 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "client.admin", 2024-03-04T09:28:46.723 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQA7k+Vllb7NFhAAmnFaNFKBcEri+iJAPnG5Gg==", 2024-03-04T09:28:46.724 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:28:46.724 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow *", 2024-03-04T09:28:46.724 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow *", 2024-03-04T09:28:46.724 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow *", 2024-03-04T09:28:46.724 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-04T09:28:46.724 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:28:46.724 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:28:46.724 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:28:46.724 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "mgr.x", 2024-03-04T09:28:46.724 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQA8k+VlPFatFBAA/o3Ps/uJrcg1738HOGy/FA==", 2024-03-04T09:28:46.724 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:28:46.724 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow *", 2024-03-04T09:28:46.724 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile mgr", 2024-03-04T09:28:46.724 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-04T09:28:46.724 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:28:46.725 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:28:46.725 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:28:46.725 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "mgr.y", 2024-03-04T09:28:46.725 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQA8k+VlEA2ACBAAXofDj3gmqItVTJ1pYLqZyQ==", 2024-03-04T09:28:46.725 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:28:46.725 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow *", 2024-03-04T09:28:46.725 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile mgr", 2024-03-04T09:28:46.725 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-04T09:28:46.725 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:28:46.725 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:28:46.725 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-04T09:28:46.725 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-04T09:28:46.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.718+0000 7f8183200640 1 -- 172.21.15.6:0/4199745299 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f815c06b4d0 msgr2=0x7f815c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:46.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.718+0000 7f8183200640 1 --2- 172.21.15.6:0/4199745299 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f815c06b4d0 0x7f815c06d990 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f816c036000 tx=0x7f816c039040 comp rx=0 tx=0).stop 2024-03-04T09:28:46.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.718+0000 7f8183200640 1 -- 172.21.15.6:0/4199745299 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f817c0b3c90 msgr2=0x7f817c0ade00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:46.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.718+0000 7f8183200640 1 --2- 172.21.15.6:0/4199745299 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f817c0b3c90 0x7f817c0ade00 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f816800b410 tx=0x7f816800b8d0 comp rx=0 tx=0).stop 2024-03-04T09:28:46.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.718+0000 7f8183200640 1 -- 172.21.15.6:0/4199745299 shutdown_connections 2024-03-04T09:28:46.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.718+0000 7f8183200640 1 --2- 172.21.15.6:0/4199745299 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f815c06b4d0 0x7f815c06d990 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:46.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.718+0000 7f8183200640 1 --2- 172.21.15.6:0/4199745299 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f817c0cf5c0 0x7f817c0ae340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:46.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.718+0000 7f8183200640 1 --2- 172.21.15.6:0/4199745299 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f817c0b3c90 0x7f817c0ade00 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:46.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.718+0000 7f8183200640 1 --2- 172.21.15.6:0/4199745299 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f817c06cb10 0x7f817c09d580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:46.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.718+0000 7f8183200640 1 -- 172.21.15.6:0/4199745299 >> 172.21.15.6:0/4199745299 conn(0x7f817c06afb0 msgr2=0x7f817c0cc790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:46.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.719+0000 7f8183200640 1 -- 172.21.15.6:0/4199745299 shutdown_connections 2024-03-04T09:28:46.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.719+0000 7f8183200640 1 -- 172.21.15.6:0/4199745299 wait complete. 2024-03-04T09:28:46.734 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T09:28:46.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.959+0000 7febb1400640 1 Processor -- start 2024-03-04T09:28:46.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.959+0000 7febb1400640 1 -- start start 2024-03-04T09:28:46.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.960+0000 7febb1400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7febac0d4a10 0x7febac0d0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:46.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.960+0000 7febb1400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7febac0d0c80 0x7febac0b44d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:46.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.960+0000 7febb1400640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febac0b3710 con 0x7febac15e0b0 2024-03-04T09:28:46.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.960+0000 7febb1400640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febac0b09e0 con 0x7febac0d4a10 2024-03-04T09:28:46.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.960+0000 7febb1400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febac0cfa20 con 0x7febac0d0c80 2024-03-04T09:28:46.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.961+0000 7febab400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7febac0d4a10 0x7febac0d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:46.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.961+0000 7febab400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7febac0d4a10 0x7febac0d0740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.6:40870/0 (socket says 172.21.15.6:40870) 2024-03-04T09:28:46.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.961+0000 7febab400640 1 -- 172.21.15.6:0/2560253990 learned_addr learned my addr 172.21.15.6:0/2560253990 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:28:46.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.961+0000 7febb0a00640 1 --2- 172.21.15.6:0/2560253990 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7febac0d0c80 0x7febac0b44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:46.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.962+0000 7febab400640 1 -- 172.21.15.6:0/2560253990 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7febac0d0c80 msgr2=0x7febac0b44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:46.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.962+0000 7febab400640 1 --2- 172.21.15.6:0/2560253990 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7febac0d0c80 0x7febac0b44d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:46.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.962+0000 7febab400640 1 -- 172.21.15.6:0/2560253990 >> v1:172.21.15.146:6789/0 conn(0x7febac15e0b0 legacy=0x7febac0cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:46.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.962+0000 7febab400640 1 -- 172.21.15.6:0/2560253990 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febac0b39f0 con 0x7febac0d4a10 2024-03-04T09:28:46.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.963+0000 7febab400640 1 --2- 172.21.15.6:0/2560253990 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7febac0d4a10 0x7febac0d0740 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7feb94009e50 tx=0x7feb9402bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=af7ad3dd13ee1612 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:46.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.964+0000 7febaaa00640 1 -- 172.21.15.6:0/2560253990 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb9402cb00 con 0x7febac0d4a10 2024-03-04T09:28:46.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.965+0000 7febaaa00640 1 -- 172.21.15.6:0/2560253990 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb9402cca0 con 0x7febac0d4a10 2024-03-04T09:28:46.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.965+0000 7febaaa00640 1 -- 172.21.15.6:0/2560253990 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb94034610 con 0x7febac0d4a10 2024-03-04T09:28:46.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.965+0000 7febb1400640 1 -- 172.21.15.6:0/2560253990 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7febac0d4a10 msgr2=0x7febac0d0740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:46.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.965+0000 7febb1400640 1 --2- 172.21.15.6:0/2560253990 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7febac0d4a10 0x7febac0d0740 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7feb94009e50 tx=0x7feb9402bf60 comp rx=0 tx=0).stop 2024-03-04T09:28:46.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.965+0000 7febb1400640 1 -- 172.21.15.6:0/2560253990 shutdown_connections 2024-03-04T09:28:46.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.965+0000 7febb1400640 1 --2- 172.21.15.6:0/2560253990 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7febac0d0c80 0x7febac0b44d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:46.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.965+0000 7febb1400640 1 --2- 172.21.15.6:0/2560253990 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7febac0d4a10 0x7febac0d0740 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:46.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.965+0000 7febb1400640 1 -- 172.21.15.6:0/2560253990 >> 172.21.15.6:0/2560253990 conn(0x7febac0d3d30 msgr2=0x7febac0d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:46.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.965+0000 7febb1400640 1 -- 172.21.15.6:0/2560253990 shutdown_connections 2024-03-04T09:28:46.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.965+0000 7febb1400640 1 -- 172.21.15.6:0/2560253990 wait complete. 2024-03-04T09:28:46.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.967+0000 7febb1400640 1 Processor -- start 2024-03-04T09:28:46.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.967+0000 7febb1400640 1 -- start start 2024-03-04T09:28:46.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.968+0000 7febb1400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7febac0d0c80 0x7febac150220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:46.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.968+0000 7febb1400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7febac0d4a10 0x7febac150c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:46.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.968+0000 7febb1400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7febac15e0b0 0x7febac0b8750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:46.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.968+0000 7febb1400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febac15e490 con 0x7febac0d0c80 2024-03-04T09:28:46.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.968+0000 7febb1400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febac0b3fb0 con 0x7febac0d4a10 2024-03-04T09:28:46.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.968+0000 7febb1400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febac0cfc60 con 0x7febac15e0b0 2024-03-04T09:28:46.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.969+0000 7febabe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7febac0d0c80 0x7febac150220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:46.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.969+0000 7febabe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7febac0d0c80 0x7febac150220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:44322/0 (socket says 172.21.15.6:44322) 2024-03-04T09:28:46.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.969+0000 7febabe00640 1 -- 172.21.15.6:0/3085063992 learned_addr learned my addr 172.21.15.6:0/3085063992 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:28:46.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.969+0000 7febab400640 1 --2- 172.21.15.6:0/3085063992 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7febac0d4a10 0x7febac150c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:46.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.969+0000 7febb0a00640 1 --2- 172.21.15.6:0/3085063992 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7febac15e0b0 0x7febac0b8750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:46.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.969+0000 7febabe00640 1 -- 172.21.15.6:0/3085063992 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7febac15e0b0 msgr2=0x7febac0b8750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:46.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.969+0000 7febabe00640 1 --2- 172.21.15.6:0/3085063992 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7febac15e0b0 0x7febac0b8750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:46.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.969+0000 7febabe00640 1 -- 172.21.15.6:0/3085063992 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7febac0d4a10 msgr2=0x7febac150c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:46.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.969+0000 7febabe00640 1 --2- 172.21.15.6:0/3085063992 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7febac0d4a10 0x7febac150c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:46.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.969+0000 7febabe00640 1 -- 172.21.15.6:0/3085063992 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febac150760 con 0x7febac0d0c80 2024-03-04T09:28:46.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.969+0000 7febb0a00640 1 --2- 172.21.15.6:0/3085063992 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7febac15e0b0 0x7febac0b8750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:28:46.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.970+0000 7febabe00640 1 --2- 172.21.15.6:0/3085063992 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7febac0d0c80 0x7febac150220 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7feb9c00c3c0 tx=0x7feb9c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:46.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.970+0000 7feba3e00640 1 -- 172.21.15.6:0/3085063992 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb9c0052d0 con 0x7febac0d0c80 2024-03-04T09:28:46.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.970+0000 7feba3e00640 1 -- 172.21.15.6:0/3085063992 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb9c005490 con 0x7febac0d0c80 2024-03-04T09:28:46.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.970+0000 7febb1400640 1 -- 172.21.15.6:0/3085063992 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febac15e790 con 0x7febac0d0c80 2024-03-04T09:28:46.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.970+0000 7feba3e00640 1 -- 172.21.15.6:0/3085063992 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb9c007260 con 0x7febac0d0c80 2024-03-04T09:28:46.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.970+0000 7febb1400640 1 -- 172.21.15.6:0/3085063992 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febac15ed50 con 0x7febac0d0c80 2024-03-04T09:28:46.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.971+0000 7febb1400640 1 -- 172.21.15.6:0/3085063992 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febac04f330 con 0x7febac0d0c80 2024-03-04T09:28:46.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.972+0000 7feba3e00640 1 -- 172.21.15.6:0/3085063992 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7feb9c007400 con 0x7febac0d0c80 2024-03-04T09:28:46.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.973+0000 7feba3e00640 1 --2- 172.21.15.6:0/3085063992 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7feb8806b520 0x7feb8806d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:46.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.973+0000 7feba3e00640 1 -- 172.21.15.6:0/3085063992 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 4857+0+0 (secure 0 0 0) 0x7feb9c0893c0 con 0x7febac0d0c80 2024-03-04T09:28:46.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.974+0000 7febab400640 1 --2- 172.21.15.6:0/3085063992 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7feb8806b520 0x7feb8806d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:46.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.975+0000 7febab400640 1 --2- 172.21.15.6:0/3085063992 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7feb8806b520 0x7feb8806d9e0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7feb94035000 tx=0x7feb94030000 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:46.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:46.978+0000 7feba3e00640 1 -- 172.21.15.6:0/3085063992 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7febac150760 con 0x7febac0d0c80 2024-03-04T09:28:47.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.177+0000 7febb1400640 1 -- 172.21.15.6:0/3085063992 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7febac0ca180 con 0x7febac0d0c80 2024-03-04T09:28:47.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.178+0000 7feba3e00640 1 -- 172.21.15.6:0/3085063992 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v64) v1 ==== 74+0+12843 (secure 0 0 0) 0x7feb9c058570 con 0x7febac0d0c80 2024-03-04T09:28:47.175 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:28:47.176 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":64,"fsid":"edce0578-5ae4-4402-b45a-552bdef84d7c","created":"2024-03-04T09:24:54.523780+0000","modified":"2024-03-04T09:28:46.183644+0000","last_up_change":"2024-03-04T09:25:07.089049+0000","last_in_change":"2024-03-04T09:24:57.982077+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T09:25:07.935198+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a96574f-81f1-4bc6-8463-757520b84a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6825","nonce":1364449401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6827","nonce":1364449401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6831","nonce":1364449401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6829","nonce":1364449401}]},"public_addr":"172.21.15.6:6825/1364449401","cluster_addr":"172.21.15.6:6827/1364449401","heartbeat_back_addr":"172.21.15.6:6831/1364449401","heartbeat_front_addr":"172.21.15.6:6829/1364449401","state":["exists","up"]},{"osd":1,"uuid":"7d79b9c1-6b87-476a-bc36-7db5ee64cd90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6809","nonce":2494103078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6811","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6813","nonce":2494103078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6821","nonce":2494103078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6817","nonce":2494103078}]},"public_addr":"172.21.15.6:6809/2494103078","cluster_addr":"172.21.15.6:6813/2494103078","heartbeat_back_addr":"172.21.15.6:6821/2494103078","heartbeat_front_addr":"172.21.15.6:6817/2494103078","state":["exists","up"]},{"osd":2,"uuid":"492331b9-4ee9-40b6-bacd-85d44cd6c45a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6812","nonce":1500066681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6816","nonce":1500066681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6824","nonce":1500066681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6820","nonce":1500066681}]},"public_addr":"172.21.15.6:6812/1500066681","cluster_addr":"172.21.15.6:6816/1500066681","heartbeat_back_addr":"172.21.15.6:6824/1500066681","heartbeat_front_addr":"172.21.15.6:6820/1500066681","state":["exists","up"]},{"osd":3,"uuid":"21bb2efe-9ba7-4bdc-bc31-180cd1102e21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6801","nonce":1558806123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6803","nonce":1558806123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6807","nonce":1558806123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6805","nonce":1558806123}]},"public_addr":"172.21.15.6:6801/1558806123","cluster_addr":"172.21.15.6:6803/1558806123","heartbeat_back_addr":"172.21.15.6:6807/1558806123","heartbeat_front_addr":"172.21.15.6:6805/1558806123","state":["exists","up"]},{"osd":4,"uuid":"6e2c4b71-19d9-4737-83d9-558705d8644a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6801","nonce":1526135780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6803","nonce":1526135780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6807","nonce":1526135780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6805","nonce":1526135780}]},"public_addr":"172.21.15.146:6801/1526135780","cluster_addr":"172.21.15.146:6803/1526135780","heartbeat_back_addr":"172.21.15.146:6807/1526135780","heartbeat_front_addr":"172.21.15.146:6805/1526135780","state":["exists","up"]},{"osd":5,"uuid":"fb69c525-f1be-4b7d-ab32-6c29b0d6fddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6809","nonce":460262507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6811","nonce":460262507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6815","nonce":460262507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6813","nonce":460262507}]},"public_addr":"172.21.15.146:6809/460262507","cluster_addr":"172.21.15.146:6811/460262507","heartbeat_back_addr":"172.21.15.146:6815/460262507","heartbeat_front_addr":"172.21.15.146:6813/460262507","state":["exists","up"]},{"osd":6,"uuid":"c32e7fed-370d-4401-8ce0-6a4335b0d168","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6823","nonce":1392897493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6827","nonce":1392897493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6831","nonce":1392897493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6829","nonce":1392897493}]},"public_addr":"172.21.15.146:6823/1392897493","cluster_addr":"172.21.15.146:6827/1392897493","heartbeat_back_addr":"172.21.15.146:6831/1392897493","heartbeat_front_addr":"172.21.15.146:6829/1392897493","state":["exists","up"]},{"osd":7,"uuid":"75a79b44-fa5f-426d-a7d3-71f21f78835c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6817","nonce":3268323156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6819","nonce":3268323156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6825","nonce":3268323156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6822","nonce":3268323156}]},"public_addr":"172.21.15.146:6817/3268323156","cluster_addr":"172.21.15.146:6819/3268323156","heartbeat_back_addr":"172.21.15.146:6825/3268323156","heartbeat_front_addr":"172.21.15.146:6822/3268323156","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.310437+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.295850+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.294674+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.526855+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.300023+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.595160+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.346046+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.558741+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T09:28:47.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.180+0000 7febb1400640 1 -- 172.21.15.6:0/3085063992 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7feb8806b520 msgr2=0x7feb8806d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:47.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.180+0000 7febb1400640 1 --2- 172.21.15.6:0/3085063992 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7feb8806b520 0x7feb8806d9e0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7feb94035000 tx=0x7feb94030000 comp rx=0 tx=0).stop 2024-03-04T09:28:47.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.180+0000 7febb1400640 1 -- 172.21.15.6:0/3085063992 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7febac0d0c80 msgr2=0x7febac150220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:47.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.180+0000 7febb1400640 1 --2- 172.21.15.6:0/3085063992 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7febac0d0c80 0x7febac150220 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7feb9c00c3c0 tx=0x7feb9c00c880 comp rx=0 tx=0).stop 2024-03-04T09:28:47.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.181+0000 7febb1400640 1 -- 172.21.15.6:0/3085063992 shutdown_connections 2024-03-04T09:28:47.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.181+0000 7febb1400640 1 --2- 172.21.15.6:0/3085063992 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7feb8806b520 0x7feb8806d9e0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:47.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.181+0000 7febb1400640 1 --2- 172.21.15.6:0/3085063992 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7febac15e0b0 0x7febac0b8750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:47.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.181+0000 7febb1400640 1 --2- 172.21.15.6:0/3085063992 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7febac0d4a10 0x7febac150c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:47.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.181+0000 7febb1400640 1 --2- 172.21.15.6:0/3085063992 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7febac0d0c80 0x7febac150220 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:47.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.181+0000 7febb1400640 1 -- 172.21.15.6:0/3085063992 >> 172.21.15.6:0/3085063992 conn(0x7febac0d3d30 msgr2=0x7febac0b1940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:47.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.181+0000 7febb1400640 1 -- 172.21.15.6:0/3085063992 shutdown_connections 2024-03-04T09:28:47.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.181+0000 7febb1400640 1 -- 172.21.15.6:0/3085063992 wait complete. 2024-03-04T09:28:47.195 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T09:25:07.935198+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-04T09:28:47.196 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-04T09:28:47.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.427+0000 7f9563200640 1 Processor -- start 2024-03-04T09:28:47.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.428+0000 7f9563200640 1 -- start start 2024-03-04T09:28:47.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.429+0000 7f9563200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f95640d4860 0x7f95640d0630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:47.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.429+0000 7f9563200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f95640cfa60 0x7f95640d0b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:47.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.429+0000 7f9563200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95640ceca0 con 0x7f956415dee0 2024-03-04T09:28:47.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.429+0000 7f9563200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95640cf5d0 con 0x7f95640cfa60 2024-03-04T09:28:47.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.429+0000 7f9563200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95640cef80 con 0x7f95640d4860 2024-03-04T09:28:47.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.429+0000 7f9561e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f956415dee0 0x7f95640d18a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:32778/0 (socket says 172.21.15.6:32778) 2024-03-04T09:28:47.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.429+0000 7f9561e00640 1 -- 172.21.15.6:0/2218345727 learned_addr learned my addr 172.21.15.6:0/2218345727 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:28:47.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.430+0000 7f9562800640 1 --2- 172.21.15.6:0/2218345727 >> v2:172.21.15.146:3300/0 conn(0x7f95640cfa60 0x7f95640d0b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:47.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.430+0000 7f9561400640 1 --2- 172.21.15.6:0/2218345727 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f95640d4860 0x7f95640d0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:47.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.430+0000 7f9560a00640 1 -- 172.21.15.6:0/2218345727 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 48931362 0 0) 0x7f95640ceca0 con 0x7f956415dee0 2024-03-04T09:28:47.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.430+0000 7f9560a00640 1 -- 172.21.15.6:0/2218345727 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9548003520 con 0x7f956415dee0 2024-03-04T09:28:47.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.430+0000 7f9560a00640 1 -- 172.21.15.6:0/2218345727 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2563580991 0 0) 0x7f9548003520 con 0x7f956415dee0 2024-03-04T09:28:47.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.430+0000 7f9560a00640 1 -- 172.21.15.6:0/2218345727 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f95640d4860 msgr2=0x7f95640d0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:47.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.430+0000 7f9560a00640 1 --2- 172.21.15.6:0/2218345727 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f95640d4860 0x7f95640d0630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:47.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.430+0000 7f9560a00640 1 -- 172.21.15.6:0/2218345727 >> v2:172.21.15.146:3300/0 conn(0x7f95640cfa60 msgr2=0x7f95640d0b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:47.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.430+0000 7f9560a00640 1 --2- 172.21.15.6:0/2218345727 >> v2:172.21.15.146:3300/0 conn(0x7f95640cfa60 0x7f95640d0b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:47.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.430+0000 7f9560a00640 1 -- 172.21.15.6:0/2218345727 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95640cf260 con 0x7f956415dee0 2024-03-04T09:28:47.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.430+0000 7f9560a00640 1 -- 172.21.15.6:0/2218345727 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7f95540019f0 con 0x7f956415dee0 2024-03-04T09:28:47.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.430+0000 7f9560a00640 1 -- 172.21.15.6:0/2218345727 >> v1:172.21.15.6:6789/0 conn(0x7f956415dee0 legacy=0x7f95640d18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:47.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.431+0000 7f9560a00640 1 --2- 172.21.15.6:0/2218345727 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f95480039b0 0x7f9548003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:47.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.431+0000 7f9560a00640 1 --2- 172.21.15.6:0/2218345727 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9548004800 0x7f9548004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:47.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.431+0000 7f9560a00640 1 --2- 172.21.15.6:0/2218345727 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f95640d4860 0x7f95480051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:47.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.431+0000 7f9560a00640 1 -- 172.21.15.6:0/2218345727 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95640cf5d0 con 0x7f9548004800 2024-03-04T09:28:47.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.431+0000 7f9560a00640 1 -- 172.21.15.6:0/2218345727 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95640cef80 con 0x7f95480039b0 2024-03-04T09:28:47.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.431+0000 7f9560a00640 1 -- 172.21.15.6:0/2218345727 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f954800b9c0 con 0x7f95640d4860 2024-03-04T09:28:47.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.431+0000 7f9561400640 1 --2- 172.21.15.6:0/2218345727 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9548004800 0x7f9548004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:47.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.431+0000 7f9561e00640 1 --2- 172.21.15.6:0/2218345727 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f95480039b0 0x7f9548003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:47.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.431+0000 7f9561400640 1 -- 172.21.15.6:0/2218345727 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f95640d4860 msgr2=0x7f95480051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:47.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.431+0000 7f9561400640 1 --2- 172.21.15.6:0/2218345727 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f95640d4860 0x7f95480051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:47.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.431+0000 7f9561400640 1 -- 172.21.15.6:0/2218345727 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f95480039b0 msgr2=0x7f9548003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:47.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.431+0000 7f9561400640 1 --2- 172.21.15.6:0/2218345727 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f95480039b0 0x7f9548003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:47.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.431+0000 7f9561400640 1 -- 172.21.15.6:0/2218345727 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9548003520 con 0x7f9548004800 2024-03-04T09:28:47.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.431+0000 7f9561e00640 1 --2- 172.21.15.6:0/2218345727 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f95480039b0 0x7f9548003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:28:47.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.432+0000 7f9561400640 1 --2- 172.21.15.6:0/2218345727 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9548004800 0x7f9548004c80 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f954c009a90 tx=0x7f9548005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=236a1a9d084ab002 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:47.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.433+0000 7f9560a00640 1 -- 172.21.15.6:0/2218345727 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f954c039070 con 0x7f9548004800 2024-03-04T09:28:47.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.433+0000 7f9560a00640 1 -- 172.21.15.6:0/2218345727 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f954c02c6b0 con 0x7f9548004800 2024-03-04T09:28:47.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.433+0000 7f9560a00640 1 -- 172.21.15.6:0/2218345727 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f954c009690 con 0x7f9548004800 2024-03-04T09:28:47.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.433+0000 7f9563200640 1 -- 172.21.15.6:0/2218345727 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9548004800 msgr2=0x7f9548004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:47.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.433+0000 7f9563200640 1 --2- 172.21.15.6:0/2218345727 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9548004800 0x7f9548004c80 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f954c009a90 tx=0x7f9548005830 comp rx=0 tx=0).stop 2024-03-04T09:28:47.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.433+0000 7f9562800640 1 -- 172.21.15.6:0/2218345727 reap_dead start 2024-03-04T09:28:47.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.433+0000 7f9563200640 1 -- 172.21.15.6:0/2218345727 shutdown_connections 2024-03-04T09:28:47.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.433+0000 7f9563200640 1 -- 172.21.15.6:0/2218345727 >> 172.21.15.6:0/2218345727 conn(0x7f95640d3cd0 msgr2=0x7f95640d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:47.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.433+0000 7f9563200640 1 -- 172.21.15.6:0/2218345727 shutdown_connections 2024-03-04T09:28:47.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.433+0000 7f9563200640 1 -- 172.21.15.6:0/2218345727 wait complete. 2024-03-04T09:28:47.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.435+0000 7f9563200640 1 Processor -- start 2024-03-04T09:28:47.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.435+0000 7f9563200640 1 -- start start 2024-03-04T09:28:47.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.436+0000 7f9563200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9548004800 0x7f956414ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:47.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.436+0000 7f9563200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f956406ccf0 0x7f956406d170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:47.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.436+0000 7f9563200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f95640c99c0 0x7f95640c9e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:47.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.436+0000 7f9563200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95640d0630 con 0x7f9548004800 2024-03-04T09:28:47.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.436+0000 7f9563200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f956415e2c0 con 0x7f956406ccf0 2024-03-04T09:28:47.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.436+0000 7f9563200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95640d07b0 con 0x7f95640c99c0 2024-03-04T09:28:47.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.436+0000 7f9561e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9548004800 0x7f956414ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:47.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.437+0000 7f9561e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9548004800 0x7f956414ef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:44340/0 (socket says 172.21.15.6:44340) 2024-03-04T09:28:47.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.437+0000 7f9561e00640 1 -- 172.21.15.6:0/2697622337 learned_addr learned my addr 172.21.15.6:0/2697622337 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:28:47.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.437+0000 7f9561400640 1 --2- 172.21.15.6:0/2697622337 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f956406ccf0 0x7f956406d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:47.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.437+0000 7f9562800640 1 --2- 172.21.15.6:0/2697622337 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f95640c99c0 0x7f95640c9e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:47.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.437+0000 7f9561e00640 1 -- 172.21.15.6:0/2697622337 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f95640c99c0 msgr2=0x7f95640c9e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:47.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.437+0000 7f9561e00640 1 --2- 172.21.15.6:0/2697622337 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f95640c99c0 0x7f95640c9e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:47.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.437+0000 7f9561e00640 1 -- 172.21.15.6:0/2697622337 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f956406ccf0 msgr2=0x7f956406d170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:47.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.437+0000 7f9561e00640 1 --2- 172.21.15.6:0/2697622337 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f956406ccf0 0x7f956406d170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:47.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.437+0000 7f9561e00640 1 -- 172.21.15.6:0/2697622337 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f956414f4c0 con 0x7f9548004800 2024-03-04T09:28:47.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.437+0000 7f9561400640 1 --2- 172.21.15.6:0/2697622337 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f956406ccf0 0x7f956406d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:28:47.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.437+0000 7f9562800640 1 --2- 172.21.15.6:0/2697622337 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f95640c99c0 0x7f95640c9e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:28:47.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.437+0000 7f9561e00640 1 --2- 172.21.15.6:0/2697622337 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9548004800 0x7f956414ef80 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f9554001af0 tx=0x7f9554001b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:47.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.437+0000 7f955a000640 1 -- 172.21.15.6:0/2697622337 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9554019070 con 0x7f9548004800 2024-03-04T09:28:47.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.438+0000 7f955a000640 1 -- 172.21.15.6:0/2697622337 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9554004d60 con 0x7f9548004800 2024-03-04T09:28:47.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.438+0000 7f955a000640 1 -- 172.21.15.6:0/2697622337 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9554007310 con 0x7f9548004800 2024-03-04T09:28:47.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.438+0000 7f9563200640 1 -- 172.21.15.6:0/2697622337 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95640ca380 con 0x7f9548004800 2024-03-04T09:28:47.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.438+0000 7f9563200640 1 -- 172.21.15.6:0/2697622337 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95640c8280 con 0x7f9548004800 2024-03-04T09:28:47.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.439+0000 7f9563200640 1 -- 172.21.15.6:0/2697622337 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f956404f330 con 0x7f9548004800 2024-03-04T09:28:47.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.440+0000 7f955a000640 1 -- 172.21.15.6:0/2697622337 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f955401c070 con 0x7f9548004800 2024-03-04T09:28:47.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.441+0000 7f955a000640 1 --2- 172.21.15.6:0/2697622337 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f954006b4d0 0x7f954006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:47.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.441+0000 7f955a000640 1 -- 172.21.15.6:0/2697622337 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f955408a220 con 0x7f9548004800 2024-03-04T09:28:47.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.442+0000 7f9561400640 1 --2- 172.21.15.6:0/2697622337 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f954006b4d0 0x7f954006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:47.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.443+0000 7f9561400640 1 --2- 172.21.15.6:0/2697622337 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f954006b4d0 0x7f954006d990 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f954c036000 tx=0x7f954c030000 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:47.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.446+0000 7f955a000640 1 -- 172.21.15.6:0/2697622337 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9554059350 con 0x7f9548004800 2024-03-04T09:28:47.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.646+0000 7f9563200640 1 -- 172.21.15.6:0/2697622337 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f956408d7d0 con 0x7f9548004800 2024-03-04T09:28:47.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.647+0000 7f955a000640 1 -- 172.21.15.6:0/2697622337 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v64) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9554088e20 con 0x7f9548004800 2024-03-04T09:28:47.644 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-04T09:28:47.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.649+0000 7f9563200640 1 -- 172.21.15.6:0/2697622337 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f954006b4d0 msgr2=0x7f954006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:47.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.649+0000 7f9563200640 1 --2- 172.21.15.6:0/2697622337 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f954006b4d0 0x7f954006d990 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f954c036000 tx=0x7f954c030000 comp rx=0 tx=0).stop 2024-03-04T09:28:47.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.649+0000 7f9563200640 1 -- 172.21.15.6:0/2697622337 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9548004800 msgr2=0x7f956414ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:47.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.649+0000 7f9563200640 1 --2- 172.21.15.6:0/2697622337 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9548004800 0x7f956414ef80 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f9554001af0 tx=0x7f9554001b20 comp rx=0 tx=0).stop 2024-03-04T09:28:47.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.650+0000 7f9563200640 1 -- 172.21.15.6:0/2697622337 shutdown_connections 2024-03-04T09:28:47.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.650+0000 7f9563200640 1 --2- 172.21.15.6:0/2697622337 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f954006b4d0 0x7f954006d990 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:47.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.650+0000 7f9563200640 1 --2- 172.21.15.6:0/2697622337 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f95640c99c0 0x7f95640c9e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:47.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.650+0000 7f9563200640 1 --2- 172.21.15.6:0/2697622337 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f956406ccf0 0x7f956406d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:47.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.650+0000 7f9563200640 1 --2- 172.21.15.6:0/2697622337 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9548004800 0x7f956414ef80 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:47.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.650+0000 7f9563200640 1 -- 172.21.15.6:0/2697622337 >> 172.21.15.6:0/2697622337 conn(0x7f95640d3cd0 msgr2=0x7f95640b8920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:47.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.650+0000 7f9563200640 1 -- 172.21.15.6:0/2697622337 shutdown_connections 2024-03-04T09:28:47.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.650+0000 7f9563200640 1 -- 172.21.15.6:0/2697622337 wait complete. 2024-03-04T09:28:47.666 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-04T09:28:47.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.894+0000 7fa4a8a00640 1 Processor -- start 2024-03-04T09:28:47.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.894+0000 7fa4a8a00640 1 -- start start 2024-03-04T09:28:47.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.896+0000 7fa4a8a00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fa4a4145300 0x7fa4a40b49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:47.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.896+0000 7fa4a8a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa4a40cd060 0x7fa4a415c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:47.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.896+0000 7fa4a8a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4a40714d0 con 0x7fa4a4070f10 2024-03-04T09:28:47.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.896+0000 7fa4a8a00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4a40717b0 con 0x7fa4a40cd060 2024-03-04T09:28:47.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.896+0000 7fa4a8a00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4a4071a90 con 0x7fa4a4145300 2024-03-04T09:28:47.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.896+0000 7fa4a3e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa4a40cd060 0x7fa4a415c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:47.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.896+0000 7fa4a3e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa4a40cd060 0x7fa4a415c570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:44342/0 (socket says 172.21.15.6:44342) 2024-03-04T09:28:47.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.896+0000 7fa4a2a00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fa4a4070f10 0x7fa4a40cc950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:32788/0 (socket says 172.21.15.6:32788) 2024-03-04T09:28:47.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.896+0000 7fa4a3e00640 1 -- 172.21.15.6:0/3044571732 learned_addr learned my addr 172.21.15.6:0/3044571732 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:28:47.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.896+0000 7fa4a3400640 1 --2- 172.21.15.6:0/3044571732 >> v2:172.21.15.146:3300/0 conn(0x7fa4a4145300 0x7fa4a40b49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:47.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.896+0000 7fa4a3e00640 1 -- 172.21.15.6:0/3044571732 >> v2:172.21.15.146:3300/0 conn(0x7fa4a4145300 msgr2=0x7fa4a40b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:47.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.897+0000 7fa4a3e00640 1 --2- 172.21.15.6:0/3044571732 >> v2:172.21.15.146:3300/0 conn(0x7fa4a4145300 0x7fa4a40b49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:47.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.897+0000 7fa4a3e00640 1 -- 172.21.15.6:0/3044571732 >> v1:172.21.15.6:6789/0 conn(0x7fa4a4070f10 legacy=0x7fa4a40cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:47.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.897+0000 7fa4a3e00640 1 -- 172.21.15.6:0/3044571732 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4a4071d70 con 0x7fa4a40cd060 2024-03-04T09:28:47.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.897+0000 7fa4a3400640 1 --2- 172.21.15.6:0/3044571732 >> v2:172.21.15.146:3300/0 conn(0x7fa4a4145300 0x7fa4a40b49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:28:47.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.898+0000 7fa4a3e00640 1 --2- 172.21.15.6:0/3044571732 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4a40cd060 0x7fa4a415c570 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fa4a40cf990 tx=0x7fa49c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=103266e7f93ac954 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:47.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.898+0000 7fa4a2000640 1 -- 172.21.15.6:0/3044571732 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa49c02ca80 con 0x7fa4a40cd060 2024-03-04T09:28:47.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.898+0000 7fa4a2000640 1 -- 172.21.15.6:0/3044571732 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa49c02cc20 con 0x7fa4a40cd060 2024-03-04T09:28:47.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.898+0000 7fa4a2000640 1 -- 172.21.15.6:0/3044571732 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa49c0343f0 con 0x7fa4a40cd060 2024-03-04T09:28:47.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.899+0000 7fa4a8a00640 1 -- 172.21.15.6:0/3044571732 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4a40cd060 msgr2=0x7fa4a415c570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:47.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.899+0000 7fa4a8a00640 1 --2- 172.21.15.6:0/3044571732 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4a40cd060 0x7fa4a415c570 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fa4a40cf990 tx=0x7fa49c02bee0 comp rx=0 tx=0).stop 2024-03-04T09:28:47.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.899+0000 7fa4a8a00640 1 -- 172.21.15.6:0/3044571732 shutdown_connections 2024-03-04T09:28:47.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.899+0000 7fa4a8a00640 1 --2- 172.21.15.6:0/3044571732 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4a40cd060 0x7fa4a415c570 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:47.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.899+0000 7fa4a8a00640 1 --2- 172.21.15.6:0/3044571732 >> v2:172.21.15.146:3300/0 conn(0x7fa4a4145300 0x7fa4a40b49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:47.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.899+0000 7fa4a8a00640 1 -- 172.21.15.6:0/3044571732 >> 172.21.15.6:0/3044571732 conn(0x7fa4a406b3d0 msgr2=0x7fa4a406b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:47.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.899+0000 7fa4a8a00640 1 -- 172.21.15.6:0/3044571732 shutdown_connections 2024-03-04T09:28:47.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.899+0000 7fa4a8a00640 1 -- 172.21.15.6:0/3044571732 wait complete. 2024-03-04T09:28:47.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.901+0000 7fa4a8a00640 1 Processor -- start 2024-03-04T09:28:47.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.901+0000 7fa4a8a00640 1 -- start start 2024-03-04T09:28:47.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.902+0000 7fa4a8a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa4a4070f10 0x7fa4a40ae2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:47.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.902+0000 7fa4a8a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4a40cd060 0x7fa4a4130f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:47.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.902+0000 7fa4a8a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa4a4145300 0x7fa4a415af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:47.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.902+0000 7fa4a8a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4a415cab0 con 0x7fa4a40cd060 2024-03-04T09:28:47.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.902+0000 7fa4a8a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4a41456e0 con 0x7fa4a4070f10 2024-03-04T09:28:47.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.902+0000 7fa4a8a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4a415cc30 con 0x7fa4a4145300 2024-03-04T09:28:47.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.902+0000 7fa4a2a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4a40cd060 0x7fa4a4130f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:47.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.902+0000 7fa4a2a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4a40cd060 0x7fa4a4130f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:44346/0 (socket says 172.21.15.6:44346) 2024-03-04T09:28:47.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.902+0000 7fa4a2a00640 1 -- 172.21.15.6:0/3132980608 learned_addr learned my addr 172.21.15.6:0/3132980608 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:28:47.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.903+0000 7fa4a3400640 1 --2- 172.21.15.6:0/3132980608 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa4a4070f10 0x7fa4a40ae2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:47.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.903+0000 7fa4a2a00640 1 -- 172.21.15.6:0/3132980608 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa4a4145300 msgr2=0x7fa4a415af80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:47.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.903+0000 7fa4a2a00640 1 --2- 172.21.15.6:0/3132980608 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa4a4145300 0x7fa4a415af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:47.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.903+0000 7fa4a2a00640 1 -- 172.21.15.6:0/3132980608 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa4a4070f10 msgr2=0x7fa4a40ae2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:47.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.903+0000 7fa4a2a00640 1 --2- 172.21.15.6:0/3132980608 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa4a4070f10 0x7fa4a40ae2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:47.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.903+0000 7fa4a2a00640 1 -- 172.21.15.6:0/3132980608 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4a41319e0 con 0x7fa4a40cd060 2024-03-04T09:28:47.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.903+0000 7fa4a2a00640 1 --2- 172.21.15.6:0/3132980608 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4a40cd060 0x7fa4a4130f20 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fa49000c430 tx=0x7fa49000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:47.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.903+0000 7fa49b400640 1 -- 172.21.15.6:0/3132980608 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4900074e0 con 0x7fa4a40cd060 2024-03-04T09:28:47.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.903+0000 7fa49b400640 1 -- 172.21.15.6:0/3132980608 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa490007680 con 0x7fa4a40cd060 2024-03-04T09:28:47.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.903+0000 7fa4a8a00640 1 -- 172.21.15.6:0/3132980608 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4a4131cc0 con 0x7fa4a40cd060 2024-03-04T09:28:47.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.903+0000 7fa49b400640 1 -- 172.21.15.6:0/3132980608 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa490004a00 con 0x7fa4a40cd060 2024-03-04T09:28:47.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.904+0000 7fa4a8a00640 1 -- 172.21.15.6:0/3132980608 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4a4132280 con 0x7fa4a40cd060 2024-03-04T09:28:47.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.905+0000 7fa4a8a00640 1 -- 172.21.15.6:0/3132980608 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4a404f330 con 0x7fa4a40cd060 2024-03-04T09:28:47.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.906+0000 7fa49b400640 1 -- 172.21.15.6:0/3132980608 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa49001a020 con 0x7fa4a40cd060 2024-03-04T09:28:47.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.907+0000 7fa49b400640 1 --2- 172.21.15.6:0/3132980608 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa48406b4d0 0x7fa48406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:47.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.907+0000 7fa49b400640 1 -- 172.21.15.6:0/3132980608 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fa4900896c0 con 0x7fa4a40cd060 2024-03-04T09:28:47.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.907+0000 7fa4a3400640 1 --2- 172.21.15.6:0/3132980608 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa48406b4d0 0x7fa48406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:47.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.909+0000 7fa4a3400640 1 --2- 172.21.15.6:0/3132980608 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa48406b4d0 0x7fa48406d990 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fa49400a950 tx=0x7fa494008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:47.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:47.912+0000 7fa49b400640 1 -- 172.21.15.6:0/3132980608 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa490058890 con 0x7fa4a40cd060 2024-03-04T09:28:48.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:48.116+0000 7fa4a8a00640 1 -- 172.21.15.6:0/3132980608 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fa4a40ce780 con 0x7fa4a40cd060 2024-03-04T09:28:48.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:48.117+0000 7fa49b400640 1 -- 172.21.15.6:0/3132980608 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v64) v1 ==== 84+0+65 (secure 0 0 0) 0x7fa490058190 con 0x7fa4a40cd060 2024-03-04T09:28:48.114 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:28:48.114 INFO:teuthology.orchestra.run.smithi006.stdout:[ 2024-03-04T09:28:48.114 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:28:48.114 INFO:teuthology.orchestra.run.smithi006.stdout: "poolnum": 1, 2024-03-04T09:28:48.114 INFO:teuthology.orchestra.run.smithi006.stdout: "poolname": ".mgr" 2024-03-04T09:28:48.115 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:28:48.115 INFO:teuthology.orchestra.run.smithi006.stdout:] 2024-03-04T09:28:48.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:48.120+0000 7fa4a8a00640 1 -- 172.21.15.6:0/3132980608 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa48406b4d0 msgr2=0x7fa48406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:48.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:48.120+0000 7fa4a8a00640 1 --2- 172.21.15.6:0/3132980608 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa48406b4d0 0x7fa48406d990 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fa49400a950 tx=0x7fa494008040 comp rx=0 tx=0).stop 2024-03-04T09:28:48.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:48.120+0000 7fa4a8a00640 1 -- 172.21.15.6:0/3132980608 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4a40cd060 msgr2=0x7fa4a4130f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:48.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:48.120+0000 7fa4a8a00640 1 --2- 172.21.15.6:0/3132980608 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4a40cd060 0x7fa4a4130f20 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fa49000c430 tx=0x7fa49000c8f0 comp rx=0 tx=0).stop 2024-03-04T09:28:48.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:48.120+0000 7fa4a8a00640 1 -- 172.21.15.6:0/3132980608 shutdown_connections 2024-03-04T09:28:48.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:48.120+0000 7fa4a8a00640 1 --2- 172.21.15.6:0/3132980608 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa48406b4d0 0x7fa48406d990 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:48.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:48.120+0000 7fa4a8a00640 1 --2- 172.21.15.6:0/3132980608 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa4a4145300 0x7fa4a415af80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:48.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:48.120+0000 7fa4a8a00640 1 --2- 172.21.15.6:0/3132980608 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4a40cd060 0x7fa4a4130f20 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:48.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:48.120+0000 7fa4a8a00640 1 --2- 172.21.15.6:0/3132980608 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa4a4070f10 0x7fa4a40ae2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:48.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:48.120+0000 7fa4a8a00640 1 -- 172.21.15.6:0/3132980608 >> 172.21.15.6:0/3132980608 conn(0x7fa4a406b3d0 msgr2=0x7fa4a40cd930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:48.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:48.120+0000 7fa4a8a00640 1 -- 172.21.15.6:0/3132980608 shutdown_connections 2024-03-04T09:28:48.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:48.120+0000 7fa4a8a00640 1 -- 172.21.15.6:0/3132980608 wait complete. 2024-03-04T09:28:48.136 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-04T09:28:48.137 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-04T09:28:48.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:48.357+0000 7fab5a600640 1 Processor -- start 2024-03-04T09:28:48.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:48.357+0000 7fab5a600640 1 -- start start 2024-03-04T09:28:48.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:48.359+0000 7fab5a600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fab541622f0 0x7fab5406daf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:48.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:48.359+0000 7fab5a600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fab541669b0 0x7fab540ccdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:48.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:48.359+0000 7fab5a600640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab54070a80 con 0x7fab5406e030 2024-03-04T09:28:48.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:48.359+0000 7fab5a600640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab54166e40 con 0x7fab541669b0 2024-03-04T09:28:48.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:48.359+0000 7fab5a600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab5406e490 con 0x7fab541622f0 2024-03-04T09:28:48.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:48.359+0000 7fab59c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fab541669b0 0x7fab540ccdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:48.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:48.359+0000 7fab59c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fab541669b0 0x7fab540ccdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:44350/0 (socket says 172.21.15.6:44350) 2024-03-04T09:28:48.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:48.359+0000 7fab59c00640 1 -- 172.21.15.6:0/3964362875 learned_addr learned my addr 172.21.15.6:0/3964362875 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:28:48.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:48.360+0000 7fab59200640 1 --2- 172.21.15.6:0/3964362875 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fab541622f0 0x7fab5406daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:48.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:48.360+0000 7fab59c00640 1 -- 172.21.15.6:0/3964362875 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fab541622f0 msgr2=0x7fab5406daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:48.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:48.360+0000 7fab59c00640 1 --2- 172.21.15.6:0/3964362875 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fab541622f0 0x7fab5406daf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:48.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:48.360+0000 7fab59c00640 1 -- 172.21.15.6:0/3964362875 >> v1:172.21.15.146:6789/0 conn(0x7fab5406e030 legacy=0x7fab541662a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:48.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:48.360+0000 7fab59c00640 1 -- 172.21.15.6:0/3964362875 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab54070d60 con 0x7fab541669b0 2024-03-04T09:28:48.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:48.360+0000 7fab59200640 1 --2- 172.21.15.6:0/3964362875 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fab541622f0 0x7fab5406daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:28:48.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:48.361+0000 7fab59c00640 1 --2- 172.21.15.6:0/3964362875 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fab541669b0 0x7fab540ccdb0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fab4c009e50 tx=0x7fab4c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33faf3a4713853ab server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:48.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:48.361+0000 7fab53400640 1 -- 172.21.15.6:0/3964362875 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab4c02ca80 con 0x7fab541669b0 2024-03-04T09:28:48.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:48.361+0000 7fab53400640 1 -- 172.21.15.6:0/3964362875 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab4c02cc20 con 0x7fab541669b0 2024-03-04T09:28:48.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:48.361+0000 7fab53400640 1 -- 172.21.15.6:0/3964362875 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab4c0343f0 con 0x7fab541669b0 2024-03-04T09:28:48.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:48.362+0000 7fab5a600640 1 -- 172.21.15.6:0/3964362875 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fab541669b0 msgr2=0x7fab540ccdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:48.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:48.362+0000 7fab5a600640 1 --2- 172.21.15.6:0/3964362875 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fab541669b0 0x7fab540ccdb0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fab4c009e50 tx=0x7fab4c02bee0 comp rx=0 tx=0).stop 2024-03-04T09:28:48.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:48.362+0000 7fab5a600640 1 -- 172.21.15.6:0/3964362875 shutdown_connections 2024-03-04T09:28:48.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:48.362+0000 7fab5a600640 1 --2- 172.21.15.6:0/3964362875 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fab541669b0 0x7fab540ccdb0 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:48.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:48.362+0000 7fab5a600640 1 --2- 172.21.15.6:0/3964362875 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fab541622f0 0x7fab5406daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:48.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:48.362+0000 7fab5a600640 1 -- 172.21.15.6:0/3964362875 >> 172.21.15.6:0/3964362875 conn(0x7fab540b17c0 msgr2=0x7fab540b1ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:48.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:48.362+0000 7fab5a600640 1 -- 172.21.15.6:0/3964362875 shutdown_connections 2024-03-04T09:28:48.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:48.362+0000 7fab5a600640 1 -- 172.21.15.6:0/3964362875 wait complete. 2024-03-04T09:28:48.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:48.365+0000 7fab5a600640 1 Processor -- start 2024-03-04T09:28:48.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:48.365+0000 7fab5a600640 1 -- start start 2024-03-04T09:28:48.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:48.366+0000 7fab5a600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fab5406e030 0x7fab5415af50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:48.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:48.366+0000 7fab5a600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fab541622f0 0x7fab541469c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:48.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:48.366+0000 7fab5a600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fab541669b0 0x7fab541633b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:48.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:48.366+0000 7fab5a600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab54071600 con 0x7fab541669b0 2024-03-04T09:28:48.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:48.366+0000 7fab5a600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab54071320 con 0x7fab5406e030 2024-03-04T09:28:48.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:48.366+0000 7fab5a600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab541626d0 con 0x7fab541622f0 2024-03-04T09:28:48.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:48.367+0000 7fab59c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fab541669b0 0x7fab541633b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:48.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:48.367+0000 7fab59c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fab541669b0 0x7fab541633b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:44364/0 (socket says 172.21.15.6:44364) 2024-03-04T09:28:48.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:48.367+0000 7fab59c00640 1 -- 172.21.15.6:0/3092337697 learned_addr learned my addr 172.21.15.6:0/3092337697 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:28:48.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:48.367+0000 7fab59200640 1 --2- 172.21.15.6:0/3092337697 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fab5406e030 0x7fab5415af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:48.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:48.367+0000 7fab53e00640 1 --2- 172.21.15.6:0/3092337697 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fab541622f0 0x7fab541469c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:48.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:48.367+0000 7fab59c00640 1 -- 172.21.15.6:0/3092337697 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fab541622f0 msgr2=0x7fab541469c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:48.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:48.367+0000 7fab59c00640 1 --2- 172.21.15.6:0/3092337697 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fab541622f0 0x7fab541469c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:48.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:48.367+0000 7fab59c00640 1 -- 172.21.15.6:0/3092337697 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fab5406e030 msgr2=0x7fab5415af50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:48.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:48.367+0000 7fab59c00640 1 --2- 172.21.15.6:0/3092337697 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fab5406e030 0x7fab5415af50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:48.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:48.367+0000 7fab59c00640 1 -- 172.21.15.6:0/3092337697 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab5415b960 con 0x7fab541669b0 2024-03-04T09:28:48.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:48.367+0000 7fab53e00640 1 --2- 172.21.15.6:0/3092337697 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fab541622f0 0x7fab541469c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:28:48.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:48.368+0000 7fab59c00640 1 --2- 172.21.15.6:0/3092337697 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fab541669b0 0x7fab541633b0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7fab4c036040 tx=0x7fab4c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:48.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:48.368+0000 7fab50c00640 1 -- 172.21.15.6:0/3092337697 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab4c043020 con 0x7fab541669b0 2024-03-04T09:28:48.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:48.368+0000 7fab50c00640 1 -- 172.21.15.6:0/3092337697 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab4c03e450 con 0x7fab541669b0 2024-03-04T09:28:48.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:48.368+0000 7fab5a600640 1 -- 172.21.15.6:0/3092337697 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab5415bc40 con 0x7fab541669b0 2024-03-04T09:28:48.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:48.368+0000 7fab50c00640 1 -- 172.21.15.6:0/3092337697 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab4c009510 con 0x7fab541669b0 2024-03-04T09:28:48.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:48.368+0000 7fab5a600640 1 -- 172.21.15.6:0/3092337697 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab54146f00 con 0x7fab541669b0 2024-03-04T09:28:48.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:48.369+0000 7fab5a600640 1 -- 172.21.15.6:0/3092337697 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab5404f330 con 0x7fab541669b0 2024-03-04T09:28:48.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:48.370+0000 7fab50c00640 1 -- 172.21.15.6:0/3092337697 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fab4c046020 con 0x7fab541669b0 2024-03-04T09:28:48.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:48.371+0000 7fab50c00640 1 --2- 172.21.15.6:0/3092337697 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fab3406b500 0x7fab3406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:48.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:48.371+0000 7fab50c00640 1 -- 172.21.15.6:0/3092337697 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fab4c0ac7d0 con 0x7fab541669b0 2024-03-04T09:28:48.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:48.372+0000 7fab59200640 1 --2- 172.21.15.6:0/3092337697 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fab3406b500 0x7fab3406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:48.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:48.373+0000 7fab59200640 1 --2- 172.21.15.6:0/3092337697 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fab3406b500 0x7fab3406d9c0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fab4800a990 tx=0x7fab48008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:48.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:48.376+0000 7fab50c00640 1 -- 172.21.15.6:0/3092337697 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fab4c04d030 con 0x7fab541669b0 2024-03-04T09:28:48.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:48.578+0000 7fab5a600640 1 -- 172.21.15.6:0/3092337697 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fab540ceeb0 con 0x7fab541669b0 2024-03-04T09:28:49.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:49.213+0000 7fab50c00640 1 -- 172.21.15.6:0/3092337697 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v65) v1 ==== 138+0+0 (secure 0 0 0) 0x7fab4c07b900 con 0x7fab541669b0 2024-03-04T09:28:49.210 INFO:teuthology.orchestra.run.smithi006.stderr:pool 'cephfs_metadata' created 2024-03-04T09:28:49.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:49.217+0000 7fab5a600640 1 -- 172.21.15.6:0/3092337697 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fab3406b500 msgr2=0x7fab3406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:49.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:49.217+0000 7fab5a600640 1 --2- 172.21.15.6:0/3092337697 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fab3406b500 0x7fab3406d9c0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fab4800a990 tx=0x7fab48008040 comp rx=0 tx=0).stop 2024-03-04T09:28:49.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:49.217+0000 7fab5a600640 1 -- 172.21.15.6:0/3092337697 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fab541669b0 msgr2=0x7fab541633b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:49.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:49.217+0000 7fab5a600640 1 --2- 172.21.15.6:0/3092337697 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fab541669b0 0x7fab541633b0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7fab4c036040 tx=0x7fab4c004a40 comp rx=0 tx=0).stop 2024-03-04T09:28:49.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:49.217+0000 7fab5a600640 1 -- 172.21.15.6:0/3092337697 shutdown_connections 2024-03-04T09:28:49.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:49.217+0000 7fab5a600640 1 --2- 172.21.15.6:0/3092337697 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fab541669b0 0x7fab541633b0 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:49.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:49.217+0000 7fab5a600640 1 --2- 172.21.15.6:0/3092337697 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fab3406b500 0x7fab3406d9c0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:49.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:49.217+0000 7fab5a600640 1 --2- 172.21.15.6:0/3092337697 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fab541622f0 0x7fab541469c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:49.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:49.217+0000 7fab5a600640 1 --2- 172.21.15.6:0/3092337697 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fab5406e030 0x7fab5415af50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:49.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:49.217+0000 7fab5a600640 1 -- 172.21.15.6:0/3092337697 >> 172.21.15.6:0/3092337697 conn(0x7fab540b17c0 msgr2=0x7fab540b0fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:49.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:49.218+0000 7fab5a600640 1 -- 172.21.15.6:0/3092337697 shutdown_connections 2024-03-04T09:28:49.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:49.218+0000 7fab5a600640 1 -- 172.21.15.6:0/3092337697 wait complete. 2024-03-04T09:28:49.239 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-04T09:28:49.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:49.460+0000 7f94c1800640 1 Processor -- start 2024-03-04T09:28:49.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:49.460+0000 7f94c1800640 1 -- start start 2024-03-04T09:28:49.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:49.461+0000 7f94c1800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f94bc145570 0x7f94bc0cef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:49.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:49.462+0000 7f94c1800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f94bc0cd430 0x7f94bc06d440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:49.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:49.462+0000 7f94c1800640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94bc0b4ae0 con 0x7f94bc0b42c0 2024-03-04T09:28:49.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:49.462+0000 7f94c1800640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94bc06c9f0 con 0x7f94bc0cd430 2024-03-04T09:28:49.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:49.462+0000 7f94c1800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94bc06cfb0 con 0x7f94bc145570 2024-03-04T09:28:49.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:49.462+0000 7f94bb400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f94bc0b42c0 0x7f94bc0ccd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:51672/0 (socket says 172.21.15.6:51672) 2024-03-04T09:28:49.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:49.462+0000 7f94bb400640 1 -- 172.21.15.6:0/611813939 learned_addr learned my addr 172.21.15.6:0/611813939 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:28:49.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:49.462+0000 7f94bbe00640 1 --2- 172.21.15.6:0/611813939 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f94bc145570 0x7f94bc0cef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:49.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:49.462+0000 7f94c0e00640 1 --2- 172.21.15.6:0/611813939 >> v2:172.21.15.146:3300/0 conn(0x7f94bc0cd430 0x7f94bc06d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:49.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:49.463+0000 7f94baa00640 1 -- 172.21.15.6:0/611813939 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1232323170 0 0) 0x7f94bc0b4ae0 con 0x7f94bc0b42c0 2024-03-04T09:28:49.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:49.463+0000 7f94baa00640 1 -- 172.21.15.6:0/611813939 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94a0003520 con 0x7f94bc0b42c0 2024-03-04T09:28:49.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:49.463+0000 7f94baa00640 1 -- 172.21.15.6:0/611813939 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2591657993 0 0) 0x7f94a0003520 con 0x7f94bc0b42c0 2024-03-04T09:28:49.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:49.463+0000 7f94baa00640 1 -- 172.21.15.6:0/611813939 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f94bc145570 msgr2=0x7f94bc0cef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:49.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:49.463+0000 7f94baa00640 1 --2- 172.21.15.6:0/611813939 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f94bc145570 0x7f94bc0cef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:49.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:49.463+0000 7f94baa00640 1 -- 172.21.15.6:0/611813939 >> v2:172.21.15.146:3300/0 conn(0x7f94bc0cd430 msgr2=0x7f94bc06d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:49.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:49.463+0000 7f94baa00640 1 --2- 172.21.15.6:0/611813939 >> v2:172.21.15.146:3300/0 conn(0x7f94bc0cd430 0x7f94bc06d440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:49.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:49.463+0000 7f94baa00640 1 -- 172.21.15.6:0/611813939 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94bc0b4dc0 con 0x7f94bc0b42c0 2024-03-04T09:28:49.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:49.463+0000 7f94baa00640 1 -- 172.21.15.6:0/611813939 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7f94a8001670 con 0x7f94bc0b42c0 2024-03-04T09:28:49.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:49.463+0000 7f94baa00640 1 -- 172.21.15.6:0/611813939 >> v1:172.21.15.6:6789/0 conn(0x7f94bc0b42c0 legacy=0x7f94bc0ccd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:49.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:49.463+0000 7f94baa00640 1 --2- 172.21.15.6:0/611813939 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f94bc145570 0x7f94a00039d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:49.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:49.463+0000 7f94baa00640 1 --2- 172.21.15.6:0/611813939 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f94a0003f50 0x7f94a0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:49.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:49.463+0000 7f94baa00640 1 --2- 172.21.15.6:0/611813939 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f94a0004d40 0x7f94a00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:49.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:49.463+0000 7f94baa00640 1 -- 172.21.15.6:0/611813939 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94bc06c9f0 con 0x7f94a0003f50 2024-03-04T09:28:49.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:49.463+0000 7f94baa00640 1 -- 172.21.15.6:0/611813939 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94bc06cfb0 con 0x7f94a0004d40 2024-03-04T09:28:49.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:49.463+0000 7f94baa00640 1 -- 172.21.15.6:0/611813939 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94a000b9c0 con 0x7f94bc145570 2024-03-04T09:28:49.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:49.464+0000 7f94bb400640 1 --2- 172.21.15.6:0/611813939 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f94a0003f50 0x7f94a0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:49.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:49.464+0000 7f94c0e00640 1 --2- 172.21.15.6:0/611813939 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f94a0004d40 0x7f94a00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:49.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:49.464+0000 7f94bbe00640 1 --2- 172.21.15.6:0/611813939 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f94bc145570 0x7f94a00039d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:49.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:49.464+0000 7f94bb400640 1 -- 172.21.15.6:0/611813939 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f94bc145570 msgr2=0x7f94a00039d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:49.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:49.464+0000 7f94bb400640 1 --2- 172.21.15.6:0/611813939 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f94bc145570 0x7f94a00039d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:49.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:49.464+0000 7f94bb400640 1 -- 172.21.15.6:0/611813939 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f94a0004d40 msgr2=0x7f94a00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:49.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:49.464+0000 7f94bb400640 1 --2- 172.21.15.6:0/611813939 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f94a0004d40 0x7f94a00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:49.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:49.464+0000 7f94bb400640 1 -- 172.21.15.6:0/611813939 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94a0003520 con 0x7f94a0003f50 2024-03-04T09:28:49.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:49.464+0000 7f94bbe00640 1 --2- 172.21.15.6:0/611813939 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f94bc145570 0x7f94a00039d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T09:28:49.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:49.466+0000 7f94bb400640 1 --2- 172.21.15.6:0/611813939 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f94a0003f50 0x7f94a0004800 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f94a800c2c0 tx=0x7f94a0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30570f86061cc39e server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:49.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:49.466+0000 7f94baa00640 1 -- 172.21.15.6:0/611813939 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94a8007d50 con 0x7f94a0003f50 2024-03-04T09:28:49.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:49.466+0000 7f94baa00640 1 -- 172.21.15.6:0/611813939 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94a8030070 con 0x7f94a0003f50 2024-03-04T09:28:49.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:49.466+0000 7f94baa00640 1 -- 172.21.15.6:0/611813939 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94a802cda0 con 0x7f94a0003f50 2024-03-04T09:28:49.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:49.467+0000 7f94c1800640 1 -- 172.21.15.6:0/611813939 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f94a0003f50 msgr2=0x7f94a0004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:49.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:49.467+0000 7f94c1800640 1 --2- 172.21.15.6:0/611813939 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f94a0003f50 0x7f94a0004800 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f94a800c2c0 tx=0x7f94a0005830 comp rx=0 tx=0).stop 2024-03-04T09:28:49.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:49.467+0000 7f94c0e00640 1 -- 172.21.15.6:0/611813939 reap_dead start 2024-03-04T09:28:49.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:49.467+0000 7f94c1800640 1 -- 172.21.15.6:0/611813939 shutdown_connections 2024-03-04T09:28:49.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:49.467+0000 7f94c1800640 1 -- 172.21.15.6:0/611813939 >> 172.21.15.6:0/611813939 conn(0x7f94bc06b190 msgr2=0x7f94bc06b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:49.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:49.467+0000 7f94c1800640 1 -- 172.21.15.6:0/611813939 shutdown_connections 2024-03-04T09:28:49.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:49.467+0000 7f94c1800640 1 -- 172.21.15.6:0/611813939 wait complete. 2024-03-04T09:28:49.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:49.469+0000 7f94c1800640 1 Processor -- start 2024-03-04T09:28:49.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:49.469+0000 7f94c1800640 1 -- start start 2024-03-04T09:28:49.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:49.470+0000 7f94c1800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f94a0003f50 0x7f94bc0ad0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:49.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:49.470+0000 7f94c1800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f94bc1fb750 0x7f94bc13d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:49.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:49.470+0000 7f94c1800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f94bc0ab0c0 0x7f94bc0ab570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:49.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:49.470+0000 7f94c1800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94bc0b4740 con 0x7f94a0003f50 2024-03-04T09:28:49.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:49.470+0000 7f94c1800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94bc15c110 con 0x7f94bc0ab0c0 2024-03-04T09:28:49.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:49.470+0000 7f94c1800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94bc0b48c0 con 0x7f94bc1fb750 2024-03-04T09:28:49.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:49.470+0000 7f94bbe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f94a0003f50 0x7f94bc0ad0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:49.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:49.470+0000 7f94bbe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f94a0003f50 0x7f94bc0ad0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49114/0 (socket says 172.21.15.6:49114) 2024-03-04T09:28:49.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:49.470+0000 7f94bbe00640 1 -- 172.21.15.6:0/2126646106 learned_addr learned my addr 172.21.15.6:0/2126646106 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:28:49.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:49.471+0000 7f94c0e00640 1 --2- 172.21.15.6:0/2126646106 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f94bc0ab0c0 0x7f94bc0ab570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:49.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:49.471+0000 7f94bbe00640 1 -- 172.21.15.6:0/2126646106 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f94bc1fb750 msgr2=0x7f94bc13d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:49.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:49.471+0000 7f94bbe00640 1 --2- 172.21.15.6:0/2126646106 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f94bc1fb750 0x7f94bc13d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:49.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:49.471+0000 7f94bbe00640 1 -- 172.21.15.6:0/2126646106 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f94bc0ab0c0 msgr2=0x7f94bc0ab570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:49.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:49.471+0000 7f94bbe00640 1 --2- 172.21.15.6:0/2126646106 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f94bc0ab0c0 0x7f94bc0ab570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:49.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:49.471+0000 7f94bbe00640 1 -- 172.21.15.6:0/2126646106 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94bc13dfe0 con 0x7f94a0003f50 2024-03-04T09:28:49.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:49.471+0000 7f94c0e00640 1 --2- 172.21.15.6:0/2126646106 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f94bc0ab0c0 0x7f94bc0ab570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:28:49.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:49.471+0000 7f94bbe00640 1 --2- 172.21.15.6:0/2126646106 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f94a0003f50 0x7f94bc0ad0e0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f94ac011630 tx=0x7f94ac011af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:49.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:49.471+0000 7f94b3e00640 1 -- 172.21.15.6:0/2126646106 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94ac00f920 con 0x7f94a0003f50 2024-03-04T09:28:49.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:49.471+0000 7f94b3e00640 1 -- 172.21.15.6:0/2126646106 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94ac00fac0 con 0x7f94a0003f50 2024-03-04T09:28:49.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:49.471+0000 7f94c1800640 1 -- 172.21.15.6:0/2126646106 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94bc1fbec0 con 0x7f94a0003f50 2024-03-04T09:28:49.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:49.471+0000 7f94b3e00640 1 -- 172.21.15.6:0/2126646106 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94ac00fe90 con 0x7f94a0003f50 2024-03-04T09:28:49.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:49.472+0000 7f94c1800640 1 -- 172.21.15.6:0/2126646106 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94bc1fc480 con 0x7f94a0003f50 2024-03-04T09:28:49.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:49.473+0000 7f94c1800640 1 -- 172.21.15.6:0/2126646106 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94bc1fc680 con 0x7f94a0003f50 2024-03-04T09:28:49.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:49.474+0000 7f94b3e00640 1 -- 172.21.15.6:0/2126646106 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f94ac015460 con 0x7f94a0003f50 2024-03-04T09:28:49.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:49.476+0000 7f94b3e00640 1 --2- 172.21.15.6:0/2126646106 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f949c06b6a0 0x7f949c06db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:49.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:49.476+0000 7f94b3e00640 1 -- 172.21.15.6:0/2126646106 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f94ac08a880 con 0x7f94a0003f50 2024-03-04T09:28:49.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:49.476+0000 7f94bb400640 1 --2- 172.21.15.6:0/2126646106 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f949c06b6a0 0x7f949c06db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:49.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:49.478+0000 7f94bb400640 1 --2- 172.21.15.6:0/2126646106 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f949c06b6a0 0x7f949c06db60 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f94a8034000 tx=0x7f94a8039040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:49.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:49.480+0000 7f94b3e00640 1 -- 172.21.15.6:0/2126646106 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f94ac019030 con 0x7f94a0003f50 2024-03-04T09:28:49.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:49.683+0000 7f94c1800640 1 -- 172.21.15.6:0/2126646106 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f94bc04f330 con 0x7f94a0003f50 2024-03-04T09:28:50.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:50.219+0000 7f94b3e00640 1 -- 172.21.15.6:0/2126646106 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v66) v1 ==== 170+0+0 (secure 0 0 0) 0x7f94ac059970 con 0x7f94a0003f50 2024-03-04T09:28:50.216 INFO:teuthology.orchestra.run.smithi006.stderr:pool 'cephfs_data' created 2024-03-04T09:28:50.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:50.223+0000 7f94c1800640 1 -- 172.21.15.6:0/2126646106 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f949c06b6a0 msgr2=0x7f949c06db60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:50.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:50.223+0000 7f94c1800640 1 --2- 172.21.15.6:0/2126646106 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f949c06b6a0 0x7f949c06db60 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f94a8034000 tx=0x7f94a8039040 comp rx=0 tx=0).stop 2024-03-04T09:28:50.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:50.223+0000 7f94c1800640 1 -- 172.21.15.6:0/2126646106 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f94a0003f50 msgr2=0x7f94bc0ad0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:50.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:50.223+0000 7f94c1800640 1 --2- 172.21.15.6:0/2126646106 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f94a0003f50 0x7f94bc0ad0e0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f94ac011630 tx=0x7f94ac011af0 comp rx=0 tx=0).stop 2024-03-04T09:28:50.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:50.223+0000 7f94c1800640 1 -- 172.21.15.6:0/2126646106 shutdown_connections 2024-03-04T09:28:50.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:50.223+0000 7f94c1800640 1 --2- 172.21.15.6:0/2126646106 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f94bc0ab0c0 0x7f94bc0ab570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:50.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:50.223+0000 7f94c1800640 1 --2- 172.21.15.6:0/2126646106 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f949c06b6a0 0x7f949c06db60 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:50.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:50.223+0000 7f94c1800640 1 --2- 172.21.15.6:0/2126646106 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f94bc1fb750 0x7f94bc13d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:50.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:50.223+0000 7f94c1800640 1 --2- 172.21.15.6:0/2126646106 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f94a0003f50 0x7f94bc0ad0e0 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:50.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:50.223+0000 7f94c1800640 1 -- 172.21.15.6:0/2126646106 >> 172.21.15.6:0/2126646106 conn(0x7f94bc06b190 msgr2=0x7f94bc0ae6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:50.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:50.225+0000 7f94c1800640 1 -- 172.21.15.6:0/2126646106 shutdown_connections 2024-03-04T09:28:50.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:50.228+0000 7f94c1800640 1 -- 172.21.15.6:0/2126646106 wait complete. 2024-03-04T09:28:50.258 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-04T09:28:50.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:50.448+0000 7f2aee600640 1 Processor -- start 2024-03-04T09:28:50.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:50.448+0000 7f2aee600640 1 -- start start 2024-03-04T09:28:50.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:50.449+0000 7f2aee600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2ae80b5340 0x7f2ae80d4a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:50.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:50.449+0000 7f2aee600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f2ae80d2660 0x7f2ae80c7f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:50.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:50.450+0000 7f2aee600640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ae8162da0 con 0x7f2ae80d2af0 2024-03-04T09:28:50.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:50.450+0000 7f2aee600640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ae80cf500 con 0x7f2ae80d2660 2024-03-04T09:28:50.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:50.450+0000 7f2aee600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ae80cf7e0 con 0x7f2ae80b5340 2024-03-04T09:28:50.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:50.450+0000 7f2ae6a00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f2ae80d2af0 0x7f2ae80d1f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:51680/0 (socket says 172.21.15.6:51680) 2024-03-04T09:28:50.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:50.450+0000 7f2ae6a00640 1 -- 172.21.15.6:0/4060793417 learned_addr learned my addr 172.21.15.6:0/4060793417 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:28:50.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:50.450+0000 7f2ae7e00640 1 --2- 172.21.15.6:0/4060793417 >> v2:172.21.15.6:3300/0 conn(0x7f2ae80d2660 0x7f2ae80c7f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:50.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:50.450+0000 7f2ae7400640 1 --2- 172.21.15.6:0/4060793417 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2ae80b5340 0x7f2ae80d4a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:50.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:50.450+0000 7f2ae7e00640 1 -- 172.21.15.6:0/4060793417 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2ae80b5340 msgr2=0x7f2ae80d4a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:50.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:50.450+0000 7f2ae7e00640 1 --2- 172.21.15.6:0/4060793417 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2ae80b5340 0x7f2ae80d4a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:50.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:50.450+0000 7f2ae7e00640 1 -- 172.21.15.6:0/4060793417 >> v1:172.21.15.6:6789/0 conn(0x7f2ae80d2af0 legacy=0x7f2ae80d1f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:50.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:50.451+0000 7f2ae7e00640 1 -- 172.21.15.6:0/4060793417 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ae80cfac0 con 0x7f2ae80d2660 2024-03-04T09:28:50.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:50.450+0000 7f2ae6000640 1 -- 172.21.15.6:0/4060793417 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2127636834 0 0) 0x7f2ae8162da0 con 0x7f2ae80d2af0 2024-03-04T09:28:50.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:50.452+0000 7f2ae7e00640 1 --2- 172.21.15.6:0/4060793417 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2ae80d2660 0x7f2ae80c7f10 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f2ae0009d90 tx=0x7f2ae002be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc7bd7b49b316bf9 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:50.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:50.452+0000 7f2ae6000640 1 -- 172.21.15.6:0/4060793417 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ae0039070 con 0x7f2ae80d2660 2024-03-04T09:28:50.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:50.452+0000 7f2ae6000640 1 -- 172.21.15.6:0/4060793417 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ae002cbc0 con 0x7f2ae80d2660 2024-03-04T09:28:50.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:50.452+0000 7f2ae6000640 1 -- 172.21.15.6:0/4060793417 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ae0034440 con 0x7f2ae80d2660 2024-03-04T09:28:50.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:50.453+0000 7f2aee600640 1 -- 172.21.15.6:0/4060793417 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2ae80d2660 msgr2=0x7f2ae80c7f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:50.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:50.453+0000 7f2aee600640 1 --2- 172.21.15.6:0/4060793417 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2ae80d2660 0x7f2ae80c7f10 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f2ae0009d90 tx=0x7f2ae002be10 comp rx=0 tx=0).stop 2024-03-04T09:28:50.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:50.453+0000 7f2aee600640 1 -- 172.21.15.6:0/4060793417 shutdown_connections 2024-03-04T09:28:50.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:50.453+0000 7f2aee600640 1 --2- 172.21.15.6:0/4060793417 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2ae80d2660 0x7f2ae80c7f10 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:50.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:50.453+0000 7f2aee600640 1 --2- 172.21.15.6:0/4060793417 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2ae80b5340 0x7f2ae80d4a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:50.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:50.453+0000 7f2aee600640 1 -- 172.21.15.6:0/4060793417 >> 172.21.15.6:0/4060793417 conn(0x7f2ae80d3ee0 msgr2=0x7f2ae80d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:50.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:50.453+0000 7f2aee600640 1 -- 172.21.15.6:0/4060793417 shutdown_connections 2024-03-04T09:28:50.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:50.453+0000 7f2aee600640 1 -- 172.21.15.6:0/4060793417 wait complete. 2024-03-04T09:28:50.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:50.454+0000 7f2aee600640 1 Processor -- start 2024-03-04T09:28:50.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:50.455+0000 7f2aee600640 1 -- start start 2024-03-04T09:28:50.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:50.456+0000 7f2aee600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2ae80b5340 0x7f2ae80cb970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:50.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:50.456+0000 7f2aee600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2ae80d2660 0x7f2ae8157340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:50.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:50.456+0000 7f2aee600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2ae80d2af0 0x7f2ae8164da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:50.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:50.456+0000 7f2aee600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ae80b5720 con 0x7f2ae80b5340 2024-03-04T09:28:50.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:50.456+0000 7f2aee600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ae80d0080 con 0x7f2ae80d2660 2024-03-04T09:28:50.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:50.456+0000 7f2aee600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ae80c8450 con 0x7f2ae80d2af0 2024-03-04T09:28:50.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:50.456+0000 7f2ae7400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2ae80b5340 0x7f2ae80cb970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:50.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:50.456+0000 7f2ae7400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2ae80b5340 0x7f2ae80cb970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49134/0 (socket says 172.21.15.6:49134) 2024-03-04T09:28:50.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:50.456+0000 7f2ae7400640 1 -- 172.21.15.6:0/1950122126 learned_addr learned my addr 172.21.15.6:0/1950122126 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:28:50.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:50.456+0000 7f2ae7400640 1 -- 172.21.15.6:0/1950122126 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2ae80d2af0 msgr2=0x7f2ae8164da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:50.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:50.456+0000 7f2ae7400640 1 --2- 172.21.15.6:0/1950122126 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2ae80d2af0 0x7f2ae8164da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:50.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:50.456+0000 7f2ae7400640 1 -- 172.21.15.6:0/1950122126 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2ae80d2660 msgr2=0x7f2ae8157340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:50.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:50.456+0000 7f2ae7400640 1 --2- 172.21.15.6:0/1950122126 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2ae80d2660 0x7f2ae8157340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:50.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:50.456+0000 7f2ae7400640 1 -- 172.21.15.6:0/1950122126 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ae8163060 con 0x7f2ae80b5340 2024-03-04T09:28:50.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:50.457+0000 7f2ae7400640 1 --2- 172.21.15.6:0/1950122126 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2ae80b5340 0x7f2ae80cb970 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f2ad800d830 tx=0x7f2ad800dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:50.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:50.457+0000 7f2adf400640 1 -- 172.21.15.6:0/1950122126 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ad800b660 con 0x7f2ae80b5340 2024-03-04T09:28:50.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:50.457+0000 7f2adf400640 1 -- 172.21.15.6:0/1950122126 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ad8019070 con 0x7f2ae80b5340 2024-03-04T09:28:50.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:50.457+0000 7f2aee600640 1 -- 172.21.15.6:0/1950122126 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ae8157880 con 0x7f2ae80b5340 2024-03-04T09:28:50.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:50.457+0000 7f2adf400640 1 -- 172.21.15.6:0/1950122126 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ad800ba60 con 0x7f2ae80b5340 2024-03-04T09:28:50.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:50.457+0000 7f2aee600640 1 -- 172.21.15.6:0/1950122126 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ae80cc380 con 0x7f2ae80b5340 2024-03-04T09:28:50.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:50.458+0000 7f2aee600640 1 -- 172.21.15.6:0/1950122126 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ae804f330 con 0x7f2ae80b5340 2024-03-04T09:28:50.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:50.460+0000 7f2adf400640 1 -- 172.21.15.6:0/1950122126 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2ad801e460 con 0x7f2ae80b5340 2024-03-04T09:28:50.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:50.460+0000 7f2adf400640 1 --2- 172.21.15.6:0/1950122126 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2ac806b4d0 0x7f2ac806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:50.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:50.460+0000 7f2adf400640 1 -- 172.21.15.6:0/1950122126 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f2ad808a8b0 con 0x7f2ae80b5340 2024-03-04T09:28:50.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:50.461+0000 7f2ae6a00640 1 --2- 172.21.15.6:0/1950122126 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2ac806b4d0 0x7f2ac806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:50.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:50.462+0000 7f2ae6a00640 1 --2- 172.21.15.6:0/1950122126 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2ac806b4d0 0x7f2ac806d990 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f2ad4007fd0 tx=0x7f2ad4007960 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:50.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:50.466+0000 7f2adf400640 1 -- 172.21.15.6:0/1950122126 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2ad80596f0 con 0x7f2ae80b5340 2024-03-04T09:28:50.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:50.705+0000 7f2aee600640 1 -- 172.21.15.6:0/1950122126 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f2ae806bc90 con 0x7f2ae80b5340 2024-03-04T09:28:51.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.260+0000 7f2adf400640 1 -- 172.21.15.6:0/1950122126 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 10 and data pool 11 v38) v1 ==== 174+0+0 (secure 0 0 0) 0x7f2ad8058ff0 con 0x7f2ae80b5340 2024-03-04T09:28:51.259 INFO:teuthology.orchestra.run.smithi006.stderr:new fs with metadata pool 10 and data pool 11 2024-03-04T09:28:51.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.270+0000 7f2aee600640 1 -- 172.21.15.6:0/1950122126 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2ac806b4d0 msgr2=0x7f2ac806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:51.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.270+0000 7f2aee600640 1 --2- 172.21.15.6:0/1950122126 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2ac806b4d0 0x7f2ac806d990 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f2ad4007fd0 tx=0x7f2ad4007960 comp rx=0 tx=0).stop 2024-03-04T09:28:51.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.270+0000 7f2aee600640 1 -- 172.21.15.6:0/1950122126 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2ae80b5340 msgr2=0x7f2ae80cb970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:51.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.270+0000 7f2aee600640 1 --2- 172.21.15.6:0/1950122126 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2ae80b5340 0x7f2ae80cb970 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f2ad800d830 tx=0x7f2ad800dcf0 comp rx=0 tx=0).stop 2024-03-04T09:28:51.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.271+0000 7f2aee600640 1 -- 172.21.15.6:0/1950122126 shutdown_connections 2024-03-04T09:28:51.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.271+0000 7f2aee600640 1 --2- 172.21.15.6:0/1950122126 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2ac806b4d0 0x7f2ac806d990 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:51.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.271+0000 7f2aee600640 1 --2- 172.21.15.6:0/1950122126 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2ae80d2af0 0x7f2ae8164da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:51.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.271+0000 7f2aee600640 1 --2- 172.21.15.6:0/1950122126 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2ae80d2660 0x7f2ae8157340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:51.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.271+0000 7f2aee600640 1 --2- 172.21.15.6:0/1950122126 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2ae80b5340 0x7f2ae80cb970 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:51.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.271+0000 7f2aee600640 1 -- 172.21.15.6:0/1950122126 >> 172.21.15.6:0/1950122126 conn(0x7f2ae80d3ee0 msgr2=0x7f2ae806d320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:51.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.271+0000 7f2aee600640 1 -- 172.21.15.6:0/1950122126 shutdown_connections 2024-03-04T09:28:51.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.272+0000 7f2aee600640 1 -- 172.21.15.6:0/1950122126 wait complete. 2024-03-04T09:28:51.303 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T09:28:51.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.486+0000 7fce01e00640 1 Processor -- start 2024-03-04T09:28:51.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.486+0000 7fce01e00640 1 -- start start 2024-03-04T09:28:51.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.488+0000 7fce01e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fcdfc0b34e0 0x7fcdfc15c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:51.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.488+0000 7fce01e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdfc15cc40 con 0x7fcdfc0af580 2024-03-04T09:28:51.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.488+0000 7fce01e00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdfc15cf20 con 0x7fcdfc0af1a0 2024-03-04T09:28:51.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.488+0000 7fce01e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcdfc15d200 con 0x7fcdfc0b34e0 2024-03-04T09:28:51.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.488+0000 7fcdfa000640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fcdfc0af580 0x7fcdfc0b2dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:51692/0 (socket says 172.21.15.6:51692) 2024-03-04T09:28:51.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.488+0000 7fcdfa000640 1 -- 172.21.15.6:0/3379193998 learned_addr learned my addr 172.21.15.6:0/3379193998 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:28:51.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.488+0000 7fcdfb400640 1 --2- 172.21.15.6:0/3379193998 >> v2:172.21.15.6:3300/0 conn(0x7fcdfc0b34e0 0x7fcdfc15c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:51.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.489+0000 7fcdf9600640 1 -- 172.21.15.6:0/3379193998 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2943029124 0 0) 0x7fcdfc15cc40 con 0x7fcdfc0af580 2024-03-04T09:28:51.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.489+0000 7fcdf9600640 1 -- 172.21.15.6:0/3379193998 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcde0003520 con 0x7fcdfc0af580 2024-03-04T09:28:51.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.489+0000 7fcdfb400640 1 -- 172.21.15.6:0/3379193998 >> v1:172.21.15.146:6789/0 conn(0x7fcdfc0af1a0 legacy=0x7fcdfc0b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:51.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.489+0000 7fcdfb400640 1 -- 172.21.15.6:0/3379193998 >> v1:172.21.15.6:6789/0 conn(0x7fcdfc0af580 legacy=0x7fcdfc0b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:51.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.489+0000 7fcdfb400640 1 -- 172.21.15.6:0/3379193998 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdfc15d4e0 con 0x7fcdfc0b34e0 2024-03-04T09:28:51.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.490+0000 7fcdfb400640 1 --2- 172.21.15.6:0/3379193998 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcdfc0b34e0 0x7fcdfc15c600 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fcdf40097a0 tx=0x7fcdf402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5a06a0176a02004 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:51.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.490+0000 7fcdf9600640 1 -- 172.21.15.6:0/3379193998 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdf4035020 con 0x7fcdfc0b34e0 2024-03-04T09:28:51.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.490+0000 7fcdf9600640 1 -- 172.21.15.6:0/3379193998 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdf4031830 con 0x7fcdfc0b34e0 2024-03-04T09:28:51.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.490+0000 7fcdf9600640 1 -- 172.21.15.6:0/3379193998 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdf4031be0 con 0x7fcdfc0b34e0 2024-03-04T09:28:51.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.491+0000 7fce01e00640 1 -- 172.21.15.6:0/3379193998 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcdfc0b34e0 msgr2=0x7fcdfc15c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:51.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.491+0000 7fce01e00640 1 --2- 172.21.15.6:0/3379193998 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcdfc0b34e0 0x7fcdfc15c600 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fcdf40097a0 tx=0x7fcdf402b8d0 comp rx=0 tx=0).stop 2024-03-04T09:28:51.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.491+0000 7fce01e00640 1 -- 172.21.15.6:0/3379193998 shutdown_connections 2024-03-04T09:28:51.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.491+0000 7fce01e00640 1 --2- 172.21.15.6:0/3379193998 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcdfc0b34e0 0x7fcdfc15c600 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:51.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.491+0000 7fce01e00640 1 -- 172.21.15.6:0/3379193998 >> 172.21.15.6:0/3379193998 conn(0x7fcdfc06afb0 msgr2=0x7fcdfc06b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:51.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.491+0000 7fce01e00640 1 -- 172.21.15.6:0/3379193998 shutdown_connections 2024-03-04T09:28:51.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.491+0000 7fce01e00640 1 -- 172.21.15.6:0/3379193998 wait complete. 2024-03-04T09:28:51.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.493+0000 7fce01e00640 1 Processor -- start 2024-03-04T09:28:51.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.493+0000 7fce01e00640 1 -- start start 2024-03-04T09:28:51.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.494+0000 7fce01e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcdfc0af1a0 0x7fcdfc1f48f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:51.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.494+0000 7fce01e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcdfc0af580 0x7fcdfc148610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:51.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.494+0000 7fce01e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcdfc0b34e0 0x7fcdfc09d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:51.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.494+0000 7fce01e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdfc155e70 con 0x7fcdfc0b34e0 2024-03-04T09:28:51.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.494+0000 7fce01e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdfc0cde30 con 0x7fcdfc0af580 2024-03-04T09:28:51.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.494+0000 7fce01e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdfc155ff0 con 0x7fcdfc0af1a0 2024-03-04T09:28:51.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.494+0000 7fcdfb400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcdfc0b34e0 0x7fcdfc09d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:51.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.494+0000 7fcdfb400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcdfc0b34e0 0x7fcdfc09d020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49150/0 (socket says 172.21.15.6:49150) 2024-03-04T09:28:51.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.494+0000 7fcdfb400640 1 -- 172.21.15.6:0/1135218516 learned_addr learned my addr 172.21.15.6:0/1135218516 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:28:51.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.494+0000 7fcdfa000640 1 --2- 172.21.15.6:0/1135218516 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcdfc0af580 0x7fcdfc148610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:51.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.495+0000 7fcdfaa00640 1 --2- 172.21.15.6:0/1135218516 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcdfc0af1a0 0x7fcdfc1f48f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:51.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.495+0000 7fcdfb400640 1 -- 172.21.15.6:0/1135218516 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcdfc0af1a0 msgr2=0x7fcdfc1f48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:51.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.495+0000 7fcdfb400640 1 --2- 172.21.15.6:0/1135218516 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcdfc0af1a0 0x7fcdfc1f48f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:51.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.495+0000 7fcdfb400640 1 -- 172.21.15.6:0/1135218516 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcdfc0af580 msgr2=0x7fcdfc148610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:51.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.495+0000 7fcdfb400640 1 --2- 172.21.15.6:0/1135218516 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcdfc0af580 0x7fcdfc148610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:51.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.495+0000 7fcdfb400640 1 -- 172.21.15.6:0/1135218516 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdfc15d880 con 0x7fcdfc0b34e0 2024-03-04T09:28:51.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.495+0000 7fcdfaa00640 1 --2- 172.21.15.6:0/1135218516 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcdfc0af1a0 0x7fcdfc1f48f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:28:51.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.495+0000 7fcdfb400640 1 --2- 172.21.15.6:0/1135218516 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcdfc0b34e0 0x7fcdfc09d020 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7fcdf4034040 tx=0x7fcdf40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:51.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.495+0000 7fcdf3400640 1 -- 172.21.15.6:0/1135218516 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdf4035040 con 0x7fcdfc0b34e0 2024-03-04T09:28:51.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.496+0000 7fcdf3400640 1 -- 172.21.15.6:0/1135218516 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdf4004230 con 0x7fcdfc0b34e0 2024-03-04T09:28:51.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.496+0000 7fcdf3400640 1 -- 172.21.15.6:0/1135218516 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdf403a640 con 0x7fcdfc0b34e0 2024-03-04T09:28:51.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.496+0000 7fce01e00640 1 -- 172.21.15.6:0/1135218516 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdfc15db40 con 0x7fcdfc0b34e0 2024-03-04T09:28:51.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.496+0000 7fce01e00640 1 -- 172.21.15.6:0/1135218516 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdfc15e100 con 0x7fcdfc0b34e0 2024-03-04T09:28:51.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.497+0000 7fce01e00640 1 -- 172.21.15.6:0/1135218516 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdfc069510 con 0x7fcdfc0b34e0 2024-03-04T09:28:51.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.498+0000 7fcdf3400640 1 -- 172.21.15.6:0/1135218516 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcdf4038020 con 0x7fcdfc0b34e0 2024-03-04T09:28:51.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.499+0000 7fcdf3400640 1 --2- 172.21.15.6:0/1135218516 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fcdd806b4d0 0x7fcdd806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:51.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.499+0000 7fcdf3400640 1 -- 172.21.15.6:0/1135218516 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcdf40abfb0 con 0x7fcdfc0b34e0 2024-03-04T09:28:51.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.500+0000 7fcdfaa00640 1 --2- 172.21.15.6:0/1135218516 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fcdd806b4d0 0x7fcdd806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:51.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.501+0000 7fcdfaa00640 1 --2- 172.21.15.6:0/1135218516 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fcdd806b4d0 0x7fcdd806d990 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fcdfc0cc020 tx=0x7fcdec009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:51.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.504+0000 7fcdf3400640 1 -- 172.21.15.6:0/1135218516 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcdf407ae20 con 0x7fcdfc0b34e0 2024-03-04T09:28:51.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.704+0000 7fce01e00640 1 -- 172.21.15.6:0/1135218516 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcdfc0aeaf0 con 0x7fcdfc0b34e0 2024-03-04T09:28:51.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.707+0000 7fcdf3400640 1 -- 172.21.15.6:0/1135218516 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v67) v1 ==== 74+0+16633 (secure 0 0 0) 0x7fcdf407a720 con 0x7fcdfc0b34e0 2024-03-04T09:28:51.705 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:28:51.705 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":67,"fsid":"edce0578-5ae4-4402-b45a-552bdef84d7c","created":"2024-03-04T09:24:54.523780+0000","modified":"2024-03-04T09:28:50.707651+0000","last_up_change":"2024-03-04T09:25:07.089049+0000","last_in_change":"2024-03-04T09:24:57.982077+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T09:25:07.935198+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-03-04T09:28:48.627427+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-03-04T09:28:49.739043+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a96574f-81f1-4bc6-8463-757520b84a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6825","nonce":1364449401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6827","nonce":1364449401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6831","nonce":1364449401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6829","nonce":1364449401}]},"public_addr":"172.21.15.6:6825/1364449401","cluster_addr":"172.21.15.6:6827/1364449401","heartbeat_back_addr":"172.21.15.6:6831/1364449401","heartbeat_front_addr":"172.21.15.6:6829/1364449401","state":["exists","up"]},{"osd":1,"uuid":"7d79b9c1-6b87-476a-bc36-7db5ee64cd90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6809","nonce":2494103078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6811","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6813","nonce":2494103078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6821","nonce":2494103078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6817","nonce":2494103078}]},"public_addr":"172.21.15.6:6809/2494103078","cluster_addr":"172.21.15.6:6813/2494103078","heartbeat_back_addr":"172.21.15.6:6821/2494103078","heartbeat_front_addr":"172.21.15.6:6817/2494103078","state":["exists","up"]},{"osd":2,"uuid":"492331b9-4ee9-40b6-bacd-85d44cd6c45a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6812","nonce":1500066681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6816","nonce":1500066681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6824","nonce":1500066681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6820","nonce":1500066681}]},"public_addr":"172.21.15.6:6812/1500066681","cluster_addr":"172.21.15.6:6816/1500066681","heartbeat_back_addr":"172.21.15.6:6824/1500066681","heartbeat_front_addr":"172.21.15.6:6820/1500066681","state":["exists","up"]},{"osd":3,"uuid":"21bb2efe-9ba7-4bdc-bc31-180cd1102e21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6801","nonce":1558806123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6803","nonce":1558806123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6807","nonce":1558806123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6805","nonce":1558806123}]},"public_addr":"172.21.15.6:6801/1558806123","cluster_addr":"172.21.15.6:6803/1558806123","heartbeat_back_addr":"172.21.15.6:6807/1558806123","heartbeat_front_addr":"172.21.15.6:6805/1558806123","state":["exists","up"]},{"osd":4,"uuid":"6e2c4b71-19d9-4737-83d9-558705d8644a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6801","nonce":1526135780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6803","nonce":1526135780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6807","nonce":1526135780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6805","nonce":1526135780}]},"public_addr":"172.21.15.146:6801/1526135780","cluster_addr":"172.21.15.146:6803/1526135780","heartbeat_back_addr":"172.21.15.146:6807/1526135780","heartbeat_front_addr":"172.21.15.146:6805/1526135780","state":["exists","up"]},{"osd":5,"uuid":"fb69c525-f1be-4b7d-ab32-6c29b0d6fddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6809","nonce":460262507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6811","nonce":460262507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6815","nonce":460262507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6813","nonce":460262507}]},"public_addr":"172.21.15.146:6809/460262507","cluster_addr":"172.21.15.146:6811/460262507","heartbeat_back_addr":"172.21.15.146:6815/460262507","heartbeat_front_addr":"172.21.15.146:6813/460262507","state":["exists","up"]},{"osd":6,"uuid":"c32e7fed-370d-4401-8ce0-6a4335b0d168","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6823","nonce":1392897493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6827","nonce":1392897493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6831","nonce":1392897493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6829","nonce":1392897493}]},"public_addr":"172.21.15.146:6823/1392897493","cluster_addr":"172.21.15.146:6827/1392897493","heartbeat_back_addr":"172.21.15.146:6831/1392897493","heartbeat_front_addr":"172.21.15.146:6829/1392897493","state":["exists","up"]},{"osd":7,"uuid":"75a79b44-fa5f-426d-a7d3-71f21f78835c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6817","nonce":3268323156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6819","nonce":3268323156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6825","nonce":3268323156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6822","nonce":3268323156}]},"public_addr":"172.21.15.146:6817/3268323156","cluster_addr":"172.21.15.146:6819/3268323156","heartbeat_back_addr":"172.21.15.146:6825/3268323156","heartbeat_front_addr":"172.21.15.146:6822/3268323156","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.310437+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.295850+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.294674+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.526855+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.300023+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.595160+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.346046+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.558741+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T09:28:51.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.709+0000 7fce01e00640 1 -- 172.21.15.6:0/1135218516 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fcdd806b4d0 msgr2=0x7fcdd806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:51.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.709+0000 7fce01e00640 1 --2- 172.21.15.6:0/1135218516 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fcdd806b4d0 0x7fcdd806d990 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fcdfc0cc020 tx=0x7fcdec009040 comp rx=0 tx=0).stop 2024-03-04T09:28:51.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.709+0000 7fce01e00640 1 -- 172.21.15.6:0/1135218516 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcdfc0b34e0 msgr2=0x7fcdfc09d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:51.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.709+0000 7fce01e00640 1 --2- 172.21.15.6:0/1135218516 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcdfc0b34e0 0x7fcdfc09d020 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7fcdf4034040 tx=0x7fcdf40046c0 comp rx=0 tx=0).stop 2024-03-04T09:28:51.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.710+0000 7fce01e00640 1 -- 172.21.15.6:0/1135218516 shutdown_connections 2024-03-04T09:28:51.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.710+0000 7fce01e00640 1 --2- 172.21.15.6:0/1135218516 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcdfc0b34e0 0x7fcdfc09d020 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:51.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.710+0000 7fce01e00640 1 --2- 172.21.15.6:0/1135218516 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcdfc0af580 0x7fcdfc148610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:51.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.710+0000 7fce01e00640 1 --2- 172.21.15.6:0/1135218516 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fcdd806b4d0 0x7fcdd806d990 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:51.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.710+0000 7fce01e00640 1 --2- 172.21.15.6:0/1135218516 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcdfc0af1a0 0x7fcdfc1f48f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:51.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.710+0000 7fce01e00640 1 -- 172.21.15.6:0/1135218516 >> 172.21.15.6:0/1135218516 conn(0x7fcdfc06afb0 msgr2=0x7fcdfc0b2ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:51.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.710+0000 7fce01e00640 1 -- 172.21.15.6:0/1135218516 shutdown_connections 2024-03-04T09:28:51.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.710+0000 7fce01e00640 1 -- 172.21.15.6:0/1135218516 wait complete. 2024-03-04T09:28:51.727 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T09:28:51.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.948+0000 7f1fc9000640 1 Processor -- start 2024-03-04T09:28:51.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.949+0000 7f1fc9000640 1 -- start start 2024-03-04T09:28:51.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.950+0000 7f1fc9000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1fc406ed60 0x7f1fc406d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:51.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.950+0000 7f1fc9000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f1fc40aedd0 0x7f1fc40af260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:51.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.950+0000 7f1fc9000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fc406ddf0 con 0x7f1fc4071760 2024-03-04T09:28:51.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.950+0000 7f1fc9000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fc406e360 con 0x7f1fc40aedd0 2024-03-04T09:28:51.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.950+0000 7f1fc9000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fc406e5b0 con 0x7f1fc406ed60 2024-03-04T09:28:51.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.951+0000 7f1fc2800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f1fc40aedd0 0x7f1fc40af260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:51.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.951+0000 7f1fc2800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f1fc40aedd0 0x7f1fc40af260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.6:59628/0 (socket says 172.21.15.6:59628) 2024-03-04T09:28:51.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.951+0000 7f1fc1e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1fc406ed60 0x7f1fc406d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:51.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.951+0000 7f1fc2800640 1 -- 172.21.15.6:0/1824737088 learned_addr learned my addr 172.21.15.6:0/1824737088 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:28:51.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.951+0000 7f1fc1400640 1 -- 172.21.15.6:0/1824737088 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4086053397 0 0) 0x7f1fc406ddf0 con 0x7f1fc4071760 2024-03-04T09:28:51.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.952+0000 7f1fc1400640 1 -- 172.21.15.6:0/1824737088 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fa8003520 con 0x7f1fc4071760 2024-03-04T09:28:51.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.952+0000 7f1fc1e00640 1 -- 172.21.15.6:0/1824737088 >> v2:172.21.15.146:3300/0 conn(0x7f1fc40aedd0 msgr2=0x7f1fc40af260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:51.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.952+0000 7f1fc1e00640 1 --2- 172.21.15.6:0/1824737088 >> v2:172.21.15.146:3300/0 conn(0x7f1fc40aedd0 0x7f1fc40af260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:51.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.952+0000 7f1fc1e00640 1 -- 172.21.15.6:0/1824737088 >> v1:172.21.15.146:6789/0 conn(0x7f1fc4071760 legacy=0x7f1fc4150e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:51.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.952+0000 7f1fc1e00640 1 -- 172.21.15.6:0/1824737088 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fc406e890 con 0x7f1fc406ed60 2024-03-04T09:28:51.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.953+0000 7f1fc1e00640 1 --2- 172.21.15.6:0/1824737088 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1fc406ed60 0x7f1fc406d6a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f1fb40097a0 tx=0x7f1fb402b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=98524ad02414b98f server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:51.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.954+0000 7f1fc1400640 1 -- 172.21.15.6:0/1824737088 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fb40315a0 con 0x7f1fc406ed60 2024-03-04T09:28:51.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.954+0000 7f1fc1400640 1 -- 172.21.15.6:0/1824737088 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fb4037070 con 0x7f1fc406ed60 2024-03-04T09:28:51.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.954+0000 7f1fc1400640 1 -- 172.21.15.6:0/1824737088 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fb4031960 con 0x7f1fc406ed60 2024-03-04T09:28:51.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.954+0000 7f1fc9000640 1 -- 172.21.15.6:0/1824737088 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1fc406ed60 msgr2=0x7f1fc406d6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:51.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.954+0000 7f1fc9000640 1 --2- 172.21.15.6:0/1824737088 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1fc406ed60 0x7f1fc406d6a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f1fb40097a0 tx=0x7f1fb402b840 comp rx=0 tx=0).stop 2024-03-04T09:28:51.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.954+0000 7f1fc9000640 1 -- 172.21.15.6:0/1824737088 shutdown_connections 2024-03-04T09:28:51.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.954+0000 7f1fc9000640 1 --2- 172.21.15.6:0/1824737088 >> v2:172.21.15.146:3300/0 conn(0x7f1fc40aedd0 0x7f1fc40af260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:51.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.954+0000 7f1fc9000640 1 --2- 172.21.15.6:0/1824737088 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1fc406ed60 0x7f1fc406d6a0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:51.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.954+0000 7f1fc9000640 1 -- 172.21.15.6:0/1824737088 >> 172.21.15.6:0/1824737088 conn(0x7f1fc406afb0 msgr2=0x7f1fc406b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:51.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.954+0000 7f1fc9000640 1 -- 172.21.15.6:0/1824737088 shutdown_connections 2024-03-04T09:28:51.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.954+0000 7f1fc9000640 1 -- 172.21.15.6:0/1824737088 wait complete. 2024-03-04T09:28:51.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.956+0000 7f1fc9000640 1 Processor -- start 2024-03-04T09:28:51.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.956+0000 7f1fc9000640 1 -- start start 2024-03-04T09:28:51.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.957+0000 7f1fc9000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1fc406ed60 0x7f1fc41f3a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:51.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.957+0000 7f1fc9000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1fc4071760 0x7f1fc414d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:51.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.957+0000 7f1fc9000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1fc40aedd0 0x7f1fc414e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:51.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.957+0000 7f1fc9000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fc4071d40 con 0x7f1fc406ed60 2024-03-04T09:28:51.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.957+0000 7f1fc9000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fc4071bc0 con 0x7f1fc40aedd0 2024-03-04T09:28:51.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.957+0000 7f1fc9000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fc415dd20 con 0x7f1fc4071760 2024-03-04T09:28:51.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.957+0000 7f1fc1e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1fc406ed60 0x7f1fc41f3a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:51.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.958+0000 7f1fc1e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1fc406ed60 0x7f1fc41f3a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49154/0 (socket says 172.21.15.6:49154) 2024-03-04T09:28:51.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.958+0000 7f1fc1e00640 1 -- 172.21.15.6:0/2169451225 learned_addr learned my addr 172.21.15.6:0/2169451225 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:28:51.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.958+0000 7f1fc2800640 1 --2- 172.21.15.6:0/2169451225 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1fc40aedd0 0x7f1fc414e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:51.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.958+0000 7f1fc1e00640 1 -- 172.21.15.6:0/2169451225 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1fc4071760 msgr2=0x7f1fc414d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:51.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.958+0000 7f1fc1e00640 1 --2- 172.21.15.6:0/2169451225 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1fc4071760 0x7f1fc414d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:51.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.958+0000 7f1fc1e00640 1 -- 172.21.15.6:0/2169451225 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1fc40aedd0 msgr2=0x7f1fc414e520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:51.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.958+0000 7f1fc1e00640 1 --2- 172.21.15.6:0/2169451225 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1fc40aedd0 0x7f1fc414e520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:51.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.958+0000 7f1fc1e00640 1 -- 172.21.15.6:0/2169451225 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fc41f3ff0 con 0x7f1fc406ed60 2024-03-04T09:28:51.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.958+0000 7f1fc2800640 1 --2- 172.21.15.6:0/2169451225 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1fc40aedd0 0x7f1fc414e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:28:51.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.958+0000 7f1fc1e00640 1 --2- 172.21.15.6:0/2169451225 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1fc406ed60 0x7f1fc41f3a10 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f1fb402b880 tx=0x7f1fb40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:51.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.959+0000 7f1fbaa00640 1 -- 172.21.15.6:0/2169451225 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fb4032630 con 0x7f1fc406ed60 2024-03-04T09:28:51.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.959+0000 7f1fbaa00640 1 -- 172.21.15.6:0/2169451225 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fb4032810 con 0x7f1fc406ed60 2024-03-04T09:28:51.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.959+0000 7f1fbaa00640 1 -- 172.21.15.6:0/2169451225 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fb4032c00 con 0x7f1fc406ed60 2024-03-04T09:28:51.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.959+0000 7f1fc9000640 1 -- 172.21.15.6:0/2169451225 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fc41f42d0 con 0x7f1fc406ed60 2024-03-04T09:28:51.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.959+0000 7f1fc9000640 1 -- 172.21.15.6:0/2169451225 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fc41f70a0 con 0x7f1fc406ed60 2024-03-04T09:28:51.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.960+0000 7f1fc9000640 1 -- 172.21.15.6:0/2169451225 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fc4069510 con 0x7f1fc406ed60 2024-03-04T09:28:51.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.961+0000 7f1fbaa00640 1 -- 172.21.15.6:0/2169451225 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1fb404d3f0 con 0x7f1fc406ed60 2024-03-04T09:28:51.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.962+0000 7f1fbaa00640 1 --2- 172.21.15.6:0/2169451225 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f1fa006b4d0 0x7f1fa006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:51.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.962+0000 7f1fbaa00640 1 -- 172.21.15.6:0/2169451225 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1fb402f030 con 0x7f1fc406ed60 2024-03-04T09:28:51.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.963+0000 7f1fb9400640 1 --2- 172.21.15.6:0/2169451225 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f1fa006b4d0 0x7f1fa006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:51.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.964+0000 7f1fb9400640 1 --2- 172.21.15.6:0/2169451225 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f1fa006b4d0 0x7f1fa006d990 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f1fac0011e0 tx=0x7f1fac0078f0 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:51.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:51.967+0000 7f1fbaa00640 1 -- 172.21.15.6:0/2169451225 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1fb4040070 con 0x7f1fc406ed60 2024-03-04T09:28:52.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:52.166+0000 7f1fc9000640 1 -- 172.21.15.6:0/2169451225 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1fc404f3a0 con 0x7f1fc406ed60 2024-03-04T09:28:52.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:52.169+0000 7f1fbaa00640 1 -- 172.21.15.6:0/2169451225 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v67) v1 ==== 74+0+16633 (secure 0 0 0) 0x7f1fb407a630 con 0x7f1fc406ed60 2024-03-04T09:28:52.167 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:28:52.167 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":67,"fsid":"edce0578-5ae4-4402-b45a-552bdef84d7c","created":"2024-03-04T09:24:54.523780+0000","modified":"2024-03-04T09:28:50.707651+0000","last_up_change":"2024-03-04T09:25:07.089049+0000","last_in_change":"2024-03-04T09:24:57.982077+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T09:25:07.935198+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-03-04T09:28:48.627427+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-03-04T09:28:49.739043+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a96574f-81f1-4bc6-8463-757520b84a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6825","nonce":1364449401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6827","nonce":1364449401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6831","nonce":1364449401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6829","nonce":1364449401}]},"public_addr":"172.21.15.6:6825/1364449401","cluster_addr":"172.21.15.6:6827/1364449401","heartbeat_back_addr":"172.21.15.6:6831/1364449401","heartbeat_front_addr":"172.21.15.6:6829/1364449401","state":["exists","up"]},{"osd":1,"uuid":"7d79b9c1-6b87-476a-bc36-7db5ee64cd90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6809","nonce":2494103078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6811","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6813","nonce":2494103078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6821","nonce":2494103078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6817","nonce":2494103078}]},"public_addr":"172.21.15.6:6809/2494103078","cluster_addr":"172.21.15.6:6813/2494103078","heartbeat_back_addr":"172.21.15.6:6821/2494103078","heartbeat_front_addr":"172.21.15.6:6817/2494103078","state":["exists","up"]},{"osd":2,"uuid":"492331b9-4ee9-40b6-bacd-85d44cd6c45a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6812","nonce":1500066681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6816","nonce":1500066681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6824","nonce":1500066681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6820","nonce":1500066681}]},"public_addr":"172.21.15.6:6812/1500066681","cluster_addr":"172.21.15.6:6816/1500066681","heartbeat_back_addr":"172.21.15.6:6824/1500066681","heartbeat_front_addr":"172.21.15.6:6820/1500066681","state":["exists","up"]},{"osd":3,"uuid":"21bb2efe-9ba7-4bdc-bc31-180cd1102e21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6801","nonce":1558806123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6803","nonce":1558806123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6807","nonce":1558806123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6805","nonce":1558806123}]},"public_addr":"172.21.15.6:6801/1558806123","cluster_addr":"172.21.15.6:6803/1558806123","heartbeat_back_addr":"172.21.15.6:6807/1558806123","heartbeat_front_addr":"172.21.15.6:6805/1558806123","state":["exists","up"]},{"osd":4,"uuid":"6e2c4b71-19d9-4737-83d9-558705d8644a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6801","nonce":1526135780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6803","nonce":1526135780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6807","nonce":1526135780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6805","nonce":1526135780}]},"public_addr":"172.21.15.146:6801/1526135780","cluster_addr":"172.21.15.146:6803/1526135780","heartbeat_back_addr":"172.21.15.146:6807/1526135780","heartbeat_front_addr":"172.21.15.146:6805/1526135780","state":["exists","up"]},{"osd":5,"uuid":"fb69c525-f1be-4b7d-ab32-6c29b0d6fddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6809","nonce":460262507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6811","nonce":460262507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6815","nonce":460262507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6813","nonce":460262507}]},"public_addr":"172.21.15.146:6809/460262507","cluster_addr":"172.21.15.146:6811/460262507","heartbeat_back_addr":"172.21.15.146:6815/460262507","heartbeat_front_addr":"172.21.15.146:6813/460262507","state":["exists","up"]},{"osd":6,"uuid":"c32e7fed-370d-4401-8ce0-6a4335b0d168","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6823","nonce":1392897493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6827","nonce":1392897493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6831","nonce":1392897493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6829","nonce":1392897493}]},"public_addr":"172.21.15.146:6823/1392897493","cluster_addr":"172.21.15.146:6827/1392897493","heartbeat_back_addr":"172.21.15.146:6831/1392897493","heartbeat_front_addr":"172.21.15.146:6829/1392897493","state":["exists","up"]},{"osd":7,"uuid":"75a79b44-fa5f-426d-a7d3-71f21f78835c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6817","nonce":3268323156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6819","nonce":3268323156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6825","nonce":3268323156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6822","nonce":3268323156}]},"public_addr":"172.21.15.146:6817/3268323156","cluster_addr":"172.21.15.146:6819/3268323156","heartbeat_back_addr":"172.21.15.146:6825/3268323156","heartbeat_front_addr":"172.21.15.146:6822/3268323156","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.310437+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.295850+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.294674+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.526855+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.300023+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.595160+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.346046+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.558741+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T09:28:52.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:52.171+0000 7f1fc9000640 1 -- 172.21.15.6:0/2169451225 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f1fa006b4d0 msgr2=0x7f1fa006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:52.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:52.171+0000 7f1fc9000640 1 --2- 172.21.15.6:0/2169451225 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f1fa006b4d0 0x7f1fa006d990 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f1fac0011e0 tx=0x7f1fac0078f0 comp rx=0 tx=0).stop 2024-03-04T09:28:52.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:52.171+0000 7f1fc9000640 1 -- 172.21.15.6:0/2169451225 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1fc406ed60 msgr2=0x7f1fc41f3a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:52.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:52.171+0000 7f1fc9000640 1 --2- 172.21.15.6:0/2169451225 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1fc406ed60 0x7f1fc41f3a10 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f1fb402b880 tx=0x7f1fb40046c0 comp rx=0 tx=0).stop 2024-03-04T09:28:52.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:52.171+0000 7f1fc9000640 1 -- 172.21.15.6:0/2169451225 shutdown_connections 2024-03-04T09:28:52.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:52.171+0000 7f1fc9000640 1 --2- 172.21.15.6:0/2169451225 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1fc40aedd0 0x7f1fc414e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:52.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:52.171+0000 7f1fc9000640 1 --2- 172.21.15.6:0/2169451225 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f1fa006b4d0 0x7f1fa006d990 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:52.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:52.171+0000 7f1fc9000640 1 --2- 172.21.15.6:0/2169451225 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1fc4071760 0x7f1fc414d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:52.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:52.171+0000 7f1fc9000640 1 --2- 172.21.15.6:0/2169451225 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1fc406ed60 0x7f1fc41f3a10 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:52.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:52.172+0000 7f1fc9000640 1 -- 172.21.15.6:0/2169451225 >> 172.21.15.6:0/2169451225 conn(0x7f1fc406afb0 msgr2=0x7f1fc415b860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:52.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:52.172+0000 7f1fc9000640 1 -- 172.21.15.6:0/2169451225 shutdown_connections 2024-03-04T09:28:52.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:52.172+0000 7f1fc9000640 1 -- 172.21.15.6:0/2169451225 wait complete. 2024-03-04T09:28:52.189 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-04T09:28:52.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:52.418+0000 7fc266a00640 1 Processor -- start 2024-03-04T09:28:52.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:52.418+0000 7fc266a00640 1 -- start start 2024-03-04T09:28:52.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:52.420+0000 7fc266a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc260072520 0x7fc2600cdfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:52.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:52.420+0000 7fc266a00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fc260071e20 0x7fc2600ce4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:52.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:52.420+0000 7fc266a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc260072df0 con 0x7fc26015bf20 2024-03-04T09:28:52.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:52.420+0000 7fc266a00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2600cdab0 con 0x7fc260072520 2024-03-04T09:28:52.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:52.420+0000 7fc266a00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2600b4d70 con 0x7fc260071e20 2024-03-04T09:28:52.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:52.420+0000 7fc25f400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc260072520 0x7fc2600cdfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:52.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:52.420+0000 7fc25f400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc260072520 0x7fc2600cdfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49166/0 (socket says 172.21.15.6:49166) 2024-03-04T09:28:52.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:52.420+0000 7fc25fe00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fc26015bf20 0x7fc2600acd40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:51704/0 (socket says 172.21.15.6:51704) 2024-03-04T09:28:52.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:52.420+0000 7fc25f400640 1 -- 172.21.15.6:0/648968256 learned_addr learned my addr 172.21.15.6:0/648968256 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:28:52.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:52.420+0000 7fc264a00640 1 --2- 172.21.15.6:0/648968256 >> v2:172.21.15.146:3300/0 conn(0x7fc260071e20 0x7fc2600ce4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:52.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:52.421+0000 7fc25f400640 1 -- 172.21.15.6:0/648968256 >> v2:172.21.15.146:3300/0 conn(0x7fc260071e20 msgr2=0x7fc2600ce4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:52.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:52.421+0000 7fc25f400640 1 --2- 172.21.15.6:0/648968256 >> v2:172.21.15.146:3300/0 conn(0x7fc260071e20 0x7fc2600ce4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:52.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:52.421+0000 7fc25f400640 1 -- 172.21.15.6:0/648968256 >> v1:172.21.15.6:6789/0 conn(0x7fc26015bf20 legacy=0x7fc2600acd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:52.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:52.421+0000 7fc25ea00640 1 -- 172.21.15.6:0/648968256 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1947375047 0 0) 0x7fc260072df0 con 0x7fc26015bf20 2024-03-04T09:28:52.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:52.421+0000 7fc25f400640 1 -- 172.21.15.6:0/648968256 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2600b5050 con 0x7fc260072520 2024-03-04T09:28:52.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:52.422+0000 7fc25f400640 1 --2- 172.21.15.6:0/648968256 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc260072520 0x7fc2600cdfb0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7fc2600af200 tx=0x7fc24c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8733da141d979c02 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:52.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:52.422+0000 7fc25ea00640 1 -- 172.21.15.6:0/648968256 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc24c034070 con 0x7fc260072520 2024-03-04T09:28:52.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:52.422+0000 7fc25ea00640 1 -- 172.21.15.6:0/648968256 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc24c030860 con 0x7fc260072520 2024-03-04T09:28:52.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:52.422+0000 7fc25ea00640 1 -- 172.21.15.6:0/648968256 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc24c030c30 con 0x7fc260072520 2024-03-04T09:28:52.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:52.423+0000 7fc266a00640 1 -- 172.21.15.6:0/648968256 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc260072520 msgr2=0x7fc2600cdfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:52.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:52.423+0000 7fc266a00640 1 --2- 172.21.15.6:0/648968256 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc260072520 0x7fc2600cdfb0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7fc2600af200 tx=0x7fc24c02b9a0 comp rx=0 tx=0).stop 2024-03-04T09:28:52.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:52.423+0000 7fc266a00640 1 -- 172.21.15.6:0/648968256 shutdown_connections 2024-03-04T09:28:52.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:52.423+0000 7fc266a00640 1 --2- 172.21.15.6:0/648968256 >> v2:172.21.15.146:3300/0 conn(0x7fc260071e20 0x7fc2600ce4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:52.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:52.423+0000 7fc266a00640 1 --2- 172.21.15.6:0/648968256 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc260072520 0x7fc2600cdfb0 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:52.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:52.423+0000 7fc266a00640 1 -- 172.21.15.6:0/648968256 >> 172.21.15.6:0/648968256 conn(0x7fc26006b3d0 msgr2=0x7fc26006b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:52.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:52.423+0000 7fc266a00640 1 -- 172.21.15.6:0/648968256 shutdown_connections 2024-03-04T09:28:52.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:52.423+0000 7fc266a00640 1 -- 172.21.15.6:0/648968256 wait complete. 2024-03-04T09:28:52.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:52.425+0000 7fc266a00640 1 Processor -- start 2024-03-04T09:28:52.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:52.425+0000 7fc266a00640 1 -- start start 2024-03-04T09:28:52.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:52.426+0000 7fc266a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc260071e20 0x7fc2601f4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:52.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:52.426+0000 7fc266a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc260072520 0x7fc26014a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:52.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:52.426+0000 7fc266a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc26015bf20 0x7fc2600b2a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:52.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:52.426+0000 7fc266a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2600ad450 con 0x7fc260072520 2024-03-04T09:28:52.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:52.426+0000 7fc266a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2600b4990 con 0x7fc260071e20 2024-03-04T09:28:52.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:52.426+0000 7fc266a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2601f57f0 con 0x7fc26015bf20 2024-03-04T09:28:52.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:52.426+0000 7fc25f400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc260072520 0x7fc26014a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:52.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:52.426+0000 7fc25f400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc260072520 0x7fc26014a530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49168/0 (socket says 172.21.15.6:49168) 2024-03-04T09:28:52.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:52.426+0000 7fc25f400640 1 -- 172.21.15.6:0/1364933891 learned_addr learned my addr 172.21.15.6:0/1364933891 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:28:52.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:52.426+0000 7fc25fe00640 1 --2- 172.21.15.6:0/1364933891 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc260071e20 0x7fc2601f4fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:52.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:52.426+0000 7fc264a00640 1 --2- 172.21.15.6:0/1364933891 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc26015bf20 0x7fc2600b2a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:52.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:52.427+0000 7fc25f400640 1 -- 172.21.15.6:0/1364933891 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc26015bf20 msgr2=0x7fc2600b2a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:52.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:52.427+0000 7fc25f400640 1 --2- 172.21.15.6:0/1364933891 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc26015bf20 0x7fc2600b2a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:52.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:52.427+0000 7fc25f400640 1 -- 172.21.15.6:0/1364933891 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc260071e20 msgr2=0x7fc2601f4fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:52.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:52.427+0000 7fc25f400640 1 --2- 172.21.15.6:0/1364933891 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc260071e20 0x7fc2601f4fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:52.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:52.427+0000 7fc25f400640 1 -- 172.21.15.6:0/1364933891 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2601f5ad0 con 0x7fc260072520 2024-03-04T09:28:52.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:52.427+0000 7fc25f400640 1 --2- 172.21.15.6:0/1364933891 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc260072520 0x7fc26014a530 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7fc24c0090b0 tx=0x7fc24c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:52.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:52.427+0000 7fc257e00640 1 -- 172.21.15.6:0/1364933891 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc24c034040 con 0x7fc260072520 2024-03-04T09:28:52.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:52.427+0000 7fc257e00640 1 -- 172.21.15.6:0/1364933891 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc24c03ab90 con 0x7fc260072520 2024-03-04T09:28:52.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:52.427+0000 7fc266a00640 1 -- 172.21.15.6:0/1364933891 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2601f5db0 con 0x7fc260072520 2024-03-04T09:28:52.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:52.427+0000 7fc257e00640 1 -- 172.21.15.6:0/1364933891 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc24c004030 con 0x7fc260072520 2024-03-04T09:28:52.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:52.428+0000 7fc266a00640 1 -- 172.21.15.6:0/1364933891 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2601f54e0 con 0x7fc260072520 2024-03-04T09:28:52.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:52.429+0000 7fc266a00640 1 -- 172.21.15.6:0/1364933891 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2600af750 con 0x7fc260072520 2024-03-04T09:28:52.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:52.430+0000 7fc257e00640 1 -- 172.21.15.6:0/1364933891 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc24c037070 con 0x7fc260072520 2024-03-04T09:28:52.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:52.430+0000 7fc257e00640 1 --2- 172.21.15.6:0/1364933891 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc24006b4d0 0x7fc24006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:52.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:52.431+0000 7fc257e00640 1 -- 172.21.15.6:0/1364933891 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc24c0ac340 con 0x7fc260072520 2024-03-04T09:28:52.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:52.431+0000 7fc25fe00640 1 --2- 172.21.15.6:0/1364933891 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc24006b4d0 0x7fc24006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:52.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:52.432+0000 7fc25fe00640 1 --2- 172.21.15.6:0/1364933891 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc24006b4d0 0x7fc24006d990 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fc250000f60 tx=0x7fc250007c00 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:52.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:52.435+0000 7fc257e00640 1 -- 172.21.15.6:0/1364933891 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc24c03f070 con 0x7fc260072520 2024-03-04T09:28:52.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:52.671+0000 7fc266a00640 1 -- 172.21.15.6:0/1364933891 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fc26004f330 con 0x7fc260072520 2024-03-04T09:28:53.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:53.281+0000 7fc257e00640 1 -- 172.21.15.6:0/1364933891 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v41) v1 ==== 118+0+0 (secure 0 0 0) 0x7fc24c07b240 con 0x7fc260072520 2024-03-04T09:28:53.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:53.284+0000 7fc266a00640 1 -- 172.21.15.6:0/1364933891 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc24006b4d0 msgr2=0x7fc24006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:53.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:53.284+0000 7fc266a00640 1 --2- 172.21.15.6:0/1364933891 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc24006b4d0 0x7fc24006d990 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fc250000f60 tx=0x7fc250007c00 comp rx=0 tx=0).stop 2024-03-04T09:28:53.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:53.284+0000 7fc266a00640 1 -- 172.21.15.6:0/1364933891 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc260072520 msgr2=0x7fc26014a530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:53.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:53.284+0000 7fc266a00640 1 --2- 172.21.15.6:0/1364933891 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc260072520 0x7fc26014a530 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7fc24c0090b0 tx=0x7fc24c0046c0 comp rx=0 tx=0).stop 2024-03-04T09:28:53.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:53.285+0000 7fc266a00640 1 -- 172.21.15.6:0/1364933891 shutdown_connections 2024-03-04T09:28:53.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:53.285+0000 7fc266a00640 1 --2- 172.21.15.6:0/1364933891 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc24006b4d0 0x7fc24006d990 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:53.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:53.285+0000 7fc266a00640 1 --2- 172.21.15.6:0/1364933891 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc26015bf20 0x7fc2600b2a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:53.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:53.285+0000 7fc266a00640 1 --2- 172.21.15.6:0/1364933891 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc260072520 0x7fc26014a530 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:53.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:53.285+0000 7fc266a00640 1 --2- 172.21.15.6:0/1364933891 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc260071e20 0x7fc2601f4fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:53.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:53.285+0000 7fc266a00640 1 -- 172.21.15.6:0/1364933891 >> 172.21.15.6:0/1364933891 conn(0x7fc26006b3d0 msgr2=0x7fc26015a520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:53.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:53.285+0000 7fc266a00640 1 -- 172.21.15.6:0/1364933891 shutdown_connections 2024-03-04T09:28:53.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:53.285+0000 7fc266a00640 1 -- 172.21.15.6:0/1364933891 wait complete. 2024-03-04T09:28:53.309 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-04T09:28:53.310 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-04T09:28:53.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:53.540+0000 7fb542600640 1 Processor -- start 2024-03-04T09:28:53.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:53.540+0000 7fb542600640 1 -- start start 2024-03-04T09:28:53.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:53.542+0000 7fb542600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fb53c072520 0x7fb53c0cdfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:53.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:53.542+0000 7fb542600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb53c071e20 0x7fb53c0ce4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:53.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:53.542+0000 7fb542600640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb53c072df0 con 0x7fb53c15bf20 2024-03-04T09:28:53.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:53.542+0000 7fb542600640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb53c0cdab0 con 0x7fb53c072520 2024-03-04T09:28:53.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:53.542+0000 7fb542600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb53c0b4d70 con 0x7fb53c071e20 2024-03-04T09:28:53.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:53.542+0000 7fb53aa00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fb53c072520 0x7fb53c0cdfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:53.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:53.542+0000 7fb53b400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fb53c15bf20 0x7fb53c0acd40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:51720/0 (socket says 172.21.15.6:51720) 2024-03-04T09:28:53.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:53.542+0000 7fb53b400640 1 -- 172.21.15.6:0/3008146907 learned_addr learned my addr 172.21.15.6:0/3008146907 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:28:53.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:53.542+0000 7fb53be00640 1 --2- 172.21.15.6:0/3008146907 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb53c071e20 0x7fb53c0ce4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:53.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:53.542+0000 7fb53aa00640 1 -- 172.21.15.6:0/3008146907 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb53c071e20 msgr2=0x7fb53c0ce4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:53.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:53.542+0000 7fb53aa00640 1 --2- 172.21.15.6:0/3008146907 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb53c071e20 0x7fb53c0ce4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:53.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:53.542+0000 7fb53aa00640 1 -- 172.21.15.6:0/3008146907 >> v1:172.21.15.6:6789/0 conn(0x7fb53c15bf20 legacy=0x7fb53c0acd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:53.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:53.542+0000 7fb53aa00640 1 -- 172.21.15.6:0/3008146907 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb53c0b5050 con 0x7fb53c072520 2024-03-04T09:28:53.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:53.542+0000 7fb53be00640 1 --2- 172.21.15.6:0/3008146907 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb53c071e20 0x7fb53c0ce4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:28:53.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:53.543+0000 7fb53aa00640 1 --2- 172.21.15.6:0/3008146907 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb53c072520 0x7fb53c0cdfb0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7fb5280097a0 tx=0x7fb52802b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bdc012d9a32dd0d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:53.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:53.543+0000 7fb53a000640 1 -- 172.21.15.6:0/3008146907 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb528035020 con 0x7fb53c072520 2024-03-04T09:28:53.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:53.543+0000 7fb53a000640 1 -- 172.21.15.6:0/3008146907 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb528031810 con 0x7fb53c072520 2024-03-04T09:28:53.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:53.543+0000 7fb53a000640 1 -- 172.21.15.6:0/3008146907 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb528031be0 con 0x7fb53c072520 2024-03-04T09:28:53.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:53.544+0000 7fb542600640 1 -- 172.21.15.6:0/3008146907 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb53c072520 msgr2=0x7fb53c0cdfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:53.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:53.544+0000 7fb542600640 1 --2- 172.21.15.6:0/3008146907 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb53c072520 0x7fb53c0cdfb0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7fb5280097a0 tx=0x7fb52802b8b0 comp rx=0 tx=0).stop 2024-03-04T09:28:53.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:53.544+0000 7fb542600640 1 -- 172.21.15.6:0/3008146907 shutdown_connections 2024-03-04T09:28:53.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:53.544+0000 7fb542600640 1 --2- 172.21.15.6:0/3008146907 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb53c071e20 0x7fb53c0ce4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:53.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:53.544+0000 7fb542600640 1 --2- 172.21.15.6:0/3008146907 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb53c072520 0x7fb53c0cdfb0 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:53.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:53.544+0000 7fb542600640 1 -- 172.21.15.6:0/3008146907 >> 172.21.15.6:0/3008146907 conn(0x7fb53c06b3d0 msgr2=0x7fb53c06b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:53.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:53.544+0000 7fb542600640 1 -- 172.21.15.6:0/3008146907 shutdown_connections 2024-03-04T09:28:53.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:53.544+0000 7fb542600640 1 -- 172.21.15.6:0/3008146907 wait complete. 2024-03-04T09:28:53.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:53.546+0000 7fb542600640 1 Processor -- start 2024-03-04T09:28:53.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:53.546+0000 7fb542600640 1 -- start start 2024-03-04T09:28:53.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:53.547+0000 7fb542600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb53c071e20 0x7fb53c09cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:53.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:53.547+0000 7fb542600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb53c072520 0x7fb53c1484f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:53.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:53.547+0000 7fb542600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb53c15bf20 0x7fb53c0cb5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:53.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:53.547+0000 7fb542600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb53c0ad450 con 0x7fb53c15bf20 2024-03-04T09:28:53.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:53.547+0000 7fb542600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb53c0b4990 con 0x7fb53c072520 2024-03-04T09:28:53.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:53.547+0000 7fb542600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb53c148a30 con 0x7fb53c071e20 2024-03-04T09:28:53.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:53.547+0000 7fb53be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb53c15bf20 0x7fb53c0cb5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:53.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:53.547+0000 7fb53be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb53c15bf20 0x7fb53c0cb5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49192/0 (socket says 172.21.15.6:49192) 2024-03-04T09:28:53.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:53.547+0000 7fb53be00640 1 -- 172.21.15.6:0/817854038 learned_addr learned my addr 172.21.15.6:0/817854038 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:28:53.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:53.548+0000 7fb53b400640 1 --2- 172.21.15.6:0/817854038 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb53c071e20 0x7fb53c09cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:53.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:53.548+0000 7fb53aa00640 1 --2- 172.21.15.6:0/817854038 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb53c072520 0x7fb53c1484f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:53.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:53.548+0000 7fb53be00640 1 -- 172.21.15.6:0/817854038 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb53c071e20 msgr2=0x7fb53c09cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:53.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:53.548+0000 7fb53be00640 1 --2- 172.21.15.6:0/817854038 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb53c071e20 0x7fb53c09cee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:53.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:53.548+0000 7fb53be00640 1 -- 172.21.15.6:0/817854038 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb53c072520 msgr2=0x7fb53c1484f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:53.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:53.548+0000 7fb53be00640 1 --2- 172.21.15.6:0/817854038 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb53c072520 0x7fb53c1484f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:53.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:53.548+0000 7fb53be00640 1 -- 172.21.15.6:0/817854038 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb53c09be40 con 0x7fb53c15bf20 2024-03-04T09:28:53.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:53.548+0000 7fb53be00640 1 --2- 172.21.15.6:0/817854038 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb53c15bf20 0x7fb53c0cb5f0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fb53400cdb0 tx=0x7fb53400a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:53.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:53.548+0000 7fb533400640 1 -- 172.21.15.6:0/817854038 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb534003b90 con 0x7fb53c15bf20 2024-03-04T09:28:53.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:53.548+0000 7fb533400640 1 -- 172.21.15.6:0/817854038 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb534003d50 con 0x7fb53c15bf20 2024-03-04T09:28:53.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:53.549+0000 7fb542600640 1 -- 172.21.15.6:0/817854038 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb53c09c120 con 0x7fb53c15bf20 2024-03-04T09:28:53.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:53.549+0000 7fb533400640 1 -- 172.21.15.6:0/817854038 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb534004260 con 0x7fb53c15bf20 2024-03-04T09:28:53.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:53.549+0000 7fb542600640 1 -- 172.21.15.6:0/817854038 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb53c09c6e0 con 0x7fb53c15bf20 2024-03-04T09:28:53.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:53.550+0000 7fb542600640 1 -- 172.21.15.6:0/817854038 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb53c04f330 con 0x7fb53c15bf20 2024-03-04T09:28:53.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:53.551+0000 7fb533400640 1 -- 172.21.15.6:0/817854038 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb534004400 con 0x7fb53c15bf20 2024-03-04T09:28:53.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:53.552+0000 7fb533400640 1 --2- 172.21.15.6:0/817854038 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb51c06b4d0 0x7fb51c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:53.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:53.552+0000 7fb533400640 1 -- 172.21.15.6:0/817854038 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb53408ac40 con 0x7fb53c15bf20 2024-03-04T09:28:53.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:53.553+0000 7fb53b400640 1 --2- 172.21.15.6:0/817854038 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb51c06b4d0 0x7fb51c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:53.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:53.554+0000 7fb53b400640 1 --2- 172.21.15.6:0/817854038 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb51c06b4d0 0x7fb51c06d990 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fb53c06e570 tx=0x7fb52c009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:53.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:53.558+0000 7fb533400640 1 -- 172.21.15.6:0/817854038 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb534014030 con 0x7fb53c15bf20 2024-03-04T09:28:53.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:53.798+0000 7fb542600640 1 -- 172.21.15.6:0/817854038 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fb53c15f720 con 0x7fb53c15bf20 2024-03-04T09:28:54.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:54.285+0000 7fb533400640 1 -- 172.21.15.6:0/817854038 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v42) v1 ==== 122+0+0 (secure 0 0 0) 0x7fb534059ab0 con 0x7fb53c15bf20 2024-03-04T09:28:54.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:54.290+0000 7fb542600640 1 -- 172.21.15.6:0/817854038 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb51c06b4d0 msgr2=0x7fb51c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:54.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:54.290+0000 7fb542600640 1 --2- 172.21.15.6:0/817854038 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb51c06b4d0 0x7fb51c06d990 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fb53c06e570 tx=0x7fb52c009040 comp rx=0 tx=0).stop 2024-03-04T09:28:54.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:54.290+0000 7fb542600640 1 -- 172.21.15.6:0/817854038 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb53c15bf20 msgr2=0x7fb53c0cb5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:54.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:54.290+0000 7fb542600640 1 --2- 172.21.15.6:0/817854038 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb53c15bf20 0x7fb53c0cb5f0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fb53400cdb0 tx=0x7fb53400a650 comp rx=0 tx=0).stop 2024-03-04T09:28:54.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:54.290+0000 7fb542600640 1 -- 172.21.15.6:0/817854038 shutdown_connections 2024-03-04T09:28:54.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:54.290+0000 7fb542600640 1 --2- 172.21.15.6:0/817854038 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb53c15bf20 0x7fb53c0cb5f0 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:54.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:54.290+0000 7fb542600640 1 --2- 172.21.15.6:0/817854038 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb53c072520 0x7fb53c1484f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:54.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:54.290+0000 7fb542600640 1 --2- 172.21.15.6:0/817854038 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb51c06b4d0 0x7fb51c06d990 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:54.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:54.290+0000 7fb542600640 1 --2- 172.21.15.6:0/817854038 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb53c071e20 0x7fb53c09cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:54.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:54.290+0000 7fb542600640 1 -- 172.21.15.6:0/817854038 >> 172.21.15.6:0/817854038 conn(0x7fb53c06b3d0 msgr2=0x7fb53c0ae770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:54.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:54.290+0000 7fb542600640 1 -- 172.21.15.6:0/817854038 shutdown_connections 2024-03-04T09:28:54.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:54.290+0000 7fb542600640 1 -- 172.21.15.6:0/817854038 wait complete. 2024-03-04T09:28:54.310 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-04T09:28:54.310 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:28:54.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:54.529+0000 7fb2e3e00640 1 Processor -- start 2024-03-04T09:28:54.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:54.529+0000 7fb2e3e00640 1 -- start start 2024-03-04T09:28:54.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:54.531+0000 7fb2e3e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fb2dc0b34e0 0x7fb2dc15c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:54.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:54.531+0000 7fb2e3e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2dc15cc40 con 0x7fb2dc0af1a0 2024-03-04T09:28:54.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:54.531+0000 7fb2e3e00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2dc15cf20 con 0x7fb2dc0af580 2024-03-04T09:28:54.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:54.531+0000 7fb2e3e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2dc15d200 con 0x7fb2dc0b34e0 2024-03-04T09:28:54.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:54.531+0000 7fb2e1e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fb2dc0b34e0 0x7fb2dc15c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:54.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:54.531+0000 7fb2e1400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fb2dc0af1a0 0x7fb2dc0b4c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:51728/0 (socket says 172.21.15.6:51728) 2024-03-04T09:28:54.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:54.531+0000 7fb2e1400640 1 -- 172.21.15.6:0/1392509622 learned_addr learned my addr 172.21.15.6:0/1392509622 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:28:54.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:54.532+0000 7fb2e1e00640 1 -- 172.21.15.6:0/1392509622 >> v1:172.21.15.146:6789/0 conn(0x7fb2dc0af580 legacy=0x7fb2dc0b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:54.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:54.532+0000 7fb2e1e00640 1 -- 172.21.15.6:0/1392509622 >> v1:172.21.15.6:6789/0 conn(0x7fb2dc0af1a0 legacy=0x7fb2dc0b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:54.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:54.532+0000 7fb2e1e00640 1 -- 172.21.15.6:0/1392509622 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2dc15d4e0 con 0x7fb2dc0b34e0 2024-03-04T09:28:54.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:54.532+0000 7fb2dbe00640 1 -- 172.21.15.6:0/1392509622 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3109556763 0 0) 0x7fb2dc15cc40 con 0x7fb2dc0af1a0 2024-03-04T09:28:54.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:54.533+0000 7fb2e1e00640 1 --2- 172.21.15.6:0/1392509622 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb2dc0b34e0 0x7fb2dc15c600 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fb2d4009d90 tx=0x7fb2d402beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8a630884ae4a86 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:54.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:54.533+0000 7fb2dbe00640 1 -- 172.21.15.6:0/1392509622 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2d4039070 con 0x7fb2dc0b34e0 2024-03-04T09:28:54.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:54.533+0000 7fb2dbe00640 1 -- 172.21.15.6:0/1392509622 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2d402cc60 con 0x7fb2dc0b34e0 2024-03-04T09:28:54.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:54.533+0000 7fb2dbe00640 1 -- 172.21.15.6:0/1392509622 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2d4034660 con 0x7fb2dc0b34e0 2024-03-04T09:28:54.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:54.534+0000 7fb2e3e00640 1 -- 172.21.15.6:0/1392509622 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb2dc0b34e0 msgr2=0x7fb2dc15c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:54.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:54.534+0000 7fb2e3e00640 1 --2- 172.21.15.6:0/1392509622 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb2dc0b34e0 0x7fb2dc15c600 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fb2d4009d90 tx=0x7fb2d402beb0 comp rx=0 tx=0).stop 2024-03-04T09:28:54.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:54.534+0000 7fb2e3e00640 1 -- 172.21.15.6:0/1392509622 shutdown_connections 2024-03-04T09:28:54.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:54.534+0000 7fb2e3e00640 1 --2- 172.21.15.6:0/1392509622 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb2dc0b34e0 0x7fb2dc15c600 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:54.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:54.534+0000 7fb2e3e00640 1 -- 172.21.15.6:0/1392509622 >> 172.21.15.6:0/1392509622 conn(0x7fb2dc06afb0 msgr2=0x7fb2dc06b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:54.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:54.534+0000 7fb2e3e00640 1 -- 172.21.15.6:0/1392509622 shutdown_connections 2024-03-04T09:28:54.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:54.534+0000 7fb2e3e00640 1 -- 172.21.15.6:0/1392509622 wait complete. 2024-03-04T09:28:54.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:54.536+0000 7fb2e3e00640 1 Processor -- start 2024-03-04T09:28:54.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:54.536+0000 7fb2e3e00640 1 -- start start 2024-03-04T09:28:54.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:54.537+0000 7fb2e3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb2dc0af1a0 0x7fb2dc153490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:54.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:54.537+0000 7fb2e3e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb2dc0af580 0x7fb2dc06cb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:54.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:54.537+0000 7fb2e3e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb2dc0b34e0 0x7fb2dc09aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:54.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:54.537+0000 7fb2e3e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2dc155e70 con 0x7fb2dc0af1a0 2024-03-04T09:28:54.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:54.537+0000 7fb2e3e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2dc0cde30 con 0x7fb2dc0b34e0 2024-03-04T09:28:54.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:54.537+0000 7fb2e3e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2dc155ff0 con 0x7fb2dc0af580 2024-03-04T09:28:54.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:54.537+0000 7fb2e1400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb2dc0af1a0 0x7fb2dc153490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:54.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:54.538+0000 7fb2e1400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb2dc0af1a0 0x7fb2dc153490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49200/0 (socket says 172.21.15.6:49200) 2024-03-04T09:28:54.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:54.538+0000 7fb2e1400640 1 -- 172.21.15.6:0/599868186 learned_addr learned my addr 172.21.15.6:0/599868186 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:28:54.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:54.538+0000 7fb2e1e00640 1 --2- 172.21.15.6:0/599868186 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb2dc0b34e0 0x7fb2dc09aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:54.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:54.538+0000 7fb2e0a00640 1 --2- 172.21.15.6:0/599868186 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb2dc0af580 0x7fb2dc06cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:54.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:54.538+0000 7fb2e1400640 1 -- 172.21.15.6:0/599868186 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb2dc0af580 msgr2=0x7fb2dc06cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:54.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:54.538+0000 7fb2e1400640 1 --2- 172.21.15.6:0/599868186 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb2dc0af580 0x7fb2dc06cb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:54.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:54.538+0000 7fb2e1400640 1 -- 172.21.15.6:0/599868186 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb2dc0b34e0 msgr2=0x7fb2dc09aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:54.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:54.538+0000 7fb2e1400640 1 --2- 172.21.15.6:0/599868186 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb2dc0b34e0 0x7fb2dc09aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:54.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:54.538+0000 7fb2e1400640 1 -- 172.21.15.6:0/599868186 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2dc1539d0 con 0x7fb2dc0af1a0 2024-03-04T09:28:54.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:54.538+0000 7fb2e1400640 1 --2- 172.21.15.6:0/599868186 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb2dc0af1a0 0x7fb2dc153490 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7fb2d000c4a0 tx=0x7fb2d000c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:54.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:54.539+0000 7fb2d9600640 1 -- 172.21.15.6:0/599868186 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2d0018070 con 0x7fb2dc0af1a0 2024-03-04T09:28:54.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:54.539+0000 7fb2d9600640 1 -- 172.21.15.6:0/599868186 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2d00074e0 con 0x7fb2dc0af1a0 2024-03-04T09:28:54.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:54.539+0000 7fb2e3e00640 1 -- 172.21.15.6:0/599868186 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2dc15d720 con 0x7fb2dc0af1a0 2024-03-04T09:28:54.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:54.539+0000 7fb2d9600640 1 -- 172.21.15.6:0/599868186 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2d0004a20 con 0x7fb2dc0af1a0 2024-03-04T09:28:54.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:54.539+0000 7fb2e3e00640 1 -- 172.21.15.6:0/599868186 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2dc15dce0 con 0x7fb2dc0af1a0 2024-03-04T09:28:54.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:54.540+0000 7fb2e3e00640 1 -- 172.21.15.6:0/599868186 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2dc069510 con 0x7fb2dc0af1a0 2024-03-04T09:28:54.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:54.541+0000 7fb2d9600640 1 -- 172.21.15.6:0/599868186 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb2d001b070 con 0x7fb2dc0af1a0 2024-03-04T09:28:54.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:54.542+0000 7fb2d9600640 1 --2- 172.21.15.6:0/599868186 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb2bc06b4d0 0x7fb2bc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:54.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:54.542+0000 7fb2d9600640 1 -- 172.21.15.6:0/599868186 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb2d0089580 con 0x7fb2dc0af1a0 2024-03-04T09:28:54.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:54.543+0000 7fb2e0a00640 1 --2- 172.21.15.6:0/599868186 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb2bc06b4d0 0x7fb2bc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:54.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:54.544+0000 7fb2e0a00640 1 --2- 172.21.15.6:0/599868186 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb2bc06b4d0 0x7fb2bc06d990 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fb2dc0cf3e0 tx=0x7fb2c8009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:54.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:54.547+0000 7fb2d9600640 1 -- 172.21.15.6:0/599868186 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb2d0058360 con 0x7fb2dc0af1a0 2024-03-04T09:28:54.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:54.787+0000 7fb2e3e00640 1 -- 172.21.15.6:0/599868186 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb2dc06d050 con 0x7fb2dc0af1a0 2024-03-04T09:28:54.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:54.788+0000 7fb2d9600640 1 -- 172.21.15.6:0/599868186 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+4979 (secure 0 0 0) 0x7fb2d0057c60 con 0x7fb2dc0af1a0 2024-03-04T09:28:54.785 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:28:54.785 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":42,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5228,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/1891254736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1891254736},{"type":"v1","addr":"172.21.15.6:6834","nonce":1891254736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":5578,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/1781219843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":1781219843},{"type":"v1","addr":"172.21.15.146:6835","nonce":1781219843}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":5587,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/1479180802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":1479180802},{"type":"v1","addr":"172.21.15.6:6835","nonce":1479180802}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:28:50.708467+0000","modified":"2024-03-04T09:28:53.801285+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5590},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5590":{"gid":5590,"name":"d","rank":0,"incarnation":38,"state":"up:active","state_seq":3,"addr":"172.21.15.146:6837/3919939129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":3919939129},{"type":"v1","addr":"172.21.15.146:6837","nonce":3919939129}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5590,"qdb_cluster":[5590]},"id":5}]} 2024-03-04T09:28:54.785 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 42 2024-03-04T09:28:54.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:54.791+0000 7fb2e3e00640 1 -- 172.21.15.6:0/599868186 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb2bc06b4d0 msgr2=0x7fb2bc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:54.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:54.791+0000 7fb2e3e00640 1 --2- 172.21.15.6:0/599868186 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb2bc06b4d0 0x7fb2bc06d990 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fb2dc0cf3e0 tx=0x7fb2c8009040 comp rx=0 tx=0).stop 2024-03-04T09:28:54.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:54.791+0000 7fb2e3e00640 1 -- 172.21.15.6:0/599868186 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb2dc0af1a0 msgr2=0x7fb2dc153490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:54.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:54.791+0000 7fb2e3e00640 1 --2- 172.21.15.6:0/599868186 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb2dc0af1a0 0x7fb2dc153490 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7fb2d000c4a0 tx=0x7fb2d000c960 comp rx=0 tx=0).stop 2024-03-04T09:28:54.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:54.791+0000 7fb2e3e00640 1 -- 172.21.15.6:0/599868186 shutdown_connections 2024-03-04T09:28:54.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:54.791+0000 7fb2e3e00640 1 --2- 172.21.15.6:0/599868186 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb2dc0b34e0 0x7fb2dc09aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:54.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:54.791+0000 7fb2e3e00640 1 --2- 172.21.15.6:0/599868186 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb2bc06b4d0 0x7fb2bc06d990 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:54.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:54.791+0000 7fb2e3e00640 1 --2- 172.21.15.6:0/599868186 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb2dc0af580 0x7fb2dc06cb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:54.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:54.791+0000 7fb2e3e00640 1 --2- 172.21.15.6:0/599868186 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb2dc0af1a0 0x7fb2dc153490 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:54.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:54.791+0000 7fb2e3e00640 1 -- 172.21.15.6:0/599868186 >> 172.21.15.6:0/599868186 conn(0x7fb2dc06afb0 msgr2=0x7fb2dc155660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:54.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:54.791+0000 7fb2e3e00640 1 -- 172.21.15.6:0/599868186 shutdown_connections 2024-03-04T09:28:54.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:54.791+0000 7fb2e3e00640 1 -- 172.21.15.6:0/599868186 wait complete. 2024-03-04T09:28:54.809 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T09:28:55.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.033+0000 7f691f400640 1 Processor -- start 2024-03-04T09:28:55.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.033+0000 7f691f400640 1 -- start start 2024-03-04T09:28:55.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.035+0000 7f691f400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f69180af580 0x7f691815c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:55.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.035+0000 7f691f400640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f691815cc40 con 0x7f69180af1a0 2024-03-04T09:28:55.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.035+0000 7f691f400640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f691815cf20 con 0x7f69180cdc20 2024-03-04T09:28:55.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.035+0000 7f691f400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f691815d200 con 0x7f69180af580 2024-03-04T09:28:55.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.035+0000 7f691ca00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f69180af1a0 0x7f69180b4c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:51744/0 (socket says 172.21.15.6:51744) 2024-03-04T09:28:55.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.035+0000 7f691ca00640 1 -- 172.21.15.6:0/59990339 learned_addr learned my addr 172.21.15.6:0/59990339 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:28:55.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.036+0000 7f6917400640 1 -- 172.21.15.6:0/59990339 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3536655015 0 0) 0x7f691815cc40 con 0x7f69180af1a0 2024-03-04T09:28:55.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.036+0000 7f6917400640 1 -- 172.21.15.6:0/59990339 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6900003520 con 0x7f69180af1a0 2024-03-04T09:28:55.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.036+0000 7f6917e00640 1 --2- 172.21.15.6:0/59990339 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f69180af580 0x7f691815c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:55.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.036+0000 7f6917400640 1 -- 172.21.15.6:0/59990339 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 671543516 0 0) 0x7f6900003520 con 0x7f69180af1a0 2024-03-04T09:28:55.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.036+0000 7f6917400640 1 -- 172.21.15.6:0/59990339 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f69180af580 msgr2=0x7f691815c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:55.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.036+0000 7f6917400640 1 --2- 172.21.15.6:0/59990339 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f69180af580 0x7f691815c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:55.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.036+0000 7f6917400640 1 -- 172.21.15.6:0/59990339 >> v1:172.21.15.146:6789/0 conn(0x7f69180cdc20 legacy=0x7f69180b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:55.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.036+0000 7f6917400640 1 -- 172.21.15.6:0/59990339 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f691815d4e0 con 0x7f69180af1a0 2024-03-04T09:28:55.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.036+0000 7f6917400640 1 -- 172.21.15.6:0/59990339 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7f690c001670 con 0x7f69180af1a0 2024-03-04T09:28:55.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.036+0000 7f6917e00640 1 --2- 172.21.15.6:0/59990339 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f69180af580 0x7f691815c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T09:28:55.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.036+0000 7f6917400640 1 -- 172.21.15.6:0/59990339 >> v1:172.21.15.6:6789/0 conn(0x7f69180af1a0 legacy=0x7f69180b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:55.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.036+0000 7f6917400640 1 --2- 172.21.15.6:0/59990339 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6900003a50 0x7f6900003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:55.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.036+0000 7f6917400640 1 --2- 172.21.15.6:0/59990339 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6900004800 0x7f6900004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:55.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.036+0000 7f6917400640 1 --2- 172.21.15.6:0/59990339 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f69180af580 0x7f690000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:55.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.036+0000 7f6917400640 1 -- 172.21.15.6:0/59990339 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f691815d200 con 0x7f6900004800 2024-03-04T09:28:55.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.036+0000 7f6917400640 1 -- 172.21.15.6:0/59990339 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6900005500 con 0x7f6900003a50 2024-03-04T09:28:55.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.036+0000 7f6917400640 1 -- 172.21.15.6:0/59990339 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f690000bf00 con 0x7f69180af580 2024-03-04T09:28:55.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.036+0000 7f6917e00640 1 --2- 172.21.15.6:0/59990339 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6900004800 0x7f6900004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:55.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.037+0000 7f6917e00640 1 -- 172.21.15.6:0/59990339 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f69180af580 msgr2=0x7f690000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:55.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.037+0000 7f6917e00640 1 --2- 172.21.15.6:0/59990339 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f69180af580 0x7f690000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:55.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.037+0000 7f6917e00640 1 -- 172.21.15.6:0/59990339 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6900003a50 msgr2=0x7f6900003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:55.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.037+0000 7f6917e00640 1 --2- 172.21.15.6:0/59990339 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6900003a50 0x7f6900003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:55.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.037+0000 7f6917e00640 1 -- 172.21.15.6:0/59990339 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6900003520 con 0x7f6900004800 2024-03-04T09:28:55.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.038+0000 7f6917e00640 1 --2- 172.21.15.6:0/59990339 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6900004800 0x7f6900004c80 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f69040098a0 tx=0x7f6900005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2fbf2bbea0d440 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:55.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.038+0000 7f6917400640 1 -- 172.21.15.6:0/59990339 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f690400f040 con 0x7f6900004800 2024-03-04T09:28:55.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.038+0000 7f6917400640 1 -- 172.21.15.6:0/59990339 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6904004230 con 0x7f6900004800 2024-03-04T09:28:55.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.038+0000 7f6917400640 1 -- 172.21.15.6:0/59990339 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6904003610 con 0x7f6900004800 2024-03-04T09:28:55.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.039+0000 7f691f400640 1 -- 172.21.15.6:0/59990339 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6900004800 msgr2=0x7f6900004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:55.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.039+0000 7f691f400640 1 --2- 172.21.15.6:0/59990339 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6900004800 0x7f6900004c80 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f69040098a0 tx=0x7f6900005830 comp rx=0 tx=0).stop 2024-03-04T09:28:55.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.039+0000 7f691d400640 1 -- 172.21.15.6:0/59990339 reap_dead start 2024-03-04T09:28:55.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.039+0000 7f691f400640 1 -- 172.21.15.6:0/59990339 shutdown_connections 2024-03-04T09:28:55.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.039+0000 7f691f400640 1 -- 172.21.15.6:0/59990339 >> 172.21.15.6:0/59990339 conn(0x7f691806afb0 msgr2=0x7f691806b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:55.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.039+0000 7f691f400640 1 -- 172.21.15.6:0/59990339 shutdown_connections 2024-03-04T09:28:55.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.039+0000 7f691f400640 1 -- 172.21.15.6:0/59990339 wait complete. 2024-03-04T09:28:55.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.041+0000 7f691f400640 1 Processor -- start 2024-03-04T09:28:55.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.041+0000 7f691f400640 1 -- start start 2024-03-04T09:28:55.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.042+0000 7f691f400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6900004800 0x7f691809aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:55.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.042+0000 7f691f400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f69181f1910 0x7f691809cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:55.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.042+0000 7f691f400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f69181f46f0 0x7f69180ce760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:55.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.042+0000 7f691f400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f691815c600 con 0x7f69181f46f0 2024-03-04T09:28:55.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.042+0000 7f691f400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69180b36f0 con 0x7f6900004800 2024-03-04T09:28:55.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.042+0000 7f691f400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f691815c780 con 0x7f69181f1910 2024-03-04T09:28:55.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.042+0000 7f691d400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f69181f46f0 0x7f69180ce760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:55.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.042+0000 7f691d400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f69181f46f0 0x7f69180ce760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49218/0 (socket says 172.21.15.6:49218) 2024-03-04T09:28:55.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.042+0000 7f691d400640 1 -- 172.21.15.6:0/1349127329 learned_addr learned my addr 172.21.15.6:0/1349127329 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:28:55.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.042+0000 7f691ca00640 1 --2- 172.21.15.6:0/1349127329 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6900004800 0x7f691809aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:55.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.043+0000 7f691d400640 1 -- 172.21.15.6:0/1349127329 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f69181f1910 msgr2=0x7f691809cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:55.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.043+0000 7f6917e00640 1 --2- 172.21.15.6:0/1349127329 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f69181f1910 0x7f691809cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:55.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.043+0000 7f691d400640 1 --2- 172.21.15.6:0/1349127329 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f69181f1910 0x7f691809cbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:55.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.043+0000 7f691d400640 1 -- 172.21.15.6:0/1349127329 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6900004800 msgr2=0x7f691809aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:55.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.043+0000 7f691d400640 1 --2- 172.21.15.6:0/1349127329 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6900004800 0x7f691809aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:55.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.043+0000 7f691d400640 1 -- 172.21.15.6:0/1349127329 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f691809d110 con 0x7f69181f46f0 2024-03-04T09:28:55.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.043+0000 7f691ca00640 1 --2- 172.21.15.6:0/1349127329 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6900004800 0x7f691809aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:28:55.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.043+0000 7f6917e00640 1 --2- 172.21.15.6:0/1349127329 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f69181f1910 0x7f691809cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:28:55.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.043+0000 7f691d400640 1 --2- 172.21.15.6:0/1349127329 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f69181f46f0 0x7f69180ce760 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f691000c700 tx=0x7f691000cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:55.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.043+0000 7f6914c00640 1 -- 172.21.15.6:0/1349127329 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6910005d40 con 0x7f69181f46f0 2024-03-04T09:28:55.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.043+0000 7f6914c00640 1 -- 172.21.15.6:0/1349127329 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f691000a640 con 0x7f69181f46f0 2024-03-04T09:28:55.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.043+0000 7f6914c00640 1 -- 172.21.15.6:0/1349127329 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f691000aa30 con 0x7f69181f46f0 2024-03-04T09:28:55.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.043+0000 7f691f400640 1 -- 172.21.15.6:0/1349127329 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f691809b940 con 0x7f69181f46f0 2024-03-04T09:28:55.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.043+0000 7f691f400640 1 -- 172.21.15.6:0/1349127329 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f691809b380 con 0x7f69181f46f0 2024-03-04T09:28:55.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.045+0000 7f691f400640 1 -- 172.21.15.6:0/1349127329 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f691804f3a0 con 0x7f69181f46f0 2024-03-04T09:28:55.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.046+0000 7f6914c00640 1 -- 172.21.15.6:0/1349127329 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6910011070 con 0x7f69181f46f0 2024-03-04T09:28:55.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.047+0000 7f6914c00640 1 --2- 172.21.15.6:0/1349127329 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f68f806b4d0 0x7f68f806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:55.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.047+0000 7f6914c00640 1 -- 172.21.15.6:0/1349127329 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6910088e20 con 0x7f69181f46f0 2024-03-04T09:28:55.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.047+0000 7f691ca00640 1 --2- 172.21.15.6:0/1349127329 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f68f806b4d0 0x7f68f806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:55.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.049+0000 7f691ca00640 1 --2- 172.21.15.6:0/1349127329 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f68f806b4d0 0x7f68f806d990 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f690c000c00 tx=0x7f690c009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:55.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.052+0000 7f6914c00640 1 -- 172.21.15.6:0/1349127329 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6910057ca0 con 0x7f69181f46f0 2024-03-04T09:28:55.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.257+0000 7f691f400640 1 -- 172.21.15.6:0/1349127329 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f69181f1cf0 con 0x7f69181f46f0 2024-03-04T09:28:55.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.260+0000 7f6914c00640 1 -- 172.21.15.6:0/1349127329 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v68) v1 ==== 74+0+16620 (secure 0 0 0) 0x7f69100575a0 con 0x7f69181f46f0 2024-03-04T09:28:55.257 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:28:55.257 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":68,"fsid":"edce0578-5ae4-4402-b45a-552bdef84d7c","created":"2024-03-04T09:24:54.523780+0000","modified":"2024-03-04T09:28:52.262569+0000","last_up_change":"2024-03-04T09:25:07.089049+0000","last_in_change":"2024-03-04T09:24:57.982077+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T09:25:07.935198+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-03-04T09:28:48.627427+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-03-04T09:28:49.739043+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a96574f-81f1-4bc6-8463-757520b84a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6825","nonce":1364449401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6827","nonce":1364449401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6831","nonce":1364449401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6829","nonce":1364449401}]},"public_addr":"172.21.15.6:6825/1364449401","cluster_addr":"172.21.15.6:6827/1364449401","heartbeat_back_addr":"172.21.15.6:6831/1364449401","heartbeat_front_addr":"172.21.15.6:6829/1364449401","state":["exists","up"]},{"osd":1,"uuid":"7d79b9c1-6b87-476a-bc36-7db5ee64cd90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6809","nonce":2494103078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6811","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6813","nonce":2494103078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6821","nonce":2494103078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6817","nonce":2494103078}]},"public_addr":"172.21.15.6:6809/2494103078","cluster_addr":"172.21.15.6:6813/2494103078","heartbeat_back_addr":"172.21.15.6:6821/2494103078","heartbeat_front_addr":"172.21.15.6:6817/2494103078","state":["exists","up"]},{"osd":2,"uuid":"492331b9-4ee9-40b6-bacd-85d44cd6c45a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6812","nonce":1500066681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6816","nonce":1500066681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6824","nonce":1500066681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6820","nonce":1500066681}]},"public_addr":"172.21.15.6:6812/1500066681","cluster_addr":"172.21.15.6:6816/1500066681","heartbeat_back_addr":"172.21.15.6:6824/1500066681","heartbeat_front_addr":"172.21.15.6:6820/1500066681","state":["exists","up"]},{"osd":3,"uuid":"21bb2efe-9ba7-4bdc-bc31-180cd1102e21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6801","nonce":1558806123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6803","nonce":1558806123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6807","nonce":1558806123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6805","nonce":1558806123}]},"public_addr":"172.21.15.6:6801/1558806123","cluster_addr":"172.21.15.6:6803/1558806123","heartbeat_back_addr":"172.21.15.6:6807/1558806123","heartbeat_front_addr":"172.21.15.6:6805/1558806123","state":["exists","up"]},{"osd":4,"uuid":"6e2c4b71-19d9-4737-83d9-558705d8644a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6801","nonce":1526135780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6803","nonce":1526135780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6807","nonce":1526135780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6805","nonce":1526135780}]},"public_addr":"172.21.15.146:6801/1526135780","cluster_addr":"172.21.15.146:6803/1526135780","heartbeat_back_addr":"172.21.15.146:6807/1526135780","heartbeat_front_addr":"172.21.15.146:6805/1526135780","state":["exists","up"]},{"osd":5,"uuid":"fb69c525-f1be-4b7d-ab32-6c29b0d6fddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6809","nonce":460262507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6811","nonce":460262507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6815","nonce":460262507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6813","nonce":460262507}]},"public_addr":"172.21.15.146:6809/460262507","cluster_addr":"172.21.15.146:6811/460262507","heartbeat_back_addr":"172.21.15.146:6815/460262507","heartbeat_front_addr":"172.21.15.146:6813/460262507","state":["exists","up"]},{"osd":6,"uuid":"c32e7fed-370d-4401-8ce0-6a4335b0d168","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6823","nonce":1392897493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6827","nonce":1392897493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6831","nonce":1392897493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6829","nonce":1392897493}]},"public_addr":"172.21.15.146:6823/1392897493","cluster_addr":"172.21.15.146:6827/1392897493","heartbeat_back_addr":"172.21.15.146:6831/1392897493","heartbeat_front_addr":"172.21.15.146:6829/1392897493","state":["exists","up"]},{"osd":7,"uuid":"75a79b44-fa5f-426d-a7d3-71f21f78835c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6817","nonce":3268323156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6819","nonce":3268323156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6825","nonce":3268323156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6822","nonce":3268323156}]},"public_addr":"172.21.15.146:6817/3268323156","cluster_addr":"172.21.15.146:6819/3268323156","heartbeat_back_addr":"172.21.15.146:6825/3268323156","heartbeat_front_addr":"172.21.15.146:6822/3268323156","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.310437+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.295850+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.294674+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.526855+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.300023+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.595160+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.346046+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.558741+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T09:28:55.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.263+0000 7f691f400640 1 -- 172.21.15.6:0/1349127329 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f68f806b4d0 msgr2=0x7f68f806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:55.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.263+0000 7f691f400640 1 --2- 172.21.15.6:0/1349127329 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f68f806b4d0 0x7f68f806d990 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f690c000c00 tx=0x7f690c009040 comp rx=0 tx=0).stop 2024-03-04T09:28:55.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.263+0000 7f691f400640 1 -- 172.21.15.6:0/1349127329 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f69181f46f0 msgr2=0x7f69180ce760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:55.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.263+0000 7f691f400640 1 --2- 172.21.15.6:0/1349127329 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f69181f46f0 0x7f69180ce760 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f691000c700 tx=0x7f691000cbc0 comp rx=0 tx=0).stop 2024-03-04T09:28:55.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.263+0000 7f691f400640 1 -- 172.21.15.6:0/1349127329 shutdown_connections 2024-03-04T09:28:55.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.263+0000 7f691f400640 1 --2- 172.21.15.6:0/1349127329 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f69181f46f0 0x7f69180ce760 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:55.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.263+0000 7f691f400640 1 --2- 172.21.15.6:0/1349127329 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f68f806b4d0 0x7f68f806d990 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:55.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.263+0000 7f691f400640 1 --2- 172.21.15.6:0/1349127329 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f69181f1910 0x7f691809cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:55.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.263+0000 7f691f400640 1 --2- 172.21.15.6:0/1349127329 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6900004800 0x7f691809aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:55.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.263+0000 7f691f400640 1 -- 172.21.15.6:0/1349127329 >> 172.21.15.6:0/1349127329 conn(0x7f691806afb0 msgr2=0x7f691815be90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:55.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.264+0000 7f691f400640 1 -- 172.21.15.6:0/1349127329 shutdown_connections 2024-03-04T09:28:55.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.264+0000 7f691f400640 1 -- 172.21.15.6:0/1349127329 wait complete. 2024-03-04T09:28:55.281 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-04T09:28:55.281 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T09:28:55.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.507+0000 7fe316e00640 1 Processor -- start 2024-03-04T09:28:55.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.507+0000 7fe316e00640 1 -- start start 2024-03-04T09:28:55.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.508+0000 7fe316e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fe310070f10 0x7fe3100cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:55.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.508+0000 7fe316e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fe3100cce90 0x7fe31015c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:55.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.508+0000 7fe316e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3100714d0 con 0x7fe310145300 2024-03-04T09:28:55.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.508+0000 7fe316e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3100cd300 con 0x7fe3100cce90 2024-03-04T09:28:55.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.508+0000 7fe316e00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3100717b0 con 0x7fe310070f10 2024-03-04T09:28:55.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.509+0000 7fe316400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fe3100cce90 0x7fe31015c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:55.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.509+0000 7fe316400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fe3100cce90 0x7fe31015c570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49232/0 (socket says 172.21.15.6:49232) 2024-03-04T09:28:55.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.509+0000 7fe316400640 1 -- 172.21.15.6:0/2814113834 learned_addr learned my addr 172.21.15.6:0/2814113834 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:28:55.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.509+0000 7fe315a00640 1 --1- 172.21.15.6:0/2814113834 >> v1:172.21.15.6:6789/0 conn(0x7fe310145300 0x7fe3100b49a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:51750/0 (socket says 172.21.15.6:51750) 2024-03-04T09:28:55.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.509+0000 7fe315000640 1 --2- 172.21.15.6:0/2814113834 >> v2:172.21.15.146:3300/0 conn(0x7fe310070f10 0x7fe3100cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:55.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.509+0000 7fe316400640 1 -- 172.21.15.6:0/2814113834 >> v2:172.21.15.146:3300/0 conn(0x7fe310070f10 msgr2=0x7fe3100cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:55.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.509+0000 7fe316400640 1 --2- 172.21.15.6:0/2814113834 >> v2:172.21.15.146:3300/0 conn(0x7fe310070f10 0x7fe3100cc950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:55.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.509+0000 7fe316400640 1 -- 172.21.15.6:0/2814113834 >> v1:172.21.15.6:6789/0 conn(0x7fe310145300 legacy=0x7fe3100b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:55.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.509+0000 7fe316400640 1 -- 172.21.15.6:0/2814113834 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe310071a90 con 0x7fe3100cce90 2024-03-04T09:28:55.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.509+0000 7fe30fe00640 1 -- 172.21.15.6:0/2814113834 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4059403194 0 0) 0x7fe3100714d0 con 0x7fe310145300 2024-03-04T09:28:55.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.510+0000 7fe316400640 1 --2- 172.21.15.6:0/2814113834 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe3100cce90 0x7fe31015c570 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7fe3100b5150 tx=0x7fe30802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b16cf57a0b36be2 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:55.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.511+0000 7fe30fe00640 1 -- 172.21.15.6:0/2814113834 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe30802ca80 con 0x7fe3100cce90 2024-03-04T09:28:55.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.511+0000 7fe30fe00640 1 -- 172.21.15.6:0/2814113834 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe30802cc20 con 0x7fe3100cce90 2024-03-04T09:28:55.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.511+0000 7fe30fe00640 1 -- 172.21.15.6:0/2814113834 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3080343f0 con 0x7fe3100cce90 2024-03-04T09:28:55.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.511+0000 7fe316e00640 1 -- 172.21.15.6:0/2814113834 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe3100cce90 msgr2=0x7fe31015c570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:55.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.511+0000 7fe316e00640 1 --2- 172.21.15.6:0/2814113834 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe3100cce90 0x7fe31015c570 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7fe3100b5150 tx=0x7fe30802bee0 comp rx=0 tx=0).stop 2024-03-04T09:28:55.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.511+0000 7fe316e00640 1 -- 172.21.15.6:0/2814113834 shutdown_connections 2024-03-04T09:28:55.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.511+0000 7fe316e00640 1 --2- 172.21.15.6:0/2814113834 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe3100cce90 0x7fe31015c570 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:55.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.511+0000 7fe316e00640 1 --2- 172.21.15.6:0/2814113834 >> v2:172.21.15.146:3300/0 conn(0x7fe310070f10 0x7fe3100cc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:55.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.511+0000 7fe316e00640 1 -- 172.21.15.6:0/2814113834 >> 172.21.15.6:0/2814113834 conn(0x7fe31006b3d0 msgr2=0x7fe31006b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:55.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.512+0000 7fe316e00640 1 -- 172.21.15.6:0/2814113834 shutdown_connections 2024-03-04T09:28:55.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.512+0000 7fe316e00640 1 -- 172.21.15.6:0/2814113834 wait complete. 2024-03-04T09:28:55.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.513+0000 7fe316e00640 1 Processor -- start 2024-03-04T09:28:55.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.514+0000 7fe316e00640 1 -- start start 2024-03-04T09:28:55.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.514+0000 7fe316e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe310070f10 0x7fe3101f61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:55.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.514+0000 7fe316e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe3100cce90 0x7fe3101e8610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:55.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.515+0000 7fe316e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe310145300 0x7fe31006d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:55.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.515+0000 7fe316e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe31015cab0 con 0x7fe310070f10 2024-03-04T09:28:55.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.515+0000 7fe316e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3101456e0 con 0x7fe3100cce90 2024-03-04T09:28:55.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.515+0000 7fe316e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe31015cc30 con 0x7fe310145300 2024-03-04T09:28:55.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.515+0000 7fe315a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe310070f10 0x7fe3101f61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:55.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.515+0000 7fe315a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe310070f10 0x7fe3101f61b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49240/0 (socket says 172.21.15.6:49240) 2024-03-04T09:28:55.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.515+0000 7fe315a00640 1 -- 172.21.15.6:0/2551699512 learned_addr learned my addr 172.21.15.6:0/2551699512 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:28:55.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.515+0000 7fe316400640 1 --2- 172.21.15.6:0/2551699512 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe310145300 0x7fe31006d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:55.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.515+0000 7fe315a00640 1 -- 172.21.15.6:0/2551699512 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe310145300 msgr2=0x7fe31006d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:55.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.515+0000 7fe315a00640 1 --2- 172.21.15.6:0/2551699512 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe310145300 0x7fe31006d810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:55.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.515+0000 7fe315a00640 1 -- 172.21.15.6:0/2551699512 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe3100cce90 msgr2=0x7fe3101e8610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:55.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.515+0000 7fe315a00640 1 --2- 172.21.15.6:0/2551699512 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe3100cce90 0x7fe3101e8610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:55.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.515+0000 7fe315a00640 1 -- 172.21.15.6:0/2551699512 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3101f66f0 con 0x7fe310070f10 2024-03-04T09:28:55.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.516+0000 7fe315a00640 1 --2- 172.21.15.6:0/2551699512 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe310070f10 0x7fe3101f61b0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fe30400c460 tx=0x7fe30400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:55.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.516+0000 7fe30d600640 1 -- 172.21.15.6:0/2551699512 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3040074e0 con 0x7fe310070f10 2024-03-04T09:28:55.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.516+0000 7fe30d600640 1 -- 172.21.15.6:0/2551699512 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3040047e0 con 0x7fe310070f10 2024-03-04T09:28:55.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.516+0000 7fe316e00640 1 -- 172.21.15.6:0/2551699512 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3101e44c0 con 0x7fe310070f10 2024-03-04T09:28:55.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.516+0000 7fe30d600640 1 -- 172.21.15.6:0/2551699512 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe304004bd0 con 0x7fe310070f10 2024-03-04T09:28:55.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.516+0000 7fe316e00640 1 -- 172.21.15.6:0/2551699512 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3101e4a80 con 0x7fe310070f10 2024-03-04T09:28:55.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.517+0000 7fe316e00640 1 -- 172.21.15.6:0/2551699512 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe31004f330 con 0x7fe310070f10 2024-03-04T09:28:55.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.518+0000 7fe30d600640 1 -- 172.21.15.6:0/2551699512 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe304011070 con 0x7fe310070f10 2024-03-04T09:28:55.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.519+0000 7fe30d600640 1 --2- 172.21.15.6:0/2551699512 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fe2f006b4d0 0x7fe2f006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:55.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.519+0000 7fe30d600640 1 -- 172.21.15.6:0/2551699512 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe304089e80 con 0x7fe310070f10 2024-03-04T09:28:55.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.520+0000 7fe315000640 1 --2- 172.21.15.6:0/2551699512 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fe2f006b4d0 0x7fe2f006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:55.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.521+0000 7fe315000640 1 --2- 172.21.15.6:0/2551699512 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fe2f006b4d0 0x7fe2f006d990 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fe31006e9a0 tx=0x7fe2fc008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:55.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.524+0000 7fe30d600640 1 -- 172.21.15.6:0/2551699512 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe304058c60 con 0x7fe310070f10 2024-03-04T09:28:55.655 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T09:28:55.659+0000 7fa856400640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-04T09:28:55.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.722+0000 7fe316e00640 1 -- 172.21.15.6:0/2551699512 --> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe3101e50d0 con 0x7fe2f006b4d0 2024-03-04T09:28:55.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.742+0000 7fe30d600640 1 -- 172.21.15.6:0/2551699512 <== mgr.4113 v2:172.21.15.146:6832/1788613334 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252066 (secure 0 0 0) 0x7fe3101e50d0 con 0x7fe2f006b4d0 2024-03-04T09:28:55.743 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:28:55.744 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-03-04T09:28:55.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.748+0000 7fe316e00640 1 -- 172.21.15.6:0/2551699512 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fe2f006b4d0 msgr2=0x7fe2f006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:55.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.748+0000 7fe316e00640 1 --2- 172.21.15.6:0/2551699512 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fe2f006b4d0 0x7fe2f006d990 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fe31006e9a0 tx=0x7fe2fc008040 comp rx=0 tx=0).stop 2024-03-04T09:28:55.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.748+0000 7fe316e00640 1 -- 172.21.15.6:0/2551699512 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe310070f10 msgr2=0x7fe3101f61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:55.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.748+0000 7fe316e00640 1 --2- 172.21.15.6:0/2551699512 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe310070f10 0x7fe3101f61b0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fe30400c460 tx=0x7fe30400c920 comp rx=0 tx=0).stop 2024-03-04T09:28:55.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.748+0000 7fe316e00640 1 -- 172.21.15.6:0/2551699512 shutdown_connections 2024-03-04T09:28:55.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.748+0000 7fe316e00640 1 --2- 172.21.15.6:0/2551699512 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fe2f006b4d0 0x7fe2f006d990 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:55.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.748+0000 7fe316e00640 1 --2- 172.21.15.6:0/2551699512 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe310145300 0x7fe31006d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:55.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.748+0000 7fe316e00640 1 --2- 172.21.15.6:0/2551699512 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe3100cce90 0x7fe3101e8610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:55.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.748+0000 7fe316e00640 1 --2- 172.21.15.6:0/2551699512 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe310070f10 0x7fe3101f61b0 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:55.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.748+0000 7fe316e00640 1 -- 172.21.15.6:0/2551699512 >> 172.21.15.6:0/2551699512 conn(0x7fe31006b3d0 msgr2=0x7fe310072470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:55.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.748+0000 7fe316e00640 1 -- 172.21.15.6:0/2551699512 shutdown_connections 2024-03-04T09:28:55.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:55.748+0000 7fe316e00640 1 -- 172.21.15.6:0/2551699512 wait complete. 2024-03-04T09:28:55.766 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":174,"stamp":"2024-03-04T09:28:53.935566+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":119,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":257096,"kb_used_data":7628,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754717624,"statfs":{"total":773094113280,"available":772830846976,"internally_reserved":0,"allocated":7811072,"data_stored":3930616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.719314"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.278157+0000","last_change":"2024-03-04T09:28:51.291006+0000","last_active":"2024-03-04T09:28:52.278157+0000","last_peered":"2024-03-04T09:28:52.278157+0000","last_clean":"2024-03-04T09:28:52.278157+0000","last_became_active":"2024-03-04T09:28:51.288593+0000","last_became_peered":"2024-03-04T09:28:51.288593+0000","last_unstale":"2024-03-04T09:28:52.278157+0000","last_undegraded":"2024-03-04T09:28:52.278157+0000","last_fullsized":"2024-03-04T09:28:52.278157+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:50:26.619245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.846313+0000","last_change":"2024-03-04T09:28:51.290989+0000","last_active":"2024-03-04T09:28:52.846313+0000","last_peered":"2024-03-04T09:28:52.846313+0000","last_clean":"2024-03-04T09:28:52.846313+0000","last_became_active":"2024-03-04T09:28:51.285332+0000","last_became_peered":"2024-03-04T09:28:51.285332+0000","last_unstale":"2024-03-04T09:28:52.846313+0000","last_undegraded":"2024-03-04T09:28:52.846313+0000","last_fullsized":"2024-03-04T09:28:52.846313+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:42:54.712273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.846662+0000","last_change":"2024-03-04T09:28:51.277640+0000","last_active":"2024-03-04T09:28:52.846662+0000","last_peered":"2024-03-04T09:28:52.846662+0000","last_clean":"2024-03-04T09:28:52.846662+0000","last_became_active":"2024-03-04T09:28:51.276654+0000","last_became_peered":"2024-03-04T09:28:51.276654+0000","last_unstale":"2024-03-04T09:28:52.846662+0000","last_undegraded":"2024-03-04T09:28:52.846662+0000","last_fullsized":"2024-03-04T09:28:52.846662+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:40:06.464005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.275291+0000","last_change":"2024-03-04T09:28:51.287933+0000","last_active":"2024-03-04T09:28:52.275291+0000","last_peered":"2024-03-04T09:28:52.275291+0000","last_clean":"2024-03-04T09:28:52.275291+0000","last_became_active":"2024-03-04T09:28:51.285522+0000","last_became_peered":"2024-03-04T09:28:51.285522+0000","last_unstale":"2024-03-04T09:28:52.275291+0000","last_undegraded":"2024-03-04T09:28:52.275291+0000","last_fullsized":"2024-03-04T09:28:52.275291+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:35:10.446104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.278228+0000","last_change":"2024-03-04T09:28:51.274879+0000","last_active":"2024-03-04T09:28:52.278228+0000","last_peered":"2024-03-04T09:28:52.278228+0000","last_clean":"2024-03-04T09:28:52.278228+0000","last_became_active":"2024-03-04T09:28:51.272881+0000","last_became_peered":"2024-03-04T09:28:51.272881+0000","last_unstale":"2024-03-04T09:28:52.278228+0000","last_undegraded":"2024-03-04T09:28:52.278228+0000","last_fullsized":"2024-03-04T09:28:52.278228+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:17:20.224241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.845696+0000","last_change":"2024-03-04T09:28:51.285099+0000","last_active":"2024-03-04T09:28:52.845696+0000","last_peered":"2024-03-04T09:28:52.845696+0000","last_clean":"2024-03-04T09:28:52.845696+0000","last_became_active":"2024-03-04T09:28:51.284032+0000","last_became_peered":"2024-03-04T09:28:51.284032+0000","last_unstale":"2024-03-04T09:28:52.845696+0000","last_undegraded":"2024-03-04T09:28:52.845696+0000","last_fullsized":"2024-03-04T09:28:52.845696+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:52:58.620617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.277892+0000","last_change":"2024-03-04T09:28:51.284413+0000","last_active":"2024-03-04T09:28:52.277892+0000","last_peered":"2024-03-04T09:28:52.277892+0000","last_clean":"2024-03-04T09:28:52.277892+0000","last_became_active":"2024-03-04T09:28:51.284108+0000","last_became_peered":"2024-03-04T09:28:51.284108+0000","last_unstale":"2024-03-04T09:28:52.277892+0000","last_undegraded":"2024-03-04T09:28:52.277892+0000","last_fullsized":"2024-03-04T09:28:52.277892+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:30:16.400951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.845944+0000","last_change":"2024-03-04T09:28:51.299723+0000","last_active":"2024-03-04T09:28:52.845944+0000","last_peered":"2024-03-04T09:28:52.845944+0000","last_clean":"2024-03-04T09:28:52.845944+0000","last_became_active":"2024-03-04T09:28:51.298902+0000","last_became_peered":"2024-03-04T09:28:51.298902+0000","last_unstale":"2024-03-04T09:28:52.845944+0000","last_undegraded":"2024-03-04T09:28:52.845944+0000","last_fullsized":"2024-03-04T09:28:52.845944+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:46:14.770359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.278820+0000","last_change":"2024-03-04T09:28:51.278381+0000","last_active":"2024-03-04T09:28:52.278820+0000","last_peered":"2024-03-04T09:28:52.278820+0000","last_clean":"2024-03-04T09:28:52.278820+0000","last_became_active":"2024-03-04T09:28:51.277596+0000","last_became_peered":"2024-03-04T09:28:51.277596+0000","last_unstale":"2024-03-04T09:28:52.278820+0000","last_undegraded":"2024-03-04T09:28:52.278820+0000","last_fullsized":"2024-03-04T09:28:52.278820+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:06:29.423392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.847179+0000","last_change":"2024-03-04T09:28:51.287253+0000","last_active":"2024-03-04T09:28:52.847179+0000","last_peered":"2024-03-04T09:28:52.847179+0000","last_clean":"2024-03-04T09:28:52.847179+0000","last_became_active":"2024-03-04T09:28:51.286981+0000","last_became_peered":"2024-03-04T09:28:51.286981+0000","last_unstale":"2024-03-04T09:28:52.847179+0000","last_undegraded":"2024-03-04T09:28:52.847179+0000","last_fullsized":"2024-03-04T09:28:52.847179+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:04:23.011844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.279565+0000","last_change":"2024-03-04T09:28:51.291266+0000","last_active":"2024-03-04T09:28:52.279565+0000","last_peered":"2024-03-04T09:28:52.279565+0000","last_clean":"2024-03-04T09:28:52.279565+0000","last_became_active":"2024-03-04T09:28:51.289515+0000","last_became_peered":"2024-03-04T09:28:51.289515+0000","last_unstale":"2024-03-04T09:28:52.279565+0000","last_undegraded":"2024-03-04T09:28:52.279565+0000","last_fullsized":"2024-03-04T09:28:52.279565+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:41:40.928565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.275461+0000","last_change":"2024-03-04T09:28:51.271412+0000","last_active":"2024-03-04T09:28:52.275461+0000","last_peered":"2024-03-04T09:28:52.275461+0000","last_clean":"2024-03-04T09:28:52.275461+0000","last_became_active":"2024-03-04T09:28:51.271117+0000","last_became_peered":"2024-03-04T09:28:51.271117+0000","last_unstale":"2024-03-04T09:28:52.275461+0000","last_undegraded":"2024-03-04T09:28:52.275461+0000","last_fullsized":"2024-03-04T09:28:52.275461+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:39:19.241679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.283815+0000","last_change":"2024-03-04T09:28:51.292356+0000","last_active":"2024-03-04T09:28:52.283815+0000","last_peered":"2024-03-04T09:28:52.283815+0000","last_clean":"2024-03-04T09:28:52.283815+0000","last_became_active":"2024-03-04T09:28:51.292029+0000","last_became_peered":"2024-03-04T09:28:51.292029+0000","last_unstale":"2024-03-04T09:28:52.283815+0000","last_undegraded":"2024-03-04T09:28:52.283815+0000","last_fullsized":"2024-03-04T09:28:52.283815+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:46:08.414226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.278005+0000","last_change":"2024-03-04T09:28:51.290626+0000","last_active":"2024-03-04T09:28:52.278005+0000","last_peered":"2024-03-04T09:28:52.278005+0000","last_clean":"2024-03-04T09:28:52.278005+0000","last_became_active":"2024-03-04T09:28:51.290330+0000","last_became_peered":"2024-03-04T09:28:51.290330+0000","last_unstale":"2024-03-04T09:28:52.278005+0000","last_undegraded":"2024-03-04T09:28:52.278005+0000","last_fullsized":"2024-03-04T09:28:52.278005+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:41:27.692459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.283742+0000","last_change":"2024-03-04T09:28:51.280290+0000","last_active":"2024-03-04T09:28:52.283742+0000","last_peered":"2024-03-04T09:28:52.283742+0000","last_clean":"2024-03-04T09:28:52.283742+0000","last_became_active":"2024-03-04T09:28:51.279939+0000","last_became_peered":"2024-03-04T09:28:51.279939+0000","last_unstale":"2024-03-04T09:28:52.283742+0000","last_undegraded":"2024-03-04T09:28:52.283742+0000","last_fullsized":"2024-03-04T09:28:52.283742+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:31:45.267982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.846322+0000","last_change":"2024-03-04T09:28:51.300526+0000","last_active":"2024-03-04T09:28:52.846322+0000","last_peered":"2024-03-04T09:28:52.846322+0000","last_clean":"2024-03-04T09:28:52.846322+0000","last_became_active":"2024-03-04T09:28:51.299888+0000","last_became_peered":"2024-03-04T09:28:51.299888+0000","last_unstale":"2024-03-04T09:28:52.846322+0000","last_undegraded":"2024-03-04T09:28:52.846322+0000","last_fullsized":"2024-03-04T09:28:52.846322+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:43:12.750957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.278898+0000","last_change":"2024-03-04T09:28:51.278151+0000","last_active":"2024-03-04T09:28:52.278898+0000","last_peered":"2024-03-04T09:28:52.278898+0000","last_clean":"2024-03-04T09:28:52.278898+0000","last_became_active":"2024-03-04T09:28:51.277223+0000","last_became_peered":"2024-03-04T09:28:51.277223+0000","last_unstale":"2024-03-04T09:28:52.278898+0000","last_undegraded":"2024-03-04T09:28:52.278898+0000","last_fullsized":"2024-03-04T09:28:52.278898+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:19:10.456432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.275656+0000","last_change":"2024-03-04T09:28:51.289240+0000","last_active":"2024-03-04T09:28:52.275656+0000","last_peered":"2024-03-04T09:28:52.275656+0000","last_clean":"2024-03-04T09:28:52.275656+0000","last_became_active":"2024-03-04T09:28:51.285104+0000","last_became_peered":"2024-03-04T09:28:51.285104+0000","last_unstale":"2024-03-04T09:28:52.275656+0000","last_undegraded":"2024-03-04T09:28:52.275656+0000","last_fullsized":"2024-03-04T09:28:52.275656+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:37:28.918787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.846054+0000","last_change":"2024-03-04T09:28:51.284530+0000","last_active":"2024-03-04T09:28:52.846054+0000","last_peered":"2024-03-04T09:28:52.846054+0000","last_clean":"2024-03-04T09:28:52.846054+0000","last_became_active":"2024-03-04T09:28:51.284234+0000","last_became_peered":"2024-03-04T09:28:51.284234+0000","last_unstale":"2024-03-04T09:28:52.846054+0000","last_undegraded":"2024-03-04T09:28:52.846054+0000","last_fullsized":"2024-03-04T09:28:52.846054+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:55:37.556856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.284073+0000","last_change":"2024-03-04T09:28:51.287316+0000","last_active":"2024-03-04T09:28:52.284073+0000","last_peered":"2024-03-04T09:28:52.284073+0000","last_clean":"2024-03-04T09:28:52.284073+0000","last_became_active":"2024-03-04T09:28:51.286937+0000","last_became_peered":"2024-03-04T09:28:51.286937+0000","last_unstale":"2024-03-04T09:28:52.284073+0000","last_undegraded":"2024-03-04T09:28:52.284073+0000","last_fullsized":"2024-03-04T09:28:52.284073+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:14:24.426848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.846943+0000","last_change":"2024-03-04T09:28:51.279950+0000","last_active":"2024-03-04T09:28:52.846943+0000","last_peered":"2024-03-04T09:28:52.846943+0000","last_clean":"2024-03-04T09:28:52.846943+0000","last_became_active":"2024-03-04T09:28:51.279628+0000","last_became_peered":"2024-03-04T09:28:51.279628+0000","last_unstale":"2024-03-04T09:28:52.846943+0000","last_undegraded":"2024-03-04T09:28:52.846943+0000","last_fullsized":"2024-03-04T09:28:52.846943+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:40:44.950933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.846744+0000","last_change":"2024-03-04T09:28:51.287505+0000","last_active":"2024-03-04T09:28:52.846744+0000","last_peered":"2024-03-04T09:28:52.846744+0000","last_clean":"2024-03-04T09:28:52.846744+0000","last_became_active":"2024-03-04T09:28:51.286364+0000","last_became_peered":"2024-03-04T09:28:51.286364+0000","last_unstale":"2024-03-04T09:28:52.846744+0000","last_undegraded":"2024-03-04T09:28:52.846744+0000","last_fullsized":"2024-03-04T09:28:52.846744+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:50:21.773645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.277427+0000","last_change":"2024-03-04T09:28:51.293297+0000","last_active":"2024-03-04T09:28:52.277427+0000","last_peered":"2024-03-04T09:28:52.277427+0000","last_clean":"2024-03-04T09:28:52.277427+0000","last_became_active":"2024-03-04T09:28:51.292187+0000","last_became_peered":"2024-03-04T09:28:51.292187+0000","last_unstale":"2024-03-04T09:28:52.277427+0000","last_undegraded":"2024-03-04T09:28:52.277427+0000","last_fullsized":"2024-03-04T09:28:52.277427+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:14:24.426848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.277338+0000","last_change":"2024-03-04T09:28:51.295074+0000","last_active":"2024-03-04T09:28:52.277338+0000","last_peered":"2024-03-04T09:28:52.277338+0000","last_clean":"2024-03-04T09:28:52.277338+0000","last_became_active":"2024-03-04T09:28:51.294173+0000","last_became_peered":"2024-03-04T09:28:51.294173+0000","last_unstale":"2024-03-04T09:28:52.277338+0000","last_undegraded":"2024-03-04T09:28:52.277338+0000","last_fullsized":"2024-03-04T09:28:52.277338+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:46:08.414226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.847212+0000","last_change":"2024-03-04T09:28:51.304555+0000","last_active":"2024-03-04T09:28:52.847212+0000","last_peered":"2024-03-04T09:28:52.847212+0000","last_clean":"2024-03-04T09:28:52.847212+0000","last_became_active":"2024-03-04T09:28:51.304180+0000","last_became_peered":"2024-03-04T09:28:51.304180+0000","last_unstale":"2024-03-04T09:28:52.847212+0000","last_undegraded":"2024-03-04T09:28:52.847212+0000","last_fullsized":"2024-03-04T09:28:52.847212+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:30:16.400951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.278678+0000","last_change":"2024-03-04T09:28:51.302858+0000","last_active":"2024-03-04T09:28:52.278678+0000","last_peered":"2024-03-04T09:28:52.278678+0000","last_clean":"2024-03-04T09:28:52.278678+0000","last_became_active":"2024-03-04T09:28:51.302536+0000","last_became_peered":"2024-03-04T09:28:51.302536+0000","last_unstale":"2024-03-04T09:28:52.278678+0000","last_undegraded":"2024-03-04T09:28:52.278678+0000","last_fullsized":"2024-03-04T09:28:52.278678+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:46:08.414226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.846915+0000","last_change":"2024-03-04T09:28:51.283168+0000","last_active":"2024-03-04T09:28:52.846915+0000","last_peered":"2024-03-04T09:28:52.846915+0000","last_clean":"2024-03-04T09:28:52.846915+0000","last_became_active":"2024-03-04T09:28:51.282883+0000","last_became_peered":"2024-03-04T09:28:51.282883+0000","last_unstale":"2024-03-04T09:28:52.846915+0000","last_undegraded":"2024-03-04T09:28:52.846915+0000","last_fullsized":"2024-03-04T09:28:52.846915+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:30:19.523187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.846945+0000","last_change":"2024-03-04T09:28:51.288242+0000","last_active":"2024-03-04T09:28:52.846945+0000","last_peered":"2024-03-04T09:28:52.846945+0000","last_clean":"2024-03-04T09:28:52.846945+0000","last_became_active":"2024-03-04T09:28:51.287545+0000","last_became_peered":"2024-03-04T09:28:51.287545+0000","last_unstale":"2024-03-04T09:28:52.846945+0000","last_undegraded":"2024-03-04T09:28:52.846945+0000","last_fullsized":"2024-03-04T09:28:52.846945+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:53:57.607241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.846835+0000","last_change":"2024-03-04T09:28:51.294887+0000","last_active":"2024-03-04T09:28:52.846835+0000","last_peered":"2024-03-04T09:28:52.846835+0000","last_clean":"2024-03-04T09:28:52.846835+0000","last_became_active":"2024-03-04T09:28:51.294549+0000","last_became_peered":"2024-03-04T09:28:51.294549+0000","last_unstale":"2024-03-04T09:28:52.846835+0000","last_undegraded":"2024-03-04T09:28:52.846835+0000","last_fullsized":"2024-03-04T09:28:52.846835+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:34:18.046888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.278977+0000","last_change":"2024-03-04T09:28:51.301289+0000","last_active":"2024-03-04T09:28:52.278977+0000","last_peered":"2024-03-04T09:28:52.278977+0000","last_clean":"2024-03-04T09:28:52.278977+0000","last_became_active":"2024-03-04T09:28:51.300987+0000","last_became_peered":"2024-03-04T09:28:51.300987+0000","last_unstale":"2024-03-04T09:28:52.278977+0000","last_undegraded":"2024-03-04T09:28:52.278977+0000","last_fullsized":"2024-03-04T09:28:52.278977+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:04:27.619250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.278577+0000","last_change":"2024-03-04T09:28:51.286243+0000","last_active":"2024-03-04T09:28:52.278577+0000","last_peered":"2024-03-04T09:28:52.278577+0000","last_clean":"2024-03-04T09:28:52.278577+0000","last_became_active":"2024-03-04T09:28:51.285945+0000","last_became_peered":"2024-03-04T09:28:51.285945+0000","last_unstale":"2024-03-04T09:28:52.278577+0000","last_undegraded":"2024-03-04T09:28:52.278577+0000","last_fullsized":"2024-03-04T09:28:52.278577+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:31:45.267982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.846871+0000","last_change":"2024-03-04T09:28:51.295866+0000","last_active":"2024-03-04T09:28:52.846871+0000","last_peered":"2024-03-04T09:28:52.846871+0000","last_clean":"2024-03-04T09:28:52.846871+0000","last_became_active":"2024-03-04T09:28:51.295588+0000","last_became_peered":"2024-03-04T09:28:51.295588+0000","last_unstale":"2024-03-04T09:28:52.846871+0000","last_undegraded":"2024-03-04T09:28:52.846871+0000","last_fullsized":"2024-03-04T09:28:52.846871+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:40:52.179638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.278119+0000","last_change":"2024-03-04T09:28:51.306539+0000","last_active":"2024-03-04T09:28:52.278119+0000","last_peered":"2024-03-04T09:28:52.278119+0000","last_clean":"2024-03-04T09:28:52.278119+0000","last_became_active":"2024-03-04T09:28:51.306207+0000","last_became_peered":"2024-03-04T09:28:51.306207+0000","last_unstale":"2024-03-04T09:28:52.278119+0000","last_undegraded":"2024-03-04T09:28:52.278119+0000","last_fullsized":"2024-03-04T09:28:52.278119+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:04:27.619250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1f","version":"67'4","reported_seq":18,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.278445+0000","last_change":"2024-03-04T09:28:50.256035+0000","last_active":"2024-03-04T09:28:52.278445+0000","last_peered":"2024-03-04T09:28:52.278445+0000","last_clean":"2024-03-04T09:28:52.278445+0000","last_became_active":"2024-03-04T09:28:50.255761+0000","last_became_peered":"2024-03-04T09:28:50.255761+0000","last_unstale":"2024-03-04T09:28:52.278445+0000","last_undegraded":"2024-03-04T09:28:52.278445+0000","last_fullsized":"2024-03-04T09:28:52.278445+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_clean_scrub_stamp":"2024-03-04T09:28:49.199103+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:47:44.767861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.279095+0000","last_change":"2024-03-04T09:28:51.277880+0000","last_active":"2024-03-04T09:28:52.279095+0000","last_peered":"2024-03-04T09:28:52.279095+0000","last_clean":"2024-03-04T09:28:52.279095+0000","last_became_active":"2024-03-04T09:28:51.276824+0000","last_became_peered":"2024-03-04T09:28:51.276824+0000","last_unstale":"2024-03-04T09:28:52.279095+0000","last_undegraded":"2024-03-04T09:28:52.279095+0000","last_fullsized":"2024-03-04T09:28:52.279095+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:39:35.069899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"67'1","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.284348+0000","last_change":"2024-03-04T09:28:50.241119+0000","last_active":"2024-03-04T09:28:52.284348+0000","last_peered":"2024-03-04T09:28:52.284348+0000","last_clean":"2024-03-04T09:28:52.284348+0000","last_became_active":"2024-03-04T09:28:50.240831+0000","last_became_peered":"2024-03-04T09:28:50.240831+0000","last_unstale":"2024-03-04T09:28:52.284348+0000","last_undegraded":"2024-03-04T09:28:52.284348+0000","last_fullsized":"2024-03-04T09:28:52.284348+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_clean_scrub_stamp":"2024-03-04T09:28:49.199103+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:43:11.736446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.846030+0000","last_change":"2024-03-04T09:28:51.307590+0000","last_active":"2024-03-04T09:28:52.846030+0000","last_peered":"2024-03-04T09:28:52.846030+0000","last_clean":"2024-03-04T09:28:52.846030+0000","last_became_active":"2024-03-04T09:28:51.306722+0000","last_became_peered":"2024-03-04T09:28:51.306722+0000","last_unstale":"2024-03-04T09:28:52.846030+0000","last_undegraded":"2024-03-04T09:28:52.846030+0000","last_fullsized":"2024-03-04T09:28:52.846030+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:13:39.526501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1d","version":"67'2","reported_seq":16,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.847107+0000","last_change":"2024-03-04T09:28:50.253226+0000","last_active":"2024-03-04T09:28:52.847107+0000","last_peered":"2024-03-04T09:28:52.847107+0000","last_clean":"2024-03-04T09:28:52.847107+0000","last_became_active":"2024-03-04T09:28:50.251509+0000","last_became_peered":"2024-03-04T09:28:50.251509+0000","last_unstale":"2024-03-04T09:28:52.847107+0000","last_undegraded":"2024-03-04T09:28:52.847107+0000","last_fullsized":"2024-03-04T09:28:52.847107+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_clean_scrub_stamp":"2024-03-04T09:28:49.199103+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:55:36.542345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.279770+0000","last_change":"2024-03-04T09:28:51.283593+0000","last_active":"2024-03-04T09:28:52.279770+0000","last_peered":"2024-03-04T09:28:52.279770+0000","last_clean":"2024-03-04T09:28:52.279770+0000","last_became_active":"2024-03-04T09:28:51.283286+0000","last_became_peered":"2024-03-04T09:28:51.283286+0000","last_unstale":"2024-03-04T09:28:52.279770+0000","last_undegraded":"2024-03-04T09:28:52.279770+0000","last_fullsized":"2024-03-04T09:28:52.279770+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:12:21.069693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1c","version":"67'2","reported_seq":16,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.846268+0000","last_change":"2024-03-04T09:28:50.250103+0000","last_active":"2024-03-04T09:28:52.846268+0000","last_peered":"2024-03-04T09:28:52.846268+0000","last_clean":"2024-03-04T09:28:52.846268+0000","last_became_active":"2024-03-04T09:28:50.249715+0000","last_became_peered":"2024-03-04T09:28:50.249715+0000","last_unstale":"2024-03-04T09:28:52.846268+0000","last_undegraded":"2024-03-04T09:28:52.846268+0000","last_fullsized":"2024-03-04T09:28:52.846268+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_clean_scrub_stamp":"2024-03-04T09:28:49.199103+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:42:39.759815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.279144+0000","last_change":"2024-03-04T09:28:51.290536+0000","last_active":"2024-03-04T09:28:52.279144+0000","last_peered":"2024-03-04T09:28:52.279144+0000","last_clean":"2024-03-04T09:28:52.279144+0000","last_became_active":"2024-03-04T09:28:51.290065+0000","last_became_peered":"2024-03-04T09:28:51.290065+0000","last_unstale":"2024-03-04T09:28:52.279144+0000","last_undegraded":"2024-03-04T09:28:52.279144+0000","last_fullsized":"2024-03-04T09:28:52.279144+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:50:26.619245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"67'1","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.847158+0000","last_change":"2024-03-04T09:28:50.250204+0000","last_active":"2024-03-04T09:28:52.847158+0000","last_peered":"2024-03-04T09:28:52.847158+0000","last_clean":"2024-03-04T09:28:52.847158+0000","last_became_active":"2024-03-04T09:28:50.247918+0000","last_became_peered":"2024-03-04T09:28:50.247918+0000","last_unstale":"2024-03-04T09:28:52.847158+0000","last_undegraded":"2024-03-04T09:28:52.847158+0000","last_fullsized":"2024-03-04T09:28:52.847158+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_clean_scrub_stamp":"2024-03-04T09:28:49.199103+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:59:04.470819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.847349+0000","last_change":"2024-03-04T09:28:51.287806+0000","last_active":"2024-03-04T09:28:52.847349+0000","last_peered":"2024-03-04T09:28:52.847349+0000","last_clean":"2024-03-04T09:28:52.847349+0000","last_became_active":"2024-03-04T09:28:51.282811+0000","last_became_peered":"2024-03-04T09:28:51.282811+0000","last_unstale":"2024-03-04T09:28:52.847349+0000","last_undegraded":"2024-03-04T09:28:52.847349+0000","last_fullsized":"2024-03-04T09:28:52.847349+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:31:45.267982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"67'1","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.845746+0000","last_change":"2024-03-04T09:28:50.255231+0000","last_active":"2024-03-04T09:28:52.845746+0000","last_peered":"2024-03-04T09:28:52.845746+0000","last_clean":"2024-03-04T09:28:52.845746+0000","last_became_active":"2024-03-04T09:28:50.254623+0000","last_became_peered":"2024-03-04T09:28:50.254623+0000","last_unstale":"2024-03-04T09:28:52.845746+0000","last_undegraded":"2024-03-04T09:28:52.845746+0000","last_fullsized":"2024-03-04T09:28:52.845746+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_clean_scrub_stamp":"2024-03-04T09:28:49.199103+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:19:09.441921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.277684+0000","last_change":"2024-03-04T09:28:51.277060+0000","last_active":"2024-03-04T09:28:52.277684+0000","last_peered":"2024-03-04T09:28:52.277684+0000","last_clean":"2024-03-04T09:28:52.277684+0000","last_became_active":"2024-03-04T09:28:51.276220+0000","last_became_peered":"2024-03-04T09:28:51.276220+0000","last_unstale":"2024-03-04T09:28:52.277684+0000","last_undegraded":"2024-03-04T09:28:52.277684+0000","last_fullsized":"2024-03-04T09:28:52.277684+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:26:47.680959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.274443+0000","last_change":"2024-03-04T09:28:50.242172+0000","last_active":"2024-03-04T09:28:52.274443+0000","last_peered":"2024-03-04T09:28:52.274443+0000","last_clean":"2024-03-04T09:28:52.274443+0000","last_became_active":"2024-03-04T09:28:50.241796+0000","last_became_peered":"2024-03-04T09:28:50.241796+0000","last_unstale":"2024-03-04T09:28:52.274443+0000","last_undegraded":"2024-03-04T09:28:52.274443+0000","last_fullsized":"2024-03-04T09:28:52.274443+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_clean_scrub_stamp":"2024-03-04T09:28:49.199103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:19:09.441921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.284476+0000","last_change":"2024-03-04T09:28:51.282196+0000","last_active":"2024-03-04T09:28:52.284476+0000","last_peered":"2024-03-04T09:28:52.284476+0000","last_clean":"2024-03-04T09:28:52.284476+0000","last_became_active":"2024-03-04T09:28:51.281804+0000","last_became_peered":"2024-03-04T09:28:51.281804+0000","last_unstale":"2024-03-04T09:28:52.284476+0000","last_undegraded":"2024-03-04T09:28:52.284476+0000","last_fullsized":"2024-03-04T09:28:52.284476+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:32:02.177278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.277678+0000","last_change":"2024-03-04T09:28:50.242171+0000","last_active":"2024-03-04T09:28:52.277678+0000","last_peered":"2024-03-04T09:28:52.277678+0000","last_clean":"2024-03-04T09:28:52.277678+0000","last_became_active":"2024-03-04T09:28:50.240397+0000","last_became_peered":"2024-03-04T09:28:50.240397+0000","last_unstale":"2024-03-04T09:28:52.277678+0000","last_undegraded":"2024-03-04T09:28:52.277678+0000","last_fullsized":"2024-03-04T09:28:52.277678+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_clean_scrub_stamp":"2024-03-04T09:28:49.199103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:27:14.886296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.278805+0000","last_change":"2024-03-04T09:28:51.287557+0000","last_active":"2024-03-04T09:28:52.278805+0000","last_peered":"2024-03-04T09:28:52.278805+0000","last_clean":"2024-03-04T09:28:52.278805+0000","last_became_active":"2024-03-04T09:28:51.287220+0000","last_became_peered":"2024-03-04T09:28:51.287220+0000","last_unstale":"2024-03-04T09:28:52.278805+0000","last_undegraded":"2024-03-04T09:28:52.278805+0000","last_fullsized":"2024-03-04T09:28:52.278805+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:14:24.426848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.847255+0000","last_change":"2024-03-04T09:28:50.242957+0000","last_active":"2024-03-04T09:28:52.847255+0000","last_peered":"2024-03-04T09:28:52.847255+0000","last_clean":"2024-03-04T09:28:52.847255+0000","last_became_active":"2024-03-04T09:28:50.242093+0000","last_became_peered":"2024-03-04T09:28:50.242093+0000","last_unstale":"2024-03-04T09:28:52.847255+0000","last_undegraded":"2024-03-04T09:28:52.847255+0000","last_fullsized":"2024-03-04T09:28:52.847255+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_clean_scrub_stamp":"2024-03-04T09:28:49.199103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:13:38.511990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.278247+0000","last_change":"2024-03-04T09:28:51.294474+0000","last_active":"2024-03-04T09:28:52.278247+0000","last_peered":"2024-03-04T09:28:52.278247+0000","last_clean":"2024-03-04T09:28:52.278247+0000","last_became_active":"2024-03-04T09:28:51.294105+0000","last_became_peered":"2024-03-04T09:28:51.294105+0000","last_unstale":"2024-03-04T09:28:52.278247+0000","last_undegraded":"2024-03-04T09:28:52.278247+0000","last_fullsized":"2024-03-04T09:28:52.278247+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:32:02.177278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.846284+0000","last_change":"2024-03-04T09:28:50.259691+0000","last_active":"2024-03-04T09:28:52.846284+0000","last_peered":"2024-03-04T09:28:52.846284+0000","last_clean":"2024-03-04T09:28:52.846284+0000","last_became_active":"2024-03-04T09:28:50.259417+0000","last_became_peered":"2024-03-04T09:28:50.259417+0000","last_unstale":"2024-03-04T09:28:52.846284+0000","last_undegraded":"2024-03-04T09:28:52.846284+0000","last_fullsized":"2024-03-04T09:28:52.846284+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_clean_scrub_stamp":"2024-03-04T09:28:49.199103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:06:28.408881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.846811+0000","last_change":"2024-03-04T09:28:51.287996+0000","last_active":"2024-03-04T09:28:52.846811+0000","last_peered":"2024-03-04T09:28:52.846811+0000","last_clean":"2024-03-04T09:28:52.846811+0000","last_became_active":"2024-03-04T09:28:51.287037+0000","last_became_peered":"2024-03-04T09:28:51.287037+0000","last_unstale":"2024-03-04T09:28:52.846811+0000","last_undegraded":"2024-03-04T09:28:52.846811+0000","last_fullsized":"2024-03-04T09:28:52.846811+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:35:08.303547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"67'3","reported_seq":17,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.847131+0000","last_change":"2024-03-04T09:28:50.258362+0000","last_active":"2024-03-04T09:28:52.847131+0000","last_peered":"2024-03-04T09:28:52.847131+0000","last_clean":"2024-03-04T09:28:52.847131+0000","last_became_active":"2024-03-04T09:28:50.257859+0000","last_became_peered":"2024-03-04T09:28:50.257859+0000","last_unstale":"2024-03-04T09:28:52.847131+0000","last_undegraded":"2024-03-04T09:28:52.847131+0000","last_fullsized":"2024-03-04T09:28:52.847131+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_clean_scrub_stamp":"2024-03-04T09:28:49.199103+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:57:09.368049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.278671+0000","last_change":"2024-03-04T09:28:51.279718+0000","last_active":"2024-03-04T09:28:52.278671+0000","last_peered":"2024-03-04T09:28:52.278671+0000","last_clean":"2024-03-04T09:28:52.278671+0000","last_became_active":"2024-03-04T09:28:51.279420+0000","last_became_peered":"2024-03-04T09:28:51.279420+0000","last_unstale":"2024-03-04T09:28:52.278671+0000","last_undegraded":"2024-03-04T09:28:52.278671+0000","last_fullsized":"2024-03-04T09:28:52.278671+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:31:29.139378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"67'6","reported_seq":20,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.275693+0000","last_change":"2024-03-04T09:28:50.245830+0000","last_active":"2024-03-04T09:28:52.275693+0000","last_peered":"2024-03-04T09:28:52.275693+0000","last_clean":"2024-03-04T09:28:52.275693+0000","last_became_active":"2024-03-04T09:28:50.245528+0000","last_became_peered":"2024-03-04T09:28:50.245528+0000","last_unstale":"2024-03-04T09:28:52.275693+0000","last_undegraded":"2024-03-04T09:28:52.275693+0000","last_fullsized":"2024-03-04T09:28:52.275693+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_clean_scrub_stamp":"2024-03-04T09:28:49.199103+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:18:20.649121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.278728+0000","last_change":"2024-03-04T09:28:51.285212+0000","last_active":"2024-03-04T09:28:52.278728+0000","last_peered":"2024-03-04T09:28:52.278728+0000","last_clean":"2024-03-04T09:28:52.278728+0000","last_became_active":"2024-03-04T09:28:51.284935+0000","last_became_peered":"2024-03-04T09:28:51.284935+0000","last_unstale":"2024-03-04T09:28:52.278728+0000","last_undegraded":"2024-03-04T09:28:52.278728+0000","last_fullsized":"2024-03-04T09:28:52.278728+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:00:55.647309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"67'2","reported_seq":16,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.275017+0000","last_change":"2024-03-04T09:28:50.253940+0000","last_active":"2024-03-04T09:28:52.275017+0000","last_peered":"2024-03-04T09:28:52.275017+0000","last_clean":"2024-03-04T09:28:52.275017+0000","last_became_active":"2024-03-04T09:28:50.253652+0000","last_became_peered":"2024-03-04T09:28:50.253652+0000","last_unstale":"2024-03-04T09:28:52.275017+0000","last_undegraded":"2024-03-04T09:28:52.275017+0000","last_fullsized":"2024-03-04T09:28:52.275017+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_clean_scrub_stamp":"2024-03-04T09:28:49.199103+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:42:50.437322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.277705+0000","last_change":"2024-03-04T09:28:51.288724+0000","last_active":"2024-03-04T09:28:52.277705+0000","last_peered":"2024-03-04T09:28:52.277705+0000","last_clean":"2024-03-04T09:28:52.277705+0000","last_became_active":"2024-03-04T09:28:51.287319+0000","last_became_peered":"2024-03-04T09:28:51.287319+0000","last_unstale":"2024-03-04T09:28:52.277705+0000","last_undegraded":"2024-03-04T09:28:52.277705+0000","last_fullsized":"2024-03-04T09:28:52.277705+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:04:23.011844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.845783+0000","last_change":"2024-03-04T09:28:50.258623+0000","last_active":"2024-03-04T09:28:52.845783+0000","last_peered":"2024-03-04T09:28:52.845783+0000","last_clean":"2024-03-04T09:28:52.845783+0000","last_became_active":"2024-03-04T09:28:50.256343+0000","last_became_peered":"2024-03-04T09:28:50.256343+0000","last_unstale":"2024-03-04T09:28:52.845783+0000","last_undegraded":"2024-03-04T09:28:52.845783+0000","last_fullsized":"2024-03-04T09:28:52.845783+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_clean_scrub_stamp":"2024-03-04T09:28:49.199103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:39:34.055388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.284096+0000","last_change":"2024-03-04T09:28:51.293951+0000","last_active":"2024-03-04T09:28:52.284096+0000","last_peered":"2024-03-04T09:28:52.284096+0000","last_clean":"2024-03-04T09:28:52.284096+0000","last_became_active":"2024-03-04T09:28:51.292713+0000","last_became_peered":"2024-03-04T09:28:51.292713+0000","last_unstale":"2024-03-04T09:28:52.284096+0000","last_undegraded":"2024-03-04T09:28:52.284096+0000","last_fullsized":"2024-03-04T09:28:52.284096+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:04:27.619250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.278154+0000","last_change":"2024-03-04T09:28:50.254527+0000","last_active":"2024-03-04T09:28:52.278154+0000","last_peered":"2024-03-04T09:28:52.278154+0000","last_clean":"2024-03-04T09:28:52.278154+0000","last_became_active":"2024-03-04T09:28:50.254264+0000","last_became_peered":"2024-03-04T09:28:50.254264+0000","last_unstale":"2024-03-04T09:28:52.278154+0000","last_undegraded":"2024-03-04T09:28:52.278154+0000","last_fullsized":"2024-03-04T09:28:52.278154+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_clean_scrub_stamp":"2024-03-04T09:28:49.199103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:40:51.165127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.277887+0000","last_change":"2024-03-04T09:28:51.271877+0000","last_active":"2024-03-04T09:28:52.277887+0000","last_peered":"2024-03-04T09:28:52.277887+0000","last_clean":"2024-03-04T09:28:52.277887+0000","last_became_active":"2024-03-04T09:28:51.270973+0000","last_became_peered":"2024-03-04T09:28:51.270973+0000","last_unstale":"2024-03-04T09:28:52.277887+0000","last_undegraded":"2024-03-04T09:28:52.277887+0000","last_fullsized":"2024-03-04T09:28:52.277887+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:46:30.848164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"67'8","reported_seq":22,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.846640+0000","last_change":"2024-03-04T09:28:50.253971+0000","last_active":"2024-03-04T09:28:52.846640+0000","last_peered":"2024-03-04T09:28:52.846640+0000","last_clean":"2024-03-04T09:28:52.846640+0000","last_became_active":"2024-03-04T09:28:50.253687+0000","last_became_peered":"2024-03-04T09:28:50.253687+0000","last_unstale":"2024-03-04T09:28:52.846640+0000","last_undegraded":"2024-03-04T09:28:52.846640+0000","last_fullsized":"2024-03-04T09:28:52.846640+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_clean_scrub_stamp":"2024-03-04T09:28:49.199103+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:46:13.755848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.847263+0000","last_change":"2024-03-04T09:28:51.279313+0000","last_active":"2024-03-04T09:28:52.847263+0000","last_peered":"2024-03-04T09:28:52.847263+0000","last_clean":"2024-03-04T09:28:52.847263+0000","last_became_active":"2024-03-04T09:28:51.278626+0000","last_became_peered":"2024-03-04T09:28:51.278626+0000","last_unstale":"2024-03-04T09:28:52.847263+0000","last_undegraded":"2024-03-04T09:28:52.847263+0000","last_fullsized":"2024-03-04T09:28:52.847263+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:55:12.363630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"67'1","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.846011+0000","last_change":"2024-03-04T09:28:50.252588+0000","last_active":"2024-03-04T09:28:52.846011+0000","last_peered":"2024-03-04T09:28:52.846011+0000","last_clean":"2024-03-04T09:28:52.846011+0000","last_became_active":"2024-03-04T09:28:50.252321+0000","last_became_peered":"2024-03-04T09:28:50.252321+0000","last_unstale":"2024-03-04T09:28:52.846011+0000","last_undegraded":"2024-03-04T09:28:52.846011+0000","last_fullsized":"2024-03-04T09:28:52.846011+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_clean_scrub_stamp":"2024-03-04T09:28:49.199103+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:17:59.355079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.278368+0000","last_change":"2024-03-04T09:28:51.306663+0000","last_active":"2024-03-04T09:28:52.278368+0000","last_peered":"2024-03-04T09:28:52.278368+0000","last_clean":"2024-03-04T09:28:52.278368+0000","last_became_active":"2024-03-04T09:28:51.305680+0000","last_became_peered":"2024-03-04T09:28:51.305680+0000","last_unstale":"2024-03-04T09:28:52.278368+0000","last_undegraded":"2024-03-04T09:28:52.278368+0000","last_fullsized":"2024-03-04T09:28:52.278368+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:10:47.851740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"67'2","reported_seq":16,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.847270+0000","last_change":"2024-03-04T09:28:50.250228+0000","last_active":"2024-03-04T09:28:52.847270+0000","last_peered":"2024-03-04T09:28:52.847270+0000","last_clean":"2024-03-04T09:28:52.847270+0000","last_became_active":"2024-03-04T09:28:50.249912+0000","last_became_peered":"2024-03-04T09:28:50.249912+0000","last_unstale":"2024-03-04T09:28:52.847270+0000","last_undegraded":"2024-03-04T09:28:52.847270+0000","last_fullsized":"2024-03-04T09:28:52.847270+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_clean_scrub_stamp":"2024-03-04T09:28:49.199103+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:42:53.697762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.275659+0000","last_change":"2024-03-04T09:28:51.273490+0000","last_active":"2024-03-04T09:28:52.275659+0000","last_peered":"2024-03-04T09:28:52.275659+0000","last_clean":"2024-03-04T09:28:52.275659+0000","last_became_active":"2024-03-04T09:28:51.272524+0000","last_became_peered":"2024-03-04T09:28:51.272524+0000","last_unstale":"2024-03-04T09:28:52.275659+0000","last_undegraded":"2024-03-04T09:28:52.275659+0000","last_fullsized":"2024-03-04T09:28:52.275659+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:59:43.290460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"67'1","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.846158+0000","last_change":"2024-03-04T09:28:50.248288+0000","last_active":"2024-03-04T09:28:52.846158+0000","last_peered":"2024-03-04T09:28:52.846158+0000","last_clean":"2024-03-04T09:28:52.846158+0000","last_became_active":"2024-03-04T09:28:50.247795+0000","last_became_peered":"2024-03-04T09:28:50.247795+0000","last_unstale":"2024-03-04T09:28:52.846158+0000","last_undegraded":"2024-03-04T09:28:52.846158+0000","last_fullsized":"2024-03-04T09:28:52.846158+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_clean_scrub_stamp":"2024-03-04T09:28:49.199103+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:17:19.209731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.278661+0000","last_change":"2024-03-04T09:28:51.282596+0000","last_active":"2024-03-04T09:28:52.278661+0000","last_peered":"2024-03-04T09:28:52.278661+0000","last_clean":"2024-03-04T09:28:52.278661+0000","last_became_active":"2024-03-04T09:28:51.282245+0000","last_became_peered":"2024-03-04T09:28:51.282245+0000","last_unstale":"2024-03-04T09:28:52.278661+0000","last_undegraded":"2024-03-04T09:28:52.278661+0000","last_fullsized":"2024-03-04T09:28:52.278661+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:42:40.774326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.277929+0000","last_change":"2024-03-04T09:28:50.259640+0000","last_active":"2024-03-04T09:28:52.277929+0000","last_peered":"2024-03-04T09:28:52.277929+0000","last_clean":"2024-03-04T09:28:52.277929+0000","last_became_active":"2024-03-04T09:28:50.259362+0000","last_became_peered":"2024-03-04T09:28:50.259362+0000","last_unstale":"2024-03-04T09:28:52.277929+0000","last_undegraded":"2024-03-04T09:28:52.277929+0000","last_fullsized":"2024-03-04T09:28:52.277929+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_clean_scrub_stamp":"2024-03-04T09:28:49.199103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:23:31.393401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.847114+0000","last_change":"2024-03-04T09:28:51.267895+0000","last_active":"2024-03-04T09:28:52.847114+0000","last_peered":"2024-03-04T09:28:52.847114+0000","last_clean":"2024-03-04T09:28:52.847114+0000","last_became_active":"2024-03-04T09:28:51.267552+0000","last_became_peered":"2024-03-04T09:28:51.267552+0000","last_unstale":"2024-03-04T09:28:52.847114+0000","last_undegraded":"2024-03-04T09:28:52.847114+0000","last_fullsized":"2024-03-04T09:28:52.847114+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:05:13.857612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.846657+0000","last_change":"2024-03-04T09:28:50.257192+0000","last_active":"2024-03-04T09:28:52.846657+0000","last_peered":"2024-03-04T09:28:52.846657+0000","last_clean":"2024-03-04T09:28:52.846657+0000","last_became_active":"2024-03-04T09:28:50.254174+0000","last_became_peered":"2024-03-04T09:28:50.254174+0000","last_unstale":"2024-03-04T09:28:52.846657+0000","last_undegraded":"2024-03-04T09:28:52.846657+0000","last_fullsized":"2024-03-04T09:28:52.846657+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_clean_scrub_stamp":"2024-03-04T09:28:49.199103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:42:50.437322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.846376+0000","last_change":"2024-03-04T09:28:51.287836+0000","last_active":"2024-03-04T09:28:52.846376+0000","last_peered":"2024-03-04T09:28:52.846376+0000","last_clean":"2024-03-04T09:28:52.846376+0000","last_became_active":"2024-03-04T09:28:51.287517+0000","last_became_peered":"2024-03-04T09:28:51.287517+0000","last_unstale":"2024-03-04T09:28:52.846376+0000","last_undegraded":"2024-03-04T09:28:52.846376+0000","last_fullsized":"2024-03-04T09:28:52.846376+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:25:43.926643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.278888+0000","last_change":"2024-03-04T09:28:50.257249+0000","last_active":"2024-03-04T09:28:52.278888+0000","last_peered":"2024-03-04T09:28:52.278888+0000","last_clean":"2024-03-04T09:28:52.278888+0000","last_became_active":"2024-03-04T09:28:50.256948+0000","last_became_peered":"2024-03-04T09:28:50.256948+0000","last_unstale":"2024-03-04T09:28:52.278888+0000","last_undegraded":"2024-03-04T09:28:52.278888+0000","last_fullsized":"2024-03-04T09:28:52.278888+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_clean_scrub_stamp":"2024-03-04T09:28:49.199103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:23:31.393401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.846481+0000","last_change":"2024-03-04T09:28:51.288816+0000","last_active":"2024-03-04T09:28:52.846481+0000","last_peered":"2024-03-04T09:28:52.846481+0000","last_clean":"2024-03-04T09:28:52.846481+0000","last_became_active":"2024-03-04T09:28:51.288574+0000","last_became_peered":"2024-03-04T09:28:51.288574+0000","last_unstale":"2024-03-04T09:28:52.846481+0000","last_undegraded":"2024-03-04T09:28:52.846481+0000","last_fullsized":"2024-03-04T09:28:52.846481+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:12:46.486857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.283884+0000","last_change":"2024-03-04T09:28:50.243400+0000","last_active":"2024-03-04T09:28:52.283884+0000","last_peered":"2024-03-04T09:28:52.283884+0000","last_clean":"2024-03-04T09:28:52.283884+0000","last_became_active":"2024-03-04T09:28:50.240709+0000","last_became_peered":"2024-03-04T09:28:50.240709+0000","last_unstale":"2024-03-04T09:28:52.283884+0000","last_undegraded":"2024-03-04T09:28:52.283884+0000","last_fullsized":"2024-03-04T09:28:52.283884+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_clean_scrub_stamp":"2024-03-04T09:28:49.199103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:12:41.549376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.284329+0000","last_change":"2024-03-04T09:28:51.279275+0000","last_active":"2024-03-04T09:28:52.284329+0000","last_peered":"2024-03-04T09:28:52.284329+0000","last_clean":"2024-03-04T09:28:52.284329+0000","last_became_active":"2024-03-04T09:28:51.278907+0000","last_became_peered":"2024-03-04T09:28:51.278907+0000","last_unstale":"2024-03-04T09:28:52.284329+0000","last_undegraded":"2024-03-04T09:28:52.284329+0000","last_fullsized":"2024-03-04T09:28:52.284329+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:50:26.619245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"67'2","reported_seq":16,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.846115+0000","last_change":"2024-03-04T09:28:50.253782+0000","last_active":"2024-03-04T09:28:52.846115+0000","last_peered":"2024-03-04T09:28:52.846115+0000","last_clean":"2024-03-04T09:28:52.846115+0000","last_became_active":"2024-03-04T09:28:50.253393+0000","last_became_peered":"2024-03-04T09:28:50.253393+0000","last_unstale":"2024-03-04T09:28:52.846115+0000","last_undegraded":"2024-03-04T09:28:52.846115+0000","last_fullsized":"2024-03-04T09:28:52.846115+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_clean_scrub_stamp":"2024-03-04T09:28:49.199103+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:31:28.124868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.278349+0000","last_change":"2024-03-04T09:28:51.288030+0000","last_active":"2024-03-04T09:28:52.278349+0000","last_peered":"2024-03-04T09:28:52.278349+0000","last_clean":"2024-03-04T09:28:52.278349+0000","last_became_active":"2024-03-04T09:28:51.287733+0000","last_became_peered":"2024-03-04T09:28:51.287733+0000","last_unstale":"2024-03-04T09:28:52.278349+0000","last_undegraded":"2024-03-04T09:28:52.278349+0000","last_fullsized":"2024-03-04T09:28:52.278349+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:55:12.363630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"68'1","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.282294+0000","last_change":"2024-03-04T09:28:50.257730+0000","last_active":"2024-03-04T09:28:52.282294+0000","last_peered":"2024-03-04T09:28:52.282294+0000","last_clean":"2024-03-04T09:28:52.282294+0000","last_became_active":"2024-03-04T09:28:50.257342+0000","last_became_peered":"2024-03-04T09:28:50.257342+0000","last_unstale":"2024-03-04T09:28:52.282294+0000","last_undegraded":"2024-03-04T09:28:52.282294+0000","last_fullsized":"2024-03-04T09:28:52.282294+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_clean_scrub_stamp":"2024-03-04T09:28:49.199103+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:35:07.289036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.279119+0000","last_change":"2024-03-04T09:28:51.297902+0000","last_active":"2024-03-04T09:28:52.279119+0000","last_peered":"2024-03-04T09:28:52.279119+0000","last_clean":"2024-03-04T09:28:52.279119+0000","last_became_active":"2024-03-04T09:28:51.297567+0000","last_became_peered":"2024-03-04T09:28:51.297567+0000","last_unstale":"2024-03-04T09:28:52.279119+0000","last_undegraded":"2024-03-04T09:28:52.279119+0000","last_fullsized":"2024-03-04T09:28:52.279119+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:40:40.882472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.278766+0000","last_change":"2024-03-04T09:28:50.247357+0000","last_active":"2024-03-04T09:28:52.278766+0000","last_peered":"2024-03-04T09:28:52.278766+0000","last_clean":"2024-03-04T09:28:52.278766+0000","last_became_active":"2024-03-04T09:28:50.247078+0000","last_became_peered":"2024-03-04T09:28:50.247078+0000","last_unstale":"2024-03-04T09:28:52.278766+0000","last_undegraded":"2024-03-04T09:28:52.278766+0000","last_fullsized":"2024-03-04T09:28:52.278766+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_clean_scrub_stamp":"2024-03-04T09:28:49.199103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:35:00.022149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.280105+0000","last_change":"2024-03-04T09:28:51.295573+0000","last_active":"2024-03-04T09:28:52.280105+0000","last_peered":"2024-03-04T09:28:52.280105+0000","last_clean":"2024-03-04T09:28:52.280105+0000","last_became_active":"2024-03-04T09:28:51.292792+0000","last_became_peered":"2024-03-04T09:28:51.292792+0000","last_unstale":"2024-03-04T09:28:52.280105+0000","last_undegraded":"2024-03-04T09:28:52.280105+0000","last_fullsized":"2024-03-04T09:28:52.280105+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:04:49.289454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.283999+0000","last_change":"2024-03-04T09:28:50.244670+0000","last_active":"2024-03-04T09:28:52.283999+0000","last_peered":"2024-03-04T09:28:52.283999+0000","last_clean":"2024-03-04T09:28:52.283999+0000","last_became_active":"2024-03-04T09:28:50.244396+0000","last_became_peered":"2024-03-04T09:28:50.244396+0000","last_unstale":"2024-03-04T09:28:52.283999+0000","last_undegraded":"2024-03-04T09:28:52.283999+0000","last_fullsized":"2024-03-04T09:28:52.283999+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_clean_scrub_stamp":"2024-03-04T09:28:49.199103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:50:23.693605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.278279+0000","last_change":"2024-03-04T09:28:51.275020+0000","last_active":"2024-03-04T09:28:52.278279+0000","last_peered":"2024-03-04T09:28:52.278279+0000","last_clean":"2024-03-04T09:28:52.278279+0000","last_became_active":"2024-03-04T09:28:51.272292+0000","last_became_peered":"2024-03-04T09:28:51.272292+0000","last_unstale":"2024-03-04T09:28:52.278279+0000","last_undegraded":"2024-03-04T09:28:52.278279+0000","last_fullsized":"2024-03-04T09:28:52.278279+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:18:00.369590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.278187+0000","last_change":"2024-03-04T09:28:50.258722+0000","last_active":"2024-03-04T09:28:52.278187+0000","last_peered":"2024-03-04T09:28:52.278187+0000","last_clean":"2024-03-04T09:28:52.278187+0000","last_became_active":"2024-03-04T09:28:50.258414+0000","last_became_peered":"2024-03-04T09:28:50.258414+0000","last_unstale":"2024-03-04T09:28:52.278187+0000","last_undegraded":"2024-03-04T09:28:52.278187+0000","last_fullsized":"2024-03-04T09:28:52.278187+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_clean_scrub_stamp":"2024-03-04T09:28:49.199103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:02:27.080325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.846348+0000","last_change":"2024-03-04T09:28:51.306716+0000","last_active":"2024-03-04T09:28:52.846348+0000","last_peered":"2024-03-04T09:28:52.846348+0000","last_clean":"2024-03-04T09:28:52.846348+0000","last_became_active":"2024-03-04T09:28:51.306381+0000","last_became_peered":"2024-03-04T09:28:51.306381+0000","last_unstale":"2024-03-04T09:28:52.846348+0000","last_undegraded":"2024-03-04T09:28:52.846348+0000","last_fullsized":"2024-03-04T09:28:52.846348+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:05:51.773362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.275312+0000","last_change":"2024-03-04T09:28:50.248128+0000","last_active":"2024-03-04T09:28:52.275312+0000","last_peered":"2024-03-04T09:28:52.275312+0000","last_clean":"2024-03-04T09:28:52.275312+0000","last_became_active":"2024-03-04T09:28:50.247783+0000","last_became_peered":"2024-03-04T09:28:50.247783+0000","last_unstale":"2024-03-04T09:28:52.275312+0000","last_undegraded":"2024-03-04T09:28:52.275312+0000","last_fullsized":"2024-03-04T09:28:52.275312+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_clean_scrub_stamp":"2024-03-04T09:28:49.199103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:39:34.055388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.274711+0000","last_change":"2024-03-04T09:28:51.274666+0000","last_active":"2024-03-04T09:28:52.274711+0000","last_peered":"2024-03-04T09:28:52.274711+0000","last_clean":"2024-03-04T09:28:52.274711+0000","last_became_active":"2024-03-04T09:28:51.272228+0000","last_became_peered":"2024-03-04T09:28:51.272228+0000","last_unstale":"2024-03-04T09:28:52.274711+0000","last_undegraded":"2024-03-04T09:28:52.274711+0000","last_fullsized":"2024-03-04T09:28:52.274711+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:46:27.964045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.846559+0000","last_change":"2024-03-04T09:28:50.257206+0000","last_active":"2024-03-04T09:28:52.846559+0000","last_peered":"2024-03-04T09:28:52.846559+0000","last_clean":"2024-03-04T09:28:52.846559+0000","last_became_active":"2024-03-04T09:28:50.256864+0000","last_became_peered":"2024-03-04T09:28:50.256864+0000","last_unstale":"2024-03-04T09:28:52.846559+0000","last_undegraded":"2024-03-04T09:28:52.846559+0000","last_fullsized":"2024-03-04T09:28:52.846559+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_clean_scrub_stamp":"2024-03-04T09:28:49.199103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:52:57.606106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.845446+0000","last_change":"2024-03-04T09:28:51.296508+0000","last_active":"2024-03-04T09:28:52.845446+0000","last_peered":"2024-03-04T09:28:52.845446+0000","last_clean":"2024-03-04T09:28:52.845446+0000","last_became_active":"2024-03-04T09:28:51.296111+0000","last_became_peered":"2024-03-04T09:28:51.296111+0000","last_unstale":"2024-03-04T09:28:52.845446+0000","last_undegraded":"2024-03-04T09:28:52.845446+0000","last_fullsized":"2024-03-04T09:28:52.845446+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:40:15.635978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.846510+0000","last_change":"2024-03-04T09:28:50.249908+0000","last_active":"2024-03-04T09:28:52.846510+0000","last_peered":"2024-03-04T09:28:52.846510+0000","last_clean":"2024-03-04T09:28:52.846510+0000","last_became_active":"2024-03-04T09:28:50.249577+0000","last_became_peered":"2024-03-04T09:28:50.249577+0000","last_unstale":"2024-03-04T09:28:52.846510+0000","last_undegraded":"2024-03-04T09:28:52.846510+0000","last_fullsized":"2024-03-04T09:28:52.846510+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_clean_scrub_stamp":"2024-03-04T09:28:49.199103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:30:03.955492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.846621+0000","last_change":"2024-03-04T09:28:51.286417+0000","last_active":"2024-03-04T09:28:52.846621+0000","last_peered":"2024-03-04T09:28:52.846621+0000","last_clean":"2024-03-04T09:28:52.846621+0000","last_became_active":"2024-03-04T09:28:51.284567+0000","last_became_peered":"2024-03-04T09:28:51.284567+0000","last_unstale":"2024-03-04T09:28:52.846621+0000","last_undegraded":"2024-03-04T09:28:52.846621+0000","last_fullsized":"2024-03-04T09:28:52.846621+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:18:24.785867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.b","version":"67'2","reported_seq":16,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.846416+0000","last_change":"2024-03-04T09:28:50.250268+0000","last_active":"2024-03-04T09:28:52.846416+0000","last_peered":"2024-03-04T09:28:52.846416+0000","last_clean":"2024-03-04T09:28:52.846416+0000","last_became_active":"2024-03-04T09:28:50.249233+0000","last_became_peered":"2024-03-04T09:28:50.249233+0000","last_unstale":"2024-03-04T09:28:52.846416+0000","last_undegraded":"2024-03-04T09:28:52.846416+0000","last_fullsized":"2024-03-04T09:28:52.846416+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_clean_scrub_stamp":"2024-03-04T09:28:49.199103+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:18:20.649121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":259,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.277282+0000","last_change":"2024-03-04T09:25:09.316159+0000","last_active":"2024-03-04T09:28:52.277282+0000","last_peered":"2024-03-04T09:28:52.277282+0000","last_clean":"2024-03-04T09:28:52.277282+0000","last_became_active":"2024-03-04T09:25:09.313767+0000","last_became_peered":"2024-03-04T09:25:09.313767+0000","last_unstale":"2024-03-04T09:28:52.277282+0000","last_undegraded":"2024-03-04T09:28:52.277282+0000","last_fullsized":"2024-03-04T09:28:52.277282+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:08.110953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:08.110953+0000","last_clean_scrub_stamp":"2024-03-04T09:25:08.110953+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:34:21.745390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574896,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32580,"kb_used_data":1396,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339260,"statfs":{"total":96636764160,"available":96603402240,"internally_reserved":0,"allocated":1429504,"data_stored":938054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574896,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31984,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339856,"statfs":{"total":96636764160,"available":96604012544,"internally_reserved":0,"allocated":823296,"data_stored":340862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574898,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31996,"kb_used_data":812,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339844,"statfs":{"total":96636764160,"available":96604000256,"internally_reserved":0,"allocated":831488,"data_stored":347160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574895,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31988,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339852,"statfs":{"total":96636764160,"available":96604008448,"internally_reserved":0,"allocated":819200,"data_stored":340840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574897,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31984,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339856,"statfs":{"total":96636764160,"available":96604012544,"internally_reserved":0,"allocated":823296,"data_stored":340882,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574896,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32004,"kb_used_data":820,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339836,"statfs":{"total":96636764160,"available":96603992064,"internally_reserved":0,"allocated":839680,"data_stored":348682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574897,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31996,"kb_used_data":816,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339844,"statfs":{"total":96636764160,"available":96604000256,"internally_reserved":0,"allocated":835584,"data_stored":342998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574896,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32564,"kb_used_data":1376,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339276,"statfs":{"total":96636764160,"available":96603418624,"internally_reserved":0,"allocated":1409024,"data_stored":931138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-04T09:28:55.774 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T09:28:56.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.036+0000 7f4778c00640 1 Processor -- start 2024-03-04T09:28:56.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.037+0000 7f4778c00640 1 -- start start 2024-03-04T09:28:56.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.038+0000 7f4778c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f47740af580 0x7f477415c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:56.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.038+0000 7f4778c00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f477415cc40 con 0x7f47740af1a0 2024-03-04T09:28:56.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.038+0000 7f4778c00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f477415cf20 con 0x7f47740cdc20 2024-03-04T09:28:56.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.038+0000 7f4778c00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f477415d200 con 0x7f47740af580 2024-03-04T09:28:56.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.038+0000 7f4771e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f47740af1a0 0x7f47740b4c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:51758/0 (socket says 172.21.15.6:51758) 2024-03-04T09:28:56.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.038+0000 7f4771e00640 1 -- 172.21.15.6:0/3859057490 learned_addr learned my addr 172.21.15.6:0/3859057490 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:28:56.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.038+0000 7f476be00640 1 --2- 172.21.15.6:0/3859057490 >> v2:172.21.15.6:3300/0 conn(0x7f47740af580 0x7f477415c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:56.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.039+0000 7f4771400640 1 -- 172.21.15.6:0/3859057490 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 74344657 0 0) 0x7f477415cc40 con 0x7f47740af1a0 2024-03-04T09:28:56.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.039+0000 7f4771400640 1 -- 172.21.15.6:0/3859057490 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4758003520 con 0x7f47740af1a0 2024-03-04T09:28:56.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.039+0000 7f476be00640 1 -- 172.21.15.6:0/3859057490 >> v1:172.21.15.146:6789/0 conn(0x7f47740cdc20 legacy=0x7f47740b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:56.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.039+0000 7f476be00640 1 -- 172.21.15.6:0/3859057490 >> v1:172.21.15.6:6789/0 conn(0x7f47740af1a0 legacy=0x7f47740b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:56.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.039+0000 7f476be00640 1 -- 172.21.15.6:0/3859057490 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f477415d4e0 con 0x7f47740af580 2024-03-04T09:28:56.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.041+0000 7f476be00640 1 --2- 172.21.15.6:0/3859057490 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f47740af580 0x7f477415c600 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f475c0097a0 tx=0x7f475c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ca7e6e293b100cb server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:56.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.041+0000 7f4771400640 1 -- 172.21.15.6:0/3859057490 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f475c035020 con 0x7f47740af580 2024-03-04T09:28:56.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.041+0000 7f4771400640 1 -- 172.21.15.6:0/3859057490 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f475c031830 con 0x7f47740af580 2024-03-04T09:28:56.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.041+0000 7f4771400640 1 -- 172.21.15.6:0/3859057490 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f475c031be0 con 0x7f47740af580 2024-03-04T09:28:56.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.042+0000 7f4778c00640 1 -- 172.21.15.6:0/3859057490 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f47740af580 msgr2=0x7f477415c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:56.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.042+0000 7f4778c00640 1 --2- 172.21.15.6:0/3859057490 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f47740af580 0x7f477415c600 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f475c0097a0 tx=0x7f475c02b8d0 comp rx=0 tx=0).stop 2024-03-04T09:28:56.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.042+0000 7f4778c00640 1 -- 172.21.15.6:0/3859057490 shutdown_connections 2024-03-04T09:28:56.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.042+0000 7f4778c00640 1 --2- 172.21.15.6:0/3859057490 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f47740af580 0x7f477415c600 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:56.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.042+0000 7f4778c00640 1 -- 172.21.15.6:0/3859057490 >> 172.21.15.6:0/3859057490 conn(0x7f477406afb0 msgr2=0x7f477406b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:56.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.042+0000 7f4778c00640 1 -- 172.21.15.6:0/3859057490 shutdown_connections 2024-03-04T09:28:56.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.042+0000 7f4778c00640 1 -- 172.21.15.6:0/3859057490 wait complete. 2024-03-04T09:28:56.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.043+0000 7f4778c00640 1 Processor -- start 2024-03-04T09:28:56.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.044+0000 7f4778c00640 1 -- start start 2024-03-04T09:28:56.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.045+0000 7f4778c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f47740af1a0 0x7f47741563b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:56.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.045+0000 7f4778c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f47740af580 0x7f4774143500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:56.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.045+0000 7f4778c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f47740cdc20 0x7f47740b0fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:56.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.045+0000 7f4778c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4774155e70 con 0x7f47740af580 2024-03-04T09:28:56.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.045+0000 7f4778c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47740b3730 con 0x7f47740af1a0 2024-03-04T09:28:56.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.045+0000 7f4778c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4774155ff0 con 0x7f47740cdc20 2024-03-04T09:28:56.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.045+0000 7f476be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f47740af580 0x7f4774143500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:56.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.045+0000 7f476be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f47740af580 0x7f4774143500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49268/0 (socket says 172.21.15.6:49268) 2024-03-04T09:28:56.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.045+0000 7f476be00640 1 -- 172.21.15.6:0/2556216804 learned_addr learned my addr 172.21.15.6:0/2556216804 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:28:56.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.045+0000 7f4771e00640 1 --2- 172.21.15.6:0/2556216804 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f47740af1a0 0x7f47741563b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:56.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.045+0000 7f476be00640 1 -- 172.21.15.6:0/2556216804 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f47740cdc20 msgr2=0x7f47740b0fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:56.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.045+0000 7f476be00640 1 --2- 172.21.15.6:0/2556216804 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f47740cdc20 0x7f47740b0fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:56.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.045+0000 7f476be00640 1 -- 172.21.15.6:0/2556216804 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f47740af1a0 msgr2=0x7f47741563b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:56.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.045+0000 7f476be00640 1 --2- 172.21.15.6:0/2556216804 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f47740af1a0 0x7f47741563b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:56.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.045+0000 7f476be00640 1 -- 172.21.15.6:0/2556216804 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47741f5140 con 0x7f47740af580 2024-03-04T09:28:56.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.046+0000 7f476be00640 1 --2- 172.21.15.6:0/2556216804 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f47740af580 0x7f4774143500 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f475c000c00 tx=0x7f475c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:56.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.046+0000 7f476a000640 1 -- 172.21.15.6:0/2556216804 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f475c035070 con 0x7f47740af580 2024-03-04T09:28:56.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.046+0000 7f476a000640 1 -- 172.21.15.6:0/2556216804 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f475c004230 con 0x7f47740af580 2024-03-04T09:28:56.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.046+0000 7f476a000640 1 -- 172.21.15.6:0/2556216804 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f475c03a630 con 0x7f47740af580 2024-03-04T09:28:56.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.046+0000 7f4778c00640 1 -- 172.21.15.6:0/2556216804 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47741f5420 con 0x7f47740af580 2024-03-04T09:28:56.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.046+0000 7f4778c00640 1 -- 172.21.15.6:0/2556216804 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47741f59e0 con 0x7f47740af580 2024-03-04T09:28:56.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.047+0000 7f4778c00640 1 -- 172.21.15.6:0/2556216804 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f477404f3a0 con 0x7f47740af580 2024-03-04T09:28:56.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.048+0000 7f476a000640 1 -- 172.21.15.6:0/2556216804 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f475c038020 con 0x7f47740af580 2024-03-04T09:28:56.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.049+0000 7f476a000640 1 --2- 172.21.15.6:0/2556216804 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f475006b4d0 0x7f475006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:56.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.049+0000 7f476a000640 1 -- 172.21.15.6:0/2556216804 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f475c0abf20 con 0x7f47740af580 2024-03-04T09:28:56.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.050+0000 7f4771e00640 1 --2- 172.21.15.6:0/2556216804 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f475006b4d0 0x7f475006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:56.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.051+0000 7f4771e00640 1 --2- 172.21.15.6:0/2556216804 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f475006b4d0 0x7f475006d990 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f47640011e0 tx=0x7f4764009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:56.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.054+0000 7f476a000640 1 -- 172.21.15.6:0/2556216804 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f475c040070 con 0x7f47740af580 2024-03-04T09:28:56.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.250+0000 7f4778c00640 1 -- 172.21.15.6:0/2556216804 --> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f47740ce210 con 0x7f475006b4d0 2024-03-04T09:28:56.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.268+0000 7f476a000640 1 -- 172.21.15.6:0/2556216804 <== mgr.4113 v2:172.21.15.146:6832/1788613334 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252066 (secure 0 0 0) 0x7f47740ce210 con 0x7f475006b4d0 2024-03-04T09:28:56.269 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:28:56.270 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-03-04T09:28:56.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.275+0000 7f4778c00640 1 -- 172.21.15.6:0/2556216804 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f475006b4d0 msgr2=0x7f475006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:56.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.275+0000 7f4778c00640 1 --2- 172.21.15.6:0/2556216804 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f475006b4d0 0x7f475006d990 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f47640011e0 tx=0x7f4764009040 comp rx=0 tx=0).stop 2024-03-04T09:28:56.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.275+0000 7f4778c00640 1 -- 172.21.15.6:0/2556216804 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f47740af580 msgr2=0x7f4774143500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:56.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.275+0000 7f4778c00640 1 --2- 172.21.15.6:0/2556216804 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f47740af580 0x7f4774143500 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f475c000c00 tx=0x7f475c0046c0 comp rx=0 tx=0).stop 2024-03-04T09:28:56.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.275+0000 7f4778c00640 1 -- 172.21.15.6:0/2556216804 shutdown_connections 2024-03-04T09:28:56.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.275+0000 7f4778c00640 1 --2- 172.21.15.6:0/2556216804 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f475006b4d0 0x7f475006d990 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:56.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.275+0000 7f4778c00640 1 --2- 172.21.15.6:0/2556216804 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f47740cdc20 0x7f47740b0fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:56.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.275+0000 7f4778c00640 1 --2- 172.21.15.6:0/2556216804 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f47740af580 0x7f4774143500 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:56.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.275+0000 7f4778c00640 1 --2- 172.21.15.6:0/2556216804 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f47740af1a0 0x7f47741563b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:56.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.275+0000 7f4778c00640 1 -- 172.21.15.6:0/2556216804 >> 172.21.15.6:0/2556216804 conn(0x7f477406afb0 msgr2=0x7f4774155420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:56.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.275+0000 7f4778c00640 1 -- 172.21.15.6:0/2556216804 shutdown_connections 2024-03-04T09:28:56.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.275+0000 7f4778c00640 1 -- 172.21.15.6:0/2556216804 wait complete. 2024-03-04T09:28:56.292 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":175,"stamp":"2024-03-04T09:28:55.936556+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":119,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":257096,"kb_used_data":7628,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754717624,"statfs":{"total":773094113280,"available":772830846976,"internally_reserved":0,"allocated":7811072,"data_stored":3930616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002160"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.278157+0000","last_change":"2024-03-04T09:28:51.291006+0000","last_active":"2024-03-04T09:28:52.278157+0000","last_peered":"2024-03-04T09:28:52.278157+0000","last_clean":"2024-03-04T09:28:52.278157+0000","last_became_active":"2024-03-04T09:28:51.288593+0000","last_became_peered":"2024-03-04T09:28:51.288593+0000","last_unstale":"2024-03-04T09:28:52.278157+0000","last_undegraded":"2024-03-04T09:28:52.278157+0000","last_fullsized":"2024-03-04T09:28:52.278157+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:50:26.619245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.846313+0000","last_change":"2024-03-04T09:28:51.290989+0000","last_active":"2024-03-04T09:28:52.846313+0000","last_peered":"2024-03-04T09:28:52.846313+0000","last_clean":"2024-03-04T09:28:52.846313+0000","last_became_active":"2024-03-04T09:28:51.285332+0000","last_became_peered":"2024-03-04T09:28:51.285332+0000","last_unstale":"2024-03-04T09:28:52.846313+0000","last_undegraded":"2024-03-04T09:28:52.846313+0000","last_fullsized":"2024-03-04T09:28:52.846313+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:42:54.712273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.846662+0000","last_change":"2024-03-04T09:28:51.277640+0000","last_active":"2024-03-04T09:28:52.846662+0000","last_peered":"2024-03-04T09:28:52.846662+0000","last_clean":"2024-03-04T09:28:52.846662+0000","last_became_active":"2024-03-04T09:28:51.276654+0000","last_became_peered":"2024-03-04T09:28:51.276654+0000","last_unstale":"2024-03-04T09:28:52.846662+0000","last_undegraded":"2024-03-04T09:28:52.846662+0000","last_fullsized":"2024-03-04T09:28:52.846662+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:40:06.464005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.275291+0000","last_change":"2024-03-04T09:28:51.287933+0000","last_active":"2024-03-04T09:28:52.275291+0000","last_peered":"2024-03-04T09:28:52.275291+0000","last_clean":"2024-03-04T09:28:52.275291+0000","last_became_active":"2024-03-04T09:28:51.285522+0000","last_became_peered":"2024-03-04T09:28:51.285522+0000","last_unstale":"2024-03-04T09:28:52.275291+0000","last_undegraded":"2024-03-04T09:28:52.275291+0000","last_fullsized":"2024-03-04T09:28:52.275291+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:35:10.446104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.278228+0000","last_change":"2024-03-04T09:28:51.274879+0000","last_active":"2024-03-04T09:28:52.278228+0000","last_peered":"2024-03-04T09:28:52.278228+0000","last_clean":"2024-03-04T09:28:52.278228+0000","last_became_active":"2024-03-04T09:28:51.272881+0000","last_became_peered":"2024-03-04T09:28:51.272881+0000","last_unstale":"2024-03-04T09:28:52.278228+0000","last_undegraded":"2024-03-04T09:28:52.278228+0000","last_fullsized":"2024-03-04T09:28:52.278228+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:17:20.224241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.845696+0000","last_change":"2024-03-04T09:28:51.285099+0000","last_active":"2024-03-04T09:28:52.845696+0000","last_peered":"2024-03-04T09:28:52.845696+0000","last_clean":"2024-03-04T09:28:52.845696+0000","last_became_active":"2024-03-04T09:28:51.284032+0000","last_became_peered":"2024-03-04T09:28:51.284032+0000","last_unstale":"2024-03-04T09:28:52.845696+0000","last_undegraded":"2024-03-04T09:28:52.845696+0000","last_fullsized":"2024-03-04T09:28:52.845696+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:52:58.620617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.277892+0000","last_change":"2024-03-04T09:28:51.284413+0000","last_active":"2024-03-04T09:28:52.277892+0000","last_peered":"2024-03-04T09:28:52.277892+0000","last_clean":"2024-03-04T09:28:52.277892+0000","last_became_active":"2024-03-04T09:28:51.284108+0000","last_became_peered":"2024-03-04T09:28:51.284108+0000","last_unstale":"2024-03-04T09:28:52.277892+0000","last_undegraded":"2024-03-04T09:28:52.277892+0000","last_fullsized":"2024-03-04T09:28:52.277892+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:30:16.400951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.845944+0000","last_change":"2024-03-04T09:28:51.299723+0000","last_active":"2024-03-04T09:28:52.845944+0000","last_peered":"2024-03-04T09:28:52.845944+0000","last_clean":"2024-03-04T09:28:52.845944+0000","last_became_active":"2024-03-04T09:28:51.298902+0000","last_became_peered":"2024-03-04T09:28:51.298902+0000","last_unstale":"2024-03-04T09:28:52.845944+0000","last_undegraded":"2024-03-04T09:28:52.845944+0000","last_fullsized":"2024-03-04T09:28:52.845944+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:46:14.770359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.278820+0000","last_change":"2024-03-04T09:28:51.278381+0000","last_active":"2024-03-04T09:28:52.278820+0000","last_peered":"2024-03-04T09:28:52.278820+0000","last_clean":"2024-03-04T09:28:52.278820+0000","last_became_active":"2024-03-04T09:28:51.277596+0000","last_became_peered":"2024-03-04T09:28:51.277596+0000","last_unstale":"2024-03-04T09:28:52.278820+0000","last_undegraded":"2024-03-04T09:28:52.278820+0000","last_fullsized":"2024-03-04T09:28:52.278820+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:06:29.423392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.847179+0000","last_change":"2024-03-04T09:28:51.287253+0000","last_active":"2024-03-04T09:28:52.847179+0000","last_peered":"2024-03-04T09:28:52.847179+0000","last_clean":"2024-03-04T09:28:52.847179+0000","last_became_active":"2024-03-04T09:28:51.286981+0000","last_became_peered":"2024-03-04T09:28:51.286981+0000","last_unstale":"2024-03-04T09:28:52.847179+0000","last_undegraded":"2024-03-04T09:28:52.847179+0000","last_fullsized":"2024-03-04T09:28:52.847179+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:04:23.011844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.279565+0000","last_change":"2024-03-04T09:28:51.291266+0000","last_active":"2024-03-04T09:28:52.279565+0000","last_peered":"2024-03-04T09:28:52.279565+0000","last_clean":"2024-03-04T09:28:52.279565+0000","last_became_active":"2024-03-04T09:28:51.289515+0000","last_became_peered":"2024-03-04T09:28:51.289515+0000","last_unstale":"2024-03-04T09:28:52.279565+0000","last_undegraded":"2024-03-04T09:28:52.279565+0000","last_fullsized":"2024-03-04T09:28:52.279565+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:41:40.928565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.275461+0000","last_change":"2024-03-04T09:28:51.271412+0000","last_active":"2024-03-04T09:28:52.275461+0000","last_peered":"2024-03-04T09:28:52.275461+0000","last_clean":"2024-03-04T09:28:52.275461+0000","last_became_active":"2024-03-04T09:28:51.271117+0000","last_became_peered":"2024-03-04T09:28:51.271117+0000","last_unstale":"2024-03-04T09:28:52.275461+0000","last_undegraded":"2024-03-04T09:28:52.275461+0000","last_fullsized":"2024-03-04T09:28:52.275461+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:39:19.241679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.283815+0000","last_change":"2024-03-04T09:28:51.292356+0000","last_active":"2024-03-04T09:28:52.283815+0000","last_peered":"2024-03-04T09:28:52.283815+0000","last_clean":"2024-03-04T09:28:52.283815+0000","last_became_active":"2024-03-04T09:28:51.292029+0000","last_became_peered":"2024-03-04T09:28:51.292029+0000","last_unstale":"2024-03-04T09:28:52.283815+0000","last_undegraded":"2024-03-04T09:28:52.283815+0000","last_fullsized":"2024-03-04T09:28:52.283815+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:46:08.414226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.278005+0000","last_change":"2024-03-04T09:28:51.290626+0000","last_active":"2024-03-04T09:28:52.278005+0000","last_peered":"2024-03-04T09:28:52.278005+0000","last_clean":"2024-03-04T09:28:52.278005+0000","last_became_active":"2024-03-04T09:28:51.290330+0000","last_became_peered":"2024-03-04T09:28:51.290330+0000","last_unstale":"2024-03-04T09:28:52.278005+0000","last_undegraded":"2024-03-04T09:28:52.278005+0000","last_fullsized":"2024-03-04T09:28:52.278005+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:41:27.692459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.283742+0000","last_change":"2024-03-04T09:28:51.280290+0000","last_active":"2024-03-04T09:28:52.283742+0000","last_peered":"2024-03-04T09:28:52.283742+0000","last_clean":"2024-03-04T09:28:52.283742+0000","last_became_active":"2024-03-04T09:28:51.279939+0000","last_became_peered":"2024-03-04T09:28:51.279939+0000","last_unstale":"2024-03-04T09:28:52.283742+0000","last_undegraded":"2024-03-04T09:28:52.283742+0000","last_fullsized":"2024-03-04T09:28:52.283742+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:31:45.267982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.846322+0000","last_change":"2024-03-04T09:28:51.300526+0000","last_active":"2024-03-04T09:28:52.846322+0000","last_peered":"2024-03-04T09:28:52.846322+0000","last_clean":"2024-03-04T09:28:52.846322+0000","last_became_active":"2024-03-04T09:28:51.299888+0000","last_became_peered":"2024-03-04T09:28:51.299888+0000","last_unstale":"2024-03-04T09:28:52.846322+0000","last_undegraded":"2024-03-04T09:28:52.846322+0000","last_fullsized":"2024-03-04T09:28:52.846322+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:43:12.750957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.278898+0000","last_change":"2024-03-04T09:28:51.278151+0000","last_active":"2024-03-04T09:28:52.278898+0000","last_peered":"2024-03-04T09:28:52.278898+0000","last_clean":"2024-03-04T09:28:52.278898+0000","last_became_active":"2024-03-04T09:28:51.277223+0000","last_became_peered":"2024-03-04T09:28:51.277223+0000","last_unstale":"2024-03-04T09:28:52.278898+0000","last_undegraded":"2024-03-04T09:28:52.278898+0000","last_fullsized":"2024-03-04T09:28:52.278898+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:19:10.456432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.275656+0000","last_change":"2024-03-04T09:28:51.289240+0000","last_active":"2024-03-04T09:28:52.275656+0000","last_peered":"2024-03-04T09:28:52.275656+0000","last_clean":"2024-03-04T09:28:52.275656+0000","last_became_active":"2024-03-04T09:28:51.285104+0000","last_became_peered":"2024-03-04T09:28:51.285104+0000","last_unstale":"2024-03-04T09:28:52.275656+0000","last_undegraded":"2024-03-04T09:28:52.275656+0000","last_fullsized":"2024-03-04T09:28:52.275656+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:37:28.918787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.846054+0000","last_change":"2024-03-04T09:28:51.284530+0000","last_active":"2024-03-04T09:28:52.846054+0000","last_peered":"2024-03-04T09:28:52.846054+0000","last_clean":"2024-03-04T09:28:52.846054+0000","last_became_active":"2024-03-04T09:28:51.284234+0000","last_became_peered":"2024-03-04T09:28:51.284234+0000","last_unstale":"2024-03-04T09:28:52.846054+0000","last_undegraded":"2024-03-04T09:28:52.846054+0000","last_fullsized":"2024-03-04T09:28:52.846054+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:55:37.556856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.284073+0000","last_change":"2024-03-04T09:28:51.287316+0000","last_active":"2024-03-04T09:28:52.284073+0000","last_peered":"2024-03-04T09:28:52.284073+0000","last_clean":"2024-03-04T09:28:52.284073+0000","last_became_active":"2024-03-04T09:28:51.286937+0000","last_became_peered":"2024-03-04T09:28:51.286937+0000","last_unstale":"2024-03-04T09:28:52.284073+0000","last_undegraded":"2024-03-04T09:28:52.284073+0000","last_fullsized":"2024-03-04T09:28:52.284073+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:14:24.426848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.846943+0000","last_change":"2024-03-04T09:28:51.279950+0000","last_active":"2024-03-04T09:28:52.846943+0000","last_peered":"2024-03-04T09:28:52.846943+0000","last_clean":"2024-03-04T09:28:52.846943+0000","last_became_active":"2024-03-04T09:28:51.279628+0000","last_became_peered":"2024-03-04T09:28:51.279628+0000","last_unstale":"2024-03-04T09:28:52.846943+0000","last_undegraded":"2024-03-04T09:28:52.846943+0000","last_fullsized":"2024-03-04T09:28:52.846943+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:40:44.950933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.846744+0000","last_change":"2024-03-04T09:28:51.287505+0000","last_active":"2024-03-04T09:28:52.846744+0000","last_peered":"2024-03-04T09:28:52.846744+0000","last_clean":"2024-03-04T09:28:52.846744+0000","last_became_active":"2024-03-04T09:28:51.286364+0000","last_became_peered":"2024-03-04T09:28:51.286364+0000","last_unstale":"2024-03-04T09:28:52.846744+0000","last_undegraded":"2024-03-04T09:28:52.846744+0000","last_fullsized":"2024-03-04T09:28:52.846744+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:50:21.773645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.277427+0000","last_change":"2024-03-04T09:28:51.293297+0000","last_active":"2024-03-04T09:28:52.277427+0000","last_peered":"2024-03-04T09:28:52.277427+0000","last_clean":"2024-03-04T09:28:52.277427+0000","last_became_active":"2024-03-04T09:28:51.292187+0000","last_became_peered":"2024-03-04T09:28:51.292187+0000","last_unstale":"2024-03-04T09:28:52.277427+0000","last_undegraded":"2024-03-04T09:28:52.277427+0000","last_fullsized":"2024-03-04T09:28:52.277427+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:14:24.426848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.277338+0000","last_change":"2024-03-04T09:28:51.295074+0000","last_active":"2024-03-04T09:28:52.277338+0000","last_peered":"2024-03-04T09:28:52.277338+0000","last_clean":"2024-03-04T09:28:52.277338+0000","last_became_active":"2024-03-04T09:28:51.294173+0000","last_became_peered":"2024-03-04T09:28:51.294173+0000","last_unstale":"2024-03-04T09:28:52.277338+0000","last_undegraded":"2024-03-04T09:28:52.277338+0000","last_fullsized":"2024-03-04T09:28:52.277338+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:46:08.414226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.847212+0000","last_change":"2024-03-04T09:28:51.304555+0000","last_active":"2024-03-04T09:28:52.847212+0000","last_peered":"2024-03-04T09:28:52.847212+0000","last_clean":"2024-03-04T09:28:52.847212+0000","last_became_active":"2024-03-04T09:28:51.304180+0000","last_became_peered":"2024-03-04T09:28:51.304180+0000","last_unstale":"2024-03-04T09:28:52.847212+0000","last_undegraded":"2024-03-04T09:28:52.847212+0000","last_fullsized":"2024-03-04T09:28:52.847212+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:30:16.400951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.278678+0000","last_change":"2024-03-04T09:28:51.302858+0000","last_active":"2024-03-04T09:28:52.278678+0000","last_peered":"2024-03-04T09:28:52.278678+0000","last_clean":"2024-03-04T09:28:52.278678+0000","last_became_active":"2024-03-04T09:28:51.302536+0000","last_became_peered":"2024-03-04T09:28:51.302536+0000","last_unstale":"2024-03-04T09:28:52.278678+0000","last_undegraded":"2024-03-04T09:28:52.278678+0000","last_fullsized":"2024-03-04T09:28:52.278678+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:46:08.414226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.846915+0000","last_change":"2024-03-04T09:28:51.283168+0000","last_active":"2024-03-04T09:28:52.846915+0000","last_peered":"2024-03-04T09:28:52.846915+0000","last_clean":"2024-03-04T09:28:52.846915+0000","last_became_active":"2024-03-04T09:28:51.282883+0000","last_became_peered":"2024-03-04T09:28:51.282883+0000","last_unstale":"2024-03-04T09:28:52.846915+0000","last_undegraded":"2024-03-04T09:28:52.846915+0000","last_fullsized":"2024-03-04T09:28:52.846915+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:30:19.523187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.846945+0000","last_change":"2024-03-04T09:28:51.288242+0000","last_active":"2024-03-04T09:28:52.846945+0000","last_peered":"2024-03-04T09:28:52.846945+0000","last_clean":"2024-03-04T09:28:52.846945+0000","last_became_active":"2024-03-04T09:28:51.287545+0000","last_became_peered":"2024-03-04T09:28:51.287545+0000","last_unstale":"2024-03-04T09:28:52.846945+0000","last_undegraded":"2024-03-04T09:28:52.846945+0000","last_fullsized":"2024-03-04T09:28:52.846945+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:53:57.607241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.846835+0000","last_change":"2024-03-04T09:28:51.294887+0000","last_active":"2024-03-04T09:28:52.846835+0000","last_peered":"2024-03-04T09:28:52.846835+0000","last_clean":"2024-03-04T09:28:52.846835+0000","last_became_active":"2024-03-04T09:28:51.294549+0000","last_became_peered":"2024-03-04T09:28:51.294549+0000","last_unstale":"2024-03-04T09:28:52.846835+0000","last_undegraded":"2024-03-04T09:28:52.846835+0000","last_fullsized":"2024-03-04T09:28:52.846835+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:34:18.046888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.278977+0000","last_change":"2024-03-04T09:28:51.301289+0000","last_active":"2024-03-04T09:28:52.278977+0000","last_peered":"2024-03-04T09:28:52.278977+0000","last_clean":"2024-03-04T09:28:52.278977+0000","last_became_active":"2024-03-04T09:28:51.300987+0000","last_became_peered":"2024-03-04T09:28:51.300987+0000","last_unstale":"2024-03-04T09:28:52.278977+0000","last_undegraded":"2024-03-04T09:28:52.278977+0000","last_fullsized":"2024-03-04T09:28:52.278977+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:04:27.619250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.278577+0000","last_change":"2024-03-04T09:28:51.286243+0000","last_active":"2024-03-04T09:28:52.278577+0000","last_peered":"2024-03-04T09:28:52.278577+0000","last_clean":"2024-03-04T09:28:52.278577+0000","last_became_active":"2024-03-04T09:28:51.285945+0000","last_became_peered":"2024-03-04T09:28:51.285945+0000","last_unstale":"2024-03-04T09:28:52.278577+0000","last_undegraded":"2024-03-04T09:28:52.278577+0000","last_fullsized":"2024-03-04T09:28:52.278577+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:31:45.267982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.846871+0000","last_change":"2024-03-04T09:28:51.295866+0000","last_active":"2024-03-04T09:28:52.846871+0000","last_peered":"2024-03-04T09:28:52.846871+0000","last_clean":"2024-03-04T09:28:52.846871+0000","last_became_active":"2024-03-04T09:28:51.295588+0000","last_became_peered":"2024-03-04T09:28:51.295588+0000","last_unstale":"2024-03-04T09:28:52.846871+0000","last_undegraded":"2024-03-04T09:28:52.846871+0000","last_fullsized":"2024-03-04T09:28:52.846871+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:40:52.179638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.278119+0000","last_change":"2024-03-04T09:28:51.306539+0000","last_active":"2024-03-04T09:28:52.278119+0000","last_peered":"2024-03-04T09:28:52.278119+0000","last_clean":"2024-03-04T09:28:52.278119+0000","last_became_active":"2024-03-04T09:28:51.306207+0000","last_became_peered":"2024-03-04T09:28:51.306207+0000","last_unstale":"2024-03-04T09:28:52.278119+0000","last_undegraded":"2024-03-04T09:28:52.278119+0000","last_fullsized":"2024-03-04T09:28:52.278119+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:04:27.619250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1f","version":"67'4","reported_seq":18,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.278445+0000","last_change":"2024-03-04T09:28:50.256035+0000","last_active":"2024-03-04T09:28:52.278445+0000","last_peered":"2024-03-04T09:28:52.278445+0000","last_clean":"2024-03-04T09:28:52.278445+0000","last_became_active":"2024-03-04T09:28:50.255761+0000","last_became_peered":"2024-03-04T09:28:50.255761+0000","last_unstale":"2024-03-04T09:28:52.278445+0000","last_undegraded":"2024-03-04T09:28:52.278445+0000","last_fullsized":"2024-03-04T09:28:52.278445+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_clean_scrub_stamp":"2024-03-04T09:28:49.199103+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:47:44.767861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.279095+0000","last_change":"2024-03-04T09:28:51.277880+0000","last_active":"2024-03-04T09:28:52.279095+0000","last_peered":"2024-03-04T09:28:52.279095+0000","last_clean":"2024-03-04T09:28:52.279095+0000","last_became_active":"2024-03-04T09:28:51.276824+0000","last_became_peered":"2024-03-04T09:28:51.276824+0000","last_unstale":"2024-03-04T09:28:52.279095+0000","last_undegraded":"2024-03-04T09:28:52.279095+0000","last_fullsized":"2024-03-04T09:28:52.279095+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:39:35.069899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"67'1","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.284348+0000","last_change":"2024-03-04T09:28:50.241119+0000","last_active":"2024-03-04T09:28:52.284348+0000","last_peered":"2024-03-04T09:28:52.284348+0000","last_clean":"2024-03-04T09:28:52.284348+0000","last_became_active":"2024-03-04T09:28:50.240831+0000","last_became_peered":"2024-03-04T09:28:50.240831+0000","last_unstale":"2024-03-04T09:28:52.284348+0000","last_undegraded":"2024-03-04T09:28:52.284348+0000","last_fullsized":"2024-03-04T09:28:52.284348+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_clean_scrub_stamp":"2024-03-04T09:28:49.199103+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:43:11.736446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.846030+0000","last_change":"2024-03-04T09:28:51.307590+0000","last_active":"2024-03-04T09:28:52.846030+0000","last_peered":"2024-03-04T09:28:52.846030+0000","last_clean":"2024-03-04T09:28:52.846030+0000","last_became_active":"2024-03-04T09:28:51.306722+0000","last_became_peered":"2024-03-04T09:28:51.306722+0000","last_unstale":"2024-03-04T09:28:52.846030+0000","last_undegraded":"2024-03-04T09:28:52.846030+0000","last_fullsized":"2024-03-04T09:28:52.846030+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:13:39.526501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1d","version":"67'2","reported_seq":16,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.847107+0000","last_change":"2024-03-04T09:28:50.253226+0000","last_active":"2024-03-04T09:28:52.847107+0000","last_peered":"2024-03-04T09:28:52.847107+0000","last_clean":"2024-03-04T09:28:52.847107+0000","last_became_active":"2024-03-04T09:28:50.251509+0000","last_became_peered":"2024-03-04T09:28:50.251509+0000","last_unstale":"2024-03-04T09:28:52.847107+0000","last_undegraded":"2024-03-04T09:28:52.847107+0000","last_fullsized":"2024-03-04T09:28:52.847107+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_clean_scrub_stamp":"2024-03-04T09:28:49.199103+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:55:36.542345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.279770+0000","last_change":"2024-03-04T09:28:51.283593+0000","last_active":"2024-03-04T09:28:52.279770+0000","last_peered":"2024-03-04T09:28:52.279770+0000","last_clean":"2024-03-04T09:28:52.279770+0000","last_became_active":"2024-03-04T09:28:51.283286+0000","last_became_peered":"2024-03-04T09:28:51.283286+0000","last_unstale":"2024-03-04T09:28:52.279770+0000","last_undegraded":"2024-03-04T09:28:52.279770+0000","last_fullsized":"2024-03-04T09:28:52.279770+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:12:21.069693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1c","version":"67'2","reported_seq":16,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.846268+0000","last_change":"2024-03-04T09:28:50.250103+0000","last_active":"2024-03-04T09:28:52.846268+0000","last_peered":"2024-03-04T09:28:52.846268+0000","last_clean":"2024-03-04T09:28:52.846268+0000","last_became_active":"2024-03-04T09:28:50.249715+0000","last_became_peered":"2024-03-04T09:28:50.249715+0000","last_unstale":"2024-03-04T09:28:52.846268+0000","last_undegraded":"2024-03-04T09:28:52.846268+0000","last_fullsized":"2024-03-04T09:28:52.846268+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_clean_scrub_stamp":"2024-03-04T09:28:49.199103+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:42:39.759815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.279144+0000","last_change":"2024-03-04T09:28:51.290536+0000","last_active":"2024-03-04T09:28:52.279144+0000","last_peered":"2024-03-04T09:28:52.279144+0000","last_clean":"2024-03-04T09:28:52.279144+0000","last_became_active":"2024-03-04T09:28:51.290065+0000","last_became_peered":"2024-03-04T09:28:51.290065+0000","last_unstale":"2024-03-04T09:28:52.279144+0000","last_undegraded":"2024-03-04T09:28:52.279144+0000","last_fullsized":"2024-03-04T09:28:52.279144+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:50:26.619245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"67'1","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.847158+0000","last_change":"2024-03-04T09:28:50.250204+0000","last_active":"2024-03-04T09:28:52.847158+0000","last_peered":"2024-03-04T09:28:52.847158+0000","last_clean":"2024-03-04T09:28:52.847158+0000","last_became_active":"2024-03-04T09:28:50.247918+0000","last_became_peered":"2024-03-04T09:28:50.247918+0000","last_unstale":"2024-03-04T09:28:52.847158+0000","last_undegraded":"2024-03-04T09:28:52.847158+0000","last_fullsized":"2024-03-04T09:28:52.847158+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_clean_scrub_stamp":"2024-03-04T09:28:49.199103+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:59:04.470819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.847349+0000","last_change":"2024-03-04T09:28:51.287806+0000","last_active":"2024-03-04T09:28:52.847349+0000","last_peered":"2024-03-04T09:28:52.847349+0000","last_clean":"2024-03-04T09:28:52.847349+0000","last_became_active":"2024-03-04T09:28:51.282811+0000","last_became_peered":"2024-03-04T09:28:51.282811+0000","last_unstale":"2024-03-04T09:28:52.847349+0000","last_undegraded":"2024-03-04T09:28:52.847349+0000","last_fullsized":"2024-03-04T09:28:52.847349+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:31:45.267982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"67'1","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.845746+0000","last_change":"2024-03-04T09:28:50.255231+0000","last_active":"2024-03-04T09:28:52.845746+0000","last_peered":"2024-03-04T09:28:52.845746+0000","last_clean":"2024-03-04T09:28:52.845746+0000","last_became_active":"2024-03-04T09:28:50.254623+0000","last_became_peered":"2024-03-04T09:28:50.254623+0000","last_unstale":"2024-03-04T09:28:52.845746+0000","last_undegraded":"2024-03-04T09:28:52.845746+0000","last_fullsized":"2024-03-04T09:28:52.845746+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_clean_scrub_stamp":"2024-03-04T09:28:49.199103+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:19:09.441921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.277684+0000","last_change":"2024-03-04T09:28:51.277060+0000","last_active":"2024-03-04T09:28:52.277684+0000","last_peered":"2024-03-04T09:28:52.277684+0000","last_clean":"2024-03-04T09:28:52.277684+0000","last_became_active":"2024-03-04T09:28:51.276220+0000","last_became_peered":"2024-03-04T09:28:51.276220+0000","last_unstale":"2024-03-04T09:28:52.277684+0000","last_undegraded":"2024-03-04T09:28:52.277684+0000","last_fullsized":"2024-03-04T09:28:52.277684+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:26:47.680959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.274443+0000","last_change":"2024-03-04T09:28:50.242172+0000","last_active":"2024-03-04T09:28:52.274443+0000","last_peered":"2024-03-04T09:28:52.274443+0000","last_clean":"2024-03-04T09:28:52.274443+0000","last_became_active":"2024-03-04T09:28:50.241796+0000","last_became_peered":"2024-03-04T09:28:50.241796+0000","last_unstale":"2024-03-04T09:28:52.274443+0000","last_undegraded":"2024-03-04T09:28:52.274443+0000","last_fullsized":"2024-03-04T09:28:52.274443+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_clean_scrub_stamp":"2024-03-04T09:28:49.199103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:19:09.441921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.284476+0000","last_change":"2024-03-04T09:28:51.282196+0000","last_active":"2024-03-04T09:28:52.284476+0000","last_peered":"2024-03-04T09:28:52.284476+0000","last_clean":"2024-03-04T09:28:52.284476+0000","last_became_active":"2024-03-04T09:28:51.281804+0000","last_became_peered":"2024-03-04T09:28:51.281804+0000","last_unstale":"2024-03-04T09:28:52.284476+0000","last_undegraded":"2024-03-04T09:28:52.284476+0000","last_fullsized":"2024-03-04T09:28:52.284476+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:32:02.177278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.277678+0000","last_change":"2024-03-04T09:28:50.242171+0000","last_active":"2024-03-04T09:28:52.277678+0000","last_peered":"2024-03-04T09:28:52.277678+0000","last_clean":"2024-03-04T09:28:52.277678+0000","last_became_active":"2024-03-04T09:28:50.240397+0000","last_became_peered":"2024-03-04T09:28:50.240397+0000","last_unstale":"2024-03-04T09:28:52.277678+0000","last_undegraded":"2024-03-04T09:28:52.277678+0000","last_fullsized":"2024-03-04T09:28:52.277678+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_clean_scrub_stamp":"2024-03-04T09:28:49.199103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:27:14.886296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.278805+0000","last_change":"2024-03-04T09:28:51.287557+0000","last_active":"2024-03-04T09:28:52.278805+0000","last_peered":"2024-03-04T09:28:52.278805+0000","last_clean":"2024-03-04T09:28:52.278805+0000","last_became_active":"2024-03-04T09:28:51.287220+0000","last_became_peered":"2024-03-04T09:28:51.287220+0000","last_unstale":"2024-03-04T09:28:52.278805+0000","last_undegraded":"2024-03-04T09:28:52.278805+0000","last_fullsized":"2024-03-04T09:28:52.278805+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:14:24.426848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.847255+0000","last_change":"2024-03-04T09:28:50.242957+0000","last_active":"2024-03-04T09:28:52.847255+0000","last_peered":"2024-03-04T09:28:52.847255+0000","last_clean":"2024-03-04T09:28:52.847255+0000","last_became_active":"2024-03-04T09:28:50.242093+0000","last_became_peered":"2024-03-04T09:28:50.242093+0000","last_unstale":"2024-03-04T09:28:52.847255+0000","last_undegraded":"2024-03-04T09:28:52.847255+0000","last_fullsized":"2024-03-04T09:28:52.847255+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_clean_scrub_stamp":"2024-03-04T09:28:49.199103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:13:38.511990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.278247+0000","last_change":"2024-03-04T09:28:51.294474+0000","last_active":"2024-03-04T09:28:52.278247+0000","last_peered":"2024-03-04T09:28:52.278247+0000","last_clean":"2024-03-04T09:28:52.278247+0000","last_became_active":"2024-03-04T09:28:51.294105+0000","last_became_peered":"2024-03-04T09:28:51.294105+0000","last_unstale":"2024-03-04T09:28:52.278247+0000","last_undegraded":"2024-03-04T09:28:52.278247+0000","last_fullsized":"2024-03-04T09:28:52.278247+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:32:02.177278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.846284+0000","last_change":"2024-03-04T09:28:50.259691+0000","last_active":"2024-03-04T09:28:52.846284+0000","last_peered":"2024-03-04T09:28:52.846284+0000","last_clean":"2024-03-04T09:28:52.846284+0000","last_became_active":"2024-03-04T09:28:50.259417+0000","last_became_peered":"2024-03-04T09:28:50.259417+0000","last_unstale":"2024-03-04T09:28:52.846284+0000","last_undegraded":"2024-03-04T09:28:52.846284+0000","last_fullsized":"2024-03-04T09:28:52.846284+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_clean_scrub_stamp":"2024-03-04T09:28:49.199103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:06:28.408881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.846811+0000","last_change":"2024-03-04T09:28:51.287996+0000","last_active":"2024-03-04T09:28:52.846811+0000","last_peered":"2024-03-04T09:28:52.846811+0000","last_clean":"2024-03-04T09:28:52.846811+0000","last_became_active":"2024-03-04T09:28:51.287037+0000","last_became_peered":"2024-03-04T09:28:51.287037+0000","last_unstale":"2024-03-04T09:28:52.846811+0000","last_undegraded":"2024-03-04T09:28:52.846811+0000","last_fullsized":"2024-03-04T09:28:52.846811+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:35:08.303547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"67'3","reported_seq":17,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.847131+0000","last_change":"2024-03-04T09:28:50.258362+0000","last_active":"2024-03-04T09:28:52.847131+0000","last_peered":"2024-03-04T09:28:52.847131+0000","last_clean":"2024-03-04T09:28:52.847131+0000","last_became_active":"2024-03-04T09:28:50.257859+0000","last_became_peered":"2024-03-04T09:28:50.257859+0000","last_unstale":"2024-03-04T09:28:52.847131+0000","last_undegraded":"2024-03-04T09:28:52.847131+0000","last_fullsized":"2024-03-04T09:28:52.847131+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_clean_scrub_stamp":"2024-03-04T09:28:49.199103+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:57:09.368049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.278671+0000","last_change":"2024-03-04T09:28:51.279718+0000","last_active":"2024-03-04T09:28:52.278671+0000","last_peered":"2024-03-04T09:28:52.278671+0000","last_clean":"2024-03-04T09:28:52.278671+0000","last_became_active":"2024-03-04T09:28:51.279420+0000","last_became_peered":"2024-03-04T09:28:51.279420+0000","last_unstale":"2024-03-04T09:28:52.278671+0000","last_undegraded":"2024-03-04T09:28:52.278671+0000","last_fullsized":"2024-03-04T09:28:52.278671+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:31:29.139378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"67'6","reported_seq":20,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.275693+0000","last_change":"2024-03-04T09:28:50.245830+0000","last_active":"2024-03-04T09:28:52.275693+0000","last_peered":"2024-03-04T09:28:52.275693+0000","last_clean":"2024-03-04T09:28:52.275693+0000","last_became_active":"2024-03-04T09:28:50.245528+0000","last_became_peered":"2024-03-04T09:28:50.245528+0000","last_unstale":"2024-03-04T09:28:52.275693+0000","last_undegraded":"2024-03-04T09:28:52.275693+0000","last_fullsized":"2024-03-04T09:28:52.275693+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_clean_scrub_stamp":"2024-03-04T09:28:49.199103+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:18:20.649121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.278728+0000","last_change":"2024-03-04T09:28:51.285212+0000","last_active":"2024-03-04T09:28:52.278728+0000","last_peered":"2024-03-04T09:28:52.278728+0000","last_clean":"2024-03-04T09:28:52.278728+0000","last_became_active":"2024-03-04T09:28:51.284935+0000","last_became_peered":"2024-03-04T09:28:51.284935+0000","last_unstale":"2024-03-04T09:28:52.278728+0000","last_undegraded":"2024-03-04T09:28:52.278728+0000","last_fullsized":"2024-03-04T09:28:52.278728+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:00:55.647309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"67'2","reported_seq":16,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.275017+0000","last_change":"2024-03-04T09:28:50.253940+0000","last_active":"2024-03-04T09:28:52.275017+0000","last_peered":"2024-03-04T09:28:52.275017+0000","last_clean":"2024-03-04T09:28:52.275017+0000","last_became_active":"2024-03-04T09:28:50.253652+0000","last_became_peered":"2024-03-04T09:28:50.253652+0000","last_unstale":"2024-03-04T09:28:52.275017+0000","last_undegraded":"2024-03-04T09:28:52.275017+0000","last_fullsized":"2024-03-04T09:28:52.275017+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_clean_scrub_stamp":"2024-03-04T09:28:49.199103+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:42:50.437322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.277705+0000","last_change":"2024-03-04T09:28:51.288724+0000","last_active":"2024-03-04T09:28:52.277705+0000","last_peered":"2024-03-04T09:28:52.277705+0000","last_clean":"2024-03-04T09:28:52.277705+0000","last_became_active":"2024-03-04T09:28:51.287319+0000","last_became_peered":"2024-03-04T09:28:51.287319+0000","last_unstale":"2024-03-04T09:28:52.277705+0000","last_undegraded":"2024-03-04T09:28:52.277705+0000","last_fullsized":"2024-03-04T09:28:52.277705+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:04:23.011844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.845783+0000","last_change":"2024-03-04T09:28:50.258623+0000","last_active":"2024-03-04T09:28:52.845783+0000","last_peered":"2024-03-04T09:28:52.845783+0000","last_clean":"2024-03-04T09:28:52.845783+0000","last_became_active":"2024-03-04T09:28:50.256343+0000","last_became_peered":"2024-03-04T09:28:50.256343+0000","last_unstale":"2024-03-04T09:28:52.845783+0000","last_undegraded":"2024-03-04T09:28:52.845783+0000","last_fullsized":"2024-03-04T09:28:52.845783+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_clean_scrub_stamp":"2024-03-04T09:28:49.199103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:39:34.055388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.284096+0000","last_change":"2024-03-04T09:28:51.293951+0000","last_active":"2024-03-04T09:28:52.284096+0000","last_peered":"2024-03-04T09:28:52.284096+0000","last_clean":"2024-03-04T09:28:52.284096+0000","last_became_active":"2024-03-04T09:28:51.292713+0000","last_became_peered":"2024-03-04T09:28:51.292713+0000","last_unstale":"2024-03-04T09:28:52.284096+0000","last_undegraded":"2024-03-04T09:28:52.284096+0000","last_fullsized":"2024-03-04T09:28:52.284096+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:04:27.619250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.278154+0000","last_change":"2024-03-04T09:28:50.254527+0000","last_active":"2024-03-04T09:28:52.278154+0000","last_peered":"2024-03-04T09:28:52.278154+0000","last_clean":"2024-03-04T09:28:52.278154+0000","last_became_active":"2024-03-04T09:28:50.254264+0000","last_became_peered":"2024-03-04T09:28:50.254264+0000","last_unstale":"2024-03-04T09:28:52.278154+0000","last_undegraded":"2024-03-04T09:28:52.278154+0000","last_fullsized":"2024-03-04T09:28:52.278154+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_clean_scrub_stamp":"2024-03-04T09:28:49.199103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:40:51.165127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.277887+0000","last_change":"2024-03-04T09:28:51.271877+0000","last_active":"2024-03-04T09:28:52.277887+0000","last_peered":"2024-03-04T09:28:52.277887+0000","last_clean":"2024-03-04T09:28:52.277887+0000","last_became_active":"2024-03-04T09:28:51.270973+0000","last_became_peered":"2024-03-04T09:28:51.270973+0000","last_unstale":"2024-03-04T09:28:52.277887+0000","last_undegraded":"2024-03-04T09:28:52.277887+0000","last_fullsized":"2024-03-04T09:28:52.277887+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:46:30.848164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"67'8","reported_seq":22,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.846640+0000","last_change":"2024-03-04T09:28:50.253971+0000","last_active":"2024-03-04T09:28:52.846640+0000","last_peered":"2024-03-04T09:28:52.846640+0000","last_clean":"2024-03-04T09:28:52.846640+0000","last_became_active":"2024-03-04T09:28:50.253687+0000","last_became_peered":"2024-03-04T09:28:50.253687+0000","last_unstale":"2024-03-04T09:28:52.846640+0000","last_undegraded":"2024-03-04T09:28:52.846640+0000","last_fullsized":"2024-03-04T09:28:52.846640+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_clean_scrub_stamp":"2024-03-04T09:28:49.199103+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:46:13.755848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.847263+0000","last_change":"2024-03-04T09:28:51.279313+0000","last_active":"2024-03-04T09:28:52.847263+0000","last_peered":"2024-03-04T09:28:52.847263+0000","last_clean":"2024-03-04T09:28:52.847263+0000","last_became_active":"2024-03-04T09:28:51.278626+0000","last_became_peered":"2024-03-04T09:28:51.278626+0000","last_unstale":"2024-03-04T09:28:52.847263+0000","last_undegraded":"2024-03-04T09:28:52.847263+0000","last_fullsized":"2024-03-04T09:28:52.847263+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:55:12.363630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"67'1","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.846011+0000","last_change":"2024-03-04T09:28:50.252588+0000","last_active":"2024-03-04T09:28:52.846011+0000","last_peered":"2024-03-04T09:28:52.846011+0000","last_clean":"2024-03-04T09:28:52.846011+0000","last_became_active":"2024-03-04T09:28:50.252321+0000","last_became_peered":"2024-03-04T09:28:50.252321+0000","last_unstale":"2024-03-04T09:28:52.846011+0000","last_undegraded":"2024-03-04T09:28:52.846011+0000","last_fullsized":"2024-03-04T09:28:52.846011+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_clean_scrub_stamp":"2024-03-04T09:28:49.199103+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:17:59.355079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.278368+0000","last_change":"2024-03-04T09:28:51.306663+0000","last_active":"2024-03-04T09:28:52.278368+0000","last_peered":"2024-03-04T09:28:52.278368+0000","last_clean":"2024-03-04T09:28:52.278368+0000","last_became_active":"2024-03-04T09:28:51.305680+0000","last_became_peered":"2024-03-04T09:28:51.305680+0000","last_unstale":"2024-03-04T09:28:52.278368+0000","last_undegraded":"2024-03-04T09:28:52.278368+0000","last_fullsized":"2024-03-04T09:28:52.278368+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:10:47.851740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"67'2","reported_seq":16,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.847270+0000","last_change":"2024-03-04T09:28:50.250228+0000","last_active":"2024-03-04T09:28:52.847270+0000","last_peered":"2024-03-04T09:28:52.847270+0000","last_clean":"2024-03-04T09:28:52.847270+0000","last_became_active":"2024-03-04T09:28:50.249912+0000","last_became_peered":"2024-03-04T09:28:50.249912+0000","last_unstale":"2024-03-04T09:28:52.847270+0000","last_undegraded":"2024-03-04T09:28:52.847270+0000","last_fullsized":"2024-03-04T09:28:52.847270+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_clean_scrub_stamp":"2024-03-04T09:28:49.199103+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:42:53.697762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.275659+0000","last_change":"2024-03-04T09:28:51.273490+0000","last_active":"2024-03-04T09:28:52.275659+0000","last_peered":"2024-03-04T09:28:52.275659+0000","last_clean":"2024-03-04T09:28:52.275659+0000","last_became_active":"2024-03-04T09:28:51.272524+0000","last_became_peered":"2024-03-04T09:28:51.272524+0000","last_unstale":"2024-03-04T09:28:52.275659+0000","last_undegraded":"2024-03-04T09:28:52.275659+0000","last_fullsized":"2024-03-04T09:28:52.275659+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:59:43.290460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"67'1","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.846158+0000","last_change":"2024-03-04T09:28:50.248288+0000","last_active":"2024-03-04T09:28:52.846158+0000","last_peered":"2024-03-04T09:28:52.846158+0000","last_clean":"2024-03-04T09:28:52.846158+0000","last_became_active":"2024-03-04T09:28:50.247795+0000","last_became_peered":"2024-03-04T09:28:50.247795+0000","last_unstale":"2024-03-04T09:28:52.846158+0000","last_undegraded":"2024-03-04T09:28:52.846158+0000","last_fullsized":"2024-03-04T09:28:52.846158+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_clean_scrub_stamp":"2024-03-04T09:28:49.199103+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:17:19.209731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.278661+0000","last_change":"2024-03-04T09:28:51.282596+0000","last_active":"2024-03-04T09:28:52.278661+0000","last_peered":"2024-03-04T09:28:52.278661+0000","last_clean":"2024-03-04T09:28:52.278661+0000","last_became_active":"2024-03-04T09:28:51.282245+0000","last_became_peered":"2024-03-04T09:28:51.282245+0000","last_unstale":"2024-03-04T09:28:52.278661+0000","last_undegraded":"2024-03-04T09:28:52.278661+0000","last_fullsized":"2024-03-04T09:28:52.278661+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:42:40.774326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.277929+0000","last_change":"2024-03-04T09:28:50.259640+0000","last_active":"2024-03-04T09:28:52.277929+0000","last_peered":"2024-03-04T09:28:52.277929+0000","last_clean":"2024-03-04T09:28:52.277929+0000","last_became_active":"2024-03-04T09:28:50.259362+0000","last_became_peered":"2024-03-04T09:28:50.259362+0000","last_unstale":"2024-03-04T09:28:52.277929+0000","last_undegraded":"2024-03-04T09:28:52.277929+0000","last_fullsized":"2024-03-04T09:28:52.277929+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_clean_scrub_stamp":"2024-03-04T09:28:49.199103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:23:31.393401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.847114+0000","last_change":"2024-03-04T09:28:51.267895+0000","last_active":"2024-03-04T09:28:52.847114+0000","last_peered":"2024-03-04T09:28:52.847114+0000","last_clean":"2024-03-04T09:28:52.847114+0000","last_became_active":"2024-03-04T09:28:51.267552+0000","last_became_peered":"2024-03-04T09:28:51.267552+0000","last_unstale":"2024-03-04T09:28:52.847114+0000","last_undegraded":"2024-03-04T09:28:52.847114+0000","last_fullsized":"2024-03-04T09:28:52.847114+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:05:13.857612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.846657+0000","last_change":"2024-03-04T09:28:50.257192+0000","last_active":"2024-03-04T09:28:52.846657+0000","last_peered":"2024-03-04T09:28:52.846657+0000","last_clean":"2024-03-04T09:28:52.846657+0000","last_became_active":"2024-03-04T09:28:50.254174+0000","last_became_peered":"2024-03-04T09:28:50.254174+0000","last_unstale":"2024-03-04T09:28:52.846657+0000","last_undegraded":"2024-03-04T09:28:52.846657+0000","last_fullsized":"2024-03-04T09:28:52.846657+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_clean_scrub_stamp":"2024-03-04T09:28:49.199103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:42:50.437322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.846376+0000","last_change":"2024-03-04T09:28:51.287836+0000","last_active":"2024-03-04T09:28:52.846376+0000","last_peered":"2024-03-04T09:28:52.846376+0000","last_clean":"2024-03-04T09:28:52.846376+0000","last_became_active":"2024-03-04T09:28:51.287517+0000","last_became_peered":"2024-03-04T09:28:51.287517+0000","last_unstale":"2024-03-04T09:28:52.846376+0000","last_undegraded":"2024-03-04T09:28:52.846376+0000","last_fullsized":"2024-03-04T09:28:52.846376+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:25:43.926643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.278888+0000","last_change":"2024-03-04T09:28:50.257249+0000","last_active":"2024-03-04T09:28:52.278888+0000","last_peered":"2024-03-04T09:28:52.278888+0000","last_clean":"2024-03-04T09:28:52.278888+0000","last_became_active":"2024-03-04T09:28:50.256948+0000","last_became_peered":"2024-03-04T09:28:50.256948+0000","last_unstale":"2024-03-04T09:28:52.278888+0000","last_undegraded":"2024-03-04T09:28:52.278888+0000","last_fullsized":"2024-03-04T09:28:52.278888+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_clean_scrub_stamp":"2024-03-04T09:28:49.199103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:23:31.393401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.846481+0000","last_change":"2024-03-04T09:28:51.288816+0000","last_active":"2024-03-04T09:28:52.846481+0000","last_peered":"2024-03-04T09:28:52.846481+0000","last_clean":"2024-03-04T09:28:52.846481+0000","last_became_active":"2024-03-04T09:28:51.288574+0000","last_became_peered":"2024-03-04T09:28:51.288574+0000","last_unstale":"2024-03-04T09:28:52.846481+0000","last_undegraded":"2024-03-04T09:28:52.846481+0000","last_fullsized":"2024-03-04T09:28:52.846481+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:12:46.486857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.283884+0000","last_change":"2024-03-04T09:28:50.243400+0000","last_active":"2024-03-04T09:28:52.283884+0000","last_peered":"2024-03-04T09:28:52.283884+0000","last_clean":"2024-03-04T09:28:52.283884+0000","last_became_active":"2024-03-04T09:28:50.240709+0000","last_became_peered":"2024-03-04T09:28:50.240709+0000","last_unstale":"2024-03-04T09:28:52.283884+0000","last_undegraded":"2024-03-04T09:28:52.283884+0000","last_fullsized":"2024-03-04T09:28:52.283884+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_clean_scrub_stamp":"2024-03-04T09:28:49.199103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:12:41.549376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.284329+0000","last_change":"2024-03-04T09:28:51.279275+0000","last_active":"2024-03-04T09:28:52.284329+0000","last_peered":"2024-03-04T09:28:52.284329+0000","last_clean":"2024-03-04T09:28:52.284329+0000","last_became_active":"2024-03-04T09:28:51.278907+0000","last_became_peered":"2024-03-04T09:28:51.278907+0000","last_unstale":"2024-03-04T09:28:52.284329+0000","last_undegraded":"2024-03-04T09:28:52.284329+0000","last_fullsized":"2024-03-04T09:28:52.284329+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:50:26.619245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"67'2","reported_seq":16,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.846115+0000","last_change":"2024-03-04T09:28:50.253782+0000","last_active":"2024-03-04T09:28:52.846115+0000","last_peered":"2024-03-04T09:28:52.846115+0000","last_clean":"2024-03-04T09:28:52.846115+0000","last_became_active":"2024-03-04T09:28:50.253393+0000","last_became_peered":"2024-03-04T09:28:50.253393+0000","last_unstale":"2024-03-04T09:28:52.846115+0000","last_undegraded":"2024-03-04T09:28:52.846115+0000","last_fullsized":"2024-03-04T09:28:52.846115+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_clean_scrub_stamp":"2024-03-04T09:28:49.199103+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:31:28.124868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.278349+0000","last_change":"2024-03-04T09:28:51.288030+0000","last_active":"2024-03-04T09:28:52.278349+0000","last_peered":"2024-03-04T09:28:52.278349+0000","last_clean":"2024-03-04T09:28:52.278349+0000","last_became_active":"2024-03-04T09:28:51.287733+0000","last_became_peered":"2024-03-04T09:28:51.287733+0000","last_unstale":"2024-03-04T09:28:52.278349+0000","last_undegraded":"2024-03-04T09:28:52.278349+0000","last_fullsized":"2024-03-04T09:28:52.278349+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:55:12.363630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"68'1","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.282294+0000","last_change":"2024-03-04T09:28:50.257730+0000","last_active":"2024-03-04T09:28:52.282294+0000","last_peered":"2024-03-04T09:28:52.282294+0000","last_clean":"2024-03-04T09:28:52.282294+0000","last_became_active":"2024-03-04T09:28:50.257342+0000","last_became_peered":"2024-03-04T09:28:50.257342+0000","last_unstale":"2024-03-04T09:28:52.282294+0000","last_undegraded":"2024-03-04T09:28:52.282294+0000","last_fullsized":"2024-03-04T09:28:52.282294+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_clean_scrub_stamp":"2024-03-04T09:28:49.199103+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:35:07.289036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.279119+0000","last_change":"2024-03-04T09:28:51.297902+0000","last_active":"2024-03-04T09:28:52.279119+0000","last_peered":"2024-03-04T09:28:52.279119+0000","last_clean":"2024-03-04T09:28:52.279119+0000","last_became_active":"2024-03-04T09:28:51.297567+0000","last_became_peered":"2024-03-04T09:28:51.297567+0000","last_unstale":"2024-03-04T09:28:52.279119+0000","last_undegraded":"2024-03-04T09:28:52.279119+0000","last_fullsized":"2024-03-04T09:28:52.279119+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:40:40.882472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.278766+0000","last_change":"2024-03-04T09:28:50.247357+0000","last_active":"2024-03-04T09:28:52.278766+0000","last_peered":"2024-03-04T09:28:52.278766+0000","last_clean":"2024-03-04T09:28:52.278766+0000","last_became_active":"2024-03-04T09:28:50.247078+0000","last_became_peered":"2024-03-04T09:28:50.247078+0000","last_unstale":"2024-03-04T09:28:52.278766+0000","last_undegraded":"2024-03-04T09:28:52.278766+0000","last_fullsized":"2024-03-04T09:28:52.278766+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_clean_scrub_stamp":"2024-03-04T09:28:49.199103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:35:00.022149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.280105+0000","last_change":"2024-03-04T09:28:51.295573+0000","last_active":"2024-03-04T09:28:52.280105+0000","last_peered":"2024-03-04T09:28:52.280105+0000","last_clean":"2024-03-04T09:28:52.280105+0000","last_became_active":"2024-03-04T09:28:51.292792+0000","last_became_peered":"2024-03-04T09:28:51.292792+0000","last_unstale":"2024-03-04T09:28:52.280105+0000","last_undegraded":"2024-03-04T09:28:52.280105+0000","last_fullsized":"2024-03-04T09:28:52.280105+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:04:49.289454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.283999+0000","last_change":"2024-03-04T09:28:50.244670+0000","last_active":"2024-03-04T09:28:52.283999+0000","last_peered":"2024-03-04T09:28:52.283999+0000","last_clean":"2024-03-04T09:28:52.283999+0000","last_became_active":"2024-03-04T09:28:50.244396+0000","last_became_peered":"2024-03-04T09:28:50.244396+0000","last_unstale":"2024-03-04T09:28:52.283999+0000","last_undegraded":"2024-03-04T09:28:52.283999+0000","last_fullsized":"2024-03-04T09:28:52.283999+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_clean_scrub_stamp":"2024-03-04T09:28:49.199103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:50:23.693605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.278279+0000","last_change":"2024-03-04T09:28:51.275020+0000","last_active":"2024-03-04T09:28:52.278279+0000","last_peered":"2024-03-04T09:28:52.278279+0000","last_clean":"2024-03-04T09:28:52.278279+0000","last_became_active":"2024-03-04T09:28:51.272292+0000","last_became_peered":"2024-03-04T09:28:51.272292+0000","last_unstale":"2024-03-04T09:28:52.278279+0000","last_undegraded":"2024-03-04T09:28:52.278279+0000","last_fullsized":"2024-03-04T09:28:52.278279+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:18:00.369590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.278187+0000","last_change":"2024-03-04T09:28:50.258722+0000","last_active":"2024-03-04T09:28:52.278187+0000","last_peered":"2024-03-04T09:28:52.278187+0000","last_clean":"2024-03-04T09:28:52.278187+0000","last_became_active":"2024-03-04T09:28:50.258414+0000","last_became_peered":"2024-03-04T09:28:50.258414+0000","last_unstale":"2024-03-04T09:28:52.278187+0000","last_undegraded":"2024-03-04T09:28:52.278187+0000","last_fullsized":"2024-03-04T09:28:52.278187+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_clean_scrub_stamp":"2024-03-04T09:28:49.199103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:02:27.080325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.846348+0000","last_change":"2024-03-04T09:28:51.306716+0000","last_active":"2024-03-04T09:28:52.846348+0000","last_peered":"2024-03-04T09:28:52.846348+0000","last_clean":"2024-03-04T09:28:52.846348+0000","last_became_active":"2024-03-04T09:28:51.306381+0000","last_became_peered":"2024-03-04T09:28:51.306381+0000","last_unstale":"2024-03-04T09:28:52.846348+0000","last_undegraded":"2024-03-04T09:28:52.846348+0000","last_fullsized":"2024-03-04T09:28:52.846348+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:05:51.773362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.275312+0000","last_change":"2024-03-04T09:28:50.248128+0000","last_active":"2024-03-04T09:28:52.275312+0000","last_peered":"2024-03-04T09:28:52.275312+0000","last_clean":"2024-03-04T09:28:52.275312+0000","last_became_active":"2024-03-04T09:28:50.247783+0000","last_became_peered":"2024-03-04T09:28:50.247783+0000","last_unstale":"2024-03-04T09:28:52.275312+0000","last_undegraded":"2024-03-04T09:28:52.275312+0000","last_fullsized":"2024-03-04T09:28:52.275312+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_clean_scrub_stamp":"2024-03-04T09:28:49.199103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:39:34.055388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.274711+0000","last_change":"2024-03-04T09:28:51.274666+0000","last_active":"2024-03-04T09:28:52.274711+0000","last_peered":"2024-03-04T09:28:52.274711+0000","last_clean":"2024-03-04T09:28:52.274711+0000","last_became_active":"2024-03-04T09:28:51.272228+0000","last_became_peered":"2024-03-04T09:28:51.272228+0000","last_unstale":"2024-03-04T09:28:52.274711+0000","last_undegraded":"2024-03-04T09:28:52.274711+0000","last_fullsized":"2024-03-04T09:28:52.274711+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:46:27.964045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.846559+0000","last_change":"2024-03-04T09:28:50.257206+0000","last_active":"2024-03-04T09:28:52.846559+0000","last_peered":"2024-03-04T09:28:52.846559+0000","last_clean":"2024-03-04T09:28:52.846559+0000","last_became_active":"2024-03-04T09:28:50.256864+0000","last_became_peered":"2024-03-04T09:28:50.256864+0000","last_unstale":"2024-03-04T09:28:52.846559+0000","last_undegraded":"2024-03-04T09:28:52.846559+0000","last_fullsized":"2024-03-04T09:28:52.846559+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_clean_scrub_stamp":"2024-03-04T09:28:49.199103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:52:57.606106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.845446+0000","last_change":"2024-03-04T09:28:51.296508+0000","last_active":"2024-03-04T09:28:52.845446+0000","last_peered":"2024-03-04T09:28:52.845446+0000","last_clean":"2024-03-04T09:28:52.845446+0000","last_became_active":"2024-03-04T09:28:51.296111+0000","last_became_peered":"2024-03-04T09:28:51.296111+0000","last_unstale":"2024-03-04T09:28:52.845446+0000","last_undegraded":"2024-03-04T09:28:52.845446+0000","last_fullsized":"2024-03-04T09:28:52.845446+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:40:15.635978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.846510+0000","last_change":"2024-03-04T09:28:50.249908+0000","last_active":"2024-03-04T09:28:52.846510+0000","last_peered":"2024-03-04T09:28:52.846510+0000","last_clean":"2024-03-04T09:28:52.846510+0000","last_became_active":"2024-03-04T09:28:50.249577+0000","last_became_peered":"2024-03-04T09:28:50.249577+0000","last_unstale":"2024-03-04T09:28:52.846510+0000","last_undegraded":"2024-03-04T09:28:52.846510+0000","last_fullsized":"2024-03-04T09:28:52.846510+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_clean_scrub_stamp":"2024-03-04T09:28:49.199103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:30:03.955492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.846621+0000","last_change":"2024-03-04T09:28:51.286417+0000","last_active":"2024-03-04T09:28:52.846621+0000","last_peered":"2024-03-04T09:28:52.846621+0000","last_clean":"2024-03-04T09:28:52.846621+0000","last_became_active":"2024-03-04T09:28:51.284567+0000","last_became_peered":"2024-03-04T09:28:51.284567+0000","last_unstale":"2024-03-04T09:28:52.846621+0000","last_undegraded":"2024-03-04T09:28:52.846621+0000","last_fullsized":"2024-03-04T09:28:52.846621+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:50.213614+0000","last_clean_scrub_stamp":"2024-03-04T09:28:50.213614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:18:24.785867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.b","version":"67'2","reported_seq":16,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.846416+0000","last_change":"2024-03-04T09:28:50.250268+0000","last_active":"2024-03-04T09:28:52.846416+0000","last_peered":"2024-03-04T09:28:52.846416+0000","last_clean":"2024-03-04T09:28:52.846416+0000","last_became_active":"2024-03-04T09:28:50.249233+0000","last_became_peered":"2024-03-04T09:28:50.249233+0000","last_unstale":"2024-03-04T09:28:52.846416+0000","last_undegraded":"2024-03-04T09:28:52.846416+0000","last_fullsized":"2024-03-04T09:28:52.846416+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:28:49.199103+0000","last_clean_scrub_stamp":"2024-03-04T09:28:49.199103+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:18:20.649121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":259,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-04T09:28:52.277282+0000","last_change":"2024-03-04T09:25:09.316159+0000","last_active":"2024-03-04T09:28:52.277282+0000","last_peered":"2024-03-04T09:28:52.277282+0000","last_clean":"2024-03-04T09:28:52.277282+0000","last_became_active":"2024-03-04T09:25:09.313767+0000","last_became_peered":"2024-03-04T09:25:09.313767+0000","last_unstale":"2024-03-04T09:28:52.277282+0000","last_undegraded":"2024-03-04T09:28:52.277282+0000","last_fullsized":"2024-03-04T09:28:52.277282+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:08.110953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:08.110953+0000","last_clean_scrub_stamp":"2024-03-04T09:25:08.110953+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:34:21.745390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574896,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32580,"kb_used_data":1396,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339260,"statfs":{"total":96636764160,"available":96603402240,"internally_reserved":0,"allocated":1429504,"data_stored":938054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574896,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31984,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339856,"statfs":{"total":96636764160,"available":96604012544,"internally_reserved":0,"allocated":823296,"data_stored":340862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574898,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31996,"kb_used_data":812,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339844,"statfs":{"total":96636764160,"available":96604000256,"internally_reserved":0,"allocated":831488,"data_stored":347160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574895,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31988,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339852,"statfs":{"total":96636764160,"available":96604008448,"internally_reserved":0,"allocated":819200,"data_stored":340840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574897,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31984,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339856,"statfs":{"total":96636764160,"available":96604012544,"internally_reserved":0,"allocated":823296,"data_stored":340882,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574896,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32004,"kb_used_data":820,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339836,"statfs":{"total":96636764160,"available":96603992064,"internally_reserved":0,"allocated":839680,"data_stored":348682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574897,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31996,"kb_used_data":816,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339844,"statfs":{"total":96636764160,"available":96604000256,"internally_reserved":0,"allocated":835584,"data_stored":342998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574896,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32564,"kb_used_data":1376,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339276,"statfs":{"total":96636764160,"available":96603418624,"internally_reserved":0,"allocated":1409024,"data_stored":931138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-04T09:28:56.296 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-04T09:28:56.296 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-04T09:28:56.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.572+0000 7ff777000640 1 Processor -- start 2024-03-04T09:28:56.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.572+0000 7ff777000640 1 -- start start 2024-03-04T09:28:56.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.573+0000 7ff777000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff7700d00c0 0x7ff77015cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:56.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.573+0000 7ff777000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7700cfd70 con 0x7ff7700cc5f0 2024-03-04T09:28:56.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.573+0000 7ff777000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff77015c120 con 0x7ff7700d04a0 2024-03-04T09:28:56.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.573+0000 7ff777000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7700cd170 con 0x7ff7700d00c0 2024-03-04T09:28:56.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.574+0000 7ff775c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff7700d00c0 0x7ff77015cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:56.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.574+0000 7ff776600640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7ff7700cc5f0 0x7ff7700cca60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:51760/0 (socket says 172.21.15.6:51760) 2024-03-04T09:28:56.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.574+0000 7ff776600640 1 -- 172.21.15.6:0/2259017196 learned_addr learned my addr 172.21.15.6:0/2259017196 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:28:56.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.574+0000 7ff775c00640 1 -- 172.21.15.6:0/2259017196 >> v1:172.21.15.146:6789/0 conn(0x7ff7700d04a0 legacy=0x7ff7700ced10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:56.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.574+0000 7ff775c00640 1 -- 172.21.15.6:0/2259017196 >> v1:172.21.15.6:6789/0 conn(0x7ff7700cc5f0 legacy=0x7ff7700cca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:56.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.574+0000 7ff76fe00640 1 -- 172.21.15.6:0/2259017196 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4046069164 0 0) 0x7ff7700cfd70 con 0x7ff7700cc5f0 2024-03-04T09:28:56.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.574+0000 7ff775c00640 1 -- 172.21.15.6:0/2259017196 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7700cfa90 con 0x7ff7700d00c0 2024-03-04T09:28:56.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.575+0000 7ff775c00640 1 --2- 172.21.15.6:0/2259017196 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff7700d00c0 0x7ff77015cd60 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7ff7640097a0 tx=0x7ff76402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8555909f53bf7f45 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:56.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.576+0000 7ff76fe00640 1 -- 172.21.15.6:0/2259017196 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff764035020 con 0x7ff7700d00c0 2024-03-04T09:28:56.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.576+0000 7ff76fe00640 1 -- 172.21.15.6:0/2259017196 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff764031830 con 0x7ff7700d00c0 2024-03-04T09:28:56.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.576+0000 7ff76fe00640 1 -- 172.21.15.6:0/2259017196 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff764031c00 con 0x7ff7700d00c0 2024-03-04T09:28:56.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.576+0000 7ff777000640 1 -- 172.21.15.6:0/2259017196 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff7700d00c0 msgr2=0x7ff77015cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:56.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.576+0000 7ff777000640 1 --2- 172.21.15.6:0/2259017196 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff7700d00c0 0x7ff77015cd60 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7ff7640097a0 tx=0x7ff76402b8d0 comp rx=0 tx=0).stop 2024-03-04T09:28:56.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.576+0000 7ff777000640 1 -- 172.21.15.6:0/2259017196 shutdown_connections 2024-03-04T09:28:56.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.576+0000 7ff777000640 1 --2- 172.21.15.6:0/2259017196 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff7700d00c0 0x7ff77015cd60 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:56.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.576+0000 7ff777000640 1 -- 172.21.15.6:0/2259017196 >> 172.21.15.6:0/2259017196 conn(0x7ff7700d1ae0 msgr2=0x7ff7700d1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:56.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.577+0000 7ff777000640 1 -- 172.21.15.6:0/2259017196 shutdown_connections 2024-03-04T09:28:56.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.577+0000 7ff777000640 1 -- 172.21.15.6:0/2259017196 wait complete. 2024-03-04T09:28:56.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.578+0000 7ff777000640 1 Processor -- start 2024-03-04T09:28:56.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.578+0000 7ff777000640 1 -- start start 2024-03-04T09:28:56.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.579+0000 7ff777000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff7700cc5f0 0x7ff770153d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:56.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.579+0000 7ff777000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff7700d00c0 0x7ff7700ca8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:56.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.579+0000 7ff777000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff7700d04a0 0x7ff7700cadf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:56.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.579+0000 7ff777000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7700ce510 con 0x7ff7700cc5f0 2024-03-04T09:28:56.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.579+0000 7ff777000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff77015d2a0 con 0x7ff7700d04a0 2024-03-04T09:28:56.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.579+0000 7ff777000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7700ce7f0 con 0x7ff7700d00c0 2024-03-04T09:28:56.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.580+0000 7ff775c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff7700cc5f0 0x7ff770153d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:56.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.580+0000 7ff775c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff7700cc5f0 0x7ff770153d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49288/0 (socket says 172.21.15.6:49288) 2024-03-04T09:28:56.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.580+0000 7ff775c00640 1 -- 172.21.15.6:0/2536614340 learned_addr learned my addr 172.21.15.6:0/2536614340 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:28:56.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.580+0000 7ff775200640 1 --2- 172.21.15.6:0/2536614340 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff7700d00c0 0x7ff7700ca8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:56.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.580+0000 7ff776600640 1 --2- 172.21.15.6:0/2536614340 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff7700d04a0 0x7ff7700cadf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:56.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.580+0000 7ff775c00640 1 -- 172.21.15.6:0/2536614340 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff7700d00c0 msgr2=0x7ff7700ca8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:56.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.580+0000 7ff775c00640 1 --2- 172.21.15.6:0/2536614340 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff7700d00c0 0x7ff7700ca8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:56.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.580+0000 7ff775c00640 1 -- 172.21.15.6:0/2536614340 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff7700d04a0 msgr2=0x7ff7700cadf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:56.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.580+0000 7ff775c00640 1 --2- 172.21.15.6:0/2536614340 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff7700d04a0 0x7ff7700cadf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:56.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.580+0000 7ff775c00640 1 -- 172.21.15.6:0/2536614340 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7700c76c0 con 0x7ff7700cc5f0 2024-03-04T09:28:56.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.580+0000 7ff776600640 1 --2- 172.21.15.6:0/2536614340 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff7700d04a0 0x7ff7700cadf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:28:56.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.581+0000 7ff775c00640 1 --2- 172.21.15.6:0/2536614340 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff7700cc5f0 0x7ff770153d70 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7ff7640097a0 tx=0x7ff7640046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:56.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.581+0000 7ff76d600640 1 -- 172.21.15.6:0/2536614340 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff76403aa70 con 0x7ff7700cc5f0 2024-03-04T09:28:56.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.581+0000 7ff76d600640 1 -- 172.21.15.6:0/2536614340 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff76403ac10 con 0x7ff7700cc5f0 2024-03-04T09:28:56.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.581+0000 7ff76d600640 1 -- 172.21.15.6:0/2536614340 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7640443f0 con 0x7ff7700cc5f0 2024-03-04T09:28:56.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.581+0000 7ff777000640 1 -- 172.21.15.6:0/2536614340 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7700c79a0 con 0x7ff7700cc5f0 2024-03-04T09:28:56.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.581+0000 7ff777000640 1 -- 172.21.15.6:0/2536614340 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7701542b0 con 0x7ff7700cc5f0 2024-03-04T09:28:56.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.582+0000 7ff777000640 1 -- 172.21.15.6:0/2536614340 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff77004f330 con 0x7ff7700cc5f0 2024-03-04T09:28:56.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.583+0000 7ff76d600640 1 -- 172.21.15.6:0/2536614340 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff76404b060 con 0x7ff7700cc5f0 2024-03-04T09:28:56.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.585+0000 7ff76d600640 1 --2- 172.21.15.6:0/2536614340 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff75006b4d0 0x7ff75006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:56.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.585+0000 7ff76d600640 1 -- 172.21.15.6:0/2536614340 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff7640abed0 con 0x7ff7700cc5f0 2024-03-04T09:28:56.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.585+0000 7ff775200640 1 --2- 172.21.15.6:0/2536614340 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff75006b4d0 0x7ff75006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:56.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.587+0000 7ff775200640 1 --2- 172.21.15.6:0/2536614340 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff75006b4d0 0x7ff75006d990 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7ff75c004ea0 tx=0x7ff75c009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:56.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.589+0000 7ff76d600640 1 -- 172.21.15.6:0/2536614340 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff76403f070 con 0x7ff7700cc5f0 2024-03-04T09:28:56.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.867+0000 7ff777000640 1 -- 172.21.15.6:0/2536614340 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7ff77006e4a0 con 0x7ff7700cc5f0 2024-03-04T09:28:56.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.872+0000 7ff76d600640 1 -- 172.21.15.6:0/2536614340 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v13) v1 ==== 190+0+147 (secure 0 0 0) 0x7ff76402f2e0 con 0x7ff7700cc5f0 2024-03-04T09:28:56.869 INFO:teuthology.orchestra.run.smithi006.stdout:[client.0] 2024-03-04T09:28:56.869 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQA9k+Vl4L+rJxAACaDbQ4/Bybp5hAZAnJzlVw== 2024-03-04T09:28:56.869 INFO:teuthology.orchestra.run.smithi006.stdout: caps mds = "allow" 2024-03-04T09:28:56.870 INFO:teuthology.orchestra.run.smithi006.stdout: caps mon = "allow r" 2024-03-04T09:28:56.870 INFO:teuthology.orchestra.run.smithi006.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-04T09:28:56.870 INFO:teuthology.orchestra.run.smithi006.stderr:updated caps for client.0 2024-03-04T09:28:56.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.875+0000 7ff777000640 1 -- 172.21.15.6:0/2536614340 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff75006b4d0 msgr2=0x7ff75006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:56.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.875+0000 7ff777000640 1 --2- 172.21.15.6:0/2536614340 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff75006b4d0 0x7ff75006d990 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7ff75c004ea0 tx=0x7ff75c009040 comp rx=0 tx=0).stop 2024-03-04T09:28:56.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.875+0000 7ff777000640 1 -- 172.21.15.6:0/2536614340 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff7700cc5f0 msgr2=0x7ff770153d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:56.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.875+0000 7ff777000640 1 --2- 172.21.15.6:0/2536614340 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff7700cc5f0 0x7ff770153d70 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7ff7640097a0 tx=0x7ff7640046c0 comp rx=0 tx=0).stop 2024-03-04T09:28:56.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.875+0000 7ff777000640 1 -- 172.21.15.6:0/2536614340 shutdown_connections 2024-03-04T09:28:56.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.875+0000 7ff777000640 1 --2- 172.21.15.6:0/2536614340 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff7700d04a0 0x7ff7700cadf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:56.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.875+0000 7ff777000640 1 --2- 172.21.15.6:0/2536614340 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff75006b4d0 0x7ff75006d990 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:56.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.875+0000 7ff777000640 1 --2- 172.21.15.6:0/2536614340 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff7700d00c0 0x7ff7700ca8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:56.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.875+0000 7ff777000640 1 --2- 172.21.15.6:0/2536614340 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff7700cc5f0 0x7ff770153d70 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:56.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.875+0000 7ff777000640 1 -- 172.21.15.6:0/2536614340 >> 172.21.15.6:0/2536614340 conn(0x7ff7700d1ae0 msgr2=0x7ff7700d1ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:56.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.875+0000 7ff777000640 1 -- 172.21.15.6:0/2536614340 shutdown_connections 2024-03-04T09:28:56.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:56.875+0000 7ff777000640 1 -- 172.21.15.6:0/2536614340 wait complete. 2024-03-04T09:28:56.893 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:28:57.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:57.117+0000 7f991e800640 1 Processor -- start 2024-03-04T09:28:57.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:57.118+0000 7f991e800640 1 -- start start 2024-03-04T09:28:57.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:57.119+0000 7f991e800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f99180af1a0 0x7f991815c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:57.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:57.119+0000 7f991e800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f99180af580 0x7f99180b4c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:57.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:57.119+0000 7f991e800640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f991815cc40 con 0x7f99180cdc20 2024-03-04T09:28:57.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:57.119+0000 7f991e800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f991815cf20 con 0x7f99180af580 2024-03-04T09:28:57.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:57.119+0000 7f991e800640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f991815d200 con 0x7f99180af1a0 2024-03-04T09:28:57.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:57.120+0000 7f9916a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f99180af580 0x7f99180b4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:57.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:57.120+0000 7f9916a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f99180af580 0x7f99180b4c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49302/0 (socket says 172.21.15.6:49302) 2024-03-04T09:28:57.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:57.120+0000 7f9916a00640 1 -- 172.21.15.6:0/4258840326 learned_addr learned my addr 172.21.15.6:0/4258840326 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:28:57.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:57.120+0000 7f9917400640 1 --2- 172.21.15.6:0/4258840326 >> v2:172.21.15.146:3300/0 conn(0x7f99180af1a0 0x7f991815c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:57.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:57.120+0000 7f9916a00640 1 -- 172.21.15.6:0/4258840326 >> v2:172.21.15.146:3300/0 conn(0x7f99180af1a0 msgr2=0x7f991815c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:57.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:57.120+0000 7f9916a00640 1 --2- 172.21.15.6:0/4258840326 >> v2:172.21.15.146:3300/0 conn(0x7f99180af1a0 0x7f991815c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:57.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:57.120+0000 7f9916a00640 1 -- 172.21.15.6:0/4258840326 >> v1:172.21.15.6:6789/0 conn(0x7f99180cdc20 legacy=0x7f99180b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:57.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:57.120+0000 7f9916a00640 1 -- 172.21.15.6:0/4258840326 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f991815d4e0 con 0x7f99180af580 2024-03-04T09:28:57.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:57.121+0000 7f9916a00640 1 --2- 172.21.15.6:0/4258840326 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f99180af580 0x7f99180b4c20 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f9918074960 tx=0x7f990402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=114bfeb4634e5112 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:57.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:57.122+0000 7f9916000640 1 -- 172.21.15.6:0/4258840326 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9904035020 con 0x7f99180af580 2024-03-04T09:28:57.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:57.122+0000 7f9916000640 1 -- 172.21.15.6:0/4258840326 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9904031870 con 0x7f99180af580 2024-03-04T09:28:57.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:57.122+0000 7f9916000640 1 -- 172.21.15.6:0/4258840326 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9904031c40 con 0x7f99180af580 2024-03-04T09:28:57.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:57.122+0000 7f991e800640 1 -- 172.21.15.6:0/4258840326 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f99180af580 msgr2=0x7f99180b4c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:57.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:57.122+0000 7f991e800640 1 --2- 172.21.15.6:0/4258840326 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f99180af580 0x7f99180b4c20 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f9918074960 tx=0x7f990402b910 comp rx=0 tx=0).stop 2024-03-04T09:28:57.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:57.122+0000 7f991e800640 1 -- 172.21.15.6:0/4258840326 shutdown_connections 2024-03-04T09:28:57.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:57.122+0000 7f991e800640 1 --2- 172.21.15.6:0/4258840326 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f99180af580 0x7f99180b4c20 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:57.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:57.122+0000 7f991e800640 1 --2- 172.21.15.6:0/4258840326 >> v2:172.21.15.146:3300/0 conn(0x7f99180af1a0 0x7f991815c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:57.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:57.122+0000 7f991e800640 1 -- 172.21.15.6:0/4258840326 >> 172.21.15.6:0/4258840326 conn(0x7f991806afb0 msgr2=0x7f991806b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:57.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:57.123+0000 7f991e800640 1 -- 172.21.15.6:0/4258840326 shutdown_connections 2024-03-04T09:28:57.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:57.123+0000 7f991e800640 1 -- 172.21.15.6:0/4258840326 wait complete. 2024-03-04T09:28:57.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:57.125+0000 7f991e800640 1 Processor -- start 2024-03-04T09:28:57.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:57.125+0000 7f991e800640 1 -- start start 2024-03-04T09:28:57.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:57.126+0000 7f991e800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f99180af1a0 0x7f991809d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:57.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:57.126+0000 7f991e800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f99180af580 0x7f991814cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:57.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:57.126+0000 7f991e800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f99180cdc20 0x7f99180cec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:57.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:57.126+0000 7f991e800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99180b3660 con 0x7f99180af580 2024-03-04T09:28:57.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:57.126+0000 7f991e800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99180b34e0 con 0x7f99180af1a0 2024-03-04T09:28:57.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:57.126+0000 7f991e800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99180b37e0 con 0x7f99180cdc20 2024-03-04T09:28:57.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:57.126+0000 7f9916a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f99180af580 0x7f991814cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:57.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:57.126+0000 7f9916a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f99180af580 0x7f991814cee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49314/0 (socket says 172.21.15.6:49314) 2024-03-04T09:28:57.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:57.126+0000 7f9916a00640 1 -- 172.21.15.6:0/2039632162 learned_addr learned my addr 172.21.15.6:0/2039632162 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:28:57.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:57.127+0000 7f9917400640 1 --2- 172.21.15.6:0/2039632162 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f99180af1a0 0x7f991809d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:57.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:57.127+0000 7f9917e00640 1 --2- 172.21.15.6:0/2039632162 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f99180cdc20 0x7f99180cec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:57.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:57.127+0000 7f9916a00640 1 -- 172.21.15.6:0/2039632162 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f99180cdc20 msgr2=0x7f99180cec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:57.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:57.127+0000 7f9916a00640 1 --2- 172.21.15.6:0/2039632162 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f99180cdc20 0x7f99180cec00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:57.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:57.127+0000 7f9916a00640 1 -- 172.21.15.6:0/2039632162 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f99180af1a0 msgr2=0x7f991809d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:57.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:57.127+0000 7f9916a00640 1 --2- 172.21.15.6:0/2039632162 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f99180af1a0 0x7f991809d330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:57.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:57.127+0000 7f9916a00640 1 -- 172.21.15.6:0/2039632162 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99180c9650 con 0x7f99180af580 2024-03-04T09:28:57.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:57.127+0000 7f9916a00640 1 --2- 172.21.15.6:0/2039632162 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f99180af580 0x7f991814cee0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f990402b950 tx=0x7f99040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:57.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:57.128+0000 7f990f400640 1 -- 172.21.15.6:0/2039632162 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9904040070 con 0x7f99180af580 2024-03-04T09:28:57.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:57.128+0000 7f990f400640 1 -- 172.21.15.6:0/2039632162 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f990403a420 con 0x7f99180af580 2024-03-04T09:28:57.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:57.128+0000 7f990f400640 1 -- 172.21.15.6:0/2039632162 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9904035040 con 0x7f99180af580 2024-03-04T09:28:57.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:57.128+0000 7f991e800640 1 -- 172.21.15.6:0/2039632162 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99180c9930 con 0x7f99180af580 2024-03-04T09:28:57.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:57.128+0000 7f991e800640 1 -- 172.21.15.6:0/2039632162 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99180ca1d0 con 0x7f99180af580 2024-03-04T09:28:57.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:57.129+0000 7f991e800640 1 -- 172.21.15.6:0/2039632162 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9918069510 con 0x7f99180af580 2024-03-04T09:28:57.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:57.130+0000 7f990f400640 1 -- 172.21.15.6:0/2039632162 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f990404b060 con 0x7f99180af580 2024-03-04T09:28:57.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:57.131+0000 7f990f400640 1 --2- 172.21.15.6:0/2039632162 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f98f806b4d0 0x7f98f806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:57.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:57.131+0000 7f990f400640 1 -- 172.21.15.6:0/2039632162 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f99040ac1b0 con 0x7f99180af580 2024-03-04T09:28:57.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:57.132+0000 7f9917400640 1 --2- 172.21.15.6:0/2039632162 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f98f806b4d0 0x7f98f806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:57.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:57.133+0000 7f9917400640 1 --2- 172.21.15.6:0/2039632162 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f98f806b4d0 0x7f98f806d990 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f99181f7720 tx=0x7f99080079a0 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:57.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:57.135+0000 7f990f400640 1 -- 172.21.15.6:0/2039632162 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f990407b020 con 0x7f99180af580 2024-03-04T09:28:57.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:57.375+0000 7f991e800640 1 -- 172.21.15.6:0/2039632162 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f991804f3a0 con 0x7f99180af580 2024-03-04T09:28:57.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:57.376+0000 7f990f400640 1 -- 172.21.15.6:0/2039632162 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+4979 (secure 0 0 0) 0x7f990407a920 con 0x7f99180af580 2024-03-04T09:28:57.373 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:28:57.373 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":42,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5228,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/1891254736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1891254736},{"type":"v1","addr":"172.21.15.6:6834","nonce":1891254736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":5578,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/1781219843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":1781219843},{"type":"v1","addr":"172.21.15.146:6835","nonce":1781219843}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":5587,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/1479180802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":1479180802},{"type":"v1","addr":"172.21.15.6:6835","nonce":1479180802}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:28:50.708467+0000","modified":"2024-03-04T09:28:53.801285+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5590},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5590":{"gid":5590,"name":"d","rank":0,"incarnation":38,"state":"up:active","state_seq":3,"addr":"172.21.15.146:6837/3919939129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":3919939129},{"type":"v1","addr":"172.21.15.146:6837","nonce":3919939129}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5590,"qdb_cluster":[5590]},"id":5}]} 2024-03-04T09:28:57.374 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 42 2024-03-04T09:28:57.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:57.379+0000 7f991e800640 1 -- 172.21.15.6:0/2039632162 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f98f806b4d0 msgr2=0x7f98f806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:57.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:57.379+0000 7f991e800640 1 --2- 172.21.15.6:0/2039632162 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f98f806b4d0 0x7f98f806d990 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f99181f7720 tx=0x7f99080079a0 comp rx=0 tx=0).stop 2024-03-04T09:28:57.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:57.379+0000 7f991e800640 1 -- 172.21.15.6:0/2039632162 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f99180af580 msgr2=0x7f991814cee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:57.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:57.379+0000 7f991e800640 1 --2- 172.21.15.6:0/2039632162 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f99180af580 0x7f991814cee0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f990402b950 tx=0x7f99040046c0 comp rx=0 tx=0).stop 2024-03-04T09:28:57.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:57.379+0000 7f991e800640 1 -- 172.21.15.6:0/2039632162 shutdown_connections 2024-03-04T09:28:57.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:57.379+0000 7f991e800640 1 --2- 172.21.15.6:0/2039632162 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f98f806b4d0 0x7f98f806d990 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:57.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:57.379+0000 7f991e800640 1 --2- 172.21.15.6:0/2039632162 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f99180cdc20 0x7f99180cec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:57.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:57.379+0000 7f991e800640 1 --2- 172.21.15.6:0/2039632162 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f99180af580 0x7f991814cee0 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:57.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:57.379+0000 7f991e800640 1 --2- 172.21.15.6:0/2039632162 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f99180af1a0 0x7f991809d330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:57.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:57.379+0000 7f991e800640 1 -- 172.21.15.6:0/2039632162 >> 172.21.15.6:0/2039632162 conn(0x7f991806afb0 msgr2=0x7f99180cc4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:57.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:57.380+0000 7f991e800640 1 -- 172.21.15.6:0/2039632162 shutdown_connections 2024-03-04T09:28:57.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:57.380+0000 7f991e800640 1 -- 172.21.15.6:0/2039632162 wait complete. 2024-03-04T09:28:57.798 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 42, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:28:50.708467+0000', 'modified': '2024-03-04T09:28:53.801285+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5590}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5590': {'gid': 5590, 'name': 'd', 'rank': 0, 'incarnation': 38, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.146:6837/3919939129', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6836', 'nonce': 3919939129}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 3919939129}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [11], 'metadata_pool': 10, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5590, 'qdb_cluster': [5590]} 2024-03-04T09:28:57.798 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-04T09:28:57.799 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-03-04T09:28:58.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.034+0000 7f2eeb600640 1 Processor -- start 2024-03-04T09:28:58.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.035+0000 7f2eeb600640 1 -- start start 2024-03-04T09:28:58.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.036+0000 7f2eeb600640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f2ee40af1a0 0x7f2ee415c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:58.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.036+0000 7f2eeb600640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ee415cc40 con 0x7f2ee40af580 2024-03-04T09:28:58.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.036+0000 7f2eeb600640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ee415cf20 con 0x7f2ee40cdc20 2024-03-04T09:28:58.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.036+0000 7f2eeb600640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ee415d200 con 0x7f2ee40af1a0 2024-03-04T09:28:58.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.037+0000 7f2ee3e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f2ee40af580 0x7f2ee40b4c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:51788/0 (socket says 172.21.15.6:51788) 2024-03-04T09:28:58.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.037+0000 7f2ee3e00640 1 -- 172.21.15.6:0/794137173 learned_addr learned my addr 172.21.15.6:0/794137173 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:28:58.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.037+0000 7f2ee8c00640 1 --2- 172.21.15.6:0/794137173 >> v2:172.21.15.146:3300/0 conn(0x7f2ee40af1a0 0x7f2ee415c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:58.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.037+0000 7f2ee3400640 1 -- 172.21.15.6:0/794137173 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 308462769 0 0) 0x7f2ee415cc40 con 0x7f2ee40af580 2024-03-04T09:28:58.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.037+0000 7f2ee3400640 1 -- 172.21.15.6:0/794137173 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ecc003520 con 0x7f2ee40af580 2024-03-04T09:28:58.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.038+0000 7f2ee3400640 1 -- 172.21.15.6:0/794137173 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 996775073 0 0) 0x7f2ecc003520 con 0x7f2ee40af580 2024-03-04T09:28:58.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.038+0000 7f2ee3400640 1 -- 172.21.15.6:0/794137173 >> v2:172.21.15.146:3300/0 conn(0x7f2ee40af1a0 msgr2=0x7f2ee415c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:58.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.038+0000 7f2ee3400640 1 --2- 172.21.15.6:0/794137173 >> v2:172.21.15.146:3300/0 conn(0x7f2ee40af1a0 0x7f2ee415c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:58.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.038+0000 7f2ee3400640 1 -- 172.21.15.6:0/794137173 >> v1:172.21.15.146:6789/0 conn(0x7f2ee40cdc20 legacy=0x7f2ee40b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:58.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.038+0000 7f2ee3400640 1 -- 172.21.15.6:0/794137173 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ee415d4e0 con 0x7f2ee40af580 2024-03-04T09:28:58.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.038+0000 7f2ee3400640 1 -- 172.21.15.6:0/794137173 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7f2ed0001670 con 0x7f2ee40af580 2024-03-04T09:28:58.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.038+0000 7f2ee3400640 1 -- 172.21.15.6:0/794137173 >> v1:172.21.15.6:6789/0 conn(0x7f2ee40af580 legacy=0x7f2ee40b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:58.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.038+0000 7f2ee3400640 1 --2- 172.21.15.6:0/794137173 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2ecc003a20 0x7f2ecc003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:58.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.038+0000 7f2ee3400640 1 --2- 172.21.15.6:0/794137173 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2ecc004800 0x7f2ecc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:58.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.038+0000 7f2ee3400640 1 --2- 172.21.15.6:0/794137173 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2ecc005230 0x7f2ecc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:58.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.038+0000 7f2ee3400640 1 -- 172.21.15.6:0/794137173 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ee415d200 con 0x7f2ecc003a20 2024-03-04T09:28:58.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.038+0000 7f2ee3400640 1 -- 172.21.15.6:0/794137173 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ecc00c190 con 0x7f2ecc004800 2024-03-04T09:28:58.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.038+0000 7f2ee3400640 1 -- 172.21.15.6:0/794137173 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ecc00c390 con 0x7f2ecc005230 2024-03-04T09:28:58.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.038+0000 7f2ee8c00640 1 --2- 172.21.15.6:0/794137173 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2ecc003a20 0x7f2ecc003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:58.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.038+0000 7f2ee3e00640 1 --2- 172.21.15.6:0/794137173 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2ecc004800 0x7f2ecc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:58.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.039+0000 7f2ee8c00640 1 -- 172.21.15.6:0/794137173 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2ecc005230 msgr2=0x7f2ecc00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:58.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.039+0000 7f2ee8c00640 1 --2- 172.21.15.6:0/794137173 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2ecc005230 0x7f2ecc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:58.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.039+0000 7f2ee8c00640 1 -- 172.21.15.6:0/794137173 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2ecc004800 msgr2=0x7f2ecc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:58.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.039+0000 7f2ee8c00640 1 --2- 172.21.15.6:0/794137173 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2ecc004800 0x7f2ecc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:58.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.039+0000 7f2ee8c00640 1 -- 172.21.15.6:0/794137173 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ecc003520 con 0x7f2ecc003a20 2024-03-04T09:28:58.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.039+0000 7f2ee9600640 1 -- 172.21.15.6:0/794137173 reap_dead start 2024-03-04T09:28:58.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.040+0000 7f2ee8c00640 1 --2- 172.21.15.6:0/794137173 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2ecc003a20 0x7f2ecc003e40 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f2ed8009ac0 tx=0x7f2ecc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1acb7e1be05b253b server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:58.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.040+0000 7f2ee3400640 1 -- 172.21.15.6:0/794137173 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ed8039070 con 0x7f2ecc003a20 2024-03-04T09:28:58.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.040+0000 7f2ee3400640 1 -- 172.21.15.6:0/794137173 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ed802c600 con 0x7f2ecc003a20 2024-03-04T09:28:58.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.040+0000 7f2ee3400640 1 -- 172.21.15.6:0/794137173 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ed8009690 con 0x7f2ecc003a20 2024-03-04T09:28:58.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.040+0000 7f2eeb600640 1 -- 172.21.15.6:0/794137173 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2ecc003a20 msgr2=0x7f2ecc003e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:58.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.040+0000 7f2eeb600640 1 --2- 172.21.15.6:0/794137173 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2ecc003a20 0x7f2ecc003e40 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f2ed8009ac0 tx=0x7f2ecc005830 comp rx=0 tx=0).stop 2024-03-04T09:28:58.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.041+0000 7f2eeb600640 1 -- 172.21.15.6:0/794137173 shutdown_connections 2024-03-04T09:28:58.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.041+0000 7f2eeb600640 1 --2- 172.21.15.6:0/794137173 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2ecc003a20 0x7f2ecc003e40 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:58.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.041+0000 7f2eeb600640 1 -- 172.21.15.6:0/794137173 >> 172.21.15.6:0/794137173 conn(0x7f2ee406afb0 msgr2=0x7f2ee406b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:58.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.041+0000 7f2eeb600640 1 -- 172.21.15.6:0/794137173 shutdown_connections 2024-03-04T09:28:58.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.041+0000 7f2eeb600640 1 -- 172.21.15.6:0/794137173 wait complete. 2024-03-04T09:28:58.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.042+0000 7f2eeb600640 1 Processor -- start 2024-03-04T09:28:58.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.043+0000 7f2eeb600640 1 -- start start 2024-03-04T09:28:58.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.043+0000 7f2eeb600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2ecc003a20 0x7f2ee4156480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:58.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.044+0000 7f2eeb600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2ee4158b50 0x7f2ee409aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:58.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.044+0000 7f2eeb600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2ee409d210 0x7f2ee409d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:58.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.044+0000 7f2eeb600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ee415c990 con 0x7f2ecc003a20 2024-03-04T09:28:58.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.044+0000 7f2eeb600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ee415c810 con 0x7f2ee4158b50 2024-03-04T09:28:58.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.044+0000 7f2eeb600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ee40b4c20 con 0x7f2ee409d210 2024-03-04T09:28:58.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.044+0000 7f2ee8c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2ecc003a20 0x7f2ee4156480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:58.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.044+0000 7f2ee8c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2ecc003a20 0x7f2ee4156480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49330/0 (socket says 172.21.15.6:49330) 2024-03-04T09:28:58.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.044+0000 7f2ee8c00640 1 -- 172.21.15.6:0/4065709101 learned_addr learned my addr 172.21.15.6:0/4065709101 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:28:58.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.044+0000 7f2ee8c00640 1 -- 172.21.15.6:0/4065709101 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2ee409d210 msgr2=0x7f2ee409d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:58.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.044+0000 7f2ee8c00640 1 --2- 172.21.15.6:0/4065709101 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2ee409d210 0x7f2ee409d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:58.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.044+0000 7f2ee8c00640 1 -- 172.21.15.6:0/4065709101 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2ee4158b50 msgr2=0x7f2ee409aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:58.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.044+0000 7f2ee3e00640 1 --2- 172.21.15.6:0/4065709101 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2ee4158b50 0x7f2ee409aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:58.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.044+0000 7f2ee8c00640 1 --2- 172.21.15.6:0/4065709101 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2ee4158b50 0x7f2ee409aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:58.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.044+0000 7f2ee8c00640 1 -- 172.21.15.6:0/4065709101 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ee409af50 con 0x7f2ecc003a20 2024-03-04T09:28:58.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.045+0000 7f2ee8c00640 1 --2- 172.21.15.6:0/4065709101 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2ecc003a20 0x7f2ee4156480 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f2ed8000c00 tx=0x7f2ed8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:58.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.045+0000 7f2ee0c00640 1 -- 172.21.15.6:0/4065709101 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ed8042020 con 0x7f2ecc003a20 2024-03-04T09:28:58.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.045+0000 7f2ee0c00640 1 -- 172.21.15.6:0/4065709101 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ed803a040 con 0x7f2ecc003a20 2024-03-04T09:28:58.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.045+0000 7f2ee0c00640 1 -- 172.21.15.6:0/4065709101 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ed8039070 con 0x7f2ecc003a20 2024-03-04T09:28:58.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.045+0000 7f2eeb600640 1 -- 172.21.15.6:0/4065709101 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ee41590f0 con 0x7f2ecc003a20 2024-03-04T09:28:58.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.045+0000 7f2eeb600640 1 -- 172.21.15.6:0/4065709101 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ee409ca10 con 0x7f2ecc003a20 2024-03-04T09:28:58.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.047+0000 7f2ee0c00640 1 -- 172.21.15.6:0/4065709101 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2ed8045020 con 0x7f2ecc003a20 2024-03-04T09:28:58.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.048+0000 7f2ee0c00640 1 --2- 172.21.15.6:0/4065709101 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2ec406b500 0x7f2ec406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:58.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.048+0000 7f2ee0c00640 1 -- 172.21.15.6:0/4065709101 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2ed80ab1e0 con 0x7f2ecc003a20 2024-03-04T09:28:58.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.049+0000 7f2ee3e00640 1 --2- 172.21.15.6:0/4065709101 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2ec406b500 0x7f2ec406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:58.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.049+0000 7f2eee394740 1 Processor -- start 2024-03-04T09:28:58.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.049+0000 7f2eee394740 1 -- start start 2024-03-04T09:28:58.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.050+0000 7f2ee3e00640 1 --2- 172.21.15.6:0/4065709101 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2ec406b500 0x7f2ec406d9c0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f2ed0001920 tx=0x7f2ed0001890 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:58.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.050+0000 7f2eee394740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5576106a4330 0x5576106a1ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:58.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.050+0000 7f2eee394740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557610683b70 0x55761065b860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:58.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.050+0000 7f2eee394740 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55761065bda0 0x557610686540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:58.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.050+0000 7f2eee394740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x5576105c0600 con 0x5576106a4330 2024-03-04T09:28:58.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.050+0000 7f2eee394740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x557610569930 con 0x557610683b70 2024-03-04T09:28:58.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.050+0000 7f2eee394740 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x5576105bc6a0 con 0x55761065bda0 2024-03-04T09:28:58.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.050+0000 7f2ee8c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5576106a4330 0x5576106a1ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:58.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.050+0000 7f2ee8c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5576106a4330 0x5576106a1ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49338/0 (socket says 172.21.15.6:49338) 2024-03-04T09:28:58.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.050+0000 7f2ee8c00640 1 -- 172.21.15.6:0/837165125 learned_addr learned my addr 172.21.15.6:0/837165125 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:28:58.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.051+0000 7f2ee3e00640 1 --2- 172.21.15.6:0/837165125 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557610683b70 0x55761065b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:58.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.051+0000 7f2ee9600640 1 --2- 172.21.15.6:0/837165125 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55761065bda0 0x557610686540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:58.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.051+0000 7f2ee8c00640 1 -- 172.21.15.6:0/837165125 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55761065bda0 msgr2=0x557610686540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:58.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.051+0000 7f2ee8c00640 1 --2- 172.21.15.6:0/837165125 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55761065bda0 0x557610686540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:58.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.051+0000 7f2ee8c00640 1 -- 172.21.15.6:0/837165125 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557610683b70 msgr2=0x55761065b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:58.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.051+0000 7f2ee8c00640 1 --2- 172.21.15.6:0/837165125 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557610683b70 0x55761065b860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:58.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.051+0000 7f2ee8c00640 1 -- 172.21.15.6:0/837165125 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5576102d52d0 con 0x5576106a4330 2024-03-04T09:28:58.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.051+0000 7f2ee8c00640 1 --2- 172.21.15.6:0/837165125 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5576106a4330 0x5576106a1ef0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f2ed8037040 tx=0x7f2ed8079850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9aa5ddf594c5356 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:58.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.051+0000 7f2ed5200640 1 -- 172.21.15.6:0/837165125 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ed807a050 con 0x5576106a4330 2024-03-04T09:28:58.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.051+0000 7f2ed5200640 1 -- 172.21.15.6:0/837165125 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ed8079950 con 0x5576106a4330 2024-03-04T09:28:58.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.051+0000 7f2ed5200640 1 -- 172.21.15.6:0/837165125 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ed807daa0 con 0x5576106a4330 2024-03-04T09:28:58.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.052+0000 7f2eee394740 1 -- 172.21.15.6:0/837165125 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5576106a4330 msgr2=0x5576106a1ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:58.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.052+0000 7f2eee394740 1 --2- 172.21.15.6:0/837165125 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5576106a4330 0x5576106a1ef0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f2ed8037040 tx=0x7f2ed8079850 comp rx=0 tx=0).stop 2024-03-04T09:28:58.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.052+0000 7f2eee394740 1 -- 172.21.15.6:0/837165125 shutdown_connections 2024-03-04T09:28:58.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.052+0000 7f2eee394740 1 --2- 172.21.15.6:0/837165125 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55761065bda0 0x557610686540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:58.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.052+0000 7f2eee394740 1 --2- 172.21.15.6:0/837165125 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557610683b70 0x55761065b860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:58.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.052+0000 7f2eee394740 1 --2- 172.21.15.6:0/837165125 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5576106a4330 0x5576106a1ef0 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:58.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.052+0000 7f2eee394740 1 -- 172.21.15.6:0/837165125 >> 172.21.15.6:0/837165125 conn(0x5576106589c0 msgr2=0x5576106a1540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:58.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.052+0000 7f2eee394740 1 -- 172.21.15.6:0/837165125 shutdown_connections 2024-03-04T09:28:58.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.052+0000 7f2eee394740 1 -- 172.21.15.6:0/837165125 wait complete. 2024-03-04T09:28:58.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.053+0000 7f2eee394740 1 -- start start 2024-03-04T09:28:58.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.053+0000 7f2eee394740 1 Processor -- start 2024-03-04T09:28:58.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.054+0000 7f2eee394740 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x557610683b70 0x55761066da70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:58.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.054+0000 7f2eee394740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5576106a4330 0x5576106841c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:58.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.054+0000 7f2eee394740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x557610684700 0x557610688b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:58.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.054+0000 7f2eee394740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x5576105bc6a0 con 0x557610684700 2024-03-04T09:28:58.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.054+0000 7f2eee394740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x55761057cd90 con 0x5576106a4330 2024-03-04T09:28:58.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.054+0000 7f2eee394740 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x5576105c0600 con 0x557610683b70 2024-03-04T09:28:58.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.054+0000 7f2eee394740 10 client.0 _ll_register_callbacks cb 0x5576105b8350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T09:28:58.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.055+0000 7f2ee9600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x557610684700 0x557610688b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:58.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.055+0000 7f2ee9600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x557610684700 0x557610688b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49346/0 (socket says 172.21.15.6:49346) 2024-03-04T09:28:58.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.055+0000 7f2ee9600640 1 -- 172.21.15.6:0/966404245 learned_addr learned my addr 172.21.15.6:0/966404245 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:28:58.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.055+0000 7f2ee3e00640 1 --2- 172.21.15.6:0/966404245 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5576106a4330 0x5576106841c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:58.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.055+0000 7f2ee8c00640 1 --2- 172.21.15.6:0/966404245 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x557610683b70 0x55761066da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:58.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.055+0000 7f2ee9600640 1 -- 172.21.15.6:0/966404245 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x557610683b70 msgr2=0x55761066da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:58.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.055+0000 7f2ee9600640 1 --2- 172.21.15.6:0/966404245 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x557610683b70 0x55761066da70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:58.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.055+0000 7f2ee9600640 1 -- 172.21.15.6:0/966404245 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5576106a4330 msgr2=0x5576106841c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:58.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.055+0000 7f2ee9600640 1 --2- 172.21.15.6:0/966404245 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5576106a4330 0x5576106841c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:58.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.055+0000 7f2ee9600640 1 -- 172.21.15.6:0/966404245 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55761039f6b0 con 0x557610684700 2024-03-04T09:28:58.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.055+0000 7f2ee3e00640 1 --2- 172.21.15.6:0/966404245 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5576106a4330 0x5576106841c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:28:58.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.055+0000 7f2ee9600640 1 --2- 172.21.15.6:0/966404245 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x557610684700 0x557610688b60 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f2ee409e100 tx=0x7f2edc00d810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2ff2d26d1e0c712 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:58.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.055+0000 7f2ec9600640 10 client.0 ms_handle_connect on v2:172.21.15.6:3300/0 2024-03-04T09:28:58.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.056+0000 7f2ec9600640 1 -- 172.21.15.6:0/966404245 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2edc018060 con 0x557610684700 2024-03-04T09:28:58.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.056+0000 7f2ec9600640 1 -- 172.21.15.6:0/966404245 <== mon.0 v2:172.21.15.6:3300/0 2 ==== osd_map(68..68 src has 1..68) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2edc0070d0 con 0x557610684700 2024-03-04T09:28:58.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.056+0000 7f2eee394740 1 -- 172.21.15.6:0/966404245 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2ed8079950 con 0x557610684700 2024-03-04T09:28:58.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.056+0000 7f2ec9600640 1 -- 172.21.15.6:0/966404245 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2edc0072d0 con 0x557610684700 2024-03-04T09:28:58.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.056+0000 7f2eee394740 10 client.6474 fetch_fsmap learned FSMap version 42 2024-03-04T09:28:58.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.056+0000 7f2eee394740 1 -- 172.21.15.6:0/966404245 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=2+}) v3 -- 0x557610685540 con 0x557610684700 2024-03-04T09:28:58.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.056+0000 7f2ec9600640 1 -- 172.21.15.6:0/966404245 <== mon.0 v2:172.21.15.6:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ed8079950 con 0x557610684700 2024-03-04T09:28:58.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.056+0000 7f2ec9600640 1 -- 172.21.15.6:0/966404245 <== mon.0 v2:172.21.15.6:3300/0 5 ==== fsmap(e 42) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f2edc007470 con 0x557610684700 2024-03-04T09:28:58.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.056+0000 7f2eee394740 10 client.6474 fetch_fsmap finished waiting for FSMap version 42 2024-03-04T09:28:58.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.056+0000 7f2eee394740 10 client.6474 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-04T09:28:58.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.056+0000 7f2eee394740 20 client.6474 populate_metadata read hostname 'smithi006' 2024-03-04T09:28:58.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.056+0000 7f2eee394740 1 --2- 172.21.15.6:0/966404245 >> [v2:172.21.15.146:6836/3919939129,v1:172.21.15.146:6837/3919939129] conn(0x5576106b96b0 0x557610686a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:58.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.057+0000 7f2eee394740 4 client.6474 mds_command: new command op to 5590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T09:28:58.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.057+0000 7f2eee394740 1 -- 172.21.15.6:0/966404245 --> [v2:172.21.15.146:6836/3919939129,v1:172.21.15.146:6837/3919939129] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5576105cdb40 con 0x5576106b96b0 2024-03-04T09:28:58.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.057+0000 7f2ee8c00640 1 --2- 172.21.15.6:0/966404245 >> [v2:172.21.15.146:6836/3919939129,v1:172.21.15.146:6837/3919939129] conn(0x5576106b96b0 0x557610686a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:58.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.058+0000 7f2ee8c00640 1 --2- 172.21.15.6:0/966404245 >> [v2:172.21.15.146:6836/3919939129,v1:172.21.15.146:6837/3919939129] conn(0x5576106b96b0 0x557610686a10 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=53d87c3e04290086 server_cookie=88a4a5f14084a25e in_seq=0 out_seq=0 2024-03-04T09:28:58.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.058+0000 7f2ec9600640 10 client.6474 ms_handle_connect on v2:172.21.15.146:6836/3919939129 2024-03-04T09:28:58.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.060+0000 7f2ec9600640 1 -- 172.21.15.6:0/966404245 <== mds.0 v2:172.21.15.146:6836/3919939129 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5576105cdb40 con 0x5576106b96b0 2024-03-04T09:28:58.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.060+0000 7f2ec9600640 10 client.6474 handle_command_reply: tid=0 2024-03-04T09:28:58.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.060+0000 7f2eee394740 1 client.6474 shutdown 2024-03-04T09:28:58.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.060+0000 7f2eee394740 1 -- 172.21.15.6:0/966404245 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x557610684700 msgr2=0x557610688b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:58.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.060+0000 7f2eee394740 1 --2- 172.21.15.6:0/966404245 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x557610684700 0x557610688b60 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f2ee409e100 tx=0x7f2edc00d810 comp rx=0 tx=0).stop 2024-03-04T09:28:58.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.060+0000 7f2eee394740 1 -- 172.21.15.6:0/966404245 shutdown_connections 2024-03-04T09:28:58.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.060+0000 7f2eee394740 1 --2- 172.21.15.6:0/966404245 >> [v2:172.21.15.146:6836/3919939129,v1:172.21.15.146:6837/3919939129] conn(0x5576106b96b0 0x557610686a10 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:58.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.060+0000 7f2eee394740 1 --2- 172.21.15.6:0/966404245 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x557610684700 0x557610688b60 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:58.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.060+0000 7f2eee394740 1 --2- 172.21.15.6:0/966404245 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5576106a4330 0x5576106841c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:58.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.060+0000 7f2eee394740 1 --2- 172.21.15.6:0/966404245 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x557610683b70 0x55761066da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:58.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.060+0000 7f2eee394740 1 -- 172.21.15.6:0/966404245 >> 172.21.15.6:0/966404245 conn(0x55761065bda0 msgr2=0x5576106a1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:58.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.061+0000 7f2ec9600640 0 client.6474 ms_handle_reset on v2:172.21.15.146:6836/3919939129 2024-03-04T09:28:58.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.061+0000 7f2eee394740 1 -- 172.21.15.6:0/966404245 shutdown_connections 2024-03-04T09:28:58.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.061+0000 7f2eee394740 1 -- 172.21.15.6:0/966404245 wait complete. 2024-03-04T09:28:58.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.061+0000 7f2eee394740 20 client.6474 trim_cache size 0 max 16384 2024-03-04T09:28:58.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.079+0000 7f2eee394740 1 Processor -- start 2024-03-04T09:28:58.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.079+0000 7f2eee394740 1 -- start start 2024-03-04T09:28:58.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.080+0000 7f2eee394740 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x557610683b70 0x557610686b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:58.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.080+0000 7f2eee394740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557610684700 0x5576106c8510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:58.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.080+0000 7f2eee394740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5576106a4330 0x5576106c8a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:58.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.080+0000 7f2eee394740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ee41fb940 con 0x5576106a4330 2024-03-04T09:28:58.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.080+0000 7f2eee394740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ee406e750 con 0x557610684700 2024-03-04T09:28:58.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.080+0000 7f2eee394740 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ee41fb370 con 0x557610683b70 2024-03-04T09:28:58.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.081+0000 7f2ee9600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5576106a4330 0x5576106c8a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:58.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.081+0000 7f2ee9600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5576106a4330 0x5576106c8a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49348/0 (socket says 172.21.15.6:49348) 2024-03-04T09:28:58.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.081+0000 7f2ee9600640 1 -- 172.21.15.6:0/1546158141 learned_addr learned my addr 172.21.15.6:0/1546158141 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:28:58.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.081+0000 7f2ee8c00640 1 --2- 172.21.15.6:0/1546158141 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x557610683b70 0x557610686b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:58.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.081+0000 7f2ee3e00640 1 --2- 172.21.15.6:0/1546158141 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557610684700 0x5576106c8510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:58.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.081+0000 7f2ee9600640 1 -- 172.21.15.6:0/1546158141 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x557610683b70 msgr2=0x557610686b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:58.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.081+0000 7f2ee9600640 1 --2- 172.21.15.6:0/1546158141 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x557610683b70 0x557610686b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:58.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.081+0000 7f2ee9600640 1 -- 172.21.15.6:0/1546158141 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557610684700 msgr2=0x5576106c8510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:58.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.081+0000 7f2ee9600640 1 --2- 172.21.15.6:0/1546158141 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557610684700 0x5576106c8510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:58.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.081+0000 7f2ee9600640 1 -- 172.21.15.6:0/1546158141 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5576105ce270 con 0x5576106a4330 2024-03-04T09:28:58.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.081+0000 7f2ee9600640 1 --2- 172.21.15.6:0/1546158141 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5576106a4330 0x5576106c8a50 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f2edc000f80 tx=0x7f2edc00b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d11d621d7bb6dc04 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:58.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.082+0000 7f2ecbe00640 1 -- 172.21.15.6:0/1546158141 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2edc01e020 con 0x5576106a4330 2024-03-04T09:28:58.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.082+0000 7f2ecbe00640 1 -- 172.21.15.6:0/1546158141 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2edc007470 con 0x5576106a4330 2024-03-04T09:28:58.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.082+0000 7f2ecbe00640 1 -- 172.21.15.6:0/1546158141 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2edc013470 con 0x5576106a4330 2024-03-04T09:28:58.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.082+0000 7f2eee394740 1 -- 172.21.15.6:0/1546158141 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5576106a4330 msgr2=0x5576106c8a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:58.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.082+0000 7f2eee394740 1 --2- 172.21.15.6:0/1546158141 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5576106a4330 0x5576106c8a50 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f2edc000f80 tx=0x7f2edc00b000 comp rx=0 tx=0).stop 2024-03-04T09:28:58.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.082+0000 7f2eee394740 1 -- 172.21.15.6:0/1546158141 shutdown_connections 2024-03-04T09:28:58.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.082+0000 7f2eee394740 1 --2- 172.21.15.6:0/1546158141 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5576106a4330 0x5576106c8a50 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:58.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.082+0000 7f2eee394740 1 --2- 172.21.15.6:0/1546158141 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557610684700 0x5576106c8510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:58.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.082+0000 7f2eee394740 1 --2- 172.21.15.6:0/1546158141 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x557610683b70 0x557610686b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:58.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.082+0000 7f2eee394740 1 -- 172.21.15.6:0/1546158141 >> 172.21.15.6:0/1546158141 conn(0x55761065bda0 msgr2=0x557610689cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:58.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.082+0000 7f2eee394740 1 -- 172.21.15.6:0/1546158141 shutdown_connections 2024-03-04T09:28:58.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.082+0000 7f2eee394740 1 -- 172.21.15.6:0/1546158141 wait complete. 2024-03-04T09:28:58.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.083+0000 7f2eee394740 1 -- start start 2024-03-04T09:28:58.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.083+0000 7f2eee394740 1 Processor -- start 2024-03-04T09:28:58.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.084+0000 7f2eee394740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x557610683b70 0x5576106b7e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:58.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.084+0000 7f2eee394740 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x557610684700 0x557610682ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:58.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.084+0000 7f2eee394740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5576106a4330 0x5576106830e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:58.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.084+0000 7f2eee394740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x5576106b9530 con 0x557610683b70 2024-03-04T09:28:58.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.084+0000 7f2eee394740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x557610569930 con 0x5576106a4330 2024-03-04T09:28:58.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.084+0000 7f2eee394740 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x5576106b9b00 con 0x557610684700 2024-03-04T09:28:58.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.084+0000 7f2eee394740 10 client.0 _ll_register_callbacks cb 0x7f2edc007ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T09:28:58.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.085+0000 7f2ee8c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x557610683b70 0x5576106b7e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:58.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.085+0000 7f2ee8c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x557610683b70 0x5576106b7e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49354/0 (socket says 172.21.15.6:49354) 2024-03-04T09:28:58.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.085+0000 7f2ee8c00640 1 -- 172.21.15.6:0/1157865833 learned_addr learned my addr 172.21.15.6:0/1157865833 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:28:58.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.085+0000 7f2ee9600640 1 --2- 172.21.15.6:0/1157865833 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5576106a4330 0x5576106830e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:58.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.085+0000 7f2ee3e00640 1 --2- 172.21.15.6:0/1157865833 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x557610684700 0x557610682ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:58.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.085+0000 7f2ee8c00640 1 -- 172.21.15.6:0/1157865833 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x557610684700 msgr2=0x557610682ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:58.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.085+0000 7f2ee8c00640 1 --2- 172.21.15.6:0/1157865833 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x557610684700 0x557610682ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:58.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.085+0000 7f2ee8c00640 1 -- 172.21.15.6:0/1157865833 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5576106a4330 msgr2=0x5576106830e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:58.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.085+0000 7f2ee8c00640 1 --2- 172.21.15.6:0/1157865833 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5576106a4330 0x5576106830e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:58.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.085+0000 7f2ee8c00640 1 -- 172.21.15.6:0/1157865833 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55761067e8d0 con 0x557610683b70 2024-03-04T09:28:58.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.085+0000 7f2ee3e00640 1 --2- 172.21.15.6:0/1157865833 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x557610684700 0x557610682ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:28:58.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.086+0000 7f2ee8c00640 1 --2- 172.21.15.6:0/1157865833 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x557610683b70 0x5576106b7e80 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f2ed80798e0 tx=0x7f2ed8006f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16cb6aa2ae5e9527 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:58.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.086+0000 7f2ec9600640 10 client.0 ms_handle_connect on v2:172.21.15.6:3300/0 2024-03-04T09:28:58.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.086+0000 7f2ec9600640 1 -- 172.21.15.6:0/1157865833 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ed8082060 con 0x557610683b70 2024-03-04T09:28:58.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.086+0000 7f2ec9600640 1 -- 172.21.15.6:0/1157865833 <== mon.0 v2:172.21.15.6:3300/0 2 ==== osd_map(68..68 src has 1..68) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2ed8075fc0 con 0x557610683b70 2024-03-04T09:28:58.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.086+0000 7f2eee394740 1 -- 172.21.15.6:0/1157865833 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2edc007470 con 0x557610683b70 2024-03-04T09:28:58.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.086+0000 7f2ec9600640 1 -- 172.21.15.6:0/1157865833 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2ed807daa0 con 0x557610683b70 2024-03-04T09:28:58.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.086+0000 7f2eee394740 10 client.6480 fetch_fsmap learned FSMap version 42 2024-03-04T09:28:58.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.086+0000 7f2eee394740 1 -- 172.21.15.6:0/1157865833 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=2+}) v3 -- 0x55761065cdf0 con 0x557610683b70 2024-03-04T09:28:58.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.086+0000 7f2ec9600640 1 -- 172.21.15.6:0/1157865833 <== mon.0 v2:172.21.15.6:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2edc007470 con 0x557610683b70 2024-03-04T09:28:58.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.086+0000 7f2ec9600640 1 -- 172.21.15.6:0/1157865833 <== mon.0 v2:172.21.15.6:3300/0 5 ==== fsmap(e 42) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f2ed8081bd0 con 0x557610683b70 2024-03-04T09:28:58.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.086+0000 7f2eee394740 10 client.6480 fetch_fsmap finished waiting for FSMap version 42 2024-03-04T09:28:58.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.086+0000 7f2eee394740 10 client.6480 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-04T09:28:58.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.086+0000 7f2eee394740 20 client.6480 populate_metadata read hostname 'smithi006' 2024-03-04T09:28:58.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.086+0000 7f2eee394740 1 --2- 172.21.15.6:0/1157865833 >> [v2:172.21.15.146:6836/3919939129,v1:172.21.15.146:6837/3919939129] conn(0x55761066c000 0x5576106a2610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:58.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.087+0000 7f2eee394740 4 client.6480 mds_command: new command op to 5590 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-04T09:28:58.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.087+0000 7f2eee394740 1 -- 172.21.15.6:0/1157865833 --> [v2:172.21.15.146:6836/3919939129,v1:172.21.15.146:6837/3919939129] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5576105c7e90 con 0x55761066c000 2024-03-04T09:28:58.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.087+0000 7f2ee3e00640 1 --2- 172.21.15.6:0/1157865833 >> [v2:172.21.15.146:6836/3919939129,v1:172.21.15.146:6837/3919939129] conn(0x55761066c000 0x5576106a2610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:58.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.088+0000 7f2ee3e00640 1 --2- 172.21.15.6:0/1157865833 >> [v2:172.21.15.146:6836/3919939129,v1:172.21.15.146:6837/3919939129] conn(0x55761066c000 0x5576106a2610 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e85d5ceef18eea92 server_cookie=2c1802a1884a17b6 in_seq=0 out_seq=0 2024-03-04T09:28:58.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.088+0000 7f2ec9600640 10 client.6480 ms_handle_connect on v2:172.21.15.146:6836/3919939129 2024-03-04T09:28:58.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.089+0000 7f2ec9600640 1 -- 172.21.15.6:0/1157865833 <== mds.0 v2:172.21.15.146:6836/3919939129 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x5576105c7e90 con 0x55761066c000 2024-03-04T09:28:58.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.089+0000 7f2ec9600640 10 client.6480 handle_command_reply: tid=0 2024-03-04T09:28:58.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.089+0000 7f2eee394740 1 client.6480 shutdown 2024-03-04T09:28:58.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.089+0000 7f2eee394740 1 -- 172.21.15.6:0/1157865833 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x557610683b70 msgr2=0x5576106b7e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:58.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.089+0000 7f2eee394740 1 --2- 172.21.15.6:0/1157865833 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x557610683b70 0x5576106b7e80 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f2ed80798e0 tx=0x7f2ed8006f40 comp rx=0 tx=0).stop 2024-03-04T09:28:58.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.089+0000 7f2eee394740 1 -- 172.21.15.6:0/1157865833 shutdown_connections 2024-03-04T09:28:58.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.089+0000 7f2eee394740 1 --2- 172.21.15.6:0/1157865833 >> [v2:172.21.15.146:6836/3919939129,v1:172.21.15.146:6837/3919939129] conn(0x55761066c000 0x5576106a2610 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:58.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.089+0000 7f2eee394740 1 --2- 172.21.15.6:0/1157865833 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5576106a4330 0x5576106830e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:58.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.089+0000 7f2eee394740 1 --2- 172.21.15.6:0/1157865833 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x557610684700 0x557610682ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:58.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.089+0000 7f2eee394740 1 --2- 172.21.15.6:0/1157865833 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x557610683b70 0x5576106b7e80 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:58.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.089+0000 7f2ec9600640 0 client.6480 ms_handle_reset on v2:172.21.15.146:6836/3919939129 2024-03-04T09:28:58.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.089+0000 7f2eee394740 1 -- 172.21.15.6:0/1157865833 >> 172.21.15.6:0/1157865833 conn(0x55761065bda0 msgr2=0x5576106a03b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:58.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.089+0000 7f2eee394740 1 -- 172.21.15.6:0/1157865833 shutdown_connections 2024-03-04T09:28:58.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.089+0000 7f2eee394740 1 -- 172.21.15.6:0/1157865833 wait complete. 2024-03-04T09:28:58.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.089+0000 7f2eee394740 20 client.6480 trim_cache size 0 max 16384 2024-03-04T09:28:58.091 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-04T09:28:58.092 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_fsid": "edce0578-5ae4-4402-b45a-552bdef84d7c", 2024-03-04T09:28:58.092 INFO:teuthology.orchestra.run.smithi006.stdout: "whoami": 0, 2024-03-04T09:28:58.092 INFO:teuthology.orchestra.run.smithi006.stdout: "id": 5590, 2024-03-04T09:28:58.092 INFO:teuthology.orchestra.run.smithi006.stdout: "want_state": "up:active", 2024-03-04T09:28:58.092 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "up:active", 2024-03-04T09:28:58.092 INFO:teuthology.orchestra.run.smithi006.stdout: "fs_name": "cephfs", 2024-03-04T09:28:58.092 INFO:teuthology.orchestra.run.smithi006.stdout: "rank_uptime": 6.8241366250000004, 2024-03-04T09:28:58.092 INFO:teuthology.orchestra.run.smithi006.stdout: "mdsmap_epoch": 42, 2024-03-04T09:28:58.092 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap_epoch": 68, 2024-03-04T09:28:58.092 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap_epoch_barrier": 68, 2024-03-04T09:28:58.092 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 13.209569773 2024-03-04T09:28:58.092 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-04T09:28:58.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.090+0000 7f2eeb600640 1 -- 172.21.15.6:0/4065709101 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2ec406b500 msgr2=0x7f2ec406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:58.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.090+0000 7f2eeb600640 1 --2- 172.21.15.6:0/4065709101 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2ec406b500 0x7f2ec406d9c0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f2ed0001920 tx=0x7f2ed0001890 comp rx=0 tx=0).stop 2024-03-04T09:28:58.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.090+0000 7f2eeb600640 1 -- 172.21.15.6:0/4065709101 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2ecc003a20 msgr2=0x7f2ee4156480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:58.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.090+0000 7f2eeb600640 1 --2- 172.21.15.6:0/4065709101 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2ecc003a20 0x7f2ee4156480 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f2ed8000c00 tx=0x7f2ed8004730 comp rx=0 tx=0).stop 2024-03-04T09:28:58.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.090+0000 7f2eeb600640 1 -- 172.21.15.6:0/4065709101 shutdown_connections 2024-03-04T09:28:58.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.090+0000 7f2eeb600640 1 --2- 172.21.15.6:0/4065709101 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2ec406b500 0x7f2ec406d9c0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:58.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.090+0000 7f2eeb600640 1 --2- 172.21.15.6:0/4065709101 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2ee409d210 0x7f2ee409d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:58.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.090+0000 7f2eeb600640 1 --2- 172.21.15.6:0/4065709101 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2ee4158b50 0x7f2ee409aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:58.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.090+0000 7f2eeb600640 1 --2- 172.21.15.6:0/4065709101 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2ecc003a20 0x7f2ee4156480 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:58.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.090+0000 7f2eeb600640 1 -- 172.21.15.6:0/4065709101 >> 172.21.15.6:0/4065709101 conn(0x7f2ee406afb0 msgr2=0x7f2ee40cec30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:58.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.090+0000 7f2eeb600640 1 -- 172.21.15.6:0/4065709101 shutdown_connections 2024-03-04T09:28:58.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.090+0000 7f2eeb600640 1 -- 172.21.15.6:0/4065709101 wait complete. 2024-03-04T09:28:58.105 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-04T09:28:58.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.320+0000 7f6640600640 1 Processor -- start 2024-03-04T09:28:58.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.321+0000 7f6640600640 1 -- start start 2024-03-04T09:28:58.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.322+0000 7f6640600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f6638161670 0x7f6638070c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:58.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.322+0000 7f6640600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f66380ae0a0 0x7f66380711b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:58.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.323+0000 7f6640600640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66380ce9a0 con 0x7f663806e240 2024-03-04T09:28:58.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.323+0000 7f6640600640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66380730d0 con 0x7f6638161670 2024-03-04T09:28:58.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.323+0000 7f6640600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66380cec80 con 0x7f66380ae0a0 2024-03-04T09:28:58.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.323+0000 7f663dc00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f6638161670 0x7f6638070c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:58.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.323+0000 7f663dc00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f6638161670 0x7f6638070c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49364/0 (socket says 172.21.15.6:49364) 2024-03-04T09:28:58.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.323+0000 7f663dc00640 1 -- 172.21.15.6:0/1401595935 learned_addr learned my addr 172.21.15.6:0/1401595935 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:28:58.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.323+0000 7f663e600640 1 --2- 172.21.15.6:0/1401595935 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f66380ae0a0 0x7f66380711b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:58.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.323+0000 7f663dc00640 1 -- 172.21.15.6:0/1401595935 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f66380ae0a0 msgr2=0x7f66380711b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:58.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.323+0000 7f663dc00640 1 --2- 172.21.15.6:0/1401595935 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f66380ae0a0 0x7f66380711b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:58.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.323+0000 7f663dc00640 1 -- 172.21.15.6:0/1401595935 >> v1:172.21.15.146:6789/0 conn(0x7f663806e240 legacy=0x7f663806cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:58.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.324+0000 7f663dc00640 1 -- 172.21.15.6:0/1401595935 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66380cef60 con 0x7f6638161670 2024-03-04T09:28:58.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.325+0000 7f663dc00640 1 --2- 172.21.15.6:0/1401595935 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6638161670 0x7f6638070c70 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f662c009870 tx=0x7f662c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b630190ac6baa640 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:58.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.325+0000 7f6637e00640 1 -- 172.21.15.6:0/1401595935 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f662c034070 con 0x7f6638161670 2024-03-04T09:28:58.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.325+0000 7f6637e00640 1 -- 172.21.15.6:0/1401595935 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f662c030840 con 0x7f6638161670 2024-03-04T09:28:58.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.325+0000 7f6637e00640 1 -- 172.21.15.6:0/1401595935 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f662c030c10 con 0x7f6638161670 2024-03-04T09:28:58.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.326+0000 7f6640600640 1 -- 172.21.15.6:0/1401595935 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6638161670 msgr2=0x7f6638070c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:58.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.326+0000 7f6640600640 1 --2- 172.21.15.6:0/1401595935 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6638161670 0x7f6638070c70 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f662c009870 tx=0x7f662c02b980 comp rx=0 tx=0).stop 2024-03-04T09:28:58.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.326+0000 7f6640600640 1 -- 172.21.15.6:0/1401595935 shutdown_connections 2024-03-04T09:28:58.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.326+0000 7f6640600640 1 --2- 172.21.15.6:0/1401595935 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f66380ae0a0 0x7f66380711b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:58.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.326+0000 7f6640600640 1 --2- 172.21.15.6:0/1401595935 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6638161670 0x7f6638070c70 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:58.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.326+0000 7f6640600640 1 -- 172.21.15.6:0/1401595935 >> 172.21.15.6:0/1401595935 conn(0x7f663806afb0 msgr2=0x7f663806b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:58.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.326+0000 7f6640600640 1 -- 172.21.15.6:0/1401595935 shutdown_connections 2024-03-04T09:28:58.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.326+0000 7f6640600640 1 -- 172.21.15.6:0/1401595935 wait complete. 2024-03-04T09:28:58.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.328+0000 7f6640600640 1 Processor -- start 2024-03-04T09:28:58.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.328+0000 7f6640600640 1 -- start start 2024-03-04T09:28:58.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.329+0000 7f6640600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f663806e240 0x7f6638148230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:58.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.329+0000 7f6640600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f66380ae0a0 0x7f663814a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:58.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.329+0000 7f6640600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6638161670 0x7f663814ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:58.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.329+0000 7f6640600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66380ce600 con 0x7f6638161670 2024-03-04T09:28:58.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.329+0000 7f6640600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66380cf520 con 0x7f663806e240 2024-03-04T09:28:58.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.329+0000 7f6640600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66380ce780 con 0x7f66380ae0a0 2024-03-04T09:28:58.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.329+0000 7f663e600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6638161670 0x7f663814ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:58.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.329+0000 7f663e600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6638161670 0x7f663814ae90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49370/0 (socket says 172.21.15.6:49370) 2024-03-04T09:28:58.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.329+0000 7f663e600640 1 -- 172.21.15.6:0/2558156012 learned_addr learned my addr 172.21.15.6:0/2558156012 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:28:58.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.330+0000 7f663dc00640 1 --2- 172.21.15.6:0/2558156012 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f663806e240 0x7f6638148230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:58.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.330+0000 7f663d200640 1 --2- 172.21.15.6:0/2558156012 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f66380ae0a0 0x7f663814a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:58.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.330+0000 7f663e600640 1 -- 172.21.15.6:0/2558156012 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f66380ae0a0 msgr2=0x7f663814a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:58.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.330+0000 7f663e600640 1 --2- 172.21.15.6:0/2558156012 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f66380ae0a0 0x7f663814a950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:58.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.330+0000 7f663e600640 1 -- 172.21.15.6:0/2558156012 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f663806e240 msgr2=0x7f6638148230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:58.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.330+0000 7f663e600640 1 --2- 172.21.15.6:0/2558156012 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f663806e240 0x7f6638148230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:58.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.330+0000 7f663e600640 1 -- 172.21.15.6:0/2558156012 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f663815da20 con 0x7f6638161670 2024-03-04T09:28:58.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.330+0000 7f663d200640 1 --2- 172.21.15.6:0/2558156012 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f66380ae0a0 0x7f663814a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:28:58.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.330+0000 7f663e600640 1 --2- 172.21.15.6:0/2558156012 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6638161670 0x7f663814ae90 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f6630007e40 tx=0x7f663000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:58.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.331+0000 7f6635600640 1 -- 172.21.15.6:0/2558156012 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6630014620 con 0x7f6638161670 2024-03-04T09:28:58.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.331+0000 7f6635600640 1 -- 172.21.15.6:0/2558156012 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6630019070 con 0x7f6638161670 2024-03-04T09:28:58.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.331+0000 7f6635600640 1 -- 172.21.15.6:0/2558156012 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6630014a20 con 0x7f6638161670 2024-03-04T09:28:58.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.331+0000 7f6640600640 1 -- 172.21.15.6:0/2558156012 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f663815dd00 con 0x7f6638161670 2024-03-04T09:28:58.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.331+0000 7f6640600640 1 -- 172.21.15.6:0/2558156012 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f663815e2c0 con 0x7f6638161670 2024-03-04T09:28:58.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.332+0000 7f6640600640 1 -- 172.21.15.6:0/2558156012 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6638069510 con 0x7f6638161670 2024-03-04T09:28:58.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.333+0000 7f6635600640 1 -- 172.21.15.6:0/2558156012 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6630014bc0 con 0x7f6638161670 2024-03-04T09:28:58.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.334+0000 7f6635600640 1 --2- 172.21.15.6:0/2558156012 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f661806b4d0 0x7f661806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:58.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.334+0000 7f6635600640 1 -- 172.21.15.6:0/2558156012 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f663008aa90 con 0x7f6638161670 2024-03-04T09:28:58.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.335+0000 7f663dc00640 1 --2- 172.21.15.6:0/2558156012 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f661806b4d0 0x7f661806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:58.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.336+0000 7f663dc00640 1 --2- 172.21.15.6:0/2558156012 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f661806b4d0 0x7f661806d990 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f662c036000 tx=0x7f662c03b040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:58.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.338+0000 7f6635600640 1 -- 172.21.15.6:0/2558156012 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f66300599a0 con 0x7f6638161670 2024-03-04T09:28:58.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.581+0000 7f6640600640 1 -- 172.21.15.6:0/2558156012 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f663804f3a0 con 0x7f6638161670 2024-03-04T09:28:58.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.582+0000 7f6635600640 1 -- 172.21.15.6:0/2558156012 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v42) v1 ==== 53+0+73 (secure 0 0 0) 0x7f66300592a0 con 0x7f6638161670 2024-03-04T09:28:58.579 INFO:teuthology.orchestra.run.smithi006.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-04T09:28:58.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.585+0000 7f6640600640 1 -- 172.21.15.6:0/2558156012 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f661806b4d0 msgr2=0x7f661806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:58.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.585+0000 7f6640600640 1 --2- 172.21.15.6:0/2558156012 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f661806b4d0 0x7f661806d990 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f662c036000 tx=0x7f662c03b040 comp rx=0 tx=0).stop 2024-03-04T09:28:58.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.585+0000 7f6640600640 1 -- 172.21.15.6:0/2558156012 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6638161670 msgr2=0x7f663814ae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:58.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.585+0000 7f6640600640 1 --2- 172.21.15.6:0/2558156012 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6638161670 0x7f663814ae90 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f6630007e40 tx=0x7f663000e4b0 comp rx=0 tx=0).stop 2024-03-04T09:28:58.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.585+0000 7f6640600640 1 -- 172.21.15.6:0/2558156012 shutdown_connections 2024-03-04T09:28:58.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.585+0000 7f6640600640 1 --2- 172.21.15.6:0/2558156012 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6638161670 0x7f663814ae90 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:58.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.585+0000 7f6640600640 1 --2- 172.21.15.6:0/2558156012 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f661806b4d0 0x7f661806d990 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:58.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.585+0000 7f6640600640 1 --2- 172.21.15.6:0/2558156012 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f66380ae0a0 0x7f663814a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:58.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.585+0000 7f6640600640 1 --2- 172.21.15.6:0/2558156012 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f663806e240 0x7f6638148230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:58.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.585+0000 7f6640600640 1 -- 172.21.15.6:0/2558156012 >> 172.21.15.6:0/2558156012 conn(0x7f663806afb0 msgr2=0x7f66380b2c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:58.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.585+0000 7f6640600640 1 -- 172.21.15.6:0/2558156012 shutdown_connections 2024-03-04T09:28:58.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:58.585+0000 7f6640600640 1 -- 172.21.15.6:0/2558156012 wait complete. 2024-03-04T09:28:58.600 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-04T09:28:58.600 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-04T09:28:58.600 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi006.front.sepia.ceph.com 2024-03-04T09:28:58.600 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-04T09:28:58.600 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-04T09:28:58.600 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-04T09:28:58.600 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-03-04T09:28:58.600 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-04T09:28:58.600 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-04T09:28:58.601 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:28:58.601 DEBUG:teuthology.orchestra.run.smithi006:> ip addr 2024-03-04T09:28:58.649 INFO:teuthology.orchestra.run.smithi006.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-04T09:28:58.649 INFO:teuthology.orchestra.run.smithi006.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-04T09:28:58.649 INFO:teuthology.orchestra.run.smithi006.stdout: inet 127.0.0.1/8 scope host lo 2024-03-04T09:28:58.649 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-04T09:28:58.650 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 ::1/128 scope host 2024-03-04T09:28:58.650 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-04T09:28:58.650 INFO:teuthology.orchestra.run.smithi006.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-04T09:28:58.650 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:6b:fe:e6 brd ff:ff:ff:ff:ff:ff 2024-03-04T09:28:58.650 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp7s0f0 2024-03-04T09:28:58.650 INFO:teuthology.orchestra.run.smithi006.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-04T09:28:58.650 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:6b:fe:e7 brd ff:ff:ff:ff:ff:ff 2024-03-04T09:28:58.650 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp7s0f1 2024-03-04T09:28:58.650 INFO:teuthology.orchestra.run.smithi006.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-04T09:28:58.650 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:bd:15:e4 brd ff:ff:ff:ff:ff:ff 2024-03-04T09:28:58.650 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp1s0f0 2024-03-04T09:28:58.650 INFO:teuthology.orchestra.run.smithi006.stdout: inet 172.21.15.6/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-04T09:28:58.650 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft 42395sec preferred_lft 42395sec 2024-03-04T09:28:58.650 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::ec4:7aff:febd:15e4/64 scope link noprefixroute 2024-03-04T09:28:58.651 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-04T09:28:58.651 INFO:teuthology.orchestra.run.smithi006.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-04T09:28:58.651 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:bd:15:e5 brd ff:ff:ff:ff:ff:ff 2024-03-04T09:28:58.651 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp1s0f1 2024-03-04T09:28:58.651 INFO:teuthology.orchestra.run.smithi006.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-04T09:28:58.651 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 5e:b4:59:dc:42:58 brd ff:ff:ff:ff:ff:ff 2024-03-04T09:28:58.651 INFO:teuthology.orchestra.run.smithi006.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-04T09:28:58.651 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-04T09:28:58.651 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::c6f:adff:fefe:e00d/64 scope link proto kernel_ll 2024-03-04T09:28:58.651 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-04T09:28:58.651 INFO:teuthology.orchestra.run.smithi006.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-04T09:28:58.651 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 7e:67:6d:ea:f9:ff brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-04T09:28:58.651 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::7c67:6dff:feea:f9ff/64 scope link proto kernel_ll 2024-03-04T09:28:58.651 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-04T09:28:58.651 INFO:teuthology.orchestra.run.smithi006.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-04T09:28:58.652 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 5e:b4:59:dc:42:58 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-04T09:28:58.652 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::5cb4:59ff:fedc:4258/64 scope link proto kernel_ll 2024-03-04T09:28:58.652 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-04T09:28:58.652 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:28:58.652 DEBUG:teuthology.orchestra.run.smithi006:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-04T09:28:58.744 INFO:teuthology.orchestra.run.smithi006.stdout:1 2024-03-04T09:28:58.749 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:28:58.749 DEBUG:teuthology.orchestra.run.smithi006:> route 2024-03-04T09:28:58.805 INFO:teuthology.orchestra.run.smithi006.stdout:Kernel IP routing table 2024-03-04T09:28:58.806 INFO:teuthology.orchestra.run.smithi006.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-04T09:28:58.806 INFO:teuthology.orchestra.run.smithi006.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-04T09:28:58.806 INFO:teuthology.orchestra.run.smithi006.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-04T09:28:58.806 INFO:teuthology.orchestra.run.smithi006.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-04T09:28:58.806 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:28:58.806 DEBUG:teuthology.orchestra.run.smithi006:> (cd / && exec bash -c ' 2024-03-04T09:28:58.807 DEBUG:teuthology.orchestra.run.smithi006:> set -e 2024-03-04T09:28:58.807 DEBUG:teuthology.orchestra.run.smithi006:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-04T09:28:58.807 DEBUG:teuthology.orchestra.run.smithi006:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-04T09:28:58.807 DEBUG:teuthology.orchestra.run.smithi006:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-04T09:28:58.807 DEBUG:teuthology.orchestra.run.smithi006:> ') 2024-03-04T09:28:59.004 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:28:59.005 DEBUG:teuthology.orchestra.run.smithi006:> ip netns list 2024-03-04T09:28:59.058 INFO:teuthology.orchestra.run.smithi006.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-04T09:28:59.058 INFO:teuthology.orchestra.run.smithi006.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-04T09:28:59.060 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-04T09:28:59.060 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:28:59.060 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip link set brx.0 up 2024-03-04T09:28:59.159 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-04T09:28:59.160 DEBUG:teuthology.orchestra.run.smithi006:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-04T09:28:59.215 INFO:teuthology.orchestra.run.smithi006.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-04T09:28:59.215 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-04T09:28:59.215 DEBUG:teuthology.orchestra.run.smithi006:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-04T09:28:59.271 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-03-04T09:28:59.271 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:28:59.271 DEBUG:teuthology.orchestra.run.smithi006:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-04T09:28:59.419 INFO:teuthology.orchestra.run.smithi006.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-04T09:28:59.419 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-04T09:28:59.458 INFO:teuthology.orchestra.run.smithi006.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-04T09:28:59.458 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-04T09:28:59.458 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: resolved to: "172.21.15.6:3300,172.21.15.146:3300,172.21.15.146:3301" 2024-03-04T09:28:59.458 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: trying mount with new device syntax: 0@edce0578-5ae4-4402-b45a-552bdef84d7c.cephfs=/ 2024-03-04T09:28:59.458 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.6:3300/172.21.15.146:3300/172.21.15.146:3301" will pass to kernel 2024-03-04T09:28:59.461 INFO:teuthology.orchestra.run.smithi006.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-04T09:28:59.462 INFO:teuthology.orchestra.run.smithi006.stdout: You just mounted a file system that supports labels which does not 2024-03-04T09:28:59.462 INFO:teuthology.orchestra.run.smithi006.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-04T09:28:59.462 INFO:teuthology.orchestra.run.smithi006.stdout: applications will generate AVC messages and not be allowed access to 2024-03-04T09:28:59.462 INFO:teuthology.orchestra.run.smithi006.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-04T09:28:59.467 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-04T09:28:59.467 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:28:59.467 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-04T09:28:59.550 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:28:59.550 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-04T09:28:59.628 INFO:teuthology.orchestra.run.smithi006.stdout:client5282 2024-03-04T09:28:59.628 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-04T09:28:59.628 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T09:28:59.685 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds mds_bal_fragment_size_max 10 2024-03-04T09:28:59.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:59.907+0000 7fc7a3e00640 1 Processor -- start 2024-03-04T09:28:59.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:59.907+0000 7fc7a3e00640 1 -- start start 2024-03-04T09:28:59.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:59.909+0000 7fc7a3e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fc79c144560 0x7fc79c06e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:59.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:59.909+0000 7fc7a3e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc79c147510 0x7fc79c06afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:59.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:59.909+0000 7fc7a3e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc79c161d20 con 0x7fc79c06b4f0 2024-03-04T09:28:59.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:59.909+0000 7fc7a3e00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc79c162ae0 con 0x7fc79c144560 2024-03-04T09:28:59.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:59.909+0000 7fc7a3e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc79c06b920 con 0x7fc79c147510 2024-03-04T09:28:59.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:59.909+0000 7fc7a1e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fc79c06b4f0 0x7fc79c0cd530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:38024/0 (socket says 172.21.15.6:38024) 2024-03-04T09:28:59.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:59.909+0000 7fc7a1e00640 1 -- 172.21.15.6:0/1979748847 learned_addr learned my addr 172.21.15.6:0/1979748847 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:28:59.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:59.910+0000 7fc7a0a00640 1 --2- 172.21.15.6:0/1979748847 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc79c147510 0x7fc79c06afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:59.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:59.910+0000 7fc7a1400640 1 --2- 172.21.15.6:0/1979748847 >> v2:172.21.15.146:3300/0 conn(0x7fc79c144560 0x7fc79c06e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:59.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:59.910+0000 7fc79be00640 1 -- 172.21.15.6:0/1979748847 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2198312466 0 0) 0x7fc79c161d20 con 0x7fc79c06b4f0 2024-03-04T09:28:59.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:59.910+0000 7fc79be00640 1 -- 172.21.15.6:0/1979748847 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc784003520 con 0x7fc79c06b4f0 2024-03-04T09:28:59.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:59.910+0000 7fc79be00640 1 -- 172.21.15.6:0/1979748847 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 925559283 0 0) 0x7fc784003520 con 0x7fc79c06b4f0 2024-03-04T09:28:59.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:59.910+0000 7fc79be00640 1 -- 172.21.15.6:0/1979748847 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc79c147510 msgr2=0x7fc79c06afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:59.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:59.910+0000 7fc79be00640 1 --2- 172.21.15.6:0/1979748847 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc79c147510 0x7fc79c06afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:59.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:59.910+0000 7fc79be00640 1 -- 172.21.15.6:0/1979748847 >> v2:172.21.15.146:3300/0 conn(0x7fc79c144560 msgr2=0x7fc79c06e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:59.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:59.910+0000 7fc79be00640 1 --2- 172.21.15.6:0/1979748847 >> v2:172.21.15.146:3300/0 conn(0x7fc79c144560 0x7fc79c06e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:59.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:59.910+0000 7fc79be00640 1 -- 172.21.15.6:0/1979748847 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc79c162000 con 0x7fc79c06b4f0 2024-03-04T09:28:59.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:59.910+0000 7fc79be00640 1 -- 172.21.15.6:0/1979748847 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7fc7940019f0 con 0x7fc79c06b4f0 2024-03-04T09:28:59.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:59.910+0000 7fc79be00640 1 -- 172.21.15.6:0/1979748847 >> v1:172.21.15.6:6789/0 conn(0x7fc79c06b4f0 legacy=0x7fc79c0cd530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:59.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:59.910+0000 7fc79be00640 1 --2- 172.21.15.6:0/1979748847 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc7840039b0 0x7fc784003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:59.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:59.910+0000 7fc79be00640 1 --2- 172.21.15.6:0/1979748847 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc784004800 0x7fc784004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:59.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:59.911+0000 7fc79be00640 1 --2- 172.21.15.6:0/1979748847 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc79c147510 0x7fc7840051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:59.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:59.911+0000 7fc79be00640 1 -- 172.21.15.6:0/1979748847 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc79c162ae0 con 0x7fc7840039b0 2024-03-04T09:28:59.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:59.911+0000 7fc79be00640 1 -- 172.21.15.6:0/1979748847 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc79c06b920 con 0x7fc784004800 2024-03-04T09:28:59.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:59.911+0000 7fc79be00640 1 -- 172.21.15.6:0/1979748847 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc78400ba40 con 0x7fc79c147510 2024-03-04T09:28:59.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:59.911+0000 7fc7a1400640 1 --2- 172.21.15.6:0/1979748847 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc7840039b0 0x7fc784003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:59.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:59.911+0000 7fc7a0a00640 1 --2- 172.21.15.6:0/1979748847 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc784004800 0x7fc784004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:59.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:59.911+0000 7fc7a1e00640 1 --2- 172.21.15.6:0/1979748847 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc79c147510 0x7fc7840051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:59.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:59.911+0000 7fc7a1400640 1 -- 172.21.15.6:0/1979748847 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc79c147510 msgr2=0x7fc7840051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:59.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:59.911+0000 7fc7a1400640 1 --2- 172.21.15.6:0/1979748847 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc79c147510 0x7fc7840051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:59.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:59.911+0000 7fc7a1400640 1 -- 172.21.15.6:0/1979748847 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc784004800 msgr2=0x7fc784004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:28:59.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:59.911+0000 7fc7a1400640 1 --2- 172.21.15.6:0/1979748847 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc784004800 0x7fc784004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:59.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:59.911+0000 7fc7a1400640 1 -- 172.21.15.6:0/1979748847 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc784003520 con 0x7fc7840039b0 2024-03-04T09:28:59.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:59.912+0000 7fc7a1400640 1 --2- 172.21.15.6:0/1979748847 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc7840039b0 0x7fc784003dd0 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7fc790009ac0 tx=0x7fc784005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e0888ac8c697010 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:59.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:59.913+0000 7fc79be00640 1 -- 172.21.15.6:0/1979748847 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc790039070 con 0x7fc7840039b0 2024-03-04T09:28:59.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:59.913+0000 7fc79be00640 1 -- 172.21.15.6:0/1979748847 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc79002c710 con 0x7fc7840039b0 2024-03-04T09:28:59.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:59.913+0000 7fc79be00640 1 -- 172.21.15.6:0/1979748847 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc790009690 con 0x7fc7840039b0 2024-03-04T09:28:59.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:59.913+0000 7fc7a3e00640 1 -- 172.21.15.6:0/1979748847 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc7840039b0 msgr2=0x7fc784003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:59.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:59.913+0000 7fc7a3e00640 1 --2- 172.21.15.6:0/1979748847 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc7840039b0 0x7fc784003dd0 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7fc790009ac0 tx=0x7fc784005830 comp rx=0 tx=0).stop 2024-03-04T09:28:59.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:59.913+0000 7fc7a1e00640 1 -- 172.21.15.6:0/1979748847 reap_dead start 2024-03-04T09:28:59.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:59.914+0000 7fc7a3e00640 1 -- 172.21.15.6:0/1979748847 shutdown_connections 2024-03-04T09:28:59.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:59.914+0000 7fc7a3e00640 1 -- 172.21.15.6:0/1979748847 >> 172.21.15.6:0/1979748847 conn(0x7fc79c0b2860 msgr2=0x7fc79c0b2c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:28:59.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:59.914+0000 7fc7a3e00640 1 -- 172.21.15.6:0/1979748847 shutdown_connections 2024-03-04T09:28:59.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:59.914+0000 7fc7a3e00640 1 -- 172.21.15.6:0/1979748847 wait complete. 2024-03-04T09:28:59.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:59.915+0000 7fc7a3e00640 1 Processor -- start 2024-03-04T09:28:59.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:59.916+0000 7fc7a3e00640 1 -- start start 2024-03-04T09:28:59.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:59.916+0000 7fc7a3e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc7840039b0 0x7fc79c145600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:59.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:59.916+0000 7fc7a3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc79c131000 0x7fc79c1f3870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:59.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:59.917+0000 7fc7a3e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc79c146880 0x7fc79c0cc340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:59.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:59.917+0000 7fc7a3e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc79c06efd0 con 0x7fc79c131000 2024-03-04T09:28:59.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:59.917+0000 7fc7a3e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc79c1625c0 con 0x7fc7840039b0 2024-03-04T09:28:59.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:59.917+0000 7fc7a3e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc79c072cb0 con 0x7fc79c146880 2024-03-04T09:28:59.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:59.917+0000 7fc7a0a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc79c131000 0x7fc79c1f3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:59.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:59.917+0000 7fc7a0a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc79c131000 0x7fc79c1f3870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:56902/0 (socket says 172.21.15.6:56902) 2024-03-04T09:28:59.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:59.917+0000 7fc7a0a00640 1 -- 172.21.15.6:0/243617613 learned_addr learned my addr 172.21.15.6:0/243617613 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:28:59.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:59.917+0000 7fc7a1e00640 1 --2- 172.21.15.6:0/243617613 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc79c146880 0x7fc79c0cc340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:59.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:59.917+0000 7fc7a1400640 1 --2- 172.21.15.6:0/243617613 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc7840039b0 0x7fc79c145600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:59.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:59.917+0000 7fc7a0a00640 1 -- 172.21.15.6:0/243617613 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc79c146880 msgr2=0x7fc79c0cc340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:59.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:59.917+0000 7fc7a0a00640 1 --2- 172.21.15.6:0/243617613 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc79c146880 0x7fc79c0cc340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:59.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:59.917+0000 7fc7a0a00640 1 -- 172.21.15.6:0/243617613 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc7840039b0 msgr2=0x7fc79c145600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:28:59.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:59.917+0000 7fc7a0a00640 1 --2- 172.21.15.6:0/243617613 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc7840039b0 0x7fc79c145600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:28:59.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:59.917+0000 7fc7a0a00640 1 -- 172.21.15.6:0/243617613 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc79c15e6f0 con 0x7fc79c131000 2024-03-04T09:28:59.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:59.918+0000 7fc7a0a00640 1 --2- 172.21.15.6:0/243617613 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc79c131000 0x7fc79c1f3870 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7fc788011600 tx=0x7fc788011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:59.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:59.918+0000 7fc799600640 1 -- 172.21.15.6:0/243617613 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc78800f920 con 0x7fc79c131000 2024-03-04T09:28:59.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:59.918+0000 7fc799600640 1 -- 172.21.15.6:0/243617613 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc78800fac0 con 0x7fc79c131000 2024-03-04T09:28:59.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:59.918+0000 7fc7a3e00640 1 -- 172.21.15.6:0/243617613 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc79c15e990 con 0x7fc79c131000 2024-03-04T09:28:59.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:59.918+0000 7fc799600640 1 -- 172.21.15.6:0/243617613 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc78800fe70 con 0x7fc79c131000 2024-03-04T09:28:59.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:59.918+0000 7fc7a3e00640 1 -- 172.21.15.6:0/243617613 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc79c15ef50 con 0x7fc79c131000 2024-03-04T09:28:59.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:59.919+0000 7fc7a3e00640 1 -- 172.21.15.6:0/243617613 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc79c04f330 con 0x7fc79c131000 2024-03-04T09:28:59.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:59.921+0000 7fc799600640 1 -- 172.21.15.6:0/243617613 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc788015460 con 0x7fc79c131000 2024-03-04T09:28:59.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:59.921+0000 7fc799600640 1 --2- 172.21.15.6:0/243617613 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc77c06b4d0 0x7fc77c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:28:59.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:59.922+0000 7fc799600640 1 -- 172.21.15.6:0/243617613 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc78808ac30 con 0x7fc79c131000 2024-03-04T09:28:59.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:59.922+0000 7fc7a1400640 1 --2- 172.21.15.6:0/243617613 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc77c06b4d0 0x7fc77c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:28:59.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:59.923+0000 7fc7a1400640 1 --2- 172.21.15.6:0/243617613 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc77c06b4d0 0x7fc77c06d990 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fc790009ac0 tx=0x7fc790030000 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:28:59.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:28:59.927+0000 7fc799600640 1 -- 172.21.15.6:0/243617613 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc788019030 con 0x7fc79c131000 2024-03-04T09:29:00.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:00.127+0000 7fc7a3e00640 1 -- 172.21.15.6:0/243617613 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command([{prefix=config set, name=mds_bal_fragment_size_max}] v 0) v1 -- 0x7fc79c147bc0 con 0x7fc79c131000 2024-03-04T09:29:00.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:00.132+0000 7fc799600640 1 -- 172.21.15.6:0/243617613 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_fragment_size_max}]=0 v23)=0 v23) v1 ==== 124+0+0 (secure 0 0 0) 0x7fc788059a10 con 0x7fc79c131000 2024-03-04T09:29:00.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:00.135+0000 7fc7a3e00640 1 -- 172.21.15.6:0/243617613 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc77c06b4d0 msgr2=0x7fc77c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:00.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:00.135+0000 7fc7a3e00640 1 --2- 172.21.15.6:0/243617613 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc77c06b4d0 0x7fc77c06d990 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fc790009ac0 tx=0x7fc790030000 comp rx=0 tx=0).stop 2024-03-04T09:29:00.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:00.135+0000 7fc7a3e00640 1 -- 172.21.15.6:0/243617613 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc79c131000 msgr2=0x7fc79c1f3870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:00.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:00.135+0000 7fc7a3e00640 1 --2- 172.21.15.6:0/243617613 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc79c131000 0x7fc79c1f3870 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7fc788011600 tx=0x7fc788011ac0 comp rx=0 tx=0).stop 2024-03-04T09:29:00.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:00.135+0000 7fc7a3e00640 1 -- 172.21.15.6:0/243617613 shutdown_connections 2024-03-04T09:29:00.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:00.135+0000 7fc7a3e00640 1 --2- 172.21.15.6:0/243617613 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc77c06b4d0 0x7fc77c06d990 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:00.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:00.135+0000 7fc7a3e00640 1 --2- 172.21.15.6:0/243617613 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc79c146880 0x7fc79c0cc340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:00.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:00.135+0000 7fc7a3e00640 1 --2- 172.21.15.6:0/243617613 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc79c131000 0x7fc79c1f3870 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:00.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:00.135+0000 7fc7a3e00640 1 --2- 172.21.15.6:0/243617613 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc7840039b0 0x7fc79c145600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:00.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:00.136+0000 7fc7a3e00640 1 -- 172.21.15.6:0/243617613 >> 172.21.15.6:0/243617613 conn(0x7fc79c0b2860 msgr2=0x7fc79c0ceec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:29:00.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:00.136+0000 7fc7a3e00640 1 -- 172.21.15.6:0/243617613 shutdown_connections 2024-03-04T09:29:00.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:00.136+0000 7fc7a3e00640 1 -- 172.21.15.6:0/243617613 wait complete. 2024-03-04T09:29:09.576 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:29:09.585 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:29:10.152 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:29:10.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:10.391+0000 7ff3ecc00640 1 Processor -- start 2024-03-04T09:29:10.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:10.391+0000 7ff3ecc00640 1 -- start start 2024-03-04T09:29:10.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:10.393+0000 7ff3ecc00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff3e80af580 0x7ff3e815c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:10.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:10.393+0000 7ff3ecc00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3e815cc40 con 0x7ff3e80cdc20 2024-03-04T09:29:10.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:10.393+0000 7ff3ecc00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3e815cf20 con 0x7ff3e80af1a0 2024-03-04T09:29:10.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:10.393+0000 7ff3ecc00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3e815d200 con 0x7ff3e80af580 2024-03-04T09:29:10.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:10.393+0000 7ff3e6800640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7ff3e80cdc20 0x7ff3e80b2dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:39772/0 (socket says 172.21.15.6:39772) 2024-03-04T09:29:10.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:10.393+0000 7ff3e6800640 1 -- 172.21.15.6:0/91773006 learned_addr learned my addr 172.21.15.6:0/91773006 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:29:10.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:10.394+0000 7ff3dfe00640 1 --2- 172.21.15.6:0/91773006 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff3e80af580 0x7ff3e815c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:10.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:10.394+0000 7ff3e5400640 1 -- 172.21.15.6:0/91773006 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 999655811 0 0) 0x7ff3e815cc40 con 0x7ff3e80cdc20 2024-03-04T09:29:10.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:10.394+0000 7ff3e5400640 1 -- 172.21.15.6:0/91773006 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3cc003520 con 0x7ff3e80cdc20 2024-03-04T09:29:10.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:10.394+0000 7ff3e5400640 1 -- 172.21.15.6:0/91773006 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1288905147 0 0) 0x7ff3cc003520 con 0x7ff3e80cdc20 2024-03-04T09:29:10.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:10.394+0000 7ff3e5400640 1 -- 172.21.15.6:0/91773006 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff3e80af580 msgr2=0x7ff3e815c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:10.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:10.394+0000 7ff3e5400640 1 --2- 172.21.15.6:0/91773006 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff3e80af580 0x7ff3e815c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:10.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:10.394+0000 7ff3e5400640 1 -- 172.21.15.6:0/91773006 >> v1:172.21.15.146:6789/0 conn(0x7ff3e80af1a0 legacy=0x7ff3e80b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:10.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:10.394+0000 7ff3e5400640 1 -- 172.21.15.6:0/91773006 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3e815d4e0 con 0x7ff3e80cdc20 2024-03-04T09:29:10.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:10.394+0000 7ff3e5400640 1 -- 172.21.15.6:0/91773006 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7ff3e00019f0 con 0x7ff3e80cdc20 2024-03-04T09:29:10.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:10.394+0000 7ff3e5400640 1 -- 172.21.15.6:0/91773006 >> v1:172.21.15.6:6789/0 conn(0x7ff3e80cdc20 legacy=0x7ff3e80b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:10.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:10.394+0000 7ff3e5400640 1 --2- 172.21.15.6:0/91773006 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff3cc003a50 0x7ff3cc003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:10.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:10.394+0000 7ff3e5400640 1 --2- 172.21.15.6:0/91773006 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff3e80af580 0x7ff3cc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:10.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:10.395+0000 7ff3e5400640 1 --2- 172.21.15.6:0/91773006 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff3cc004d40 0x7ff3cc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:10.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:10.395+0000 7ff3e5400640 1 -- 172.21.15.6:0/91773006 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3e815d200 con 0x7ff3cc003a50 2024-03-04T09:29:10.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:10.395+0000 7ff3e5400640 1 -- 172.21.15.6:0/91773006 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3cc00bb80 con 0x7ff3cc004d40 2024-03-04T09:29:10.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:10.395+0000 7ff3e5400640 1 -- 172.21.15.6:0/91773006 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3cc00bd80 con 0x7ff3e80af580 2024-03-04T09:29:10.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:10.395+0000 7ff3e5e00640 1 --2- 172.21.15.6:0/91773006 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff3cc003a50 0x7ff3cc003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:10.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:10.395+0000 7ff3e6800640 1 --2- 172.21.15.6:0/91773006 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff3cc004d40 0x7ff3cc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:10.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:10.395+0000 7ff3dfe00640 1 --2- 172.21.15.6:0/91773006 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff3e80af580 0x7ff3cc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:10.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:10.395+0000 7ff3e5e00640 1 -- 172.21.15.6:0/91773006 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff3e80af580 msgr2=0x7ff3cc004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:10.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:10.395+0000 7ff3e5e00640 1 --2- 172.21.15.6:0/91773006 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff3e80af580 0x7ff3cc004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:10.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:10.395+0000 7ff3e5e00640 1 -- 172.21.15.6:0/91773006 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff3cc004d40 msgr2=0x7ff3cc0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:10.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:10.395+0000 7ff3e5e00640 1 --2- 172.21.15.6:0/91773006 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff3cc004d40 0x7ff3cc0051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:10.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:10.395+0000 7ff3e5e00640 1 -- 172.21.15.6:0/91773006 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3cc003520 con 0x7ff3cc003a50 2024-03-04T09:29:10.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:10.396+0000 7ff3e5e00640 1 --2- 172.21.15.6:0/91773006 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff3cc003a50 0x7ff3cc003e70 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7ff3d8001750 tx=0x7ff3cc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4af389214992da04 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:10.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:10.397+0000 7ff3e5400640 1 -- 172.21.15.6:0/91773006 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3d8001370 con 0x7ff3cc003a50 2024-03-04T09:29:10.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:10.397+0000 7ff3e5400640 1 -- 172.21.15.6:0/91773006 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3d80052d0 con 0x7ff3cc003a50 2024-03-04T09:29:10.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:10.397+0000 7ff3e5400640 1 -- 172.21.15.6:0/91773006 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3d8003830 con 0x7ff3cc003a50 2024-03-04T09:29:10.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:10.397+0000 7ff3ecc00640 1 -- 172.21.15.6:0/91773006 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff3cc003a50 msgr2=0x7ff3cc003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:10.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:10.397+0000 7ff3ecc00640 1 --2- 172.21.15.6:0/91773006 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff3cc003a50 0x7ff3cc003e70 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7ff3d8001750 tx=0x7ff3cc005830 comp rx=0 tx=0).stop 2024-03-04T09:29:10.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:10.397+0000 7ff3e6800640 1 -- 172.21.15.6:0/91773006 reap_dead start 2024-03-04T09:29:10.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:10.397+0000 7ff3ecc00640 1 -- 172.21.15.6:0/91773006 shutdown_connections 2024-03-04T09:29:10.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:10.397+0000 7ff3ecc00640 1 -- 172.21.15.6:0/91773006 >> 172.21.15.6:0/91773006 conn(0x7ff3e806afb0 msgr2=0x7ff3e806b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:29:10.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:10.397+0000 7ff3ecc00640 1 -- 172.21.15.6:0/91773006 shutdown_connections 2024-03-04T09:29:10.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:10.397+0000 7ff3ecc00640 1 -- 172.21.15.6:0/91773006 wait complete. 2024-03-04T09:29:10.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:10.399+0000 7ff3ecc00640 1 Processor -- start 2024-03-04T09:29:10.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:10.399+0000 7ff3ecc00640 1 -- start start 2024-03-04T09:29:10.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:10.400+0000 7ff3ecc00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff3cc003a50 0x7ff3e80b0c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:10.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:10.400+0000 7ff3ecc00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff3e80c5140 0x7ff3e8159280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:10.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:10.400+0000 7ff3ecc00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff3e80c55a0 0x7ff3e81597c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:10.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:10.400+0000 7ff3ecc00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3e815c600 con 0x7ff3cc003a50 2024-03-04T09:29:10.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:10.400+0000 7ff3ecc00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3e80b36f0 con 0x7ff3e80c55a0 2024-03-04T09:29:10.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:10.400+0000 7ff3ecc00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3e815c780 con 0x7ff3e80c5140 2024-03-04T09:29:10.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:10.401+0000 7ff3e5e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff3cc003a50 0x7ff3e80b0c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:10.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:10.401+0000 7ff3e5e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff3cc003a50 0x7ff3e80b0c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:39498/0 (socket says 172.21.15.6:39498) 2024-03-04T09:29:10.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:10.401+0000 7ff3e5e00640 1 -- 172.21.15.6:0/2039371479 learned_addr learned my addr 172.21.15.6:0/2039371479 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:29:10.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:10.401+0000 7ff3dfe00640 1 --2- 172.21.15.6:0/2039371479 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff3e80c5140 0x7ff3e8159280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:10.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:10.401+0000 7ff3e6800640 1 --2- 172.21.15.6:0/2039371479 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff3e80c55a0 0x7ff3e81597c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:10.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:10.401+0000 7ff3e5e00640 1 -- 172.21.15.6:0/2039371479 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff3e80c5140 msgr2=0x7ff3e8159280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:10.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:10.401+0000 7ff3e5e00640 1 --2- 172.21.15.6:0/2039371479 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff3e80c5140 0x7ff3e8159280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:10.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:10.401+0000 7ff3e5e00640 1 -- 172.21.15.6:0/2039371479 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff3e80c55a0 msgr2=0x7ff3e81597c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:10.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:10.401+0000 7ff3e5e00640 1 --2- 172.21.15.6:0/2039371479 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff3e80c55a0 0x7ff3e81597c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:10.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:10.401+0000 7ff3e5e00640 1 -- 172.21.15.6:0/2039371479 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3e809ae60 con 0x7ff3cc003a50 2024-03-04T09:29:10.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:10.402+0000 7ff3e5e00640 1 --2- 172.21.15.6:0/2039371479 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff3cc003a50 0x7ff3e80b0c20 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7ff3d8008fd0 tx=0x7ff3d8001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:10.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:10.402+0000 7ff3de000640 1 -- 172.21.15.6:0/2039371479 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3d8041070 con 0x7ff3cc003a50 2024-03-04T09:29:10.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:10.402+0000 7ff3de000640 1 -- 172.21.15.6:0/2039371479 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3d8005480 con 0x7ff3cc003a50 2024-03-04T09:29:10.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:10.402+0000 7ff3de000640 1 -- 172.21.15.6:0/2039371479 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3d8001370 con 0x7ff3cc003a50 2024-03-04T09:29:10.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:10.402+0000 7ff3ecc00640 1 -- 172.21.15.6:0/2039371479 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3e809b140 con 0x7ff3cc003a50 2024-03-04T09:29:10.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:10.402+0000 7ff3ecc00640 1 -- 172.21.15.6:0/2039371479 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3e809b9e0 con 0x7ff3cc003a50 2024-03-04T09:29:10.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:10.403+0000 7ff3ecc00640 1 -- 172.21.15.6:0/2039371479 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3e8069510 con 0x7ff3cc003a50 2024-03-04T09:29:10.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:10.404+0000 7ff3de000640 1 -- 172.21.15.6:0/2039371479 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff3d8044070 con 0x7ff3cc003a50 2024-03-04T09:29:10.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:10.405+0000 7ff3de000640 1 --2- 172.21.15.6:0/2039371479 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff3c406b500 0x7ff3c406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:10.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:10.405+0000 7ff3de000640 1 -- 172.21.15.6:0/2039371479 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5825+0+0 (secure 0 0 0) 0x7ff3d80abdd0 con 0x7ff3cc003a50 2024-03-04T09:29:10.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:10.406+0000 7ff3dfe00640 1 --2- 172.21.15.6:0/2039371479 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff3c406b500 0x7ff3c406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:10.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:10.407+0000 7ff3dfe00640 1 --2- 172.21.15.6:0/2039371479 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff3c406b500 0x7ff3c406d9c0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7ff3e80728b0 tx=0x7ff3d0008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:10.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:10.410+0000 7ff3de000640 1 -- 172.21.15.6:0/2039371479 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff3d807ab30 con 0x7ff3cc003a50 2024-03-04T09:29:10.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:10.651+0000 7ff3ecc00640 1 -- 172.21.15.6:0/2039371479 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff3e809b4d0 con 0x7ff3cc003a50 2024-03-04T09:29:10.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:10.652+0000 7ff3de000640 1 -- 172.21.15.6:0/2039371479 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+4979 (secure 0 0 0) 0x7ff3d807a430 con 0x7ff3cc003a50 2024-03-04T09:29:10.652 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:29:10.653 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":42,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5228,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/1891254736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1891254736},{"type":"v1","addr":"172.21.15.6:6834","nonce":1891254736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":5578,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/1781219843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":1781219843},{"type":"v1","addr":"172.21.15.146:6835","nonce":1781219843}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":5587,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/1479180802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":1479180802},{"type":"v1","addr":"172.21.15.6:6835","nonce":1479180802}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:28:50.708467+0000","modified":"2024-03-04T09:28:53.801285+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5590},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5590":{"gid":5590,"name":"d","rank":0,"incarnation":38,"state":"up:active","state_seq":3,"addr":"172.21.15.146:6837/3919939129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":3919939129},{"type":"v1","addr":"172.21.15.146:6837","nonce":3919939129}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5590,"qdb_cluster":[5590]},"id":5}]} 2024-03-04T09:29:10.654 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 42 2024-03-04T09:29:10.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:10.655+0000 7ff3ecc00640 1 -- 172.21.15.6:0/2039371479 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff3c406b500 msgr2=0x7ff3c406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:10.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:10.655+0000 7ff3ecc00640 1 --2- 172.21.15.6:0/2039371479 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff3c406b500 0x7ff3c406d9c0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7ff3e80728b0 tx=0x7ff3d0008040 comp rx=0 tx=0).stop 2024-03-04T09:29:10.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:10.655+0000 7ff3ecc00640 1 -- 172.21.15.6:0/2039371479 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff3cc003a50 msgr2=0x7ff3e80b0c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:10.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:10.655+0000 7ff3ecc00640 1 --2- 172.21.15.6:0/2039371479 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff3cc003a50 0x7ff3e80b0c20 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7ff3d8008fd0 tx=0x7ff3d8001120 comp rx=0 tx=0).stop 2024-03-04T09:29:10.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:10.655+0000 7ff3ecc00640 1 -- 172.21.15.6:0/2039371479 shutdown_connections 2024-03-04T09:29:10.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:10.655+0000 7ff3ecc00640 1 --2- 172.21.15.6:0/2039371479 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff3e80c55a0 0x7ff3e81597c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:10.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:10.655+0000 7ff3ecc00640 1 --2- 172.21.15.6:0/2039371479 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff3c406b500 0x7ff3c406d9c0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:10.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:10.655+0000 7ff3ecc00640 1 --2- 172.21.15.6:0/2039371479 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff3e80c5140 0x7ff3e8159280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:10.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:10.655+0000 7ff3ecc00640 1 --2- 172.21.15.6:0/2039371479 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff3cc003a50 0x7ff3e80b0c20 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:10.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:10.655+0000 7ff3ecc00640 1 -- 172.21.15.6:0/2039371479 >> 172.21.15.6:0/2039371479 conn(0x7ff3e806afb0 msgr2=0x7ff3e80cc750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:29:10.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:10.655+0000 7ff3ecc00640 1 -- 172.21.15.6:0/2039371479 shutdown_connections 2024-03-04T09:29:10.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:10.655+0000 7ff3ecc00640 1 -- 172.21.15.6:0/2039371479 wait complete. 2024-03-04T09:29:10.673 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json perf dump mds_cache strays_created 2024-03-04T09:29:10.922 INFO:teuthology.orchestra.run.smithi146.stdout:{"mds_cache":{"strays_created":0}} 2024-03-04T09:29:10.922 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_created": 0 } } 2024-03-04T09:29:10.922 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-04T09:29:10.922 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T09:29:10.982 DEBUG:teuthology.orchestra.run.smithi006:> stdin-killer -- python3 -c ' 2024-03-04T09:29:10.983 DEBUG:teuthology.orchestra.run.smithi006:> import os 2024-03-04T09:29:10.983 DEBUG:teuthology.orchestra.run.smithi006:> import uuid 2024-03-04T09:29:10.983 DEBUG:teuthology.orchestra.run.smithi006:> 2024-03-04T09:29:10.983 DEBUG:teuthology.orchestra.run.smithi006:> n = 100 2024-03-04T09:29:10.983 DEBUG:teuthology.orchestra.run.smithi006:> create_hard_links = False 2024-03-04T09:29:10.983 DEBUG:teuthology.orchestra.run.smithi006:> if 0 > 0: 2024-03-04T09:29:10.983 DEBUG:teuthology.orchestra.run.smithi006:> create_hard_links = True 2024-03-04T09:29:10.983 DEBUG:teuthology.orchestra.run.smithi006:> path = "/home/ubuntu/cephtest/mnt.0/subdir/file" 2024-03-04T09:29:10.983 DEBUG:teuthology.orchestra.run.smithi006:> 2024-03-04T09:29:10.984 DEBUG:teuthology.orchestra.run.smithi006:> dpath = os.path.dirname(path) 2024-03-04T09:29:10.984 DEBUG:teuthology.orchestra.run.smithi006:> fnameprefix = os.path.basename(path) 2024-03-04T09:29:10.984 DEBUG:teuthology.orchestra.run.smithi006:> os.makedirs(dpath, exist_ok=True) 2024-03-04T09:29:10.984 DEBUG:teuthology.orchestra.run.smithi006:> 2024-03-04T09:29:10.984 DEBUG:teuthology.orchestra.run.smithi006:> try: 2024-03-04T09:29:10.984 DEBUG:teuthology.orchestra.run.smithi006:> dirfd = os.open(dpath, os.O_DIRECTORY) 2024-03-04T09:29:10.984 DEBUG:teuthology.orchestra.run.smithi006:> 2024-03-04T09:29:10.984 DEBUG:teuthology.orchestra.run.smithi006:> for i in range(n): 2024-03-04T09:29:10.984 DEBUG:teuthology.orchestra.run.smithi006:> fpath = os.path.join(dpath, f"{fnameprefix}_{i}") 2024-03-04T09:29:10.985 DEBUG:teuthology.orchestra.run.smithi006:> with open(fpath, '"'"'w'"'"') as f: 2024-03-04T09:29:10.985 DEBUG:teuthology.orchestra.run.smithi006:> f.write(f"{i}") 2024-03-04T09:29:10.985 DEBUG:teuthology.orchestra.run.smithi006:> if False: 2024-03-04T09:29:10.985 DEBUG:teuthology.orchestra.run.smithi006:> f.flush() 2024-03-04T09:29:10.985 DEBUG:teuthology.orchestra.run.smithi006:> os.fsync(f.fileno()) 2024-03-04T09:29:10.985 DEBUG:teuthology.orchestra.run.smithi006:> if True: 2024-03-04T09:29:10.985 DEBUG:teuthology.orchestra.run.smithi006:> os.unlink(fpath) 2024-03-04T09:29:10.985 DEBUG:teuthology.orchestra.run.smithi006:> if False: 2024-03-04T09:29:10.985 DEBUG:teuthology.orchestra.run.smithi006:> os.fsync(dirfd) 2024-03-04T09:29:10.985 DEBUG:teuthology.orchestra.run.smithi006:> if create_hard_links: 2024-03-04T09:29:10.985 DEBUG:teuthology.orchestra.run.smithi006:> for j in range(0): 2024-03-04T09:29:10.985 DEBUG:teuthology.orchestra.run.smithi006:> os.system(f"ln {fpath} {dpath}/{fnameprefix}_{i}_{uuid.uuid4()}") 2024-03-04T09:29:10.986 DEBUG:teuthology.orchestra.run.smithi006:> if True: 2024-03-04T09:29:10.986 DEBUG:teuthology.orchestra.run.smithi006:> os.fsync(dirfd) 2024-03-04T09:29:10.986 DEBUG:teuthology.orchestra.run.smithi006:> finally: 2024-03-04T09:29:10.986 DEBUG:teuthology.orchestra.run.smithi006:> os.close(dirfd) 2024-03-04T09:29:10.986 DEBUG:teuthology.orchestra.run.smithi006:> ' 2024-03-04T09:29:11.240 INFO:teuthology.orchestra.run.smithi006.stderr:Traceback (most recent call last): 2024-03-04T09:29:11.240 INFO:teuthology.orchestra.run.smithi006.stderr: File "", line 26, in 2024-03-04T09:29:11.240 INFO:teuthology.orchestra.run.smithi006.stderr:OSError: [Errno 28] No space left on device: '/home/ubuntu/cephtest/mnt.0/subdir/file_10' 2024-03-04T09:29:11.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:11 stdin-killer INFO: command exited with status 1: exiting normally with same code! 2024-03-04T09:29:11.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T09:29:11.266 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:29:11.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:11.545+0000 7f324b600640 1 Processor -- start 2024-03-04T09:29:11.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:11.545+0000 7f324b600640 1 -- start start 2024-03-04T09:29:11.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:11.547+0000 7f324b600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f32440cf5c0 0x7f32440cdc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:11.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:11.547+0000 7f324b600640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32440b4860 con 0x7f32440b3c90 2024-03-04T09:29:11.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:11.547+0000 7f324b600640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32440b4b40 con 0x7f3244071050 2024-03-04T09:29:11.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:11.547+0000 7f324b600640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32440ce160 con 0x7f32440cf5c0 2024-03-04T09:29:11.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:11.547+0000 7f3243e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f32440cf5c0 0x7f32440cdc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:11.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:11.547+0000 7f3243e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f32440cf5c0 0x7f32440cdc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:39512/0 (socket says 172.21.15.6:39512) 2024-03-04T09:29:11.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:11.547+0000 7f3243e00640 1 -- 172.21.15.6:0/1716235798 learned_addr learned my addr 172.21.15.6:0/1716235798 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:29:11.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:11.547+0000 7f3248c00640 1 --1- 172.21.15.6:0/1716235798 >> v1:172.21.15.6:6789/0 conn(0x7f32440b3c90 0x7f32440b4070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:39788/0 (socket says 172.21.15.6:39788) 2024-03-04T09:29:11.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:11.548+0000 7f3243e00640 1 -- 172.21.15.6:0/1716235798 >> v1:172.21.15.146:6789/0 conn(0x7f3244071050 legacy=0x7f3244071490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:11.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:11.548+0000 7f3243e00640 1 -- 172.21.15.6:0/1716235798 >> v1:172.21.15.6:6789/0 conn(0x7f32440b3c90 legacy=0x7f32440b4070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:11.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:11.548+0000 7f3243e00640 1 -- 172.21.15.6:0/1716235798 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32440b4e20 con 0x7f32440cf5c0 2024-03-04T09:29:11.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:11.548+0000 7f3243400640 1 -- 172.21.15.6:0/1716235798 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3745580237 0 0) 0x7f32440b4860 con 0x7f32440b3c90 2024-03-04T09:29:11.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:11.549+0000 7f3243e00640 1 --2- 172.21.15.6:0/1716235798 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f32440cf5c0 0x7f32440cdc20 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f3230009870 tx=0x7f323002ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a7d246031ca4238 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:11.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:11.549+0000 7f3243400640 1 -- 172.21.15.6:0/1716235798 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f323002f840 con 0x7f32440cf5c0 2024-03-04T09:29:11.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:11.549+0000 7f3243400640 1 -- 172.21.15.6:0/1716235798 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f323002f9e0 con 0x7f32440cf5c0 2024-03-04T09:29:11.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:11.549+0000 7f3243400640 1 -- 172.21.15.6:0/1716235798 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f323002fdb0 con 0x7f32440cf5c0 2024-03-04T09:29:11.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:11.550+0000 7f324b600640 1 -- 172.21.15.6:0/1716235798 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f32440cf5c0 msgr2=0x7f32440cdc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:11.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:11.550+0000 7f324b600640 1 --2- 172.21.15.6:0/1716235798 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f32440cf5c0 0x7f32440cdc20 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f3230009870 tx=0x7f323002ba20 comp rx=0 tx=0).stop 2024-03-04T09:29:11.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:11.550+0000 7f324b600640 1 -- 172.21.15.6:0/1716235798 shutdown_connections 2024-03-04T09:29:11.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:11.550+0000 7f324b600640 1 --2- 172.21.15.6:0/1716235798 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f32440cf5c0 0x7f32440cdc20 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:11.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:11.550+0000 7f324b600640 1 -- 172.21.15.6:0/1716235798 >> 172.21.15.6:0/1716235798 conn(0x7f324406afb0 msgr2=0x7f324406b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:29:11.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:11.550+0000 7f324b600640 1 -- 172.21.15.6:0/1716235798 shutdown_connections 2024-03-04T09:29:11.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:11.550+0000 7f324b600640 1 -- 172.21.15.6:0/1716235798 wait complete. 2024-03-04T09:29:11.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:11.552+0000 7f324b600640 1 Processor -- start 2024-03-04T09:29:11.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:11.552+0000 7f324b600640 1 -- start start 2024-03-04T09:29:11.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:11.553+0000 7f324b600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3244071050 0x7f32441e4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:11.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:11.553+0000 7f324b600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f32440b3c90 0x7f32441f40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:11.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:11.553+0000 7f324b600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f32440cf5c0 0x7f3244155fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:11.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:11.553+0000 7f324b600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3244071ba0 con 0x7f32440cf5c0 2024-03-04T09:29:11.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:11.553+0000 7f324b600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f324415b340 con 0x7f32440b3c90 2024-03-04T09:29:11.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:11.553+0000 7f324b600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3244071d20 con 0x7f3244071050 2024-03-04T09:29:11.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:11.553+0000 7f3249600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f32440cf5c0 0x7f3244155fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:11.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:11.554+0000 7f3249600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f32440cf5c0 0x7f3244155fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:39520/0 (socket says 172.21.15.6:39520) 2024-03-04T09:29:11.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:11.554+0000 7f3249600640 1 -- 172.21.15.6:0/1142387122 learned_addr learned my addr 172.21.15.6:0/1142387122 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:29:11.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:11.554+0000 7f3243e00640 1 --2- 172.21.15.6:0/1142387122 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f32440b3c90 0x7f32441f40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:11.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:11.554+0000 7f3248c00640 1 --2- 172.21.15.6:0/1142387122 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3244071050 0x7f32441e4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:11.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:11.554+0000 7f3249600640 1 -- 172.21.15.6:0/1142387122 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3244071050 msgr2=0x7f32441e4610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:11.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:11.554+0000 7f3249600640 1 --2- 172.21.15.6:0/1142387122 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3244071050 0x7f32441e4610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:11.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:11.554+0000 7f3249600640 1 -- 172.21.15.6:0/1142387122 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f32440b3c90 msgr2=0x7f32441f40e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:11.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:11.554+0000 7f3249600640 1 --2- 172.21.15.6:0/1142387122 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f32440b3c90 0x7f32441f40e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:11.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:11.554+0000 7f3249600640 1 -- 172.21.15.6:0/1142387122 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f324409aab0 con 0x7f32440cf5c0 2024-03-04T09:29:11.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:11.554+0000 7f3243e00640 1 --2- 172.21.15.6:0/1142387122 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f32440b3c90 0x7f32441f40e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:29:11.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:11.554+0000 7f3248c00640 1 --2- 172.21.15.6:0/1142387122 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3244071050 0x7f32441e4610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:29:11.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:11.554+0000 7f3249600640 1 --2- 172.21.15.6:0/1142387122 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f32440cf5c0 0x7f3244155fd0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f323c00c780 tx=0x7f323c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:11.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:11.555+0000 7f3240c00640 1 -- 172.21.15.6:0/1142387122 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f323c004b60 con 0x7f32440cf5c0 2024-03-04T09:29:11.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:11.555+0000 7f3240c00640 1 -- 172.21.15.6:0/1142387122 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f323c004d00 con 0x7f32440cf5c0 2024-03-04T09:29:11.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:11.555+0000 7f3240c00640 1 -- 172.21.15.6:0/1142387122 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f323c00aa90 con 0x7f32440cf5c0 2024-03-04T09:29:11.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:11.555+0000 7f324b600640 1 -- 172.21.15.6:0/1142387122 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32441f4730 con 0x7f32440cf5c0 2024-03-04T09:29:11.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:11.555+0000 7f324b600640 1 -- 172.21.15.6:0/1142387122 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32441f4cf0 con 0x7f32440cf5c0 2024-03-04T09:29:11.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:11.556+0000 7f324b600640 1 -- 172.21.15.6:0/1142387122 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3244069510 con 0x7f32440cf5c0 2024-03-04T09:29:11.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:11.557+0000 7f3240c00640 1 -- 172.21.15.6:0/1142387122 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f323c00ac30 con 0x7f32440cf5c0 2024-03-04T09:29:11.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:11.558+0000 7f3240c00640 1 --2- 172.21.15.6:0/1142387122 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f322406b4d0 0x7f322406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:11.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:11.559+0000 7f3240c00640 1 -- 172.21.15.6:0/1142387122 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f323c08a4e0 con 0x7f32440cf5c0 2024-03-04T09:29:11.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:11.559+0000 7f3248c00640 1 --2- 172.21.15.6:0/1142387122 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f322406b4d0 0x7f322406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:11.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:11.561+0000 7f3248c00640 1 --2- 172.21.15.6:0/1142387122 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f322406b4d0 0x7f322406d990 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f3238000f60 tx=0x7f3238009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:11.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:11.563+0000 7f3240c00640 1 -- 172.21.15.6:0/1142387122 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f323c020070 con 0x7f32440cf5c0 2024-03-04T09:29:11.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:11.800+0000 7f324b600640 1 -- 172.21.15.6:0/1142387122 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3244152e90 con 0x7f32440cf5c0 2024-03-04T09:29:11.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:11.802+0000 7f3240c00640 1 -- 172.21.15.6:0/1142387122 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+4979 (secure 0 0 0) 0x7f323c0592e0 con 0x7f32440cf5c0 2024-03-04T09:29:11.799 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:29:11.799 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":42,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5228,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/1891254736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1891254736},{"type":"v1","addr":"172.21.15.6:6834","nonce":1891254736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":5578,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/1781219843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":1781219843},{"type":"v1","addr":"172.21.15.146:6835","nonce":1781219843}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":5587,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/1479180802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":1479180802},{"type":"v1","addr":"172.21.15.6:6835","nonce":1479180802}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:28:50.708467+0000","modified":"2024-03-04T09:28:53.801285+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5590},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5590":{"gid":5590,"name":"d","rank":0,"incarnation":38,"state":"up:active","state_seq":3,"addr":"172.21.15.146:6837/3919939129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":3919939129},{"type":"v1","addr":"172.21.15.146:6837","nonce":3919939129}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5590,"qdb_cluster":[5590]},"id":5}]} 2024-03-04T09:29:11.799 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 42 2024-03-04T09:29:11.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:11.804+0000 7f324b600640 1 -- 172.21.15.6:0/1142387122 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f322406b4d0 msgr2=0x7f322406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:11.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:11.804+0000 7f324b600640 1 --2- 172.21.15.6:0/1142387122 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f322406b4d0 0x7f322406d990 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f3238000f60 tx=0x7f3238009040 comp rx=0 tx=0).stop 2024-03-04T09:29:11.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:11.804+0000 7f324b600640 1 -- 172.21.15.6:0/1142387122 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f32440cf5c0 msgr2=0x7f3244155fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:11.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:11.804+0000 7f324b600640 1 --2- 172.21.15.6:0/1142387122 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f32440cf5c0 0x7f3244155fd0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f323c00c780 tx=0x7f323c00cc40 comp rx=0 tx=0).stop 2024-03-04T09:29:11.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:11.805+0000 7f324b600640 1 -- 172.21.15.6:0/1142387122 shutdown_connections 2024-03-04T09:29:11.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:11.805+0000 7f324b600640 1 --2- 172.21.15.6:0/1142387122 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f32440cf5c0 0x7f3244155fd0 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:11.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:11.805+0000 7f324b600640 1 --2- 172.21.15.6:0/1142387122 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f32440b3c90 0x7f32441f40e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:11.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:11.805+0000 7f324b600640 1 --2- 172.21.15.6:0/1142387122 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f322406b4d0 0x7f322406d990 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:11.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:11.805+0000 7f324b600640 1 --2- 172.21.15.6:0/1142387122 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3244071050 0x7f32441e4610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:11.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:11.805+0000 7f324b600640 1 -- 172.21.15.6:0/1142387122 >> 172.21.15.6:0/1142387122 conn(0x7f324406afb0 msgr2=0x7f324406cba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:29:11.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:11.805+0000 7f324b600640 1 -- 172.21.15.6:0/1142387122 shutdown_connections 2024-03-04T09:29:11.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:11.805+0000 7f324b600640 1 -- 172.21.15.6:0/1142387122 wait complete. 2024-03-04T09:29:11.822 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json perf dump mds_cache strays_created 2024-03-04T09:29:12.049 INFO:teuthology.orchestra.run.smithi146.stdout:{"mds_cache":{"strays_created":10}} 2024-03-04T09:29:12.050 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_created": 10 } } 2024-03-04T09:29:12.050 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:29:12.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:12.287+0000 7fa976600640 1 Processor -- start 2024-03-04T09:29:12.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:12.287+0000 7fa976600640 1 -- start start 2024-03-04T09:29:12.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:12.288+0000 7fa976600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa97015e0b0 0x7fa9700cf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:12.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:12.288+0000 7fa976600640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fa9700d4a10 0x7fa9700d0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:12.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:12.288+0000 7fa976600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa9700d0c80 0x7fa9700d1900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:12.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:12.288+0000 7fa976600640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9700b09e0 con 0x7fa97015e0b0 2024-03-04T09:29:12.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:12.288+0000 7fa976600640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9700b3710 con 0x7fa9700d4a10 2024-03-04T09:29:12.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:12.288+0000 7fa976600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9700b39f0 con 0x7fa9700d0c80 2024-03-04T09:29:12.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:12.289+0000 7fa975200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa97015e0b0 0x7fa9700cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:12.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:12.289+0000 7fa975200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa97015e0b0 0x7fa9700cf310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:39528/0 (socket says 172.21.15.6:39528) 2024-03-04T09:29:12.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:12.289+0000 7fa975200640 1 -- 172.21.15.6:0/2060124757 learned_addr learned my addr 172.21.15.6:0/2060124757 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:29:12.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:12.289+0000 7fa96fe00640 1 --2- 172.21.15.6:0/2060124757 >> v2:172.21.15.146:3300/0 conn(0x7fa9700d4a10 0x7fa9700d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:12.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:12.289+0000 7fa975c00640 1 --2- 172.21.15.6:0/2060124757 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa9700d0c80 0x7fa9700d1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:12.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:12.289+0000 7fa975200640 1 -- 172.21.15.6:0/2060124757 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa9700d0c80 msgr2=0x7fa9700d1900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:12.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:12.289+0000 7fa975200640 1 --2- 172.21.15.6:0/2060124757 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa9700d0c80 0x7fa9700d1900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:12.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:12.289+0000 7fa975200640 1 -- 172.21.15.6:0/2060124757 >> v2:172.21.15.146:3300/0 conn(0x7fa9700d4a10 msgr2=0x7fa9700d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:12.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:12.289+0000 7fa975200640 1 --2- 172.21.15.6:0/2060124757 >> v2:172.21.15.146:3300/0 conn(0x7fa9700d4a10 0x7fa9700d0740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:12.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:12.289+0000 7fa975200640 1 -- 172.21.15.6:0/2060124757 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9700b3cd0 con 0x7fa97015e0b0 2024-03-04T09:29:12.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:12.290+0000 7fa975200640 1 --2- 172.21.15.6:0/2060124757 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa97015e0b0 0x7fa9700cf310 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7fa97006b8c0 tx=0x7fa96002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49100f9a56a1a20e server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:12.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:12.291+0000 7fa96f400640 1 -- 172.21.15.6:0/2060124757 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa96002ca80 con 0x7fa97015e0b0 2024-03-04T09:29:12.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:12.291+0000 7fa96f400640 1 -- 172.21.15.6:0/2060124757 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa96002cc20 con 0x7fa97015e0b0 2024-03-04T09:29:12.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:12.291+0000 7fa96f400640 1 -- 172.21.15.6:0/2060124757 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9600343f0 con 0x7fa97015e0b0 2024-03-04T09:29:12.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:12.291+0000 7fa976600640 1 -- 172.21.15.6:0/2060124757 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa97015e0b0 msgr2=0x7fa9700cf310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:12.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:12.291+0000 7fa976600640 1 --2- 172.21.15.6:0/2060124757 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa97015e0b0 0x7fa9700cf310 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7fa97006b8c0 tx=0x7fa96002bee0 comp rx=0 tx=0).stop 2024-03-04T09:29:12.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:12.291+0000 7fa976600640 1 -- 172.21.15.6:0/2060124757 shutdown_connections 2024-03-04T09:29:12.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:12.291+0000 7fa976600640 1 --2- 172.21.15.6:0/2060124757 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa9700d0c80 0x7fa9700d1900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:12.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:12.291+0000 7fa976600640 1 --2- 172.21.15.6:0/2060124757 >> v2:172.21.15.146:3300/0 conn(0x7fa9700d4a10 0x7fa9700d0740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:12.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:12.291+0000 7fa976600640 1 --2- 172.21.15.6:0/2060124757 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa97015e0b0 0x7fa9700cf310 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:12.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:12.292+0000 7fa976600640 1 -- 172.21.15.6:0/2060124757 >> 172.21.15.6:0/2060124757 conn(0x7fa9700d3d30 msgr2=0x7fa9700d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:29:12.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:12.292+0000 7fa976600640 1 -- 172.21.15.6:0/2060124757 shutdown_connections 2024-03-04T09:29:12.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:12.292+0000 7fa976600640 1 -- 172.21.15.6:0/2060124757 wait complete. 2024-03-04T09:29:12.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:12.293+0000 7fa976600640 1 Processor -- start 2024-03-04T09:29:12.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:12.293+0000 7fa976600640 1 -- start start 2024-03-04T09:29:12.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:12.294+0000 7fa976600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa9700d0c80 0x7fa9700b6950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:12.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:12.294+0000 7fa976600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa9700d4a10 0x7fa9700b8490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:12.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:12.294+0000 7fa976600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa97015e0b0 0x7fa9700b89d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:12.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:12.294+0000 7fa976600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9700cfc60 con 0x7fa97015e0b0 2024-03-04T09:29:12.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:12.294+0000 7fa976600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa97015e490 con 0x7fa9700d0c80 2024-03-04T09:29:12.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:12.294+0000 7fa976600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9700cf850 con 0x7fa9700d4a10 2024-03-04T09:29:12.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:12.295+0000 7fa975c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa97015e0b0 0x7fa9700b89d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:12.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:12.295+0000 7fa975c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa97015e0b0 0x7fa9700b89d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:39532/0 (socket says 172.21.15.6:39532) 2024-03-04T09:29:12.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:12.295+0000 7fa975c00640 1 -- 172.21.15.6:0/1043059186 learned_addr learned my addr 172.21.15.6:0/1043059186 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:29:12.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:12.295+0000 7fa96fe00640 1 --2- 172.21.15.6:0/1043059186 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa9700d4a10 0x7fa9700b8490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:12.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:12.295+0000 7fa975200640 1 --2- 172.21.15.6:0/1043059186 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa9700d0c80 0x7fa9700b6950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:12.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:12.295+0000 7fa975c00640 1 -- 172.21.15.6:0/1043059186 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa9700d4a10 msgr2=0x7fa9700b8490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:12.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:12.295+0000 7fa975c00640 1 --2- 172.21.15.6:0/1043059186 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa9700d4a10 0x7fa9700b8490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:12.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:12.295+0000 7fa975c00640 1 -- 172.21.15.6:0/1043059186 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa9700d0c80 msgr2=0x7fa9700b6950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:12.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:12.295+0000 7fa975c00640 1 --2- 172.21.15.6:0/1043059186 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa9700d0c80 0x7fa9700b6950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:12.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:12.295+0000 7fa975c00640 1 -- 172.21.15.6:0/1043059186 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa970160570 con 0x7fa97015e0b0 2024-03-04T09:29:12.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:12.296+0000 7fa975c00640 1 --2- 172.21.15.6:0/1043059186 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa97015e0b0 0x7fa9700b89d0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7fa96800d800 tx=0x7fa96800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:12.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:12.296+0000 7fa96cc00640 1 -- 172.21.15.6:0/1043059186 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa96800b6c0 con 0x7fa97015e0b0 2024-03-04T09:29:12.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:12.296+0000 7fa96cc00640 1 -- 172.21.15.6:0/1043059186 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa968019070 con 0x7fa97015e0b0 2024-03-04T09:29:12.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:12.296+0000 7fa96cc00640 1 -- 172.21.15.6:0/1043059186 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa96800bac0 con 0x7fa97015e0b0 2024-03-04T09:29:12.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:12.296+0000 7fa976600640 1 -- 172.21.15.6:0/1043059186 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa970160850 con 0x7fa97015e0b0 2024-03-04T09:29:12.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:12.296+0000 7fa976600640 1 -- 172.21.15.6:0/1043059186 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa970160e10 con 0x7fa97015e0b0 2024-03-04T09:29:12.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:12.297+0000 7fa976600640 1 -- 172.21.15.6:0/1043059186 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa97004f330 con 0x7fa97015e0b0 2024-03-04T09:29:12.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:12.298+0000 7fa96cc00640 1 -- 172.21.15.6:0/1043059186 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa96800bc60 con 0x7fa97015e0b0 2024-03-04T09:29:12.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:12.299+0000 7fa96cc00640 1 --2- 172.21.15.6:0/1043059186 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa94c06b4d0 0x7fa94c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:12.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:12.299+0000 7fa96cc00640 1 -- 172.21.15.6:0/1043059186 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fa96808a790 con 0x7fa97015e0b0 2024-03-04T09:29:12.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:12.300+0000 7fa975200640 1 --2- 172.21.15.6:0/1043059186 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa94c06b4d0 0x7fa94c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:12.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:12.301+0000 7fa975200640 1 --2- 172.21.15.6:0/1043059186 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa94c06b4d0 0x7fa94c06d990 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fa960035000 tx=0x7fa960030000 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:12.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:12.304+0000 7fa96cc00640 1 -- 172.21.15.6:0/1043059186 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa9680594f0 con 0x7fa97015e0b0 2024-03-04T09:29:12.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:12.542+0000 7fa976600640 1 -- 172.21.15.6:0/1043059186 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa97006cb10 con 0x7fa97015e0b0 2024-03-04T09:29:12.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:12.543+0000 7fa96cc00640 1 -- 172.21.15.6:0/1043059186 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+4979 (secure 0 0 0) 0x7fa968058df0 con 0x7fa97015e0b0 2024-03-04T09:29:12.541 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:29:12.541 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":42,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5228,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/1891254736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1891254736},{"type":"v1","addr":"172.21.15.6:6834","nonce":1891254736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":5578,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/1781219843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":1781219843},{"type":"v1","addr":"172.21.15.146:6835","nonce":1781219843}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":5587,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/1479180802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":1479180802},{"type":"v1","addr":"172.21.15.6:6835","nonce":1479180802}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:28:50.708467+0000","modified":"2024-03-04T09:28:53.801285+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5590},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5590":{"gid":5590,"name":"d","rank":0,"incarnation":38,"state":"up:active","state_seq":3,"addr":"172.21.15.146:6837/3919939129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":3919939129},{"type":"v1","addr":"172.21.15.146:6837","nonce":3919939129}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5590,"qdb_cluster":[5590]},"id":5}]} 2024-03-04T09:29:12.541 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 42 2024-03-04T09:29:12.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:12.546+0000 7fa976600640 1 -- 172.21.15.6:0/1043059186 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa94c06b4d0 msgr2=0x7fa94c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:12.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:12.546+0000 7fa976600640 1 --2- 172.21.15.6:0/1043059186 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa94c06b4d0 0x7fa94c06d990 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fa960035000 tx=0x7fa960030000 comp rx=0 tx=0).stop 2024-03-04T09:29:12.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:12.546+0000 7fa976600640 1 -- 172.21.15.6:0/1043059186 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa97015e0b0 msgr2=0x7fa9700b89d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:12.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:12.546+0000 7fa976600640 1 --2- 172.21.15.6:0/1043059186 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa97015e0b0 0x7fa9700b89d0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7fa96800d800 tx=0x7fa96800dcc0 comp rx=0 tx=0).stop 2024-03-04T09:29:12.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:12.546+0000 7fa976600640 1 -- 172.21.15.6:0/1043059186 shutdown_connections 2024-03-04T09:29:12.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:12.546+0000 7fa976600640 1 --2- 172.21.15.6:0/1043059186 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa97015e0b0 0x7fa9700b89d0 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:12.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:12.546+0000 7fa976600640 1 --2- 172.21.15.6:0/1043059186 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa94c06b4d0 0x7fa94c06d990 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:12.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:12.546+0000 7fa976600640 1 --2- 172.21.15.6:0/1043059186 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa9700d4a10 0x7fa9700b8490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:12.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:12.546+0000 7fa976600640 1 --2- 172.21.15.6:0/1043059186 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa9700d0c80 0x7fa9700b6950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:12.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:12.546+0000 7fa976600640 1 -- 172.21.15.6:0/1043059186 >> 172.21.15.6:0/1043059186 conn(0x7fa9700d3d30 msgr2=0x7fa9700b5c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:29:12.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:12.546+0000 7fa976600640 1 -- 172.21.15.6:0/1043059186 shutdown_connections 2024-03-04T09:29:12.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:12.546+0000 7fa976600640 1 -- 172.21.15.6:0/1043059186 wait complete. 2024-03-04T09:29:12.562 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json perf dump mds_cache strays_enqueued 2024-03-04T09:29:12.804 INFO:teuthology.orchestra.run.smithi146.stdout:{"mds_cache":{"strays_enqueued":0}} 2024-03-04T09:29:12.805 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 0 } } 2024-03-04T09:29:12.805 DEBUG:tasks.ceph_test_case:wait_until_equal: 0 != 10, waiting (timeout=60)... 2024-03-04T09:29:17.806 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:29:18.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:18.042+0000 7f588f800640 1 Processor -- start 2024-03-04T09:29:18.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:18.043+0000 7f588f800640 1 -- start start 2024-03-04T09:29:18.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:18.044+0000 7f588f800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f58880af580 0x7f588815c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:18.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:18.044+0000 7f588f800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f58880cdc20 0x7f58880b2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:18.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:18.044+0000 7f588f800640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f588815cc40 con 0x7f58880af1a0 2024-03-04T09:29:18.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:18.044+0000 7f588f800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f588815cf20 con 0x7f58880af580 2024-03-04T09:29:18.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:18.044+0000 7f588f800640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f588815d200 con 0x7f58880cdc20 2024-03-04T09:29:18.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:18.045+0000 7f588ce00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f58880af1a0 0x7f58880b4c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:39802/0 (socket says 172.21.15.6:39802) 2024-03-04T09:29:18.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:18.045+0000 7f588ce00640 1 -- 172.21.15.6:0/736353380 learned_addr learned my addr 172.21.15.6:0/736353380 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:29:18.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:18.045+0000 7f5887e00640 1 --2- 172.21.15.6:0/736353380 >> v2:172.21.15.6:3300/0 conn(0x7f58880af580 0x7f588815c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:18.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:18.045+0000 7f588d800640 1 --2- 172.21.15.6:0/736353380 >> v2:172.21.15.146:3300/0 conn(0x7f58880cdc20 0x7f58880b2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:18.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:18.045+0000 7f5887e00640 1 -- 172.21.15.6:0/736353380 >> v2:172.21.15.146:3300/0 conn(0x7f58880cdc20 msgr2=0x7f58880b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:18.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:18.045+0000 7f5887e00640 1 --2- 172.21.15.6:0/736353380 >> v2:172.21.15.146:3300/0 conn(0x7f58880cdc20 0x7f58880b2dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:18.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:18.045+0000 7f5887e00640 1 -- 172.21.15.6:0/736353380 >> v1:172.21.15.6:6789/0 conn(0x7f58880af1a0 legacy=0x7f58880b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:18.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:18.045+0000 7f5887e00640 1 -- 172.21.15.6:0/736353380 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f588815d4e0 con 0x7f58880af580 2024-03-04T09:29:18.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:18.045+0000 7f5887400640 1 -- 172.21.15.6:0/736353380 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 607802100 0 0) 0x7f588815cc40 con 0x7f58880af1a0 2024-03-04T09:29:18.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:18.046+0000 7f5887e00640 1 --2- 172.21.15.6:0/736353380 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f58880af580 0x7f588815c600 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f588806c0c0 tx=0x7f587402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bd05d4149f3e807 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:18.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:18.047+0000 7f5887400640 1 -- 172.21.15.6:0/736353380 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58740315a0 con 0x7f58880af580 2024-03-04T09:29:18.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:18.047+0000 7f5887400640 1 -- 172.21.15.6:0/736353380 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5874037070 con 0x7f58880af580 2024-03-04T09:29:18.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:18.047+0000 7f5887400640 1 -- 172.21.15.6:0/736353380 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5874031960 con 0x7f58880af580 2024-03-04T09:29:18.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:18.047+0000 7f588f800640 1 -- 172.21.15.6:0/736353380 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f58880af580 msgr2=0x7f588815c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:18.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:18.047+0000 7f588f800640 1 --2- 172.21.15.6:0/736353380 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f58880af580 0x7f588815c600 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f588806c0c0 tx=0x7f587402b840 comp rx=0 tx=0).stop 2024-03-04T09:29:18.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:18.047+0000 7f588f800640 1 -- 172.21.15.6:0/736353380 shutdown_connections 2024-03-04T09:29:18.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:18.047+0000 7f588f800640 1 --2- 172.21.15.6:0/736353380 >> v2:172.21.15.146:3300/0 conn(0x7f58880cdc20 0x7f58880b2dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:18.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:18.047+0000 7f588f800640 1 --2- 172.21.15.6:0/736353380 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f58880af580 0x7f588815c600 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:18.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:18.047+0000 7f588f800640 1 -- 172.21.15.6:0/736353380 >> 172.21.15.6:0/736353380 conn(0x7f588806afb0 msgr2=0x7f588806b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:29:18.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:18.048+0000 7f588f800640 1 -- 172.21.15.6:0/736353380 shutdown_connections 2024-03-04T09:29:18.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:18.048+0000 7f588f800640 1 -- 172.21.15.6:0/736353380 wait complete. 2024-03-04T09:29:18.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:18.049+0000 7f588f800640 1 Processor -- start 2024-03-04T09:29:18.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:18.049+0000 7f588f800640 1 -- start start 2024-03-04T09:29:18.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:18.050+0000 7f588f800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f58880af1a0 0x7f5888154af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:18.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:18.050+0000 7f588f800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f58880af580 0x7f58881f1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:18.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:18.050+0000 7f588f800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f58880cdc20 0x7f588809b640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:18.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:18.050+0000 7f588f800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58880b36a0 con 0x7f58880cdc20 2024-03-04T09:29:18.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:18.050+0000 7f588f800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58880b3520 con 0x7f58880af580 2024-03-04T09:29:18.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:18.050+0000 7f588f800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58880b3820 con 0x7f58880af1a0 2024-03-04T09:29:18.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:18.051+0000 7f588d800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f58880cdc20 0x7f588809b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:18.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:18.051+0000 7f588d800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f58880cdc20 0x7f588809b640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:39540/0 (socket says 172.21.15.6:39540) 2024-03-04T09:29:18.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:18.051+0000 7f588d800640 1 -- 172.21.15.6:0/2353081968 learned_addr learned my addr 172.21.15.6:0/2353081968 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:29:18.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:18.051+0000 7f588ce00640 1 --2- 172.21.15.6:0/2353081968 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f58880af1a0 0x7f5888154af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:18.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:18.051+0000 7f5887e00640 1 --2- 172.21.15.6:0/2353081968 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f58880af580 0x7f58881f1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:18.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:18.051+0000 7f588d800640 1 -- 172.21.15.6:0/2353081968 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f58880af1a0 msgr2=0x7f5888154af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:18.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:18.051+0000 7f588d800640 1 --2- 172.21.15.6:0/2353081968 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f58880af1a0 0x7f5888154af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:18.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:18.051+0000 7f588d800640 1 -- 172.21.15.6:0/2353081968 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f58880af580 msgr2=0x7f58881f1870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:18.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:18.051+0000 7f588d800640 1 --2- 172.21.15.6:0/2353081968 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f58880af580 0x7f58881f1870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:18.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:18.051+0000 7f588d800640 1 -- 172.21.15.6:0/2353081968 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58881f2360 con 0x7f58880cdc20 2024-03-04T09:29:18.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:18.051+0000 7f588d800640 1 --2- 172.21.15.6:0/2353081968 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f58880cdc20 0x7f588809b640 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f5880007e60 tx=0x7f588000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:18.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:18.052+0000 7f5884c00640 1 -- 172.21.15.6:0/2353081968 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5880014420 con 0x7f58880cdc20 2024-03-04T09:29:18.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:18.052+0000 7f5884c00640 1 -- 172.21.15.6:0/2353081968 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5880019070 con 0x7f58880cdc20 2024-03-04T09:29:18.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:18.052+0000 7f5884c00640 1 -- 172.21.15.6:0/2353081968 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5880014820 con 0x7f58880cdc20 2024-03-04T09:29:18.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:18.052+0000 7f588f800640 1 -- 172.21.15.6:0/2353081968 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58881f2640 con 0x7f58880cdc20 2024-03-04T09:29:18.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:18.052+0000 7f588f800640 1 -- 172.21.15.6:0/2353081968 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5888158530 con 0x7f58880cdc20 2024-03-04T09:29:18.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:18.053+0000 7f588f800640 1 -- 172.21.15.6:0/2353081968 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5888069510 con 0x7f58880cdc20 2024-03-04T09:29:18.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:18.054+0000 7f5884c00640 1 -- 172.21.15.6:0/2353081968 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f58800149c0 con 0x7f58880cdc20 2024-03-04T09:29:18.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:18.055+0000 7f5884c00640 1 --2- 172.21.15.6:0/2353081968 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f586806b500 0x7f586806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:18.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:18.055+0000 7f5884c00640 1 -- 172.21.15.6:0/2353081968 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f588008ab90 con 0x7f58880cdc20 2024-03-04T09:29:18.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:18.056+0000 7f588ce00640 1 --2- 172.21.15.6:0/2353081968 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f586806b500 0x7f586806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:18.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:18.057+0000 7f588ce00640 1 --2- 172.21.15.6:0/2353081968 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f586806b500 0x7f586806d9c0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f587c000f60 tx=0x7f587c007b70 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:18.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:18.060+0000 7f5884c00640 1 -- 172.21.15.6:0/2353081968 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f58800598f0 con 0x7f58880cdc20 2024-03-04T09:29:18.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:18.300+0000 7f588f800640 1 -- 172.21.15.6:0/2353081968 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f588804f3a0 con 0x7f58880cdc20 2024-03-04T09:29:18.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:18.301+0000 7f5884c00640 1 -- 172.21.15.6:0/2353081968 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+4979 (secure 0 0 0) 0x7f58800591f0 con 0x7f58880cdc20 2024-03-04T09:29:18.299 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:29:18.299 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":42,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5228,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/1891254736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1891254736},{"type":"v1","addr":"172.21.15.6:6834","nonce":1891254736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":5578,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/1781219843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":1781219843},{"type":"v1","addr":"172.21.15.146:6835","nonce":1781219843}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":5587,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/1479180802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":1479180802},{"type":"v1","addr":"172.21.15.6:6835","nonce":1479180802}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:28:50.708467+0000","modified":"2024-03-04T09:28:53.801285+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5590},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5590":{"gid":5590,"name":"d","rank":0,"incarnation":38,"state":"up:active","state_seq":3,"addr":"172.21.15.146:6837/3919939129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":3919939129},{"type":"v1","addr":"172.21.15.146:6837","nonce":3919939129}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5590,"qdb_cluster":[5590]},"id":5}]} 2024-03-04T09:29:18.299 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 42 2024-03-04T09:29:18.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:18.304+0000 7f588f800640 1 -- 172.21.15.6:0/2353081968 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f586806b500 msgr2=0x7f586806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:18.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:18.304+0000 7f588f800640 1 --2- 172.21.15.6:0/2353081968 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f586806b500 0x7f586806d9c0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f587c000f60 tx=0x7f587c007b70 comp rx=0 tx=0).stop 2024-03-04T09:29:18.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:18.304+0000 7f588f800640 1 -- 172.21.15.6:0/2353081968 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f58880cdc20 msgr2=0x7f588809b640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:18.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:18.304+0000 7f588f800640 1 --2- 172.21.15.6:0/2353081968 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f58880cdc20 0x7f588809b640 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f5880007e60 tx=0x7f588000e4b0 comp rx=0 tx=0).stop 2024-03-04T09:29:18.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:18.304+0000 7f588f800640 1 -- 172.21.15.6:0/2353081968 shutdown_connections 2024-03-04T09:29:18.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:18.304+0000 7f588f800640 1 --2- 172.21.15.6:0/2353081968 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f58880cdc20 0x7f588809b640 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:18.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:18.304+0000 7f588f800640 1 --2- 172.21.15.6:0/2353081968 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f58880af580 0x7f58881f1870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:18.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:18.304+0000 7f588f800640 1 --2- 172.21.15.6:0/2353081968 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f586806b500 0x7f586806d9c0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:18.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:18.304+0000 7f588f800640 1 --2- 172.21.15.6:0/2353081968 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f58880af1a0 0x7f5888154af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:18.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:18.305+0000 7f588f800640 1 -- 172.21.15.6:0/2353081968 >> 172.21.15.6:0/2353081968 conn(0x7f588806afb0 msgr2=0x7f5888072020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:29:18.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:18.305+0000 7f588f800640 1 -- 172.21.15.6:0/2353081968 shutdown_connections 2024-03-04T09:29:18.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:18.305+0000 7f588f800640 1 -- 172.21.15.6:0/2353081968 wait complete. 2024-03-04T09:29:18.322 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json perf dump mds_cache strays_enqueued 2024-03-04T09:29:18.555 INFO:teuthology.orchestra.run.smithi146.stdout:{"mds_cache":{"strays_enqueued":0}} 2024-03-04T09:29:18.555 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 0 } } 2024-03-04T09:29:18.555 DEBUG:tasks.ceph_test_case:wait_until_equal: 0 != 10, waiting (timeout=60)... 2024-03-04T09:29:23.557 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:29:23.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:23.788+0000 7f605ae00640 1 Processor -- start 2024-03-04T09:29:23.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:23.788+0000 7f605ae00640 1 -- start start 2024-03-04T09:29:23.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:23.790+0000 7f605ae00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f605415e070 0x7f60540d2960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:23.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:23.790+0000 7f605ae00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f60540cf410 0x7f60540cf870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:23.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:23.790+0000 7f605ae00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f60540d05c0 0x7f60540d0a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:23.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:23.790+0000 7f605ae00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60540d18c0 con 0x7f605415e070 2024-03-04T09:29:23.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:23.790+0000 7f605ae00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60540d1ba0 con 0x7f60540cf410 2024-03-04T09:29:23.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:23.790+0000 7f605ae00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60540d1e80 con 0x7f60540d05c0 2024-03-04T09:29:23.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:23.790+0000 7f6059a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f605415e070 0x7f60540d2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:23.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:23.791+0000 7f6059a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f605415e070 0x7f60540d2960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:46178/0 (socket says 172.21.15.6:46178) 2024-03-04T09:29:23.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:23.791+0000 7f6059a00640 1 -- 172.21.15.6:0/1033785197 learned_addr learned my addr 172.21.15.6:0/1033785197 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:29:23.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:23.791+0000 7f6059000640 1 --2- 172.21.15.6:0/1033785197 >> v2:172.21.15.146:3300/0 conn(0x7f60540cf410 0x7f60540cf870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:23.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:23.791+0000 7f605a400640 1 --2- 172.21.15.6:0/1033785197 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f60540d05c0 0x7f60540d0a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:23.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:23.791+0000 7f6059a00640 1 -- 172.21.15.6:0/1033785197 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f60540d05c0 msgr2=0x7f60540d0a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:23.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:23.791+0000 7f6059a00640 1 --2- 172.21.15.6:0/1033785197 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f60540d05c0 0x7f60540d0a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:23.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:23.791+0000 7f6059a00640 1 -- 172.21.15.6:0/1033785197 >> v2:172.21.15.146:3300/0 conn(0x7f60540cf410 msgr2=0x7f60540cf870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:23.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:23.791+0000 7f6059a00640 1 --2- 172.21.15.6:0/1033785197 >> v2:172.21.15.146:3300/0 conn(0x7f60540cf410 0x7f60540cf870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:23.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:23.791+0000 7f6059a00640 1 -- 172.21.15.6:0/1033785197 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60540d2160 con 0x7f605415e070 2024-03-04T09:29:23.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:23.792+0000 7f6059a00640 1 --2- 172.21.15.6:0/1033785197 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f605415e070 0x7f60540d2960 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f60540b6880 tx=0x7f604802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fe458cd11743cdd server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:23.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:23.792+0000 7f6053e00640 1 -- 172.21.15.6:0/1033785197 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6048035020 con 0x7f605415e070 2024-03-04T09:29:23.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:23.793+0000 7f6053e00640 1 -- 172.21.15.6:0/1033785197 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6048031870 con 0x7f605415e070 2024-03-04T09:29:23.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:23.793+0000 7f6053e00640 1 -- 172.21.15.6:0/1033785197 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6048031c40 con 0x7f605415e070 2024-03-04T09:29:23.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:23.793+0000 7f605ae00640 1 -- 172.21.15.6:0/1033785197 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f605415e070 msgr2=0x7f60540d2960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:23.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:23.793+0000 7f605ae00640 1 --2- 172.21.15.6:0/1033785197 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f605415e070 0x7f60540d2960 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f60540b6880 tx=0x7f604802b910 comp rx=0 tx=0).stop 2024-03-04T09:29:23.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:23.793+0000 7f605ae00640 1 -- 172.21.15.6:0/1033785197 shutdown_connections 2024-03-04T09:29:23.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:23.793+0000 7f605ae00640 1 --2- 172.21.15.6:0/1033785197 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f60540d05c0 0x7f60540d0a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:23.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:23.793+0000 7f605ae00640 1 --2- 172.21.15.6:0/1033785197 >> v2:172.21.15.146:3300/0 conn(0x7f60540cf410 0x7f60540cf870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:23.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:23.793+0000 7f605ae00640 1 --2- 172.21.15.6:0/1033785197 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f605415e070 0x7f60540d2960 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:23.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:23.793+0000 7f605ae00640 1 -- 172.21.15.6:0/1033785197 >> 172.21.15.6:0/1033785197 conn(0x7f605408d2a0 msgr2=0x7f60540d3cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:29:23.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:23.793+0000 7f605ae00640 1 -- 172.21.15.6:0/1033785197 shutdown_connections 2024-03-04T09:29:23.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:23.793+0000 7f605ae00640 1 -- 172.21.15.6:0/1033785197 wait complete. 2024-03-04T09:29:23.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:23.795+0000 7f605ae00640 1 Processor -- start 2024-03-04T09:29:23.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:23.795+0000 7f605ae00640 1 -- start start 2024-03-04T09:29:23.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:23.796+0000 7f605ae00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f60540cf410 0x7f605415c130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:23.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:23.796+0000 7f605ae00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f60540d05c0 0x7f60540b6910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:23.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:23.796+0000 7f605ae00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f605415e070 0x7f60540b6e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:23.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:23.796+0000 7f605ae00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60540d4730 con 0x7f60540cf410 2024-03-04T09:29:23.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:23.796+0000 7f605ae00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60540d45b0 con 0x7f60540d05c0 2024-03-04T09:29:23.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:23.796+0000 7f605ae00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60540b3630 con 0x7f605415e070 2024-03-04T09:29:23.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:23.796+0000 7f6059a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f60540cf410 0x7f605415c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:23.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:23.796+0000 7f6059a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f60540cf410 0x7f605415c130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:46194/0 (socket says 172.21.15.6:46194) 2024-03-04T09:29:23.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:23.796+0000 7f6059a00640 1 -- 172.21.15.6:0/651949574 learned_addr learned my addr 172.21.15.6:0/651949574 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:29:23.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:23.797+0000 7f605a400640 1 --2- 172.21.15.6:0/651949574 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f605415e070 0x7f60540b6e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:23.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:23.797+0000 7f6059000640 1 --2- 172.21.15.6:0/651949574 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f60540d05c0 0x7f60540b6910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:23.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:23.797+0000 7f6059a00640 1 -- 172.21.15.6:0/651949574 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f605415e070 msgr2=0x7f60540b6e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:23.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:23.797+0000 7f6059a00640 1 --2- 172.21.15.6:0/651949574 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f605415e070 0x7f60540b6e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:23.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:23.797+0000 7f6059a00640 1 -- 172.21.15.6:0/651949574 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f60540d05c0 msgr2=0x7f60540b6910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:23.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:23.797+0000 7f6059a00640 1 --2- 172.21.15.6:0/651949574 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f60540d05c0 0x7f60540b6910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:23.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:23.797+0000 7f6059a00640 1 -- 172.21.15.6:0/651949574 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60540cc7a0 con 0x7f60540cf410 2024-03-04T09:29:23.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:23.797+0000 7f6059000640 1 --2- 172.21.15.6:0/651949574 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f60540d05c0 0x7f60540b6910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:29:23.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:23.797+0000 7f6059a00640 1 --2- 172.21.15.6:0/651949574 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f60540cf410 0x7f605415c130 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f6048000c00 tx=0x7f60480046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:23.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:23.798+0000 7f6051600640 1 -- 172.21.15.6:0/651949574 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6048035040 con 0x7f60540cf410 2024-03-04T09:29:23.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:23.798+0000 7f6051600640 1 -- 172.21.15.6:0/651949574 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6048030d00 con 0x7f60540cf410 2024-03-04T09:29:23.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:23.798+0000 7f6051600640 1 -- 172.21.15.6:0/651949574 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60480456c0 con 0x7f60540cf410 2024-03-04T09:29:23.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:23.798+0000 7f605ae00640 1 -- 172.21.15.6:0/651949574 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60540cbb90 con 0x7f60540cf410 2024-03-04T09:29:23.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:23.798+0000 7f605ae00640 1 -- 172.21.15.6:0/651949574 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60540cc150 con 0x7f60540cf410 2024-03-04T09:29:23.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:23.799+0000 7f605ae00640 1 -- 172.21.15.6:0/651949574 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f605404f330 con 0x7f60540cf410 2024-03-04T09:29:23.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:23.800+0000 7f6051600640 1 -- 172.21.15.6:0/651949574 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6048038020 con 0x7f60540cf410 2024-03-04T09:29:23.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:23.801+0000 7f6051600640 1 --2- 172.21.15.6:0/651949574 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f603406b4d0 0x7f603406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:23.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:23.801+0000 7f6051600640 1 -- 172.21.15.6:0/651949574 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f60480ac4e0 con 0x7f60540cf410 2024-03-04T09:29:23.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:23.802+0000 7f6059000640 1 --2- 172.21.15.6:0/651949574 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f603406b4d0 0x7f603406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:23.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:23.803+0000 7f6059000640 1 --2- 172.21.15.6:0/651949574 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f603406b4d0 0x7f603406d990 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f60540d3300 tx=0x7f6040008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:23.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:23.807+0000 7f6051600640 1 -- 172.21.15.6:0/651949574 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f604807b2d0 con 0x7f60540cf410 2024-03-04T09:29:24.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:24.046+0000 7f605ae00640 1 -- 172.21.15.6:0/651949574 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6054159e20 con 0x7f60540cf410 2024-03-04T09:29:24.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:24.047+0000 7f6051600640 1 -- 172.21.15.6:0/651949574 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+4979 (secure 0 0 0) 0x7f604807abd0 con 0x7f60540cf410 2024-03-04T09:29:24.044 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:29:24.044 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":42,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5228,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/1891254736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1891254736},{"type":"v1","addr":"172.21.15.6:6834","nonce":1891254736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":5578,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/1781219843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":1781219843},{"type":"v1","addr":"172.21.15.146:6835","nonce":1781219843}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":5587,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/1479180802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":1479180802},{"type":"v1","addr":"172.21.15.6:6835","nonce":1479180802}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:28:50.708467+0000","modified":"2024-03-04T09:28:53.801285+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5590},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5590":{"gid":5590,"name":"d","rank":0,"incarnation":38,"state":"up:active","state_seq":3,"addr":"172.21.15.146:6837/3919939129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":3919939129},{"type":"v1","addr":"172.21.15.146:6837","nonce":3919939129}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5590,"qdb_cluster":[5590]},"id":5}]} 2024-03-04T09:29:24.044 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 42 2024-03-04T09:29:24.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:24.049+0000 7f605ae00640 1 -- 172.21.15.6:0/651949574 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f603406b4d0 msgr2=0x7f603406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:24.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:24.049+0000 7f605ae00640 1 --2- 172.21.15.6:0/651949574 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f603406b4d0 0x7f603406d990 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f60540d3300 tx=0x7f6040008040 comp rx=0 tx=0).stop 2024-03-04T09:29:24.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:24.049+0000 7f605ae00640 1 -- 172.21.15.6:0/651949574 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f60540cf410 msgr2=0x7f605415c130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:24.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:24.049+0000 7f605ae00640 1 --2- 172.21.15.6:0/651949574 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f60540cf410 0x7f605415c130 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f6048000c00 tx=0x7f60480046c0 comp rx=0 tx=0).stop 2024-03-04T09:29:24.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:24.049+0000 7f605ae00640 1 -- 172.21.15.6:0/651949574 shutdown_connections 2024-03-04T09:29:24.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:24.049+0000 7f605ae00640 1 --2- 172.21.15.6:0/651949574 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f603406b4d0 0x7f603406d990 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:24.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:24.050+0000 7f605ae00640 1 --2- 172.21.15.6:0/651949574 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f605415e070 0x7f60540b6e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:24.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:24.050+0000 7f605ae00640 1 --2- 172.21.15.6:0/651949574 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f60540d05c0 0x7f60540b6910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:24.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:24.050+0000 7f605ae00640 1 --2- 172.21.15.6:0/651949574 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f60540cf410 0x7f605415c130 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:24.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:24.050+0000 7f605ae00640 1 -- 172.21.15.6:0/651949574 >> 172.21.15.6:0/651949574 conn(0x7f605408d2a0 msgr2=0x7f605415d180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:29:24.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:24.050+0000 7f605ae00640 1 -- 172.21.15.6:0/651949574 shutdown_connections 2024-03-04T09:29:24.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:24.050+0000 7f605ae00640 1 -- 172.21.15.6:0/651949574 wait complete. 2024-03-04T09:29:24.067 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json perf dump mds_cache strays_enqueued 2024-03-04T09:29:24.298 INFO:teuthology.orchestra.run.smithi146.stdout:{"mds_cache":{"strays_enqueued":10}} 2024-03-04T09:29:24.298 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 10 } } 2024-03-04T09:29:24.298 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:29:24.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:24.533+0000 7f3f57e00640 1 Processor -- start 2024-03-04T09:29:24.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:24.533+0000 7f3f57e00640 1 -- start start 2024-03-04T09:29:24.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:24.535+0000 7f3f57e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f3f500cf5c0 0x7f3f500cdc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:24.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:24.535+0000 7f3f57e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f3f50071050 0x7f3f50071490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:24.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:24.535+0000 7f3f57e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f500b4860 con 0x7f3f500b3c90 2024-03-04T09:29:24.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:24.535+0000 7f3f57e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f500ce160 con 0x7f3f50071050 2024-03-04T09:29:24.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:24.535+0000 7f3f57e00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f500b4b40 con 0x7f3f500cf5c0 2024-03-04T09:29:24.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:24.535+0000 7f3f55400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f3f500b3c90 0x7f3f500b4070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:56636/0 (socket says 172.21.15.6:56636) 2024-03-04T09:29:24.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:24.535+0000 7f3f55400640 1 -- 172.21.15.6:0/2321665212 learned_addr learned my addr 172.21.15.6:0/2321665212 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:29:24.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:24.535+0000 7f3f55e00640 1 --2- 172.21.15.6:0/2321665212 >> v2:172.21.15.6:3300/0 conn(0x7f3f50071050 0x7f3f50071490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:24.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:24.536+0000 7f3f54a00640 1 --2- 172.21.15.6:0/2321665212 >> v2:172.21.15.146:3300/0 conn(0x7f3f500cf5c0 0x7f3f500cdc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:24.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:24.536+0000 7f3f4fe00640 1 -- 172.21.15.6:0/2321665212 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3160658258 0 0) 0x7f3f500b4860 con 0x7f3f500b3c90 2024-03-04T09:29:24.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:24.536+0000 7f3f55e00640 1 -- 172.21.15.6:0/2321665212 >> v2:172.21.15.146:3300/0 conn(0x7f3f500cf5c0 msgr2=0x7f3f500cdc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:24.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:24.536+0000 7f3f55e00640 1 --2- 172.21.15.6:0/2321665212 >> v2:172.21.15.146:3300/0 conn(0x7f3f500cf5c0 0x7f3f500cdc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:24.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:24.536+0000 7f3f55e00640 1 -- 172.21.15.6:0/2321665212 >> v1:172.21.15.6:6789/0 conn(0x7f3f500b3c90 legacy=0x7f3f500b4070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:24.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:24.536+0000 7f3f55e00640 1 -- 172.21.15.6:0/2321665212 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f500b4e20 con 0x7f3f50071050 2024-03-04T09:29:24.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:24.537+0000 7f3f55e00640 1 --2- 172.21.15.6:0/2321665212 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3f50071050 0x7f3f50071490 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f3f5006be30 tx=0x7f3f4802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b024d219c231f99 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:24.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:24.537+0000 7f3f4fe00640 1 -- 172.21.15.6:0/2321665212 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f48039070 con 0x7f3f50071050 2024-03-04T09:29:24.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:24.537+0000 7f3f4fe00640 1 -- 172.21.15.6:0/2321665212 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f4802cd10 con 0x7f3f50071050 2024-03-04T09:29:24.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:24.537+0000 7f3f4fe00640 1 -- 172.21.15.6:0/2321665212 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f48034660 con 0x7f3f50071050 2024-03-04T09:29:24.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:24.538+0000 7f3f57e00640 1 -- 172.21.15.6:0/2321665212 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3f50071050 msgr2=0x7f3f50071490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:24.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:24.538+0000 7f3f57e00640 1 --2- 172.21.15.6:0/2321665212 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3f50071050 0x7f3f50071490 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f3f5006be30 tx=0x7f3f4802bf60 comp rx=0 tx=0).stop 2024-03-04T09:29:24.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:24.538+0000 7f3f57e00640 1 -- 172.21.15.6:0/2321665212 shutdown_connections 2024-03-04T09:29:24.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:24.538+0000 7f3f57e00640 1 --2- 172.21.15.6:0/2321665212 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3f50071050 0x7f3f50071490 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:24.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:24.538+0000 7f3f57e00640 1 --2- 172.21.15.6:0/2321665212 >> v2:172.21.15.146:3300/0 conn(0x7f3f500cf5c0 0x7f3f500cdc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:24.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:24.538+0000 7f3f57e00640 1 -- 172.21.15.6:0/2321665212 >> 172.21.15.6:0/2321665212 conn(0x7f3f5006afb0 msgr2=0x7f3f5006b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:29:24.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:24.538+0000 7f3f57e00640 1 -- 172.21.15.6:0/2321665212 shutdown_connections 2024-03-04T09:29:24.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:24.538+0000 7f3f57e00640 1 -- 172.21.15.6:0/2321665212 wait complete. 2024-03-04T09:29:24.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:24.540+0000 7f3f57e00640 1 Processor -- start 2024-03-04T09:29:24.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:24.540+0000 7f3f57e00640 1 -- start start 2024-03-04T09:29:24.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:24.541+0000 7f3f57e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3f50071050 0x7f3f5014cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:24.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:24.541+0000 7f3f57e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3f500b3c90 0x7f3f5014e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:24.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:24.541+0000 7f3f57e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3f500cf5c0 0x7f3f500ce2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:24.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:24.541+0000 7f3f57e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f500719d0 con 0x7f3f500cf5c0 2024-03-04T09:29:24.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:24.541+0000 7f3f57e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f5015b340 con 0x7f3f500b3c90 2024-03-04T09:29:24.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:24.541+0000 7f3f57e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f50071b50 con 0x7f3f50071050 2024-03-04T09:29:24.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:24.541+0000 7f3f55e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3f500cf5c0 0x7f3f500ce2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:24.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:24.541+0000 7f3f55e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3f500cf5c0 0x7f3f500ce2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:46212/0 (socket says 172.21.15.6:46212) 2024-03-04T09:29:24.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:24.541+0000 7f3f55e00640 1 -- 172.21.15.6:0/2214161908 learned_addr learned my addr 172.21.15.6:0/2214161908 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:29:24.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:24.542+0000 7f3f55e00640 1 -- 172.21.15.6:0/2214161908 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3f50071050 msgr2=0x7f3f5014cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:24.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:24.542+0000 7f3f55400640 1 --2- 172.21.15.6:0/2214161908 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3f50071050 0x7f3f5014cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:24.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:24.542+0000 7f3f54a00640 1 --2- 172.21.15.6:0/2214161908 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3f500b3c90 0x7f3f5014e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:24.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:24.542+0000 7f3f55e00640 1 --2- 172.21.15.6:0/2214161908 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3f50071050 0x7f3f5014cee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:24.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:24.542+0000 7f3f55e00640 1 -- 172.21.15.6:0/2214161908 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3f500b3c90 msgr2=0x7f3f5014e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:24.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:24.542+0000 7f3f55e00640 1 --2- 172.21.15.6:0/2214161908 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3f500b3c90 0x7f3f5014e860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:24.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:24.542+0000 7f3f55e00640 1 -- 172.21.15.6:0/2214161908 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f5014d4c0 con 0x7f3f500cf5c0 2024-03-04T09:29:24.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:24.542+0000 7f3f54a00640 1 --2- 172.21.15.6:0/2214161908 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3f500b3c90 0x7f3f5014e860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:29:24.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:24.542+0000 7f3f55e00640 1 --2- 172.21.15.6:0/2214161908 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3f500cf5c0 0x7f3f500ce2e0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f3f48009e80 tx=0x7f3f48004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:24.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:24.542+0000 7f3f4d600640 1 -- 172.21.15.6:0/2214161908 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f48039050 con 0x7f3f500cf5c0 2024-03-04T09:29:24.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:24.542+0000 7f3f4d600640 1 -- 172.21.15.6:0/2214161908 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f48045070 con 0x7f3f500cf5c0 2024-03-04T09:29:24.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:24.542+0000 7f3f57e00640 1 -- 172.21.15.6:0/2214161908 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f5009d360 con 0x7f3f500cf5c0 2024-03-04T09:29:24.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:24.542+0000 7f3f4d600640 1 -- 172.21.15.6:0/2214161908 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f4803e830 con 0x7f3f500cf5c0 2024-03-04T09:29:24.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:24.543+0000 7f3f57e00640 1 -- 172.21.15.6:0/2214161908 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f50149f90 con 0x7f3f500cf5c0 2024-03-04T09:29:24.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:24.544+0000 7f3f57e00640 1 -- 172.21.15.6:0/2214161908 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f50069510 con 0x7f3f500cf5c0 2024-03-04T09:29:24.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:24.545+0000 7f3f4d600640 1 -- 172.21.15.6:0/2214161908 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3f4803e9d0 con 0x7f3f500cf5c0 2024-03-04T09:29:24.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:24.546+0000 7f3f4d600640 1 --2- 172.21.15.6:0/2214161908 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f3f3006b500 0x7f3f3006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:24.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:24.546+0000 7f3f4d600640 1 -- 172.21.15.6:0/2214161908 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f3f480ad110 con 0x7f3f500cf5c0 2024-03-04T09:29:24.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:24.546+0000 7f3f55400640 1 --2- 172.21.15.6:0/2214161908 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f3f3006b500 0x7f3f3006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:24.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:24.547+0000 7f3f55400640 1 --2- 172.21.15.6:0/2214161908 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f3f3006b500 0x7f3f3006d9c0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f3f44000f60 tx=0x7f3f44009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:24.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:24.550+0000 7f3f4d600640 1 -- 172.21.15.6:0/2214161908 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3f48032e20 con 0x7f3f500cf5c0 2024-03-04T09:29:24.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:24.793+0000 7f3f57e00640 1 -- 172.21.15.6:0/2214161908 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3f5009de70 con 0x7f3f500cf5c0 2024-03-04T09:29:24.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:24.794+0000 7f3f4d600640 1 -- 172.21.15.6:0/2214161908 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+4979 (secure 0 0 0) 0x7f3f4807be70 con 0x7f3f500cf5c0 2024-03-04T09:29:24.791 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:29:24.791 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":42,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5228,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/1891254736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1891254736},{"type":"v1","addr":"172.21.15.6:6834","nonce":1891254736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":5578,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/1781219843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":1781219843},{"type":"v1","addr":"172.21.15.146:6835","nonce":1781219843}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":5587,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/1479180802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":1479180802},{"type":"v1","addr":"172.21.15.6:6835","nonce":1479180802}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:28:50.708467+0000","modified":"2024-03-04T09:28:53.801285+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5590},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5590":{"gid":5590,"name":"d","rank":0,"incarnation":38,"state":"up:active","state_seq":3,"addr":"172.21.15.146:6837/3919939129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":3919939129},{"type":"v1","addr":"172.21.15.146:6837","nonce":3919939129}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5590,"qdb_cluster":[5590]},"id":5}]} 2024-03-04T09:29:24.791 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 42 2024-03-04T09:29:24.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:24.796+0000 7f3f57e00640 1 -- 172.21.15.6:0/2214161908 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f3f3006b500 msgr2=0x7f3f3006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:24.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:24.796+0000 7f3f57e00640 1 --2- 172.21.15.6:0/2214161908 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f3f3006b500 0x7f3f3006d9c0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f3f44000f60 tx=0x7f3f44009040 comp rx=0 tx=0).stop 2024-03-04T09:29:24.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:24.796+0000 7f3f57e00640 1 -- 172.21.15.6:0/2214161908 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3f500cf5c0 msgr2=0x7f3f500ce2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:24.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:24.796+0000 7f3f57e00640 1 --2- 172.21.15.6:0/2214161908 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3f500cf5c0 0x7f3f500ce2e0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f3f48009e80 tx=0x7f3f48004a40 comp rx=0 tx=0).stop 2024-03-04T09:29:24.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:24.797+0000 7f3f57e00640 1 -- 172.21.15.6:0/2214161908 shutdown_connections 2024-03-04T09:29:24.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:24.797+0000 7f3f57e00640 1 --2- 172.21.15.6:0/2214161908 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3f500cf5c0 0x7f3f500ce2e0 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:24.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:24.797+0000 7f3f57e00640 1 --2- 172.21.15.6:0/2214161908 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3f500b3c90 0x7f3f5014e860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:24.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:24.797+0000 7f3f57e00640 1 --2- 172.21.15.6:0/2214161908 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f3f3006b500 0x7f3f3006d9c0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:24.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:24.797+0000 7f3f57e00640 1 --2- 172.21.15.6:0/2214161908 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3f50071050 0x7f3f5014cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:24.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:24.797+0000 7f3f57e00640 1 -- 172.21.15.6:0/2214161908 >> 172.21.15.6:0/2214161908 conn(0x7f3f5006afb0 msgr2=0x7f3f5014fc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:29:24.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:24.797+0000 7f3f57e00640 1 -- 172.21.15.6:0/2214161908 shutdown_connections 2024-03-04T09:29:24.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:24.797+0000 7f3f57e00640 1 -- 172.21.15.6:0/2214161908 wait complete. 2024-03-04T09:29:24.813 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json perf dump purge_queue pq_executed 2024-03-04T09:29:25.043 INFO:teuthology.orchestra.run.smithi146.stdout:{"purge_queue":{"pq_executed":10}} 2024-03-04T09:29:25.043 DEBUG:tasks.cephfs.filesystem:_json_asok output { "purge_queue": { "pq_executed": 10 } } 2024-03-04T09:29:25.044 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-04T09:29:25.044 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T09:29:25.105 DEBUG:teuthology.orchestra.run.smithi006:> stdin-killer -- python3 -c ' 2024-03-04T09:29:25.105 DEBUG:teuthology.orchestra.run.smithi006:> import os 2024-03-04T09:29:25.105 DEBUG:teuthology.orchestra.run.smithi006:> import uuid 2024-03-04T09:29:25.105 DEBUG:teuthology.orchestra.run.smithi006:> 2024-03-04T09:29:25.105 DEBUG:teuthology.orchestra.run.smithi006:> n = 10 2024-03-04T09:29:25.105 DEBUG:teuthology.orchestra.run.smithi006:> create_hard_links = False 2024-03-04T09:29:25.106 DEBUG:teuthology.orchestra.run.smithi006:> if 0 > 0: 2024-03-04T09:29:25.106 DEBUG:teuthology.orchestra.run.smithi006:> create_hard_links = True 2024-03-04T09:29:25.106 DEBUG:teuthology.orchestra.run.smithi006:> path = "/home/ubuntu/cephtest/mnt.0/subdir/file" 2024-03-04T09:29:25.106 DEBUG:teuthology.orchestra.run.smithi006:> 2024-03-04T09:29:25.106 DEBUG:teuthology.orchestra.run.smithi006:> dpath = os.path.dirname(path) 2024-03-04T09:29:25.106 DEBUG:teuthology.orchestra.run.smithi006:> fnameprefix = os.path.basename(path) 2024-03-04T09:29:25.106 DEBUG:teuthology.orchestra.run.smithi006:> os.makedirs(dpath, exist_ok=True) 2024-03-04T09:29:25.106 DEBUG:teuthology.orchestra.run.smithi006:> 2024-03-04T09:29:25.106 DEBUG:teuthology.orchestra.run.smithi006:> try: 2024-03-04T09:29:25.106 DEBUG:teuthology.orchestra.run.smithi006:> dirfd = os.open(dpath, os.O_DIRECTORY) 2024-03-04T09:29:25.106 DEBUG:teuthology.orchestra.run.smithi006:> 2024-03-04T09:29:25.106 DEBUG:teuthology.orchestra.run.smithi006:> for i in range(n): 2024-03-04T09:29:25.106 DEBUG:teuthology.orchestra.run.smithi006:> fpath = os.path.join(dpath, f"{fnameprefix}_{i}") 2024-03-04T09:29:25.107 DEBUG:teuthology.orchestra.run.smithi006:> with open(fpath, '"'"'w'"'"') as f: 2024-03-04T09:29:25.107 DEBUG:teuthology.orchestra.run.smithi006:> f.write(f"{i}") 2024-03-04T09:29:25.107 DEBUG:teuthology.orchestra.run.smithi006:> if False: 2024-03-04T09:29:25.107 DEBUG:teuthology.orchestra.run.smithi006:> f.flush() 2024-03-04T09:29:25.107 DEBUG:teuthology.orchestra.run.smithi006:> os.fsync(f.fileno()) 2024-03-04T09:29:25.107 DEBUG:teuthology.orchestra.run.smithi006:> if True: 2024-03-04T09:29:25.107 DEBUG:teuthology.orchestra.run.smithi006:> os.unlink(fpath) 2024-03-04T09:29:25.107 DEBUG:teuthology.orchestra.run.smithi006:> if True: 2024-03-04T09:29:25.107 DEBUG:teuthology.orchestra.run.smithi006:> os.fsync(dirfd) 2024-03-04T09:29:25.107 DEBUG:teuthology.orchestra.run.smithi006:> if create_hard_links: 2024-03-04T09:29:25.107 DEBUG:teuthology.orchestra.run.smithi006:> for j in range(0): 2024-03-04T09:29:25.108 DEBUG:teuthology.orchestra.run.smithi006:> os.system(f"ln {fpath} {dpath}/{fnameprefix}_{i}_{uuid.uuid4()}") 2024-03-04T09:29:25.108 DEBUG:teuthology.orchestra.run.smithi006:> if False: 2024-03-04T09:29:25.108 DEBUG:teuthology.orchestra.run.smithi006:> os.fsync(dirfd) 2024-03-04T09:29:25.108 DEBUG:teuthology.orchestra.run.smithi006:> finally: 2024-03-04T09:29:25.108 DEBUG:teuthology.orchestra.run.smithi006:> os.close(dirfd) 2024-03-04T09:29:25.108 DEBUG:teuthology.orchestra.run.smithi006:> ' 2024-03-04T09:29:25.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:25 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-04T09:29:25.414 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-04T09:29:25.415 DEBUG:teuthology.orchestra.run.smithi006:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-04T09:29:25.573 DEBUG:teuthology.orchestra.run.smithi146:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-04T09:29:25.684 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-04T09:29:25.684 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-04T09:29:25.684 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T09:29:25.745 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-04T09:29:25.746 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T09:29:25.803 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T09:29:26.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.031+0000 7f5459a00640 1 Processor -- start 2024-03-04T09:29:26.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.032+0000 7f5459a00640 1 -- start start 2024-03-04T09:29:26.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.033+0000 7f5459a00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f54540cf5c0 0x7f54540cdc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:26.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.033+0000 7f5459a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f5454071050 0x7f5454071490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:26.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.033+0000 7f5459a00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54540b4860 con 0x7f54540b3c90 2024-03-04T09:29:26.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.033+0000 7f5459a00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54540ce160 con 0x7f5454071050 2024-03-04T09:29:26.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.033+0000 7f5459a00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54540b4b40 con 0x7f54540cf5c0 2024-03-04T09:29:26.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.034+0000 7f5453400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f5454071050 0x7f5454071490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:26.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.034+0000 7f5453400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f5454071050 0x7f5454071490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:46228/0 (socket says 172.21.15.6:46228) 2024-03-04T09:29:26.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.034+0000 7f5453400640 1 -- 172.21.15.6:0/4177662760 learned_addr learned my addr 172.21.15.6:0/4177662760 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:29:26.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.034+0000 7f544a000640 1 --2- 172.21.15.6:0/4177662760 >> v2:172.21.15.146:3300/0 conn(0x7f54540cf5c0 0x7f54540cdc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:26.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.034+0000 7f5453400640 1 -- 172.21.15.6:0/4177662760 >> v2:172.21.15.146:3300/0 conn(0x7f54540cf5c0 msgr2=0x7f54540cdc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:26.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.034+0000 7f5453400640 1 --2- 172.21.15.6:0/4177662760 >> v2:172.21.15.146:3300/0 conn(0x7f54540cf5c0 0x7f54540cdc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:26.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.034+0000 7f5453400640 1 -- 172.21.15.6:0/4177662760 >> v1:172.21.15.146:6789/0 conn(0x7f54540b3c90 legacy=0x7f54540b4070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:26.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.034+0000 7f5453400640 1 -- 172.21.15.6:0/4177662760 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54540b4e20 con 0x7f5454071050 2024-03-04T09:29:26.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.036+0000 7f5453400640 1 --2- 172.21.15.6:0/4177662760 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5454071050 0x7f5454071490 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f545406be30 tx=0x7f544c02e210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1abeebbcedd238b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:26.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.036+0000 7f5452000640 1 -- 172.21.15.6:0/4177662760 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f544c00b0d0 con 0x7f5454071050 2024-03-04T09:29:26.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.036+0000 7f5452000640 1 -- 172.21.15.6:0/4177662760 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f544c00b270 con 0x7f5454071050 2024-03-04T09:29:26.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.036+0000 7f5452000640 1 -- 172.21.15.6:0/4177662760 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f544c009600 con 0x7f5454071050 2024-03-04T09:29:26.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.036+0000 7f5459a00640 1 -- 172.21.15.6:0/4177662760 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5454071050 msgr2=0x7f5454071490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:26.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.036+0000 7f5459a00640 1 --2- 172.21.15.6:0/4177662760 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5454071050 0x7f5454071490 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f545406be30 tx=0x7f544c02e210 comp rx=0 tx=0).stop 2024-03-04T09:29:26.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.037+0000 7f5459a00640 1 -- 172.21.15.6:0/4177662760 shutdown_connections 2024-03-04T09:29:26.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.037+0000 7f5459a00640 1 --2- 172.21.15.6:0/4177662760 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5454071050 0x7f5454071490 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:26.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.037+0000 7f5459a00640 1 --2- 172.21.15.6:0/4177662760 >> v2:172.21.15.146:3300/0 conn(0x7f54540cf5c0 0x7f54540cdc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:26.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.037+0000 7f5459a00640 1 -- 172.21.15.6:0/4177662760 >> 172.21.15.6:0/4177662760 conn(0x7f545406afb0 msgr2=0x7f545406b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:29:26.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.037+0000 7f5459a00640 1 -- 172.21.15.6:0/4177662760 shutdown_connections 2024-03-04T09:29:26.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.037+0000 7f5459a00640 1 -- 172.21.15.6:0/4177662760 wait complete. 2024-03-04T09:29:26.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.038+0000 7f5459a00640 1 Processor -- start 2024-03-04T09:29:26.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.039+0000 7f5459a00640 1 -- start start 2024-03-04T09:29:26.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.040+0000 7f5459a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5454071050 0x7f545409d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:26.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.040+0000 7f5459a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54540b3c90 0x7f545409b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:26.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.040+0000 7f5459a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f54540cf5c0 0x7f545414d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:26.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.040+0000 7f5459a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54540719d0 con 0x7f54540b3c90 2024-03-04T09:29:26.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.040+0000 7f5459a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f545415b340 con 0x7f54540cf5c0 2024-03-04T09:29:26.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.040+0000 7f5459a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5454071b50 con 0x7f5454071050 2024-03-04T09:29:26.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.040+0000 7f544a000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54540b3c90 0x7f545409b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:26.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.040+0000 7f544a000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54540b3c90 0x7f545409b420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:46238/0 (socket says 172.21.15.6:46238) 2024-03-04T09:29:26.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.040+0000 7f544a000640 1 -- 172.21.15.6:0/983358403 learned_addr learned my addr 172.21.15.6:0/983358403 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:29:26.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.040+0000 7f544a000640 1 -- 172.21.15.6:0/983358403 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5454071050 msgr2=0x7f545409d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:26.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.040+0000 7f5453400640 1 --2- 172.21.15.6:0/983358403 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f54540cf5c0 0x7f545414d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:26.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.040+0000 7f5452a00640 1 --2- 172.21.15.6:0/983358403 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5454071050 0x7f545409d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:26.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.041+0000 7f544a000640 1 --2- 172.21.15.6:0/983358403 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5454071050 0x7f545409d7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:26.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.041+0000 7f544a000640 1 -- 172.21.15.6:0/983358403 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f54540cf5c0 msgr2=0x7f545414d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:26.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.041+0000 7f544a000640 1 --2- 172.21.15.6:0/983358403 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f54540cf5c0 0x7f545414d9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:26.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.041+0000 7f544a000640 1 -- 172.21.15.6:0/983358403 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f545409b960 con 0x7f54540b3c90 2024-03-04T09:29:26.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.041+0000 7f5453400640 1 --2- 172.21.15.6:0/983358403 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f54540cf5c0 0x7f545414d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:29:26.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.041+0000 7f5452a00640 1 --2- 172.21.15.6:0/983358403 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5454071050 0x7f545409d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:29:26.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.041+0000 7f544a000640 1 --2- 172.21.15.6:0/983358403 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54540b3c90 0x7f545409b420 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f543c00d5d0 tx=0x7f543c00da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:26.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.041+0000 7f544b400640 1 -- 172.21.15.6:0/983358403 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f543c0125f0 con 0x7f54540b3c90 2024-03-04T09:29:26.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.041+0000 7f544b400640 1 -- 172.21.15.6:0/983358403 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f543c019070 con 0x7f54540b3c90 2024-03-04T09:29:26.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.041+0000 7f544b400640 1 -- 172.21.15.6:0/983358403 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f543c0129f0 con 0x7f54540b3c90 2024-03-04T09:29:26.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.041+0000 7f5459a00640 1 -- 172.21.15.6:0/983358403 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f545409cc80 con 0x7f54540b3c90 2024-03-04T09:29:26.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.042+0000 7f5459a00640 1 -- 172.21.15.6:0/983358403 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f545409bec0 con 0x7f54540b3c90 2024-03-04T09:29:26.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.043+0000 7f5459a00640 1 -- 172.21.15.6:0/983358403 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f545404f3a0 con 0x7f54540b3c90 2024-03-04T09:29:26.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.044+0000 7f544b400640 1 -- 172.21.15.6:0/983358403 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f543c012b90 con 0x7f54540b3c90 2024-03-04T09:29:26.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.045+0000 7f544b400640 1 --2- 172.21.15.6:0/983358403 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f543006b4d0 0x7f543006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:26.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.045+0000 7f544b400640 1 -- 172.21.15.6:0/983358403 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f543c08a760 con 0x7f54540b3c90 2024-03-04T09:29:26.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.045+0000 7f5452a00640 1 --2- 172.21.15.6:0/983358403 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f543006b4d0 0x7f543006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:26.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.046+0000 7f5452a00640 1 --2- 172.21.15.6:0/983358403 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f543006b4d0 0x7f543006d990 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f5444000f60 tx=0x7f5444009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:26.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.049+0000 7f544b400640 1 -- 172.21.15.6:0/983358403 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f545409b960 con 0x7f54540b3c90 2024-03-04T09:29:26.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.247+0000 7f5459a00640 1 -- 172.21.15.6:0/983358403 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f54540cefb0 con 0x7f54540b3c90 2024-03-04T09:29:26.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.250+0000 7f544b400640 1 -- 172.21.15.6:0/983358403 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v70) v1 ==== 74+0+16813 (secure 0 0 0) 0x7f543c0594c0 con 0x7f54540b3c90 2024-03-04T09:29:26.247 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:29:26.247 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":70,"fsid":"edce0578-5ae4-4402-b45a-552bdef84d7c","created":"2024-03-04T09:24:54.523780+0000","modified":"2024-03-04T09:29:02.333052+0000","last_up_change":"2024-03-04T09:25:07.089049+0000","last_in_change":"2024-03-04T09:24:57.982077+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T09:25:07.935198+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-03-04T09:28:48.627427+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-03-04T09:28:49.739043+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a96574f-81f1-4bc6-8463-757520b84a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6825","nonce":1364449401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6827","nonce":1364449401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6831","nonce":1364449401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6829","nonce":1364449401}]},"public_addr":"172.21.15.6:6825/1364449401","cluster_addr":"172.21.15.6:6827/1364449401","heartbeat_back_addr":"172.21.15.6:6831/1364449401","heartbeat_front_addr":"172.21.15.6:6829/1364449401","state":["exists","up"]},{"osd":1,"uuid":"7d79b9c1-6b87-476a-bc36-7db5ee64cd90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6809","nonce":2494103078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6811","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6813","nonce":2494103078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6821","nonce":2494103078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6817","nonce":2494103078}]},"public_addr":"172.21.15.6:6809/2494103078","cluster_addr":"172.21.15.6:6813/2494103078","heartbeat_back_addr":"172.21.15.6:6821/2494103078","heartbeat_front_addr":"172.21.15.6:6817/2494103078","state":["exists","up"]},{"osd":2,"uuid":"492331b9-4ee9-40b6-bacd-85d44cd6c45a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6812","nonce":1500066681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6816","nonce":1500066681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6824","nonce":1500066681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6820","nonce":1500066681}]},"public_addr":"172.21.15.6:6812/1500066681","cluster_addr":"172.21.15.6:6816/1500066681","heartbeat_back_addr":"172.21.15.6:6824/1500066681","heartbeat_front_addr":"172.21.15.6:6820/1500066681","state":["exists","up"]},{"osd":3,"uuid":"21bb2efe-9ba7-4bdc-bc31-180cd1102e21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6801","nonce":1558806123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6803","nonce":1558806123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6807","nonce":1558806123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6805","nonce":1558806123}]},"public_addr":"172.21.15.6:6801/1558806123","cluster_addr":"172.21.15.6:6803/1558806123","heartbeat_back_addr":"172.21.15.6:6807/1558806123","heartbeat_front_addr":"172.21.15.6:6805/1558806123","state":["exists","up"]},{"osd":4,"uuid":"6e2c4b71-19d9-4737-83d9-558705d8644a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6801","nonce":1526135780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6803","nonce":1526135780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6807","nonce":1526135780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6805","nonce":1526135780}]},"public_addr":"172.21.15.146:6801/1526135780","cluster_addr":"172.21.15.146:6803/1526135780","heartbeat_back_addr":"172.21.15.146:6807/1526135780","heartbeat_front_addr":"172.21.15.146:6805/1526135780","state":["exists","up"]},{"osd":5,"uuid":"fb69c525-f1be-4b7d-ab32-6c29b0d6fddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6809","nonce":460262507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6811","nonce":460262507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6815","nonce":460262507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6813","nonce":460262507}]},"public_addr":"172.21.15.146:6809/460262507","cluster_addr":"172.21.15.146:6811/460262507","heartbeat_back_addr":"172.21.15.146:6815/460262507","heartbeat_front_addr":"172.21.15.146:6813/460262507","state":["exists","up"]},{"osd":6,"uuid":"c32e7fed-370d-4401-8ce0-6a4335b0d168","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6823","nonce":1392897493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6827","nonce":1392897493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6831","nonce":1392897493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6829","nonce":1392897493}]},"public_addr":"172.21.15.146:6823/1392897493","cluster_addr":"172.21.15.146:6827/1392897493","heartbeat_back_addr":"172.21.15.146:6831/1392897493","heartbeat_front_addr":"172.21.15.146:6829/1392897493","state":["exists","up"]},{"osd":7,"uuid":"75a79b44-fa5f-426d-a7d3-71f21f78835c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6817","nonce":3268323156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6819","nonce":3268323156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6825","nonce":3268323156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6822","nonce":3268323156}]},"public_addr":"172.21.15.146:6817/3268323156","cluster_addr":"172.21.15.146:6819/3268323156","heartbeat_back_addr":"172.21.15.146:6825/3268323156","heartbeat_front_addr":"172.21.15.146:6822/3268323156","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.310437+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.295850+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.294674+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.526855+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.300023+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.595160+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.346046+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.558741+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"11.8","mappings":[{"from":6,"to":5}]},{"pgid":"11.10","mappings":[{"from":6,"to":1}]},{"pgid":"11.14","mappings":[{"from":6,"to":0}]},{"pgid":"11.3e","mappings":[{"from":6,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T09:29:26.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.253+0000 7f5459a00640 1 -- 172.21.15.6:0/983358403 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f543006b4d0 msgr2=0x7f543006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:26.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.253+0000 7f5459a00640 1 --2- 172.21.15.6:0/983358403 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f543006b4d0 0x7f543006d990 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f5444000f60 tx=0x7f5444009040 comp rx=0 tx=0).stop 2024-03-04T09:29:26.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.253+0000 7f5459a00640 1 -- 172.21.15.6:0/983358403 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54540b3c90 msgr2=0x7f545409b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:26.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.253+0000 7f5459a00640 1 --2- 172.21.15.6:0/983358403 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54540b3c90 0x7f545409b420 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f543c00d5d0 tx=0x7f543c00da90 comp rx=0 tx=0).stop 2024-03-04T09:29:26.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.253+0000 7f5459a00640 1 -- 172.21.15.6:0/983358403 shutdown_connections 2024-03-04T09:29:26.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.253+0000 7f5459a00640 1 --2- 172.21.15.6:0/983358403 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f54540cf5c0 0x7f545414d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:26.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.253+0000 7f5459a00640 1 --2- 172.21.15.6:0/983358403 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54540b3c90 0x7f545409b420 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:26.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.253+0000 7f5459a00640 1 --2- 172.21.15.6:0/983358403 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f543006b4d0 0x7f543006d990 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:26.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.253+0000 7f5459a00640 1 --2- 172.21.15.6:0/983358403 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5454071050 0x7f545409d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:26.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.253+0000 7f5459a00640 1 -- 172.21.15.6:0/983358403 >> 172.21.15.6:0/983358403 conn(0x7f545406afb0 msgr2=0x7f545406d0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:29:26.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.253+0000 7f5459a00640 1 -- 172.21.15.6:0/983358403 shutdown_connections 2024-03-04T09:29:26.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.253+0000 7f5459a00640 1 -- 172.21.15.6:0/983358403 wait complete. 2024-03-04T09:29:26.269 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T09:25:07.935198+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-04T09:28:48.627427+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '67', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_data', 'create_time': '2024-03-04T09:28:49.739043+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-04T09:29:26.270 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-04T09:29:26.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.499+0000 7fbb52400640 1 Processor -- start 2024-03-04T09:29:26.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.499+0000 7fbb52400640 1 -- start start 2024-03-04T09:29:26.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.500+0000 7fbb52400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbb4c06da40 0x7fbb4c06e340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:26.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.500+0000 7fbb52400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fbb4c0ccad0 0x7fbb4c0ccf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:26.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.500+0000 7fbb52400640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb4c06d090 con 0x7fbb4c15e680 2024-03-04T09:29:26.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.500+0000 7fbb52400640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb4c161c70 con 0x7fbb4c0ccad0 2024-03-04T09:29:26.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.500+0000 7fbb52400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb4c15eb80 con 0x7fbb4c06da40 2024-03-04T09:29:26.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.501+0000 7fbb4be00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fbb4c0ccad0 0x7fbb4c0ccf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:26.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.501+0000 7fbb4be00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fbb4c0ccad0 0x7fbb4c0ccf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:46252/0 (socket says 172.21.15.6:46252) 2024-03-04T09:29:26.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.501+0000 7fbb4be00640 1 -- 172.21.15.6:0/2691699088 learned_addr learned my addr 172.21.15.6:0/2691699088 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:29:26.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.501+0000 7fbb4b400640 1 --2- 172.21.15.6:0/2691699088 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbb4c06da40 0x7fbb4c06e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:26.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.501+0000 7fbb4be00640 1 -- 172.21.15.6:0/2691699088 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbb4c06da40 msgr2=0x7fbb4c06e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:26.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.501+0000 7fbb4be00640 1 --2- 172.21.15.6:0/2691699088 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbb4c06da40 0x7fbb4c06e340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:26.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.501+0000 7fbb4be00640 1 -- 172.21.15.6:0/2691699088 >> v1:172.21.15.146:6789/0 conn(0x7fbb4c15e680 legacy=0x7fbb4c06c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:26.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.502+0000 7fbb4be00640 1 -- 172.21.15.6:0/2691699088 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb4c15ee60 con 0x7fbb4c0ccad0 2024-03-04T09:29:26.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.502+0000 7fbb4b400640 1 --2- 172.21.15.6:0/2691699088 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbb4c06da40 0x7fbb4c06e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T09:29:26.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.503+0000 7fbb4be00640 1 --2- 172.21.15.6:0/2691699088 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbb4c0ccad0 0x7fbb4c0ccf80 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7fbb44009e50 tx=0x7fbb4402bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27a534fcffe07ac server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:26.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.503+0000 7fbb4aa00640 1 -- 172.21.15.6:0/2691699088 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb4402caf0 con 0x7fbb4c0ccad0 2024-03-04T09:29:26.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.504+0000 7fbb4aa00640 1 -- 172.21.15.6:0/2691699088 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb4402cc90 con 0x7fbb4c0ccad0 2024-03-04T09:29:26.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.504+0000 7fbb4aa00640 1 -- 172.21.15.6:0/2691699088 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb44034610 con 0x7fbb4c0ccad0 2024-03-04T09:29:26.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.504+0000 7fbb52400640 1 -- 172.21.15.6:0/2691699088 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbb4c0ccad0 msgr2=0x7fbb4c0ccf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:26.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.504+0000 7fbb52400640 1 --2- 172.21.15.6:0/2691699088 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbb4c0ccad0 0x7fbb4c0ccf80 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7fbb44009e50 tx=0x7fbb4402bf50 comp rx=0 tx=0).stop 2024-03-04T09:29:26.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.504+0000 7fbb52400640 1 -- 172.21.15.6:0/2691699088 shutdown_connections 2024-03-04T09:29:26.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.504+0000 7fbb52400640 1 --2- 172.21.15.6:0/2691699088 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbb4c0ccad0 0x7fbb4c0ccf80 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:26.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.504+0000 7fbb52400640 1 --2- 172.21.15.6:0/2691699088 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbb4c06da40 0x7fbb4c06e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:26.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.504+0000 7fbb52400640 1 -- 172.21.15.6:0/2691699088 >> 172.21.15.6:0/2691699088 conn(0x7fbb4c06b3d0 msgr2=0x7fbb4c06b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:29:26.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.504+0000 7fbb52400640 1 -- 172.21.15.6:0/2691699088 shutdown_connections 2024-03-04T09:29:26.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.505+0000 7fbb52400640 1 -- 172.21.15.6:0/2691699088 wait complete. 2024-03-04T09:29:26.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.506+0000 7fbb52400640 1 Processor -- start 2024-03-04T09:29:26.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.506+0000 7fbb52400640 1 -- start start 2024-03-04T09:29:26.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.507+0000 7fbb52400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbb4c06da40 0x7fbb4c09c740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:26.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.507+0000 7fbb52400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbb4c0ccad0 0x7fbb4c09aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:26.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.507+0000 7fbb52400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbb4c15e680 0x7fbb4c160a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:26.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.507+0000 7fbb52400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb4c06e880 con 0x7fbb4c06da40 2024-03-04T09:29:26.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.507+0000 7fbb52400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb4c15f420 con 0x7fbb4c15e680 2024-03-04T09:29:26.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.507+0000 7fbb52400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb4c06ea00 con 0x7fbb4c0ccad0 2024-03-04T09:29:26.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.508+0000 7fbb4b400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbb4c06da40 0x7fbb4c09c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:26.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.508+0000 7fbb4b400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbb4c06da40 0x7fbb4c09c740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:46254/0 (socket says 172.21.15.6:46254) 2024-03-04T09:29:26.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.508+0000 7fbb4b400640 1 -- 172.21.15.6:0/132805945 learned_addr learned my addr 172.21.15.6:0/132805945 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:29:26.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.508+0000 7fbb4be00640 1 --2- 172.21.15.6:0/132805945 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbb4c15e680 0x7fbb4c160a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:26.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.508+0000 7fbb43e00640 1 --2- 172.21.15.6:0/132805945 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbb4c0ccad0 0x7fbb4c09aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:26.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.508+0000 7fbb4b400640 1 -- 172.21.15.6:0/132805945 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbb4c0ccad0 msgr2=0x7fbb4c09aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:26.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.508+0000 7fbb4b400640 1 --2- 172.21.15.6:0/132805945 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbb4c0ccad0 0x7fbb4c09aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:26.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.508+0000 7fbb4b400640 1 -- 172.21.15.6:0/132805945 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbb4c15e680 msgr2=0x7fbb4c160a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:26.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.508+0000 7fbb4b400640 1 --2- 172.21.15.6:0/132805945 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbb4c15e680 0x7fbb4c160a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:26.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.508+0000 7fbb4b400640 1 -- 172.21.15.6:0/132805945 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb4c09b1b0 con 0x7fbb4c06da40 2024-03-04T09:29:26.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.508+0000 7fbb4b400640 1 --2- 172.21.15.6:0/132805945 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbb4c06da40 0x7fbb4c09c740 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7fbb3c00e810 tx=0x7fbb3c00ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:26.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.509+0000 7fbb43400640 1 -- 172.21.15.6:0/132805945 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb3c00f040 con 0x7fbb4c06da40 2024-03-04T09:29:26.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.509+0000 7fbb43400640 1 -- 172.21.15.6:0/132805945 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb3c004030 con 0x7fbb4c06da40 2024-03-04T09:29:26.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.509+0000 7fbb43400640 1 -- 172.21.15.6:0/132805945 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb3c004400 con 0x7fbb4c06da40 2024-03-04T09:29:26.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.509+0000 7fbb52400640 1 -- 172.21.15.6:0/132805945 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb4c09b490 con 0x7fbb4c06da40 2024-03-04T09:29:26.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.509+0000 7fbb52400640 1 -- 172.21.15.6:0/132805945 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb4c09ba50 con 0x7fbb4c06da40 2024-03-04T09:29:26.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.510+0000 7fbb52400640 1 -- 172.21.15.6:0/132805945 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb4c04f330 con 0x7fbb4c06da40 2024-03-04T09:29:26.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.511+0000 7fbb43400640 1 -- 172.21.15.6:0/132805945 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbb3c00c6d0 con 0x7fbb4c06da40 2024-03-04T09:29:26.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.512+0000 7fbb43400640 1 --2- 172.21.15.6:0/132805945 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fbb2806b550 0x7fbb2806da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:26.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.513+0000 7fbb43400640 1 -- 172.21.15.6:0/132805945 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fbb3c089eb0 con 0x7fbb4c06da40 2024-03-04T09:29:26.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.513+0000 7fbb43e00640 1 --2- 172.21.15.6:0/132805945 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fbb2806b550 0x7fbb2806da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:26.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.514+0000 7fbb43e00640 1 --2- 172.21.15.6:0/132805945 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fbb2806b550 0x7fbb2806da10 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fbb34007fd0 tx=0x7fbb34009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:26.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.517+0000 7fbb43400640 1 -- 172.21.15.6:0/132805945 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbb3c017020 con 0x7fbb4c06da40 2024-03-04T09:29:26.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.728+0000 7fbb52400640 1 -- 172.21.15.6:0/132805945 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fbb4c160850 con 0x7fbb4c06da40 2024-03-04T09:29:26.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.729+0000 7fbb43400640 1 -- 172.21.15.6:0/132805945 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v70) v1 ==== 93+0+10 (secure 0 0 0) 0x7fbb3c058e20 con 0x7fbb4c06da40 2024-03-04T09:29:26.726 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-04T09:29:26.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.732+0000 7fbb52400640 1 -- 172.21.15.6:0/132805945 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fbb2806b550 msgr2=0x7fbb2806da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:26.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.732+0000 7fbb52400640 1 --2- 172.21.15.6:0/132805945 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fbb2806b550 0x7fbb2806da10 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fbb34007fd0 tx=0x7fbb34009040 comp rx=0 tx=0).stop 2024-03-04T09:29:26.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.732+0000 7fbb52400640 1 -- 172.21.15.6:0/132805945 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbb4c06da40 msgr2=0x7fbb4c09c740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:26.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.732+0000 7fbb52400640 1 --2- 172.21.15.6:0/132805945 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbb4c06da40 0x7fbb4c09c740 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7fbb3c00e810 tx=0x7fbb3c00ecd0 comp rx=0 tx=0).stop 2024-03-04T09:29:26.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.732+0000 7fbb52400640 1 -- 172.21.15.6:0/132805945 shutdown_connections 2024-03-04T09:29:26.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.732+0000 7fbb52400640 1 --2- 172.21.15.6:0/132805945 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbb4c15e680 0x7fbb4c160a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:26.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.732+0000 7fbb52400640 1 --2- 172.21.15.6:0/132805945 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fbb2806b550 0x7fbb2806da10 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:26.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.732+0000 7fbb52400640 1 --2- 172.21.15.6:0/132805945 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbb4c0ccad0 0x7fbb4c09aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:26.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.732+0000 7fbb52400640 1 --2- 172.21.15.6:0/132805945 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbb4c06da40 0x7fbb4c09c740 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:26.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.732+0000 7fbb52400640 1 -- 172.21.15.6:0/132805945 >> 172.21.15.6:0/132805945 conn(0x7fbb4c06b3d0 msgr2=0x7fbb4c073f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:29:26.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.732+0000 7fbb52400640 1 -- 172.21.15.6:0/132805945 shutdown_connections 2024-03-04T09:29:26.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.733+0000 7fbb52400640 1 -- 172.21.15.6:0/132805945 wait complete. 2024-03-04T09:29:26.750 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-04T09:29:26.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.982+0000 7f712e600640 1 Processor -- start 2024-03-04T09:29:26.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.982+0000 7f712e600640 1 -- start start 2024-03-04T09:29:26.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.984+0000 7f712e600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f71280c4f10 0x7f71280b4340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:26.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.984+0000 7f712e600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f71280c52f0 0x7f7128166c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:26.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.984+0000 7f712e600640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71280b17f0 con 0x7f71280b2890 2024-03-04T09:29:26.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.984+0000 7f712e600640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71280b3570 con 0x7f71280c4f10 2024-03-04T09:29:26.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.984+0000 7f712e600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71280b0900 con 0x7f71280c52f0 2024-03-04T09:29:26.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.984+0000 7f7126a00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f71280b2890 0x7f71280aea90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:56638/0 (socket says 172.21.15.6:56638) 2024-03-04T09:29:26.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.984+0000 7f7126a00640 1 -- 172.21.15.6:0/2796280712 learned_addr learned my addr 172.21.15.6:0/2796280712 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:29:26.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.984+0000 7f7127400640 1 --2- 172.21.15.6:0/2796280712 >> v2:172.21.15.6:3300/0 conn(0x7f71280c4f10 0x7f71280b4340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:26.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.984+0000 7f7127e00640 1 --2- 172.21.15.6:0/2796280712 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f71280c52f0 0x7f7128166c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:26.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.985+0000 7f7126000640 1 -- 172.21.15.6:0/2796280712 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3703067819 0 0) 0x7f71280b17f0 con 0x7f71280b2890 2024-03-04T09:29:26.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.985+0000 7f7126000640 1 -- 172.21.15.6:0/2796280712 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f710c003520 con 0x7f71280b2890 2024-03-04T09:29:26.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.985+0000 7f7127400640 1 -- 172.21.15.6:0/2796280712 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f71280c52f0 msgr2=0x7f7128166c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:26.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.985+0000 7f7127400640 1 --2- 172.21.15.6:0/2796280712 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f71280c52f0 0x7f7128166c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:26.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.985+0000 7f7127400640 1 -- 172.21.15.6:0/2796280712 >> v1:172.21.15.6:6789/0 conn(0x7f71280b2890 legacy=0x7f71280aea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:26.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.985+0000 7f7127400640 1 -- 172.21.15.6:0/2796280712 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71280b1ad0 con 0x7f71280c4f10 2024-03-04T09:29:26.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.986+0000 7f7127400640 1 --2- 172.21.15.6:0/2796280712 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f71280c4f10 0x7f71280b4340 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f7118009d90 tx=0x7f711802bd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7f9d0f5e4280932 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:26.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.986+0000 7f7126000640 1 -- 172.21.15.6:0/2796280712 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7118039070 con 0x7f71280c4f10 2024-03-04T09:29:26.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.986+0000 7f7126000640 1 -- 172.21.15.6:0/2796280712 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f711802cb30 con 0x7f71280c4f10 2024-03-04T09:29:26.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.986+0000 7f7126000640 1 -- 172.21.15.6:0/2796280712 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7118034440 con 0x7f71280c4f10 2024-03-04T09:29:26.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.987+0000 7f712e600640 1 -- 172.21.15.6:0/2796280712 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f71280c4f10 msgr2=0x7f71280b4340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:26.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.987+0000 7f712e600640 1 --2- 172.21.15.6:0/2796280712 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f71280c4f10 0x7f71280b4340 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f7118009d90 tx=0x7f711802bd80 comp rx=0 tx=0).stop 2024-03-04T09:29:26.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.987+0000 7f712e600640 1 -- 172.21.15.6:0/2796280712 shutdown_connections 2024-03-04T09:29:26.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.987+0000 7f712e600640 1 --2- 172.21.15.6:0/2796280712 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f71280c52f0 0x7f7128166c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:26.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.987+0000 7f712e600640 1 --2- 172.21.15.6:0/2796280712 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f71280c4f10 0x7f71280b4340 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:26.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.987+0000 7f712e600640 1 -- 172.21.15.6:0/2796280712 >> 172.21.15.6:0/2796280712 conn(0x7f71280b3820 msgr2=0x7f71280b3c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:29:26.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.987+0000 7f712e600640 1 -- 172.21.15.6:0/2796280712 shutdown_connections 2024-03-04T09:29:26.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.987+0000 7f712e600640 1 -- 172.21.15.6:0/2796280712 wait complete. 2024-03-04T09:29:26.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.989+0000 7f712e600640 1 Processor -- start 2024-03-04T09:29:26.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.989+0000 7f712e600640 1 -- start start 2024-03-04T09:29:26.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.990+0000 7f712e600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f71280b2890 0x7f712809e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:26.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.990+0000 7f712e600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f71280c4f10 0x7f712809e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:26.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.990+0000 7f712e600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f71280c52f0 0x7f712809ca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:26.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.990+0000 7f712e600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71280b2370 con 0x7f71280c4f10 2024-03-04T09:29:26.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.990+0000 7f712e600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71280b2090 con 0x7f71280c52f0 2024-03-04T09:29:26.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.990+0000 7f712e600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71280b2cd0 con 0x7f71280b2890 2024-03-04T09:29:26.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.990+0000 7f7126a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f71280c4f10 0x7f712809e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:26.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.990+0000 7f7126a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f71280c4f10 0x7f712809e900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:46270/0 (socket says 172.21.15.6:46270) 2024-03-04T09:29:26.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.990+0000 7f7126a00640 1 -- 172.21.15.6:0/2157012223 learned_addr learned my addr 172.21.15.6:0/2157012223 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:29:26.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.990+0000 7f7127400640 1 --2- 172.21.15.6:0/2157012223 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f71280b2890 0x7f712809e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:26.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.990+0000 7f7126a00640 1 -- 172.21.15.6:0/2157012223 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f71280b2890 msgr2=0x7f712809e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:26.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.990+0000 7f7126a00640 1 --2- 172.21.15.6:0/2157012223 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f71280b2890 0x7f712809e3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:26.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.990+0000 7f7127e00640 1 --2- 172.21.15.6:0/2157012223 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f71280c52f0 0x7f712809ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:26.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.990+0000 7f7126a00640 1 -- 172.21.15.6:0/2157012223 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f71280c52f0 msgr2=0x7f712809ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:26.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.991+0000 7f7126a00640 1 --2- 172.21.15.6:0/2157012223 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f71280c52f0 0x7f712809ca00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:26.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.991+0000 7f7126a00640 1 -- 172.21.15.6:0/2157012223 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71280c5e20 con 0x7f71280c4f10 2024-03-04T09:29:26.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.991+0000 7f7126a00640 1 --2- 172.21.15.6:0/2157012223 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f71280c4f10 0x7f712809e900 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f711000c430 tx=0x7f711000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:26.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.991+0000 7f711f400640 1 -- 172.21.15.6:0/2157012223 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7110017070 con 0x7f71280c4f10 2024-03-04T09:29:26.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.991+0000 7f711f400640 1 -- 172.21.15.6:0/2157012223 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7110005ce0 con 0x7f71280c4f10 2024-03-04T09:29:26.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.991+0000 7f711f400640 1 -- 172.21.15.6:0/2157012223 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71100072f0 con 0x7f71280c4f10 2024-03-04T09:29:26.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.991+0000 7f712e600640 1 -- 172.21.15.6:0/2157012223 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71280c6100 con 0x7f71280c4f10 2024-03-04T09:29:26.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.991+0000 7f712e600640 1 -- 172.21.15.6:0/2157012223 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71280c66c0 con 0x7f71280c4f10 2024-03-04T09:29:26.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.992+0000 7f712e600640 1 -- 172.21.15.6:0/2157012223 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f712804f330 con 0x7f71280c4f10 2024-03-04T09:29:26.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.993+0000 7f711f400640 1 -- 172.21.15.6:0/2157012223 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f711001a070 con 0x7f71280c4f10 2024-03-04T09:29:26.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.994+0000 7f711f400640 1 --2- 172.21.15.6:0/2157012223 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f710406b4d0 0x7f710406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:26.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.994+0000 7f711f400640 1 -- 172.21.15.6:0/2157012223 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f7110089760 con 0x7f71280c4f10 2024-03-04T09:29:26.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.995+0000 7f7127400640 1 --2- 172.21.15.6:0/2157012223 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f710406b4d0 0x7f710406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:26.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.996+0000 7f7127400640 1 --2- 172.21.15.6:0/2157012223 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f710406b4d0 0x7f710406d990 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f7118036000 tx=0x7f7118030000 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:26.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:26.999+0000 7f711f400640 1 -- 172.21.15.6:0/2157012223 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7110058560 con 0x7f71280c4f10 2024-03-04T09:29:27.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.196+0000 7f712e600640 1 -- 172.21.15.6:0/2157012223 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f71280b7c50 con 0x7f71280c4f10 2024-03-04T09:29:27.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.197+0000 7f711f400640 1 -- 172.21.15.6:0/2157012223 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v70) v1 ==== 104+0+11 (secure 0 0 0) 0x7f7110057e60 con 0x7f71280c4f10 2024-03-04T09:29:27.194 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-03-04T09:29:27.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.199+0000 7f712e600640 1 -- 172.21.15.6:0/2157012223 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f710406b4d0 msgr2=0x7f710406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:27.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.199+0000 7f712e600640 1 --2- 172.21.15.6:0/2157012223 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f710406b4d0 0x7f710406d990 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f7118036000 tx=0x7f7118030000 comp rx=0 tx=0).stop 2024-03-04T09:29:27.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.199+0000 7f712e600640 1 -- 172.21.15.6:0/2157012223 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f71280c4f10 msgr2=0x7f712809e900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:27.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.199+0000 7f712e600640 1 --2- 172.21.15.6:0/2157012223 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f71280c4f10 0x7f712809e900 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f711000c430 tx=0x7f711000c8f0 comp rx=0 tx=0).stop 2024-03-04T09:29:27.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.200+0000 7f712e600640 1 -- 172.21.15.6:0/2157012223 shutdown_connections 2024-03-04T09:29:27.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.200+0000 7f712e600640 1 --2- 172.21.15.6:0/2157012223 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f71280c52f0 0x7f712809ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:27.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.200+0000 7f712e600640 1 --2- 172.21.15.6:0/2157012223 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f71280c4f10 0x7f712809e900 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:27.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.200+0000 7f712e600640 1 --2- 172.21.15.6:0/2157012223 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f710406b4d0 0x7f710406d990 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:27.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.200+0000 7f712e600640 1 --2- 172.21.15.6:0/2157012223 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f71280b2890 0x7f712809e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:27.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.200+0000 7f712e600640 1 -- 172.21.15.6:0/2157012223 >> 172.21.15.6:0/2157012223 conn(0x7f71280b3820 msgr2=0x7f71281ec250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:29:27.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.200+0000 7f712e600640 1 -- 172.21.15.6:0/2157012223 shutdown_connections 2024-03-04T09:29:27.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.200+0000 7f712e600640 1 -- 172.21.15.6:0/2157012223 wait complete. 2024-03-04T09:29:27.214 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-04T09:29:27.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.437+0000 7f0b12600640 1 Processor -- start 2024-03-04T09:29:27.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.438+0000 7f0b12600640 1 -- start start 2024-03-04T09:29:27.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.439+0000 7f0b12600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0b0c06d8d0 0x7f0b0c0b1b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:27.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.439+0000 7f0b12600640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f0b0c0b2080 0x7f0b0c0b3640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:27.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.439+0000 7f0b12600640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b0c06de40 con 0x7f0b0c147590 2024-03-04T09:29:27.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.439+0000 7f0b12600640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b0c147970 con 0x7f0b0c0b2080 2024-03-04T09:29:27.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.439+0000 7f0b12600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b0c0b31b0 con 0x7f0b0c06d8d0 2024-03-04T09:29:27.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.440+0000 7f0b11200640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f0b0c147590 0x7f0b0c071480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:56654/0 (socket says 172.21.15.6:56654) 2024-03-04T09:29:27.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.440+0000 7f0b11200640 1 -- 172.21.15.6:0/1091537445 learned_addr learned my addr 172.21.15.6:0/1091537445 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:29:27.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.440+0000 7f0b11c00640 1 --2- 172.21.15.6:0/1091537445 >> v2:172.21.15.146:3300/0 conn(0x7f0b0c0b2080 0x7f0b0c0b3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:27.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.440+0000 7f0b0b400640 1 -- 172.21.15.6:0/1091537445 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 987639013 0 0) 0x7f0b0c06de40 con 0x7f0b0c147590 2024-03-04T09:29:27.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.440+0000 7f0b0be00640 1 --2- 172.21.15.6:0/1091537445 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0b0c06d8d0 0x7f0b0c0b1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:27.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.440+0000 7f0b0b400640 1 -- 172.21.15.6:0/1091537445 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0af4003520 con 0x7f0b0c147590 2024-03-04T09:29:27.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.440+0000 7f0b0b400640 1 -- 172.21.15.6:0/1091537445 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 584465446 0 0) 0x7f0af4003520 con 0x7f0b0c147590 2024-03-04T09:29:27.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.440+0000 7f0b0b400640 1 -- 172.21.15.6:0/1091537445 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0b0c06d8d0 msgr2=0x7f0b0c0b1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:27.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.440+0000 7f0b0b400640 1 --2- 172.21.15.6:0/1091537445 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0b0c06d8d0 0x7f0b0c0b1b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:27.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.440+0000 7f0b0b400640 1 -- 172.21.15.6:0/1091537445 >> v2:172.21.15.146:3300/0 conn(0x7f0b0c0b2080 msgr2=0x7f0b0c0b3640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:27.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.440+0000 7f0b0b400640 1 --2- 172.21.15.6:0/1091537445 >> v2:172.21.15.146:3300/0 conn(0x7f0b0c0b2080 0x7f0b0c0b3640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:27.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.440+0000 7f0b0b400640 1 -- 172.21.15.6:0/1091537445 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b0c06e120 con 0x7f0b0c147590 2024-03-04T09:29:27.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.441+0000 7f0b0b400640 1 -- 172.21.15.6:0/1091537445 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7f0b00001670 con 0x7f0b0c147590 2024-03-04T09:29:27.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.441+0000 7f0b0b400640 1 -- 172.21.15.6:0/1091537445 >> v1:172.21.15.6:6789/0 conn(0x7f0b0c147590 legacy=0x7f0b0c071480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:27.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.441+0000 7f0b0b400640 1 --2- 172.21.15.6:0/1091537445 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0b0c06d8d0 0x7f0af40039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:27.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.441+0000 7f0b0b400640 1 --2- 172.21.15.6:0/1091537445 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0af4003f80 0x7f0af4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:27.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.441+0000 7f0b0b400640 1 --2- 172.21.15.6:0/1091537445 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0af4004d40 0x7f0af40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:27.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.441+0000 7f0b0b400640 1 -- 172.21.15.6:0/1091537445 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b0c147970 con 0x7f0af4003f80 2024-03-04T09:29:27.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.441+0000 7f0b0b400640 1 -- 172.21.15.6:0/1091537445 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b0c0b31b0 con 0x7f0af4004d40 2024-03-04T09:29:27.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.441+0000 7f0b0b400640 1 -- 172.21.15.6:0/1091537445 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0af400ba70 con 0x7f0b0c06d8d0 2024-03-04T09:29:27.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.441+0000 7f0b11c00640 1 --2- 172.21.15.6:0/1091537445 >> v2:172.21.15.146:3300/0 conn(0x7f0b0c0b2080 0x7f0b0c0b3640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T09:29:27.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.441+0000 7f0b0be00640 1 --2- 172.21.15.6:0/1091537445 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0af4003f80 0x7f0af4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:27.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.441+0000 7f0b11200640 1 --2- 172.21.15.6:0/1091537445 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0b0c06d8d0 0x7f0af40039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:27.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.441+0000 7f0b0be00640 1 -- 172.21.15.6:0/1091537445 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0b0c06d8d0 msgr2=0x7f0af40039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:27.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.441+0000 7f0b0be00640 1 --2- 172.21.15.6:0/1091537445 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0b0c06d8d0 0x7f0af40039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:27.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.441+0000 7f0b0be00640 1 -- 172.21.15.6:0/1091537445 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0af4004d40 msgr2=0x7f0af40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:27.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.441+0000 7f0b0be00640 1 --2- 172.21.15.6:0/1091537445 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0af4004d40 0x7f0af40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:27.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.441+0000 7f0b0be00640 1 -- 172.21.15.6:0/1091537445 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0af4003520 con 0x7f0af4003f80 2024-03-04T09:29:27.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.442+0000 7f0b11200640 1 --2- 172.21.15.6:0/1091537445 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0b0c06d8d0 0x7f0af40039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:29:27.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.443+0000 7f0b0be00640 1 --2- 172.21.15.6:0/1091537445 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0af4003f80 0x7f0af4004800 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f0af8009ac0 tx=0x7f0af4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=917adf0fedac0b65 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:27.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.443+0000 7f0b0b400640 1 -- 172.21.15.6:0/1091537445 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0af8039070 con 0x7f0af4003f80 2024-03-04T09:29:27.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.443+0000 7f0b0b400640 1 -- 172.21.15.6:0/1091537445 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0af802c6b0 con 0x7f0af4003f80 2024-03-04T09:29:27.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.443+0000 7f0b0b400640 1 -- 172.21.15.6:0/1091537445 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0af8009690 con 0x7f0af4003f80 2024-03-04T09:29:27.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.444+0000 7f0b12600640 1 -- 172.21.15.6:0/1091537445 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0af4003f80 msgr2=0x7f0af4004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:27.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.444+0000 7f0b12600640 1 --2- 172.21.15.6:0/1091537445 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0af4003f80 0x7f0af4004800 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f0af8009ac0 tx=0x7f0af4005830 comp rx=0 tx=0).stop 2024-03-04T09:29:27.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.444+0000 7f0b11c00640 1 -- 172.21.15.6:0/1091537445 reap_dead start 2024-03-04T09:29:27.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.444+0000 7f0b12600640 1 -- 172.21.15.6:0/1091537445 shutdown_connections 2024-03-04T09:29:27.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.444+0000 7f0b12600640 1 -- 172.21.15.6:0/1091537445 >> 172.21.15.6:0/1091537445 conn(0x7f0b0c06b3a0 msgr2=0x7f0b0c06b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:29:27.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.444+0000 7f0b12600640 1 -- 172.21.15.6:0/1091537445 shutdown_connections 2024-03-04T09:29:27.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.444+0000 7f0b12600640 1 -- 172.21.15.6:0/1091537445 wait complete. 2024-03-04T09:29:27.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.446+0000 7f0b12600640 1 Processor -- start 2024-03-04T09:29:27.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.446+0000 7f0b12600640 1 -- start start 2024-03-04T09:29:27.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.447+0000 7f0b12600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0af4003f80 0x7f0b0c142da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:27.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.447+0000 7f0b12600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0b0c12ff90 0x7f0b0c1432e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:27.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.447+0000 7f0b12600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0b0c160830 0x7f0b0c160c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:27.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.447+0000 7f0b12600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b0c0b1b40 con 0x7f0b0c160830 2024-03-04T09:29:27.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.447+0000 7f0b12600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b0c0b2b50 con 0x7f0af4003f80 2024-03-04T09:29:27.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.447+0000 7f0b12600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b0c0b1cc0 con 0x7f0b0c12ff90 2024-03-04T09:29:27.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.447+0000 7f0b11c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0b0c160830 0x7f0b0c160c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:27.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.447+0000 7f0b11c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0b0c160830 0x7f0b0c160c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:46290/0 (socket says 172.21.15.6:46290) 2024-03-04T09:29:27.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.447+0000 7f0b11c00640 1 -- 172.21.15.6:0/2756373494 learned_addr learned my addr 172.21.15.6:0/2756373494 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:29:27.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.447+0000 7f0b11200640 1 --2- 172.21.15.6:0/2756373494 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0af4003f80 0x7f0b0c142da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:27.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.447+0000 7f0b0be00640 1 --2- 172.21.15.6:0/2756373494 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0b0c12ff90 0x7f0b0c1432e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:27.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.448+0000 7f0b11c00640 1 -- 172.21.15.6:0/2756373494 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0b0c12ff90 msgr2=0x7f0b0c1432e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:27.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.448+0000 7f0b11c00640 1 --2- 172.21.15.6:0/2756373494 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0b0c12ff90 0x7f0b0c1432e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:27.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.448+0000 7f0b11c00640 1 -- 172.21.15.6:0/2756373494 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0af4003f80 msgr2=0x7f0b0c142da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:27.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.448+0000 7f0b11c00640 1 --2- 172.21.15.6:0/2756373494 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0af4003f80 0x7f0b0c142da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:27.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.448+0000 7f0b11c00640 1 -- 172.21.15.6:0/2756373494 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b0c1611d0 con 0x7f0b0c160830 2024-03-04T09:29:27.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.448+0000 7f0b0be00640 1 --2- 172.21.15.6:0/2756373494 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0b0c12ff90 0x7f0b0c1432e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:29:27.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.448+0000 7f0b11c00640 1 --2- 172.21.15.6:0/2756373494 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0b0c160830 0x7f0b0c160c90 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f0b0c0af220 tx=0x7f0b0400d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:27.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.448+0000 7f0b08c00640 1 -- 172.21.15.6:0/2756373494 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b0401a070 con 0x7f0b0c160830 2024-03-04T09:29:27.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.448+0000 7f0b08c00640 1 -- 172.21.15.6:0/2756373494 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b0400dcd0 con 0x7f0b0c160830 2024-03-04T09:29:27.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.448+0000 7f0b08c00640 1 -- 172.21.15.6:0/2756373494 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b04003bd0 con 0x7f0b0c160830 2024-03-04T09:29:27.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.448+0000 7f0b12600640 1 -- 172.21.15.6:0/2756373494 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b0c140400 con 0x7f0b0c160830 2024-03-04T09:29:27.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.448+0000 7f0b12600640 1 -- 172.21.15.6:0/2756373494 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b0c1e6930 con 0x7f0b0c160830 2024-03-04T09:29:27.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.450+0000 7f0b12600640 1 -- 172.21.15.6:0/2756373494 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b0c04f330 con 0x7f0b0c160830 2024-03-04T09:29:27.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.451+0000 7f0b08c00640 1 -- 172.21.15.6:0/2756373494 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0b0401d070 con 0x7f0b0c160830 2024-03-04T09:29:27.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.452+0000 7f0b08c00640 1 --2- 172.21.15.6:0/2756373494 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f0aec06b4d0 0x7f0aec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:27.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.452+0000 7f0b08c00640 1 -- 172.21.15.6:0/2756373494 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f0b0408a070 con 0x7f0b0c160830 2024-03-04T09:29:27.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.452+0000 7f0b11200640 1 --2- 172.21.15.6:0/2756373494 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f0aec06b4d0 0x7f0aec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:27.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.453+0000 7f0b11200640 1 --2- 172.21.15.6:0/2756373494 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f0aec06b4d0 0x7f0aec06d990 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f0b00001eb0 tx=0x7f0b00009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:27.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.456+0000 7f0b08c00640 1 -- 172.21.15.6:0/2756373494 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0b04010070 con 0x7f0b0c160830 2024-03-04T09:29:27.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.661+0000 7f0b12600640 1 -- 172.21.15.6:0/2756373494 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f0b0c1e7320 con 0x7f0b0c160830 2024-03-04T09:29:27.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.661+0000 7f0b08c00640 1 -- 172.21.15.6:0/2756373494 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v70) v1 ==== 100+0+11 (secure 0 0 0) 0x7f0b04058f00 con 0x7f0b0c160830 2024-03-04T09:29:27.658 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 64 2024-03-04T09:29:27.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.664+0000 7f0b12600640 1 -- 172.21.15.6:0/2756373494 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f0aec06b4d0 msgr2=0x7f0aec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:27.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.664+0000 7f0b12600640 1 --2- 172.21.15.6:0/2756373494 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f0aec06b4d0 0x7f0aec06d990 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f0b00001eb0 tx=0x7f0b00009040 comp rx=0 tx=0).stop 2024-03-04T09:29:27.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.664+0000 7f0b12600640 1 -- 172.21.15.6:0/2756373494 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0b0c160830 msgr2=0x7f0b0c160c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:27.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.664+0000 7f0b12600640 1 --2- 172.21.15.6:0/2756373494 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0b0c160830 0x7f0b0c160c90 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f0b0c0af220 tx=0x7f0b0400d4a0 comp rx=0 tx=0).stop 2024-03-04T09:29:27.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.664+0000 7f0b12600640 1 -- 172.21.15.6:0/2756373494 shutdown_connections 2024-03-04T09:29:27.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.664+0000 7f0b12600640 1 --2- 172.21.15.6:0/2756373494 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0b0c160830 0x7f0b0c160c90 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:27.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.664+0000 7f0b12600640 1 --2- 172.21.15.6:0/2756373494 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f0aec06b4d0 0x7f0aec06d990 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:27.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.664+0000 7f0b12600640 1 --2- 172.21.15.6:0/2756373494 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0b0c12ff90 0x7f0b0c1432e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:27.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.664+0000 7f0b12600640 1 --2- 172.21.15.6:0/2756373494 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0af4003f80 0x7f0b0c142da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:27.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.664+0000 7f0b12600640 1 -- 172.21.15.6:0/2756373494 >> 172.21.15.6:0/2756373494 conn(0x7f0b0c06b3a0 msgr2=0x7f0b0c0b3dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:29:27.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.664+0000 7f0b12600640 1 -- 172.21.15.6:0/2756373494 shutdown_connections 2024-03-04T09:29:27.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.664+0000 7f0b12600640 1 -- 172.21.15.6:0/2756373494 wait complete. 2024-03-04T09:29:27.681 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-04T09:29:27.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.906+0000 7f59a9000640 1 Processor -- start 2024-03-04T09:29:27.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.906+0000 7f59a9000640 1 -- start start 2024-03-04T09:29:27.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.908+0000 7f59a9000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f59a40b3c90 0x7f59a40cdc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:27.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.908+0000 7f59a9000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59a40b4860 con 0x7f59a4071050 2024-03-04T09:29:27.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.908+0000 7f59a9000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59a40b4b40 con 0x7f59a40cf5c0 2024-03-04T09:29:27.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.908+0000 7f59a9000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59a40ce160 con 0x7f59a40b3c90 2024-03-04T09:29:27.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.908+0000 7f59a2800640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f59a4071050 0x7f59a4071490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:56662/0 (socket says 172.21.15.6:56662) 2024-03-04T09:29:27.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.908+0000 7f59a2800640 1 -- 172.21.15.6:0/3862934759 learned_addr learned my addr 172.21.15.6:0/3862934759 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:29:27.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.909+0000 7f59a1e00640 1 --2- 172.21.15.6:0/3862934759 >> v2:172.21.15.146:3300/0 conn(0x7f59a40b3c90 0x7f59a40cdc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:27.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.909+0000 7f59a1400640 1 -- 172.21.15.6:0/3862934759 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2910368223 0 0) 0x7f59a40b4860 con 0x7f59a4071050 2024-03-04T09:29:27.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.909+0000 7f59a1400640 1 -- 172.21.15.6:0/3862934759 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5988003520 con 0x7f59a4071050 2024-03-04T09:29:27.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.909+0000 7f59a1400640 1 -- 172.21.15.6:0/3862934759 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3805588825 0 0) 0x7f5988003520 con 0x7f59a4071050 2024-03-04T09:29:27.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.909+0000 7f59a1400640 1 -- 172.21.15.6:0/3862934759 >> v2:172.21.15.146:3300/0 conn(0x7f59a40b3c90 msgr2=0x7f59a40cdc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:27.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.909+0000 7f59a1400640 1 --2- 172.21.15.6:0/3862934759 >> v2:172.21.15.146:3300/0 conn(0x7f59a40b3c90 0x7f59a40cdc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:27.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.909+0000 7f59a1400640 1 -- 172.21.15.6:0/3862934759 >> v1:172.21.15.146:6789/0 conn(0x7f59a40cf5c0 legacy=0x7f59a40b4070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:27.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.909+0000 7f59a1400640 1 -- 172.21.15.6:0/3862934759 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59a40b4e20 con 0x7f59a4071050 2024-03-04T09:29:27.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.909+0000 7f59a1400640 1 -- 172.21.15.6:0/3862934759 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7f599c0019f0 con 0x7f59a4071050 2024-03-04T09:29:27.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.909+0000 7f59a1400640 1 -- 172.21.15.6:0/3862934759 >> v1:172.21.15.6:6789/0 conn(0x7f59a4071050 legacy=0x7f59a4071490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:27.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.909+0000 7f59a1400640 1 --2- 172.21.15.6:0/3862934759 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5988003a20 0x7f5988003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:27.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.909+0000 7f59a1400640 1 --2- 172.21.15.6:0/3862934759 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5988004800 0x7f5988004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:27.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.909+0000 7f59a1400640 1 --2- 172.21.15.6:0/3862934759 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f59880051c0 0x7f598800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:27.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.909+0000 7f59a1400640 1 -- 172.21.15.6:0/3862934759 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59a40ce160 con 0x7f5988004800 2024-03-04T09:29:27.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.909+0000 7f59a1400640 1 -- 172.21.15.6:0/3862934759 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f598800c0d0 con 0x7f59880051c0 2024-03-04T09:29:27.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.909+0000 7f59a1400640 1 -- 172.21.15.6:0/3862934759 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f598800c2d0 con 0x7f5988003a20 2024-03-04T09:29:27.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.910+0000 7f5999400640 1 --2- 172.21.15.6:0/3862934759 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5988004800 0x7f5988004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:27.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.910+0000 7f59a2800640 1 --2- 172.21.15.6:0/3862934759 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f59880051c0 0x7f598800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:27.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.910+0000 7f59a1e00640 1 --2- 172.21.15.6:0/3862934759 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5988003a20 0x7f5988003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:27.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.910+0000 7f5999400640 1 -- 172.21.15.6:0/3862934759 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5988003a20 msgr2=0x7f5988003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:27.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.910+0000 7f5999400640 1 --2- 172.21.15.6:0/3862934759 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5988003a20 0x7f5988003e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:27.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.910+0000 7f5999400640 1 -- 172.21.15.6:0/3862934759 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f59880051c0 msgr2=0x7f598800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:27.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.910+0000 7f5999400640 1 --2- 172.21.15.6:0/3862934759 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f59880051c0 0x7f598800b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:27.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.910+0000 7f59a2800640 1 -- 172.21.15.6:0/3862934759 reap_dead start 2024-03-04T09:29:27.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.910+0000 7f5999400640 1 -- 172.21.15.6:0/3862934759 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5988003520 con 0x7f5988004800 2024-03-04T09:29:27.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.911+0000 7f5999400640 1 --2- 172.21.15.6:0/3862934759 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5988004800 0x7f5988004c80 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f598c001750 tx=0x7f5988005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa4f5a5ad62f298b server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:27.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.911+0000 7f59a1400640 1 -- 172.21.15.6:0/3862934759 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f598c001370 con 0x7f5988004800 2024-03-04T09:29:27.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.912+0000 7f59a1400640 1 -- 172.21.15.6:0/3862934759 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f598c0052d0 con 0x7f5988004800 2024-03-04T09:29:27.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.912+0000 7f59a1400640 1 -- 172.21.15.6:0/3862934759 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f598c003830 con 0x7f5988004800 2024-03-04T09:29:27.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.912+0000 7f59a9000640 1 -- 172.21.15.6:0/3862934759 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5988004800 msgr2=0x7f5988004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:27.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.912+0000 7f59a9000640 1 --2- 172.21.15.6:0/3862934759 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5988004800 0x7f5988004c80 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f598c001750 tx=0x7f5988005830 comp rx=0 tx=0).stop 2024-03-04T09:29:27.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.912+0000 7f59a9000640 1 -- 172.21.15.6:0/3862934759 shutdown_connections 2024-03-04T09:29:27.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.912+0000 7f59a9000640 1 --2- 172.21.15.6:0/3862934759 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5988004800 0x7f5988004c80 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:27.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.912+0000 7f59a9000640 1 -- 172.21.15.6:0/3862934759 >> 172.21.15.6:0/3862934759 conn(0x7f59a406afb0 msgr2=0x7f59a406b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:29:27.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.912+0000 7f59a9000640 1 -- 172.21.15.6:0/3862934759 shutdown_connections 2024-03-04T09:29:27.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.912+0000 7f59a9000640 1 -- 172.21.15.6:0/3862934759 wait complete. 2024-03-04T09:29:27.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.914+0000 7f59a9000640 1 Processor -- start 2024-03-04T09:29:27.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.914+0000 7f59a9000640 1 -- start start 2024-03-04T09:29:27.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.915+0000 7f59a9000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5988004800 0x7f59a4158570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:27.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.915+0000 7f59a9000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f59a4151b90 0x7f59a4144480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:27.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.915+0000 7f59a9000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f59a41556b0 0x7f59a414a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:27.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.915+0000 7f59a9000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59a40cdc20 con 0x7f59a41556b0 2024-03-04T09:29:27.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.915+0000 7f59a9000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59a415b340 con 0x7f59a4151b90 2024-03-04T09:29:27.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.915+0000 7f59a9000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59a40cdda0 con 0x7f5988004800 2024-03-04T09:29:27.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.915+0000 7f59a2800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f59a41556b0 0x7f59a414a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:27.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.915+0000 7f59a2800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f59a41556b0 0x7f59a414a820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:46302/0 (socket says 172.21.15.6:46302) 2024-03-04T09:29:27.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.915+0000 7f59a2800640 1 -- 172.21.15.6:0/2137678415 learned_addr learned my addr 172.21.15.6:0/2137678415 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:29:27.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.916+0000 7f59a1e00640 1 --2- 172.21.15.6:0/2137678415 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5988004800 0x7f59a4158570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:27.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.916+0000 7f5999400640 1 --2- 172.21.15.6:0/2137678415 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f59a4151b90 0x7f59a4144480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:27.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.916+0000 7f59a2800640 1 -- 172.21.15.6:0/2137678415 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5988004800 msgr2=0x7f59a4158570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:27.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.916+0000 7f59a2800640 1 --2- 172.21.15.6:0/2137678415 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5988004800 0x7f59a4158570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:27.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.916+0000 7f59a2800640 1 -- 172.21.15.6:0/2137678415 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f59a4151b90 msgr2=0x7f59a4144480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:27.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.916+0000 7f59a2800640 1 --2- 172.21.15.6:0/2137678415 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f59a4151b90 0x7f59a4144480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:27.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.916+0000 7f59a2800640 1 -- 172.21.15.6:0/2137678415 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59a41449c0 con 0x7f59a41556b0 2024-03-04T09:29:27.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.916+0000 7f59a2800640 1 --2- 172.21.15.6:0/2137678415 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f59a41556b0 0x7f59a414a820 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f599c007c40 tx=0x7f599c00b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:27.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.916+0000 7f599aa00640 1 -- 172.21.15.6:0/2137678415 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f599c004d60 con 0x7f59a41556b0 2024-03-04T09:29:27.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.917+0000 7f599aa00640 1 -- 172.21.15.6:0/2137678415 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f599c00bb20 con 0x7f59a41556b0 2024-03-04T09:29:27.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.917+0000 7f59a9000640 1 -- 172.21.15.6:0/2137678415 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59a41577b0 con 0x7f59a41556b0 2024-03-04T09:29:27.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.917+0000 7f599aa00640 1 -- 172.21.15.6:0/2137678415 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f599c007110 con 0x7f59a41556b0 2024-03-04T09:29:27.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.917+0000 7f59a9000640 1 -- 172.21.15.6:0/2137678415 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59a4157d70 con 0x7f59a41556b0 2024-03-04T09:29:27.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.918+0000 7f59a9000640 1 -- 172.21.15.6:0/2137678415 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59a404f3a0 con 0x7f59a41556b0 2024-03-04T09:29:27.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.919+0000 7f599aa00640 1 -- 172.21.15.6:0/2137678415 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f599c0072b0 con 0x7f59a41556b0 2024-03-04T09:29:27.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.920+0000 7f599aa00640 1 --2- 172.21.15.6:0/2137678415 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f598006b440 0x7f598006d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:27.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.920+0000 7f599aa00640 1 -- 172.21.15.6:0/2137678415 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f599c08a1c0 con 0x7f59a41556b0 2024-03-04T09:29:27.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.921+0000 7f59a1e00640 1 --2- 172.21.15.6:0/2137678415 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f598006b440 0x7f598006d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:27.919 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.922+0000 7f59a1e00640 1 --2- 172.21.15.6:0/2137678415 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f598006b440 0x7f598006d900 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f59a4050c30 tx=0x7f5994008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:27.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:27.925+0000 7f599aa00640 1 -- 172.21.15.6:0/2137678415 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f599c058fb0 con 0x7f59a41556b0 2024-03-04T09:29:28.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:28.137+0000 7f59a9000640 1 -- 172.21.15.6:0/2137678415 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f59a40cc9d0 con 0x7f59a41556b0 2024-03-04T09:29:28.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:28.138+0000 7f599aa00640 1 -- 172.21.15.6:0/2137678415 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v70) v1 ==== 80+0+0 (secure 0 0 0) 0x7f599c0588b0 con 0x7f59a41556b0 2024-03-04T09:29:28.135 INFO:teuthology.orchestra.run.smithi006.stderr:listed 0 entries 2024-03-04T09:29:28.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:28.141+0000 7f59a9000640 1 -- 172.21.15.6:0/2137678415 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f598006b440 msgr2=0x7f598006d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:28.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:28.141+0000 7f59a9000640 1 --2- 172.21.15.6:0/2137678415 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f598006b440 0x7f598006d900 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f59a4050c30 tx=0x7f5994008040 comp rx=0 tx=0).stop 2024-03-04T09:29:28.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:28.141+0000 7f59a9000640 1 -- 172.21.15.6:0/2137678415 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f59a41556b0 msgr2=0x7f59a414a820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:28.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:28.141+0000 7f59a9000640 1 --2- 172.21.15.6:0/2137678415 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f59a41556b0 0x7f59a414a820 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f599c007c40 tx=0x7f599c00b4a0 comp rx=0 tx=0).stop 2024-03-04T09:29:28.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:28.141+0000 7f59a9000640 1 -- 172.21.15.6:0/2137678415 shutdown_connections 2024-03-04T09:29:28.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:28.141+0000 7f59a9000640 1 --2- 172.21.15.6:0/2137678415 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f59a41556b0 0x7f59a414a820 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:28.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:28.141+0000 7f59a9000640 1 --2- 172.21.15.6:0/2137678415 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f59a4151b90 0x7f59a4144480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:28.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:28.141+0000 7f59a9000640 1 --2- 172.21.15.6:0/2137678415 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f598006b440 0x7f598006d900 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:28.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:28.141+0000 7f59a9000640 1 --2- 172.21.15.6:0/2137678415 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5988004800 0x7f59a4158570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:28.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:28.141+0000 7f59a9000640 1 -- 172.21.15.6:0/2137678415 >> 172.21.15.6:0/2137678415 conn(0x7f59a406afb0 msgr2=0x7f59a406e180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:29:28.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:28.141+0000 7f59a9000640 1 -- 172.21.15.6:0/2137678415 shutdown_connections 2024-03-04T09:29:28.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:28.141+0000 7f59a9000640 1 -- 172.21.15.6:0/2137678415 wait complete. 2024-03-04T09:29:28.155 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-04T09:29:28.156 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:29:28.156 DEBUG:teuthology.orchestra.run.smithi006:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-04T09:29:28.306 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi006.front.sepia.ceph.com 2024-03-04T09:29:28.307 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:29:28.307 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-04T09:29:28.353 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-04T09:29:28.353 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T09:29:28.391 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053429,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249180k,nr_inodes=812295,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-04T09:29:28.391 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-04T09:29:28.391 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T09:29:28.433 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054202,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249800k,nr_inodes=812450,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-04T09:29:28.433 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-04T09:29:28.433 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T09:29:28.473 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053429,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249180k,nr_inodes=812295,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-04T09:29:28.474 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:29:28.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:28.697+0000 7fa453e00640 1 Processor -- start 2024-03-04T09:29:28.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:28.697+0000 7fa453e00640 1 -- start start 2024-03-04T09:29:28.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:28.699+0000 7fa453e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa45415e0b0 0x7fa4540cf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:28.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:28.699+0000 7fa453e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa4540d0e50 0x7fa4540b44d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:28.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:28.699+0000 7fa453e00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4540b3710 con 0x7fa4540d4a10 2024-03-04T09:29:28.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:28.699+0000 7fa453e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4540b09e0 con 0x7fa4540d0e50 2024-03-04T09:29:28.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:28.699+0000 7fa453e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4540b39f0 con 0x7fa45415e0b0 2024-03-04T09:29:28.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:28.699+0000 7fa453400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa4540d0e50 0x7fa4540b44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:28.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:28.699+0000 7fa453400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa4540d0e50 0x7fa4540b44d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:46316/0 (socket says 172.21.15.6:46316) 2024-03-04T09:29:28.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:28.699+0000 7fa453400640 1 -- 172.21.15.6:0/1439859303 learned_addr learned my addr 172.21.15.6:0/1439859303 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:29:28.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:28.700+0000 7fa452a00640 1 --2- 172.21.15.6:0/1439859303 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa45415e0b0 0x7fa4540cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:28.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:28.700+0000 7fa453400640 1 -- 172.21.15.6:0/1439859303 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa45415e0b0 msgr2=0x7fa4540cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:28.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:28.700+0000 7fa453400640 1 --2- 172.21.15.6:0/1439859303 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa45415e0b0 0x7fa4540cf310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:28.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:28.700+0000 7fa453400640 1 -- 172.21.15.6:0/1439859303 >> v1:172.21.15.146:6789/0 conn(0x7fa4540d4a10 legacy=0x7fa4540d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:28.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:28.700+0000 7fa453400640 1 -- 172.21.15.6:0/1439859303 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4540b3cd0 con 0x7fa4540d0e50 2024-03-04T09:29:28.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:28.700+0000 7fa452a00640 1 --2- 172.21.15.6:0/1439859303 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa45415e0b0 0x7fa4540cf310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:29:28.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:28.701+0000 7fa453400640 1 --2- 172.21.15.6:0/1439859303 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4540d0e50 0x7fa4540b44d0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7fa44c009870 tx=0x7fa44c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23d73c218000007 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:28.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:28.701+0000 7fa451600640 1 -- 172.21.15.6:0/1439859303 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa44c035020 con 0x7fa4540d0e50 2024-03-04T09:29:28.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:28.701+0000 7fa451600640 1 -- 172.21.15.6:0/1439859303 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa44c031640 con 0x7fa4540d0e50 2024-03-04T09:29:28.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:28.701+0000 7fa451600640 1 -- 172.21.15.6:0/1439859303 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa44c02d040 con 0x7fa4540d0e50 2024-03-04T09:29:28.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:28.702+0000 7fa453e00640 1 -- 172.21.15.6:0/1439859303 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4540d0e50 msgr2=0x7fa4540b44d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:28.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:28.702+0000 7fa453e00640 1 --2- 172.21.15.6:0/1439859303 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4540d0e50 0x7fa4540b44d0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7fa44c009870 tx=0x7fa44c02b8f0 comp rx=0 tx=0).stop 2024-03-04T09:29:28.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:28.702+0000 7fa453e00640 1 -- 172.21.15.6:0/1439859303 shutdown_connections 2024-03-04T09:29:28.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:28.702+0000 7fa453e00640 1 --2- 172.21.15.6:0/1439859303 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4540d0e50 0x7fa4540b44d0 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:28.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:28.702+0000 7fa453e00640 1 --2- 172.21.15.6:0/1439859303 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa45415e0b0 0x7fa4540cf310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:28.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:28.702+0000 7fa453e00640 1 -- 172.21.15.6:0/1439859303 >> 172.21.15.6:0/1439859303 conn(0x7fa4540d3d30 msgr2=0x7fa4540d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:29:28.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:28.702+0000 7fa453e00640 1 -- 172.21.15.6:0/1439859303 shutdown_connections 2024-03-04T09:29:28.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:28.702+0000 7fa453e00640 1 -- 172.21.15.6:0/1439859303 wait complete. 2024-03-04T09:29:28.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:28.704+0000 7fa453e00640 1 Processor -- start 2024-03-04T09:29:28.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:28.704+0000 7fa453e00640 1 -- start start 2024-03-04T09:29:28.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:28.705+0000 7fa453e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa4540d0e50 0x7fa454158310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:28.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:28.705+0000 7fa453e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4540d4a10 0x7fa4541f6fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:28.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:28.705+0000 7fa453e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa45415e0b0 0x7fa45406bde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:28.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:28.705+0000 7fa453e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4540cfc60 con 0x7fa4540d4a10 2024-03-04T09:29:28.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:28.705+0000 7fa453e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa45415e490 con 0x7fa4540d0e50 2024-03-04T09:29:28.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:28.705+0000 7fa453e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4540cf850 con 0x7fa45415e0b0 2024-03-04T09:29:28.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:28.705+0000 7fa452000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4540d4a10 0x7fa4541f6fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:28.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:28.705+0000 7fa452000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4540d4a10 0x7fa4541f6fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:46320/0 (socket says 172.21.15.6:46320) 2024-03-04T09:29:28.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:28.705+0000 7fa452000640 1 -- 172.21.15.6:0/4252783120 learned_addr learned my addr 172.21.15.6:0/4252783120 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:29:28.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:28.705+0000 7fa452a00640 1 --2- 172.21.15.6:0/4252783120 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa4540d0e50 0x7fa454158310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:28.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:28.705+0000 7fa453400640 1 --2- 172.21.15.6:0/4252783120 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa45415e0b0 0x7fa45406bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:28.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:28.706+0000 7fa452000640 1 -- 172.21.15.6:0/4252783120 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa45415e0b0 msgr2=0x7fa45406bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:28.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:28.706+0000 7fa452000640 1 --2- 172.21.15.6:0/4252783120 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa45415e0b0 0x7fa45406bde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:28.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:28.706+0000 7fa452000640 1 -- 172.21.15.6:0/4252783120 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa4540d0e50 msgr2=0x7fa454158310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:28.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:28.706+0000 7fa452000640 1 --2- 172.21.15.6:0/4252783120 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa4540d0e50 0x7fa454158310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:28.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:28.706+0000 7fa452000640 1 -- 172.21.15.6:0/4252783120 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4541589c0 con 0x7fa4540d4a10 2024-03-04T09:29:28.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:28.706+0000 7fa452000640 1 --2- 172.21.15.6:0/4252783120 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4540d4a10 0x7fa4541f6fa0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7fa44400c7b0 tx=0x7fa44400cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:28.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:28.706+0000 7fa44aa00640 1 -- 172.21.15.6:0/4252783120 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa444016020 con 0x7fa4540d4a10 2024-03-04T09:29:28.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:28.706+0000 7fa44aa00640 1 -- 172.21.15.6:0/4252783120 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa44400aa50 con 0x7fa4540d4a10 2024-03-04T09:29:28.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:28.706+0000 7fa44aa00640 1 -- 172.21.15.6:0/4252783120 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa44400ae40 con 0x7fa4540d4a10 2024-03-04T09:29:28.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:28.706+0000 7fa453e00640 1 -- 172.21.15.6:0/4252783120 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4541f7720 con 0x7fa4540d4a10 2024-03-04T09:29:28.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:28.706+0000 7fa453e00640 1 -- 172.21.15.6:0/4252783120 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4541f7ce0 con 0x7fa4540d4a10 2024-03-04T09:29:28.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:28.708+0000 7fa453e00640 1 -- 172.21.15.6:0/4252783120 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa45404f330 con 0x7fa4540d4a10 2024-03-04T09:29:28.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:28.709+0000 7fa44aa00640 1 -- 172.21.15.6:0/4252783120 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa444019020 con 0x7fa4540d4a10 2024-03-04T09:29:28.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:28.710+0000 7fa44aa00640 1 --2- 172.21.15.6:0/4252783120 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa43006b520 0x7fa43006d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:28.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:28.710+0000 7fa44aa00640 1 -- 172.21.15.6:0/4252783120 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fa44408a1e0 con 0x7fa4540d4a10 2024-03-04T09:29:28.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:28.711+0000 7fa452a00640 1 --2- 172.21.15.6:0/4252783120 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa43006b520 0x7fa43006d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:28.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:28.712+0000 7fa452a00640 1 --2- 172.21.15.6:0/4252783120 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa43006b520 0x7fa43006d9e0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fa43c00aac0 tx=0x7fa43c008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:28.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:28.715+0000 7fa44aa00640 1 -- 172.21.15.6:0/4252783120 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa444020030 con 0x7fa4540d4a10 2024-03-04T09:29:28.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:28.954+0000 7fa453e00640 1 -- 172.21.15.6:0/4252783120 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa454074110 con 0x7fa4540d4a10 2024-03-04T09:29:28.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:28.955+0000 7fa44aa00640 1 -- 172.21.15.6:0/4252783120 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+4979 (secure 0 0 0) 0x7fa444059050 con 0x7fa4540d4a10 2024-03-04T09:29:28.952 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:29:28.952 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":42,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5228,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/1891254736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1891254736},{"type":"v1","addr":"172.21.15.6:6834","nonce":1891254736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":5578,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/1781219843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":1781219843},{"type":"v1","addr":"172.21.15.146:6835","nonce":1781219843}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":5587,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/1479180802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":1479180802},{"type":"v1","addr":"172.21.15.6:6835","nonce":1479180802}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:28:50.708467+0000","modified":"2024-03-04T09:28:53.801285+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5590},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5590":{"gid":5590,"name":"d","rank":0,"incarnation":38,"state":"up:active","state_seq":3,"addr":"172.21.15.146:6837/3919939129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":3919939129},{"type":"v1","addr":"172.21.15.146:6837","nonce":3919939129}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5590,"qdb_cluster":[5590]},"id":5}]} 2024-03-04T09:29:28.952 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 42 2024-03-04T09:29:28.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:28.957+0000 7fa453e00640 1 -- 172.21.15.6:0/4252783120 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa43006b520 msgr2=0x7fa43006d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:28.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:28.957+0000 7fa453e00640 1 --2- 172.21.15.6:0/4252783120 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa43006b520 0x7fa43006d9e0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fa43c00aac0 tx=0x7fa43c008040 comp rx=0 tx=0).stop 2024-03-04T09:29:28.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:28.957+0000 7fa453e00640 1 -- 172.21.15.6:0/4252783120 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4540d4a10 msgr2=0x7fa4541f6fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:28.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:28.957+0000 7fa453e00640 1 --2- 172.21.15.6:0/4252783120 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4540d4a10 0x7fa4541f6fa0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7fa44400c7b0 tx=0x7fa44400cc70 comp rx=0 tx=0).stop 2024-03-04T09:29:28.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:28.957+0000 7fa453e00640 1 -- 172.21.15.6:0/4252783120 shutdown_connections 2024-03-04T09:29:28.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:28.957+0000 7fa453e00640 1 --2- 172.21.15.6:0/4252783120 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa43006b520 0x7fa43006d9e0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:28.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:28.957+0000 7fa453e00640 1 --2- 172.21.15.6:0/4252783120 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa45415e0b0 0x7fa45406bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:28.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:28.957+0000 7fa453e00640 1 --2- 172.21.15.6:0/4252783120 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4540d4a10 0x7fa4541f6fa0 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:28.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:28.957+0000 7fa453e00640 1 --2- 172.21.15.6:0/4252783120 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa4540d0e50 0x7fa454158310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:28.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:28.957+0000 7fa453e00640 1 -- 172.21.15.6:0/4252783120 >> 172.21.15.6:0/4252783120 conn(0x7fa4540d3d30 msgr2=0x7fa4540d4210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:29:28.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:28.957+0000 7fa453e00640 1 -- 172.21.15.6:0/4252783120 shutdown_connections 2024-03-04T09:29:28.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:28.957+0000 7fa453e00640 1 -- 172.21.15.6:0/4252783120 wait complete. 2024-03-04T09:29:28.974 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T09:29:29.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.196+0000 7f51a4a00640 1 Processor -- start 2024-03-04T09:29:29.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.196+0000 7f51a4a00640 1 -- start start 2024-03-04T09:29:29.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.198+0000 7f51a4a00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f51a00cd760 0x7f51a015d430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:29.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.198+0000 7f51a4a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51a0159cc0 con 0x7f51a01598e0 2024-03-04T09:29:29.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.198+0000 7f51a4a00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51a00729b0 con 0x7f51a015d970 2024-03-04T09:29:29.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.198+0000 7f51a4a00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51a006e3f0 con 0x7f51a00cd760 2024-03-04T09:29:29.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.198+0000 7f519e800640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f51a01598e0 0x7f51a006f620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:42852/0 (socket says 172.21.15.6:42852) 2024-03-04T09:29:29.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.198+0000 7f519e800640 1 -- 172.21.15.6:0/4022932541 learned_addr learned my addr 172.21.15.6:0/4022932541 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:29:29.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.198+0000 7f519d400640 1 -- 172.21.15.6:0/4022932541 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2195728305 0 0) 0x7f51a0159cc0 con 0x7f51a01598e0 2024-03-04T09:29:29.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.199+0000 7f519d400640 1 -- 172.21.15.6:0/4022932541 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5184003520 con 0x7f51a01598e0 2024-03-04T09:29:29.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.199+0000 7f519de00640 1 --2- 172.21.15.6:0/4022932541 >> v2:172.21.15.146:3300/0 conn(0x7f51a00cd760 0x7f51a015d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:29.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.199+0000 7f519d400640 1 -- 172.21.15.6:0/4022932541 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2583905744 0 0) 0x7f5184003520 con 0x7f51a01598e0 2024-03-04T09:29:29.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.199+0000 7f519d400640 1 -- 172.21.15.6:0/4022932541 >> v2:172.21.15.146:3300/0 conn(0x7f51a00cd760 msgr2=0x7f51a015d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:29.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.199+0000 7f519d400640 1 --2- 172.21.15.6:0/4022932541 >> v2:172.21.15.146:3300/0 conn(0x7f51a00cd760 0x7f51a015d430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:29.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.199+0000 7f519d400640 1 -- 172.21.15.6:0/4022932541 >> v1:172.21.15.146:6789/0 conn(0x7f51a015d970 legacy=0x7f51a006afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:29.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.199+0000 7f519d400640 1 -- 172.21.15.6:0/4022932541 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51a0072c50 con 0x7f51a01598e0 2024-03-04T09:29:29.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.199+0000 7f519d400640 1 -- 172.21.15.6:0/4022932541 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7f5190001670 con 0x7f51a01598e0 2024-03-04T09:29:29.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.199+0000 7f519d400640 1 -- 172.21.15.6:0/4022932541 >> v1:172.21.15.6:6789/0 conn(0x7f51a01598e0 legacy=0x7f51a006f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:29.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.199+0000 7f519d400640 1 --2- 172.21.15.6:0/4022932541 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5184003a00 0x7f5184003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:29.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.199+0000 7f519d400640 1 --2- 172.21.15.6:0/4022932541 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5184004800 0x7f5184004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:29.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.199+0000 7f519d400640 1 --2- 172.21.15.6:0/4022932541 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5184005210 0x7f518400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:29.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.199+0000 7f519d400640 1 -- 172.21.15.6:0/4022932541 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51a006e3f0 con 0x7f5184004800 2024-03-04T09:29:29.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.199+0000 7f519d400640 1 -- 172.21.15.6:0/4022932541 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f518400c140 con 0x7f5184003a00 2024-03-04T09:29:29.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.199+0000 7f519d400640 1 -- 172.21.15.6:0/4022932541 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f518400c340 con 0x7f5184005210 2024-03-04T09:29:29.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.199+0000 7f519de00640 1 --2- 172.21.15.6:0/4022932541 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5184004800 0x7f5184004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:29.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.200+0000 7f519e800640 1 --2- 172.21.15.6:0/4022932541 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5184003a00 0x7f5184003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:29.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.200+0000 7f519de00640 1 -- 172.21.15.6:0/4022932541 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5184005210 msgr2=0x7f518400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:29.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.200+0000 7f519de00640 1 --2- 172.21.15.6:0/4022932541 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5184005210 0x7f518400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:29.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.200+0000 7f519de00640 1 -- 172.21.15.6:0/4022932541 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5184003a00 msgr2=0x7f5184003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:29.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.200+0000 7f519de00640 1 --2- 172.21.15.6:0/4022932541 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5184003a00 0x7f5184003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:29.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.200+0000 7f519de00640 1 -- 172.21.15.6:0/4022932541 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5184003520 con 0x7f5184004800 2024-03-04T09:29:29.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.200+0000 7f519f200640 1 -- 172.21.15.6:0/4022932541 reap_dead start 2024-03-04T09:29:29.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.201+0000 7f519de00640 1 --2- 172.21.15.6:0/4022932541 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5184004800 0x7f5184004c80 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f518c00ba00 tx=0x7f5184005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=420cc8ee22852925 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:29.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.201+0000 7f519d400640 1 -- 172.21.15.6:0/4022932541 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f518c036020 con 0x7f5184004800 2024-03-04T09:29:29.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.201+0000 7f519d400640 1 -- 172.21.15.6:0/4022932541 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f518c0325f0 con 0x7f5184004800 2024-03-04T09:29:29.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.201+0000 7f519d400640 1 -- 172.21.15.6:0/4022932541 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f518c00b600 con 0x7f5184004800 2024-03-04T09:29:29.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.202+0000 7f51a4a00640 1 -- 172.21.15.6:0/4022932541 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5184004800 msgr2=0x7f5184004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:29.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.202+0000 7f51a4a00640 1 --2- 172.21.15.6:0/4022932541 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5184004800 0x7f5184004c80 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f518c00ba00 tx=0x7f5184005830 comp rx=0 tx=0).stop 2024-03-04T09:29:29.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.202+0000 7f51a4a00640 1 -- 172.21.15.6:0/4022932541 shutdown_connections 2024-03-04T09:29:29.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.202+0000 7f51a4a00640 1 --2- 172.21.15.6:0/4022932541 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5184004800 0x7f5184004c80 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:29.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.202+0000 7f51a4a00640 1 -- 172.21.15.6:0/4022932541 >> 172.21.15.6:0/4022932541 conn(0x7f51a006c0c0 msgr2=0x7f51a006c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:29:29.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.202+0000 7f51a4a00640 1 -- 172.21.15.6:0/4022932541 shutdown_connections 2024-03-04T09:29:29.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.202+0000 7f51a4a00640 1 -- 172.21.15.6:0/4022932541 wait complete. 2024-03-04T09:29:29.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.204+0000 7f51a4a00640 1 Processor -- start 2024-03-04T09:29:29.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.204+0000 7f51a4a00640 1 -- start start 2024-03-04T09:29:29.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.205+0000 7f51a4a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5184004800 0x7f51a0156a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:29.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.205+0000 7f51a4a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f51a0158560 0x7f51a00cf080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:29.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.205+0000 7f51a4a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f51a0157a40 0x7f51a00b1910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:29.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.205+0000 7f51a4a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51a006e080 con 0x7f51a0157a40 2024-03-04T09:29:29.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.205+0000 7f51a4a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51a006dda0 con 0x7f5184004800 2024-03-04T09:29:29.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.205+0000 7f51a4a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51a015d430 con 0x7f51a0158560 2024-03-04T09:29:29.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.205+0000 7f519f200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f51a0157a40 0x7f51a00b1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:29.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.205+0000 7f519f200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f51a0157a40 0x7f51a00b1910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:52844/0 (socket says 172.21.15.6:52844) 2024-03-04T09:29:29.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.205+0000 7f519f200640 1 -- 172.21.15.6:0/644198022 learned_addr learned my addr 172.21.15.6:0/644198022 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:29:29.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.205+0000 7f519e800640 1 --2- 172.21.15.6:0/644198022 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5184004800 0x7f51a0156a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:29.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.205+0000 7f519de00640 1 --2- 172.21.15.6:0/644198022 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f51a0158560 0x7f51a00cf080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:29.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.205+0000 7f519f200640 1 -- 172.21.15.6:0/644198022 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f51a0158560 msgr2=0x7f51a00cf080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:29.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.206+0000 7f519f200640 1 --2- 172.21.15.6:0/644198022 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f51a0158560 0x7f51a00cf080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:29.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.206+0000 7f519f200640 1 -- 172.21.15.6:0/644198022 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5184004800 msgr2=0x7f51a0156a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:29.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.206+0000 7f519f200640 1 --2- 172.21.15.6:0/644198022 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5184004800 0x7f51a0156a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:29.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.206+0000 7f519f200640 1 -- 172.21.15.6:0/644198022 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51a00cf5c0 con 0x7f51a0157a40 2024-03-04T09:29:29.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.206+0000 7f519de00640 1 --2- 172.21.15.6:0/644198022 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f51a0158560 0x7f51a00cf080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:29:29.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.206+0000 7f519f200640 1 --2- 172.21.15.6:0/644198022 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f51a0157a40 0x7f51a00b1910 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f51a00b3c30 tx=0x7f519800c980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:29.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.206+0000 7f5196a00640 1 -- 172.21.15.6:0/644198022 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5198017070 con 0x7f51a0157a40 2024-03-04T09:29:29.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.206+0000 7f5196a00640 1 -- 172.21.15.6:0/644198022 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5198007070 con 0x7f51a0157a40 2024-03-04T09:29:29.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.206+0000 7f5196a00640 1 -- 172.21.15.6:0/644198022 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5198007460 con 0x7f51a0157a40 2024-03-04T09:29:29.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.206+0000 7f51a4a00640 1 -- 172.21.15.6:0/644198022 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51a00b3de0 con 0x7f51a0157a40 2024-03-04T09:29:29.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.206+0000 7f51a4a00640 1 -- 172.21.15.6:0/644198022 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51a00b43a0 con 0x7f51a0157a40 2024-03-04T09:29:29.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.207+0000 7f51a4a00640 1 -- 172.21.15.6:0/644198022 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51a004f330 con 0x7f51a0157a40 2024-03-04T09:29:29.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.208+0000 7f5196a00640 1 -- 172.21.15.6:0/644198022 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f519801a070 con 0x7f51a0157a40 2024-03-04T09:29:29.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.209+0000 7f5196a00640 1 --2- 172.21.15.6:0/644198022 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f518006b4d0 0x7f518006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:29.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.209+0000 7f5196a00640 1 -- 172.21.15.6:0/644198022 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f5198089590 con 0x7f51a0157a40 2024-03-04T09:29:29.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.210+0000 7f519e800640 1 --2- 172.21.15.6:0/644198022 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f518006b4d0 0x7f518006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:29.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.211+0000 7f519e800640 1 --2- 172.21.15.6:0/644198022 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f518006b4d0 0x7f518006d990 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f5190001050 tx=0x7f5190009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:29.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.213+0000 7f5196a00640 1 -- 172.21.15.6:0/644198022 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f51980582f0 con 0x7f51a0157a40 2024-03-04T09:29:29.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.416+0000 7f51a4a00640 1 -- 172.21.15.6:0/644198022 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f51a0072340 con 0x7f51a0157a40 2024-03-04T09:29:29.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.419+0000 7f5196a00640 1 -- 172.21.15.6:0/644198022 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v70) v1 ==== 74+0+16813 (secure 0 0 0) 0x7f5198057bf0 con 0x7f51a0157a40 2024-03-04T09:29:29.416 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:29:29.416 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":70,"fsid":"edce0578-5ae4-4402-b45a-552bdef84d7c","created":"2024-03-04T09:24:54.523780+0000","modified":"2024-03-04T09:29:02.333052+0000","last_up_change":"2024-03-04T09:25:07.089049+0000","last_in_change":"2024-03-04T09:24:57.982077+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T09:25:07.935198+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-03-04T09:28:48.627427+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-03-04T09:28:49.739043+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a96574f-81f1-4bc6-8463-757520b84a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6825","nonce":1364449401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6827","nonce":1364449401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6831","nonce":1364449401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6829","nonce":1364449401}]},"public_addr":"172.21.15.6:6825/1364449401","cluster_addr":"172.21.15.6:6827/1364449401","heartbeat_back_addr":"172.21.15.6:6831/1364449401","heartbeat_front_addr":"172.21.15.6:6829/1364449401","state":["exists","up"]},{"osd":1,"uuid":"7d79b9c1-6b87-476a-bc36-7db5ee64cd90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6809","nonce":2494103078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6811","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6813","nonce":2494103078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6821","nonce":2494103078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6817","nonce":2494103078}]},"public_addr":"172.21.15.6:6809/2494103078","cluster_addr":"172.21.15.6:6813/2494103078","heartbeat_back_addr":"172.21.15.6:6821/2494103078","heartbeat_front_addr":"172.21.15.6:6817/2494103078","state":["exists","up"]},{"osd":2,"uuid":"492331b9-4ee9-40b6-bacd-85d44cd6c45a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6812","nonce":1500066681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6816","nonce":1500066681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6824","nonce":1500066681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6820","nonce":1500066681}]},"public_addr":"172.21.15.6:6812/1500066681","cluster_addr":"172.21.15.6:6816/1500066681","heartbeat_back_addr":"172.21.15.6:6824/1500066681","heartbeat_front_addr":"172.21.15.6:6820/1500066681","state":["exists","up"]},{"osd":3,"uuid":"21bb2efe-9ba7-4bdc-bc31-180cd1102e21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6801","nonce":1558806123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6803","nonce":1558806123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6807","nonce":1558806123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6805","nonce":1558806123}]},"public_addr":"172.21.15.6:6801/1558806123","cluster_addr":"172.21.15.6:6803/1558806123","heartbeat_back_addr":"172.21.15.6:6807/1558806123","heartbeat_front_addr":"172.21.15.6:6805/1558806123","state":["exists","up"]},{"osd":4,"uuid":"6e2c4b71-19d9-4737-83d9-558705d8644a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6801","nonce":1526135780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6803","nonce":1526135780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6807","nonce":1526135780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6805","nonce":1526135780}]},"public_addr":"172.21.15.146:6801/1526135780","cluster_addr":"172.21.15.146:6803/1526135780","heartbeat_back_addr":"172.21.15.146:6807/1526135780","heartbeat_front_addr":"172.21.15.146:6805/1526135780","state":["exists","up"]},{"osd":5,"uuid":"fb69c525-f1be-4b7d-ab32-6c29b0d6fddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6809","nonce":460262507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6811","nonce":460262507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6815","nonce":460262507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6813","nonce":460262507}]},"public_addr":"172.21.15.146:6809/460262507","cluster_addr":"172.21.15.146:6811/460262507","heartbeat_back_addr":"172.21.15.146:6815/460262507","heartbeat_front_addr":"172.21.15.146:6813/460262507","state":["exists","up"]},{"osd":6,"uuid":"c32e7fed-370d-4401-8ce0-6a4335b0d168","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6823","nonce":1392897493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6827","nonce":1392897493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6831","nonce":1392897493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6829","nonce":1392897493}]},"public_addr":"172.21.15.146:6823/1392897493","cluster_addr":"172.21.15.146:6827/1392897493","heartbeat_back_addr":"172.21.15.146:6831/1392897493","heartbeat_front_addr":"172.21.15.146:6829/1392897493","state":["exists","up"]},{"osd":7,"uuid":"75a79b44-fa5f-426d-a7d3-71f21f78835c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6817","nonce":3268323156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6819","nonce":3268323156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6825","nonce":3268323156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6822","nonce":3268323156}]},"public_addr":"172.21.15.146:6817/3268323156","cluster_addr":"172.21.15.146:6819/3268323156","heartbeat_back_addr":"172.21.15.146:6825/3268323156","heartbeat_front_addr":"172.21.15.146:6822/3268323156","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.310437+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.295850+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.294674+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.526855+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.300023+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.595160+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.346046+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.558741+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"11.8","mappings":[{"from":6,"to":5}]},{"pgid":"11.10","mappings":[{"from":6,"to":1}]},{"pgid":"11.14","mappings":[{"from":6,"to":0}]},{"pgid":"11.3e","mappings":[{"from":6,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T09:29:29.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.422+0000 7f51a4a00640 1 -- 172.21.15.6:0/644198022 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f518006b4d0 msgr2=0x7f518006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:29.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.422+0000 7f51a4a00640 1 --2- 172.21.15.6:0/644198022 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f518006b4d0 0x7f518006d990 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f5190001050 tx=0x7f5190009040 comp rx=0 tx=0).stop 2024-03-04T09:29:29.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.422+0000 7f51a4a00640 1 -- 172.21.15.6:0/644198022 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f51a0157a40 msgr2=0x7f51a00b1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:29.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.422+0000 7f51a4a00640 1 --2- 172.21.15.6:0/644198022 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f51a0157a40 0x7f51a00b1910 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f51a00b3c30 tx=0x7f519800c980 comp rx=0 tx=0).stop 2024-03-04T09:29:29.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.422+0000 7f51a4a00640 1 -- 172.21.15.6:0/644198022 shutdown_connections 2024-03-04T09:29:29.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.422+0000 7f51a4a00640 1 --2- 172.21.15.6:0/644198022 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f51a0157a40 0x7f51a00b1910 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:29.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.422+0000 7f51a4a00640 1 --2- 172.21.15.6:0/644198022 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f518006b4d0 0x7f518006d990 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:29.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.422+0000 7f51a4a00640 1 --2- 172.21.15.6:0/644198022 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f51a0158560 0x7f51a00cf080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:29.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.422+0000 7f51a4a00640 1 --2- 172.21.15.6:0/644198022 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5184004800 0x7f51a0156a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:29.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.422+0000 7f51a4a00640 1 -- 172.21.15.6:0/644198022 >> 172.21.15.6:0/644198022 conn(0x7f51a006c0c0 msgr2=0x7f51a006e880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:29:29.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.422+0000 7f51a4a00640 1 -- 172.21.15.6:0/644198022 shutdown_connections 2024-03-04T09:29:29.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.422+0000 7f51a4a00640 1 -- 172.21.15.6:0/644198022 wait complete. 2024-03-04T09:29:29.437 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T09:25:07.935198+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-04T09:28:48.627427+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '67', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_data', 'create_time': '2024-03-04T09:28:49.739043+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-04T09:29:29.438 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-04T09:29:29.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.669+0000 7fbf61c00640 1 Processor -- start 2024-03-04T09:29:29.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.669+0000 7fbf61c00640 1 -- start start 2024-03-04T09:29:29.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.671+0000 7fbf61c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbf5c15e680 0x7fbf5c06c980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:29.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.671+0000 7fbf61c00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf5c15eb80 con 0x7fbf5c06da40 2024-03-04T09:29:29.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.671+0000 7fbf61c00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf5c15ee60 con 0x7fbf5c0ccad0 2024-03-04T09:29:29.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.671+0000 7fbf61c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf5c06ea50 con 0x7fbf5c15e680 2024-03-04T09:29:29.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.671+0000 7fbf5aa00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fbf5c06da40 0x7fbf5c06e340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:42854/0 (socket says 172.21.15.6:42854) 2024-03-04T09:29:29.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.671+0000 7fbf5aa00640 1 -- 172.21.15.6:0/1024030361 learned_addr learned my addr 172.21.15.6:0/1024030361 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:29:29.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.672+0000 7fbf5a000640 1 --2- 172.21.15.6:0/1024030361 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbf5c15e680 0x7fbf5c06c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:29.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.672+0000 7fbf59600640 1 -- 172.21.15.6:0/1024030361 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2067668113 0 0) 0x7fbf5c15eb80 con 0x7fbf5c06da40 2024-03-04T09:29:29.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.672+0000 7fbf59600640 1 -- 172.21.15.6:0/1024030361 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf40003520 con 0x7fbf5c06da40 2024-03-04T09:29:29.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.672+0000 7fbf59600640 1 -- 172.21.15.6:0/1024030361 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2896657050 0 0) 0x7fbf40003520 con 0x7fbf5c06da40 2024-03-04T09:29:29.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.672+0000 7fbf59600640 1 -- 172.21.15.6:0/1024030361 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbf5c15e680 msgr2=0x7fbf5c06c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:29.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.672+0000 7fbf59600640 1 --2- 172.21.15.6:0/1024030361 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbf5c15e680 0x7fbf5c06c980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:29.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.672+0000 7fbf59600640 1 -- 172.21.15.6:0/1024030361 >> v1:172.21.15.146:6789/0 conn(0x7fbf5c0ccad0 legacy=0x7fbf5c162100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:29.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.672+0000 7fbf59600640 1 -- 172.21.15.6:0/1024030361 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf5c15f140 con 0x7fbf5c06da40 2024-03-04T09:29:29.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.672+0000 7fbf59600640 1 -- 172.21.15.6:0/1024030361 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7fbf4c001670 con 0x7fbf5c06da40 2024-03-04T09:29:29.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.672+0000 7fbf59600640 1 -- 172.21.15.6:0/1024030361 >> v1:172.21.15.6:6789/0 conn(0x7fbf5c06da40 legacy=0x7fbf5c06e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:29.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.672+0000 7fbf59600640 1 --2- 172.21.15.6:0/1024030361 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbf5c15e680 0x7fbf400039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:29.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.672+0000 7fbf59600640 1 --2- 172.21.15.6:0/1024030361 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbf40003fb0 0x7fbf40004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:29.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.672+0000 7fbf59600640 1 --2- 172.21.15.6:0/1024030361 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbf40004d40 0x7fbf400051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:29.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.672+0000 7fbf59600640 1 -- 172.21.15.6:0/1024030361 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf4000bad0 con 0x7fbf40003fb0 2024-03-04T09:29:29.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.672+0000 7fbf59600640 1 -- 172.21.15.6:0/1024030361 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf4000bc80 con 0x7fbf40004d40 2024-03-04T09:29:29.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.672+0000 7fbf59600640 1 -- 172.21.15.6:0/1024030361 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf4000be80 con 0x7fbf5c15e680 2024-03-04T09:29:29.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.673+0000 7fbf5a000640 1 --2- 172.21.15.6:0/1024030361 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbf40003fb0 0x7fbf40004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:29.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.673+0000 7fbf5aa00640 1 --2- 172.21.15.6:0/1024030361 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbf5c15e680 0x7fbf400039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:29.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.673+0000 7fbf5b400640 1 --2- 172.21.15.6:0/1024030361 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbf40004d40 0x7fbf400051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:29.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.673+0000 7fbf5a000640 1 -- 172.21.15.6:0/1024030361 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbf5c15e680 msgr2=0x7fbf400039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:29.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.673+0000 7fbf5a000640 1 --2- 172.21.15.6:0/1024030361 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbf5c15e680 0x7fbf400039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:29.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.673+0000 7fbf5a000640 1 -- 172.21.15.6:0/1024030361 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbf40004d40 msgr2=0x7fbf400051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:29.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.673+0000 7fbf5a000640 1 --2- 172.21.15.6:0/1024030361 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbf40004d40 0x7fbf400051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:29.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.673+0000 7fbf5a000640 1 -- 172.21.15.6:0/1024030361 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf40003520 con 0x7fbf40003fb0 2024-03-04T09:29:29.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.674+0000 7fbf5a000640 1 --2- 172.21.15.6:0/1024030361 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbf40003fb0 0x7fbf40004800 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7fbf48009ac0 tx=0x7fbf40005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15dee7a9e2fb4d3a server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:29.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.674+0000 7fbf59600640 1 -- 172.21.15.6:0/1024030361 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf48039070 con 0x7fbf40003fb0 2024-03-04T09:29:29.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.674+0000 7fbf59600640 1 -- 172.21.15.6:0/1024030361 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf4802c600 con 0x7fbf40003fb0 2024-03-04T09:29:29.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.675+0000 7fbf59600640 1 -- 172.21.15.6:0/1024030361 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf48009690 con 0x7fbf40003fb0 2024-03-04T09:29:29.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.675+0000 7fbf61c00640 1 -- 172.21.15.6:0/1024030361 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbf40003fb0 msgr2=0x7fbf40004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:29.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.675+0000 7fbf61c00640 1 --2- 172.21.15.6:0/1024030361 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbf40003fb0 0x7fbf40004800 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7fbf48009ac0 tx=0x7fbf40005830 comp rx=0 tx=0).stop 2024-03-04T09:29:29.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.675+0000 7fbf5b400640 1 -- 172.21.15.6:0/1024030361 reap_dead start 2024-03-04T09:29:29.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.675+0000 7fbf61c00640 1 -- 172.21.15.6:0/1024030361 shutdown_connections 2024-03-04T09:29:29.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.675+0000 7fbf61c00640 1 -- 172.21.15.6:0/1024030361 >> 172.21.15.6:0/1024030361 conn(0x7fbf5c06b3d0 msgr2=0x7fbf5c06b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:29:29.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.675+0000 7fbf61c00640 1 -- 172.21.15.6:0/1024030361 shutdown_connections 2024-03-04T09:29:29.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.675+0000 7fbf61c00640 1 -- 172.21.15.6:0/1024030361 wait complete. 2024-03-04T09:29:29.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.677+0000 7fbf61c00640 1 Processor -- start 2024-03-04T09:29:29.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.677+0000 7fbf61c00640 1 -- start start 2024-03-04T09:29:29.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.678+0000 7fbf61c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbf5c151470 0x7fbf5c0b3440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:29.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.678+0000 7fbf61c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbf5c151850 0x7fbf5c0cac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:29.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.678+0000 7fbf61c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbf5c0cb1c0 0x7fbf5c09bf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:29.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.678+0000 7fbf61c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf5c162810 con 0x7fbf5c151850 2024-03-04T09:29:29.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.678+0000 7fbf61c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf5c161c70 con 0x7fbf5c151470 2024-03-04T09:29:29.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.678+0000 7fbf61c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf5c162990 con 0x7fbf5c0cb1c0 2024-03-04T09:29:29.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.678+0000 7fbf5a000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbf5c151850 0x7fbf5c0cac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:29.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.678+0000 7fbf5a000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbf5c151850 0x7fbf5c0cac80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:52862/0 (socket says 172.21.15.6:52862) 2024-03-04T09:29:29.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.678+0000 7fbf5a000640 1 -- 172.21.15.6:0/1131340256 learned_addr learned my addr 172.21.15.6:0/1131340256 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:29:29.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.679+0000 7fbf5aa00640 1 --2- 172.21.15.6:0/1131340256 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbf5c151470 0x7fbf5c0b3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:29.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.679+0000 7fbf5b400640 1 --2- 172.21.15.6:0/1131340256 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbf5c0cb1c0 0x7fbf5c09bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:29.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.679+0000 7fbf5a000640 1 -- 172.21.15.6:0/1131340256 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbf5c0cb1c0 msgr2=0x7fbf5c09bf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:29.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.679+0000 7fbf5a000640 1 --2- 172.21.15.6:0/1131340256 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbf5c0cb1c0 0x7fbf5c09bf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:29.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.679+0000 7fbf5a000640 1 -- 172.21.15.6:0/1131340256 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbf5c151470 msgr2=0x7fbf5c0b3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:29.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.679+0000 7fbf5a000640 1 --2- 172.21.15.6:0/1131340256 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbf5c151470 0x7fbf5c0b3440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:29.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.679+0000 7fbf5a000640 1 -- 172.21.15.6:0/1131340256 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf5c09ba50 con 0x7fbf5c151850 2024-03-04T09:29:29.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.679+0000 7fbf5b400640 1 --2- 172.21.15.6:0/1131340256 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbf5c0cb1c0 0x7fbf5c09bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:29:29.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.679+0000 7fbf5a000640 1 --2- 172.21.15.6:0/1131340256 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbf5c151850 0x7fbf5c0cac80 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7fbf480099e0 tx=0x7fbf48004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:29.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.679+0000 7fbf52a00640 1 -- 172.21.15.6:0/1131340256 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf48039070 con 0x7fbf5c151850 2024-03-04T09:29:29.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.680+0000 7fbf52a00640 1 -- 172.21.15.6:0/1131340256 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf4803a040 con 0x7fbf5c151850 2024-03-04T09:29:29.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.680+0000 7fbf61c00640 1 -- 172.21.15.6:0/1131340256 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf5c0b3a20 con 0x7fbf5c151850 2024-03-04T09:29:29.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.680+0000 7fbf52a00640 1 -- 172.21.15.6:0/1131340256 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf480044e0 con 0x7fbf5c151850 2024-03-04T09:29:29.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.680+0000 7fbf61c00640 1 -- 172.21.15.6:0/1131340256 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf5c0b3fe0 con 0x7fbf5c151850 2024-03-04T09:29:29.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.681+0000 7fbf61c00640 1 -- 172.21.15.6:0/1131340256 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf5c0cb5d0 con 0x7fbf5c151850 2024-03-04T09:29:29.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.682+0000 7fbf52a00640 1 -- 172.21.15.6:0/1131340256 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbf48045030 con 0x7fbf5c151850 2024-03-04T09:29:29.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.682+0000 7fbf52a00640 1 --2- 172.21.15.6:0/1131340256 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fbf3c06b4d0 0x7fbf3c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:29.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.683+0000 7fbf52a00640 1 -- 172.21.15.6:0/1131340256 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fbf480aacb0 con 0x7fbf5c151850 2024-03-04T09:29:29.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.683+0000 7fbf5aa00640 1 --2- 172.21.15.6:0/1131340256 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fbf3c06b4d0 0x7fbf3c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:29.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.684+0000 7fbf5aa00640 1 --2- 172.21.15.6:0/1131340256 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fbf3c06b4d0 0x7fbf3c06d990 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fbf4c001050 tx=0x7fbf4c009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:29.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.688+0000 7fbf52a00640 1 -- 172.21.15.6:0/1131340256 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbf5c0cb5d0 con 0x7fbf5c151850 2024-03-04T09:29:29.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.897+0000 7fbf61c00640 1 -- 172.21.15.6:0/1131340256 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fbf5c04f330 con 0x7fbf5c151850 2024-03-04T09:29:29.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.898+0000 7fbf52a00640 1 -- 172.21.15.6:0/1131340256 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v70) v1 ==== 93+0+10 (secure 0 0 0) 0x7fbf48079aa0 con 0x7fbf5c151850 2024-03-04T09:29:29.895 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-04T09:29:29.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.900+0000 7fbf61c00640 1 -- 172.21.15.6:0/1131340256 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fbf3c06b4d0 msgr2=0x7fbf3c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:29.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.901+0000 7fbf61c00640 1 --2- 172.21.15.6:0/1131340256 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fbf3c06b4d0 0x7fbf3c06d990 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fbf4c001050 tx=0x7fbf4c009040 comp rx=0 tx=0).stop 2024-03-04T09:29:29.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.901+0000 7fbf61c00640 1 -- 172.21.15.6:0/1131340256 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbf5c151850 msgr2=0x7fbf5c0cac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:29.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.901+0000 7fbf61c00640 1 --2- 172.21.15.6:0/1131340256 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbf5c151850 0x7fbf5c0cac80 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7fbf480099e0 tx=0x7fbf48004730 comp rx=0 tx=0).stop 2024-03-04T09:29:29.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.901+0000 7fbf61c00640 1 -- 172.21.15.6:0/1131340256 shutdown_connections 2024-03-04T09:29:29.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.901+0000 7fbf61c00640 1 --2- 172.21.15.6:0/1131340256 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fbf3c06b4d0 0x7fbf3c06d990 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:29.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.901+0000 7fbf61c00640 1 --2- 172.21.15.6:0/1131340256 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbf5c0cb1c0 0x7fbf5c09bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:29.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.901+0000 7fbf61c00640 1 --2- 172.21.15.6:0/1131340256 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbf5c151850 0x7fbf5c0cac80 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:29.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.901+0000 7fbf61c00640 1 --2- 172.21.15.6:0/1131340256 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbf5c151470 0x7fbf5c0b3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:29.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.901+0000 7fbf61c00640 1 -- 172.21.15.6:0/1131340256 >> 172.21.15.6:0/1131340256 conn(0x7fbf5c06b3d0 msgr2=0x7fbf5c15fe90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:29:29.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.901+0000 7fbf61c00640 1 -- 172.21.15.6:0/1131340256 shutdown_connections 2024-03-04T09:29:29.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:29.901+0000 7fbf61c00640 1 -- 172.21.15.6:0/1131340256 wait complete. 2024-03-04T09:29:29.915 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-04T09:29:30.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.138+0000 7f33f3400640 1 Processor -- start 2024-03-04T09:29:30.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.138+0000 7f33f3400640 1 -- start start 2024-03-04T09:29:30.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.140+0000 7f33f3400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f33ec06da40 0x7f33ec06e340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:30.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.140+0000 7f33f3400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f33ec15e680 0x7f33ec06c980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:30.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.140+0000 7f33f3400640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33ec15eb80 con 0x7f33ec0ccad0 2024-03-04T09:29:30.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.140+0000 7f33f3400640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33ec161c70 con 0x7f33ec06da40 2024-03-04T09:29:30.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.140+0000 7f33f3400640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33ec15ee60 con 0x7f33ec15e680 2024-03-04T09:29:30.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.140+0000 7f33f0a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f33ec06da40 0x7f33ec06e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:30.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.140+0000 7f33f0a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f33ec06da40 0x7f33ec06e340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:52868/0 (socket says 172.21.15.6:52868) 2024-03-04T09:29:30.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.140+0000 7f33f1400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f33ec0ccad0 0x7f33ec162100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:42864/0 (socket says 172.21.15.6:42864) 2024-03-04T09:29:30.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.140+0000 7f33f0a00640 1 -- 172.21.15.6:0/382178283 learned_addr learned my addr 172.21.15.6:0/382178283 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:29:30.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.141+0000 7f33ebe00640 1 --2- 172.21.15.6:0/382178283 >> v2:172.21.15.146:3300/0 conn(0x7f33ec15e680 0x7f33ec06c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:30.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.141+0000 7f33f0a00640 1 -- 172.21.15.6:0/382178283 >> v2:172.21.15.146:3300/0 conn(0x7f33ec15e680 msgr2=0x7f33ec06c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:30.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.141+0000 7f33f0a00640 1 --2- 172.21.15.6:0/382178283 >> v2:172.21.15.146:3300/0 conn(0x7f33ec15e680 0x7f33ec06c980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:30.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.141+0000 7f33f0a00640 1 -- 172.21.15.6:0/382178283 >> v1:172.21.15.6:6789/0 conn(0x7f33ec0ccad0 legacy=0x7f33ec162100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:30.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.141+0000 7f33f0a00640 1 -- 172.21.15.6:0/382178283 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33ec15f140 con 0x7f33ec06da40 2024-03-04T09:29:30.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.142+0000 7f33f0a00640 1 --2- 172.21.15.6:0/382178283 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f33ec06da40 0x7f33ec06e340 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f33ec06bfd0 tx=0x7f33dc02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1533016f3a7ad03 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:30.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.142+0000 7f33eb400640 1 -- 172.21.15.6:0/382178283 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33dc034070 con 0x7f33ec06da40 2024-03-04T09:29:30.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.142+0000 7f33eb400640 1 -- 172.21.15.6:0/382178283 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33dc030860 con 0x7f33ec06da40 2024-03-04T09:29:30.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.142+0000 7f33eb400640 1 -- 172.21.15.6:0/382178283 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33dc030c30 con 0x7f33ec06da40 2024-03-04T09:29:30.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.143+0000 7f33f3400640 1 -- 172.21.15.6:0/382178283 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f33ec06da40 msgr2=0x7f33ec06e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:30.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.143+0000 7f33f3400640 1 --2- 172.21.15.6:0/382178283 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f33ec06da40 0x7f33ec06e340 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f33ec06bfd0 tx=0x7f33dc02b9a0 comp rx=0 tx=0).stop 2024-03-04T09:29:30.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.143+0000 7f33f3400640 1 -- 172.21.15.6:0/382178283 shutdown_connections 2024-03-04T09:29:30.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.143+0000 7f33f3400640 1 --2- 172.21.15.6:0/382178283 >> v2:172.21.15.146:3300/0 conn(0x7f33ec15e680 0x7f33ec06c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:30.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.143+0000 7f33f3400640 1 --2- 172.21.15.6:0/382178283 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f33ec06da40 0x7f33ec06e340 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:30.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.143+0000 7f33f3400640 1 -- 172.21.15.6:0/382178283 >> 172.21.15.6:0/382178283 conn(0x7f33ec06b3d0 msgr2=0x7f33ec06b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:29:30.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.143+0000 7f33f3400640 1 -- 172.21.15.6:0/382178283 shutdown_connections 2024-03-04T09:29:30.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.143+0000 7f33f3400640 1 -- 172.21.15.6:0/382178283 wait complete. 2024-03-04T09:29:30.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.145+0000 7f33f3400640 1 Processor -- start 2024-03-04T09:29:30.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.145+0000 7f33f3400640 1 -- start start 2024-03-04T09:29:30.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.146+0000 7f33f3400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f33ec06da40 0x7f33ec0b3af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:30.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.146+0000 7f33f3400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f33ec0ccad0 0x7f33ec14cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:30.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.146+0000 7f33f3400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f33ec15e680 0x7f33ec1f3310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:30.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.146+0000 7f33f3400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33ec06ea00 con 0x7f33ec0ccad0 2024-03-04T09:29:30.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.146+0000 7f33f3400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33ec06e880 con 0x7f33ec06da40 2024-03-04T09:29:30.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.146+0000 7f33f3400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33ec06cec0 con 0x7f33ec15e680 2024-03-04T09:29:30.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.146+0000 7f33ebe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f33ec0ccad0 0x7f33ec14cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:30.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.146+0000 7f33ebe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f33ec0ccad0 0x7f33ec14cbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:52880/0 (socket says 172.21.15.6:52880) 2024-03-04T09:29:30.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.146+0000 7f33ebe00640 1 -- 172.21.15.6:0/276011050 learned_addr learned my addr 172.21.15.6:0/276011050 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:29:30.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.147+0000 7f33ebe00640 1 -- 172.21.15.6:0/276011050 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f33ec15e680 msgr2=0x7f33ec1f3310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:30.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.147+0000 7f33ebe00640 1 --2- 172.21.15.6:0/276011050 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f33ec15e680 0x7f33ec1f3310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:30.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.147+0000 7f33ebe00640 1 -- 172.21.15.6:0/276011050 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f33ec06da40 msgr2=0x7f33ec0b3af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:30.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.147+0000 7f33f0a00640 1 --2- 172.21.15.6:0/276011050 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f33ec06da40 0x7f33ec0b3af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:30.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.147+0000 7f33ebe00640 1 --2- 172.21.15.6:0/276011050 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f33ec06da40 0x7f33ec0b3af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:30.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.147+0000 7f33ebe00640 1 -- 172.21.15.6:0/276011050 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33ec1f3850 con 0x7f33ec0ccad0 2024-03-04T09:29:30.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.147+0000 7f33ebe00640 1 --2- 172.21.15.6:0/276011050 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f33ec0ccad0 0x7f33ec14cbe0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f33d800d5f0 tx=0x7f33d800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:30.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.147+0000 7f33e8c00640 1 -- 172.21.15.6:0/276011050 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33d80125f0 con 0x7f33ec0ccad0 2024-03-04T09:29:30.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.147+0000 7f33e8c00640 1 -- 172.21.15.6:0/276011050 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33d8019070 con 0x7f33ec0ccad0 2024-03-04T09:29:30.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.147+0000 7f33f3400640 1 -- 172.21.15.6:0/276011050 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33ec0b4200 con 0x7f33ec0ccad0 2024-03-04T09:29:30.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.147+0000 7f33e8c00640 1 -- 172.21.15.6:0/276011050 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33d80129d0 con 0x7f33ec0ccad0 2024-03-04T09:29:30.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.147+0000 7f33f3400640 1 -- 172.21.15.6:0/276011050 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33ec0b47c0 con 0x7f33ec0ccad0 2024-03-04T09:29:30.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.149+0000 7f33f3400640 1 -- 172.21.15.6:0/276011050 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33ec06e340 con 0x7f33ec0ccad0 2024-03-04T09:29:30.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.149+0000 7f33e8c00640 1 -- 172.21.15.6:0/276011050 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f33d8012b70 con 0x7f33ec0ccad0 2024-03-04T09:29:30.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.150+0000 7f33e8c00640 1 --2- 172.21.15.6:0/276011050 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f33cc06b4d0 0x7f33cc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:30.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.150+0000 7f33e8c00640 1 -- 172.21.15.6:0/276011050 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f33d808a6a0 con 0x7f33ec0ccad0 2024-03-04T09:29:30.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.151+0000 7f33f0a00640 1 --2- 172.21.15.6:0/276011050 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f33cc06b4d0 0x7f33cc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:30.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.152+0000 7f33f0a00640 1 --2- 172.21.15.6:0/276011050 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f33cc06b4d0 0x7f33cc06d990 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f33dc036000 tx=0x7f33dc03b040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:30.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.155+0000 7f33e8c00640 1 -- 172.21.15.6:0/276011050 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f33ec06e340 con 0x7f33ec0ccad0 2024-03-04T09:29:30.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.357+0000 7f33f3400640 1 -- 172.21.15.6:0/276011050 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f33ec04f330 con 0x7f33ec0ccad0 2024-03-04T09:29:30.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.358+0000 7f33e8c00640 1 -- 172.21.15.6:0/276011050 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v70) v1 ==== 104+0+11 (secure 0 0 0) 0x7f33d8059490 con 0x7f33ec0ccad0 2024-03-04T09:29:30.355 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-03-04T09:29:30.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.360+0000 7f33f3400640 1 -- 172.21.15.6:0/276011050 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f33cc06b4d0 msgr2=0x7f33cc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:30.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.360+0000 7f33f3400640 1 --2- 172.21.15.6:0/276011050 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f33cc06b4d0 0x7f33cc06d990 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f33dc036000 tx=0x7f33dc03b040 comp rx=0 tx=0).stop 2024-03-04T09:29:30.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.361+0000 7f33f3400640 1 -- 172.21.15.6:0/276011050 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f33ec0ccad0 msgr2=0x7f33ec14cbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:30.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.361+0000 7f33f3400640 1 --2- 172.21.15.6:0/276011050 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f33ec0ccad0 0x7f33ec14cbe0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f33d800d5f0 tx=0x7f33d800dab0 comp rx=0 tx=0).stop 2024-03-04T09:29:30.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.361+0000 7f33f3400640 1 -- 172.21.15.6:0/276011050 shutdown_connections 2024-03-04T09:29:30.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.361+0000 7f33f3400640 1 --2- 172.21.15.6:0/276011050 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f33cc06b4d0 0x7f33cc06d990 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:30.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.361+0000 7f33f3400640 1 --2- 172.21.15.6:0/276011050 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f33ec15e680 0x7f33ec1f3310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:30.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.361+0000 7f33f3400640 1 --2- 172.21.15.6:0/276011050 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f33ec0ccad0 0x7f33ec14cbe0 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:30.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.361+0000 7f33f3400640 1 --2- 172.21.15.6:0/276011050 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f33ec06da40 0x7f33ec0b3af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:30.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.361+0000 7f33f3400640 1 -- 172.21.15.6:0/276011050 >> 172.21.15.6:0/276011050 conn(0x7f33ec06b3d0 msgr2=0x7f33ec162100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:29:30.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.361+0000 7f33f3400640 1 -- 172.21.15.6:0/276011050 shutdown_connections 2024-03-04T09:29:30.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.361+0000 7f33f3400640 1 -- 172.21.15.6:0/276011050 wait complete. 2024-03-04T09:29:30.378 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-04T09:29:30.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.605+0000 7f7599000640 1 Processor -- start 2024-03-04T09:29:30.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.606+0000 7f7599000640 1 -- start start 2024-03-04T09:29:30.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.607+0000 7f7599000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f759415afa0 0x7f75940cf3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:30.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.607+0000 7f7599000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f7594070ce0 0x7f75940ccad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:30.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.607+0000 7f7599000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f75940cd010 0x7f759406c980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:30.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.607+0000 7f7599000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75940cfa10 con 0x7f75940cd010 2024-03-04T09:29:30.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.607+0000 7f7599000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7594071c60 con 0x7f7594070ce0 2024-03-04T09:29:30.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.607+0000 7f7599000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7594071f40 con 0x7f759415afa0 2024-03-04T09:29:30.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.607+0000 7f7592800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f75940cd010 0x7f759406c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:30.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.607+0000 7f7592800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f75940cd010 0x7f759406c980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:52888/0 (socket says 172.21.15.6:52888) 2024-03-04T09:29:30.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.607+0000 7f7592800640 1 -- 172.21.15.6:0/3542899420 learned_addr learned my addr 172.21.15.6:0/3542899420 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:29:30.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.608+0000 7f7591e00640 1 --2- 172.21.15.6:0/3542899420 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f759415afa0 0x7f75940cf3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:30.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.608+0000 7f7592800640 1 -- 172.21.15.6:0/3542899420 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f759415afa0 msgr2=0x7f75940cf3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:30.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.608+0000 7f7591400640 1 --2- 172.21.15.6:0/3542899420 >> v2:172.21.15.146:3300/0 conn(0x7f7594070ce0 0x7f75940ccad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:30.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.608+0000 7f7592800640 1 --2- 172.21.15.6:0/3542899420 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f759415afa0 0x7f75940cf3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:30.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.608+0000 7f7592800640 1 -- 172.21.15.6:0/3542899420 >> v2:172.21.15.146:3300/0 conn(0x7f7594070ce0 msgr2=0x7f75940ccad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:30.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.608+0000 7f7592800640 1 --2- 172.21.15.6:0/3542899420 >> v2:172.21.15.146:3300/0 conn(0x7f7594070ce0 0x7f75940ccad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:30.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.608+0000 7f7592800640 1 -- 172.21.15.6:0/3542899420 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7594072220 con 0x7f75940cd010 2024-03-04T09:29:30.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.608+0000 7f7591400640 1 --2- 172.21.15.6:0/3542899420 >> v2:172.21.15.146:3300/0 conn(0x7f7594070ce0 0x7f75940ccad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:29:30.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.608+0000 7f7591e00640 1 --2- 172.21.15.6:0/3542899420 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f759415afa0 0x7f75940cf3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:29:30.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.609+0000 7f7592800640 1 --2- 172.21.15.6:0/3542899420 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f75940cd010 0x7f759406c980 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f759406c050 tx=0x7f758c02b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6b286ae2a8a0921 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:30.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.609+0000 7f7590a00640 1 -- 172.21.15.6:0/3542899420 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f758c0317f0 con 0x7f75940cd010 2024-03-04T09:29:30.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.609+0000 7f7590a00640 1 -- 172.21.15.6:0/3542899420 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f758c037070 con 0x7f75940cd010 2024-03-04T09:29:30.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.609+0000 7f7590a00640 1 -- 172.21.15.6:0/3542899420 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f758c031bb0 con 0x7f75940cd010 2024-03-04T09:29:30.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.610+0000 7f7599000640 1 -- 172.21.15.6:0/3542899420 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f75940cd010 msgr2=0x7f759406c980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:30.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.610+0000 7f7599000640 1 --2- 172.21.15.6:0/3542899420 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f75940cd010 0x7f759406c980 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f759406c050 tx=0x7f758c02b890 comp rx=0 tx=0).stop 2024-03-04T09:29:30.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.610+0000 7f7599000640 1 -- 172.21.15.6:0/3542899420 shutdown_connections 2024-03-04T09:29:30.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.610+0000 7f7599000640 1 --2- 172.21.15.6:0/3542899420 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f75940cd010 0x7f759406c980 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:30.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.610+0000 7f7599000640 1 --2- 172.21.15.6:0/3542899420 >> v2:172.21.15.146:3300/0 conn(0x7f7594070ce0 0x7f75940ccad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:30.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.610+0000 7f7599000640 1 --2- 172.21.15.6:0/3542899420 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f759415afa0 0x7f75940cf3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:30.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.610+0000 7f7599000640 1 -- 172.21.15.6:0/3542899420 >> 172.21.15.6:0/3542899420 conn(0x7f759406b3d0 msgr2=0x7f759406b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:29:30.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.610+0000 7f7599000640 1 -- 172.21.15.6:0/3542899420 shutdown_connections 2024-03-04T09:29:30.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.610+0000 7f7599000640 1 -- 172.21.15.6:0/3542899420 wait complete. 2024-03-04T09:29:30.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.612+0000 7f7599000640 1 Processor -- start 2024-03-04T09:29:30.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.612+0000 7f7599000640 1 -- start start 2024-03-04T09:29:30.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.613+0000 7f7599000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7594070ce0 0x7f7594150890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:30.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.613+0000 7f7599000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f75940cd010 0x7f759409d060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:30.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.613+0000 7f7599000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f759415afa0 0x7f759409d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:30.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.613+0000 7f7599000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f759406cec0 con 0x7f7594070ce0 2024-03-04T09:29:30.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.613+0000 7f7599000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7594071140 con 0x7f75940cd010 2024-03-04T09:29:30.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.613+0000 7f7599000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f759406d040 con 0x7f759415afa0 2024-03-04T09:29:30.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.613+0000 7f7591e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7594070ce0 0x7f7594150890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:30.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.613+0000 7f7591e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7594070ce0 0x7f7594150890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:52890/0 (socket says 172.21.15.6:52890) 2024-03-04T09:29:30.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.613+0000 7f7591e00640 1 -- 172.21.15.6:0/901119618 learned_addr learned my addr 172.21.15.6:0/901119618 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:29:30.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.613+0000 7f7591400640 1 --2- 172.21.15.6:0/901119618 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f75940cd010 0x7f759409d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:30.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.614+0000 7f7591e00640 1 -- 172.21.15.6:0/901119618 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f759415afa0 msgr2=0x7f759409d5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:30.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.614+0000 7f7591e00640 1 --2- 172.21.15.6:0/901119618 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f759415afa0 0x7f759409d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:30.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.614+0000 7f7591e00640 1 -- 172.21.15.6:0/901119618 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f75940cd010 msgr2=0x7f759409d060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:30.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.614+0000 7f7591e00640 1 --2- 172.21.15.6:0/901119618 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f75940cd010 0x7f759409d060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:30.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.614+0000 7f7591e00640 1 -- 172.21.15.6:0/901119618 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75940af7c0 con 0x7f7594070ce0 2024-03-04T09:29:30.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.614+0000 7f7591400640 1 --2- 172.21.15.6:0/901119618 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f75940cd010 0x7f759409d060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:29:30.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.614+0000 7f7591e00640 1 --2- 172.21.15.6:0/901119618 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7594070ce0 0x7f7594150890 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f758400c7f0 tx=0x7f758400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:30.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.614+0000 7f758a000640 1 -- 172.21.15.6:0/901119618 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7584015070 con 0x7f7594070ce0 2024-03-04T09:29:30.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.614+0000 7f758a000640 1 -- 172.21.15.6:0/901119618 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7584007820 con 0x7f7594070ce0 2024-03-04T09:29:30.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.614+0000 7f758a000640 1 -- 172.21.15.6:0/901119618 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7584007bd0 con 0x7f7594070ce0 2024-03-04T09:29:30.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.614+0000 7f7599000640 1 -- 172.21.15.6:0/901119618 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75940afaa0 con 0x7f7594070ce0 2024-03-04T09:29:30.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.614+0000 7f7599000640 1 -- 172.21.15.6:0/901119618 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75940b0060 con 0x7f7594070ce0 2024-03-04T09:29:30.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.616+0000 7f7599000640 1 -- 172.21.15.6:0/901119618 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f759404f330 con 0x7f7594070ce0 2024-03-04T09:29:30.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.616+0000 7f758a000640 1 -- 172.21.15.6:0/901119618 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7584018070 con 0x7f7594070ce0 2024-03-04T09:29:30.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.618+0000 7f758a000640 1 --2- 172.21.15.6:0/901119618 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f757006b4d0 0x7f757006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:30.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.618+0000 7f758a000640 1 -- 172.21.15.6:0/901119618 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f758408a350 con 0x7f7594070ce0 2024-03-04T09:29:30.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.618+0000 7f7591400640 1 --2- 172.21.15.6:0/901119618 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f757006b4d0 0x7f757006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:30.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.620+0000 7f7591400640 1 --2- 172.21.15.6:0/901119618 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f757006b4d0 0x7f757006d990 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f75940b4eb0 tx=0x7f757c007530 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:30.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.622+0000 7f758a000640 1 -- 172.21.15.6:0/901119618 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7584059150 con 0x7f7594070ce0 2024-03-04T09:29:30.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.823+0000 7f7599000640 1 -- 172.21.15.6:0/901119618 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f759406ea20 con 0x7f7594070ce0 2024-03-04T09:29:30.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.824+0000 7f758a000640 1 -- 172.21.15.6:0/901119618 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v70) v1 ==== 100+0+11 (secure 0 0 0) 0x7f7584058a50 con 0x7f7594070ce0 2024-03-04T09:29:30.821 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 64 2024-03-04T09:29:30.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.826+0000 7f7599000640 1 -- 172.21.15.6:0/901119618 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f757006b4d0 msgr2=0x7f757006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:30.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.826+0000 7f7599000640 1 --2- 172.21.15.6:0/901119618 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f757006b4d0 0x7f757006d990 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f75940b4eb0 tx=0x7f757c007530 comp rx=0 tx=0).stop 2024-03-04T09:29:30.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.826+0000 7f7599000640 1 -- 172.21.15.6:0/901119618 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7594070ce0 msgr2=0x7f7594150890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:30.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.826+0000 7f7599000640 1 --2- 172.21.15.6:0/901119618 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7594070ce0 0x7f7594150890 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f758400c7f0 tx=0x7f758400ccb0 comp rx=0 tx=0).stop 2024-03-04T09:29:30.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.826+0000 7f7599000640 1 -- 172.21.15.6:0/901119618 shutdown_connections 2024-03-04T09:29:30.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.826+0000 7f7599000640 1 --2- 172.21.15.6:0/901119618 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f757006b4d0 0x7f757006d990 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:30.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.826+0000 7f7599000640 1 --2- 172.21.15.6:0/901119618 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f759415afa0 0x7f759409d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:30.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.826+0000 7f7599000640 1 --2- 172.21.15.6:0/901119618 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f75940cd010 0x7f759409d060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:30.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.826+0000 7f7599000640 1 --2- 172.21.15.6:0/901119618 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7594070ce0 0x7f7594150890 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:30.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.826+0000 7f7599000640 1 -- 172.21.15.6:0/901119618 >> 172.21.15.6:0/901119618 conn(0x7f759406b3d0 msgr2=0x7f75940ce340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:29:30.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.827+0000 7f7599000640 1 -- 172.21.15.6:0/901119618 shutdown_connections 2024-03-04T09:29:30.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:30.827+0000 7f7599000640 1 -- 172.21.15.6:0/901119618 wait complete. 2024-03-04T09:29:30.842 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:29:31.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.062+0000 7f914d400640 1 Processor -- start 2024-03-04T09:29:31.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.062+0000 7f914d400640 1 -- start start 2024-03-04T09:29:31.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.064+0000 7f914d400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9148070f10 0x7f91480cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:31.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.064+0000 7f914d400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f91480cce90 0x7f914815c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:31.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.064+0000 7f914d400640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91480714d0 con 0x7f9148145300 2024-03-04T09:29:31.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.064+0000 7f914d400640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91480cd300 con 0x7f91480cce90 2024-03-04T09:29:31.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.064+0000 7f914d400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91480717b0 con 0x7f9148070f10 2024-03-04T09:29:31.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.064+0000 7f9147e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f9148145300 0x7f91480b49a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:42866/0 (socket says 172.21.15.6:42866) 2024-03-04T09:29:31.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.064+0000 7f9147e00640 1 -- 172.21.15.6:0/527054989 learned_addr learned my addr 172.21.15.6:0/527054989 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:29:31.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.065+0000 7f914ca00640 1 --2- 172.21.15.6:0/527054989 >> v2:172.21.15.146:3300/0 conn(0x7f91480cce90 0x7f914815c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:31.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.065+0000 7f9147400640 1 -- 172.21.15.6:0/527054989 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3633354553 0 0) 0x7f91480714d0 con 0x7f9148145300 2024-03-04T09:29:31.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.065+0000 7f913f400640 1 --2- 172.21.15.6:0/527054989 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9148070f10 0x7f91480cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:31.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.065+0000 7f9147400640 1 -- 172.21.15.6:0/527054989 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f912c003520 con 0x7f9148145300 2024-03-04T09:29:31.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.065+0000 7f9147400640 1 -- 172.21.15.6:0/527054989 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3697737792 0 0) 0x7f912c003520 con 0x7f9148145300 2024-03-04T09:29:31.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.065+0000 7f9147400640 1 -- 172.21.15.6:0/527054989 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9148070f10 msgr2=0x7f91480cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:31.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.065+0000 7f9147400640 1 --2- 172.21.15.6:0/527054989 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9148070f10 0x7f91480cc950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:31.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.065+0000 7f9147400640 1 -- 172.21.15.6:0/527054989 >> v2:172.21.15.146:3300/0 conn(0x7f91480cce90 msgr2=0x7f914815c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:31.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.065+0000 7f9147400640 1 --2- 172.21.15.6:0/527054989 >> v2:172.21.15.146:3300/0 conn(0x7f91480cce90 0x7f914815c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:31.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.065+0000 7f9147400640 1 -- 172.21.15.6:0/527054989 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9148071a90 con 0x7f9148145300 2024-03-04T09:29:31.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.065+0000 7f9147400640 1 -- 172.21.15.6:0/527054989 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7f9138001670 con 0x7f9148145300 2024-03-04T09:29:31.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.065+0000 7f9147400640 1 -- 172.21.15.6:0/527054989 >> v1:172.21.15.6:6789/0 conn(0x7f9148145300 legacy=0x7f91480b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:31.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.065+0000 7f9147400640 1 --2- 172.21.15.6:0/527054989 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f912c0039b0 0x7f912c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:31.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.065+0000 7f9147400640 1 --2- 172.21.15.6:0/527054989 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9148070f10 0x7f912c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:31.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.065+0000 7f9147400640 1 --2- 172.21.15.6:0/527054989 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f912c004d40 0x7f912c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:31.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.065+0000 7f9147400640 1 -- 172.21.15.6:0/527054989 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91480cd300 con 0x7f912c0039b0 2024-03-04T09:29:31.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.065+0000 7f9147400640 1 -- 172.21.15.6:0/527054989 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91480717b0 con 0x7f912c004d40 2024-03-04T09:29:31.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.065+0000 7f9147400640 1 -- 172.21.15.6:0/527054989 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f912c00b9c0 con 0x7f9148070f10 2024-03-04T09:29:31.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.066+0000 7f9147e00640 1 --2- 172.21.15.6:0/527054989 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f912c0039b0 0x7f912c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:31.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.066+0000 7f913f400640 1 --2- 172.21.15.6:0/527054989 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9148070f10 0x7f912c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:31.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.066+0000 7f914ca00640 1 --2- 172.21.15.6:0/527054989 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f912c004d40 0x7f912c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:31.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.066+0000 7f9147e00640 1 -- 172.21.15.6:0/527054989 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9148070f10 msgr2=0x7f912c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:31.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.066+0000 7f9147e00640 1 --2- 172.21.15.6:0/527054989 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9148070f10 0x7f912c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:31.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.066+0000 7f9147e00640 1 -- 172.21.15.6:0/527054989 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f912c004d40 msgr2=0x7f912c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:31.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.066+0000 7f9147e00640 1 --2- 172.21.15.6:0/527054989 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f912c004d40 0x7f912c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:31.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.066+0000 7f9147e00640 1 -- 172.21.15.6:0/527054989 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f912c003520 con 0x7f912c0039b0 2024-03-04T09:29:31.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.067+0000 7f9147e00640 1 --2- 172.21.15.6:0/527054989 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f912c0039b0 0x7f912c003dd0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f913800c210 tx=0x7f912c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16c352f887eb7e55 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:31.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.068+0000 7f9147400640 1 -- 172.21.15.6:0/527054989 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9138007d50 con 0x7f912c0039b0 2024-03-04T09:29:31.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.068+0000 7f9147400640 1 -- 172.21.15.6:0/527054989 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9138030070 con 0x7f912c0039b0 2024-03-04T09:29:31.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.068+0000 7f9147400640 1 -- 172.21.15.6:0/527054989 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f913802cd20 con 0x7f912c0039b0 2024-03-04T09:29:31.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.068+0000 7f914d400640 1 -- 172.21.15.6:0/527054989 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f912c0039b0 msgr2=0x7f912c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:31.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.068+0000 7f914d400640 1 --2- 172.21.15.6:0/527054989 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f912c0039b0 0x7f912c003dd0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f913800c210 tx=0x7f912c005830 comp rx=0 tx=0).stop 2024-03-04T09:29:31.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.068+0000 7f914ca00640 1 -- 172.21.15.6:0/527054989 reap_dead start 2024-03-04T09:29:31.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.068+0000 7f914d400640 1 -- 172.21.15.6:0/527054989 shutdown_connections 2024-03-04T09:29:31.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.068+0000 7f914d400640 1 -- 172.21.15.6:0/527054989 >> 172.21.15.6:0/527054989 conn(0x7f914806b3d0 msgr2=0x7f914806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:29:31.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.068+0000 7f914d400640 1 -- 172.21.15.6:0/527054989 shutdown_connections 2024-03-04T09:29:31.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.068+0000 7f914d400640 1 -- 172.21.15.6:0/527054989 wait complete. 2024-03-04T09:29:31.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.070+0000 7f914d400640 1 Processor -- start 2024-03-04T09:29:31.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.070+0000 7f914d400640 1 -- start start 2024-03-04T09:29:31.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.071+0000 7f914d400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f912c0039b0 0x7f9148141120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:31.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.071+0000 7f914d400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f914813fae0 0x7f914813def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:31.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.071+0000 7f914d400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f914806c7c0 0x7f914806cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:31.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.071+0000 7f914d400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91480cc950 con 0x7f914813fae0 2024-03-04T09:29:31.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.071+0000 7f914d400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91481456e0 con 0x7f914806c7c0 2024-03-04T09:29:31.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.071+0000 7f914d400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91480ccad0 con 0x7f912c0039b0 2024-03-04T09:29:31.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.071+0000 7f913f400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f914813fae0 0x7f914813def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:31.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.072+0000 7f913f400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f914813fae0 0x7f914813def0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:52902/0 (socket says 172.21.15.6:52902) 2024-03-04T09:29:31.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.072+0000 7f913f400640 1 -- 172.21.15.6:0/4246427282 learned_addr learned my addr 172.21.15.6:0/4246427282 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:29:31.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.072+0000 7f9147e00640 1 --2- 172.21.15.6:0/4246427282 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f912c0039b0 0x7f9148141120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:31.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.072+0000 7f913f400640 1 -- 172.21.15.6:0/4246427282 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f912c0039b0 msgr2=0x7f9148141120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:31.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.072+0000 7f913f400640 1 --2- 172.21.15.6:0/4246427282 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f912c0039b0 0x7f9148141120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:31.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.072+0000 7f914ca00640 1 --2- 172.21.15.6:0/4246427282 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f914806c7c0 0x7f914806cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:31.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.072+0000 7f913f400640 1 -- 172.21.15.6:0/4246427282 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f914806c7c0 msgr2=0x7f914806cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:31.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.072+0000 7f913f400640 1 --2- 172.21.15.6:0/4246427282 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f914806c7c0 0x7f914806cc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:31.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.072+0000 7f913f400640 1 -- 172.21.15.6:0/4246427282 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f914813e430 con 0x7f914813fae0 2024-03-04T09:29:31.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.072+0000 7f9147e00640 1 --2- 172.21.15.6:0/4246427282 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f912c0039b0 0x7f9148141120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:29:31.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.072+0000 7f913f400640 1 --2- 172.21.15.6:0/4246427282 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f914813fae0 0x7f914813def0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f913000f5e0 tx=0x7f913000faa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:31.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.073+0000 7f9144c00640 1 -- 172.21.15.6:0/4246427282 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9130013810 con 0x7f914813fae0 2024-03-04T09:29:31.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.073+0000 7f9144c00640 1 -- 172.21.15.6:0/4246427282 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f913001a070 con 0x7f914813fae0 2024-03-04T09:29:31.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.073+0000 7f9144c00640 1 -- 172.21.15.6:0/4246427282 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9130013bb0 con 0x7f914813fae0 2024-03-04T09:29:31.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.073+0000 7f914d400640 1 -- 172.21.15.6:0/4246427282 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9148140080 con 0x7f914813fae0 2024-03-04T09:29:31.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.073+0000 7f914d400640 1 -- 172.21.15.6:0/4246427282 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9148140640 con 0x7f914813fae0 2024-03-04T09:29:31.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.074+0000 7f914d400640 1 -- 172.21.15.6:0/4246427282 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91480b1560 con 0x7f914813fae0 2024-03-04T09:29:31.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.075+0000 7f9144c00640 1 -- 172.21.15.6:0/4246427282 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9130012070 con 0x7f914813fae0 2024-03-04T09:29:31.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.076+0000 7f9144c00640 1 --2- 172.21.15.6:0/4246427282 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f912406b4d0 0x7f912406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:31.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.076+0000 7f9144c00640 1 -- 172.21.15.6:0/4246427282 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f913008a320 con 0x7f914813fae0 2024-03-04T09:29:31.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.077+0000 7f9147e00640 1 --2- 172.21.15.6:0/4246427282 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f912406b4d0 0x7f912406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:31.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.078+0000 7f9147e00640 1 --2- 172.21.15.6:0/4246427282 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f912406b4d0 0x7f912406d990 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f9138034000 tx=0x7f9138007000 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:31.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.081+0000 7f9144c00640 1 -- 172.21.15.6:0/4246427282 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f91480b1560 con 0x7f914813fae0 2024-03-04T09:29:31.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.324+0000 7f914d400640 1 -- 172.21.15.6:0/4246427282 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f914804f330 con 0x7f914813fae0 2024-03-04T09:29:31.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.325+0000 7f9144c00640 1 -- 172.21.15.6:0/4246427282 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+4979 (secure 0 0 0) 0x7f913001c310 con 0x7f914813fae0 2024-03-04T09:29:31.322 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:29:31.322 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":42,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5228,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/1891254736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1891254736},{"type":"v1","addr":"172.21.15.6:6834","nonce":1891254736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":5578,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/1781219843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":1781219843},{"type":"v1","addr":"172.21.15.146:6835","nonce":1781219843}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":5587,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/1479180802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":1479180802},{"type":"v1","addr":"172.21.15.6:6835","nonce":1479180802}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:28:50.708467+0000","modified":"2024-03-04T09:28:53.801285+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5590},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5590":{"gid":5590,"name":"d","rank":0,"incarnation":38,"state":"up:active","state_seq":3,"addr":"172.21.15.146:6837/3919939129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":3919939129},{"type":"v1","addr":"172.21.15.146:6837","nonce":3919939129}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5590,"qdb_cluster":[5590]},"id":5}]} 2024-03-04T09:29:31.323 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 42 2024-03-04T09:29:31.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.327+0000 7f914d400640 1 -- 172.21.15.6:0/4246427282 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f912406b4d0 msgr2=0x7f912406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:31.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.327+0000 7f914d400640 1 --2- 172.21.15.6:0/4246427282 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f912406b4d0 0x7f912406d990 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f9138034000 tx=0x7f9138007000 comp rx=0 tx=0).stop 2024-03-04T09:29:31.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.327+0000 7f914d400640 1 -- 172.21.15.6:0/4246427282 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f914813fae0 msgr2=0x7f914813def0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:31.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.327+0000 7f914d400640 1 --2- 172.21.15.6:0/4246427282 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f914813fae0 0x7f914813def0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f913000f5e0 tx=0x7f913000faa0 comp rx=0 tx=0).stop 2024-03-04T09:29:31.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.328+0000 7f914d400640 1 -- 172.21.15.6:0/4246427282 shutdown_connections 2024-03-04T09:29:31.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.328+0000 7f914d400640 1 --2- 172.21.15.6:0/4246427282 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f914806c7c0 0x7f914806cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:31.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.328+0000 7f914d400640 1 --2- 172.21.15.6:0/4246427282 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f914813fae0 0x7f914813def0 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:31.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.328+0000 7f914d400640 1 --2- 172.21.15.6:0/4246427282 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f912406b4d0 0x7f912406d990 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:31.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.328+0000 7f914d400640 1 --2- 172.21.15.6:0/4246427282 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f912c0039b0 0x7f9148141120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:31.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.328+0000 7f914d400640 1 -- 172.21.15.6:0/4246427282 >> 172.21.15.6:0/4246427282 conn(0x7f914806b3d0 msgr2=0x7f91480af750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:29:31.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.328+0000 7f914d400640 1 -- 172.21.15.6:0/4246427282 shutdown_connections 2024-03-04T09:29:31.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.328+0000 7f914d400640 1 -- 172.21.15.6:0/4246427282 wait complete. 2024-03-04T09:29:31.344 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T09:29:31.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.560+0000 7fb48e800640 1 Processor -- start 2024-03-04T09:29:31.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.560+0000 7fb48e800640 1 -- start start 2024-03-04T09:29:31.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.562+0000 7fb48e800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb48815e0b0 0x7fb4880cf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:31.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.562+0000 7fb48e800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fb4880d0e50 0x7fb4880b44d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:31.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.562+0000 7fb48e800640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4880b3710 con 0x7fb4880d4a10 2024-03-04T09:29:31.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.562+0000 7fb48e800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4880b09e0 con 0x7fb4880d0e50 2024-03-04T09:29:31.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.562+0000 7fb48e800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4880b39f0 con 0x7fb48815e0b0 2024-03-04T09:29:31.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.562+0000 7fb48de00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fb4880d0e50 0x7fb4880b44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:31.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.562+0000 7fb48de00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fb4880d0e50 0x7fb4880b44d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:52904/0 (socket says 172.21.15.6:52904) 2024-03-04T09:29:31.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.563+0000 7fb48de00640 1 -- 172.21.15.6:0/4071515752 learned_addr learned my addr 172.21.15.6:0/4071515752 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:29:31.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.563+0000 7fb48d400640 1 --2- 172.21.15.6:0/4071515752 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb48815e0b0 0x7fb4880cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:31.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.563+0000 7fb48de00640 1 -- 172.21.15.6:0/4071515752 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb48815e0b0 msgr2=0x7fb4880cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:31.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.563+0000 7fb48de00640 1 --2- 172.21.15.6:0/4071515752 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb48815e0b0 0x7fb4880cf310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:31.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.563+0000 7fb48de00640 1 -- 172.21.15.6:0/4071515752 >> v1:172.21.15.146:6789/0 conn(0x7fb4880d4a10 legacy=0x7fb4880d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:31.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.563+0000 7fb48de00640 1 -- 172.21.15.6:0/4071515752 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4880b3cd0 con 0x7fb4880d0e50 2024-03-04T09:29:31.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.564+0000 7fb48de00640 1 --2- 172.21.15.6:0/4071515752 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb4880d0e50 0x7fb4880b44d0 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7fb480009870 tx=0x7fb48002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3baaeb37b602687 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:31.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.564+0000 7fb487e00640 1 -- 172.21.15.6:0/4071515752 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb480034070 con 0x7fb4880d0e50 2024-03-04T09:29:31.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.565+0000 7fb487e00640 1 -- 172.21.15.6:0/4071515752 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb480030840 con 0x7fb4880d0e50 2024-03-04T09:29:31.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.565+0000 7fb487e00640 1 -- 172.21.15.6:0/4071515752 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb480030c10 con 0x7fb4880d0e50 2024-03-04T09:29:31.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.565+0000 7fb48e800640 1 -- 172.21.15.6:0/4071515752 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb4880d0e50 msgr2=0x7fb4880b44d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:31.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.565+0000 7fb48e800640 1 --2- 172.21.15.6:0/4071515752 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb4880d0e50 0x7fb4880b44d0 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7fb480009870 tx=0x7fb48002b980 comp rx=0 tx=0).stop 2024-03-04T09:29:31.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.565+0000 7fb48e800640 1 -- 172.21.15.6:0/4071515752 shutdown_connections 2024-03-04T09:29:31.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.565+0000 7fb48e800640 1 --2- 172.21.15.6:0/4071515752 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb4880d0e50 0x7fb4880b44d0 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:31.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.565+0000 7fb48e800640 1 --2- 172.21.15.6:0/4071515752 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb48815e0b0 0x7fb4880cf310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:31.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.565+0000 7fb48e800640 1 -- 172.21.15.6:0/4071515752 >> 172.21.15.6:0/4071515752 conn(0x7fb4880d3d30 msgr2=0x7fb4880d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:29:31.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.565+0000 7fb48e800640 1 -- 172.21.15.6:0/4071515752 shutdown_connections 2024-03-04T09:29:31.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.565+0000 7fb48e800640 1 -- 172.21.15.6:0/4071515752 wait complete. 2024-03-04T09:29:31.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.567+0000 7fb48e800640 1 Processor -- start 2024-03-04T09:29:31.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.567+0000 7fb48e800640 1 -- start start 2024-03-04T09:29:31.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.568+0000 7fb48e800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb4880d0e50 0x7fb4880cbe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:31.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.568+0000 7fb48e800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb4880d4a10 0x7fb4880cc3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:31.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.568+0000 7fb48e800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb48815e0b0 0x7fb4881f3ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:31.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.568+0000 7fb48e800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4880cfc60 con 0x7fb48815e0b0 2024-03-04T09:29:31.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.568+0000 7fb48e800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb48815e490 con 0x7fb4880d4a10 2024-03-04T09:29:31.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.568+0000 7fb48e800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4880cf850 con 0x7fb4880d0e50 2024-03-04T09:29:31.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.568+0000 7fb48de00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb48815e0b0 0x7fb4881f3ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:31.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.568+0000 7fb48de00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb48815e0b0 0x7fb4881f3ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:52906/0 (socket says 172.21.15.6:52906) 2024-03-04T09:29:31.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.568+0000 7fb48de00640 1 -- 172.21.15.6:0/2792704681 learned_addr learned my addr 172.21.15.6:0/2792704681 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:29:31.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.569+0000 7fb48ca00640 1 --2- 172.21.15.6:0/2792704681 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb4880d4a10 0x7fb4880cc3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:31.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.569+0000 7fb48d400640 1 --2- 172.21.15.6:0/2792704681 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb4880d0e50 0x7fb4880cbe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:31.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.569+0000 7fb48de00640 1 -- 172.21.15.6:0/2792704681 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb4880d0e50 msgr2=0x7fb4880cbe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:31.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.569+0000 7fb48de00640 1 --2- 172.21.15.6:0/2792704681 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb4880d0e50 0x7fb4880cbe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:31.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.569+0000 7fb48de00640 1 -- 172.21.15.6:0/2792704681 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb4880d4a10 msgr2=0x7fb4880cc3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:31.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.569+0000 7fb48de00640 1 --2- 172.21.15.6:0/2792704681 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb4880d4a10 0x7fb4880cc3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:31.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.569+0000 7fb48de00640 1 -- 172.21.15.6:0/2792704681 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4881f4850 con 0x7fb48815e0b0 2024-03-04T09:29:31.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.569+0000 7fb48ca00640 1 --2- 172.21.15.6:0/2792704681 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb4880d4a10 0x7fb4880cc3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:29:31.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.569+0000 7fb48d400640 1 --2- 172.21.15.6:0/2792704681 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb4880d0e50 0x7fb4880cbe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:29:31.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.569+0000 7fb48de00640 1 --2- 172.21.15.6:0/2792704681 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb48815e0b0 0x7fb4881f3ad0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7fb480009870 tx=0x7fb480004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:31.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.570+0000 7fb485600640 1 -- 172.21.15.6:0/2792704681 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb480048070 con 0x7fb48815e0b0 2024-03-04T09:29:31.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.570+0000 7fb485600640 1 -- 172.21.15.6:0/2792704681 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb480030640 con 0x7fb48815e0b0 2024-03-04T09:29:31.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.570+0000 7fb485600640 1 -- 172.21.15.6:0/2792704681 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb480034040 con 0x7fb48815e0b0 2024-03-04T09:29:31.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.570+0000 7fb48e800640 1 -- 172.21.15.6:0/2792704681 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb488148f90 con 0x7fb48815e0b0 2024-03-04T09:29:31.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.570+0000 7fb48e800640 1 -- 172.21.15.6:0/2792704681 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb488149550 con 0x7fb48815e0b0 2024-03-04T09:29:31.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.571+0000 7fb48e800640 1 -- 172.21.15.6:0/2792704681 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb48804f330 con 0x7fb48815e0b0 2024-03-04T09:29:31.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.572+0000 7fb485600640 1 -- 172.21.15.6:0/2792704681 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb480037070 con 0x7fb48815e0b0 2024-03-04T09:29:31.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.573+0000 7fb485600640 1 --2- 172.21.15.6:0/2792704681 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb46806b500 0x7fb46806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:31.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.573+0000 7fb485600640 1 -- 172.21.15.6:0/2792704681 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fb4800abf20 con 0x7fb48815e0b0 2024-03-04T09:29:31.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.573+0000 7fb48d400640 1 --2- 172.21.15.6:0/2792704681 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb46806b500 0x7fb46806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:31.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.575+0000 7fb48d400640 1 --2- 172.21.15.6:0/2792704681 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb46806b500 0x7fb46806d9c0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fb478006fd0 tx=0x7fb478008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:31.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.577+0000 7fb485600640 1 -- 172.21.15.6:0/2792704681 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb48007ada0 con 0x7fb48815e0b0 2024-03-04T09:29:31.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.776+0000 7fb48e800640 1 -- 172.21.15.6:0/2792704681 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb488161640 con 0x7fb48815e0b0 2024-03-04T09:29:31.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.778+0000 7fb485600640 1 -- 172.21.15.6:0/2792704681 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v70) v1 ==== 74+0+16813 (secure 0 0 0) 0x7fb48007a6a0 con 0x7fb48815e0b0 2024-03-04T09:29:31.775 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:29:31.776 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":70,"fsid":"edce0578-5ae4-4402-b45a-552bdef84d7c","created":"2024-03-04T09:24:54.523780+0000","modified":"2024-03-04T09:29:02.333052+0000","last_up_change":"2024-03-04T09:25:07.089049+0000","last_in_change":"2024-03-04T09:24:57.982077+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T09:25:07.935198+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-03-04T09:28:48.627427+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-03-04T09:28:49.739043+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a96574f-81f1-4bc6-8463-757520b84a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6825","nonce":1364449401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6827","nonce":1364449401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6831","nonce":1364449401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6829","nonce":1364449401}]},"public_addr":"172.21.15.6:6825/1364449401","cluster_addr":"172.21.15.6:6827/1364449401","heartbeat_back_addr":"172.21.15.6:6831/1364449401","heartbeat_front_addr":"172.21.15.6:6829/1364449401","state":["exists","up"]},{"osd":1,"uuid":"7d79b9c1-6b87-476a-bc36-7db5ee64cd90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6809","nonce":2494103078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6811","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6813","nonce":2494103078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6821","nonce":2494103078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6817","nonce":2494103078}]},"public_addr":"172.21.15.6:6809/2494103078","cluster_addr":"172.21.15.6:6813/2494103078","heartbeat_back_addr":"172.21.15.6:6821/2494103078","heartbeat_front_addr":"172.21.15.6:6817/2494103078","state":["exists","up"]},{"osd":2,"uuid":"492331b9-4ee9-40b6-bacd-85d44cd6c45a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6812","nonce":1500066681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6816","nonce":1500066681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6824","nonce":1500066681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6820","nonce":1500066681}]},"public_addr":"172.21.15.6:6812/1500066681","cluster_addr":"172.21.15.6:6816/1500066681","heartbeat_back_addr":"172.21.15.6:6824/1500066681","heartbeat_front_addr":"172.21.15.6:6820/1500066681","state":["exists","up"]},{"osd":3,"uuid":"21bb2efe-9ba7-4bdc-bc31-180cd1102e21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6801","nonce":1558806123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6803","nonce":1558806123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6807","nonce":1558806123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6805","nonce":1558806123}]},"public_addr":"172.21.15.6:6801/1558806123","cluster_addr":"172.21.15.6:6803/1558806123","heartbeat_back_addr":"172.21.15.6:6807/1558806123","heartbeat_front_addr":"172.21.15.6:6805/1558806123","state":["exists","up"]},{"osd":4,"uuid":"6e2c4b71-19d9-4737-83d9-558705d8644a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6801","nonce":1526135780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6803","nonce":1526135780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6807","nonce":1526135780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6805","nonce":1526135780}]},"public_addr":"172.21.15.146:6801/1526135780","cluster_addr":"172.21.15.146:6803/1526135780","heartbeat_back_addr":"172.21.15.146:6807/1526135780","heartbeat_front_addr":"172.21.15.146:6805/1526135780","state":["exists","up"]},{"osd":5,"uuid":"fb69c525-f1be-4b7d-ab32-6c29b0d6fddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6809","nonce":460262507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6811","nonce":460262507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6815","nonce":460262507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6813","nonce":460262507}]},"public_addr":"172.21.15.146:6809/460262507","cluster_addr":"172.21.15.146:6811/460262507","heartbeat_back_addr":"172.21.15.146:6815/460262507","heartbeat_front_addr":"172.21.15.146:6813/460262507","state":["exists","up"]},{"osd":6,"uuid":"c32e7fed-370d-4401-8ce0-6a4335b0d168","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6823","nonce":1392897493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6827","nonce":1392897493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6831","nonce":1392897493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6829","nonce":1392897493}]},"public_addr":"172.21.15.146:6823/1392897493","cluster_addr":"172.21.15.146:6827/1392897493","heartbeat_back_addr":"172.21.15.146:6831/1392897493","heartbeat_front_addr":"172.21.15.146:6829/1392897493","state":["exists","up"]},{"osd":7,"uuid":"75a79b44-fa5f-426d-a7d3-71f21f78835c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6817","nonce":3268323156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6819","nonce":3268323156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6825","nonce":3268323156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6822","nonce":3268323156}]},"public_addr":"172.21.15.146:6817/3268323156","cluster_addr":"172.21.15.146:6819/3268323156","heartbeat_back_addr":"172.21.15.146:6825/3268323156","heartbeat_front_addr":"172.21.15.146:6822/3268323156","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.310437+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.295850+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.294674+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.526855+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.300023+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.595160+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.346046+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.558741+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"11.8","mappings":[{"from":6,"to":5}]},{"pgid":"11.10","mappings":[{"from":6,"to":1}]},{"pgid":"11.14","mappings":[{"from":6,"to":0}]},{"pgid":"11.3e","mappings":[{"from":6,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T09:29:31.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.781+0000 7fb48e800640 1 -- 172.21.15.6:0/2792704681 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb46806b500 msgr2=0x7fb46806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:31.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.781+0000 7fb48e800640 1 --2- 172.21.15.6:0/2792704681 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb46806b500 0x7fb46806d9c0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fb478006fd0 tx=0x7fb478008040 comp rx=0 tx=0).stop 2024-03-04T09:29:31.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.781+0000 7fb48e800640 1 -- 172.21.15.6:0/2792704681 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb48815e0b0 msgr2=0x7fb4881f3ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:31.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.781+0000 7fb48e800640 1 --2- 172.21.15.6:0/2792704681 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb48815e0b0 0x7fb4881f3ad0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7fb480009870 tx=0x7fb480004730 comp rx=0 tx=0).stop 2024-03-04T09:29:31.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.781+0000 7fb48e800640 1 -- 172.21.15.6:0/2792704681 shutdown_connections 2024-03-04T09:29:31.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.781+0000 7fb48e800640 1 --2- 172.21.15.6:0/2792704681 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb48815e0b0 0x7fb4881f3ad0 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:31.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.781+0000 7fb48e800640 1 --2- 172.21.15.6:0/2792704681 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb4880d4a10 0x7fb4880cc3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:31.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.781+0000 7fb48e800640 1 --2- 172.21.15.6:0/2792704681 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb46806b500 0x7fb46806d9c0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:31.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.781+0000 7fb48e800640 1 --2- 172.21.15.6:0/2792704681 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb4880d0e50 0x7fb4880cbe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:31.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.781+0000 7fb48e800640 1 -- 172.21.15.6:0/2792704681 >> 172.21.15.6:0/2792704681 conn(0x7fb4880d3d30 msgr2=0x7fb4880ce7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:29:31.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.781+0000 7fb48e800640 1 -- 172.21.15.6:0/2792704681 shutdown_connections 2024-03-04T09:29:31.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:31.781+0000 7fb48e800640 1 -- 172.21.15.6:0/2792704681 wait complete. 2024-03-04T09:29:31.797 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-04T09:29:31.798 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:29:32.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.018+0000 7f5fcf000640 1 Processor -- start 2024-03-04T09:29:32.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.019+0000 7f5fcf000640 1 -- start start 2024-03-04T09:29:32.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.020+0000 7f5fcf000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5fc80d4a10 0x7f5fc80d0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:32.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.020+0000 7f5fcf000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fc80b3710 con 0x7f5fc815e0b0 2024-03-04T09:29:32.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.020+0000 7f5fcf000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fc80b39f0 con 0x7f5fc80d0c80 2024-03-04T09:29:32.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.020+0000 7f5fcf000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fc80b09e0 con 0x7f5fc80d4a10 2024-03-04T09:29:32.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.021+0000 7f5fcdc00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f5fc815e0b0 0x7f5fc80cf310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:42868/0 (socket says 172.21.15.6:42868) 2024-03-04T09:29:32.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.021+0000 7f5fcdc00640 1 -- 172.21.15.6:0/2153922185 learned_addr learned my addr 172.21.15.6:0/2153922185 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:29:32.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.021+0000 7f5fcd200640 1 --2- 172.21.15.6:0/2153922185 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5fc80d4a10 0x7f5fc80d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:32.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.021+0000 7f5fc7e00640 1 -- 172.21.15.6:0/2153922185 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4062714578 0 0) 0x7f5fc80b3710 con 0x7f5fc815e0b0 2024-03-04T09:29:32.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.021+0000 7f5fc7e00640 1 -- 172.21.15.6:0/2153922185 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fb0003520 con 0x7f5fc815e0b0 2024-03-04T09:29:32.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.022+0000 7f5fcd200640 1 -- 172.21.15.6:0/2153922185 >> v1:172.21.15.146:6789/0 conn(0x7f5fc80d0c80 legacy=0x7f5fc80b44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:32.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.022+0000 7f5fc7e00640 1 -- 172.21.15.6:0/2153922185 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1728998863 0 0) 0x7f5fb0003520 con 0x7f5fc815e0b0 2024-03-04T09:29:32.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.022+0000 7f5fcd200640 1 -- 172.21.15.6:0/2153922185 >> v1:172.21.15.6:6789/0 conn(0x7f5fc815e0b0 legacy=0x7f5fc80cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:32.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.022+0000 7f5fcd200640 1 -- 172.21.15.6:0/2153922185 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fc80b3cd0 con 0x7f5fc80d4a10 2024-03-04T09:29:32.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.023+0000 7f5fcd200640 1 --2- 172.21.15.6:0/2153922185 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5fc80d4a10 0x7f5fc80d0740 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f5fb40097d0 tx=0x7f5fb402b800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b94d5f16a8090cea server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:32.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.024+0000 7f5fc7e00640 1 -- 172.21.15.6:0/2153922185 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fb4037070 con 0x7f5fc80d4a10 2024-03-04T09:29:32.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.024+0000 7f5fc7e00640 1 -- 172.21.15.6:0/2153922185 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fb4030510 con 0x7f5fc80d4a10 2024-03-04T09:29:32.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.024+0000 7f5fc7e00640 1 -- 172.21.15.6:0/2153922185 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fbc0019f0 con 0x7f5fc80d4a10 2024-03-04T09:29:32.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.024+0000 7f5fcf000640 1 -- 172.21.15.6:0/2153922185 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5fc80d4a10 msgr2=0x7f5fc80d0740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:32.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.024+0000 7f5fcf000640 1 --2- 172.21.15.6:0/2153922185 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5fc80d4a10 0x7f5fc80d0740 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f5fb40097d0 tx=0x7f5fb402b800 comp rx=0 tx=0).stop 2024-03-04T09:29:32.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.025+0000 7f5fcf000640 1 -- 172.21.15.6:0/2153922185 shutdown_connections 2024-03-04T09:29:32.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.025+0000 7f5fcf000640 1 --2- 172.21.15.6:0/2153922185 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5fc80d4a10 0x7f5fc80d0740 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:32.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.025+0000 7f5fcf000640 1 -- 172.21.15.6:0/2153922185 >> 172.21.15.6:0/2153922185 conn(0x7f5fc80d3d30 msgr2=0x7f5fc80d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:29:32.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.025+0000 7f5fcf000640 1 -- 172.21.15.6:0/2153922185 shutdown_connections 2024-03-04T09:29:32.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.025+0000 7f5fcf000640 1 -- 172.21.15.6:0/2153922185 wait complete. 2024-03-04T09:29:32.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.026+0000 7f5fcf000640 1 Processor -- start 2024-03-04T09:29:32.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.027+0000 7f5fcf000640 1 -- start start 2024-03-04T09:29:32.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.028+0000 7f5fcf000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5fc80d0c80 0x7f5fc80ccd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:32.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.028+0000 7f5fcf000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5fc80d4a10 0x7f5fc80cae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:32.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.028+0000 7f5fcf000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5fc815e0b0 0x7f5fc80b68f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:32.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.028+0000 7f5fcf000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fc815e490 con 0x7f5fc80d0c80 2024-03-04T09:29:32.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.028+0000 7f5fcf000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fc80cfa20 con 0x7f5fc815e0b0 2024-03-04T09:29:32.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.028+0000 7f5fcf000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fc80cfc60 con 0x7f5fc80d4a10 2024-03-04T09:29:32.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.028+0000 7f5fcdc00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5fc80d0c80 0x7f5fc80ccd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:32.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.028+0000 7f5fcdc00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5fc80d0c80 0x7f5fc80ccd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:52908/0 (socket says 172.21.15.6:52908) 2024-03-04T09:29:32.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.028+0000 7f5fcdc00640 1 -- 172.21.15.6:0/521395432 learned_addr learned my addr 172.21.15.6:0/521395432 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:29:32.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.028+0000 7f5fcd200640 1 --2- 172.21.15.6:0/521395432 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5fc80d4a10 0x7f5fc80cae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:32.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.028+0000 7f5fce600640 1 --2- 172.21.15.6:0/521395432 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5fc815e0b0 0x7f5fc80b68f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:32.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.028+0000 7f5fcdc00640 1 -- 172.21.15.6:0/521395432 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5fc80d4a10 msgr2=0x7f5fc80cae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:32.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.028+0000 7f5fcdc00640 1 --2- 172.21.15.6:0/521395432 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5fc80d4a10 0x7f5fc80cae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:32.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.028+0000 7f5fcdc00640 1 -- 172.21.15.6:0/521395432 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5fc815e0b0 msgr2=0x7f5fc80b68f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:32.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.028+0000 7f5fcdc00640 1 --2- 172.21.15.6:0/521395432 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5fc815e0b0 0x7f5fc80b68f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:32.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.028+0000 7f5fcdc00640 1 -- 172.21.15.6:0/521395432 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fc80cb8c0 con 0x7f5fc80d0c80 2024-03-04T09:29:32.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.028+0000 7f5fce600640 1 --2- 172.21.15.6:0/521395432 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5fc815e0b0 0x7f5fc80b68f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:29:32.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.029+0000 7f5fcdc00640 1 --2- 172.21.15.6:0/521395432 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5fc80d0c80 0x7f5fc80ccd40 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f5fbc00cc50 tx=0x7f5fbc00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:32.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.029+0000 7f5fc5600640 1 -- 172.21.15.6:0/521395432 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fbc017070 con 0x7f5fc80d0c80 2024-03-04T09:29:32.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.029+0000 7f5fc5600640 1 -- 172.21.15.6:0/521395432 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fbc012400 con 0x7f5fc80d0c80 2024-03-04T09:29:32.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.029+0000 7f5fc5600640 1 -- 172.21.15.6:0/521395432 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fbc00ce60 con 0x7f5fc80d0c80 2024-03-04T09:29:32.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.029+0000 7f5fcf000640 1 -- 172.21.15.6:0/521395432 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fc80cbba0 con 0x7f5fc80d0c80 2024-03-04T09:29:32.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.029+0000 7f5fcf000640 1 -- 172.21.15.6:0/521395432 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fc80cc160 con 0x7f5fc80d0c80 2024-03-04T09:29:32.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.030+0000 7f5fcf000640 1 -- 172.21.15.6:0/521395432 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fc804f330 con 0x7f5fc80d0c80 2024-03-04T09:29:32.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.031+0000 7f5fc5600640 1 -- 172.21.15.6:0/521395432 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5fbc01a070 con 0x7f5fc80d0c80 2024-03-04T09:29:32.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.032+0000 7f5fc5600640 1 --2- 172.21.15.6:0/521395432 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f5fa806b4d0 0x7f5fa806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:32.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.032+0000 7f5fc5600640 1 -- 172.21.15.6:0/521395432 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f5fbc08a380 con 0x7f5fc80d0c80 2024-03-04T09:29:32.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.033+0000 7f5fcd200640 1 --2- 172.21.15.6:0/521395432 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f5fa806b4d0 0x7f5fa806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:32.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.034+0000 7f5fcd200640 1 --2- 172.21.15.6:0/521395432 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f5fa806b4d0 0x7f5fa806d990 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f5fb4034000 tx=0x7f5fb4008030 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:32.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.037+0000 7f5fc5600640 1 -- 172.21.15.6:0/521395432 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5fbc059050 con 0x7f5fc80d0c80 2024-03-04T09:29:32.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.277+0000 7f5fcf000640 1 -- 172.21.15.6:0/521395432 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5fc8073f70 con 0x7f5fc80d0c80 2024-03-04T09:29:32.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.278+0000 7f5fc5600640 1 -- 172.21.15.6:0/521395432 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+4979 (secure 0 0 0) 0x7f5fbc058950 con 0x7f5fc80d0c80 2024-03-04T09:29:32.274 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:29:32.275 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":42,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5228,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/1891254736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1891254736},{"type":"v1","addr":"172.21.15.6:6834","nonce":1891254736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":5578,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/1781219843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":1781219843},{"type":"v1","addr":"172.21.15.146:6835","nonce":1781219843}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":5587,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/1479180802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":1479180802},{"type":"v1","addr":"172.21.15.6:6835","nonce":1479180802}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:28:50.708467+0000","modified":"2024-03-04T09:28:53.801285+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5590},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5590":{"gid":5590,"name":"d","rank":0,"incarnation":38,"state":"up:active","state_seq":3,"addr":"172.21.15.146:6837/3919939129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":3919939129},{"type":"v1","addr":"172.21.15.146:6837","nonce":3919939129}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5590,"qdb_cluster":[5590]},"id":5}]} 2024-03-04T09:29:32.275 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 42 2024-03-04T09:29:32.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.280+0000 7f5fcf000640 1 -- 172.21.15.6:0/521395432 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f5fa806b4d0 msgr2=0x7f5fa806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:32.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.280+0000 7f5fcf000640 1 --2- 172.21.15.6:0/521395432 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f5fa806b4d0 0x7f5fa806d990 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f5fb4034000 tx=0x7f5fb4008030 comp rx=0 tx=0).stop 2024-03-04T09:29:32.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.280+0000 7f5fcf000640 1 -- 172.21.15.6:0/521395432 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5fc80d0c80 msgr2=0x7f5fc80ccd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:32.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.280+0000 7f5fcf000640 1 --2- 172.21.15.6:0/521395432 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5fc80d0c80 0x7f5fc80ccd40 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f5fbc00cc50 tx=0x7f5fbc00a430 comp rx=0 tx=0).stop 2024-03-04T09:29:32.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.280+0000 7f5fcf000640 1 -- 172.21.15.6:0/521395432 shutdown_connections 2024-03-04T09:29:32.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.280+0000 7f5fcf000640 1 --2- 172.21.15.6:0/521395432 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5fc815e0b0 0x7f5fc80b68f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:32.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.280+0000 7f5fcf000640 1 --2- 172.21.15.6:0/521395432 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f5fa806b4d0 0x7f5fa806d990 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:32.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.280+0000 7f5fcf000640 1 --2- 172.21.15.6:0/521395432 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5fc80d4a10 0x7f5fc80cae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:32.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.280+0000 7f5fcf000640 1 --2- 172.21.15.6:0/521395432 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5fc80d0c80 0x7f5fc80ccd40 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:32.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.280+0000 7f5fcf000640 1 -- 172.21.15.6:0/521395432 >> 172.21.15.6:0/521395432 conn(0x7f5fc80d3d30 msgr2=0x7f5fc80ca5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:29:32.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.281+0000 7f5fcf000640 1 -- 172.21.15.6:0/521395432 shutdown_connections 2024-03-04T09:29:32.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.281+0000 7f5fcf000640 1 -- 172.21.15.6:0/521395432 wait complete. 2024-03-04T09:29:32.296 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:29:32.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.527+0000 7fd666400640 1 Processor -- start 2024-03-04T09:29:32.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.527+0000 7fd666400640 1 -- start start 2024-03-04T09:29:32.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.528+0000 7fd666400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd660070f10 0x7fd6600cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:32.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.529+0000 7fd666400640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6600714d0 con 0x7fd6600cce90 2024-03-04T09:29:32.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.529+0000 7fd666400640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6600717b0 con 0x7fd660145300 2024-03-04T09:29:32.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.529+0000 7fd666400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6600cd300 con 0x7fd660070f10 2024-03-04T09:29:32.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.529+0000 7fd665a00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fd6600cce90 0x7fd66015c570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:42884/0 (socket says 172.21.15.6:42884) 2024-03-04T09:29:32.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.529+0000 7fd665a00640 1 -- 172.21.15.6:0/3854579377 learned_addr learned my addr 172.21.15.6:0/3854579377 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:29:32.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.529+0000 7fd65fe00640 1 --2- 172.21.15.6:0/3854579377 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd660070f10 0x7fd6600cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:32.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.529+0000 7fd65f400640 1 -- 172.21.15.6:0/3854579377 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3848385776 0 0) 0x7fd6600714d0 con 0x7fd6600cce90 2024-03-04T09:29:32.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.530+0000 7fd65f400640 1 -- 172.21.15.6:0/3854579377 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd648003520 con 0x7fd6600cce90 2024-03-04T09:29:32.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.530+0000 7fd65f400640 1 -- 172.21.15.6:0/3854579377 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2227141624 0 0) 0x7fd648003520 con 0x7fd6600cce90 2024-03-04T09:29:32.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.530+0000 7fd65f400640 1 -- 172.21.15.6:0/3854579377 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd660070f10 msgr2=0x7fd6600cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:32.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.530+0000 7fd65f400640 1 --2- 172.21.15.6:0/3854579377 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd660070f10 0x7fd6600cc950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:32.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.530+0000 7fd65f400640 1 -- 172.21.15.6:0/3854579377 >> v1:172.21.15.146:6789/0 conn(0x7fd660145300 legacy=0x7fd6600b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:32.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.530+0000 7fd65f400640 1 -- 172.21.15.6:0/3854579377 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd660071a90 con 0x7fd6600cce90 2024-03-04T09:29:32.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.530+0000 7fd65f400640 1 -- 172.21.15.6:0/3854579377 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7fd6580019f0 con 0x7fd6600cce90 2024-03-04T09:29:32.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.530+0000 7fd65f400640 1 -- 172.21.15.6:0/3854579377 >> v1:172.21.15.6:6789/0 conn(0x7fd6600cce90 legacy=0x7fd66015c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:32.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.530+0000 7fd65f400640 1 --2- 172.21.15.6:0/3854579377 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd6480039e0 0x7fd648003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:32.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.530+0000 7fd65f400640 1 --2- 172.21.15.6:0/3854579377 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd648004800 0x7fd648004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:32.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.530+0000 7fd65f400640 1 --2- 172.21.15.6:0/3854579377 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd660070f10 0x7fd6480051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:32.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.530+0000 7fd65f400640 1 -- 172.21.15.6:0/3854579377 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6600cd300 con 0x7fd648004800 2024-03-04T09:29:32.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.530+0000 7fd65f400640 1 -- 172.21.15.6:0/3854579377 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd64800baa0 con 0x7fd6480039e0 2024-03-04T09:29:32.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.530+0000 7fd65f400640 1 -- 172.21.15.6:0/3854579377 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd64800bca0 con 0x7fd660070f10 2024-03-04T09:29:32.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.530+0000 7fd65fe00640 1 --2- 172.21.15.6:0/3854579377 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd648004800 0x7fd648004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:32.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.531+0000 7fd665000640 1 --2- 172.21.15.6:0/3854579377 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd6480039e0 0x7fd648003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:32.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.531+0000 7fd665a00640 1 --2- 172.21.15.6:0/3854579377 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd660070f10 0x7fd6480051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:32.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.531+0000 7fd65fe00640 1 -- 172.21.15.6:0/3854579377 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd660070f10 msgr2=0x7fd6480051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:32.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.531+0000 7fd65fe00640 1 --2- 172.21.15.6:0/3854579377 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd660070f10 0x7fd6480051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:32.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.531+0000 7fd65fe00640 1 -- 172.21.15.6:0/3854579377 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd6480039e0 msgr2=0x7fd648003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:32.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.531+0000 7fd65fe00640 1 --2- 172.21.15.6:0/3854579377 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd6480039e0 0x7fd648003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:32.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.531+0000 7fd65fe00640 1 -- 172.21.15.6:0/3854579377 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd648003520 con 0x7fd648004800 2024-03-04T09:29:32.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.532+0000 7fd65fe00640 1 --2- 172.21.15.6:0/3854579377 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd648004800 0x7fd648004c80 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7fd64c009a90 tx=0x7fd648005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72304d74b99b0032 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:32.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.532+0000 7fd65f400640 1 -- 172.21.15.6:0/3854579377 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd64c039070 con 0x7fd648004800 2024-03-04T09:29:32.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.532+0000 7fd65f400640 1 -- 172.21.15.6:0/3854579377 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd64c02c680 con 0x7fd648004800 2024-03-04T09:29:32.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.532+0000 7fd65f400640 1 -- 172.21.15.6:0/3854579377 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd64c009690 con 0x7fd648004800 2024-03-04T09:29:32.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.533+0000 7fd666400640 1 -- 172.21.15.6:0/3854579377 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd648004800 msgr2=0x7fd648004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:32.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.533+0000 7fd666400640 1 --2- 172.21.15.6:0/3854579377 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd648004800 0x7fd648004c80 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7fd64c009a90 tx=0x7fd648005830 comp rx=0 tx=0).stop 2024-03-04T09:29:32.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.533+0000 7fd665a00640 1 -- 172.21.15.6:0/3854579377 reap_dead start 2024-03-04T09:29:32.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.533+0000 7fd666400640 1 -- 172.21.15.6:0/3854579377 shutdown_connections 2024-03-04T09:29:32.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.533+0000 7fd666400640 1 -- 172.21.15.6:0/3854579377 >> 172.21.15.6:0/3854579377 conn(0x7fd66006b3d0 msgr2=0x7fd66006b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:29:32.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.533+0000 7fd666400640 1 -- 172.21.15.6:0/3854579377 shutdown_connections 2024-03-04T09:29:32.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.533+0000 7fd666400640 1 -- 172.21.15.6:0/3854579377 wait complete. 2024-03-04T09:29:32.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.535+0000 7fd666400640 1 Processor -- start 2024-03-04T09:29:32.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.535+0000 7fd666400640 1 -- start start 2024-03-04T09:29:32.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.536+0000 7fd666400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd648004800 0x7fd66015b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:32.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.536+0000 7fd666400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd6600b1050 0x7fd6600aec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:32.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.536+0000 7fd666400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd6600b14d0 0x7fd6600ac040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:32.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.536+0000 7fd666400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6600cc950 con 0x7fd648004800 2024-03-04T09:29:32.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.536+0000 7fd666400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6601456e0 con 0x7fd6600b1050 2024-03-04T09:29:32.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.536+0000 7fd666400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6600ccad0 con 0x7fd6600b14d0 2024-03-04T09:29:32.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.536+0000 7fd665000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd648004800 0x7fd66015b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:32.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.536+0000 7fd665000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd648004800 0x7fd66015b210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:52926/0 (socket says 172.21.15.6:52926) 2024-03-04T09:29:32.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.536+0000 7fd665000640 1 -- 172.21.15.6:0/3855115826 learned_addr learned my addr 172.21.15.6:0/3855115826 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:29:32.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.537+0000 7fd65fe00640 1 --2- 172.21.15.6:0/3855115826 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd6600b1050 0x7fd6600aec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:32.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.537+0000 7fd665000640 1 -- 172.21.15.6:0/3855115826 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd6600b14d0 msgr2=0x7fd6600ac040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:32.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.537+0000 7fd665a00640 1 --2- 172.21.15.6:0/3855115826 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd6600b14d0 0x7fd6600ac040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:32.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.537+0000 7fd665000640 1 --2- 172.21.15.6:0/3855115826 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd6600b14d0 0x7fd6600ac040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:32.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.537+0000 7fd665000640 1 -- 172.21.15.6:0/3855115826 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd6600b1050 msgr2=0x7fd6600aec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:32.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.537+0000 7fd665000640 1 --2- 172.21.15.6:0/3855115826 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd6600b1050 0x7fd6600aec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:32.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.537+0000 7fd665000640 1 -- 172.21.15.6:0/3855115826 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6600afcd0 con 0x7fd648004800 2024-03-04T09:29:32.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.537+0000 7fd65fe00640 1 --2- 172.21.15.6:0/3855115826 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd6600b1050 0x7fd6600aec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:29:32.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.537+0000 7fd665000640 1 --2- 172.21.15.6:0/3855115826 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd648004800 0x7fd66015b210 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7fd65400e550 tx=0x7fd65400ea10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:32.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.537+0000 7fd65cc00640 1 -- 172.21.15.6:0/3855115826 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6540052e0 con 0x7fd648004800 2024-03-04T09:29:32.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.537+0000 7fd65cc00640 1 -- 172.21.15.6:0/3855115826 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd654018070 con 0x7fd648004800 2024-03-04T09:29:32.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.537+0000 7fd666400640 1 -- 172.21.15.6:0/3855115826 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6600affb0 con 0x7fd648004800 2024-03-04T09:29:32.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.537+0000 7fd65cc00640 1 -- 172.21.15.6:0/3855115826 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd654003810 con 0x7fd648004800 2024-03-04T09:29:32.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.538+0000 7fd666400640 1 -- 172.21.15.6:0/3855115826 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6600b0570 con 0x7fd648004800 2024-03-04T09:29:32.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.539+0000 7fd666400640 1 -- 172.21.15.6:0/3855115826 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd66004f330 con 0x7fd648004800 2024-03-04T09:29:32.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.540+0000 7fd65cc00640 1 -- 172.21.15.6:0/3855115826 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd6540039b0 con 0x7fd648004800 2024-03-04T09:29:32.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.540+0000 7fd65cc00640 1 --2- 172.21.15.6:0/3855115826 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd64006b4d0 0x7fd64006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:32.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.540+0000 7fd65cc00640 1 -- 172.21.15.6:0/3855115826 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fd654089f70 con 0x7fd648004800 2024-03-04T09:29:32.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.541+0000 7fd65fe00640 1 --2- 172.21.15.6:0/3855115826 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd64006b4d0 0x7fd64006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:32.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.543+0000 7fd65fe00640 1 --2- 172.21.15.6:0/3855115826 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd64006b4d0 0x7fd64006d990 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fd64c036000 tx=0x7fd64c030000 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:32.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.545+0000 7fd65cc00640 1 -- 172.21.15.6:0/3855115826 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd654058cd0 con 0x7fd648004800 2024-03-04T09:29:32.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.790+0000 7fd666400640 1 -- 172.21.15.6:0/3855115826 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd6601446f0 con 0x7fd648004800 2024-03-04T09:29:32.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.791+0000 7fd65cc00640 1 -- 172.21.15.6:0/3855115826 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+4979 (secure 0 0 0) 0x7fd6540585d0 con 0x7fd648004800 2024-03-04T09:29:32.788 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:29:32.788 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":42,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5228,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/1891254736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1891254736},{"type":"v1","addr":"172.21.15.6:6834","nonce":1891254736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":5578,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/1781219843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":1781219843},{"type":"v1","addr":"172.21.15.146:6835","nonce":1781219843}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":5587,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/1479180802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":1479180802},{"type":"v1","addr":"172.21.15.6:6835","nonce":1479180802}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:28:50.708467+0000","modified":"2024-03-04T09:28:53.801285+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5590},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5590":{"gid":5590,"name":"d","rank":0,"incarnation":38,"state":"up:active","state_seq":3,"addr":"172.21.15.146:6837/3919939129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":3919939129},{"type":"v1","addr":"172.21.15.146:6837","nonce":3919939129}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5590,"qdb_cluster":[5590]},"id":5}]} 2024-03-04T09:29:32.789 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 42 2024-03-04T09:29:32.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.793+0000 7fd666400640 1 -- 172.21.15.6:0/3855115826 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd64006b4d0 msgr2=0x7fd64006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:32.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.794+0000 7fd666400640 1 --2- 172.21.15.6:0/3855115826 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd64006b4d0 0x7fd64006d990 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fd64c036000 tx=0x7fd64c030000 comp rx=0 tx=0).stop 2024-03-04T09:29:32.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.794+0000 7fd666400640 1 -- 172.21.15.6:0/3855115826 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd648004800 msgr2=0x7fd66015b210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:32.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.794+0000 7fd666400640 1 --2- 172.21.15.6:0/3855115826 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd648004800 0x7fd66015b210 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7fd65400e550 tx=0x7fd65400ea10 comp rx=0 tx=0).stop 2024-03-04T09:29:32.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.794+0000 7fd666400640 1 -- 172.21.15.6:0/3855115826 shutdown_connections 2024-03-04T09:29:32.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.794+0000 7fd666400640 1 --2- 172.21.15.6:0/3855115826 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd64006b4d0 0x7fd64006d990 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:32.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.794+0000 7fd666400640 1 --2- 172.21.15.6:0/3855115826 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd6600b14d0 0x7fd6600ac040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:32.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.794+0000 7fd666400640 1 --2- 172.21.15.6:0/3855115826 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd6600b1050 0x7fd6600aec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:32.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.794+0000 7fd666400640 1 --2- 172.21.15.6:0/3855115826 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd648004800 0x7fd66015b210 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:32.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.794+0000 7fd666400640 1 -- 172.21.15.6:0/3855115826 >> 172.21.15.6:0/3855115826 conn(0x7fd66006b3d0 msgr2=0x7fd66006c9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:29:32.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.794+0000 7fd666400640 1 -- 172.21.15.6:0/3855115826 shutdown_connections 2024-03-04T09:29:32.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:32.794+0000 7fd666400640 1 -- 172.21.15.6:0/3855115826 wait complete. 2024-03-04T09:29:32.811 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T09:29:33.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.025+0000 7f2567e00640 1 Processor -- start 2024-03-04T09:29:33.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.025+0000 7f2567e00640 1 -- start start 2024-03-04T09:29:33.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.026+0000 7f2567e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f2568145300 0x7f25680b49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:33.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.026+0000 7f2567e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25680714d0 con 0x7f2568070f10 2024-03-04T09:29:33.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.026+0000 7f2567e00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25680717b0 con 0x7f25680cd060 2024-03-04T09:29:33.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.026+0000 7f2567e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2568071a90 con 0x7f2568145300 2024-03-04T09:29:33.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.027+0000 7f2566a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f2568145300 0x7f25680b49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:33.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.027+0000 7f2566a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f2568145300 0x7f25680b49a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:52930/0 (socket says 172.21.15.6:52930) 2024-03-04T09:29:33.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.027+0000 7f255fe00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f2568070f10 0x7f25680cc950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:42886/0 (socket says 172.21.15.6:42886) 2024-03-04T09:29:33.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.027+0000 7f2566a00640 1 -- 172.21.15.6:0/1453365763 learned_addr learned my addr 172.21.15.6:0/1453365763 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:29:33.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.027+0000 7f2566a00640 1 -- 172.21.15.6:0/1453365763 >> v1:172.21.15.146:6789/0 conn(0x7f25680cd060 legacy=0x7f256815c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:33.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.027+0000 7f2566a00640 1 -- 172.21.15.6:0/1453365763 >> v1:172.21.15.6:6789/0 conn(0x7f2568070f10 legacy=0x7f25680cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:33.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.027+0000 7f2566a00640 1 -- 172.21.15.6:0/1453365763 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2568071d70 con 0x7f2568145300 2024-03-04T09:29:33.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.028+0000 7f2566a00640 1 --2- 172.21.15.6:0/1453365763 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2568145300 0x7f25680b49a0 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f2558009870 tx=0x7f255802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2303a96afefbd90 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:33.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.029+0000 7f2566000640 1 -- 172.21.15.6:0/1453365763 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2558035020 con 0x7f2568145300 2024-03-04T09:29:33.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.029+0000 7f2566000640 1 -- 172.21.15.6:0/1453365763 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2558031870 con 0x7f2568145300 2024-03-04T09:29:33.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.029+0000 7f2566000640 1 -- 172.21.15.6:0/1453365763 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2558031c40 con 0x7f2568145300 2024-03-04T09:29:33.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.029+0000 7f2567e00640 1 -- 172.21.15.6:0/1453365763 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2568145300 msgr2=0x7f25680b49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:33.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.029+0000 7f2567e00640 1 --2- 172.21.15.6:0/1453365763 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2568145300 0x7f25680b49a0 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f2558009870 tx=0x7f255802b910 comp rx=0 tx=0).stop 2024-03-04T09:29:33.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.029+0000 7f2567e00640 1 -- 172.21.15.6:0/1453365763 shutdown_connections 2024-03-04T09:29:33.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.029+0000 7f2567e00640 1 --2- 172.21.15.6:0/1453365763 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2568145300 0x7f25680b49a0 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:33.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.029+0000 7f2567e00640 1 -- 172.21.15.6:0/1453365763 >> 172.21.15.6:0/1453365763 conn(0x7f256806b3d0 msgr2=0x7f256806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:29:33.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.030+0000 7f2567e00640 1 -- 172.21.15.6:0/1453365763 shutdown_connections 2024-03-04T09:29:33.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.030+0000 7f2567e00640 1 -- 172.21.15.6:0/1453365763 wait complete. 2024-03-04T09:29:33.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.031+0000 7f2567e00640 1 Processor -- start 2024-03-04T09:29:33.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.032+0000 7f2567e00640 1 -- start start 2024-03-04T09:29:33.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.032+0000 7f2567e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2568070f10 0x7f2568140ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:33.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.032+0000 7f2567e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f25680cd060 0x7f256806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:33.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.032+0000 7f2567e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2568145300 0x7f25681f3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:33.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.032+0000 7f2567e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f256815cc80 con 0x7f25680cd060 2024-03-04T09:29:33.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.032+0000 7f2567e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25681456e0 con 0x7f2568145300 2024-03-04T09:29:33.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.032+0000 7f2567e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f256815ce00 con 0x7f2568070f10 2024-03-04T09:29:33.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.033+0000 7f255fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f25680cd060 0x7f256806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:33.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.033+0000 7f255fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f25680cd060 0x7f256806d3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:52936/0 (socket says 172.21.15.6:52936) 2024-03-04T09:29:33.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.033+0000 7f255fe00640 1 -- 172.21.15.6:0/3904978690 learned_addr learned my addr 172.21.15.6:0/3904978690 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:29:33.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.033+0000 7f2567400640 1 --2- 172.21.15.6:0/3904978690 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2568145300 0x7f25681f3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:33.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.033+0000 7f2566a00640 1 --2- 172.21.15.6:0/3904978690 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2568070f10 0x7f2568140ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:33.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.033+0000 7f255fe00640 1 -- 172.21.15.6:0/3904978690 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2568070f10 msgr2=0x7f2568140ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:33.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.033+0000 7f255fe00640 1 --2- 172.21.15.6:0/3904978690 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2568070f10 0x7f2568140ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:33.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.033+0000 7f255fe00640 1 -- 172.21.15.6:0/3904978690 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2568145300 msgr2=0x7f25681f3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:33.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.033+0000 7f255fe00640 1 --2- 172.21.15.6:0/3904978690 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2568145300 0x7f25681f3bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:33.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.033+0000 7f255fe00640 1 -- 172.21.15.6:0/3904978690 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25681e63a0 con 0x7f25680cd060 2024-03-04T09:29:33.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.034+0000 7f255fe00640 1 --2- 172.21.15.6:0/3904978690 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f25680cd060 0x7f256806d3c0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f255000c400 tx=0x7f255000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:33.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.034+0000 7f255ea00640 1 -- 172.21.15.6:0/3904978690 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25500074e0 con 0x7f25680cd060 2024-03-04T09:29:33.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.034+0000 7f255ea00640 1 -- 172.21.15.6:0/3904978690 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2550007680 con 0x7f25680cd060 2024-03-04T09:29:33.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.034+0000 7f2567e00640 1 -- 172.21.15.6:0/3904978690 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25681e6680 con 0x7f25680cd060 2024-03-04T09:29:33.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.034+0000 7f255ea00640 1 -- 172.21.15.6:0/3904978690 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2550004a00 con 0x7f25680cd060 2024-03-04T09:29:33.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.034+0000 7f2567e00640 1 -- 172.21.15.6:0/3904978690 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25681e6c40 con 0x7f25680cd060 2024-03-04T09:29:33.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.035+0000 7f2567e00640 1 -- 172.21.15.6:0/3904978690 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f256804f330 con 0x7f25680cd060 2024-03-04T09:29:33.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.036+0000 7f255ea00640 1 -- 172.21.15.6:0/3904978690 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f255001a020 con 0x7f25680cd060 2024-03-04T09:29:33.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.037+0000 7f255ea00640 1 --2- 172.21.15.6:0/3904978690 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f254406b4d0 0x7f254406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:33.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.037+0000 7f255ea00640 1 -- 172.21.15.6:0/3904978690 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f2550089b40 con 0x7f25680cd060 2024-03-04T09:29:33.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.038+0000 7f2566a00640 1 --2- 172.21.15.6:0/3904978690 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f254406b4d0 0x7f254406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:33.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.039+0000 7f2566a00640 1 --2- 172.21.15.6:0/3904978690 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f254406b4d0 0x7f254406d990 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f2558036000 tx=0x7f255803c040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:33.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.041+0000 7f255ea00640 1 -- 172.21.15.6:0/3904978690 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f25500588a0 con 0x7f25680cd060 2024-03-04T09:29:33.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.238+0000 7f2567e00640 1 -- 172.21.15.6:0/3904978690 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2568141400 con 0x7f25680cd060 2024-03-04T09:29:33.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.240+0000 7f255ea00640 1 -- 172.21.15.6:0/3904978690 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v70) v1 ==== 74+0+16813 (secure 0 0 0) 0x7f25500581a0 con 0x7f25680cd060 2024-03-04T09:29:33.237 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:29:33.237 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":70,"fsid":"edce0578-5ae4-4402-b45a-552bdef84d7c","created":"2024-03-04T09:24:54.523780+0000","modified":"2024-03-04T09:29:02.333052+0000","last_up_change":"2024-03-04T09:25:07.089049+0000","last_in_change":"2024-03-04T09:24:57.982077+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T09:25:07.935198+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-03-04T09:28:48.627427+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-03-04T09:28:49.739043+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a96574f-81f1-4bc6-8463-757520b84a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6825","nonce":1364449401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6827","nonce":1364449401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6831","nonce":1364449401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6829","nonce":1364449401}]},"public_addr":"172.21.15.6:6825/1364449401","cluster_addr":"172.21.15.6:6827/1364449401","heartbeat_back_addr":"172.21.15.6:6831/1364449401","heartbeat_front_addr":"172.21.15.6:6829/1364449401","state":["exists","up"]},{"osd":1,"uuid":"7d79b9c1-6b87-476a-bc36-7db5ee64cd90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6809","nonce":2494103078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6811","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6813","nonce":2494103078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6821","nonce":2494103078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6817","nonce":2494103078}]},"public_addr":"172.21.15.6:6809/2494103078","cluster_addr":"172.21.15.6:6813/2494103078","heartbeat_back_addr":"172.21.15.6:6821/2494103078","heartbeat_front_addr":"172.21.15.6:6817/2494103078","state":["exists","up"]},{"osd":2,"uuid":"492331b9-4ee9-40b6-bacd-85d44cd6c45a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6812","nonce":1500066681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6816","nonce":1500066681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6824","nonce":1500066681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6820","nonce":1500066681}]},"public_addr":"172.21.15.6:6812/1500066681","cluster_addr":"172.21.15.6:6816/1500066681","heartbeat_back_addr":"172.21.15.6:6824/1500066681","heartbeat_front_addr":"172.21.15.6:6820/1500066681","state":["exists","up"]},{"osd":3,"uuid":"21bb2efe-9ba7-4bdc-bc31-180cd1102e21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6801","nonce":1558806123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6803","nonce":1558806123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6807","nonce":1558806123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6805","nonce":1558806123}]},"public_addr":"172.21.15.6:6801/1558806123","cluster_addr":"172.21.15.6:6803/1558806123","heartbeat_back_addr":"172.21.15.6:6807/1558806123","heartbeat_front_addr":"172.21.15.6:6805/1558806123","state":["exists","up"]},{"osd":4,"uuid":"6e2c4b71-19d9-4737-83d9-558705d8644a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6801","nonce":1526135780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6803","nonce":1526135780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6807","nonce":1526135780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6805","nonce":1526135780}]},"public_addr":"172.21.15.146:6801/1526135780","cluster_addr":"172.21.15.146:6803/1526135780","heartbeat_back_addr":"172.21.15.146:6807/1526135780","heartbeat_front_addr":"172.21.15.146:6805/1526135780","state":["exists","up"]},{"osd":5,"uuid":"fb69c525-f1be-4b7d-ab32-6c29b0d6fddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6809","nonce":460262507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6811","nonce":460262507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6815","nonce":460262507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6813","nonce":460262507}]},"public_addr":"172.21.15.146:6809/460262507","cluster_addr":"172.21.15.146:6811/460262507","heartbeat_back_addr":"172.21.15.146:6815/460262507","heartbeat_front_addr":"172.21.15.146:6813/460262507","state":["exists","up"]},{"osd":6,"uuid":"c32e7fed-370d-4401-8ce0-6a4335b0d168","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6823","nonce":1392897493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6827","nonce":1392897493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6831","nonce":1392897493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6829","nonce":1392897493}]},"public_addr":"172.21.15.146:6823/1392897493","cluster_addr":"172.21.15.146:6827/1392897493","heartbeat_back_addr":"172.21.15.146:6831/1392897493","heartbeat_front_addr":"172.21.15.146:6829/1392897493","state":["exists","up"]},{"osd":7,"uuid":"75a79b44-fa5f-426d-a7d3-71f21f78835c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6817","nonce":3268323156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6819","nonce":3268323156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6825","nonce":3268323156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6822","nonce":3268323156}]},"public_addr":"172.21.15.146:6817/3268323156","cluster_addr":"172.21.15.146:6819/3268323156","heartbeat_back_addr":"172.21.15.146:6825/3268323156","heartbeat_front_addr":"172.21.15.146:6822/3268323156","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.310437+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.295850+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.294674+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.526855+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.300023+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.595160+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.346046+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.558741+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"11.8","mappings":[{"from":6,"to":5}]},{"pgid":"11.10","mappings":[{"from":6,"to":1}]},{"pgid":"11.14","mappings":[{"from":6,"to":0}]},{"pgid":"11.3e","mappings":[{"from":6,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T09:29:33.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.242+0000 7f2567e00640 1 -- 172.21.15.6:0/3904978690 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f254406b4d0 msgr2=0x7f254406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:33.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.242+0000 7f2567e00640 1 --2- 172.21.15.6:0/3904978690 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f254406b4d0 0x7f254406d990 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f2558036000 tx=0x7f255803c040 comp rx=0 tx=0).stop 2024-03-04T09:29:33.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.242+0000 7f2567e00640 1 -- 172.21.15.6:0/3904978690 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f25680cd060 msgr2=0x7f256806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:33.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.242+0000 7f2567e00640 1 --2- 172.21.15.6:0/3904978690 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f25680cd060 0x7f256806d3c0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f255000c400 tx=0x7f255000c8c0 comp rx=0 tx=0).stop 2024-03-04T09:29:33.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.242+0000 7f2567e00640 1 -- 172.21.15.6:0/3904978690 shutdown_connections 2024-03-04T09:29:33.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.242+0000 7f2567e00640 1 --2- 172.21.15.6:0/3904978690 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2568145300 0x7f25681f3bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:33.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.242+0000 7f2567e00640 1 --2- 172.21.15.6:0/3904978690 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f25680cd060 0x7f256806d3c0 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:33.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.242+0000 7f2567e00640 1 --2- 172.21.15.6:0/3904978690 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f254406b4d0 0x7f254406d990 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:33.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.242+0000 7f2567e00640 1 --2- 172.21.15.6:0/3904978690 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2568070f10 0x7f2568140ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:33.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.243+0000 7f2567e00640 1 -- 172.21.15.6:0/3904978690 >> 172.21.15.6:0/3904978690 conn(0x7f256806b3d0 msgr2=0x7f25680cbd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:29:33.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.243+0000 7f2567e00640 1 -- 172.21.15.6:0/3904978690 shutdown_connections 2024-03-04T09:29:33.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.243+0000 7f2567e00640 1 -- 172.21.15.6:0/3904978690 wait complete. 2024-03-04T09:29:33.259 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-04T09:29:33.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.482+0000 7f8a3a600640 1 Processor -- start 2024-03-04T09:29:33.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.482+0000 7f8a3a600640 1 -- start start 2024-03-04T09:29:33.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.484+0000 7f8a3a600640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f8a34161b40 0x7f8a340cfd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:33.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.484+0000 7f8a3a600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8a340d2930 0x7f8a340b3740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:33.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.484+0000 7f8a3a600640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a340cea50 con 0x7f8a340b5060 2024-03-04T09:29:33.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.484+0000 7f8a3a600640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a340cf900 con 0x7f8a34161b40 2024-03-04T09:29:33.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.484+0000 7f8a3a600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a340cecf0 con 0x7f8a340d2930 2024-03-04T09:29:33.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.484+0000 7f8a32a00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f8a340b5060 0x7f8a340d2220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:42900/0 (socket says 172.21.15.6:42900) 2024-03-04T09:29:33.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.484+0000 7f8a32a00640 1 -- 172.21.15.6:0/4143943551 learned_addr learned my addr 172.21.15.6:0/4143943551 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:29:33.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.484+0000 7f8a33400640 1 --2- 172.21.15.6:0/4143943551 >> v2:172.21.15.146:3300/0 conn(0x7f8a34161b40 0x7f8a340cfd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:33.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.484+0000 7f8a33e00640 1 --2- 172.21.15.6:0/4143943551 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8a340d2930 0x7f8a340b3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:33.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.485+0000 7f8a32000640 1 -- 172.21.15.6:0/4143943551 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3141257028 0 0) 0x7f8a340cea50 con 0x7f8a340b5060 2024-03-04T09:29:33.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.485+0000 7f8a32000640 1 -- 172.21.15.6:0/4143943551 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a18003520 con 0x7f8a340b5060 2024-03-04T09:29:33.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.485+0000 7f8a32000640 1 -- 172.21.15.6:0/4143943551 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2522990342 0 0) 0x7f8a18003520 con 0x7f8a340b5060 2024-03-04T09:29:33.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.485+0000 7f8a32000640 1 -- 172.21.15.6:0/4143943551 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8a340d2930 msgr2=0x7f8a340b3740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:33.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.485+0000 7f8a32000640 1 --2- 172.21.15.6:0/4143943551 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8a340d2930 0x7f8a340b3740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:33.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.485+0000 7f8a32000640 1 -- 172.21.15.6:0/4143943551 >> v2:172.21.15.146:3300/0 conn(0x7f8a34161b40 msgr2=0x7f8a340cfd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:33.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.485+0000 7f8a32000640 1 --2- 172.21.15.6:0/4143943551 >> v2:172.21.15.146:3300/0 conn(0x7f8a34161b40 0x7f8a340cfd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:33.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.485+0000 7f8a32000640 1 -- 172.21.15.6:0/4143943551 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a340cefd0 con 0x7f8a340b5060 2024-03-04T09:29:33.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.485+0000 7f8a32000640 1 -- 172.21.15.6:0/4143943551 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7f8a20001670 con 0x7f8a340b5060 2024-03-04T09:29:33.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.485+0000 7f8a32000640 1 -- 172.21.15.6:0/4143943551 >> v1:172.21.15.6:6789/0 conn(0x7f8a340b5060 legacy=0x7f8a340d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:33.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.485+0000 7f8a32000640 1 --2- 172.21.15.6:0/4143943551 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8a18003a20 0x7f8a18003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:33.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.485+0000 7f8a32000640 1 --2- 172.21.15.6:0/4143943551 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8a340d2930 0x7f8a18004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:33.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.485+0000 7f8a32000640 1 --2- 172.21.15.6:0/4143943551 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8a18004d40 0x7f8a180051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:33.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.485+0000 7f8a32000640 1 -- 172.21.15.6:0/4143943551 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a340cf900 con 0x7f8a18004d40 2024-03-04T09:29:33.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.485+0000 7f8a32000640 1 -- 172.21.15.6:0/4143943551 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a340cecf0 con 0x7f8a18003a20 2024-03-04T09:29:33.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.485+0000 7f8a32000640 1 -- 172.21.15.6:0/4143943551 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a1800b9f0 con 0x7f8a340d2930 2024-03-04T09:29:33.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.486+0000 7f8a33e00640 1 --2- 172.21.15.6:0/4143943551 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8a18004d40 0x7f8a180051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:33.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.486+0000 7f8a33400640 1 --2- 172.21.15.6:0/4143943551 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8a18003a20 0x7f8a18003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:33.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.486+0000 7f8a32a00640 1 --2- 172.21.15.6:0/4143943551 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8a340d2930 0x7f8a18004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:33.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.486+0000 7f8a33e00640 1 -- 172.21.15.6:0/4143943551 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8a340d2930 msgr2=0x7f8a18004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:33.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.486+0000 7f8a33e00640 1 --2- 172.21.15.6:0/4143943551 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8a340d2930 0x7f8a18004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:33.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.486+0000 7f8a33e00640 1 -- 172.21.15.6:0/4143943551 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8a18003a20 msgr2=0x7f8a18003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:33.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.486+0000 7f8a33e00640 1 --2- 172.21.15.6:0/4143943551 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8a18003a20 0x7f8a18003e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:33.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.486+0000 7f8a33e00640 1 -- 172.21.15.6:0/4143943551 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a18003520 con 0x7f8a18004d40 2024-03-04T09:29:33.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.487+0000 7f8a33e00640 1 --2- 172.21.15.6:0/4143943551 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8a18004d40 0x7f8a180051c0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f8a2c009ac0 tx=0x7f8a18005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1333f303225f1d37 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:33.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.488+0000 7f8a32000640 1 -- 172.21.15.6:0/4143943551 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a2c039070 con 0x7f8a18004d40 2024-03-04T09:29:33.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.488+0000 7f8a32000640 1 -- 172.21.15.6:0/4143943551 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a2c02c6b0 con 0x7f8a18004d40 2024-03-04T09:29:33.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.488+0000 7f8a32000640 1 -- 172.21.15.6:0/4143943551 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a2c009690 con 0x7f8a18004d40 2024-03-04T09:29:33.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.488+0000 7f8a3a600640 1 -- 172.21.15.6:0/4143943551 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8a18004d40 msgr2=0x7f8a180051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:33.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.488+0000 7f8a3a600640 1 --2- 172.21.15.6:0/4143943551 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8a18004d40 0x7f8a180051c0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f8a2c009ac0 tx=0x7f8a18005830 comp rx=0 tx=0).stop 2024-03-04T09:29:33.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.488+0000 7f8a33e00640 1 -- 172.21.15.6:0/4143943551 reap_dead start 2024-03-04T09:29:33.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.488+0000 7f8a3a600640 1 -- 172.21.15.6:0/4143943551 shutdown_connections 2024-03-04T09:29:33.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.488+0000 7f8a3a600640 1 -- 172.21.15.6:0/4143943551 >> 172.21.15.6:0/4143943551 conn(0x7f8a3409d220 msgr2=0x7f8a340d3ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:29:33.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.488+0000 7f8a3a600640 1 -- 172.21.15.6:0/4143943551 shutdown_connections 2024-03-04T09:29:33.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.488+0000 7f8a3a600640 1 -- 172.21.15.6:0/4143943551 wait complete. 2024-03-04T09:29:33.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.490+0000 7f8a3a600640 1 Processor -- start 2024-03-04T09:29:33.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.490+0000 7f8a3a600640 1 -- start start 2024-03-04T09:29:33.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.491+0000 7f8a3a600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8a18004d40 0x7f8a3415fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:33.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.491+0000 7f8a3a600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8a340ca100 0x7f8a3415afd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:33.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.491+0000 7f8a3a600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8a340ca580 0x7f8a3415b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:33.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.491+0000 7f8a3a600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a340b2ff0 con 0x7f8a18004d40 2024-03-04T09:29:33.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.491+0000 7f8a3a600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a340cf590 con 0x7f8a340ca100 2024-03-04T09:29:33.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.491+0000 7f8a3a600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a340b3170 con 0x7f8a340ca580 2024-03-04T09:29:33.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.491+0000 7f8a33400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8a18004d40 0x7f8a3415fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:33.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.491+0000 7f8a33400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8a18004d40 0x7f8a3415fcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:52962/0 (socket says 172.21.15.6:52962) 2024-03-04T09:29:33.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.491+0000 7f8a33400640 1 -- 172.21.15.6:0/1598950053 learned_addr learned my addr 172.21.15.6:0/1598950053 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:29:33.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.492+0000 7f8a33400640 1 -- 172.21.15.6:0/1598950053 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8a340ca580 msgr2=0x7f8a3415b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:33.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.492+0000 7f8a33400640 1 --2- 172.21.15.6:0/1598950053 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8a340ca580 0x7f8a3415b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:33.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.492+0000 7f8a33400640 1 -- 172.21.15.6:0/1598950053 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8a340ca100 msgr2=0x7f8a3415afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:33.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.492+0000 7f8a33400640 1 --2- 172.21.15.6:0/1598950053 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8a340ca100 0x7f8a3415afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:33.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.492+0000 7f8a33400640 1 -- 172.21.15.6:0/1598950053 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a340cbcb0 con 0x7f8a18004d40 2024-03-04T09:29:33.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.492+0000 7f8a33400640 1 --2- 172.21.15.6:0/1598950053 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8a18004d40 0x7f8a3415fcf0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f8a24007f60 tx=0x7f8a24007f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:33.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.493+0000 7f8a2b400640 1 -- 172.21.15.6:0/1598950053 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a2401a020 con 0x7f8a18004d40 2024-03-04T09:29:33.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.493+0000 7f8a2b400640 1 -- 172.21.15.6:0/1598950053 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a24016470 con 0x7f8a18004d40 2024-03-04T09:29:33.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.493+0000 7f8a2b400640 1 -- 172.21.15.6:0/1598950053 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a24016860 con 0x7f8a18004d40 2024-03-04T09:29:33.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.493+0000 7f8a3a600640 1 -- 172.21.15.6:0/1598950053 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a340cbf90 con 0x7f8a18004d40 2024-03-04T09:29:33.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.493+0000 7f8a3a600640 1 -- 172.21.15.6:0/1598950053 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a340cc550 con 0x7f8a18004d40 2024-03-04T09:29:33.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.494+0000 7f8a3a600640 1 -- 172.21.15.6:0/1598950053 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a3404f3a0 con 0x7f8a18004d40 2024-03-04T09:29:33.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.496+0000 7f8a2b400640 1 -- 172.21.15.6:0/1598950053 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8a24005d10 con 0x7f8a18004d40 2024-03-04T09:29:33.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.497+0000 7f8a2b400640 1 --2- 172.21.15.6:0/1598950053 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f8a1406b4d0 0x7f8a1406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:33.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.497+0000 7f8a2b400640 1 -- 172.21.15.6:0/1598950053 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f8a2408b1e0 con 0x7f8a18004d40 2024-03-04T09:29:33.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.498+0000 7f8a32a00640 1 --2- 172.21.15.6:0/1598950053 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f8a1406b4d0 0x7f8a1406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:33.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.499+0000 7f8a32a00640 1 --2- 172.21.15.6:0/1598950053 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f8a1406b4d0 0x7f8a1406d990 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f8a20001200 tx=0x7f8a20001130 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:33.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.502+0000 7f8a2b400640 1 -- 172.21.15.6:0/1598950053 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8a24024030 con 0x7f8a18004d40 2024-03-04T09:29:33.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.744+0000 7f8a3a600640 1 -- 172.21.15.6:0/1598950053 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f8a340d1df0 con 0x7f8a18004d40 2024-03-04T09:29:33.742 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-03-04T09:29:33.745+0000 7f1606a00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-04T09:29:33.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.749+0000 7f8a2b400640 1 -- 172.21.15.6:0/1598950053 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v43) v1 ==== 161+0+0 (secure 0 0 0) 0x7f8a24059fe0 con 0x7f8a18004d40 2024-03-04T09:29:33.746 INFO:teuthology.orchestra.run.smithi006.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-04T09:29:33.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.752+0000 7f8a3a600640 1 -- 172.21.15.6:0/1598950053 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f8a1406b4d0 msgr2=0x7f8a1406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:33.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.752+0000 7f8a3a600640 1 --2- 172.21.15.6:0/1598950053 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f8a1406b4d0 0x7f8a1406d990 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f8a20001200 tx=0x7f8a20001130 comp rx=0 tx=0).stop 2024-03-04T09:29:33.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.752+0000 7f8a3a600640 1 -- 172.21.15.6:0/1598950053 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8a18004d40 msgr2=0x7f8a3415fcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:33.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.752+0000 7f8a3a600640 1 --2- 172.21.15.6:0/1598950053 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8a18004d40 0x7f8a3415fcf0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f8a24007f60 tx=0x7f8a24007f90 comp rx=0 tx=0).stop 2024-03-04T09:29:33.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.752+0000 7f8a3a600640 1 -- 172.21.15.6:0/1598950053 shutdown_connections 2024-03-04T09:29:33.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.752+0000 7f8a3a600640 1 --2- 172.21.15.6:0/1598950053 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f8a1406b4d0 0x7f8a1406d990 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:33.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.752+0000 7f8a3a600640 1 --2- 172.21.15.6:0/1598950053 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8a340ca580 0x7f8a3415b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:33.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.752+0000 7f8a3a600640 1 --2- 172.21.15.6:0/1598950053 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8a340ca100 0x7f8a3415afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:33.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.752+0000 7f8a3a600640 1 --2- 172.21.15.6:0/1598950053 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8a18004d40 0x7f8a3415fcf0 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:33.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.752+0000 7f8a3a600640 1 -- 172.21.15.6:0/1598950053 >> 172.21.15.6:0/1598950053 conn(0x7f8a3409d220 msgr2=0x7f8a340b0b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:29:33.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.752+0000 7f8a3a600640 1 -- 172.21.15.6:0/1598950053 shutdown_connections 2024-03-04T09:29:33.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.752+0000 7f8a3a600640 1 -- 172.21.15.6:0/1598950053 wait complete. 2024-03-04T09:29:33.772 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-04T09:29:33.866 INFO:tasks.ceph.mds.d.smithi146.stdout:starting mds.d at 2024-03-04T09:29:33.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.993+0000 7f9e16000640 1 Processor -- start 2024-03-04T09:29:33.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.993+0000 7f9e16000640 1 -- start start 2024-03-04T09:29:33.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.995+0000 7f9e16000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9e10147590 0x7f9e10071480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:33.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.995+0000 7f9e16000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e100719c0 con 0x7f9e100b2250 2024-03-04T09:29:33.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.995+0000 7f9e16000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e1006de40 con 0x7f9e1006d8d0 2024-03-04T09:29:33.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.995+0000 7f9e16000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e10147970 con 0x7f9e10147590 2024-03-04T09:29:33.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.995+0000 7f9e15600640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f9e100b2250 0x7f9e100b3640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:42916/0 (socket says 172.21.15.6:42916) 2024-03-04T09:29:33.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.995+0000 7f9e15600640 1 -- 172.21.15.6:0/1313558555 learned_addr learned my addr 172.21.15.6:0/1313558555 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:29:33.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.996+0000 7f9e14c00640 1 --2- 172.21.15.6:0/1313558555 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9e10147590 0x7f9e10071480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:33.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.996+0000 7f9e0f400640 1 -- 172.21.15.6:0/1313558555 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2579633231 0 0) 0x7f9e100719c0 con 0x7f9e100b2250 2024-03-04T09:29:33.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.996+0000 7f9e0f400640 1 -- 172.21.15.6:0/1313558555 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9df8003520 con 0x7f9e100b2250 2024-03-04T09:29:33.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.996+0000 7f9e0f400640 1 -- 172.21.15.6:0/1313558555 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1656791320 0 0) 0x7f9df8003520 con 0x7f9e100b2250 2024-03-04T09:29:33.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.996+0000 7f9e0f400640 1 -- 172.21.15.6:0/1313558555 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9e10147590 msgr2=0x7f9e10071480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:33.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.996+0000 7f9e0f400640 1 --2- 172.21.15.6:0/1313558555 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9e10147590 0x7f9e10071480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:33.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.996+0000 7f9e0f400640 1 -- 172.21.15.6:0/1313558555 >> v1:172.21.15.146:6789/0 conn(0x7f9e1006d8d0 legacy=0x7f9e100b1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:33.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.996+0000 7f9e0f400640 1 -- 172.21.15.6:0/1313558555 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e1006e120 con 0x7f9e100b2250 2024-03-04T09:29:33.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.996+0000 7f9e0f400640 1 -- 172.21.15.6:0/1313558555 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7f9e080019f0 con 0x7f9e100b2250 2024-03-04T09:29:33.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.996+0000 7f9e0f400640 1 -- 172.21.15.6:0/1313558555 >> v1:172.21.15.6:6789/0 conn(0x7f9e100b2250 legacy=0x7f9e100b3640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:33.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.996+0000 7f9e0f400640 1 --2- 172.21.15.6:0/1313558555 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9df80039e0 0x7f9df8003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:33.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.996+0000 7f9e0f400640 1 --2- 172.21.15.6:0/1313558555 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9df8004800 0x7f9df8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:33.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.996+0000 7f9e0f400640 1 --2- 172.21.15.6:0/1313558555 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9e10147590 0x7f9df8005250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:33.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.996+0000 7f9e0f400640 1 -- 172.21.15.6:0/1313558555 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e10147970 con 0x7f9df8004800 2024-03-04T09:29:33.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.996+0000 7f9e0f400640 1 -- 172.21.15.6:0/1313558555 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9df800bb50 con 0x7f9df80039e0 2024-03-04T09:29:33.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.996+0000 7f9e0f400640 1 -- 172.21.15.6:0/1313558555 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9df800bd50 con 0x7f9e10147590 2024-03-04T09:29:33.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.997+0000 7f9e0fe00640 1 --2- 172.21.15.6:0/1313558555 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9df8004800 0x7f9df8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:33.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.997+0000 7f9e15600640 1 --2- 172.21.15.6:0/1313558555 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9e10147590 0x7f9df8005250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:33.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.997+0000 7f9e14c00640 1 --2- 172.21.15.6:0/1313558555 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9df80039e0 0x7f9df8003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:33.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.997+0000 7f9e0fe00640 1 -- 172.21.15.6:0/1313558555 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9e10147590 msgr2=0x7f9df8005250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:33.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.997+0000 7f9e0fe00640 1 --2- 172.21.15.6:0/1313558555 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9e10147590 0x7f9df8005250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:33.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.997+0000 7f9e0fe00640 1 -- 172.21.15.6:0/1313558555 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9df80039e0 msgr2=0x7f9df8003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:33.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.997+0000 7f9e0fe00640 1 --2- 172.21.15.6:0/1313558555 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9df80039e0 0x7f9df8003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:33.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.997+0000 7f9e0fe00640 1 -- 172.21.15.6:0/1313558555 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9df8003520 con 0x7f9df8004800 2024-03-04T09:29:33.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.997+0000 7f9e14c00640 1 --2- 172.21.15.6:0/1313558555 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9df80039e0 0x7f9df8003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:29:33.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.998+0000 7f9e0fe00640 1 --2- 172.21.15.6:0/1313558555 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9df8004800 0x7f9df8004c80 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f9dfc001750 tx=0x7f9df8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c6b3de877af33b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:33.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.999+0000 7f9e0f400640 1 -- 172.21.15.6:0/1313558555 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dfc001370 con 0x7f9df8004800 2024-03-04T09:29:33.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.999+0000 7f9e0f400640 1 -- 172.21.15.6:0/1313558555 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dfc0052d0 con 0x7f9df8004800 2024-03-04T09:29:33.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.999+0000 7f9e0f400640 1 -- 172.21.15.6:0/1313558555 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dfc003830 con 0x7f9df8004800 2024-03-04T09:29:33.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.999+0000 7f9e16000640 1 -- 172.21.15.6:0/1313558555 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9df8004800 msgr2=0x7f9df8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:33.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.999+0000 7f9e16000640 1 --2- 172.21.15.6:0/1313558555 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9df8004800 0x7f9df8004c80 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f9dfc001750 tx=0x7f9df8005830 comp rx=0 tx=0).stop 2024-03-04T09:29:33.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.999+0000 7f9e15600640 1 -- 172.21.15.6:0/1313558555 reap_dead start 2024-03-04T09:29:33.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.999+0000 7f9e16000640 1 -- 172.21.15.6:0/1313558555 shutdown_connections 2024-03-04T09:29:33.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.999+0000 7f9e16000640 1 -- 172.21.15.6:0/1313558555 >> 172.21.15.6:0/1313558555 conn(0x7f9e1006b3a0 msgr2=0x7f9e1006b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:29:33.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.999+0000 7f9e16000640 1 -- 172.21.15.6:0/1313558555 shutdown_connections 2024-03-04T09:29:33.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:33.999+0000 7f9e16000640 1 -- 172.21.15.6:0/1313558555 wait complete. 2024-03-04T09:29:33.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:34.001+0000 7f9e16000640 1 Processor -- start 2024-03-04T09:29:33.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:34.001+0000 7f9e16000640 1 -- start start 2024-03-04T09:29:34.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:34.002+0000 7f9e16000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9df8004800 0x7f9e101f4690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:34.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:34.002+0000 7f9e16000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9e101429e0 0x7f9e101f3540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:34.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:34.002+0000 7f9e16000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9e10142e60 0x7f9e101f3a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:34.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:34.002+0000 7f9e16000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e100b2b50 con 0x7f9df8004800 2024-03-04T09:29:34.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:34.002+0000 7f9e16000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e100b31b0 con 0x7f9e10142e60 2024-03-04T09:29:34.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:34.002+0000 7f9e16000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e10071480 con 0x7f9e101429e0 2024-03-04T09:29:34.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:34.002+0000 7f9e14c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9df8004800 0x7f9e101f4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:34.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:34.002+0000 7f9e14c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9df8004800 0x7f9e101f4690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:52974/0 (socket says 172.21.15.6:52974) 2024-03-04T09:29:34.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:34.002+0000 7f9e14c00640 1 -- 172.21.15.6:0/895955845 learned_addr learned my addr 172.21.15.6:0/895955845 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:29:34.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:34.003+0000 7f9e0fe00640 1 --2- 172.21.15.6:0/895955845 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9e101429e0 0x7f9e101f3540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:34.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:34.003+0000 7f9e14c00640 1 -- 172.21.15.6:0/895955845 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9e101429e0 msgr2=0x7f9e101f3540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:34.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:34.003+0000 7f9e14c00640 1 --2- 172.21.15.6:0/895955845 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9e101429e0 0x7f9e101f3540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:34.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:34.003+0000 7f9e14c00640 1 -- 172.21.15.6:0/895955845 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9e10142e60 msgr2=0x7f9e101f3a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:34.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:34.003+0000 7f9e14c00640 1 --2- 172.21.15.6:0/895955845 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9e10142e60 0x7f9e101f3a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:34.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:34.003+0000 7f9e14c00640 1 -- 172.21.15.6:0/895955845 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e101f40b0 con 0x7f9df8004800 2024-03-04T09:29:34.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:34.003+0000 7f9e14c00640 1 --2- 172.21.15.6:0/895955845 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9df8004800 0x7f9e101f4690 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f9e0400e7e0 tx=0x7f9e0400eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:34.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:34.003+0000 7f9e0cc00640 1 -- 172.21.15.6:0/895955845 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e04017070 con 0x7f9df8004800 2024-03-04T09:29:34.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:34.003+0000 7f9e0cc00640 1 -- 172.21.15.6:0/895955845 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e0400c670 con 0x7f9df8004800 2024-03-04T09:29:34.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:34.003+0000 7f9e0cc00640 1 -- 172.21.15.6:0/895955845 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e0400f040 con 0x7f9df8004800 2024-03-04T09:29:34.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:34.003+0000 7f9e16000640 1 -- 172.21.15.6:0/895955845 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e101f4390 con 0x7f9df8004800 2024-03-04T09:29:34.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:34.004+0000 7f9e16000640 1 -- 172.21.15.6:0/895955845 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e10160240 con 0x7f9df8004800 2024-03-04T09:29:34.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:34.005+0000 7f9e16000640 1 -- 172.21.15.6:0/895955845 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e1004f330 con 0x7f9df8004800 2024-03-04T09:29:34.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:34.006+0000 7f9e0cc00640 1 -- 172.21.15.6:0/895955845 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9e0401a070 con 0x7f9df8004800 2024-03-04T09:29:34.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:34.008+0000 7f9e0cc00640 1 --2- 172.21.15.6:0/895955845 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f9df006b4d0 0x7f9df006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:34.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:34.008+0000 7f9e0cc00640 1 -- 172.21.15.6:0/895955845 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5911+0+0 (secure 0 0 0) 0x7f9e04089e80 con 0x7f9df8004800 2024-03-04T09:29:34.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:34.009+0000 7f9e0fe00640 1 --2- 172.21.15.6:0/895955845 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f9df006b4d0 0x7f9df006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:34.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:34.012+0000 7f9e0fe00640 1 --2- 172.21.15.6:0/895955845 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f9df006b4d0 0x7f9df006d990 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f9dfc001750 tx=0x7f9dfc009030 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:34.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:34.012+0000 7f9e0cc00640 1 -- 172.21.15.6:0/895955845 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9e04058c30 con 0x7f9df8004800 2024-03-04T09:29:34.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:34.252+0000 7f9e16000640 1 -- 172.21.15.6:0/895955845 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f9e10074ab0 con 0x7f9df8004800 2024-03-04T09:29:34.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:34.762+0000 7f9e0cc00640 1 -- 172.21.15.6:0/895955845 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v44) v1 ==== 104+0+0 (secure 0 0 0) 0x7f9e04058530 con 0x7f9df8004800 2024-03-04T09:29:34.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:34.765+0000 7f9e16000640 1 -- 172.21.15.6:0/895955845 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f9df006b4d0 msgr2=0x7f9df006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:34.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:34.765+0000 7f9e16000640 1 --2- 172.21.15.6:0/895955845 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f9df006b4d0 0x7f9df006d990 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f9dfc001750 tx=0x7f9dfc009030 comp rx=0 tx=0).stop 2024-03-04T09:29:34.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:34.765+0000 7f9e16000640 1 -- 172.21.15.6:0/895955845 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9df8004800 msgr2=0x7f9e101f4690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:34.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:34.765+0000 7f9e16000640 1 --2- 172.21.15.6:0/895955845 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9df8004800 0x7f9e101f4690 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f9e0400e7e0 tx=0x7f9e0400eca0 comp rx=0 tx=0).stop 2024-03-04T09:29:34.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:34.765+0000 7f9e16000640 1 -- 172.21.15.6:0/895955845 shutdown_connections 2024-03-04T09:29:34.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:34.765+0000 7f9e16000640 1 --2- 172.21.15.6:0/895955845 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9e10142e60 0x7f9e101f3a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:34.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:34.765+0000 7f9e16000640 1 --2- 172.21.15.6:0/895955845 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f9df006b4d0 0x7f9df006d990 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:34.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:34.765+0000 7f9e16000640 1 --2- 172.21.15.6:0/895955845 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9e101429e0 0x7f9e101f3540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:34.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:34.765+0000 7f9e16000640 1 --2- 172.21.15.6:0/895955845 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9df8004800 0x7f9e101f4690 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:34.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:34.765+0000 7f9e16000640 1 -- 172.21.15.6:0/895955845 >> 172.21.15.6:0/895955845 conn(0x7f9e1006b3a0 msgr2=0x7f9e100af270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:29:34.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:34.765+0000 7f9e16000640 1 -- 172.21.15.6:0/895955845 shutdown_connections 2024-03-04T09:29:34.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:34.766+0000 7f9e16000640 1 -- 172.21.15.6:0/895955845 wait complete. 2024-03-04T09:29:34.783 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-04T09:29:35.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:35.013+0000 7f7c76a00640 1 Processor -- start 2024-03-04T09:29:35.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:35.013+0000 7f7c76a00640 1 -- start start 2024-03-04T09:29:35.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:35.015+0000 7f7c76a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7c70161af0 0x7f7c7006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:35.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:35.015+0000 7f7c76a00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f7c7006b290 0x7f7c7006b720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:35.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:35.015+0000 7f7c76a00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c70162960 con 0x7f7c700b4040 2024-03-04T09:29:35.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:35.015+0000 7f7c76a00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c700b5150 con 0x7f7c7006b290 2024-03-04T09:29:35.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:35.015+0000 7f7c76a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c7006f410 con 0x7f7c70161af0 2024-03-04T09:29:35.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:35.016+0000 7f7c74a00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f7c7006b290 0x7f7c7006b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:35.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:35.016+0000 7f7c6fe00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7c70161af0 0x7f7c7006eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:35.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:35.016+0000 7f7c74a00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f7c7006b290 0x7f7c7006b720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.6:39764/0 (socket says 172.21.15.6:39764) 2024-03-04T09:29:35.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:35.016+0000 7f7c74a00640 1 -- 172.21.15.6:0/3423275977 learned_addr learned my addr 172.21.15.6:0/3423275977 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:29:35.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:35.017+0000 7f7c6fe00640 1 -- 172.21.15.6:0/3423275977 >> v2:172.21.15.146:3300/0 conn(0x7f7c7006b290 msgr2=0x7f7c7006b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:35.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:35.017+0000 7f7c6fe00640 1 --2- 172.21.15.6:0/3423275977 >> v2:172.21.15.146:3300/0 conn(0x7f7c7006b290 0x7f7c7006b720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:35.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:35.017+0000 7f7c6fe00640 1 -- 172.21.15.6:0/3423275977 >> v1:172.21.15.146:6789/0 conn(0x7f7c700b4040 legacy=0x7f7c70161ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:35.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:35.017+0000 7f7c6fe00640 1 -- 172.21.15.6:0/3423275977 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c700b4540 con 0x7f7c70161af0 2024-03-04T09:29:35.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:35.017+0000 7f7c74a00640 1 --2- 172.21.15.6:0/3423275977 >> v2:172.21.15.146:3300/0 conn(0x7f7c7006b290 0x7f7c7006b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T09:29:35.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:35.018+0000 7f7c6fe00640 1 --2- 172.21.15.6:0/3423275977 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7c70161af0 0x7f7c7006eed0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f7c60009d90 tx=0x7f7c6002beb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=47a3e0d7116e533b server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:35.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:35.019+0000 7f7c6ea00640 1 -- 172.21.15.6:0/3423275977 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c60039070 con 0x7f7c70161af0 2024-03-04T09:29:35.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:35.019+0000 7f7c6ea00640 1 -- 172.21.15.6:0/3423275977 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c6002cc60 con 0x7f7c70161af0 2024-03-04T09:29:35.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:35.019+0000 7f7c6ea00640 1 -- 172.21.15.6:0/3423275977 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c60034660 con 0x7f7c70161af0 2024-03-04T09:29:35.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:35.020+0000 7f7c76a00640 1 -- 172.21.15.6:0/3423275977 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7c70161af0 msgr2=0x7f7c7006eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:35.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:35.020+0000 7f7c76a00640 1 --2- 172.21.15.6:0/3423275977 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7c70161af0 0x7f7c7006eed0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f7c60009d90 tx=0x7f7c6002beb0 comp rx=0 tx=0).stop 2024-03-04T09:29:35.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:35.020+0000 7f7c76a00640 1 -- 172.21.15.6:0/3423275977 shutdown_connections 2024-03-04T09:29:35.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:35.020+0000 7f7c76a00640 1 --2- 172.21.15.6:0/3423275977 >> v2:172.21.15.146:3300/0 conn(0x7f7c7006b290 0x7f7c7006b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:35.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:35.020+0000 7f7c76a00640 1 --2- 172.21.15.6:0/3423275977 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7c70161af0 0x7f7c7006eed0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:35.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:35.020+0000 7f7c76a00640 1 -- 172.21.15.6:0/3423275977 >> 172.21.15.6:0/3423275977 conn(0x7f7c700b2400 msgr2=0x7f7c700b27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:29:35.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:35.020+0000 7f7c76a00640 1 -- 172.21.15.6:0/3423275977 shutdown_connections 2024-03-04T09:29:35.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:35.020+0000 7f7c76a00640 1 -- 172.21.15.6:0/3423275977 wait complete. 2024-03-04T09:29:35.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:35.022+0000 7f7c76a00640 1 Processor -- start 2024-03-04T09:29:35.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:35.023+0000 7f7c76a00640 1 -- start start 2024-03-04T09:29:35.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:35.024+0000 7f7c76a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7c7006b290 0x7f7c701418d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:35.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:35.024+0000 7f7c76a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7c700b4040 0x7f7c7015adf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:35.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:35.024+0000 7f7c76a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7c70161af0 0x7f7c7015d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:35.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:35.024+0000 7f7c76a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c700b4de0 con 0x7f7c70161af0 2024-03-04T09:29:35.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:35.024+0000 7f7c76a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c700b4b00 con 0x7f7c7006b290 2024-03-04T09:29:35.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:35.024+0000 7f7c76a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c700afe70 con 0x7f7c700b4040 2024-03-04T09:29:35.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:35.024+0000 7f7c74a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7c70161af0 0x7f7c7015d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:35.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:35.024+0000 7f7c74a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7c70161af0 0x7f7c7015d5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:52984/0 (socket says 172.21.15.6:52984) 2024-03-04T09:29:35.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:35.024+0000 7f7c74a00640 1 -- 172.21.15.6:0/3309660715 learned_addr learned my addr 172.21.15.6:0/3309660715 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:29:35.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:35.024+0000 7f7c6f400640 1 --2- 172.21.15.6:0/3309660715 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7c700b4040 0x7f7c7015adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:35.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:35.024+0000 7f7c6fe00640 1 --2- 172.21.15.6:0/3309660715 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7c7006b290 0x7f7c701418d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:35.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:35.025+0000 7f7c74a00640 1 -- 172.21.15.6:0/3309660715 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7c700b4040 msgr2=0x7f7c7015adf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:35.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:35.025+0000 7f7c74a00640 1 --2- 172.21.15.6:0/3309660715 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7c700b4040 0x7f7c7015adf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:35.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:35.025+0000 7f7c74a00640 1 -- 172.21.15.6:0/3309660715 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7c7006b290 msgr2=0x7f7c701418d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:35.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:35.025+0000 7f7c74a00640 1 --2- 172.21.15.6:0/3309660715 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7c7006b290 0x7f7c701418d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:35.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:35.025+0000 7f7c74a00640 1 -- 172.21.15.6:0/3309660715 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c701557a0 con 0x7f7c70161af0 2024-03-04T09:29:35.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:35.025+0000 7f7c74a00640 1 --2- 172.21.15.6:0/3309660715 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7c70161af0 0x7f7c7015d5a0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f7c6800e890 tx=0x7f7c6800ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:35.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:35.025+0000 7f7c67e00640 1 -- 172.21.15.6:0/3309660715 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c68017070 con 0x7f7c70161af0 2024-03-04T09:29:35.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:35.025+0000 7f7c67e00640 1 -- 172.21.15.6:0/3309660715 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c68004020 con 0x7f7c70161af0 2024-03-04T09:29:35.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:35.025+0000 7f7c67e00640 1 -- 172.21.15.6:0/3309660715 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c680043f0 con 0x7f7c70161af0 2024-03-04T09:29:35.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:35.025+0000 7f7c76a00640 1 -- 172.21.15.6:0/3309660715 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c7015e3b0 con 0x7f7c70161af0 2024-03-04T09:29:35.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:35.026+0000 7f7c76a00640 1 -- 172.21.15.6:0/3309660715 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c7015b330 con 0x7f7c70161af0 2024-03-04T09:29:35.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:35.027+0000 7f7c76a00640 1 -- 172.21.15.6:0/3309660715 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c70068010 con 0x7f7c70161af0 2024-03-04T09:29:35.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:35.028+0000 7f7c67e00640 1 -- 172.21.15.6:0/3309660715 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7c6800c940 con 0x7f7c70161af0 2024-03-04T09:29:35.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:35.029+0000 7f7c67e00640 1 --2- 172.21.15.6:0/3309660715 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f7c4c06b4d0 0x7f7c4c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:35.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:35.029+0000 7f7c67e00640 1 -- 172.21.15.6:0/3309660715 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5911+0+0 (secure 0 0 0) 0x7f7c6808a3c0 con 0x7f7c70161af0 2024-03-04T09:29:35.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:35.030+0000 7f7c6fe00640 1 --2- 172.21.15.6:0/3309660715 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f7c4c06b4d0 0x7f7c4c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:35.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:35.031+0000 7f7c6fe00640 1 --2- 172.21.15.6:0/3309660715 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f7c4c06b4d0 0x7f7c4c06d990 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f7c60036000 tx=0x7f7c60030000 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:35.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:35.034+0000 7f7c67e00640 1 -- 172.21.15.6:0/3309660715 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7c680591e0 con 0x7f7c70161af0 2024-03-04T09:29:35.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:35.239+0000 7f7c76a00640 1 -- 172.21.15.6:0/3309660715 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f7c7006dd90 con 0x7f7c70161af0 2024-03-04T09:29:35.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:35.765+0000 7f7c67e00640 1 -- 172.21.15.6:0/3309660715 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v72) v1 ==== 181+0+0 (secure 0 0 0) 0x7f7c68058ae0 con 0x7f7c70161af0 2024-03-04T09:29:35.762 INFO:teuthology.orchestra.run.smithi006.stderr:pool 'cephfs_metadata' removed 2024-03-04T09:29:35.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:35.769+0000 7f7c76a00640 1 -- 172.21.15.6:0/3309660715 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f7c4c06b4d0 msgr2=0x7f7c4c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:35.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:35.769+0000 7f7c76a00640 1 --2- 172.21.15.6:0/3309660715 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f7c4c06b4d0 0x7f7c4c06d990 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f7c60036000 tx=0x7f7c60030000 comp rx=0 tx=0).stop 2024-03-04T09:29:35.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:35.769+0000 7f7c76a00640 1 -- 172.21.15.6:0/3309660715 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7c70161af0 msgr2=0x7f7c7015d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:35.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:35.769+0000 7f7c76a00640 1 --2- 172.21.15.6:0/3309660715 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7c70161af0 0x7f7c7015d5a0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f7c6800e890 tx=0x7f7c6800ed50 comp rx=0 tx=0).stop 2024-03-04T09:29:35.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:35.770+0000 7f7c76a00640 1 -- 172.21.15.6:0/3309660715 shutdown_connections 2024-03-04T09:29:35.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:35.770+0000 7f7c76a00640 1 --2- 172.21.15.6:0/3309660715 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7c70161af0 0x7f7c7015d5a0 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:35.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:35.770+0000 7f7c76a00640 1 --2- 172.21.15.6:0/3309660715 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f7c4c06b4d0 0x7f7c4c06d990 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:35.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:35.770+0000 7f7c76a00640 1 --2- 172.21.15.6:0/3309660715 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7c700b4040 0x7f7c7015adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:35.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:35.770+0000 7f7c76a00640 1 --2- 172.21.15.6:0/3309660715 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7c7006b290 0x7f7c701418d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:35.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:35.770+0000 7f7c76a00640 1 -- 172.21.15.6:0/3309660715 >> 172.21.15.6:0/3309660715 conn(0x7f7c700b2400 msgr2=0x7f7c700b3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:29:35.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:35.770+0000 7f7c76a00640 1 -- 172.21.15.6:0/3309660715 shutdown_connections 2024-03-04T09:29:35.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:35.770+0000 7f7c76a00640 1 -- 172.21.15.6:0/3309660715 wait complete. 2024-03-04T09:29:35.794 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-04T09:29:35.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:35.995+0000 7f401ae00640 1 Processor -- start 2024-03-04T09:29:35.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:35.996+0000 7f401ae00640 1 -- start start 2024-03-04T09:29:35.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:35.997+0000 7f401ae00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f4014159b10 0x7f401406ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:35.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:35.997+0000 7f401ae00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f401406f460 con 0x7f40140b5e30 2024-03-04T09:29:35.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:35.997+0000 7f401ae00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40140af550 con 0x7f40140b4ba0 2024-03-04T09:29:35.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:35.997+0000 7f401ae00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40140af320 con 0x7f4014159b10 2024-03-04T09:29:35.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:35.998+0000 7f4018e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f40140b5e30 0x7f40140b05f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:42932/0 (socket says 172.21.15.6:42932) 2024-03-04T09:29:35.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:35.998+0000 7f4018e00640 1 -- 172.21.15.6:0/208065510 learned_addr learned my addr 172.21.15.6:0/208065510 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:29:35.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:35.998+0000 7f4013e00640 1 --2- 172.21.15.6:0/208065510 >> v2:172.21.15.146:3300/0 conn(0x7f4014159b10 0x7f401406ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:35.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:35.998+0000 7f4012a00640 1 -- 172.21.15.6:0/208065510 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2362762101 0 0) 0x7f401406f460 con 0x7f40140b5e30 2024-03-04T09:29:35.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:35.998+0000 7f4012a00640 1 -- 172.21.15.6:0/208065510 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ff4003520 con 0x7f40140b5e30 2024-03-04T09:29:35.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:35.998+0000 7f4012a00640 1 -- 172.21.15.6:0/208065510 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1928337477 0 0) 0x7f3ff4003520 con 0x7f40140b5e30 2024-03-04T09:29:35.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:35.998+0000 7f4012a00640 1 -- 172.21.15.6:0/208065510 >> v2:172.21.15.146:3300/0 conn(0x7f4014159b10 msgr2=0x7f401406ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:35.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:35.999+0000 7f4012a00640 1 --2- 172.21.15.6:0/208065510 >> v2:172.21.15.146:3300/0 conn(0x7f4014159b10 0x7f401406ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:35.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:35.999+0000 7f4012a00640 1 -- 172.21.15.6:0/208065510 >> v1:172.21.15.146:6789/0 conn(0x7f40140b4ba0 legacy=0x7f40140b5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:35.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:35.999+0000 7f4012a00640 1 -- 172.21.15.6:0/208065510 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40140af830 con 0x7f40140b5e30 2024-03-04T09:29:35.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:35.999+0000 7f4012a00640 1 -- 172.21.15.6:0/208065510 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7f400c001670 con 0x7f40140b5e30 2024-03-04T09:29:35.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:35.999+0000 7f4012a00640 1 -- 172.21.15.6:0/208065510 >> v1:172.21.15.6:6789/0 conn(0x7f40140b5e30 legacy=0x7f40140b05f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:35.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:35.999+0000 7f4012a00640 1 --2- 172.21.15.6:0/208065510 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3ff40039b0 0x7f3ff4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:35.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:35.999+0000 7f4012a00640 1 --2- 172.21.15.6:0/208065510 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3ff4004800 0x7f3ff4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:35.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:35.999+0000 7f4012a00640 1 --2- 172.21.15.6:0/208065510 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3ff40051c0 0x7f3ff400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:35.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:35.999+0000 7f4012a00640 1 -- 172.21.15.6:0/208065510 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ff400c130 con 0x7f3ff4004800 2024-03-04T09:29:35.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:35.999+0000 7f4012a00640 1 -- 172.21.15.6:0/208065510 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ff400c2e0 con 0x7f3ff40051c0 2024-03-04T09:29:35.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:35.999+0000 7f4012a00640 1 -- 172.21.15.6:0/208065510 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ff400c4e0 con 0x7f3ff40039b0 2024-03-04T09:29:35.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:35.999+0000 7f4013400640 1 --2- 172.21.15.6:0/208065510 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3ff4004800 0x7f3ff4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:35.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:35.999+0000 7f4013400640 1 -- 172.21.15.6:0/208065510 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3ff40039b0 msgr2=0x7f3ff4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:35.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:35.999+0000 7f4013400640 1 --2- 172.21.15.6:0/208065510 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3ff40039b0 0x7f3ff4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:35.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:35.999+0000 7f4013400640 1 -- 172.21.15.6:0/208065510 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3ff40051c0 msgr2=0x7f3ff400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:35.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:35.999+0000 7f4013400640 1 --2- 172.21.15.6:0/208065510 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3ff40051c0 0x7f3ff400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:35.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:35.999+0000 7f4013400640 1 -- 172.21.15.6:0/208065510 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ff4003520 con 0x7f3ff4004800 2024-03-04T09:29:35.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:35.999+0000 7f4018e00640 1 -- 172.21.15.6:0/208065510 reap_dead start 2024-03-04T09:29:35.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:36.001+0000 7f4013400640 1 --2- 172.21.15.6:0/208065510 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3ff4004800 0x7f3ff4004c80 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f3ffc001e50 tx=0x7f3ff4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0311ea6f0e5c7ba server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:35.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:36.001+0000 7f4012a00640 1 -- 172.21.15.6:0/208065510 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ffc001a70 con 0x7f3ff4004800 2024-03-04T09:29:35.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:36.001+0000 7f4012a00640 1 -- 172.21.15.6:0/208065510 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ffc003b70 con 0x7f3ff4004800 2024-03-04T09:29:35.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:36.001+0000 7f4012a00640 1 -- 172.21.15.6:0/208065510 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ffc0050d0 con 0x7f3ff4004800 2024-03-04T09:29:35.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:36.001+0000 7f401ae00640 1 -- 172.21.15.6:0/208065510 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3ff4004800 msgr2=0x7f3ff4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:35.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:36.001+0000 7f401ae00640 1 --2- 172.21.15.6:0/208065510 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3ff4004800 0x7f3ff4004c80 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f3ffc001e50 tx=0x7f3ff4005830 comp rx=0 tx=0).stop 2024-03-04T09:29:35.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:36.001+0000 7f401ae00640 1 -- 172.21.15.6:0/208065510 shutdown_connections 2024-03-04T09:29:35.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:36.001+0000 7f401ae00640 1 --2- 172.21.15.6:0/208065510 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3ff4004800 0x7f3ff4004c80 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:35.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:36.001+0000 7f401ae00640 1 -- 172.21.15.6:0/208065510 >> 172.21.15.6:0/208065510 conn(0x7f401409aa10 msgr2=0x7f401409ae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:29:36.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:36.002+0000 7f401ae00640 1 -- 172.21.15.6:0/208065510 shutdown_connections 2024-03-04T09:29:36.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:36.002+0000 7f401ae00640 1 -- 172.21.15.6:0/208065510 wait complete. 2024-03-04T09:29:36.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:36.003+0000 7f401ae00640 1 Processor -- start 2024-03-04T09:29:36.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:36.003+0000 7f401ae00640 1 -- start start 2024-03-04T09:29:36.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:36.004+0000 7f401ae00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3ff4004800 0x7f40140cd390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:36.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:36.004+0000 7f401ae00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f401406dd40 0x7f4014148750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:36.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:36.004+0000 7f401ae00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f401415acb0 0x7f401415b160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:36.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:36.004+0000 7f401ae00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40140b0160 con 0x7f401406dd40 2024-03-04T09:29:36.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:36.004+0000 7f401ae00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4014159ef0 con 0x7f3ff4004800 2024-03-04T09:29:36.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:36.004+0000 7f401ae00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40140afdf0 con 0x7f401415acb0 2024-03-04T09:29:36.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:36.005+0000 7f4013400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f401406dd40 0x7f4014148750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:36.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:36.005+0000 7f4013400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f401406dd40 0x7f4014148750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53010/0 (socket says 172.21.15.6:53010) 2024-03-04T09:29:36.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:36.005+0000 7f4013400640 1 -- 172.21.15.6:0/4076685727 learned_addr learned my addr 172.21.15.6:0/4076685727 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:29:36.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:36.005+0000 7f4013e00640 1 --2- 172.21.15.6:0/4076685727 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3ff4004800 0x7f40140cd390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:36.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:36.005+0000 7f4018e00640 1 --2- 172.21.15.6:0/4076685727 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f401415acb0 0x7f401415b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:36.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:36.005+0000 7f4013400640 1 -- 172.21.15.6:0/4076685727 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f401415acb0 msgr2=0x7f401415b160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:36.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:36.005+0000 7f4013400640 1 --2- 172.21.15.6:0/4076685727 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f401415acb0 0x7f401415b160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:36.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:36.005+0000 7f4013400640 1 -- 172.21.15.6:0/4076685727 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3ff4004800 msgr2=0x7f40140cd390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:36.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:36.005+0000 7f4013400640 1 --2- 172.21.15.6:0/4076685727 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3ff4004800 0x7f40140cd390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:36.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:36.005+0000 7f4013400640 1 -- 172.21.15.6:0/4076685727 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40140b0720 con 0x7f401406dd40 2024-03-04T09:29:36.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:36.006+0000 7f4013400640 1 --2- 172.21.15.6:0/4076685727 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f401406dd40 0x7f4014148750 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f3ffc000f80 tx=0x7f3ffc0014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:36.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:36.006+0000 7f400be00640 1 -- 172.21.15.6:0/4076685727 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ffc003970 con 0x7f401406dd40 2024-03-04T09:29:36.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:36.006+0000 7f400be00640 1 -- 172.21.15.6:0/4076685727 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ffc006b50 con 0x7f401406dd40 2024-03-04T09:29:36.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:36.006+0000 7f400be00640 1 -- 172.21.15.6:0/4076685727 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ffc007020 con 0x7f401406dd40 2024-03-04T09:29:36.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:36.006+0000 7f401ae00640 1 -- 172.21.15.6:0/4076685727 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4014148c90 con 0x7f401406dd40 2024-03-04T09:29:36.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:36.006+0000 7f401ae00640 1 -- 172.21.15.6:0/4076685727 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40140cc180 con 0x7f401406dd40 2024-03-04T09:29:36.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:36.007+0000 7f401ae00640 1 -- 172.21.15.6:0/4076685727 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4014068010 con 0x7f401406dd40 2024-03-04T09:29:36.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:36.008+0000 7f400be00640 1 -- 172.21.15.6:0/4076685727 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3ffc049060 con 0x7f401406dd40 2024-03-04T09:29:36.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:36.009+0000 7f400be00640 1 --2- 172.21.15.6:0/4076685727 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f3ff006b4d0 0x7f3ff006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:36.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:36.009+0000 7f400be00640 1 -- 172.21.15.6:0/4076685727 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5473+0+0 (secure 0 0 0) 0x7f3ffc0ab440 con 0x7f401406dd40 2024-03-04T09:29:36.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:36.010+0000 7f4013e00640 1 --2- 172.21.15.6:0/4076685727 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f3ff006b4d0 0x7f3ff006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:36.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:36.011+0000 7f4013e00640 1 --2- 172.21.15.6:0/4076685727 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f3ff006b4d0 0x7f3ff006d990 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f4014050c30 tx=0x7f4004008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:36.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:36.014+0000 7f400be00640 1 -- 172.21.15.6:0/4076685727 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3ffc033020 con 0x7f401406dd40 2024-03-04T09:29:36.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:36.218+0000 7f401ae00640 1 -- 172.21.15.6:0/4076685727 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f40140cb1d0 con 0x7f401406dd40 2024-03-04T09:29:36.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:36.777+0000 7f400be00640 1 -- 172.21.15.6:0/4076685727 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v73) v1 ==== 169+0+0 (secure 0 0 0) 0x7f3ffc07a410 con 0x7f401406dd40 2024-03-04T09:29:36.775 INFO:teuthology.orchestra.run.smithi006.stderr:pool 'cephfs_data' removed 2024-03-04T09:29:36.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:36.781+0000 7f401ae00640 1 -- 172.21.15.6:0/4076685727 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f3ff006b4d0 msgr2=0x7f3ff006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:36.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:36.781+0000 7f401ae00640 1 --2- 172.21.15.6:0/4076685727 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f3ff006b4d0 0x7f3ff006d990 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f4014050c30 tx=0x7f4004008040 comp rx=0 tx=0).stop 2024-03-04T09:29:36.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:36.781+0000 7f401ae00640 1 -- 172.21.15.6:0/4076685727 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f401406dd40 msgr2=0x7f4014148750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:36.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:36.781+0000 7f401ae00640 1 --2- 172.21.15.6:0/4076685727 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f401406dd40 0x7f4014148750 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f3ffc000f80 tx=0x7f3ffc0014a0 comp rx=0 tx=0).stop 2024-03-04T09:29:36.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:36.782+0000 7f401ae00640 1 -- 172.21.15.6:0/4076685727 shutdown_connections 2024-03-04T09:29:36.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:36.782+0000 7f401ae00640 1 --2- 172.21.15.6:0/4076685727 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f3ff006b4d0 0x7f3ff006d990 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:36.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:36.782+0000 7f401ae00640 1 --2- 172.21.15.6:0/4076685727 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f401415acb0 0x7f401415b160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:36.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:36.782+0000 7f401ae00640 1 --2- 172.21.15.6:0/4076685727 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f401406dd40 0x7f4014148750 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:36.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:36.782+0000 7f401ae00640 1 --2- 172.21.15.6:0/4076685727 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3ff4004800 0x7f40140cd390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:36.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:36.782+0000 7f401ae00640 1 -- 172.21.15.6:0/4076685727 >> 172.21.15.6:0/4076685727 conn(0x7f401409aa10 msgr2=0x7f4014154b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:29:36.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:36.782+0000 7f401ae00640 1 -- 172.21.15.6:0/4076685727 shutdown_connections 2024-03-04T09:29:36.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:36.783+0000 7f401ae00640 1 -- 172.21.15.6:0/4076685727 wait complete. 2024-03-04T09:29:36.806 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds mds_bal_fragment_size_max 2024-03-04T09:29:37.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.040+0000 7fa2e6600640 1 Processor -- start 2024-03-04T09:29:37.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.040+0000 7fa2e6600640 1 -- start start 2024-03-04T09:29:37.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.041+0000 7fa2e6600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa2e0161b10 0x7fa2e006ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:37.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.042+0000 7fa2e6600640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fa2e00b3960 0x7fa2e00b3df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:37.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.042+0000 7fa2e6600640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2e0067030 con 0x7fa2e0161ef0 2024-03-04T09:29:37.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.042+0000 7fa2e6600640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2e006f3e0 con 0x7fa2e00b3960 2024-03-04T09:29:37.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.042+0000 7fa2e6600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e006f6c0 con 0x7fa2e0161b10 2024-03-04T09:29:37.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.042+0000 7fa2dea00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fa2e0161ef0 0x7fa2e0162350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:42948/0 (socket says 172.21.15.6:42948) 2024-03-04T09:29:37.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.042+0000 7fa2dea00640 1 -- 172.21.15.6:0/3900534235 learned_addr learned my addr 172.21.15.6:0/3900534235 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:29:37.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.042+0000 7fa2dfe00640 1 --2- 172.21.15.6:0/3900534235 >> v2:172.21.15.146:3300/0 conn(0x7fa2e00b3960 0x7fa2e00b3df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:37.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.042+0000 7fa2df400640 1 --2- 172.21.15.6:0/3900534235 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa2e0161b10 0x7fa2e006ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:37.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.042+0000 7fa2de000640 1 -- 172.21.15.6:0/3900534235 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3879462503 0 0) 0x7fa2e0067030 con 0x7fa2e0161ef0 2024-03-04T09:29:37.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.042+0000 7fa2de000640 1 -- 172.21.15.6:0/3900534235 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2c4003520 con 0x7fa2e0161ef0 2024-03-04T09:29:37.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.043+0000 7fa2de000640 1 -- 172.21.15.6:0/3900534235 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1321182744 0 0) 0x7fa2c4003520 con 0x7fa2e0161ef0 2024-03-04T09:29:37.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.043+0000 7fa2de000640 1 -- 172.21.15.6:0/3900534235 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa2e0161b10 msgr2=0x7fa2e006ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:37.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.043+0000 7fa2de000640 1 --2- 172.21.15.6:0/3900534235 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa2e0161b10 0x7fa2e006ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:37.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.043+0000 7fa2de000640 1 -- 172.21.15.6:0/3900534235 >> v2:172.21.15.146:3300/0 conn(0x7fa2e00b3960 msgr2=0x7fa2e00b3df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:37.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.043+0000 7fa2de000640 1 --2- 172.21.15.6:0/3900534235 >> v2:172.21.15.146:3300/0 conn(0x7fa2e00b3960 0x7fa2e00b3df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:37.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.043+0000 7fa2de000640 1 -- 172.21.15.6:0/3900534235 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2e006f9a0 con 0x7fa2e0161ef0 2024-03-04T09:29:37.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.043+0000 7fa2de000640 1 -- 172.21.15.6:0/3900534235 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7fa2c8001670 con 0x7fa2e0161ef0 2024-03-04T09:29:37.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.043+0000 7fa2de000640 1 -- 172.21.15.6:0/3900534235 >> v1:172.21.15.6:6789/0 conn(0x7fa2e0161ef0 legacy=0x7fa2e0162350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:37.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.043+0000 7fa2de000640 1 --2- 172.21.15.6:0/3900534235 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa2c40039b0 0x7fa2c4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:37.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.043+0000 7fa2de000640 1 --2- 172.21.15.6:0/3900534235 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa2c4004800 0x7fa2c4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:37.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.043+0000 7fa2de000640 1 --2- 172.21.15.6:0/3900534235 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa2e0161b10 0x7fa2c40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:37.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.043+0000 7fa2de000640 1 -- 172.21.15.6:0/3900534235 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e006f3e0 con 0x7fa2c4004800 2024-03-04T09:29:37.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.043+0000 7fa2de000640 1 -- 172.21.15.6:0/3900534235 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e006f6c0 con 0x7fa2c40039b0 2024-03-04T09:29:37.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.043+0000 7fa2de000640 1 -- 172.21.15.6:0/3900534235 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2c400b9c0 con 0x7fa2e0161b10 2024-03-04T09:29:37.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.043+0000 7fa2dea00640 1 --2- 172.21.15.6:0/3900534235 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa2c4004800 0x7fa2c4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:37.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.044+0000 7fa2dfe00640 1 --2- 172.21.15.6:0/3900534235 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa2e0161b10 0x7fa2c40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:37.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.044+0000 7fa2df400640 1 --2- 172.21.15.6:0/3900534235 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa2c40039b0 0x7fa2c4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:37.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.044+0000 7fa2dea00640 1 -- 172.21.15.6:0/3900534235 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa2e0161b10 msgr2=0x7fa2c40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:37.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.044+0000 7fa2dea00640 1 --2- 172.21.15.6:0/3900534235 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa2e0161b10 0x7fa2c40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:37.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.044+0000 7fa2dea00640 1 -- 172.21.15.6:0/3900534235 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa2c40039b0 msgr2=0x7fa2c4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:37.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.044+0000 7fa2dea00640 1 --2- 172.21.15.6:0/3900534235 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa2c40039b0 0x7fa2c4003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:37.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.044+0000 7fa2dea00640 1 -- 172.21.15.6:0/3900534235 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2c4003520 con 0x7fa2c4004800 2024-03-04T09:29:37.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.045+0000 7fa2dea00640 1 --2- 172.21.15.6:0/3900534235 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa2c4004800 0x7fa2c4004c80 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7fa2c800c210 tx=0x7fa2c4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7eaa274baf8a542 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:37.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.045+0000 7fa2de000640 1 -- 172.21.15.6:0/3900534235 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2c8007d50 con 0x7fa2c4004800 2024-03-04T09:29:37.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.045+0000 7fa2de000640 1 -- 172.21.15.6:0/3900534235 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2c8030070 con 0x7fa2c4004800 2024-03-04T09:29:37.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.045+0000 7fa2de000640 1 -- 172.21.15.6:0/3900534235 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2c802cd80 con 0x7fa2c4004800 2024-03-04T09:29:37.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.046+0000 7fa2e6600640 1 -- 172.21.15.6:0/3900534235 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa2c4004800 msgr2=0x7fa2c4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:37.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.046+0000 7fa2e6600640 1 --2- 172.21.15.6:0/3900534235 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa2c4004800 0x7fa2c4004c80 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7fa2c800c210 tx=0x7fa2c4005830 comp rx=0 tx=0).stop 2024-03-04T09:29:37.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.046+0000 7fa2dfe00640 1 -- 172.21.15.6:0/3900534235 reap_dead start 2024-03-04T09:29:37.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.046+0000 7fa2e6600640 1 -- 172.21.15.6:0/3900534235 shutdown_connections 2024-03-04T09:29:37.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.046+0000 7fa2e6600640 1 -- 172.21.15.6:0/3900534235 >> 172.21.15.6:0/3900534235 conn(0x7fa2e00722d0 msgr2=0x7fa2e006afb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:29:37.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.046+0000 7fa2e6600640 1 -- 172.21.15.6:0/3900534235 shutdown_connections 2024-03-04T09:29:37.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.046+0000 7fa2e6600640 1 -- 172.21.15.6:0/3900534235 wait complete. 2024-03-04T09:29:37.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.048+0000 7fa2e6600640 1 Processor -- start 2024-03-04T09:29:37.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.048+0000 7fa2e6600640 1 -- start start 2024-03-04T09:29:37.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.049+0000 7fa2e6600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa2c4004800 0x7fa2e01f40a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:37.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.049+0000 7fa2e6600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa2e0150d50 0x7fa2e0153b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:37.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.049+0000 7fa2e6600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa2e01511d0 0x7fa2e0154090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:37.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.049+0000 7fa2e6600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e01551e0 con 0x7fa2c4004800 2024-03-04T09:29:37.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.049+0000 7fa2e6600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e00b3440 con 0x7fa2e0150d50 2024-03-04T09:29:37.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.049+0000 7fa2e6600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e0155360 con 0x7fa2e01511d0 2024-03-04T09:29:37.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.049+0000 7fa2df400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa2c4004800 0x7fa2e01f40a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:37.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.049+0000 7fa2df400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa2c4004800 0x7fa2e01f40a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53036/0 (socket says 172.21.15.6:53036) 2024-03-04T09:29:37.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.049+0000 7fa2df400640 1 -- 172.21.15.6:0/601826493 learned_addr learned my addr 172.21.15.6:0/601826493 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:29:37.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.050+0000 7fa2dea00640 1 --2- 172.21.15.6:0/601826493 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa2e0150d50 0x7fa2e0153b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:37.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.050+0000 7fa2dfe00640 1 --2- 172.21.15.6:0/601826493 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa2e01511d0 0x7fa2e0154090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:37.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.050+0000 7fa2df400640 1 -- 172.21.15.6:0/601826493 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa2e01511d0 msgr2=0x7fa2e0154090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:37.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.050+0000 7fa2df400640 1 --2- 172.21.15.6:0/601826493 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa2e01511d0 0x7fa2e0154090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:37.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.050+0000 7fa2df400640 1 -- 172.21.15.6:0/601826493 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa2e0150d50 msgr2=0x7fa2e0153b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:37.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.050+0000 7fa2df400640 1 --2- 172.21.15.6:0/601826493 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa2e0150d50 0x7fa2e0153b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:37.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.050+0000 7fa2df400640 1 -- 172.21.15.6:0/601826493 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2e0152ab0 con 0x7fa2c4004800 2024-03-04T09:29:37.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.050+0000 7fa2df400640 1 --2- 172.21.15.6:0/601826493 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa2c4004800 0x7fa2e01f40a0 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7fa2d8009fb0 tx=0x7fa2d80115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:37.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.051+0000 7fa2d7400640 1 -- 172.21.15.6:0/601826493 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2d801a020 con 0x7fa2c4004800 2024-03-04T09:29:37.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.051+0000 7fa2d7400640 1 -- 172.21.15.6:0/601826493 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2d8016470 con 0x7fa2c4004800 2024-03-04T09:29:37.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.051+0000 7fa2d7400640 1 -- 172.21.15.6:0/601826493 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2d8016860 con 0x7fa2c4004800 2024-03-04T09:29:37.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.051+0000 7fa2e6600640 1 -- 172.21.15.6:0/601826493 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2e0152d90 con 0x7fa2c4004800 2024-03-04T09:29:37.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.051+0000 7fa2e6600640 1 -- 172.21.15.6:0/601826493 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2e0153350 con 0x7fa2c4004800 2024-03-04T09:29:37.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.052+0000 7fa2e6600640 1 -- 172.21.15.6:0/601826493 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2e004f330 con 0x7fa2c4004800 2024-03-04T09:29:37.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.053+0000 7fa2d7400640 1 -- 172.21.15.6:0/601826493 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa2d8005d10 con 0x7fa2c4004800 2024-03-04T09:29:37.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.054+0000 7fa2d7400640 1 --2- 172.21.15.6:0/601826493 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa2bc06b4d0 0x7fa2bc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:37.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.054+0000 7fa2d7400640 1 -- 172.21.15.6:0/601826493 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fa2d808ad10 con 0x7fa2c4004800 2024-03-04T09:29:37.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.055+0000 7fa2dea00640 1 --2- 172.21.15.6:0/601826493 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa2bc06b4d0 0x7fa2bc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:37.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.056+0000 7fa2dea00640 1 --2- 172.21.15.6:0/601826493 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa2bc06b4d0 0x7fa2bc06d990 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fa2c8034000 tx=0x7fa2c8007000 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:37.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.059+0000 7fa2d7400640 1 -- 172.21.15.6:0/601826493 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa2d8090020 con 0x7fa2c4004800 2024-03-04T09:29:37.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.260+0000 7fa2e6600640 1 -- 172.21.15.6:0/601826493 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "config rm", "who": "mds", "name": "mds_bal_fragment_size_max"} v 0) v1 -- 0x7fa2e0154a00 con 0x7fa2c4004800 2024-03-04T09:29:37.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.267+0000 7fa2d7400640 1 -- 172.21.15.6:0/601826493 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds", "name": "mds_bal_fragment_size_max"}]=0 v24) v1 ==== 108+0+0 (secure 0 0 0) 0x7fa2d8024030 con 0x7fa2c4004800 2024-03-04T09:29:37.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.270+0000 7fa2e6600640 1 -- 172.21.15.6:0/601826493 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa2bc06b4d0 msgr2=0x7fa2bc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:37.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.270+0000 7fa2e6600640 1 --2- 172.21.15.6:0/601826493 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa2bc06b4d0 0x7fa2bc06d990 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fa2c8034000 tx=0x7fa2c8007000 comp rx=0 tx=0).stop 2024-03-04T09:29:37.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.270+0000 7fa2e6600640 1 -- 172.21.15.6:0/601826493 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa2c4004800 msgr2=0x7fa2e01f40a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:37.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.270+0000 7fa2e6600640 1 --2- 172.21.15.6:0/601826493 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa2c4004800 0x7fa2e01f40a0 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7fa2d8009fb0 tx=0x7fa2d80115a0 comp rx=0 tx=0).stop 2024-03-04T09:29:37.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.270+0000 7fa2e6600640 1 -- 172.21.15.6:0/601826493 shutdown_connections 2024-03-04T09:29:37.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.270+0000 7fa2e6600640 1 --2- 172.21.15.6:0/601826493 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa2bc06b4d0 0x7fa2bc06d990 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:37.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.270+0000 7fa2e6600640 1 --2- 172.21.15.6:0/601826493 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa2e01511d0 0x7fa2e0154090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:37.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.270+0000 7fa2e6600640 1 --2- 172.21.15.6:0/601826493 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa2e0150d50 0x7fa2e0153b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:37.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.270+0000 7fa2e6600640 1 --2- 172.21.15.6:0/601826493 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa2c4004800 0x7fa2e01f40a0 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:37.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.270+0000 7fa2e6600640 1 -- 172.21.15.6:0/601826493 >> 172.21.15.6:0/601826493 conn(0x7fa2e00722d0 msgr2=0x7fa2e00b0980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:29:37.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.270+0000 7fa2e6600640 1 -- 172.21.15.6:0/601826493 shutdown_connections 2024-03-04T09:29:37.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.270+0000 7fa2e6600640 1 -- 172.21.15.6:0/601826493 wait complete. 2024-03-04T09:29:37.286 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-04T09:29:37.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.501+0000 7fb036e00640 1 Processor -- start 2024-03-04T09:29:37.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.501+0000 7fb036e00640 1 -- start start 2024-03-04T09:29:37.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.503+0000 7fb036e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fb0300cf3a0 0x7fb030070ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:37.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.503+0000 7fb036e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb030071c60 con 0x7fb0300cd1e0 2024-03-04T09:29:37.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.503+0000 7fb036e00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb030071f40 con 0x7fb03015afa0 2024-03-04T09:29:37.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.503+0000 7fb036e00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0300cfa10 con 0x7fb0300cf3a0 2024-03-04T09:29:37.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.504+0000 7fb027e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fb0300cd1e0 0x7fb0300b3ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:42954/0 (socket says 172.21.15.6:42954) 2024-03-04T09:29:37.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.504+0000 7fb027e00640 1 -- 172.21.15.6:0/3904785644 learned_addr learned my addr 172.21.15.6:0/3904785644 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:29:37.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.504+0000 7fb034e00640 1 --2- 172.21.15.6:0/3904785644 >> v2:172.21.15.146:3300/0 conn(0x7fb0300cf3a0 0x7fb030070ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:37.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.504+0000 7fb02f400640 1 -- 172.21.15.6:0/3904785644 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1873014792 0 0) 0x7fb030071c60 con 0x7fb0300cd1e0 2024-03-04T09:29:37.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.504+0000 7fb02f400640 1 -- 172.21.15.6:0/3904785644 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb014005730 con 0x7fb0300cd1e0 2024-03-04T09:29:37.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.504+0000 7fb02f400640 1 -- 172.21.15.6:0/3904785644 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2688179973 0 0) 0x7fb014005730 con 0x7fb0300cd1e0 2024-03-04T09:29:37.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.504+0000 7fb02f400640 1 -- 172.21.15.6:0/3904785644 >> v2:172.21.15.146:3300/0 conn(0x7fb0300cf3a0 msgr2=0x7fb030070ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:37.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.504+0000 7fb02f400640 1 --2- 172.21.15.6:0/3904785644 >> v2:172.21.15.146:3300/0 conn(0x7fb0300cf3a0 0x7fb030070ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:37.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.505+0000 7fb02f400640 1 -- 172.21.15.6:0/3904785644 >> v1:172.21.15.146:6789/0 conn(0x7fb03015afa0 legacy=0x7fb0300ccad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:37.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.505+0000 7fb02f400640 1 -- 172.21.15.6:0/3904785644 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb030072220 con 0x7fb0300cd1e0 2024-03-04T09:29:37.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.505+0000 7fb02f400640 1 -- 172.21.15.6:0/3904785644 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7fb018001670 con 0x7fb0300cd1e0 2024-03-04T09:29:37.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.505+0000 7fb02f400640 1 -- 172.21.15.6:0/3904785644 >> v1:172.21.15.6:6789/0 conn(0x7fb0300cd1e0 legacy=0x7fb0300b3ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:37.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.505+0000 7fb02f400640 1 --2- 172.21.15.6:0/3904785644 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb014005c10 0x7fb014006030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:37.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.505+0000 7fb02f400640 1 --2- 172.21.15.6:0/3904785644 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb014006a10 0x7fb014006e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:37.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.505+0000 7fb02f400640 1 --2- 172.21.15.6:0/3904785644 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb0140073d0 0x7fb01400dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:37.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.505+0000 7fb02f400640 1 -- 172.21.15.6:0/3904785644 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0300cfa10 con 0x7fb014005c10 2024-03-04T09:29:37.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.505+0000 7fb02f400640 1 -- 172.21.15.6:0/3904785644 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb01400e2d0 con 0x7fb014006a10 2024-03-04T09:29:37.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.505+0000 7fb02f400640 1 -- 172.21.15.6:0/3904785644 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb01400e4d0 con 0x7fb0140073d0 2024-03-04T09:29:37.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.505+0000 7fb02fe00640 1 --2- 172.21.15.6:0/3904785644 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb014005c10 0x7fb014006030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:37.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.505+0000 7fb027e00640 1 --2- 172.21.15.6:0/3904785644 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb014006a10 0x7fb014006e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:37.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.505+0000 7fb02fe00640 1 -- 172.21.15.6:0/3904785644 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb0140073d0 msgr2=0x7fb01400dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:37.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.505+0000 7fb02fe00640 1 --2- 172.21.15.6:0/3904785644 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb0140073d0 0x7fb01400dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:37.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.505+0000 7fb02fe00640 1 -- 172.21.15.6:0/3904785644 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb014006a10 msgr2=0x7fb014006e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:37.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.505+0000 7fb02fe00640 1 --2- 172.21.15.6:0/3904785644 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb014006a10 0x7fb014006e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:37.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.505+0000 7fb02fe00640 1 -- 172.21.15.6:0/3904785644 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb014005730 con 0x7fb014005c10 2024-03-04T09:29:37.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.505+0000 7fb034e00640 1 -- 172.21.15.6:0/3904785644 reap_dead start 2024-03-04T09:29:37.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.506+0000 7fb02fe00640 1 --2- 172.21.15.6:0/3904785644 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb014005c10 0x7fb014006030 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7fb020001750 tx=0x7fb014007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa7ea29b35d46688 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:37.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.507+0000 7fb02f400640 1 -- 172.21.15.6:0/3904785644 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb020001370 con 0x7fb014005c10 2024-03-04T09:29:37.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.507+0000 7fb02f400640 1 -- 172.21.15.6:0/3904785644 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0200052d0 con 0x7fb014005c10 2024-03-04T09:29:37.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.507+0000 7fb02f400640 1 -- 172.21.15.6:0/3904785644 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb020003830 con 0x7fb014005c10 2024-03-04T09:29:37.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.507+0000 7fb036e00640 1 -- 172.21.15.6:0/3904785644 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb014005c10 msgr2=0x7fb014006030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:37.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.507+0000 7fb036e00640 1 --2- 172.21.15.6:0/3904785644 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb014005c10 0x7fb014006030 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7fb020001750 tx=0x7fb014007a40 comp rx=0 tx=0).stop 2024-03-04T09:29:37.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.507+0000 7fb036e00640 1 -- 172.21.15.6:0/3904785644 shutdown_connections 2024-03-04T09:29:37.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.507+0000 7fb036e00640 1 --2- 172.21.15.6:0/3904785644 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb014005c10 0x7fb014006030 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:37.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.507+0000 7fb036e00640 1 -- 172.21.15.6:0/3904785644 >> 172.21.15.6:0/3904785644 conn(0x7fb03006b3d0 msgr2=0x7fb03006b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:29:37.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.507+0000 7fb036e00640 1 -- 172.21.15.6:0/3904785644 shutdown_connections 2024-03-04T09:29:37.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.507+0000 7fb036e00640 1 -- 172.21.15.6:0/3904785644 wait complete. 2024-03-04T09:29:37.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.509+0000 7fb036e00640 1 Processor -- start 2024-03-04T09:29:37.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.509+0000 7fb036e00640 1 -- start start 2024-03-04T09:29:37.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.510+0000 7fb036e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb014005c10 0x7fb03015d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:37.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.510+0000 7fb036e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb030156010 0x7fb030154650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:37.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.510+0000 7fb036e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb03014a000 0x7fb03014a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:37.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.510+0000 7fb036e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb030070ce0 con 0x7fb014005c10 2024-03-04T09:29:37.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.510+0000 7fb036e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0300cf7d0 con 0x7fb030156010 2024-03-04T09:29:37.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.510+0000 7fb036e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb030070e60 con 0x7fb03014a000 2024-03-04T09:29:37.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.510+0000 7fb02fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb014005c10 0x7fb03015d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:37.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.511+0000 7fb02fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb014005c10 0x7fb03015d410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53042/0 (socket says 172.21.15.6:53042) 2024-03-04T09:29:37.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.511+0000 7fb02fe00640 1 -- 172.21.15.6:0/1271992658 learned_addr learned my addr 172.21.15.6:0/1271992658 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:29:37.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.511+0000 7fb027e00640 1 --2- 172.21.15.6:0/1271992658 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb030156010 0x7fb030154650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:37.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.511+0000 7fb02fe00640 1 -- 172.21.15.6:0/1271992658 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb03014a000 msgr2=0x7fb03014a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:37.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.511+0000 7fb02fe00640 1 --2- 172.21.15.6:0/1271992658 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb03014a000 0x7fb03014a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:37.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.511+0000 7fb02fe00640 1 -- 172.21.15.6:0/1271992658 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb030156010 msgr2=0x7fb030154650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:37.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.511+0000 7fb02fe00640 1 --2- 172.21.15.6:0/1271992658 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb030156010 0x7fb030154650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:37.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.511+0000 7fb02fe00640 1 -- 172.21.15.6:0/1271992658 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb030156e70 con 0x7fb014005c10 2024-03-04T09:29:37.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.511+0000 7fb027e00640 1 --2- 172.21.15.6:0/1271992658 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb030156010 0x7fb030154650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:29:37.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.511+0000 7fb02fe00640 1 --2- 172.21.15.6:0/1271992658 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb014005c10 0x7fb03015d410 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7fb020005040 tx=0x7fb020001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:37.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.512+0000 7fb02cc00640 1 -- 172.21.15.6:0/1271992658 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb020041020 con 0x7fb014005c10 2024-03-04T09:29:37.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.512+0000 7fb02cc00640 1 -- 172.21.15.6:0/1271992658 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb020005480 con 0x7fb014005c10 2024-03-04T09:29:37.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.512+0000 7fb02cc00640 1 -- 172.21.15.6:0/1271992658 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb020001370 con 0x7fb014005c10 2024-03-04T09:29:37.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.512+0000 7fb036e00640 1 -- 172.21.15.6:0/1271992658 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb03014a9c0 con 0x7fb014005c10 2024-03-04T09:29:37.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.512+0000 7fb036e00640 1 -- 172.21.15.6:0/1271992658 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0301568b0 con 0x7fb014005c10 2024-03-04T09:29:37.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.513+0000 7fb036e00640 1 -- 172.21.15.6:0/1271992658 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb03004f330 con 0x7fb014005c10 2024-03-04T09:29:37.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.514+0000 7fb02cc00640 1 -- 172.21.15.6:0/1271992658 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb020044020 con 0x7fb014005c10 2024-03-04T09:29:37.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.515+0000 7fb02cc00640 1 --2- 172.21.15.6:0/1271992658 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb00c06b500 0x7fb00c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:37.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.515+0000 7fb02cc00640 1 -- 172.21.15.6:0/1271992658 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fb0200ab830 con 0x7fb014005c10 2024-03-04T09:29:37.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.516+0000 7fb027e00640 1 --2- 172.21.15.6:0/1271992658 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb00c06b500 0x7fb00c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:37.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.517+0000 7fb027e00640 1 --2- 172.21.15.6:0/1271992658 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb00c06b500 0x7fb00c06d9c0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fb018000c00 tx=0x7fb018009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:37.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.519+0000 7fb02cc00640 1 -- 172.21.15.6:0/1271992658 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb02004b030 con 0x7fb014005c10 2024-03-04T09:29:37.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.717+0000 7fb036e00640 1 -- 172.21.15.6:0/1271992658 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fb030159060 con 0x7fb014005c10 2024-03-04T09:29:37.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.723+0000 7fb02cc00640 1 -- 172.21.15.6:0/1271992658 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v25) v1 ==== 104+0+0 (secure 0 0 0) 0x7fb030156e70 con 0x7fb014005c10 2024-03-04T09:29:37.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.725+0000 7fb036e00640 1 -- 172.21.15.6:0/1271992658 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb00c06b500 msgr2=0x7fb00c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:37.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.725+0000 7fb036e00640 1 --2- 172.21.15.6:0/1271992658 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb00c06b500 0x7fb00c06d9c0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fb018000c00 tx=0x7fb018009040 comp rx=0 tx=0).stop 2024-03-04T09:29:37.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.725+0000 7fb036e00640 1 -- 172.21.15.6:0/1271992658 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb014005c10 msgr2=0x7fb03015d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:37.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.725+0000 7fb036e00640 1 --2- 172.21.15.6:0/1271992658 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb014005c10 0x7fb03015d410 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7fb020005040 tx=0x7fb020001120 comp rx=0 tx=0).stop 2024-03-04T09:29:37.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.725+0000 7fb036e00640 1 -- 172.21.15.6:0/1271992658 shutdown_connections 2024-03-04T09:29:37.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.726+0000 7fb036e00640 1 --2- 172.21.15.6:0/1271992658 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb00c06b500 0x7fb00c06d9c0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:37.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.726+0000 7fb036e00640 1 --2- 172.21.15.6:0/1271992658 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb03014a000 0x7fb03014a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:37.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.726+0000 7fb036e00640 1 --2- 172.21.15.6:0/1271992658 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb030156010 0x7fb030154650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:37.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.726+0000 7fb036e00640 1 --2- 172.21.15.6:0/1271992658 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb014005c10 0x7fb03015d410 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:37.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.726+0000 7fb036e00640 1 -- 172.21.15.6:0/1271992658 >> 172.21.15.6:0/1271992658 conn(0x7fb03006b3d0 msgr2=0x7fb030159840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:29:37.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.726+0000 7fb036e00640 1 -- 172.21.15.6:0/1271992658 shutdown_connections 2024-03-04T09:29:37.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.726+0000 7fb036e00640 1 -- 172.21.15.6:0/1271992658 wait complete. 2024-03-04T09:29:37.743 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_strays' 2024-03-04T09:29:37.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.964+0000 7f8ebc000640 1 Processor -- start 2024-03-04T09:29:37.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.965+0000 7f8ebc000640 1 -- start start 2024-03-04T09:29:37.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.966+0000 7f8ebc000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f8eb40b34e0 0x7f8eb415c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:37.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.966+0000 7f8ebc000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8eb415cc40 con 0x7f8eb40af580 2024-03-04T09:29:37.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.966+0000 7f8ebc000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8eb415cf20 con 0x7f8eb40af1a0 2024-03-04T09:29:37.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.966+0000 7f8ebc000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8eb415d200 con 0x7f8eb40b34e0 2024-03-04T09:29:37.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.967+0000 7f8eba000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f8eb40b34e0 0x7f8eb415c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:37.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.967+0000 7f8eb8c00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f8eb40af580 0x7f8eb40b2dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:42966/0 (socket says 172.21.15.6:42966) 2024-03-04T09:29:37.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.967+0000 7f8eb8c00640 1 -- 172.21.15.6:0/1318808835 learned_addr learned my addr 172.21.15.6:0/1318808835 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:29:37.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.967+0000 7f8eba000640 1 -- 172.21.15.6:0/1318808835 >> v1:172.21.15.146:6789/0 conn(0x7f8eb40af1a0 legacy=0x7f8eb40b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:37.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.967+0000 7f8eba000640 1 -- 172.21.15.6:0/1318808835 >> v1:172.21.15.6:6789/0 conn(0x7f8eb40af580 legacy=0x7f8eb40b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:37.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.967+0000 7f8eba000640 1 -- 172.21.15.6:0/1318808835 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8eb415d4e0 con 0x7f8eb40b34e0 2024-03-04T09:29:37.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.967+0000 7f8eb3e00640 1 -- 172.21.15.6:0/1318808835 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1897580587 0 0) 0x7f8eb415cc40 con 0x7f8eb40af580 2024-03-04T09:29:37.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.968+0000 7f8eba000640 1 --2- 172.21.15.6:0/1318808835 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8eb40b34e0 0x7f8eb415c600 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f8eac0097a0 tx=0x7f8eac02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c1afba8ee949855 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:37.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.969+0000 7f8eb3e00640 1 -- 172.21.15.6:0/1318808835 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8eac0315a0 con 0x7f8eb40b34e0 2024-03-04T09:29:37.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.969+0000 7f8eb3e00640 1 -- 172.21.15.6:0/1318808835 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8eac037070 con 0x7f8eb40b34e0 2024-03-04T09:29:37.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.969+0000 7f8eb3e00640 1 -- 172.21.15.6:0/1318808835 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8eac031960 con 0x7f8eb40b34e0 2024-03-04T09:29:37.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.969+0000 7f8ebc000640 1 -- 172.21.15.6:0/1318808835 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8eb40b34e0 msgr2=0x7f8eb415c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:37.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.969+0000 7f8ebc000640 1 --2- 172.21.15.6:0/1318808835 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8eb40b34e0 0x7f8eb415c600 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f8eac0097a0 tx=0x7f8eac02b840 comp rx=0 tx=0).stop 2024-03-04T09:29:37.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.969+0000 7f8ebc000640 1 -- 172.21.15.6:0/1318808835 shutdown_connections 2024-03-04T09:29:37.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.969+0000 7f8ebc000640 1 --2- 172.21.15.6:0/1318808835 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8eb40b34e0 0x7f8eb415c600 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:37.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.969+0000 7f8ebc000640 1 -- 172.21.15.6:0/1318808835 >> 172.21.15.6:0/1318808835 conn(0x7f8eb406afb0 msgr2=0x7f8eb406b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:29:37.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.969+0000 7f8ebc000640 1 -- 172.21.15.6:0/1318808835 shutdown_connections 2024-03-04T09:29:37.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.969+0000 7f8ebc000640 1 -- 172.21.15.6:0/1318808835 wait complete. 2024-03-04T09:29:37.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.971+0000 7f8ebc000640 1 Processor -- start 2024-03-04T09:29:37.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.971+0000 7f8ebc000640 1 -- start start 2024-03-04T09:29:37.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.972+0000 7f8ebc000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8eb40af1a0 0x7f8eb41f2150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:37.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.972+0000 7f8ebc000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8eb40af580 0x7f8eb41521b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:37.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.972+0000 7f8ebc000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8eb40b34e0 0x7f8eb415e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:37.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.972+0000 7f8ebc000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8eb4155e70 con 0x7f8eb40af580 2024-03-04T09:29:37.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.972+0000 7f8ebc000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8eb40cde30 con 0x7f8eb40b34e0 2024-03-04T09:29:37.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.972+0000 7f8ebc000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8eb4155ff0 con 0x7f8eb40af1a0 2024-03-04T09:29:37.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.972+0000 7f8eb8c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8eb40af580 0x7f8eb41521b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:37.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.972+0000 7f8eb8c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8eb40af580 0x7f8eb41521b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53056/0 (socket says 172.21.15.6:53056) 2024-03-04T09:29:37.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.972+0000 7f8eb8c00640 1 -- 172.21.15.6:0/1993350362 learned_addr learned my addr 172.21.15.6:0/1993350362 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:29:37.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.973+0000 7f8eba000640 1 --2- 172.21.15.6:0/1993350362 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8eb40b34e0 0x7f8eb415e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:37.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.973+0000 7f8eb9600640 1 --2- 172.21.15.6:0/1993350362 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8eb40af1a0 0x7f8eb41f2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:37.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.973+0000 7f8eb8c00640 1 -- 172.21.15.6:0/1993350362 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8eb40af1a0 msgr2=0x7f8eb41f2150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:37.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.973+0000 7f8eb8c00640 1 --2- 172.21.15.6:0/1993350362 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8eb40af1a0 0x7f8eb41f2150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:37.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.973+0000 7f8eb8c00640 1 -- 172.21.15.6:0/1993350362 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8eb40b34e0 msgr2=0x7f8eb415e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:37.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.973+0000 7f8eb8c00640 1 --2- 172.21.15.6:0/1993350362 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8eb40b34e0 0x7f8eb415e9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:37.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.973+0000 7f8eb8c00640 1 -- 172.21.15.6:0/1993350362 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8eb41f16b0 con 0x7f8eb40af580 2024-03-04T09:29:37.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.973+0000 7f8eba000640 1 --2- 172.21.15.6:0/1993350362 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8eb40b34e0 0x7f8eb415e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:29:37.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.973+0000 7f8eb8c00640 1 --2- 172.21.15.6:0/1993350362 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8eb40af580 0x7f8eb41521b0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f8ea400c820 tx=0x7f8ea400cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:37.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.973+0000 7f8eb1600640 1 -- 172.21.15.6:0/1993350362 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ea4018070 con 0x7f8eb40af580 2024-03-04T09:29:37.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.973+0000 7f8eb1600640 1 -- 172.21.15.6:0/1993350362 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ea400a640 con 0x7f8eb40af580 2024-03-04T09:29:37.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.973+0000 7f8eb1600640 1 -- 172.21.15.6:0/1993350362 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ea400aa30 con 0x7f8eb40af580 2024-03-04T09:29:37.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.974+0000 7f8ebc000640 1 -- 172.21.15.6:0/1993350362 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8eb4152a90 con 0x7f8eb40af580 2024-03-04T09:29:37.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.974+0000 7f8ebc000640 1 -- 172.21.15.6:0/1993350362 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8eb41f1c30 con 0x7f8eb40af580 2024-03-04T09:29:37.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.975+0000 7f8ebc000640 1 -- 172.21.15.6:0/1993350362 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8eb4069510 con 0x7f8eb40af580 2024-03-04T09:29:37.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.976+0000 7f8eb1600640 1 -- 172.21.15.6:0/1993350362 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8ea401b070 con 0x7f8eb40af580 2024-03-04T09:29:37.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.977+0000 7f8eb1600640 1 --2- 172.21.15.6:0/1993350362 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f8e9806b4d0 0x7f8e9806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:37.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.977+0000 7f8eb1600640 1 -- 172.21.15.6:0/1993350362 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f8ea40895e0 con 0x7f8eb40af580 2024-03-04T09:29:37.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.978+0000 7f8eb9600640 1 --2- 172.21.15.6:0/1993350362 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f8e9806b4d0 0x7f8e9806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:37.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.980+0000 7f8eb9600640 1 --2- 172.21.15.6:0/1993350362 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f8e9806b4d0 0x7f8e9806d990 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f8ea8003d90 tx=0x7f8ea8009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:37.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:37.982+0000 7f8eb1600640 1 -- 172.21.15.6:0/1993350362 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8ea40586c0 con 0x7f8eb40af580 2024-03-04T09:29:38.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:38.256+0000 7f8ebc000640 1 -- 172.21.15.6:0/1993350362 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_strays"]} v 0) v1 -- 0x7f8eb41f1e00 con 0x7f8eb40af580 2024-03-04T09:29:38.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:38.790+0000 7f8eb1600640 1 -- 172.21.15.6:0/1993350362 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_strays"]}]=0 v248) v1 ==== 140+0+0 (secure 0 0 0) 0x7f8ea4057fc0 con 0x7f8eb40af580 2024-03-04T09:29:38.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:38.793+0000 7f8ebc000640 1 -- 172.21.15.6:0/1993350362 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f8e9806b4d0 msgr2=0x7f8e9806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:38.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:38.793+0000 7f8ebc000640 1 --2- 172.21.15.6:0/1993350362 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f8e9806b4d0 0x7f8e9806d990 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f8ea8003d90 tx=0x7f8ea8009040 comp rx=0 tx=0).stop 2024-03-04T09:29:38.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:38.793+0000 7f8ebc000640 1 -- 172.21.15.6:0/1993350362 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8eb40af580 msgr2=0x7f8eb41521b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:38.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:38.793+0000 7f8ebc000640 1 --2- 172.21.15.6:0/1993350362 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8eb40af580 0x7f8eb41521b0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f8ea400c820 tx=0x7f8ea400cce0 comp rx=0 tx=0).stop 2024-03-04T09:29:38.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:38.794+0000 7f8ebc000640 1 -- 172.21.15.6:0/1993350362 shutdown_connections 2024-03-04T09:29:38.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:38.794+0000 7f8ebc000640 1 --2- 172.21.15.6:0/1993350362 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8eb40b34e0 0x7f8eb415e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:38.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:38.794+0000 7f8ebc000640 1 --2- 172.21.15.6:0/1993350362 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8eb40af580 0x7f8eb41521b0 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:38.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:38.794+0000 7f8ebc000640 1 --2- 172.21.15.6:0/1993350362 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f8e9806b4d0 0x7f8e9806d990 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:38.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:38.794+0000 7f8ebc000640 1 --2- 172.21.15.6:0/1993350362 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8eb40af1a0 0x7f8eb41f2150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:38.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:38.794+0000 7f8ebc000640 1 -- 172.21.15.6:0/1993350362 >> 172.21.15.6:0/1993350362 conn(0x7f8eb406afb0 msgr2=0x7f8eb414a6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:29:38.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:38.794+0000 7f8ebc000640 1 -- 172.21.15.6:0/1993350362 shutdown_connections 2024-03-04T09:29:38.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:38.794+0000 7f8ebc000640 1 -- 172.21.15.6:0/1993350362 wait complete. 2024-03-04T09:29:38.812 INFO:tasks.cephfs_test_runner:That unlinking fails when the stray directory fragment becomes too ... ok 2024-03-04T09:29:38.813 INFO:tasks.cephfs_test_runner:Starting test: test_fancy_layout (tasks.cephfs.test_strays.TestStrays) purge stray file with fancy layout 2024-03-04T09:29:38.813 INFO:tasks.cephfs_test_runner:test_fancy_layout (tasks.cephfs.test_strays.TestStrays) 2024-03-04T09:29:38.813 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T09:29:39.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.040+0000 7f7adfe00640 1 Processor -- start 2024-03-04T09:29:39.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.041+0000 7f7adfe00640 1 -- start start 2024-03-04T09:29:39.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.042+0000 7f7adfe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f7ae015e0b0 0x7f7ae00cf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:39.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.042+0000 7f7adfe00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7ae00d4a10 0x7f7ae00d0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:39.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.042+0000 7f7adfe00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f7ae00d0c80 0x7f7ae00d1900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:39.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.042+0000 7f7adfe00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ae00b09e0 con 0x7f7ae015e0b0 2024-03-04T09:29:39.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.042+0000 7f7adfe00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ae00b3710 con 0x7f7ae00d0c80 2024-03-04T09:29:39.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.042+0000 7f7adfe00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ae00b39f0 con 0x7f7ae00d4a10 2024-03-04T09:29:39.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.042+0000 7f7adea00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f7ae015e0b0 0x7f7ae00cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:39.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.042+0000 7f7adea00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f7ae015e0b0 0x7f7ae00cf310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38648/0 (socket says 172.21.15.6:38648) 2024-03-04T09:29:39.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.042+0000 7f7adea00640 1 -- 172.21.15.6:0/1180115565 learned_addr learned my addr 172.21.15.6:0/1180115565 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:29:39.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.043+0000 7f7ade000640 1 --2- 172.21.15.6:0/1180115565 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7ae00d4a10 0x7f7ae00d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:39.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.043+0000 7f7adf400640 1 --2- 172.21.15.6:0/1180115565 >> v2:172.21.15.146:3300/0 conn(0x7f7ae00d0c80 0x7f7ae00d1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:39.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.043+0000 7f7adea00640 1 -- 172.21.15.6:0/1180115565 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7ae00d4a10 msgr2=0x7f7ae00d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:39.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.043+0000 7f7adea00640 1 --2- 172.21.15.6:0/1180115565 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7ae00d4a10 0x7f7ae00d0740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:39.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.043+0000 7f7adea00640 1 -- 172.21.15.6:0/1180115565 >> v2:172.21.15.146:3300/0 conn(0x7f7ae00d0c80 msgr2=0x7f7ae00d1900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:39.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.043+0000 7f7adea00640 1 --2- 172.21.15.6:0/1180115565 >> v2:172.21.15.146:3300/0 conn(0x7f7ae00d0c80 0x7f7ae00d1900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:39.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.043+0000 7f7adea00640 1 -- 172.21.15.6:0/1180115565 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ae00b3cd0 con 0x7f7ae015e0b0 2024-03-04T09:29:39.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.043+0000 7f7ade000640 1 --2- 172.21.15.6:0/1180115565 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7ae00d4a10 0x7f7ae00d0740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:29:39.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.043+0000 7f7adf400640 1 --2- 172.21.15.6:0/1180115565 >> v2:172.21.15.146:3300/0 conn(0x7f7ae00d0c80 0x7f7ae00d1900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:29:39.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.044+0000 7f7adea00640 1 --2- 172.21.15.6:0/1180115565 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7ae015e0b0 0x7f7ae00cf310 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f7ae00d06c0 tx=0x7f7ad002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88d7889ed938f139 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:39.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.044+0000 7f7add600640 1 -- 172.21.15.6:0/1180115565 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ad002ca80 con 0x7f7ae015e0b0 2024-03-04T09:29:39.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.044+0000 7f7add600640 1 -- 172.21.15.6:0/1180115565 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ad002cc20 con 0x7f7ae015e0b0 2024-03-04T09:29:39.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.044+0000 7f7add600640 1 -- 172.21.15.6:0/1180115565 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ad00343f0 con 0x7f7ae015e0b0 2024-03-04T09:29:39.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.045+0000 7f7adfe00640 1 -- 172.21.15.6:0/1180115565 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7ae015e0b0 msgr2=0x7f7ae00cf310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:39.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.045+0000 7f7adfe00640 1 --2- 172.21.15.6:0/1180115565 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7ae015e0b0 0x7f7ae00cf310 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f7ae00d06c0 tx=0x7f7ad002bee0 comp rx=0 tx=0).stop 2024-03-04T09:29:39.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.045+0000 7f7adfe00640 1 -- 172.21.15.6:0/1180115565 shutdown_connections 2024-03-04T09:29:39.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.045+0000 7f7adfe00640 1 --2- 172.21.15.6:0/1180115565 >> v2:172.21.15.146:3300/0 conn(0x7f7ae00d0c80 0x7f7ae00d1900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:39.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.045+0000 7f7adfe00640 1 --2- 172.21.15.6:0/1180115565 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7ae00d4a10 0x7f7ae00d0740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:39.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.045+0000 7f7adfe00640 1 --2- 172.21.15.6:0/1180115565 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7ae015e0b0 0x7f7ae00cf310 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:39.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.045+0000 7f7adfe00640 1 -- 172.21.15.6:0/1180115565 >> 172.21.15.6:0/1180115565 conn(0x7f7ae00d3d30 msgr2=0x7f7ae00d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:29:39.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.045+0000 7f7adfe00640 1 -- 172.21.15.6:0/1180115565 shutdown_connections 2024-03-04T09:29:39.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.045+0000 7f7adfe00640 1 -- 172.21.15.6:0/1180115565 wait complete. 2024-03-04T09:29:39.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.047+0000 7f7adfe00640 1 Processor -- start 2024-03-04T09:29:39.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.047+0000 7f7adfe00640 1 -- start start 2024-03-04T09:29:39.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.048+0000 7f7adfe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7ae00d0c80 0x7f7ae015b700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:39.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.048+0000 7f7adfe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7ae00d4a10 0x7f7ae00b56d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:39.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.048+0000 7f7adfe00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7ae015e0b0 0x7f7ae00b5c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:39.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.048+0000 7f7adfe00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ae00cfc60 con 0x7f7ae00d0c80 2024-03-04T09:29:39.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.048+0000 7f7adfe00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ae015e490 con 0x7f7ae00d4a10 2024-03-04T09:29:39.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.048+0000 7f7adfe00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ae00cf850 con 0x7f7ae015e0b0 2024-03-04T09:29:39.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.048+0000 7f7adea00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7ae00d0c80 0x7f7ae015b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:39.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.048+0000 7f7adea00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7ae00d0c80 0x7f7ae015b700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38662/0 (socket says 172.21.15.6:38662) 2024-03-04T09:29:39.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.048+0000 7f7adea00640 1 -- 172.21.15.6:0/2449686501 learned_addr learned my addr 172.21.15.6:0/2449686501 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:29:39.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.049+0000 7f7adea00640 1 -- 172.21.15.6:0/2449686501 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7ae015e0b0 msgr2=0x7f7ae00b5c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:39.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.049+0000 7f7adea00640 1 --2- 172.21.15.6:0/2449686501 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7ae015e0b0 0x7f7ae00b5c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:39.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.049+0000 7f7adea00640 1 -- 172.21.15.6:0/2449686501 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7ae00d4a10 msgr2=0x7f7ae00b56d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:39.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.049+0000 7f7adea00640 1 --2- 172.21.15.6:0/2449686501 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7ae00d4a10 0x7f7ae00b56d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:39.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.049+0000 7f7adea00640 1 -- 172.21.15.6:0/2449686501 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ae01f3a40 con 0x7f7ae00d0c80 2024-03-04T09:29:39.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.049+0000 7f7adea00640 1 --2- 172.21.15.6:0/2449686501 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7ae00d0c80 0x7f7ae015b700 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f7ad0000f80 tx=0x7f7ad0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:39.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.049+0000 7f7ad6a00640 1 -- 172.21.15.6:0/2449686501 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ad0043020 con 0x7f7ae00d0c80 2024-03-04T09:29:39.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.050+0000 7f7ad6a00640 1 -- 172.21.15.6:0/2449686501 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ad003e450 con 0x7f7ae00d0c80 2024-03-04T09:29:39.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.050+0000 7f7ad6a00640 1 -- 172.21.15.6:0/2449686501 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ad003e840 con 0x7f7ae00d0c80 2024-03-04T09:29:39.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.050+0000 7f7adfe00640 1 -- 172.21.15.6:0/2449686501 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ae01f3d20 con 0x7f7ae00d0c80 2024-03-04T09:29:39.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.050+0000 7f7adfe00640 1 -- 172.21.15.6:0/2449686501 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ae01f42e0 con 0x7f7ae00d0c80 2024-03-04T09:29:39.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.051+0000 7f7adfe00640 1 -- 172.21.15.6:0/2449686501 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ae00682b0 con 0x7f7ae00d0c80 2024-03-04T09:29:39.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.052+0000 7f7ad6a00640 1 -- 172.21.15.6:0/2449686501 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7ad0046020 con 0x7f7ae00d0c80 2024-03-04T09:29:39.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.053+0000 7f7ad6a00640 1 --2- 172.21.15.6:0/2449686501 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f7abc06b4d0 0x7f7abc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:39.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.053+0000 7f7ad6a00640 1 -- 172.21.15.6:0/2449686501 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f7ad00ac910 con 0x7f7ae00d0c80 2024-03-04T09:29:39.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.053+0000 7f7ade000640 1 --2- 172.21.15.6:0/2449686501 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f7abc06b4d0 0x7f7abc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:39.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.054+0000 7f7ade000640 1 --2- 172.21.15.6:0/2449686501 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f7abc06b4d0 0x7f7abc06d990 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f7ae00b9740 tx=0x7f7ac8008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:39.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.057+0000 7f7ad6a00640 1 -- 172.21.15.6:0/2449686501 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7ae00682b0 con 0x7f7ae00d0c80 2024-03-04T09:29:39.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.257+0000 7f7adfe00640 1 -- 172.21.15.6:0/2449686501 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7ae004f330 con 0x7f7ae00d0c80 2024-03-04T09:29:39.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.258+0000 7f7ad6a00640 1 -- 172.21.15.6:0/2449686501 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v73) v1 ==== 74+0+12975 (secure 0 0 0) 0x7f7ad004d030 con 0x7f7ae00d0c80 2024-03-04T09:29:39.255 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:29:39.255 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":73,"fsid":"edce0578-5ae4-4402-b45a-552bdef84d7c","created":"2024-03-04T09:24:54.523780+0000","modified":"2024-03-04T09:29:36.767075+0000","last_up_change":"2024-03-04T09:25:07.089049+0000","last_in_change":"2024-03-04T09:24:57.982077+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T09:25:07.935198+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a96574f-81f1-4bc6-8463-757520b84a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6825","nonce":1364449401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6827","nonce":1364449401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6831","nonce":1364449401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6829","nonce":1364449401}]},"public_addr":"172.21.15.6:6825/1364449401","cluster_addr":"172.21.15.6:6827/1364449401","heartbeat_back_addr":"172.21.15.6:6831/1364449401","heartbeat_front_addr":"172.21.15.6:6829/1364449401","state":["exists","up"]},{"osd":1,"uuid":"7d79b9c1-6b87-476a-bc36-7db5ee64cd90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6809","nonce":2494103078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6811","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6813","nonce":2494103078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6821","nonce":2494103078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6817","nonce":2494103078}]},"public_addr":"172.21.15.6:6809/2494103078","cluster_addr":"172.21.15.6:6813/2494103078","heartbeat_back_addr":"172.21.15.6:6821/2494103078","heartbeat_front_addr":"172.21.15.6:6817/2494103078","state":["exists","up"]},{"osd":2,"uuid":"492331b9-4ee9-40b6-bacd-85d44cd6c45a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6812","nonce":1500066681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6816","nonce":1500066681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6824","nonce":1500066681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6820","nonce":1500066681}]},"public_addr":"172.21.15.6:6812/1500066681","cluster_addr":"172.21.15.6:6816/1500066681","heartbeat_back_addr":"172.21.15.6:6824/1500066681","heartbeat_front_addr":"172.21.15.6:6820/1500066681","state":["exists","up"]},{"osd":3,"uuid":"21bb2efe-9ba7-4bdc-bc31-180cd1102e21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6801","nonce":1558806123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6803","nonce":1558806123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6807","nonce":1558806123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6805","nonce":1558806123}]},"public_addr":"172.21.15.6:6801/1558806123","cluster_addr":"172.21.15.6:6803/1558806123","heartbeat_back_addr":"172.21.15.6:6807/1558806123","heartbeat_front_addr":"172.21.15.6:6805/1558806123","state":["exists","up"]},{"osd":4,"uuid":"6e2c4b71-19d9-4737-83d9-558705d8644a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6801","nonce":1526135780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6803","nonce":1526135780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6807","nonce":1526135780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6805","nonce":1526135780}]},"public_addr":"172.21.15.146:6801/1526135780","cluster_addr":"172.21.15.146:6803/1526135780","heartbeat_back_addr":"172.21.15.146:6807/1526135780","heartbeat_front_addr":"172.21.15.146:6805/1526135780","state":["exists","up"]},{"osd":5,"uuid":"fb69c525-f1be-4b7d-ab32-6c29b0d6fddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6809","nonce":460262507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6811","nonce":460262507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6815","nonce":460262507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6813","nonce":460262507}]},"public_addr":"172.21.15.146:6809/460262507","cluster_addr":"172.21.15.146:6811/460262507","heartbeat_back_addr":"172.21.15.146:6815/460262507","heartbeat_front_addr":"172.21.15.146:6813/460262507","state":["exists","up"]},{"osd":6,"uuid":"c32e7fed-370d-4401-8ce0-6a4335b0d168","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6823","nonce":1392897493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6827","nonce":1392897493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6831","nonce":1392897493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6829","nonce":1392897493}]},"public_addr":"172.21.15.146:6823/1392897493","cluster_addr":"172.21.15.146:6827/1392897493","heartbeat_back_addr":"172.21.15.146:6831/1392897493","heartbeat_front_addr":"172.21.15.146:6829/1392897493","state":["exists","up"]},{"osd":7,"uuid":"75a79b44-fa5f-426d-a7d3-71f21f78835c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6817","nonce":3268323156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6819","nonce":3268323156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6825","nonce":3268323156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6822","nonce":3268323156}]},"public_addr":"172.21.15.146:6817/3268323156","cluster_addr":"172.21.15.146:6819/3268323156","heartbeat_back_addr":"172.21.15.146:6825/3268323156","heartbeat_front_addr":"172.21.15.146:6822/3268323156","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.310437+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.295850+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.294674+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.526855+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.300023+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.595160+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.346046+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.558741+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.146:6837/3919939129":"2024-03-05T09:29:33.745766+0000","172.21.15.146:6836/3919939129":"2024-03-05T09:29:33.745766+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T09:29:39.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.261+0000 7f7adfe00640 1 -- 172.21.15.6:0/2449686501 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f7abc06b4d0 msgr2=0x7f7abc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:39.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.261+0000 7f7adfe00640 1 --2- 172.21.15.6:0/2449686501 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f7abc06b4d0 0x7f7abc06d990 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f7ae00b9740 tx=0x7f7ac8008040 comp rx=0 tx=0).stop 2024-03-04T09:29:39.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.261+0000 7f7adfe00640 1 -- 172.21.15.6:0/2449686501 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7ae00d0c80 msgr2=0x7f7ae015b700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:39.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.261+0000 7f7adfe00640 1 --2- 172.21.15.6:0/2449686501 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7ae00d0c80 0x7f7ae015b700 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f7ad0000f80 tx=0x7f7ad0004a40 comp rx=0 tx=0).stop 2024-03-04T09:29:39.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.261+0000 7f7adfe00640 1 -- 172.21.15.6:0/2449686501 shutdown_connections 2024-03-04T09:29:39.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.261+0000 7f7adfe00640 1 --2- 172.21.15.6:0/2449686501 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f7abc06b4d0 0x7f7abc06d990 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:39.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.261+0000 7f7adfe00640 1 --2- 172.21.15.6:0/2449686501 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7ae015e0b0 0x7f7ae00b5c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:39.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.261+0000 7f7adfe00640 1 --2- 172.21.15.6:0/2449686501 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7ae00d4a10 0x7f7ae00b56d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:39.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.261+0000 7f7adfe00640 1 --2- 172.21.15.6:0/2449686501 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7ae00d0c80 0x7f7ae015b700 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:39.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.261+0000 7f7adfe00640 1 -- 172.21.15.6:0/2449686501 >> 172.21.15.6:0/2449686501 conn(0x7f7ae00d3d30 msgr2=0x7f7ae00b1f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:29:39.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.261+0000 7f7adfe00640 1 -- 172.21.15.6:0/2449686501 shutdown_connections 2024-03-04T09:29:39.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.261+0000 7f7adfe00640 1 -- 172.21.15.6:0/2449686501 wait complete. 2024-03-04T09:29:39.279 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T09:25:07.935198+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-04T09:29:39.279 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-04T09:29:39.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.513+0000 7f666e800640 1 Processor -- start 2024-03-04T09:29:39.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.513+0000 7f666e800640 1 -- start start 2024-03-04T09:29:39.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.515+0000 7f666e800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6668145430 0x7f666815f810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:39.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.515+0000 7f666e800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f66680b2250 0x7f66680b2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:39.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.515+0000 7f666e800640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f666815fe40 con 0x7f666815cc90 2024-03-04T09:29:39.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.515+0000 7f666e800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6668160120 con 0x7f66680b2250 2024-03-04T09:29:39.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.515+0000 7f666e800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6668160400 con 0x7f6668145430 2024-03-04T09:29:39.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.515+0000 7f666de00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f66680b2250 0x7f66680b2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:39.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.515+0000 7f666de00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f66680b2250 0x7f66680b2fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38668/0 (socket says 172.21.15.6:38668) 2024-03-04T09:29:39.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.515+0000 7f666de00640 1 -- 172.21.15.6:0/1797525449 learned_addr learned my addr 172.21.15.6:0/1797525449 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:29:39.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.516+0000 7f666d400640 1 --2- 172.21.15.6:0/1797525449 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6668145430 0x7f666815f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:39.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.516+0000 7f666de00640 1 -- 172.21.15.6:0/1797525449 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6668145430 msgr2=0x7f666815f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:39.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.516+0000 7f666de00640 1 --2- 172.21.15.6:0/1797525449 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6668145430 0x7f666815f810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:39.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.516+0000 7f666de00640 1 -- 172.21.15.6:0/1797525449 >> v1:172.21.15.146:6789/0 conn(0x7f666815cc90 legacy=0x7f66680b1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:39.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.516+0000 7f666de00640 1 -- 172.21.15.6:0/1797525449 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66681606e0 con 0x7f66680b2250 2024-03-04T09:29:39.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.517+0000 7f666de00640 1 --2- 172.21.15.6:0/1797525449 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f66680b2250 0x7f66680b2fd0 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f6660009e50 tx=0x7f666002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c753d8c1864ca16 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:39.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.517+0000 7f6667e00640 1 -- 172.21.15.6:0/1797525449 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f666002cb00 con 0x7f66680b2250 2024-03-04T09:29:39.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.517+0000 7f6667e00640 1 -- 172.21.15.6:0/1797525449 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f666002cca0 con 0x7f66680b2250 2024-03-04T09:29:39.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.517+0000 7f6667e00640 1 -- 172.21.15.6:0/1797525449 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6660034610 con 0x7f66680b2250 2024-03-04T09:29:39.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.518+0000 7f666e800640 1 -- 172.21.15.6:0/1797525449 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f66680b2250 msgr2=0x7f66680b2fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:39.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.518+0000 7f666e800640 1 --2- 172.21.15.6:0/1797525449 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f66680b2250 0x7f66680b2fd0 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f6660009e50 tx=0x7f666002bf60 comp rx=0 tx=0).stop 2024-03-04T09:29:39.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.518+0000 7f666e800640 1 -- 172.21.15.6:0/1797525449 shutdown_connections 2024-03-04T09:29:39.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.518+0000 7f666e800640 1 --2- 172.21.15.6:0/1797525449 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f66680b2250 0x7f66680b2fd0 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:39.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.518+0000 7f666e800640 1 --2- 172.21.15.6:0/1797525449 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6668145430 0x7f666815f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:39.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.518+0000 7f666e800640 1 -- 172.21.15.6:0/1797525449 >> 172.21.15.6:0/1797525449 conn(0x7f666806b3a0 msgr2=0x7f666806b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:29:39.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.518+0000 7f666e800640 1 -- 172.21.15.6:0/1797525449 shutdown_connections 2024-03-04T09:29:39.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.518+0000 7f666e800640 1 -- 172.21.15.6:0/1797525449 wait complete. 2024-03-04T09:29:39.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.520+0000 7f666e800640 1 Processor -- start 2024-03-04T09:29:39.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.520+0000 7f666e800640 1 -- start start 2024-03-04T09:29:39.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.521+0000 7f666e800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f66680b2250 0x7f6668141f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:39.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.521+0000 7f666e800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6668145430 0x7f66681e6540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:39.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.521+0000 7f666e800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f666815cc90 0x7f6668071450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:39.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.521+0000 7f666e800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6668145990 con 0x7f66680b2250 2024-03-04T09:29:39.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.521+0000 7f666e800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6668145810 con 0x7f6668145430 2024-03-04T09:29:39.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.521+0000 7f666e800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66680cde00 con 0x7f666815cc90 2024-03-04T09:29:39.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.521+0000 7f666d400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f66680b2250 0x7f6668141f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:39.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.521+0000 7f666d400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f66680b2250 0x7f6668141f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38678/0 (socket says 172.21.15.6:38678) 2024-03-04T09:29:39.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.521+0000 7f666d400640 1 -- 172.21.15.6:0/1393188129 learned_addr learned my addr 172.21.15.6:0/1393188129 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:29:39.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.522+0000 7f666ca00640 1 --2- 172.21.15.6:0/1393188129 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6668145430 0x7f66681e6540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:39.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.522+0000 7f666de00640 1 --2- 172.21.15.6:0/1393188129 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f666815cc90 0x7f6668071450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:39.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.522+0000 7f666d400640 1 -- 172.21.15.6:0/1393188129 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f666815cc90 msgr2=0x7f6668071450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:39.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.522+0000 7f666d400640 1 --2- 172.21.15.6:0/1393188129 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f666815cc90 0x7f6668071450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:39.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.522+0000 7f666d400640 1 -- 172.21.15.6:0/1393188129 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6668145430 msgr2=0x7f66681e6540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:39.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.522+0000 7f666d400640 1 --2- 172.21.15.6:0/1393188129 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6668145430 0x7f66681e6540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:39.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.522+0000 7f666d400640 1 -- 172.21.15.6:0/1393188129 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6668142790 con 0x7f66680b2250 2024-03-04T09:29:39.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.522+0000 7f666ca00640 1 --2- 172.21.15.6:0/1393188129 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6668145430 0x7f66681e6540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:29:39.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.522+0000 7f666d400640 1 --2- 172.21.15.6:0/1393188129 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f66680b2250 0x7f6668141f30 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f665c00d830 tx=0x7f665c00dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:39.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.522+0000 7f6665600640 1 -- 172.21.15.6:0/1393188129 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f665c00b6c0 con 0x7f66680b2250 2024-03-04T09:29:39.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.522+0000 7f6665600640 1 -- 172.21.15.6:0/1393188129 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f665c019070 con 0x7f66680b2250 2024-03-04T09:29:39.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.522+0000 7f6665600640 1 -- 172.21.15.6:0/1393188129 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f665c00ba80 con 0x7f66680b2250 2024-03-04T09:29:39.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.522+0000 7f666e800640 1 -- 172.21.15.6:0/1393188129 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66681e5780 con 0x7f66680b2250 2024-03-04T09:29:39.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.523+0000 7f666e800640 1 -- 172.21.15.6:0/1393188129 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66681e5d40 con 0x7f66680b2250 2024-03-04T09:29:39.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.524+0000 7f666e800640 1 -- 172.21.15.6:0/1393188129 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f666804f330 con 0x7f66680b2250 2024-03-04T09:29:39.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.525+0000 7f6665600640 1 -- 172.21.15.6:0/1393188129 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f665c01e460 con 0x7f66680b2250 2024-03-04T09:29:39.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.525+0000 7f6665600640 1 --2- 172.21.15.6:0/1393188129 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f664806b500 0x7f664806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:39.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.526+0000 7f6665600640 1 -- 172.21.15.6:0/1393188129 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f665c08a5b0 con 0x7f66680b2250 2024-03-04T09:29:39.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.526+0000 7f666ca00640 1 --2- 172.21.15.6:0/1393188129 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f664806b500 0x7f664806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:39.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.527+0000 7f666ca00640 1 --2- 172.21.15.6:0/1393188129 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f664806b500 0x7f664806d9c0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f66681f88a0 tx=0x7f6654009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:39.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.530+0000 7f6665600640 1 -- 172.21.15.6:0/1393188129 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f665c059720 con 0x7f66680b2250 2024-03-04T09:29:39.726 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:29:39.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.728+0000 7f666e800640 1 -- 172.21.15.6:0/1393188129 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f666806e240 con 0x7f66680b2250 2024-03-04T09:29:39.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.729+0000 7f6665600640 1 -- 172.21.15.6:0/1393188129 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v73) v1 ==== 93+0+10 (secure 0 0 0) 0x7f665c059020 con 0x7f66680b2250 2024-03-04T09:29:39.727 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-04T09:29:39.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.731+0000 7f666e800640 1 -- 172.21.15.6:0/1393188129 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f664806b500 msgr2=0x7f664806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:39.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.731+0000 7f666e800640 1 --2- 172.21.15.6:0/1393188129 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f664806b500 0x7f664806d9c0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f66681f88a0 tx=0x7f6654009040 comp rx=0 tx=0).stop 2024-03-04T09:29:39.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.732+0000 7f666e800640 1 -- 172.21.15.6:0/1393188129 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f66680b2250 msgr2=0x7f6668141f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:39.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.732+0000 7f666e800640 1 --2- 172.21.15.6:0/1393188129 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f66680b2250 0x7f6668141f30 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f665c00d830 tx=0x7f665c00dcf0 comp rx=0 tx=0).stop 2024-03-04T09:29:39.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.732+0000 7f666e800640 1 -- 172.21.15.6:0/1393188129 shutdown_connections 2024-03-04T09:29:39.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.732+0000 7f666e800640 1 --2- 172.21.15.6:0/1393188129 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f664806b500 0x7f664806d9c0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:39.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.732+0000 7f666e800640 1 --2- 172.21.15.6:0/1393188129 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f666815cc90 0x7f6668071450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:39.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.732+0000 7f666e800640 1 --2- 172.21.15.6:0/1393188129 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6668145430 0x7f66681e6540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:39.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.732+0000 7f666e800640 1 --2- 172.21.15.6:0/1393188129 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f66680b2250 0x7f6668141f30 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:39.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.732+0000 7f666e800640 1 -- 172.21.15.6:0/1393188129 >> 172.21.15.6:0/1393188129 conn(0x7f666806b3a0 msgr2=0x7f66680ae500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:29:39.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.732+0000 7f666e800640 1 -- 172.21.15.6:0/1393188129 shutdown_connections 2024-03-04T09:29:39.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:39.732+0000 7f666e800640 1 -- 172.21.15.6:0/1393188129 wait complete. 2024-03-04T09:29:39.736 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:29:39.749 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_fancy_layout' 2024-03-04T09:29:40.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:40.006+0000 7f095e800640 1 Processor -- start 2024-03-04T09:29:40.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:40.006+0000 7f095e800640 1 -- start start 2024-03-04T09:29:40.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:40.007+0000 7f095e800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f095815e810 0x7f095815b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:40.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:40.007+0000 7f095e800640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f095815ed10 con 0x7f09580cd060 2024-03-04T09:29:40.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:40.007+0000 7f095e800640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f095815eff0 con 0x7f09581552b0 2024-03-04T09:29:40.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:40.007+0000 7f095e800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f095815f2d0 con 0x7f095815e810 2024-03-04T09:29:40.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:40.008+0000 7f095ca00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f09580cd060 0x7f09580b3cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:59400/0 (socket says 172.21.15.6:59400) 2024-03-04T09:29:40.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:40.008+0000 7f095ca00640 1 -- 172.21.15.6:0/1054210835 learned_addr learned my addr 172.21.15.6:0/1054210835 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:29:40.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:40.008+0000 7f095de00640 1 --2- 172.21.15.6:0/1054210835 >> v2:172.21.15.6:3300/0 conn(0x7f095815e810 0x7f095815b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:40.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:40.008+0000 7f095de00640 1 -- 172.21.15.6:0/1054210835 >> v1:172.21.15.146:6789/0 conn(0x7f09581552b0 legacy=0x7f09580cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:40.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:40.008+0000 7f095de00640 1 -- 172.21.15.6:0/1054210835 >> v1:172.21.15.6:6789/0 conn(0x7f09580cd060 legacy=0x7f09580b3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:40.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:40.008+0000 7f095de00640 1 -- 172.21.15.6:0/1054210835 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f095815f5b0 con 0x7f095815e810 2024-03-04T09:29:40.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:40.008+0000 7f0957e00640 1 -- 172.21.15.6:0/1054210835 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3589182325 0 0) 0x7f095815ed10 con 0x7f09580cd060 2024-03-04T09:29:40.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:40.009+0000 7f095de00640 1 --2- 172.21.15.6:0/1054210835 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f095815e810 0x7f095815b1d0 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f0950009d90 tx=0x7f095002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88411c143432b116 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:40.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:40.009+0000 7f0957e00640 1 -- 172.21.15.6:0/1054210835 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0950039070 con 0x7f095815e810 2024-03-04T09:29:40.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:40.010+0000 7f0957e00640 1 -- 172.21.15.6:0/1054210835 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f095002cc60 con 0x7f095815e810 2024-03-04T09:29:40.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:40.010+0000 7f0957e00640 1 -- 172.21.15.6:0/1054210835 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0950034660 con 0x7f095815e810 2024-03-04T09:29:40.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:40.010+0000 7f095e800640 1 -- 172.21.15.6:0/1054210835 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f095815e810 msgr2=0x7f095815b1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:40.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:40.010+0000 7f095e800640 1 --2- 172.21.15.6:0/1054210835 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f095815e810 0x7f095815b1d0 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f0950009d90 tx=0x7f095002beb0 comp rx=0 tx=0).stop 2024-03-04T09:29:40.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:40.010+0000 7f095e800640 1 -- 172.21.15.6:0/1054210835 shutdown_connections 2024-03-04T09:29:40.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:40.010+0000 7f095e800640 1 --2- 172.21.15.6:0/1054210835 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f095815e810 0x7f095815b1d0 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:40.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:40.010+0000 7f095e800640 1 -- 172.21.15.6:0/1054210835 >> 172.21.15.6:0/1054210835 conn(0x7f095806b3d0 msgr2=0x7f095806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:29:40.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:40.010+0000 7f095e800640 1 -- 172.21.15.6:0/1054210835 shutdown_connections 2024-03-04T09:29:40.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:40.010+0000 7f095e800640 1 -- 172.21.15.6:0/1054210835 wait complete. 2024-03-04T09:29:40.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:40.012+0000 7f095e800640 1 Processor -- start 2024-03-04T09:29:40.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:40.012+0000 7f095e800640 1 -- start start 2024-03-04T09:29:40.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:40.013+0000 7f095e800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f09580cd060 0x7f0958147390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:40.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:40.013+0000 7f095e800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f09581552b0 0x7f0958152900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:40.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:40.013+0000 7f095e800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f095815e810 0x7f0958152e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:40.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:40.013+0000 7f095e800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f095806dd30 con 0x7f09581552b0 2024-03-04T09:29:40.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:40.013+0000 7f095e800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09580b3760 con 0x7f095815e810 2024-03-04T09:29:40.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:40.013+0000 7f095e800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f095806deb0 con 0x7f09580cd060 2024-03-04T09:29:40.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:40.013+0000 7f095ca00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f09581552b0 0x7f0958152900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:40.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:40.013+0000 7f095ca00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f09581552b0 0x7f0958152900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38694/0 (socket says 172.21.15.6:38694) 2024-03-04T09:29:40.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:40.013+0000 7f095ca00640 1 -- 172.21.15.6:0/280159366 learned_addr learned my addr 172.21.15.6:0/280159366 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:29:40.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:40.014+0000 7f095d400640 1 --2- 172.21.15.6:0/280159366 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f09580cd060 0x7f0958147390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:40.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:40.014+0000 7f095de00640 1 --2- 172.21.15.6:0/280159366 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f095815e810 0x7f0958152e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:40.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:40.014+0000 7f095ca00640 1 -- 172.21.15.6:0/280159366 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f09580cd060 msgr2=0x7f0958147390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:40.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:40.014+0000 7f095ca00640 1 --2- 172.21.15.6:0/280159366 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f09580cd060 0x7f0958147390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:40.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:40.014+0000 7f095ca00640 1 -- 172.21.15.6:0/280159366 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f095815e810 msgr2=0x7f0958152e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:40.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:40.014+0000 7f095ca00640 1 --2- 172.21.15.6:0/280159366 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f095815e810 0x7f0958152e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:40.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:40.014+0000 7f095ca00640 1 -- 172.21.15.6:0/280159366 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09581478d0 con 0x7f09581552b0 2024-03-04T09:29:40.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:40.014+0000 7f095de00640 1 --2- 172.21.15.6:0/280159366 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f095815e810 0x7f0958152e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:29:40.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:40.014+0000 7f095d400640 1 --2- 172.21.15.6:0/280159366 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f09580cd060 0x7f0958147390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:29:40.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:40.014+0000 7f095ca00640 1 --2- 172.21.15.6:0/280159366 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f09581552b0 0x7f0958152900 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f094400c4a0 tx=0x7f094400c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:40.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:40.014+0000 7f0955600640 1 -- 172.21.15.6:0/280159366 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0944017070 con 0x7f09581552b0 2024-03-04T09:29:40.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:40.014+0000 7f0955600640 1 -- 172.21.15.6:0/280159366 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0944005ce0 con 0x7f09581552b0 2024-03-04T09:29:40.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:40.014+0000 7f095e800640 1 -- 172.21.15.6:0/280159366 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09580ae4e0 con 0x7f09581552b0 2024-03-04T09:29:40.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:40.015+0000 7f0955600640 1 -- 172.21.15.6:0/280159366 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09440072b0 con 0x7f09581552b0 2024-03-04T09:29:40.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:40.015+0000 7f095e800640 1 -- 172.21.15.6:0/280159366 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09580aeaa0 con 0x7f09581552b0 2024-03-04T09:29:40.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:40.016+0000 7f095e800640 1 -- 172.21.15.6:0/280159366 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f095804f330 con 0x7f09581552b0 2024-03-04T09:29:40.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:40.017+0000 7f0955600640 1 -- 172.21.15.6:0/280159366 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f094401a070 con 0x7f09581552b0 2024-03-04T09:29:40.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:40.018+0000 7f0955600640 1 --2- 172.21.15.6:0/280159366 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f093806b4d0 0x7f093806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:40.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:40.018+0000 7f0955600640 1 -- 172.21.15.6:0/280159366 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f09440893e0 con 0x7f09581552b0 2024-03-04T09:29:40.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:40.018+0000 7f095d400640 1 --2- 172.21.15.6:0/280159366 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f093806b4d0 0x7f093806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:40.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:40.019+0000 7f095d400640 1 --2- 172.21.15.6:0/280159366 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f093806b4d0 0x7f093806d990 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f094c003eb0 tx=0x7f094c009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:40.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:40.022+0000 7f0955600640 1 -- 172.21.15.6:0/280159366 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f09581478d0 con 0x7f09581552b0 2024-03-04T09:29:40.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:40.294+0000 7f095e800640 1 -- 172.21.15.6:0/280159366 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_fancy_layout"]} v 0) v1 -- 0x7f09580b2160 con 0x7f09581552b0 2024-03-04T09:29:40.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:40.802+0000 7f0955600640 1 -- 172.21.15.6:0/280159366 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_fancy_layout"]}]=0 v250) v1 ==== 135+0+0 (secure 0 0 0) 0x7f0944058430 con 0x7f09581552b0 2024-03-04T09:29:40.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:40.805+0000 7f095e800640 1 -- 172.21.15.6:0/280159366 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f093806b4d0 msgr2=0x7f093806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:40.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:40.805+0000 7f095e800640 1 --2- 172.21.15.6:0/280159366 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f093806b4d0 0x7f093806d990 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f094c003eb0 tx=0x7f094c009040 comp rx=0 tx=0).stop 2024-03-04T09:29:40.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:40.805+0000 7f095e800640 1 -- 172.21.15.6:0/280159366 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f09581552b0 msgr2=0x7f0958152900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:40.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:40.805+0000 7f095e800640 1 --2- 172.21.15.6:0/280159366 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f09581552b0 0x7f0958152900 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f094400c4a0 tx=0x7f094400c960 comp rx=0 tx=0).stop 2024-03-04T09:29:40.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:40.805+0000 7f095e800640 1 -- 172.21.15.6:0/280159366 shutdown_connections 2024-03-04T09:29:40.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:40.805+0000 7f095e800640 1 --2- 172.21.15.6:0/280159366 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f095815e810 0x7f0958152e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:40.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:40.805+0000 7f095e800640 1 --2- 172.21.15.6:0/280159366 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f09581552b0 0x7f0958152900 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:40.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:40.805+0000 7f095e800640 1 --2- 172.21.15.6:0/280159366 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f093806b4d0 0x7f093806d990 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:40.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:40.805+0000 7f095e800640 1 --2- 172.21.15.6:0/280159366 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f09580cd060 0x7f0958147390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:40.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:40.805+0000 7f095e800640 1 -- 172.21.15.6:0/280159366 >> 172.21.15.6:0/280159366 conn(0x7f095806b3d0 msgr2=0x7f09580ce820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:29:40.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:40.806+0000 7f095e800640 1 -- 172.21.15.6:0/280159366 shutdown_connections 2024-03-04T09:29:40.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:40.806+0000 7f095e800640 1 -- 172.21.15.6:0/280159366 wait complete. 2024-03-04T09:29:40.825 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-04T09:29:41.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:41.053+0000 7f309e400640 1 Processor -- start 2024-03-04T09:29:41.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:41.053+0000 7f309e400640 1 -- start start 2024-03-04T09:29:41.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:41.054+0000 7f309e400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f309806da40 0x7f309806e340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:41.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:41.054+0000 7f309e400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f30980ccad0 0x7f30980ccf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:41.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:41.054+0000 7f309e400640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f309806d090 con 0x7f309815e680 2024-03-04T09:29:41.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:41.054+0000 7f309e400640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3098161c70 con 0x7f30980ccad0 2024-03-04T09:29:41.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:41.054+0000 7f309e400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f309815eb80 con 0x7f309806da40 2024-03-04T09:29:41.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:41.054+0000 7f308ea00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f309815e680 0x7f309806c980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:59416/0 (socket says 172.21.15.6:59416) 2024-03-04T09:29:41.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:41.054+0000 7f308ea00640 1 -- 172.21.15.6:0/1017469266 learned_addr learned my addr 172.21.15.6:0/1017469266 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:29:41.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:41.055+0000 7f3097e00640 1 --2- 172.21.15.6:0/1017469266 >> v2:172.21.15.146:3300/0 conn(0x7f30980ccad0 0x7f30980ccf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:41.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:41.055+0000 7f3097400640 1 --2- 172.21.15.6:0/1017469266 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f309806da40 0x7f309806e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:41.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:41.055+0000 7f3096a00640 1 -- 172.21.15.6:0/1017469266 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3998750653 0 0) 0x7f309806d090 con 0x7f309815e680 2024-03-04T09:29:41.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:41.055+0000 7f3096a00640 1 -- 172.21.15.6:0/1017469266 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f307c003520 con 0x7f309815e680 2024-03-04T09:29:41.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:41.055+0000 7f3096a00640 1 -- 172.21.15.6:0/1017469266 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2277275082 0 0) 0x7f307c003520 con 0x7f309815e680 2024-03-04T09:29:41.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:41.055+0000 7f3096a00640 1 -- 172.21.15.6:0/1017469266 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f309806da40 msgr2=0x7f309806e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:41.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:41.055+0000 7f3096a00640 1 --2- 172.21.15.6:0/1017469266 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f309806da40 0x7f309806e340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:41.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:41.055+0000 7f3096a00640 1 -- 172.21.15.6:0/1017469266 >> v2:172.21.15.146:3300/0 conn(0x7f30980ccad0 msgr2=0x7f30980ccf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:41.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:41.055+0000 7f3096a00640 1 --2- 172.21.15.6:0/1017469266 >> v2:172.21.15.146:3300/0 conn(0x7f30980ccad0 0x7f30980ccf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:41.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:41.055+0000 7f3096a00640 1 -- 172.21.15.6:0/1017469266 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f309815ee60 con 0x7f309815e680 2024-03-04T09:29:41.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:41.055+0000 7f3096a00640 1 -- 172.21.15.6:0/1017469266 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7f30800019f0 con 0x7f309815e680 2024-03-04T09:29:41.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:41.056+0000 7f3096a00640 1 -- 172.21.15.6:0/1017469266 >> v1:172.21.15.6:6789/0 conn(0x7f309815e680 legacy=0x7f309806c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:41.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:41.056+0000 7f3096a00640 1 --2- 172.21.15.6:0/1017469266 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f309806da40 0x7f307c0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:41.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:41.056+0000 7f3096a00640 1 --2- 172.21.15.6:0/1017469266 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f307c003f30 0x7f307c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:41.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:41.056+0000 7f3096a00640 1 --2- 172.21.15.6:0/1017469266 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f307c004d40 0x7f307c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:41.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:41.056+0000 7f3096a00640 1 -- 172.21.15.6:0/1017469266 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3098161c70 con 0x7f307c004d40 2024-03-04T09:29:41.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:41.056+0000 7f3096a00640 1 -- 172.21.15.6:0/1017469266 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f309815eb80 con 0x7f307c003f30 2024-03-04T09:29:41.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:41.056+0000 7f3096a00640 1 -- 172.21.15.6:0/1017469266 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f307c00b9c0 con 0x7f309806da40 2024-03-04T09:29:41.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:41.056+0000 7f3097e00640 1 --2- 172.21.15.6:0/1017469266 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f307c004d40 0x7f307c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:41.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:41.056+0000 7f3097400640 1 --2- 172.21.15.6:0/1017469266 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f309806da40 0x7f307c0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:41.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:41.056+0000 7f3097e00640 1 -- 172.21.15.6:0/1017469266 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f309806da40 msgr2=0x7f307c0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:41.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:41.056+0000 7f3097e00640 1 --2- 172.21.15.6:0/1017469266 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f309806da40 0x7f307c0039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:41.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:41.056+0000 7f3097e00640 1 -- 172.21.15.6:0/1017469266 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f307c003f30 msgr2=0x7f307c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:41.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:41.056+0000 7f3097e00640 1 --2- 172.21.15.6:0/1017469266 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f307c003f30 0x7f307c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:41.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:41.056+0000 7f3097e00640 1 -- 172.21.15.6:0/1017469266 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f307c003520 con 0x7f307c004d40 2024-03-04T09:29:41.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:41.058+0000 7f3097e00640 1 --2- 172.21.15.6:0/1017469266 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f307c004d40 0x7f307c0051c0 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f3090009ac0 tx=0x7f307c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82293c22b9b35a87 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:41.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:41.058+0000 7f3096a00640 1 -- 172.21.15.6:0/1017469266 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3090039070 con 0x7f307c004d40 2024-03-04T09:29:41.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:41.058+0000 7f3096a00640 1 -- 172.21.15.6:0/1017469266 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f309002c6d0 con 0x7f307c004d40 2024-03-04T09:29:41.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:41.058+0000 7f3096a00640 1 -- 172.21.15.6:0/1017469266 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3090009690 con 0x7f307c004d40 2024-03-04T09:29:41.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:41.058+0000 7f309e400640 1 -- 172.21.15.6:0/1017469266 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f307c004d40 msgr2=0x7f307c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:41.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:41.058+0000 7f309e400640 1 --2- 172.21.15.6:0/1017469266 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f307c004d40 0x7f307c0051c0 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f3090009ac0 tx=0x7f307c005830 comp rx=0 tx=0).stop 2024-03-04T09:29:41.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:41.058+0000 7f3097e00640 1 -- 172.21.15.6:0/1017469266 reap_dead start 2024-03-04T09:29:41.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:41.058+0000 7f309e400640 1 -- 172.21.15.6:0/1017469266 shutdown_connections 2024-03-04T09:29:41.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:41.058+0000 7f309e400640 1 -- 172.21.15.6:0/1017469266 >> 172.21.15.6:0/1017469266 conn(0x7f309806b3d0 msgr2=0x7f309806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:29:41.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:41.059+0000 7f309e400640 1 -- 172.21.15.6:0/1017469266 shutdown_connections 2024-03-04T09:29:41.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:41.059+0000 7f309e400640 1 -- 172.21.15.6:0/1017469266 wait complete. 2024-03-04T09:29:41.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:41.060+0000 7f309e400640 1 Processor -- start 2024-03-04T09:29:41.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:41.061+0000 7f309e400640 1 -- start start 2024-03-04T09:29:41.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:41.062+0000 7f309e400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f307c004d40 0x7f30980cbe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:41.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:41.062+0000 7f309e400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f309814f610 0x7f30980c8fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:41.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:41.062+0000 7f309e400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f30980b07f0 0x7f30980ad470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:41.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:41.062+0000 7f309e400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f309806e340 con 0x7f307c004d40 2024-03-04T09:29:41.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:41.062+0000 7f309e400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f309815f420 con 0x7f30980b07f0 2024-03-04T09:29:41.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:41.062+0000 7f309e400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f309806e4c0 con 0x7f309814f610 2024-03-04T09:29:41.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:41.062+0000 7f3097400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f307c004d40 0x7f30980cbe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:41.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:41.062+0000 7f3097400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f307c004d40 0x7f30980cbe50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38720/0 (socket says 172.21.15.6:38720) 2024-03-04T09:29:41.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:41.062+0000 7f3097400640 1 -- 172.21.15.6:0/2202978993 learned_addr learned my addr 172.21.15.6:0/2202978993 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:29:41.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:41.062+0000 7f3097400640 1 -- 172.21.15.6:0/2202978993 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f309814f610 msgr2=0x7f30980c8fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:41.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:41.062+0000 7f308ea00640 1 --2- 172.21.15.6:0/2202978993 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f309814f610 0x7f30980c8fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:41.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:41.062+0000 7f3097400640 1 --2- 172.21.15.6:0/2202978993 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f309814f610 0x7f30980c8fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:41.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:41.062+0000 7f3097400640 1 -- 172.21.15.6:0/2202978993 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f30980b07f0 msgr2=0x7f30980ad470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:41.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:41.062+0000 7f3097e00640 1 --2- 172.21.15.6:0/2202978993 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f30980b07f0 0x7f30980ad470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:41.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:41.062+0000 7f3097400640 1 --2- 172.21.15.6:0/2202978993 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f30980b07f0 0x7f30980ad470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:41.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:41.062+0000 7f3097400640 1 -- 172.21.15.6:0/2202978993 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f309814fa90 con 0x7f307c004d40 2024-03-04T09:29:41.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:41.063+0000 7f3097400640 1 --2- 172.21.15.6:0/2202978993 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f307c004d40 0x7f30980cbe50 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f3088011660 tx=0x7f3088011b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:41.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:41.063+0000 7f308fe00640 1 -- 172.21.15.6:0/2202978993 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f308800f9c0 con 0x7f307c004d40 2024-03-04T09:29:41.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:41.063+0000 7f308fe00640 1 -- 172.21.15.6:0/2202978993 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f308800fb60 con 0x7f307c004d40 2024-03-04T09:29:41.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:41.063+0000 7f308fe00640 1 -- 172.21.15.6:0/2202978993 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3088015460 con 0x7f307c004d40 2024-03-04T09:29:41.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:41.063+0000 7f309e400640 1 -- 172.21.15.6:0/2202978993 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30980afa30 con 0x7f307c004d40 2024-03-04T09:29:41.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:41.063+0000 7f309e400640 1 -- 172.21.15.6:0/2202978993 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30980afff0 con 0x7f307c004d40 2024-03-04T09:29:41.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:41.064+0000 7f309e400640 1 -- 172.21.15.6:0/2202978993 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f309804f330 con 0x7f307c004d40 2024-03-04T09:29:41.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:41.065+0000 7f308fe00640 1 -- 172.21.15.6:0/2202978993 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3088015600 con 0x7f307c004d40 2024-03-04T09:29:41.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:41.066+0000 7f308fe00640 1 --2- 172.21.15.6:0/2202978993 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f307406b4d0 0x7f307406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:41.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:41.066+0000 7f308fe00640 1 -- 172.21.15.6:0/2202978993 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f308808a550 con 0x7f307c004d40 2024-03-04T09:29:41.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:41.067+0000 7f308ea00640 1 --2- 172.21.15.6:0/2202978993 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f307406b4d0 0x7f307406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:41.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:41.068+0000 7f308ea00640 1 --2- 172.21.15.6:0/2202978993 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f307406b4d0 0x7f307406d990 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f30800012e0 tx=0x7f3080009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:41.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:41.071+0000 7f308fe00640 1 -- 172.21.15.6:0/2202978993 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3088019030 con 0x7f307c004d40 2024-03-04T09:29:41.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:41.271+0000 7f309e400640 1 -- 172.21.15.6:0/2202978993 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f309815c210 con 0x7f307c004d40 2024-03-04T09:29:41.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:41.278+0000 7f308fe00640 1 -- 172.21.15.6:0/2202978993 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v26)=0 v26) v1 ==== 122+0+0 (secure 0 0 0) 0x7f3088059630 con 0x7f307c004d40 2024-03-04T09:29:41.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:41.280+0000 7f309e400640 1 -- 172.21.15.6:0/2202978993 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f307406b4d0 msgr2=0x7f307406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:41.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:41.280+0000 7f309e400640 1 --2- 172.21.15.6:0/2202978993 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f307406b4d0 0x7f307406d990 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f30800012e0 tx=0x7f3080009040 comp rx=0 tx=0).stop 2024-03-04T09:29:41.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:41.280+0000 7f309e400640 1 -- 172.21.15.6:0/2202978993 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f307c004d40 msgr2=0x7f30980cbe50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:41.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:41.280+0000 7f309e400640 1 --2- 172.21.15.6:0/2202978993 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f307c004d40 0x7f30980cbe50 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f3088011660 tx=0x7f3088011b20 comp rx=0 tx=0).stop 2024-03-04T09:29:41.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:41.281+0000 7f309e400640 1 -- 172.21.15.6:0/2202978993 shutdown_connections 2024-03-04T09:29:41.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:41.281+0000 7f309e400640 1 --2- 172.21.15.6:0/2202978993 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f30980b07f0 0x7f30980ad470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:41.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:41.281+0000 7f309e400640 1 --2- 172.21.15.6:0/2202978993 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f307406b4d0 0x7f307406d990 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:41.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:41.281+0000 7f309e400640 1 --2- 172.21.15.6:0/2202978993 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f309814f610 0x7f30980c8fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:41.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:41.281+0000 7f309e400640 1 --2- 172.21.15.6:0/2202978993 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f307c004d40 0x7f30980cbe50 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:41.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:41.281+0000 7f309e400640 1 -- 172.21.15.6:0/2202978993 >> 172.21.15.6:0/2202978993 conn(0x7f309806b3d0 msgr2=0x7f309815dcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:29:41.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:41.281+0000 7f309e400640 1 -- 172.21.15.6:0/2202978993 shutdown_connections 2024-03-04T09:29:41.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:41.281+0000 7f309e400640 1 -- 172.21.15.6:0/2202978993 wait complete. 2024-03-04T09:29:41.297 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-04T09:29:41.298 DEBUG:teuthology.orchestra.run.smithi006:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-04T09:29:41.416 DEBUG:teuthology.orchestra.run.smithi146:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-04T09:29:41.534 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-04T09:29:41.534 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-04T09:29:41.534 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T09:29:41.594 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054202,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249800k,nr_inodes=812450,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-04T09:29:41.595 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-04T09:29:41.595 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T09:29:41.652 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053429,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249180k,nr_inodes=812295,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-04T09:29:41.652 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-04T09:29:41.652 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T09:29:41.710 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054202,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249800k,nr_inodes=812450,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-04T09:29:41.711 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-04T09:29:41.711 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T09:29:41.767 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053429,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249180k,nr_inodes=812295,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-04T09:29:41.768 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-04T09:29:41.768 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:29:42.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.004+0000 7f3d42c00640 1 Processor -- start 2024-03-04T09:29:42.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.004+0000 7f3d42c00640 1 -- start start 2024-03-04T09:29:42.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.006+0000 7f3d42c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3d3c161670 0x7f3d3c070c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:42.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.006+0000 7f3d42c00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f3d3c06e240 0x7f3d3c0711b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:42.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.006+0000 7f3d42c00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d3c0ce9a0 con 0x7f3d3c06cfe0 2024-03-04T09:29:42.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.006+0000 7f3d42c00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d3c0730d0 con 0x7f3d3c06e240 2024-03-04T09:29:42.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.006+0000 7f3d42c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d3c0cec80 con 0x7f3d3c161670 2024-03-04T09:29:42.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.006+0000 7f3d40c00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f3d3c06cfe0 0x7f3d3c06d420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:59418/0 (socket says 172.21.15.6:59418) 2024-03-04T09:29:42.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.006+0000 7f3d40c00640 1 -- 172.21.15.6:0/3101477852 learned_addr learned my addr 172.21.15.6:0/3101477852 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:29:42.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.007+0000 7f3d3b400640 1 --2- 172.21.15.6:0/3101477852 >> v2:172.21.15.146:3300/0 conn(0x7f3d3c06e240 0x7f3d3c0711b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:42.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.007+0000 7f3d3be00640 1 --2- 172.21.15.6:0/3101477852 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3d3c161670 0x7f3d3c070c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:42.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.007+0000 7f3d3aa00640 1 -- 172.21.15.6:0/3101477852 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4060496912 0 0) 0x7f3d3c0ce9a0 con 0x7f3d3c06cfe0 2024-03-04T09:29:42.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.007+0000 7f3d3aa00640 1 -- 172.21.15.6:0/3101477852 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d20003520 con 0x7f3d3c06cfe0 2024-03-04T09:29:42.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.007+0000 7f3d3aa00640 1 -- 172.21.15.6:0/3101477852 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 98489487 0 0) 0x7f3d20003520 con 0x7f3d3c06cfe0 2024-03-04T09:29:42.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.007+0000 7f3d3aa00640 1 -- 172.21.15.6:0/3101477852 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3d3c161670 msgr2=0x7f3d3c070c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:42.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.007+0000 7f3d3aa00640 1 --2- 172.21.15.6:0/3101477852 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3d3c161670 0x7f3d3c070c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:42.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.007+0000 7f3d3aa00640 1 -- 172.21.15.6:0/3101477852 >> v2:172.21.15.146:3300/0 conn(0x7f3d3c06e240 msgr2=0x7f3d3c0711b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:42.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.007+0000 7f3d3aa00640 1 --2- 172.21.15.6:0/3101477852 >> v2:172.21.15.146:3300/0 conn(0x7f3d3c06e240 0x7f3d3c0711b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:42.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.007+0000 7f3d3aa00640 1 -- 172.21.15.6:0/3101477852 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d3c0cef60 con 0x7f3d3c06cfe0 2024-03-04T09:29:42.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.007+0000 7f3d3aa00640 1 -- 172.21.15.6:0/3101477852 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7f3d340019f0 con 0x7f3d3c06cfe0 2024-03-04T09:29:42.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.007+0000 7f3d3aa00640 1 -- 172.21.15.6:0/3101477852 >> v1:172.21.15.6:6789/0 conn(0x7f3d3c06cfe0 legacy=0x7f3d3c06d420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:42.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.007+0000 7f3d3aa00640 1 --2- 172.21.15.6:0/3101477852 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3d20003a20 0x7f3d20003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:42.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.007+0000 7f3d3aa00640 1 --2- 172.21.15.6:0/3101477852 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3d20004800 0x7f3d20004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:42.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.007+0000 7f3d3aa00640 1 --2- 172.21.15.6:0/3101477852 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3d3c161670 0x7f3d200051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:42.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.007+0000 7f3d3aa00640 1 -- 172.21.15.6:0/3101477852 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d3c0730d0 con 0x7f3d20004800 2024-03-04T09:29:42.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.007+0000 7f3d3aa00640 1 -- 172.21.15.6:0/3101477852 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d3c0cec80 con 0x7f3d20003a20 2024-03-04T09:29:42.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.007+0000 7f3d3aa00640 1 -- 172.21.15.6:0/3101477852 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d2000b9c0 con 0x7f3d3c161670 2024-03-04T09:29:42.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.008+0000 7f3d3b400640 1 --2- 172.21.15.6:0/3101477852 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3d20004800 0x7f3d20004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:42.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.008+0000 7f3d3be00640 1 --2- 172.21.15.6:0/3101477852 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3d20003a20 0x7f3d20003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:42.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.008+0000 7f3d3b400640 1 -- 172.21.15.6:0/3101477852 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3d3c161670 msgr2=0x7f3d200051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:42.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.008+0000 7f3d3b400640 1 --2- 172.21.15.6:0/3101477852 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3d3c161670 0x7f3d200051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:42.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.008+0000 7f3d3b400640 1 -- 172.21.15.6:0/3101477852 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3d20003a20 msgr2=0x7f3d20003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:42.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.008+0000 7f3d3b400640 1 --2- 172.21.15.6:0/3101477852 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3d20003a20 0x7f3d20003e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:42.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.008+0000 7f3d3b400640 1 -- 172.21.15.6:0/3101477852 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d20003520 con 0x7f3d20004800 2024-03-04T09:29:42.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.009+0000 7f3d3b400640 1 --2- 172.21.15.6:0/3101477852 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3d20004800 0x7f3d20004c80 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f3d28009af0 tx=0x7f3d20005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea6e49283354602a server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:42.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.010+0000 7f3d3aa00640 1 -- 172.21.15.6:0/3101477852 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d28039070 con 0x7f3d20004800 2024-03-04T09:29:42.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.010+0000 7f3d3aa00640 1 -- 172.21.15.6:0/3101477852 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d2802c710 con 0x7f3d20004800 2024-03-04T09:29:42.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.010+0000 7f3d3aa00640 1 -- 172.21.15.6:0/3101477852 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d28009690 con 0x7f3d20004800 2024-03-04T09:29:42.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.010+0000 7f3d42c00640 1 -- 172.21.15.6:0/3101477852 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3d20004800 msgr2=0x7f3d20004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:42.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.010+0000 7f3d42c00640 1 --2- 172.21.15.6:0/3101477852 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3d20004800 0x7f3d20004c80 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f3d28009af0 tx=0x7f3d20005830 comp rx=0 tx=0).stop 2024-03-04T09:29:42.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.010+0000 7f3d40c00640 1 -- 172.21.15.6:0/3101477852 reap_dead start 2024-03-04T09:29:42.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.011+0000 7f3d42c00640 1 -- 172.21.15.6:0/3101477852 shutdown_connections 2024-03-04T09:29:42.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.011+0000 7f3d42c00640 1 -- 172.21.15.6:0/3101477852 >> 172.21.15.6:0/3101477852 conn(0x7f3d3c06afb0 msgr2=0x7f3d3c06b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:29:42.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.011+0000 7f3d42c00640 1 -- 172.21.15.6:0/3101477852 shutdown_connections 2024-03-04T09:29:42.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.011+0000 7f3d42c00640 1 -- 172.21.15.6:0/3101477852 wait complete. 2024-03-04T09:29:42.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.013+0000 7f3d42c00640 1 Processor -- start 2024-03-04T09:29:42.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.013+0000 7f3d42c00640 1 -- start start 2024-03-04T09:29:42.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.014+0000 7f3d42c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3d20004800 0x7f3d3c09aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:42.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.014+0000 7f3d42c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3d3c09af50 0x7f3d3c1543d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:42.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.014+0000 7f3d42c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3d3c154910 0x7f3d3c09c710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:42.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.014+0000 7f3d42c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d3c0ce600 con 0x7f3d20004800 2024-03-04T09:29:42.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.014+0000 7f3d42c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d3c0cf520 con 0x7f3d3c154910 2024-03-04T09:29:42.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.014+0000 7f3d42c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d3c0ce780 con 0x7f3d3c09af50 2024-03-04T09:29:42.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.015+0000 7f3d3be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3d20004800 0x7f3d3c09aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:42.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.015+0000 7f3d3be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3d20004800 0x7f3d3c09aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38734/0 (socket says 172.21.15.6:38734) 2024-03-04T09:29:42.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.015+0000 7f3d3be00640 1 -- 172.21.15.6:0/243868398 learned_addr learned my addr 172.21.15.6:0/243868398 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:29:42.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.015+0000 7f3d40c00640 1 --2- 172.21.15.6:0/243868398 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3d3c154910 0x7f3d3c09c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:42.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.015+0000 7f3d3b400640 1 --2- 172.21.15.6:0/243868398 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3d3c09af50 0x7f3d3c1543d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:42.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.015+0000 7f3d3be00640 1 -- 172.21.15.6:0/243868398 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3d3c09af50 msgr2=0x7f3d3c1543d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:42.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.015+0000 7f3d3be00640 1 --2- 172.21.15.6:0/243868398 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3d3c09af50 0x7f3d3c1543d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:42.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.015+0000 7f3d3be00640 1 -- 172.21.15.6:0/243868398 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3d3c154910 msgr2=0x7f3d3c09c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:42.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.015+0000 7f3d3be00640 1 --2- 172.21.15.6:0/243868398 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3d3c154910 0x7f3d3c09c710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:42.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.015+0000 7f3d3be00640 1 -- 172.21.15.6:0/243868398 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d3c15dbd0 con 0x7f3d20004800 2024-03-04T09:29:42.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.015+0000 7f3d40c00640 1 --2- 172.21.15.6:0/243868398 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3d3c154910 0x7f3d3c09c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:29:42.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.016+0000 7f3d3be00640 1 --2- 172.21.15.6:0/243868398 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3d20004800 0x7f3d3c09aa10 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f3d2c011600 tx=0x7f3d2c011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:42.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.016+0000 7f3d33e00640 1 -- 172.21.15.6:0/243868398 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d2c00f920 con 0x7f3d20004800 2024-03-04T09:29:42.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.016+0000 7f3d33e00640 1 -- 172.21.15.6:0/243868398 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d2c00fac0 con 0x7f3d20004800 2024-03-04T09:29:42.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.016+0000 7f3d33e00640 1 -- 172.21.15.6:0/243868398 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d2c00fe70 con 0x7f3d20004800 2024-03-04T09:29:42.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.016+0000 7f3d42c00640 1 -- 172.21.15.6:0/243868398 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d3c15deb0 con 0x7f3d20004800 2024-03-04T09:29:42.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.016+0000 7f3d42c00640 1 -- 172.21.15.6:0/243868398 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d3c15e470 con 0x7f3d20004800 2024-03-04T09:29:42.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.017+0000 7f3d42c00640 1 -- 172.21.15.6:0/243868398 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d3c069510 con 0x7f3d20004800 2024-03-04T09:29:42.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.018+0000 7f3d33e00640 1 -- 172.21.15.6:0/243868398 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3d2c015460 con 0x7f3d20004800 2024-03-04T09:29:42.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.019+0000 7f3d33e00640 1 --2- 172.21.15.6:0/243868398 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f3d1c06b500 0x7f3d1c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:42.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.019+0000 7f3d33e00640 1 -- 172.21.15.6:0/243868398 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f3d2c08a6a0 con 0x7f3d20004800 2024-03-04T09:29:42.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.020+0000 7f3d3b400640 1 --2- 172.21.15.6:0/243868398 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f3d1c06b500 0x7f3d1c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:42.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.021+0000 7f3d3b400640 1 --2- 172.21.15.6:0/243868398 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f3d1c06b500 0x7f3d1c06d9c0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f3d28036000 tx=0x7f3d28030000 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:42.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.024+0000 7f3d33e00640 1 -- 172.21.15.6:0/243868398 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3d2c019030 con 0x7f3d20004800 2024-03-04T09:29:42.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.263+0000 7f3d42c00640 1 -- 172.21.15.6:0/243868398 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3d3c0b52d0 con 0x7f3d20004800 2024-03-04T09:29:42.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.264+0000 7f3d33e00640 1 -- 172.21.15.6:0/243868398 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) v1 ==== 94+0+3699 (secure 0 0 0) 0x7f3d2c0598b0 con 0x7f3d20004800 2024-03-04T09:29:42.262 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:29:42.262 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":44,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5228,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/1891254736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1891254736},{"type":"v1","addr":"172.21.15.6:6834","nonce":1891254736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":5578,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/1781219843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":1781219843},{"type":"v1","addr":"172.21.15.146:6835","nonce":1781219843}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":5587,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/1479180802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":1479180802},{"type":"v1","addr":"172.21.15.6:6835","nonce":1479180802}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":5767,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/1950694644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":1950694644},{"type":"v1","addr":"172.21.15.146:6837","nonce":1950694644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":44}],"filesystems":[]} 2024-03-04T09:29:42.262 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 44 2024-03-04T09:29:42.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.267+0000 7f3d42c00640 1 -- 172.21.15.6:0/243868398 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f3d1c06b500 msgr2=0x7f3d1c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:42.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.267+0000 7f3d42c00640 1 --2- 172.21.15.6:0/243868398 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f3d1c06b500 0x7f3d1c06d9c0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f3d28036000 tx=0x7f3d28030000 comp rx=0 tx=0).stop 2024-03-04T09:29:42.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.267+0000 7f3d42c00640 1 -- 172.21.15.6:0/243868398 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3d20004800 msgr2=0x7f3d3c09aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:42.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.267+0000 7f3d42c00640 1 --2- 172.21.15.6:0/243868398 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3d20004800 0x7f3d3c09aa10 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f3d2c011600 tx=0x7f3d2c011ac0 comp rx=0 tx=0).stop 2024-03-04T09:29:42.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.267+0000 7f3d42c00640 1 -- 172.21.15.6:0/243868398 shutdown_connections 2024-03-04T09:29:42.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.267+0000 7f3d42c00640 1 --2- 172.21.15.6:0/243868398 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3d3c154910 0x7f3d3c09c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:42.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.267+0000 7f3d42c00640 1 --2- 172.21.15.6:0/243868398 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f3d1c06b500 0x7f3d1c06d9c0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:42.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.267+0000 7f3d42c00640 1 --2- 172.21.15.6:0/243868398 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3d3c09af50 0x7f3d3c1543d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:42.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.267+0000 7f3d42c00640 1 --2- 172.21.15.6:0/243868398 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3d20004800 0x7f3d3c09aa10 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:42.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.267+0000 7f3d42c00640 1 -- 172.21.15.6:0/243868398 >> 172.21.15.6:0/243868398 conn(0x7f3d3c06afb0 msgr2=0x7f3d3c0adec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:29:42.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.268+0000 7f3d42c00640 1 -- 172.21.15.6:0/243868398 shutdown_connections 2024-03-04T09:29:42.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.268+0000 7f3d42c00640 1 -- 172.21.15.6:0/243868398 wait complete. 2024-03-04T09:29:42.286 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-04T09:29:42.286 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-04T09:29:42.286 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-04T09:29:42.287 INFO:teuthology.orchestra.run:waiting for 300 2024-03-04T09:29:42.287 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-04T09:29:42.287 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-04T09:29:42.287 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-04T09:29:42.287 INFO:teuthology.orchestra.run:waiting for 300 2024-03-04T09:29:42.287 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-04T09:29:42.288 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-04T09:29:42.288 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-04T09:29:42.288 INFO:teuthology.orchestra.run:waiting for 300 2024-03-04T09:29:42.288 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-04T09:29:42.288 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-04T09:29:42.288 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-04T09:29:42.288 INFO:teuthology.orchestra.run:waiting for 300 2024-03-04T09:29:42.307 INFO:tasks.ceph.mds.c:Stopped 2024-03-04T09:29:42.308 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-04T09:29:42.309 INFO:tasks.ceph.mds.a:Stopped 2024-03-04T09:29:42.309 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-04T09:29:42.314 INFO:tasks.ceph.mds.d:Stopped 2024-03-04T09:29:42.314 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-04T09:29:42.315 INFO:tasks.ceph.mds.b:Stopped 2024-03-04T09:29:42.315 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-04T09:29:42.357 INFO:tasks.ceph.mds.c:Started 2024-03-04T09:29:42.358 DEBUG:teuthology.parallel:result is None 2024-03-04T09:29:42.361 INFO:tasks.ceph.mds.d:Started 2024-03-04T09:29:42.362 DEBUG:teuthology.parallel:result is None 2024-03-04T09:29:42.364 INFO:tasks.ceph.mds.a:Started 2024-03-04T09:29:42.364 DEBUG:teuthology.parallel:result is None 2024-03-04T09:29:42.366 INFO:tasks.ceph.mds.b:Started 2024-03-04T09:29:42.367 DEBUG:teuthology.parallel:result is None 2024-03-04T09:29:42.367 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-04T09:29:42.520 INFO:tasks.ceph.mds.c.smithi006.stdout:starting mds.c at 2024-03-04T09:29:42.760 INFO:tasks.ceph.mds.b.smithi146.stdout:starting mds.b at 2024-03-04T09:29:42.760 INFO:tasks.ceph.mds.a.smithi006.stdout:starting mds.a at 2024-03-04T09:29:42.761 INFO:tasks.ceph.mds.d.smithi146.stdout:starting mds.d at 2024-03-04T09:29:42.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.561+0000 7fdffa400640 1 Processor -- start 2024-03-04T09:29:42.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.562+0000 7fdffa400640 1 -- start start 2024-03-04T09:29:42.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.563+0000 7fdffa400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fdff4070ee0 0x7fdff4162c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:42.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.563+0000 7fdffa400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fdff4072da0 0x7fdff40cf130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:42.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.563+0000 7fdffa400640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdff40ce090 con 0x7fdff40712c0 2024-03-04T09:29:42.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.563+0000 7fdffa400640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdff40ceca0 con 0x7fdff4070ee0 2024-03-04T09:29:42.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.563+0000 7fdffa400640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdff40ce370 con 0x7fdff4072da0 2024-03-04T09:29:42.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.563+0000 7fdff3400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fdff4070ee0 0x7fdff4162c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:42.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.563+0000 7fdff3400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fdff4070ee0 0x7fdff4162c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38792/0 (socket says 172.21.15.6:38792) 2024-03-04T09:29:42.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.563+0000 7fdff3400640 1 -- 172.21.15.6:0/1932553201 learned_addr learned my addr 172.21.15.6:0/1932553201 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:29:42.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.563+0000 7fdfebe00640 1 --2- 172.21.15.6:0/1932553201 >> v2:172.21.15.146:3300/0 conn(0x7fdff4072da0 0x7fdff40cf130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:42.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.564+0000 7fdff3400640 1 -- 172.21.15.6:0/1932553201 >> v2:172.21.15.146:3300/0 conn(0x7fdff4072da0 msgr2=0x7fdff40cf130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:42.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.564+0000 7fdff3400640 1 --2- 172.21.15.6:0/1932553201 >> v2:172.21.15.146:3300/0 conn(0x7fdff4072da0 0x7fdff40cf130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:42.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.564+0000 7fdff3400640 1 -- 172.21.15.6:0/1932553201 >> v1:172.21.15.6:6789/0 conn(0x7fdff40712c0 legacy=0x7fdff406e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:42.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.564+0000 7fdff3400640 1 -- 172.21.15.6:0/1932553201 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdff40ce650 con 0x7fdff4070ee0 2024-03-04T09:29:42.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.565+0000 7fdff3400640 1 --2- 172.21.15.6:0/1932553201 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdff4070ee0 0x7fdff4162c40 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7fdff406c570 tx=0x7fdfe402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9998b67d8b8961df server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:42.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.565+0000 7fdff2a00640 1 -- 172.21.15.6:0/1932553201 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfe4034070 con 0x7fdff4070ee0 2024-03-04T09:29:42.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.565+0000 7fdff2a00640 1 -- 172.21.15.6:0/1932553201 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfe4030860 con 0x7fdff4070ee0 2024-03-04T09:29:42.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.565+0000 7fdff2a00640 1 -- 172.21.15.6:0/1932553201 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfe4030c30 con 0x7fdff4070ee0 2024-03-04T09:29:42.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.566+0000 7fdffa400640 1 -- 172.21.15.6:0/1932553201 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdff4070ee0 msgr2=0x7fdff4162c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:42.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.566+0000 7fdffa400640 1 --2- 172.21.15.6:0/1932553201 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdff4070ee0 0x7fdff4162c40 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7fdff406c570 tx=0x7fdfe402b9a0 comp rx=0 tx=0).stop 2024-03-04T09:29:42.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.566+0000 7fdffa400640 1 -- 172.21.15.6:0/1932553201 shutdown_connections 2024-03-04T09:29:42.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.566+0000 7fdffa400640 1 --2- 172.21.15.6:0/1932553201 >> v2:172.21.15.146:3300/0 conn(0x7fdff4072da0 0x7fdff40cf130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:42.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.566+0000 7fdffa400640 1 --2- 172.21.15.6:0/1932553201 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdff4070ee0 0x7fdff4162c40 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:42.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.566+0000 7fdffa400640 1 -- 172.21.15.6:0/1932553201 >> 172.21.15.6:0/1932553201 conn(0x7fdff406afb0 msgr2=0x7fdff406b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:29:42.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.566+0000 7fdffa400640 1 -- 172.21.15.6:0/1932553201 shutdown_connections 2024-03-04T09:29:42.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.566+0000 7fdffa400640 1 -- 172.21.15.6:0/1932553201 wait complete. 2024-03-04T09:29:42.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.568+0000 7fdffa400640 1 Processor -- start 2024-03-04T09:29:42.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.568+0000 7fdffa400640 1 -- start start 2024-03-04T09:29:42.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.569+0000 7fdffa400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdff4070ee0 0x7fdff415cb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:42.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.569+0000 7fdffa400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdff40712c0 0x7fdff40ae130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:42.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.569+0000 7fdffa400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdff4072da0 0x7fdff40b41c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:42.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.569+0000 7fdffa400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdff40cdda0 con 0x7fdff4072da0 2024-03-04T09:29:42.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.569+0000 7fdffa400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdff40cdc20 con 0x7fdff4070ee0 2024-03-04T09:29:42.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.569+0000 7fdffa400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdff406e7d0 con 0x7fdff40712c0 2024-03-04T09:29:42.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.569+0000 7fdff3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdff4072da0 0x7fdff40b41c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:42.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.569+0000 7fdff3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdff4072da0 0x7fdff40b41c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38808/0 (socket says 172.21.15.6:38808) 2024-03-04T09:29:42.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.569+0000 7fdff3e00640 1 -- 172.21.15.6:0/1265806678 learned_addr learned my addr 172.21.15.6:0/1265806678 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:29:42.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.569+0000 7fdfebe00640 1 --2- 172.21.15.6:0/1265806678 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdff40712c0 0x7fdff40ae130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:42.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.569+0000 7fdff3400640 1 --2- 172.21.15.6:0/1265806678 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdff4070ee0 0x7fdff415cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:42.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.569+0000 7fdff3e00640 1 -- 172.21.15.6:0/1265806678 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdff40712c0 msgr2=0x7fdff40ae130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:42.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.569+0000 7fdff3e00640 1 --2- 172.21.15.6:0/1265806678 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdff40712c0 0x7fdff40ae130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:42.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.569+0000 7fdff3e00640 1 -- 172.21.15.6:0/1265806678 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdff4070ee0 msgr2=0x7fdff415cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:42.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.569+0000 7fdff3e00640 1 --2- 172.21.15.6:0/1265806678 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdff4070ee0 0x7fdff415cb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:42.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.569+0000 7fdff3e00640 1 -- 172.21.15.6:0/1265806678 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdff40b2b50 con 0x7fdff4072da0 2024-03-04T09:29:42.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.569+0000 7fdfebe00640 1 --2- 172.21.15.6:0/1265806678 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdff40712c0 0x7fdff40ae130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:29:42.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.569+0000 7fdff3400640 1 --2- 172.21.15.6:0/1265806678 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdff4070ee0 0x7fdff415cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:29:42.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.570+0000 7fdff3e00640 1 --2- 172.21.15.6:0/1265806678 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdff4072da0 0x7fdff40b41c0 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7fdfec00c740 tx=0x7fdfec00cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:42.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.570+0000 7fdfeb400640 1 -- 172.21.15.6:0/1265806678 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfec016020 con 0x7fdff4072da0 2024-03-04T09:29:42.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.570+0000 7fdfeb400640 1 -- 172.21.15.6:0/1265806678 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfec00ce50 con 0x7fdff4072da0 2024-03-04T09:29:42.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.570+0000 7fdffa400640 1 -- 172.21.15.6:0/1265806678 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdff40b2e30 con 0x7fdff4072da0 2024-03-04T09:29:42.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.570+0000 7fdfeb400640 1 -- 172.21.15.6:0/1265806678 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfec003970 con 0x7fdff4072da0 2024-03-04T09:29:42.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.570+0000 7fdffa400640 1 -- 172.21.15.6:0/1265806678 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdff40b33f0 con 0x7fdff4072da0 2024-03-04T09:29:42.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.571+0000 7fdffa400640 1 -- 172.21.15.6:0/1265806678 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdff40b38d0 con 0x7fdff4072da0 2024-03-04T09:29:42.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.572+0000 7fdfeb400640 1 -- 172.21.15.6:0/1265806678 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdfec003b10 con 0x7fdff4072da0 2024-03-04T09:29:42.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.573+0000 7fdfeb400640 1 --2- 172.21.15.6:0/1265806678 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fdfd006b4d0 0x7fdfd006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:42.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.573+0000 7fdfeb400640 1 -- 172.21.15.6:0/1265806678 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fdfec089bf0 con 0x7fdff4072da0 2024-03-04T09:29:42.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.573+0000 7fdff3400640 1 --2- 172.21.15.6:0/1265806678 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fdfd006b4d0 0x7fdfd006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:42.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.574+0000 7fdff3400640 1 --2- 172.21.15.6:0/1265806678 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fdfd006b4d0 0x7fdfd006d990 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fdfe4036000 tx=0x7fdfe403b040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:42.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.578+0000 7fdfeb400640 1 -- 172.21.15.6:0/1265806678 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdff40b38d0 con 0x7fdff4072da0 2024-03-04T09:29:42.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.791+0000 7fdffa400640 1 -- 172.21.15.6:0/1265806678 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fdff40b52d0 con 0x7fdff4072da0 2024-03-04T09:29:42.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.831+0000 7fdfeb400640 1 -- 172.21.15.6:0/1265806678 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v75) v1 ==== 97+0+0 (secure 0 0 0) 0x7fdfec020020 con 0x7fdff4072da0 2024-03-04T09:29:42.841 INFO:teuthology.orchestra.run.smithi006.stderr: removed all blocklist entries 2024-03-04T09:29:42.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.835+0000 7fdffa400640 1 -- 172.21.15.6:0/1265806678 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fdfd006b4d0 msgr2=0x7fdfd006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:42.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.835+0000 7fdffa400640 1 --2- 172.21.15.6:0/1265806678 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fdfd006b4d0 0x7fdfd006d990 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fdfe4036000 tx=0x7fdfe403b040 comp rx=0 tx=0).stop 2024-03-04T09:29:42.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.835+0000 7fdffa400640 1 -- 172.21.15.6:0/1265806678 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdff4072da0 msgr2=0x7fdff40b41c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:42.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.835+0000 7fdffa400640 1 --2- 172.21.15.6:0/1265806678 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdff4072da0 0x7fdff40b41c0 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7fdfec00c740 tx=0x7fdfec00cc00 comp rx=0 tx=0).stop 2024-03-04T09:29:42.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.835+0000 7fdffa400640 1 -- 172.21.15.6:0/1265806678 shutdown_connections 2024-03-04T09:29:42.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.835+0000 7fdffa400640 1 --2- 172.21.15.6:0/1265806678 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdff4072da0 0x7fdff40b41c0 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:42.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.835+0000 7fdffa400640 1 --2- 172.21.15.6:0/1265806678 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fdfd006b4d0 0x7fdfd006d990 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:42.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.835+0000 7fdffa400640 1 --2- 172.21.15.6:0/1265806678 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdff40712c0 0x7fdff40ae130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:42.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.835+0000 7fdffa400640 1 --2- 172.21.15.6:0/1265806678 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdff4070ee0 0x7fdff415cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:42.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.835+0000 7fdffa400640 1 -- 172.21.15.6:0/1265806678 >> 172.21.15.6:0/1265806678 conn(0x7fdff406afb0 msgr2=0x7fdff4166030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:29:42.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.835+0000 7fdffa400640 1 -- 172.21.15.6:0/1265806678 shutdown_connections 2024-03-04T09:29:42.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:42.836+0000 7fdffa400640 1 -- 172.21.15.6:0/1265806678 wait complete. 2024-03-04T09:29:42.854 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-04T09:29:43.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.085+0000 7f50cf200640 1 Processor -- start 2024-03-04T09:29:43.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.085+0000 7f50cf200640 1 -- start start 2024-03-04T09:29:43.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.086+0000 7f50cf200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f50d015e2a0 0x7f50d00cef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:43.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.086+0000 7f50cf200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f50d00cf460 0x7f50d00b1330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:43.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.086+0000 7f50cf200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50d00d1120 con 0x7f50d015dec0 2024-03-04T09:29:43.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.086+0000 7f50cf200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50d00b0ea0 con 0x7f50d015e2a0 2024-03-04T09:29:43.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.086+0000 7f50cf200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50d00d1400 con 0x7f50d00cf460 2024-03-04T09:29:43.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.086+0000 7f50cde00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f50d015dec0 0x7f50d00ce810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:59434/0 (socket says 172.21.15.6:59434) 2024-03-04T09:29:43.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.086+0000 7f50cde00640 1 -- 172.21.15.6:0/2958160960 learned_addr learned my addr 172.21.15.6:0/2958160960 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:29:43.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.087+0000 7f50cd400640 1 --2- 172.21.15.6:0/2958160960 >> v2:172.21.15.6:3300/0 conn(0x7f50d015e2a0 0x7f50d00cef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:43.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.087+0000 7f50ce800640 1 --2- 172.21.15.6:0/2958160960 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f50d00cf460 0x7f50d00b1330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:43.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.087+0000 7f50cca00640 1 -- 172.21.15.6:0/2958160960 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3434106623 0 0) 0x7f50d00d1120 con 0x7f50d015dec0 2024-03-04T09:29:43.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.087+0000 7f50cca00640 1 -- 172.21.15.6:0/2958160960 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50b4003520 con 0x7f50d015dec0 2024-03-04T09:29:43.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.087+0000 7f50cd400640 1 -- 172.21.15.6:0/2958160960 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f50d00cf460 msgr2=0x7f50d00b1330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:43.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.087+0000 7f50cd400640 1 --2- 172.21.15.6:0/2958160960 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f50d00cf460 0x7f50d00b1330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:43.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.087+0000 7f50cd400640 1 -- 172.21.15.6:0/2958160960 >> v1:172.21.15.6:6789/0 conn(0x7f50d015dec0 legacy=0x7f50d00ce810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:43.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.087+0000 7f50cd400640 1 -- 172.21.15.6:0/2958160960 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50d00b0b30 con 0x7f50d015e2a0 2024-03-04T09:29:43.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.088+0000 7f50cd400640 1 --2- 172.21.15.6:0/2958160960 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f50d015e2a0 0x7f50d00cef20 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f50bc0097a0 tx=0x7f50bc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=193057a484d2de8a server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:43.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.089+0000 7f50cca00640 1 -- 172.21.15.6:0/2958160960 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50bc035020 con 0x7f50d015e2a0 2024-03-04T09:29:43.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.089+0000 7f50cca00640 1 -- 172.21.15.6:0/2958160960 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50bc031830 con 0x7f50d015e2a0 2024-03-04T09:29:43.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.089+0000 7f50cca00640 1 -- 172.21.15.6:0/2958160960 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50bc031be0 con 0x7f50d015e2a0 2024-03-04T09:29:43.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.089+0000 7f50cf200640 1 -- 172.21.15.6:0/2958160960 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f50d015e2a0 msgr2=0x7f50d00cef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:43.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.089+0000 7f50cf200640 1 --2- 172.21.15.6:0/2958160960 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f50d015e2a0 0x7f50d00cef20 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f50bc0097a0 tx=0x7f50bc02b8d0 comp rx=0 tx=0).stop 2024-03-04T09:29:43.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.089+0000 7f50cf200640 1 -- 172.21.15.6:0/2958160960 shutdown_connections 2024-03-04T09:29:43.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.089+0000 7f50cf200640 1 --2- 172.21.15.6:0/2958160960 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f50d00cf460 0x7f50d00b1330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:43.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.089+0000 7f50cf200640 1 --2- 172.21.15.6:0/2958160960 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f50d015e2a0 0x7f50d00cef20 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:43.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.089+0000 7f50cf200640 1 -- 172.21.15.6:0/2958160960 >> 172.21.15.6:0/2958160960 conn(0x7f50d00d3d40 msgr2=0x7f50d00d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:29:43.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.090+0000 7f50cf200640 1 -- 172.21.15.6:0/2958160960 shutdown_connections 2024-03-04T09:29:43.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.090+0000 7f50cf200640 1 -- 172.21.15.6:0/2958160960 wait complete. 2024-03-04T09:29:43.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.091+0000 7f50cf200640 1 Processor -- start 2024-03-04T09:29:43.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.092+0000 7f50cf200640 1 -- start start 2024-03-04T09:29:43.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.092+0000 7f50cf200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f50d00cf460 0x7f50d00cc900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:43.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.093+0000 7f50cf200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f50d015dec0 0x7f50d014b950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:43.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.093+0000 7f50cf200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f50d015e2a0 0x7f50d00cae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:43.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.093+0000 7f50cf200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50d00d0ec0 con 0x7f50d00cf460 2024-03-04T09:29:43.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.093+0000 7f50cf200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50d00d0d40 con 0x7f50d015dec0 2024-03-04T09:29:43.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.093+0000 7f50cf200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50d00b1870 con 0x7f50d015e2a0 2024-03-04T09:29:43.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.093+0000 7f50cde00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f50d00cf460 0x7f50d00cc900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:43.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.093+0000 7f50cde00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f50d00cf460 0x7f50d00cc900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38832/0 (socket says 172.21.15.6:38832) 2024-03-04T09:29:43.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.093+0000 7f50cde00640 1 -- 172.21.15.6:0/2187755062 learned_addr learned my addr 172.21.15.6:0/2187755062 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:29:43.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.093+0000 7f50ce800640 1 --2- 172.21.15.6:0/2187755062 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f50d015e2a0 0x7f50d00cae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:43.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.093+0000 7f50cde00640 1 -- 172.21.15.6:0/2187755062 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f50d015e2a0 msgr2=0x7f50d00cae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:43.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.093+0000 7f50cde00640 1 --2- 172.21.15.6:0/2187755062 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f50d015e2a0 0x7f50d00cae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:43.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.093+0000 7f50cde00640 1 -- 172.21.15.6:0/2187755062 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f50d015dec0 msgr2=0x7f50d014b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:43.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.093+0000 7f50cde00640 1 --2- 172.21.15.6:0/2187755062 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f50d015dec0 0x7f50d014b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:43.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.093+0000 7f50cde00640 1 -- 172.21.15.6:0/2187755062 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50d00cb860 con 0x7f50d00cf460 2024-03-04T09:29:43.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.093+0000 7f50ce800640 1 --2- 172.21.15.6:0/2187755062 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f50d015e2a0 0x7f50d00cae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:29:43.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.094+0000 7f50cde00640 1 --2- 172.21.15.6:0/2187755062 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f50d00cf460 0x7f50d00cc900 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f50c800cb70 tx=0x7f50c800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:43.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.094+0000 7f50c6000640 1 -- 172.21.15.6:0/2187755062 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50c8017070 con 0x7f50d00cf460 2024-03-04T09:29:43.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.094+0000 7f50c6000640 1 -- 172.21.15.6:0/2187755062 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50c800ad40 con 0x7f50d00cf460 2024-03-04T09:29:43.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.094+0000 7f50c6000640 1 -- 172.21.15.6:0/2187755062 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50c8012620 con 0x7f50d00cf460 2024-03-04T09:29:43.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.094+0000 7f50cf200640 1 -- 172.21.15.6:0/2187755062 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50d00cbb40 con 0x7f50d00cf460 2024-03-04T09:29:43.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.094+0000 7f50cf200640 1 -- 172.21.15.6:0/2187755062 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50d00cc100 con 0x7f50d00cf460 2024-03-04T09:29:43.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.095+0000 7f50cf200640 1 -- 172.21.15.6:0/2187755062 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50d004f330 con 0x7f50d00cf460 2024-03-04T09:29:43.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.096+0000 7f50c6000640 1 -- 172.21.15.6:0/2187755062 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f50c801a070 con 0x7f50d00cf460 2024-03-04T09:29:43.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.097+0000 7f50c6000640 1 --2- 172.21.15.6:0/2187755062 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f50b006b4d0 0x7f50b006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:43.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.097+0000 7f50c6000640 1 -- 172.21.15.6:0/2187755062 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f50c8089da0 con 0x7f50d00cf460 2024-03-04T09:29:43.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.098+0000 7f50cd400640 1 --2- 172.21.15.6:0/2187755062 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f50b006b4d0 0x7f50b006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:43.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.099+0000 7f50cd400640 1 --2- 172.21.15.6:0/2187755062 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f50b006b4d0 0x7f50b006d990 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f50bc036000 tx=0x7f50bc030000 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:43.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.102+0000 7f50c6000640 1 -- 172.21.15.6:0/2187755062 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f50c8058f60 con 0x7f50d00cf460 2024-03-04T09:29:43.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.378+0000 7f50cf200640 1 -- 172.21.15.6:0/2187755062 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f50d00d09f0 con 0x7f50d00cf460 2024-03-04T09:29:43.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.379+0000 7f50c6000640 1 -- 172.21.15.6:0/2187755062 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v13) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f50c8058860 con 0x7f50d00cf460 2024-03-04T09:29:43.378 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:29:43.378 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-04T09:29:43.378 INFO:teuthology.orchestra.run.smithi006.stdout: "auth_dump": [ 2024-03-04T09:29:43.379 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:29:43.379 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "mds.a", 2024-03-04T09:29:43.379 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQA8k+VlHCzFIBAAnKYlmCMDw45LqnSjxnZY5Q==", 2024-03-04T09:29:43.379 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:29:43.379 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-04T09:29:43.379 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow *", 2024-03-04T09:29:43.379 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow *", 2024-03-04T09:29:43.379 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-04T09:29:43.379 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:29:43.379 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:29:43.379 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:29:43.379 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "mds.b", 2024-03-04T09:29:43.379 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQA9k+VlNY7fCBAAX3z8X72/6LkQD6mwGYT69g==", 2024-03-04T09:29:43.380 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:29:43.380 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-04T09:29:43.380 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow *", 2024-03-04T09:29:43.380 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow *", 2024-03-04T09:29:43.380 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-04T09:29:43.380 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:29:43.380 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:29:43.380 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:29:43.380 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "mds.c", 2024-03-04T09:29:43.380 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQA8k+Vl0pjOMhAA7WcTw8awv4EAsrpP4yncNA==", 2024-03-04T09:29:43.380 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:29:43.380 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-04T09:29:43.381 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow *", 2024-03-04T09:29:43.381 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow *", 2024-03-04T09:29:43.381 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-04T09:29:43.381 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:29:43.381 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:29:43.381 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:29:43.381 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "mds.d", 2024-03-04T09:29:43.381 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQA9k+VlbladGRAABNgUcaqTVYuJfCqy5c/G0g==", 2024-03-04T09:29:43.381 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:29:43.381 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-04T09:29:43.382 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow *", 2024-03-04T09:29:43.382 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow *", 2024-03-04T09:29:43.382 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-04T09:29:43.382 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:29:43.382 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:29:43.382 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:29:43.382 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.0", 2024-03-04T09:29:43.382 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQBCk+VlyjMMExAAqb2FJ9n5PWehfx/6PB19ig==", 2024-03-04T09:29:43.382 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:29:43.382 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-04T09:29:43.383 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-04T09:29:43.383 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-04T09:29:43.383 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:29:43.383 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:29:43.383 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:29:43.383 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.1", 2024-03-04T09:29:43.383 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQBFk+VlXD3cLBAAGclSivcYWDslqN8MLlj7eQ==", 2024-03-04T09:29:43.383 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:29:43.383 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-04T09:29:43.383 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-04T09:29:43.384 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-04T09:29:43.384 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:29:43.384 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:29:43.384 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:29:43.384 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.2", 2024-03-04T09:29:43.384 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQBJk+Vl7d9nCxAAnb3W4EI3Yf3XY8ht3VnKEQ==", 2024-03-04T09:29:43.384 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:29:43.384 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-04T09:29:43.384 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-04T09:29:43.384 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-04T09:29:43.384 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:29:43.384 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:29:43.384 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:29:43.384 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.3", 2024-03-04T09:29:43.385 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQBMk+Vlv+Y1NBAAiRYSUqov2i2sj3iInfpHvA==", 2024-03-04T09:29:43.385 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:29:43.385 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-04T09:29:43.385 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-04T09:29:43.385 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-04T09:29:43.385 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:29:43.385 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:29:43.385 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:29:43.385 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.4", 2024-03-04T09:29:43.385 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQBUk+Vl1OlAARAAxMV4GUGS+RJ6XlrRhrVKcg==", 2024-03-04T09:29:43.385 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:29:43.386 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-04T09:29:43.386 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-04T09:29:43.386 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-04T09:29:43.386 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:29:43.386 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:29:43.386 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:29:43.386 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.5", 2024-03-04T09:29:43.386 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQBXk+Vl1lKLGhAAHlsiRr+1+OSRWhkB+weIJg==", 2024-03-04T09:29:43.386 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:29:43.386 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-04T09:29:43.387 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-04T09:29:43.387 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-04T09:29:43.387 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:29:43.387 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:29:43.387 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:29:43.387 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.6", 2024-03-04T09:29:43.387 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQBak+VlwB+GJBAA3f9KUz+zZlQuUR/ywQfkDA==", 2024-03-04T09:29:43.387 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:29:43.387 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-04T09:29:43.387 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-04T09:29:43.388 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-04T09:29:43.388 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:29:43.388 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:29:43.388 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:29:43.388 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.7", 2024-03-04T09:29:43.388 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQBdk+Vle4o8LhAAko+iH5VqqCsfWkNfKZjGVA==", 2024-03-04T09:29:43.388 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:29:43.388 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-04T09:29:43.388 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-04T09:29:43.389 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-04T09:29:43.389 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:29:43.389 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:29:43.389 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:29:43.389 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "client.0", 2024-03-04T09:29:43.389 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQA9k+Vl4L+rJxAACaDbQ4/Bybp5hAZAnJzlVw==", 2024-03-04T09:29:43.389 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:29:43.389 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-04T09:29:43.389 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow r", 2024-03-04T09:29:43.389 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-04T09:29:43.389 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:29:43.389 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:29:43.389 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:29:43.390 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "client.1", 2024-03-04T09:29:43.390 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQA+k+VlGjkOAhAAJNzB8+tQl3mqCnTgMn9EVw==", 2024-03-04T09:29:43.390 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:29:43.390 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-04T09:29:43.390 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow r", 2024-03-04T09:29:43.390 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow rw", 2024-03-04T09:29:43.390 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow rwx" 2024-03-04T09:29:43.390 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:29:43.390 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:29:43.390 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:29:43.390 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "client.2", 2024-03-04T09:29:43.390 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQA9k+VlUc8DMxAA8pGUtIh+BO6UKrrYnCqVgQ==", 2024-03-04T09:29:43.390 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:29:43.391 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-04T09:29:43.391 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow r", 2024-03-04T09:29:43.391 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow rw", 2024-03-04T09:29:43.391 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow rwx" 2024-03-04T09:29:43.391 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:29:43.391 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:29:43.391 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:29:43.391 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "client.3", 2024-03-04T09:29:43.391 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQA+k+VlzMFKDBAAGr713I2hcmGMuHMIjNknYA==", 2024-03-04T09:29:43.391 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:29:43.391 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-04T09:29:43.391 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow r", 2024-03-04T09:29:43.391 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow rw", 2024-03-04T09:29:43.392 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow rwx" 2024-03-04T09:29:43.392 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:29:43.392 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:29:43.392 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:29:43.392 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "client.admin", 2024-03-04T09:29:43.392 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQA7k+Vllb7NFhAAmnFaNFKBcEri+iJAPnG5Gg==", 2024-03-04T09:29:43.392 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:29:43.392 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow *", 2024-03-04T09:29:43.392 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow *", 2024-03-04T09:29:43.392 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow *", 2024-03-04T09:29:43.392 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-04T09:29:43.392 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:29:43.392 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:29:43.392 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:29:43.393 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "mgr.x", 2024-03-04T09:29:43.393 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQA8k+VlPFatFBAA/o3Ps/uJrcg1738HOGy/FA==", 2024-03-04T09:29:43.393 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:29:43.393 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow *", 2024-03-04T09:29:43.393 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile mgr", 2024-03-04T09:29:43.393 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-04T09:29:43.393 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:29:43.393 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:29:43.393 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:29:43.393 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "mgr.y", 2024-03-04T09:29:43.393 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQA8k+VlEA2ACBAAXofDj3gmqItVTJ1pYLqZyQ==", 2024-03-04T09:29:43.393 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:29:43.393 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow *", 2024-03-04T09:29:43.393 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile mgr", 2024-03-04T09:29:43.394 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-04T09:29:43.394 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:29:43.394 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:29:43.394 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-04T09:29:43.394 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-04T09:29:43.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.381+0000 7f50cf200640 1 -- 172.21.15.6:0/2187755062 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f50b006b4d0 msgr2=0x7f50b006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:43.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.381+0000 7f50cf200640 1 --2- 172.21.15.6:0/2187755062 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f50b006b4d0 0x7f50b006d990 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f50bc036000 tx=0x7f50bc030000 comp rx=0 tx=0).stop 2024-03-04T09:29:43.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.382+0000 7f50cf200640 1 -- 172.21.15.6:0/2187755062 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f50d00cf460 msgr2=0x7f50d00cc900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:43.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.382+0000 7f50cf200640 1 --2- 172.21.15.6:0/2187755062 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f50d00cf460 0x7f50d00cc900 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f50c800cb70 tx=0x7f50c800a430 comp rx=0 tx=0).stop 2024-03-04T09:29:43.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.382+0000 7f50cf200640 1 -- 172.21.15.6:0/2187755062 shutdown_connections 2024-03-04T09:29:43.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.382+0000 7f50cf200640 1 --2- 172.21.15.6:0/2187755062 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f50b006b4d0 0x7f50b006d990 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:43.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.382+0000 7f50cf200640 1 --2- 172.21.15.6:0/2187755062 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f50d015e2a0 0x7f50d00cae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:43.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.382+0000 7f50cf200640 1 --2- 172.21.15.6:0/2187755062 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f50d015dec0 0x7f50d014b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:43.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.382+0000 7f50cf200640 1 --2- 172.21.15.6:0/2187755062 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f50d00cf460 0x7f50d00cc900 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:43.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.382+0000 7f50cf200640 1 -- 172.21.15.6:0/2187755062 >> 172.21.15.6:0/2187755062 conn(0x7f50d00d3d40 msgr2=0x7f50d015e680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:29:43.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.382+0000 7f50cf200640 1 -- 172.21.15.6:0/2187755062 shutdown_connections 2024-03-04T09:29:43.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.382+0000 7f50cf200640 1 -- 172.21.15.6:0/2187755062 wait complete. 2024-03-04T09:29:43.401 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T09:29:43.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.631+0000 7f547f200640 1 Processor -- start 2024-03-04T09:29:43.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.631+0000 7f547f200640 1 -- start start 2024-03-04T09:29:43.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.633+0000 7f547f200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f548015e070 0x7f54800d2960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:43.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.633+0000 7f547f200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54800d18c0 con 0x7f54800cf410 2024-03-04T09:29:43.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.633+0000 7f547f200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54800d1ba0 con 0x7f54800d0cd0 2024-03-04T09:29:43.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.633+0000 7f547f200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54800d1e80 con 0x7f548015e070 2024-03-04T09:29:43.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.633+0000 7f5475400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f54800cf410 0x7f54800d05c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:59450/0 (socket says 172.21.15.6:59450) 2024-03-04T09:29:43.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.633+0000 7f5475400640 1 -- 172.21.15.6:0/1438021709 learned_addr learned my addr 172.21.15.6:0/1438021709 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:29:43.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.634+0000 7f547de00640 1 --2- 172.21.15.6:0/1438021709 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f548015e070 0x7f54800d2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:43.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.634+0000 7f547d400640 1 -- 172.21.15.6:0/1438021709 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 281116144 0 0) 0x7f54800d18c0 con 0x7f54800cf410 2024-03-04T09:29:43.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.634+0000 7f547d400640 1 -- 172.21.15.6:0/1438021709 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5464003520 con 0x7f54800cf410 2024-03-04T09:29:43.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.634+0000 7f547d400640 1 -- 172.21.15.6:0/1438021709 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2201999647 0 0) 0x7f5464003520 con 0x7f54800cf410 2024-03-04T09:29:43.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.634+0000 7f547d400640 1 -- 172.21.15.6:0/1438021709 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f548015e070 msgr2=0x7f54800d2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:43.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.634+0000 7f547d400640 1 --2- 172.21.15.6:0/1438021709 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f548015e070 0x7f54800d2960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:43.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.634+0000 7f547d400640 1 -- 172.21.15.6:0/1438021709 >> v1:172.21.15.146:6789/0 conn(0x7f54800d0cd0 legacy=0x7f54800b3630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:43.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.634+0000 7f547d400640 1 -- 172.21.15.6:0/1438021709 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54800d2160 con 0x7f54800cf410 2024-03-04T09:29:43.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.634+0000 7f547d400640 1 -- 172.21.15.6:0/1438021709 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7f5468001670 con 0x7f54800cf410 2024-03-04T09:29:43.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.634+0000 7f547d400640 1 -- 172.21.15.6:0/1438021709 >> v1:172.21.15.6:6789/0 conn(0x7f54800cf410 legacy=0x7f54800d05c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:43.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.634+0000 7f547d400640 1 --2- 172.21.15.6:0/1438021709 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54640039e0 0x7f5464003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:43.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.634+0000 7f547d400640 1 --2- 172.21.15.6:0/1438021709 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f548015e070 0x7f5464004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:43.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.634+0000 7f547d400640 1 --2- 172.21.15.6:0/1438021709 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5464004d40 0x7f54640051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:43.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.634+0000 7f547d400640 1 -- 172.21.15.6:0/1438021709 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54800d1e80 con 0x7f54640039e0 2024-03-04T09:29:43.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.634+0000 7f547d400640 1 -- 172.21.15.6:0/1438021709 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f546400ba70 con 0x7f5464004d40 2024-03-04T09:29:43.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.634+0000 7f547d400640 1 -- 172.21.15.6:0/1438021709 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f546400bc70 con 0x7f548015e070 2024-03-04T09:29:43.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.634+0000 7f547de00640 1 --2- 172.21.15.6:0/1438021709 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54640039e0 0x7f5464003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:43.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.635+0000 7f547de00640 1 -- 172.21.15.6:0/1438021709 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f548015e070 msgr2=0x7f5464004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:43.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.635+0000 7f547de00640 1 --2- 172.21.15.6:0/1438021709 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f548015e070 0x7f5464004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:43.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.635+0000 7f547de00640 1 -- 172.21.15.6:0/1438021709 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5464004d40 msgr2=0x7f54640051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:43.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.635+0000 7f547de00640 1 --2- 172.21.15.6:0/1438021709 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5464004d40 0x7f54640051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:43.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.635+0000 7f547de00640 1 -- 172.21.15.6:0/1438021709 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5464003520 con 0x7f54640039e0 2024-03-04T09:29:43.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.636+0000 7f547de00640 1 --2- 172.21.15.6:0/1438021709 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54640039e0 0x7f5464003e00 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f5470009a90 tx=0x7f5464005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74c7bc58771f7b1d server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:43.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.636+0000 7f547d400640 1 -- 172.21.15.6:0/1438021709 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5470039070 con 0x7f54640039e0 2024-03-04T09:29:43.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.636+0000 7f547d400640 1 -- 172.21.15.6:0/1438021709 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f547002c6d0 con 0x7f54640039e0 2024-03-04T09:29:43.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.636+0000 7f547d400640 1 -- 172.21.15.6:0/1438021709 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5470009690 con 0x7f54640039e0 2024-03-04T09:29:43.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.637+0000 7f547f200640 1 -- 172.21.15.6:0/1438021709 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54640039e0 msgr2=0x7f5464003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:43.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.637+0000 7f547f200640 1 --2- 172.21.15.6:0/1438021709 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54640039e0 0x7f5464003e00 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f5470009a90 tx=0x7f5464005830 comp rx=0 tx=0).stop 2024-03-04T09:29:43.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.637+0000 7f547e800640 1 -- 172.21.15.6:0/1438021709 reap_dead start 2024-03-04T09:29:43.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.637+0000 7f547f200640 1 -- 172.21.15.6:0/1438021709 shutdown_connections 2024-03-04T09:29:43.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.637+0000 7f547f200640 1 -- 172.21.15.6:0/1438021709 >> 172.21.15.6:0/1438021709 conn(0x7f548008d2a0 msgr2=0x7f54800d3cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:29:43.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.637+0000 7f547f200640 1 -- 172.21.15.6:0/1438021709 shutdown_connections 2024-03-04T09:29:43.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.637+0000 7f547f200640 1 -- 172.21.15.6:0/1438021709 wait complete. 2024-03-04T09:29:43.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.639+0000 7f547f200640 1 Processor -- start 2024-03-04T09:29:43.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.639+0000 7f547f200640 1 -- start start 2024-03-04T09:29:43.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.640+0000 7f547f200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f54640039e0 0x7f5480149650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:43.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.640+0000 7f547f200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f54801f87d0 0x7f5480160c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:43.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.640+0000 7f547f200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54800c77e0 0x7f548015b740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:43.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.640+0000 7f547f200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54800d4730 con 0x7f54800c77e0 2024-03-04T09:29:43.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.640+0000 7f547f200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54800d45b0 con 0x7f54801f87d0 2024-03-04T09:29:43.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.640+0000 7f547f200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54800cf870 con 0x7f54640039e0 2024-03-04T09:29:43.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.640+0000 7f547e800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54800c77e0 0x7f548015b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:43.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.640+0000 7f547e800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54800c77e0 0x7f548015b740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38836/0 (socket says 172.21.15.6:38836) 2024-03-04T09:29:43.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.640+0000 7f547e800640 1 -- 172.21.15.6:0/2569129382 learned_addr learned my addr 172.21.15.6:0/2569129382 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:29:43.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.641+0000 7f547de00640 1 --2- 172.21.15.6:0/2569129382 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f54640039e0 0x7f5480149650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:43.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.641+0000 7f5475400640 1 --2- 172.21.15.6:0/2569129382 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f54801f87d0 0x7f5480160c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:43.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.641+0000 7f547e800640 1 -- 172.21.15.6:0/2569129382 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f54640039e0 msgr2=0x7f5480149650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:43.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.641+0000 7f547e800640 1 --2- 172.21.15.6:0/2569129382 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f54640039e0 0x7f5480149650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:43.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.641+0000 7f547e800640 1 -- 172.21.15.6:0/2569129382 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f54801f87d0 msgr2=0x7f5480160c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:43.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.641+0000 7f547e800640 1 --2- 172.21.15.6:0/2569129382 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f54801f87d0 0x7f5480160c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:43.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.641+0000 7f547e800640 1 -- 172.21.15.6:0/2569129382 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54800c7d80 con 0x7f54800c77e0 2024-03-04T09:29:43.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.641+0000 7f547e800640 1 --2- 172.21.15.6:0/2569129382 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54800c77e0 0x7f548015b740 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f548008d270 tx=0x7f547800cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:43.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.642+0000 7f5476a00640 1 -- 172.21.15.6:0/2569129382 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5478017070 con 0x7f54800c77e0 2024-03-04T09:29:43.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.642+0000 7f5476a00640 1 -- 172.21.15.6:0/2569129382 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5478005d60 con 0x7f54800c77e0 2024-03-04T09:29:43.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.642+0000 7f547f200640 1 -- 172.21.15.6:0/2569129382 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54801f7cf0 con 0x7f54800c77e0 2024-03-04T09:29:43.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.642+0000 7f5476a00640 1 -- 172.21.15.6:0/2569129382 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f547800a850 con 0x7f54800c77e0 2024-03-04T09:29:43.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.642+0000 7f547f200640 1 -- 172.21.15.6:0/2569129382 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5480149b90 con 0x7f54800c77e0 2024-03-04T09:29:43.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.643+0000 7f547f200640 1 -- 172.21.15.6:0/2569129382 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54801f7ae0 con 0x7f54800c77e0 2024-03-04T09:29:43.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.644+0000 7f5476a00640 1 -- 172.21.15.6:0/2569129382 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f547801a070 con 0x7f54800c77e0 2024-03-04T09:29:43.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.645+0000 7f5476a00640 1 --2- 172.21.15.6:0/2569129382 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f545c06b500 0x7f545c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:43.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.645+0000 7f5476a00640 1 -- 172.21.15.6:0/2569129382 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f54780892d0 con 0x7f54800c77e0 2024-03-04T09:29:43.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.646+0000 7f547de00640 1 --2- 172.21.15.6:0/2569129382 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f545c06b500 0x7f545c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:43.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.647+0000 7f547de00640 1 --2- 172.21.15.6:0/2569129382 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f545c06b500 0x7f545c06d9c0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f5470036000 tx=0x7f5470030000 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:43.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.649+0000 7f5476a00640 1 -- 172.21.15.6:0/2569129382 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5478058400 con 0x7f54800c77e0 2024-03-04T09:29:43.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.849+0000 7f547f200640 1 -- 172.21.15.6:0/2569129382 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f548004f330 con 0x7f54800c77e0 2024-03-04T09:29:43.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.850+0000 7f5476a00640 1 -- 172.21.15.6:0/2569129382 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v77) v1 ==== 74+0+12844 (secure 0 0 0) 0x7f5478057d00 con 0x7f54800c77e0 2024-03-04T09:29:43.849 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:29:43.849 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":77,"fsid":"edce0578-5ae4-4402-b45a-552bdef84d7c","created":"2024-03-04T09:24:54.523780+0000","modified":"2024-03-04T09:29:42.849533+0000","last_up_change":"2024-03-04T09:25:07.089049+0000","last_in_change":"2024-03-04T09:24:57.982077+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T09:25:07.935198+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a96574f-81f1-4bc6-8463-757520b84a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6825","nonce":1364449401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6827","nonce":1364449401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6831","nonce":1364449401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6829","nonce":1364449401}]},"public_addr":"172.21.15.6:6825/1364449401","cluster_addr":"172.21.15.6:6827/1364449401","heartbeat_back_addr":"172.21.15.6:6831/1364449401","heartbeat_front_addr":"172.21.15.6:6829/1364449401","state":["exists","up"]},{"osd":1,"uuid":"7d79b9c1-6b87-476a-bc36-7db5ee64cd90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6809","nonce":2494103078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6811","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6813","nonce":2494103078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6821","nonce":2494103078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6817","nonce":2494103078}]},"public_addr":"172.21.15.6:6809/2494103078","cluster_addr":"172.21.15.6:6813/2494103078","heartbeat_back_addr":"172.21.15.6:6821/2494103078","heartbeat_front_addr":"172.21.15.6:6817/2494103078","state":["exists","up"]},{"osd":2,"uuid":"492331b9-4ee9-40b6-bacd-85d44cd6c45a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6812","nonce":1500066681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6816","nonce":1500066681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6824","nonce":1500066681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6820","nonce":1500066681}]},"public_addr":"172.21.15.6:6812/1500066681","cluster_addr":"172.21.15.6:6816/1500066681","heartbeat_back_addr":"172.21.15.6:6824/1500066681","heartbeat_front_addr":"172.21.15.6:6820/1500066681","state":["exists","up"]},{"osd":3,"uuid":"21bb2efe-9ba7-4bdc-bc31-180cd1102e21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6801","nonce":1558806123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6803","nonce":1558806123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6807","nonce":1558806123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6805","nonce":1558806123}]},"public_addr":"172.21.15.6:6801/1558806123","cluster_addr":"172.21.15.6:6803/1558806123","heartbeat_back_addr":"172.21.15.6:6807/1558806123","heartbeat_front_addr":"172.21.15.6:6805/1558806123","state":["exists","up"]},{"osd":4,"uuid":"6e2c4b71-19d9-4737-83d9-558705d8644a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6801","nonce":1526135780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6803","nonce":1526135780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6807","nonce":1526135780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6805","nonce":1526135780}]},"public_addr":"172.21.15.146:6801/1526135780","cluster_addr":"172.21.15.146:6803/1526135780","heartbeat_back_addr":"172.21.15.146:6807/1526135780","heartbeat_front_addr":"172.21.15.146:6805/1526135780","state":["exists","up"]},{"osd":5,"uuid":"fb69c525-f1be-4b7d-ab32-6c29b0d6fddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6809","nonce":460262507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6811","nonce":460262507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6815","nonce":460262507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6813","nonce":460262507}]},"public_addr":"172.21.15.146:6809/460262507","cluster_addr":"172.21.15.146:6811/460262507","heartbeat_back_addr":"172.21.15.146:6815/460262507","heartbeat_front_addr":"172.21.15.146:6813/460262507","state":["exists","up"]},{"osd":6,"uuid":"c32e7fed-370d-4401-8ce0-6a4335b0d168","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6823","nonce":1392897493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6827","nonce":1392897493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6831","nonce":1392897493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6829","nonce":1392897493}]},"public_addr":"172.21.15.146:6823/1392897493","cluster_addr":"172.21.15.146:6827/1392897493","heartbeat_back_addr":"172.21.15.146:6831/1392897493","heartbeat_front_addr":"172.21.15.146:6829/1392897493","state":["exists","up"]},{"osd":7,"uuid":"75a79b44-fa5f-426d-a7d3-71f21f78835c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6817","nonce":3268323156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6819","nonce":3268323156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6825","nonce":3268323156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6822","nonce":3268323156}]},"public_addr":"172.21.15.146:6817/3268323156","cluster_addr":"172.21.15.146:6819/3268323156","heartbeat_back_addr":"172.21.15.146:6825/3268323156","heartbeat_front_addr":"172.21.15.146:6822/3268323156","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.310437+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.295850+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.294674+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.526855+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.300023+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.595160+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.346046+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.558741+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T09:29:43.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.853+0000 7f547f200640 1 -- 172.21.15.6:0/2569129382 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f545c06b500 msgr2=0x7f545c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:43.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.853+0000 7f547f200640 1 --2- 172.21.15.6:0/2569129382 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f545c06b500 0x7f545c06d9c0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f5470036000 tx=0x7f5470030000 comp rx=0 tx=0).stop 2024-03-04T09:29:43.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.853+0000 7f547f200640 1 -- 172.21.15.6:0/2569129382 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54800c77e0 msgr2=0x7f548015b740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:43.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.853+0000 7f547f200640 1 --2- 172.21.15.6:0/2569129382 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54800c77e0 0x7f548015b740 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f548008d270 tx=0x7f547800cb90 comp rx=0 tx=0).stop 2024-03-04T09:29:43.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.853+0000 7f547f200640 1 -- 172.21.15.6:0/2569129382 shutdown_connections 2024-03-04T09:29:43.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.853+0000 7f547f200640 1 --2- 172.21.15.6:0/2569129382 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54800c77e0 0x7f548015b740 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:43.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.853+0000 7f547f200640 1 --2- 172.21.15.6:0/2569129382 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f54801f87d0 0x7f5480160c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:43.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.853+0000 7f547f200640 1 --2- 172.21.15.6:0/2569129382 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f545c06b500 0x7f545c06d9c0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:43.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.853+0000 7f547f200640 1 --2- 172.21.15.6:0/2569129382 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f54640039e0 0x7f5480149650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:43.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.853+0000 7f547f200640 1 -- 172.21.15.6:0/2569129382 >> 172.21.15.6:0/2569129382 conn(0x7f548008d2a0 msgr2=0x7f54800b3160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:29:43.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.853+0000 7f547f200640 1 -- 172.21.15.6:0/2569129382 shutdown_connections 2024-03-04T09:29:43.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:43.853+0000 7f547f200640 1 -- 172.21.15.6:0/2569129382 wait complete. 2024-03-04T09:29:43.868 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T09:25:07.935198+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-04T09:29:43.869 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-04T09:29:44.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.080+0000 7effc6000640 1 Processor -- start 2024-03-04T09:29:44.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.080+0000 7effc6000640 1 -- start start 2024-03-04T09:29:44.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.081+0000 7effc6000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7effc00d29e0 0x7effc00b4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:44.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.081+0000 7effc6000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effc00b6be0 con 0x7effc015df70 2024-03-04T09:29:44.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.081+0000 7effc6000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effc00b6ec0 con 0x7effc015e350 2024-03-04T09:29:44.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.081+0000 7effc6000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effc00b71a0 con 0x7effc00d29e0 2024-03-04T09:29:44.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.082+0000 7effc4c00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7effc015df70 0x7effc00cf3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:59460/0 (socket says 172.21.15.6:59460) 2024-03-04T09:29:44.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.082+0000 7effc4c00640 1 -- 172.21.15.6:0/4263640599 learned_addr learned my addr 172.21.15.6:0/4263640599 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:29:44.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.082+0000 7effbf400640 1 -- 172.21.15.6:0/4263640599 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1449645031 0 0) 0x7effc00b6be0 con 0x7effc015df70 2024-03-04T09:29:44.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.082+0000 7effbf400640 1 -- 172.21.15.6:0/4263640599 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7effa4003520 con 0x7effc015df70 2024-03-04T09:29:44.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.082+0000 7effc5600640 1 --2- 172.21.15.6:0/4263640599 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7effc00d29e0 0x7effc00b4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:44.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.082+0000 7effbf400640 1 -- 172.21.15.6:0/4263640599 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2708363057 0 0) 0x7effa4003520 con 0x7effc015df70 2024-03-04T09:29:44.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.082+0000 7effbf400640 1 -- 172.21.15.6:0/4263640599 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7effc00d29e0 msgr2=0x7effc00b4900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:44.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.082+0000 7effbf400640 1 --2- 172.21.15.6:0/4263640599 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7effc00d29e0 0x7effc00b4900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:44.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.082+0000 7effbf400640 1 -- 172.21.15.6:0/4263640599 >> v1:172.21.15.146:6789/0 conn(0x7effc015e350 legacy=0x7effc00d22d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:44.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.083+0000 7effbf400640 1 -- 172.21.15.6:0/4263640599 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effc00b7480 con 0x7effc015df70 2024-03-04T09:29:44.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.083+0000 7effbf400640 1 -- 172.21.15.6:0/4263640599 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7effb0001670 con 0x7effc015df70 2024-03-04T09:29:44.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.083+0000 7effbf400640 1 -- 172.21.15.6:0/4263640599 >> v1:172.21.15.6:6789/0 conn(0x7effc015df70 legacy=0x7effc00cf3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:44.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.083+0000 7effbf400640 1 --2- 172.21.15.6:0/4263640599 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7effa40039e0 0x7effa4003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:44.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.083+0000 7effbf400640 1 --2- 172.21.15.6:0/4263640599 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7effa4004800 0x7effa4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:44.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.083+0000 7effbf400640 1 --2- 172.21.15.6:0/4263640599 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7effc00d29e0 0x7effa4005270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:44.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.083+0000 7effbf400640 1 -- 172.21.15.6:0/4263640599 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effc00b71a0 con 0x7effa40039e0 2024-03-04T09:29:44.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.083+0000 7effbf400640 1 -- 172.21.15.6:0/4263640599 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effa400bbe0 con 0x7effa4004800 2024-03-04T09:29:44.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.083+0000 7effbf400640 1 -- 172.21.15.6:0/4263640599 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effa400bde0 con 0x7effc00d29e0 2024-03-04T09:29:44.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.083+0000 7effc4c00640 1 --2- 172.21.15.6:0/4263640599 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7effa40039e0 0x7effa4003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:44.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.083+0000 7effbfe00640 1 --2- 172.21.15.6:0/4263640599 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7effa4004800 0x7effa4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:44.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.083+0000 7effc5600640 1 --2- 172.21.15.6:0/4263640599 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7effc00d29e0 0x7effa4005270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:44.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.083+0000 7effc4c00640 1 -- 172.21.15.6:0/4263640599 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7effc00d29e0 msgr2=0x7effa4005270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:44.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.083+0000 7effc4c00640 1 --2- 172.21.15.6:0/4263640599 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7effc00d29e0 0x7effa4005270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:44.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.083+0000 7effc4c00640 1 -- 172.21.15.6:0/4263640599 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7effa4004800 msgr2=0x7effa4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:44.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.083+0000 7effc4c00640 1 --2- 172.21.15.6:0/4263640599 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7effa4004800 0x7effa4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:44.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.083+0000 7effc4c00640 1 -- 172.21.15.6:0/4263640599 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effa4003520 con 0x7effa40039e0 2024-03-04T09:29:44.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.084+0000 7effc5600640 1 --2- 172.21.15.6:0/4263640599 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7effc00d29e0 0x7effa4005270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:29:44.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.085+0000 7effc4c00640 1 --2- 172.21.15.6:0/4263640599 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7effa40039e0 0x7effa4003de0 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7effb000c110 tx=0x7effa4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9857eb8d072cc80 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:44.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.085+0000 7effbf400640 1 -- 172.21.15.6:0/4263640599 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effb0007c40 con 0x7effa40039e0 2024-03-04T09:29:44.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.085+0000 7effbf400640 1 -- 172.21.15.6:0/4263640599 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effb0030070 con 0x7effa40039e0 2024-03-04T09:29:44.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.085+0000 7effbf400640 1 -- 172.21.15.6:0/4263640599 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effb0005d30 con 0x7effa40039e0 2024-03-04T09:29:44.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.085+0000 7effc6000640 1 -- 172.21.15.6:0/4263640599 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7effa40039e0 msgr2=0x7effa4003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:44.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.085+0000 7effc6000640 1 --2- 172.21.15.6:0/4263640599 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7effa40039e0 0x7effa4003de0 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7effb000c110 tx=0x7effa4005830 comp rx=0 tx=0).stop 2024-03-04T09:29:44.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.085+0000 7effc5600640 1 -- 172.21.15.6:0/4263640599 reap_dead start 2024-03-04T09:29:44.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.086+0000 7effc6000640 1 -- 172.21.15.6:0/4263640599 shutdown_connections 2024-03-04T09:29:44.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.086+0000 7effc6000640 1 -- 172.21.15.6:0/4263640599 >> 172.21.15.6:0/4263640599 conn(0x7effc008d2a0 msgr2=0x7effc00d3cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:29:44.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.086+0000 7effc6000640 1 -- 172.21.15.6:0/4263640599 shutdown_connections 2024-03-04T09:29:44.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.086+0000 7effc6000640 1 -- 172.21.15.6:0/4263640599 wait complete. 2024-03-04T09:29:44.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.087+0000 7effc6000640 1 Processor -- start 2024-03-04T09:29:44.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.088+0000 7effc6000640 1 -- start start 2024-03-04T09:29:44.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.089+0000 7effc6000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7effa40039e0 0x7effc01533a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:44.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.089+0000 7effc6000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7effc0153b50 0x7effc00c8b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:44.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.089+0000 7effc6000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7effc0158c70 0x7effc01590f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:44.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.089+0000 7effc6000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effc00cfab0 con 0x7effc0158c70 2024-03-04T09:29:44.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.089+0000 7effc6000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effc00d1d60 con 0x7effa40039e0 2024-03-04T09:29:44.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.089+0000 7effc6000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effc00cfc30 con 0x7effc0153b50 2024-03-04T09:29:44.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.089+0000 7effc5600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7effc0158c70 0x7effc01590f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:44.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.089+0000 7effc5600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7effc0158c70 0x7effc01590f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38846/0 (socket says 172.21.15.6:38846) 2024-03-04T09:29:44.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.089+0000 7effc5600640 1 -- 172.21.15.6:0/3095671365 learned_addr learned my addr 172.21.15.6:0/3095671365 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:29:44.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.089+0000 7effc4c00640 1 --2- 172.21.15.6:0/3095671365 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7effa40039e0 0x7effc01533a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:44.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.089+0000 7effbfe00640 1 --2- 172.21.15.6:0/3095671365 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7effc0153b50 0x7effc00c8b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:44.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.089+0000 7effc5600640 1 -- 172.21.15.6:0/3095671365 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7effc0153b50 msgr2=0x7effc00c8b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:44.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.089+0000 7effc5600640 1 --2- 172.21.15.6:0/3095671365 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7effc0153b50 0x7effc00c8b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:44.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.089+0000 7effc5600640 1 -- 172.21.15.6:0/3095671365 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7effa40039e0 msgr2=0x7effc01533a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:44.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.089+0000 7effc5600640 1 --2- 172.21.15.6:0/3095671365 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7effa40039e0 0x7effc01533a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:44.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.089+0000 7effc5600640 1 -- 172.21.15.6:0/3095671365 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effc00ca0d0 con 0x7effc0158c70 2024-03-04T09:29:44.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.090+0000 7effc5600640 1 --2- 172.21.15.6:0/3095671365 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7effc0158c70 0x7effc01590f0 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7effc00ce820 tx=0x7effb8009d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:44.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.090+0000 7effbcc00640 1 -- 172.21.15.6:0/3095671365 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effb8019070 con 0x7effc0158c70 2024-03-04T09:29:44.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.090+0000 7effbcc00640 1 -- 172.21.15.6:0/3095671365 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effb8007ce0 con 0x7effc0158c70 2024-03-04T09:29:44.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.090+0000 7effbcc00640 1 -- 172.21.15.6:0/3095671365 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effb8014640 con 0x7effc0158c70 2024-03-04T09:29:44.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.090+0000 7effc6000640 1 -- 172.21.15.6:0/3095671365 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effc00ca3b0 con 0x7effc0158c70 2024-03-04T09:29:44.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.090+0000 7effc6000640 1 -- 172.21.15.6:0/3095671365 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effc00c8640 con 0x7effc0158c70 2024-03-04T09:29:44.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.091+0000 7effc6000640 1 -- 172.21.15.6:0/3095671365 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effc00c8840 con 0x7effc0158c70 2024-03-04T09:29:44.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.092+0000 7effbcc00640 1 -- 172.21.15.6:0/3095671365 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7effb801c070 con 0x7effc0158c70 2024-03-04T09:29:44.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.093+0000 7effbcc00640 1 --2- 172.21.15.6:0/3095671365 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7eff9c06b500 0x7eff9c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:44.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.093+0000 7effbcc00640 1 -- 172.21.15.6:0/3095671365 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 4857+0+0 (secure 0 0 0) 0x7effb80898a0 con 0x7effc0158c70 2024-03-04T09:29:44.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.094+0000 7effc4c00640 1 --2- 172.21.15.6:0/3095671365 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7eff9c06b500 0x7eff9c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:44.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.095+0000 7effc4c00640 1 --2- 172.21.15.6:0/3095671365 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7eff9c06b500 0x7eff9c06d9c0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7effb0034000 tx=0x7effb0007000 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:44.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.098+0000 7effbcc00640 1 -- 172.21.15.6:0/3095671365 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7effc00c8840 con 0x7effc0158c70 2024-03-04T09:29:44.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.299+0000 7effc6000640 1 -- 172.21.15.6:0/3095671365 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7effc00d20b0 con 0x7effc0158c70 2024-03-04T09:29:44.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.300+0000 7effbcc00640 1 -- 172.21.15.6:0/3095671365 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v77) v1 ==== 93+0+10 (secure 0 0 0) 0x7effb8058ae0 con 0x7effc0158c70 2024-03-04T09:29:44.298 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-04T09:29:44.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.303+0000 7effc6000640 1 -- 172.21.15.6:0/3095671365 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7eff9c06b500 msgr2=0x7eff9c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:44.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.303+0000 7effc6000640 1 --2- 172.21.15.6:0/3095671365 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7eff9c06b500 0x7eff9c06d9c0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7effb0034000 tx=0x7effb0007000 comp rx=0 tx=0).stop 2024-03-04T09:29:44.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.303+0000 7effc6000640 1 -- 172.21.15.6:0/3095671365 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7effc0158c70 msgr2=0x7effc01590f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:44.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.303+0000 7effc6000640 1 --2- 172.21.15.6:0/3095671365 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7effc0158c70 0x7effc01590f0 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7effc00ce820 tx=0x7effb8009d30 comp rx=0 tx=0).stop 2024-03-04T09:29:44.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.303+0000 7effc6000640 1 -- 172.21.15.6:0/3095671365 shutdown_connections 2024-03-04T09:29:44.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.303+0000 7effc6000640 1 --2- 172.21.15.6:0/3095671365 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7effc0158c70 0x7effc01590f0 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:44.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.303+0000 7effc6000640 1 --2- 172.21.15.6:0/3095671365 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7eff9c06b500 0x7eff9c06d9c0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:44.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.303+0000 7effc6000640 1 --2- 172.21.15.6:0/3095671365 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7effc0153b50 0x7effc00c8b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:44.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.303+0000 7effc6000640 1 --2- 172.21.15.6:0/3095671365 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7effa40039e0 0x7effc01533a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:44.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.303+0000 7effc6000640 1 -- 172.21.15.6:0/3095671365 >> 172.21.15.6:0/3095671365 conn(0x7effc008d2a0 msgr2=0x7effc00d0da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:29:44.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.303+0000 7effc6000640 1 -- 172.21.15.6:0/3095671365 shutdown_connections 2024-03-04T09:29:44.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.303+0000 7effc6000640 1 -- 172.21.15.6:0/3095671365 wait complete. 2024-03-04T09:29:44.318 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-04T09:29:44.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.533+0000 7f3009800640 1 Processor -- start 2024-03-04T09:29:44.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.533+0000 7f3009800640 1 -- start start 2024-03-04T09:29:44.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.534+0000 7f3009800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3004161b40 0x7f30040cfd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:44.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.534+0000 7f3009800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f30040b5060 0x7f30040d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:44.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.534+0000 7f3009800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f30040b54a0 0x7f30040d2760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:44.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.534+0000 7f3009800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30040cf900 con 0x7f30040b5060 2024-03-04T09:29:44.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.534+0000 7f3009800640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30040cea50 con 0x7f30040b54a0 2024-03-04T09:29:44.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.534+0000 7f3009800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30040cecf0 con 0x7f3004161b40 2024-03-04T09:29:44.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.535+0000 7f3002000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f30040b5060 0x7f30040d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:44.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.535+0000 7f3002000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f30040b5060 0x7f30040d2220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38850/0 (socket says 172.21.15.6:38850) 2024-03-04T09:29:44.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.535+0000 7f3002000640 1 -- 172.21.15.6:0/2405276186 learned_addr learned my addr 172.21.15.6:0/2405276186 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:29:44.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.535+0000 7f3003400640 1 --2- 172.21.15.6:0/2405276186 >> v2:172.21.15.146:3300/0 conn(0x7f30040b54a0 0x7f30040d2760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:44.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.535+0000 7f3002a00640 1 --2- 172.21.15.6:0/2405276186 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3004161b40 0x7f30040cfd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:44.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.535+0000 7f3002000640 1 -- 172.21.15.6:0/2405276186 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3004161b40 msgr2=0x7f30040cfd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:44.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.536+0000 7f3002000640 1 --2- 172.21.15.6:0/2405276186 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3004161b40 0x7f30040cfd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:44.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.536+0000 7f3002000640 1 -- 172.21.15.6:0/2405276186 >> v2:172.21.15.146:3300/0 conn(0x7f30040b54a0 msgr2=0x7f30040d2760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:44.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.536+0000 7f3002000640 1 --2- 172.21.15.6:0/2405276186 >> v2:172.21.15.146:3300/0 conn(0x7f30040b54a0 0x7f30040d2760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:44.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.536+0000 7f3002000640 1 -- 172.21.15.6:0/2405276186 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30040cefd0 con 0x7f30040b5060 2024-03-04T09:29:44.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.537+0000 7f3002000640 1 --2- 172.21.15.6:0/2405276186 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f30040b5060 0x7f30040d2220 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f300406b7f0 tx=0x7f2ff002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be2fc89eed7c2b3c server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:44.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.537+0000 7f3001600640 1 -- 172.21.15.6:0/2405276186 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ff0035020 con 0x7f30040b5060 2024-03-04T09:29:44.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.537+0000 7f3001600640 1 -- 172.21.15.6:0/2405276186 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ff0031870 con 0x7f30040b5060 2024-03-04T09:29:44.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.537+0000 7f3001600640 1 -- 172.21.15.6:0/2405276186 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ff0031c40 con 0x7f30040b5060 2024-03-04T09:29:44.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.538+0000 7f3009800640 1 -- 172.21.15.6:0/2405276186 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f30040b5060 msgr2=0x7f30040d2220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:44.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.538+0000 7f3009800640 1 --2- 172.21.15.6:0/2405276186 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f30040b5060 0x7f30040d2220 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f300406b7f0 tx=0x7f2ff002b910 comp rx=0 tx=0).stop 2024-03-04T09:29:44.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.538+0000 7f3009800640 1 -- 172.21.15.6:0/2405276186 shutdown_connections 2024-03-04T09:29:44.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.538+0000 7f3009800640 1 --2- 172.21.15.6:0/2405276186 >> v2:172.21.15.146:3300/0 conn(0x7f30040b54a0 0x7f30040d2760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:44.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.538+0000 7f3009800640 1 --2- 172.21.15.6:0/2405276186 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f30040b5060 0x7f30040d2220 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:44.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.538+0000 7f3009800640 1 --2- 172.21.15.6:0/2405276186 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3004161b40 0x7f30040cfd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:44.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.538+0000 7f3009800640 1 -- 172.21.15.6:0/2405276186 >> 172.21.15.6:0/2405276186 conn(0x7f300409d220 msgr2=0x7f30040d3ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:29:44.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.538+0000 7f3009800640 1 -- 172.21.15.6:0/2405276186 shutdown_connections 2024-03-04T09:29:44.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.538+0000 7f3009800640 1 -- 172.21.15.6:0/2405276186 wait complete. 2024-03-04T09:29:44.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.539+0000 7f3009800640 1 Processor -- start 2024-03-04T09:29:44.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.540+0000 7f3009800640 1 -- start start 2024-03-04T09:29:44.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.541+0000 7f3009800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f30040b5060 0x7f30040b80c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:44.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.541+0000 7f3009800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f30040b54a0 0x7f3004152220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:44.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.541+0000 7f3009800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3004161b40 0x7f30041f4110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:44.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.541+0000 7f3009800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30040b2ff0 con 0x7f3004161b40 2024-03-04T09:29:44.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.541+0000 7f3009800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30040cf590 con 0x7f30040b5060 2024-03-04T09:29:44.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.541+0000 7f3009800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30040b3170 con 0x7f30040b54a0 2024-03-04T09:29:44.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.541+0000 7f3003400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3004161b40 0x7f30041f4110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:44.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.541+0000 7f3003400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3004161b40 0x7f30041f4110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38856/0 (socket says 172.21.15.6:38856) 2024-03-04T09:29:44.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.541+0000 7f3003400640 1 -- 172.21.15.6:0/2103159647 learned_addr learned my addr 172.21.15.6:0/2103159647 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:29:44.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.541+0000 7f3002a00640 1 --2- 172.21.15.6:0/2103159647 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f30040b5060 0x7f30040b80c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:44.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.541+0000 7f3002000640 1 --2- 172.21.15.6:0/2103159647 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f30040b54a0 0x7f3004152220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:44.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.541+0000 7f3003400640 1 -- 172.21.15.6:0/2103159647 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f30040b54a0 msgr2=0x7f3004152220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:44.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.541+0000 7f3003400640 1 --2- 172.21.15.6:0/2103159647 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f30040b54a0 0x7f3004152220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:44.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.541+0000 7f3003400640 1 -- 172.21.15.6:0/2103159647 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f30040b5060 msgr2=0x7f30040b80c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:44.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.541+0000 7f3003400640 1 --2- 172.21.15.6:0/2103159647 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f30040b5060 0x7f30040b80c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:44.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.541+0000 7f3003400640 1 -- 172.21.15.6:0/2103159647 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30040cc8d0 con 0x7f3004161b40 2024-03-04T09:29:44.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.542+0000 7f3003400640 1 --2- 172.21.15.6:0/2103159647 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3004161b40 0x7f30041f4110 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f2ffc007d20 tx=0x7f2ffc00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:44.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.542+0000 7f2ffaa00640 1 -- 172.21.15.6:0/2103159647 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ffc014420 con 0x7f3004161b40 2024-03-04T09:29:44.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.542+0000 7f2ffaa00640 1 -- 172.21.15.6:0/2103159647 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ffc019070 con 0x7f3004161b40 2024-03-04T09:29:44.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.542+0000 7f2ffaa00640 1 -- 172.21.15.6:0/2103159647 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ffc014820 con 0x7f3004161b40 2024-03-04T09:29:44.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.542+0000 7f3009800640 1 -- 172.21.15.6:0/2103159647 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30040ca5c0 con 0x7f3004161b40 2024-03-04T09:29:44.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.542+0000 7f3009800640 1 -- 172.21.15.6:0/2103159647 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30040ca8a0 con 0x7f3004161b40 2024-03-04T09:29:44.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.543+0000 7f3009800640 1 -- 172.21.15.6:0/2103159647 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f300404f3a0 con 0x7f3004161b40 2024-03-04T09:29:44.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.544+0000 7f2ffaa00640 1 -- 172.21.15.6:0/2103159647 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2ffc0149c0 con 0x7f3004161b40 2024-03-04T09:29:44.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.545+0000 7f2ffaa00640 1 --2- 172.21.15.6:0/2103159647 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2fe406b4d0 0x7f2fe406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:44.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.545+0000 7f2ffaa00640 1 -- 172.21.15.6:0/2103159647 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f2ffc08a710 con 0x7f3004161b40 2024-03-04T09:29:44.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.546+0000 7f3002a00640 1 --2- 172.21.15.6:0/2103159647 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2fe406b4d0 0x7f2fe406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:44.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.547+0000 7f3002a00640 1 --2- 172.21.15.6:0/2103159647 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2fe406b4d0 0x7f2fe406d990 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f2ff4006fd0 tx=0x7f2ff4008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:44.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.550+0000 7f2ffaa00640 1 -- 172.21.15.6:0/2103159647 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2ffc059840 con 0x7f3004161b40 2024-03-04T09:29:44.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.754+0000 7f3009800640 1 -- 172.21.15.6:0/2103159647 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f30040b45c0 con 0x7f3004161b40 2024-03-04T09:29:44.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.755+0000 7f2ffaa00640 1 -- 172.21.15.6:0/2103159647 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v77) v1 ==== 84+0+65 (secure 0 0 0) 0x7f2ffc05f090 con 0x7f3004161b40 2024-03-04T09:29:44.752 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:29:44.752 INFO:teuthology.orchestra.run.smithi006.stdout:[ 2024-03-04T09:29:44.752 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:29:44.752 INFO:teuthology.orchestra.run.smithi006.stdout: "poolnum": 1, 2024-03-04T09:29:44.752 INFO:teuthology.orchestra.run.smithi006.stdout: "poolname": ".mgr" 2024-03-04T09:29:44.753 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:29:44.753 INFO:teuthology.orchestra.run.smithi006.stdout:] 2024-03-04T09:29:44.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.758+0000 7f3009800640 1 -- 172.21.15.6:0/2103159647 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2fe406b4d0 msgr2=0x7f2fe406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:44.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.758+0000 7f3009800640 1 --2- 172.21.15.6:0/2103159647 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2fe406b4d0 0x7f2fe406d990 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f2ff4006fd0 tx=0x7f2ff4008040 comp rx=0 tx=0).stop 2024-03-04T09:29:44.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.758+0000 7f3009800640 1 -- 172.21.15.6:0/2103159647 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3004161b40 msgr2=0x7f30041f4110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:44.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.758+0000 7f3009800640 1 --2- 172.21.15.6:0/2103159647 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3004161b40 0x7f30041f4110 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f2ffc007d20 tx=0x7f2ffc00e4b0 comp rx=0 tx=0).stop 2024-03-04T09:29:44.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.758+0000 7f3009800640 1 -- 172.21.15.6:0/2103159647 shutdown_connections 2024-03-04T09:29:44.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.758+0000 7f3009800640 1 --2- 172.21.15.6:0/2103159647 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3004161b40 0x7f30041f4110 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:44.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.758+0000 7f3009800640 1 --2- 172.21.15.6:0/2103159647 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2fe406b4d0 0x7f2fe406d990 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:44.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.758+0000 7f3009800640 1 --2- 172.21.15.6:0/2103159647 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f30040b54a0 0x7f3004152220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:44.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.758+0000 7f3009800640 1 --2- 172.21.15.6:0/2103159647 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f30040b5060 0x7f30040b80c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:44.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.758+0000 7f3009800640 1 -- 172.21.15.6:0/2103159647 >> 172.21.15.6:0/2103159647 conn(0x7f300409d220 msgr2=0x7f30040b3590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:29:44.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.758+0000 7f3009800640 1 -- 172.21.15.6:0/2103159647 shutdown_connections 2024-03-04T09:29:44.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:44.758+0000 7f3009800640 1 -- 172.21.15.6:0/2103159647 wait complete. 2024-03-04T09:29:44.774 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-04T09:29:44.774 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-04T09:29:45.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:45.006+0000 7ff18a800640 1 Processor -- start 2024-03-04T09:29:45.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:45.007+0000 7ff18a800640 1 -- start start 2024-03-04T09:29:45.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:45.008+0000 7ff18a800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff1840ae7b0 0x7ff1840aced0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:45.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:45.008+0000 7ff18a800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ff1840b3640 0x7ff18415cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:45.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:45.008+0000 7ff18a800640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1841531f0 con 0x7ff1840aeb90 2024-03-04T09:29:45.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:45.008+0000 7ff18a800640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1840d04d0 con 0x7ff1840b3640 2024-03-04T09:29:45.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:45.008+0000 7ff18a800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff18415d300 con 0x7ff1840ae7b0 2024-03-04T09:29:45.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:45.009+0000 7ff183e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7ff1840aeb90 0x7ff18406cc70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:59464/0 (socket says 172.21.15.6:59464) 2024-03-04T09:29:45.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:45.009+0000 7ff183e00640 1 -- 172.21.15.6:0/2019617791 learned_addr learned my addr 172.21.15.6:0/2019617791 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:29:45.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:45.009+0000 7ff182a00640 1 --2- 172.21.15.6:0/2019617791 >> v2:172.21.15.146:3300/0 conn(0x7ff1840b3640 0x7ff18415cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:45.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:45.009+0000 7ff183400640 1 --2- 172.21.15.6:0/2019617791 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff1840ae7b0 0x7ff1840aced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:45.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:45.009+0000 7ff182000640 1 -- 172.21.15.6:0/2019617791 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3953236650 0 0) 0x7ff1841531f0 con 0x7ff1840aeb90 2024-03-04T09:29:45.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:45.009+0000 7ff182000640 1 -- 172.21.15.6:0/2019617791 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff168003520 con 0x7ff1840aeb90 2024-03-04T09:29:45.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:45.010+0000 7ff182000640 1 -- 172.21.15.6:0/2019617791 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1314484143 0 0) 0x7ff168003520 con 0x7ff1840aeb90 2024-03-04T09:29:45.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:45.010+0000 7ff182000640 1 -- 172.21.15.6:0/2019617791 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff1840ae7b0 msgr2=0x7ff1840aced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:45.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:45.010+0000 7ff182000640 1 --2- 172.21.15.6:0/2019617791 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff1840ae7b0 0x7ff1840aced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:45.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:45.010+0000 7ff182000640 1 -- 172.21.15.6:0/2019617791 >> v2:172.21.15.146:3300/0 conn(0x7ff1840b3640 msgr2=0x7ff18415cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:45.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:45.010+0000 7ff182000640 1 --2- 172.21.15.6:0/2019617791 >> v2:172.21.15.146:3300/0 conn(0x7ff1840b3640 0x7ff18415cdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:45.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:45.010+0000 7ff182000640 1 -- 172.21.15.6:0/2019617791 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1840b2b80 con 0x7ff1840aeb90 2024-03-04T09:29:45.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:45.010+0000 7ff182000640 1 -- 172.21.15.6:0/2019617791 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7ff17c001670 con 0x7ff1840aeb90 2024-03-04T09:29:45.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:45.010+0000 7ff182000640 1 -- 172.21.15.6:0/2019617791 >> v1:172.21.15.6:6789/0 conn(0x7ff1840aeb90 legacy=0x7ff18406cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:45.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:45.010+0000 7ff182000640 1 --2- 172.21.15.6:0/2019617791 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff1840ae7b0 0x7ff1680039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:45.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:45.010+0000 7ff182000640 1 --2- 172.21.15.6:0/2019617791 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff168003f60 0x7ff168004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:45.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:45.010+0000 7ff182000640 1 --2- 172.21.15.6:0/2019617791 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff168004d40 0x7ff1680051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:45.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:45.010+0000 7ff182000640 1 -- 172.21.15.6:0/2019617791 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1840d04d0 con 0x7ff168003f60 2024-03-04T09:29:45.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:45.010+0000 7ff182000640 1 -- 172.21.15.6:0/2019617791 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff18415d300 con 0x7ff168004d40 2024-03-04T09:29:45.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:45.010+0000 7ff182000640 1 -- 172.21.15.6:0/2019617791 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff16800b9c0 con 0x7ff1840ae7b0 2024-03-04T09:29:45.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:45.010+0000 7ff182a00640 1 --2- 172.21.15.6:0/2019617791 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff168003f60 0x7ff168004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:45.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:45.010+0000 7ff183400640 1 --2- 172.21.15.6:0/2019617791 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff1840ae7b0 0x7ff1680039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:45.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:45.011+0000 7ff183e00640 1 --2- 172.21.15.6:0/2019617791 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff168004d40 0x7ff1680051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:45.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:45.011+0000 7ff182a00640 1 -- 172.21.15.6:0/2019617791 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff1840ae7b0 msgr2=0x7ff1680039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:45.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:45.011+0000 7ff182a00640 1 --2- 172.21.15.6:0/2019617791 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff1840ae7b0 0x7ff1680039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:45.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:45.011+0000 7ff182a00640 1 -- 172.21.15.6:0/2019617791 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff168004d40 msgr2=0x7ff1680051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:45.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:45.011+0000 7ff182a00640 1 --2- 172.21.15.6:0/2019617791 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff168004d40 0x7ff1680051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:45.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:45.011+0000 7ff182a00640 1 -- 172.21.15.6:0/2019617791 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff168003520 con 0x7ff168003f60 2024-03-04T09:29:45.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:45.011+0000 7ff183e00640 1 --2- 172.21.15.6:0/2019617791 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff168004d40 0x7ff1680051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:29:45.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:45.012+0000 7ff182a00640 1 --2- 172.21.15.6:0/2019617791 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff168003f60 0x7ff168004800 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7ff16c009ac0 tx=0x7ff168005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90d203f154bc1867 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:45.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:45.012+0000 7ff182000640 1 -- 172.21.15.6:0/2019617791 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff16c039070 con 0x7ff168003f60 2024-03-04T09:29:45.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:45.012+0000 7ff182000640 1 -- 172.21.15.6:0/2019617791 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff16c02c710 con 0x7ff168003f60 2024-03-04T09:29:45.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:45.012+0000 7ff182000640 1 -- 172.21.15.6:0/2019617791 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff16c009690 con 0x7ff168003f60 2024-03-04T09:29:45.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:45.013+0000 7ff18a800640 1 -- 172.21.15.6:0/2019617791 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff168003f60 msgr2=0x7ff168004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:45.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:45.013+0000 7ff18a800640 1 --2- 172.21.15.6:0/2019617791 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff168003f60 0x7ff168004800 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7ff16c009ac0 tx=0x7ff168005830 comp rx=0 tx=0).stop 2024-03-04T09:29:45.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:45.013+0000 7ff183e00640 1 -- 172.21.15.6:0/2019617791 reap_dead start 2024-03-04T09:29:45.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:45.013+0000 7ff18a800640 1 -- 172.21.15.6:0/2019617791 shutdown_connections 2024-03-04T09:29:45.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:45.013+0000 7ff18a800640 1 -- 172.21.15.6:0/2019617791 >> 172.21.15.6:0/2019617791 conn(0x7ff18406afb0 msgr2=0x7ff18406b3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:29:45.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:45.013+0000 7ff18a800640 1 -- 172.21.15.6:0/2019617791 shutdown_connections 2024-03-04T09:29:45.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:45.013+0000 7ff18a800640 1 -- 172.21.15.6:0/2019617791 wait complete. 2024-03-04T09:29:45.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:45.015+0000 7ff18a800640 1 Processor -- start 2024-03-04T09:29:45.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:45.015+0000 7ff18a800640 1 -- start start 2024-03-04T09:29:45.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:45.016+0000 7ff18a800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff168003f60 0x7ff18414a440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:45.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:45.016+0000 7ff18a800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff184148d10 0x7ff18414a980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:45.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:45.016+0000 7ff18a800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff18413a080 0x7ff18413a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:45.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:45.016+0000 7ff18a800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1840b2e40 con 0x7ff184148d10 2024-03-04T09:29:45.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:45.016+0000 7ff18a800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff18406eeb0 con 0x7ff168003f60 2024-03-04T09:29:45.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:45.016+0000 7ff18a800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1840b3120 con 0x7ff18413a080 2024-03-04T09:29:45.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:45.016+0000 7ff182a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff184148d10 0x7ff18414a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:45.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:45.016+0000 7ff182a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff184148d10 0x7ff18414a980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38864/0 (socket says 172.21.15.6:38864) 2024-03-04T09:29:45.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:45.016+0000 7ff182a00640 1 -- 172.21.15.6:0/264834246 learned_addr learned my addr 172.21.15.6:0/264834246 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:29:45.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:45.017+0000 7ff183400640 1 --2- 172.21.15.6:0/264834246 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff168003f60 0x7ff18414a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:45.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:45.017+0000 7ff182a00640 1 -- 172.21.15.6:0/264834246 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff18413a080 msgr2=0x7ff18413a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:45.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:45.017+0000 7ff183e00640 1 --2- 172.21.15.6:0/264834246 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff18413a080 0x7ff18413a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:45.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:45.017+0000 7ff182a00640 1 --2- 172.21.15.6:0/264834246 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff18413a080 0x7ff18413a500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:45.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:45.017+0000 7ff182a00640 1 -- 172.21.15.6:0/264834246 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff168003f60 msgr2=0x7ff18414a440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:45.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:45.017+0000 7ff182a00640 1 --2- 172.21.15.6:0/264834246 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff168003f60 0x7ff18414a440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:45.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:45.017+0000 7ff182a00640 1 -- 172.21.15.6:0/264834246 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff18414b8a0 con 0x7ff184148d10 2024-03-04T09:29:45.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:45.017+0000 7ff183400640 1 --2- 172.21.15.6:0/264834246 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff168003f60 0x7ff18414a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:29:45.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:45.017+0000 7ff183e00640 1 --2- 172.21.15.6:0/264834246 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff18413a080 0x7ff18413a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:29:45.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:45.017+0000 7ff182a00640 1 --2- 172.21.15.6:0/264834246 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff184148d10 0x7ff18414a980 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7ff16c037040 tx=0x7ff16c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:45.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:45.017+0000 7ff17b400640 1 -- 172.21.15.6:0/264834246 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff16c042020 con 0x7ff184148d10 2024-03-04T09:29:45.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:45.017+0000 7ff17b400640 1 -- 172.21.15.6:0/264834246 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff16c03a040 con 0x7ff184148d10 2024-03-04T09:29:45.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:45.017+0000 7ff17b400640 1 -- 172.21.15.6:0/264834246 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff16c039070 con 0x7ff184148d10 2024-03-04T09:29:45.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:45.017+0000 7ff18a800640 1 -- 172.21.15.6:0/264834246 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff18414b000 con 0x7ff184148d10 2024-03-04T09:29:45.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:45.017+0000 7ff18a800640 1 -- 172.21.15.6:0/264834246 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff18414b5c0 con 0x7ff184148d10 2024-03-04T09:29:45.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:45.019+0000 7ff18a800640 1 -- 172.21.15.6:0/264834246 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff184137d10 con 0x7ff184148d10 2024-03-04T09:29:45.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:45.020+0000 7ff17b400640 1 -- 172.21.15.6:0/264834246 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff16c045020 con 0x7ff184148d10 2024-03-04T09:29:45.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:45.021+0000 7ff17b400640 1 --2- 172.21.15.6:0/264834246 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff16006b500 0x7ff16006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:45.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:45.021+0000 7ff17b400640 1 -- 172.21.15.6:0/264834246 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 4857+0+0 (secure 0 0 0) 0x7ff16c0aaf90 con 0x7ff184148d10 2024-03-04T09:29:45.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:45.021+0000 7ff183400640 1 --2- 172.21.15.6:0/264834246 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff16006b500 0x7ff16006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:45.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:45.023+0000 7ff183400640 1 --2- 172.21.15.6:0/264834246 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff16006b500 0x7ff16006d9c0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7ff18409ade0 tx=0x7ff17400b040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:45.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:45.026+0000 7ff17b400640 1 -- 172.21.15.6:0/264834246 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff184137d10 con 0x7ff184148d10 2024-03-04T09:29:45.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:45.226+0000 7ff18a800640 1 -- 172.21.15.6:0/264834246 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7ff184068010 con 0x7ff184148d10 2024-03-04T09:29:45.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:45.859+0000 7ff17b400640 1 -- 172.21.15.6:0/264834246 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v78) v1 ==== 138+0+0 (secure 0 0 0) 0x7ff16c04c020 con 0x7ff184148d10 2024-03-04T09:29:45.860 INFO:teuthology.orchestra.run.smithi006.stderr:pool 'cephfs_metadata' created 2024-03-04T09:29:45.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:45.863+0000 7ff18a800640 1 -- 172.21.15.6:0/264834246 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff16006b500 msgr2=0x7ff16006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:45.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:45.863+0000 7ff18a800640 1 --2- 172.21.15.6:0/264834246 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff16006b500 0x7ff16006d9c0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7ff18409ade0 tx=0x7ff17400b040 comp rx=0 tx=0).stop 2024-03-04T09:29:45.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:45.863+0000 7ff18a800640 1 -- 172.21.15.6:0/264834246 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff184148d10 msgr2=0x7ff18414a980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:45.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:45.863+0000 7ff18a800640 1 --2- 172.21.15.6:0/264834246 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff184148d10 0x7ff18414a980 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7ff16c037040 tx=0x7ff16c004730 comp rx=0 tx=0).stop 2024-03-04T09:29:45.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:45.864+0000 7ff18a800640 1 -- 172.21.15.6:0/264834246 shutdown_connections 2024-03-04T09:29:45.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:45.864+0000 7ff18a800640 1 --2- 172.21.15.6:0/264834246 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff16006b500 0x7ff16006d9c0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:45.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:45.864+0000 7ff18a800640 1 --2- 172.21.15.6:0/264834246 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff18413a080 0x7ff18413a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:45.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:45.864+0000 7ff18a800640 1 --2- 172.21.15.6:0/264834246 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff184148d10 0x7ff18414a980 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:45.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:45.864+0000 7ff18a800640 1 --2- 172.21.15.6:0/264834246 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff168003f60 0x7ff18414a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:45.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:45.864+0000 7ff18a800640 1 -- 172.21.15.6:0/264834246 >> 172.21.15.6:0/264834246 conn(0x7ff18406afb0 msgr2=0x7ff18406cc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:29:45.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:45.864+0000 7ff18a800640 1 -- 172.21.15.6:0/264834246 shutdown_connections 2024-03-04T09:29:45.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:45.864+0000 7ff18a800640 1 -- 172.21.15.6:0/264834246 wait complete. 2024-03-04T09:29:45.891 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-04T09:29:46.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:46.105+0000 7f62c2400640 1 Processor -- start 2024-03-04T09:29:46.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:46.105+0000 7f62c2400640 1 -- start start 2024-03-04T09:29:46.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:46.106+0000 7f62c2400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f62bc15e170 0x7f62bc0d2870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:46.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:46.107+0000 7f62c2400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f62bc1625c0 0x7f62bc1681e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:46.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:46.107+0000 7f62c2400640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62bc0b3d90 con 0x7f62bc0b6280 2024-03-04T09:29:46.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:46.107+0000 7f62c2400640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62bc0cfee0 con 0x7f62bc15e170 2024-03-04T09:29:46.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:46.107+0000 7f62c2400640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62bc0cf640 con 0x7f62bc1625c0 2024-03-04T09:29:46.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:46.107+0000 7f62bb400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f62bc15e170 0x7f62bc0d2870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:46.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:46.107+0000 7f62baa00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f62bc0b6280 0x7f62bc161eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:59470/0 (socket says 172.21.15.6:59470) 2024-03-04T09:29:46.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:46.107+0000 7f62baa00640 1 -- 172.21.15.6:0/4069128069 learned_addr learned my addr 172.21.15.6:0/4069128069 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:29:46.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:46.107+0000 7f62bb400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f62bc15e170 0x7f62bc0d2870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38876/0 (socket says 172.21.15.6:38876) 2024-03-04T09:29:46.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:46.107+0000 7f62bbe00640 1 --2- 172.21.15.6:0/4069128069 >> v2:172.21.15.146:3300/0 conn(0x7f62bc1625c0 0x7f62bc1681e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:46.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:46.108+0000 7f62bb400640 1 -- 172.21.15.6:0/4069128069 >> v2:172.21.15.146:3300/0 conn(0x7f62bc1625c0 msgr2=0x7f62bc1681e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:46.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:46.108+0000 7f62bb400640 1 --2- 172.21.15.6:0/4069128069 >> v2:172.21.15.146:3300/0 conn(0x7f62bc1625c0 0x7f62bc1681e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:46.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:46.108+0000 7f62bb400640 1 -- 172.21.15.6:0/4069128069 >> v1:172.21.15.6:6789/0 conn(0x7f62bc0b6280 legacy=0x7f62bc161eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:46.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:46.108+0000 7f62bb400640 1 -- 172.21.15.6:0/4069128069 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62bc0cf920 con 0x7f62bc15e170 2024-03-04T09:29:46.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:46.108+0000 7f62ba000640 1 -- 172.21.15.6:0/4069128069 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2213573705 0 0) 0x7f62bc0b3d90 con 0x7f62bc0b6280 2024-03-04T09:29:46.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:46.109+0000 7f62bb400640 1 --2- 172.21.15.6:0/4069128069 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f62bc15e170 0x7f62bc0d2870 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f62bc0d19c0 tx=0x7f62b402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3027449003703651 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:46.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:46.109+0000 7f62ba000640 1 -- 172.21.15.6:0/4069128069 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62b4039070 con 0x7f62bc15e170 2024-03-04T09:29:46.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:46.109+0000 7f62ba000640 1 -- 172.21.15.6:0/4069128069 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62b402cc80 con 0x7f62bc15e170 2024-03-04T09:29:46.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:46.109+0000 7f62ba000640 1 -- 172.21.15.6:0/4069128069 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62b4034660 con 0x7f62bc15e170 2024-03-04T09:29:46.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:46.110+0000 7f62c2400640 1 -- 172.21.15.6:0/4069128069 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f62bc15e170 msgr2=0x7f62bc0d2870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:46.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:46.110+0000 7f62c2400640 1 --2- 172.21.15.6:0/4069128069 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f62bc15e170 0x7f62bc0d2870 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f62bc0d19c0 tx=0x7f62b402bed0 comp rx=0 tx=0).stop 2024-03-04T09:29:46.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:46.110+0000 7f62c2400640 1 -- 172.21.15.6:0/4069128069 shutdown_connections 2024-03-04T09:29:46.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:46.110+0000 7f62c2400640 1 --2- 172.21.15.6:0/4069128069 >> v2:172.21.15.146:3300/0 conn(0x7f62bc1625c0 0x7f62bc1681e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:46.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:46.110+0000 7f62c2400640 1 --2- 172.21.15.6:0/4069128069 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f62bc15e170 0x7f62bc0d2870 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:46.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:46.110+0000 7f62c2400640 1 -- 172.21.15.6:0/4069128069 >> 172.21.15.6:0/4069128069 conn(0x7f62bc09d290 msgr2=0x7f62bc08d4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:29:46.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:46.110+0000 7f62c2400640 1 -- 172.21.15.6:0/4069128069 shutdown_connections 2024-03-04T09:29:46.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:46.110+0000 7f62c2400640 1 -- 172.21.15.6:0/4069128069 wait complete. 2024-03-04T09:29:46.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:46.112+0000 7f62c2400640 1 Processor -- start 2024-03-04T09:29:46.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:46.112+0000 7f62c2400640 1 -- start start 2024-03-04T09:29:46.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:46.113+0000 7f62c2400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f62bc0b6280 0x7f62bc15b550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:46.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:46.113+0000 7f62c2400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f62bc15e170 0x7f62bc161040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:46.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:46.113+0000 7f62c2400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f62bc1625c0 0x7f62bc0d0290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:46.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:46.113+0000 7f62c2400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62bc0d2f30 con 0x7f62bc15e170 2024-03-04T09:29:46.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:46.113+0000 7f62c2400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62bc0d2db0 con 0x7f62bc0b6280 2024-03-04T09:29:46.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:46.113+0000 7f62c2400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62bc15e550 con 0x7f62bc1625c0 2024-03-04T09:29:46.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:46.113+0000 7f62baa00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f62bc15e170 0x7f62bc161040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:46.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:46.113+0000 7f62baa00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f62bc15e170 0x7f62bc161040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38878/0 (socket says 172.21.15.6:38878) 2024-03-04T09:29:46.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:46.113+0000 7f62baa00640 1 -- 172.21.15.6:0/2830254297 learned_addr learned my addr 172.21.15.6:0/2830254297 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:29:46.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:46.114+0000 7f62bb400640 1 --2- 172.21.15.6:0/2830254297 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f62bc0b6280 0x7f62bc15b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:46.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:46.114+0000 7f62bbe00640 1 --2- 172.21.15.6:0/2830254297 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f62bc1625c0 0x7f62bc0d0290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:46.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:46.114+0000 7f62baa00640 1 -- 172.21.15.6:0/2830254297 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f62bc1625c0 msgr2=0x7f62bc0d0290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:46.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:46.114+0000 7f62baa00640 1 --2- 172.21.15.6:0/2830254297 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f62bc1625c0 0x7f62bc0d0290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:46.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:46.114+0000 7f62baa00640 1 -- 172.21.15.6:0/2830254297 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f62bc0b6280 msgr2=0x7f62bc15b550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:46.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:46.114+0000 7f62baa00640 1 --2- 172.21.15.6:0/2830254297 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f62bc0b6280 0x7f62bc15b550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:46.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:46.114+0000 7f62baa00640 1 -- 172.21.15.6:0/2830254297 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62bc0c9380 con 0x7f62bc15e170 2024-03-04T09:29:46.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:46.114+0000 7f62bbe00640 1 --2- 172.21.15.6:0/2830254297 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f62bc1625c0 0x7f62bc0d0290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:29:46.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:46.114+0000 7f62baa00640 1 --2- 172.21.15.6:0/2830254297 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f62bc15e170 0x7f62bc161040 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f62a400c4a0 tx=0x7f62a400c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:46.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:46.114+0000 7f62b3400640 1 -- 172.21.15.6:0/2830254297 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62a4017070 con 0x7f62bc15e170 2024-03-04T09:29:46.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:46.114+0000 7f62b3400640 1 -- 172.21.15.6:0/2830254297 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62a4005ce0 con 0x7f62bc15e170 2024-03-04T09:29:46.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:46.114+0000 7f62b3400640 1 -- 172.21.15.6:0/2830254297 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62a40072b0 con 0x7f62bc15e170 2024-03-04T09:29:46.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:46.115+0000 7f62c2400640 1 -- 172.21.15.6:0/2830254297 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62bc0c9620 con 0x7f62bc15e170 2024-03-04T09:29:46.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:46.115+0000 7f62c2400640 1 -- 172.21.15.6:0/2830254297 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62bc0c9be0 con 0x7f62bc15e170 2024-03-04T09:29:46.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:46.116+0000 7f62c2400640 1 -- 172.21.15.6:0/2830254297 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62bc068010 con 0x7f62bc15e170 2024-03-04T09:29:46.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:46.117+0000 7f62b3400640 1 -- 172.21.15.6:0/2830254297 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f62a401a070 con 0x7f62bc15e170 2024-03-04T09:29:46.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:46.118+0000 7f62b3400640 1 --2- 172.21.15.6:0/2830254297 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f629806b4d0 0x7f629806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:46.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:46.118+0000 7f62b3400640 1 -- 172.21.15.6:0/2830254297 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f62a4089440 con 0x7f62bc15e170 2024-03-04T09:29:46.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:46.118+0000 7f62bb400640 1 --2- 172.21.15.6:0/2830254297 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f629806b4d0 0x7f629806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:46.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:46.120+0000 7f62bb400640 1 --2- 172.21.15.6:0/2830254297 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f629806b4d0 0x7f629806d990 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f62b4036000 tx=0x7f62b4030000 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:46.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:46.123+0000 7f62b3400640 1 -- 172.21.15.6:0/2830254297 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f62a4058400 con 0x7f62bc15e170 2024-03-04T09:29:46.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:46.324+0000 7f62c2400640 1 -- 172.21.15.6:0/2830254297 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f62bc0743f0 con 0x7f62bc15e170 2024-03-04T09:29:46.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:46.866+0000 7f62b3400640 1 -- 172.21.15.6:0/2830254297 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v79) v1 ==== 170+0+0 (secure 0 0 0) 0x7f62a4057d00 con 0x7f62bc15e170 2024-03-04T09:29:46.863 INFO:teuthology.orchestra.run.smithi006.stderr:pool 'cephfs_data' created 2024-03-04T09:29:46.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:46.870+0000 7f62c2400640 1 -- 172.21.15.6:0/2830254297 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f629806b4d0 msgr2=0x7f629806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:46.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:46.870+0000 7f62c2400640 1 --2- 172.21.15.6:0/2830254297 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f629806b4d0 0x7f629806d990 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f62b4036000 tx=0x7f62b4030000 comp rx=0 tx=0).stop 2024-03-04T09:29:46.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:46.870+0000 7f62c2400640 1 -- 172.21.15.6:0/2830254297 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f62bc15e170 msgr2=0x7f62bc161040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:46.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:46.870+0000 7f62c2400640 1 --2- 172.21.15.6:0/2830254297 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f62bc15e170 0x7f62bc161040 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f62a400c4a0 tx=0x7f62a400c960 comp rx=0 tx=0).stop 2024-03-04T09:29:46.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:46.870+0000 7f62c2400640 1 -- 172.21.15.6:0/2830254297 shutdown_connections 2024-03-04T09:29:46.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:46.870+0000 7f62c2400640 1 --2- 172.21.15.6:0/2830254297 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f629806b4d0 0x7f629806d990 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:46.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:46.870+0000 7f62c2400640 1 --2- 172.21.15.6:0/2830254297 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f62bc1625c0 0x7f62bc0d0290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:46.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:46.870+0000 7f62c2400640 1 --2- 172.21.15.6:0/2830254297 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f62bc15e170 0x7f62bc161040 secure :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f62a400c4a0 tx=0x7f62a400c960 comp rx=0 tx=0).stop 2024-03-04T09:29:46.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:46.870+0000 7f62c2400640 1 --2- 172.21.15.6:0/2830254297 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f62bc0b6280 0x7f62bc15b550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:46.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:46.870+0000 7f62c2400640 1 -- 172.21.15.6:0/2830254297 >> 172.21.15.6:0/2830254297 conn(0x7f62bc09d290 msgr2=0x7f62bc0d31d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:29:46.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:46.871+0000 7f62c2400640 1 -- 172.21.15.6:0/2830254297 shutdown_connections 2024-03-04T09:29:46.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:46.872+0000 7f62c2400640 1 -- 172.21.15.6:0/2830254297 wait complete. 2024-03-04T09:29:46.905 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-04T09:29:47.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:47.097+0000 7f8689600640 1 Processor -- start 2024-03-04T09:29:47.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:47.098+0000 7f8689600640 1 -- start start 2024-03-04T09:29:47.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:47.099+0000 7f8689600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f868415bf20 0x7f86840acd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:47.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:47.099+0000 7f8689600640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8684072df0 con 0x7f8684072520 2024-03-04T09:29:47.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:47.099+0000 7f8689600640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86840cdab0 con 0x7f8684071e20 2024-03-04T09:29:47.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:47.099+0000 7f8689600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86840b4d70 con 0x7f868415bf20 2024-03-04T09:29:47.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:47.100+0000 7f8682000640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f8684072520 0x7f86840cdfb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:59484/0 (socket says 172.21.15.6:59484) 2024-03-04T09:29:47.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:47.100+0000 7f8682000640 1 -- 172.21.15.6:0/2729147326 learned_addr learned my addr 172.21.15.6:0/2729147326 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:29:47.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:47.100+0000 7f8682a00640 1 --2- 172.21.15.6:0/2729147326 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f868415bf20 0x7f86840acd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:47.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:47.100+0000 7f8681600640 1 -- 172.21.15.6:0/2729147326 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1624168957 0 0) 0x7f8684072df0 con 0x7f8684072520 2024-03-04T09:29:47.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:47.100+0000 7f8681600640 1 -- 172.21.15.6:0/2729147326 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8668003520 con 0x7f8684072520 2024-03-04T09:29:47.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:47.101+0000 7f8681600640 1 -- 172.21.15.6:0/2729147326 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1285466129 0 0) 0x7f8668003520 con 0x7f8684072520 2024-03-04T09:29:47.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:47.101+0000 7f8681600640 1 -- 172.21.15.6:0/2729147326 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f868415bf20 msgr2=0x7f86840acd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:47.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:47.101+0000 7f8681600640 1 --2- 172.21.15.6:0/2729147326 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f868415bf20 0x7f86840acd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:47.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:47.101+0000 7f8681600640 1 -- 172.21.15.6:0/2729147326 >> v1:172.21.15.146:6789/0 conn(0x7f8684071e20 legacy=0x7f86840ccad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:47.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:47.101+0000 7f8681600640 1 -- 172.21.15.6:0/2729147326 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86840b5050 con 0x7f8684072520 2024-03-04T09:29:47.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:47.101+0000 7f8681600640 1 -- 172.21.15.6:0/2729147326 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7f866c001670 con 0x7f8684072520 2024-03-04T09:29:47.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:47.101+0000 7f8681600640 1 -- 172.21.15.6:0/2729147326 >> v1:172.21.15.6:6789/0 conn(0x7f8684072520 legacy=0x7f86840cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:47.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:47.101+0000 7f8681600640 1 --2- 172.21.15.6:0/2729147326 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f86680039e0 0x7f8668003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:47.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:47.101+0000 7f8681600640 1 --2- 172.21.15.6:0/2729147326 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8668004800 0x7f8668004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:47.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:47.101+0000 7f8681600640 1 --2- 172.21.15.6:0/2729147326 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f868415bf20 0x7f86680051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:47.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:47.101+0000 7f8681600640 1 -- 172.21.15.6:0/2729147326 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86840b4d70 con 0x7f8668004800 2024-03-04T09:29:47.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:47.101+0000 7f8681600640 1 -- 172.21.15.6:0/2729147326 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f866800ba70 con 0x7f86680039e0 2024-03-04T09:29:47.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:47.101+0000 7f8681600640 1 -- 172.21.15.6:0/2729147326 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f866800bc70 con 0x7f868415bf20 2024-03-04T09:29:47.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:47.101+0000 7f8682000640 1 --2- 172.21.15.6:0/2729147326 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8668004800 0x7f8668004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:47.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:47.102+0000 7f8682000640 1 -- 172.21.15.6:0/2729147326 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f868415bf20 msgr2=0x7f86680051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:47.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:47.101+0000 7f8683400640 1 --2- 172.21.15.6:0/2729147326 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f868415bf20 0x7f86680051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:47.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:47.102+0000 7f8682a00640 1 --2- 172.21.15.6:0/2729147326 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f86680039e0 0x7f8668003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:47.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:47.102+0000 7f8682000640 1 --2- 172.21.15.6:0/2729147326 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f868415bf20 0x7f86680051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:47.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:47.102+0000 7f8682000640 1 -- 172.21.15.6:0/2729147326 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f86680039e0 msgr2=0x7f8668003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:47.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:47.102+0000 7f8682000640 1 --2- 172.21.15.6:0/2729147326 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f86680039e0 0x7f8668003e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:47.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:47.102+0000 7f8682000640 1 -- 172.21.15.6:0/2729147326 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8668003520 con 0x7f8668004800 2024-03-04T09:29:47.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:47.102+0000 7f8683400640 1 --2- 172.21.15.6:0/2729147326 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f868415bf20 0x7f86680051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:29:47.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:47.103+0000 7f8682000640 1 --2- 172.21.15.6:0/2729147326 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8668004800 0x7f8668004c80 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f866c00c210 tx=0x7f8668005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bef5d41afea29cc server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:47.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:47.103+0000 7f8681600640 1 -- 172.21.15.6:0/2729147326 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f866c007d10 con 0x7f8668004800 2024-03-04T09:29:47.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:47.103+0000 7f8681600640 1 -- 172.21.15.6:0/2729147326 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f866c030070 con 0x7f8668004800 2024-03-04T09:29:47.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:47.103+0000 7f8681600640 1 -- 172.21.15.6:0/2729147326 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f866c005d30 con 0x7f8668004800 2024-03-04T09:29:47.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:47.104+0000 7f8689600640 1 -- 172.21.15.6:0/2729147326 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8668004800 msgr2=0x7f8668004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:47.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:47.104+0000 7f8689600640 1 --2- 172.21.15.6:0/2729147326 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8668004800 0x7f8668004c80 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f866c00c210 tx=0x7f8668005830 comp rx=0 tx=0).stop 2024-03-04T09:29:47.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:47.104+0000 7f8683400640 1 -- 172.21.15.6:0/2729147326 reap_dead start 2024-03-04T09:29:47.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:47.104+0000 7f8689600640 1 -- 172.21.15.6:0/2729147326 shutdown_connections 2024-03-04T09:29:47.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:47.104+0000 7f8689600640 1 -- 172.21.15.6:0/2729147326 >> 172.21.15.6:0/2729147326 conn(0x7f868406b3d0 msgr2=0x7f868406b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:29:47.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:47.104+0000 7f8689600640 1 -- 172.21.15.6:0/2729147326 shutdown_connections 2024-03-04T09:29:47.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:47.104+0000 7f8689600640 1 -- 172.21.15.6:0/2729147326 wait complete. 2024-03-04T09:29:47.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:47.106+0000 7f8689600640 1 Processor -- start 2024-03-04T09:29:47.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:47.106+0000 7f8689600640 1 -- start start 2024-03-04T09:29:47.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:47.107+0000 7f8689600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8668004800 0x7f86841501e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:47.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:47.107+0000 7f8689600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f868406dd80 0x7f868415c570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:47.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:47.107+0000 7f8689600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f86841450b0 0x7f8684145510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:47.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:47.107+0000 7f8689600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86840b4b10 con 0x7f8668004800 2024-03-04T09:29:47.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:47.107+0000 7f8689600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86840b4990 con 0x7f86841450b0 2024-03-04T09:29:47.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:47.107+0000 7f8689600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86840ce6c0 con 0x7f868406dd80 2024-03-04T09:29:47.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:47.107+0000 7f8682a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8668004800 0x7f86841501e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:47.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:47.107+0000 7f8682a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8668004800 0x7f86841501e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38892/0 (socket says 172.21.15.6:38892) 2024-03-04T09:29:47.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:47.107+0000 7f8682a00640 1 -- 172.21.15.6:0/2771770312 learned_addr learned my addr 172.21.15.6:0/2771770312 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:29:47.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:47.107+0000 7f8682000640 1 --2- 172.21.15.6:0/2771770312 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f868406dd80 0x7f868415c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:47.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:47.108+0000 7f8682a00640 1 -- 172.21.15.6:0/2771770312 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f868406dd80 msgr2=0x7f868415c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:47.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:47.108+0000 7f8682a00640 1 --2- 172.21.15.6:0/2771770312 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f868406dd80 0x7f868415c570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:47.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:47.108+0000 7f8683400640 1 --2- 172.21.15.6:0/2771770312 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f86841450b0 0x7f8684145510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:47.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:47.108+0000 7f8682a00640 1 -- 172.21.15.6:0/2771770312 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f86841450b0 msgr2=0x7f8684145510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:47.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:47.108+0000 7f8682a00640 1 --2- 172.21.15.6:0/2771770312 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f86841450b0 0x7f8684145510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:47.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:47.108+0000 7f8682a00640 1 -- 172.21.15.6:0/2771770312 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8684150830 con 0x7f8668004800 2024-03-04T09:29:47.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:47.108+0000 7f8682a00640 1 --2- 172.21.15.6:0/2771770312 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8668004800 0x7f86841501e0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f8674010890 tx=0x7f8674010d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:47.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:47.108+0000 7f867aa00640 1 -- 172.21.15.6:0/2771770312 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f867400f040 con 0x7f8668004800 2024-03-04T09:29:47.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:47.108+0000 7f867aa00640 1 -- 172.21.15.6:0/2771770312 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f867400eb60 con 0x7f8668004800 2024-03-04T09:29:47.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:47.108+0000 7f867aa00640 1 -- 172.21.15.6:0/2771770312 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f867401c400 con 0x7f8668004800 2024-03-04T09:29:47.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:47.108+0000 7f8689600640 1 -- 172.21.15.6:0/2771770312 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8684150b10 con 0x7f8668004800 2024-03-04T09:29:47.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:47.108+0000 7f8689600640 1 -- 172.21.15.6:0/2771770312 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86841510d0 con 0x7f8668004800 2024-03-04T09:29:47.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:47.110+0000 7f8689600640 1 -- 172.21.15.6:0/2771770312 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f868404f330 con 0x7f8668004800 2024-03-04T09:29:47.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:47.111+0000 7f867aa00640 1 -- 172.21.15.6:0/2771770312 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f867401c5a0 con 0x7f8668004800 2024-03-04T09:29:47.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:47.111+0000 7f867aa00640 1 --2- 172.21.15.6:0/2771770312 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f866006b4d0 0x7f866006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:47.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:47.112+0000 7f867aa00640 1 -- 172.21.15.6:0/2771770312 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f867408a7f0 con 0x7f8668004800 2024-03-04T09:29:47.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:47.112+0000 7f8682000640 1 --2- 172.21.15.6:0/2771770312 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f866006b4d0 0x7f866006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:47.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:47.113+0000 7f8682000640 1 --2- 172.21.15.6:0/2771770312 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f866006b4d0 0x7f866006d990 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f866c034000 tx=0x7f866c009030 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:47.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:47.116+0000 7f867aa00640 1 -- 172.21.15.6:0/2771770312 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8674018030 con 0x7f8668004800 2024-03-04T09:29:47.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:47.355+0000 7f8689600640 1 -- 172.21.15.6:0/2771770312 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f868415f720 con 0x7f8668004800 2024-03-04T09:29:47.901 INFO:teuthology.orchestra.run.smithi006.stderr:new fs with metadata pool 12 and data pool 13 2024-03-04T09:29:47.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:47.903+0000 7f867aa00640 1 -- 172.21.15.6:0/2771770312 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 12 and data pool 13 v47) v1 ==== 174+0+0 (secure 0 0 0) 0x7f86740596c0 con 0x7f8668004800 2024-03-04T09:29:47.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:47.916+0000 7f8689600640 1 -- 172.21.15.6:0/2771770312 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f866006b4d0 msgr2=0x7f866006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:47.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:47.916+0000 7f8689600640 1 --2- 172.21.15.6:0/2771770312 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f866006b4d0 0x7f866006d990 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f866c034000 tx=0x7f866c009030 comp rx=0 tx=0).stop 2024-03-04T09:29:47.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:47.916+0000 7f8689600640 1 -- 172.21.15.6:0/2771770312 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8668004800 msgr2=0x7f86841501e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:47.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:47.916+0000 7f8689600640 1 --2- 172.21.15.6:0/2771770312 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8668004800 0x7f86841501e0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f8674010890 tx=0x7f8674010d50 comp rx=0 tx=0).stop 2024-03-04T09:29:47.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:47.917+0000 7f8689600640 1 -- 172.21.15.6:0/2771770312 shutdown_connections 2024-03-04T09:29:47.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:47.917+0000 7f8689600640 1 --2- 172.21.15.6:0/2771770312 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f86841450b0 0x7f8684145510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:47.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:47.917+0000 7f8689600640 1 --2- 172.21.15.6:0/2771770312 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f866006b4d0 0x7f866006d990 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:47.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:47.917+0000 7f8689600640 1 --2- 172.21.15.6:0/2771770312 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f868406dd80 0x7f868415c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:47.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:47.917+0000 7f8689600640 1 --2- 172.21.15.6:0/2771770312 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8668004800 0x7f86841501e0 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:47.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:47.917+0000 7f8689600640 1 -- 172.21.15.6:0/2771770312 >> 172.21.15.6:0/2771770312 conn(0x7f868406b3d0 msgr2=0x7f868415a3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:29:47.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:47.918+0000 7f8689600640 1 -- 172.21.15.6:0/2771770312 shutdown_connections 2024-03-04T09:29:47.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:47.918+0000 7f8689600640 1 -- 172.21.15.6:0/2771770312 wait complete. 2024-03-04T09:29:47.951 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T09:29:48.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.135+0000 7f9071000640 1 Processor -- start 2024-03-04T09:29:48.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.135+0000 7f9071000640 1 -- start start 2024-03-04T09:29:48.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.136+0000 7f9071000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f906c15b290 0x7f906c15a580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:48.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.136+0000 7f9071000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f906c06e760 0x7f906c06cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:48.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.136+0000 7f9071000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f906c073360 con 0x7f906c071090 2024-03-04T09:29:48.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.136+0000 7f9071000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f906c06e2d0 con 0x7f906c15b290 2024-03-04T09:29:48.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.136+0000 7f9071000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f906c06d6c0 con 0x7f906c06e760 2024-03-04T09:29:48.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.137+0000 7f9069e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f906c15b290 0x7f906c15a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:48.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.137+0000 7f9069e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f906c15b290 0x7f906c15a580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38906/0 (socket says 172.21.15.6:38906) 2024-03-04T09:29:48.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.137+0000 7f9069e00640 1 -- 172.21.15.6:0/3503488540 learned_addr learned my addr 172.21.15.6:0/3503488540 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:29:48.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.137+0000 7f9069e00640 1 -- 172.21.15.6:0/3503488540 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f906c06e760 msgr2=0x7f906c06cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:48.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.137+0000 7f9069e00640 1 --2- 172.21.15.6:0/3503488540 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f906c06e760 0x7f906c06cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:48.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.137+0000 7f9069e00640 1 -- 172.21.15.6:0/3503488540 >> v1:172.21.15.6:6789/0 conn(0x7f906c071090 legacy=0x7f906c15db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:48.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.137+0000 7f9069e00640 1 -- 172.21.15.6:0/3503488540 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f906c06d9a0 con 0x7f906c15b290 2024-03-04T09:29:48.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.137+0000 7f9069400640 1 -- 172.21.15.6:0/3503488540 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2973991950 0 0) 0x7f9064001490 con 0x7f906c071090 2024-03-04T09:29:48.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.138+0000 7f9069e00640 1 --2- 172.21.15.6:0/3503488540 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f906c15b290 0x7f906c15a580 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f905c009870 tx=0x7f905c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=718e656eb1129a2a server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:48.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.139+0000 7f9069400640 1 -- 172.21.15.6:0/3503488540 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f905c034070 con 0x7f906c15b290 2024-03-04T09:29:48.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.139+0000 7f9069400640 1 -- 172.21.15.6:0/3503488540 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f905c030840 con 0x7f906c15b290 2024-03-04T09:29:48.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.139+0000 7f9069400640 1 -- 172.21.15.6:0/3503488540 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f905c030c10 con 0x7f906c15b290 2024-03-04T09:29:48.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.139+0000 7f9071000640 1 -- 172.21.15.6:0/3503488540 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f906c15b290 msgr2=0x7f906c15a580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:48.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.139+0000 7f9071000640 1 --2- 172.21.15.6:0/3503488540 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f906c15b290 0x7f906c15a580 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f905c009870 tx=0x7f905c02b980 comp rx=0 tx=0).stop 2024-03-04T09:29:48.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.139+0000 7f9071000640 1 -- 172.21.15.6:0/3503488540 shutdown_connections 2024-03-04T09:29:48.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.139+0000 7f9071000640 1 --2- 172.21.15.6:0/3503488540 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f906c06e760 0x7f906c06cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:48.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.139+0000 7f9071000640 1 --2- 172.21.15.6:0/3503488540 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f906c15b290 0x7f906c15a580 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:48.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.139+0000 7f9071000640 1 -- 172.21.15.6:0/3503488540 >> 172.21.15.6:0/3503488540 conn(0x7f906c0b1bb0 msgr2=0x7f906c0b1fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:29:48.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.139+0000 7f9071000640 1 -- 172.21.15.6:0/3503488540 shutdown_connections 2024-03-04T09:29:48.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.140+0000 7f9071000640 1 -- 172.21.15.6:0/3503488540 wait complete. 2024-03-04T09:29:48.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.141+0000 7f9071000640 1 Processor -- start 2024-03-04T09:29:48.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.141+0000 7f9071000640 1 -- start start 2024-03-04T09:29:48.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.142+0000 7f9071000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f906c06e760 0x7f906c14b5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:48.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.142+0000 7f9071000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f906c071090 0x7f906c0ae280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:48.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.142+0000 7f9071000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f906c15b290 0x7f906c146550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:48.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.142+0000 7f9071000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f906c069060 con 0x7f906c15b290 2024-03-04T09:29:48.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.142+0000 7f9071000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f906c06df60 con 0x7f906c06e760 2024-03-04T09:29:48.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.142+0000 7f9071000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f906c0691e0 con 0x7f906c071090 2024-03-04T09:29:48.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.142+0000 7f906a800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f906c15b290 0x7f906c146550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:48.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.142+0000 7f906a800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f906c15b290 0x7f906c146550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38920/0 (socket says 172.21.15.6:38920) 2024-03-04T09:29:48.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.142+0000 7f906a800640 1 -- 172.21.15.6:0/2215394510 learned_addr learned my addr 172.21.15.6:0/2215394510 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:29:48.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.143+0000 7f9063e00640 1 --2- 172.21.15.6:0/2215394510 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f906c071090 0x7f906c0ae280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:48.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.143+0000 7f9069e00640 1 --2- 172.21.15.6:0/2215394510 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f906c06e760 0x7f906c14b5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:48.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.143+0000 7f906a800640 1 -- 172.21.15.6:0/2215394510 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f906c071090 msgr2=0x7f906c0ae280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:48.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.143+0000 7f906a800640 1 --2- 172.21.15.6:0/2215394510 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f906c071090 0x7f906c0ae280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:48.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.143+0000 7f906a800640 1 -- 172.21.15.6:0/2215394510 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f906c06e760 msgr2=0x7f906c14b5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:48.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.143+0000 7f906a800640 1 --2- 172.21.15.6:0/2215394510 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f906c06e760 0x7f906c14b5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:48.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.143+0000 7f906a800640 1 -- 172.21.15.6:0/2215394510 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f906c149a40 con 0x7f906c15b290 2024-03-04T09:29:48.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.143+0000 7f9069e00640 1 --2- 172.21.15.6:0/2215394510 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f906c06e760 0x7f906c14b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:29:48.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.143+0000 7f906a800640 1 --2- 172.21.15.6:0/2215394510 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f906c15b290 0x7f906c146550 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f906400c7c0 tx=0x7f906400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:48.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.143+0000 7f9062000640 1 -- 172.21.15.6:0/2215394510 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9064003da0 con 0x7f906c15b290 2024-03-04T09:29:48.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.143+0000 7f9062000640 1 -- 172.21.15.6:0/2215394510 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9064005070 con 0x7f906c15b290 2024-03-04T09:29:48.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.143+0000 7f9062000640 1 -- 172.21.15.6:0/2215394510 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9064005460 con 0x7f906c15b290 2024-03-04T09:29:48.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.144+0000 7f9071000640 1 -- 172.21.15.6:0/2215394510 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f906c149d20 con 0x7f906c15b290 2024-03-04T09:29:48.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.144+0000 7f9071000640 1 -- 172.21.15.6:0/2215394510 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f906c14a2e0 con 0x7f906c15b290 2024-03-04T09:29:48.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.145+0000 7f9071000640 1 -- 172.21.15.6:0/2215394510 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f906c04f3a0 con 0x7f906c15b290 2024-03-04T09:29:48.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.146+0000 7f9062000640 1 -- 172.21.15.6:0/2215394510 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9064005600 con 0x7f906c15b290 2024-03-04T09:29:48.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.147+0000 7f9062000640 1 --2- 172.21.15.6:0/2215394510 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f904806b520 0x7f904806d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:48.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.147+0000 7f9062000640 1 -- 172.21.15.6:0/2215394510 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f906408a060 con 0x7f906c15b290 2024-03-04T09:29:48.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.147+0000 7f9069e00640 1 --2- 172.21.15.6:0/2215394510 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f904806b520 0x7f904806d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:48.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.149+0000 7f9069e00640 1 --2- 172.21.15.6:0/2215394510 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f904806b520 0x7f904806d9e0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f905c036000 tx=0x7f905c03b040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:48.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.151+0000 7f9062000640 1 -- 172.21.15.6:0/2215394510 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f906c073360 con 0x7f906c15b290 2024-03-04T09:29:48.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.351+0000 7f9071000640 1 -- 172.21.15.6:0/2215394510 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f906c0b21e0 con 0x7f906c15b290 2024-03-04T09:29:48.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.354+0000 7f9062000640 1 -- 172.21.15.6:0/2215394510 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v80) v1 ==== 74+0+16625 (secure 0 0 0) 0x7f9064059050 con 0x7f906c15b290 2024-03-04T09:29:48.351 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:29:48.352 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":80,"fsid":"edce0578-5ae4-4402-b45a-552bdef84d7c","created":"2024-03-04T09:24:54.523780+0000","modified":"2024-03-04T09:29:47.356843+0000","last_up_change":"2024-03-04T09:25:07.089049+0000","last_in_change":"2024-03-04T09:24:57.982077+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T09:25:07.935198+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-03-04T09:29:45.280632+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-03-04T09:29:46.378588+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a96574f-81f1-4bc6-8463-757520b84a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6825","nonce":1364449401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6827","nonce":1364449401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6831","nonce":1364449401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6829","nonce":1364449401}]},"public_addr":"172.21.15.6:6825/1364449401","cluster_addr":"172.21.15.6:6827/1364449401","heartbeat_back_addr":"172.21.15.6:6831/1364449401","heartbeat_front_addr":"172.21.15.6:6829/1364449401","state":["exists","up"]},{"osd":1,"uuid":"7d79b9c1-6b87-476a-bc36-7db5ee64cd90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6809","nonce":2494103078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6811","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6813","nonce":2494103078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6821","nonce":2494103078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6817","nonce":2494103078}]},"public_addr":"172.21.15.6:6809/2494103078","cluster_addr":"172.21.15.6:6813/2494103078","heartbeat_back_addr":"172.21.15.6:6821/2494103078","heartbeat_front_addr":"172.21.15.6:6817/2494103078","state":["exists","up"]},{"osd":2,"uuid":"492331b9-4ee9-40b6-bacd-85d44cd6c45a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6812","nonce":1500066681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6816","nonce":1500066681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6824","nonce":1500066681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6820","nonce":1500066681}]},"public_addr":"172.21.15.6:6812/1500066681","cluster_addr":"172.21.15.6:6816/1500066681","heartbeat_back_addr":"172.21.15.6:6824/1500066681","heartbeat_front_addr":"172.21.15.6:6820/1500066681","state":["exists","up"]},{"osd":3,"uuid":"21bb2efe-9ba7-4bdc-bc31-180cd1102e21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6801","nonce":1558806123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6803","nonce":1558806123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6807","nonce":1558806123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6805","nonce":1558806123}]},"public_addr":"172.21.15.6:6801/1558806123","cluster_addr":"172.21.15.6:6803/1558806123","heartbeat_back_addr":"172.21.15.6:6807/1558806123","heartbeat_front_addr":"172.21.15.6:6805/1558806123","state":["exists","up"]},{"osd":4,"uuid":"6e2c4b71-19d9-4737-83d9-558705d8644a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6801","nonce":1526135780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6803","nonce":1526135780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6807","nonce":1526135780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6805","nonce":1526135780}]},"public_addr":"172.21.15.146:6801/1526135780","cluster_addr":"172.21.15.146:6803/1526135780","heartbeat_back_addr":"172.21.15.146:6807/1526135780","heartbeat_front_addr":"172.21.15.146:6805/1526135780","state":["exists","up"]},{"osd":5,"uuid":"fb69c525-f1be-4b7d-ab32-6c29b0d6fddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6809","nonce":460262507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6811","nonce":460262507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6815","nonce":460262507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6813","nonce":460262507}]},"public_addr":"172.21.15.146:6809/460262507","cluster_addr":"172.21.15.146:6811/460262507","heartbeat_back_addr":"172.21.15.146:6815/460262507","heartbeat_front_addr":"172.21.15.146:6813/460262507","state":["exists","up"]},{"osd":6,"uuid":"c32e7fed-370d-4401-8ce0-6a4335b0d168","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6823","nonce":1392897493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6827","nonce":1392897493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6831","nonce":1392897493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6829","nonce":1392897493}]},"public_addr":"172.21.15.146:6823/1392897493","cluster_addr":"172.21.15.146:6827/1392897493","heartbeat_back_addr":"172.21.15.146:6831/1392897493","heartbeat_front_addr":"172.21.15.146:6829/1392897493","state":["exists","up"]},{"osd":7,"uuid":"75a79b44-fa5f-426d-a7d3-71f21f78835c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6817","nonce":3268323156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6819","nonce":3268323156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6825","nonce":3268323156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6822","nonce":3268323156}]},"public_addr":"172.21.15.146:6817/3268323156","cluster_addr":"172.21.15.146:6819/3268323156","heartbeat_back_addr":"172.21.15.146:6825/3268323156","heartbeat_front_addr":"172.21.15.146:6822/3268323156","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.310437+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.295850+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.294674+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.526855+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.300023+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.595160+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.346046+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.558741+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T09:29:48.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.356+0000 7f9071000640 1 -- 172.21.15.6:0/2215394510 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f904806b520 msgr2=0x7f904806d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:48.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.356+0000 7f9071000640 1 --2- 172.21.15.6:0/2215394510 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f904806b520 0x7f904806d9e0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f905c036000 tx=0x7f905c03b040 comp rx=0 tx=0).stop 2024-03-04T09:29:48.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.356+0000 7f9071000640 1 -- 172.21.15.6:0/2215394510 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f906c15b290 msgr2=0x7f906c146550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:48.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.356+0000 7f9071000640 1 --2- 172.21.15.6:0/2215394510 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f906c15b290 0x7f906c146550 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f906400c7c0 tx=0x7f906400cc80 comp rx=0 tx=0).stop 2024-03-04T09:29:48.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.356+0000 7f9071000640 1 -- 172.21.15.6:0/2215394510 shutdown_connections 2024-03-04T09:29:48.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.356+0000 7f9071000640 1 --2- 172.21.15.6:0/2215394510 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f906c15b290 0x7f906c146550 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:48.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.356+0000 7f9071000640 1 --2- 172.21.15.6:0/2215394510 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f904806b520 0x7f904806d9e0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:48.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.356+0000 7f9071000640 1 --2- 172.21.15.6:0/2215394510 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f906c071090 0x7f906c0ae280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:48.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.356+0000 7f9071000640 1 --2- 172.21.15.6:0/2215394510 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f906c06e760 0x7f906c14b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:48.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.356+0000 7f9071000640 1 -- 172.21.15.6:0/2215394510 >> 172.21.15.6:0/2215394510 conn(0x7f906c0b1bb0 msgr2=0x7f906c14fef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:29:48.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.356+0000 7f9071000640 1 -- 172.21.15.6:0/2215394510 shutdown_connections 2024-03-04T09:29:48.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.356+0000 7f9071000640 1 -- 172.21.15.6:0/2215394510 wait complete. 2024-03-04T09:29:48.372 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T09:29:48.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.589+0000 7f5aaa000640 1 Processor -- start 2024-03-04T09:29:48.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.590+0000 7f5aaa000640 1 -- start start 2024-03-04T09:29:48.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.591+0000 7f5aaa000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f5aa415e0b0 0x7f5aa40cf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:48.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.591+0000 7f5aaa000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f5aa40d0e50 0x7f5aa40b44d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:48.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.591+0000 7f5aaa000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5aa40b3710 con 0x7f5aa40d4a10 2024-03-04T09:29:48.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.591+0000 7f5aaa000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5aa40b09e0 con 0x7f5aa40d0e50 2024-03-04T09:29:48.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.591+0000 7f5aaa000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5aa40b39f0 con 0x7f5aa415e0b0 2024-03-04T09:29:48.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.592+0000 7f5aa3e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f5aa40d4a10 0x7f5aa40d0740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:59502/0 (socket says 172.21.15.6:59502) 2024-03-04T09:29:48.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.592+0000 7f5aa3e00640 1 -- 172.21.15.6:0/4273938232 learned_addr learned my addr 172.21.15.6:0/4273938232 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:29:48.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.592+0000 7f5aa9600640 1 --2- 172.21.15.6:0/4273938232 >> v2:172.21.15.6:3300/0 conn(0x7f5aa40d0e50 0x7f5aa40b44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:48.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.592+0000 7f5aa8c00640 1 --2- 172.21.15.6:0/4273938232 >> v2:172.21.15.146:3300/0 conn(0x7f5aa415e0b0 0x7f5aa40cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:48.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.592+0000 7f5aa3400640 1 -- 172.21.15.6:0/4273938232 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2866804112 0 0) 0x7f5aa40b3710 con 0x7f5aa40d4a10 2024-03-04T09:29:48.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.592+0000 7f5aa3400640 1 -- 172.21.15.6:0/4273938232 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a84003520 con 0x7f5aa40d4a10 2024-03-04T09:29:48.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.592+0000 7f5aa9600640 1 -- 172.21.15.6:0/4273938232 >> v2:172.21.15.146:3300/0 conn(0x7f5aa415e0b0 msgr2=0x7f5aa40cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:48.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.592+0000 7f5aa9600640 1 --2- 172.21.15.6:0/4273938232 >> v2:172.21.15.146:3300/0 conn(0x7f5aa415e0b0 0x7f5aa40cf310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:48.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.592+0000 7f5aa9600640 1 -- 172.21.15.6:0/4273938232 >> v1:172.21.15.6:6789/0 conn(0x7f5aa40d4a10 legacy=0x7f5aa40d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:48.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.592+0000 7f5aa9600640 1 -- 172.21.15.6:0/4273938232 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5aa40b3cd0 con 0x7f5aa40d0e50 2024-03-04T09:29:48.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.593+0000 7f5aa9600640 1 --2- 172.21.15.6:0/4273938232 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5aa40d0e50 0x7f5aa40b44d0 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f5aa40b68c0 tx=0x7f5a9c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0bea59e05df7257 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:48.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.594+0000 7f5aa3400640 1 -- 172.21.15.6:0/4273938232 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a9c035020 con 0x7f5aa40d0e50 2024-03-04T09:29:48.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.594+0000 7f5aa3400640 1 -- 172.21.15.6:0/4273938232 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a9c031830 con 0x7f5aa40d0e50 2024-03-04T09:29:48.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.594+0000 7f5aa3400640 1 -- 172.21.15.6:0/4273938232 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a9c031c00 con 0x7f5aa40d0e50 2024-03-04T09:29:48.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.594+0000 7f5aaa000640 1 -- 172.21.15.6:0/4273938232 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5aa40d0e50 msgr2=0x7f5aa40b44d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:48.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.594+0000 7f5aaa000640 1 --2- 172.21.15.6:0/4273938232 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5aa40d0e50 0x7f5aa40b44d0 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f5aa40b68c0 tx=0x7f5a9c02b8d0 comp rx=0 tx=0).stop 2024-03-04T09:29:48.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.594+0000 7f5aaa000640 1 -- 172.21.15.6:0/4273938232 shutdown_connections 2024-03-04T09:29:48.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.594+0000 7f5aaa000640 1 --2- 172.21.15.6:0/4273938232 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5aa40d0e50 0x7f5aa40b44d0 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:48.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.594+0000 7f5aaa000640 1 --2- 172.21.15.6:0/4273938232 >> v2:172.21.15.146:3300/0 conn(0x7f5aa415e0b0 0x7f5aa40cf310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:48.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.594+0000 7f5aaa000640 1 -- 172.21.15.6:0/4273938232 >> 172.21.15.6:0/4273938232 conn(0x7f5aa40d3d30 msgr2=0x7f5aa40d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:29:48.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.595+0000 7f5aaa000640 1 -- 172.21.15.6:0/4273938232 shutdown_connections 2024-03-04T09:29:48.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.595+0000 7f5aaa000640 1 -- 172.21.15.6:0/4273938232 wait complete. 2024-03-04T09:29:48.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.596+0000 7f5aaa000640 1 Processor -- start 2024-03-04T09:29:48.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.597+0000 7f5aaa000640 1 -- start start 2024-03-04T09:29:48.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.598+0000 7f5aaa000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5aa40d0e50 0x7f5aa415bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:48.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.598+0000 7f5aaa000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5aa40d4a10 0x7f5aa4160d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:48.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.598+0000 7f5aaa000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5aa415e0b0 0x7f5aa40b8120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:48.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.598+0000 7f5aaa000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5aa40cfc60 con 0x7f5aa415e0b0 2024-03-04T09:29:48.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.598+0000 7f5aaa000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5aa415e490 con 0x7f5aa40d4a10 2024-03-04T09:29:48.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.598+0000 7f5aaa000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5aa40cf850 con 0x7f5aa40d0e50 2024-03-04T09:29:48.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.598+0000 7f5aa9600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5aa415e0b0 0x7f5aa40b8120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:48.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.598+0000 7f5aa9600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5aa415e0b0 0x7f5aa40b8120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38932/0 (socket says 172.21.15.6:38932) 2024-03-04T09:29:48.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.598+0000 7f5aa9600640 1 -- 172.21.15.6:0/2155043041 learned_addr learned my addr 172.21.15.6:0/2155043041 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:29:48.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.598+0000 7f5aa8c00640 1 --2- 172.21.15.6:0/2155043041 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5aa40d0e50 0x7f5aa415bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:48.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.598+0000 7f5aa3e00640 1 --2- 172.21.15.6:0/2155043041 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5aa40d4a10 0x7f5aa4160d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:48.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.598+0000 7f5aa9600640 1 -- 172.21.15.6:0/2155043041 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5aa40d0e50 msgr2=0x7f5aa415bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:48.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.598+0000 7f5aa9600640 1 --2- 172.21.15.6:0/2155043041 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5aa40d0e50 0x7f5aa415bfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:48.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.598+0000 7f5aa9600640 1 -- 172.21.15.6:0/2155043041 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5aa40d4a10 msgr2=0x7f5aa4160d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:48.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.598+0000 7f5aa9600640 1 --2- 172.21.15.6:0/2155043041 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5aa40d4a10 0x7f5aa4160d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:48.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.598+0000 7f5aa9600640 1 -- 172.21.15.6:0/2155043041 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5aa4160960 con 0x7f5aa415e0b0 2024-03-04T09:29:48.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.599+0000 7f5aa9600640 1 --2- 172.21.15.6:0/2155043041 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5aa415e0b0 0x7f5aa40b8120 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f5a9c02bee0 tx=0x7f5a9c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:48.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.599+0000 7f5aa0c00640 1 -- 172.21.15.6:0/2155043041 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a9c035040 con 0x7f5aa415e0b0 2024-03-04T09:29:48.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.599+0000 7f5aa0c00640 1 -- 172.21.15.6:0/2155043041 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a9c03b860 con 0x7f5aa415e0b0 2024-03-04T09:29:48.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.599+0000 7f5aa0c00640 1 -- 172.21.15.6:0/2155043041 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a9c03bc50 con 0x7f5aa415e0b0 2024-03-04T09:29:48.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.599+0000 7f5aaa000640 1 -- 172.21.15.6:0/2155043041 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5aa40b7360 con 0x7f5aa415e0b0 2024-03-04T09:29:48.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.599+0000 7f5aaa000640 1 -- 172.21.15.6:0/2155043041 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5aa40b7920 con 0x7f5aa415e0b0 2024-03-04T09:29:48.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.600+0000 7f5aaa000640 1 -- 172.21.15.6:0/2155043041 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5aa404f330 con 0x7f5aa415e0b0 2024-03-04T09:29:48.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.602+0000 7f5aa0c00640 1 -- 172.21.15.6:0/2155043041 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5a9c038020 con 0x7f5aa415e0b0 2024-03-04T09:29:48.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.602+0000 7f5aa0c00640 1 --2- 172.21.15.6:0/2155043041 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f5a8006b4d0 0x7f5a8006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:48.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.602+0000 7f5aa0c00640 1 -- 172.21.15.6:0/2155043041 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5a9c0abf40 con 0x7f5aa415e0b0 2024-03-04T09:29:48.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.603+0000 7f5aa8c00640 1 --2- 172.21.15.6:0/2155043041 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f5a8006b4d0 0x7f5a8006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:48.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.604+0000 7f5aa8c00640 1 --2- 172.21.15.6:0/2155043041 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f5a8006b4d0 0x7f5a8006d990 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f5a94006fd0 tx=0x7f5a94008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:48.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.607+0000 7f5aa0c00640 1 -- 172.21.15.6:0/2155043041 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5a9c031800 con 0x7f5aa415e0b0 2024-03-04T09:29:48.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.803+0000 7f5aaa000640 1 -- 172.21.15.6:0/2155043041 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5aa4074920 con 0x7f5aa415e0b0 2024-03-04T09:29:48.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.805+0000 7f5aa0c00640 1 -- 172.21.15.6:0/2155043041 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v80) v1 ==== 74+0+16625 (secure 0 0 0) 0x7f5a9c07ad20 con 0x7f5aa415e0b0 2024-03-04T09:29:48.802 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:29:48.802 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":80,"fsid":"edce0578-5ae4-4402-b45a-552bdef84d7c","created":"2024-03-04T09:24:54.523780+0000","modified":"2024-03-04T09:29:47.356843+0000","last_up_change":"2024-03-04T09:25:07.089049+0000","last_in_change":"2024-03-04T09:24:57.982077+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T09:25:07.935198+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-03-04T09:29:45.280632+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-03-04T09:29:46.378588+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a96574f-81f1-4bc6-8463-757520b84a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6825","nonce":1364449401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6827","nonce":1364449401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6831","nonce":1364449401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6829","nonce":1364449401}]},"public_addr":"172.21.15.6:6825/1364449401","cluster_addr":"172.21.15.6:6827/1364449401","heartbeat_back_addr":"172.21.15.6:6831/1364449401","heartbeat_front_addr":"172.21.15.6:6829/1364449401","state":["exists","up"]},{"osd":1,"uuid":"7d79b9c1-6b87-476a-bc36-7db5ee64cd90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6809","nonce":2494103078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6811","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6813","nonce":2494103078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6821","nonce":2494103078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6817","nonce":2494103078}]},"public_addr":"172.21.15.6:6809/2494103078","cluster_addr":"172.21.15.6:6813/2494103078","heartbeat_back_addr":"172.21.15.6:6821/2494103078","heartbeat_front_addr":"172.21.15.6:6817/2494103078","state":["exists","up"]},{"osd":2,"uuid":"492331b9-4ee9-40b6-bacd-85d44cd6c45a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6812","nonce":1500066681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6816","nonce":1500066681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6824","nonce":1500066681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6820","nonce":1500066681}]},"public_addr":"172.21.15.6:6812/1500066681","cluster_addr":"172.21.15.6:6816/1500066681","heartbeat_back_addr":"172.21.15.6:6824/1500066681","heartbeat_front_addr":"172.21.15.6:6820/1500066681","state":["exists","up"]},{"osd":3,"uuid":"21bb2efe-9ba7-4bdc-bc31-180cd1102e21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6801","nonce":1558806123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6803","nonce":1558806123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6807","nonce":1558806123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6805","nonce":1558806123}]},"public_addr":"172.21.15.6:6801/1558806123","cluster_addr":"172.21.15.6:6803/1558806123","heartbeat_back_addr":"172.21.15.6:6807/1558806123","heartbeat_front_addr":"172.21.15.6:6805/1558806123","state":["exists","up"]},{"osd":4,"uuid":"6e2c4b71-19d9-4737-83d9-558705d8644a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6801","nonce":1526135780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6803","nonce":1526135780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6807","nonce":1526135780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6805","nonce":1526135780}]},"public_addr":"172.21.15.146:6801/1526135780","cluster_addr":"172.21.15.146:6803/1526135780","heartbeat_back_addr":"172.21.15.146:6807/1526135780","heartbeat_front_addr":"172.21.15.146:6805/1526135780","state":["exists","up"]},{"osd":5,"uuid":"fb69c525-f1be-4b7d-ab32-6c29b0d6fddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6809","nonce":460262507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6811","nonce":460262507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6815","nonce":460262507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6813","nonce":460262507}]},"public_addr":"172.21.15.146:6809/460262507","cluster_addr":"172.21.15.146:6811/460262507","heartbeat_back_addr":"172.21.15.146:6815/460262507","heartbeat_front_addr":"172.21.15.146:6813/460262507","state":["exists","up"]},{"osd":6,"uuid":"c32e7fed-370d-4401-8ce0-6a4335b0d168","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6823","nonce":1392897493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6827","nonce":1392897493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6831","nonce":1392897493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6829","nonce":1392897493}]},"public_addr":"172.21.15.146:6823/1392897493","cluster_addr":"172.21.15.146:6827/1392897493","heartbeat_back_addr":"172.21.15.146:6831/1392897493","heartbeat_front_addr":"172.21.15.146:6829/1392897493","state":["exists","up"]},{"osd":7,"uuid":"75a79b44-fa5f-426d-a7d3-71f21f78835c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6817","nonce":3268323156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6819","nonce":3268323156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6825","nonce":3268323156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6822","nonce":3268323156}]},"public_addr":"172.21.15.146:6817/3268323156","cluster_addr":"172.21.15.146:6819/3268323156","heartbeat_back_addr":"172.21.15.146:6825/3268323156","heartbeat_front_addr":"172.21.15.146:6822/3268323156","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.310437+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.295850+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.294674+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.526855+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.300023+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.595160+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.346046+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.558741+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T09:29:48.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.807+0000 7f5aaa000640 1 -- 172.21.15.6:0/2155043041 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f5a8006b4d0 msgr2=0x7f5a8006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:48.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.807+0000 7f5aaa000640 1 --2- 172.21.15.6:0/2155043041 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f5a8006b4d0 0x7f5a8006d990 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f5a94006fd0 tx=0x7f5a94008040 comp rx=0 tx=0).stop 2024-03-04T09:29:48.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.807+0000 7f5aaa000640 1 -- 172.21.15.6:0/2155043041 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5aa415e0b0 msgr2=0x7f5aa40b8120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:48.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.807+0000 7f5aaa000640 1 --2- 172.21.15.6:0/2155043041 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5aa415e0b0 0x7f5aa40b8120 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f5a9c02bee0 tx=0x7f5a9c0046c0 comp rx=0 tx=0).stop 2024-03-04T09:29:48.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.807+0000 7f5aaa000640 1 -- 172.21.15.6:0/2155043041 shutdown_connections 2024-03-04T09:29:48.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.807+0000 7f5aaa000640 1 --2- 172.21.15.6:0/2155043041 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5aa415e0b0 0x7f5aa40b8120 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:48.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.807+0000 7f5aaa000640 1 --2- 172.21.15.6:0/2155043041 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5aa40d4a10 0x7f5aa4160d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:48.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.807+0000 7f5aaa000640 1 --2- 172.21.15.6:0/2155043041 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f5a8006b4d0 0x7f5a8006d990 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:48.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.807+0000 7f5aaa000640 1 --2- 172.21.15.6:0/2155043041 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5aa40d0e50 0x7f5aa415bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:48.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.807+0000 7f5aaa000640 1 -- 172.21.15.6:0/2155043041 >> 172.21.15.6:0/2155043041 conn(0x7f5aa40d3d30 msgr2=0x7f5aa40cfde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:29:48.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.807+0000 7f5aaa000640 1 -- 172.21.15.6:0/2155043041 shutdown_connections 2024-03-04T09:29:48.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:48.807+0000 7f5aaa000640 1 -- 172.21.15.6:0/2155043041 wait complete. 2024-03-04T09:29:48.824 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-04T09:29:49.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:49.045+0000 7fed66c00640 1 Processor -- start 2024-03-04T09:29:49.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:49.045+0000 7fed66c00640 1 -- start start 2024-03-04T09:29:49.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:49.046+0000 7fed66c00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fed600d07d0 0x7fed600b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:49.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:49.046+0000 7fed66c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fed600d4930 0x7fed6015dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:49.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:49.046+0000 7fed66c00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed600cf780 con 0x7fed600d0bb0 2024-03-04T09:29:49.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:49.046+0000 7fed66c00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed600b3770 con 0x7fed600d4930 2024-03-04T09:29:49.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:49.046+0000 7fed66c00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed600cfa60 con 0x7fed600d07d0 2024-03-04T09:29:49.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:49.047+0000 7fed5f400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fed600d4930 0x7fed6015dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:49.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:49.047+0000 7fed5f400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fed600d4930 0x7fed6015dfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38628/0 (socket says 172.21.15.6:38628) 2024-03-04T09:29:49.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:49.047+0000 7fed5f400640 1 -- 172.21.15.6:0/1065352370 learned_addr learned my addr 172.21.15.6:0/1065352370 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:29:49.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:49.047+0000 7fed5fe00640 1 --2- 172.21.15.6:0/1065352370 >> v2:172.21.15.146:3300/0 conn(0x7fed600d07d0 0x7fed600b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:49.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:49.047+0000 7fed5f400640 1 -- 172.21.15.6:0/1065352370 >> v2:172.21.15.146:3300/0 conn(0x7fed600d07d0 msgr2=0x7fed600b4f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:49.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:49.047+0000 7fed5f400640 1 --2- 172.21.15.6:0/1065352370 >> v2:172.21.15.146:3300/0 conn(0x7fed600d07d0 0x7fed600b4f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:49.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:49.047+0000 7fed5f400640 1 -- 172.21.15.6:0/1065352370 >> v1:172.21.15.146:6789/0 conn(0x7fed600d0bb0 legacy=0x7fed600d1bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:49.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:49.047+0000 7fed5f400640 1 -- 172.21.15.6:0/1065352370 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed600cfd40 con 0x7fed600d4930 2024-03-04T09:29:49.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:49.050+0000 7fed5f400640 1 --2- 172.21.15.6:0/1065352370 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fed600d4930 0x7fed6015dfb0 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7fed600b69a0 tx=0x7fed4c02ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f306006b51f6ecfe server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:49.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:49.050+0000 7fed5ea00640 1 -- 172.21.15.6:0/1065352370 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed4c02f850 con 0x7fed600d4930 2024-03-04T09:29:49.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:49.050+0000 7fed5ea00640 1 -- 172.21.15.6:0/1065352370 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed4c02f9f0 con 0x7fed600d4930 2024-03-04T09:29:49.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:49.050+0000 7fed5ea00640 1 -- 172.21.15.6:0/1065352370 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed4c02fdc0 con 0x7fed600d4930 2024-03-04T09:29:49.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:49.051+0000 7fed66c00640 1 -- 172.21.15.6:0/1065352370 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fed600d4930 msgr2=0x7fed6015dfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:49.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:49.051+0000 7fed66c00640 1 --2- 172.21.15.6:0/1065352370 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fed600d4930 0x7fed6015dfb0 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7fed600b69a0 tx=0x7fed4c02ba30 comp rx=0 tx=0).stop 2024-03-04T09:29:49.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:49.051+0000 7fed66c00640 1 -- 172.21.15.6:0/1065352370 shutdown_connections 2024-03-04T09:29:49.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:49.051+0000 7fed66c00640 1 --2- 172.21.15.6:0/1065352370 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fed600d4930 0x7fed6015dfb0 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:49.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:49.051+0000 7fed66c00640 1 --2- 172.21.15.6:0/1065352370 >> v2:172.21.15.146:3300/0 conn(0x7fed600d07d0 0x7fed600b4f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:49.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:49.051+0000 7fed66c00640 1 -- 172.21.15.6:0/1065352370 >> 172.21.15.6:0/1065352370 conn(0x7fed600d3e10 msgr2=0x7fed600d4220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:29:49.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:49.051+0000 7fed66c00640 1 -- 172.21.15.6:0/1065352370 shutdown_connections 2024-03-04T09:29:49.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:49.051+0000 7fed66c00640 1 -- 172.21.15.6:0/1065352370 wait complete. 2024-03-04T09:29:49.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:49.053+0000 7fed66c00640 1 Processor -- start 2024-03-04T09:29:49.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:49.053+0000 7fed66c00640 1 -- start start 2024-03-04T09:29:49.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:49.054+0000 7fed66c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fed600d07d0 0x7fed601f21a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:49.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:49.054+0000 7fed66c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fed600d0bb0 0x7fed600b6970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:49.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:49.054+0000 7fed66c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fed600d4930 0x7fed60160140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:49.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:49.054+0000 7fed66c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed600b4d90 con 0x7fed600d07d0 2024-03-04T09:29:49.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:49.054+0000 7fed66c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed600b4c10 con 0x7fed600d0bb0 2024-03-04T09:29:49.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:49.054+0000 7fed66c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed6015e4f0 con 0x7fed600d4930 2024-03-04T09:29:49.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:49.054+0000 7fed5fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fed600d07d0 0x7fed601f21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:49.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:49.054+0000 7fed5fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fed600d07d0 0x7fed601f21a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38642/0 (socket says 172.21.15.6:38642) 2024-03-04T09:29:49.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:49.054+0000 7fed5fe00640 1 -- 172.21.15.6:0/4230269363 learned_addr learned my addr 172.21.15.6:0/4230269363 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:29:49.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:49.054+0000 7fed5fe00640 1 -- 172.21.15.6:0/4230269363 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fed600d4930 msgr2=0x7fed60160140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:49.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:49.054+0000 7fed5fe00640 1 --2- 172.21.15.6:0/4230269363 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fed600d4930 0x7fed60160140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:49.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:49.055+0000 7fed5fe00640 1 -- 172.21.15.6:0/4230269363 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fed600d0bb0 msgr2=0x7fed600b6970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:49.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:49.055+0000 7fed5fe00640 1 --2- 172.21.15.6:0/4230269363 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fed600d0bb0 0x7fed600b6970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:49.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:49.055+0000 7fed5fe00640 1 -- 172.21.15.6:0/4230269363 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed600b7330 con 0x7fed600d07d0 2024-03-04T09:29:49.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:49.055+0000 7fed5fe00640 1 --2- 172.21.15.6:0/4230269363 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fed600d07d0 0x7fed601f21a0 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7fed5000d800 tx=0x7fed5000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:49.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:49.055+0000 7fed57e00640 1 -- 172.21.15.6:0/4230269363 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed5000b6c0 con 0x7fed600d07d0 2024-03-04T09:29:49.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:49.055+0000 7fed57e00640 1 -- 172.21.15.6:0/4230269363 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed50019070 con 0x7fed600d07d0 2024-03-04T09:29:49.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:49.055+0000 7fed66c00640 1 -- 172.21.15.6:0/4230269363 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed600b7610 con 0x7fed600d07d0 2024-03-04T09:29:49.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:49.055+0000 7fed57e00640 1 -- 172.21.15.6:0/4230269363 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed5000ba60 con 0x7fed600d07d0 2024-03-04T09:29:49.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:49.055+0000 7fed66c00640 1 -- 172.21.15.6:0/4230269363 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed6015f940 con 0x7fed600d07d0 2024-03-04T09:29:49.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:49.057+0000 7fed66c00640 1 -- 172.21.15.6:0/4230269363 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed60068a50 con 0x7fed600d07d0 2024-03-04T09:29:49.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:49.057+0000 7fed57e00640 1 -- 172.21.15.6:0/4230269363 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fed5000bc00 con 0x7fed600d07d0 2024-03-04T09:29:49.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:49.059+0000 7fed57e00640 1 --2- 172.21.15.6:0/4230269363 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fed4006b4d0 0x7fed4006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:49.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:49.059+0000 7fed57e00640 1 -- 172.21.15.6:0/4230269363 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fed5008a780 con 0x7fed600d07d0 2024-03-04T09:29:49.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:49.060+0000 7fed5f400640 1 --2- 172.21.15.6:0/4230269363 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fed4006b4d0 0x7fed4006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:49.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:49.061+0000 7fed5f400640 1 --2- 172.21.15.6:0/4230269363 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fed4006b4d0 0x7fed4006d990 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fed4c036000 tx=0x7fed4c039040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:49.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:49.063+0000 7fed57e00640 1 -- 172.21.15.6:0/4230269363 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fed60068a50 con 0x7fed600d07d0 2024-03-04T09:29:49.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:49.298+0000 7fed66c00640 1 -- 172.21.15.6:0/4230269363 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fed601f26e0 con 0x7fed600d07d0 2024-03-04T09:29:49.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:49.929+0000 7fed57e00640 1 -- 172.21.15.6:0/4230269363 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v50) v1 ==== 118+0+0 (secure 0 0 0) 0x7fed50011340 con 0x7fed600d07d0 2024-03-04T09:29:49.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:49.934+0000 7fed66c00640 1 -- 172.21.15.6:0/4230269363 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fed4006b4d0 msgr2=0x7fed4006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:49.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:49.934+0000 7fed66c00640 1 --2- 172.21.15.6:0/4230269363 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fed4006b4d0 0x7fed4006d990 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fed4c036000 tx=0x7fed4c039040 comp rx=0 tx=0).stop 2024-03-04T09:29:49.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:49.934+0000 7fed66c00640 1 -- 172.21.15.6:0/4230269363 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fed600d07d0 msgr2=0x7fed601f21a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:49.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:49.934+0000 7fed66c00640 1 --2- 172.21.15.6:0/4230269363 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fed600d07d0 0x7fed601f21a0 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7fed5000d800 tx=0x7fed5000dcc0 comp rx=0 tx=0).stop 2024-03-04T09:29:49.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:49.934+0000 7fed66c00640 1 -- 172.21.15.6:0/4230269363 shutdown_connections 2024-03-04T09:29:49.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:49.934+0000 7fed66c00640 1 --2- 172.21.15.6:0/4230269363 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fed4006b4d0 0x7fed4006d990 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:49.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:49.934+0000 7fed66c00640 1 --2- 172.21.15.6:0/4230269363 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fed600d4930 0x7fed60160140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:49.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:49.934+0000 7fed66c00640 1 --2- 172.21.15.6:0/4230269363 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fed600d0bb0 0x7fed600b6970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:49.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:49.934+0000 7fed66c00640 1 --2- 172.21.15.6:0/4230269363 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fed600d07d0 0x7fed601f21a0 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:49.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:49.934+0000 7fed66c00640 1 -- 172.21.15.6:0/4230269363 >> 172.21.15.6:0/4230269363 conn(0x7fed600d3e10 msgr2=0x7fed600b1080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:29:49.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:49.934+0000 7fed66c00640 1 -- 172.21.15.6:0/4230269363 shutdown_connections 2024-03-04T09:29:49.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:49.934+0000 7fed66c00640 1 -- 172.21.15.6:0/4230269363 wait complete. 2024-03-04T09:29:49.953 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-04T09:29:49.953 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-04T09:29:50.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:50.206+0000 7f109e600640 1 Processor -- start 2024-03-04T09:29:50.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:50.206+0000 7f109e600640 1 -- start start 2024-03-04T09:29:50.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:50.207+0000 7f109e600640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f10980d4930 0x7f109815dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:50.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:50.207+0000 7f109e600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f10980d0bb0 0x7f109815e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:50.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:50.207+0000 7f109e600640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10980cf780 con 0x7f10980d07d0 2024-03-04T09:29:50.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:50.207+0000 7f109e600640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10980b3770 con 0x7f10980d4930 2024-03-04T09:29:50.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:50.207+0000 7f109e600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10980cfa60 con 0x7f10980d0bb0 2024-03-04T09:29:50.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:50.208+0000 7f1097400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f10980d07d0 0x7f10980b4f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:41960/0 (socket says 172.21.15.6:41960) 2024-03-04T09:29:50.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:50.208+0000 7f1097400640 1 -- 172.21.15.6:0/3531672099 learned_addr learned my addr 172.21.15.6:0/3531672099 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:29:50.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:50.208+0000 7f1096a00640 1 --2- 172.21.15.6:0/3531672099 >> v2:172.21.15.146:3300/0 conn(0x7f10980d4930 0x7f109815dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:50.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:50.208+0000 7f1097e00640 1 --2- 172.21.15.6:0/3531672099 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f10980d0bb0 0x7f109815e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:50.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:50.208+0000 7f1096000640 1 -- 172.21.15.6:0/3531672099 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1091377236 0 0) 0x7f10980cf780 con 0x7f10980d07d0 2024-03-04T09:29:50.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:50.208+0000 7f1096000640 1 -- 172.21.15.6:0/3531672099 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f107c003520 con 0x7f10980d07d0 2024-03-04T09:29:50.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:50.209+0000 7f1096000640 1 -- 172.21.15.6:0/3531672099 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2687515566 0 0) 0x7f107c003520 con 0x7f10980d07d0 2024-03-04T09:29:50.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:50.209+0000 7f1096000640 1 -- 172.21.15.6:0/3531672099 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f10980d0bb0 msgr2=0x7f109815e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:50.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:50.209+0000 7f1096000640 1 --2- 172.21.15.6:0/3531672099 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f10980d0bb0 0x7f109815e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:50.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:50.209+0000 7f1096000640 1 -- 172.21.15.6:0/3531672099 >> v2:172.21.15.146:3300/0 conn(0x7f10980d4930 msgr2=0x7f109815dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:50.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:50.209+0000 7f1096000640 1 --2- 172.21.15.6:0/3531672099 >> v2:172.21.15.146:3300/0 conn(0x7f10980d4930 0x7f109815dfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:50.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:50.209+0000 7f1096000640 1 -- 172.21.15.6:0/3531672099 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10980cfd40 con 0x7f10980d07d0 2024-03-04T09:29:50.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:50.209+0000 7f1096000640 1 -- 172.21.15.6:0/3531672099 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7f1088001670 con 0x7f10980d07d0 2024-03-04T09:29:50.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:50.209+0000 7f1096000640 1 -- 172.21.15.6:0/3531672099 >> v1:172.21.15.6:6789/0 conn(0x7f10980d07d0 legacy=0x7f10980b4f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:50.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:50.209+0000 7f1096000640 1 --2- 172.21.15.6:0/3531672099 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f10980d0bb0 0x7f107c0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:50.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:50.209+0000 7f1096000640 1 --2- 172.21.15.6:0/3531672099 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f107c003f80 0x7f107c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:50.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:50.209+0000 7f1096000640 1 --2- 172.21.15.6:0/3531672099 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f107c004d40 0x7f107c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:50.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:50.209+0000 7f1096000640 1 -- 172.21.15.6:0/3531672099 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10980b3770 con 0x7f107c003f80 2024-03-04T09:29:50.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:50.209+0000 7f1096000640 1 -- 172.21.15.6:0/3531672099 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10980cfa60 con 0x7f107c004d40 2024-03-04T09:29:50.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:50.209+0000 7f1096000640 1 -- 172.21.15.6:0/3531672099 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f107c00ba40 con 0x7f10980d0bb0 2024-03-04T09:29:50.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:50.209+0000 7f1096a00640 1 --2- 172.21.15.6:0/3531672099 >> v2:172.21.15.146:3300/0 conn(0x7f10980d4930 0x7f109815dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T09:29:50.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:50.209+0000 7f1096a00640 1 --2- 172.21.15.6:0/3531672099 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f107c003f80 0x7f107c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:50.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:50.210+0000 7f1097e00640 1 --2- 172.21.15.6:0/3531672099 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f107c004d40 0x7f107c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:50.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:50.210+0000 7f1097400640 1 --2- 172.21.15.6:0/3531672099 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f10980d0bb0 0x7f107c0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:50.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:50.210+0000 7f1096a00640 1 -- 172.21.15.6:0/3531672099 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f10980d0bb0 msgr2=0x7f107c0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:50.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:50.210+0000 7f1096a00640 1 --2- 172.21.15.6:0/3531672099 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f10980d0bb0 0x7f107c0039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:50.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:50.210+0000 7f1096a00640 1 -- 172.21.15.6:0/3531672099 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f107c004d40 msgr2=0x7f107c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:50.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:50.210+0000 7f1096a00640 1 --2- 172.21.15.6:0/3531672099 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f107c004d40 0x7f107c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:50.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:50.210+0000 7f1096a00640 1 -- 172.21.15.6:0/3531672099 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f107c003520 con 0x7f107c003f80 2024-03-04T09:29:50.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:50.210+0000 7f1097e00640 1 --2- 172.21.15.6:0/3531672099 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f107c004d40 0x7f107c0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T09:29:50.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:50.211+0000 7f1096a00640 1 --2- 172.21.15.6:0/3531672099 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f107c003f80 0x7f107c004800 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f1084009c70 tx=0x7f107c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=daab4379d90c600e server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:50.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:50.211+0000 7f1096000640 1 -- 172.21.15.6:0/3531672099 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1084009840 con 0x7f107c003f80 2024-03-04T09:29:50.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:50.211+0000 7f1096000640 1 -- 172.21.15.6:0/3531672099 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1084004020 con 0x7f107c003f80 2024-03-04T09:29:50.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:50.211+0000 7f1096000640 1 -- 172.21.15.6:0/3531672099 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1084004450 con 0x7f107c003f80 2024-03-04T09:29:50.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:50.212+0000 7f109e600640 1 -- 172.21.15.6:0/3531672099 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f107c003f80 msgr2=0x7f107c004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:50.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:50.212+0000 7f109e600640 1 --2- 172.21.15.6:0/3531672099 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f107c003f80 0x7f107c004800 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f1084009c70 tx=0x7f107c005830 comp rx=0 tx=0).stop 2024-03-04T09:29:50.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:50.212+0000 7f1097e00640 1 -- 172.21.15.6:0/3531672099 reap_dead start 2024-03-04T09:29:50.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:50.212+0000 7f109e600640 1 -- 172.21.15.6:0/3531672099 shutdown_connections 2024-03-04T09:29:50.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:50.212+0000 7f109e600640 1 -- 172.21.15.6:0/3531672099 >> 172.21.15.6:0/3531672099 conn(0x7f10980d3e10 msgr2=0x7f10980d4220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:29:50.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:50.212+0000 7f109e600640 1 -- 172.21.15.6:0/3531672099 shutdown_connections 2024-03-04T09:29:50.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:50.212+0000 7f109e600640 1 -- 172.21.15.6:0/3531672099 wait complete. 2024-03-04T09:29:50.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:50.214+0000 7f109e600640 1 Processor -- start 2024-03-04T09:29:50.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:50.214+0000 7f109e600640 1 -- start start 2024-03-04T09:29:50.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:50.215+0000 7f109e600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f107c003f80 0x7f1098150400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:50.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:50.215+0000 7f109e600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1098150a90 0x7f10980b8660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:50.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:50.215+0000 7f109e600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f10980b8ba0 0x7f10980b73b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:50.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:50.215+0000 7f109e600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f109815e880 con 0x7f1098150a90 2024-03-04T09:29:50.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:50.215+0000 7f109e600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f109815e700 con 0x7f10980b8ba0 2024-03-04T09:29:50.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:50.215+0000 7f109e600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10980b3c00 con 0x7f107c003f80 2024-03-04T09:29:50.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:50.215+0000 7f1096a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1098150a90 0x7f10980b8660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:50.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:50.215+0000 7f1096a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1098150a90 0x7f10980b8660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38654/0 (socket says 172.21.15.6:38654) 2024-03-04T09:29:50.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:50.215+0000 7f1096a00640 1 -- 172.21.15.6:0/2067741206 learned_addr learned my addr 172.21.15.6:0/2067741206 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:29:50.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:50.216+0000 7f1097400640 1 --2- 172.21.15.6:0/2067741206 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f107c003f80 0x7f1098150400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:50.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:50.216+0000 7f1097e00640 1 --2- 172.21.15.6:0/2067741206 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f10980b8ba0 0x7f10980b73b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:50.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:50.216+0000 7f1096a00640 1 -- 172.21.15.6:0/2067741206 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f107c003f80 msgr2=0x7f1098150400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:50.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:50.216+0000 7f1096a00640 1 --2- 172.21.15.6:0/2067741206 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f107c003f80 0x7f1098150400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:50.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:50.216+0000 7f1096a00640 1 -- 172.21.15.6:0/2067741206 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f10980b8ba0 msgr2=0x7f10980b73b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:50.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:50.216+0000 7f1096a00640 1 --2- 172.21.15.6:0/2067741206 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f10980b8ba0 0x7f10980b73b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:50.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:50.216+0000 7f1096a00640 1 -- 172.21.15.6:0/2067741206 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f109815eb30 con 0x7f1098150a90 2024-03-04T09:29:50.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:50.216+0000 7f1096a00640 1 --2- 172.21.15.6:0/2067741206 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1098150a90 0x7f10980b8660 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f1084009d60 tx=0x7f1084004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:50.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:50.217+0000 7f108f400640 1 -- 172.21.15.6:0/2067741206 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1084043020 con 0x7f1098150a90 2024-03-04T09:29:50.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:50.217+0000 7f108f400640 1 -- 172.21.15.6:0/2067741206 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f108402cb70 con 0x7f1098150a90 2024-03-04T09:29:50.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:50.217+0000 7f108f400640 1 -- 172.21.15.6:0/2067741206 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1084006e10 con 0x7f1098150a90 2024-03-04T09:29:50.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:50.217+0000 7f109e600640 1 -- 172.21.15.6:0/2067741206 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f109815ee10 con 0x7f1098150a90 2024-03-04T09:29:50.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:50.217+0000 7f109e600640 1 -- 172.21.15.6:0/2067741206 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1098150f10 con 0x7f1098150a90 2024-03-04T09:29:50.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:50.219+0000 7f109e600640 1 -- 172.21.15.6:0/2067741206 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f109804f330 con 0x7f1098150a90 2024-03-04T09:29:50.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:50.219+0000 7f108f400640 1 -- 172.21.15.6:0/2067741206 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1084046020 con 0x7f1098150a90 2024-03-04T09:29:50.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:50.220+0000 7f108f400640 1 --2- 172.21.15.6:0/2067741206 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f107806b500 0x7f107806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:50.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:50.221+0000 7f108f400640 1 -- 172.21.15.6:0/2067741206 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f10840ac260 con 0x7f1098150a90 2024-03-04T09:29:50.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:50.221+0000 7f1097400640 1 --2- 172.21.15.6:0/2067741206 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f107806b500 0x7f107806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:50.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:50.222+0000 7f1097400640 1 --2- 172.21.15.6:0/2067741206 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f107806b500 0x7f107806d9c0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f1088000c00 tx=0x7f1088009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:50.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:50.225+0000 7f108f400640 1 -- 172.21.15.6:0/2067741206 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f108404d030 con 0x7f1098150a90 2024-03-04T09:29:50.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:50.577+0000 7f109e600640 1 -- 172.21.15.6:0/2067741206 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f10980b8f80 con 0x7f1098150a90 2024-03-04T09:29:50.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:50.932+0000 7f108f400640 1 -- 172.21.15.6:0/2067741206 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v51) v1 ==== 122+0+0 (secure 0 0 0) 0x7f108407b1f0 con 0x7f1098150a90 2024-03-04T09:29:50.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:50.935+0000 7f109e600640 1 -- 172.21.15.6:0/2067741206 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f107806b500 msgr2=0x7f107806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:50.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:50.935+0000 7f109e600640 1 --2- 172.21.15.6:0/2067741206 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f107806b500 0x7f107806d9c0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f1088000c00 tx=0x7f1088009040 comp rx=0 tx=0).stop 2024-03-04T09:29:50.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:50.935+0000 7f109e600640 1 -- 172.21.15.6:0/2067741206 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1098150a90 msgr2=0x7f10980b8660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:50.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:50.935+0000 7f109e600640 1 --2- 172.21.15.6:0/2067741206 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1098150a90 0x7f10980b8660 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f1084009d60 tx=0x7f1084004730 comp rx=0 tx=0).stop 2024-03-04T09:29:50.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:50.936+0000 7f109e600640 1 -- 172.21.15.6:0/2067741206 shutdown_connections 2024-03-04T09:29:50.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:50.936+0000 7f109e600640 1 --2- 172.21.15.6:0/2067741206 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f10980b8ba0 0x7f10980b73b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:50.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:50.936+0000 7f109e600640 1 --2- 172.21.15.6:0/2067741206 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1098150a90 0x7f10980b8660 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:50.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:50.936+0000 7f109e600640 1 --2- 172.21.15.6:0/2067741206 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f107806b500 0x7f107806d9c0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:50.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:50.936+0000 7f109e600640 1 --2- 172.21.15.6:0/2067741206 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f107c003f80 0x7f1098150400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:50.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:50.936+0000 7f109e600640 1 -- 172.21.15.6:0/2067741206 >> 172.21.15.6:0/2067741206 conn(0x7f10980d3e10 msgr2=0x7f109806ced0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:29:50.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:50.936+0000 7f109e600640 1 -- 172.21.15.6:0/2067741206 shutdown_connections 2024-03-04T09:29:50.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:50.936+0000 7f109e600640 1 -- 172.21.15.6:0/2067741206 wait complete. 2024-03-04T09:29:50.955 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-04T09:29:50.956 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:29:51.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.171+0000 7fd2dce00640 1 Processor -- start 2024-03-04T09:29:51.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.171+0000 7fd2dce00640 1 -- start start 2024-03-04T09:29:51.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.173+0000 7fd2dce00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd2d80d0e50 0x7fd2d8168c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:51.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.173+0000 7fd2dce00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2d80b3710 con 0x7fd2d815e0b0 2024-03-04T09:29:51.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.173+0000 7fd2dce00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2d80b39f0 con 0x7fd2d80d4a10 2024-03-04T09:29:51.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.173+0000 7fd2dce00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2d80b09e0 con 0x7fd2d80d0e50 2024-03-04T09:29:51.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.173+0000 7fd2d7400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fd2d815e0b0 0x7fd2d80cf310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:41974/0 (socket says 172.21.15.6:41974) 2024-03-04T09:29:51.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.173+0000 7fd2d7400640 1 -- 172.21.15.6:0/1225919941 learned_addr learned my addr 172.21.15.6:0/1225919941 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:29:51.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.174+0000 7fd2d6a00640 1 -- 172.21.15.6:0/1225919941 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1020320617 0 0) 0x7fd2d80b3710 con 0x7fd2d815e0b0 2024-03-04T09:29:51.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.174+0000 7fd2d6a00640 1 -- 172.21.15.6:0/1225919941 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2bc003520 con 0x7fd2d815e0b0 2024-03-04T09:29:51.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.174+0000 7fd2d7e00640 1 --2- 172.21.15.6:0/1225919941 >> v2:172.21.15.146:3300/0 conn(0x7fd2d80d0e50 0x7fd2d8168c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:51.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.174+0000 7fd2d6a00640 1 -- 172.21.15.6:0/1225919941 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 255861754 0 0) 0x7fd2bc003520 con 0x7fd2d815e0b0 2024-03-04T09:29:51.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.174+0000 7fd2d6a00640 1 -- 172.21.15.6:0/1225919941 >> v2:172.21.15.146:3300/0 conn(0x7fd2d80d0e50 msgr2=0x7fd2d8168c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:51.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.174+0000 7fd2d6a00640 1 --2- 172.21.15.6:0/1225919941 >> v2:172.21.15.146:3300/0 conn(0x7fd2d80d0e50 0x7fd2d8168c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:51.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.174+0000 7fd2d6a00640 1 -- 172.21.15.6:0/1225919941 >> v1:172.21.15.146:6789/0 conn(0x7fd2d80d4a10 legacy=0x7fd2d80d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:51.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.174+0000 7fd2d6a00640 1 -- 172.21.15.6:0/1225919941 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2d80b3cd0 con 0x7fd2d815e0b0 2024-03-04T09:29:51.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.174+0000 7fd2d6a00640 1 -- 172.21.15.6:0/1225919941 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7fd2c80019f0 con 0x7fd2d815e0b0 2024-03-04T09:29:51.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.174+0000 7fd2d6a00640 1 -- 172.21.15.6:0/1225919941 >> v1:172.21.15.6:6789/0 conn(0x7fd2d815e0b0 legacy=0x7fd2d80cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:51.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.174+0000 7fd2d6a00640 1 --2- 172.21.15.6:0/1225919941 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd2bc0039b0 0x7fd2bc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:51.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.174+0000 7fd2d6a00640 1 --2- 172.21.15.6:0/1225919941 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd2bc004800 0x7fd2bc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:51.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.174+0000 7fd2d6a00640 1 --2- 172.21.15.6:0/1225919941 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd2bc0051c0 0x7fd2bc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:51.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.174+0000 7fd2d6a00640 1 -- 172.21.15.6:0/1225919941 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2bc00c0e0 con 0x7fd2bc0039b0 2024-03-04T09:29:51.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.174+0000 7fd2d6a00640 1 -- 172.21.15.6:0/1225919941 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2bc00c290 con 0x7fd2bc004800 2024-03-04T09:29:51.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.174+0000 7fd2d6a00640 1 -- 172.21.15.6:0/1225919941 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2bc00c490 con 0x7fd2bc0051c0 2024-03-04T09:29:51.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.175+0000 7fd2d7400640 1 --2- 172.21.15.6:0/1225919941 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd2bc0039b0 0x7fd2bc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:51.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.175+0000 7fd2cea00640 1 --2- 172.21.15.6:0/1225919941 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd2bc004800 0x7fd2bc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:51.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.175+0000 7fd2d7400640 1 -- 172.21.15.6:0/1225919941 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd2bc0051c0 msgr2=0x7fd2bc00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:51.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.175+0000 7fd2d7400640 1 --2- 172.21.15.6:0/1225919941 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd2bc0051c0 0x7fd2bc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:51.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.175+0000 7fd2d7400640 1 -- 172.21.15.6:0/1225919941 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd2bc004800 msgr2=0x7fd2bc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:51.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.175+0000 7fd2d7400640 1 --2- 172.21.15.6:0/1225919941 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd2bc004800 0x7fd2bc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:51.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.175+0000 7fd2d7400640 1 -- 172.21.15.6:0/1225919941 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2bc003520 con 0x7fd2bc0039b0 2024-03-04T09:29:51.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.175+0000 7fd2cea00640 1 --2- 172.21.15.6:0/1225919941 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd2bc004800 0x7fd2bc004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:29:51.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.175+0000 7fd2d7e00640 1 -- 172.21.15.6:0/1225919941 reap_dead start 2024-03-04T09:29:51.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.176+0000 7fd2d7400640 1 --2- 172.21.15.6:0/1225919941 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd2bc0039b0 0x7fd2bc003dd0 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7fd2c800a860 tx=0x7fd2bc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fef3d96efac47ebd server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:51.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.176+0000 7fd2d6a00640 1 -- 172.21.15.6:0/1225919941 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2c800a480 con 0x7fd2bc0039b0 2024-03-04T09:29:51.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.177+0000 7fd2d6a00640 1 -- 172.21.15.6:0/1225919941 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2c8034630 con 0x7fd2bc0039b0 2024-03-04T09:29:51.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.177+0000 7fd2d6a00640 1 -- 172.21.15.6:0/1225919941 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2c8034a80 con 0x7fd2bc0039b0 2024-03-04T09:29:51.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.177+0000 7fd2dce00640 1 -- 172.21.15.6:0/1225919941 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd2bc0039b0 msgr2=0x7fd2bc003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:51.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.177+0000 7fd2dce00640 1 --2- 172.21.15.6:0/1225919941 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd2bc0039b0 0x7fd2bc003dd0 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7fd2c800a860 tx=0x7fd2bc005830 comp rx=0 tx=0).stop 2024-03-04T09:29:51.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.177+0000 7fd2dce00640 1 -- 172.21.15.6:0/1225919941 shutdown_connections 2024-03-04T09:29:51.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.177+0000 7fd2dce00640 1 --2- 172.21.15.6:0/1225919941 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd2bc0039b0 0x7fd2bc003dd0 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:51.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.177+0000 7fd2dce00640 1 -- 172.21.15.6:0/1225919941 >> 172.21.15.6:0/1225919941 conn(0x7fd2d80d3d30 msgr2=0x7fd2d80d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:29:51.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.178+0000 7fd2dce00640 1 -- 172.21.15.6:0/1225919941 shutdown_connections 2024-03-04T09:29:51.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.178+0000 7fd2dce00640 1 -- 172.21.15.6:0/1225919941 wait complete. 2024-03-04T09:29:51.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.179+0000 7fd2dce00640 1 Processor -- start 2024-03-04T09:29:51.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.179+0000 7fd2dce00640 1 -- start start 2024-03-04T09:29:51.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.180+0000 7fd2dce00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd2bc0039b0 0x7fd2d80cddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:51.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.180+0000 7fd2dce00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd2d80b7cd0 0x7fd2d80b8f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:51.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.180+0000 7fd2dce00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd2d80b8150 0x7fd2d80b94a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:51.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.180+0000 7fd2dce00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2d80cfc60 con 0x7fd2d80b8150 2024-03-04T09:29:51.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.180+0000 7fd2dce00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2d815e490 con 0x7fd2d80b7cd0 2024-03-04T09:29:51.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.180+0000 7fd2dce00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2d80d0740 con 0x7fd2bc0039b0 2024-03-04T09:29:51.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.181+0000 7fd2d7e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd2d80b8150 0x7fd2d80b94a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:51.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.181+0000 7fd2d7e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd2d80b8150 0x7fd2d80b94a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38682/0 (socket says 172.21.15.6:38682) 2024-03-04T09:29:51.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.181+0000 7fd2d7e00640 1 -- 172.21.15.6:0/3062626542 learned_addr learned my addr 172.21.15.6:0/3062626542 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:29:51.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.181+0000 7fd2d7400640 1 --2- 172.21.15.6:0/3062626542 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd2bc0039b0 0x7fd2d80cddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:51.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.181+0000 7fd2cea00640 1 --2- 172.21.15.6:0/3062626542 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd2d80b7cd0 0x7fd2d80b8f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:51.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.181+0000 7fd2d7e00640 1 -- 172.21.15.6:0/3062626542 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd2bc0039b0 msgr2=0x7fd2d80cddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:51.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.181+0000 7fd2d7e00640 1 --2- 172.21.15.6:0/3062626542 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd2bc0039b0 0x7fd2d80cddd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:51.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.181+0000 7fd2d7e00640 1 -- 172.21.15.6:0/3062626542 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd2d80b7cd0 msgr2=0x7fd2d80b8f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:51.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.181+0000 7fd2d7e00640 1 --2- 172.21.15.6:0/3062626542 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd2d80b7cd0 0x7fd2d80b8f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:51.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.181+0000 7fd2d7e00640 1 -- 172.21.15.6:0/3062626542 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2d80b6950 con 0x7fd2d80b8150 2024-03-04T09:29:51.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.182+0000 7fd2d7e00640 1 --2- 172.21.15.6:0/3062626542 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd2d80b8150 0x7fd2d80b94a0 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7fd2d80d3c80 tx=0x7fd2d000d8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:51.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.182+0000 7fd2cfe00640 1 -- 172.21.15.6:0/3062626542 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2d0004020 con 0x7fd2d80b8150 2024-03-04T09:29:51.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.182+0000 7fd2cfe00640 1 -- 172.21.15.6:0/3062626542 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2d00041c0 con 0x7fd2d80b8150 2024-03-04T09:29:51.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.182+0000 7fd2cfe00640 1 -- 172.21.15.6:0/3062626542 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2d00043b0 con 0x7fd2d80b8150 2024-03-04T09:29:51.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.182+0000 7fd2dce00640 1 -- 172.21.15.6:0/3062626542 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2d80b6c30 con 0x7fd2d80b8150 2024-03-04T09:29:51.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.182+0000 7fd2dce00640 1 -- 172.21.15.6:0/3062626542 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2d80b71f0 con 0x7fd2d80b8150 2024-03-04T09:29:51.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.183+0000 7fd2dce00640 1 -- 172.21.15.6:0/3062626542 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2d804f330 con 0x7fd2d80b8150 2024-03-04T09:29:51.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.184+0000 7fd2cfe00640 1 -- 172.21.15.6:0/3062626542 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd2d001a030 con 0x7fd2d80b8150 2024-03-04T09:29:51.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.185+0000 7fd2cfe00640 1 --2- 172.21.15.6:0/3062626542 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd2b406b4d0 0x7fd2b406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:51.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.185+0000 7fd2cfe00640 1 -- 172.21.15.6:0/3062626542 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd2d0089870 con 0x7fd2d80b8150 2024-03-04T09:29:51.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.186+0000 7fd2d7400640 1 --2- 172.21.15.6:0/3062626542 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd2b406b4d0 0x7fd2b406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:51.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.187+0000 7fd2d7400640 1 --2- 172.21.15.6:0/3062626542 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd2b406b4d0 0x7fd2b406d990 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fd2c8034000 tx=0x7fd2c802f000 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:51.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.189+0000 7fd2cfe00640 1 -- 172.21.15.6:0/3062626542 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd2d0021020 con 0x7fd2d80b8150 2024-03-04T09:29:51.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.433+0000 7fd2dce00640 1 -- 172.21.15.6:0/3062626542 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd2d80b2fe0 con 0x7fd2d80b8150 2024-03-04T09:29:51.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.434+0000 7fd2cfe00640 1 -- 172.21.15.6:0/3062626542 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) v1 ==== 94+0+4976 (secure 0 0 0) 0x7fd2d00586e0 con 0x7fd2d80b8150 2024-03-04T09:29:51.431 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:29:51.431 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":51,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5827,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6836/3447344578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":3447344578},{"type":"v1","addr":"172.21.15.146:6836","nonce":3447344578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":46},{"gid":5830,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/168994142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":168994142},{"type":"v1","addr":"172.21.15.146:6837","nonce":168994142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":46},{"gid":6735,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/3369489999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":3369489999},{"type":"v1","addr":"172.21.15.6:6833","nonce":3369489999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":45}],"filesystems":[{"mdsmap":{"epoch":51,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:29:47.357677+0000","modified":"2024-03-04T09:29:50.579389+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6744},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6744":{"gid":6744,"name":"a","rank":0,"incarnation":47,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6835/1667002065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1667002065},{"type":"v1","addr":"172.21.15.6:6835","nonce":1667002065}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6744,"qdb_cluster":[6744]},"id":6}]} 2024-03-04T09:29:51.432 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 51 2024-03-04T09:29:51.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.436+0000 7fd2dce00640 1 -- 172.21.15.6:0/3062626542 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd2b406b4d0 msgr2=0x7fd2b406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:51.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.436+0000 7fd2dce00640 1 --2- 172.21.15.6:0/3062626542 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd2b406b4d0 0x7fd2b406d990 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fd2c8034000 tx=0x7fd2c802f000 comp rx=0 tx=0).stop 2024-03-04T09:29:51.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.436+0000 7fd2dce00640 1 -- 172.21.15.6:0/3062626542 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd2d80b8150 msgr2=0x7fd2d80b94a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:51.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.436+0000 7fd2dce00640 1 --2- 172.21.15.6:0/3062626542 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd2d80b8150 0x7fd2d80b94a0 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7fd2d80d3c80 tx=0x7fd2d000d8a0 comp rx=0 tx=0).stop 2024-03-04T09:29:51.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.437+0000 7fd2dce00640 1 -- 172.21.15.6:0/3062626542 shutdown_connections 2024-03-04T09:29:51.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.437+0000 7fd2dce00640 1 --2- 172.21.15.6:0/3062626542 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd2d80b8150 0x7fd2d80b94a0 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:51.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.437+0000 7fd2dce00640 1 --2- 172.21.15.6:0/3062626542 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd2d80b7cd0 0x7fd2d80b8f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:51.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.437+0000 7fd2dce00640 1 --2- 172.21.15.6:0/3062626542 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd2b406b4d0 0x7fd2b406d990 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:51.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.437+0000 7fd2dce00640 1 --2- 172.21.15.6:0/3062626542 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd2bc0039b0 0x7fd2d80cddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:51.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.437+0000 7fd2dce00640 1 -- 172.21.15.6:0/3062626542 >> 172.21.15.6:0/3062626542 conn(0x7fd2d80d3d30 msgr2=0x7fd2d80cfee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:29:51.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.437+0000 7fd2dce00640 1 -- 172.21.15.6:0/3062626542 shutdown_connections 2024-03-04T09:29:51.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.437+0000 7fd2dce00640 1 -- 172.21.15.6:0/3062626542 wait complete. 2024-03-04T09:29:51.452 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T09:29:51.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.655+0000 7fb705400640 1 Processor -- start 2024-03-04T09:29:51.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.655+0000 7fb705400640 1 -- start start 2024-03-04T09:29:51.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.657+0000 7fb705400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fb7000d0e50 0x7fb700168c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:51.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.657+0000 7fb705400640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7000b3710 con 0x7fb70015e0b0 2024-03-04T09:29:51.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.657+0000 7fb705400640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7000b39f0 con 0x7fb7000d4a10 2024-03-04T09:29:51.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.657+0000 7fb705400640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7000b09e0 con 0x7fb7000d0e50 2024-03-04T09:29:51.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.657+0000 7fb6ffe00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fb70015e0b0 0x7fb7000cf310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:41978/0 (socket says 172.21.15.6:41978) 2024-03-04T09:29:51.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.657+0000 7fb6ffe00640 1 -- 172.21.15.6:0/3252256019 learned_addr learned my addr 172.21.15.6:0/3252256019 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:29:51.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.658+0000 7fb6ff400640 1 -- 172.21.15.6:0/3252256019 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 296753812 0 0) 0x7fb7000b3710 con 0x7fb70015e0b0 2024-03-04T09:29:51.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.658+0000 7fb704a00640 1 --2- 172.21.15.6:0/3252256019 >> v2:172.21.15.146:3300/0 conn(0x7fb7000d0e50 0x7fb700168c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:51.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.658+0000 7fb6ff400640 1 -- 172.21.15.6:0/3252256019 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6e4003520 con 0x7fb70015e0b0 2024-03-04T09:29:51.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.658+0000 7fb6ff400640 1 -- 172.21.15.6:0/3252256019 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2204921102 0 0) 0x7fb6e4003520 con 0x7fb70015e0b0 2024-03-04T09:29:51.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.658+0000 7fb6ff400640 1 -- 172.21.15.6:0/3252256019 >> v2:172.21.15.146:3300/0 conn(0x7fb7000d0e50 msgr2=0x7fb700168c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:51.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.658+0000 7fb6ff400640 1 --2- 172.21.15.6:0/3252256019 >> v2:172.21.15.146:3300/0 conn(0x7fb7000d0e50 0x7fb700168c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:51.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.658+0000 7fb6ff400640 1 -- 172.21.15.6:0/3252256019 >> v1:172.21.15.146:6789/0 conn(0x7fb7000d4a10 legacy=0x7fb7000d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:51.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.658+0000 7fb6ff400640 1 -- 172.21.15.6:0/3252256019 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7000b3cd0 con 0x7fb70015e0b0 2024-03-04T09:29:51.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.658+0000 7fb6ff400640 1 -- 172.21.15.6:0/3252256019 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7fb6f0001670 con 0x7fb70015e0b0 2024-03-04T09:29:51.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.658+0000 7fb6ff400640 1 -- 172.21.15.6:0/3252256019 >> v1:172.21.15.6:6789/0 conn(0x7fb70015e0b0 legacy=0x7fb7000cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:51.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.658+0000 7fb6ff400640 1 --2- 172.21.15.6:0/3252256019 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb6e40039b0 0x7fb6e4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:51.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.658+0000 7fb6ff400640 1 --2- 172.21.15.6:0/3252256019 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb6e4004800 0x7fb6e4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:51.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.658+0000 7fb6ff400640 1 --2- 172.21.15.6:0/3252256019 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb6e40051c0 0x7fb6e400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:51.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.658+0000 7fb6ff400640 1 -- 172.21.15.6:0/3252256019 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6e400c0e0 con 0x7fb6e40051c0 2024-03-04T09:29:51.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.658+0000 7fb6ff400640 1 -- 172.21.15.6:0/3252256019 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6e400c290 con 0x7fb6e40039b0 2024-03-04T09:29:51.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.658+0000 7fb6ff400640 1 -- 172.21.15.6:0/3252256019 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6e400c490 con 0x7fb6e4004800 2024-03-04T09:29:51.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.659+0000 7fb704a00640 1 --2- 172.21.15.6:0/3252256019 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb6e40051c0 0x7fb6e400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:51.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.659+0000 7fb6ffe00640 1 --2- 172.21.15.6:0/3252256019 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb6e40039b0 0x7fb6e4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:51.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.659+0000 7fb6f7400640 1 --2- 172.21.15.6:0/3252256019 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb6e4004800 0x7fb6e4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:51.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.659+0000 7fb704a00640 1 -- 172.21.15.6:0/3252256019 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb6e4004800 msgr2=0x7fb6e4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:51.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.659+0000 7fb704a00640 1 --2- 172.21.15.6:0/3252256019 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb6e4004800 0x7fb6e4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:51.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.659+0000 7fb704a00640 1 -- 172.21.15.6:0/3252256019 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb6e40039b0 msgr2=0x7fb6e4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:51.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.659+0000 7fb704a00640 1 --2- 172.21.15.6:0/3252256019 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb6e40039b0 0x7fb6e4003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:51.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.659+0000 7fb704a00640 1 -- 172.21.15.6:0/3252256019 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6e4003520 con 0x7fb6e40051c0 2024-03-04T09:29:51.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.660+0000 7fb704a00640 1 -- 172.21.15.6:0/3252256019 reap_dead start 2024-03-04T09:29:51.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.660+0000 7fb704a00640 1 --2- 172.21.15.6:0/3252256019 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb6e40051c0 0x7fb6e400b9c0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7fb6f8007330 tx=0x7fb6e4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f457dd6797b74d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:51.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.660+0000 7fb6ff400640 1 -- 172.21.15.6:0/3252256019 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6f8009be0 con 0x7fb6e40051c0 2024-03-04T09:29:51.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.660+0000 7fb6ff400640 1 -- 172.21.15.6:0/3252256019 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6f802ccd0 con 0x7fb6e40051c0 2024-03-04T09:29:51.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.660+0000 7fb6ff400640 1 -- 172.21.15.6:0/3252256019 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6f8003bd0 con 0x7fb6e40051c0 2024-03-04T09:29:51.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.661+0000 7fb705400640 1 -- 172.21.15.6:0/3252256019 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb6e40051c0 msgr2=0x7fb6e400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:51.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.661+0000 7fb705400640 1 --2- 172.21.15.6:0/3252256019 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb6e40051c0 0x7fb6e400b9c0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7fb6f8007330 tx=0x7fb6e4005830 comp rx=0 tx=0).stop 2024-03-04T09:29:51.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.661+0000 7fb705400640 1 -- 172.21.15.6:0/3252256019 shutdown_connections 2024-03-04T09:29:51.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.661+0000 7fb705400640 1 --2- 172.21.15.6:0/3252256019 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb6e40051c0 0x7fb6e400b9c0 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:51.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.661+0000 7fb705400640 1 -- 172.21.15.6:0/3252256019 >> 172.21.15.6:0/3252256019 conn(0x7fb7000d3d30 msgr2=0x7fb7000d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:29:51.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.661+0000 7fb705400640 1 -- 172.21.15.6:0/3252256019 shutdown_connections 2024-03-04T09:29:51.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.661+0000 7fb705400640 1 -- 172.21.15.6:0/3252256019 wait complete. 2024-03-04T09:29:51.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.663+0000 7fb705400640 1 Processor -- start 2024-03-04T09:29:51.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.663+0000 7fb705400640 1 -- start start 2024-03-04T09:29:51.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.664+0000 7fb705400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb6e40051c0 0x7fb7001f3aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:51.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.664+0000 7fb705400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb70015a3b0 0x7fb7000cc9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:51.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.664+0000 7fb705400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb7000ccef0 0x7fb7000cb810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:51.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.664+0000 7fb705400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7000cfc60 con 0x7fb7000ccef0 2024-03-04T09:29:51.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.664+0000 7fb705400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb70015e490 con 0x7fb70015a3b0 2024-03-04T09:29:51.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.664+0000 7fb705400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7000d0740 con 0x7fb6e40051c0 2024-03-04T09:29:51.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.664+0000 7fb704a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb7000ccef0 0x7fb7000cb810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:51.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.664+0000 7fb704a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb7000ccef0 0x7fb7000cb810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38698/0 (socket says 172.21.15.6:38698) 2024-03-04T09:29:51.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.664+0000 7fb704a00640 1 -- 172.21.15.6:0/2211837816 learned_addr learned my addr 172.21.15.6:0/2211837816 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:29:51.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.664+0000 7fb704a00640 1 -- 172.21.15.6:0/2211837816 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb6e40051c0 msgr2=0x7fb7001f3aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:51.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.664+0000 7fb704a00640 1 --2- 172.21.15.6:0/2211837816 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb6e40051c0 0x7fb7001f3aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:51.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.664+0000 7fb704a00640 1 -- 172.21.15.6:0/2211837816 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb70015a3b0 msgr2=0x7fb7000cc9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:51.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.664+0000 7fb704a00640 1 --2- 172.21.15.6:0/2211837816 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb70015a3b0 0x7fb7000cc9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:51.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.664+0000 7fb704a00640 1 -- 172.21.15.6:0/2211837816 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7001f3fe0 con 0x7fb7000ccef0 2024-03-04T09:29:51.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.665+0000 7fb704a00640 1 --2- 172.21.15.6:0/2211837816 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb7000ccef0 0x7fb7000cb810 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7fb6f8007420 tx=0x7fb6f8004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:51.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.665+0000 7fb6fcc00640 1 -- 172.21.15.6:0/2211837816 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6f8003e10 con 0x7fb7000ccef0 2024-03-04T09:29:51.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.665+0000 7fb6fcc00640 1 -- 172.21.15.6:0/2211837816 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6f8040480 con 0x7fb7000ccef0 2024-03-04T09:29:51.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.665+0000 7fb705400640 1 -- 172.21.15.6:0/2211837816 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7000c7760 con 0x7fb7000ccef0 2024-03-04T09:29:51.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.665+0000 7fb6fcc00640 1 -- 172.21.15.6:0/2211837816 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6f8040870 con 0x7fb7000ccef0 2024-03-04T09:29:51.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.665+0000 7fb705400640 1 -- 172.21.15.6:0/2211837816 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7000c7d20 con 0x7fb7000ccef0 2024-03-04T09:29:51.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.666+0000 7fb705400640 1 -- 172.21.15.6:0/2211837816 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb70004f330 con 0x7fb7000ccef0 2024-03-04T09:29:51.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.668+0000 7fb6fcc00640 1 -- 172.21.15.6:0/2211837816 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb6f8034440 con 0x7fb7000ccef0 2024-03-04T09:29:51.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.669+0000 7fb6fcc00640 1 --2- 172.21.15.6:0/2211837816 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb6dc06b4d0 0x7fb6dc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:51.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.669+0000 7fb6fcc00640 1 -- 172.21.15.6:0/2211837816 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb6f80abfc0 con 0x7fb7000ccef0 2024-03-04T09:29:51.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.670+0000 7fb6ffe00640 1 --2- 172.21.15.6:0/2211837816 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb6dc06b4d0 0x7fb6dc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:51.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.671+0000 7fb6ffe00640 1 --2- 172.21.15.6:0/2211837816 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb6dc06b4d0 0x7fb6dc06d990 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fb6f0001eb0 tx=0x7fb6f0000f30 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:51.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.674+0000 7fb6fcc00640 1 -- 172.21.15.6:0/2211837816 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb6f807a750 con 0x7fb7000ccef0 2024-03-04T09:29:51.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.873+0000 7fb705400640 1 -- 172.21.15.6:0/2211837816 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb7000d2af0 con 0x7fb7000ccef0 2024-03-04T09:29:51.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.874+0000 7fb6fcc00640 1 -- 172.21.15.6:0/2211837816 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+16612 (secure 0 0 0) 0x7fb6f807e7f0 con 0x7fb7000ccef0 2024-03-04T09:29:51.871 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:29:51.872 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":81,"fsid":"edce0578-5ae4-4402-b45a-552bdef84d7c","created":"2024-03-04T09:24:54.523780+0000","modified":"2024-03-04T09:29:48.885831+0000","last_up_change":"2024-03-04T09:25:07.089049+0000","last_in_change":"2024-03-04T09:24:57.982077+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T09:25:07.935198+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-03-04T09:29:45.280632+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-03-04T09:29:46.378588+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a96574f-81f1-4bc6-8463-757520b84a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6825","nonce":1364449401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6827","nonce":1364449401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6831","nonce":1364449401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6829","nonce":1364449401}]},"public_addr":"172.21.15.6:6825/1364449401","cluster_addr":"172.21.15.6:6827/1364449401","heartbeat_back_addr":"172.21.15.6:6831/1364449401","heartbeat_front_addr":"172.21.15.6:6829/1364449401","state":["exists","up"]},{"osd":1,"uuid":"7d79b9c1-6b87-476a-bc36-7db5ee64cd90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6809","nonce":2494103078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6811","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6813","nonce":2494103078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6821","nonce":2494103078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6817","nonce":2494103078}]},"public_addr":"172.21.15.6:6809/2494103078","cluster_addr":"172.21.15.6:6813/2494103078","heartbeat_back_addr":"172.21.15.6:6821/2494103078","heartbeat_front_addr":"172.21.15.6:6817/2494103078","state":["exists","up"]},{"osd":2,"uuid":"492331b9-4ee9-40b6-bacd-85d44cd6c45a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6812","nonce":1500066681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6816","nonce":1500066681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6824","nonce":1500066681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6820","nonce":1500066681}]},"public_addr":"172.21.15.6:6812/1500066681","cluster_addr":"172.21.15.6:6816/1500066681","heartbeat_back_addr":"172.21.15.6:6824/1500066681","heartbeat_front_addr":"172.21.15.6:6820/1500066681","state":["exists","up"]},{"osd":3,"uuid":"21bb2efe-9ba7-4bdc-bc31-180cd1102e21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6801","nonce":1558806123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6803","nonce":1558806123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6807","nonce":1558806123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6805","nonce":1558806123}]},"public_addr":"172.21.15.6:6801/1558806123","cluster_addr":"172.21.15.6:6803/1558806123","heartbeat_back_addr":"172.21.15.6:6807/1558806123","heartbeat_front_addr":"172.21.15.6:6805/1558806123","state":["exists","up"]},{"osd":4,"uuid":"6e2c4b71-19d9-4737-83d9-558705d8644a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6801","nonce":1526135780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6803","nonce":1526135780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6807","nonce":1526135780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6805","nonce":1526135780}]},"public_addr":"172.21.15.146:6801/1526135780","cluster_addr":"172.21.15.146:6803/1526135780","heartbeat_back_addr":"172.21.15.146:6807/1526135780","heartbeat_front_addr":"172.21.15.146:6805/1526135780","state":["exists","up"]},{"osd":5,"uuid":"fb69c525-f1be-4b7d-ab32-6c29b0d6fddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6809","nonce":460262507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6811","nonce":460262507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6815","nonce":460262507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6813","nonce":460262507}]},"public_addr":"172.21.15.146:6809/460262507","cluster_addr":"172.21.15.146:6811/460262507","heartbeat_back_addr":"172.21.15.146:6815/460262507","heartbeat_front_addr":"172.21.15.146:6813/460262507","state":["exists","up"]},{"osd":6,"uuid":"c32e7fed-370d-4401-8ce0-6a4335b0d168","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6823","nonce":1392897493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6827","nonce":1392897493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6831","nonce":1392897493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6829","nonce":1392897493}]},"public_addr":"172.21.15.146:6823/1392897493","cluster_addr":"172.21.15.146:6827/1392897493","heartbeat_back_addr":"172.21.15.146:6831/1392897493","heartbeat_front_addr":"172.21.15.146:6829/1392897493","state":["exists","up"]},{"osd":7,"uuid":"75a79b44-fa5f-426d-a7d3-71f21f78835c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6817","nonce":3268323156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6819","nonce":3268323156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6825","nonce":3268323156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6822","nonce":3268323156}]},"public_addr":"172.21.15.146:6817/3268323156","cluster_addr":"172.21.15.146:6819/3268323156","heartbeat_back_addr":"172.21.15.146:6825/3268323156","heartbeat_front_addr":"172.21.15.146:6822/3268323156","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.310437+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.295850+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.294674+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.526855+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.300023+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.595160+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.346046+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.558741+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T09:29:51.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.877+0000 7fb705400640 1 -- 172.21.15.6:0/2211837816 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb6dc06b4d0 msgr2=0x7fb6dc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:51.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.877+0000 7fb705400640 1 --2- 172.21.15.6:0/2211837816 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb6dc06b4d0 0x7fb6dc06d990 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fb6f0001eb0 tx=0x7fb6f0000f30 comp rx=0 tx=0).stop 2024-03-04T09:29:51.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.877+0000 7fb705400640 1 -- 172.21.15.6:0/2211837816 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb7000ccef0 msgr2=0x7fb7000cb810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:51.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.877+0000 7fb705400640 1 --2- 172.21.15.6:0/2211837816 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb7000ccef0 0x7fb7000cb810 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7fb6f8007420 tx=0x7fb6f8004ab0 comp rx=0 tx=0).stop 2024-03-04T09:29:51.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.877+0000 7fb705400640 1 -- 172.21.15.6:0/2211837816 shutdown_connections 2024-03-04T09:29:51.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.877+0000 7fb705400640 1 --2- 172.21.15.6:0/2211837816 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb7000ccef0 0x7fb7000cb810 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:51.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.877+0000 7fb705400640 1 --2- 172.21.15.6:0/2211837816 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb70015a3b0 0x7fb7000cc9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:51.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.877+0000 7fb705400640 1 --2- 172.21.15.6:0/2211837816 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb6dc06b4d0 0x7fb6dc06d990 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:51.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.877+0000 7fb705400640 1 --2- 172.21.15.6:0/2211837816 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb6e40051c0 0x7fb7001f3aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:51.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.877+0000 7fb705400640 1 -- 172.21.15.6:0/2211837816 >> 172.21.15.6:0/2211837816 conn(0x7fb7000d3d30 msgr2=0x7fb7000cf070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:29:51.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.877+0000 7fb705400640 1 -- 172.21.15.6:0/2211837816 shutdown_connections 2024-03-04T09:29:51.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:51.877+0000 7fb705400640 1 -- 172.21.15.6:0/2211837816 wait complete. 2024-03-04T09:29:51.892 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-04T09:29:51.892 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T09:29:52.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.109+0000 7f2203000640 1 Processor -- start 2024-03-04T09:29:52.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.109+0000 7f2203000640 1 -- start start 2024-03-04T09:29:52.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.111+0000 7f2203000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f21fc06c8b0 0x7f21fc06ef10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:52.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.111+0000 7f2203000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f21fc0ce370 0x7f21fc0ce7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:52.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.111+0000 7f2203000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21fc0cf1b0 con 0x7f21fc066d00 2024-03-04T09:29:52.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.111+0000 7f2203000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21fc15e470 con 0x7f21fc06c8b0 2024-03-04T09:29:52.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.111+0000 7f2203000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21fc15ea00 con 0x7f21fc0ce370 2024-03-04T09:29:52.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.111+0000 7f2202600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f21fc0ce370 0x7f21fc0ce7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:52.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.112+0000 7f2202600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f21fc0ce370 0x7f21fc0ce7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.6:52004/0 (socket says 172.21.15.6:52004) 2024-03-04T09:29:52.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.112+0000 7f2202600640 1 -- 172.21.15.6:0/1611211322 learned_addr learned my addr 172.21.15.6:0/1611211322 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:29:52.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.112+0000 7f2201c00640 1 --2- 172.21.15.6:0/1611211322 >> v2:172.21.15.146:3300/0 conn(0x7f21fc06c8b0 0x7f21fc06ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:52.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.112+0000 7f2202600640 1 -- 172.21.15.6:0/1611211322 >> v2:172.21.15.146:3300/0 conn(0x7f21fc06c8b0 msgr2=0x7f21fc06ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:52.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.112+0000 7f2202600640 1 --2- 172.21.15.6:0/1611211322 >> v2:172.21.15.146:3300/0 conn(0x7f21fc06c8b0 0x7f21fc06ef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:52.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.112+0000 7f2202600640 1 -- 172.21.15.6:0/1611211322 >> v1:172.21.15.146:6789/0 conn(0x7f21fc066d00 legacy=0x7f21fc15ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:52.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.112+0000 7f2202600640 1 -- 172.21.15.6:0/1611211322 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21fc06d420 con 0x7f21fc0ce370 2024-03-04T09:29:52.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.114+0000 7f2202600640 1 --2- 172.21.15.6:0/1611211322 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f21fc0ce370 0x7f21fc0ce7e0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f21f4009e50 tx=0x7f21f402bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c0d3dee6285176c4 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:52.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.114+0000 7f21fbe00640 1 -- 172.21.15.6:0/1611211322 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21f402cb00 con 0x7f21fc0ce370 2024-03-04T09:29:52.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.114+0000 7f21fbe00640 1 -- 172.21.15.6:0/1611211322 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21f402cca0 con 0x7f21fc0ce370 2024-03-04T09:29:52.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.114+0000 7f21fbe00640 1 -- 172.21.15.6:0/1611211322 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21f4034610 con 0x7f21fc0ce370 2024-03-04T09:29:52.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.115+0000 7f2203000640 1 -- 172.21.15.6:0/1611211322 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f21fc0ce370 msgr2=0x7f21fc0ce7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:52.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.115+0000 7f2203000640 1 --2- 172.21.15.6:0/1611211322 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f21fc0ce370 0x7f21fc0ce7e0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f21f4009e50 tx=0x7f21f402bf60 comp rx=0 tx=0).stop 2024-03-04T09:29:52.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.115+0000 7f2203000640 1 -- 172.21.15.6:0/1611211322 shutdown_connections 2024-03-04T09:29:52.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.115+0000 7f2203000640 1 --2- 172.21.15.6:0/1611211322 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f21fc0ce370 0x7f21fc0ce7e0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:52.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.115+0000 7f2203000640 1 --2- 172.21.15.6:0/1611211322 >> v2:172.21.15.146:3300/0 conn(0x7f21fc06c8b0 0x7f21fc06ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:52.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.115+0000 7f2203000640 1 -- 172.21.15.6:0/1611211322 >> 172.21.15.6:0/1611211322 conn(0x7f21fc0ccf70 msgr2=0x7f21fc0cd350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:29:52.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.115+0000 7f2203000640 1 -- 172.21.15.6:0/1611211322 shutdown_connections 2024-03-04T09:29:52.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.115+0000 7f2203000640 1 -- 172.21.15.6:0/1611211322 wait complete. 2024-03-04T09:29:52.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.119+0000 7f2203000640 1 Processor -- start 2024-03-04T09:29:52.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.119+0000 7f2203000640 1 -- start start 2024-03-04T09:29:52.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.121+0000 7f2203000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f21fc066d00 0x7f21fc148860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:52.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.121+0000 7f2203000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f21fc06c8b0 0x7f21fc1455a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:52.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.121+0000 7f2203000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f21fc0ce370 0x7f21fc15d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:52.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.121+0000 7f2203000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21fc0ceea0 con 0x7f21fc06c8b0 2024-03-04T09:29:52.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.121+0000 7f2203000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21fc0ced20 con 0x7f21fc066d00 2024-03-04T09:29:52.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.121+0000 7f2203000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21fc06afb0 con 0x7f21fc0ce370 2024-03-04T09:29:52.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.122+0000 7f2201200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f21fc06c8b0 0x7f21fc1455a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:52.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.122+0000 7f2201200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f21fc06c8b0 0x7f21fc1455a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38714/0 (socket says 172.21.15.6:38714) 2024-03-04T09:29:52.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.122+0000 7f2201200640 1 -- 172.21.15.6:0/1085812661 learned_addr learned my addr 172.21.15.6:0/1085812661 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:29:52.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.122+0000 7f2202600640 1 --2- 172.21.15.6:0/1085812661 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f21fc0ce370 0x7f21fc15d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:52.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.122+0000 7f2201c00640 1 --2- 172.21.15.6:0/1085812661 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f21fc066d00 0x7f21fc148860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:52.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.122+0000 7f2201200640 1 -- 172.21.15.6:0/1085812661 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f21fc0ce370 msgr2=0x7f21fc15d550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:52.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.122+0000 7f2201200640 1 --2- 172.21.15.6:0/1085812661 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f21fc0ce370 0x7f21fc15d550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:52.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.122+0000 7f2201200640 1 -- 172.21.15.6:0/1085812661 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f21fc066d00 msgr2=0x7f21fc148860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:52.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.122+0000 7f2201200640 1 --2- 172.21.15.6:0/1085812661 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f21fc066d00 0x7f21fc148860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:52.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.122+0000 7f2201200640 1 -- 172.21.15.6:0/1085812661 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21fc148da0 con 0x7f21fc06c8b0 2024-03-04T09:29:52.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.122+0000 7f2201200640 1 --2- 172.21.15.6:0/1085812661 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f21fc06c8b0 0x7f21fc1455a0 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f21e800c400 tx=0x7f21e800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:52.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.123+0000 7f21f9600640 1 -- 172.21.15.6:0/1085812661 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e80052d0 con 0x7f21fc06c8b0 2024-03-04T09:29:52.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.123+0000 7f21f9600640 1 -- 172.21.15.6:0/1085812661 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e8005490 con 0x7f21fc06c8b0 2024-03-04T09:29:52.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.123+0000 7f2203000640 1 -- 172.21.15.6:0/1085812661 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21fc145ae0 con 0x7f21fc06c8b0 2024-03-04T09:29:52.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.123+0000 7f21f9600640 1 -- 172.21.15.6:0/1085812661 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e8007260 con 0x7f21fc06c8b0 2024-03-04T09:29:52.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.123+0000 7f2203000640 1 -- 172.21.15.6:0/1085812661 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21fc15f630 con 0x7f21fc06c8b0 2024-03-04T09:29:52.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.124+0000 7f2203000640 1 -- 172.21.15.6:0/1085812661 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21fc04f330 con 0x7f21fc06c8b0 2024-03-04T09:29:52.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.126+0000 7f21f9600640 1 -- 172.21.15.6:0/1085812661 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f21e8007400 con 0x7f21fc06c8b0 2024-03-04T09:29:52.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.127+0000 7f21f9600640 1 --2- 172.21.15.6:0/1085812661 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f21dc06b4d0 0x7f21dc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:52.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.127+0000 7f21f9600640 1 -- 172.21.15.6:0/1085812661 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f21e8089780 con 0x7f21fc06c8b0 2024-03-04T09:29:52.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.127+0000 7f2201c00640 1 --2- 172.21.15.6:0/1085812661 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f21dc06b4d0 0x7f21dc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:52.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.128+0000 7f2201c00640 1 --2- 172.21.15.6:0/1085812661 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f21dc06b4d0 0x7f21dc06d990 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f21f0009ad0 tx=0x7f21f0008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:52.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.131+0000 7f21f9600640 1 -- 172.21.15.6:0/1085812661 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f21e8058600 con 0x7f21fc06c8b0 2024-03-04T09:29:52.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.331+0000 7f2203000640 1 -- 172.21.15.6:0/1085812661 --> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f21fc0cdde0 con 0x7f21dc06b4d0 2024-03-04T09:29:52.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.351+0000 7f21f9600640 1 -- 172.21.15.6:0/1085812661 <== mgr.4113 v2:172.21.15.146:6832/1788613334 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252000 (secure 0 0 0) 0x7f21fc0cdde0 con 0x7f21dc06b4d0 2024-03-04T09:29:52.351 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:29:52.353 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-03-04T09:29:52.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.356+0000 7f2203000640 1 -- 172.21.15.6:0/1085812661 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f21dc06b4d0 msgr2=0x7f21dc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:52.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.356+0000 7f2203000640 1 --2- 172.21.15.6:0/1085812661 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f21dc06b4d0 0x7f21dc06d990 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f21f0009ad0 tx=0x7f21f0008040 comp rx=0 tx=0).stop 2024-03-04T09:29:52.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.356+0000 7f2203000640 1 -- 172.21.15.6:0/1085812661 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f21fc06c8b0 msgr2=0x7f21fc1455a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:52.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.356+0000 7f2203000640 1 --2- 172.21.15.6:0/1085812661 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f21fc06c8b0 0x7f21fc1455a0 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f21e800c400 tx=0x7f21e800c8c0 comp rx=0 tx=0).stop 2024-03-04T09:29:52.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.356+0000 7f2203000640 1 -- 172.21.15.6:0/1085812661 shutdown_connections 2024-03-04T09:29:52.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.356+0000 7f2203000640 1 --2- 172.21.15.6:0/1085812661 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f21dc06b4d0 0x7f21dc06d990 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:52.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.356+0000 7f2203000640 1 --2- 172.21.15.6:0/1085812661 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f21fc0ce370 0x7f21fc15d550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:52.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.356+0000 7f2203000640 1 --2- 172.21.15.6:0/1085812661 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f21fc06c8b0 0x7f21fc1455a0 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:52.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.356+0000 7f2203000640 1 --2- 172.21.15.6:0/1085812661 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f21fc066d00 0x7f21fc148860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:52.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.356+0000 7f2203000640 1 -- 172.21.15.6:0/1085812661 >> 172.21.15.6:0/1085812661 conn(0x7f21fc0ccf70 msgr2=0x7f21fc161800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:29:52.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.356+0000 7f2203000640 1 -- 172.21.15.6:0/1085812661 shutdown_connections 2024-03-04T09:29:52.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.356+0000 7f2203000640 1 -- 172.21.15.6:0/1085812661 wait complete. 2024-03-04T09:29:52.381 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":216,"stamp":"2024-03-04T09:29:51.957304+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":43,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":258208,"kb_used_data":8744,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754716512,"statfs":{"total":773094113280,"available":772829708288,"internally_reserved":0,"allocated":8953856,"data_stored":4458152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001694"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.931056+0000","last_change":"2024-03-04T09:29:47.926163+0000","last_active":"2024-03-04T09:29:47.931056+0000","last_peered":"2024-03-04T09:29:47.931056+0000","last_clean":"2024-03-04T09:29:47.931056+0000","last_became_active":"2024-03-04T09:29:47.925792+0000","last_became_peered":"2024-03-04T09:29:47.925792+0000","last_unstale":"2024-03-04T09:29:47.931056+0000","last_undegraded":"2024-03-04T09:29:47.931056+0000","last_fullsized":"2024-03-04T09:29:47.931056+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:32:32.654222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.927761+0000","last_change":"2024-03-04T09:29:47.921497+0000","last_active":"2024-03-04T09:29:47.927761+0000","last_peered":"2024-03-04T09:29:47.927761+0000","last_clean":"2024-03-04T09:29:47.927761+0000","last_became_active":"2024-03-04T09:29:47.921167+0000","last_became_peered":"2024-03-04T09:29:47.921167+0000","last_unstale":"2024-03-04T09:29:47.927761+0000","last_undegraded":"2024-03-04T09:29:47.927761+0000","last_fullsized":"2024-03-04T09:29:47.927761+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:41:37.529066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.929202+0000","last_change":"2024-03-04T09:29:47.924582+0000","last_active":"2024-03-04T09:29:47.929202+0000","last_peered":"2024-03-04T09:29:47.929202+0000","last_clean":"2024-03-04T09:29:47.929202+0000","last_became_active":"2024-03-04T09:29:47.924275+0000","last_became_peered":"2024-03-04T09:29:47.924275+0000","last_unstale":"2024-03-04T09:29:47.929202+0000","last_undegraded":"2024-03-04T09:29:47.929202+0000","last_fullsized":"2024-03-04T09:29:47.929202+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:11:36.963320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.934625+0000","last_change":"2024-03-04T09:29:47.927066+0000","last_active":"2024-03-04T09:29:47.934625+0000","last_peered":"2024-03-04T09:29:47.934625+0000","last_clean":"2024-03-04T09:29:47.934625+0000","last_became_active":"2024-03-04T09:29:47.926721+0000","last_became_peered":"2024-03-04T09:29:47.926721+0000","last_unstale":"2024-03-04T09:29:47.934625+0000","last_undegraded":"2024-03-04T09:29:47.934625+0000","last_fullsized":"2024-03-04T09:29:47.934625+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:13:39.210481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.938976+0000","last_change":"2024-03-04T09:29:47.935195+0000","last_active":"2024-03-04T09:29:47.938976+0000","last_peered":"2024-03-04T09:29:47.938976+0000","last_clean":"2024-03-04T09:29:47.938976+0000","last_became_active":"2024-03-04T09:29:47.934932+0000","last_became_peered":"2024-03-04T09:29:47.934932+0000","last_unstale":"2024-03-04T09:29:47.938976+0000","last_undegraded":"2024-03-04T09:29:47.938976+0000","last_fullsized":"2024-03-04T09:29:47.938976+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:49:19.951736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.932066+0000","last_change":"2024-03-04T09:29:47.929146+0000","last_active":"2024-03-04T09:29:47.932066+0000","last_peered":"2024-03-04T09:29:47.932066+0000","last_clean":"2024-03-04T09:29:47.932066+0000","last_became_active":"2024-03-04T09:29:47.928691+0000","last_became_peered":"2024-03-04T09:29:47.928691+0000","last_unstale":"2024-03-04T09:29:47.932066+0000","last_undegraded":"2024-03-04T09:29:47.932066+0000","last_fullsized":"2024-03-04T09:29:47.932066+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:13:42.546100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.934459+0000","last_change":"2024-03-04T09:29:47.929052+0000","last_active":"2024-03-04T09:29:47.934459+0000","last_peered":"2024-03-04T09:29:47.934459+0000","last_clean":"2024-03-04T09:29:47.934459+0000","last_became_active":"2024-03-04T09:29:47.928609+0000","last_became_peered":"2024-03-04T09:29:47.928609+0000","last_unstale":"2024-03-04T09:29:47.934459+0000","last_undegraded":"2024-03-04T09:29:47.934459+0000","last_fullsized":"2024-03-04T09:29:47.934459+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:44:09.397551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.920344+0000","last_change":"2024-03-04T09:29:47.911061+0000","last_active":"2024-03-04T09:29:47.920344+0000","last_peered":"2024-03-04T09:29:47.920344+0000","last_clean":"2024-03-04T09:29:47.920344+0000","last_became_active":"2024-03-04T09:29:47.909106+0000","last_became_peered":"2024-03-04T09:29:47.909106+0000","last_unstale":"2024-03-04T09:29:47.920344+0000","last_undegraded":"2024-03-04T09:29:47.920344+0000","last_fullsized":"2024-03-04T09:29:47.920344+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:35:14.693482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.921330+0000","last_change":"2024-03-04T09:29:47.917439+0000","last_active":"2024-03-04T09:29:47.921330+0000","last_peered":"2024-03-04T09:29:47.921330+0000","last_clean":"2024-03-04T09:29:47.921330+0000","last_became_active":"2024-03-04T09:29:47.917005+0000","last_became_peered":"2024-03-04T09:29:47.917005+0000","last_unstale":"2024-03-04T09:29:47.921330+0000","last_undegraded":"2024-03-04T09:29:47.921330+0000","last_fullsized":"2024-03-04T09:29:47.921330+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:32:42.357098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.930971+0000","last_change":"2024-03-04T09:29:47.927014+0000","last_active":"2024-03-04T09:29:47.930971+0000","last_peered":"2024-03-04T09:29:47.930971+0000","last_clean":"2024-03-04T09:29:47.930971+0000","last_became_active":"2024-03-04T09:29:47.926689+0000","last_became_peered":"2024-03-04T09:29:47.926689+0000","last_unstale":"2024-03-04T09:29:47.930971+0000","last_undegraded":"2024-03-04T09:29:47.930971+0000","last_fullsized":"2024-03-04T09:29:47.930971+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:13:42.546100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.928021+0000","last_change":"2024-03-04T09:29:47.920696+0000","last_active":"2024-03-04T09:29:47.928021+0000","last_peered":"2024-03-04T09:29:47.928021+0000","last_clean":"2024-03-04T09:29:47.928021+0000","last_became_active":"2024-03-04T09:29:47.918831+0000","last_became_peered":"2024-03-04T09:29:47.918831+0000","last_unstale":"2024-03-04T09:29:47.928021+0000","last_undegraded":"2024-03-04T09:29:47.928021+0000","last_fullsized":"2024-03-04T09:29:47.928021+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:01:52.293904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.931170+0000","last_change":"2024-03-04T09:29:47.927745+0000","last_active":"2024-03-04T09:29:47.931170+0000","last_peered":"2024-03-04T09:29:47.931170+0000","last_clean":"2024-03-04T09:29:47.931170+0000","last_became_active":"2024-03-04T09:29:47.927105+0000","last_became_peered":"2024-03-04T09:29:47.927105+0000","last_unstale":"2024-03-04T09:29:47.931170+0000","last_undegraded":"2024-03-04T09:29:47.931170+0000","last_fullsized":"2024-03-04T09:29:47.931170+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:50:00.557690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.936182+0000","last_change":"2024-03-04T09:29:47.929993+0000","last_active":"2024-03-04T09:29:47.936182+0000","last_peered":"2024-03-04T09:29:47.936182+0000","last_clean":"2024-03-04T09:29:47.936182+0000","last_became_active":"2024-03-04T09:29:47.929381+0000","last_became_peered":"2024-03-04T09:29:47.929381+0000","last_unstale":"2024-03-04T09:29:47.936182+0000","last_undegraded":"2024-03-04T09:29:47.936182+0000","last_fullsized":"2024-03-04T09:29:47.936182+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:43:42.397015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.935177+0000","last_change":"2024-03-04T09:29:47.926135+0000","last_active":"2024-03-04T09:29:47.935177+0000","last_peered":"2024-03-04T09:29:47.935177+0000","last_clean":"2024-03-04T09:29:47.935177+0000","last_became_active":"2024-03-04T09:29:47.925848+0000","last_became_peered":"2024-03-04T09:29:47.925848+0000","last_unstale":"2024-03-04T09:29:47.935177+0000","last_undegraded":"2024-03-04T09:29:47.935177+0000","last_fullsized":"2024-03-04T09:29:47.935177+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:30:18.836859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.929741+0000","last_change":"2024-03-04T09:29:47.920506+0000","last_active":"2024-03-04T09:29:47.929741+0000","last_peered":"2024-03-04T09:29:47.929741+0000","last_clean":"2024-03-04T09:29:47.929741+0000","last_became_active":"2024-03-04T09:29:47.919954+0000","last_became_peered":"2024-03-04T09:29:47.919954+0000","last_unstale":"2024-03-04T09:29:47.929741+0000","last_undegraded":"2024-03-04T09:29:47.929741+0000","last_fullsized":"2024-03-04T09:29:47.929741+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:51:58.796350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.932909+0000","last_change":"2024-03-04T09:29:47.930263+0000","last_active":"2024-03-04T09:29:47.932909+0000","last_peered":"2024-03-04T09:29:47.932909+0000","last_clean":"2024-03-04T09:29:47.932909+0000","last_became_active":"2024-03-04T09:29:47.928895+0000","last_became_peered":"2024-03-04T09:29:47.928895+0000","last_unstale":"2024-03-04T09:29:47.932909+0000","last_undegraded":"2024-03-04T09:29:47.932909+0000","last_fullsized":"2024-03-04T09:29:47.932909+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:32:32.654222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.941355+0000","last_change":"2024-03-04T09:29:47.937624+0000","last_active":"2024-03-04T09:29:47.941355+0000","last_peered":"2024-03-04T09:29:47.941355+0000","last_clean":"2024-03-04T09:29:47.941355+0000","last_became_active":"2024-03-04T09:29:47.937306+0000","last_became_peered":"2024-03-04T09:29:47.937306+0000","last_unstale":"2024-03-04T09:29:47.941355+0000","last_undegraded":"2024-03-04T09:29:47.941355+0000","last_fullsized":"2024-03-04T09:29:47.941355+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:21:18.831139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.938481+0000","last_change":"2024-03-04T09:29:47.933722+0000","last_active":"2024-03-04T09:29:47.938481+0000","last_peered":"2024-03-04T09:29:47.938481+0000","last_clean":"2024-03-04T09:29:47.938481+0000","last_became_active":"2024-03-04T09:29:47.933363+0000","last_became_peered":"2024-03-04T09:29:47.933363+0000","last_unstale":"2024-03-04T09:29:47.938481+0000","last_undegraded":"2024-03-04T09:29:47.938481+0000","last_fullsized":"2024-03-04T09:29:47.938481+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:51:18.420239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.931407+0000","last_change":"2024-03-04T09:29:47.923456+0000","last_active":"2024-03-04T09:29:47.931407+0000","last_peered":"2024-03-04T09:29:47.931407+0000","last_clean":"2024-03-04T09:29:47.931407+0000","last_became_active":"2024-03-04T09:29:47.923113+0000","last_became_peered":"2024-03-04T09:29:47.923113+0000","last_unstale":"2024-03-04T09:29:47.931407+0000","last_undegraded":"2024-03-04T09:29:47.931407+0000","last_fullsized":"2024-03-04T09:29:47.931407+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:27:13.253074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.933645+0000","last_change":"2024-03-04T09:29:47.922349+0000","last_active":"2024-03-04T09:29:47.933645+0000","last_peered":"2024-03-04T09:29:47.933645+0000","last_clean":"2024-03-04T09:29:47.933645+0000","last_became_active":"2024-03-04T09:29:47.918752+0000","last_became_peered":"2024-03-04T09:29:47.918752+0000","last_unstale":"2024-03-04T09:29:47.933645+0000","last_undegraded":"2024-03-04T09:29:47.933645+0000","last_fullsized":"2024-03-04T09:29:47.933645+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:24:29.054506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.932862+0000","last_change":"2024-03-04T09:29:47.920982+0000","last_active":"2024-03-04T09:29:47.932862+0000","last_peered":"2024-03-04T09:29:47.932862+0000","last_clean":"2024-03-04T09:29:47.932862+0000","last_became_active":"2024-03-04T09:29:47.920480+0000","last_became_peered":"2024-03-04T09:29:47.920480+0000","last_unstale":"2024-03-04T09:29:47.932862+0000","last_undegraded":"2024-03-04T09:29:47.932862+0000","last_fullsized":"2024-03-04T09:29:47.932862+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:41:48.826232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.923083+0000","last_change":"2024-03-04T09:29:47.916934+0000","last_active":"2024-03-04T09:29:47.923083+0000","last_peered":"2024-03-04T09:29:47.923083+0000","last_clean":"2024-03-04T09:29:47.923083+0000","last_became_active":"2024-03-04T09:29:47.916618+0000","last_became_peered":"2024-03-04T09:29:47.916618+0000","last_unstale":"2024-03-04T09:29:47.923083+0000","last_undegraded":"2024-03-04T09:29:47.923083+0000","last_fullsized":"2024-03-04T09:29:47.923083+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:11:44.498334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.924238+0000","last_change":"2024-03-04T09:29:47.920358+0000","last_active":"2024-03-04T09:29:47.924238+0000","last_peered":"2024-03-04T09:29:47.924238+0000","last_clean":"2024-03-04T09:29:47.924238+0000","last_became_active":"2024-03-04T09:29:47.919453+0000","last_became_peered":"2024-03-04T09:29:47.919453+0000","last_unstale":"2024-03-04T09:29:47.924238+0000","last_undegraded":"2024-03-04T09:29:47.924238+0000","last_fullsized":"2024-03-04T09:29:47.924238+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:41:03.110600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.932790+0000","last_change":"2024-03-04T09:29:47.920639+0000","last_active":"2024-03-04T09:29:47.932790+0000","last_peered":"2024-03-04T09:29:47.932790+0000","last_clean":"2024-03-04T09:29:47.932790+0000","last_became_active":"2024-03-04T09:29:47.920361+0000","last_became_peered":"2024-03-04T09:29:47.920361+0000","last_unstale":"2024-03-04T09:29:47.932790+0000","last_undegraded":"2024-03-04T09:29:47.932790+0000","last_fullsized":"2024-03-04T09:29:47.932790+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:36:04.950142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.927617+0000","last_change":"2024-03-04T09:29:47.925548+0000","last_active":"2024-03-04T09:29:47.927617+0000","last_peered":"2024-03-04T09:29:47.927617+0000","last_clean":"2024-03-04T09:29:47.927617+0000","last_became_active":"2024-03-04T09:29:47.925214+0000","last_became_peered":"2024-03-04T09:29:47.925214+0000","last_unstale":"2024-03-04T09:29:47.927617+0000","last_undegraded":"2024-03-04T09:29:47.927617+0000","last_fullsized":"2024-03-04T09:29:47.927617+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:42:37.575160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.933675+0000","last_change":"2024-03-04T09:29:47.924998+0000","last_active":"2024-03-04T09:29:47.933675+0000","last_peered":"2024-03-04T09:29:47.933675+0000","last_clean":"2024-03-04T09:29:47.933675+0000","last_became_active":"2024-03-04T09:29:47.924658+0000","last_became_peered":"2024-03-04T09:29:47.924658+0000","last_unstale":"2024-03-04T09:29:47.933675+0000","last_undegraded":"2024-03-04T09:29:47.933675+0000","last_fullsized":"2024-03-04T09:29:47.933675+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:21:18.831139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.938246+0000","last_change":"2024-03-04T09:29:47.934371+0000","last_active":"2024-03-04T09:29:47.938246+0000","last_peered":"2024-03-04T09:29:47.938246+0000","last_clean":"2024-03-04T09:29:47.938246+0000","last_became_active":"2024-03-04T09:29:47.933995+0000","last_became_peered":"2024-03-04T09:29:47.933995+0000","last_unstale":"2024-03-04T09:29:47.938246+0000","last_undegraded":"2024-03-04T09:29:47.938246+0000","last_fullsized":"2024-03-04T09:29:47.938246+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:50:00.557690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.924893+0000","last_change":"2024-03-04T09:29:47.920745+0000","last_active":"2024-03-04T09:29:47.924893+0000","last_peered":"2024-03-04T09:29:47.924893+0000","last_clean":"2024-03-04T09:29:47.924893+0000","last_became_active":"2024-03-04T09:29:47.920426+0000","last_became_peered":"2024-03-04T09:29:47.920426+0000","last_unstale":"2024-03-04T09:29:47.924893+0000","last_undegraded":"2024-03-04T09:29:47.924893+0000","last_fullsized":"2024-03-04T09:29:47.924893+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:35:14.693482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.919501+0000","last_change":"2024-03-04T09:29:47.913136+0000","last_active":"2024-03-04T09:29:47.919501+0000","last_peered":"2024-03-04T09:29:47.919501+0000","last_clean":"2024-03-04T09:29:47.919501+0000","last_became_active":"2024-03-04T09:29:47.911494+0000","last_became_peered":"2024-03-04T09:29:47.911494+0000","last_unstale":"2024-03-04T09:29:47.919501+0000","last_undegraded":"2024-03-04T09:29:47.919501+0000","last_fullsized":"2024-03-04T09:29:47.919501+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:32:58.823873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.937050+0000","last_change":"2024-03-04T09:29:47.930841+0000","last_active":"2024-03-04T09:29:47.937050+0000","last_peered":"2024-03-04T09:29:47.937050+0000","last_clean":"2024-03-04T09:29:47.937050+0000","last_became_active":"2024-03-04T09:29:47.928656+0000","last_became_peered":"2024-03-04T09:29:47.928656+0000","last_unstale":"2024-03-04T09:29:47.937050+0000","last_undegraded":"2024-03-04T09:29:47.937050+0000","last_fullsized":"2024-03-04T09:29:47.937050+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:00:02.131925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.933106+0000","last_change":"2024-03-04T09:29:47.928549+0000","last_active":"2024-03-04T09:29:47.933106+0000","last_peered":"2024-03-04T09:29:47.933106+0000","last_clean":"2024-03-04T09:29:47.933106+0000","last_became_active":"2024-03-04T09:29:47.928247+0000","last_became_peered":"2024-03-04T09:29:47.928247+0000","last_unstale":"2024-03-04T09:29:47.933106+0000","last_undegraded":"2024-03-04T09:29:47.933106+0000","last_fullsized":"2024-03-04T09:29:47.933106+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:32:42.357098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.922907+0000","last_change":"2024-03-04T09:29:47.916919+0000","last_active":"2024-03-04T09:29:47.922907+0000","last_peered":"2024-03-04T09:29:47.922907+0000","last_clean":"2024-03-04T09:29:47.922907+0000","last_became_active":"2024-03-04T09:29:47.916462+0000","last_became_peered":"2024-03-04T09:29:47.916462+0000","last_unstale":"2024-03-04T09:29:47.922907+0000","last_undegraded":"2024-03-04T09:29:47.922907+0000","last_fullsized":"2024-03-04T09:29:47.922907+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:19:21.432462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.939032+0000","last_change":"2024-03-04T09:29:47.934744+0000","last_active":"2024-03-04T09:29:47.939032+0000","last_peered":"2024-03-04T09:29:47.939032+0000","last_clean":"2024-03-04T09:29:47.939032+0000","last_became_active":"2024-03-04T09:29:47.934453+0000","last_became_peered":"2024-03-04T09:29:47.934453+0000","last_unstale":"2024-03-04T09:29:47.939032+0000","last_undegraded":"2024-03-04T09:29:47.939032+0000","last_fullsized":"2024-03-04T09:29:47.939032+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:18:06.867475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"80'4","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.945186+0000","last_change":"2024-03-04T09:29:46.905791+0000","last_active":"2024-03-04T09:29:47.945186+0000","last_peered":"2024-03-04T09:29:47.945186+0000","last_clean":"2024-03-04T09:29:47.945186+0000","last_became_active":"2024-03-04T09:29:46.905461+0000","last_became_peered":"2024-03-04T09:29:46.905461+0000","last_unstale":"2024-03-04T09:29:47.945186+0000","last_undegraded":"2024-03-04T09:29:47.945186+0000","last_fullsized":"2024-03-04T09:29:47.945186+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_clean_scrub_stamp":"2024-03-04T09:29:45.844895+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:38:50.353890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.931037+0000","last_change":"2024-03-04T09:29:47.925958+0000","last_active":"2024-03-04T09:29:47.931037+0000","last_peered":"2024-03-04T09:29:47.931037+0000","last_clean":"2024-03-04T09:29:47.931037+0000","last_became_active":"2024-03-04T09:29:47.925407+0000","last_became_peered":"2024-03-04T09:29:47.925407+0000","last_unstale":"2024-03-04T09:29:47.931037+0000","last_undegraded":"2024-03-04T09:29:47.931037+0000","last_fullsized":"2024-03-04T09:29:47.931037+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:32:42.357098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"80'1","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.944661+0000","last_change":"2024-03-04T09:29:46.905848+0000","last_active":"2024-03-04T09:29:47.944661+0000","last_peered":"2024-03-04T09:29:47.944661+0000","last_clean":"2024-03-04T09:29:47.944661+0000","last_became_active":"2024-03-04T09:29:46.904393+0000","last_became_peered":"2024-03-04T09:29:46.904393+0000","last_unstale":"2024-03-04T09:29:47.944661+0000","last_undegraded":"2024-03-04T09:29:47.944661+0000","last_fullsized":"2024-03-04T09:29:47.944661+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_clean_scrub_stamp":"2024-03-04T09:29:45.844895+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:35:13.678169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.933295+0000","last_change":"2024-03-04T09:29:47.930427+0000","last_active":"2024-03-04T09:29:47.933295+0000","last_peered":"2024-03-04T09:29:47.933295+0000","last_clean":"2024-03-04T09:29:47.933295+0000","last_became_active":"2024-03-04T09:29:47.930015+0000","last_became_peered":"2024-03-04T09:29:47.930015+0000","last_unstale":"2024-03-04T09:29:47.933295+0000","last_undegraded":"2024-03-04T09:29:47.933295+0000","last_fullsized":"2024-03-04T09:29:47.933295+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:18:06.867475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"80'2","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.945158+0000","last_change":"2024-03-04T09:29:46.901411+0000","last_active":"2024-03-04T09:29:47.945158+0000","last_peered":"2024-03-04T09:29:47.945158+0000","last_clean":"2024-03-04T09:29:47.945158+0000","last_became_active":"2024-03-04T09:29:46.899957+0000","last_became_peered":"2024-03-04T09:29:46.899957+0000","last_unstale":"2024-03-04T09:29:47.945158+0000","last_undegraded":"2024-03-04T09:29:47.945158+0000","last_fullsized":"2024-03-04T09:29:47.945158+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_clean_scrub_stamp":"2024-03-04T09:29:45.844895+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:43:41.381702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.936925+0000","last_change":"2024-03-04T09:29:47.933993+0000","last_active":"2024-03-04T09:29:47.936925+0000","last_peered":"2024-03-04T09:29:47.936925+0000","last_clean":"2024-03-04T09:29:47.936925+0000","last_became_active":"2024-03-04T09:29:47.933664+0000","last_became_peered":"2024-03-04T09:29:47.933664+0000","last_unstale":"2024-03-04T09:29:47.936925+0000","last_undegraded":"2024-03-04T09:29:47.936925+0000","last_fullsized":"2024-03-04T09:29:47.936925+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:44:08.304833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"80'2","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.944800+0000","last_change":"2024-03-04T09:29:46.896760+0000","last_active":"2024-03-04T09:29:47.944800+0000","last_peered":"2024-03-04T09:29:47.944800+0000","last_clean":"2024-03-04T09:29:47.944800+0000","last_became_active":"2024-03-04T09:29:46.896407+0000","last_became_peered":"2024-03-04T09:29:46.896407+0000","last_unstale":"2024-03-04T09:29:47.944800+0000","last_undegraded":"2024-03-04T09:29:47.944800+0000","last_fullsized":"2024-03-04T09:29:47.944800+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_clean_scrub_stamp":"2024-03-04T09:29:45.844895+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:31:16.574548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.937026+0000","last_change":"2024-03-04T09:29:47.930603+0000","last_active":"2024-03-04T09:29:47.937026+0000","last_peered":"2024-03-04T09:29:47.937026+0000","last_clean":"2024-03-04T09:29:47.937026+0000","last_became_active":"2024-03-04T09:29:47.930056+0000","last_became_peered":"2024-03-04T09:29:47.930056+0000","last_unstale":"2024-03-04T09:29:47.937026+0000","last_undegraded":"2024-03-04T09:29:47.937026+0000","last_fullsized":"2024-03-04T09:29:47.937026+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:05:45.936048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"80'1","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.944620+0000","last_change":"2024-03-04T09:29:46.906308+0000","last_active":"2024-03-04T09:29:47.944620+0000","last_peered":"2024-03-04T09:29:47.944620+0000","last_clean":"2024-03-04T09:29:47.944620+0000","last_became_active":"2024-03-04T09:29:46.905284+0000","last_became_peered":"2024-03-04T09:29:46.905284+0000","last_unstale":"2024-03-04T09:29:47.944620+0000","last_undegraded":"2024-03-04T09:29:47.944620+0000","last_fullsized":"2024-03-04T09:29:47.944620+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_clean_scrub_stamp":"2024-03-04T09:29:45.844895+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:19:20.417148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.938998+0000","last_change":"2024-03-04T09:29:47.935922+0000","last_active":"2024-03-04T09:29:47.938998+0000","last_peered":"2024-03-04T09:29:47.938998+0000","last_clean":"2024-03-04T09:29:47.938998+0000","last_became_active":"2024-03-04T09:29:47.935607+0000","last_became_peered":"2024-03-04T09:29:47.935607+0000","last_unstale":"2024-03-04T09:29:47.938998+0000","last_undegraded":"2024-03-04T09:29:47.938998+0000","last_fullsized":"2024-03-04T09:29:47.938998+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:11:36.963320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"80'1","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.941894+0000","last_change":"2024-03-04T09:29:46.904602+0000","last_active":"2024-03-04T09:29:47.941894+0000","last_peered":"2024-03-04T09:29:47.941894+0000","last_clean":"2024-03-04T09:29:47.941894+0000","last_became_active":"2024-03-04T09:29:46.903831+0000","last_became_peered":"2024-03-04T09:29:46.903831+0000","last_unstale":"2024-03-04T09:29:47.941894+0000","last_undegraded":"2024-03-04T09:29:47.941894+0000","last_fullsized":"2024-03-04T09:29:47.941894+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_clean_scrub_stamp":"2024-03-04T09:29:45.844895+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:26:39.557924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.930529+0000","last_change":"2024-03-04T09:29:47.922387+0000","last_active":"2024-03-04T09:29:47.930529+0000","last_peered":"2024-03-04T09:29:47.930529+0000","last_clean":"2024-03-04T09:29:47.930529+0000","last_became_active":"2024-03-04T09:29:47.921863+0000","last_became_peered":"2024-03-04T09:29:47.921863+0000","last_unstale":"2024-03-04T09:29:47.930529+0000","last_undegraded":"2024-03-04T09:29:47.930529+0000","last_fullsized":"2024-03-04T09:29:47.930529+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:49:19.951736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.907403+0000","last_change":"2024-03-04T09:29:46.895184+0000","last_active":"2024-03-04T09:29:47.907403+0000","last_peered":"2024-03-04T09:29:47.907403+0000","last_clean":"2024-03-04T09:29:47.907403+0000","last_became_active":"2024-03-04T09:29:46.894192+0000","last_became_peered":"2024-03-04T09:29:46.894192+0000","last_unstale":"2024-03-04T09:29:47.907403+0000","last_undegraded":"2024-03-04T09:29:47.907403+0000","last_fullsized":"2024-03-04T09:29:47.907403+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_clean_scrub_stamp":"2024-03-04T09:29:45.844895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:49:18.936422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.928199+0000","last_change":"2024-03-04T09:29:47.920278+0000","last_active":"2024-03-04T09:29:47.928199+0000","last_peered":"2024-03-04T09:29:47.928199+0000","last_clean":"2024-03-04T09:29:47.928199+0000","last_became_active":"2024-03-04T09:29:47.919571+0000","last_became_peered":"2024-03-04T09:29:47.919571+0000","last_unstale":"2024-03-04T09:29:47.928199+0000","last_undegraded":"2024-03-04T09:29:47.928199+0000","last_fullsized":"2024-03-04T09:29:47.928199+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:48:59.581604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.891360+0000","last_change":"2024-03-04T09:29:46.908478+0000","last_active":"2024-03-04T09:29:47.891360+0000","last_peered":"2024-03-04T09:29:47.891360+0000","last_clean":"2024-03-04T09:29:47.891360+0000","last_became_active":"2024-03-04T09:29:46.908115+0000","last_became_peered":"2024-03-04T09:29:46.908115+0000","last_unstale":"2024-03-04T09:29:47.891360+0000","last_undegraded":"2024-03-04T09:29:47.891360+0000","last_fullsized":"2024-03-04T09:29:47.891360+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_clean_scrub_stamp":"2024-03-04T09:29:45.844895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:32:40.899263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.928004+0000","last_change":"2024-03-04T09:29:47.917693+0000","last_active":"2024-03-04T09:29:47.928004+0000","last_peered":"2024-03-04T09:29:47.928004+0000","last_clean":"2024-03-04T09:29:47.928004+0000","last_became_active":"2024-03-04T09:29:47.917341+0000","last_became_peered":"2024-03-04T09:29:47.917341+0000","last_unstale":"2024-03-04T09:29:47.928004+0000","last_undegraded":"2024-03-04T09:29:47.928004+0000","last_fullsized":"2024-03-04T09:29:47.928004+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:00:59.073724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.908665+0000","last_change":"2024-03-04T09:29:46.910036+0000","last_active":"2024-03-04T09:29:47.908665+0000","last_peered":"2024-03-04T09:29:47.908665+0000","last_clean":"2024-03-04T09:29:47.908665+0000","last_became_active":"2024-03-04T09:29:46.909728+0000","last_became_peered":"2024-03-04T09:29:46.909728+0000","last_unstale":"2024-03-04T09:29:47.908665+0000","last_undegraded":"2024-03-04T09:29:47.908665+0000","last_fullsized":"2024-03-04T09:29:47.908665+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_clean_scrub_stamp":"2024-03-04T09:29:45.844895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:29:49.767886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.925561+0000","last_change":"2024-03-04T09:29:47.923787+0000","last_active":"2024-03-04T09:29:47.925561+0000","last_peered":"2024-03-04T09:29:47.925561+0000","last_clean":"2024-03-04T09:29:47.925561+0000","last_became_active":"2024-03-04T09:29:47.923446+0000","last_became_peered":"2024-03-04T09:29:47.923446+0000","last_unstale":"2024-03-04T09:29:47.925561+0000","last_undegraded":"2024-03-04T09:29:47.925561+0000","last_fullsized":"2024-03-04T09:29:47.925561+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:32:32.654222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.910642+0000","last_change":"2024-03-04T09:29:46.908443+0000","last_active":"2024-03-04T09:29:47.910642+0000","last_peered":"2024-03-04T09:29:47.910642+0000","last_clean":"2024-03-04T09:29:47.910642+0000","last_became_active":"2024-03-04T09:29:46.908174+0000","last_became_peered":"2024-03-04T09:29:46.908174+0000","last_unstale":"2024-03-04T09:29:47.910642+0000","last_undegraded":"2024-03-04T09:29:47.910642+0000","last_fullsized":"2024-03-04T09:29:47.910642+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_clean_scrub_stamp":"2024-03-04T09:29:45.844895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:26:39.557924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.933883+0000","last_change":"2024-03-04T09:29:47.929520+0000","last_active":"2024-03-04T09:29:47.933883+0000","last_peered":"2024-03-04T09:29:47.933883+0000","last_clean":"2024-03-04T09:29:47.933883+0000","last_became_active":"2024-03-04T09:29:47.929216+0000","last_became_peered":"2024-03-04T09:29:47.929216+0000","last_unstale":"2024-03-04T09:29:47.933883+0000","last_undegraded":"2024-03-04T09:29:47.933883+0000","last_fullsized":"2024-03-04T09:29:47.933883+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:55:33.422040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"80'3","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.945240+0000","last_change":"2024-03-04T09:29:46.899987+0000","last_active":"2024-03-04T09:29:47.945240+0000","last_peered":"2024-03-04T09:29:47.945240+0000","last_clean":"2024-03-04T09:29:47.945240+0000","last_became_active":"2024-03-04T09:29:46.899560+0000","last_became_peered":"2024-03-04T09:29:46.899560+0000","last_unstale":"2024-03-04T09:29:47.945240+0000","last_undegraded":"2024-03-04T09:29:47.945240+0000","last_fullsized":"2024-03-04T09:29:47.945240+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_clean_scrub_stamp":"2024-03-04T09:29:45.844895+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:14:05.615270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.934188+0000","last_change":"2024-03-04T09:29:47.927722+0000","last_active":"2024-03-04T09:29:47.934188+0000","last_peered":"2024-03-04T09:29:47.934188+0000","last_clean":"2024-03-04T09:29:47.934188+0000","last_became_active":"2024-03-04T09:29:47.927232+0000","last_became_peered":"2024-03-04T09:29:47.927232+0000","last_unstale":"2024-03-04T09:29:47.934188+0000","last_undegraded":"2024-03-04T09:29:47.934188+0000","last_fullsized":"2024-03-04T09:29:47.934188+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:10:47.491712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"80'6","reported_seq":18,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.946983+0000","last_change":"2024-03-04T09:29:46.889387+0000","last_active":"2024-03-04T09:29:47.946983+0000","last_peered":"2024-03-04T09:29:47.946983+0000","last_clean":"2024-03-04T09:29:47.946983+0000","last_became_active":"2024-03-04T09:29:46.887601+0000","last_became_peered":"2024-03-04T09:29:46.887601+0000","last_unstale":"2024-03-04T09:29:47.946983+0000","last_undegraded":"2024-03-04T09:29:47.946983+0000","last_fullsized":"2024-03-04T09:29:47.946983+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_clean_scrub_stamp":"2024-03-04T09:29:45.844895+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:41:02.095286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.932178+0000","last_change":"2024-03-04T09:29:47.927748+0000","last_active":"2024-03-04T09:29:47.932178+0000","last_peered":"2024-03-04T09:29:47.932178+0000","last_clean":"2024-03-04T09:29:47.932178+0000","last_became_active":"2024-03-04T09:29:47.926299+0000","last_became_peered":"2024-03-04T09:29:47.926299+0000","last_unstale":"2024-03-04T09:29:47.932178+0000","last_undegraded":"2024-03-04T09:29:47.932178+0000","last_fullsized":"2024-03-04T09:29:47.932178+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:14:23.576549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"80'2","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.895387+0000","last_change":"2024-03-04T09:29:46.899282+0000","last_active":"2024-03-04T09:29:47.895387+0000","last_peered":"2024-03-04T09:29:47.895387+0000","last_clean":"2024-03-04T09:29:47.895387+0000","last_became_active":"2024-03-04T09:29:46.897870+0000","last_became_peered":"2024-03-04T09:29:46.897870+0000","last_unstale":"2024-03-04T09:29:47.895387+0000","last_undegraded":"2024-03-04T09:29:47.895387+0000","last_fullsized":"2024-03-04T09:29:47.895387+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_clean_scrub_stamp":"2024-03-04T09:29:45.844895+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:05:23.250531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.924875+0000","last_change":"2024-03-04T09:29:47.917902+0000","last_active":"2024-03-04T09:29:47.924875+0000","last_peered":"2024-03-04T09:29:47.924875+0000","last_clean":"2024-03-04T09:29:47.924875+0000","last_became_active":"2024-03-04T09:29:47.917601+0000","last_became_peered":"2024-03-04T09:29:47.917601+0000","last_unstale":"2024-03-04T09:29:47.924875+0000","last_undegraded":"2024-03-04T09:29:47.924875+0000","last_fullsized":"2024-03-04T09:29:47.924875+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:18:06.867475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.903997+0000","last_change":"2024-03-04T09:29:46.878519+0000","last_active":"2024-03-04T09:29:47.903997+0000","last_peered":"2024-03-04T09:29:47.903997+0000","last_clean":"2024-03-04T09:29:47.903997+0000","last_became_active":"2024-03-04T09:29:46.877481+0000","last_became_peered":"2024-03-04T09:29:46.877481+0000","last_unstale":"2024-03-04T09:29:47.903997+0000","last_undegraded":"2024-03-04T09:29:47.903997+0000","last_fullsized":"2024-03-04T09:29:47.903997+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_clean_scrub_stamp":"2024-03-04T09:29:45.844895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:20:58.488539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.941271+0000","last_change":"2024-03-04T09:29:47.932464+0000","last_active":"2024-03-04T09:29:47.941271+0000","last_peered":"2024-03-04T09:29:47.941271+0000","last_clean":"2024-03-04T09:29:47.941271+0000","last_became_active":"2024-03-04T09:29:47.932190+0000","last_became_peered":"2024-03-04T09:29:47.932190+0000","last_unstale":"2024-03-04T09:29:47.941271+0000","last_undegraded":"2024-03-04T09:29:47.941271+0000","last_fullsized":"2024-03-04T09:29:47.941271+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:58:07.029154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.895695+0000","last_change":"2024-03-04T09:29:46.907772+0000","last_active":"2024-03-04T09:29:47.895695+0000","last_peered":"2024-03-04T09:29:47.895695+0000","last_clean":"2024-03-04T09:29:47.895695+0000","last_became_active":"2024-03-04T09:29:46.907454+0000","last_became_peered":"2024-03-04T09:29:46.907454+0000","last_unstale":"2024-03-04T09:29:47.895695+0000","last_undegraded":"2024-03-04T09:29:47.895695+0000","last_fullsized":"2024-03-04T09:29:47.895695+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_clean_scrub_stamp":"2024-03-04T09:29:45.844895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:19:20.417148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.930206+0000","last_change":"2024-03-04T09:29:47.925561+0000","last_active":"2024-03-04T09:29:47.930206+0000","last_peered":"2024-03-04T09:29:47.930206+0000","last_clean":"2024-03-04T09:29:47.930206+0000","last_became_active":"2024-03-04T09:29:47.925240+0000","last_became_peered":"2024-03-04T09:29:47.925240+0000","last_unstale":"2024-03-04T09:29:47.930206+0000","last_undegraded":"2024-03-04T09:29:47.930206+0000","last_fullsized":"2024-03-04T09:29:47.930206+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:13:17.716288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"80'8","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.944801+0000","last_change":"2024-03-04T09:29:46.904792+0000","last_active":"2024-03-04T09:29:47.944801+0000","last_peered":"2024-03-04T09:29:47.944801+0000","last_clean":"2024-03-04T09:29:47.944801+0000","last_became_active":"2024-03-04T09:29:46.904415+0000","last_became_peered":"2024-03-04T09:29:46.904415+0000","last_unstale":"2024-03-04T09:29:47.944801+0000","last_undegraded":"2024-03-04T09:29:47.944801+0000","last_fullsized":"2024-03-04T09:29:47.944801+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_clean_scrub_stamp":"2024-03-04T09:29:45.844895+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:33:08.484663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.923132+0000","last_change":"2024-03-04T09:29:47.917900+0000","last_active":"2024-03-04T09:29:47.923132+0000","last_peered":"2024-03-04T09:29:47.923132+0000","last_clean":"2024-03-04T09:29:47.923132+0000","last_became_active":"2024-03-04T09:29:47.917613+0000","last_became_peered":"2024-03-04T09:29:47.917613+0000","last_unstale":"2024-03-04T09:29:47.923132+0000","last_undegraded":"2024-03-04T09:29:47.923132+0000","last_fullsized":"2024-03-04T09:29:47.923132+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:50:00.557690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"80'1","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.942145+0000","last_change":"2024-03-04T09:29:46.899872+0000","last_active":"2024-03-04T09:29:47.942145+0000","last_peered":"2024-03-04T09:29:47.942145+0000","last_clean":"2024-03-04T09:29:47.942145+0000","last_became_active":"2024-03-04T09:29:46.899479+0000","last_became_peered":"2024-03-04T09:29:46.899479+0000","last_unstale":"2024-03-04T09:29:47.942145+0000","last_undegraded":"2024-03-04T09:29:47.942145+0000","last_fullsized":"2024-03-04T09:29:47.942145+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_clean_scrub_stamp":"2024-03-04T09:29:45.844895+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:57:45.922627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.931017+0000","last_change":"2024-03-04T09:29:47.925683+0000","last_active":"2024-03-04T09:29:47.931017+0000","last_peered":"2024-03-04T09:29:47.931017+0000","last_clean":"2024-03-04T09:29:47.931017+0000","last_became_active":"2024-03-04T09:29:47.924331+0000","last_became_peered":"2024-03-04T09:29:47.924331+0000","last_unstale":"2024-03-04T09:29:47.931017+0000","last_undegraded":"2024-03-04T09:29:47.931017+0000","last_fullsized":"2024-03-04T09:29:47.931017+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:10:47.491712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"80'2","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.944990+0000","last_change":"2024-03-04T09:29:46.907165+0000","last_active":"2024-03-04T09:29:47.944990+0000","last_peered":"2024-03-04T09:29:47.944990+0000","last_clean":"2024-03-04T09:29:47.944990+0000","last_became_active":"2024-03-04T09:29:46.906840+0000","last_became_peered":"2024-03-04T09:29:46.906840+0000","last_unstale":"2024-03-04T09:29:47.944990+0000","last_undegraded":"2024-03-04T09:29:47.944990+0000","last_fullsized":"2024-03-04T09:29:47.944990+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_clean_scrub_stamp":"2024-03-04T09:29:45.844895+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:44:07.289519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.934206+0000","last_change":"2024-03-04T09:29:47.930350+0000","last_active":"2024-03-04T09:29:47.934206+0000","last_peered":"2024-03-04T09:29:47.934206+0000","last_clean":"2024-03-04T09:29:47.934206+0000","last_became_active":"2024-03-04T09:29:47.929950+0000","last_became_peered":"2024-03-04T09:29:47.929950+0000","last_unstale":"2024-03-04T09:29:47.934206+0000","last_undegraded":"2024-03-04T09:29:47.934206+0000","last_fullsized":"2024-03-04T09:29:47.934206+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:13:42.546100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.909032+0000","last_change":"2024-03-04T09:29:46.893430+0000","last_active":"2024-03-04T09:29:47.909032+0000","last_peered":"2024-03-04T09:29:47.909032+0000","last_clean":"2024-03-04T09:29:47.909032+0000","last_became_active":"2024-03-04T09:29:46.893173+0000","last_became_peered":"2024-03-04T09:29:46.893173+0000","last_unstale":"2024-03-04T09:29:47.909032+0000","last_undegraded":"2024-03-04T09:29:47.909032+0000","last_fullsized":"2024-03-04T09:29:47.909032+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_clean_scrub_stamp":"2024-03-04T09:29:45.844895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:21:17.815826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.935154+0000","last_change":"2024-03-04T09:29:47.924678+0000","last_active":"2024-03-04T09:29:47.935154+0000","last_peered":"2024-03-04T09:29:47.935154+0000","last_clean":"2024-03-04T09:29:47.935154+0000","last_became_active":"2024-03-04T09:29:47.924404+0000","last_became_peered":"2024-03-04T09:29:47.924404+0000","last_unstale":"2024-03-04T09:29:47.935154+0000","last_undegraded":"2024-03-04T09:29:47.935154+0000","last_fullsized":"2024-03-04T09:29:47.935154+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:48:42.428966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"80'2","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.937300+0000","last_change":"2024-03-04T09:29:46.895983+0000","last_active":"2024-03-04T09:29:47.937300+0000","last_peered":"2024-03-04T09:29:47.937300+0000","last_clean":"2024-03-04T09:29:47.937300+0000","last_became_active":"2024-03-04T09:29:46.895680+0000","last_became_peered":"2024-03-04T09:29:46.895680+0000","last_unstale":"2024-03-04T09:29:47.937300+0000","last_undegraded":"2024-03-04T09:29:47.937300+0000","last_fullsized":"2024-03-04T09:29:47.937300+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_clean_scrub_stamp":"2024-03-04T09:29:45.844895+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:55:04.068008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.919738+0000","last_change":"2024-03-04T09:29:47.913695+0000","last_active":"2024-03-04T09:29:47.919738+0000","last_peered":"2024-03-04T09:29:47.919738+0000","last_clean":"2024-03-04T09:29:47.919738+0000","last_became_active":"2024-03-04T09:29:47.913345+0000","last_became_peered":"2024-03-04T09:29:47.913345+0000","last_unstale":"2024-03-04T09:29:47.919738+0000","last_undegraded":"2024-03-04T09:29:47.919738+0000","last_fullsized":"2024-03-04T09:29:47.919738+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:06:10.504206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.917068+0000","last_change":"2024-03-04T09:29:46.893771+0000","last_active":"2024-03-04T09:29:47.917068+0000","last_peered":"2024-03-04T09:29:47.917068+0000","last_clean":"2024-03-04T09:29:47.917068+0000","last_became_active":"2024-03-04T09:29:46.893410+0000","last_became_peered":"2024-03-04T09:29:46.893410+0000","last_unstale":"2024-03-04T09:29:47.917068+0000","last_undegraded":"2024-03-04T09:29:47.917068+0000","last_fullsized":"2024-03-04T09:29:47.917068+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_clean_scrub_stamp":"2024-03-04T09:29:45.844895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:06:47.404643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.934339+0000","last_change":"2024-03-04T09:29:47.927843+0000","last_active":"2024-03-04T09:29:47.934339+0000","last_peered":"2024-03-04T09:29:47.934339+0000","last_clean":"2024-03-04T09:29:47.934339+0000","last_became_active":"2024-03-04T09:29:47.927548+0000","last_became_peered":"2024-03-04T09:29:47.927548+0000","last_unstale":"2024-03-04T09:29:47.934339+0000","last_undegraded":"2024-03-04T09:29:47.934339+0000","last_fullsized":"2024-03-04T09:29:47.934339+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:51:21.354710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.912065+0000","last_change":"2024-03-04T09:29:46.900643+0000","last_active":"2024-03-04T09:29:47.912065+0000","last_peered":"2024-03-04T09:29:47.912065+0000","last_clean":"2024-03-04T09:29:47.912065+0000","last_became_active":"2024-03-04T09:29:46.900368+0000","last_became_peered":"2024-03-04T09:29:46.900368+0000","last_unstale":"2024-03-04T09:29:47.912065+0000","last_undegraded":"2024-03-04T09:29:47.912065+0000","last_fullsized":"2024-03-04T09:29:47.912065+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_clean_scrub_stamp":"2024-03-04T09:29:45.844895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:43:50.343554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.920128+0000","last_change":"2024-03-04T09:29:47.915790+0000","last_active":"2024-03-04T09:29:47.920128+0000","last_peered":"2024-03-04T09:29:47.920128+0000","last_clean":"2024-03-04T09:29:47.920128+0000","last_became_active":"2024-03-04T09:29:47.915338+0000","last_became_peered":"2024-03-04T09:29:47.915338+0000","last_unstale":"2024-03-04T09:29:47.920128+0000","last_undegraded":"2024-03-04T09:29:47.920128+0000","last_fullsized":"2024-03-04T09:29:47.920128+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:10:47.491712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.903647+0000","last_change":"2024-03-04T09:29:46.893503+0000","last_active":"2024-03-04T09:29:47.903647+0000","last_peered":"2024-03-04T09:29:47.903647+0000","last_clean":"2024-03-04T09:29:47.903647+0000","last_became_active":"2024-03-04T09:29:46.893069+0000","last_became_peered":"2024-03-04T09:29:46.893069+0000","last_unstale":"2024-03-04T09:29:47.903647+0000","last_undegraded":"2024-03-04T09:29:47.903647+0000","last_fullsized":"2024-03-04T09:29:47.903647+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_clean_scrub_stamp":"2024-03-04T09:29:45.844895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:00:54.135399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.932928+0000","last_change":"2024-03-04T09:29:47.926973+0000","last_active":"2024-03-04T09:29:47.932928+0000","last_peered":"2024-03-04T09:29:47.932928+0000","last_clean":"2024-03-04T09:29:47.932928+0000","last_became_active":"2024-03-04T09:29:47.926569+0000","last_became_peered":"2024-03-04T09:29:47.926569+0000","last_unstale":"2024-03-04T09:29:47.932928+0000","last_undegraded":"2024-03-04T09:29:47.932928+0000","last_fullsized":"2024-03-04T09:29:47.932928+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:31:16.169781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"80'2","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.893342+0000","last_change":"2024-03-04T09:29:46.896017+0000","last_active":"2024-03-04T09:29:47.893342+0000","last_peered":"2024-03-04T09:29:47.893342+0000","last_clean":"2024-03-04T09:29:47.893342+0000","last_became_active":"2024-03-04T09:29:46.893411+0000","last_became_peered":"2024-03-04T09:29:46.893411+0000","last_unstale":"2024-03-04T09:29:47.893342+0000","last_undegraded":"2024-03-04T09:29:47.893342+0000","last_fullsized":"2024-03-04T09:29:47.893342+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_clean_scrub_stamp":"2024-03-04T09:29:45.844895+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:47:04.045507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.936967+0000","last_change":"2024-03-04T09:29:47.930500+0000","last_active":"2024-03-04T09:29:47.936967+0000","last_peered":"2024-03-04T09:29:47.936967+0000","last_clean":"2024-03-04T09:29:47.936967+0000","last_became_active":"2024-03-04T09:29:47.930219+0000","last_became_peered":"2024-03-04T09:29:47.930219+0000","last_unstale":"2024-03-04T09:29:47.936967+0000","last_undegraded":"2024-03-04T09:29:47.936967+0000","last_fullsized":"2024-03-04T09:29:47.936967+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:33:09.499977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.895041+0000","last_change":"2024-03-04T09:29:46.888517+0000","last_active":"2024-03-04T09:29:47.895041+0000","last_peered":"2024-03-04T09:29:47.895041+0000","last_clean":"2024-03-04T09:29:47.895041+0000","last_became_active":"2024-03-04T09:29:46.888165+0000","last_became_peered":"2024-03-04T09:29:46.888165+0000","last_unstale":"2024-03-04T09:29:47.895041+0000","last_undegraded":"2024-03-04T09:29:47.895041+0000","last_fullsized":"2024-03-04T09:29:47.895041+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_clean_scrub_stamp":"2024-03-04T09:29:45.844895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:15:20.058129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.924913+0000","last_change":"2024-03-04T09:29:47.922765+0000","last_active":"2024-03-04T09:29:47.924913+0000","last_peered":"2024-03-04T09:29:47.924913+0000","last_clean":"2024-03-04T09:29:47.924913+0000","last_became_active":"2024-03-04T09:29:47.922427+0000","last_became_peered":"2024-03-04T09:29:47.922427+0000","last_unstale":"2024-03-04T09:29:47.924913+0000","last_undegraded":"2024-03-04T09:29:47.924913+0000","last_fullsized":"2024-03-04T09:29:47.924913+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:31:16.169781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.903453+0000","last_change":"2024-03-04T09:29:46.905513+0000","last_active":"2024-03-04T09:29:47.903453+0000","last_peered":"2024-03-04T09:29:47.903453+0000","last_clean":"2024-03-04T09:29:47.903453+0000","last_became_active":"2024-03-04T09:29:46.903818+0000","last_became_peered":"2024-03-04T09:29:46.903818+0000","last_unstale":"2024-03-04T09:29:47.903453+0000","last_undegraded":"2024-03-04T09:29:47.903453+0000","last_fullsized":"2024-03-04T09:29:47.903453+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_clean_scrub_stamp":"2024-03-04T09:29:45.844895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:35:07.573589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.934167+0000","last_change":"2024-03-04T09:29:47.927552+0000","last_active":"2024-03-04T09:29:47.934167+0000","last_peered":"2024-03-04T09:29:47.934167+0000","last_clean":"2024-03-04T09:29:47.934167+0000","last_became_active":"2024-03-04T09:29:47.926751+0000","last_became_peered":"2024-03-04T09:29:47.926751+0000","last_unstale":"2024-03-04T09:29:47.934167+0000","last_undegraded":"2024-03-04T09:29:47.934167+0000","last_fullsized":"2024-03-04T09:29:47.934167+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:06:10.504206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.908627+0000","last_change":"2024-03-04T09:29:46.907506+0000","last_active":"2024-03-04T09:29:47.908627+0000","last_peered":"2024-03-04T09:29:47.908627+0000","last_clean":"2024-03-04T09:29:47.908627+0000","last_became_active":"2024-03-04T09:29:46.906967+0000","last_became_peered":"2024-03-04T09:29:46.906967+0000","last_unstale":"2024-03-04T09:29:47.908627+0000","last_undegraded":"2024-03-04T09:29:47.908627+0000","last_fullsized":"2024-03-04T09:29:47.908627+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_clean_scrub_stamp":"2024-03-04T09:29:45.844895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:26:07.581293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.918098+0000","last_change":"2024-03-04T09:29:47.911980+0000","last_active":"2024-03-04T09:29:47.918098+0000","last_peered":"2024-03-04T09:29:47.918098+0000","last_clean":"2024-03-04T09:29:47.918098+0000","last_became_active":"2024-03-04T09:29:47.911653+0000","last_became_peered":"2024-03-04T09:29:47.911653+0000","last_unstale":"2024-03-04T09:29:47.918098+0000","last_undegraded":"2024-03-04T09:29:47.918098+0000","last_fullsized":"2024-03-04T09:29:47.918098+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:31:16.169781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.908321+0000","last_change":"2024-03-04T09:29:46.909862+0000","last_active":"2024-03-04T09:29:47.908321+0000","last_peered":"2024-03-04T09:29:47.908321+0000","last_clean":"2024-03-04T09:29:47.908321+0000","last_became_active":"2024-03-04T09:29:46.909610+0000","last_became_peered":"2024-03-04T09:29:46.909610+0000","last_unstale":"2024-03-04T09:29:47.908321+0000","last_undegraded":"2024-03-04T09:29:47.908321+0000","last_fullsized":"2024-03-04T09:29:47.908321+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_clean_scrub_stamp":"2024-03-04T09:29:45.844895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:11:35.948007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.932275+0000","last_change":"2024-03-04T09:29:47.925936+0000","last_active":"2024-03-04T09:29:47.932275+0000","last_peered":"2024-03-04T09:29:47.932275+0000","last_clean":"2024-03-04T09:29:47.932275+0000","last_became_active":"2024-03-04T09:29:47.925668+0000","last_became_peered":"2024-03-04T09:29:47.925668+0000","last_unstale":"2024-03-04T09:29:47.932275+0000","last_undegraded":"2024-03-04T09:29:47.932275+0000","last_fullsized":"2024-03-04T09:29:47.932275+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:09:08.575816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.913044+0000","last_change":"2024-03-04T09:29:46.904786+0000","last_active":"2024-03-04T09:29:47.913044+0000","last_peered":"2024-03-04T09:29:47.913044+0000","last_clean":"2024-03-04T09:29:47.913044+0000","last_became_active":"2024-03-04T09:29:46.904494+0000","last_became_peered":"2024-03-04T09:29:46.904494+0000","last_unstale":"2024-03-04T09:29:47.913044+0000","last_undegraded":"2024-03-04T09:29:47.913044+0000","last_fullsized":"2024-03-04T09:29:47.913044+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_clean_scrub_stamp":"2024-03-04T09:29:45.844895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:13:42.118138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.935326+0000","last_change":"2024-03-04T09:29:47.927853+0000","last_active":"2024-03-04T09:29:47.935326+0000","last_peered":"2024-03-04T09:29:47.935326+0000","last_clean":"2024-03-04T09:29:47.935326+0000","last_became_active":"2024-03-04T09:29:47.926384+0000","last_became_peered":"2024-03-04T09:29:47.926384+0000","last_unstale":"2024-03-04T09:29:47.935326+0000","last_undegraded":"2024-03-04T09:29:47.935326+0000","last_fullsized":"2024-03-04T09:29:47.935326+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:31:17.589861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.911259+0000","last_change":"2024-03-04T09:29:46.907893+0000","last_active":"2024-03-04T09:29:47.911259+0000","last_peered":"2024-03-04T09:29:47.911259+0000","last_clean":"2024-03-04T09:29:47.911259+0000","last_became_active":"2024-03-04T09:29:46.906807+0000","last_became_peered":"2024-03-04T09:29:46.906807+0000","last_unstale":"2024-03-04T09:29:47.911259+0000","last_undegraded":"2024-03-04T09:29:47.911259+0000","last_fullsized":"2024-03-04T09:29:47.911259+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_clean_scrub_stamp":"2024-03-04T09:29:45.844895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:13:42.118138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.927515+0000","last_change":"2024-03-04T09:29:47.923223+0000","last_active":"2024-03-04T09:29:47.927515+0000","last_peered":"2024-03-04T09:29:47.927515+0000","last_clean":"2024-03-04T09:29:47.927515+0000","last_became_active":"2024-03-04T09:29:47.921790+0000","last_became_peered":"2024-03-04T09:29:47.921790+0000","last_unstale":"2024-03-04T09:29:47.927515+0000","last_undegraded":"2024-03-04T09:29:47.927515+0000","last_fullsized":"2024-03-04T09:29:47.927515+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:06:10.504206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.917465+0000","last_change":"2024-03-04T09:29:46.906266+0000","last_active":"2024-03-04T09:29:47.917465+0000","last_peered":"2024-03-04T09:29:47.917465+0000","last_clean":"2024-03-04T09:29:47.917465+0000","last_became_active":"2024-03-04T09:29:46.905334+0000","last_became_peered":"2024-03-04T09:29:46.905334+0000","last_unstale":"2024-03-04T09:29:47.917465+0000","last_undegraded":"2024-03-04T09:29:47.917465+0000","last_fullsized":"2024-03-04T09:29:47.917465+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_clean_scrub_stamp":"2024-03-04T09:29:45.844895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:41:02.095286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.905853+0000","last_change":"2024-03-04T09:29:47.902516+0000","last_active":"2024-03-04T09:29:47.905853+0000","last_peered":"2024-03-04T09:29:47.905853+0000","last_clean":"2024-03-04T09:29:47.905853+0000","last_became_active":"2024-03-04T09:29:47.901685+0000","last_became_peered":"2024-03-04T09:29:47.901685+0000","last_unstale":"2024-03-04T09:29:47.905853+0000","last_undegraded":"2024-03-04T09:29:47.905853+0000","last_fullsized":"2024-03-04T09:29:47.905853+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:51:23.265839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"80'1","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.944935+0000","last_change":"2024-03-04T09:29:46.908186+0000","last_active":"2024-03-04T09:29:47.944935+0000","last_peered":"2024-03-04T09:29:47.944935+0000","last_clean":"2024-03-04T09:29:47.944935+0000","last_became_active":"2024-03-04T09:29:46.907897+0000","last_became_peered":"2024-03-04T09:29:46.907897+0000","last_unstale":"2024-03-04T09:29:47.944935+0000","last_undegraded":"2024-03-04T09:29:47.944935+0000","last_fullsized":"2024-03-04T09:29:47.944935+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_clean_scrub_stamp":"2024-03-04T09:29:45.844895+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:13:41.530786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":283,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.905923+0000","last_change":"2024-03-04T09:25:09.316159+0000","last_active":"2024-03-04T09:29:47.905923+0000","last_peered":"2024-03-04T09:29:47.905923+0000","last_clean":"2024-03-04T09:29:47.905923+0000","last_became_active":"2024-03-04T09:25:09.313767+0000","last_became_peered":"2024-03-04T09:25:09.313767+0000","last_unstale":"2024-03-04T09:29:47.905923+0000","last_undegraded":"2024-03-04T09:29:47.905923+0000","last_fullsized":"2024-03-04T09:29:47.905923+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:08.110953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:08.110953+0000","last_clean_scrub_stamp":"2024-03-04T09:25:08.110953+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:34:21.745390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":3880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574907,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32712,"kb_used_data":1528,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339128,"statfs":{"total":96636764160,"available":96603267072,"internally_reserved":0,"allocated":1564672,"data_stored":1000045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574907,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32132,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339708,"statfs":{"total":96636764160,"available":96603860992,"internally_reserved":0,"allocated":970752,"data_stored":409677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32132,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339708,"statfs":{"total":96636764160,"available":96603860992,"internally_reserved":0,"allocated":970752,"data_stored":409677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574906,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32128,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339712,"statfs":{"total":96636764160,"available":96603865088,"internally_reserved":0,"allocated":970752,"data_stored":409677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574908,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32132,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339708,"statfs":{"total":96636764160,"available":96603860992,"internally_reserved":0,"allocated":970752,"data_stored":409677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574907,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32132,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339708,"statfs":{"total":96636764160,"available":96603860992,"internally_reserved":0,"allocated":970752,"data_stored":409677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574908,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32132,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339708,"statfs":{"total":96636764160,"available":96603860992,"internally_reserved":0,"allocated":970752,"data_stored":409677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574907,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32708,"kb_used_data":1528,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339132,"statfs":{"total":96636764160,"available":96603271168,"internally_reserved":0,"allocated":1564672,"data_stored":1000045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-04T09:29:52.385 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T09:29:52.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.665+0000 7f0b07200640 1 Processor -- start 2024-03-04T09:29:52.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.665+0000 7f0b07200640 1 -- start start 2024-03-04T09:29:52.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.667+0000 7f0b07200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f0b080b5740 0x7f0b080b5b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:52.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.667+0000 7f0b07200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f0b0809b120 0x7f0b080b1690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:52.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.667+0000 7f0b07200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0b0815f760 0x7f0b0815ea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:52.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.667+0000 7f0b07200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b080b6060 con 0x7f0b0809b120 2024-03-04T09:29:52.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.667+0000 7f0b07200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b0815d9b0 con 0x7f0b080b5740 2024-03-04T09:29:52.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.667+0000 7f0b07200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b0815dc90 con 0x7f0b0815f760 2024-03-04T09:29:52.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.668+0000 7f0afd400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f0b0809b120 0x7f0b080b1690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:52.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.668+0000 7f0afd400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f0b0809b120 0x7f0b080b1690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38728/0 (socket says 172.21.15.6:38728) 2024-03-04T09:29:52.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.668+0000 7f0afd400640 1 -- 172.21.15.6:0/3188568974 learned_addr learned my addr 172.21.15.6:0/3188568974 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:29:52.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.668+0000 7f0b05e00640 1 --2- 172.21.15.6:0/3188568974 >> v2:172.21.15.146:3300/0 conn(0x7f0b080b5740 0x7f0b080b5b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:52.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.668+0000 7f0b06800640 1 --2- 172.21.15.6:0/3188568974 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0b0815f760 0x7f0b0815ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:52.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.668+0000 7f0afd400640 1 -- 172.21.15.6:0/3188568974 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0b0815f760 msgr2=0x7f0b0815ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:52.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.668+0000 7f0afd400640 1 --2- 172.21.15.6:0/3188568974 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0b0815f760 0x7f0b0815ea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:52.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.668+0000 7f0afd400640 1 -- 172.21.15.6:0/3188568974 >> v2:172.21.15.146:3300/0 conn(0x7f0b080b5740 msgr2=0x7f0b080b5b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:52.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.668+0000 7f0afd400640 1 --2- 172.21.15.6:0/3188568974 >> v2:172.21.15.146:3300/0 conn(0x7f0b080b5740 0x7f0b080b5b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:52.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.668+0000 7f0afd400640 1 -- 172.21.15.6:0/3188568974 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b0815df70 con 0x7f0b0809b120 2024-03-04T09:29:52.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.669+0000 7f0afd400640 1 --2- 172.21.15.6:0/3188568974 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0b0809b120 0x7f0b080b1690 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f0b08074870 tx=0x7f0af002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2745b84807afc39 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:52.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.670+0000 7f0b05400640 1 -- 172.21.15.6:0/3188568974 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0af002e070 con 0x7f0b0809b120 2024-03-04T09:29:52.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.670+0000 7f0b05400640 1 -- 172.21.15.6:0/3188568974 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0af0037070 con 0x7f0b0809b120 2024-03-04T09:29:52.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.670+0000 7f0b05400640 1 -- 172.21.15.6:0/3188568974 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0af0031a00 con 0x7f0b0809b120 2024-03-04T09:29:52.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.670+0000 7f0b07200640 1 -- 172.21.15.6:0/3188568974 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0b0809b120 msgr2=0x7f0b080b1690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:52.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.671+0000 7f0b07200640 1 --2- 172.21.15.6:0/3188568974 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0b0809b120 0x7f0b080b1690 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f0b08074870 tx=0x7f0af002b880 comp rx=0 tx=0).stop 2024-03-04T09:29:52.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.671+0000 7f0b07200640 1 -- 172.21.15.6:0/3188568974 shutdown_connections 2024-03-04T09:29:52.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.671+0000 7f0b07200640 1 --2- 172.21.15.6:0/3188568974 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0b0815f760 0x7f0b0815ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:52.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.671+0000 7f0b07200640 1 --2- 172.21.15.6:0/3188568974 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0b0809b120 0x7f0b080b1690 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:52.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.671+0000 7f0b07200640 1 --2- 172.21.15.6:0/3188568974 >> v2:172.21.15.146:3300/0 conn(0x7f0b080b5740 0x7f0b080b5b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:52.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.671+0000 7f0b07200640 1 -- 172.21.15.6:0/3188568974 >> 172.21.15.6:0/3188568974 conn(0x7f0b080731f0 msgr2=0x7f0b0809aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:29:52.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.671+0000 7f0b07200640 1 -- 172.21.15.6:0/3188568974 shutdown_connections 2024-03-04T09:29:52.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.671+0000 7f0b07200640 1 -- 172.21.15.6:0/3188568974 wait complete. 2024-03-04T09:29:52.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.674+0000 7f0b07200640 1 Processor -- start 2024-03-04T09:29:52.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.674+0000 7f0b07200640 1 -- start start 2024-03-04T09:29:52.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.675+0000 7f0b07200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0b0809b120 0x7f0b081495c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:52.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.675+0000 7f0b07200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0b080b5740 0x7f0b081462c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:52.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.675+0000 7f0b07200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0b0815f760 0x7f0b080b0240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:52.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.675+0000 7f0b07200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b080b1bd0 con 0x7f0b0815f760 2024-03-04T09:29:52.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.675+0000 7f0b07200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b0815d660 con 0x7f0b080b5740 2024-03-04T09:29:52.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.675+0000 7f0b07200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b080b1d50 con 0x7f0b0809b120 2024-03-04T09:29:52.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.676+0000 7f0b06800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0b0815f760 0x7f0b080b0240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:52.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.676+0000 7f0b06800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0b0815f760 0x7f0b080b0240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38736/0 (socket says 172.21.15.6:38736) 2024-03-04T09:29:52.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.676+0000 7f0b06800640 1 -- 172.21.15.6:0/3449452945 learned_addr learned my addr 172.21.15.6:0/3449452945 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:29:52.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.676+0000 7f0afd400640 1 --2- 172.21.15.6:0/3449452945 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0b080b5740 0x7f0b081462c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:52.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.676+0000 7f0b05e00640 1 --2- 172.21.15.6:0/3449452945 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0b0809b120 0x7f0b081495c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:52.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.676+0000 7f0b06800640 1 -- 172.21.15.6:0/3449452945 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0b0809b120 msgr2=0x7f0b081495c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:52.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.676+0000 7f0b06800640 1 --2- 172.21.15.6:0/3449452945 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0b0809b120 0x7f0b081495c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:52.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.676+0000 7f0b06800640 1 -- 172.21.15.6:0/3449452945 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0b080b5740 msgr2=0x7f0b081462c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:52.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.676+0000 7f0b06800640 1 --2- 172.21.15.6:0/3449452945 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0b080b5740 0x7f0b081462c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:52.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.676+0000 7f0b06800640 1 -- 172.21.15.6:0/3449452945 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b080cd0c0 con 0x7f0b0815f760 2024-03-04T09:29:52.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.677+0000 7f0b06800640 1 --2- 172.21.15.6:0/3449452945 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0b0815f760 0x7f0b080b0240 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f0b00007e90 tx=0x7f0b0000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:52.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.677+0000 7f0afea00640 1 -- 172.21.15.6:0/3449452945 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b00014420 con 0x7f0b0815f760 2024-03-04T09:29:52.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.677+0000 7f0afea00640 1 -- 172.21.15.6:0/3449452945 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b00019070 con 0x7f0b0815f760 2024-03-04T09:29:52.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.677+0000 7f0afea00640 1 -- 172.21.15.6:0/3449452945 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b00014800 con 0x7f0b0815f760 2024-03-04T09:29:52.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.677+0000 7f0b07200640 1 -- 172.21.15.6:0/3449452945 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b080cd3a0 con 0x7f0b0815f760 2024-03-04T09:29:52.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.677+0000 7f0b07200640 1 -- 172.21.15.6:0/3449452945 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b080cd960 con 0x7f0b0815f760 2024-03-04T09:29:52.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.678+0000 7f0b07200640 1 -- 172.21.15.6:0/3449452945 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b080cde50 con 0x7f0b0815f760 2024-03-04T09:29:52.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.680+0000 7f0afea00640 1 -- 172.21.15.6:0/3449452945 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0b000149a0 con 0x7f0b0815f760 2024-03-04T09:29:52.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.681+0000 7f0afea00640 1 --2- 172.21.15.6:0/3449452945 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f0ae406b4d0 0x7f0ae406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:52.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.681+0000 7f0afea00640 1 -- 172.21.15.6:0/3449452945 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0b0008aa40 con 0x7f0b0815f760 2024-03-04T09:29:52.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.682+0000 7f0b05e00640 1 --2- 172.21.15.6:0/3449452945 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f0ae406b4d0 0x7f0ae406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:52.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.683+0000 7f0b05e00640 1 --2- 172.21.15.6:0/3449452945 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f0ae406b4d0 0x7f0ae406d990 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f0af8006fd0 tx=0x7f0af8008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:52.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.686+0000 7f0afea00640 1 -- 172.21.15.6:0/3449452945 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0b000598b0 con 0x7f0b0815f760 2024-03-04T09:29:52.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.886+0000 7f0b07200640 1 -- 172.21.15.6:0/3449452945 --> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0b08072f90 con 0x7f0ae406b4d0 2024-03-04T09:29:52.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.905+0000 7f0afea00640 1 -- 172.21.15.6:0/3449452945 <== mgr.4113 v2:172.21.15.146:6832/1788613334 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252000 (secure 0 0 0) 0x7f0b08072f90 con 0x7f0ae406b4d0 2024-03-04T09:29:52.906 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:29:52.907 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-03-04T09:29:52.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.912+0000 7f0b07200640 1 -- 172.21.15.6:0/3449452945 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f0ae406b4d0 msgr2=0x7f0ae406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:52.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.912+0000 7f0b07200640 1 --2- 172.21.15.6:0/3449452945 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f0ae406b4d0 0x7f0ae406d990 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f0af8006fd0 tx=0x7f0af8008040 comp rx=0 tx=0).stop 2024-03-04T09:29:52.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.912+0000 7f0b07200640 1 -- 172.21.15.6:0/3449452945 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0b0815f760 msgr2=0x7f0b080b0240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:52.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.912+0000 7f0b07200640 1 --2- 172.21.15.6:0/3449452945 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0b0815f760 0x7f0b080b0240 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f0b00007e90 tx=0x7f0b0000e4b0 comp rx=0 tx=0).stop 2024-03-04T09:29:52.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.912+0000 7f0b07200640 1 -- 172.21.15.6:0/3449452945 shutdown_connections 2024-03-04T09:29:52.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.912+0000 7f0b07200640 1 --2- 172.21.15.6:0/3449452945 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0b0815f760 0x7f0b080b0240 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:52.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.912+0000 7f0b07200640 1 --2- 172.21.15.6:0/3449452945 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0b080b5740 0x7f0b081462c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:52.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.912+0000 7f0b07200640 1 --2- 172.21.15.6:0/3449452945 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f0ae406b4d0 0x7f0ae406d990 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:52.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.912+0000 7f0b07200640 1 --2- 172.21.15.6:0/3449452945 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0b0809b120 0x7f0b081495c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:52.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.912+0000 7f0b07200640 1 -- 172.21.15.6:0/3449452945 >> 172.21.15.6:0/3449452945 conn(0x7f0b080731f0 msgr2=0x7f0b080b2c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:29:52.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.912+0000 7f0b07200640 1 -- 172.21.15.6:0/3449452945 shutdown_connections 2024-03-04T09:29:52.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:52.912+0000 7f0b07200640 1 -- 172.21.15.6:0/3449452945 wait complete. 2024-03-04T09:29:52.928 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":216,"stamp":"2024-03-04T09:29:51.957304+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":43,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":258208,"kb_used_data":8744,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754716512,"statfs":{"total":773094113280,"available":772829708288,"internally_reserved":0,"allocated":8953856,"data_stored":4458152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001694"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.931056+0000","last_change":"2024-03-04T09:29:47.926163+0000","last_active":"2024-03-04T09:29:47.931056+0000","last_peered":"2024-03-04T09:29:47.931056+0000","last_clean":"2024-03-04T09:29:47.931056+0000","last_became_active":"2024-03-04T09:29:47.925792+0000","last_became_peered":"2024-03-04T09:29:47.925792+0000","last_unstale":"2024-03-04T09:29:47.931056+0000","last_undegraded":"2024-03-04T09:29:47.931056+0000","last_fullsized":"2024-03-04T09:29:47.931056+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:32:32.654222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.927761+0000","last_change":"2024-03-04T09:29:47.921497+0000","last_active":"2024-03-04T09:29:47.927761+0000","last_peered":"2024-03-04T09:29:47.927761+0000","last_clean":"2024-03-04T09:29:47.927761+0000","last_became_active":"2024-03-04T09:29:47.921167+0000","last_became_peered":"2024-03-04T09:29:47.921167+0000","last_unstale":"2024-03-04T09:29:47.927761+0000","last_undegraded":"2024-03-04T09:29:47.927761+0000","last_fullsized":"2024-03-04T09:29:47.927761+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:41:37.529066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.929202+0000","last_change":"2024-03-04T09:29:47.924582+0000","last_active":"2024-03-04T09:29:47.929202+0000","last_peered":"2024-03-04T09:29:47.929202+0000","last_clean":"2024-03-04T09:29:47.929202+0000","last_became_active":"2024-03-04T09:29:47.924275+0000","last_became_peered":"2024-03-04T09:29:47.924275+0000","last_unstale":"2024-03-04T09:29:47.929202+0000","last_undegraded":"2024-03-04T09:29:47.929202+0000","last_fullsized":"2024-03-04T09:29:47.929202+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:11:36.963320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.934625+0000","last_change":"2024-03-04T09:29:47.927066+0000","last_active":"2024-03-04T09:29:47.934625+0000","last_peered":"2024-03-04T09:29:47.934625+0000","last_clean":"2024-03-04T09:29:47.934625+0000","last_became_active":"2024-03-04T09:29:47.926721+0000","last_became_peered":"2024-03-04T09:29:47.926721+0000","last_unstale":"2024-03-04T09:29:47.934625+0000","last_undegraded":"2024-03-04T09:29:47.934625+0000","last_fullsized":"2024-03-04T09:29:47.934625+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:13:39.210481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.938976+0000","last_change":"2024-03-04T09:29:47.935195+0000","last_active":"2024-03-04T09:29:47.938976+0000","last_peered":"2024-03-04T09:29:47.938976+0000","last_clean":"2024-03-04T09:29:47.938976+0000","last_became_active":"2024-03-04T09:29:47.934932+0000","last_became_peered":"2024-03-04T09:29:47.934932+0000","last_unstale":"2024-03-04T09:29:47.938976+0000","last_undegraded":"2024-03-04T09:29:47.938976+0000","last_fullsized":"2024-03-04T09:29:47.938976+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:49:19.951736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.932066+0000","last_change":"2024-03-04T09:29:47.929146+0000","last_active":"2024-03-04T09:29:47.932066+0000","last_peered":"2024-03-04T09:29:47.932066+0000","last_clean":"2024-03-04T09:29:47.932066+0000","last_became_active":"2024-03-04T09:29:47.928691+0000","last_became_peered":"2024-03-04T09:29:47.928691+0000","last_unstale":"2024-03-04T09:29:47.932066+0000","last_undegraded":"2024-03-04T09:29:47.932066+0000","last_fullsized":"2024-03-04T09:29:47.932066+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:13:42.546100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.934459+0000","last_change":"2024-03-04T09:29:47.929052+0000","last_active":"2024-03-04T09:29:47.934459+0000","last_peered":"2024-03-04T09:29:47.934459+0000","last_clean":"2024-03-04T09:29:47.934459+0000","last_became_active":"2024-03-04T09:29:47.928609+0000","last_became_peered":"2024-03-04T09:29:47.928609+0000","last_unstale":"2024-03-04T09:29:47.934459+0000","last_undegraded":"2024-03-04T09:29:47.934459+0000","last_fullsized":"2024-03-04T09:29:47.934459+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:44:09.397551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.920344+0000","last_change":"2024-03-04T09:29:47.911061+0000","last_active":"2024-03-04T09:29:47.920344+0000","last_peered":"2024-03-04T09:29:47.920344+0000","last_clean":"2024-03-04T09:29:47.920344+0000","last_became_active":"2024-03-04T09:29:47.909106+0000","last_became_peered":"2024-03-04T09:29:47.909106+0000","last_unstale":"2024-03-04T09:29:47.920344+0000","last_undegraded":"2024-03-04T09:29:47.920344+0000","last_fullsized":"2024-03-04T09:29:47.920344+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:35:14.693482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.921330+0000","last_change":"2024-03-04T09:29:47.917439+0000","last_active":"2024-03-04T09:29:47.921330+0000","last_peered":"2024-03-04T09:29:47.921330+0000","last_clean":"2024-03-04T09:29:47.921330+0000","last_became_active":"2024-03-04T09:29:47.917005+0000","last_became_peered":"2024-03-04T09:29:47.917005+0000","last_unstale":"2024-03-04T09:29:47.921330+0000","last_undegraded":"2024-03-04T09:29:47.921330+0000","last_fullsized":"2024-03-04T09:29:47.921330+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:32:42.357098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.930971+0000","last_change":"2024-03-04T09:29:47.927014+0000","last_active":"2024-03-04T09:29:47.930971+0000","last_peered":"2024-03-04T09:29:47.930971+0000","last_clean":"2024-03-04T09:29:47.930971+0000","last_became_active":"2024-03-04T09:29:47.926689+0000","last_became_peered":"2024-03-04T09:29:47.926689+0000","last_unstale":"2024-03-04T09:29:47.930971+0000","last_undegraded":"2024-03-04T09:29:47.930971+0000","last_fullsized":"2024-03-04T09:29:47.930971+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:13:42.546100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.928021+0000","last_change":"2024-03-04T09:29:47.920696+0000","last_active":"2024-03-04T09:29:47.928021+0000","last_peered":"2024-03-04T09:29:47.928021+0000","last_clean":"2024-03-04T09:29:47.928021+0000","last_became_active":"2024-03-04T09:29:47.918831+0000","last_became_peered":"2024-03-04T09:29:47.918831+0000","last_unstale":"2024-03-04T09:29:47.928021+0000","last_undegraded":"2024-03-04T09:29:47.928021+0000","last_fullsized":"2024-03-04T09:29:47.928021+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:01:52.293904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.931170+0000","last_change":"2024-03-04T09:29:47.927745+0000","last_active":"2024-03-04T09:29:47.931170+0000","last_peered":"2024-03-04T09:29:47.931170+0000","last_clean":"2024-03-04T09:29:47.931170+0000","last_became_active":"2024-03-04T09:29:47.927105+0000","last_became_peered":"2024-03-04T09:29:47.927105+0000","last_unstale":"2024-03-04T09:29:47.931170+0000","last_undegraded":"2024-03-04T09:29:47.931170+0000","last_fullsized":"2024-03-04T09:29:47.931170+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:50:00.557690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.936182+0000","last_change":"2024-03-04T09:29:47.929993+0000","last_active":"2024-03-04T09:29:47.936182+0000","last_peered":"2024-03-04T09:29:47.936182+0000","last_clean":"2024-03-04T09:29:47.936182+0000","last_became_active":"2024-03-04T09:29:47.929381+0000","last_became_peered":"2024-03-04T09:29:47.929381+0000","last_unstale":"2024-03-04T09:29:47.936182+0000","last_undegraded":"2024-03-04T09:29:47.936182+0000","last_fullsized":"2024-03-04T09:29:47.936182+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:43:42.397015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.935177+0000","last_change":"2024-03-04T09:29:47.926135+0000","last_active":"2024-03-04T09:29:47.935177+0000","last_peered":"2024-03-04T09:29:47.935177+0000","last_clean":"2024-03-04T09:29:47.935177+0000","last_became_active":"2024-03-04T09:29:47.925848+0000","last_became_peered":"2024-03-04T09:29:47.925848+0000","last_unstale":"2024-03-04T09:29:47.935177+0000","last_undegraded":"2024-03-04T09:29:47.935177+0000","last_fullsized":"2024-03-04T09:29:47.935177+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:30:18.836859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.929741+0000","last_change":"2024-03-04T09:29:47.920506+0000","last_active":"2024-03-04T09:29:47.929741+0000","last_peered":"2024-03-04T09:29:47.929741+0000","last_clean":"2024-03-04T09:29:47.929741+0000","last_became_active":"2024-03-04T09:29:47.919954+0000","last_became_peered":"2024-03-04T09:29:47.919954+0000","last_unstale":"2024-03-04T09:29:47.929741+0000","last_undegraded":"2024-03-04T09:29:47.929741+0000","last_fullsized":"2024-03-04T09:29:47.929741+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:51:58.796350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.932909+0000","last_change":"2024-03-04T09:29:47.930263+0000","last_active":"2024-03-04T09:29:47.932909+0000","last_peered":"2024-03-04T09:29:47.932909+0000","last_clean":"2024-03-04T09:29:47.932909+0000","last_became_active":"2024-03-04T09:29:47.928895+0000","last_became_peered":"2024-03-04T09:29:47.928895+0000","last_unstale":"2024-03-04T09:29:47.932909+0000","last_undegraded":"2024-03-04T09:29:47.932909+0000","last_fullsized":"2024-03-04T09:29:47.932909+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:32:32.654222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.941355+0000","last_change":"2024-03-04T09:29:47.937624+0000","last_active":"2024-03-04T09:29:47.941355+0000","last_peered":"2024-03-04T09:29:47.941355+0000","last_clean":"2024-03-04T09:29:47.941355+0000","last_became_active":"2024-03-04T09:29:47.937306+0000","last_became_peered":"2024-03-04T09:29:47.937306+0000","last_unstale":"2024-03-04T09:29:47.941355+0000","last_undegraded":"2024-03-04T09:29:47.941355+0000","last_fullsized":"2024-03-04T09:29:47.941355+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:21:18.831139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.938481+0000","last_change":"2024-03-04T09:29:47.933722+0000","last_active":"2024-03-04T09:29:47.938481+0000","last_peered":"2024-03-04T09:29:47.938481+0000","last_clean":"2024-03-04T09:29:47.938481+0000","last_became_active":"2024-03-04T09:29:47.933363+0000","last_became_peered":"2024-03-04T09:29:47.933363+0000","last_unstale":"2024-03-04T09:29:47.938481+0000","last_undegraded":"2024-03-04T09:29:47.938481+0000","last_fullsized":"2024-03-04T09:29:47.938481+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:51:18.420239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.931407+0000","last_change":"2024-03-04T09:29:47.923456+0000","last_active":"2024-03-04T09:29:47.931407+0000","last_peered":"2024-03-04T09:29:47.931407+0000","last_clean":"2024-03-04T09:29:47.931407+0000","last_became_active":"2024-03-04T09:29:47.923113+0000","last_became_peered":"2024-03-04T09:29:47.923113+0000","last_unstale":"2024-03-04T09:29:47.931407+0000","last_undegraded":"2024-03-04T09:29:47.931407+0000","last_fullsized":"2024-03-04T09:29:47.931407+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:27:13.253074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.933645+0000","last_change":"2024-03-04T09:29:47.922349+0000","last_active":"2024-03-04T09:29:47.933645+0000","last_peered":"2024-03-04T09:29:47.933645+0000","last_clean":"2024-03-04T09:29:47.933645+0000","last_became_active":"2024-03-04T09:29:47.918752+0000","last_became_peered":"2024-03-04T09:29:47.918752+0000","last_unstale":"2024-03-04T09:29:47.933645+0000","last_undegraded":"2024-03-04T09:29:47.933645+0000","last_fullsized":"2024-03-04T09:29:47.933645+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:24:29.054506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.932862+0000","last_change":"2024-03-04T09:29:47.920982+0000","last_active":"2024-03-04T09:29:47.932862+0000","last_peered":"2024-03-04T09:29:47.932862+0000","last_clean":"2024-03-04T09:29:47.932862+0000","last_became_active":"2024-03-04T09:29:47.920480+0000","last_became_peered":"2024-03-04T09:29:47.920480+0000","last_unstale":"2024-03-04T09:29:47.932862+0000","last_undegraded":"2024-03-04T09:29:47.932862+0000","last_fullsized":"2024-03-04T09:29:47.932862+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:41:48.826232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.923083+0000","last_change":"2024-03-04T09:29:47.916934+0000","last_active":"2024-03-04T09:29:47.923083+0000","last_peered":"2024-03-04T09:29:47.923083+0000","last_clean":"2024-03-04T09:29:47.923083+0000","last_became_active":"2024-03-04T09:29:47.916618+0000","last_became_peered":"2024-03-04T09:29:47.916618+0000","last_unstale":"2024-03-04T09:29:47.923083+0000","last_undegraded":"2024-03-04T09:29:47.923083+0000","last_fullsized":"2024-03-04T09:29:47.923083+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:11:44.498334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.924238+0000","last_change":"2024-03-04T09:29:47.920358+0000","last_active":"2024-03-04T09:29:47.924238+0000","last_peered":"2024-03-04T09:29:47.924238+0000","last_clean":"2024-03-04T09:29:47.924238+0000","last_became_active":"2024-03-04T09:29:47.919453+0000","last_became_peered":"2024-03-04T09:29:47.919453+0000","last_unstale":"2024-03-04T09:29:47.924238+0000","last_undegraded":"2024-03-04T09:29:47.924238+0000","last_fullsized":"2024-03-04T09:29:47.924238+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:41:03.110600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.932790+0000","last_change":"2024-03-04T09:29:47.920639+0000","last_active":"2024-03-04T09:29:47.932790+0000","last_peered":"2024-03-04T09:29:47.932790+0000","last_clean":"2024-03-04T09:29:47.932790+0000","last_became_active":"2024-03-04T09:29:47.920361+0000","last_became_peered":"2024-03-04T09:29:47.920361+0000","last_unstale":"2024-03-04T09:29:47.932790+0000","last_undegraded":"2024-03-04T09:29:47.932790+0000","last_fullsized":"2024-03-04T09:29:47.932790+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:36:04.950142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.927617+0000","last_change":"2024-03-04T09:29:47.925548+0000","last_active":"2024-03-04T09:29:47.927617+0000","last_peered":"2024-03-04T09:29:47.927617+0000","last_clean":"2024-03-04T09:29:47.927617+0000","last_became_active":"2024-03-04T09:29:47.925214+0000","last_became_peered":"2024-03-04T09:29:47.925214+0000","last_unstale":"2024-03-04T09:29:47.927617+0000","last_undegraded":"2024-03-04T09:29:47.927617+0000","last_fullsized":"2024-03-04T09:29:47.927617+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:42:37.575160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.933675+0000","last_change":"2024-03-04T09:29:47.924998+0000","last_active":"2024-03-04T09:29:47.933675+0000","last_peered":"2024-03-04T09:29:47.933675+0000","last_clean":"2024-03-04T09:29:47.933675+0000","last_became_active":"2024-03-04T09:29:47.924658+0000","last_became_peered":"2024-03-04T09:29:47.924658+0000","last_unstale":"2024-03-04T09:29:47.933675+0000","last_undegraded":"2024-03-04T09:29:47.933675+0000","last_fullsized":"2024-03-04T09:29:47.933675+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:21:18.831139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.938246+0000","last_change":"2024-03-04T09:29:47.934371+0000","last_active":"2024-03-04T09:29:47.938246+0000","last_peered":"2024-03-04T09:29:47.938246+0000","last_clean":"2024-03-04T09:29:47.938246+0000","last_became_active":"2024-03-04T09:29:47.933995+0000","last_became_peered":"2024-03-04T09:29:47.933995+0000","last_unstale":"2024-03-04T09:29:47.938246+0000","last_undegraded":"2024-03-04T09:29:47.938246+0000","last_fullsized":"2024-03-04T09:29:47.938246+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:50:00.557690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.924893+0000","last_change":"2024-03-04T09:29:47.920745+0000","last_active":"2024-03-04T09:29:47.924893+0000","last_peered":"2024-03-04T09:29:47.924893+0000","last_clean":"2024-03-04T09:29:47.924893+0000","last_became_active":"2024-03-04T09:29:47.920426+0000","last_became_peered":"2024-03-04T09:29:47.920426+0000","last_unstale":"2024-03-04T09:29:47.924893+0000","last_undegraded":"2024-03-04T09:29:47.924893+0000","last_fullsized":"2024-03-04T09:29:47.924893+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:35:14.693482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.919501+0000","last_change":"2024-03-04T09:29:47.913136+0000","last_active":"2024-03-04T09:29:47.919501+0000","last_peered":"2024-03-04T09:29:47.919501+0000","last_clean":"2024-03-04T09:29:47.919501+0000","last_became_active":"2024-03-04T09:29:47.911494+0000","last_became_peered":"2024-03-04T09:29:47.911494+0000","last_unstale":"2024-03-04T09:29:47.919501+0000","last_undegraded":"2024-03-04T09:29:47.919501+0000","last_fullsized":"2024-03-04T09:29:47.919501+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:32:58.823873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.937050+0000","last_change":"2024-03-04T09:29:47.930841+0000","last_active":"2024-03-04T09:29:47.937050+0000","last_peered":"2024-03-04T09:29:47.937050+0000","last_clean":"2024-03-04T09:29:47.937050+0000","last_became_active":"2024-03-04T09:29:47.928656+0000","last_became_peered":"2024-03-04T09:29:47.928656+0000","last_unstale":"2024-03-04T09:29:47.937050+0000","last_undegraded":"2024-03-04T09:29:47.937050+0000","last_fullsized":"2024-03-04T09:29:47.937050+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:00:02.131925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.933106+0000","last_change":"2024-03-04T09:29:47.928549+0000","last_active":"2024-03-04T09:29:47.933106+0000","last_peered":"2024-03-04T09:29:47.933106+0000","last_clean":"2024-03-04T09:29:47.933106+0000","last_became_active":"2024-03-04T09:29:47.928247+0000","last_became_peered":"2024-03-04T09:29:47.928247+0000","last_unstale":"2024-03-04T09:29:47.933106+0000","last_undegraded":"2024-03-04T09:29:47.933106+0000","last_fullsized":"2024-03-04T09:29:47.933106+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:32:42.357098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.922907+0000","last_change":"2024-03-04T09:29:47.916919+0000","last_active":"2024-03-04T09:29:47.922907+0000","last_peered":"2024-03-04T09:29:47.922907+0000","last_clean":"2024-03-04T09:29:47.922907+0000","last_became_active":"2024-03-04T09:29:47.916462+0000","last_became_peered":"2024-03-04T09:29:47.916462+0000","last_unstale":"2024-03-04T09:29:47.922907+0000","last_undegraded":"2024-03-04T09:29:47.922907+0000","last_fullsized":"2024-03-04T09:29:47.922907+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:19:21.432462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.939032+0000","last_change":"2024-03-04T09:29:47.934744+0000","last_active":"2024-03-04T09:29:47.939032+0000","last_peered":"2024-03-04T09:29:47.939032+0000","last_clean":"2024-03-04T09:29:47.939032+0000","last_became_active":"2024-03-04T09:29:47.934453+0000","last_became_peered":"2024-03-04T09:29:47.934453+0000","last_unstale":"2024-03-04T09:29:47.939032+0000","last_undegraded":"2024-03-04T09:29:47.939032+0000","last_fullsized":"2024-03-04T09:29:47.939032+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:18:06.867475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"80'4","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.945186+0000","last_change":"2024-03-04T09:29:46.905791+0000","last_active":"2024-03-04T09:29:47.945186+0000","last_peered":"2024-03-04T09:29:47.945186+0000","last_clean":"2024-03-04T09:29:47.945186+0000","last_became_active":"2024-03-04T09:29:46.905461+0000","last_became_peered":"2024-03-04T09:29:46.905461+0000","last_unstale":"2024-03-04T09:29:47.945186+0000","last_undegraded":"2024-03-04T09:29:47.945186+0000","last_fullsized":"2024-03-04T09:29:47.945186+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_clean_scrub_stamp":"2024-03-04T09:29:45.844895+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:38:50.353890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.931037+0000","last_change":"2024-03-04T09:29:47.925958+0000","last_active":"2024-03-04T09:29:47.931037+0000","last_peered":"2024-03-04T09:29:47.931037+0000","last_clean":"2024-03-04T09:29:47.931037+0000","last_became_active":"2024-03-04T09:29:47.925407+0000","last_became_peered":"2024-03-04T09:29:47.925407+0000","last_unstale":"2024-03-04T09:29:47.931037+0000","last_undegraded":"2024-03-04T09:29:47.931037+0000","last_fullsized":"2024-03-04T09:29:47.931037+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:32:42.357098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"80'1","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.944661+0000","last_change":"2024-03-04T09:29:46.905848+0000","last_active":"2024-03-04T09:29:47.944661+0000","last_peered":"2024-03-04T09:29:47.944661+0000","last_clean":"2024-03-04T09:29:47.944661+0000","last_became_active":"2024-03-04T09:29:46.904393+0000","last_became_peered":"2024-03-04T09:29:46.904393+0000","last_unstale":"2024-03-04T09:29:47.944661+0000","last_undegraded":"2024-03-04T09:29:47.944661+0000","last_fullsized":"2024-03-04T09:29:47.944661+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_clean_scrub_stamp":"2024-03-04T09:29:45.844895+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:35:13.678169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.933295+0000","last_change":"2024-03-04T09:29:47.930427+0000","last_active":"2024-03-04T09:29:47.933295+0000","last_peered":"2024-03-04T09:29:47.933295+0000","last_clean":"2024-03-04T09:29:47.933295+0000","last_became_active":"2024-03-04T09:29:47.930015+0000","last_became_peered":"2024-03-04T09:29:47.930015+0000","last_unstale":"2024-03-04T09:29:47.933295+0000","last_undegraded":"2024-03-04T09:29:47.933295+0000","last_fullsized":"2024-03-04T09:29:47.933295+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:18:06.867475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"80'2","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.945158+0000","last_change":"2024-03-04T09:29:46.901411+0000","last_active":"2024-03-04T09:29:47.945158+0000","last_peered":"2024-03-04T09:29:47.945158+0000","last_clean":"2024-03-04T09:29:47.945158+0000","last_became_active":"2024-03-04T09:29:46.899957+0000","last_became_peered":"2024-03-04T09:29:46.899957+0000","last_unstale":"2024-03-04T09:29:47.945158+0000","last_undegraded":"2024-03-04T09:29:47.945158+0000","last_fullsized":"2024-03-04T09:29:47.945158+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_clean_scrub_stamp":"2024-03-04T09:29:45.844895+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:43:41.381702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.936925+0000","last_change":"2024-03-04T09:29:47.933993+0000","last_active":"2024-03-04T09:29:47.936925+0000","last_peered":"2024-03-04T09:29:47.936925+0000","last_clean":"2024-03-04T09:29:47.936925+0000","last_became_active":"2024-03-04T09:29:47.933664+0000","last_became_peered":"2024-03-04T09:29:47.933664+0000","last_unstale":"2024-03-04T09:29:47.936925+0000","last_undegraded":"2024-03-04T09:29:47.936925+0000","last_fullsized":"2024-03-04T09:29:47.936925+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:44:08.304833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"80'2","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.944800+0000","last_change":"2024-03-04T09:29:46.896760+0000","last_active":"2024-03-04T09:29:47.944800+0000","last_peered":"2024-03-04T09:29:47.944800+0000","last_clean":"2024-03-04T09:29:47.944800+0000","last_became_active":"2024-03-04T09:29:46.896407+0000","last_became_peered":"2024-03-04T09:29:46.896407+0000","last_unstale":"2024-03-04T09:29:47.944800+0000","last_undegraded":"2024-03-04T09:29:47.944800+0000","last_fullsized":"2024-03-04T09:29:47.944800+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_clean_scrub_stamp":"2024-03-04T09:29:45.844895+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:31:16.574548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.937026+0000","last_change":"2024-03-04T09:29:47.930603+0000","last_active":"2024-03-04T09:29:47.937026+0000","last_peered":"2024-03-04T09:29:47.937026+0000","last_clean":"2024-03-04T09:29:47.937026+0000","last_became_active":"2024-03-04T09:29:47.930056+0000","last_became_peered":"2024-03-04T09:29:47.930056+0000","last_unstale":"2024-03-04T09:29:47.937026+0000","last_undegraded":"2024-03-04T09:29:47.937026+0000","last_fullsized":"2024-03-04T09:29:47.937026+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:05:45.936048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"80'1","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.944620+0000","last_change":"2024-03-04T09:29:46.906308+0000","last_active":"2024-03-04T09:29:47.944620+0000","last_peered":"2024-03-04T09:29:47.944620+0000","last_clean":"2024-03-04T09:29:47.944620+0000","last_became_active":"2024-03-04T09:29:46.905284+0000","last_became_peered":"2024-03-04T09:29:46.905284+0000","last_unstale":"2024-03-04T09:29:47.944620+0000","last_undegraded":"2024-03-04T09:29:47.944620+0000","last_fullsized":"2024-03-04T09:29:47.944620+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_clean_scrub_stamp":"2024-03-04T09:29:45.844895+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:19:20.417148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.938998+0000","last_change":"2024-03-04T09:29:47.935922+0000","last_active":"2024-03-04T09:29:47.938998+0000","last_peered":"2024-03-04T09:29:47.938998+0000","last_clean":"2024-03-04T09:29:47.938998+0000","last_became_active":"2024-03-04T09:29:47.935607+0000","last_became_peered":"2024-03-04T09:29:47.935607+0000","last_unstale":"2024-03-04T09:29:47.938998+0000","last_undegraded":"2024-03-04T09:29:47.938998+0000","last_fullsized":"2024-03-04T09:29:47.938998+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:11:36.963320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"80'1","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.941894+0000","last_change":"2024-03-04T09:29:46.904602+0000","last_active":"2024-03-04T09:29:47.941894+0000","last_peered":"2024-03-04T09:29:47.941894+0000","last_clean":"2024-03-04T09:29:47.941894+0000","last_became_active":"2024-03-04T09:29:46.903831+0000","last_became_peered":"2024-03-04T09:29:46.903831+0000","last_unstale":"2024-03-04T09:29:47.941894+0000","last_undegraded":"2024-03-04T09:29:47.941894+0000","last_fullsized":"2024-03-04T09:29:47.941894+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_clean_scrub_stamp":"2024-03-04T09:29:45.844895+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:26:39.557924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.930529+0000","last_change":"2024-03-04T09:29:47.922387+0000","last_active":"2024-03-04T09:29:47.930529+0000","last_peered":"2024-03-04T09:29:47.930529+0000","last_clean":"2024-03-04T09:29:47.930529+0000","last_became_active":"2024-03-04T09:29:47.921863+0000","last_became_peered":"2024-03-04T09:29:47.921863+0000","last_unstale":"2024-03-04T09:29:47.930529+0000","last_undegraded":"2024-03-04T09:29:47.930529+0000","last_fullsized":"2024-03-04T09:29:47.930529+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:49:19.951736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.907403+0000","last_change":"2024-03-04T09:29:46.895184+0000","last_active":"2024-03-04T09:29:47.907403+0000","last_peered":"2024-03-04T09:29:47.907403+0000","last_clean":"2024-03-04T09:29:47.907403+0000","last_became_active":"2024-03-04T09:29:46.894192+0000","last_became_peered":"2024-03-04T09:29:46.894192+0000","last_unstale":"2024-03-04T09:29:47.907403+0000","last_undegraded":"2024-03-04T09:29:47.907403+0000","last_fullsized":"2024-03-04T09:29:47.907403+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_clean_scrub_stamp":"2024-03-04T09:29:45.844895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:49:18.936422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.928199+0000","last_change":"2024-03-04T09:29:47.920278+0000","last_active":"2024-03-04T09:29:47.928199+0000","last_peered":"2024-03-04T09:29:47.928199+0000","last_clean":"2024-03-04T09:29:47.928199+0000","last_became_active":"2024-03-04T09:29:47.919571+0000","last_became_peered":"2024-03-04T09:29:47.919571+0000","last_unstale":"2024-03-04T09:29:47.928199+0000","last_undegraded":"2024-03-04T09:29:47.928199+0000","last_fullsized":"2024-03-04T09:29:47.928199+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:48:59.581604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.891360+0000","last_change":"2024-03-04T09:29:46.908478+0000","last_active":"2024-03-04T09:29:47.891360+0000","last_peered":"2024-03-04T09:29:47.891360+0000","last_clean":"2024-03-04T09:29:47.891360+0000","last_became_active":"2024-03-04T09:29:46.908115+0000","last_became_peered":"2024-03-04T09:29:46.908115+0000","last_unstale":"2024-03-04T09:29:47.891360+0000","last_undegraded":"2024-03-04T09:29:47.891360+0000","last_fullsized":"2024-03-04T09:29:47.891360+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_clean_scrub_stamp":"2024-03-04T09:29:45.844895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:32:40.899263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.928004+0000","last_change":"2024-03-04T09:29:47.917693+0000","last_active":"2024-03-04T09:29:47.928004+0000","last_peered":"2024-03-04T09:29:47.928004+0000","last_clean":"2024-03-04T09:29:47.928004+0000","last_became_active":"2024-03-04T09:29:47.917341+0000","last_became_peered":"2024-03-04T09:29:47.917341+0000","last_unstale":"2024-03-04T09:29:47.928004+0000","last_undegraded":"2024-03-04T09:29:47.928004+0000","last_fullsized":"2024-03-04T09:29:47.928004+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:00:59.073724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.908665+0000","last_change":"2024-03-04T09:29:46.910036+0000","last_active":"2024-03-04T09:29:47.908665+0000","last_peered":"2024-03-04T09:29:47.908665+0000","last_clean":"2024-03-04T09:29:47.908665+0000","last_became_active":"2024-03-04T09:29:46.909728+0000","last_became_peered":"2024-03-04T09:29:46.909728+0000","last_unstale":"2024-03-04T09:29:47.908665+0000","last_undegraded":"2024-03-04T09:29:47.908665+0000","last_fullsized":"2024-03-04T09:29:47.908665+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_clean_scrub_stamp":"2024-03-04T09:29:45.844895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:29:49.767886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.925561+0000","last_change":"2024-03-04T09:29:47.923787+0000","last_active":"2024-03-04T09:29:47.925561+0000","last_peered":"2024-03-04T09:29:47.925561+0000","last_clean":"2024-03-04T09:29:47.925561+0000","last_became_active":"2024-03-04T09:29:47.923446+0000","last_became_peered":"2024-03-04T09:29:47.923446+0000","last_unstale":"2024-03-04T09:29:47.925561+0000","last_undegraded":"2024-03-04T09:29:47.925561+0000","last_fullsized":"2024-03-04T09:29:47.925561+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:32:32.654222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.910642+0000","last_change":"2024-03-04T09:29:46.908443+0000","last_active":"2024-03-04T09:29:47.910642+0000","last_peered":"2024-03-04T09:29:47.910642+0000","last_clean":"2024-03-04T09:29:47.910642+0000","last_became_active":"2024-03-04T09:29:46.908174+0000","last_became_peered":"2024-03-04T09:29:46.908174+0000","last_unstale":"2024-03-04T09:29:47.910642+0000","last_undegraded":"2024-03-04T09:29:47.910642+0000","last_fullsized":"2024-03-04T09:29:47.910642+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_clean_scrub_stamp":"2024-03-04T09:29:45.844895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:26:39.557924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.933883+0000","last_change":"2024-03-04T09:29:47.929520+0000","last_active":"2024-03-04T09:29:47.933883+0000","last_peered":"2024-03-04T09:29:47.933883+0000","last_clean":"2024-03-04T09:29:47.933883+0000","last_became_active":"2024-03-04T09:29:47.929216+0000","last_became_peered":"2024-03-04T09:29:47.929216+0000","last_unstale":"2024-03-04T09:29:47.933883+0000","last_undegraded":"2024-03-04T09:29:47.933883+0000","last_fullsized":"2024-03-04T09:29:47.933883+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:55:33.422040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"80'3","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.945240+0000","last_change":"2024-03-04T09:29:46.899987+0000","last_active":"2024-03-04T09:29:47.945240+0000","last_peered":"2024-03-04T09:29:47.945240+0000","last_clean":"2024-03-04T09:29:47.945240+0000","last_became_active":"2024-03-04T09:29:46.899560+0000","last_became_peered":"2024-03-04T09:29:46.899560+0000","last_unstale":"2024-03-04T09:29:47.945240+0000","last_undegraded":"2024-03-04T09:29:47.945240+0000","last_fullsized":"2024-03-04T09:29:47.945240+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_clean_scrub_stamp":"2024-03-04T09:29:45.844895+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:14:05.615270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.934188+0000","last_change":"2024-03-04T09:29:47.927722+0000","last_active":"2024-03-04T09:29:47.934188+0000","last_peered":"2024-03-04T09:29:47.934188+0000","last_clean":"2024-03-04T09:29:47.934188+0000","last_became_active":"2024-03-04T09:29:47.927232+0000","last_became_peered":"2024-03-04T09:29:47.927232+0000","last_unstale":"2024-03-04T09:29:47.934188+0000","last_undegraded":"2024-03-04T09:29:47.934188+0000","last_fullsized":"2024-03-04T09:29:47.934188+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:10:47.491712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"80'6","reported_seq":18,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.946983+0000","last_change":"2024-03-04T09:29:46.889387+0000","last_active":"2024-03-04T09:29:47.946983+0000","last_peered":"2024-03-04T09:29:47.946983+0000","last_clean":"2024-03-04T09:29:47.946983+0000","last_became_active":"2024-03-04T09:29:46.887601+0000","last_became_peered":"2024-03-04T09:29:46.887601+0000","last_unstale":"2024-03-04T09:29:47.946983+0000","last_undegraded":"2024-03-04T09:29:47.946983+0000","last_fullsized":"2024-03-04T09:29:47.946983+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_clean_scrub_stamp":"2024-03-04T09:29:45.844895+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:41:02.095286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.932178+0000","last_change":"2024-03-04T09:29:47.927748+0000","last_active":"2024-03-04T09:29:47.932178+0000","last_peered":"2024-03-04T09:29:47.932178+0000","last_clean":"2024-03-04T09:29:47.932178+0000","last_became_active":"2024-03-04T09:29:47.926299+0000","last_became_peered":"2024-03-04T09:29:47.926299+0000","last_unstale":"2024-03-04T09:29:47.932178+0000","last_undegraded":"2024-03-04T09:29:47.932178+0000","last_fullsized":"2024-03-04T09:29:47.932178+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:14:23.576549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"80'2","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.895387+0000","last_change":"2024-03-04T09:29:46.899282+0000","last_active":"2024-03-04T09:29:47.895387+0000","last_peered":"2024-03-04T09:29:47.895387+0000","last_clean":"2024-03-04T09:29:47.895387+0000","last_became_active":"2024-03-04T09:29:46.897870+0000","last_became_peered":"2024-03-04T09:29:46.897870+0000","last_unstale":"2024-03-04T09:29:47.895387+0000","last_undegraded":"2024-03-04T09:29:47.895387+0000","last_fullsized":"2024-03-04T09:29:47.895387+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_clean_scrub_stamp":"2024-03-04T09:29:45.844895+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:05:23.250531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.924875+0000","last_change":"2024-03-04T09:29:47.917902+0000","last_active":"2024-03-04T09:29:47.924875+0000","last_peered":"2024-03-04T09:29:47.924875+0000","last_clean":"2024-03-04T09:29:47.924875+0000","last_became_active":"2024-03-04T09:29:47.917601+0000","last_became_peered":"2024-03-04T09:29:47.917601+0000","last_unstale":"2024-03-04T09:29:47.924875+0000","last_undegraded":"2024-03-04T09:29:47.924875+0000","last_fullsized":"2024-03-04T09:29:47.924875+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:18:06.867475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.903997+0000","last_change":"2024-03-04T09:29:46.878519+0000","last_active":"2024-03-04T09:29:47.903997+0000","last_peered":"2024-03-04T09:29:47.903997+0000","last_clean":"2024-03-04T09:29:47.903997+0000","last_became_active":"2024-03-04T09:29:46.877481+0000","last_became_peered":"2024-03-04T09:29:46.877481+0000","last_unstale":"2024-03-04T09:29:47.903997+0000","last_undegraded":"2024-03-04T09:29:47.903997+0000","last_fullsized":"2024-03-04T09:29:47.903997+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_clean_scrub_stamp":"2024-03-04T09:29:45.844895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:20:58.488539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.941271+0000","last_change":"2024-03-04T09:29:47.932464+0000","last_active":"2024-03-04T09:29:47.941271+0000","last_peered":"2024-03-04T09:29:47.941271+0000","last_clean":"2024-03-04T09:29:47.941271+0000","last_became_active":"2024-03-04T09:29:47.932190+0000","last_became_peered":"2024-03-04T09:29:47.932190+0000","last_unstale":"2024-03-04T09:29:47.941271+0000","last_undegraded":"2024-03-04T09:29:47.941271+0000","last_fullsized":"2024-03-04T09:29:47.941271+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:58:07.029154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.895695+0000","last_change":"2024-03-04T09:29:46.907772+0000","last_active":"2024-03-04T09:29:47.895695+0000","last_peered":"2024-03-04T09:29:47.895695+0000","last_clean":"2024-03-04T09:29:47.895695+0000","last_became_active":"2024-03-04T09:29:46.907454+0000","last_became_peered":"2024-03-04T09:29:46.907454+0000","last_unstale":"2024-03-04T09:29:47.895695+0000","last_undegraded":"2024-03-04T09:29:47.895695+0000","last_fullsized":"2024-03-04T09:29:47.895695+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_clean_scrub_stamp":"2024-03-04T09:29:45.844895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:19:20.417148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.930206+0000","last_change":"2024-03-04T09:29:47.925561+0000","last_active":"2024-03-04T09:29:47.930206+0000","last_peered":"2024-03-04T09:29:47.930206+0000","last_clean":"2024-03-04T09:29:47.930206+0000","last_became_active":"2024-03-04T09:29:47.925240+0000","last_became_peered":"2024-03-04T09:29:47.925240+0000","last_unstale":"2024-03-04T09:29:47.930206+0000","last_undegraded":"2024-03-04T09:29:47.930206+0000","last_fullsized":"2024-03-04T09:29:47.930206+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:13:17.716288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"80'8","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.944801+0000","last_change":"2024-03-04T09:29:46.904792+0000","last_active":"2024-03-04T09:29:47.944801+0000","last_peered":"2024-03-04T09:29:47.944801+0000","last_clean":"2024-03-04T09:29:47.944801+0000","last_became_active":"2024-03-04T09:29:46.904415+0000","last_became_peered":"2024-03-04T09:29:46.904415+0000","last_unstale":"2024-03-04T09:29:47.944801+0000","last_undegraded":"2024-03-04T09:29:47.944801+0000","last_fullsized":"2024-03-04T09:29:47.944801+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_clean_scrub_stamp":"2024-03-04T09:29:45.844895+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:33:08.484663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.923132+0000","last_change":"2024-03-04T09:29:47.917900+0000","last_active":"2024-03-04T09:29:47.923132+0000","last_peered":"2024-03-04T09:29:47.923132+0000","last_clean":"2024-03-04T09:29:47.923132+0000","last_became_active":"2024-03-04T09:29:47.917613+0000","last_became_peered":"2024-03-04T09:29:47.917613+0000","last_unstale":"2024-03-04T09:29:47.923132+0000","last_undegraded":"2024-03-04T09:29:47.923132+0000","last_fullsized":"2024-03-04T09:29:47.923132+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:50:00.557690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"80'1","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.942145+0000","last_change":"2024-03-04T09:29:46.899872+0000","last_active":"2024-03-04T09:29:47.942145+0000","last_peered":"2024-03-04T09:29:47.942145+0000","last_clean":"2024-03-04T09:29:47.942145+0000","last_became_active":"2024-03-04T09:29:46.899479+0000","last_became_peered":"2024-03-04T09:29:46.899479+0000","last_unstale":"2024-03-04T09:29:47.942145+0000","last_undegraded":"2024-03-04T09:29:47.942145+0000","last_fullsized":"2024-03-04T09:29:47.942145+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_clean_scrub_stamp":"2024-03-04T09:29:45.844895+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:57:45.922627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.931017+0000","last_change":"2024-03-04T09:29:47.925683+0000","last_active":"2024-03-04T09:29:47.931017+0000","last_peered":"2024-03-04T09:29:47.931017+0000","last_clean":"2024-03-04T09:29:47.931017+0000","last_became_active":"2024-03-04T09:29:47.924331+0000","last_became_peered":"2024-03-04T09:29:47.924331+0000","last_unstale":"2024-03-04T09:29:47.931017+0000","last_undegraded":"2024-03-04T09:29:47.931017+0000","last_fullsized":"2024-03-04T09:29:47.931017+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:10:47.491712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"80'2","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.944990+0000","last_change":"2024-03-04T09:29:46.907165+0000","last_active":"2024-03-04T09:29:47.944990+0000","last_peered":"2024-03-04T09:29:47.944990+0000","last_clean":"2024-03-04T09:29:47.944990+0000","last_became_active":"2024-03-04T09:29:46.906840+0000","last_became_peered":"2024-03-04T09:29:46.906840+0000","last_unstale":"2024-03-04T09:29:47.944990+0000","last_undegraded":"2024-03-04T09:29:47.944990+0000","last_fullsized":"2024-03-04T09:29:47.944990+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_clean_scrub_stamp":"2024-03-04T09:29:45.844895+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:44:07.289519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.934206+0000","last_change":"2024-03-04T09:29:47.930350+0000","last_active":"2024-03-04T09:29:47.934206+0000","last_peered":"2024-03-04T09:29:47.934206+0000","last_clean":"2024-03-04T09:29:47.934206+0000","last_became_active":"2024-03-04T09:29:47.929950+0000","last_became_peered":"2024-03-04T09:29:47.929950+0000","last_unstale":"2024-03-04T09:29:47.934206+0000","last_undegraded":"2024-03-04T09:29:47.934206+0000","last_fullsized":"2024-03-04T09:29:47.934206+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:13:42.546100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.909032+0000","last_change":"2024-03-04T09:29:46.893430+0000","last_active":"2024-03-04T09:29:47.909032+0000","last_peered":"2024-03-04T09:29:47.909032+0000","last_clean":"2024-03-04T09:29:47.909032+0000","last_became_active":"2024-03-04T09:29:46.893173+0000","last_became_peered":"2024-03-04T09:29:46.893173+0000","last_unstale":"2024-03-04T09:29:47.909032+0000","last_undegraded":"2024-03-04T09:29:47.909032+0000","last_fullsized":"2024-03-04T09:29:47.909032+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_clean_scrub_stamp":"2024-03-04T09:29:45.844895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:21:17.815826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.935154+0000","last_change":"2024-03-04T09:29:47.924678+0000","last_active":"2024-03-04T09:29:47.935154+0000","last_peered":"2024-03-04T09:29:47.935154+0000","last_clean":"2024-03-04T09:29:47.935154+0000","last_became_active":"2024-03-04T09:29:47.924404+0000","last_became_peered":"2024-03-04T09:29:47.924404+0000","last_unstale":"2024-03-04T09:29:47.935154+0000","last_undegraded":"2024-03-04T09:29:47.935154+0000","last_fullsized":"2024-03-04T09:29:47.935154+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:48:42.428966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"80'2","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.937300+0000","last_change":"2024-03-04T09:29:46.895983+0000","last_active":"2024-03-04T09:29:47.937300+0000","last_peered":"2024-03-04T09:29:47.937300+0000","last_clean":"2024-03-04T09:29:47.937300+0000","last_became_active":"2024-03-04T09:29:46.895680+0000","last_became_peered":"2024-03-04T09:29:46.895680+0000","last_unstale":"2024-03-04T09:29:47.937300+0000","last_undegraded":"2024-03-04T09:29:47.937300+0000","last_fullsized":"2024-03-04T09:29:47.937300+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_clean_scrub_stamp":"2024-03-04T09:29:45.844895+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:55:04.068008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.919738+0000","last_change":"2024-03-04T09:29:47.913695+0000","last_active":"2024-03-04T09:29:47.919738+0000","last_peered":"2024-03-04T09:29:47.919738+0000","last_clean":"2024-03-04T09:29:47.919738+0000","last_became_active":"2024-03-04T09:29:47.913345+0000","last_became_peered":"2024-03-04T09:29:47.913345+0000","last_unstale":"2024-03-04T09:29:47.919738+0000","last_undegraded":"2024-03-04T09:29:47.919738+0000","last_fullsized":"2024-03-04T09:29:47.919738+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:06:10.504206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.917068+0000","last_change":"2024-03-04T09:29:46.893771+0000","last_active":"2024-03-04T09:29:47.917068+0000","last_peered":"2024-03-04T09:29:47.917068+0000","last_clean":"2024-03-04T09:29:47.917068+0000","last_became_active":"2024-03-04T09:29:46.893410+0000","last_became_peered":"2024-03-04T09:29:46.893410+0000","last_unstale":"2024-03-04T09:29:47.917068+0000","last_undegraded":"2024-03-04T09:29:47.917068+0000","last_fullsized":"2024-03-04T09:29:47.917068+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_clean_scrub_stamp":"2024-03-04T09:29:45.844895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:06:47.404643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.934339+0000","last_change":"2024-03-04T09:29:47.927843+0000","last_active":"2024-03-04T09:29:47.934339+0000","last_peered":"2024-03-04T09:29:47.934339+0000","last_clean":"2024-03-04T09:29:47.934339+0000","last_became_active":"2024-03-04T09:29:47.927548+0000","last_became_peered":"2024-03-04T09:29:47.927548+0000","last_unstale":"2024-03-04T09:29:47.934339+0000","last_undegraded":"2024-03-04T09:29:47.934339+0000","last_fullsized":"2024-03-04T09:29:47.934339+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:51:21.354710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.912065+0000","last_change":"2024-03-04T09:29:46.900643+0000","last_active":"2024-03-04T09:29:47.912065+0000","last_peered":"2024-03-04T09:29:47.912065+0000","last_clean":"2024-03-04T09:29:47.912065+0000","last_became_active":"2024-03-04T09:29:46.900368+0000","last_became_peered":"2024-03-04T09:29:46.900368+0000","last_unstale":"2024-03-04T09:29:47.912065+0000","last_undegraded":"2024-03-04T09:29:47.912065+0000","last_fullsized":"2024-03-04T09:29:47.912065+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_clean_scrub_stamp":"2024-03-04T09:29:45.844895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:43:50.343554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.920128+0000","last_change":"2024-03-04T09:29:47.915790+0000","last_active":"2024-03-04T09:29:47.920128+0000","last_peered":"2024-03-04T09:29:47.920128+0000","last_clean":"2024-03-04T09:29:47.920128+0000","last_became_active":"2024-03-04T09:29:47.915338+0000","last_became_peered":"2024-03-04T09:29:47.915338+0000","last_unstale":"2024-03-04T09:29:47.920128+0000","last_undegraded":"2024-03-04T09:29:47.920128+0000","last_fullsized":"2024-03-04T09:29:47.920128+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:10:47.491712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.903647+0000","last_change":"2024-03-04T09:29:46.893503+0000","last_active":"2024-03-04T09:29:47.903647+0000","last_peered":"2024-03-04T09:29:47.903647+0000","last_clean":"2024-03-04T09:29:47.903647+0000","last_became_active":"2024-03-04T09:29:46.893069+0000","last_became_peered":"2024-03-04T09:29:46.893069+0000","last_unstale":"2024-03-04T09:29:47.903647+0000","last_undegraded":"2024-03-04T09:29:47.903647+0000","last_fullsized":"2024-03-04T09:29:47.903647+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_clean_scrub_stamp":"2024-03-04T09:29:45.844895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:00:54.135399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.932928+0000","last_change":"2024-03-04T09:29:47.926973+0000","last_active":"2024-03-04T09:29:47.932928+0000","last_peered":"2024-03-04T09:29:47.932928+0000","last_clean":"2024-03-04T09:29:47.932928+0000","last_became_active":"2024-03-04T09:29:47.926569+0000","last_became_peered":"2024-03-04T09:29:47.926569+0000","last_unstale":"2024-03-04T09:29:47.932928+0000","last_undegraded":"2024-03-04T09:29:47.932928+0000","last_fullsized":"2024-03-04T09:29:47.932928+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:31:16.169781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"80'2","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.893342+0000","last_change":"2024-03-04T09:29:46.896017+0000","last_active":"2024-03-04T09:29:47.893342+0000","last_peered":"2024-03-04T09:29:47.893342+0000","last_clean":"2024-03-04T09:29:47.893342+0000","last_became_active":"2024-03-04T09:29:46.893411+0000","last_became_peered":"2024-03-04T09:29:46.893411+0000","last_unstale":"2024-03-04T09:29:47.893342+0000","last_undegraded":"2024-03-04T09:29:47.893342+0000","last_fullsized":"2024-03-04T09:29:47.893342+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_clean_scrub_stamp":"2024-03-04T09:29:45.844895+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:47:04.045507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.936967+0000","last_change":"2024-03-04T09:29:47.930500+0000","last_active":"2024-03-04T09:29:47.936967+0000","last_peered":"2024-03-04T09:29:47.936967+0000","last_clean":"2024-03-04T09:29:47.936967+0000","last_became_active":"2024-03-04T09:29:47.930219+0000","last_became_peered":"2024-03-04T09:29:47.930219+0000","last_unstale":"2024-03-04T09:29:47.936967+0000","last_undegraded":"2024-03-04T09:29:47.936967+0000","last_fullsized":"2024-03-04T09:29:47.936967+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:33:09.499977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.895041+0000","last_change":"2024-03-04T09:29:46.888517+0000","last_active":"2024-03-04T09:29:47.895041+0000","last_peered":"2024-03-04T09:29:47.895041+0000","last_clean":"2024-03-04T09:29:47.895041+0000","last_became_active":"2024-03-04T09:29:46.888165+0000","last_became_peered":"2024-03-04T09:29:46.888165+0000","last_unstale":"2024-03-04T09:29:47.895041+0000","last_undegraded":"2024-03-04T09:29:47.895041+0000","last_fullsized":"2024-03-04T09:29:47.895041+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_clean_scrub_stamp":"2024-03-04T09:29:45.844895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:15:20.058129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.924913+0000","last_change":"2024-03-04T09:29:47.922765+0000","last_active":"2024-03-04T09:29:47.924913+0000","last_peered":"2024-03-04T09:29:47.924913+0000","last_clean":"2024-03-04T09:29:47.924913+0000","last_became_active":"2024-03-04T09:29:47.922427+0000","last_became_peered":"2024-03-04T09:29:47.922427+0000","last_unstale":"2024-03-04T09:29:47.924913+0000","last_undegraded":"2024-03-04T09:29:47.924913+0000","last_fullsized":"2024-03-04T09:29:47.924913+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:31:16.169781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.903453+0000","last_change":"2024-03-04T09:29:46.905513+0000","last_active":"2024-03-04T09:29:47.903453+0000","last_peered":"2024-03-04T09:29:47.903453+0000","last_clean":"2024-03-04T09:29:47.903453+0000","last_became_active":"2024-03-04T09:29:46.903818+0000","last_became_peered":"2024-03-04T09:29:46.903818+0000","last_unstale":"2024-03-04T09:29:47.903453+0000","last_undegraded":"2024-03-04T09:29:47.903453+0000","last_fullsized":"2024-03-04T09:29:47.903453+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_clean_scrub_stamp":"2024-03-04T09:29:45.844895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:35:07.573589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.934167+0000","last_change":"2024-03-04T09:29:47.927552+0000","last_active":"2024-03-04T09:29:47.934167+0000","last_peered":"2024-03-04T09:29:47.934167+0000","last_clean":"2024-03-04T09:29:47.934167+0000","last_became_active":"2024-03-04T09:29:47.926751+0000","last_became_peered":"2024-03-04T09:29:47.926751+0000","last_unstale":"2024-03-04T09:29:47.934167+0000","last_undegraded":"2024-03-04T09:29:47.934167+0000","last_fullsized":"2024-03-04T09:29:47.934167+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:06:10.504206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.908627+0000","last_change":"2024-03-04T09:29:46.907506+0000","last_active":"2024-03-04T09:29:47.908627+0000","last_peered":"2024-03-04T09:29:47.908627+0000","last_clean":"2024-03-04T09:29:47.908627+0000","last_became_active":"2024-03-04T09:29:46.906967+0000","last_became_peered":"2024-03-04T09:29:46.906967+0000","last_unstale":"2024-03-04T09:29:47.908627+0000","last_undegraded":"2024-03-04T09:29:47.908627+0000","last_fullsized":"2024-03-04T09:29:47.908627+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_clean_scrub_stamp":"2024-03-04T09:29:45.844895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:26:07.581293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.918098+0000","last_change":"2024-03-04T09:29:47.911980+0000","last_active":"2024-03-04T09:29:47.918098+0000","last_peered":"2024-03-04T09:29:47.918098+0000","last_clean":"2024-03-04T09:29:47.918098+0000","last_became_active":"2024-03-04T09:29:47.911653+0000","last_became_peered":"2024-03-04T09:29:47.911653+0000","last_unstale":"2024-03-04T09:29:47.918098+0000","last_undegraded":"2024-03-04T09:29:47.918098+0000","last_fullsized":"2024-03-04T09:29:47.918098+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:31:16.169781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.908321+0000","last_change":"2024-03-04T09:29:46.909862+0000","last_active":"2024-03-04T09:29:47.908321+0000","last_peered":"2024-03-04T09:29:47.908321+0000","last_clean":"2024-03-04T09:29:47.908321+0000","last_became_active":"2024-03-04T09:29:46.909610+0000","last_became_peered":"2024-03-04T09:29:46.909610+0000","last_unstale":"2024-03-04T09:29:47.908321+0000","last_undegraded":"2024-03-04T09:29:47.908321+0000","last_fullsized":"2024-03-04T09:29:47.908321+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_clean_scrub_stamp":"2024-03-04T09:29:45.844895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:11:35.948007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.932275+0000","last_change":"2024-03-04T09:29:47.925936+0000","last_active":"2024-03-04T09:29:47.932275+0000","last_peered":"2024-03-04T09:29:47.932275+0000","last_clean":"2024-03-04T09:29:47.932275+0000","last_became_active":"2024-03-04T09:29:47.925668+0000","last_became_peered":"2024-03-04T09:29:47.925668+0000","last_unstale":"2024-03-04T09:29:47.932275+0000","last_undegraded":"2024-03-04T09:29:47.932275+0000","last_fullsized":"2024-03-04T09:29:47.932275+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:09:08.575816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.913044+0000","last_change":"2024-03-04T09:29:46.904786+0000","last_active":"2024-03-04T09:29:47.913044+0000","last_peered":"2024-03-04T09:29:47.913044+0000","last_clean":"2024-03-04T09:29:47.913044+0000","last_became_active":"2024-03-04T09:29:46.904494+0000","last_became_peered":"2024-03-04T09:29:46.904494+0000","last_unstale":"2024-03-04T09:29:47.913044+0000","last_undegraded":"2024-03-04T09:29:47.913044+0000","last_fullsized":"2024-03-04T09:29:47.913044+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_clean_scrub_stamp":"2024-03-04T09:29:45.844895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:13:42.118138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.935326+0000","last_change":"2024-03-04T09:29:47.927853+0000","last_active":"2024-03-04T09:29:47.935326+0000","last_peered":"2024-03-04T09:29:47.935326+0000","last_clean":"2024-03-04T09:29:47.935326+0000","last_became_active":"2024-03-04T09:29:47.926384+0000","last_became_peered":"2024-03-04T09:29:47.926384+0000","last_unstale":"2024-03-04T09:29:47.935326+0000","last_undegraded":"2024-03-04T09:29:47.935326+0000","last_fullsized":"2024-03-04T09:29:47.935326+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:31:17.589861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.911259+0000","last_change":"2024-03-04T09:29:46.907893+0000","last_active":"2024-03-04T09:29:47.911259+0000","last_peered":"2024-03-04T09:29:47.911259+0000","last_clean":"2024-03-04T09:29:47.911259+0000","last_became_active":"2024-03-04T09:29:46.906807+0000","last_became_peered":"2024-03-04T09:29:46.906807+0000","last_unstale":"2024-03-04T09:29:47.911259+0000","last_undegraded":"2024-03-04T09:29:47.911259+0000","last_fullsized":"2024-03-04T09:29:47.911259+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_clean_scrub_stamp":"2024-03-04T09:29:45.844895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:13:42.118138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.927515+0000","last_change":"2024-03-04T09:29:47.923223+0000","last_active":"2024-03-04T09:29:47.927515+0000","last_peered":"2024-03-04T09:29:47.927515+0000","last_clean":"2024-03-04T09:29:47.927515+0000","last_became_active":"2024-03-04T09:29:47.921790+0000","last_became_peered":"2024-03-04T09:29:47.921790+0000","last_unstale":"2024-03-04T09:29:47.927515+0000","last_undegraded":"2024-03-04T09:29:47.927515+0000","last_fullsized":"2024-03-04T09:29:47.927515+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:06:10.504206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.917465+0000","last_change":"2024-03-04T09:29:46.906266+0000","last_active":"2024-03-04T09:29:47.917465+0000","last_peered":"2024-03-04T09:29:47.917465+0000","last_clean":"2024-03-04T09:29:47.917465+0000","last_became_active":"2024-03-04T09:29:46.905334+0000","last_became_peered":"2024-03-04T09:29:46.905334+0000","last_unstale":"2024-03-04T09:29:47.917465+0000","last_undegraded":"2024-03-04T09:29:47.917465+0000","last_fullsized":"2024-03-04T09:29:47.917465+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_clean_scrub_stamp":"2024-03-04T09:29:45.844895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:41:02.095286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.905853+0000","last_change":"2024-03-04T09:29:47.902516+0000","last_active":"2024-03-04T09:29:47.905853+0000","last_peered":"2024-03-04T09:29:47.905853+0000","last_clean":"2024-03-04T09:29:47.905853+0000","last_became_active":"2024-03-04T09:29:47.901685+0000","last_became_peered":"2024-03-04T09:29:47.901685+0000","last_unstale":"2024-03-04T09:29:47.905853+0000","last_undegraded":"2024-03-04T09:29:47.905853+0000","last_fullsized":"2024-03-04T09:29:47.905853+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:46.860209+0000","last_clean_scrub_stamp":"2024-03-04T09:29:46.860209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:51:23.265839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"80'1","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.944935+0000","last_change":"2024-03-04T09:29:46.908186+0000","last_active":"2024-03-04T09:29:47.944935+0000","last_peered":"2024-03-04T09:29:47.944935+0000","last_clean":"2024-03-04T09:29:47.944935+0000","last_became_active":"2024-03-04T09:29:46.907897+0000","last_became_peered":"2024-03-04T09:29:46.907897+0000","last_unstale":"2024-03-04T09:29:47.944935+0000","last_undegraded":"2024-03-04T09:29:47.944935+0000","last_fullsized":"2024-03-04T09:29:47.944935+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:29:45.844895+0000","last_clean_scrub_stamp":"2024-03-04T09:29:45.844895+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:13:41.530786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":283,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T09:29:47.905923+0000","last_change":"2024-03-04T09:25:09.316159+0000","last_active":"2024-03-04T09:29:47.905923+0000","last_peered":"2024-03-04T09:29:47.905923+0000","last_clean":"2024-03-04T09:29:47.905923+0000","last_became_active":"2024-03-04T09:25:09.313767+0000","last_became_peered":"2024-03-04T09:25:09.313767+0000","last_unstale":"2024-03-04T09:29:47.905923+0000","last_undegraded":"2024-03-04T09:29:47.905923+0000","last_fullsized":"2024-03-04T09:29:47.905923+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:08.110953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:08.110953+0000","last_clean_scrub_stamp":"2024-03-04T09:25:08.110953+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:34:21.745390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":3880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574907,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32712,"kb_used_data":1528,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339128,"statfs":{"total":96636764160,"available":96603267072,"internally_reserved":0,"allocated":1564672,"data_stored":1000045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574907,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32132,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339708,"statfs":{"total":96636764160,"available":96603860992,"internally_reserved":0,"allocated":970752,"data_stored":409677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32132,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339708,"statfs":{"total":96636764160,"available":96603860992,"internally_reserved":0,"allocated":970752,"data_stored":409677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574906,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32128,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339712,"statfs":{"total":96636764160,"available":96603865088,"internally_reserved":0,"allocated":970752,"data_stored":409677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574908,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32132,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339708,"statfs":{"total":96636764160,"available":96603860992,"internally_reserved":0,"allocated":970752,"data_stored":409677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574907,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32132,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339708,"statfs":{"total":96636764160,"available":96603860992,"internally_reserved":0,"allocated":970752,"data_stored":409677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574908,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32132,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339708,"statfs":{"total":96636764160,"available":96603860992,"internally_reserved":0,"allocated":970752,"data_stored":409677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574907,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32708,"kb_used_data":1528,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339132,"statfs":{"total":96636764160,"available":96603271168,"internally_reserved":0,"allocated":1564672,"data_stored":1000045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-04T09:29:52.932 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-04T09:29:52.933 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-04T09:29:53.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.208+0000 7fde67200640 1 Processor -- start 2024-03-04T09:29:53.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.208+0000 7fde67200640 1 -- start start 2024-03-04T09:29:53.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.210+0000 7fde67200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fde68161320 0x7fde680d2320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:53.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.210+0000 7fde67200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde680d48d0 0x7fde680d0610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:53.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.210+0000 7fde67200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fde680d0b50 0x7fde680ce8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:53.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.210+0000 7fde67200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde680d0fc0 con 0x7fde68161320 2024-03-04T09:29:53.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.210+0000 7fde67200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde680d2be0 con 0x7fde680d0b50 2024-03-04T09:29:53.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.210+0000 7fde67200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde680b3550 con 0x7fde680d48d0 2024-03-04T09:29:53.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.210+0000 7fde65e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fde68161320 0x7fde680d2320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:53.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.210+0000 7fde65e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fde68161320 0x7fde680d2320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38750/0 (socket says 172.21.15.6:38750) 2024-03-04T09:29:53.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.210+0000 7fde65e00640 1 -- 172.21.15.6:0/2434727420 learned_addr learned my addr 172.21.15.6:0/2434727420 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:29:53.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.211+0000 7fde65400640 1 --2- 172.21.15.6:0/2434727420 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde680d48d0 0x7fde680d0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:53.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.211+0000 7fde66800640 1 --2- 172.21.15.6:0/2434727420 >> v2:172.21.15.146:3300/0 conn(0x7fde680d0b50 0x7fde680ce8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:53.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.211+0000 7fde65e00640 1 -- 172.21.15.6:0/2434727420 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde680d48d0 msgr2=0x7fde680d0610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:53.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.211+0000 7fde65e00640 1 --2- 172.21.15.6:0/2434727420 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde680d48d0 0x7fde680d0610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:53.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.211+0000 7fde65e00640 1 -- 172.21.15.6:0/2434727420 >> v2:172.21.15.146:3300/0 conn(0x7fde680d0b50 msgr2=0x7fde680ce8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:53.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.211+0000 7fde65e00640 1 --2- 172.21.15.6:0/2434727420 >> v2:172.21.15.146:3300/0 conn(0x7fde680d0b50 0x7fde680ce8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:53.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.211+0000 7fde65e00640 1 -- 172.21.15.6:0/2434727420 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde680b3830 con 0x7fde68161320 2024-03-04T09:29:53.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.211+0000 7fde66800640 1 --2- 172.21.15.6:0/2434727420 >> v2:172.21.15.146:3300/0 conn(0x7fde680d0b50 0x7fde680ce8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:29:53.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.211+0000 7fde65400640 1 --2- 172.21.15.6:0/2434727420 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde680d48d0 0x7fde680d0610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:29:53.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.212+0000 7fde65e00640 1 --2- 172.21.15.6:0/2434727420 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fde68161320 0x7fde680d2320 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7fde6808d620 tx=0x7fde5802b920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91a13f5d1c0b48ce server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:53.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.212+0000 7fde64a00640 1 -- 172.21.15.6:0/2434727420 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde58035020 con 0x7fde68161320 2024-03-04T09:29:53.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.213+0000 7fde64a00640 1 -- 172.21.15.6:0/2434727420 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde58031880 con 0x7fde68161320 2024-03-04T09:29:53.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.213+0000 7fde64a00640 1 -- 172.21.15.6:0/2434727420 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde58031c50 con 0x7fde68161320 2024-03-04T09:29:53.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.213+0000 7fde67200640 1 -- 172.21.15.6:0/2434727420 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fde68161320 msgr2=0x7fde680d2320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:53.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.213+0000 7fde67200640 1 --2- 172.21.15.6:0/2434727420 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fde68161320 0x7fde680d2320 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7fde6808d620 tx=0x7fde5802b920 comp rx=0 tx=0).stop 2024-03-04T09:29:53.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.213+0000 7fde67200640 1 -- 172.21.15.6:0/2434727420 shutdown_connections 2024-03-04T09:29:53.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.213+0000 7fde67200640 1 --2- 172.21.15.6:0/2434727420 >> v2:172.21.15.146:3300/0 conn(0x7fde680d0b50 0x7fde680ce8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:53.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.213+0000 7fde67200640 1 --2- 172.21.15.6:0/2434727420 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde680d48d0 0x7fde680d0610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:53.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.213+0000 7fde67200640 1 --2- 172.21.15.6:0/2434727420 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fde68161320 0x7fde680d2320 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:53.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.213+0000 7fde67200640 1 -- 172.21.15.6:0/2434727420 >> 172.21.15.6:0/2434727420 conn(0x7fde680d3d40 msgr2=0x7fde680d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:29:53.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.213+0000 7fde67200640 1 -- 172.21.15.6:0/2434727420 shutdown_connections 2024-03-04T09:29:53.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.213+0000 7fde67200640 1 -- 172.21.15.6:0/2434727420 wait complete. 2024-03-04T09:29:53.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.215+0000 7fde67200640 1 Processor -- start 2024-03-04T09:29:53.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.215+0000 7fde67200640 1 -- start start 2024-03-04T09:29:53.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.216+0000 7fde67200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fde680d0b50 0x7fde680a0070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:53.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.216+0000 7fde67200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fde680d48d0 0x7fde681546e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:53.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.216+0000 7fde67200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde68161320 0x7fde68154c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:53.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.216+0000 7fde67200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde680d2860 con 0x7fde680d48d0 2024-03-04T09:29:53.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.216+0000 7fde67200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde680b3df0 con 0x7fde680d0b50 2024-03-04T09:29:53.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.216+0000 7fde67200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde68161700 con 0x7fde68161320 2024-03-04T09:29:53.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.216+0000 7fde65400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fde680d48d0 0x7fde681546e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:53.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.216+0000 7fde65400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fde680d48d0 0x7fde681546e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38762/0 (socket says 172.21.15.6:38762) 2024-03-04T09:29:53.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.216+0000 7fde65400640 1 -- 172.21.15.6:0/766532985 learned_addr learned my addr 172.21.15.6:0/766532985 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:29:53.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.217+0000 7fde65e00640 1 --2- 172.21.15.6:0/766532985 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fde680d0b50 0x7fde680a0070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:53.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.217+0000 7fde66800640 1 --2- 172.21.15.6:0/766532985 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde68161320 0x7fde68154c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:53.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.217+0000 7fde65400640 1 -- 172.21.15.6:0/766532985 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde68161320 msgr2=0x7fde68154c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:53.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.217+0000 7fde65400640 1 --2- 172.21.15.6:0/766532985 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde68161320 0x7fde68154c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:53.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.217+0000 7fde65400640 1 -- 172.21.15.6:0/766532985 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fde680d0b50 msgr2=0x7fde680a0070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:53.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.217+0000 7fde65400640 1 --2- 172.21.15.6:0/766532985 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fde680d0b50 0x7fde680a0070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:53.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.217+0000 7fde65400640 1 -- 172.21.15.6:0/766532985 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde6809f200 con 0x7fde680d48d0 2024-03-04T09:29:53.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.217+0000 7fde66800640 1 --2- 172.21.15.6:0/766532985 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde68161320 0x7fde68154c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:29:53.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.217+0000 7fde65e00640 1 --2- 172.21.15.6:0/766532985 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fde680d0b50 0x7fde680a0070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:29:53.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.217+0000 7fde65400640 1 --2- 172.21.15.6:0/766532985 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fde680d48d0 0x7fde681546e0 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7fde5400c7f0 tx=0x7fde5400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:53.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.218+0000 7fde5e000640 1 -- 172.21.15.6:0/766532985 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde54015070 con 0x7fde680d48d0 2024-03-04T09:29:53.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.218+0000 7fde5e000640 1 -- 172.21.15.6:0/766532985 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde54007820 con 0x7fde680d48d0 2024-03-04T09:29:53.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.218+0000 7fde5e000640 1 -- 172.21.15.6:0/766532985 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde54007c10 con 0x7fde680d48d0 2024-03-04T09:29:53.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.218+0000 7fde67200640 1 -- 172.21.15.6:0/766532985 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde6809ef20 con 0x7fde680d48d0 2024-03-04T09:29:53.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.218+0000 7fde67200640 1 -- 172.21.15.6:0/766532985 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde680c8830 con 0x7fde680d48d0 2024-03-04T09:29:53.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.219+0000 7fde67200640 1 -- 172.21.15.6:0/766532985 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde6804f330 con 0x7fde680d48d0 2024-03-04T09:29:53.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.220+0000 7fde5e000640 1 -- 172.21.15.6:0/766532985 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fde54018070 con 0x7fde680d48d0 2024-03-04T09:29:53.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.221+0000 7fde5e000640 1 --2- 172.21.15.6:0/766532985 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fde4806b500 0x7fde4806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:53.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.221+0000 7fde5e000640 1 -- 172.21.15.6:0/766532985 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fde54089e40 con 0x7fde680d48d0 2024-03-04T09:29:53.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.222+0000 7fde65e00640 1 --2- 172.21.15.6:0/766532985 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fde4806b500 0x7fde4806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:53.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.223+0000 7fde65e00640 1 --2- 172.21.15.6:0/766532985 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fde4806b500 0x7fde4806d9c0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fde58036000 tx=0x7fde58008030 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:53.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.230+0000 7fde5e000640 1 -- 172.21.15.6:0/766532985 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fde54058d50 con 0x7fde680d48d0 2024-03-04T09:29:53.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.505+0000 7fde67200640 1 -- 172.21.15.6:0/766532985 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fde680b25e0 con 0x7fde680d48d0 2024-03-04T09:29:53.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.510+0000 7fde5e000640 1 -- 172.21.15.6:0/766532985 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v14) v1 ==== 190+0+147 (secure 0 0 0) 0x7fde54058650 con 0x7fde680d48d0 2024-03-04T09:29:53.507 INFO:teuthology.orchestra.run.smithi006.stdout:[client.0] 2024-03-04T09:29:53.507 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQA9k+Vl4L+rJxAACaDbQ4/Bybp5hAZAnJzlVw== 2024-03-04T09:29:53.507 INFO:teuthology.orchestra.run.smithi006.stdout: caps mds = "allow" 2024-03-04T09:29:53.507 INFO:teuthology.orchestra.run.smithi006.stdout: caps mon = "allow r" 2024-03-04T09:29:53.507 INFO:teuthology.orchestra.run.smithi006.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-04T09:29:53.507 INFO:teuthology.orchestra.run.smithi006.stderr:updated caps for client.0 2024-03-04T09:29:53.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.513+0000 7fde67200640 1 -- 172.21.15.6:0/766532985 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fde4806b500 msgr2=0x7fde4806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:53.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.513+0000 7fde67200640 1 --2- 172.21.15.6:0/766532985 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fde4806b500 0x7fde4806d9c0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fde58036000 tx=0x7fde58008030 comp rx=0 tx=0).stop 2024-03-04T09:29:53.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.513+0000 7fde67200640 1 -- 172.21.15.6:0/766532985 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fde680d48d0 msgr2=0x7fde681546e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:53.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.513+0000 7fde67200640 1 --2- 172.21.15.6:0/766532985 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fde680d48d0 0x7fde681546e0 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7fde5400c7f0 tx=0x7fde5400ccb0 comp rx=0 tx=0).stop 2024-03-04T09:29:53.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.513+0000 7fde67200640 1 -- 172.21.15.6:0/766532985 shutdown_connections 2024-03-04T09:29:53.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.513+0000 7fde67200640 1 --2- 172.21.15.6:0/766532985 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fde4806b500 0x7fde4806d9c0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:53.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.513+0000 7fde67200640 1 --2- 172.21.15.6:0/766532985 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde68161320 0x7fde68154c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:53.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.513+0000 7fde67200640 1 --2- 172.21.15.6:0/766532985 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fde680d48d0 0x7fde681546e0 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:53.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.513+0000 7fde67200640 1 --2- 172.21.15.6:0/766532985 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fde680d0b50 0x7fde680a0070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:53.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.513+0000 7fde67200640 1 -- 172.21.15.6:0/766532985 >> 172.21.15.6:0/766532985 conn(0x7fde680d3d40 msgr2=0x7fde680cce20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:29:53.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.513+0000 7fde67200640 1 -- 172.21.15.6:0/766532985 shutdown_connections 2024-03-04T09:29:53.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.513+0000 7fde67200640 1 -- 172.21.15.6:0/766532985 wait complete. 2024-03-04T09:29:53.530 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:29:53.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.738+0000 7f617b200640 1 Processor -- start 2024-03-04T09:29:53.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.738+0000 7f617b200640 1 -- start start 2024-03-04T09:29:53.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.739+0000 7f617b200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f61740b0f90 0x7f61740cf080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:53.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.739+0000 7f617b200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f61740cc730 0x7f61740d00a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:53.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.739+0000 7f617b200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f61740d05e0 0x7f617415d750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:53.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.739+0000 7f617b200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61740d0a50 con 0x7f61740b0f90 2024-03-04T09:29:53.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.739+0000 7f617b200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61740b1370 con 0x7f61740cc730 2024-03-04T09:29:53.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.739+0000 7f617b200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61740ccf80 con 0x7f61740d05e0 2024-03-04T09:29:53.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.740+0000 7f6179e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f61740b0f90 0x7f61740cf080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:53.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.740+0000 7f6179e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f61740b0f90 0x7f61740cf080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38774/0 (socket says 172.21.15.6:38774) 2024-03-04T09:29:53.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.740+0000 7f6179e00640 1 -- 172.21.15.6:0/2192328991 learned_addr learned my addr 172.21.15.6:0/2192328991 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:29:53.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.740+0000 7f617a800640 1 --2- 172.21.15.6:0/2192328991 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f61740d05e0 0x7f617415d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:53.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.740+0000 7f6179400640 1 --2- 172.21.15.6:0/2192328991 >> v2:172.21.15.146:3300/0 conn(0x7f61740cc730 0x7f61740d00a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:53.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.740+0000 7f6179e00640 1 -- 172.21.15.6:0/2192328991 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f61740d05e0 msgr2=0x7f617415d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:53.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.740+0000 7f6179e00640 1 --2- 172.21.15.6:0/2192328991 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f61740d05e0 0x7f617415d750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:53.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.740+0000 7f6179e00640 1 -- 172.21.15.6:0/2192328991 >> v2:172.21.15.146:3300/0 conn(0x7f61740cc730 msgr2=0x7f61740d00a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:53.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.740+0000 7f6179e00640 1 --2- 172.21.15.6:0/2192328991 >> v2:172.21.15.146:3300/0 conn(0x7f61740cc730 0x7f61740d00a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:53.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.740+0000 7f6179e00640 1 -- 172.21.15.6:0/2192328991 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61740cd260 con 0x7f61740b0f90 2024-03-04T09:29:53.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.740+0000 7f6179400640 1 --2- 172.21.15.6:0/2192328991 >> v2:172.21.15.146:3300/0 conn(0x7f61740cc730 0x7f61740d00a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:29:53.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.740+0000 7f617a800640 1 --2- 172.21.15.6:0/2192328991 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f61740d05e0 0x7f617415d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:29:53.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.741+0000 7f6179e00640 1 --2- 172.21.15.6:0/2192328991 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f61740b0f90 0x7f61740cf080 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f61740d1a40 tx=0x7f616802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f7b84956e8e8e50 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:53.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.742+0000 7f6178a00640 1 -- 172.21.15.6:0/2192328991 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f616802e070 con 0x7f61740b0f90 2024-03-04T09:29:53.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.742+0000 7f6178a00640 1 -- 172.21.15.6:0/2192328991 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6168037070 con 0x7f61740b0f90 2024-03-04T09:29:53.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.742+0000 7f6178a00640 1 -- 172.21.15.6:0/2192328991 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6168031a00 con 0x7f61740b0f90 2024-03-04T09:29:53.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.742+0000 7f617b200640 1 -- 172.21.15.6:0/2192328991 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f61740b0f90 msgr2=0x7f61740cf080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:53.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.742+0000 7f617b200640 1 --2- 172.21.15.6:0/2192328991 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f61740b0f90 0x7f61740cf080 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f61740d1a40 tx=0x7f616802b880 comp rx=0 tx=0).stop 2024-03-04T09:29:53.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.743+0000 7f617b200640 1 -- 172.21.15.6:0/2192328991 shutdown_connections 2024-03-04T09:29:53.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.743+0000 7f617b200640 1 --2- 172.21.15.6:0/2192328991 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f61740d05e0 0x7f617415d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:53.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.743+0000 7f617b200640 1 --2- 172.21.15.6:0/2192328991 >> v2:172.21.15.146:3300/0 conn(0x7f61740cc730 0x7f61740d00a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:53.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.743+0000 7f617b200640 1 --2- 172.21.15.6:0/2192328991 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f61740b0f90 0x7f61740cf080 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:53.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.743+0000 7f617b200640 1 -- 172.21.15.6:0/2192328991 >> 172.21.15.6:0/2192328991 conn(0x7f61740d1ae0 msgr2=0x7f61740d1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:29:53.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.743+0000 7f617b200640 1 -- 172.21.15.6:0/2192328991 shutdown_connections 2024-03-04T09:29:53.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.743+0000 7f617b200640 1 -- 172.21.15.6:0/2192328991 wait complete. 2024-03-04T09:29:53.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.745+0000 7f617b200640 1 Processor -- start 2024-03-04T09:29:53.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.746+0000 7f617b200640 1 -- start start 2024-03-04T09:29:53.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.747+0000 7f617b200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f61740b0f90 0x7f6174157c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:53.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.747+0000 7f617b200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f61740cc730 0x7f6174156830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:53.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.747+0000 7f617b200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f61740d05e0 0x7f617415a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:53.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.747+0000 7f617b200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61740ccba0 con 0x7f61740b0f90 2024-03-04T09:29:53.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.747+0000 7f617b200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61740cd820 con 0x7f61740d05e0 2024-03-04T09:29:53.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.747+0000 7f617b200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61740ccd20 con 0x7f61740cc730 2024-03-04T09:29:53.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.748+0000 7f6179e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f61740b0f90 0x7f6174157c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:53.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.748+0000 7f6179e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f61740b0f90 0x7f6174157c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38776/0 (socket says 172.21.15.6:38776) 2024-03-04T09:29:53.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.748+0000 7f6179e00640 1 -- 172.21.15.6:0/719946333 learned_addr learned my addr 172.21.15.6:0/719946333 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:29:53.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.748+0000 7f617a800640 1 --2- 172.21.15.6:0/719946333 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f61740d05e0 0x7f617415a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:53.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.748+0000 7f6179400640 1 --2- 172.21.15.6:0/719946333 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f61740cc730 0x7f6174156830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:53.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.748+0000 7f6179e00640 1 -- 172.21.15.6:0/719946333 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f61740cc730 msgr2=0x7f6174156830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:53.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.748+0000 7f6179e00640 1 --2- 172.21.15.6:0/719946333 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f61740cc730 0x7f6174156830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:53.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.748+0000 7f6179e00640 1 -- 172.21.15.6:0/719946333 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f61740d05e0 msgr2=0x7f617415a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:53.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.748+0000 7f6179e00640 1 --2- 172.21.15.6:0/719946333 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f61740d05e0 0x7f617415a210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:53.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.748+0000 7f6179e00640 1 -- 172.21.15.6:0/719946333 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61740cb6a0 con 0x7f61740b0f90 2024-03-04T09:29:53.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.749+0000 7f6179e00640 1 --2- 172.21.15.6:0/719946333 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f61740b0f90 0x7f6174157c80 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f616802b8c0 tx=0x7f61680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:53.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.749+0000 7f616e000640 1 -- 172.21.15.6:0/719946333 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6168032880 con 0x7f61740b0f90 2024-03-04T09:29:53.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.749+0000 7f616e000640 1 -- 172.21.15.6:0/719946333 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6168032a20 con 0x7f61740b0f90 2024-03-04T09:29:53.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.749+0000 7f616e000640 1 -- 172.21.15.6:0/719946333 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6168032e10 con 0x7f61740b0f90 2024-03-04T09:29:53.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.749+0000 7f617b200640 1 -- 172.21.15.6:0/719946333 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61740c67c0 con 0x7f61740b0f90 2024-03-04T09:29:53.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.749+0000 7f617b200640 1 -- 172.21.15.6:0/719946333 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6174156d70 con 0x7f61740b0f90 2024-03-04T09:29:53.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.750+0000 7f617b200640 1 -- 172.21.15.6:0/719946333 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61740cb980 con 0x7f61740b0f90 2024-03-04T09:29:53.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.751+0000 7f616e000640 1 -- 172.21.15.6:0/719946333 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f61680453f0 con 0x7f61740b0f90 2024-03-04T09:29:53.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.752+0000 7f616e000640 1 --2- 172.21.15.6:0/719946333 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f615806b500 0x7f615806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:53.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.752+0000 7f616e000640 1 -- 172.21.15.6:0/719946333 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f61680ac250 con 0x7f61740b0f90 2024-03-04T09:29:53.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.753+0000 7f6179400640 1 --2- 172.21.15.6:0/719946333 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f615806b500 0x7f615806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:53.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.754+0000 7f6179400640 1 --2- 172.21.15.6:0/719946333 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f615806b500 0x7f615806d9c0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f61740c65d0 tx=0x7f6164008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:53.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.756+0000 7f616e000640 1 -- 172.21.15.6:0/719946333 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6168040070 con 0x7f61740b0f90 2024-03-04T09:29:53.841 INFO:tasks.ceph.mds.a.smithi006.stderr:2024-03-04T09:29:53.844+0000 7ff086000640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-04T09:29:53.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.998+0000 7f617b200640 1 -- 172.21.15.6:0/719946333 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f617404f330 con 0x7f61740b0f90 2024-03-04T09:29:53.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:53.999+0000 7f616e000640 1 -- 172.21.15.6:0/719946333 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) v1 ==== 94+0+4976 (secure 0 0 0) 0x7f616807b150 con 0x7f61740b0f90 2024-03-04T09:29:53.996 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:29:53.996 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":51,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5827,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6836/3447344578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":3447344578},{"type":"v1","addr":"172.21.15.146:6836","nonce":3447344578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":46},{"gid":5830,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/168994142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":168994142},{"type":"v1","addr":"172.21.15.146:6837","nonce":168994142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":46},{"gid":6735,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/3369489999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":3369489999},{"type":"v1","addr":"172.21.15.6:6833","nonce":3369489999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":45}],"filesystems":[{"mdsmap":{"epoch":51,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:29:47.357677+0000","modified":"2024-03-04T09:29:50.579389+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6744},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6744":{"gid":6744,"name":"a","rank":0,"incarnation":47,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6835/1667002065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1667002065},{"type":"v1","addr":"172.21.15.6:6835","nonce":1667002065}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6744,"qdb_cluster":[6744]},"id":6}]} 2024-03-04T09:29:53.997 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 51 2024-03-04T09:29:53.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.002+0000 7f617b200640 1 -- 172.21.15.6:0/719946333 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f615806b500 msgr2=0x7f615806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:53.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.002+0000 7f617b200640 1 --2- 172.21.15.6:0/719946333 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f615806b500 0x7f615806d9c0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f61740c65d0 tx=0x7f6164008040 comp rx=0 tx=0).stop 2024-03-04T09:29:53.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.002+0000 7f617b200640 1 -- 172.21.15.6:0/719946333 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f61740b0f90 msgr2=0x7f6174157c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:53.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.002+0000 7f617b200640 1 --2- 172.21.15.6:0/719946333 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f61740b0f90 0x7f6174157c80 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f616802b8c0 tx=0x7f61680046c0 comp rx=0 tx=0).stop 2024-03-04T09:29:53.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.002+0000 7f617b200640 1 -- 172.21.15.6:0/719946333 shutdown_connections 2024-03-04T09:29:53.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.002+0000 7f617b200640 1 --2- 172.21.15.6:0/719946333 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f61740d05e0 0x7f617415a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:53.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.002+0000 7f617b200640 1 --2- 172.21.15.6:0/719946333 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f615806b500 0x7f615806d9c0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:53.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.002+0000 7f617b200640 1 --2- 172.21.15.6:0/719946333 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f61740cc730 0x7f6174156830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:53.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.002+0000 7f617b200640 1 --2- 172.21.15.6:0/719946333 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f61740b0f90 0x7f6174157c80 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:53.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.002+0000 7f617b200640 1 -- 172.21.15.6:0/719946333 >> 172.21.15.6:0/719946333 conn(0x7f61740d1ae0 msgr2=0x7f6174160eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:29:53.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.002+0000 7f617b200640 1 -- 172.21.15.6:0/719946333 shutdown_connections 2024-03-04T09:29:53.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.002+0000 7f617b200640 1 -- 172.21.15.6:0/719946333 wait complete. 2024-03-04T09:29:54.427 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 51, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:29:47.357677+0000', 'modified': '2024-03-04T09:29:50.579389+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6744}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6744': {'gid': 6744, 'name': 'a', 'rank': 0, 'incarnation': 47, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6835/1667002065', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1667002065}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1667002065}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 6744, 'qdb_cluster': [6744]} 2024-03-04T09:29:54.428 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-04T09:29:54.428 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-04T09:29:54.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.655+0000 7fec1c800640 1 Processor -- start 2024-03-04T09:29:54.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.655+0000 7fec1c800640 1 -- start start 2024-03-04T09:29:54.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.656+0000 7fec1c800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fec140af1a0 0x7fec1415c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:54.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.657+0000 7fec1c800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fec140af580 0x7fec140b4c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:54.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.657+0000 7fec1c800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fec140cdc20 0x7fec140b2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:54.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.657+0000 7fec1c800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec1415cc40 con 0x7fec140cdc20 2024-03-04T09:29:54.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.657+0000 7fec1c800640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec1415cf20 con 0x7fec140af580 2024-03-04T09:29:54.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.657+0000 7fec1c800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec1415d200 con 0x7fec140af1a0 2024-03-04T09:29:54.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.657+0000 7fec1a800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fec140cdc20 0x7fec140b2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:54.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.657+0000 7fec1a800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fec140cdc20 0x7fec140b2dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38778/0 (socket says 172.21.15.6:38778) 2024-03-04T09:29:54.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.657+0000 7fec1a800640 1 -- 172.21.15.6:0/1411260837 learned_addr learned my addr 172.21.15.6:0/1411260837 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:29:54.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.657+0000 7fec19e00640 1 --2- 172.21.15.6:0/1411260837 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fec140af1a0 0x7fec1415c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:54.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.657+0000 7fec19400640 1 --2- 172.21.15.6:0/1411260837 >> v2:172.21.15.146:3300/0 conn(0x7fec140af580 0x7fec140b4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:54.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.658+0000 7fec1a800640 1 -- 172.21.15.6:0/1411260837 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fec140af1a0 msgr2=0x7fec1415c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:54.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.658+0000 7fec1a800640 1 --2- 172.21.15.6:0/1411260837 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fec140af1a0 0x7fec1415c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:54.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.658+0000 7fec1a800640 1 -- 172.21.15.6:0/1411260837 >> v2:172.21.15.146:3300/0 conn(0x7fec140af580 msgr2=0x7fec140b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:54.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.658+0000 7fec1a800640 1 --2- 172.21.15.6:0/1411260837 >> v2:172.21.15.146:3300/0 conn(0x7fec140af580 0x7fec140b4c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:54.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.658+0000 7fec1a800640 1 -- 172.21.15.6:0/1411260837 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec1415d4e0 con 0x7fec140cdc20 2024-03-04T09:29:54.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.658+0000 7fec19400640 1 --2- 172.21.15.6:0/1411260837 >> v2:172.21.15.146:3300/0 conn(0x7fec140af580 0x7fec140b4c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T09:29:54.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.659+0000 7fec1a800640 1 --2- 172.21.15.6:0/1411260837 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fec140cdc20 0x7fec140b2dd0 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7fec140748c0 tx=0x7fec1002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27349eb3c26c65e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:54.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.660+0000 7fec18a00640 1 -- 172.21.15.6:0/1411260837 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec1002c9e0 con 0x7fec140cdc20 2024-03-04T09:29:54.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.660+0000 7fec18a00640 1 -- 172.21.15.6:0/1411260837 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec1002cb80 con 0x7fec140cdc20 2024-03-04T09:29:54.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.660+0000 7fec18a00640 1 -- 172.21.15.6:0/1411260837 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec100343f0 con 0x7fec140cdc20 2024-03-04T09:29:54.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.660+0000 7fec1c800640 1 -- 172.21.15.6:0/1411260837 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fec140cdc20 msgr2=0x7fec140b2dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:54.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.660+0000 7fec1c800640 1 --2- 172.21.15.6:0/1411260837 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fec140cdc20 0x7fec140b2dd0 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7fec140748c0 tx=0x7fec1002be40 comp rx=0 tx=0).stop 2024-03-04T09:29:54.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.661+0000 7fec1c800640 1 -- 172.21.15.6:0/1411260837 shutdown_connections 2024-03-04T09:29:54.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.661+0000 7fec1c800640 1 --2- 172.21.15.6:0/1411260837 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fec140cdc20 0x7fec140b2dd0 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:54.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.661+0000 7fec1c800640 1 --2- 172.21.15.6:0/1411260837 >> v2:172.21.15.146:3300/0 conn(0x7fec140af580 0x7fec140b4c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:54.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.661+0000 7fec1c800640 1 --2- 172.21.15.6:0/1411260837 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fec140af1a0 0x7fec1415c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:54.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.661+0000 7fec1c800640 1 -- 172.21.15.6:0/1411260837 >> 172.21.15.6:0/1411260837 conn(0x7fec1406afb0 msgr2=0x7fec1406b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:29:54.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.661+0000 7fec1c800640 1 -- 172.21.15.6:0/1411260837 shutdown_connections 2024-03-04T09:29:54.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.661+0000 7fec1c800640 1 -- 172.21.15.6:0/1411260837 wait complete. 2024-03-04T09:29:54.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.663+0000 7fec1c800640 1 Processor -- start 2024-03-04T09:29:54.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.664+0000 7fec1c800640 1 -- start start 2024-03-04T09:29:54.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.665+0000 7fec1c800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fec140af1a0 0x7fec1414b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:54.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.665+0000 7fec1c800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fec140af580 0x7fec14158090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:54.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.665+0000 7fec1c800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fec140cdc20 0x7fec14158e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:54.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.665+0000 7fec1c800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec140b3490 con 0x7fec140cdc20 2024-03-04T09:29:54.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.665+0000 7fec1c800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec140b3310 con 0x7fec140af580 2024-03-04T09:29:54.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.665+0000 7fec1c800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec140b3610 con 0x7fec140af1a0 2024-03-04T09:29:54.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.666+0000 7fec1a800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fec140cdc20 0x7fec14158e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:54.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.666+0000 7fec1a800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fec140cdc20 0x7fec14158e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38780/0 (socket says 172.21.15.6:38780) 2024-03-04T09:29:54.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.666+0000 7fec1a800640 1 -- 172.21.15.6:0/2013769141 learned_addr learned my addr 172.21.15.6:0/2013769141 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:29:54.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.666+0000 7fec19e00640 1 --2- 172.21.15.6:0/2013769141 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fec140af1a0 0x7fec1414b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:54.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.666+0000 7fec19400640 1 --2- 172.21.15.6:0/2013769141 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fec140af580 0x7fec14158090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:54.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.666+0000 7fec1a800640 1 -- 172.21.15.6:0/2013769141 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fec140af1a0 msgr2=0x7fec1414b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:54.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.666+0000 7fec1a800640 1 --2- 172.21.15.6:0/2013769141 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fec140af1a0 0x7fec1414b580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:54.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.666+0000 7fec1a800640 1 -- 172.21.15.6:0/2013769141 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fec140af580 msgr2=0x7fec14158090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:54.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.666+0000 7fec1a800640 1 --2- 172.21.15.6:0/2013769141 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fec140af580 0x7fec14158090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:54.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.666+0000 7fec1a800640 1 -- 172.21.15.6:0/2013769141 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec140cabe0 con 0x7fec140cdc20 2024-03-04T09:29:54.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.667+0000 7fec1a800640 1 --2- 172.21.15.6:0/2013769141 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fec140cdc20 0x7fec14158e10 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7fec10036040 tx=0x7fec10004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:54.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.667+0000 7fec0e000640 1 -- 172.21.15.6:0/2013769141 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec10044020 con 0x7fec140cdc20 2024-03-04T09:29:54.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.667+0000 7fec0e000640 1 -- 172.21.15.6:0/2013769141 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec10040690 con 0x7fec140cdc20 2024-03-04T09:29:54.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.667+0000 7fec1c800640 1 -- 172.21.15.6:0/2013769141 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec140caec0 con 0x7fec140cdc20 2024-03-04T09:29:54.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.667+0000 7fec0e000640 1 -- 172.21.15.6:0/2013769141 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec10009510 con 0x7fec140cdc20 2024-03-04T09:29:54.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.667+0000 7fec1c800640 1 -- 172.21.15.6:0/2013769141 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec140cb480 con 0x7fec140cdc20 2024-03-04T09:29:54.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.670+0000 7fec0e000640 1 -- 172.21.15.6:0/2013769141 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fec10047020 con 0x7fec140cdc20 2024-03-04T09:29:54.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.672+0000 7fec0e000640 1 --2- 172.21.15.6:0/2013769141 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7febf806b4d0 0x7febf806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:54.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.672+0000 7fec1f5cf740 1 Processor -- start 2024-03-04T09:29:54.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.672+0000 7fec0e000640 1 -- 172.21.15.6:0/2013769141 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fec100acdb0 con 0x7fec140cdc20 2024-03-04T09:29:54.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.672+0000 7fec1f5cf740 1 -- start start 2024-03-04T09:29:54.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.673+0000 7fec19e00640 1 --2- 172.21.15.6:0/2013769141 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7febf806b4d0 0x7febf806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:54.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.673+0000 7fec1f5cf740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56033cc7b7b0 0x56033cc4e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:54.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.673+0000 7fec1f5cf740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56033cc7bb90 0x56033cc7e540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:54.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.673+0000 7fec1f5cf740 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x56033cc4ec90 0x56033cc7ea80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:54.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.673+0000 7fec1f5cf740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x56033cbb8600 con 0x56033cc7b7b0 2024-03-04T09:29:54.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.673+0000 7fec1f5cf740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x56033cb61930 con 0x56033cc7bb90 2024-03-04T09:29:54.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.673+0000 7fec1f5cf740 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x56033cbb46a0 con 0x56033cc4ec90 2024-03-04T09:29:54.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.674+0000 7fec19e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56033cc7b7b0 0x56033cc4e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:54.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.674+0000 7fec19e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56033cc7b7b0 0x56033cc4e750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38790/0 (socket says 172.21.15.6:38790) 2024-03-04T09:29:54.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.674+0000 7fec19e00640 1 -- 172.21.15.6:0/1633534740 learned_addr learned my addr 172.21.15.6:0/1633534740 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:29:54.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.674+0000 7fec19400640 1 --2- 172.21.15.6:0/1633534740 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56033cc7bb90 0x56033cc7e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:54.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.674+0000 7fec1a800640 1 --2- 172.21.15.6:0/1633534740 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x56033cc4ec90 0x56033cc7ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:54.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.674+0000 7fec19e00640 1 -- 172.21.15.6:0/1633534740 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x56033cc4ec90 msgr2=0x56033cc7ea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:54.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.674+0000 7fec19e00640 1 --2- 172.21.15.6:0/1633534740 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x56033cc4ec90 0x56033cc7ea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:54.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.674+0000 7fec19e00640 1 -- 172.21.15.6:0/1633534740 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56033cc7bb90 msgr2=0x56033cc7e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:54.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.674+0000 7fec19e00640 1 --2- 172.21.15.6:0/1633534740 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56033cc7bb90 0x56033cc7e540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:54.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.674+0000 7fec19e00640 1 -- 172.21.15.6:0/1633534740 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56033c8cd2d0 con 0x56033cc7b7b0 2024-03-04T09:29:54.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.674+0000 7fec19e00640 1 --2- 172.21.15.6:0/2013769141 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7febf806b4d0 0x7febf806d990 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fec08006fd0 tx=0x7fec0800be70 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:54.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.675+0000 7fec19e00640 1 --2- 172.21.15.6:0/1633534740 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56033cc7b7b0 0x56033cc4e750 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7fec0800fab0 tx=0x7fec0801a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec457e68dbca09d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:54.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.675+0000 7fec02600640 1 -- 172.21.15.6:0/1633534740 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec08025070 con 0x56033cc7b7b0 2024-03-04T09:29:54.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.675+0000 7fec02600640 1 -- 172.21.15.6:0/1633534740 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec08018cc0 con 0x56033cc7b7b0 2024-03-04T09:29:54.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.675+0000 7fec02600640 1 -- 172.21.15.6:0/1633534740 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec0801c040 con 0x56033cc7b7b0 2024-03-04T09:29:54.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.676+0000 7fec1f5cf740 1 -- 172.21.15.6:0/1633534740 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56033cc7b7b0 msgr2=0x56033cc4e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:54.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.676+0000 7fec1f5cf740 1 --2- 172.21.15.6:0/1633534740 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56033cc7b7b0 0x56033cc4e750 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7fec0800fab0 tx=0x7fec0801a000 comp rx=0 tx=0).stop 2024-03-04T09:29:54.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.676+0000 7fec1f5cf740 1 -- 172.21.15.6:0/1633534740 shutdown_connections 2024-03-04T09:29:54.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.676+0000 7fec1f5cf740 1 --2- 172.21.15.6:0/1633534740 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x56033cc4ec90 0x56033cc7ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:54.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.676+0000 7fec1f5cf740 1 --2- 172.21.15.6:0/1633534740 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56033cc7bb90 0x56033cc7e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:54.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.676+0000 7fec1f5cf740 1 --2- 172.21.15.6:0/1633534740 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56033cc7b7b0 0x56033cc4e750 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:54.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.676+0000 7fec1f5cf740 1 -- 172.21.15.6:0/1633534740 >> 172.21.15.6:0/1633534740 conn(0x56033cc509c0 msgr2=0x56033cc99540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:29:54.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.676+0000 7fec1f5cf740 1 -- 172.21.15.6:0/1633534740 shutdown_connections 2024-03-04T09:29:54.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.676+0000 7fec1f5cf740 1 -- 172.21.15.6:0/1633534740 wait complete. 2024-03-04T09:29:54.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.677+0000 7fec1f5cf740 1 -- start start 2024-03-04T09:29:54.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.677+0000 7fec1f5cf740 1 Processor -- start 2024-03-04T09:29:54.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.679+0000 7fec1f5cf740 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x56033cc7b7b0 0x56033cc65a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:54.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.679+0000 7fec1f5cf740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56033cc7bb90 0x56033cc81e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:54.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.679+0000 7fec1f5cf740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56033cc7a5b0 0x56033cc75060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:54.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.679+0000 7fec1f5cf740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x56033cc52f60 con 0x56033cc7a5b0 2024-03-04T09:29:54.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.679+0000 7fec1f5cf740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x56033cb74d90 con 0x56033cc7bb90 2024-03-04T09:29:54.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.679+0000 7fec1f5cf740 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x56033cc52350 con 0x56033cc7b7b0 2024-03-04T09:29:54.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.679+0000 7fec1f5cf740 10 client.0 _ll_register_callbacks cb 0x56033cbb0350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T09:29:54.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.679+0000 7fec1a800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56033cc7a5b0 0x56033cc75060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:54.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.680+0000 7fec1a800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56033cc7a5b0 0x56033cc75060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38796/0 (socket says 172.21.15.6:38796) 2024-03-04T09:29:54.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.680+0000 7fec1a800640 1 -- 172.21.15.6:0/1820859 learned_addr learned my addr 172.21.15.6:0/1820859 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:29:54.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.680+0000 7fec1a800640 1 -- 172.21.15.6:0/1820859 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x56033cc7b7b0 msgr2=0x56033cc65a90 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-04T09:29:54.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.680+0000 7fec1a800640 1 --2- 172.21.15.6:0/1820859 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x56033cc7b7b0 0x56033cc65a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:54.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.680+0000 7fec1a800640 1 -- 172.21.15.6:0/1820859 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56033cc7bb90 msgr2=0x56033cc81e00 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-04T09:29:54.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.680+0000 7fec1a800640 1 --2- 172.21.15.6:0/1820859 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56033cc7bb90 0x56033cc81e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:54.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.680+0000 7fec1a800640 1 -- 172.21.15.6:0/1820859 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56033c9976b0 con 0x56033cc7a5b0 2024-03-04T09:29:54.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.680+0000 7fec1a800640 1 --2- 172.21.15.6:0/1820859 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56033cc7a5b0 0x56033cc75060 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7fec100ab300 tx=0x7fec1007bcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=448577b31ad67fa0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:54.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.680+0000 7febf6a00640 10 client.0 ms_handle_connect on v2:172.21.15.6:3300/0 2024-03-04T09:29:54.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.680+0000 7febf6a00640 1 -- 172.21.15.6:0/1820859 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec10083c40 con 0x56033cc7a5b0 2024-03-04T09:29:54.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.680+0000 7febf6a00640 1 -- 172.21.15.6:0/1820859 <== mon.0 v2:172.21.15.6:3300/0 2 ==== osd_map(81..81 src has 1..81) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fec1002c9f0 con 0x56033cc7a5b0 2024-03-04T09:29:54.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.680+0000 7fec1f5cf740 1 -- 172.21.15.6:0/1820859 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fec08018cc0 con 0x56033cc7a5b0 2024-03-04T09:29:54.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.681+0000 7febf6a00640 1 -- 172.21.15.6:0/1820859 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) v2 ==== 24+0+0 (secure 0 0 0) 0x7fec100a9050 con 0x56033cc7a5b0 2024-03-04T09:29:54.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.681+0000 7fec1f5cf740 10 client.6876 fetch_fsmap learned FSMap version 51 2024-03-04T09:29:54.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.681+0000 7fec1f5cf740 1 -- 172.21.15.6:0/1820859 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=2+}) v3 -- 0x56033cc81600 con 0x56033cc7a5b0 2024-03-04T09:29:54.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.681+0000 7febf6a00640 1 -- 172.21.15.6:0/1820859 <== mon.0 v2:172.21.15.6:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec08018cc0 con 0x56033cc7a5b0 2024-03-04T09:29:54.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.681+0000 7febf6a00640 1 -- 172.21.15.6:0/1820859 <== mon.0 v2:172.21.15.6:3300/0 5 ==== fsmap(e 51) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fec1007b790 con 0x56033cc7a5b0 2024-03-04T09:29:54.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.681+0000 7fec1f5cf740 10 client.6876 fetch_fsmap finished waiting for FSMap version 51 2024-03-04T09:29:54.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.681+0000 7fec1f5cf740 10 client.6876 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-04T09:29:54.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.681+0000 7fec1f5cf740 20 client.6876 populate_metadata read hostname 'smithi006' 2024-03-04T09:29:54.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.681+0000 7fec1f5cf740 1 --2- 172.21.15.6:0/1820859 >> [v2:172.21.15.6:6834/1667002065,v1:172.21.15.6:6835/1667002065] conn(0x56033ccace80 0x56033ccad260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:54.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.681+0000 7fec1f5cf740 4 client.6876 mds_command: new command op to 6744 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T09:29:54.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.681+0000 7fec1f5cf740 1 -- 172.21.15.6:0/1820859 --> [v2:172.21.15.6:6834/1667002065,v1:172.21.15.6:6835/1667002065] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56033cbc5b40 con 0x56033ccace80 2024-03-04T09:29:54.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.681+0000 7fec19e00640 1 --2- 172.21.15.6:0/1820859 >> [v2:172.21.15.6:6834/1667002065,v1:172.21.15.6:6835/1667002065] conn(0x56033ccace80 0x56033ccad260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:54.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.682+0000 7fec19e00640 1 --2- 172.21.15.6:0/1820859 >> [v2:172.21.15.6:6834/1667002065,v1:172.21.15.6:6835/1667002065] conn(0x56033ccace80 0x56033ccad260 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bb5f12e1d027e8b7 server_cookie=22151196a217bf43 in_seq=0 out_seq=0 2024-03-04T09:29:54.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.682+0000 7febf6a00640 10 client.6876 ms_handle_connect on v2:172.21.15.6:6834/1667002065 2024-03-04T09:29:54.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.683+0000 7febf6a00640 1 -- 172.21.15.6:0/1820859 <== mds.0 v2:172.21.15.6:6834/1667002065 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x56033cbc5b40 con 0x56033ccace80 2024-03-04T09:29:54.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.683+0000 7febf6a00640 10 client.6876 handle_command_reply: tid=0 2024-03-04T09:29:54.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.683+0000 7fec1f5cf740 1 client.6876 shutdown 2024-03-04T09:29:54.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.683+0000 7fec1f5cf740 1 -- 172.21.15.6:0/1820859 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56033cc7a5b0 msgr2=0x56033cc75060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:54.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.683+0000 7fec1f5cf740 1 --2- 172.21.15.6:0/1820859 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56033cc7a5b0 0x56033cc75060 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7fec100ab300 tx=0x7fec1007bcf0 comp rx=0 tx=0).stop 2024-03-04T09:29:54.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.683+0000 7fec1f5cf740 1 -- 172.21.15.6:0/1820859 shutdown_connections 2024-03-04T09:29:54.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.683+0000 7fec1f5cf740 1 --2- 172.21.15.6:0/1820859 >> [v2:172.21.15.6:6834/1667002065,v1:172.21.15.6:6835/1667002065] conn(0x56033ccace80 0x56033ccad260 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:54.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.683+0000 7fec1f5cf740 1 --2- 172.21.15.6:0/1820859 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56033cc7a5b0 0x56033cc75060 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:54.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.683+0000 7fec1f5cf740 1 --2- 172.21.15.6:0/1820859 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56033cc7bb90 0x56033cc81e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:54.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.683+0000 7fec1f5cf740 1 --2- 172.21.15.6:0/1820859 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x56033cc7b7b0 0x56033cc65a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:54.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.683+0000 7fec1f5cf740 1 -- 172.21.15.6:0/1820859 >> 172.21.15.6:0/1820859 conn(0x56033cc4ec90 msgr2=0x56033cc7e530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:29:54.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.683+0000 7febf6a00640 0 client.6876 ms_handle_reset on v2:172.21.15.6:6834/1667002065 2024-03-04T09:29:54.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.683+0000 7fec1f5cf740 1 -- 172.21.15.6:0/1820859 shutdown_connections 2024-03-04T09:29:54.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.683+0000 7fec1f5cf740 1 -- 172.21.15.6:0/1820859 wait complete. 2024-03-04T09:29:54.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.684+0000 7fec1f5cf740 20 client.6876 trim_cache size 0 max 16384 2024-03-04T09:29:54.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.702+0000 7fec1f5cf740 1 Processor -- start 2024-03-04T09:29:54.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.702+0000 7fec1f5cf740 1 -- start start 2024-03-04T09:29:54.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.703+0000 7fec1f5cf740 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x56033cc7a5b0 0x56033cc54820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:54.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.703+0000 7fec1f5cf740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56033cc7b7b0 0x56033ccab780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:54.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.703+0000 7fec1f5cf740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56033cc7bb90 0x56033cc9ba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:54.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.703+0000 7fec1f5cf740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec141ff380 con 0x56033cc7bb90 2024-03-04T09:29:54.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.703+0000 7fec1f5cf740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec141fe7b0 con 0x56033cc7b7b0 2024-03-04T09:29:54.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.703+0000 7fec1f5cf740 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec141fedb0 con 0x56033cc7a5b0 2024-03-04T09:29:54.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.703+0000 7fec1a800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56033cc7bb90 0x56033cc9ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:54.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.703+0000 7fec1a800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56033cc7bb90 0x56033cc9ba40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38806/0 (socket says 172.21.15.6:38806) 2024-03-04T09:29:54.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.703+0000 7fec1a800640 1 -- 172.21.15.6:0/2566916206 learned_addr learned my addr 172.21.15.6:0/2566916206 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:29:54.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.704+0000 7fec19e00640 1 --2- 172.21.15.6:0/2566916206 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x56033cc7a5b0 0x56033cc54820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:54.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.704+0000 7fec19400640 1 --2- 172.21.15.6:0/2566916206 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56033cc7b7b0 0x56033ccab780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:54.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.704+0000 7fec1a800640 1 -- 172.21.15.6:0/2566916206 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x56033cc7a5b0 msgr2=0x56033cc54820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:54.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.704+0000 7fec1a800640 1 --2- 172.21.15.6:0/2566916206 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x56033cc7a5b0 0x56033cc54820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:54.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.704+0000 7fec1a800640 1 -- 172.21.15.6:0/2566916206 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56033cc7b7b0 msgr2=0x56033ccab780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:54.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.704+0000 7fec1a800640 1 --2- 172.21.15.6:0/2566916206 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56033cc7b7b0 0x56033ccab780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:54.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.704+0000 7fec1a800640 1 -- 172.21.15.6:0/2566916206 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56033cc7b160 con 0x56033cc7bb90 2024-03-04T09:29:54.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.704+0000 7fec1a800640 1 --2- 172.21.15.6:0/2566916206 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56033cc7bb90 0x56033cc9ba40 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7fec10077990 tx=0x7fec1004a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=597484f4fbb9302d server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:54.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.704+0000 7fec01c00640 1 -- 172.21.15.6:0/2566916206 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec10083c40 con 0x56033cc7bb90 2024-03-04T09:29:54.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.704+0000 7fec01c00640 1 -- 172.21.15.6:0/2566916206 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec1007b790 con 0x56033cc7bb90 2024-03-04T09:29:54.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.705+0000 7fec01c00640 1 -- 172.21.15.6:0/2566916206 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec100837b0 con 0x56033cc7bb90 2024-03-04T09:29:54.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.705+0000 7fec1f5cf740 1 -- 172.21.15.6:0/2566916206 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56033cc7bb90 msgr2=0x56033cc9ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:54.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.705+0000 7fec1f5cf740 1 --2- 172.21.15.6:0/2566916206 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56033cc7bb90 0x56033cc9ba40 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7fec10077990 tx=0x7fec1004a000 comp rx=0 tx=0).stop 2024-03-04T09:29:54.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.705+0000 7fec1f5cf740 1 -- 172.21.15.6:0/2566916206 shutdown_connections 2024-03-04T09:29:54.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.705+0000 7fec1f5cf740 1 --2- 172.21.15.6:0/2566916206 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56033cc7bb90 0x56033cc9ba40 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:54.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.705+0000 7fec1f5cf740 1 --2- 172.21.15.6:0/2566916206 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56033cc7b7b0 0x56033ccab780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:54.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.705+0000 7fec1f5cf740 1 --2- 172.21.15.6:0/2566916206 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x56033cc7a5b0 0x56033cc54820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:54.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.705+0000 7fec1f5cf740 1 -- 172.21.15.6:0/2566916206 >> 172.21.15.6:0/2566916206 conn(0x56033cc4ec90 msgr2=0x56033cc7e530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:29:54.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.705+0000 7fec1f5cf740 1 -- 172.21.15.6:0/2566916206 shutdown_connections 2024-03-04T09:29:54.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.705+0000 7fec1f5cf740 1 -- 172.21.15.6:0/2566916206 wait complete. 2024-03-04T09:29:54.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.706+0000 7fec1f5cf740 1 -- start start 2024-03-04T09:29:54.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.706+0000 7fec1f5cf740 1 Processor -- start 2024-03-04T09:29:54.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.707+0000 7fec1f5cf740 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x56033cc7a5b0 0x56033cc637e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:54.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.707+0000 7fec1f5cf740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56033cc7b7b0 0x56033cc63d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:54.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.707+0000 7fec1f5cf740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56033cc7bb90 0x56033cc9ad10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:54.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.707+0000 7fec1f5cf740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x56033cc52f60 con 0x56033cc7bb90 2024-03-04T09:29:54.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.707+0000 7fec1f5cf740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x56033cb74d90 con 0x56033cc7b7b0 2024-03-04T09:29:54.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.707+0000 7fec1f5cf740 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x56033cc77b30 con 0x56033cc7a5b0 2024-03-04T09:29:54.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.707+0000 7fec1f5cf740 10 client.0 _ll_register_callbacks cb 0x7fec10009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T09:29:54.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.707+0000 7fec1a800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56033cc7bb90 0x56033cc9ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:54.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.707+0000 7fec1a800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56033cc7bb90 0x56033cc9ad10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38818/0 (socket says 172.21.15.6:38818) 2024-03-04T09:29:54.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.707+0000 7fec1a800640 1 -- 172.21.15.6:0/3469187509 learned_addr learned my addr 172.21.15.6:0/3469187509 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:29:54.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.708+0000 7fec19e00640 1 --2- 172.21.15.6:0/3469187509 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x56033cc7a5b0 0x56033cc637e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:54.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.708+0000 7fec19400640 1 --2- 172.21.15.6:0/3469187509 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56033cc7b7b0 0x56033cc63d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:54.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.708+0000 7fec1a800640 1 -- 172.21.15.6:0/3469187509 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x56033cc7a5b0 msgr2=0x56033cc637e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:54.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.708+0000 7fec1a800640 1 --2- 172.21.15.6:0/3469187509 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x56033cc7a5b0 0x56033cc637e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:54.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.708+0000 7fec1a800640 1 -- 172.21.15.6:0/3469187509 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56033cc7b7b0 msgr2=0x56033cc63d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:54.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.708+0000 7fec1a800640 1 --2- 172.21.15.6:0/3469187509 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56033cc7b7b0 0x56033cc63d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:54.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.708+0000 7fec1a800640 1 -- 172.21.15.6:0/3469187509 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56033cca9030 con 0x56033cc7bb90 2024-03-04T09:29:54.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.708+0000 7fec19e00640 1 --2- 172.21.15.6:0/3469187509 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x56033cc7a5b0 0x56033cc637e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:29:54.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.708+0000 7fec1a800640 1 --2- 172.21.15.6:0/3469187509 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56033cc7bb90 0x56033cc9ad10 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7fec10077eb0 tx=0x7fec100ab1b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f5f6d9ee6aeb08 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:54.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.708+0000 7febf6a00640 10 client.0 ms_handle_connect on v2:172.21.15.6:3300/0 2024-03-04T09:29:54.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.708+0000 7febf6a00640 1 -- 172.21.15.6:0/3469187509 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec1009e030 con 0x56033cc7bb90 2024-03-04T09:29:54.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.708+0000 7febf6a00640 1 -- 172.21.15.6:0/3469187509 <== mon.0 v2:172.21.15.6:3300/0 2 ==== osd_map(81..81 src has 1..81) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fec10079830 con 0x56033cc7bb90 2024-03-04T09:29:54.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.708+0000 7fec1f5cf740 1 -- 172.21.15.6:0/3469187509 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fec1007b790 con 0x56033cc7bb90 2024-03-04T09:29:54.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.709+0000 7febf6a00640 1 -- 172.21.15.6:0/3469187509 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) v2 ==== 24+0+0 (secure 0 0 0) 0x7fec10099070 con 0x56033cc7bb90 2024-03-04T09:29:54.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.709+0000 7fec1f5cf740 10 client.6882 fetch_fsmap learned FSMap version 51 2024-03-04T09:29:54.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.709+0000 7fec1f5cf740 1 -- 172.21.15.6:0/3469187509 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=2+}) v3 -- 0x56033cca9030 con 0x56033cc7bb90 2024-03-04T09:29:54.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.709+0000 7febf6a00640 1 -- 172.21.15.6:0/3469187509 <== mon.0 v2:172.21.15.6:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec1007b790 con 0x56033cc7bb90 2024-03-04T09:29:54.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.709+0000 7febf6a00640 1 -- 172.21.15.6:0/3469187509 <== mon.0 v2:172.21.15.6:3300/0 5 ==== fsmap(e 51) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fec10077280 con 0x56033cc7bb90 2024-03-04T09:29:54.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.709+0000 7fec1f5cf740 10 client.6882 fetch_fsmap finished waiting for FSMap version 51 2024-03-04T09:29:54.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.709+0000 7fec1f5cf740 10 client.6882 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-04T09:29:54.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.709+0000 7fec1f5cf740 20 client.6882 populate_metadata read hostname 'smithi006' 2024-03-04T09:29:54.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.709+0000 7fec1f5cf740 1 --2- 172.21.15.6:0/3469187509 >> [v2:172.21.15.6:6834/1667002065,v1:172.21.15.6:6835/1667002065] conn(0x56033cc9b250 0x56033ccbbf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:54.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.709+0000 7fec1f5cf740 4 client.6882 mds_command: new command op to 6744 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-04T09:29:54.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.709+0000 7fec1f5cf740 1 -- 172.21.15.6:0/3469187509 --> [v2:172.21.15.6:6834/1667002065,v1:172.21.15.6:6835/1667002065] -- command(tid 0: {"prefix": "status"}) v1 -- 0x56033cbbfe90 con 0x56033cc9b250 2024-03-04T09:29:54.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.709+0000 7fec19e00640 1 --2- 172.21.15.6:0/3469187509 >> [v2:172.21.15.6:6834/1667002065,v1:172.21.15.6:6835/1667002065] conn(0x56033cc9b250 0x56033ccbbf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:54.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.710+0000 7fec19e00640 1 --2- 172.21.15.6:0/3469187509 >> [v2:172.21.15.6:6834/1667002065,v1:172.21.15.6:6835/1667002065] conn(0x56033cc9b250 0x56033ccbbf50 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=36a34bdadb05b9a6 server_cookie=48bae8c02b22d8b3 in_seq=0 out_seq=0 2024-03-04T09:29:54.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.710+0000 7febf6a00640 10 client.6882 ms_handle_connect on v2:172.21.15.6:6834/1667002065 2024-03-04T09:29:54.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.710+0000 7febf6a00640 1 -- 172.21.15.6:0/3469187509 <== mds.0 v2:172.21.15.6:6834/1667002065 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x56033cbbfe90 con 0x56033cc9b250 2024-03-04T09:29:54.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.710+0000 7febf6a00640 10 client.6882 handle_command_reply: tid=0 2024-03-04T09:29:54.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.710+0000 7fec1f5cf740 1 client.6882 shutdown 2024-03-04T09:29:54.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.710+0000 7fec1f5cf740 1 -- 172.21.15.6:0/3469187509 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56033cc7bb90 msgr2=0x56033cc9ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:54.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.710+0000 7fec1f5cf740 1 --2- 172.21.15.6:0/3469187509 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56033cc7bb90 0x56033cc9ad10 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7fec10077eb0 tx=0x7fec100ab1b0 comp rx=0 tx=0).stop 2024-03-04T09:29:54.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.711+0000 7fec1f5cf740 1 -- 172.21.15.6:0/3469187509 shutdown_connections 2024-03-04T09:29:54.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.711+0000 7fec1f5cf740 1 --2- 172.21.15.6:0/3469187509 >> [v2:172.21.15.6:6834/1667002065,v1:172.21.15.6:6835/1667002065] conn(0x56033cc9b250 0x56033ccbbf50 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:54.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.711+0000 7fec1f5cf740 1 --2- 172.21.15.6:0/3469187509 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56033cc7bb90 0x56033cc9ad10 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:54.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.711+0000 7fec1f5cf740 1 --2- 172.21.15.6:0/3469187509 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56033cc7b7b0 0x56033cc63d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:54.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.711+0000 7fec1f5cf740 1 --2- 172.21.15.6:0/3469187509 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x56033cc7a5b0 0x56033cc637e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:54.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.711+0000 7febf6a00640 0 client.6882 ms_handle_reset on v2:172.21.15.6:6834/1667002065 2024-03-04T09:29:54.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.711+0000 7fec1f5cf740 1 -- 172.21.15.6:0/3469187509 >> 172.21.15.6:0/3469187509 conn(0x56033cc4ec90 msgr2=0x56033ccab780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:29:54.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.711+0000 7fec1f5cf740 1 -- 172.21.15.6:0/3469187509 shutdown_connections 2024-03-04T09:29:54.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.711+0000 7fec1f5cf740 1 -- 172.21.15.6:0/3469187509 wait complete. 2024-03-04T09:29:54.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.711+0000 7fec1f5cf740 20 client.6882 trim_cache size 0 max 16384 2024-03-04T09:29:54.713 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-04T09:29:54.713 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_fsid": "edce0578-5ae4-4402-b45a-552bdef84d7c", 2024-03-04T09:29:54.713 INFO:teuthology.orchestra.run.smithi006.stdout: "whoami": 0, 2024-03-04T09:29:54.713 INFO:teuthology.orchestra.run.smithi006.stdout: "id": 6744, 2024-03-04T09:29:54.713 INFO:teuthology.orchestra.run.smithi006.stdout: "want_state": "up:active", 2024-03-04T09:29:54.713 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "up:active", 2024-03-04T09:29:54.713 INFO:teuthology.orchestra.run.smithi006.stdout: "fs_name": "cephfs", 2024-03-04T09:29:54.713 INFO:teuthology.orchestra.run.smithi006.stdout: "rank_uptime": 6.8060788050000003, 2024-03-04T09:29:54.713 INFO:teuthology.orchestra.run.smithi006.stdout: "mdsmap_epoch": 51, 2024-03-04T09:29:54.713 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap_epoch": 81, 2024-03-04T09:29:54.713 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap_epoch_barrier": 81, 2024-03-04T09:29:54.713 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 12.177848869 2024-03-04T09:29:54.713 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-04T09:29:54.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.712+0000 7fec1c800640 1 -- 172.21.15.6:0/2013769141 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7febf806b4d0 msgr2=0x7febf806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:54.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.712+0000 7fec1c800640 1 --2- 172.21.15.6:0/2013769141 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7febf806b4d0 0x7febf806d990 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fec08006fd0 tx=0x7fec0800be70 comp rx=0 tx=0).stop 2024-03-04T09:29:54.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.712+0000 7fec1c800640 1 -- 172.21.15.6:0/2013769141 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fec140cdc20 msgr2=0x7fec14158e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:54.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.712+0000 7fec1c800640 1 --2- 172.21.15.6:0/2013769141 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fec140cdc20 0x7fec14158e10 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7fec10036040 tx=0x7fec10004a40 comp rx=0 tx=0).stop 2024-03-04T09:29:54.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.712+0000 7fec1c800640 1 -- 172.21.15.6:0/2013769141 shutdown_connections 2024-03-04T09:29:54.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.712+0000 7fec1c800640 1 --2- 172.21.15.6:0/2013769141 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fec140cdc20 0x7fec14158e10 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:54.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.712+0000 7fec1c800640 1 --2- 172.21.15.6:0/2013769141 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fec140af580 0x7fec14158090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:54.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.712+0000 7fec1c800640 1 --2- 172.21.15.6:0/2013769141 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7febf806b4d0 0x7febf806d990 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:54.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.712+0000 7fec1c800640 1 --2- 172.21.15.6:0/2013769141 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fec140af1a0 0x7fec1414b580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:54.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.712+0000 7fec1c800640 1 -- 172.21.15.6:0/2013769141 >> 172.21.15.6:0/2013769141 conn(0x7fec1406afb0 msgr2=0x7fec1415a140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:29:54.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.712+0000 7fec1c800640 1 -- 172.21.15.6:0/2013769141 shutdown_connections 2024-03-04T09:29:54.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.712+0000 7fec1c800640 1 -- 172.21.15.6:0/2013769141 wait complete. 2024-03-04T09:29:54.726 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-04T09:29:54.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.925+0000 7feef4400640 1 Processor -- start 2024-03-04T09:29:54.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.925+0000 7feef4400640 1 -- start start 2024-03-04T09:29:54.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.927+0000 7feef4400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7feeec0af580 0x7feeec15c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:54.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.927+0000 7feef4400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7feeec0cdc20 0x7feeec0b2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:54.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.927+0000 7feef4400640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feeec15cc40 con 0x7feeec0af1a0 2024-03-04T09:29:54.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.927+0000 7feef4400640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feeec15cf20 con 0x7feeec0af580 2024-03-04T09:29:54.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.927+0000 7feef4400640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feeec15d200 con 0x7feeec0cdc20 2024-03-04T09:29:54.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.927+0000 7feef1000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7feeec0af580 0x7feeec15c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:54.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.927+0000 7feef1a00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7feeec0af1a0 0x7feeec0b4c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:41986/0 (socket says 172.21.15.6:41986) 2024-03-04T09:29:54.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.927+0000 7feef1a00640 1 -- 172.21.15.6:0/3730426559 learned_addr learned my addr 172.21.15.6:0/3730426559 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:29:54.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.928+0000 7feef2400640 1 --2- 172.21.15.6:0/3730426559 >> v2:172.21.15.146:3300/0 conn(0x7feeec0cdc20 0x7feeec0b2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:54.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.928+0000 7feef1000640 1 -- 172.21.15.6:0/3730426559 >> v2:172.21.15.146:3300/0 conn(0x7feeec0cdc20 msgr2=0x7feeec0b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:54.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.928+0000 7feef1000640 1 --2- 172.21.15.6:0/3730426559 >> v2:172.21.15.146:3300/0 conn(0x7feeec0cdc20 0x7feeec0b2dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:54.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.928+0000 7feef1000640 1 -- 172.21.15.6:0/3730426559 >> v1:172.21.15.6:6789/0 conn(0x7feeec0af1a0 legacy=0x7feeec0b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:54.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.928+0000 7feef1000640 1 -- 172.21.15.6:0/3730426559 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feeec15d4e0 con 0x7feeec0af580 2024-03-04T09:29:54.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.929+0000 7feef1000640 1 --2- 172.21.15.6:0/3730426559 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feeec0af580 0x7feeec15c600 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7feeec06c0c0 tx=0x7feed802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c1def1872020bab server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:54.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.929+0000 7feeebe00640 1 -- 172.21.15.6:0/3730426559 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feed80315a0 con 0x7feeec0af580 2024-03-04T09:29:54.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.929+0000 7feeebe00640 1 -- 172.21.15.6:0/3730426559 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feed8037070 con 0x7feeec0af580 2024-03-04T09:29:54.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.929+0000 7feeebe00640 1 -- 172.21.15.6:0/3730426559 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feed8031960 con 0x7feeec0af580 2024-03-04T09:29:54.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.930+0000 7feef4400640 1 -- 172.21.15.6:0/3730426559 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feeec0af580 msgr2=0x7feeec15c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:54.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.930+0000 7feef4400640 1 --2- 172.21.15.6:0/3730426559 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feeec0af580 0x7feeec15c600 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7feeec06c0c0 tx=0x7feed802b840 comp rx=0 tx=0).stop 2024-03-04T09:29:54.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.930+0000 7feef4400640 1 -- 172.21.15.6:0/3730426559 shutdown_connections 2024-03-04T09:29:54.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.930+0000 7feef4400640 1 --2- 172.21.15.6:0/3730426559 >> v2:172.21.15.146:3300/0 conn(0x7feeec0cdc20 0x7feeec0b2dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:54.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.930+0000 7feef4400640 1 --2- 172.21.15.6:0/3730426559 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feeec0af580 0x7feeec15c600 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:54.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.930+0000 7feef4400640 1 -- 172.21.15.6:0/3730426559 >> 172.21.15.6:0/3730426559 conn(0x7feeec06afb0 msgr2=0x7feeec06b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:29:54.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.930+0000 7feef4400640 1 -- 172.21.15.6:0/3730426559 shutdown_connections 2024-03-04T09:29:54.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.930+0000 7feef4400640 1 -- 172.21.15.6:0/3730426559 wait complete. 2024-03-04T09:29:54.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.932+0000 7feef4400640 1 Processor -- start 2024-03-04T09:29:54.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.932+0000 7feef4400640 1 -- start start 2024-03-04T09:29:54.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.933+0000 7feef4400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feeec0af1a0 0x7feeec14a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:54.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.933+0000 7feef4400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feeec0af580 0x7feeec1f14c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:54.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.933+0000 7feef4400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feeec0cdc20 0x7feeec14cc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:54.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.933+0000 7feef4400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feeec0b36a0 con 0x7feeec0cdc20 2024-03-04T09:29:54.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.933+0000 7feef4400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feeec0b3520 con 0x7feeec0af580 2024-03-04T09:29:54.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.933+0000 7feef4400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feeec0b3820 con 0x7feeec0af1a0 2024-03-04T09:29:54.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.934+0000 7feef2400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feeec0cdc20 0x7feeec14cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:54.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.934+0000 7feef2400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feeec0cdc20 0x7feeec14cc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38838/0 (socket says 172.21.15.6:38838) 2024-03-04T09:29:54.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.934+0000 7feef2400640 1 -- 172.21.15.6:0/2036714246 learned_addr learned my addr 172.21.15.6:0/2036714246 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:29:54.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.934+0000 7feef1000640 1 --2- 172.21.15.6:0/2036714246 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feeec0af580 0x7feeec1f14c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:54.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.934+0000 7feef1a00640 1 --2- 172.21.15.6:0/2036714246 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feeec0af1a0 0x7feeec14a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:54.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.934+0000 7feef2400640 1 -- 172.21.15.6:0/2036714246 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feeec0af1a0 msgr2=0x7feeec14a4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:54.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.934+0000 7feef2400640 1 --2- 172.21.15.6:0/2036714246 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feeec0af1a0 0x7feeec14a4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:54.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.934+0000 7feef2400640 1 -- 172.21.15.6:0/2036714246 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feeec0af580 msgr2=0x7feeec1f14c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:54.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.934+0000 7feef2400640 1 --2- 172.21.15.6:0/2036714246 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feeec0af580 0x7feeec1f14c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:54.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.934+0000 7feef2400640 1 -- 172.21.15.6:0/2036714246 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feeec146fb0 con 0x7feeec0cdc20 2024-03-04T09:29:54.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.935+0000 7feef2400640 1 --2- 172.21.15.6:0/2036714246 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feeec0cdc20 0x7feeec14cc20 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7feee400d5d0 tx=0x7feee400da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:54.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.935+0000 7feee9600640 1 -- 172.21.15.6:0/2036714246 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feee40125f0 con 0x7feeec0cdc20 2024-03-04T09:29:54.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.935+0000 7feee9600640 1 -- 172.21.15.6:0/2036714246 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feee4019070 con 0x7feeec0cdc20 2024-03-04T09:29:54.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.935+0000 7feee9600640 1 -- 172.21.15.6:0/2036714246 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feee40129f0 con 0x7feeec0cdc20 2024-03-04T09:29:54.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.935+0000 7feef4400640 1 -- 172.21.15.6:0/2036714246 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feeec147290 con 0x7feeec0cdc20 2024-03-04T09:29:54.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.935+0000 7feef4400640 1 -- 172.21.15.6:0/2036714246 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feeec147850 con 0x7feeec0cdc20 2024-03-04T09:29:54.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.937+0000 7feef4400640 1 -- 172.21.15.6:0/2036714246 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feeec04f3a0 con 0x7feeec0cdc20 2024-03-04T09:29:54.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.937+0000 7feee9600640 1 -- 172.21.15.6:0/2036714246 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7feee4012b90 con 0x7feeec0cdc20 2024-03-04T09:29:54.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.939+0000 7feee9600640 1 --2- 172.21.15.6:0/2036714246 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7feecc06b4d0 0x7feecc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:54.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.939+0000 7feee9600640 1 -- 172.21.15.6:0/2036714246 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5709+0+0 (secure 0 0 0) 0x7feee408a6b0 con 0x7feeec0cdc20 2024-03-04T09:29:54.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.940+0000 7feef1a00640 1 --2- 172.21.15.6:0/2036714246 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7feecc06b4d0 0x7feecc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:54.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.941+0000 7feef1a00640 1 --2- 172.21.15.6:0/2036714246 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7feecc06b4d0 0x7feecc06d990 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7feee00012e0 tx=0x7feee0009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:54.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:54.944+0000 7feee9600640 1 -- 172.21.15.6:0/2036714246 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7feee40594a0 con 0x7feeec0cdc20 2024-03-04T09:29:55.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:55.191+0000 7feef4400640 1 -- 172.21.15.6:0/2036714246 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7feeec06b920 con 0x7feeec0cdc20 2024-03-04T09:29:55.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:55.191+0000 7feee9600640 1 -- 172.21.15.6:0/2036714246 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v51) v1 ==== 53+0+73 (secure 0 0 0) 0x7feee4058da0 con 0x7feeec0cdc20 2024-03-04T09:29:55.188 INFO:teuthology.orchestra.run.smithi006.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-04T09:29:55.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:55.194+0000 7feef4400640 1 -- 172.21.15.6:0/2036714246 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7feecc06b4d0 msgr2=0x7feecc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:55.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:55.194+0000 7feef4400640 1 --2- 172.21.15.6:0/2036714246 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7feecc06b4d0 0x7feecc06d990 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7feee00012e0 tx=0x7feee0009040 comp rx=0 tx=0).stop 2024-03-04T09:29:55.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:55.194+0000 7feef4400640 1 -- 172.21.15.6:0/2036714246 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feeec0cdc20 msgr2=0x7feeec14cc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:55.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:55.194+0000 7feef4400640 1 --2- 172.21.15.6:0/2036714246 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feeec0cdc20 0x7feeec14cc20 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7feee400d5d0 tx=0x7feee400da90 comp rx=0 tx=0).stop 2024-03-04T09:29:55.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:55.194+0000 7feef4400640 1 -- 172.21.15.6:0/2036714246 shutdown_connections 2024-03-04T09:29:55.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:55.194+0000 7feef4400640 1 --2- 172.21.15.6:0/2036714246 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feeec0cdc20 0x7feeec14cc20 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:55.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:55.194+0000 7feef4400640 1 --2- 172.21.15.6:0/2036714246 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feeec0af580 0x7feeec1f14c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:55.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:55.194+0000 7feef4400640 1 --2- 172.21.15.6:0/2036714246 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7feecc06b4d0 0x7feecc06d990 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:55.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:55.194+0000 7feef4400640 1 --2- 172.21.15.6:0/2036714246 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feeec0af1a0 0x7feeec14a4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:55.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:55.194+0000 7feef4400640 1 -- 172.21.15.6:0/2036714246 >> 172.21.15.6:0/2036714246 conn(0x7feeec06afb0 msgr2=0x7feeec0ad470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:29:55.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:55.194+0000 7feef4400640 1 -- 172.21.15.6:0/2036714246 shutdown_connections 2024-03-04T09:29:55.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:55.194+0000 7feef4400640 1 -- 172.21.15.6:0/2036714246 wait complete. 2024-03-04T09:29:55.211 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-04T09:29:55.212 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-04T09:29:55.212 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi006.front.sepia.ceph.com 2024-03-04T09:29:55.212 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-04T09:29:55.212 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-04T09:29:55.212 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-04T09:29:55.213 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-03-04T09:29:55.213 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-04T09:29:55.213 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-04T09:29:55.213 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:29:55.214 DEBUG:teuthology.orchestra.run.smithi006:> ip addr 2024-03-04T09:29:55.267 INFO:teuthology.orchestra.run.smithi006.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-04T09:29:55.268 INFO:teuthology.orchestra.run.smithi006.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-04T09:29:55.268 INFO:teuthology.orchestra.run.smithi006.stdout: inet 127.0.0.1/8 scope host lo 2024-03-04T09:29:55.269 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-04T09:29:55.269 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 ::1/128 scope host 2024-03-04T09:29:55.270 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-04T09:29:55.270 INFO:teuthology.orchestra.run.smithi006.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-04T09:29:55.270 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:6b:fe:e6 brd ff:ff:ff:ff:ff:ff 2024-03-04T09:29:55.271 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp7s0f0 2024-03-04T09:29:55.271 INFO:teuthology.orchestra.run.smithi006.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-04T09:29:55.271 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:6b:fe:e7 brd ff:ff:ff:ff:ff:ff 2024-03-04T09:29:55.271 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp7s0f1 2024-03-04T09:29:55.271 INFO:teuthology.orchestra.run.smithi006.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-04T09:29:55.271 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:bd:15:e4 brd ff:ff:ff:ff:ff:ff 2024-03-04T09:29:55.272 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp1s0f0 2024-03-04T09:29:55.272 INFO:teuthology.orchestra.run.smithi006.stdout: inet 172.21.15.6/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-04T09:29:55.272 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft 42339sec preferred_lft 42339sec 2024-03-04T09:29:55.272 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::ec4:7aff:febd:15e4/64 scope link noprefixroute 2024-03-04T09:29:55.273 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-04T09:29:55.273 INFO:teuthology.orchestra.run.smithi006.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-04T09:29:55.273 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:bd:15:e5 brd ff:ff:ff:ff:ff:ff 2024-03-04T09:29:55.273 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp1s0f1 2024-03-04T09:29:55.273 INFO:teuthology.orchestra.run.smithi006.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-04T09:29:55.273 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 5e:b4:59:dc:42:58 brd ff:ff:ff:ff:ff:ff 2024-03-04T09:29:55.273 INFO:teuthology.orchestra.run.smithi006.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-04T09:29:55.273 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-04T09:29:55.274 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::c6f:adff:fefe:e00d/64 scope link proto kernel_ll 2024-03-04T09:29:55.274 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-04T09:29:55.274 INFO:teuthology.orchestra.run.smithi006.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-04T09:29:55.274 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 7e:67:6d:ea:f9:ff brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-04T09:29:55.274 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::7c67:6dff:feea:f9ff/64 scope link proto kernel_ll 2024-03-04T09:29:55.275 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-04T09:29:55.275 INFO:teuthology.orchestra.run.smithi006.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-04T09:29:55.275 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 5e:b4:59:dc:42:58 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-04T09:29:55.275 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::5cb4:59ff:fedc:4258/64 scope link proto kernel_ll 2024-03-04T09:29:55.275 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-04T09:29:55.276 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:29:55.276 DEBUG:teuthology.orchestra.run.smithi006:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-04T09:29:55.356 INFO:teuthology.orchestra.run.smithi006.stdout:1 2024-03-04T09:29:55.361 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:29:55.361 DEBUG:teuthology.orchestra.run.smithi006:> route 2024-03-04T09:29:55.413 INFO:teuthology.orchestra.run.smithi006.stdout:Kernel IP routing table 2024-03-04T09:29:55.414 INFO:teuthology.orchestra.run.smithi006.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-04T09:29:55.414 INFO:teuthology.orchestra.run.smithi006.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-04T09:29:55.414 INFO:teuthology.orchestra.run.smithi006.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-04T09:29:55.414 INFO:teuthology.orchestra.run.smithi006.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-04T09:29:55.414 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:29:55.414 DEBUG:teuthology.orchestra.run.smithi006:> (cd / && exec bash -c ' 2024-03-04T09:29:55.415 DEBUG:teuthology.orchestra.run.smithi006:> set -e 2024-03-04T09:29:55.415 DEBUG:teuthology.orchestra.run.smithi006:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-04T09:29:55.415 DEBUG:teuthology.orchestra.run.smithi006:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-04T09:29:55.415 DEBUG:teuthology.orchestra.run.smithi006:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-04T09:29:55.415 DEBUG:teuthology.orchestra.run.smithi006:> ') 2024-03-04T09:29:55.614 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:29:55.615 DEBUG:teuthology.orchestra.run.smithi006:> ip netns list 2024-03-04T09:29:55.666 INFO:teuthology.orchestra.run.smithi006.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-04T09:29:55.666 INFO:teuthology.orchestra.run.smithi006.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-04T09:29:55.667 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-04T09:29:55.667 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:29:55.667 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip link set brx.0 up 2024-03-04T09:29:55.777 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-04T09:29:55.777 DEBUG:teuthology.orchestra.run.smithi006:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-04T09:29:55.828 INFO:teuthology.orchestra.run.smithi006.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-04T09:29:55.828 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-04T09:29:55.829 DEBUG:teuthology.orchestra.run.smithi006:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-04T09:29:55.878 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-03-04T09:29:55.878 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:29:55.878 DEBUG:teuthology.orchestra.run.smithi006:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-04T09:29:56.016 INFO:teuthology.orchestra.run.smithi006.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-04T09:29:56.016 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-04T09:29:56.049 INFO:teuthology.orchestra.run.smithi006.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-04T09:29:56.049 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-04T09:29:56.049 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: resolved to: "172.21.15.6:3300,172.21.15.146:3300,172.21.15.146:3301" 2024-03-04T09:29:56.049 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: trying mount with new device syntax: 0@edce0578-5ae4-4402-b45a-552bdef84d7c.cephfs=/ 2024-03-04T09:29:56.049 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.6:3300/172.21.15.146:3300/172.21.15.146:3301" will pass to kernel 2024-03-04T09:29:56.051 INFO:teuthology.orchestra.run.smithi006.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-04T09:29:56.051 INFO:teuthology.orchestra.run.smithi006.stdout: You just mounted a file system that supports labels which does not 2024-03-04T09:29:56.051 INFO:teuthology.orchestra.run.smithi006.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-04T09:29:56.051 INFO:teuthology.orchestra.run.smithi006.stdout: applications will generate AVC messages and not be allowed access to 2024-03-04T09:29:56.052 INFO:teuthology.orchestra.run.smithi006.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-04T09:29:56.055 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-04T09:29:56.055 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:29:56.055 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-04T09:29:56.133 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:29:56.133 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-04T09:29:56.194 INFO:teuthology.orchestra.run.smithi006.stdout:client5468 2024-03-04T09:29:56.195 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-04T09:29:56.195 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T09:29:56.245 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:29:56.245 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest/mnt.0 && exec touch fancy_layout_file) 2024-03-04T09:29:56.348 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:29:56.349 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.file.layout -v 'stripe_unit=1048576 stripe_count=4 object_size=8388608' fancy_layout_file) 2024-03-04T09:29:56.407 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-04T09:29:56.407 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T09:29:56.456 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:29:56.456 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=fancy_layout_file bs=1M conv=fdatasync count=35 seek=0) 2024-03-04T09:29:56.832 INFO:teuthology.orchestra.run.smithi006.stderr:35+0 records in 2024-03-04T09:29:56.832 INFO:teuthology.orchestra.run.smithi006.stderr:35+0 records out 2024-03-04T09:29:56.832 INFO:teuthology.orchestra.run.smithi006.stderr:36700160 bytes (37 MB, 35 MiB) copied, 0.325218 s, 113 MB/s 2024-03-04T09:29:56.833 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:29:56.833 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest/mnt.0 && exec rm -f fancy_layout_file) 2024-03-04T09:29:56.874 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:29:57.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.117+0000 7faee9a00640 1 Processor -- start 2024-03-04T09:29:57.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.117+0000 7faee9a00640 1 -- start start 2024-03-04T09:29:57.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.119+0000 7faee9a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7faee4162420 0x7faee406afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:57.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.119+0000 7faee9a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faee406b4f0 0x7faee40d0910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:57.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.119+0000 7faee9a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faee4147bc0 con 0x7faee41477e0 2024-03-04T09:29:57.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.119+0000 7faee9a00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faee406b960 con 0x7faee4162420 2024-03-04T09:29:57.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.119+0000 7faee9a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faee4161f90 con 0x7faee406b4f0 2024-03-04T09:29:57.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.120+0000 7faee3400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7faee4162420 0x7faee406afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:57.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.120+0000 7faee3400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7faee4162420 0x7faee406afb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38840/0 (socket says 172.21.15.6:38840) 2024-03-04T09:29:57.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.120+0000 7faee3400640 1 -- 172.21.15.6:0/1441726830 learned_addr learned my addr 172.21.15.6:0/1441726830 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:29:57.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.120+0000 7faee3400640 1 -- 172.21.15.6:0/1441726830 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faee406b4f0 msgr2=0x7faee40d0910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:57.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.120+0000 7faee3400640 1 --2- 172.21.15.6:0/1441726830 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faee406b4f0 0x7faee40d0910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:57.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.120+0000 7faee3400640 1 -- 172.21.15.6:0/1441726830 >> v1:172.21.15.6:6789/0 conn(0x7faee41477e0 legacy=0x7faee406ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:57.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.120+0000 7faee3400640 1 -- 172.21.15.6:0/1441726830 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faee40b3d70 con 0x7faee4162420 2024-03-04T09:29:57.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.121+0000 7faee3400640 1 --2- 172.21.15.6:0/1441726830 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faee4162420 0x7faee406afb0 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7faed0009870 tx=0x7faed002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7a185cd1c2c53fd server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:57.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.122+0000 7faee2a00640 1 -- 172.21.15.6:0/1441726830 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faed0035020 con 0x7faee4162420 2024-03-04T09:29:57.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.122+0000 7faee2a00640 1 -- 172.21.15.6:0/1441726830 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faed0031640 con 0x7faee4162420 2024-03-04T09:29:57.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.122+0000 7faee2a00640 1 -- 172.21.15.6:0/1441726830 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faed002d040 con 0x7faee4162420 2024-03-04T09:29:57.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.122+0000 7faee9a00640 1 -- 172.21.15.6:0/1441726830 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faee4162420 msgr2=0x7faee406afb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:57.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.122+0000 7faee9a00640 1 --2- 172.21.15.6:0/1441726830 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faee4162420 0x7faee406afb0 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7faed0009870 tx=0x7faed002b8f0 comp rx=0 tx=0).stop 2024-03-04T09:29:57.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.123+0000 7faee9a00640 1 -- 172.21.15.6:0/1441726830 shutdown_connections 2024-03-04T09:29:57.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.123+0000 7faee9a00640 1 --2- 172.21.15.6:0/1441726830 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faee406b4f0 0x7faee40d0910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:57.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.123+0000 7faee9a00640 1 --2- 172.21.15.6:0/1441726830 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faee4162420 0x7faee406afb0 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:57.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.123+0000 7faee9a00640 1 -- 172.21.15.6:0/1441726830 >> 172.21.15.6:0/1441726830 conn(0x7faee406c0c0 msgr2=0x7faee406c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:29:57.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.123+0000 7faee9a00640 1 -- 172.21.15.6:0/1441726830 shutdown_connections 2024-03-04T09:29:57.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.123+0000 7faee9a00640 1 -- 172.21.15.6:0/1441726830 wait complete. 2024-03-04T09:29:57.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.125+0000 7faee9a00640 1 Processor -- start 2024-03-04T09:29:57.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.126+0000 7faee9a00640 1 -- start start 2024-03-04T09:29:57.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.127+0000 7faee9a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faee406b4f0 0x7faee413efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:57.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.127+0000 7faee9a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faee41477e0 0x7faee415f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:57.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.127+0000 7faee9a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faee4162420 0x7faee415faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:57.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.127+0000 7faee9a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faee4161c20 con 0x7faee41477e0 2024-03-04T09:29:57.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.127+0000 7faee9a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faee40b4330 con 0x7faee4162420 2024-03-04T09:29:57.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.127+0000 7faee9a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faee41628a0 con 0x7faee406b4f0 2024-03-04T09:29:57.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.128+0000 7faee3400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faee41477e0 0x7faee415f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:57.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.128+0000 7faee3400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faee41477e0 0x7faee415f5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38854/0 (socket says 172.21.15.6:38854) 2024-03-04T09:29:57.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.128+0000 7faee3400640 1 -- 172.21.15.6:0/833574242 learned_addr learned my addr 172.21.15.6:0/833574242 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:29:57.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.128+0000 7faee3e00640 1 --2- 172.21.15.6:0/833574242 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faee406b4f0 0x7faee413efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:57.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.128+0000 7faee3400640 1 -- 172.21.15.6:0/833574242 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faee406b4f0 msgr2=0x7faee413efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:57.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.128+0000 7faee3400640 1 --2- 172.21.15.6:0/833574242 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faee406b4f0 0x7faee413efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:57.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.128+0000 7faee9000640 1 --2- 172.21.15.6:0/833574242 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faee4162420 0x7faee415faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:57.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.128+0000 7faee3400640 1 -- 172.21.15.6:0/833574242 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faee4162420 msgr2=0x7faee415faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:57.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.128+0000 7faee3400640 1 --2- 172.21.15.6:0/833574242 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faee4162420 0x7faee415faf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:57.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.128+0000 7faee3400640 1 -- 172.21.15.6:0/833574242 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faee4150050 con 0x7faee41477e0 2024-03-04T09:29:57.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.128+0000 7faee3e00640 1 --2- 172.21.15.6:0/833574242 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faee406b4f0 0x7faee413efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:29:57.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.128+0000 7faee9000640 1 --2- 172.21.15.6:0/833574242 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faee4162420 0x7faee415faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:29:57.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.128+0000 7faee3400640 1 --2- 172.21.15.6:0/833574242 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faee41477e0 0x7faee415f5b0 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7faed002b930 tx=0x7faed00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:57.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.129+0000 7faedbe00640 1 -- 172.21.15.6:0/833574242 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faed0040070 con 0x7faee41477e0 2024-03-04T09:29:57.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.129+0000 7faedbe00640 1 -- 172.21.15.6:0/833574242 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faed0039bf0 con 0x7faee41477e0 2024-03-04T09:29:57.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.129+0000 7faee9a00640 1 -- 172.21.15.6:0/833574242 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faee41505e0 con 0x7faee41477e0 2024-03-04T09:29:57.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.129+0000 7faedbe00640 1 -- 172.21.15.6:0/833574242 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faed0035040 con 0x7faee41477e0 2024-03-04T09:29:57.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.129+0000 7faee9a00640 1 -- 172.21.15.6:0/833574242 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faee4150ba0 con 0x7faee41477e0 2024-03-04T09:29:57.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.130+0000 7faee9a00640 1 -- 172.21.15.6:0/833574242 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faee404f330 con 0x7faee41477e0 2024-03-04T09:29:57.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.131+0000 7faedbe00640 1 -- 172.21.15.6:0/833574242 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7faed004b060 con 0x7faee41477e0 2024-03-04T09:29:57.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.132+0000 7faedbe00640 1 --2- 172.21.15.6:0/833574242 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7faec406b4d0 0x7faec406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:57.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.132+0000 7faedbe00640 1 -- 172.21.15.6:0/833574242 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5709+0+0 (secure 0 0 0) 0x7faed00abd50 con 0x7faee41477e0 2024-03-04T09:29:57.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.133+0000 7faee3e00640 1 --2- 172.21.15.6:0/833574242 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7faec406b4d0 0x7faec406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:57.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.134+0000 7faee3e00640 1 --2- 172.21.15.6:0/833574242 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7faec406b4d0 0x7faec406d990 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7faed4001680 tx=0x7faed4009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:57.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.137+0000 7faedbe00640 1 -- 172.21.15.6:0/833574242 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7faed00af030 con 0x7faee41477e0 2024-03-04T09:29:57.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.378+0000 7faee9a00640 1 -- 172.21.15.6:0/833574242 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faee40724b0 con 0x7faee41477e0 2024-03-04T09:29:57.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.379+0000 7faedbe00640 1 -- 172.21.15.6:0/833574242 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) v1 ==== 94+0+4976 (secure 0 0 0) 0x7faed0031610 con 0x7faee41477e0 2024-03-04T09:29:57.376 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:29:57.377 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":51,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5827,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6836/3447344578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":3447344578},{"type":"v1","addr":"172.21.15.146:6836","nonce":3447344578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":46},{"gid":5830,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/168994142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":168994142},{"type":"v1","addr":"172.21.15.146:6837","nonce":168994142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":46},{"gid":6735,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/3369489999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":3369489999},{"type":"v1","addr":"172.21.15.6:6833","nonce":3369489999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":45}],"filesystems":[{"mdsmap":{"epoch":51,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:29:47.357677+0000","modified":"2024-03-04T09:29:50.579389+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6744},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6744":{"gid":6744,"name":"a","rank":0,"incarnation":47,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6835/1667002065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1667002065},{"type":"v1","addr":"172.21.15.6:6835","nonce":1667002065}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6744,"qdb_cluster":[6744]},"id":6}]} 2024-03-04T09:29:57.377 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 51 2024-03-04T09:29:57.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.381+0000 7faee9a00640 1 -- 172.21.15.6:0/833574242 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7faec406b4d0 msgr2=0x7faec406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:57.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.381+0000 7faee9a00640 1 --2- 172.21.15.6:0/833574242 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7faec406b4d0 0x7faec406d990 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7faed4001680 tx=0x7faed4009040 comp rx=0 tx=0).stop 2024-03-04T09:29:57.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.381+0000 7faee9a00640 1 -- 172.21.15.6:0/833574242 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faee41477e0 msgr2=0x7faee415f5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:57.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.381+0000 7faee9a00640 1 --2- 172.21.15.6:0/833574242 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faee41477e0 0x7faee415f5b0 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7faed002b930 tx=0x7faed00046c0 comp rx=0 tx=0).stop 2024-03-04T09:29:57.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.382+0000 7faee9a00640 1 -- 172.21.15.6:0/833574242 shutdown_connections 2024-03-04T09:29:57.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.382+0000 7faee9a00640 1 --2- 172.21.15.6:0/833574242 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faee4162420 0x7faee415faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:57.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.382+0000 7faee9a00640 1 --2- 172.21.15.6:0/833574242 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faee41477e0 0x7faee415f5b0 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:57.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.382+0000 7faee9a00640 1 --2- 172.21.15.6:0/833574242 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7faec406b4d0 0x7faec406d990 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:57.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.382+0000 7faee9a00640 1 --2- 172.21.15.6:0/833574242 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faee406b4f0 0x7faee413efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:57.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.382+0000 7faee9a00640 1 -- 172.21.15.6:0/833574242 >> 172.21.15.6:0/833574242 conn(0x7faee406c0c0 msgr2=0x7faee40b0750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:29:57.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.382+0000 7faee9a00640 1 -- 172.21.15.6:0/833574242 shutdown_connections 2024-03-04T09:29:57.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.382+0000 7faee9a00640 1 -- 172.21.15.6:0/833574242 wait complete. 2024-03-04T09:29:57.399 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json flush journal 2024-03-04T09:29:57.646 INFO:teuthology.orchestra.run.smithi006.stdout:{"message":"","return_code":0} 2024-03-04T09:29:57.647 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-04T09:29:57.647 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph df --format=json-pretty 2024-03-04T09:29:57.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.853+0000 7f3ccf400640 1 Processor -- start 2024-03-04T09:29:57.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.853+0000 7f3ccf400640 1 -- start start 2024-03-04T09:29:57.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.855+0000 7f3ccf400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f3cc80ce1b0 0x7f3cc80afa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:57.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.855+0000 7f3ccf400640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cc80cd4a0 con 0x7f3cc815bd50 2024-03-04T09:29:57.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.855+0000 7f3ccf400640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cc806e420 con 0x7f3cc80ccf00 2024-03-04T09:29:57.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.855+0000 7f3ccf400640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3cc80affa0 con 0x7f3cc80ce1b0 2024-03-04T09:29:57.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.855+0000 7f3ccca00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f3cc815bd50 0x7f3cc80cdaa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:42006/0 (socket says 172.21.15.6:42006) 2024-03-04T09:29:57.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.856+0000 7f3ccca00640 1 -- 172.21.15.6:0/758502750 learned_addr learned my addr 172.21.15.6:0/758502750 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:29:57.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.856+0000 7f3cc7400640 1 -- 172.21.15.6:0/758502750 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1617113032 0 0) 0x7f3cc80cd4a0 con 0x7f3cc815bd50 2024-03-04T09:29:57.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.856+0000 7f3cc7e00640 1 --2- 172.21.15.6:0/758502750 >> v2:172.21.15.146:3300/0 conn(0x7f3cc80ce1b0 0x7f3cc80afa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:57.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.856+0000 7f3cc7400640 1 -- 172.21.15.6:0/758502750 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3cb0003520 con 0x7f3cc815bd50 2024-03-04T09:29:57.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.856+0000 7f3cc7400640 1 -- 172.21.15.6:0/758502750 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3420462838 0 0) 0x7f3cb0003520 con 0x7f3cc815bd50 2024-03-04T09:29:57.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.856+0000 7f3cc7400640 1 -- 172.21.15.6:0/758502750 >> v2:172.21.15.146:3300/0 conn(0x7f3cc80ce1b0 msgr2=0x7f3cc80afa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:57.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.856+0000 7f3cc7400640 1 --2- 172.21.15.6:0/758502750 >> v2:172.21.15.146:3300/0 conn(0x7f3cc80ce1b0 0x7f3cc80afa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:57.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.856+0000 7f3cc7400640 1 -- 172.21.15.6:0/758502750 >> v1:172.21.15.146:6789/0 conn(0x7f3cc80ccf00 legacy=0x7f3cc806b290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:57.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.856+0000 7f3cc7400640 1 -- 172.21.15.6:0/758502750 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cc806e700 con 0x7f3cc815bd50 2024-03-04T09:29:57.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.856+0000 7f3cc7400640 1 -- 172.21.15.6:0/758502750 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7f3cbc001670 con 0x7f3cc815bd50 2024-03-04T09:29:57.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.857+0000 7f3cc7400640 1 -- 172.21.15.6:0/758502750 >> v1:172.21.15.6:6789/0 conn(0x7f3cc815bd50 legacy=0x7f3cc80cdaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:57.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.857+0000 7f3cc7400640 1 --2- 172.21.15.6:0/758502750 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3cb0003a20 0x7f3cb0003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:57.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.857+0000 7f3cc7400640 1 --2- 172.21.15.6:0/758502750 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3cb0004800 0x7f3cb0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:57.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.857+0000 7f3cc7400640 1 --2- 172.21.15.6:0/758502750 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3cb00052e0 0x7f3cb000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:57.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.857+0000 7f3cc7400640 1 -- 172.21.15.6:0/758502750 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cc80affa0 con 0x7f3cb0004800 2024-03-04T09:29:57.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.857+0000 7f3cc7400640 1 -- 172.21.15.6:0/758502750 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cb000c230 con 0x7f3cb0003a20 2024-03-04T09:29:57.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.857+0000 7f3cc7400640 1 -- 172.21.15.6:0/758502750 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cb000c430 con 0x7f3cb00052e0 2024-03-04T09:29:57.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.857+0000 7f3cc7e00640 1 --2- 172.21.15.6:0/758502750 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3cb0004800 0x7f3cb0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:57.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.857+0000 7f3ccca00640 1 --2- 172.21.15.6:0/758502750 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3cb0003a20 0x7f3cb0003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:57.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.857+0000 7f3ccd400640 1 --2- 172.21.15.6:0/758502750 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3cb00052e0 0x7f3cb000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:57.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.858+0000 7f3cc7e00640 1 -- 172.21.15.6:0/758502750 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3cb00052e0 msgr2=0x7f3cb000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:57.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.858+0000 7f3cc7e00640 1 --2- 172.21.15.6:0/758502750 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3cb00052e0 0x7f3cb000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:57.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.858+0000 7f3cc7e00640 1 -- 172.21.15.6:0/758502750 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3cb0003a20 msgr2=0x7f3cb0003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:57.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.858+0000 7f3cc7e00640 1 --2- 172.21.15.6:0/758502750 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3cb0003a20 0x7f3cb0003e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:57.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.858+0000 7f3cc7e00640 1 -- 172.21.15.6:0/758502750 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cb0003520 con 0x7f3cb0004800 2024-03-04T09:29:57.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.858+0000 7f3ccd400640 1 --2- 172.21.15.6:0/758502750 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3cb00052e0 0x7f3cb000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:29:57.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.858+0000 7f3ccd400640 1 -- 172.21.15.6:0/758502750 reap_dead start 2024-03-04T09:29:57.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.859+0000 7f3cc7e00640 1 --2- 172.21.15.6:0/758502750 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3cb0004800 0x7f3cb0004c80 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f3cb400ba00 tx=0x7f3cb0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32bcfd536043abfb server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:57.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.860+0000 7f3cc7400640 1 -- 172.21.15.6:0/758502750 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cb400b600 con 0x7f3cb0004800 2024-03-04T09:29:57.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.860+0000 7f3cc7400640 1 -- 172.21.15.6:0/758502750 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cb4038070 con 0x7f3cb0004800 2024-03-04T09:29:57.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.860+0000 7f3cc7400640 1 -- 172.21.15.6:0/758502750 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cb40338a0 con 0x7f3cb0004800 2024-03-04T09:29:57.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.860+0000 7f3ccf400640 1 -- 172.21.15.6:0/758502750 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3cb0004800 msgr2=0x7f3cb0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:57.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.860+0000 7f3ccf400640 1 --2- 172.21.15.6:0/758502750 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3cb0004800 0x7f3cb0004c80 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f3cb400ba00 tx=0x7f3cb0005830 comp rx=0 tx=0).stop 2024-03-04T09:29:57.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.861+0000 7f3ccf400640 1 -- 172.21.15.6:0/758502750 shutdown_connections 2024-03-04T09:29:57.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.861+0000 7f3ccf400640 1 --2- 172.21.15.6:0/758502750 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3cb0004800 0x7f3cb0004c80 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:57.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.861+0000 7f3ccf400640 1 -- 172.21.15.6:0/758502750 >> 172.21.15.6:0/758502750 conn(0x7f3cc80b2640 msgr2=0x7f3cc80b2a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:29:57.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.861+0000 7f3ccf400640 1 -- 172.21.15.6:0/758502750 shutdown_connections 2024-03-04T09:29:57.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.861+0000 7f3ccf400640 1 -- 172.21.15.6:0/758502750 wait complete. 2024-03-04T09:29:57.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.862+0000 7f3ccf400640 1 Processor -- start 2024-03-04T09:29:57.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.863+0000 7f3ccf400640 1 -- start start 2024-03-04T09:29:57.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.864+0000 7f3ccf400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3cb0004800 0x7f3cc814ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:57.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.864+0000 7f3ccf400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3cc815a4c0 0x7f3cc8156260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:57.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.864+0000 7f3ccf400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3cc81f3a40 0x7f3cc815e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:57.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.864+0000 7f3ccf400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cc815c130 con 0x7f3cc81f3a40 2024-03-04T09:29:57.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.864+0000 7f3ccf400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cc806e1d0 con 0x7f3cc815a4c0 2024-03-04T09:29:57.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.864+0000 7f3ccf400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cc815c2b0 con 0x7f3cb0004800 2024-03-04T09:29:57.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.864+0000 7f3ccd400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3cc81f3a40 0x7f3cc815e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:57.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.864+0000 7f3ccd400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3cc81f3a40 0x7f3cc815e5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38866/0 (socket says 172.21.15.6:38866) 2024-03-04T09:29:57.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.864+0000 7f3ccd400640 1 -- 172.21.15.6:0/619244303 learned_addr learned my addr 172.21.15.6:0/619244303 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:29:57.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.864+0000 7f3cc7e00640 1 --2- 172.21.15.6:0/619244303 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3cc815a4c0 0x7f3cc8156260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:57.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.864+0000 7f3ccca00640 1 --2- 172.21.15.6:0/619244303 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3cb0004800 0x7f3cc814ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:57.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.865+0000 7f3ccd400640 1 -- 172.21.15.6:0/619244303 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3cb0004800 msgr2=0x7f3cc814ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:57.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.865+0000 7f3ccd400640 1 --2- 172.21.15.6:0/619244303 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3cb0004800 0x7f3cc814ccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:57.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.865+0000 7f3ccd400640 1 -- 172.21.15.6:0/619244303 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3cc815a4c0 msgr2=0x7f3cc8156260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:57.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.865+0000 7f3ccd400640 1 --2- 172.21.15.6:0/619244303 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3cc815a4c0 0x7f3cc8156260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:57.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.865+0000 7f3ccd400640 1 -- 172.21.15.6:0/619244303 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cc815d520 con 0x7f3cc81f3a40 2024-03-04T09:29:57.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.865+0000 7f3ccd400640 1 --2- 172.21.15.6:0/619244303 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3cc81f3a40 0x7f3cc815e5c0 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f3cc806cb80 tx=0x7f3cc000dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:57.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.865+0000 7f3cc4c00640 1 -- 172.21.15.6:0/619244303 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cc0018070 con 0x7f3cc81f3a40 2024-03-04T09:29:57.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.865+0000 7f3cc4c00640 1 -- 172.21.15.6:0/619244303 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cc0003b70 con 0x7f3cc81f3a40 2024-03-04T09:29:57.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.865+0000 7f3cc4c00640 1 -- 172.21.15.6:0/619244303 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cc0005070 con 0x7f3cc81f3a40 2024-03-04T09:29:57.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.865+0000 7f3ccf400640 1 -- 172.21.15.6:0/619244303 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cc815d800 con 0x7f3cc81f3a40 2024-03-04T09:29:57.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.865+0000 7f3ccf400640 1 -- 172.21.15.6:0/619244303 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cc815ddc0 con 0x7f3cc81f3a40 2024-03-04T09:29:57.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.867+0000 7f3ccf400640 1 -- 172.21.15.6:0/619244303 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cc804f3a0 con 0x7f3cc81f3a40 2024-03-04T09:29:57.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.867+0000 7f3cc4c00640 1 -- 172.21.15.6:0/619244303 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3cc001b070 con 0x7f3cc81f3a40 2024-03-04T09:29:57.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.868+0000 7f3cc4c00640 1 --2- 172.21.15.6:0/619244303 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f3ca806b500 0x7f3ca806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:57.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.868+0000 7f3cc4c00640 1 -- 172.21.15.6:0/619244303 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3cc00899a0 con 0x7f3cc81f3a40 2024-03-04T09:29:57.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.869+0000 7f3ccca00640 1 --2- 172.21.15.6:0/619244303 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f3ca806b500 0x7f3ca806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:57.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.870+0000 7f3ccca00640 1 --2- 172.21.15.6:0/619244303 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f3ca806b500 0x7f3ca806d9c0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f3cbc007c10 tx=0x7f3cbc009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:57.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:57.873+0000 7f3cc4c00640 1 -- 172.21.15.6:0/619244303 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3cc0058780 con 0x7f3cc81f3a40 2024-03-04T09:29:58.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:58.380+0000 7f3ccf400640 1 -- 172.21.15.6:0/619244303 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "df", "format": "json-pretty"} v 0) v1 -- 0x7f3cc8067800 con 0x7f3cc81f3a40 2024-03-04T09:29:58.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:58.381+0000 7f3cc4c00640 1 -- 172.21.15.6:0/619244303 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "df", "format": "json-pretty"}]=0 v0) v1 ==== 75+0+1654 (secure 0 0 0) 0x7f3cc0058080 con 0x7f3cc81f3a40 2024-03-04T09:29:58.378 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:29:58.378 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-04T09:29:58.378 INFO:teuthology.orchestra.run.smithi006.stdout: "stats": { 2024-03-04T09:29:58.378 INFO:teuthology.orchestra.run.smithi006.stdout: "total_bytes": 773094113280, 2024-03-04T09:29:58.378 INFO:teuthology.orchestra.run.smithi006.stdout: "total_avail_bytes": 772829466624, 2024-03-04T09:29:58.378 INFO:teuthology.orchestra.run.smithi006.stdout: "total_used_bytes": 264646656, 2024-03-04T09:29:58.378 INFO:teuthology.orchestra.run.smithi006.stdout: "total_used_raw_bytes": 264646656, 2024-03-04T09:29:58.379 INFO:teuthology.orchestra.run.smithi006.stdout: "total_used_raw_ratio": 0.00034232140751555562, 2024-03-04T09:29:58.379 INFO:teuthology.orchestra.run.smithi006.stdout: "num_osds": 8, 2024-03-04T09:29:58.379 INFO:teuthology.orchestra.run.smithi006.stdout: "num_per_pool_osds": 8, 2024-03-04T09:29:58.379 INFO:teuthology.orchestra.run.smithi006.stdout: "num_per_pool_omap_osds": 8 2024-03-04T09:29:58.379 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:29:58.379 INFO:teuthology.orchestra.run.smithi006.stdout: "stats_by_class": { 2024-03-04T09:29:58.379 INFO:teuthology.orchestra.run.smithi006.stdout: "ssd": { 2024-03-04T09:29:58.379 INFO:teuthology.orchestra.run.smithi006.stdout: "total_bytes": 773094113280, 2024-03-04T09:29:58.379 INFO:teuthology.orchestra.run.smithi006.stdout: "total_avail_bytes": 772829466624, 2024-03-04T09:29:58.379 INFO:teuthology.orchestra.run.smithi006.stdout: "total_used_bytes": 264646656, 2024-03-04T09:29:58.379 INFO:teuthology.orchestra.run.smithi006.stdout: "total_used_raw_bytes": 264646656, 2024-03-04T09:29:58.379 INFO:teuthology.orchestra.run.smithi006.stdout: "total_used_raw_ratio": 0.00034232140751555562 2024-03-04T09:29:58.379 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:29:58.379 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:29:58.379 INFO:teuthology.orchestra.run.smithi006.stdout: "pools": [ 2024-03-04T09:29:58.380 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:29:58.380 INFO:teuthology.orchestra.run.smithi006.stdout: "name": ".mgr", 2024-03-04T09:29:58.380 INFO:teuthology.orchestra.run.smithi006.stdout: "id": 1, 2024-03-04T09:29:58.380 INFO:teuthology.orchestra.run.smithi006.stdout: "stats": { 2024-03-04T09:29:58.380 INFO:teuthology.orchestra.run.smithi006.stdout: "stored": 590368, 2024-03-04T09:29:58.380 INFO:teuthology.orchestra.run.smithi006.stdout: "objects": 2, 2024-03-04T09:29:58.380 INFO:teuthology.orchestra.run.smithi006.stdout: "kb_used": 1160, 2024-03-04T09:29:58.380 INFO:teuthology.orchestra.run.smithi006.stdout: "bytes_used": 1187840, 2024-03-04T09:29:58.380 INFO:teuthology.orchestra.run.smithi006.stdout: "percent_used": 1.6179305930563714e-06, 2024-03-04T09:29:58.380 INFO:teuthology.orchestra.run.smithi006.stdout: "max_avail": 367085617152 2024-03-04T09:29:58.380 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:29:58.380 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:29:58.380 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:29:58.380 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "cephfs_metadata", 2024-03-04T09:29:58.381 INFO:teuthology.orchestra.run.smithi006.stdout: "id": 12, 2024-03-04T09:29:58.381 INFO:teuthology.orchestra.run.smithi006.stdout: "stats": { 2024-03-04T09:29:58.381 INFO:teuthology.orchestra.run.smithi006.stdout: "stored": 12619, 2024-03-04T09:29:58.381 INFO:teuthology.orchestra.run.smithi006.stdout: "objects": 23, 2024-03-04T09:29:58.381 INFO:teuthology.orchestra.run.smithi006.stdout: "kb_used": 92, 2024-03-04T09:29:58.381 INFO:teuthology.orchestra.run.smithi006.stdout: "bytes_used": 94208, 2024-03-04T09:29:58.381 INFO:teuthology.orchestra.run.smithi006.stdout: "percent_used": 1.2831883111630304e-07, 2024-03-04T09:29:58.381 INFO:teuthology.orchestra.run.smithi006.stdout: "max_avail": 367085617152 2024-03-04T09:29:58.381 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:29:58.381 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:29:58.381 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:29:58.381 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "cephfs_data", 2024-03-04T09:29:58.381 INFO:teuthology.orchestra.run.smithi006.stdout: "id": 13, 2024-03-04T09:29:58.381 INFO:teuthology.orchestra.run.smithi006.stdout: "stats": { 2024-03-04T09:29:58.381 INFO:teuthology.orchestra.run.smithi006.stdout: "stored": 0, 2024-03-04T09:29:58.382 INFO:teuthology.orchestra.run.smithi006.stdout: "objects": 0, 2024-03-04T09:29:58.382 INFO:teuthology.orchestra.run.smithi006.stdout: "kb_used": 0, 2024-03-04T09:29:58.382 INFO:teuthology.orchestra.run.smithi006.stdout: "bytes_used": 0, 2024-03-04T09:29:58.382 INFO:teuthology.orchestra.run.smithi006.stdout: "percent_used": 0, 2024-03-04T09:29:58.382 INFO:teuthology.orchestra.run.smithi006.stdout: "max_avail": 367085617152 2024-03-04T09:29:58.382 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:29:58.382 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:29:58.382 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-04T09:29:58.382 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-04T09:29:58.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:58.384+0000 7f3ccf400640 1 -- 172.21.15.6:0/619244303 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f3ca806b500 msgr2=0x7f3ca806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:58.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:58.384+0000 7f3ccf400640 1 --2- 172.21.15.6:0/619244303 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f3ca806b500 0x7f3ca806d9c0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f3cbc007c10 tx=0x7f3cbc009040 comp rx=0 tx=0).stop 2024-03-04T09:29:58.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:58.384+0000 7f3ccf400640 1 -- 172.21.15.6:0/619244303 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3cc81f3a40 msgr2=0x7f3cc815e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:58.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:58.384+0000 7f3ccf400640 1 --2- 172.21.15.6:0/619244303 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3cc81f3a40 0x7f3cc815e5c0 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f3cc806cb80 tx=0x7f3cc000dc00 comp rx=0 tx=0).stop 2024-03-04T09:29:58.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:58.384+0000 7f3ccf400640 1 -- 172.21.15.6:0/619244303 shutdown_connections 2024-03-04T09:29:58.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:58.384+0000 7f3ccf400640 1 --2- 172.21.15.6:0/619244303 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3cc81f3a40 0x7f3cc815e5c0 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:58.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:58.384+0000 7f3ccf400640 1 --2- 172.21.15.6:0/619244303 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3cc815a4c0 0x7f3cc8156260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:58.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:58.384+0000 7f3ccf400640 1 --2- 172.21.15.6:0/619244303 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f3ca806b500 0x7f3ca806d9c0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:58.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:58.384+0000 7f3ccf400640 1 --2- 172.21.15.6:0/619244303 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3cb0004800 0x7f3cc814ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:58.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:58.384+0000 7f3ccf400640 1 -- 172.21.15.6:0/619244303 >> 172.21.15.6:0/619244303 conn(0x7f3cc80b2640 msgr2=0x7f3cc806bf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:29:58.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:58.385+0000 7f3ccf400640 1 -- 172.21.15.6:0/619244303 shutdown_connections 2024-03-04T09:29:58.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:58.385+0000 7f3ccf400640 1 -- 172.21.15.6:0/619244303 wait complete. 2024-03-04T09:29:58.400 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-03-04T09:29:58.400 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-04T09:29:58.400 DEBUG:teuthology.orchestra.run.smithi006:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-04T09:29:58.500 DEBUG:teuthology.orchestra.run.smithi146:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-04T09:29:58.617 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-04T09:29:58.618 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-04T09:29:58.618 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T09:29:58.674 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-04T09:29:58.674 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T09:29:58.725 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T09:29:58.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:58.943+0000 7fec3a600640 1 Processor -- start 2024-03-04T09:29:58.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:58.943+0000 7fec3a600640 1 -- start start 2024-03-04T09:29:58.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:58.945+0000 7fec3a600640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fec34147870 0x7fec340b5860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:58.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:58.945+0000 7fec3a600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fec3406b6c0 0x7fec340d0c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:58.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:58.945+0000 7fec3a600640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec3406fb90 con 0x7fec3406e710 2024-03-04T09:29:58.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:58.945+0000 7fec3a600640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec34147c50 con 0x7fec34147870 2024-03-04T09:29:58.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:58.945+0000 7fec3a600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec340b3b60 con 0x7fec3406b6c0 2024-03-04T09:29:58.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:58.945+0000 7fec2be00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fec3406e710 0x7fec3406afb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:37176/0 (socket says 172.21.15.6:37176) 2024-03-04T09:29:58.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:58.945+0000 7fec2be00640 1 -- 172.21.15.6:0/1472308315 learned_addr learned my addr 172.21.15.6:0/1472308315 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:29:58.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:58.946+0000 7fec39200640 1 --2- 172.21.15.6:0/1472308315 >> v2:172.21.15.146:3300/0 conn(0x7fec34147870 0x7fec340b5860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:58.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:58.946+0000 7fec39c00640 1 --2- 172.21.15.6:0/1472308315 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fec3406b6c0 0x7fec340d0c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:58.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:58.946+0000 7fec33e00640 1 -- 172.21.15.6:0/1472308315 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1625760361 0 0) 0x7fec3406fb90 con 0x7fec3406e710 2024-03-04T09:29:58.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:58.946+0000 7fec33e00640 1 -- 172.21.15.6:0/1472308315 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec10003520 con 0x7fec3406e710 2024-03-04T09:29:58.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:58.946+0000 7fec33e00640 1 -- 172.21.15.6:0/1472308315 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 360863707 0 0) 0x7fec10003520 con 0x7fec3406e710 2024-03-04T09:29:58.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:58.946+0000 7fec33e00640 1 -- 172.21.15.6:0/1472308315 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fec3406b6c0 msgr2=0x7fec340d0c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:58.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:58.946+0000 7fec33e00640 1 --2- 172.21.15.6:0/1472308315 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fec3406b6c0 0x7fec340d0c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:58.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:58.946+0000 7fec33e00640 1 -- 172.21.15.6:0/1472308315 >> v2:172.21.15.146:3300/0 conn(0x7fec34147870 msgr2=0x7fec340b5860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:58.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:58.946+0000 7fec33e00640 1 --2- 172.21.15.6:0/1472308315 >> v2:172.21.15.146:3300/0 conn(0x7fec34147870 0x7fec340b5860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:58.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:58.946+0000 7fec33e00640 1 -- 172.21.15.6:0/1472308315 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec3406ebf0 con 0x7fec3406e710 2024-03-04T09:29:58.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:58.946+0000 7fec33e00640 1 -- 172.21.15.6:0/1472308315 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7fec20001670 con 0x7fec3406e710 2024-03-04T09:29:58.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:58.947+0000 7fec39c00640 1 --2- 172.21.15.6:0/1472308315 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fec3406b6c0 0x7fec340d0c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T09:29:58.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:58.947+0000 7fec33e00640 1 -- 172.21.15.6:0/1472308315 >> v1:172.21.15.6:6789/0 conn(0x7fec3406e710 legacy=0x7fec3406afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:58.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:58.947+0000 7fec33e00640 1 --2- 172.21.15.6:0/1472308315 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fec100039b0 0x7fec10003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:58.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:58.947+0000 7fec33e00640 1 --2- 172.21.15.6:0/1472308315 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fec3406b6c0 0x7fec10004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:58.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:58.947+0000 7fec33e00640 1 --2- 172.21.15.6:0/1472308315 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fec10004d40 0x7fec100051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:58.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:58.947+0000 7fec33e00640 1 -- 172.21.15.6:0/1472308315 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec34147c50 con 0x7fec10004d40 2024-03-04T09:29:58.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:58.947+0000 7fec33e00640 1 -- 172.21.15.6:0/1472308315 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec340b3b60 con 0x7fec100039b0 2024-03-04T09:29:58.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:58.947+0000 7fec33e00640 1 -- 172.21.15.6:0/1472308315 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec1000b9c0 con 0x7fec3406b6c0 2024-03-04T09:29:58.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:58.947+0000 7fec39c00640 1 --2- 172.21.15.6:0/1472308315 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fec10004d40 0x7fec100051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:58.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:58.947+0000 7fec39200640 1 --2- 172.21.15.6:0/1472308315 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fec100039b0 0x7fec10003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:58.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:58.947+0000 7fec2be00640 1 --2- 172.21.15.6:0/1472308315 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fec3406b6c0 0x7fec10004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:58.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:58.948+0000 7fec39c00640 1 -- 172.21.15.6:0/1472308315 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fec3406b6c0 msgr2=0x7fec10004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:58.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:58.948+0000 7fec39c00640 1 --2- 172.21.15.6:0/1472308315 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fec3406b6c0 0x7fec10004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:58.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:58.948+0000 7fec39c00640 1 -- 172.21.15.6:0/1472308315 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fec100039b0 msgr2=0x7fec10003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:58.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:58.948+0000 7fec39c00640 1 --2- 172.21.15.6:0/1472308315 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fec100039b0 0x7fec10003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:58.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:58.948+0000 7fec39c00640 1 -- 172.21.15.6:0/1472308315 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec10003520 con 0x7fec10004d40 2024-03-04T09:29:58.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:58.949+0000 7fec39c00640 1 --2- 172.21.15.6:0/1472308315 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fec10004d40 0x7fec100051c0 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7fec34162ad0 tx=0x7fec10005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e7b298a15dca4ed server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:58.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:58.949+0000 7fec33e00640 1 -- 172.21.15.6:0/1472308315 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec2c009de0 con 0x7fec10004d40 2024-03-04T09:29:58.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:58.949+0000 7fec33e00640 1 -- 172.21.15.6:0/1472308315 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec2c02cd00 con 0x7fec10004d40 2024-03-04T09:29:58.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:58.949+0000 7fec33e00640 1 -- 172.21.15.6:0/1472308315 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec2c003b90 con 0x7fec10004d40 2024-03-04T09:29:58.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:58.949+0000 7fec3a600640 1 -- 172.21.15.6:0/1472308315 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fec10004d40 msgr2=0x7fec100051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:58.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:58.949+0000 7fec3a600640 1 --2- 172.21.15.6:0/1472308315 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fec10004d40 0x7fec100051c0 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7fec34162ad0 tx=0x7fec10005830 comp rx=0 tx=0).stop 2024-03-04T09:29:58.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:58.949+0000 7fec39c00640 1 -- 172.21.15.6:0/1472308315 reap_dead start 2024-03-04T09:29:58.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:58.950+0000 7fec3a600640 1 -- 172.21.15.6:0/1472308315 shutdown_connections 2024-03-04T09:29:58.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:58.950+0000 7fec3a600640 1 -- 172.21.15.6:0/1472308315 >> 172.21.15.6:0/1472308315 conn(0x7fec3406c0c0 msgr2=0x7fec3406c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:29:58.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:58.950+0000 7fec3a600640 1 -- 172.21.15.6:0/1472308315 shutdown_connections 2024-03-04T09:29:58.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:58.950+0000 7fec3a600640 1 -- 172.21.15.6:0/1472308315 wait complete. 2024-03-04T09:29:58.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:58.951+0000 7fec3a600640 1 Processor -- start 2024-03-04T09:29:58.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:58.952+0000 7fec3a600640 1 -- start start 2024-03-04T09:29:58.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:58.952+0000 7fec3a600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fec10004d40 0x7fec340b0640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:58.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:58.952+0000 7fec3a600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fec34145bd0 0x7fec34146030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:58.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:58.952+0000 7fec3a600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fec34131750 0x7fec34131bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:58.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:58.952+0000 7fec3a600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec340b4370 con 0x7fec34145bd0 2024-03-04T09:29:58.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:58.952+0000 7fec3a600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec340b4090 con 0x7fec10004d40 2024-03-04T09:29:58.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:58.953+0000 7fec3a600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec340b5da0 con 0x7fec34131750 2024-03-04T09:29:58.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:58.953+0000 7fec2be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fec34145bd0 0x7fec34146030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:58.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:58.953+0000 7fec2be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fec34145bd0 0x7fec34146030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:34020/0 (socket says 172.21.15.6:34020) 2024-03-04T09:29:58.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:58.953+0000 7fec2be00640 1 -- 172.21.15.6:0/4183244170 learned_addr learned my addr 172.21.15.6:0/4183244170 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:29:58.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:58.953+0000 7fec39200640 1 --2- 172.21.15.6:0/4183244170 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fec10004d40 0x7fec340b0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:58.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:58.953+0000 7fec39c00640 1 --2- 172.21.15.6:0/4183244170 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fec34131750 0x7fec34131bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:58.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:58.953+0000 7fec2be00640 1 -- 172.21.15.6:0/4183244170 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fec34131750 msgr2=0x7fec34131bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:58.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:58.953+0000 7fec2be00640 1 --2- 172.21.15.6:0/4183244170 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fec34131750 0x7fec34131bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:58.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:58.953+0000 7fec2be00640 1 -- 172.21.15.6:0/4183244170 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fec10004d40 msgr2=0x7fec340b0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:58.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:58.953+0000 7fec2be00640 1 --2- 172.21.15.6:0/4183244170 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fec10004d40 0x7fec340b0640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:58.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:58.953+0000 7fec2be00640 1 -- 172.21.15.6:0/4183244170 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec340ad880 con 0x7fec34145bd0 2024-03-04T09:29:58.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:58.953+0000 7fec39c00640 1 --2- 172.21.15.6:0/4183244170 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fec34131750 0x7fec34131bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:29:58.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:58.954+0000 7fec2be00640 1 --2- 172.21.15.6:0/4183244170 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fec34145bd0 0x7fec34146030 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7fec2000e840 tx=0x7fec2000ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:58.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:58.954+0000 7fec31600640 1 -- 172.21.15.6:0/4183244170 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec2000c8d0 con 0x7fec34145bd0 2024-03-04T09:29:58.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:58.954+0000 7fec31600640 1 -- 172.21.15.6:0/4183244170 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec20018070 con 0x7fec34145bd0 2024-03-04T09:29:58.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:58.954+0000 7fec31600640 1 -- 172.21.15.6:0/4183244170 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec2000ccd0 con 0x7fec34145bd0 2024-03-04T09:29:58.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:58.954+0000 7fec3a600640 1 -- 172.21.15.6:0/4183244170 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec341324f0 con 0x7fec34145bd0 2024-03-04T09:29:58.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:58.954+0000 7fec3a600640 1 -- 172.21.15.6:0/4183244170 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec3412fbd0 con 0x7fec34145bd0 2024-03-04T09:29:58.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:58.955+0000 7fec3a600640 1 -- 172.21.15.6:0/4183244170 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec3404f330 con 0x7fec34145bd0 2024-03-04T09:29:58.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:58.956+0000 7fec31600640 1 -- 172.21.15.6:0/4183244170 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fec2001d460 con 0x7fec34145bd0 2024-03-04T09:29:58.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:58.958+0000 7fec31600640 1 --2- 172.21.15.6:0/4183244170 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fec0806b500 0x7fec0806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:58.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:58.958+0000 7fec31600640 1 -- 172.21.15.6:0/4183244170 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fec2008a220 con 0x7fec34145bd0 2024-03-04T09:29:58.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:58.959+0000 7fec39200640 1 --2- 172.21.15.6:0/4183244170 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fec0806b500 0x7fec0806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:58.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:58.960+0000 7fec39200640 1 --2- 172.21.15.6:0/4183244170 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fec0806b500 0x7fec0806d9c0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fec18009a10 tx=0x7fec18008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:58.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:58.961+0000 7fec31600640 1 -- 172.21.15.6:0/4183244170 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fec340ad880 con 0x7fec34145bd0 2024-03-04T09:29:59.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.164+0000 7fec3a600640 1 -- 172.21.15.6:0/4183244170 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fec340724f0 con 0x7fec34145bd0 2024-03-04T09:29:59.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.166+0000 7fec31600640 1 -- 172.21.15.6:0/4183244170 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+16612 (secure 0 0 0) 0x7fec20059090 con 0x7fec34145bd0 2024-03-04T09:29:59.163 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:29:59.164 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":81,"fsid":"edce0578-5ae4-4402-b45a-552bdef84d7c","created":"2024-03-04T09:24:54.523780+0000","modified":"2024-03-04T09:29:48.885831+0000","last_up_change":"2024-03-04T09:25:07.089049+0000","last_in_change":"2024-03-04T09:24:57.982077+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T09:25:07.935198+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-03-04T09:29:45.280632+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-03-04T09:29:46.378588+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a96574f-81f1-4bc6-8463-757520b84a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6825","nonce":1364449401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6827","nonce":1364449401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6831","nonce":1364449401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6829","nonce":1364449401}]},"public_addr":"172.21.15.6:6825/1364449401","cluster_addr":"172.21.15.6:6827/1364449401","heartbeat_back_addr":"172.21.15.6:6831/1364449401","heartbeat_front_addr":"172.21.15.6:6829/1364449401","state":["exists","up"]},{"osd":1,"uuid":"7d79b9c1-6b87-476a-bc36-7db5ee64cd90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6809","nonce":2494103078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6811","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6813","nonce":2494103078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6821","nonce":2494103078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6817","nonce":2494103078}]},"public_addr":"172.21.15.6:6809/2494103078","cluster_addr":"172.21.15.6:6813/2494103078","heartbeat_back_addr":"172.21.15.6:6821/2494103078","heartbeat_front_addr":"172.21.15.6:6817/2494103078","state":["exists","up"]},{"osd":2,"uuid":"492331b9-4ee9-40b6-bacd-85d44cd6c45a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6812","nonce":1500066681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6816","nonce":1500066681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6824","nonce":1500066681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6820","nonce":1500066681}]},"public_addr":"172.21.15.6:6812/1500066681","cluster_addr":"172.21.15.6:6816/1500066681","heartbeat_back_addr":"172.21.15.6:6824/1500066681","heartbeat_front_addr":"172.21.15.6:6820/1500066681","state":["exists","up"]},{"osd":3,"uuid":"21bb2efe-9ba7-4bdc-bc31-180cd1102e21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6801","nonce":1558806123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6803","nonce":1558806123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6807","nonce":1558806123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6805","nonce":1558806123}]},"public_addr":"172.21.15.6:6801/1558806123","cluster_addr":"172.21.15.6:6803/1558806123","heartbeat_back_addr":"172.21.15.6:6807/1558806123","heartbeat_front_addr":"172.21.15.6:6805/1558806123","state":["exists","up"]},{"osd":4,"uuid":"6e2c4b71-19d9-4737-83d9-558705d8644a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6801","nonce":1526135780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6803","nonce":1526135780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6807","nonce":1526135780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6805","nonce":1526135780}]},"public_addr":"172.21.15.146:6801/1526135780","cluster_addr":"172.21.15.146:6803/1526135780","heartbeat_back_addr":"172.21.15.146:6807/1526135780","heartbeat_front_addr":"172.21.15.146:6805/1526135780","state":["exists","up"]},{"osd":5,"uuid":"fb69c525-f1be-4b7d-ab32-6c29b0d6fddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6809","nonce":460262507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6811","nonce":460262507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6815","nonce":460262507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6813","nonce":460262507}]},"public_addr":"172.21.15.146:6809/460262507","cluster_addr":"172.21.15.146:6811/460262507","heartbeat_back_addr":"172.21.15.146:6815/460262507","heartbeat_front_addr":"172.21.15.146:6813/460262507","state":["exists","up"]},{"osd":6,"uuid":"c32e7fed-370d-4401-8ce0-6a4335b0d168","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6823","nonce":1392897493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6827","nonce":1392897493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6831","nonce":1392897493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6829","nonce":1392897493}]},"public_addr":"172.21.15.146:6823/1392897493","cluster_addr":"172.21.15.146:6827/1392897493","heartbeat_back_addr":"172.21.15.146:6831/1392897493","heartbeat_front_addr":"172.21.15.146:6829/1392897493","state":["exists","up"]},{"osd":7,"uuid":"75a79b44-fa5f-426d-a7d3-71f21f78835c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6817","nonce":3268323156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6819","nonce":3268323156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6825","nonce":3268323156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6822","nonce":3268323156}]},"public_addr":"172.21.15.146:6817/3268323156","cluster_addr":"172.21.15.146:6819/3268323156","heartbeat_back_addr":"172.21.15.146:6825/3268323156","heartbeat_front_addr":"172.21.15.146:6822/3268323156","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.310437+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.295850+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.294674+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.526855+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.300023+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.595160+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.346046+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.558741+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T09:29:59.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.169+0000 7fec3a600640 1 -- 172.21.15.6:0/4183244170 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fec0806b500 msgr2=0x7fec0806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:59.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.169+0000 7fec3a600640 1 --2- 172.21.15.6:0/4183244170 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fec0806b500 0x7fec0806d9c0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fec18009a10 tx=0x7fec18008040 comp rx=0 tx=0).stop 2024-03-04T09:29:59.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.169+0000 7fec3a600640 1 -- 172.21.15.6:0/4183244170 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fec34145bd0 msgr2=0x7fec34146030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:59.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.169+0000 7fec3a600640 1 --2- 172.21.15.6:0/4183244170 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fec34145bd0 0x7fec34146030 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7fec2000e840 tx=0x7fec2000ed00 comp rx=0 tx=0).stop 2024-03-04T09:29:59.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.169+0000 7fec3a600640 1 -- 172.21.15.6:0/4183244170 shutdown_connections 2024-03-04T09:29:59.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.169+0000 7fec3a600640 1 --2- 172.21.15.6:0/4183244170 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fec0806b500 0x7fec0806d9c0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:59.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.169+0000 7fec3a600640 1 --2- 172.21.15.6:0/4183244170 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fec34131750 0x7fec34131bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:59.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.169+0000 7fec3a600640 1 --2- 172.21.15.6:0/4183244170 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fec34145bd0 0x7fec34146030 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:59.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.169+0000 7fec3a600640 1 --2- 172.21.15.6:0/4183244170 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fec10004d40 0x7fec340b0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:59.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.169+0000 7fec3a600640 1 -- 172.21.15.6:0/4183244170 >> 172.21.15.6:0/4183244170 conn(0x7fec3406c0c0 msgr2=0x7fec3415fd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:29:59.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.169+0000 7fec3a600640 1 -- 172.21.15.6:0/4183244170 shutdown_connections 2024-03-04T09:29:59.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.169+0000 7fec3a600640 1 -- 172.21.15.6:0/4183244170 wait complete. 2024-03-04T09:29:59.186 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T09:25:07.935198+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-04T09:29:45.280632+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '80', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_data', 'create_time': '2024-03-04T09:29:46.378588+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-04T09:29:59.187 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-04T09:29:59.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.407+0000 7fa767e00640 1 Processor -- start 2024-03-04T09:29:59.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.408+0000 7fa767e00640 1 -- start start 2024-03-04T09:29:59.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.409+0000 7fa767e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fa7680d4860 0x7fa7680d0630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:59.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.409+0000 7fa767e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa7680cfa60 0x7fa7680d0b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:59.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.409+0000 7fa767e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7680ceca0 con 0x7fa76815dee0 2024-03-04T09:29:59.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.409+0000 7fa767e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7680cf5d0 con 0x7fa7680cfa60 2024-03-04T09:29:59.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.409+0000 7fa767e00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7680cef80 con 0x7fa7680d4860 2024-03-04T09:29:59.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.410+0000 7fa766a00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fa76815dee0 0x7fa7680d18a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:37186/0 (socket says 172.21.15.6:37186) 2024-03-04T09:29:59.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.410+0000 7fa766a00640 1 -- 172.21.15.6:0/2499102175 learned_addr learned my addr 172.21.15.6:0/2499102175 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:29:59.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.410+0000 7fa767400640 1 --2- 172.21.15.6:0/2499102175 >> v2:172.21.15.6:3300/0 conn(0x7fa7680cfa60 0x7fa7680d0b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:59.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.410+0000 7fa75e000640 1 --2- 172.21.15.6:0/2499102175 >> v2:172.21.15.146:3300/0 conn(0x7fa7680d4860 0x7fa7680d0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:59.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.410+0000 7fa766000640 1 -- 172.21.15.6:0/2499102175 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 174122232 0 0) 0x7fa7680ceca0 con 0x7fa76815dee0 2024-03-04T09:29:59.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.410+0000 7fa766000640 1 -- 172.21.15.6:0/2499102175 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa74c003520 con 0x7fa76815dee0 2024-03-04T09:29:59.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.410+0000 7fa767400640 1 -- 172.21.15.6:0/2499102175 >> v2:172.21.15.146:3300/0 conn(0x7fa7680d4860 msgr2=0x7fa7680d0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:59.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.410+0000 7fa767400640 1 --2- 172.21.15.6:0/2499102175 >> v2:172.21.15.146:3300/0 conn(0x7fa7680d4860 0x7fa7680d0630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:59.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.410+0000 7fa767400640 1 -- 172.21.15.6:0/2499102175 >> v1:172.21.15.6:6789/0 conn(0x7fa76815dee0 legacy=0x7fa7680d18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:59.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.410+0000 7fa767400640 1 -- 172.21.15.6:0/2499102175 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7680cf260 con 0x7fa7680cfa60 2024-03-04T09:29:59.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.411+0000 7fa767400640 1 --2- 172.21.15.6:0/2499102175 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa7680cfa60 0x7fa7680d0b70 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7fa768162b70 tx=0x7fa76002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c88b1f532967d93 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:59.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.412+0000 7fa766000640 1 -- 172.21.15.6:0/2499102175 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa760035020 con 0x7fa7680cfa60 2024-03-04T09:29:59.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.412+0000 7fa766000640 1 -- 172.21.15.6:0/2499102175 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa760031830 con 0x7fa7680cfa60 2024-03-04T09:29:59.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.412+0000 7fa766000640 1 -- 172.21.15.6:0/2499102175 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa760031c00 con 0x7fa7680cfa60 2024-03-04T09:29:59.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.413+0000 7fa767e00640 1 -- 172.21.15.6:0/2499102175 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa7680cfa60 msgr2=0x7fa7680d0b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:59.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.413+0000 7fa767e00640 1 --2- 172.21.15.6:0/2499102175 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa7680cfa60 0x7fa7680d0b70 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7fa768162b70 tx=0x7fa76002b8d0 comp rx=0 tx=0).stop 2024-03-04T09:29:59.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.413+0000 7fa767e00640 1 -- 172.21.15.6:0/2499102175 shutdown_connections 2024-03-04T09:29:59.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.413+0000 7fa767e00640 1 --2- 172.21.15.6:0/2499102175 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa7680cfa60 0x7fa7680d0b70 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:59.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.413+0000 7fa767e00640 1 --2- 172.21.15.6:0/2499102175 >> v2:172.21.15.146:3300/0 conn(0x7fa7680d4860 0x7fa7680d0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:59.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.413+0000 7fa767e00640 1 -- 172.21.15.6:0/2499102175 >> 172.21.15.6:0/2499102175 conn(0x7fa7680d3cd0 msgr2=0x7fa7680d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:29:59.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.413+0000 7fa767e00640 1 -- 172.21.15.6:0/2499102175 shutdown_connections 2024-03-04T09:29:59.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.413+0000 7fa767e00640 1 -- 172.21.15.6:0/2499102175 wait complete. 2024-03-04T09:29:59.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.414+0000 7fa767e00640 1 Processor -- start 2024-03-04T09:29:59.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.415+0000 7fa767e00640 1 -- start start 2024-03-04T09:29:59.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.416+0000 7fa767e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa7680cfa60 0x7fa76815b280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:59.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.416+0000 7fa767e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa7680d4860 0x7fa76815be20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:59.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.416+0000 7fa767e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa76815dee0 0x7fa7680ca020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:59.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.416+0000 7fa767e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7680b3510 con 0x7fa7680d4860 2024-03-04T09:29:59.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.416+0000 7fa767e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa76815e2c0 con 0x7fa76815dee0 2024-03-04T09:29:59.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.416+0000 7fa767e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7680b3690 con 0x7fa7680cfa60 2024-03-04T09:29:59.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.416+0000 7fa75e000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa7680d4860 0x7fa76815be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:59.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.416+0000 7fa75e000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa7680d4860 0x7fa76815be20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:34036/0 (socket says 172.21.15.6:34036) 2024-03-04T09:29:59.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.416+0000 7fa75e000640 1 -- 172.21.15.6:0/4020878267 learned_addr learned my addr 172.21.15.6:0/4020878267 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:29:59.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.416+0000 7fa767400640 1 --2- 172.21.15.6:0/4020878267 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa76815dee0 0x7fa7680ca020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:59.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.416+0000 7fa75e000640 1 -- 172.21.15.6:0/4020878267 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa7680cfa60 msgr2=0x7fa76815b280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:59.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.416+0000 7fa75e000640 1 --2- 172.21.15.6:0/4020878267 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa7680cfa60 0x7fa76815b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:59.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.416+0000 7fa75e000640 1 -- 172.21.15.6:0/4020878267 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa76815dee0 msgr2=0x7fa7680ca020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:59.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.416+0000 7fa75e000640 1 --2- 172.21.15.6:0/4020878267 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa76815dee0 0x7fa7680ca020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:59.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.416+0000 7fa75e000640 1 -- 172.21.15.6:0/4020878267 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7680c9260 con 0x7fa7680d4860 2024-03-04T09:29:59.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.417+0000 7fa75e000640 1 --2- 172.21.15.6:0/4020878267 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa7680d4860 0x7fa76815be20 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7fa75000efc0 tx=0x7fa75000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:59.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.417+0000 7fa75f400640 1 -- 172.21.15.6:0/4020878267 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa750018020 con 0x7fa7680d4860 2024-03-04T09:29:59.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.417+0000 7fa75f400640 1 -- 172.21.15.6:0/4020878267 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa750014480 con 0x7fa7680d4860 2024-03-04T09:29:59.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.417+0000 7fa75f400640 1 -- 172.21.15.6:0/4020878267 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa750014850 con 0x7fa7680d4860 2024-03-04T09:29:59.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.417+0000 7fa767e00640 1 -- 172.21.15.6:0/4020878267 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7680c9540 con 0x7fa7680d4860 2024-03-04T09:29:59.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.417+0000 7fa767e00640 1 -- 172.21.15.6:0/4020878267 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7680c9b00 con 0x7fa7680d4860 2024-03-04T09:29:59.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.418+0000 7fa767e00640 1 -- 172.21.15.6:0/4020878267 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa76804f330 con 0x7fa7680d4860 2024-03-04T09:29:59.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.419+0000 7fa75f400640 1 -- 172.21.15.6:0/4020878267 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa750005d10 con 0x7fa7680d4860 2024-03-04T09:29:59.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.420+0000 7fa75f400640 1 --2- 172.21.15.6:0/4020878267 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa74406b4d0 0x7fa74406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:59.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.420+0000 7fa75f400640 1 -- 172.21.15.6:0/4020878267 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa75008ad80 con 0x7fa7680d4860 2024-03-04T09:29:59.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.421+0000 7fa766a00640 1 --2- 172.21.15.6:0/4020878267 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa74406b4d0 0x7fa74406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:59.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.422+0000 7fa766a00640 1 --2- 172.21.15.6:0/4020878267 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa74406b4d0 0x7fa74406d990 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fa758000f60 tx=0x7fa7580078f0 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:59.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.425+0000 7fa75f400640 1 -- 172.21.15.6:0/4020878267 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa750022070 con 0x7fa7680d4860 2024-03-04T09:29:59.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.628+0000 7fa767e00640 1 -- 172.21.15.6:0/4020878267 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa76808d7d0 con 0x7fa7680d4860 2024-03-04T09:29:59.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.629+0000 7fa75f400640 1 -- 172.21.15.6:0/4020878267 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa750010040 con 0x7fa7680d4860 2024-03-04T09:29:59.626 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-04T09:29:59.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.631+0000 7fa767e00640 1 -- 172.21.15.6:0/4020878267 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa74406b4d0 msgr2=0x7fa74406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:59.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.631+0000 7fa767e00640 1 --2- 172.21.15.6:0/4020878267 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa74406b4d0 0x7fa74406d990 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fa758000f60 tx=0x7fa7580078f0 comp rx=0 tx=0).stop 2024-03-04T09:29:59.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.631+0000 7fa767e00640 1 -- 172.21.15.6:0/4020878267 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa7680d4860 msgr2=0x7fa76815be20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:59.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.631+0000 7fa767e00640 1 --2- 172.21.15.6:0/4020878267 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa7680d4860 0x7fa76815be20 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7fa75000efc0 tx=0x7fa75000c490 comp rx=0 tx=0).stop 2024-03-04T09:29:59.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.631+0000 7fa767e00640 1 -- 172.21.15.6:0/4020878267 shutdown_connections 2024-03-04T09:29:59.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.631+0000 7fa767e00640 1 --2- 172.21.15.6:0/4020878267 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa76815dee0 0x7fa7680ca020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:59.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.631+0000 7fa767e00640 1 --2- 172.21.15.6:0/4020878267 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa7680d4860 0x7fa76815be20 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:59.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.631+0000 7fa767e00640 1 --2- 172.21.15.6:0/4020878267 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa74406b4d0 0x7fa74406d990 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:59.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.631+0000 7fa767e00640 1 --2- 172.21.15.6:0/4020878267 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa7680cfa60 0x7fa76815b280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:59.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.631+0000 7fa767e00640 1 -- 172.21.15.6:0/4020878267 >> 172.21.15.6:0/4020878267 conn(0x7fa7680d3cd0 msgr2=0x7fa76806cf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:29:59.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.632+0000 7fa767e00640 1 -- 172.21.15.6:0/4020878267 shutdown_connections 2024-03-04T09:29:59.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.632+0000 7fa767e00640 1 -- 172.21.15.6:0/4020878267 wait complete. 2024-03-04T09:29:59.647 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-04T09:29:59.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.879+0000 7fb77a200640 1 Processor -- start 2024-03-04T09:29:59.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.879+0000 7fb77a200640 1 -- start start 2024-03-04T09:29:59.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.881+0000 7fb77a200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fb774072520 0x7fb7740cdfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:59.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.882+0000 7fb77a200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb774072df0 con 0x7fb77415bf20 2024-03-04T09:29:59.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.882+0000 7fb77a200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7740cdab0 con 0x7fb774071e20 2024-03-04T09:29:59.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.882+0000 7fb77a200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7740b4d70 con 0x7fb774072520 2024-03-04T09:29:59.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.882+0000 7fb773400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fb77415bf20 0x7fb7740acd40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:37190/0 (socket says 172.21.15.6:37190) 2024-03-04T09:29:59.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.882+0000 7fb773400640 1 -- 172.21.15.6:0/2671756460 learned_addr learned my addr 172.21.15.6:0/2671756460 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:29:59.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.882+0000 7fb772a00640 1 --2- 172.21.15.6:0/2671756460 >> v2:172.21.15.146:3300/0 conn(0x7fb774072520 0x7fb7740cdfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:59.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.882+0000 7fb772000640 1 -- 172.21.15.6:0/2671756460 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2129086654 0 0) 0x7fb774072df0 con 0x7fb77415bf20 2024-03-04T09:29:59.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.883+0000 7fb772000640 1 -- 172.21.15.6:0/2671756460 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb758003520 con 0x7fb77415bf20 2024-03-04T09:29:59.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.883+0000 7fb772000640 1 -- 172.21.15.6:0/2671756460 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1053003121 0 0) 0x7fb758003520 con 0x7fb77415bf20 2024-03-04T09:29:59.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.883+0000 7fb772000640 1 -- 172.21.15.6:0/2671756460 >> v2:172.21.15.146:3300/0 conn(0x7fb774072520 msgr2=0x7fb7740cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:59.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.883+0000 7fb772000640 1 --2- 172.21.15.6:0/2671756460 >> v2:172.21.15.146:3300/0 conn(0x7fb774072520 0x7fb7740cdfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:59.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.883+0000 7fb772000640 1 -- 172.21.15.6:0/2671756460 >> v1:172.21.15.146:6789/0 conn(0x7fb774071e20 legacy=0x7fb7740ccad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:59.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.883+0000 7fb772000640 1 -- 172.21.15.6:0/2671756460 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7740b5050 con 0x7fb77415bf20 2024-03-04T09:29:59.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.883+0000 7fb772000640 1 -- 172.21.15.6:0/2671756460 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7fb764001670 con 0x7fb77415bf20 2024-03-04T09:29:59.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.883+0000 7fb772000640 1 -- 172.21.15.6:0/2671756460 >> v1:172.21.15.6:6789/0 conn(0x7fb77415bf20 legacy=0x7fb7740acd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:59.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.883+0000 7fb772000640 1 --2- 172.21.15.6:0/2671756460 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb7580039b0 0x7fb758003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:59.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.883+0000 7fb772000640 1 --2- 172.21.15.6:0/2671756460 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb758004800 0x7fb758004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:59.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.883+0000 7fb772000640 1 --2- 172.21.15.6:0/2671756460 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb7580051c0 0x7fb75800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:59.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.883+0000 7fb772000640 1 -- 172.21.15.6:0/2671756460 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7740b4d70 con 0x7fb7580051c0 2024-03-04T09:29:59.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.883+0000 7fb772000640 1 -- 172.21.15.6:0/2671756460 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb75800c060 con 0x7fb758004800 2024-03-04T09:29:59.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.883+0000 7fb772000640 1 -- 172.21.15.6:0/2671756460 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb75800c260 con 0x7fb7580039b0 2024-03-04T09:29:59.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.884+0000 7fb773e00640 1 --2- 172.21.15.6:0/2671756460 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb7580051c0 0x7fb75800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:59.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.884+0000 7fb772a00640 1 --2- 172.21.15.6:0/2671756460 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb758004800 0x7fb758004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:59.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.884+0000 7fb773400640 1 --2- 172.21.15.6:0/2671756460 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb7580039b0 0x7fb758003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:59.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.884+0000 7fb773e00640 1 -- 172.21.15.6:0/2671756460 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb7580039b0 msgr2=0x7fb758003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:59.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.884+0000 7fb773e00640 1 --2- 172.21.15.6:0/2671756460 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb7580039b0 0x7fb758003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:59.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.884+0000 7fb773e00640 1 -- 172.21.15.6:0/2671756460 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb758004800 msgr2=0x7fb758004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:29:59.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.884+0000 7fb773e00640 1 --2- 172.21.15.6:0/2671756460 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb758004800 0x7fb758004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:59.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.884+0000 7fb773e00640 1 -- 172.21.15.6:0/2671756460 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb758003520 con 0x7fb7580051c0 2024-03-04T09:29:59.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.884+0000 7fb773400640 1 --2- 172.21.15.6:0/2671756460 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb7580039b0 0x7fb758003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:29:59.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.885+0000 7fb773e00640 1 -- 172.21.15.6:0/2671756460 reap_dead start 2024-03-04T09:29:59.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.885+0000 7fb773e00640 1 --2- 172.21.15.6:0/2671756460 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb7580051c0 0x7fb75800b9c0 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7fb76c001e50 tx=0x7fb758005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc25b361adb3ce83 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:59.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.885+0000 7fb772000640 1 -- 172.21.15.6:0/2671756460 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb76c001a70 con 0x7fb7580051c0 2024-03-04T09:29:59.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.886+0000 7fb772000640 1 -- 172.21.15.6:0/2671756460 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb76c003bb0 con 0x7fb7580051c0 2024-03-04T09:29:59.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.886+0000 7fb772000640 1 -- 172.21.15.6:0/2671756460 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb76c0050d0 con 0x7fb7580051c0 2024-03-04T09:29:59.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.886+0000 7fb77a200640 1 -- 172.21.15.6:0/2671756460 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb7580051c0 msgr2=0x7fb75800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:59.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.886+0000 7fb77a200640 1 --2- 172.21.15.6:0/2671756460 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb7580051c0 0x7fb75800b9c0 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7fb76c001e50 tx=0x7fb758005830 comp rx=0 tx=0).stop 2024-03-04T09:29:59.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.886+0000 7fb77a200640 1 -- 172.21.15.6:0/2671756460 shutdown_connections 2024-03-04T09:29:59.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.886+0000 7fb77a200640 1 --2- 172.21.15.6:0/2671756460 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb7580051c0 0x7fb75800b9c0 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:59.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.886+0000 7fb77a200640 1 -- 172.21.15.6:0/2671756460 >> 172.21.15.6:0/2671756460 conn(0x7fb77406b3d0 msgr2=0x7fb77406b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:29:59.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.886+0000 7fb77a200640 1 -- 172.21.15.6:0/2671756460 shutdown_connections 2024-03-04T09:29:59.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.886+0000 7fb77a200640 1 -- 172.21.15.6:0/2671756460 wait complete. 2024-03-04T09:29:59.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.888+0000 7fb77a200640 1 Processor -- start 2024-03-04T09:29:59.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.888+0000 7fb77a200640 1 -- start start 2024-03-04T09:29:59.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.889+0000 7fb77a200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb7580051c0 0x7fb7740cbac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:59.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.889+0000 7fb77a200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb77415ab00 0x7fb7740c99f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:59.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.889+0000 7fb77a200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb77415af80 0x7fb7740c9f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:59.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.889+0000 7fb77a200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7740b4b10 con 0x7fb77415af80 2024-03-04T09:29:59.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.889+0000 7fb77a200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7740b4990 con 0x7fb7580051c0 2024-03-04T09:29:59.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.889+0000 7fb77a200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7740acd40 con 0x7fb77415ab00 2024-03-04T09:29:59.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.889+0000 7fb773e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb77415af80 0x7fb7740c9f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:59.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.889+0000 7fb773e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb77415af80 0x7fb7740c9f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:34060/0 (socket says 172.21.15.6:34060) 2024-03-04T09:29:59.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.889+0000 7fb773e00640 1 -- 172.21.15.6:0/2639060241 learned_addr learned my addr 172.21.15.6:0/2639060241 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:29:59.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.890+0000 7fb772a00640 1 --2- 172.21.15.6:0/2639060241 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb77415ab00 0x7fb7740c99f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:59.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.890+0000 7fb773400640 1 --2- 172.21.15.6:0/2639060241 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb7580051c0 0x7fb7740cbac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:59.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.890+0000 7fb773e00640 1 -- 172.21.15.6:0/2639060241 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb77415ab00 msgr2=0x7fb7740c99f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:59.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.890+0000 7fb773e00640 1 --2- 172.21.15.6:0/2639060241 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb77415ab00 0x7fb7740c99f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:59.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.890+0000 7fb773e00640 1 -- 172.21.15.6:0/2639060241 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb7580051c0 msgr2=0x7fb7740cbac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:29:59.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.890+0000 7fb773e00640 1 --2- 172.21.15.6:0/2639060241 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb7580051c0 0x7fb7740cbac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:29:59.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.890+0000 7fb773e00640 1 -- 172.21.15.6:0/2639060241 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7740caa60 con 0x7fb77415af80 2024-03-04T09:29:59.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.890+0000 7fb773400640 1 --2- 172.21.15.6:0/2639060241 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb7580051c0 0x7fb7740cbac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:29:59.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.890+0000 7fb773e00640 1 --2- 172.21.15.6:0/2639060241 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb77415af80 0x7fb7740c9f30 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7fb76c02c060 tx=0x7fb76c0014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:59.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.891+0000 7fb76b400640 1 -- 172.21.15.6:0/2639060241 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb76c007db0 con 0x7fb77415af80 2024-03-04T09:29:59.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.891+0000 7fb76b400640 1 -- 172.21.15.6:0/2639060241 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb76c006d60 con 0x7fb77415af80 2024-03-04T09:29:59.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.891+0000 7fb76b400640 1 -- 172.21.15.6:0/2639060241 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb76c0396a0 con 0x7fb77415af80 2024-03-04T09:29:59.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.891+0000 7fb77a200640 1 -- 172.21.15.6:0/2639060241 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb77409d540 con 0x7fb77415af80 2024-03-04T09:29:59.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.891+0000 7fb77a200640 1 -- 172.21.15.6:0/2639060241 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7740cafe0 con 0x7fb77415af80 2024-03-04T09:29:59.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.892+0000 7fb77a200640 1 -- 172.21.15.6:0/2639060241 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb77404f330 con 0x7fb77415af80 2024-03-04T09:29:59.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.893+0000 7fb76b400640 1 -- 172.21.15.6:0/2639060241 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb76c039840 con 0x7fb77415af80 2024-03-04T09:29:59.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.894+0000 7fb76b400640 1 --2- 172.21.15.6:0/2639060241 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb75406b4d0 0x7fb75406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:29:59.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.894+0000 7fb76b400640 1 -- 172.21.15.6:0/2639060241 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb76c0ac170 con 0x7fb77415af80 2024-03-04T09:29:59.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.895+0000 7fb773400640 1 --2- 172.21.15.6:0/2639060241 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb75406b4d0 0x7fb75406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:29:59.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.896+0000 7fb773400640 1 --2- 172.21.15.6:0/2639060241 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb75406b4d0 0x7fb75406d990 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fb764001200 tx=0x7fb764009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:29:59.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:29:59.899+0000 7fb76b400640 1 -- 172.21.15.6:0/2639060241 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb76c033030 con 0x7fb77415af80 2024-03-04T09:30:00.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.100+0000 7fb77a200640 1 -- 172.21.15.6:0/2639060241 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb7740b3190 con 0x7fb77415af80 2024-03-04T09:30:00.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.101+0000 7fb76b400640 1 -- 172.21.15.6:0/2639060241 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v81) v1 ==== 104+0+11 (secure 0 0 0) 0x7fb76c07a900 con 0x7fb77415af80 2024-03-04T09:30:00.098 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-03-04T09:30:00.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.103+0000 7fb77a200640 1 -- 172.21.15.6:0/2639060241 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb75406b4d0 msgr2=0x7fb75406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:00.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.103+0000 7fb77a200640 1 --2- 172.21.15.6:0/2639060241 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb75406b4d0 0x7fb75406d990 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fb764001200 tx=0x7fb764009040 comp rx=0 tx=0).stop 2024-03-04T09:30:00.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.103+0000 7fb77a200640 1 -- 172.21.15.6:0/2639060241 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb77415af80 msgr2=0x7fb7740c9f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:00.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.103+0000 7fb77a200640 1 --2- 172.21.15.6:0/2639060241 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb77415af80 0x7fb7740c9f30 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7fb76c02c060 tx=0x7fb76c0014a0 comp rx=0 tx=0).stop 2024-03-04T09:30:00.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.103+0000 7fb77a200640 1 -- 172.21.15.6:0/2639060241 shutdown_connections 2024-03-04T09:30:00.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.103+0000 7fb77a200640 1 --2- 172.21.15.6:0/2639060241 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb77415af80 0x7fb7740c9f30 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:00.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.103+0000 7fb77a200640 1 --2- 172.21.15.6:0/2639060241 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb75406b4d0 0x7fb75406d990 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:00.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.103+0000 7fb77a200640 1 --2- 172.21.15.6:0/2639060241 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb77415ab00 0x7fb7740c99f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:00.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.103+0000 7fb77a200640 1 --2- 172.21.15.6:0/2639060241 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb7580051c0 0x7fb7740cbac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:00.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.103+0000 7fb77a200640 1 -- 172.21.15.6:0/2639060241 >> 172.21.15.6:0/2639060241 conn(0x7fb77406b3d0 msgr2=0x7fb7740ad8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:00.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.103+0000 7fb77a200640 1 -- 172.21.15.6:0/2639060241 shutdown_connections 2024-03-04T09:30:00.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.103+0000 7fb77a200640 1 -- 172.21.15.6:0/2639060241 wait complete. 2024-03-04T09:30:00.127 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-04T09:30:00.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.342+0000 7f4340a00640 1 Processor -- start 2024-03-04T09:30:00.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.342+0000 7f4340a00640 1 -- start start 2024-03-04T09:30:00.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.344+0000 7f4340a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f433c15bf20 0x7f433c0acd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:00.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.344+0000 7f4340a00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f433c071e20 0x7f433c0ccad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:00.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.344+0000 7f4340a00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f433c072df0 con 0x7f433c072520 2024-03-04T09:30:00.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.344+0000 7f4340a00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f433c0cdab0 con 0x7f433c15bf20 2024-03-04T09:30:00.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.344+0000 7f4340a00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f433c0b4d70 con 0x7f433c071e20 2024-03-04T09:30:00.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.345+0000 7f4339e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f433c15bf20 0x7f433c0acd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:00.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.345+0000 7f4339e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f433c15bf20 0x7f433c0acd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:34070/0 (socket says 172.21.15.6:34070) 2024-03-04T09:30:00.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.345+0000 7f4339e00640 1 -- 172.21.15.6:0/122270564 learned_addr learned my addr 172.21.15.6:0/122270564 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:30:00.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.345+0000 7f433a800640 1 --2- 172.21.15.6:0/122270564 >> v2:172.21.15.146:3300/0 conn(0x7f433c071e20 0x7f433c0ccad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:00.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.345+0000 7f4339e00640 1 -- 172.21.15.6:0/122270564 >> v2:172.21.15.146:3300/0 conn(0x7f433c071e20 msgr2=0x7f433c0ccad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:00.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.345+0000 7f4339e00640 1 --2- 172.21.15.6:0/122270564 >> v2:172.21.15.146:3300/0 conn(0x7f433c071e20 0x7f433c0ccad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:00.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.345+0000 7f4339e00640 1 -- 172.21.15.6:0/122270564 >> v1:172.21.15.146:6789/0 conn(0x7f433c072520 legacy=0x7f433c0cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:00.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.345+0000 7f4339e00640 1 -- 172.21.15.6:0/122270564 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f433c0b5050 con 0x7f433c15bf20 2024-03-04T09:30:00.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.345+0000 7f433a800640 1 --2- 172.21.15.6:0/122270564 >> v2:172.21.15.146:3300/0 conn(0x7f433c071e20 0x7f433c0ccad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:30:00.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.346+0000 7f4339e00640 1 --2- 172.21.15.6:0/122270564 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f433c15bf20 0x7f433c0acd40 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f433c0af200 tx=0x7f432c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3bd303c6461fe9c server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:00.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.347+0000 7f4339400640 1 -- 172.21.15.6:0/122270564 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f432c02ca80 con 0x7f433c15bf20 2024-03-04T09:30:00.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.347+0000 7f4339400640 1 -- 172.21.15.6:0/122270564 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f432c02cc20 con 0x7f433c15bf20 2024-03-04T09:30:00.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.347+0000 7f4339400640 1 -- 172.21.15.6:0/122270564 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f432c0343f0 con 0x7f433c15bf20 2024-03-04T09:30:00.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.347+0000 7f4340a00640 1 -- 172.21.15.6:0/122270564 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f433c15bf20 msgr2=0x7f433c0acd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:00.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.347+0000 7f4340a00640 1 --2- 172.21.15.6:0/122270564 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f433c15bf20 0x7f433c0acd40 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f433c0af200 tx=0x7f432c02bee0 comp rx=0 tx=0).stop 2024-03-04T09:30:00.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.347+0000 7f4340a00640 1 -- 172.21.15.6:0/122270564 shutdown_connections 2024-03-04T09:30:00.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.347+0000 7f4340a00640 1 --2- 172.21.15.6:0/122270564 >> v2:172.21.15.146:3300/0 conn(0x7f433c071e20 0x7f433c0ccad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:00.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.347+0000 7f4340a00640 1 --2- 172.21.15.6:0/122270564 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f433c15bf20 0x7f433c0acd40 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:00.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.347+0000 7f4340a00640 1 -- 172.21.15.6:0/122270564 >> 172.21.15.6:0/122270564 conn(0x7f433c06b3d0 msgr2=0x7f433c06b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:00.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.347+0000 7f4340a00640 1 -- 172.21.15.6:0/122270564 shutdown_connections 2024-03-04T09:30:00.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.348+0000 7f4340a00640 1 -- 172.21.15.6:0/122270564 wait complete. 2024-03-04T09:30:00.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.349+0000 7f4340a00640 1 Processor -- start 2024-03-04T09:30:00.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.349+0000 7f4340a00640 1 -- start start 2024-03-04T09:30:00.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.350+0000 7f4340a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f433c071e20 0x7f433c1f51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:00.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.350+0000 7f4340a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f433c072520 0x7f433c1f3fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:00.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.350+0000 7f4340a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f433c15bf20 0x7f433c15a170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:00.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.350+0000 7f4340a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f433c0b4b10 con 0x7f433c071e20 2024-03-04T09:30:00.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.350+0000 7f4340a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f433c0b4990 con 0x7f433c072520 2024-03-04T09:30:00.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.350+0000 7f4340a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f433c0ce6c0 con 0x7f433c15bf20 2024-03-04T09:30:00.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.351+0000 7f4339e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f433c071e20 0x7f433c1f51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:00.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.351+0000 7f4339e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f433c071e20 0x7f433c1f51c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:34076/0 (socket says 172.21.15.6:34076) 2024-03-04T09:30:00.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.351+0000 7f4339e00640 1 -- 172.21.15.6:0/3874495793 learned_addr learned my addr 172.21.15.6:0/3874495793 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:30:00.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.351+0000 7f4331400640 1 --2- 172.21.15.6:0/3874495793 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f433c072520 0x7f433c1f3fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:00.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.351+0000 7f4339e00640 1 -- 172.21.15.6:0/3874495793 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f433c15bf20 msgr2=0x7f433c15a170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:00.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.351+0000 7f4339e00640 1 --2- 172.21.15.6:0/3874495793 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f433c15bf20 0x7f433c15a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:00.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.351+0000 7f4339e00640 1 -- 172.21.15.6:0/3874495793 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f433c072520 msgr2=0x7f433c1f3fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:00.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.351+0000 7f4339e00640 1 --2- 172.21.15.6:0/3874495793 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f433c072520 0x7f433c1f3fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:00.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.351+0000 7f4339e00640 1 -- 172.21.15.6:0/3874495793 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f433c1f5700 con 0x7f433c071e20 2024-03-04T09:30:00.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.351+0000 7f4331400640 1 --2- 172.21.15.6:0/3874495793 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f433c072520 0x7f433c1f3fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:30:00.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.352+0000 7f4339e00640 1 --2- 172.21.15.6:0/3874495793 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f433c071e20 0x7f433c1f51c0 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f432c000f80 tx=0x7f432c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:00.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.352+0000 7f4332a00640 1 -- 172.21.15.6:0/3874495793 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f432c043020 con 0x7f433c071e20 2024-03-04T09:30:00.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.352+0000 7f4332a00640 1 -- 172.21.15.6:0/3874495793 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f432c03e450 con 0x7f433c071e20 2024-03-04T09:30:00.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.352+0000 7f4332a00640 1 -- 172.21.15.6:0/3874495793 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f432c03e840 con 0x7f433c071e20 2024-03-04T09:30:00.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.352+0000 7f4340a00640 1 -- 172.21.15.6:0/3874495793 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f433c15e070 con 0x7f433c071e20 2024-03-04T09:30:00.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.352+0000 7f4340a00640 1 -- 172.21.15.6:0/3874495793 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f433c15e630 con 0x7f433c071e20 2024-03-04T09:30:00.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.354+0000 7f4340a00640 1 -- 172.21.15.6:0/3874495793 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f433c04f330 con 0x7f433c071e20 2024-03-04T09:30:00.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.354+0000 7f4332a00640 1 -- 172.21.15.6:0/3874495793 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f432c046020 con 0x7f433c071e20 2024-03-04T09:30:00.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.355+0000 7f4332a00640 1 --2- 172.21.15.6:0/3874495793 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f431806b4d0 0x7f431806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:00.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.355+0000 7f4332a00640 1 -- 172.21.15.6:0/3874495793 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f432c0acb60 con 0x7f433c071e20 2024-03-04T09:30:00.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.356+0000 7f4331400640 1 --2- 172.21.15.6:0/3874495793 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f431806b4d0 0x7f431806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:00.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.357+0000 7f4331400640 1 --2- 172.21.15.6:0/3874495793 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f431806b4d0 0x7f431806d990 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f433c09cb80 tx=0x7f4324008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:00.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.360+0000 7f4332a00640 1 -- 172.21.15.6:0/3874495793 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f432c04d030 con 0x7f433c071e20 2024-03-04T09:30:00.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.707+0000 7f4340a00640 1 -- 172.21.15.6:0/3874495793 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f433c15f720 con 0x7f433c071e20 2024-03-04T09:30:00.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.708+0000 7f4332a00640 1 -- 172.21.15.6:0/3874495793 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v81) v1 ==== 100+0+11 (secure 0 0 0) 0x7f433c1f5700 con 0x7f433c071e20 2024-03-04T09:30:00.705 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 64 2024-03-04T09:30:00.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.710+0000 7f4340a00640 1 -- 172.21.15.6:0/3874495793 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f431806b4d0 msgr2=0x7f431806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:00.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.710+0000 7f4340a00640 1 --2- 172.21.15.6:0/3874495793 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f431806b4d0 0x7f431806d990 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f433c09cb80 tx=0x7f4324008040 comp rx=0 tx=0).stop 2024-03-04T09:30:00.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.710+0000 7f4340a00640 1 -- 172.21.15.6:0/3874495793 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f433c071e20 msgr2=0x7f433c1f51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:00.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.710+0000 7f4340a00640 1 --2- 172.21.15.6:0/3874495793 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f433c071e20 0x7f433c1f51c0 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f432c000f80 tx=0x7f432c004a40 comp rx=0 tx=0).stop 2024-03-04T09:30:00.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.710+0000 7f4340a00640 1 -- 172.21.15.6:0/3874495793 shutdown_connections 2024-03-04T09:30:00.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.710+0000 7f4340a00640 1 --2- 172.21.15.6:0/3874495793 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f431806b4d0 0x7f431806d990 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:00.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.710+0000 7f4340a00640 1 --2- 172.21.15.6:0/3874495793 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f433c15bf20 0x7f433c15a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:00.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.710+0000 7f4340a00640 1 --2- 172.21.15.6:0/3874495793 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f433c072520 0x7f433c1f3fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:00.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.710+0000 7f4340a00640 1 --2- 172.21.15.6:0/3874495793 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f433c071e20 0x7f433c1f51c0 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:00.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.710+0000 7f4340a00640 1 -- 172.21.15.6:0/3874495793 >> 172.21.15.6:0/3874495793 conn(0x7f433c06b3d0 msgr2=0x7f433c159120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:00.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.711+0000 7f4340a00640 1 -- 172.21.15.6:0/3874495793 shutdown_connections 2024-03-04T09:30:00.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.711+0000 7f4340a00640 1 -- 172.21.15.6:0/3874495793 wait complete. 2024-03-04T09:30:00.727 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-04T09:30:00.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.934+0000 7f0926e00640 1 Processor -- start 2024-03-04T09:30:00.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.934+0000 7f0926e00640 1 -- start start 2024-03-04T09:30:00.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.936+0000 7f0926e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f09200d4a10 0x7f09200d0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:00.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.936+0000 7f0926e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f09200d0c80 0x7f09200b44d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:00.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.936+0000 7f0926e00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09200b3710 con 0x7f092015e0b0 2024-03-04T09:30:00.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.936+0000 7f0926e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09200d10f0 con 0x7f09200d0c80 2024-03-04T09:30:00.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.936+0000 7f0926e00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09200b09e0 con 0x7f09200d4a10 2024-03-04T09:30:00.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.936+0000 7f0926400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f09200d0c80 0x7f09200b44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:00.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.936+0000 7f0926400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f09200d0c80 0x7f09200b44d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:34086/0 (socket says 172.21.15.6:34086) 2024-03-04T09:30:00.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.936+0000 7f0926400640 1 -- 172.21.15.6:0/3712514995 learned_addr learned my addr 172.21.15.6:0/3712514995 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:30:00.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.937+0000 7f0925000640 1 --2- 172.21.15.6:0/3712514995 >> v2:172.21.15.146:3300/0 conn(0x7f09200d4a10 0x7f09200d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:00.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.937+0000 7f0926400640 1 -- 172.21.15.6:0/3712514995 >> v2:172.21.15.146:3300/0 conn(0x7f09200d4a10 msgr2=0x7f09200d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:00.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.937+0000 7f0926400640 1 --2- 172.21.15.6:0/3712514995 >> v2:172.21.15.146:3300/0 conn(0x7f09200d4a10 0x7f09200d0740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:00.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.937+0000 7f0926400640 1 -- 172.21.15.6:0/3712514995 >> v1:172.21.15.146:6789/0 conn(0x7f092015e0b0 legacy=0x7f09200cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:00.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.937+0000 7f0926400640 1 -- 172.21.15.6:0/3712514995 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09200b39f0 con 0x7f09200d0c80 2024-03-04T09:30:00.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.937+0000 7f0925000640 1 --2- 172.21.15.6:0/3712514995 >> v2:172.21.15.146:3300/0 conn(0x7f09200d4a10 0x7f09200d0740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:30:00.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.938+0000 7f0926400640 1 --2- 172.21.15.6:0/3712514995 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f09200d0c80 0x7f09200b44d0 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f092008d6d0 tx=0x7f091802bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6207ac2fc355c107 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:00.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.938+0000 7f091fe00640 1 -- 172.21.15.6:0/3712514995 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f091802ca40 con 0x7f09200d0c80 2024-03-04T09:30:00.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.938+0000 7f091fe00640 1 -- 172.21.15.6:0/3712514995 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f091802cbe0 con 0x7f09200d0c80 2024-03-04T09:30:00.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.938+0000 7f091fe00640 1 -- 172.21.15.6:0/3712514995 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09180343f0 con 0x7f09200d0c80 2024-03-04T09:30:00.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.939+0000 7f0926e00640 1 -- 172.21.15.6:0/3712514995 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f09200d0c80 msgr2=0x7f09200b44d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:00.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.939+0000 7f0926e00640 1 --2- 172.21.15.6:0/3712514995 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f09200d0c80 0x7f09200b44d0 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f092008d6d0 tx=0x7f091802bea0 comp rx=0 tx=0).stop 2024-03-04T09:30:00.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.939+0000 7f0926e00640 1 -- 172.21.15.6:0/3712514995 shutdown_connections 2024-03-04T09:30:00.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.939+0000 7f0926e00640 1 --2- 172.21.15.6:0/3712514995 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f09200d0c80 0x7f09200b44d0 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:00.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.939+0000 7f0926e00640 1 --2- 172.21.15.6:0/3712514995 >> v2:172.21.15.146:3300/0 conn(0x7f09200d4a10 0x7f09200d0740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:00.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.939+0000 7f0926e00640 1 -- 172.21.15.6:0/3712514995 >> 172.21.15.6:0/3712514995 conn(0x7f09200d3d30 msgr2=0x7f09200d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:00.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.939+0000 7f0926e00640 1 -- 172.21.15.6:0/3712514995 shutdown_connections 2024-03-04T09:30:00.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.939+0000 7f0926e00640 1 -- 172.21.15.6:0/3712514995 wait complete. 2024-03-04T09:30:00.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.941+0000 7f0926e00640 1 Processor -- start 2024-03-04T09:30:00.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.941+0000 7f0926e00640 1 -- start start 2024-03-04T09:30:00.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.942+0000 7f0926e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f09200d0c80 0x7f09200b8f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:00.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.942+0000 7f0926e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f09200d4a10 0x7f092015f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:00.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.942+0000 7f0926e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f092015e0b0 0x7f09201f3030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:00.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.942+0000 7f0926e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09200b3fb0 con 0x7f092015e0b0 2024-03-04T09:30:00.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.942+0000 7f0926e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09200cfa20 con 0x7f09200d4a10 2024-03-04T09:30:00.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.942+0000 7f0926e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f092015e490 con 0x7f09200d0c80 2024-03-04T09:30:00.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.942+0000 7f0926400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f092015e0b0 0x7f09201f3030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:00.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.942+0000 7f0926400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f092015e0b0 0x7f09201f3030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:34100/0 (socket says 172.21.15.6:34100) 2024-03-04T09:30:00.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.943+0000 7f0926400640 1 -- 172.21.15.6:0/1420839429 learned_addr learned my addr 172.21.15.6:0/1420839429 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:30:00.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.943+0000 7f0925a00640 1 --2- 172.21.15.6:0/1420839429 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f09200d0c80 0x7f09200b8f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:00.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.943+0000 7f0925000640 1 --2- 172.21.15.6:0/1420839429 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f09200d4a10 0x7f092015f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:00.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.943+0000 7f0926400640 1 -- 172.21.15.6:0/1420839429 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f09200d0c80 msgr2=0x7f09200b8f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:00.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.943+0000 7f0926400640 1 --2- 172.21.15.6:0/1420839429 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f09200d0c80 0x7f09200b8f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:00.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.943+0000 7f0926400640 1 -- 172.21.15.6:0/1420839429 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f09200d4a10 msgr2=0x7f092015f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:00.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.943+0000 7f0926400640 1 --2- 172.21.15.6:0/1420839429 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f09200d4a10 0x7f092015f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:00.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.943+0000 7f0926400640 1 -- 172.21.15.6:0/1420839429 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f092015f8d0 con 0x7f092015e0b0 2024-03-04T09:30:00.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.943+0000 7f0925000640 1 --2- 172.21.15.6:0/1420839429 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f09200d4a10 0x7f092015f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:30:00.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.943+0000 7f0926400640 1 --2- 172.21.15.6:0/1420839429 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f092015e0b0 0x7f09201f3030 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f0918036040 tx=0x7f0918004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:00.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.944+0000 7f091d600640 1 -- 172.21.15.6:0/1420839429 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0918043020 con 0x7f092015e0b0 2024-03-04T09:30:00.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.944+0000 7f091d600640 1 -- 172.21.15.6:0/1420839429 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f091803e450 con 0x7f092015e0b0 2024-03-04T09:30:00.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.944+0000 7f0926e00640 1 -- 172.21.15.6:0/1420839429 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f092015fbb0 con 0x7f092015e0b0 2024-03-04T09:30:00.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.944+0000 7f091d600640 1 -- 172.21.15.6:0/1420839429 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f091803e860 con 0x7f092015e0b0 2024-03-04T09:30:00.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.944+0000 7f0926e00640 1 -- 172.21.15.6:0/1420839429 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0920160170 con 0x7f092015e0b0 2024-03-04T09:30:00.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.945+0000 7f0926e00640 1 -- 172.21.15.6:0/1420839429 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f092004f330 con 0x7f092015e0b0 2024-03-04T09:30:00.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.946+0000 7f091d600640 1 -- 172.21.15.6:0/1420839429 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0918046020 con 0x7f092015e0b0 2024-03-04T09:30:00.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.947+0000 7f091d600640 1 --2- 172.21.15.6:0/1420839429 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f090006b500 0x7f090006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:00.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.947+0000 7f091d600640 1 -- 172.21.15.6:0/1420839429 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f09180ac8f0 con 0x7f092015e0b0 2024-03-04T09:30:00.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.947+0000 7f0925a00640 1 --2- 172.21.15.6:0/1420839429 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f090006b500 0x7f090006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:00.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.948+0000 7f0925a00640 1 --2- 172.21.15.6:0/1420839429 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f090006b500 0x7f090006d9c0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f0910004ea0 tx=0x7f0910009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:00.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:00.951+0000 7f091d600640 1 -- 172.21.15.6:0/1420839429 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f091804d070 con 0x7f092015e0b0 2024-03-04T09:30:01.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:01.166+0000 7f0926e00640 1 -- 172.21.15.6:0/1420839429 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f092015f5c0 con 0x7f092015e0b0 2024-03-04T09:30:01.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:01.167+0000 7f091d600640 1 -- 172.21.15.6:0/1420839429 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v81) v1 ==== 80+0+0 (secure 0 0 0) 0x7f091807b6d0 con 0x7f092015e0b0 2024-03-04T09:30:01.164 INFO:teuthology.orchestra.run.smithi006.stderr:listed 0 entries 2024-03-04T09:30:01.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:01.169+0000 7f0926e00640 1 -- 172.21.15.6:0/1420839429 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f090006b500 msgr2=0x7f090006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:01.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:01.169+0000 7f0926e00640 1 --2- 172.21.15.6:0/1420839429 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f090006b500 0x7f090006d9c0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f0910004ea0 tx=0x7f0910009040 comp rx=0 tx=0).stop 2024-03-04T09:30:01.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:01.169+0000 7f0926e00640 1 -- 172.21.15.6:0/1420839429 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f092015e0b0 msgr2=0x7f09201f3030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:01.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:01.169+0000 7f0926e00640 1 --2- 172.21.15.6:0/1420839429 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f092015e0b0 0x7f09201f3030 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f0918036040 tx=0x7f0918004a40 comp rx=0 tx=0).stop 2024-03-04T09:30:01.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:01.169+0000 7f0926e00640 1 -- 172.21.15.6:0/1420839429 shutdown_connections 2024-03-04T09:30:01.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:01.169+0000 7f0926e00640 1 --2- 172.21.15.6:0/1420839429 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f092015e0b0 0x7f09201f3030 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:01.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:01.169+0000 7f0926e00640 1 --2- 172.21.15.6:0/1420839429 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f09200d4a10 0x7f092015f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:01.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:01.169+0000 7f0926e00640 1 --2- 172.21.15.6:0/1420839429 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f090006b500 0x7f090006d9c0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:01.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:01.169+0000 7f0926e00640 1 --2- 172.21.15.6:0/1420839429 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f09200d0c80 0x7f09200b8f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:01.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:01.169+0000 7f0926e00640 1 -- 172.21.15.6:0/1420839429 >> 172.21.15.6:0/1420839429 conn(0x7f09200d3d30 msgr2=0x7f09200b1f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:01.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:01.169+0000 7f0926e00640 1 -- 172.21.15.6:0/1420839429 shutdown_connections 2024-03-04T09:30:01.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:01.169+0000 7f0926e00640 1 -- 172.21.15.6:0/1420839429 wait complete. 2024-03-04T09:30:01.183 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-04T09:30:01.183 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:30:01.183 DEBUG:teuthology.orchestra.run.smithi006:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-04T09:30:01.274 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi006.front.sepia.ceph.com 2024-03-04T09:30:01.274 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:30:01.274 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-04T09:30:01.317 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-04T09:30:01.317 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T09:30:01.374 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053429,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249180k,nr_inodes=812295,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-04T09:30:01.375 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-04T09:30:01.375 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T09:30:01.428 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054202,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249800k,nr_inodes=812450,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-04T09:30:01.428 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-04T09:30:01.429 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T09:30:01.486 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053429,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249180k,nr_inodes=812295,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-04T09:30:01.487 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:30:01.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:01.713+0000 7f17f7400640 1 Processor -- start 2024-03-04T09:30:01.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:01.713+0000 7f17f7400640 1 -- start start 2024-03-04T09:30:01.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:01.715+0000 7f17f7400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f17f0161b40 0x7f17f00cfd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:01.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:01.715+0000 7f17f7400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f17f00d2930 0x7f17f00b3740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:01.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:01.715+0000 7f17f7400640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17f00cea50 con 0x7f17f00b5060 2024-03-04T09:30:01.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:01.715+0000 7f17f7400640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17f00cf900 con 0x7f17f00d2930 2024-03-04T09:30:01.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:01.715+0000 7f17f7400640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17f00cecf0 con 0x7f17f0161b40 2024-03-04T09:30:01.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:01.715+0000 7f17f5400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f17f00d2930 0x7f17f00b3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:01.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:01.715+0000 7f17f5400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f17f00d2930 0x7f17f00b3740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:34110/0 (socket says 172.21.15.6:34110) 2024-03-04T09:30:01.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:01.715+0000 7f17f5400640 1 -- 172.21.15.6:0/3388005539 learned_addr learned my addr 172.21.15.6:0/3388005539 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:30:01.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:01.716+0000 7f17f5400640 1 -- 172.21.15.6:0/3388005539 >> v2:172.21.15.146:3300/0 conn(0x7f17f0161b40 msgr2=0x7f17f00cfd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:01.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:01.716+0000 7f17f4a00640 1 --2- 172.21.15.6:0/3388005539 >> v2:172.21.15.146:3300/0 conn(0x7f17f0161b40 0x7f17f00cfd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:01.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:01.716+0000 7f17f5400640 1 --2- 172.21.15.6:0/3388005539 >> v2:172.21.15.146:3300/0 conn(0x7f17f0161b40 0x7f17f00cfd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:01.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:01.716+0000 7f17f5400640 1 -- 172.21.15.6:0/3388005539 >> v1:172.21.15.146:6789/0 conn(0x7f17f00b5060 legacy=0x7f17f00d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:01.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:01.716+0000 7f17f5400640 1 -- 172.21.15.6:0/3388005539 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17f00cefd0 con 0x7f17f00d2930 2024-03-04T09:30:01.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:01.716+0000 7f17f4a00640 1 --2- 172.21.15.6:0/3388005539 >> v2:172.21.15.146:3300/0 conn(0x7f17f0161b40 0x7f17f00cfd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:30:01.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:01.717+0000 7f17f5400640 1 --2- 172.21.15.6:0/3388005539 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f17f00d2930 0x7f17f00b3740 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f17f008db00 tx=0x7f17e802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6a498ac85b72fe6 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:01.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:01.717+0000 7f17ef400640 1 -- 172.21.15.6:0/3388005539 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17e802ca80 con 0x7f17f00d2930 2024-03-04T09:30:01.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:01.717+0000 7f17ef400640 1 -- 172.21.15.6:0/3388005539 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17e802cc20 con 0x7f17f00d2930 2024-03-04T09:30:01.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:01.717+0000 7f17ef400640 1 -- 172.21.15.6:0/3388005539 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17e80343f0 con 0x7f17f00d2930 2024-03-04T09:30:01.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:01.718+0000 7f17f7400640 1 -- 172.21.15.6:0/3388005539 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f17f00d2930 msgr2=0x7f17f00b3740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:01.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:01.718+0000 7f17f7400640 1 --2- 172.21.15.6:0/3388005539 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f17f00d2930 0x7f17f00b3740 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f17f008db00 tx=0x7f17e802bee0 comp rx=0 tx=0).stop 2024-03-04T09:30:01.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:01.718+0000 7f17f7400640 1 -- 172.21.15.6:0/3388005539 shutdown_connections 2024-03-04T09:30:01.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:01.718+0000 7f17f7400640 1 --2- 172.21.15.6:0/3388005539 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f17f00d2930 0x7f17f00b3740 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:01.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:01.718+0000 7f17f7400640 1 --2- 172.21.15.6:0/3388005539 >> v2:172.21.15.146:3300/0 conn(0x7f17f0161b40 0x7f17f00cfd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:01.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:01.718+0000 7f17f7400640 1 -- 172.21.15.6:0/3388005539 >> 172.21.15.6:0/3388005539 conn(0x7f17f009d220 msgr2=0x7f17f00d3ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:01.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:01.719+0000 7f17f7400640 1 -- 172.21.15.6:0/3388005539 shutdown_connections 2024-03-04T09:30:01.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:01.719+0000 7f17f7400640 1 -- 172.21.15.6:0/3388005539 wait complete. 2024-03-04T09:30:01.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:01.720+0000 7f17f7400640 1 Processor -- start 2024-03-04T09:30:01.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:01.720+0000 7f17f7400640 1 -- start start 2024-03-04T09:30:01.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:01.721+0000 7f17f7400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f17f00b5060 0x7f17f00c8410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:01.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:01.721+0000 7f17f7400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f17f00d2930 0x7f17f0152300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:01.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:01.721+0000 7f17f7400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f17f0161b40 0x7f17f015fef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:01.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:01.721+0000 7f17f7400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17f00b2ff0 con 0x7f17f00d2930 2024-03-04T09:30:01.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:01.721+0000 7f17f7400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17f00cf590 con 0x7f17f0161b40 2024-03-04T09:30:01.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:01.721+0000 7f17f7400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17f00b3170 con 0x7f17f00b5060 2024-03-04T09:30:01.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:01.722+0000 7f17efe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f17f00d2930 0x7f17f0152300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:01.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:01.722+0000 7f17efe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f17f00d2930 0x7f17f0152300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:34112/0 (socket says 172.21.15.6:34112) 2024-03-04T09:30:01.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:01.722+0000 7f17efe00640 1 -- 172.21.15.6:0/2970918146 learned_addr learned my addr 172.21.15.6:0/2970918146 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:30:01.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:01.722+0000 7f17f5400640 1 --2- 172.21.15.6:0/2970918146 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f17f0161b40 0x7f17f015fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:01.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:01.722+0000 7f17f4a00640 1 --2- 172.21.15.6:0/2970918146 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f17f00b5060 0x7f17f00c8410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:01.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:01.722+0000 7f17efe00640 1 -- 172.21.15.6:0/2970918146 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f17f00b5060 msgr2=0x7f17f00c8410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:01.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:01.722+0000 7f17efe00640 1 --2- 172.21.15.6:0/2970918146 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f17f00b5060 0x7f17f00c8410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:01.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:01.722+0000 7f17efe00640 1 -- 172.21.15.6:0/2970918146 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f17f0161b40 msgr2=0x7f17f015fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:01.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:01.722+0000 7f17efe00640 1 --2- 172.21.15.6:0/2970918146 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f17f0161b40 0x7f17f015fef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:01.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:01.722+0000 7f17efe00640 1 -- 172.21.15.6:0/2970918146 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17f00c9410 con 0x7f17f00d2930 2024-03-04T09:30:01.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:01.723+0000 7f17efe00640 1 --2- 172.21.15.6:0/2970918146 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f17f00d2930 0x7f17f0152300 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f17dc00b3f0 tx=0x7f17dc00b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:01.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:01.723+0000 7f17ecc00640 1 -- 172.21.15.6:0/2970918146 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17dc015020 con 0x7f17f00d2930 2024-03-04T09:30:01.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:01.723+0000 7f17ecc00640 1 -- 172.21.15.6:0/2970918146 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17dc011830 con 0x7f17f00d2930 2024-03-04T09:30:01.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:01.723+0000 7f17ecc00640 1 -- 172.21.15.6:0/2970918146 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17dc011c20 con 0x7f17f00d2930 2024-03-04T09:30:01.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:01.723+0000 7f17f7400640 1 -- 172.21.15.6:0/2970918146 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17f01f7a80 con 0x7f17f00d2930 2024-03-04T09:30:01.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:01.723+0000 7f17f7400640 1 -- 172.21.15.6:0/2970918146 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17f01f8040 con 0x7f17f00d2930 2024-03-04T09:30:01.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:01.725+0000 7f17f7400640 1 -- 172.21.15.6:0/2970918146 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17f00b9d40 con 0x7f17f00d2930 2024-03-04T09:30:01.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:01.726+0000 7f17ecc00640 1 -- 172.21.15.6:0/2970918146 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f17dc018020 con 0x7f17f00d2930 2024-03-04T09:30:01.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:01.727+0000 7f17ecc00640 1 --2- 172.21.15.6:0/2970918146 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f17d006b500 0x7f17d006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:01.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:01.727+0000 7f17ecc00640 1 -- 172.21.15.6:0/2970918146 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f17dc089e10 con 0x7f17f00d2930 2024-03-04T09:30:01.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:01.727+0000 7f17f4a00640 1 --2- 172.21.15.6:0/2970918146 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f17d006b500 0x7f17d006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:01.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:01.729+0000 7f17f4a00640 1 --2- 172.21.15.6:0/2970918146 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f17d006b500 0x7f17d006d9c0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f17f00c76d0 tx=0x7f17e4008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:01.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:01.733+0000 7f17ecc00640 1 -- 172.21.15.6:0/2970918146 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f17dc08d030 con 0x7f17f00d2930 2024-03-04T09:30:01.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:01.972+0000 7f17f7400640 1 -- 172.21.15.6:0/2970918146 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f17f00c95e0 con 0x7f17f00d2930 2024-03-04T09:30:01.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:01.972+0000 7f17ecc00640 1 -- 172.21.15.6:0/2970918146 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) v1 ==== 94+0+4976 (secure 0 0 0) 0x7f17dc01f030 con 0x7f17f00d2930 2024-03-04T09:30:01.970 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:30:01.970 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":51,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5827,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6836/3447344578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":3447344578},{"type":"v1","addr":"172.21.15.146:6836","nonce":3447344578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":46},{"gid":5830,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/168994142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":168994142},{"type":"v1","addr":"172.21.15.146:6837","nonce":168994142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":46},{"gid":6735,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/3369489999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":3369489999},{"type":"v1","addr":"172.21.15.6:6833","nonce":3369489999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":45}],"filesystems":[{"mdsmap":{"epoch":51,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:29:47.357677+0000","modified":"2024-03-04T09:29:50.579389+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6744},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6744":{"gid":6744,"name":"a","rank":0,"incarnation":47,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6835/1667002065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1667002065},{"type":"v1","addr":"172.21.15.6:6835","nonce":1667002065}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6744,"qdb_cluster":[6744]},"id":6}]} 2024-03-04T09:30:01.970 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 51 2024-03-04T09:30:01.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:01.975+0000 7f17f7400640 1 -- 172.21.15.6:0/2970918146 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f17d006b500 msgr2=0x7f17d006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:01.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:01.975+0000 7f17f7400640 1 --2- 172.21.15.6:0/2970918146 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f17d006b500 0x7f17d006d9c0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f17f00c76d0 tx=0x7f17e4008040 comp rx=0 tx=0).stop 2024-03-04T09:30:01.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:01.975+0000 7f17f7400640 1 -- 172.21.15.6:0/2970918146 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f17f00d2930 msgr2=0x7f17f0152300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:01.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:01.975+0000 7f17f7400640 1 --2- 172.21.15.6:0/2970918146 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f17f00d2930 0x7f17f0152300 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f17dc00b3f0 tx=0x7f17dc00b8b0 comp rx=0 tx=0).stop 2024-03-04T09:30:01.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:01.975+0000 7f17f7400640 1 -- 172.21.15.6:0/2970918146 shutdown_connections 2024-03-04T09:30:01.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:01.975+0000 7f17f7400640 1 --2- 172.21.15.6:0/2970918146 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f17f0161b40 0x7f17f015fef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:01.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:01.975+0000 7f17f7400640 1 --2- 172.21.15.6:0/2970918146 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f17f00d2930 0x7f17f0152300 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:01.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:01.975+0000 7f17f7400640 1 --2- 172.21.15.6:0/2970918146 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f17d006b500 0x7f17d006d9c0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:01.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:01.975+0000 7f17f7400640 1 --2- 172.21.15.6:0/2970918146 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f17f00b5060 0x7f17f00c8410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:01.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:01.975+0000 7f17f7400640 1 -- 172.21.15.6:0/2970918146 >> 172.21.15.6:0/2970918146 conn(0x7f17f009d220 msgr2=0x7f17f00b17b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:01.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:01.975+0000 7f17f7400640 1 -- 172.21.15.6:0/2970918146 shutdown_connections 2024-03-04T09:30:01.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:01.975+0000 7f17f7400640 1 -- 172.21.15.6:0/2970918146 wait complete. 2024-03-04T09:30:01.991 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T09:30:02.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.199+0000 7f1bfbe00640 1 Processor -- start 2024-03-04T09:30:02.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.199+0000 7f1bfbe00640 1 -- start start 2024-03-04T09:30:02.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.201+0000 7f1bfbe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f1bf40af1a0 0x7f1bf415c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:02.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.201+0000 7f1bfbe00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1bf40cdc20 0x7f1bf40b2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:02.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.201+0000 7f1bfbe00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bf415cc40 con 0x7f1bf40af580 2024-03-04T09:30:02.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.201+0000 7f1bfbe00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1bf415cf20 con 0x7f1bf40af1a0 2024-03-04T09:30:02.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.201+0000 7f1bfbe00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bf415d200 con 0x7f1bf40cdc20 2024-03-04T09:30:02.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.201+0000 7f1bf9400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f1bf40af1a0 0x7f1bf415c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:02.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.201+0000 7f1bf9400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f1bf40af1a0 0x7f1bf415c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:34120/0 (socket says 172.21.15.6:34120) 2024-03-04T09:30:02.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.201+0000 7f1bf9400640 1 -- 172.21.15.6:0/1543013943 learned_addr learned my addr 172.21.15.6:0/1543013943 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:30:02.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.201+0000 7f1bf8a00640 1 --1- 172.21.15.6:0/1543013943 >> v1:172.21.15.6:6789/0 conn(0x7f1bf40af580 0x7f1bf40b4c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:37196/0 (socket says 172.21.15.6:37196) 2024-03-04T09:30:02.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.202+0000 7f1bf9e00640 1 --2- 172.21.15.6:0/1543013943 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1bf40cdc20 0x7f1bf40b2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:02.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.202+0000 7f1bf9400640 1 -- 172.21.15.6:0/1543013943 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1bf40cdc20 msgr2=0x7f1bf40b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:02.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.202+0000 7f1bf9400640 1 --2- 172.21.15.6:0/1543013943 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1bf40cdc20 0x7f1bf40b2dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:02.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.202+0000 7f1bf9400640 1 -- 172.21.15.6:0/1543013943 >> v1:172.21.15.6:6789/0 conn(0x7f1bf40af580 legacy=0x7f1bf40b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:02.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.202+0000 7f1bf9400640 1 -- 172.21.15.6:0/1543013943 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bf415d4e0 con 0x7f1bf40af1a0 2024-03-04T09:30:02.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.203+0000 7f1bf9400640 1 --2- 172.21.15.6:0/1543013943 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1bf40af1a0 0x7f1bf415c600 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f1be8009870 tx=0x7f1be802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b13cb8094dd4ab9 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:02.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.203+0000 7f1bf3e00640 1 -- 172.21.15.6:0/1543013943 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1be8035020 con 0x7f1bf40af1a0 2024-03-04T09:30:02.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.203+0000 7f1bf3e00640 1 -- 172.21.15.6:0/1543013943 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1be8031640 con 0x7f1bf40af1a0 2024-03-04T09:30:02.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.203+0000 7f1bf3e00640 1 -- 172.21.15.6:0/1543013943 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1be802d040 con 0x7f1bf40af1a0 2024-03-04T09:30:02.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.204+0000 7f1bfbe00640 1 -- 172.21.15.6:0/1543013943 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1bf40af1a0 msgr2=0x7f1bf415c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:02.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.204+0000 7f1bfbe00640 1 --2- 172.21.15.6:0/1543013943 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1bf40af1a0 0x7f1bf415c600 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f1be8009870 tx=0x7f1be802b8f0 comp rx=0 tx=0).stop 2024-03-04T09:30:02.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.204+0000 7f1bfbe00640 1 -- 172.21.15.6:0/1543013943 shutdown_connections 2024-03-04T09:30:02.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.204+0000 7f1bfbe00640 1 --2- 172.21.15.6:0/1543013943 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1bf40cdc20 0x7f1bf40b2dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:02.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.204+0000 7f1bfbe00640 1 --2- 172.21.15.6:0/1543013943 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1bf40af1a0 0x7f1bf415c600 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:02.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.204+0000 7f1bfbe00640 1 -- 172.21.15.6:0/1543013943 >> 172.21.15.6:0/1543013943 conn(0x7f1bf406afb0 msgr2=0x7f1bf406b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:02.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.204+0000 7f1bfbe00640 1 -- 172.21.15.6:0/1543013943 shutdown_connections 2024-03-04T09:30:02.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.204+0000 7f1bfbe00640 1 -- 172.21.15.6:0/1543013943 wait complete. 2024-03-04T09:30:02.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.206+0000 7f1bfbe00640 1 Processor -- start 2024-03-04T09:30:02.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.206+0000 7f1bfbe00640 1 -- start start 2024-03-04T09:30:02.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.207+0000 7f1bfbe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1bf40af1a0 0x7f1bf4153810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:02.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.207+0000 7f1bfbe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1bf40af580 0x7f1bf415e750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:02.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.207+0000 7f1bfbe00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1bf40cdc20 0x7f1bf415ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:02.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.207+0000 7f1bfbe00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bf40b3720 con 0x7f1bf40af1a0 2024-03-04T09:30:02.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.207+0000 7f1bfbe00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bf40b35a0 con 0x7f1bf40af580 2024-03-04T09:30:02.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.207+0000 7f1bfbe00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bf4155e70 con 0x7f1bf40cdc20 2024-03-04T09:30:02.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.207+0000 7f1bf9400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1bf40af1a0 0x7f1bf4153810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:02.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.207+0000 7f1bf9400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1bf40af1a0 0x7f1bf4153810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:34130/0 (socket says 172.21.15.6:34130) 2024-03-04T09:30:02.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.207+0000 7f1bf9400640 1 -- 172.21.15.6:0/3857607154 learned_addr learned my addr 172.21.15.6:0/3857607154 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:30:02.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.208+0000 7f1bf9400640 1 -- 172.21.15.6:0/3857607154 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1bf40cdc20 msgr2=0x7f1bf415ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:02.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.208+0000 7f1bf9400640 1 --2- 172.21.15.6:0/3857607154 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1bf40cdc20 0x7f1bf415ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:02.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.208+0000 7f1bf9400640 1 -- 172.21.15.6:0/3857607154 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1bf40af580 msgr2=0x7f1bf415e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:02.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.208+0000 7f1bf8a00640 1 --2- 172.21.15.6:0/3857607154 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1bf40af580 0x7f1bf415e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:02.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.208+0000 7f1bf9400640 1 --2- 172.21.15.6:0/3857607154 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1bf40af580 0x7f1bf415e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:02.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.208+0000 7f1bf9400640 1 -- 172.21.15.6:0/3857607154 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bf409d7c0 con 0x7f1bf40af1a0 2024-03-04T09:30:02.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.208+0000 7f1bf8a00640 1 --2- 172.21.15.6:0/3857607154 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1bf40af580 0x7f1bf415e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:30:02.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.208+0000 7f1bf9400640 1 --2- 172.21.15.6:0/3857607154 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1bf40af1a0 0x7f1bf4153810 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f1be8000c00 tx=0x7f1be80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:02.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.208+0000 7f1bf1600640 1 -- 172.21.15.6:0/3857607154 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1be8040070 con 0x7f1bf40af1a0 2024-03-04T09:30:02.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.208+0000 7f1bf1600640 1 -- 172.21.15.6:0/3857607154 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1be8039bf0 con 0x7f1bf40af1a0 2024-03-04T09:30:02.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.208+0000 7f1bf1600640 1 -- 172.21.15.6:0/3857607154 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1be8035040 con 0x7f1bf40af1a0 2024-03-04T09:30:02.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.208+0000 7f1bfbe00640 1 -- 172.21.15.6:0/3857607154 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bf415a860 con 0x7f1bf40af1a0 2024-03-04T09:30:02.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.209+0000 7f1bfbe00640 1 -- 172.21.15.6:0/3857607154 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bf415ae20 con 0x7f1bf40af1a0 2024-03-04T09:30:02.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.210+0000 7f1bfbe00640 1 -- 172.21.15.6:0/3857607154 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bf4069510 con 0x7f1bf40af1a0 2024-03-04T09:30:02.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.211+0000 7f1bf1600640 1 -- 172.21.15.6:0/3857607154 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1be804b060 con 0x7f1bf40af1a0 2024-03-04T09:30:02.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.212+0000 7f1bf1600640 1 --2- 172.21.15.6:0/3857607154 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f1bd406b520 0x7f1bd406d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:02.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.212+0000 7f1bf1600640 1 -- 172.21.15.6:0/3857607154 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f1be80abd60 con 0x7f1bf40af1a0 2024-03-04T09:30:02.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.212+0000 7f1bf8a00640 1 --2- 172.21.15.6:0/3857607154 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f1bd406b520 0x7f1bd406d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:02.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.214+0000 7f1bf8a00640 1 --2- 172.21.15.6:0/3857607154 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f1bd406b520 0x7f1bd406d9e0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f1bf406e230 tx=0x7f1be0009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:02.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.217+0000 7f1bf1600640 1 -- 172.21.15.6:0/3857607154 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1be8031610 con 0x7f1bf40af1a0 2024-03-04T09:30:02.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.414+0000 7f1bfbe00640 1 -- 172.21.15.6:0/3857607154 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1bf404f3a0 con 0x7f1bf40af1a0 2024-03-04T09:30:02.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.417+0000 7f1bf1600640 1 -- 172.21.15.6:0/3857607154 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) v1 ==== 74+0+16662 (secure 0 0 0) 0x7f1be807aba0 con 0x7f1bf40af1a0 2024-03-04T09:30:02.414 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:30:02.414 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":82,"fsid":"edce0578-5ae4-4402-b45a-552bdef84d7c","created":"2024-03-04T09:24:54.523780+0000","modified":"2024-03-04T09:30:01.996269+0000","last_up_change":"2024-03-04T09:25:07.089049+0000","last_in_change":"2024-03-04T09:24:57.982077+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T09:25:07.935198+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-03-04T09:29:45.280632+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-03-04T09:29:46.378588+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a96574f-81f1-4bc6-8463-757520b84a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6825","nonce":1364449401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6827","nonce":1364449401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6831","nonce":1364449401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6829","nonce":1364449401}]},"public_addr":"172.21.15.6:6825/1364449401","cluster_addr":"172.21.15.6:6827/1364449401","heartbeat_back_addr":"172.21.15.6:6831/1364449401","heartbeat_front_addr":"172.21.15.6:6829/1364449401","state":["exists","up"]},{"osd":1,"uuid":"7d79b9c1-6b87-476a-bc36-7db5ee64cd90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6809","nonce":2494103078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6811","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6813","nonce":2494103078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6821","nonce":2494103078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6817","nonce":2494103078}]},"public_addr":"172.21.15.6:6809/2494103078","cluster_addr":"172.21.15.6:6813/2494103078","heartbeat_back_addr":"172.21.15.6:6821/2494103078","heartbeat_front_addr":"172.21.15.6:6817/2494103078","state":["exists","up"]},{"osd":2,"uuid":"492331b9-4ee9-40b6-bacd-85d44cd6c45a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6812","nonce":1500066681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6816","nonce":1500066681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6824","nonce":1500066681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6820","nonce":1500066681}]},"public_addr":"172.21.15.6:6812/1500066681","cluster_addr":"172.21.15.6:6816/1500066681","heartbeat_back_addr":"172.21.15.6:6824/1500066681","heartbeat_front_addr":"172.21.15.6:6820/1500066681","state":["exists","up"]},{"osd":3,"uuid":"21bb2efe-9ba7-4bdc-bc31-180cd1102e21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6801","nonce":1558806123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6803","nonce":1558806123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6807","nonce":1558806123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6805","nonce":1558806123}]},"public_addr":"172.21.15.6:6801/1558806123","cluster_addr":"172.21.15.6:6803/1558806123","heartbeat_back_addr":"172.21.15.6:6807/1558806123","heartbeat_front_addr":"172.21.15.6:6805/1558806123","state":["exists","up"]},{"osd":4,"uuid":"6e2c4b71-19d9-4737-83d9-558705d8644a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6801","nonce":1526135780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6803","nonce":1526135780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6807","nonce":1526135780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6805","nonce":1526135780}]},"public_addr":"172.21.15.146:6801/1526135780","cluster_addr":"172.21.15.146:6803/1526135780","heartbeat_back_addr":"172.21.15.146:6807/1526135780","heartbeat_front_addr":"172.21.15.146:6805/1526135780","state":["exists","up"]},{"osd":5,"uuid":"fb69c525-f1be-4b7d-ab32-6c29b0d6fddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6809","nonce":460262507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6811","nonce":460262507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6815","nonce":460262507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6813","nonce":460262507}]},"public_addr":"172.21.15.146:6809/460262507","cluster_addr":"172.21.15.146:6811/460262507","heartbeat_back_addr":"172.21.15.146:6815/460262507","heartbeat_front_addr":"172.21.15.146:6813/460262507","state":["exists","up"]},{"osd":6,"uuid":"c32e7fed-370d-4401-8ce0-6a4335b0d168","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6823","nonce":1392897493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6827","nonce":1392897493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6831","nonce":1392897493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6829","nonce":1392897493}]},"public_addr":"172.21.15.146:6823/1392897493","cluster_addr":"172.21.15.146:6827/1392897493","heartbeat_back_addr":"172.21.15.146:6831/1392897493","heartbeat_front_addr":"172.21.15.146:6829/1392897493","state":["exists","up"]},{"osd":7,"uuid":"75a79b44-fa5f-426d-a7d3-71f21f78835c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6817","nonce":3268323156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6819","nonce":3268323156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6825","nonce":3268323156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6822","nonce":3268323156}]},"public_addr":"172.21.15.146:6817/3268323156","cluster_addr":"172.21.15.146:6819/3268323156","heartbeat_back_addr":"172.21.15.146:6825/3268323156","heartbeat_front_addr":"172.21.15.146:6822/3268323156","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.310437+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.295850+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.294674+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.526855+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.300023+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.595160+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.346046+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.558741+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"13.10","mappings":[{"from":1,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T09:30:02.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.419+0000 7f1bfbe00640 1 -- 172.21.15.6:0/3857607154 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f1bd406b520 msgr2=0x7f1bd406d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:02.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.419+0000 7f1bfbe00640 1 --2- 172.21.15.6:0/3857607154 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f1bd406b520 0x7f1bd406d9e0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f1bf406e230 tx=0x7f1be0009040 comp rx=0 tx=0).stop 2024-03-04T09:30:02.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.419+0000 7f1bfbe00640 1 -- 172.21.15.6:0/3857607154 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1bf40af1a0 msgr2=0x7f1bf4153810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:02.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.419+0000 7f1bfbe00640 1 --2- 172.21.15.6:0/3857607154 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1bf40af1a0 0x7f1bf4153810 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f1be8000c00 tx=0x7f1be80046c0 comp rx=0 tx=0).stop 2024-03-04T09:30:02.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.419+0000 7f1bfbe00640 1 -- 172.21.15.6:0/3857607154 shutdown_connections 2024-03-04T09:30:02.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.419+0000 7f1bfbe00640 1 --2- 172.21.15.6:0/3857607154 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f1bd406b520 0x7f1bd406d9e0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:02.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.419+0000 7f1bfbe00640 1 --2- 172.21.15.6:0/3857607154 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1bf40cdc20 0x7f1bf415ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:02.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.419+0000 7f1bfbe00640 1 --2- 172.21.15.6:0/3857607154 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1bf40af580 0x7f1bf415e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:02.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.419+0000 7f1bfbe00640 1 --2- 172.21.15.6:0/3857607154 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1bf40af1a0 0x7f1bf4153810 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:02.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.419+0000 7f1bfbe00640 1 -- 172.21.15.6:0/3857607154 >> 172.21.15.6:0/3857607154 conn(0x7f1bf406afb0 msgr2=0x7f1bf41556d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:02.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.419+0000 7f1bfbe00640 1 -- 172.21.15.6:0/3857607154 shutdown_connections 2024-03-04T09:30:02.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.419+0000 7f1bfbe00640 1 -- 172.21.15.6:0/3857607154 wait complete. 2024-03-04T09:30:02.435 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T09:25:07.935198+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-04T09:29:45.280632+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '80', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_data', 'create_time': '2024-03-04T09:29:46.378588+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-04T09:30:02.436 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-04T09:30:02.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.650+0000 7f5913e00640 1 Processor -- start 2024-03-04T09:30:02.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.650+0000 7f5913e00640 1 -- start start 2024-03-04T09:30:02.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.652+0000 7f5913e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f591415dee0 0x7f59140d18a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:02.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.652+0000 7f5913e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f59140d4860 0x7f59140d0630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:02.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.652+0000 7f5913e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59140ceca0 con 0x7f59140cfa60 2024-03-04T09:30:02.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.652+0000 7f5913e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59140cf5d0 con 0x7f591415dee0 2024-03-04T09:30:02.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.652+0000 7f5913e00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59140cef80 con 0x7f59140d4860 2024-03-04T09:30:02.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.652+0000 7f5912a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f591415dee0 0x7f59140d18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:02.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.652+0000 7f5912a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f591415dee0 0x7f59140d18a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:34134/0 (socket says 172.21.15.6:34134) 2024-03-04T09:30:02.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.652+0000 7f5912a00640 1 -- 172.21.15.6:0/2467094706 learned_addr learned my addr 172.21.15.6:0/2467094706 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:30:02.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.652+0000 7f5913400640 1 --1- 172.21.15.6:0/2467094706 >> v1:172.21.15.6:6789/0 conn(0x7f59140cfa60 0x7f59140b3510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:37206/0 (socket says 172.21.15.6:37206) 2024-03-04T09:30:02.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.652+0000 7f590a000640 1 --2- 172.21.15.6:0/2467094706 >> v2:172.21.15.146:3300/0 conn(0x7f59140d4860 0x7f59140d0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:02.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.653+0000 7f5912a00640 1 -- 172.21.15.6:0/2467094706 >> v2:172.21.15.146:3300/0 conn(0x7f59140d4860 msgr2=0x7f59140d0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:02.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.653+0000 7f5912a00640 1 --2- 172.21.15.6:0/2467094706 >> v2:172.21.15.146:3300/0 conn(0x7f59140d4860 0x7f59140d0630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:02.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.653+0000 7f5912a00640 1 -- 172.21.15.6:0/2467094706 >> v1:172.21.15.6:6789/0 conn(0x7f59140cfa60 legacy=0x7f59140b3510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:02.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.653+0000 7f5912a00640 1 -- 172.21.15.6:0/2467094706 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59140cf260 con 0x7f591415dee0 2024-03-04T09:30:02.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.653+0000 7f590a000640 1 --2- 172.21.15.6:0/2467094706 >> v2:172.21.15.146:3300/0 conn(0x7f59140d4860 0x7f59140d0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:30:02.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.654+0000 7f5912a00640 1 --2- 172.21.15.6:0/2467094706 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f591415dee0 0x7f59140d18a0 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f5914162b70 tx=0x7f590402bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abcf0d5f1db437e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:02.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.654+0000 7f5912000640 1 -- 172.21.15.6:0/2467094706 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f590402cb10 con 0x7f591415dee0 2024-03-04T09:30:02.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.654+0000 7f5912000640 1 -- 172.21.15.6:0/2467094706 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f590402ccb0 con 0x7f591415dee0 2024-03-04T09:30:02.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.654+0000 7f5912000640 1 -- 172.21.15.6:0/2467094706 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f590402ce60 con 0x7f591415dee0 2024-03-04T09:30:02.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.655+0000 7f5913e00640 1 -- 172.21.15.6:0/2467094706 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f591415dee0 msgr2=0x7f59140d18a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:02.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.655+0000 7f5913e00640 1 --2- 172.21.15.6:0/2467094706 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f591415dee0 0x7f59140d18a0 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f5914162b70 tx=0x7f590402bf70 comp rx=0 tx=0).stop 2024-03-04T09:30:02.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.655+0000 7f5913e00640 1 -- 172.21.15.6:0/2467094706 shutdown_connections 2024-03-04T09:30:02.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.655+0000 7f5913e00640 1 --2- 172.21.15.6:0/2467094706 >> v2:172.21.15.146:3300/0 conn(0x7f59140d4860 0x7f59140d0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:02.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.655+0000 7f5913e00640 1 --2- 172.21.15.6:0/2467094706 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f591415dee0 0x7f59140d18a0 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:02.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.655+0000 7f5913e00640 1 -- 172.21.15.6:0/2467094706 >> 172.21.15.6:0/2467094706 conn(0x7f59140d3cd0 msgr2=0x7f59140d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:02.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.655+0000 7f5913e00640 1 -- 172.21.15.6:0/2467094706 shutdown_connections 2024-03-04T09:30:02.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.655+0000 7f5913e00640 1 -- 172.21.15.6:0/2467094706 wait complete. 2024-03-04T09:30:02.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.658+0000 7f5913e00640 1 Processor -- start 2024-03-04T09:30:02.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.658+0000 7f5913e00640 1 -- start start 2024-03-04T09:30:02.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.659+0000 7f5913e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f59140cfa60 0x7f59140a1340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:02.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.659+0000 7f5913e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f59140d4860 0x7f591415f480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:02.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.659+0000 7f5913e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f591415dee0 0x7f59140cd450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:02.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.659+0000 7f5913e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59140d1de0 con 0x7f591415dee0 2024-03-04T09:30:02.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.659+0000 7f5913e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f591415e2c0 con 0x7f59140cfa60 2024-03-04T09:30:02.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.659+0000 7f5913e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59140d1f60 con 0x7f59140d4860 2024-03-04T09:30:02.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.660+0000 7f5913400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f591415dee0 0x7f59140cd450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:02.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.660+0000 7f5913400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f591415dee0 0x7f59140cd450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:34148/0 (socket says 172.21.15.6:34148) 2024-03-04T09:30:02.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.660+0000 7f5913400640 1 -- 172.21.15.6:0/1410333157 learned_addr learned my addr 172.21.15.6:0/1410333157 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:30:02.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.660+0000 7f5912a00640 1 --2- 172.21.15.6:0/1410333157 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f59140cfa60 0x7f59140a1340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:02.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.660+0000 7f590a000640 1 --2- 172.21.15.6:0/1410333157 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f59140d4860 0x7f591415f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:02.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.660+0000 7f5913400640 1 -- 172.21.15.6:0/1410333157 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f59140d4860 msgr2=0x7f591415f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:02.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.660+0000 7f5913400640 1 --2- 172.21.15.6:0/1410333157 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f59140d4860 0x7f591415f480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:02.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.660+0000 7f5913400640 1 -- 172.21.15.6:0/1410333157 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f59140cfa60 msgr2=0x7f59140a1340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:02.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.660+0000 7f5913400640 1 --2- 172.21.15.6:0/1410333157 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f59140cfa60 0x7f59140a1340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:02.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.660+0000 7f5913400640 1 -- 172.21.15.6:0/1410333157 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59140cc3b0 con 0x7f591415dee0 2024-03-04T09:30:02.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.660+0000 7f590a000640 1 --2- 172.21.15.6:0/1410333157 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f59140d4860 0x7f591415f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:30:02.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.660+0000 7f5912a00640 1 --2- 172.21.15.6:0/1410333157 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f59140cfa60 0x7f59140a1340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:30:02.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.660+0000 7f5913400640 1 --2- 172.21.15.6:0/1410333157 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f591415dee0 0x7f59140cd450 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f590c00c430 tx=0x7f590c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:02.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.661+0000 7f590b400640 1 -- 172.21.15.6:0/1410333157 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f590c0074e0 con 0x7f591415dee0 2024-03-04T09:30:02.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.661+0000 7f590b400640 1 -- 172.21.15.6:0/1410333157 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f590c007680 con 0x7f591415dee0 2024-03-04T09:30:02.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.661+0000 7f590b400640 1 -- 172.21.15.6:0/1410333157 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f590c0049e0 con 0x7f591415dee0 2024-03-04T09:30:02.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.661+0000 7f5913e00640 1 -- 172.21.15.6:0/1410333157 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59140cc690 con 0x7f591415dee0 2024-03-04T09:30:02.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.661+0000 7f5913e00640 1 -- 172.21.15.6:0/1410333157 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59140ccc50 con 0x7f591415dee0 2024-03-04T09:30:02.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.662+0000 7f5913e00640 1 -- 172.21.15.6:0/1410333157 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f591404f330 con 0x7f591415dee0 2024-03-04T09:30:02.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.663+0000 7f590b400640 1 -- 172.21.15.6:0/1410333157 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f590c01a020 con 0x7f591415dee0 2024-03-04T09:30:02.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.664+0000 7f590b400640 1 --2- 172.21.15.6:0/1410333157 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f58f006b4d0 0x7f58f006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:02.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.664+0000 7f590b400640 1 -- 172.21.15.6:0/1410333157 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f590c089a00 con 0x7f591415dee0 2024-03-04T09:30:02.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.665+0000 7f5912a00640 1 --2- 172.21.15.6:0/1410333157 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f58f006b4d0 0x7f58f006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:02.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.666+0000 7f5912a00640 1 --2- 172.21.15.6:0/1410333157 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f58f006b4d0 0x7f58f006d990 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f5904035000 tx=0x7f5904030000 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:02.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.669+0000 7f590b400640 1 -- 172.21.15.6:0/1410333157 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f590c058860 con 0x7f591415dee0 2024-03-04T09:30:02.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.870+0000 7f5913e00640 1 -- 172.21.15.6:0/1410333157 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f591408d7d0 con 0x7f591415dee0 2024-03-04T09:30:02.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.871+0000 7f590b400640 1 -- 172.21.15.6:0/1410333157 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) v1 ==== 93+0+10 (secure 0 0 0) 0x7f590c058160 con 0x7f591415dee0 2024-03-04T09:30:02.867 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-04T09:30:02.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.873+0000 7f5913e00640 1 -- 172.21.15.6:0/1410333157 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f58f006b4d0 msgr2=0x7f58f006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:02.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.873+0000 7f5913e00640 1 --2- 172.21.15.6:0/1410333157 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f58f006b4d0 0x7f58f006d990 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f5904035000 tx=0x7f5904030000 comp rx=0 tx=0).stop 2024-03-04T09:30:02.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.873+0000 7f5913e00640 1 -- 172.21.15.6:0/1410333157 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f591415dee0 msgr2=0x7f59140cd450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:02.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.873+0000 7f5913e00640 1 --2- 172.21.15.6:0/1410333157 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f591415dee0 0x7f59140cd450 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f590c00c430 tx=0x7f590c00c8f0 comp rx=0 tx=0).stop 2024-03-04T09:30:02.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.874+0000 7f5913e00640 1 -- 172.21.15.6:0/1410333157 shutdown_connections 2024-03-04T09:30:02.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.874+0000 7f5913e00640 1 --2- 172.21.15.6:0/1410333157 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f591415dee0 0x7f59140cd450 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:02.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.874+0000 7f5913e00640 1 --2- 172.21.15.6:0/1410333157 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f58f006b4d0 0x7f58f006d990 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:02.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.874+0000 7f5913e00640 1 --2- 172.21.15.6:0/1410333157 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f59140d4860 0x7f591415f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:02.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.874+0000 7f5913e00640 1 --2- 172.21.15.6:0/1410333157 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f59140cfa60 0x7f59140a1340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:02.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.874+0000 7f5913e00640 1 -- 172.21.15.6:0/1410333157 >> 172.21.15.6:0/1410333157 conn(0x7f59140d3cd0 msgr2=0x7f59140b9660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:02.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.874+0000 7f5913e00640 1 -- 172.21.15.6:0/1410333157 shutdown_connections 2024-03-04T09:30:02.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:02.874+0000 7f5913e00640 1 -- 172.21.15.6:0/1410333157 wait complete. 2024-03-04T09:30:02.892 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-04T09:30:03.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.155+0000 7f5c0ae00640 1 Processor -- start 2024-03-04T09:30:03.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.156+0000 7f5c0ae00640 1 -- start start 2024-03-04T09:30:03.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.157+0000 7f5c0ae00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5c0406ed40 0x7f5c040b3520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:03.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.157+0000 7f5c0ae00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f5c040cc710 0x7f5c040ccb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:03.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.157+0000 7f5c0ae00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c0406dee0 con 0x7f5c0406f120 2024-03-04T09:30:03.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.157+0000 7f5c0ae00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c0406e180 con 0x7f5c040cc710 2024-03-04T09:30:03.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.157+0000 7f5c0ae00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c0406e460 con 0x7f5c0406ed40 2024-03-04T09:30:03.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.158+0000 7f5c0a400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f5c0406f120 0x7f5c0406f5b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:37220/0 (socket says 172.21.15.6:37220) 2024-03-04T09:30:03.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.158+0000 7f5c0a400640 1 -- 172.21.15.6:0/2775607933 learned_addr learned my addr 172.21.15.6:0/2775607933 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:30:03.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.158+0000 7f5c09000640 1 --2- 172.21.15.6:0/2775607933 >> v2:172.21.15.146:3300/0 conn(0x7f5c040cc710 0x7f5c040ccb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:03.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.158+0000 7f5c09a00640 1 --2- 172.21.15.6:0/2775607933 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5c0406ed40 0x7f5c040b3520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:03.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.158+0000 7f5c03e00640 1 -- 172.21.15.6:0/2775607933 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3586703813 0 0) 0x7f5c0406dee0 con 0x7f5c0406f120 2024-03-04T09:30:03.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.158+0000 7f5c03e00640 1 -- 172.21.15.6:0/2775607933 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bec003520 con 0x7f5c0406f120 2024-03-04T09:30:03.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.159+0000 7f5c03e00640 1 -- 172.21.15.6:0/2775607933 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1363136355 0 0) 0x7f5bec003520 con 0x7f5c0406f120 2024-03-04T09:30:03.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.159+0000 7f5c03e00640 1 -- 172.21.15.6:0/2775607933 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5c0406ed40 msgr2=0x7f5c040b3520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:03.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.159+0000 7f5c03e00640 1 --2- 172.21.15.6:0/2775607933 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5c0406ed40 0x7f5c040b3520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:03.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.159+0000 7f5c03e00640 1 -- 172.21.15.6:0/2775607933 >> v2:172.21.15.146:3300/0 conn(0x7f5c040cc710 msgr2=0x7f5c040ccb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:03.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.159+0000 7f5c03e00640 1 --2- 172.21.15.6:0/2775607933 >> v2:172.21.15.146:3300/0 conn(0x7f5c040cc710 0x7f5c040ccb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:03.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.159+0000 7f5c03e00640 1 -- 172.21.15.6:0/2775607933 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c040ce400 con 0x7f5c0406f120 2024-03-04T09:30:03.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.159+0000 7f5c03e00640 1 -- 172.21.15.6:0/2775607933 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7f5bfc0019f0 con 0x7f5c0406f120 2024-03-04T09:30:03.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.159+0000 7f5c03e00640 1 -- 172.21.15.6:0/2775607933 >> v1:172.21.15.6:6789/0 conn(0x7f5c0406f120 legacy=0x7f5c0406f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:03.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.159+0000 7f5c03e00640 1 --2- 172.21.15.6:0/2775607933 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5bec0039b0 0x7f5bec003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:03.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.159+0000 7f5c03e00640 1 --2- 172.21.15.6:0/2775607933 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5c0406ed40 0x7f5bec004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:03.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.159+0000 7f5c03e00640 1 --2- 172.21.15.6:0/2775607933 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5bec004d40 0x7f5bec0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:03.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.159+0000 7f5c03e00640 1 -- 172.21.15.6:0/2775607933 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c0406e180 con 0x7f5bec004d40 2024-03-04T09:30:03.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.159+0000 7f5c03e00640 1 -- 172.21.15.6:0/2775607933 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c0406e460 con 0x7f5bec0039b0 2024-03-04T09:30:03.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.159+0000 7f5c03e00640 1 -- 172.21.15.6:0/2775607933 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bec00dca0 con 0x7f5c0406ed40 2024-03-04T09:30:03.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.159+0000 7f5c0a400640 1 --2- 172.21.15.6:0/2775607933 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5bec004d40 0x7f5bec0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:03.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.159+0000 7f5c09000640 1 --2- 172.21.15.6:0/2775607933 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5c0406ed40 0x7f5bec004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:03.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.160+0000 7f5c09a00640 1 --2- 172.21.15.6:0/2775607933 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5bec0039b0 0x7f5bec003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:03.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.160+0000 7f5c0a400640 1 -- 172.21.15.6:0/2775607933 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5c0406ed40 msgr2=0x7f5bec004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:03.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.160+0000 7f5c0a400640 1 --2- 172.21.15.6:0/2775607933 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5c0406ed40 0x7f5bec004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:03.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.160+0000 7f5c0a400640 1 -- 172.21.15.6:0/2775607933 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5bec0039b0 msgr2=0x7f5bec003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:03.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.160+0000 7f5c0a400640 1 --2- 172.21.15.6:0/2775607933 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5bec0039b0 0x7f5bec003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:03.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.160+0000 7f5c0a400640 1 -- 172.21.15.6:0/2775607933 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bec003520 con 0x7f5bec004d40 2024-03-04T09:30:03.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.160+0000 7f5c09a00640 1 --2- 172.21.15.6:0/2775607933 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5bec0039b0 0x7f5bec003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:30:03.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.161+0000 7f5c0a400640 1 --2- 172.21.15.6:0/2775607933 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5bec004d40 0x7f5bec0051c0 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f5bfc00a7a0 tx=0x7f5bec005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aee95a6af8bba489 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:03.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.161+0000 7f5c03e00640 1 -- 172.21.15.6:0/2775607933 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bfc00a3f0 con 0x7f5bec004d40 2024-03-04T09:30:03.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.161+0000 7f5c03e00640 1 -- 172.21.15.6:0/2775607933 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bfc034650 con 0x7f5bec004d40 2024-03-04T09:30:03.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.161+0000 7f5c03e00640 1 -- 172.21.15.6:0/2775607933 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bfc034a40 con 0x7f5bec004d40 2024-03-04T09:30:03.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.161+0000 7f5c0ae00640 1 -- 172.21.15.6:0/2775607933 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5bec004d40 msgr2=0x7f5bec0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:03.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.161+0000 7f5c0ae00640 1 --2- 172.21.15.6:0/2775607933 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5bec004d40 0x7f5bec0051c0 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f5bfc00a7a0 tx=0x7f5bec005830 comp rx=0 tx=0).stop 2024-03-04T09:30:03.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.161+0000 7f5c0a400640 1 -- 172.21.15.6:0/2775607933 reap_dead start 2024-03-04T09:30:03.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.162+0000 7f5c0ae00640 1 -- 172.21.15.6:0/2775607933 shutdown_connections 2024-03-04T09:30:03.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.162+0000 7f5c0ae00640 1 -- 172.21.15.6:0/2775607933 >> 172.21.15.6:0/2775607933 conn(0x7f5c040723d0 msgr2=0x7f5c0406afb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:03.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.162+0000 7f5c0ae00640 1 -- 172.21.15.6:0/2775607933 shutdown_connections 2024-03-04T09:30:03.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.162+0000 7f5c0ae00640 1 -- 172.21.15.6:0/2775607933 wait complete. 2024-03-04T09:30:03.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.163+0000 7f5c0ae00640 1 Processor -- start 2024-03-04T09:30:03.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.164+0000 7f5c0ae00640 1 -- start start 2024-03-04T09:30:03.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.165+0000 7f5c0ae00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5bec004d40 0x7f5c0415c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:03.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.165+0000 7f5c0ae00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5c0414abd0 0x7f5c0415a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:03.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.165+0000 7f5c0ae00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5c0415ada0 0x7f5c0414ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:03.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.165+0000 7f5c0ae00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c0406ea10 con 0x7f5c0415ada0 2024-03-04T09:30:03.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.165+0000 7f5c0ae00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c040b3520 con 0x7f5c0414abd0 2024-03-04T09:30:03.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.165+0000 7f5c0ae00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c0415bf30 con 0x7f5bec004d40 2024-03-04T09:30:03.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.166+0000 7f5c0a400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5c0415ada0 0x7f5c0414ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:03.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.166+0000 7f5c0a400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5c0415ada0 0x7f5c0414ece0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:34174/0 (socket says 172.21.15.6:34174) 2024-03-04T09:30:03.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.166+0000 7f5c0a400640 1 -- 172.21.15.6:0/504098675 learned_addr learned my addr 172.21.15.6:0/504098675 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:30:03.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.166+0000 7f5c09a00640 1 --2- 172.21.15.6:0/504098675 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5bec004d40 0x7f5c0415c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:03.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.166+0000 7f5c09000640 1 --2- 172.21.15.6:0/504098675 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5c0414abd0 0x7f5c0415a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:03.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.166+0000 7f5c0a400640 1 -- 172.21.15.6:0/504098675 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5bec004d40 msgr2=0x7f5c0415c3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:03.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.166+0000 7f5c0a400640 1 --2- 172.21.15.6:0/504098675 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5bec004d40 0x7f5c0415c3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:03.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.166+0000 7f5c0a400640 1 -- 172.21.15.6:0/504098675 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5c0414abd0 msgr2=0x7f5c0415a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:03.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.166+0000 7f5c0a400640 1 --2- 172.21.15.6:0/504098675 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5c0414abd0 0x7f5c0415a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:03.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.166+0000 7f5c0a400640 1 -- 172.21.15.6:0/504098675 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c0415b320 con 0x7f5c0415ada0 2024-03-04T09:30:03.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.166+0000 7f5c09000640 1 --2- 172.21.15.6:0/504098675 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5c0414abd0 0x7f5c0415a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:30:03.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.166+0000 7f5c0a400640 1 --2- 172.21.15.6:0/504098675 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5c0415ada0 0x7f5c0414ece0 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f5bfc00a7a0 tx=0x7f5c0406c780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:03.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.167+0000 7f5c01600640 1 -- 172.21.15.6:0/504098675 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bfc001f60 con 0x7f5c0415ada0 2024-03-04T09:30:03.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.167+0000 7f5c01600640 1 -- 172.21.15.6:0/504098675 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bfc044070 con 0x7f5c0415ada0 2024-03-04T09:30:03.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.167+0000 7f5c01600640 1 -- 172.21.15.6:0/504098675 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bfc034a40 con 0x7f5c0415ada0 2024-03-04T09:30:03.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.167+0000 7f5c0ae00640 1 -- 172.21.15.6:0/504098675 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c0415b600 con 0x7f5c0415ada0 2024-03-04T09:30:03.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.167+0000 7f5c0ae00640 1 -- 172.21.15.6:0/504098675 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c0415bbc0 con 0x7f5c0415ada0 2024-03-04T09:30:03.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.168+0000 7f5c0ae00640 1 -- 172.21.15.6:0/504098675 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c0414afb0 con 0x7f5c0415ada0 2024-03-04T09:30:03.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.169+0000 7f5c01600640 1 -- 172.21.15.6:0/504098675 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5bfc034be0 con 0x7f5c0415ada0 2024-03-04T09:30:03.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.171+0000 7f5c01600640 1 --2- 172.21.15.6:0/504098675 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f5be406b4d0 0x7f5be406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:03.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.171+0000 7f5c01600640 1 -- 172.21.15.6:0/504098675 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f5bfc0ac6d0 con 0x7f5c0415ada0 2024-03-04T09:30:03.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.171+0000 7f5c09a00640 1 --2- 172.21.15.6:0/504098675 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f5be406b4d0 0x7f5be406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:03.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.173+0000 7f5c09a00640 1 --2- 172.21.15.6:0/504098675 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f5be406b4d0 0x7f5be406d990 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f5bf800ab20 tx=0x7f5bf8008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:03.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.174+0000 7f5c01600640 1 -- 172.21.15.6:0/504098675 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5bfc07b5b0 con 0x7f5c0415ada0 2024-03-04T09:30:03.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.375+0000 7f5c0ae00640 1 -- 172.21.15.6:0/504098675 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f5c0404f330 con 0x7f5c0415ada0 2024-03-04T09:30:03.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.376+0000 7f5c01600640 1 -- 172.21.15.6:0/504098675 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v83) v1 ==== 104+0+11 (secure 0 0 0) 0x7f5bfc07aeb0 con 0x7f5c0415ada0 2024-03-04T09:30:03.372 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-03-04T09:30:03.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.378+0000 7f5c0ae00640 1 -- 172.21.15.6:0/504098675 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f5be406b4d0 msgr2=0x7f5be406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:03.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.378+0000 7f5c0ae00640 1 --2- 172.21.15.6:0/504098675 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f5be406b4d0 0x7f5be406d990 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f5bf800ab20 tx=0x7f5bf8008040 comp rx=0 tx=0).stop 2024-03-04T09:30:03.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.378+0000 7f5c0ae00640 1 -- 172.21.15.6:0/504098675 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5c0415ada0 msgr2=0x7f5c0414ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:03.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.378+0000 7f5c0ae00640 1 --2- 172.21.15.6:0/504098675 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5c0415ada0 0x7f5c0414ece0 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f5bfc00a7a0 tx=0x7f5c0406c780 comp rx=0 tx=0).stop 2024-03-04T09:30:03.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.378+0000 7f5c0ae00640 1 -- 172.21.15.6:0/504098675 shutdown_connections 2024-03-04T09:30:03.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.378+0000 7f5c0ae00640 1 --2- 172.21.15.6:0/504098675 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5c0415ada0 0x7f5c0414ece0 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:03.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.378+0000 7f5c0ae00640 1 --2- 172.21.15.6:0/504098675 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5c0414abd0 0x7f5c0415a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:03.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.378+0000 7f5c0ae00640 1 --2- 172.21.15.6:0/504098675 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f5be406b4d0 0x7f5be406d990 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:03.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.378+0000 7f5c0ae00640 1 --2- 172.21.15.6:0/504098675 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5bec004d40 0x7f5c0415c3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:03.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.378+0000 7f5c0ae00640 1 -- 172.21.15.6:0/504098675 >> 172.21.15.6:0/504098675 conn(0x7f5c040723d0 msgr2=0x7f5c04150320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:03.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.379+0000 7f5c0ae00640 1 -- 172.21.15.6:0/504098675 shutdown_connections 2024-03-04T09:30:03.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.379+0000 7f5c0ae00640 1 -- 172.21.15.6:0/504098675 wait complete. 2024-03-04T09:30:03.395 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-04T09:30:03.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.605+0000 7f69e3e00640 1 Processor -- start 2024-03-04T09:30:03.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.605+0000 7f69e3e00640 1 -- start start 2024-03-04T09:30:03.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.607+0000 7f69e3e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f69e4161700 0x7f69e40b4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:03.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.607+0000 7f69e3e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f69e40d4890 0x7f69e40d1de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:03.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.607+0000 7f69e3e00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69e40b2c00 con 0x7f69e40d2320 2024-03-04T09:30:03.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.607+0000 7f69e3e00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69e4161ae0 con 0x7f69e40d4890 2024-03-04T09:30:03.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.607+0000 7f69e3e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69e40b2ea0 con 0x7f69e4161700 2024-03-04T09:30:03.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.608+0000 7f69e2a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f69e4161700 0x7f69e40b4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:03.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.608+0000 7f69e2a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f69e4161700 0x7f69e40b4fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.6:38686/0 (socket says 172.21.15.6:38686) 2024-03-04T09:30:03.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.608+0000 7f69e2a00640 1 -- 172.21.15.6:0/3219029149 learned_addr learned my addr 172.21.15.6:0/3219029149 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:30:03.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.608+0000 7f69dbe00640 1 --2- 172.21.15.6:0/3219029149 >> v2:172.21.15.146:3300/0 conn(0x7f69e40d4890 0x7f69e40d1de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:03.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.609+0000 7f69e2a00640 1 -- 172.21.15.6:0/3219029149 >> v2:172.21.15.146:3300/0 conn(0x7f69e40d4890 msgr2=0x7f69e40d1de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:03.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.609+0000 7f69e2a00640 1 --2- 172.21.15.6:0/3219029149 >> v2:172.21.15.146:3300/0 conn(0x7f69e40d4890 0x7f69e40d1de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:03.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.609+0000 7f69e2a00640 1 -- 172.21.15.6:0/3219029149 >> v1:172.21.15.146:6789/0 conn(0x7f69e40d2320 legacy=0x7f69e40b3600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:03.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.609+0000 7f69e2a00640 1 -- 172.21.15.6:0/3219029149 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69e40b3180 con 0x7f69e4161700 2024-03-04T09:30:03.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.609+0000 7f69dbe00640 1 --2- 172.21.15.6:0/3219029149 >> v2:172.21.15.146:3300/0 conn(0x7f69e40d4890 0x7f69e40d1de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T09:30:03.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.610+0000 7f69e2a00640 1 --2- 172.21.15.6:0/3219029149 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f69e4161700 0x7f69e40b4fd0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f69d4009870 tx=0x7f69d402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2dfc908708f78e7e server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:03.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.611+0000 7f69e2000640 1 -- 172.21.15.6:0/3219029149 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69d4034070 con 0x7f69e4161700 2024-03-04T09:30:03.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.611+0000 7f69e2000640 1 -- 172.21.15.6:0/3219029149 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69d4030840 con 0x7f69e4161700 2024-03-04T09:30:03.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.611+0000 7f69e2000640 1 -- 172.21.15.6:0/3219029149 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69d4030c10 con 0x7f69e4161700 2024-03-04T09:30:03.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.611+0000 7f69e3e00640 1 -- 172.21.15.6:0/3219029149 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f69e4161700 msgr2=0x7f69e40b4fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:03.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.611+0000 7f69e3e00640 1 --2- 172.21.15.6:0/3219029149 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f69e4161700 0x7f69e40b4fd0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f69d4009870 tx=0x7f69d402b980 comp rx=0 tx=0).stop 2024-03-04T09:30:03.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.612+0000 7f69e3e00640 1 -- 172.21.15.6:0/3219029149 shutdown_connections 2024-03-04T09:30:03.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.612+0000 7f69e3e00640 1 --2- 172.21.15.6:0/3219029149 >> v2:172.21.15.146:3300/0 conn(0x7f69e40d4890 0x7f69e40d1de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:03.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.612+0000 7f69e3e00640 1 --2- 172.21.15.6:0/3219029149 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f69e4161700 0x7f69e40b4fd0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:03.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.612+0000 7f69e3e00640 1 -- 172.21.15.6:0/3219029149 >> 172.21.15.6:0/3219029149 conn(0x7f69e40d3d70 msgr2=0x7f69e40d4180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:03.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.612+0000 7f69e3e00640 1 -- 172.21.15.6:0/3219029149 shutdown_connections 2024-03-04T09:30:03.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.612+0000 7f69e3e00640 1 -- 172.21.15.6:0/3219029149 wait complete. 2024-03-04T09:30:03.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.614+0000 7f69e3e00640 1 Processor -- start 2024-03-04T09:30:03.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.614+0000 7f69e3e00640 1 -- start start 2024-03-04T09:30:03.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.616+0000 7f69e3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f69e40d2320 0x7f69e40b9440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:03.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.616+0000 7f69e3e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f69e40d4890 0x7f69e414db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:03.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.616+0000 7f69e3e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f69e4161700 0x7f69e40c9a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:03.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.616+0000 7f69e3e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69e40b5510 con 0x7f69e40d2320 2024-03-04T09:30:03.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.616+0000 7f69e3e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69e40b0d00 con 0x7f69e4161700 2024-03-04T09:30:03.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.616+0000 7f69e3e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69e40b5690 con 0x7f69e40d4890 2024-03-04T09:30:03.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.616+0000 7f69e2a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f69e40d2320 0x7f69e40b9440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:03.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.616+0000 7f69e2a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f69e40d2320 0x7f69e40b9440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:34190/0 (socket says 172.21.15.6:34190) 2024-03-04T09:30:03.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.616+0000 7f69e2a00640 1 -- 172.21.15.6:0/2090983529 learned_addr learned my addr 172.21.15.6:0/2090983529 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:30:03.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.617+0000 7f69e2a00640 1 -- 172.21.15.6:0/2090983529 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f69e40d4890 msgr2=0x7f69e414db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:03.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.617+0000 7f69e2a00640 1 --2- 172.21.15.6:0/2090983529 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f69e40d4890 0x7f69e414db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:03.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.617+0000 7f69e2a00640 1 -- 172.21.15.6:0/2090983529 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f69e4161700 msgr2=0x7f69e40c9a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:03.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.617+0000 7f69e2a00640 1 --2- 172.21.15.6:0/2090983529 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f69e4161700 0x7f69e40c9a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:03.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.617+0000 7f69e2a00640 1 -- 172.21.15.6:0/2090983529 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69e414e070 con 0x7f69e40d2320 2024-03-04T09:30:03.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.617+0000 7f69e2a00640 1 --2- 172.21.15.6:0/2090983529 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f69e40d2320 0x7f69e40b9440 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f69d4000c00 tx=0x7f69d40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:03.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.617+0000 7f69daa00640 1 -- 172.21.15.6:0/2090983529 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69d4034040 con 0x7f69e40d2320 2024-03-04T09:30:03.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.617+0000 7f69daa00640 1 -- 172.21.15.6:0/2090983529 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69d403ada0 con 0x7f69e40d2320 2024-03-04T09:30:03.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.617+0000 7f69e3e00640 1 -- 172.21.15.6:0/2090983529 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69e40c9290 con 0x7f69e40d2320 2024-03-04T09:30:03.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.617+0000 7f69daa00640 1 -- 172.21.15.6:0/2090983529 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69d4044630 con 0x7f69e40d2320 2024-03-04T09:30:03.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.618+0000 7f69e3e00640 1 -- 172.21.15.6:0/2090983529 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69e40cde00 con 0x7f69e40d2320 2024-03-04T09:30:03.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.619+0000 7f69e3e00640 1 -- 172.21.15.6:0/2090983529 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69e404f330 con 0x7f69e40d2320 2024-03-04T09:30:03.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.620+0000 7f69daa00640 1 -- 172.21.15.6:0/2090983529 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f69d404b060 con 0x7f69e40d2320 2024-03-04T09:30:03.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.621+0000 7f69daa00640 1 --2- 172.21.15.6:0/2090983529 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f69c006b4d0 0x7f69c006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:03.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.622+0000 7f69daa00640 1 -- 172.21.15.6:0/2090983529 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f69d40ac1f0 con 0x7f69e40d2320 2024-03-04T09:30:03.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.622+0000 7f69dbe00640 1 --2- 172.21.15.6:0/2090983529 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f69c006b4d0 0x7f69c006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:03.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.623+0000 7f69dbe00640 1 --2- 172.21.15.6:0/2090983529 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f69c006b4d0 0x7f69c006d990 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f69cc006e80 tx=0x7f69cc006dd0 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:03.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.625+0000 7f69daa00640 1 -- 172.21.15.6:0/2090983529 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f69d403f070 con 0x7f69e40d2320 2024-03-04T09:30:03.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.850+0000 7f69e3e00640 1 -- 172.21.15.6:0/2090983529 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f69e406c970 con 0x7f69e40d2320 2024-03-04T09:30:03.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.851+0000 7f69daa00640 1 -- 172.21.15.6:0/2090983529 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v83) v1 ==== 100+0+11 (secure 0 0 0) 0x7f69d407b050 con 0x7f69e40d2320 2024-03-04T09:30:03.847 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 64 2024-03-04T09:30:03.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.853+0000 7f69e3e00640 1 -- 172.21.15.6:0/2090983529 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f69c006b4d0 msgr2=0x7f69c006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:03.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.853+0000 7f69e3e00640 1 --2- 172.21.15.6:0/2090983529 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f69c006b4d0 0x7f69c006d990 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f69cc006e80 tx=0x7f69cc006dd0 comp rx=0 tx=0).stop 2024-03-04T09:30:03.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.853+0000 7f69e3e00640 1 -- 172.21.15.6:0/2090983529 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f69e40d2320 msgr2=0x7f69e40b9440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:03.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.853+0000 7f69e3e00640 1 --2- 172.21.15.6:0/2090983529 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f69e40d2320 0x7f69e40b9440 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f69d4000c00 tx=0x7f69d40046c0 comp rx=0 tx=0).stop 2024-03-04T09:30:03.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.853+0000 7f69e3e00640 1 -- 172.21.15.6:0/2090983529 shutdown_connections 2024-03-04T09:30:03.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.853+0000 7f69e3e00640 1 --2- 172.21.15.6:0/2090983529 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f69e4161700 0x7f69e40c9a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:03.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.853+0000 7f69e3e00640 1 --2- 172.21.15.6:0/2090983529 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f69c006b4d0 0x7f69c006d990 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:03.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.853+0000 7f69e3e00640 1 --2- 172.21.15.6:0/2090983529 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f69e40d4890 0x7f69e414db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:03.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.853+0000 7f69e3e00640 1 --2- 172.21.15.6:0/2090983529 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f69e40d2320 0x7f69e40b9440 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:03.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.853+0000 7f69e3e00640 1 -- 172.21.15.6:0/2090983529 >> 172.21.15.6:0/2090983529 conn(0x7f69e40d3d70 msgr2=0x7f69e41605e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:03.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.853+0000 7f69e3e00640 1 -- 172.21.15.6:0/2090983529 shutdown_connections 2024-03-04T09:30:03.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:03.853+0000 7f69e3e00640 1 -- 172.21.15.6:0/2090983529 wait complete. 2024-03-04T09:30:03.870 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:30:04.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.105+0000 7effc2c00640 1 Processor -- start 2024-03-04T09:30:04.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.105+0000 7effc2c00640 1 -- start start 2024-03-04T09:30:04.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.106+0000 7effc2c00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7effbc0d0e50 0x7effbc168c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:04.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.107+0000 7effc2c00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effbc0b3710 con 0x7effbc15e0b0 2024-03-04T09:30:04.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.107+0000 7effc2c00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effbc0b39f0 con 0x7effbc0d4a10 2024-03-04T09:30:04.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.107+0000 7effc2c00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7effbc0b09e0 con 0x7effbc0d0e50 2024-03-04T09:30:04.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.107+0000 7effc1800640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7effbc15e0b0 0x7effbc0cf310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:37222/0 (socket says 172.21.15.6:37222) 2024-03-04T09:30:04.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.107+0000 7effc1800640 1 -- 172.21.15.6:0/2840050258 learned_addr learned my addr 172.21.15.6:0/2840050258 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:30:04.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.107+0000 7effbbe00640 1 -- 172.21.15.6:0/2840050258 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3186594898 0 0) 0x7effbc0b3710 con 0x7effbc15e0b0 2024-03-04T09:30:04.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.107+0000 7effbbe00640 1 -- 172.21.15.6:0/2840050258 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7effa4003520 con 0x7effbc15e0b0 2024-03-04T09:30:04.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.107+0000 7effc2200640 1 --2- 172.21.15.6:0/2840050258 >> v2:172.21.15.146:3300/0 conn(0x7effbc0d0e50 0x7effbc168c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:04.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.108+0000 7effbbe00640 1 -- 172.21.15.6:0/2840050258 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 276289378 0 0) 0x7effa4003520 con 0x7effbc15e0b0 2024-03-04T09:30:04.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.108+0000 7effbbe00640 1 -- 172.21.15.6:0/2840050258 >> v2:172.21.15.146:3300/0 conn(0x7effbc0d0e50 msgr2=0x7effbc168c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:04.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.108+0000 7effbbe00640 1 --2- 172.21.15.6:0/2840050258 >> v2:172.21.15.146:3300/0 conn(0x7effbc0d0e50 0x7effbc168c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:04.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.108+0000 7effbbe00640 1 -- 172.21.15.6:0/2840050258 >> v1:172.21.15.146:6789/0 conn(0x7effbc0d4a10 legacy=0x7effbc0d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:04.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.108+0000 7effbbe00640 1 -- 172.21.15.6:0/2840050258 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effbc0b3cd0 con 0x7effbc15e0b0 2024-03-04T09:30:04.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.108+0000 7effbbe00640 1 -- 172.21.15.6:0/2840050258 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7effb0001670 con 0x7effbc15e0b0 2024-03-04T09:30:04.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.108+0000 7effbbe00640 1 -- 172.21.15.6:0/2840050258 >> v1:172.21.15.6:6789/0 conn(0x7effbc15e0b0 legacy=0x7effbc0cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:04.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.108+0000 7effbbe00640 1 --2- 172.21.15.6:0/2840050258 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7effa4003a00 0x7effa4003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:04.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.108+0000 7effbbe00640 1 --2- 172.21.15.6:0/2840050258 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7effa4004800 0x7effa4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:04.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.108+0000 7effbbe00640 1 --2- 172.21.15.6:0/2840050258 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7effa4005210 0x7effa400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:04.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.108+0000 7effbbe00640 1 -- 172.21.15.6:0/2840050258 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effa400c130 con 0x7effa4003a00 2024-03-04T09:30:04.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.108+0000 7effbbe00640 1 -- 172.21.15.6:0/2840050258 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effa400c2e0 con 0x7effa4005210 2024-03-04T09:30:04.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.108+0000 7effbbe00640 1 -- 172.21.15.6:0/2840050258 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effa400c4e0 con 0x7effa4004800 2024-03-04T09:30:04.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.108+0000 7effc1800640 1 --2- 172.21.15.6:0/2840050258 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7effa4003a00 0x7effa4003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:04.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.109+0000 7effc0e00640 1 --2- 172.21.15.6:0/2840050258 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7effa4004800 0x7effa4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:04.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.109+0000 7effc1800640 1 -- 172.21.15.6:0/2840050258 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7effa4004800 msgr2=0x7effa4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:04.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.109+0000 7effc2200640 1 --2- 172.21.15.6:0/2840050258 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7effa4005210 0x7effa400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:04.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.109+0000 7effc1800640 1 --2- 172.21.15.6:0/2840050258 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7effa4004800 0x7effa4004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:04.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.109+0000 7effc1800640 1 -- 172.21.15.6:0/2840050258 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7effa4005210 msgr2=0x7effa400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:04.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.109+0000 7effc1800640 1 --2- 172.21.15.6:0/2840050258 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7effa4005210 0x7effa400b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:04.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.109+0000 7effc1800640 1 -- 172.21.15.6:0/2840050258 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effa4003520 con 0x7effa4003a00 2024-03-04T09:30:04.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.109+0000 7effc2200640 1 -- 172.21.15.6:0/2840050258 reap_dead start 2024-03-04T09:30:04.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.110+0000 7effc1800640 1 --2- 172.21.15.6:0/2840050258 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7effa4003a00 0x7effa4003e20 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7effb000c110 tx=0x7effa4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e3361c0a5e17c6d server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:04.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.110+0000 7effbbe00640 1 -- 172.21.15.6:0/2840050258 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effb0007c40 con 0x7effa4003a00 2024-03-04T09:30:04.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.110+0000 7effbbe00640 1 -- 172.21.15.6:0/2840050258 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effb0030070 con 0x7effa4003a00 2024-03-04T09:30:04.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.110+0000 7effbbe00640 1 -- 172.21.15.6:0/2840050258 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effb0005d70 con 0x7effa4003a00 2024-03-04T09:30:04.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.111+0000 7effc2c00640 1 -- 172.21.15.6:0/2840050258 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7effa4003a00 msgr2=0x7effa4003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:04.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.111+0000 7effc2c00640 1 --2- 172.21.15.6:0/2840050258 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7effa4003a00 0x7effa4003e20 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7effb000c110 tx=0x7effa4005830 comp rx=0 tx=0).stop 2024-03-04T09:30:04.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.111+0000 7effc2c00640 1 -- 172.21.15.6:0/2840050258 shutdown_connections 2024-03-04T09:30:04.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.111+0000 7effc2c00640 1 --2- 172.21.15.6:0/2840050258 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7effa4003a00 0x7effa4003e20 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:04.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.111+0000 7effc2c00640 1 -- 172.21.15.6:0/2840050258 >> 172.21.15.6:0/2840050258 conn(0x7effbc0d3d30 msgr2=0x7effbc0d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:04.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.111+0000 7effc2c00640 1 -- 172.21.15.6:0/2840050258 shutdown_connections 2024-03-04T09:30:04.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.111+0000 7effc2c00640 1 -- 172.21.15.6:0/2840050258 wait complete. 2024-03-04T09:30:04.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.113+0000 7effc2c00640 1 Processor -- start 2024-03-04T09:30:04.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.113+0000 7effc2c00640 1 -- start start 2024-03-04T09:30:04.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.114+0000 7effc2c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7effa4003a00 0x7effbc153240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:04.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.114+0000 7effc2c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7effbc152b00 0x7effbc160f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:04.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.114+0000 7effc2c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7effbc0ccc30 0x7effbc0cd0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:04.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.114+0000 7effc2c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effbc0cfc60 con 0x7effbc152b00 2024-03-04T09:30:04.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.114+0000 7effc2c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effbc15e490 con 0x7effa4003a00 2024-03-04T09:30:04.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.114+0000 7effc2c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effbc0d0740 con 0x7effbc0ccc30 2024-03-04T09:30:04.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.114+0000 7effc0e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7effbc152b00 0x7effbc160f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:04.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.114+0000 7effc0e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7effbc152b00 0x7effbc160f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:34214/0 (socket says 172.21.15.6:34214) 2024-03-04T09:30:04.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.114+0000 7effc0e00640 1 -- 172.21.15.6:0/2006863635 learned_addr learned my addr 172.21.15.6:0/2006863635 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:30:04.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.114+0000 7effc2200640 1 --2- 172.21.15.6:0/2006863635 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7effbc0ccc30 0x7effbc0cd0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:04.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.114+0000 7effc1800640 1 --2- 172.21.15.6:0/2006863635 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7effa4003a00 0x7effbc153240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:04.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.114+0000 7effc0e00640 1 -- 172.21.15.6:0/2006863635 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7effbc0ccc30 msgr2=0x7effbc0cd0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:04.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.114+0000 7effc0e00640 1 --2- 172.21.15.6:0/2006863635 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7effbc0ccc30 0x7effbc0cd0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:04.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.114+0000 7effc0e00640 1 -- 172.21.15.6:0/2006863635 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7effa4003a00 msgr2=0x7effbc153240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:04.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.114+0000 7effc0e00640 1 --2- 172.21.15.6:0/2006863635 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7effa4003a00 0x7effbc153240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:04.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.115+0000 7effc0e00640 1 -- 172.21.15.6:0/2006863635 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effbc161490 con 0x7effbc152b00 2024-03-04T09:30:04.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.115+0000 7effc0e00640 1 --2- 172.21.15.6:0/2006863635 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7effbc152b00 0x7effbc160f50 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7effa800d4a0 tx=0x7effa800d960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:04.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.115+0000 7effb9600640 1 -- 172.21.15.6:0/2006863635 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effa8005480 con 0x7effbc152b00 2024-03-04T09:30:04.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.115+0000 7effb9600640 1 -- 172.21.15.6:0/2006863635 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effa8017070 con 0x7effbc152b00 2024-03-04T09:30:04.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.115+0000 7effb9600640 1 -- 172.21.15.6:0/2006863635 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effa8003810 con 0x7effbc152b00 2024-03-04T09:30:04.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.115+0000 7effc2c00640 1 -- 172.21.15.6:0/2006863635 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effbc15f270 con 0x7effbc152b00 2024-03-04T09:30:04.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.115+0000 7effc2c00640 1 -- 172.21.15.6:0/2006863635 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effbc15f830 con 0x7effbc152b00 2024-03-04T09:30:04.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.117+0000 7effc2c00640 1 -- 172.21.15.6:0/2006863635 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effbc04f330 con 0x7effbc152b00 2024-03-04T09:30:04.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.118+0000 7effb9600640 1 -- 172.21.15.6:0/2006863635 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7effa80039b0 con 0x7effbc152b00 2024-03-04T09:30:04.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.119+0000 7effb9600640 1 --2- 172.21.15.6:0/2006863635 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7eff9c06b500 0x7eff9c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:04.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.119+0000 7effb9600640 1 -- 172.21.15.6:0/2006863635 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5738+0+0 (secure 0 0 0) 0x7effa8089e90 con 0x7effbc152b00 2024-03-04T09:30:04.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.120+0000 7effc1800640 1 --2- 172.21.15.6:0/2006863635 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7eff9c06b500 0x7eff9c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:04.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.121+0000 7effc1800640 1 --2- 172.21.15.6:0/2006863635 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7eff9c06b500 0x7eff9c06d9c0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7effb0034000 tx=0x7effb0039040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:04.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.123+0000 7effb9600640 1 -- 172.21.15.6:0/2006863635 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7effa8058e40 con 0x7effbc152b00 2024-03-04T09:30:04.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.363+0000 7effc2c00640 1 -- 172.21.15.6:0/2006863635 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7effbc0d3260 con 0x7effbc152b00 2024-03-04T09:30:04.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.365+0000 7effb9600640 1 -- 172.21.15.6:0/2006863635 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) v1 ==== 94+0+4976 (secure 0 0 0) 0x7effa8058740 con 0x7effbc152b00 2024-03-04T09:30:04.362 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:30:04.362 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":51,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5827,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6836/3447344578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":3447344578},{"type":"v1","addr":"172.21.15.146:6836","nonce":3447344578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":46},{"gid":5830,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/168994142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":168994142},{"type":"v1","addr":"172.21.15.146:6837","nonce":168994142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":46},{"gid":6735,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/3369489999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":3369489999},{"type":"v1","addr":"172.21.15.6:6833","nonce":3369489999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":45}],"filesystems":[{"mdsmap":{"epoch":51,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:29:47.357677+0000","modified":"2024-03-04T09:29:50.579389+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6744},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6744":{"gid":6744,"name":"a","rank":0,"incarnation":47,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6835/1667002065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1667002065},{"type":"v1","addr":"172.21.15.6:6835","nonce":1667002065}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6744,"qdb_cluster":[6744]},"id":6}]} 2024-03-04T09:30:04.362 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 51 2024-03-04T09:30:04.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.367+0000 7effc2c00640 1 -- 172.21.15.6:0/2006863635 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7eff9c06b500 msgr2=0x7eff9c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:04.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.367+0000 7effc2c00640 1 --2- 172.21.15.6:0/2006863635 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7eff9c06b500 0x7eff9c06d9c0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7effb0034000 tx=0x7effb0039040 comp rx=0 tx=0).stop 2024-03-04T09:30:04.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.367+0000 7effc2c00640 1 -- 172.21.15.6:0/2006863635 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7effbc152b00 msgr2=0x7effbc160f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:04.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.367+0000 7effc2c00640 1 --2- 172.21.15.6:0/2006863635 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7effbc152b00 0x7effbc160f50 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7effa800d4a0 tx=0x7effa800d960 comp rx=0 tx=0).stop 2024-03-04T09:30:04.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.367+0000 7effc2c00640 1 -- 172.21.15.6:0/2006863635 shutdown_connections 2024-03-04T09:30:04.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.367+0000 7effc2c00640 1 --2- 172.21.15.6:0/2006863635 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7eff9c06b500 0x7eff9c06d9c0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:04.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.367+0000 7effc2c00640 1 --2- 172.21.15.6:0/2006863635 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7effbc0ccc30 0x7effbc0cd0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:04.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.367+0000 7effc2c00640 1 --2- 172.21.15.6:0/2006863635 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7effbc152b00 0x7effbc160f50 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:04.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.367+0000 7effc2c00640 1 --2- 172.21.15.6:0/2006863635 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7effa4003a00 0x7effbc153240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:04.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.367+0000 7effc2c00640 1 -- 172.21.15.6:0/2006863635 >> 172.21.15.6:0/2006863635 conn(0x7effbc0d3d30 msgr2=0x7effbc0b1c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:04.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.368+0000 7effc2c00640 1 -- 172.21.15.6:0/2006863635 shutdown_connections 2024-03-04T09:30:04.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.368+0000 7effc2c00640 1 -- 172.21.15.6:0/2006863635 wait complete. 2024-03-04T09:30:04.386 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T09:30:04.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.606+0000 7f9f2a200640 1 Processor -- start 2024-03-04T09:30:04.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.606+0000 7f9f2a200640 1 -- start start 2024-03-04T09:30:04.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.608+0000 7f9f2a200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f9f240af1a0 0x7f9f2415c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:04.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.608+0000 7f9f2a200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9f240cdc20 0x7f9f240b2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:04.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.608+0000 7f9f2a200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f2415cc40 con 0x7f9f240af580 2024-03-04T09:30:04.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.608+0000 7f9f2a200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f2415cf20 con 0x7f9f240af1a0 2024-03-04T09:30:04.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.608+0000 7f9f2a200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f2415d200 con 0x7f9f240cdc20 2024-03-04T09:30:04.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.608+0000 7f9f23400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f9f240af1a0 0x7f9f2415c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:04.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.608+0000 7f9f23400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f9f240af1a0 0x7f9f2415c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:34218/0 (socket says 172.21.15.6:34218) 2024-03-04T09:30:04.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.608+0000 7f9f23400640 1 -- 172.21.15.6:0/372840383 learned_addr learned my addr 172.21.15.6:0/372840383 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:30:04.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.609+0000 7f9f23e00640 1 --2- 172.21.15.6:0/372840383 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9f240cdc20 0x7f9f240b2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:04.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.609+0000 7f9f23400640 1 -- 172.21.15.6:0/372840383 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9f240cdc20 msgr2=0x7f9f240b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:04.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.609+0000 7f9f23400640 1 --2- 172.21.15.6:0/372840383 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9f240cdc20 0x7f9f240b2dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:04.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.609+0000 7f9f23400640 1 -- 172.21.15.6:0/372840383 >> v1:172.21.15.146:6789/0 conn(0x7f9f240af580 legacy=0x7f9f240b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:04.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.609+0000 7f9f23400640 1 -- 172.21.15.6:0/372840383 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f2415d4e0 con 0x7f9f240af1a0 2024-03-04T09:30:04.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.610+0000 7f9f23400640 1 --2- 172.21.15.6:0/372840383 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9f240af1a0 0x7f9f2415c600 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f9f14009870 tx=0x7f9f1402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7bd53a79e8c424a server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:04.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.611+0000 7f9f22000640 1 -- 172.21.15.6:0/372840383 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f14035020 con 0x7f9f240af1a0 2024-03-04T09:30:04.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.611+0000 7f9f22000640 1 -- 172.21.15.6:0/372840383 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f14031640 con 0x7f9f240af1a0 2024-03-04T09:30:04.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.611+0000 7f9f22000640 1 -- 172.21.15.6:0/372840383 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f1402d040 con 0x7f9f240af1a0 2024-03-04T09:30:04.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.611+0000 7f9f2a200640 1 -- 172.21.15.6:0/372840383 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9f240af1a0 msgr2=0x7f9f2415c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:04.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.611+0000 7f9f2a200640 1 --2- 172.21.15.6:0/372840383 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9f240af1a0 0x7f9f2415c600 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f9f14009870 tx=0x7f9f1402b8f0 comp rx=0 tx=0).stop 2024-03-04T09:30:04.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.611+0000 7f9f2a200640 1 -- 172.21.15.6:0/372840383 shutdown_connections 2024-03-04T09:30:04.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.611+0000 7f9f2a200640 1 --2- 172.21.15.6:0/372840383 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9f240cdc20 0x7f9f240b2dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:04.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.611+0000 7f9f2a200640 1 --2- 172.21.15.6:0/372840383 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9f240af1a0 0x7f9f2415c600 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:04.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.611+0000 7f9f2a200640 1 -- 172.21.15.6:0/372840383 >> 172.21.15.6:0/372840383 conn(0x7f9f2406afb0 msgr2=0x7f9f2406b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:04.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.612+0000 7f9f2a200640 1 -- 172.21.15.6:0/372840383 shutdown_connections 2024-03-04T09:30:04.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.612+0000 7f9f2a200640 1 -- 172.21.15.6:0/372840383 wait complete. 2024-03-04T09:30:04.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.614+0000 7f9f2a200640 1 Processor -- start 2024-03-04T09:30:04.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.615+0000 7f9f2a200640 1 -- start start 2024-03-04T09:30:04.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.616+0000 7f9f2a200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9f240af1a0 0x7f9f24151b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:04.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.616+0000 7f9f2a200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9f240af580 0x7f9f24152f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:04.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.616+0000 7f9f2a200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9f240cdc20 0x7f9f2414eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:04.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.616+0000 7f9f2a200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f240b3720 con 0x7f9f240af1a0 2024-03-04T09:30:04.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.616+0000 7f9f2a200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f240b35a0 con 0x7f9f240cdc20 2024-03-04T09:30:04.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.616+0000 7f9f2a200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f24155e70 con 0x7f9f240af580 2024-03-04T09:30:04.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.616+0000 7f9f23400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9f240af1a0 0x7f9f24151b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:04.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.616+0000 7f9f23400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9f240af1a0 0x7f9f24151b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:34220/0 (socket says 172.21.15.6:34220) 2024-03-04T09:30:04.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.616+0000 7f9f23400640 1 -- 172.21.15.6:0/2031536677 learned_addr learned my addr 172.21.15.6:0/2031536677 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:30:04.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.617+0000 7f9f23e00640 1 --2- 172.21.15.6:0/2031536677 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9f240cdc20 0x7f9f2414eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:04.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.617+0000 7f9f23400640 1 -- 172.21.15.6:0/2031536677 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9f240af580 msgr2=0x7f9f24152f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:04.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.617+0000 7f9f22a00640 1 --2- 172.21.15.6:0/2031536677 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9f240af580 0x7f9f24152f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:04.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.617+0000 7f9f23400640 1 --2- 172.21.15.6:0/2031536677 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9f240af580 0x7f9f24152f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:04.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.617+0000 7f9f23400640 1 -- 172.21.15.6:0/2031536677 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9f240cdc20 msgr2=0x7f9f2414eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:04.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.617+0000 7f9f23400640 1 --2- 172.21.15.6:0/2031536677 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9f240cdc20 0x7f9f2414eb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:04.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.617+0000 7f9f23400640 1 -- 172.21.15.6:0/2031536677 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f24153450 con 0x7f9f240af1a0 2024-03-04T09:30:04.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.617+0000 7f9f23e00640 1 --2- 172.21.15.6:0/2031536677 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9f240cdc20 0x7f9f2414eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:30:04.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.617+0000 7f9f22a00640 1 --2- 172.21.15.6:0/2031536677 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9f240af580 0x7f9f24152f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:30:04.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.617+0000 7f9f23400640 1 --2- 172.21.15.6:0/2031536677 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9f240af1a0 0x7f9f24151b00 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f9f1402b930 tx=0x7f9f140046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:04.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.618+0000 7f9f1b400640 1 -- 172.21.15.6:0/2031536677 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f14040070 con 0x7f9f240af1a0 2024-03-04T09:30:04.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.618+0000 7f9f1b400640 1 -- 172.21.15.6:0/2031536677 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f1403e070 con 0x7f9f240af1a0 2024-03-04T09:30:04.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.618+0000 7f9f1b400640 1 -- 172.21.15.6:0/2031536677 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f14035040 con 0x7f9f240af1a0 2024-03-04T09:30:04.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.618+0000 7f9f2a200640 1 -- 172.21.15.6:0/2031536677 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f241536f0 con 0x7f9f240af1a0 2024-03-04T09:30:04.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.618+0000 7f9f2a200640 1 -- 172.21.15.6:0/2031536677 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f24153cb0 con 0x7f9f240af1a0 2024-03-04T09:30:04.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.619+0000 7f9f2a200640 1 -- 172.21.15.6:0/2031536677 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f2404f3a0 con 0x7f9f240af1a0 2024-03-04T09:30:04.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.620+0000 7f9f1b400640 1 -- 172.21.15.6:0/2031536677 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9f14038020 con 0x7f9f240af1a0 2024-03-04T09:30:04.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.621+0000 7f9f1b400640 1 --2- 172.21.15.6:0/2031536677 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f9f0406b4d0 0x7f9f0406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:04.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.621+0000 7f9f1b400640 1 -- 172.21.15.6:0/2031536677 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f9f140abd40 con 0x7f9f240af1a0 2024-03-04T09:30:04.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.622+0000 7f9f22a00640 1 --2- 172.21.15.6:0/2031536677 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f9f0406b4d0 0x7f9f0406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:04.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.623+0000 7f9f22a00640 1 --2- 172.21.15.6:0/2031536677 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f9f0406b4d0 0x7f9f0406d990 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f9f2406b640 tx=0x7f9f10009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:04.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.626+0000 7f9f1b400640 1 -- 172.21.15.6:0/2031536677 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9f1407ab00 con 0x7f9f240af1a0 2024-03-04T09:30:04.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.823+0000 7f9f2a200640 1 -- 172.21.15.6:0/2031536677 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9f2406b390 con 0x7f9f240af1a0 2024-03-04T09:30:04.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.826+0000 7f9f1b400640 1 -- 172.21.15.6:0/2031536677 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) v1 ==== 74+0+16662 (secure 0 0 0) 0x7f9f1407a400 con 0x7f9f240af1a0 2024-03-04T09:30:04.824 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:30:04.824 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":83,"fsid":"edce0578-5ae4-4402-b45a-552bdef84d7c","created":"2024-03-04T09:24:54.523780+0000","modified":"2024-03-04T09:30:03.005383+0000","last_up_change":"2024-03-04T09:25:07.089049+0000","last_in_change":"2024-03-04T09:24:57.982077+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T09:25:07.935198+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-03-04T09:29:45.280632+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-03-04T09:29:46.378588+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a96574f-81f1-4bc6-8463-757520b84a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6825","nonce":1364449401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6827","nonce":1364449401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6831","nonce":1364449401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6829","nonce":1364449401}]},"public_addr":"172.21.15.6:6825/1364449401","cluster_addr":"172.21.15.6:6827/1364449401","heartbeat_back_addr":"172.21.15.6:6831/1364449401","heartbeat_front_addr":"172.21.15.6:6829/1364449401","state":["exists","up"]},{"osd":1,"uuid":"7d79b9c1-6b87-476a-bc36-7db5ee64cd90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6809","nonce":2494103078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6811","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6813","nonce":2494103078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6821","nonce":2494103078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6817","nonce":2494103078}]},"public_addr":"172.21.15.6:6809/2494103078","cluster_addr":"172.21.15.6:6813/2494103078","heartbeat_back_addr":"172.21.15.6:6821/2494103078","heartbeat_front_addr":"172.21.15.6:6817/2494103078","state":["exists","up"]},{"osd":2,"uuid":"492331b9-4ee9-40b6-bacd-85d44cd6c45a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6812","nonce":1500066681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6816","nonce":1500066681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6824","nonce":1500066681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6820","nonce":1500066681}]},"public_addr":"172.21.15.6:6812/1500066681","cluster_addr":"172.21.15.6:6816/1500066681","heartbeat_back_addr":"172.21.15.6:6824/1500066681","heartbeat_front_addr":"172.21.15.6:6820/1500066681","state":["exists","up"]},{"osd":3,"uuid":"21bb2efe-9ba7-4bdc-bc31-180cd1102e21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6801","nonce":1558806123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6803","nonce":1558806123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6807","nonce":1558806123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6805","nonce":1558806123}]},"public_addr":"172.21.15.6:6801/1558806123","cluster_addr":"172.21.15.6:6803/1558806123","heartbeat_back_addr":"172.21.15.6:6807/1558806123","heartbeat_front_addr":"172.21.15.6:6805/1558806123","state":["exists","up"]},{"osd":4,"uuid":"6e2c4b71-19d9-4737-83d9-558705d8644a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6801","nonce":1526135780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6803","nonce":1526135780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6807","nonce":1526135780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6805","nonce":1526135780}]},"public_addr":"172.21.15.146:6801/1526135780","cluster_addr":"172.21.15.146:6803/1526135780","heartbeat_back_addr":"172.21.15.146:6807/1526135780","heartbeat_front_addr":"172.21.15.146:6805/1526135780","state":["exists","up"]},{"osd":5,"uuid":"fb69c525-f1be-4b7d-ab32-6c29b0d6fddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6809","nonce":460262507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6811","nonce":460262507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6815","nonce":460262507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6813","nonce":460262507}]},"public_addr":"172.21.15.146:6809/460262507","cluster_addr":"172.21.15.146:6811/460262507","heartbeat_back_addr":"172.21.15.146:6815/460262507","heartbeat_front_addr":"172.21.15.146:6813/460262507","state":["exists","up"]},{"osd":6,"uuid":"c32e7fed-370d-4401-8ce0-6a4335b0d168","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6823","nonce":1392897493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6827","nonce":1392897493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6831","nonce":1392897493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6829","nonce":1392897493}]},"public_addr":"172.21.15.146:6823/1392897493","cluster_addr":"172.21.15.146:6827/1392897493","heartbeat_back_addr":"172.21.15.146:6831/1392897493","heartbeat_front_addr":"172.21.15.146:6829/1392897493","state":["exists","up"]},{"osd":7,"uuid":"75a79b44-fa5f-426d-a7d3-71f21f78835c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6817","nonce":3268323156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6819","nonce":3268323156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6825","nonce":3268323156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6822","nonce":3268323156}]},"public_addr":"172.21.15.146:6817/3268323156","cluster_addr":"172.21.15.146:6819/3268323156","heartbeat_back_addr":"172.21.15.146:6825/3268323156","heartbeat_front_addr":"172.21.15.146:6822/3268323156","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.310437+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.295850+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.294674+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.526855+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.300023+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.595160+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.346046+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.558741+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"13.10","mappings":[{"from":1,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T09:30:04.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.829+0000 7f9f2a200640 1 -- 172.21.15.6:0/2031536677 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f9f0406b4d0 msgr2=0x7f9f0406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:04.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.829+0000 7f9f2a200640 1 --2- 172.21.15.6:0/2031536677 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f9f0406b4d0 0x7f9f0406d990 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f9f2406b640 tx=0x7f9f10009040 comp rx=0 tx=0).stop 2024-03-04T09:30:04.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.829+0000 7f9f2a200640 1 -- 172.21.15.6:0/2031536677 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9f240af1a0 msgr2=0x7f9f24151b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:04.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.829+0000 7f9f2a200640 1 --2- 172.21.15.6:0/2031536677 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9f240af1a0 0x7f9f24151b00 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f9f1402b930 tx=0x7f9f140046c0 comp rx=0 tx=0).stop 2024-03-04T09:30:04.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.829+0000 7f9f2a200640 1 -- 172.21.15.6:0/2031536677 shutdown_connections 2024-03-04T09:30:04.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.829+0000 7f9f2a200640 1 --2- 172.21.15.6:0/2031536677 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9f240cdc20 0x7f9f2414eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:04.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.829+0000 7f9f2a200640 1 --2- 172.21.15.6:0/2031536677 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f9f0406b4d0 0x7f9f0406d990 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:04.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.829+0000 7f9f2a200640 1 --2- 172.21.15.6:0/2031536677 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9f240af580 0x7f9f24152f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:04.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.829+0000 7f9f2a200640 1 --2- 172.21.15.6:0/2031536677 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9f240af1a0 0x7f9f24151b00 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:04.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.829+0000 7f9f2a200640 1 -- 172.21.15.6:0/2031536677 >> 172.21.15.6:0/2031536677 conn(0x7f9f2406afb0 msgr2=0x7f9f241526a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:04.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.830+0000 7f9f2a200640 1 -- 172.21.15.6:0/2031536677 shutdown_connections 2024-03-04T09:30:04.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:04.830+0000 7f9f2a200640 1 -- 172.21.15.6:0/2031536677 wait complete. 2024-03-04T09:30:04.843 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-04T09:30:04.843 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:30:05.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.065+0000 7fe8c7200640 1 Processor -- start 2024-03-04T09:30:05.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.066+0000 7fe8c7200640 1 -- start start 2024-03-04T09:30:05.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.068+0000 7fe8c7200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fe8c80d4a10 0x7fe8c80d0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:05.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.068+0000 7fe8c7200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fe8c80d0c80 0x7fe8c80b44d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:05.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.068+0000 7fe8c7200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8c80b3710 con 0x7fe8c815e0b0 2024-03-04T09:30:05.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.068+0000 7fe8c7200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8c80d10f0 con 0x7fe8c80d4a10 2024-03-04T09:30:05.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.068+0000 7fe8c7200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8c80b09e0 con 0x7fe8c80d0c80 2024-03-04T09:30:05.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.068+0000 7fe8c5e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fe8c815e0b0 0x7fe8c80cf310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:37232/0 (socket says 172.21.15.6:37232) 2024-03-04T09:30:05.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.068+0000 7fe8c5e00640 1 -- 172.21.15.6:0/2333923715 learned_addr learned my addr 172.21.15.6:0/2333923715 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:30:05.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.068+0000 7fe8c5400640 1 --2- 172.21.15.6:0/2333923715 >> v2:172.21.15.6:3300/0 conn(0x7fe8c80d4a10 0x7fe8c80d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:05.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.068+0000 7fe8c6800640 1 --2- 172.21.15.6:0/2333923715 >> v2:172.21.15.146:3300/0 conn(0x7fe8c80d0c80 0x7fe8c80b44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:05.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.069+0000 7fe8c4a00640 1 -- 172.21.15.6:0/2333923715 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1791576892 0 0) 0x7fe8c80b3710 con 0x7fe8c815e0b0 2024-03-04T09:30:05.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.069+0000 7fe8c5400640 1 -- 172.21.15.6:0/2333923715 >> v2:172.21.15.146:3300/0 conn(0x7fe8c80d0c80 msgr2=0x7fe8c80b44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:05.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.069+0000 7fe8c5400640 1 --2- 172.21.15.6:0/2333923715 >> v2:172.21.15.146:3300/0 conn(0x7fe8c80d0c80 0x7fe8c80b44d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:05.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.069+0000 7fe8c5400640 1 -- 172.21.15.6:0/2333923715 >> v1:172.21.15.6:6789/0 conn(0x7fe8c815e0b0 legacy=0x7fe8c80cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:05.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.069+0000 7fe8c5400640 1 -- 172.21.15.6:0/2333923715 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8c80b39f0 con 0x7fe8c80d4a10 2024-03-04T09:30:05.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.070+0000 7fe8c5400640 1 --2- 172.21.15.6:0/2333923715 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe8c80d4a10 0x7fe8c80d0740 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7fe8c808d6d0 tx=0x7fe8b402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea3399c79a9f4c89 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:05.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.071+0000 7fe8c4a00640 1 -- 172.21.15.6:0/2333923715 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8b4035020 con 0x7fe8c80d4a10 2024-03-04T09:30:05.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.071+0000 7fe8c4a00640 1 -- 172.21.15.6:0/2333923715 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8b4031830 con 0x7fe8c80d4a10 2024-03-04T09:30:05.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.071+0000 7fe8c4a00640 1 -- 172.21.15.6:0/2333923715 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8b4031c00 con 0x7fe8c80d4a10 2024-03-04T09:30:05.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.071+0000 7fe8c7200640 1 -- 172.21.15.6:0/2333923715 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe8c80d4a10 msgr2=0x7fe8c80d0740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:05.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.071+0000 7fe8c7200640 1 --2- 172.21.15.6:0/2333923715 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe8c80d4a10 0x7fe8c80d0740 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7fe8c808d6d0 tx=0x7fe8b402b8d0 comp rx=0 tx=0).stop 2024-03-04T09:30:05.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.072+0000 7fe8c7200640 1 -- 172.21.15.6:0/2333923715 shutdown_connections 2024-03-04T09:30:05.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.072+0000 7fe8c7200640 1 --2- 172.21.15.6:0/2333923715 >> v2:172.21.15.146:3300/0 conn(0x7fe8c80d0c80 0x7fe8c80b44d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:05.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.072+0000 7fe8c7200640 1 --2- 172.21.15.6:0/2333923715 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe8c80d4a10 0x7fe8c80d0740 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:05.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.072+0000 7fe8c7200640 1 -- 172.21.15.6:0/2333923715 >> 172.21.15.6:0/2333923715 conn(0x7fe8c80d3d30 msgr2=0x7fe8c80d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:05.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.072+0000 7fe8c7200640 1 -- 172.21.15.6:0/2333923715 shutdown_connections 2024-03-04T09:30:05.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.072+0000 7fe8c7200640 1 -- 172.21.15.6:0/2333923715 wait complete. 2024-03-04T09:30:05.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.073+0000 7fe8c7200640 1 Processor -- start 2024-03-04T09:30:05.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.073+0000 7fe8c7200640 1 -- start start 2024-03-04T09:30:05.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.074+0000 7fe8c7200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe8c80d0c80 0x7fe8c814f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:05.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.074+0000 7fe8c7200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe8c80d4a10 0x7fe8c80cd8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:05.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.074+0000 7fe8c7200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe8c815e0b0 0x7fe8c80cae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:05.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.074+0000 7fe8c7200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8c80b3fb0 con 0x7fe8c80d4a10 2024-03-04T09:30:05.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.074+0000 7fe8c7200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8c80cfa20 con 0x7fe8c80d0c80 2024-03-04T09:30:05.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.074+0000 7fe8c7200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8c815e490 con 0x7fe8c815e0b0 2024-03-04T09:30:05.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.075+0000 7fe8c5400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe8c80d4a10 0x7fe8c80cd8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:05.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.075+0000 7fe8c5400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe8c80d4a10 0x7fe8c80cd8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:34244/0 (socket says 172.21.15.6:34244) 2024-03-04T09:30:05.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.075+0000 7fe8c5400640 1 -- 172.21.15.6:0/2385548033 learned_addr learned my addr 172.21.15.6:0/2385548033 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:30:05.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.075+0000 7fe8c5e00640 1 --2- 172.21.15.6:0/2385548033 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe8c80d0c80 0x7fe8c814f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:05.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.075+0000 7fe8c6800640 1 --2- 172.21.15.6:0/2385548033 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe8c815e0b0 0x7fe8c80cae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:05.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.075+0000 7fe8c5400640 1 -- 172.21.15.6:0/2385548033 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe8c815e0b0 msgr2=0x7fe8c80cae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:05.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.075+0000 7fe8c5400640 1 --2- 172.21.15.6:0/2385548033 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe8c815e0b0 0x7fe8c80cae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:05.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.075+0000 7fe8c5400640 1 -- 172.21.15.6:0/2385548033 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe8c80d0c80 msgr2=0x7fe8c814f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:05.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.075+0000 7fe8c5400640 1 --2- 172.21.15.6:0/2385548033 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe8c80d0c80 0x7fe8c814f310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:05.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.075+0000 7fe8c5400640 1 -- 172.21.15.6:0/2385548033 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8c8160430 con 0x7fe8c80d4a10 2024-03-04T09:30:05.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.075+0000 7fe8c5e00640 1 --2- 172.21.15.6:0/2385548033 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe8c80d0c80 0x7fe8c814f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T09:30:05.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.076+0000 7fe8c5400640 1 --2- 172.21.15.6:0/2385548033 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe8c80d4a10 0x7fe8c80cd8e0 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7fe8b4009720 tx=0x7fe8b40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:05.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.076+0000 7fe8be000640 1 -- 172.21.15.6:0/2385548033 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8b4035070 con 0x7fe8c80d4a10 2024-03-04T09:30:05.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.076+0000 7fe8be000640 1 -- 172.21.15.6:0/2385548033 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8b4004230 con 0x7fe8c80d4a10 2024-03-04T09:30:05.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.076+0000 7fe8be000640 1 -- 172.21.15.6:0/2385548033 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8b403a6f0 con 0x7fe8c80d4a10 2024-03-04T09:30:05.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.076+0000 7fe8c7200640 1 -- 172.21.15.6:0/2385548033 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8c8160710 con 0x7fe8c80d4a10 2024-03-04T09:30:05.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.076+0000 7fe8c7200640 1 -- 172.21.15.6:0/2385548033 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8c81611f0 con 0x7fe8c80d4a10 2024-03-04T09:30:05.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.077+0000 7fe8c7200640 1 -- 172.21.15.6:0/2385548033 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8c804f330 con 0x7fe8c80d4a10 2024-03-04T09:30:05.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.078+0000 7fe8be000640 1 -- 172.21.15.6:0/2385548033 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe8b4038020 con 0x7fe8c80d4a10 2024-03-04T09:30:05.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.079+0000 7fe8be000640 1 --2- 172.21.15.6:0/2385548033 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fe8a806b520 0x7fe8a806d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:05.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.079+0000 7fe8be000640 1 -- 172.21.15.6:0/2385548033 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fe8b40abee0 con 0x7fe8c80d4a10 2024-03-04T09:30:05.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.080+0000 7fe8c5e00640 1 --2- 172.21.15.6:0/2385548033 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fe8a806b520 0x7fe8a806d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:05.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.081+0000 7fe8c5e00640 1 --2- 172.21.15.6:0/2385548033 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fe8a806b520 0x7fe8a806d9e0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fe8b8000f60 tx=0x7fe8b8009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:05.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.088+0000 7fe8be000640 1 -- 172.21.15.6:0/2385548033 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe8b4040070 con 0x7fe8c80d4a10 2024-03-04T09:30:05.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.324+0000 7fe8c7200640 1 -- 172.21.15.6:0/2385548033 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe8c80b29b0 con 0x7fe8c80d4a10 2024-03-04T09:30:05.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.326+0000 7fe8be000640 1 -- 172.21.15.6:0/2385548033 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) v1 ==== 94+0+4976 (secure 0 0 0) 0x7fe8b407adb0 con 0x7fe8c80d4a10 2024-03-04T09:30:05.323 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:30:05.323 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":51,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5827,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6836/3447344578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":3447344578},{"type":"v1","addr":"172.21.15.146:6836","nonce":3447344578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":46},{"gid":5830,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/168994142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":168994142},{"type":"v1","addr":"172.21.15.146:6837","nonce":168994142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":46},{"gid":6735,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/3369489999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":3369489999},{"type":"v1","addr":"172.21.15.6:6833","nonce":3369489999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":45}],"filesystems":[{"mdsmap":{"epoch":51,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:29:47.357677+0000","modified":"2024-03-04T09:29:50.579389+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6744},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6744":{"gid":6744,"name":"a","rank":0,"incarnation":47,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6835/1667002065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1667002065},{"type":"v1","addr":"172.21.15.6:6835","nonce":1667002065}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6744,"qdb_cluster":[6744]},"id":6}]} 2024-03-04T09:30:05.323 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 51 2024-03-04T09:30:05.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.328+0000 7fe8c7200640 1 -- 172.21.15.6:0/2385548033 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fe8a806b520 msgr2=0x7fe8a806d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:05.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.328+0000 7fe8c7200640 1 --2- 172.21.15.6:0/2385548033 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fe8a806b520 0x7fe8a806d9e0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fe8b8000f60 tx=0x7fe8b8009040 comp rx=0 tx=0).stop 2024-03-04T09:30:05.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.328+0000 7fe8c7200640 1 -- 172.21.15.6:0/2385548033 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe8c80d4a10 msgr2=0x7fe8c80cd8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:05.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.328+0000 7fe8c7200640 1 --2- 172.21.15.6:0/2385548033 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe8c80d4a10 0x7fe8c80cd8e0 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7fe8b4009720 tx=0x7fe8b40046c0 comp rx=0 tx=0).stop 2024-03-04T09:30:05.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.328+0000 7fe8c7200640 1 -- 172.21.15.6:0/2385548033 shutdown_connections 2024-03-04T09:30:05.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.328+0000 7fe8c7200640 1 --2- 172.21.15.6:0/2385548033 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fe8a806b520 0x7fe8a806d9e0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:05.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.328+0000 7fe8c7200640 1 --2- 172.21.15.6:0/2385548033 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe8c815e0b0 0x7fe8c80cae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:05.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.328+0000 7fe8c7200640 1 --2- 172.21.15.6:0/2385548033 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe8c80d4a10 0x7fe8c80cd8e0 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:05.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.328+0000 7fe8c7200640 1 --2- 172.21.15.6:0/2385548033 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe8c80d0c80 0x7fe8c814f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:05.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.328+0000 7fe8c7200640 1 -- 172.21.15.6:0/2385548033 >> 172.21.15.6:0/2385548033 conn(0x7fe8c80d3d30 msgr2=0x7fe8c80cf070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:05.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.328+0000 7fe8c7200640 1 -- 172.21.15.6:0/2385548033 shutdown_connections 2024-03-04T09:30:05.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.329+0000 7fe8c7200640 1 -- 172.21.15.6:0/2385548033 wait complete. 2024-03-04T09:30:05.350 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:30:05.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.565+0000 7f7d6e800640 1 Processor -- start 2024-03-04T09:30:05.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.566+0000 7f7d6e800640 1 -- start start 2024-03-04T09:30:05.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.567+0000 7f7d6e800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f7d68145300 0x7f7d680b49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:05.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.567+0000 7f7d6e800640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d680714d0 con 0x7f7d680cd060 2024-03-04T09:30:05.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.567+0000 7f7d6e800640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d680717b0 con 0x7f7d68070f10 2024-03-04T09:30:05.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.567+0000 7f7d6e800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d68071a90 con 0x7f7d68145300 2024-03-04T09:30:05.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.568+0000 7f7d6d400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f7d68145300 0x7f7d680b49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:05.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.568+0000 7f7d6d400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f7d68145300 0x7f7d680b49a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:34258/0 (socket says 172.21.15.6:34258) 2024-03-04T09:30:05.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.568+0000 7f7d6d400640 1 -- 172.21.15.6:0/613798462 learned_addr learned my addr 172.21.15.6:0/613798462 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:30:05.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.568+0000 7f7d6de00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f7d680cd060 0x7f7d6815c570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:37238/0 (socket says 172.21.15.6:37238) 2024-03-04T09:30:05.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.568+0000 7f7d6d400640 1 -- 172.21.15.6:0/613798462 >> v1:172.21.15.146:6789/0 conn(0x7f7d68070f10 legacy=0x7f7d680cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:05.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.568+0000 7f7d6d400640 1 -- 172.21.15.6:0/613798462 >> v1:172.21.15.6:6789/0 conn(0x7f7d680cd060 legacy=0x7f7d6815c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:05.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.568+0000 7f7d6d400640 1 -- 172.21.15.6:0/613798462 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d68071d70 con 0x7f7d68145300 2024-03-04T09:30:05.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.568+0000 7f7d67e00640 1 -- 172.21.15.6:0/613798462 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1467602652 0 0) 0x7f7d680714d0 con 0x7f7d680cd060 2024-03-04T09:30:05.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.569+0000 7f7d6d400640 1 --2- 172.21.15.6:0/613798462 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7d68145300 0x7f7d680b49a0 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f7d5c009870 tx=0x7f7d5c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10379deda5e48170 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:05.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.570+0000 7f7d67e00640 1 -- 172.21.15.6:0/613798462 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d5c035020 con 0x7f7d68145300 2024-03-04T09:30:05.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.570+0000 7f7d67e00640 1 -- 172.21.15.6:0/613798462 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d5c031870 con 0x7f7d68145300 2024-03-04T09:30:05.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.570+0000 7f7d67e00640 1 -- 172.21.15.6:0/613798462 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d5c031c40 con 0x7f7d68145300 2024-03-04T09:30:05.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.570+0000 7f7d6e800640 1 -- 172.21.15.6:0/613798462 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7d68145300 msgr2=0x7f7d680b49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:05.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.570+0000 7f7d6e800640 1 --2- 172.21.15.6:0/613798462 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7d68145300 0x7f7d680b49a0 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f7d5c009870 tx=0x7f7d5c02b910 comp rx=0 tx=0).stop 2024-03-04T09:30:05.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.570+0000 7f7d6e800640 1 -- 172.21.15.6:0/613798462 shutdown_connections 2024-03-04T09:30:05.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.570+0000 7f7d6e800640 1 --2- 172.21.15.6:0/613798462 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7d68145300 0x7f7d680b49a0 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:05.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.570+0000 7f7d6e800640 1 -- 172.21.15.6:0/613798462 >> 172.21.15.6:0/613798462 conn(0x7f7d6806b3d0 msgr2=0x7f7d6806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:05.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.570+0000 7f7d6e800640 1 -- 172.21.15.6:0/613798462 shutdown_connections 2024-03-04T09:30:05.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.570+0000 7f7d6e800640 1 -- 172.21.15.6:0/613798462 wait complete. 2024-03-04T09:30:05.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.573+0000 7f7d6e800640 1 Processor -- start 2024-03-04T09:30:05.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.573+0000 7f7d6e800640 1 -- start start 2024-03-04T09:30:05.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.575+0000 7f7d6e800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7d68070f10 0x7f7d681f6b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:05.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.575+0000 7f7d6e800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7d680cd060 0x7f7d681e6ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:05.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.575+0000 7f7d6e800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7d68145300 0x7f7d681e8d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:05.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.575+0000 7f7d6e800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d6815cc80 con 0x7f7d68070f10 2024-03-04T09:30:05.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.575+0000 7f7d6e800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d681456e0 con 0x7f7d68145300 2024-03-04T09:30:05.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.575+0000 7f7d6e800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d6815ce00 con 0x7f7d680cd060 2024-03-04T09:30:05.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.575+0000 7f7d6d400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7d68070f10 0x7f7d681f6b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:05.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.575+0000 7f7d6d400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7d68070f10 0x7f7d681f6b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:34274/0 (socket says 172.21.15.6:34274) 2024-03-04T09:30:05.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.575+0000 7f7d6d400640 1 -- 172.21.15.6:0/281567792 learned_addr learned my addr 172.21.15.6:0/281567792 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:30:05.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.575+0000 7f7d6d400640 1 -- 172.21.15.6:0/281567792 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7d680cd060 msgr2=0x7f7d681e6ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:05.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.575+0000 7f7d6d400640 1 --2- 172.21.15.6:0/281567792 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7d680cd060 0x7f7d681e6ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:05.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.575+0000 7f7d6d400640 1 -- 172.21.15.6:0/281567792 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7d68145300 msgr2=0x7f7d681e8d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:05.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.575+0000 7f7d6d400640 1 --2- 172.21.15.6:0/281567792 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7d68145300 0x7f7d681e8d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:05.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.575+0000 7f7d6d400640 1 -- 172.21.15.6:0/281567792 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d681e7990 con 0x7f7d68070f10 2024-03-04T09:30:05.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.576+0000 7f7d6d400640 1 --2- 172.21.15.6:0/281567792 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7d68070f10 0x7f7d681f6b90 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f7d5c0090b0 tx=0x7f7d5c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:05.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.576+0000 7f7d65600640 1 -- 172.21.15.6:0/281567792 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d5c03ab50 con 0x7f7d68070f10 2024-03-04T09:30:05.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.576+0000 7f7d65600640 1 -- 172.21.15.6:0/281567792 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d5c03ad10 con 0x7f7d68070f10 2024-03-04T09:30:05.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.576+0000 7f7d65600640 1 -- 172.21.15.6:0/281567792 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d5c004270 con 0x7f7d68070f10 2024-03-04T09:30:05.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.576+0000 7f7d6e800640 1 -- 172.21.15.6:0/281567792 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d681e7c70 con 0x7f7d68070f10 2024-03-04T09:30:05.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.576+0000 7f7d6e800640 1 -- 172.21.15.6:0/281567792 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d681e8230 con 0x7f7d68070f10 2024-03-04T09:30:05.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.578+0000 7f7d6e800640 1 -- 172.21.15.6:0/281567792 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d6804f330 con 0x7f7d68070f10 2024-03-04T09:30:05.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.578+0000 7f7d65600640 1 -- 172.21.15.6:0/281567792 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7d5c038020 con 0x7f7d68070f10 2024-03-04T09:30:05.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.579+0000 7f7d65600640 1 --2- 172.21.15.6:0/281567792 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f7d4806b4d0 0x7f7d4806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:05.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.579+0000 7f7d65600640 1 -- 172.21.15.6:0/281567792 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f7d5c0ac200 con 0x7f7d68070f10 2024-03-04T09:30:05.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.580+0000 7f7d6ca00640 1 --2- 172.21.15.6:0/281567792 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f7d4806b4d0 0x7f7d4806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:05.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.581+0000 7f7d6ca00640 1 --2- 172.21.15.6:0/281567792 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f7d4806b4d0 0x7f7d4806d990 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f7d68145060 tx=0x7f7d540079b0 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:05.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.584+0000 7f7d65600640 1 -- 172.21.15.6:0/281567792 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7d5c03f070 con 0x7f7d68070f10 2024-03-04T09:30:05.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.953+0000 7f7d6e800640 1 -- 172.21.15.6:0/281567792 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7d6815be10 con 0x7f7d68070f10 2024-03-04T09:30:05.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.954+0000 7f7d65600640 1 -- 172.21.15.6:0/281567792 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) v1 ==== 94+0+4976 (secure 0 0 0) 0x7f7d5c07b060 con 0x7f7d68070f10 2024-03-04T09:30:05.951 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:30:05.951 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":51,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5827,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6836/3447344578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":3447344578},{"type":"v1","addr":"172.21.15.146:6836","nonce":3447344578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":46},{"gid":5830,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/168994142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":168994142},{"type":"v1","addr":"172.21.15.146:6837","nonce":168994142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":46},{"gid":6735,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/3369489999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":3369489999},{"type":"v1","addr":"172.21.15.6:6833","nonce":3369489999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":45}],"filesystems":[{"mdsmap":{"epoch":51,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:29:47.357677+0000","modified":"2024-03-04T09:29:50.579389+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6744},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6744":{"gid":6744,"name":"a","rank":0,"incarnation":47,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6835/1667002065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1667002065},{"type":"v1","addr":"172.21.15.6:6835","nonce":1667002065}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6744,"qdb_cluster":[6744]},"id":6}]} 2024-03-04T09:30:05.952 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 51 2024-03-04T09:30:05.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.957+0000 7f7d6e800640 1 -- 172.21.15.6:0/281567792 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f7d4806b4d0 msgr2=0x7f7d4806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:05.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.957+0000 7f7d6e800640 1 --2- 172.21.15.6:0/281567792 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f7d4806b4d0 0x7f7d4806d990 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f7d68145060 tx=0x7f7d540079b0 comp rx=0 tx=0).stop 2024-03-04T09:30:05.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.957+0000 7f7d6e800640 1 -- 172.21.15.6:0/281567792 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7d68070f10 msgr2=0x7f7d681f6b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:05.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.957+0000 7f7d6e800640 1 --2- 172.21.15.6:0/281567792 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7d68070f10 0x7f7d681f6b90 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f7d5c0090b0 tx=0x7f7d5c0046c0 comp rx=0 tx=0).stop 2024-03-04T09:30:05.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.958+0000 7f7d6e800640 1 -- 172.21.15.6:0/281567792 shutdown_connections 2024-03-04T09:30:05.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.958+0000 7f7d6e800640 1 --2- 172.21.15.6:0/281567792 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7d68145300 0x7f7d681e8d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:05.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.958+0000 7f7d6e800640 1 --2- 172.21.15.6:0/281567792 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f7d4806b4d0 0x7f7d4806d990 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:05.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.958+0000 7f7d6e800640 1 --2- 172.21.15.6:0/281567792 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7d680cd060 0x7f7d681e6ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:05.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.958+0000 7f7d6e800640 1 --2- 172.21.15.6:0/281567792 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7d68070f10 0x7f7d681f6b90 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:05.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.958+0000 7f7d6e800640 1 -- 172.21.15.6:0/281567792 >> 172.21.15.6:0/281567792 conn(0x7f7d6806b3d0 msgr2=0x7f7d680cbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:05.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.958+0000 7f7d6e800640 1 -- 172.21.15.6:0/281567792 shutdown_connections 2024-03-04T09:30:05.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:05.958+0000 7f7d6e800640 1 -- 172.21.15.6:0/281567792 wait complete. 2024-03-04T09:30:05.985 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T09:30:06.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.179+0000 7fece1400640 1 Processor -- start 2024-03-04T09:30:06.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.179+0000 7fece1400640 1 -- start start 2024-03-04T09:30:06.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.180+0000 7fece1400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fecdc0af580 0x7fecdc15c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:06.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.181+0000 7fece1400640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecdc15cc40 con 0x7fecdc0af1a0 2024-03-04T09:30:06.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.181+0000 7fece1400640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecdc15cf20 con 0x7fecdc0cdc20 2024-03-04T09:30:06.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.181+0000 7fece1400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecdc15d200 con 0x7fecdc0af580 2024-03-04T09:30:06.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.181+0000 7fecda800640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fecdc0af1a0 0x7fecdc0b4c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:37244/0 (socket says 172.21.15.6:37244) 2024-03-04T09:30:06.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.181+0000 7fecda800640 1 -- 172.21.15.6:0/2938762332 learned_addr learned my addr 172.21.15.6:0/2938762332 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:30:06.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.181+0000 7fecd3e00640 1 --2- 172.21.15.6:0/2938762332 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fecdc0af580 0x7fecdc15c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:06.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.181+0000 7fecd9e00640 1 -- 172.21.15.6:0/2938762332 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2848299125 0 0) 0x7fecdc15cc40 con 0x7fecdc0af1a0 2024-03-04T09:30:06.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.182+0000 7fecd9e00640 1 -- 172.21.15.6:0/2938762332 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fecc0003520 con 0x7fecdc0af1a0 2024-03-04T09:30:06.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.182+0000 7fecd9e00640 1 -- 172.21.15.6:0/2938762332 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 179609255 0 0) 0x7fecc0003520 con 0x7fecdc0af1a0 2024-03-04T09:30:06.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.182+0000 7fecd9e00640 1 -- 172.21.15.6:0/2938762332 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fecdc0af580 msgr2=0x7fecdc15c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:06.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.182+0000 7fecd9e00640 1 --2- 172.21.15.6:0/2938762332 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fecdc0af580 0x7fecdc15c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:06.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.182+0000 7fecd9e00640 1 -- 172.21.15.6:0/2938762332 >> v1:172.21.15.146:6789/0 conn(0x7fecdc0cdc20 legacy=0x7fecdc0b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:06.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.182+0000 7fecd9e00640 1 -- 172.21.15.6:0/2938762332 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecdc15d4e0 con 0x7fecdc0af1a0 2024-03-04T09:30:06.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.182+0000 7fecd9e00640 1 -- 172.21.15.6:0/2938762332 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7feccc001670 con 0x7fecdc0af1a0 2024-03-04T09:30:06.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.182+0000 7fecd9e00640 1 -- 172.21.15.6:0/2938762332 >> v1:172.21.15.6:6789/0 conn(0x7fecdc0af1a0 legacy=0x7fecdc0b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:06.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.182+0000 7fecd9e00640 1 --2- 172.21.15.6:0/2938762332 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fecc0003a50 0x7fecc0003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:06.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.182+0000 7fecd9e00640 1 --2- 172.21.15.6:0/2938762332 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fecc0004800 0x7fecc0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:06.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.182+0000 7fecd9e00640 1 --2- 172.21.15.6:0/2938762332 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fecdc0af580 0x7fecc00052a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:06.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.182+0000 7fecd9e00640 1 -- 172.21.15.6:0/2938762332 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecdc15d200 con 0x7fecc0003a50 2024-03-04T09:30:06.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.182+0000 7fecd9e00640 1 -- 172.21.15.6:0/2938762332 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecc000bba0 con 0x7fecc0004800 2024-03-04T09:30:06.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.182+0000 7fecd9e00640 1 -- 172.21.15.6:0/2938762332 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecc000bda0 con 0x7fecdc0af580 2024-03-04T09:30:06.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.182+0000 7fecda800640 1 --2- 172.21.15.6:0/2938762332 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fecc0003a50 0x7fecc0003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:06.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.183+0000 7fecdb200640 1 --2- 172.21.15.6:0/2938762332 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fecdc0af580 0x7fecc00052a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:06.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.183+0000 7fecd3e00640 1 --2- 172.21.15.6:0/2938762332 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fecc0004800 0x7fecc0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:06.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.183+0000 7fecda800640 1 -- 172.21.15.6:0/2938762332 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fecdc0af580 msgr2=0x7fecc00052a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:06.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.183+0000 7fecda800640 1 --2- 172.21.15.6:0/2938762332 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fecdc0af580 0x7fecc00052a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:06.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.183+0000 7fecda800640 1 -- 172.21.15.6:0/2938762332 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fecc0004800 msgr2=0x7fecc0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:06.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.183+0000 7fecda800640 1 --2- 172.21.15.6:0/2938762332 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fecc0004800 0x7fecc0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:06.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.183+0000 7fecda800640 1 -- 172.21.15.6:0/2938762332 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecc0003520 con 0x7fecc0003a50 2024-03-04T09:30:06.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.183+0000 7fecd3e00640 1 --2- 172.21.15.6:0/2938762332 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fecc0004800 0x7fecc0004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:30:06.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.184+0000 7fecda800640 1 --2- 172.21.15.6:0/2938762332 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fecc0003a50 0x7fecc0003e70 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7feccc001eb0 tx=0x7fecc0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c596c42343f5f648 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:06.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.184+0000 7fecd9e00640 1 -- 172.21.15.6:0/2938762332 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feccc007d30 con 0x7fecc0003a50 2024-03-04T09:30:06.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.185+0000 7fecd9e00640 1 -- 172.21.15.6:0/2938762332 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feccc030070 con 0x7fecc0003a50 2024-03-04T09:30:06.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.185+0000 7fecd9e00640 1 -- 172.21.15.6:0/2938762332 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feccc005d30 con 0x7fecc0003a50 2024-03-04T09:30:06.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.185+0000 7fece1400640 1 -- 172.21.15.6:0/2938762332 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fecc0003a50 msgr2=0x7fecc0003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:06.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.185+0000 7fece1400640 1 --2- 172.21.15.6:0/2938762332 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fecc0003a50 0x7fecc0003e70 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7feccc001eb0 tx=0x7fecc0005830 comp rx=0 tx=0).stop 2024-03-04T09:30:06.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.185+0000 7fecdb200640 1 -- 172.21.15.6:0/2938762332 reap_dead start 2024-03-04T09:30:06.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.185+0000 7fece1400640 1 -- 172.21.15.6:0/2938762332 shutdown_connections 2024-03-04T09:30:06.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.185+0000 7fece1400640 1 -- 172.21.15.6:0/2938762332 >> 172.21.15.6:0/2938762332 conn(0x7fecdc06afb0 msgr2=0x7fecdc06b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:06.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.186+0000 7fece1400640 1 -- 172.21.15.6:0/2938762332 shutdown_connections 2024-03-04T09:30:06.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.186+0000 7fece1400640 1 -- 172.21.15.6:0/2938762332 wait complete. 2024-03-04T09:30:06.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.187+0000 7fece1400640 1 Processor -- start 2024-03-04T09:30:06.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.187+0000 7fece1400640 1 -- start start 2024-03-04T09:30:06.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.188+0000 7fece1400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fecc0003a50 0x7fecdc14b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:06.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.188+0000 7fece1400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fecdc09d8e0 0x7fecdc071930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:06.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.188+0000 7fece1400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fecdc071e70 0x7fecdc14e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:06.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.188+0000 7fece1400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecdc15c600 con 0x7fecdc071e70 2024-03-04T09:30:06.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.188+0000 7fece1400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecdc0b36f0 con 0x7fecdc09d8e0 2024-03-04T09:30:06.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.188+0000 7fece1400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecdc15c780 con 0x7fecc0003a50 2024-03-04T09:30:06.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.189+0000 7fecdb200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fecdc071e70 0x7fecdc14e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:06.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.189+0000 7fecdb200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fecdc071e70 0x7fecdc14e550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:34294/0 (socket says 172.21.15.6:34294) 2024-03-04T09:30:06.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.189+0000 7fecdb200640 1 -- 172.21.15.6:0/984988988 learned_addr learned my addr 172.21.15.6:0/984988988 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:30:06.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.189+0000 7fecda800640 1 --2- 172.21.15.6:0/984988988 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fecc0003a50 0x7fecdc14b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:06.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.189+0000 7fecd3e00640 1 --2- 172.21.15.6:0/984988988 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fecdc09d8e0 0x7fecdc071930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:06.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.189+0000 7fecdb200640 1 -- 172.21.15.6:0/984988988 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fecc0003a50 msgr2=0x7fecdc14b9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:06.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.189+0000 7fecdb200640 1 --2- 172.21.15.6:0/984988988 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fecc0003a50 0x7fecdc14b9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:06.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.189+0000 7fecdb200640 1 -- 172.21.15.6:0/984988988 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fecdc09d8e0 msgr2=0x7fecdc071930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:06.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.189+0000 7fecdb200640 1 --2- 172.21.15.6:0/984988988 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fecdc09d8e0 0x7fecdc071930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:06.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.189+0000 7fecdb200640 1 -- 172.21.15.6:0/984988988 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecdc09c880 con 0x7fecdc071e70 2024-03-04T09:30:06.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.190+0000 7fecdb200640 1 --2- 172.21.15.6:0/984988988 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fecdc071e70 0x7fecdc14e550 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7fecdc06c0c0 tx=0x7fecd400ea00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:06.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.190+0000 7fecd2a00640 1 -- 172.21.15.6:0/984988988 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecd4003b80 con 0x7fecdc071e70 2024-03-04T09:30:06.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.190+0000 7fecd2a00640 1 -- 172.21.15.6:0/984988988 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecd4018070 con 0x7fecdc071e70 2024-03-04T09:30:06.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.190+0000 7fecd2a00640 1 -- 172.21.15.6:0/984988988 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecd40050d0 con 0x7fecdc071e70 2024-03-04T09:30:06.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.190+0000 7fece1400640 1 -- 172.21.15.6:0/984988988 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecdc14bef0 con 0x7fecdc071e70 2024-03-04T09:30:06.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.190+0000 7fece1400640 1 -- 172.21.15.6:0/984988988 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecdc09ce00 con 0x7fecdc071e70 2024-03-04T09:30:06.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.191+0000 7fece1400640 1 -- 172.21.15.6:0/984988988 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fecdc069510 con 0x7fecdc071e70 2024-03-04T09:30:06.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.193+0000 7fecd2a00640 1 -- 172.21.15.6:0/984988988 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fecd4005270 con 0x7fecdc071e70 2024-03-04T09:30:06.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.194+0000 7fecd2a00640 1 --2- 172.21.15.6:0/984988988 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fecb806b4d0 0x7fecb806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:06.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.194+0000 7fecd2a00640 1 -- 172.21.15.6:0/984988988 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fecd4089f40 con 0x7fecdc071e70 2024-03-04T09:30:06.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.195+0000 7fecda800640 1 --2- 172.21.15.6:0/984988988 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fecb806b4d0 0x7fecb806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:06.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.196+0000 7fecda800640 1 --2- 172.21.15.6:0/984988988 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fecb806b4d0 0x7fecb806d990 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7feccc034000 tx=0x7feccc009030 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:06.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.199+0000 7fecd2a00640 1 -- 172.21.15.6:0/984988988 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fecd4058d90 con 0x7fecdc071e70 2024-03-04T09:30:06.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.396+0000 7fece1400640 1 -- 172.21.15.6:0/984988988 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fecdc04f3a0 con 0x7fecdc071e70 2024-03-04T09:30:06.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.398+0000 7fecd2a00640 1 -- 172.21.15.6:0/984988988 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) v1 ==== 74+0+16662 (secure 0 0 0) 0x7fecd4058690 con 0x7fecdc071e70 2024-03-04T09:30:06.395 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:30:06.395 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":83,"fsid":"edce0578-5ae4-4402-b45a-552bdef84d7c","created":"2024-03-04T09:24:54.523780+0000","modified":"2024-03-04T09:30:03.005383+0000","last_up_change":"2024-03-04T09:25:07.089049+0000","last_in_change":"2024-03-04T09:24:57.982077+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T09:25:07.935198+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-03-04T09:29:45.280632+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-03-04T09:29:46.378588+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a96574f-81f1-4bc6-8463-757520b84a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6825","nonce":1364449401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6827","nonce":1364449401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6831","nonce":1364449401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6829","nonce":1364449401}]},"public_addr":"172.21.15.6:6825/1364449401","cluster_addr":"172.21.15.6:6827/1364449401","heartbeat_back_addr":"172.21.15.6:6831/1364449401","heartbeat_front_addr":"172.21.15.6:6829/1364449401","state":["exists","up"]},{"osd":1,"uuid":"7d79b9c1-6b87-476a-bc36-7db5ee64cd90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6809","nonce":2494103078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6811","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6813","nonce":2494103078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6821","nonce":2494103078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6817","nonce":2494103078}]},"public_addr":"172.21.15.6:6809/2494103078","cluster_addr":"172.21.15.6:6813/2494103078","heartbeat_back_addr":"172.21.15.6:6821/2494103078","heartbeat_front_addr":"172.21.15.6:6817/2494103078","state":["exists","up"]},{"osd":2,"uuid":"492331b9-4ee9-40b6-bacd-85d44cd6c45a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6812","nonce":1500066681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6816","nonce":1500066681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6824","nonce":1500066681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6820","nonce":1500066681}]},"public_addr":"172.21.15.6:6812/1500066681","cluster_addr":"172.21.15.6:6816/1500066681","heartbeat_back_addr":"172.21.15.6:6824/1500066681","heartbeat_front_addr":"172.21.15.6:6820/1500066681","state":["exists","up"]},{"osd":3,"uuid":"21bb2efe-9ba7-4bdc-bc31-180cd1102e21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6801","nonce":1558806123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6803","nonce":1558806123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6807","nonce":1558806123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6805","nonce":1558806123}]},"public_addr":"172.21.15.6:6801/1558806123","cluster_addr":"172.21.15.6:6803/1558806123","heartbeat_back_addr":"172.21.15.6:6807/1558806123","heartbeat_front_addr":"172.21.15.6:6805/1558806123","state":["exists","up"]},{"osd":4,"uuid":"6e2c4b71-19d9-4737-83d9-558705d8644a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6801","nonce":1526135780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6803","nonce":1526135780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6807","nonce":1526135780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6805","nonce":1526135780}]},"public_addr":"172.21.15.146:6801/1526135780","cluster_addr":"172.21.15.146:6803/1526135780","heartbeat_back_addr":"172.21.15.146:6807/1526135780","heartbeat_front_addr":"172.21.15.146:6805/1526135780","state":["exists","up"]},{"osd":5,"uuid":"fb69c525-f1be-4b7d-ab32-6c29b0d6fddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6809","nonce":460262507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6811","nonce":460262507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6815","nonce":460262507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6813","nonce":460262507}]},"public_addr":"172.21.15.146:6809/460262507","cluster_addr":"172.21.15.146:6811/460262507","heartbeat_back_addr":"172.21.15.146:6815/460262507","heartbeat_front_addr":"172.21.15.146:6813/460262507","state":["exists","up"]},{"osd":6,"uuid":"c32e7fed-370d-4401-8ce0-6a4335b0d168","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6823","nonce":1392897493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6827","nonce":1392897493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6831","nonce":1392897493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6829","nonce":1392897493}]},"public_addr":"172.21.15.146:6823/1392897493","cluster_addr":"172.21.15.146:6827/1392897493","heartbeat_back_addr":"172.21.15.146:6831/1392897493","heartbeat_front_addr":"172.21.15.146:6829/1392897493","state":["exists","up"]},{"osd":7,"uuid":"75a79b44-fa5f-426d-a7d3-71f21f78835c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6817","nonce":3268323156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6819","nonce":3268323156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6825","nonce":3268323156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6822","nonce":3268323156}]},"public_addr":"172.21.15.146:6817/3268323156","cluster_addr":"172.21.15.146:6819/3268323156","heartbeat_back_addr":"172.21.15.146:6825/3268323156","heartbeat_front_addr":"172.21.15.146:6822/3268323156","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.310437+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.295850+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.294674+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.526855+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.300023+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.595160+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.346046+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.558741+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"13.10","mappings":[{"from":1,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T09:30:06.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.400+0000 7fece1400640 1 -- 172.21.15.6:0/984988988 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fecb806b4d0 msgr2=0x7fecb806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:06.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.400+0000 7fece1400640 1 --2- 172.21.15.6:0/984988988 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fecb806b4d0 0x7fecb806d990 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7feccc034000 tx=0x7feccc009030 comp rx=0 tx=0).stop 2024-03-04T09:30:06.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.400+0000 7fece1400640 1 -- 172.21.15.6:0/984988988 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fecdc071e70 msgr2=0x7fecdc14e550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:06.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.400+0000 7fece1400640 1 --2- 172.21.15.6:0/984988988 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fecdc071e70 0x7fecdc14e550 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7fecdc06c0c0 tx=0x7fecd400ea00 comp rx=0 tx=0).stop 2024-03-04T09:30:06.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.400+0000 7fece1400640 1 -- 172.21.15.6:0/984988988 shutdown_connections 2024-03-04T09:30:06.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.400+0000 7fece1400640 1 --2- 172.21.15.6:0/984988988 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fecdc071e70 0x7fecdc14e550 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:06.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.400+0000 7fece1400640 1 --2- 172.21.15.6:0/984988988 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fecdc09d8e0 0x7fecdc071930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:06.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.400+0000 7fece1400640 1 --2- 172.21.15.6:0/984988988 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fecb806b4d0 0x7fecb806d990 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:06.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.400+0000 7fece1400640 1 --2- 172.21.15.6:0/984988988 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fecc0003a50 0x7fecdc14b9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:06.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.401+0000 7fece1400640 1 -- 172.21.15.6:0/984988988 >> 172.21.15.6:0/984988988 conn(0x7fecdc06afb0 msgr2=0x7fecdc06b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:06.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.401+0000 7fece1400640 1 -- 172.21.15.6:0/984988988 shutdown_connections 2024-03-04T09:30:06.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.401+0000 7fece1400640 1 -- 172.21.15.6:0/984988988 wait complete. 2024-03-04T09:30:06.417 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-04T09:30:06.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.626+0000 7f9b8e400640 1 Processor -- start 2024-03-04T09:30:06.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.626+0000 7f9b8e400640 1 -- start start 2024-03-04T09:30:06.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.627+0000 7f9b8e400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f9b88145300 0x7f9b880b49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:06.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.627+0000 7f9b8e400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9b88070f10 0x7f9b880cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:06.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.628+0000 7f9b8e400640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b880714d0 con 0x7f9b880cce90 2024-03-04T09:30:06.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.628+0000 7f9b8e400640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b880cd300 con 0x7f9b88145300 2024-03-04T09:30:06.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.628+0000 7f9b8e400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b880717b0 con 0x7f9b88070f10 2024-03-04T09:30:06.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.628+0000 7f9b87e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9b88070f10 0x7f9b880cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:06.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.628+0000 7f9b87e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9b88070f10 0x7f9b880cc950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.6:38786/0 (socket says 172.21.15.6:38786) 2024-03-04T09:30:06.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.628+0000 7f9b87e00640 1 -- 172.21.15.6:0/1974432901 learned_addr learned my addr 172.21.15.6:0/1974432901 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:30:06.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.629+0000 7f9b8d000640 1 --2- 172.21.15.6:0/1974432901 >> v2:172.21.15.146:3300/0 conn(0x7f9b88145300 0x7f9b880b49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:06.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.629+0000 7f9b87e00640 1 -- 172.21.15.6:0/1974432901 >> v2:172.21.15.146:3300/0 conn(0x7f9b88145300 msgr2=0x7f9b880b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:06.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.629+0000 7f9b87e00640 1 --2- 172.21.15.6:0/1974432901 >> v2:172.21.15.146:3300/0 conn(0x7f9b88145300 0x7f9b880b49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:06.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.629+0000 7f9b87e00640 1 -- 172.21.15.6:0/1974432901 >> v1:172.21.15.146:6789/0 conn(0x7f9b880cce90 legacy=0x7f9b880cd930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:06.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.629+0000 7f9b87e00640 1 -- 172.21.15.6:0/1974432901 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b88071a90 con 0x7f9b88070f10 2024-03-04T09:30:06.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.631+0000 7f9b87e00640 1 --2- 172.21.15.6:0/1974432901 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9b88070f10 0x7f9b880cc950 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f9b74009870 tx=0x7f9b7402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=653a9176bb5f5fd5 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:06.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.632+0000 7f9b87400640 1 -- 172.21.15.6:0/1974432901 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b74034070 con 0x7f9b88070f10 2024-03-04T09:30:06.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.632+0000 7f9b87400640 1 -- 172.21.15.6:0/1974432901 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b74030840 con 0x7f9b88070f10 2024-03-04T09:30:06.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.632+0000 7f9b87400640 1 -- 172.21.15.6:0/1974432901 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b74030c10 con 0x7f9b88070f10 2024-03-04T09:30:06.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.632+0000 7f9b8e400640 1 -- 172.21.15.6:0/1974432901 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9b88070f10 msgr2=0x7f9b880cc950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:06.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.632+0000 7f9b8e400640 1 --2- 172.21.15.6:0/1974432901 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9b88070f10 0x7f9b880cc950 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f9b74009870 tx=0x7f9b7402b980 comp rx=0 tx=0).stop 2024-03-04T09:30:06.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.632+0000 7f9b8e400640 1 -- 172.21.15.6:0/1974432901 shutdown_connections 2024-03-04T09:30:06.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.632+0000 7f9b8e400640 1 --2- 172.21.15.6:0/1974432901 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9b88070f10 0x7f9b880cc950 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:06.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.632+0000 7f9b8e400640 1 --2- 172.21.15.6:0/1974432901 >> v2:172.21.15.146:3300/0 conn(0x7f9b88145300 0x7f9b880b49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:06.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.633+0000 7f9b8e400640 1 -- 172.21.15.6:0/1974432901 >> 172.21.15.6:0/1974432901 conn(0x7f9b8806b3d0 msgr2=0x7f9b8806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:06.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.633+0000 7f9b8e400640 1 -- 172.21.15.6:0/1974432901 shutdown_connections 2024-03-04T09:30:06.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.633+0000 7f9b8e400640 1 -- 172.21.15.6:0/1974432901 wait complete. 2024-03-04T09:30:06.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.634+0000 7f9b8e400640 1 Processor -- start 2024-03-04T09:30:06.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.635+0000 7f9b8e400640 1 -- start start 2024-03-04T09:30:06.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.636+0000 7f9b8e400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9b88070f10 0x7f9b8813c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:06.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.636+0000 7f9b8e400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9b880cce90 0x7f9b8813f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:06.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.636+0000 7f9b8e400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9b88145300 0x7f9b880ae260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:06.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.636+0000 7f9b8e400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b881456e0 con 0x7f9b88070f10 2024-03-04T09:30:06.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.636+0000 7f9b8e400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b880b4ee0 con 0x7f9b88145300 2024-03-04T09:30:06.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.636+0000 7f9b8e400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b8815d580 con 0x7f9b880cce90 2024-03-04T09:30:06.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.636+0000 7f9b8d000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9b88070f10 0x7f9b8813c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:06.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.636+0000 7f9b8d000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9b88070f10 0x7f9b8813c670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:34302/0 (socket says 172.21.15.6:34302) 2024-03-04T09:30:06.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.636+0000 7f9b8d000640 1 -- 172.21.15.6:0/1992146784 learned_addr learned my addr 172.21.15.6:0/1992146784 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:30:06.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.636+0000 7f9b87e00640 1 --2- 172.21.15.6:0/1992146784 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9b880cce90 0x7f9b8813f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:06.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.636+0000 7f9b8d000640 1 -- 172.21.15.6:0/1992146784 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9b880cce90 msgr2=0x7f9b8813f740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:06.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.636+0000 7f9b8d000640 1 --2- 172.21.15.6:0/1992146784 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9b880cce90 0x7f9b8813f740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:06.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.636+0000 7f9b8da00640 1 --2- 172.21.15.6:0/1992146784 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9b88145300 0x7f9b880ae260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:06.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.636+0000 7f9b8d000640 1 -- 172.21.15.6:0/1992146784 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9b88145300 msgr2=0x7f9b880ae260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:06.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.637+0000 7f9b8d000640 1 --2- 172.21.15.6:0/1992146784 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9b88145300 0x7f9b880ae260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:06.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.637+0000 7f9b8d000640 1 -- 172.21.15.6:0/1992146784 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b8813ced0 con 0x7f9b88070f10 2024-03-04T09:30:06.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.637+0000 7f9b87e00640 1 --2- 172.21.15.6:0/1992146784 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9b880cce90 0x7f9b8813f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:30:06.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.637+0000 7f9b8da00640 1 --2- 172.21.15.6:0/1992146784 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9b88145300 0x7f9b880ae260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:30:06.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.637+0000 7f9b8d000640 1 --2- 172.21.15.6:0/1992146784 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9b88070f10 0x7f9b8813c670 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f9b7c00e870 tx=0x7f9b7c00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:06.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.637+0000 7f9b84c00640 1 -- 172.21.15.6:0/1992146784 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b7c017070 con 0x7f9b88070f10 2024-03-04T09:30:06.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.637+0000 7f9b84c00640 1 -- 172.21.15.6:0/1992146784 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b7c00c940 con 0x7f9b88070f10 2024-03-04T09:30:06.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.637+0000 7f9b84c00640 1 -- 172.21.15.6:0/1992146784 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b7c00cd30 con 0x7f9b88070f10 2024-03-04T09:30:06.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.637+0000 7f9b8e400640 1 -- 172.21.15.6:0/1992146784 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b880ae990 con 0x7f9b88070f10 2024-03-04T09:30:06.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.637+0000 7f9b8e400640 1 -- 172.21.15.6:0/1992146784 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b880aef50 con 0x7f9b88070f10 2024-03-04T09:30:06.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.639+0000 7f9b8e400640 1 -- 172.21.15.6:0/1992146784 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b880af190 con 0x7f9b88070f10 2024-03-04T09:30:06.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.639+0000 7f9b84c00640 1 -- 172.21.15.6:0/1992146784 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9b7c01a070 con 0x7f9b88070f10 2024-03-04T09:30:06.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.641+0000 7f9b84c00640 1 --2- 172.21.15.6:0/1992146784 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f9b6806b4d0 0x7f9b6806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:06.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.641+0000 7f9b84c00640 1 -- 172.21.15.6:0/1992146784 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f9b7c08a540 con 0x7f9b88070f10 2024-03-04T09:30:06.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.641+0000 7f9b87e00640 1 --2- 172.21.15.6:0/1992146784 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f9b6806b4d0 0x7f9b6806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:06.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.642+0000 7f9b87e00640 1 --2- 172.21.15.6:0/1992146784 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f9b6806b4d0 0x7f9b6806d990 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f9b74036000 tx=0x7f9b7403b040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:06.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.645+0000 7f9b84c00640 1 -- 172.21.15.6:0/1992146784 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9b880af190 con 0x7f9b88070f10 2024-03-04T09:30:06.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.877+0000 7f9b8e400640 1 -- 172.21.15.6:0/1992146784 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f9b88067830 con 0x7f9b88070f10 2024-03-04T09:30:06.874 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-03-04T09:30:06.878+0000 7f1606a00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-04T09:30:06.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.881+0000 7f9b84c00640 1 -- 172.21.15.6:0/1992146784 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v52) v1 ==== 161+0+0 (secure 0 0 0) 0x7f9b88067830 con 0x7f9b88070f10 2024-03-04T09:30:06.878 INFO:teuthology.orchestra.run.smithi006.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-04T09:30:06.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.884+0000 7f9b8e400640 1 -- 172.21.15.6:0/1992146784 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f9b6806b4d0 msgr2=0x7f9b6806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:06.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.884+0000 7f9b8e400640 1 --2- 172.21.15.6:0/1992146784 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f9b6806b4d0 0x7f9b6806d990 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f9b74036000 tx=0x7f9b7403b040 comp rx=0 tx=0).stop 2024-03-04T09:30:06.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.885+0000 7f9b8e400640 1 -- 172.21.15.6:0/1992146784 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9b88070f10 msgr2=0x7f9b8813c670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:06.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.885+0000 7f9b8e400640 1 --2- 172.21.15.6:0/1992146784 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9b88070f10 0x7f9b8813c670 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f9b7c00e870 tx=0x7f9b7c00ed30 comp rx=0 tx=0).stop 2024-03-04T09:30:06.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.885+0000 7f9b8e400640 1 -- 172.21.15.6:0/1992146784 shutdown_connections 2024-03-04T09:30:06.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.885+0000 7f9b8e400640 1 --2- 172.21.15.6:0/1992146784 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9b88145300 0x7f9b880ae260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:06.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.885+0000 7f9b8e400640 1 --2- 172.21.15.6:0/1992146784 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f9b6806b4d0 0x7f9b6806d990 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:06.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.885+0000 7f9b8e400640 1 --2- 172.21.15.6:0/1992146784 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9b880cce90 0x7f9b8813f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:06.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.885+0000 7f9b8e400640 1 --2- 172.21.15.6:0/1992146784 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9b88070f10 0x7f9b8813c670 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:06.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.885+0000 7f9b8e400640 1 -- 172.21.15.6:0/1992146784 >> 172.21.15.6:0/1992146784 conn(0x7f9b8806b3d0 msgr2=0x7f9b880b0120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:06.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.885+0000 7f9b8e400640 1 -- 172.21.15.6:0/1992146784 shutdown_connections 2024-03-04T09:30:06.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:06.885+0000 7f9b8e400640 1 -- 172.21.15.6:0/1992146784 wait complete. 2024-03-04T09:30:06.902 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-04T09:30:06.908 INFO:tasks.ceph.mds.a.smithi006.stderr: -3974> 2024-03-04T09:29:53.844+0000 7ff086000640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-04T09:30:06.974 INFO:tasks.ceph.mds.a.smithi006.stdout:starting mds.a at 2024-03-04T09:30:07.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:07.089+0000 7f0726e00640 1 Processor -- start 2024-03-04T09:30:07.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:07.090+0000 7f0726e00640 1 -- start start 2024-03-04T09:30:07.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:07.091+0000 7f0726e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f07200ccad0 0x7f07200cceb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:07.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:07.091+0000 7f0726e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f0720162bb0 0x7f0720163040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:07.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:07.091+0000 7f0726e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f07200cdab0 0x7f07200cdee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:07.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:07.091+0000 7f0726e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07200cd3f0 con 0x7f07200ccad0 2024-03-04T09:30:07.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:07.091+0000 7f0726e00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0720163950 con 0x7f0720162bb0 2024-03-04T09:30:07.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:07.091+0000 7f0726e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07200acd40 con 0x7f07200cdab0 2024-03-04T09:30:07.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:07.092+0000 7f071fe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f07200ccad0 0x7f07200cceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:07.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:07.092+0000 7f071fe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f07200ccad0 0x7f07200cceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:34332/0 (socket says 172.21.15.6:34332) 2024-03-04T09:30:07.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:07.092+0000 7f071fe00640 1 -- 172.21.15.6:0/2358104147 learned_addr learned my addr 172.21.15.6:0/2358104147 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:30:07.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:07.092+0000 7f0717400640 1 --2- 172.21.15.6:0/2358104147 >> v2:172.21.15.146:3300/0 conn(0x7f0720162bb0 0x7f0720163040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:07.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:07.092+0000 7f0724e00640 1 --2- 172.21.15.6:0/2358104147 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f07200cdab0 0x7f07200cdee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:07.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:07.092+0000 7f071fe00640 1 -- 172.21.15.6:0/2358104147 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f07200cdab0 msgr2=0x7f07200cdee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:07.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:07.092+0000 7f071fe00640 1 --2- 172.21.15.6:0/2358104147 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f07200cdab0 0x7f07200cdee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:07.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:07.092+0000 7f071fe00640 1 -- 172.21.15.6:0/2358104147 >> v2:172.21.15.146:3300/0 conn(0x7f0720162bb0 msgr2=0x7f0720163040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:07.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:07.092+0000 7f071fe00640 1 --2- 172.21.15.6:0/2358104147 >> v2:172.21.15.146:3300/0 conn(0x7f0720162bb0 0x7f0720163040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:07.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:07.092+0000 7f071fe00640 1 -- 172.21.15.6:0/2358104147 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07200ad020 con 0x7f07200ccad0 2024-03-04T09:30:07.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:07.093+0000 7f071fe00640 1 --2- 172.21.15.6:0/2358104147 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f07200ccad0 0x7f07200cceb0 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f0720070b90 tx=0x7f071002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12aefdbef26e6b07 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:07.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:07.094+0000 7f071f400640 1 -- 172.21.15.6:0/2358104147 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f071002ca70 con 0x7f07200ccad0 2024-03-04T09:30:07.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:07.094+0000 7f071f400640 1 -- 172.21.15.6:0/2358104147 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f071002cc10 con 0x7f07200ccad0 2024-03-04T09:30:07.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:07.094+0000 7f071f400640 1 -- 172.21.15.6:0/2358104147 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07100343f0 con 0x7f07200ccad0 2024-03-04T09:30:07.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:07.094+0000 7f0726e00640 1 -- 172.21.15.6:0/2358104147 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f07200ccad0 msgr2=0x7f07200cceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:07.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:07.094+0000 7f0726e00640 1 --2- 172.21.15.6:0/2358104147 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f07200ccad0 0x7f07200cceb0 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f0720070b90 tx=0x7f071002bed0 comp rx=0 tx=0).stop 2024-03-04T09:30:07.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:07.094+0000 7f0726e00640 1 -- 172.21.15.6:0/2358104147 shutdown_connections 2024-03-04T09:30:07.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:07.094+0000 7f0726e00640 1 --2- 172.21.15.6:0/2358104147 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f07200cdab0 0x7f07200cdee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:07.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:07.094+0000 7f0726e00640 1 --2- 172.21.15.6:0/2358104147 >> v2:172.21.15.146:3300/0 conn(0x7f0720162bb0 0x7f0720163040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:07.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:07.094+0000 7f0726e00640 1 --2- 172.21.15.6:0/2358104147 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f07200ccad0 0x7f07200cceb0 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:07.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:07.094+0000 7f0726e00640 1 -- 172.21.15.6:0/2358104147 >> 172.21.15.6:0/2358104147 conn(0x7f072006b3d0 msgr2=0x7f072006b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:07.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:07.094+0000 7f0726e00640 1 -- 172.21.15.6:0/2358104147 shutdown_connections 2024-03-04T09:30:07.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:07.094+0000 7f0726e00640 1 -- 172.21.15.6:0/2358104147 wait complete. 2024-03-04T09:30:07.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:07.096+0000 7f0726e00640 1 Processor -- start 2024-03-04T09:30:07.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:07.096+0000 7f0726e00640 1 -- start start 2024-03-04T09:30:07.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:07.097+0000 7f0726e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f07200ccad0 0x7f0720150d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:07.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:07.097+0000 7f0726e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f07200cdab0 0x7f072009d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:07.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:07.097+0000 7f0726e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0720162bb0 0x7f072009b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:07.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:07.097+0000 7f0726e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f072006e7d0 con 0x7f07200cdab0 2024-03-04T09:30:07.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:07.097+0000 7f0726e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07200ad5e0 con 0x7f0720162bb0 2024-03-04T09:30:07.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:07.097+0000 7f0726e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f072006c980 con 0x7f07200ccad0 2024-03-04T09:30:07.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:07.097+0000 7f0717400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f07200cdab0 0x7f072009d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:07.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:07.098+0000 7f0717400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f07200cdab0 0x7f072009d480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:34338/0 (socket says 172.21.15.6:34338) 2024-03-04T09:30:07.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:07.098+0000 7f0717400640 1 -- 172.21.15.6:0/2592712372 learned_addr learned my addr 172.21.15.6:0/2592712372 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:30:07.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:07.098+0000 7f071fe00640 1 --2- 172.21.15.6:0/2592712372 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f07200ccad0 0x7f0720150d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:07.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:07.098+0000 7f0724e00640 1 --2- 172.21.15.6:0/2592712372 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0720162bb0 0x7f072009b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:07.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:07.098+0000 7f0717400640 1 -- 172.21.15.6:0/2592712372 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f07200ccad0 msgr2=0x7f0720150d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:07.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:07.098+0000 7f0717400640 1 --2- 172.21.15.6:0/2592712372 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f07200ccad0 0x7f0720150d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:07.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:07.098+0000 7f0717400640 1 -- 172.21.15.6:0/2592712372 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0720162bb0 msgr2=0x7f072009b490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:07.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:07.098+0000 7f0717400640 1 --2- 172.21.15.6:0/2592712372 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0720162bb0 0x7f072009b490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:07.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:07.098+0000 7f0717400640 1 -- 172.21.15.6:0/2592712372 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0720151260 con 0x7f07200cdab0 2024-03-04T09:30:07.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:07.098+0000 7f0717400640 1 --2- 172.21.15.6:0/2592712372 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f07200cdab0 0x7f072009d480 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f070800d5f0 tx=0x7f070800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:07.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:07.098+0000 7f071cc00640 1 -- 172.21.15.6:0/2592712372 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07080125f0 con 0x7f07200cdab0 2024-03-04T09:30:07.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:07.098+0000 7f071cc00640 1 -- 172.21.15.6:0/2592712372 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0708019070 con 0x7f07200cdab0 2024-03-04T09:30:07.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:07.098+0000 7f071cc00640 1 -- 172.21.15.6:0/2592712372 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07080129d0 con 0x7f07200cdab0 2024-03-04T09:30:07.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:07.098+0000 7f0726e00640 1 -- 172.21.15.6:0/2592712372 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07200b3c00 con 0x7f07200cdab0 2024-03-04T09:30:07.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:07.099+0000 7f0726e00640 1 -- 172.21.15.6:0/2592712372 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07200b41c0 con 0x7f07200cdab0 2024-03-04T09:30:07.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:07.100+0000 7f0726e00640 1 -- 172.21.15.6:0/2592712372 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0720067830 con 0x7f07200cdab0 2024-03-04T09:30:07.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:07.101+0000 7f071cc00640 1 -- 172.21.15.6:0/2592712372 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0708012b70 con 0x7f07200cdab0 2024-03-04T09:30:07.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:07.101+0000 7f071cc00640 1 --2- 172.21.15.6:0/2592712372 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f06fc06b4d0 0x7f06fc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:07.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:07.102+0000 7f071cc00640 1 -- 172.21.15.6:0/2592712372 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5824+0+0 (secure 0 0 0) 0x7f070808a660 con 0x7f07200cdab0 2024-03-04T09:30:07.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:07.102+0000 7f071fe00640 1 --2- 172.21.15.6:0/2592712372 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f06fc06b4d0 0x7f06fc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:07.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:07.103+0000 7f071fe00640 1 --2- 172.21.15.6:0/2592712372 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f06fc06b4d0 0x7f06fc06d990 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f0710035000 tx=0x7f0710030000 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:07.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:07.106+0000 7f071cc00640 1 -- 172.21.15.6:0/2592712372 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0720067830 con 0x7f07200cdab0 2024-03-04T09:30:07.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:07.336+0000 7f0726e00640 1 -- 172.21.15.6:0/2592712372 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f072004f330 con 0x7f07200cdab0 2024-03-04T09:30:07.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:07.884+0000 7f071cc00640 1 -- 172.21.15.6:0/2592712372 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v53) v1 ==== 104+0+0 (secure 0 0 0) 0x7f0720151260 con 0x7f07200cdab0 2024-03-04T09:30:07.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:07.888+0000 7f0726e00640 1 -- 172.21.15.6:0/2592712372 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f06fc06b4d0 msgr2=0x7f06fc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:07.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:07.888+0000 7f0726e00640 1 --2- 172.21.15.6:0/2592712372 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f06fc06b4d0 0x7f06fc06d990 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f0710035000 tx=0x7f0710030000 comp rx=0 tx=0).stop 2024-03-04T09:30:07.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:07.888+0000 7f0726e00640 1 -- 172.21.15.6:0/2592712372 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f07200cdab0 msgr2=0x7f072009d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:07.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:07.888+0000 7f0726e00640 1 --2- 172.21.15.6:0/2592712372 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f07200cdab0 0x7f072009d480 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f070800d5f0 tx=0x7f070800dab0 comp rx=0 tx=0).stop 2024-03-04T09:30:07.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:07.888+0000 7f0726e00640 1 -- 172.21.15.6:0/2592712372 shutdown_connections 2024-03-04T09:30:07.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:07.888+0000 7f0726e00640 1 --2- 172.21.15.6:0/2592712372 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0720162bb0 0x7f072009b490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:07.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:07.888+0000 7f0726e00640 1 --2- 172.21.15.6:0/2592712372 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f07200cdab0 0x7f072009d480 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:07.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:07.888+0000 7f0726e00640 1 --2- 172.21.15.6:0/2592712372 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f06fc06b4d0 0x7f06fc06d990 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:07.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:07.888+0000 7f0726e00640 1 --2- 172.21.15.6:0/2592712372 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f07200ccad0 0x7f0720150d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:07.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:07.888+0000 7f0726e00640 1 -- 172.21.15.6:0/2592712372 >> 172.21.15.6:0/2592712372 conn(0x7f072006b3d0 msgr2=0x7f07200ae560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:07.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:07.889+0000 7f0726e00640 1 -- 172.21.15.6:0/2592712372 shutdown_connections 2024-03-04T09:30:07.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:07.889+0000 7f0726e00640 1 -- 172.21.15.6:0/2592712372 wait complete. 2024-03-04T09:30:07.910 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-04T09:30:08.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:08.188+0000 7fb984a00640 1 Processor -- start 2024-03-04T09:30:08.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:08.188+0000 7fb984a00640 1 -- start start 2024-03-04T09:30:08.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:08.190+0000 7fb984a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb9800cf8a0 0x7fb9800cfd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:08.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:08.190+0000 7fb984a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb98015f5f0 con 0x7fb98015e8c0 2024-03-04T09:30:08.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:08.190+0000 7fb984a00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9800ad8f0 con 0x7fb980072860 2024-03-04T09:30:08.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:08.190+0000 7fb984a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9800d0240 con 0x7fb9800cf8a0 2024-03-04T09:30:08.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:08.190+0000 7fb97e800640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fb98015e8c0 0x7fb98015ed50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:37258/0 (socket says 172.21.15.6:37258) 2024-03-04T09:30:08.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:08.190+0000 7fb97e800640 1 -- 172.21.15.6:0/3575727797 learned_addr learned my addr 172.21.15.6:0/3575727797 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:30:08.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:08.190+0000 7fb97d400640 1 -- 172.21.15.6:0/3575727797 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 901442555 0 0) 0x7fb98015f5f0 con 0x7fb98015e8c0 2024-03-04T09:30:08.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:08.190+0000 7fb977e00640 1 --2- 172.21.15.6:0/3575727797 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb9800cf8a0 0x7fb9800cfd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:08.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:08.191+0000 7fb97d400640 1 -- 172.21.15.6:0/3575727797 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb964005730 con 0x7fb98015e8c0 2024-03-04T09:30:08.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:08.191+0000 7fb97d400640 1 -- 172.21.15.6:0/3575727797 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 580088040 0 0) 0x7fb964005730 con 0x7fb98015e8c0 2024-03-04T09:30:08.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:08.191+0000 7fb97d400640 1 -- 172.21.15.6:0/3575727797 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb9800cf8a0 msgr2=0x7fb9800cfd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:08.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:08.191+0000 7fb97d400640 1 --2- 172.21.15.6:0/3575727797 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb9800cf8a0 0x7fb9800cfd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:08.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:08.191+0000 7fb97d400640 1 -- 172.21.15.6:0/3575727797 >> v1:172.21.15.146:6789/0 conn(0x7fb980072860 legacy=0x7fb9800ae3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:08.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:08.191+0000 7fb97d400640 1 -- 172.21.15.6:0/3575727797 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9800adbd0 con 0x7fb98015e8c0 2024-03-04T09:30:08.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:08.191+0000 7fb97d400640 1 -- 172.21.15.6:0/3575727797 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7fb9780019f0 con 0x7fb98015e8c0 2024-03-04T09:30:08.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:08.191+0000 7fb97d400640 1 -- 172.21.15.6:0/3575727797 >> v1:172.21.15.6:6789/0 conn(0x7fb98015e8c0 legacy=0x7fb98015ed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:08.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:08.191+0000 7fb97d400640 1 --2- 172.21.15.6:0/3575727797 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb964005bf0 0x7fb964006010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:08.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:08.191+0000 7fb97d400640 1 --2- 172.21.15.6:0/3575727797 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb964006a10 0x7fb964006e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:08.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:08.191+0000 7fb97d400640 1 --2- 172.21.15.6:0/3575727797 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb9800cf8a0 0x7fb96400dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:08.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:08.191+0000 7fb97d400640 1 -- 172.21.15.6:0/3575727797 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb964007700 con 0x7fb964005bf0 2024-03-04T09:30:08.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:08.191+0000 7fb97d400640 1 -- 172.21.15.6:0/3575727797 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb96400e110 con 0x7fb964006a10 2024-03-04T09:30:08.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:08.191+0000 7fb97d400640 1 -- 172.21.15.6:0/3575727797 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb96400e290 con 0x7fb9800cf8a0 2024-03-04T09:30:08.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:08.191+0000 7fb97de00640 1 --2- 172.21.15.6:0/3575727797 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb964005bf0 0x7fb964006010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:08.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:08.192+0000 7fb977e00640 1 --2- 172.21.15.6:0/3575727797 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb964006a10 0x7fb964006e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:08.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:08.192+0000 7fb97e800640 1 --2- 172.21.15.6:0/3575727797 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb9800cf8a0 0x7fb96400dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:08.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:08.192+0000 7fb97de00640 1 -- 172.21.15.6:0/3575727797 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb9800cf8a0 msgr2=0x7fb96400dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:08.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:08.192+0000 7fb97de00640 1 --2- 172.21.15.6:0/3575727797 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb9800cf8a0 0x7fb96400dbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:08.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:08.192+0000 7fb97de00640 1 -- 172.21.15.6:0/3575727797 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb964006a10 msgr2=0x7fb964006e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:08.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:08.192+0000 7fb97de00640 1 --2- 172.21.15.6:0/3575727797 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb964006a10 0x7fb964006e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:08.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:08.192+0000 7fb97de00640 1 -- 172.21.15.6:0/3575727797 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb964005730 con 0x7fb964005bf0 2024-03-04T09:30:08.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:08.192+0000 7fb977e00640 1 --2- 172.21.15.6:0/3575727797 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb964006a10 0x7fb964006e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:30:08.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:08.193+0000 7fb97de00640 1 --2- 172.21.15.6:0/3575727797 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb964005bf0 0x7fb964006010 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7fb970001750 tx=0x7fb964007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ac571d066524d89 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:08.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:08.193+0000 7fb97d400640 1 -- 172.21.15.6:0/3575727797 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb970001370 con 0x7fb964005bf0 2024-03-04T09:30:08.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:08.193+0000 7fb97d400640 1 -- 172.21.15.6:0/3575727797 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9700052d0 con 0x7fb964005bf0 2024-03-04T09:30:08.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:08.193+0000 7fb97d400640 1 -- 172.21.15.6:0/3575727797 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb970003830 con 0x7fb964005bf0 2024-03-04T09:30:08.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:08.194+0000 7fb984a00640 1 -- 172.21.15.6:0/3575727797 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb964005bf0 msgr2=0x7fb964006010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:08.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:08.194+0000 7fb984a00640 1 --2- 172.21.15.6:0/3575727797 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb964005bf0 0x7fb964006010 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7fb970001750 tx=0x7fb964007a40 comp rx=0 tx=0).stop 2024-03-04T09:30:08.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:08.194+0000 7fb97e800640 1 -- 172.21.15.6:0/3575727797 reap_dead start 2024-03-04T09:30:08.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:08.194+0000 7fb984a00640 1 -- 172.21.15.6:0/3575727797 shutdown_connections 2024-03-04T09:30:08.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:08.194+0000 7fb984a00640 1 -- 172.21.15.6:0/3575727797 >> 172.21.15.6:0/3575727797 conn(0x7fb98006afb0 msgr2=0x7fb98006b3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:08.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:08.194+0000 7fb984a00640 1 -- 172.21.15.6:0/3575727797 shutdown_connections 2024-03-04T09:30:08.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:08.194+0000 7fb984a00640 1 -- 172.21.15.6:0/3575727797 wait complete. 2024-03-04T09:30:08.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:08.196+0000 7fb984a00640 1 Processor -- start 2024-03-04T09:30:08.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:08.196+0000 7fb984a00640 1 -- start start 2024-03-04T09:30:08.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:08.197+0000 7fb984a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb964005bf0 0x7fb9800af6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:08.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:08.197+0000 7fb984a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb980157200 0x7fb9800b1060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:08.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:08.197+0000 7fb984a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb98009b2e0 0x7fb98014e230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:08.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:08.197+0000 7fb984a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9800cfd00 con 0x7fb964005bf0 2024-03-04T09:30:08.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:08.197+0000 7fb984a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9800adf40 con 0x7fb98009b2e0 2024-03-04T09:30:08.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:08.197+0000 7fb984a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9800cfe80 con 0x7fb980157200 2024-03-04T09:30:08.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:08.197+0000 7fb97de00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb964005bf0 0x7fb9800af6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:08.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:08.197+0000 7fb97de00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb964005bf0 0x7fb9800af6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:34346/0 (socket says 172.21.15.6:34346) 2024-03-04T09:30:08.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:08.197+0000 7fb97de00640 1 -- 172.21.15.6:0/3388446046 learned_addr learned my addr 172.21.15.6:0/3388446046 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:30:08.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:08.197+0000 7fb97de00640 1 -- 172.21.15.6:0/3388446046 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb980157200 msgr2=0x7fb9800b1060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:08.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:08.197+0000 7fb97de00640 1 --2- 172.21.15.6:0/3388446046 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb980157200 0x7fb9800b1060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:08.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:08.197+0000 7fb97de00640 1 -- 172.21.15.6:0/3388446046 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb98009b2e0 msgr2=0x7fb98014e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:08.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:08.197+0000 7fb97de00640 1 --2- 172.21.15.6:0/3388446046 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb98009b2e0 0x7fb98014e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:08.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:08.197+0000 7fb97de00640 1 -- 172.21.15.6:0/3388446046 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9800b15a0 con 0x7fb964005bf0 2024-03-04T09:30:08.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:08.198+0000 7fb97de00640 1 --2- 172.21.15.6:0/3388446046 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb964005bf0 0x7fb9800af6b0 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7fb970001750 tx=0x7fb970001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:08.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:08.198+0000 7fb976000640 1 -- 172.21.15.6:0/3388446046 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb970042070 con 0x7fb964005bf0 2024-03-04T09:30:08.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:08.198+0000 7fb976000640 1 -- 172.21.15.6:0/3388446046 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb970005480 con 0x7fb964005bf0 2024-03-04T09:30:08.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:08.198+0000 7fb976000640 1 -- 172.21.15.6:0/3388446046 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb970001370 con 0x7fb964005bf0 2024-03-04T09:30:08.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:08.198+0000 7fb984a00640 1 -- 172.21.15.6:0/3388446046 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9800afc90 con 0x7fb964005bf0 2024-03-04T09:30:08.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:08.198+0000 7fb984a00640 1 -- 172.21.15.6:0/3388446046 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb980156720 con 0x7fb964005bf0 2024-03-04T09:30:08.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:08.199+0000 7fb984a00640 1 -- 172.21.15.6:0/3388446046 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb980068010 con 0x7fb964005bf0 2024-03-04T09:30:08.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:08.200+0000 7fb976000640 1 -- 172.21.15.6:0/3388446046 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb970045070 con 0x7fb964005bf0 2024-03-04T09:30:08.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:08.201+0000 7fb976000640 1 --2- 172.21.15.6:0/3388446046 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb95c06b500 0x7fb95c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:08.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:08.201+0000 7fb976000640 1 -- 172.21.15.6:0/3388446046 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5824+0+0 (secure 0 0 0) 0x7fb9700abc50 con 0x7fb964005bf0 2024-03-04T09:30:08.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:08.202+0000 7fb977e00640 1 --2- 172.21.15.6:0/3388446046 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb95c06b500 0x7fb95c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:08.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:08.203+0000 7fb977e00640 1 --2- 172.21.15.6:0/3388446046 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb95c06b500 0x7fb95c06d9c0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fb968006f00 tx=0x7fb968006e10 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:08.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:08.206+0000 7fb976000640 1 -- 172.21.15.6:0/3388446046 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb97007ab90 con 0x7fb964005bf0 2024-03-04T09:30:08.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:08.405+0000 7fb984a00640 1 -- 172.21.15.6:0/3388446046 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fb98006eb10 con 0x7fb964005bf0 2024-03-04T09:30:09.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:09.059+0000 7fb976000640 1 -- 172.21.15.6:0/3388446046 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v85) v1 ==== 181+0+0 (secure 0 0 0) 0x7fb97007a490 con 0x7fb964005bf0 2024-03-04T09:30:09.056 INFO:teuthology.orchestra.run.smithi006.stderr:pool 'cephfs_metadata' removed 2024-03-04T09:30:09.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:09.063+0000 7fb984a00640 1 -- 172.21.15.6:0/3388446046 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb95c06b500 msgr2=0x7fb95c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:09.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:09.063+0000 7fb984a00640 1 --2- 172.21.15.6:0/3388446046 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb95c06b500 0x7fb95c06d9c0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fb968006f00 tx=0x7fb968006e10 comp rx=0 tx=0).stop 2024-03-04T09:30:09.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:09.063+0000 7fb984a00640 1 -- 172.21.15.6:0/3388446046 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb964005bf0 msgr2=0x7fb9800af6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:09.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:09.063+0000 7fb984a00640 1 --2- 172.21.15.6:0/3388446046 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb964005bf0 0x7fb9800af6b0 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7fb970001750 tx=0x7fb970001120 comp rx=0 tx=0).stop 2024-03-04T09:30:09.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:09.063+0000 7fb984a00640 1 -- 172.21.15.6:0/3388446046 shutdown_connections 2024-03-04T09:30:09.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:09.063+0000 7fb984a00640 1 --2- 172.21.15.6:0/3388446046 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb98009b2e0 0x7fb98014e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:09.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:09.063+0000 7fb984a00640 1 --2- 172.21.15.6:0/3388446046 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb95c06b500 0x7fb95c06d9c0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:09.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:09.063+0000 7fb984a00640 1 --2- 172.21.15.6:0/3388446046 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb980157200 0x7fb9800b1060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:09.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:09.063+0000 7fb984a00640 1 --2- 172.21.15.6:0/3388446046 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb964005bf0 0x7fb9800af6b0 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:09.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:09.063+0000 7fb984a00640 1 -- 172.21.15.6:0/3388446046 >> 172.21.15.6:0/3388446046 conn(0x7fb98006afb0 msgr2=0x7fb98015ce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:09.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:09.064+0000 7fb984a00640 1 -- 172.21.15.6:0/3388446046 shutdown_connections 2024-03-04T09:30:09.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:09.064+0000 7fb984a00640 1 -- 172.21.15.6:0/3388446046 wait complete. 2024-03-04T09:30:09.090 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-04T09:30:09.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:09.294+0000 7fdbcbe00640 1 Processor -- start 2024-03-04T09:30:09.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:09.294+0000 7fdbcbe00640 1 -- start start 2024-03-04T09:30:09.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:09.295+0000 7fdbcbe00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdbcc145570 0x7fdbcc0cf420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:09.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:09.295+0000 7fdbcbe00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fdbcc071450 0x7fdbcc06daf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:09.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:09.296+0000 7fdbcbe00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbcc071c50 con 0x7fdbcc06e030 2024-03-04T09:30:09.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:09.296+0000 7fdbcbe00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbcc06e4a0 con 0x7fdbcc071450 2024-03-04T09:30:09.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:09.296+0000 7fdbcbe00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbcc071f30 con 0x7fdbcc145570 2024-03-04T09:30:09.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:09.296+0000 7fdbcaa00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdbcc145570 0x7fdbcc0cf420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:09.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:09.297+0000 7fdbcaa00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdbcc145570 0x7fdbcc0cf420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.6:56584/0 (socket says 172.21.15.6:56584) 2024-03-04T09:30:09.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:09.297+0000 7fdbcaa00640 1 -- 172.21.15.6:0/3329268768 learned_addr learned my addr 172.21.15.6:0/3329268768 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:30:09.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:09.297+0000 7fdbca000640 1 --2- 172.21.15.6:0/3329268768 >> v2:172.21.15.146:3300/0 conn(0x7fdbcc071450 0x7fdbcc06daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:09.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:09.297+0000 7fdbcaa00640 1 -- 172.21.15.6:0/3329268768 >> v2:172.21.15.146:3300/0 conn(0x7fdbcc071450 msgr2=0x7fdbcc06daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:09.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:09.297+0000 7fdbcaa00640 1 --2- 172.21.15.6:0/3329268768 >> v2:172.21.15.146:3300/0 conn(0x7fdbcc071450 0x7fdbcc06daf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:09.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:09.297+0000 7fdbcaa00640 1 -- 172.21.15.6:0/3329268768 >> v1:172.21.15.146:6789/0 conn(0x7fdbcc06e030 legacy=0x7fdbcc0ccc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:09.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:09.297+0000 7fdbcaa00640 1 -- 172.21.15.6:0/3329268768 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbcc072210 con 0x7fdbcc145570 2024-03-04T09:30:09.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:09.299+0000 7fdbcaa00640 1 --2- 172.21.15.6:0/3329268768 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdbcc145570 0x7fdbcc0cf420 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fdbbc009e50 tx=0x7fdbbc02bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=70ce5326303236f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:09.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:09.299+0000 7fdbc9600640 1 -- 172.21.15.6:0/3329268768 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbbc02ca70 con 0x7fdbcc145570 2024-03-04T09:30:09.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:09.299+0000 7fdbc9600640 1 -- 172.21.15.6:0/3329268768 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbbc02cc10 con 0x7fdbcc145570 2024-03-04T09:30:09.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:09.299+0000 7fdbc9600640 1 -- 172.21.15.6:0/3329268768 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbbc0343f0 con 0x7fdbcc145570 2024-03-04T09:30:09.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:09.300+0000 7fdbcbe00640 1 -- 172.21.15.6:0/3329268768 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdbcc145570 msgr2=0x7fdbcc0cf420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:09.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:09.300+0000 7fdbcbe00640 1 --2- 172.21.15.6:0/3329268768 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdbcc145570 0x7fdbcc0cf420 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fdbbc009e50 tx=0x7fdbbc02bed0 comp rx=0 tx=0).stop 2024-03-04T09:30:09.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:09.300+0000 7fdbcbe00640 1 -- 172.21.15.6:0/3329268768 shutdown_connections 2024-03-04T09:30:09.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:09.300+0000 7fdbcbe00640 1 --2- 172.21.15.6:0/3329268768 >> v2:172.21.15.146:3300/0 conn(0x7fdbcc071450 0x7fdbcc06daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:09.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:09.300+0000 7fdbcbe00640 1 --2- 172.21.15.6:0/3329268768 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdbcc145570 0x7fdbcc0cf420 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:09.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:09.300+0000 7fdbcbe00640 1 -- 172.21.15.6:0/3329268768 >> 172.21.15.6:0/3329268768 conn(0x7fdbcc0b17c0 msgr2=0x7fdbcc0b1ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:09.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:09.300+0000 7fdbcbe00640 1 -- 172.21.15.6:0/3329268768 shutdown_connections 2024-03-04T09:30:09.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:09.300+0000 7fdbcbe00640 1 -- 172.21.15.6:0/3329268768 wait complete. 2024-03-04T09:30:09.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:09.302+0000 7fdbcbe00640 1 Processor -- start 2024-03-04T09:30:09.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:09.302+0000 7fdbcbe00640 1 -- start start 2024-03-04T09:30:09.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:09.303+0000 7fdbcbe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdbcc06e030 0x7fdbcc1f4180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:09.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:09.303+0000 7fdbcbe00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdbcc071450 0x7fdbcc0b35d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:09.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:09.303+0000 7fdbcbe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdbcc145570 0x7fdbcc1f0a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:09.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:09.303+0000 7fdbcbe00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbcc0718b0 con 0x7fdbcc145570 2024-03-04T09:30:09.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:09.303+0000 7fdbcbe00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbcc0727d0 con 0x7fdbcc06e030 2024-03-04T09:30:09.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:09.303+0000 7fdbcbe00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbcc071a30 con 0x7fdbcc071450 2024-03-04T09:30:09.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:09.303+0000 7fdbcb400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdbcc145570 0x7fdbcc1f0a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:09.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:09.303+0000 7fdbcb400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdbcc145570 0x7fdbcc1f0a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51818/0 (socket says 172.21.15.6:51818) 2024-03-04T09:30:09.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:09.303+0000 7fdbcb400640 1 -- 172.21.15.6:0/3928314690 learned_addr learned my addr 172.21.15.6:0/3928314690 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:30:09.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:09.304+0000 7fdbca000640 1 --2- 172.21.15.6:0/3928314690 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdbcc071450 0x7fdbcc0b35d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:09.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:09.304+0000 7fdbcaa00640 1 --2- 172.21.15.6:0/3928314690 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdbcc06e030 0x7fdbcc1f4180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:09.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:09.304+0000 7fdbcb400640 1 -- 172.21.15.6:0/3928314690 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdbcc071450 msgr2=0x7fdbcc0b35d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:09.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:09.304+0000 7fdbcb400640 1 --2- 172.21.15.6:0/3928314690 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdbcc071450 0x7fdbcc0b35d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:09.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:09.304+0000 7fdbcb400640 1 -- 172.21.15.6:0/3928314690 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdbcc06e030 msgr2=0x7fdbcc1f4180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:09.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:09.304+0000 7fdbcb400640 1 --2- 172.21.15.6:0/3928314690 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdbcc06e030 0x7fdbcc1f4180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:09.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:09.304+0000 7fdbcb400640 1 -- 172.21.15.6:0/3928314690 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbcc0b3b10 con 0x7fdbcc145570 2024-03-04T09:30:09.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:09.304+0000 7fdbcaa00640 1 --2- 172.21.15.6:0/3928314690 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdbcc06e030 0x7fdbcc1f4180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:30:09.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:09.304+0000 7fdbcb400640 1 --2- 172.21.15.6:0/3928314690 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdbcc145570 0x7fdbcc1f0a20 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7fdbc4001eb0 tx=0x7fdbc400c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:09.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:09.304+0000 7fdbc2a00640 1 -- 172.21.15.6:0/3928314690 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbc4017070 con 0x7fdbcc145570 2024-03-04T09:30:09.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:09.304+0000 7fdbc2a00640 1 -- 172.21.15.6:0/3928314690 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbc40052d0 con 0x7fdbcc145570 2024-03-04T09:30:09.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:09.304+0000 7fdbc2a00640 1 -- 172.21.15.6:0/3928314690 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbc4003810 con 0x7fdbcc145570 2024-03-04T09:30:09.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:09.304+0000 7fdbcbe00640 1 -- 172.21.15.6:0/3928314690 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbcc0b3db0 con 0x7fdbcc145570 2024-03-04T09:30:09.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:09.305+0000 7fdbcbe00640 1 -- 172.21.15.6:0/3928314690 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbcc0b4370 con 0x7fdbcc145570 2024-03-04T09:30:09.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:09.306+0000 7fdbcbe00640 1 -- 172.21.15.6:0/3928314690 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbcc1f4760 con 0x7fdbcc145570 2024-03-04T09:30:09.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:09.307+0000 7fdbc2a00640 1 -- 172.21.15.6:0/3928314690 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdbc401a070 con 0x7fdbcc145570 2024-03-04T09:30:09.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:09.308+0000 7fdbc2a00640 1 --2- 172.21.15.6:0/3928314690 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fdba806b4d0 0x7fdba806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:09.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:09.308+0000 7fdbc2a00640 1 -- 172.21.15.6:0/3928314690 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 5386+0+0 (secure 0 0 0) 0x7fdbc40894c0 con 0x7fdbcc145570 2024-03-04T09:30:09.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:09.309+0000 7fdbcaa00640 1 --2- 172.21.15.6:0/3928314690 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fdba806b4d0 0x7fdba806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:09.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:09.310+0000 7fdbcaa00640 1 --2- 172.21.15.6:0/3928314690 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fdba806b4d0 0x7fdba806d990 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fdbbc035000 tx=0x7fdbbc030000 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:09.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:09.313+0000 7fdbc2a00640 1 -- 172.21.15.6:0/3928314690 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdbc40583e0 con 0x7fdbcc145570 2024-03-04T09:30:09.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:09.518+0000 7fdbcbe00640 1 -- 172.21.15.6:0/3928314690 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fdbcc071100 con 0x7fdbcc145570 2024-03-04T09:30:09.877 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:30:09.885 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:30:10.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.072+0000 7fdbc2a00640 1 -- 172.21.15.6:0/3928314690 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v86) v1 ==== 169+0+0 (secure 0 0 0) 0x7fdbc4057ce0 con 0x7fdbcc145570 2024-03-04T09:30:10.069 INFO:teuthology.orchestra.run.smithi006.stderr:pool 'cephfs_data' removed 2024-03-04T09:30:10.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.075+0000 7fdbcbe00640 1 -- 172.21.15.6:0/3928314690 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fdba806b4d0 msgr2=0x7fdba806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:10.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.075+0000 7fdbcbe00640 1 --2- 172.21.15.6:0/3928314690 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fdba806b4d0 0x7fdba806d990 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fdbbc035000 tx=0x7fdbbc030000 comp rx=0 tx=0).stop 2024-03-04T09:30:10.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.075+0000 7fdbcbe00640 1 -- 172.21.15.6:0/3928314690 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdbcc145570 msgr2=0x7fdbcc1f0a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:10.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.075+0000 7fdbcbe00640 1 --2- 172.21.15.6:0/3928314690 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdbcc145570 0x7fdbcc1f0a20 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7fdbc4001eb0 tx=0x7fdbc400c890 comp rx=0 tx=0).stop 2024-03-04T09:30:10.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.076+0000 7fdbcbe00640 1 -- 172.21.15.6:0/3928314690 shutdown_connections 2024-03-04T09:30:10.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.076+0000 7fdbcbe00640 1 --2- 172.21.15.6:0/3928314690 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdbcc145570 0x7fdbcc1f0a20 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:10.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.076+0000 7fdbcbe00640 1 --2- 172.21.15.6:0/3928314690 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fdba806b4d0 0x7fdba806d990 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:10.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.076+0000 7fdbcbe00640 1 --2- 172.21.15.6:0/3928314690 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdbcc071450 0x7fdbcc0b35d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:10.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.076+0000 7fdbcbe00640 1 --2- 172.21.15.6:0/3928314690 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdbcc06e030 0x7fdbcc1f4180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:10.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.076+0000 7fdbcbe00640 1 -- 172.21.15.6:0/3928314690 >> 172.21.15.6:0/3928314690 conn(0x7fdbcc0b17c0 msgr2=0x7fdbcc15ed30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:10.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.079+0000 7fdbcbe00640 1 -- 172.21.15.6:0/3928314690 shutdown_connections 2024-03-04T09:30:10.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.079+0000 7fdbcbe00640 1 -- 172.21.15.6:0/3928314690 wait complete. 2024-03-04T09:30:10.099 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-04T09:30:10.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.309+0000 7f2704000640 1 Processor -- start 2024-03-04T09:30:10.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.310+0000 7f2704000640 1 -- start start 2024-03-04T09:30:10.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.311+0000 7f2704000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f26fc15bf20 0x7f26fc0acd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:10.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.311+0000 7f2704000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f26fc072520 0x7f26fc0cdfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:10.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.311+0000 7f2704000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26fc072df0 con 0x7f26fc071e20 2024-03-04T09:30:10.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.311+0000 7f2704000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26fc0cdab0 con 0x7f26fc15bf20 2024-03-04T09:30:10.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.311+0000 7f2704000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26fc0b4d70 con 0x7f26fc072520 2024-03-04T09:30:10.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.312+0000 7f2702000640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f26fc071e20 0x7f26fc0ccad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:34822/0 (socket says 172.21.15.6:34822) 2024-03-04T09:30:10.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.312+0000 7f2702000640 1 -- 172.21.15.6:0/3121098191 learned_addr learned my addr 172.21.15.6:0/3121098191 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:30:10.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.312+0000 7f2701600640 1 --2- 172.21.15.6:0/3121098191 >> v2:172.21.15.146:3300/0 conn(0x7f26fc15bf20 0x7f26fc0acd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:10.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.312+0000 7f2700c00640 1 --2- 172.21.15.6:0/3121098191 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f26fc072520 0x7f26fc0cdfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:10.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.312+0000 7f26fbe00640 1 -- 172.21.15.6:0/3121098191 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2006691906 0 0) 0x7f26fc072df0 con 0x7f26fc071e20 2024-03-04T09:30:10.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.312+0000 7f26fbe00640 1 -- 172.21.15.6:0/3121098191 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26e4003520 con 0x7f26fc071e20 2024-03-04T09:30:10.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.313+0000 7f26fbe00640 1 -- 172.21.15.6:0/3121098191 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2602020695 0 0) 0x7f26e4003520 con 0x7f26fc071e20 2024-03-04T09:30:10.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.313+0000 7f26fbe00640 1 -- 172.21.15.6:0/3121098191 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f26fc072520 msgr2=0x7f26fc0cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:10.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.313+0000 7f26fbe00640 1 --2- 172.21.15.6:0/3121098191 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f26fc072520 0x7f26fc0cdfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:10.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.313+0000 7f26fbe00640 1 -- 172.21.15.6:0/3121098191 >> v2:172.21.15.146:3300/0 conn(0x7f26fc15bf20 msgr2=0x7f26fc0acd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:10.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.313+0000 7f26fbe00640 1 --2- 172.21.15.6:0/3121098191 >> v2:172.21.15.146:3300/0 conn(0x7f26fc15bf20 0x7f26fc0acd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:10.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.313+0000 7f26fbe00640 1 -- 172.21.15.6:0/3121098191 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26fc0b5050 con 0x7f26fc071e20 2024-03-04T09:30:10.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.313+0000 7f26fbe00640 1 -- 172.21.15.6:0/3121098191 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7f26f40019f0 con 0x7f26fc071e20 2024-03-04T09:30:10.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.313+0000 7f26fbe00640 1 -- 172.21.15.6:0/3121098191 >> v1:172.21.15.6:6789/0 conn(0x7f26fc071e20 legacy=0x7f26fc0ccad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:10.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.313+0000 7f26fbe00640 1 --2- 172.21.15.6:0/3121098191 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f26e40039b0 0x7f26e4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:10.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.313+0000 7f26fbe00640 1 --2- 172.21.15.6:0/3121098191 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f26fc072520 0x7f26e4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:10.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.313+0000 7f26fbe00640 1 --2- 172.21.15.6:0/3121098191 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f26e4004d40 0x7f26e40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:10.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.313+0000 7f26fbe00640 1 -- 172.21.15.6:0/3121098191 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26fc0cdab0 con 0x7f26e4004d40 2024-03-04T09:30:10.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.313+0000 7f26fbe00640 1 -- 172.21.15.6:0/3121098191 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26fc0b4d70 con 0x7f26e40039b0 2024-03-04T09:30:10.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.313+0000 7f26fbe00640 1 -- 172.21.15.6:0/3121098191 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26e400b9c0 con 0x7f26fc072520 2024-03-04T09:30:10.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.313+0000 7f2702000640 1 --2- 172.21.15.6:0/3121098191 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f26e4004d40 0x7f26e40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:10.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.313+0000 7f2700c00640 1 --2- 172.21.15.6:0/3121098191 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f26fc072520 0x7f26e4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:10.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.313+0000 7f2701600640 1 --2- 172.21.15.6:0/3121098191 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f26e40039b0 0x7f26e4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:10.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.314+0000 7f2700c00640 1 -- 172.21.15.6:0/3121098191 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f26e40039b0 msgr2=0x7f26e4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:10.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.314+0000 7f2700c00640 1 --2- 172.21.15.6:0/3121098191 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f26e40039b0 0x7f26e4003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:10.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.314+0000 7f2700c00640 1 -- 172.21.15.6:0/3121098191 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f26e4004d40 msgr2=0x7f26e40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:10.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.314+0000 7f2700c00640 1 --2- 172.21.15.6:0/3121098191 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f26e4004d40 0x7f26e40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:10.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.314+0000 7f2700c00640 1 -- 172.21.15.6:0/3121098191 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26e4003520 con 0x7f26fc072520 2024-03-04T09:30:10.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.314+0000 7f2701600640 1 --2- 172.21.15.6:0/3121098191 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f26e40039b0 0x7f26e4003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T09:30:10.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.314+0000 7f2702000640 1 --2- 172.21.15.6:0/3121098191 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f26e4004d40 0x7f26e40051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T09:30:10.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.315+0000 7f2700c00640 1 --2- 172.21.15.6:0/3121098191 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f26fc072520 0x7f26e4004800 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f26e8009ac0 tx=0x7f26e4005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=27511f89e099f81 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:10.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.315+0000 7f26fbe00640 1 -- 172.21.15.6:0/3121098191 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26e8039070 con 0x7f26fc072520 2024-03-04T09:30:10.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.315+0000 7f26fbe00640 1 -- 172.21.15.6:0/3121098191 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26e802c710 con 0x7f26fc072520 2024-03-04T09:30:10.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.315+0000 7f26fbe00640 1 -- 172.21.15.6:0/3121098191 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26e8009690 con 0x7f26fc072520 2024-03-04T09:30:10.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.316+0000 7f2704000640 1 -- 172.21.15.6:0/3121098191 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f26fc072520 msgr2=0x7f26e4004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:10.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.316+0000 7f2704000640 1 --2- 172.21.15.6:0/3121098191 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f26fc072520 0x7f26e4004800 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f26e8009ac0 tx=0x7f26e4005830 comp rx=0 tx=0).stop 2024-03-04T09:30:10.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.316+0000 7f2702000640 1 -- 172.21.15.6:0/3121098191 reap_dead start 2024-03-04T09:30:10.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.316+0000 7f2704000640 1 -- 172.21.15.6:0/3121098191 shutdown_connections 2024-03-04T09:30:10.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.316+0000 7f2704000640 1 -- 172.21.15.6:0/3121098191 >> 172.21.15.6:0/3121098191 conn(0x7f26fc06b3d0 msgr2=0x7f26fc06b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:10.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.316+0000 7f2704000640 1 -- 172.21.15.6:0/3121098191 shutdown_connections 2024-03-04T09:30:10.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.316+0000 7f2704000640 1 -- 172.21.15.6:0/3121098191 wait complete. 2024-03-04T09:30:10.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.318+0000 7f2704000640 1 Processor -- start 2024-03-04T09:30:10.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.318+0000 7f2704000640 1 -- start start 2024-03-04T09:30:10.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.319+0000 7f2704000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f26fc072520 0x7f26fc09ba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:10.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.319+0000 7f2704000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f26fc14c650 0x7f26fc0b21d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:10.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.319+0000 7f2704000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f26fc09d190 0x7f26fc0b00a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:10.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.319+0000 7f2704000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26fc0b4b10 con 0x7f26fc072520 2024-03-04T09:30:10.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.319+0000 7f2704000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26fc0b4990 con 0x7f26fc09d190 2024-03-04T09:30:10.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.319+0000 7f2704000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26fc0cd1e0 con 0x7f26fc14c650 2024-03-04T09:30:10.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.319+0000 7f2701600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f26fc072520 0x7f26fc09ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:10.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.319+0000 7f2701600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f26fc072520 0x7f26fc09ba60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51834/0 (socket says 172.21.15.6:51834) 2024-03-04T09:30:10.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.319+0000 7f2701600640 1 -- 172.21.15.6:0/4082864238 learned_addr learned my addr 172.21.15.6:0/4082864238 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:30:10.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.319+0000 7f2700c00640 1 --2- 172.21.15.6:0/4082864238 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f26fc14c650 0x7f26fc0b21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:10.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.319+0000 7f2702000640 1 --2- 172.21.15.6:0/4082864238 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f26fc09d190 0x7f26fc0b00a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:10.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.320+0000 7f2701600640 1 -- 172.21.15.6:0/4082864238 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f26fc14c650 msgr2=0x7f26fc0b21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:10.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.320+0000 7f2701600640 1 --2- 172.21.15.6:0/4082864238 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f26fc14c650 0x7f26fc0b21d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:10.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.320+0000 7f2701600640 1 -- 172.21.15.6:0/4082864238 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f26fc09d190 msgr2=0x7f26fc0b00a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:10.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.320+0000 7f2701600640 1 --2- 172.21.15.6:0/4082864238 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f26fc09d190 0x7f26fc0b00a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:10.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.320+0000 7f2701600640 1 -- 172.21.15.6:0/4082864238 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26fc09c0f0 con 0x7f26fc072520 2024-03-04T09:30:10.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.320+0000 7f2702000640 1 --2- 172.21.15.6:0/4082864238 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f26fc09d190 0x7f26fc0b00a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T09:30:10.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.320+0000 7f2701600640 1 --2- 172.21.15.6:0/4082864238 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f26fc072520 0x7f26fc09ba60 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f26f0011600 tx=0x7f26f0011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:10.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.320+0000 7f26f9600640 1 -- 172.21.15.6:0/4082864238 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26f000f8c0 con 0x7f26fc072520 2024-03-04T09:30:10.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.320+0000 7f26f9600640 1 -- 172.21.15.6:0/4082864238 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26f000fa60 con 0x7f26fc072520 2024-03-04T09:30:10.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.320+0000 7f2704000640 1 -- 172.21.15.6:0/4082864238 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26fc09c3d0 con 0x7f26fc072520 2024-03-04T09:30:10.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.320+0000 7f26f9600640 1 -- 172.21.15.6:0/4082864238 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26f000fe30 con 0x7f26fc072520 2024-03-04T09:30:10.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.320+0000 7f2704000640 1 -- 172.21.15.6:0/4082864238 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26fc09c990 con 0x7f26fc072520 2024-03-04T09:30:10.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.322+0000 7f2704000640 1 -- 172.21.15.6:0/4082864238 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26fc04f330 con 0x7f26fc072520 2024-03-04T09:30:10.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.323+0000 7f26f9600640 1 -- 172.21.15.6:0/4082864238 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f26f0015460 con 0x7f26fc072520 2024-03-04T09:30:10.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.323+0000 7f26f9600640 1 --2- 172.21.15.6:0/4082864238 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f26dc06b500 0x7f26dc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:10.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.323+0000 7f26f9600640 1 -- 172.21.15.6:0/4082864238 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f26f008a7c0 con 0x7f26fc072520 2024-03-04T09:30:10.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.324+0000 7f2700c00640 1 --2- 172.21.15.6:0/4082864238 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f26dc06b500 0x7f26dc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:10.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.325+0000 7f2700c00640 1 --2- 172.21.15.6:0/4082864238 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f26dc06b500 0x7f26dc06d9c0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f26e8009ac0 tx=0x7f26e8030000 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:10.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.328+0000 7f26f9600640 1 -- 172.21.15.6:0/4082864238 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f26f0019030 con 0x7f26fc072520 2024-03-04T09:30:10.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.528+0000 7f2704000640 1 -- 172.21.15.6:0/4082864238 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f26fc15f720 con 0x7f26fc072520 2024-03-04T09:30:10.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.535+0000 7f26f9600640 1 -- 172.21.15.6:0/4082864238 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v27) v1 ==== 104+0+0 (secure 0 0 0) 0x7f26f00598a0 con 0x7f26fc072520 2024-03-04T09:30:10.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.538+0000 7f2704000640 1 -- 172.21.15.6:0/4082864238 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f26dc06b500 msgr2=0x7f26dc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:10.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.538+0000 7f2704000640 1 --2- 172.21.15.6:0/4082864238 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f26dc06b500 0x7f26dc06d9c0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f26e8009ac0 tx=0x7f26e8030000 comp rx=0 tx=0).stop 2024-03-04T09:30:10.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.538+0000 7f2704000640 1 -- 172.21.15.6:0/4082864238 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f26fc072520 msgr2=0x7f26fc09ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:10.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.538+0000 7f2704000640 1 --2- 172.21.15.6:0/4082864238 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f26fc072520 0x7f26fc09ba60 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f26f0011600 tx=0x7f26f0011ac0 comp rx=0 tx=0).stop 2024-03-04T09:30:10.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.539+0000 7f2704000640 1 -- 172.21.15.6:0/4082864238 shutdown_connections 2024-03-04T09:30:10.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.539+0000 7f2704000640 1 --2- 172.21.15.6:0/4082864238 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f26fc09d190 0x7f26fc0b00a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:10.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.539+0000 7f2704000640 1 --2- 172.21.15.6:0/4082864238 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f26dc06b500 0x7f26dc06d9c0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:10.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.539+0000 7f2704000640 1 --2- 172.21.15.6:0/4082864238 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f26fc14c650 0x7f26fc0b21d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:10.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.539+0000 7f2704000640 1 --2- 172.21.15.6:0/4082864238 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f26fc072520 0x7f26fc09ba60 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:10.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.539+0000 7f2704000640 1 -- 172.21.15.6:0/4082864238 >> 172.21.15.6:0/4082864238 conn(0x7f26fc06b3d0 msgr2=0x7f26fc0ade40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:10.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.539+0000 7f2704000640 1 -- 172.21.15.6:0/4082864238 shutdown_connections 2024-03-04T09:30:10.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.539+0000 7f2704000640 1 -- 172.21.15.6:0/4082864238 wait complete. 2024-03-04T09:30:10.554 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_fancy_layout' 2024-03-04T09:30:10.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.788+0000 7effed800640 1 Processor -- start 2024-03-04T09:30:10.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.788+0000 7effed800640 1 -- start start 2024-03-04T09:30:10.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.790+0000 7effed800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7effe80af580 0x7effe815c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:10.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.790+0000 7effed800640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effe815cc40 con 0x7effe80af1a0 2024-03-04T09:30:10.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.790+0000 7effed800640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effe815cf20 con 0x7effe80cdc20 2024-03-04T09:30:10.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.790+0000 7effed800640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7effe815d200 con 0x7effe80af580 2024-03-04T09:30:10.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.791+0000 7effe6a00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7effe80af1a0 0x7effe80b4c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:34834/0 (socket says 172.21.15.6:34834) 2024-03-04T09:30:10.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.791+0000 7effe6a00640 1 -- 172.21.15.6:0/1941287522 learned_addr learned my addr 172.21.15.6:0/1941287522 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:30:10.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.791+0000 7effe6000640 1 --2- 172.21.15.6:0/1941287522 >> v2:172.21.15.146:3300/0 conn(0x7effe80af580 0x7effe815c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:10.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.791+0000 7effe5600640 1 -- 172.21.15.6:0/1941287522 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2148314735 0 0) 0x7effe815cc40 con 0x7effe80af1a0 2024-03-04T09:30:10.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.791+0000 7effe5600640 1 -- 172.21.15.6:0/1941287522 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7effcc003520 con 0x7effe80af1a0 2024-03-04T09:30:10.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.792+0000 7effe5600640 1 -- 172.21.15.6:0/1941287522 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2049826541 0 0) 0x7effcc003520 con 0x7effe80af1a0 2024-03-04T09:30:10.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.792+0000 7effe6000640 1 -- 172.21.15.6:0/1941287522 >> v1:172.21.15.146:6789/0 conn(0x7effe80cdc20 legacy=0x7effe80b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:10.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.792+0000 7effe6000640 1 -- 172.21.15.6:0/1941287522 >> v1:172.21.15.6:6789/0 conn(0x7effe80af1a0 legacy=0x7effe80b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:10.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.792+0000 7effe6000640 1 -- 172.21.15.6:0/1941287522 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effe815d4e0 con 0x7effe80af580 2024-03-04T09:30:10.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.793+0000 7effe6000640 1 --2- 172.21.15.6:0/1941287522 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7effe80af580 0x7effe815c600 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7effd40097a0 tx=0x7effd402b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8884b7ff608745ef server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:10.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.794+0000 7effe5600640 1 -- 172.21.15.6:0/1941287522 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effd4037070 con 0x7effe80af580 2024-03-04T09:30:10.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.794+0000 7effe5600640 1 -- 172.21.15.6:0/1941287522 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effd4030510 con 0x7effe80af580 2024-03-04T09:30:10.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.794+0000 7effe5600640 1 -- 172.21.15.6:0/1941287522 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effd8001670 con 0x7effe80af580 2024-03-04T09:30:10.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.795+0000 7effed800640 1 -- 172.21.15.6:0/1941287522 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7effe80af580 msgr2=0x7effe815c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:10.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.795+0000 7effed800640 1 --2- 172.21.15.6:0/1941287522 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7effe80af580 0x7effe815c600 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7effd40097a0 tx=0x7effd402b790 comp rx=0 tx=0).stop 2024-03-04T09:30:10.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.795+0000 7effed800640 1 -- 172.21.15.6:0/1941287522 shutdown_connections 2024-03-04T09:30:10.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.795+0000 7effed800640 1 --2- 172.21.15.6:0/1941287522 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7effe80af580 0x7effe815c600 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:10.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.795+0000 7effed800640 1 -- 172.21.15.6:0/1941287522 >> 172.21.15.6:0/1941287522 conn(0x7effe806afb0 msgr2=0x7effe806b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:10.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.795+0000 7effed800640 1 -- 172.21.15.6:0/1941287522 shutdown_connections 2024-03-04T09:30:10.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.795+0000 7effed800640 1 -- 172.21.15.6:0/1941287522 wait complete. 2024-03-04T09:30:10.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.796+0000 7effed800640 1 Processor -- start 2024-03-04T09:30:10.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.797+0000 7effed800640 1 -- start start 2024-03-04T09:30:10.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.797+0000 7effed800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7effe80af1a0 0x7effe80ce5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:10.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.797+0000 7effed800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7effe80af580 0x7effe814c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:10.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.798+0000 7effed800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7effe80cdc20 0x7effe814cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:10.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.798+0000 7effed800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effe8155e70 con 0x7effe80af1a0 2024-03-04T09:30:10.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.798+0000 7effed800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effe80b3730 con 0x7effe80af580 2024-03-04T09:30:10.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.798+0000 7effed800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effe8155ff0 con 0x7effe80cdc20 2024-03-04T09:30:10.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.798+0000 7effe6a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7effe80af1a0 0x7effe80ce5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:10.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.798+0000 7effe6a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7effe80af1a0 0x7effe80ce5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51840/0 (socket says 172.21.15.6:51840) 2024-03-04T09:30:10.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.798+0000 7effe6a00640 1 -- 172.21.15.6:0/48956211 learned_addr learned my addr 172.21.15.6:0/48956211 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:30:10.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.798+0000 7effe6000640 1 --2- 172.21.15.6:0/48956211 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7effe80af580 0x7effe814c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:10.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.798+0000 7effe6a00640 1 -- 172.21.15.6:0/48956211 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7effe80cdc20 msgr2=0x7effe814cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:10.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.798+0000 7effe6a00640 1 --2- 172.21.15.6:0/48956211 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7effe80cdc20 0x7effe814cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:10.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.798+0000 7effe6a00640 1 -- 172.21.15.6:0/48956211 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7effe80af580 msgr2=0x7effe814c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:10.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.798+0000 7effe6a00640 1 --2- 172.21.15.6:0/48956211 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7effe80af580 0x7effe814c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:10.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.798+0000 7effe6a00640 1 -- 172.21.15.6:0/48956211 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effe814a180 con 0x7effe80af1a0 2024-03-04T09:30:10.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.798+0000 7effe6000640 1 --2- 172.21.15.6:0/48956211 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7effe80af580 0x7effe814c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:30:10.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.798+0000 7effe6a00640 1 --2- 172.21.15.6:0/48956211 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7effe80af1a0 0x7effe80ce5e0 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7effd800c6f0 tx=0x7effd800cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:10.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.799+0000 7effdea00640 1 -- 172.21.15.6:0/48956211 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effd8005ad0 con 0x7effe80af1a0 2024-03-04T09:30:10.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.799+0000 7effdea00640 1 -- 172.21.15.6:0/48956211 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effd8005c70 con 0x7effe80af1a0 2024-03-04T09:30:10.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.799+0000 7effed800640 1 -- 172.21.15.6:0/48956211 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effe814a460 con 0x7effe80af1a0 2024-03-04T09:30:10.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.799+0000 7effdea00640 1 -- 172.21.15.6:0/48956211 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effd8007270 con 0x7effe80af1a0 2024-03-04T09:30:10.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.799+0000 7effed800640 1 -- 172.21.15.6:0/48956211 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effe814aa20 con 0x7effe80af1a0 2024-03-04T09:30:10.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.800+0000 7effed800640 1 -- 172.21.15.6:0/48956211 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effe804f3a0 con 0x7effe80af1a0 2024-03-04T09:30:10.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.801+0000 7effdea00640 1 -- 172.21.15.6:0/48956211 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7effd8007410 con 0x7effe80af1a0 2024-03-04T09:30:10.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.802+0000 7effdea00640 1 --2- 172.21.15.6:0/48956211 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7effc806b4d0 0x7effc806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:10.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.802+0000 7effdea00640 1 -- 172.21.15.6:0/48956211 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 4943+0+0 (secure 0 0 0) 0x7effd8089e20 con 0x7effe80af1a0 2024-03-04T09:30:10.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.803+0000 7effe6000640 1 --2- 172.21.15.6:0/48956211 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7effc806b4d0 0x7effc806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:10.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.804+0000 7effe6000640 1 --2- 172.21.15.6:0/48956211 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7effc806b4d0 0x7effc806d990 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7effd4034000 tx=0x7effd402f000 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:10.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:10.807+0000 7effdea00640 1 -- 172.21.15.6:0/48956211 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7effd8058f00 con 0x7effe80af1a0 2024-03-04T09:30:11.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:11.079+0000 7effed800640 1 -- 172.21.15.6:0/48956211 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_fancy_layout"]} v 0) v1 -- 0x7effe80cf700 con 0x7effe80af1a0 2024-03-04T09:30:12.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.085+0000 7effdea00640 1 -- 172.21.15.6:0/48956211 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_fancy_layout"]}]=0 v281) v1 ==== 132+0+0 (secure 0 0 0) 0x7effd8058800 con 0x7effe80af1a0 2024-03-04T09:30:12.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.088+0000 7effed800640 1 -- 172.21.15.6:0/48956211 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7effc806b4d0 msgr2=0x7effc806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:12.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.088+0000 7effed800640 1 --2- 172.21.15.6:0/48956211 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7effc806b4d0 0x7effc806d990 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7effd4034000 tx=0x7effd402f000 comp rx=0 tx=0).stop 2024-03-04T09:30:12.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.088+0000 7effed800640 1 -- 172.21.15.6:0/48956211 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7effe80af1a0 msgr2=0x7effe80ce5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:12.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.088+0000 7effed800640 1 --2- 172.21.15.6:0/48956211 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7effe80af1a0 0x7effe80ce5e0 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7effd800c6f0 tx=0x7effd800cbb0 comp rx=0 tx=0).stop 2024-03-04T09:30:12.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.088+0000 7effed800640 1 -- 172.21.15.6:0/48956211 shutdown_connections 2024-03-04T09:30:12.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.089+0000 7effed800640 1 --2- 172.21.15.6:0/48956211 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7effc806b4d0 0x7effc806d990 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:12.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.089+0000 7effed800640 1 --2- 172.21.15.6:0/48956211 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7effe80cdc20 0x7effe814cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:12.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.089+0000 7effed800640 1 --2- 172.21.15.6:0/48956211 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7effe80af580 0x7effe814c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:12.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.089+0000 7effed800640 1 --2- 172.21.15.6:0/48956211 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7effe80af1a0 0x7effe80ce5e0 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:12.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.089+0000 7effed800640 1 -- 172.21.15.6:0/48956211 >> 172.21.15.6:0/48956211 conn(0x7effe806afb0 msgr2=0x7effe80ccec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:12.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.089+0000 7effed800640 1 -- 172.21.15.6:0/48956211 shutdown_connections 2024-03-04T09:30:12.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.089+0000 7effed800640 1 -- 172.21.15.6:0/48956211 wait complete. 2024-03-04T09:30:12.106 INFO:tasks.cephfs_test_runner:purge stray file with fancy layout ... ok 2024-03-04T09:30:12.107 INFO:tasks.cephfs_test_runner:Starting test: test_files_throttle (tasks.cephfs.test_strays.TestStrays) 2024-03-04T09:30:12.107 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T09:30:12.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.332+0000 7f3b2b600640 1 Processor -- start 2024-03-04T09:30:12.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.332+0000 7f3b2b600640 1 -- start start 2024-03-04T09:30:12.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.333+0000 7f3b2b600640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f3b240b3c90 0x7f3b240cdc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:12.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.333+0000 7f3b2b600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f3b240cf5c0 0x7f3b240b4070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:12.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.333+0000 7f3b2b600640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b240b45b0 con 0x7f3b2406cb10 2024-03-04T09:30:12.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.333+0000 7f3b2b600640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b240ce160 con 0x7f3b240cf5c0 2024-03-04T09:30:12.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.333+0000 7f3b2b600640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b240b4860 con 0x7f3b240b3c90 2024-03-04T09:30:12.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.334+0000 7f3b23e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f3b240cf5c0 0x7f3b240b4070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:12.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.334+0000 7f3b23e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f3b240cf5c0 0x7f3b240b4070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51842/0 (socket says 172.21.15.6:51842) 2024-03-04T09:30:12.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.334+0000 7f3b23e00640 1 -- 172.21.15.6:0/1653872390 learned_addr learned my addr 172.21.15.6:0/1653872390 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:30:12.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.334+0000 7f3b28c00640 1 --2- 172.21.15.6:0/1653872390 >> v2:172.21.15.146:3300/0 conn(0x7f3b240b3c90 0x7f3b240cdc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:12.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.334+0000 7f3b23e00640 1 -- 172.21.15.6:0/1653872390 >> v2:172.21.15.146:3300/0 conn(0x7f3b240b3c90 msgr2=0x7f3b240cdc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:12.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.334+0000 7f3b23e00640 1 --2- 172.21.15.6:0/1653872390 >> v2:172.21.15.146:3300/0 conn(0x7f3b240b3c90 0x7f3b240cdc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:12.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.334+0000 7f3b23e00640 1 -- 172.21.15.6:0/1653872390 >> v1:172.21.15.6:6789/0 conn(0x7f3b2406cb10 legacy=0x7f3b2406cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:12.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.334+0000 7f3b23e00640 1 -- 172.21.15.6:0/1653872390 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b240b4b40 con 0x7f3b240cf5c0 2024-03-04T09:30:12.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.334+0000 7f3b28c00640 1 --2- 172.21.15.6:0/1653872390 >> v2:172.21.15.146:3300/0 conn(0x7f3b240b3c90 0x7f3b240cdc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:30:12.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.334+0000 7f3b23400640 1 -- 172.21.15.6:0/1653872390 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4034707118 0 0) 0x7f3b240b45b0 con 0x7f3b2406cb10 2024-03-04T09:30:12.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.336+0000 7f3b23e00640 1 --2- 172.21.15.6:0/1653872390 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3b240cf5c0 0x7f3b240b4070 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f3b24070800 tx=0x7f3b1002ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f03e16193744f6ca server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:12.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.336+0000 7f3b23400640 1 -- 172.21.15.6:0/1653872390 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b1002f850 con 0x7f3b240cf5c0 2024-03-04T09:30:12.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.336+0000 7f3b23400640 1 -- 172.21.15.6:0/1653872390 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b1002f9f0 con 0x7f3b240cf5c0 2024-03-04T09:30:12.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.336+0000 7f3b23400640 1 -- 172.21.15.6:0/1653872390 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b1002fdc0 con 0x7f3b240cf5c0 2024-03-04T09:30:12.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.336+0000 7f3b2b600640 1 -- 172.21.15.6:0/1653872390 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3b240cf5c0 msgr2=0x7f3b240b4070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:12.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.336+0000 7f3b2b600640 1 --2- 172.21.15.6:0/1653872390 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3b240cf5c0 0x7f3b240b4070 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f3b24070800 tx=0x7f3b1002ba30 comp rx=0 tx=0).stop 2024-03-04T09:30:12.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.336+0000 7f3b2b600640 1 -- 172.21.15.6:0/1653872390 shutdown_connections 2024-03-04T09:30:12.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.336+0000 7f3b2b600640 1 --2- 172.21.15.6:0/1653872390 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3b240cf5c0 0x7f3b240b4070 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:12.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.336+0000 7f3b2b600640 1 --2- 172.21.15.6:0/1653872390 >> v2:172.21.15.146:3300/0 conn(0x7f3b240b3c90 0x7f3b240cdc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:12.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.336+0000 7f3b2b600640 1 -- 172.21.15.6:0/1653872390 >> 172.21.15.6:0/1653872390 conn(0x7f3b2406afb0 msgr2=0x7f3b2406b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:12.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.337+0000 7f3b2b600640 1 -- 172.21.15.6:0/1653872390 shutdown_connections 2024-03-04T09:30:12.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.337+0000 7f3b2b600640 1 -- 172.21.15.6:0/1653872390 wait complete. 2024-03-04T09:30:12.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.338+0000 7f3b2b600640 1 Processor -- start 2024-03-04T09:30:12.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.338+0000 7f3b2b600640 1 -- start start 2024-03-04T09:30:12.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.339+0000 7f3b2b600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3b2406cb10 0x7f3b24154b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:12.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.339+0000 7f3b2b600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3b240b3c90 0x7f3b24152f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:12.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.339+0000 7f3b2b600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3b240cf5c0 0x7f3b24149db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:12.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.339+0000 7f3b2b600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b2415b340 con 0x7f3b240cf5c0 2024-03-04T09:30:12.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.339+0000 7f3b2b600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b240b5100 con 0x7f3b240b3c90 2024-03-04T09:30:12.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.339+0000 7f3b2b600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b2406d690 con 0x7f3b2406cb10 2024-03-04T09:30:12.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.340+0000 7f3b29600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3b240cf5c0 0x7f3b24149db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:12.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.340+0000 7f3b29600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3b240cf5c0 0x7f3b24149db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51858/0 (socket says 172.21.15.6:51858) 2024-03-04T09:30:12.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.340+0000 7f3b29600640 1 -- 172.21.15.6:0/3445012617 learned_addr learned my addr 172.21.15.6:0/3445012617 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:30:12.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.340+0000 7f3b28c00640 1 --2- 172.21.15.6:0/3445012617 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3b2406cb10 0x7f3b24154b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:12.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.340+0000 7f3b23e00640 1 --2- 172.21.15.6:0/3445012617 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3b240b3c90 0x7f3b24152f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:12.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.340+0000 7f3b29600640 1 -- 172.21.15.6:0/3445012617 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3b2406cb10 msgr2=0x7f3b24154b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:12.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.340+0000 7f3b29600640 1 --2- 172.21.15.6:0/3445012617 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3b2406cb10 0x7f3b24154b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:12.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.340+0000 7f3b29600640 1 -- 172.21.15.6:0/3445012617 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3b240b3c90 msgr2=0x7f3b24152f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:12.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.340+0000 7f3b29600640 1 --2- 172.21.15.6:0/3445012617 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3b240b3c90 0x7f3b24152f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:12.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.340+0000 7f3b29600640 1 -- 172.21.15.6:0/3445012617 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b24155280 con 0x7f3b240cf5c0 2024-03-04T09:30:12.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.340+0000 7f3b23e00640 1 --2- 172.21.15.6:0/3445012617 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3b240b3c90 0x7f3b24152f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:30:12.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.341+0000 7f3b29600640 1 --2- 172.21.15.6:0/3445012617 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3b240cf5c0 0x7f3b24149db0 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f3b1c00c7e0 tx=0x7f3b1c00cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:12.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.341+0000 7f3b20c00640 1 -- 172.21.15.6:0/3445012617 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b1c017070 con 0x7f3b240cf5c0 2024-03-04T09:30:12.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.341+0000 7f3b20c00640 1 -- 172.21.15.6:0/3445012617 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b1c003dc0 con 0x7f3b240cf5c0 2024-03-04T09:30:12.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.341+0000 7f3b2b600640 1 -- 172.21.15.6:0/3445012617 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b24155560 con 0x7f3b240cf5c0 2024-03-04T09:30:12.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.341+0000 7f3b20c00640 1 -- 172.21.15.6:0/3445012617 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b1c0052b0 con 0x7f3b240cf5c0 2024-03-04T09:30:12.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.341+0000 7f3b2b600640 1 -- 172.21.15.6:0/3445012617 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b24155b20 con 0x7f3b240cf5c0 2024-03-04T09:30:12.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.342+0000 7f3b2b600640 1 -- 172.21.15.6:0/3445012617 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b24069510 con 0x7f3b240cf5c0 2024-03-04T09:30:12.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.343+0000 7f3b20c00640 1 -- 172.21.15.6:0/3445012617 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3b1c01a070 con 0x7f3b240cf5c0 2024-03-04T09:30:12.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.344+0000 7f3b20c00640 1 --2- 172.21.15.6:0/3445012617 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f3b0406b4d0 0x7f3b0406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:12.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.344+0000 7f3b20c00640 1 -- 172.21.15.6:0/3445012617 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f3b1c089670 con 0x7f3b240cf5c0 2024-03-04T09:30:12.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.345+0000 7f3b28c00640 1 --2- 172.21.15.6:0/3445012617 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f3b0406b4d0 0x7f3b0406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:12.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.346+0000 7f3b28c00640 1 --2- 172.21.15.6:0/3445012617 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f3b0406b4d0 0x7f3b0406d990 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f3b1800a990 tx=0x7f3b18008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:12.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.349+0000 7f3b20c00640 1 -- 172.21.15.6:0/3445012617 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3b1c058750 con 0x7f3b240cf5c0 2024-03-04T09:30:12.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.553+0000 7f3b2b600640 1 -- 172.21.15.6:0/3445012617 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3b240b18b0 con 0x7f3b240cf5c0 2024-03-04T09:30:12.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.555+0000 7f3b20c00640 1 -- 172.21.15.6:0/3445012617 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) v1 ==== 74+0+12971 (secure 0 0 0) 0x7f3b1c058050 con 0x7f3b240cf5c0 2024-03-04T09:30:12.552 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:30:12.552 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":86,"fsid":"edce0578-5ae4-4402-b45a-552bdef84d7c","created":"2024-03-04T09:24:54.523780+0000","modified":"2024-03-04T09:30:10.060755+0000","last_up_change":"2024-03-04T09:25:07.089049+0000","last_in_change":"2024-03-04T09:24:57.982077+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T09:25:07.935198+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a96574f-81f1-4bc6-8463-757520b84a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6825","nonce":1364449401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6827","nonce":1364449401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6831","nonce":1364449401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6829","nonce":1364449401}]},"public_addr":"172.21.15.6:6825/1364449401","cluster_addr":"172.21.15.6:6827/1364449401","heartbeat_back_addr":"172.21.15.6:6831/1364449401","heartbeat_front_addr":"172.21.15.6:6829/1364449401","state":["exists","up"]},{"osd":1,"uuid":"7d79b9c1-6b87-476a-bc36-7db5ee64cd90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6809","nonce":2494103078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6811","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6813","nonce":2494103078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6821","nonce":2494103078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6817","nonce":2494103078}]},"public_addr":"172.21.15.6:6809/2494103078","cluster_addr":"172.21.15.6:6813/2494103078","heartbeat_back_addr":"172.21.15.6:6821/2494103078","heartbeat_front_addr":"172.21.15.6:6817/2494103078","state":["exists","up"]},{"osd":2,"uuid":"492331b9-4ee9-40b6-bacd-85d44cd6c45a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6812","nonce":1500066681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6816","nonce":1500066681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6824","nonce":1500066681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6820","nonce":1500066681}]},"public_addr":"172.21.15.6:6812/1500066681","cluster_addr":"172.21.15.6:6816/1500066681","heartbeat_back_addr":"172.21.15.6:6824/1500066681","heartbeat_front_addr":"172.21.15.6:6820/1500066681","state":["exists","up"]},{"osd":3,"uuid":"21bb2efe-9ba7-4bdc-bc31-180cd1102e21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6801","nonce":1558806123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6803","nonce":1558806123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6807","nonce":1558806123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6805","nonce":1558806123}]},"public_addr":"172.21.15.6:6801/1558806123","cluster_addr":"172.21.15.6:6803/1558806123","heartbeat_back_addr":"172.21.15.6:6807/1558806123","heartbeat_front_addr":"172.21.15.6:6805/1558806123","state":["exists","up"]},{"osd":4,"uuid":"6e2c4b71-19d9-4737-83d9-558705d8644a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6801","nonce":1526135780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6803","nonce":1526135780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6807","nonce":1526135780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6805","nonce":1526135780}]},"public_addr":"172.21.15.146:6801/1526135780","cluster_addr":"172.21.15.146:6803/1526135780","heartbeat_back_addr":"172.21.15.146:6807/1526135780","heartbeat_front_addr":"172.21.15.146:6805/1526135780","state":["exists","up"]},{"osd":5,"uuid":"fb69c525-f1be-4b7d-ab32-6c29b0d6fddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6809","nonce":460262507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6811","nonce":460262507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6815","nonce":460262507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6813","nonce":460262507}]},"public_addr":"172.21.15.146:6809/460262507","cluster_addr":"172.21.15.146:6811/460262507","heartbeat_back_addr":"172.21.15.146:6815/460262507","heartbeat_front_addr":"172.21.15.146:6813/460262507","state":["exists","up"]},{"osd":6,"uuid":"c32e7fed-370d-4401-8ce0-6a4335b0d168","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6823","nonce":1392897493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6827","nonce":1392897493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6831","nonce":1392897493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6829","nonce":1392897493}]},"public_addr":"172.21.15.146:6823/1392897493","cluster_addr":"172.21.15.146:6827/1392897493","heartbeat_back_addr":"172.21.15.146:6831/1392897493","heartbeat_front_addr":"172.21.15.146:6829/1392897493","state":["exists","up"]},{"osd":7,"uuid":"75a79b44-fa5f-426d-a7d3-71f21f78835c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6817","nonce":3268323156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6819","nonce":3268323156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6825","nonce":3268323156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6822","nonce":3268323156}]},"public_addr":"172.21.15.146:6817/3268323156","cluster_addr":"172.21.15.146:6819/3268323156","heartbeat_back_addr":"172.21.15.146:6825/3268323156","heartbeat_front_addr":"172.21.15.146:6822/3268323156","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.310437+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.295850+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.294674+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.526855+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.300023+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.595160+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.346046+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.558741+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.6:6835/1667002065":"2024-03-05T09:30:06.879071+0000","172.21.15.6:6834/1667002065":"2024-03-05T09:30:06.879071+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T09:30:12.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.557+0000 7f3b2b600640 1 -- 172.21.15.6:0/3445012617 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f3b0406b4d0 msgr2=0x7f3b0406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:12.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.557+0000 7f3b2b600640 1 --2- 172.21.15.6:0/3445012617 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f3b0406b4d0 0x7f3b0406d990 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f3b1800a990 tx=0x7f3b18008040 comp rx=0 tx=0).stop 2024-03-04T09:30:12.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.557+0000 7f3b2b600640 1 -- 172.21.15.6:0/3445012617 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3b240cf5c0 msgr2=0x7f3b24149db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:12.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.557+0000 7f3b2b600640 1 --2- 172.21.15.6:0/3445012617 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3b240cf5c0 0x7f3b24149db0 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f3b1c00c7e0 tx=0x7f3b1c00cca0 comp rx=0 tx=0).stop 2024-03-04T09:30:12.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.557+0000 7f3b2b600640 1 -- 172.21.15.6:0/3445012617 shutdown_connections 2024-03-04T09:30:12.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.557+0000 7f3b2b600640 1 --2- 172.21.15.6:0/3445012617 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3b240cf5c0 0x7f3b24149db0 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:12.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.557+0000 7f3b2b600640 1 --2- 172.21.15.6:0/3445012617 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3b240b3c90 0x7f3b24152f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:12.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.557+0000 7f3b2b600640 1 --2- 172.21.15.6:0/3445012617 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f3b0406b4d0 0x7f3b0406d990 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:12.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.557+0000 7f3b2b600640 1 --2- 172.21.15.6:0/3445012617 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3b2406cb10 0x7f3b24154b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:12.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.557+0000 7f3b2b600640 1 -- 172.21.15.6:0/3445012617 >> 172.21.15.6:0/3445012617 conn(0x7f3b2406afb0 msgr2=0x7f3b240cec80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:12.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.557+0000 7f3b2b600640 1 -- 172.21.15.6:0/3445012617 shutdown_connections 2024-03-04T09:30:12.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.557+0000 7f3b2b600640 1 -- 172.21.15.6:0/3445012617 wait complete. 2024-03-04T09:30:12.572 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T09:25:07.935198+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-04T09:30:12.573 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-04T09:30:12.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.798+0000 7ffaadc00640 1 Processor -- start 2024-03-04T09:30:12.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.798+0000 7ffaadc00640 1 -- start start 2024-03-04T09:30:12.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.800+0000 7ffaadc00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ffaa806da40 0x7ffaa806e340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:12.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.800+0000 7ffaadc00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ffaa80ccad0 0x7ffaa80ccf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:12.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.800+0000 7ffaadc00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffaa806d090 con 0x7ffaa815e680 2024-03-04T09:30:12.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.800+0000 7ffaadc00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffaa8161c70 con 0x7ffaa806da40 2024-03-04T09:30:12.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.800+0000 7ffaadc00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffaa815eb80 con 0x7ffaa80ccad0 2024-03-04T09:30:12.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.800+0000 7ffaa6a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ffaa806da40 0x7ffaa806e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:12.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.800+0000 7ffaa6a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ffaa806da40 0x7ffaa806e340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51862/0 (socket says 172.21.15.6:51862) 2024-03-04T09:30:12.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.800+0000 7ffaa6a00640 1 -- 172.21.15.6:0/2027497401 learned_addr learned my addr 172.21.15.6:0/2027497401 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:30:12.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.801+0000 7ffaa6a00640 1 -- 172.21.15.6:0/2027497401 >> v2:172.21.15.146:3300/0 conn(0x7ffaa80ccad0 msgr2=0x7ffaa80ccf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:12.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.801+0000 7ffaa6a00640 1 --2- 172.21.15.6:0/2027497401 >> v2:172.21.15.146:3300/0 conn(0x7ffaa80ccad0 0x7ffaa80ccf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:12.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.801+0000 7ffaa6a00640 1 -- 172.21.15.6:0/2027497401 >> v1:172.21.15.146:6789/0 conn(0x7ffaa815e680 legacy=0x7ffaa806c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:12.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.801+0000 7ffaa6a00640 1 -- 172.21.15.6:0/2027497401 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaa815ee60 con 0x7ffaa806da40 2024-03-04T09:30:12.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.802+0000 7ffaa6a00640 1 --2- 172.21.15.6:0/2027497401 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffaa806da40 0x7ffaa806e340 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7ffaa80cf210 tx=0x7ffa9802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56b8009eb982ea83 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:12.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.803+0000 7ffaa5600640 1 -- 172.21.15.6:0/2027497401 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa98035020 con 0x7ffaa806da40 2024-03-04T09:30:12.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.803+0000 7ffaa5600640 1 -- 172.21.15.6:0/2027497401 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa98031870 con 0x7ffaa806da40 2024-03-04T09:30:12.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.803+0000 7ffaa5600640 1 -- 172.21.15.6:0/2027497401 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa98031c40 con 0x7ffaa806da40 2024-03-04T09:30:12.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.803+0000 7ffaadc00640 1 -- 172.21.15.6:0/2027497401 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffaa806da40 msgr2=0x7ffaa806e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:12.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.803+0000 7ffaadc00640 1 --2- 172.21.15.6:0/2027497401 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffaa806da40 0x7ffaa806e340 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7ffaa80cf210 tx=0x7ffa9802b910 comp rx=0 tx=0).stop 2024-03-04T09:30:12.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.804+0000 7ffaadc00640 1 -- 172.21.15.6:0/2027497401 shutdown_connections 2024-03-04T09:30:12.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.804+0000 7ffaadc00640 1 --2- 172.21.15.6:0/2027497401 >> v2:172.21.15.146:3300/0 conn(0x7ffaa80ccad0 0x7ffaa80ccf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:12.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.804+0000 7ffaadc00640 1 --2- 172.21.15.6:0/2027497401 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffaa806da40 0x7ffaa806e340 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:12.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.804+0000 7ffaadc00640 1 -- 172.21.15.6:0/2027497401 >> 172.21.15.6:0/2027497401 conn(0x7ffaa806b3d0 msgr2=0x7ffaa806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:12.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.804+0000 7ffaadc00640 1 -- 172.21.15.6:0/2027497401 shutdown_connections 2024-03-04T09:30:12.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.804+0000 7ffaadc00640 1 -- 172.21.15.6:0/2027497401 wait complete. 2024-03-04T09:30:12.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.805+0000 7ffaadc00640 1 Processor -- start 2024-03-04T09:30:12.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.806+0000 7ffaadc00640 1 -- start start 2024-03-04T09:30:12.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.806+0000 7ffaadc00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffaa806da40 0x7ffaa80ae000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:12.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.806+0000 7ffaadc00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ffaa80ccad0 0x7ffaa809c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:12.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.806+0000 7ffaadc00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffaa815e680 0x7ffaa809cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:12.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.806+0000 7ffaadc00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaa806e880 con 0x7ffaa806da40 2024-03-04T09:30:12.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.806+0000 7ffaadc00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaa815f420 con 0x7ffaa815e680 2024-03-04T09:30:12.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.806+0000 7ffaadc00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffaa806ea00 con 0x7ffaa80ccad0 2024-03-04T09:30:12.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.807+0000 7ffaa6a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffaa806da40 0x7ffaa80ae000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:12.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.807+0000 7ffaa6a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffaa806da40 0x7ffaa80ae000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51864/0 (socket says 172.21.15.6:51864) 2024-03-04T09:30:12.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.807+0000 7ffaa6a00640 1 -- 172.21.15.6:0/2417183097 learned_addr learned my addr 172.21.15.6:0/2417183097 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:30:12.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.807+0000 7ffaa6000640 1 --2- 172.21.15.6:0/2417183097 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ffaa80ccad0 0x7ffaa809c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:12.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.807+0000 7ffaa7400640 1 --2- 172.21.15.6:0/2417183097 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffaa815e680 0x7ffaa809cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:12.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.807+0000 7ffaa6a00640 1 -- 172.21.15.6:0/2417183097 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ffaa80ccad0 msgr2=0x7ffaa809c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:12.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.807+0000 7ffaa6a00640 1 --2- 172.21.15.6:0/2417183097 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ffaa80ccad0 0x7ffaa809c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:12.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.807+0000 7ffaa6a00640 1 -- 172.21.15.6:0/2417183097 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffaa815e680 msgr2=0x7ffaa809cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:12.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.807+0000 7ffaa6a00640 1 --2- 172.21.15.6:0/2417183097 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffaa815e680 0x7ffaa809cd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:12.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.807+0000 7ffaa6a00640 1 -- 172.21.15.6:0/2417183097 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaa81f7730 con 0x7ffaa806da40 2024-03-04T09:30:12.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.807+0000 7ffaa6000640 1 --2- 172.21.15.6:0/2417183097 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ffaa80ccad0 0x7ffaa809c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:30:12.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.808+0000 7ffaa6a00640 1 --2- 172.21.15.6:0/2417183097 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffaa806da40 0x7ffaa80ae000 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7ffa98000c00 tx=0x7ffa980046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:12.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.808+0000 7ffa9ea00640 1 -- 172.21.15.6:0/2417183097 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa980443f0 con 0x7ffaa806da40 2024-03-04T09:30:12.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.808+0000 7ffa9ea00640 1 -- 172.21.15.6:0/2417183097 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa98044590 con 0x7ffaa806da40 2024-03-04T09:30:12.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.808+0000 7ffa9ea00640 1 -- 172.21.15.6:0/2417183097 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa98044980 con 0x7ffaa806da40 2024-03-04T09:30:12.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.808+0000 7ffaadc00640 1 -- 172.21.15.6:0/2417183097 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffaa81f7a10 con 0x7ffaa806da40 2024-03-04T09:30:12.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.808+0000 7ffaadc00640 1 -- 172.21.15.6:0/2417183097 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffaa81f7fd0 con 0x7ffaa806da40 2024-03-04T09:30:12.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.809+0000 7ffaadc00640 1 -- 172.21.15.6:0/2417183097 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffaa804f330 con 0x7ffaa806da40 2024-03-04T09:30:12.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.810+0000 7ffa9ea00640 1 -- 172.21.15.6:0/2417183097 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ffa9804b060 con 0x7ffaa806da40 2024-03-04T09:30:12.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.811+0000 7ffa9ea00640 1 --2- 172.21.15.6:0/2417183097 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ffa8406b4d0 0x7ffa8406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:12.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.811+0000 7ffa9ea00640 1 -- 172.21.15.6:0/2417183097 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 4943+0+0 (secure 0 0 0) 0x7ffa980ac2d0 con 0x7ffaa806da40 2024-03-04T09:30:12.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.812+0000 7ffaa6000640 1 --2- 172.21.15.6:0/2417183097 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ffa8406b4d0 0x7ffa8406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:12.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.813+0000 7ffaa6000640 1 --2- 172.21.15.6:0/2417183097 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ffa8406b4d0 0x7ffa8406d990 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7ffaa80ae950 tx=0x7ffa90008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:12.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:12.816+0000 7ffa9ea00640 1 -- 172.21.15.6:0/2417183097 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ffa9803f070 con 0x7ffaa806da40 2024-03-04T09:30:13.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:13.017+0000 7ffaadc00640 1 -- 172.21.15.6:0/2417183097 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ffaa806b9c0 con 0x7ffaa806da40 2024-03-04T09:30:13.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:13.017+0000 7ffa9ea00640 1 -- 172.21.15.6:0/2417183097 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v86) v1 ==== 93+0+10 (secure 0 0 0) 0x7ffa9807b3b0 con 0x7ffaa806da40 2024-03-04T09:30:13.014 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-04T09:30:13.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:13.020+0000 7ffaadc00640 1 -- 172.21.15.6:0/2417183097 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ffa8406b4d0 msgr2=0x7ffa8406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:13.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:13.020+0000 7ffaadc00640 1 --2- 172.21.15.6:0/2417183097 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ffa8406b4d0 0x7ffa8406d990 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7ffaa80ae950 tx=0x7ffa90008040 comp rx=0 tx=0).stop 2024-03-04T09:30:13.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:13.020+0000 7ffaadc00640 1 -- 172.21.15.6:0/2417183097 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffaa806da40 msgr2=0x7ffaa80ae000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:13.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:13.020+0000 7ffaadc00640 1 --2- 172.21.15.6:0/2417183097 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffaa806da40 0x7ffaa80ae000 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7ffa98000c00 tx=0x7ffa980046c0 comp rx=0 tx=0).stop 2024-03-04T09:30:13.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:13.020+0000 7ffaadc00640 1 -- 172.21.15.6:0/2417183097 shutdown_connections 2024-03-04T09:30:13.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:13.020+0000 7ffaadc00640 1 --2- 172.21.15.6:0/2417183097 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffaa815e680 0x7ffaa809cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:13.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:13.020+0000 7ffaadc00640 1 --2- 172.21.15.6:0/2417183097 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ffa8406b4d0 0x7ffa8406d990 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:13.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:13.020+0000 7ffaadc00640 1 --2- 172.21.15.6:0/2417183097 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ffaa80ccad0 0x7ffaa809c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:13.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:13.020+0000 7ffaadc00640 1 --2- 172.21.15.6:0/2417183097 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffaa806da40 0x7ffaa80ae000 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:13.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:13.020+0000 7ffaadc00640 1 -- 172.21.15.6:0/2417183097 >> 172.21.15.6:0/2417183097 conn(0x7ffaa806b3d0 msgr2=0x7ffaa806e100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:13.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:13.021+0000 7ffaadc00640 1 -- 172.21.15.6:0/2417183097 shutdown_connections 2024-03-04T09:30:13.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:13.021+0000 7ffaadc00640 1 -- 172.21.15.6:0/2417183097 wait complete. 2024-03-04T09:30:13.037 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_files_throttle' 2024-03-04T09:30:13.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:13.261+0000 7f310e800640 1 Processor -- start 2024-03-04T09:30:13.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:13.261+0000 7f310e800640 1 -- start start 2024-03-04T09:30:13.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:13.263+0000 7f310e800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f31080cd060 0x7f310815b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:13.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:13.263+0000 7f310e800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f310815e810 0x7f31080b3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:13.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:13.263+0000 7f310e800640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f310815ed10 con 0x7f31081552b0 2024-03-04T09:30:13.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:13.263+0000 7f310e800640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f310815eff0 con 0x7f31080cd060 2024-03-04T09:30:13.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:13.263+0000 7f310e800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f310815f2d0 con 0x7f310815e810 2024-03-04T09:30:13.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:13.264+0000 7f310de00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f310815e810 0x7f31080b3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:13.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:13.264+0000 7f310de00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f310815e810 0x7f31080b3cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.6:56650/0 (socket says 172.21.15.6:56650) 2024-03-04T09:30:13.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:13.264+0000 7f310de00640 1 -- 172.21.15.6:0/1421252218 learned_addr learned my addr 172.21.15.6:0/1421252218 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:30:13.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:13.264+0000 7f310ca00640 1 --2- 172.21.15.6:0/1421252218 >> v2:172.21.15.146:3300/0 conn(0x7f31080cd060 0x7f310815b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:13.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:13.265+0000 7f310ca00640 1 -- 172.21.15.6:0/1421252218 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f310815e810 msgr2=0x7f31080b3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:13.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:13.265+0000 7f310ca00640 1 --2- 172.21.15.6:0/1421252218 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f310815e810 0x7f31080b3cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:13.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:13.265+0000 7f310ca00640 1 -- 172.21.15.6:0/1421252218 >> v1:172.21.15.146:6789/0 conn(0x7f31081552b0 legacy=0x7f31080cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:13.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:13.265+0000 7f310ca00640 1 -- 172.21.15.6:0/1421252218 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f310815f5b0 con 0x7f31080cd060 2024-03-04T09:30:13.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:13.265+0000 7f310de00640 1 --2- 172.21.15.6:0/1421252218 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f310815e810 0x7f31080b3cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T09:30:13.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:13.266+0000 7f310ca00640 1 --2- 172.21.15.6:0/1421252218 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f31080cd060 0x7f310815b1d0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f30f40097a0 tx=0x7f30f402b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fbb826ffff341a56 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:13.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:13.267+0000 7f3107e00640 1 -- 172.21.15.6:0/1421252218 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30f4031780 con 0x7f31080cd060 2024-03-04T09:30:13.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:13.267+0000 7f3107e00640 1 -- 172.21.15.6:0/1421252218 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30f4037070 con 0x7f31080cd060 2024-03-04T09:30:13.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:13.267+0000 7f3107e00640 1 -- 172.21.15.6:0/1421252218 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30f4031b40 con 0x7f31080cd060 2024-03-04T09:30:13.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:13.267+0000 7f310e800640 1 -- 172.21.15.6:0/1421252218 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f31080cd060 msgr2=0x7f310815b1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:13.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:13.267+0000 7f310e800640 1 --2- 172.21.15.6:0/1421252218 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f31080cd060 0x7f310815b1d0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f30f40097a0 tx=0x7f30f402b820 comp rx=0 tx=0).stop 2024-03-04T09:30:13.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:13.267+0000 7f310e800640 1 -- 172.21.15.6:0/1421252218 shutdown_connections 2024-03-04T09:30:13.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:13.267+0000 7f310e800640 1 --2- 172.21.15.6:0/1421252218 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f310815e810 0x7f31080b3cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:13.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:13.267+0000 7f310e800640 1 --2- 172.21.15.6:0/1421252218 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f31080cd060 0x7f310815b1d0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:13.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:13.267+0000 7f310e800640 1 -- 172.21.15.6:0/1421252218 >> 172.21.15.6:0/1421252218 conn(0x7f310806b3d0 msgr2=0x7f310806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:13.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:13.267+0000 7f310e800640 1 -- 172.21.15.6:0/1421252218 shutdown_connections 2024-03-04T09:30:13.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:13.267+0000 7f310e800640 1 -- 172.21.15.6:0/1421252218 wait complete. 2024-03-04T09:30:13.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:13.269+0000 7f310e800640 1 Processor -- start 2024-03-04T09:30:13.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:13.269+0000 7f310e800640 1 -- start start 2024-03-04T09:30:13.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:13.270+0000 7f310e800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f31080cd060 0x7f310814df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:13.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:13.270+0000 7f310e800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f31081552b0 0x7f3108150060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:13.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:13.270+0000 7f310e800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f310815e810 0x7f310815d120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:13.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:13.270+0000 7f310e800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f310806dd30 con 0x7f310815e810 2024-03-04T09:30:13.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:13.270+0000 7f310e800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31080b3760 con 0x7f31081552b0 2024-03-04T09:30:13.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:13.270+0000 7f310e800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f310806deb0 con 0x7f31080cd060 2024-03-04T09:30:13.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:13.270+0000 7f310de00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f310815e810 0x7f310815d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:13.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:13.271+0000 7f310de00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f310815e810 0x7f310815d120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51872/0 (socket says 172.21.15.6:51872) 2024-03-04T09:30:13.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:13.271+0000 7f310de00640 1 -- 172.21.15.6:0/3416828932 learned_addr learned my addr 172.21.15.6:0/3416828932 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:30:13.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:13.271+0000 7f310ca00640 1 --2- 172.21.15.6:0/3416828932 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f31081552b0 0x7f3108150060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:13.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:13.271+0000 7f310d400640 1 --2- 172.21.15.6:0/3416828932 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f31080cd060 0x7f310814df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:13.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:13.271+0000 7f310de00640 1 -- 172.21.15.6:0/3416828932 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f31080cd060 msgr2=0x7f310814df60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:13.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:13.271+0000 7f310de00640 1 --2- 172.21.15.6:0/3416828932 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f31080cd060 0x7f310814df60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:13.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:13.271+0000 7f310de00640 1 -- 172.21.15.6:0/3416828932 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f31081552b0 msgr2=0x7f3108150060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:13.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:13.271+0000 7f310de00640 1 --2- 172.21.15.6:0/3416828932 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f31081552b0 0x7f3108150060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:13.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:13.271+0000 7f310de00640 1 -- 172.21.15.6:0/3416828932 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f310814e4a0 con 0x7f310815e810 2024-03-04T09:30:13.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:13.271+0000 7f310d400640 1 --2- 172.21.15.6:0/3416828932 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f31080cd060 0x7f310814df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:30:13.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:13.271+0000 7f310de00640 1 --2- 172.21.15.6:0/3416828932 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f310815e810 0x7f310815d120 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f310000e370 tx=0x7f310000e830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:13.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:13.272+0000 7f3105600640 1 -- 172.21.15.6:0/3416828932 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f310000b2d0 con 0x7f310815e810 2024-03-04T09:30:13.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:13.272+0000 7f3105600640 1 -- 172.21.15.6:0/3416828932 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f310000b470 con 0x7f310815e810 2024-03-04T09:30:13.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:13.272+0000 7f310e800640 1 -- 172.21.15.6:0/3416828932 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f310814e740 con 0x7f310815e810 2024-03-04T09:30:13.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:13.272+0000 7f3105600640 1 -- 172.21.15.6:0/3416828932 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3100003bd0 con 0x7f310815e810 2024-03-04T09:30:13.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:13.272+0000 7f310e800640 1 -- 172.21.15.6:0/3416828932 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f310814ed00 con 0x7f310815e810 2024-03-04T09:30:13.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:13.273+0000 7f310e800640 1 -- 172.21.15.6:0/3416828932 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f310804f330 con 0x7f310815e810 2024-03-04T09:30:13.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:13.274+0000 7f3105600640 1 -- 172.21.15.6:0/3416828932 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3100005d10 con 0x7f310815e810 2024-03-04T09:30:13.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:13.275+0000 7f3105600640 1 --2- 172.21.15.6:0/3416828932 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f30e806b4d0 0x7f30e806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:13.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:13.275+0000 7f3105600640 1 -- 172.21.15.6:0/3416828932 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f310008a520 con 0x7f310815e810 2024-03-04T09:30:13.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:13.276+0000 7f310d400640 1 --2- 172.21.15.6:0/3416828932 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f30e806b4d0 0x7f30e806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:13.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:13.277+0000 7f310d400640 1 --2- 172.21.15.6:0/3416828932 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f30e806b4d0 0x7f30e806d990 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f30fc000f60 tx=0x7f30fc009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:13.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:13.279+0000 7f3105600640 1 -- 172.21.15.6:0/3416828932 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3100059610 con 0x7f310815e810 2024-03-04T09:30:13.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:13.547+0000 7f310e800640 1 -- 172.21.15.6:0/3416828932 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_files_throttle"]} v 0) v1 -- 0x7f31080b42c0 con 0x7f310815e810 2024-03-04T09:30:14.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:14.097+0000 7f3105600640 1 -- 172.21.15.6:0/3416828932 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_files_throttle"]}]=0 v283) v1 ==== 137+0+0 (secure 0 0 0) 0x7f3100058f10 con 0x7f310815e810 2024-03-04T09:30:14.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:14.100+0000 7f310e800640 1 -- 172.21.15.6:0/3416828932 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f30e806b4d0 msgr2=0x7f30e806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:14.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:14.100+0000 7f310e800640 1 --2- 172.21.15.6:0/3416828932 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f30e806b4d0 0x7f30e806d990 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f30fc000f60 tx=0x7f30fc009040 comp rx=0 tx=0).stop 2024-03-04T09:30:14.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:14.100+0000 7f310e800640 1 -- 172.21.15.6:0/3416828932 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f310815e810 msgr2=0x7f310815d120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:14.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:14.100+0000 7f310e800640 1 --2- 172.21.15.6:0/3416828932 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f310815e810 0x7f310815d120 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f310000e370 tx=0x7f310000e830 comp rx=0 tx=0).stop 2024-03-04T09:30:14.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:14.101+0000 7f310e800640 1 -- 172.21.15.6:0/3416828932 shutdown_connections 2024-03-04T09:30:14.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:14.101+0000 7f310e800640 1 --2- 172.21.15.6:0/3416828932 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f310815e810 0x7f310815d120 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:14.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:14.101+0000 7f310e800640 1 --2- 172.21.15.6:0/3416828932 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f31081552b0 0x7f3108150060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:14.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:14.101+0000 7f310e800640 1 --2- 172.21.15.6:0/3416828932 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f30e806b4d0 0x7f30e806d990 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:14.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:14.101+0000 7f310e800640 1 --2- 172.21.15.6:0/3416828932 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f31080cd060 0x7f310814df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:14.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:14.101+0000 7f310e800640 1 -- 172.21.15.6:0/3416828932 >> 172.21.15.6:0/3416828932 conn(0x7f310806b3d0 msgr2=0x7f31080cc920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:14.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:14.101+0000 7f310e800640 1 -- 172.21.15.6:0/3416828932 shutdown_connections 2024-03-04T09:30:14.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:14.101+0000 7f310e800640 1 -- 172.21.15.6:0/3416828932 wait complete. 2024-03-04T09:30:14.120 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-04T09:30:14.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:14.342+0000 7f65b3e00640 1 Processor -- start 2024-03-04T09:30:14.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:14.342+0000 7f65b3e00640 1 -- start start 2024-03-04T09:30:14.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:14.344+0000 7f65b3e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f65b415f810 0x7f65b40b1b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:14.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:14.344+0000 7f65b3e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f65b40b2080 0x7f65b40b2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:14.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:14.344+0000 7f65b3e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65b415fe40 con 0x7f65b4145430 2024-03-04T09:30:14.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:14.344+0000 7f65b3e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65b4160120 con 0x7f65b40b2080 2024-03-04T09:30:14.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:14.344+0000 7f65b3e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65b4160400 con 0x7f65b415f810 2024-03-04T09:30:14.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:14.345+0000 7f65b3400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f65b40b2080 0x7f65b40b2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:14.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:14.345+0000 7f65b2a00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f65b4145430 0x7f65b415cc90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:34842/0 (socket says 172.21.15.6:34842) 2024-03-04T09:30:14.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:14.345+0000 7f65b2a00640 1 -- 172.21.15.6:0/1627028401 learned_addr learned my addr 172.21.15.6:0/1627028401 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:30:14.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:14.345+0000 7f65abe00640 1 --2- 172.21.15.6:0/1627028401 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f65b415f810 0x7f65b40b1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:14.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:14.345+0000 7f65b3400640 1 -- 172.21.15.6:0/1627028401 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f65b415f810 msgr2=0x7f65b40b1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:14.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:14.345+0000 7f65b3400640 1 --2- 172.21.15.6:0/1627028401 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f65b415f810 0x7f65b40b1b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:14.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:14.345+0000 7f65b3400640 1 -- 172.21.15.6:0/1627028401 >> v1:172.21.15.6:6789/0 conn(0x7f65b4145430 legacy=0x7f65b415cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:14.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:14.345+0000 7f65b2000640 1 -- 172.21.15.6:0/1627028401 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1498972161 0 0) 0x7f65b415fe40 con 0x7f65b4145430 2024-03-04T09:30:14.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:14.345+0000 7f65b3400640 1 -- 172.21.15.6:0/1627028401 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65b41606e0 con 0x7f65b40b2080 2024-03-04T09:30:14.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:14.347+0000 7f65b3400640 1 --2- 172.21.15.6:0/1627028401 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f65b40b2080 0x7f65b40b2fd0 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f65ac009d90 tx=0x7f65ac02bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0d2167f7e407182 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:14.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:14.347+0000 7f65b2000640 1 -- 172.21.15.6:0/1627028401 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65ac039070 con 0x7f65b40b2080 2024-03-04T09:30:14.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:14.347+0000 7f65b2000640 1 -- 172.21.15.6:0/1627028401 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65ac02cc50 con 0x7f65b40b2080 2024-03-04T09:30:14.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:14.347+0000 7f65b2000640 1 -- 172.21.15.6:0/1627028401 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65ac034440 con 0x7f65b40b2080 2024-03-04T09:30:14.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:14.348+0000 7f65b3e00640 1 -- 172.21.15.6:0/1627028401 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f65b40b2080 msgr2=0x7f65b40b2fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:14.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:14.348+0000 7f65b3e00640 1 --2- 172.21.15.6:0/1627028401 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f65b40b2080 0x7f65b40b2fd0 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f65ac009d90 tx=0x7f65ac02bea0 comp rx=0 tx=0).stop 2024-03-04T09:30:14.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:14.348+0000 7f65b3e00640 1 -- 172.21.15.6:0/1627028401 shutdown_connections 2024-03-04T09:30:14.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:14.348+0000 7f65b3e00640 1 --2- 172.21.15.6:0/1627028401 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f65b40b2080 0x7f65b40b2fd0 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:14.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:14.348+0000 7f65b3e00640 1 --2- 172.21.15.6:0/1627028401 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f65b415f810 0x7f65b40b1b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:14.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:14.348+0000 7f65b3e00640 1 -- 172.21.15.6:0/1627028401 >> 172.21.15.6:0/1627028401 conn(0x7f65b406b3a0 msgr2=0x7f65b406b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:14.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:14.348+0000 7f65b3e00640 1 -- 172.21.15.6:0/1627028401 shutdown_connections 2024-03-04T09:30:14.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:14.348+0000 7f65b3e00640 1 -- 172.21.15.6:0/1627028401 wait complete. 2024-03-04T09:30:14.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:14.350+0000 7f65b3e00640 1 Processor -- start 2024-03-04T09:30:14.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:14.350+0000 7f65b3e00640 1 -- start start 2024-03-04T09:30:14.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:14.351+0000 7f65b3e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f65b40b2080 0x7f65b4156320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:14.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:14.351+0000 7f65b3e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f65b4145430 0x7f65b41436d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:14.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:14.351+0000 7f65b3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f65b415f810 0x7f65b413dd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:14.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:14.351+0000 7f65b3e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65b40cdf80 con 0x7f65b415f810 2024-03-04T09:30:14.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:14.351+0000 7f65b3e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65b40cde00 con 0x7f65b4145430 2024-03-04T09:30:14.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:14.351+0000 7f65b3e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65b40b3510 con 0x7f65b40b2080 2024-03-04T09:30:14.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:14.352+0000 7f65b3400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f65b415f810 0x7f65b413dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:14.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:14.352+0000 7f65b3400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f65b415f810 0x7f65b413dd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51890/0 (socket says 172.21.15.6:51890) 2024-03-04T09:30:14.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:14.352+0000 7f65b3400640 1 -- 172.21.15.6:0/137947252 learned_addr learned my addr 172.21.15.6:0/137947252 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:30:14.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:14.352+0000 7f65b2a00640 1 --2- 172.21.15.6:0/137947252 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f65b40b2080 0x7f65b4156320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:14.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:14.352+0000 7f65abe00640 1 --2- 172.21.15.6:0/137947252 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f65b4145430 0x7f65b41436d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:14.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:14.352+0000 7f65b3400640 1 -- 172.21.15.6:0/137947252 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f65b40b2080 msgr2=0x7f65b4156320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:14.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:14.352+0000 7f65b3400640 1 --2- 172.21.15.6:0/137947252 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f65b40b2080 0x7f65b4156320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:14.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:14.352+0000 7f65b3400640 1 -- 172.21.15.6:0/137947252 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f65b4145430 msgr2=0x7f65b41436d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:14.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:14.352+0000 7f65b3400640 1 --2- 172.21.15.6:0/137947252 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f65b4145430 0x7f65b41436d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:14.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:14.352+0000 7f65b3400640 1 -- 172.21.15.6:0/137947252 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65b41f1f60 con 0x7f65b415f810 2024-03-04T09:30:14.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:14.352+0000 7f65abe00640 1 --2- 172.21.15.6:0/137947252 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f65b4145430 0x7f65b41436d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:30:14.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:14.353+0000 7f65b3400640 1 --2- 172.21.15.6:0/137947252 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f65b415f810 0x7f65b413dd40 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f65ac009eb0 tx=0x7f65ac004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:14.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:14.353+0000 7f65ab400640 1 -- 172.21.15.6:0/137947252 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65ac044020 con 0x7f65b415f810 2024-03-04T09:30:14.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:14.353+0000 7f65ab400640 1 -- 172.21.15.6:0/137947252 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65ac02ca50 con 0x7f65b415f810 2024-03-04T09:30:14.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:14.353+0000 7f65ab400640 1 -- 172.21.15.6:0/137947252 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65ac039040 con 0x7f65b415f810 2024-03-04T09:30:14.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:14.353+0000 7f65b3e00640 1 -- 172.21.15.6:0/137947252 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65b41f2240 con 0x7f65b415f810 2024-03-04T09:30:14.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:14.353+0000 7f65b3e00640 1 -- 172.21.15.6:0/137947252 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65b41f2ae0 con 0x7f65b415f810 2024-03-04T09:30:14.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:14.354+0000 7f65b3e00640 1 -- 172.21.15.6:0/137947252 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65b404f330 con 0x7f65b415f810 2024-03-04T09:30:14.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:14.355+0000 7f65ab400640 1 -- 172.21.15.6:0/137947252 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f65ac047020 con 0x7f65b415f810 2024-03-04T09:30:14.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:14.356+0000 7f65ab400640 1 --2- 172.21.15.6:0/137947252 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f659006b520 0x7f659006d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:14.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:14.356+0000 7f65ab400640 1 -- 172.21.15.6:0/137947252 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f65ac0ac880 con 0x7f65b415f810 2024-03-04T09:30:14.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:14.357+0000 7f65b2a00640 1 --2- 172.21.15.6:0/137947252 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f659006b520 0x7f659006d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:14.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:14.358+0000 7f65b2a00640 1 --2- 172.21.15.6:0/137947252 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f659006b520 0x7f659006d9e0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f65a4000f60 tx=0x7f65a4007b50 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:14.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:14.360+0000 7f65ab400640 1 -- 172.21.15.6:0/137947252 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f65ac04e030 con 0x7f65b415f810 2024-03-04T09:30:14.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:14.557+0000 7f65b3e00640 1 -- 172.21.15.6:0/137947252 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f65b406e240 con 0x7f65b415f810 2024-03-04T09:30:14.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:14.565+0000 7f65ab400640 1 -- 172.21.15.6:0/137947252 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v28)=0 v28) v1 ==== 122+0+0 (secure 0 0 0) 0x7f65ac02cc20 con 0x7f65b415f810 2024-03-04T09:30:14.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:14.568+0000 7f65b3e00640 1 -- 172.21.15.6:0/137947252 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f659006b520 msgr2=0x7f659006d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:14.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:14.568+0000 7f65b3e00640 1 --2- 172.21.15.6:0/137947252 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f659006b520 0x7f659006d9e0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f65a4000f60 tx=0x7f65a4007b50 comp rx=0 tx=0).stop 2024-03-04T09:30:14.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:14.568+0000 7f65b3e00640 1 -- 172.21.15.6:0/137947252 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f65b415f810 msgr2=0x7f65b413dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:14.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:14.568+0000 7f65b3e00640 1 --2- 172.21.15.6:0/137947252 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f65b415f810 0x7f65b413dd40 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f65ac009eb0 tx=0x7f65ac004a40 comp rx=0 tx=0).stop 2024-03-04T09:30:14.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:14.568+0000 7f65b3e00640 1 -- 172.21.15.6:0/137947252 shutdown_connections 2024-03-04T09:30:14.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:14.568+0000 7f65b3e00640 1 --2- 172.21.15.6:0/137947252 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f65b415f810 0x7f65b413dd40 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:14.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:14.568+0000 7f65b3e00640 1 --2- 172.21.15.6:0/137947252 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f65b4145430 0x7f65b41436d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:14.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:14.568+0000 7f65b3e00640 1 --2- 172.21.15.6:0/137947252 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f659006b520 0x7f659006d9e0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:14.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:14.568+0000 7f65b3e00640 1 --2- 172.21.15.6:0/137947252 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f65b40b2080 0x7f65b4156320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:14.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:14.568+0000 7f65b3e00640 1 -- 172.21.15.6:0/137947252 >> 172.21.15.6:0/137947252 conn(0x7f65b406b3a0 msgr2=0x7f65b40b0290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:14.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:14.568+0000 7f65b3e00640 1 -- 172.21.15.6:0/137947252 shutdown_connections 2024-03-04T09:30:14.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:14.568+0000 7f65b3e00640 1 -- 172.21.15.6:0/137947252 wait complete. 2024-03-04T09:30:14.585 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-04T09:30:14.585 DEBUG:teuthology.orchestra.run.smithi006:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-04T09:30:14.705 DEBUG:teuthology.orchestra.run.smithi146:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-04T09:30:14.838 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-04T09:30:14.838 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-04T09:30:14.838 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T09:30:14.901 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054202,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249800k,nr_inodes=812450,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-04T09:30:14.901 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-04T09:30:14.901 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T09:30:14.958 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053429,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249180k,nr_inodes=812295,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-04T09:30:14.958 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-04T09:30:14.958 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T09:30:15.019 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054202,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249800k,nr_inodes=812450,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-04T09:30:15.020 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-04T09:30:15.020 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T09:30:15.078 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053429,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249180k,nr_inodes=812295,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-04T09:30:15.078 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-04T09:30:15.078 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:30:15.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.309+0000 7f20d0a00640 1 Processor -- start 2024-03-04T09:30:15.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.309+0000 7f20d0a00640 1 -- start start 2024-03-04T09:30:15.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.310+0000 7f20d0a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f20cc15e0b0 0x7f20cc0cf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:15.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.310+0000 7f20d0a00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f20cc0d4a10 0x7f20cc0d0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:15.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.311+0000 7f20d0a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20cc0b3710 con 0x7f20cc0d0c80 2024-03-04T09:30:15.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.311+0000 7f20d0a00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20cc0d10f0 con 0x7f20cc15e0b0 2024-03-04T09:30:15.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.311+0000 7f20d0a00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20cc0b09e0 con 0x7f20cc0d4a10 2024-03-04T09:30:15.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.311+0000 7f20cb200640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f20cc0d0c80 0x7f20cc0d1900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:34848/0 (socket says 172.21.15.6:34848) 2024-03-04T09:30:15.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.311+0000 7f20cb200640 1 -- 172.21.15.6:0/2710667769 learned_addr learned my addr 172.21.15.6:0/2710667769 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:30:15.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.311+0000 7f20ca800640 1 --2- 172.21.15.6:0/2710667769 >> v2:172.21.15.6:3300/0 conn(0x7f20cc15e0b0 0x7f20cc0cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:15.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.311+0000 7f20c9e00640 1 --2- 172.21.15.6:0/2710667769 >> v2:172.21.15.146:3300/0 conn(0x7f20cc0d4a10 0x7f20cc0d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:15.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.311+0000 7f20c9400640 1 -- 172.21.15.6:0/2710667769 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1574099536 0 0) 0x7f20cc0b3710 con 0x7f20cc0d0c80 2024-03-04T09:30:15.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.312+0000 7f20c9400640 1 -- 172.21.15.6:0/2710667769 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20b0003520 con 0x7f20cc0d0c80 2024-03-04T09:30:15.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.312+0000 7f20ca800640 1 -- 172.21.15.6:0/2710667769 >> v2:172.21.15.146:3300/0 conn(0x7f20cc0d4a10 msgr2=0x7f20cc0d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:15.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.312+0000 7f20ca800640 1 --2- 172.21.15.6:0/2710667769 >> v2:172.21.15.146:3300/0 conn(0x7f20cc0d4a10 0x7f20cc0d0740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:15.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.312+0000 7f20ca800640 1 -- 172.21.15.6:0/2710667769 >> v1:172.21.15.6:6789/0 conn(0x7f20cc0d0c80 legacy=0x7f20cc0d1900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:15.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.312+0000 7f20ca800640 1 -- 172.21.15.6:0/2710667769 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20cc0b39f0 con 0x7f20cc15e0b0 2024-03-04T09:30:15.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.313+0000 7f20ca800640 1 --2- 172.21.15.6:0/2710667769 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f20cc15e0b0 0x7f20cc0cf310 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f20cc06b8c0 tx=0x7f20bc02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f276492d1da6f8d9 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:15.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.313+0000 7f20c9400640 1 -- 172.21.15.6:0/2710667769 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20bc039070 con 0x7f20cc15e0b0 2024-03-04T09:30:15.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.313+0000 7f20c9400640 1 -- 172.21.15.6:0/2710667769 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20bc02cc60 con 0x7f20cc15e0b0 2024-03-04T09:30:15.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.313+0000 7f20c9400640 1 -- 172.21.15.6:0/2710667769 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20bc034660 con 0x7f20cc15e0b0 2024-03-04T09:30:15.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.314+0000 7f20d0a00640 1 -- 172.21.15.6:0/2710667769 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f20cc15e0b0 msgr2=0x7f20cc0cf310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:15.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.314+0000 7f20d0a00640 1 --2- 172.21.15.6:0/2710667769 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f20cc15e0b0 0x7f20cc0cf310 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f20cc06b8c0 tx=0x7f20bc02beb0 comp rx=0 tx=0).stop 2024-03-04T09:30:15.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.314+0000 7f20d0a00640 1 -- 172.21.15.6:0/2710667769 shutdown_connections 2024-03-04T09:30:15.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.314+0000 7f20d0a00640 1 --2- 172.21.15.6:0/2710667769 >> v2:172.21.15.146:3300/0 conn(0x7f20cc0d4a10 0x7f20cc0d0740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:15.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.314+0000 7f20d0a00640 1 --2- 172.21.15.6:0/2710667769 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f20cc15e0b0 0x7f20cc0cf310 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:15.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.314+0000 7f20d0a00640 1 -- 172.21.15.6:0/2710667769 >> 172.21.15.6:0/2710667769 conn(0x7f20cc0d3d30 msgr2=0x7f20cc0d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:15.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.314+0000 7f20d0a00640 1 -- 172.21.15.6:0/2710667769 shutdown_connections 2024-03-04T09:30:15.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.314+0000 7f20d0a00640 1 -- 172.21.15.6:0/2710667769 wait complete. 2024-03-04T09:30:15.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.316+0000 7f20d0a00640 1 Processor -- start 2024-03-04T09:30:15.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.316+0000 7f20d0a00640 1 -- start start 2024-03-04T09:30:15.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.317+0000 7f20d0a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f20cc0d0c80 0x7f20cc159f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:15.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.317+0000 7f20d0a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f20cc0d4a10 0x7f20cc0cb680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:15.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.317+0000 7f20d0a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f20cc15e0b0 0x7f20cc0cd1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:15.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.317+0000 7f20d0a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20cc15e490 con 0x7f20cc0d0c80 2024-03-04T09:30:15.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.317+0000 7f20d0a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20cc0b3fb0 con 0x7f20cc15e0b0 2024-03-04T09:30:15.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.317+0000 7f20d0a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20cc0cfc60 con 0x7f20cc0d4a10 2024-03-04T09:30:15.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.317+0000 7f20ca800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f20cc0d0c80 0x7f20cc159f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:15.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.317+0000 7f20ca800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f20cc0d0c80 0x7f20cc159f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51906/0 (socket says 172.21.15.6:51906) 2024-03-04T09:30:15.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.317+0000 7f20ca800640 1 -- 172.21.15.6:0/3108968467 learned_addr learned my addr 172.21.15.6:0/3108968467 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:30:15.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.317+0000 7f20ca800640 1 -- 172.21.15.6:0/3108968467 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f20cc0d4a10 msgr2=0x7f20cc0cb680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:15.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.317+0000 7f20ca800640 1 --2- 172.21.15.6:0/3108968467 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f20cc0d4a10 0x7f20cc0cb680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:15.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.317+0000 7f20ca800640 1 -- 172.21.15.6:0/3108968467 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f20cc15e0b0 msgr2=0x7f20cc0cd1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:15.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.317+0000 7f20ca800640 1 --2- 172.21.15.6:0/3108968467 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f20cc15e0b0 0x7f20cc0cd1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:15.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.317+0000 7f20ca800640 1 -- 172.21.15.6:0/3108968467 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20cc0b76a0 con 0x7f20cc0d0c80 2024-03-04T09:30:15.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.318+0000 7f20ca800640 1 --2- 172.21.15.6:0/3108968467 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f20cc0d0c80 0x7f20cc159f70 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7f20bc009d90 tx=0x7f20bc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:15.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.318+0000 7f20c2a00640 1 -- 172.21.15.6:0/3108968467 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20bc039050 con 0x7f20cc0d0c80 2024-03-04T09:30:15.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.318+0000 7f20c2a00640 1 -- 172.21.15.6:0/3108968467 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20bc045070 con 0x7f20cc0d0c80 2024-03-04T09:30:15.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.318+0000 7f20c2a00640 1 -- 172.21.15.6:0/3108968467 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20bc009450 con 0x7f20cc0d0c80 2024-03-04T09:30:15.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.318+0000 7f20d0a00640 1 -- 172.21.15.6:0/3108968467 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20cc0a1470 con 0x7f20cc0d0c80 2024-03-04T09:30:15.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.318+0000 7f20d0a00640 1 -- 172.21.15.6:0/3108968467 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20cc0b7910 con 0x7f20cc0d0c80 2024-03-04T09:30:15.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.319+0000 7f20d0a00640 1 -- 172.21.15.6:0/3108968467 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20cc04f330 con 0x7f20cc0d0c80 2024-03-04T09:30:15.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.320+0000 7f20c2a00640 1 -- 172.21.15.6:0/3108968467 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f20bc03e850 con 0x7f20cc0d0c80 2024-03-04T09:30:15.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.321+0000 7f20c2a00640 1 --2- 172.21.15.6:0/3108968467 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f20ac06b4d0 0x7f20ac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:15.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.321+0000 7f20c2a00640 1 -- 172.21.15.6:0/3108968467 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f20bc0ace00 con 0x7f20cc0d0c80 2024-03-04T09:30:15.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.322+0000 7f20c9e00640 1 --2- 172.21.15.6:0/3108968467 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f20ac06b4d0 0x7f20ac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:15.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.323+0000 7f20c9e00640 1 --2- 172.21.15.6:0/3108968467 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f20ac06b4d0 0x7f20ac06d990 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f20b8006a90 tx=0x7f20b8006a20 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:15.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.325+0000 7f20c2a00640 1 -- 172.21.15.6:0/3108968467 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f20bc07c150 con 0x7f20cc0d0c80 2024-03-04T09:30:15.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.564+0000 7f20d0a00640 1 -- 172.21.15.6:0/3108968467 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f20cc06c9b0 con 0x7f20cc0d0c80 2024-03-04T09:30:15.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.565+0000 7f20c2a00640 1 -- 172.21.15.6:0/3108968467 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) v1 ==== 94+0+3696 (secure 0 0 0) 0x7f20bc07ba50 con 0x7f20cc0d0c80 2024-03-04T09:30:15.570 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:30:15.570 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":53,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5827,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6836/3447344578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":3447344578},{"type":"v1","addr":"172.21.15.146:6836","nonce":3447344578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":46},{"gid":5830,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/168994142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":168994142},{"type":"v1","addr":"172.21.15.146:6837","nonce":168994142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":46},{"gid":6735,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/3369489999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":3369489999},{"type":"v1","addr":"172.21.15.6:6833","nonce":3369489999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":45},{"gid":7020,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/2306206705","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":2306206705},{"type":"v1","addr":"172.21.15.6:6835","nonce":2306206705}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53}],"filesystems":[]} 2024-03-04T09:30:15.571 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 53 2024-03-04T09:30:15.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.568+0000 7f20d0a00640 1 -- 172.21.15.6:0/3108968467 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f20ac06b4d0 msgr2=0x7f20ac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:15.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.568+0000 7f20d0a00640 1 --2- 172.21.15.6:0/3108968467 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f20ac06b4d0 0x7f20ac06d990 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f20b8006a90 tx=0x7f20b8006a20 comp rx=0 tx=0).stop 2024-03-04T09:30:15.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.568+0000 7f20d0a00640 1 -- 172.21.15.6:0/3108968467 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f20cc0d0c80 msgr2=0x7f20cc159f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:15.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.568+0000 7f20d0a00640 1 --2- 172.21.15.6:0/3108968467 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f20cc0d0c80 0x7f20cc159f70 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7f20bc009d90 tx=0x7f20bc004a40 comp rx=0 tx=0).stop 2024-03-04T09:30:15.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.568+0000 7f20d0a00640 1 -- 172.21.15.6:0/3108968467 shutdown_connections 2024-03-04T09:30:15.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.568+0000 7f20d0a00640 1 --2- 172.21.15.6:0/3108968467 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f20cc15e0b0 0x7f20cc0cd1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:15.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.568+0000 7f20d0a00640 1 --2- 172.21.15.6:0/3108968467 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f20ac06b4d0 0x7f20ac06d990 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:15.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.568+0000 7f20d0a00640 1 --2- 172.21.15.6:0/3108968467 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f20cc0d4a10 0x7f20cc0cb680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:15.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.568+0000 7f20d0a00640 1 --2- 172.21.15.6:0/3108968467 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f20cc0d0c80 0x7f20cc159f70 unknown :-1 s=CLOSED pgs=1037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:15.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.568+0000 7f20d0a00640 1 -- 172.21.15.6:0/3108968467 >> 172.21.15.6:0/3108968467 conn(0x7f20cc0d3d30 msgr2=0x7f20cc073f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:15.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.568+0000 7f20d0a00640 1 -- 172.21.15.6:0/3108968467 shutdown_connections 2024-03-04T09:30:15.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.568+0000 7f20d0a00640 1 -- 172.21.15.6:0/3108968467 wait complete. 2024-03-04T09:30:15.583 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-04T09:30:15.583 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-04T09:30:15.583 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-04T09:30:15.583 INFO:teuthology.orchestra.run:waiting for 300 2024-03-04T09:30:15.584 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-04T09:30:15.584 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-04T09:30:15.584 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-04T09:30:15.585 INFO:teuthology.orchestra.run:waiting for 300 2024-03-04T09:30:15.585 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-04T09:30:15.585 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-04T09:30:15.585 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-04T09:30:15.585 INFO:teuthology.orchestra.run:waiting for 300 2024-03-04T09:30:15.586 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-04T09:30:15.586 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-04T09:30:15.586 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-04T09:30:15.587 INFO:teuthology.orchestra.run:waiting for 300 2024-03-04T09:30:15.600 INFO:tasks.ceph.mds.a:Stopped 2024-03-04T09:30:15.601 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-04T09:30:15.604 INFO:tasks.ceph.mds.c:Stopped 2024-03-04T09:30:15.604 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-04T09:30:15.608 INFO:tasks.ceph.mds.b:Stopped 2024-03-04T09:30:15.608 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-04T09:30:15.612 INFO:tasks.ceph.mds.d:Stopped 2024-03-04T09:30:15.613 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-04T09:30:15.651 INFO:tasks.ceph.mds.a:Started 2024-03-04T09:30:15.651 DEBUG:teuthology.parallel:result is None 2024-03-04T09:30:15.658 INFO:tasks.ceph.mds.c:Started 2024-03-04T09:30:15.658 DEBUG:teuthology.parallel:result is None 2024-03-04T09:30:15.661 INFO:tasks.ceph.mds.b:Started 2024-03-04T09:30:15.661 DEBUG:teuthology.parallel:result is None 2024-03-04T09:30:15.669 INFO:tasks.ceph.mds.d:Started 2024-03-04T09:30:15.669 DEBUG:teuthology.parallel:result is None 2024-03-04T09:30:15.670 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-04T09:30:15.805 INFO:tasks.ceph.mds.a.smithi006.stdout:starting mds.a at 2024-03-04T09:30:15.806 INFO:tasks.ceph.mds.c.smithi006.stdout:starting mds.c at 2024-03-04T09:30:15.824 INFO:tasks.ceph.mds.b.smithi146.stdout:starting mds.b at 2024-03-04T09:30:15.832 INFO:tasks.ceph.mds.d.smithi146.stdout:starting mds.d at 2024-03-04T09:30:15.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.848+0000 7f4b0fe00640 1 Processor -- start 2024-03-04T09:30:15.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.848+0000 7f4b0fe00640 1 -- start start 2024-03-04T09:30:15.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.850+0000 7f4b0fe00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4b08161cc0 0x7f4b08162130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:15.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.850+0000 7f4b0fe00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b0806f0a0 con 0x7f4b0815f360 2024-03-04T09:30:15.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.850+0000 7f4b0fe00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b0806f380 con 0x7f4b0815df70 2024-03-04T09:30:15.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.850+0000 7f4b0fe00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b08162670 con 0x7f4b08161cc0 2024-03-04T09:30:15.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.850+0000 7f4b0ca00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f4b0815f360 0x7f4b080b6230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:34860/0 (socket says 172.21.15.6:34860) 2024-03-04T09:30:15.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.850+0000 7f4b0ca00640 1 -- 172.21.15.6:0/3055333350 learned_addr learned my addr 172.21.15.6:0/3055333350 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:30:15.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.850+0000 7f4b0de00640 1 --2- 172.21.15.6:0/3055333350 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4b08161cc0 0x7f4b08162130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:15.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.850+0000 7f4b07e00640 1 -- 172.21.15.6:0/3055333350 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 906937753 0 0) 0x7f4b0806f0a0 con 0x7f4b0815f360 2024-03-04T09:30:15.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.850+0000 7f4b07e00640 1 -- 172.21.15.6:0/3055333350 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4af0003520 con 0x7f4b0815f360 2024-03-04T09:30:15.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.850+0000 7f4b07e00640 1 -- 172.21.15.6:0/3055333350 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 924764981 0 0) 0x7f4b0806f380 con 0x7f4b0815df70 2024-03-04T09:30:15.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.850+0000 7f4b07e00640 1 -- 172.21.15.6:0/3055333350 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b0806f0a0 con 0x7f4b0815df70 2024-03-04T09:30:15.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.850+0000 7f4b0de00640 1 -- 172.21.15.6:0/3055333350 >> v1:172.21.15.146:6789/0 conn(0x7f4b0815df70 legacy=0x7f4b08070140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:15.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.851+0000 7f4b0de00640 1 -- 172.21.15.6:0/3055333350 >> v1:172.21.15.6:6789/0 conn(0x7f4b0815f360 legacy=0x7f4b080b6230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:15.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.851+0000 7f4b0de00640 1 -- 172.21.15.6:0/3055333350 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b0806f660 con 0x7f4b08161cc0 2024-03-04T09:30:15.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.852+0000 7f4b0de00640 1 --2- 172.21.15.6:0/3055333350 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4b08161cc0 0x7f4b08162130 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f4b00009dc0 tx=0x7f4b0002bf10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=402ff790198d8810 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:15.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.852+0000 7f4b07e00640 1 -- 172.21.15.6:0/3055333350 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b00039070 con 0x7f4b08161cc0 2024-03-04T09:30:15.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.852+0000 7f4b07e00640 1 -- 172.21.15.6:0/3055333350 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b0002ccc0 con 0x7f4b08161cc0 2024-03-04T09:30:15.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.852+0000 7f4b07e00640 1 -- 172.21.15.6:0/3055333350 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b00034640 con 0x7f4b08161cc0 2024-03-04T09:30:15.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.852+0000 7f4b0fe00640 1 -- 172.21.15.6:0/3055333350 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4b08161cc0 msgr2=0x7f4b08162130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:15.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.852+0000 7f4b0fe00640 1 --2- 172.21.15.6:0/3055333350 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4b08161cc0 0x7f4b08162130 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f4b00009dc0 tx=0x7f4b0002bf10 comp rx=0 tx=0).stop 2024-03-04T09:30:15.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.852+0000 7f4b0fe00640 1 -- 172.21.15.6:0/3055333350 shutdown_connections 2024-03-04T09:30:15.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.853+0000 7f4b0fe00640 1 --2- 172.21.15.6:0/3055333350 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4b08161cc0 0x7f4b08162130 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:15.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.853+0000 7f4b0fe00640 1 -- 172.21.15.6:0/3055333350 >> 172.21.15.6:0/3055333350 conn(0x7f4b0809aa10 msgr2=0x7f4b0809ae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:15.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.853+0000 7f4b0fe00640 1 -- 172.21.15.6:0/3055333350 shutdown_connections 2024-03-04T09:30:15.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.853+0000 7f4b0fe00640 1 -- 172.21.15.6:0/3055333350 wait complete. 2024-03-04T09:30:15.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.854+0000 7f4b0fe00640 1 Processor -- start 2024-03-04T09:30:15.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.854+0000 7f4b0fe00640 1 -- start start 2024-03-04T09:30:15.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.855+0000 7f4b0fe00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4b0815df70 0x7f4b0814e810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:15.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.855+0000 7f4b0fe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4b08161cc0 0x7f4b081505e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:15.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.855+0000 7f4b0fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4b08151080 0x7f4b080afe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:15.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.855+0000 7f4b0fe00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b080b6940 con 0x7f4b08151080 2024-03-04T09:30:15.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.855+0000 7f4b0fe00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b0806fcb0 con 0x7f4b08161cc0 2024-03-04T09:30:15.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.855+0000 7f4b0fe00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b080b6ac0 con 0x7f4b0815df70 2024-03-04T09:30:15.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.855+0000 7f4b0de00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4b08151080 0x7f4b080afe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:15.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.856+0000 7f4b0de00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4b08151080 0x7f4b080afe30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51948/0 (socket says 172.21.15.6:51948) 2024-03-04T09:30:15.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.856+0000 7f4b0de00640 1 -- 172.21.15.6:0/1683470426 learned_addr learned my addr 172.21.15.6:0/1683470426 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:30:15.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.856+0000 7f4b0d400640 1 --2- 172.21.15.6:0/1683470426 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4b0815df70 0x7f4b0814e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:15.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.856+0000 7f4b0ca00640 1 --2- 172.21.15.6:0/1683470426 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4b08161cc0 0x7f4b081505e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:15.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.856+0000 7f4b0de00640 1 -- 172.21.15.6:0/1683470426 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4b0815df70 msgr2=0x7f4b0814e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:15.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.856+0000 7f4b0de00640 1 --2- 172.21.15.6:0/1683470426 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4b0815df70 0x7f4b0814e810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:15.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.856+0000 7f4b0de00640 1 -- 172.21.15.6:0/1683470426 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4b08161cc0 msgr2=0x7f4b081505e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:15.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.856+0000 7f4b0de00640 1 --2- 172.21.15.6:0/1683470426 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4b08161cc0 0x7f4b081505e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:15.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.856+0000 7f4b0de00640 1 -- 172.21.15.6:0/1683470426 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b08151560 con 0x7f4b08151080 2024-03-04T09:30:15.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.856+0000 7f4b0de00640 1 --2- 172.21.15.6:0/1683470426 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4b08151080 0x7f4b080afe30 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f4b0002c520 tx=0x7f4b00004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:15.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.857+0000 7f4b05600640 1 -- 172.21.15.6:0/1683470426 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b00039050 con 0x7f4b08151080 2024-03-04T09:30:15.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.857+0000 7f4b05600640 1 -- 172.21.15.6:0/1683470426 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b00045070 con 0x7f4b08151080 2024-03-04T09:30:15.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.857+0000 7f4b0fe00640 1 -- 172.21.15.6:0/1683470426 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b08151840 con 0x7f4b08151080 2024-03-04T09:30:15.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.857+0000 7f4b05600640 1 -- 172.21.15.6:0/1683470426 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b00009450 con 0x7f4b08151080 2024-03-04T09:30:15.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.857+0000 7f4b0fe00640 1 -- 172.21.15.6:0/1683470426 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b08151e00 con 0x7f4b08151080 2024-03-04T09:30:15.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.858+0000 7f4b0fe00640 1 -- 172.21.15.6:0/1683470426 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b0804f3a0 con 0x7f4b08151080 2024-03-04T09:30:15.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.859+0000 7f4b05600640 1 -- 172.21.15.6:0/1683470426 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4b0003e850 con 0x7f4b08151080 2024-03-04T09:30:15.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.859+0000 7f4b05600640 1 --2- 172.21.15.6:0/1683470426 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f4ae806b4d0 0x7f4ae806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:15.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.860+0000 7f4b05600640 1 -- 172.21.15.6:0/1683470426 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f4b000acff0 con 0x7f4b08151080 2024-03-04T09:30:15.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.860+0000 7f4b0d400640 1 --2- 172.21.15.6:0/1683470426 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f4ae806b4d0 0x7f4ae806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:15.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.860+0000 7f4b0d400640 1 --2- 172.21.15.6:0/1683470426 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f4ae806b4d0 0x7f4ae806d990 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f4afc0011e0 tx=0x7f4afc009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:15.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:15.864+0000 7f4b05600640 1 -- 172.21.15.6:0/1683470426 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4b0007bca0 con 0x7f4b08151080 2024-03-04T09:30:16.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.070+0000 7f4b0fe00640 1 -- 172.21.15.6:0/1683470426 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f4b0809b670 con 0x7f4b08151080 2024-03-04T09:30:16.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.124+0000 7f4b05600640 1 -- 172.21.15.6:0/1683470426 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v88) v1 ==== 97+0+0 (secure 0 0 0) 0x7f4b0007fd40 con 0x7f4b08151080 2024-03-04T09:30:16.122 INFO:teuthology.orchestra.run.smithi006.stderr: removed all blocklist entries 2024-03-04T09:30:16.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.127+0000 7f4b0fe00640 1 -- 172.21.15.6:0/1683470426 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f4ae806b4d0 msgr2=0x7f4ae806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:16.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.127+0000 7f4b0fe00640 1 --2- 172.21.15.6:0/1683470426 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f4ae806b4d0 0x7f4ae806d990 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f4afc0011e0 tx=0x7f4afc009040 comp rx=0 tx=0).stop 2024-03-04T09:30:16.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.127+0000 7f4b0fe00640 1 -- 172.21.15.6:0/1683470426 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4b08151080 msgr2=0x7f4b080afe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:16.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.127+0000 7f4b0fe00640 1 --2- 172.21.15.6:0/1683470426 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4b08151080 0x7f4b080afe30 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f4b0002c520 tx=0x7f4b00004a40 comp rx=0 tx=0).stop 2024-03-04T09:30:16.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.128+0000 7f4b0fe00640 1 -- 172.21.15.6:0/1683470426 shutdown_connections 2024-03-04T09:30:16.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.128+0000 7f4b0fe00640 1 --2- 172.21.15.6:0/1683470426 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4b08151080 0x7f4b080afe30 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:16.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.128+0000 7f4b0fe00640 1 --2- 172.21.15.6:0/1683470426 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4b08161cc0 0x7f4b081505e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:16.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.128+0000 7f4b0fe00640 1 --2- 172.21.15.6:0/1683470426 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f4ae806b4d0 0x7f4ae806d990 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:16.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.128+0000 7f4b0fe00640 1 --2- 172.21.15.6:0/1683470426 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4b0815df70 0x7f4b0814e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:16.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.128+0000 7f4b0fe00640 1 -- 172.21.15.6:0/1683470426 >> 172.21.15.6:0/1683470426 conn(0x7f4b0809aa10 msgr2=0x7f4b0806cda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:16.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.128+0000 7f4b0fe00640 1 -- 172.21.15.6:0/1683470426 shutdown_connections 2024-03-04T09:30:16.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.128+0000 7f4b0fe00640 1 -- 172.21.15.6:0/1683470426 wait complete. 2024-03-04T09:30:16.147 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-04T09:30:16.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.353+0000 7fc62ae00640 1 Processor -- start 2024-03-04T09:30:16.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.353+0000 7fc62ae00640 1 -- start start 2024-03-04T09:30:16.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.354+0000 7fc62ae00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc624161670 0x7fc624070c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:16.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.355+0000 7fc62ae00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc62406de00 con 0x7fc6240ae0a0 2024-03-04T09:30:16.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.355+0000 7fc62ae00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6240ce9a0 con 0x7fc62406e240 2024-03-04T09:30:16.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.355+0000 7fc62ae00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6240730d0 con 0x7fc624161670 2024-03-04T09:30:16.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.355+0000 7fc623e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc624161670 0x7fc624070c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:16.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.355+0000 7fc623e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc624161670 0x7fc624070c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51954/0 (socket says 172.21.15.6:51954) 2024-03-04T09:30:16.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.355+0000 7fc628e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fc6240ae0a0 0x7fc62406d6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:34870/0 (socket says 172.21.15.6:34870) 2024-03-04T09:30:16.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.355+0000 7fc623e00640 1 -- 172.21.15.6:0/2096358287 learned_addr learned my addr 172.21.15.6:0/2096358287 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:30:16.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.355+0000 7fc623e00640 1 -- 172.21.15.6:0/2096358287 >> v1:172.21.15.146:6789/0 conn(0x7fc62406e240 legacy=0x7fc62406cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:16.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.355+0000 7fc623e00640 1 -- 172.21.15.6:0/2096358287 >> v1:172.21.15.6:6789/0 conn(0x7fc6240ae0a0 legacy=0x7fc62406d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:16.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.355+0000 7fc623e00640 1 -- 172.21.15.6:0/2096358287 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6240cec80 con 0x7fc624161670 2024-03-04T09:30:16.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.355+0000 7fc623400640 1 -- 172.21.15.6:0/2096358287 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3014266940 0 0) 0x7fc62406de00 con 0x7fc6240ae0a0 2024-03-04T09:30:16.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.357+0000 7fc623e00640 1 --2- 172.21.15.6:0/2096358287 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc624161670 0x7fc624070c70 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7fc614009870 tx=0x7fc61402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74b479bf1679ae2a server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:16.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.357+0000 7fc623400640 1 -- 172.21.15.6:0/2096358287 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc614035020 con 0x7fc624161670 2024-03-04T09:30:16.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.357+0000 7fc623400640 1 -- 172.21.15.6:0/2096358287 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc614031870 con 0x7fc624161670 2024-03-04T09:30:16.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.357+0000 7fc623400640 1 -- 172.21.15.6:0/2096358287 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc614031c40 con 0x7fc624161670 2024-03-04T09:30:16.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.357+0000 7fc62ae00640 1 -- 172.21.15.6:0/2096358287 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc624161670 msgr2=0x7fc624070c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:16.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.357+0000 7fc62ae00640 1 --2- 172.21.15.6:0/2096358287 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc624161670 0x7fc624070c70 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7fc614009870 tx=0x7fc61402b910 comp rx=0 tx=0).stop 2024-03-04T09:30:16.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.358+0000 7fc62ae00640 1 -- 172.21.15.6:0/2096358287 shutdown_connections 2024-03-04T09:30:16.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.358+0000 7fc62ae00640 1 --2- 172.21.15.6:0/2096358287 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc624161670 0x7fc624070c70 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:16.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.358+0000 7fc62ae00640 1 -- 172.21.15.6:0/2096358287 >> 172.21.15.6:0/2096358287 conn(0x7fc62406afb0 msgr2=0x7fc62406b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:16.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.358+0000 7fc62ae00640 1 -- 172.21.15.6:0/2096358287 shutdown_connections 2024-03-04T09:30:16.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.358+0000 7fc62ae00640 1 -- 172.21.15.6:0/2096358287 wait complete. 2024-03-04T09:30:16.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.360+0000 7fc62ae00640 1 Processor -- start 2024-03-04T09:30:16.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.360+0000 7fc62ae00640 1 -- start start 2024-03-04T09:30:16.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.362+0000 7fc62ae00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc62406e240 0x7fc624155a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:16.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.362+0000 7fc62ae00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc6240ae0a0 0x7fc6241f5290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:16.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.362+0000 7fc62ae00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc624161670 0x7fc6241f86a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:16.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.362+0000 7fc62ae00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6240cf520 con 0x7fc62406e240 2024-03-04T09:30:16.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.362+0000 7fc62ae00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6240cf240 con 0x7fc6240ae0a0 2024-03-04T09:30:16.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.362+0000 7fc62ae00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6240ce600 con 0x7fc624161670 2024-03-04T09:30:16.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.362+0000 7fc623e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc62406e240 0x7fc624155a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:16.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.362+0000 7fc623e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc62406e240 0x7fc624155a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51968/0 (socket says 172.21.15.6:51968) 2024-03-04T09:30:16.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.362+0000 7fc623e00640 1 -- 172.21.15.6:0/3474729419 learned_addr learned my addr 172.21.15.6:0/3474729419 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:30:16.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.362+0000 7fc623e00640 1 -- 172.21.15.6:0/3474729419 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc624161670 msgr2=0x7fc6241f86a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:16.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.362+0000 7fc623e00640 1 --2- 172.21.15.6:0/3474729419 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc624161670 0x7fc6241f86a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:16.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.362+0000 7fc623e00640 1 -- 172.21.15.6:0/3474729419 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc6240ae0a0 msgr2=0x7fc6241f5290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:16.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.362+0000 7fc623e00640 1 --2- 172.21.15.6:0/3474729419 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc6240ae0a0 0x7fc6241f5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:16.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.362+0000 7fc623e00640 1 -- 172.21.15.6:0/3474729419 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6241f8be0 con 0x7fc62406e240 2024-03-04T09:30:16.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.363+0000 7fc623e00640 1 --2- 172.21.15.6:0/3474729419 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc62406e240 0x7fc624155a70 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7fc614009870 tx=0x7fc6140046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:16.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.363+0000 7fc620c00640 1 -- 172.21.15.6:0/3474729419 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc61403ab50 con 0x7fc62406e240 2024-03-04T09:30:16.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.363+0000 7fc620c00640 1 -- 172.21.15.6:0/3474729419 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc61403acf0 con 0x7fc62406e240 2024-03-04T09:30:16.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.363+0000 7fc620c00640 1 -- 172.21.15.6:0/3474729419 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc614004270 con 0x7fc62406e240 2024-03-04T09:30:16.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.363+0000 7fc62ae00640 1 -- 172.21.15.6:0/3474729419 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6240cb060 con 0x7fc62406e240 2024-03-04T09:30:16.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.363+0000 7fc62ae00640 1 -- 172.21.15.6:0/3474729419 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6241f47b0 con 0x7fc62406e240 2024-03-04T09:30:16.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.364+0000 7fc62ae00640 1 -- 172.21.15.6:0/3474729419 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc624069510 con 0x7fc62406e240 2024-03-04T09:30:16.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.365+0000 7fc620c00640 1 -- 172.21.15.6:0/3474729419 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc61404b060 con 0x7fc62406e240 2024-03-04T09:30:16.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.366+0000 7fc620c00640 1 --2- 172.21.15.6:0/3474729419 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc60006b500 0x7fc60006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:16.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.366+0000 7fc620c00640 1 -- 172.21.15.6:0/3474729419 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fc6140abe50 con 0x7fc62406e240 2024-03-04T09:30:16.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.367+0000 7fc61be00640 1 --2- 172.21.15.6:0/3474729419 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc60006b500 0x7fc60006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:16.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.368+0000 7fc61be00640 1 --2- 172.21.15.6:0/3474729419 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc60006b500 0x7fc60006d9c0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fc6240b0f50 tx=0x7fc60c007990 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:16.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.371+0000 7fc620c00640 1 -- 172.21.15.6:0/3474729419 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc6241f8be0 con 0x7fc62406e240 2024-03-04T09:30:16.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.640+0000 7fc62ae00640 1 -- 172.21.15.6:0/3474729419 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fc62404f3a0 con 0x7fc62406e240 2024-03-04T09:30:16.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.642+0000 7fc620c00640 1 -- 172.21.15.6:0/3474729419 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v14) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fc61403f070 con 0x7fc62406e240 2024-03-04T09:30:16.640 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:30:16.640 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-04T09:30:16.640 INFO:teuthology.orchestra.run.smithi006.stdout: "auth_dump": [ 2024-03-04T09:30:16.640 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:30:16.640 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "mds.a", 2024-03-04T09:30:16.640 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQA8k+VlHCzFIBAAnKYlmCMDw45LqnSjxnZY5Q==", 2024-03-04T09:30:16.640 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:30:16.641 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-04T09:30:16.641 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow *", 2024-03-04T09:30:16.641 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow *", 2024-03-04T09:30:16.641 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-04T09:30:16.641 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:30:16.641 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:30:16.641 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:30:16.641 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "mds.b", 2024-03-04T09:30:16.641 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQA9k+VlNY7fCBAAX3z8X72/6LkQD6mwGYT69g==", 2024-03-04T09:30:16.641 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:30:16.641 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-04T09:30:16.641 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow *", 2024-03-04T09:30:16.641 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow *", 2024-03-04T09:30:16.641 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-04T09:30:16.642 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:30:16.642 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:30:16.642 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:30:16.642 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "mds.c", 2024-03-04T09:30:16.642 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQA8k+Vl0pjOMhAA7WcTw8awv4EAsrpP4yncNA==", 2024-03-04T09:30:16.642 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:30:16.642 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-04T09:30:16.642 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow *", 2024-03-04T09:30:16.642 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow *", 2024-03-04T09:30:16.642 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-04T09:30:16.642 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:30:16.642 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:30:16.642 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:30:16.643 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "mds.d", 2024-03-04T09:30:16.643 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQA9k+VlbladGRAABNgUcaqTVYuJfCqy5c/G0g==", 2024-03-04T09:30:16.643 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:30:16.643 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-04T09:30:16.643 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow *", 2024-03-04T09:30:16.643 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow *", 2024-03-04T09:30:16.643 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-04T09:30:16.643 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:30:16.643 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:30:16.643 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:30:16.643 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.0", 2024-03-04T09:30:16.643 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQBCk+VlyjMMExAAqb2FJ9n5PWehfx/6PB19ig==", 2024-03-04T09:30:16.643 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:30:16.643 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-04T09:30:16.643 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-04T09:30:16.644 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-04T09:30:16.644 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:30:16.644 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:30:16.644 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:30:16.644 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.1", 2024-03-04T09:30:16.644 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQBFk+VlXD3cLBAAGclSivcYWDslqN8MLlj7eQ==", 2024-03-04T09:30:16.644 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:30:16.644 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-04T09:30:16.644 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-04T09:30:16.644 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-04T09:30:16.644 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:30:16.644 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:30:16.644 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:30:16.644 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.2", 2024-03-04T09:30:16.645 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQBJk+Vl7d9nCxAAnb3W4EI3Yf3XY8ht3VnKEQ==", 2024-03-04T09:30:16.645 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:30:16.645 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-04T09:30:16.645 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-04T09:30:16.645 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-04T09:30:16.645 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:30:16.645 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:30:16.645 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:30:16.645 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.3", 2024-03-04T09:30:16.645 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQBMk+Vlv+Y1NBAAiRYSUqov2i2sj3iInfpHvA==", 2024-03-04T09:30:16.645 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:30:16.645 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-04T09:30:16.645 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-04T09:30:16.645 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-04T09:30:16.645 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:30:16.646 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:30:16.646 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:30:16.646 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.4", 2024-03-04T09:30:16.646 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQBUk+Vl1OlAARAAxMV4GUGS+RJ6XlrRhrVKcg==", 2024-03-04T09:30:16.646 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:30:16.646 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-04T09:30:16.646 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-04T09:30:16.646 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-04T09:30:16.646 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:30:16.646 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:30:16.646 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:30:16.646 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.5", 2024-03-04T09:30:16.646 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQBXk+Vl1lKLGhAAHlsiRr+1+OSRWhkB+weIJg==", 2024-03-04T09:30:16.646 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:30:16.646 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-04T09:30:16.647 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-04T09:30:16.647 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-04T09:30:16.647 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:30:16.647 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:30:16.647 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:30:16.647 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.6", 2024-03-04T09:30:16.647 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQBak+VlwB+GJBAA3f9KUz+zZlQuUR/ywQfkDA==", 2024-03-04T09:30:16.647 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:30:16.647 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-04T09:30:16.647 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-04T09:30:16.647 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-04T09:30:16.647 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:30:16.647 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:30:16.647 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:30:16.647 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.7", 2024-03-04T09:30:16.648 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQBdk+Vle4o8LhAAko+iH5VqqCsfWkNfKZjGVA==", 2024-03-04T09:30:16.648 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:30:16.648 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-04T09:30:16.648 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-04T09:30:16.648 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-04T09:30:16.648 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:30:16.648 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:30:16.648 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:30:16.648 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "client.0", 2024-03-04T09:30:16.648 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQA9k+Vl4L+rJxAACaDbQ4/Bybp5hAZAnJzlVw==", 2024-03-04T09:30:16.648 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:30:16.648 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-04T09:30:16.648 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow r", 2024-03-04T09:30:16.648 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-04T09:30:16.649 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:30:16.649 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:30:16.649 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:30:16.649 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "client.1", 2024-03-04T09:30:16.649 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQA+k+VlGjkOAhAAJNzB8+tQl3mqCnTgMn9EVw==", 2024-03-04T09:30:16.649 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:30:16.649 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-04T09:30:16.649 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow r", 2024-03-04T09:30:16.649 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow rw", 2024-03-04T09:30:16.649 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow rwx" 2024-03-04T09:30:16.649 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:30:16.650 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:30:16.650 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:30:16.650 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "client.2", 2024-03-04T09:30:16.650 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQA9k+VlUc8DMxAA8pGUtIh+BO6UKrrYnCqVgQ==", 2024-03-04T09:30:16.650 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:30:16.650 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-04T09:30:16.650 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow r", 2024-03-04T09:30:16.650 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow rw", 2024-03-04T09:30:16.650 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow rwx" 2024-03-04T09:30:16.650 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:30:16.650 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:30:16.650 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:30:16.650 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "client.3", 2024-03-04T09:30:16.650 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQA+k+VlzMFKDBAAGr713I2hcmGMuHMIjNknYA==", 2024-03-04T09:30:16.651 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:30:16.651 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-04T09:30:16.651 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow r", 2024-03-04T09:30:16.651 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow rw", 2024-03-04T09:30:16.651 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow rwx" 2024-03-04T09:30:16.651 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:30:16.651 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:30:16.651 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:30:16.651 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "client.admin", 2024-03-04T09:30:16.651 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQA7k+Vllb7NFhAAmnFaNFKBcEri+iJAPnG5Gg==", 2024-03-04T09:30:16.651 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:30:16.651 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow *", 2024-03-04T09:30:16.651 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow *", 2024-03-04T09:30:16.651 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow *", 2024-03-04T09:30:16.652 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-04T09:30:16.652 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:30:16.652 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:30:16.652 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:30:16.652 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "mgr.x", 2024-03-04T09:30:16.652 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQA8k+VlPFatFBAA/o3Ps/uJrcg1738HOGy/FA==", 2024-03-04T09:30:16.652 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:30:16.652 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow *", 2024-03-04T09:30:16.652 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile mgr", 2024-03-04T09:30:16.652 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-04T09:30:16.652 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:30:16.652 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:30:16.652 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:30:16.652 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "mgr.y", 2024-03-04T09:30:16.653 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQA8k+VlEA2ACBAAXofDj3gmqItVTJ1pYLqZyQ==", 2024-03-04T09:30:16.653 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:30:16.653 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow *", 2024-03-04T09:30:16.653 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile mgr", 2024-03-04T09:30:16.653 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-04T09:30:16.653 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:30:16.653 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:30:16.653 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-04T09:30:16.653 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-04T09:30:16.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.644+0000 7fc62ae00640 1 -- 172.21.15.6:0/3474729419 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc60006b500 msgr2=0x7fc60006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:16.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.644+0000 7fc62ae00640 1 --2- 172.21.15.6:0/3474729419 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc60006b500 0x7fc60006d9c0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fc6240b0f50 tx=0x7fc60c007990 comp rx=0 tx=0).stop 2024-03-04T09:30:16.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.644+0000 7fc62ae00640 1 -- 172.21.15.6:0/3474729419 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc62406e240 msgr2=0x7fc624155a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:16.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.644+0000 7fc62ae00640 1 --2- 172.21.15.6:0/3474729419 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc62406e240 0x7fc624155a70 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7fc614009870 tx=0x7fc6140046c0 comp rx=0 tx=0).stop 2024-03-04T09:30:16.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.645+0000 7fc62ae00640 1 -- 172.21.15.6:0/3474729419 shutdown_connections 2024-03-04T09:30:16.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.645+0000 7fc62ae00640 1 --2- 172.21.15.6:0/3474729419 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc60006b500 0x7fc60006d9c0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:16.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.645+0000 7fc62ae00640 1 --2- 172.21.15.6:0/3474729419 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc624161670 0x7fc6241f86a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:16.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.645+0000 7fc62ae00640 1 --2- 172.21.15.6:0/3474729419 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc6240ae0a0 0x7fc6241f5290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:16.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.645+0000 7fc62ae00640 1 --2- 172.21.15.6:0/3474729419 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc62406e240 0x7fc624155a70 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:16.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.645+0000 7fc62ae00640 1 -- 172.21.15.6:0/3474729419 >> 172.21.15.6:0/3474729419 conn(0x7fc62406afb0 msgr2=0x7fc6240b2ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:16.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.645+0000 7fc62ae00640 1 -- 172.21.15.6:0/3474729419 shutdown_connections 2024-03-04T09:30:16.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.645+0000 7fc62ae00640 1 -- 172.21.15.6:0/3474729419 wait complete. 2024-03-04T09:30:16.660 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T09:30:16.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.871+0000 7f42eaa00640 1 Processor -- start 2024-03-04T09:30:16.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.871+0000 7f42eaa00640 1 -- start start 2024-03-04T09:30:16.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.874+0000 7f42eaa00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f42e40b3c90 0x7f42e40cdc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:16.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.874+0000 7f42eaa00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f42e4071050 0x7f42e4071490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:16.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.874+0000 7f42eaa00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42e40b4860 con 0x7f42e40cf5c0 2024-03-04T09:30:16.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.874+0000 7f42eaa00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42e40ce160 con 0x7f42e40b3c90 2024-03-04T09:30:16.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.874+0000 7f42eaa00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42e40b4b40 con 0x7f42e4071050 2024-03-04T09:30:16.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.874+0000 7f42e8a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f42e4071050 0x7f42e4071490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:16.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.874+0000 7f42e3e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f42e40b3c90 0x7f42e40cdc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:16.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.874+0000 7f42e8a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f42e4071050 0x7f42e4071490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.6:56750/0 (socket says 172.21.15.6:56750) 2024-03-04T09:30:16.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.874+0000 7f42e8a00640 1 -- 172.21.15.6:0/2550117691 learned_addr learned my addr 172.21.15.6:0/2550117691 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:30:16.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.874+0000 7f42e8a00640 1 -- 172.21.15.6:0/2550117691 >> v2:172.21.15.146:3300/0 conn(0x7f42e40b3c90 msgr2=0x7f42e40cdc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:16.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.874+0000 7f42e8a00640 1 --2- 172.21.15.6:0/2550117691 >> v2:172.21.15.146:3300/0 conn(0x7f42e40b3c90 0x7f42e40cdc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:16.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.874+0000 7f42e8a00640 1 -- 172.21.15.6:0/2550117691 >> v1:172.21.15.146:6789/0 conn(0x7f42e40cf5c0 legacy=0x7f42e40b4070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:16.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.874+0000 7f42e8a00640 1 -- 172.21.15.6:0/2550117691 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42e40b4e20 con 0x7f42e4071050 2024-03-04T09:30:16.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.876+0000 7f42e8a00640 1 --2- 172.21.15.6:0/2550117691 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f42e4071050 0x7f42e4071490 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f42dc009e50 tx=0x7f42dc02bff0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3912966f025b06ed server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:16.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.876+0000 7f42e2a00640 1 -- 172.21.15.6:0/2550117691 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42dc02cb90 con 0x7f42e4071050 2024-03-04T09:30:16.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.876+0000 7f42e2a00640 1 -- 172.21.15.6:0/2550117691 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42dc02cd30 con 0x7f42e4071050 2024-03-04T09:30:16.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.876+0000 7f42e2a00640 1 -- 172.21.15.6:0/2550117691 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42dc034610 con 0x7f42e4071050 2024-03-04T09:30:16.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.877+0000 7f42eaa00640 1 -- 172.21.15.6:0/2550117691 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f42e4071050 msgr2=0x7f42e4071490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:16.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.877+0000 7f42eaa00640 1 --2- 172.21.15.6:0/2550117691 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f42e4071050 0x7f42e4071490 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f42dc009e50 tx=0x7f42dc02bff0 comp rx=0 tx=0).stop 2024-03-04T09:30:16.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.877+0000 7f42eaa00640 1 -- 172.21.15.6:0/2550117691 shutdown_connections 2024-03-04T09:30:16.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.877+0000 7f42eaa00640 1 --2- 172.21.15.6:0/2550117691 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f42e4071050 0x7f42e4071490 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:16.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.877+0000 7f42eaa00640 1 --2- 172.21.15.6:0/2550117691 >> v2:172.21.15.146:3300/0 conn(0x7f42e40b3c90 0x7f42e40cdc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:16.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.877+0000 7f42eaa00640 1 -- 172.21.15.6:0/2550117691 >> 172.21.15.6:0/2550117691 conn(0x7f42e406afb0 msgr2=0x7f42e406b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:16.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.877+0000 7f42eaa00640 1 -- 172.21.15.6:0/2550117691 shutdown_connections 2024-03-04T09:30:16.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.877+0000 7f42eaa00640 1 -- 172.21.15.6:0/2550117691 wait complete. 2024-03-04T09:30:16.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.879+0000 7f42eaa00640 1 Processor -- start 2024-03-04T09:30:16.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.879+0000 7f42eaa00640 1 -- start start 2024-03-04T09:30:16.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.880+0000 7f42eaa00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f42e4071050 0x7f42e41469f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:16.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.880+0000 7f42eaa00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f42e40b3c90 0x7f42e4166f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:16.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.880+0000 7f42eaa00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f42e40cf5c0 0x7f42e40af940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:16.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.880+0000 7f42eaa00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42e40719d0 con 0x7f42e40b3c90 2024-03-04T09:30:16.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.880+0000 7f42eaa00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42e415b340 con 0x7f42e4071050 2024-03-04T09:30:16.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.880+0000 7f42eaa00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42e4071b50 con 0x7f42e40cf5c0 2024-03-04T09:30:16.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.880+0000 7f42e3400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f42e40b3c90 0x7f42e4166f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:16.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.880+0000 7f42e3400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f42e40b3c90 0x7f42e4166f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51972/0 (socket says 172.21.15.6:51972) 2024-03-04T09:30:16.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.880+0000 7f42e3400640 1 -- 172.21.15.6:0/2216414748 learned_addr learned my addr 172.21.15.6:0/2216414748 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:30:16.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.880+0000 7f42e8a00640 1 --2- 172.21.15.6:0/2216414748 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f42e40cf5c0 0x7f42e40af940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:16.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.880+0000 7f42e3e00640 1 --2- 172.21.15.6:0/2216414748 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f42e4071050 0x7f42e41469f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:16.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.881+0000 7f42e3400640 1 -- 172.21.15.6:0/2216414748 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f42e40cf5c0 msgr2=0x7f42e40af940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:16.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.881+0000 7f42e3400640 1 --2- 172.21.15.6:0/2216414748 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f42e40cf5c0 0x7f42e40af940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:16.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.881+0000 7f42e3400640 1 -- 172.21.15.6:0/2216414748 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f42e4071050 msgr2=0x7f42e41469f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:16.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.881+0000 7f42e3400640 1 --2- 172.21.15.6:0/2216414748 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f42e4071050 0x7f42e41469f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:16.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.881+0000 7f42e3400640 1 -- 172.21.15.6:0/2216414748 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42e4146f30 con 0x7f42e40b3c90 2024-03-04T09:30:16.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.881+0000 7f42e3e00640 1 --2- 172.21.15.6:0/2216414748 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f42e4071050 0x7f42e41469f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:30:16.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.881+0000 7f42e3400640 1 --2- 172.21.15.6:0/2216414748 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f42e40b3c90 0x7f42e4166f00 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f42cc00c3c0 tx=0x7f42cc00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:16.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.881+0000 7f42dbe00640 1 -- 172.21.15.6:0/2216414748 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42cc0052d0 con 0x7f42e40b3c90 2024-03-04T09:30:16.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.881+0000 7f42dbe00640 1 -- 172.21.15.6:0/2216414748 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42cc005490 con 0x7f42e40b3c90 2024-03-04T09:30:16.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.881+0000 7f42dbe00640 1 -- 172.21.15.6:0/2216414748 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42cc007260 con 0x7f42e40b3c90 2024-03-04T09:30:16.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.881+0000 7f42eaa00640 1 -- 172.21.15.6:0/2216414748 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42e4167ff0 con 0x7f42e40b3c90 2024-03-04T09:30:16.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.881+0000 7f42eaa00640 1 -- 172.21.15.6:0/2216414748 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42e4166700 con 0x7f42e40b3c90 2024-03-04T09:30:16.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.883+0000 7f42eaa00640 1 -- 172.21.15.6:0/2216414748 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42e404f3a0 con 0x7f42e40b3c90 2024-03-04T09:30:16.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.884+0000 7f42dbe00640 1 -- 172.21.15.6:0/2216414748 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f42cc007400 con 0x7f42e40b3c90 2024-03-04T09:30:16.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.885+0000 7f42dbe00640 1 --2- 172.21.15.6:0/2216414748 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f42c006b4d0 0x7f42c006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:16.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.885+0000 7f42dbe00640 1 -- 172.21.15.6:0/2216414748 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f42cc089430 con 0x7f42e40b3c90 2024-03-04T09:30:16.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.885+0000 7f42e3e00640 1 --2- 172.21.15.6:0/2216414748 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f42c006b4d0 0x7f42c006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:16.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.886+0000 7f42e3e00640 1 --2- 172.21.15.6:0/2216414748 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f42c006b4d0 0x7f42c006d990 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f42d4009ac0 tx=0x7f42d4008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:16.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:16.889+0000 7f42dbe00640 1 -- 172.21.15.6:0/2216414748 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f42cc058560 con 0x7f42e40b3c90 2024-03-04T09:30:17.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.086+0000 7f42eaa00640 1 -- 172.21.15.6:0/2216414748 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f42e414e5a0 con 0x7f42e40b3c90 2024-03-04T09:30:17.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.088+0000 7f42dbe00640 1 -- 172.21.15.6:0/2216414748 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) v1 ==== 74+0+12844 (secure 0 0 0) 0x7f42cc057e60 con 0x7f42e40b3c90 2024-03-04T09:30:17.086 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:30:17.086 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":90,"fsid":"edce0578-5ae4-4402-b45a-552bdef84d7c","created":"2024-03-04T09:24:54.523780+0000","modified":"2024-03-04T09:30:16.142206+0000","last_up_change":"2024-03-04T09:25:07.089049+0000","last_in_change":"2024-03-04T09:24:57.982077+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T09:25:07.935198+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a96574f-81f1-4bc6-8463-757520b84a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6825","nonce":1364449401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6827","nonce":1364449401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6831","nonce":1364449401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6829","nonce":1364449401}]},"public_addr":"172.21.15.6:6825/1364449401","cluster_addr":"172.21.15.6:6827/1364449401","heartbeat_back_addr":"172.21.15.6:6831/1364449401","heartbeat_front_addr":"172.21.15.6:6829/1364449401","state":["exists","up"]},{"osd":1,"uuid":"7d79b9c1-6b87-476a-bc36-7db5ee64cd90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6809","nonce":2494103078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6811","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6813","nonce":2494103078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6821","nonce":2494103078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6817","nonce":2494103078}]},"public_addr":"172.21.15.6:6809/2494103078","cluster_addr":"172.21.15.6:6813/2494103078","heartbeat_back_addr":"172.21.15.6:6821/2494103078","heartbeat_front_addr":"172.21.15.6:6817/2494103078","state":["exists","up"]},{"osd":2,"uuid":"492331b9-4ee9-40b6-bacd-85d44cd6c45a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6812","nonce":1500066681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6816","nonce":1500066681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6824","nonce":1500066681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6820","nonce":1500066681}]},"public_addr":"172.21.15.6:6812/1500066681","cluster_addr":"172.21.15.6:6816/1500066681","heartbeat_back_addr":"172.21.15.6:6824/1500066681","heartbeat_front_addr":"172.21.15.6:6820/1500066681","state":["exists","up"]},{"osd":3,"uuid":"21bb2efe-9ba7-4bdc-bc31-180cd1102e21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6801","nonce":1558806123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6803","nonce":1558806123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6807","nonce":1558806123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6805","nonce":1558806123}]},"public_addr":"172.21.15.6:6801/1558806123","cluster_addr":"172.21.15.6:6803/1558806123","heartbeat_back_addr":"172.21.15.6:6807/1558806123","heartbeat_front_addr":"172.21.15.6:6805/1558806123","state":["exists","up"]},{"osd":4,"uuid":"6e2c4b71-19d9-4737-83d9-558705d8644a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6801","nonce":1526135780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6803","nonce":1526135780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6807","nonce":1526135780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6805","nonce":1526135780}]},"public_addr":"172.21.15.146:6801/1526135780","cluster_addr":"172.21.15.146:6803/1526135780","heartbeat_back_addr":"172.21.15.146:6807/1526135780","heartbeat_front_addr":"172.21.15.146:6805/1526135780","state":["exists","up"]},{"osd":5,"uuid":"fb69c525-f1be-4b7d-ab32-6c29b0d6fddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6809","nonce":460262507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6811","nonce":460262507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6815","nonce":460262507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6813","nonce":460262507}]},"public_addr":"172.21.15.146:6809/460262507","cluster_addr":"172.21.15.146:6811/460262507","heartbeat_back_addr":"172.21.15.146:6815/460262507","heartbeat_front_addr":"172.21.15.146:6813/460262507","state":["exists","up"]},{"osd":6,"uuid":"c32e7fed-370d-4401-8ce0-6a4335b0d168","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6823","nonce":1392897493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6827","nonce":1392897493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6831","nonce":1392897493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6829","nonce":1392897493}]},"public_addr":"172.21.15.146:6823/1392897493","cluster_addr":"172.21.15.146:6827/1392897493","heartbeat_back_addr":"172.21.15.146:6831/1392897493","heartbeat_front_addr":"172.21.15.146:6829/1392897493","state":["exists","up"]},{"osd":7,"uuid":"75a79b44-fa5f-426d-a7d3-71f21f78835c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6817","nonce":3268323156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6819","nonce":3268323156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6825","nonce":3268323156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6822","nonce":3268323156}]},"public_addr":"172.21.15.146:6817/3268323156","cluster_addr":"172.21.15.146:6819/3268323156","heartbeat_back_addr":"172.21.15.146:6825/3268323156","heartbeat_front_addr":"172.21.15.146:6822/3268323156","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.310437+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.295850+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.294674+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.526855+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.300023+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.595160+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.346046+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.558741+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T09:30:17.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.090+0000 7f42eaa00640 1 -- 172.21.15.6:0/2216414748 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f42c006b4d0 msgr2=0x7f42c006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:17.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.090+0000 7f42eaa00640 1 --2- 172.21.15.6:0/2216414748 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f42c006b4d0 0x7f42c006d990 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f42d4009ac0 tx=0x7f42d4008040 comp rx=0 tx=0).stop 2024-03-04T09:30:17.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.090+0000 7f42eaa00640 1 -- 172.21.15.6:0/2216414748 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f42e40b3c90 msgr2=0x7f42e4166f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:17.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.090+0000 7f42eaa00640 1 --2- 172.21.15.6:0/2216414748 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f42e40b3c90 0x7f42e4166f00 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f42cc00c3c0 tx=0x7f42cc00c880 comp rx=0 tx=0).stop 2024-03-04T09:30:17.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.091+0000 7f42eaa00640 1 -- 172.21.15.6:0/2216414748 shutdown_connections 2024-03-04T09:30:17.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.091+0000 7f42eaa00640 1 --2- 172.21.15.6:0/2216414748 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f42c006b4d0 0x7f42c006d990 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:17.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.091+0000 7f42eaa00640 1 --2- 172.21.15.6:0/2216414748 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f42e40cf5c0 0x7f42e40af940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:17.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.091+0000 7f42eaa00640 1 --2- 172.21.15.6:0/2216414748 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f42e40b3c90 0x7f42e4166f00 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:17.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.091+0000 7f42eaa00640 1 --2- 172.21.15.6:0/2216414748 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f42e4071050 0x7f42e41469f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:17.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.091+0000 7f42eaa00640 1 -- 172.21.15.6:0/2216414748 >> 172.21.15.6:0/2216414748 conn(0x7f42e406afb0 msgr2=0x7f42e414d980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:17.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.091+0000 7f42eaa00640 1 -- 172.21.15.6:0/2216414748 shutdown_connections 2024-03-04T09:30:17.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.091+0000 7f42eaa00640 1 -- 172.21.15.6:0/2216414748 wait complete. 2024-03-04T09:30:17.106 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T09:25:07.935198+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-04T09:30:17.106 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-04T09:30:17.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.316+0000 7f6343600640 1 Processor -- start 2024-03-04T09:30:17.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.317+0000 7f6343600640 1 -- start start 2024-03-04T09:30:17.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.318+0000 7f6343600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f633c071e20 0x7f633c0ccad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:17.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.318+0000 7f6343600640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f633c072df0 con 0x7f633c072520 2024-03-04T09:30:17.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.318+0000 7f6343600640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f633c0cdab0 con 0x7f633c15bf20 2024-03-04T09:30:17.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.318+0000 7f6343600640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f633c0b4d70 con 0x7f633c071e20 2024-03-04T09:30:17.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.319+0000 7f633be00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f633c072520 0x7f633c0cdfb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:34872/0 (socket says 172.21.15.6:34872) 2024-03-04T09:30:17.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.319+0000 7f633be00640 1 -- 172.21.15.6:0/1066451756 learned_addr learned my addr 172.21.15.6:0/1066451756 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:30:17.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.319+0000 7f6341600640 1 --2- 172.21.15.6:0/1066451756 >> v2:172.21.15.6:3300/0 conn(0x7f633c071e20 0x7f633c0ccad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:17.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.319+0000 7f633b400640 1 -- 172.21.15.6:0/1066451756 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3443232716 0 0) 0x7f633c072df0 con 0x7f633c072520 2024-03-04T09:30:17.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.319+0000 7f633b400640 1 -- 172.21.15.6:0/1066451756 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6320003520 con 0x7f633c072520 2024-03-04T09:30:17.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.319+0000 7f6341600640 1 -- 172.21.15.6:0/1066451756 >> v1:172.21.15.146:6789/0 conn(0x7f633c15bf20 legacy=0x7f633c0acd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:17.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.319+0000 7f6341600640 1 -- 172.21.15.6:0/1066451756 >> v1:172.21.15.6:6789/0 conn(0x7f633c072520 legacy=0x7f633c0cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:17.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.319+0000 7f6341600640 1 -- 172.21.15.6:0/1066451756 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f633c0b5050 con 0x7f633c071e20 2024-03-04T09:30:17.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.320+0000 7f6341600640 1 --2- 172.21.15.6:0/1066451756 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f633c071e20 0x7f633c0ccad0 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f63340097a0 tx=0x7f633402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=777e67ebb9b99109 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:17.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.321+0000 7f633b400640 1 -- 172.21.15.6:0/1066451756 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6334035020 con 0x7f633c071e20 2024-03-04T09:30:17.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.321+0000 7f633b400640 1 -- 172.21.15.6:0/1066451756 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6334031830 con 0x7f633c071e20 2024-03-04T09:30:17.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.321+0000 7f633b400640 1 -- 172.21.15.6:0/1066451756 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6334031be0 con 0x7f633c071e20 2024-03-04T09:30:17.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.321+0000 7f6343600640 1 -- 172.21.15.6:0/1066451756 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f633c071e20 msgr2=0x7f633c0ccad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:17.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.321+0000 7f6343600640 1 --2- 172.21.15.6:0/1066451756 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f633c071e20 0x7f633c0ccad0 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f63340097a0 tx=0x7f633402b8d0 comp rx=0 tx=0).stop 2024-03-04T09:30:17.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.321+0000 7f6343600640 1 -- 172.21.15.6:0/1066451756 shutdown_connections 2024-03-04T09:30:17.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.321+0000 7f6343600640 1 --2- 172.21.15.6:0/1066451756 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f633c071e20 0x7f633c0ccad0 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:17.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.321+0000 7f6343600640 1 -- 172.21.15.6:0/1066451756 >> 172.21.15.6:0/1066451756 conn(0x7f633c06b3d0 msgr2=0x7f633c06b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:17.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.321+0000 7f6343600640 1 -- 172.21.15.6:0/1066451756 shutdown_connections 2024-03-04T09:30:17.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.322+0000 7f6343600640 1 -- 172.21.15.6:0/1066451756 wait complete. 2024-03-04T09:30:17.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.323+0000 7f6343600640 1 Processor -- start 2024-03-04T09:30:17.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.323+0000 7f6343600640 1 -- start start 2024-03-04T09:30:17.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.324+0000 7f6343600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f633c071e20 0x7f633c150670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:17.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.324+0000 7f6343600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f633c072520 0x7f633c09b990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:17.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.324+0000 7f6343600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f633c15bf20 0x7f633c1e6b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:17.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.324+0000 7f6343600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f633c0ad450 con 0x7f633c071e20 2024-03-04T09:30:17.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.324+0000 7f6343600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f633c0b4990 con 0x7f633c15bf20 2024-03-04T09:30:17.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.324+0000 7f6343600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f633c156100 con 0x7f633c072520 2024-03-04T09:30:17.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.325+0000 7f6340c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f633c071e20 0x7f633c150670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:17.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.325+0000 7f6340c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f633c071e20 0x7f633c150670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51994/0 (socket says 172.21.15.6:51994) 2024-03-04T09:30:17.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.325+0000 7f6340c00640 1 -- 172.21.15.6:0/2548264909 learned_addr learned my addr 172.21.15.6:0/2548264909 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:30:17.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.325+0000 7f6340c00640 1 -- 172.21.15.6:0/2548264909 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f633c072520 msgr2=0x7f633c09b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:17.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.325+0000 7f6340c00640 1 --2- 172.21.15.6:0/2548264909 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f633c072520 0x7f633c09b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:17.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.325+0000 7f6341600640 1 --2- 172.21.15.6:0/2548264909 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f633c15bf20 0x7f633c1e6b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:17.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.325+0000 7f6340c00640 1 -- 172.21.15.6:0/2548264909 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f633c15bf20 msgr2=0x7f633c1e6b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:17.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.325+0000 7f6340c00640 1 --2- 172.21.15.6:0/2548264909 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f633c15bf20 0x7f633c1e6b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:17.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.325+0000 7f6340c00640 1 -- 172.21.15.6:0/2548264909 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f633c1554f0 con 0x7f633c071e20 2024-03-04T09:30:17.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.325+0000 7f6341600640 1 --2- 172.21.15.6:0/2548264909 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f633c15bf20 0x7f633c1e6b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:30:17.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.325+0000 7f6340c00640 1 --2- 172.21.15.6:0/2548264909 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f633c071e20 0x7f633c150670 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f632c00c7c0 tx=0x7f632c00cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:17.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.326+0000 7f6338c00640 1 -- 172.21.15.6:0/2548264909 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f632c016020 con 0x7f633c071e20 2024-03-04T09:30:17.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.326+0000 7f6338c00640 1 -- 172.21.15.6:0/2548264909 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f632c00aa50 con 0x7f633c071e20 2024-03-04T09:30:17.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.326+0000 7f6338c00640 1 -- 172.21.15.6:0/2548264909 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f632c012420 con 0x7f633c071e20 2024-03-04T09:30:17.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.326+0000 7f6343600640 1 -- 172.21.15.6:0/2548264909 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f633c1557d0 con 0x7f633c071e20 2024-03-04T09:30:17.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.326+0000 7f6343600640 1 -- 172.21.15.6:0/2548264909 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f633c155d90 con 0x7f633c071e20 2024-03-04T09:30:17.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.327+0000 7f6343600640 1 -- 172.21.15.6:0/2548264909 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f633c067830 con 0x7f633c071e20 2024-03-04T09:30:17.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.328+0000 7f6338c00640 1 -- 172.21.15.6:0/2548264909 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f632c019020 con 0x7f633c071e20 2024-03-04T09:30:17.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.329+0000 7f6338c00640 1 --2- 172.21.15.6:0/2548264909 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f631c06b4d0 0x7f631c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:17.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.329+0000 7f6338c00640 1 -- 172.21.15.6:0/2548264909 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f632c089bf0 con 0x7f633c071e20 2024-03-04T09:30:17.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.330+0000 7f633be00640 1 --2- 172.21.15.6:0/2548264909 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f631c06b4d0 0x7f631c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:17.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.331+0000 7f633be00640 1 --2- 172.21.15.6:0/2548264909 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f631c06b4d0 0x7f631c06d990 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f6328000f60 tx=0x7f63280078f0 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:17.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.333+0000 7f6338c00640 1 -- 172.21.15.6:0/2548264909 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f633c067830 con 0x7f633c071e20 2024-03-04T09:30:17.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.529+0000 7f6343600640 1 -- 172.21.15.6:0/2548264909 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f633c04f330 con 0x7f633c071e20 2024-03-04T09:30:17.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.530+0000 7f6338c00640 1 -- 172.21.15.6:0/2548264909 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v90) v1 ==== 93+0+10 (secure 0 0 0) 0x7f632c020070 con 0x7f633c071e20 2024-03-04T09:30:17.528 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-04T09:30:17.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.532+0000 7f6343600640 1 -- 172.21.15.6:0/2548264909 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f631c06b4d0 msgr2=0x7f631c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:17.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.532+0000 7f6343600640 1 --2- 172.21.15.6:0/2548264909 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f631c06b4d0 0x7f631c06d990 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f6328000f60 tx=0x7f63280078f0 comp rx=0 tx=0).stop 2024-03-04T09:30:17.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.532+0000 7f6343600640 1 -- 172.21.15.6:0/2548264909 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f633c071e20 msgr2=0x7f633c150670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:17.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.532+0000 7f6343600640 1 --2- 172.21.15.6:0/2548264909 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f633c071e20 0x7f633c150670 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f632c00c7c0 tx=0x7f632c00cc80 comp rx=0 tx=0).stop 2024-03-04T09:30:17.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.533+0000 7f6343600640 1 -- 172.21.15.6:0/2548264909 shutdown_connections 2024-03-04T09:30:17.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.533+0000 7f6343600640 1 --2- 172.21.15.6:0/2548264909 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f633c15bf20 0x7f633c1e6b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:17.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.533+0000 7f6343600640 1 --2- 172.21.15.6:0/2548264909 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f631c06b4d0 0x7f631c06d990 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:17.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.533+0000 7f6343600640 1 --2- 172.21.15.6:0/2548264909 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f633c072520 0x7f633c09b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:17.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.533+0000 7f6343600640 1 --2- 172.21.15.6:0/2548264909 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f633c071e20 0x7f633c150670 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:17.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.533+0000 7f6343600640 1 -- 172.21.15.6:0/2548264909 >> 172.21.15.6:0/2548264909 conn(0x7f633c06b3d0 msgr2=0x7f633c0ae090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:17.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.533+0000 7f6343600640 1 -- 172.21.15.6:0/2548264909 shutdown_connections 2024-03-04T09:30:17.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.533+0000 7f6343600640 1 -- 172.21.15.6:0/2548264909 wait complete. 2024-03-04T09:30:17.549 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-04T09:30:17.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.766+0000 7f54d8c00640 1 Processor -- start 2024-03-04T09:30:17.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.766+0000 7f54d8c00640 1 -- start start 2024-03-04T09:30:17.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.768+0000 7f54d8c00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f54d415dec0 0x7f54d40ce810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:17.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.768+0000 7f54d8c00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54d40d1120 con 0x7f54d415e2a0 2024-03-04T09:30:17.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.768+0000 7f54d8c00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54d40d1400 con 0x7f54d40cf460 2024-03-04T09:30:17.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.768+0000 7f54d8c00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54d40b0ea0 con 0x7f54d415dec0 2024-03-04T09:30:17.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.768+0000 7f54d2a00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f54d415e2a0 0x7f54d40ced50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:34876/0 (socket says 172.21.15.6:34876) 2024-03-04T09:30:17.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.768+0000 7f54d2a00640 1 -- 172.21.15.6:0/1732265365 learned_addr learned my addr 172.21.15.6:0/1732265365 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:30:17.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.768+0000 7f54d3400640 1 --2- 172.21.15.6:0/1732265365 >> v2:172.21.15.146:3300/0 conn(0x7f54d415dec0 0x7f54d40ce810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:17.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.768+0000 7f54d2000640 1 -- 172.21.15.6:0/1732265365 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2175690366 0 0) 0x7f54d40d1120 con 0x7f54d415e2a0 2024-03-04T09:30:17.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.768+0000 7f54d2000640 1 -- 172.21.15.6:0/1732265365 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54b8003520 con 0x7f54d415e2a0 2024-03-04T09:30:17.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.769+0000 7f54d2000640 1 -- 172.21.15.6:0/1732265365 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4050102458 0 0) 0x7f54b8003520 con 0x7f54d415e2a0 2024-03-04T09:30:17.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.769+0000 7f54d2000640 1 -- 172.21.15.6:0/1732265365 >> v2:172.21.15.146:3300/0 conn(0x7f54d415dec0 msgr2=0x7f54d40ce810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:17.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.769+0000 7f54d2000640 1 --2- 172.21.15.6:0/1732265365 >> v2:172.21.15.146:3300/0 conn(0x7f54d415dec0 0x7f54d40ce810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:17.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.769+0000 7f54d2000640 1 -- 172.21.15.6:0/1732265365 >> v1:172.21.15.146:6789/0 conn(0x7f54d40cf460 legacy=0x7f54d40b1330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:17.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.769+0000 7f54d2000640 1 -- 172.21.15.6:0/1732265365 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54d40b0b30 con 0x7f54d415e2a0 2024-03-04T09:30:17.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.769+0000 7f54d2000640 1 -- 172.21.15.6:0/1732265365 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7f54c0001670 con 0x7f54d415e2a0 2024-03-04T09:30:17.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.769+0000 7f54d2000640 1 -- 172.21.15.6:0/1732265365 >> v1:172.21.15.6:6789/0 conn(0x7f54d415e2a0 legacy=0x7f54d40ced50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:17.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.769+0000 7f54d2000640 1 --2- 172.21.15.6:0/1732265365 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f54b80039b0 0x7f54b8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:17.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.769+0000 7f54d2000640 1 --2- 172.21.15.6:0/1732265365 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54b8004800 0x7f54b8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:17.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.769+0000 7f54d2000640 1 --2- 172.21.15.6:0/1732265365 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f54b80051c0 0x7f54b800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:17.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.769+0000 7f54d2000640 1 -- 172.21.15.6:0/1732265365 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54d40b0ea0 con 0x7f54b8004800 2024-03-04T09:30:17.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.769+0000 7f54d2000640 1 -- 172.21.15.6:0/1732265365 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54b800c060 con 0x7f54b80051c0 2024-03-04T09:30:17.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.769+0000 7f54d2000640 1 -- 172.21.15.6:0/1732265365 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54b800c260 con 0x7f54b80039b0 2024-03-04T09:30:17.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.769+0000 7f54d2a00640 1 --2- 172.21.15.6:0/1732265365 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54b8004800 0x7f54b8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:17.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.769+0000 7f54d3400640 1 --2- 172.21.15.6:0/1732265365 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f54b80039b0 0x7f54b8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:17.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.770+0000 7f54d2a00640 1 -- 172.21.15.6:0/1732265365 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f54b80039b0 msgr2=0x7f54b8003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:17.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.770+0000 7f54d2a00640 1 --2- 172.21.15.6:0/1732265365 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f54b80039b0 0x7f54b8003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:17.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.770+0000 7f54d2a00640 1 -- 172.21.15.6:0/1732265365 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f54b80051c0 msgr2=0x7f54b800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:17.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.770+0000 7f54d2a00640 1 --2- 172.21.15.6:0/1732265365 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f54b80051c0 0x7f54b800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:17.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.770+0000 7f54d2a00640 1 -- 172.21.15.6:0/1732265365 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54b8003520 con 0x7f54b8004800 2024-03-04T09:30:17.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.770+0000 7f54d3e00640 1 -- 172.21.15.6:0/1732265365 reap_dead start 2024-03-04T09:30:17.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.770+0000 7f54d3400640 1 --2- 172.21.15.6:0/1732265365 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f54b80039b0 0x7f54b8003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:30:17.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.771+0000 7f54d2a00640 1 --2- 172.21.15.6:0/1732265365 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54b8004800 0x7f54b8004c80 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f54c000c210 tx=0x7f54b8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42b0392ff435213b server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:17.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.771+0000 7f54d2000640 1 -- 172.21.15.6:0/1732265365 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54c0007d30 con 0x7f54b8004800 2024-03-04T09:30:17.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.771+0000 7f54d2000640 1 -- 172.21.15.6:0/1732265365 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54c0030070 con 0x7f54b8004800 2024-03-04T09:30:17.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.771+0000 7f54d2000640 1 -- 172.21.15.6:0/1732265365 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54c002cda0 con 0x7f54b8004800 2024-03-04T09:30:17.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.772+0000 7f54d8c00640 1 -- 172.21.15.6:0/1732265365 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54b8004800 msgr2=0x7f54b8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:17.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.772+0000 7f54d8c00640 1 --2- 172.21.15.6:0/1732265365 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54b8004800 0x7f54b8004c80 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f54c000c210 tx=0x7f54b8005830 comp rx=0 tx=0).stop 2024-03-04T09:30:17.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.772+0000 7f54d8c00640 1 -- 172.21.15.6:0/1732265365 shutdown_connections 2024-03-04T09:30:17.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.772+0000 7f54d8c00640 1 --2- 172.21.15.6:0/1732265365 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54b8004800 0x7f54b8004c80 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:17.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.772+0000 7f54d8c00640 1 -- 172.21.15.6:0/1732265365 >> 172.21.15.6:0/1732265365 conn(0x7f54d40d3d40 msgr2=0x7f54d40d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:17.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.772+0000 7f54d8c00640 1 -- 172.21.15.6:0/1732265365 shutdown_connections 2024-03-04T09:30:17.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.772+0000 7f54d8c00640 1 -- 172.21.15.6:0/1732265365 wait complete. 2024-03-04T09:30:17.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.776+0000 7f54d8c00640 1 Processor -- start 2024-03-04T09:30:17.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.776+0000 7f54d8c00640 1 -- start start 2024-03-04T09:30:17.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.777+0000 7f54d8c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f54b8004800 0x7f54d40cae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:17.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.777+0000 7f54d8c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f54d40b9350 0x7f54d415bd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:17.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.777+0000 7f54d8c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54d40b17d0 0x7f54d40b1c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:17.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.777+0000 7f54d8c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54d40ce990 con 0x7f54d40b17d0 2024-03-04T09:30:17.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.777+0000 7f54d8c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54d40ce810 con 0x7f54d40b9350 2024-03-04T09:30:17.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.777+0000 7f54d8c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54d40ceb10 con 0x7f54b8004800 2024-03-04T09:30:17.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.777+0000 7f54d3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54d40b17d0 0x7f54d40b1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:17.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.777+0000 7f54d3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54d40b17d0 0x7f54d40b1c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:52014/0 (socket says 172.21.15.6:52014) 2024-03-04T09:30:17.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.777+0000 7f54d3e00640 1 -- 172.21.15.6:0/4261304235 learned_addr learned my addr 172.21.15.6:0/4261304235 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:30:17.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.777+0000 7f54d2a00640 1 --2- 172.21.15.6:0/4261304235 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f54d40b9350 0x7f54d415bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:17.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.777+0000 7f54d3400640 1 --2- 172.21.15.6:0/4261304235 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f54b8004800 0x7f54d40cae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:17.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.777+0000 7f54d3e00640 1 -- 172.21.15.6:0/4261304235 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f54b8004800 msgr2=0x7f54d40cae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:17.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.777+0000 7f54d3e00640 1 --2- 172.21.15.6:0/4261304235 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f54b8004800 0x7f54d40cae10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:17.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.777+0000 7f54d3e00640 1 -- 172.21.15.6:0/4261304235 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f54d40b9350 msgr2=0x7f54d415bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:17.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.777+0000 7f54d3e00640 1 --2- 172.21.15.6:0/4261304235 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f54d40b9350 0x7f54d415bd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:17.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.777+0000 7f54d3e00640 1 -- 172.21.15.6:0/4261304235 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54d40cb660 con 0x7f54d40b17d0 2024-03-04T09:30:17.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.777+0000 7f54d3e00640 1 --2- 172.21.15.6:0/4261304235 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54d40b17d0 0x7f54d40b1c50 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f54d40cf8d0 tx=0x7f54cc00c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:17.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.777+0000 7f54cb400640 1 -- 172.21.15.6:0/4261304235 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54cc017070 con 0x7f54d40b17d0 2024-03-04T09:30:17.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.777+0000 7f54cb400640 1 -- 172.21.15.6:0/4261304235 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54cc0070d0 con 0x7f54d40b17d0 2024-03-04T09:30:17.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.778+0000 7f54cb400640 1 -- 172.21.15.6:0/4261304235 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54cc0074a0 con 0x7f54d40b17d0 2024-03-04T09:30:17.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.778+0000 7f54d8c00640 1 -- 172.21.15.6:0/4261304235 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54d40cb940 con 0x7f54d40b17d0 2024-03-04T09:30:17.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.778+0000 7f54d8c00640 1 -- 172.21.15.6:0/4261304235 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54d40cbf00 con 0x7f54d40b17d0 2024-03-04T09:30:17.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.779+0000 7f54d8c00640 1 -- 172.21.15.6:0/4261304235 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54d404f330 con 0x7f54d40b17d0 2024-03-04T09:30:17.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.780+0000 7f54cb400640 1 -- 172.21.15.6:0/4261304235 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f54cc01a070 con 0x7f54d40b17d0 2024-03-04T09:30:17.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.781+0000 7f54cb400640 1 --2- 172.21.15.6:0/4261304235 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f54b406b4d0 0x7f54b406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:17.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.781+0000 7f54cb400640 1 -- 172.21.15.6:0/4261304235 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f54cc0890a0 con 0x7f54d40b17d0 2024-03-04T09:30:17.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.781+0000 7f54d3400640 1 --2- 172.21.15.6:0/4261304235 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f54b406b4d0 0x7f54b406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:17.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.783+0000 7f54d3400640 1 --2- 172.21.15.6:0/4261304235 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f54b406b4d0 0x7f54b406d990 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f54d4069ad0 tx=0x7f54c4008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:17.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.786+0000 7f54cb400640 1 -- 172.21.15.6:0/4261304235 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f54cc058270 con 0x7f54d40b17d0 2024-03-04T09:30:17.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.986+0000 7f54d8c00640 1 -- 172.21.15.6:0/4261304235 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f54d41608b0 con 0x7f54d40b17d0 2024-03-04T09:30:17.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.986+0000 7f54cb400640 1 -- 172.21.15.6:0/4261304235 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v90) v1 ==== 84+0+65 (secure 0 0 0) 0x7f54cc057b70 con 0x7f54d40b17d0 2024-03-04T09:30:17.984 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:30:17.984 INFO:teuthology.orchestra.run.smithi006.stdout:[ 2024-03-04T09:30:17.984 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:30:17.985 INFO:teuthology.orchestra.run.smithi006.stdout: "poolnum": 1, 2024-03-04T09:30:17.985 INFO:teuthology.orchestra.run.smithi006.stdout: "poolname": ".mgr" 2024-03-04T09:30:17.985 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:30:17.985 INFO:teuthology.orchestra.run.smithi006.stdout:] 2024-03-04T09:30:17.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.989+0000 7f54d8c00640 1 -- 172.21.15.6:0/4261304235 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f54b406b4d0 msgr2=0x7f54b406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:17.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.989+0000 7f54d8c00640 1 --2- 172.21.15.6:0/4261304235 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f54b406b4d0 0x7f54b406d990 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f54d4069ad0 tx=0x7f54c4008040 comp rx=0 tx=0).stop 2024-03-04T09:30:17.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.989+0000 7f54d8c00640 1 -- 172.21.15.6:0/4261304235 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54d40b17d0 msgr2=0x7f54d40b1c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:17.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.989+0000 7f54d8c00640 1 --2- 172.21.15.6:0/4261304235 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54d40b17d0 0x7f54d40b1c50 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f54d40cf8d0 tx=0x7f54cc00c960 comp rx=0 tx=0).stop 2024-03-04T09:30:17.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.989+0000 7f54d8c00640 1 -- 172.21.15.6:0/4261304235 shutdown_connections 2024-03-04T09:30:17.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.989+0000 7f54d8c00640 1 --2- 172.21.15.6:0/4261304235 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54d40b17d0 0x7f54d40b1c50 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:17.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.989+0000 7f54d8c00640 1 --2- 172.21.15.6:0/4261304235 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f54d40b9350 0x7f54d415bd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:17.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.989+0000 7f54d8c00640 1 --2- 172.21.15.6:0/4261304235 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f54b406b4d0 0x7f54b406d990 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:17.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.989+0000 7f54d8c00640 1 --2- 172.21.15.6:0/4261304235 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f54b8004800 0x7f54d40cae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:17.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.989+0000 7f54d8c00640 1 -- 172.21.15.6:0/4261304235 >> 172.21.15.6:0/4261304235 conn(0x7f54d40d3d40 msgr2=0x7f54d415ec90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:17.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.989+0000 7f54d8c00640 1 -- 172.21.15.6:0/4261304235 shutdown_connections 2024-03-04T09:30:17.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:17.989+0000 7f54d8c00640 1 -- 172.21.15.6:0/4261304235 wait complete. 2024-03-04T09:30:18.007 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-04T09:30:18.008 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-04T09:30:18.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:18.239+0000 7fd386800640 1 Processor -- start 2024-03-04T09:30:18.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:18.239+0000 7fd386800640 1 -- start start 2024-03-04T09:30:18.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:18.241+0000 7fd386800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd3800b3640 0x7fd38015cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:18.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:18.241+0000 7fd386800640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3801531f0 con 0x7fd3800aced0 2024-03-04T09:30:18.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:18.241+0000 7fd386800640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3800b2b80 con 0x7fd3800ae7b0 2024-03-04T09:30:18.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:18.241+0000 7fd386800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3800d04d0 con 0x7fd3800b3640 2024-03-04T09:30:18.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:18.241+0000 7fd37ea00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fd3800aced0 0x7fd38006cc70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:34890/0 (socket says 172.21.15.6:34890) 2024-03-04T09:30:18.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:18.241+0000 7fd37ea00640 1 -- 172.21.15.6:0/3032008627 learned_addr learned my addr 172.21.15.6:0/3032008627 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:30:18.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:18.242+0000 7fd37fe00640 1 --2- 172.21.15.6:0/3032008627 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd3800b3640 0x7fd38015cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:18.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:18.242+0000 7fd37e000640 1 -- 172.21.15.6:0/3032008627 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3447404246 0 0) 0x7fd3801531f0 con 0x7fd3800aced0 2024-03-04T09:30:18.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:18.242+0000 7fd37e000640 1 -- 172.21.15.6:0/3032008627 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd364003520 con 0x7fd3800aced0 2024-03-04T09:30:18.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:18.242+0000 7fd37e000640 1 -- 172.21.15.6:0/3032008627 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3242088086 0 0) 0x7fd364003520 con 0x7fd3800aced0 2024-03-04T09:30:18.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:18.242+0000 7fd37e000640 1 -- 172.21.15.6:0/3032008627 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd3800b3640 msgr2=0x7fd38015cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:18.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:18.243+0000 7fd37e000640 1 --2- 172.21.15.6:0/3032008627 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd3800b3640 0x7fd38015cdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:18.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:18.243+0000 7fd37e000640 1 -- 172.21.15.6:0/3032008627 >> v1:172.21.15.146:6789/0 conn(0x7fd3800ae7b0 legacy=0x7fd3800aeb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:18.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:18.243+0000 7fd37e000640 1 -- 172.21.15.6:0/3032008627 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3800b3ad0 con 0x7fd3800aced0 2024-03-04T09:30:18.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:18.243+0000 7fd37e000640 1 -- 172.21.15.6:0/3032008627 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7fd36c001670 con 0x7fd3800aced0 2024-03-04T09:30:18.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:18.243+0000 7fd37e000640 1 -- 172.21.15.6:0/3032008627 >> v1:172.21.15.6:6789/0 conn(0x7fd3800aced0 legacy=0x7fd38006cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:18.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:18.243+0000 7fd37e000640 1 --2- 172.21.15.6:0/3032008627 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd3800b3640 0x7fd3640039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:18.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:18.243+0000 7fd37e000640 1 --2- 172.21.15.6:0/3032008627 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd364003f60 0x7fd364004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:18.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:18.243+0000 7fd37e000640 1 --2- 172.21.15.6:0/3032008627 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd364004d40 0x7fd3640051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:18.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:18.243+0000 7fd37e000640 1 -- 172.21.15.6:0/3032008627 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3800d04d0 con 0x7fd364004d40 2024-03-04T09:30:18.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:18.243+0000 7fd37e000640 1 -- 172.21.15.6:0/3032008627 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd36400baa0 con 0x7fd364003f60 2024-03-04T09:30:18.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:18.243+0000 7fd37e000640 1 -- 172.21.15.6:0/3032008627 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd36400bca0 con 0x7fd3800b3640 2024-03-04T09:30:18.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:18.243+0000 7fd37fe00640 1 --2- 172.21.15.6:0/3032008627 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd364004d40 0x7fd3640051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:18.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:18.243+0000 7fd37ea00640 1 --2- 172.21.15.6:0/3032008627 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd364003f60 0x7fd364004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:18.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:18.243+0000 7fd37f400640 1 --2- 172.21.15.6:0/3032008627 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd3800b3640 0x7fd3640039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:18.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:18.243+0000 7fd37fe00640 1 -- 172.21.15.6:0/3032008627 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd3800b3640 msgr2=0x7fd3640039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:18.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:18.243+0000 7fd37fe00640 1 --2- 172.21.15.6:0/3032008627 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd3800b3640 0x7fd3640039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:18.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:18.243+0000 7fd37fe00640 1 -- 172.21.15.6:0/3032008627 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd364003f60 msgr2=0x7fd364004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:18.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:18.243+0000 7fd37fe00640 1 --2- 172.21.15.6:0/3032008627 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd364003f60 0x7fd364004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:18.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:18.243+0000 7fd37fe00640 1 -- 172.21.15.6:0/3032008627 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd364003520 con 0x7fd364004d40 2024-03-04T09:30:18.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:18.243+0000 7fd37f400640 1 --2- 172.21.15.6:0/3032008627 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd3800b3640 0x7fd3640039e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:30:18.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:18.244+0000 7fd37fe00640 1 --2- 172.21.15.6:0/3032008627 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd364004d40 0x7fd3640051c0 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7fd378009ac0 tx=0x7fd364005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10edc7cc11f977fc server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:18.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:18.244+0000 7fd37e000640 1 -- 172.21.15.6:0/3032008627 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd378039070 con 0x7fd364004d40 2024-03-04T09:30:18.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:18.244+0000 7fd37e000640 1 -- 172.21.15.6:0/3032008627 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd37802c6b0 con 0x7fd364004d40 2024-03-04T09:30:18.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:18.244+0000 7fd37e000640 1 -- 172.21.15.6:0/3032008627 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd378009690 con 0x7fd364004d40 2024-03-04T09:30:18.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:18.246+0000 7fd386800640 1 -- 172.21.15.6:0/3032008627 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd364004d40 msgr2=0x7fd3640051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:18.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:18.246+0000 7fd386800640 1 --2- 172.21.15.6:0/3032008627 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd364004d40 0x7fd3640051c0 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7fd378009ac0 tx=0x7fd364005830 comp rx=0 tx=0).stop 2024-03-04T09:30:18.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:18.246+0000 7fd37fe00640 1 -- 172.21.15.6:0/3032008627 reap_dead start 2024-03-04T09:30:18.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:18.246+0000 7fd386800640 1 -- 172.21.15.6:0/3032008627 shutdown_connections 2024-03-04T09:30:18.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:18.246+0000 7fd386800640 1 -- 172.21.15.6:0/3032008627 >> 172.21.15.6:0/3032008627 conn(0x7fd38006afb0 msgr2=0x7fd38006b3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:18.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:18.246+0000 7fd386800640 1 -- 172.21.15.6:0/3032008627 shutdown_connections 2024-03-04T09:30:18.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:18.246+0000 7fd386800640 1 -- 172.21.15.6:0/3032008627 wait complete. 2024-03-04T09:30:18.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:18.248+0000 7fd386800640 1 Processor -- start 2024-03-04T09:30:18.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:18.248+0000 7fd386800640 1 -- start start 2024-03-04T09:30:18.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:18.249+0000 7fd386800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd364004d40 0x7fd3801403e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:18.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:18.249+0000 7fd386800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd38009c620 0x7fd38009aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:18.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:18.249+0000 7fd386800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd38009af50 0x7fd380151a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:18.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:18.249+0000 7fd386800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3800b2e40 con 0x7fd38009af50 2024-03-04T09:30:18.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:18.249+0000 7fd386800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd38006eeb0 con 0x7fd364004d40 2024-03-04T09:30:18.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:18.249+0000 7fd386800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3800b3120 con 0x7fd38009c620 2024-03-04T09:30:18.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:18.249+0000 7fd37fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd38009af50 0x7fd380151a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:18.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:18.249+0000 7fd37fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd38009af50 0x7fd380151a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:52026/0 (socket says 172.21.15.6:52026) 2024-03-04T09:30:18.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:18.249+0000 7fd37fe00640 1 -- 172.21.15.6:0/4179866687 learned_addr learned my addr 172.21.15.6:0/4179866687 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:30:18.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:18.249+0000 7fd37fe00640 1 -- 172.21.15.6:0/4179866687 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd38009c620 msgr2=0x7fd38009aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:18.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:18.249+0000 7fd37fe00640 1 --2- 172.21.15.6:0/4179866687 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd38009c620 0x7fd38009aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:18.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:18.249+0000 7fd37f400640 1 --2- 172.21.15.6:0/4179866687 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd364004d40 0x7fd3801403e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:18.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:18.249+0000 7fd37fe00640 1 -- 172.21.15.6:0/4179866687 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd364004d40 msgr2=0x7fd3801403e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:18.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:18.250+0000 7fd37fe00640 1 --2- 172.21.15.6:0/4179866687 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd364004d40 0x7fd3801403e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:18.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:18.250+0000 7fd37fe00640 1 -- 172.21.15.6:0/4179866687 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd38014c9b0 con 0x7fd38009af50 2024-03-04T09:30:18.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:18.250+0000 7fd37fe00640 1 --2- 172.21.15.6:0/4179866687 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd38009af50 0x7fd380151a10 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7fd378009be0 tx=0x7fd378004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:18.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:18.250+0000 7fd377400640 1 -- 172.21.15.6:0/4179866687 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd378039070 con 0x7fd38009af50 2024-03-04T09:30:18.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:18.250+0000 7fd377400640 1 -- 172.21.15.6:0/4179866687 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd37803a040 con 0x7fd38009af50 2024-03-04T09:30:18.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:18.250+0000 7fd377400640 1 -- 172.21.15.6:0/4179866687 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd378009690 con 0x7fd38009af50 2024-03-04T09:30:18.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:18.250+0000 7fd386800640 1 -- 172.21.15.6:0/4179866687 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd38009b580 con 0x7fd38009af50 2024-03-04T09:30:18.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:18.250+0000 7fd386800640 1 -- 172.21.15.6:0/4179866687 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd38009bb40 con 0x7fd38009af50 2024-03-04T09:30:18.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:18.252+0000 7fd386800640 1 -- 172.21.15.6:0/4179866687 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd380068010 con 0x7fd38009af50 2024-03-04T09:30:18.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:18.252+0000 7fd377400640 1 -- 172.21.15.6:0/4179866687 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd378003610 con 0x7fd38009af50 2024-03-04T09:30:18.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:18.253+0000 7fd377400640 1 --2- 172.21.15.6:0/4179866687 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd36006b4d0 0x7fd36006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:18.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:18.253+0000 7fd377400640 1 -- 172.21.15.6:0/4179866687 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fd3780ab1f0 con 0x7fd38009af50 2024-03-04T09:30:18.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:18.254+0000 7fd37f400640 1 --2- 172.21.15.6:0/4179866687 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd36006b4d0 0x7fd36006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:18.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:18.255+0000 7fd37f400640 1 --2- 172.21.15.6:0/4179866687 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd36006b4d0 0x7fd36006d990 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fd370001d20 tx=0x7fd370009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:18.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:18.258+0000 7fd377400640 1 -- 172.21.15.6:0/4179866687 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd378079cd0 con 0x7fd38009af50 2024-03-04T09:30:18.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:18.456+0000 7fd386800640 1 -- 172.21.15.6:0/4179866687 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fd3800b4670 con 0x7fd38009af50 2024-03-04T09:30:19.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:19.152+0000 7fd377400640 1 -- 172.21.15.6:0/4179866687 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v91) v1 ==== 138+0+0 (secure 0 0 0) 0x7fd37807dd70 con 0x7fd38009af50 2024-03-04T09:30:19.149 INFO:teuthology.orchestra.run.smithi006.stderr:pool 'cephfs_metadata' created 2024-03-04T09:30:19.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:19.156+0000 7fd386800640 1 -- 172.21.15.6:0/4179866687 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd36006b4d0 msgr2=0x7fd36006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:19.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:19.156+0000 7fd386800640 1 --2- 172.21.15.6:0/4179866687 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd36006b4d0 0x7fd36006d990 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fd370001d20 tx=0x7fd370009040 comp rx=0 tx=0).stop 2024-03-04T09:30:19.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:19.156+0000 7fd386800640 1 -- 172.21.15.6:0/4179866687 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd38009af50 msgr2=0x7fd380151a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:19.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:19.156+0000 7fd386800640 1 --2- 172.21.15.6:0/4179866687 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd38009af50 0x7fd380151a10 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7fd378009be0 tx=0x7fd378004730 comp rx=0 tx=0).stop 2024-03-04T09:30:19.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:19.156+0000 7fd386800640 1 -- 172.21.15.6:0/4179866687 shutdown_connections 2024-03-04T09:30:19.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:19.156+0000 7fd386800640 1 --2- 172.21.15.6:0/4179866687 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd38009af50 0x7fd380151a10 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:19.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:19.156+0000 7fd386800640 1 --2- 172.21.15.6:0/4179866687 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd36006b4d0 0x7fd36006d990 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:19.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:19.156+0000 7fd386800640 1 --2- 172.21.15.6:0/4179866687 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd38009c620 0x7fd38009aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:19.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:19.156+0000 7fd386800640 1 --2- 172.21.15.6:0/4179866687 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd364004d40 0x7fd3801403e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:19.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:19.156+0000 7fd386800640 1 -- 172.21.15.6:0/4179866687 >> 172.21.15.6:0/4179866687 conn(0x7fd38006afb0 msgr2=0x7fd38014f2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:19.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:19.157+0000 7fd386800640 1 -- 172.21.15.6:0/4179866687 shutdown_connections 2024-03-04T09:30:19.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:19.157+0000 7fd386800640 1 -- 172.21.15.6:0/4179866687 wait complete. 2024-03-04T09:30:19.183 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-04T09:30:19.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:19.409+0000 7ff4c8000640 1 Processor -- start 2024-03-04T09:30:19.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:19.409+0000 7ff4c8000640 1 -- start start 2024-03-04T09:30:19.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:19.411+0000 7ff4c8000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff4c00ad6f0 0x7ff4c015c370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:19.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:19.411+0000 7ff4c8000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4c0156830 con 0x7ff4c006d5d0 2024-03-04T09:30:19.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:19.411+0000 7ff4c8000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4c0160120 con 0x7ff4c015f960 2024-03-04T09:30:19.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:19.411+0000 7ff4c8000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4c006ce60 con 0x7ff4c00ad6f0 2024-03-04T09:30:19.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:19.412+0000 7ff4c6000640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7ff4c006d5d0 0x7ff4c006da40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:55242/0 (socket says 172.21.15.6:55242) 2024-03-04T09:30:19.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:19.412+0000 7ff4c6000640 1 -- 172.21.15.6:0/417315940 learned_addr learned my addr 172.21.15.6:0/417315940 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:30:19.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:19.412+0000 7ff4c5600640 1 --2- 172.21.15.6:0/417315940 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff4c00ad6f0 0x7ff4c015c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:19.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:19.412+0000 7ff4bfe00640 1 -- 172.21.15.6:0/417315940 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4143072481 0 0) 0x7ff4c0156830 con 0x7ff4c006d5d0 2024-03-04T09:30:19.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:19.412+0000 7ff4bfe00640 1 -- 172.21.15.6:0/417315940 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4a8003520 con 0x7ff4c006d5d0 2024-03-04T09:30:19.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:19.413+0000 7ff4bfe00640 1 -- 172.21.15.6:0/417315940 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3949323037 0 0) 0x7ff4a8003520 con 0x7ff4c006d5d0 2024-03-04T09:30:19.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:19.413+0000 7ff4bfe00640 1 -- 172.21.15.6:0/417315940 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff4c00ad6f0 msgr2=0x7ff4c015c370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:19.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:19.413+0000 7ff4bfe00640 1 --2- 172.21.15.6:0/417315940 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff4c00ad6f0 0x7ff4c015c370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:19.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:19.413+0000 7ff4bfe00640 1 -- 172.21.15.6:0/417315940 >> v1:172.21.15.146:6789/0 conn(0x7ff4c015f960 legacy=0x7ff4c00adad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:19.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:19.413+0000 7ff4bfe00640 1 -- 172.21.15.6:0/417315940 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4c0160400 con 0x7ff4c006d5d0 2024-03-04T09:30:19.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:19.413+0000 7ff4bfe00640 1 -- 172.21.15.6:0/417315940 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7ff4b80019f0 con 0x7ff4c006d5d0 2024-03-04T09:30:19.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:19.413+0000 7ff4bfe00640 1 -- 172.21.15.6:0/417315940 >> v1:172.21.15.6:6789/0 conn(0x7ff4c006d5d0 legacy=0x7ff4c006da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:19.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:19.413+0000 7ff4bfe00640 1 --2- 172.21.15.6:0/417315940 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff4c00ad6f0 0x7ff4a80039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:19.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:19.413+0000 7ff4bfe00640 1 --2- 172.21.15.6:0/417315940 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff4a8003f60 0x7ff4a8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:19.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:19.413+0000 7ff4bfe00640 1 --2- 172.21.15.6:0/417315940 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff4a8004d40 0x7ff4a80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:19.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:19.413+0000 7ff4bfe00640 1 -- 172.21.15.6:0/417315940 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4c006ce60 con 0x7ff4a8003f60 2024-03-04T09:30:19.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:19.413+0000 7ff4bfe00640 1 -- 172.21.15.6:0/417315940 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4a800baa0 con 0x7ff4a8004d40 2024-03-04T09:30:19.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:19.413+0000 7ff4bfe00640 1 -- 172.21.15.6:0/417315940 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4a800bca0 con 0x7ff4c00ad6f0 2024-03-04T09:30:19.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:19.413+0000 7ff4c4c00640 1 --2- 172.21.15.6:0/417315940 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff4a8003f60 0x7ff4a8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:19.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:19.413+0000 7ff4c6000640 1 --2- 172.21.15.6:0/417315940 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff4a8004d40 0x7ff4a80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:19.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:19.413+0000 7ff4c4c00640 1 -- 172.21.15.6:0/417315940 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff4c00ad6f0 msgr2=0x7ff4a80039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:19.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:19.413+0000 7ff4c4c00640 1 --2- 172.21.15.6:0/417315940 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff4c00ad6f0 0x7ff4a80039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:19.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:19.413+0000 7ff4c4c00640 1 -- 172.21.15.6:0/417315940 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff4a8004d40 msgr2=0x7ff4a80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:19.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:19.414+0000 7ff4c4c00640 1 --2- 172.21.15.6:0/417315940 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff4a8004d40 0x7ff4a80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:19.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:19.414+0000 7ff4c4c00640 1 -- 172.21.15.6:0/417315940 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4a8003520 con 0x7ff4a8003f60 2024-03-04T09:30:19.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:19.414+0000 7ff4c6000640 1 --2- 172.21.15.6:0/417315940 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff4a8004d40 0x7ff4a80051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:30:19.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:19.415+0000 7ff4c4c00640 1 --2- 172.21.15.6:0/417315940 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff4a8003f60 0x7ff4a8004800 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7ff4ac001750 tx=0x7ff4a8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e84f2e122b7a47a1 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:19.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:19.415+0000 7ff4bfe00640 1 -- 172.21.15.6:0/417315940 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4ac001370 con 0x7ff4a8003f60 2024-03-04T09:30:19.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:19.415+0000 7ff4bfe00640 1 -- 172.21.15.6:0/417315940 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4ac0052d0 con 0x7ff4a8003f60 2024-03-04T09:30:19.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:19.415+0000 7ff4bfe00640 1 -- 172.21.15.6:0/417315940 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4ac003830 con 0x7ff4a8003f60 2024-03-04T09:30:19.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:19.415+0000 7ff4c8000640 1 -- 172.21.15.6:0/417315940 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff4a8003f60 msgr2=0x7ff4a8004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:19.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:19.415+0000 7ff4c8000640 1 --2- 172.21.15.6:0/417315940 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff4a8003f60 0x7ff4a8004800 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7ff4ac001750 tx=0x7ff4a8005830 comp rx=0 tx=0).stop 2024-03-04T09:30:19.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:19.416+0000 7ff4c6000640 1 -- 172.21.15.6:0/417315940 reap_dead start 2024-03-04T09:30:19.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:19.416+0000 7ff4c8000640 1 -- 172.21.15.6:0/417315940 shutdown_connections 2024-03-04T09:30:19.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:19.416+0000 7ff4c8000640 1 -- 172.21.15.6:0/417315940 >> 172.21.15.6:0/417315940 conn(0x7ff4c006b3d0 msgr2=0x7ff4c006b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:19.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:19.416+0000 7ff4c8000640 1 -- 172.21.15.6:0/417315940 shutdown_connections 2024-03-04T09:30:19.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:19.416+0000 7ff4c8000640 1 -- 172.21.15.6:0/417315940 wait complete. 2024-03-04T09:30:19.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:19.417+0000 7ff4c8000640 1 Processor -- start 2024-03-04T09:30:19.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:19.418+0000 7ff4c8000640 1 -- start start 2024-03-04T09:30:19.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:19.419+0000 7ff4c8000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff4a8003f60 0x7ff4c009d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:19.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:19.419+0000 7ff4c8000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff4c013e400 0x7ff4c01e60a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:19.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:19.419+0000 7ff4c8000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff4c013e990 0x7ff4c009aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:19.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:19.419+0000 7ff4c8000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4c015fdc0 con 0x7ff4c013e400 2024-03-04T09:30:19.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:19.419+0000 7ff4c8000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4c006d0b0 con 0x7ff4a8003f60 2024-03-04T09:30:19.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:19.419+0000 7ff4c8000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4c015c370 con 0x7ff4c013e990 2024-03-04T09:30:19.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:19.419+0000 7ff4c4c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff4c013e400 0x7ff4c01e60a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:19.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:19.419+0000 7ff4c4c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff4c013e400 0x7ff4c01e60a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38290/0 (socket says 172.21.15.6:38290) 2024-03-04T09:30:19.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:19.419+0000 7ff4c4c00640 1 -- 172.21.15.6:0/3173881976 learned_addr learned my addr 172.21.15.6:0/3173881976 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:30:19.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:19.419+0000 7ff4c5600640 1 --2- 172.21.15.6:0/3173881976 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff4a8003f60 0x7ff4c009d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:19.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:19.419+0000 7ff4c6000640 1 --2- 172.21.15.6:0/3173881976 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff4c013e990 0x7ff4c009aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:19.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:19.419+0000 7ff4c4c00640 1 -- 172.21.15.6:0/3173881976 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff4c013e990 msgr2=0x7ff4c009aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:19.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:19.419+0000 7ff4c4c00640 1 --2- 172.21.15.6:0/3173881976 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff4c013e990 0x7ff4c009aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:19.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:19.419+0000 7ff4c4c00640 1 -- 172.21.15.6:0/3173881976 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff4a8003f60 msgr2=0x7ff4c009d860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:19.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:19.419+0000 7ff4c4c00640 1 --2- 172.21.15.6:0/3173881976 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff4a8003f60 0x7ff4c009d860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:19.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:19.419+0000 7ff4c4c00640 1 -- 172.21.15.6:0/3173881976 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4c01e65e0 con 0x7ff4c013e400 2024-03-04T09:30:19.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:19.420+0000 7ff4c4c00640 1 --2- 172.21.15.6:0/3173881976 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff4c013e400 0x7ff4c01e60a0 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7ff4ac005040 tx=0x7ff4ac001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:19.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:19.420+0000 7ff4bd600640 1 -- 172.21.15.6:0/3173881976 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4ac041070 con 0x7ff4c013e400 2024-03-04T09:30:19.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:19.420+0000 7ff4bd600640 1 -- 172.21.15.6:0/3173881976 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4ac005480 con 0x7ff4c013e400 2024-03-04T09:30:19.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:19.420+0000 7ff4c8000640 1 -- 172.21.15.6:0/3173881976 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4c00cd900 con 0x7ff4c013e400 2024-03-04T09:30:19.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:19.420+0000 7ff4bd600640 1 -- 172.21.15.6:0/3173881976 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4ac001370 con 0x7ff4c013e400 2024-03-04T09:30:19.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:19.420+0000 7ff4c8000640 1 -- 172.21.15.6:0/3173881976 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4c00cdec0 con 0x7ff4c013e400 2024-03-04T09:30:19.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:19.422+0000 7ff4c8000640 1 -- 172.21.15.6:0/3173881976 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4c0068010 con 0x7ff4c013e400 2024-03-04T09:30:19.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:19.424+0000 7ff4bd600640 1 -- 172.21.15.6:0/3173881976 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff4ac044070 con 0x7ff4c013e400 2024-03-04T09:30:19.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:19.425+0000 7ff4bd600640 1 --2- 172.21.15.6:0/3173881976 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff4a006b4d0 0x7ff4a006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:19.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:19.425+0000 7ff4bd600640 1 -- 172.21.15.6:0/3173881976 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5227+0+0 (secure 0 0 0) 0x7ff4ac0ac070 con 0x7ff4c013e400 2024-03-04T09:30:19.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:19.426+0000 7ff4c5600640 1 --2- 172.21.15.6:0/3173881976 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff4a006b4d0 0x7ff4a006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:19.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:19.427+0000 7ff4c5600640 1 --2- 172.21.15.6:0/3173881976 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff4a006b4d0 0x7ff4a006d990 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7ff4b4009a50 tx=0x7ff4b4008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:19.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:19.429+0000 7ff4bd600640 1 -- 172.21.15.6:0/3173881976 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff4ac07b0c0 con 0x7ff4c013e400 2024-03-04T09:30:19.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:19.629+0000 7ff4c8000640 1 -- 172.21.15.6:0/3173881976 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7ff4c009de70 con 0x7ff4c013e400 2024-03-04T09:30:20.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:20.166+0000 7ff4bd600640 1 -- 172.21.15.6:0/3173881976 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v92) v1 ==== 170+0+0 (secure 0 0 0) 0x7ff4ac07a9c0 con 0x7ff4c013e400 2024-03-04T09:30:20.164 INFO:teuthology.orchestra.run.smithi006.stderr:pool 'cephfs_data' created 2024-03-04T09:30:20.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:20.170+0000 7ff4c8000640 1 -- 172.21.15.6:0/3173881976 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff4a006b4d0 msgr2=0x7ff4a006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:20.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:20.170+0000 7ff4c8000640 1 --2- 172.21.15.6:0/3173881976 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff4a006b4d0 0x7ff4a006d990 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7ff4b4009a50 tx=0x7ff4b4008040 comp rx=0 tx=0).stop 2024-03-04T09:30:20.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:20.171+0000 7ff4c8000640 1 -- 172.21.15.6:0/3173881976 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff4c013e400 msgr2=0x7ff4c01e60a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:20.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:20.171+0000 7ff4c8000640 1 --2- 172.21.15.6:0/3173881976 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff4c013e400 0x7ff4c01e60a0 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7ff4ac005040 tx=0x7ff4ac001120 comp rx=0 tx=0).stop 2024-03-04T09:30:20.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:20.171+0000 7ff4c8000640 1 -- 172.21.15.6:0/3173881976 shutdown_connections 2024-03-04T09:30:20.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:20.171+0000 7ff4c8000640 1 --2- 172.21.15.6:0/3173881976 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff4a006b4d0 0x7ff4a006d990 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:20.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:20.171+0000 7ff4c8000640 1 --2- 172.21.15.6:0/3173881976 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff4c013e990 0x7ff4c009aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:20.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:20.171+0000 7ff4c8000640 1 --2- 172.21.15.6:0/3173881976 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff4c013e400 0x7ff4c01e60a0 unknown :-1 s=CLOSED pgs=1062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:20.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:20.171+0000 7ff4c8000640 1 --2- 172.21.15.6:0/3173881976 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff4a8003f60 0x7ff4c009d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:20.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:20.171+0000 7ff4c8000640 1 -- 172.21.15.6:0/3173881976 >> 172.21.15.6:0/3173881976 conn(0x7ff4c006b3d0 msgr2=0x7ff4c00b1960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:20.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:20.171+0000 7ff4c8000640 1 -- 172.21.15.6:0/3173881976 shutdown_connections 2024-03-04T09:30:20.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:20.171+0000 7ff4c8000640 1 -- 172.21.15.6:0/3173881976 wait complete. 2024-03-04T09:30:20.205 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-04T09:30:20.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:20.396+0000 7fb38b000640 1 Processor -- start 2024-03-04T09:30:20.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:20.397+0000 7fb38b000640 1 -- start start 2024-03-04T09:30:20.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:20.399+0000 7fb38b000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb384162bb0 0x7fb384163040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:20.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:20.399+0000 7fb38b000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb384163950 con 0x7fb3840adb00 2024-03-04T09:30:20.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:20.399+0000 7fb38b000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3840acd40 con 0x7fb3840ccad0 2024-03-04T09:30:20.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:20.399+0000 7fb38b000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3840ad020 con 0x7fb384162bb0 2024-03-04T09:30:20.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:20.399+0000 7fb389000640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fb3840adb00 0x7fb3840adf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:55258/0 (socket says 172.21.15.6:55258) 2024-03-04T09:30:20.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:20.399+0000 7fb389000640 1 -- 172.21.15.6:0/1618967231 learned_addr learned my addr 172.21.15.6:0/1618967231 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:30:20.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:20.399+0000 7fb37be00640 1 --2- 172.21.15.6:0/1618967231 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb384162bb0 0x7fb384163040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:20.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:20.400+0000 7fb383400640 1 -- 172.21.15.6:0/1618967231 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2576579170 0 0) 0x7fb384163950 con 0x7fb3840adb00 2024-03-04T09:30:20.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:20.400+0000 7fb383400640 1 -- 172.21.15.6:0/1618967231 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb368003520 con 0x7fb3840adb00 2024-03-04T09:30:20.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:20.400+0000 7fb383400640 1 -- 172.21.15.6:0/1618967231 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1836953981 0 0) 0x7fb368003520 con 0x7fb3840adb00 2024-03-04T09:30:20.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:20.400+0000 7fb383400640 1 -- 172.21.15.6:0/1618967231 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb384162bb0 msgr2=0x7fb384163040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:20.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:20.400+0000 7fb383400640 1 --2- 172.21.15.6:0/1618967231 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb384162bb0 0x7fb384163040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:20.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:20.400+0000 7fb383400640 1 -- 172.21.15.6:0/1618967231 >> v1:172.21.15.146:6789/0 conn(0x7fb3840ccad0 legacy=0x7fb3840cceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:20.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:20.400+0000 7fb383400640 1 -- 172.21.15.6:0/1618967231 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3840ad300 con 0x7fb3840adb00 2024-03-04T09:30:20.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:20.400+0000 7fb383400640 1 -- 172.21.15.6:0/1618967231 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7fb37c0019f0 con 0x7fb3840adb00 2024-03-04T09:30:20.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:20.400+0000 7fb383400640 1 -- 172.21.15.6:0/1618967231 >> v1:172.21.15.6:6789/0 conn(0x7fb3840adb00 legacy=0x7fb3840adf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:20.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:20.400+0000 7fb383400640 1 --2- 172.21.15.6:0/1618967231 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb384162bb0 0x7fb3680039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:20.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:20.400+0000 7fb383400640 1 --2- 172.21.15.6:0/1618967231 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb368003f60 0x7fb368004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:20.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:20.400+0000 7fb383400640 1 --2- 172.21.15.6:0/1618967231 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb368004d40 0x7fb3680051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:20.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:20.400+0000 7fb383400640 1 -- 172.21.15.6:0/1618967231 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3840ad020 con 0x7fb368003f60 2024-03-04T09:30:20.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:20.400+0000 7fb383400640 1 -- 172.21.15.6:0/1618967231 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb36800baa0 con 0x7fb368004d40 2024-03-04T09:30:20.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:20.400+0000 7fb383400640 1 -- 172.21.15.6:0/1618967231 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb36800bca0 con 0x7fb384162bb0 2024-03-04T09:30:20.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:20.401+0000 7fb37be00640 1 --2- 172.21.15.6:0/1618967231 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb368003f60 0x7fb368004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:20.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:20.401+0000 7fb383e00640 1 --2- 172.21.15.6:0/1618967231 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb384162bb0 0x7fb3680039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:20.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:20.401+0000 7fb389000640 1 --2- 172.21.15.6:0/1618967231 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb368004d40 0x7fb3680051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:20.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:20.401+0000 7fb37be00640 1 -- 172.21.15.6:0/1618967231 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb384162bb0 msgr2=0x7fb3680039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:20.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:20.401+0000 7fb37be00640 1 --2- 172.21.15.6:0/1618967231 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb384162bb0 0x7fb3680039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:20.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:20.401+0000 7fb37be00640 1 -- 172.21.15.6:0/1618967231 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb368004d40 msgr2=0x7fb3680051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:20.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:20.401+0000 7fb37be00640 1 --2- 172.21.15.6:0/1618967231 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb368004d40 0x7fb3680051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:20.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:20.401+0000 7fb37be00640 1 -- 172.21.15.6:0/1618967231 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb368003520 con 0x7fb368003f60 2024-03-04T09:30:20.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:20.403+0000 7fb37be00640 1 --2- 172.21.15.6:0/1618967231 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb368003f60 0x7fb368004800 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7fb36c009ac0 tx=0x7fb368005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=743d0c5b56fa246d server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:20.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:20.403+0000 7fb383400640 1 -- 172.21.15.6:0/1618967231 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb36c039070 con 0x7fb368003f60 2024-03-04T09:30:20.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:20.403+0000 7fb383400640 1 -- 172.21.15.6:0/1618967231 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb36c02c6b0 con 0x7fb368003f60 2024-03-04T09:30:20.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:20.403+0000 7fb383400640 1 -- 172.21.15.6:0/1618967231 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb36c009690 con 0x7fb368003f60 2024-03-04T09:30:20.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:20.403+0000 7fb38b000640 1 -- 172.21.15.6:0/1618967231 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb368003f60 msgr2=0x7fb368004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:20.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:20.403+0000 7fb38b000640 1 --2- 172.21.15.6:0/1618967231 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb368003f60 0x7fb368004800 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7fb36c009ac0 tx=0x7fb368005830 comp rx=0 tx=0).stop 2024-03-04T09:30:20.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:20.403+0000 7fb389000640 1 -- 172.21.15.6:0/1618967231 reap_dead start 2024-03-04T09:30:20.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:20.404+0000 7fb38b000640 1 -- 172.21.15.6:0/1618967231 shutdown_connections 2024-03-04T09:30:20.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:20.404+0000 7fb38b000640 1 -- 172.21.15.6:0/1618967231 >> 172.21.15.6:0/1618967231 conn(0x7fb38406b3d0 msgr2=0x7fb38406b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:20.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:20.404+0000 7fb38b000640 1 -- 172.21.15.6:0/1618967231 shutdown_connections 2024-03-04T09:30:20.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:20.404+0000 7fb38b000640 1 -- 172.21.15.6:0/1618967231 wait complete. 2024-03-04T09:30:20.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:20.405+0000 7fb38b000640 1 Processor -- start 2024-03-04T09:30:20.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:20.406+0000 7fb38b000640 1 -- start start 2024-03-04T09:30:20.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:20.407+0000 7fb38b000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb368003f60 0x7fb38415b770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:20.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:20.407+0000 7fb38b000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb38415cdf0 0x7fb3840b1940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:20.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:20.407+0000 7fb38b000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb384165200 0x7fb38414e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:20.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:20.407+0000 7fb38b000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb38406c980 con 0x7fb38415cdf0 2024-03-04T09:30:20.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:20.407+0000 7fb38b000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb38406e7d0 con 0x7fb368003f60 2024-03-04T09:30:20.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:20.407+0000 7fb38b000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3840ae660 con 0x7fb384165200 2024-03-04T09:30:20.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:20.407+0000 7fb37be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb38415cdf0 0x7fb3840b1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:20.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:20.407+0000 7fb37be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb38415cdf0 0x7fb3840b1940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38304/0 (socket says 172.21.15.6:38304) 2024-03-04T09:30:20.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:20.407+0000 7fb37be00640 1 -- 172.21.15.6:0/912528460 learned_addr learned my addr 172.21.15.6:0/912528460 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:30:20.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:20.407+0000 7fb389000640 1 --2- 172.21.15.6:0/912528460 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb384165200 0x7fb38414e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:20.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:20.407+0000 7fb383e00640 1 --2- 172.21.15.6:0/912528460 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb368003f60 0x7fb38415b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:20.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:20.407+0000 7fb37be00640 1 -- 172.21.15.6:0/912528460 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb384165200 msgr2=0x7fb38414e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:20.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:20.407+0000 7fb37be00640 1 --2- 172.21.15.6:0/912528460 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb384165200 0x7fb38414e8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:20.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:20.407+0000 7fb37be00640 1 -- 172.21.15.6:0/912528460 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb368003f60 msgr2=0x7fb38415b770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:20.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:20.407+0000 7fb37be00640 1 --2- 172.21.15.6:0/912528460 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb368003f60 0x7fb38415b770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:20.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:20.407+0000 7fb37be00640 1 -- 172.21.15.6:0/912528460 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb38415bd50 con 0x7fb38415cdf0 2024-03-04T09:30:20.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:20.408+0000 7fb37be00640 1 --2- 172.21.15.6:0/912528460 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb38415cdf0 0x7fb3840b1940 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7fb36c009ac0 tx=0x7fb36c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:20.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:20.408+0000 7fb380c00640 1 -- 172.21.15.6:0/912528460 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb36c039050 con 0x7fb38415cdf0 2024-03-04T09:30:20.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:20.408+0000 7fb380c00640 1 -- 172.21.15.6:0/912528460 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb36c045070 con 0x7fb38415cdf0 2024-03-04T09:30:20.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:20.408+0000 7fb380c00640 1 -- 172.21.15.6:0/912528460 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb36c0044e0 con 0x7fb38415cdf0 2024-03-04T09:30:20.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:20.408+0000 7fb38b000640 1 -- 172.21.15.6:0/912528460 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb38415c030 con 0x7fb38415cdf0 2024-03-04T09:30:20.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:20.408+0000 7fb38b000640 1 -- 172.21.15.6:0/912528460 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb38415c5f0 con 0x7fb38415cdf0 2024-03-04T09:30:20.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:20.410+0000 7fb38b000640 1 -- 172.21.15.6:0/912528460 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb38404f330 con 0x7fb38415cdf0 2024-03-04T09:30:20.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:20.411+0000 7fb380c00640 1 -- 172.21.15.6:0/912528460 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb36c003850 con 0x7fb38415cdf0 2024-03-04T09:30:20.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:20.412+0000 7fb380c00640 1 --2- 172.21.15.6:0/912528460 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb36006b500 0x7fb36006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:20.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:20.412+0000 7fb380c00640 1 -- 172.21.15.6:0/912528460 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5609+0+0 (secure 0 0 0) 0x7fb36c0abe10 con 0x7fb38415cdf0 2024-03-04T09:30:20.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:20.413+0000 7fb383e00640 1 --2- 172.21.15.6:0/912528460 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb36006b500 0x7fb36006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:20.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:20.414+0000 7fb383e00640 1 --2- 172.21.15.6:0/912528460 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb36006b500 0x7fb36006d9c0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fb3740035f0 tx=0x7fb374009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:20.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:20.417+0000 7fb380c00640 1 -- 172.21.15.6:0/912528460 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb36c0af030 con 0x7fb38415cdf0 2024-03-04T09:30:20.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:20.657+0000 7fb38b000640 1 -- 172.21.15.6:0/912528460 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fb38406e010 con 0x7fb38415cdf0 2024-03-04T09:30:21.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.188+0000 7fb380c00640 1 -- 172.21.15.6:0/912528460 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 14 and data pool 15 v56) v1 ==== 174+0+0 (secure 0 0 0) 0x7fb36c04e020 con 0x7fb38415cdf0 2024-03-04T09:30:21.186 INFO:teuthology.orchestra.run.smithi006.stderr:new fs with metadata pool 14 and data pool 15 2024-03-04T09:30:21.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.199+0000 7fb38b000640 1 -- 172.21.15.6:0/912528460 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb36006b500 msgr2=0x7fb36006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:21.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.199+0000 7fb38b000640 1 --2- 172.21.15.6:0/912528460 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb36006b500 0x7fb36006d9c0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fb3740035f0 tx=0x7fb374009040 comp rx=0 tx=0).stop 2024-03-04T09:30:21.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.199+0000 7fb38b000640 1 -- 172.21.15.6:0/912528460 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb38415cdf0 msgr2=0x7fb3840b1940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:21.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.199+0000 7fb38b000640 1 --2- 172.21.15.6:0/912528460 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb38415cdf0 0x7fb3840b1940 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7fb36c009ac0 tx=0x7fb36c004730 comp rx=0 tx=0).stop 2024-03-04T09:30:21.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.199+0000 7fb38b000640 1 -- 172.21.15.6:0/912528460 shutdown_connections 2024-03-04T09:30:21.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.199+0000 7fb38b000640 1 --2- 172.21.15.6:0/912528460 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb36006b500 0x7fb36006d9c0 secure :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fb3740035f0 tx=0x7fb374009040 comp rx=0 tx=0).stop 2024-03-04T09:30:21.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.199+0000 7fb38b000640 1 --2- 172.21.15.6:0/912528460 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb384165200 0x7fb38414e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:21.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.199+0000 7fb38b000640 1 --2- 172.21.15.6:0/912528460 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb38415cdf0 0x7fb3840b1940 unknown :-1 s=CLOSED pgs=1065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:21.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.199+0000 7fb38b000640 1 --2- 172.21.15.6:0/912528460 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb368003f60 0x7fb38415b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:21.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.199+0000 7fb38b000640 1 -- 172.21.15.6:0/912528460 >> 172.21.15.6:0/912528460 conn(0x7fb38406b3d0 msgr2=0x7fb38406b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:21.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.203+0000 7fb38b000640 1 -- 172.21.15.6:0/912528460 shutdown_connections 2024-03-04T09:30:21.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.204+0000 7fb38b000640 1 -- 172.21.15.6:0/912528460 wait complete. 2024-03-04T09:30:21.234 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T09:30:21.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.412+0000 7f25be200640 1 Processor -- start 2024-03-04T09:30:21.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.412+0000 7f25be200640 1 -- start start 2024-03-04T09:30:21.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.414+0000 7f25be200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f25b815efc0 0x7f25b80cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:21.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.414+0000 7f25be200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f25b81621d0 0x7f25b8070c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:21.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.414+0000 7f25be200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25b81627f0 con 0x7f25b80cce90 2024-03-04T09:30:21.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.414+0000 7f25be200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25b815f3a0 con 0x7f25b81621d0 2024-03-04T09:30:21.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.414+0000 7f25be200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25b806d190 con 0x7f25b815efc0 2024-03-04T09:30:21.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.415+0000 7f25bd800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f25b81621d0 0x7f25b8070c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:21.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.415+0000 7f25bd800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f25b81621d0 0x7f25b8070c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.6:36986/0 (socket says 172.21.15.6:36986) 2024-03-04T09:30:21.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.415+0000 7f25bd800640 1 -- 172.21.15.6:0/2290648727 learned_addr learned my addr 172.21.15.6:0/2290648727 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:30:21.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.415+0000 7f25bce00640 1 --2- 172.21.15.6:0/2290648727 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f25b815efc0 0x7f25b80cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:21.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.416+0000 7f25bd800640 1 -- 172.21.15.6:0/2290648727 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f25b815efc0 msgr2=0x7f25b80cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:21.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.416+0000 7f25bd800640 1 --2- 172.21.15.6:0/2290648727 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f25b815efc0 0x7f25b80cc950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:21.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.416+0000 7f25bd800640 1 -- 172.21.15.6:0/2290648727 >> v1:172.21.15.146:6789/0 conn(0x7f25b80cce90 legacy=0x7f25b8161ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:21.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.416+0000 7f25bd800640 1 -- 172.21.15.6:0/2290648727 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25b806c860 con 0x7f25b81621d0 2024-03-04T09:30:21.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.417+0000 7f25bd800640 1 --2- 172.21.15.6:0/2290648727 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f25b81621d0 0x7f25b8070c20 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f25b0009e50 tx=0x7f25b002bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9a6899cda8f3497a server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:21.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.417+0000 7f25b7e00640 1 -- 172.21.15.6:0/2290648727 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25b002cb00 con 0x7f25b81621d0 2024-03-04T09:30:21.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.417+0000 7f25b7e00640 1 -- 172.21.15.6:0/2290648727 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25b002cca0 con 0x7f25b81621d0 2024-03-04T09:30:21.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.417+0000 7f25b7e00640 1 -- 172.21.15.6:0/2290648727 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25b0034610 con 0x7f25b81621d0 2024-03-04T09:30:21.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.418+0000 7f25be200640 1 -- 172.21.15.6:0/2290648727 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f25b81621d0 msgr2=0x7f25b8070c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:21.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.418+0000 7f25be200640 1 --2- 172.21.15.6:0/2290648727 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f25b81621d0 0x7f25b8070c20 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f25b0009e50 tx=0x7f25b002bf60 comp rx=0 tx=0).stop 2024-03-04T09:30:21.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.418+0000 7f25be200640 1 -- 172.21.15.6:0/2290648727 shutdown_connections 2024-03-04T09:30:21.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.418+0000 7f25be200640 1 --2- 172.21.15.6:0/2290648727 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f25b81621d0 0x7f25b8070c20 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:21.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.418+0000 7f25be200640 1 --2- 172.21.15.6:0/2290648727 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f25b815efc0 0x7f25b80cc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:21.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.418+0000 7f25be200640 1 -- 172.21.15.6:0/2290648727 >> 172.21.15.6:0/2290648727 conn(0x7f25b806b3d0 msgr2=0x7f25b806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:21.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.418+0000 7f25be200640 1 -- 172.21.15.6:0/2290648727 shutdown_connections 2024-03-04T09:30:21.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.418+0000 7f25be200640 1 -- 172.21.15.6:0/2290648727 wait complete. 2024-03-04T09:30:21.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.420+0000 7f25be200640 1 Processor -- start 2024-03-04T09:30:21.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.420+0000 7f25be200640 1 -- start start 2024-03-04T09:30:21.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.421+0000 7f25be200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f25b80cce90 0x7f25b813a350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:21.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.421+0000 7f25be200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f25b815efc0 0x7f25b81601a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:21.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.421+0000 7f25be200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f25b81621d0 0x7f25b8146550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:21.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.421+0000 7f25be200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25b806d620 con 0x7f25b815efc0 2024-03-04T09:30:21.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.421+0000 7f25be200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25b806ce20 con 0x7f25b81621d0 2024-03-04T09:30:21.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.421+0000 7f25be200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25b8159ae0 con 0x7f25b80cce90 2024-03-04T09:30:21.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.421+0000 7f25afe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f25b815efc0 0x7f25b81601a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:21.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.421+0000 7f25afe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f25b815efc0 0x7f25b81601a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38312/0 (socket says 172.21.15.6:38312) 2024-03-04T09:30:21.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.421+0000 7f25afe00640 1 -- 172.21.15.6:0/1861537128 learned_addr learned my addr 172.21.15.6:0/1861537128 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:30:21.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.422+0000 7f25bd800640 1 --2- 172.21.15.6:0/1861537128 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f25b81621d0 0x7f25b8146550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:21.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.422+0000 7f25afe00640 1 -- 172.21.15.6:0/1861537128 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f25b80cce90 msgr2=0x7f25b813a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:21.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.422+0000 7f25bce00640 1 --2- 172.21.15.6:0/1861537128 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f25b80cce90 0x7f25b813a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:21.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.422+0000 7f25afe00640 1 --2- 172.21.15.6:0/1861537128 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f25b80cce90 0x7f25b813a350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:21.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.422+0000 7f25afe00640 1 -- 172.21.15.6:0/1861537128 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f25b81621d0 msgr2=0x7f25b8146550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:21.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.422+0000 7f25afe00640 1 --2- 172.21.15.6:0/1861537128 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f25b81621d0 0x7f25b8146550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:21.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.422+0000 7f25afe00640 1 -- 172.21.15.6:0/1861537128 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25b8159800 con 0x7f25b815efc0 2024-03-04T09:30:21.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.422+0000 7f25bd800640 1 --2- 172.21.15.6:0/1861537128 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f25b81621d0 0x7f25b8146550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:30:21.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.422+0000 7f25afe00640 1 --2- 172.21.15.6:0/1861537128 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f25b815efc0 0x7f25b81601a0 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f25a000c400 tx=0x7f25a000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:21.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.422+0000 7f25b5600640 1 -- 172.21.15.6:0/1861537128 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25a00052d0 con 0x7f25b815efc0 2024-03-04T09:30:21.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.422+0000 7f25b5600640 1 -- 172.21.15.6:0/1861537128 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25a0005490 con 0x7f25b815efc0 2024-03-04T09:30:21.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.422+0000 7f25b5600640 1 -- 172.21.15.6:0/1861537128 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25a0007260 con 0x7f25b815efc0 2024-03-04T09:30:21.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.422+0000 7f25be200640 1 -- 172.21.15.6:0/1861537128 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25b815f6c0 con 0x7f25b815efc0 2024-03-04T09:30:21.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.422+0000 7f25be200640 1 -- 172.21.15.6:0/1861537128 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25b815fc80 con 0x7f25b815efc0 2024-03-04T09:30:21.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.423+0000 7f25be200640 1 -- 172.21.15.6:0/1861537128 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25b804f330 con 0x7f25b815efc0 2024-03-04T09:30:21.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.425+0000 7f25b5600640 1 -- 172.21.15.6:0/1861537128 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f25a0007400 con 0x7f25b815efc0 2024-03-04T09:30:21.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.425+0000 7f25b5600640 1 --2- 172.21.15.6:0/1861537128 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f259406b500 0x7f259406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:21.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.426+0000 7f25b5600640 1 -- 172.21.15.6:0/1861537128 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f25a0089780 con 0x7f25b815efc0 2024-03-04T09:30:21.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.426+0000 7f25bce00640 1 --2- 172.21.15.6:0/1861537128 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f259406b500 0x7f259406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:21.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.427+0000 7f25bce00640 1 --2- 172.21.15.6:0/1861537128 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f259406b500 0x7f259406d9c0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f25a8009a10 tx=0x7f25a8008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:21.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.430+0000 7f25b5600640 1 -- 172.21.15.6:0/1861537128 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f25a0058560 con 0x7f25b815efc0 2024-03-04T09:30:21.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.627+0000 7f25be200640 1 -- 172.21.15.6:0/1861537128 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f25b80af2f0 con 0x7f25b815efc0 2024-03-04T09:30:21.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.630+0000 7f25b5600640 1 -- 172.21.15.6:0/1861537128 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) v1 ==== 74+0+16573 (secure 0 0 0) 0x7f25a0057e60 con 0x7f25b815efc0 2024-03-04T09:30:21.627 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:30:21.627 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":93,"fsid":"edce0578-5ae4-4402-b45a-552bdef84d7c","created":"2024-03-04T09:24:54.523780+0000","modified":"2024-03-04T09:30:20.659466+0000","last_up_change":"2024-03-04T09:25:07.089049+0000","last_in_change":"2024-03-04T09:24:57.982077+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T09:25:07.935198+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-03-04T09:30:18.510767+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"93","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-03-04T09:30:19.686114+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"93","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a96574f-81f1-4bc6-8463-757520b84a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6825","nonce":1364449401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6827","nonce":1364449401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6831","nonce":1364449401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6829","nonce":1364449401}]},"public_addr":"172.21.15.6:6825/1364449401","cluster_addr":"172.21.15.6:6827/1364449401","heartbeat_back_addr":"172.21.15.6:6831/1364449401","heartbeat_front_addr":"172.21.15.6:6829/1364449401","state":["exists","up"]},{"osd":1,"uuid":"7d79b9c1-6b87-476a-bc36-7db5ee64cd90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6809","nonce":2494103078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6811","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6813","nonce":2494103078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6821","nonce":2494103078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6817","nonce":2494103078}]},"public_addr":"172.21.15.6:6809/2494103078","cluster_addr":"172.21.15.6:6813/2494103078","heartbeat_back_addr":"172.21.15.6:6821/2494103078","heartbeat_front_addr":"172.21.15.6:6817/2494103078","state":["exists","up"]},{"osd":2,"uuid":"492331b9-4ee9-40b6-bacd-85d44cd6c45a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6812","nonce":1500066681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6816","nonce":1500066681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6824","nonce":1500066681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6820","nonce":1500066681}]},"public_addr":"172.21.15.6:6812/1500066681","cluster_addr":"172.21.15.6:6816/1500066681","heartbeat_back_addr":"172.21.15.6:6824/1500066681","heartbeat_front_addr":"172.21.15.6:6820/1500066681","state":["exists","up"]},{"osd":3,"uuid":"21bb2efe-9ba7-4bdc-bc31-180cd1102e21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6801","nonce":1558806123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6803","nonce":1558806123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6807","nonce":1558806123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6805","nonce":1558806123}]},"public_addr":"172.21.15.6:6801/1558806123","cluster_addr":"172.21.15.6:6803/1558806123","heartbeat_back_addr":"172.21.15.6:6807/1558806123","heartbeat_front_addr":"172.21.15.6:6805/1558806123","state":["exists","up"]},{"osd":4,"uuid":"6e2c4b71-19d9-4737-83d9-558705d8644a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6801","nonce":1526135780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6803","nonce":1526135780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6807","nonce":1526135780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6805","nonce":1526135780}]},"public_addr":"172.21.15.146:6801/1526135780","cluster_addr":"172.21.15.146:6803/1526135780","heartbeat_back_addr":"172.21.15.146:6807/1526135780","heartbeat_front_addr":"172.21.15.146:6805/1526135780","state":["exists","up"]},{"osd":5,"uuid":"fb69c525-f1be-4b7d-ab32-6c29b0d6fddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6809","nonce":460262507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6811","nonce":460262507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6815","nonce":460262507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6813","nonce":460262507}]},"public_addr":"172.21.15.146:6809/460262507","cluster_addr":"172.21.15.146:6811/460262507","heartbeat_back_addr":"172.21.15.146:6815/460262507","heartbeat_front_addr":"172.21.15.146:6813/460262507","state":["exists","up"]},{"osd":6,"uuid":"c32e7fed-370d-4401-8ce0-6a4335b0d168","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6823","nonce":1392897493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6827","nonce":1392897493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6831","nonce":1392897493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6829","nonce":1392897493}]},"public_addr":"172.21.15.146:6823/1392897493","cluster_addr":"172.21.15.146:6827/1392897493","heartbeat_back_addr":"172.21.15.146:6831/1392897493","heartbeat_front_addr":"172.21.15.146:6829/1392897493","state":["exists","up"]},{"osd":7,"uuid":"75a79b44-fa5f-426d-a7d3-71f21f78835c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6817","nonce":3268323156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6819","nonce":3268323156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6825","nonce":3268323156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6822","nonce":3268323156}]},"public_addr":"172.21.15.146:6817/3268323156","cluster_addr":"172.21.15.146:6819/3268323156","heartbeat_back_addr":"172.21.15.146:6825/3268323156","heartbeat_front_addr":"172.21.15.146:6822/3268323156","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.310437+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.295850+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.294674+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.526855+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.300023+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.595160+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.346046+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.558741+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T09:30:21.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.633+0000 7f25be200640 1 -- 172.21.15.6:0/1861537128 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f259406b500 msgr2=0x7f259406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:21.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.633+0000 7f25be200640 1 --2- 172.21.15.6:0/1861537128 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f259406b500 0x7f259406d9c0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f25a8009a10 tx=0x7f25a8008040 comp rx=0 tx=0).stop 2024-03-04T09:30:21.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.633+0000 7f25be200640 1 -- 172.21.15.6:0/1861537128 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f25b815efc0 msgr2=0x7f25b81601a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:21.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.633+0000 7f25be200640 1 --2- 172.21.15.6:0/1861537128 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f25b815efc0 0x7f25b81601a0 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f25a000c400 tx=0x7f25a000c8c0 comp rx=0 tx=0).stop 2024-03-04T09:30:21.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.633+0000 7f25be200640 1 -- 172.21.15.6:0/1861537128 shutdown_connections 2024-03-04T09:30:21.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.633+0000 7f25be200640 1 --2- 172.21.15.6:0/1861537128 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f25b81621d0 0x7f25b8146550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:21.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.633+0000 7f25be200640 1 --2- 172.21.15.6:0/1861537128 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f25b815efc0 0x7f25b81601a0 unknown :-1 s=CLOSED pgs=1066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:21.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.633+0000 7f25be200640 1 --2- 172.21.15.6:0/1861537128 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f259406b500 0x7f259406d9c0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:21.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.633+0000 7f25be200640 1 --2- 172.21.15.6:0/1861537128 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f25b80cce90 0x7f25b813a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:21.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.633+0000 7f25be200640 1 -- 172.21.15.6:0/1861537128 >> 172.21.15.6:0/1861537128 conn(0x7f25b806b3d0 msgr2=0x7f25b80b0830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:21.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.633+0000 7f25be200640 1 -- 172.21.15.6:0/1861537128 shutdown_connections 2024-03-04T09:30:21.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.633+0000 7f25be200640 1 -- 172.21.15.6:0/1861537128 wait complete. 2024-03-04T09:30:21.649 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T09:30:21.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.876+0000 7f1b2a800640 1 Processor -- start 2024-03-04T09:30:21.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.876+0000 7f1b2a800640 1 -- start start 2024-03-04T09:30:21.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.877+0000 7f1b2a800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f1b240d2930 0x7f1b240b3740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:21.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.877+0000 7f1b2a800640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b240cea50 con 0x7f1b240cfd90 2024-03-04T09:30:21.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.878+0000 7f1b2a800640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b240cecf0 con 0x7f1b24161b40 2024-03-04T09:30:21.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.878+0000 7f1b2a800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b240b5770 con 0x7f1b240d2930 2024-03-04T09:30:21.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.878+0000 7f1b23e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f1b240d2930 0x7f1b240b3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:21.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.878+0000 7f1b22a00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f1b240cfd90 0x7f1b240d2220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:55274/0 (socket says 172.21.15.6:55274) 2024-03-04T09:30:21.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.878+0000 7f1b23e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f1b240d2930 0x7f1b240b3740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38316/0 (socket says 172.21.15.6:38316) 2024-03-04T09:30:21.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.878+0000 7f1b22a00640 1 -- 172.21.15.6:0/1834395541 learned_addr learned my addr 172.21.15.6:0/1834395541 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:30:21.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.878+0000 7f1b23e00640 1 -- 172.21.15.6:0/1834395541 >> v1:172.21.15.146:6789/0 conn(0x7f1b24161b40 legacy=0x7f1b240b5060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:21.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.878+0000 7f1b23e00640 1 -- 172.21.15.6:0/1834395541 >> v1:172.21.15.6:6789/0 conn(0x7f1b240cfd90 legacy=0x7f1b240d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:21.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.878+0000 7f1b23e00640 1 -- 172.21.15.6:0/1834395541 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b240cefd0 con 0x7f1b240d2930 2024-03-04T09:30:21.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.880+0000 7f1b23e00640 1 --2- 172.21.15.6:0/1834395541 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1b240d2930 0x7f1b240b3740 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7f1b1c0097a0 tx=0x7f1b1c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27d2d8d067e609eb server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:21.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.880+0000 7f1b22000640 1 -- 172.21.15.6:0/1834395541 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b1c035020 con 0x7f1b240d2930 2024-03-04T09:30:21.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.880+0000 7f1b22000640 1 -- 172.21.15.6:0/1834395541 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b1c031830 con 0x7f1b240d2930 2024-03-04T09:30:21.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.880+0000 7f1b22000640 1 -- 172.21.15.6:0/1834395541 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b1c031c00 con 0x7f1b240d2930 2024-03-04T09:30:21.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.881+0000 7f1b2a800640 1 -- 172.21.15.6:0/1834395541 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1b240d2930 msgr2=0x7f1b240b3740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:21.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.881+0000 7f1b2a800640 1 --2- 172.21.15.6:0/1834395541 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1b240d2930 0x7f1b240b3740 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7f1b1c0097a0 tx=0x7f1b1c02b8d0 comp rx=0 tx=0).stop 2024-03-04T09:30:21.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.881+0000 7f1b2a800640 1 -- 172.21.15.6:0/1834395541 shutdown_connections 2024-03-04T09:30:21.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.881+0000 7f1b2a800640 1 --2- 172.21.15.6:0/1834395541 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1b240d2930 0x7f1b240b3740 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:21.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.881+0000 7f1b2a800640 1 -- 172.21.15.6:0/1834395541 >> 172.21.15.6:0/1834395541 conn(0x7f1b2409d220 msgr2=0x7f1b240d3ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:21.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.881+0000 7f1b2a800640 1 -- 172.21.15.6:0/1834395541 shutdown_connections 2024-03-04T09:30:21.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.881+0000 7f1b2a800640 1 -- 172.21.15.6:0/1834395541 wait complete. 2024-03-04T09:30:21.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.883+0000 7f1b2a800640 1 Processor -- start 2024-03-04T09:30:21.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.883+0000 7f1b2a800640 1 -- start start 2024-03-04T09:30:21.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.884+0000 7f1b2a800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1b240cfd90 0x7f1b24158a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:21.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.884+0000 7f1b2a800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1b240d2930 0x7f1b240b6b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:21.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.884+0000 7f1b2a800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1b24161b40 0x7f1b240cd380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:21.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.884+0000 7f1b2a800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b240cf590 con 0x7f1b240cfd90 2024-03-04T09:30:21.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.884+0000 7f1b2a800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b240cf900 con 0x7f1b24161b40 2024-03-04T09:30:21.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.884+0000 7f1b2a800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b240b2ff0 con 0x7f1b240d2930 2024-03-04T09:30:21.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.884+0000 7f1b23400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1b240cfd90 0x7f1b24158a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:21.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.884+0000 7f1b23400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1b240cfd90 0x7f1b24158a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38326/0 (socket says 172.21.15.6:38326) 2024-03-04T09:30:21.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.884+0000 7f1b23400640 1 -- 172.21.15.6:0/3949935376 learned_addr learned my addr 172.21.15.6:0/3949935376 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:30:21.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.884+0000 7f1b23400640 1 -- 172.21.15.6:0/3949935376 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1b240d2930 msgr2=0x7f1b240b6b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:21.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.884+0000 7f1b23400640 1 --2- 172.21.15.6:0/3949935376 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1b240d2930 0x7f1b240b6b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:21.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.884+0000 7f1b23400640 1 -- 172.21.15.6:0/3949935376 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1b24161b40 msgr2=0x7f1b240cd380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:21.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.884+0000 7f1b23400640 1 --2- 172.21.15.6:0/3949935376 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1b24161b40 0x7f1b240cd380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:21.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.884+0000 7f1b23400640 1 -- 172.21.15.6:0/3949935376 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b240c8660 con 0x7f1b240cfd90 2024-03-04T09:30:21.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.885+0000 7f1b23400640 1 --2- 172.21.15.6:0/3949935376 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1b240cfd90 0x7f1b24158a20 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f1b1400c7f0 tx=0x7f1b1400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:21.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.885+0000 7f1b1b400640 1 -- 172.21.15.6:0/3949935376 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b14016020 con 0x7f1b240cfd90 2024-03-04T09:30:21.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.885+0000 7f1b1b400640 1 -- 172.21.15.6:0/3949935376 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b1400aa50 con 0x7f1b240cfd90 2024-03-04T09:30:21.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.885+0000 7f1b1b400640 1 -- 172.21.15.6:0/3949935376 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b1400ae40 con 0x7f1b240cfd90 2024-03-04T09:30:21.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.885+0000 7f1b2a800640 1 -- 172.21.15.6:0/3949935376 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b240c8940 con 0x7f1b240cfd90 2024-03-04T09:30:21.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.885+0000 7f1b2a800640 1 -- 172.21.15.6:0/3949935376 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b24158f60 con 0x7f1b240cfd90 2024-03-04T09:30:21.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.886+0000 7f1b2a800640 1 -- 172.21.15.6:0/3949935376 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b2404f3a0 con 0x7f1b240cfd90 2024-03-04T09:30:21.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.887+0000 7f1b1b400640 1 -- 172.21.15.6:0/3949935376 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1b14019020 con 0x7f1b240cfd90 2024-03-04T09:30:21.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.888+0000 7f1b1b400640 1 --2- 172.21.15.6:0/3949935376 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f1b0006b520 0x7f1b0006d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:21.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.888+0000 7f1b1b400640 1 -- 172.21.15.6:0/3949935376 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1b1408a330 con 0x7f1b240cfd90 2024-03-04T09:30:21.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.889+0000 7f1b22a00640 1 --2- 172.21.15.6:0/3949935376 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f1b0006b520 0x7f1b0006d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:21.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.890+0000 7f1b22a00640 1 --2- 172.21.15.6:0/3949935376 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f1b0006b520 0x7f1b0006d9e0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f1b0c000f60 tx=0x7f1b0c007a80 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:21.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:21.892+0000 7f1b1b400640 1 -- 172.21.15.6:0/3949935376 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1b14020030 con 0x7f1b240cfd90 2024-03-04T09:30:22.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:22.087+0000 7f1b2a800640 1 -- 172.21.15.6:0/3949935376 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1b240d1df0 con 0x7f1b240cfd90 2024-03-04T09:30:22.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:22.090+0000 7f1b1b400640 1 -- 172.21.15.6:0/3949935376 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) v1 ==== 74+0+16573 (secure 0 0 0) 0x7f1b14059100 con 0x7f1b240cfd90 2024-03-04T09:30:22.087 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:30:22.087 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":93,"fsid":"edce0578-5ae4-4402-b45a-552bdef84d7c","created":"2024-03-04T09:24:54.523780+0000","modified":"2024-03-04T09:30:20.659466+0000","last_up_change":"2024-03-04T09:25:07.089049+0000","last_in_change":"2024-03-04T09:24:57.982077+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T09:25:07.935198+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-03-04T09:30:18.510767+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"93","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-03-04T09:30:19.686114+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"93","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a96574f-81f1-4bc6-8463-757520b84a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6825","nonce":1364449401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6827","nonce":1364449401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6831","nonce":1364449401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6829","nonce":1364449401}]},"public_addr":"172.21.15.6:6825/1364449401","cluster_addr":"172.21.15.6:6827/1364449401","heartbeat_back_addr":"172.21.15.6:6831/1364449401","heartbeat_front_addr":"172.21.15.6:6829/1364449401","state":["exists","up"]},{"osd":1,"uuid":"7d79b9c1-6b87-476a-bc36-7db5ee64cd90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6809","nonce":2494103078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6811","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6813","nonce":2494103078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6821","nonce":2494103078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6817","nonce":2494103078}]},"public_addr":"172.21.15.6:6809/2494103078","cluster_addr":"172.21.15.6:6813/2494103078","heartbeat_back_addr":"172.21.15.6:6821/2494103078","heartbeat_front_addr":"172.21.15.6:6817/2494103078","state":["exists","up"]},{"osd":2,"uuid":"492331b9-4ee9-40b6-bacd-85d44cd6c45a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6812","nonce":1500066681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6816","nonce":1500066681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6824","nonce":1500066681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6820","nonce":1500066681}]},"public_addr":"172.21.15.6:6812/1500066681","cluster_addr":"172.21.15.6:6816/1500066681","heartbeat_back_addr":"172.21.15.6:6824/1500066681","heartbeat_front_addr":"172.21.15.6:6820/1500066681","state":["exists","up"]},{"osd":3,"uuid":"21bb2efe-9ba7-4bdc-bc31-180cd1102e21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6801","nonce":1558806123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6803","nonce":1558806123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6807","nonce":1558806123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6805","nonce":1558806123}]},"public_addr":"172.21.15.6:6801/1558806123","cluster_addr":"172.21.15.6:6803/1558806123","heartbeat_back_addr":"172.21.15.6:6807/1558806123","heartbeat_front_addr":"172.21.15.6:6805/1558806123","state":["exists","up"]},{"osd":4,"uuid":"6e2c4b71-19d9-4737-83d9-558705d8644a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6801","nonce":1526135780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6803","nonce":1526135780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6807","nonce":1526135780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6805","nonce":1526135780}]},"public_addr":"172.21.15.146:6801/1526135780","cluster_addr":"172.21.15.146:6803/1526135780","heartbeat_back_addr":"172.21.15.146:6807/1526135780","heartbeat_front_addr":"172.21.15.146:6805/1526135780","state":["exists","up"]},{"osd":5,"uuid":"fb69c525-f1be-4b7d-ab32-6c29b0d6fddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6809","nonce":460262507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6811","nonce":460262507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6815","nonce":460262507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6813","nonce":460262507}]},"public_addr":"172.21.15.146:6809/460262507","cluster_addr":"172.21.15.146:6811/460262507","heartbeat_back_addr":"172.21.15.146:6815/460262507","heartbeat_front_addr":"172.21.15.146:6813/460262507","state":["exists","up"]},{"osd":6,"uuid":"c32e7fed-370d-4401-8ce0-6a4335b0d168","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6823","nonce":1392897493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6827","nonce":1392897493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6831","nonce":1392897493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6829","nonce":1392897493}]},"public_addr":"172.21.15.146:6823/1392897493","cluster_addr":"172.21.15.146:6827/1392897493","heartbeat_back_addr":"172.21.15.146:6831/1392897493","heartbeat_front_addr":"172.21.15.146:6829/1392897493","state":["exists","up"]},{"osd":7,"uuid":"75a79b44-fa5f-426d-a7d3-71f21f78835c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6817","nonce":3268323156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6819","nonce":3268323156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6825","nonce":3268323156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6822","nonce":3268323156}]},"public_addr":"172.21.15.146:6817/3268323156","cluster_addr":"172.21.15.146:6819/3268323156","heartbeat_back_addr":"172.21.15.146:6825/3268323156","heartbeat_front_addr":"172.21.15.146:6822/3268323156","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.310437+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.295850+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.294674+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.526855+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.300023+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.595160+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.346046+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.558741+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T09:30:22.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:22.093+0000 7f1b2a800640 1 -- 172.21.15.6:0/3949935376 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f1b0006b520 msgr2=0x7f1b0006d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:22.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:22.093+0000 7f1b2a800640 1 --2- 172.21.15.6:0/3949935376 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f1b0006b520 0x7f1b0006d9e0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f1b0c000f60 tx=0x7f1b0c007a80 comp rx=0 tx=0).stop 2024-03-04T09:30:22.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:22.093+0000 7f1b2a800640 1 -- 172.21.15.6:0/3949935376 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1b240cfd90 msgr2=0x7f1b24158a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:22.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:22.093+0000 7f1b2a800640 1 --2- 172.21.15.6:0/3949935376 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1b240cfd90 0x7f1b24158a20 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f1b1400c7f0 tx=0x7f1b1400ccb0 comp rx=0 tx=0).stop 2024-03-04T09:30:22.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:22.093+0000 7f1b2a800640 1 -- 172.21.15.6:0/3949935376 shutdown_connections 2024-03-04T09:30:22.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:22.093+0000 7f1b2a800640 1 --2- 172.21.15.6:0/3949935376 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1b24161b40 0x7f1b240cd380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:22.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:22.093+0000 7f1b2a800640 1 --2- 172.21.15.6:0/3949935376 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f1b0006b520 0x7f1b0006d9e0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:22.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:22.093+0000 7f1b2a800640 1 --2- 172.21.15.6:0/3949935376 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1b240d2930 0x7f1b240b6b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:22.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:22.093+0000 7f1b2a800640 1 --2- 172.21.15.6:0/3949935376 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1b240cfd90 0x7f1b24158a20 unknown :-1 s=CLOSED pgs=1069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:22.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:22.093+0000 7f1b2a800640 1 -- 172.21.15.6:0/3949935376 >> 172.21.15.6:0/3949935376 conn(0x7f1b2409d220 msgr2=0x7f1b2406bb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:22.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:22.093+0000 7f1b2a800640 1 -- 172.21.15.6:0/3949935376 shutdown_connections 2024-03-04T09:30:22.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:22.093+0000 7f1b2a800640 1 -- 172.21.15.6:0/3949935376 wait complete. 2024-03-04T09:30:22.109 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-04T09:30:22.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:22.329+0000 7f134fe00640 1 Processor -- start 2024-03-04T09:30:22.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:22.329+0000 7f134fe00640 1 -- start start 2024-03-04T09:30:22.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:22.331+0000 7f134fe00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f1350072040 0x7f13500b3630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:22.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:22.331+0000 7f134fe00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f13500713f0 0x7f1350071840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:22.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:22.331+0000 7f134fe00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1350154ce0 con 0x7f1350154900 2024-03-04T09:30:22.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:22.331+0000 7f134fe00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1350071d80 con 0x7f1350072040 2024-03-04T09:30:22.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:22.331+0000 7f134fe00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13500b2bf0 con 0x7f13500713f0 2024-03-04T09:30:22.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:22.331+0000 7f134ea00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f1350154900 0x7f13500cf1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:55286/0 (socket says 172.21.15.6:55286) 2024-03-04T09:30:22.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:22.331+0000 7f134ea00640 1 -- 172.21.15.6:0/1251811816 learned_addr learned my addr 172.21.15.6:0/1251811816 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:30:22.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:22.332+0000 7f134f400640 1 --2- 172.21.15.6:0/1251811816 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f13500713f0 0x7f1350071840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:22.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:22.332+0000 7f1346000640 1 --2- 172.21.15.6:0/1251811816 >> v2:172.21.15.146:3300/0 conn(0x7f1350072040 0x7f13500b3630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:22.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:22.332+0000 7f134e000640 1 -- 172.21.15.6:0/1251811816 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4113067673 0 0) 0x7f1350154ce0 con 0x7f1350154900 2024-03-04T09:30:22.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:22.332+0000 7f134e000640 1 -- 172.21.15.6:0/1251811816 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1334003520 con 0x7f1350154900 2024-03-04T09:30:22.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:22.332+0000 7f134e000640 1 -- 172.21.15.6:0/1251811816 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 778320493 0 0) 0x7f1334003520 con 0x7f1350154900 2024-03-04T09:30:22.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:22.332+0000 7f134e000640 1 -- 172.21.15.6:0/1251811816 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f13500713f0 msgr2=0x7f1350071840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:22.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:22.332+0000 7f134e000640 1 --2- 172.21.15.6:0/1251811816 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f13500713f0 0x7f1350071840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:22.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:22.332+0000 7f134e000640 1 -- 172.21.15.6:0/1251811816 >> v2:172.21.15.146:3300/0 conn(0x7f1350072040 msgr2=0x7f13500b3630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:22.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:22.332+0000 7f134e000640 1 --2- 172.21.15.6:0/1251811816 >> v2:172.21.15.146:3300/0 conn(0x7f1350072040 0x7f13500b3630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:22.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:22.332+0000 7f134e000640 1 -- 172.21.15.6:0/1251811816 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13500b2ed0 con 0x7f1350154900 2024-03-04T09:30:22.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:22.332+0000 7f134e000640 1 -- 172.21.15.6:0/1251811816 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7f1340001670 con 0x7f1350154900 2024-03-04T09:30:22.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:22.332+0000 7f1346000640 1 --2- 172.21.15.6:0/1251811816 >> v2:172.21.15.146:3300/0 conn(0x7f1350072040 0x7f13500b3630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T09:30:22.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:22.332+0000 7f134e000640 1 -- 172.21.15.6:0/1251811816 >> v1:172.21.15.6:6789/0 conn(0x7f1350154900 legacy=0x7f13500cf1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:22.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:22.332+0000 7f134e000640 1 --2- 172.21.15.6:0/1251811816 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f13500713f0 0x7f13340039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:22.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:22.333+0000 7f134e000640 1 --2- 172.21.15.6:0/1251811816 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1334003f80 0x7f1334004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:22.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:22.333+0000 7f134e000640 1 --2- 172.21.15.6:0/1251811816 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1334004d40 0x7f13340051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:22.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:22.333+0000 7f134e000640 1 -- 172.21.15.6:0/1251811816 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1350071d80 con 0x7f1334003f80 2024-03-04T09:30:22.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:22.333+0000 7f134e000640 1 -- 172.21.15.6:0/1251811816 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13500b2bf0 con 0x7f1334004d40 2024-03-04T09:30:22.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:22.333+0000 7f134e000640 1 -- 172.21.15.6:0/1251811816 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f133400bb20 con 0x7f13500713f0 2024-03-04T09:30:22.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:22.333+0000 7f1346000640 1 --2- 172.21.15.6:0/1251811816 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1334003f80 0x7f1334004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:22.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:22.333+0000 7f134f400640 1 --2- 172.21.15.6:0/1251811816 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1334004d40 0x7f13340051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:22.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:22.333+0000 7f134ea00640 1 --2- 172.21.15.6:0/1251811816 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f13500713f0 0x7f13340039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:22.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:22.333+0000 7f1346000640 1 -- 172.21.15.6:0/1251811816 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f13500713f0 msgr2=0x7f13340039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:22.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:22.333+0000 7f1346000640 1 --2- 172.21.15.6:0/1251811816 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f13500713f0 0x7f13340039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:22.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:22.333+0000 7f1346000640 1 -- 172.21.15.6:0/1251811816 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1334004d40 msgr2=0x7f13340051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:22.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:22.333+0000 7f1346000640 1 --2- 172.21.15.6:0/1251811816 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1334004d40 0x7f13340051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:22.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:22.333+0000 7f1346000640 1 -- 172.21.15.6:0/1251811816 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1334003520 con 0x7f1334003f80 2024-03-04T09:30:22.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:22.333+0000 7f134ea00640 1 --2- 172.21.15.6:0/1251811816 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f13500713f0 0x7f13340039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:30:22.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:22.335+0000 7f1346000640 1 --2- 172.21.15.6:0/1251811816 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1334003f80 0x7f1334004800 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7f133800aab0 tx=0x7f1334005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d97bc7ca3507cf9c server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:22.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:22.335+0000 7f134e000640 1 -- 172.21.15.6:0/1251811816 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f133800b040 con 0x7f1334003f80 2024-03-04T09:30:22.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:22.335+0000 7f134e000640 1 -- 172.21.15.6:0/1251811816 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1338004230 con 0x7f1334003f80 2024-03-04T09:30:22.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:22.335+0000 7f134e000640 1 -- 172.21.15.6:0/1251811816 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f133802c8f0 con 0x7f1334003f80 2024-03-04T09:30:22.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:22.336+0000 7f134fe00640 1 -- 172.21.15.6:0/1251811816 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1334003f80 msgr2=0x7f1334004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:22.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:22.336+0000 7f134fe00640 1 --2- 172.21.15.6:0/1251811816 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1334003f80 0x7f1334004800 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7f133800aab0 tx=0x7f1334005830 comp rx=0 tx=0).stop 2024-03-04T09:30:22.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:22.336+0000 7f134f400640 1 -- 172.21.15.6:0/1251811816 reap_dead start 2024-03-04T09:30:22.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:22.336+0000 7f134fe00640 1 -- 172.21.15.6:0/1251811816 shutdown_connections 2024-03-04T09:30:22.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:22.336+0000 7f134fe00640 1 -- 172.21.15.6:0/1251811816 >> 172.21.15.6:0/1251811816 conn(0x7f135006b3a0 msgr2=0x7f135006b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:22.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:22.336+0000 7f134fe00640 1 -- 172.21.15.6:0/1251811816 shutdown_connections 2024-03-04T09:30:22.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:22.336+0000 7f134fe00640 1 -- 172.21.15.6:0/1251811816 wait complete. 2024-03-04T09:30:22.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:22.338+0000 7f134fe00640 1 Processor -- start 2024-03-04T09:30:22.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:22.339+0000 7f134fe00640 1 -- start start 2024-03-04T09:30:22.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:22.340+0000 7f134fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1334003f80 0x7f13501f4e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:22.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:22.340+0000 7f134fe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f13501f4660 0x7f13501f3550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:22.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:22.340+0000 7f134fe00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f135015dcd0 0x7f135015e150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:22.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:22.340+0000 7f134fe00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13500b3cf0 con 0x7f1334003f80 2024-03-04T09:30:22.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:22.340+0000 7f134fe00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13500b3b70 con 0x7f13501f4660 2024-03-04T09:30:22.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:22.340+0000 7f134fe00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1350071840 con 0x7f135015dcd0 2024-03-04T09:30:22.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:22.340+0000 7f134ea00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1334003f80 0x7f13501f4e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:22.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:22.340+0000 7f134ea00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1334003f80 0x7f13501f4e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38354/0 (socket says 172.21.15.6:38354) 2024-03-04T09:30:22.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:22.340+0000 7f134ea00640 1 -- 172.21.15.6:0/2147862526 learned_addr learned my addr 172.21.15.6:0/2147862526 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:30:22.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:22.341+0000 7f134ea00640 1 -- 172.21.15.6:0/2147862526 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f135015dcd0 msgr2=0x7f135015e150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:22.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:22.341+0000 7f134f400640 1 --2- 172.21.15.6:0/2147862526 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f135015dcd0 0x7f135015e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:22.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:22.341+0000 7f1346000640 1 --2- 172.21.15.6:0/2147862526 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f13501f4660 0x7f13501f3550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:22.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:22.341+0000 7f134ea00640 1 --2- 172.21.15.6:0/2147862526 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f135015dcd0 0x7f135015e150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:22.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:22.341+0000 7f134ea00640 1 -- 172.21.15.6:0/2147862526 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f13501f4660 msgr2=0x7f13501f3550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:22.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:22.341+0000 7f134ea00640 1 --2- 172.21.15.6:0/2147862526 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f13501f4660 0x7f13501f3550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:22.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:22.341+0000 7f134ea00640 1 -- 172.21.15.6:0/2147862526 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13501421b0 con 0x7f1334003f80 2024-03-04T09:30:22.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:22.341+0000 7f134ea00640 1 --2- 172.21.15.6:0/2147862526 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1334003f80 0x7f13501f4e80 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7f134000d770 tx=0x7f134000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:22.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:22.341+0000 7f1347e00640 1 -- 172.21.15.6:0/2147862526 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1340019070 con 0x7f1334003f80 2024-03-04T09:30:22.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:22.341+0000 7f1347e00640 1 -- 172.21.15.6:0/2147862526 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13400074e0 con 0x7f1334003f80 2024-03-04T09:30:22.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:22.341+0000 7f134fe00640 1 -- 172.21.15.6:0/2147862526 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1350142490 con 0x7f1334003f80 2024-03-04T09:30:22.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:22.341+0000 7f1347e00640 1 -- 172.21.15.6:0/2147862526 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13400049e0 con 0x7f1334003f80 2024-03-04T09:30:22.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:22.342+0000 7f134fe00640 1 -- 172.21.15.6:0/2147862526 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1350142a50 con 0x7f1334003f80 2024-03-04T09:30:22.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:22.343+0000 7f134fe00640 1 -- 172.21.15.6:0/2147862526 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f135015e690 con 0x7f1334003f80 2024-03-04T09:30:22.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:22.344+0000 7f1347e00640 1 -- 172.21.15.6:0/2147862526 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f134001c070 con 0x7f1334003f80 2024-03-04T09:30:22.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:22.345+0000 7f1347e00640 1 --2- 172.21.15.6:0/2147862526 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f132c06b500 0x7f132c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:22.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:22.345+0000 7f1347e00640 1 -- 172.21.15.6:0/2147862526 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1340089ca0 con 0x7f1334003f80 2024-03-04T09:30:22.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:22.346+0000 7f1346000640 1 --2- 172.21.15.6:0/2147862526 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f132c06b500 0x7f132c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:22.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:22.347+0000 7f1346000640 1 --2- 172.21.15.6:0/2147862526 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f132c06b500 0x7f132c06d9c0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f1338035000 tx=0x7f1338030000 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:22.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:22.349+0000 7f1347e00640 1 -- 172.21.15.6:0/2147862526 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1340058a80 con 0x7f1334003f80 2024-03-04T09:30:22.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:22.579+0000 7f134fe00640 1 -- 172.21.15.6:0/2147862526 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f135004f330 con 0x7f1334003f80 2024-03-04T09:30:23.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:23.201+0000 7f1347e00640 1 -- 172.21.15.6:0/2147862526 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v59) v1 ==== 118+0+0 (secure 0 0 0) 0x7f1340058380 con 0x7f1334003f80 2024-03-04T09:30:23.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:23.205+0000 7f134fe00640 1 -- 172.21.15.6:0/2147862526 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f132c06b500 msgr2=0x7f132c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:23.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:23.205+0000 7f134fe00640 1 --2- 172.21.15.6:0/2147862526 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f132c06b500 0x7f132c06d9c0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f1338035000 tx=0x7f1338030000 comp rx=0 tx=0).stop 2024-03-04T09:30:23.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:23.205+0000 7f134fe00640 1 -- 172.21.15.6:0/2147862526 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1334003f80 msgr2=0x7f13501f4e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:23.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:23.205+0000 7f134fe00640 1 --2- 172.21.15.6:0/2147862526 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1334003f80 0x7f13501f4e80 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7f134000d770 tx=0x7f134000dc30 comp rx=0 tx=0).stop 2024-03-04T09:30:23.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:23.205+0000 7f134fe00640 1 -- 172.21.15.6:0/2147862526 shutdown_connections 2024-03-04T09:30:23.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:23.205+0000 7f134fe00640 1 --2- 172.21.15.6:0/2147862526 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f132c06b500 0x7f132c06d9c0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:23.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:23.205+0000 7f134fe00640 1 --2- 172.21.15.6:0/2147862526 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f135015dcd0 0x7f135015e150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:23.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:23.205+0000 7f134fe00640 1 --2- 172.21.15.6:0/2147862526 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f13501f4660 0x7f13501f3550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:23.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:23.205+0000 7f134fe00640 1 --2- 172.21.15.6:0/2147862526 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1334003f80 0x7f13501f4e80 unknown :-1 s=CLOSED pgs=1072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:23.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:23.205+0000 7f134fe00640 1 -- 172.21.15.6:0/2147862526 >> 172.21.15.6:0/2147862526 conn(0x7f135006b3a0 msgr2=0x7f1350151870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:23.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:23.205+0000 7f134fe00640 1 -- 172.21.15.6:0/2147862526 shutdown_connections 2024-03-04T09:30:23.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:23.205+0000 7f134fe00640 1 -- 172.21.15.6:0/2147862526 wait complete. 2024-03-04T09:30:23.227 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-04T09:30:23.227 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-04T09:30:23.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:23.459+0000 7f37c8000640 1 Processor -- start 2024-03-04T09:30:23.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:23.459+0000 7f37c8000640 1 -- start start 2024-03-04T09:30:23.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:23.461+0000 7f37c8000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f37c0066d00 0x7f37c00b4a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:23.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:23.461+0000 7f37c8000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37c00b4f70 con 0x7f37c00b4650 2024-03-04T09:30:23.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:23.461+0000 7f37c8000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37c0158bb0 con 0x7f37c00cf6e0 2024-03-04T09:30:23.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:23.461+0000 7f37c8000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37c00671c0 con 0x7f37c0066d00 2024-03-04T09:30:23.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:23.461+0000 7f37c5600640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f37c00b4650 0x7f37c015bc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:55302/0 (socket says 172.21.15.6:55302) 2024-03-04T09:30:23.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:23.461+0000 7f37c5600640 1 -- 172.21.15.6:0/2008953405 learned_addr learned my addr 172.21.15.6:0/2008953405 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:30:23.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:23.462+0000 7f37c4c00640 1 --2- 172.21.15.6:0/2008953405 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f37c0066d00 0x7f37c00b4a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:23.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:23.462+0000 7f37bfe00640 1 -- 172.21.15.6:0/2008953405 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2551578195 0 0) 0x7f37c00b4f70 con 0x7f37c00b4650 2024-03-04T09:30:23.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:23.462+0000 7f37bfe00640 1 -- 172.21.15.6:0/2008953405 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37a8003520 con 0x7f37c00b4650 2024-03-04T09:30:23.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:23.462+0000 7f37bfe00640 1 -- 172.21.15.6:0/2008953405 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1379387805 0 0) 0x7f37a8003520 con 0x7f37c00b4650 2024-03-04T09:30:23.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:23.462+0000 7f37bfe00640 1 -- 172.21.15.6:0/2008953405 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f37c0066d00 msgr2=0x7f37c00b4a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:23.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:23.462+0000 7f37bfe00640 1 --2- 172.21.15.6:0/2008953405 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f37c0066d00 0x7f37c00b4a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:23.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:23.462+0000 7f37bfe00640 1 -- 172.21.15.6:0/2008953405 >> v1:172.21.15.146:6789/0 conn(0x7f37c00cf6e0 legacy=0x7f37c006d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:23.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:23.462+0000 7f37bfe00640 1 -- 172.21.15.6:0/2008953405 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37c00674a0 con 0x7f37c00b4650 2024-03-04T09:30:23.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:23.462+0000 7f37bfe00640 1 -- 172.21.15.6:0/2008953405 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7f37b4001670 con 0x7f37c00b4650 2024-03-04T09:30:23.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:23.463+0000 7f37bfe00640 1 -- 172.21.15.6:0/2008953405 >> v1:172.21.15.6:6789/0 conn(0x7f37c00b4650 legacy=0x7f37c015bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:23.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:23.463+0000 7f37bfe00640 1 --2- 172.21.15.6:0/2008953405 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f37a80039e0 0x7f37a8003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:23.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:23.463+0000 7f37bfe00640 1 --2- 172.21.15.6:0/2008953405 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f37a8004800 0x7f37a8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:23.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:23.463+0000 7f37bfe00640 1 --2- 172.21.15.6:0/2008953405 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f37c0066d00 0x7f37a80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:23.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:23.463+0000 7f37bfe00640 1 -- 172.21.15.6:0/2008953405 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37c00671c0 con 0x7f37a8004800 2024-03-04T09:30:23.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:23.463+0000 7f37bfe00640 1 -- 172.21.15.6:0/2008953405 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37a800ba70 con 0x7f37a80039e0 2024-03-04T09:30:23.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:23.463+0000 7f37bfe00640 1 -- 172.21.15.6:0/2008953405 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37a800bc70 con 0x7f37c0066d00 2024-03-04T09:30:23.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:23.463+0000 7f37c4c00640 1 --2- 172.21.15.6:0/2008953405 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f37a8004800 0x7f37a8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:23.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:23.463+0000 7f37c5600640 1 --2- 172.21.15.6:0/2008953405 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f37a80039e0 0x7f37a8003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:23.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:23.463+0000 7f37c4c00640 1 -- 172.21.15.6:0/2008953405 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f37c0066d00 msgr2=0x7f37a80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:23.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:23.463+0000 7f37c4c00640 1 --2- 172.21.15.6:0/2008953405 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f37c0066d00 0x7f37a80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:23.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:23.463+0000 7f37c4c00640 1 -- 172.21.15.6:0/2008953405 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f37a80039e0 msgr2=0x7f37a8003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:23.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:23.463+0000 7f37c4c00640 1 --2- 172.21.15.6:0/2008953405 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f37a80039e0 0x7f37a8003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:23.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:23.463+0000 7f37c4c00640 1 -- 172.21.15.6:0/2008953405 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37a8003520 con 0x7f37a8004800 2024-03-04T09:30:23.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:23.463+0000 7f37c5600640 1 --2- 172.21.15.6:0/2008953405 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f37a80039e0 0x7f37a8003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:30:23.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:23.464+0000 7f37c4c00640 1 --2- 172.21.15.6:0/2008953405 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f37a8004800 0x7f37a8004c80 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7f37ac009650 tx=0x7f37a8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=861d3f373bcfc002 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:23.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:23.465+0000 7f37bfe00640 1 -- 172.21.15.6:0/2008953405 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37ac03b070 con 0x7f37a8004800 2024-03-04T09:30:23.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:23.465+0000 7f37bfe00640 1 -- 172.21.15.6:0/2008953405 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37ac02e8c0 con 0x7f37a8004800 2024-03-04T09:30:23.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:23.465+0000 7f37bfe00640 1 -- 172.21.15.6:0/2008953405 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37ac009690 con 0x7f37a8004800 2024-03-04T09:30:23.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:23.465+0000 7f37c8000640 1 -- 172.21.15.6:0/2008953405 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f37a8004800 msgr2=0x7f37a8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:23.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:23.465+0000 7f37c8000640 1 --2- 172.21.15.6:0/2008953405 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f37a8004800 0x7f37a8004c80 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7f37ac009650 tx=0x7f37a8005830 comp rx=0 tx=0).stop 2024-03-04T09:30:23.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:23.465+0000 7f37c6000640 1 -- 172.21.15.6:0/2008953405 reap_dead start 2024-03-04T09:30:23.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:23.465+0000 7f37c8000640 1 -- 172.21.15.6:0/2008953405 shutdown_connections 2024-03-04T09:30:23.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:23.465+0000 7f37c8000640 1 -- 172.21.15.6:0/2008953405 >> 172.21.15.6:0/2008953405 conn(0x7f37c0072280 msgr2=0x7f37c006afb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:23.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:23.466+0000 7f37c8000640 1 -- 172.21.15.6:0/2008953405 shutdown_connections 2024-03-04T09:30:23.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:23.466+0000 7f37c8000640 1 -- 172.21.15.6:0/2008953405 wait complete. 2024-03-04T09:30:23.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:23.467+0000 7f37c8000640 1 Processor -- start 2024-03-04T09:30:23.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:23.467+0000 7f37c8000640 1 -- start start 2024-03-04T09:30:23.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:23.468+0000 7f37c8000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f37a8004800 0x7f37c01f0180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:23.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:23.468+0000 7f37c8000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f37c0145dd0 0x7f37c01f41e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:23.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:23.468+0000 7f37c8000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f37c0150be0 0x7f37c015c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:23.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:23.468+0000 7f37c8000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37c00cf1c0 con 0x7f37a8004800 2024-03-04T09:30:23.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:23.468+0000 7f37c8000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37c00ceee0 con 0x7f37c0145dd0 2024-03-04T09:30:23.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:23.468+0000 7f37c8000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37c006db50 con 0x7f37c0150be0 2024-03-04T09:30:23.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:23.469+0000 7f37c5600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f37a8004800 0x7f37c01f0180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:23.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:23.469+0000 7f37c5600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f37a8004800 0x7f37c01f0180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38374/0 (socket says 172.21.15.6:38374) 2024-03-04T09:30:23.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:23.469+0000 7f37c5600640 1 -- 172.21.15.6:0/700513770 learned_addr learned my addr 172.21.15.6:0/700513770 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:30:23.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:23.469+0000 7f37c4c00640 1 --2- 172.21.15.6:0/700513770 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f37c0145dd0 0x7f37c01f41e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:23.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:23.469+0000 7f37c5600640 1 -- 172.21.15.6:0/700513770 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f37c0150be0 msgr2=0x7f37c015c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:23.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:23.469+0000 7f37c6000640 1 --2- 172.21.15.6:0/700513770 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f37c0150be0 0x7f37c015c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:23.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:23.469+0000 7f37c5600640 1 --2- 172.21.15.6:0/700513770 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f37c0150be0 0x7f37c015c970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:23.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:23.469+0000 7f37c5600640 1 -- 172.21.15.6:0/700513770 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f37c0145dd0 msgr2=0x7f37c01f41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:23.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:23.469+0000 7f37c5600640 1 --2- 172.21.15.6:0/700513770 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f37c0145dd0 0x7f37c01f41e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:23.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:23.469+0000 7f37c5600640 1 -- 172.21.15.6:0/700513770 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37c01f4720 con 0x7f37a8004800 2024-03-04T09:30:23.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:23.469+0000 7f37c4c00640 1 --2- 172.21.15.6:0/700513770 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f37c0145dd0 0x7f37c01f41e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:30:23.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:23.469+0000 7f37c5600640 1 --2- 172.21.15.6:0/700513770 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f37a8004800 0x7f37c01f0180 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f37b400d940 tx=0x7f37b400de00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:23.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:23.470+0000 7f37bd600640 1 -- 172.21.15.6:0/700513770 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37b4019070 con 0x7f37a8004800 2024-03-04T09:30:23.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:23.470+0000 7f37bd600640 1 -- 172.21.15.6:0/700513770 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37b40074e0 con 0x7f37a8004800 2024-03-04T09:30:23.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:23.470+0000 7f37bd600640 1 -- 172.21.15.6:0/700513770 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37b40049e0 con 0x7f37a8004800 2024-03-04T09:30:23.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:23.470+0000 7f37c8000640 1 -- 172.21.15.6:0/700513770 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37c01f07f0 con 0x7f37a8004800 2024-03-04T09:30:23.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:23.470+0000 7f37c8000640 1 -- 172.21.15.6:0/700513770 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37c01f0db0 con 0x7f37a8004800 2024-03-04T09:30:23.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:23.471+0000 7f37c8000640 1 -- 172.21.15.6:0/700513770 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37c01461b0 con 0x7f37a8004800 2024-03-04T09:30:23.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:23.472+0000 7f37bd600640 1 -- 172.21.15.6:0/700513770 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f37b401c070 con 0x7f37a8004800 2024-03-04T09:30:23.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:23.473+0000 7f37bd600640 1 --2- 172.21.15.6:0/700513770 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f37a006b4d0 0x7f37a006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:23.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:23.473+0000 7f37bd600640 1 -- 172.21.15.6:0/700513770 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f37b4089de0 con 0x7f37a8004800 2024-03-04T09:30:23.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:23.474+0000 7f37c4c00640 1 --2- 172.21.15.6:0/700513770 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f37a006b4d0 0x7f37a006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:23.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:23.475+0000 7f37c4c00640 1 --2- 172.21.15.6:0/700513770 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f37a006b4d0 0x7f37a006d990 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f37ac038000 tx=0x7f37ac032000 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:23.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:23.481+0000 7f37bd600640 1 -- 172.21.15.6:0/700513770 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f37c01f4720 con 0x7f37a8004800 2024-03-04T09:30:23.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:23.724+0000 7f37c8000640 1 -- 172.21.15.6:0/700513770 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f37c004f330 con 0x7f37a8004800 2024-03-04T09:30:24.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.211+0000 7f37bd600640 1 -- 172.21.15.6:0/700513770 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v60) v1 ==== 122+0+0 (secure 0 0 0) 0x7f37b4058c50 con 0x7f37a8004800 2024-03-04T09:30:24.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.214+0000 7f37c8000640 1 -- 172.21.15.6:0/700513770 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f37a006b4d0 msgr2=0x7f37a006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:24.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.214+0000 7f37c8000640 1 --2- 172.21.15.6:0/700513770 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f37a006b4d0 0x7f37a006d990 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f37ac038000 tx=0x7f37ac032000 comp rx=0 tx=0).stop 2024-03-04T09:30:24.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.214+0000 7f37c8000640 1 -- 172.21.15.6:0/700513770 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f37a8004800 msgr2=0x7f37c01f0180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:24.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.214+0000 7f37c8000640 1 --2- 172.21.15.6:0/700513770 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f37a8004800 0x7f37c01f0180 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f37b400d940 tx=0x7f37b400de00 comp rx=0 tx=0).stop 2024-03-04T09:30:24.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.214+0000 7f37c8000640 1 -- 172.21.15.6:0/700513770 shutdown_connections 2024-03-04T09:30:24.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.214+0000 7f37c8000640 1 --2- 172.21.15.6:0/700513770 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f37a006b4d0 0x7f37a006d990 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:24.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.214+0000 7f37c8000640 1 --2- 172.21.15.6:0/700513770 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f37c0150be0 0x7f37c015c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:24.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.214+0000 7f37c8000640 1 --2- 172.21.15.6:0/700513770 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f37c0145dd0 0x7f37c01f41e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:24.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.214+0000 7f37c8000640 1 --2- 172.21.15.6:0/700513770 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f37a8004800 0x7f37c01f0180 unknown :-1 s=CLOSED pgs=1075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:24.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.214+0000 7f37c8000640 1 -- 172.21.15.6:0/700513770 >> 172.21.15.6:0/700513770 conn(0x7f37c0072280 msgr2=0x7f37c006d420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:24.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.214+0000 7f37c8000640 1 -- 172.21.15.6:0/700513770 shutdown_connections 2024-03-04T09:30:24.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.214+0000 7f37c8000640 1 -- 172.21.15.6:0/700513770 wait complete. 2024-03-04T09:30:24.232 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-04T09:30:24.232 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:30:24.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.455+0000 7f1469a00640 1 Processor -- start 2024-03-04T09:30:24.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.455+0000 7f1469a00640 1 -- start start 2024-03-04T09:30:24.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.457+0000 7f1469a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1464161670 0x7f1464070c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:24.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.457+0000 7f1469a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f14640ae0a0 0x7f14640711b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:24.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.457+0000 7f1469a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14640ce9a0 con 0x7f146406e240 2024-03-04T09:30:24.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.457+0000 7f1469a00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14640730d0 con 0x7f14640ae0a0 2024-03-04T09:30:24.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.457+0000 7f1469a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14640cec80 con 0x7f1464161670 2024-03-04T09:30:24.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.457+0000 7f1463400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f14640ae0a0 0x7f14640711b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:24.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.457+0000 7f1462000640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f146406e240 0x7f146406cfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:55314/0 (socket says 172.21.15.6:55314) 2024-03-04T09:30:24.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.457+0000 7f1462000640 1 -- 172.21.15.6:0/3484852050 learned_addr learned my addr 172.21.15.6:0/3484852050 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:30:24.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.458+0000 7f1462a00640 1 --2- 172.21.15.6:0/3484852050 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1464161670 0x7f1464070c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:24.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.458+0000 7f1463400640 1 -- 172.21.15.6:0/3484852050 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1464161670 msgr2=0x7f1464070c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:24.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.458+0000 7f1463400640 1 --2- 172.21.15.6:0/3484852050 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1464161670 0x7f1464070c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:24.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.458+0000 7f1461600640 1 -- 172.21.15.6:0/3484852050 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1445048524 0 0) 0x7f14640ce9a0 con 0x7f146406e240 2024-03-04T09:30:24.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.458+0000 7f1463400640 1 -- 172.21.15.6:0/3484852050 >> v1:172.21.15.6:6789/0 conn(0x7f146406e240 legacy=0x7f146406cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:24.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.458+0000 7f1463400640 1 -- 172.21.15.6:0/3484852050 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14640cef60 con 0x7f14640ae0a0 2024-03-04T09:30:24.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.460+0000 7f1463400640 1 --2- 172.21.15.6:0/3484852050 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f14640ae0a0 0x7f14640711b0 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f145c009d90 tx=0x7f145c02bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d678ea875efa8e27 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:24.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.460+0000 7f1461600640 1 -- 172.21.15.6:0/3484852050 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f145c039070 con 0x7f14640ae0a0 2024-03-04T09:30:24.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.460+0000 7f1461600640 1 -- 172.21.15.6:0/3484852050 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f145c02cc50 con 0x7f14640ae0a0 2024-03-04T09:30:24.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.460+0000 7f1461600640 1 -- 172.21.15.6:0/3484852050 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f145c034440 con 0x7f14640ae0a0 2024-03-04T09:30:24.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.461+0000 7f1469a00640 1 -- 172.21.15.6:0/3484852050 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f14640ae0a0 msgr2=0x7f14640711b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:24.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.461+0000 7f1469a00640 1 --2- 172.21.15.6:0/3484852050 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f14640ae0a0 0x7f14640711b0 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f145c009d90 tx=0x7f145c02bea0 comp rx=0 tx=0).stop 2024-03-04T09:30:24.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.461+0000 7f1469a00640 1 -- 172.21.15.6:0/3484852050 shutdown_connections 2024-03-04T09:30:24.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.461+0000 7f1469a00640 1 --2- 172.21.15.6:0/3484852050 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f14640ae0a0 0x7f14640711b0 unknown :-1 s=CLOSED pgs=1077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:24.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.461+0000 7f1469a00640 1 --2- 172.21.15.6:0/3484852050 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1464161670 0x7f1464070c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:24.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.461+0000 7f1469a00640 1 -- 172.21.15.6:0/3484852050 >> 172.21.15.6:0/3484852050 conn(0x7f146406afb0 msgr2=0x7f146406b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:24.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.461+0000 7f1469a00640 1 -- 172.21.15.6:0/3484852050 shutdown_connections 2024-03-04T09:30:24.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.461+0000 7f1469a00640 1 -- 172.21.15.6:0/3484852050 wait complete. 2024-03-04T09:30:24.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.463+0000 7f1469a00640 1 Processor -- start 2024-03-04T09:30:24.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.463+0000 7f1469a00640 1 -- start start 2024-03-04T09:30:24.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.464+0000 7f1469a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f146406e240 0x7f14641f3e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:24.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.464+0000 7f1469a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f14640ae0a0 0x7f14640c9150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:24.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.464+0000 7f1469a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1464161670 0x7f1464150770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:24.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.464+0000 7f1469a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14640ce600 con 0x7f1464161670 2024-03-04T09:30:24.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.464+0000 7f1469a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14640cf520 con 0x7f14640ae0a0 2024-03-04T09:30:24.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.464+0000 7f1469a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14640ce780 con 0x7f146406e240 2024-03-04T09:30:24.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.464+0000 7f1463400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1464161670 0x7f1464150770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:24.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.464+0000 7f1463400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1464161670 0x7f1464150770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38388/0 (socket says 172.21.15.6:38388) 2024-03-04T09:30:24.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.464+0000 7f1463400640 1 -- 172.21.15.6:0/3764613539 learned_addr learned my addr 172.21.15.6:0/3764613539 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:30:24.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.465+0000 7f1462a00640 1 --2- 172.21.15.6:0/3764613539 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f146406e240 0x7f14641f3e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:24.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.465+0000 7f1462000640 1 --2- 172.21.15.6:0/3764613539 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f14640ae0a0 0x7f14640c9150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:24.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.465+0000 7f1463400640 1 -- 172.21.15.6:0/3764613539 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f146406e240 msgr2=0x7f14641f3e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:24.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.465+0000 7f1463400640 1 --2- 172.21.15.6:0/3764613539 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f146406e240 0x7f14641f3e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:24.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.465+0000 7f1463400640 1 -- 172.21.15.6:0/3764613539 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f14640ae0a0 msgr2=0x7f14640c9150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:24.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.465+0000 7f1463400640 1 --2- 172.21.15.6:0/3764613539 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f14640ae0a0 0x7f14640c9150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:24.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.465+0000 7f1463400640 1 -- 172.21.15.6:0/3764613539 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1464150cb0 con 0x7f1464161670 2024-03-04T09:30:24.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.465+0000 7f1462000640 1 --2- 172.21.15.6:0/3764613539 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f14640ae0a0 0x7f14640c9150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:30:24.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.465+0000 7f1463400640 1 --2- 172.21.15.6:0/3764613539 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1464161670 0x7f1464150770 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f145c009eb0 tx=0x7f145c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:24.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.465+0000 7f145aa00640 1 -- 172.21.15.6:0/3764613539 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f145c039050 con 0x7f1464161670 2024-03-04T09:30:24.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.466+0000 7f145aa00640 1 -- 172.21.15.6:0/3764613539 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f145c045070 con 0x7f1464161670 2024-03-04T09:30:24.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.466+0000 7f1469a00640 1 -- 172.21.15.6:0/3764613539 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14641f4460 con 0x7f1464161670 2024-03-04T09:30:24.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.466+0000 7f145aa00640 1 -- 172.21.15.6:0/3764613539 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f145c03e610 con 0x7f1464161670 2024-03-04T09:30:24.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.466+0000 7f1469a00640 1 -- 172.21.15.6:0/3764613539 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14641f4a20 con 0x7f1464161670 2024-03-04T09:30:24.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.467+0000 7f1469a00640 1 -- 172.21.15.6:0/3764613539 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1464069510 con 0x7f1464161670 2024-03-04T09:30:24.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.468+0000 7f145aa00640 1 -- 172.21.15.6:0/3764613539 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f145c03e7b0 con 0x7f1464161670 2024-03-04T09:30:24.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.469+0000 7f145aa00640 1 --2- 172.21.15.6:0/3764613539 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f144406b4d0 0x7f144406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:24.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.469+0000 7f145aa00640 1 -- 172.21.15.6:0/3764613539 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f145c0acce0 con 0x7f1464161670 2024-03-04T09:30:24.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.469+0000 7f1462a00640 1 --2- 172.21.15.6:0/3764613539 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f144406b4d0 0x7f144406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:24.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.471+0000 7f1462a00640 1 --2- 172.21.15.6:0/3764613539 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f144406b4d0 0x7f144406d990 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f1454006fd0 tx=0x7f1454008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:24.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.473+0000 7f145aa00640 1 -- 172.21.15.6:0/3764613539 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f145c032e20 con 0x7f1464161670 2024-03-04T09:30:24.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.712+0000 7f1469a00640 1 -- 172.21.15.6:0/3764613539 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f14640cf9b0 con 0x7f1464161670 2024-03-04T09:30:24.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.714+0000 7f145aa00640 1 -- 172.21.15.6:0/3764613539 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+4976 (secure 0 0 0) 0x7f14640cf9b0 con 0x7f1464161670 2024-03-04T09:30:24.711 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:30:24.711 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":60,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5555,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/847815316","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":847815316},{"type":"v1","addr":"172.21.15.146:6835","nonce":847815316}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55},{"gid":6046,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/1371176866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":1371176866},{"type":"v1","addr":"172.21.15.146:6837","nonce":1371176866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55},{"gid":7092,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/1696083602","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":1696083602},{"type":"v1","addr":"172.21.15.6:6834","nonce":1696083602}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":54}],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:30:20.660103+0000","modified":"2024-03-04T09:30:23.726533+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7095},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7095":{"gid":7095,"name":"a","rank":0,"incarnation":56,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6835/2836916196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2836916196},{"type":"v1","addr":"172.21.15.6:6835","nonce":2836916196}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7095,"qdb_cluster":[7095]},"id":7}]} 2024-03-04T09:30:24.711 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 60 2024-03-04T09:30:24.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.717+0000 7f1469a00640 1 -- 172.21.15.6:0/3764613539 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f144406b4d0 msgr2=0x7f144406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:24.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.717+0000 7f1469a00640 1 --2- 172.21.15.6:0/3764613539 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f144406b4d0 0x7f144406d990 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f1454006fd0 tx=0x7f1454008040 comp rx=0 tx=0).stop 2024-03-04T09:30:24.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.717+0000 7f1469a00640 1 -- 172.21.15.6:0/3764613539 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1464161670 msgr2=0x7f1464150770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:24.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.717+0000 7f1469a00640 1 --2- 172.21.15.6:0/3764613539 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1464161670 0x7f1464150770 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f145c009eb0 tx=0x7f145c004a40 comp rx=0 tx=0).stop 2024-03-04T09:30:24.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.717+0000 7f1469a00640 1 -- 172.21.15.6:0/3764613539 shutdown_connections 2024-03-04T09:30:24.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.717+0000 7f1469a00640 1 --2- 172.21.15.6:0/3764613539 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1464161670 0x7f1464150770 unknown :-1 s=CLOSED pgs=1078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:24.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.717+0000 7f1469a00640 1 --2- 172.21.15.6:0/3764613539 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f14640ae0a0 0x7f14640c9150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:24.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.717+0000 7f1469a00640 1 --2- 172.21.15.6:0/3764613539 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f144406b4d0 0x7f144406d990 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:24.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.717+0000 7f1469a00640 1 --2- 172.21.15.6:0/3764613539 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f146406e240 0x7f14641f3e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:24.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.717+0000 7f1469a00640 1 -- 172.21.15.6:0/3764613539 >> 172.21.15.6:0/3764613539 conn(0x7f146406afb0 msgr2=0x7f146406b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:24.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.717+0000 7f1469a00640 1 -- 172.21.15.6:0/3764613539 shutdown_connections 2024-03-04T09:30:24.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.717+0000 7f1469a00640 1 -- 172.21.15.6:0/3764613539 wait complete. 2024-03-04T09:30:24.736 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T09:30:24.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.960+0000 7f745d400640 1 Processor -- start 2024-03-04T09:30:24.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.960+0000 7f745d400640 1 -- start start 2024-03-04T09:30:24.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.962+0000 7f745d400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f74580d0e50 0x7f7458168c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:24.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.962+0000 7f745d400640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74580b3710 con 0x7f74580d4a10 2024-03-04T09:30:24.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.962+0000 7f745d400640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74580b39f0 con 0x7f745815e0b0 2024-03-04T09:30:24.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.962+0000 7f745d400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74580b09e0 con 0x7f74580d0e50 2024-03-04T09:30:24.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.962+0000 7f7457400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f74580d4a10 0x7f74580d0740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:55320/0 (socket says 172.21.15.6:55320) 2024-03-04T09:30:24.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.962+0000 7f7457400640 1 -- 172.21.15.6:0/197955214 learned_addr learned my addr 172.21.15.6:0/197955214 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:30:24.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.963+0000 7f745ca00640 1 --2- 172.21.15.6:0/197955214 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f74580d0e50 0x7f7458168c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:24.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.963+0000 7f7456a00640 1 -- 172.21.15.6:0/197955214 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4192992162 0 0) 0x7f74580b3710 con 0x7f74580d4a10 2024-03-04T09:30:24.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.963+0000 7f7456a00640 1 -- 172.21.15.6:0/197955214 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f743c003520 con 0x7f74580d4a10 2024-03-04T09:30:24.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.963+0000 7f7456a00640 1 -- 172.21.15.6:0/197955214 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2137457868 0 0) 0x7f743c003520 con 0x7f74580d4a10 2024-03-04T09:30:24.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.963+0000 7f7456a00640 1 -- 172.21.15.6:0/197955214 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f74580d0e50 msgr2=0x7f7458168c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:24.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.963+0000 7f7456a00640 1 --2- 172.21.15.6:0/197955214 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f74580d0e50 0x7f7458168c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:24.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.963+0000 7f7456a00640 1 -- 172.21.15.6:0/197955214 >> v1:172.21.15.146:6789/0 conn(0x7f745815e0b0 legacy=0x7f74580cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:24.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.963+0000 7f7456a00640 1 -- 172.21.15.6:0/197955214 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74580b3cd0 con 0x7f74580d4a10 2024-03-04T09:30:24.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.963+0000 7f7456a00640 1 -- 172.21.15.6:0/197955214 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7f7444001690 con 0x7f74580d4a10 2024-03-04T09:30:24.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.963+0000 7f7456a00640 1 -- 172.21.15.6:0/197955214 >> v1:172.21.15.6:6789/0 conn(0x7f74580d4a10 legacy=0x7f74580d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:24.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.963+0000 7f7456a00640 1 --2- 172.21.15.6:0/197955214 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f743c0039e0 0x7f743c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:24.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.963+0000 7f7456a00640 1 --2- 172.21.15.6:0/197955214 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f743c004800 0x7f743c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:24.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.963+0000 7f7456a00640 1 --2- 172.21.15.6:0/197955214 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f74580d0e50 0x7f743c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:24.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.963+0000 7f7456a00640 1 -- 172.21.15.6:0/197955214 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f743c00baa0 con 0x7f743c0039e0 2024-03-04T09:30:24.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.963+0000 7f7456a00640 1 -- 172.21.15.6:0/197955214 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f743c00bc50 con 0x7f743c004800 2024-03-04T09:30:24.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.963+0000 7f7456a00640 1 -- 172.21.15.6:0/197955214 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f743c00be50 con 0x7f74580d0e50 2024-03-04T09:30:24.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.964+0000 7f7457e00640 1 --2- 172.21.15.6:0/197955214 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f743c0039e0 0x7f743c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:24.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.964+0000 7f7457400640 1 --2- 172.21.15.6:0/197955214 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f743c004800 0x7f743c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:24.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.964+0000 7f745ca00640 1 --2- 172.21.15.6:0/197955214 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f74580d0e50 0x7f743c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:24.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.964+0000 7f7457e00640 1 -- 172.21.15.6:0/197955214 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f74580d0e50 msgr2=0x7f743c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:24.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.964+0000 7f7457e00640 1 --2- 172.21.15.6:0/197955214 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f74580d0e50 0x7f743c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:24.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.964+0000 7f7457e00640 1 -- 172.21.15.6:0/197955214 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f743c004800 msgr2=0x7f743c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:24.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.964+0000 7f7457e00640 1 --2- 172.21.15.6:0/197955214 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f743c004800 0x7f743c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:24.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.964+0000 7f7457e00640 1 -- 172.21.15.6:0/197955214 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f743c003520 con 0x7f743c0039e0 2024-03-04T09:30:24.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.964+0000 7f7457400640 1 --2- 172.21.15.6:0/197955214 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f743c004800 0x7f743c004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:30:24.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.964+0000 7f745ca00640 1 --2- 172.21.15.6:0/197955214 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f74580d0e50 0x7f743c0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:30:24.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.965+0000 7f7457e00640 1 --2- 172.21.15.6:0/197955214 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f743c0039e0 0x7f743c003e00 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7f7448001e20 tx=0x7f743c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6084bc7dd3007b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:24.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.965+0000 7f7456a00640 1 -- 172.21.15.6:0/197955214 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7448001a70 con 0x7f743c0039e0 2024-03-04T09:30:24.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.965+0000 7f7456a00640 1 -- 172.21.15.6:0/197955214 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7448003b70 con 0x7f743c0039e0 2024-03-04T09:30:24.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.965+0000 7f7456a00640 1 -- 172.21.15.6:0/197955214 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74480050d0 con 0x7f743c0039e0 2024-03-04T09:30:24.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.966+0000 7f745d400640 1 -- 172.21.15.6:0/197955214 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f743c0039e0 msgr2=0x7f743c003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:24.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.966+0000 7f745d400640 1 --2- 172.21.15.6:0/197955214 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f743c0039e0 0x7f743c003e00 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7f7448001e20 tx=0x7f743c005830 comp rx=0 tx=0).stop 2024-03-04T09:30:24.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.966+0000 7f745ca00640 1 -- 172.21.15.6:0/197955214 reap_dead start 2024-03-04T09:30:24.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.966+0000 7f745d400640 1 -- 172.21.15.6:0/197955214 shutdown_connections 2024-03-04T09:30:24.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.966+0000 7f745d400640 1 -- 172.21.15.6:0/197955214 >> 172.21.15.6:0/197955214 conn(0x7f74580d3d30 msgr2=0x7f74580d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:24.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.966+0000 7f745d400640 1 -- 172.21.15.6:0/197955214 shutdown_connections 2024-03-04T09:30:24.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.966+0000 7f745d400640 1 -- 172.21.15.6:0/197955214 wait complete. 2024-03-04T09:30:24.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.968+0000 7f745d400640 1 Processor -- start 2024-03-04T09:30:24.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.968+0000 7f745d400640 1 -- start start 2024-03-04T09:30:24.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.969+0000 7f745d400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f743c0039e0 0x7f74580b9230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:24.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.969+0000 7f745d400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f745814d5a0 0x7f745814da20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:24.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.969+0000 7f745d400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f745814b5f0 0x7f745814ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:24.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.969+0000 7f745d400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f745815e490 con 0x7f743c0039e0 2024-03-04T09:30:24.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.969+0000 7f745d400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74580cfa20 con 0x7f745814d5a0 2024-03-04T09:30:24.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.969+0000 7f745d400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74580cfc60 con 0x7f745814b5f0 2024-03-04T09:30:24.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.969+0000 7f7457e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f743c0039e0 0x7f74580b9230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:24.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.969+0000 7f7457e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f743c0039e0 0x7f74580b9230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38402/0 (socket says 172.21.15.6:38402) 2024-03-04T09:30:24.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.969+0000 7f7457e00640 1 -- 172.21.15.6:0/927256734 learned_addr learned my addr 172.21.15.6:0/927256734 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:30:24.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.970+0000 7f745ca00640 1 --2- 172.21.15.6:0/927256734 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f745814b5f0 0x7f745814ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:24.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.970+0000 7f7457400640 1 --2- 172.21.15.6:0/927256734 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f745814d5a0 0x7f745814da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:24.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.970+0000 7f7457e00640 1 -- 172.21.15.6:0/927256734 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f745814b5f0 msgr2=0x7f745814ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:24.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.970+0000 7f7457e00640 1 --2- 172.21.15.6:0/927256734 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f745814b5f0 0x7f745814ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:24.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.970+0000 7f7457e00640 1 -- 172.21.15.6:0/927256734 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f745814d5a0 msgr2=0x7f745814da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:24.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.970+0000 7f7457e00640 1 --2- 172.21.15.6:0/927256734 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f745814d5a0 0x7f745814da20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:24.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.970+0000 7f7457e00640 1 -- 172.21.15.6:0/927256734 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f745814c360 con 0x7f743c0039e0 2024-03-04T09:30:24.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.970+0000 7f745ca00640 1 --2- 172.21.15.6:0/927256734 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f745814b5f0 0x7f745814ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:30:24.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.970+0000 7f7457e00640 1 --2- 172.21.15.6:0/927256734 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f743c0039e0 0x7f74580b9230 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7f7448000f80 tx=0x7f74480014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:24.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.970+0000 7f744fe00640 1 -- 172.21.15.6:0/927256734 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7448003970 con 0x7f743c0039e0 2024-03-04T09:30:24.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.970+0000 7f744fe00640 1 -- 172.21.15.6:0/927256734 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7448006b50 con 0x7f743c0039e0 2024-03-04T09:30:24.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.971+0000 7f744fe00640 1 -- 172.21.15.6:0/927256734 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7448007030 con 0x7f743c0039e0 2024-03-04T09:30:24.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.971+0000 7f745d400640 1 -- 172.21.15.6:0/927256734 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f745814c640 con 0x7f743c0039e0 2024-03-04T09:30:24.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.971+0000 7f745d400640 1 -- 172.21.15.6:0/927256734 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f745814cc00 con 0x7f743c0039e0 2024-03-04T09:30:24.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.972+0000 7f745d400640 1 -- 172.21.15.6:0/927256734 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7458072bd0 con 0x7f743c0039e0 2024-03-04T09:30:24.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.973+0000 7f744fe00640 1 -- 172.21.15.6:0/927256734 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f74480071d0 con 0x7f743c0039e0 2024-03-04T09:30:24.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.974+0000 7f744fe00640 1 --2- 172.21.15.6:0/927256734 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f743806b500 0x7f743806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:24.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.974+0000 7f744fe00640 1 -- 172.21.15.6:0/927256734 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f74480abe00 con 0x7f743c0039e0 2024-03-04T09:30:24.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.974+0000 7f7457400640 1 --2- 172.21.15.6:0/927256734 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f743806b500 0x7f743806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:24.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.975+0000 7f7457400640 1 --2- 172.21.15.6:0/927256734 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f743806b500 0x7f743806d9c0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f745814e6a0 tx=0x7f7444009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:24.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:24.979+0000 7f744fe00640 1 -- 172.21.15.6:0/927256734 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7458072bd0 con 0x7f743c0039e0 2024-03-04T09:30:25.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.182+0000 7f745d400640 1 -- 172.21.15.6:0/927256734 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f745804f330 con 0x7f743c0039e0 2024-03-04T09:30:25.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.184+0000 7f744fe00640 1 -- 172.21.15.6:0/927256734 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) v1 ==== 74+0+16560 (secure 0 0 0) 0x7f7448033020 con 0x7f743c0039e0 2024-03-04T09:30:25.182 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:30:25.182 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":94,"fsid":"edce0578-5ae4-4402-b45a-552bdef84d7c","created":"2024-03-04T09:24:54.523780+0000","modified":"2024-03-04T09:30:22.189189+0000","last_up_change":"2024-03-04T09:25:07.089049+0000","last_in_change":"2024-03-04T09:24:57.982077+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T09:25:07.935198+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-03-04T09:30:18.510767+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"93","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-03-04T09:30:19.686114+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"94","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a96574f-81f1-4bc6-8463-757520b84a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6825","nonce":1364449401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6827","nonce":1364449401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6831","nonce":1364449401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6829","nonce":1364449401}]},"public_addr":"172.21.15.6:6825/1364449401","cluster_addr":"172.21.15.6:6827/1364449401","heartbeat_back_addr":"172.21.15.6:6831/1364449401","heartbeat_front_addr":"172.21.15.6:6829/1364449401","state":["exists","up"]},{"osd":1,"uuid":"7d79b9c1-6b87-476a-bc36-7db5ee64cd90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6809","nonce":2494103078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6811","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6813","nonce":2494103078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6821","nonce":2494103078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6817","nonce":2494103078}]},"public_addr":"172.21.15.6:6809/2494103078","cluster_addr":"172.21.15.6:6813/2494103078","heartbeat_back_addr":"172.21.15.6:6821/2494103078","heartbeat_front_addr":"172.21.15.6:6817/2494103078","state":["exists","up"]},{"osd":2,"uuid":"492331b9-4ee9-40b6-bacd-85d44cd6c45a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6812","nonce":1500066681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6816","nonce":1500066681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6824","nonce":1500066681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6820","nonce":1500066681}]},"public_addr":"172.21.15.6:6812/1500066681","cluster_addr":"172.21.15.6:6816/1500066681","heartbeat_back_addr":"172.21.15.6:6824/1500066681","heartbeat_front_addr":"172.21.15.6:6820/1500066681","state":["exists","up"]},{"osd":3,"uuid":"21bb2efe-9ba7-4bdc-bc31-180cd1102e21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6801","nonce":1558806123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6803","nonce":1558806123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6807","nonce":1558806123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6805","nonce":1558806123}]},"public_addr":"172.21.15.6:6801/1558806123","cluster_addr":"172.21.15.6:6803/1558806123","heartbeat_back_addr":"172.21.15.6:6807/1558806123","heartbeat_front_addr":"172.21.15.6:6805/1558806123","state":["exists","up"]},{"osd":4,"uuid":"6e2c4b71-19d9-4737-83d9-558705d8644a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6801","nonce":1526135780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6803","nonce":1526135780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6807","nonce":1526135780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6805","nonce":1526135780}]},"public_addr":"172.21.15.146:6801/1526135780","cluster_addr":"172.21.15.146:6803/1526135780","heartbeat_back_addr":"172.21.15.146:6807/1526135780","heartbeat_front_addr":"172.21.15.146:6805/1526135780","state":["exists","up"]},{"osd":5,"uuid":"fb69c525-f1be-4b7d-ab32-6c29b0d6fddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6809","nonce":460262507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6811","nonce":460262507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6815","nonce":460262507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6813","nonce":460262507}]},"public_addr":"172.21.15.146:6809/460262507","cluster_addr":"172.21.15.146:6811/460262507","heartbeat_back_addr":"172.21.15.146:6815/460262507","heartbeat_front_addr":"172.21.15.146:6813/460262507","state":["exists","up"]},{"osd":6,"uuid":"c32e7fed-370d-4401-8ce0-6a4335b0d168","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6823","nonce":1392897493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6827","nonce":1392897493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6831","nonce":1392897493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6829","nonce":1392897493}]},"public_addr":"172.21.15.146:6823/1392897493","cluster_addr":"172.21.15.146:6827/1392897493","heartbeat_back_addr":"172.21.15.146:6831/1392897493","heartbeat_front_addr":"172.21.15.146:6829/1392897493","state":["exists","up"]},{"osd":7,"uuid":"75a79b44-fa5f-426d-a7d3-71f21f78835c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6817","nonce":3268323156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6819","nonce":3268323156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6825","nonce":3268323156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6822","nonce":3268323156}]},"public_addr":"172.21.15.146:6817/3268323156","cluster_addr":"172.21.15.146:6819/3268323156","heartbeat_back_addr":"172.21.15.146:6825/3268323156","heartbeat_front_addr":"172.21.15.146:6822/3268323156","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.310437+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.295850+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.294674+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.526855+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.300023+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.595160+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.346046+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.558741+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T09:30:25.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.187+0000 7f745d400640 1 -- 172.21.15.6:0/927256734 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f743806b500 msgr2=0x7f743806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:25.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.187+0000 7f745d400640 1 --2- 172.21.15.6:0/927256734 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f743806b500 0x7f743806d9c0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f745814e6a0 tx=0x7f7444009040 comp rx=0 tx=0).stop 2024-03-04T09:30:25.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.187+0000 7f745d400640 1 -- 172.21.15.6:0/927256734 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f743c0039e0 msgr2=0x7f74580b9230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:25.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.187+0000 7f745d400640 1 --2- 172.21.15.6:0/927256734 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f743c0039e0 0x7f74580b9230 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7f7448000f80 tx=0x7f74480014a0 comp rx=0 tx=0).stop 2024-03-04T09:30:25.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.187+0000 7f745d400640 1 -- 172.21.15.6:0/927256734 shutdown_connections 2024-03-04T09:30:25.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.187+0000 7f745d400640 1 --2- 172.21.15.6:0/927256734 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f743806b500 0x7f743806d9c0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:25.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.187+0000 7f745d400640 1 --2- 172.21.15.6:0/927256734 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f745814b5f0 0x7f745814ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:25.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.187+0000 7f745d400640 1 --2- 172.21.15.6:0/927256734 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f745814d5a0 0x7f745814da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:25.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.187+0000 7f745d400640 1 --2- 172.21.15.6:0/927256734 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f743c0039e0 0x7f74580b9230 unknown :-1 s=CLOSED pgs=1081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:25.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.187+0000 7f745d400640 1 -- 172.21.15.6:0/927256734 >> 172.21.15.6:0/927256734 conn(0x7f74580d3d30 msgr2=0x7f74580b1b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:25.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.187+0000 7f745d400640 1 -- 172.21.15.6:0/927256734 shutdown_connections 2024-03-04T09:30:25.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.187+0000 7f745d400640 1 -- 172.21.15.6:0/927256734 wait complete. 2024-03-04T09:30:25.206 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-04T09:30:25.207 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T09:30:25.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.424+0000 7f6265400640 1 Processor -- start 2024-03-04T09:30:25.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.424+0000 7f6265400640 1 -- start start 2024-03-04T09:30:25.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.426+0000 7f6265400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f62600cd060 0x7f6260161ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:25.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.426+0000 7f6265400640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62601627f0 con 0x7f6260162000 2024-03-04T09:30:25.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.426+0000 7f6265400640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f626006c860 con 0x7f626015efc0 2024-03-04T09:30:25.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.426+0000 7f6265400640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f626015f3a0 con 0x7f62600cd060 2024-03-04T09:30:25.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.426+0000 7f6257400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f62600cd060 0x7f6260161ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:25.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.426+0000 7f6257400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f62600cd060 0x7f6260161ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38408/0 (socket says 172.21.15.6:38408) 2024-03-04T09:30:25.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.426+0000 7f6257400640 1 -- 172.21.15.6:0/3073795369 learned_addr learned my addr 172.21.15.6:0/3073795369 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:30:25.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.426+0000 7f6264a00640 1 --1- 172.21.15.6:0/3073795369 >> v1:172.21.15.6:6789/0 conn(0x7f6260162000 0x7f6260070c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:55322/0 (socket says 172.21.15.6:55322) 2024-03-04T09:30:25.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.427+0000 7f6257400640 1 -- 172.21.15.6:0/3073795369 >> v1:172.21.15.146:6789/0 conn(0x7f626015efc0 legacy=0x7f62600cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:25.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.427+0000 7f6257400640 1 -- 172.21.15.6:0/3073795369 >> v1:172.21.15.6:6789/0 conn(0x7f6260162000 legacy=0x7f6260070c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:25.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.427+0000 7f6257400640 1 -- 172.21.15.6:0/3073795369 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f626006cb40 con 0x7f62600cd060 2024-03-04T09:30:25.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.429+0000 7f6257400640 1 --2- 172.21.15.6:0/3073795369 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f62600cd060 0x7f6260161ac0 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f6248009870 tx=0x7f624802b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54d3ca6565cbcf73 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:25.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.429+0000 7f625f400640 1 -- 172.21.15.6:0/3073795369 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6248034070 con 0x7f62600cd060 2024-03-04T09:30:25.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.429+0000 7f625f400640 1 -- 172.21.15.6:0/3073795369 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6248030850 con 0x7f62600cd060 2024-03-04T09:30:25.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.429+0000 7f625f400640 1 -- 172.21.15.6:0/3073795369 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6248030c20 con 0x7f62600cd060 2024-03-04T09:30:25.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.430+0000 7f6265400640 1 -- 172.21.15.6:0/3073795369 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f62600cd060 msgr2=0x7f6260161ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:25.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.430+0000 7f6265400640 1 --2- 172.21.15.6:0/3073795369 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f62600cd060 0x7f6260161ac0 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f6248009870 tx=0x7f624802b990 comp rx=0 tx=0).stop 2024-03-04T09:30:25.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.430+0000 7f6265400640 1 -- 172.21.15.6:0/3073795369 shutdown_connections 2024-03-04T09:30:25.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.430+0000 7f6265400640 1 --2- 172.21.15.6:0/3073795369 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f62600cd060 0x7f6260161ac0 unknown :-1 s=CLOSED pgs=1083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:25.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.430+0000 7f6265400640 1 -- 172.21.15.6:0/3073795369 >> 172.21.15.6:0/3073795369 conn(0x7f626006b3d0 msgr2=0x7f626006b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:25.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.430+0000 7f6265400640 1 -- 172.21.15.6:0/3073795369 shutdown_connections 2024-03-04T09:30:25.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.430+0000 7f6265400640 1 -- 172.21.15.6:0/3073795369 wait complete. 2024-03-04T09:30:25.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.433+0000 7f6265400640 1 Processor -- start 2024-03-04T09:30:25.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.433+0000 7f6265400640 1 -- start start 2024-03-04T09:30:25.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.434+0000 7f6265400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f62600cd060 0x7f6260159410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:25.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.434+0000 7f6265400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f626015efc0 0x7f626015a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:25.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.434+0000 7f6265400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6260162000 0x7f626015aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:25.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.435+0000 7f6265400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6260162470 con 0x7f62600cd060 2024-03-04T09:30:25.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.435+0000 7f6265400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f626006d190 con 0x7f626015efc0 2024-03-04T09:30:25.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.435+0000 7f6265400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f626006d620 con 0x7f6260162000 2024-03-04T09:30:25.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.435+0000 7f625fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f62600cd060 0x7f6260159410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:25.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.435+0000 7f625fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f62600cd060 0x7f6260159410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38424/0 (socket says 172.21.15.6:38424) 2024-03-04T09:30:25.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.435+0000 7f625fe00640 1 -- 172.21.15.6:0/3910065688 learned_addr learned my addr 172.21.15.6:0/3910065688 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:30:25.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.435+0000 7f6257400640 1 --2- 172.21.15.6:0/3910065688 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f626015efc0 0x7f626015a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:25.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.435+0000 7f6264a00640 1 --2- 172.21.15.6:0/3910065688 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6260162000 0x7f626015aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:25.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.435+0000 7f625fe00640 1 -- 172.21.15.6:0/3910065688 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6260162000 msgr2=0x7f626015aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:25.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.435+0000 7f625fe00640 1 --2- 172.21.15.6:0/3910065688 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6260162000 0x7f626015aaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:25.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.435+0000 7f625fe00640 1 -- 172.21.15.6:0/3910065688 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f626015efc0 msgr2=0x7f626015a5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:25.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.435+0000 7f625fe00640 1 --2- 172.21.15.6:0/3910065688 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f626015efc0 0x7f626015a5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:25.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.435+0000 7f625fe00640 1 -- 172.21.15.6:0/3910065688 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6260159950 con 0x7f62600cd060 2024-03-04T09:30:25.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.435+0000 7f6264a00640 1 --2- 172.21.15.6:0/3910065688 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6260162000 0x7f626015aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:30:25.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.436+0000 7f625fe00640 1 --2- 172.21.15.6:0/3910065688 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f62600cd060 0x7f6260159410 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f625000ba50 tx=0x7f625000bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:25.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.436+0000 7f625cc00640 1 -- 172.21.15.6:0/3910065688 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6250019070 con 0x7f62600cd060 2024-03-04T09:30:25.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.436+0000 7f625cc00640 1 -- 172.21.15.6:0/3910065688 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f625000cc20 con 0x7f62600cd060 2024-03-04T09:30:25.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.436+0000 7f6265400640 1 -- 172.21.15.6:0/3910065688 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6260159bf0 con 0x7f62600cd060 2024-03-04T09:30:25.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.436+0000 7f625cc00640 1 -- 172.21.15.6:0/3910065688 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6250014440 con 0x7f62600cd060 2024-03-04T09:30:25.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.436+0000 7f6265400640 1 -- 172.21.15.6:0/3910065688 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62601e7b70 con 0x7f62600cd060 2024-03-04T09:30:25.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.437+0000 7f6265400640 1 -- 172.21.15.6:0/3910065688 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f626004f330 con 0x7f62600cd060 2024-03-04T09:30:25.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.439+0000 7f625cc00640 1 -- 172.21.15.6:0/3910065688 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f625001c070 con 0x7f62600cd060 2024-03-04T09:30:25.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.440+0000 7f625cc00640 1 --2- 172.21.15.6:0/3910065688 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f623c06b4d0 0x7f623c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:25.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.440+0000 7f625cc00640 1 -- 172.21.15.6:0/3910065688 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6250089ec0 con 0x7f62600cd060 2024-03-04T09:30:25.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.441+0000 7f6257400640 1 --2- 172.21.15.6:0/3910065688 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f623c06b4d0 0x7f623c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:25.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.442+0000 7f6257400640 1 --2- 172.21.15.6:0/3910065688 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f623c06b4d0 0x7f623c06d990 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f6248036000 tx=0x7f624803b040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:25.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.445+0000 7f625cc00640 1 -- 172.21.15.6:0/3910065688 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6250058650 con 0x7f62600cd060 2024-03-04T09:30:25.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.641+0000 7f6265400640 1 -- 172.21.15.6:0/3910065688 --> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6260072330 con 0x7f623c06b4d0 2024-03-04T09:30:25.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.662+0000 7f625cc00640 1 -- 172.21.15.6:0/3910065688 <== mgr.4113 v2:172.21.15.146:6832/1788613334 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252084 (secure 0 0 0) 0x7f6260072330 con 0x7f623c06b4d0 2024-03-04T09:30:25.662 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:30:25.663 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-03-04T09:30:25.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.666+0000 7f6265400640 1 -- 172.21.15.6:0/3910065688 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f623c06b4d0 msgr2=0x7f623c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:25.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.666+0000 7f6265400640 1 --2- 172.21.15.6:0/3910065688 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f623c06b4d0 0x7f623c06d990 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f6248036000 tx=0x7f624803b040 comp rx=0 tx=0).stop 2024-03-04T09:30:25.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.666+0000 7f6265400640 1 -- 172.21.15.6:0/3910065688 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f62600cd060 msgr2=0x7f6260159410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:25.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.666+0000 7f6265400640 1 --2- 172.21.15.6:0/3910065688 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f62600cd060 0x7f6260159410 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f625000ba50 tx=0x7f625000bf10 comp rx=0 tx=0).stop 2024-03-04T09:30:25.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.667+0000 7f6265400640 1 -- 172.21.15.6:0/3910065688 shutdown_connections 2024-03-04T09:30:25.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.667+0000 7f6265400640 1 --2- 172.21.15.6:0/3910065688 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f623c06b4d0 0x7f623c06d990 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:25.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.667+0000 7f6265400640 1 --2- 172.21.15.6:0/3910065688 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6260162000 0x7f626015aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:25.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.667+0000 7f6265400640 1 --2- 172.21.15.6:0/3910065688 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f626015efc0 0x7f626015a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:25.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.667+0000 7f6265400640 1 --2- 172.21.15.6:0/3910065688 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f62600cd060 0x7f6260159410 unknown :-1 s=CLOSED pgs=1084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:25.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.667+0000 7f6265400640 1 -- 172.21.15.6:0/3910065688 >> 172.21.15.6:0/3910065688 conn(0x7f626006b3d0 msgr2=0x7f6260070be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:25.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.667+0000 7f6265400640 1 -- 172.21.15.6:0/3910065688 shutdown_connections 2024-03-04T09:30:25.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.667+0000 7f6265400640 1 -- 172.21.15.6:0/3910065688 wait complete. 2024-03-04T09:30:25.685 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":245,"stamp":"2024-03-04T09:30:23.968540+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":136,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":259756,"kb_used_data":10288,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754714964,"statfs":{"total":773094113280,"available":772828123136,"internally_reserved":0,"allocated":10534912,"data_stored":5156476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.811351"},"pg_stats":[{"pgid":"15.3f","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.203689+0000","last_change":"2024-03-04T09:30:21.188523+0000","last_active":"2024-03-04T09:30:22.203689+0000","last_peered":"2024-03-04T09:30:22.203689+0000","last_clean":"2024-03-04T09:30:22.203689+0000","last_became_active":"2024-03-04T09:30:21.188002+0000","last_became_peered":"2024-03-04T09:30:21.188002+0000","last_unstale":"2024-03-04T09:30:22.203689+0000","last_undegraded":"2024-03-04T09:30:22.203689+0000","last_fullsized":"2024-03-04T09:30:22.203689+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:39:07.447728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3e","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.207793+0000","last_change":"2024-03-04T09:30:21.204386+0000","last_active":"2024-03-04T09:30:22.207793+0000","last_peered":"2024-03-04T09:30:22.207793+0000","last_clean":"2024-03-04T09:30:22.207793+0000","last_became_active":"2024-03-04T09:30:21.202933+0000","last_became_peered":"2024-03-04T09:30:21.202933+0000","last_unstale":"2024-03-04T09:30:22.207793+0000","last_undegraded":"2024-03-04T09:30:22.207793+0000","last_fullsized":"2024-03-04T09:30:22.207793+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:17:11.508691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3d","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.203617+0000","last_change":"2024-03-04T09:30:21.195261+0000","last_active":"2024-03-04T09:30:22.203617+0000","last_peered":"2024-03-04T09:30:22.203617+0000","last_clean":"2024-03-04T09:30:22.203617+0000","last_became_active":"2024-03-04T09:30:21.194932+0000","last_became_peered":"2024-03-04T09:30:21.194932+0000","last_unstale":"2024-03-04T09:30:22.203617+0000","last_undegraded":"2024-03-04T09:30:22.203617+0000","last_fullsized":"2024-03-04T09:30:22.203617+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:45:28.640838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3c","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.600344+0000","last_change":"2024-03-04T09:30:21.218498+0000","last_active":"2024-03-04T09:30:22.600344+0000","last_peered":"2024-03-04T09:30:22.600344+0000","last_clean":"2024-03-04T09:30:22.600344+0000","last_became_active":"2024-03-04T09:30:21.217784+0000","last_became_peered":"2024-03-04T09:30:21.217784+0000","last_unstale":"2024-03-04T09:30:22.600344+0000","last_undegraded":"2024-03-04T09:30:22.600344+0000","last_fullsized":"2024-03-04T09:30:22.600344+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:58:07.611882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.3b","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.603491+0000","last_change":"2024-03-04T09:30:21.218477+0000","last_active":"2024-03-04T09:30:22.603491+0000","last_peered":"2024-03-04T09:30:22.603491+0000","last_clean":"2024-03-04T09:30:22.603491+0000","last_became_active":"2024-03-04T09:30:21.218149+0000","last_became_peered":"2024-03-04T09:30:21.218149+0000","last_unstale":"2024-03-04T09:30:22.603491+0000","last_undegraded":"2024-03-04T09:30:22.603491+0000","last_fullsized":"2024-03-04T09:30:22.603491+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:22:44.546244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.3a","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.602235+0000","last_change":"2024-03-04T09:30:21.213698+0000","last_active":"2024-03-04T09:30:22.602235+0000","last_peered":"2024-03-04T09:30:22.602235+0000","last_clean":"2024-03-04T09:30:22.602235+0000","last_became_active":"2024-03-04T09:30:21.213386+0000","last_became_peered":"2024-03-04T09:30:21.213386+0000","last_unstale":"2024-03-04T09:30:22.602235+0000","last_undegraded":"2024-03-04T09:30:22.602235+0000","last_fullsized":"2024-03-04T09:30:22.602235+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:38:54.890507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.39","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.601971+0000","last_change":"2024-03-04T09:30:21.212163+0000","last_active":"2024-03-04T09:30:22.601971+0000","last_peered":"2024-03-04T09:30:22.601971+0000","last_clean":"2024-03-04T09:30:22.601971+0000","last_became_active":"2024-03-04T09:30:21.211896+0000","last_became_peered":"2024-03-04T09:30:21.211896+0000","last_unstale":"2024-03-04T09:30:22.601971+0000","last_undegraded":"2024-03-04T09:30:22.601971+0000","last_fullsized":"2024-03-04T09:30:22.601971+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:47:42.246844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.38","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.207989+0000","last_change":"2024-03-04T09:30:21.202283+0000","last_active":"2024-03-04T09:30:22.207989+0000","last_peered":"2024-03-04T09:30:22.207989+0000","last_clean":"2024-03-04T09:30:22.207989+0000","last_became_active":"2024-03-04T09:30:21.201683+0000","last_became_peered":"2024-03-04T09:30:21.201683+0000","last_unstale":"2024-03-04T09:30:22.207989+0000","last_undegraded":"2024-03-04T09:30:22.207989+0000","last_fullsized":"2024-03-04T09:30:22.207989+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:02:26.808507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.37","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.207193+0000","last_change":"2024-03-04T09:30:21.207087+0000","last_active":"2024-03-04T09:30:22.207193+0000","last_peered":"2024-03-04T09:30:22.207193+0000","last_clean":"2024-03-04T09:30:22.207193+0000","last_became_active":"2024-03-04T09:30:21.205513+0000","last_became_peered":"2024-03-04T09:30:21.205513+0000","last_unstale":"2024-03-04T09:30:22.207193+0000","last_undegraded":"2024-03-04T09:30:22.207193+0000","last_fullsized":"2024-03-04T09:30:22.207193+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:22:44.546244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.36","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.603148+0000","last_change":"2024-03-04T09:30:21.210554+0000","last_active":"2024-03-04T09:30:22.603148+0000","last_peered":"2024-03-04T09:30:22.603148+0000","last_clean":"2024-03-04T09:30:22.603148+0000","last_became_active":"2024-03-04T09:30:21.210151+0000","last_became_peered":"2024-03-04T09:30:21.210151+0000","last_unstale":"2024-03-04T09:30:22.603148+0000","last_undegraded":"2024-03-04T09:30:22.603148+0000","last_fullsized":"2024-03-04T09:30:22.603148+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:35:54.773629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.35","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.601765+0000","last_change":"2024-03-04T09:30:21.211218+0000","last_active":"2024-03-04T09:30:22.601765+0000","last_peered":"2024-03-04T09:30:22.601765+0000","last_clean":"2024-03-04T09:30:22.601765+0000","last_became_active":"2024-03-04T09:30:21.207302+0000","last_became_peered":"2024-03-04T09:30:21.207302+0000","last_unstale":"2024-03-04T09:30:22.601765+0000","last_undegraded":"2024-03-04T09:30:22.601765+0000","last_fullsized":"2024-03-04T09:30:22.601765+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:22:44.546244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.34","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.207209+0000","last_change":"2024-03-04T09:30:21.211419+0000","last_active":"2024-03-04T09:30:22.207209+0000","last_peered":"2024-03-04T09:30:22.207209+0000","last_clean":"2024-03-04T09:30:22.207209+0000","last_became_active":"2024-03-04T09:30:21.211060+0000","last_became_peered":"2024-03-04T09:30:21.211060+0000","last_unstale":"2024-03-04T09:30:22.207209+0000","last_undegraded":"2024-03-04T09:30:22.207209+0000","last_fullsized":"2024-03-04T09:30:22.207209+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:47:42.246844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.33","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.599833+0000","last_change":"2024-03-04T09:30:21.207596+0000","last_active":"2024-03-04T09:30:22.599833+0000","last_peered":"2024-03-04T09:30:22.599833+0000","last_clean":"2024-03-04T09:30:22.599833+0000","last_became_active":"2024-03-04T09:30:21.207323+0000","last_became_peered":"2024-03-04T09:30:21.207323+0000","last_unstale":"2024-03-04T09:30:22.599833+0000","last_undegraded":"2024-03-04T09:30:22.599833+0000","last_fullsized":"2024-03-04T09:30:22.599833+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:53:52.836749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.32","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.211090+0000","last_change":"2024-03-04T09:30:21.210114+0000","last_active":"2024-03-04T09:30:22.211090+0000","last_peered":"2024-03-04T09:30:22.211090+0000","last_clean":"2024-03-04T09:30:22.211090+0000","last_became_active":"2024-03-04T09:30:21.209823+0000","last_became_peered":"2024-03-04T09:30:21.209823+0000","last_unstale":"2024-03-04T09:30:22.211090+0000","last_undegraded":"2024-03-04T09:30:22.211090+0000","last_fullsized":"2024-03-04T09:30:22.211090+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:49:53.245627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.31","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.599906+0000","last_change":"2024-03-04T09:30:21.209101+0000","last_active":"2024-03-04T09:30:22.599906+0000","last_peered":"2024-03-04T09:30:22.599906+0000","last_clean":"2024-03-04T09:30:22.599906+0000","last_became_active":"2024-03-04T09:30:21.208703+0000","last_became_peered":"2024-03-04T09:30:21.208703+0000","last_unstale":"2024-03-04T09:30:22.599906+0000","last_undegraded":"2024-03-04T09:30:22.599906+0000","last_fullsized":"2024-03-04T09:30:22.599906+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:25:05.050243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.30","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.601160+0000","last_change":"2024-03-04T09:30:21.210486+0000","last_active":"2024-03-04T09:30:22.601160+0000","last_peered":"2024-03-04T09:30:22.601160+0000","last_clean":"2024-03-04T09:30:22.601160+0000","last_became_active":"2024-03-04T09:30:21.207407+0000","last_became_peered":"2024-03-04T09:30:21.207407+0000","last_unstale":"2024-03-04T09:30:22.601160+0000","last_undegraded":"2024-03-04T09:30:22.601160+0000","last_fullsized":"2024-03-04T09:30:22.601160+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:05:34.112195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.2f","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.208448+0000","last_change":"2024-03-04T09:30:21.199293+0000","last_active":"2024-03-04T09:30:22.208448+0000","last_peered":"2024-03-04T09:30:22.208448+0000","last_clean":"2024-03-04T09:30:22.208448+0000","last_became_active":"2024-03-04T09:30:21.199020+0000","last_became_peered":"2024-03-04T09:30:21.199020+0000","last_unstale":"2024-03-04T09:30:22.208448+0000","last_undegraded":"2024-03-04T09:30:22.208448+0000","last_fullsized":"2024-03-04T09:30:22.208448+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:49:16.693009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.2e","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.600547+0000","last_change":"2024-03-04T09:30:21.214466+0000","last_active":"2024-03-04T09:30:22.600547+0000","last_peered":"2024-03-04T09:30:22.600547+0000","last_clean":"2024-03-04T09:30:22.600547+0000","last_became_active":"2024-03-04T09:30:21.214193+0000","last_became_peered":"2024-03-04T09:30:21.214193+0000","last_unstale":"2024-03-04T09:30:22.600547+0000","last_undegraded":"2024-03-04T09:30:22.600547+0000","last_fullsized":"2024-03-04T09:30:22.600547+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:40:13.536981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.2d","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.211447+0000","last_change":"2024-03-04T09:30:21.219224+0000","last_active":"2024-03-04T09:30:22.211447+0000","last_peered":"2024-03-04T09:30:22.211447+0000","last_clean":"2024-03-04T09:30:22.211447+0000","last_became_active":"2024-03-04T09:30:21.218894+0000","last_became_peered":"2024-03-04T09:30:21.218894+0000","last_unstale":"2024-03-04T09:30:22.211447+0000","last_undegraded":"2024-03-04T09:30:22.211447+0000","last_fullsized":"2024-03-04T09:30:22.211447+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:27:46.546965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2c","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.207619+0000","last_change":"2024-03-04T09:30:21.210653+0000","last_active":"2024-03-04T09:30:22.207619+0000","last_peered":"2024-03-04T09:30:22.207619+0000","last_clean":"2024-03-04T09:30:22.207619+0000","last_became_active":"2024-03-04T09:30:21.210251+0000","last_became_peered":"2024-03-04T09:30:21.210251+0000","last_unstale":"2024-03-04T09:30:22.207619+0000","last_undegraded":"2024-03-04T09:30:22.207619+0000","last_fullsized":"2024-03-04T09:30:22.207619+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:53:52.836749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.2b","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.211219+0000","last_change":"2024-03-04T09:30:21.215051+0000","last_active":"2024-03-04T09:30:22.211219+0000","last_peered":"2024-03-04T09:30:22.211219+0000","last_clean":"2024-03-04T09:30:22.211219+0000","last_became_active":"2024-03-04T09:30:21.214690+0000","last_became_peered":"2024-03-04T09:30:21.214690+0000","last_unstale":"2024-03-04T09:30:22.211219+0000","last_undegraded":"2024-03-04T09:30:22.211219+0000","last_fullsized":"2024-03-04T09:30:22.211219+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:31:50.883753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2a","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.208053+0000","last_change":"2024-03-04T09:30:21.203403+0000","last_active":"2024-03-04T09:30:22.208053+0000","last_peered":"2024-03-04T09:30:22.208053+0000","last_clean":"2024-03-04T09:30:22.208053+0000","last_became_active":"2024-03-04T09:30:21.202934+0000","last_became_peered":"2024-03-04T09:30:21.202934+0000","last_unstale":"2024-03-04T09:30:22.208053+0000","last_undegraded":"2024-03-04T09:30:22.208053+0000","last_fullsized":"2024-03-04T09:30:22.208053+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:48:19.061200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.29","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.207876+0000","last_change":"2024-03-04T09:30:21.198463+0000","last_active":"2024-03-04T09:30:22.207876+0000","last_peered":"2024-03-04T09:30:22.207876+0000","last_clean":"2024-03-04T09:30:22.207876+0000","last_became_active":"2024-03-04T09:30:21.198172+0000","last_became_peered":"2024-03-04T09:30:21.198172+0000","last_unstale":"2024-03-04T09:30:22.207876+0000","last_undegraded":"2024-03-04T09:30:22.207876+0000","last_fullsized":"2024-03-04T09:30:22.207876+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:35:41.436658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.28","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.601374+0000","last_change":"2024-03-04T09:30:21.210170+0000","last_active":"2024-03-04T09:30:22.601374+0000","last_peered":"2024-03-04T09:30:22.601374+0000","last_clean":"2024-03-04T09:30:22.601374+0000","last_became_active":"2024-03-04T09:30:21.209873+0000","last_became_peered":"2024-03-04T09:30:21.209873+0000","last_unstale":"2024-03-04T09:30:22.601374+0000","last_undegraded":"2024-03-04T09:30:22.601374+0000","last_fullsized":"2024-03-04T09:30:22.601374+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:35:54.773629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.27","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.602868+0000","last_change":"2024-03-04T09:30:21.214806+0000","last_active":"2024-03-04T09:30:22.602868+0000","last_peered":"2024-03-04T09:30:22.602868+0000","last_clean":"2024-03-04T09:30:22.602868+0000","last_became_active":"2024-03-04T09:30:21.214161+0000","last_became_peered":"2024-03-04T09:30:21.214161+0000","last_unstale":"2024-03-04T09:30:22.602868+0000","last_undegraded":"2024-03-04T09:30:22.602868+0000","last_fullsized":"2024-03-04T09:30:22.602868+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:05:34.112195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.26","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.208046+0000","last_change":"2024-03-04T09:30:21.194645+0000","last_active":"2024-03-04T09:30:22.208046+0000","last_peered":"2024-03-04T09:30:22.208046+0000","last_clean":"2024-03-04T09:30:22.208046+0000","last_became_active":"2024-03-04T09:30:21.194362+0000","last_became_peered":"2024-03-04T09:30:21.194362+0000","last_unstale":"2024-03-04T09:30:22.208046+0000","last_undegraded":"2024-03-04T09:30:22.208046+0000","last_fullsized":"2024-03-04T09:30:22.208046+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:30:23.608760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.25","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.602145+0000","last_change":"2024-03-04T09:30:21.220051+0000","last_active":"2024-03-04T09:30:22.602145+0000","last_peered":"2024-03-04T09:30:22.602145+0000","last_clean":"2024-03-04T09:30:22.602145+0000","last_became_active":"2024-03-04T09:30:21.219729+0000","last_became_peered":"2024-03-04T09:30:21.219729+0000","last_unstale":"2024-03-04T09:30:22.602145+0000","last_undegraded":"2024-03-04T09:30:22.602145+0000","last_fullsized":"2024-03-04T09:30:22.602145+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:04:51.260343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.24","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.604357+0000","last_change":"2024-03-04T09:30:21.214188+0000","last_active":"2024-03-04T09:30:22.604357+0000","last_peered":"2024-03-04T09:30:22.604357+0000","last_clean":"2024-03-04T09:30:22.604357+0000","last_became_active":"2024-03-04T09:30:21.213867+0000","last_became_peered":"2024-03-04T09:30:21.213867+0000","last_unstale":"2024-03-04T09:30:22.604357+0000","last_undegraded":"2024-03-04T09:30:22.604357+0000","last_fullsized":"2024-03-04T09:30:22.604357+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:47:42.246844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.23","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.207656+0000","last_change":"2024-03-04T09:30:21.211260+0000","last_active":"2024-03-04T09:30:22.207656+0000","last_peered":"2024-03-04T09:30:22.207656+0000","last_clean":"2024-03-04T09:30:22.207656+0000","last_became_active":"2024-03-04T09:30:21.210927+0000","last_became_peered":"2024-03-04T09:30:21.210927+0000","last_unstale":"2024-03-04T09:30:22.207656+0000","last_undegraded":"2024-03-04T09:30:22.207656+0000","last_fullsized":"2024-03-04T09:30:22.207656+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:00:49.856992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.22","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.211146+0000","last_change":"2024-03-04T09:30:21.221279+0000","last_active":"2024-03-04T09:30:22.211146+0000","last_peered":"2024-03-04T09:30:22.211146+0000","last_clean":"2024-03-04T09:30:22.211146+0000","last_became_active":"2024-03-04T09:30:21.220582+0000","last_became_peered":"2024-03-04T09:30:21.220582+0000","last_unstale":"2024-03-04T09:30:22.211146+0000","last_undegraded":"2024-03-04T09:30:22.211146+0000","last_fullsized":"2024-03-04T09:30:22.211146+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:12:10.257211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.21","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.600391+0000","last_change":"2024-03-04T09:30:21.213950+0000","last_active":"2024-03-04T09:30:22.600391+0000","last_peered":"2024-03-04T09:30:22.600391+0000","last_clean":"2024-03-04T09:30:22.600391+0000","last_became_active":"2024-03-04T09:30:21.213595+0000","last_became_peered":"2024-03-04T09:30:21.213595+0000","last_unstale":"2024-03-04T09:30:22.600391+0000","last_undegraded":"2024-03-04T09:30:22.600391+0000","last_fullsized":"2024-03-04T09:30:22.600391+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:52:21.419597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.20","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.202906+0000","last_change":"2024-03-04T09:30:21.191402+0000","last_active":"2024-03-04T09:30:22.202906+0000","last_peered":"2024-03-04T09:30:22.202906+0000","last_clean":"2024-03-04T09:30:22.202906+0000","last_became_active":"2024-03-04T09:30:21.190931+0000","last_became_peered":"2024-03-04T09:30:21.190931+0000","last_unstale":"2024-03-04T09:30:22.202906+0000","last_undegraded":"2024-03-04T09:30:22.202906+0000","last_fullsized":"2024-03-04T09:30:22.202906+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:55:38.377212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.211543+0000","last_change":"2024-03-04T09:30:21.205326+0000","last_active":"2024-03-04T09:30:22.211543+0000","last_peered":"2024-03-04T09:30:22.211543+0000","last_clean":"2024-03-04T09:30:22.211543+0000","last_became_active":"2024-03-04T09:30:21.205025+0000","last_became_peered":"2024-03-04T09:30:21.205025+0000","last_unstale":"2024-03-04T09:30:22.211543+0000","last_undegraded":"2024-03-04T09:30:22.211543+0000","last_fullsized":"2024-03-04T09:30:22.211543+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:09:41.869707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1f","version":"93'4","reported_seq":18,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.208447+0000","last_change":"2024-03-04T09:30:20.202696+0000","last_active":"2024-03-04T09:30:22.208447+0000","last_peered":"2024-03-04T09:30:22.208447+0000","last_clean":"2024-03-04T09:30:22.208447+0000","last_became_active":"2024-03-04T09:30:20.201031+0000","last_became_peered":"2024-03-04T09:30:20.201031+0000","last_unstale":"2024-03-04T09:30:22.208447+0000","last_undegraded":"2024-03-04T09:30:22.208447+0000","last_fullsized":"2024-03-04T09:30:22.208447+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_clean_scrub_stamp":"2024-03-04T09:30:19.137440+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:35:53.756969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.600779+0000","last_change":"2024-03-04T09:30:21.208078+0000","last_active":"2024-03-04T09:30:22.600779+0000","last_peered":"2024-03-04T09:30:22.600779+0000","last_clean":"2024-03-04T09:30:22.600779+0000","last_became_active":"2024-03-04T09:30:21.207737+0000","last_became_peered":"2024-03-04T09:30:21.207737+0000","last_unstale":"2024-03-04T09:30:22.600779+0000","last_undegraded":"2024-03-04T09:30:22.600779+0000","last_fullsized":"2024-03-04T09:30:22.600779+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:06:54.905510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1e","version":"93'1","reported_seq":15,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.602684+0000","last_change":"2024-03-04T09:30:20.202396+0000","last_active":"2024-03-04T09:30:22.602684+0000","last_peered":"2024-03-04T09:30:22.602684+0000","last_clean":"2024-03-04T09:30:22.602684+0000","last_became_active":"2024-03-04T09:30:20.202080+0000","last_became_peered":"2024-03-04T09:30:20.202080+0000","last_unstale":"2024-03-04T09:30:22.602684+0000","last_undegraded":"2024-03-04T09:30:22.602684+0000","last_fullsized":"2024-03-04T09:30:22.602684+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_clean_scrub_stamp":"2024-03-04T09:30:19.137440+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:43:31.237510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.203381+0000","last_change":"2024-03-04T09:30:21.208566+0000","last_active":"2024-03-04T09:30:22.203381+0000","last_peered":"2024-03-04T09:30:22.203381+0000","last_clean":"2024-03-04T09:30:22.203381+0000","last_became_active":"2024-03-04T09:30:21.207592+0000","last_became_peered":"2024-03-04T09:30:21.207592+0000","last_unstale":"2024-03-04T09:30:22.203381+0000","last_undegraded":"2024-03-04T09:30:22.203381+0000","last_fullsized":"2024-03-04T09:30:22.203381+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:49:32.875495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1d","version":"93'2","reported_seq":16,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.207671+0000","last_change":"2024-03-04T09:30:20.199898+0000","last_active":"2024-03-04T09:30:22.207671+0000","last_peered":"2024-03-04T09:30:22.207671+0000","last_clean":"2024-03-04T09:30:22.207671+0000","last_became_active":"2024-03-04T09:30:20.199607+0000","last_became_peered":"2024-03-04T09:30:20.199607+0000","last_unstale":"2024-03-04T09:30:22.207671+0000","last_undegraded":"2024-03-04T09:30:22.207671+0000","last_fullsized":"2024-03-04T09:30:22.207671+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_clean_scrub_stamp":"2024-03-04T09:30:19.137440+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:59:30.574909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.206573+0000","last_change":"2024-03-04T09:30:21.214934+0000","last_active":"2024-03-04T09:30:22.206573+0000","last_peered":"2024-03-04T09:30:22.206573+0000","last_clean":"2024-03-04T09:30:22.206573+0000","last_became_active":"2024-03-04T09:30:21.214642+0000","last_became_peered":"2024-03-04T09:30:21.214642+0000","last_unstale":"2024-03-04T09:30:22.206573+0000","last_undegraded":"2024-03-04T09:30:22.206573+0000","last_fullsized":"2024-03-04T09:30:22.206573+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:05:34.112195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1c","version":"93'2","reported_seq":16,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.207658+0000","last_change":"2024-03-04T09:30:20.201824+0000","last_active":"2024-03-04T09:30:22.207658+0000","last_peered":"2024-03-04T09:30:22.207658+0000","last_clean":"2024-03-04T09:30:22.207658+0000","last_became_active":"2024-03-04T09:30:20.201263+0000","last_became_peered":"2024-03-04T09:30:20.201263+0000","last_unstale":"2024-03-04T09:30:22.207658+0000","last_undegraded":"2024-03-04T09:30:22.207658+0000","last_fullsized":"2024-03-04T09:30:22.207658+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_clean_scrub_stamp":"2024-03-04T09:30:19.137440+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:26:11.379928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.208700+0000","last_change":"2024-03-04T09:30:21.212815+0000","last_active":"2024-03-04T09:30:22.208700+0000","last_peered":"2024-03-04T09:30:22.208700+0000","last_clean":"2024-03-04T09:30:22.208700+0000","last_became_active":"2024-03-04T09:30:21.212499+0000","last_became_peered":"2024-03-04T09:30:21.212499+0000","last_unstale":"2024-03-04T09:30:22.208700+0000","last_undegraded":"2024-03-04T09:30:22.208700+0000","last_fullsized":"2024-03-04T09:30:22.208700+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:40:36.757300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1b","version":"93'1","reported_seq":15,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.203260+0000","last_change":"2024-03-04T09:30:20.196754+0000","last_active":"2024-03-04T09:30:22.203260+0000","last_peered":"2024-03-04T09:30:22.203260+0000","last_clean":"2024-03-04T09:30:22.203260+0000","last_became_active":"2024-03-04T09:30:20.196436+0000","last_became_peered":"2024-03-04T09:30:20.196436+0000","last_unstale":"2024-03-04T09:30:22.203260+0000","last_undegraded":"2024-03-04T09:30:22.203260+0000","last_fullsized":"2024-03-04T09:30:22.203260+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_clean_scrub_stamp":"2024-03-04T09:30:19.137440+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:27:45.530306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.603485+0000","last_change":"2024-03-04T09:30:21.209123+0000","last_active":"2024-03-04T09:30:22.603485+0000","last_peered":"2024-03-04T09:30:22.603485+0000","last_clean":"2024-03-04T09:30:22.603485+0000","last_became_active":"2024-03-04T09:30:21.207804+0000","last_became_peered":"2024-03-04T09:30:21.207804+0000","last_unstale":"2024-03-04T09:30:22.603485+0000","last_undegraded":"2024-03-04T09:30:22.603485+0000","last_fullsized":"2024-03-04T09:30:22.603485+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:25:05.050243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1a","version":"93'1","reported_seq":15,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.207584+0000","last_change":"2024-03-04T09:30:20.205820+0000","last_active":"2024-03-04T09:30:22.207584+0000","last_peered":"2024-03-04T09:30:22.207584+0000","last_clean":"2024-03-04T09:30:22.207584+0000","last_became_active":"2024-03-04T09:30:20.205544+0000","last_became_peered":"2024-03-04T09:30:20.205544+0000","last_unstale":"2024-03-04T09:30:22.207584+0000","last_undegraded":"2024-03-04T09:30:22.207584+0000","last_fullsized":"2024-03-04T09:30:22.207584+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_clean_scrub_stamp":"2024-03-04T09:30:19.137440+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:05:33.095535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.599877+0000","last_change":"2024-03-04T09:30:21.209440+0000","last_active":"2024-03-04T09:30:22.599877+0000","last_peered":"2024-03-04T09:30:22.599877+0000","last_clean":"2024-03-04T09:30:22.599877+0000","last_became_active":"2024-03-04T09:30:21.209149+0000","last_became_peered":"2024-03-04T09:30:21.209149+0000","last_unstale":"2024-03-04T09:30:22.599877+0000","last_undegraded":"2024-03-04T09:30:22.599877+0000","last_fullsized":"2024-03-04T09:30:22.599877+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:38:54.890507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":14,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.601645+0000","last_change":"2024-03-04T09:30:20.204817+0000","last_active":"2024-03-04T09:30:22.601645+0000","last_peered":"2024-03-04T09:30:22.601645+0000","last_clean":"2024-03-04T09:30:22.601645+0000","last_became_active":"2024-03-04T09:30:20.204526+0000","last_became_peered":"2024-03-04T09:30:20.204526+0000","last_unstale":"2024-03-04T09:30:22.601645+0000","last_undegraded":"2024-03-04T09:30:22.601645+0000","last_fullsized":"2024-03-04T09:30:22.601645+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_clean_scrub_stamp":"2024-03-04T09:30:19.137440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:31:56.088975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.206665+0000","last_change":"2024-03-04T09:30:21.215369+0000","last_active":"2024-03-04T09:30:22.206665+0000","last_peered":"2024-03-04T09:30:22.206665+0000","last_clean":"2024-03-04T09:30:22.206665+0000","last_became_active":"2024-03-04T09:30:21.215098+0000","last_became_peered":"2024-03-04T09:30:21.215098+0000","last_unstale":"2024-03-04T09:30:22.206665+0000","last_undegraded":"2024-03-04T09:30:22.206665+0000","last_fullsized":"2024-03-04T09:30:22.206665+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:35:54.773629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":14,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.202898+0000","last_change":"2024-03-04T09:30:20.188458+0000","last_active":"2024-03-04T09:30:22.202898+0000","last_peered":"2024-03-04T09:30:22.202898+0000","last_clean":"2024-03-04T09:30:22.202898+0000","last_became_active":"2024-03-04T09:30:20.188160+0000","last_became_peered":"2024-03-04T09:30:20.188160+0000","last_unstale":"2024-03-04T09:30:22.202898+0000","last_undegraded":"2024-03-04T09:30:22.202898+0000","last_fullsized":"2024-03-04T09:30:22.202898+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_clean_scrub_stamp":"2024-03-04T09:30:19.137440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:44:14.674247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.208239+0000","last_change":"2024-03-04T09:30:21.206057+0000","last_active":"2024-03-04T09:30:22.208239+0000","last_peered":"2024-03-04T09:30:22.208239+0000","last_clean":"2024-03-04T09:30:22.208239+0000","last_became_active":"2024-03-04T09:30:21.205516+0000","last_became_peered":"2024-03-04T09:30:21.205516+0000","last_unstale":"2024-03-04T09:30:22.208239+0000","last_undegraded":"2024-03-04T09:30:22.208239+0000","last_fullsized":"2024-03-04T09:30:22.208239+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:47:59.445932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":14,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.203793+0000","last_change":"2024-03-04T09:30:20.203539+0000","last_active":"2024-03-04T09:30:22.203793+0000","last_peered":"2024-03-04T09:30:22.203793+0000","last_clean":"2024-03-04T09:30:22.203793+0000","last_became_active":"2024-03-04T09:30:20.202487+0000","last_became_peered":"2024-03-04T09:30:20.202487+0000","last_unstale":"2024-03-04T09:30:22.203793+0000","last_undegraded":"2024-03-04T09:30:22.203793+0000","last_fullsized":"2024-03-04T09:30:22.203793+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_clean_scrub_stamp":"2024-03-04T09:30:19.137440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:44:40.582064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.207324+0000","last_change":"2024-03-04T09:30:21.214060+0000","last_active":"2024-03-04T09:30:22.207324+0000","last_peered":"2024-03-04T09:30:22.207324+0000","last_clean":"2024-03-04T09:30:22.207324+0000","last_became_active":"2024-03-04T09:30:21.212861+0000","last_became_peered":"2024-03-04T09:30:21.212861+0000","last_unstale":"2024-03-04T09:30:22.207324+0000","last_undegraded":"2024-03-04T09:30:22.207324+0000","last_fullsized":"2024-03-04T09:30:22.207324+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:04:51.260343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":14,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.600464+0000","last_change":"2024-03-04T09:30:20.203934+0000","last_active":"2024-03-04T09:30:22.600464+0000","last_peered":"2024-03-04T09:30:22.600464+0000","last_clean":"2024-03-04T09:30:22.600464+0000","last_became_active":"2024-03-04T09:30:20.203656+0000","last_became_peered":"2024-03-04T09:30:20.203656+0000","last_unstale":"2024-03-04T09:30:22.600464+0000","last_undegraded":"2024-03-04T09:30:22.600464+0000","last_fullsized":"2024-03-04T09:30:22.600464+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_clean_scrub_stamp":"2024-03-04T09:30:19.137440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:59:30.574909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.604462+0000","last_change":"2024-03-04T09:30:21.214168+0000","last_active":"2024-03-04T09:30:22.604462+0000","last_peered":"2024-03-04T09:30:22.604462+0000","last_clean":"2024-03-04T09:30:22.604462+0000","last_became_active":"2024-03-04T09:30:21.213406+0000","last_became_peered":"2024-03-04T09:30:21.213406+0000","last_unstale":"2024-03-04T09:30:22.604462+0000","last_undegraded":"2024-03-04T09:30:22.604462+0000","last_fullsized":"2024-03-04T09:30:22.604462+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:53:52.836749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.15","version":"93'3","reported_seq":17,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.602467+0000","last_change":"2024-03-04T09:30:20.204159+0000","last_active":"2024-03-04T09:30:22.602467+0000","last_peered":"2024-03-04T09:30:22.602467+0000","last_clean":"2024-03-04T09:30:22.602467+0000","last_became_active":"2024-03-04T09:30:20.203855+0000","last_became_peered":"2024-03-04T09:30:20.203855+0000","last_unstale":"2024-03-04T09:30:22.602467+0000","last_undegraded":"2024-03-04T09:30:22.602467+0000","last_fullsized":"2024-03-04T09:30:22.602467+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_clean_scrub_stamp":"2024-03-04T09:30:19.137440+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:15:51.774866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.602167+0000","last_change":"2024-03-04T09:30:21.203568+0000","last_active":"2024-03-04T09:30:22.602167+0000","last_peered":"2024-03-04T09:30:22.602167+0000","last_clean":"2024-03-04T09:30:22.602167+0000","last_became_active":"2024-03-04T09:30:21.202806+0000","last_became_peered":"2024-03-04T09:30:21.202806+0000","last_unstale":"2024-03-04T09:30:22.602167+0000","last_undegraded":"2024-03-04T09:30:22.602167+0000","last_fullsized":"2024-03-04T09:30:22.602167+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:53:52.836749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"93'6","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.207893+0000","last_change":"2024-03-04T09:30:20.203706+0000","last_active":"2024-03-04T09:30:22.207893+0000","last_peered":"2024-03-04T09:30:22.207893+0000","last_clean":"2024-03-04T09:30:22.207893+0000","last_became_active":"2024-03-04T09:30:20.203345+0000","last_became_peered":"2024-03-04T09:30:20.203345+0000","last_unstale":"2024-03-04T09:30:22.207893+0000","last_undegraded":"2024-03-04T09:30:22.207893+0000","last_fullsized":"2024-03-04T09:30:22.207893+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_clean_scrub_stamp":"2024-03-04T09:30:19.137440+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:43:31.237510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.599878+0000","last_change":"2024-03-04T09:30:21.202680+0000","last_active":"2024-03-04T09:30:22.599878+0000","last_peered":"2024-03-04T09:30:22.599878+0000","last_clean":"2024-03-04T09:30:22.599878+0000","last_became_active":"2024-03-04T09:30:21.202453+0000","last_became_peered":"2024-03-04T09:30:21.202453+0000","last_unstale":"2024-03-04T09:30:22.599878+0000","last_undegraded":"2024-03-04T09:30:22.599878+0000","last_fullsized":"2024-03-04T09:30:22.599878+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:30:09.483436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.13","version":"93'2","reported_seq":16,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.211387+0000","last_change":"2024-03-04T09:30:20.202977+0000","last_active":"2024-03-04T09:30:22.211387+0000","last_peered":"2024-03-04T09:30:22.211387+0000","last_clean":"2024-03-04T09:30:22.211387+0000","last_became_active":"2024-03-04T09:30:20.201161+0000","last_became_peered":"2024-03-04T09:30:20.201161+0000","last_unstale":"2024-03-04T09:30:22.211387+0000","last_undegraded":"2024-03-04T09:30:22.211387+0000","last_fullsized":"2024-03-04T09:30:22.211387+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_clean_scrub_stamp":"2024-03-04T09:30:19.137440+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:39:23.646435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.203295+0000","last_change":"2024-03-04T09:30:21.210999+0000","last_active":"2024-03-04T09:30:22.203295+0000","last_peered":"2024-03-04T09:30:22.203295+0000","last_clean":"2024-03-04T09:30:22.203295+0000","last_became_active":"2024-03-04T09:30:21.207217+0000","last_became_peered":"2024-03-04T09:30:21.207217+0000","last_unstale":"2024-03-04T09:30:22.203295+0000","last_undegraded":"2024-03-04T09:30:22.203295+0000","last_fullsized":"2024-03-04T09:30:22.203295+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:01:32.367615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":14,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.211507+0000","last_change":"2024-03-04T09:30:20.184266+0000","last_active":"2024-03-04T09:30:22.211507+0000","last_peered":"2024-03-04T09:30:22.211507+0000","last_clean":"2024-03-04T09:30:22.211507+0000","last_became_active":"2024-03-04T09:30:20.183857+0000","last_became_peered":"2024-03-04T09:30:20.183857+0000","last_unstale":"2024-03-04T09:30:22.211507+0000","last_undegraded":"2024-03-04T09:30:22.211507+0000","last_fullsized":"2024-03-04T09:30:22.211507+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_clean_scrub_stamp":"2024-03-04T09:30:19.137440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:26:40.873839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.205422+0000","last_change":"2024-03-04T09:30:21.215813+0000","last_active":"2024-03-04T09:30:22.205422+0000","last_peered":"2024-03-04T09:30:22.205422+0000","last_clean":"2024-03-04T09:30:22.205422+0000","last_became_active":"2024-03-04T09:30:21.215547+0000","last_became_peered":"2024-03-04T09:30:21.215547+0000","last_unstale":"2024-03-04T09:30:22.205422+0000","last_undegraded":"2024-03-04T09:30:22.205422+0000","last_fullsized":"2024-03-04T09:30:22.205422+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:51:18.057309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":14,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.205777+0000","last_change":"2024-03-04T09:30:20.206681+0000","last_active":"2024-03-04T09:30:22.205777+0000","last_peered":"2024-03-04T09:30:22.205777+0000","last_clean":"2024-03-04T09:30:22.205777+0000","last_became_active":"2024-03-04T09:30:20.206212+0000","last_became_peered":"2024-03-04T09:30:20.206212+0000","last_unstale":"2024-03-04T09:30:22.205777+0000","last_undegraded":"2024-03-04T09:30:22.205777+0000","last_fullsized":"2024-03-04T09:30:22.205777+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_clean_scrub_stamp":"2024-03-04T09:30:19.137440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:31:56.088975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.210983+0000","last_change":"2024-03-04T09:30:21.217458+0000","last_active":"2024-03-04T09:30:22.210983+0000","last_peered":"2024-03-04T09:30:22.210983+0000","last_clean":"2024-03-04T09:30:22.210983+0000","last_became_active":"2024-03-04T09:30:21.217201+0000","last_became_peered":"2024-03-04T09:30:21.217201+0000","last_unstale":"2024-03-04T09:30:22.210983+0000","last_undegraded":"2024-03-04T09:30:22.210983+0000","last_fullsized":"2024-03-04T09:30:22.210983+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:18:40.161366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.10","version":"93'8","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.203115+0000","last_change":"2024-03-04T09:30:20.193928+0000","last_active":"2024-03-04T09:30:22.203115+0000","last_peered":"2024-03-04T09:30:22.203115+0000","last_clean":"2024-03-04T09:30:22.203115+0000","last_became_active":"2024-03-04T09:30:20.193601+0000","last_became_peered":"2024-03-04T09:30:20.193601+0000","last_unstale":"2024-03-04T09:30:22.203115+0000","last_undegraded":"2024-03-04T09:30:22.203115+0000","last_fullsized":"2024-03-04T09:30:22.203115+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_clean_scrub_stamp":"2024-03-04T09:30:19.137440+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:33:41.777208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.604363+0000","last_change":"2024-03-04T09:30:21.209945+0000","last_active":"2024-03-04T09:30:22.604363+0000","last_peered":"2024-03-04T09:30:22.604363+0000","last_clean":"2024-03-04T09:30:22.604363+0000","last_became_active":"2024-03-04T09:30:21.209231+0000","last_became_peered":"2024-03-04T09:30:21.209231+0000","last_unstale":"2024-03-04T09:30:22.604363+0000","last_undegraded":"2024-03-04T09:30:22.604363+0000","last_fullsized":"2024-03-04T09:30:22.604363+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:04:51.260343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.e","version":"93'2","reported_seq":16,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.206181+0000","last_change":"2024-03-04T09:30:20.197183+0000","last_active":"2024-03-04T09:30:22.206181+0000","last_peered":"2024-03-04T09:30:22.206181+0000","last_clean":"2024-03-04T09:30:22.206181+0000","last_became_active":"2024-03-04T09:30:20.196884+0000","last_became_peered":"2024-03-04T09:30:20.196884+0000","last_unstale":"2024-03-04T09:30:22.206181+0000","last_undegraded":"2024-03-04T09:30:22.206181+0000","last_fullsized":"2024-03-04T09:30:22.206181+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_clean_scrub_stamp":"2024-03-04T09:30:19.137440+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:42:56.761983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.208741+0000","last_change":"2024-03-04T09:30:21.207297+0000","last_active":"2024-03-04T09:30:22.208741+0000","last_peered":"2024-03-04T09:30:22.208741+0000","last_clean":"2024-03-04T09:30:22.208741+0000","last_became_active":"2024-03-04T09:30:21.206895+0000","last_became_peered":"2024-03-04T09:30:21.206895+0000","last_unstale":"2024-03-04T09:30:22.208741+0000","last_undegraded":"2024-03-04T09:30:22.208741+0000","last_fullsized":"2024-03-04T09:30:22.208741+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:40:11.984649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.d","version":"93'1","reported_seq":15,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.602718+0000","last_change":"2024-03-04T09:30:20.201461+0000","last_active":"2024-03-04T09:30:22.602718+0000","last_peered":"2024-03-04T09:30:22.602718+0000","last_clean":"2024-03-04T09:30:22.602718+0000","last_became_active":"2024-03-04T09:30:20.201181+0000","last_became_peered":"2024-03-04T09:30:20.201181+0000","last_unstale":"2024-03-04T09:30:22.602718+0000","last_undegraded":"2024-03-04T09:30:22.602718+0000","last_fullsized":"2024-03-04T09:30:22.602718+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_clean_scrub_stamp":"2024-03-04T09:30:19.137440+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:40:10.967990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.602630+0000","last_change":"2024-03-04T09:30:21.215009+0000","last_active":"2024-03-04T09:30:22.602630+0000","last_peered":"2024-03-04T09:30:22.602630+0000","last_clean":"2024-03-04T09:30:22.602630+0000","last_became_active":"2024-03-04T09:30:21.214716+0000","last_became_peered":"2024-03-04T09:30:21.214716+0000","last_unstale":"2024-03-04T09:30:22.602630+0000","last_undegraded":"2024-03-04T09:30:22.602630+0000","last_fullsized":"2024-03-04T09:30:22.602630+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:30:09.483436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":14,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.207817+0000","last_change":"2024-03-04T09:30:20.202007+0000","last_active":"2024-03-04T09:30:22.207817+0000","last_peered":"2024-03-04T09:30:22.207817+0000","last_clean":"2024-03-04T09:30:22.207817+0000","last_became_active":"2024-03-04T09:30:20.201633+0000","last_became_peered":"2024-03-04T09:30:20.201633+0000","last_unstale":"2024-03-04T09:30:22.207817+0000","last_undegraded":"2024-03-04T09:30:22.207817+0000","last_fullsized":"2024-03-04T09:30:22.207817+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_clean_scrub_stamp":"2024-03-04T09:30:19.137440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:56:56.967774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.211265+0000","last_change":"2024-03-04T09:30:21.220213+0000","last_active":"2024-03-04T09:30:22.211265+0000","last_peered":"2024-03-04T09:30:22.211265+0000","last_clean":"2024-03-04T09:30:22.211265+0000","last_became_active":"2024-03-04T09:30:21.217126+0000","last_became_peered":"2024-03-04T09:30:21.217126+0000","last_unstale":"2024-03-04T09:30:22.211265+0000","last_undegraded":"2024-03-04T09:30:22.211265+0000","last_fullsized":"2024-03-04T09:30:22.211265+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:44:15.690906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.b","version":"93'2","reported_seq":16,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.207756+0000","last_change":"2024-03-04T09:30:20.203741+0000","last_active":"2024-03-04T09:30:22.207756+0000","last_peered":"2024-03-04T09:30:22.207756+0000","last_clean":"2024-03-04T09:30:22.207756+0000","last_became_active":"2024-03-04T09:30:20.203455+0000","last_became_peered":"2024-03-04T09:30:20.203455+0000","last_unstale":"2024-03-04T09:30:22.207756+0000","last_undegraded":"2024-03-04T09:30:22.207756+0000","last_fullsized":"2024-03-04T09:30:22.207756+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_clean_scrub_stamp":"2024-03-04T09:30:19.137440+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:22:43.529585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.211333+0000","last_change":"2024-03-04T09:30:21.208688+0000","last_active":"2024-03-04T09:30:22.211333+0000","last_peered":"2024-03-04T09:30:22.211333+0000","last_clean":"2024-03-04T09:30:22.211333+0000","last_became_active":"2024-03-04T09:30:21.208387+0000","last_became_peered":"2024-03-04T09:30:21.208387+0000","last_unstale":"2024-03-04T09:30:22.211333+0000","last_undegraded":"2024-03-04T09:30:22.211333+0000","last_fullsized":"2024-03-04T09:30:22.211333+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:33:42.793868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":14,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.208765+0000","last_change":"2024-03-04T09:30:20.201935+0000","last_active":"2024-03-04T09:30:22.208765+0000","last_peered":"2024-03-04T09:30:22.208765+0000","last_clean":"2024-03-04T09:30:22.208765+0000","last_became_active":"2024-03-04T09:30:20.201641+0000","last_became_peered":"2024-03-04T09:30:20.201641+0000","last_unstale":"2024-03-04T09:30:22.208765+0000","last_undegraded":"2024-03-04T09:30:22.208765+0000","last_fullsized":"2024-03-04T09:30:22.208765+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_clean_scrub_stamp":"2024-03-04T09:30:19.137440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:39:06.431069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.210904+0000","last_change":"2024-03-04T09:30:21.216656+0000","last_active":"2024-03-04T09:30:22.210904+0000","last_peered":"2024-03-04T09:30:22.210904+0000","last_clean":"2024-03-04T09:30:22.210904+0000","last_became_active":"2024-03-04T09:30:21.216112+0000","last_became_peered":"2024-03-04T09:30:21.216112+0000","last_unstale":"2024-03-04T09:30:22.210904+0000","last_undegraded":"2024-03-04T09:30:22.210904+0000","last_fullsized":"2024-03-04T09:30:22.210904+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:50:33.851581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":14,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.208621+0000","last_change":"2024-03-04T09:30:20.200225+0000","last_active":"2024-03-04T09:30:22.208621+0000","last_peered":"2024-03-04T09:30:22.208621+0000","last_clean":"2024-03-04T09:30:22.208621+0000","last_became_active":"2024-03-04T09:30:20.199914+0000","last_became_peered":"2024-03-04T09:30:20.199914+0000","last_unstale":"2024-03-04T09:30:22.208621+0000","last_undegraded":"2024-03-04T09:30:22.208621+0000","last_fullsized":"2024-03-04T09:30:22.208621+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_clean_scrub_stamp":"2024-03-04T09:30:19.137440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:52:31.073581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.601373+0000","last_change":"2024-03-04T09:30:21.210036+0000","last_active":"2024-03-04T09:30:22.601373+0000","last_peered":"2024-03-04T09:30:22.601373+0000","last_clean":"2024-03-04T09:30:22.601373+0000","last_became_active":"2024-03-04T09:30:21.209732+0000","last_became_peered":"2024-03-04T09:30:21.209732+0000","last_unstale":"2024-03-04T09:30:22.601373+0000","last_undegraded":"2024-03-04T09:30:22.601373+0000","last_fullsized":"2024-03-04T09:30:22.601373+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:26:12.396587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":14,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.207572+0000","last_change":"2024-03-04T09:30:20.204803+0000","last_active":"2024-03-04T09:30:22.207572+0000","last_peered":"2024-03-04T09:30:22.207572+0000","last_clean":"2024-03-04T09:30:22.207572+0000","last_became_active":"2024-03-04T09:30:20.203153+0000","last_became_peered":"2024-03-04T09:30:20.203153+0000","last_unstale":"2024-03-04T09:30:22.207572+0000","last_undegraded":"2024-03-04T09:30:22.207572+0000","last_fullsized":"2024-03-04T09:30:22.207572+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_clean_scrub_stamp":"2024-03-04T09:30:19.137440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:36:55.596046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.208303+0000","last_change":"2024-03-04T09:30:21.201417+0000","last_active":"2024-03-04T09:30:22.208303+0000","last_peered":"2024-03-04T09:30:22.208303+0000","last_clean":"2024-03-04T09:30:22.208303+0000","last_became_active":"2024-03-04T09:30:21.201134+0000","last_became_peered":"2024-03-04T09:30:21.201134+0000","last_unstale":"2024-03-04T09:30:22.208303+0000","last_undegraded":"2024-03-04T09:30:22.208303+0000","last_fullsized":"2024-03-04T09:30:22.208303+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:03:49.406310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.7","version":"93'2","reported_seq":16,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.208564+0000","last_change":"2024-03-04T09:30:20.204428+0000","last_active":"2024-03-04T09:30:22.208564+0000","last_peered":"2024-03-04T09:30:22.208564+0000","last_clean":"2024-03-04T09:30:22.208564+0000","last_became_active":"2024-03-04T09:30:20.204109+0000","last_became_peered":"2024-03-04T09:30:20.204109+0000","last_unstale":"2024-03-04T09:30:22.208564+0000","last_undegraded":"2024-03-04T09:30:22.208564+0000","last_fullsized":"2024-03-04T09:30:22.208564+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_clean_scrub_stamp":"2024-03-04T09:30:19.137440+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:45:27.624178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.203731+0000","last_change":"2024-03-04T09:30:21.212032+0000","last_active":"2024-03-04T09:30:22.203731+0000","last_peered":"2024-03-04T09:30:22.203731+0000","last_clean":"2024-03-04T09:30:22.203731+0000","last_became_active":"2024-03-04T09:30:21.211735+0000","last_became_peered":"2024-03-04T09:30:21.211735+0000","last_unstale":"2024-03-04T09:30:22.203731+0000","last_undegraded":"2024-03-04T09:30:22.203731+0000","last_fullsized":"2024-03-04T09:30:22.203731+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:03:48.340632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.6","version":"94'1","reported_seq":15,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.210061+0000","last_change":"2024-03-04T09:30:20.202239+0000","last_active":"2024-03-04T09:30:22.210061+0000","last_peered":"2024-03-04T09:30:22.210061+0000","last_clean":"2024-03-04T09:30:22.210061+0000","last_became_active":"2024-03-04T09:30:20.201889+0000","last_became_peered":"2024-03-04T09:30:20.201889+0000","last_unstale":"2024-03-04T09:30:22.210061+0000","last_undegraded":"2024-03-04T09:30:22.210061+0000","last_fullsized":"2024-03-04T09:30:22.210061+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_clean_scrub_stamp":"2024-03-04T09:30:19.137440+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:47:41.230184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.203451+0000","last_change":"2024-03-04T09:30:21.212635+0000","last_active":"2024-03-04T09:30:22.203451+0000","last_peered":"2024-03-04T09:30:22.203451+0000","last_clean":"2024-03-04T09:30:22.203451+0000","last_became_active":"2024-03-04T09:30:21.212346+0000","last_became_peered":"2024-03-04T09:30:21.212346+0000","last_unstale":"2024-03-04T09:30:22.203451+0000","last_undegraded":"2024-03-04T09:30:22.203451+0000","last_fullsized":"2024-03-04T09:30:22.203451+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:52:32.090241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":14,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.207918+0000","last_change":"2024-03-04T09:30:20.206409+0000","last_active":"2024-03-04T09:30:22.207918+0000","last_peered":"2024-03-04T09:30:22.207918+0000","last_clean":"2024-03-04T09:30:22.207918+0000","last_became_active":"2024-03-04T09:30:20.204756+0000","last_became_peered":"2024-03-04T09:30:20.204756+0000","last_unstale":"2024-03-04T09:30:22.207918+0000","last_undegraded":"2024-03-04T09:30:22.207918+0000","last_fullsized":"2024-03-04T09:30:22.207918+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_clean_scrub_stamp":"2024-03-04T09:30:19.137440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:51:17.040650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.602450+0000","last_change":"2024-03-04T09:30:21.216899+0000","last_active":"2024-03-04T09:30:22.602450+0000","last_peered":"2024-03-04T09:30:22.602450+0000","last_clean":"2024-03-04T09:30:22.602450+0000","last_became_active":"2024-03-04T09:30:21.215437+0000","last_became_peered":"2024-03-04T09:30:21.215437+0000","last_unstale":"2024-03-04T09:30:22.602450+0000","last_undegraded":"2024-03-04T09:30:22.602450+0000","last_fullsized":"2024-03-04T09:30:22.602450+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:56:57.984433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":14,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.604548+0000","last_change":"2024-03-04T09:30:20.200047+0000","last_active":"2024-03-04T09:30:22.604548+0000","last_peered":"2024-03-04T09:30:22.604548+0000","last_clean":"2024-03-04T09:30:22.604548+0000","last_became_active":"2024-03-04T09:30:20.199490+0000","last_became_peered":"2024-03-04T09:30:20.199490+0000","last_unstale":"2024-03-04T09:30:22.604548+0000","last_undegraded":"2024-03-04T09:30:22.604548+0000","last_fullsized":"2024-03-04T09:30:22.604548+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_clean_scrub_stamp":"2024-03-04T09:30:19.137440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:43:31.237510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.211418+0000","last_change":"2024-03-04T09:30:21.221508+0000","last_active":"2024-03-04T09:30:22.211418+0000","last_peered":"2024-03-04T09:30:22.211418+0000","last_clean":"2024-03-04T09:30:22.211418+0000","last_became_active":"2024-03-04T09:30:21.220985+0000","last_became_peered":"2024-03-04T09:30:21.220985+0000","last_unstale":"2024-03-04T09:30:22.211418+0000","last_undegraded":"2024-03-04T09:30:22.211418+0000","last_fullsized":"2024-03-04T09:30:22.211418+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:44:41.598724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":14,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.207858+0000","last_change":"2024-03-04T09:30:20.202203+0000","last_active":"2024-03-04T09:30:22.207858+0000","last_peered":"2024-03-04T09:30:22.207858+0000","last_clean":"2024-03-04T09:30:22.207858+0000","last_became_active":"2024-03-04T09:30:20.201554+0000","last_became_peered":"2024-03-04T09:30:20.201554+0000","last_unstale":"2024-03-04T09:30:22.207858+0000","last_undegraded":"2024-03-04T09:30:22.207858+0000","last_fullsized":"2024-03-04T09:30:22.207858+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_clean_scrub_stamp":"2024-03-04T09:30:19.137440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:04:50.243684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.206501+0000","last_change":"2024-03-04T09:30:21.208592+0000","last_active":"2024-03-04T09:30:22.206501+0000","last_peered":"2024-03-04T09:30:22.206501+0000","last_clean":"2024-03-04T09:30:22.206501+0000","last_became_active":"2024-03-04T09:30:21.208299+0000","last_became_peered":"2024-03-04T09:30:21.208299+0000","last_unstale":"2024-03-04T09:30:22.206501+0000","last_undegraded":"2024-03-04T09:30:22.206501+0000","last_fullsized":"2024-03-04T09:30:22.206501+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:56:57.984433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":14,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.602052+0000","last_change":"2024-03-04T09:30:20.193110+0000","last_active":"2024-03-04T09:30:22.602052+0000","last_peered":"2024-03-04T09:30:22.602052+0000","last_clean":"2024-03-04T09:30:22.602052+0000","last_became_active":"2024-03-04T09:30:20.192504+0000","last_became_peered":"2024-03-04T09:30:20.192504+0000","last_unstale":"2024-03-04T09:30:22.602052+0000","last_undegraded":"2024-03-04T09:30:22.602052+0000","last_fullsized":"2024-03-04T09:30:22.602052+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_clean_scrub_stamp":"2024-03-04T09:30:19.137440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:40:35.740641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.211210+0000","last_change":"2024-03-04T09:30:21.216422+0000","last_active":"2024-03-04T09:30:22.211210+0000","last_peered":"2024-03-04T09:30:22.211210+0000","last_clean":"2024-03-04T09:30:22.211210+0000","last_became_active":"2024-03-04T09:30:21.215611+0000","last_became_peered":"2024-03-04T09:30:21.215611+0000","last_unstale":"2024-03-04T09:30:22.211210+0000","last_undegraded":"2024-03-04T09:30:22.211210+0000","last_fullsized":"2024-03-04T09:30:22.211210+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:21:52.125031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":14,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.600183+0000","last_change":"2024-03-04T09:30:20.189556+0000","last_active":"2024-03-04T09:30:22.600183+0000","last_peered":"2024-03-04T09:30:22.600183+0000","last_clean":"2024-03-04T09:30:22.600183+0000","last_became_active":"2024-03-04T09:30:20.188384+0000","last_became_peered":"2024-03-04T09:30:20.188384+0000","last_unstale":"2024-03-04T09:30:22.600183+0000","last_undegraded":"2024-03-04T09:30:22.600183+0000","last_fullsized":"2024-03-04T09:30:22.600183+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_clean_scrub_stamp":"2024-03-04T09:30:19.137440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:36:55.596046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.601407+0000","last_change":"2024-03-04T09:30:21.213566+0000","last_active":"2024-03-04T09:30:22.601407+0000","last_peered":"2024-03-04T09:30:22.601407+0000","last_clean":"2024-03-04T09:30:22.601407+0000","last_became_active":"2024-03-04T09:30:21.213248+0000","last_became_peered":"2024-03-04T09:30:21.213248+0000","last_unstale":"2024-03-04T09:30:22.601407+0000","last_undegraded":"2024-03-04T09:30:22.601407+0000","last_fullsized":"2024-03-04T09:30:22.601407+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:51:18.057309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":14,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.211017+0000","last_change":"2024-03-04T09:30:20.196985+0000","last_active":"2024-03-04T09:30:22.211017+0000","last_peered":"2024-03-04T09:30:22.211017+0000","last_clean":"2024-03-04T09:30:22.211017+0000","last_became_active":"2024-03-04T09:30:20.195708+0000","last_became_peered":"2024-03-04T09:30:20.195708+0000","last_unstale":"2024-03-04T09:30:22.211017+0000","last_undegraded":"2024-03-04T09:30:22.211017+0000","last_fullsized":"2024-03-04T09:30:22.211017+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_clean_scrub_stamp":"2024-03-04T09:30:19.137440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:30:23.060431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.211718+0000","last_change":"2024-03-04T09:30:21.218217+0000","last_active":"2024-03-04T09:30:22.211718+0000","last_peered":"2024-03-04T09:30:22.211718+0000","last_clean":"2024-03-04T09:30:22.211718+0000","last_became_active":"2024-03-04T09:30:21.217067+0000","last_became_peered":"2024-03-04T09:30:21.217067+0000","last_unstale":"2024-03-04T09:30:22.211718+0000","last_undegraded":"2024-03-04T09:30:22.211718+0000","last_fullsized":"2024-03-04T09:30:22.211718+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:14:56.870440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.f","version":"93'1","reported_seq":15,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.601642+0000","last_change":"2024-03-04T09:30:20.204692+0000","last_active":"2024-03-04T09:30:22.601642+0000","last_peered":"2024-03-04T09:30:22.601642+0000","last_clean":"2024-03-04T09:30:22.601642+0000","last_became_active":"2024-03-04T09:30:20.204363+0000","last_became_peered":"2024-03-04T09:30:20.204363+0000","last_unstale":"2024-03-04T09:30:22.601642+0000","last_undegraded":"2024-03-04T09:30:22.601642+0000","last_fullsized":"2024-03-04T09:30:22.601642+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_clean_scrub_stamp":"2024-03-04T09:30:19.137440+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:42:56.761983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":310,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.601531+0000","last_change":"2024-03-04T09:25:09.316159+0000","last_active":"2024-03-04T09:30:22.601531+0000","last_peered":"2024-03-04T09:30:22.601531+0000","last_clean":"2024-03-04T09:30:22.601531+0000","last_became_active":"2024-03-04T09:25:09.313767+0000","last_became_peered":"2024-03-04T09:25:09.313767+0000","last_unstale":"2024-03-04T09:30:22.601531+0000","last_undegraded":"2024-03-04T09:30:22.601531+0000","last_fullsized":"2024-03-04T09:30:22.601531+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:08.110953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:08.110953+0000","last_clean_scrub_stamp":"2024-03-04T09:25:08.110953+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:34:21.745390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":15,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574914,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32908,"kb_used_data":1724,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338932,"statfs":{"total":96636764160,"available":96603066368,"internally_reserved":0,"allocated":1765376,"data_stored":1088420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574914,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32324,"kb_used_data":1144,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339516,"statfs":{"total":96636764160,"available":96603664384,"internally_reserved":0,"allocated":1171456,"data_stored":498592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574916,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32324,"kb_used_data":1140,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339516,"statfs":{"total":96636764160,"available":96603664384,"internally_reserved":0,"allocated":1167360,"data_stored":498010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574913,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32344,"kb_used_data":1156,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339496,"statfs":{"total":96636764160,"available":96603643904,"internally_reserved":0,"allocated":1183744,"data_stored":499676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574915,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32320,"kb_used_data":1140,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339520,"statfs":{"total":96636764160,"available":96603668480,"internally_reserved":0,"allocated":1167360,"data_stored":493274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574914,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32308,"kb_used_data":1128,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339532,"statfs":{"total":96636764160,"available":96603680768,"internally_reserved":0,"allocated":1155072,"data_stored":491674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574915,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32320,"kb_used_data":1136,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339520,"statfs":{"total":96636764160,"available":96603668480,"internally_reserved":0,"allocated":1163264,"data_stored":497940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574914,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32908,"kb_used_data":1720,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338932,"statfs":{"total":96636764160,"available":96603066368,"internally_reserved":0,"allocated":1761280,"data_stored":1088890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-04T09:30:25.691 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T09:30:25.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.948+0000 7fa1a7200640 1 Processor -- start 2024-03-04T09:30:25.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.949+0000 7fa1a7200640 1 -- start start 2024-03-04T09:30:25.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.950+0000 7fa1a7200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa1a015e070 0x7fa1a00d2960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:25.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.950+0000 7fa1a7200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fa1a00cf410 0x7fa1a00cf870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:25.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.950+0000 7fa1a7200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1a00d18c0 con 0x7fa1a00d05c0 2024-03-04T09:30:25.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.950+0000 7fa1a7200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1a00d1ba0 con 0x7fa1a00cf410 2024-03-04T09:30:25.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.950+0000 7fa1a7200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1a00d1e80 con 0x7fa1a015e070 2024-03-04T09:30:25.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.951+0000 7fa1a5e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa1a015e070 0x7fa1a00d2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:25.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.951+0000 7fa1a5400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fa1a00cf410 0x7fa1a00cf870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:25.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.951+0000 7fa1a5e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa1a015e070 0x7fa1a00d2960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.6:42132/0 (socket says 172.21.15.6:42132) 2024-03-04T09:30:25.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.951+0000 7fa1a5e00640 1 -- 172.21.15.6:0/1814388153 learned_addr learned my addr 172.21.15.6:0/1814388153 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:30:25.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.952+0000 7fa1a5400640 1 -- 172.21.15.6:0/1814388153 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa1a015e070 msgr2=0x7fa1a00d2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:25.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.952+0000 7fa1a5400640 1 --2- 172.21.15.6:0/1814388153 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa1a015e070 0x7fa1a00d2960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:25.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.952+0000 7fa1a5400640 1 -- 172.21.15.6:0/1814388153 >> v1:172.21.15.146:6789/0 conn(0x7fa1a00d05c0 legacy=0x7fa1a00b3630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:25.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.952+0000 7fa1a5400640 1 -- 172.21.15.6:0/1814388153 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1a00d2160 con 0x7fa1a00cf410 2024-03-04T09:30:25.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.952+0000 7fa1a5e00640 1 --2- 172.21.15.6:0/1814388153 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa1a015e070 0x7fa1a00d2960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T09:30:25.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.953+0000 7fa1a5400640 1 --2- 172.21.15.6:0/1814388153 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa1a00cf410 0x7fa1a00cf870 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fa1900097a0 tx=0x7fa19002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4f7a88042d06b547 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:25.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.954+0000 7fa1a4a00640 1 -- 172.21.15.6:0/1814388153 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa190035020 con 0x7fa1a00cf410 2024-03-04T09:30:25.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.954+0000 7fa1a4a00640 1 -- 172.21.15.6:0/1814388153 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa190031830 con 0x7fa1a00cf410 2024-03-04T09:30:25.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.954+0000 7fa1a4a00640 1 -- 172.21.15.6:0/1814388153 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa190031c00 con 0x7fa1a00cf410 2024-03-04T09:30:25.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.955+0000 7fa1a7200640 1 -- 172.21.15.6:0/1814388153 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa1a00cf410 msgr2=0x7fa1a00cf870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:25.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.955+0000 7fa1a7200640 1 --2- 172.21.15.6:0/1814388153 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa1a00cf410 0x7fa1a00cf870 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fa1900097a0 tx=0x7fa19002b8d0 comp rx=0 tx=0).stop 2024-03-04T09:30:25.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.955+0000 7fa1a7200640 1 -- 172.21.15.6:0/1814388153 shutdown_connections 2024-03-04T09:30:25.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.955+0000 7fa1a7200640 1 --2- 172.21.15.6:0/1814388153 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa1a00cf410 0x7fa1a00cf870 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:25.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.955+0000 7fa1a7200640 1 --2- 172.21.15.6:0/1814388153 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa1a015e070 0x7fa1a00d2960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:25.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.955+0000 7fa1a7200640 1 -- 172.21.15.6:0/1814388153 >> 172.21.15.6:0/1814388153 conn(0x7fa1a008d2a0 msgr2=0x7fa1a00d3cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:25.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.955+0000 7fa1a7200640 1 -- 172.21.15.6:0/1814388153 shutdown_connections 2024-03-04T09:30:25.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.955+0000 7fa1a7200640 1 -- 172.21.15.6:0/1814388153 wait complete. 2024-03-04T09:30:25.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.957+0000 7fa1a7200640 1 Processor -- start 2024-03-04T09:30:25.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.957+0000 7fa1a7200640 1 -- start start 2024-03-04T09:30:25.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.958+0000 7fa1a7200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa1a00cf410 0x7fa1a014e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:25.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.958+0000 7fa1a7200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa1a00d05c0 0x7fa1a00ccd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:25.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.958+0000 7fa1a7200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa1a015e070 0x7fa1a00cd280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:25.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.958+0000 7fa1a7200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1a00d4730 con 0x7fa1a00d05c0 2024-03-04T09:30:25.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.958+0000 7fa1a7200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1a00d45b0 con 0x7fa1a00cf410 2024-03-04T09:30:25.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.958+0000 7fa1a7200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1a00d0a50 con 0x7fa1a015e070 2024-03-04T09:30:25.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.958+0000 7fa1a5400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa1a00d05c0 0x7fa1a00ccd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:25.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.958+0000 7fa1a5400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa1a00d05c0 0x7fa1a00ccd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38426/0 (socket says 172.21.15.6:38426) 2024-03-04T09:30:25.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.958+0000 7fa1a5400640 1 -- 172.21.15.6:0/812528402 learned_addr learned my addr 172.21.15.6:0/812528402 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:30:25.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.959+0000 7fa1a5e00640 1 --2- 172.21.15.6:0/812528402 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa1a00cf410 0x7fa1a014e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:25.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.959+0000 7fa1a6800640 1 --2- 172.21.15.6:0/812528402 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa1a015e070 0x7fa1a00cd280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:25.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.959+0000 7fa1a5400640 1 -- 172.21.15.6:0/812528402 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa1a015e070 msgr2=0x7fa1a00cd280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:25.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.959+0000 7fa1a5400640 1 --2- 172.21.15.6:0/812528402 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa1a015e070 0x7fa1a00cd280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:25.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.959+0000 7fa1a5400640 1 -- 172.21.15.6:0/812528402 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa1a00cf410 msgr2=0x7fa1a014e2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:25.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.959+0000 7fa1a5400640 1 --2- 172.21.15.6:0/812528402 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa1a00cf410 0x7fa1a014e2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:25.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.959+0000 7fa1a5400640 1 -- 172.21.15.6:0/812528402 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1a015edf0 con 0x7fa1a00d05c0 2024-03-04T09:30:25.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.959+0000 7fa1a5e00640 1 --2- 172.21.15.6:0/812528402 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa1a00cf410 0x7fa1a014e2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T09:30:25.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.959+0000 7fa1a6800640 1 --2- 172.21.15.6:0/812528402 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa1a015e070 0x7fa1a00cd280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:30:25.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.959+0000 7fa1a5400640 1 --2- 172.21.15.6:0/812528402 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa1a00d05c0 0x7fa1a00ccd40 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7fa19002bee0 tx=0x7fa1900046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:25.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.960+0000 7fa19a000640 1 -- 172.21.15.6:0/812528402 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa190035040 con 0x7fa1a00d05c0 2024-03-04T09:30:25.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.960+0000 7fa19a000640 1 -- 172.21.15.6:0/812528402 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa19003b920 con 0x7fa1a00d05c0 2024-03-04T09:30:25.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.960+0000 7fa19a000640 1 -- 172.21.15.6:0/812528402 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa19003bd10 con 0x7fa1a00d05c0 2024-03-04T09:30:25.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.960+0000 7fa1a7200640 1 -- 172.21.15.6:0/812528402 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1a00cd8e0 con 0x7fa1a00d05c0 2024-03-04T09:30:25.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.960+0000 7fa1a7200640 1 -- 172.21.15.6:0/812528402 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1a00cdea0 con 0x7fa1a00d05c0 2024-03-04T09:30:25.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.961+0000 7fa1a7200640 1 -- 172.21.15.6:0/812528402 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1a004f330 con 0x7fa1a00d05c0 2024-03-04T09:30:25.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.962+0000 7fa19a000640 1 -- 172.21.15.6:0/812528402 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa190038020 con 0x7fa1a00d05c0 2024-03-04T09:30:25.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.963+0000 7fa19a000640 1 --2- 172.21.15.6:0/812528402 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa18406b4d0 0x7fa18406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:25.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.963+0000 7fa19a000640 1 -- 172.21.15.6:0/812528402 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa1900ac390 con 0x7fa1a00d05c0 2024-03-04T09:30:25.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.963+0000 7fa1a5e00640 1 --2- 172.21.15.6:0/812528402 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa18406b4d0 0x7fa18406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:25.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.965+0000 7fa1a5e00640 1 --2- 172.21.15.6:0/812528402 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa18406b4d0 0x7fa18406d990 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fa194009b80 tx=0x7fa194008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:25.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:25.967+0000 7fa19a000640 1 -- 172.21.15.6:0/812528402 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa190044380 con 0x7fa1a00d05c0 2024-03-04T09:30:26.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:26.167+0000 7fa1a7200640 1 -- 172.21.15.6:0/812528402 --> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa1a00c75c0 con 0x7fa18406b4d0 2024-03-04T09:30:26.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:26.185+0000 7fa19a000640 1 -- 172.21.15.6:0/812528402 <== mgr.4113 v2:172.21.15.146:6832/1788613334 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252084 (secure 0 0 0) 0x7fa1a00c75c0 con 0x7fa18406b4d0 2024-03-04T09:30:26.185 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:30:26.187 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-03-04T09:30:26.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:26.191+0000 7fa1a7200640 1 -- 172.21.15.6:0/812528402 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa18406b4d0 msgr2=0x7fa18406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:26.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:26.191+0000 7fa1a7200640 1 --2- 172.21.15.6:0/812528402 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa18406b4d0 0x7fa18406d990 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fa194009b80 tx=0x7fa194008040 comp rx=0 tx=0).stop 2024-03-04T09:30:26.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:26.191+0000 7fa1a7200640 1 -- 172.21.15.6:0/812528402 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa1a00d05c0 msgr2=0x7fa1a00ccd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:26.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:26.191+0000 7fa1a7200640 1 --2- 172.21.15.6:0/812528402 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa1a00d05c0 0x7fa1a00ccd40 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7fa19002bee0 tx=0x7fa1900046c0 comp rx=0 tx=0).stop 2024-03-04T09:30:26.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:26.191+0000 7fa1a7200640 1 -- 172.21.15.6:0/812528402 shutdown_connections 2024-03-04T09:30:26.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:26.191+0000 7fa1a7200640 1 --2- 172.21.15.6:0/812528402 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa18406b4d0 0x7fa18406d990 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:26.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:26.191+0000 7fa1a7200640 1 --2- 172.21.15.6:0/812528402 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa1a015e070 0x7fa1a00cd280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:26.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:26.191+0000 7fa1a7200640 1 --2- 172.21.15.6:0/812528402 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa1a00d05c0 0x7fa1a00ccd40 unknown :-1 s=CLOSED pgs=1085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:26.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:26.191+0000 7fa1a7200640 1 --2- 172.21.15.6:0/812528402 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa1a00cf410 0x7fa1a014e2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:26.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:26.191+0000 7fa1a7200640 1 -- 172.21.15.6:0/812528402 >> 172.21.15.6:0/812528402 conn(0x7fa1a008d2a0 msgr2=0x7fa1a00ce8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:26.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:26.192+0000 7fa1a7200640 1 -- 172.21.15.6:0/812528402 shutdown_connections 2024-03-04T09:30:26.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:26.192+0000 7fa1a7200640 1 -- 172.21.15.6:0/812528402 wait complete. 2024-03-04T09:30:26.207 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":246,"stamp":"2024-03-04T09:30:25.969479+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":136,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":259756,"kb_used_data":10288,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754714964,"statfs":{"total":773094113280,"available":772828123136,"internally_reserved":0,"allocated":10534912,"data_stored":5156476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002150"},"pg_stats":[{"pgid":"15.3f","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.203689+0000","last_change":"2024-03-04T09:30:21.188523+0000","last_active":"2024-03-04T09:30:22.203689+0000","last_peered":"2024-03-04T09:30:22.203689+0000","last_clean":"2024-03-04T09:30:22.203689+0000","last_became_active":"2024-03-04T09:30:21.188002+0000","last_became_peered":"2024-03-04T09:30:21.188002+0000","last_unstale":"2024-03-04T09:30:22.203689+0000","last_undegraded":"2024-03-04T09:30:22.203689+0000","last_fullsized":"2024-03-04T09:30:22.203689+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:39:07.447728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3e","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.207793+0000","last_change":"2024-03-04T09:30:21.204386+0000","last_active":"2024-03-04T09:30:22.207793+0000","last_peered":"2024-03-04T09:30:22.207793+0000","last_clean":"2024-03-04T09:30:22.207793+0000","last_became_active":"2024-03-04T09:30:21.202933+0000","last_became_peered":"2024-03-04T09:30:21.202933+0000","last_unstale":"2024-03-04T09:30:22.207793+0000","last_undegraded":"2024-03-04T09:30:22.207793+0000","last_fullsized":"2024-03-04T09:30:22.207793+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:17:11.508691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3d","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.203617+0000","last_change":"2024-03-04T09:30:21.195261+0000","last_active":"2024-03-04T09:30:22.203617+0000","last_peered":"2024-03-04T09:30:22.203617+0000","last_clean":"2024-03-04T09:30:22.203617+0000","last_became_active":"2024-03-04T09:30:21.194932+0000","last_became_peered":"2024-03-04T09:30:21.194932+0000","last_unstale":"2024-03-04T09:30:22.203617+0000","last_undegraded":"2024-03-04T09:30:22.203617+0000","last_fullsized":"2024-03-04T09:30:22.203617+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:45:28.640838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3c","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.600344+0000","last_change":"2024-03-04T09:30:21.218498+0000","last_active":"2024-03-04T09:30:22.600344+0000","last_peered":"2024-03-04T09:30:22.600344+0000","last_clean":"2024-03-04T09:30:22.600344+0000","last_became_active":"2024-03-04T09:30:21.217784+0000","last_became_peered":"2024-03-04T09:30:21.217784+0000","last_unstale":"2024-03-04T09:30:22.600344+0000","last_undegraded":"2024-03-04T09:30:22.600344+0000","last_fullsized":"2024-03-04T09:30:22.600344+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:58:07.611882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.3b","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.603491+0000","last_change":"2024-03-04T09:30:21.218477+0000","last_active":"2024-03-04T09:30:22.603491+0000","last_peered":"2024-03-04T09:30:22.603491+0000","last_clean":"2024-03-04T09:30:22.603491+0000","last_became_active":"2024-03-04T09:30:21.218149+0000","last_became_peered":"2024-03-04T09:30:21.218149+0000","last_unstale":"2024-03-04T09:30:22.603491+0000","last_undegraded":"2024-03-04T09:30:22.603491+0000","last_fullsized":"2024-03-04T09:30:22.603491+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:22:44.546244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.3a","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.602235+0000","last_change":"2024-03-04T09:30:21.213698+0000","last_active":"2024-03-04T09:30:22.602235+0000","last_peered":"2024-03-04T09:30:22.602235+0000","last_clean":"2024-03-04T09:30:22.602235+0000","last_became_active":"2024-03-04T09:30:21.213386+0000","last_became_peered":"2024-03-04T09:30:21.213386+0000","last_unstale":"2024-03-04T09:30:22.602235+0000","last_undegraded":"2024-03-04T09:30:22.602235+0000","last_fullsized":"2024-03-04T09:30:22.602235+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:38:54.890507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.39","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.601971+0000","last_change":"2024-03-04T09:30:21.212163+0000","last_active":"2024-03-04T09:30:22.601971+0000","last_peered":"2024-03-04T09:30:22.601971+0000","last_clean":"2024-03-04T09:30:22.601971+0000","last_became_active":"2024-03-04T09:30:21.211896+0000","last_became_peered":"2024-03-04T09:30:21.211896+0000","last_unstale":"2024-03-04T09:30:22.601971+0000","last_undegraded":"2024-03-04T09:30:22.601971+0000","last_fullsized":"2024-03-04T09:30:22.601971+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:47:42.246844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.38","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.207989+0000","last_change":"2024-03-04T09:30:21.202283+0000","last_active":"2024-03-04T09:30:22.207989+0000","last_peered":"2024-03-04T09:30:22.207989+0000","last_clean":"2024-03-04T09:30:22.207989+0000","last_became_active":"2024-03-04T09:30:21.201683+0000","last_became_peered":"2024-03-04T09:30:21.201683+0000","last_unstale":"2024-03-04T09:30:22.207989+0000","last_undegraded":"2024-03-04T09:30:22.207989+0000","last_fullsized":"2024-03-04T09:30:22.207989+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:02:26.808507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.37","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.207193+0000","last_change":"2024-03-04T09:30:21.207087+0000","last_active":"2024-03-04T09:30:22.207193+0000","last_peered":"2024-03-04T09:30:22.207193+0000","last_clean":"2024-03-04T09:30:22.207193+0000","last_became_active":"2024-03-04T09:30:21.205513+0000","last_became_peered":"2024-03-04T09:30:21.205513+0000","last_unstale":"2024-03-04T09:30:22.207193+0000","last_undegraded":"2024-03-04T09:30:22.207193+0000","last_fullsized":"2024-03-04T09:30:22.207193+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:22:44.546244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.36","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.603148+0000","last_change":"2024-03-04T09:30:21.210554+0000","last_active":"2024-03-04T09:30:22.603148+0000","last_peered":"2024-03-04T09:30:22.603148+0000","last_clean":"2024-03-04T09:30:22.603148+0000","last_became_active":"2024-03-04T09:30:21.210151+0000","last_became_peered":"2024-03-04T09:30:21.210151+0000","last_unstale":"2024-03-04T09:30:22.603148+0000","last_undegraded":"2024-03-04T09:30:22.603148+0000","last_fullsized":"2024-03-04T09:30:22.603148+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:35:54.773629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.35","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.601765+0000","last_change":"2024-03-04T09:30:21.211218+0000","last_active":"2024-03-04T09:30:22.601765+0000","last_peered":"2024-03-04T09:30:22.601765+0000","last_clean":"2024-03-04T09:30:22.601765+0000","last_became_active":"2024-03-04T09:30:21.207302+0000","last_became_peered":"2024-03-04T09:30:21.207302+0000","last_unstale":"2024-03-04T09:30:22.601765+0000","last_undegraded":"2024-03-04T09:30:22.601765+0000","last_fullsized":"2024-03-04T09:30:22.601765+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:22:44.546244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.34","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.207209+0000","last_change":"2024-03-04T09:30:21.211419+0000","last_active":"2024-03-04T09:30:22.207209+0000","last_peered":"2024-03-04T09:30:22.207209+0000","last_clean":"2024-03-04T09:30:22.207209+0000","last_became_active":"2024-03-04T09:30:21.211060+0000","last_became_peered":"2024-03-04T09:30:21.211060+0000","last_unstale":"2024-03-04T09:30:22.207209+0000","last_undegraded":"2024-03-04T09:30:22.207209+0000","last_fullsized":"2024-03-04T09:30:22.207209+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:47:42.246844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.33","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.599833+0000","last_change":"2024-03-04T09:30:21.207596+0000","last_active":"2024-03-04T09:30:22.599833+0000","last_peered":"2024-03-04T09:30:22.599833+0000","last_clean":"2024-03-04T09:30:22.599833+0000","last_became_active":"2024-03-04T09:30:21.207323+0000","last_became_peered":"2024-03-04T09:30:21.207323+0000","last_unstale":"2024-03-04T09:30:22.599833+0000","last_undegraded":"2024-03-04T09:30:22.599833+0000","last_fullsized":"2024-03-04T09:30:22.599833+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:53:52.836749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.32","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.211090+0000","last_change":"2024-03-04T09:30:21.210114+0000","last_active":"2024-03-04T09:30:22.211090+0000","last_peered":"2024-03-04T09:30:22.211090+0000","last_clean":"2024-03-04T09:30:22.211090+0000","last_became_active":"2024-03-04T09:30:21.209823+0000","last_became_peered":"2024-03-04T09:30:21.209823+0000","last_unstale":"2024-03-04T09:30:22.211090+0000","last_undegraded":"2024-03-04T09:30:22.211090+0000","last_fullsized":"2024-03-04T09:30:22.211090+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:49:53.245627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.31","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.599906+0000","last_change":"2024-03-04T09:30:21.209101+0000","last_active":"2024-03-04T09:30:22.599906+0000","last_peered":"2024-03-04T09:30:22.599906+0000","last_clean":"2024-03-04T09:30:22.599906+0000","last_became_active":"2024-03-04T09:30:21.208703+0000","last_became_peered":"2024-03-04T09:30:21.208703+0000","last_unstale":"2024-03-04T09:30:22.599906+0000","last_undegraded":"2024-03-04T09:30:22.599906+0000","last_fullsized":"2024-03-04T09:30:22.599906+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:25:05.050243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.30","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.601160+0000","last_change":"2024-03-04T09:30:21.210486+0000","last_active":"2024-03-04T09:30:22.601160+0000","last_peered":"2024-03-04T09:30:22.601160+0000","last_clean":"2024-03-04T09:30:22.601160+0000","last_became_active":"2024-03-04T09:30:21.207407+0000","last_became_peered":"2024-03-04T09:30:21.207407+0000","last_unstale":"2024-03-04T09:30:22.601160+0000","last_undegraded":"2024-03-04T09:30:22.601160+0000","last_fullsized":"2024-03-04T09:30:22.601160+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:05:34.112195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.2f","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.208448+0000","last_change":"2024-03-04T09:30:21.199293+0000","last_active":"2024-03-04T09:30:22.208448+0000","last_peered":"2024-03-04T09:30:22.208448+0000","last_clean":"2024-03-04T09:30:22.208448+0000","last_became_active":"2024-03-04T09:30:21.199020+0000","last_became_peered":"2024-03-04T09:30:21.199020+0000","last_unstale":"2024-03-04T09:30:22.208448+0000","last_undegraded":"2024-03-04T09:30:22.208448+0000","last_fullsized":"2024-03-04T09:30:22.208448+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:49:16.693009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.2e","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.600547+0000","last_change":"2024-03-04T09:30:21.214466+0000","last_active":"2024-03-04T09:30:22.600547+0000","last_peered":"2024-03-04T09:30:22.600547+0000","last_clean":"2024-03-04T09:30:22.600547+0000","last_became_active":"2024-03-04T09:30:21.214193+0000","last_became_peered":"2024-03-04T09:30:21.214193+0000","last_unstale":"2024-03-04T09:30:22.600547+0000","last_undegraded":"2024-03-04T09:30:22.600547+0000","last_fullsized":"2024-03-04T09:30:22.600547+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:40:13.536981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.2d","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.211447+0000","last_change":"2024-03-04T09:30:21.219224+0000","last_active":"2024-03-04T09:30:22.211447+0000","last_peered":"2024-03-04T09:30:22.211447+0000","last_clean":"2024-03-04T09:30:22.211447+0000","last_became_active":"2024-03-04T09:30:21.218894+0000","last_became_peered":"2024-03-04T09:30:21.218894+0000","last_unstale":"2024-03-04T09:30:22.211447+0000","last_undegraded":"2024-03-04T09:30:22.211447+0000","last_fullsized":"2024-03-04T09:30:22.211447+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:27:46.546965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2c","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.207619+0000","last_change":"2024-03-04T09:30:21.210653+0000","last_active":"2024-03-04T09:30:22.207619+0000","last_peered":"2024-03-04T09:30:22.207619+0000","last_clean":"2024-03-04T09:30:22.207619+0000","last_became_active":"2024-03-04T09:30:21.210251+0000","last_became_peered":"2024-03-04T09:30:21.210251+0000","last_unstale":"2024-03-04T09:30:22.207619+0000","last_undegraded":"2024-03-04T09:30:22.207619+0000","last_fullsized":"2024-03-04T09:30:22.207619+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:53:52.836749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.2b","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.211219+0000","last_change":"2024-03-04T09:30:21.215051+0000","last_active":"2024-03-04T09:30:22.211219+0000","last_peered":"2024-03-04T09:30:22.211219+0000","last_clean":"2024-03-04T09:30:22.211219+0000","last_became_active":"2024-03-04T09:30:21.214690+0000","last_became_peered":"2024-03-04T09:30:21.214690+0000","last_unstale":"2024-03-04T09:30:22.211219+0000","last_undegraded":"2024-03-04T09:30:22.211219+0000","last_fullsized":"2024-03-04T09:30:22.211219+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:31:50.883753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2a","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.208053+0000","last_change":"2024-03-04T09:30:21.203403+0000","last_active":"2024-03-04T09:30:22.208053+0000","last_peered":"2024-03-04T09:30:22.208053+0000","last_clean":"2024-03-04T09:30:22.208053+0000","last_became_active":"2024-03-04T09:30:21.202934+0000","last_became_peered":"2024-03-04T09:30:21.202934+0000","last_unstale":"2024-03-04T09:30:22.208053+0000","last_undegraded":"2024-03-04T09:30:22.208053+0000","last_fullsized":"2024-03-04T09:30:22.208053+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:48:19.061200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.29","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.207876+0000","last_change":"2024-03-04T09:30:21.198463+0000","last_active":"2024-03-04T09:30:22.207876+0000","last_peered":"2024-03-04T09:30:22.207876+0000","last_clean":"2024-03-04T09:30:22.207876+0000","last_became_active":"2024-03-04T09:30:21.198172+0000","last_became_peered":"2024-03-04T09:30:21.198172+0000","last_unstale":"2024-03-04T09:30:22.207876+0000","last_undegraded":"2024-03-04T09:30:22.207876+0000","last_fullsized":"2024-03-04T09:30:22.207876+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:35:41.436658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.28","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.601374+0000","last_change":"2024-03-04T09:30:21.210170+0000","last_active":"2024-03-04T09:30:22.601374+0000","last_peered":"2024-03-04T09:30:22.601374+0000","last_clean":"2024-03-04T09:30:22.601374+0000","last_became_active":"2024-03-04T09:30:21.209873+0000","last_became_peered":"2024-03-04T09:30:21.209873+0000","last_unstale":"2024-03-04T09:30:22.601374+0000","last_undegraded":"2024-03-04T09:30:22.601374+0000","last_fullsized":"2024-03-04T09:30:22.601374+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:35:54.773629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.27","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.602868+0000","last_change":"2024-03-04T09:30:21.214806+0000","last_active":"2024-03-04T09:30:22.602868+0000","last_peered":"2024-03-04T09:30:22.602868+0000","last_clean":"2024-03-04T09:30:22.602868+0000","last_became_active":"2024-03-04T09:30:21.214161+0000","last_became_peered":"2024-03-04T09:30:21.214161+0000","last_unstale":"2024-03-04T09:30:22.602868+0000","last_undegraded":"2024-03-04T09:30:22.602868+0000","last_fullsized":"2024-03-04T09:30:22.602868+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:05:34.112195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.26","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.208046+0000","last_change":"2024-03-04T09:30:21.194645+0000","last_active":"2024-03-04T09:30:22.208046+0000","last_peered":"2024-03-04T09:30:22.208046+0000","last_clean":"2024-03-04T09:30:22.208046+0000","last_became_active":"2024-03-04T09:30:21.194362+0000","last_became_peered":"2024-03-04T09:30:21.194362+0000","last_unstale":"2024-03-04T09:30:22.208046+0000","last_undegraded":"2024-03-04T09:30:22.208046+0000","last_fullsized":"2024-03-04T09:30:22.208046+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:30:23.608760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.25","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.602145+0000","last_change":"2024-03-04T09:30:21.220051+0000","last_active":"2024-03-04T09:30:22.602145+0000","last_peered":"2024-03-04T09:30:22.602145+0000","last_clean":"2024-03-04T09:30:22.602145+0000","last_became_active":"2024-03-04T09:30:21.219729+0000","last_became_peered":"2024-03-04T09:30:21.219729+0000","last_unstale":"2024-03-04T09:30:22.602145+0000","last_undegraded":"2024-03-04T09:30:22.602145+0000","last_fullsized":"2024-03-04T09:30:22.602145+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:04:51.260343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.24","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.604357+0000","last_change":"2024-03-04T09:30:21.214188+0000","last_active":"2024-03-04T09:30:22.604357+0000","last_peered":"2024-03-04T09:30:22.604357+0000","last_clean":"2024-03-04T09:30:22.604357+0000","last_became_active":"2024-03-04T09:30:21.213867+0000","last_became_peered":"2024-03-04T09:30:21.213867+0000","last_unstale":"2024-03-04T09:30:22.604357+0000","last_undegraded":"2024-03-04T09:30:22.604357+0000","last_fullsized":"2024-03-04T09:30:22.604357+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:47:42.246844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.23","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.207656+0000","last_change":"2024-03-04T09:30:21.211260+0000","last_active":"2024-03-04T09:30:22.207656+0000","last_peered":"2024-03-04T09:30:22.207656+0000","last_clean":"2024-03-04T09:30:22.207656+0000","last_became_active":"2024-03-04T09:30:21.210927+0000","last_became_peered":"2024-03-04T09:30:21.210927+0000","last_unstale":"2024-03-04T09:30:22.207656+0000","last_undegraded":"2024-03-04T09:30:22.207656+0000","last_fullsized":"2024-03-04T09:30:22.207656+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:00:49.856992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.22","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.211146+0000","last_change":"2024-03-04T09:30:21.221279+0000","last_active":"2024-03-04T09:30:22.211146+0000","last_peered":"2024-03-04T09:30:22.211146+0000","last_clean":"2024-03-04T09:30:22.211146+0000","last_became_active":"2024-03-04T09:30:21.220582+0000","last_became_peered":"2024-03-04T09:30:21.220582+0000","last_unstale":"2024-03-04T09:30:22.211146+0000","last_undegraded":"2024-03-04T09:30:22.211146+0000","last_fullsized":"2024-03-04T09:30:22.211146+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:12:10.257211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.21","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.600391+0000","last_change":"2024-03-04T09:30:21.213950+0000","last_active":"2024-03-04T09:30:22.600391+0000","last_peered":"2024-03-04T09:30:22.600391+0000","last_clean":"2024-03-04T09:30:22.600391+0000","last_became_active":"2024-03-04T09:30:21.213595+0000","last_became_peered":"2024-03-04T09:30:21.213595+0000","last_unstale":"2024-03-04T09:30:22.600391+0000","last_undegraded":"2024-03-04T09:30:22.600391+0000","last_fullsized":"2024-03-04T09:30:22.600391+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:52:21.419597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.20","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.202906+0000","last_change":"2024-03-04T09:30:21.191402+0000","last_active":"2024-03-04T09:30:22.202906+0000","last_peered":"2024-03-04T09:30:22.202906+0000","last_clean":"2024-03-04T09:30:22.202906+0000","last_became_active":"2024-03-04T09:30:21.190931+0000","last_became_peered":"2024-03-04T09:30:21.190931+0000","last_unstale":"2024-03-04T09:30:22.202906+0000","last_undegraded":"2024-03-04T09:30:22.202906+0000","last_fullsized":"2024-03-04T09:30:22.202906+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:55:38.377212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.211543+0000","last_change":"2024-03-04T09:30:21.205326+0000","last_active":"2024-03-04T09:30:22.211543+0000","last_peered":"2024-03-04T09:30:22.211543+0000","last_clean":"2024-03-04T09:30:22.211543+0000","last_became_active":"2024-03-04T09:30:21.205025+0000","last_became_peered":"2024-03-04T09:30:21.205025+0000","last_unstale":"2024-03-04T09:30:22.211543+0000","last_undegraded":"2024-03-04T09:30:22.211543+0000","last_fullsized":"2024-03-04T09:30:22.211543+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:09:41.869707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1f","version":"93'4","reported_seq":18,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.208447+0000","last_change":"2024-03-04T09:30:20.202696+0000","last_active":"2024-03-04T09:30:22.208447+0000","last_peered":"2024-03-04T09:30:22.208447+0000","last_clean":"2024-03-04T09:30:22.208447+0000","last_became_active":"2024-03-04T09:30:20.201031+0000","last_became_peered":"2024-03-04T09:30:20.201031+0000","last_unstale":"2024-03-04T09:30:22.208447+0000","last_undegraded":"2024-03-04T09:30:22.208447+0000","last_fullsized":"2024-03-04T09:30:22.208447+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_clean_scrub_stamp":"2024-03-04T09:30:19.137440+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:35:53.756969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.600779+0000","last_change":"2024-03-04T09:30:21.208078+0000","last_active":"2024-03-04T09:30:22.600779+0000","last_peered":"2024-03-04T09:30:22.600779+0000","last_clean":"2024-03-04T09:30:22.600779+0000","last_became_active":"2024-03-04T09:30:21.207737+0000","last_became_peered":"2024-03-04T09:30:21.207737+0000","last_unstale":"2024-03-04T09:30:22.600779+0000","last_undegraded":"2024-03-04T09:30:22.600779+0000","last_fullsized":"2024-03-04T09:30:22.600779+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:06:54.905510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1e","version":"93'1","reported_seq":15,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.602684+0000","last_change":"2024-03-04T09:30:20.202396+0000","last_active":"2024-03-04T09:30:22.602684+0000","last_peered":"2024-03-04T09:30:22.602684+0000","last_clean":"2024-03-04T09:30:22.602684+0000","last_became_active":"2024-03-04T09:30:20.202080+0000","last_became_peered":"2024-03-04T09:30:20.202080+0000","last_unstale":"2024-03-04T09:30:22.602684+0000","last_undegraded":"2024-03-04T09:30:22.602684+0000","last_fullsized":"2024-03-04T09:30:22.602684+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_clean_scrub_stamp":"2024-03-04T09:30:19.137440+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:43:31.237510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.203381+0000","last_change":"2024-03-04T09:30:21.208566+0000","last_active":"2024-03-04T09:30:22.203381+0000","last_peered":"2024-03-04T09:30:22.203381+0000","last_clean":"2024-03-04T09:30:22.203381+0000","last_became_active":"2024-03-04T09:30:21.207592+0000","last_became_peered":"2024-03-04T09:30:21.207592+0000","last_unstale":"2024-03-04T09:30:22.203381+0000","last_undegraded":"2024-03-04T09:30:22.203381+0000","last_fullsized":"2024-03-04T09:30:22.203381+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:49:32.875495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1d","version":"93'2","reported_seq":16,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.207671+0000","last_change":"2024-03-04T09:30:20.199898+0000","last_active":"2024-03-04T09:30:22.207671+0000","last_peered":"2024-03-04T09:30:22.207671+0000","last_clean":"2024-03-04T09:30:22.207671+0000","last_became_active":"2024-03-04T09:30:20.199607+0000","last_became_peered":"2024-03-04T09:30:20.199607+0000","last_unstale":"2024-03-04T09:30:22.207671+0000","last_undegraded":"2024-03-04T09:30:22.207671+0000","last_fullsized":"2024-03-04T09:30:22.207671+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_clean_scrub_stamp":"2024-03-04T09:30:19.137440+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:59:30.574909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.206573+0000","last_change":"2024-03-04T09:30:21.214934+0000","last_active":"2024-03-04T09:30:22.206573+0000","last_peered":"2024-03-04T09:30:22.206573+0000","last_clean":"2024-03-04T09:30:22.206573+0000","last_became_active":"2024-03-04T09:30:21.214642+0000","last_became_peered":"2024-03-04T09:30:21.214642+0000","last_unstale":"2024-03-04T09:30:22.206573+0000","last_undegraded":"2024-03-04T09:30:22.206573+0000","last_fullsized":"2024-03-04T09:30:22.206573+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:05:34.112195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1c","version":"93'2","reported_seq":16,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.207658+0000","last_change":"2024-03-04T09:30:20.201824+0000","last_active":"2024-03-04T09:30:22.207658+0000","last_peered":"2024-03-04T09:30:22.207658+0000","last_clean":"2024-03-04T09:30:22.207658+0000","last_became_active":"2024-03-04T09:30:20.201263+0000","last_became_peered":"2024-03-04T09:30:20.201263+0000","last_unstale":"2024-03-04T09:30:22.207658+0000","last_undegraded":"2024-03-04T09:30:22.207658+0000","last_fullsized":"2024-03-04T09:30:22.207658+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_clean_scrub_stamp":"2024-03-04T09:30:19.137440+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:26:11.379928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.208700+0000","last_change":"2024-03-04T09:30:21.212815+0000","last_active":"2024-03-04T09:30:22.208700+0000","last_peered":"2024-03-04T09:30:22.208700+0000","last_clean":"2024-03-04T09:30:22.208700+0000","last_became_active":"2024-03-04T09:30:21.212499+0000","last_became_peered":"2024-03-04T09:30:21.212499+0000","last_unstale":"2024-03-04T09:30:22.208700+0000","last_undegraded":"2024-03-04T09:30:22.208700+0000","last_fullsized":"2024-03-04T09:30:22.208700+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:40:36.757300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1b","version":"93'1","reported_seq":15,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.203260+0000","last_change":"2024-03-04T09:30:20.196754+0000","last_active":"2024-03-04T09:30:22.203260+0000","last_peered":"2024-03-04T09:30:22.203260+0000","last_clean":"2024-03-04T09:30:22.203260+0000","last_became_active":"2024-03-04T09:30:20.196436+0000","last_became_peered":"2024-03-04T09:30:20.196436+0000","last_unstale":"2024-03-04T09:30:22.203260+0000","last_undegraded":"2024-03-04T09:30:22.203260+0000","last_fullsized":"2024-03-04T09:30:22.203260+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_clean_scrub_stamp":"2024-03-04T09:30:19.137440+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:27:45.530306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.603485+0000","last_change":"2024-03-04T09:30:21.209123+0000","last_active":"2024-03-04T09:30:22.603485+0000","last_peered":"2024-03-04T09:30:22.603485+0000","last_clean":"2024-03-04T09:30:22.603485+0000","last_became_active":"2024-03-04T09:30:21.207804+0000","last_became_peered":"2024-03-04T09:30:21.207804+0000","last_unstale":"2024-03-04T09:30:22.603485+0000","last_undegraded":"2024-03-04T09:30:22.603485+0000","last_fullsized":"2024-03-04T09:30:22.603485+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:25:05.050243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1a","version":"93'1","reported_seq":15,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.207584+0000","last_change":"2024-03-04T09:30:20.205820+0000","last_active":"2024-03-04T09:30:22.207584+0000","last_peered":"2024-03-04T09:30:22.207584+0000","last_clean":"2024-03-04T09:30:22.207584+0000","last_became_active":"2024-03-04T09:30:20.205544+0000","last_became_peered":"2024-03-04T09:30:20.205544+0000","last_unstale":"2024-03-04T09:30:22.207584+0000","last_undegraded":"2024-03-04T09:30:22.207584+0000","last_fullsized":"2024-03-04T09:30:22.207584+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_clean_scrub_stamp":"2024-03-04T09:30:19.137440+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:05:33.095535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.599877+0000","last_change":"2024-03-04T09:30:21.209440+0000","last_active":"2024-03-04T09:30:22.599877+0000","last_peered":"2024-03-04T09:30:22.599877+0000","last_clean":"2024-03-04T09:30:22.599877+0000","last_became_active":"2024-03-04T09:30:21.209149+0000","last_became_peered":"2024-03-04T09:30:21.209149+0000","last_unstale":"2024-03-04T09:30:22.599877+0000","last_undegraded":"2024-03-04T09:30:22.599877+0000","last_fullsized":"2024-03-04T09:30:22.599877+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:38:54.890507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":14,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.601645+0000","last_change":"2024-03-04T09:30:20.204817+0000","last_active":"2024-03-04T09:30:22.601645+0000","last_peered":"2024-03-04T09:30:22.601645+0000","last_clean":"2024-03-04T09:30:22.601645+0000","last_became_active":"2024-03-04T09:30:20.204526+0000","last_became_peered":"2024-03-04T09:30:20.204526+0000","last_unstale":"2024-03-04T09:30:22.601645+0000","last_undegraded":"2024-03-04T09:30:22.601645+0000","last_fullsized":"2024-03-04T09:30:22.601645+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_clean_scrub_stamp":"2024-03-04T09:30:19.137440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:31:56.088975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.206665+0000","last_change":"2024-03-04T09:30:21.215369+0000","last_active":"2024-03-04T09:30:22.206665+0000","last_peered":"2024-03-04T09:30:22.206665+0000","last_clean":"2024-03-04T09:30:22.206665+0000","last_became_active":"2024-03-04T09:30:21.215098+0000","last_became_peered":"2024-03-04T09:30:21.215098+0000","last_unstale":"2024-03-04T09:30:22.206665+0000","last_undegraded":"2024-03-04T09:30:22.206665+0000","last_fullsized":"2024-03-04T09:30:22.206665+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:35:54.773629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":14,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.202898+0000","last_change":"2024-03-04T09:30:20.188458+0000","last_active":"2024-03-04T09:30:22.202898+0000","last_peered":"2024-03-04T09:30:22.202898+0000","last_clean":"2024-03-04T09:30:22.202898+0000","last_became_active":"2024-03-04T09:30:20.188160+0000","last_became_peered":"2024-03-04T09:30:20.188160+0000","last_unstale":"2024-03-04T09:30:22.202898+0000","last_undegraded":"2024-03-04T09:30:22.202898+0000","last_fullsized":"2024-03-04T09:30:22.202898+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_clean_scrub_stamp":"2024-03-04T09:30:19.137440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:44:14.674247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.208239+0000","last_change":"2024-03-04T09:30:21.206057+0000","last_active":"2024-03-04T09:30:22.208239+0000","last_peered":"2024-03-04T09:30:22.208239+0000","last_clean":"2024-03-04T09:30:22.208239+0000","last_became_active":"2024-03-04T09:30:21.205516+0000","last_became_peered":"2024-03-04T09:30:21.205516+0000","last_unstale":"2024-03-04T09:30:22.208239+0000","last_undegraded":"2024-03-04T09:30:22.208239+0000","last_fullsized":"2024-03-04T09:30:22.208239+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:47:59.445932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":14,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.203793+0000","last_change":"2024-03-04T09:30:20.203539+0000","last_active":"2024-03-04T09:30:22.203793+0000","last_peered":"2024-03-04T09:30:22.203793+0000","last_clean":"2024-03-04T09:30:22.203793+0000","last_became_active":"2024-03-04T09:30:20.202487+0000","last_became_peered":"2024-03-04T09:30:20.202487+0000","last_unstale":"2024-03-04T09:30:22.203793+0000","last_undegraded":"2024-03-04T09:30:22.203793+0000","last_fullsized":"2024-03-04T09:30:22.203793+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_clean_scrub_stamp":"2024-03-04T09:30:19.137440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:44:40.582064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.207324+0000","last_change":"2024-03-04T09:30:21.214060+0000","last_active":"2024-03-04T09:30:22.207324+0000","last_peered":"2024-03-04T09:30:22.207324+0000","last_clean":"2024-03-04T09:30:22.207324+0000","last_became_active":"2024-03-04T09:30:21.212861+0000","last_became_peered":"2024-03-04T09:30:21.212861+0000","last_unstale":"2024-03-04T09:30:22.207324+0000","last_undegraded":"2024-03-04T09:30:22.207324+0000","last_fullsized":"2024-03-04T09:30:22.207324+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:04:51.260343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":14,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.600464+0000","last_change":"2024-03-04T09:30:20.203934+0000","last_active":"2024-03-04T09:30:22.600464+0000","last_peered":"2024-03-04T09:30:22.600464+0000","last_clean":"2024-03-04T09:30:22.600464+0000","last_became_active":"2024-03-04T09:30:20.203656+0000","last_became_peered":"2024-03-04T09:30:20.203656+0000","last_unstale":"2024-03-04T09:30:22.600464+0000","last_undegraded":"2024-03-04T09:30:22.600464+0000","last_fullsized":"2024-03-04T09:30:22.600464+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_clean_scrub_stamp":"2024-03-04T09:30:19.137440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:59:30.574909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.604462+0000","last_change":"2024-03-04T09:30:21.214168+0000","last_active":"2024-03-04T09:30:22.604462+0000","last_peered":"2024-03-04T09:30:22.604462+0000","last_clean":"2024-03-04T09:30:22.604462+0000","last_became_active":"2024-03-04T09:30:21.213406+0000","last_became_peered":"2024-03-04T09:30:21.213406+0000","last_unstale":"2024-03-04T09:30:22.604462+0000","last_undegraded":"2024-03-04T09:30:22.604462+0000","last_fullsized":"2024-03-04T09:30:22.604462+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:53:52.836749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.15","version":"93'3","reported_seq":17,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.602467+0000","last_change":"2024-03-04T09:30:20.204159+0000","last_active":"2024-03-04T09:30:22.602467+0000","last_peered":"2024-03-04T09:30:22.602467+0000","last_clean":"2024-03-04T09:30:22.602467+0000","last_became_active":"2024-03-04T09:30:20.203855+0000","last_became_peered":"2024-03-04T09:30:20.203855+0000","last_unstale":"2024-03-04T09:30:22.602467+0000","last_undegraded":"2024-03-04T09:30:22.602467+0000","last_fullsized":"2024-03-04T09:30:22.602467+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_clean_scrub_stamp":"2024-03-04T09:30:19.137440+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:15:51.774866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.602167+0000","last_change":"2024-03-04T09:30:21.203568+0000","last_active":"2024-03-04T09:30:22.602167+0000","last_peered":"2024-03-04T09:30:22.602167+0000","last_clean":"2024-03-04T09:30:22.602167+0000","last_became_active":"2024-03-04T09:30:21.202806+0000","last_became_peered":"2024-03-04T09:30:21.202806+0000","last_unstale":"2024-03-04T09:30:22.602167+0000","last_undegraded":"2024-03-04T09:30:22.602167+0000","last_fullsized":"2024-03-04T09:30:22.602167+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:53:52.836749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"93'6","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.207893+0000","last_change":"2024-03-04T09:30:20.203706+0000","last_active":"2024-03-04T09:30:22.207893+0000","last_peered":"2024-03-04T09:30:22.207893+0000","last_clean":"2024-03-04T09:30:22.207893+0000","last_became_active":"2024-03-04T09:30:20.203345+0000","last_became_peered":"2024-03-04T09:30:20.203345+0000","last_unstale":"2024-03-04T09:30:22.207893+0000","last_undegraded":"2024-03-04T09:30:22.207893+0000","last_fullsized":"2024-03-04T09:30:22.207893+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_clean_scrub_stamp":"2024-03-04T09:30:19.137440+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:43:31.237510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.599878+0000","last_change":"2024-03-04T09:30:21.202680+0000","last_active":"2024-03-04T09:30:22.599878+0000","last_peered":"2024-03-04T09:30:22.599878+0000","last_clean":"2024-03-04T09:30:22.599878+0000","last_became_active":"2024-03-04T09:30:21.202453+0000","last_became_peered":"2024-03-04T09:30:21.202453+0000","last_unstale":"2024-03-04T09:30:22.599878+0000","last_undegraded":"2024-03-04T09:30:22.599878+0000","last_fullsized":"2024-03-04T09:30:22.599878+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:30:09.483436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.13","version":"93'2","reported_seq":16,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.211387+0000","last_change":"2024-03-04T09:30:20.202977+0000","last_active":"2024-03-04T09:30:22.211387+0000","last_peered":"2024-03-04T09:30:22.211387+0000","last_clean":"2024-03-04T09:30:22.211387+0000","last_became_active":"2024-03-04T09:30:20.201161+0000","last_became_peered":"2024-03-04T09:30:20.201161+0000","last_unstale":"2024-03-04T09:30:22.211387+0000","last_undegraded":"2024-03-04T09:30:22.211387+0000","last_fullsized":"2024-03-04T09:30:22.211387+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_clean_scrub_stamp":"2024-03-04T09:30:19.137440+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:39:23.646435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.203295+0000","last_change":"2024-03-04T09:30:21.210999+0000","last_active":"2024-03-04T09:30:22.203295+0000","last_peered":"2024-03-04T09:30:22.203295+0000","last_clean":"2024-03-04T09:30:22.203295+0000","last_became_active":"2024-03-04T09:30:21.207217+0000","last_became_peered":"2024-03-04T09:30:21.207217+0000","last_unstale":"2024-03-04T09:30:22.203295+0000","last_undegraded":"2024-03-04T09:30:22.203295+0000","last_fullsized":"2024-03-04T09:30:22.203295+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:01:32.367615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":14,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.211507+0000","last_change":"2024-03-04T09:30:20.184266+0000","last_active":"2024-03-04T09:30:22.211507+0000","last_peered":"2024-03-04T09:30:22.211507+0000","last_clean":"2024-03-04T09:30:22.211507+0000","last_became_active":"2024-03-04T09:30:20.183857+0000","last_became_peered":"2024-03-04T09:30:20.183857+0000","last_unstale":"2024-03-04T09:30:22.211507+0000","last_undegraded":"2024-03-04T09:30:22.211507+0000","last_fullsized":"2024-03-04T09:30:22.211507+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_clean_scrub_stamp":"2024-03-04T09:30:19.137440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:26:40.873839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.205422+0000","last_change":"2024-03-04T09:30:21.215813+0000","last_active":"2024-03-04T09:30:22.205422+0000","last_peered":"2024-03-04T09:30:22.205422+0000","last_clean":"2024-03-04T09:30:22.205422+0000","last_became_active":"2024-03-04T09:30:21.215547+0000","last_became_peered":"2024-03-04T09:30:21.215547+0000","last_unstale":"2024-03-04T09:30:22.205422+0000","last_undegraded":"2024-03-04T09:30:22.205422+0000","last_fullsized":"2024-03-04T09:30:22.205422+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:51:18.057309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":14,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.205777+0000","last_change":"2024-03-04T09:30:20.206681+0000","last_active":"2024-03-04T09:30:22.205777+0000","last_peered":"2024-03-04T09:30:22.205777+0000","last_clean":"2024-03-04T09:30:22.205777+0000","last_became_active":"2024-03-04T09:30:20.206212+0000","last_became_peered":"2024-03-04T09:30:20.206212+0000","last_unstale":"2024-03-04T09:30:22.205777+0000","last_undegraded":"2024-03-04T09:30:22.205777+0000","last_fullsized":"2024-03-04T09:30:22.205777+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_clean_scrub_stamp":"2024-03-04T09:30:19.137440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:31:56.088975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.210983+0000","last_change":"2024-03-04T09:30:21.217458+0000","last_active":"2024-03-04T09:30:22.210983+0000","last_peered":"2024-03-04T09:30:22.210983+0000","last_clean":"2024-03-04T09:30:22.210983+0000","last_became_active":"2024-03-04T09:30:21.217201+0000","last_became_peered":"2024-03-04T09:30:21.217201+0000","last_unstale":"2024-03-04T09:30:22.210983+0000","last_undegraded":"2024-03-04T09:30:22.210983+0000","last_fullsized":"2024-03-04T09:30:22.210983+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:18:40.161366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.10","version":"93'8","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.203115+0000","last_change":"2024-03-04T09:30:20.193928+0000","last_active":"2024-03-04T09:30:22.203115+0000","last_peered":"2024-03-04T09:30:22.203115+0000","last_clean":"2024-03-04T09:30:22.203115+0000","last_became_active":"2024-03-04T09:30:20.193601+0000","last_became_peered":"2024-03-04T09:30:20.193601+0000","last_unstale":"2024-03-04T09:30:22.203115+0000","last_undegraded":"2024-03-04T09:30:22.203115+0000","last_fullsized":"2024-03-04T09:30:22.203115+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_clean_scrub_stamp":"2024-03-04T09:30:19.137440+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:33:41.777208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.604363+0000","last_change":"2024-03-04T09:30:21.209945+0000","last_active":"2024-03-04T09:30:22.604363+0000","last_peered":"2024-03-04T09:30:22.604363+0000","last_clean":"2024-03-04T09:30:22.604363+0000","last_became_active":"2024-03-04T09:30:21.209231+0000","last_became_peered":"2024-03-04T09:30:21.209231+0000","last_unstale":"2024-03-04T09:30:22.604363+0000","last_undegraded":"2024-03-04T09:30:22.604363+0000","last_fullsized":"2024-03-04T09:30:22.604363+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:04:51.260343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.e","version":"93'2","reported_seq":16,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.206181+0000","last_change":"2024-03-04T09:30:20.197183+0000","last_active":"2024-03-04T09:30:22.206181+0000","last_peered":"2024-03-04T09:30:22.206181+0000","last_clean":"2024-03-04T09:30:22.206181+0000","last_became_active":"2024-03-04T09:30:20.196884+0000","last_became_peered":"2024-03-04T09:30:20.196884+0000","last_unstale":"2024-03-04T09:30:22.206181+0000","last_undegraded":"2024-03-04T09:30:22.206181+0000","last_fullsized":"2024-03-04T09:30:22.206181+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_clean_scrub_stamp":"2024-03-04T09:30:19.137440+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:42:56.761983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.208741+0000","last_change":"2024-03-04T09:30:21.207297+0000","last_active":"2024-03-04T09:30:22.208741+0000","last_peered":"2024-03-04T09:30:22.208741+0000","last_clean":"2024-03-04T09:30:22.208741+0000","last_became_active":"2024-03-04T09:30:21.206895+0000","last_became_peered":"2024-03-04T09:30:21.206895+0000","last_unstale":"2024-03-04T09:30:22.208741+0000","last_undegraded":"2024-03-04T09:30:22.208741+0000","last_fullsized":"2024-03-04T09:30:22.208741+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:40:11.984649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.d","version":"93'1","reported_seq":15,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.602718+0000","last_change":"2024-03-04T09:30:20.201461+0000","last_active":"2024-03-04T09:30:22.602718+0000","last_peered":"2024-03-04T09:30:22.602718+0000","last_clean":"2024-03-04T09:30:22.602718+0000","last_became_active":"2024-03-04T09:30:20.201181+0000","last_became_peered":"2024-03-04T09:30:20.201181+0000","last_unstale":"2024-03-04T09:30:22.602718+0000","last_undegraded":"2024-03-04T09:30:22.602718+0000","last_fullsized":"2024-03-04T09:30:22.602718+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_clean_scrub_stamp":"2024-03-04T09:30:19.137440+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:40:10.967990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.602630+0000","last_change":"2024-03-04T09:30:21.215009+0000","last_active":"2024-03-04T09:30:22.602630+0000","last_peered":"2024-03-04T09:30:22.602630+0000","last_clean":"2024-03-04T09:30:22.602630+0000","last_became_active":"2024-03-04T09:30:21.214716+0000","last_became_peered":"2024-03-04T09:30:21.214716+0000","last_unstale":"2024-03-04T09:30:22.602630+0000","last_undegraded":"2024-03-04T09:30:22.602630+0000","last_fullsized":"2024-03-04T09:30:22.602630+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:30:09.483436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":14,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.207817+0000","last_change":"2024-03-04T09:30:20.202007+0000","last_active":"2024-03-04T09:30:22.207817+0000","last_peered":"2024-03-04T09:30:22.207817+0000","last_clean":"2024-03-04T09:30:22.207817+0000","last_became_active":"2024-03-04T09:30:20.201633+0000","last_became_peered":"2024-03-04T09:30:20.201633+0000","last_unstale":"2024-03-04T09:30:22.207817+0000","last_undegraded":"2024-03-04T09:30:22.207817+0000","last_fullsized":"2024-03-04T09:30:22.207817+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_clean_scrub_stamp":"2024-03-04T09:30:19.137440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:56:56.967774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.211265+0000","last_change":"2024-03-04T09:30:21.220213+0000","last_active":"2024-03-04T09:30:22.211265+0000","last_peered":"2024-03-04T09:30:22.211265+0000","last_clean":"2024-03-04T09:30:22.211265+0000","last_became_active":"2024-03-04T09:30:21.217126+0000","last_became_peered":"2024-03-04T09:30:21.217126+0000","last_unstale":"2024-03-04T09:30:22.211265+0000","last_undegraded":"2024-03-04T09:30:22.211265+0000","last_fullsized":"2024-03-04T09:30:22.211265+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:44:15.690906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.b","version":"93'2","reported_seq":16,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.207756+0000","last_change":"2024-03-04T09:30:20.203741+0000","last_active":"2024-03-04T09:30:22.207756+0000","last_peered":"2024-03-04T09:30:22.207756+0000","last_clean":"2024-03-04T09:30:22.207756+0000","last_became_active":"2024-03-04T09:30:20.203455+0000","last_became_peered":"2024-03-04T09:30:20.203455+0000","last_unstale":"2024-03-04T09:30:22.207756+0000","last_undegraded":"2024-03-04T09:30:22.207756+0000","last_fullsized":"2024-03-04T09:30:22.207756+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_clean_scrub_stamp":"2024-03-04T09:30:19.137440+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:22:43.529585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.211333+0000","last_change":"2024-03-04T09:30:21.208688+0000","last_active":"2024-03-04T09:30:22.211333+0000","last_peered":"2024-03-04T09:30:22.211333+0000","last_clean":"2024-03-04T09:30:22.211333+0000","last_became_active":"2024-03-04T09:30:21.208387+0000","last_became_peered":"2024-03-04T09:30:21.208387+0000","last_unstale":"2024-03-04T09:30:22.211333+0000","last_undegraded":"2024-03-04T09:30:22.211333+0000","last_fullsized":"2024-03-04T09:30:22.211333+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:33:42.793868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":14,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.208765+0000","last_change":"2024-03-04T09:30:20.201935+0000","last_active":"2024-03-04T09:30:22.208765+0000","last_peered":"2024-03-04T09:30:22.208765+0000","last_clean":"2024-03-04T09:30:22.208765+0000","last_became_active":"2024-03-04T09:30:20.201641+0000","last_became_peered":"2024-03-04T09:30:20.201641+0000","last_unstale":"2024-03-04T09:30:22.208765+0000","last_undegraded":"2024-03-04T09:30:22.208765+0000","last_fullsized":"2024-03-04T09:30:22.208765+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_clean_scrub_stamp":"2024-03-04T09:30:19.137440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:39:06.431069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.210904+0000","last_change":"2024-03-04T09:30:21.216656+0000","last_active":"2024-03-04T09:30:22.210904+0000","last_peered":"2024-03-04T09:30:22.210904+0000","last_clean":"2024-03-04T09:30:22.210904+0000","last_became_active":"2024-03-04T09:30:21.216112+0000","last_became_peered":"2024-03-04T09:30:21.216112+0000","last_unstale":"2024-03-04T09:30:22.210904+0000","last_undegraded":"2024-03-04T09:30:22.210904+0000","last_fullsized":"2024-03-04T09:30:22.210904+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:50:33.851581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":14,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.208621+0000","last_change":"2024-03-04T09:30:20.200225+0000","last_active":"2024-03-04T09:30:22.208621+0000","last_peered":"2024-03-04T09:30:22.208621+0000","last_clean":"2024-03-04T09:30:22.208621+0000","last_became_active":"2024-03-04T09:30:20.199914+0000","last_became_peered":"2024-03-04T09:30:20.199914+0000","last_unstale":"2024-03-04T09:30:22.208621+0000","last_undegraded":"2024-03-04T09:30:22.208621+0000","last_fullsized":"2024-03-04T09:30:22.208621+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_clean_scrub_stamp":"2024-03-04T09:30:19.137440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:52:31.073581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.601373+0000","last_change":"2024-03-04T09:30:21.210036+0000","last_active":"2024-03-04T09:30:22.601373+0000","last_peered":"2024-03-04T09:30:22.601373+0000","last_clean":"2024-03-04T09:30:22.601373+0000","last_became_active":"2024-03-04T09:30:21.209732+0000","last_became_peered":"2024-03-04T09:30:21.209732+0000","last_unstale":"2024-03-04T09:30:22.601373+0000","last_undegraded":"2024-03-04T09:30:22.601373+0000","last_fullsized":"2024-03-04T09:30:22.601373+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:26:12.396587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":14,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.207572+0000","last_change":"2024-03-04T09:30:20.204803+0000","last_active":"2024-03-04T09:30:22.207572+0000","last_peered":"2024-03-04T09:30:22.207572+0000","last_clean":"2024-03-04T09:30:22.207572+0000","last_became_active":"2024-03-04T09:30:20.203153+0000","last_became_peered":"2024-03-04T09:30:20.203153+0000","last_unstale":"2024-03-04T09:30:22.207572+0000","last_undegraded":"2024-03-04T09:30:22.207572+0000","last_fullsized":"2024-03-04T09:30:22.207572+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_clean_scrub_stamp":"2024-03-04T09:30:19.137440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:36:55.596046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.208303+0000","last_change":"2024-03-04T09:30:21.201417+0000","last_active":"2024-03-04T09:30:22.208303+0000","last_peered":"2024-03-04T09:30:22.208303+0000","last_clean":"2024-03-04T09:30:22.208303+0000","last_became_active":"2024-03-04T09:30:21.201134+0000","last_became_peered":"2024-03-04T09:30:21.201134+0000","last_unstale":"2024-03-04T09:30:22.208303+0000","last_undegraded":"2024-03-04T09:30:22.208303+0000","last_fullsized":"2024-03-04T09:30:22.208303+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:03:49.406310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.7","version":"93'2","reported_seq":16,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.208564+0000","last_change":"2024-03-04T09:30:20.204428+0000","last_active":"2024-03-04T09:30:22.208564+0000","last_peered":"2024-03-04T09:30:22.208564+0000","last_clean":"2024-03-04T09:30:22.208564+0000","last_became_active":"2024-03-04T09:30:20.204109+0000","last_became_peered":"2024-03-04T09:30:20.204109+0000","last_unstale":"2024-03-04T09:30:22.208564+0000","last_undegraded":"2024-03-04T09:30:22.208564+0000","last_fullsized":"2024-03-04T09:30:22.208564+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_clean_scrub_stamp":"2024-03-04T09:30:19.137440+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:45:27.624178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.203731+0000","last_change":"2024-03-04T09:30:21.212032+0000","last_active":"2024-03-04T09:30:22.203731+0000","last_peered":"2024-03-04T09:30:22.203731+0000","last_clean":"2024-03-04T09:30:22.203731+0000","last_became_active":"2024-03-04T09:30:21.211735+0000","last_became_peered":"2024-03-04T09:30:21.211735+0000","last_unstale":"2024-03-04T09:30:22.203731+0000","last_undegraded":"2024-03-04T09:30:22.203731+0000","last_fullsized":"2024-03-04T09:30:22.203731+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:03:48.340632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.6","version":"94'1","reported_seq":15,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.210061+0000","last_change":"2024-03-04T09:30:20.202239+0000","last_active":"2024-03-04T09:30:22.210061+0000","last_peered":"2024-03-04T09:30:22.210061+0000","last_clean":"2024-03-04T09:30:22.210061+0000","last_became_active":"2024-03-04T09:30:20.201889+0000","last_became_peered":"2024-03-04T09:30:20.201889+0000","last_unstale":"2024-03-04T09:30:22.210061+0000","last_undegraded":"2024-03-04T09:30:22.210061+0000","last_fullsized":"2024-03-04T09:30:22.210061+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_clean_scrub_stamp":"2024-03-04T09:30:19.137440+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:47:41.230184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.203451+0000","last_change":"2024-03-04T09:30:21.212635+0000","last_active":"2024-03-04T09:30:22.203451+0000","last_peered":"2024-03-04T09:30:22.203451+0000","last_clean":"2024-03-04T09:30:22.203451+0000","last_became_active":"2024-03-04T09:30:21.212346+0000","last_became_peered":"2024-03-04T09:30:21.212346+0000","last_unstale":"2024-03-04T09:30:22.203451+0000","last_undegraded":"2024-03-04T09:30:22.203451+0000","last_fullsized":"2024-03-04T09:30:22.203451+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:52:32.090241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":14,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.207918+0000","last_change":"2024-03-04T09:30:20.206409+0000","last_active":"2024-03-04T09:30:22.207918+0000","last_peered":"2024-03-04T09:30:22.207918+0000","last_clean":"2024-03-04T09:30:22.207918+0000","last_became_active":"2024-03-04T09:30:20.204756+0000","last_became_peered":"2024-03-04T09:30:20.204756+0000","last_unstale":"2024-03-04T09:30:22.207918+0000","last_undegraded":"2024-03-04T09:30:22.207918+0000","last_fullsized":"2024-03-04T09:30:22.207918+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_clean_scrub_stamp":"2024-03-04T09:30:19.137440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:51:17.040650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.602450+0000","last_change":"2024-03-04T09:30:21.216899+0000","last_active":"2024-03-04T09:30:22.602450+0000","last_peered":"2024-03-04T09:30:22.602450+0000","last_clean":"2024-03-04T09:30:22.602450+0000","last_became_active":"2024-03-04T09:30:21.215437+0000","last_became_peered":"2024-03-04T09:30:21.215437+0000","last_unstale":"2024-03-04T09:30:22.602450+0000","last_undegraded":"2024-03-04T09:30:22.602450+0000","last_fullsized":"2024-03-04T09:30:22.602450+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:56:57.984433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":14,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.604548+0000","last_change":"2024-03-04T09:30:20.200047+0000","last_active":"2024-03-04T09:30:22.604548+0000","last_peered":"2024-03-04T09:30:22.604548+0000","last_clean":"2024-03-04T09:30:22.604548+0000","last_became_active":"2024-03-04T09:30:20.199490+0000","last_became_peered":"2024-03-04T09:30:20.199490+0000","last_unstale":"2024-03-04T09:30:22.604548+0000","last_undegraded":"2024-03-04T09:30:22.604548+0000","last_fullsized":"2024-03-04T09:30:22.604548+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_clean_scrub_stamp":"2024-03-04T09:30:19.137440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:43:31.237510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.211418+0000","last_change":"2024-03-04T09:30:21.221508+0000","last_active":"2024-03-04T09:30:22.211418+0000","last_peered":"2024-03-04T09:30:22.211418+0000","last_clean":"2024-03-04T09:30:22.211418+0000","last_became_active":"2024-03-04T09:30:21.220985+0000","last_became_peered":"2024-03-04T09:30:21.220985+0000","last_unstale":"2024-03-04T09:30:22.211418+0000","last_undegraded":"2024-03-04T09:30:22.211418+0000","last_fullsized":"2024-03-04T09:30:22.211418+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:44:41.598724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":14,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.207858+0000","last_change":"2024-03-04T09:30:20.202203+0000","last_active":"2024-03-04T09:30:22.207858+0000","last_peered":"2024-03-04T09:30:22.207858+0000","last_clean":"2024-03-04T09:30:22.207858+0000","last_became_active":"2024-03-04T09:30:20.201554+0000","last_became_peered":"2024-03-04T09:30:20.201554+0000","last_unstale":"2024-03-04T09:30:22.207858+0000","last_undegraded":"2024-03-04T09:30:22.207858+0000","last_fullsized":"2024-03-04T09:30:22.207858+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_clean_scrub_stamp":"2024-03-04T09:30:19.137440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:04:50.243684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.206501+0000","last_change":"2024-03-04T09:30:21.208592+0000","last_active":"2024-03-04T09:30:22.206501+0000","last_peered":"2024-03-04T09:30:22.206501+0000","last_clean":"2024-03-04T09:30:22.206501+0000","last_became_active":"2024-03-04T09:30:21.208299+0000","last_became_peered":"2024-03-04T09:30:21.208299+0000","last_unstale":"2024-03-04T09:30:22.206501+0000","last_undegraded":"2024-03-04T09:30:22.206501+0000","last_fullsized":"2024-03-04T09:30:22.206501+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:56:57.984433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":14,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.602052+0000","last_change":"2024-03-04T09:30:20.193110+0000","last_active":"2024-03-04T09:30:22.602052+0000","last_peered":"2024-03-04T09:30:22.602052+0000","last_clean":"2024-03-04T09:30:22.602052+0000","last_became_active":"2024-03-04T09:30:20.192504+0000","last_became_peered":"2024-03-04T09:30:20.192504+0000","last_unstale":"2024-03-04T09:30:22.602052+0000","last_undegraded":"2024-03-04T09:30:22.602052+0000","last_fullsized":"2024-03-04T09:30:22.602052+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_clean_scrub_stamp":"2024-03-04T09:30:19.137440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:40:35.740641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.211210+0000","last_change":"2024-03-04T09:30:21.216422+0000","last_active":"2024-03-04T09:30:22.211210+0000","last_peered":"2024-03-04T09:30:22.211210+0000","last_clean":"2024-03-04T09:30:22.211210+0000","last_became_active":"2024-03-04T09:30:21.215611+0000","last_became_peered":"2024-03-04T09:30:21.215611+0000","last_unstale":"2024-03-04T09:30:22.211210+0000","last_undegraded":"2024-03-04T09:30:22.211210+0000","last_fullsized":"2024-03-04T09:30:22.211210+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:21:52.125031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":14,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.600183+0000","last_change":"2024-03-04T09:30:20.189556+0000","last_active":"2024-03-04T09:30:22.600183+0000","last_peered":"2024-03-04T09:30:22.600183+0000","last_clean":"2024-03-04T09:30:22.600183+0000","last_became_active":"2024-03-04T09:30:20.188384+0000","last_became_peered":"2024-03-04T09:30:20.188384+0000","last_unstale":"2024-03-04T09:30:22.600183+0000","last_undegraded":"2024-03-04T09:30:22.600183+0000","last_fullsized":"2024-03-04T09:30:22.600183+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_clean_scrub_stamp":"2024-03-04T09:30:19.137440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:36:55.596046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.601407+0000","last_change":"2024-03-04T09:30:21.213566+0000","last_active":"2024-03-04T09:30:22.601407+0000","last_peered":"2024-03-04T09:30:22.601407+0000","last_clean":"2024-03-04T09:30:22.601407+0000","last_became_active":"2024-03-04T09:30:21.213248+0000","last_became_peered":"2024-03-04T09:30:21.213248+0000","last_unstale":"2024-03-04T09:30:22.601407+0000","last_undegraded":"2024-03-04T09:30:22.601407+0000","last_fullsized":"2024-03-04T09:30:22.601407+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:51:18.057309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":14,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.211017+0000","last_change":"2024-03-04T09:30:20.196985+0000","last_active":"2024-03-04T09:30:22.211017+0000","last_peered":"2024-03-04T09:30:22.211017+0000","last_clean":"2024-03-04T09:30:22.211017+0000","last_became_active":"2024-03-04T09:30:20.195708+0000","last_became_peered":"2024-03-04T09:30:20.195708+0000","last_unstale":"2024-03-04T09:30:22.211017+0000","last_undegraded":"2024-03-04T09:30:22.211017+0000","last_fullsized":"2024-03-04T09:30:22.211017+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_clean_scrub_stamp":"2024-03-04T09:30:19.137440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:30:23.060431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.211718+0000","last_change":"2024-03-04T09:30:21.218217+0000","last_active":"2024-03-04T09:30:22.211718+0000","last_peered":"2024-03-04T09:30:22.211718+0000","last_clean":"2024-03-04T09:30:22.211718+0000","last_became_active":"2024-03-04T09:30:21.217067+0000","last_became_peered":"2024-03-04T09:30:21.217067+0000","last_unstale":"2024-03-04T09:30:22.211718+0000","last_undegraded":"2024-03-04T09:30:22.211718+0000","last_fullsized":"2024-03-04T09:30:22.211718+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:20.154100+0000","last_clean_scrub_stamp":"2024-03-04T09:30:20.154100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:14:56.870440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.f","version":"93'1","reported_seq":15,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.601642+0000","last_change":"2024-03-04T09:30:20.204692+0000","last_active":"2024-03-04T09:30:22.601642+0000","last_peered":"2024-03-04T09:30:22.601642+0000","last_clean":"2024-03-04T09:30:22.601642+0000","last_became_active":"2024-03-04T09:30:20.204363+0000","last_became_peered":"2024-03-04T09:30:20.204363+0000","last_unstale":"2024-03-04T09:30:22.601642+0000","last_undegraded":"2024-03-04T09:30:22.601642+0000","last_fullsized":"2024-03-04T09:30:22.601642+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:30:19.137440+0000","last_clean_scrub_stamp":"2024-03-04T09:30:19.137440+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:42:56.761983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":310,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-04T09:30:22.601531+0000","last_change":"2024-03-04T09:25:09.316159+0000","last_active":"2024-03-04T09:30:22.601531+0000","last_peered":"2024-03-04T09:30:22.601531+0000","last_clean":"2024-03-04T09:30:22.601531+0000","last_became_active":"2024-03-04T09:25:09.313767+0000","last_became_peered":"2024-03-04T09:25:09.313767+0000","last_unstale":"2024-03-04T09:30:22.601531+0000","last_undegraded":"2024-03-04T09:30:22.601531+0000","last_fullsized":"2024-03-04T09:30:22.601531+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:08.110953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:08.110953+0000","last_clean_scrub_stamp":"2024-03-04T09:25:08.110953+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:34:21.745390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":15,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574914,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32908,"kb_used_data":1724,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338932,"statfs":{"total":96636764160,"available":96603066368,"internally_reserved":0,"allocated":1765376,"data_stored":1088420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574914,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32324,"kb_used_data":1144,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339516,"statfs":{"total":96636764160,"available":96603664384,"internally_reserved":0,"allocated":1171456,"data_stored":498592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574916,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32324,"kb_used_data":1140,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339516,"statfs":{"total":96636764160,"available":96603664384,"internally_reserved":0,"allocated":1167360,"data_stored":498010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574913,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32344,"kb_used_data":1156,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339496,"statfs":{"total":96636764160,"available":96603643904,"internally_reserved":0,"allocated":1183744,"data_stored":499676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574915,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32320,"kb_used_data":1140,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339520,"statfs":{"total":96636764160,"available":96603668480,"internally_reserved":0,"allocated":1167360,"data_stored":493274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574914,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32308,"kb_used_data":1128,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339532,"statfs":{"total":96636764160,"available":96603680768,"internally_reserved":0,"allocated":1155072,"data_stored":491674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574915,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32320,"kb_used_data":1136,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339520,"statfs":{"total":96636764160,"available":96603668480,"internally_reserved":0,"allocated":1163264,"data_stored":497940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574914,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32908,"kb_used_data":1720,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338932,"statfs":{"total":96636764160,"available":96603066368,"internally_reserved":0,"allocated":1761280,"data_stored":1088890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-04T09:30:26.211 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-04T09:30:26.211 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-04T09:30:26.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:26.486+0000 7fb819c00640 1 Processor -- start 2024-03-04T09:30:26.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:26.486+0000 7fb819c00640 1 -- start start 2024-03-04T09:30:26.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:26.487+0000 7fb819c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb81406d6f0 0x7fb81406db50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:26.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:26.487+0000 7fb819c00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fb8140b2c80 0x7fb8140b3110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:26.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:26.487+0000 7fb819c00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8140ae2d0 con 0x7fb81415a630 2024-03-04T09:30:26.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:26.487+0000 7fb819c00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb81406e090 con 0x7fb8140b2c80 2024-03-04T09:30:26.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:26.487+0000 7fb819c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8140ae5b0 con 0x7fb81406d6f0 2024-03-04T09:30:26.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:26.488+0000 7fb812a00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fb81415a630 0x7fb8140720a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:55336/0 (socket says 172.21.15.6:55336) 2024-03-04T09:30:26.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:26.488+0000 7fb812a00640 1 -- 172.21.15.6:0/2008522818 learned_addr learned my addr 172.21.15.6:0/2008522818 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:30:26.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:26.488+0000 7fb80be00640 1 --2- 172.21.15.6:0/2008522818 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb81406d6f0 0x7fb81406db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:26.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:26.488+0000 7fb813400640 1 --2- 172.21.15.6:0/2008522818 >> v2:172.21.15.146:3300/0 conn(0x7fb8140b2c80 0x7fb8140b3110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:26.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:26.488+0000 7fb812000640 1 -- 172.21.15.6:0/2008522818 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 178231122 0 0) 0x7fb8140ae2d0 con 0x7fb81415a630 2024-03-04T09:30:26.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:26.488+0000 7fb812000640 1 -- 172.21.15.6:0/2008522818 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7f8003440 con 0x7fb81415a630 2024-03-04T09:30:26.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:26.489+0000 7fb812000640 1 -- 172.21.15.6:0/2008522818 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4198092275 0 0) 0x7fb7f8003440 con 0x7fb81415a630 2024-03-04T09:30:26.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:26.489+0000 7fb812000640 1 -- 172.21.15.6:0/2008522818 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb81406d6f0 msgr2=0x7fb81406db50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:26.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:26.489+0000 7fb812000640 1 --2- 172.21.15.6:0/2008522818 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb81406d6f0 0x7fb81406db50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:26.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:26.489+0000 7fb812000640 1 -- 172.21.15.6:0/2008522818 >> v2:172.21.15.146:3300/0 conn(0x7fb8140b2c80 msgr2=0x7fb8140b3110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:26.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:26.489+0000 7fb812000640 1 --2- 172.21.15.6:0/2008522818 >> v2:172.21.15.146:3300/0 conn(0x7fb8140b2c80 0x7fb8140b3110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:26.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:26.489+0000 7fb812000640 1 -- 172.21.15.6:0/2008522818 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8140ae890 con 0x7fb81415a630 2024-03-04T09:30:26.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:26.489+0000 7fb812000640 1 -- 172.21.15.6:0/2008522818 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7fb804001670 con 0x7fb81415a630 2024-03-04T09:30:26.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:26.489+0000 7fb812000640 1 -- 172.21.15.6:0/2008522818 >> v1:172.21.15.6:6789/0 conn(0x7fb81415a630 legacy=0x7fb8140720a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:26.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:26.489+0000 7fb812000640 1 --2- 172.21.15.6:0/2008522818 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb7f8003aa0 0x7fb7f8003ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:26.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:26.489+0000 7fb812000640 1 --2- 172.21.15.6:0/2008522818 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb7f8008850 0x7fb7f800ac80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:26.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:26.489+0000 7fb812000640 1 --2- 172.21.15.6:0/2008522818 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb81406d6f0 0x7fb7f800d410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:26.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:26.489+0000 7fb812000640 1 -- 172.21.15.6:0/2008522818 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8140ae5b0 con 0x7fb7f8003aa0 2024-03-04T09:30:26.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:26.489+0000 7fb812000640 1 -- 172.21.15.6:0/2008522818 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7f800db50 con 0x7fb7f8008850 2024-03-04T09:30:26.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:26.489+0000 7fb812000640 1 -- 172.21.15.6:0/2008522818 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7f800dd50 con 0x7fb81406d6f0 2024-03-04T09:30:26.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:26.489+0000 7fb812a00640 1 --2- 172.21.15.6:0/2008522818 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb7f8003aa0 0x7fb7f8003ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:26.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:26.489+0000 7fb812a00640 1 -- 172.21.15.6:0/2008522818 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb81406d6f0 msgr2=0x7fb7f800d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:26.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:26.489+0000 7fb812a00640 1 --2- 172.21.15.6:0/2008522818 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb81406d6f0 0x7fb7f800d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:26.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:26.489+0000 7fb812a00640 1 -- 172.21.15.6:0/2008522818 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb7f8008850 msgr2=0x7fb7f800ac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:26.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:26.489+0000 7fb812a00640 1 --2- 172.21.15.6:0/2008522818 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb7f8008850 0x7fb7f800ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:26.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:26.490+0000 7fb812a00640 1 -- 172.21.15.6:0/2008522818 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7f8003440 con 0x7fb7f8003aa0 2024-03-04T09:30:26.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:26.491+0000 7fb812a00640 1 --2- 172.21.15.6:0/2008522818 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb7f8003aa0 0x7fb7f8003ec0 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7fb80400c290 tx=0x7fb7f8003730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b174217038b0bd96 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:26.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:26.491+0000 7fb812000640 1 -- 172.21.15.6:0/2008522818 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb804007d50 con 0x7fb7f8003aa0 2024-03-04T09:30:26.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:26.491+0000 7fb812000640 1 -- 172.21.15.6:0/2008522818 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb804030070 con 0x7fb7f8003aa0 2024-03-04T09:30:26.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:26.491+0000 7fb812000640 1 -- 172.21.15.6:0/2008522818 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb80402cd80 con 0x7fb7f8003aa0 2024-03-04T09:30:26.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:26.491+0000 7fb819c00640 1 -- 172.21.15.6:0/2008522818 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb7f8003aa0 msgr2=0x7fb7f8003ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:26.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:26.491+0000 7fb819c00640 1 --2- 172.21.15.6:0/2008522818 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb7f8003aa0 0x7fb7f8003ec0 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7fb80400c290 tx=0x7fb7f8003730 comp rx=0 tx=0).stop 2024-03-04T09:30:26.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:26.492+0000 7fb813400640 1 -- 172.21.15.6:0/2008522818 reap_dead start 2024-03-04T09:30:26.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:26.492+0000 7fb819c00640 1 -- 172.21.15.6:0/2008522818 shutdown_connections 2024-03-04T09:30:26.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:26.492+0000 7fb819c00640 1 -- 172.21.15.6:0/2008522818 >> 172.21.15.6:0/2008522818 conn(0x7fb81406b3d0 msgr2=0x7fb81406b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:26.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:26.492+0000 7fb819c00640 1 -- 172.21.15.6:0/2008522818 shutdown_connections 2024-03-04T09:30:26.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:26.492+0000 7fb819c00640 1 -- 172.21.15.6:0/2008522818 wait complete. 2024-03-04T09:30:26.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:26.494+0000 7fb819c00640 1 Processor -- start 2024-03-04T09:30:26.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:26.494+0000 7fb819c00640 1 -- start start 2024-03-04T09:30:26.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:26.495+0000 7fb819c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb7f8003aa0 0x7fb8141f3540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:26.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:26.495+0000 7fb819c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb81415aac0 0x7fb8141f4560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:26.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:26.495+0000 7fb819c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb81415ccd0 0x7fb81415d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:26.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:26.495+0000 7fb819c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8140727b0 con 0x7fb81415aac0 2024-03-04T09:30:26.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:26.495+0000 7fb819c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8140aee50 con 0x7fb7f8003aa0 2024-03-04T09:30:26.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:26.495+0000 7fb819c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb814072930 con 0x7fb81415ccd0 2024-03-04T09:30:26.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:26.495+0000 7fb80be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb81415aac0 0x7fb8141f4560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:26.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:26.495+0000 7fb80be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb81415aac0 0x7fb8141f4560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38444/0 (socket says 172.21.15.6:38444) 2024-03-04T09:30:26.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:26.495+0000 7fb80be00640 1 -- 172.21.15.6:0/3215500612 learned_addr learned my addr 172.21.15.6:0/3215500612 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:30:26.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:26.495+0000 7fb812a00640 1 --2- 172.21.15.6:0/3215500612 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb7f8003aa0 0x7fb8141f3540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:26.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:26.496+0000 7fb813400640 1 --2- 172.21.15.6:0/3215500612 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb81415ccd0 0x7fb81415d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:26.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:26.496+0000 7fb80be00640 1 -- 172.21.15.6:0/3215500612 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb81415ccd0 msgr2=0x7fb81415d180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:26.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:26.496+0000 7fb80be00640 1 --2- 172.21.15.6:0/3215500612 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb81415ccd0 0x7fb81415d180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:26.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:26.496+0000 7fb80be00640 1 -- 172.21.15.6:0/3215500612 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb7f8003aa0 msgr2=0x7fb8141f3540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:26.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:26.496+0000 7fb80be00640 1 --2- 172.21.15.6:0/3215500612 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb7f8003aa0 0x7fb8141f3540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:26.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:26.496+0000 7fb80be00640 1 -- 172.21.15.6:0/3215500612 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8141f4aa0 con 0x7fb81415aac0 2024-03-04T09:30:26.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:26.496+0000 7fb813400640 1 --2- 172.21.15.6:0/3215500612 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb81415ccd0 0x7fb81415d180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:30:26.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:26.496+0000 7fb812a00640 1 --2- 172.21.15.6:0/3215500612 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb7f8003aa0 0x7fb8141f3540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:30:26.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:26.496+0000 7fb80be00640 1 --2- 172.21.15.6:0/3215500612 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb81415aac0 0x7fb8141f4560 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7fb7fc00e7c0 tx=0x7fb7fc00ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:26.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:26.496+0000 7fb80aa00640 1 -- 172.21.15.6:0/3215500612 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7fc017070 con 0x7fb81415aac0 2024-03-04T09:30:26.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:26.496+0000 7fb80aa00640 1 -- 172.21.15.6:0/3215500612 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7fc00c8c0 con 0x7fb81415aac0 2024-03-04T09:30:26.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:26.496+0000 7fb80aa00640 1 -- 172.21.15.6:0/3215500612 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7fc00ccb0 con 0x7fb81415aac0 2024-03-04T09:30:26.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:26.496+0000 7fb819c00640 1 -- 172.21.15.6:0/3215500612 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb81415bfe0 con 0x7fb81415aac0 2024-03-04T09:30:26.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:26.497+0000 7fb819c00640 1 -- 172.21.15.6:0/3215500612 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb81415b220 con 0x7fb81415aac0 2024-03-04T09:30:26.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:26.498+0000 7fb819c00640 1 -- 172.21.15.6:0/3215500612 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb814068010 con 0x7fb81415aac0 2024-03-04T09:30:26.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:26.499+0000 7fb80aa00640 1 -- 172.21.15.6:0/3215500612 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb7fc01a070 con 0x7fb81415aac0 2024-03-04T09:30:26.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:26.501+0000 7fb80aa00640 1 --2- 172.21.15.6:0/3215500612 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb7f006b4d0 0x7fb7f006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:26.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:26.501+0000 7fb80aa00640 1 -- 172.21.15.6:0/3215500612 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb7fc08a190 con 0x7fb81415aac0 2024-03-04T09:30:26.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:26.501+0000 7fb812a00640 1 --2- 172.21.15.6:0/3215500612 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb7f006b4d0 0x7fb7f006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:26.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:26.503+0000 7fb812a00640 1 --2- 172.21.15.6:0/3215500612 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb7f006b4d0 0x7fb7f006d990 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fb804034000 tx=0x7fb804007000 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:26.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:26.504+0000 7fb80aa00640 1 -- 172.21.15.6:0/3215500612 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb7fc058f70 con 0x7fb81415aac0 2024-03-04T09:30:26.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:26.991+0000 7fb819c00640 1 -- 172.21.15.6:0/3215500612 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fb814158770 con 0x7fb81415aac0 2024-03-04T09:30:26.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:26.995+0000 7fb80aa00640 1 -- 172.21.15.6:0/3215500612 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v15) v1 ==== 190+0+147 (secure 0 0 0) 0x7fb7fc058870 con 0x7fb81415aac0 2024-03-04T09:30:26.992 INFO:teuthology.orchestra.run.smithi006.stdout:[client.0] 2024-03-04T09:30:26.992 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQA9k+Vl4L+rJxAACaDbQ4/Bybp5hAZAnJzlVw== 2024-03-04T09:30:26.992 INFO:teuthology.orchestra.run.smithi006.stdout: caps mds = "allow" 2024-03-04T09:30:26.992 INFO:teuthology.orchestra.run.smithi006.stdout: caps mon = "allow r" 2024-03-04T09:30:26.992 INFO:teuthology.orchestra.run.smithi006.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-04T09:30:26.992 INFO:teuthology.orchestra.run.smithi006.stderr:updated caps for client.0 2024-03-04T09:30:26.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:26.997+0000 7fb819c00640 1 -- 172.21.15.6:0/3215500612 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb7f006b4d0 msgr2=0x7fb7f006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:26.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:26.997+0000 7fb819c00640 1 --2- 172.21.15.6:0/3215500612 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb7f006b4d0 0x7fb7f006d990 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fb804034000 tx=0x7fb804007000 comp rx=0 tx=0).stop 2024-03-04T09:30:26.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:26.997+0000 7fb819c00640 1 -- 172.21.15.6:0/3215500612 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb81415aac0 msgr2=0x7fb8141f4560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:26.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:26.997+0000 7fb819c00640 1 --2- 172.21.15.6:0/3215500612 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb81415aac0 0x7fb8141f4560 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7fb7fc00e7c0 tx=0x7fb7fc00ec80 comp rx=0 tx=0).stop 2024-03-04T09:30:26.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:26.997+0000 7fb819c00640 1 -- 172.21.15.6:0/3215500612 shutdown_connections 2024-03-04T09:30:26.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:26.997+0000 7fb819c00640 1 --2- 172.21.15.6:0/3215500612 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb7f006b4d0 0x7fb7f006d990 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:26.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:26.998+0000 7fb819c00640 1 --2- 172.21.15.6:0/3215500612 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb81415ccd0 0x7fb81415d180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:26.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:26.998+0000 7fb819c00640 1 --2- 172.21.15.6:0/3215500612 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb81415aac0 0x7fb8141f4560 unknown :-1 s=CLOSED pgs=1088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:26.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:26.998+0000 7fb819c00640 1 --2- 172.21.15.6:0/3215500612 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb7f8003aa0 0x7fb8141f3540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:26.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:26.998+0000 7fb819c00640 1 -- 172.21.15.6:0/3215500612 >> 172.21.15.6:0/3215500612 conn(0x7fb81406b3d0 msgr2=0x7fb814156f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:26.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:26.998+0000 7fb819c00640 1 -- 172.21.15.6:0/3215500612 shutdown_connections 2024-03-04T09:30:26.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:26.998+0000 7fb819c00640 1 -- 172.21.15.6:0/3215500612 wait complete. 2024-03-04T09:30:27.012 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:30:27.123 INFO:tasks.ceph.mds.a.smithi006.stderr:2024-03-04T09:30:27.126+0000 7f805d200640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-04T09:30:27.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:27.222+0000 7ff6f6c00640 1 Processor -- start 2024-03-04T09:30:27.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:27.222+0000 7ff6f6c00640 1 -- start start 2024-03-04T09:30:27.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:27.224+0000 7ff6f6c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff6f006ed90 0x7ff6f006bf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:27.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:27.224+0000 7ff6f6c00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ff6f006c4c0 0x7ff6f015c740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:27.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:27.224+0000 7ff6f6c00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6f00afb40 con 0x7ff6f015c360 2024-03-04T09:30:27.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:27.224+0000 7ff6f6c00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6f006c950 con 0x7ff6f006ed90 2024-03-04T09:30:27.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:27.224+0000 7ff6f6c00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6f006d310 con 0x7ff6f006c4c0 2024-03-04T09:30:27.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:27.224+0000 7ff6f4e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff6f006ed90 0x7ff6f006bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:27.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:27.224+0000 7ff6f4e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff6f006ed90 0x7ff6f006bf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38450/0 (socket says 172.21.15.6:38450) 2024-03-04T09:30:27.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:27.224+0000 7ff6f4e00640 1 -- 172.21.15.6:0/3611038515 learned_addr learned my addr 172.21.15.6:0/3611038515 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:30:27.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:27.224+0000 7ff6f6200640 1 --2- 172.21.15.6:0/3611038515 >> v2:172.21.15.146:3300/0 conn(0x7ff6f006c4c0 0x7ff6f015c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:27.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:27.224+0000 7ff6f4e00640 1 -- 172.21.15.6:0/3611038515 >> v2:172.21.15.146:3300/0 conn(0x7ff6f006c4c0 msgr2=0x7ff6f015c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:27.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:27.224+0000 7ff6f4e00640 1 --2- 172.21.15.6:0/3611038515 >> v2:172.21.15.146:3300/0 conn(0x7ff6f006c4c0 0x7ff6f015c740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:27.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:27.224+0000 7ff6f4e00640 1 -- 172.21.15.6:0/3611038515 >> v1:172.21.15.146:6789/0 conn(0x7ff6f015c360 legacy=0x7ff6f00ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:27.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:27.225+0000 7ff6f4e00640 1 -- 172.21.15.6:0/3611038515 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6f00afe20 con 0x7ff6f006ed90 2024-03-04T09:30:27.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:27.225+0000 7ff6f6200640 1 --2- 172.21.15.6:0/3611038515 >> v2:172.21.15.146:3300/0 conn(0x7ff6f006c4c0 0x7ff6f015c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:30:27.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:27.226+0000 7ff6f4e00640 1 --2- 172.21.15.6:0/3611038515 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff6f006ed90 0x7ff6f006bf80 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7ff6f006cee0 tx=0x7ff6dc02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2408fa2f029f06d9 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:27.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:27.229+0000 7ff6efe00640 1 -- 172.21.15.6:0/3611038515 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6dc02e070 con 0x7ff6f006ed90 2024-03-04T09:30:27.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:27.229+0000 7ff6efe00640 1 -- 172.21.15.6:0/3611038515 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6dc037070 con 0x7ff6f006ed90 2024-03-04T09:30:27.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:27.229+0000 7ff6efe00640 1 -- 172.21.15.6:0/3611038515 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6dc031a00 con 0x7ff6f006ed90 2024-03-04T09:30:27.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:27.229+0000 7ff6f6c00640 1 -- 172.21.15.6:0/3611038515 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff6f006ed90 msgr2=0x7ff6f006bf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:27.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:27.229+0000 7ff6f6c00640 1 --2- 172.21.15.6:0/3611038515 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff6f006ed90 0x7ff6f006bf80 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7ff6f006cee0 tx=0x7ff6dc02b880 comp rx=0 tx=0).stop 2024-03-04T09:30:27.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:27.229+0000 7ff6f6c00640 1 -- 172.21.15.6:0/3611038515 shutdown_connections 2024-03-04T09:30:27.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:27.229+0000 7ff6f6c00640 1 --2- 172.21.15.6:0/3611038515 >> v2:172.21.15.146:3300/0 conn(0x7ff6f006c4c0 0x7ff6f015c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:27.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:27.229+0000 7ff6f6c00640 1 --2- 172.21.15.6:0/3611038515 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff6f006ed90 0x7ff6f006bf80 unknown :-1 s=CLOSED pgs=1089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:27.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:27.229+0000 7ff6f6c00640 1 -- 172.21.15.6:0/3611038515 >> 172.21.15.6:0/3611038515 conn(0x7ff6f00ccf70 msgr2=0x7ff6f00cd350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:27.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:27.230+0000 7ff6f6c00640 1 -- 172.21.15.6:0/3611038515 shutdown_connections 2024-03-04T09:30:27.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:27.230+0000 7ff6f6c00640 1 -- 172.21.15.6:0/3611038515 wait complete. 2024-03-04T09:30:27.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:27.231+0000 7ff6f6c00640 1 Processor -- start 2024-03-04T09:30:27.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:27.231+0000 7ff6f6c00640 1 -- start start 2024-03-04T09:30:27.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:27.232+0000 7ff6f6c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff6f006c4c0 0x7ff6f0156ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:27.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:27.232+0000 7ff6f6c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff6f006ed90 0x7ff6f01586c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:27.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:27.232+0000 7ff6f6c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff6f015c360 0x7ff6f0158c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:27.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:27.232+0000 7ff6f6c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f006ba60 con 0x7ff6f006ed90 2024-03-04T09:30:27.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:27.232+0000 7ff6f6c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f006d840 con 0x7ff6f006c4c0 2024-03-04T09:30:27.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:27.232+0000 7ff6f6c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f00cea80 con 0x7ff6f015c360 2024-03-04T09:30:27.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:27.233+0000 7ff6f4e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff6f006ed90 0x7ff6f01586c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:27.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:27.233+0000 7ff6f4e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff6f006ed90 0x7ff6f01586c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38454/0 (socket says 172.21.15.6:38454) 2024-03-04T09:30:27.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:27.233+0000 7ff6f4e00640 1 -- 172.21.15.6:0/487887402 learned_addr learned my addr 172.21.15.6:0/487887402 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:30:27.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:27.233+0000 7ff6f5800640 1 --2- 172.21.15.6:0/487887402 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff6f006c4c0 0x7ff6f0156ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:27.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:27.233+0000 7ff6f6200640 1 --2- 172.21.15.6:0/487887402 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff6f015c360 0x7ff6f0158c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:27.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:27.233+0000 7ff6f4e00640 1 -- 172.21.15.6:0/487887402 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff6f015c360 msgr2=0x7ff6f0158c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:27.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:27.233+0000 7ff6f4e00640 1 --2- 172.21.15.6:0/487887402 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff6f015c360 0x7ff6f0158c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:27.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:27.233+0000 7ff6f4e00640 1 -- 172.21.15.6:0/487887402 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff6f006c4c0 msgr2=0x7ff6f0156ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:27.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:27.233+0000 7ff6f4e00640 1 --2- 172.21.15.6:0/487887402 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff6f006c4c0 0x7ff6f0156ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:27.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:27.233+0000 7ff6f4e00640 1 -- 172.21.15.6:0/487887402 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6f00ac540 con 0x7ff6f006ed90 2024-03-04T09:30:27.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:27.234+0000 7ff6f4e00640 1 --2- 172.21.15.6:0/487887402 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff6f006ed90 0x7ff6f01586c0 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7ff6dc000c00 tx=0x7ff6dc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:27.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:27.234+0000 7ff6ed600640 1 -- 172.21.15.6:0/487887402 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6dc032880 con 0x7ff6f006ed90 2024-03-04T09:30:27.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:27.234+0000 7ff6ed600640 1 -- 172.21.15.6:0/487887402 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6dc032a20 con 0x7ff6f006ed90 2024-03-04T09:30:27.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:27.234+0000 7ff6f6c00640 1 -- 172.21.15.6:0/487887402 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6f00ac820 con 0x7ff6f006ed90 2024-03-04T09:30:27.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:27.234+0000 7ff6ed600640 1 -- 172.21.15.6:0/487887402 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6dc032e10 con 0x7ff6f006ed90 2024-03-04T09:30:27.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:27.234+0000 7ff6f6c00640 1 -- 172.21.15.6:0/487887402 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6f00acde0 con 0x7ff6f006ed90 2024-03-04T09:30:27.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:27.236+0000 7ff6f6c00640 1 -- 172.21.15.6:0/487887402 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6f004f330 con 0x7ff6f006ed90 2024-03-04T09:30:27.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:27.236+0000 7ff6ed600640 1 -- 172.21.15.6:0/487887402 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff6dc0453f0 con 0x7ff6f006ed90 2024-03-04T09:30:27.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:27.237+0000 7ff6ed600640 1 --2- 172.21.15.6:0/487887402 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff6d006b500 0x7ff6d006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:27.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:27.237+0000 7ff6ed600640 1 -- 172.21.15.6:0/487887402 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff6dc0ac140 con 0x7ff6f006ed90 2024-03-04T09:30:27.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:27.238+0000 7ff6f5800640 1 --2- 172.21.15.6:0/487887402 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff6d006b500 0x7ff6d006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:27.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:27.239+0000 7ff6f5800640 1 --2- 172.21.15.6:0/487887402 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff6d006b500 0x7ff6d006d9c0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7ff6e4006fd0 tx=0x7ff6e4009060 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:27.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:27.242+0000 7ff6ed600640 1 -- 172.21.15.6:0/487887402 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff6dc040070 con 0x7ff6f006ed90 2024-03-04T09:30:27.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:27.482+0000 7ff6f6c00640 1 -- 172.21.15.6:0/487887402 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff6f006caf0 con 0x7ff6f006ed90 2024-03-04T09:30:27.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:27.483+0000 7ff6ed600640 1 -- 172.21.15.6:0/487887402 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+4976 (secure 0 0 0) 0x7ff6dc07a9f0 con 0x7ff6f006ed90 2024-03-04T09:30:27.480 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:30:27.480 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":60,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5555,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/847815316","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":847815316},{"type":"v1","addr":"172.21.15.146:6835","nonce":847815316}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55},{"gid":6046,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/1371176866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":1371176866},{"type":"v1","addr":"172.21.15.146:6837","nonce":1371176866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55},{"gid":7092,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/1696083602","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":1696083602},{"type":"v1","addr":"172.21.15.6:6834","nonce":1696083602}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":54}],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:30:20.660103+0000","modified":"2024-03-04T09:30:23.726533+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7095},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7095":{"gid":7095,"name":"a","rank":0,"incarnation":56,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6835/2836916196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2836916196},{"type":"v1","addr":"172.21.15.6:6835","nonce":2836916196}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7095,"qdb_cluster":[7095]},"id":7}]} 2024-03-04T09:30:27.481 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 60 2024-03-04T09:30:27.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:27.485+0000 7ff6f6c00640 1 -- 172.21.15.6:0/487887402 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff6d006b500 msgr2=0x7ff6d006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:27.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:27.485+0000 7ff6f6c00640 1 --2- 172.21.15.6:0/487887402 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff6d006b500 0x7ff6d006d9c0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7ff6e4006fd0 tx=0x7ff6e4009060 comp rx=0 tx=0).stop 2024-03-04T09:30:27.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:27.486+0000 7ff6f6c00640 1 -- 172.21.15.6:0/487887402 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff6f006ed90 msgr2=0x7ff6f01586c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:27.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:27.486+0000 7ff6f6c00640 1 --2- 172.21.15.6:0/487887402 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff6f006ed90 0x7ff6f01586c0 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7ff6dc000c00 tx=0x7ff6dc0046c0 comp rx=0 tx=0).stop 2024-03-04T09:30:27.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:27.486+0000 7ff6f6c00640 1 -- 172.21.15.6:0/487887402 shutdown_connections 2024-03-04T09:30:27.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:27.486+0000 7ff6f6c00640 1 --2- 172.21.15.6:0/487887402 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff6d006b500 0x7ff6d006d9c0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:27.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:27.486+0000 7ff6f6c00640 1 --2- 172.21.15.6:0/487887402 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff6f015c360 0x7ff6f0158c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:27.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:27.486+0000 7ff6f6c00640 1 --2- 172.21.15.6:0/487887402 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff6f006ed90 0x7ff6f01586c0 unknown :-1 s=CLOSED pgs=1090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:27.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:27.486+0000 7ff6f6c00640 1 --2- 172.21.15.6:0/487887402 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff6f006c4c0 0x7ff6f0156ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:27.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:27.486+0000 7ff6f6c00640 1 -- 172.21.15.6:0/487887402 >> 172.21.15.6:0/487887402 conn(0x7ff6f00ccf70 msgr2=0x7ff6f0074050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:27.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:27.486+0000 7ff6f6c00640 1 -- 172.21.15.6:0/487887402 shutdown_connections 2024-03-04T09:30:27.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:27.486+0000 7ff6f6c00640 1 -- 172.21.15.6:0/487887402 wait complete. 2024-03-04T09:30:27.904 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 60, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:30:20.660103+0000', 'modified': '2024-03-04T09:30:23.726533+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7095}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7095': {'gid': 7095, 'name': 'a', 'rank': 0, 'incarnation': 56, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6835/2836916196', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2836916196}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 2836916196}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7095, 'qdb_cluster': [7095]} 2024-03-04T09:30:27.905 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-04T09:30:27.905 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-04T09:30:28.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.166+0000 7fa5c2c00640 1 Processor -- start 2024-03-04T09:30:28.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.167+0000 7fa5c2c00640 1 -- start start 2024-03-04T09:30:28.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.170+0000 7fa5c2c00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fa5bc0b3c90 0x7fa5bc0cdc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:28.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.170+0000 7fa5c2c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa5bc0cf5c0 0x7fa5bc0b4070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:28.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.170+0000 7fa5c2c00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5bc0b45b0 con 0x7fa5bc06cb10 2024-03-04T09:30:28.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.170+0000 7fa5c2c00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5bc0ce160 con 0x7fa5bc0cf5c0 2024-03-04T09:30:28.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.170+0000 7fa5c2c00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5bc0b4860 con 0x7fa5bc0b3c90 2024-03-04T09:30:28.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.170+0000 7fa5bb400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa5bc0cf5c0 0x7fa5bc0b4070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:28.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.170+0000 7fa5bb400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa5bc0cf5c0 0x7fa5bc0b4070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38470/0 (socket says 172.21.15.6:38470) 2024-03-04T09:30:28.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.170+0000 7fa5c0c00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fa5bc06cb10 0x7fa5bc06cf80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:55346/0 (socket says 172.21.15.6:55346) 2024-03-04T09:30:28.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.170+0000 7fa5bb400640 1 -- 172.21.15.6:0/1243436465 learned_addr learned my addr 172.21.15.6:0/1243436465 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:30:28.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.171+0000 7fa5bbe00640 1 --2- 172.21.15.6:0/1243436465 >> v2:172.21.15.146:3300/0 conn(0x7fa5bc0b3c90 0x7fa5bc0cdc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:28.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.171+0000 7fa5bb400640 1 -- 172.21.15.6:0/1243436465 >> v2:172.21.15.146:3300/0 conn(0x7fa5bc0b3c90 msgr2=0x7fa5bc0cdc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:28.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.171+0000 7fa5bb400640 1 --2- 172.21.15.6:0/1243436465 >> v2:172.21.15.146:3300/0 conn(0x7fa5bc0b3c90 0x7fa5bc0cdc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:28.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.171+0000 7fa5bb400640 1 -- 172.21.15.6:0/1243436465 >> v1:172.21.15.6:6789/0 conn(0x7fa5bc06cb10 legacy=0x7fa5bc06cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:28.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.171+0000 7fa5bb400640 1 -- 172.21.15.6:0/1243436465 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5bc0b4b40 con 0x7fa5bc0cf5c0 2024-03-04T09:30:28.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.172+0000 7fa5bb400640 1 --2- 172.21.15.6:0/1243436465 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa5bc0cf5c0 0x7fa5bc0b4070 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7fa5bc070800 tx=0x7fa5a802ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8886644dbc2efe7c server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:28.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.172+0000 7fa5baa00640 1 -- 172.21.15.6:0/1243436465 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5a802f850 con 0x7fa5bc0cf5c0 2024-03-04T09:30:28.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.172+0000 7fa5baa00640 1 -- 172.21.15.6:0/1243436465 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5a802f9f0 con 0x7fa5bc0cf5c0 2024-03-04T09:30:28.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.172+0000 7fa5baa00640 1 -- 172.21.15.6:0/1243436465 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5a802fdc0 con 0x7fa5bc0cf5c0 2024-03-04T09:30:28.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.173+0000 7fa5c2c00640 1 -- 172.21.15.6:0/1243436465 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa5bc0cf5c0 msgr2=0x7fa5bc0b4070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:28.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.173+0000 7fa5c2c00640 1 --2- 172.21.15.6:0/1243436465 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa5bc0cf5c0 0x7fa5bc0b4070 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7fa5bc070800 tx=0x7fa5a802ba30 comp rx=0 tx=0).stop 2024-03-04T09:30:28.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.173+0000 7fa5c2c00640 1 -- 172.21.15.6:0/1243436465 shutdown_connections 2024-03-04T09:30:28.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.173+0000 7fa5c2c00640 1 --2- 172.21.15.6:0/1243436465 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa5bc0cf5c0 0x7fa5bc0b4070 unknown :-1 s=CLOSED pgs=1092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:28.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.173+0000 7fa5c2c00640 1 --2- 172.21.15.6:0/1243436465 >> v2:172.21.15.146:3300/0 conn(0x7fa5bc0b3c90 0x7fa5bc0cdc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:28.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.173+0000 7fa5c2c00640 1 -- 172.21.15.6:0/1243436465 >> 172.21.15.6:0/1243436465 conn(0x7fa5bc06afb0 msgr2=0x7fa5bc06b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:28.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.173+0000 7fa5c2c00640 1 -- 172.21.15.6:0/1243436465 shutdown_connections 2024-03-04T09:30:28.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.173+0000 7fa5c2c00640 1 -- 172.21.15.6:0/1243436465 wait complete. 2024-03-04T09:30:28.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.176+0000 7fa5c2c00640 1 Processor -- start 2024-03-04T09:30:28.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.176+0000 7fa5c2c00640 1 -- start start 2024-03-04T09:30:28.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.178+0000 7fa5c2c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa5bc06cb10 0x7fa5bc14bed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:28.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.178+0000 7fa5c2c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa5bc0b3c90 0x7fa5bc09cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:28.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.178+0000 7fa5c2c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa5bc0cf5c0 0x7fa5bc151a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:28.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.178+0000 7fa5c2c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5bc15b340 con 0x7fa5bc0cf5c0 2024-03-04T09:30:28.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.178+0000 7fa5c2c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5bc0b5100 con 0x7fa5bc06cb10 2024-03-04T09:30:28.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.178+0000 7fa5c2c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5bc06d690 con 0x7fa5bc0b3c90 2024-03-04T09:30:28.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.178+0000 7fa5c0c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa5bc0cf5c0 0x7fa5bc151a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:28.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.178+0000 7fa5c0c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa5bc0cf5c0 0x7fa5bc151a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38480/0 (socket says 172.21.15.6:38480) 2024-03-04T09:30:28.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.178+0000 7fa5c0c00640 1 -- 172.21.15.6:0/3218137812 learned_addr learned my addr 172.21.15.6:0/3218137812 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:30:28.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.179+0000 7fa5bbe00640 1 --2- 172.21.15.6:0/3218137812 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa5bc06cb10 0x7fa5bc14bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:28.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.179+0000 7fa5bb400640 1 --2- 172.21.15.6:0/3218137812 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa5bc0b3c90 0x7fa5bc09cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:28.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.179+0000 7fa5c0c00640 1 -- 172.21.15.6:0/3218137812 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa5bc0b3c90 msgr2=0x7fa5bc09cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:28.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.179+0000 7fa5c0c00640 1 --2- 172.21.15.6:0/3218137812 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa5bc0b3c90 0x7fa5bc09cfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:28.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.179+0000 7fa5c0c00640 1 -- 172.21.15.6:0/3218137812 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa5bc06cb10 msgr2=0x7fa5bc14bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:28.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.179+0000 7fa5c0c00640 1 --2- 172.21.15.6:0/3218137812 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa5bc06cb10 0x7fa5bc14bed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:28.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.179+0000 7fa5c0c00640 1 -- 172.21.15.6:0/3218137812 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5bc151f70 con 0x7fa5bc0cf5c0 2024-03-04T09:30:28.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.179+0000 7fa5c0c00640 1 --2- 172.21.15.6:0/3218137812 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa5bc0cf5c0 0x7fa5bc151a30 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7fa5b400c780 tx=0x7fa5b400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:28.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.179+0000 7fa5b3e00640 1 -- 172.21.15.6:0/3218137812 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5b4017070 con 0x7fa5bc0cf5c0 2024-03-04T09:30:28.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.180+0000 7fa5b3e00640 1 -- 172.21.15.6:0/3218137812 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5b400a640 con 0x7fa5bc0cf5c0 2024-03-04T09:30:28.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.180+0000 7fa5c2c00640 1 -- 172.21.15.6:0/3218137812 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5bc09d6a0 con 0x7fa5bc0cf5c0 2024-03-04T09:30:28.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.180+0000 7fa5b3e00640 1 -- 172.21.15.6:0/3218137812 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5b400aa30 con 0x7fa5bc0cf5c0 2024-03-04T09:30:28.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.180+0000 7fa5c2c00640 1 -- 172.21.15.6:0/3218137812 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5bc071360 con 0x7fa5bc0cf5c0 2024-03-04T09:30:28.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.182+0000 7fa5b3e00640 1 -- 172.21.15.6:0/3218137812 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa5b401a070 con 0x7fa5bc0cf5c0 2024-03-04T09:30:28.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.183+0000 7fa5b3e00640 1 --2- 172.21.15.6:0/3218137812 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa59c06b4d0 0x7fa59c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:28.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.183+0000 7fa5b3e00640 1 -- 172.21.15.6:0/3218137812 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa5b4089730 con 0x7fa5bc0cf5c0 2024-03-04T09:30:28.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.184+0000 7fa5bbe00640 1 --2- 172.21.15.6:0/3218137812 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa59c06b4d0 0x7fa59c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:28.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.185+0000 7fa5bbe00640 1 --2- 172.21.15.6:0/3218137812 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa59c06b4d0 0x7fa59c06d990 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fa5ac00a970 tx=0x7fa5ac008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:28.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.186+0000 7fa5c5986740 1 Processor -- start 2024-03-04T09:30:28.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.186+0000 7fa5c5986740 1 -- start start 2024-03-04T09:30:28.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.187+0000 7fa5c5986740 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5638d190f7b0 0x5638d18e2750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:28.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.187+0000 7fa5c5986740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5638d190fb90 0x5638d1912540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:28.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.187+0000 7fa5c5986740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5638d18e2c90 0x5638d1912a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:28.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.187+0000 7fa5c5986740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x5638d184c600 con 0x5638d190fb90 2024-03-04T09:30:28.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.187+0000 7fa5c5986740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x5638d17f5930 con 0x5638d18e2c90 2024-03-04T09:30:28.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.187+0000 7fa5c5986740 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x5638d18486a0 con 0x5638d190f7b0 2024-03-04T09:30:28.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.188+0000 7fa5bb400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5638d190fb90 0x5638d1912540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:28.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.188+0000 7fa5bb400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5638d190fb90 0x5638d1912540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38494/0 (socket says 172.21.15.6:38494) 2024-03-04T09:30:28.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.188+0000 7fa5bb400640 1 -- 172.21.15.6:0/176109113 learned_addr learned my addr 172.21.15.6:0/176109113 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:30:28.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.188+0000 7fa5bbe00640 1 --2- 172.21.15.6:0/176109113 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5638d190f7b0 0x5638d18e2750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:28.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.188+0000 7fa5c0c00640 1 --2- 172.21.15.6:0/176109113 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5638d18e2c90 0x5638d1912a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:28.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.188+0000 7fa5bb400640 1 -- 172.21.15.6:0/176109113 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5638d190f7b0 msgr2=0x5638d18e2750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:28.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.188+0000 7fa5bb400640 1 --2- 172.21.15.6:0/176109113 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5638d190f7b0 0x5638d18e2750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:28.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.188+0000 7fa5bb400640 1 -- 172.21.15.6:0/176109113 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5638d18e2c90 msgr2=0x5638d1912a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:28.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.188+0000 7fa5bb400640 1 --2- 172.21.15.6:0/176109113 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5638d18e2c90 0x5638d1912a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:28.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.188+0000 7fa5bb400640 1 -- 172.21.15.6:0/176109113 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5638d15612d0 con 0x5638d190fb90 2024-03-04T09:30:28.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.189+0000 7fa5bb400640 1 --2- 172.21.15.6:0/176109113 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5638d190fb90 0x5638d1912540 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7fa5a8000c00 tx=0x7fa5a8030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d448192341b7d900 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:28.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.189+0000 7fa5a7e00640 1 -- 172.21.15.6:0/176109113 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5a8047070 con 0x5638d190fb90 2024-03-04T09:30:28.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.189+0000 7fa5a7e00640 1 -- 172.21.15.6:0/176109113 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5a802fc90 con 0x5638d190fb90 2024-03-04T09:30:28.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.189+0000 7fa5a7e00640 1 -- 172.21.15.6:0/176109113 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5a8042640 con 0x5638d190fb90 2024-03-04T09:30:28.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.190+0000 7fa5c5986740 1 -- 172.21.15.6:0/176109113 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5638d190fb90 msgr2=0x5638d1912540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:28.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.190+0000 7fa5c5986740 1 --2- 172.21.15.6:0/176109113 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5638d190fb90 0x5638d1912540 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7fa5a8000c00 tx=0x7fa5a8030000 comp rx=0 tx=0).stop 2024-03-04T09:30:28.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.190+0000 7fa5c5986740 1 -- 172.21.15.6:0/176109113 shutdown_connections 2024-03-04T09:30:28.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.190+0000 7fa5c5986740 1 --2- 172.21.15.6:0/176109113 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5638d18e2c90 0x5638d1912a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:28.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.190+0000 7fa5c5986740 1 --2- 172.21.15.6:0/176109113 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5638d190fb90 0x5638d1912540 unknown :-1 s=CLOSED pgs=1094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:28.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.190+0000 7fa5c5986740 1 --2- 172.21.15.6:0/176109113 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5638d190f7b0 0x5638d18e2750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:28.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.190+0000 7fa5c5986740 1 -- 172.21.15.6:0/176109113 >> 172.21.15.6:0/176109113 conn(0x5638d18e49c0 msgr2=0x5638d192d540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:28.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.190+0000 7fa5c5986740 1 -- 172.21.15.6:0/176109113 shutdown_connections 2024-03-04T09:30:28.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.190+0000 7fa5c5986740 1 -- 172.21.15.6:0/176109113 wait complete. 2024-03-04T09:30:28.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.191+0000 7fa5c5986740 1 -- start start 2024-03-04T09:30:28.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.191+0000 7fa5c5986740 1 Processor -- start 2024-03-04T09:30:28.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.192+0000 7fa5c5986740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5638d190f7b0 0x5638d193e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:28.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.192+0000 7fa5c5986740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5638d190fb90 0x5638d190abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:28.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.192+0000 7fa5c5986740 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5638d192e000 0x5638d1940360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:28.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.192+0000 7fa5c5986740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x5638d18e6f60 con 0x5638d190f7b0 2024-03-04T09:30:28.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.192+0000 7fa5c5986740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x5638d1808d90 con 0x5638d190fb90 2024-03-04T09:30:28.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.192+0000 7fa5c5986740 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x5638d18e6350 con 0x5638d192e000 2024-03-04T09:30:28.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.192+0000 7fa5c5986740 10 client.0 _ll_register_callbacks cb 0x5638d1844350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T09:30:28.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.192+0000 7fa5bbe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5638d190f7b0 0x5638d193e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:28.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.192+0000 7fa5bbe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5638d190f7b0 0x5638d193e3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38508/0 (socket says 172.21.15.6:38508) 2024-03-04T09:30:28.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.192+0000 7fa5bbe00640 1 -- 172.21.15.6:0/3250846345 learned_addr learned my addr 172.21.15.6:0/3250846345 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:30:28.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.193+0000 7fa5bb400640 1 --2- 172.21.15.6:0/3250846345 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5638d190fb90 0x5638d190abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:28.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.193+0000 7fa5c0c00640 1 --2- 172.21.15.6:0/3250846345 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5638d192e000 0x5638d1940360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:28.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.193+0000 7fa5bbe00640 1 -- 172.21.15.6:0/3250846345 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5638d192e000 msgr2=0x5638d1940360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:28.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.193+0000 7fa5bbe00640 1 --2- 172.21.15.6:0/3250846345 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5638d192e000 0x5638d1940360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:28.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.193+0000 7fa5bbe00640 1 -- 172.21.15.6:0/3250846345 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5638d190fb90 msgr2=0x5638d190abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:28.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.193+0000 7fa5bbe00640 1 --2- 172.21.15.6:0/3250846345 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5638d190fb90 0x5638d190abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:28.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.193+0000 7fa5bbe00640 1 -- 172.21.15.6:0/3250846345 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5638d162b6b0 con 0x5638d190f7b0 2024-03-04T09:30:28.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.193+0000 7fa5bb400640 1 --2- 172.21.15.6:0/3250846345 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5638d190fb90 0x5638d190abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:30:28.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.193+0000 7fa5bbe00640 1 --2- 172.21.15.6:0/3250846345 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5638d190f7b0 0x5638d193e3e0 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7fa5ac0137d0 tx=0x7fa5ac004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b9bb496c86be708 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:28.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.193+0000 7fa5a4c00640 10 client.0 ms_handle_connect on v2:172.21.15.6:3300/0 2024-03-04T09:30:28.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.193+0000 7fa5a4c00640 1 -- 172.21.15.6:0/3250846345 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5ac024020 con 0x5638d190f7b0 2024-03-04T09:30:28.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.193+0000 7fa5a4c00640 1 -- 172.21.15.6:0/3250846345 <== mon.0 v2:172.21.15.6:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa5ac014980 con 0x5638d190f7b0 2024-03-04T09:30:28.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.193+0000 7fa5c5986740 1 -- 172.21.15.6:0/3250846345 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa5a802fc90 con 0x5638d190f7b0 2024-03-04T09:30:28.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.194+0000 7fa5a4c00640 1 -- 172.21.15.6:0/3250846345 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_get_version_reply(handle=1 version=60) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa5ac02e050 con 0x5638d190f7b0 2024-03-04T09:30:28.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.194+0000 7fa5c5986740 10 client.7239 fetch_fsmap learned FSMap version 60 2024-03-04T09:30:28.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.194+0000 7fa5c5986740 1 -- 172.21.15.6:0/3250846345 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,fsmap=60,monmap=2+}) v3 -- 0x5638d19113b0 con 0x5638d190f7b0 2024-03-04T09:30:28.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.194+0000 7fa5a4c00640 1 -- 172.21.15.6:0/3250846345 <== mon.0 v2:172.21.15.6:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5a802fc90 con 0x5638d190f7b0 2024-03-04T09:30:28.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.194+0000 7fa5a4c00640 1 -- 172.21.15.6:0/3250846345 <== mon.0 v2:172.21.15.6:3300/0 5 ==== fsmap(e 60) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fa5ac031020 con 0x5638d190f7b0 2024-03-04T09:30:28.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.194+0000 7fa5c5986740 10 client.7239 fetch_fsmap finished waiting for FSMap version 60 2024-03-04T09:30:28.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.194+0000 7fa5c5986740 10 client.7239 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-04T09:30:28.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.194+0000 7fa5c5986740 20 client.7239 populate_metadata read hostname 'smithi006' 2024-03-04T09:30:28.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.194+0000 7fa5c5986740 1 --2- 172.21.15.6:0/3250846345 >> [v2:172.21.15.6:6832/2836916196,v1:172.21.15.6:6835/2836916196] conn(0x5638d1907850 0x5638d1941680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:28.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.194+0000 7fa5c5986740 4 client.7239 mds_command: new command op to 7095 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T09:30:28.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.194+0000 7fa5c5986740 1 -- 172.21.15.6:0/3250846345 --> [v2:172.21.15.6:6832/2836916196,v1:172.21.15.6:6835/2836916196] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5638d1859b40 con 0x5638d1907850 2024-03-04T09:30:28.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.195+0000 7fa5bb400640 1 --2- 172.21.15.6:0/3250846345 >> [v2:172.21.15.6:6832/2836916196,v1:172.21.15.6:6835/2836916196] conn(0x5638d1907850 0x5638d1941680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:28.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.195+0000 7fa5bb400640 1 --2- 172.21.15.6:0/3250846345 >> [v2:172.21.15.6:6832/2836916196,v1:172.21.15.6:6835/2836916196] conn(0x5638d1907850 0x5638d1941680 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6e5eb9ed6088009c server_cookie=b0408580f6fa5c45 in_seq=0 out_seq=0 2024-03-04T09:30:28.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.195+0000 7fa5a4c00640 10 client.7239 ms_handle_connect on v2:172.21.15.6:6832/2836916196 2024-03-04T09:30:28.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.196+0000 7fa5a4c00640 1 -- 172.21.15.6:0/3250846345 <== mds.0 v2:172.21.15.6:6832/2836916196 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5638d1859b40 con 0x5638d1907850 2024-03-04T09:30:28.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.196+0000 7fa5a4c00640 10 client.7239 handle_command_reply: tid=0 2024-03-04T09:30:28.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.196+0000 7fa5c5986740 1 client.7239 shutdown 2024-03-04T09:30:28.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.196+0000 7fa5c5986740 1 -- 172.21.15.6:0/3250846345 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5638d190f7b0 msgr2=0x5638d193e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:28.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.196+0000 7fa5c5986740 1 --2- 172.21.15.6:0/3250846345 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5638d190f7b0 0x5638d193e3e0 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7fa5ac0137d0 tx=0x7fa5ac004730 comp rx=0 tx=0).stop 2024-03-04T09:30:28.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.196+0000 7fa5c5986740 1 -- 172.21.15.6:0/3250846345 shutdown_connections 2024-03-04T09:30:28.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.196+0000 7fa5c5986740 1 --2- 172.21.15.6:0/3250846345 >> [v2:172.21.15.6:6832/2836916196,v1:172.21.15.6:6835/2836916196] conn(0x5638d1907850 0x5638d1941680 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:28.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.196+0000 7fa5c5986740 1 --2- 172.21.15.6:0/3250846345 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5638d192e000 0x5638d1940360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:28.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.197+0000 7fa5a4c00640 0 client.7239 ms_handle_reset on v2:172.21.15.6:6832/2836916196 2024-03-04T09:30:28.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.196+0000 7fa5c5986740 1 --2- 172.21.15.6:0/3250846345 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5638d190fb90 0x5638d190abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:28.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.197+0000 7fa5c5986740 1 --2- 172.21.15.6:0/3250846345 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5638d190f7b0 0x5638d193e3e0 unknown :-1 s=CLOSED pgs=1095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:28.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.197+0000 7fa5c5986740 1 -- 172.21.15.6:0/3250846345 >> 172.21.15.6:0/3250846345 conn(0x5638d18e2c90 msgr2=0x5638d1930a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:28.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.197+0000 7fa5c5986740 1 -- 172.21.15.6:0/3250846345 shutdown_connections 2024-03-04T09:30:28.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.197+0000 7fa5c5986740 1 -- 172.21.15.6:0/3250846345 wait complete. 2024-03-04T09:30:28.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.197+0000 7fa5c5986740 20 client.7239 trim_cache size 0 max 16384 2024-03-04T09:30:28.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.217+0000 7fa5c5986740 1 Processor -- start 2024-03-04T09:30:28.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.217+0000 7fa5c5986740 1 -- start start 2024-03-04T09:30:28.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.218+0000 7fa5c5986740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5638d190f7b0 0x5638d1910160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:28.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.218+0000 7fa5c5986740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5638d190fb90 0x5638d193ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:28.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.218+0000 7fa5c5986740 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5638d192e000 0x5638d193d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:28.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.218+0000 7fa5c5986740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5bc14df00 con 0x5638d190f7b0 2024-03-04T09:30:28.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.218+0000 7fa5c5986740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5bc071d50 con 0x5638d190fb90 2024-03-04T09:30:28.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.218+0000 7fa5c5986740 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5bc072350 con 0x5638d192e000 2024-03-04T09:30:28.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.218+0000 7fa5bbe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5638d190f7b0 0x5638d1910160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:28.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.218+0000 7fa5bbe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5638d190f7b0 0x5638d1910160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38518/0 (socket says 172.21.15.6:38518) 2024-03-04T09:30:28.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.218+0000 7fa5bbe00640 1 -- 172.21.15.6:0/4075601536 learned_addr learned my addr 172.21.15.6:0/4075601536 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:30:28.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.219+0000 7fa5bbe00640 1 -- 172.21.15.6:0/4075601536 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5638d192e000 msgr2=0x5638d193d1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:28.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.219+0000 7fa5bbe00640 1 --2- 172.21.15.6:0/4075601536 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5638d192e000 0x5638d193d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:28.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.219+0000 7fa5bbe00640 1 -- 172.21.15.6:0/4075601536 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5638d190fb90 msgr2=0x5638d193ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:28.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.219+0000 7fa5bbe00640 1 --2- 172.21.15.6:0/4075601536 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5638d190fb90 0x5638d193ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:28.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.219+0000 7fa5bbe00640 1 -- 172.21.15.6:0/4075601536 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5638d18e3200 con 0x5638d190f7b0 2024-03-04T09:30:28.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.219+0000 7fa5bbe00640 1 --2- 172.21.15.6:0/4075601536 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5638d190f7b0 0x5638d1910160 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7fa5ac00afa0 tx=0x7fa5ac01f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40877eb0fd00394e server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:28.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.219+0000 7fa5a7400640 1 -- 172.21.15.6:0/4075601536 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5ac024070 con 0x5638d190f7b0 2024-03-04T09:30:28.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.219+0000 7fa5a7400640 1 -- 172.21.15.6:0/4075601536 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5ac01b040 con 0x5638d190f7b0 2024-03-04T09:30:28.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.219+0000 7fa5a7400640 1 -- 172.21.15.6:0/4075601536 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5ac01a070 con 0x5638d190f7b0 2024-03-04T09:30:28.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.220+0000 7fa5c5986740 1 -- 172.21.15.6:0/4075601536 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5638d190f7b0 msgr2=0x5638d1910160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:28.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.220+0000 7fa5c5986740 1 --2- 172.21.15.6:0/4075601536 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5638d190f7b0 0x5638d1910160 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7fa5ac00afa0 tx=0x7fa5ac01f000 comp rx=0 tx=0).stop 2024-03-04T09:30:28.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.220+0000 7fa5c5986740 1 -- 172.21.15.6:0/4075601536 shutdown_connections 2024-03-04T09:30:28.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.220+0000 7fa5c5986740 1 --2- 172.21.15.6:0/4075601536 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5638d192e000 0x5638d193d1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:28.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.220+0000 7fa5c5986740 1 --2- 172.21.15.6:0/4075601536 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5638d190fb90 0x5638d193ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:28.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.220+0000 7fa5c5986740 1 --2- 172.21.15.6:0/4075601536 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5638d190f7b0 0x5638d1910160 unknown :-1 s=CLOSED pgs=1096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:28.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.220+0000 7fa5c5986740 1 -- 172.21.15.6:0/4075601536 >> 172.21.15.6:0/4075601536 conn(0x5638d18e2c90 msgr2=0x5638d18e1620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:28.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.220+0000 7fa5c5986740 1 -- 172.21.15.6:0/4075601536 shutdown_connections 2024-03-04T09:30:28.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.220+0000 7fa5c5986740 1 -- 172.21.15.6:0/4075601536 wait complete. 2024-03-04T09:30:28.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.221+0000 7fa5c5986740 1 -- start start 2024-03-04T09:30:28.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.221+0000 7fa5c5986740 1 Processor -- start 2024-03-04T09:30:28.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.222+0000 7fa5c5986740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5638d190f7b0 0x5638d18e7450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:28.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.222+0000 7fa5c5986740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5638d190fb90 0x5638d18e7990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:28.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.222+0000 7fa5c5986740 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5638d192e000 0x5638d192e790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:28.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.222+0000 7fa5c5986740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x5638d19076d0 con 0x5638d190fb90 2024-03-04T09:30:28.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.222+0000 7fa5c5986740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x5638d19091a0 con 0x5638d190f7b0 2024-03-04T09:30:28.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.222+0000 7fa5c5986740 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x5638d1907c40 con 0x5638d192e000 2024-03-04T09:30:28.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.222+0000 7fa5c5986740 10 client.0 _ll_register_callbacks cb 0x7fa5ac00a720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T09:30:28.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.222+0000 7fa5bb400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5638d190fb90 0x5638d18e7990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:28.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.222+0000 7fa5bb400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5638d190fb90 0x5638d18e7990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38528/0 (socket says 172.21.15.6:38528) 2024-03-04T09:30:28.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.222+0000 7fa5bb400640 1 -- 172.21.15.6:0/1509701924 learned_addr learned my addr 172.21.15.6:0/1509701924 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:30:28.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.223+0000 7fa5bb400640 1 -- 172.21.15.6:0/1509701924 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5638d192e000 msgr2=0x5638d192e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:28.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.223+0000 7fa5bb400640 1 --2- 172.21.15.6:0/1509701924 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5638d192e000 0x5638d192e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:28.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.223+0000 7fa5bb400640 1 -- 172.21.15.6:0/1509701924 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5638d190f7b0 msgr2=0x5638d18e7450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:28.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.223+0000 7fa5bb400640 1 --2- 172.21.15.6:0/1509701924 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5638d190f7b0 0x5638d18e7450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:28.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.223+0000 7fa5bb400640 1 -- 172.21.15.6:0/1509701924 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5638d18de050 con 0x5638d190fb90 2024-03-04T09:30:28.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.223+0000 7fa5bb400640 1 --2- 172.21.15.6:0/1509701924 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5638d190fb90 0x5638d18e7990 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7fa5a80090b0 tx=0x7fa5a80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5822454083596725 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:28.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.223+0000 7fa5a4c00640 10 client.0 ms_handle_connect on v2:172.21.15.6:3300/0 2024-03-04T09:30:28.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.223+0000 7fa5a4c00640 1 -- 172.21.15.6:0/1509701924 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5a8047040 con 0x5638d190fb90 2024-03-04T09:30:28.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.223+0000 7fa5a4c00640 1 -- 172.21.15.6:0/1509701924 <== mon.0 v2:172.21.15.6:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa5a802fb10 con 0x5638d190fb90 2024-03-04T09:30:28.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.223+0000 7fa5c5986740 1 -- 172.21.15.6:0/1509701924 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa5ac01b040 con 0x5638d190fb90 2024-03-04T09:30:28.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.224+0000 7fa5a4c00640 1 -- 172.21.15.6:0/1509701924 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_get_version_reply(handle=1 version=60) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa5a8004040 con 0x5638d190fb90 2024-03-04T09:30:28.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.224+0000 7fa5c5986740 10 client.7245 fetch_fsmap learned FSMap version 60 2024-03-04T09:30:28.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.224+0000 7fa5c5986740 1 -- 172.21.15.6:0/1509701924 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,fsmap=60,monmap=2+}) v3 -- 0x5638d19065f0 con 0x5638d190fb90 2024-03-04T09:30:28.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.224+0000 7fa5a4c00640 1 -- 172.21.15.6:0/1509701924 <== mon.0 v2:172.21.15.6:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5a8061070 con 0x5638d190fb90 2024-03-04T09:30:28.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.224+0000 7fa5a4c00640 1 -- 172.21.15.6:0/1509701924 <== mon.0 v2:172.21.15.6:3300/0 5 ==== fsmap(e 60) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fa5a80041e0 con 0x5638d190fb90 2024-03-04T09:30:28.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.224+0000 7fa5c5986740 10 client.7245 fetch_fsmap finished waiting for FSMap version 60 2024-03-04T09:30:28.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.224+0000 7fa5c5986740 10 client.7245 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-04T09:30:28.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.224+0000 7fa5c5986740 20 client.7245 populate_metadata read hostname 'smithi006' 2024-03-04T09:30:28.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.224+0000 7fa5c5986740 1 --2- 172.21.15.6:0/1509701924 >> [v2:172.21.15.6:6832/2836916196,v1:172.21.15.6:6835/2836916196] conn(0x5638d192ecd0 0x5638d19476b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:28.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.224+0000 7fa5c5986740 4 client.7245 mds_command: new command op to 7095 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-04T09:30:28.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.224+0000 7fa5c5986740 1 -- 172.21.15.6:0/1509701924 --> [v2:172.21.15.6:6832/2836916196,v1:172.21.15.6:6835/2836916196] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5638d1853e90 con 0x5638d192ecd0 2024-03-04T09:30:28.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.224+0000 7fa5bbe00640 1 --2- 172.21.15.6:0/1509701924 >> [v2:172.21.15.6:6832/2836916196,v1:172.21.15.6:6835/2836916196] conn(0x5638d192ecd0 0x5638d19476b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:28.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.225+0000 7fa5bbe00640 1 --2- 172.21.15.6:0/1509701924 >> [v2:172.21.15.6:6832/2836916196,v1:172.21.15.6:6835/2836916196] conn(0x5638d192ecd0 0x5638d19476b0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2e560b2dec987820 server_cookie=57a6d81f36d3f81d in_seq=0 out_seq=0 2024-03-04T09:30:28.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.225+0000 7fa5a4c00640 10 client.7245 ms_handle_connect on v2:172.21.15.6:6832/2836916196 2024-03-04T09:30:28.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.225+0000 7fa5a4c00640 1 -- 172.21.15.6:0/1509701924 <== mds.0 v2:172.21.15.6:6832/2836916196 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x5638d1853e90 con 0x5638d192ecd0 2024-03-04T09:30:28.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.225+0000 7fa5a4c00640 10 client.7245 handle_command_reply: tid=0 2024-03-04T09:30:28.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.225+0000 7fa5c5986740 1 client.7245 shutdown 2024-03-04T09:30:28.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.225+0000 7fa5c5986740 1 -- 172.21.15.6:0/1509701924 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5638d190fb90 msgr2=0x5638d18e7990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:28.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.225+0000 7fa5c5986740 1 --2- 172.21.15.6:0/1509701924 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5638d190fb90 0x5638d18e7990 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7fa5a80090b0 tx=0x7fa5a80046c0 comp rx=0 tx=0).stop 2024-03-04T09:30:28.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.225+0000 7fa5c5986740 1 -- 172.21.15.6:0/1509701924 shutdown_connections 2024-03-04T09:30:28.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.225+0000 7fa5c5986740 1 --2- 172.21.15.6:0/1509701924 >> [v2:172.21.15.6:6832/2836916196,v1:172.21.15.6:6835/2836916196] conn(0x5638d192ecd0 0x5638d19476b0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:28.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.225+0000 7fa5c5986740 1 --2- 172.21.15.6:0/1509701924 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5638d192e000 0x5638d192e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:28.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.225+0000 7fa5c5986740 1 --2- 172.21.15.6:0/1509701924 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5638d190fb90 0x5638d18e7990 unknown :-1 s=CLOSED pgs=1097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:28.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.225+0000 7fa5c5986740 1 --2- 172.21.15.6:0/1509701924 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5638d190f7b0 0x5638d18e7450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:28.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.225+0000 7fa5a4c00640 0 client.7245 ms_handle_reset on v2:172.21.15.6:6832/2836916196 2024-03-04T09:30:28.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.225+0000 7fa5c5986740 1 -- 172.21.15.6:0/1509701924 >> 172.21.15.6:0/1509701924 conn(0x5638d18e2c90 msgr2=0x5638d18e1620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:28.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.226+0000 7fa5c5986740 1 -- 172.21.15.6:0/1509701924 shutdown_connections 2024-03-04T09:30:28.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.226+0000 7fa5c5986740 1 -- 172.21.15.6:0/1509701924 wait complete. 2024-03-04T09:30:28.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.226+0000 7fa5c5986740 20 client.7245 trim_cache size 0 max 16384 2024-03-04T09:30:28.228 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-04T09:30:28.228 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_fsid": "edce0578-5ae4-4402-b45a-552bdef84d7c", 2024-03-04T09:30:28.228 INFO:teuthology.orchestra.run.smithi006.stdout: "whoami": 0, 2024-03-04T09:30:28.228 INFO:teuthology.orchestra.run.smithi006.stdout: "id": 7095, 2024-03-04T09:30:28.229 INFO:teuthology.orchestra.run.smithi006.stdout: "want_state": "up:active", 2024-03-04T09:30:28.229 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "up:active", 2024-03-04T09:30:28.229 INFO:teuthology.orchestra.run.smithi006.stdout: "fs_name": "cephfs", 2024-03-04T09:30:28.229 INFO:teuthology.orchestra.run.smithi006.stdout: "rank_uptime": 7.0331872129999997, 2024-03-04T09:30:28.229 INFO:teuthology.orchestra.run.smithi006.stdout: "mdsmap_epoch": 60, 2024-03-04T09:30:28.229 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap_epoch": 94, 2024-03-04T09:30:28.229 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap_epoch_barrier": 94, 2024-03-04T09:30:28.229 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 12.411326146 2024-03-04T09:30:28.229 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-04T09:30:28.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.226+0000 7fa5c2c00640 1 -- 172.21.15.6:0/3218137812 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa59c06b4d0 msgr2=0x7fa59c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:28.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.226+0000 7fa5c2c00640 1 --2- 172.21.15.6:0/3218137812 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa59c06b4d0 0x7fa59c06d990 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fa5ac00a970 tx=0x7fa5ac008040 comp rx=0 tx=0).stop 2024-03-04T09:30:28.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.226+0000 7fa5c2c00640 1 -- 172.21.15.6:0/3218137812 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa5bc0cf5c0 msgr2=0x7fa5bc151a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:28.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.226+0000 7fa5c2c00640 1 --2- 172.21.15.6:0/3218137812 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa5bc0cf5c0 0x7fa5bc151a30 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7fa5b400c780 tx=0x7fa5b400cc40 comp rx=0 tx=0).stop 2024-03-04T09:30:28.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.227+0000 7fa5c2c00640 1 -- 172.21.15.6:0/3218137812 shutdown_connections 2024-03-04T09:30:28.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.227+0000 7fa5c2c00640 1 --2- 172.21.15.6:0/3218137812 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa5bc0cf5c0 0x7fa5bc151a30 unknown :-1 s=CLOSED pgs=1093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:28.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.227+0000 7fa5c2c00640 1 --2- 172.21.15.6:0/3218137812 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa59c06b4d0 0x7fa59c06d990 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:28.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.227+0000 7fa5c2c00640 1 --2- 172.21.15.6:0/3218137812 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa5bc0b3c90 0x7fa5bc09cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:28.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.227+0000 7fa5c2c00640 1 --2- 172.21.15.6:0/3218137812 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa5bc06cb10 0x7fa5bc14bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:28.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.227+0000 7fa5c2c00640 1 -- 172.21.15.6:0/3218137812 >> 172.21.15.6:0/3218137812 conn(0x7fa5bc06afb0 msgr2=0x7fa5bc0ce4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:28.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.227+0000 7fa5c2c00640 1 -- 172.21.15.6:0/3218137812 shutdown_connections 2024-03-04T09:30:28.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.227+0000 7fa5c2c00640 1 -- 172.21.15.6:0/3218137812 wait complete. 2024-03-04T09:30:28.244 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-04T09:30:28.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.463+0000 7fa346000640 1 Processor -- start 2024-03-04T09:30:28.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.463+0000 7fa346000640 1 -- start start 2024-03-04T09:30:28.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.464+0000 7fa346000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa34015e0b0 0x7fa3400cf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:28.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.464+0000 7fa346000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa3400d4a10 0x7fa3400d0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:28.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.464+0000 7fa346000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3400b3710 con 0x7fa3400d0c80 2024-03-04T09:30:28.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.464+0000 7fa346000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3400b09e0 con 0x7fa3400d4a10 2024-03-04T09:30:28.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.464+0000 7fa346000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3400b39f0 con 0x7fa34015e0b0 2024-03-04T09:30:28.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.465+0000 7fa337e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa3400d4a10 0x7fa3400d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:28.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.465+0000 7fa337e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa3400d4a10 0x7fa3400d0740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38534/0 (socket says 172.21.15.6:38534) 2024-03-04T09:30:28.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.465+0000 7fa337e00640 1 -- 172.21.15.6:0/2671823142 learned_addr learned my addr 172.21.15.6:0/2671823142 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:30:28.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.465+0000 7fa344c00640 1 --2- 172.21.15.6:0/2671823142 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa34015e0b0 0x7fa3400cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:28.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.465+0000 7fa337e00640 1 -- 172.21.15.6:0/2671823142 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa34015e0b0 msgr2=0x7fa3400cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:28.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.465+0000 7fa337e00640 1 --2- 172.21.15.6:0/2671823142 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa34015e0b0 0x7fa3400cf310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:28.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.465+0000 7fa337e00640 1 -- 172.21.15.6:0/2671823142 >> v1:172.21.15.146:6789/0 conn(0x7fa3400d0c80 legacy=0x7fa3400d1900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:28.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.465+0000 7fa337e00640 1 -- 172.21.15.6:0/2671823142 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3400b3cd0 con 0x7fa3400d4a10 2024-03-04T09:30:28.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.465+0000 7fa344c00640 1 --2- 172.21.15.6:0/2671823142 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa34015e0b0 0x7fa3400cf310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:30:28.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.466+0000 7fa337e00640 1 --2- 172.21.15.6:0/2671823142 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa3400d4a10 0x7fa3400d0740 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7fa330009870 tx=0x7fa33002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3dfe1998f2b8891 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:28.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.467+0000 7fa33fe00640 1 -- 172.21.15.6:0/2671823142 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa330035020 con 0x7fa3400d4a10 2024-03-04T09:30:28.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.467+0000 7fa33fe00640 1 -- 172.21.15.6:0/2671823142 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa330031640 con 0x7fa3400d4a10 2024-03-04T09:30:28.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.467+0000 7fa33fe00640 1 -- 172.21.15.6:0/2671823142 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa33002d040 con 0x7fa3400d4a10 2024-03-04T09:30:28.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.467+0000 7fa346000640 1 -- 172.21.15.6:0/2671823142 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa3400d4a10 msgr2=0x7fa3400d0740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:28.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.467+0000 7fa346000640 1 --2- 172.21.15.6:0/2671823142 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa3400d4a10 0x7fa3400d0740 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7fa330009870 tx=0x7fa33002b8f0 comp rx=0 tx=0).stop 2024-03-04T09:30:28.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.468+0000 7fa346000640 1 -- 172.21.15.6:0/2671823142 shutdown_connections 2024-03-04T09:30:28.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.468+0000 7fa346000640 1 --2- 172.21.15.6:0/2671823142 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa3400d4a10 0x7fa3400d0740 unknown :-1 s=CLOSED pgs=1098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:28.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.468+0000 7fa346000640 1 --2- 172.21.15.6:0/2671823142 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa34015e0b0 0x7fa3400cf310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:28.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.468+0000 7fa346000640 1 -- 172.21.15.6:0/2671823142 >> 172.21.15.6:0/2671823142 conn(0x7fa3400d3d30 msgr2=0x7fa3400d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:28.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.468+0000 7fa346000640 1 -- 172.21.15.6:0/2671823142 shutdown_connections 2024-03-04T09:30:28.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.468+0000 7fa346000640 1 -- 172.21.15.6:0/2671823142 wait complete. 2024-03-04T09:30:28.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.470+0000 7fa346000640 1 Processor -- start 2024-03-04T09:30:28.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.470+0000 7fa346000640 1 -- start start 2024-03-04T09:30:28.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.471+0000 7fa346000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa3400d0c80 0x7fa340157270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:28.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.471+0000 7fa346000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa3400d4a10 0x7fa3400b0c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:28.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.472+0000 7fa346000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa34015e0b0 0x7fa3400b8ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:28.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.472+0000 7fa346000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3400cfc60 con 0x7fa3400d0c80 2024-03-04T09:30:28.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.472+0000 7fa346000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa34015e490 con 0x7fa34015e0b0 2024-03-04T09:30:28.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.472+0000 7fa346000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3400b54e0 con 0x7fa3400d4a10 2024-03-04T09:30:28.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.472+0000 7fa344c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa3400d0c80 0x7fa340157270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:28.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.472+0000 7fa344c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa3400d0c80 0x7fa340157270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38548/0 (socket says 172.21.15.6:38548) 2024-03-04T09:30:28.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.472+0000 7fa344c00640 1 -- 172.21.15.6:0/766227117 learned_addr learned my addr 172.21.15.6:0/766227117 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:30:28.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.472+0000 7fa337e00640 1 --2- 172.21.15.6:0/766227117 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa3400d4a10 0x7fa3400b0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:28.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.472+0000 7fa345600640 1 --2- 172.21.15.6:0/766227117 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa34015e0b0 0x7fa3400b8ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:28.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.472+0000 7fa344c00640 1 -- 172.21.15.6:0/766227117 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa3400d4a10 msgr2=0x7fa3400b0c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:28.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.472+0000 7fa344c00640 1 --2- 172.21.15.6:0/766227117 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa3400d4a10 0x7fa3400b0c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:28.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.472+0000 7fa344c00640 1 -- 172.21.15.6:0/766227117 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa34015e0b0 msgr2=0x7fa3400b8ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:28.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.472+0000 7fa344c00640 1 --2- 172.21.15.6:0/766227117 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa34015e0b0 0x7fa3400b8ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:28.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.472+0000 7fa344c00640 1 -- 172.21.15.6:0/766227117 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3400b25f0 con 0x7fa3400d0c80 2024-03-04T09:30:28.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.473+0000 7fa344c00640 1 --2- 172.21.15.6:0/766227117 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa3400d0c80 0x7fa340157270 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7fa32800c7f0 tx=0x7fa32800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:28.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.473+0000 7fa33d600640 1 -- 172.21.15.6:0/766227117 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa328015070 con 0x7fa3400d0c80 2024-03-04T09:30:28.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.473+0000 7fa33d600640 1 -- 172.21.15.6:0/766227117 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa328007820 con 0x7fa3400d0c80 2024-03-04T09:30:28.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.473+0000 7fa33d600640 1 -- 172.21.15.6:0/766227117 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa328007bd0 con 0x7fa3400d0c80 2024-03-04T09:30:28.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.473+0000 7fa346000640 1 -- 172.21.15.6:0/766227117 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3400b1550 con 0x7fa3400d0c80 2024-03-04T09:30:28.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.473+0000 7fa346000640 1 -- 172.21.15.6:0/766227117 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3400b1b10 con 0x7fa3400d0c80 2024-03-04T09:30:28.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.475+0000 7fa346000640 1 -- 172.21.15.6:0/766227117 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa34004f330 con 0x7fa3400d0c80 2024-03-04T09:30:28.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.475+0000 7fa33d600640 1 -- 172.21.15.6:0/766227117 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa328018070 con 0x7fa3400d0c80 2024-03-04T09:30:28.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.476+0000 7fa33d600640 1 --2- 172.21.15.6:0/766227117 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa31c06b500 0x7fa31c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:28.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.476+0000 7fa33d600640 1 -- 172.21.15.6:0/766227117 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa32808a220 con 0x7fa3400d0c80 2024-03-04T09:30:28.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.477+0000 7fa337e00640 1 --2- 172.21.15.6:0/766227117 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa31c06b500 0x7fa31c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:28.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.478+0000 7fa337e00640 1 --2- 172.21.15.6:0/766227117 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa31c06b500 0x7fa31c06d9c0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fa330036000 tx=0x7fa33003c040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:28.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.481+0000 7fa33d600640 1 -- 172.21.15.6:0/766227117 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa328059130 con 0x7fa3400d0c80 2024-03-04T09:30:28.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.721+0000 7fa346000640 1 -- 172.21.15.6:0/766227117 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fa3400d22b0 con 0x7fa3400d0c80 2024-03-04T09:30:28.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.722+0000 7fa33d600640 1 -- 172.21.15.6:0/766227117 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v60) v1 ==== 53+0+73 (secure 0 0 0) 0x7fa328058a30 con 0x7fa3400d0c80 2024-03-04T09:30:28.718 INFO:teuthology.orchestra.run.smithi006.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-04T09:30:28.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.724+0000 7fa346000640 1 -- 172.21.15.6:0/766227117 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa31c06b500 msgr2=0x7fa31c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:28.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.724+0000 7fa346000640 1 --2- 172.21.15.6:0/766227117 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa31c06b500 0x7fa31c06d9c0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fa330036000 tx=0x7fa33003c040 comp rx=0 tx=0).stop 2024-03-04T09:30:28.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.724+0000 7fa346000640 1 -- 172.21.15.6:0/766227117 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa3400d0c80 msgr2=0x7fa340157270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:28.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.724+0000 7fa346000640 1 --2- 172.21.15.6:0/766227117 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa3400d0c80 0x7fa340157270 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7fa32800c7f0 tx=0x7fa32800ccb0 comp rx=0 tx=0).stop 2024-03-04T09:30:28.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.724+0000 7fa346000640 1 -- 172.21.15.6:0/766227117 shutdown_connections 2024-03-04T09:30:28.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.724+0000 7fa346000640 1 --2- 172.21.15.6:0/766227117 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa34015e0b0 0x7fa3400b8ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:28.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.724+0000 7fa346000640 1 --2- 172.21.15.6:0/766227117 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa31c06b500 0x7fa31c06d9c0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:28.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.724+0000 7fa346000640 1 --2- 172.21.15.6:0/766227117 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa3400d4a10 0x7fa3400b0c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:28.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.724+0000 7fa346000640 1 --2- 172.21.15.6:0/766227117 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa3400d0c80 0x7fa340157270 unknown :-1 s=CLOSED pgs=1099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:28.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.724+0000 7fa346000640 1 -- 172.21.15.6:0/766227117 >> 172.21.15.6:0/766227117 conn(0x7fa3400d3d30 msgr2=0x7fa34006cca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:28.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.724+0000 7fa346000640 1 -- 172.21.15.6:0/766227117 shutdown_connections 2024-03-04T09:30:28.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:28.725+0000 7fa346000640 1 -- 172.21.15.6:0/766227117 wait complete. 2024-03-04T09:30:28.738 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-04T09:30:28.738 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-04T09:30:28.738 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi006.front.sepia.ceph.com 2024-03-04T09:30:28.738 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-04T09:30:28.739 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-04T09:30:28.739 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-04T09:30:28.739 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-03-04T09:30:28.739 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-04T09:30:28.739 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-04T09:30:28.739 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:30:28.739 DEBUG:teuthology.orchestra.run.smithi006:> ip addr 2024-03-04T09:30:28.785 INFO:teuthology.orchestra.run.smithi006.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-04T09:30:28.785 INFO:teuthology.orchestra.run.smithi006.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-04T09:30:28.785 INFO:teuthology.orchestra.run.smithi006.stdout: inet 127.0.0.1/8 scope host lo 2024-03-04T09:30:28.786 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-04T09:30:28.786 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 ::1/128 scope host 2024-03-04T09:30:28.786 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-04T09:30:28.786 INFO:teuthology.orchestra.run.smithi006.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-04T09:30:28.786 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:6b:fe:e6 brd ff:ff:ff:ff:ff:ff 2024-03-04T09:30:28.786 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp7s0f0 2024-03-04T09:30:28.786 INFO:teuthology.orchestra.run.smithi006.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-04T09:30:28.786 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:6b:fe:e7 brd ff:ff:ff:ff:ff:ff 2024-03-04T09:30:28.786 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp7s0f1 2024-03-04T09:30:28.786 INFO:teuthology.orchestra.run.smithi006.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-04T09:30:28.786 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:bd:15:e4 brd ff:ff:ff:ff:ff:ff 2024-03-04T09:30:28.786 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp1s0f0 2024-03-04T09:30:28.786 INFO:teuthology.orchestra.run.smithi006.stdout: inet 172.21.15.6/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-04T09:30:28.787 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft 42305sec preferred_lft 42305sec 2024-03-04T09:30:28.787 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::ec4:7aff:febd:15e4/64 scope link noprefixroute 2024-03-04T09:30:28.787 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-04T09:30:28.787 INFO:teuthology.orchestra.run.smithi006.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-04T09:30:28.787 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:bd:15:e5 brd ff:ff:ff:ff:ff:ff 2024-03-04T09:30:28.787 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp1s0f1 2024-03-04T09:30:28.787 INFO:teuthology.orchestra.run.smithi006.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-04T09:30:28.787 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 5e:b4:59:dc:42:58 brd ff:ff:ff:ff:ff:ff 2024-03-04T09:30:28.787 INFO:teuthology.orchestra.run.smithi006.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-04T09:30:28.788 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-04T09:30:28.788 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::c6f:adff:fefe:e00d/64 scope link proto kernel_ll 2024-03-04T09:30:28.788 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-04T09:30:28.788 INFO:teuthology.orchestra.run.smithi006.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-04T09:30:28.788 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 7e:67:6d:ea:f9:ff brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-04T09:30:28.788 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::7c67:6dff:feea:f9ff/64 scope link proto kernel_ll 2024-03-04T09:30:28.788 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-04T09:30:28.788 INFO:teuthology.orchestra.run.smithi006.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-04T09:30:28.789 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 5e:b4:59:dc:42:58 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-04T09:30:28.789 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::5cb4:59ff:fedc:4258/64 scope link proto kernel_ll 2024-03-04T09:30:28.789 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-04T09:30:28.789 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:30:28.789 DEBUG:teuthology.orchestra.run.smithi006:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-04T09:30:28.919 INFO:teuthology.orchestra.run.smithi006.stdout:1 2024-03-04T09:30:28.924 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:30:28.924 DEBUG:teuthology.orchestra.run.smithi006:> route 2024-03-04T09:30:28.975 INFO:teuthology.orchestra.run.smithi006.stdout:Kernel IP routing table 2024-03-04T09:30:28.975 INFO:teuthology.orchestra.run.smithi006.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-04T09:30:28.975 INFO:teuthology.orchestra.run.smithi006.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-04T09:30:28.975 INFO:teuthology.orchestra.run.smithi006.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-04T09:30:28.975 INFO:teuthology.orchestra.run.smithi006.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-04T09:30:28.975 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:30:28.976 DEBUG:teuthology.orchestra.run.smithi006:> (cd / && exec bash -c ' 2024-03-04T09:30:28.976 DEBUG:teuthology.orchestra.run.smithi006:> set -e 2024-03-04T09:30:28.976 DEBUG:teuthology.orchestra.run.smithi006:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-04T09:30:28.976 DEBUG:teuthology.orchestra.run.smithi006:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-04T09:30:28.976 DEBUG:teuthology.orchestra.run.smithi006:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-04T09:30:28.976 DEBUG:teuthology.orchestra.run.smithi006:> ') 2024-03-04T09:30:29.162 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:30:29.162 DEBUG:teuthology.orchestra.run.smithi006:> ip netns list 2024-03-04T09:30:29.216 INFO:teuthology.orchestra.run.smithi006.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-04T09:30:29.216 INFO:teuthology.orchestra.run.smithi006.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-04T09:30:29.216 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-04T09:30:29.216 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:30:29.217 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip link set brx.0 up 2024-03-04T09:30:29.300 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-04T09:30:29.300 DEBUG:teuthology.orchestra.run.smithi006:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-04T09:30:29.348 INFO:teuthology.orchestra.run.smithi006.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-04T09:30:29.349 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-04T09:30:29.349 DEBUG:teuthology.orchestra.run.smithi006:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-04T09:30:29.397 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-03-04T09:30:29.397 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:30:29.398 DEBUG:teuthology.orchestra.run.smithi006:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-04T09:30:29.533 INFO:teuthology.orchestra.run.smithi006.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-04T09:30:29.533 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-04T09:30:29.568 INFO:teuthology.orchestra.run.smithi006.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-04T09:30:29.569 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-04T09:30:29.569 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: resolved to: "172.21.15.6:3300,172.21.15.146:3300,172.21.15.146:3301" 2024-03-04T09:30:29.569 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: trying mount with new device syntax: 0@edce0578-5ae4-4402-b45a-552bdef84d7c.cephfs=/ 2024-03-04T09:30:29.569 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.6:3300/172.21.15.146:3300/172.21.15.146:3301" will pass to kernel 2024-03-04T09:30:29.572 INFO:teuthology.orchestra.run.smithi006.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-04T09:30:29.572 INFO:teuthology.orchestra.run.smithi006.stdout: You just mounted a file system that supports labels which does not 2024-03-04T09:30:29.572 INFO:teuthology.orchestra.run.smithi006.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-04T09:30:29.572 INFO:teuthology.orchestra.run.smithi006.stdout: applications will generate AVC messages and not be allowed access to 2024-03-04T09:30:29.572 INFO:teuthology.orchestra.run.smithi006.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-04T09:30:29.576 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-04T09:30:29.576 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:30:29.576 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-04T09:30:29.658 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:30:29.661 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-04T09:30:29.724 INFO:teuthology.orchestra.run.smithi006.stdout:client5624 2024-03-04T09:30:29.725 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-04T09:30:29.725 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T09:30:29.778 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-04T09:30:29.783 DEBUG:teuthology.orchestra.run.smithi146:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-04T09:30:29.955 INFO:teuthology.orchestra.run.smithi006.stdout:[global] 2024-03-04T09:30:29.955 INFO:teuthology.orchestra.run.smithi006.stdout: chdir = "" 2024-03-04T09:30:29.956 INFO:teuthology.orchestra.run.smithi006.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-04T09:30:29.956 INFO:teuthology.orchestra.run.smithi006.stdout: auth supported = cephx 2024-03-04T09:30:29.956 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:30:29.956 INFO:teuthology.orchestra.run.smithi006.stdout: filestore xattr use omap = true 2024-03-04T09:30:29.956 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:30:29.956 INFO:teuthology.orchestra.run.smithi006.stdout: mon clock drift allowed = 1.000 2024-03-04T09:30:29.956 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:30:29.956 INFO:teuthology.orchestra.run.smithi006.stdout: osd crush chooseleaf type = 0 2024-03-04T09:30:29.956 INFO:teuthology.orchestra.run.smithi006.stdout: auth debug = true 2024-03-04T09:30:29.956 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:30:29.956 INFO:teuthology.orchestra.run.smithi006.stdout: ms die on old message = true 2024-03-04T09:30:29.956 INFO:teuthology.orchestra.run.smithi006.stdout: ms die on bug = true 2024-03-04T09:30:29.957 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:30:29.957 INFO:teuthology.orchestra.run.smithi006.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-04T09:30:29.957 INFO:teuthology.orchestra.run.smithi006.stdout: mon pg warn max object skew = 0 2024-03-04T09:30:29.957 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:30:29.957 INFO:teuthology.orchestra.run.smithi006.stdout: # disable pg_autoscaler by default for new pools 2024-03-04T09:30:29.957 INFO:teuthology.orchestra.run.smithi006.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-04T09:30:29.957 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:30:29.957 INFO:teuthology.orchestra.run.smithi006.stdout: osd pool default size = 2 2024-03-04T09:30:29.957 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:30:29.957 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd allow primary affinity = true 2024-03-04T09:30:29.957 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd allow pg remap = true 2024-03-04T09:30:29.957 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on legacy crush tunables = false 2024-03-04T09:30:29.957 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on crush straw calc version zero = false 2024-03-04T09:30:29.957 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on no sortbitwise = false 2024-03-04T09:30:29.958 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on osd down out interval zero = false 2024-03-04T09:30:29.958 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on too few osds = false 2024-03-04T09:30:29.958 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-04T09:30:29.958 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-04T09:30:29.958 INFO:teuthology.orchestra.run.smithi006.stdout: mon_allow_pool_size_one = true 2024-03-04T09:30:29.958 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:30:29.958 INFO:teuthology.orchestra.run.smithi006.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-04T09:30:29.958 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:30:29.958 INFO:teuthology.orchestra.run.smithi006.stdout: osd default data pool replay window = 5 2024-03-04T09:30:29.958 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:30:29.958 INFO:teuthology.orchestra.run.smithi006.stdout: mon allow pool delete = true 2024-03-04T09:30:29.958 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:30:29.958 INFO:teuthology.orchestra.run.smithi006.stdout: mon cluster log file level = debug 2024-03-04T09:30:29.958 INFO:teuthology.orchestra.run.smithi006.stdout: debug asserts on shutdown = true 2024-03-04T09:30:29.959 INFO:teuthology.orchestra.run.smithi006.stdout: mon health detail to clog = false 2024-03-04T09:30:29.959 INFO:teuthology.orchestra.run.smithi006.stdout: mon host = "172.21.15.6,172.21.15.146,[v2:172.21.15.146:3301,v1:172.21.15.146:6790]" 2024-03-04T09:30:29.959 INFO:teuthology.orchestra.run.smithi006.stdout: ms die on skipped message = False 2024-03-04T09:30:29.959 INFO:teuthology.orchestra.run.smithi006.stdout: fsid = edce0578-5ae4-4402-b45a-552bdef84d7c 2024-03-04T09:30:29.959 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:30:29.959 INFO:teuthology.orchestra.run.smithi006.stdout:[osd] 2024-03-04T09:30:29.959 INFO:teuthology.orchestra.run.smithi006.stdout: osd journal size = 100 2024-03-04T09:30:29.959 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:30:29.959 INFO:teuthology.orchestra.run.smithi006.stdout: osd scrub load threshold = 5.0 2024-03-04T09:30:29.959 INFO:teuthology.orchestra.run.smithi006.stdout: osd scrub max interval = 600 2024-03-04T09:30:29.959 INFO:teuthology.orchestra.run.smithi006.stdout: osd mclock profile = high_recovery_ops 2024-03-04T09:30:29.959 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:30:29.959 INFO:teuthology.orchestra.run.smithi006.stdout: osd recover clone overlap = true 2024-03-04T09:30:29.960 INFO:teuthology.orchestra.run.smithi006.stdout: osd recovery max chunk = 1048576 2024-03-04T09:30:29.960 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:30:29.960 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug shutdown = true 2024-03-04T09:30:29.960 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug op order = true 2024-03-04T09:30:29.960 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug verify stray on activate = true 2024-03-04T09:30:29.960 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug trim objects = true 2024-03-04T09:30:29.960 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:30:29.960 INFO:teuthology.orchestra.run.smithi006.stdout: osd open classes on start = true 2024-03-04T09:30:29.960 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug pg log writeout = true 2024-03-04T09:30:29.960 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:30:29.960 INFO:teuthology.orchestra.run.smithi006.stdout: osd deep scrub update digest min age = 30 2024-03-04T09:30:29.960 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:30:29.960 INFO:teuthology.orchestra.run.smithi006.stdout: osd map max advance = 10 2024-03-04T09:30:29.961 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:30:29.961 INFO:teuthology.orchestra.run.smithi006.stdout: journal zero on create = true 2024-03-04T09:30:29.961 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:30:29.961 INFO:teuthology.orchestra.run.smithi006.stdout: filestore ondisk finisher threads = 3 2024-03-04T09:30:29.961 INFO:teuthology.orchestra.run.smithi006.stdout: filestore apply finisher threads = 3 2024-03-04T09:30:29.961 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:30:29.961 INFO:teuthology.orchestra.run.smithi006.stdout: bdev debug aio = true 2024-03-04T09:30:29.961 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug misdirected ops = true 2024-03-04T09:30:29.961 INFO:teuthology.orchestra.run.smithi006.stdout: bdev async discard = True 2024-03-04T09:30:29.961 INFO:teuthology.orchestra.run.smithi006.stdout: bdev enable discard = True 2024-03-04T09:30:29.961 INFO:teuthology.orchestra.run.smithi006.stdout: bluestore allocator = bitmap 2024-03-04T09:30:29.961 INFO:teuthology.orchestra.run.smithi006.stdout: bluestore block size = 96636764160 2024-03-04T09:30:29.961 INFO:teuthology.orchestra.run.smithi006.stdout: bluestore fsck on mount = True 2024-03-04T09:30:29.961 INFO:teuthology.orchestra.run.smithi006.stdout: debug bluefs = 1/20 2024-03-04T09:30:29.962 INFO:teuthology.orchestra.run.smithi006.stdout: debug bluestore = 1/20 2024-03-04T09:30:29.962 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2024-03-04T09:30:29.962 INFO:teuthology.orchestra.run.smithi006.stdout: debug osd = 20 2024-03-04T09:30:29.962 INFO:teuthology.orchestra.run.smithi006.stdout: debug rocksdb = 4/10 2024-03-04T09:30:29.962 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd backfillfull_ratio = 0.85 2024-03-04T09:30:29.962 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd full ratio = 0.9 2024-03-04T09:30:29.962 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd nearfull ratio = 0.8 2024-03-04T09:30:29.962 INFO:teuthology.orchestra.run.smithi006.stdout: osd failsafe full ratio = 0.95 2024-03-04T09:30:29.962 INFO:teuthology.orchestra.run.smithi006.stdout: osd objectstore = bluestore 2024-03-04T09:30:29.962 INFO:teuthology.orchestra.run.smithi006.stdout: osd op complaint time = 180 2024-03-04T09:30:29.962 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:30:29.962 INFO:teuthology.orchestra.run.smithi006.stdout:[mgr] 2024-03-04T09:30:29.962 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2024-03-04T09:30:29.963 INFO:teuthology.orchestra.run.smithi006.stdout: debug mgr = 20 2024-03-04T09:30:29.963 INFO:teuthology.orchestra.run.smithi006.stdout: debug mon = 20 2024-03-04T09:30:29.963 INFO:teuthology.orchestra.run.smithi006.stdout: debug auth = 20 2024-03-04T09:30:29.963 INFO:teuthology.orchestra.run.smithi006.stdout: mon reweight min pgs per osd = 4 2024-03-04T09:30:29.963 INFO:teuthology.orchestra.run.smithi006.stdout: mon reweight min bytes per osd = 10 2024-03-04T09:30:29.963 INFO:teuthology.orchestra.run.smithi006.stdout: mgr/telemetry/nag = false 2024-03-04T09:30:29.963 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:30:29.963 INFO:teuthology.orchestra.run.smithi006.stdout:[mon] 2024-03-04T09:30:29.963 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2024-03-04T09:30:29.963 INFO:teuthology.orchestra.run.smithi006.stdout: debug mon = 20 2024-03-04T09:30:29.963 INFO:teuthology.orchestra.run.smithi006.stdout: debug paxos = 20 2024-03-04T09:30:29.963 INFO:teuthology.orchestra.run.smithi006.stdout: debug auth = 20 2024-03-04T09:30:29.963 INFO:teuthology.orchestra.run.smithi006.stdout: mon data avail warn = 5 2024-03-04T09:30:29.964 INFO:teuthology.orchestra.run.smithi006.stdout: mon mgr mkfs grace = 240 2024-03-04T09:30:29.964 INFO:teuthology.orchestra.run.smithi006.stdout: mon reweight min pgs per osd = 4 2024-03-04T09:30:29.964 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd reporter subtree level = osd 2024-03-04T09:30:29.964 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd prime pg temp = true 2024-03-04T09:30:29.964 INFO:teuthology.orchestra.run.smithi006.stdout: mon reweight min bytes per osd = 10 2024-03-04T09:30:29.964 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:30:29.964 INFO:teuthology.orchestra.run.smithi006.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-04T09:30:29.964 INFO:teuthology.orchestra.run.smithi006.stdout: auth mon ticket ttl = 660 # 11m 2024-03-04T09:30:29.964 INFO:teuthology.orchestra.run.smithi006.stdout: auth service ticket ttl = 240 # 4m 2024-03-04T09:30:29.964 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:30:29.964 INFO:teuthology.orchestra.run.smithi006.stdout: # don't complain about insecure global_id in the test suite 2024-03-04T09:30:29.964 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-04T09:30:29.965 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-04T09:30:29.965 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:30:29.965 INFO:teuthology.orchestra.run.smithi006.stdout: # 1m isn't quite enough 2024-03-04T09:30:29.965 INFO:teuthology.orchestra.run.smithi006.stdout: mon_down_mkfs_grace = 2m 2024-03-04T09:30:29.965 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:30:29.965 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_filestore_osds = false 2024-03-04T09:30:29.965 INFO:teuthology.orchestra.run.smithi006.stdout: mon op complaint time = 120 2024-03-04T09:30:29.965 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:30:29.965 INFO:teuthology.orchestra.run.smithi006.stdout:[client] 2024-03-04T09:30:29.965 INFO:teuthology.orchestra.run.smithi006.stdout: rgw cache enabled = true 2024-03-04T09:30:29.965 INFO:teuthology.orchestra.run.smithi006.stdout: rgw enable ops log = true 2024-03-04T09:30:29.965 INFO:teuthology.orchestra.run.smithi006.stdout: rgw enable usage log = true 2024-03-04T09:30:29.966 INFO:teuthology.orchestra.run.smithi006.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-04T09:30:29.966 INFO:teuthology.orchestra.run.smithi006.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-04T09:30:29.966 INFO:teuthology.orchestra.run.smithi006.stdout: client mount timeout = 600 2024-03-04T09:30:29.966 INFO:teuthology.orchestra.run.smithi006.stdout: debug client = 20 2024-03-04T09:30:29.966 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2024-03-04T09:30:29.966 INFO:teuthology.orchestra.run.smithi006.stdout: pid file = "" 2024-03-04T09:30:29.966 INFO:teuthology.orchestra.run.smithi006.stdout: rados mon op timeout = 900 2024-03-04T09:30:29.966 INFO:teuthology.orchestra.run.smithi006.stdout: rados osd op timeout = 900 2024-03-04T09:30:29.966 INFO:teuthology.orchestra.run.smithi006.stdout:[mon.a] 2024-03-04T09:30:29.966 INFO:teuthology.orchestra.run.smithi006.stdout:[mon.b] 2024-03-04T09:30:29.966 INFO:teuthology.orchestra.run.smithi006.stdout:[mon.c] 2024-03-04T09:30:29.967 INFO:teuthology.orchestra.run.smithi006.stdout:[mds.a] 2024-03-04T09:30:29.967 INFO:teuthology.orchestra.run.smithi006.stdout:[mds.c] 2024-03-04T09:30:29.967 INFO:teuthology.orchestra.run.smithi006.stdout:[mds.b] 2024-03-04T09:30:29.967 INFO:teuthology.orchestra.run.smithi006.stdout:[mds.d] 2024-03-04T09:30:29.967 INFO:teuthology.orchestra.run.smithi006.stdout:[mds] 2024-03-04T09:30:29.967 INFO:teuthology.orchestra.run.smithi006.stdout: debug mds = 20 2024-03-04T09:30:29.967 INFO:teuthology.orchestra.run.smithi006.stdout: debug mds balancer = 20 2024-03-04T09:30:29.967 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2024-03-04T09:30:29.967 INFO:teuthology.orchestra.run.smithi006.stdout: mds debug frag = True 2024-03-04T09:30:29.967 INFO:teuthology.orchestra.run.smithi006.stdout: mds debug scatterstat = True 2024-03-04T09:30:29.967 INFO:teuthology.orchestra.run.smithi006.stdout: mds op complaint time = 180 2024-03-04T09:30:29.967 INFO:teuthology.orchestra.run.smithi006.stdout: mds verify scatter = True 2024-03-04T09:30:29.967 INFO:teuthology.orchestra.run.smithi006.stdout: osd op complaint time = 180 2024-03-04T09:30:29.968 INFO:teuthology.orchestra.run.smithi006.stdout: rados mon op timeout = 900 2024-03-04T09:30:29.968 INFO:teuthology.orchestra.run.smithi006.stdout: rados osd op timeout = 900 2024-03-04T09:30:29.968 INFO:teuthology.orchestra.run.smithi006.stdout: mds_max_purge_files = 6 2024-03-04T09:30:29.968 INFO:teuthology.orchestra.run.smithi146.stdout:[global] 2024-03-04T09:30:29.968 INFO:teuthology.orchestra.run.smithi146.stdout: chdir = "" 2024-03-04T09:30:29.968 INFO:teuthology.orchestra.run.smithi146.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-04T09:30:29.968 INFO:teuthology.orchestra.run.smithi146.stdout: auth supported = cephx 2024-03-04T09:30:29.969 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:30:29.969 INFO:teuthology.orchestra.run.smithi146.stdout: filestore xattr use omap = true 2024-03-04T09:30:29.969 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:30:29.969 INFO:teuthology.orchestra.run.smithi146.stdout: mon clock drift allowed = 1.000 2024-03-04T09:30:29.969 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:30:29.969 INFO:teuthology.orchestra.run.smithi146.stdout: osd crush chooseleaf type = 0 2024-03-04T09:30:29.969 INFO:teuthology.orchestra.run.smithi146.stdout: auth debug = true 2024-03-04T09:30:29.969 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:30:29.969 INFO:teuthology.orchestra.run.smithi146.stdout: ms die on old message = true 2024-03-04T09:30:29.969 INFO:teuthology.orchestra.run.smithi146.stdout: ms die on bug = true 2024-03-04T09:30:29.969 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:30:29.969 INFO:teuthology.orchestra.run.smithi146.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-04T09:30:29.970 INFO:teuthology.orchestra.run.smithi146.stdout: mon pg warn max object skew = 0 2024-03-04T09:30:29.970 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:30:29.970 INFO:teuthology.orchestra.run.smithi146.stdout: # disable pg_autoscaler by default for new pools 2024-03-04T09:30:29.970 INFO:teuthology.orchestra.run.smithi146.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-04T09:30:29.970 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:30:29.970 INFO:teuthology.orchestra.run.smithi146.stdout: osd pool default size = 2 2024-03-04T09:30:29.970 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:30:29.970 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd allow primary affinity = true 2024-03-04T09:30:29.970 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd allow pg remap = true 2024-03-04T09:30:29.970 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on legacy crush tunables = false 2024-03-04T09:30:29.970 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on crush straw calc version zero = false 2024-03-04T09:30:29.970 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on no sortbitwise = false 2024-03-04T09:30:29.970 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on osd down out interval zero = false 2024-03-04T09:30:29.971 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on too few osds = false 2024-03-04T09:30:29.971 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-04T09:30:29.971 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-04T09:30:29.971 INFO:teuthology.orchestra.run.smithi146.stdout: mon_allow_pool_size_one = true 2024-03-04T09:30:29.971 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:30:29.971 INFO:teuthology.orchestra.run.smithi146.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-04T09:30:29.971 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:30:29.971 INFO:teuthology.orchestra.run.smithi146.stdout: osd default data pool replay window = 5 2024-03-04T09:30:29.971 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:30:29.971 INFO:teuthology.orchestra.run.smithi146.stdout: mon allow pool delete = true 2024-03-04T09:30:29.971 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:30:29.971 INFO:teuthology.orchestra.run.smithi146.stdout: mon cluster log file level = debug 2024-03-04T09:30:29.971 INFO:teuthology.orchestra.run.smithi146.stdout: debug asserts on shutdown = true 2024-03-04T09:30:29.972 INFO:teuthology.orchestra.run.smithi146.stdout: mon health detail to clog = false 2024-03-04T09:30:29.972 INFO:teuthology.orchestra.run.smithi146.stdout: mon host = "172.21.15.6,172.21.15.146,[v2:172.21.15.146:3301,v1:172.21.15.146:6790]" 2024-03-04T09:30:29.972 INFO:teuthology.orchestra.run.smithi146.stdout: ms die on skipped message = False 2024-03-04T09:30:29.972 INFO:teuthology.orchestra.run.smithi146.stdout: fsid = edce0578-5ae4-4402-b45a-552bdef84d7c 2024-03-04T09:30:29.972 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:30:29.972 INFO:teuthology.orchestra.run.smithi146.stdout:[osd] 2024-03-04T09:30:29.972 INFO:teuthology.orchestra.run.smithi146.stdout: osd journal size = 100 2024-03-04T09:30:29.972 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:30:29.972 INFO:teuthology.orchestra.run.smithi146.stdout: osd scrub load threshold = 5.0 2024-03-04T09:30:29.972 INFO:teuthology.orchestra.run.smithi146.stdout: osd scrub max interval = 600 2024-03-04T09:30:29.972 INFO:teuthology.orchestra.run.smithi146.stdout: osd mclock profile = high_recovery_ops 2024-03-04T09:30:29.972 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:30:29.972 INFO:teuthology.orchestra.run.smithi146.stdout: osd recover clone overlap = true 2024-03-04T09:30:29.972 INFO:teuthology.orchestra.run.smithi146.stdout: osd recovery max chunk = 1048576 2024-03-04T09:30:29.973 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:30:29.973 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug shutdown = true 2024-03-04T09:30:29.973 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug op order = true 2024-03-04T09:30:29.973 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug verify stray on activate = true 2024-03-04T09:30:29.973 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug trim objects = true 2024-03-04T09:30:29.973 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:30:29.973 INFO:teuthology.orchestra.run.smithi146.stdout: osd open classes on start = true 2024-03-04T09:30:29.973 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug pg log writeout = true 2024-03-04T09:30:29.973 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:30:29.973 INFO:teuthology.orchestra.run.smithi146.stdout: osd deep scrub update digest min age = 30 2024-03-04T09:30:29.973 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:30:29.973 INFO:teuthology.orchestra.run.smithi146.stdout: osd map max advance = 10 2024-03-04T09:30:29.973 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:30:29.973 INFO:teuthology.orchestra.run.smithi146.stdout: journal zero on create = true 2024-03-04T09:30:29.974 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:30:29.974 INFO:teuthology.orchestra.run.smithi146.stdout: filestore ondisk finisher threads = 3 2024-03-04T09:30:29.974 INFO:teuthology.orchestra.run.smithi146.stdout: filestore apply finisher threads = 3 2024-03-04T09:30:29.974 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:30:29.974 INFO:teuthology.orchestra.run.smithi146.stdout: bdev debug aio = true 2024-03-04T09:30:29.974 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug misdirected ops = true 2024-03-04T09:30:29.974 INFO:teuthology.orchestra.run.smithi146.stdout: bdev async discard = True 2024-03-04T09:30:29.974 INFO:teuthology.orchestra.run.smithi146.stdout: bdev enable discard = True 2024-03-04T09:30:29.974 INFO:teuthology.orchestra.run.smithi146.stdout: bluestore allocator = bitmap 2024-03-04T09:30:29.974 INFO:teuthology.orchestra.run.smithi146.stdout: bluestore block size = 96636764160 2024-03-04T09:30:29.974 INFO:teuthology.orchestra.run.smithi146.stdout: bluestore fsck on mount = True 2024-03-04T09:30:29.974 INFO:teuthology.orchestra.run.smithi146.stdout: debug bluefs = 1/20 2024-03-04T09:30:29.974 INFO:teuthology.orchestra.run.smithi146.stdout: debug bluestore = 1/20 2024-03-04T09:30:29.974 INFO:teuthology.orchestra.run.smithi146.stdout: debug ms = 1 2024-03-04T09:30:29.975 INFO:teuthology.orchestra.run.smithi146.stdout: debug osd = 20 2024-03-04T09:30:29.975 INFO:teuthology.orchestra.run.smithi146.stdout: debug rocksdb = 4/10 2024-03-04T09:30:29.975 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd backfillfull_ratio = 0.85 2024-03-04T09:30:29.975 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd full ratio = 0.9 2024-03-04T09:30:29.975 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd nearfull ratio = 0.8 2024-03-04T09:30:29.975 INFO:teuthology.orchestra.run.smithi146.stdout: osd failsafe full ratio = 0.95 2024-03-04T09:30:29.975 INFO:teuthology.orchestra.run.smithi146.stdout: osd objectstore = bluestore 2024-03-04T09:30:29.975 INFO:teuthology.orchestra.run.smithi146.stdout: osd op complaint time = 180 2024-03-04T09:30:29.975 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:30:29.975 INFO:teuthology.orchestra.run.smithi146.stdout:[mgr] 2024-03-04T09:30:29.975 INFO:teuthology.orchestra.run.smithi146.stdout: debug ms = 1 2024-03-04T09:30:29.975 INFO:teuthology.orchestra.run.smithi146.stdout: debug mgr = 20 2024-03-04T09:30:29.975 INFO:teuthology.orchestra.run.smithi146.stdout: debug mon = 20 2024-03-04T09:30:29.976 INFO:teuthology.orchestra.run.smithi146.stdout: debug auth = 20 2024-03-04T09:30:29.976 INFO:teuthology.orchestra.run.smithi146.stdout: mon reweight min pgs per osd = 4 2024-03-04T09:30:29.976 INFO:teuthology.orchestra.run.smithi146.stdout: mon reweight min bytes per osd = 10 2024-03-04T09:30:29.976 INFO:teuthology.orchestra.run.smithi146.stdout: mgr/telemetry/nag = false 2024-03-04T09:30:29.976 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:30:29.976 INFO:teuthology.orchestra.run.smithi146.stdout:[mon] 2024-03-04T09:30:29.976 INFO:teuthology.orchestra.run.smithi146.stdout: debug ms = 1 2024-03-04T09:30:29.976 INFO:teuthology.orchestra.run.smithi146.stdout: debug mon = 20 2024-03-04T09:30:29.976 INFO:teuthology.orchestra.run.smithi146.stdout: debug paxos = 20 2024-03-04T09:30:29.976 INFO:teuthology.orchestra.run.smithi146.stdout: debug auth = 20 2024-03-04T09:30:29.976 INFO:teuthology.orchestra.run.smithi146.stdout: mon data avail warn = 5 2024-03-04T09:30:29.976 INFO:teuthology.orchestra.run.smithi146.stdout: mon mgr mkfs grace = 240 2024-03-04T09:30:29.976 INFO:teuthology.orchestra.run.smithi146.stdout: mon reweight min pgs per osd = 4 2024-03-04T09:30:29.977 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd reporter subtree level = osd 2024-03-04T09:30:29.977 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd prime pg temp = true 2024-03-04T09:30:29.977 INFO:teuthology.orchestra.run.smithi146.stdout: mon reweight min bytes per osd = 10 2024-03-04T09:30:29.977 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:30:29.977 INFO:teuthology.orchestra.run.smithi146.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-04T09:30:29.977 INFO:teuthology.orchestra.run.smithi146.stdout: auth mon ticket ttl = 660 # 11m 2024-03-04T09:30:29.977 INFO:teuthology.orchestra.run.smithi146.stdout: auth service ticket ttl = 240 # 4m 2024-03-04T09:30:29.977 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:30:29.977 INFO:teuthology.orchestra.run.smithi146.stdout: # don't complain about insecure global_id in the test suite 2024-03-04T09:30:29.977 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-04T09:30:29.977 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-04T09:30:29.977 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:30:29.977 INFO:teuthology.orchestra.run.smithi146.stdout: # 1m isn't quite enough 2024-03-04T09:30:29.978 INFO:teuthology.orchestra.run.smithi146.stdout: mon_down_mkfs_grace = 2m 2024-03-04T09:30:29.978 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:30:29.978 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_filestore_osds = false 2024-03-04T09:30:29.978 INFO:teuthology.orchestra.run.smithi146.stdout: mon op complaint time = 120 2024-03-04T09:30:29.978 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:30:29.978 INFO:teuthology.orchestra.run.smithi146.stdout:[client] 2024-03-04T09:30:29.978 INFO:teuthology.orchestra.run.smithi146.stdout: rgw cache enabled = true 2024-03-04T09:30:29.978 INFO:teuthology.orchestra.run.smithi146.stdout: rgw enable ops log = true 2024-03-04T09:30:29.978 INFO:teuthology.orchestra.run.smithi146.stdout: rgw enable usage log = true 2024-03-04T09:30:29.978 INFO:teuthology.orchestra.run.smithi146.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-04T09:30:29.978 INFO:teuthology.orchestra.run.smithi146.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-04T09:30:29.978 INFO:teuthology.orchestra.run.smithi146.stdout: client mount timeout = 600 2024-03-04T09:30:29.978 INFO:teuthology.orchestra.run.smithi146.stdout: debug client = 20 2024-03-04T09:30:29.978 INFO:teuthology.orchestra.run.smithi146.stdout: debug ms = 1 2024-03-04T09:30:29.979 INFO:teuthology.orchestra.run.smithi146.stdout: pid file = "" 2024-03-04T09:30:29.979 INFO:teuthology.orchestra.run.smithi146.stdout: rados mon op timeout = 900 2024-03-04T09:30:29.979 INFO:teuthology.orchestra.run.smithi146.stdout: rados osd op timeout = 900 2024-03-04T09:30:29.979 INFO:teuthology.orchestra.run.smithi146.stdout:[mon.a] 2024-03-04T09:30:29.979 INFO:teuthology.orchestra.run.smithi146.stdout:[mon.b] 2024-03-04T09:30:29.979 INFO:teuthology.orchestra.run.smithi146.stdout:[mon.c] 2024-03-04T09:30:29.979 INFO:teuthology.orchestra.run.smithi146.stdout:[mds.a] 2024-03-04T09:30:29.979 INFO:teuthology.orchestra.run.smithi146.stdout:[mds.c] 2024-03-04T09:30:29.979 INFO:teuthology.orchestra.run.smithi146.stdout:[mds.b] 2024-03-04T09:30:29.979 INFO:teuthology.orchestra.run.smithi146.stdout:[mds.d] 2024-03-04T09:30:29.979 INFO:teuthology.orchestra.run.smithi146.stdout:[mds] 2024-03-04T09:30:29.979 INFO:teuthology.orchestra.run.smithi146.stdout: debug mds = 20 2024-03-04T09:30:29.979 INFO:teuthology.orchestra.run.smithi146.stdout: debug mds balancer = 20 2024-03-04T09:30:29.980 INFO:teuthology.orchestra.run.smithi146.stdout: debug ms = 1 2024-03-04T09:30:29.980 INFO:teuthology.orchestra.run.smithi146.stdout: mds debug frag = True 2024-03-04T09:30:29.980 INFO:teuthology.orchestra.run.smithi146.stdout: mds debug scatterstat = True 2024-03-04T09:30:29.980 INFO:teuthology.orchestra.run.smithi146.stdout: mds op complaint time = 180 2024-03-04T09:30:29.980 INFO:teuthology.orchestra.run.smithi146.stdout: mds verify scatter = True 2024-03-04T09:30:29.980 INFO:teuthology.orchestra.run.smithi146.stdout: osd op complaint time = 180 2024-03-04T09:30:29.980 INFO:teuthology.orchestra.run.smithi146.stdout: rados mon op timeout = 900 2024-03-04T09:30:29.980 INFO:teuthology.orchestra.run.smithi146.stdout: rados osd op timeout = 900 2024-03-04T09:30:29.980 INFO:teuthology.orchestra.run.smithi146.stdout: mds_max_purge_files = 6 2024-03-04T09:30:30.007 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-04T09:30:30.015 DEBUG:teuthology.orchestra.run.smithi146:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-04T09:30:30.179 INFO:teuthology.orchestra.run.smithi006.stdout:[global] 2024-03-04T09:30:30.180 INFO:teuthology.orchestra.run.smithi006.stdout: chdir = "" 2024-03-04T09:30:30.180 INFO:teuthology.orchestra.run.smithi006.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-04T09:30:30.180 INFO:teuthology.orchestra.run.smithi006.stdout: auth supported = cephx 2024-03-04T09:30:30.180 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:30:30.180 INFO:teuthology.orchestra.run.smithi006.stdout: filestore xattr use omap = true 2024-03-04T09:30:30.180 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:30:30.180 INFO:teuthology.orchestra.run.smithi006.stdout: mon clock drift allowed = 1.000 2024-03-04T09:30:30.180 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:30:30.180 INFO:teuthology.orchestra.run.smithi006.stdout: osd crush chooseleaf type = 0 2024-03-04T09:30:30.180 INFO:teuthology.orchestra.run.smithi006.stdout: auth debug = true 2024-03-04T09:30:30.180 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:30:30.180 INFO:teuthology.orchestra.run.smithi006.stdout: ms die on old message = true 2024-03-04T09:30:30.181 INFO:teuthology.orchestra.run.smithi006.stdout: ms die on bug = true 2024-03-04T09:30:30.181 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:30:30.181 INFO:teuthology.orchestra.run.smithi006.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-04T09:30:30.181 INFO:teuthology.orchestra.run.smithi006.stdout: mon pg warn max object skew = 0 2024-03-04T09:30:30.181 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:30:30.181 INFO:teuthology.orchestra.run.smithi006.stdout: # disable pg_autoscaler by default for new pools 2024-03-04T09:30:30.181 INFO:teuthology.orchestra.run.smithi006.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-04T09:30:30.181 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:30:30.181 INFO:teuthology.orchestra.run.smithi006.stdout: osd pool default size = 2 2024-03-04T09:30:30.181 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:30:30.181 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd allow primary affinity = true 2024-03-04T09:30:30.181 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd allow pg remap = true 2024-03-04T09:30:30.181 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on legacy crush tunables = false 2024-03-04T09:30:30.182 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on crush straw calc version zero = false 2024-03-04T09:30:30.182 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on no sortbitwise = false 2024-03-04T09:30:30.182 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on osd down out interval zero = false 2024-03-04T09:30:30.182 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on too few osds = false 2024-03-04T09:30:30.182 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-04T09:30:30.182 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-04T09:30:30.182 INFO:teuthology.orchestra.run.smithi006.stdout: mon_allow_pool_size_one = true 2024-03-04T09:30:30.182 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:30:30.182 INFO:teuthology.orchestra.run.smithi006.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-04T09:30:30.182 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:30:30.182 INFO:teuthology.orchestra.run.smithi006.stdout: osd default data pool replay window = 5 2024-03-04T09:30:30.182 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:30:30.182 INFO:teuthology.orchestra.run.smithi006.stdout: mon allow pool delete = true 2024-03-04T09:30:30.182 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:30:30.183 INFO:teuthology.orchestra.run.smithi006.stdout: mon cluster log file level = debug 2024-03-04T09:30:30.183 INFO:teuthology.orchestra.run.smithi006.stdout: debug asserts on shutdown = true 2024-03-04T09:30:30.183 INFO:teuthology.orchestra.run.smithi006.stdout: mon health detail to clog = false 2024-03-04T09:30:30.183 INFO:teuthology.orchestra.run.smithi006.stdout: mon host = "172.21.15.6,172.21.15.146,[v2:172.21.15.146:3301,v1:172.21.15.146:6790]" 2024-03-04T09:30:30.183 INFO:teuthology.orchestra.run.smithi006.stdout: ms die on skipped message = False 2024-03-04T09:30:30.183 INFO:teuthology.orchestra.run.smithi006.stdout: fsid = edce0578-5ae4-4402-b45a-552bdef84d7c 2024-03-04T09:30:30.183 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:30:30.183 INFO:teuthology.orchestra.run.smithi006.stdout:[osd] 2024-03-04T09:30:30.183 INFO:teuthology.orchestra.run.smithi006.stdout: osd journal size = 100 2024-03-04T09:30:30.183 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:30:30.183 INFO:teuthology.orchestra.run.smithi006.stdout: osd scrub load threshold = 5.0 2024-03-04T09:30:30.183 INFO:teuthology.orchestra.run.smithi006.stdout: osd scrub max interval = 600 2024-03-04T09:30:30.183 INFO:teuthology.orchestra.run.smithi006.stdout: osd mclock profile = high_recovery_ops 2024-03-04T09:30:30.183 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:30:30.184 INFO:teuthology.orchestra.run.smithi006.stdout: osd recover clone overlap = true 2024-03-04T09:30:30.184 INFO:teuthology.orchestra.run.smithi006.stdout: osd recovery max chunk = 1048576 2024-03-04T09:30:30.184 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:30:30.184 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug shutdown = true 2024-03-04T09:30:30.184 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug op order = true 2024-03-04T09:30:30.184 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug verify stray on activate = true 2024-03-04T09:30:30.184 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug trim objects = true 2024-03-04T09:30:30.184 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:30:30.184 INFO:teuthology.orchestra.run.smithi006.stdout: osd open classes on start = true 2024-03-04T09:30:30.184 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug pg log writeout = true 2024-03-04T09:30:30.184 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:30:30.184 INFO:teuthology.orchestra.run.smithi006.stdout: osd deep scrub update digest min age = 30 2024-03-04T09:30:30.185 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:30:30.185 INFO:teuthology.orchestra.run.smithi006.stdout: osd map max advance = 10 2024-03-04T09:30:30.185 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:30:30.185 INFO:teuthology.orchestra.run.smithi006.stdout: journal zero on create = true 2024-03-04T09:30:30.185 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:30:30.185 INFO:teuthology.orchestra.run.smithi006.stdout: filestore ondisk finisher threads = 3 2024-03-04T09:30:30.185 INFO:teuthology.orchestra.run.smithi006.stdout: filestore apply finisher threads = 3 2024-03-04T09:30:30.185 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:30:30.185 INFO:teuthology.orchestra.run.smithi006.stdout: bdev debug aio = true 2024-03-04T09:30:30.185 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug misdirected ops = true 2024-03-04T09:30:30.185 INFO:teuthology.orchestra.run.smithi006.stdout: bdev async discard = True 2024-03-04T09:30:30.185 INFO:teuthology.orchestra.run.smithi006.stdout: bdev enable discard = True 2024-03-04T09:30:30.185 INFO:teuthology.orchestra.run.smithi006.stdout: bluestore allocator = bitmap 2024-03-04T09:30:30.185 INFO:teuthology.orchestra.run.smithi006.stdout: bluestore block size = 96636764160 2024-03-04T09:30:30.186 INFO:teuthology.orchestra.run.smithi006.stdout: bluestore fsck on mount = True 2024-03-04T09:30:30.186 INFO:teuthology.orchestra.run.smithi006.stdout: debug bluefs = 1/20 2024-03-04T09:30:30.186 INFO:teuthology.orchestra.run.smithi006.stdout: debug bluestore = 1/20 2024-03-04T09:30:30.186 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2024-03-04T09:30:30.186 INFO:teuthology.orchestra.run.smithi006.stdout: debug osd = 20 2024-03-04T09:30:30.186 INFO:teuthology.orchestra.run.smithi006.stdout: debug rocksdb = 4/10 2024-03-04T09:30:30.186 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd backfillfull_ratio = 0.85 2024-03-04T09:30:30.186 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd full ratio = 0.9 2024-03-04T09:30:30.186 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd nearfull ratio = 0.8 2024-03-04T09:30:30.186 INFO:teuthology.orchestra.run.smithi006.stdout: osd failsafe full ratio = 0.95 2024-03-04T09:30:30.186 INFO:teuthology.orchestra.run.smithi006.stdout: osd objectstore = bluestore 2024-03-04T09:30:30.186 INFO:teuthology.orchestra.run.smithi006.stdout: osd op complaint time = 180 2024-03-04T09:30:30.186 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:30:30.186 INFO:teuthology.orchestra.run.smithi006.stdout:[mgr] 2024-03-04T09:30:30.187 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2024-03-04T09:30:30.187 INFO:teuthology.orchestra.run.smithi006.stdout: debug mgr = 20 2024-03-04T09:30:30.187 INFO:teuthology.orchestra.run.smithi006.stdout: debug mon = 20 2024-03-04T09:30:30.187 INFO:teuthology.orchestra.run.smithi006.stdout: debug auth = 20 2024-03-04T09:30:30.187 INFO:teuthology.orchestra.run.smithi006.stdout: mon reweight min pgs per osd = 4 2024-03-04T09:30:30.187 INFO:teuthology.orchestra.run.smithi006.stdout: mon reweight min bytes per osd = 10 2024-03-04T09:30:30.187 INFO:teuthology.orchestra.run.smithi006.stdout: mgr/telemetry/nag = false 2024-03-04T09:30:30.187 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:30:30.187 INFO:teuthology.orchestra.run.smithi006.stdout:[mon] 2024-03-04T09:30:30.187 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2024-03-04T09:30:30.187 INFO:teuthology.orchestra.run.smithi006.stdout: debug mon = 20 2024-03-04T09:30:30.187 INFO:teuthology.orchestra.run.smithi006.stdout: debug paxos = 20 2024-03-04T09:30:30.187 INFO:teuthology.orchestra.run.smithi006.stdout: debug auth = 20 2024-03-04T09:30:30.187 INFO:teuthology.orchestra.run.smithi006.stdout: mon data avail warn = 5 2024-03-04T09:30:30.188 INFO:teuthology.orchestra.run.smithi006.stdout: mon mgr mkfs grace = 240 2024-03-04T09:30:30.188 INFO:teuthology.orchestra.run.smithi006.stdout: mon reweight min pgs per osd = 4 2024-03-04T09:30:30.188 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd reporter subtree level = osd 2024-03-04T09:30:30.188 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd prime pg temp = true 2024-03-04T09:30:30.188 INFO:teuthology.orchestra.run.smithi006.stdout: mon reweight min bytes per osd = 10 2024-03-04T09:30:30.188 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:30:30.188 INFO:teuthology.orchestra.run.smithi006.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-04T09:30:30.188 INFO:teuthology.orchestra.run.smithi006.stdout: auth mon ticket ttl = 660 # 11m 2024-03-04T09:30:30.188 INFO:teuthology.orchestra.run.smithi006.stdout: auth service ticket ttl = 240 # 4m 2024-03-04T09:30:30.189 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:30:30.189 INFO:teuthology.orchestra.run.smithi006.stdout: # don't complain about insecure global_id in the test suite 2024-03-04T09:30:30.189 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-04T09:30:30.189 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-04T09:30:30.189 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:30:30.189 INFO:teuthology.orchestra.run.smithi006.stdout: # 1m isn't quite enough 2024-03-04T09:30:30.189 INFO:teuthology.orchestra.run.smithi006.stdout: mon_down_mkfs_grace = 2m 2024-03-04T09:30:30.189 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:30:30.189 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_filestore_osds = false 2024-03-04T09:30:30.189 INFO:teuthology.orchestra.run.smithi006.stdout: mon op complaint time = 120 2024-03-04T09:30:30.189 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:30:30.189 INFO:teuthology.orchestra.run.smithi006.stdout:[client] 2024-03-04T09:30:30.189 INFO:teuthology.orchestra.run.smithi006.stdout: rgw cache enabled = true 2024-03-04T09:30:30.190 INFO:teuthology.orchestra.run.smithi006.stdout: rgw enable ops log = true 2024-03-04T09:30:30.190 INFO:teuthology.orchestra.run.smithi006.stdout: rgw enable usage log = true 2024-03-04T09:30:30.190 INFO:teuthology.orchestra.run.smithi006.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-04T09:30:30.190 INFO:teuthology.orchestra.run.smithi006.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-04T09:30:30.190 INFO:teuthology.orchestra.run.smithi006.stdout: client mount timeout = 600 2024-03-04T09:30:30.190 INFO:teuthology.orchestra.run.smithi006.stdout: debug client = 20 2024-03-04T09:30:30.190 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2024-03-04T09:30:30.190 INFO:teuthology.orchestra.run.smithi006.stdout: pid file = "" 2024-03-04T09:30:30.190 INFO:teuthology.orchestra.run.smithi006.stdout: rados mon op timeout = 900 2024-03-04T09:30:30.190 INFO:teuthology.orchestra.run.smithi006.stdout: rados osd op timeout = 900 2024-03-04T09:30:30.190 INFO:teuthology.orchestra.run.smithi006.stdout:[mon.a] 2024-03-04T09:30:30.190 INFO:teuthology.orchestra.run.smithi006.stdout:[mon.b] 2024-03-04T09:30:30.190 INFO:teuthology.orchestra.run.smithi006.stdout:[mon.c] 2024-03-04T09:30:30.191 INFO:teuthology.orchestra.run.smithi006.stdout:[mds.a] 2024-03-04T09:30:30.191 INFO:teuthology.orchestra.run.smithi006.stdout:[mds.c] 2024-03-04T09:30:30.191 INFO:teuthology.orchestra.run.smithi006.stdout:[mds.b] 2024-03-04T09:30:30.191 INFO:teuthology.orchestra.run.smithi006.stdout:[mds.d] 2024-03-04T09:30:30.191 INFO:teuthology.orchestra.run.smithi006.stdout:[mds] 2024-03-04T09:30:30.191 INFO:teuthology.orchestra.run.smithi006.stdout: debug mds = 20 2024-03-04T09:30:30.191 INFO:teuthology.orchestra.run.smithi006.stdout: debug mds balancer = 20 2024-03-04T09:30:30.191 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2024-03-04T09:30:30.191 INFO:teuthology.orchestra.run.smithi006.stdout: mds debug frag = True 2024-03-04T09:30:30.191 INFO:teuthology.orchestra.run.smithi006.stdout: mds debug scatterstat = True 2024-03-04T09:30:30.191 INFO:teuthology.orchestra.run.smithi006.stdout: mds op complaint time = 180 2024-03-04T09:30:30.191 INFO:teuthology.orchestra.run.smithi006.stdout: mds verify scatter = True 2024-03-04T09:30:30.191 INFO:teuthology.orchestra.run.smithi006.stdout: osd op complaint time = 180 2024-03-04T09:30:30.191 INFO:teuthology.orchestra.run.smithi006.stdout: rados mon op timeout = 900 2024-03-04T09:30:30.192 INFO:teuthology.orchestra.run.smithi006.stdout: rados osd op timeout = 900 2024-03-04T09:30:30.192 INFO:teuthology.orchestra.run.smithi006.stdout: mds_max_purge_files = 6 2024-03-04T09:30:30.192 INFO:teuthology.orchestra.run.smithi006.stdout: mds_max_purge_ops = 100000000 2024-03-04T09:30:30.192 INFO:teuthology.orchestra.run.smithi146.stdout:[global] 2024-03-04T09:30:30.192 INFO:teuthology.orchestra.run.smithi146.stdout: chdir = "" 2024-03-04T09:30:30.192 INFO:teuthology.orchestra.run.smithi146.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-04T09:30:30.193 INFO:teuthology.orchestra.run.smithi146.stdout: auth supported = cephx 2024-03-04T09:30:30.193 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:30:30.193 INFO:teuthology.orchestra.run.smithi146.stdout: filestore xattr use omap = true 2024-03-04T09:30:30.193 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:30:30.193 INFO:teuthology.orchestra.run.smithi146.stdout: mon clock drift allowed = 1.000 2024-03-04T09:30:30.193 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:30:30.193 INFO:teuthology.orchestra.run.smithi146.stdout: osd crush chooseleaf type = 0 2024-03-04T09:30:30.193 INFO:teuthology.orchestra.run.smithi146.stdout: auth debug = true 2024-03-04T09:30:30.193 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:30:30.193 INFO:teuthology.orchestra.run.smithi146.stdout: ms die on old message = true 2024-03-04T09:30:30.193 INFO:teuthology.orchestra.run.smithi146.stdout: ms die on bug = true 2024-03-04T09:30:30.193 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:30:30.193 INFO:teuthology.orchestra.run.smithi146.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-04T09:30:30.193 INFO:teuthology.orchestra.run.smithi146.stdout: mon pg warn max object skew = 0 2024-03-04T09:30:30.194 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:30:30.194 INFO:teuthology.orchestra.run.smithi146.stdout: # disable pg_autoscaler by default for new pools 2024-03-04T09:30:30.194 INFO:teuthology.orchestra.run.smithi146.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-04T09:30:30.194 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:30:30.194 INFO:teuthology.orchestra.run.smithi146.stdout: osd pool default size = 2 2024-03-04T09:30:30.194 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:30:30.194 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd allow primary affinity = true 2024-03-04T09:30:30.194 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd allow pg remap = true 2024-03-04T09:30:30.194 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on legacy crush tunables = false 2024-03-04T09:30:30.194 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on crush straw calc version zero = false 2024-03-04T09:30:30.194 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on no sortbitwise = false 2024-03-04T09:30:30.194 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on osd down out interval zero = false 2024-03-04T09:30:30.194 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on too few osds = false 2024-03-04T09:30:30.194 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-04T09:30:30.195 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-04T09:30:30.195 INFO:teuthology.orchestra.run.smithi146.stdout: mon_allow_pool_size_one = true 2024-03-04T09:30:30.195 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:30:30.195 INFO:teuthology.orchestra.run.smithi146.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-04T09:30:30.195 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:30:30.195 INFO:teuthology.orchestra.run.smithi146.stdout: osd default data pool replay window = 5 2024-03-04T09:30:30.195 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:30:30.195 INFO:teuthology.orchestra.run.smithi146.stdout: mon allow pool delete = true 2024-03-04T09:30:30.195 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:30:30.195 INFO:teuthology.orchestra.run.smithi146.stdout: mon cluster log file level = debug 2024-03-04T09:30:30.195 INFO:teuthology.orchestra.run.smithi146.stdout: debug asserts on shutdown = true 2024-03-04T09:30:30.195 INFO:teuthology.orchestra.run.smithi146.stdout: mon health detail to clog = false 2024-03-04T09:30:30.195 INFO:teuthology.orchestra.run.smithi146.stdout: mon host = "172.21.15.6,172.21.15.146,[v2:172.21.15.146:3301,v1:172.21.15.146:6790]" 2024-03-04T09:30:30.195 INFO:teuthology.orchestra.run.smithi146.stdout: ms die on skipped message = False 2024-03-04T09:30:30.195 INFO:teuthology.orchestra.run.smithi146.stdout: fsid = edce0578-5ae4-4402-b45a-552bdef84d7c 2024-03-04T09:30:30.196 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:30:30.196 INFO:teuthology.orchestra.run.smithi146.stdout:[osd] 2024-03-04T09:30:30.196 INFO:teuthology.orchestra.run.smithi146.stdout: osd journal size = 100 2024-03-04T09:30:30.196 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:30:30.196 INFO:teuthology.orchestra.run.smithi146.stdout: osd scrub load threshold = 5.0 2024-03-04T09:30:30.196 INFO:teuthology.orchestra.run.smithi146.stdout: osd scrub max interval = 600 2024-03-04T09:30:30.196 INFO:teuthology.orchestra.run.smithi146.stdout: osd mclock profile = high_recovery_ops 2024-03-04T09:30:30.196 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:30:30.196 INFO:teuthology.orchestra.run.smithi146.stdout: osd recover clone overlap = true 2024-03-04T09:30:30.196 INFO:teuthology.orchestra.run.smithi146.stdout: osd recovery max chunk = 1048576 2024-03-04T09:30:30.196 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:30:30.196 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug shutdown = true 2024-03-04T09:30:30.196 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug op order = true 2024-03-04T09:30:30.196 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug verify stray on activate = true 2024-03-04T09:30:30.197 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug trim objects = true 2024-03-04T09:30:30.197 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:30:30.197 INFO:teuthology.orchestra.run.smithi146.stdout: osd open classes on start = true 2024-03-04T09:30:30.197 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug pg log writeout = true 2024-03-04T09:30:30.197 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:30:30.197 INFO:teuthology.orchestra.run.smithi146.stdout: osd deep scrub update digest min age = 30 2024-03-04T09:30:30.197 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:30:30.197 INFO:teuthology.orchestra.run.smithi146.stdout: osd map max advance = 10 2024-03-04T09:30:30.197 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:30:30.197 INFO:teuthology.orchestra.run.smithi146.stdout: journal zero on create = true 2024-03-04T09:30:30.197 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:30:30.197 INFO:teuthology.orchestra.run.smithi146.stdout: filestore ondisk finisher threads = 3 2024-03-04T09:30:30.197 INFO:teuthology.orchestra.run.smithi146.stdout: filestore apply finisher threads = 3 2024-03-04T09:30:30.197 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:30:30.198 INFO:teuthology.orchestra.run.smithi146.stdout: bdev debug aio = true 2024-03-04T09:30:30.198 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug misdirected ops = true 2024-03-04T09:30:30.198 INFO:teuthology.orchestra.run.smithi146.stdout: bdev async discard = True 2024-03-04T09:30:30.198 INFO:teuthology.orchestra.run.smithi146.stdout: bdev enable discard = True 2024-03-04T09:30:30.198 INFO:teuthology.orchestra.run.smithi146.stdout: bluestore allocator = bitmap 2024-03-04T09:30:30.198 INFO:teuthology.orchestra.run.smithi146.stdout: bluestore block size = 96636764160 2024-03-04T09:30:30.198 INFO:teuthology.orchestra.run.smithi146.stdout: bluestore fsck on mount = True 2024-03-04T09:30:30.198 INFO:teuthology.orchestra.run.smithi146.stdout: debug bluefs = 1/20 2024-03-04T09:30:30.198 INFO:teuthology.orchestra.run.smithi146.stdout: debug bluestore = 1/20 2024-03-04T09:30:30.198 INFO:teuthology.orchestra.run.smithi146.stdout: debug ms = 1 2024-03-04T09:30:30.198 INFO:teuthology.orchestra.run.smithi146.stdout: debug osd = 20 2024-03-04T09:30:30.198 INFO:teuthology.orchestra.run.smithi146.stdout: debug rocksdb = 4/10 2024-03-04T09:30:30.198 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd backfillfull_ratio = 0.85 2024-03-04T09:30:30.198 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd full ratio = 0.9 2024-03-04T09:30:30.198 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd nearfull ratio = 0.8 2024-03-04T09:30:30.199 INFO:teuthology.orchestra.run.smithi146.stdout: osd failsafe full ratio = 0.95 2024-03-04T09:30:30.199 INFO:teuthology.orchestra.run.smithi146.stdout: osd objectstore = bluestore 2024-03-04T09:30:30.199 INFO:teuthology.orchestra.run.smithi146.stdout: osd op complaint time = 180 2024-03-04T09:30:30.199 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:30:30.199 INFO:teuthology.orchestra.run.smithi146.stdout:[mgr] 2024-03-04T09:30:30.199 INFO:teuthology.orchestra.run.smithi146.stdout: debug ms = 1 2024-03-04T09:30:30.199 INFO:teuthology.orchestra.run.smithi146.stdout: debug mgr = 20 2024-03-04T09:30:30.199 INFO:teuthology.orchestra.run.smithi146.stdout: debug mon = 20 2024-03-04T09:30:30.199 INFO:teuthology.orchestra.run.smithi146.stdout: debug auth = 20 2024-03-04T09:30:30.199 INFO:teuthology.orchestra.run.smithi146.stdout: mon reweight min pgs per osd = 4 2024-03-04T09:30:30.199 INFO:teuthology.orchestra.run.smithi146.stdout: mon reweight min bytes per osd = 10 2024-03-04T09:30:30.199 INFO:teuthology.orchestra.run.smithi146.stdout: mgr/telemetry/nag = false 2024-03-04T09:30:30.199 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:30:30.199 INFO:teuthology.orchestra.run.smithi146.stdout:[mon] 2024-03-04T09:30:30.199 INFO:teuthology.orchestra.run.smithi146.stdout: debug ms = 1 2024-03-04T09:30:30.200 INFO:teuthology.orchestra.run.smithi146.stdout: debug mon = 20 2024-03-04T09:30:30.200 INFO:teuthology.orchestra.run.smithi146.stdout: debug paxos = 20 2024-03-04T09:30:30.200 INFO:teuthology.orchestra.run.smithi146.stdout: debug auth = 20 2024-03-04T09:30:30.200 INFO:teuthology.orchestra.run.smithi146.stdout: mon data avail warn = 5 2024-03-04T09:30:30.200 INFO:teuthology.orchestra.run.smithi146.stdout: mon mgr mkfs grace = 240 2024-03-04T09:30:30.200 INFO:teuthology.orchestra.run.smithi146.stdout: mon reweight min pgs per osd = 4 2024-03-04T09:30:30.200 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd reporter subtree level = osd 2024-03-04T09:30:30.200 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd prime pg temp = true 2024-03-04T09:30:30.200 INFO:teuthology.orchestra.run.smithi146.stdout: mon reweight min bytes per osd = 10 2024-03-04T09:30:30.200 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:30:30.200 INFO:teuthology.orchestra.run.smithi146.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-04T09:30:30.200 INFO:teuthology.orchestra.run.smithi146.stdout: auth mon ticket ttl = 660 # 11m 2024-03-04T09:30:30.200 INFO:teuthology.orchestra.run.smithi146.stdout: auth service ticket ttl = 240 # 4m 2024-03-04T09:30:30.200 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:30:30.200 INFO:teuthology.orchestra.run.smithi146.stdout: # don't complain about insecure global_id in the test suite 2024-03-04T09:30:30.201 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-04T09:30:30.201 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-04T09:30:30.201 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:30:30.201 INFO:teuthology.orchestra.run.smithi146.stdout: # 1m isn't quite enough 2024-03-04T09:30:30.201 INFO:teuthology.orchestra.run.smithi146.stdout: mon_down_mkfs_grace = 2m 2024-03-04T09:30:30.201 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:30:30.201 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_filestore_osds = false 2024-03-04T09:30:30.201 INFO:teuthology.orchestra.run.smithi146.stdout: mon op complaint time = 120 2024-03-04T09:30:30.201 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:30:30.201 INFO:teuthology.orchestra.run.smithi146.stdout:[client] 2024-03-04T09:30:30.201 INFO:teuthology.orchestra.run.smithi146.stdout: rgw cache enabled = true 2024-03-04T09:30:30.201 INFO:teuthology.orchestra.run.smithi146.stdout: rgw enable ops log = true 2024-03-04T09:30:30.201 INFO:teuthology.orchestra.run.smithi146.stdout: rgw enable usage log = true 2024-03-04T09:30:30.201 INFO:teuthology.orchestra.run.smithi146.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-04T09:30:30.201 INFO:teuthology.orchestra.run.smithi146.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-04T09:30:30.201 INFO:teuthology.orchestra.run.smithi146.stdout: client mount timeout = 600 2024-03-04T09:30:30.202 INFO:teuthology.orchestra.run.smithi146.stdout: debug client = 20 2024-03-04T09:30:30.202 INFO:teuthology.orchestra.run.smithi146.stdout: debug ms = 1 2024-03-04T09:30:30.202 INFO:teuthology.orchestra.run.smithi146.stdout: pid file = "" 2024-03-04T09:30:30.202 INFO:teuthology.orchestra.run.smithi146.stdout: rados mon op timeout = 900 2024-03-04T09:30:30.202 INFO:teuthology.orchestra.run.smithi146.stdout: rados osd op timeout = 900 2024-03-04T09:30:30.202 INFO:teuthology.orchestra.run.smithi146.stdout:[mon.a] 2024-03-04T09:30:30.202 INFO:teuthology.orchestra.run.smithi146.stdout:[mon.b] 2024-03-04T09:30:30.202 INFO:teuthology.orchestra.run.smithi146.stdout:[mon.c] 2024-03-04T09:30:30.202 INFO:teuthology.orchestra.run.smithi146.stdout:[mds.a] 2024-03-04T09:30:30.202 INFO:teuthology.orchestra.run.smithi146.stdout:[mds.c] 2024-03-04T09:30:30.202 INFO:teuthology.orchestra.run.smithi146.stdout:[mds.b] 2024-03-04T09:30:30.202 INFO:teuthology.orchestra.run.smithi146.stdout:[mds.d] 2024-03-04T09:30:30.202 INFO:teuthology.orchestra.run.smithi146.stdout:[mds] 2024-03-04T09:30:30.202 INFO:teuthology.orchestra.run.smithi146.stdout: debug mds = 20 2024-03-04T09:30:30.202 INFO:teuthology.orchestra.run.smithi146.stdout: debug mds balancer = 20 2024-03-04T09:30:30.202 INFO:teuthology.orchestra.run.smithi146.stdout: debug ms = 1 2024-03-04T09:30:30.203 INFO:teuthology.orchestra.run.smithi146.stdout: mds debug frag = True 2024-03-04T09:30:30.203 INFO:teuthology.orchestra.run.smithi146.stdout: mds debug scatterstat = True 2024-03-04T09:30:30.203 INFO:teuthology.orchestra.run.smithi146.stdout: mds op complaint time = 180 2024-03-04T09:30:30.203 INFO:teuthology.orchestra.run.smithi146.stdout: mds verify scatter = True 2024-03-04T09:30:30.203 INFO:teuthology.orchestra.run.smithi146.stdout: osd op complaint time = 180 2024-03-04T09:30:30.203 INFO:teuthology.orchestra.run.smithi146.stdout: rados mon op timeout = 900 2024-03-04T09:30:30.203 INFO:teuthology.orchestra.run.smithi146.stdout: rados osd op timeout = 900 2024-03-04T09:30:30.203 INFO:teuthology.orchestra.run.smithi146.stdout: mds_max_purge_files = 6 2024-03-04T09:30:30.203 INFO:teuthology.orchestra.run.smithi146.stdout: mds_max_purge_ops = 100000000 2024-03-04T09:30:30.230 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-04T09:30:30.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:30.441+0000 7f71e5200640 1 Processor -- start 2024-03-04T09:30:30.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:30.442+0000 7f71e5200640 1 -- start start 2024-03-04T09:30:30.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:30.444+0000 7f71e5200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f71e00b5cf0 0x7f71e00ce2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:30.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:30.444+0000 7f71e5200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71e00afb10 con 0x7f71e015bed0 2024-03-04T09:30:30.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:30.444+0000 7f71e5200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71e006d580 con 0x7f71e00cfbe0 2024-03-04T09:30:30.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:30.444+0000 7f71e5200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71e006e190 con 0x7f71e00b5cf0 2024-03-04T09:30:30.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:30.444+0000 7f71dde00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f71e015bed0 0x7f71e015c2b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:48400/0 (socket says 172.21.15.6:48400) 2024-03-04T09:30:30.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:30.444+0000 7f71dde00640 1 -- 172.21.15.6:0/2385681485 learned_addr learned my addr 172.21.15.6:0/2385681485 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:30:30.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:30.445+0000 7f71de800640 1 --2- 172.21.15.6:0/2385681485 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f71e00b5cf0 0x7f71e00ce2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:30.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:30.445+0000 7f71dca00640 1 -- 172.21.15.6:0/2385681485 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3734399562 0 0) 0x7f71e00afb10 con 0x7f71e015bed0 2024-03-04T09:30:30.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:30.445+0000 7f71dca00640 1 -- 172.21.15.6:0/2385681485 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71c4003520 con 0x7f71e015bed0 2024-03-04T09:30:30.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:30.445+0000 7f71dca00640 1 -- 172.21.15.6:0/2385681485 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1625785313 0 0) 0x7f71c4003520 con 0x7f71e015bed0 2024-03-04T09:30:30.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:30.445+0000 7f71dca00640 1 -- 172.21.15.6:0/2385681485 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f71e00b5cf0 msgr2=0x7f71e00ce2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:30.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:30.445+0000 7f71dca00640 1 --2- 172.21.15.6:0/2385681485 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f71e00b5cf0 0x7f71e00ce2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:30.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:30.445+0000 7f71dca00640 1 -- 172.21.15.6:0/2385681485 >> v1:172.21.15.146:6789/0 conn(0x7f71e00cfbe0 legacy=0x7f71e00b55e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:30.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:30.445+0000 7f71dca00640 1 -- 172.21.15.6:0/2385681485 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71e006d860 con 0x7f71e015bed0 2024-03-04T09:30:30.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:30.445+0000 7f71dca00640 1 -- 172.21.15.6:0/2385681485 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7f71d0001670 con 0x7f71e015bed0 2024-03-04T09:30:30.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:30.445+0000 7f71dca00640 1 -- 172.21.15.6:0/2385681485 >> v1:172.21.15.6:6789/0 conn(0x7f71e015bed0 legacy=0x7f71e015c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:30.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:30.445+0000 7f71dca00640 1 --2- 172.21.15.6:0/2385681485 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f71c40039e0 0x7f71c4003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:30.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:30.446+0000 7f71dca00640 1 --2- 172.21.15.6:0/2385681485 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f71e00b5cf0 0x7f71c4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:30.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:30.446+0000 7f71dca00640 1 --2- 172.21.15.6:0/2385681485 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f71c4004d40 0x7f71c40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:30.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:30.446+0000 7f71dca00640 1 -- 172.21.15.6:0/2385681485 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71e006e190 con 0x7f71c40039e0 2024-03-04T09:30:30.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:30.446+0000 7f71dca00640 1 -- 172.21.15.6:0/2385681485 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71c400bad0 con 0x7f71c4004d40 2024-03-04T09:30:30.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:30.446+0000 7f71dca00640 1 -- 172.21.15.6:0/2385681485 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71c400bcd0 con 0x7f71e00b5cf0 2024-03-04T09:30:30.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:30.446+0000 7f71dde00640 1 --2- 172.21.15.6:0/2385681485 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f71c40039e0 0x7f71c4003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:30.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:30.446+0000 7f71dd400640 1 --2- 172.21.15.6:0/2385681485 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f71e00b5cf0 0x7f71c4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:30.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:30.446+0000 7f71dde00640 1 -- 172.21.15.6:0/2385681485 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f71e00b5cf0 msgr2=0x7f71c4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:30.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:30.446+0000 7f71dde00640 1 --2- 172.21.15.6:0/2385681485 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f71e00b5cf0 0x7f71c4004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:30.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:30.446+0000 7f71dde00640 1 -- 172.21.15.6:0/2385681485 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f71c4004d40 msgr2=0x7f71c40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:30.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:30.446+0000 7f71de800640 1 --2- 172.21.15.6:0/2385681485 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f71c4004d40 0x7f71c40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:30.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:30.446+0000 7f71dde00640 1 --2- 172.21.15.6:0/2385681485 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f71c4004d40 0x7f71c40051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:30.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:30.447+0000 7f71dde00640 1 -- 172.21.15.6:0/2385681485 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71c4003520 con 0x7f71c40039e0 2024-03-04T09:30:30.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:30.447+0000 7f71dd400640 1 --2- 172.21.15.6:0/2385681485 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f71e00b5cf0 0x7f71c4004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:30:30.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:30.448+0000 7f71dde00640 1 --2- 172.21.15.6:0/2385681485 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f71c40039e0 0x7f71c4003e00 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7f71d000c160 tx=0x7f71c4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2424880b2461ece server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:30.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:30.448+0000 7f71dca00640 1 -- 172.21.15.6:0/2385681485 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71d0007c60 con 0x7f71c40039e0 2024-03-04T09:30:30.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:30.448+0000 7f71dca00640 1 -- 172.21.15.6:0/2385681485 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71d0030070 con 0x7f71c40039e0 2024-03-04T09:30:30.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:30.448+0000 7f71dca00640 1 -- 172.21.15.6:0/2385681485 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71d0005d70 con 0x7f71c40039e0 2024-03-04T09:30:30.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:30.449+0000 7f71e5200640 1 -- 172.21.15.6:0/2385681485 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f71c40039e0 msgr2=0x7f71c4003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:30.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:30.449+0000 7f71e5200640 1 --2- 172.21.15.6:0/2385681485 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f71c40039e0 0x7f71c4003e00 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7f71d000c160 tx=0x7f71c4005830 comp rx=0 tx=0).stop 2024-03-04T09:30:30.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:30.449+0000 7f71de800640 1 -- 172.21.15.6:0/2385681485 reap_dead start 2024-03-04T09:30:30.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:30.449+0000 7f71e5200640 1 -- 172.21.15.6:0/2385681485 shutdown_connections 2024-03-04T09:30:30.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:30.449+0000 7f71e5200640 1 -- 172.21.15.6:0/2385681485 >> 172.21.15.6:0/2385681485 conn(0x7f71e00b33b0 msgr2=0x7f71e00b3790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:30.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:30.449+0000 7f71e5200640 1 -- 172.21.15.6:0/2385681485 shutdown_connections 2024-03-04T09:30:30.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:30.449+0000 7f71e5200640 1 -- 172.21.15.6:0/2385681485 wait complete. 2024-03-04T09:30:30.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:30.452+0000 7f71e5200640 1 Processor -- start 2024-03-04T09:30:30.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:30.452+0000 7f71e5200640 1 -- start start 2024-03-04T09:30:30.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:30.454+0000 7f71e5200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f71c40039e0 0x7f71e014ff70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:30.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:30.454+0000 7f71e5200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f71e014d5b0 0x7f71e014be80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:30.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:30.454+0000 7f71e5200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f71e015a9a0 0x7f71e015ae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:30.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:30.454+0000 7f71e5200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71e00ce2f0 con 0x7f71c40039e0 2024-03-04T09:30:30.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:30.454+0000 7f71e5200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71e006de20 con 0x7f71e014d5b0 2024-03-04T09:30:30.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:30.454+0000 7f71e5200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71e00ce470 con 0x7f71e015a9a0 2024-03-04T09:30:30.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:30.454+0000 7f71dde00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f71c40039e0 0x7f71e014ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:30.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:30.454+0000 7f71dde00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f71c40039e0 0x7f71e014ff70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:39828/0 (socket says 172.21.15.6:39828) 2024-03-04T09:30:30.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:30.454+0000 7f71dde00640 1 -- 172.21.15.6:0/3009449618 learned_addr learned my addr 172.21.15.6:0/3009449618 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:30:30.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:30.454+0000 7f71dd400640 1 --2- 172.21.15.6:0/3009449618 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f71e014d5b0 0x7f71e014be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:30.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:30.455+0000 7f71de800640 1 --2- 172.21.15.6:0/3009449618 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f71e015a9a0 0x7f71e015ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:30.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:30.455+0000 7f71dde00640 1 -- 172.21.15.6:0/3009449618 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f71e015a9a0 msgr2=0x7f71e015ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:30.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:30.455+0000 7f71dde00640 1 --2- 172.21.15.6:0/3009449618 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f71e015a9a0 0x7f71e015ae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:30.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:30.455+0000 7f71dde00640 1 -- 172.21.15.6:0/3009449618 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f71e014d5b0 msgr2=0x7f71e014be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:30.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:30.455+0000 7f71dde00640 1 --2- 172.21.15.6:0/3009449618 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f71e014d5b0 0x7f71e014be80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:30.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:30.455+0000 7f71dde00640 1 -- 172.21.15.6:0/3009449618 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71e0150850 con 0x7f71c40039e0 2024-03-04T09:30:30.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:30.455+0000 7f71dd400640 1 --2- 172.21.15.6:0/3009449618 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f71e014d5b0 0x7f71e014be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:30:30.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:30.455+0000 7f71dde00640 1 --2- 172.21.15.6:0/3009449618 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f71c40039e0 0x7f71e014ff70 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7f71d000c280 tx=0x7f71d0001970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:30.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:30.455+0000 7f71d6000640 1 -- 172.21.15.6:0/3009449618 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71d0004bf0 con 0x7f71c40039e0 2024-03-04T09:30:30.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:30.455+0000 7f71d6000640 1 -- 172.21.15.6:0/3009449618 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71d0000ed0 con 0x7f71c40039e0 2024-03-04T09:30:30.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:30.455+0000 7f71d6000640 1 -- 172.21.15.6:0/3009449618 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71d0007070 con 0x7f71c40039e0 2024-03-04T09:30:30.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:30.455+0000 7f71e5200640 1 -- 172.21.15.6:0/3009449618 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71e014da10 con 0x7f71c40039e0 2024-03-04T09:30:30.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:30.456+0000 7f71e5200640 1 -- 172.21.15.6:0/3009449618 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71e0070790 con 0x7f71c40039e0 2024-03-04T09:30:30.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:30.457+0000 7f71e5200640 1 -- 172.21.15.6:0/3009449618 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71e0073fb0 con 0x7f71c40039e0 2024-03-04T09:30:30.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:30.459+0000 7f71d6000640 1 -- 172.21.15.6:0/3009449618 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f71d0048020 con 0x7f71c40039e0 2024-03-04T09:30:30.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:30.460+0000 7f71d6000640 1 --2- 172.21.15.6:0/3009449618 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f71c006b4d0 0x7f71c006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:30.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:30.460+0000 7f71d6000640 1 -- 172.21.15.6:0/3009449618 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f71d00ab5b0 con 0x7f71c40039e0 2024-03-04T09:30:30.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:30.461+0000 7f71dd400640 1 --2- 172.21.15.6:0/3009449618 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f71c006b4d0 0x7f71c006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:30.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:30.462+0000 7f71dd400640 1 --2- 172.21.15.6:0/3009449618 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f71c006b4d0 0x7f71c006d990 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f71e00aeef0 tx=0x7f71cc009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:30.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:30.469+0000 7f71d6000640 1 -- 172.21.15.6:0/3009449618 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f71d0007d00 con 0x7f71c40039e0 2024-03-04T09:30:30.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:30.673+0000 7f71e5200640 1 -- 172.21.15.6:0/3009449618 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f71e0075460 con 0x7f71c40039e0 2024-03-04T09:30:30.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:30.674+0000 7f71d6000640 1 -- 172.21.15.6:0/3009449618 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v94) v1 ==== 100+0+11 (secure 0 0 0) 0x7f71d007a4c0 con 0x7f71c40039e0 2024-03-04T09:30:30.671 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 64 2024-03-04T09:30:30.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:30.676+0000 7f71e5200640 1 -- 172.21.15.6:0/3009449618 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f71c006b4d0 msgr2=0x7f71c006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:30.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:30.676+0000 7f71e5200640 1 --2- 172.21.15.6:0/3009449618 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f71c006b4d0 0x7f71c006d990 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f71e00aeef0 tx=0x7f71cc009040 comp rx=0 tx=0).stop 2024-03-04T09:30:30.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:30.676+0000 7f71e5200640 1 -- 172.21.15.6:0/3009449618 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f71c40039e0 msgr2=0x7f71e014ff70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:30.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:30.676+0000 7f71e5200640 1 --2- 172.21.15.6:0/3009449618 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f71c40039e0 0x7f71e014ff70 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7f71d000c280 tx=0x7f71d0001970 comp rx=0 tx=0).stop 2024-03-04T09:30:30.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:30.676+0000 7f71e5200640 1 -- 172.21.15.6:0/3009449618 shutdown_connections 2024-03-04T09:30:30.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:30.676+0000 7f71e5200640 1 --2- 172.21.15.6:0/3009449618 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f71c006b4d0 0x7f71c006d990 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:30.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:30.676+0000 7f71e5200640 1 --2- 172.21.15.6:0/3009449618 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f71e015a9a0 0x7f71e015ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:30.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:30.676+0000 7f71e5200640 1 --2- 172.21.15.6:0/3009449618 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f71e014d5b0 0x7f71e014be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:30.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:30.676+0000 7f71e5200640 1 --2- 172.21.15.6:0/3009449618 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f71c40039e0 0x7f71e014ff70 unknown :-1 s=CLOSED pgs=1102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:30.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:30.676+0000 7f71e5200640 1 -- 172.21.15.6:0/3009449618 >> 172.21.15.6:0/3009449618 conn(0x7f71e00b33b0 msgr2=0x7f71e0158420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:30.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:30.676+0000 7f71e5200640 1 -- 172.21.15.6:0/3009449618 shutdown_connections 2024-03-04T09:30:30.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:30.676+0000 7f71e5200640 1 -- 172.21.15.6:0/3009449618 wait complete. 2024-03-04T09:30:30.693 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-04T09:30:30.701 DEBUG:teuthology.orchestra.run.smithi146:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-04T09:30:30.858 INFO:teuthology.orchestra.run.smithi006.stdout:[global] 2024-03-04T09:30:30.858 INFO:teuthology.orchestra.run.smithi006.stdout: chdir = "" 2024-03-04T09:30:30.858 INFO:teuthology.orchestra.run.smithi006.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-04T09:30:30.858 INFO:teuthology.orchestra.run.smithi006.stdout: auth supported = cephx 2024-03-04T09:30:30.859 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:30:30.859 INFO:teuthology.orchestra.run.smithi006.stdout: filestore xattr use omap = true 2024-03-04T09:30:30.859 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:30:30.859 INFO:teuthology.orchestra.run.smithi006.stdout: mon clock drift allowed = 1.000 2024-03-04T09:30:30.859 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:30:30.859 INFO:teuthology.orchestra.run.smithi006.stdout: osd crush chooseleaf type = 0 2024-03-04T09:30:30.859 INFO:teuthology.orchestra.run.smithi006.stdout: auth debug = true 2024-03-04T09:30:30.859 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:30:30.859 INFO:teuthology.orchestra.run.smithi006.stdout: ms die on old message = true 2024-03-04T09:30:30.859 INFO:teuthology.orchestra.run.smithi006.stdout: ms die on bug = true 2024-03-04T09:30:30.859 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:30:30.859 INFO:teuthology.orchestra.run.smithi006.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-04T09:30:30.859 INFO:teuthology.orchestra.run.smithi006.stdout: mon pg warn max object skew = 0 2024-03-04T09:30:30.859 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:30:30.859 INFO:teuthology.orchestra.run.smithi006.stdout: # disable pg_autoscaler by default for new pools 2024-03-04T09:30:30.860 INFO:teuthology.orchestra.run.smithi006.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-04T09:30:30.860 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:30:30.860 INFO:teuthology.orchestra.run.smithi006.stdout: osd pool default size = 2 2024-03-04T09:30:30.860 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:30:30.860 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd allow primary affinity = true 2024-03-04T09:30:30.860 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd allow pg remap = true 2024-03-04T09:30:30.860 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on legacy crush tunables = false 2024-03-04T09:30:30.860 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on crush straw calc version zero = false 2024-03-04T09:30:30.860 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on no sortbitwise = false 2024-03-04T09:30:30.860 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on osd down out interval zero = false 2024-03-04T09:30:30.860 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on too few osds = false 2024-03-04T09:30:30.860 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-04T09:30:30.860 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-04T09:30:30.860 INFO:teuthology.orchestra.run.smithi006.stdout: mon_allow_pool_size_one = true 2024-03-04T09:30:30.860 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:30:30.861 INFO:teuthology.orchestra.run.smithi006.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-04T09:30:30.861 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:30:30.861 INFO:teuthology.orchestra.run.smithi006.stdout: osd default data pool replay window = 5 2024-03-04T09:30:30.861 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:30:30.861 INFO:teuthology.orchestra.run.smithi006.stdout: mon allow pool delete = true 2024-03-04T09:30:30.861 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:30:30.861 INFO:teuthology.orchestra.run.smithi006.stdout: mon cluster log file level = debug 2024-03-04T09:30:30.861 INFO:teuthology.orchestra.run.smithi006.stdout: debug asserts on shutdown = true 2024-03-04T09:30:30.861 INFO:teuthology.orchestra.run.smithi006.stdout: mon health detail to clog = false 2024-03-04T09:30:30.861 INFO:teuthology.orchestra.run.smithi006.stdout: mon host = "172.21.15.6,172.21.15.146,[v2:172.21.15.146:3301,v1:172.21.15.146:6790]" 2024-03-04T09:30:30.861 INFO:teuthology.orchestra.run.smithi006.stdout: ms die on skipped message = False 2024-03-04T09:30:30.861 INFO:teuthology.orchestra.run.smithi006.stdout: fsid = edce0578-5ae4-4402-b45a-552bdef84d7c 2024-03-04T09:30:30.861 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:30:30.861 INFO:teuthology.orchestra.run.smithi006.stdout:[osd] 2024-03-04T09:30:30.861 INFO:teuthology.orchestra.run.smithi006.stdout: osd journal size = 100 2024-03-04T09:30:30.861 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:30:30.862 INFO:teuthology.orchestra.run.smithi006.stdout: osd scrub load threshold = 5.0 2024-03-04T09:30:30.862 INFO:teuthology.orchestra.run.smithi006.stdout: osd scrub max interval = 600 2024-03-04T09:30:30.862 INFO:teuthology.orchestra.run.smithi006.stdout: osd mclock profile = high_recovery_ops 2024-03-04T09:30:30.862 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:30:30.862 INFO:teuthology.orchestra.run.smithi006.stdout: osd recover clone overlap = true 2024-03-04T09:30:30.862 INFO:teuthology.orchestra.run.smithi006.stdout: osd recovery max chunk = 1048576 2024-03-04T09:30:30.862 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:30:30.862 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug shutdown = true 2024-03-04T09:30:30.862 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug op order = true 2024-03-04T09:30:30.862 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug verify stray on activate = true 2024-03-04T09:30:30.862 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug trim objects = true 2024-03-04T09:30:30.862 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:30:30.862 INFO:teuthology.orchestra.run.smithi006.stdout: osd open classes on start = true 2024-03-04T09:30:30.862 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug pg log writeout = true 2024-03-04T09:30:30.862 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:30:30.863 INFO:teuthology.orchestra.run.smithi006.stdout: osd deep scrub update digest min age = 30 2024-03-04T09:30:30.863 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:30:30.863 INFO:teuthology.orchestra.run.smithi006.stdout: osd map max advance = 10 2024-03-04T09:30:30.863 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:30:30.863 INFO:teuthology.orchestra.run.smithi006.stdout: journal zero on create = true 2024-03-04T09:30:30.863 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:30:30.863 INFO:teuthology.orchestra.run.smithi006.stdout: filestore ondisk finisher threads = 3 2024-03-04T09:30:30.863 INFO:teuthology.orchestra.run.smithi006.stdout: filestore apply finisher threads = 3 2024-03-04T09:30:30.863 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:30:30.863 INFO:teuthology.orchestra.run.smithi006.stdout: bdev debug aio = true 2024-03-04T09:30:30.863 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug misdirected ops = true 2024-03-04T09:30:30.863 INFO:teuthology.orchestra.run.smithi006.stdout: bdev async discard = True 2024-03-04T09:30:30.863 INFO:teuthology.orchestra.run.smithi006.stdout: bdev enable discard = True 2024-03-04T09:30:30.863 INFO:teuthology.orchestra.run.smithi006.stdout: bluestore allocator = bitmap 2024-03-04T09:30:30.863 INFO:teuthology.orchestra.run.smithi006.stdout: bluestore block size = 96636764160 2024-03-04T09:30:30.864 INFO:teuthology.orchestra.run.smithi006.stdout: bluestore fsck on mount = True 2024-03-04T09:30:30.864 INFO:teuthology.orchestra.run.smithi006.stdout: debug bluefs = 1/20 2024-03-04T09:30:30.864 INFO:teuthology.orchestra.run.smithi006.stdout: debug bluestore = 1/20 2024-03-04T09:30:30.864 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2024-03-04T09:30:30.864 INFO:teuthology.orchestra.run.smithi006.stdout: debug osd = 20 2024-03-04T09:30:30.864 INFO:teuthology.orchestra.run.smithi006.stdout: debug rocksdb = 4/10 2024-03-04T09:30:30.864 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd backfillfull_ratio = 0.85 2024-03-04T09:30:30.864 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd full ratio = 0.9 2024-03-04T09:30:30.864 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd nearfull ratio = 0.8 2024-03-04T09:30:30.864 INFO:teuthology.orchestra.run.smithi006.stdout: osd failsafe full ratio = 0.95 2024-03-04T09:30:30.864 INFO:teuthology.orchestra.run.smithi006.stdout: osd objectstore = bluestore 2024-03-04T09:30:30.864 INFO:teuthology.orchestra.run.smithi006.stdout: osd op complaint time = 180 2024-03-04T09:30:30.864 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:30:30.864 INFO:teuthology.orchestra.run.smithi006.stdout:[mgr] 2024-03-04T09:30:30.864 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2024-03-04T09:30:30.865 INFO:teuthology.orchestra.run.smithi006.stdout: debug mgr = 20 2024-03-04T09:30:30.865 INFO:teuthology.orchestra.run.smithi006.stdout: debug mon = 20 2024-03-04T09:30:30.865 INFO:teuthology.orchestra.run.smithi006.stdout: debug auth = 20 2024-03-04T09:30:30.865 INFO:teuthology.orchestra.run.smithi006.stdout: mon reweight min pgs per osd = 4 2024-03-04T09:30:30.865 INFO:teuthology.orchestra.run.smithi006.stdout: mon reweight min bytes per osd = 10 2024-03-04T09:30:30.865 INFO:teuthology.orchestra.run.smithi006.stdout: mgr/telemetry/nag = false 2024-03-04T09:30:30.865 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:30:30.865 INFO:teuthology.orchestra.run.smithi006.stdout:[mon] 2024-03-04T09:30:30.865 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2024-03-04T09:30:30.865 INFO:teuthology.orchestra.run.smithi006.stdout: debug mon = 20 2024-03-04T09:30:30.865 INFO:teuthology.orchestra.run.smithi006.stdout: debug paxos = 20 2024-03-04T09:30:30.865 INFO:teuthology.orchestra.run.smithi006.stdout: debug auth = 20 2024-03-04T09:30:30.865 INFO:teuthology.orchestra.run.smithi006.stdout: mon data avail warn = 5 2024-03-04T09:30:30.865 INFO:teuthology.orchestra.run.smithi006.stdout: mon mgr mkfs grace = 240 2024-03-04T09:30:30.865 INFO:teuthology.orchestra.run.smithi006.stdout: mon reweight min pgs per osd = 4 2024-03-04T09:30:30.865 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd reporter subtree level = osd 2024-03-04T09:30:30.866 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd prime pg temp = true 2024-03-04T09:30:30.866 INFO:teuthology.orchestra.run.smithi006.stdout: mon reweight min bytes per osd = 10 2024-03-04T09:30:30.866 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:30:30.866 INFO:teuthology.orchestra.run.smithi006.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-04T09:30:30.866 INFO:teuthology.orchestra.run.smithi006.stdout: auth mon ticket ttl = 660 # 11m 2024-03-04T09:30:30.866 INFO:teuthology.orchestra.run.smithi006.stdout: auth service ticket ttl = 240 # 4m 2024-03-04T09:30:30.866 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:30:30.866 INFO:teuthology.orchestra.run.smithi006.stdout: # don't complain about insecure global_id in the test suite 2024-03-04T09:30:30.866 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-04T09:30:30.866 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-04T09:30:30.866 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:30:30.866 INFO:teuthology.orchestra.run.smithi006.stdout: # 1m isn't quite enough 2024-03-04T09:30:30.866 INFO:teuthology.orchestra.run.smithi006.stdout: mon_down_mkfs_grace = 2m 2024-03-04T09:30:30.866 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:30:30.866 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_filestore_osds = false 2024-03-04T09:30:30.866 INFO:teuthology.orchestra.run.smithi006.stdout: mon op complaint time = 120 2024-03-04T09:30:30.867 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:30:30.867 INFO:teuthology.orchestra.run.smithi006.stdout:[client] 2024-03-04T09:30:30.867 INFO:teuthology.orchestra.run.smithi006.stdout: rgw cache enabled = true 2024-03-04T09:30:30.867 INFO:teuthology.orchestra.run.smithi006.stdout: rgw enable ops log = true 2024-03-04T09:30:30.867 INFO:teuthology.orchestra.run.smithi006.stdout: rgw enable usage log = true 2024-03-04T09:30:30.867 INFO:teuthology.orchestra.run.smithi006.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-04T09:30:30.867 INFO:teuthology.orchestra.run.smithi006.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-04T09:30:30.867 INFO:teuthology.orchestra.run.smithi006.stdout: client mount timeout = 600 2024-03-04T09:30:30.867 INFO:teuthology.orchestra.run.smithi006.stdout: debug client = 20 2024-03-04T09:30:30.867 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2024-03-04T09:30:30.867 INFO:teuthology.orchestra.run.smithi006.stdout: pid file = "" 2024-03-04T09:30:30.867 INFO:teuthology.orchestra.run.smithi006.stdout: rados mon op timeout = 900 2024-03-04T09:30:30.867 INFO:teuthology.orchestra.run.smithi006.stdout: rados osd op timeout = 900 2024-03-04T09:30:30.868 INFO:teuthology.orchestra.run.smithi006.stdout:[mon.a] 2024-03-04T09:30:30.868 INFO:teuthology.orchestra.run.smithi006.stdout:[mon.b] 2024-03-04T09:30:30.868 INFO:teuthology.orchestra.run.smithi006.stdout:[mon.c] 2024-03-04T09:30:30.868 INFO:teuthology.orchestra.run.smithi006.stdout:[mds.a] 2024-03-04T09:30:30.868 INFO:teuthology.orchestra.run.smithi006.stdout:[mds.c] 2024-03-04T09:30:30.868 INFO:teuthology.orchestra.run.smithi006.stdout:[mds.b] 2024-03-04T09:30:30.868 INFO:teuthology.orchestra.run.smithi006.stdout:[mds.d] 2024-03-04T09:30:30.868 INFO:teuthology.orchestra.run.smithi006.stdout:[mds] 2024-03-04T09:30:30.868 INFO:teuthology.orchestra.run.smithi006.stdout: debug mds = 20 2024-03-04T09:30:30.868 INFO:teuthology.orchestra.run.smithi006.stdout: debug mds balancer = 20 2024-03-04T09:30:30.868 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2024-03-04T09:30:30.868 INFO:teuthology.orchestra.run.smithi006.stdout: mds debug frag = True 2024-03-04T09:30:30.869 INFO:teuthology.orchestra.run.smithi006.stdout: mds debug scatterstat = True 2024-03-04T09:30:30.869 INFO:teuthology.orchestra.run.smithi006.stdout: mds op complaint time = 180 2024-03-04T09:30:30.869 INFO:teuthology.orchestra.run.smithi006.stdout: mds verify scatter = True 2024-03-04T09:30:30.869 INFO:teuthology.orchestra.run.smithi006.stdout: osd op complaint time = 180 2024-03-04T09:30:30.869 INFO:teuthology.orchestra.run.smithi006.stdout: rados mon op timeout = 900 2024-03-04T09:30:30.869 INFO:teuthology.orchestra.run.smithi006.stdout: rados osd op timeout = 900 2024-03-04T09:30:30.869 INFO:teuthology.orchestra.run.smithi006.stdout: mds_max_purge_files = 6 2024-03-04T09:30:30.869 INFO:teuthology.orchestra.run.smithi006.stdout: mds_max_purge_ops = 100000000 2024-03-04T09:30:30.869 INFO:teuthology.orchestra.run.smithi006.stdout: mds_max_purge_ops_per_pg = 1562500.0 2024-03-04T09:30:30.880 INFO:teuthology.orchestra.run.smithi146.stdout:[global] 2024-03-04T09:30:30.880 INFO:teuthology.orchestra.run.smithi146.stdout: chdir = "" 2024-03-04T09:30:30.880 INFO:teuthology.orchestra.run.smithi146.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-04T09:30:30.880 INFO:teuthology.orchestra.run.smithi146.stdout: auth supported = cephx 2024-03-04T09:30:30.880 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:30:30.880 INFO:teuthology.orchestra.run.smithi146.stdout: filestore xattr use omap = true 2024-03-04T09:30:30.880 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:30:30.880 INFO:teuthology.orchestra.run.smithi146.stdout: mon clock drift allowed = 1.000 2024-03-04T09:30:30.880 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:30:30.880 INFO:teuthology.orchestra.run.smithi146.stdout: osd crush chooseleaf type = 0 2024-03-04T09:30:30.880 INFO:teuthology.orchestra.run.smithi146.stdout: auth debug = true 2024-03-04T09:30:30.881 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:30:30.881 INFO:teuthology.orchestra.run.smithi146.stdout: ms die on old message = true 2024-03-04T09:30:30.881 INFO:teuthology.orchestra.run.smithi146.stdout: ms die on bug = true 2024-03-04T09:30:30.881 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:30:30.881 INFO:teuthology.orchestra.run.smithi146.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-04T09:30:30.881 INFO:teuthology.orchestra.run.smithi146.stdout: mon pg warn max object skew = 0 2024-03-04T09:30:30.881 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:30:30.881 INFO:teuthology.orchestra.run.smithi146.stdout: # disable pg_autoscaler by default for new pools 2024-03-04T09:30:30.881 INFO:teuthology.orchestra.run.smithi146.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-04T09:30:30.881 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:30:30.881 INFO:teuthology.orchestra.run.smithi146.stdout: osd pool default size = 2 2024-03-04T09:30:30.881 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:30:30.881 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd allow primary affinity = true 2024-03-04T09:30:30.881 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd allow pg remap = true 2024-03-04T09:30:30.881 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on legacy crush tunables = false 2024-03-04T09:30:30.882 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on crush straw calc version zero = false 2024-03-04T09:30:30.882 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on no sortbitwise = false 2024-03-04T09:30:30.882 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on osd down out interval zero = false 2024-03-04T09:30:30.882 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on too few osds = false 2024-03-04T09:30:30.882 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-04T09:30:30.882 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-04T09:30:30.882 INFO:teuthology.orchestra.run.smithi146.stdout: mon_allow_pool_size_one = true 2024-03-04T09:30:30.882 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:30:30.882 INFO:teuthology.orchestra.run.smithi146.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-04T09:30:30.882 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:30:30.882 INFO:teuthology.orchestra.run.smithi146.stdout: osd default data pool replay window = 5 2024-03-04T09:30:30.882 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:30:30.882 INFO:teuthology.orchestra.run.smithi146.stdout: mon allow pool delete = true 2024-03-04T09:30:30.882 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:30:30.882 INFO:teuthology.orchestra.run.smithi146.stdout: mon cluster log file level = debug 2024-03-04T09:30:30.883 INFO:teuthology.orchestra.run.smithi146.stdout: debug asserts on shutdown = true 2024-03-04T09:30:30.883 INFO:teuthology.orchestra.run.smithi146.stdout: mon health detail to clog = false 2024-03-04T09:30:30.883 INFO:teuthology.orchestra.run.smithi146.stdout: mon host = "172.21.15.6,172.21.15.146,[v2:172.21.15.146:3301,v1:172.21.15.146:6790]" 2024-03-04T09:30:30.883 INFO:teuthology.orchestra.run.smithi146.stdout: ms die on skipped message = False 2024-03-04T09:30:30.883 INFO:teuthology.orchestra.run.smithi146.stdout: fsid = edce0578-5ae4-4402-b45a-552bdef84d7c 2024-03-04T09:30:30.883 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:30:30.883 INFO:teuthology.orchestra.run.smithi146.stdout:[osd] 2024-03-04T09:30:30.883 INFO:teuthology.orchestra.run.smithi146.stdout: osd journal size = 100 2024-03-04T09:30:30.883 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:30:30.883 INFO:teuthology.orchestra.run.smithi146.stdout: osd scrub load threshold = 5.0 2024-03-04T09:30:30.883 INFO:teuthology.orchestra.run.smithi146.stdout: osd scrub max interval = 600 2024-03-04T09:30:30.883 INFO:teuthology.orchestra.run.smithi146.stdout: osd mclock profile = high_recovery_ops 2024-03-04T09:30:30.884 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:30:30.884 INFO:teuthology.orchestra.run.smithi146.stdout: osd recover clone overlap = true 2024-03-04T09:30:30.884 INFO:teuthology.orchestra.run.smithi146.stdout: osd recovery max chunk = 1048576 2024-03-04T09:30:30.884 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:30:30.884 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug shutdown = true 2024-03-04T09:30:30.884 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug op order = true 2024-03-04T09:30:30.884 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug verify stray on activate = true 2024-03-04T09:30:30.884 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug trim objects = true 2024-03-04T09:30:30.884 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:30:30.884 INFO:teuthology.orchestra.run.smithi146.stdout: osd open classes on start = true 2024-03-04T09:30:30.884 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug pg log writeout = true 2024-03-04T09:30:30.884 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:30:30.885 INFO:teuthology.orchestra.run.smithi146.stdout: osd deep scrub update digest min age = 30 2024-03-04T09:30:30.885 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:30:30.885 INFO:teuthology.orchestra.run.smithi146.stdout: osd map max advance = 10 2024-03-04T09:30:30.885 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:30:30.885 INFO:teuthology.orchestra.run.smithi146.stdout: journal zero on create = true 2024-03-04T09:30:30.885 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:30:30.885 INFO:teuthology.orchestra.run.smithi146.stdout: filestore ondisk finisher threads = 3 2024-03-04T09:30:30.885 INFO:teuthology.orchestra.run.smithi146.stdout: filestore apply finisher threads = 3 2024-03-04T09:30:30.885 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:30:30.885 INFO:teuthology.orchestra.run.smithi146.stdout: bdev debug aio = true 2024-03-04T09:30:30.885 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug misdirected ops = true 2024-03-04T09:30:30.885 INFO:teuthology.orchestra.run.smithi146.stdout: bdev async discard = True 2024-03-04T09:30:30.885 INFO:teuthology.orchestra.run.smithi146.stdout: bdev enable discard = True 2024-03-04T09:30:30.885 INFO:teuthology.orchestra.run.smithi146.stdout: bluestore allocator = bitmap 2024-03-04T09:30:30.885 INFO:teuthology.orchestra.run.smithi146.stdout: bluestore block size = 96636764160 2024-03-04T09:30:30.886 INFO:teuthology.orchestra.run.smithi146.stdout: bluestore fsck on mount = True 2024-03-04T09:30:30.886 INFO:teuthology.orchestra.run.smithi146.stdout: debug bluefs = 1/20 2024-03-04T09:30:30.886 INFO:teuthology.orchestra.run.smithi146.stdout: debug bluestore = 1/20 2024-03-04T09:30:30.886 INFO:teuthology.orchestra.run.smithi146.stdout: debug ms = 1 2024-03-04T09:30:30.886 INFO:teuthology.orchestra.run.smithi146.stdout: debug osd = 20 2024-03-04T09:30:30.886 INFO:teuthology.orchestra.run.smithi146.stdout: debug rocksdb = 4/10 2024-03-04T09:30:30.886 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd backfillfull_ratio = 0.85 2024-03-04T09:30:30.886 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd full ratio = 0.9 2024-03-04T09:30:30.886 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd nearfull ratio = 0.8 2024-03-04T09:30:30.886 INFO:teuthology.orchestra.run.smithi146.stdout: osd failsafe full ratio = 0.95 2024-03-04T09:30:30.886 INFO:teuthology.orchestra.run.smithi146.stdout: osd objectstore = bluestore 2024-03-04T09:30:30.886 INFO:teuthology.orchestra.run.smithi146.stdout: osd op complaint time = 180 2024-03-04T09:30:30.886 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:30:30.886 INFO:teuthology.orchestra.run.smithi146.stdout:[mgr] 2024-03-04T09:30:30.886 INFO:teuthology.orchestra.run.smithi146.stdout: debug ms = 1 2024-03-04T09:30:30.887 INFO:teuthology.orchestra.run.smithi146.stdout: debug mgr = 20 2024-03-04T09:30:30.887 INFO:teuthology.orchestra.run.smithi146.stdout: debug mon = 20 2024-03-04T09:30:30.887 INFO:teuthology.orchestra.run.smithi146.stdout: debug auth = 20 2024-03-04T09:30:30.887 INFO:teuthology.orchestra.run.smithi146.stdout: mon reweight min pgs per osd = 4 2024-03-04T09:30:30.887 INFO:teuthology.orchestra.run.smithi146.stdout: mon reweight min bytes per osd = 10 2024-03-04T09:30:30.887 INFO:teuthology.orchestra.run.smithi146.stdout: mgr/telemetry/nag = false 2024-03-04T09:30:30.887 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:30:30.887 INFO:teuthology.orchestra.run.smithi146.stdout:[mon] 2024-03-04T09:30:30.887 INFO:teuthology.orchestra.run.smithi146.stdout: debug ms = 1 2024-03-04T09:30:30.887 INFO:teuthology.orchestra.run.smithi146.stdout: debug mon = 20 2024-03-04T09:30:30.887 INFO:teuthology.orchestra.run.smithi146.stdout: debug paxos = 20 2024-03-04T09:30:30.887 INFO:teuthology.orchestra.run.smithi146.stdout: debug auth = 20 2024-03-04T09:30:30.887 INFO:teuthology.orchestra.run.smithi146.stdout: mon data avail warn = 5 2024-03-04T09:30:30.887 INFO:teuthology.orchestra.run.smithi146.stdout: mon mgr mkfs grace = 240 2024-03-04T09:30:30.887 INFO:teuthology.orchestra.run.smithi146.stdout: mon reweight min pgs per osd = 4 2024-03-04T09:30:30.887 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd reporter subtree level = osd 2024-03-04T09:30:30.888 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd prime pg temp = true 2024-03-04T09:30:30.888 INFO:teuthology.orchestra.run.smithi146.stdout: mon reweight min bytes per osd = 10 2024-03-04T09:30:30.888 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:30:30.888 INFO:teuthology.orchestra.run.smithi146.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-04T09:30:30.888 INFO:teuthology.orchestra.run.smithi146.stdout: auth mon ticket ttl = 660 # 11m 2024-03-04T09:30:30.888 INFO:teuthology.orchestra.run.smithi146.stdout: auth service ticket ttl = 240 # 4m 2024-03-04T09:30:30.888 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:30:30.888 INFO:teuthology.orchestra.run.smithi146.stdout: # don't complain about insecure global_id in the test suite 2024-03-04T09:30:30.888 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-04T09:30:30.888 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-04T09:30:30.888 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:30:30.888 INFO:teuthology.orchestra.run.smithi146.stdout: # 1m isn't quite enough 2024-03-04T09:30:30.888 INFO:teuthology.orchestra.run.smithi146.stdout: mon_down_mkfs_grace = 2m 2024-03-04T09:30:30.888 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:30:30.889 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_filestore_osds = false 2024-03-04T09:30:30.889 INFO:teuthology.orchestra.run.smithi146.stdout: mon op complaint time = 120 2024-03-04T09:30:30.889 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:30:30.889 INFO:teuthology.orchestra.run.smithi146.stdout:[client] 2024-03-04T09:30:30.889 INFO:teuthology.orchestra.run.smithi146.stdout: rgw cache enabled = true 2024-03-04T09:30:30.889 INFO:teuthology.orchestra.run.smithi146.stdout: rgw enable ops log = true 2024-03-04T09:30:30.889 INFO:teuthology.orchestra.run.smithi146.stdout: rgw enable usage log = true 2024-03-04T09:30:30.889 INFO:teuthology.orchestra.run.smithi146.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-04T09:30:30.889 INFO:teuthology.orchestra.run.smithi146.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-04T09:30:30.889 INFO:teuthology.orchestra.run.smithi146.stdout: client mount timeout = 600 2024-03-04T09:30:30.889 INFO:teuthology.orchestra.run.smithi146.stdout: debug client = 20 2024-03-04T09:30:30.889 INFO:teuthology.orchestra.run.smithi146.stdout: debug ms = 1 2024-03-04T09:30:30.889 INFO:teuthology.orchestra.run.smithi146.stdout: pid file = "" 2024-03-04T09:30:30.889 INFO:teuthology.orchestra.run.smithi146.stdout: rados mon op timeout = 900 2024-03-04T09:30:30.889 INFO:teuthology.orchestra.run.smithi146.stdout: rados osd op timeout = 900 2024-03-04T09:30:30.889 INFO:teuthology.orchestra.run.smithi146.stdout:[mon.a] 2024-03-04T09:30:30.890 INFO:teuthology.orchestra.run.smithi146.stdout:[mon.b] 2024-03-04T09:30:30.890 INFO:teuthology.orchestra.run.smithi146.stdout:[mon.c] 2024-03-04T09:30:30.890 INFO:teuthology.orchestra.run.smithi146.stdout:[mds.a] 2024-03-04T09:30:30.890 INFO:teuthology.orchestra.run.smithi146.stdout:[mds.c] 2024-03-04T09:30:30.890 INFO:teuthology.orchestra.run.smithi146.stdout:[mds.b] 2024-03-04T09:30:30.890 INFO:teuthology.orchestra.run.smithi146.stdout:[mds.d] 2024-03-04T09:30:30.890 INFO:teuthology.orchestra.run.smithi146.stdout:[mds] 2024-03-04T09:30:30.890 INFO:teuthology.orchestra.run.smithi146.stdout: debug mds = 20 2024-03-04T09:30:30.890 INFO:teuthology.orchestra.run.smithi146.stdout: debug mds balancer = 20 2024-03-04T09:30:30.890 INFO:teuthology.orchestra.run.smithi146.stdout: debug ms = 1 2024-03-04T09:30:30.890 INFO:teuthology.orchestra.run.smithi146.stdout: mds debug frag = True 2024-03-04T09:30:30.890 INFO:teuthology.orchestra.run.smithi146.stdout: mds debug scatterstat = True 2024-03-04T09:30:30.890 INFO:teuthology.orchestra.run.smithi146.stdout: mds op complaint time = 180 2024-03-04T09:30:30.890 INFO:teuthology.orchestra.run.smithi146.stdout: mds verify scatter = True 2024-03-04T09:30:30.890 INFO:teuthology.orchestra.run.smithi146.stdout: osd op complaint time = 180 2024-03-04T09:30:30.891 INFO:teuthology.orchestra.run.smithi146.stdout: rados mon op timeout = 900 2024-03-04T09:30:30.891 INFO:teuthology.orchestra.run.smithi146.stdout: rados osd op timeout = 900 2024-03-04T09:30:30.891 INFO:teuthology.orchestra.run.smithi146.stdout: mds_max_purge_files = 6 2024-03-04T09:30:30.891 INFO:teuthology.orchestra.run.smithi146.stdout: mds_max_purge_ops = 100000000 2024-03-04T09:30:30.891 INFO:teuthology.orchestra.run.smithi146.stdout: mds_max_purge_ops_per_pg = 1562500.0 2024-03-04T09:30:30.925 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-04T09:30:30.926 INFO:teuthology.orchestra.run:waiting for 300 2024-03-04T09:30:30.926 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-04T09:30:30.926 INFO:teuthology.orchestra.run:waiting for 300 2024-03-04T09:30:30.926 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-04T09:30:30.926 INFO:teuthology.orchestra.run:waiting for 300 2024-03-04T09:30:30.927 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-04T09:30:30.927 INFO:teuthology.orchestra.run:waiting for 300 2024-03-04T09:30:30.943 INFO:tasks.ceph.mds.c:Stopped 2024-03-04T09:30:30.943 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail c 2024-03-04T09:30:30.947 INFO:tasks.ceph.mds.d:Stopped 2024-03-04T09:30:30.947 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail d 2024-03-04T09:30:30.950 INFO:tasks.ceph.mds.b:Stopped 2024-03-04T09:30:30.950 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail b 2024-03-04T09:30:30.952 INFO:tasks.ceph.mds.a:Stopped 2024-03-04T09:30:30.952 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail a 2024-03-04T09:30:31.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.193+0000 7efd2a800640 1 Processor -- start 2024-03-04T09:30:31.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.193+0000 7efd2a800640 1 -- start start 2024-03-04T09:30:31.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.193+0000 7f5764200640 1 Processor -- start 2024-03-04T09:30:31.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.193+0000 7f5764200640 1 -- start start 2024-03-04T09:30:31.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.194+0000 7efd2a800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efd240d2db0 0x7efd240d0c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:31.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.194+0000 7efd2a800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7efd240b2fa0 0x7efd240b12a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:31.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.194+0000 7efd2a800640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd240cf730 con 0x7efd2415e4e0 2024-03-04T09:30:31.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.194+0000 7efd2a800640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd240b0b30 con 0x7efd240b2fa0 2024-03-04T09:30:31.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.194+0000 7efd2a800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd240b0e10 con 0x7efd240d2db0 2024-03-04T09:30:31.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.195+0000 7f5764200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f575c0d2e30 0x7f575c0d4b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:31.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.195+0000 7f5764200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f575c0cf720 0x7f575c0d1bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:31.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.195+0000 7efd29e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7efd240b2fa0 0x7efd240b12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:31.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.195+0000 7efd29e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7efd240b2fa0 0x7efd240b12a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.6:36882/0 (socket says 172.21.15.6:36882) 2024-03-04T09:30:31.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.195+0000 7efd29e00640 1 -- 172.21.15.6:0/113771113 learned_addr learned my addr 172.21.15.6:0/113771113 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:30:31.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.195+0000 7f5764200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f575c0cfbc0 con 0x7f575c0d09d0 2024-03-04T09:30:31.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.195+0000 7f5764200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f575c161e20 con 0x7f575c0d2e30 2024-03-04T09:30:31.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.195+0000 7f5764200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f575c0b38f0 con 0x7f575c0cf720 2024-03-04T09:30:31.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.195+0000 7efd28a00640 1 --2- 172.21.15.6:0/113771113 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efd240d2db0 0x7efd240d0c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:31.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.196+0000 7f5760e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f575c0cf720 0x7f575c0d1bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:31.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.196+0000 7f5760e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f575c0cf720 0x7f575c0d1bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.6:51410/0 (socket says 172.21.15.6:51410) 2024-03-04T09:30:31.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.196+0000 7f5760e00640 1 -- 172.21.15.6:0/2779352827 learned_addr learned my addr 172.21.15.6:0/2779352827 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:30:31.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.196+0000 7f5761800640 1 --2- 172.21.15.6:0/2779352827 >> v2:172.21.15.146:3300/0 conn(0x7f575c0d2e30 0x7f575c0d4b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:31.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.196+0000 7efd29e00640 1 -- 172.21.15.6:0/113771113 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efd240d2db0 msgr2=0x7efd240d0c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:31.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.196+0000 7efd29e00640 1 --2- 172.21.15.6:0/113771113 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efd240d2db0 0x7efd240d0c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:31.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.196+0000 7efd29e00640 1 -- 172.21.15.6:0/113771113 >> v1:172.21.15.146:6789/0 conn(0x7efd2415e4e0 legacy=0x7efd240d0560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:31.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.196+0000 7efd29e00640 1 -- 172.21.15.6:0/113771113 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd240cfa10 con 0x7efd240b2fa0 2024-03-04T09:30:31.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.197+0000 7f5760e00640 1 -- 172.21.15.6:0/2779352827 >> v2:172.21.15.146:3300/0 conn(0x7f575c0d2e30 msgr2=0x7f575c0d4b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:31.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.197+0000 7f5760e00640 1 --2- 172.21.15.6:0/2779352827 >> v2:172.21.15.146:3300/0 conn(0x7f575c0d2e30 0x7f575c0d4b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:31.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.197+0000 7f5760e00640 1 -- 172.21.15.6:0/2779352827 >> v1:172.21.15.146:6789/0 conn(0x7f575c0d09d0 legacy=0x7f575c0b4340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:31.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.197+0000 7f5760e00640 1 -- 172.21.15.6:0/2779352827 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f575c0cfea0 con 0x7f575c0cf720 2024-03-04T09:30:31.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.198+0000 7efd29e00640 1 --2- 172.21.15.6:0/113771113 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efd240b2fa0 0x7efd240b12a0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7efd1c009e50 tx=0x7efd1c02bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7d7710d9b20c35f5 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:31.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.198+0000 7efd23e00640 1 -- 172.21.15.6:0/113771113 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd1c02cb00 con 0x7efd240b2fa0 2024-03-04T09:30:31.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.199+0000 7f5760e00640 1 --2- 172.21.15.6:0/2779352827 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f575c0cf720 0x7f575c0d1bd0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f5748009870 tx=0x7f574802b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7ef22ad39deca7be server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:31.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.199+0000 7efd23e00640 1 -- 172.21.15.6:0/113771113 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd1c02cca0 con 0x7efd240b2fa0 2024-03-04T09:30:31.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.199+0000 7efd23e00640 1 -- 172.21.15.6:0/113771113 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd1c034610 con 0x7efd240b2fa0 2024-03-04T09:30:31.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.199+0000 7f575be00640 1 -- 172.21.15.6:0/2779352827 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5748034070 con 0x7f575c0cf720 2024-03-04T09:30:31.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.199+0000 7f575be00640 1 -- 172.21.15.6:0/2779352827 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5748030840 con 0x7f575c0cf720 2024-03-04T09:30:31.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.199+0000 7f575be00640 1 -- 172.21.15.6:0/2779352827 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5748030c10 con 0x7f575c0cf720 2024-03-04T09:30:31.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.199+0000 7efd2a800640 1 -- 172.21.15.6:0/113771113 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efd240b2fa0 msgr2=0x7efd240b12a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:31.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.199+0000 7efd2a800640 1 --2- 172.21.15.6:0/113771113 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efd240b2fa0 0x7efd240b12a0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7efd1c009e50 tx=0x7efd1c02bf60 comp rx=0 tx=0).stop 2024-03-04T09:30:31.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.200+0000 7efd2a800640 1 -- 172.21.15.6:0/113771113 shutdown_connections 2024-03-04T09:30:31.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.200+0000 7efd2a800640 1 --2- 172.21.15.6:0/113771113 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efd240b2fa0 0x7efd240b12a0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:31.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.200+0000 7efd2a800640 1 --2- 172.21.15.6:0/113771113 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efd240d2db0 0x7efd240d0c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:31.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.200+0000 7efd2a800640 1 -- 172.21.15.6:0/113771113 >> 172.21.15.6:0/113771113 conn(0x7efd240d3ea0 msgr2=0x7efd240d42b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:31.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.200+0000 7efd2a800640 1 -- 172.21.15.6:0/113771113 shutdown_connections 2024-03-04T09:30:31.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.200+0000 7efd2a800640 1 -- 172.21.15.6:0/113771113 wait complete. 2024-03-04T09:30:31.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.200+0000 7f5764200640 1 -- 172.21.15.6:0/2779352827 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f575c0cf720 msgr2=0x7f575c0d1bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:31.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.200+0000 7f5764200640 1 --2- 172.21.15.6:0/2779352827 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f575c0cf720 0x7f575c0d1bd0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f5748009870 tx=0x7f574802b980 comp rx=0 tx=0).stop 2024-03-04T09:30:31.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.200+0000 7f5764200640 1 -- 172.21.15.6:0/2779352827 shutdown_connections 2024-03-04T09:30:31.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.200+0000 7f5764200640 1 --2- 172.21.15.6:0/2779352827 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f575c0cf720 0x7f575c0d1bd0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:31.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.200+0000 7f5764200640 1 --2- 172.21.15.6:0/2779352827 >> v2:172.21.15.146:3300/0 conn(0x7f575c0d2e30 0x7f575c0d4b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:31.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.201+0000 7f5764200640 1 -- 172.21.15.6:0/2779352827 >> 172.21.15.6:0/2779352827 conn(0x7f575c0d4000 msgr2=0x7f575c0d4410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:31.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.201+0000 7f5764200640 1 -- 172.21.15.6:0/2779352827 shutdown_connections 2024-03-04T09:30:31.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.201+0000 7f5764200640 1 -- 172.21.15.6:0/2779352827 wait complete. 2024-03-04T09:30:31.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.202+0000 7efd2a800640 1 Processor -- start 2024-03-04T09:30:31.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.202+0000 7efd2a800640 1 -- start start 2024-03-04T09:30:31.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.203+0000 7f5764200640 1 Processor -- start 2024-03-04T09:30:31.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.203+0000 7f5764200640 1 -- start start 2024-03-04T09:30:31.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.203+0000 7efd2a800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efd240b2fa0 0x7efd240cd350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:31.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.203+0000 7efd2a800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efd2415e4e0 0x7efd241603a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:31.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.203+0000 7efd2a800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efd240cd890 0x7efd240b6a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:31.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.203+0000 7efd2a800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd2415ea40 con 0x7efd2415e4e0 2024-03-04T09:30:31.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.203+0000 7efd2a800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd2415e8c0 con 0x7efd240b2fa0 2024-03-04T09:30:31.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.203+0000 7efd2a800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd240d11b0 con 0x7efd240cd890 2024-03-04T09:30:31.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.204+0000 7efd28a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efd2415e4e0 0x7efd241603a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:31.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.204+0000 7efd28a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efd2415e4e0 0x7efd241603a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:39840/0 (socket says 172.21.15.6:39840) 2024-03-04T09:30:31.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.204+0000 7efd28a00640 1 -- 172.21.15.6:0/1967906821 learned_addr learned my addr 172.21.15.6:0/1967906821 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:30:31.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.204+0000 7efd29400640 1 --2- 172.21.15.6:0/1967906821 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efd240b2fa0 0x7efd240cd350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:31.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.204+0000 7efd29e00640 1 --2- 172.21.15.6:0/1967906821 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efd240cd890 0x7efd240b6a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:31.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.204+0000 7f5764200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f575c0cf720 0x7f575c15d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:31.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.204+0000 7f5764200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f575c0d09d0 0x7f575c0a0e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:31.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.204+0000 7f5764200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f575c0d2e30 0x7f575c0c9d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:31.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.204+0000 7f5764200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f575c0b50e0 con 0x7f575c0d09d0 2024-03-04T09:30:31.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.204+0000 7f5764200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f575c0b3e20 con 0x7f575c0cf720 2024-03-04T09:30:31.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.204+0000 7f5764200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f575c0b5260 con 0x7f575c0d2e30 2024-03-04T09:30:31.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.205+0000 7efd28a00640 1 -- 172.21.15.6:0/1967906821 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efd240cd890 msgr2=0x7efd240b6a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:31.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.205+0000 7efd28a00640 1 --2- 172.21.15.6:0/1967906821 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efd240cd890 0x7efd240b6a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:31.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.205+0000 7efd28a00640 1 -- 172.21.15.6:0/1967906821 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efd240b2fa0 msgr2=0x7efd240cd350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:31.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.205+0000 7efd28a00640 1 --2- 172.21.15.6:0/1967906821 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efd240b2fa0 0x7efd240cd350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:31.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.205+0000 7efd28a00640 1 -- 172.21.15.6:0/1967906821 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd24160e40 con 0x7efd2415e4e0 2024-03-04T09:30:31.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.205+0000 7f5760e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f575c0d09d0 0x7f575c0a0e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:31.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.205+0000 7f5760e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f575c0d09d0 0x7f575c0a0e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:39844/0 (socket says 172.21.15.6:39844) 2024-03-04T09:30:31.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.205+0000 7f5760e00640 1 -- 172.21.15.6:0/951301177 learned_addr learned my addr 172.21.15.6:0/951301177 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:30:31.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.205+0000 7f5761800640 1 --2- 172.21.15.6:0/951301177 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f575c0cf720 0x7f575c15d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:31.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.205+0000 7f5762200640 1 --2- 172.21.15.6:0/951301177 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f575c0d2e30 0x7f575c0c9d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:31.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.205+0000 7efd28a00640 1 --2- 172.21.15.6:0/1967906821 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efd2415e4e0 0x7efd241603a0 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7efd1000e8a0 tx=0x7efd1000eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:31.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.206+0000 7efd21600640 1 -- 172.21.15.6:0/1967906821 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd10017070 con 0x7efd2415e4e0 2024-03-04T09:30:31.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.206+0000 7efd21600640 1 -- 172.21.15.6:0/1967906821 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd1000c940 con 0x7efd2415e4e0 2024-03-04T09:30:31.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.206+0000 7efd21600640 1 -- 172.21.15.6:0/1967906821 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd1000cd30 con 0x7efd2415e4e0 2024-03-04T09:30:31.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.206+0000 7f5760e00640 1 -- 172.21.15.6:0/951301177 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f575c0d2e30 msgr2=0x7f575c0c9d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:31.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.206+0000 7f5760e00640 1 --2- 172.21.15.6:0/951301177 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f575c0d2e30 0x7f575c0c9d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:31.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.206+0000 7f5760e00640 1 -- 172.21.15.6:0/951301177 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f575c0cf720 msgr2=0x7f575c15d5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:31.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.206+0000 7f5760e00640 1 --2- 172.21.15.6:0/951301177 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f575c0cf720 0x7f575c15d5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:31.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.206+0000 7f5760e00640 1 -- 172.21.15.6:0/951301177 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f575c0a00b0 con 0x7f575c0d09d0 2024-03-04T09:30:31.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.207+0000 7efd2a800640 1 -- 172.21.15.6:0/1967906821 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd240cc870 con 0x7efd2415e4e0 2024-03-04T09:30:31.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.207+0000 7efd2a800640 1 -- 172.21.15.6:0/1967906821 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd240cce30 con 0x7efd2415e4e0 2024-03-04T09:30:31.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.207+0000 7f5760e00640 1 --2- 172.21.15.6:0/951301177 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f575c0d09d0 0x7f575c0a0e70 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7f574802b9c0 tx=0x7f5748004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:31.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.208+0000 7f5759600640 1 -- 172.21.15.6:0/951301177 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5748048070 con 0x7f575c0d09d0 2024-03-04T09:30:31.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.208+0000 7f5759600640 1 -- 172.21.15.6:0/951301177 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5748030640 con 0x7f575c0d09d0 2024-03-04T09:30:31.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.208+0000 7f5759600640 1 -- 172.21.15.6:0/951301177 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5748034040 con 0x7f575c0d09d0 2024-03-04T09:30:31.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.208+0000 7f5764200640 1 -- 172.21.15.6:0/951301177 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f575c0a0390 con 0x7f575c0d09d0 2024-03-04T09:30:31.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.208+0000 7f5764200640 1 -- 172.21.15.6:0/951301177 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f575c09fd80 con 0x7f575c0d09d0 2024-03-04T09:30:31.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.209+0000 7efd2a800640 1 -- 172.21.15.6:0/1967906821 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd2404f330 con 0x7efd2415e4e0 2024-03-04T09:30:31.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.209+0000 7f5764200640 1 -- 172.21.15.6:0/951301177 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f575c04f410 con 0x7f575c0d09d0 2024-03-04T09:30:31.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.211+0000 7efd21600640 1 -- 172.21.15.6:0/1967906821 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efd1001a070 con 0x7efd2415e4e0 2024-03-04T09:30:31.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.211+0000 7f5759600640 1 -- 172.21.15.6:0/951301177 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f574804b070 con 0x7f575c0d09d0 2024-03-04T09:30:31.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.212+0000 7f5759600640 1 --2- 172.21.15.6:0/951301177 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f573c06b550 0x7f573c06da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:31.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.212+0000 7f5759600640 1 -- 172.21.15.6:0/951301177 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f57480ac760 con 0x7f575c0d09d0 2024-03-04T09:30:31.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.212+0000 7efd21600640 1 --2- 172.21.15.6:0/1967906821 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7efd0406b520 0x7efd0406d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:31.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.212+0000 7efd21600640 1 -- 172.21.15.6:0/1967906821 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efd1008a2a0 con 0x7efd2415e4e0 2024-03-04T09:30:31.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.213+0000 7f5761800640 1 --2- 172.21.15.6:0/951301177 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f573c06b550 0x7f573c06da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:31.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.213+0000 7efd29400640 1 --2- 172.21.15.6:0/1967906821 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7efd0406b520 0x7efd0406d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:31.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.214+0000 7efd29400640 1 --2- 172.21.15.6:0/1967906821 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7efd0406b520 0x7efd0406d9e0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7efd18001780 tx=0x7efd18009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:31.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.214+0000 7f5761800640 1 --2- 172.21.15.6:0/951301177 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f573c06b550 0x7f573c06da10 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f575c0a1e30 tx=0x7f5750009190 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:31.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.220+0000 7f0bb7000640 1 Processor -- start 2024-03-04T09:30:31.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.220+0000 7f0bb7000640 1 -- start start 2024-03-04T09:30:31.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.220+0000 7f5759600640 1 -- 172.21.15.6:0/951301177 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5748052020 con 0x7f575c0d09d0 2024-03-04T09:30:31.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.222+0000 7f0bb7000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f0bb006e170 0x7f0bb015d040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:31.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.222+0000 7f0bb7000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f0bb00cf650 0x7f0bb0070c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:31.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.222+0000 7f0bb7000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bb00ad390 con 0x7f0bb00af2e0 2024-03-04T09:30:31.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.222+0000 7f0bb7000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bb006e550 con 0x7f0bb006e170 2024-03-04T09:30:31.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.222+0000 7f0bb7000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bb015d580 con 0x7f0bb00cf650 2024-03-04T09:30:31.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.222+0000 7f0bafe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f0bb006e170 0x7f0bb015d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:31.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.222+0000 7f0bafe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f0bb006e170 0x7f0bb015d040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:39856/0 (socket says 172.21.15.6:39856) 2024-03-04T09:30:31.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.222+0000 7f0bafe00640 1 -- 172.21.15.6:0/3822323476 learned_addr learned my addr 172.21.15.6:0/3822323476 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:30:31.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.223+0000 7f0bb5000640 1 --2- 172.21.15.6:0/3822323476 >> v2:172.21.15.146:3300/0 conn(0x7f0bb00cf650 0x7f0bb0070c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:31.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.223+0000 7f0bafe00640 1 -- 172.21.15.6:0/3822323476 >> v2:172.21.15.146:3300/0 conn(0x7f0bb00cf650 msgr2=0x7f0bb0070c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:31.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.223+0000 7f0bafe00640 1 --2- 172.21.15.6:0/3822323476 >> v2:172.21.15.146:3300/0 conn(0x7f0bb00cf650 0x7f0bb0070c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:31.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.223+0000 7f0bafe00640 1 -- 172.21.15.6:0/3822323476 >> v1:172.21.15.6:6789/0 conn(0x7f0bb00af2e0 legacy=0x7f0bb00cef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:31.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.223+0000 7f0bafe00640 1 -- 172.21.15.6:0/3822323476 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bb00ace10 con 0x7f0bb006e170 2024-03-04T09:30:31.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.224+0000 7f0bafe00640 1 --2- 172.21.15.6:0/3822323476 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0bb006e170 0x7f0bb015d040 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7f0bb006f830 tx=0x7f0ba002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1bcf64e92ac21b2 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:31.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.229+0000 7f0baea00640 1 -- 172.21.15.6:0/3822323476 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ba0034070 con 0x7f0bb006e170 2024-03-04T09:30:31.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.229+0000 7f0baea00640 1 -- 172.21.15.6:0/3822323476 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ba0030860 con 0x7f0bb006e170 2024-03-04T09:30:31.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.229+0000 7f0baea00640 1 -- 172.21.15.6:0/3822323476 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ba0030c30 con 0x7f0bb006e170 2024-03-04T09:30:31.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.230+0000 7f0bb7000640 1 -- 172.21.15.6:0/3822323476 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0bb006e170 msgr2=0x7f0bb015d040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:31.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.230+0000 7f0bb7000640 1 --2- 172.21.15.6:0/3822323476 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0bb006e170 0x7f0bb015d040 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7f0bb006f830 tx=0x7f0ba002b9a0 comp rx=0 tx=0).stop 2024-03-04T09:30:31.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.230+0000 7f0bb7000640 1 -- 172.21.15.6:0/3822323476 shutdown_connections 2024-03-04T09:30:31.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.230+0000 7f0bb7000640 1 --2- 172.21.15.6:0/3822323476 >> v2:172.21.15.146:3300/0 conn(0x7f0bb00cf650 0x7f0bb0070c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:31.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.230+0000 7f0bb7000640 1 --2- 172.21.15.6:0/3822323476 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0bb006e170 0x7f0bb015d040 unknown :-1 s=CLOSED pgs=1106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:31.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.230+0000 7f0bb7000640 1 -- 172.21.15.6:0/3822323476 >> 172.21.15.6:0/3822323476 conn(0x7f0bb00b1990 msgr2=0x7f0bb00b1da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:31.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.230+0000 7f0bb7000640 1 -- 172.21.15.6:0/3822323476 shutdown_connections 2024-03-04T09:30:31.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.230+0000 7f0bb7000640 1 -- 172.21.15.6:0/3822323476 wait complete. 2024-03-04T09:30:31.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.231+0000 7efd21600640 1 -- 172.21.15.6:0/1967906821 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7efd10058c40 con 0x7efd2415e4e0 2024-03-04T09:30:31.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.232+0000 7f0bb7000640 1 Processor -- start 2024-03-04T09:30:31.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.233+0000 7f0bb7000640 1 -- start start 2024-03-04T09:30:31.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.234+0000 7f0bb7000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0bb006e170 0x7f0bb00b3920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:31.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.234+0000 7f0bb7000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0bb00af2e0 0x7f0bb009aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:31.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.234+0000 7f0bb7000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0bb00cf650 0x7f0bb00cd970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:31.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.234+0000 7f0bb7000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bb009ddd0 con 0x7f0bb00cf650 2024-03-04T09:30:31.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.234+0000 7f0bb7000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bb015c840 con 0x7f0bb00af2e0 2024-03-04T09:30:31.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.234+0000 7f0bb7000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bb009df50 con 0x7f0bb006e170 2024-03-04T09:30:31.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.234+0000 7f0bb5000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0bb00cf650 0x7f0bb00cd970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:31.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.234+0000 7f0bb5000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0bb00cf650 0x7f0bb00cd970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:39864/0 (socket says 172.21.15.6:39864) 2024-03-04T09:30:31.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.234+0000 7f0bb5000640 1 -- 172.21.15.6:0/2754232654 learned_addr learned my addr 172.21.15.6:0/2754232654 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:30:31.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.234+0000 7f0baf400640 1 --2- 172.21.15.6:0/2754232654 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0bb00af2e0 0x7f0bb009aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:31.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.235+0000 7f0bafe00640 1 --2- 172.21.15.6:0/2754232654 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0bb006e170 0x7f0bb00b3920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:31.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.235+0000 7f0bb5000640 1 -- 172.21.15.6:0/2754232654 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0bb006e170 msgr2=0x7f0bb00b3920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:31.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.235+0000 7f0bb5000640 1 --2- 172.21.15.6:0/2754232654 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0bb006e170 0x7f0bb00b3920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:31.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.235+0000 7f0bb5000640 1 -- 172.21.15.6:0/2754232654 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0bb00af2e0 msgr2=0x7f0bb009aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:31.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.235+0000 7f0bb5000640 1 --2- 172.21.15.6:0/2754232654 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0bb00af2e0 0x7f0bb009aa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:31.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.235+0000 7f0bb5000640 1 -- 172.21.15.6:0/2754232654 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bb009c850 con 0x7f0bb00cf650 2024-03-04T09:30:31.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.236+0000 7f0bb5000640 1 --2- 172.21.15.6:0/2754232654 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0bb00cf650 0x7f0bb00cd970 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7f0ba8007e90 tx=0x7f0ba800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:31.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.236+0000 7f0ba7e00640 1 -- 172.21.15.6:0/2754232654 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ba8014420 con 0x7f0bb00cf650 2024-03-04T09:30:31.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.236+0000 7f0ba7e00640 1 -- 172.21.15.6:0/2754232654 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ba8019070 con 0x7f0bb00cf650 2024-03-04T09:30:31.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.236+0000 7f0ba7e00640 1 -- 172.21.15.6:0/2754232654 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ba80147c0 con 0x7f0bb00cf650 2024-03-04T09:30:31.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.236+0000 7f0bb7000640 1 -- 172.21.15.6:0/2754232654 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bb009cb30 con 0x7f0bb00cf650 2024-03-04T09:30:31.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.236+0000 7f0bb7000640 1 -- 172.21.15.6:0/2754232654 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bb009b170 con 0x7f0bb00cf650 2024-03-04T09:30:31.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.238+0000 7f0bb7000640 1 -- 172.21.15.6:0/2754232654 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bb009e0f0 con 0x7f0bb00cf650 2024-03-04T09:30:31.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.239+0000 7f0ba7e00640 1 -- 172.21.15.6:0/2754232654 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0ba8014960 con 0x7f0bb00cf650 2024-03-04T09:30:31.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.241+0000 7f0ba7e00640 1 --2- 172.21.15.6:0/2754232654 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f0b9006b4d0 0x7f0b9006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:31.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.241+0000 7f0ba7e00640 1 -- 172.21.15.6:0/2754232654 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0ba808a8f0 con 0x7f0bb00cf650 2024-03-04T09:30:31.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.241+0000 7f0bafe00640 1 --2- 172.21.15.6:0/2754232654 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f0b9006b4d0 0x7f0b9006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:31.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.243+0000 7f0bafe00640 1 --2- 172.21.15.6:0/2754232654 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f0b9006b4d0 0x7f0b9006d990 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f0ba0036000 tx=0x7f0ba0008030 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:31.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.245+0000 7f0ba7e00640 1 -- 172.21.15.6:0/2754232654 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0ba8059120 con 0x7f0bb00cf650 2024-03-04T09:30:31.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.300+0000 7fd8ab000640 1 Processor -- start 2024-03-04T09:30:31.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.301+0000 7fd8ab000640 1 -- start start 2024-03-04T09:30:31.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.302+0000 7fd8ab000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd8a415e4e0 0x7fd8a40d0560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:31.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.302+0000 7fd8ab000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd8a40d2db0 0x7fd8a40b12a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:31.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.302+0000 7fd8ab000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8a40cf730 con 0x7fd8a40b2fa0 2024-03-04T09:30:31.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.302+0000 7fd8ab000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8a40b0b30 con 0x7fd8a40d2db0 2024-03-04T09:30:31.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.302+0000 7fd8ab000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8a40b0e10 con 0x7fd8a415e4e0 2024-03-04T09:30:31.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.302+0000 7fd8a9200640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fd8a40b2fa0 0x7fd8a40d0aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:48426/0 (socket says 172.21.15.6:48426) 2024-03-04T09:30:31.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.302+0000 7fd8a9200640 1 -- 172.21.15.6:0/3367334476 learned_addr learned my addr 172.21.15.6:0/3367334476 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:30:31.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.303+0000 7fd8aa600640 1 --2- 172.21.15.6:0/3367334476 >> v2:172.21.15.146:3300/0 conn(0x7fd8a40d2db0 0x7fd8a40b12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:31.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.303+0000 7fd8a9c00640 1 --2- 172.21.15.6:0/3367334476 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd8a415e4e0 0x7fd8a40d0560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:31.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.303+0000 7fd8a3e00640 1 -- 172.21.15.6:0/3367334476 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2877170274 0 0) 0x7fd8a40cf730 con 0x7fd8a40b2fa0 2024-03-04T09:30:31.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.303+0000 7fd8a3e00640 1 -- 172.21.15.6:0/3367334476 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd88c003520 con 0x7fd8a40b2fa0 2024-03-04T09:30:31.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.303+0000 7fd8a3e00640 1 -- 172.21.15.6:0/3367334476 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1856382185 0 0) 0x7fd88c003520 con 0x7fd8a40b2fa0 2024-03-04T09:30:31.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.303+0000 7fd8a3e00640 1 -- 172.21.15.6:0/3367334476 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd8a415e4e0 msgr2=0x7fd8a40d0560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:31.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.303+0000 7fd8a3e00640 1 --2- 172.21.15.6:0/3367334476 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd8a415e4e0 0x7fd8a40d0560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:31.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.303+0000 7fd8a3e00640 1 -- 172.21.15.6:0/3367334476 >> v2:172.21.15.146:3300/0 conn(0x7fd8a40d2db0 msgr2=0x7fd8a40b12a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:31.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.303+0000 7fd8a3e00640 1 --2- 172.21.15.6:0/3367334476 >> v2:172.21.15.146:3300/0 conn(0x7fd8a40d2db0 0x7fd8a40b12a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:31.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.303+0000 7fd8a3e00640 1 -- 172.21.15.6:0/3367334476 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8a40cfa10 con 0x7fd8a40b2fa0 2024-03-04T09:30:31.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.303+0000 7fd8a3e00640 1 -- 172.21.15.6:0/3367334476 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7fd890001670 con 0x7fd8a40b2fa0 2024-03-04T09:30:31.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.304+0000 7fd8a3e00640 1 -- 172.21.15.6:0/3367334476 >> v1:172.21.15.6:6789/0 conn(0x7fd8a40b2fa0 legacy=0x7fd8a40d0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:31.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.304+0000 7fd8a3e00640 1 --2- 172.21.15.6:0/3367334476 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd88c0039b0 0x7fd88c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:31.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.304+0000 7fd8a3e00640 1 --2- 172.21.15.6:0/3367334476 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd8a415e4e0 0x7fd88c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:31.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.304+0000 7fd8a3e00640 1 --2- 172.21.15.6:0/3367334476 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd88c004d40 0x7fd88c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:31.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.304+0000 7fd8a3e00640 1 -- 172.21.15.6:0/3367334476 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8a40b0b30 con 0x7fd88c004d40 2024-03-04T09:30:31.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.304+0000 7fd8a3e00640 1 -- 172.21.15.6:0/3367334476 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8a40b0e10 con 0x7fd88c0039b0 2024-03-04T09:30:31.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.304+0000 7fd8a3e00640 1 -- 172.21.15.6:0/3367334476 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd88c00b9c0 con 0x7fd8a415e4e0 2024-03-04T09:30:31.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.304+0000 7fd8aa600640 1 --2- 172.21.15.6:0/3367334476 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd88c004d40 0x7fd88c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:31.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.304+0000 7fd8a9200640 1 --2- 172.21.15.6:0/3367334476 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd8a415e4e0 0x7fd88c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:31.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.304+0000 7fd8a9c00640 1 --2- 172.21.15.6:0/3367334476 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd88c0039b0 0x7fd88c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:31.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.305+0000 7fd8aa600640 1 -- 172.21.15.6:0/3367334476 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd8a415e4e0 msgr2=0x7fd88c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:31.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.305+0000 7fd8aa600640 1 --2- 172.21.15.6:0/3367334476 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd8a415e4e0 0x7fd88c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:31.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.305+0000 7fd8aa600640 1 -- 172.21.15.6:0/3367334476 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd88c0039b0 msgr2=0x7fd88c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:31.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.305+0000 7fd8aa600640 1 --2- 172.21.15.6:0/3367334476 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd88c0039b0 0x7fd88c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:31.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.305+0000 7fd8aa600640 1 -- 172.21.15.6:0/3367334476 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8a40cfa10 con 0x7fd88c004d40 2024-03-04T09:30:31.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.305+0000 7fd8a9c00640 1 --2- 172.21.15.6:0/3367334476 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd88c0039b0 0x7fd88c003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T09:30:31.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.307+0000 7fd8aa600640 1 --2- 172.21.15.6:0/3367334476 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd88c004d40 0x7fd88c0051c0 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7fd89c009fc0 tx=0x7fd88c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20496e9268cf509e server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:31.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.307+0000 7fd8a3e00640 1 -- 172.21.15.6:0/3367334476 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd89c039070 con 0x7fd88c004d40 2024-03-04T09:30:31.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.307+0000 7fd8a3e00640 1 -- 172.21.15.6:0/3367334476 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd89c02cc60 con 0x7fd88c004d40 2024-03-04T09:30:31.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.307+0000 7fd8a3e00640 1 -- 172.21.15.6:0/3367334476 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd89c009c10 con 0x7fd88c004d40 2024-03-04T09:30:31.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.308+0000 7fd8ab000640 1 -- 172.21.15.6:0/3367334476 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd88c004d40 msgr2=0x7fd88c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:31.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.308+0000 7fd8ab000640 1 --2- 172.21.15.6:0/3367334476 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd88c004d40 0x7fd88c0051c0 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7fd89c009fc0 tx=0x7fd88c005830 comp rx=0 tx=0).stop 2024-03-04T09:30:31.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.308+0000 7fd8aa600640 1 -- 172.21.15.6:0/3367334476 reap_dead start 2024-03-04T09:30:31.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.308+0000 7fd8ab000640 1 -- 172.21.15.6:0/3367334476 shutdown_connections 2024-03-04T09:30:31.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.308+0000 7fd8ab000640 1 -- 172.21.15.6:0/3367334476 >> 172.21.15.6:0/3367334476 conn(0x7fd8a40d3ea0 msgr2=0x7fd8a40d42b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:31.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.308+0000 7fd8ab000640 1 -- 172.21.15.6:0/3367334476 shutdown_connections 2024-03-04T09:30:31.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.308+0000 7fd8ab000640 1 -- 172.21.15.6:0/3367334476 wait complete. 2024-03-04T09:30:31.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.310+0000 7fd8ab000640 1 Processor -- start 2024-03-04T09:30:31.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.310+0000 7fd8ab000640 1 -- start start 2024-03-04T09:30:31.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.311+0000 7fd8ab000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd88c004d40 0x7fd8a414e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:31.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.311+0000 7fd8ab000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd8a40caf70 0x7fd8a40cb3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:31.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.311+0000 7fd8ab000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd8a4160620 0x7fd8a4160aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:31.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.311+0000 7fd8ab000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8a415ea40 con 0x7fd8a4160620 2024-03-04T09:30:31.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.311+0000 7fd8ab000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8a415e8c0 con 0x7fd8a40caf70 2024-03-04T09:30:31.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.311+0000 7fd8ab000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8a40d11b0 con 0x7fd88c004d40 2024-03-04T09:30:31.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.312+0000 7fd8aa600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd8a4160620 0x7fd8a4160aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:31.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.312+0000 7fd8aa600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd8a4160620 0x7fd8a4160aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:39872/0 (socket says 172.21.15.6:39872) 2024-03-04T09:30:31.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.312+0000 7fd8aa600640 1 -- 172.21.15.6:0/1404458802 learned_addr learned my addr 172.21.15.6:0/1404458802 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:30:31.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.312+0000 7fd8a9c00640 1 --2- 172.21.15.6:0/1404458802 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd88c004d40 0x7fd8a414e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:31.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.312+0000 7fd8aa600640 1 -- 172.21.15.6:0/1404458802 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd88c004d40 msgr2=0x7fd8a414e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:31.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.312+0000 7fd8aa600640 1 --2- 172.21.15.6:0/1404458802 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd88c004d40 0x7fd8a414e500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:31.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.312+0000 7fd8aa600640 1 -- 172.21.15.6:0/1404458802 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd8a40caf70 msgr2=0x7fd8a40cb3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:31.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.312+0000 7fd8aa600640 1 --2- 172.21.15.6:0/1404458802 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd8a40caf70 0x7fd8a40cb3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:31.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.312+0000 7fd8aa600640 1 -- 172.21.15.6:0/1404458802 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8a4161350 con 0x7fd8a4160620 2024-03-04T09:30:31.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.313+0000 7fd8aa600640 1 --2- 172.21.15.6:0/1404458802 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd8a4160620 0x7fd8a4160aa0 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7fd89c02c6d0 tx=0x7fd89c009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:31.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.314+0000 7fd8a1600640 1 -- 172.21.15.6:0/1404458802 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd89c044070 con 0x7fd8a4160620 2024-03-04T09:30:31.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.314+0000 7fd8a1600640 1 -- 172.21.15.6:0/1404458802 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd89c03a040 con 0x7fd8a4160620 2024-03-04T09:30:31.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.314+0000 7fd8a1600640 1 -- 172.21.15.6:0/1404458802 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd89c039030 con 0x7fd8a4160620 2024-03-04T09:30:31.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.314+0000 7fd8ab000640 1 -- 172.21.15.6:0/1404458802 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8a40b7660 con 0x7fd8a4160620 2024-03-04T09:30:31.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.314+0000 7fd8ab000640 1 -- 172.21.15.6:0/1404458802 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8a40b7c20 con 0x7fd8a4160620 2024-03-04T09:30:31.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.315+0000 7fd8ab000640 1 -- 172.21.15.6:0/1404458802 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8a40b7e20 con 0x7fd8a4160620 2024-03-04T09:30:31.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.318+0000 7fd8a1600640 1 -- 172.21.15.6:0/1404458802 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd89c047070 con 0x7fd8a4160620 2024-03-04T09:30:31.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.319+0000 7fd8a1600640 1 --2- 172.21.15.6:0/1404458802 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd88406b550 0x7fd88406da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:31.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.319+0000 7fd8a1600640 1 -- 172.21.15.6:0/1404458802 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd89c0abc90 con 0x7fd8a4160620 2024-03-04T09:30:31.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.319+0000 7fd8a9c00640 1 --2- 172.21.15.6:0/1404458802 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd88406b550 0x7fd88406da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:31.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.321+0000 7fd8a9c00640 1 --2- 172.21.15.6:0/1404458802 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd88406b550 0x7fd88406da10 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fd898006f80 tx=0x7fd898008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:31.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.328+0000 7fd8a1600640 1 -- 172.21.15.6:0/1404458802 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd8a40b7e20 con 0x7fd8a4160620 2024-03-04T09:30:31.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.530+0000 7f0bb7000640 1 -- 172.21.15.6:0/2754232654 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "c"} v 0) v1 -- 0x7f0bb00b2720 con 0x7f0bb00cf650 2024-03-04T09:30:31.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.601+0000 7fd8ab000640 1 -- 172.21.15.6:0/1404458802 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "a"} v 0) v1 -- 0x7fd8a404f330 con 0x7fd8a4160620 2024-03-04T09:30:31.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.728+0000 7f5764200640 1 -- 172.21.15.6:0/951301177 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "d"} v 0) v1 -- 0x7f575c08c0a0 con 0x7f575c0d09d0 2024-03-04T09:30:31.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:31.738+0000 7efd2a800640 1 -- 172.21.15.6:0/1967906821 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "b"} v 0) v1 -- 0x7efd2415fec0 con 0x7efd2415e4e0 2024-03-04T09:30:32.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:32.262+0000 7f0ba7e00640 1 -- 172.21.15.6:0/2754232654 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "c"}]=0 failed mds gid 7092 v61) v1 ==== 95+0+0 (secure 0 0 0) 0x7f0ba805d1c0 con 0x7f0bb00cf650 2024-03-04T09:30:32.259 INFO:teuthology.orchestra.run.smithi006.stderr:failed mds gid 7092 2024-03-04T09:30:32.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:32.266+0000 7f0bb7000640 1 -- 172.21.15.6:0/2754232654 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f0b9006b4d0 msgr2=0x7f0b9006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:32.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:32.266+0000 7f0bb7000640 1 --2- 172.21.15.6:0/2754232654 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f0b9006b4d0 0x7f0b9006d990 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f0ba0036000 tx=0x7f0ba0008030 comp rx=0 tx=0).stop 2024-03-04T09:30:32.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:32.266+0000 7f0bb7000640 1 -- 172.21.15.6:0/2754232654 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0bb00cf650 msgr2=0x7f0bb00cd970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:32.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:32.266+0000 7f0bb7000640 1 --2- 172.21.15.6:0/2754232654 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0bb00cf650 0x7f0bb00cd970 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7f0ba8007e90 tx=0x7f0ba800e4b0 comp rx=0 tx=0).stop 2024-03-04T09:30:32.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:32.266+0000 7f0bb7000640 1 -- 172.21.15.6:0/2754232654 shutdown_connections 2024-03-04T09:30:32.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:32.266+0000 7f0bb7000640 1 --2- 172.21.15.6:0/2754232654 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0bb00cf650 0x7f0bb00cd970 unknown :-1 s=CLOSED pgs=1107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:32.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:32.266+0000 7f0bb7000640 1 --2- 172.21.15.6:0/2754232654 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0bb00af2e0 0x7f0bb009aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:32.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:32.266+0000 7f0bb7000640 1 --2- 172.21.15.6:0/2754232654 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f0b9006b4d0 0x7f0b9006d990 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:32.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:32.266+0000 7f0bb7000640 1 --2- 172.21.15.6:0/2754232654 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0bb006e170 0x7f0bb00b3920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:32.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:32.266+0000 7f0bb7000640 1 -- 172.21.15.6:0/2754232654 >> 172.21.15.6:0/2754232654 conn(0x7f0bb00b1990 msgr2=0x7f0bb00ae8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:32.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:32.267+0000 7f0bb7000640 1 -- 172.21.15.6:0/2754232654 shutdown_connections 2024-03-04T09:30:32.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:32.267+0000 7f0bb7000640 1 -- 172.21.15.6:0/2754232654 wait complete. 2024-03-04T09:30:32.264 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-03-04T09:30:32.267+0000 7f1603800640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-04T09:30:32.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:32.270+0000 7fd8a1600640 1 -- 172.21.15.6:0/1404458802 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "a"}]=0 failed mds gid 7095 v62) v1 ==== 95+0+0 (secure 0 0 0) 0x7fd89c07a520 con 0x7fd8a4160620 2024-03-04T09:30:32.267 INFO:teuthology.orchestra.run.smithi006.stderr:failed mds gid 7095 2024-03-04T09:30:32.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:32.273+0000 7fd8ab000640 1 -- 172.21.15.6:0/1404458802 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd88406b550 msgr2=0x7fd88406da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:32.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:32.273+0000 7fd8ab000640 1 --2- 172.21.15.6:0/1404458802 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd88406b550 0x7fd88406da10 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fd898006f80 tx=0x7fd898008040 comp rx=0 tx=0).stop 2024-03-04T09:30:32.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:32.273+0000 7fd8ab000640 1 -- 172.21.15.6:0/1404458802 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd8a4160620 msgr2=0x7fd8a4160aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:32.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:32.273+0000 7fd8ab000640 1 --2- 172.21.15.6:0/1404458802 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd8a4160620 0x7fd8a4160aa0 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7fd89c02c6d0 tx=0x7fd89c009750 comp rx=0 tx=0).stop 2024-03-04T09:30:32.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:32.274+0000 7fd8ab000640 1 -- 172.21.15.6:0/1404458802 shutdown_connections 2024-03-04T09:30:32.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:32.274+0000 7fd8ab000640 1 --2- 172.21.15.6:0/1404458802 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd8a4160620 0x7fd8a4160aa0 unknown :-1 s=CLOSED pgs=1110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:32.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:32.274+0000 7fd8ab000640 1 --2- 172.21.15.6:0/1404458802 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd8a40caf70 0x7fd8a40cb3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:32.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:32.274+0000 7fd8ab000640 1 --2- 172.21.15.6:0/1404458802 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd88406b550 0x7fd88406da10 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:32.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:32.274+0000 7fd8ab000640 1 --2- 172.21.15.6:0/1404458802 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd88c004d40 0x7fd8a414e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:32.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:32.274+0000 7fd8ab000640 1 -- 172.21.15.6:0/1404458802 >> 172.21.15.6:0/1404458802 conn(0x7fd8a40d3ea0 msgr2=0x7fd8a40b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:32.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:32.274+0000 7fd8ab000640 1 -- 172.21.15.6:0/1404458802 shutdown_connections 2024-03-04T09:30:32.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:32.274+0000 7fd8ab000640 1 -- 172.21.15.6:0/1404458802 wait complete. 2024-03-04T09:30:32.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:32.284+0000 7f5759600640 1 -- 172.21.15.6:0/951301177 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "d"}]=0 failed mds gid 6046 v63) v1 ==== 95+0+0 (secure 0 0 0) 0x7f574807b1a0 con 0x7f575c0d09d0 2024-03-04T09:30:32.281 INFO:teuthology.orchestra.run.smithi006.stderr:failed mds gid 6046 2024-03-04T09:30:32.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:32.287+0000 7f5764200640 1 -- 172.21.15.6:0/951301177 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f573c06b550 msgr2=0x7f573c06da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:32.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:32.287+0000 7f5764200640 1 --2- 172.21.15.6:0/951301177 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f573c06b550 0x7f573c06da10 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f575c0a1e30 tx=0x7f5750009190 comp rx=0 tx=0).stop 2024-03-04T09:30:32.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:32.287+0000 7f5764200640 1 -- 172.21.15.6:0/951301177 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f575c0d09d0 msgr2=0x7f575c0a0e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:32.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:32.287+0000 7f5764200640 1 --2- 172.21.15.6:0/951301177 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f575c0d09d0 0x7f575c0a0e70 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7f574802b9c0 tx=0x7f5748004730 comp rx=0 tx=0).stop 2024-03-04T09:30:32.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:32.287+0000 7f5764200640 1 -- 172.21.15.6:0/951301177 shutdown_connections 2024-03-04T09:30:32.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:32.287+0000 7f5764200640 1 --2- 172.21.15.6:0/951301177 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f573c06b550 0x7f573c06da10 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:32.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:32.287+0000 7f5764200640 1 --2- 172.21.15.6:0/951301177 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f575c0d2e30 0x7f575c0c9d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:32.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:32.287+0000 7f5764200640 1 --2- 172.21.15.6:0/951301177 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f575c0d09d0 0x7f575c0a0e70 unknown :-1 s=CLOSED pgs=1104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:32.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:32.287+0000 7f5764200640 1 --2- 172.21.15.6:0/951301177 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f575c0cf720 0x7f575c15d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:32.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:32.287+0000 7f5764200640 1 -- 172.21.15.6:0/951301177 >> 172.21.15.6:0/951301177 conn(0x7f575c0d4000 msgr2=0x7f575c0b4190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:32.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:32.287+0000 7f5764200640 1 -- 172.21.15.6:0/951301177 shutdown_connections 2024-03-04T09:30:32.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:32.287+0000 7f5764200640 1 -- 172.21.15.6:0/951301177 wait complete. 2024-03-04T09:30:32.291 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-04T09:30:32.292 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-04T09:30:32.293 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-04T09:30:32.293 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-04T09:30:32.304 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-04T09:30:32.304 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-04T09:30:32.308 INFO:tasks.ceph.mds.c:Started 2024-03-04T09:30:32.311 DEBUG:teuthology.parallel:result is None 2024-03-04T09:30:32.312 INFO:tasks.ceph.mds.d:Started 2024-03-04T09:30:32.312 DEBUG:teuthology.parallel:result is None 2024-03-04T09:30:32.352 INFO:tasks.ceph.mds.a:Started 2024-03-04T09:30:32.353 DEBUG:teuthology.parallel:result is None 2024-03-04T09:30:32.465 INFO:tasks.ceph.mds.c.smithi006.stdout:starting mds.c at 2024-03-04T09:30:32.484 INFO:tasks.ceph.mds.d.smithi146.stdout:starting mds.d at 2024-03-04T09:30:32.517 INFO:tasks.ceph.mds.a.smithi006.stdout:starting mds.a at 2024-03-04T09:30:33.257 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-03-04T09:30:33.260+0000 7f1603800640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-04T09:30:33.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:33.264+0000 7efd21600640 1 -- 172.21.15.6:0/1967906821 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "b"}]=0 failed mds gid 5555 v64) v1 ==== 95+0+0 (secure 0 0 0) 0x7efd1005cce0 con 0x7efd2415e4e0 2024-03-04T09:30:33.261 INFO:teuthology.orchestra.run.smithi006.stderr:failed mds gid 5555 2024-03-04T09:30:33.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:33.268+0000 7efd2a800640 1 -- 172.21.15.6:0/1967906821 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7efd0406b520 msgr2=0x7efd0406d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:33.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:33.268+0000 7efd2a800640 1 --2- 172.21.15.6:0/1967906821 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7efd0406b520 0x7efd0406d9e0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7efd18001780 tx=0x7efd18009040 comp rx=0 tx=0).stop 2024-03-04T09:30:33.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:33.268+0000 7efd2a800640 1 -- 172.21.15.6:0/1967906821 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efd2415e4e0 msgr2=0x7efd241603a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:33.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:33.268+0000 7efd2a800640 1 --2- 172.21.15.6:0/1967906821 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efd2415e4e0 0x7efd241603a0 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7efd1000e8a0 tx=0x7efd1000eca0 comp rx=0 tx=0).stop 2024-03-04T09:30:33.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:33.268+0000 7efd2a800640 1 -- 172.21.15.6:0/1967906821 shutdown_connections 2024-03-04T09:30:33.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:33.268+0000 7efd2a800640 1 --2- 172.21.15.6:0/1967906821 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7efd0406b520 0x7efd0406d9e0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:33.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:33.268+0000 7efd2a800640 1 --2- 172.21.15.6:0/1967906821 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efd240cd890 0x7efd240b6a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:33.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:33.268+0000 7efd2a800640 1 --2- 172.21.15.6:0/1967906821 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efd2415e4e0 0x7efd241603a0 unknown :-1 s=CLOSED pgs=1103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:33.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:33.268+0000 7efd2a800640 1 --2- 172.21.15.6:0/1967906821 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efd240b2fa0 0x7efd240cd350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:33.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:33.268+0000 7efd2a800640 1 -- 172.21.15.6:0/1967906821 >> 172.21.15.6:0/1967906821 conn(0x7efd240d3ea0 msgr2=0x7efd2406bf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:33.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:33.269+0000 7efd2a800640 1 -- 172.21.15.6:0/1967906821 shutdown_connections 2024-03-04T09:30:33.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:33.269+0000 7efd2a800640 1 -- 172.21.15.6:0/1967906821 wait complete. 2024-03-04T09:30:33.291 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-04T09:30:33.291 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-04T09:30:33.299 INFO:tasks.ceph.mds.b:Started 2024-03-04T09:30:33.300 DEBUG:teuthology.parallel:result is None 2024-03-04T09:30:33.300 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:30:33.468 INFO:tasks.ceph.mds.b.smithi146.stdout:starting mds.b at 2024-03-04T09:30:33.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:33.539+0000 7f6f32000640 1 Processor -- start 2024-03-04T09:30:33.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:33.539+0000 7f6f32000640 1 -- start start 2024-03-04T09:30:33.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:33.541+0000 7f6f32000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f6f2c15fba0 0x7f6f2c0b5160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:33.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:33.541+0000 7f6f32000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f2c145bc0 con 0x7f6f2c0b56a0 2024-03-04T09:30:33.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:33.541+0000 7f6f32000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f2c160040 con 0x7f6f2c1457e0 2024-03-04T09:30:33.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:33.541+0000 7f6f32000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f2c0b5af0 con 0x7f6f2c15fba0 2024-03-04T09:30:33.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:33.541+0000 7f6f2aa00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f6f2c15fba0 0x7f6f2c0b5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:33.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:33.541+0000 7f6f2aa00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f6f2c15fba0 0x7f6f2c0b5160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:39910/0 (socket says 172.21.15.6:39910) 2024-03-04T09:30:33.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:33.541+0000 7f6f2aa00640 1 -- 172.21.15.6:0/4135449116 learned_addr learned my addr 172.21.15.6:0/4135449116 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:30:33.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:33.542+0000 7f6f2aa00640 1 -- 172.21.15.6:0/4135449116 >> v1:172.21.15.146:6789/0 conn(0x7f6f2c1457e0 legacy=0x7f6f2c0cf5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:33.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:33.542+0000 7f6f2aa00640 1 -- 172.21.15.6:0/4135449116 >> v1:172.21.15.6:6789/0 conn(0x7f6f2c0b56a0 legacy=0x7f6f2c0710b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:33.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:33.542+0000 7f6f2aa00640 1 -- 172.21.15.6:0/4135449116 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f2c160320 con 0x7f6f2c15fba0 2024-03-04T09:30:33.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:33.543+0000 7f6f2aa00640 1 --2- 172.21.15.6:0/4135449116 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6f2c15fba0 0x7f6f2c0b5160 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7f6f18009870 tx=0x7f6f1802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c777e0c000b4461 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:33.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:33.543+0000 7f6f2a000640 1 -- 172.21.15.6:0/4135449116 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f18034070 con 0x7f6f2c15fba0 2024-03-04T09:30:33.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:33.543+0000 7f6f2a000640 1 -- 172.21.15.6:0/4135449116 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f18030860 con 0x7f6f2c15fba0 2024-03-04T09:30:33.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:33.543+0000 7f6f2a000640 1 -- 172.21.15.6:0/4135449116 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f18030c30 con 0x7f6f2c15fba0 2024-03-04T09:30:33.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:33.544+0000 7f6f32000640 1 -- 172.21.15.6:0/4135449116 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6f2c15fba0 msgr2=0x7f6f2c0b5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:33.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:33.544+0000 7f6f32000640 1 --2- 172.21.15.6:0/4135449116 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6f2c15fba0 0x7f6f2c0b5160 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7f6f18009870 tx=0x7f6f1802b9a0 comp rx=0 tx=0).stop 2024-03-04T09:30:33.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:33.544+0000 7f6f32000640 1 -- 172.21.15.6:0/4135449116 shutdown_connections 2024-03-04T09:30:33.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:33.544+0000 7f6f32000640 1 --2- 172.21.15.6:0/4135449116 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6f2c15fba0 0x7f6f2c0b5160 unknown :-1 s=CLOSED pgs=1118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:33.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:33.544+0000 7f6f32000640 1 -- 172.21.15.6:0/4135449116 >> 172.21.15.6:0/4135449116 conn(0x7f6f2c06cf40 msgr2=0x7f6f2c06d320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:33.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:33.544+0000 7f6f32000640 1 -- 172.21.15.6:0/4135449116 shutdown_connections 2024-03-04T09:30:33.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:33.544+0000 7f6f32000640 1 -- 172.21.15.6:0/4135449116 wait complete. 2024-03-04T09:30:33.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:33.546+0000 7f6f32000640 1 Processor -- start 2024-03-04T09:30:33.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:33.546+0000 7f6f32000640 1 -- start start 2024-03-04T09:30:33.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:33.547+0000 7f6f32000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6f2c0b56a0 0x7f6f2c0ae080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:33.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:33.547+0000 7f6f32000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6f2c1457e0 0x7f6f2c13ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:33.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:33.547+0000 7f6f32000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6f2c15fba0 0x7f6f2c0aedf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:33.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:33.547+0000 7f6f32000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f2c0cca70 con 0x7f6f2c1457e0 2024-03-04T09:30:33.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:33.547+0000 7f6f32000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f2c1608e0 con 0x7f6f2c0b56a0 2024-03-04T09:30:33.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:33.547+0000 7f6f32000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f2c0717c0 con 0x7f6f2c15fba0 2024-03-04T09:30:33.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:33.548+0000 7f6f2aa00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6f2c1457e0 0x7f6f2c13ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:33.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:33.548+0000 7f6f2aa00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6f2c1457e0 0x7f6f2c13ced0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:39922/0 (socket says 172.21.15.6:39922) 2024-03-04T09:30:33.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:33.548+0000 7f6f2aa00640 1 -- 172.21.15.6:0/3112840638 learned_addr learned my addr 172.21.15.6:0/3112840638 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:30:33.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:33.548+0000 7f6f2be00640 1 --2- 172.21.15.6:0/3112840638 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6f2c15fba0 0x7f6f2c0aedf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:33.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:33.548+0000 7f6f2b400640 1 --2- 172.21.15.6:0/3112840638 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6f2c0b56a0 0x7f6f2c0ae080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:33.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:33.548+0000 7f6f2aa00640 1 -- 172.21.15.6:0/3112840638 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6f2c15fba0 msgr2=0x7f6f2c0aedf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:33.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:33.548+0000 7f6f2aa00640 1 --2- 172.21.15.6:0/3112840638 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6f2c15fba0 0x7f6f2c0aedf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:33.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:33.548+0000 7f6f2aa00640 1 -- 172.21.15.6:0/3112840638 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6f2c0b56a0 msgr2=0x7f6f2c0ae080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:33.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:33.548+0000 7f6f2aa00640 1 --2- 172.21.15.6:0/3112840638 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6f2c0b56a0 0x7f6f2c0ae080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:33.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:33.548+0000 7f6f2aa00640 1 -- 172.21.15.6:0/3112840638 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f2c0af330 con 0x7f6f2c1457e0 2024-03-04T09:30:33.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:33.549+0000 7f6f2aa00640 1 --2- 172.21.15.6:0/3112840638 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6f2c1457e0 0x7f6f2c13ced0 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f6f1802b9e0 tx=0x7f6f180046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:33.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:33.549+0000 7f6f23400640 1 -- 172.21.15.6:0/3112840638 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f18034040 con 0x7f6f2c1457e0 2024-03-04T09:30:33.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:33.549+0000 7f6f23400640 1 -- 172.21.15.6:0/3112840638 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f180433f0 con 0x7f6f2c1457e0 2024-03-04T09:30:33.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:33.549+0000 7f6f23400640 1 -- 172.21.15.6:0/3112840638 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f180437e0 con 0x7f6f2c1457e0 2024-03-04T09:30:33.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:33.549+0000 7f6f32000640 1 -- 172.21.15.6:0/3112840638 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f2c0ae660 con 0x7f6f2c1457e0 2024-03-04T09:30:33.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:33.549+0000 7f6f32000640 1 -- 172.21.15.6:0/3112840638 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f2c1435c0 con 0x7f6f2c1457e0 2024-03-04T09:30:33.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:33.550+0000 7f6f32000640 1 -- 172.21.15.6:0/3112840638 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f2c0692a0 con 0x7f6f2c1457e0 2024-03-04T09:30:33.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:33.551+0000 7f6f23400640 1 -- 172.21.15.6:0/3112840638 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6f1804a060 con 0x7f6f2c1457e0 2024-03-04T09:30:33.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:33.552+0000 7f6f23400640 1 --2- 172.21.15.6:0/3112840638 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f6f0c06b4d0 0x7f6f0c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:33.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:33.552+0000 7f6f23400640 1 -- 172.21.15.6:0/3112840638 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f6f180ac590 con 0x7f6f2c1457e0 2024-03-04T09:30:33.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:33.553+0000 7f6f2b400640 1 --2- 172.21.15.6:0/3112840638 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f6f0c06b4d0 0x7f6f0c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:33.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:33.554+0000 7f6f2b400640 1 --2- 172.21.15.6:0/3112840638 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f6f0c06b4d0 0x7f6f0c06d990 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f6f1c000f60 tx=0x7f6f1c009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:33.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:33.558+0000 7f6f23400640 1 -- 172.21.15.6:0/3112840638 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6f1803e070 con 0x7f6f2c1457e0 2024-03-04T09:30:33.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:33.797+0000 7f6f32000640 1 -- 172.21.15.6:0/3112840638 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6f2c06d8a0 con 0x7f6f2c1457e0 2024-03-04T09:30:33.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:33.798+0000 7f6f23400640 1 -- 172.21.15.6:0/3112840638 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) v1 ==== 94+0+4179 (secure 0 0 0) 0x7f6f1804be90 con 0x7f6f2c1457e0 2024-03-04T09:30:33.795 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:30:33.795 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":65,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5654,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/1370302030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":1370302030},{"type":"v1","addr":"172.21.15.146:6835","nonce":1370302030}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":7284,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/1583413840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1583413840},{"type":"v1","addr":"172.21.15.6:6833","nonce":1583413840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:30:20.660103+0000","modified":"2024-03-04T09:30:33.272685+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":98,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7296},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7296":{"gid":7296,"name":"a","rank":0,"incarnation":65,"state":"up:replay","state_seq":1,"addr":"172.21.15.6:6835/1732383836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1732383836},{"type":"v1","addr":"172.21.15.6:6835","nonce":1732383836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":7}]} 2024-03-04T09:30:33.796 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 65 2024-03-04T09:30:33.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:33.801+0000 7f6f32000640 1 -- 172.21.15.6:0/3112840638 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f6f0c06b4d0 msgr2=0x7f6f0c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:33.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:33.801+0000 7f6f32000640 1 --2- 172.21.15.6:0/3112840638 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f6f0c06b4d0 0x7f6f0c06d990 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f6f1c000f60 tx=0x7f6f1c009040 comp rx=0 tx=0).stop 2024-03-04T09:30:33.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:33.801+0000 7f6f32000640 1 -- 172.21.15.6:0/3112840638 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6f2c1457e0 msgr2=0x7f6f2c13ced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:33.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:33.801+0000 7f6f32000640 1 --2- 172.21.15.6:0/3112840638 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6f2c1457e0 0x7f6f2c13ced0 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f6f1802b9e0 tx=0x7f6f180046c0 comp rx=0 tx=0).stop 2024-03-04T09:30:33.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:33.801+0000 7f6f32000640 1 -- 172.21.15.6:0/3112840638 shutdown_connections 2024-03-04T09:30:33.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:33.801+0000 7f6f32000640 1 --2- 172.21.15.6:0/3112840638 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f6f0c06b4d0 0x7f6f0c06d990 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:33.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:33.801+0000 7f6f32000640 1 --2- 172.21.15.6:0/3112840638 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6f2c15fba0 0x7f6f2c0aedf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:33.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:33.801+0000 7f6f32000640 1 --2- 172.21.15.6:0/3112840638 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6f2c1457e0 0x7f6f2c13ced0 unknown :-1 s=CLOSED pgs=1119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:33.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:33.801+0000 7f6f32000640 1 --2- 172.21.15.6:0/3112840638 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6f2c0b56a0 0x7f6f2c0ae080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:33.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:33.801+0000 7f6f32000640 1 -- 172.21.15.6:0/3112840638 >> 172.21.15.6:0/3112840638 conn(0x7f6f2c06cf40 msgr2=0x7f6f2c071b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:33.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:33.801+0000 7f6f32000640 1 -- 172.21.15.6:0/3112840638 shutdown_connections 2024-03-04T09:30:33.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:33.802+0000 7f6f32000640 1 -- 172.21.15.6:0/3112840638 wait complete. 2024-03-04T09:30:34.221 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 65, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:30:20.660103+0000', 'modified': '2024-03-04T09:30:33.272685+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 98, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7296}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7296': {'gid': 7296, 'name': 'a', 'rank': 0, 'incarnation': 65, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.6:6835/1732383836', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1732383836}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1732383836}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-04T09:30:34.221 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7296:up:replay 2024-03-04T09:30:35.222 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:30:35.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:35.446+0000 7fc92e000640 1 Processor -- start 2024-03-04T09:30:35.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:35.446+0000 7fc92e000640 1 -- start start 2024-03-04T09:30:35.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:35.447+0000 7fc92e000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc92815f630 0x7fc928161ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:35.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:35.447+0000 7fc92e000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc9280d1be0 0x7fc9280d2070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:35.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:35.447+0000 7fc92e000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9280b50d0 con 0x7fc9280b4b30 2024-03-04T09:30:35.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:35.447+0000 7fc92e000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9280cf450 con 0x7fc92815f630 2024-03-04T09:30:35.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:35.447+0000 7fc92e000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9280cf6a0 con 0x7fc9280d1be0 2024-03-04T09:30:35.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:35.448+0000 7fc92cc00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc92815f630 0x7fc928161ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:35.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:35.448+0000 7fc92cc00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc92815f630 0x7fc928161ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:39938/0 (socket says 172.21.15.6:39938) 2024-03-04T09:30:35.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:35.448+0000 7fc92cc00640 1 -- 172.21.15.6:0/3464514086 learned_addr learned my addr 172.21.15.6:0/3464514086 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:30:35.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:35.448+0000 7fc92cc00640 1 -- 172.21.15.6:0/3464514086 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc9280d1be0 msgr2=0x7fc9280d2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:35.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:35.448+0000 7fc92d600640 1 --2- 172.21.15.6:0/3464514086 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc9280d1be0 0x7fc9280d2070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:35.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:35.448+0000 7fc92cc00640 1 --2- 172.21.15.6:0/3464514086 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc9280d1be0 0x7fc9280d2070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:35.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:35.448+0000 7fc92cc00640 1 -- 172.21.15.6:0/3464514086 >> v1:172.21.15.146:6789/0 conn(0x7fc9280b4b30 legacy=0x7fc9281621e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:35.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:35.448+0000 7fc92cc00640 1 -- 172.21.15.6:0/3464514086 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9280cf980 con 0x7fc92815f630 2024-03-04T09:30:35.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:35.448+0000 7fc92d600640 1 --2- 172.21.15.6:0/3464514086 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc9280d1be0 0x7fc9280d2070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:30:35.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:35.449+0000 7fc92cc00640 1 --2- 172.21.15.6:0/3464514086 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc92815f630 0x7fc928161ca0 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7fc9100098a0 tx=0x7fc91002b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc341d351205a035 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:35.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:35.450+0000 7fc927400640 1 -- 172.21.15.6:0/3464514086 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc910038070 con 0x7fc92815f630 2024-03-04T09:30:35.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:35.450+0000 7fc927400640 1 -- 172.21.15.6:0/3464514086 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc910031740 con 0x7fc92815f630 2024-03-04T09:30:35.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:35.450+0000 7fc927400640 1 -- 172.21.15.6:0/3464514086 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc91002f050 con 0x7fc92815f630 2024-03-04T09:30:35.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:35.450+0000 7fc92e000640 1 -- 172.21.15.6:0/3464514086 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc92815f630 msgr2=0x7fc928161ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:35.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:35.450+0000 7fc92e000640 1 --2- 172.21.15.6:0/3464514086 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc92815f630 0x7fc928161ca0 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7fc9100098a0 tx=0x7fc91002b7e0 comp rx=0 tx=0).stop 2024-03-04T09:30:35.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:35.450+0000 7fc92e000640 1 -- 172.21.15.6:0/3464514086 shutdown_connections 2024-03-04T09:30:35.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:35.450+0000 7fc92e000640 1 --2- 172.21.15.6:0/3464514086 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc9280d1be0 0x7fc9280d2070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:35.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:35.450+0000 7fc92e000640 1 --2- 172.21.15.6:0/3464514086 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc92815f630 0x7fc928161ca0 unknown :-1 s=CLOSED pgs=1120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:35.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:35.450+0000 7fc92e000640 1 -- 172.21.15.6:0/3464514086 >> 172.21.15.6:0/3464514086 conn(0x7fc9280d3e30 msgr2=0x7fc9280d4240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:35.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:35.451+0000 7fc92e000640 1 -- 172.21.15.6:0/3464514086 shutdown_connections 2024-03-04T09:30:35.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:35.451+0000 7fc92e000640 1 -- 172.21.15.6:0/3464514086 wait complete. 2024-03-04T09:30:35.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:35.452+0000 7fc92e000640 1 Processor -- start 2024-03-04T09:30:35.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:35.452+0000 7fc92e000640 1 -- start start 2024-03-04T09:30:35.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:35.453+0000 7fc92e000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc9280b4b30 0x7fc92806d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:35.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:35.453+0000 7fc92e000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc9280d1be0 0x7fc9280ccb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:35.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:35.453+0000 7fc92e000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc92815f630 0x7fc9280cd070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:35.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:35.453+0000 7fc92e000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9281628f0 con 0x7fc9280b4b30 2024-03-04T09:30:35.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:35.453+0000 7fc92e000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9280d4c40 con 0x7fc9280d1be0 2024-03-04T09:30:35.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:35.453+0000 7fc92e000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc928162a70 con 0x7fc92815f630 2024-03-04T09:30:35.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:35.454+0000 7fc92cc00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc9280b4b30 0x7fc92806d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:35.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:35.454+0000 7fc92cc00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc9280b4b30 0x7fc92806d220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:39944/0 (socket says 172.21.15.6:39944) 2024-03-04T09:30:35.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:35.454+0000 7fc92cc00640 1 -- 172.21.15.6:0/2435240426 learned_addr learned my addr 172.21.15.6:0/2435240426 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:30:35.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:35.454+0000 7fc927e00640 1 --2- 172.21.15.6:0/2435240426 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc9280d1be0 0x7fc9280ccb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:35.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:35.454+0000 7fc92cc00640 1 -- 172.21.15.6:0/2435240426 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc92815f630 msgr2=0x7fc9280cd070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:35.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:35.454+0000 7fc92cc00640 1 --2- 172.21.15.6:0/2435240426 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc92815f630 0x7fc9280cd070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:35.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:35.454+0000 7fc92cc00640 1 -- 172.21.15.6:0/2435240426 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc9280d1be0 msgr2=0x7fc9280ccb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:35.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:35.454+0000 7fc92cc00640 1 --2- 172.21.15.6:0/2435240426 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc9280d1be0 0x7fc9280ccb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:35.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:35.454+0000 7fc92cc00640 1 -- 172.21.15.6:0/2435240426 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc928144350 con 0x7fc9280b4b30 2024-03-04T09:30:35.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:35.455+0000 7fc92cc00640 1 --2- 172.21.15.6:0/2435240426 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc9280b4b30 0x7fc92806d220 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7fc9100098a0 tx=0x7fc9100046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:35.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:35.455+0000 7fc924c00640 1 -- 172.21.15.6:0/2435240426 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc910042070 con 0x7fc9280b4b30 2024-03-04T09:30:35.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:35.455+0000 7fc924c00640 1 -- 172.21.15.6:0/2435240426 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc910033a70 con 0x7fc9280b4b30 2024-03-04T09:30:35.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:35.455+0000 7fc924c00640 1 -- 172.21.15.6:0/2435240426 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc910038040 con 0x7fc9280b4b30 2024-03-04T09:30:35.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:35.455+0000 7fc92e000640 1 -- 172.21.15.6:0/2435240426 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9280cd6d0 con 0x7fc9280b4b30 2024-03-04T09:30:35.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:35.455+0000 7fc92e000640 1 -- 172.21.15.6:0/2435240426 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9280cdc90 con 0x7fc9280b4b30 2024-03-04T09:30:35.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:35.456+0000 7fc92e000640 1 -- 172.21.15.6:0/2435240426 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc928144520 con 0x7fc9280b4b30 2024-03-04T09:30:35.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:35.457+0000 7fc924c00640 1 -- 172.21.15.6:0/2435240426 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc910045070 con 0x7fc9280b4b30 2024-03-04T09:30:35.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:35.458+0000 7fc924c00640 1 --2- 172.21.15.6:0/2435240426 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc90406b500 0x7fc90406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:35.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:35.458+0000 7fc924c00640 1 -- 172.21.15.6:0/2435240426 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5881+0+0 (secure 0 0 0) 0x7fc9100abd70 con 0x7fc9280b4b30 2024-03-04T09:30:35.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:35.459+0000 7fc927e00640 1 --2- 172.21.15.6:0/2435240426 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc90406b500 0x7fc90406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:35.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:35.460+0000 7fc927e00640 1 --2- 172.21.15.6:0/2435240426 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc90406b500 0x7fc90406d9c0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fc9280b58b0 tx=0x7fc918007240 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:35.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:35.463+0000 7fc924c00640 1 -- 172.21.15.6:0/2435240426 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc91004c070 con 0x7fc9280b4b30 2024-03-04T09:30:35.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:35.703+0000 7fc92e000640 1 -- 172.21.15.6:0/2435240426 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc92808d950 con 0x7fc9280b4b30 2024-03-04T09:30:35.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:35.705+0000 7fc924c00640 1 -- 172.21.15.6:0/2435240426 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+4970 (secure 0 0 0) 0x7fc91007ab30 con 0x7fc9280b4b30 2024-03-04T09:30:35.702 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:30:35.702 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":67,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5654,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/1370302030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":1370302030},{"type":"v1","addr":"172.21.15.146:6835","nonce":1370302030}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":6163,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/809822004","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":809822004},{"type":"v1","addr":"172.21.15.146:6837","nonce":809822004}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":66},{"gid":7284,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/1583413840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1583413840},{"type":"v1","addr":"172.21.15.6:6833","nonce":1583413840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:30:20.660103+0000","modified":"2024-03-04T09:30:34.299801+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":98,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7296},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7296":{"gid":7296,"name":"a","rank":0,"incarnation":65,"state":"up:rejoin","state_seq":3,"addr":"172.21.15.6:6835/1732383836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1732383836},{"type":"v1","addr":"172.21.15.6:6835","nonce":1732383836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":7}]} 2024-03-04T09:30:35.702 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 67 2024-03-04T09:30:35.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:35.707+0000 7fc92e000640 1 -- 172.21.15.6:0/2435240426 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc90406b500 msgr2=0x7fc90406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:35.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:35.707+0000 7fc92e000640 1 --2- 172.21.15.6:0/2435240426 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc90406b500 0x7fc90406d9c0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fc9280b58b0 tx=0x7fc918007240 comp rx=0 tx=0).stop 2024-03-04T09:30:35.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:35.707+0000 7fc92e000640 1 -- 172.21.15.6:0/2435240426 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc9280b4b30 msgr2=0x7fc92806d220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:35.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:35.707+0000 7fc92e000640 1 --2- 172.21.15.6:0/2435240426 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc9280b4b30 0x7fc92806d220 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7fc9100098a0 tx=0x7fc9100046c0 comp rx=0 tx=0).stop 2024-03-04T09:30:35.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:35.707+0000 7fc92e000640 1 -- 172.21.15.6:0/2435240426 shutdown_connections 2024-03-04T09:30:35.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:35.707+0000 7fc92e000640 1 --2- 172.21.15.6:0/2435240426 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc90406b500 0x7fc90406d9c0 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:35.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:35.707+0000 7fc92e000640 1 --2- 172.21.15.6:0/2435240426 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc92815f630 0x7fc9280cd070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:35.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:35.707+0000 7fc92e000640 1 --2- 172.21.15.6:0/2435240426 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc9280d1be0 0x7fc9280ccb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:35.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:35.707+0000 7fc92e000640 1 --2- 172.21.15.6:0/2435240426 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc9280b4b30 0x7fc92806d220 unknown :-1 s=CLOSED pgs=1121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:35.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:35.707+0000 7fc92e000640 1 -- 172.21.15.6:0/2435240426 >> 172.21.15.6:0/2435240426 conn(0x7fc9280d3e30 msgr2=0x7fc9280d08b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:35.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:35.708+0000 7fc92e000640 1 -- 172.21.15.6:0/2435240426 shutdown_connections 2024-03-04T09:30:35.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:35.708+0000 7fc92e000640 1 -- 172.21.15.6:0/2435240426 wait complete. 2024-03-04T09:30:36.126 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 67, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:30:20.660103+0000', 'modified': '2024-03-04T09:30:34.299801+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 98, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7296}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7296': {'gid': 7296, 'name': 'a', 'rank': 0, 'incarnation': 65, 'state': 'up:rejoin', 'state_seq': 3, 'addr': '172.21.15.6:6835/1732383836', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1732383836}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1732383836}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-04T09:30:36.127 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7296:up:rejoin 2024-03-04T09:30:37.128 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:30:37.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:37.360+0000 7fdd8f200640 1 Processor -- start 2024-03-04T09:30:37.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:37.360+0000 7fdd8f200640 1 -- start start 2024-03-04T09:30:37.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:37.362+0000 7fdd8f200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fdd90166fc0 0x7fdd901693b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:37.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:37.362+0000 7fdd8f200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd90162ac0 con 0x7fdd900b4b30 2024-03-04T09:30:37.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:37.362+0000 7fdd8f200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd900b50d0 con 0x7fdd9015f630 2024-03-04T09:30:37.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:37.362+0000 7fdd8f200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd900cf450 con 0x7fdd90166fc0 2024-03-04T09:30:37.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:37.362+0000 7fdd8e800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fdd90166fc0 0x7fdd901693b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:37.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:37.362+0000 7fdd8e800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fdd90166fc0 0x7fdd901693b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:39954/0 (socket says 172.21.15.6:39954) 2024-03-04T09:30:37.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:37.362+0000 7fdd8e800640 1 -- 172.21.15.6:0/2694367005 learned_addr learned my addr 172.21.15.6:0/2694367005 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:30:37.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:37.363+0000 7fdd8e800640 1 -- 172.21.15.6:0/2694367005 >> v1:172.21.15.146:6789/0 conn(0x7fdd9015f630 legacy=0x7fdd90161ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:37.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:37.363+0000 7fdd8e800640 1 -- 172.21.15.6:0/2694367005 >> v1:172.21.15.6:6789/0 conn(0x7fdd900b4b30 legacy=0x7fdd901623b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:37.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:37.363+0000 7fdd8e800640 1 -- 172.21.15.6:0/2694367005 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd900cf6a0 con 0x7fdd90166fc0 2024-03-04T09:30:37.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:37.364+0000 7fdd8e800640 1 --2- 172.21.15.6:0/2694367005 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdd90166fc0 0x7fdd901693b0 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7fdd880098a0 tx=0x7fdd8802b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef641751eae17b65 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:37.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:37.364+0000 7fdd8d400640 1 -- 172.21.15.6:0/2694367005 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd880317c0 con 0x7fdd90166fc0 2024-03-04T09:30:37.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:37.364+0000 7fdd8d400640 1 -- 172.21.15.6:0/2694367005 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd88037070 con 0x7fdd90166fc0 2024-03-04T09:30:37.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:37.364+0000 7fdd8d400640 1 -- 172.21.15.6:0/2694367005 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd88031b80 con 0x7fdd90166fc0 2024-03-04T09:30:37.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:37.365+0000 7fdd8f200640 1 -- 172.21.15.6:0/2694367005 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdd90166fc0 msgr2=0x7fdd901693b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:37.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:37.365+0000 7fdd8f200640 1 --2- 172.21.15.6:0/2694367005 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdd90166fc0 0x7fdd901693b0 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7fdd880098a0 tx=0x7fdd8802b860 comp rx=0 tx=0).stop 2024-03-04T09:30:37.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:37.365+0000 7fdd8f200640 1 -- 172.21.15.6:0/2694367005 shutdown_connections 2024-03-04T09:30:37.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:37.365+0000 7fdd8f200640 1 --2- 172.21.15.6:0/2694367005 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdd90166fc0 0x7fdd901693b0 unknown :-1 s=CLOSED pgs=1123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:37.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:37.365+0000 7fdd8f200640 1 -- 172.21.15.6:0/2694367005 >> 172.21.15.6:0/2694367005 conn(0x7fdd900d3e30 msgr2=0x7fdd900d4240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:37.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:37.365+0000 7fdd8f200640 1 -- 172.21.15.6:0/2694367005 shutdown_connections 2024-03-04T09:30:37.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:37.365+0000 7fdd8f200640 1 -- 172.21.15.6:0/2694367005 wait complete. 2024-03-04T09:30:37.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:37.366+0000 7fdd8f200640 1 Processor -- start 2024-03-04T09:30:37.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:37.367+0000 7fdd8f200640 1 -- start start 2024-03-04T09:30:37.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:37.368+0000 7fdd8f200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdd900b4b30 0x7fdd9015b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:37.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:37.368+0000 7fdd8f200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdd9015f630 0x7fdd90158700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:37.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:37.368+0000 7fdd8f200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdd90166fc0 0x7fdd90158c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:37.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:37.368+0000 7fdd8f200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd900d4c40 con 0x7fdd9015f630 2024-03-04T09:30:37.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:37.368+0000 7fdd8f200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd900cfc60 con 0x7fdd900b4b30 2024-03-04T09:30:37.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:37.368+0000 7fdd8f200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd900d2bf0 con 0x7fdd90166fc0 2024-03-04T09:30:37.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:37.368+0000 7fdd85400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdd9015f630 0x7fdd90158700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:37.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:37.368+0000 7fdd85400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdd9015f630 0x7fdd90158700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:39966/0 (socket says 172.21.15.6:39966) 2024-03-04T09:30:37.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:37.368+0000 7fdd85400640 1 -- 172.21.15.6:0/887943824 learned_addr learned my addr 172.21.15.6:0/887943824 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:30:37.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:37.368+0000 7fdd8de00640 1 --2- 172.21.15.6:0/887943824 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdd900b4b30 0x7fdd9015b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:37.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:37.368+0000 7fdd8e800640 1 --2- 172.21.15.6:0/887943824 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdd90166fc0 0x7fdd90158c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:37.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:37.369+0000 7fdd85400640 1 -- 172.21.15.6:0/887943824 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdd90166fc0 msgr2=0x7fdd90158c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:37.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:37.369+0000 7fdd85400640 1 --2- 172.21.15.6:0/887943824 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdd90166fc0 0x7fdd90158c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:37.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:37.369+0000 7fdd85400640 1 -- 172.21.15.6:0/887943824 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdd900b4b30 msgr2=0x7fdd9015b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:37.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:37.369+0000 7fdd85400640 1 --2- 172.21.15.6:0/887943824 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdd900b4b30 0x7fdd9015b7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:37.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:37.369+0000 7fdd85400640 1 -- 172.21.15.6:0/887943824 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd901420e0 con 0x7fdd9015f630 2024-03-04T09:30:37.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:37.369+0000 7fdd8de00640 1 --2- 172.21.15.6:0/887943824 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdd900b4b30 0x7fdd9015b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:30:37.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:37.369+0000 7fdd8e800640 1 --2- 172.21.15.6:0/887943824 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdd90166fc0 0x7fdd90158c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:30:37.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:37.369+0000 7fdd85400640 1 --2- 172.21.15.6:0/887943824 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdd9015f630 0x7fdd90158700 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7fdd7800c3f0 tx=0x7fdd7800c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:37.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:37.369+0000 7fdd86a00640 1 -- 172.21.15.6:0/887943824 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd780052d0 con 0x7fdd9015f630 2024-03-04T09:30:37.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:37.369+0000 7fdd86a00640 1 -- 172.21.15.6:0/887943824 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd78005490 con 0x7fdd9015f630 2024-03-04T09:30:37.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:37.369+0000 7fdd86a00640 1 -- 172.21.15.6:0/887943824 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd78007260 con 0x7fdd9015f630 2024-03-04T09:30:37.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:37.369+0000 7fdd8f200640 1 -- 172.21.15.6:0/887943824 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd9015bd10 con 0x7fdd9015f630 2024-03-04T09:30:37.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:37.369+0000 7fdd8f200640 1 -- 172.21.15.6:0/887943824 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd90140070 con 0x7fdd9015f630 2024-03-04T09:30:37.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:37.371+0000 7fdd8f200640 1 -- 172.21.15.6:0/887943824 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd9004f330 con 0x7fdd9015f630 2024-03-04T09:30:37.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:37.371+0000 7fdd86a00640 1 -- 172.21.15.6:0/887943824 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdd78007400 con 0x7fdd9015f630 2024-03-04T09:30:37.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:37.372+0000 7fdd86a00640 1 --2- 172.21.15.6:0/887943824 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fdd6c06b4d0 0x7fdd6c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:37.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:37.372+0000 7fdd86a00640 1 -- 172.21.15.6:0/887943824 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5881+0+0 (secure 0 0 0) 0x7fdd78089700 con 0x7fdd9015f630 2024-03-04T09:30:37.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:37.373+0000 7fdd8de00640 1 --2- 172.21.15.6:0/887943824 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fdd6c06b4d0 0x7fdd6c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:37.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:37.374+0000 7fdd8de00640 1 --2- 172.21.15.6:0/887943824 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fdd6c06b4d0 0x7fdd6c06d990 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fdd80007eb0 tx=0x7fdd800073c0 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:37.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:37.378+0000 7fdd86a00640 1 -- 172.21.15.6:0/887943824 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdd78058560 con 0x7fdd9015f630 2024-03-04T09:30:37.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:37.617+0000 7fdd8f200640 1 -- 172.21.15.6:0/887943824 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdd9008d950 con 0x7fdd9015f630 2024-03-04T09:30:37.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:37.618+0000 7fdd86a00640 1 -- 172.21.15.6:0/887943824 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) v1 ==== 94+0+4977 (secure 0 0 0) 0x7fdd78057e60 con 0x7fdd9015f630 2024-03-04T09:30:37.616 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:30:37.616 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":68,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5654,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/1370302030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":1370302030},{"type":"v1","addr":"172.21.15.146:6835","nonce":1370302030}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":6163,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/809822004","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":809822004},{"type":"v1","addr":"172.21.15.146:6837","nonce":809822004}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":66},{"gid":7284,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/1583413840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1583413840},{"type":"v1","addr":"172.21.15.6:6833","nonce":1583413840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64}],"filesystems":[{"mdsmap":{"epoch":68,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:30:20.660103+0000","modified":"2024-03-04T09:30:36.289706+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":98,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7296},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7296":{"gid":7296,"name":"a","rank":0,"incarnation":65,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1732383836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1732383836},{"type":"v1","addr":"172.21.15.6:6835","nonce":1732383836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7296,"qdb_cluster":[7296]},"id":7}]} 2024-03-04T09:30:37.616 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 68 2024-03-04T09:30:37.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:37.622+0000 7fdd8f200640 1 -- 172.21.15.6:0/887943824 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fdd6c06b4d0 msgr2=0x7fdd6c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:37.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:37.622+0000 7fdd8f200640 1 --2- 172.21.15.6:0/887943824 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fdd6c06b4d0 0x7fdd6c06d990 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fdd80007eb0 tx=0x7fdd800073c0 comp rx=0 tx=0).stop 2024-03-04T09:30:37.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:37.622+0000 7fdd8f200640 1 -- 172.21.15.6:0/887943824 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdd9015f630 msgr2=0x7fdd90158700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:37.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:37.622+0000 7fdd8f200640 1 --2- 172.21.15.6:0/887943824 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdd9015f630 0x7fdd90158700 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7fdd7800c3f0 tx=0x7fdd7800c8b0 comp rx=0 tx=0).stop 2024-03-04T09:30:37.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:37.622+0000 7fdd8f200640 1 -- 172.21.15.6:0/887943824 shutdown_connections 2024-03-04T09:30:37.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:37.622+0000 7fdd8f200640 1 --2- 172.21.15.6:0/887943824 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fdd6c06b4d0 0x7fdd6c06d990 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:37.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:37.622+0000 7fdd8f200640 1 --2- 172.21.15.6:0/887943824 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdd90166fc0 0x7fdd90158c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:37.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:37.622+0000 7fdd8f200640 1 --2- 172.21.15.6:0/887943824 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdd9015f630 0x7fdd90158700 unknown :-1 s=CLOSED pgs=1124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:37.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:37.622+0000 7fdd8f200640 1 --2- 172.21.15.6:0/887943824 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdd900b4b30 0x7fdd9015b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:37.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:37.622+0000 7fdd8f200640 1 -- 172.21.15.6:0/887943824 >> 172.21.15.6:0/887943824 conn(0x7fdd900d3e30 msgr2=0x7fdd900d4210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:37.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:37.622+0000 7fdd8f200640 1 -- 172.21.15.6:0/887943824 shutdown_connections 2024-03-04T09:30:37.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:37.622+0000 7fdd8f200640 1 -- 172.21.15.6:0/887943824 wait complete. 2024-03-04T09:30:38.040 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 68, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:30:20.660103+0000', 'modified': '2024-03-04T09:30:36.289706+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 98, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7296}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7296': {'gid': 7296, 'name': 'a', 'rank': 0, 'incarnation': 65, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1732383836', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1732383836}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1732383836}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7296, 'qdb_cluster': [7296]} 2024-03-04T09:30:38.040 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-04T09:30:38.040 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-04T09:30:38.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.270+0000 7f0e6be00640 1 Processor -- start 2024-03-04T09:30:38.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.270+0000 7f0e6be00640 1 -- start start 2024-03-04T09:30:38.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.271+0000 7f0e6be00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f0e6c0b5560 0x7f0e6c15fc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:38.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.271+0000 7f0e6be00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0e6c0d1cc0 0x7f0e6c0d24c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:38.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.271+0000 7f0e6be00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e6c160170 con 0x7f0e6c0d2a00 2024-03-04T09:30:38.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.271+0000 7f0e6be00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e6c0d12e0 con 0x7f0e6c0b5560 2024-03-04T09:30:38.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.271+0000 7f0e6be00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e6c0ced90 con 0x7f0e6c0d1cc0 2024-03-04T09:30:38.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.271+0000 7f0e6aa00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f0e6c0b5560 0x7f0e6c15fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:38.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.271+0000 7f0e6b400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f0e6c0d2a00 0x7f0e6c0cf220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:48462/0 (socket says 172.21.15.6:48462) 2024-03-04T09:30:38.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.271+0000 7f0e6b400640 1 -- 172.21.15.6:0/1422378765 learned_addr learned my addr 172.21.15.6:0/1422378765 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:30:38.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.272+0000 7f0e63e00640 1 --2- 172.21.15.6:0/1422378765 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0e6c0d1cc0 0x7f0e6c0d24c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:38.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.272+0000 7f0e6aa00640 1 -- 172.21.15.6:0/1422378765 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0e6c0d1cc0 msgr2=0x7f0e6c0d24c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:38.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.272+0000 7f0e6aa00640 1 --2- 172.21.15.6:0/1422378765 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0e6c0d1cc0 0x7f0e6c0d24c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:38.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.272+0000 7f0e6aa00640 1 -- 172.21.15.6:0/1422378765 >> v1:172.21.15.6:6789/0 conn(0x7f0e6c0d2a00 legacy=0x7f0e6c0cf220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:38.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.272+0000 7f0e6aa00640 1 -- 172.21.15.6:0/1422378765 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e6c0cea60 con 0x7f0e6c0b5560 2024-03-04T09:30:38.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.273+0000 7f0e6aa00640 1 --2- 172.21.15.6:0/1422378765 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0e6c0b5560 0x7f0e6c15fc30 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7f0e5c0097a0 tx=0x7f0e5c02b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9574e87a06fcb27 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:38.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.274+0000 7f0e6a000640 1 -- 172.21.15.6:0/1422378765 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e5c031780 con 0x7f0e6c0b5560 2024-03-04T09:30:38.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.274+0000 7f0e6a000640 1 -- 172.21.15.6:0/1422378765 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e5c037070 con 0x7f0e6c0b5560 2024-03-04T09:30:38.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.274+0000 7f0e6a000640 1 -- 172.21.15.6:0/1422378765 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e5c031b40 con 0x7f0e6c0b5560 2024-03-04T09:30:38.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.274+0000 7f0e6be00640 1 -- 172.21.15.6:0/1422378765 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0e6c0b5560 msgr2=0x7f0e6c15fc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:38.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.274+0000 7f0e6be00640 1 --2- 172.21.15.6:0/1422378765 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0e6c0b5560 0x7f0e6c15fc30 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7f0e5c0097a0 tx=0x7f0e5c02b820 comp rx=0 tx=0).stop 2024-03-04T09:30:38.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.274+0000 7f0e6be00640 1 -- 172.21.15.6:0/1422378765 shutdown_connections 2024-03-04T09:30:38.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.274+0000 7f0e6be00640 1 --2- 172.21.15.6:0/1422378765 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0e6c0d1cc0 0x7f0e6c0d24c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:38.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.274+0000 7f0e6be00640 1 --2- 172.21.15.6:0/1422378765 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0e6c0b5560 0x7f0e6c15fc30 unknown :-1 s=CLOSED pgs=1126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:38.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.274+0000 7f0e6be00640 1 -- 172.21.15.6:0/1422378765 >> 172.21.15.6:0/1422378765 conn(0x7f0e6c09d1b0 msgr2=0x7f0e6c0d3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:38.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.275+0000 7f0e6be00640 1 -- 172.21.15.6:0/1422378765 shutdown_connections 2024-03-04T09:30:38.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.275+0000 7f0e6be00640 1 -- 172.21.15.6:0/1422378765 wait complete. 2024-03-04T09:30:38.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.276+0000 7f0e6be00640 1 Processor -- start 2024-03-04T09:30:38.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.277+0000 7f0e6be00640 1 -- start start 2024-03-04T09:30:38.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.277+0000 7f0e6be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0e6c0b5560 0x7f0e6c14d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:38.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.277+0000 7f0e6be00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0e6c0d1cc0 0x7f0e6c0ca250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:38.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.278+0000 7f0e6be00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0e6c0d2a00 0x7f0e6c0b8b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:38.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.278+0000 7f0e6be00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e6c15f2f0 con 0x7f0e6c0b5560 2024-03-04T09:30:38.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.278+0000 7f0e6be00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e6c0d2e30 con 0x7f0e6c0d1cc0 2024-03-04T09:30:38.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.278+0000 7f0e6be00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e6c15f5d0 con 0x7f0e6c0d2a00 2024-03-04T09:30:38.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.278+0000 7f0e6aa00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0e6c0b5560 0x7f0e6c14d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:38.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.278+0000 7f0e6aa00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0e6c0b5560 0x7f0e6c14d410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:39984/0 (socket says 172.21.15.6:39984) 2024-03-04T09:30:38.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.278+0000 7f0e6aa00640 1 -- 172.21.15.6:0/3173049941 learned_addr learned my addr 172.21.15.6:0/3173049941 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:30:38.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.278+0000 7f0e6b400640 1 --2- 172.21.15.6:0/3173049941 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0e6c0d2a00 0x7f0e6c0b8b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:38.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.278+0000 7f0e63e00640 1 --2- 172.21.15.6:0/3173049941 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0e6c0d1cc0 0x7f0e6c0ca250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:38.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.278+0000 7f0e6aa00640 1 -- 172.21.15.6:0/3173049941 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0e6c0d2a00 msgr2=0x7f0e6c0b8b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:38.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.278+0000 7f0e6aa00640 1 --2- 172.21.15.6:0/3173049941 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0e6c0d2a00 0x7f0e6c0b8b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:38.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.278+0000 7f0e6aa00640 1 -- 172.21.15.6:0/3173049941 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0e6c0d1cc0 msgr2=0x7f0e6c0ca250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:38.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.278+0000 7f0e6aa00640 1 --2- 172.21.15.6:0/3173049941 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0e6c0d1cc0 0x7f0e6c0ca250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:38.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.278+0000 7f0e6aa00640 1 -- 172.21.15.6:0/3173049941 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e6c150980 con 0x7f0e6c0b5560 2024-03-04T09:30:38.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.279+0000 7f0e6aa00640 1 --2- 172.21.15.6:0/3173049941 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0e6c0b5560 0x7f0e6c14d410 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7f0e5c034040 tx=0x7f0e5c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:38.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.279+0000 7f0e62a00640 1 -- 172.21.15.6:0/3173049941 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e5c032610 con 0x7f0e6c0b5560 2024-03-04T09:30:38.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.279+0000 7f0e62a00640 1 -- 172.21.15.6:0/3173049941 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e5c0327b0 con 0x7f0e6c0b5560 2024-03-04T09:30:38.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.279+0000 7f0e6be00640 1 -- 172.21.15.6:0/3173049941 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e6c0c8dc0 con 0x7f0e6c0b5560 2024-03-04T09:30:38.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.279+0000 7f0e62a00640 1 -- 172.21.15.6:0/3173049941 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e5c032ba0 con 0x7f0e6c0b5560 2024-03-04T09:30:38.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.279+0000 7f0e6be00640 1 -- 172.21.15.6:0/3173049941 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e6c0c9380 con 0x7f0e6c0b5560 2024-03-04T09:30:38.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.283+0000 7f0e62a00640 1 -- 172.21.15.6:0/3173049941 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0e5c04e430 con 0x7f0e6c0b5560 2024-03-04T09:30:38.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.284+0000 7f0e7464c740 1 Processor -- start 2024-03-04T09:30:38.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.284+0000 7f0e7464c740 1 -- start start 2024-03-04T09:30:38.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.284+0000 7f0e62a00640 1 --2- 172.21.15.6:0/3173049941 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f0e4806b4d0 0x7f0e4806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:38.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.284+0000 7f0e62a00640 1 -- 172.21.15.6:0/3173049941 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f0e5c0ac200 con 0x7f0e6c0b5560 2024-03-04T09:30:38.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.285+0000 7f0e7464c740 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5565a4ce1990 0x5565a4cb4930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:38.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.285+0000 7f0e7464c740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5565a4ce1d70 0x5565a4ce4720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:38.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.285+0000 7f0e7464c740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5565a4cb4e70 0x5565a4ce4c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:38.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.285+0000 7f0e7464c740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x5565a4c1e600 con 0x5565a4ce1d70 2024-03-04T09:30:38.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.285+0000 7f0e7464c740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x5565a4bc7930 con 0x5565a4cb4e70 2024-03-04T09:30:38.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.285+0000 7f0e7464c740 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x5565a4c1a6a0 con 0x5565a4ce1990 2024-03-04T09:30:38.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.285+0000 7f0e63e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5565a4ce1d70 0x5565a4ce4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:38.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.285+0000 7f0e6b400640 1 --2- 172.21.15.6:0/3173049941 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f0e4806b4d0 0x7f0e4806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:38.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.285+0000 7f0e63e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5565a4ce1d70 0x5565a4ce4720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:39992/0 (socket says 172.21.15.6:39992) 2024-03-04T09:30:38.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.285+0000 7f0e63e00640 1 -- 172.21.15.6:0/1299697017 learned_addr learned my addr 172.21.15.6:0/1299697017 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:30:38.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.285+0000 7f0e63e00640 1 -- 172.21.15.6:0/1299697017 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5565a4ce1990 msgr2=0x5565a4cb4930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:38.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.285+0000 7f0e63e00640 1 --2- 172.21.15.6:0/1299697017 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5565a4ce1990 0x5565a4cb4930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:38.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.285+0000 7f0e63e00640 1 -- 172.21.15.6:0/1299697017 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5565a4cb4e70 msgr2=0x5565a4ce4c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:38.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.285+0000 7f0e63e00640 1 --2- 172.21.15.6:0/1299697017 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5565a4cb4e70 0x5565a4ce4c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:38.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.285+0000 7f0e63e00640 1 -- 172.21.15.6:0/1299697017 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5565a49332d0 con 0x5565a4ce1d70 2024-03-04T09:30:38.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.286+0000 7f0e63e00640 1 --2- 172.21.15.6:0/1299697017 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5565a4ce1d70 0x5565a4ce4720 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f0e54009840 tx=0x7f0e54009d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8806b164beb80b9f server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:38.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.286+0000 7f0e5b400640 1 -- 172.21.15.6:0/1299697017 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e54017030 con 0x5565a4ce1d70 2024-03-04T09:30:38.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.286+0000 7f0e5b400640 1 -- 172.21.15.6:0/1299697017 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e54004020 con 0x5565a4ce1d70 2024-03-04T09:30:38.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.286+0000 7f0e5b400640 1 -- 172.21.15.6:0/1299697017 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e5400f040 con 0x5565a4ce1d70 2024-03-04T09:30:38.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.287+0000 7f0e7464c740 1 -- 172.21.15.6:0/1299697017 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5565a4ce1d70 msgr2=0x5565a4ce4720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:38.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.287+0000 7f0e7464c740 1 --2- 172.21.15.6:0/1299697017 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5565a4ce1d70 0x5565a4ce4720 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f0e54009840 tx=0x7f0e54009d00 comp rx=0 tx=0).stop 2024-03-04T09:30:38.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.287+0000 7f0e6b400640 1 --2- 172.21.15.6:0/3173049941 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f0e4806b4d0 0x7f0e4806d990 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x5565a4cb73b0 tx=0x7f0e64001a40 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:38.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.287+0000 7f0e7464c740 1 -- 172.21.15.6:0/1299697017 shutdown_connections 2024-03-04T09:30:38.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.287+0000 7f0e7464c740 1 --2- 172.21.15.6:0/1299697017 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5565a4cb4e70 0x5565a4ce4c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:38.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.287+0000 7f0e7464c740 1 --2- 172.21.15.6:0/1299697017 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5565a4ce1d70 0x5565a4ce4720 unknown :-1 s=CLOSED pgs=1128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:38.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.287+0000 7f0e7464c740 1 --2- 172.21.15.6:0/1299697017 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5565a4ce1990 0x5565a4cb4930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:38.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.287+0000 7f0e7464c740 1 -- 172.21.15.6:0/1299697017 >> 172.21.15.6:0/1299697017 conn(0x5565a4cb6ba0 msgr2=0x5565a4cff720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:38.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.287+0000 7f0e7464c740 1 -- 172.21.15.6:0/1299697017 shutdown_connections 2024-03-04T09:30:38.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.287+0000 7f0e7464c740 1 -- 172.21.15.6:0/1299697017 wait complete. 2024-03-04T09:30:38.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.288+0000 7f0e7464c740 1 -- start start 2024-03-04T09:30:38.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.288+0000 7f0e7464c740 1 Processor -- start 2024-03-04T09:30:38.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.289+0000 7f0e7464c740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5565a4ce1990 0x5565a4cf9f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:38.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.289+0000 7f0e7464c740 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5565a4ce1d70 0x5565a4ce7a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:38.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.289+0000 7f0e7464c740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5565a4cd27b0 0x5565a4cd2c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:38.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.289+0000 7f0e7464c740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x5565a4cb9140 con 0x5565a4ce1990 2024-03-04T09:30:38.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.289+0000 7f0e7464c740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x5565a4bdad90 con 0x5565a4cd27b0 2024-03-04T09:30:38.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.289+0000 7f0e7464c740 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x5565a4cb8530 con 0x5565a4ce1d70 2024-03-04T09:30:38.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.289+0000 7f0e7464c740 10 client.0 _ll_register_callbacks cb 0x5565a4c16350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T09:30:38.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.289+0000 7f0e6aa00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5565a4ce1990 0x5565a4cf9f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:38.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.289+0000 7f0e6aa00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5565a4ce1990 0x5565a4cf9f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:40006/0 (socket says 172.21.15.6:40006) 2024-03-04T09:30:38.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.289+0000 7f0e6aa00640 1 -- 172.21.15.6:0/1091312472 learned_addr learned my addr 172.21.15.6:0/1091312472 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:30:38.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.289+0000 7f0e6aa00640 1 -- 172.21.15.6:0/1091312472 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5565a4ce1d70 msgr2=0x5565a4ce7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:38.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.290+0000 7f0e6aa00640 1 --2- 172.21.15.6:0/1091312472 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5565a4ce1d70 0x5565a4ce7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:38.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.290+0000 7f0e6aa00640 1 -- 172.21.15.6:0/1091312472 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5565a4cd27b0 msgr2=0x5565a4cd2c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:38.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.290+0000 7f0e6aa00640 1 --2- 172.21.15.6:0/1091312472 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5565a4cd27b0 0x5565a4cd2c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:38.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.290+0000 7f0e6aa00640 1 -- 172.21.15.6:0/1091312472 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5565a49fd6b0 con 0x5565a4ce1990 2024-03-04T09:30:38.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.290+0000 7f0e6aa00640 1 --2- 172.21.15.6:0/1091312472 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5565a4ce1990 0x5565a4cf9f70 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7f0e5c07a850 tx=0x7f0e5c07afc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=659e6c92f15c4bf9 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:38.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.290+0000 7f0e4fe00640 10 client.0 ms_handle_connect on v2:172.21.15.6:3300/0 2024-03-04T09:30:38.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.290+0000 7f0e4fe00640 1 -- 172.21.15.6:0/1091312472 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e5c082d40 con 0x5565a4ce1990 2024-03-04T09:30:38.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.290+0000 7f0e4fe00640 1 -- 172.21.15.6:0/1091312472 <== mon.0 v2:172.21.15.6:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f0e5c092170 con 0x5565a4ce1990 2024-03-04T09:30:38.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.290+0000 7f0e7464c740 1 -- 172.21.15.6:0/1091312472 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0e54004020 con 0x5565a4ce1990 2024-03-04T09:30:38.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.290+0000 7f0e4fe00640 1 -- 172.21.15.6:0/1091312472 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_get_version_reply(handle=1 version=68) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0e5c07ea10 con 0x5565a4ce1990 2024-03-04T09:30:38.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.290+0000 7f0e7464c740 10 client.7332 fetch_fsmap learned FSMap version 68 2024-03-04T09:30:38.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.290+0000 7f0e7464c740 1 -- 172.21.15.6:0/1091312472 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,fsmap=68,monmap=2+}) v3 -- 0x5565a4cfa4b0 con 0x5565a4ce1990 2024-03-04T09:30:38.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.291+0000 7f0e4fe00640 1 -- 172.21.15.6:0/1091312472 <== mon.0 v2:172.21.15.6:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e54004020 con 0x5565a4ce1990 2024-03-04T09:30:38.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.291+0000 7f0e4fe00640 1 -- 172.21.15.6:0/1091312472 <== mon.0 v2:172.21.15.6:3300/0 5 ==== fsmap(e 68) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f0e5c079d10 con 0x5565a4ce1990 2024-03-04T09:30:38.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.291+0000 7f0e7464c740 10 client.7332 fetch_fsmap finished waiting for FSMap version 68 2024-03-04T09:30:38.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.291+0000 7f0e7464c740 10 client.7332 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-04T09:30:38.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.291+0000 7f0e7464c740 20 client.7332 populate_metadata read hostname 'smithi006' 2024-03-04T09:30:38.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.291+0000 7f0e7464c740 1 --2- 172.21.15.6:0/1091312472 >> [v2:172.21.15.6:6834/1732383836,v1:172.21.15.6:6835/1732383836] conn(0x5565a4cf8710 0x5565a4cf8af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:38.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.291+0000 7f0e7464c740 4 client.7332 mds_command: new command op to 7296 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T09:30:38.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.291+0000 7f0e7464c740 1 -- 172.21.15.6:0/1091312472 --> [v2:172.21.15.6:6834/1732383836,v1:172.21.15.6:6835/1732383836] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5565a4c2bb40 con 0x5565a4cf8710 2024-03-04T09:30:38.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.291+0000 7f0e63e00640 1 --2- 172.21.15.6:0/1091312472 >> [v2:172.21.15.6:6834/1732383836,v1:172.21.15.6:6835/1732383836] conn(0x5565a4cf8710 0x5565a4cf8af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:38.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.292+0000 7f0e63e00640 1 --2- 172.21.15.6:0/1091312472 >> [v2:172.21.15.6:6834/1732383836,v1:172.21.15.6:6835/1732383836] conn(0x5565a4cf8710 0x5565a4cf8af0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c4fe7771a8b0f5dd server_cookie=e7edecb4caea89b6 in_seq=0 out_seq=0 2024-03-04T09:30:38.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.292+0000 7f0e4fe00640 10 client.7332 ms_handle_connect on v2:172.21.15.6:6834/1732383836 2024-03-04T09:30:38.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.293+0000 7f0e4fe00640 1 -- 172.21.15.6:0/1091312472 <== mds.0 v2:172.21.15.6:6834/1732383836 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5565a4c2bb40 con 0x5565a4cf8710 2024-03-04T09:30:38.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.293+0000 7f0e4fe00640 10 client.7332 handle_command_reply: tid=0 2024-03-04T09:30:38.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.293+0000 7f0e7464c740 1 client.7332 shutdown 2024-03-04T09:30:38.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.293+0000 7f0e7464c740 1 -- 172.21.15.6:0/1091312472 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5565a4ce1990 msgr2=0x5565a4cf9f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:38.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.293+0000 7f0e7464c740 1 --2- 172.21.15.6:0/1091312472 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5565a4ce1990 0x5565a4cf9f70 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7f0e5c07a850 tx=0x7f0e5c07afc0 comp rx=0 tx=0).stop 2024-03-04T09:30:38.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.293+0000 7f0e7464c740 1 -- 172.21.15.6:0/1091312472 shutdown_connections 2024-03-04T09:30:38.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.293+0000 7f0e7464c740 1 --2- 172.21.15.6:0/1091312472 >> [v2:172.21.15.6:6834/1732383836,v1:172.21.15.6:6835/1732383836] conn(0x5565a4cf8710 0x5565a4cf8af0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:38.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.293+0000 7f0e7464c740 1 --2- 172.21.15.6:0/1091312472 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5565a4cd27b0 0x5565a4cd2c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:38.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.293+0000 7f0e7464c740 1 --2- 172.21.15.6:0/1091312472 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5565a4ce1d70 0x5565a4ce7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:38.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.294+0000 7f0e4fe00640 0 client.7332 ms_handle_reset on v2:172.21.15.6:6834/1732383836 2024-03-04T09:30:38.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.293+0000 7f0e7464c740 1 --2- 172.21.15.6:0/1091312472 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5565a4ce1990 0x5565a4cf9f70 unknown :-1 s=CLOSED pgs=1129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:38.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.294+0000 7f0e7464c740 1 -- 172.21.15.6:0/1091312472 >> 172.21.15.6:0/1091312472 conn(0x5565a4cb4e70 msgr2=0x5565a4cb4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:38.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.294+0000 7f0e7464c740 1 -- 172.21.15.6:0/1091312472 shutdown_connections 2024-03-04T09:30:38.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.294+0000 7f0e7464c740 1 -- 172.21.15.6:0/1091312472 wait complete. 2024-03-04T09:30:38.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.294+0000 7f0e7464c740 20 client.7332 trim_cache size 0 max 16384 2024-03-04T09:30:38.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.312+0000 7f0e7464c740 1 Processor -- start 2024-03-04T09:30:38.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.312+0000 7f0e7464c740 1 -- start start 2024-03-04T09:30:38.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.313+0000 7f0e7464c740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5565a4cd27b0 0x5565a4cb8530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:38.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.313+0000 7f0e7464c740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5565a4ce1990 0x5565a4cb8a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:38.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.313+0000 7f0e7464c740 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5565a4ce1d70 0x5565a4ce4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:38.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.313+0000 7f0e7464c740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e6c0cc4a0 con 0x5565a4ce1990 2024-03-04T09:30:38.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.313+0000 7f0e7464c740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e6c1f8410 con 0x5565a4cd27b0 2024-03-04T09:30:38.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.313+0000 7f0e7464c740 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e6c0cbea0 con 0x5565a4ce1d70 2024-03-04T09:30:38.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.314+0000 7f0e63e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5565a4ce1990 0x5565a4cb8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:38.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.314+0000 7f0e63e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5565a4ce1990 0x5565a4cb8a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:40012/0 (socket says 172.21.15.6:40012) 2024-03-04T09:30:38.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.314+0000 7f0e63e00640 1 -- 172.21.15.6:0/2289174925 learned_addr learned my addr 172.21.15.6:0/2289174925 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:30:38.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.314+0000 7f0e6aa00640 1 --2- 172.21.15.6:0/2289174925 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5565a4cd27b0 0x5565a4cb8530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:38.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.314+0000 7f0e6b400640 1 --2- 172.21.15.6:0/2289174925 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5565a4ce1d70 0x5565a4ce4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:38.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.314+0000 7f0e63e00640 1 -- 172.21.15.6:0/2289174925 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5565a4ce1d70 msgr2=0x5565a4ce4900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:38.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.314+0000 7f0e63e00640 1 --2- 172.21.15.6:0/2289174925 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5565a4ce1d70 0x5565a4ce4900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:38.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.314+0000 7f0e63e00640 1 -- 172.21.15.6:0/2289174925 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5565a4cd27b0 msgr2=0x5565a4cb8530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:38.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.314+0000 7f0e63e00640 1 --2- 172.21.15.6:0/2289174925 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5565a4cd27b0 0x5565a4cb8530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:38.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.314+0000 7f0e63e00640 1 -- 172.21.15.6:0/2289174925 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5565a4cb01b0 con 0x5565a4ce1990 2024-03-04T09:30:38.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.314+0000 7f0e6b400640 1 --2- 172.21.15.6:0/2289174925 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5565a4ce1d70 0x5565a4ce4900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T09:30:38.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.315+0000 7f0e63e00640 1 --2- 172.21.15.6:0/2289174925 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5565a4ce1990 0x5565a4cb8a70 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7f0e54016040 tx=0x7f0e54013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1869a8f92002df51 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:38.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.315+0000 7f0e5aa00640 1 -- 172.21.15.6:0/2289174925 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e54004240 con 0x5565a4ce1990 2024-03-04T09:30:38.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.315+0000 7f0e5aa00640 1 -- 172.21.15.6:0/2289174925 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e54004400 con 0x5565a4ce1990 2024-03-04T09:30:38.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.315+0000 7f0e5aa00640 1 -- 172.21.15.6:0/2289174925 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e5400e990 con 0x5565a4ce1990 2024-03-04T09:30:38.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.315+0000 7f0e7464c740 1 -- 172.21.15.6:0/2289174925 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5565a4ce1990 msgr2=0x5565a4cb8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:38.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.315+0000 7f0e7464c740 1 --2- 172.21.15.6:0/2289174925 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5565a4ce1990 0x5565a4cb8a70 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7f0e54016040 tx=0x7f0e54013000 comp rx=0 tx=0).stop 2024-03-04T09:30:38.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.315+0000 7f0e7464c740 1 -- 172.21.15.6:0/2289174925 shutdown_connections 2024-03-04T09:30:38.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.315+0000 7f0e7464c740 1 --2- 172.21.15.6:0/2289174925 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5565a4ce1d70 0x5565a4ce4900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:38.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.315+0000 7f0e7464c740 1 --2- 172.21.15.6:0/2289174925 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5565a4ce1990 0x5565a4cb8a70 unknown :-1 s=CLOSED pgs=1130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:38.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.315+0000 7f0e7464c740 1 --2- 172.21.15.6:0/2289174925 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5565a4cd27b0 0x5565a4cb8530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:38.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.315+0000 7f0e7464c740 1 -- 172.21.15.6:0/2289174925 >> 172.21.15.6:0/2289174925 conn(0x5565a4cb4e70 msgr2=0x5565a4cf9d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:38.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.316+0000 7f0e7464c740 1 -- 172.21.15.6:0/2289174925 shutdown_connections 2024-03-04T09:30:38.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.316+0000 7f0e7464c740 1 -- 172.21.15.6:0/2289174925 wait complete. 2024-03-04T09:30:38.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.316+0000 7f0e7464c740 1 -- start start 2024-03-04T09:30:38.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.316+0000 7f0e7464c740 1 Processor -- start 2024-03-04T09:30:38.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.317+0000 7f0e7464c740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5565a4cd27b0 0x5565a4d71940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:38.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.317+0000 7f0e7464c740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5565a4ce1990 0x5565a4ce08e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:38.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.317+0000 7f0e7464c740 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5565a4ce1d70 0x5565a4cb8530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:38.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.318+0000 7f0e7464c740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x5565a4cb9140 con 0x5565a4ce1990 2024-03-04T09:30:38.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.318+0000 7f0e7464c740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x5565a4bdad90 con 0x5565a4cd27b0 2024-03-04T09:30:38.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.318+0000 7f0e7464c740 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x5565a4cf7a40 con 0x5565a4ce1d70 2024-03-04T09:30:38.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.318+0000 7f0e7464c740 10 client.0 _ll_register_callbacks cb 0x7f0e5c009400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T09:30:38.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.318+0000 7f0e63e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5565a4ce1990 0x5565a4ce08e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:38.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.318+0000 7f0e63e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5565a4ce1990 0x5565a4ce08e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:40026/0 (socket says 172.21.15.6:40026) 2024-03-04T09:30:38.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.318+0000 7f0e63e00640 1 -- 172.21.15.6:0/449603198 learned_addr learned my addr 172.21.15.6:0/449603198 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:30:38.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.318+0000 7f0e6b400640 1 --2- 172.21.15.6:0/449603198 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5565a4ce1d70 0x5565a4cb8530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:38.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.318+0000 7f0e63e00640 1 -- 172.21.15.6:0/449603198 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5565a4ce1d70 msgr2=0x5565a4cb8530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:38.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.318+0000 7f0e63e00640 1 --2- 172.21.15.6:0/449603198 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5565a4ce1d70 0x5565a4cb8530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:38.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.318+0000 7f0e63e00640 1 -- 172.21.15.6:0/449603198 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5565a4cd27b0 msgr2=0x5565a4d71940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:38.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.318+0000 7f0e63e00640 1 --2- 172.21.15.6:0/449603198 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5565a4cd27b0 0x5565a4d71940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:38.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.318+0000 7f0e63e00640 1 -- 172.21.15.6:0/449603198 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5565a4d22aa0 con 0x5565a4ce1990 2024-03-04T09:30:38.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.319+0000 7f0e63e00640 1 --2- 172.21.15.6:0/449603198 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5565a4ce1990 0x5565a4ce08e0 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7f0e540095d0 tx=0x7f0e54004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3fb1e7f6734831a server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:38.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.319+0000 7f0e4fe00640 10 client.0 ms_handle_connect on v2:172.21.15.6:3300/0 2024-03-04T09:30:38.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.321+0000 7f0e4fe00640 1 -- 172.21.15.6:0/449603198 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e54017050 con 0x5565a4ce1990 2024-03-04T09:30:38.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.321+0000 7f0e4fe00640 1 -- 172.21.15.6:0/449603198 <== mon.0 v2:172.21.15.6:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f0e540092f0 con 0x5565a4ce1990 2024-03-04T09:30:38.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.321+0000 7f0e7464c740 1 -- 172.21.15.6:0/449603198 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0e54004400 con 0x5565a4ce1990 2024-03-04T09:30:38.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.322+0000 7f0e4fe00640 1 -- 172.21.15.6:0/449603198 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_get_version_reply(handle=1 version=68) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0e54034050 con 0x5565a4ce1990 2024-03-04T09:30:38.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.322+0000 7f0e7464c740 10 client.7338 fetch_fsmap learned FSMap version 68 2024-03-04T09:30:38.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.322+0000 7f0e7464c740 1 -- 172.21.15.6:0/449603198 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,fsmap=68,monmap=2+}) v3 -- 0x5565a4d72170 con 0x5565a4ce1990 2024-03-04T09:30:38.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.322+0000 7f0e4fe00640 1 -- 172.21.15.6:0/449603198 <== mon.0 v2:172.21.15.6:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e54004400 con 0x5565a4ce1990 2024-03-04T09:30:38.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.322+0000 7f0e4fe00640 1 -- 172.21.15.6:0/449603198 <== mon.0 v2:172.21.15.6:3300/0 5 ==== fsmap(e 68) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f0e54037020 con 0x5565a4ce1990 2024-03-04T09:30:38.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.322+0000 7f0e7464c740 10 client.7338 fetch_fsmap finished waiting for FSMap version 68 2024-03-04T09:30:38.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.322+0000 7f0e7464c740 10 client.7338 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-04T09:30:38.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.322+0000 7f0e7464c740 20 client.7338 populate_metadata read hostname 'smithi006' 2024-03-04T09:30:38.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.322+0000 7f0e7464c740 1 --2- 172.21.15.6:0/449603198 >> [v2:172.21.15.6:6834/1732383836,v1:172.21.15.6:6835/1732383836] conn(0x5565a4d713b0 0x5565a4cb8a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:38.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.322+0000 7f0e7464c740 4 client.7338 mds_command: new command op to 7296 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-04T09:30:38.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.322+0000 7f0e7464c740 1 -- 172.21.15.6:0/449603198 --> [v2:172.21.15.6:6834/1732383836,v1:172.21.15.6:6835/1732383836] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5565a4c25e90 con 0x5565a4d713b0 2024-03-04T09:30:38.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.322+0000 7f0e6aa00640 1 --2- 172.21.15.6:0/449603198 >> [v2:172.21.15.6:6834/1732383836,v1:172.21.15.6:6835/1732383836] conn(0x5565a4d713b0 0x5565a4cb8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:38.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.323+0000 7f0e6aa00640 1 --2- 172.21.15.6:0/449603198 >> [v2:172.21.15.6:6834/1732383836,v1:172.21.15.6:6835/1732383836] conn(0x5565a4d713b0 0x5565a4cb8a70 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4b8c426e9c8ad1f server_cookie=9befe28b54474a52 in_seq=0 out_seq=0 2024-03-04T09:30:38.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.323+0000 7f0e4fe00640 10 client.7338 ms_handle_connect on v2:172.21.15.6:6834/1732383836 2024-03-04T09:30:38.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.323+0000 7f0e4fe00640 1 -- 172.21.15.6:0/449603198 <== mds.0 v2:172.21.15.6:6834/1732383836 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+324 (crc 0 0 0) 0x5565a4c25e90 con 0x5565a4d713b0 2024-03-04T09:30:38.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.323+0000 7f0e4fe00640 10 client.7338 handle_command_reply: tid=0 2024-03-04T09:30:38.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.323+0000 7f0e7464c740 1 client.7338 shutdown 2024-03-04T09:30:38.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.324+0000 7f0e7464c740 1 -- 172.21.15.6:0/449603198 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5565a4ce1990 msgr2=0x5565a4ce08e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:38.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.324+0000 7f0e7464c740 1 --2- 172.21.15.6:0/449603198 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5565a4ce1990 0x5565a4ce08e0 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7f0e540095d0 tx=0x7f0e54004730 comp rx=0 tx=0).stop 2024-03-04T09:30:38.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.324+0000 7f0e7464c740 1 -- 172.21.15.6:0/449603198 shutdown_connections 2024-03-04T09:30:38.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.324+0000 7f0e7464c740 1 --2- 172.21.15.6:0/449603198 >> [v2:172.21.15.6:6834/1732383836,v1:172.21.15.6:6835/1732383836] conn(0x5565a4d713b0 0x5565a4cb8a70 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:38.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.324+0000 7f0e7464c740 1 --2- 172.21.15.6:0/449603198 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5565a4ce1d70 0x5565a4cb8530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:38.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.324+0000 7f0e7464c740 1 --2- 172.21.15.6:0/449603198 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5565a4ce1990 0x5565a4ce08e0 unknown :-1 s=CLOSED pgs=1131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:38.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.324+0000 7f0e7464c740 1 --2- 172.21.15.6:0/449603198 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5565a4cd27b0 0x5565a4d71940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:38.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.324+0000 7f0e7464c740 1 -- 172.21.15.6:0/449603198 >> 172.21.15.6:0/449603198 conn(0x5565a4cb4e70 msgr2=0x5565a4cf6710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:38.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.324+0000 7f0e4fe00640 0 client.7338 ms_handle_reset on v2:172.21.15.6:6834/1732383836 2024-03-04T09:30:38.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.324+0000 7f0e7464c740 1 -- 172.21.15.6:0/449603198 shutdown_connections 2024-03-04T09:30:38.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.324+0000 7f0e7464c740 1 -- 172.21.15.6:0/449603198 wait complete. 2024-03-04T09:30:38.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.324+0000 7f0e7464c740 20 client.7338 trim_cache size 0 max 16384 2024-03-04T09:30:38.323 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-04T09:30:38.323 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_fsid": "edce0578-5ae4-4402-b45a-552bdef84d7c", 2024-03-04T09:30:38.323 INFO:teuthology.orchestra.run.smithi006.stdout: "whoami": 0, 2024-03-04T09:30:38.324 INFO:teuthology.orchestra.run.smithi006.stdout: "id": 7296, 2024-03-04T09:30:38.324 INFO:teuthology.orchestra.run.smithi006.stdout: "want_state": "up:active", 2024-03-04T09:30:38.324 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "up:active", 2024-03-04T09:30:38.324 INFO:teuthology.orchestra.run.smithi006.stdout: "fs_name": "cephfs", 2024-03-04T09:30:38.324 INFO:teuthology.orchestra.run.smithi006.stdout: "rank_uptime": 5.044987055, 2024-03-04T09:30:38.324 INFO:teuthology.orchestra.run.smithi006.stdout: "mdsmap_epoch": 68, 2024-03-04T09:30:38.324 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap_epoch": 98, 2024-03-04T09:30:38.324 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap_epoch_barrier": 98, 2024-03-04T09:30:38.324 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 5.7989493699999999 2024-03-04T09:30:38.324 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-04T09:30:38.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.325+0000 7f0e6be00640 1 -- 172.21.15.6:0/3173049941 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f0e4806b4d0 msgr2=0x7f0e4806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:38.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.325+0000 7f0e6be00640 1 --2- 172.21.15.6:0/3173049941 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f0e4806b4d0 0x7f0e4806d990 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x5565a4cb73b0 tx=0x7f0e64001a40 comp rx=0 tx=0).stop 2024-03-04T09:30:38.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.325+0000 7f0e6be00640 1 -- 172.21.15.6:0/3173049941 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0e6c0b5560 msgr2=0x7f0e6c14d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:38.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.325+0000 7f0e6be00640 1 --2- 172.21.15.6:0/3173049941 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0e6c0b5560 0x7f0e6c14d410 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7f0e5c034040 tx=0x7f0e5c0046c0 comp rx=0 tx=0).stop 2024-03-04T09:30:38.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.325+0000 7f0e6be00640 1 -- 172.21.15.6:0/3173049941 shutdown_connections 2024-03-04T09:30:38.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.325+0000 7f0e6be00640 1 --2- 172.21.15.6:0/3173049941 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f0e4806b4d0 0x7f0e4806d990 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:38.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.325+0000 7f0e6be00640 1 --2- 172.21.15.6:0/3173049941 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0e6c0d2a00 0x7f0e6c0b8b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:38.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.325+0000 7f0e6be00640 1 --2- 172.21.15.6:0/3173049941 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0e6c0d1cc0 0x7f0e6c0ca250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:38.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.325+0000 7f0e6be00640 1 --2- 172.21.15.6:0/3173049941 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0e6c0b5560 0x7f0e6c14d410 unknown :-1 s=CLOSED pgs=1127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:38.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.325+0000 7f0e6be00640 1 -- 172.21.15.6:0/3173049941 >> 172.21.15.6:0/3173049941 conn(0x7f0e6c09d1b0 msgr2=0x7f0e6c160c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:38.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.325+0000 7f0e6be00640 1 -- 172.21.15.6:0/3173049941 shutdown_connections 2024-03-04T09:30:38.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:38.325+0000 7f0e6be00640 1 -- 172.21.15.6:0/3173049941 wait complete. 2024-03-04T09:30:38.340 DEBUG:teuthology.orchestra.run.smithi006:> stdin-killer -- python3 -c ' 2024-03-04T09:30:38.340 DEBUG:teuthology.orchestra.run.smithi006:> import os 2024-03-04T09:30:38.341 DEBUG:teuthology.orchestra.run.smithi006:> 2024-03-04T09:30:38.341 DEBUG:teuthology.orchestra.run.smithi006:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-04T09:30:38.341 DEBUG:teuthology.orchestra.run.smithi006:> subdir = "delete_me" 2024-03-04T09:30:38.341 DEBUG:teuthology.orchestra.run.smithi006:> size_unit = 1024 2024-03-04T09:30:38.341 DEBUG:teuthology.orchestra.run.smithi006:> file_multiplier = 200 2024-03-04T09:30:38.341 DEBUG:teuthology.orchestra.run.smithi006:> os.mkdir(os.path.join(mountpoint, subdir)) 2024-03-04T09:30:38.341 DEBUG:teuthology.orchestra.run.smithi006:> for i in range(0, file_multiplier): 2024-03-04T09:30:38.341 DEBUG:teuthology.orchestra.run.smithi006:> for size in range(0, 16*size_unit, size_unit): 2024-03-04T09:30:38.341 DEBUG:teuthology.orchestra.run.smithi006:> filename = "{0}_{1}.bin".format(i, size // size_unit) 2024-03-04T09:30:38.341 DEBUG:teuthology.orchestra.run.smithi006:> with open(os.path.join(mountpoint, subdir, filename), '"'"'w'"'"') as f: 2024-03-04T09:30:38.341 DEBUG:teuthology.orchestra.run.smithi006:> f.write(size * '"'"'x'"'"') 2024-03-04T09:30:38.341 DEBUG:teuthology.orchestra.run.smithi006:> ' 2024-03-04T09:30:39.280 INFO:tasks.ceph.mds.a.smithi006.stderr:2024-03-04T09:30:39.283+0000 7f42d2c00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-04T09:30:40.019 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:30:40.028 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:30:42.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:42 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-04T09:30:42.653 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json config get mds_max_purge_ops 2024-03-04T09:30:42.654 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:30:42.654 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest/mnt.0 && exec rm -rf delete_me) 2024-03-04T09:30:42.918 INFO:teuthology.orchestra.run.smithi006.stdout:{"mds_max_purge_ops":"100000000"} 2024-03-04T09:30:42.918 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_max_purge_ops": "100000000" } 2024-03-04T09:30:42.919 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json config get mds_max_purge_files 2024-03-04T09:30:43.141 INFO:teuthology.orchestra.run.smithi006.stdout:{"mds_max_purge_files":"6"} 2024-03-04T09:30:43.141 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_max_purge_files": "6" } 2024-03-04T09:30:43.141 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:30:43.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:43.393+0000 7f7052a00640 1 Processor -- start 2024-03-04T09:30:43.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:43.393+0000 7f7052a00640 1 -- start start 2024-03-04T09:30:43.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:43.395+0000 7f7052a00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f704c06fc80 0x7f704c0b3f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:43.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:43.395+0000 7f7052a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f704c0ae4e0 con 0x7f704c06afc0 2024-03-04T09:30:43.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:43.395+0000 7f7052a00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f704c15ec30 con 0x7f704c15e140 2024-03-04T09:30:43.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:43.395+0000 7f7052a00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f704c15eef0 con 0x7f704c06fc80 2024-03-04T09:30:43.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:43.396+0000 7f7050a00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f704c06afc0 0x7f704c0aeca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:45268/0 (socket says 172.21.15.6:45268) 2024-03-04T09:30:43.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:43.396+0000 7f7050a00640 1 -- 172.21.15.6:0/2825790590 learned_addr learned my addr 172.21.15.6:0/2825790590 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:30:43.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:43.396+0000 7f704b400640 1 --2- 172.21.15.6:0/2825790590 >> v2:172.21.15.146:3300/0 conn(0x7f704c06fc80 0x7f704c0b3f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:43.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:43.396+0000 7f704aa00640 1 -- 172.21.15.6:0/2825790590 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1795715014 0 0) 0x7f704c0ae4e0 con 0x7f704c06afc0 2024-03-04T09:30:43.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:43.396+0000 7f704aa00640 1 -- 172.21.15.6:0/2825790590 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7030003520 con 0x7f704c06afc0 2024-03-04T09:30:43.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:43.397+0000 7f704b400640 1 -- 172.21.15.6:0/2825790590 >> v1:172.21.15.146:6789/0 conn(0x7f704c15e140 legacy=0x7f704c15e520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:43.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:43.397+0000 7f704b400640 1 -- 172.21.15.6:0/2825790590 >> v1:172.21.15.6:6789/0 conn(0x7f704c06afc0 legacy=0x7f704c0aeca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:43.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:43.397+0000 7f704b400640 1 -- 172.21.15.6:0/2825790590 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f704c15f1d0 con 0x7f704c06fc80 2024-03-04T09:30:43.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:43.398+0000 7f704b400640 1 --2- 172.21.15.6:0/2825790590 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f704c06fc80 0x7f704c0b3f40 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f70380097a0 tx=0x7f703802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d9adc642d8b30824 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:43.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:43.399+0000 7f704aa00640 1 -- 172.21.15.6:0/2825790590 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7038035020 con 0x7f704c06fc80 2024-03-04T09:30:43.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:43.399+0000 7f704aa00640 1 -- 172.21.15.6:0/2825790590 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7038031830 con 0x7f704c06fc80 2024-03-04T09:30:43.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:43.399+0000 7f704aa00640 1 -- 172.21.15.6:0/2825790590 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7038031c00 con 0x7f704c06fc80 2024-03-04T09:30:43.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:43.399+0000 7f7052a00640 1 -- 172.21.15.6:0/2825790590 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f704c06fc80 msgr2=0x7f704c0b3f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:43.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:43.399+0000 7f7052a00640 1 --2- 172.21.15.6:0/2825790590 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f704c06fc80 0x7f704c0b3f40 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f70380097a0 tx=0x7f703802b8d0 comp rx=0 tx=0).stop 2024-03-04T09:30:43.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:43.400+0000 7f7052a00640 1 -- 172.21.15.6:0/2825790590 shutdown_connections 2024-03-04T09:30:43.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:43.400+0000 7f7052a00640 1 --2- 172.21.15.6:0/2825790590 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f704c06fc80 0x7f704c0b3f40 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:43.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:43.400+0000 7f7052a00640 1 -- 172.21.15.6:0/2825790590 >> 172.21.15.6:0/2825790590 conn(0x7f704c0b2c40 msgr2=0x7f704c0b3020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:43.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:43.400+0000 7f7052a00640 1 -- 172.21.15.6:0/2825790590 shutdown_connections 2024-03-04T09:30:43.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:43.400+0000 7f7052a00640 1 -- 172.21.15.6:0/2825790590 wait complete. 2024-03-04T09:30:43.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:43.402+0000 7f7052a00640 1 Processor -- start 2024-03-04T09:30:43.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:43.402+0000 7f7052a00640 1 -- start start 2024-03-04T09:30:43.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:43.404+0000 7f7052a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f704c06afc0 0x7f704c151200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:43.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:43.404+0000 7f7052a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f704c06fc80 0x7f704c153900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:43.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:43.404+0000 7f7052a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f704c15e140 0x7f704c153e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:43.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:43.404+0000 7f7052a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f704c06b430 con 0x7f704c06fc80 2024-03-04T09:30:43.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:43.404+0000 7f7052a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f704c0ae780 con 0x7f704c15e140 2024-03-04T09:30:43.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:43.404+0000 7f7052a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f704c06b5b0 con 0x7f704c06afc0 2024-03-04T09:30:43.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:43.404+0000 7f704b400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f704c06fc80 0x7f704c153900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:43.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:43.404+0000 7f704b400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f704c06fc80 0x7f704c153900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37198/0 (socket says 172.21.15.6:37198) 2024-03-04T09:30:43.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:43.404+0000 7f704b400640 1 -- 172.21.15.6:0/3653798254 learned_addr learned my addr 172.21.15.6:0/3653798254 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:30:43.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:43.405+0000 7f7050a00640 1 --2- 172.21.15.6:0/3653798254 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f704c15e140 0x7f704c153e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:43.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:43.405+0000 7f704be00640 1 --2- 172.21.15.6:0/3653798254 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f704c06afc0 0x7f704c151200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:43.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:43.405+0000 7f704b400640 1 -- 172.21.15.6:0/3653798254 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f704c06afc0 msgr2=0x7f704c151200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:43.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:43.405+0000 7f704b400640 1 --2- 172.21.15.6:0/3653798254 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f704c06afc0 0x7f704c151200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:43.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:43.405+0000 7f704b400640 1 -- 172.21.15.6:0/3653798254 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f704c15e140 msgr2=0x7f704c153e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:43.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:43.405+0000 7f704b400640 1 --2- 172.21.15.6:0/3653798254 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f704c15e140 0x7f704c153e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:43.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:43.405+0000 7f704b400640 1 -- 172.21.15.6:0/3653798254 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f704c1f7060 con 0x7f704c06fc80 2024-03-04T09:30:43.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:43.405+0000 7f704b400640 1 --2- 172.21.15.6:0/3653798254 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f704c06fc80 0x7f704c153900 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7f703802bee0 tx=0x7f70380046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:43.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:43.406+0000 7f7043e00640 1 -- 172.21.15.6:0/3653798254 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7038035040 con 0x7f704c06fc80 2024-03-04T09:30:43.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:43.406+0000 7f7043e00640 1 -- 172.21.15.6:0/3653798254 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f703803b920 con 0x7f704c06fc80 2024-03-04T09:30:43.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:43.406+0000 7f7052a00640 1 -- 172.21.15.6:0/3653798254 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f704c1f7340 con 0x7f704c06fc80 2024-03-04T09:30:43.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:43.406+0000 7f7043e00640 1 -- 172.21.15.6:0/3653798254 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f703803bd10 con 0x7f704c06fc80 2024-03-04T09:30:43.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:43.406+0000 7f7052a00640 1 -- 172.21.15.6:0/3653798254 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f704c1f7900 con 0x7f704c06fc80 2024-03-04T09:30:43.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:43.408+0000 7f7052a00640 1 -- 172.21.15.6:0/3653798254 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f704c0692a0 con 0x7f704c06fc80 2024-03-04T09:30:43.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:43.409+0000 7f7043e00640 1 -- 172.21.15.6:0/3653798254 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7038038020 con 0x7f704c06fc80 2024-03-04T09:30:43.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:43.410+0000 7f7043e00640 1 --2- 172.21.15.6:0/3653798254 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f702c06b4d0 0x7f702c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:43.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:43.410+0000 7f7043e00640 1 -- 172.21.15.6:0/3653798254 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f70380ac410 con 0x7f704c06fc80 2024-03-04T09:30:43.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:43.411+0000 7f704be00640 1 --2- 172.21.15.6:0/3653798254 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f702c06b4d0 0x7f702c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:43.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:43.412+0000 7f704be00640 1 --2- 172.21.15.6:0/3653798254 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f702c06b4d0 0x7f702c06d990 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f704c15b350 tx=0x7f703c009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:43.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:43.415+0000 7f7043e00640 1 -- 172.21.15.6:0/3653798254 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f703807b150 con 0x7f704c06fc80 2024-03-04T09:30:43.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:43.730+0000 7f7052a00640 1 -- 172.21.15.6:0/3653798254 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f704c0af970 con 0x7f704c06fc80 2024-03-04T09:30:43.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:43.731+0000 7f7043e00640 1 -- 172.21.15.6:0/3653798254 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) v1 ==== 94+0+4977 (secure 0 0 0) 0x7f7038031800 con 0x7f704c06fc80 2024-03-04T09:30:43.728 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:30:43.729 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":68,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5654,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/1370302030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":1370302030},{"type":"v1","addr":"172.21.15.146:6835","nonce":1370302030}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":6163,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/809822004","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":809822004},{"type":"v1","addr":"172.21.15.146:6837","nonce":809822004}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":66},{"gid":7284,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/1583413840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1583413840},{"type":"v1","addr":"172.21.15.6:6833","nonce":1583413840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64}],"filesystems":[{"mdsmap":{"epoch":68,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:30:20.660103+0000","modified":"2024-03-04T09:30:36.289706+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":98,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7296},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7296":{"gid":7296,"name":"a","rank":0,"incarnation":65,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1732383836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1732383836},{"type":"v1","addr":"172.21.15.6:6835","nonce":1732383836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7296,"qdb_cluster":[7296]},"id":7}]} 2024-03-04T09:30:43.729 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 68 2024-03-04T09:30:43.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:43.734+0000 7f7052a00640 1 -- 172.21.15.6:0/3653798254 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f702c06b4d0 msgr2=0x7f702c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:43.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:43.734+0000 7f7052a00640 1 --2- 172.21.15.6:0/3653798254 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f702c06b4d0 0x7f702c06d990 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f704c15b350 tx=0x7f703c009040 comp rx=0 tx=0).stop 2024-03-04T09:30:43.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:43.734+0000 7f7052a00640 1 -- 172.21.15.6:0/3653798254 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f704c06fc80 msgr2=0x7f704c153900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:43.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:43.734+0000 7f7052a00640 1 --2- 172.21.15.6:0/3653798254 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f704c06fc80 0x7f704c153900 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7f703802bee0 tx=0x7f70380046c0 comp rx=0 tx=0).stop 2024-03-04T09:30:43.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:43.735+0000 7f7052a00640 1 -- 172.21.15.6:0/3653798254 shutdown_connections 2024-03-04T09:30:43.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:43.735+0000 7f7052a00640 1 --2- 172.21.15.6:0/3653798254 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f704c15e140 0x7f704c153e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:43.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:43.735+0000 7f7052a00640 1 --2- 172.21.15.6:0/3653798254 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f704c06fc80 0x7f704c153900 unknown :-1 s=CLOSED pgs=1133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:43.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:43.735+0000 7f7052a00640 1 --2- 172.21.15.6:0/3653798254 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f702c06b4d0 0x7f702c06d990 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:43.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:43.735+0000 7f7052a00640 1 --2- 172.21.15.6:0/3653798254 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f704c06afc0 0x7f704c151200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:43.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:43.735+0000 7f7052a00640 1 -- 172.21.15.6:0/3653798254 >> 172.21.15.6:0/3653798254 conn(0x7f704c0b2c40 msgr2=0x7f704c0aeba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:43.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:43.735+0000 7f7052a00640 1 -- 172.21.15.6:0/3653798254 shutdown_connections 2024-03-04T09:30:43.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:43.735+0000 7f7052a00640 1 -- 172.21.15.6:0/3653798254 wait complete. 2024-03-04T09:30:43.752 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump 2024-03-04T09:30:44.099 INFO:teuthology.orchestra.run.smithi006.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":44,"msgr_send_messages":46,"msgr_recv_bytes":18757,"msgr_send_bytes":6010857,"msgr_created_connections":9,"msgr_active_connections":5,"msgr_running_total_time":0.065935842,"msgr_running_send_time":0.044489703,"msgr_running_recv_time":0.008247501,"msgr_running_fast_dispatch_time":0.009453358,"msgr_send_messages_queue_lat":{"avgcount":45,"sum":0.022358331,"avgtime":0.000496851},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000208,"avgtime":0.000000104},"msgr_recv_encrypted_bytes":1728,"msgr_send_encrypted_bytes":59456},"AsyncMessenger::Worker-1":{"msgr_recv_messages":6817,"msgr_send_messages":12493,"msgr_recv_bytes":2464784,"msgr_send_bytes":5889153,"msgr_created_connections":7,"msgr_active_connections":5,"msgr_running_total_time":1.457705581,"msgr_running_send_time":0.971951030,"msgr_running_recv_time":1.040138390,"msgr_running_fast_dispatch_time":0.000910752,"msgr_send_messages_queue_lat":{"avgcount":12488,"sum":0.558389866,"avgtime":0.000044714},"msgr_handle_ack_lat":{"avgcount":15196,"sum":0.007398721,"avgtime":0.000000486},"msgr_recv_encrypted_bytes":0,"msgr_send_encrypted_bytes":46336},"AsyncMessenger::Worker-2":{"msgr_recv_messages":35,"msgr_send_messages":31,"msgr_recv_bytes":114006,"msgr_send_bytes":4206043,"msgr_created_connections":5,"msgr_active_connections":4,"msgr_running_total_time":0.029149163,"msgr_running_send_time":0.018962561,"msgr_running_recv_time":0.010060073,"msgr_running_fast_dispatch_time":0.003467555,"msgr_send_messages_queue_lat":{"avgcount":30,"sum":0.010539860,"avgtime":0.000351328},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":106000,"msgr_send_encrypted_bytes":6048},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":924,"complete_latency":{"avgcount":67,"sum":2.839853087,"avgtime":0.042385866}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":4,"sum":0.000237730,"avgtime":0.000059432}},"mds":{"request":3204,"reply":3204,"reply_latency":{"avgcount":3204,"sum":680.651966133,"avgtime":0.212438191},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":0,"dir_split":0,"dir_merge":0,"inodes":3211,"inodes_top":0,"inodes_bottom":3211,"inodes_pin_tail":0,"inodes_pinned":3211,"inodes_expired":0,"inodes_with_caps":3202,"caps":3202,"subtrees":2,"traverse":3206,"traverse_hit":3206,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":3001,"root_rbytes":19239936,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":1,"ceph_cap_op_grant":3199,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":3014,"handle_client_caps_dirty":2993,"handle_client_cap_release":0,"process_request_cap_release":3},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":0,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":0,"strays_enqueued":0,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":6217,"evex":0,"evtrm":0,"ev":6222,"evexg":0,"evexd":0,"segadd":6,"segex":0,"segtrm":0,"seg":8,"segmjr":2,"segexg":0,"segexd":0,"expos":4194304,"wrpos":17201544,"rdpos":4195674,"jlat":{"avgcount":10,"sum":1.040504545,"avgtime":0.104050454},"replayed":5},"mds_mem":{"ino":3214,"ino+":3214,"ino-":0,"dir":13,"dir+":13,"dir-":0,"dn":3211,"dn+":3211,"dn-":0,"cap":3202,"cap+":3202,"cap-":0,"rss":95984,"heap":248164},"mds_server":{"dispatch_client_request":3205,"dispatch_server_request":0,"handle_client_request":3204,"handle_client_session":0,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":679.388313371,"avgtime":0.212308847},"req_getattr_latency":{"avgcount":2,"sum":0.665016344,"avgtime":0.332508172},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.005488280,"avgtime":0.005488280},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":1,"sum":0.593148138,"avgtime":0.593148138},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":3084,"average_load":3084,"avg_session_uptime":926,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":15895036,"buffer_anon_items":9005,"buffer_meta_bytes":81664,"buffer_meta_items":928,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5672,"osdmap_items":92,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":12061524,"mds_co_items":120082,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":62,"op_send_bytes":13237100,"op_resend":0,"op_reply":62,"op_latency":{"avgcount":62,"sum":0.370995574,"avgtime":0.005983799},"op_inflight":0,"oplen_avg":{"avgcount":62,"sum":113},"op":62,"op_r":36,"op_w":26,"op_rmw":0,"op_pg":0,"osdop_stat":2,"osdop_create":0,"osdop_read":8,"osdop_write":10,"osdop_writefull":1,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":8,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":0,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":11,"omap_rd":48,"omap_del":0},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":0,"pq_executing":0,"pq_executing_high_water":0,"pq_executed_ops":0,"pq_executed":0,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":155693,"max":104857600,"get_started":0,"get":6892,"get_sum":1974698,"get_or_fail_fail":0,"get_or_fail_success":6892,"take":0,"take_sum":0,"put":6311,"put_sum":1819005,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":62,"take_sum":13239021,"put":62,"put_sum":13239021,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":62,"take_sum":62,"put":62,"put_sum":62,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":0,"take_sum":0,"put":0,"put_sum":0,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x5610c362caa0":{"val":33196,"max":3758096384,"get_started":0,"get":6217,"get_sum":13005055,"get_or_fail_fail":0,"get_or_fail_success":6217,"take":0,"take_sum":0,"put":10,"put_sum":12971859,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-03-04T09:30:44.101 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 5, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.04e-07, "sum": 2.08e-07 }, "msgr_recv_bytes": 18757, "msgr_recv_encrypted_bytes": 1728, "msgr_recv_messages": 44, "msgr_running_fast_dispatch_time": 0.009453358, "msgr_running_recv_time": 0.008247501, "msgr_running_send_time": 0.044489703, "msgr_running_total_time": 0.065935842, "msgr_send_bytes": 6010857, "msgr_send_encrypted_bytes": 59456, "msgr_send_messages": 46, "msgr_send_messages_queue_lat": { "avgcount": 45, "avgtime": 0.000496851, "sum": 0.022358331 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 5, "msgr_created_connections": 7, "msgr_handle_ack_lat": { "avgcount": 15196, "avgtime": 4.86e-07, "sum": 0.007398721 }, "msgr_recv_bytes": 2464784, "msgr_recv_encrypted_bytes": 0, "msgr_recv_messages": 6817, "msgr_running_fast_dispatch_time": 0.000910752, "msgr_running_recv_time": 1.04013839, "msgr_running_send_time": 0.97195103, "msgr_running_total_time": 1.457705581, "msgr_send_bytes": 5889153, "msgr_send_encrypted_bytes": 46336, "msgr_send_messages": 12493, "msgr_send_messages_queue_lat": { "avgcount": 12488, "avgtime": 4.4714e-05, "sum": 0.558389866 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 4, "msgr_created_connections": 5, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 114006, "msgr_recv_encrypted_bytes": 106000, "msgr_recv_messages": 35, "msgr_running_fast_dispatch_time": 0.003467555, "msgr_running_recv_time": 0.010060073, "msgr_running_send_time": 0.018962561, "msgr_running_total_time": 0.029149163, "msgr_send_bytes": 4206043, "msgr_send_encrypted_bytes": 6048, "msgr_send_messages": 31, "msgr_send_messages_queue_lat": { "avgcount": 30, "avgtime": 0.000351328, "sum": 0.01053986 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 67, "avgtime": 0.042385866, "sum": 2.839853087 }, "queue_len": 924 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 4, "avgtime": 5.9432e-05, "sum": 0.00023773 }, "queue_len": 0 }, "mds": { "caps": 3202, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 3199, "ceph_cap_op_revoke": 1, "ceph_cap_op_trunc": 0, "dir_commit": 0, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 0, "handle_client_caps": 3014, "handle_client_caps_dirty": 2993, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 3211, "inodes_bottom": 3211, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 3211, "inodes_top": 0, "inodes_with_caps": 3202, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 3, "q": 0, "reply": 3204, "reply_latency": { "avgcount": 3204, "avgtime": 0.212438191, "sum": 680.651966133 }, "request": 3204, "root_rbytes": 19239936, "root_rfiles": 3001, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 3206, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 3206, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 0, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 0, "strays_enqueued": 0, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 6222, "evadd": 6217, "evex": 0, "evexd": 0, "evexg": 0, "evlrg": 0, "evtrm": 0, "expos": 4194304, "jlat": { "avgcount": 10, "avgtime": 0.104050454, "sum": 1.040504545 }, "rdpos": 4195674, "replayed": 5, "seg": 8, "segadd": 6, "segex": 0, "segexd": 0, "segexg": 0, "segmjr": 2, "segtrm": 0, "wrpos": 17201544 }, "mds_mem": { "cap": 3202, "cap+": 3202, "cap-": 0, "dir": 13, "dir+": 13, "dir-": 0, "dn": 3211, "dn+": 3211, "dn-": 0, "heap": 248164, "ino": 3214, "ino+": 3214, "ino-": 0, "rss": 95984 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 3205, "dispatch_server_request": 0, "handle_client_request": 3204, "handle_client_session": 0, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.212308847, "sum": 679.388313371 }, "req_getattr_latency": { "avgcount": 2, "avgtime": 0.332508172, "sum": 0.665016344 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.00548828, "sum": 0.00548828 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 1, "avgtime": 0.593148138, "sum": 0.593148138 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "mds_sessions": { "average_load": 3084, "avg_session_uptime": 926, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 3084 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 15895036, "buffer_anon_items": 9005, "buffer_meta_bytes": 81664, "buffer_meta_items": 928, "mds_co_bytes": 12061524, "mds_co_items": 120082, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5672, "osdmap_items": 92, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 0, "omap_rd": 48, "omap_wr": 11, "op": 62, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 62, "avgtime": 0.005983799, "sum": 0.370995574 }, "op_pg": 0, "op_r": 36, "op_reply": 62, "op_resend": 0, "op_rmw": 0, "op_send": 62, "op_send_bytes": 13237100, "op_w": 26, "oplen_avg": { "avgcount": 62, "sum": 113 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 0, "osdop_delete": 8, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 8, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 0, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 2, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 10, "osdop_writefull": 1, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 0, "pq_executed_ops": 0, "pq_executing": 0, "pq_executing_high_water": 0, "pq_executing_ops": 0, "pq_executing_ops_high_water": 0, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 6892, "get_or_fail_fail": 0, "get_or_fail_success": 6892, "get_started": 0, "get_sum": 1974698, "max": 104857600, "put": 6311, "put_sum": 1819005, "take": 0, "take_sum": 0, "val": 155693, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 62, "put_sum": 13239021, "take": 62, "take_sum": 13239021, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 62, "put_sum": 62, "take": 62, "take_sum": 62, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 3758096384, "put": 0, "put_sum": 0, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x5610c362caa0": { "get": 6217, "get_or_fail_fail": 0, "get_or_fail_success": 6217, "get_started": 0, "get_sum": 13005055, "max": 3758096384, "put": 10, "put_sum": 12971859, "take": 0, "take_sum": 0, "val": 33196, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-03-04T09:30:44.101 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/0, 0/0 2024-03-04T09:30:45.103 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:30:45.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:45.325+0000 7f903a800640 1 Processor -- start 2024-03-04T09:30:45.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:45.325+0000 7f903a800640 1 -- start start 2024-03-04T09:30:45.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:45.327+0000 7f903a800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f90340b4b30 0x7f90341623b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:45.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:45.327+0000 7f903a800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f90340d1be0 0x7f90340d2070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:45.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:45.327+0000 7f903a800640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90340b50d0 con 0x7f903415f630 2024-03-04T09:30:45.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:45.327+0000 7f903a800640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90340cf450 con 0x7f90340d1be0 2024-03-04T09:30:45.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:45.327+0000 7f903a800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90340cf6a0 con 0x7f90340b4b30 2024-03-04T09:30:45.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:45.328+0000 7f9038a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f90340b4b30 0x7f90341623b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:45.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:45.328+0000 7f9039e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f90340d1be0 0x7f90340d2070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:45.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:45.328+0000 7f9038a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f90340b4b30 0x7f90341623b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.6:34700/0 (socket says 172.21.15.6:34700) 2024-03-04T09:30:45.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:45.328+0000 7f9038a00640 1 -- 172.21.15.6:0/4040757832 learned_addr learned my addr 172.21.15.6:0/4040757832 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:30:45.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:45.329+0000 7f9039e00640 1 -- 172.21.15.6:0/4040757832 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f90340b4b30 msgr2=0x7f90341623b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:45.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:45.329+0000 7f9039e00640 1 --2- 172.21.15.6:0/4040757832 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f90340b4b30 0x7f90341623b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:45.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:45.329+0000 7f9039e00640 1 -- 172.21.15.6:0/4040757832 >> v1:172.21.15.146:6789/0 conn(0x7f903415f630 legacy=0x7f9034161ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:45.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:45.329+0000 7f9039e00640 1 -- 172.21.15.6:0/4040757832 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90340cf980 con 0x7f90340d1be0 2024-03-04T09:30:45.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:45.330+0000 7f9039e00640 1 --2- 172.21.15.6:0/4040757832 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f90340d1be0 0x7f90340d2070 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f902c009d90 tx=0x7f902c02bd80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3ffa32da8659893b server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:45.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:45.331+0000 7f9033e00640 1 -- 172.21.15.6:0/4040757832 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f902c039070 con 0x7f90340d1be0 2024-03-04T09:30:45.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:45.331+0000 7f9033e00640 1 -- 172.21.15.6:0/4040757832 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f902c02cb30 con 0x7f90340d1be0 2024-03-04T09:30:45.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:45.331+0000 7f9033e00640 1 -- 172.21.15.6:0/4040757832 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f902c034440 con 0x7f90340d1be0 2024-03-04T09:30:45.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:45.332+0000 7f903a800640 1 -- 172.21.15.6:0/4040757832 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f90340d1be0 msgr2=0x7f90340d2070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:45.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:45.332+0000 7f903a800640 1 --2- 172.21.15.6:0/4040757832 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f90340d1be0 0x7f90340d2070 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f902c009d90 tx=0x7f902c02bd80 comp rx=0 tx=0).stop 2024-03-04T09:30:45.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:45.332+0000 7f903a800640 1 -- 172.21.15.6:0/4040757832 shutdown_connections 2024-03-04T09:30:45.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:45.332+0000 7f903a800640 1 --2- 172.21.15.6:0/4040757832 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f90340d1be0 0x7f90340d2070 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:45.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:45.332+0000 7f903a800640 1 --2- 172.21.15.6:0/4040757832 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f90340b4b30 0x7f90341623b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:45.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:45.332+0000 7f903a800640 1 -- 172.21.15.6:0/4040757832 >> 172.21.15.6:0/4040757832 conn(0x7f90340d3e30 msgr2=0x7f90340d4240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:45.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:45.332+0000 7f903a800640 1 -- 172.21.15.6:0/4040757832 shutdown_connections 2024-03-04T09:30:45.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:45.332+0000 7f903a800640 1 -- 172.21.15.6:0/4040757832 wait complete. 2024-03-04T09:30:45.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:45.334+0000 7f903a800640 1 Processor -- start 2024-03-04T09:30:45.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:45.334+0000 7f903a800640 1 -- start start 2024-03-04T09:30:45.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:45.335+0000 7f903a800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f90340b4b30 0x7f90340ca370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:45.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:45.335+0000 7f903a800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f90340d1be0 0x7f903414d130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:45.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:45.335+0000 7f903a800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f903415f630 0x7f90340cc800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:45.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:45.335+0000 7f903a800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90341628f0 con 0x7f90340d1be0 2024-03-04T09:30:45.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:45.335+0000 7f903a800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90340d4c40 con 0x7f903415f630 2024-03-04T09:30:45.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:45.335+0000 7f903a800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9034162a70 con 0x7f90340b4b30 2024-03-04T09:30:45.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:45.336+0000 7f9038a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f90340d1be0 0x7f903414d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:45.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:45.336+0000 7f9038a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f90340d1be0 0x7f903414d130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37200/0 (socket says 172.21.15.6:37200) 2024-03-04T09:30:45.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:45.336+0000 7f9038a00640 1 -- 172.21.15.6:0/2681789178 learned_addr learned my addr 172.21.15.6:0/2681789178 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:30:45.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:45.336+0000 7f9039e00640 1 --2- 172.21.15.6:0/2681789178 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f903415f630 0x7f90340cc800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:45.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:45.336+0000 7f9039400640 1 --2- 172.21.15.6:0/2681789178 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f90340b4b30 0x7f90340ca370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:45.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:45.336+0000 7f9038a00640 1 -- 172.21.15.6:0/2681789178 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f90340b4b30 msgr2=0x7f90340ca370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:45.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:45.336+0000 7f9038a00640 1 --2- 172.21.15.6:0/2681789178 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f90340b4b30 0x7f90340ca370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:45.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:45.336+0000 7f9038a00640 1 -- 172.21.15.6:0/2681789178 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f903415f630 msgr2=0x7f90340cc800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:45.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:45.336+0000 7f9038a00640 1 --2- 172.21.15.6:0/2681789178 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f903415f630 0x7f90340cc800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:45.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:45.336+0000 7f9038a00640 1 -- 172.21.15.6:0/2681789178 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90340c7980 con 0x7f90340d1be0 2024-03-04T09:30:45.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:45.337+0000 7f9038a00640 1 --2- 172.21.15.6:0/2681789178 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f90340d1be0 0x7f903414d130 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7f902000e870 tx=0x7f902000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:45.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:45.337+0000 7f9031600640 1 -- 172.21.15.6:0/2681789178 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9020017070 con 0x7f90340d1be0 2024-03-04T09:30:45.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:45.337+0000 7f9031600640 1 -- 172.21.15.6:0/2681789178 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f902000c8e0 con 0x7f90340d1be0 2024-03-04T09:30:45.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:45.337+0000 7f903a800640 1 -- 172.21.15.6:0/2681789178 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90340c7c40 con 0x7f90340d1be0 2024-03-04T09:30:45.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:45.337+0000 7f9031600640 1 -- 172.21.15.6:0/2681789178 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f902000ccd0 con 0x7f90340d1be0 2024-03-04T09:30:45.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:45.337+0000 7f903a800640 1 -- 172.21.15.6:0/2681789178 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f903415ce30 con 0x7f90340d1be0 2024-03-04T09:30:45.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:45.339+0000 7f903a800640 1 -- 172.21.15.6:0/2681789178 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f903415cc20 con 0x7f90340d1be0 2024-03-04T09:30:45.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:45.340+0000 7f9031600640 1 -- 172.21.15.6:0/2681789178 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f902001a070 con 0x7f90340d1be0 2024-03-04T09:30:45.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:45.341+0000 7f9031600640 1 --2- 172.21.15.6:0/2681789178 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f901406b520 0x7f901406d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:45.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:45.341+0000 7f9031600640 1 -- 172.21.15.6:0/2681789178 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f902008a110 con 0x7f90340d1be0 2024-03-04T09:30:45.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:45.342+0000 7f9039400640 1 --2- 172.21.15.6:0/2681789178 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f901406b520 0x7f901406d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:45.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:45.343+0000 7f9039400640 1 --2- 172.21.15.6:0/2681789178 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f901406b520 0x7f901406d9e0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f9028000f60 tx=0x7f9028009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:45.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:45.351+0000 7f9031600640 1 -- 172.21.15.6:0/2681789178 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9020059010 con 0x7f90340d1be0 2024-03-04T09:30:45.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:45.736+0000 7f903a800640 1 -- 172.21.15.6:0/2681789178 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f903404f330 con 0x7f90340d1be0 2024-03-04T09:30:45.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:45.737+0000 7f9031600640 1 -- 172.21.15.6:0/2681789178 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) v1 ==== 94+0+4977 (secure 0 0 0) 0x7f9020058910 con 0x7f90340d1be0 2024-03-04T09:30:45.737 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:30:45.737 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":68,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5654,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/1370302030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":1370302030},{"type":"v1","addr":"172.21.15.146:6835","nonce":1370302030}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":6163,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/809822004","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":809822004},{"type":"v1","addr":"172.21.15.146:6837","nonce":809822004}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":66},{"gid":7284,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/1583413840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1583413840},{"type":"v1","addr":"172.21.15.6:6833","nonce":1583413840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64}],"filesystems":[{"mdsmap":{"epoch":68,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:30:20.660103+0000","modified":"2024-03-04T09:30:36.289706+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":98,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7296},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7296":{"gid":7296,"name":"a","rank":0,"incarnation":65,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1732383836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1732383836},{"type":"v1","addr":"172.21.15.6:6835","nonce":1732383836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7296,"qdb_cluster":[7296]},"id":7}]} 2024-03-04T09:30:45.738 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 68 2024-03-04T09:30:45.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:45.739+0000 7f903a800640 1 -- 172.21.15.6:0/2681789178 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f901406b520 msgr2=0x7f901406d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:45.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:45.739+0000 7f903a800640 1 --2- 172.21.15.6:0/2681789178 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f901406b520 0x7f901406d9e0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f9028000f60 tx=0x7f9028009040 comp rx=0 tx=0).stop 2024-03-04T09:30:45.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:45.739+0000 7f903a800640 1 -- 172.21.15.6:0/2681789178 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f90340d1be0 msgr2=0x7f903414d130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:45.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:45.739+0000 7f903a800640 1 --2- 172.21.15.6:0/2681789178 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f90340d1be0 0x7f903414d130 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7f902000e870 tx=0x7f902000ed30 comp rx=0 tx=0).stop 2024-03-04T09:30:45.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:45.739+0000 7f903a800640 1 -- 172.21.15.6:0/2681789178 shutdown_connections 2024-03-04T09:30:45.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:45.739+0000 7f903a800640 1 --2- 172.21.15.6:0/2681789178 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f903415f630 0x7f90340cc800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:45.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:45.740+0000 7f903a800640 1 --2- 172.21.15.6:0/2681789178 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f90340d1be0 0x7f903414d130 unknown :-1 s=CLOSED pgs=1134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:45.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:45.740+0000 7f903a800640 1 --2- 172.21.15.6:0/2681789178 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f901406b520 0x7f901406d9e0 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:45.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:45.740+0000 7f903a800640 1 --2- 172.21.15.6:0/2681789178 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f90340b4b30 0x7f90340ca370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:45.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:45.740+0000 7f903a800640 1 -- 172.21.15.6:0/2681789178 >> 172.21.15.6:0/2681789178 conn(0x7f90340d3e30 msgr2=0x7f90340d4210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:45.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:45.740+0000 7f903a800640 1 -- 172.21.15.6:0/2681789178 shutdown_connections 2024-03-04T09:30:45.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:45.740+0000 7f903a800640 1 -- 172.21.15.6:0/2681789178 wait complete. 2024-03-04T09:30:45.761 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump 2024-03-04T09:30:45.996 INFO:teuthology.orchestra.run.smithi006.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":47,"msgr_send_messages":50,"msgr_recv_bytes":19629,"msgr_send_bytes":9868245,"msgr_created_connections":9,"msgr_active_connections":5,"msgr_running_total_time":0.088915439,"msgr_running_send_time":0.064243103,"msgr_running_recv_time":0.008329092,"msgr_running_fast_dispatch_time":0.012528990,"msgr_send_messages_queue_lat":{"avgcount":49,"sum":0.024668366,"avgtime":0.000503436},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000208,"avgtime":0.000000104},"msgr_recv_encrypted_bytes":1728,"msgr_send_encrypted_bytes":61248},"AsyncMessenger::Worker-1":{"msgr_recv_messages":7877,"msgr_send_messages":13730,"msgr_recv_bytes":2829829,"msgr_send_bytes":8074690,"msgr_created_connections":7,"msgr_active_connections":5,"msgr_running_total_time":1.700870977,"msgr_running_send_time":1.106553269,"msgr_running_recv_time":1.198803724,"msgr_running_fast_dispatch_time":0.001109232,"msgr_send_messages_queue_lat":{"avgcount":13728,"sum":0.709462085,"avgtime":0.000051679},"msgr_handle_ack_lat":{"avgcount":17443,"sum":0.008331549,"avgtime":0.000000477},"msgr_recv_encrypted_bytes":0,"msgr_send_encrypted_bytes":46336},"AsyncMessenger::Worker-2":{"msgr_recv_messages":35,"msgr_send_messages":31,"msgr_recv_bytes":114006,"msgr_send_bytes":4206043,"msgr_created_connections":5,"msgr_active_connections":4,"msgr_running_total_time":0.029243028,"msgr_running_send_time":0.019007846,"msgr_running_recv_time":0.010082606,"msgr_running_fast_dispatch_time":0.003467555,"msgr_send_messages_queue_lat":{"avgcount":30,"sum":0.010539860,"avgtime":0.000351328},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":106000,"msgr_send_encrypted_bytes":6048},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":1411,"complete_latency":{"avgcount":72,"sum":3.316724928,"avgtime":0.046065624}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":4,"sum":0.000237730,"avgtime":0.000059432}},"mds":{"request":4155,"reply":4155,"reply_latency":{"avgcount":4155,"sum":682.263383439,"avgtime":0.164202980},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":0,"dir_split":0,"dir_merge":0,"inodes":3211,"inodes_top":0,"inodes_bottom":3211,"inodes_pin_tail":0,"inodes_pinned":3211,"inodes_expired":0,"inodes_with_caps":3202,"caps":3202,"subtrees":2,"traverse":4157,"traverse_hit":4157,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":3001,"root_rbytes":19239936,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":1,"ceph_cap_op_grant":3367,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":3700,"handle_client_caps_dirty":2996,"handle_client_cap_release":0,"process_request_cap_release":954},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":946,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":946,"strays_enqueued":0,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":7790,"evex":0,"evtrm":0,"ev":7795,"evexg":0,"evexd":0,"segadd":8,"segex":0,"segtrm":0,"seg":10,"segmjr":2,"segexg":0,"segexd":0,"expos":4194304,"wrpos":21521929,"rdpos":4195674,"jlat":{"avgcount":11,"sum":1.104145275,"avgtime":0.100376843},"replayed":5},"mds_mem":{"ino":3214,"ino+":3214,"ino-":0,"dir":13,"dir+":13,"dir-":0,"dn":4157,"dn+":4157,"dn-":0,"cap":3202,"cap+":3202,"cap-":0,"rss":96496,"heap":248164},"mds_server":{"dispatch_client_request":4156,"dispatch_server_request":0,"handle_client_request":4155,"handle_client_session":0,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":679.388313371,"avgtime":0.212308847},"req_getattr_latency":{"avgcount":2,"sum":0.665016344,"avgtime":0.332508172},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.005488280,"avgtime":0.005488280},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":6,"sum":1.043772762,"avgtime":0.173962127},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":946,"sum":1.160792682,"avgtime":0.001227053},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":3955,"average_load":3955,"avg_session_uptime":926,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":15895036,"buffer_anon_items":9005,"buffer_meta_bytes":81664,"buffer_meta_items":928,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5672,"osdmap_items":92,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":12061524,"mds_co_items":120082,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":66,"op_send_bytes":17091620,"op_resend":0,"op_reply":66,"op_latency":{"avgcount":66,"sum":0.442994714,"avgtime":0.006712041},"op_inflight":0,"oplen_avg":{"avgcount":66,"sum":120},"op":66,"op_r":36,"op_w":30,"op_rmw":0,"op_pg":0,"osdop_stat":2,"osdop_create":0,"osdop_read":8,"osdop_write":11,"osdop_writefull":1,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":9,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":0,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":14,"omap_rd":48,"omap_del":2},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":0,"pq_executing":0,"pq_executing_high_water":0,"pq_executed_ops":0,"pq_executed":0,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":252,"max":104857600,"get_started":0,"get":7956,"get_sum":2245197,"get_or_fail_fail":0,"get_or_fail_success":7956,"take":0,"take_sum":0,"put":7955,"put_sum":2244945,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":66,"take_sum":17093541,"put":66,"put_sum":17093541,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":66,"take_sum":66,"put":66,"put_sum":66,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":0,"take_sum":0,"put":0,"put_sum":0,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x5610c362caa0":{"val":550409,"max":3758096384,"get_started":0,"get":7790,"get_sum":17325440,"get_or_fail_fail":0,"get_or_fail_success":7790,"take":0,"take_sum":0,"put":11,"put_sum":16775031,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-03-04T09:30:45.999 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 5, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.04e-07, "sum": 2.08e-07 }, "msgr_recv_bytes": 19629, "msgr_recv_encrypted_bytes": 1728, "msgr_recv_messages": 47, "msgr_running_fast_dispatch_time": 0.01252899, "msgr_running_recv_time": 0.008329092, "msgr_running_send_time": 0.064243103, "msgr_running_total_time": 0.088915439, "msgr_send_bytes": 9868245, "msgr_send_encrypted_bytes": 61248, "msgr_send_messages": 50, "msgr_send_messages_queue_lat": { "avgcount": 49, "avgtime": 0.000503436, "sum": 0.024668366 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 5, "msgr_created_connections": 7, "msgr_handle_ack_lat": { "avgcount": 17443, "avgtime": 4.77e-07, "sum": 0.008331549 }, "msgr_recv_bytes": 2829829, "msgr_recv_encrypted_bytes": 0, "msgr_recv_messages": 7877, "msgr_running_fast_dispatch_time": 0.001109232, "msgr_running_recv_time": 1.198803724, "msgr_running_send_time": 1.106553269, "msgr_running_total_time": 1.700870977, "msgr_send_bytes": 8074690, "msgr_send_encrypted_bytes": 46336, "msgr_send_messages": 13730, "msgr_send_messages_queue_lat": { "avgcount": 13728, "avgtime": 5.1679e-05, "sum": 0.709462085 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 4, "msgr_created_connections": 5, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 114006, "msgr_recv_encrypted_bytes": 106000, "msgr_recv_messages": 35, "msgr_running_fast_dispatch_time": 0.003467555, "msgr_running_recv_time": 0.010082606, "msgr_running_send_time": 0.019007846, "msgr_running_total_time": 0.029243028, "msgr_send_bytes": 4206043, "msgr_send_encrypted_bytes": 6048, "msgr_send_messages": 31, "msgr_send_messages_queue_lat": { "avgcount": 30, "avgtime": 0.000351328, "sum": 0.01053986 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 72, "avgtime": 0.046065624, "sum": 3.316724928 }, "queue_len": 1411 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 4, "avgtime": 5.9432e-05, "sum": 0.00023773 }, "queue_len": 0 }, "mds": { "caps": 3202, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 3367, "ceph_cap_op_revoke": 1, "ceph_cap_op_trunc": 0, "dir_commit": 0, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 0, "handle_client_caps": 3700, "handle_client_caps_dirty": 2996, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 3211, "inodes_bottom": 3211, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 3211, "inodes_top": 0, "inodes_with_caps": 3202, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 954, "q": 0, "reply": 4155, "reply_latency": { "avgcount": 4155, "avgtime": 0.16420298, "sum": 682.263383439 }, "request": 4155, "root_rbytes": 19239936, "root_rfiles": 3001, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 4157, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 4157, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 946, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 946, "strays_enqueued": 0, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 7795, "evadd": 7790, "evex": 0, "evexd": 0, "evexg": 0, "evlrg": 0, "evtrm": 0, "expos": 4194304, "jlat": { "avgcount": 11, "avgtime": 0.100376843, "sum": 1.104145275 }, "rdpos": 4195674, "replayed": 5, "seg": 10, "segadd": 8, "segex": 0, "segexd": 0, "segexg": 0, "segmjr": 2, "segtrm": 0, "wrpos": 21521929 }, "mds_mem": { "cap": 3202, "cap+": 3202, "cap-": 0, "dir": 13, "dir+": 13, "dir-": 0, "dn": 4157, "dn+": 4157, "dn-": 0, "heap": 248164, "ino": 3214, "ino+": 3214, "ino-": 0, "rss": 96496 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 4156, "dispatch_server_request": 0, "handle_client_request": 4155, "handle_client_session": 0, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.212308847, "sum": 679.388313371 }, "req_getattr_latency": { "avgcount": 2, "avgtime": 0.332508172, "sum": 0.665016344 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.00548828, "sum": 0.00548828 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 6, "avgtime": 0.173962127, "sum": 1.043772762 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 946, "avgtime": 0.001227053, "sum": 1.160792682 } }, "mds_sessions": { "average_load": 3955, "avg_session_uptime": 926, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 3955 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 15895036, "buffer_anon_items": 9005, "buffer_meta_bytes": 81664, "buffer_meta_items": 928, "mds_co_bytes": 12061524, "mds_co_items": 120082, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5672, "osdmap_items": 92, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 2, "omap_rd": 48, "omap_wr": 14, "op": 66, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 66, "avgtime": 0.006712041, "sum": 0.442994714 }, "op_pg": 0, "op_r": 36, "op_reply": 66, "op_resend": 0, "op_rmw": 0, "op_send": 66, "op_send_bytes": 17091620, "op_w": 30, "oplen_avg": { "avgcount": 66, "sum": 120 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 0, "osdop_delete": 9, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 8, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 0, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 2, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 11, "osdop_writefull": 1, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 0, "pq_executed_ops": 0, "pq_executing": 0, "pq_executing_high_water": 0, "pq_executing_ops": 0, "pq_executing_ops_high_water": 0, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 7956, "get_or_fail_fail": 0, "get_or_fail_success": 7956, "get_started": 0, "get_sum": 2245197, "max": 104857600, "put": 7955, "put_sum": 2244945, "take": 0, "take_sum": 0, "val": 252, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 66, "put_sum": 17093541, "take": 66, "take_sum": 17093541, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 66, "put_sum": 66, "take": 66, "take_sum": 66, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 3758096384, "put": 0, "put_sum": 0, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x5610c362caa0": { "get": 7790, "get_or_fail_fail": 0, "get_or_fail_success": 7790, "get_started": 0, "get_sum": 17325440, "max": 3758096384, "put": 11, "put_sum": 16775031, "take": 0, "take_sum": 0, "val": 550409, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-03-04T09:30:45.999 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/946, 0/946 2024-03-04T09:30:47.000 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:30:47.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:47.212+0000 7fdb58c00640 1 Processor -- start 2024-03-04T09:30:47.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:47.213+0000 7fdb58c00640 1 -- start start 2024-03-04T09:30:47.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:47.215+0000 7fdb58c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fdb541457e0 0x7fdb540cf5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:47.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:47.215+0000 7fdb58c00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fdb540b5870 0x7fdb540710b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:47.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:47.215+0000 7fdb58c00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb54145bc0 con 0x7fdb5415fba0 2024-03-04T09:30:47.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:47.215+0000 7fdb58c00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb540cfb20 con 0x7fdb541457e0 2024-03-04T09:30:47.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:47.215+0000 7fdb58c00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb54160040 con 0x7fdb540b5870 2024-03-04T09:30:47.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:47.215+0000 7fdb51e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fdb541457e0 0x7fdb540cf5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:47.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:47.215+0000 7fdb51e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fdb541457e0 0x7fdb540cf5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37212/0 (socket says 172.21.15.6:37212) 2024-03-04T09:30:47.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:47.215+0000 7fdb51e00640 1 -- 172.21.15.6:0/18130041 learned_addr learned my addr 172.21.15.6:0/18130041 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:30:47.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:47.216+0000 7fdb52800640 1 --2- 172.21.15.6:0/18130041 >> v2:172.21.15.146:3300/0 conn(0x7fdb540b5870 0x7fdb540710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:47.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:47.216+0000 7fdb51e00640 1 -- 172.21.15.6:0/18130041 >> v2:172.21.15.146:3300/0 conn(0x7fdb540b5870 msgr2=0x7fdb540710b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:47.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:47.216+0000 7fdb51e00640 1 --2- 172.21.15.6:0/18130041 >> v2:172.21.15.146:3300/0 conn(0x7fdb540b5870 0x7fdb540710b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:47.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:47.216+0000 7fdb51e00640 1 -- 172.21.15.6:0/18130041 >> v1:172.21.15.146:6789/0 conn(0x7fdb5415fba0 legacy=0x7fdb540b5160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:47.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:47.216+0000 7fdb51e00640 1 -- 172.21.15.6:0/18130041 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb54160320 con 0x7fdb541457e0 2024-03-04T09:30:47.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:47.217+0000 7fdb51e00640 1 --2- 172.21.15.6:0/18130041 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdb541457e0 0x7fdb540cf5e0 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7fdb5406f870 tx=0x7fdb4402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f92ee9d10955d22 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:47.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:47.218+0000 7fdb50a00640 1 -- 172.21.15.6:0/18130041 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb4402e070 con 0x7fdb541457e0 2024-03-04T09:30:47.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:47.218+0000 7fdb50a00640 1 -- 172.21.15.6:0/18130041 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb44037070 con 0x7fdb541457e0 2024-03-04T09:30:47.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:47.218+0000 7fdb50a00640 1 -- 172.21.15.6:0/18130041 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb44031a00 con 0x7fdb541457e0 2024-03-04T09:30:47.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:47.219+0000 7fdb58c00640 1 -- 172.21.15.6:0/18130041 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdb541457e0 msgr2=0x7fdb540cf5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:47.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:47.219+0000 7fdb58c00640 1 --2- 172.21.15.6:0/18130041 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdb541457e0 0x7fdb540cf5e0 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7fdb5406f870 tx=0x7fdb4402b880 comp rx=0 tx=0).stop 2024-03-04T09:30:47.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:47.219+0000 7fdb58c00640 1 -- 172.21.15.6:0/18130041 shutdown_connections 2024-03-04T09:30:47.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:47.219+0000 7fdb58c00640 1 --2- 172.21.15.6:0/18130041 >> v2:172.21.15.146:3300/0 conn(0x7fdb540b5870 0x7fdb540710b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:47.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:47.219+0000 7fdb58c00640 1 --2- 172.21.15.6:0/18130041 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdb541457e0 0x7fdb540cf5e0 unknown :-1 s=CLOSED pgs=1135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:47.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:47.219+0000 7fdb58c00640 1 -- 172.21.15.6:0/18130041 >> 172.21.15.6:0/18130041 conn(0x7fdb5406cf40 msgr2=0x7fdb5406d320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:47.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:47.219+0000 7fdb58c00640 1 -- 172.21.15.6:0/18130041 shutdown_connections 2024-03-04T09:30:47.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:47.219+0000 7fdb58c00640 1 -- 172.21.15.6:0/18130041 wait complete. 2024-03-04T09:30:47.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:47.221+0000 7fdb58c00640 1 Processor -- start 2024-03-04T09:30:47.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:47.221+0000 7fdb58c00640 1 -- start start 2024-03-04T09:30:47.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:47.222+0000 7fdb58c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdb540b5870 0x7fdb540aef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:47.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:47.222+0000 7fdb58c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdb541457e0 0x7fdb5413d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:47.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:47.222+0000 7fdb58c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdb5415fba0 0x7fdb5406ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:47.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:47.222+0000 7fdb58c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb540cca70 con 0x7fdb5415fba0 2024-03-04T09:30:47.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:47.222+0000 7fdb58c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb541608e0 con 0x7fdb540b5870 2024-03-04T09:30:47.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:47.222+0000 7fdb58c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb540715f0 con 0x7fdb541457e0 2024-03-04T09:30:47.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:47.223+0000 7fdb52800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdb5415fba0 0x7fdb5406ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:47.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:47.223+0000 7fdb52800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdb5415fba0 0x7fdb5406ddc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37214/0 (socket says 172.21.15.6:37214) 2024-03-04T09:30:47.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:47.223+0000 7fdb52800640 1 -- 172.21.15.6:0/4160926168 learned_addr learned my addr 172.21.15.6:0/4160926168 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:30:47.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:47.223+0000 7fdb51e00640 1 --2- 172.21.15.6:0/4160926168 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdb540b5870 0x7fdb540aef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:47.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:47.223+0000 7fdb51400640 1 --2- 172.21.15.6:0/4160926168 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdb541457e0 0x7fdb5413d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:47.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:47.223+0000 7fdb52800640 1 -- 172.21.15.6:0/4160926168 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdb541457e0 msgr2=0x7fdb5413d1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:47.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:47.223+0000 7fdb52800640 1 --2- 172.21.15.6:0/4160926168 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdb541457e0 0x7fdb5413d1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:47.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:47.223+0000 7fdb52800640 1 -- 172.21.15.6:0/4160926168 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdb540b5870 msgr2=0x7fdb540aef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:47.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:47.223+0000 7fdb52800640 1 --2- 172.21.15.6:0/4160926168 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdb540b5870 0x7fdb540aef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:47.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:47.223+0000 7fdb52800640 1 -- 172.21.15.6:0/4160926168 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb54140cd0 con 0x7fdb5415fba0 2024-03-04T09:30:47.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:47.223+0000 7fdb51400640 1 --2- 172.21.15.6:0/4160926168 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdb541457e0 0x7fdb5413d1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T09:30:47.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:47.223+0000 7fdb51e00640 1 --2- 172.21.15.6:0/4160926168 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdb540b5870 0x7fdb540aef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T09:30:47.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:47.224+0000 7fdb52800640 1 --2- 172.21.15.6:0/4160926168 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdb5415fba0 0x7fdb5406ddc0 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7fdb4c007e40 tx=0x7fdb4c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:47.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:47.224+0000 7fdb4a000640 1 -- 172.21.15.6:0/4160926168 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb4c014620 con 0x7fdb5415fba0 2024-03-04T09:30:47.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:47.224+0000 7fdb58c00640 1 -- 172.21.15.6:0/4160926168 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb54140fb0 con 0x7fdb5415fba0 2024-03-04T09:30:47.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:47.224+0000 7fdb58c00640 1 -- 172.21.15.6:0/4160926168 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb5413aac0 con 0x7fdb5415fba0 2024-03-04T09:30:47.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:47.224+0000 7fdb4a000640 1 -- 172.21.15.6:0/4160926168 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb4c019070 con 0x7fdb5415fba0 2024-03-04T09:30:47.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:47.224+0000 7fdb4a000640 1 -- 172.21.15.6:0/4160926168 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb4c014a20 con 0x7fdb5415fba0 2024-03-04T09:30:47.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:47.226+0000 7fdb58c00640 1 -- 172.21.15.6:0/4160926168 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb2c005290 con 0x7fdb5415fba0 2024-03-04T09:30:47.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:47.227+0000 7fdb4a000640 1 -- 172.21.15.6:0/4160926168 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdb4c014bc0 con 0x7fdb5415fba0 2024-03-04T09:30:47.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:47.228+0000 7fdb4a000640 1 --2- 172.21.15.6:0/4160926168 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fdb3406b500 0x7fdb3406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:47.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:47.228+0000 7fdb4a000640 1 -- 172.21.15.6:0/4160926168 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5881+0+0 (secure 0 0 0) 0x7fdb4c08abd0 con 0x7fdb5415fba0 2024-03-04T09:30:47.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:47.229+0000 7fdb51e00640 1 --2- 172.21.15.6:0/4160926168 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fdb3406b500 0x7fdb3406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:47.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:47.230+0000 7fdb51e00640 1 --2- 172.21.15.6:0/4160926168 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fdb3406b500 0x7fdb3406d9c0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fdb44036000 tx=0x7fdb4403d040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:47.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:47.232+0000 7fdb4a000640 1 -- 172.21.15.6:0/4160926168 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdb4c059900 con 0x7fdb5415fba0 2024-03-04T09:30:47.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:47.579+0000 7fdb58c00640 1 -- 172.21.15.6:0/4160926168 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdb2c005580 con 0x7fdb5415fba0 2024-03-04T09:30:47.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:47.580+0000 7fdb4a000640 1 -- 172.21.15.6:0/4160926168 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) v1 ==== 94+0+4977 (secure 0 0 0) 0x7fdb4c059200 con 0x7fdb5415fba0 2024-03-04T09:30:47.577 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:30:47.577 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":68,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5654,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/1370302030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":1370302030},{"type":"v1","addr":"172.21.15.146:6835","nonce":1370302030}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":6163,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/809822004","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":809822004},{"type":"v1","addr":"172.21.15.146:6837","nonce":809822004}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":66},{"gid":7284,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/1583413840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1583413840},{"type":"v1","addr":"172.21.15.6:6833","nonce":1583413840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64}],"filesystems":[{"mdsmap":{"epoch":68,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:30:20.660103+0000","modified":"2024-03-04T09:30:36.289706+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":98,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7296},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7296":{"gid":7296,"name":"a","rank":0,"incarnation":65,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1732383836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1732383836},{"type":"v1","addr":"172.21.15.6:6835","nonce":1732383836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7296,"qdb_cluster":[7296]},"id":7}]} 2024-03-04T09:30:47.578 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 68 2024-03-04T09:30:47.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:47.583+0000 7fdb58c00640 1 -- 172.21.15.6:0/4160926168 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fdb3406b500 msgr2=0x7fdb3406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:47.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:47.583+0000 7fdb58c00640 1 --2- 172.21.15.6:0/4160926168 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fdb3406b500 0x7fdb3406d9c0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fdb44036000 tx=0x7fdb4403d040 comp rx=0 tx=0).stop 2024-03-04T09:30:47.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:47.583+0000 7fdb58c00640 1 -- 172.21.15.6:0/4160926168 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdb5415fba0 msgr2=0x7fdb5406ddc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:47.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:47.583+0000 7fdb58c00640 1 --2- 172.21.15.6:0/4160926168 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdb5415fba0 0x7fdb5406ddc0 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7fdb4c007e40 tx=0x7fdb4c00e4b0 comp rx=0 tx=0).stop 2024-03-04T09:30:47.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:47.583+0000 7fdb58c00640 1 -- 172.21.15.6:0/4160926168 shutdown_connections 2024-03-04T09:30:47.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:47.583+0000 7fdb58c00640 1 --2- 172.21.15.6:0/4160926168 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdb5415fba0 0x7fdb5406ddc0 unknown :-1 s=CLOSED pgs=1136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:47.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:47.583+0000 7fdb58c00640 1 --2- 172.21.15.6:0/4160926168 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fdb3406b500 0x7fdb3406d9c0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:47.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:47.583+0000 7fdb58c00640 1 --2- 172.21.15.6:0/4160926168 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdb541457e0 0x7fdb5413d1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:47.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:47.583+0000 7fdb58c00640 1 --2- 172.21.15.6:0/4160926168 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdb540b5870 0x7fdb540aef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:47.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:47.583+0000 7fdb58c00640 1 -- 172.21.15.6:0/4160926168 >> 172.21.15.6:0/4160926168 conn(0x7fdb5406cf40 msgr2=0x7fdb5406d320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:47.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:47.584+0000 7fdb58c00640 1 -- 172.21.15.6:0/4160926168 shutdown_connections 2024-03-04T09:30:47.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:47.584+0000 7fdb58c00640 1 -- 172.21.15.6:0/4160926168 wait complete. 2024-03-04T09:30:47.601 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump 2024-03-04T09:30:47.862 INFO:teuthology.orchestra.run.smithi006.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":76,"msgr_send_messages":79,"msgr_recv_bytes":26951,"msgr_send_bytes":12642753,"msgr_created_connections":9,"msgr_active_connections":5,"msgr_running_total_time":0.102609054,"msgr_running_send_time":0.074341033,"msgr_running_recv_time":0.009213939,"msgr_running_fast_dispatch_time":0.014976120,"msgr_send_messages_queue_lat":{"avgcount":78,"sum":0.026950963,"avgtime":0.000345525},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000208,"avgtime":0.000000104},"msgr_recv_encrypted_bytes":1728,"msgr_send_encrypted_bytes":61248},"AsyncMessenger::Worker-1":{"msgr_recv_messages":8672,"msgr_send_messages":15809,"msgr_recv_bytes":3107396,"msgr_send_bytes":8835879,"msgr_created_connections":7,"msgr_active_connections":5,"msgr_running_total_time":1.982712409,"msgr_running_send_time":1.277557539,"msgr_running_recv_time":1.332730107,"msgr_running_fast_dispatch_time":0.002176974,"msgr_send_messages_queue_lat":{"avgcount":15807,"sum":0.812504138,"avgtime":0.000051401},"msgr_handle_ack_lat":{"avgcount":19700,"sum":0.009439098,"avgtime":0.000000479},"msgr_recv_encrypted_bytes":0,"msgr_send_encrypted_bytes":46336},"AsyncMessenger::Worker-2":{"msgr_recv_messages":51,"msgr_send_messages":47,"msgr_recv_bytes":118080,"msgr_send_bytes":4211549,"msgr_created_connections":5,"msgr_active_connections":4,"msgr_running_total_time":0.030873229,"msgr_running_send_time":0.019720616,"msgr_running_recv_time":0.010434835,"msgr_running_fast_dispatch_time":0.003906265,"msgr_send_messages_queue_lat":{"avgcount":46,"sum":0.012034187,"avgtime":0.000261612},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":106272,"msgr_send_encrypted_bytes":6672},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":1413,"complete_latency":{"avgcount":72,"sum":3.316724928,"avgtime":0.046065624}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":119,"sum":0.132642545,"avgtime":0.001114643}},"mds":{"request":4909,"reply":4909,"reply_latency":{"avgcount":4909,"sum":683.861862170,"avgtime":0.139307773},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":0,"dir_split":0,"dir_merge":0,"inodes":4910,"inodes_top":0,"inodes_bottom":4910,"inodes_pin_tail":0,"inodes_pinned":4910,"inodes_expired":0,"inodes_with_caps":3136,"caps":3136,"subtrees":2,"traverse":4911,"traverse_hit":4911,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":2894,"root_rbytes":22259712,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":1,"ceph_cap_op_grant":4116,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":3703,"handle_client_caps_dirty":2999,"handle_client_cap_release":1,"process_request_cap_release":1708},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":1634,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":1700,"strays_enqueued":66,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":8686,"evex":0,"evtrm":0,"ev":8691,"evexg":0,"evexd":0,"segadd":9,"segex":0,"segtrm":0,"seg":11,"segmjr":2,"segexg":0,"segexd":0,"expos":4194304,"wrpos":24172697,"rdpos":4195674,"jlat":{"avgcount":11,"sum":1.104145275,"avgtime":0.100376843},"replayed":5},"mds_mem":{"ino":3214,"ino+":3214,"ino-":0,"dir":13,"dir+":13,"dir-":0,"dn":4911,"dn+":4911,"dn-":0,"cap":3136,"cap+":3202,"cap-":66,"rss":107120,"heap":248164},"mds_server":{"dispatch_client_request":4910,"dispatch_server_request":0,"handle_client_request":4909,"handle_client_session":1,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":679.388313371,"avgtime":0.212308847},"req_getattr_latency":{"avgcount":2,"sum":0.665016344,"avgtime":0.332508172},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.005488280,"avgtime":0.005488280},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":6,"sum":1.043772762,"avgtime":0.173962127},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":1700,"sum":2.759271413,"avgtime":0.001623100},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":4617,"average_load":4617,"avg_session_uptime":931,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":17593477,"buffer_anon_items":8672,"buffer_meta_bytes":0,"buffer_meta_items":0,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5672,"osdmap_items":92,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":16252754,"mds_co_items":205751,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":144,"op_send_bytes":19863564,"op_resend":0,"op_reply":144,"op_latency":{"avgcount":144,"sum":0.734991224,"avgtime":0.005104105},"op_inflight":0,"oplen_avg":{"avgcount":144,"sum":198},"op":144,"op_r":37,"op_w":107,"op_rmw":0,"op_pg":0,"osdop_stat":2,"osdop_create":0,"osdop_read":9,"osdop_write":13,"osdop_writefull":3,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":81,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":0,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":15,"omap_rd":48,"omap_del":2},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":6,"pq_executing":0,"pq_executing_high_water":6,"pq_executed_ops":66,"pq_executed":66,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":254,"max":104857600,"get_started":0,"get":8795,"get_sum":2458256,"get_or_fail_fail":0,"get_or_fail_success":8795,"take":0,"take_sum":0,"put":8794,"put_sum":2458002,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":144,"take_sum":19872151,"put":144,"put_sum":19872151,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":144,"take_sum":144,"put":144,"put_sum":144,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":66,"get_sum":6666,"get_or_fail_fail":0,"get_or_fail_success":66,"take":0,"take_sum":0,"put":1,"put_sum":6666,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x5610c362caa0":{"val":436121,"max":3758096384,"get_started":0,"get":8686,"get_sum":19976208,"get_or_fail_fail":0,"get_or_fail_success":8686,"take":0,"take_sum":0,"put":12,"put_sum":19540087,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-03-04T09:30:47.863 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 5, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.04e-07, "sum": 2.08e-07 }, "msgr_recv_bytes": 26951, "msgr_recv_encrypted_bytes": 1728, "msgr_recv_messages": 76, "msgr_running_fast_dispatch_time": 0.01497612, "msgr_running_recv_time": 0.009213939, "msgr_running_send_time": 0.074341033, "msgr_running_total_time": 0.102609054, "msgr_send_bytes": 12642753, "msgr_send_encrypted_bytes": 61248, "msgr_send_messages": 79, "msgr_send_messages_queue_lat": { "avgcount": 78, "avgtime": 0.000345525, "sum": 0.026950963 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 5, "msgr_created_connections": 7, "msgr_handle_ack_lat": { "avgcount": 19700, "avgtime": 4.79e-07, "sum": 0.009439098 }, "msgr_recv_bytes": 3107396, "msgr_recv_encrypted_bytes": 0, "msgr_recv_messages": 8672, "msgr_running_fast_dispatch_time": 0.002176974, "msgr_running_recv_time": 1.332730107, "msgr_running_send_time": 1.277557539, "msgr_running_total_time": 1.982712409, "msgr_send_bytes": 8835879, "msgr_send_encrypted_bytes": 46336, "msgr_send_messages": 15809, "msgr_send_messages_queue_lat": { "avgcount": 15807, "avgtime": 5.1401e-05, "sum": 0.812504138 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 4, "msgr_created_connections": 5, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 118080, "msgr_recv_encrypted_bytes": 106272, "msgr_recv_messages": 51, "msgr_running_fast_dispatch_time": 0.003906265, "msgr_running_recv_time": 0.010434835, "msgr_running_send_time": 0.019720616, "msgr_running_total_time": 0.030873229, "msgr_send_bytes": 4211549, "msgr_send_encrypted_bytes": 6672, "msgr_send_messages": 47, "msgr_send_messages_queue_lat": { "avgcount": 46, "avgtime": 0.000261612, "sum": 0.012034187 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 72, "avgtime": 0.046065624, "sum": 3.316724928 }, "queue_len": 1413 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 119, "avgtime": 0.001114643, "sum": 0.132642545 }, "queue_len": 0 }, "mds": { "caps": 3136, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 4116, "ceph_cap_op_revoke": 1, "ceph_cap_op_trunc": 0, "dir_commit": 0, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 1, "handle_client_caps": 3703, "handle_client_caps_dirty": 2999, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 4910, "inodes_bottom": 4910, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 4910, "inodes_top": 0, "inodes_with_caps": 3136, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 1708, "q": 0, "reply": 4909, "reply_latency": { "avgcount": 4909, "avgtime": 0.139307773, "sum": 683.86186217 }, "request": 4909, "root_rbytes": 22259712, "root_rfiles": 2894, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 4911, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 4911, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 1634, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 1700, "strays_enqueued": 66, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 8691, "evadd": 8686, "evex": 0, "evexd": 0, "evexg": 0, "evlrg": 0, "evtrm": 0, "expos": 4194304, "jlat": { "avgcount": 11, "avgtime": 0.100376843, "sum": 1.104145275 }, "rdpos": 4195674, "replayed": 5, "seg": 11, "segadd": 9, "segex": 0, "segexd": 0, "segexg": 0, "segmjr": 2, "segtrm": 0, "wrpos": 24172697 }, "mds_mem": { "cap": 3136, "cap+": 3202, "cap-": 66, "dir": 13, "dir+": 13, "dir-": 0, "dn": 4911, "dn+": 4911, "dn-": 0, "heap": 248164, "ino": 3214, "ino+": 3214, "ino-": 0, "rss": 107120 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 4910, "dispatch_server_request": 0, "handle_client_request": 4909, "handle_client_session": 1, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.212308847, "sum": 679.388313371 }, "req_getattr_latency": { "avgcount": 2, "avgtime": 0.332508172, "sum": 0.665016344 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.00548828, "sum": 0.00548828 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 6, "avgtime": 0.173962127, "sum": 1.043772762 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 1700, "avgtime": 0.0016231, "sum": 2.759271413 } }, "mds_sessions": { "average_load": 4617, "avg_session_uptime": 931, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 4617 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 17593477, "buffer_anon_items": 8672, "buffer_meta_bytes": 0, "buffer_meta_items": 0, "mds_co_bytes": 16252754, "mds_co_items": 205751, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5672, "osdmap_items": 92, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 2, "omap_rd": 48, "omap_wr": 15, "op": 144, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 144, "avgtime": 0.005104105, "sum": 0.734991224 }, "op_pg": 0, "op_r": 37, "op_reply": 144, "op_resend": 0, "op_rmw": 0, "op_send": 144, "op_send_bytes": 19863564, "op_w": 107, "oplen_avg": { "avgcount": 144, "sum": 198 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 0, "osdop_delete": 81, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 9, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 0, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 2, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 13, "osdop_writefull": 3, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 66, "pq_executed_ops": 66, "pq_executing": 0, "pq_executing_high_water": 6, "pq_executing_ops": 0, "pq_executing_ops_high_water": 6, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 8795, "get_or_fail_fail": 0, "get_or_fail_success": 8795, "get_started": 0, "get_sum": 2458256, "max": 104857600, "put": 8794, "put_sum": 2458002, "take": 0, "take_sum": 0, "val": 254, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 144, "put_sum": 19872151, "take": 144, "take_sum": 19872151, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 144, "put_sum": 144, "take": 144, "take_sum": 144, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 66, "get_or_fail_fail": 0, "get_or_fail_success": 66, "get_started": 0, "get_sum": 6666, "max": 3758096384, "put": 1, "put_sum": 6666, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x5610c362caa0": { "get": 8686, "get_or_fail_fail": 0, "get_or_fail_success": 8686, "get_started": 0, "get_sum": 19976208, "max": 3758096384, "put": 12, "put_sum": 19540087, "take": 0, "take_sum": 0, "val": 436121, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-03-04T09:30:47.863 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/1634, 66/1700 2024-03-04T09:30:48.864 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:30:49.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:49.156+0000 7fed91200640 1 Processor -- start 2024-03-04T09:30:49.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:49.157+0000 7fed91200640 1 -- start start 2024-03-04T09:30:49.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:49.159+0000 7fed91200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fed8c14b4a0 0x7fed8c0b4a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:49.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:49.159+0000 7fed91200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fed8c0cd3b0 0x7fed8c0b4f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:49.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:49.159+0000 7fed91200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed8c0b3b60 con 0x7fed8c0b2e50 2024-03-04T09:30:49.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:49.159+0000 7fed91200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed8c06e8f0 con 0x7fed8c0cd3b0 2024-03-04T09:30:49.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:49.159+0000 7fed91200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed8c06d450 con 0x7fed8c14b4a0 2024-03-04T09:30:49.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:49.160+0000 7fed89400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fed8c0cd3b0 0x7fed8c0b4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:49.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:49.160+0000 7fed8a800640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fed8c0b2e50 0x7fed8c0b32a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.6:57484/0 (socket says 172.21.15.6:57484) 2024-03-04T09:30:49.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:49.160+0000 7fed8a800640 1 -- 172.21.15.6:0/436446611 learned_addr learned my addr 172.21.15.6:0/436446611 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:30:49.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:49.161+0000 7fed89e00640 1 --2- 172.21.15.6:0/436446611 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fed8c14b4a0 0x7fed8c0b4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:49.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:49.163+0000 7fed89e00640 1 -- 172.21.15.6:0/436446611 >> v2:172.21.15.146:3300/0 conn(0x7fed8c0cd3b0 msgr2=0x7fed8c0b4f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:49.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:49.163+0000 7fed89e00640 1 --2- 172.21.15.6:0/436446611 >> v2:172.21.15.146:3300/0 conn(0x7fed8c0cd3b0 0x7fed8c0b4f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:49.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:49.163+0000 7fed88a00640 1 -- 172.21.15.6:0/436446611 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4060934675 0 0) 0x7fed8c0b3b60 con 0x7fed8c0b2e50 2024-03-04T09:30:49.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:49.164+0000 7fed89e00640 1 -- 172.21.15.6:0/436446611 >> v1:172.21.15.146:6789/0 conn(0x7fed8c0b2e50 legacy=0x7fed8c0b32a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:49.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:49.164+0000 7fed89e00640 1 -- 172.21.15.6:0/436446611 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed8c0b3e40 con 0x7fed8c14b4a0 2024-03-04T09:30:49.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:49.170+0000 7fed89e00640 1 --2- 172.21.15.6:0/436446611 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fed8c14b4a0 0x7fed8c0b4a10 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fed7c0097a0 tx=0x7fed7c02b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f98e7d270d3ef464 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:49.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:49.171+0000 7fed88a00640 1 -- 172.21.15.6:0/436446611 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed7c035020 con 0x7fed8c14b4a0 2024-03-04T09:30:49.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:49.171+0000 7fed88a00640 1 -- 172.21.15.6:0/436446611 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed7c031830 con 0x7fed8c14b4a0 2024-03-04T09:30:49.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:49.171+0000 7fed88a00640 1 -- 172.21.15.6:0/436446611 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed7c031c00 con 0x7fed8c14b4a0 2024-03-04T09:30:49.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:49.172+0000 7fed91200640 1 -- 172.21.15.6:0/436446611 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fed8c14b4a0 msgr2=0x7fed8c0b4a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:49.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:49.172+0000 7fed91200640 1 --2- 172.21.15.6:0/436446611 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fed8c14b4a0 0x7fed8c0b4a10 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fed7c0097a0 tx=0x7fed7c02b8d0 comp rx=0 tx=0).stop 2024-03-04T09:30:49.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:49.173+0000 7fed91200640 1 -- 172.21.15.6:0/436446611 shutdown_connections 2024-03-04T09:30:49.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:49.173+0000 7fed91200640 1 --2- 172.21.15.6:0/436446611 >> v2:172.21.15.146:3300/0 conn(0x7fed8c0cd3b0 0x7fed8c0b4f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:49.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:49.173+0000 7fed91200640 1 --2- 172.21.15.6:0/436446611 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fed8c14b4a0 0x7fed8c0b4a10 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:49.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:49.173+0000 7fed91200640 1 -- 172.21.15.6:0/436446611 >> 172.21.15.6:0/436446611 conn(0x7fed8c0afac0 msgr2=0x7fed8c0afea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:49.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:49.177+0000 7fed91200640 1 -- 172.21.15.6:0/436446611 shutdown_connections 2024-03-04T09:30:49.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:49.177+0000 7fed91200640 1 -- 172.21.15.6:0/436446611 wait complete. 2024-03-04T09:30:49.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:49.185+0000 7fed91200640 1 Processor -- start 2024-03-04T09:30:49.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:49.186+0000 7fed91200640 1 -- start start 2024-03-04T09:30:49.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:49.195+0000 7fed91200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fed8c0b2e50 0x7fed8c1f8940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:49.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:49.195+0000 7fed91200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fed8c0cd3b0 0x7fed8c0aab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:49.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:49.197+0000 7fed91200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fed8c14b4a0 0x7fed8c0ab510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:49.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:49.197+0000 7fed91200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed8c06da60 con 0x7fed8c0cd3b0 2024-03-04T09:30:49.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:49.197+0000 7fed91200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed8c06d8e0 con 0x7fed8c0b2e50 2024-03-04T09:30:49.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:49.198+0000 7fed89400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fed8c0cd3b0 0x7fed8c0aab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:49.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:49.198+0000 7fed89400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fed8c0cd3b0 0x7fed8c0aab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41430/0 (socket says 172.21.15.6:41430) 2024-03-04T09:30:49.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:49.198+0000 7fed89400640 1 -- 172.21.15.6:0/746376686 learned_addr learned my addr 172.21.15.6:0/746376686 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:30:49.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:49.198+0000 7fed89e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fed8c0b2e50 0x7fed8c1f8940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:49.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:49.198+0000 7fed91200640 1 -- 172.21.15.6:0/746376686 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed8c06dbe0 con 0x7fed8c14b4a0 2024-03-04T09:30:49.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:49.198+0000 7fed8a800640 1 --2- 172.21.15.6:0/746376686 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fed8c14b4a0 0x7fed8c0ab510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:49.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:49.202+0000 7fed89e00640 1 -- 172.21.15.6:0/746376686 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fed8c14b4a0 msgr2=0x7fed8c0ab510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:49.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:49.202+0000 7fed89e00640 1 --2- 172.21.15.6:0/746376686 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fed8c14b4a0 0x7fed8c0ab510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:49.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:49.202+0000 7fed89e00640 1 -- 172.21.15.6:0/746376686 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fed8c0cd3b0 msgr2=0x7fed8c0aab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:49.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:49.202+0000 7fed89e00640 1 --2- 172.21.15.6:0/746376686 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fed8c0cd3b0 0x7fed8c0aab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:49.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:49.202+0000 7fed89e00640 1 -- 172.21.15.6:0/746376686 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed8c0adb50 con 0x7fed8c0b2e50 2024-03-04T09:30:49.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:49.203+0000 7fed89400640 1 --2- 172.21.15.6:0/746376686 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fed8c0cd3b0 0x7fed8c0aab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T09:30:49.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:49.203+0000 7fed89e00640 1 --2- 172.21.15.6:0/746376686 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fed8c0b2e50 0x7fed8c1f8940 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fed7c033000 tx=0x7fed7c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:49.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:49.207+0000 7fed82000640 1 -- 172.21.15.6:0/746376686 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed7c035040 con 0x7fed8c0b2e50 2024-03-04T09:30:49.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:49.207+0000 7fed91200640 1 -- 172.21.15.6:0/746376686 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed8c0ade30 con 0x7fed8c0b2e50 2024-03-04T09:30:49.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:49.207+0000 7fed91200640 1 -- 172.21.15.6:0/746376686 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed8c0ae3f0 con 0x7fed8c0b2e50 2024-03-04T09:30:49.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:49.208+0000 7fed82000640 1 -- 172.21.15.6:0/746376686 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed7c03b840 con 0x7fed8c0b2e50 2024-03-04T09:30:49.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:49.208+0000 7fed82000640 1 -- 172.21.15.6:0/746376686 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed7c035040 con 0x7fed8c0b2e50 2024-03-04T09:30:49.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:49.212+0000 7fed7b400640 1 -- 172.21.15.6:0/746376686 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed5c005290 con 0x7fed8c0b2e50 2024-03-04T09:30:49.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:49.220+0000 7fed82000640 1 -- 172.21.15.6:0/746376686 <== mon.1 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fed7c038020 con 0x7fed8c0b2e50 2024-03-04T09:30:49.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:49.228+0000 7fed82000640 1 --2- 172.21.15.6:0/746376686 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fed6806b6d0 0x7fed6806db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:49.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:49.229+0000 7fed82000640 1 -- 172.21.15.6:0/746376686 <== mon.1 v2:172.21.15.146:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5881+0+0 (secure 0 0 0) 0x7fed7c0abdd0 con 0x7fed8c0b2e50 2024-03-04T09:30:49.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:49.230+0000 7fed89400640 1 --2- 172.21.15.6:0/746376686 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fed6806b6d0 0x7fed6806db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:49.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:49.232+0000 7fed89400640 1 --2- 172.21.15.6:0/746376686 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fed6806b6d0 0x7fed6806db90 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fed8c0b3880 tx=0x7fed74009190 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:49.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:49.234+0000 7fed82000640 1 -- 172.21.15.6:0/746376686 <== mon.1 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fed7c031800 con 0x7fed8c0b2e50 2024-03-04T09:30:49.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:49.765+0000 7fed7b400640 1 -- 172.21.15.6:0/746376686 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fed5c005850 con 0x7fed8c0b2e50 2024-03-04T09:30:49.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:49.766+0000 7fed82000640 1 -- 172.21.15.6:0/746376686 <== mon.1 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) v1 ==== 94+0+4977 (secure 0 0 0) 0x7fed7c07a560 con 0x7fed8c0b2e50 2024-03-04T09:30:49.763 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:30:49.763 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":68,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5654,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/1370302030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":1370302030},{"type":"v1","addr":"172.21.15.146:6835","nonce":1370302030}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":6163,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/809822004","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":809822004},{"type":"v1","addr":"172.21.15.146:6837","nonce":809822004}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":66},{"gid":7284,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/1583413840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1583413840},{"type":"v1","addr":"172.21.15.6:6833","nonce":1583413840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64}],"filesystems":[{"mdsmap":{"epoch":68,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:30:20.660103+0000","modified":"2024-03-04T09:30:36.289706+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":98,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7296},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7296":{"gid":7296,"name":"a","rank":0,"incarnation":65,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1732383836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1732383836},{"type":"v1","addr":"172.21.15.6:6835","nonce":1732383836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7296,"qdb_cluster":[7296]},"id":7}]} 2024-03-04T09:30:49.764 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 68 2024-03-04T09:30:49.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:49.769+0000 7fed7b400640 1 -- 172.21.15.6:0/746376686 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fed6806b6d0 msgr2=0x7fed6806db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:49.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:49.769+0000 7fed7b400640 1 --2- 172.21.15.6:0/746376686 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fed6806b6d0 0x7fed6806db90 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fed8c0b3880 tx=0x7fed74009190 comp rx=0 tx=0).stop 2024-03-04T09:30:49.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:49.769+0000 7fed7b400640 1 -- 172.21.15.6:0/746376686 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fed8c0b2e50 msgr2=0x7fed8c1f8940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:49.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:49.769+0000 7fed7b400640 1 --2- 172.21.15.6:0/746376686 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fed8c0b2e50 0x7fed8c1f8940 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fed7c033000 tx=0x7fed7c0046c0 comp rx=0 tx=0).stop 2024-03-04T09:30:49.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:49.769+0000 7fed7b400640 1 -- 172.21.15.6:0/746376686 shutdown_connections 2024-03-04T09:30:49.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:49.769+0000 7fed7b400640 1 --2- 172.21.15.6:0/746376686 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fed6806b6d0 0x7fed6806db90 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:49.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:49.769+0000 7fed7b400640 1 --2- 172.21.15.6:0/746376686 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fed8c14b4a0 0x7fed8c0ab510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:49.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:49.769+0000 7fed7b400640 1 --2- 172.21.15.6:0/746376686 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fed8c0cd3b0 0x7fed8c0aab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:49.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:49.769+0000 7fed7b400640 1 --2- 172.21.15.6:0/746376686 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fed8c0b2e50 0x7fed8c1f8940 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:49.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:49.769+0000 7fed7b400640 1 -- 172.21.15.6:0/746376686 >> 172.21.15.6:0/746376686 conn(0x7fed8c0afac0 msgr2=0x7fed8c164aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:49.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:49.769+0000 7fed7b400640 1 -- 172.21.15.6:0/746376686 shutdown_connections 2024-03-04T09:30:49.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:49.770+0000 7fed7b400640 1 -- 172.21.15.6:0/746376686 wait complete. 2024-03-04T09:30:49.789 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump 2024-03-04T09:30:50.015 INFO:teuthology.orchestra.run.smithi006.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":126,"msgr_send_messages":130,"msgr_recv_bytes":39723,"msgr_send_bytes":14150101,"msgr_created_connections":9,"msgr_active_connections":5,"msgr_running_total_time":0.120609173,"msgr_running_send_time":0.085732806,"msgr_running_recv_time":0.010694107,"msgr_running_fast_dispatch_time":0.019588359,"msgr_send_messages_queue_lat":{"avgcount":129,"sum":0.031536363,"avgtime":0.000244467},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000208,"avgtime":0.000000104},"msgr_recv_encrypted_bytes":1728,"msgr_send_encrypted_bytes":63040},"AsyncMessenger::Worker-1":{"msgr_recv_messages":11181,"msgr_send_messages":18389,"msgr_recv_bytes":4016268,"msgr_send_bytes":9608818,"msgr_created_connections":7,"msgr_active_connections":5,"msgr_running_total_time":2.305880941,"msgr_running_send_time":1.488594357,"msgr_running_recv_time":2.521821423,"msgr_running_fast_dispatch_time":0.004325028,"msgr_send_messages_queue_lat":{"avgcount":18387,"sum":1.223644355,"avgtime":0.000066549},"msgr_handle_ack_lat":{"avgcount":23550,"sum":0.010521081,"avgtime":0.000000446},"msgr_recv_encrypted_bytes":0,"msgr_send_encrypted_bytes":46336},"AsyncMessenger::Worker-2":{"msgr_recv_messages":107,"msgr_send_messages":103,"msgr_recv_bytes":132296,"msgr_send_bytes":4229797,"msgr_created_connections":5,"msgr_active_connections":4,"msgr_running_total_time":0.037073213,"msgr_running_send_time":0.022358535,"msgr_running_recv_time":0.011962124,"msgr_running_fast_dispatch_time":0.005444152,"msgr_send_messages_queue_lat":{"avgcount":102,"sum":0.015583362,"avgtime":0.000152778},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":106272,"msgr_send_encrypted_bytes":6672},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":935,"complete_latency":{"avgcount":74,"sum":6.209740326,"avgtime":0.083915409}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":433,"sum":0.606175883,"avgtime":0.001399944}},"mds":{"request":5306,"reply":5306,"reply_latency":{"avgcount":5306,"sum":685.670882483,"avgtime":0.129225571},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":0,"dir_split":0,"dir_merge":0,"inodes":5251,"inodes_top":0,"inodes_bottom":5251,"inodes_pin_tail":0,"inodes_pinned":5251,"inodes_expired":0,"inodes_with_caps":2719,"caps":2719,"subtrees":2,"traverse":5308,"traverse_hit":5308,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":1857,"root_rbytes":14353408,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":9,"ceph_cap_op_grant":5220,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":4313,"handle_client_caps_dirty":2999,"handle_client_cap_release":4,"process_request_cap_release":2105},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":1861,"num_strays_delayed":183,"num_strays_enqueuing":0,"strays_created":2097,"strays_enqueued":236,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":9877,"evex":0,"evtrm":0,"ev":9882,"evexg":0,"evexd":0,"segadd":10,"segex":0,"segtrm":0,"seg":12,"segmjr":2,"segexg":0,"segexd":0,"expos":4194304,"wrpos":26805728,"rdpos":4195674,"jlat":{"avgcount":12,"sum":2.238811436,"avgtime":0.186567619},"replayed":5},"mds_mem":{"ino":3157,"ino+":3214,"ino-":57,"dir":13,"dir+":13,"dir-":0,"dn":5251,"dn+":5308,"dn-":57,"cap":2719,"cap+":3202,"cap-":483,"rss":107120,"heap":248164},"mds_server":{"dispatch_client_request":5307,"dispatch_server_request":0,"handle_client_request":5306,"handle_client_session":1,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":679.388313371,"avgtime":0.212308847},"req_getattr_latency":{"avgcount":2,"sum":0.665016344,"avgtime":0.332508172},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.005488280,"avgtime":0.005488280},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":6,"sum":1.043772762,"avgtime":0.173962127},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":2097,"sum":4.568291726,"avgtime":0.002178489},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":4901,"average_load":4901,"avg_session_uptime":931,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":17593477,"buffer_anon_items":8672,"buffer_meta_bytes":0,"buffer_meta_items":0,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5672,"osdmap_items":92,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":16252754,"mds_co_items":205751,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":324,"op_send_bytes":21370016,"op_resend":0,"op_reply":324,"op_latency":{"avgcount":324,"sum":1.388983412,"avgtime":0.004286985},"op_inflight":0,"oplen_avg":{"avgcount":324,"sum":380},"op":324,"op_r":40,"op_w":284,"op_rmw":0,"op_pg":0,"osdop_stat":2,"osdop_create":0,"osdop_read":12,"osdop_write":17,"osdop_writefull":4,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":252,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":0,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":17,"omap_rd":48,"omap_del":3},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":6,"pq_executing":0,"pq_executing_high_water":6,"pq_executed_ops":236,"pq_executed":236,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":379172,"max":104857600,"get_started":0,"get":11410,"get_sum":3158766,"get_or_fail_fail":0,"get_or_fail_success":11410,"take":0,"take_sum":0,"put":9987,"put_sum":2779594,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":324,"take_sum":21395773,"put":324,"put_sum":21395773,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":324,"take_sum":324,"put":324,"put_sum":324,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":236,"get_sum":23836,"get_or_fail_fail":0,"get_or_fail_success":236,"take":0,"take_sum":0,"put":4,"put_sum":23836,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x5610c362caa0":{"val":1639904,"max":3758096384,"get_started":0,"get":9877,"get_sum":22609239,"get_or_fail_fail":0,"get_or_fail_success":9877,"take":0,"take_sum":0,"put":13,"put_sum":20969335,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-03-04T09:30:50.017 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 5, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.04e-07, "sum": 2.08e-07 }, "msgr_recv_bytes": 39723, "msgr_recv_encrypted_bytes": 1728, "msgr_recv_messages": 126, "msgr_running_fast_dispatch_time": 0.019588359, "msgr_running_recv_time": 0.010694107, "msgr_running_send_time": 0.085732806, "msgr_running_total_time": 0.120609173, "msgr_send_bytes": 14150101, "msgr_send_encrypted_bytes": 63040, "msgr_send_messages": 130, "msgr_send_messages_queue_lat": { "avgcount": 129, "avgtime": 0.000244467, "sum": 0.031536363 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 5, "msgr_created_connections": 7, "msgr_handle_ack_lat": { "avgcount": 23550, "avgtime": 4.46e-07, "sum": 0.010521081 }, "msgr_recv_bytes": 4016268, "msgr_recv_encrypted_bytes": 0, "msgr_recv_messages": 11181, "msgr_running_fast_dispatch_time": 0.004325028, "msgr_running_recv_time": 2.521821423, "msgr_running_send_time": 1.488594357, "msgr_running_total_time": 2.305880941, "msgr_send_bytes": 9608818, "msgr_send_encrypted_bytes": 46336, "msgr_send_messages": 18389, "msgr_send_messages_queue_lat": { "avgcount": 18387, "avgtime": 6.6549e-05, "sum": 1.223644355 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 4, "msgr_created_connections": 5, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 132296, "msgr_recv_encrypted_bytes": 106272, "msgr_recv_messages": 107, "msgr_running_fast_dispatch_time": 0.005444152, "msgr_running_recv_time": 0.011962124, "msgr_running_send_time": 0.022358535, "msgr_running_total_time": 0.037073213, "msgr_send_bytes": 4229797, "msgr_send_encrypted_bytes": 6672, "msgr_send_messages": 103, "msgr_send_messages_queue_lat": { "avgcount": 102, "avgtime": 0.000152778, "sum": 0.015583362 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 74, "avgtime": 0.083915409, "sum": 6.209740326 }, "queue_len": 935 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 433, "avgtime": 0.001399944, "sum": 0.606175883 }, "queue_len": 0 }, "mds": { "caps": 2719, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 5220, "ceph_cap_op_revoke": 9, "ceph_cap_op_trunc": 0, "dir_commit": 0, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 4, "handle_client_caps": 4313, "handle_client_caps_dirty": 2999, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 5251, "inodes_bottom": 5251, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 5251, "inodes_top": 0, "inodes_with_caps": 2719, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 2105, "q": 0, "reply": 5306, "reply_latency": { "avgcount": 5306, "avgtime": 0.129225571, "sum": 685.670882483 }, "request": 5306, "root_rbytes": 14353408, "root_rfiles": 1857, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 5308, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 5308, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 1861, "num_strays_delayed": 183, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 2097, "strays_enqueued": 236, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 9882, "evadd": 9877, "evex": 0, "evexd": 0, "evexg": 0, "evlrg": 0, "evtrm": 0, "expos": 4194304, "jlat": { "avgcount": 12, "avgtime": 0.186567619, "sum": 2.238811436 }, "rdpos": 4195674, "replayed": 5, "seg": 12, "segadd": 10, "segex": 0, "segexd": 0, "segexg": 0, "segmjr": 2, "segtrm": 0, "wrpos": 26805728 }, "mds_mem": { "cap": 2719, "cap+": 3202, "cap-": 483, "dir": 13, "dir+": 13, "dir-": 0, "dn": 5251, "dn+": 5308, "dn-": 57, "heap": 248164, "ino": 3157, "ino+": 3214, "ino-": 57, "rss": 107120 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 5307, "dispatch_server_request": 0, "handle_client_request": 5306, "handle_client_session": 1, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.212308847, "sum": 679.388313371 }, "req_getattr_latency": { "avgcount": 2, "avgtime": 0.332508172, "sum": 0.665016344 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.00548828, "sum": 0.00548828 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 6, "avgtime": 0.173962127, "sum": 1.043772762 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 2097, "avgtime": 0.002178489, "sum": 4.568291726 } }, "mds_sessions": { "average_load": 4901, "avg_session_uptime": 931, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 4901 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 17593477, "buffer_anon_items": 8672, "buffer_meta_bytes": 0, "buffer_meta_items": 0, "mds_co_bytes": 16252754, "mds_co_items": 205751, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5672, "osdmap_items": 92, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 3, "omap_rd": 48, "omap_wr": 17, "op": 324, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 324, "avgtime": 0.004286985, "sum": 1.388983412 }, "op_pg": 0, "op_r": 40, "op_reply": 324, "op_resend": 0, "op_rmw": 0, "op_send": 324, "op_send_bytes": 21370016, "op_w": 284, "oplen_avg": { "avgcount": 324, "sum": 380 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 0, "osdop_delete": 252, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 12, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 0, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 2, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 17, "osdop_writefull": 4, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 236, "pq_executed_ops": 236, "pq_executing": 0, "pq_executing_high_water": 6, "pq_executing_ops": 0, "pq_executing_ops_high_water": 6, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 11410, "get_or_fail_fail": 0, "get_or_fail_success": 11410, "get_started": 0, "get_sum": 3158766, "max": 104857600, "put": 9987, "put_sum": 2779594, "take": 0, "take_sum": 0, "val": 379172, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 324, "put_sum": 21395773, "take": 324, "take_sum": 21395773, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 324, "put_sum": 324, "take": 324, "take_sum": 324, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 236, "get_or_fail_fail": 0, "get_or_fail_success": 236, "get_started": 0, "get_sum": 23836, "max": 3758096384, "put": 4, "put_sum": 23836, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x5610c362caa0": { "get": 9877, "get_or_fail_fail": 0, "get_or_fail_success": 9877, "get_started": 0, "get_sum": 22609239, "max": 3758096384, "put": 13, "put_sum": 20969335, "take": 0, "take_sum": 0, "val": 1639904, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-03-04T09:30:50.018 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/1861, 236/2097 2024-03-04T09:30:51.019 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:30:51.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:51.250+0000 7f656b600640 1 Processor -- start 2024-03-04T09:30:51.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:51.250+0000 7f656b600640 1 -- start start 2024-03-04T09:30:51.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:51.252+0000 7f656b600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6564147770 0x7f6564071130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:51.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:51.252+0000 7f656b600640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65640717f0 con 0x7f6564147b50 2024-03-04T09:30:51.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:51.252+0000 7f656b600640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6564071ad0 con 0x7f6564162350 2024-03-04T09:30:51.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:51.252+0000 7f656b600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6564072400 con 0x7f6564147770 2024-03-04T09:30:51.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:51.253+0000 7f6563e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f6564147b50 0x7f6564161c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:43120/0 (socket says 172.21.15.6:43120) 2024-03-04T09:30:51.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:51.253+0000 7f6563e00640 1 -- 172.21.15.6:0/3101091633 learned_addr learned my addr 172.21.15.6:0/3101091633 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:30:51.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:51.253+0000 7f6568c00640 1 --2- 172.21.15.6:0/3101091633 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6564147770 0x7f6564071130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:51.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:51.253+0000 7f6563400640 1 -- 172.21.15.6:0/3101091633 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3539204324 0 0) 0x7f65640717f0 con 0x7f6564147b50 2024-03-04T09:30:51.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:51.253+0000 7f6563400640 1 -- 172.21.15.6:0/3101091633 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f654c003520 con 0x7f6564147b50 2024-03-04T09:30:51.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:51.254+0000 7f6563400640 1 -- 172.21.15.6:0/3101091633 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3005194312 0 0) 0x7f654c003520 con 0x7f6564147b50 2024-03-04T09:30:51.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:51.254+0000 7f6563400640 1 -- 172.21.15.6:0/3101091633 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6564147770 msgr2=0x7f6564071130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:51.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:51.254+0000 7f6563400640 1 --2- 172.21.15.6:0/3101091633 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6564147770 0x7f6564071130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:51.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:51.254+0000 7f6563400640 1 -- 172.21.15.6:0/3101091633 >> v1:172.21.15.146:6789/0 conn(0x7f6564162350 legacy=0x7f65640ad8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:51.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:51.254+0000 7f6563400640 1 -- 172.21.15.6:0/3101091633 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6564071db0 con 0x7f6564147b50 2024-03-04T09:30:51.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:51.254+0000 7f6563400640 1 -- 172.21.15.6:0/3101091633 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7f6550001670 con 0x7f6564147b50 2024-03-04T09:30:51.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:51.254+0000 7f6563400640 1 -- 172.21.15.6:0/3101091633 >> v1:172.21.15.6:6789/0 conn(0x7f6564147b50 legacy=0x7f6564161c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:51.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:51.254+0000 7f6563400640 1 --2- 172.21.15.6:0/3101091633 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6564147770 0x7f654c0039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:51.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:51.254+0000 7f6563400640 1 --2- 172.21.15.6:0/3101091633 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f654c003f60 0x7f654c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:51.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:51.254+0000 7f6563400640 1 --2- 172.21.15.6:0/3101091633 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f654c004d40 0x7f654c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:51.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:51.254+0000 7f6563400640 1 -- 172.21.15.6:0/3101091633 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f654c00ba70 con 0x7f654c004d40 2024-03-04T09:30:51.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:51.254+0000 7f6563400640 1 -- 172.21.15.6:0/3101091633 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f654c00bc20 con 0x7f654c003f60 2024-03-04T09:30:51.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:51.254+0000 7f6563400640 1 -- 172.21.15.6:0/3101091633 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f654c00be20 con 0x7f6564147770 2024-03-04T09:30:51.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:51.254+0000 7f6569600640 1 --2- 172.21.15.6:0/3101091633 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f654c004d40 0x7f654c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:51.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:51.254+0000 7f6563e00640 1 --2- 172.21.15.6:0/3101091633 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f654c003f60 0x7f654c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:51.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:51.254+0000 7f6568c00640 1 --2- 172.21.15.6:0/3101091633 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6564147770 0x7f654c0039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:51.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:51.255+0000 7f6569600640 1 -- 172.21.15.6:0/3101091633 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6564147770 msgr2=0x7f654c0039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:51.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:51.255+0000 7f6569600640 1 --2- 172.21.15.6:0/3101091633 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6564147770 0x7f654c0039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:51.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:51.255+0000 7f6569600640 1 -- 172.21.15.6:0/3101091633 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f654c003f60 msgr2=0x7f654c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:51.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:51.255+0000 7f6569600640 1 --2- 172.21.15.6:0/3101091633 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f654c003f60 0x7f654c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:51.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:51.255+0000 7f6569600640 1 -- 172.21.15.6:0/3101091633 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6564071db0 con 0x7f654c004d40 2024-03-04T09:30:51.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:51.256+0000 7f6569600640 1 --2- 172.21.15.6:0/3101091633 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f654c004d40 0x7f654c0051c0 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f655c001e50 tx=0x7f654c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a838437f6ad07b19 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:51.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:51.256+0000 7f6563400640 1 -- 172.21.15.6:0/3101091633 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f655c001a70 con 0x7f654c004d40 2024-03-04T09:30:51.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:51.256+0000 7f6563400640 1 -- 172.21.15.6:0/3101091633 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f655c003b70 con 0x7f654c004d40 2024-03-04T09:30:51.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:51.256+0000 7f6563400640 1 -- 172.21.15.6:0/3101091633 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f655c0050d0 con 0x7f654c004d40 2024-03-04T09:30:51.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:51.257+0000 7f656b600640 1 -- 172.21.15.6:0/3101091633 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f654c004d40 msgr2=0x7f654c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:51.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:51.257+0000 7f656b600640 1 --2- 172.21.15.6:0/3101091633 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f654c004d40 0x7f654c0051c0 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f655c001e50 tx=0x7f654c005830 comp rx=0 tx=0).stop 2024-03-04T09:30:51.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:51.257+0000 7f6569600640 1 -- 172.21.15.6:0/3101091633 reap_dead start 2024-03-04T09:30:51.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:51.257+0000 7f656b600640 1 -- 172.21.15.6:0/3101091633 shutdown_connections 2024-03-04T09:30:51.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:51.257+0000 7f656b600640 1 -- 172.21.15.6:0/3101091633 >> 172.21.15.6:0/3101091633 conn(0x7f656406cf40 msgr2=0x7f656406d320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:51.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:51.257+0000 7f656b600640 1 -- 172.21.15.6:0/3101091633 shutdown_connections 2024-03-04T09:30:51.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:51.257+0000 7f656b600640 1 -- 172.21.15.6:0/3101091633 wait complete. 2024-03-04T09:30:51.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:51.259+0000 7f656b600640 1 Processor -- start 2024-03-04T09:30:51.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:51.260+0000 7f656b600640 1 -- start start 2024-03-04T09:30:51.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:51.261+0000 7f656b600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f654c004d40 0x7f656414f930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:51.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:51.261+0000 7f656b600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f656414fe70 0x7f656415f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:51.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:51.261+0000 7f656b600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f656415fc60 0x7f656415d280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:51.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:51.261+0000 7f656b600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6564162980 con 0x7f656415fc60 2024-03-04T09:30:51.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:51.261+0000 7f656b600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6564162800 con 0x7f654c004d40 2024-03-04T09:30:51.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:51.261+0000 7f656b600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6564162b00 con 0x7f656414fe70 2024-03-04T09:30:51.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:51.262+0000 7f6569600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f656415fc60 0x7f656415d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:51.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:51.262+0000 7f6569600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f656415fc60 0x7f656415d280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41448/0 (socket says 172.21.15.6:41448) 2024-03-04T09:30:51.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:51.262+0000 7f6569600640 1 -- 172.21.15.6:0/1454062974 learned_addr learned my addr 172.21.15.6:0/1454062974 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:30:51.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:51.262+0000 7f6568c00640 1 --2- 172.21.15.6:0/1454062974 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f654c004d40 0x7f656414f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:51.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:51.262+0000 7f6563e00640 1 --2- 172.21.15.6:0/1454062974 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f656414fe70 0x7f656415f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:51.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:51.262+0000 7f6569600640 1 -- 172.21.15.6:0/1454062974 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f656414fe70 msgr2=0x7f656415f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:51.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:51.262+0000 7f6569600640 1 --2- 172.21.15.6:0/1454062974 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f656414fe70 0x7f656415f720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:51.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:51.262+0000 7f6569600640 1 -- 172.21.15.6:0/1454062974 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f654c004d40 msgr2=0x7f656414f930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:51.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:51.262+0000 7f6569600640 1 --2- 172.21.15.6:0/1454062974 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f654c004d40 0x7f656414f930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:51.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:51.262+0000 7f6569600640 1 -- 172.21.15.6:0/1454062974 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65641603e0 con 0x7f656415fc60 2024-03-04T09:30:51.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:51.262+0000 7f6568c00640 1 --2- 172.21.15.6:0/1454062974 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f654c004d40 0x7f656414f930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T09:30:51.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:51.263+0000 7f6569600640 1 --2- 172.21.15.6:0/1454062974 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f656415fc60 0x7f656415d280 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f655c02c060 tx=0x7f655c0014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:51.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:51.263+0000 7f6560c00640 1 -- 172.21.15.6:0/1454062974 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f655c007db0 con 0x7f656415fc60 2024-03-04T09:30:51.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:51.263+0000 7f6560c00640 1 -- 172.21.15.6:0/1454062974 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f655c003970 con 0x7f656415fc60 2024-03-04T09:30:51.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:51.263+0000 7f656b600640 1 -- 172.21.15.6:0/1454062974 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65641606c0 con 0x7f656415fc60 2024-03-04T09:30:51.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:51.263+0000 7f656b600640 1 -- 172.21.15.6:0/1454062974 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6564160c80 con 0x7f656415fc60 2024-03-04T09:30:51.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:51.264+0000 7f6560c00640 1 -- 172.21.15.6:0/1454062974 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f655c006d70 con 0x7f656415fc60 2024-03-04T09:30:51.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:51.265+0000 7f656b600640 1 -- 172.21.15.6:0/1454062974 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6540005290 con 0x7f656415fc60 2024-03-04T09:30:51.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:51.267+0000 7f6560c00640 1 -- 172.21.15.6:0/1454062974 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f655c007030 con 0x7f656415fc60 2024-03-04T09:30:51.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:51.267+0000 7f6560c00640 1 --2- 172.21.15.6:0/1454062974 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f654406b520 0x7f654406d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:51.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:51.268+0000 7f6560c00640 1 -- 172.21.15.6:0/1454062974 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f655c0ab410 con 0x7f656415fc60 2024-03-04T09:30:51.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:51.268+0000 7f6568c00640 1 --2- 172.21.15.6:0/1454062974 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f654406b520 0x7f654406d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:51.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:51.269+0000 7f6568c00640 1 --2- 172.21.15.6:0/1454062974 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f654406b520 0x7f654406d9e0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f656414d770 tx=0x7f6558008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:51.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:51.273+0000 7f6560c00640 1 -- 172.21.15.6:0/1454062974 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f655c033020 con 0x7f656415fc60 2024-03-04T09:30:51.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:51.637+0000 7f656b600640 1 -- 172.21.15.6:0/1454062974 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6540005580 con 0x7f656415fc60 2024-03-04T09:30:51.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:51.637+0000 7f6560c00640 1 -- 172.21.15.6:0/1454062974 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) v1 ==== 94+0+4977 (secure 0 0 0) 0x7f655c003b40 con 0x7f656415fc60 2024-03-04T09:30:51.634 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:30:51.634 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":68,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5654,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/1370302030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":1370302030},{"type":"v1","addr":"172.21.15.146:6835","nonce":1370302030}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":6163,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/809822004","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":809822004},{"type":"v1","addr":"172.21.15.146:6837","nonce":809822004}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":66},{"gid":7284,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/1583413840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1583413840},{"type":"v1","addr":"172.21.15.6:6833","nonce":1583413840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64}],"filesystems":[{"mdsmap":{"epoch":68,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:30:20.660103+0000","modified":"2024-03-04T09:30:36.289706+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":98,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7296},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7296":{"gid":7296,"name":"a","rank":0,"incarnation":65,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1732383836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1732383836},{"type":"v1","addr":"172.21.15.6:6835","nonce":1732383836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7296,"qdb_cluster":[7296]},"id":7}]} 2024-03-04T09:30:51.635 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 68 2024-03-04T09:30:51.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:51.641+0000 7f656b600640 1 -- 172.21.15.6:0/1454062974 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f654406b520 msgr2=0x7f654406d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:51.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:51.641+0000 7f656b600640 1 --2- 172.21.15.6:0/1454062974 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f654406b520 0x7f654406d9e0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f656414d770 tx=0x7f6558008040 comp rx=0 tx=0).stop 2024-03-04T09:30:51.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:51.641+0000 7f656b600640 1 -- 172.21.15.6:0/1454062974 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f656415fc60 msgr2=0x7f656415d280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:51.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:51.641+0000 7f656b600640 1 --2- 172.21.15.6:0/1454062974 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f656415fc60 0x7f656415d280 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f655c02c060 tx=0x7f655c0014a0 comp rx=0 tx=0).stop 2024-03-04T09:30:51.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:51.641+0000 7f656b600640 1 -- 172.21.15.6:0/1454062974 shutdown_connections 2024-03-04T09:30:51.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:51.641+0000 7f656b600640 1 --2- 172.21.15.6:0/1454062974 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f656415fc60 0x7f656415d280 unknown :-1 s=CLOSED pgs=1139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:51.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:51.641+0000 7f656b600640 1 --2- 172.21.15.6:0/1454062974 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f654406b520 0x7f654406d9e0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:51.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:51.641+0000 7f656b600640 1 --2- 172.21.15.6:0/1454062974 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f656414fe70 0x7f656415f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:51.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:51.641+0000 7f656b600640 1 --2- 172.21.15.6:0/1454062974 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f654c004d40 0x7f656414f930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:51.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:51.641+0000 7f656b600640 1 -- 172.21.15.6:0/1454062974 >> 172.21.15.6:0/1454062974 conn(0x7f656406cf40 msgr2=0x7f6564145bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:51.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:51.641+0000 7f656b600640 1 -- 172.21.15.6:0/1454062974 shutdown_connections 2024-03-04T09:30:51.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:51.641+0000 7f656b600640 1 -- 172.21.15.6:0/1454062974 wait complete. 2024-03-04T09:30:51.657 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump 2024-03-04T09:30:51.910 INFO:teuthology.orchestra.run.smithi006.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":241,"msgr_send_messages":245,"msgr_recv_bytes":69005,"msgr_send_bytes":14228686,"msgr_created_connections":9,"msgr_active_connections":5,"msgr_running_total_time":0.134003529,"msgr_running_send_time":0.091167114,"msgr_running_recv_time":0.013869230,"msgr_running_fast_dispatch_time":0.022981984,"msgr_send_messages_queue_lat":{"avgcount":244,"sum":0.041075945,"avgtime":0.000168344},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000208,"avgtime":0.000000104},"msgr_recv_encrypted_bytes":1728,"msgr_send_encrypted_bytes":63040},"AsyncMessenger::Worker-1":{"msgr_recv_messages":12221,"msgr_send_messages":20557,"msgr_recv_bytes":4264846,"msgr_send_bytes":12834084,"msgr_created_connections":7,"msgr_active_connections":5,"msgr_running_total_time":2.575314280,"msgr_running_send_time":1.653400622,"msgr_running_recv_time":2.637149411,"msgr_running_fast_dispatch_time":0.013885236,"msgr_send_messages_queue_lat":{"avgcount":20555,"sum":1.336968841,"avgtime":0.000065043},"msgr_handle_ack_lat":{"avgcount":25811,"sum":0.011265656,"avgtime":0.000000436},"msgr_recv_encrypted_bytes":0,"msgr_send_encrypted_bytes":46336},"AsyncMessenger::Worker-2":{"msgr_recv_messages":217,"msgr_send_messages":213,"msgr_recv_bytes":160254,"msgr_send_bytes":4265955,"msgr_created_connections":5,"msgr_active_connections":4,"msgr_running_total_time":0.048679778,"msgr_running_send_time":0.027092981,"msgr_running_recv_time":0.014664307,"msgr_running_fast_dispatch_time":0.008499516,"msgr_send_messages_queue_lat":{"avgcount":212,"sum":0.024049328,"avgtime":0.000113440},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":106544,"msgr_send_encrypted_bytes":7296},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":1242,"complete_latency":{"avgcount":80,"sum":8.982571340,"avgtime":0.112282141}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":1129,"sum":1.084050378,"avgtime":0.000960186}},"mds":{"request":5493,"reply":5493,"reply_latency":{"avgcount":5493,"sum":687.652869045,"avgtime":0.125187123},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":0,"dir_split":0,"dir_merge":0,"inodes":5072,"inodes_top":0,"inodes_bottom":5072,"inodes_pin_tail":0,"inodes_pinned":5072,"inodes_expired":0,"inodes_with_caps":2140,"caps":2140,"subtrees":2,"traverse":5495,"traverse_hit":5495,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":1857,"root_rbytes":14353408,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":102,"ceph_cap_op_grant":6345,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":5619,"handle_client_caps_dirty":3000,"handle_client_cap_release":774,"process_request_cap_release":2292},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":1647,"num_strays_delayed":1349,"num_strays_enqueuing":0,"strays_created":2284,"strays_enqueued":637,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":11558,"evex":0,"evtrm":0,"ev":11563,"evexg":0,"evexd":0,"segadd":13,"segex":0,"segtrm":0,"seg":15,"segmjr":2,"segexg":0,"segexd":0,"expos":4194304,"wrpos":29720076,"rdpos":4195674,"jlat":{"avgcount":16,"sum":4.274523971,"avgtime":0.267157748},"replayed":5},"mds_mem":{"ino":2779,"ino+":3214,"ino-":435,"dir":13,"dir+":13,"dir-":0,"dn":5060,"dn+":5495,"dn-":435,"cap":1105,"cap+":3202,"cap-":2097,"rss":107120,"heap":248164},"mds_server":{"dispatch_client_request":5494,"dispatch_server_request":0,"handle_client_request":5493,"handle_client_session":1,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":679.388313371,"avgtime":0.212308847},"req_getattr_latency":{"avgcount":2,"sum":0.665016344,"avgtime":0.332508172},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.005488280,"avgtime":0.005488280},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":6,"sum":1.043772762,"avgtime":0.173962127},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":2284,"sum":6.550278288,"avgtime":0.002867897},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":4974,"average_load":4974,"avg_session_uptime":931,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":17593477,"buffer_anon_items":8672,"buffer_meta_bytes":0,"buffer_meta_items":0,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5672,"osdmap_items":92,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":16252754,"mds_co_items":205751,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":746,"op_send_bytes":25646032,"op_resend":0,"op_reply":746,"op_latency":{"avgcount":746,"sum":2.862965806,"avgtime":0.003837755},"op_inflight":0,"oplen_avg":{"avgcount":746,"sum":804},"op":746,"op_r":46,"op_w":700,"op_rmw":0,"op_pg":0,"osdop_stat":2,"osdop_create":0,"osdop_read":18,"osdop_write":27,"osdop_writefull":7,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":654,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":0,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":19,"omap_rd":48,"omap_del":4},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":6,"pq_executing":0,"pq_executing_high_water":6,"pq_executed_ops":637,"pq_executed":637,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":253,"max":104857600,"get_started":0,"get":12676,"get_sum":3350935,"get_or_fail_fail":0,"get_or_fail_success":12676,"take":0,"take_sum":0,"put":12675,"put_sum":3350682,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":746,"take_sum":25712290,"put":746,"put_sum":25712290,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":746,"take_sum":746,"put":746,"put_sum":746,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":637,"get_sum":64337,"get_or_fail_fail":0,"get_or_fail_success":637,"take":0,"take_sum":0,"put":10,"put_sum":64337,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x5610c362caa0":{"val":359948,"max":3758096384,"get_started":0,"get":11558,"get_sum":25523587,"get_or_fail_fail":0,"get_or_fail_success":11558,"take":0,"take_sum":0,"put":17,"put_sum":25163639,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-03-04T09:30:51.911 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 5, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.04e-07, "sum": 2.08e-07 }, "msgr_recv_bytes": 69005, "msgr_recv_encrypted_bytes": 1728, "msgr_recv_messages": 241, "msgr_running_fast_dispatch_time": 0.022981984, "msgr_running_recv_time": 0.01386923, "msgr_running_send_time": 0.091167114, "msgr_running_total_time": 0.134003529, "msgr_send_bytes": 14228686, "msgr_send_encrypted_bytes": 63040, "msgr_send_messages": 245, "msgr_send_messages_queue_lat": { "avgcount": 244, "avgtime": 0.000168344, "sum": 0.041075945 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 5, "msgr_created_connections": 7, "msgr_handle_ack_lat": { "avgcount": 25811, "avgtime": 4.36e-07, "sum": 0.011265656 }, "msgr_recv_bytes": 4264846, "msgr_recv_encrypted_bytes": 0, "msgr_recv_messages": 12221, "msgr_running_fast_dispatch_time": 0.013885236, "msgr_running_recv_time": 2.637149411, "msgr_running_send_time": 1.653400622, "msgr_running_total_time": 2.57531428, "msgr_send_bytes": 12834084, "msgr_send_encrypted_bytes": 46336, "msgr_send_messages": 20557, "msgr_send_messages_queue_lat": { "avgcount": 20555, "avgtime": 6.5043e-05, "sum": 1.336968841 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 4, "msgr_created_connections": 5, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 160254, "msgr_recv_encrypted_bytes": 106544, "msgr_recv_messages": 217, "msgr_running_fast_dispatch_time": 0.008499516, "msgr_running_recv_time": 0.014664307, "msgr_running_send_time": 0.027092981, "msgr_running_total_time": 0.048679778, "msgr_send_bytes": 4265955, "msgr_send_encrypted_bytes": 7296, "msgr_send_messages": 213, "msgr_send_messages_queue_lat": { "avgcount": 212, "avgtime": 0.00011344, "sum": 0.024049328 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 80, "avgtime": 0.112282141, "sum": 8.98257134 }, "queue_len": 1242 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 1129, "avgtime": 0.000960186, "sum": 1.084050378 }, "queue_len": 0 }, "mds": { "caps": 2140, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 6345, "ceph_cap_op_revoke": 102, "ceph_cap_op_trunc": 0, "dir_commit": 0, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 774, "handle_client_caps": 5619, "handle_client_caps_dirty": 3000, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 5072, "inodes_bottom": 5072, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 5072, "inodes_top": 0, "inodes_with_caps": 2140, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 2292, "q": 0, "reply": 5493, "reply_latency": { "avgcount": 5493, "avgtime": 0.125187123, "sum": 687.652869045 }, "request": 5493, "root_rbytes": 14353408, "root_rfiles": 1857, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 5495, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 5495, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 1647, "num_strays_delayed": 1349, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 2284, "strays_enqueued": 637, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 11563, "evadd": 11558, "evex": 0, "evexd": 0, "evexg": 0, "evlrg": 0, "evtrm": 0, "expos": 4194304, "jlat": { "avgcount": 16, "avgtime": 0.267157748, "sum": 4.274523971 }, "rdpos": 4195674, "replayed": 5, "seg": 15, "segadd": 13, "segex": 0, "segexd": 0, "segexg": 0, "segmjr": 2, "segtrm": 0, "wrpos": 29720076 }, "mds_mem": { "cap": 1105, "cap+": 3202, "cap-": 2097, "dir": 13, "dir+": 13, "dir-": 0, "dn": 5060, "dn+": 5495, "dn-": 435, "heap": 248164, "ino": 2779, "ino+": 3214, "ino-": 435, "rss": 107120 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 5494, "dispatch_server_request": 0, "handle_client_request": 5493, "handle_client_session": 1, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.212308847, "sum": 679.388313371 }, "req_getattr_latency": { "avgcount": 2, "avgtime": 0.332508172, "sum": 0.665016344 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.00548828, "sum": 0.00548828 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 6, "avgtime": 0.173962127, "sum": 1.043772762 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 2284, "avgtime": 0.002867897, "sum": 6.550278288 } }, "mds_sessions": { "average_load": 4974, "avg_session_uptime": 931, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 4974 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 17593477, "buffer_anon_items": 8672, "buffer_meta_bytes": 0, "buffer_meta_items": 0, "mds_co_bytes": 16252754, "mds_co_items": 205751, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5672, "osdmap_items": 92, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 4, "omap_rd": 48, "omap_wr": 19, "op": 746, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 746, "avgtime": 0.003837755, "sum": 2.862965806 }, "op_pg": 0, "op_r": 46, "op_reply": 746, "op_resend": 0, "op_rmw": 0, "op_send": 746, "op_send_bytes": 25646032, "op_w": 700, "oplen_avg": { "avgcount": 746, "sum": 804 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 0, "osdop_delete": 654, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 18, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 0, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 2, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 27, "osdop_writefull": 7, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 637, "pq_executed_ops": 637, "pq_executing": 0, "pq_executing_high_water": 6, "pq_executing_ops": 0, "pq_executing_ops_high_water": 6, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 12676, "get_or_fail_fail": 0, "get_or_fail_success": 12676, "get_started": 0, "get_sum": 3350935, "max": 104857600, "put": 12675, "put_sum": 3350682, "take": 0, "take_sum": 0, "val": 253, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 746, "put_sum": 25712290, "take": 746, "take_sum": 25712290, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 746, "put_sum": 746, "take": 746, "take_sum": 746, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 637, "get_or_fail_fail": 0, "get_or_fail_success": 637, "get_started": 0, "get_sum": 64337, "max": 3758096384, "put": 10, "put_sum": 64337, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x5610c362caa0": { "get": 11558, "get_or_fail_fail": 0, "get_or_fail_success": 11558, "get_started": 0, "get_sum": 25523587, "max": 3758096384, "put": 17, "put_sum": 25163639, "take": 0, "take_sum": 0, "val": 359948, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-03-04T09:30:51.912 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/1647, 637/2284 2024-03-04T09:30:52.912 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:30:53.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:53.153+0000 7f9b33000640 1 Processor -- start 2024-03-04T09:30:53.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:53.154+0000 7f9b33000640 1 -- start start 2024-03-04T09:30:53.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:53.156+0000 7f9b33000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f9b2c06e090 0x7f9b2c0cedd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:53.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:53.156+0000 7f9b33000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9b2c0cf310 0x7f9b2c15d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:53.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:53.156+0000 7f9b33000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b2c160830 con 0x7f9b2c156340 2024-03-04T09:30:53.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:53.156+0000 7f9b33000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b2c15fa70 con 0x7f9b2c06e090 2024-03-04T09:30:53.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:53.156+0000 7f9b33000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b2c15fd50 con 0x7f9b2c0cf310 2024-03-04T09:30:53.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:53.156+0000 7f9b31200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f9b2c06e090 0x7f9b2c0cedd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:53.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:53.156+0000 7f9b31200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f9b2c06e090 0x7f9b2c0cedd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41458/0 (socket says 172.21.15.6:41458) 2024-03-04T09:30:53.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:53.156+0000 7f9b31200640 1 -- 172.21.15.6:0/1279096715 learned_addr learned my addr 172.21.15.6:0/1279096715 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:30:53.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:53.157+0000 7f9b32600640 1 --2- 172.21.15.6:0/1279096715 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9b2c0cf310 0x7f9b2c15d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:53.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:53.157+0000 7f9b31200640 1 -- 172.21.15.6:0/1279096715 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9b2c0cf310 msgr2=0x7f9b2c15d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:53.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:53.157+0000 7f9b31200640 1 --2- 172.21.15.6:0/1279096715 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9b2c0cf310 0x7f9b2c15d4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:53.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:53.157+0000 7f9b31200640 1 -- 172.21.15.6:0/1279096715 >> v1:172.21.15.6:6789/0 conn(0x7f9b2c156340 legacy=0x7f9b2c06d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:53.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:53.157+0000 7f9b31200640 1 -- 172.21.15.6:0/1279096715 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b2c160030 con 0x7f9b2c06e090 2024-03-04T09:30:53.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:53.157+0000 7f9b32600640 1 --2- 172.21.15.6:0/1279096715 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9b2c0cf310 0x7f9b2c15d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:30:53.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:53.158+0000 7f9b31200640 1 --2- 172.21.15.6:0/1279096715 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9b2c06e090 0x7f9b2c0cedd0 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7f9b18009870 tx=0x7f9b1802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8783817af747b51a server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:53.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:53.159+0000 7f9b2be00640 1 -- 172.21.15.6:0/1279096715 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b18035020 con 0x7f9b2c06e090 2024-03-04T09:30:53.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:53.159+0000 7f9b2be00640 1 -- 172.21.15.6:0/1279096715 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b18031640 con 0x7f9b2c06e090 2024-03-04T09:30:53.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:53.159+0000 7f9b2be00640 1 -- 172.21.15.6:0/1279096715 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b1802d040 con 0x7f9b2c06e090 2024-03-04T09:30:53.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:53.159+0000 7f9b33000640 1 -- 172.21.15.6:0/1279096715 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9b2c06e090 msgr2=0x7f9b2c0cedd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:53.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:53.159+0000 7f9b33000640 1 --2- 172.21.15.6:0/1279096715 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9b2c06e090 0x7f9b2c0cedd0 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7f9b18009870 tx=0x7f9b1802b8f0 comp rx=0 tx=0).stop 2024-03-04T09:30:53.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:53.160+0000 7f9b33000640 1 -- 172.21.15.6:0/1279096715 shutdown_connections 2024-03-04T09:30:53.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:53.160+0000 7f9b33000640 1 --2- 172.21.15.6:0/1279096715 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9b2c0cf310 0x7f9b2c15d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:53.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:53.160+0000 7f9b33000640 1 --2- 172.21.15.6:0/1279096715 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9b2c06e090 0x7f9b2c0cedd0 unknown :-1 s=CLOSED pgs=1141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:53.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:53.160+0000 7f9b33000640 1 -- 172.21.15.6:0/1279096715 >> 172.21.15.6:0/1279096715 conn(0x7f9b2c0b1990 msgr2=0x7f9b2c0b1d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:53.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:53.160+0000 7f9b33000640 1 -- 172.21.15.6:0/1279096715 shutdown_connections 2024-03-04T09:30:53.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:53.160+0000 7f9b33000640 1 -- 172.21.15.6:0/1279096715 wait complete. 2024-03-04T09:30:53.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:53.162+0000 7f9b33000640 1 Processor -- start 2024-03-04T09:30:53.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:53.162+0000 7f9b33000640 1 -- start start 2024-03-04T09:30:53.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:53.163+0000 7f9b33000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9b2c06e090 0x7f9b2c071ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:53.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:53.163+0000 7f9b33000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9b2c0cf310 0x7f9b2c0cd730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:53.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:53.163+0000 7f9b33000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9b2c156340 0x7f9b2c15ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:53.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:53.163+0000 7f9b33000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b2c0cf730 con 0x7f9b2c0cf310 2024-03-04T09:30:53.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:53.163+0000 7f9b33000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b2c156720 con 0x7f9b2c156340 2024-03-04T09:30:53.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:53.163+0000 7f9b33000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b2c153360 con 0x7f9b2c06e090 2024-03-04T09:30:53.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:53.163+0000 7f9b31200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9b2c0cf310 0x7f9b2c0cd730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:53.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:53.164+0000 7f9b31200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9b2c0cf310 0x7f9b2c0cd730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41462/0 (socket says 172.21.15.6:41462) 2024-03-04T09:30:53.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:53.164+0000 7f9b31200640 1 -- 172.21.15.6:0/4092799706 learned_addr learned my addr 172.21.15.6:0/4092799706 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:30:53.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:53.164+0000 7f9b31c00640 1 --2- 172.21.15.6:0/4092799706 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9b2c06e090 0x7f9b2c071ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:53.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:53.164+0000 7f9b32600640 1 --2- 172.21.15.6:0/4092799706 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9b2c156340 0x7f9b2c15ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:53.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:53.164+0000 7f9b31200640 1 -- 172.21.15.6:0/4092799706 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9b2c06e090 msgr2=0x7f9b2c071ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:53.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:53.164+0000 7f9b31200640 1 --2- 172.21.15.6:0/4092799706 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9b2c06e090 0x7f9b2c071ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:53.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:53.164+0000 7f9b31200640 1 -- 172.21.15.6:0/4092799706 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9b2c156340 msgr2=0x7f9b2c15ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:53.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:53.164+0000 7f9b31200640 1 --2- 172.21.15.6:0/4092799706 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9b2c156340 0x7f9b2c15ea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:53.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:53.164+0000 7f9b31200640 1 -- 172.21.15.6:0/4092799706 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b2c0722a0 con 0x7f9b2c0cf310 2024-03-04T09:30:53.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:53.165+0000 7f9b31200640 1 --2- 172.21.15.6:0/4092799706 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9b2c0cf310 0x7f9b2c0cd730 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7f9b18000c00 tx=0x7f9b180046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:53.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:53.165+0000 7f9b29600640 1 -- 172.21.15.6:0/4092799706 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b18040070 con 0x7f9b2c0cf310 2024-03-04T09:30:53.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:53.165+0000 7f9b29600640 1 -- 172.21.15.6:0/4092799706 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b18039bf0 con 0x7f9b2c0cf310 2024-03-04T09:30:53.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:53.165+0000 7f9b29600640 1 -- 172.21.15.6:0/4092799706 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b18035040 con 0x7f9b2c0cf310 2024-03-04T09:30:53.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:53.165+0000 7f9b33000640 1 -- 172.21.15.6:0/4092799706 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b2c072580 con 0x7f9b2c0cf310 2024-03-04T09:30:53.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:53.165+0000 7f9b33000640 1 -- 172.21.15.6:0/4092799706 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b2c0cdc70 con 0x7f9b2c0cf310 2024-03-04T09:30:53.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:53.167+0000 7f9b33000640 1 -- 172.21.15.6:0/4092799706 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b2c04f330 con 0x7f9b2c0cf310 2024-03-04T09:30:53.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:53.168+0000 7f9b29600640 1 -- 172.21.15.6:0/4092799706 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9b1804b060 con 0x7f9b2c0cf310 2024-03-04T09:30:53.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:53.169+0000 7f9b29600640 1 --2- 172.21.15.6:0/4092799706 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f9b0c06b500 0x7f9b0c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:53.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:53.169+0000 7f9b29600640 1 -- 172.21.15.6:0/4092799706 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f9b180abd00 con 0x7f9b2c0cf310 2024-03-04T09:30:53.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:53.170+0000 7f9b31c00640 1 --2- 172.21.15.6:0/4092799706 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f9b0c06b500 0x7f9b0c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:53.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:53.171+0000 7f9b31c00640 1 --2- 172.21.15.6:0/4092799706 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f9b0c06b500 0x7f9b0c06d9c0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f9b20001640 tx=0x7f9b20009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:53.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:53.178+0000 7f9b29600640 1 -- 172.21.15.6:0/4092799706 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9b1802f300 con 0x7f9b2c0cf310 2024-03-04T09:30:53.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:53.526+0000 7f9b33000640 1 -- 172.21.15.6:0/4092799706 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9b2c0b2790 con 0x7f9b2c0cf310 2024-03-04T09:30:53.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:53.526+0000 7f9b29600640 1 -- 172.21.15.6:0/4092799706 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) v1 ==== 94+0+4977 (secure 0 0 0) 0x7f9b1807aad0 con 0x7f9b2c0cf310 2024-03-04T09:30:53.523 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:30:53.524 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":68,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5654,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/1370302030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":1370302030},{"type":"v1","addr":"172.21.15.146:6835","nonce":1370302030}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":6163,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/809822004","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":809822004},{"type":"v1","addr":"172.21.15.146:6837","nonce":809822004}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":66},{"gid":7284,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/1583413840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1583413840},{"type":"v1","addr":"172.21.15.6:6833","nonce":1583413840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64}],"filesystems":[{"mdsmap":{"epoch":68,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:30:20.660103+0000","modified":"2024-03-04T09:30:36.289706+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":98,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7296},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7296":{"gid":7296,"name":"a","rank":0,"incarnation":65,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1732383836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1732383836},{"type":"v1","addr":"172.21.15.6:6835","nonce":1732383836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7296,"qdb_cluster":[7296]},"id":7}]} 2024-03-04T09:30:53.524 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 68 2024-03-04T09:30:53.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:53.529+0000 7f9b33000640 1 -- 172.21.15.6:0/4092799706 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f9b0c06b500 msgr2=0x7f9b0c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:53.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:53.529+0000 7f9b33000640 1 --2- 172.21.15.6:0/4092799706 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f9b0c06b500 0x7f9b0c06d9c0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f9b20001640 tx=0x7f9b20009040 comp rx=0 tx=0).stop 2024-03-04T09:30:53.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:53.529+0000 7f9b33000640 1 -- 172.21.15.6:0/4092799706 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9b2c0cf310 msgr2=0x7f9b2c0cd730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:53.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:53.529+0000 7f9b33000640 1 --2- 172.21.15.6:0/4092799706 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9b2c0cf310 0x7f9b2c0cd730 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7f9b18000c00 tx=0x7f9b180046c0 comp rx=0 tx=0).stop 2024-03-04T09:30:53.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:53.530+0000 7f9b33000640 1 -- 172.21.15.6:0/4092799706 shutdown_connections 2024-03-04T09:30:53.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:53.530+0000 7f9b33000640 1 --2- 172.21.15.6:0/4092799706 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9b2c156340 0x7f9b2c15ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:53.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:53.530+0000 7f9b33000640 1 --2- 172.21.15.6:0/4092799706 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9b2c0cf310 0x7f9b2c0cd730 unknown :-1 s=CLOSED pgs=1142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:53.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:53.530+0000 7f9b33000640 1 --2- 172.21.15.6:0/4092799706 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f9b0c06b500 0x7f9b0c06d9c0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:53.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:53.530+0000 7f9b33000640 1 --2- 172.21.15.6:0/4092799706 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9b2c06e090 0x7f9b2c071ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:53.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:53.530+0000 7f9b33000640 1 -- 172.21.15.6:0/4092799706 >> 172.21.15.6:0/4092799706 conn(0x7f9b2c0b1990 msgr2=0x7f9b2c0b0e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:53.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:53.530+0000 7f9b33000640 1 -- 172.21.15.6:0/4092799706 shutdown_connections 2024-03-04T09:30:53.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:53.530+0000 7f9b33000640 1 -- 172.21.15.6:0/4092799706 wait complete. 2024-03-04T09:30:53.549 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump 2024-03-04T09:30:53.823 INFO:teuthology.orchestra.run.smithi006.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":246,"msgr_send_messages":251,"msgr_recv_bytes":70411,"msgr_send_bytes":18460073,"msgr_created_connections":9,"msgr_active_connections":5,"msgr_running_total_time":0.156660745,"msgr_running_send_time":0.110198461,"msgr_running_recv_time":0.013995949,"msgr_running_fast_dispatch_time":0.026406389,"msgr_send_messages_queue_lat":{"avgcount":250,"sum":0.046633482,"avgtime":0.000186533},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000208,"avgtime":0.000000104},"msgr_recv_encrypted_bytes":1728,"msgr_send_encrypted_bytes":63040},"AsyncMessenger::Worker-1":{"msgr_recv_messages":13041,"msgr_send_messages":22355,"msgr_recv_bytes":4700296,"msgr_send_bytes":13692734,"msgr_created_connections":7,"msgr_active_connections":5,"msgr_running_total_time":2.807940742,"msgr_running_send_time":1.791773039,"msgr_running_recv_time":2.752934174,"msgr_running_fast_dispatch_time":0.014869686,"msgr_send_messages_queue_lat":{"avgcount":22353,"sum":1.417080504,"avgtime":0.000063395},"msgr_handle_ack_lat":{"avgcount":27913,"sum":0.011924149,"avgtime":0.000000427},"msgr_recv_encrypted_bytes":0,"msgr_send_encrypted_bytes":46336},"AsyncMessenger::Worker-2":{"msgr_recv_messages":217,"msgr_send_messages":214,"msgr_recv_bytes":160254,"msgr_send_bytes":4266281,"msgr_created_connections":5,"msgr_active_connections":4,"msgr_running_total_time":0.048747141,"msgr_running_send_time":0.027150988,"msgr_running_recv_time":0.014664307,"msgr_running_fast_dispatch_time":0.008499516,"msgr_send_messages_queue_lat":{"avgcount":213,"sum":0.024333417,"avgtime":0.000114241},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":106544,"msgr_send_encrypted_bytes":7296},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":1939,"complete_latency":{"avgcount":88,"sum":11.131545561,"avgtime":0.126494835}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":1137,"sum":2.525143593,"avgtime":0.002220882}},"mds":{"request":6301,"reply":6300,"reply_latency":{"avgcount":6300,"sum":689.291418846,"avgtime":0.109411336},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":0,"dir_split":0,"dir_merge":0,"inodes":5573,"inodes_top":0,"inodes_bottom":5573,"inodes_pin_tail":0,"inodes_pinned":5573,"inodes_expired":0,"inodes_with_caps":765,"caps":765,"subtrees":2,"traverse":6303,"traverse_hit":6302,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":702,"root_rbytes":5537792,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":102,"ceph_cap_op_grant":6833,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":5619,"handle_client_caps_dirty":3000,"handle_client_cap_release":776,"process_request_cap_release":3100},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":993,"num_strays_delayed":0,"num_strays_enqueuing":339,"strays_created":3091,"strays_enqueued":2098,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":14493,"evex":0,"evtrm":0,"ev":14498,"evexg":0,"evexd":0,"segadd":17,"segex":0,"segtrm":0,"seg":19,"segmjr":3,"segexg":0,"segexd":0,"expos":4194304,"wrpos":34015405,"rdpos":4195674,"jlat":{"avgcount":20,"sum":5.079343412,"avgtime":0.253967170},"replayed":5},"mds_mem":{"ino":2485,"ino+":3214,"ino-":729,"dir":13,"dir+":13,"dir-":0,"dn":5573,"dn+":6302,"dn-":729,"cap":765,"cap+":3202,"cap-":2437,"rss":103368,"heap":248164},"mds_server":{"dispatch_client_request":6302,"dispatch_server_request":0,"handle_client_request":6301,"handle_client_session":1,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":679.388313371,"avgtime":0.212308847},"req_getattr_latency":{"avgcount":2,"sum":0.665016344,"avgtime":0.332508172},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.005488280,"avgtime":0.005488280},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":6,"sum":1.043772762,"avgtime":0.173962127},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":3091,"sum":8.188828089,"avgtime":0.002649248},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":5664,"average_load":5664,"avg_session_uptime":936,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":10377661,"buffer_anon_items":5542,"buffer_meta_bytes":176,"buffer_meta_items":2,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5672,"osdmap_items":92,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":11570177,"mds_co_items":178363,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":8,"op_laggy":0,"op_send":768,"op_send_bytes":30057143,"op_resend":0,"op_reply":760,"op_latency":{"avgcount":760,"sum":2.977964431,"avgtime":0.003918374},"op_inflight":0,"oplen_avg":{"avgcount":768,"sum":830},"op":768,"op_r":50,"op_w":718,"op_rmw":0,"op_pg":0,"osdop_stat":2,"osdop_create":0,"osdop_read":22,"osdop_write":35,"osdop_writefull":7,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":662,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":0,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":23,"omap_rd":48,"omap_del":6},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":6,"pq_executing_ops_high_water":6,"pq_executing":6,"pq_executing_high_water":6,"pq_executed_ops":638,"pq_executed":638,"pq_item_in_journal":1799},"throttle-msgr_dispatch_throttler-mds":{"val":253,"max":104857600,"get_started":0,"get":13500,"get_sum":3713288,"get_or_fail_fail":0,"get_or_fail_success":13500,"take":0,"take_sum":0,"put":13499,"put_sum":3713035,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":34340,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":768,"take_sum":30270962,"put":760,"put_sum":30236622,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":8,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":768,"take_sum":768,"put":760,"put_sum":760,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":2437,"get_sum":246137,"get_or_fail_fail":0,"get_or_fail_success":2437,"take":0,"take_sum":0,"put":15,"put_sum":246137,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x5610c362caa0":{"val":460973,"max":3758096384,"get_started":0,"get":14493,"get_sum":29818916,"get_or_fail_fail":0,"get_or_fail_success":14493,"take":0,"take_sum":0,"put":20,"put_sum":29357943,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-03-04T09:30:53.825 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 5, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.04e-07, "sum": 2.08e-07 }, "msgr_recv_bytes": 70411, "msgr_recv_encrypted_bytes": 1728, "msgr_recv_messages": 246, "msgr_running_fast_dispatch_time": 0.026406389, "msgr_running_recv_time": 0.013995949, "msgr_running_send_time": 0.110198461, "msgr_running_total_time": 0.156660745, "msgr_send_bytes": 18460073, "msgr_send_encrypted_bytes": 63040, "msgr_send_messages": 251, "msgr_send_messages_queue_lat": { "avgcount": 250, "avgtime": 0.000186533, "sum": 0.046633482 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 5, "msgr_created_connections": 7, "msgr_handle_ack_lat": { "avgcount": 27913, "avgtime": 4.27e-07, "sum": 0.011924149 }, "msgr_recv_bytes": 4700296, "msgr_recv_encrypted_bytes": 0, "msgr_recv_messages": 13041, "msgr_running_fast_dispatch_time": 0.014869686, "msgr_running_recv_time": 2.752934174, "msgr_running_send_time": 1.791773039, "msgr_running_total_time": 2.807940742, "msgr_send_bytes": 13692734, "msgr_send_encrypted_bytes": 46336, "msgr_send_messages": 22355, "msgr_send_messages_queue_lat": { "avgcount": 22353, "avgtime": 6.3395e-05, "sum": 1.417080504 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 4, "msgr_created_connections": 5, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 160254, "msgr_recv_encrypted_bytes": 106544, "msgr_recv_messages": 217, "msgr_running_fast_dispatch_time": 0.008499516, "msgr_running_recv_time": 0.014664307, "msgr_running_send_time": 0.027150988, "msgr_running_total_time": 0.048747141, "msgr_send_bytes": 4266281, "msgr_send_encrypted_bytes": 7296, "msgr_send_messages": 214, "msgr_send_messages_queue_lat": { "avgcount": 213, "avgtime": 0.000114241, "sum": 0.024333417 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 88, "avgtime": 0.126494835, "sum": 11.131545561 }, "queue_len": 1939 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 1137, "avgtime": 0.002220882, "sum": 2.525143593 }, "queue_len": 0 }, "mds": { "caps": 765, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 6833, "ceph_cap_op_revoke": 102, "ceph_cap_op_trunc": 0, "dir_commit": 0, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 776, "handle_client_caps": 5619, "handle_client_caps_dirty": 3000, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 5573, "inodes_bottom": 5573, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 5573, "inodes_top": 0, "inodes_with_caps": 765, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 3100, "q": 0, "reply": 6300, "reply_latency": { "avgcount": 6300, "avgtime": 0.109411336, "sum": 689.291418846 }, "request": 6301, "root_rbytes": 5537792, "root_rfiles": 702, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 6303, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 6302, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 993, "num_strays_delayed": 0, "num_strays_enqueuing": 339, "recovery_completed": 0, "recovery_started": 0, "strays_created": 3091, "strays_enqueued": 2098, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 14498, "evadd": 14493, "evex": 0, "evexd": 0, "evexg": 0, "evlrg": 0, "evtrm": 0, "expos": 4194304, "jlat": { "avgcount": 20, "avgtime": 0.25396717, "sum": 5.079343412 }, "rdpos": 4195674, "replayed": 5, "seg": 19, "segadd": 17, "segex": 0, "segexd": 0, "segexg": 0, "segmjr": 3, "segtrm": 0, "wrpos": 34015405 }, "mds_mem": { "cap": 765, "cap+": 3202, "cap-": 2437, "dir": 13, "dir+": 13, "dir-": 0, "dn": 5573, "dn+": 6302, "dn-": 729, "heap": 248164, "ino": 2485, "ino+": 3214, "ino-": 729, "rss": 103368 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 6302, "dispatch_server_request": 0, "handle_client_request": 6301, "handle_client_session": 1, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.212308847, "sum": 679.388313371 }, "req_getattr_latency": { "avgcount": 2, "avgtime": 0.332508172, "sum": 0.665016344 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.00548828, "sum": 0.00548828 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 6, "avgtime": 0.173962127, "sum": 1.043772762 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 3091, "avgtime": 0.002649248, "sum": 8.188828089 } }, "mds_sessions": { "average_load": 5664, "avg_session_uptime": 936, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 5664 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 10377661, "buffer_anon_items": 5542, "buffer_meta_bytes": 176, "buffer_meta_items": 2, "mds_co_bytes": 11570177, "mds_co_items": 178363, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5672, "osdmap_items": 92, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 6, "omap_rd": 48, "omap_wr": 23, "op": 768, "op_Infinitylight": 0, "op_active": 8, "op_laggy": 0, "op_latency": { "avgcount": 760, "avgtime": 0.003918374, "sum": 2.977964431 }, "op_pg": 0, "op_r": 50, "op_reply": 760, "op_resend": 0, "op_rmw": 0, "op_send": 768, "op_send_bytes": 30057143, "op_w": 718, "oplen_avg": { "avgcount": 768, "sum": 830 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 0, "osdop_delete": 662, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 22, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 0, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 2, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 35, "osdop_writefull": 7, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 638, "pq_executed_ops": 638, "pq_executing": 6, "pq_executing_high_water": 6, "pq_executing_ops": 6, "pq_executing_ops_high_water": 6, "pq_item_in_journal": 1799 }, "throttle-msgr_dispatch_throttler-mds": { "get": 13500, "get_or_fail_fail": 0, "get_or_fail_success": 13500, "get_started": 0, "get_sum": 3713288, "max": 104857600, "put": 13499, "put_sum": 3713035, "take": 0, "take_sum": 0, "val": 253, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 760, "put_sum": 30236622, "take": 768, "take_sum": 30270962, "val": 34340, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 760, "put_sum": 760, "take": 768, "take_sum": 768, "val": 8, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 2437, "get_or_fail_fail": 0, "get_or_fail_success": 2437, "get_started": 0, "get_sum": 246137, "max": 3758096384, "put": 15, "put_sum": 246137, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x5610c362caa0": { "get": 14493, "get_or_fail_fail": 0, "get_or_fail_success": 14493, "get_started": 0, "get_sum": 29818916, "max": 3758096384, "put": 20, "put_sum": 29357943, "take": 0, "take_sum": 0, "val": 460973, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-03-04T09:30:53.825 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 6/993, 638/3091 2024-03-04T09:30:54.613 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:30:54.826 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:30:55.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.038+0000 7fb110a00640 1 Processor -- start 2024-03-04T09:30:55.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:54.887+0000 7f787ea00640 1 Processor -- start 2024-03-04T09:30:55.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:54.887+0000 7f787ea00640 1 -- start start 2024-03-04T09:30:55.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.038+0000 7fb110a00640 1 -- start start 2024-03-04T09:30:55.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.039+0000 7fb110a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fb10c16e140 0x7fb10c0ad4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:55.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:54.890+0000 7f787ea00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f78780b5120 0x7f78780b2e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:55.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:54.890+0000 7f787ea00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f787816bcc0 0x7f7878167e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:55.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:54.890+0000 7f787ea00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78780b34e0 con 0x7f787808ba40 2024-03-04T09:30:55.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:54.890+0000 7f787ea00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78780b37c0 con 0x7f787816bcc0 2024-03-04T09:30:55.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:54.890+0000 7f787ea00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78780b3aa0 con 0x7f78780b5120 2024-03-04T09:30:55.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.039+0000 7fb110a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb10c16e520 con 0x7fb10c16e7b0 2024-03-04T09:30:55.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.039+0000 7fb110a00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb10c0b4ac0 con 0x7fb10c16f3c0 2024-03-04T09:30:55.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.039+0000 7fb110a00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb10c0b4da0 con 0x7fb10c16e140 2024-03-04T09:30:55.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.040+0000 7fb10a800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fb10c16e140 0x7fb10c0ad4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:55.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.040+0000 7fb10b200640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fb10c16e7b0 0x7fb10c06d580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:43138/0 (socket says 172.21.15.6:43138) 2024-03-04T09:30:55.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.040+0000 7fb10b200640 1 -- 172.21.15.6:0/1656654338 learned_addr learned my addr 172.21.15.6:0/1656654338 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:30:55.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.040+0000 7fb109400640 1 -- 172.21.15.6:0/1656654338 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 945468783 0 0) 0x7fb10c16e520 con 0x7fb10c16e7b0 2024-03-04T09:30:55.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:54.891+0000 7f787ca00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f787816bcc0 0x7f7878167e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:55.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:54.891+0000 7f787ca00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f787816bcc0 0x7f7878167e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.6:39376/0 (socket says 172.21.15.6:39376) 2024-03-04T09:30:55.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:54.891+0000 7f787ca00640 1 -- 172.21.15.6:0/4140694212 learned_addr learned my addr 172.21.15.6:0/4140694212 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:30:55.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.106+0000 7fb109400640 1 -- 172.21.15.6:0/1656654338 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0f0003500 con 0x7fb10c16e7b0 2024-03-04T09:30:55.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.106+0000 7fb109400640 1 -- 172.21.15.6:0/1656654338 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1281032169 0 0) 0x7fb10c0b4ac0 con 0x7fb10c16f3c0 2024-03-04T09:30:55.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.106+0000 7fb109400640 1 -- 172.21.15.6:0/1656654338 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb10c16e520 con 0x7fb10c16f3c0 2024-03-04T09:30:55.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:54.891+0000 7f7877400640 1 --2- 172.21.15.6:0/4140694212 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f78780b5120 0x7f78780b2e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:55.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:54.893+0000 7f7876a00640 1 -- 172.21.15.6:0/4140694212 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1944967706 0 0) 0x7f78780b34e0 con 0x7f787808ba40 2024-03-04T09:30:55.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.106+0000 7f7876a00640 1 -- 172.21.15.6:0/4140694212 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f785c003420 con 0x7f787808ba40 2024-03-04T09:30:55.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.107+0000 7fb10a800640 1 -- 172.21.15.6:0/1656654338 >> v1:172.21.15.146:6789/0 conn(0x7fb10c16f3c0 legacy=0x7fb10c0b4150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:55.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.107+0000 7fb10a800640 1 -- 172.21.15.6:0/1656654338 >> v1:172.21.15.6:6789/0 conn(0x7fb10c16e7b0 legacy=0x7fb10c06d580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:55.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.107+0000 7fb10a800640 1 -- 172.21.15.6:0/1656654338 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb10c06caa0 con 0x7fb10c16e140 2024-03-04T09:30:55.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.107+0000 7f787ca00640 1 -- 172.21.15.6:0/4140694212 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f78780b5120 msgr2=0x7f78780b2e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:55.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.107+0000 7f787ca00640 1 --2- 172.21.15.6:0/4140694212 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f78780b5120 0x7f78780b2e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:55.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.107+0000 7f787ca00640 1 -- 172.21.15.6:0/4140694212 >> v1:172.21.15.146:6789/0 conn(0x7f787808ba40 legacy=0x7f78780b4a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:55.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.107+0000 7f787ca00640 1 -- 172.21.15.6:0/4140694212 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78780b3d80 con 0x7f787816bcc0 2024-03-04T09:30:55.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.109+0000 7fb10a800640 1 --2- 172.21.15.6:0/1656654338 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb10c16e140 0x7fb10c0ad4d0 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7fb0fc009d90 tx=0x7fb0fc02bec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5153741bb68afc77 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:55.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.110+0000 7fb109400640 1 -- 172.21.15.6:0/1656654338 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0fc0094e0 con 0x7fb10c16e140 2024-03-04T09:30:55.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.110+0000 7fb109400640 1 -- 172.21.15.6:0/1656654338 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0fc003990 con 0x7fb10c16e140 2024-03-04T09:30:55.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.110+0000 7fb109400640 1 -- 172.21.15.6:0/1656654338 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0fc003dc0 con 0x7fb10c16e140 2024-03-04T09:30:55.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.111+0000 7f787ca00640 1 --2- 172.21.15.6:0/4140694212 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f787816bcc0 0x7f7878167e10 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f78640083f0 tx=0x7f786400abe0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f93eb20f214138c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:55.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.112+0000 7f7876a00640 1 -- 172.21.15.6:0/4140694212 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7864034020 con 0x7f787816bcc0 2024-03-04T09:30:55.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.112+0000 7fb110a00640 1 -- 172.21.15.6:0/1656654338 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb10c16e140 msgr2=0x7fb10c0ad4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:55.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.112+0000 7fb110a00640 1 --2- 172.21.15.6:0/1656654338 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb10c16e140 0x7fb10c0ad4d0 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7fb0fc009d90 tx=0x7fb0fc02bec0 comp rx=0 tx=0).stop 2024-03-04T09:30:55.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.112+0000 7f7876a00640 1 -- 172.21.15.6:0/4140694212 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7864008df0 con 0x7f787816bcc0 2024-03-04T09:30:55.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.112+0000 7f7876a00640 1 -- 172.21.15.6:0/4140694212 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f786400a2f0 con 0x7f787816bcc0 2024-03-04T09:30:55.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.112+0000 7fb110a00640 1 -- 172.21.15.6:0/1656654338 shutdown_connections 2024-03-04T09:30:55.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.112+0000 7fb110a00640 1 --2- 172.21.15.6:0/1656654338 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb10c16e140 0x7fb10c0ad4d0 unknown :-1 s=CLOSED pgs=1144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:55.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.112+0000 7fb110a00640 1 -- 172.21.15.6:0/1656654338 >> 172.21.15.6:0/1656654338 conn(0x7fb10c06afc0 msgr2=0x7fb10c06b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:55.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.113+0000 7f787ea00640 1 -- 172.21.15.6:0/4140694212 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f787816bcc0 msgr2=0x7f7878167e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:55.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.113+0000 7f787ea00640 1 --2- 172.21.15.6:0/4140694212 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f787816bcc0 0x7f7878167e10 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f78640083f0 tx=0x7f786400abe0 comp rx=0 tx=0).stop 2024-03-04T09:30:55.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.113+0000 7fb110a00640 1 -- 172.21.15.6:0/1656654338 shutdown_connections 2024-03-04T09:30:55.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.113+0000 7f787ea00640 1 -- 172.21.15.6:0/4140694212 shutdown_connections 2024-03-04T09:30:55.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.113+0000 7f787ea00640 1 --2- 172.21.15.6:0/4140694212 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f787816bcc0 0x7f7878167e10 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:55.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.113+0000 7f787ea00640 1 --2- 172.21.15.6:0/4140694212 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f78780b5120 0x7f78780b2e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:55.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.113+0000 7fb110a00640 1 -- 172.21.15.6:0/1656654338 wait complete. 2024-03-04T09:30:55.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.113+0000 7f787ea00640 1 -- 172.21.15.6:0/4140694212 >> 172.21.15.6:0/4140694212 conn(0x7f78780afac0 msgr2=0x7f78780afea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:55.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.114+0000 7f787ea00640 1 -- 172.21.15.6:0/4140694212 shutdown_connections 2024-03-04T09:30:55.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.114+0000 7f787ea00640 1 -- 172.21.15.6:0/4140694212 wait complete. 2024-03-04T09:30:55.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.116+0000 7f787ea00640 1 Processor -- start 2024-03-04T09:30:55.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.116+0000 7fb110a00640 1 Processor -- start 2024-03-04T09:30:55.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.116+0000 7fb110a00640 1 -- start start 2024-03-04T09:30:55.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.116+0000 7f787ea00640 1 -- start start 2024-03-04T09:30:55.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.118+0000 7f787ea00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f787808ba40 0x7f787815a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:55.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.118+0000 7f787ea00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f78780b5120 0x7f787816d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:55.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.118+0000 7f787ea00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f787816bcc0 0x7f787806da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:55.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.118+0000 7f787ea00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7878168350 con 0x7f787816bcc0 2024-03-04T09:30:55.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.118+0000 7f787ea00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78780b4060 con 0x7f787808ba40 2024-03-04T09:30:55.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.118+0000 7f787ea00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78781684d0 con 0x7f78780b5120 2024-03-04T09:30:55.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.118+0000 7f787ca00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f787816bcc0 0x7f787806da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:55.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.118+0000 7f787ca00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f787816bcc0 0x7f787806da50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41494/0 (socket says 172.21.15.6:41494) 2024-03-04T09:30:55.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.118+0000 7f787ca00640 1 -- 172.21.15.6:0/1938218364 learned_addr learned my addr 172.21.15.6:0/1938218364 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:30:55.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.119+0000 7f7877400640 1 --2- 172.21.15.6:0/1938218364 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f78780b5120 0x7f787816d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:55.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.119+0000 7f7877e00640 1 --2- 172.21.15.6:0/1938218364 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f787808ba40 0x7f787815a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:55.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.119+0000 7fb110a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb10c16e7b0 0x7fb10c20d740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:55.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.119+0000 7fb110a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb10c16f3c0 0x7fb10c0c77e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:55.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.119+0000 7fb110a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb10c16c4e0 0x7fb10c164ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:55.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.119+0000 7fb110a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb10c0b4860 con 0x7fb10c16e7b0 2024-03-04T09:30:55.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.119+0000 7fb110a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb10c06d0f0 con 0x7fb10c16f3c0 2024-03-04T09:30:55.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.119+0000 7fb110a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb10c16ec20 con 0x7fb10c16c4e0 2024-03-04T09:30:55.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.120+0000 7f787ca00640 1 -- 172.21.15.6:0/1938218364 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f78780b5120 msgr2=0x7f787816d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:55.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.120+0000 7f787ca00640 1 --2- 172.21.15.6:0/1938218364 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f78780b5120 0x7f787816d510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:55.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.120+0000 7f787ca00640 1 -- 172.21.15.6:0/1938218364 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f787808ba40 msgr2=0x7f787815a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:55.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.120+0000 7f787ca00640 1 --2- 172.21.15.6:0/1938218364 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f787808ba40 0x7f787815a620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:55.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.120+0000 7f787ca00640 1 -- 172.21.15.6:0/1938218364 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f787816def0 con 0x7f787816bcc0 2024-03-04T09:30:55.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.120+0000 7fb10b200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb10c16c4e0 0x7fb10c164ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:55.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.120+0000 7fb10b200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb10c16c4e0 0x7fb10c164ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.6:49448/0 (socket says 172.21.15.6:49448) 2024-03-04T09:30:55.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.120+0000 7fb10b200640 1 -- 172.21.15.6:0/2211567955 learned_addr learned my addr 172.21.15.6:0/2211567955 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:30:55.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.121+0000 7fb10a800640 1 --2- 172.21.15.6:0/2211567955 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb10c16e7b0 0x7fb10c20d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:55.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.121+0000 7fb109e00640 1 --2- 172.21.15.6:0/2211567955 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb10c16f3c0 0x7fb10c0c77e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:55.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.121+0000 7fb10a800640 1 -- 172.21.15.6:0/2211567955 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb10c16c4e0 msgr2=0x7fb10c164ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:55.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.121+0000 7fb10a800640 1 --2- 172.21.15.6:0/2211567955 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb10c16c4e0 0x7fb10c164ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:55.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.121+0000 7f787ca00640 1 --2- 172.21.15.6:0/1938218364 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f787816bcc0 0x7f787806da50 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7f7864033040 tx=0x7f78640046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:55.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.121+0000 7fb10a800640 1 -- 172.21.15.6:0/2211567955 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb10c16f3c0 msgr2=0x7fb10c0c77e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:55.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.121+0000 7fb10a800640 1 --2- 172.21.15.6:0/2211567955 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb10c16f3c0 0x7fb10c0c77e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:55.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.121+0000 7fb10a800640 1 -- 172.21.15.6:0/2211567955 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb10c20dc80 con 0x7fb10c16e7b0 2024-03-04T09:30:55.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.122+0000 7f786fe00640 1 -- 172.21.15.6:0/1938218364 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7864034040 con 0x7f787816bcc0 2024-03-04T09:30:55.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.122+0000 7f786fe00640 1 -- 172.21.15.6:0/1938218364 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7864030850 con 0x7f787816bcc0 2024-03-04T09:30:55.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.123+0000 7f786fe00640 1 -- 172.21.15.6:0/1938218364 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7864030c80 con 0x7f787816bcc0 2024-03-04T09:30:55.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.123+0000 7fb10a800640 1 --2- 172.21.15.6:0/2211567955 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb10c16e7b0 0x7fb10c20d740 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7fb0fc004000 tx=0x7fb0fc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:55.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.124+0000 7f787ea00640 1 -- 172.21.15.6:0/1938218364 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f787816e1d0 con 0x7f787816bcc0 2024-03-04T09:30:55.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.123+0000 7fb102a00640 1 -- 172.21.15.6:0/2211567955 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0fc041020 con 0x7fb10c16e7b0 2024-03-04T09:30:55.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.124+0000 7fb102a00640 1 -- 172.21.15.6:0/2211567955 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0fc02ca70 con 0x7fb10c16e7b0 2024-03-04T09:30:55.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.124+0000 7fb102a00640 1 -- 172.21.15.6:0/2211567955 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0fc02ce60 con 0x7fb10c16e7b0 2024-03-04T09:30:55.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.124+0000 7f787ea00640 1 -- 172.21.15.6:0/1938218364 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f787816e790 con 0x7f787816bcc0 2024-03-04T09:30:55.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.126+0000 7f787ea00640 1 -- 172.21.15.6:0/1938218364 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78780692a0 con 0x7f787816bcc0 2024-03-04T09:30:55.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.126+0000 7fb110a00640 1 -- 172.21.15.6:0/2211567955 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb10c16c9e0 con 0x7fb10c16e7b0 2024-03-04T09:30:55.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.127+0000 7fb110a00640 1 -- 172.21.15.6:0/2211567955 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb10c16cfa0 con 0x7fb10c16e7b0 2024-03-04T09:30:55.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.128+0000 7fb110a00640 1 -- 172.21.15.6:0/2211567955 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb10c0cf9f0 con 0x7fb10c16e7b0 2024-03-04T09:30:55.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.128+0000 7f786fe00640 1 -- 172.21.15.6:0/1938218364 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f786404a060 con 0x7f787816bcc0 2024-03-04T09:30:55.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.130+0000 7f786fe00640 1 --2- 172.21.15.6:0/1938218364 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f785806b4d0 0x7f785806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:55.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.130+0000 7f786fe00640 1 -- 172.21.15.6:0/1938218364 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f78640abfa0 con 0x7f787816bcc0 2024-03-04T09:30:55.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.131+0000 7f7877e00640 1 --2- 172.21.15.6:0/1938218364 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f785806b4d0 0x7f785806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:55.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.133+0000 7fb102a00640 1 -- 172.21.15.6:0/2211567955 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb0fc044070 con 0x7fb10c16e7b0 2024-03-04T09:30:55.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.133+0000 7f7877e00640 1 --2- 172.21.15.6:0/1938218364 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f785806b4d0 0x7f785806d990 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f7870001ea0 tx=0x7f7870009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:55.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.134+0000 7fb102a00640 1 --2- 172.21.15.6:0/2211567955 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb0e806b550 0x7fb0e806da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:55.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.135+0000 7fb109e00640 1 --2- 172.21.15.6:0/2211567955 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb0e806b550 0x7fb0e806da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:55.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.137+0000 7fb109e00640 1 --2- 172.21.15.6:0/2211567955 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb0e806b550 0x7fb0e806da10 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fb10c0c7540 tx=0x7fb0f4009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:55.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.141+0000 7fb102a00640 1 -- 172.21.15.6:0/2211567955 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5881+0+0 (secure 0 0 0) 0x7fb0fc078360 con 0x7fb10c16e7b0 2024-03-04T09:30:55.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.141+0000 7f786fe00640 1 -- 172.21.15.6:0/1938218364 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f786403e070 con 0x7f787816bcc0 2024-03-04T09:30:55.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.150+0000 7fb102a00640 1 -- 172.21.15.6:0/2211567955 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb10c20dc80 con 0x7fb10c16e7b0 2024-03-04T09:30:55.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.568+0000 7f787ea00640 1 -- 172.21.15.6:0/1938218364 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f787816b0b0 con 0x7f787816bcc0 2024-03-04T09:30:55.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.571+0000 7f786fe00640 1 -- 172.21.15.6:0/1938218364 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) v1 ==== 94+0+4977 (secure 0 0 0) 0x7f7864030e20 con 0x7f787816bcc0 2024-03-04T09:30:55.569 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:30:55.569 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":68,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5654,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/1370302030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":1370302030},{"type":"v1","addr":"172.21.15.146:6835","nonce":1370302030}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":6163,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/809822004","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":809822004},{"type":"v1","addr":"172.21.15.146:6837","nonce":809822004}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":66},{"gid":7284,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/1583413840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1583413840},{"type":"v1","addr":"172.21.15.6:6833","nonce":1583413840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64}],"filesystems":[{"mdsmap":{"epoch":68,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:30:20.660103+0000","modified":"2024-03-04T09:30:36.289706+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":98,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7296},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7296":{"gid":7296,"name":"a","rank":0,"incarnation":65,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1732383836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1732383836},{"type":"v1","addr":"172.21.15.6:6835","nonce":1732383836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7296,"qdb_cluster":[7296]},"id":7}]} 2024-03-04T09:30:55.569 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 68 2024-03-04T09:30:55.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.575+0000 7f787ea00640 1 -- 172.21.15.6:0/1938218364 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f785806b4d0 msgr2=0x7f785806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:55.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.575+0000 7f787ea00640 1 --2- 172.21.15.6:0/1938218364 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f785806b4d0 0x7f785806d990 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f7870001ea0 tx=0x7f7870009040 comp rx=0 tx=0).stop 2024-03-04T09:30:55.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.575+0000 7f787ea00640 1 -- 172.21.15.6:0/1938218364 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f787816bcc0 msgr2=0x7f787806da50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:55.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.575+0000 7f787ea00640 1 --2- 172.21.15.6:0/1938218364 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f787816bcc0 0x7f787806da50 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7f7864033040 tx=0x7f78640046c0 comp rx=0 tx=0).stop 2024-03-04T09:30:55.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.575+0000 7f787ea00640 1 -- 172.21.15.6:0/1938218364 shutdown_connections 2024-03-04T09:30:55.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.576+0000 7f787ea00640 1 --2- 172.21.15.6:0/1938218364 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f787816bcc0 0x7f787806da50 unknown :-1 s=CLOSED pgs=1145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:55.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.576+0000 7f787ea00640 1 --2- 172.21.15.6:0/1938218364 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f785806b4d0 0x7f785806d990 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:55.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.576+0000 7f787ea00640 1 --2- 172.21.15.6:0/1938218364 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f78780b5120 0x7f787816d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:55.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.576+0000 7f787ea00640 1 --2- 172.21.15.6:0/1938218364 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f787808ba40 0x7f787815a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:55.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.576+0000 7f787ea00640 1 -- 172.21.15.6:0/1938218364 >> 172.21.15.6:0/1938218364 conn(0x7f78780afac0 msgr2=0x7f78780afea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:55.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.576+0000 7f787ea00640 1 -- 172.21.15.6:0/1938218364 shutdown_connections 2024-03-04T09:30:55.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.576+0000 7f787ea00640 1 -- 172.21.15.6:0/1938218364 wait complete. 2024-03-04T09:30:55.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.590+0000 7fb110a00640 1 -- 172.21.15.6:0/2211567955 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb10c04f330 con 0x7fb10c16e7b0 2024-03-04T09:30:55.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.591+0000 7fb102a00640 1 -- 172.21.15.6:0/2211567955 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) v1 ==== 94+0+4977 (secure 0 0 0) 0x7fb0fc07a2c0 con 0x7fb10c16e7b0 2024-03-04T09:30:55.588 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:30:55.588 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":68,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5654,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/1370302030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":1370302030},{"type":"v1","addr":"172.21.15.146:6835","nonce":1370302030}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":6163,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/809822004","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":809822004},{"type":"v1","addr":"172.21.15.146:6837","nonce":809822004}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":66},{"gid":7284,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/1583413840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1583413840},{"type":"v1","addr":"172.21.15.6:6833","nonce":1583413840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64}],"filesystems":[{"mdsmap":{"epoch":68,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:30:20.660103+0000","modified":"2024-03-04T09:30:36.289706+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":98,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7296},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7296":{"gid":7296,"name":"a","rank":0,"incarnation":65,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1732383836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1732383836},{"type":"v1","addr":"172.21.15.6:6835","nonce":1732383836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7296,"qdb_cluster":[7296]},"id":7}]} 2024-03-04T09:30:55.589 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 68 2024-03-04T09:30:55.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.595+0000 7fb110a00640 1 -- 172.21.15.6:0/2211567955 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb0e806b550 msgr2=0x7fb0e806da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:55.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.595+0000 7fb110a00640 1 --2- 172.21.15.6:0/2211567955 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb0e806b550 0x7fb0e806da10 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fb10c0c7540 tx=0x7fb0f4009040 comp rx=0 tx=0).stop 2024-03-04T09:30:55.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.595+0000 7fb110a00640 1 -- 172.21.15.6:0/2211567955 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb10c16e7b0 msgr2=0x7fb10c20d740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:55.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.595+0000 7fb110a00640 1 --2- 172.21.15.6:0/2211567955 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb10c16e7b0 0x7fb10c20d740 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7fb0fc004000 tx=0x7fb0fc004a40 comp rx=0 tx=0).stop 2024-03-04T09:30:55.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.595+0000 7fb110a00640 1 -- 172.21.15.6:0/2211567955 shutdown_connections 2024-03-04T09:30:55.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.595+0000 7fb110a00640 1 --2- 172.21.15.6:0/2211567955 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb0e806b550 0x7fb0e806da10 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:55.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.595+0000 7fb110a00640 1 --2- 172.21.15.6:0/2211567955 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb10c16c4e0 0x7fb10c164ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:55.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.595+0000 7fb110a00640 1 --2- 172.21.15.6:0/2211567955 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb10c16f3c0 0x7fb10c0c77e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:55.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.595+0000 7fb110a00640 1 --2- 172.21.15.6:0/2211567955 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb10c16e7b0 0x7fb10c20d740 unknown :-1 s=CLOSED pgs=1146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:55.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.595+0000 7fb110a00640 1 -- 172.21.15.6:0/2211567955 >> 172.21.15.6:0/2211567955 conn(0x7fb10c06afc0 msgr2=0x7fb10c071ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:55.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.596+0000 7fb110a00640 1 -- 172.21.15.6:0/2211567955 shutdown_connections 2024-03-04T09:30:55.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:55.596+0000 7fb110a00640 1 -- 172.21.15.6:0/2211567955 wait complete. 2024-03-04T09:30:55.600 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json flush journal 2024-03-04T09:30:55.617 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump 2024-03-04T09:30:55.914 INFO:teuthology.orchestra.run.smithi006.stdout:{"message":"","return_code":0} 2024-03-04T09:30:55.915 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-04T09:30:55.931 INFO:teuthology.orchestra.run.smithi006.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":964,"msgr_send_messages":969,"msgr_recv_bytes":255492,"msgr_send_bytes":20333343,"msgr_created_connections":9,"msgr_active_connections":5,"msgr_running_total_time":0.431663008,"msgr_running_send_time":0.158842157,"msgr_running_recv_time":0.031962417,"msgr_running_fast_dispatch_time":0.227784973,"msgr_send_messages_queue_lat":{"avgcount":968,"sum":0.235937273,"avgtime":0.000243736},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000208,"avgtime":0.000000104},"msgr_recv_encrypted_bytes":1728,"msgr_send_encrypted_bytes":64832},"AsyncMessenger::Worker-1":{"msgr_recv_messages":14352,"msgr_send_messages":24887,"msgr_recv_bytes":5165138,"msgr_send_bytes":14513402,"msgr_created_connections":7,"msgr_active_connections":5,"msgr_running_total_time":3.238215031,"msgr_running_send_time":1.940077223,"msgr_running_recv_time":2.808287580,"msgr_running_fast_dispatch_time":0.229851457,"msgr_send_messages_queue_lat":{"avgcount":24885,"sum":1.685241687,"avgtime":0.000067721},"msgr_handle_ack_lat":{"avgcount":28797,"sum":0.012254664,"avgtime":0.000000425},"msgr_recv_encrypted_bytes":0,"msgr_send_encrypted_bytes":46336},"AsyncMessenger::Worker-2":{"msgr_recv_messages":928,"msgr_send_messages":924,"msgr_recv_bytes":344012,"msgr_send_bytes":4507138,"msgr_created_connections":5,"msgr_active_connections":4,"msgr_running_total_time":0.307855089,"msgr_running_send_time":0.060419249,"msgr_running_recv_time":0.032919556,"msgr_running_fast_dispatch_time":0.208979010,"msgr_send_messages_queue_lat":{"avgcount":923,"sum":0.121708980,"avgtime":0.000131862},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":106816,"msgr_send_encrypted_bytes":7920},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":129,"sum":12.104633783,"avgtime":0.093834370}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":5303,"sum":3.266211347,"avgtime":0.000615917}},"mds":{"request":6410,"reply":6410,"reply_latency":{"avgcount":6410,"sum":690.066655152,"avgtime":0.107654704},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":12,"dir_split":0,"dir_merge":0,"inodes":3465,"inodes_top":0,"inodes_bottom":3465,"inodes_pin_tail":0,"inodes_pinned":3465,"inodes_expired":0,"inodes_with_caps":85,"caps":85,"subtrees":2,"traverse":6412,"traverse_hit":6412,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":0,"root_rbytes":0,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":105,"ceph_cap_op_grant":7450,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":5622,"handle_client_caps_dirty":3000,"handle_client_cap_release":780,"process_request_cap_release":3210},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":84,"num_strays_delayed":84,"num_strays_enqueuing":0,"strays_created":3201,"strays_enqueued":3117,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":15721,"evex":15723,"evtrm":15723,"ev":15725,"evexg":0,"evexd":15722,"segadd":19,"segex":20,"segtrm":20,"seg":1,"segmjr":1,"segexg":0,"segexd":19,"expos":35096584,"wrpos":35100778,"rdpos":4195674,"jlat":{"avgcount":24,"sum":5.529588877,"avgtime":0.230399536},"replayed":5},"mds_mem":{"ino":914,"ino+":3214,"ino-":2300,"dir":13,"dir+":13,"dir-":0,"dn":4112,"dn+":6412,"dn-":2300,"cap":85,"cap+":3202,"cap-":3117,"rss":114376,"heap":248164},"mds_server":{"dispatch_client_request":6414,"dispatch_server_request":0,"handle_client_request":6410,"handle_client_session":1,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":679.388313371,"avgtime":0.212308847},"req_getattr_latency":{"avgcount":2,"sum":0.665016344,"avgtime":0.332508172},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.005488280,"avgtime":0.005488280},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":6,"sum":1.043772762,"avgtime":0.173962127},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":1,"sum":0.588745986,"avgtime":0.588745986},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":3200,"sum":8.375318409,"avgtime":0.002617287},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":5720,"average_load":5720,"avg_session_uptime":936,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":10377661,"buffer_anon_items":5542,"buffer_meta_bytes":176,"buffer_meta_items":2,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5672,"osdmap_items":92,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":11570177,"mds_co_items":178363,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":3379,"op_send_bytes":31776895,"op_resend":0,"op_reply":3379,"op_latency":{"avgcount":3379,"sum":13.228841973,"avgtime":0.003915016},"op_inflight":0,"oplen_avg":{"avgcount":3379,"sum":3649},"op":3379,"op_r":56,"op_w":3323,"op_rmw":0,"op_pg":0,"osdop_stat":12,"osdop_create":94,"osdop_read":28,"osdop_write":44,"osdop_writefull":13,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":3142,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":188,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":45,"omap_rd":48,"omap_del":10},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":6,"pq_executing":0,"pq_executing_high_water":6,"pq_executed_ops":3117,"pq_executed":3117,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":16240,"get_sum":4300317,"get_or_fail_fail":0,"get_or_fail_success":16240,"take":0,"take_sum":0,"put":16240,"put_sum":4300317,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":3379,"take_sum":32093633,"put":3379,"put_sum":32093633,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":3379,"take_sum":3379,"put":3379,"put_sum":3379,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":3117,"get_sum":314817,"get_or_fail_fail":0,"get_or_fail_success":3117,"take":0,"take_sum":0,"put":20,"put_sum":314817,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x5610c362caa0":{"val":0,"max":3758096384,"get_started":0,"get":15721,"get_sum":30904289,"get_or_fail_fail":0,"get_or_fail_success":15721,"take":0,"take_sum":0,"put":24,"put_sum":30904289,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-03-04T09:30:55.932 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 5, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.04e-07, "sum": 2.08e-07 }, "msgr_recv_bytes": 255492, "msgr_recv_encrypted_bytes": 1728, "msgr_recv_messages": 964, "msgr_running_fast_dispatch_time": 0.227784973, "msgr_running_recv_time": 0.031962417, "msgr_running_send_time": 0.158842157, "msgr_running_total_time": 0.431663008, "msgr_send_bytes": 20333343, "msgr_send_encrypted_bytes": 64832, "msgr_send_messages": 969, "msgr_send_messages_queue_lat": { "avgcount": 968, "avgtime": 0.000243736, "sum": 0.235937273 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 5, "msgr_created_connections": 7, "msgr_handle_ack_lat": { "avgcount": 28797, "avgtime": 4.25e-07, "sum": 0.012254664 }, "msgr_recv_bytes": 5165138, "msgr_recv_encrypted_bytes": 0, "msgr_recv_messages": 14352, "msgr_running_fast_dispatch_time": 0.229851457, "msgr_running_recv_time": 2.80828758, "msgr_running_send_time": 1.940077223, "msgr_running_total_time": 3.238215031, "msgr_send_bytes": 14513402, "msgr_send_encrypted_bytes": 46336, "msgr_send_messages": 24887, "msgr_send_messages_queue_lat": { "avgcount": 24885, "avgtime": 6.7721e-05, "sum": 1.685241687 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 4, "msgr_created_connections": 5, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 344012, "msgr_recv_encrypted_bytes": 106816, "msgr_recv_messages": 928, "msgr_running_fast_dispatch_time": 0.20897901, "msgr_running_recv_time": 0.032919556, "msgr_running_send_time": 0.060419249, "msgr_running_total_time": 0.307855089, "msgr_send_bytes": 4507138, "msgr_send_encrypted_bytes": 7920, "msgr_send_messages": 924, "msgr_send_messages_queue_lat": { "avgcount": 923, "avgtime": 0.000131862, "sum": 0.12170898 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 129, "avgtime": 0.09383437, "sum": 12.104633783 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 5303, "avgtime": 0.000615917, "sum": 3.266211347 }, "queue_len": 0 }, "mds": { "caps": 85, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 7450, "ceph_cap_op_revoke": 105, "ceph_cap_op_trunc": 0, "dir_commit": 12, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 780, "handle_client_caps": 5622, "handle_client_caps_dirty": 3000, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 3465, "inodes_bottom": 3465, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 3465, "inodes_top": 0, "inodes_with_caps": 85, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 3210, "q": 0, "reply": 6410, "reply_latency": { "avgcount": 6410, "avgtime": 0.107654704, "sum": 690.066655152 }, "request": 6410, "root_rbytes": 0, "root_rfiles": 0, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 6412, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 6412, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 84, "num_strays_delayed": 84, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 3201, "strays_enqueued": 3117, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 15725, "evadd": 15721, "evex": 15723, "evexd": 15722, "evexg": 0, "evlrg": 0, "evtrm": 15723, "expos": 35096584, "jlat": { "avgcount": 24, "avgtime": 0.230399536, "sum": 5.529588877 }, "rdpos": 4195674, "replayed": 5, "seg": 1, "segadd": 19, "segex": 20, "segexd": 19, "segexg": 0, "segmjr": 1, "segtrm": 20, "wrpos": 35100778 }, "mds_mem": { "cap": 85, "cap+": 3202, "cap-": 3117, "dir": 13, "dir+": 13, "dir-": 0, "dn": 4112, "dn+": 6412, "dn-": 2300, "heap": 248164, "ino": 914, "ino+": 3214, "ino-": 2300, "rss": 114376 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 6414, "dispatch_server_request": 0, "handle_client_request": 6410, "handle_client_session": 1, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.212308847, "sum": 679.388313371 }, "req_getattr_latency": { "avgcount": 2, "avgtime": 0.332508172, "sum": 0.665016344 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.00548828, "sum": 0.00548828 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 6, "avgtime": 0.173962127, "sum": 1.043772762 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 1, "avgtime": 0.588745986, "sum": 0.588745986 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 3200, "avgtime": 0.002617287, "sum": 8.375318409 } }, "mds_sessions": { "average_load": 5720, "avg_session_uptime": 936, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 5720 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 10377661, "buffer_anon_items": 5542, "buffer_meta_bytes": 176, "buffer_meta_items": 2, "mds_co_bytes": 11570177, "mds_co_items": 178363, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5672, "osdmap_items": 92, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 10, "omap_rd": 48, "omap_wr": 45, "op": 3379, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 3379, "avgtime": 0.003915016, "sum": 13.228841973 }, "op_pg": 0, "op_r": 56, "op_reply": 3379, "op_resend": 0, "op_rmw": 0, "op_send": 3379, "op_send_bytes": 31776895, "op_w": 3323, "oplen_avg": { "avgcount": 3379, "sum": 3649 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 94, "osdop_delete": 3142, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 28, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 188, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 12, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 44, "osdop_writefull": 13, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 3117, "pq_executed_ops": 3117, "pq_executing": 0, "pq_executing_high_water": 6, "pq_executing_ops": 0, "pq_executing_ops_high_water": 6, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 16240, "get_or_fail_fail": 0, "get_or_fail_success": 16240, "get_started": 0, "get_sum": 4300317, "max": 104857600, "put": 16240, "put_sum": 4300317, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 3379, "put_sum": 32093633, "take": 3379, "take_sum": 32093633, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 3379, "put_sum": 3379, "take": 3379, "take_sum": 3379, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 3117, "get_or_fail_fail": 0, "get_or_fail_success": 3117, "get_started": 0, "get_sum": 314817, "max": 3758096384, "put": 20, "put_sum": 314817, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x5610c362caa0": { "get": 15721, "get_or_fail_fail": 0, "get_or_fail_success": 15721, "get_started": 0, "get_sum": 30904289, "max": 3758096384, "put": 24, "put_sum": 30904289, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-03-04T09:30:55.933 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/84, 3117/3201 2024-03-04T09:30:56.933 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:30:57.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.152+0000 7f2544e00640 1 Processor -- start 2024-03-04T09:30:57.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.152+0000 7f2544e00640 1 -- start start 2024-03-04T09:30:57.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.154+0000 7f2544e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f25400d2e20 0x7f25401690c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:57.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.154+0000 7f2544e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25400b5120 con 0x7f25400b30b0 2024-03-04T09:30:57.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.154+0000 7f2544e00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f254015e950 con 0x7f254015e570 2024-03-04T09:30:57.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.154+0000 7f2544e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25400b4080 con 0x7f25400d2e20 2024-03-04T09:30:57.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.154+0000 7f2537e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f25400b30b0 0x7f25400d2710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:43148/0 (socket says 172.21.15.6:43148) 2024-03-04T09:30:57.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.154+0000 7f2537e00640 1 -- 172.21.15.6:0/2296288422 learned_addr learned my addr 172.21.15.6:0/2296288422 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:30:57.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.154+0000 7f253e800640 1 --2- 172.21.15.6:0/2296288422 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f25400d2e20 0x7f25401690c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:57.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.155+0000 7f253d400640 1 -- 172.21.15.6:0/2296288422 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4266489258 0 0) 0x7f25400b5120 con 0x7f25400b30b0 2024-03-04T09:30:57.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.155+0000 7f253d400640 1 -- 172.21.15.6:0/2296288422 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2524003520 con 0x7f25400b30b0 2024-03-04T09:30:57.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.155+0000 7f253d400640 1 -- 172.21.15.6:0/2296288422 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3705855137 0 0) 0x7f2524003520 con 0x7f25400b30b0 2024-03-04T09:30:57.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.155+0000 7f253d400640 1 -- 172.21.15.6:0/2296288422 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f25400d2e20 msgr2=0x7f25401690c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:57.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.155+0000 7f253d400640 1 --2- 172.21.15.6:0/2296288422 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f25400d2e20 0x7f25401690c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:57.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.155+0000 7f253d400640 1 -- 172.21.15.6:0/2296288422 >> v1:172.21.15.146:6789/0 conn(0x7f254015e570 legacy=0x7f25400b3570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:57.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.155+0000 7f253d400640 1 -- 172.21.15.6:0/2296288422 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25400b4360 con 0x7f25400b30b0 2024-03-04T09:30:57.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.155+0000 7f253d400640 1 -- 172.21.15.6:0/2296288422 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7f2528001670 con 0x7f25400b30b0 2024-03-04T09:30:57.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.155+0000 7f253d400640 1 -- 172.21.15.6:0/2296288422 >> v1:172.21.15.6:6789/0 conn(0x7f25400b30b0 legacy=0x7f25400d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:57.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.155+0000 7f253d400640 1 --2- 172.21.15.6:0/2296288422 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f25400d2e20 0x7f25240039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:57.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.155+0000 7f253d400640 1 --2- 172.21.15.6:0/2296288422 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2524003f60 0x7f2524004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:57.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.155+0000 7f253d400640 1 --2- 172.21.15.6:0/2296288422 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2524004d40 0x7f25240051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:57.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.155+0000 7f253d400640 1 -- 172.21.15.6:0/2296288422 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25400b4080 con 0x7f2524004d40 2024-03-04T09:30:57.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.155+0000 7f253d400640 1 -- 172.21.15.6:0/2296288422 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f252400baa0 con 0x7f2524003f60 2024-03-04T09:30:57.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.155+0000 7f253d400640 1 -- 172.21.15.6:0/2296288422 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f252400bca0 con 0x7f25400d2e20 2024-03-04T09:30:57.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.156+0000 7f253e800640 1 --2- 172.21.15.6:0/2296288422 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2524004d40 0x7f25240051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:57.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.156+0000 7f2537e00640 1 --2- 172.21.15.6:0/2296288422 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2524003f60 0x7f2524004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:57.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.156+0000 7f253e800640 1 -- 172.21.15.6:0/2296288422 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f25400d2e20 msgr2=0x7f25240039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:57.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.156+0000 7f253de00640 1 --2- 172.21.15.6:0/2296288422 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f25400d2e20 0x7f25240039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:57.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.156+0000 7f253e800640 1 --2- 172.21.15.6:0/2296288422 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f25400d2e20 0x7f25240039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:57.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.156+0000 7f253e800640 1 -- 172.21.15.6:0/2296288422 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2524003f60 msgr2=0x7f2524004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:57.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.156+0000 7f253e800640 1 --2- 172.21.15.6:0/2296288422 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2524003f60 0x7f2524004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:57.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.156+0000 7f253e800640 1 -- 172.21.15.6:0/2296288422 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2524003520 con 0x7f2524004d40 2024-03-04T09:30:57.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.156+0000 7f2537e00640 1 --2- 172.21.15.6:0/2296288422 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2524003f60 0x7f2524004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:30:57.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.156+0000 7f253de00640 1 --2- 172.21.15.6:0/2296288422 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f25400d2e20 0x7f25240039e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:30:57.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.157+0000 7f253e800640 1 --2- 172.21.15.6:0/2296288422 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2524004d40 0x7f25240051c0 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f2538009f90 tx=0x7f2524005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3478bff57522d696 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:57.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.158+0000 7f253d400640 1 -- 172.21.15.6:0/2296288422 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2538039070 con 0x7f2524004d40 2024-03-04T09:30:57.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.158+0000 7f253d400640 1 -- 172.21.15.6:0/2296288422 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f253802cbd0 con 0x7f2524004d40 2024-03-04T09:30:57.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.158+0000 7f253d400640 1 -- 172.21.15.6:0/2296288422 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2538009c10 con 0x7f2524004d40 2024-03-04T09:30:57.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.158+0000 7f2544e00640 1 -- 172.21.15.6:0/2296288422 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2524004d40 msgr2=0x7f25240051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:57.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.158+0000 7f2544e00640 1 --2- 172.21.15.6:0/2296288422 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2524004d40 0x7f25240051c0 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f2538009f90 tx=0x7f2524005830 comp rx=0 tx=0).stop 2024-03-04T09:30:57.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.158+0000 7f253e800640 1 -- 172.21.15.6:0/2296288422 reap_dead start 2024-03-04T09:30:57.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.158+0000 7f2544e00640 1 -- 172.21.15.6:0/2296288422 shutdown_connections 2024-03-04T09:30:57.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.158+0000 7f2544e00640 1 -- 172.21.15.6:0/2296288422 >> 172.21.15.6:0/2296288422 conn(0x7f2540073070 msgr2=0x7f254009d410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:57.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.159+0000 7f2544e00640 1 -- 172.21.15.6:0/2296288422 shutdown_connections 2024-03-04T09:30:57.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.159+0000 7f2544e00640 1 -- 172.21.15.6:0/2296288422 wait complete. 2024-03-04T09:30:57.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.160+0000 7f2544e00640 1 Processor -- start 2024-03-04T09:30:57.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.160+0000 7f2544e00640 1 -- start start 2024-03-04T09:30:57.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.161+0000 7f2544e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2524004d40 0x7f25400cdc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:57.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.161+0000 7f2544e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f254014afd0 0x7f25400b8670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:57.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.161+0000 7f2544e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2540160dd0 0x7f2540161250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:57.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.161+0000 7f2544e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25400b4c00 con 0x7f2540160dd0 2024-03-04T09:30:57.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.161+0000 7f2544e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25400b4920 con 0x7f254014afd0 2024-03-04T09:30:57.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.161+0000 7f2544e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25400cfcc0 con 0x7f2524004d40 2024-03-04T09:30:57.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.162+0000 7f253e800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2540160dd0 0x7f2540161250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:57.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.162+0000 7f253e800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2540160dd0 0x7f2540161250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41522/0 (socket says 172.21.15.6:41522) 2024-03-04T09:30:57.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.162+0000 7f253e800640 1 -- 172.21.15.6:0/1436230995 learned_addr learned my addr 172.21.15.6:0/1436230995 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:30:57.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.162+0000 7f253de00640 1 --2- 172.21.15.6:0/1436230995 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2524004d40 0x7f25400cdc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:57.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.162+0000 7f2537e00640 1 --2- 172.21.15.6:0/1436230995 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f254014afd0 0x7f25400b8670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:57.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.162+0000 7f253e800640 1 -- 172.21.15.6:0/1436230995 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2524004d40 msgr2=0x7f25400cdc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:57.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.162+0000 7f253e800640 1 --2- 172.21.15.6:0/1436230995 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2524004d40 0x7f25400cdc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:57.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.162+0000 7f253e800640 1 -- 172.21.15.6:0/1436230995 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f254014afd0 msgr2=0x7f25400b8670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:57.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.162+0000 7f253e800640 1 --2- 172.21.15.6:0/1436230995 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f254014afd0 0x7f25400b8670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:57.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.162+0000 7f253e800640 1 -- 172.21.15.6:0/1436230995 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25400b8bb0 con 0x7f2540160dd0 2024-03-04T09:30:57.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.162+0000 7f2537e00640 1 --2- 172.21.15.6:0/1436230995 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f254014afd0 0x7f25400b8670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:30:57.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.163+0000 7f253e800640 1 --2- 172.21.15.6:0/1436230995 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2540160dd0 0x7f2540161250 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7f2538009f90 tx=0x7f2538009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:57.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.163+0000 7f2536000640 1 -- 172.21.15.6:0/1436230995 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2538039070 con 0x7f2540160dd0 2024-03-04T09:30:57.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.163+0000 7f2536000640 1 -- 172.21.15.6:0/1436230995 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2538044070 con 0x7f2540160dd0 2024-03-04T09:30:57.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.163+0000 7f2536000640 1 -- 172.21.15.6:0/1436230995 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2538009c10 con 0x7f2540160dd0 2024-03-04T09:30:57.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.163+0000 7f2544e00640 1 -- 172.21.15.6:0/1436230995 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25400b8e30 con 0x7f2540160dd0 2024-03-04T09:30:57.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.163+0000 7f2544e00640 1 -- 172.21.15.6:0/1436230995 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25400b93f0 con 0x7f2540160dd0 2024-03-04T09:30:57.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.164+0000 7f2544e00640 1 -- 172.21.15.6:0/1436230995 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2540068070 con 0x7f2540160dd0 2024-03-04T09:30:57.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.165+0000 7f2536000640 1 -- 172.21.15.6:0/1436230995 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f253803e830 con 0x7f2540160dd0 2024-03-04T09:30:57.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.166+0000 7f2536000640 1 --2- 172.21.15.6:0/1436230995 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f251c06b500 0x7f251c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:57.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.166+0000 7f2536000640 1 -- 172.21.15.6:0/1436230995 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f25380ac730 con 0x7f2540160dd0 2024-03-04T09:30:57.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.167+0000 7f253de00640 1 --2- 172.21.15.6:0/1436230995 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f251c06b500 0x7f251c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:57.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.168+0000 7f253de00640 1 --2- 172.21.15.6:0/1436230995 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f251c06b500 0x7f251c06d9c0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f2530001750 tx=0x7f2530009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:57.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.171+0000 7f2536000640 1 -- 172.21.15.6:0/1436230995 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f25400b8bb0 con 0x7f2540160dd0 2024-03-04T09:30:57.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.436+0000 7f2544e00640 1 -- 172.21.15.6:0/1436230995 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f25400d24f0 con 0x7f2540160dd0 2024-03-04T09:30:57.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.438+0000 7f2536000640 1 -- 172.21.15.6:0/1436230995 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) v1 ==== 94+0+4977 (secure 0 0 0) 0x7f253807b3d0 con 0x7f2540160dd0 2024-03-04T09:30:57.435 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:30:57.435 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":68,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5654,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/1370302030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":1370302030},{"type":"v1","addr":"172.21.15.146:6835","nonce":1370302030}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":6163,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/809822004","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":809822004},{"type":"v1","addr":"172.21.15.146:6837","nonce":809822004}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":66},{"gid":7284,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/1583413840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1583413840},{"type":"v1","addr":"172.21.15.6:6833","nonce":1583413840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64}],"filesystems":[{"mdsmap":{"epoch":68,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:30:20.660103+0000","modified":"2024-03-04T09:30:36.289706+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":98,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7296},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7296":{"gid":7296,"name":"a","rank":0,"incarnation":65,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1732383836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1732383836},{"type":"v1","addr":"172.21.15.6:6835","nonce":1732383836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7296,"qdb_cluster":[7296]},"id":7}]} 2024-03-04T09:30:57.435 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 68 2024-03-04T09:30:57.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.440+0000 7f2544e00640 1 -- 172.21.15.6:0/1436230995 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f251c06b500 msgr2=0x7f251c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:57.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.440+0000 7f2544e00640 1 --2- 172.21.15.6:0/1436230995 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f251c06b500 0x7f251c06d9c0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f2530001750 tx=0x7f2530009040 comp rx=0 tx=0).stop 2024-03-04T09:30:57.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.440+0000 7f2544e00640 1 -- 172.21.15.6:0/1436230995 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2540160dd0 msgr2=0x7f2540161250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:57.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.440+0000 7f2544e00640 1 --2- 172.21.15.6:0/1436230995 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2540160dd0 0x7f2540161250 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7f2538009f90 tx=0x7f2538009750 comp rx=0 tx=0).stop 2024-03-04T09:30:57.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.441+0000 7f2544e00640 1 -- 172.21.15.6:0/1436230995 shutdown_connections 2024-03-04T09:30:57.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.441+0000 7f2544e00640 1 --2- 172.21.15.6:0/1436230995 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2540160dd0 0x7f2540161250 unknown :-1 s=CLOSED pgs=1149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:57.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.441+0000 7f2544e00640 1 --2- 172.21.15.6:0/1436230995 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f254014afd0 0x7f25400b8670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:57.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.441+0000 7f2544e00640 1 --2- 172.21.15.6:0/1436230995 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f251c06b500 0x7f251c06d9c0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:57.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.441+0000 7f2544e00640 1 --2- 172.21.15.6:0/1436230995 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2524004d40 0x7f25400cdc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:57.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.441+0000 7f2544e00640 1 -- 172.21.15.6:0/1436230995 >> 172.21.15.6:0/1436230995 conn(0x7f2540073070 msgr2=0x7f25400b3720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:57.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.441+0000 7f2544e00640 1 -- 172.21.15.6:0/1436230995 shutdown_connections 2024-03-04T09:30:57.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.441+0000 7f2544e00640 1 -- 172.21.15.6:0/1436230995 wait complete. 2024-03-04T09:30:57.457 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump 2024-03-04T09:30:57.690 INFO:teuthology.orchestra.run.smithi006.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":990,"msgr_send_messages":995,"msgr_recv_bytes":262168,"msgr_send_bytes":20393165,"msgr_created_connections":9,"msgr_active_connections":5,"msgr_running_total_time":0.436033032,"msgr_running_send_time":0.161358021,"msgr_running_recv_time":0.032725575,"msgr_running_fast_dispatch_time":0.228612663,"msgr_send_messages_queue_lat":{"avgcount":994,"sum":0.238019031,"avgtime":0.000239455},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000208,"avgtime":0.000000104},"msgr_recv_encrypted_bytes":1728,"msgr_send_encrypted_bytes":64832},"AsyncMessenger::Worker-1":{"msgr_recv_messages":14390,"msgr_send_messages":24923,"msgr_recv_bytes":5185242,"msgr_send_bytes":14533656,"msgr_created_connections":7,"msgr_active_connections":5,"msgr_running_total_time":3.242700787,"msgr_running_send_time":1.942029079,"msgr_running_recv_time":2.809453583,"msgr_running_fast_dispatch_time":0.230961122,"msgr_send_messages_queue_lat":{"avgcount":24921,"sum":1.687663654,"avgtime":0.000067720},"msgr_handle_ack_lat":{"avgcount":28799,"sum":0.012254965,"avgtime":0.000000425},"msgr_recv_encrypted_bytes":0,"msgr_send_encrypted_bytes":46336},"AsyncMessenger::Worker-2":{"msgr_recv_messages":959,"msgr_send_messages":955,"msgr_recv_bytes":351886,"msgr_send_bytes":4517244,"msgr_created_connections":5,"msgr_active_connections":4,"msgr_running_total_time":0.311061772,"msgr_running_send_time":0.061659273,"msgr_running_recv_time":0.033755739,"msgr_running_fast_dispatch_time":0.209843231,"msgr_send_messages_queue_lat":{"avgcount":954,"sum":0.124162436,"avgtime":0.000130149},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":106816,"msgr_send_encrypted_bytes":7920},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":132,"sum":12.112039175,"avgtime":0.091757872}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":5464,"sum":3.276418394,"avgtime":0.000599637}},"mds":{"request":6410,"reply":6410,"reply_latency":{"avgcount":6410,"sum":690.066655152,"avgtime":0.107654704},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":12,"dir_split":0,"dir_merge":0,"inodes":95,"inodes_top":0,"inodes_bottom":95,"inodes_pin_tail":0,"inodes_pinned":95,"inodes_expired":0,"inodes_with_caps":1,"caps":1,"subtrees":2,"traverse":6412,"traverse_hit":6412,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":0,"root_rbytes":0,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":105,"ceph_cap_op_grant":7450,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":5622,"handle_client_caps_dirty":3000,"handle_client_cap_release":781,"process_request_cap_release":3210},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":0,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":3201,"strays_enqueued":3201,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":15805,"evex":15723,"evtrm":15723,"ev":87,"evexg":0,"evexd":15722,"segadd":19,"segex":20,"segtrm":20,"seg":1,"segmjr":1,"segexg":0,"segexd":19,"expos":35096584,"wrpos":35151943,"rdpos":4195674,"jlat":{"avgcount":25,"sum":5.532321132,"avgtime":0.221292845},"replayed":5},"mds_mem":{"ino":97,"ino+":3214,"ino-":3117,"dir":13,"dir+":13,"dir-":0,"dn":95,"dn+":6412,"dn-":6317,"cap":1,"cap+":3202,"cap-":3201,"rss":114632,"heap":248164},"mds_server":{"dispatch_client_request":6414,"dispatch_server_request":0,"handle_client_request":6410,"handle_client_session":1,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":679.388313371,"avgtime":0.212308847},"req_getattr_latency":{"avgcount":2,"sum":0.665016344,"avgtime":0.332508172},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.005488280,"avgtime":0.005488280},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":6,"sum":1.043772762,"avgtime":0.173962127},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":1,"sum":0.588745986,"avgtime":0.588745986},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":3200,"sum":8.375318409,"avgtime":0.002617287},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":5720,"average_load":5720,"avg_session_uptime":941,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":935168,"buffer_anon_items":348,"buffer_meta_bytes":176,"buffer_meta_items":2,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5672,"osdmap_items":92,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":337193,"mds_co_items":6297,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":3472,"op_send_bytes":31836751,"op_resend":0,"op_reply":3472,"op_latency":{"avgcount":3472,"sum":13.565837947,"avgtime":0.003907211},"op_inflight":0,"oplen_avg":{"avgcount":3472,"sum":3744},"op":3472,"op_r":59,"op_w":3413,"op_rmw":0,"op_pg":0,"osdop_stat":12,"osdop_create":94,"osdop_read":31,"osdop_write":48,"osdop_writefull":14,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":3226,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":188,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":47,"omap_rd":48,"omap_del":11},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":6,"pq_executing":0,"pq_executing_high_water":6,"pq_executed_ops":3201,"pq_executed":3201,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":16335,"get_sum":4326421,"get_or_fail_fail":0,"get_or_fail_success":16335,"take":0,"take_sum":0,"put":16335,"put_sum":4326421,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":3472,"take_sum":32161973,"put":3472,"put_sum":32161973,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":3472,"take_sum":3472,"put":3472,"put_sum":3472,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":3201,"get_sum":323301,"get_or_fail_fail":0,"get_or_fail_success":3201,"take":0,"take_sum":0,"put":23,"put_sum":323301,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x5610c362caa0":{"val":0,"max":3758096384,"get_started":0,"get":15805,"get_sum":30955454,"get_or_fail_fail":0,"get_or_fail_success":15805,"take":0,"take_sum":0,"put":25,"put_sum":30955454,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-03-04T09:30:57.692 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 5, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.04e-07, "sum": 2.08e-07 }, "msgr_recv_bytes": 262168, "msgr_recv_encrypted_bytes": 1728, "msgr_recv_messages": 990, "msgr_running_fast_dispatch_time": 0.228612663, "msgr_running_recv_time": 0.032725575, "msgr_running_send_time": 0.161358021, "msgr_running_total_time": 0.436033032, "msgr_send_bytes": 20393165, "msgr_send_encrypted_bytes": 64832, "msgr_send_messages": 995, "msgr_send_messages_queue_lat": { "avgcount": 994, "avgtime": 0.000239455, "sum": 0.238019031 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 5, "msgr_created_connections": 7, "msgr_handle_ack_lat": { "avgcount": 28799, "avgtime": 4.25e-07, "sum": 0.012254965 }, "msgr_recv_bytes": 5185242, "msgr_recv_encrypted_bytes": 0, "msgr_recv_messages": 14390, "msgr_running_fast_dispatch_time": 0.230961122, "msgr_running_recv_time": 2.809453583, "msgr_running_send_time": 1.942029079, "msgr_running_total_time": 3.242700787, "msgr_send_bytes": 14533656, "msgr_send_encrypted_bytes": 46336, "msgr_send_messages": 24923, "msgr_send_messages_queue_lat": { "avgcount": 24921, "avgtime": 6.772e-05, "sum": 1.687663654 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 4, "msgr_created_connections": 5, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 351886, "msgr_recv_encrypted_bytes": 106816, "msgr_recv_messages": 959, "msgr_running_fast_dispatch_time": 0.209843231, "msgr_running_recv_time": 0.033755739, "msgr_running_send_time": 0.061659273, "msgr_running_total_time": 0.311061772, "msgr_send_bytes": 4517244, "msgr_send_encrypted_bytes": 7920, "msgr_send_messages": 955, "msgr_send_messages_queue_lat": { "avgcount": 954, "avgtime": 0.000130149, "sum": 0.124162436 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 132, "avgtime": 0.091757872, "sum": 12.112039175 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 5464, "avgtime": 0.000599637, "sum": 3.276418394 }, "queue_len": 0 }, "mds": { "caps": 1, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 7450, "ceph_cap_op_revoke": 105, "ceph_cap_op_trunc": 0, "dir_commit": 12, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 781, "handle_client_caps": 5622, "handle_client_caps_dirty": 3000, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 95, "inodes_bottom": 95, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 95, "inodes_top": 0, "inodes_with_caps": 1, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 3210, "q": 0, "reply": 6410, "reply_latency": { "avgcount": 6410, "avgtime": 0.107654704, "sum": 690.066655152 }, "request": 6410, "root_rbytes": 0, "root_rfiles": 0, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 6412, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 6412, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 0, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 3201, "strays_enqueued": 3201, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 87, "evadd": 15805, "evex": 15723, "evexd": 15722, "evexg": 0, "evlrg": 0, "evtrm": 15723, "expos": 35096584, "jlat": { "avgcount": 25, "avgtime": 0.221292845, "sum": 5.532321132 }, "rdpos": 4195674, "replayed": 5, "seg": 1, "segadd": 19, "segex": 20, "segexd": 19, "segexg": 0, "segmjr": 1, "segtrm": 20, "wrpos": 35151943 }, "mds_mem": { "cap": 1, "cap+": 3202, "cap-": 3201, "dir": 13, "dir+": 13, "dir-": 0, "dn": 95, "dn+": 6412, "dn-": 6317, "heap": 248164, "ino": 97, "ino+": 3214, "ino-": 3117, "rss": 114632 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 6414, "dispatch_server_request": 0, "handle_client_request": 6410, "handle_client_session": 1, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.212308847, "sum": 679.388313371 }, "req_getattr_latency": { "avgcount": 2, "avgtime": 0.332508172, "sum": 0.665016344 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.00548828, "sum": 0.00548828 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 6, "avgtime": 0.173962127, "sum": 1.043772762 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 1, "avgtime": 0.588745986, "sum": 0.588745986 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 3200, "avgtime": 0.002617287, "sum": 8.375318409 } }, "mds_sessions": { "average_load": 5720, "avg_session_uptime": 941, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 5720 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 935168, "buffer_anon_items": 348, "buffer_meta_bytes": 176, "buffer_meta_items": 2, "mds_co_bytes": 337193, "mds_co_items": 6297, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5672, "osdmap_items": 92, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 11, "omap_rd": 48, "omap_wr": 47, "op": 3472, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 3472, "avgtime": 0.003907211, "sum": 13.565837947 }, "op_pg": 0, "op_r": 59, "op_reply": 3472, "op_resend": 0, "op_rmw": 0, "op_send": 3472, "op_send_bytes": 31836751, "op_w": 3413, "oplen_avg": { "avgcount": 3472, "sum": 3744 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 94, "osdop_delete": 3226, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 31, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 188, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 12, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 48, "osdop_writefull": 14, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 3201, "pq_executed_ops": 3201, "pq_executing": 0, "pq_executing_high_water": 6, "pq_executing_ops": 0, "pq_executing_ops_high_water": 6, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 16335, "get_or_fail_fail": 0, "get_or_fail_success": 16335, "get_started": 0, "get_sum": 4326421, "max": 104857600, "put": 16335, "put_sum": 4326421, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 3472, "put_sum": 32161973, "take": 3472, "take_sum": 32161973, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 3472, "put_sum": 3472, "take": 3472, "take_sum": 3472, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 3201, "get_or_fail_fail": 0, "get_or_fail_success": 3201, "get_started": 0, "get_sum": 323301, "max": 3758096384, "put": 23, "put_sum": 323301, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x5610c362caa0": { "get": 15805, "get_or_fail_fail": 0, "get_or_fail_success": 15805, "get_started": 0, "get_sum": 30955454, "max": 3758096384, "put": 25, "put_sum": 30955454, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-03-04T09:30:57.692 INFO:tasks.cephfs.test_strays:Complete purge in 7 seconds 2024-03-04T09:30:57.695 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:30:57.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.902+0000 7f4d6f200640 1 Processor -- start 2024-03-04T09:30:57.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.903+0000 7f4d6f200640 1 -- start start 2024-03-04T09:30:57.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.904+0000 7f4d6f200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f4d6815e1c0 0x7f4d6806cae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:57.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.904+0000 7f4d6f200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d680d1180 con 0x7f4d6815e5a0 2024-03-04T09:30:57.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.904+0000 7f4d6f200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d680d1460 con 0x7f4d680d0ca0 2024-03-04T09:30:57.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.904+0000 7f4d6f200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d680b17f0 con 0x7f4d6815e1c0 2024-03-04T09:30:57.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.904+0000 7f4d67e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f4d6815e1c0 0x7f4d6806cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:57.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.905+0000 7f4d67e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f4d6815e1c0 0x7f4d6806cae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41536/0 (socket says 172.21.15.6:41536) 2024-03-04T09:30:57.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.905+0000 7f4d67e00640 1 -- 172.21.15.6:0/40345563 learned_addr learned my addr 172.21.15.6:0/40345563 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:30:57.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.905+0000 7f4d67e00640 1 -- 172.21.15.6:0/40345563 >> v1:172.21.15.146:6789/0 conn(0x7f4d680d0ca0 legacy=0x7f4d680ce520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:57.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.905+0000 7f4d67e00640 1 -- 172.21.15.6:0/40345563 >> v1:172.21.15.6:6789/0 conn(0x7f4d6815e5a0 legacy=0x7f4d680cec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:57.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.905+0000 7f4d67e00640 1 -- 172.21.15.6:0/40345563 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d680b1a40 con 0x7f4d6815e1c0 2024-03-04T09:30:57.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.906+0000 7f4d67e00640 1 --2- 172.21.15.6:0/40345563 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4d6815e1c0 0x7f4d6806cae0 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f4d58009e50 tx=0x7f4d5802c010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=170c43cff03f636b server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:57.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.906+0000 7f4d66a00640 1 -- 172.21.15.6:0/40345563 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d5802cbb0 con 0x7f4d6815e1c0 2024-03-04T09:30:57.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.907+0000 7f4d66a00640 1 -- 172.21.15.6:0/40345563 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d5802cd50 con 0x7f4d6815e1c0 2024-03-04T09:30:57.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.907+0000 7f4d66a00640 1 -- 172.21.15.6:0/40345563 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d58034610 con 0x7f4d6815e1c0 2024-03-04T09:30:57.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.907+0000 7f4d6f200640 1 -- 172.21.15.6:0/40345563 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4d6815e1c0 msgr2=0x7f4d6806cae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:57.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.907+0000 7f4d6f200640 1 --2- 172.21.15.6:0/40345563 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4d6815e1c0 0x7f4d6806cae0 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f4d58009e50 tx=0x7f4d5802c010 comp rx=0 tx=0).stop 2024-03-04T09:30:57.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.907+0000 7f4d6f200640 1 -- 172.21.15.6:0/40345563 shutdown_connections 2024-03-04T09:30:57.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.907+0000 7f4d6f200640 1 --2- 172.21.15.6:0/40345563 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4d6815e1c0 0x7f4d6806cae0 unknown :-1 s=CLOSED pgs=1151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:57.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.907+0000 7f4d6f200640 1 -- 172.21.15.6:0/40345563 >> 172.21.15.6:0/40345563 conn(0x7f4d680d1eb0 msgr2=0x7f4d680d22c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:57.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.907+0000 7f4d6f200640 1 -- 172.21.15.6:0/40345563 shutdown_connections 2024-03-04T09:30:57.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.907+0000 7f4d6f200640 1 -- 172.21.15.6:0/40345563 wait complete. 2024-03-04T09:30:57.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.910+0000 7f4d6f200640 1 Processor -- start 2024-03-04T09:30:57.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.910+0000 7f4d6f200640 1 -- start start 2024-03-04T09:30:57.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.912+0000 7f4d6f200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4d680d0ca0 0x7f4d680b5710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:57.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.912+0000 7f4d6f200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4d6815e1c0 0x7f4d68160b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:57.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.912+0000 7f4d6f200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4d6815e5a0 0x7f4d68151a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:57.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.912+0000 7f4d6f200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d680d1c60 con 0x7f4d680d0ca0 2024-03-04T09:30:57.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.912+0000 7f4d6f200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d680d1ae0 con 0x7f4d6815e5a0 2024-03-04T09:30:57.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.912+0000 7f4d6f200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d680b3250 con 0x7f4d6815e1c0 2024-03-04T09:30:57.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.912+0000 7f4d67e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4d680d0ca0 0x7f4d680b5710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:57.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.912+0000 7f4d67e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4d680d0ca0 0x7f4d680b5710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41550/0 (socket says 172.21.15.6:41550) 2024-03-04T09:30:57.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.912+0000 7f4d67e00640 1 -- 172.21.15.6:0/1280164577 learned_addr learned my addr 172.21.15.6:0/1280164577 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:30:57.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.913+0000 7f4d67e00640 1 -- 172.21.15.6:0/1280164577 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4d6815e1c0 msgr2=0x7f4d68160b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:57.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.913+0000 7f4d67e00640 1 --2- 172.21.15.6:0/1280164577 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4d6815e1c0 0x7f4d68160b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:57.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.913+0000 7f4d67e00640 1 -- 172.21.15.6:0/1280164577 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4d6815e5a0 msgr2=0x7f4d68151a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:57.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.913+0000 7f4d6d200640 1 --2- 172.21.15.6:0/1280164577 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4d6815e5a0 0x7f4d68151a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:57.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.913+0000 7f4d67e00640 1 --2- 172.21.15.6:0/1280164577 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4d6815e5a0 0x7f4d68151a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:57.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.913+0000 7f4d67e00640 1 -- 172.21.15.6:0/1280164577 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d680b5e10 con 0x7f4d680d0ca0 2024-03-04T09:30:57.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.913+0000 7f4d6d200640 1 --2- 172.21.15.6:0/1280164577 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4d6815e5a0 0x7f4d68151a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:30:57.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.913+0000 7f4d67e00640 1 --2- 172.21.15.6:0/1280164577 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4d680d0ca0 0x7f4d680b5710 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f4d58009e50 tx=0x7f4d58004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:57.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.913+0000 7f4d5fe00640 1 -- 172.21.15.6:0/1280164577 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d58043020 con 0x7f4d680d0ca0 2024-03-04T09:30:57.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.913+0000 7f4d5fe00640 1 -- 172.21.15.6:0/1280164577 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d5803e450 con 0x7f4d680d0ca0 2024-03-04T09:30:57.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.913+0000 7f4d5fe00640 1 -- 172.21.15.6:0/1280164577 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d5803e840 con 0x7f4d680d0ca0 2024-03-04T09:30:57.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.913+0000 7f4d6f200640 1 -- 172.21.15.6:0/1280164577 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d680b60f0 con 0x7f4d680d0ca0 2024-03-04T09:30:57.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.914+0000 7f4d6f200640 1 -- 172.21.15.6:0/1280164577 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d680b66b0 con 0x7f4d680d0ca0 2024-03-04T09:30:57.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.915+0000 7f4d5d600640 1 -- 172.21.15.6:0/1280164577 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d3c005290 con 0x7f4d680d0ca0 2024-03-04T09:30:57.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.916+0000 7f4d5fe00640 1 -- 172.21.15.6:0/1280164577 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4d58046020 con 0x7f4d680d0ca0 2024-03-04T09:30:57.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.916+0000 7f4d5fe00640 1 --2- 172.21.15.6:0/1280164577 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f4d4806b4d0 0x7f4d4806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:57.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.917+0000 7f4d5fe00640 1 -- 172.21.15.6:0/1280164577 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f4d580accd0 con 0x7f4d680d0ca0 2024-03-04T09:30:57.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.917+0000 7f4d67400640 1 --2- 172.21.15.6:0/1280164577 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f4d4806b4d0 0x7f4d4806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:57.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.918+0000 7f4d67400640 1 --2- 172.21.15.6:0/1280164577 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f4d4806b4d0 0x7f4d4806d990 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f4d681616b0 tx=0x7f4d540079b0 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:57.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:57.921+0000 7f4d5fe00640 1 -- 172.21.15.6:0/1280164577 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4d5804d030 con 0x7f4d680d0ca0 2024-03-04T09:30:58.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:58.160+0000 7f4d5d600640 1 -- 172.21.15.6:0/1280164577 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4d3c005580 con 0x7f4d680d0ca0 2024-03-04T09:30:58.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:58.161+0000 7f4d5fe00640 1 -- 172.21.15.6:0/1280164577 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) v1 ==== 94+0+4977 (secure 0 0 0) 0x7f4d5807be90 con 0x7f4d680d0ca0 2024-03-04T09:30:58.158 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:30:58.159 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":68,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5654,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/1370302030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":1370302030},{"type":"v1","addr":"172.21.15.146:6835","nonce":1370302030}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":6163,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/809822004","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":809822004},{"type":"v1","addr":"172.21.15.146:6837","nonce":809822004}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":66},{"gid":7284,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/1583413840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1583413840},{"type":"v1","addr":"172.21.15.6:6833","nonce":1583413840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64}],"filesystems":[{"mdsmap":{"epoch":68,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:30:20.660103+0000","modified":"2024-03-04T09:30:36.289706+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":98,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7296},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7296":{"gid":7296,"name":"a","rank":0,"incarnation":65,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1732383836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1732383836},{"type":"v1","addr":"172.21.15.6:6835","nonce":1732383836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7296,"qdb_cluster":[7296]},"id":7}]} 2024-03-04T09:30:58.159 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 68 2024-03-04T09:30:58.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:58.165+0000 7f4d5d600640 1 -- 172.21.15.6:0/1280164577 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f4d4806b4d0 msgr2=0x7f4d4806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:58.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:58.165+0000 7f4d5d600640 1 --2- 172.21.15.6:0/1280164577 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f4d4806b4d0 0x7f4d4806d990 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f4d681616b0 tx=0x7f4d540079b0 comp rx=0 tx=0).stop 2024-03-04T09:30:58.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:58.165+0000 7f4d5d600640 1 -- 172.21.15.6:0/1280164577 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4d680d0ca0 msgr2=0x7f4d680b5710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:58.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:58.165+0000 7f4d5d600640 1 --2- 172.21.15.6:0/1280164577 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4d680d0ca0 0x7f4d680b5710 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f4d58009e50 tx=0x7f4d58004a40 comp rx=0 tx=0).stop 2024-03-04T09:30:58.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:58.165+0000 7f4d5d600640 1 -- 172.21.15.6:0/1280164577 shutdown_connections 2024-03-04T09:30:58.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:58.165+0000 7f4d5d600640 1 --2- 172.21.15.6:0/1280164577 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4d6815e5a0 0x7f4d68151a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:58.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:58.165+0000 7f4d5d600640 1 --2- 172.21.15.6:0/1280164577 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f4d4806b4d0 0x7f4d4806d990 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:58.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:58.165+0000 7f4d5d600640 1 --2- 172.21.15.6:0/1280164577 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4d6815e1c0 0x7f4d68160b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:58.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:58.165+0000 7f4d5d600640 1 --2- 172.21.15.6:0/1280164577 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4d680d0ca0 0x7f4d680b5710 unknown :-1 s=CLOSED pgs=1152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:58.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:58.165+0000 7f4d5d600640 1 -- 172.21.15.6:0/1280164577 >> 172.21.15.6:0/1280164577 conn(0x7f4d680d1eb0 msgr2=0x7f4d68161f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:58.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:58.165+0000 7f4d5d600640 1 -- 172.21.15.6:0/1280164577 shutdown_connections 2024-03-04T09:30:58.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:58.165+0000 7f4d5d600640 1 -- 172.21.15.6:0/1280164577 wait complete. 2024-03-04T09:30:58.180 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump 2024-03-04T09:30:58.399 INFO:teuthology.orchestra.run.smithi006.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":991,"msgr_send_messages":996,"msgr_recv_bytes":262460,"msgr_send_bytes":20393720,"msgr_created_connections":9,"msgr_active_connections":5,"msgr_running_total_time":0.436265730,"msgr_running_send_time":0.161484790,"msgr_running_recv_time":0.032763726,"msgr_running_fast_dispatch_time":0.228656341,"msgr_send_messages_queue_lat":{"avgcount":995,"sum":0.238099618,"avgtime":0.000239296},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000208,"avgtime":0.000000104},"msgr_recv_encrypted_bytes":1728,"msgr_send_encrypted_bytes":64832},"AsyncMessenger::Worker-1":{"msgr_recv_messages":14391,"msgr_send_messages":24923,"msgr_recv_bytes":5185660,"msgr_send_bytes":14533656,"msgr_created_connections":7,"msgr_active_connections":5,"msgr_running_total_time":3.242979214,"msgr_running_send_time":1.942150871,"msgr_running_recv_time":2.809516072,"msgr_running_fast_dispatch_time":0.231050848,"msgr_send_messages_queue_lat":{"avgcount":24921,"sum":1.687663654,"avgtime":0.000067720},"msgr_handle_ack_lat":{"avgcount":28800,"sum":0.012255169,"avgtime":0.000000425},"msgr_recv_encrypted_bytes":0,"msgr_send_encrypted_bytes":46336},"AsyncMessenger::Worker-2":{"msgr_recv_messages":959,"msgr_send_messages":955,"msgr_recv_bytes":351886,"msgr_send_bytes":4517244,"msgr_created_connections":5,"msgr_active_connections":4,"msgr_running_total_time":0.311061772,"msgr_running_send_time":0.061659273,"msgr_running_recv_time":0.033755739,"msgr_running_fast_dispatch_time":0.209843231,"msgr_send_messages_queue_lat":{"avgcount":954,"sum":0.124162436,"avgtime":0.000130149},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":106816,"msgr_send_encrypted_bytes":7920},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":133,"sum":12.112057031,"avgtime":0.091068097}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":5464,"sum":3.276418394,"avgtime":0.000599637}},"mds":{"request":6410,"reply":6410,"reply_latency":{"avgcount":6410,"sum":690.066655152,"avgtime":0.107654704},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":12,"dir_split":0,"dir_merge":0,"inodes":95,"inodes_top":0,"inodes_bottom":95,"inodes_pin_tail":0,"inodes_pinned":95,"inodes_expired":0,"inodes_with_caps":1,"caps":1,"subtrees":2,"traverse":6412,"traverse_hit":6412,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":0,"root_rbytes":0,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":105,"ceph_cap_op_grant":7450,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":5622,"handle_client_caps_dirty":3000,"handle_client_cap_release":781,"process_request_cap_release":3210},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":0,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":3201,"strays_enqueued":3201,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":15805,"evex":15723,"evtrm":15723,"ev":87,"evexg":0,"evexd":15722,"segadd":19,"segex":20,"segtrm":20,"seg":1,"segmjr":1,"segexg":0,"segexd":19,"expos":35096584,"wrpos":35151943,"rdpos":4195674,"jlat":{"avgcount":25,"sum":5.532321132,"avgtime":0.221292845},"replayed":5},"mds_mem":{"ino":97,"ino+":3214,"ino-":3117,"dir":13,"dir+":13,"dir-":0,"dn":95,"dn+":6412,"dn-":6317,"cap":1,"cap+":3202,"cap-":3201,"rss":114632,"heap":248164},"mds_server":{"dispatch_client_request":6414,"dispatch_server_request":0,"handle_client_request":6410,"handle_client_session":1,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":679.388313371,"avgtime":0.212308847},"req_getattr_latency":{"avgcount":2,"sum":0.665016344,"avgtime":0.332508172},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.005488280,"avgtime":0.005488280},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":6,"sum":1.043772762,"avgtime":0.173962127},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":1,"sum":0.588745986,"avgtime":0.588745986},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":3200,"sum":8.375318409,"avgtime":0.002617287},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":5720,"average_load":5720,"avg_session_uptime":941,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":935168,"buffer_anon_items":348,"buffer_meta_bytes":176,"buffer_meta_items":2,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5672,"osdmap_items":92,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":337193,"mds_co_items":6297,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":3473,"op_send_bytes":31836946,"op_resend":0,"op_reply":3473,"op_latency":{"avgcount":3473,"sum":13.568837911,"avgtime":0.003906950},"op_inflight":0,"oplen_avg":{"avgcount":3473,"sum":3746},"op":3473,"op_r":59,"op_w":3414,"op_rmw":0,"op_pg":0,"osdop_stat":12,"osdop_create":94,"osdop_read":31,"osdop_write":48,"osdop_writefull":14,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":3226,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":188,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":48,"omap_rd":48,"omap_del":12},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":6,"pq_executing":0,"pq_executing_high_water":6,"pq_executed_ops":3201,"pq_executed":3201,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":16337,"get_sum":4326951,"get_or_fail_fail":0,"get_or_fail_success":16337,"take":0,"take_sum":0,"put":16337,"put_sum":4326951,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":3473,"take_sum":32162168,"put":3473,"put_sum":32162168,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":3473,"take_sum":3473,"put":3473,"put_sum":3473,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":3201,"get_sum":323301,"get_or_fail_fail":0,"get_or_fail_success":3201,"take":0,"take_sum":0,"put":23,"put_sum":323301,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x5610c362caa0":{"val":0,"max":3758096384,"get_started":0,"get":15805,"get_sum":30955454,"get_or_fail_fail":0,"get_or_fail_success":15805,"take":0,"take_sum":0,"put":25,"put_sum":30955454,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-03-04T09:30:58.401 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 5, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.04e-07, "sum": 2.08e-07 }, "msgr_recv_bytes": 262460, "msgr_recv_encrypted_bytes": 1728, "msgr_recv_messages": 991, "msgr_running_fast_dispatch_time": 0.228656341, "msgr_running_recv_time": 0.032763726, "msgr_running_send_time": 0.16148479, "msgr_running_total_time": 0.43626573, "msgr_send_bytes": 20393720, "msgr_send_encrypted_bytes": 64832, "msgr_send_messages": 996, "msgr_send_messages_queue_lat": { "avgcount": 995, "avgtime": 0.000239296, "sum": 0.238099618 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 5, "msgr_created_connections": 7, "msgr_handle_ack_lat": { "avgcount": 28800, "avgtime": 4.25e-07, "sum": 0.012255169 }, "msgr_recv_bytes": 5185660, "msgr_recv_encrypted_bytes": 0, "msgr_recv_messages": 14391, "msgr_running_fast_dispatch_time": 0.231050848, "msgr_running_recv_time": 2.809516072, "msgr_running_send_time": 1.942150871, "msgr_running_total_time": 3.242979214, "msgr_send_bytes": 14533656, "msgr_send_encrypted_bytes": 46336, "msgr_send_messages": 24923, "msgr_send_messages_queue_lat": { "avgcount": 24921, "avgtime": 6.772e-05, "sum": 1.687663654 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 4, "msgr_created_connections": 5, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 351886, "msgr_recv_encrypted_bytes": 106816, "msgr_recv_messages": 959, "msgr_running_fast_dispatch_time": 0.209843231, "msgr_running_recv_time": 0.033755739, "msgr_running_send_time": 0.061659273, "msgr_running_total_time": 0.311061772, "msgr_send_bytes": 4517244, "msgr_send_encrypted_bytes": 7920, "msgr_send_messages": 955, "msgr_send_messages_queue_lat": { "avgcount": 954, "avgtime": 0.000130149, "sum": 0.124162436 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 133, "avgtime": 0.091068097, "sum": 12.112057031 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 5464, "avgtime": 0.000599637, "sum": 3.276418394 }, "queue_len": 0 }, "mds": { "caps": 1, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 7450, "ceph_cap_op_revoke": 105, "ceph_cap_op_trunc": 0, "dir_commit": 12, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 781, "handle_client_caps": 5622, "handle_client_caps_dirty": 3000, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 95, "inodes_bottom": 95, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 95, "inodes_top": 0, "inodes_with_caps": 1, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 3210, "q": 0, "reply": 6410, "reply_latency": { "avgcount": 6410, "avgtime": 0.107654704, "sum": 690.066655152 }, "request": 6410, "root_rbytes": 0, "root_rfiles": 0, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 6412, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 6412, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 0, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 3201, "strays_enqueued": 3201, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 87, "evadd": 15805, "evex": 15723, "evexd": 15722, "evexg": 0, "evlrg": 0, "evtrm": 15723, "expos": 35096584, "jlat": { "avgcount": 25, "avgtime": 0.221292845, "sum": 5.532321132 }, "rdpos": 4195674, "replayed": 5, "seg": 1, "segadd": 19, "segex": 20, "segexd": 19, "segexg": 0, "segmjr": 1, "segtrm": 20, "wrpos": 35151943 }, "mds_mem": { "cap": 1, "cap+": 3202, "cap-": 3201, "dir": 13, "dir+": 13, "dir-": 0, "dn": 95, "dn+": 6412, "dn-": 6317, "heap": 248164, "ino": 97, "ino+": 3214, "ino-": 3117, "rss": 114632 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 6414, "dispatch_server_request": 0, "handle_client_request": 6410, "handle_client_session": 1, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.212308847, "sum": 679.388313371 }, "req_getattr_latency": { "avgcount": 2, "avgtime": 0.332508172, "sum": 0.665016344 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.00548828, "sum": 0.00548828 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 6, "avgtime": 0.173962127, "sum": 1.043772762 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 1, "avgtime": 0.588745986, "sum": 0.588745986 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 3200, "avgtime": 0.002617287, "sum": 8.375318409 } }, "mds_sessions": { "average_load": 5720, "avg_session_uptime": 941, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 5720 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 935168, "buffer_anon_items": 348, "buffer_meta_bytes": 176, "buffer_meta_items": 2, "mds_co_bytes": 337193, "mds_co_items": 6297, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5672, "osdmap_items": 92, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 12, "omap_rd": 48, "omap_wr": 48, "op": 3473, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 3473, "avgtime": 0.00390695, "sum": 13.568837911 }, "op_pg": 0, "op_r": 59, "op_reply": 3473, "op_resend": 0, "op_rmw": 0, "op_send": 3473, "op_send_bytes": 31836946, "op_w": 3414, "oplen_avg": { "avgcount": 3473, "sum": 3746 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 94, "osdop_delete": 3226, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 31, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 188, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 12, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 48, "osdop_writefull": 14, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 3201, "pq_executed_ops": 3201, "pq_executing": 0, "pq_executing_high_water": 6, "pq_executing_ops": 0, "pq_executing_ops_high_water": 6, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 16337, "get_or_fail_fail": 0, "get_or_fail_success": 16337, "get_started": 0, "get_sum": 4326951, "max": 104857600, "put": 16337, "put_sum": 4326951, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 3473, "put_sum": 32162168, "take": 3473, "take_sum": 32162168, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 3473, "put_sum": 3473, "take": 3473, "take_sum": 3473, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 3201, "get_or_fail_fail": 0, "get_or_fail_success": 3201, "get_started": 0, "get_sum": 323301, "max": 3758096384, "put": 23, "put_sum": 323301, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x5610c362caa0": { "get": 15805, "get_or_fail_fail": 0, "get_or_fail_success": 15805, "get_started": 0, "get_sum": 30955454, "max": 3758096384, "put": 25, "put_sum": 30955454, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-03-04T09:30:58.401 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-04T09:30:58.401 DEBUG:teuthology.orchestra.run.smithi006:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-04T09:30:58.550 DEBUG:teuthology.orchestra.run.smithi146:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-04T09:30:58.671 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-04T09:30:58.671 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-04T09:30:58.671 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T09:30:58.732 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-04T09:30:58.732 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T09:30:58.787 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T09:30:58.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.000+0000 7f88c8c00640 1 Processor -- start 2024-03-04T09:30:58.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.000+0000 7f88c8c00640 1 -- start start 2024-03-04T09:30:58.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.002+0000 7f88c8c00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f88c40b0580 0x7f88c40b0960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:58.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.002+0000 7f88c8c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f88c4072ac0 0x7f88c406e130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:58.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.002+0000 7f88c8c00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88c406d3b0 con 0x7f88c40b5110 2024-03-04T09:30:58.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.002+0000 7f88c8c00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88c40b4350 con 0x7f88c4072ac0 2024-03-04T09:30:58.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.002+0000 7f88c8c00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88c40b4630 con 0x7f88c40b0580 2024-03-04T09:30:59.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.003+0000 7f88c3200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f88c4072ac0 0x7f88c406e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:59.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.003+0000 7f88c3200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f88c4072ac0 0x7f88c406e130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51456/0 (socket says 172.21.15.6:51456) 2024-03-04T09:30:59.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.003+0000 7f88c3200640 1 -- 172.21.15.6:0/3084077433 learned_addr learned my addr 172.21.15.6:0/3084077433 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:30:59.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.003+0000 7f88c1e00640 1 --2- 172.21.15.6:0/3084077433 >> v2:172.21.15.146:3300/0 conn(0x7f88c40b0580 0x7f88c40b0960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:59.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.003+0000 7f88c3200640 1 -- 172.21.15.6:0/3084077433 >> v2:172.21.15.146:3300/0 conn(0x7f88c40b0580 msgr2=0x7f88c40b0960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:59.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.003+0000 7f88c3200640 1 --2- 172.21.15.6:0/3084077433 >> v2:172.21.15.146:3300/0 conn(0x7f88c40b0580 0x7f88c40b0960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:59.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.003+0000 7f88c3200640 1 -- 172.21.15.6:0/3084077433 >> v1:172.21.15.146:6789/0 conn(0x7f88c40b5110 legacy=0x7f88c40723b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:59.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.003+0000 7f88c3200640 1 -- 172.21.15.6:0/3084077433 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88c40b4910 con 0x7f88c4072ac0 2024-03-04T09:30:59.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.005+0000 7f88c3200640 1 --2- 172.21.15.6:0/3084077433 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f88c4072ac0 0x7f88c406e130 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7f88c4073070 tx=0x7f88bc02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e78843e9a82e888 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:59.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.005+0000 7f88c0a00640 1 -- 172.21.15.6:0/3084077433 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88bc02ca80 con 0x7f88c4072ac0 2024-03-04T09:30:59.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.005+0000 7f88c0a00640 1 -- 172.21.15.6:0/3084077433 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88bc02cc20 con 0x7f88c4072ac0 2024-03-04T09:30:59.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.006+0000 7f88c0a00640 1 -- 172.21.15.6:0/3084077433 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88bc0343f0 con 0x7f88c4072ac0 2024-03-04T09:30:59.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.006+0000 7f88c8c00640 1 -- 172.21.15.6:0/3084077433 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f88c4072ac0 msgr2=0x7f88c406e130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:59.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.006+0000 7f88c8c00640 1 --2- 172.21.15.6:0/3084077433 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f88c4072ac0 0x7f88c406e130 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7f88c4073070 tx=0x7f88bc02bee0 comp rx=0 tx=0).stop 2024-03-04T09:30:59.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.006+0000 7f88c8c00640 1 -- 172.21.15.6:0/3084077433 shutdown_connections 2024-03-04T09:30:59.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.006+0000 7f88c8c00640 1 --2- 172.21.15.6:0/3084077433 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f88c4072ac0 0x7f88c406e130 unknown :-1 s=CLOSED pgs=1153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:59.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.006+0000 7f88c8c00640 1 --2- 172.21.15.6:0/3084077433 >> v2:172.21.15.146:3300/0 conn(0x7f88c40b0580 0x7f88c40b0960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:59.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.006+0000 7f88c8c00640 1 -- 172.21.15.6:0/3084077433 >> 172.21.15.6:0/3084077433 conn(0x7f88c409ac20 msgr2=0x7f88c40b3050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:59.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.006+0000 7f88c8c00640 1 -- 172.21.15.6:0/3084077433 shutdown_connections 2024-03-04T09:30:59.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.007+0000 7f88c8c00640 1 -- 172.21.15.6:0/3084077433 wait complete. 2024-03-04T09:30:59.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.008+0000 7f88c8c00640 1 Processor -- start 2024-03-04T09:30:59.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.008+0000 7f88c8c00640 1 -- start start 2024-03-04T09:30:59.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.009+0000 7f88c8c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f88c4072ac0 0x7f88c414e730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:59.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.009+0000 7f88c8c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f88c40b0580 0x7f88c409eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:59.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.009+0000 7f88c8c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f88c40b5110 0x7f88c409f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:59.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.009+0000 7f88c8c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88c40b5720 con 0x7f88c40b5110 2024-03-04T09:30:59.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.009+0000 7f88c8c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88c40b55a0 con 0x7f88c4072ac0 2024-03-04T09:30:59.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.009+0000 7f88c8c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88c40b58a0 con 0x7f88c40b0580 2024-03-04T09:30:59.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.010+0000 7f88c3200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f88c40b5110 0x7f88c409f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:59.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.010+0000 7f88c3200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f88c40b5110 0x7f88c409f0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51470/0 (socket says 172.21.15.6:51470) 2024-03-04T09:30:59.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.010+0000 7f88c3200640 1 -- 172.21.15.6:0/267871650 learned_addr learned my addr 172.21.15.6:0/267871650 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:30:59.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.010+0000 7f88c1e00640 1 --2- 172.21.15.6:0/267871650 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f88c4072ac0 0x7f88c414e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:59.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.010+0000 7f88c3200640 1 -- 172.21.15.6:0/267871650 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f88c40b0580 msgr2=0x7f88c409eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:59.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.010+0000 7f88c1400640 1 --2- 172.21.15.6:0/267871650 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f88c40b0580 0x7f88c409eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:59.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.010+0000 7f88c3200640 1 --2- 172.21.15.6:0/267871650 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f88c40b0580 0x7f88c409eb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:59.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.010+0000 7f88c3200640 1 -- 172.21.15.6:0/267871650 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f88c4072ac0 msgr2=0x7f88c414e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:59.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.010+0000 7f88c3200640 1 --2- 172.21.15.6:0/267871650 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f88c4072ac0 0x7f88c414e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:59.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.010+0000 7f88c3200640 1 -- 172.21.15.6:0/267871650 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88c414ec70 con 0x7f88c40b5110 2024-03-04T09:30:59.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.010+0000 7f88c1e00640 1 --2- 172.21.15.6:0/267871650 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f88c4072ac0 0x7f88c414e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:30:59.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.010+0000 7f88c3200640 1 --2- 172.21.15.6:0/267871650 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f88c40b5110 0x7f88c409f0a0 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7f88bc036040 tx=0x7f88bc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:59.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.011+0000 7f88ba000640 1 -- 172.21.15.6:0/267871650 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88bc043020 con 0x7f88c40b5110 2024-03-04T09:30:59.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.011+0000 7f88ba000640 1 -- 172.21.15.6:0/267871650 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88bc03e450 con 0x7f88c40b5110 2024-03-04T09:30:59.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.011+0000 7f88ba000640 1 -- 172.21.15.6:0/267871650 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88bc009510 con 0x7f88c40b5110 2024-03-04T09:30:59.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.011+0000 7f88c8c00640 1 -- 172.21.15.6:0/267871650 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88c4151280 con 0x7f88c40b5110 2024-03-04T09:30:59.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.011+0000 7f88c8c00640 1 -- 172.21.15.6:0/267871650 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88c4151840 con 0x7f88c40b5110 2024-03-04T09:30:59.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.012+0000 7f88c8c00640 1 -- 172.21.15.6:0/267871650 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88c40692a0 con 0x7f88c40b5110 2024-03-04T09:30:59.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.013+0000 7f88ba000640 1 -- 172.21.15.6:0/267871650 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f88bc046020 con 0x7f88c40b5110 2024-03-04T09:30:59.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.014+0000 7f88ba000640 1 --2- 172.21.15.6:0/267871650 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f88a406b4d0 0x7f88a406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:59.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.014+0000 7f88ba000640 1 -- 172.21.15.6:0/267871650 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f88bc0acba0 con 0x7f88c40b5110 2024-03-04T09:30:59.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.015+0000 7f88c1e00640 1 --2- 172.21.15.6:0/267871650 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f88a406b4d0 0x7f88a406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:59.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.016+0000 7f88c1e00640 1 --2- 172.21.15.6:0/267871650 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f88a406b4d0 0x7f88a406d990 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f88b4006fd0 tx=0x7f88b4008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:59.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.023+0000 7f88ba000640 1 -- 172.21.15.6:0/267871650 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f88bc04d030 con 0x7f88c40b5110 2024-03-04T09:30:59.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.223+0000 7f88c8c00640 1 -- 172.21.15.6:0/267871650 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f88c4074d40 con 0x7f88c40b5110 2024-03-04T09:30:59.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.225+0000 7f88ba000640 1 -- 172.21.15.6:0/267871650 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v98) v1 ==== 74+0+16817 (secure 0 0 0) 0x7f88bc07b9f0 con 0x7f88c40b5110 2024-03-04T09:30:59.223 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:30:59.223 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":98,"fsid":"edce0578-5ae4-4402-b45a-552bdef84d7c","created":"2024-03-04T09:24:54.523780+0000","modified":"2024-03-04T09:30:33.260648+0000","last_up_change":"2024-03-04T09:25:07.089049+0000","last_in_change":"2024-03-04T09:24:57.982077+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T09:25:07.935198+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-03-04T09:30:18.510767+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"93","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-03-04T09:30:19.686114+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"94","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a96574f-81f1-4bc6-8463-757520b84a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6825","nonce":1364449401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6827","nonce":1364449401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6831","nonce":1364449401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6829","nonce":1364449401}]},"public_addr":"172.21.15.6:6825/1364449401","cluster_addr":"172.21.15.6:6827/1364449401","heartbeat_back_addr":"172.21.15.6:6831/1364449401","heartbeat_front_addr":"172.21.15.6:6829/1364449401","state":["exists","up"]},{"osd":1,"uuid":"7d79b9c1-6b87-476a-bc36-7db5ee64cd90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6809","nonce":2494103078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6811","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6813","nonce":2494103078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6821","nonce":2494103078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6817","nonce":2494103078}]},"public_addr":"172.21.15.6:6809/2494103078","cluster_addr":"172.21.15.6:6813/2494103078","heartbeat_back_addr":"172.21.15.6:6821/2494103078","heartbeat_front_addr":"172.21.15.6:6817/2494103078","state":["exists","up"]},{"osd":2,"uuid":"492331b9-4ee9-40b6-bacd-85d44cd6c45a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6812","nonce":1500066681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6816","nonce":1500066681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6824","nonce":1500066681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6820","nonce":1500066681}]},"public_addr":"172.21.15.6:6812/1500066681","cluster_addr":"172.21.15.6:6816/1500066681","heartbeat_back_addr":"172.21.15.6:6824/1500066681","heartbeat_front_addr":"172.21.15.6:6820/1500066681","state":["exists","up"]},{"osd":3,"uuid":"21bb2efe-9ba7-4bdc-bc31-180cd1102e21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6801","nonce":1558806123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6803","nonce":1558806123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6807","nonce":1558806123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6805","nonce":1558806123}]},"public_addr":"172.21.15.6:6801/1558806123","cluster_addr":"172.21.15.6:6803/1558806123","heartbeat_back_addr":"172.21.15.6:6807/1558806123","heartbeat_front_addr":"172.21.15.6:6805/1558806123","state":["exists","up"]},{"osd":4,"uuid":"6e2c4b71-19d9-4737-83d9-558705d8644a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6801","nonce":1526135780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6803","nonce":1526135780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6807","nonce":1526135780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6805","nonce":1526135780}]},"public_addr":"172.21.15.146:6801/1526135780","cluster_addr":"172.21.15.146:6803/1526135780","heartbeat_back_addr":"172.21.15.146:6807/1526135780","heartbeat_front_addr":"172.21.15.146:6805/1526135780","state":["exists","up"]},{"osd":5,"uuid":"fb69c525-f1be-4b7d-ab32-6c29b0d6fddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6809","nonce":460262507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6811","nonce":460262507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6815","nonce":460262507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6813","nonce":460262507}]},"public_addr":"172.21.15.146:6809/460262507","cluster_addr":"172.21.15.146:6811/460262507","heartbeat_back_addr":"172.21.15.146:6815/460262507","heartbeat_front_addr":"172.21.15.146:6813/460262507","state":["exists","up"]},{"osd":6,"uuid":"c32e7fed-370d-4401-8ce0-6a4335b0d168","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6823","nonce":1392897493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6827","nonce":1392897493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6831","nonce":1392897493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6829","nonce":1392897493}]},"public_addr":"172.21.15.146:6823/1392897493","cluster_addr":"172.21.15.146:6827/1392897493","heartbeat_back_addr":"172.21.15.146:6831/1392897493","heartbeat_front_addr":"172.21.15.146:6829/1392897493","state":["exists","up"]},{"osd":7,"uuid":"75a79b44-fa5f-426d-a7d3-71f21f78835c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6817","nonce":3268323156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6819","nonce":3268323156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6825","nonce":3268323156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6822","nonce":3268323156}]},"public_addr":"172.21.15.146:6817/3268323156","cluster_addr":"172.21.15.146:6819/3268323156","heartbeat_back_addr":"172.21.15.146:6825/3268323156","heartbeat_front_addr":"172.21.15.146:6822/3268323156","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.310437+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.295850+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.294674+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.526855+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.300023+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.595160+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.346046+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.558741+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.146:6835/847815316":"2024-03-05T09:30:33.260598+0000","172.21.15.146:6834/847815316":"2024-03-05T09:30:33.260598+0000","172.21.15.6:6835/2836916196":"2024-03-05T09:30:32.267572+0000","172.21.15.6:6832/2836916196":"2024-03-05T09:30:32.267572+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T09:30:59.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.228+0000 7f88c8c00640 1 -- 172.21.15.6:0/267871650 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f88a406b4d0 msgr2=0x7f88a406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:59.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.228+0000 7f88c8c00640 1 --2- 172.21.15.6:0/267871650 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f88a406b4d0 0x7f88a406d990 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f88b4006fd0 tx=0x7f88b4008040 comp rx=0 tx=0).stop 2024-03-04T09:30:59.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.228+0000 7f88c8c00640 1 -- 172.21.15.6:0/267871650 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f88c40b5110 msgr2=0x7f88c409f0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:59.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.228+0000 7f88c8c00640 1 --2- 172.21.15.6:0/267871650 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f88c40b5110 0x7f88c409f0a0 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7f88bc036040 tx=0x7f88bc004a40 comp rx=0 tx=0).stop 2024-03-04T09:30:59.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.228+0000 7f88c8c00640 1 -- 172.21.15.6:0/267871650 shutdown_connections 2024-03-04T09:30:59.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.228+0000 7f88c8c00640 1 --2- 172.21.15.6:0/267871650 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f88c40b5110 0x7f88c409f0a0 unknown :-1 s=CLOSED pgs=1154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:59.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.228+0000 7f88c8c00640 1 --2- 172.21.15.6:0/267871650 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f88a406b4d0 0x7f88a406d990 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:59.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.228+0000 7f88c8c00640 1 --2- 172.21.15.6:0/267871650 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f88c40b0580 0x7f88c409eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:59.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.228+0000 7f88c8c00640 1 --2- 172.21.15.6:0/267871650 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f88c4072ac0 0x7f88c414e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:59.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.228+0000 7f88c8c00640 1 -- 172.21.15.6:0/267871650 >> 172.21.15.6:0/267871650 conn(0x7f88c409ac20 msgr2=0x7f88c406cb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:59.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.228+0000 7f88c8c00640 1 -- 172.21.15.6:0/267871650 shutdown_connections 2024-03-04T09:30:59.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.228+0000 7f88c8c00640 1 -- 172.21.15.6:0/267871650 wait complete. 2024-03-04T09:30:59.245 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T09:25:07.935198+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-04T09:30:18.510767+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '93', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_data', 'create_time': '2024-03-04T09:30:19.686114+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '94', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-04T09:30:59.246 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-04T09:30:59.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.464+0000 7f7f33400640 1 Processor -- start 2024-03-04T09:30:59.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.465+0000 7f7f33400640 1 -- start start 2024-03-04T09:30:59.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.466+0000 7f7f33400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f7f2c13df20 0x7f7f2c13e640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:59.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.466+0000 7f7f33400640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f2c13e300 con 0x7f7f2c072700 2024-03-04T09:30:59.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.466+0000 7f7f33400640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f2c15be80 con 0x7f7f2c13eb80 2024-03-04T09:30:59.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.466+0000 7f7f33400640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f2c0ce780 con 0x7f7f2c13df20 2024-03-04T09:30:59.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.467+0000 7f7f31400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f7f2c072700 0x7f7f2c0cf1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:35264/0 (socket says 172.21.15.6:35264) 2024-03-04T09:30:59.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.467+0000 7f7f31400640 1 -- 172.21.15.6:0/1068566806 learned_addr learned my addr 172.21.15.6:0/1068566806 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:30:59.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.467+0000 7f7f30a00640 1 --2- 172.21.15.6:0/1068566806 >> v2:172.21.15.146:3300/0 conn(0x7f7f2c13df20 0x7f7f2c13e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:59.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.467+0000 7f7f2b400640 1 -- 172.21.15.6:0/1068566806 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1407560422 0 0) 0x7f7f2c13e300 con 0x7f7f2c072700 2024-03-04T09:30:59.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.468+0000 7f7f2b400640 1 -- 172.21.15.6:0/1068566806 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f10005730 con 0x7f7f2c072700 2024-03-04T09:30:59.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.468+0000 7f7f2b400640 1 -- 172.21.15.6:0/1068566806 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3680500284 0 0) 0x7f7f10005730 con 0x7f7f2c072700 2024-03-04T09:30:59.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.468+0000 7f7f2b400640 1 -- 172.21.15.6:0/1068566806 >> v2:172.21.15.146:3300/0 conn(0x7f7f2c13df20 msgr2=0x7f7f2c13e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:59.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.468+0000 7f7f2b400640 1 --2- 172.21.15.6:0/1068566806 >> v2:172.21.15.146:3300/0 conn(0x7f7f2c13df20 0x7f7f2c13e640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:59.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.468+0000 7f7f2b400640 1 -- 172.21.15.6:0/1068566806 >> v1:172.21.15.146:6789/0 conn(0x7f7f2c13eb80 legacy=0x7f7f2c15b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:59.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.468+0000 7f7f2b400640 1 -- 172.21.15.6:0/1068566806 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f2c15c160 con 0x7f7f2c072700 2024-03-04T09:30:59.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.468+0000 7f7f2b400640 1 -- 172.21.15.6:0/1068566806 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7f7f240019f0 con 0x7f7f2c072700 2024-03-04T09:30:59.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.468+0000 7f7f2b400640 1 -- 172.21.15.6:0/1068566806 >> v1:172.21.15.6:6789/0 conn(0x7f7f2c072700 legacy=0x7f7f2c0cf1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:59.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.468+0000 7f7f2b400640 1 --2- 172.21.15.6:0/1068566806 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7f10005c10 0x7f7f10006030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:59.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.468+0000 7f7f2b400640 1 --2- 172.21.15.6:0/1068566806 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7f10006a10 0x7f7f10006e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:59.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.468+0000 7f7f2b400640 1 --2- 172.21.15.6:0/1068566806 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7f100073d0 0x7f7f1000dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:59.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.468+0000 7f7f2b400640 1 -- 172.21.15.6:0/1068566806 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f1000e2a0 con 0x7f7f10006a10 2024-03-04T09:30:59.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.468+0000 7f7f2b400640 1 -- 172.21.15.6:0/1068566806 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f1000e450 con 0x7f7f100073d0 2024-03-04T09:30:59.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.468+0000 7f7f2b400640 1 -- 172.21.15.6:0/1068566806 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f1000e650 con 0x7f7f10005c10 2024-03-04T09:30:59.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.468+0000 7f7f2be00640 1 --2- 172.21.15.6:0/1068566806 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7f10006a10 0x7f7f10006e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:59.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.469+0000 7f7f31400640 1 --2- 172.21.15.6:0/1068566806 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7f100073d0 0x7f7f1000dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:59.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.469+0000 7f7f30a00640 1 --2- 172.21.15.6:0/1068566806 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7f10005c10 0x7f7f10006030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:59.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.469+0000 7f7f2be00640 1 -- 172.21.15.6:0/1068566806 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7f10005c10 msgr2=0x7f7f10006030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:59.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.469+0000 7f7f2be00640 1 --2- 172.21.15.6:0/1068566806 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7f10005c10 0x7f7f10006030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:59.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.469+0000 7f7f2be00640 1 -- 172.21.15.6:0/1068566806 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7f100073d0 msgr2=0x7f7f1000dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:59.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.469+0000 7f7f2be00640 1 --2- 172.21.15.6:0/1068566806 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7f100073d0 0x7f7f1000dbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:59.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.469+0000 7f7f2be00640 1 -- 172.21.15.6:0/1068566806 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f10005730 con 0x7f7f10006a10 2024-03-04T09:30:59.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.469+0000 7f7f31400640 1 -- 172.21.15.6:0/1068566806 reap_dead start 2024-03-04T09:30:59.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.469+0000 7f7f30a00640 1 --2- 172.21.15.6:0/1068566806 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7f10005c10 0x7f7f10006030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:30:59.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.470+0000 7f7f2be00640 1 --2- 172.21.15.6:0/1068566806 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7f10006a10 0x7f7f10006e90 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7f7f14001720 tx=0x7f7f10007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9eddc041e57dc794 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:59.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.470+0000 7f7f2b400640 1 -- 172.21.15.6:0/1068566806 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f14001370 con 0x7f7f10006a10 2024-03-04T09:30:59.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.470+0000 7f7f2b400640 1 -- 172.21.15.6:0/1068566806 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f140052d0 con 0x7f7f10006a10 2024-03-04T09:30:59.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.470+0000 7f7f2b400640 1 -- 172.21.15.6:0/1068566806 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f14003830 con 0x7f7f10006a10 2024-03-04T09:30:59.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.471+0000 7f7f33400640 1 -- 172.21.15.6:0/1068566806 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7f10006a10 msgr2=0x7f7f10006e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:59.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.471+0000 7f7f33400640 1 --2- 172.21.15.6:0/1068566806 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7f10006a10 0x7f7f10006e90 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7f7f14001720 tx=0x7f7f10007a40 comp rx=0 tx=0).stop 2024-03-04T09:30:59.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.471+0000 7f7f33400640 1 -- 172.21.15.6:0/1068566806 shutdown_connections 2024-03-04T09:30:59.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.471+0000 7f7f33400640 1 --2- 172.21.15.6:0/1068566806 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7f10006a10 0x7f7f10006e90 unknown :-1 s=CLOSED pgs=1156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:59.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.471+0000 7f7f33400640 1 -- 172.21.15.6:0/1068566806 >> 172.21.15.6:0/1068566806 conn(0x7f7f2c06cfb0 msgr2=0x7f7f2c06d390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:59.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.471+0000 7f7f33400640 1 -- 172.21.15.6:0/1068566806 shutdown_connections 2024-03-04T09:30:59.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.471+0000 7f7f33400640 1 -- 172.21.15.6:0/1068566806 wait complete. 2024-03-04T09:30:59.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.473+0000 7f7f33400640 1 Processor -- start 2024-03-04T09:30:59.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.473+0000 7f7f33400640 1 -- start start 2024-03-04T09:30:59.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.474+0000 7f7f33400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7f10006a10 0x7f7f2c139620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:59.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.474+0000 7f7f33400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7f2c0b4590 0x7f7f2c13a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:59.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.474+0000 7f7f33400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7f2c13a7b0 0x7f7f2c140b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:59.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.474+0000 7f7f33400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f2c0cec90 con 0x7f7f2c13a7b0 2024-03-04T09:30:59.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.474+0000 7f7f33400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f2c15c440 con 0x7f7f10006a10 2024-03-04T09:30:59.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.474+0000 7f7f33400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f2c070e80 con 0x7f7f2c0b4590 2024-03-04T09:30:59.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.474+0000 7f7f31400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7f2c13a7b0 0x7f7f2c140b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:59.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.474+0000 7f7f31400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7f2c13a7b0 0x7f7f2c140b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51486/0 (socket says 172.21.15.6:51486) 2024-03-04T09:30:59.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.474+0000 7f7f31400640 1 -- 172.21.15.6:0/335164340 learned_addr learned my addr 172.21.15.6:0/335164340 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:30:59.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.475+0000 7f7f30a00640 1 --2- 172.21.15.6:0/335164340 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7f10006a10 0x7f7f2c139620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:59.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.475+0000 7f7f2be00640 1 --2- 172.21.15.6:0/335164340 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7f2c0b4590 0x7f7f2c13a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:59.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.475+0000 7f7f31400640 1 -- 172.21.15.6:0/335164340 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7f2c0b4590 msgr2=0x7f7f2c13a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:59.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.475+0000 7f7f31400640 1 --2- 172.21.15.6:0/335164340 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7f2c0b4590 0x7f7f2c13a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:59.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.475+0000 7f7f31400640 1 -- 172.21.15.6:0/335164340 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7f10006a10 msgr2=0x7f7f2c139620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:59.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.475+0000 7f7f31400640 1 --2- 172.21.15.6:0/335164340 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7f10006a10 0x7f7f2c139620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:59.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.475+0000 7f7f31400640 1 -- 172.21.15.6:0/335164340 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f2c13b100 con 0x7f7f2c13a7b0 2024-03-04T09:30:59.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.475+0000 7f7f31400640 1 --2- 172.21.15.6:0/335164340 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7f2c13a7b0 0x7f7f2c140b50 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7f7f2c0af9b0 tx=0x7f7f24007bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:59.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.476+0000 7f7f28c00640 1 -- 172.21.15.6:0/335164340 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f2400eb00 con 0x7f7f2c13a7b0 2024-03-04T09:30:59.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.476+0000 7f7f28c00640 1 -- 172.21.15.6:0/335164340 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f24018070 con 0x7f7f2c13a7b0 2024-03-04T09:30:59.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.476+0000 7f7f33400640 1 -- 172.21.15.6:0/335164340 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f2c13b3e0 con 0x7f7f2c13a7b0 2024-03-04T09:30:59.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.476+0000 7f7f28c00640 1 -- 172.21.15.6:0/335164340 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f24013420 con 0x7f7f2c13a7b0 2024-03-04T09:30:59.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.476+0000 7f7f33400640 1 -- 172.21.15.6:0/335164340 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f2c13b9a0 con 0x7f7f2c13a7b0 2024-03-04T09:30:59.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.477+0000 7f7f33400640 1 -- 172.21.15.6:0/335164340 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f2c0692a0 con 0x7f7f2c13a7b0 2024-03-04T09:30:59.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.478+0000 7f7f28c00640 1 -- 172.21.15.6:0/335164340 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7f240135c0 con 0x7f7f2c13a7b0 2024-03-04T09:30:59.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.479+0000 7f7f28c00640 1 --2- 172.21.15.6:0/335164340 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f7f0806b4d0 0x7f7f0806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:59.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.480+0000 7f7f28c00640 1 -- 172.21.15.6:0/335164340 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f7f2408a640 con 0x7f7f2c13a7b0 2024-03-04T09:30:59.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.480+0000 7f7f30a00640 1 --2- 172.21.15.6:0/335164340 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f7f0806b4d0 0x7f7f0806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:59.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.481+0000 7f7f30a00640 1 --2- 172.21.15.6:0/335164340 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f7f0806b4d0 0x7f7f0806d990 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f7f2c050b10 tx=0x7f7f1c008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:59.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.484+0000 7f7f28c00640 1 -- 172.21.15.6:0/335164340 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7f24059370 con 0x7f7f2c13a7b0 2024-03-04T09:30:59.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.686+0000 7f7f33400640 1 -- 172.21.15.6:0/335164340 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7f2c0b61a0 con 0x7f7f2c13a7b0 2024-03-04T09:30:59.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.687+0000 7f7f28c00640 1 -- 172.21.15.6:0/335164340 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v98) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7f24058c70 con 0x7f7f2c13a7b0 2024-03-04T09:30:59.684 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-04T09:30:59.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.690+0000 7f7f33400640 1 -- 172.21.15.6:0/335164340 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f7f0806b4d0 msgr2=0x7f7f0806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:59.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.690+0000 7f7f33400640 1 --2- 172.21.15.6:0/335164340 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f7f0806b4d0 0x7f7f0806d990 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f7f2c050b10 tx=0x7f7f1c008040 comp rx=0 tx=0).stop 2024-03-04T09:30:59.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.690+0000 7f7f33400640 1 -- 172.21.15.6:0/335164340 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7f2c13a7b0 msgr2=0x7f7f2c140b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:59.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.690+0000 7f7f33400640 1 --2- 172.21.15.6:0/335164340 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7f2c13a7b0 0x7f7f2c140b50 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7f7f2c0af9b0 tx=0x7f7f24007bc0 comp rx=0 tx=0).stop 2024-03-04T09:30:59.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.690+0000 7f7f33400640 1 -- 172.21.15.6:0/335164340 shutdown_connections 2024-03-04T09:30:59.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.690+0000 7f7f33400640 1 --2- 172.21.15.6:0/335164340 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7f2c13a7b0 0x7f7f2c140b50 unknown :-1 s=CLOSED pgs=1157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:59.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.690+0000 7f7f33400640 1 --2- 172.21.15.6:0/335164340 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f7f0806b4d0 0x7f7f0806d990 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:59.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.690+0000 7f7f33400640 1 --2- 172.21.15.6:0/335164340 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7f2c0b4590 0x7f7f2c13a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:59.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.690+0000 7f7f33400640 1 --2- 172.21.15.6:0/335164340 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7f10006a10 0x7f7f2c139620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:59.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.690+0000 7f7f33400640 1 -- 172.21.15.6:0/335164340 >> 172.21.15.6:0/335164340 conn(0x7f7f2c06cfb0 msgr2=0x7f7f2c15daf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:59.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.690+0000 7f7f33400640 1 -- 172.21.15.6:0/335164340 shutdown_connections 2024-03-04T09:30:59.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.690+0000 7f7f33400640 1 -- 172.21.15.6:0/335164340 wait complete. 2024-03-04T09:30:59.708 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-04T09:30:59.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.924+0000 7f577b200640 1 Processor -- start 2024-03-04T09:30:59.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.924+0000 7f577b200640 1 -- start start 2024-03-04T09:30:59.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.926+0000 7f577b200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f577c147850 0x7f577c072990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:59.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.926+0000 7f577b200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f577c0b4cc0 0x7f577c0b3b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:59.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.926+0000 7f577b200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f577c073070 con 0x7f577c0b4080 2024-03-04T09:30:59.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.926+0000 7f577b200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f577c0b3060 con 0x7f577c147850 2024-03-04T09:30:59.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.926+0000 7f577b200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f577c0b3340 con 0x7f577c0b4cc0 2024-03-04T09:30:59.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.926+0000 7f5779e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f577c147850 0x7f577c072990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:59.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.926+0000 7f5779e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f577c147850 0x7f577c072990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51502/0 (socket says 172.21.15.6:51502) 2024-03-04T09:30:59.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.926+0000 7f5779e00640 1 -- 172.21.15.6:0/78870059 learned_addr learned my addr 172.21.15.6:0/78870059 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:30:59.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.927+0000 7f5779400640 1 --2- 172.21.15.6:0/78870059 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f577c0b4cc0 0x7f577c0b3b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:59.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.927+0000 7f5779e00640 1 -- 172.21.15.6:0/78870059 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f577c0b4cc0 msgr2=0x7f577c0b3b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:59.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.927+0000 7f5779e00640 1 --2- 172.21.15.6:0/78870059 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f577c0b4cc0 0x7f577c0b3b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:59.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.927+0000 7f5779e00640 1 -- 172.21.15.6:0/78870059 >> v1:172.21.15.146:6789/0 conn(0x7f577c0b4080 legacy=0x7f577c06d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:30:59.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.927+0000 7f5779e00640 1 -- 172.21.15.6:0/78870059 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f577c0b3620 con 0x7f577c147850 2024-03-04T09:30:59.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.928+0000 7f5779e00640 1 --2- 172.21.15.6:0/78870059 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f577c147850 0x7f577c072990 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7f576c009870 tx=0x7f576c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75c3ccd7bc6d7f7b server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:59.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.928+0000 7f5778a00640 1 -- 172.21.15.6:0/78870059 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f576c035020 con 0x7f577c147850 2024-03-04T09:30:59.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.928+0000 7f5778a00640 1 -- 172.21.15.6:0/78870059 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f576c031640 con 0x7f577c147850 2024-03-04T09:30:59.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.929+0000 7f5778a00640 1 -- 172.21.15.6:0/78870059 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f576c02d040 con 0x7f577c147850 2024-03-04T09:30:59.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.929+0000 7f577b200640 1 -- 172.21.15.6:0/78870059 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f577c147850 msgr2=0x7f577c072990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:59.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.929+0000 7f577b200640 1 --2- 172.21.15.6:0/78870059 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f577c147850 0x7f577c072990 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7f576c009870 tx=0x7f576c02b8f0 comp rx=0 tx=0).stop 2024-03-04T09:30:59.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.929+0000 7f577b200640 1 -- 172.21.15.6:0/78870059 shutdown_connections 2024-03-04T09:30:59.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.929+0000 7f577b200640 1 --2- 172.21.15.6:0/78870059 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f577c0b4cc0 0x7f577c0b3b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:59.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.929+0000 7f577b200640 1 --2- 172.21.15.6:0/78870059 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f577c147850 0x7f577c072990 unknown :-1 s=CLOSED pgs=1158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:59.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.929+0000 7f577b200640 1 -- 172.21.15.6:0/78870059 >> 172.21.15.6:0/78870059 conn(0x7f577c06b3e0 msgr2=0x7f577c06b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:30:59.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.929+0000 7f577b200640 1 -- 172.21.15.6:0/78870059 shutdown_connections 2024-03-04T09:30:59.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.929+0000 7f577b200640 1 -- 172.21.15.6:0/78870059 wait complete. 2024-03-04T09:30:59.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.931+0000 7f577b200640 1 Processor -- start 2024-03-04T09:30:59.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.931+0000 7f577b200640 1 -- start start 2024-03-04T09:30:59.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.932+0000 7f577b200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f577c0b4080 0x7f577c13fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:59.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.932+0000 7f577b200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f577c0b4cc0 0x7f577c13d160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:59.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.932+0000 7f577b200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f577c147850 0x7f577c13e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:59.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.932+0000 7f577b200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f577c0b5280 con 0x7f577c0b4080 2024-03-04T09:30:59.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.932+0000 7f577b200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f577c0b5100 con 0x7f577c0b4cc0 2024-03-04T09:30:59.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.932+0000 7f577b200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f577c06dae0 con 0x7f577c147850 2024-03-04T09:30:59.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.933+0000 7f5779e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f577c0b4080 0x7f577c13fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:59.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.933+0000 7f5779e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f577c0b4080 0x7f577c13fdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51504/0 (socket says 172.21.15.6:51504) 2024-03-04T09:30:59.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.933+0000 7f5779e00640 1 -- 172.21.15.6:0/1784613972 learned_addr learned my addr 172.21.15.6:0/1784613972 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:30:59.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.933+0000 7f5779400640 1 --2- 172.21.15.6:0/1784613972 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f577c0b4cc0 0x7f577c13d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:59.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.933+0000 7f577a800640 1 --2- 172.21.15.6:0/1784613972 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f577c147850 0x7f577c13e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:59.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.933+0000 7f5779e00640 1 -- 172.21.15.6:0/1784613972 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f577c147850 msgr2=0x7f577c13e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:59.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.933+0000 7f5779e00640 1 --2- 172.21.15.6:0/1784613972 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f577c147850 0x7f577c13e340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:59.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.933+0000 7f5779e00640 1 -- 172.21.15.6:0/1784613972 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f577c0b4cc0 msgr2=0x7f577c13d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:30:59.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.933+0000 7f5779e00640 1 --2- 172.21.15.6:0/1784613972 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f577c0b4cc0 0x7f577c13d160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:30:59.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.933+0000 7f5779e00640 1 -- 172.21.15.6:0/1784613972 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f577c13ed20 con 0x7f577c0b4080 2024-03-04T09:30:59.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.933+0000 7f577a800640 1 --2- 172.21.15.6:0/1784613972 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f577c147850 0x7f577c13e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:30:59.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.933+0000 7f5779e00640 1 --2- 172.21.15.6:0/1784613972 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f577c0b4080 0x7f577c13fdc0 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f576c000c00 tx=0x7f576c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:59.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.934+0000 7f5772000640 1 -- 172.21.15.6:0/1784613972 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f576c040070 con 0x7f577c0b4080 2024-03-04T09:30:59.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.934+0000 7f5772000640 1 -- 172.21.15.6:0/1784613972 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f576c0399f0 con 0x7f577c0b4080 2024-03-04T09:30:59.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.934+0000 7f5772000640 1 -- 172.21.15.6:0/1784613972 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f576c035040 con 0x7f577c0b4080 2024-03-04T09:30:59.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.934+0000 7f577b200640 1 -- 172.21.15.6:0/1784613972 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f577c13f000 con 0x7f577c0b4080 2024-03-04T09:30:59.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.934+0000 7f577b200640 1 -- 172.21.15.6:0/1784613972 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f577c13f5c0 con 0x7f577c0b4080 2024-03-04T09:30:59.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.935+0000 7f577b200640 1 -- 172.21.15.6:0/1784613972 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f577c13fa00 con 0x7f577c0b4080 2024-03-04T09:30:59.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.936+0000 7f5772000640 1 -- 172.21.15.6:0/1784613972 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f576c04b060 con 0x7f577c0b4080 2024-03-04T09:30:59.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.937+0000 7f5772000640 1 --2- 172.21.15.6:0/1784613972 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f575c06b4d0 0x7f575c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:30:59.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.937+0000 7f5772000640 1 -- 172.21.15.6:0/1784613972 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f576c0abba0 con 0x7f577c0b4080 2024-03-04T09:30:59.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.938+0000 7f5779400640 1 --2- 172.21.15.6:0/1784613972 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f575c06b4d0 0x7f575c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:30:59.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.939+0000 7f5779400640 1 --2- 172.21.15.6:0/1784613972 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f575c06b4d0 0x7f575c06d990 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f577c0ae9e0 tx=0x7f5768008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:30:59.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:30:59.942+0000 7f5772000640 1 -- 172.21.15.6:0/1784613972 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f576c07a970 con 0x7f577c0b4080 2024-03-04T09:31:00.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.140+0000 7f577b200640 1 -- 172.21.15.6:0/1784613972 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f577c04f330 con 0x7f577c0b4080 2024-03-04T09:31:00.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.141+0000 7f5772000640 1 -- 172.21.15.6:0/1784613972 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v98) v1 ==== 104+0+11 (secure 0 0 0) 0x7f576c07a270 con 0x7f577c0b4080 2024-03-04T09:31:00.138 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-03-04T09:31:00.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.144+0000 7f577b200640 1 -- 172.21.15.6:0/1784613972 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f575c06b4d0 msgr2=0x7f575c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:00.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.144+0000 7f577b200640 1 --2- 172.21.15.6:0/1784613972 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f575c06b4d0 0x7f575c06d990 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f577c0ae9e0 tx=0x7f5768008040 comp rx=0 tx=0).stop 2024-03-04T09:31:00.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.144+0000 7f577b200640 1 -- 172.21.15.6:0/1784613972 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f577c0b4080 msgr2=0x7f577c13fdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:00.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.144+0000 7f577b200640 1 --2- 172.21.15.6:0/1784613972 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f577c0b4080 0x7f577c13fdc0 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f576c000c00 tx=0x7f576c0046c0 comp rx=0 tx=0).stop 2024-03-04T09:31:00.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.144+0000 7f577b200640 1 -- 172.21.15.6:0/1784613972 shutdown_connections 2024-03-04T09:31:00.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.144+0000 7f577b200640 1 --2- 172.21.15.6:0/1784613972 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f575c06b4d0 0x7f575c06d990 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:00.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.144+0000 7f577b200640 1 --2- 172.21.15.6:0/1784613972 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f577c147850 0x7f577c13e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:00.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.144+0000 7f577b200640 1 --2- 172.21.15.6:0/1784613972 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f577c0b4cc0 0x7f577c13d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:00.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.144+0000 7f577b200640 1 --2- 172.21.15.6:0/1784613972 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f577c0b4080 0x7f577c13fdc0 unknown :-1 s=CLOSED pgs=1159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:00.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.144+0000 7f577b200640 1 -- 172.21.15.6:0/1784613972 >> 172.21.15.6:0/1784613972 conn(0x7f577c06b3e0 msgr2=0x7f577c0afb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:00.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.144+0000 7f577b200640 1 -- 172.21.15.6:0/1784613972 shutdown_connections 2024-03-04T09:31:00.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.144+0000 7f577b200640 1 -- 172.21.15.6:0/1784613972 wait complete. 2024-03-04T09:31:00.160 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-04T09:31:00.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.377+0000 7f24bfe00640 1 Processor -- start 2024-03-04T09:31:00.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.377+0000 7f24bfe00640 1 -- start start 2024-03-04T09:31:00.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.379+0000 7f24bfe00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f24c00cf640 0x7f24c00b40c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:00.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.379+0000 7f24bfe00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24c015b110 con 0x7f24c00af5f0 2024-03-04T09:31:00.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.379+0000 7f24bfe00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24c00b4e90 con 0x7f24c015e6f0 2024-03-04T09:31:00.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.379+0000 7f24bfe00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24c015e260 con 0x7f24c00cf640 2024-03-04T09:31:00.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.380+0000 7f24bea00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f24c00af5f0 0x7f24c015a620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:35274/0 (socket says 172.21.15.6:35274) 2024-03-04T09:31:00.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.380+0000 7f24bea00640 1 -- 172.21.15.6:0/1636136482 learned_addr learned my addr 172.21.15.6:0/1636136482 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:31:00.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.380+0000 7f24bf400640 1 --2- 172.21.15.6:0/1636136482 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f24c00cf640 0x7f24c00b40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:00.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.380+0000 7f24bd600640 1 -- 172.21.15.6:0/1636136482 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1478599033 0 0) 0x7f24c015b110 con 0x7f24c00af5f0 2024-03-04T09:31:00.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.380+0000 7f24bd600640 1 -- 172.21.15.6:0/1636136482 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24a4005730 con 0x7f24c00af5f0 2024-03-04T09:31:00.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.381+0000 7f24bd600640 1 -- 172.21.15.6:0/1636136482 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2052361259 0 0) 0x7f24a4005730 con 0x7f24c00af5f0 2024-03-04T09:31:00.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.381+0000 7f24bd600640 1 -- 172.21.15.6:0/1636136482 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f24c00cf640 msgr2=0x7f24c00b40c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:00.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.381+0000 7f24bd600640 1 --2- 172.21.15.6:0/1636136482 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f24c00cf640 0x7f24c00b40c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:00.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.381+0000 7f24bd600640 1 -- 172.21.15.6:0/1636136482 >> v1:172.21.15.146:6789/0 conn(0x7f24c015e6f0 legacy=0x7f24c00cef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:00.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.381+0000 7f24bd600640 1 -- 172.21.15.6:0/1636136482 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24c015b3f0 con 0x7f24c00af5f0 2024-03-04T09:31:00.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.381+0000 7f24bd600640 1 -- 172.21.15.6:0/1636136482 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7f24b0001670 con 0x7f24c00af5f0 2024-03-04T09:31:00.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.381+0000 7f24bd600640 1 -- 172.21.15.6:0/1636136482 >> v1:172.21.15.6:6789/0 conn(0x7f24c00af5f0 legacy=0x7f24c015a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:00.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.381+0000 7f24bd600640 1 --2- 172.21.15.6:0/1636136482 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f24c00cf640 0x7f24a4005bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:00.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.381+0000 7f24bd600640 1 --2- 172.21.15.6:0/1636136482 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f24a4006170 0x7f24a4006a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:00.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.381+0000 7f24bd600640 1 --2- 172.21.15.6:0/1636136482 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f24a4006f50 0x7f24a40073d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:00.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.381+0000 7f24bd600640 1 -- 172.21.15.6:0/1636136482 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24c015e260 con 0x7f24a4006170 2024-03-04T09:31:00.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.381+0000 7f24bd600640 1 -- 172.21.15.6:0/1636136482 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24a400dcb0 con 0x7f24a4006f50 2024-03-04T09:31:00.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.381+0000 7f24bd600640 1 -- 172.21.15.6:0/1636136482 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24a400deb0 con 0x7f24c00cf640 2024-03-04T09:31:00.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.381+0000 7f24be000640 1 --2- 172.21.15.6:0/1636136482 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f24a4006170 0x7f24a4006a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:00.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.382+0000 7f24bea00640 1 --2- 172.21.15.6:0/1636136482 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f24c00cf640 0x7f24a4005bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:00.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.382+0000 7f24be000640 1 -- 172.21.15.6:0/1636136482 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f24c00cf640 msgr2=0x7f24a4005bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:00.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.382+0000 7f24bf400640 1 --2- 172.21.15.6:0/1636136482 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f24a4006f50 0x7f24a40073d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:00.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.382+0000 7f24be000640 1 --2- 172.21.15.6:0/1636136482 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f24c00cf640 0x7f24a4005bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:00.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.382+0000 7f24be000640 1 -- 172.21.15.6:0/1636136482 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f24a4006f50 msgr2=0x7f24a40073d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:00.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.382+0000 7f24be000640 1 --2- 172.21.15.6:0/1636136482 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f24a4006f50 0x7f24a40073d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:00.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.382+0000 7f24be000640 1 -- 172.21.15.6:0/1636136482 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24a4005730 con 0x7f24a4006170 2024-03-04T09:31:00.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.382+0000 7f24bea00640 1 --2- 172.21.15.6:0/1636136482 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f24c00cf640 0x7f24a4005bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:31:00.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.382+0000 7f24bf400640 1 --2- 172.21.15.6:0/1636136482 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f24a4006f50 0x7f24a40073d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:31:00.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.383+0000 7f24be000640 1 --2- 172.21.15.6:0/1636136482 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f24a4006170 0x7f24a4006a10 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f24ac001750 tx=0x7f24a4007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3552e88f56f961fa server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:00.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.383+0000 7f24bd600640 1 -- 172.21.15.6:0/1636136482 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24ac001370 con 0x7f24a4006170 2024-03-04T09:31:00.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.383+0000 7f24bd600640 1 -- 172.21.15.6:0/1636136482 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24ac0052d0 con 0x7f24a4006170 2024-03-04T09:31:00.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.383+0000 7f24bd600640 1 -- 172.21.15.6:0/1636136482 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24ac003830 con 0x7f24a4006170 2024-03-04T09:31:00.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.384+0000 7f24bfe00640 1 -- 172.21.15.6:0/1636136482 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f24a4006170 msgr2=0x7f24a4006a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:00.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.384+0000 7f24bfe00640 1 --2- 172.21.15.6:0/1636136482 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f24a4006170 0x7f24a4006a10 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f24ac001750 tx=0x7f24a4007a40 comp rx=0 tx=0).stop 2024-03-04T09:31:00.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.384+0000 7f24bf400640 1 -- 172.21.15.6:0/1636136482 reap_dead start 2024-03-04T09:31:00.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.384+0000 7f24bfe00640 1 -- 172.21.15.6:0/1636136482 shutdown_connections 2024-03-04T09:31:00.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.384+0000 7f24bfe00640 1 -- 172.21.15.6:0/1636136482 >> 172.21.15.6:0/1636136482 conn(0x7f24c00b1cc0 msgr2=0x7f24c00b20d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:00.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.384+0000 7f24bfe00640 1 -- 172.21.15.6:0/1636136482 shutdown_connections 2024-03-04T09:31:00.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.384+0000 7f24bfe00640 1 -- 172.21.15.6:0/1636136482 wait complete. 2024-03-04T09:31:00.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.386+0000 7f24bfe00640 1 Processor -- start 2024-03-04T09:31:00.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.386+0000 7f24bfe00640 1 -- start start 2024-03-04T09:31:00.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.387+0000 7f24bfe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f24a4006170 0x7f24c0157c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:00.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.387+0000 7f24bfe00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f24c00b04b0 0x7f24c00b0930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:00.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.387+0000 7f24bfe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f24c01e5730 0x7f24c01e5bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:00.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.387+0000 7f24bfe00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24c00cca20 con 0x7f24a4006170 2024-03-04T09:31:00.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.387+0000 7f24bfe00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24c015def0 con 0x7f24c01e5730 2024-03-04T09:31:00.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.387+0000 7f24bfe00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24c015ad30 con 0x7f24c00b04b0 2024-03-04T09:31:00.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.387+0000 7f24bea00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f24a4006170 0x7f24c0157c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:00.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.387+0000 7f24bea00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f24a4006170 0x7f24c0157c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51534/0 (socket says 172.21.15.6:51534) 2024-03-04T09:31:00.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.387+0000 7f24bea00640 1 -- 172.21.15.6:0/2368730839 learned_addr learned my addr 172.21.15.6:0/2368730839 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:31:00.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.387+0000 7f24be000640 1 --2- 172.21.15.6:0/2368730839 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f24c00b04b0 0x7f24c00b0930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:00.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.387+0000 7f24bea00640 1 -- 172.21.15.6:0/2368730839 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f24c00b04b0 msgr2=0x7f24c00b0930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:00.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.387+0000 7f24bea00640 1 --2- 172.21.15.6:0/2368730839 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f24c00b04b0 0x7f24c00b0930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:00.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.388+0000 7f24bf400640 1 --2- 172.21.15.6:0/2368730839 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f24c01e5730 0x7f24c01e5bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:00.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.388+0000 7f24bea00640 1 -- 172.21.15.6:0/2368730839 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f24c01e5730 msgr2=0x7f24c01e5bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:00.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.388+0000 7f24bea00640 1 --2- 172.21.15.6:0/2368730839 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f24c01e5730 0x7f24c01e5bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:00.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.388+0000 7f24bea00640 1 -- 172.21.15.6:0/2368730839 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24c009b7a0 con 0x7f24a4006170 2024-03-04T09:31:00.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.388+0000 7f24be000640 1 --2- 172.21.15.6:0/2368730839 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f24c00b04b0 0x7f24c00b0930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:31:00.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.388+0000 7f24bea00640 1 --2- 172.21.15.6:0/2368730839 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f24a4006170 0x7f24c0157c40 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f24b000d810 tx=0x7f24b000dcd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:00.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.388+0000 7f24b6a00640 1 -- 172.21.15.6:0/2368730839 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24b0019070 con 0x7f24a4006170 2024-03-04T09:31:00.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.388+0000 7f24b6a00640 1 -- 172.21.15.6:0/2368730839 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24b00074e0 con 0x7f24a4006170 2024-03-04T09:31:00.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.388+0000 7f24b6a00640 1 -- 172.21.15.6:0/2368730839 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24b00049e0 con 0x7f24a4006170 2024-03-04T09:31:00.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.388+0000 7f24bfe00640 1 -- 172.21.15.6:0/2368730839 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24c009ba20 con 0x7f24a4006170 2024-03-04T09:31:00.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.388+0000 7f24bfe00640 1 -- 172.21.15.6:0/2368730839 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24c009bfc0 con 0x7f24a4006170 2024-03-04T09:31:00.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.390+0000 7f24bfe00640 1 -- 172.21.15.6:0/2368730839 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24c004f330 con 0x7f24a4006170 2024-03-04T09:31:00.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.391+0000 7f24b6a00640 1 -- 172.21.15.6:0/2368730839 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f24b0004b80 con 0x7f24a4006170 2024-03-04T09:31:00.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.391+0000 7f24b6a00640 1 --2- 172.21.15.6:0/2368730839 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f24a006b500 0x7f24a006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:00.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.391+0000 7f24b6a00640 1 -- 172.21.15.6:0/2368730839 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f24b0089bf0 con 0x7f24a4006170 2024-03-04T09:31:00.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.392+0000 7f24be000640 1 --2- 172.21.15.6:0/2368730839 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f24a006b500 0x7f24a006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:00.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.393+0000 7f24be000640 1 --2- 172.21.15.6:0/2368730839 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f24a006b500 0x7f24a006d9c0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f24ac033000 tx=0x7f24ac009030 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:00.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.396+0000 7f24b6a00640 1 -- 172.21.15.6:0/2368730839 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f24c009b7a0 con 0x7f24a4006170 2024-03-04T09:31:00.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.754+0000 7f24bfe00640 1 -- 172.21.15.6:0/2368730839 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f24c00b3960 con 0x7f24a4006170 2024-03-04T09:31:00.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.755+0000 7f24b6a00640 1 -- 172.21.15.6:0/2368730839 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v98) v1 ==== 100+0+11 (secure 0 0 0) 0x7f24b00589c0 con 0x7f24a4006170 2024-03-04T09:31:00.752 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 64 2024-03-04T09:31:00.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.757+0000 7f24bfe00640 1 -- 172.21.15.6:0/2368730839 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f24a006b500 msgr2=0x7f24a006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:00.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.757+0000 7f24bfe00640 1 --2- 172.21.15.6:0/2368730839 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f24a006b500 0x7f24a006d9c0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f24ac033000 tx=0x7f24ac009030 comp rx=0 tx=0).stop 2024-03-04T09:31:00.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.757+0000 7f24bfe00640 1 -- 172.21.15.6:0/2368730839 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f24a4006170 msgr2=0x7f24c0157c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:00.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.757+0000 7f24bfe00640 1 --2- 172.21.15.6:0/2368730839 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f24a4006170 0x7f24c0157c40 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f24b000d810 tx=0x7f24b000dcd0 comp rx=0 tx=0).stop 2024-03-04T09:31:00.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.758+0000 7f24bfe00640 1 -- 172.21.15.6:0/2368730839 shutdown_connections 2024-03-04T09:31:00.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.758+0000 7f24bfe00640 1 --2- 172.21.15.6:0/2368730839 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f24c01e5730 0x7f24c01e5bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:00.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.758+0000 7f24bfe00640 1 --2- 172.21.15.6:0/2368730839 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f24a006b500 0x7f24a006d9c0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:00.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.758+0000 7f24bfe00640 1 --2- 172.21.15.6:0/2368730839 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f24c00b04b0 0x7f24c00b0930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:00.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.758+0000 7f24bfe00640 1 --2- 172.21.15.6:0/2368730839 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f24a4006170 0x7f24c0157c40 unknown :-1 s=CLOSED pgs=1162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:00.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.758+0000 7f24bfe00640 1 -- 172.21.15.6:0/2368730839 >> 172.21.15.6:0/2368730839 conn(0x7f24c00b1cc0 msgr2=0x7f24c00cdeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:00.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.758+0000 7f24bfe00640 1 -- 172.21.15.6:0/2368730839 shutdown_connections 2024-03-04T09:31:00.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.758+0000 7f24bfe00640 1 -- 172.21.15.6:0/2368730839 wait complete. 2024-03-04T09:31:00.773 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-04T09:31:00.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.981+0000 7f4c14c00640 1 Processor -- start 2024-03-04T09:31:00.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.981+0000 7f4c14c00640 1 -- start start 2024-03-04T09:31:00.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.982+0000 7f4c14c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4c1015e370 0x7f4c100d2ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:00.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.982+0000 7f4c14c00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f4c100cf8a0 0x7f4c100cfd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:00.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.982+0000 7f4c14c00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c100d1e40 con 0x7f4c100d0a50 2024-03-04T09:31:00.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.982+0000 7f4c14c00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c100d2a50 con 0x7f4c100cf8a0 2024-03-04T09:31:00.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.982+0000 7f4c14c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c100d2120 con 0x7f4c1015e370 2024-03-04T09:31:00.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.983+0000 7f4c0e800640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f4c100d0a50 0x7f4c10161eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:35280/0 (socket says 172.21.15.6:35280) 2024-03-04T09:31:00.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.983+0000 7f4c0e800640 1 -- 172.21.15.6:0/2636653579 learned_addr learned my addr 172.21.15.6:0/2636653579 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:31:00.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.983+0000 7f4c05400640 1 --2- 172.21.15.6:0/2636653579 >> v2:172.21.15.146:3300/0 conn(0x7f4c100cf8a0 0x7f4c100cfd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:00.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.983+0000 7f4c0de00640 1 --2- 172.21.15.6:0/2636653579 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4c1015e370 0x7f4c100d2ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:00.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.983+0000 7f4c0d400640 1 -- 172.21.15.6:0/2636653579 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1368577549 0 0) 0x7f4c100d1e40 con 0x7f4c100d0a50 2024-03-04T09:31:00.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.983+0000 7f4c0d400640 1 -- 172.21.15.6:0/2636653579 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4bf4003520 con 0x7f4c100d0a50 2024-03-04T09:31:00.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.984+0000 7f4c0d400640 1 -- 172.21.15.6:0/2636653579 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3135279870 0 0) 0x7f4bf4003520 con 0x7f4c100d0a50 2024-03-04T09:31:00.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.984+0000 7f4c0d400640 1 -- 172.21.15.6:0/2636653579 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4c1015e370 msgr2=0x7f4c100d2ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:00.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.984+0000 7f4c0d400640 1 --2- 172.21.15.6:0/2636653579 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4c1015e370 0x7f4c100d2ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:00.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.984+0000 7f4c0d400640 1 -- 172.21.15.6:0/2636653579 >> v2:172.21.15.146:3300/0 conn(0x7f4c100cf8a0 msgr2=0x7f4c100cfd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:00.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.984+0000 7f4c0d400640 1 --2- 172.21.15.6:0/2636653579 >> v2:172.21.15.146:3300/0 conn(0x7f4c100cf8a0 0x7f4c100cfd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:00.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.984+0000 7f4c0d400640 1 -- 172.21.15.6:0/2636653579 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c100d2400 con 0x7f4c100d0a50 2024-03-04T09:31:00.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.984+0000 7f4c0d400640 1 -- 172.21.15.6:0/2636653579 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7f4c08001670 con 0x7f4c100d0a50 2024-03-04T09:31:00.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.984+0000 7f4c0d400640 1 -- 172.21.15.6:0/2636653579 >> v1:172.21.15.6:6789/0 conn(0x7f4c100d0a50 legacy=0x7f4c10161eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:00.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.984+0000 7f4c0d400640 1 --2- 172.21.15.6:0/2636653579 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4bf40039b0 0x7f4bf4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:00.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.984+0000 7f4c0d400640 1 --2- 172.21.15.6:0/2636653579 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4bf4004800 0x7f4bf4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:00.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.984+0000 7f4c0d400640 1 --2- 172.21.15.6:0/2636653579 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4c1015e370 0x7f4bf40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:00.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.984+0000 7f4c0d400640 1 -- 172.21.15.6:0/2636653579 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c100d2a50 con 0x7f4bf4004800 2024-03-04T09:31:00.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.984+0000 7f4c0d400640 1 -- 172.21.15.6:0/2636653579 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c100d2120 con 0x7f4bf40039b0 2024-03-04T09:31:00.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.984+0000 7f4c0d400640 1 -- 172.21.15.6:0/2636653579 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bf400b9c0 con 0x7f4c1015e370 2024-03-04T09:31:00.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.984+0000 7f4c05400640 1 --2- 172.21.15.6:0/2636653579 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4bf4004800 0x7f4bf4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:00.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.985+0000 7f4c0de00640 1 --2- 172.21.15.6:0/2636653579 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4bf40039b0 0x7f4bf4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:00.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.985+0000 7f4c0e800640 1 --2- 172.21.15.6:0/2636653579 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4c1015e370 0x7f4bf40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:00.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.985+0000 7f4c05400640 1 -- 172.21.15.6:0/2636653579 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4c1015e370 msgr2=0x7f4bf40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:00.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.985+0000 7f4c05400640 1 --2- 172.21.15.6:0/2636653579 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4c1015e370 0x7f4bf40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:00.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.985+0000 7f4c05400640 1 -- 172.21.15.6:0/2636653579 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4bf40039b0 msgr2=0x7f4bf4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:00.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.985+0000 7f4c05400640 1 --2- 172.21.15.6:0/2636653579 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4bf40039b0 0x7f4bf4003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:00.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.985+0000 7f4c05400640 1 -- 172.21.15.6:0/2636653579 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bf4003520 con 0x7f4bf4004800 2024-03-04T09:31:00.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.986+0000 7f4c05400640 1 --2- 172.21.15.6:0/2636653579 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4bf4004800 0x7f4bf4004c80 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7f4bf8009ac0 tx=0x7f4bf4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d4f728f409b583 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:00.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.986+0000 7f4c0d400640 1 -- 172.21.15.6:0/2636653579 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bf8039070 con 0x7f4bf4004800 2024-03-04T09:31:00.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.986+0000 7f4c0d400640 1 -- 172.21.15.6:0/2636653579 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bf802c710 con 0x7f4bf4004800 2024-03-04T09:31:00.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.986+0000 7f4c0d400640 1 -- 172.21.15.6:0/2636653579 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bf8009690 con 0x7f4bf4004800 2024-03-04T09:31:00.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.987+0000 7f4c14c00640 1 -- 172.21.15.6:0/2636653579 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4bf4004800 msgr2=0x7f4bf4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:00.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.987+0000 7f4c14c00640 1 --2- 172.21.15.6:0/2636653579 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4bf4004800 0x7f4bf4004c80 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7f4bf8009ac0 tx=0x7f4bf4005830 comp rx=0 tx=0).stop 2024-03-04T09:31:00.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.987+0000 7f4c0e800640 1 -- 172.21.15.6:0/2636653579 reap_dead start 2024-03-04T09:31:00.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.987+0000 7f4c14c00640 1 -- 172.21.15.6:0/2636653579 shutdown_connections 2024-03-04T09:31:00.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.987+0000 7f4c14c00640 1 -- 172.21.15.6:0/2636653579 >> 172.21.15.6:0/2636653579 conn(0x7f4c1009d3b0 msgr2=0x7f4c100d40d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:00.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.987+0000 7f4c14c00640 1 -- 172.21.15.6:0/2636653579 shutdown_connections 2024-03-04T09:31:00.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.987+0000 7f4c14c00640 1 -- 172.21.15.6:0/2636653579 wait complete. 2024-03-04T09:31:00.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.989+0000 7f4c14c00640 1 Processor -- start 2024-03-04T09:31:00.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.989+0000 7f4c14c00640 1 -- start start 2024-03-04T09:31:00.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.990+0000 7f4c14c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4bf4004800 0x7f4c100b4450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:00.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.990+0000 7f4c14c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4c100b38a0 0x7f4c100b1c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:00.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.990+0000 7f4c14c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4c100b21d0 0x7f4c1006cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:00.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.990+0000 7f4c14c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c100d2ee0 con 0x7f4c100b38a0 2024-03-04T09:31:00.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.990+0000 7f4c14c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c1015e750 con 0x7f4bf4004800 2024-03-04T09:31:00.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.990+0000 7f4c14c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c100d3060 con 0x7f4c100b21d0 2024-03-04T09:31:00.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.990+0000 7f4c05400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4c100b38a0 0x7f4c100b1c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:00.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.991+0000 7f4c05400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4c100b38a0 0x7f4c100b1c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51562/0 (socket says 172.21.15.6:51562) 2024-03-04T09:31:00.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.991+0000 7f4c05400640 1 -- 172.21.15.6:0/1214985798 learned_addr learned my addr 172.21.15.6:0/1214985798 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:31:00.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.991+0000 7f4c0e800640 1 --2- 172.21.15.6:0/1214985798 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4c100b21d0 0x7f4c1006cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:00.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.991+0000 7f4c0de00640 1 --2- 172.21.15.6:0/1214985798 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4bf4004800 0x7f4c100b4450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:00.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.991+0000 7f4c05400640 1 -- 172.21.15.6:0/1214985798 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4c100b21d0 msgr2=0x7f4c1006cdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:00.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.991+0000 7f4c05400640 1 --2- 172.21.15.6:0/1214985798 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4c100b21d0 0x7f4c1006cdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:00.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.991+0000 7f4c05400640 1 -- 172.21.15.6:0/1214985798 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4bf4004800 msgr2=0x7f4c100b4450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:00.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.991+0000 7f4c05400640 1 --2- 172.21.15.6:0/1214985798 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4bf4004800 0x7f4c100b4450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:00.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.991+0000 7f4c05400640 1 -- 172.21.15.6:0/1214985798 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c100b15a0 con 0x7f4c100b38a0 2024-03-04T09:31:00.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.991+0000 7f4c0e800640 1 --2- 172.21.15.6:0/1214985798 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4c100b21d0 0x7f4c1006cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:31:00.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.991+0000 7f4c0de00640 1 --2- 172.21.15.6:0/1214985798 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4bf4004800 0x7f4c100b4450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:31:00.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.991+0000 7f4c05400640 1 --2- 172.21.15.6:0/1214985798 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4c100b38a0 0x7f4c100b1c90 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7f4bf8009be0 tx=0x7f4bf8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:00.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.992+0000 7f4c06a00640 1 -- 172.21.15.6:0/1214985798 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bf8042020 con 0x7f4c100b38a0 2024-03-04T09:31:00.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.992+0000 7f4c06a00640 1 -- 172.21.15.6:0/1214985798 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bf803a040 con 0x7f4c100b38a0 2024-03-04T09:31:00.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.992+0000 7f4c14c00640 1 -- 172.21.15.6:0/1214985798 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c100b2800 con 0x7f4c100b38a0 2024-03-04T09:31:00.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.992+0000 7f4c06a00640 1 -- 172.21.15.6:0/1214985798 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bf8039070 con 0x7f4c100b38a0 2024-03-04T09:31:00.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.992+0000 7f4c14c00640 1 -- 172.21.15.6:0/1214985798 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c100b2dc0 con 0x7f4c100b38a0 2024-03-04T09:31:00.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.993+0000 7f4c14c00640 1 -- 172.21.15.6:0/1214985798 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c1004f410 con 0x7f4c100b38a0 2024-03-04T09:31:00.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.994+0000 7f4c06a00640 1 -- 172.21.15.6:0/1214985798 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4bf8045020 con 0x7f4c100b38a0 2024-03-04T09:31:00.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.996+0000 7f4c06a00640 1 --2- 172.21.15.6:0/1214985798 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f4bec06b500 0x7f4bec06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:00.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.996+0000 7f4c06a00640 1 -- 172.21.15.6:0/1214985798 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f4bf80abb20 con 0x7f4c100b38a0 2024-03-04T09:31:00.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.996+0000 7f4c0de00640 1 --2- 172.21.15.6:0/1214985798 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f4bec06b500 0x7f4bec06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:00.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.997+0000 7f4c0de00640 1 --2- 172.21.15.6:0/1214985798 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f4bec06b500 0x7f4bec06d9c0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f4c00012430 tx=0x7f4c0000b040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:00.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:00.999+0000 7f4c06a00640 1 -- 172.21.15.6:0/1214985798 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4bf804c030 con 0x7f4c100b38a0 2024-03-04T09:31:01.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:01.232+0000 7f4c14c00640 1 -- 172.21.15.6:0/1214985798 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f4c100d1710 con 0x7f4c100b38a0 2024-03-04T09:31:01.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:01.233+0000 7f4c06a00640 1 -- 172.21.15.6:0/1214985798 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 4 entries v98) v1 ==== 80+0+242 (secure 0 0 0) 0x7f4bf807a850 con 0x7f4c100b38a0 2024-03-04T09:31:01.230 INFO:teuthology.orchestra.run.smithi006.stdout:172.21.15.146:6835/847815316 2024-03-05T09:30:33.260598+0000 2024-03-04T09:31:01.230 INFO:teuthology.orchestra.run.smithi006.stdout:172.21.15.146:6834/847815316 2024-03-05T09:30:33.260598+0000 2024-03-04T09:31:01.230 INFO:teuthology.orchestra.run.smithi006.stdout:172.21.15.6:6835/2836916196 2024-03-05T09:30:32.267572+0000 2024-03-04T09:31:01.230 INFO:teuthology.orchestra.run.smithi006.stdout:172.21.15.6:6832/2836916196 2024-03-05T09:30:32.267572+0000 2024-03-04T09:31:01.230 INFO:teuthology.orchestra.run.smithi006.stderr:listed 4 entries 2024-03-04T09:31:01.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:01.235+0000 7f4c14c00640 1 -- 172.21.15.6:0/1214985798 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f4bec06b500 msgr2=0x7f4bec06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:01.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:01.235+0000 7f4c14c00640 1 --2- 172.21.15.6:0/1214985798 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f4bec06b500 0x7f4bec06d9c0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f4c00012430 tx=0x7f4c0000b040 comp rx=0 tx=0).stop 2024-03-04T09:31:01.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:01.235+0000 7f4c14c00640 1 -- 172.21.15.6:0/1214985798 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4c100b38a0 msgr2=0x7f4c100b1c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:01.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:01.235+0000 7f4c14c00640 1 --2- 172.21.15.6:0/1214985798 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4c100b38a0 0x7f4c100b1c90 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7f4bf8009be0 tx=0x7f4bf8004730 comp rx=0 tx=0).stop 2024-03-04T09:31:01.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:01.235+0000 7f4c14c00640 1 -- 172.21.15.6:0/1214985798 shutdown_connections 2024-03-04T09:31:01.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:01.235+0000 7f4c14c00640 1 --2- 172.21.15.6:0/1214985798 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f4bec06b500 0x7f4bec06d9c0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:01.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:01.235+0000 7f4c14c00640 1 --2- 172.21.15.6:0/1214985798 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4c100b21d0 0x7f4c1006cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:01.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:01.235+0000 7f4c14c00640 1 --2- 172.21.15.6:0/1214985798 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4c100b38a0 0x7f4c100b1c90 unknown :-1 s=CLOSED pgs=1165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:01.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:01.235+0000 7f4c14c00640 1 --2- 172.21.15.6:0/1214985798 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4bf4004800 0x7f4c100b4450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:01.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:01.235+0000 7f4c14c00640 1 -- 172.21.15.6:0/1214985798 >> 172.21.15.6:0/1214985798 conn(0x7f4c1009d3b0 msgr2=0x7f4c100ced50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:01.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:01.236+0000 7f4c14c00640 1 -- 172.21.15.6:0/1214985798 shutdown_connections 2024-03-04T09:31:01.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:01.236+0000 7f4c14c00640 1 -- 172.21.15.6:0/1214985798 wait complete. 2024-03-04T09:31:01.258 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-04T09:31:01.258 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:31:01.259 DEBUG:teuthology.orchestra.run.smithi006:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-04T09:31:01.392 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi006.front.sepia.ceph.com 2024-03-04T09:31:01.392 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:31:01.392 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-04T09:31:01.444 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-04T09:31:01.445 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T09:31:01.486 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053429,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249180k,nr_inodes=812295,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-04T09:31:01.486 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-04T09:31:01.486 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T09:31:01.524 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054202,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249800k,nr_inodes=812450,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-04T09:31:01.524 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-04T09:31:01.524 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T09:31:01.574 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053429,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249180k,nr_inodes=812295,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-04T09:31:01.574 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:31:01.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:01.783+0000 7f9112600640 1 Processor -- start 2024-03-04T09:31:01.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:01.783+0000 7f9112600640 1 -- start start 2024-03-04T09:31:01.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:01.785+0000 7f9112600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f910c0b5870 0x7f910c166f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:01.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:01.785+0000 7f9112600640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f910c145bc0 con 0x7f910c15fba0 2024-03-04T09:31:01.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:01.785+0000 7f9112600640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f910c160040 con 0x7f910c1457e0 2024-03-04T09:31:01.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:01.785+0000 7f9112600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f910c160320 con 0x7f910c0b5870 2024-03-04T09:31:01.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:01.785+0000 7f910aa00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f910c15fba0 0x7f910c0b5160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:35286/0 (socket says 172.21.15.6:35286) 2024-03-04T09:31:01.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:01.785+0000 7f910aa00640 1 -- 172.21.15.6:0/1870326923 learned_addr learned my addr 172.21.15.6:0/1870326923 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:31:01.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:01.786+0000 7f910be00640 1 --2- 172.21.15.6:0/1870326923 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f910c0b5870 0x7f910c166f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:01.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:01.786+0000 7f910a000640 1 -- 172.21.15.6:0/1870326923 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4151596577 0 0) 0x7f910c145bc0 con 0x7f910c15fba0 2024-03-04T09:31:01.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:01.786+0000 7f910a000640 1 -- 172.21.15.6:0/1870326923 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90f0003520 con 0x7f910c15fba0 2024-03-04T09:31:01.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:01.786+0000 7f910a000640 1 -- 172.21.15.6:0/1870326923 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2590982346 0 0) 0x7f910c160040 con 0x7f910c1457e0 2024-03-04T09:31:01.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:01.786+0000 7f910a000640 1 -- 172.21.15.6:0/1870326923 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f910c145bc0 con 0x7f910c1457e0 2024-03-04T09:31:01.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:01.786+0000 7f910a000640 1 -- 172.21.15.6:0/1870326923 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3046266214 0 0) 0x7f90f0003520 con 0x7f910c15fba0 2024-03-04T09:31:01.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:01.786+0000 7f910a000640 1 -- 172.21.15.6:0/1870326923 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f910c0b5870 msgr2=0x7f910c166f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:01.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:01.786+0000 7f910a000640 1 --2- 172.21.15.6:0/1870326923 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f910c0b5870 0x7f910c166f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:01.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:01.786+0000 7f910a000640 1 -- 172.21.15.6:0/1870326923 >> v1:172.21.15.146:6789/0 conn(0x7f910c1457e0 legacy=0x7f910c0cf5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:01.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:01.786+0000 7f910a000640 1 -- 172.21.15.6:0/1870326923 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f910c160600 con 0x7f910c15fba0 2024-03-04T09:31:01.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:01.786+0000 7f910a000640 1 -- 172.21.15.6:0/1870326923 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7f90f8001670 con 0x7f910c15fba0 2024-03-04T09:31:01.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:01.786+0000 7f910a000640 1 -- 172.21.15.6:0/1870326923 >> v1:172.21.15.6:6789/0 conn(0x7f910c15fba0 legacy=0x7f910c0b5160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:01.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:01.786+0000 7f910a000640 1 --2- 172.21.15.6:0/1870326923 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f90f0003b70 0x7f90f0003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:01.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:01.786+0000 7f910a000640 1 --2- 172.21.15.6:0/1870326923 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f90f0004800 0x7f90f0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:01.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:01.786+0000 7f910a000640 1 --2- 172.21.15.6:0/1870326923 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f910c0b5870 0x7f90f0005220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:01.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:01.787+0000 7f910a000640 1 -- 172.21.15.6:0/1870326923 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f910c160320 con 0x7f90f0003b70 2024-03-04T09:31:01.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:01.787+0000 7f910a000640 1 -- 172.21.15.6:0/1870326923 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90f000bb50 con 0x7f90f0004800 2024-03-04T09:31:01.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:01.787+0000 7f910a000640 1 -- 172.21.15.6:0/1870326923 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90f000bd50 con 0x7f910c0b5870 2024-03-04T09:31:01.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:01.787+0000 7f910b400640 1 --2- 172.21.15.6:0/1870326923 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f90f0003b70 0x7f90f0003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:01.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:01.787+0000 7f910be00640 1 --2- 172.21.15.6:0/1870326923 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f910c0b5870 0x7f90f0005220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:01.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:01.787+0000 7f910aa00640 1 --2- 172.21.15.6:0/1870326923 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f90f0004800 0x7f90f0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:01.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:01.787+0000 7f910b400640 1 -- 172.21.15.6:0/1870326923 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f910c0b5870 msgr2=0x7f90f0005220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:01.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:01.787+0000 7f910b400640 1 --2- 172.21.15.6:0/1870326923 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f910c0b5870 0x7f90f0005220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:01.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:01.787+0000 7f910b400640 1 -- 172.21.15.6:0/1870326923 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f90f0004800 msgr2=0x7f90f0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:01.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:01.787+0000 7f910b400640 1 --2- 172.21.15.6:0/1870326923 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f90f0004800 0x7f90f0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:01.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:01.787+0000 7f910b400640 1 -- 172.21.15.6:0/1870326923 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90f0003520 con 0x7f90f0003b70 2024-03-04T09:31:01.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:01.788+0000 7f910b400640 1 --2- 172.21.15.6:0/1870326923 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f90f0003b70 0x7f90f0003f70 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7f90fc0018d0 tx=0x7f90f0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be623c51cd46362e server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:01.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:01.789+0000 7f910a000640 1 -- 172.21.15.6:0/1870326923 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90fc037070 con 0x7f90f0003b70 2024-03-04T09:31:01.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:01.789+0000 7f910a000640 1 -- 172.21.15.6:0/1870326923 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90fc0052d0 con 0x7f90f0003b70 2024-03-04T09:31:01.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:01.789+0000 7f910a000640 1 -- 172.21.15.6:0/1870326923 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90fc0014f0 con 0x7f90f0003b70 2024-03-04T09:31:01.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:01.789+0000 7f9112600640 1 -- 172.21.15.6:0/1870326923 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f90f0003b70 msgr2=0x7f90f0003f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:01.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:01.789+0000 7f9112600640 1 --2- 172.21.15.6:0/1870326923 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f90f0003b70 0x7f90f0003f70 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7f90fc0018d0 tx=0x7f90f0005830 comp rx=0 tx=0).stop 2024-03-04T09:31:01.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:01.789+0000 7f910be00640 1 -- 172.21.15.6:0/1870326923 reap_dead start 2024-03-04T09:31:01.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:01.789+0000 7f9112600640 1 -- 172.21.15.6:0/1870326923 shutdown_connections 2024-03-04T09:31:01.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:01.789+0000 7f9112600640 1 -- 172.21.15.6:0/1870326923 >> 172.21.15.6:0/1870326923 conn(0x7f910c06cf40 msgr2=0x7f910c06d320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:01.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:01.790+0000 7f9112600640 1 -- 172.21.15.6:0/1870326923 shutdown_connections 2024-03-04T09:31:01.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:01.790+0000 7f9112600640 1 -- 172.21.15.6:0/1870326923 wait complete. 2024-03-04T09:31:01.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:01.792+0000 7f9112600640 1 Processor -- start 2024-03-04T09:31:01.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:01.793+0000 7f9112600640 1 -- start start 2024-03-04T09:31:01.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:01.794+0000 7f9112600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f90f0003b70 0x7f910c0c4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:01.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:01.794+0000 7f9112600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f910c0c6640 0x7f910c13a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:01.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:01.794+0000 7f9112600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f910c142490 0x7f910c0c74e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:01.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:01.794+0000 7f9112600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f910c0720c0 con 0x7f910c142490 2024-03-04T09:31:01.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:01.794+0000 7f9112600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f910c0cca70 con 0x7f90f0003b70 2024-03-04T09:31:01.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:01.794+0000 7f9112600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f910c072240 con 0x7f910c0c6640 2024-03-04T09:31:01.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:01.794+0000 7f910be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f910c142490 0x7f910c0c74e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:01.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:01.795+0000 7f910be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f910c142490 0x7f910c0c74e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51574/0 (socket says 172.21.15.6:51574) 2024-03-04T09:31:01.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:01.795+0000 7f910be00640 1 -- 172.21.15.6:0/3714777711 learned_addr learned my addr 172.21.15.6:0/3714777711 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:31:01.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:01.795+0000 7f910aa00640 1 --2- 172.21.15.6:0/3714777711 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f910c0c6640 0x7f910c13a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:01.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:01.795+0000 7f910b400640 1 --2- 172.21.15.6:0/3714777711 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f90f0003b70 0x7f910c0c4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:01.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:01.795+0000 7f910be00640 1 -- 172.21.15.6:0/3714777711 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f910c0c6640 msgr2=0x7f910c13a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:01.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:01.795+0000 7f910be00640 1 --2- 172.21.15.6:0/3714777711 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f910c0c6640 0x7f910c13a410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:01.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:01.795+0000 7f910be00640 1 -- 172.21.15.6:0/3714777711 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f90f0003b70 msgr2=0x7f910c0c4dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:01.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:01.795+0000 7f910be00640 1 --2- 172.21.15.6:0/3714777711 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f90f0003b70 0x7f910c0c4dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:01.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:01.795+0000 7f910be00640 1 -- 172.21.15.6:0/3714777711 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f910c0c5300 con 0x7f910c142490 2024-03-04T09:31:01.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:01.795+0000 7f910be00640 1 --2- 172.21.15.6:0/3714777711 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f910c142490 0x7f910c0c74e0 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7f910c0b5db0 tx=0x7f91040115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:01.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:01.796+0000 7f9103400640 1 -- 172.21.15.6:0/3714777711 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9104003990 con 0x7f910c142490 2024-03-04T09:31:01.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:01.796+0000 7f9103400640 1 -- 172.21.15.6:0/3714777711 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f910401b070 con 0x7f910c142490 2024-03-04T09:31:01.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:01.796+0000 7f9112600640 1 -- 172.21.15.6:0/3714777711 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f910c0c55a0 con 0x7f910c142490 2024-03-04T09:31:01.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:01.796+0000 7f9103400640 1 -- 172.21.15.6:0/3714777711 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9104003b70 con 0x7f910c142490 2024-03-04T09:31:01.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:01.796+0000 7f9112600640 1 -- 172.21.15.6:0/3714777711 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f910c0c5b60 con 0x7f910c142490 2024-03-04T09:31:01.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:01.797+0000 7f9112600640 1 -- 172.21.15.6:0/3714777711 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f910c0692a0 con 0x7f910c142490 2024-03-04T09:31:01.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:01.798+0000 7f9103400640 1 -- 172.21.15.6:0/3714777711 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9104005d10 con 0x7f910c142490 2024-03-04T09:31:01.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:01.799+0000 7f9103400640 1 --2- 172.21.15.6:0/3714777711 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f90ec06b4d0 0x7f90ec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:01.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:01.800+0000 7f9103400640 1 -- 172.21.15.6:0/3714777711 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f910408aa10 con 0x7f910c142490 2024-03-04T09:31:01.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:01.800+0000 7f910b400640 1 --2- 172.21.15.6:0/3714777711 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f90ec06b4d0 0x7f90ec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:01.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:01.801+0000 7f910b400640 1 --2- 172.21.15.6:0/3714777711 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f90ec06b4d0 0x7f90ec06d990 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f90fc0018d0 tx=0x7f90fc031000 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:01.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:01.803+0000 7f9103400640 1 -- 172.21.15.6:0/3714777711 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9104025020 con 0x7f910c142490 2024-03-04T09:31:02.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.045+0000 7f9112600640 1 -- 172.21.15.6:0/3714777711 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f910c0adb00 con 0x7f910c142490 2024-03-04T09:31:02.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.046+0000 7f9103400640 1 -- 172.21.15.6:0/3714777711 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) v1 ==== 94+0+4977 (secure 0 0 0) 0x7f9104059740 con 0x7f910c142490 2024-03-04T09:31:02.043 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:31:02.044 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":68,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5654,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/1370302030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":1370302030},{"type":"v1","addr":"172.21.15.146:6835","nonce":1370302030}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":6163,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/809822004","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":809822004},{"type":"v1","addr":"172.21.15.146:6837","nonce":809822004}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":66},{"gid":7284,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/1583413840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1583413840},{"type":"v1","addr":"172.21.15.6:6833","nonce":1583413840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64}],"filesystems":[{"mdsmap":{"epoch":68,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:30:20.660103+0000","modified":"2024-03-04T09:30:36.289706+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":98,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7296},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7296":{"gid":7296,"name":"a","rank":0,"incarnation":65,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1732383836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1732383836},{"type":"v1","addr":"172.21.15.6:6835","nonce":1732383836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7296,"qdb_cluster":[7296]},"id":7}]} 2024-03-04T09:31:02.044 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 68 2024-03-04T09:31:02.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.049+0000 7f9112600640 1 -- 172.21.15.6:0/3714777711 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f90ec06b4d0 msgr2=0x7f90ec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:02.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.049+0000 7f9112600640 1 --2- 172.21.15.6:0/3714777711 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f90ec06b4d0 0x7f90ec06d990 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f90fc0018d0 tx=0x7f90fc031000 comp rx=0 tx=0).stop 2024-03-04T09:31:02.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.049+0000 7f9112600640 1 -- 172.21.15.6:0/3714777711 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f910c142490 msgr2=0x7f910c0c74e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:02.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.049+0000 7f9112600640 1 --2- 172.21.15.6:0/3714777711 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f910c142490 0x7f910c0c74e0 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7f910c0b5db0 tx=0x7f91040115a0 comp rx=0 tx=0).stop 2024-03-04T09:31:02.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.049+0000 7f9112600640 1 -- 172.21.15.6:0/3714777711 shutdown_connections 2024-03-04T09:31:02.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.049+0000 7f9112600640 1 --2- 172.21.15.6:0/3714777711 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f910c142490 0x7f910c0c74e0 unknown :-1 s=CLOSED pgs=1168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:02.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.049+0000 7f9112600640 1 --2- 172.21.15.6:0/3714777711 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f90ec06b4d0 0x7f90ec06d990 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:02.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.049+0000 7f9112600640 1 --2- 172.21.15.6:0/3714777711 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f910c0c6640 0x7f910c13a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:02.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.049+0000 7f9112600640 1 --2- 172.21.15.6:0/3714777711 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f90f0003b70 0x7f910c0c4dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:02.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.049+0000 7f9112600640 1 -- 172.21.15.6:0/3714777711 >> 172.21.15.6:0/3714777711 conn(0x7f910c06cf40 msgr2=0x7f910c0ccbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:02.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.050+0000 7f9112600640 1 -- 172.21.15.6:0/3714777711 shutdown_connections 2024-03-04T09:31:02.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.050+0000 7f9112600640 1 -- 172.21.15.6:0/3714777711 wait complete. 2024-03-04T09:31:02.066 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T09:31:02.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.278+0000 7f8335c00640 1 Processor -- start 2024-03-04T09:31:02.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.278+0000 7f8335c00640 1 -- start start 2024-03-04T09:31:02.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.279+0000 7f8335c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f83300d0530 0x7f83300d09e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:02.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.279+0000 7f8335c00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f83300cc9e0 0x7f83300b34b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:02.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.279+0000 7f8335c00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83300b18c0 con 0x7f833015f9e0 2024-03-04T09:31:02.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.279+0000 7f8335c00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83300b1bc0 con 0x7f83300d0530 2024-03-04T09:31:02.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.279+0000 7f8335c00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83300ccdf0 con 0x7f83300cc9e0 2024-03-04T09:31:02.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.279+0000 7f832e000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f83300d0530 0x7f83300d09e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:02.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.279+0000 7f832e000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f83300d0530 0x7f83300d09e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51586/0 (socket says 172.21.15.6:51586) 2024-03-04T09:31:02.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.279+0000 7f832e000640 1 -- 172.21.15.6:0/3940301339 learned_addr learned my addr 172.21.15.6:0/3940301339 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:31:02.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.280+0000 7f832f400640 1 --2- 172.21.15.6:0/3940301339 >> v2:172.21.15.146:3300/0 conn(0x7f83300cc9e0 0x7f83300b34b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:02.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.280+0000 7f832e000640 1 -- 172.21.15.6:0/3940301339 >> v2:172.21.15.146:3300/0 conn(0x7f83300cc9e0 msgr2=0x7f83300b34b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:02.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.280+0000 7f832e000640 1 --2- 172.21.15.6:0/3940301339 >> v2:172.21.15.146:3300/0 conn(0x7f83300cc9e0 0x7f83300b34b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:02.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.280+0000 7f832e000640 1 -- 172.21.15.6:0/3940301339 >> v1:172.21.15.6:6789/0 conn(0x7f833015f9e0 legacy=0x7f83300cfe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:02.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.280+0000 7f832e000640 1 -- 172.21.15.6:0/3940301339 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f833009b4f0 con 0x7f83300d0530 2024-03-04T09:31:02.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.282+0000 7f832e000640 1 --2- 172.21.15.6:0/3940301339 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f83300d0530 0x7f83300d09e0 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7f83300b3440 tx=0x7f831802ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=baa05b2e0105b1eb server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:02.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.282+0000 7f832d600640 1 -- 172.21.15.6:0/3940301339 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f831802f850 con 0x7f83300d0530 2024-03-04T09:31:02.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.282+0000 7f832d600640 1 -- 172.21.15.6:0/3940301339 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f831802f9f0 con 0x7f83300d0530 2024-03-04T09:31:02.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.282+0000 7f832d600640 1 -- 172.21.15.6:0/3940301339 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f831802fdc0 con 0x7f83300d0530 2024-03-04T09:31:02.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.283+0000 7f8335c00640 1 -- 172.21.15.6:0/3940301339 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f83300d0530 msgr2=0x7f83300d09e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:02.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.283+0000 7f8335c00640 1 --2- 172.21.15.6:0/3940301339 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f83300d0530 0x7f83300d09e0 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7f83300b3440 tx=0x7f831802ba30 comp rx=0 tx=0).stop 2024-03-04T09:31:02.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.283+0000 7f8335c00640 1 -- 172.21.15.6:0/3940301339 shutdown_connections 2024-03-04T09:31:02.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.283+0000 7f8335c00640 1 --2- 172.21.15.6:0/3940301339 >> v2:172.21.15.146:3300/0 conn(0x7f83300cc9e0 0x7f83300b34b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:02.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.283+0000 7f8335c00640 1 --2- 172.21.15.6:0/3940301339 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f83300d0530 0x7f83300d09e0 unknown :-1 s=CLOSED pgs=1170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:02.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.283+0000 7f8335c00640 1 -- 172.21.15.6:0/3940301339 >> 172.21.15.6:0/3940301339 conn(0x7f83300d1f20 msgr2=0x7f83300d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:02.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.283+0000 7f8335c00640 1 -- 172.21.15.6:0/3940301339 shutdown_connections 2024-03-04T09:31:02.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.283+0000 7f8335c00640 1 -- 172.21.15.6:0/3940301339 wait complete. 2024-03-04T09:31:02.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.285+0000 7f8335c00640 1 Processor -- start 2024-03-04T09:31:02.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.285+0000 7f8335c00640 1 -- start start 2024-03-04T09:31:02.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.286+0000 7f8335c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f83300cc9e0 0x7f83301593a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:02.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.286+0000 7f8335c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f83300d0530 0x7f83300c66d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:02.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.286+0000 7f8335c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833015f9e0 0x7f83300c48f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:02.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.286+0000 7f8335c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83300d2a40 con 0x7f83300cc9e0 2024-03-04T09:31:02.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.286+0000 7f8335c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83300d2db0 con 0x7f833015f9e0 2024-03-04T09:31:02.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.286+0000 7f8335c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83300b2050 con 0x7f83300d0530 2024-03-04T09:31:02.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.286+0000 7f832ea00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f83300cc9e0 0x7f83301593a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:02.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.286+0000 7f832ea00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f83300cc9e0 0x7f83301593a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51592/0 (socket says 172.21.15.6:51592) 2024-03-04T09:31:02.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.286+0000 7f832ea00640 1 -- 172.21.15.6:0/193405955 learned_addr learned my addr 172.21.15.6:0/193405955 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:31:02.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.287+0000 7f832ea00640 1 -- 172.21.15.6:0/193405955 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f83300d0530 msgr2=0x7f83300c66d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:02.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.287+0000 7f832ea00640 1 --2- 172.21.15.6:0/193405955 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f83300d0530 0x7f83300c66d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:02.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.287+0000 7f832ea00640 1 -- 172.21.15.6:0/193405955 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833015f9e0 msgr2=0x7f83300c48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:02.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.287+0000 7f832ea00640 1 --2- 172.21.15.6:0/193405955 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833015f9e0 0x7f83300c48f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:02.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.287+0000 7f832ea00640 1 -- 172.21.15.6:0/193405955 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83300c8630 con 0x7f83300cc9e0 2024-03-04T09:31:02.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.287+0000 7f832ea00640 1 --2- 172.21.15.6:0/193405955 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f83300cc9e0 0x7f83301593a0 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f832000c730 tx=0x7f832000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:02.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.287+0000 7f8326a00640 1 -- 172.21.15.6:0/193405955 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8320016020 con 0x7f83300cc9e0 2024-03-04T09:31:02.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.287+0000 7f8326a00640 1 -- 172.21.15.6:0/193405955 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8320006d50 con 0x7f83300cc9e0 2024-03-04T09:31:02.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.287+0000 7f8326a00640 1 -- 172.21.15.6:0/193405955 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8320003bb0 con 0x7f83300cc9e0 2024-03-04T09:31:02.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.287+0000 7f8335c00640 1 -- 172.21.15.6:0/193405955 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83300c9390 con 0x7f83300cc9e0 2024-03-04T09:31:02.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.287+0000 7f8335c00640 1 -- 172.21.15.6:0/193405955 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83300c9950 con 0x7f83300cc9e0 2024-03-04T09:31:02.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.289+0000 7f8335c00640 1 -- 172.21.15.6:0/193405955 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8330068070 con 0x7f83300cc9e0 2024-03-04T09:31:02.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.290+0000 7f8326a00640 1 -- 172.21.15.6:0/193405955 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f832000a7f0 con 0x7f83300cc9e0 2024-03-04T09:31:02.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.291+0000 7f8326a00640 1 --2- 172.21.15.6:0/193405955 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f830c06b4d0 0x7f830c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:02.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.291+0000 7f8326a00640 1 -- 172.21.15.6:0/193405955 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f8320089e80 con 0x7f83300cc9e0 2024-03-04T09:31:02.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.291+0000 7f832e000640 1 --2- 172.21.15.6:0/193405955 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f830c06b4d0 0x7f830c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:02.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.292+0000 7f832e000640 1 --2- 172.21.15.6:0/193405955 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f830c06b4d0 0x7f830c06d990 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f8318036000 tx=0x7f8318039040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:02.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.295+0000 7f8326a00640 1 -- 172.21.15.6:0/193405955 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8320020020 con 0x7f83300cc9e0 2024-03-04T09:31:02.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.497+0000 7f8335c00640 1 -- 172.21.15.6:0/193405955 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8330159aa0 con 0x7f83300cc9e0 2024-03-04T09:31:02.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.498+0000 7f8326a00640 1 -- 172.21.15.6:0/193405955 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v98) v1 ==== 74+0+16817 (secure 0 0 0) 0x7f8320058ce0 con 0x7f83300cc9e0 2024-03-04T09:31:02.495 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:31:02.496 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":98,"fsid":"edce0578-5ae4-4402-b45a-552bdef84d7c","created":"2024-03-04T09:24:54.523780+0000","modified":"2024-03-04T09:30:33.260648+0000","last_up_change":"2024-03-04T09:25:07.089049+0000","last_in_change":"2024-03-04T09:24:57.982077+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T09:25:07.935198+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-03-04T09:30:18.510767+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"93","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-03-04T09:30:19.686114+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"94","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a96574f-81f1-4bc6-8463-757520b84a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6825","nonce":1364449401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6827","nonce":1364449401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6831","nonce":1364449401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6829","nonce":1364449401}]},"public_addr":"172.21.15.6:6825/1364449401","cluster_addr":"172.21.15.6:6827/1364449401","heartbeat_back_addr":"172.21.15.6:6831/1364449401","heartbeat_front_addr":"172.21.15.6:6829/1364449401","state":["exists","up"]},{"osd":1,"uuid":"7d79b9c1-6b87-476a-bc36-7db5ee64cd90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6809","nonce":2494103078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6811","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6813","nonce":2494103078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6821","nonce":2494103078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6817","nonce":2494103078}]},"public_addr":"172.21.15.6:6809/2494103078","cluster_addr":"172.21.15.6:6813/2494103078","heartbeat_back_addr":"172.21.15.6:6821/2494103078","heartbeat_front_addr":"172.21.15.6:6817/2494103078","state":["exists","up"]},{"osd":2,"uuid":"492331b9-4ee9-40b6-bacd-85d44cd6c45a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6812","nonce":1500066681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6816","nonce":1500066681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6824","nonce":1500066681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6820","nonce":1500066681}]},"public_addr":"172.21.15.6:6812/1500066681","cluster_addr":"172.21.15.6:6816/1500066681","heartbeat_back_addr":"172.21.15.6:6824/1500066681","heartbeat_front_addr":"172.21.15.6:6820/1500066681","state":["exists","up"]},{"osd":3,"uuid":"21bb2efe-9ba7-4bdc-bc31-180cd1102e21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6801","nonce":1558806123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6803","nonce":1558806123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6807","nonce":1558806123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6805","nonce":1558806123}]},"public_addr":"172.21.15.6:6801/1558806123","cluster_addr":"172.21.15.6:6803/1558806123","heartbeat_back_addr":"172.21.15.6:6807/1558806123","heartbeat_front_addr":"172.21.15.6:6805/1558806123","state":["exists","up"]},{"osd":4,"uuid":"6e2c4b71-19d9-4737-83d9-558705d8644a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6801","nonce":1526135780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6803","nonce":1526135780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6807","nonce":1526135780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6805","nonce":1526135780}]},"public_addr":"172.21.15.146:6801/1526135780","cluster_addr":"172.21.15.146:6803/1526135780","heartbeat_back_addr":"172.21.15.146:6807/1526135780","heartbeat_front_addr":"172.21.15.146:6805/1526135780","state":["exists","up"]},{"osd":5,"uuid":"fb69c525-f1be-4b7d-ab32-6c29b0d6fddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6809","nonce":460262507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6811","nonce":460262507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6815","nonce":460262507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6813","nonce":460262507}]},"public_addr":"172.21.15.146:6809/460262507","cluster_addr":"172.21.15.146:6811/460262507","heartbeat_back_addr":"172.21.15.146:6815/460262507","heartbeat_front_addr":"172.21.15.146:6813/460262507","state":["exists","up"]},{"osd":6,"uuid":"c32e7fed-370d-4401-8ce0-6a4335b0d168","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6823","nonce":1392897493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6827","nonce":1392897493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6831","nonce":1392897493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6829","nonce":1392897493}]},"public_addr":"172.21.15.146:6823/1392897493","cluster_addr":"172.21.15.146:6827/1392897493","heartbeat_back_addr":"172.21.15.146:6831/1392897493","heartbeat_front_addr":"172.21.15.146:6829/1392897493","state":["exists","up"]},{"osd":7,"uuid":"75a79b44-fa5f-426d-a7d3-71f21f78835c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6817","nonce":3268323156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6819","nonce":3268323156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6825","nonce":3268323156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6822","nonce":3268323156}]},"public_addr":"172.21.15.146:6817/3268323156","cluster_addr":"172.21.15.146:6819/3268323156","heartbeat_back_addr":"172.21.15.146:6825/3268323156","heartbeat_front_addr":"172.21.15.146:6822/3268323156","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.310437+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.295850+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.294674+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.526855+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.300023+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.595160+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.346046+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.558741+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.146:6835/847815316":"2024-03-05T09:30:33.260598+0000","172.21.15.146:6834/847815316":"2024-03-05T09:30:33.260598+0000","172.21.15.6:6835/2836916196":"2024-03-05T09:30:32.267572+0000","172.21.15.6:6832/2836916196":"2024-03-05T09:30:32.267572+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T09:31:02.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.501+0000 7f8335c00640 1 -- 172.21.15.6:0/193405955 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f830c06b4d0 msgr2=0x7f830c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:02.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.501+0000 7f8335c00640 1 --2- 172.21.15.6:0/193405955 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f830c06b4d0 0x7f830c06d990 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f8318036000 tx=0x7f8318039040 comp rx=0 tx=0).stop 2024-03-04T09:31:02.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.501+0000 7f8335c00640 1 -- 172.21.15.6:0/193405955 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f83300cc9e0 msgr2=0x7f83301593a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:02.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.501+0000 7f8335c00640 1 --2- 172.21.15.6:0/193405955 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f83300cc9e0 0x7f83301593a0 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f832000c730 tx=0x7f832000cbf0 comp rx=0 tx=0).stop 2024-03-04T09:31:02.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.501+0000 7f8335c00640 1 -- 172.21.15.6:0/193405955 shutdown_connections 2024-03-04T09:31:02.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.501+0000 7f8335c00640 1 --2- 172.21.15.6:0/193405955 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833015f9e0 0x7f83300c48f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:02.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.501+0000 7f8335c00640 1 --2- 172.21.15.6:0/193405955 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f830c06b4d0 0x7f830c06d990 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:02.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.501+0000 7f8335c00640 1 --2- 172.21.15.6:0/193405955 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f83300d0530 0x7f83300c66d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:02.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.501+0000 7f8335c00640 1 --2- 172.21.15.6:0/193405955 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f83300cc9e0 0x7f83301593a0 unknown :-1 s=CLOSED pgs=1171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:02.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.501+0000 7f8335c00640 1 -- 172.21.15.6:0/193405955 >> 172.21.15.6:0/193405955 conn(0x7f83300d1f20 msgr2=0x7f833006f0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:02.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.501+0000 7f8335c00640 1 -- 172.21.15.6:0/193405955 shutdown_connections 2024-03-04T09:31:02.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.501+0000 7f8335c00640 1 -- 172.21.15.6:0/193405955 wait complete. 2024-03-04T09:31:02.519 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T09:25:07.935198+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-04T09:30:18.510767+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '93', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_data', 'create_time': '2024-03-04T09:30:19.686114+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '94', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-04T09:31:02.519 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-04T09:31:02.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.741+0000 7ff1e3600640 1 Processor -- start 2024-03-04T09:31:02.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.742+0000 7ff1e3600640 1 -- start start 2024-03-04T09:31:02.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.743+0000 7ff1e3600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff1dc0aec40 0x7ff1dc0cf730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:02.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.743+0000 7ff1e3600640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ff1dc0b58e0 0x7ff1dc159220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:02.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.743+0000 7ff1e3600640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1dc15d500 con 0x7ff1dc0af020 2024-03-04T09:31:02.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.743+0000 7ff1e3600640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1dc15d070 con 0x7ff1dc0aec40 2024-03-04T09:31:02.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.743+0000 7ff1e3600640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1dc071d70 con 0x7ff1dc0b58e0 2024-03-04T09:31:02.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.744+0000 7ff1e0c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff1dc0aec40 0x7ff1dc0cf730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:02.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.744+0000 7ff1e0c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff1dc0aec40 0x7ff1dc0cf730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51598/0 (socket says 172.21.15.6:51598) 2024-03-04T09:31:02.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.744+0000 7ff1e0c00640 1 -- 172.21.15.6:0/590756717 learned_addr learned my addr 172.21.15.6:0/590756717 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:31:02.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.744+0000 7ff1e1600640 1 --2- 172.21.15.6:0/590756717 >> v2:172.21.15.146:3300/0 conn(0x7ff1dc0b58e0 0x7ff1dc159220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:02.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.744+0000 7ff1e0c00640 1 -- 172.21.15.6:0/590756717 >> v2:172.21.15.146:3300/0 conn(0x7ff1dc0b58e0 msgr2=0x7ff1dc159220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:02.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.744+0000 7ff1e0c00640 1 --2- 172.21.15.6:0/590756717 >> v2:172.21.15.146:3300/0 conn(0x7ff1dc0b58e0 0x7ff1dc159220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:02.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.744+0000 7ff1e0c00640 1 -- 172.21.15.6:0/590756717 >> v1:172.21.15.146:6789/0 conn(0x7ff1dc0af020 legacy=0x7ff1dc0b51d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:02.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.744+0000 7ff1e0c00640 1 -- 172.21.15.6:0/590756717 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1dc15c740 con 0x7ff1dc0aec40 2024-03-04T09:31:02.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.745+0000 7ff1e0c00640 1 --2- 172.21.15.6:0/590756717 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff1dc0aec40 0x7ff1dc0cf730 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7ff1dc0afb00 tx=0x7ff1cc02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a4a79d9b5e32a3b server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:02.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.746+0000 7ff1db400640 1 -- 172.21.15.6:0/590756717 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1cc02e070 con 0x7ff1dc0aec40 2024-03-04T09:31:02.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.746+0000 7ff1db400640 1 -- 172.21.15.6:0/590756717 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1cc037070 con 0x7ff1dc0aec40 2024-03-04T09:31:02.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.746+0000 7ff1db400640 1 -- 172.21.15.6:0/590756717 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1cc031a00 con 0x7ff1dc0aec40 2024-03-04T09:31:02.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.746+0000 7ff1e3600640 1 -- 172.21.15.6:0/590756717 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff1dc0aec40 msgr2=0x7ff1dc0cf730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:02.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.746+0000 7ff1e3600640 1 --2- 172.21.15.6:0/590756717 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff1dc0aec40 0x7ff1dc0cf730 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7ff1dc0afb00 tx=0x7ff1cc02b880 comp rx=0 tx=0).stop 2024-03-04T09:31:02.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.747+0000 7ff1e3600640 1 -- 172.21.15.6:0/590756717 shutdown_connections 2024-03-04T09:31:02.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.747+0000 7ff1e3600640 1 --2- 172.21.15.6:0/590756717 >> v2:172.21.15.146:3300/0 conn(0x7ff1dc0b58e0 0x7ff1dc159220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:02.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.747+0000 7ff1e3600640 1 --2- 172.21.15.6:0/590756717 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff1dc0aec40 0x7ff1dc0cf730 unknown :-1 s=CLOSED pgs=1172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:02.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.747+0000 7ff1e3600640 1 -- 172.21.15.6:0/590756717 >> 172.21.15.6:0/590756717 conn(0x7ff1dc06cfb0 msgr2=0x7ff1dc06d390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:02.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.747+0000 7ff1e3600640 1 -- 172.21.15.6:0/590756717 shutdown_connections 2024-03-04T09:31:02.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.747+0000 7ff1e3600640 1 -- 172.21.15.6:0/590756717 wait complete. 2024-03-04T09:31:02.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.749+0000 7ff1e3600640 1 Processor -- start 2024-03-04T09:31:02.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.749+0000 7ff1e3600640 1 -- start start 2024-03-04T09:31:02.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.751+0000 7ff1e3600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff1dc0aec40 0x7ff1dc149660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:02.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.751+0000 7ff1e3600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff1dc0af020 0x7ff1dc156ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:02.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.751+0000 7ff1e3600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff1dc0b58e0 0x7ff1dc0a9db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:02.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.751+0000 7ff1e3600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1dc0ccbb0 con 0x7ff1dc0af020 2024-03-04T09:31:02.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.751+0000 7ff1e3600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1dc15cd00 con 0x7ff1dc0aec40 2024-03-04T09:31:02.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.751+0000 7ff1e3600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1dc0cc810 con 0x7ff1dc0b58e0 2024-03-04T09:31:02.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.751+0000 7ff1dbe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff1dc0af020 0x7ff1dc156ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:02.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.751+0000 7ff1dbe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff1dc0af020 0x7ff1dc156ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51600/0 (socket says 172.21.15.6:51600) 2024-03-04T09:31:02.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.751+0000 7ff1dbe00640 1 -- 172.21.15.6:0/2477305429 learned_addr learned my addr 172.21.15.6:0/2477305429 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:31:02.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.751+0000 7ff1e0c00640 1 --2- 172.21.15.6:0/2477305429 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff1dc0aec40 0x7ff1dc149660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:02.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.751+0000 7ff1dbe00640 1 -- 172.21.15.6:0/2477305429 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff1dc0b58e0 msgr2=0x7ff1dc0a9db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:02.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.751+0000 7ff1dbe00640 1 --2- 172.21.15.6:0/2477305429 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff1dc0b58e0 0x7ff1dc0a9db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:02.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.751+0000 7ff1dbe00640 1 -- 172.21.15.6:0/2477305429 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff1dc0aec40 msgr2=0x7ff1dc149660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:02.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.751+0000 7ff1dbe00640 1 --2- 172.21.15.6:0/2477305429 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff1dc0aec40 0x7ff1dc149660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:02.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.751+0000 7ff1dbe00640 1 -- 172.21.15.6:0/2477305429 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1dc155f50 con 0x7ff1dc0af020 2024-03-04T09:31:02.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.752+0000 7ff1dbe00640 1 --2- 172.21.15.6:0/2477305429 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff1dc0af020 0x7ff1dc156ff0 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7ff1c800c400 tx=0x7ff1c800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:02.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.752+0000 7ff1d8c00640 1 -- 172.21.15.6:0/2477305429 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1c8011790 con 0x7ff1dc0af020 2024-03-04T09:31:02.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.752+0000 7ff1d8c00640 1 -- 172.21.15.6:0/2477305429 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1c8017070 con 0x7ff1dc0af020 2024-03-04T09:31:02.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.752+0000 7ff1e3600640 1 -- 172.21.15.6:0/2477305429 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1dc156230 con 0x7ff1dc0af020 2024-03-04T09:31:02.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.752+0000 7ff1d8c00640 1 -- 172.21.15.6:0/2477305429 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1c8011bb0 con 0x7ff1dc0af020 2024-03-04T09:31:02.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.752+0000 7ff1e3600640 1 -- 172.21.15.6:0/2477305429 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1dc156ad0 con 0x7ff1dc0af020 2024-03-04T09:31:02.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.753+0000 7ff1e3600640 1 -- 172.21.15.6:0/2477305429 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1dc0692a0 con 0x7ff1dc0af020 2024-03-04T09:31:02.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.754+0000 7ff1d8c00640 1 -- 172.21.15.6:0/2477305429 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff1c801c460 con 0x7ff1dc0af020 2024-03-04T09:31:02.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.755+0000 7ff1d8c00640 1 --2- 172.21.15.6:0/2477305429 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff1bc06b4d0 0x7ff1bc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:02.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.755+0000 7ff1d8c00640 1 -- 172.21.15.6:0/2477305429 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5881+0+0 (secure 0 0 0) 0x7ff1c808a070 con 0x7ff1dc0af020 2024-03-04T09:31:02.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.756+0000 7ff1e0c00640 1 --2- 172.21.15.6:0/2477305429 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff1bc06b4d0 0x7ff1bc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:02.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.757+0000 7ff1e0c00640 1 --2- 172.21.15.6:0/2477305429 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff1bc06b4d0 0x7ff1bc06d990 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7ff1cc036000 tx=0x7ff1cc03d040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:02.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:02.760+0000 7ff1d8c00640 1 -- 172.21.15.6:0/2477305429 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff1c8058ed0 con 0x7ff1dc0af020 2024-03-04T09:31:03.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.009+0000 7ff1e3600640 1 -- 172.21.15.6:0/2477305429 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff1dc156ca0 con 0x7ff1dc0af020 2024-03-04T09:31:03.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.010+0000 7ff1d8c00640 1 -- 172.21.15.6:0/2477305429 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v98) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff1c80587d0 con 0x7ff1dc0af020 2024-03-04T09:31:03.006 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-04T09:31:03.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.012+0000 7ff1e3600640 1 -- 172.21.15.6:0/2477305429 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff1bc06b4d0 msgr2=0x7ff1bc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:03.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.012+0000 7ff1e3600640 1 --2- 172.21.15.6:0/2477305429 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff1bc06b4d0 0x7ff1bc06d990 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7ff1cc036000 tx=0x7ff1cc03d040 comp rx=0 tx=0).stop 2024-03-04T09:31:03.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.012+0000 7ff1e3600640 1 -- 172.21.15.6:0/2477305429 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff1dc0af020 msgr2=0x7ff1dc156ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:03.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.012+0000 7ff1e3600640 1 --2- 172.21.15.6:0/2477305429 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff1dc0af020 0x7ff1dc156ff0 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7ff1c800c400 tx=0x7ff1c800c8c0 comp rx=0 tx=0).stop 2024-03-04T09:31:03.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.012+0000 7ff1e3600640 1 -- 172.21.15.6:0/2477305429 shutdown_connections 2024-03-04T09:31:03.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.012+0000 7ff1e3600640 1 --2- 172.21.15.6:0/2477305429 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff1bc06b4d0 0x7ff1bc06d990 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:03.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.012+0000 7ff1e3600640 1 --2- 172.21.15.6:0/2477305429 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff1dc0b58e0 0x7ff1dc0a9db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:03.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.012+0000 7ff1e3600640 1 --2- 172.21.15.6:0/2477305429 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff1dc0af020 0x7ff1dc156ff0 unknown :-1 s=CLOSED pgs=1173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:03.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.012+0000 7ff1e3600640 1 --2- 172.21.15.6:0/2477305429 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff1dc0aec40 0x7ff1dc149660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:03.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.012+0000 7ff1e3600640 1 -- 172.21.15.6:0/2477305429 >> 172.21.15.6:0/2477305429 conn(0x7ff1dc06cfb0 msgr2=0x7ff1dc06d610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:03.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.012+0000 7ff1e3600640 1 -- 172.21.15.6:0/2477305429 shutdown_connections 2024-03-04T09:31:03.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.012+0000 7ff1e3600640 1 -- 172.21.15.6:0/2477305429 wait complete. 2024-03-04T09:31:03.029 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-04T09:31:03.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.275+0000 7f6457000640 1 Processor -- start 2024-03-04T09:31:03.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.275+0000 7f6457000640 1 -- start start 2024-03-04T09:31:03.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.278+0000 7f6457000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f64500af9f0 0x7f64500b5990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:03.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.278+0000 7f6457000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f645006afc0 0x7f645006b450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:03.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.278+0000 7f6457000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64500b00b0 con 0x7f64500af2b0 2024-03-04T09:31:03.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.278+0000 7f6457000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64500afe60 con 0x7f64500af9f0 2024-03-04T09:31:03.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.278+0000 7f6457000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64500b5ed0 con 0x7f645006afc0 2024-03-04T09:31:03.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.279+0000 7f6456600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f645006afc0 0x7f645006b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:03.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.279+0000 7f6455200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f64500af9f0 0x7f64500b5990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:03.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.279+0000 7f6456600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f645006afc0 0x7f645006b450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.6:51686/0 (socket says 172.21.15.6:51686) 2024-03-04T09:31:03.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.279+0000 7f6456600640 1 -- 172.21.15.6:0/870309887 learned_addr learned my addr 172.21.15.6:0/870309887 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:31:03.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.280+0000 7f6455200640 1 -- 172.21.15.6:0/870309887 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f645006afc0 msgr2=0x7f645006b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:03.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.280+0000 7f6455200640 1 --2- 172.21.15.6:0/870309887 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f645006afc0 0x7f645006b450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:03.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.280+0000 7f6455200640 1 -- 172.21.15.6:0/870309887 >> v1:172.21.15.146:6789/0 conn(0x7f64500af2b0 legacy=0x7f645015f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:03.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.280+0000 7f6455200640 1 -- 172.21.15.6:0/870309887 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64500b0390 con 0x7f64500af9f0 2024-03-04T09:31:03.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.280+0000 7f6456600640 1 --2- 172.21.15.6:0/870309887 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f645006afc0 0x7f645006b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T09:31:03.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.281+0000 7f6455200640 1 --2- 172.21.15.6:0/870309887 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f64500af9f0 0x7f64500b5990 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f643c0097a0 tx=0x7f643c02b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=df026901b271d5d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:03.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.282+0000 7f644fe00640 1 -- 172.21.15.6:0/870309887 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f643c035020 con 0x7f64500af9f0 2024-03-04T09:31:03.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.282+0000 7f644fe00640 1 -- 172.21.15.6:0/870309887 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f643c031810 con 0x7f64500af9f0 2024-03-04T09:31:03.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.282+0000 7f644fe00640 1 -- 172.21.15.6:0/870309887 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f643c031be0 con 0x7f64500af9f0 2024-03-04T09:31:03.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.283+0000 7f6457000640 1 -- 172.21.15.6:0/870309887 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f64500af9f0 msgr2=0x7f64500b5990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:03.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.283+0000 7f6457000640 1 --2- 172.21.15.6:0/870309887 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f64500af9f0 0x7f64500b5990 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f643c0097a0 tx=0x7f643c02b8b0 comp rx=0 tx=0).stop 2024-03-04T09:31:03.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.283+0000 7f6457000640 1 -- 172.21.15.6:0/870309887 shutdown_connections 2024-03-04T09:31:03.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.283+0000 7f6457000640 1 --2- 172.21.15.6:0/870309887 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f645006afc0 0x7f645006b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:03.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.283+0000 7f6457000640 1 --2- 172.21.15.6:0/870309887 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f64500af9f0 0x7f64500b5990 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:03.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.283+0000 7f6457000640 1 -- 172.21.15.6:0/870309887 >> 172.21.15.6:0/870309887 conn(0x7f645006c0d0 msgr2=0x7f645006c4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:03.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.283+0000 7f6457000640 1 -- 172.21.15.6:0/870309887 shutdown_connections 2024-03-04T09:31:03.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.283+0000 7f6457000640 1 -- 172.21.15.6:0/870309887 wait complete. 2024-03-04T09:31:03.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.285+0000 7f6457000640 1 Processor -- start 2024-03-04T09:31:03.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.285+0000 7f6457000640 1 -- start start 2024-03-04T09:31:03.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.287+0000 7f6457000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f645006afc0 0x7f6450158930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:03.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.287+0000 7f6457000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f64500af2b0 0x7f64500b19b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:03.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.287+0000 7f6457000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f64500af9f0 0x7f64501607c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:03.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.287+0000 7f6457000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64500d08e0 con 0x7f64500af2b0 2024-03-04T09:31:03.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.287+0000 7f6457000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f645015eae0 con 0x7f645006afc0 2024-03-04T09:31:03.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.287+0000 7f6457000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64500d0a60 con 0x7f64500af9f0 2024-03-04T09:31:03.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.287+0000 7f6455200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f64500af2b0 0x7f64500b19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:03.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.287+0000 7f6455200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f64500af2b0 0x7f64500b19b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51610/0 (socket says 172.21.15.6:51610) 2024-03-04T09:31:03.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.287+0000 7f6455200640 1 -- 172.21.15.6:0/2142508903 learned_addr learned my addr 172.21.15.6:0/2142508903 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:31:03.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.288+0000 7f6455c00640 1 --2- 172.21.15.6:0/2142508903 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f645006afc0 0x7f6450158930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:03.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.288+0000 7f6456600640 1 --2- 172.21.15.6:0/2142508903 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f64500af9f0 0x7f64501607c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:03.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.288+0000 7f6455200640 1 -- 172.21.15.6:0/2142508903 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f64500af9f0 msgr2=0x7f64501607c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:03.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.288+0000 7f6455200640 1 --2- 172.21.15.6:0/2142508903 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f64500af9f0 0x7f64501607c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:03.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.288+0000 7f6455200640 1 -- 172.21.15.6:0/2142508903 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f645006afc0 msgr2=0x7f6450158930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:03.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.288+0000 7f6455200640 1 --2- 172.21.15.6:0/2142508903 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f645006afc0 0x7f6450158930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:03.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.288+0000 7f6455200640 1 -- 172.21.15.6:0/2142508903 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6450159590 con 0x7f64500af2b0 2024-03-04T09:31:03.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.288+0000 7f6456600640 1 --2- 172.21.15.6:0/2142508903 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f64500af9f0 0x7f64501607c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:31:03.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.288+0000 7f6455c00640 1 --2- 172.21.15.6:0/2142508903 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f645006afc0 0x7f6450158930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T09:31:03.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.288+0000 7f6455200640 1 --2- 172.21.15.6:0/2142508903 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f64500af2b0 0x7f64500b19b0 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7f643c0098c0 tx=0x7f643c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:03.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.289+0000 7f644d600640 1 -- 172.21.15.6:0/2142508903 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f643c035040 con 0x7f64500af2b0 2024-03-04T09:31:03.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.289+0000 7f644d600640 1 -- 172.21.15.6:0/2142508903 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f643c039cf0 con 0x7f64500af2b0 2024-03-04T09:31:03.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.289+0000 7f644d600640 1 -- 172.21.15.6:0/2142508903 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f643c039ea0 con 0x7f64500af2b0 2024-03-04T09:31:03.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.289+0000 7f6457000640 1 -- 172.21.15.6:0/2142508903 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6450159830 con 0x7f64500af2b0 2024-03-04T09:31:03.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.289+0000 7f6457000640 1 -- 172.21.15.6:0/2142508903 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6450143920 con 0x7f64500af2b0 2024-03-04T09:31:03.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.290+0000 7f6457000640 1 -- 172.21.15.6:0/2142508903 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f645004f330 con 0x7f64500af2b0 2024-03-04T09:31:03.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.291+0000 7f644d600640 1 -- 172.21.15.6:0/2142508903 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f643c038020 con 0x7f64500af2b0 2024-03-04T09:31:03.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.292+0000 7f644d600640 1 --2- 172.21.15.6:0/2142508903 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f643006b4d0 0x7f643006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:03.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.292+0000 7f644d600640 1 -- 172.21.15.6:0/2142508903 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f643c0ac0d0 con 0x7f64500af2b0 2024-03-04T09:31:03.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.293+0000 7f6455c00640 1 --2- 172.21.15.6:0/2142508903 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f643006b4d0 0x7f643006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:03.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.294+0000 7f6455c00640 1 --2- 172.21.15.6:0/2142508903 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f643006b4d0 0x7f643006d990 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f6444007fd0 tx=0x7f6444009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:03.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.301+0000 7f644d600640 1 -- 172.21.15.6:0/2142508903 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f643c07ad70 con 0x7f64500af2b0 2024-03-04T09:31:03.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.502+0000 7f6457000640 1 -- 172.21.15.6:0/2142508903 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f645015bad0 con 0x7f64500af2b0 2024-03-04T09:31:03.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.503+0000 7f644d600640 1 -- 172.21.15.6:0/2142508903 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v98) v1 ==== 104+0+11 (secure 0 0 0) 0x7f643c07a670 con 0x7f64500af2b0 2024-03-04T09:31:03.500 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-03-04T09:31:03.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.506+0000 7f6457000640 1 -- 172.21.15.6:0/2142508903 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f643006b4d0 msgr2=0x7f643006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:03.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.506+0000 7f6457000640 1 --2- 172.21.15.6:0/2142508903 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f643006b4d0 0x7f643006d990 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f6444007fd0 tx=0x7f6444009040 comp rx=0 tx=0).stop 2024-03-04T09:31:03.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.506+0000 7f6457000640 1 -- 172.21.15.6:0/2142508903 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f64500af2b0 msgr2=0x7f64500b19b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:03.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.506+0000 7f6457000640 1 --2- 172.21.15.6:0/2142508903 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f64500af2b0 0x7f64500b19b0 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7f643c0098c0 tx=0x7f643c0046c0 comp rx=0 tx=0).stop 2024-03-04T09:31:03.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.506+0000 7f6457000640 1 -- 172.21.15.6:0/2142508903 shutdown_connections 2024-03-04T09:31:03.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.506+0000 7f6457000640 1 --2- 172.21.15.6:0/2142508903 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f643006b4d0 0x7f643006d990 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:03.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.506+0000 7f6457000640 1 --2- 172.21.15.6:0/2142508903 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f64500af9f0 0x7f64501607c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:03.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.506+0000 7f6457000640 1 --2- 172.21.15.6:0/2142508903 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f64500af2b0 0x7f64500b19b0 unknown :-1 s=CLOSED pgs=1174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:03.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.506+0000 7f6457000640 1 --2- 172.21.15.6:0/2142508903 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f645006afc0 0x7f6450158930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:03.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.506+0000 7f6457000640 1 -- 172.21.15.6:0/2142508903 >> 172.21.15.6:0/2142508903 conn(0x7f645006c0d0 msgr2=0x7f645006f010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:03.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.506+0000 7f6457000640 1 -- 172.21.15.6:0/2142508903 shutdown_connections 2024-03-04T09:31:03.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.506+0000 7f6457000640 1 -- 172.21.15.6:0/2142508903 wait complete. 2024-03-04T09:31:03.521 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-04T09:31:03.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.753+0000 7f3c2a000640 1 Processor -- start 2024-03-04T09:31:03.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.754+0000 7f3c2a000640 1 -- start start 2024-03-04T09:31:03.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.755+0000 7f3c2a000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f3c2415e360 0x7f3c240cf890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:03.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.755+0000 7f3c2a000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c240d1d20 con 0x7f3c240d11c0 2024-03-04T09:31:03.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.755+0000 7f3c2a000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c240d2000 con 0x7f3c240cfdd0 2024-03-04T09:31:03.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.755+0000 7f3c2a000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c240d22e0 con 0x7f3c2415e360 2024-03-04T09:31:03.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.756+0000 7f3c23e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f3c240d11c0 0x7f3c240b9250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:35304/0 (socket says 172.21.15.6:35304) 2024-03-04T09:31:03.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.756+0000 7f3c23e00640 1 -- 172.21.15.6:0/956306386 learned_addr learned my addr 172.21.15.6:0/956306386 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:31:03.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.756+0000 7f3c23400640 1 --2- 172.21.15.6:0/956306386 >> v2:172.21.15.146:3300/0 conn(0x7f3c2415e360 0x7f3c240cf890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:03.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.756+0000 7f3c22a00640 1 -- 172.21.15.6:0/956306386 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4021830171 0 0) 0x7f3c240d1d20 con 0x7f3c240d11c0 2024-03-04T09:31:03.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.756+0000 7f3c22a00640 1 -- 172.21.15.6:0/956306386 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c08003520 con 0x7f3c240d11c0 2024-03-04T09:31:03.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.757+0000 7f3c22a00640 1 -- 172.21.15.6:0/956306386 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 297482622 0 0) 0x7f3c08003520 con 0x7f3c240d11c0 2024-03-04T09:31:03.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.757+0000 7f3c22a00640 1 -- 172.21.15.6:0/956306386 >> v2:172.21.15.146:3300/0 conn(0x7f3c2415e360 msgr2=0x7f3c240cf890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:03.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.757+0000 7f3c22a00640 1 --2- 172.21.15.6:0/956306386 >> v2:172.21.15.146:3300/0 conn(0x7f3c2415e360 0x7f3c240cf890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:03.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.757+0000 7f3c22a00640 1 -- 172.21.15.6:0/956306386 >> v1:172.21.15.146:6789/0 conn(0x7f3c240cfdd0 legacy=0x7f3c240d0ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:03.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.757+0000 7f3c22a00640 1 -- 172.21.15.6:0/956306386 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c240d25c0 con 0x7f3c240d11c0 2024-03-04T09:31:03.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.757+0000 7f3c22a00640 1 -- 172.21.15.6:0/956306386 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7f3c1c0019f0 con 0x7f3c240d11c0 2024-03-04T09:31:03.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.757+0000 7f3c22a00640 1 -- 172.21.15.6:0/956306386 >> v1:172.21.15.6:6789/0 conn(0x7f3c240d11c0 legacy=0x7f3c240b9250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:03.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.757+0000 7f3c22a00640 1 --2- 172.21.15.6:0/956306386 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3c08003a00 0x7f3c08003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:03.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.757+0000 7f3c22a00640 1 --2- 172.21.15.6:0/956306386 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3c08004800 0x7f3c08004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:03.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.757+0000 7f3c22a00640 1 --2- 172.21.15.6:0/956306386 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3c080051c0 0x7f3c0800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:03.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.757+0000 7f3c22a00640 1 -- 172.21.15.6:0/956306386 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c240d22e0 con 0x7f3c08003a00 2024-03-04T09:31:03.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.757+0000 7f3c22a00640 1 -- 172.21.15.6:0/956306386 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c0800c0c0 con 0x7f3c080051c0 2024-03-04T09:31:03.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.757+0000 7f3c22a00640 1 -- 172.21.15.6:0/956306386 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c0800c2c0 con 0x7f3c08004800 2024-03-04T09:31:03.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.757+0000 7f3c23400640 1 --2- 172.21.15.6:0/956306386 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3c08003a00 0x7f3c08003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:03.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.758+0000 7f3c23e00640 1 --2- 172.21.15.6:0/956306386 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3c080051c0 0x7f3c0800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:03.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.758+0000 7f3c1be00640 1 --2- 172.21.15.6:0/956306386 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3c08004800 0x7f3c08004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:03.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.758+0000 7f3c23e00640 1 -- 172.21.15.6:0/956306386 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3c08004800 msgr2=0x7f3c08004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:03.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.758+0000 7f3c23e00640 1 --2- 172.21.15.6:0/956306386 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3c08004800 0x7f3c08004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:03.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.758+0000 7f3c23e00640 1 -- 172.21.15.6:0/956306386 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3c08003a00 msgr2=0x7f3c08003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:03.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.758+0000 7f3c23e00640 1 --2- 172.21.15.6:0/956306386 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3c08003a00 0x7f3c08003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:03.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.758+0000 7f3c23e00640 1 -- 172.21.15.6:0/956306386 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c08003520 con 0x7f3c080051c0 2024-03-04T09:31:03.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.759+0000 7f3c23e00640 1 -- 172.21.15.6:0/956306386 reap_dead start 2024-03-04T09:31:03.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.760+0000 7f3c23e00640 1 --2- 172.21.15.6:0/956306386 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3c080051c0 0x7f3c0800b9c0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f3c1c00a740 tx=0x7f3c08005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8d8068ae1cb1a8fc server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:03.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.760+0000 7f3c22a00640 1 -- 172.21.15.6:0/956306386 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c1c00a390 con 0x7f3c080051c0 2024-03-04T09:31:03.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.760+0000 7f3c22a00640 1 -- 172.21.15.6:0/956306386 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c1c034650 con 0x7f3c080051c0 2024-03-04T09:31:03.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.760+0000 7f3c22a00640 1 -- 172.21.15.6:0/956306386 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c1c034a40 con 0x7f3c080051c0 2024-03-04T09:31:03.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.761+0000 7f3c2a000640 1 -- 172.21.15.6:0/956306386 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3c080051c0 msgr2=0x7f3c0800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:03.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.761+0000 7f3c2a000640 1 --2- 172.21.15.6:0/956306386 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3c080051c0 0x7f3c0800b9c0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f3c1c00a740 tx=0x7f3c08005830 comp rx=0 tx=0).stop 2024-03-04T09:31:03.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.761+0000 7f3c2a000640 1 -- 172.21.15.6:0/956306386 shutdown_connections 2024-03-04T09:31:03.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.761+0000 7f3c2a000640 1 --2- 172.21.15.6:0/956306386 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3c080051c0 0x7f3c0800b9c0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:03.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.761+0000 7f3c2a000640 1 -- 172.21.15.6:0/956306386 >> 172.21.15.6:0/956306386 conn(0x7f3c2409d430 msgr2=0x7f3c240d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:03.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.761+0000 7f3c2a000640 1 -- 172.21.15.6:0/956306386 shutdown_connections 2024-03-04T09:31:03.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.761+0000 7f3c2a000640 1 -- 172.21.15.6:0/956306386 wait complete. 2024-03-04T09:31:03.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.763+0000 7f3c2a000640 1 Processor -- start 2024-03-04T09:31:03.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.764+0000 7f3c2a000640 1 -- start start 2024-03-04T09:31:03.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.765+0000 7f3c2a000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3c080051c0 0x7f3c2415a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:03.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.765+0000 7f3c2a000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3c240cdfc0 0x7f3c24160060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:03.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.765+0000 7f3c2a000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3c240ccd00 0x7f3c241605a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:03.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.765+0000 7f3c2a000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c240d30a0 con 0x7f3c240ccd00 2024-03-04T09:31:03.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.765+0000 7f3c2a000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c2415e740 con 0x7f3c080051c0 2024-03-04T09:31:03.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.765+0000 7f3c2a000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c240d3220 con 0x7f3c240cdfc0 2024-03-04T09:31:03.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.765+0000 7f3c23e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3c240ccd00 0x7f3c241605a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:03.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.765+0000 7f3c23e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3c240ccd00 0x7f3c241605a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51628/0 (socket says 172.21.15.6:51628) 2024-03-04T09:31:03.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.765+0000 7f3c23e00640 1 -- 172.21.15.6:0/188610547 learned_addr learned my addr 172.21.15.6:0/188610547 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:31:03.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.765+0000 7f3c23400640 1 --2- 172.21.15.6:0/188610547 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3c080051c0 0x7f3c2415a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:03.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.766+0000 7f3c1be00640 1 --2- 172.21.15.6:0/188610547 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3c240cdfc0 0x7f3c24160060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:03.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.766+0000 7f3c23e00640 1 -- 172.21.15.6:0/188610547 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3c240cdfc0 msgr2=0x7f3c24160060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:03.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.766+0000 7f3c23e00640 1 --2- 172.21.15.6:0/188610547 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3c240cdfc0 0x7f3c24160060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:03.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.766+0000 7f3c23e00640 1 -- 172.21.15.6:0/188610547 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3c080051c0 msgr2=0x7f3c2415a1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:03.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.766+0000 7f3c23e00640 1 --2- 172.21.15.6:0/188610547 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3c080051c0 0x7f3c2415a1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:03.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.766+0000 7f3c23e00640 1 -- 172.21.15.6:0/188610547 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c240cb5c0 con 0x7f3c240ccd00 2024-03-04T09:31:03.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.766+0000 7f3c1be00640 1 --2- 172.21.15.6:0/188610547 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3c240cdfc0 0x7f3c24160060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:31:03.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.766+0000 7f3c23e00640 1 --2- 172.21.15.6:0/188610547 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3c240ccd00 0x7f3c241605a0 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7f3c1c000f80 tx=0x7f3c240d1ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:03.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.766+0000 7f3c1b400640 1 -- 172.21.15.6:0/188610547 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c1c039040 con 0x7f3c240ccd00 2024-03-04T09:31:03.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.766+0000 7f3c1b400640 1 -- 172.21.15.6:0/188610547 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c1c043070 con 0x7f3c240ccd00 2024-03-04T09:31:03.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.766+0000 7f3c1b400640 1 -- 172.21.15.6:0/188610547 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c1c004d60 con 0x7f3c240ccd00 2024-03-04T09:31:03.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.766+0000 7f3c2a000640 1 -- 172.21.15.6:0/188610547 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c240cb8a0 con 0x7f3c240ccd00 2024-03-04T09:31:03.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.767+0000 7f3c2a000640 1 -- 172.21.15.6:0/188610547 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c240cbe60 con 0x7f3c240ccd00 2024-03-04T09:31:03.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.768+0000 7f3c2a000640 1 -- 172.21.15.6:0/188610547 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c240692a0 con 0x7f3c240ccd00 2024-03-04T09:31:03.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.769+0000 7f3c1b400640 1 -- 172.21.15.6:0/188610547 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3c1c034650 con 0x7f3c240ccd00 2024-03-04T09:31:03.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.771+0000 7f3c1b400640 1 --2- 172.21.15.6:0/188610547 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f3c0006b4d0 0x7f3c0006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:03.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.771+0000 7f3c1b400640 1 -- 172.21.15.6:0/188610547 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f3c1c0ac280 con 0x7f3c240ccd00 2024-03-04T09:31:03.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.771+0000 7f3c23400640 1 --2- 172.21.15.6:0/188610547 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f3c0006b4d0 0x7f3c0006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:03.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.772+0000 7f3c23400640 1 --2- 172.21.15.6:0/188610547 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f3c0006b4d0 0x7f3c0006d990 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f3c14009a80 tx=0x7f3c14008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:03.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.775+0000 7f3c1b400640 1 -- 172.21.15.6:0/188610547 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3c1c0333d0 con 0x7f3c240ccd00 2024-03-04T09:31:03.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.976+0000 7f3c2a000640 1 -- 172.21.15.6:0/188610547 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f3c240d1760 con 0x7f3c240ccd00 2024-03-04T09:31:03.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.977+0000 7f3c1b400640 1 -- 172.21.15.6:0/188610547 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v98) v1 ==== 100+0+11 (secure 0 0 0) 0x7f3c1c07b050 con 0x7f3c240ccd00 2024-03-04T09:31:03.974 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 64 2024-03-04T09:31:03.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.979+0000 7f3c2a000640 1 -- 172.21.15.6:0/188610547 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f3c0006b4d0 msgr2=0x7f3c0006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:03.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.979+0000 7f3c2a000640 1 --2- 172.21.15.6:0/188610547 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f3c0006b4d0 0x7f3c0006d990 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f3c14009a80 tx=0x7f3c14008040 comp rx=0 tx=0).stop 2024-03-04T09:31:03.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.979+0000 7f3c2a000640 1 -- 172.21.15.6:0/188610547 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3c240ccd00 msgr2=0x7f3c241605a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:03.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.979+0000 7f3c2a000640 1 --2- 172.21.15.6:0/188610547 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3c240ccd00 0x7f3c241605a0 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7f3c1c000f80 tx=0x7f3c240d1ef0 comp rx=0 tx=0).stop 2024-03-04T09:31:03.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.979+0000 7f3c2a000640 1 -- 172.21.15.6:0/188610547 shutdown_connections 2024-03-04T09:31:03.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.979+0000 7f3c2a000640 1 --2- 172.21.15.6:0/188610547 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3c240ccd00 0x7f3c241605a0 unknown :-1 s=CLOSED pgs=1176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:03.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.979+0000 7f3c2a000640 1 --2- 172.21.15.6:0/188610547 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f3c0006b4d0 0x7f3c0006d990 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:03.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.979+0000 7f3c2a000640 1 --2- 172.21.15.6:0/188610547 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3c240cdfc0 0x7f3c24160060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:03.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.979+0000 7f3c2a000640 1 --2- 172.21.15.6:0/188610547 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3c080051c0 0x7f3c2415a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:03.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.979+0000 7f3c2a000640 1 -- 172.21.15.6:0/188610547 >> 172.21.15.6:0/188610547 conn(0x7f3c2409d430 msgr2=0x7f3c240c9d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:03.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.980+0000 7f3c2a000640 1 -- 172.21.15.6:0/188610547 shutdown_connections 2024-03-04T09:31:03.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:03.980+0000 7f3c2a000640 1 -- 172.21.15.6:0/188610547 wait complete. 2024-03-04T09:31:03.995 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:31:04.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.201+0000 7f3335200640 1 Processor -- start 2024-03-04T09:31:04.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.202+0000 7f3335200640 1 -- start start 2024-03-04T09:31:04.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.203+0000 7f3335200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3330147770 0x7f3330071130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:04.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.203+0000 7f3335200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f3330147b50 0x7f3330161c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:04.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.203+0000 7f3335200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33300717f0 con 0x7f3330162180 2024-03-04T09:31:04.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.203+0000 7f3335200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3330072400 con 0x7f3330147b50 2024-03-04T09:31:04.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.203+0000 7f3335200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3330071ad0 con 0x7f3330147770 2024-03-04T09:31:04.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.204+0000 7f3325400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f3330147b50 0x7f3330161c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:04.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.204+0000 7f3325400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f3330147b50 0x7f3330161c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51632/0 (socket says 172.21.15.6:51632) 2024-03-04T09:31:04.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.204+0000 7f3325400640 1 -- 172.21.15.6:0/1952905787 learned_addr learned my addr 172.21.15.6:0/1952905787 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:31:04.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.204+0000 7f332de00640 1 --2- 172.21.15.6:0/1952905787 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3330147770 0x7f3330071130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:04.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.204+0000 7f3325400640 1 -- 172.21.15.6:0/1952905787 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3330147770 msgr2=0x7f3330071130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:04.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.204+0000 7f3325400640 1 --2- 172.21.15.6:0/1952905787 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3330147770 0x7f3330071130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:04.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.204+0000 7f3325400640 1 -- 172.21.15.6:0/1952905787 >> v1:172.21.15.6:6789/0 conn(0x7f3330162180 legacy=0x7f33300b4bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:04.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.204+0000 7f3325400640 1 -- 172.21.15.6:0/1952905787 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3330071db0 con 0x7f3330147b50 2024-03-04T09:31:04.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.206+0000 7f3325400640 1 --2- 172.21.15.6:0/1952905787 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3330147b50 0x7f3330161c40 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7f3318009870 tx=0x7f331802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a70a6a399d60d9a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:04.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.206+0000 7f332d400640 1 -- 172.21.15.6:0/1952905787 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3318034070 con 0x7f3330147b50 2024-03-04T09:31:04.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.206+0000 7f332d400640 1 -- 172.21.15.6:0/1952905787 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3318030840 con 0x7f3330147b50 2024-03-04T09:31:04.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.206+0000 7f332d400640 1 -- 172.21.15.6:0/1952905787 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3318030c10 con 0x7f3330147b50 2024-03-04T09:31:04.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.206+0000 7f3335200640 1 -- 172.21.15.6:0/1952905787 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3330147b50 msgr2=0x7f3330161c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:04.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.206+0000 7f3335200640 1 --2- 172.21.15.6:0/1952905787 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3330147b50 0x7f3330161c40 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7f3318009870 tx=0x7f331802b980 comp rx=0 tx=0).stop 2024-03-04T09:31:04.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.207+0000 7f3335200640 1 -- 172.21.15.6:0/1952905787 shutdown_connections 2024-03-04T09:31:04.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.207+0000 7f3335200640 1 --2- 172.21.15.6:0/1952905787 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3330147b50 0x7f3330161c40 unknown :-1 s=CLOSED pgs=1178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:04.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.207+0000 7f3335200640 1 --2- 172.21.15.6:0/1952905787 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3330147770 0x7f3330071130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:04.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.207+0000 7f3335200640 1 -- 172.21.15.6:0/1952905787 >> 172.21.15.6:0/1952905787 conn(0x7f333006cf40 msgr2=0x7f333006d320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:04.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.207+0000 7f3335200640 1 -- 172.21.15.6:0/1952905787 shutdown_connections 2024-03-04T09:31:04.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.207+0000 7f3335200640 1 -- 172.21.15.6:0/1952905787 wait complete. 2024-03-04T09:31:04.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.208+0000 7f3335200640 1 Processor -- start 2024-03-04T09:31:04.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.209+0000 7f3335200640 1 -- start start 2024-03-04T09:31:04.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.209+0000 7f3335200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3330147770 0x7f33300c4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:04.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.209+0000 7f3335200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3330162180 0x7f3330141e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:04.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.209+0000 7f3335200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f33301e9cd0 0x7f33301e8450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:04.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.209+0000 7f3335200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33300aea40 con 0x7f3330147770 2024-03-04T09:31:04.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.209+0000 7f3335200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33300ae8c0 con 0x7f3330162180 2024-03-04T09:31:04.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.209+0000 7f3335200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33300aebc0 con 0x7f33301e9cd0 2024-03-04T09:31:04.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.210+0000 7f332de00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3330147770 0x7f33300c4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:04.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.210+0000 7f332de00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3330147770 0x7f33300c4f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51640/0 (socket says 172.21.15.6:51640) 2024-03-04T09:31:04.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.210+0000 7f332de00640 1 -- 172.21.15.6:0/2083480684 learned_addr learned my addr 172.21.15.6:0/2083480684 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:31:04.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.210+0000 7f3325400640 1 --2- 172.21.15.6:0/2083480684 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3330162180 0x7f3330141e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:04.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.210+0000 7f332de00640 1 -- 172.21.15.6:0/2083480684 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f33301e9cd0 msgr2=0x7f33301e8450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:04.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.210+0000 7f332e800640 1 --2- 172.21.15.6:0/2083480684 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f33301e9cd0 0x7f33301e8450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:04.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.210+0000 7f332de00640 1 --2- 172.21.15.6:0/2083480684 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f33301e9cd0 0x7f33301e8450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:04.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.210+0000 7f332de00640 1 -- 172.21.15.6:0/2083480684 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3330162180 msgr2=0x7f3330141e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:04.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.210+0000 7f332de00640 1 --2- 172.21.15.6:0/2083480684 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3330162180 0x7f3330141e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:04.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.210+0000 7f332de00640 1 -- 172.21.15.6:0/2083480684 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33301ea160 con 0x7f3330147770 2024-03-04T09:31:04.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.210+0000 7f3325400640 1 --2- 172.21.15.6:0/2083480684 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3330162180 0x7f3330141e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:31:04.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.210+0000 7f332e800640 1 --2- 172.21.15.6:0/2083480684 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f33301e9cd0 0x7f33301e8450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:31:04.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.211+0000 7f332de00640 1 --2- 172.21.15.6:0/2083480684 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3330147770 0x7f33300c4f50 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7f332000e9b0 tx=0x7f332000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:04.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.211+0000 7f3326a00640 1 -- 172.21.15.6:0/2083480684 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3320017070 con 0x7f3330147770 2024-03-04T09:31:04.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.211+0000 7f3326a00640 1 -- 172.21.15.6:0/2083480684 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f332000c9a0 con 0x7f3330147770 2024-03-04T09:31:04.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.211+0000 7f3326a00640 1 -- 172.21.15.6:0/2083480684 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f332000cd90 con 0x7f3330147770 2024-03-04T09:31:04.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.211+0000 7f3335200640 1 -- 172.21.15.6:0/2083480684 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33301e8990 con 0x7f3330147770 2024-03-04T09:31:04.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.211+0000 7f3335200640 1 -- 172.21.15.6:0/2083480684 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33301e8f10 con 0x7f3330147770 2024-03-04T09:31:04.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.213+0000 7f3335200640 1 -- 172.21.15.6:0/2083480684 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33300692a0 con 0x7f3330147770 2024-03-04T09:31:04.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.214+0000 7f3326a00640 1 -- 172.21.15.6:0/2083480684 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f332001a070 con 0x7f3330147770 2024-03-04T09:31:04.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.215+0000 7f3326a00640 1 --2- 172.21.15.6:0/2083480684 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f330c06b4d0 0x7f330c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:04.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.215+0000 7f3326a00640 1 -- 172.21.15.6:0/2083480684 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f332008a340 con 0x7f3330147770 2024-03-04T09:31:04.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.216+0000 7f3325400640 1 --2- 172.21.15.6:0/2083480684 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f330c06b4d0 0x7f330c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:04.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.217+0000 7f3325400640 1 --2- 172.21.15.6:0/2083480684 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f330c06b4d0 0x7f330c06d990 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f3318036000 tx=0x7f331803b040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:04.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.220+0000 7f3326a00640 1 -- 172.21.15.6:0/2083480684 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f33301ea160 con 0x7f3330147770 2024-03-04T09:31:04.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.461+0000 7f3335200640 1 -- 172.21.15.6:0/2083480684 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f333006e010 con 0x7f3330147770 2024-03-04T09:31:04.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.462+0000 7f3326a00640 1 -- 172.21.15.6:0/2083480684 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) v1 ==== 94+0+4977 (secure 0 0 0) 0x7f3320059110 con 0x7f3330147770 2024-03-04T09:31:04.459 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:31:04.459 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":68,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5654,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/1370302030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":1370302030},{"type":"v1","addr":"172.21.15.146:6835","nonce":1370302030}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":6163,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/809822004","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":809822004},{"type":"v1","addr":"172.21.15.146:6837","nonce":809822004}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":66},{"gid":7284,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/1583413840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1583413840},{"type":"v1","addr":"172.21.15.6:6833","nonce":1583413840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64}],"filesystems":[{"mdsmap":{"epoch":68,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:30:20.660103+0000","modified":"2024-03-04T09:30:36.289706+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":98,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7296},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7296":{"gid":7296,"name":"a","rank":0,"incarnation":65,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1732383836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1732383836},{"type":"v1","addr":"172.21.15.6:6835","nonce":1732383836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7296,"qdb_cluster":[7296]},"id":7}]} 2024-03-04T09:31:04.459 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 68 2024-03-04T09:31:04.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.464+0000 7f3335200640 1 -- 172.21.15.6:0/2083480684 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f330c06b4d0 msgr2=0x7f330c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:04.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.464+0000 7f3335200640 1 --2- 172.21.15.6:0/2083480684 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f330c06b4d0 0x7f330c06d990 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f3318036000 tx=0x7f331803b040 comp rx=0 tx=0).stop 2024-03-04T09:31:04.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.464+0000 7f3335200640 1 -- 172.21.15.6:0/2083480684 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3330147770 msgr2=0x7f33300c4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:04.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.464+0000 7f3335200640 1 --2- 172.21.15.6:0/2083480684 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3330147770 0x7f33300c4f50 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7f332000e9b0 tx=0x7f332000ee70 comp rx=0 tx=0).stop 2024-03-04T09:31:04.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.464+0000 7f3335200640 1 -- 172.21.15.6:0/2083480684 shutdown_connections 2024-03-04T09:31:04.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.464+0000 7f3335200640 1 --2- 172.21.15.6:0/2083480684 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f330c06b4d0 0x7f330c06d990 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:04.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.464+0000 7f3335200640 1 --2- 172.21.15.6:0/2083480684 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f33301e9cd0 0x7f33301e8450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:04.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.464+0000 7f3335200640 1 --2- 172.21.15.6:0/2083480684 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3330162180 0x7f3330141e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:04.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.464+0000 7f3335200640 1 --2- 172.21.15.6:0/2083480684 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3330147770 0x7f33300c4f50 unknown :-1 s=CLOSED pgs=1179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:04.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.464+0000 7f3335200640 1 -- 172.21.15.6:0/2083480684 >> 172.21.15.6:0/2083480684 conn(0x7f333006cf40 msgr2=0x7f33300aed40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:04.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.465+0000 7f3335200640 1 -- 172.21.15.6:0/2083480684 shutdown_connections 2024-03-04T09:31:04.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.465+0000 7f3335200640 1 -- 172.21.15.6:0/2083480684 wait complete. 2024-03-04T09:31:04.489 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T09:31:04.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.698+0000 7fd236a00640 1 Processor -- start 2024-03-04T09:31:04.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.699+0000 7fd236a00640 1 -- start start 2024-03-04T09:31:04.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.700+0000 7fd236a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd230147b50 0x7fd230071130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:04.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.700+0000 7fd236a00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd230162350 0x7fd2300ad8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:04.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.700+0000 7fd236a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2300717f0 con 0x7fd230147770 2024-03-04T09:31:04.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.700+0000 7fd236a00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd230072400 con 0x7fd230162350 2024-03-04T09:31:04.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.700+0000 7fd236a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd230071ad0 con 0x7fd230147b50 2024-03-04T09:31:04.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.700+0000 7fd22fe00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fd230147770 0x7fd230161c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:35324/0 (socket says 172.21.15.6:35324) 2024-03-04T09:31:04.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.700+0000 7fd22fe00640 1 -- 172.21.15.6:0/3897390692 learned_addr learned my addr 172.21.15.6:0/3897390692 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:31:04.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.701+0000 7fd234a00640 1 --2- 172.21.15.6:0/3897390692 >> v2:172.21.15.146:3300/0 conn(0x7fd230162350 0x7fd2300ad8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:04.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.701+0000 7fd227400640 1 --2- 172.21.15.6:0/3897390692 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd230147b50 0x7fd230071130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:04.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.701+0000 7fd22f400640 1 -- 172.21.15.6:0/3897390692 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2105205906 0 0) 0x7fd2300717f0 con 0x7fd230147770 2024-03-04T09:31:04.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.701+0000 7fd22f400640 1 -- 172.21.15.6:0/3897390692 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd214003520 con 0x7fd230147770 2024-03-04T09:31:04.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.702+0000 7fd22f400640 1 -- 172.21.15.6:0/3897390692 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1598972629 0 0) 0x7fd214003520 con 0x7fd230147770 2024-03-04T09:31:04.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.702+0000 7fd22f400640 1 -- 172.21.15.6:0/3897390692 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd230147b50 msgr2=0x7fd230071130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:04.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.702+0000 7fd22f400640 1 --2- 172.21.15.6:0/3897390692 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd230147b50 0x7fd230071130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:04.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.702+0000 7fd22f400640 1 -- 172.21.15.6:0/3897390692 >> v2:172.21.15.146:3300/0 conn(0x7fd230162350 msgr2=0x7fd2300ad8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:04.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.702+0000 7fd22f400640 1 --2- 172.21.15.6:0/3897390692 >> v2:172.21.15.146:3300/0 conn(0x7fd230162350 0x7fd2300ad8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:04.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.702+0000 7fd22f400640 1 -- 172.21.15.6:0/3897390692 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd230071db0 con 0x7fd230147770 2024-03-04T09:31:04.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.702+0000 7fd22f400640 1 -- 172.21.15.6:0/3897390692 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7fd2200019f0 con 0x7fd230147770 2024-03-04T09:31:04.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.702+0000 7fd22f400640 1 -- 172.21.15.6:0/3897390692 >> v1:172.21.15.6:6789/0 conn(0x7fd230147770 legacy=0x7fd230161c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:04.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.702+0000 7fd22f400640 1 --2- 172.21.15.6:0/3897390692 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd2140039b0 0x7fd214003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:04.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.702+0000 7fd22f400640 1 --2- 172.21.15.6:0/3897390692 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd214004310 0x7fd214004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:04.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.702+0000 7fd22f400640 1 --2- 172.21.15.6:0/3897390692 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd214004e50 0x7fd2140052d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:04.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.702+0000 7fd22f400640 1 -- 172.21.15.6:0/3897390692 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd230071ad0 con 0x7fd214004310 2024-03-04T09:31:04.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.702+0000 7fd22f400640 1 -- 172.21.15.6:0/3897390692 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd21400bcc0 con 0x7fd214004e50 2024-03-04T09:31:04.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.702+0000 7fd22f400640 1 -- 172.21.15.6:0/3897390692 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd21400bec0 con 0x7fd2140039b0 2024-03-04T09:31:04.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.702+0000 7fd227400640 1 --2- 172.21.15.6:0/3897390692 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd214004310 0x7fd214004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:04.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.703+0000 7fd22fe00640 1 --2- 172.21.15.6:0/3897390692 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd2140039b0 0x7fd214003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:04.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.703+0000 7fd227400640 1 -- 172.21.15.6:0/3897390692 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd2140039b0 msgr2=0x7fd214003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:04.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.703+0000 7fd227400640 1 --2- 172.21.15.6:0/3897390692 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd2140039b0 0x7fd214003d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:04.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.703+0000 7fd227400640 1 -- 172.21.15.6:0/3897390692 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd214004e50 msgr2=0x7fd2140052d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:04.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.703+0000 7fd234a00640 1 --2- 172.21.15.6:0/3897390692 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd214004e50 0x7fd2140052d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:04.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.703+0000 7fd227400640 1 --2- 172.21.15.6:0/3897390692 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd214004e50 0x7fd2140052d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:04.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.703+0000 7fd227400640 1 -- 172.21.15.6:0/3897390692 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd214003520 con 0x7fd214004310 2024-03-04T09:31:04.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.704+0000 7fd227400640 1 --2- 172.21.15.6:0/3897390692 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd214004310 0x7fd214004800 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7fd218009ac0 tx=0x7fd214005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=630e6a0e02717fe6 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:04.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.704+0000 7fd22f400640 1 -- 172.21.15.6:0/3897390692 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd218039070 con 0x7fd214004310 2024-03-04T09:31:04.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.704+0000 7fd22f400640 1 -- 172.21.15.6:0/3897390692 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd21802c6b0 con 0x7fd214004310 2024-03-04T09:31:04.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.704+0000 7fd22f400640 1 -- 172.21.15.6:0/3897390692 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd218009690 con 0x7fd214004310 2024-03-04T09:31:04.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.705+0000 7fd236a00640 1 -- 172.21.15.6:0/3897390692 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd214004310 msgr2=0x7fd214004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:04.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.705+0000 7fd236a00640 1 --2- 172.21.15.6:0/3897390692 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd214004310 0x7fd214004800 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7fd218009ac0 tx=0x7fd214005830 comp rx=0 tx=0).stop 2024-03-04T09:31:04.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.705+0000 7fd234a00640 1 -- 172.21.15.6:0/3897390692 reap_dead start 2024-03-04T09:31:04.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.705+0000 7fd236a00640 1 -- 172.21.15.6:0/3897390692 shutdown_connections 2024-03-04T09:31:04.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.705+0000 7fd236a00640 1 -- 172.21.15.6:0/3897390692 >> 172.21.15.6:0/3897390692 conn(0x7fd23006cf40 msgr2=0x7fd23006d320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:04.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.705+0000 7fd236a00640 1 -- 172.21.15.6:0/3897390692 shutdown_connections 2024-03-04T09:31:04.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.705+0000 7fd236a00640 1 -- 172.21.15.6:0/3897390692 wait complete. 2024-03-04T09:31:04.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.709+0000 7fd236a00640 1 Processor -- start 2024-03-04T09:31:04.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.709+0000 7fd236a00640 1 -- start start 2024-03-04T09:31:04.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.711+0000 7fd236a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd214004310 0x7fd23015d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:04.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.711+0000 7fd236a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd230144ef0 0x7fd230145370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:04.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.711+0000 7fd236a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd2300c9150 0x7fd2300c95d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:04.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.711+0000 7fd236a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd230162980 con 0x7fd2300c9150 2024-03-04T09:31:04.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.711+0000 7fd236a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd230162800 con 0x7fd230144ef0 2024-03-04T09:31:04.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.711+0000 7fd236a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd230162b00 con 0x7fd214004310 2024-03-04T09:31:04.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.711+0000 7fd234a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd2300c9150 0x7fd2300c95d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:04.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.711+0000 7fd234a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd2300c9150 0x7fd2300c95d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51654/0 (socket says 172.21.15.6:51654) 2024-03-04T09:31:04.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.711+0000 7fd234a00640 1 -- 172.21.15.6:0/460643144 learned_addr learned my addr 172.21.15.6:0/460643144 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:31:04.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.712+0000 7fd22fe00640 1 --2- 172.21.15.6:0/460643144 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd214004310 0x7fd23015d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:04.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.712+0000 7fd227400640 1 --2- 172.21.15.6:0/460643144 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd230144ef0 0x7fd230145370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:04.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.712+0000 7fd234a00640 1 -- 172.21.15.6:0/460643144 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd214004310 msgr2=0x7fd23015d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:04.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.712+0000 7fd234a00640 1 --2- 172.21.15.6:0/460643144 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd214004310 0x7fd23015d600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:04.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.712+0000 7fd234a00640 1 -- 172.21.15.6:0/460643144 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd230144ef0 msgr2=0x7fd230145370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:04.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.712+0000 7fd234a00640 1 --2- 172.21.15.6:0/460643144 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd230144ef0 0x7fd230145370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:04.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.712+0000 7fd234a00640 1 -- 172.21.15.6:0/460643144 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2300c6ad0 con 0x7fd2300c9150 2024-03-04T09:31:04.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.712+0000 7fd227400640 1 --2- 172.21.15.6:0/460643144 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd230144ef0 0x7fd230145370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:31:04.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.712+0000 7fd234a00640 1 --2- 172.21.15.6:0/460643144 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd2300c9150 0x7fd2300c95d0 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7fd230162730 tx=0x7fd228010c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:04.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.712+0000 7fd22cc00640 1 -- 172.21.15.6:0/460643144 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd22800f040 con 0x7fd2300c9150 2024-03-04T09:31:04.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.713+0000 7fd22cc00640 1 -- 172.21.15.6:0/460643144 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd22801c070 con 0x7fd2300c9150 2024-03-04T09:31:04.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.713+0000 7fd22cc00640 1 -- 172.21.15.6:0/460643144 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd22800eaa0 con 0x7fd2300c9150 2024-03-04T09:31:04.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.713+0000 7fd236a00640 1 -- 172.21.15.6:0/460643144 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2300c6db0 con 0x7fd2300c9150 2024-03-04T09:31:04.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.713+0000 7fd236a00640 1 -- 172.21.15.6:0/460643144 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2300c9b10 con 0x7fd2300c9150 2024-03-04T09:31:04.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.714+0000 7fd236a00640 1 -- 172.21.15.6:0/460643144 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2300747e0 con 0x7fd2300c9150 2024-03-04T09:31:04.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.715+0000 7fd22cc00640 1 -- 172.21.15.6:0/460643144 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd22800ec40 con 0x7fd2300c9150 2024-03-04T09:31:04.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.716+0000 7fd22cc00640 1 --2- 172.21.15.6:0/460643144 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd20c06b4d0 0x7fd20c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:04.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.716+0000 7fd22cc00640 1 -- 172.21.15.6:0/460643144 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5881+0+0 (secure 0 0 0) 0x7fd22808a200 con 0x7fd2300c9150 2024-03-04T09:31:04.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.717+0000 7fd22fe00640 1 --2- 172.21.15.6:0/460643144 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd20c06b4d0 0x7fd20c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:04.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.718+0000 7fd22fe00640 1 --2- 172.21.15.6:0/460643144 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd20c06b4d0 0x7fd20c06d990 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fd220001330 tx=0x7fd220009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:04.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.721+0000 7fd22cc00640 1 -- 172.21.15.6:0/460643144 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd2280588e0 con 0x7fd2300c9150 2024-03-04T09:31:04.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.921+0000 7fd236a00640 1 -- 172.21.15.6:0/460643144 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd2300692a0 con 0x7fd2300c9150 2024-03-04T09:31:04.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.924+0000 7fd22cc00640 1 -- 172.21.15.6:0/460643144 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v98) v1 ==== 74+0+16817 (secure 0 0 0) 0x7fd22805c980 con 0x7fd2300c9150 2024-03-04T09:31:04.921 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:31:04.922 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":98,"fsid":"edce0578-5ae4-4402-b45a-552bdef84d7c","created":"2024-03-04T09:24:54.523780+0000","modified":"2024-03-04T09:30:33.260648+0000","last_up_change":"2024-03-04T09:25:07.089049+0000","last_in_change":"2024-03-04T09:24:57.982077+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T09:25:07.935198+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-03-04T09:30:18.510767+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"93","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-03-04T09:30:19.686114+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"94","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a96574f-81f1-4bc6-8463-757520b84a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6825","nonce":1364449401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6827","nonce":1364449401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6831","nonce":1364449401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6829","nonce":1364449401}]},"public_addr":"172.21.15.6:6825/1364449401","cluster_addr":"172.21.15.6:6827/1364449401","heartbeat_back_addr":"172.21.15.6:6831/1364449401","heartbeat_front_addr":"172.21.15.6:6829/1364449401","state":["exists","up"]},{"osd":1,"uuid":"7d79b9c1-6b87-476a-bc36-7db5ee64cd90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6809","nonce":2494103078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6811","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6813","nonce":2494103078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6821","nonce":2494103078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6817","nonce":2494103078}]},"public_addr":"172.21.15.6:6809/2494103078","cluster_addr":"172.21.15.6:6813/2494103078","heartbeat_back_addr":"172.21.15.6:6821/2494103078","heartbeat_front_addr":"172.21.15.6:6817/2494103078","state":["exists","up"]},{"osd":2,"uuid":"492331b9-4ee9-40b6-bacd-85d44cd6c45a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6812","nonce":1500066681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6816","nonce":1500066681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6824","nonce":1500066681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6820","nonce":1500066681}]},"public_addr":"172.21.15.6:6812/1500066681","cluster_addr":"172.21.15.6:6816/1500066681","heartbeat_back_addr":"172.21.15.6:6824/1500066681","heartbeat_front_addr":"172.21.15.6:6820/1500066681","state":["exists","up"]},{"osd":3,"uuid":"21bb2efe-9ba7-4bdc-bc31-180cd1102e21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6801","nonce":1558806123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6803","nonce":1558806123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6807","nonce":1558806123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6805","nonce":1558806123}]},"public_addr":"172.21.15.6:6801/1558806123","cluster_addr":"172.21.15.6:6803/1558806123","heartbeat_back_addr":"172.21.15.6:6807/1558806123","heartbeat_front_addr":"172.21.15.6:6805/1558806123","state":["exists","up"]},{"osd":4,"uuid":"6e2c4b71-19d9-4737-83d9-558705d8644a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6801","nonce":1526135780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6803","nonce":1526135780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6807","nonce":1526135780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6805","nonce":1526135780}]},"public_addr":"172.21.15.146:6801/1526135780","cluster_addr":"172.21.15.146:6803/1526135780","heartbeat_back_addr":"172.21.15.146:6807/1526135780","heartbeat_front_addr":"172.21.15.146:6805/1526135780","state":["exists","up"]},{"osd":5,"uuid":"fb69c525-f1be-4b7d-ab32-6c29b0d6fddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6809","nonce":460262507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6811","nonce":460262507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6815","nonce":460262507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6813","nonce":460262507}]},"public_addr":"172.21.15.146:6809/460262507","cluster_addr":"172.21.15.146:6811/460262507","heartbeat_back_addr":"172.21.15.146:6815/460262507","heartbeat_front_addr":"172.21.15.146:6813/460262507","state":["exists","up"]},{"osd":6,"uuid":"c32e7fed-370d-4401-8ce0-6a4335b0d168","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6823","nonce":1392897493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6827","nonce":1392897493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6831","nonce":1392897493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6829","nonce":1392897493}]},"public_addr":"172.21.15.146:6823/1392897493","cluster_addr":"172.21.15.146:6827/1392897493","heartbeat_back_addr":"172.21.15.146:6831/1392897493","heartbeat_front_addr":"172.21.15.146:6829/1392897493","state":["exists","up"]},{"osd":7,"uuid":"75a79b44-fa5f-426d-a7d3-71f21f78835c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6817","nonce":3268323156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6819","nonce":3268323156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6825","nonce":3268323156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6822","nonce":3268323156}]},"public_addr":"172.21.15.146:6817/3268323156","cluster_addr":"172.21.15.146:6819/3268323156","heartbeat_back_addr":"172.21.15.146:6825/3268323156","heartbeat_front_addr":"172.21.15.146:6822/3268323156","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.310437+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.295850+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.294674+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.526855+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.300023+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.595160+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.346046+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.558741+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.146:6835/847815316":"2024-03-05T09:30:33.260598+0000","172.21.15.146:6834/847815316":"2024-03-05T09:30:33.260598+0000","172.21.15.6:6835/2836916196":"2024-03-05T09:30:32.267572+0000","172.21.15.6:6832/2836916196":"2024-03-05T09:30:32.267572+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T09:31:04.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.927+0000 7fd236a00640 1 -- 172.21.15.6:0/460643144 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd20c06b4d0 msgr2=0x7fd20c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:04.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.927+0000 7fd236a00640 1 --2- 172.21.15.6:0/460643144 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd20c06b4d0 0x7fd20c06d990 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fd220001330 tx=0x7fd220009040 comp rx=0 tx=0).stop 2024-03-04T09:31:04.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.927+0000 7fd236a00640 1 -- 172.21.15.6:0/460643144 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd2300c9150 msgr2=0x7fd2300c95d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:04.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.927+0000 7fd236a00640 1 --2- 172.21.15.6:0/460643144 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd2300c9150 0x7fd2300c95d0 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7fd230162730 tx=0x7fd228010c20 comp rx=0 tx=0).stop 2024-03-04T09:31:04.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.927+0000 7fd236a00640 1 -- 172.21.15.6:0/460643144 shutdown_connections 2024-03-04T09:31:04.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.927+0000 7fd236a00640 1 --2- 172.21.15.6:0/460643144 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd2300c9150 0x7fd2300c95d0 unknown :-1 s=CLOSED pgs=1182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:04.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.927+0000 7fd236a00640 1 --2- 172.21.15.6:0/460643144 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd230144ef0 0x7fd230145370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:04.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.927+0000 7fd236a00640 1 --2- 172.21.15.6:0/460643144 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd20c06b4d0 0x7fd20c06d990 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:04.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.927+0000 7fd236a00640 1 --2- 172.21.15.6:0/460643144 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd214004310 0x7fd23015d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:04.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.927+0000 7fd236a00640 1 -- 172.21.15.6:0/460643144 >> 172.21.15.6:0/460643144 conn(0x7fd23006cf40 msgr2=0x7fd2300b2bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:04.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.927+0000 7fd236a00640 1 -- 172.21.15.6:0/460643144 shutdown_connections 2024-03-04T09:31:04.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:04.927+0000 7fd236a00640 1 -- 172.21.15.6:0/460643144 wait complete. 2024-03-04T09:31:04.944 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-04T09:31:04.944 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:31:05.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.153+0000 7f9c2f200640 1 Processor -- start 2024-03-04T09:31:05.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.154+0000 7f9c2f200640 1 -- start start 2024-03-04T09:31:05.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.155+0000 7f9c2f200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9c2815f630 0x7f9c28161ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:05.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.155+0000 7f9c2f200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f9c280b4b30 0x7f9c281621e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:05.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.155+0000 7f9c2f200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c280b50d0 con 0x7f9c28162720 2024-03-04T09:31:05.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.155+0000 7f9c2f200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c280cf450 con 0x7f9c280b4b30 2024-03-04T09:31:05.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.155+0000 7f9c2f200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c280cf6a0 con 0x7f9c2815f630 2024-03-04T09:31:05.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.155+0000 7f9c2d400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f9c280b4b30 0x7f9c281621e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:05.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.156+0000 7f9c2d400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f9c280b4b30 0x7f9c281621e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.6:46744/0 (socket says 172.21.15.6:46744) 2024-03-04T09:31:05.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.156+0000 7f9c2d400640 1 -- 172.21.15.6:0/2618833537 learned_addr learned my addr 172.21.15.6:0/2618833537 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:31:05.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.156+0000 7f9c2de00640 1 --2- 172.21.15.6:0/2618833537 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9c2815f630 0x7f9c28161ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:05.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.156+0000 7f9c2d400640 1 -- 172.21.15.6:0/2618833537 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9c2815f630 msgr2=0x7f9c28161ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:05.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.156+0000 7f9c2d400640 1 --2- 172.21.15.6:0/2618833537 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9c2815f630 0x7f9c28161ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:05.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.156+0000 7f9c2d400640 1 -- 172.21.15.6:0/2618833537 >> v1:172.21.15.146:6789/0 conn(0x7f9c28162720 legacy=0x7f9c280b58b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:05.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.156+0000 7f9c2d400640 1 -- 172.21.15.6:0/2618833537 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c280cf980 con 0x7f9c280b4b30 2024-03-04T09:31:05.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.158+0000 7f9c2d400640 1 --2- 172.21.15.6:0/2618833537 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9c280b4b30 0x7f9c281621e0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f9c18009870 tx=0x7f9c1802b860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5febbbe55b4c29d4 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:05.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.158+0000 7f9c2ca00640 1 -- 172.21.15.6:0/2618833537 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c180317c0 con 0x7f9c280b4b30 2024-03-04T09:31:05.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.158+0000 7f9c2ca00640 1 -- 172.21.15.6:0/2618833537 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c18037070 con 0x7f9c280b4b30 2024-03-04T09:31:05.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.158+0000 7f9c2ca00640 1 -- 172.21.15.6:0/2618833537 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c18031b80 con 0x7f9c280b4b30 2024-03-04T09:31:05.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.159+0000 7f9c2f200640 1 -- 172.21.15.6:0/2618833537 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9c280b4b30 msgr2=0x7f9c281621e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:05.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.159+0000 7f9c2f200640 1 --2- 172.21.15.6:0/2618833537 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9c280b4b30 0x7f9c281621e0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f9c18009870 tx=0x7f9c1802b860 comp rx=0 tx=0).stop 2024-03-04T09:31:05.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.159+0000 7f9c2f200640 1 -- 172.21.15.6:0/2618833537 shutdown_connections 2024-03-04T09:31:05.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.159+0000 7f9c2f200640 1 --2- 172.21.15.6:0/2618833537 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9c280b4b30 0x7f9c281621e0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:05.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.159+0000 7f9c2f200640 1 --2- 172.21.15.6:0/2618833537 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9c2815f630 0x7f9c28161ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:05.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.159+0000 7f9c2f200640 1 -- 172.21.15.6:0/2618833537 >> 172.21.15.6:0/2618833537 conn(0x7f9c280d3e30 msgr2=0x7f9c280d4240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:05.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.159+0000 7f9c2f200640 1 -- 172.21.15.6:0/2618833537 shutdown_connections 2024-03-04T09:31:05.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.159+0000 7f9c2f200640 1 -- 172.21.15.6:0/2618833537 wait complete. 2024-03-04T09:31:05.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.160+0000 7f9c2f200640 1 Processor -- start 2024-03-04T09:31:05.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.161+0000 7f9c2f200640 1 -- start start 2024-03-04T09:31:05.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.162+0000 7f9c2f200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9c280b4b30 0x7f9c280b9620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:05.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.162+0000 7f9c2f200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9c2815f630 0x7f9c280cd5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:05.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.162+0000 7f9c2f200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9c28162720 0x7f9c28156a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:05.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.162+0000 7f9c2f200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c280d2bf0 con 0x7f9c2815f630 2024-03-04T09:31:05.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.162+0000 7f9c2f200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c280d4c40 con 0x7f9c28162720 2024-03-04T09:31:05.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.162+0000 7f9c2f200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c280d2d70 con 0x7f9c280b4b30 2024-03-04T09:31:05.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.162+0000 7f9c2d400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9c2815f630 0x7f9c280cd5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:05.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.162+0000 7f9c2d400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9c2815f630 0x7f9c280cd5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51664/0 (socket says 172.21.15.6:51664) 2024-03-04T09:31:05.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.162+0000 7f9c2d400640 1 -- 172.21.15.6:0/2648335901 learned_addr learned my addr 172.21.15.6:0/2648335901 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:31:05.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.162+0000 7f9c2e800640 1 --2- 172.21.15.6:0/2648335901 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9c28162720 0x7f9c28156a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:05.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.162+0000 7f9c2de00640 1 --2- 172.21.15.6:0/2648335901 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9c280b4b30 0x7f9c280b9620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:05.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.162+0000 7f9c2d400640 1 -- 172.21.15.6:0/2648335901 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9c280b4b30 msgr2=0x7f9c280b9620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:05.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.162+0000 7f9c2d400640 1 --2- 172.21.15.6:0/2648335901 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9c280b4b30 0x7f9c280b9620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:05.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.163+0000 7f9c2d400640 1 -- 172.21.15.6:0/2648335901 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9c28162720 msgr2=0x7f9c28156a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:05.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.163+0000 7f9c2d400640 1 --2- 172.21.15.6:0/2648335901 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9c28162720 0x7f9c28156a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:05.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.163+0000 7f9c2d400640 1 -- 172.21.15.6:0/2648335901 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c2815db90 con 0x7f9c2815f630 2024-03-04T09:31:05.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.163+0000 7f9c2d400640 1 --2- 172.21.15.6:0/2648335901 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9c2815f630 0x7f9c280cd5e0 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7f9c18000c00 tx=0x7f9c180046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:05.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.163+0000 7f9c22000640 1 -- 172.21.15.6:0/2648335901 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c18032650 con 0x7f9c2815f630 2024-03-04T09:31:05.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.163+0000 7f9c22000640 1 -- 172.21.15.6:0/2648335901 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c180327f0 con 0x7f9c2815f630 2024-03-04T09:31:05.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.163+0000 7f9c2f200640 1 -- 172.21.15.6:0/2648335901 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c2815de70 con 0x7f9c2815f630 2024-03-04T09:31:05.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.163+0000 7f9c22000640 1 -- 172.21.15.6:0/2648335901 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c18032be0 con 0x7f9c2815f630 2024-03-04T09:31:05.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.164+0000 7f9c2f200640 1 -- 172.21.15.6:0/2648335901 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c2815e430 con 0x7f9c2815f630 2024-03-04T09:31:05.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.165+0000 7f9c2f200640 1 -- 172.21.15.6:0/2648335901 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c2804f330 con 0x7f9c2815f630 2024-03-04T09:31:05.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.166+0000 7f9c22000640 1 -- 172.21.15.6:0/2648335901 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9c1804c060 con 0x7f9c2815f630 2024-03-04T09:31:05.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.167+0000 7f9c22000640 1 --2- 172.21.15.6:0/2648335901 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f9c0c06b4d0 0x7f9c0c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:05.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.167+0000 7f9c22000640 1 -- 172.21.15.6:0/2648335901 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f9c180abc20 con 0x7f9c2815f630 2024-03-04T09:31:05.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.168+0000 7f9c2de00640 1 --2- 172.21.15.6:0/2648335901 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f9c0c06b4d0 0x7f9c0c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:05.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.169+0000 7f9c2de00640 1 --2- 172.21.15.6:0/2648335901 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f9c0c06b4d0 0x7f9c0c06d990 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f9c2815eb40 tx=0x7f9c1c008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:05.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.171+0000 7f9c22000640 1 -- 172.21.15.6:0/2648335901 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9c18040070 con 0x7f9c2815f630 2024-03-04T09:31:05.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.416+0000 7f9c2f200640 1 -- 172.21.15.6:0/2648335901 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9c2808d950 con 0x7f9c2815f630 2024-03-04T09:31:05.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.417+0000 7f9c22000640 1 -- 172.21.15.6:0/2648335901 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) v1 ==== 94+0+4977 (secure 0 0 0) 0x7f9c1807a8d0 con 0x7f9c2815f630 2024-03-04T09:31:05.414 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:31:05.414 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":68,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5654,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/1370302030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":1370302030},{"type":"v1","addr":"172.21.15.146:6835","nonce":1370302030}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":6163,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/809822004","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":809822004},{"type":"v1","addr":"172.21.15.146:6837","nonce":809822004}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":66},{"gid":7284,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/1583413840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1583413840},{"type":"v1","addr":"172.21.15.6:6833","nonce":1583413840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64}],"filesystems":[{"mdsmap":{"epoch":68,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:30:20.660103+0000","modified":"2024-03-04T09:30:36.289706+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":98,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7296},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7296":{"gid":7296,"name":"a","rank":0,"incarnation":65,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1732383836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1732383836},{"type":"v1","addr":"172.21.15.6:6835","nonce":1732383836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7296,"qdb_cluster":[7296]},"id":7}]} 2024-03-04T09:31:05.415 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 68 2024-03-04T09:31:05.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.420+0000 7f9c2f200640 1 -- 172.21.15.6:0/2648335901 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f9c0c06b4d0 msgr2=0x7f9c0c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:05.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.420+0000 7f9c2f200640 1 --2- 172.21.15.6:0/2648335901 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f9c0c06b4d0 0x7f9c0c06d990 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f9c2815eb40 tx=0x7f9c1c008040 comp rx=0 tx=0).stop 2024-03-04T09:31:05.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.420+0000 7f9c2f200640 1 -- 172.21.15.6:0/2648335901 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9c2815f630 msgr2=0x7f9c280cd5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:05.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.420+0000 7f9c2f200640 1 --2- 172.21.15.6:0/2648335901 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9c2815f630 0x7f9c280cd5e0 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7f9c18000c00 tx=0x7f9c180046c0 comp rx=0 tx=0).stop 2024-03-04T09:31:05.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.420+0000 7f9c2f200640 1 -- 172.21.15.6:0/2648335901 shutdown_connections 2024-03-04T09:31:05.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.420+0000 7f9c2f200640 1 --2- 172.21.15.6:0/2648335901 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9c28162720 0x7f9c28156a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:05.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.420+0000 7f9c2f200640 1 --2- 172.21.15.6:0/2648335901 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9c2815f630 0x7f9c280cd5e0 unknown :-1 s=CLOSED pgs=1183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:05.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.420+0000 7f9c2f200640 1 --2- 172.21.15.6:0/2648335901 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f9c0c06b4d0 0x7f9c0c06d990 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:05.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.420+0000 7f9c2f200640 1 --2- 172.21.15.6:0/2648335901 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9c280b4b30 0x7f9c280b9620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:05.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.420+0000 7f9c2f200640 1 -- 172.21.15.6:0/2648335901 >> 172.21.15.6:0/2648335901 conn(0x7f9c280d3e30 msgr2=0x7f9c2815fba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:05.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.420+0000 7f9c2f200640 1 -- 172.21.15.6:0/2648335901 shutdown_connections 2024-03-04T09:31:05.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.421+0000 7f9c2f200640 1 -- 172.21.15.6:0/2648335901 wait complete. 2024-03-04T09:31:05.437 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:31:05.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.643+0000 7f04b5e00640 1 Processor -- start 2024-03-04T09:31:05.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.643+0000 7f04b5e00640 1 -- start start 2024-03-04T09:31:05.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.645+0000 7f04b5e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f04b01587e0 0x7f04b0161ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:05.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.645+0000 7f04b5e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f04b006c6e0 0x7f04b00aedd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:05.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.645+0000 7f04b5e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04b0158bc0 con 0x7f04b006e5f0 2024-03-04T09:31:05.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.645+0000 7f04b5e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04b0162c10 con 0x7f04b006c6e0 2024-03-04T09:31:05.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.645+0000 7f04b5e00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04b0162650 con 0x7f04b01587e0 2024-03-04T09:31:05.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.645+0000 7f04af400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f04b006c6e0 0x7f04b00aedd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:05.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.645+0000 7f04af400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f04b006c6e0 0x7f04b00aedd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51670/0 (socket says 172.21.15.6:51670) 2024-03-04T09:31:05.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.645+0000 7f04af400640 1 -- 172.21.15.6:0/1510311571 learned_addr learned my addr 172.21.15.6:0/1510311571 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:31:05.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.645+0000 7f04ae000640 1 --1- 172.21.15.6:0/1510311571 >> v1:172.21.15.6:6789/0 conn(0x7f04b006e5f0 0x7f04b006bfd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:35332/0 (socket says 172.21.15.6:35332) 2024-03-04T09:31:05.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.646+0000 7f04aea00640 1 --2- 172.21.15.6:0/1510311571 >> v2:172.21.15.146:3300/0 conn(0x7f04b01587e0 0x7f04b0161ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:05.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.646+0000 7f04af400640 1 -- 172.21.15.6:0/1510311571 >> v2:172.21.15.146:3300/0 conn(0x7f04b01587e0 msgr2=0x7f04b0161ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:05.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.646+0000 7f04af400640 1 --2- 172.21.15.6:0/1510311571 >> v2:172.21.15.146:3300/0 conn(0x7f04b01587e0 0x7f04b0161ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:05.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.646+0000 7f04af400640 1 -- 172.21.15.6:0/1510311571 >> v1:172.21.15.6:6789/0 conn(0x7f04b006e5f0 legacy=0x7f04b006bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:05.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.646+0000 7f04af400640 1 -- 172.21.15.6:0/1510311571 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04b0162930 con 0x7f04b006c6e0 2024-03-04T09:31:05.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.647+0000 7f04af400640 1 --2- 172.21.15.6:0/1510311571 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f04b006c6e0 0x7f04b00aedd0 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7f04b0066e30 tx=0x7f04a802bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8689db03eeeeb0b4 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:05.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.647+0000 7f04ad600640 1 -- 172.21.15.6:0/1510311571 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04a802cb10 con 0x7f04b006c6e0 2024-03-04T09:31:05.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.647+0000 7f04ad600640 1 -- 172.21.15.6:0/1510311571 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04a802ccb0 con 0x7f04b006c6e0 2024-03-04T09:31:05.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.647+0000 7f04ad600640 1 -- 172.21.15.6:0/1510311571 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04a802ce60 con 0x7f04b006c6e0 2024-03-04T09:31:05.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.648+0000 7f04b5e00640 1 -- 172.21.15.6:0/1510311571 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f04b006c6e0 msgr2=0x7f04b00aedd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:05.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.648+0000 7f04b5e00640 1 --2- 172.21.15.6:0/1510311571 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f04b006c6e0 0x7f04b00aedd0 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7f04b0066e30 tx=0x7f04a802bf70 comp rx=0 tx=0).stop 2024-03-04T09:31:05.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.648+0000 7f04b5e00640 1 -- 172.21.15.6:0/1510311571 shutdown_connections 2024-03-04T09:31:05.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.648+0000 7f04b5e00640 1 --2- 172.21.15.6:0/1510311571 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f04b006c6e0 0x7f04b00aedd0 unknown :-1 s=CLOSED pgs=1185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:05.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.648+0000 7f04b5e00640 1 --2- 172.21.15.6:0/1510311571 >> v2:172.21.15.146:3300/0 conn(0x7f04b01587e0 0x7f04b0161ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:05.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.648+0000 7f04b5e00640 1 -- 172.21.15.6:0/1510311571 >> 172.21.15.6:0/1510311571 conn(0x7f04b00b2980 msgr2=0x7f04b00715c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:05.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.648+0000 7f04b5e00640 1 -- 172.21.15.6:0/1510311571 shutdown_connections 2024-03-04T09:31:05.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.648+0000 7f04b5e00640 1 -- 172.21.15.6:0/1510311571 wait complete. 2024-03-04T09:31:05.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.651+0000 7f04b5e00640 1 Processor -- start 2024-03-04T09:31:05.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.651+0000 7f04b5e00640 1 -- start start 2024-03-04T09:31:05.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.653+0000 7f04b5e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f04b006c6e0 0x7f04b014d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:05.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.653+0000 7f04b5e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f04b006e5f0 0x7f04b0153830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:05.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.653+0000 7f04b5e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f04b01587e0 0x7f04b0153d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:05.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.653+0000 7f04b5e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04b0162220 con 0x7f04b006c6e0 2024-03-04T09:31:05.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.653+0000 7f04b5e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04b00ae8b0 con 0x7f04b01587e0 2024-03-04T09:31:05.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.653+0000 7f04b5e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04b01623a0 con 0x7f04b006e5f0 2024-03-04T09:31:05.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.653+0000 7f04aea00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f04b006c6e0 0x7f04b014d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:05.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.653+0000 7f04aea00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f04b006c6e0 0x7f04b014d530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51678/0 (socket says 172.21.15.6:51678) 2024-03-04T09:31:05.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.653+0000 7f04aea00640 1 -- 172.21.15.6:0/4050716738 learned_addr learned my addr 172.21.15.6:0/4050716738 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:31:05.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.653+0000 7f04aea00640 1 -- 172.21.15.6:0/4050716738 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f04b006e5f0 msgr2=0x7f04b0153830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:05.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.653+0000 7f04aea00640 1 --2- 172.21.15.6:0/4050716738 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f04b006e5f0 0x7f04b0153830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:05.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.653+0000 7f04aea00640 1 -- 172.21.15.6:0/4050716738 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f04b01587e0 msgr2=0x7f04b0153d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:05.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.653+0000 7f04aea00640 1 --2- 172.21.15.6:0/4050716738 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f04b01587e0 0x7f04b0153d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:05.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.653+0000 7f04aea00640 1 -- 172.21.15.6:0/4050716738 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04b01f7670 con 0x7f04b006c6e0 2024-03-04T09:31:05.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.654+0000 7f04aea00640 1 --2- 172.21.15.6:0/4050716738 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f04b006c6e0 0x7f04b014d530 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7f04a000d830 tx=0x7f04a000dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:05.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.654+0000 7f04a7400640 1 -- 172.21.15.6:0/4050716738 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04a000b6c0 con 0x7f04b006c6e0 2024-03-04T09:31:05.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.654+0000 7f04a7400640 1 -- 172.21.15.6:0/4050716738 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04a0019070 con 0x7f04b006c6e0 2024-03-04T09:31:05.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.654+0000 7f04a7400640 1 -- 172.21.15.6:0/4050716738 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04a000ba80 con 0x7f04b006c6e0 2024-03-04T09:31:05.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.654+0000 7f04b5e00640 1 -- 172.21.15.6:0/4050716738 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04b01f7910 con 0x7f04b006c6e0 2024-03-04T09:31:05.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.654+0000 7f04b5e00640 1 -- 172.21.15.6:0/4050716738 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04b01f7ed0 con 0x7f04b006c6e0 2024-03-04T09:31:05.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.656+0000 7f04b5e00640 1 -- 172.21.15.6:0/4050716738 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04b00692a0 con 0x7f04b006c6e0 2024-03-04T09:31:05.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.656+0000 7f04a7400640 1 -- 172.21.15.6:0/4050716738 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f04a001e460 con 0x7f04b006c6e0 2024-03-04T09:31:05.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.657+0000 7f04a7400640 1 --2- 172.21.15.6:0/4050716738 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f049006b500 0x7f049006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:05.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.657+0000 7f04a7400640 1 -- 172.21.15.6:0/4050716738 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f04a008a9f0 con 0x7f04b006c6e0 2024-03-04T09:31:05.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.658+0000 7f04ae000640 1 --2- 172.21.15.6:0/4050716738 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f049006b500 0x7f049006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:05.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.659+0000 7f04ae000640 1 --2- 172.21.15.6:0/4050716738 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f049006b500 0x7f049006d9c0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f049c000f60 tx=0x7f049c007a00 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:05.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:05.661+0000 7f04a7400640 1 -- 172.21.15.6:0/4050716738 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f04a0059720 con 0x7f04b006c6e0 2024-03-04T09:31:06.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.099+0000 7f04b5e00640 1 -- 172.21.15.6:0/4050716738 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f04b00726b0 con 0x7f04b006c6e0 2024-03-04T09:31:06.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.100+0000 7f04a7400640 1 -- 172.21.15.6:0/4050716738 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) v1 ==== 94+0+4977 (secure 0 0 0) 0x7f04a0059020 con 0x7f04b006c6e0 2024-03-04T09:31:06.097 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:31:06.097 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":68,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5654,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/1370302030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":1370302030},{"type":"v1","addr":"172.21.15.146:6835","nonce":1370302030}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":6163,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/809822004","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":809822004},{"type":"v1","addr":"172.21.15.146:6837","nonce":809822004}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":66},{"gid":7284,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/1583413840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1583413840},{"type":"v1","addr":"172.21.15.6:6833","nonce":1583413840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64}],"filesystems":[{"mdsmap":{"epoch":68,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:30:20.660103+0000","modified":"2024-03-04T09:30:36.289706+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":98,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7296},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7296":{"gid":7296,"name":"a","rank":0,"incarnation":65,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1732383836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1732383836},{"type":"v1","addr":"172.21.15.6:6835","nonce":1732383836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7296,"qdb_cluster":[7296]},"id":7}]} 2024-03-04T09:31:06.097 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 68 2024-03-04T09:31:06.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.103+0000 7f04b5e00640 1 -- 172.21.15.6:0/4050716738 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f049006b500 msgr2=0x7f049006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:06.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.103+0000 7f04b5e00640 1 --2- 172.21.15.6:0/4050716738 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f049006b500 0x7f049006d9c0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f049c000f60 tx=0x7f049c007a00 comp rx=0 tx=0).stop 2024-03-04T09:31:06.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.103+0000 7f04b5e00640 1 -- 172.21.15.6:0/4050716738 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f04b006c6e0 msgr2=0x7f04b014d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:06.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.103+0000 7f04b5e00640 1 --2- 172.21.15.6:0/4050716738 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f04b006c6e0 0x7f04b014d530 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7f04a000d830 tx=0x7f04a000dcf0 comp rx=0 tx=0).stop 2024-03-04T09:31:06.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.103+0000 7f04b5e00640 1 -- 172.21.15.6:0/4050716738 shutdown_connections 2024-03-04T09:31:06.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.103+0000 7f04b5e00640 1 --2- 172.21.15.6:0/4050716738 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f04b01587e0 0x7f04b0153d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:06.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.103+0000 7f04b5e00640 1 --2- 172.21.15.6:0/4050716738 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f049006b500 0x7f049006d9c0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:06.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.103+0000 7f04b5e00640 1 --2- 172.21.15.6:0/4050716738 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f04b006e5f0 0x7f04b0153830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:06.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.103+0000 7f04b5e00640 1 --2- 172.21.15.6:0/4050716738 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f04b006c6e0 0x7f04b014d530 unknown :-1 s=CLOSED pgs=1186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:06.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.103+0000 7f04b5e00640 1 -- 172.21.15.6:0/4050716738 >> 172.21.15.6:0/4050716738 conn(0x7f04b00b2980 msgr2=0x7f04b00b3410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:06.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.104+0000 7f04b5e00640 1 -- 172.21.15.6:0/4050716738 shutdown_connections 2024-03-04T09:31:06.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.104+0000 7f04b5e00640 1 -- 172.21.15.6:0/4050716738 wait complete. 2024-03-04T09:31:06.118 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T09:31:06.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.328+0000 7f842cc00640 1 Processor -- start 2024-03-04T09:31:06.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.329+0000 7f842cc00640 1 -- start start 2024-03-04T09:31:06.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.330+0000 7f842cc00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f84281457e0 0x7f84280cf5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:06.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.330+0000 7f842cc00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f842815fba0 0x7f84280b5160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:06.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.330+0000 7f842cc00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f84280b56a0 0x7f842815c590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:06.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.330+0000 7f842cc00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84280b5ab0 con 0x7f84280b56a0 2024-03-04T09:31:06.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.330+0000 7f842cc00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8428145bc0 con 0x7f842815fba0 2024-03-04T09:31:06.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.330+0000 7f842cc00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84280cfb20 con 0x7f84281457e0 2024-03-04T09:31:06.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.331+0000 7f8426800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f84280b56a0 0x7f842815c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:06.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.331+0000 7f8426800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f84280b56a0 0x7f842815c590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51692/0 (socket says 172.21.15.6:51692) 2024-03-04T09:31:06.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.331+0000 7f8426800640 1 -- 172.21.15.6:0/2831490048 learned_addr learned my addr 172.21.15.6:0/2831490048 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:31:06.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.331+0000 7f841d400640 1 --2- 172.21.15.6:0/2831490048 >> v2:172.21.15.146:3300/0 conn(0x7f842815fba0 0x7f84280b5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:06.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.331+0000 7f8425e00640 1 --2- 172.21.15.6:0/2831490048 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f84281457e0 0x7f84280cf5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:06.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.331+0000 7f8426800640 1 -- 172.21.15.6:0/2831490048 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f84281457e0 msgr2=0x7f84280cf5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:06.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.331+0000 7f8426800640 1 --2- 172.21.15.6:0/2831490048 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f84281457e0 0x7f84280cf5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:06.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.331+0000 7f8426800640 1 -- 172.21.15.6:0/2831490048 >> v2:172.21.15.146:3300/0 conn(0x7f842815fba0 msgr2=0x7f84280b5160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:06.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.331+0000 7f8426800640 1 --2- 172.21.15.6:0/2831490048 >> v2:172.21.15.146:3300/0 conn(0x7f842815fba0 0x7f84280b5160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:06.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.331+0000 7f8426800640 1 -- 172.21.15.6:0/2831490048 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8428160040 con 0x7f84280b56a0 2024-03-04T09:31:06.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.331+0000 7f841d400640 1 --2- 172.21.15.6:0/2831490048 >> v2:172.21.15.146:3300/0 conn(0x7f842815fba0 0x7f84280b5160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:31:06.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.332+0000 7f8426800640 1 --2- 172.21.15.6:0/2831490048 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f84280b56a0 0x7f842815c590 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7f842806beb0 tx=0x7f842002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19a8c79ec8f06cef server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:06.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.333+0000 7f8425400640 1 -- 172.21.15.6:0/2831490048 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f842002c9e0 con 0x7f84280b56a0 2024-03-04T09:31:06.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.333+0000 7f8425400640 1 -- 172.21.15.6:0/2831490048 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f842002cb80 con 0x7f84280b56a0 2024-03-04T09:31:06.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.333+0000 7f8425400640 1 -- 172.21.15.6:0/2831490048 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84200343f0 con 0x7f84280b56a0 2024-03-04T09:31:06.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.333+0000 7f842cc00640 1 -- 172.21.15.6:0/2831490048 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f84280b56a0 msgr2=0x7f842815c590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:06.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.333+0000 7f842cc00640 1 --2- 172.21.15.6:0/2831490048 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f84280b56a0 0x7f842815c590 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7f842806beb0 tx=0x7f842002be40 comp rx=0 tx=0).stop 2024-03-04T09:31:06.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.333+0000 7f842cc00640 1 -- 172.21.15.6:0/2831490048 shutdown_connections 2024-03-04T09:31:06.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.333+0000 7f842cc00640 1 --2- 172.21.15.6:0/2831490048 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f84280b56a0 0x7f842815c590 unknown :-1 s=CLOSED pgs=1187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:06.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.333+0000 7f842cc00640 1 --2- 172.21.15.6:0/2831490048 >> v2:172.21.15.146:3300/0 conn(0x7f842815fba0 0x7f84280b5160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:06.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.333+0000 7f842cc00640 1 --2- 172.21.15.6:0/2831490048 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f84281457e0 0x7f84280cf5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:06.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.333+0000 7f842cc00640 1 -- 172.21.15.6:0/2831490048 >> 172.21.15.6:0/2831490048 conn(0x7f842806cf40 msgr2=0x7f842806d320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:06.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.333+0000 7f842cc00640 1 -- 172.21.15.6:0/2831490048 shutdown_connections 2024-03-04T09:31:06.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.333+0000 7f842cc00640 1 -- 172.21.15.6:0/2831490048 wait complete. 2024-03-04T09:31:06.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.336+0000 7f842cc00640 1 Processor -- start 2024-03-04T09:31:06.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.336+0000 7f842cc00640 1 -- start start 2024-03-04T09:31:06.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.338+0000 7f842cc00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f84280b56a0 0x7f842815b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:06.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.338+0000 7f842cc00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f84281457e0 0x7f84281e62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:06.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.338+0000 7f842cc00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f842815fba0 0x7f84281e7a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:06.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.338+0000 7f842cc00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84281608e0 con 0x7f84280b56a0 2024-03-04T09:31:06.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.338+0000 7f842cc00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8428160600 con 0x7f84281457e0 2024-03-04T09:31:06.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.338+0000 7f842cc00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84280cca70 con 0x7f842815fba0 2024-03-04T09:31:06.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.338+0000 7f8425e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f84280b56a0 0x7f842815b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:06.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.339+0000 7f8425e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f84280b56a0 0x7f842815b920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51700/0 (socket says 172.21.15.6:51700) 2024-03-04T09:31:06.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.339+0000 7f8425e00640 1 -- 172.21.15.6:0/928991951 learned_addr learned my addr 172.21.15.6:0/928991951 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:31:06.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.339+0000 7f841d400640 1 --2- 172.21.15.6:0/928991951 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f84281457e0 0x7f84281e62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:06.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.339+0000 7f8426800640 1 --2- 172.21.15.6:0/928991951 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f842815fba0 0x7f84281e7a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:06.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.339+0000 7f8425e00640 1 -- 172.21.15.6:0/928991951 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f842815fba0 msgr2=0x7f84281e7a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:06.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.339+0000 7f8425e00640 1 --2- 172.21.15.6:0/928991951 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f842815fba0 0x7f84281e7a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:06.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.339+0000 7f8425e00640 1 -- 172.21.15.6:0/928991951 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f84281457e0 msgr2=0x7f84281e62e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:06.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.339+0000 7f8425e00640 1 --2- 172.21.15.6:0/928991951 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f84281457e0 0x7f84281e62e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:06.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.339+0000 7f8425e00640 1 -- 172.21.15.6:0/928991951 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84281e6da0 con 0x7f84280b56a0 2024-03-04T09:31:06.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.339+0000 7f8426800640 1 --2- 172.21.15.6:0/928991951 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f842815fba0 0x7f84281e7a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:31:06.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.339+0000 7f841d400640 1 --2- 172.21.15.6:0/928991951 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f84281457e0 0x7f84281e62e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:31:06.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.339+0000 7f8425e00640 1 --2- 172.21.15.6:0/928991951 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f84280b56a0 0x7f842815b920 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7f841800c7f0 tx=0x7f841800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:06.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.340+0000 7f841ea00640 1 -- 172.21.15.6:0/928991951 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8418015070 con 0x7f84280b56a0 2024-03-04T09:31:06.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.340+0000 7f841ea00640 1 -- 172.21.15.6:0/928991951 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8418004220 con 0x7f84280b56a0 2024-03-04T09:31:06.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.340+0000 7f842cc00640 1 -- 172.21.15.6:0/928991951 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84281e7080 con 0x7f84280b56a0 2024-03-04T09:31:06.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.340+0000 7f841ea00640 1 -- 172.21.15.6:0/928991951 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8418007620 con 0x7f84280b56a0 2024-03-04T09:31:06.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.340+0000 7f842cc00640 1 -- 172.21.15.6:0/928991951 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f842815be60 con 0x7f84280b56a0 2024-03-04T09:31:06.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.341+0000 7f842cc00640 1 -- 172.21.15.6:0/928991951 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84280692a0 con 0x7f84280b56a0 2024-03-04T09:31:06.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.342+0000 7f841ea00640 1 -- 172.21.15.6:0/928991951 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f84180079d0 con 0x7f84280b56a0 2024-03-04T09:31:06.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.344+0000 7f841ea00640 1 --2- 172.21.15.6:0/928991951 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f840406b4d0 0x7f840406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:06.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.344+0000 7f841ea00640 1 -- 172.21.15.6:0/928991951 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f841808a3f0 con 0x7f84280b56a0 2024-03-04T09:31:06.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.344+0000 7f841d400640 1 --2- 172.21.15.6:0/928991951 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f840406b4d0 0x7f840406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:06.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.345+0000 7f841d400640 1 --2- 172.21.15.6:0/928991951 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f840406b4d0 0x7f840406d990 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f8410006fd0 tx=0x7f8410008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:06.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.347+0000 7f841ea00640 1 -- 172.21.15.6:0/928991951 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8418059130 con 0x7f84280b56a0 2024-03-04T09:31:06.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.553+0000 7f842cc00640 1 -- 172.21.15.6:0/928991951 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f84280ae920 con 0x7f84280b56a0 2024-03-04T09:31:06.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.555+0000 7f841ea00640 1 -- 172.21.15.6:0/928991951 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v98) v1 ==== 74+0+16817 (secure 0 0 0) 0x7f8418058a30 con 0x7f84280b56a0 2024-03-04T09:31:06.552 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:31:06.552 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":98,"fsid":"edce0578-5ae4-4402-b45a-552bdef84d7c","created":"2024-03-04T09:24:54.523780+0000","modified":"2024-03-04T09:30:33.260648+0000","last_up_change":"2024-03-04T09:25:07.089049+0000","last_in_change":"2024-03-04T09:24:57.982077+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T09:25:07.935198+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-03-04T09:30:18.510767+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"93","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-03-04T09:30:19.686114+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"94","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a96574f-81f1-4bc6-8463-757520b84a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6825","nonce":1364449401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6827","nonce":1364449401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6831","nonce":1364449401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6829","nonce":1364449401}]},"public_addr":"172.21.15.6:6825/1364449401","cluster_addr":"172.21.15.6:6827/1364449401","heartbeat_back_addr":"172.21.15.6:6831/1364449401","heartbeat_front_addr":"172.21.15.6:6829/1364449401","state":["exists","up"]},{"osd":1,"uuid":"7d79b9c1-6b87-476a-bc36-7db5ee64cd90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6809","nonce":2494103078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6811","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6813","nonce":2494103078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6821","nonce":2494103078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6817","nonce":2494103078}]},"public_addr":"172.21.15.6:6809/2494103078","cluster_addr":"172.21.15.6:6813/2494103078","heartbeat_back_addr":"172.21.15.6:6821/2494103078","heartbeat_front_addr":"172.21.15.6:6817/2494103078","state":["exists","up"]},{"osd":2,"uuid":"492331b9-4ee9-40b6-bacd-85d44cd6c45a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6812","nonce":1500066681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6816","nonce":1500066681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6824","nonce":1500066681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6820","nonce":1500066681}]},"public_addr":"172.21.15.6:6812/1500066681","cluster_addr":"172.21.15.6:6816/1500066681","heartbeat_back_addr":"172.21.15.6:6824/1500066681","heartbeat_front_addr":"172.21.15.6:6820/1500066681","state":["exists","up"]},{"osd":3,"uuid":"21bb2efe-9ba7-4bdc-bc31-180cd1102e21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6801","nonce":1558806123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6803","nonce":1558806123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6807","nonce":1558806123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6805","nonce":1558806123}]},"public_addr":"172.21.15.6:6801/1558806123","cluster_addr":"172.21.15.6:6803/1558806123","heartbeat_back_addr":"172.21.15.6:6807/1558806123","heartbeat_front_addr":"172.21.15.6:6805/1558806123","state":["exists","up"]},{"osd":4,"uuid":"6e2c4b71-19d9-4737-83d9-558705d8644a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6801","nonce":1526135780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6803","nonce":1526135780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6807","nonce":1526135780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6805","nonce":1526135780}]},"public_addr":"172.21.15.146:6801/1526135780","cluster_addr":"172.21.15.146:6803/1526135780","heartbeat_back_addr":"172.21.15.146:6807/1526135780","heartbeat_front_addr":"172.21.15.146:6805/1526135780","state":["exists","up"]},{"osd":5,"uuid":"fb69c525-f1be-4b7d-ab32-6c29b0d6fddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6809","nonce":460262507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6811","nonce":460262507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6815","nonce":460262507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6813","nonce":460262507}]},"public_addr":"172.21.15.146:6809/460262507","cluster_addr":"172.21.15.146:6811/460262507","heartbeat_back_addr":"172.21.15.146:6815/460262507","heartbeat_front_addr":"172.21.15.146:6813/460262507","state":["exists","up"]},{"osd":6,"uuid":"c32e7fed-370d-4401-8ce0-6a4335b0d168","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6823","nonce":1392897493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6827","nonce":1392897493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6831","nonce":1392897493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6829","nonce":1392897493}]},"public_addr":"172.21.15.146:6823/1392897493","cluster_addr":"172.21.15.146:6827/1392897493","heartbeat_back_addr":"172.21.15.146:6831/1392897493","heartbeat_front_addr":"172.21.15.146:6829/1392897493","state":["exists","up"]},{"osd":7,"uuid":"75a79b44-fa5f-426d-a7d3-71f21f78835c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6817","nonce":3268323156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6819","nonce":3268323156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6825","nonce":3268323156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6822","nonce":3268323156}]},"public_addr":"172.21.15.146:6817/3268323156","cluster_addr":"172.21.15.146:6819/3268323156","heartbeat_back_addr":"172.21.15.146:6825/3268323156","heartbeat_front_addr":"172.21.15.146:6822/3268323156","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.310437+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.295850+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.294674+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.526855+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.300023+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.595160+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.346046+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.558741+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.146:6835/847815316":"2024-03-05T09:30:33.260598+0000","172.21.15.146:6834/847815316":"2024-03-05T09:30:33.260598+0000","172.21.15.6:6835/2836916196":"2024-03-05T09:30:32.267572+0000","172.21.15.6:6832/2836916196":"2024-03-05T09:30:32.267572+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T09:31:06.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.558+0000 7f842cc00640 1 -- 172.21.15.6:0/928991951 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f840406b4d0 msgr2=0x7f840406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:06.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.558+0000 7f842cc00640 1 --2- 172.21.15.6:0/928991951 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f840406b4d0 0x7f840406d990 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f8410006fd0 tx=0x7f8410008040 comp rx=0 tx=0).stop 2024-03-04T09:31:06.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.558+0000 7f842cc00640 1 -- 172.21.15.6:0/928991951 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f84280b56a0 msgr2=0x7f842815b920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:06.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.558+0000 7f842cc00640 1 --2- 172.21.15.6:0/928991951 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f84280b56a0 0x7f842815b920 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7f841800c7f0 tx=0x7f841800ccb0 comp rx=0 tx=0).stop 2024-03-04T09:31:06.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.559+0000 7f842cc00640 1 -- 172.21.15.6:0/928991951 shutdown_connections 2024-03-04T09:31:06.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.559+0000 7f842cc00640 1 --2- 172.21.15.6:0/928991951 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f840406b4d0 0x7f840406d990 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:06.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.559+0000 7f842cc00640 1 --2- 172.21.15.6:0/928991951 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f842815fba0 0x7f84281e7a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:06.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.559+0000 7f842cc00640 1 --2- 172.21.15.6:0/928991951 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f84281457e0 0x7f84281e62e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:06.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.559+0000 7f842cc00640 1 --2- 172.21.15.6:0/928991951 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f84280b56a0 0x7f842815b920 unknown :-1 s=CLOSED pgs=1188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:06.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.559+0000 7f842cc00640 1 -- 172.21.15.6:0/928991951 >> 172.21.15.6:0/928991951 conn(0x7f842806cf40 msgr2=0x7f84280ccca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:06.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.559+0000 7f842cc00640 1 -- 172.21.15.6:0/928991951 shutdown_connections 2024-03-04T09:31:06.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.559+0000 7f842cc00640 1 -- 172.21.15.6:0/928991951 wait complete. 2024-03-04T09:31:06.574 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-04T09:31:06.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.795+0000 7f4257200640 1 Processor -- start 2024-03-04T09:31:06.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.795+0000 7f4257200640 1 -- start start 2024-03-04T09:31:06.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.797+0000 7f4257200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f42500af2e0 0x7f42500cef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:06.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.797+0000 7f4257200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42500ad390 con 0x7f4250072a50 2024-03-04T09:31:06.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.797+0000 7f4257200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42500ace10 con 0x7f425006e170 2024-03-04T09:31:06.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.797+0000 7f4257200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f425006e550 con 0x7f42500af2e0 2024-03-04T09:31:06.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.798+0000 7f4255200640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f4250072a50 0x7f4250070c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:35338/0 (socket says 172.21.15.6:35338) 2024-03-04T09:31:06.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.798+0000 7f4255200640 1 -- 172.21.15.6:0/1896838834 learned_addr learned my addr 172.21.15.6:0/1896838834 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:31:06.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.798+0000 7f4247400640 1 --2- 172.21.15.6:0/1896838834 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f42500af2e0 0x7f42500cef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:06.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.798+0000 7f424f400640 1 -- 172.21.15.6:0/1896838834 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3587218599 0 0) 0x7f4248001110 con 0x7f4250072a50 2024-03-04T09:31:06.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.798+0000 7f424f400640 1 -- 172.21.15.6:0/1896838834 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4234003520 con 0x7f4250072a50 2024-03-04T09:31:06.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.799+0000 7f424f400640 1 -- 172.21.15.6:0/1896838834 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4237423850 0 0) 0x7f4234003520 con 0x7f4250072a50 2024-03-04T09:31:06.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.799+0000 7f424f400640 1 -- 172.21.15.6:0/1896838834 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f42500af2e0 msgr2=0x7f42500cef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:06.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.799+0000 7f424f400640 1 --2- 172.21.15.6:0/1896838834 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f42500af2e0 0x7f42500cef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:06.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.799+0000 7f424f400640 1 -- 172.21.15.6:0/1896838834 >> v1:172.21.15.146:6789/0 conn(0x7f425006e170 legacy=0x7f425015d040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:06.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.799+0000 7f424f400640 1 -- 172.21.15.6:0/1896838834 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42500ad0b0 con 0x7f4250072a50 2024-03-04T09:31:06.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.799+0000 7f424f400640 1 -- 172.21.15.6:0/1896838834 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7f4248001840 con 0x7f4250072a50 2024-03-04T09:31:06.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.799+0000 7f424f400640 1 -- 172.21.15.6:0/1896838834 >> v1:172.21.15.6:6789/0 conn(0x7f4250072a50 legacy=0x7f4250070c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:06.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.799+0000 7f424f400640 1 --2- 172.21.15.6:0/1896838834 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f42340039e0 0x7f4234003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:06.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.799+0000 7f424f400640 1 --2- 172.21.15.6:0/1896838834 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f42500af2e0 0x7f4234004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:06.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.799+0000 7f424f400640 1 --2- 172.21.15.6:0/1896838834 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4234004d40 0x7f42340051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:06.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.799+0000 7f424f400640 1 -- 172.21.15.6:0/1896838834 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f425006e550 con 0x7f4234004d40 2024-03-04T09:31:06.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.799+0000 7f424f400640 1 -- 172.21.15.6:0/1896838834 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f423400b9c0 con 0x7f42340039e0 2024-03-04T09:31:06.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.799+0000 7f424f400640 1 -- 172.21.15.6:0/1896838834 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f423400bbc0 con 0x7f42500af2e0 2024-03-04T09:31:06.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.799+0000 7f4255200640 1 --2- 172.21.15.6:0/1896838834 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4234004d40 0x7f42340051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:06.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.799+0000 7f424fe00640 1 --2- 172.21.15.6:0/1896838834 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f42340039e0 0x7f4234003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:06.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.800+0000 7f4247400640 1 --2- 172.21.15.6:0/1896838834 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f42500af2e0 0x7f4234004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:06.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.800+0000 7f4255200640 1 -- 172.21.15.6:0/1896838834 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f42500af2e0 msgr2=0x7f4234004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:06.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.800+0000 7f4255200640 1 --2- 172.21.15.6:0/1896838834 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f42500af2e0 0x7f4234004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:06.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.800+0000 7f4255200640 1 -- 172.21.15.6:0/1896838834 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f42340039e0 msgr2=0x7f4234003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:06.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.800+0000 7f4255200640 1 --2- 172.21.15.6:0/1896838834 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f42340039e0 0x7f4234003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:06.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.800+0000 7f4255200640 1 -- 172.21.15.6:0/1896838834 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4234003520 con 0x7f4234004d40 2024-03-04T09:31:06.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.800+0000 7f4247400640 1 --2- 172.21.15.6:0/1896838834 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f42500af2e0 0x7f4234004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T09:31:06.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.802+0000 7f4255200640 1 --2- 172.21.15.6:0/1896838834 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4234004d40 0x7f42340051c0 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7f424800c4c0 tx=0x7f4234005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b46facdb4dbf1039 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:06.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.803+0000 7f424f400640 1 -- 172.21.15.6:0/1896838834 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f424800c0e0 con 0x7f4234004d40 2024-03-04T09:31:06.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.803+0000 7f424f400640 1 -- 172.21.15.6:0/1896838834 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4248030070 con 0x7f4234004d40 2024-03-04T09:31:06.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.803+0000 7f424f400640 1 -- 172.21.15.6:0/1896838834 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42480058d0 con 0x7f4234004d40 2024-03-04T09:31:06.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.803+0000 7f4257200640 1 -- 172.21.15.6:0/1896838834 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4234004d40 msgr2=0x7f42340051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:06.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.803+0000 7f4257200640 1 --2- 172.21.15.6:0/1896838834 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4234004d40 0x7f42340051c0 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7f424800c4c0 tx=0x7f4234005830 comp rx=0 tx=0).stop 2024-03-04T09:31:06.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.803+0000 7f4255200640 1 -- 172.21.15.6:0/1896838834 reap_dead start 2024-03-04T09:31:06.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.804+0000 7f4257200640 1 -- 172.21.15.6:0/1896838834 shutdown_connections 2024-03-04T09:31:06.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.804+0000 7f4257200640 1 -- 172.21.15.6:0/1896838834 >> 172.21.15.6:0/1896838834 conn(0x7f42500b1990 msgr2=0x7f42500b1da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:06.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.804+0000 7f4257200640 1 -- 172.21.15.6:0/1896838834 shutdown_connections 2024-03-04T09:31:06.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.804+0000 7f4257200640 1 -- 172.21.15.6:0/1896838834 wait complete. 2024-03-04T09:31:06.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.805+0000 7f4257200640 1 Processor -- start 2024-03-04T09:31:06.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.805+0000 7f4257200640 1 -- start start 2024-03-04T09:31:06.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.806+0000 7f4257200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f425014c650 0x7f425015e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:06.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.806+0000 7f4257200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f425014ca30 0x7f425015ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:06.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.806+0000 7f4257200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f425015f140 0x7f425015ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:06.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.806+0000 7f4257200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f425009df50 con 0x7f425014ca30 2024-03-04T09:31:06.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.806+0000 7f4257200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f425009ddd0 con 0x7f425014c650 2024-03-04T09:31:06.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.806+0000 7f4257200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4250071390 con 0x7f425015f140 2024-03-04T09:31:06.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.807+0000 7f4247400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f425014ca30 0x7f425015ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:06.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.807+0000 7f4247400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f425014ca30 0x7f425015ec00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51706/0 (socket says 172.21.15.6:51706) 2024-03-04T09:31:06.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.807+0000 7f4247400640 1 -- 172.21.15.6:0/106955812 learned_addr learned my addr 172.21.15.6:0/106955812 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:31:06.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.807+0000 7f4247400640 1 -- 172.21.15.6:0/106955812 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f425015f140 msgr2=0x7f425015ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:06.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.807+0000 7f424fe00640 1 --2- 172.21.15.6:0/106955812 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f425014c650 0x7f425015e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:06.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.807+0000 7f4247400640 1 --2- 172.21.15.6:0/106955812 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f425015f140 0x7f425015ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:06.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.807+0000 7f4247400640 1 -- 172.21.15.6:0/106955812 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f425014c650 msgr2=0x7f425015e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:06.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.807+0000 7f4247400640 1 --2- 172.21.15.6:0/106955812 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f425014c650 0x7f425015e050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:06.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.807+0000 7f4247400640 1 -- 172.21.15.6:0/106955812 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f425015d290 con 0x7f425014ca30 2024-03-04T09:31:06.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.807+0000 7f4247400640 1 --2- 172.21.15.6:0/106955812 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f425014ca30 0x7f425015ec00 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7f4238011700 tx=0x7f4238011bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:06.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.808+0000 7f424cc00640 1 -- 172.21.15.6:0/106955812 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f423800f940 con 0x7f425014ca30 2024-03-04T09:31:06.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.808+0000 7f424cc00640 1 -- 172.21.15.6:0/106955812 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f423800fae0 con 0x7f425014ca30 2024-03-04T09:31:06.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.808+0000 7f424cc00640 1 -- 172.21.15.6:0/106955812 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4238015460 con 0x7f425014ca30 2024-03-04T09:31:06.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.808+0000 7f4257200640 1 -- 172.21.15.6:0/106955812 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f425015d570 con 0x7f425014ca30 2024-03-04T09:31:06.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.808+0000 7f4257200640 1 -- 172.21.15.6:0/106955812 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f425015db30 con 0x7f425014ca30 2024-03-04T09:31:06.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.809+0000 7f4257200640 1 -- 172.21.15.6:0/106955812 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f425015f540 con 0x7f425014ca30 2024-03-04T09:31:06.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.810+0000 7f424cc00640 1 -- 172.21.15.6:0/106955812 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4238015600 con 0x7f425014ca30 2024-03-04T09:31:06.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.811+0000 7f424cc00640 1 --2- 172.21.15.6:0/106955812 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f422c06b4d0 0x7f422c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:06.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.811+0000 7f424cc00640 1 -- 172.21.15.6:0/106955812 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f423808ac50 con 0x7f425014ca30 2024-03-04T09:31:06.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.812+0000 7f424fe00640 1 --2- 172.21.15.6:0/106955812 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f422c06b4d0 0x7f422c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:06.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.813+0000 7f424fe00640 1 --2- 172.21.15.6:0/106955812 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f422c06b4d0 0x7f422c06d990 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f4240003f00 tx=0x7f4240009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:06.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:06.816+0000 7f424cc00640 1 -- 172.21.15.6:0/106955812 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4238019030 con 0x7f425014ca30 2024-03-04T09:31:07.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:07.043+0000 7f4257200640 1 -- 172.21.15.6:0/106955812 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f425009e0f0 con 0x7f425014ca30 2024-03-04T09:31:07.041 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-03-04T09:31:07.044+0000 7f1606a00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-04T09:31:07.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:07.047+0000 7f424cc00640 1 -- 172.21.15.6:0/106955812 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v69) v1 ==== 161+0+0 (secure 0 0 0) 0x7f4238059980 con 0x7f425014ca30 2024-03-04T09:31:07.045 INFO:teuthology.orchestra.run.smithi006.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-04T09:31:07.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:07.051+0000 7f4257200640 1 -- 172.21.15.6:0/106955812 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f422c06b4d0 msgr2=0x7f422c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:07.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:07.051+0000 7f4257200640 1 --2- 172.21.15.6:0/106955812 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f422c06b4d0 0x7f422c06d990 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f4240003f00 tx=0x7f4240009040 comp rx=0 tx=0).stop 2024-03-04T09:31:07.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:07.051+0000 7f4257200640 1 -- 172.21.15.6:0/106955812 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f425014ca30 msgr2=0x7f425015ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:07.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:07.051+0000 7f4257200640 1 --2- 172.21.15.6:0/106955812 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f425014ca30 0x7f425015ec00 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7f4238011700 tx=0x7f4238011bc0 comp rx=0 tx=0).stop 2024-03-04T09:31:07.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:07.051+0000 7f4257200640 1 -- 172.21.15.6:0/106955812 shutdown_connections 2024-03-04T09:31:07.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:07.051+0000 7f4257200640 1 --2- 172.21.15.6:0/106955812 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f422c06b4d0 0x7f422c06d990 secure :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f4240003f00 tx=0x7f4240009040 comp rx=0 tx=0).stop 2024-03-04T09:31:07.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:07.051+0000 7f4257200640 1 --2- 172.21.15.6:0/106955812 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f425015f140 0x7f425015ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:07.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:07.051+0000 7f4257200640 1 --2- 172.21.15.6:0/106955812 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f425014ca30 0x7f425015ec00 unknown :-1 s=CLOSED pgs=1191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:07.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:07.051+0000 7f4257200640 1 --2- 172.21.15.6:0/106955812 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f425014c650 0x7f425015e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:07.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:07.051+0000 7f4257200640 1 -- 172.21.15.6:0/106955812 >> 172.21.15.6:0/106955812 conn(0x7f42500b1990 msgr2=0x7f425006ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:07.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:07.052+0000 7f4257200640 1 -- 172.21.15.6:0/106955812 shutdown_connections 2024-03-04T09:31:07.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:07.052+0000 7f4257200640 1 -- 172.21.15.6:0/106955812 wait complete. 2024-03-04T09:31:07.073 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-04T09:31:07.123 INFO:tasks.ceph.mds.a.smithi006.stdout:starting mds.a at 2024-03-04T09:31:07.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:07.267+0000 7fc67b600640 1 Processor -- start 2024-03-04T09:31:07.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:07.267+0000 7fc67b600640 1 -- start start 2024-03-04T09:31:07.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:07.268+0000 7fc67b600640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fc6740adfc0 0x7fc6740b5c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:07.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:07.268+0000 7fc67b600640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc674071010 con 0x7fc6740b5790 2024-03-04T09:31:07.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:07.268+0000 7fc67b600640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc674072750 con 0x7fc67415e780 2024-03-04T09:31:07.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:07.268+0000 7fc67b600640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc674071bb0 con 0x7fc6740adfc0 2024-03-04T09:31:07.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:07.269+0000 7fc673e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fc6740b5790 0x7fc6740ad8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:35352/0 (socket says 172.21.15.6:35352) 2024-03-04T09:31:07.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:07.269+0000 7fc673e00640 1 -- 172.21.15.6:0/3019139893 learned_addr learned my addr 172.21.15.6:0/3019139893 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:31:07.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:07.269+0000 7fc679600640 1 --2- 172.21.15.6:0/3019139893 >> v2:172.21.15.146:3300/0 conn(0x7fc6740adfc0 0x7fc6740b5c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:07.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:07.269+0000 7fc673400640 1 -- 172.21.15.6:0/3019139893 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1256715694 0 0) 0x7fc674071010 con 0x7fc6740b5790 2024-03-04T09:31:07.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:07.269+0000 7fc673400640 1 -- 172.21.15.6:0/3019139893 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc658003520 con 0x7fc6740b5790 2024-03-04T09:31:07.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:07.270+0000 7fc673400640 1 -- 172.21.15.6:0/3019139893 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1615741349 0 0) 0x7fc658003520 con 0x7fc6740b5790 2024-03-04T09:31:07.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:07.270+0000 7fc673400640 1 -- 172.21.15.6:0/3019139893 >> v2:172.21.15.146:3300/0 conn(0x7fc6740adfc0 msgr2=0x7fc6740b5c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:07.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:07.270+0000 7fc673400640 1 --2- 172.21.15.6:0/3019139893 >> v2:172.21.15.146:3300/0 conn(0x7fc6740adfc0 0x7fc6740b5c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:07.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:07.270+0000 7fc673400640 1 -- 172.21.15.6:0/3019139893 >> v1:172.21.15.146:6789/0 conn(0x7fc67415e780 legacy=0x7fc674072040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:07.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:07.270+0000 7fc673400640 1 -- 172.21.15.6:0/3019139893 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc674071280 con 0x7fc6740b5790 2024-03-04T09:31:07.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:07.270+0000 7fc673400640 1 -- 172.21.15.6:0/3019139893 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7fc65c001670 con 0x7fc6740b5790 2024-03-04T09:31:07.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:07.270+0000 7fc673400640 1 -- 172.21.15.6:0/3019139893 >> v1:172.21.15.6:6789/0 conn(0x7fc6740b5790 legacy=0x7fc6740ad8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:07.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:07.270+0000 7fc673400640 1 --2- 172.21.15.6:0/3019139893 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc6580039b0 0x7fc658003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:07.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:07.270+0000 7fc673400640 1 --2- 172.21.15.6:0/3019139893 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc6580043b0 0x7fc658004880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:07.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:07.270+0000 7fc673400640 1 --2- 172.21.15.6:0/3019139893 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc658005120 0x7fc65800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:07.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:07.270+0000 7fc673400640 1 -- 172.21.15.6:0/3019139893 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc674071bb0 con 0x7fc658005120 2024-03-04T09:31:07.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:07.270+0000 7fc673400640 1 -- 172.21.15.6:0/3019139893 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc65800c060 con 0x7fc6580043b0 2024-03-04T09:31:07.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:07.270+0000 7fc673400640 1 -- 172.21.15.6:0/3019139893 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc65800c260 con 0x7fc6580039b0 2024-03-04T09:31:07.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:07.270+0000 7fc679600640 1 --2- 172.21.15.6:0/3019139893 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc658005120 0x7fc65800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:07.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:07.271+0000 7fc679600640 1 -- 172.21.15.6:0/3019139893 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc6580039b0 msgr2=0x7fc658003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:07.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:07.271+0000 7fc679600640 1 --2- 172.21.15.6:0/3019139893 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc6580039b0 0x7fc658003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:07.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:07.271+0000 7fc679600640 1 -- 172.21.15.6:0/3019139893 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc6580043b0 msgr2=0x7fc658004880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:07.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:07.271+0000 7fc679600640 1 --2- 172.21.15.6:0/3019139893 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc6580043b0 0x7fc658004880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:07.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:07.271+0000 7fc679600640 1 -- 172.21.15.6:0/3019139893 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc658003520 con 0x7fc658005120 2024-03-04T09:31:07.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:07.271+0000 7fc679600640 1 -- 172.21.15.6:0/3019139893 reap_dead start 2024-03-04T09:31:07.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:07.272+0000 7fc679600640 1 --2- 172.21.15.6:0/3019139893 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc658005120 0x7fc65800b9c0 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7fc66c009fc0 tx=0x7fc658005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=178e6aba2fc8a33a server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:07.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:07.272+0000 7fc673400640 1 -- 172.21.15.6:0/3019139893 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc66c009c10 con 0x7fc658005120 2024-03-04T09:31:07.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:07.272+0000 7fc673400640 1 -- 172.21.15.6:0/3019139893 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc66c02cca0 con 0x7fc658005120 2024-03-04T09:31:07.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:07.272+0000 7fc673400640 1 -- 172.21.15.6:0/3019139893 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc66c003bd0 con 0x7fc658005120 2024-03-04T09:31:07.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:07.272+0000 7fc67b600640 1 -- 172.21.15.6:0/3019139893 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc658005120 msgr2=0x7fc65800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:07.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:07.272+0000 7fc67b600640 1 --2- 172.21.15.6:0/3019139893 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc658005120 0x7fc65800b9c0 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7fc66c009fc0 tx=0x7fc658005830 comp rx=0 tx=0).stop 2024-03-04T09:31:07.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:07.273+0000 7fc67b600640 1 -- 172.21.15.6:0/3019139893 shutdown_connections 2024-03-04T09:31:07.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:07.273+0000 7fc67b600640 1 --2- 172.21.15.6:0/3019139893 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc658005120 0x7fc65800b9c0 unknown :-1 s=CLOSED pgs=1196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:07.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:07.273+0000 7fc67b600640 1 -- 172.21.15.6:0/3019139893 >> 172.21.15.6:0/3019139893 conn(0x7fc67406cfb0 msgr2=0x7fc67406d390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:07.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:07.273+0000 7fc67b600640 1 -- 172.21.15.6:0/3019139893 shutdown_connections 2024-03-04T09:31:07.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:07.273+0000 7fc67b600640 1 -- 172.21.15.6:0/3019139893 wait complete. 2024-03-04T09:31:07.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:07.275+0000 7fc67b600640 1 Processor -- start 2024-03-04T09:31:07.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:07.275+0000 7fc67b600640 1 -- start start 2024-03-04T09:31:07.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:07.277+0000 7fc67b600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc658005120 0x7fc67415dc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:07.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:07.277+0000 7fc67b600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc674141990 0x7fc6740c7850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:07.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:07.277+0000 7fc67b600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc674142890 0x7fc674142d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:07.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:07.277+0000 7fc67b600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6740b3080 con 0x7fc674142890 2024-03-04T09:31:07.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:07.277+0000 7fc67b600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc674071840 con 0x7fc674141990 2024-03-04T09:31:07.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:07.277+0000 7fc67b600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6740b3200 con 0x7fc658005120 2024-03-04T09:31:07.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:07.277+0000 7fc679600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc674142890 0x7fc674142d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:07.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:07.277+0000 7fc679600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc674142890 0x7fc674142d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51766/0 (socket says 172.21.15.6:51766) 2024-03-04T09:31:07.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:07.277+0000 7fc679600640 1 -- 172.21.15.6:0/1196793853 learned_addr learned my addr 172.21.15.6:0/1196793853 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:31:07.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:07.278+0000 7fc673e00640 1 --2- 172.21.15.6:0/1196793853 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc674141990 0x7fc6740c7850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:07.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:07.278+0000 7fc678c00640 1 --2- 172.21.15.6:0/1196793853 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc658005120 0x7fc67415dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:07.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:07.278+0000 7fc679600640 1 -- 172.21.15.6:0/1196793853 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc658005120 msgr2=0x7fc67415dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:07.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:07.278+0000 7fc679600640 1 --2- 172.21.15.6:0/1196793853 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc658005120 0x7fc67415dc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:07.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:07.278+0000 7fc679600640 1 -- 172.21.15.6:0/1196793853 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc674141990 msgr2=0x7fc6740c7850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:07.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:07.278+0000 7fc679600640 1 --2- 172.21.15.6:0/1196793853 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc674141990 0x7fc6740c7850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:07.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:07.278+0000 7fc679600640 1 -- 172.21.15.6:0/1196793853 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc67415e180 con 0x7fc674142890 2024-03-04T09:31:07.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:07.278+0000 7fc678c00640 1 --2- 172.21.15.6:0/1196793853 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc658005120 0x7fc67415dc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:31:07.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:07.278+0000 7fc673e00640 1 --2- 172.21.15.6:0/1196793853 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc674141990 0x7fc6740c7850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:31:07.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:07.278+0000 7fc679600640 1 --2- 172.21.15.6:0/1196793853 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc674142890 0x7fc674142d20 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7fc66c00e200 tx=0x7fc66c009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:07.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:07.278+0000 7fc670c00640 1 -- 172.21.15.6:0/1196793853 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc66c003990 con 0x7fc674142890 2024-03-04T09:31:07.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:07.278+0000 7fc670c00640 1 -- 172.21.15.6:0/1196793853 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc66c044070 con 0x7fc674142890 2024-03-04T09:31:07.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:07.278+0000 7fc670c00640 1 -- 172.21.15.6:0/1196793853 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc66c003d90 con 0x7fc674142890 2024-03-04T09:31:07.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:07.278+0000 7fc67b600640 1 -- 172.21.15.6:0/1196793853 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6740c67b0 con 0x7fc674142890 2024-03-04T09:31:07.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:07.279+0000 7fc67b600640 1 -- 172.21.15.6:0/1196793853 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6740c6d70 con 0x7fc674142890 2024-03-04T09:31:07.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:07.280+0000 7fc67b600640 1 -- 172.21.15.6:0/1196793853 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6740692a0 con 0x7fc674142890 2024-03-04T09:31:07.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:07.281+0000 7fc670c00640 1 -- 172.21.15.6:0/1196793853 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc66c005d10 con 0x7fc674142890 2024-03-04T09:31:07.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:07.281+0000 7fc670c00640 1 --2- 172.21.15.6:0/1196793853 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc65006b500 0x7fc65006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:07.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:07.282+0000 7fc670c00640 1 -- 172.21.15.6:0/1196793853 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5967+0+0 (secure 0 0 0) 0x7fc66c0aca10 con 0x7fc674142890 2024-03-04T09:31:07.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:07.282+0000 7fc678c00640 1 --2- 172.21.15.6:0/1196793853 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc65006b500 0x7fc65006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:07.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:07.283+0000 7fc678c00640 1 --2- 172.21.15.6:0/1196793853 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc65006b500 0x7fc65006d9c0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fc6640017c0 tx=0x7fc664009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:07.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:07.286+0000 7fc670c00640 1 -- 172.21.15.6:0/1196793853 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc67415e180 con 0x7fc674142890 2024-03-04T09:31:07.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:07.529+0000 7fc67b600640 1 -- 172.21.15.6:0/1196793853 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fc67415e380 con 0x7fc674142890 2024-03-04T09:31:08.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:08.051+0000 7fc670c00640 1 -- 172.21.15.6:0/1196793853 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v70) v1 ==== 104+0+0 (secure 0 0 0) 0x7fc66c07b780 con 0x7fc674142890 2024-03-04T09:31:08.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:08.056+0000 7fc67b600640 1 -- 172.21.15.6:0/1196793853 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc65006b500 msgr2=0x7fc65006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:08.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:08.056+0000 7fc67b600640 1 --2- 172.21.15.6:0/1196793853 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc65006b500 0x7fc65006d9c0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fc6640017c0 tx=0x7fc664009040 comp rx=0 tx=0).stop 2024-03-04T09:31:08.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:08.056+0000 7fc67b600640 1 -- 172.21.15.6:0/1196793853 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc674142890 msgr2=0x7fc674142d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:08.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:08.056+0000 7fc67b600640 1 --2- 172.21.15.6:0/1196793853 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc674142890 0x7fc674142d20 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7fc66c00e200 tx=0x7fc66c009750 comp rx=0 tx=0).stop 2024-03-04T09:31:08.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:08.056+0000 7fc67b600640 1 -- 172.21.15.6:0/1196793853 shutdown_connections 2024-03-04T09:31:08.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:08.056+0000 7fc67b600640 1 --2- 172.21.15.6:0/1196793853 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc674142890 0x7fc674142d20 unknown :-1 s=CLOSED pgs=1197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:08.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:08.056+0000 7fc67b600640 1 --2- 172.21.15.6:0/1196793853 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc674141990 0x7fc6740c7850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:08.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:08.056+0000 7fc67b600640 1 --2- 172.21.15.6:0/1196793853 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc65006b500 0x7fc65006d9c0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:08.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:08.056+0000 7fc67b600640 1 --2- 172.21.15.6:0/1196793853 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc658005120 0x7fc67415dc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:08.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:08.056+0000 7fc67b600640 1 -- 172.21.15.6:0/1196793853 >> 172.21.15.6:0/1196793853 conn(0x7fc67406cfb0 msgr2=0x7fc67415f770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:08.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:08.056+0000 7fc67b600640 1 -- 172.21.15.6:0/1196793853 shutdown_connections 2024-03-04T09:31:08.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:08.056+0000 7fc67b600640 1 -- 172.21.15.6:0/1196793853 wait complete. 2024-03-04T09:31:08.076 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-04T09:31:08.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:08.303+0000 7fa9f6a00640 1 Processor -- start 2024-03-04T09:31:08.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:08.303+0000 7fa9f6a00640 1 -- start start 2024-03-04T09:31:08.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:08.305+0000 7fa9f6a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa9f00aef00 0x7fa9f0072bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:08.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:08.305+0000 7fa9f6a00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fa9f00cc8c0 0x7fa9f0071480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:08.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:08.305+0000 7fa9f6a00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9f00719c0 con 0x7fa9f00b50f0 2024-03-04T09:31:08.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:08.305+0000 7fa9f6a00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9f00af2e0 con 0x7fa9f00cc8c0 2024-03-04T09:31:08.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:08.305+0000 7fa9f6a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9f015d380 con 0x7fa9f00aef00 2024-03-04T09:31:08.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:08.306+0000 7fa9ef400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fa9f00cc8c0 0x7fa9f0071480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:08.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:08.306+0000 7fa9efe00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa9f00aef00 0x7fa9f0072bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:08.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:08.306+0000 7fa9efe00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa9f00aef00 0x7fa9f0072bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.6:51922/0 (socket says 172.21.15.6:51922) 2024-03-04T09:31:08.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:08.306+0000 7fa9efe00640 1 -- 172.21.15.6:0/1515891269 learned_addr learned my addr 172.21.15.6:0/1515891269 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:31:08.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:08.306+0000 7fa9ef400640 1 -- 172.21.15.6:0/1515891269 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa9f00aef00 msgr2=0x7fa9f0072bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:08.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:08.306+0000 7fa9ef400640 1 --2- 172.21.15.6:0/1515891269 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa9f00aef00 0x7fa9f0072bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:08.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:08.306+0000 7fa9ef400640 1 -- 172.21.15.6:0/1515891269 >> v1:172.21.15.146:6789/0 conn(0x7fa9f00b50f0 legacy=0x7fa9f015fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:08.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:08.306+0000 7fa9ef400640 1 -- 172.21.15.6:0/1515891269 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9f015c800 con 0x7fa9f00cc8c0 2024-03-04T09:31:08.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:08.308+0000 7fa9ef400640 1 --2- 172.21.15.6:0/1515891269 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa9f00cc8c0 0x7fa9f0071480 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fa9f015d560 tx=0x7fa9dc02b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=656231968dc7a06 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:08.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:08.309+0000 7fa9eea00640 1 -- 172.21.15.6:0/1515891269 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9dc039070 con 0x7fa9f00cc8c0 2024-03-04T09:31:08.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:08.309+0000 7fa9eea00640 1 -- 172.21.15.6:0/1515891269 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9dc0316f0 con 0x7fa9f00cc8c0 2024-03-04T09:31:08.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:08.309+0000 7fa9eea00640 1 -- 172.21.15.6:0/1515891269 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9dc031ac0 con 0x7fa9f00cc8c0 2024-03-04T09:31:08.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:08.309+0000 7fa9f6a00640 1 -- 172.21.15.6:0/1515891269 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa9f00cc8c0 msgr2=0x7fa9f0071480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:08.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:08.309+0000 7fa9f6a00640 1 --2- 172.21.15.6:0/1515891269 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa9f00cc8c0 0x7fa9f0071480 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fa9f015d560 tx=0x7fa9dc02b790 comp rx=0 tx=0).stop 2024-03-04T09:31:08.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:08.310+0000 7fa9f6a00640 1 -- 172.21.15.6:0/1515891269 shutdown_connections 2024-03-04T09:31:08.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:08.310+0000 7fa9f6a00640 1 --2- 172.21.15.6:0/1515891269 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa9f00cc8c0 0x7fa9f0071480 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:08.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:08.310+0000 7fa9f6a00640 1 --2- 172.21.15.6:0/1515891269 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa9f00aef00 0x7fa9f0072bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:08.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:08.310+0000 7fa9f6a00640 1 -- 172.21.15.6:0/1515891269 >> 172.21.15.6:0/1515891269 conn(0x7fa9f00b1ae0 msgr2=0x7fa9f00b1ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:08.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:08.310+0000 7fa9f6a00640 1 -- 172.21.15.6:0/1515891269 shutdown_connections 2024-03-04T09:31:08.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:08.310+0000 7fa9f6a00640 1 -- 172.21.15.6:0/1515891269 wait complete. 2024-03-04T09:31:08.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:08.311+0000 7fa9f6a00640 1 Processor -- start 2024-03-04T09:31:08.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:08.312+0000 7fa9f6a00640 1 -- start start 2024-03-04T09:31:08.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:08.312+0000 7fa9f6a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa9f00aef00 0x7fa9f00ac440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:08.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:08.312+0000 7fa9f6a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa9f00b50f0 0x7fa9f00a9f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:08.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:08.312+0000 7fa9f6a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa9f00cc8c0 0x7fa9f0142710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:08.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:08.312+0000 7fa9f6a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9f015d0a0 con 0x7fa9f00b50f0 2024-03-04T09:31:08.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:08.312+0000 7fa9f6a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9f015cdc0 con 0x7fa9f00aef00 2024-03-04T09:31:08.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:08.312+0000 7fa9f6a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9f015c460 con 0x7fa9f00cc8c0 2024-03-04T09:31:08.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:08.313+0000 7fa9ef400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa9f00b50f0 0x7fa9f00a9f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:08.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:08.313+0000 7fa9ef400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa9f00b50f0 0x7fa9f00a9f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51776/0 (socket says 172.21.15.6:51776) 2024-03-04T09:31:08.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:08.313+0000 7fa9ef400640 1 -- 172.21.15.6:0/589881777 learned_addr learned my addr 172.21.15.6:0/589881777 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:31:08.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:08.313+0000 7fa9efe00640 1 --2- 172.21.15.6:0/589881777 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa9f00aef00 0x7fa9f00ac440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:08.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:08.313+0000 7fa9f4a00640 1 --2- 172.21.15.6:0/589881777 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa9f00cc8c0 0x7fa9f0142710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:08.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:08.313+0000 7fa9ef400640 1 -- 172.21.15.6:0/589881777 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa9f00cc8c0 msgr2=0x7fa9f0142710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:08.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:08.313+0000 7fa9ef400640 1 --2- 172.21.15.6:0/589881777 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa9f00cc8c0 0x7fa9f0142710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:08.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:08.313+0000 7fa9ef400640 1 -- 172.21.15.6:0/589881777 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa9f00aef00 msgr2=0x7fa9f00ac440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:08.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:08.313+0000 7fa9ef400640 1 --2- 172.21.15.6:0/589881777 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa9f00aef00 0x7fa9f00ac440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:08.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:08.313+0000 7fa9ef400640 1 -- 172.21.15.6:0/589881777 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9f00acaa0 con 0x7fa9f00b50f0 2024-03-04T09:31:08.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:08.313+0000 7fa9f4a00640 1 --2- 172.21.15.6:0/589881777 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa9f00cc8c0 0x7fa9f0142710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:31:08.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:08.314+0000 7fa9ef400640 1 --2- 172.21.15.6:0/589881777 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa9f00b50f0 0x7fa9f00a9f20 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7fa9dc000c00 tx=0x7fa9dc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:08.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:08.314+0000 7fa9e7e00640 1 -- 172.21.15.6:0/589881777 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9dc043070 con 0x7fa9f00b50f0 2024-03-04T09:31:08.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:08.314+0000 7fa9e7e00640 1 -- 172.21.15.6:0/589881777 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9dc034680 con 0x7fa9f00b50f0 2024-03-04T09:31:08.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:08.314+0000 7fa9e7e00640 1 -- 172.21.15.6:0/589881777 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9dc039040 con 0x7fa9f00b50f0 2024-03-04T09:31:08.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:08.314+0000 7fa9f6a00640 1 -- 172.21.15.6:0/589881777 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9f00ad060 con 0x7fa9f00b50f0 2024-03-04T09:31:08.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:08.314+0000 7fa9f6a00640 1 -- 172.21.15.6:0/589881777 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9f00ad620 con 0x7fa9f00b50f0 2024-03-04T09:31:08.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:08.315+0000 7fa9f6a00640 1 -- 172.21.15.6:0/589881777 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9f00692a0 con 0x7fa9f00b50f0 2024-03-04T09:31:08.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:08.316+0000 7fa9e7e00640 1 -- 172.21.15.6:0/589881777 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa9dc03c070 con 0x7fa9f00b50f0 2024-03-04T09:31:08.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:08.318+0000 7fa9e7e00640 1 --2- 172.21.15.6:0/589881777 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa9d006b4d0 0x7fa9d006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:08.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:08.318+0000 7fa9e7e00640 1 -- 172.21.15.6:0/589881777 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5967+0+0 (secure 0 0 0) 0x7fa9dc0abaf0 con 0x7fa9f00b50f0 2024-03-04T09:31:08.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:08.318+0000 7fa9efe00640 1 --2- 172.21.15.6:0/589881777 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa9d006b4d0 0x7fa9d006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:08.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:08.320+0000 7fa9efe00640 1 --2- 172.21.15.6:0/589881777 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa9d006b4d0 0x7fa9d006d990 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fa9e0009b60 tx=0x7fa9e0008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:08.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:08.322+0000 7fa9e7e00640 1 -- 172.21.15.6:0/589881777 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa9dc07a7d0 con 0x7fa9f00b50f0 2024-03-04T09:31:08.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:08.522+0000 7fa9f6a00640 1 -- 172.21.15.6:0/589881777 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fa9f00ace40 con 0x7fa9f00b50f0 2024-03-04T09:31:09.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:09.522+0000 7fa9e7e00640 1 -- 172.21.15.6:0/589881777 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v100) v1 ==== 181+0+0 (secure 0 0 0) 0x7fa9dc07a0d0 con 0x7fa9f00b50f0 2024-03-04T09:31:09.520 INFO:teuthology.orchestra.run.smithi006.stderr:pool 'cephfs_metadata' removed 2024-03-04T09:31:09.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:09.526+0000 7fa9f6a00640 1 -- 172.21.15.6:0/589881777 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa9d006b4d0 msgr2=0x7fa9d006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:09.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:09.526+0000 7fa9f6a00640 1 --2- 172.21.15.6:0/589881777 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa9d006b4d0 0x7fa9d006d990 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fa9e0009b60 tx=0x7fa9e0008040 comp rx=0 tx=0).stop 2024-03-04T09:31:09.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:09.526+0000 7fa9f6a00640 1 -- 172.21.15.6:0/589881777 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa9f00b50f0 msgr2=0x7fa9f00a9f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:09.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:09.526+0000 7fa9f6a00640 1 --2- 172.21.15.6:0/589881777 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa9f00b50f0 0x7fa9f00a9f20 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7fa9dc000c00 tx=0x7fa9dc0046c0 comp rx=0 tx=0).stop 2024-03-04T09:31:09.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:09.526+0000 7fa9f6a00640 1 -- 172.21.15.6:0/589881777 shutdown_connections 2024-03-04T09:31:09.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:09.526+0000 7fa9f6a00640 1 --2- 172.21.15.6:0/589881777 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa9d006b4d0 0x7fa9d006d990 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:09.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:09.526+0000 7fa9f6a00640 1 --2- 172.21.15.6:0/589881777 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa9f00cc8c0 0x7fa9f0142710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:09.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:09.526+0000 7fa9f6a00640 1 --2- 172.21.15.6:0/589881777 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa9f00b50f0 0x7fa9f00a9f20 unknown :-1 s=CLOSED pgs=1198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:09.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:09.526+0000 7fa9f6a00640 1 --2- 172.21.15.6:0/589881777 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa9f00aef00 0x7fa9f00ac440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:09.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:09.526+0000 7fa9f6a00640 1 -- 172.21.15.6:0/589881777 >> 172.21.15.6:0/589881777 conn(0x7fa9f00b1ae0 msgr2=0x7fa9f00cd470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:09.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:09.527+0000 7fa9f6a00640 1 -- 172.21.15.6:0/589881777 shutdown_connections 2024-03-04T09:31:09.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:09.527+0000 7fa9f6a00640 1 -- 172.21.15.6:0/589881777 wait complete. 2024-03-04T09:31:09.553 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-04T09:31:09.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:09.756+0000 7f7bf0800640 1 Processor -- start 2024-03-04T09:31:09.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:09.756+0000 7f7bf0800640 1 -- start start 2024-03-04T09:31:09.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:09.757+0000 7f7bf0800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f7be80aef00 0x7f7be8072bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:09.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:09.757+0000 7f7bf0800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f7be80cc8c0 0x7f7be8071480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:09.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:09.757+0000 7f7bf0800640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7be80719c0 con 0x7f7be80b50f0 2024-03-04T09:31:09.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:09.757+0000 7f7bf0800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7be80af2e0 con 0x7f7be80cc8c0 2024-03-04T09:31:09.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:09.757+0000 7f7bf0800640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7be815d380 con 0x7f7be80aef00 2024-03-04T09:31:09.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:09.757+0000 7f7bed400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f7be80cc8c0 0x7f7be8071480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:09.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:09.758+0000 7f7bed400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f7be80cc8c0 0x7f7be8071480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:54976/0 (socket says 172.21.15.6:54976) 2024-03-04T09:31:09.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:09.758+0000 7f7bed400640 1 -- 172.21.15.6:0/3180268533 learned_addr learned my addr 172.21.15.6:0/3180268533 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:31:09.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:09.758+0000 7f7bee800640 1 --1- 172.21.15.6:0/3180268533 >> v1:172.21.15.6:6789/0 conn(0x7f7be80b50f0 0x7f7be815fd80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:48350/0 (socket says 172.21.15.6:48350) 2024-03-04T09:31:09.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:09.758+0000 7f7bede00640 1 --2- 172.21.15.6:0/3180268533 >> v2:172.21.15.146:3300/0 conn(0x7f7be80aef00 0x7f7be8072bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:09.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:09.758+0000 7f7bed400640 1 -- 172.21.15.6:0/3180268533 >> v2:172.21.15.146:3300/0 conn(0x7f7be80aef00 msgr2=0x7f7be8072bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:09.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:09.758+0000 7f7bed400640 1 --2- 172.21.15.6:0/3180268533 >> v2:172.21.15.146:3300/0 conn(0x7f7be80aef00 0x7f7be8072bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:09.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:09.758+0000 7f7bed400640 1 -- 172.21.15.6:0/3180268533 >> v1:172.21.15.6:6789/0 conn(0x7f7be80b50f0 legacy=0x7f7be815fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:09.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:09.758+0000 7f7bed400640 1 -- 172.21.15.6:0/3180268533 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7be815c800 con 0x7f7be80cc8c0 2024-03-04T09:31:09.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:09.758+0000 7f7beca00640 1 -- 172.21.15.6:0/3180268533 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3207503394 0 0) 0x7f7be80719c0 con 0x7f7be80b50f0 2024-03-04T09:31:09.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:09.758+0000 7f7bede00640 1 --2- 172.21.15.6:0/3180268533 >> v2:172.21.15.146:3300/0 conn(0x7f7be80aef00 0x7f7be8072bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:31:09.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:09.759+0000 7f7bed400640 1 --2- 172.21.15.6:0/3180268533 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7be80cc8c0 0x7f7be8071480 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7f7be815d560 tx=0x7f7bd402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b2f491e4e167e53 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:09.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:09.760+0000 7f7beca00640 1 -- 172.21.15.6:0/3180268533 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bd402e070 con 0x7f7be80cc8c0 2024-03-04T09:31:09.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:09.760+0000 7f7beca00640 1 -- 172.21.15.6:0/3180268533 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bd4037070 con 0x7f7be80cc8c0 2024-03-04T09:31:09.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:09.760+0000 7f7beca00640 1 -- 172.21.15.6:0/3180268533 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bd4031a00 con 0x7f7be80cc8c0 2024-03-04T09:31:09.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:09.760+0000 7f7bf0800640 1 -- 172.21.15.6:0/3180268533 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7be80cc8c0 msgr2=0x7f7be8071480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:09.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:09.760+0000 7f7bf0800640 1 --2- 172.21.15.6:0/3180268533 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7be80cc8c0 0x7f7be8071480 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7f7be815d560 tx=0x7f7bd402b880 comp rx=0 tx=0).stop 2024-03-04T09:31:09.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:09.760+0000 7f7bf0800640 1 -- 172.21.15.6:0/3180268533 shutdown_connections 2024-03-04T09:31:09.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:09.760+0000 7f7bf0800640 1 --2- 172.21.15.6:0/3180268533 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7be80cc8c0 0x7f7be8071480 unknown :-1 s=CLOSED pgs=1200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:09.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:09.760+0000 7f7bf0800640 1 --2- 172.21.15.6:0/3180268533 >> v2:172.21.15.146:3300/0 conn(0x7f7be80aef00 0x7f7be8072bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:09.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:09.760+0000 7f7bf0800640 1 -- 172.21.15.6:0/3180268533 >> 172.21.15.6:0/3180268533 conn(0x7f7be80b1ae0 msgr2=0x7f7be80b1ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:09.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:09.760+0000 7f7bf0800640 1 -- 172.21.15.6:0/3180268533 shutdown_connections 2024-03-04T09:31:09.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:09.760+0000 7f7bf0800640 1 -- 172.21.15.6:0/3180268533 wait complete. 2024-03-04T09:31:09.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:09.762+0000 7f7bf0800640 1 Processor -- start 2024-03-04T09:31:09.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:09.762+0000 7f7bf0800640 1 -- start start 2024-03-04T09:31:09.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:09.763+0000 7f7bf0800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7be80aef00 0x7f7be81f39c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:09.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:09.763+0000 7f7bf0800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7be80b50f0 0x7f7be80b2ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:09.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:09.763+0000 7f7bf0800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7be80cc8c0 0x7f7be80adb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:09.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:09.763+0000 7f7bf0800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7be815d0a0 con 0x7f7be80aef00 2024-03-04T09:31:09.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:09.763+0000 7f7bf0800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7be815cdc0 con 0x7f7be80b50f0 2024-03-04T09:31:09.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:09.763+0000 7f7bf0800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7be815c460 con 0x7f7be80cc8c0 2024-03-04T09:31:09.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:09.763+0000 7f7bede00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7be80aef00 0x7f7be81f39c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:09.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:09.763+0000 7f7bede00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7be80aef00 0x7f7be81f39c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:54984/0 (socket says 172.21.15.6:54984) 2024-03-04T09:31:09.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:09.763+0000 7f7bede00640 1 -- 172.21.15.6:0/547508280 learned_addr learned my addr 172.21.15.6:0/547508280 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:31:09.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:09.764+0000 7f7bee800640 1 --2- 172.21.15.6:0/547508280 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7be80cc8c0 0x7f7be80adb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:09.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:09.764+0000 7f7bed400640 1 --2- 172.21.15.6:0/547508280 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7be80b50f0 0x7f7be80b2ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:09.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:09.764+0000 7f7bede00640 1 -- 172.21.15.6:0/547508280 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7be80cc8c0 msgr2=0x7f7be80adb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:09.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:09.764+0000 7f7bede00640 1 --2- 172.21.15.6:0/547508280 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7be80cc8c0 0x7f7be80adb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:09.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:09.764+0000 7f7bede00640 1 -- 172.21.15.6:0/547508280 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7be80b50f0 msgr2=0x7f7be80b2ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:09.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:09.764+0000 7f7bede00640 1 --2- 172.21.15.6:0/547508280 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7be80b50f0 0x7f7be80b2ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:09.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:09.764+0000 7f7bede00640 1 -- 172.21.15.6:0/547508280 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7be81f3f00 con 0x7f7be80aef00 2024-03-04T09:31:09.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:09.764+0000 7f7bee800640 1 --2- 172.21.15.6:0/547508280 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7be80cc8c0 0x7f7be80adb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:31:09.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:09.764+0000 7f7bede00640 1 --2- 172.21.15.6:0/547508280 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7be80aef00 0x7f7be81f39c0 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f7bdc00c7f0 tx=0x7f7bdc00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:09.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:09.764+0000 7f7be2000640 1 -- 172.21.15.6:0/547508280 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bdc015070 con 0x7f7be80aef00 2024-03-04T09:31:09.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:09.765+0000 7f7be2000640 1 -- 172.21.15.6:0/547508280 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bdc007a30 con 0x7f7be80aef00 2024-03-04T09:31:09.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:09.765+0000 7f7be2000640 1 -- 172.21.15.6:0/547508280 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bdc007e00 con 0x7f7be80aef00 2024-03-04T09:31:09.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:09.765+0000 7f7bf0800640 1 -- 172.21.15.6:0/547508280 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7be80b3730 con 0x7f7be80aef00 2024-03-04T09:31:09.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:09.765+0000 7f7bf0800640 1 -- 172.21.15.6:0/547508280 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7be80b3cf0 con 0x7f7be80aef00 2024-03-04T09:31:09.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:09.766+0000 7f7bf0800640 1 -- 172.21.15.6:0/547508280 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7be80692a0 con 0x7f7be80aef00 2024-03-04T09:31:09.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:09.767+0000 7f7be2000640 1 -- 172.21.15.6:0/547508280 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7bdc018070 con 0x7f7be80aef00 2024-03-04T09:31:09.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:09.768+0000 7f7be2000640 1 --2- 172.21.15.6:0/547508280 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f7bc806b4d0 0x7f7bc806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:09.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:09.768+0000 7f7be2000640 1 -- 172.21.15.6:0/547508280 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5529+0+0 (secure 0 0 0) 0x7f7bdc08a160 con 0x7f7be80aef00 2024-03-04T09:31:09.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:09.768+0000 7f7bed400640 1 --2- 172.21.15.6:0/547508280 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f7bc806b4d0 0x7f7bc806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:09.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:09.769+0000 7f7bed400640 1 --2- 172.21.15.6:0/547508280 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f7bc806b4d0 0x7f7bc806d990 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f7bd4036000 tx=0x7f7bd4008030 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:09.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:09.772+0000 7f7be2000640 1 -- 172.21.15.6:0/547508280 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7bdc058ff0 con 0x7f7be80aef00 2024-03-04T09:31:09.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:09.975+0000 7f7bf0800640 1 -- 172.21.15.6:0/547508280 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f7be80cd8c0 con 0x7f7be80aef00 2024-03-04T09:31:10.156 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:31:10.164 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:31:10.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:10.526+0000 7f7be2000640 1 -- 172.21.15.6:0/547508280 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v101) v1 ==== 169+0+0 (secure 0 0 0) 0x7f7bdc0588f0 con 0x7f7be80aef00 2024-03-04T09:31:10.523 INFO:teuthology.orchestra.run.smithi006.stderr:pool 'cephfs_data' removed 2024-03-04T09:31:10.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:10.529+0000 7f7bf0800640 1 -- 172.21.15.6:0/547508280 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f7bc806b4d0 msgr2=0x7f7bc806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:10.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:10.529+0000 7f7bf0800640 1 --2- 172.21.15.6:0/547508280 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f7bc806b4d0 0x7f7bc806d990 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f7bd4036000 tx=0x7f7bd4008030 comp rx=0 tx=0).stop 2024-03-04T09:31:10.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:10.529+0000 7f7bf0800640 1 -- 172.21.15.6:0/547508280 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7be80aef00 msgr2=0x7f7be81f39c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:10.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:10.529+0000 7f7bf0800640 1 --2- 172.21.15.6:0/547508280 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7be80aef00 0x7f7be81f39c0 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f7bdc00c7f0 tx=0x7f7bdc00cbf0 comp rx=0 tx=0).stop 2024-03-04T09:31:10.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:10.530+0000 7f7bf0800640 1 -- 172.21.15.6:0/547508280 shutdown_connections 2024-03-04T09:31:10.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:10.530+0000 7f7bf0800640 1 --2- 172.21.15.6:0/547508280 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f7bc806b4d0 0x7f7bc806d990 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:10.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:10.530+0000 7f7bf0800640 1 --2- 172.21.15.6:0/547508280 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7be80cc8c0 0x7f7be80adb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:10.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:10.530+0000 7f7bf0800640 1 --2- 172.21.15.6:0/547508280 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7be80b50f0 0x7f7be80b2ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:10.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:10.530+0000 7f7bf0800640 1 --2- 172.21.15.6:0/547508280 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7be80aef00 0x7f7be81f39c0 unknown :-1 s=CLOSED pgs=1201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:10.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:10.530+0000 7f7bf0800640 1 -- 172.21.15.6:0/547508280 >> 172.21.15.6:0/547508280 conn(0x7f7be80b1ae0 msgr2=0x7f7be815fc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:10.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:10.530+0000 7f7bf0800640 1 -- 172.21.15.6:0/547508280 shutdown_connections 2024-03-04T09:31:10.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:10.531+0000 7f7bf0800640 1 -- 172.21.15.6:0/547508280 wait complete. 2024-03-04T09:31:10.557 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-04T09:31:10.562 DEBUG:teuthology.orchestra.run.smithi146:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-04T09:31:10.700 INFO:teuthology.orchestra.run.smithi006.stdout:[global] 2024-03-04T09:31:10.700 INFO:teuthology.orchestra.run.smithi006.stdout: chdir = "" 2024-03-04T09:31:10.700 INFO:teuthology.orchestra.run.smithi006.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-04T09:31:10.700 INFO:teuthology.orchestra.run.smithi006.stdout: auth supported = cephx 2024-03-04T09:31:10.700 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:31:10.700 INFO:teuthology.orchestra.run.smithi006.stdout: filestore xattr use omap = true 2024-03-04T09:31:10.700 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:31:10.701 INFO:teuthology.orchestra.run.smithi006.stdout: mon clock drift allowed = 1.000 2024-03-04T09:31:10.701 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:31:10.701 INFO:teuthology.orchestra.run.smithi006.stdout: osd crush chooseleaf type = 0 2024-03-04T09:31:10.701 INFO:teuthology.orchestra.run.smithi006.stdout: auth debug = true 2024-03-04T09:31:10.701 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:31:10.701 INFO:teuthology.orchestra.run.smithi006.stdout: ms die on old message = true 2024-03-04T09:31:10.701 INFO:teuthology.orchestra.run.smithi006.stdout: ms die on bug = true 2024-03-04T09:31:10.701 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:31:10.701 INFO:teuthology.orchestra.run.smithi006.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-04T09:31:10.701 INFO:teuthology.orchestra.run.smithi006.stdout: mon pg warn max object skew = 0 2024-03-04T09:31:10.701 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:31:10.701 INFO:teuthology.orchestra.run.smithi006.stdout: # disable pg_autoscaler by default for new pools 2024-03-04T09:31:10.701 INFO:teuthology.orchestra.run.smithi006.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-04T09:31:10.701 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:31:10.701 INFO:teuthology.orchestra.run.smithi006.stdout: osd pool default size = 2 2024-03-04T09:31:10.702 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:31:10.702 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd allow primary affinity = true 2024-03-04T09:31:10.702 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd allow pg remap = true 2024-03-04T09:31:10.702 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on legacy crush tunables = false 2024-03-04T09:31:10.702 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on crush straw calc version zero = false 2024-03-04T09:31:10.702 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on no sortbitwise = false 2024-03-04T09:31:10.702 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on osd down out interval zero = false 2024-03-04T09:31:10.702 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on too few osds = false 2024-03-04T09:31:10.702 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-04T09:31:10.702 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-04T09:31:10.702 INFO:teuthology.orchestra.run.smithi006.stdout: mon_allow_pool_size_one = true 2024-03-04T09:31:10.702 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:31:10.703 INFO:teuthology.orchestra.run.smithi006.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-04T09:31:10.703 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:31:10.703 INFO:teuthology.orchestra.run.smithi006.stdout: osd default data pool replay window = 5 2024-03-04T09:31:10.703 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:31:10.703 INFO:teuthology.orchestra.run.smithi006.stdout: mon allow pool delete = true 2024-03-04T09:31:10.703 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:31:10.703 INFO:teuthology.orchestra.run.smithi006.stdout: mon cluster log file level = debug 2024-03-04T09:31:10.703 INFO:teuthology.orchestra.run.smithi006.stdout: debug asserts on shutdown = true 2024-03-04T09:31:10.703 INFO:teuthology.orchestra.run.smithi006.stdout: mon health detail to clog = false 2024-03-04T09:31:10.703 INFO:teuthology.orchestra.run.smithi006.stdout: mon host = "172.21.15.6,172.21.15.146,[v2:172.21.15.146:3301,v1:172.21.15.146:6790]" 2024-03-04T09:31:10.703 INFO:teuthology.orchestra.run.smithi006.stdout: ms die on skipped message = False 2024-03-04T09:31:10.703 INFO:teuthology.orchestra.run.smithi006.stdout: fsid = edce0578-5ae4-4402-b45a-552bdef84d7c 2024-03-04T09:31:10.703 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:31:10.703 INFO:teuthology.orchestra.run.smithi006.stdout:[osd] 2024-03-04T09:31:10.704 INFO:teuthology.orchestra.run.smithi006.stdout: osd journal size = 100 2024-03-04T09:31:10.704 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:31:10.704 INFO:teuthology.orchestra.run.smithi006.stdout: osd scrub load threshold = 5.0 2024-03-04T09:31:10.704 INFO:teuthology.orchestra.run.smithi006.stdout: osd scrub max interval = 600 2024-03-04T09:31:10.704 INFO:teuthology.orchestra.run.smithi006.stdout: osd mclock profile = high_recovery_ops 2024-03-04T09:31:10.704 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:31:10.704 INFO:teuthology.orchestra.run.smithi006.stdout: osd recover clone overlap = true 2024-03-04T09:31:10.704 INFO:teuthology.orchestra.run.smithi006.stdout: osd recovery max chunk = 1048576 2024-03-04T09:31:10.704 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:31:10.704 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug shutdown = true 2024-03-04T09:31:10.704 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug op order = true 2024-03-04T09:31:10.704 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug verify stray on activate = true 2024-03-04T09:31:10.704 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug trim objects = true 2024-03-04T09:31:10.705 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:31:10.705 INFO:teuthology.orchestra.run.smithi006.stdout: osd open classes on start = true 2024-03-04T09:31:10.705 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug pg log writeout = true 2024-03-04T09:31:10.705 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:31:10.705 INFO:teuthology.orchestra.run.smithi006.stdout: osd deep scrub update digest min age = 30 2024-03-04T09:31:10.705 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:31:10.705 INFO:teuthology.orchestra.run.smithi006.stdout: osd map max advance = 10 2024-03-04T09:31:10.705 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:31:10.705 INFO:teuthology.orchestra.run.smithi006.stdout: journal zero on create = true 2024-03-04T09:31:10.705 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:31:10.706 INFO:teuthology.orchestra.run.smithi006.stdout: filestore ondisk finisher threads = 3 2024-03-04T09:31:10.706 INFO:teuthology.orchestra.run.smithi006.stdout: filestore apply finisher threads = 3 2024-03-04T09:31:10.706 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:31:10.706 INFO:teuthology.orchestra.run.smithi006.stdout: bdev debug aio = true 2024-03-04T09:31:10.706 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug misdirected ops = true 2024-03-04T09:31:10.706 INFO:teuthology.orchestra.run.smithi006.stdout: bdev async discard = True 2024-03-04T09:31:10.706 INFO:teuthology.orchestra.run.smithi006.stdout: bdev enable discard = True 2024-03-04T09:31:10.706 INFO:teuthology.orchestra.run.smithi006.stdout: bluestore allocator = bitmap 2024-03-04T09:31:10.706 INFO:teuthology.orchestra.run.smithi006.stdout: bluestore block size = 96636764160 2024-03-04T09:31:10.706 INFO:teuthology.orchestra.run.smithi006.stdout: bluestore fsck on mount = True 2024-03-04T09:31:10.706 INFO:teuthology.orchestra.run.smithi006.stdout: debug bluefs = 1/20 2024-03-04T09:31:10.706 INFO:teuthology.orchestra.run.smithi006.stdout: debug bluestore = 1/20 2024-03-04T09:31:10.707 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2024-03-04T09:31:10.707 INFO:teuthology.orchestra.run.smithi006.stdout: debug osd = 20 2024-03-04T09:31:10.707 INFO:teuthology.orchestra.run.smithi006.stdout: debug rocksdb = 4/10 2024-03-04T09:31:10.707 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd backfillfull_ratio = 0.85 2024-03-04T09:31:10.707 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd full ratio = 0.9 2024-03-04T09:31:10.707 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd nearfull ratio = 0.8 2024-03-04T09:31:10.707 INFO:teuthology.orchestra.run.smithi006.stdout: osd failsafe full ratio = 0.95 2024-03-04T09:31:10.707 INFO:teuthology.orchestra.run.smithi006.stdout: osd objectstore = bluestore 2024-03-04T09:31:10.707 INFO:teuthology.orchestra.run.smithi006.stdout: osd op complaint time = 180 2024-03-04T09:31:10.707 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:31:10.707 INFO:teuthology.orchestra.run.smithi006.stdout:[mgr] 2024-03-04T09:31:10.707 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2024-03-04T09:31:10.707 INFO:teuthology.orchestra.run.smithi006.stdout: debug mgr = 20 2024-03-04T09:31:10.707 INFO:teuthology.orchestra.run.smithi006.stdout: debug mon = 20 2024-03-04T09:31:10.708 INFO:teuthology.orchestra.run.smithi006.stdout: debug auth = 20 2024-03-04T09:31:10.708 INFO:teuthology.orchestra.run.smithi006.stdout: mon reweight min pgs per osd = 4 2024-03-04T09:31:10.708 INFO:teuthology.orchestra.run.smithi006.stdout: mon reweight min bytes per osd = 10 2024-03-04T09:31:10.708 INFO:teuthology.orchestra.run.smithi006.stdout: mgr/telemetry/nag = false 2024-03-04T09:31:10.708 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:31:10.708 INFO:teuthology.orchestra.run.smithi006.stdout:[mon] 2024-03-04T09:31:10.708 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2024-03-04T09:31:10.708 INFO:teuthology.orchestra.run.smithi006.stdout: debug mon = 20 2024-03-04T09:31:10.708 INFO:teuthology.orchestra.run.smithi006.stdout: debug paxos = 20 2024-03-04T09:31:10.708 INFO:teuthology.orchestra.run.smithi006.stdout: debug auth = 20 2024-03-04T09:31:10.708 INFO:teuthology.orchestra.run.smithi006.stdout: mon data avail warn = 5 2024-03-04T09:31:10.708 INFO:teuthology.orchestra.run.smithi006.stdout: mon mgr mkfs grace = 240 2024-03-04T09:31:10.708 INFO:teuthology.orchestra.run.smithi006.stdout: mon reweight min pgs per osd = 4 2024-03-04T09:31:10.708 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd reporter subtree level = osd 2024-03-04T09:31:10.708 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd prime pg temp = true 2024-03-04T09:31:10.709 INFO:teuthology.orchestra.run.smithi006.stdout: mon reweight min bytes per osd = 10 2024-03-04T09:31:10.709 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:31:10.709 INFO:teuthology.orchestra.run.smithi006.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-04T09:31:10.709 INFO:teuthology.orchestra.run.smithi006.stdout: auth mon ticket ttl = 660 # 11m 2024-03-04T09:31:10.709 INFO:teuthology.orchestra.run.smithi006.stdout: auth service ticket ttl = 240 # 4m 2024-03-04T09:31:10.709 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:31:10.709 INFO:teuthology.orchestra.run.smithi006.stdout: # don't complain about insecure global_id in the test suite 2024-03-04T09:31:10.709 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-04T09:31:10.709 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-04T09:31:10.709 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:31:10.709 INFO:teuthology.orchestra.run.smithi006.stdout: # 1m isn't quite enough 2024-03-04T09:31:10.709 INFO:teuthology.orchestra.run.smithi006.stdout: mon_down_mkfs_grace = 2m 2024-03-04T09:31:10.709 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:31:10.709 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_filestore_osds = false 2024-03-04T09:31:10.710 INFO:teuthology.orchestra.run.smithi006.stdout: mon op complaint time = 120 2024-03-04T09:31:10.710 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:31:10.710 INFO:teuthology.orchestra.run.smithi006.stdout:[client] 2024-03-04T09:31:10.710 INFO:teuthology.orchestra.run.smithi006.stdout: rgw cache enabled = true 2024-03-04T09:31:10.710 INFO:teuthology.orchestra.run.smithi006.stdout: rgw enable ops log = true 2024-03-04T09:31:10.710 INFO:teuthology.orchestra.run.smithi006.stdout: rgw enable usage log = true 2024-03-04T09:31:10.710 INFO:teuthology.orchestra.run.smithi006.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-04T09:31:10.710 INFO:teuthology.orchestra.run.smithi006.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-04T09:31:10.710 INFO:teuthology.orchestra.run.smithi006.stdout: client mount timeout = 600 2024-03-04T09:31:10.710 INFO:teuthology.orchestra.run.smithi006.stdout: debug client = 20 2024-03-04T09:31:10.710 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2024-03-04T09:31:10.710 INFO:teuthology.orchestra.run.smithi006.stdout: pid file = "" 2024-03-04T09:31:10.710 INFO:teuthology.orchestra.run.smithi006.stdout: rados mon op timeout = 900 2024-03-04T09:31:10.710 INFO:teuthology.orchestra.run.smithi006.stdout: rados osd op timeout = 900 2024-03-04T09:31:10.710 INFO:teuthology.orchestra.run.smithi006.stdout:[mon.a] 2024-03-04T09:31:10.710 INFO:teuthology.orchestra.run.smithi006.stdout:[mon.b] 2024-03-04T09:31:10.711 INFO:teuthology.orchestra.run.smithi006.stdout:[mon.c] 2024-03-04T09:31:10.711 INFO:teuthology.orchestra.run.smithi006.stdout:[mds.a] 2024-03-04T09:31:10.711 INFO:teuthology.orchestra.run.smithi006.stdout:[mds.c] 2024-03-04T09:31:10.711 INFO:teuthology.orchestra.run.smithi006.stdout:[mds.b] 2024-03-04T09:31:10.711 INFO:teuthology.orchestra.run.smithi006.stdout:[mds.d] 2024-03-04T09:31:10.711 INFO:teuthology.orchestra.run.smithi006.stdout:[mds] 2024-03-04T09:31:10.711 INFO:teuthology.orchestra.run.smithi006.stdout: debug mds = 20 2024-03-04T09:31:10.711 INFO:teuthology.orchestra.run.smithi006.stdout: debug mds balancer = 20 2024-03-04T09:31:10.711 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2024-03-04T09:31:10.711 INFO:teuthology.orchestra.run.smithi006.stdout: mds debug frag = True 2024-03-04T09:31:10.711 INFO:teuthology.orchestra.run.smithi006.stdout: mds debug scatterstat = True 2024-03-04T09:31:10.711 INFO:teuthology.orchestra.run.smithi006.stdout: mds op complaint time = 180 2024-03-04T09:31:10.711 INFO:teuthology.orchestra.run.smithi006.stdout: mds verify scatter = True 2024-03-04T09:31:10.711 INFO:teuthology.orchestra.run.smithi006.stdout: osd op complaint time = 180 2024-03-04T09:31:10.711 INFO:teuthology.orchestra.run.smithi006.stdout: rados mon op timeout = 900 2024-03-04T09:31:10.711 INFO:teuthology.orchestra.run.smithi006.stdout: rados osd op timeout = 900 2024-03-04T09:31:10.712 INFO:teuthology.orchestra.run.smithi006.stdout: mds_max_purge_files = 6 2024-03-04T09:31:10.712 INFO:teuthology.orchestra.run.smithi006.stdout: mds_max_purge_ops = 100000000 2024-03-04T09:31:10.718 INFO:teuthology.orchestra.run.smithi146.stdout:[global] 2024-03-04T09:31:10.718 INFO:teuthology.orchestra.run.smithi146.stdout: chdir = "" 2024-03-04T09:31:10.718 INFO:teuthology.orchestra.run.smithi146.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-04T09:31:10.719 INFO:teuthology.orchestra.run.smithi146.stdout: auth supported = cephx 2024-03-04T09:31:10.719 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:31:10.719 INFO:teuthology.orchestra.run.smithi146.stdout: filestore xattr use omap = true 2024-03-04T09:31:10.719 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:31:10.719 INFO:teuthology.orchestra.run.smithi146.stdout: mon clock drift allowed = 1.000 2024-03-04T09:31:10.719 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:31:10.719 INFO:teuthology.orchestra.run.smithi146.stdout: osd crush chooseleaf type = 0 2024-03-04T09:31:10.719 INFO:teuthology.orchestra.run.smithi146.stdout: auth debug = true 2024-03-04T09:31:10.719 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:31:10.719 INFO:teuthology.orchestra.run.smithi146.stdout: ms die on old message = true 2024-03-04T09:31:10.719 INFO:teuthology.orchestra.run.smithi146.stdout: ms die on bug = true 2024-03-04T09:31:10.719 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:31:10.719 INFO:teuthology.orchestra.run.smithi146.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-04T09:31:10.719 INFO:teuthology.orchestra.run.smithi146.stdout: mon pg warn max object skew = 0 2024-03-04T09:31:10.719 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:31:10.720 INFO:teuthology.orchestra.run.smithi146.stdout: # disable pg_autoscaler by default for new pools 2024-03-04T09:31:10.720 INFO:teuthology.orchestra.run.smithi146.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-04T09:31:10.720 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:31:10.720 INFO:teuthology.orchestra.run.smithi146.stdout: osd pool default size = 2 2024-03-04T09:31:10.720 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:31:10.720 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd allow primary affinity = true 2024-03-04T09:31:10.720 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd allow pg remap = true 2024-03-04T09:31:10.720 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on legacy crush tunables = false 2024-03-04T09:31:10.720 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on crush straw calc version zero = false 2024-03-04T09:31:10.720 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on no sortbitwise = false 2024-03-04T09:31:10.720 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on osd down out interval zero = false 2024-03-04T09:31:10.720 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on too few osds = false 2024-03-04T09:31:10.720 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-04T09:31:10.720 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-04T09:31:10.720 INFO:teuthology.orchestra.run.smithi146.stdout: mon_allow_pool_size_one = true 2024-03-04T09:31:10.721 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:31:10.721 INFO:teuthology.orchestra.run.smithi146.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-04T09:31:10.721 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:31:10.721 INFO:teuthology.orchestra.run.smithi146.stdout: osd default data pool replay window = 5 2024-03-04T09:31:10.721 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:31:10.721 INFO:teuthology.orchestra.run.smithi146.stdout: mon allow pool delete = true 2024-03-04T09:31:10.721 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:31:10.721 INFO:teuthology.orchestra.run.smithi146.stdout: mon cluster log file level = debug 2024-03-04T09:31:10.721 INFO:teuthology.orchestra.run.smithi146.stdout: debug asserts on shutdown = true 2024-03-04T09:31:10.721 INFO:teuthology.orchestra.run.smithi146.stdout: mon health detail to clog = false 2024-03-04T09:31:10.721 INFO:teuthology.orchestra.run.smithi146.stdout: mon host = "172.21.15.6,172.21.15.146,[v2:172.21.15.146:3301,v1:172.21.15.146:6790]" 2024-03-04T09:31:10.721 INFO:teuthology.orchestra.run.smithi146.stdout: ms die on skipped message = False 2024-03-04T09:31:10.721 INFO:teuthology.orchestra.run.smithi146.stdout: fsid = edce0578-5ae4-4402-b45a-552bdef84d7c 2024-03-04T09:31:10.721 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:31:10.722 INFO:teuthology.orchestra.run.smithi146.stdout:[osd] 2024-03-04T09:31:10.722 INFO:teuthology.orchestra.run.smithi146.stdout: osd journal size = 100 2024-03-04T09:31:10.722 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:31:10.722 INFO:teuthology.orchestra.run.smithi146.stdout: osd scrub load threshold = 5.0 2024-03-04T09:31:10.722 INFO:teuthology.orchestra.run.smithi146.stdout: osd scrub max interval = 600 2024-03-04T09:31:10.722 INFO:teuthology.orchestra.run.smithi146.stdout: osd mclock profile = high_recovery_ops 2024-03-04T09:31:10.722 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:31:10.722 INFO:teuthology.orchestra.run.smithi146.stdout: osd recover clone overlap = true 2024-03-04T09:31:10.722 INFO:teuthology.orchestra.run.smithi146.stdout: osd recovery max chunk = 1048576 2024-03-04T09:31:10.722 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:31:10.722 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug shutdown = true 2024-03-04T09:31:10.722 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug op order = true 2024-03-04T09:31:10.722 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug verify stray on activate = true 2024-03-04T09:31:10.722 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug trim objects = true 2024-03-04T09:31:10.722 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:31:10.722 INFO:teuthology.orchestra.run.smithi146.stdout: osd open classes on start = true 2024-03-04T09:31:10.722 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug pg log writeout = true 2024-03-04T09:31:10.723 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:31:10.723 INFO:teuthology.orchestra.run.smithi146.stdout: osd deep scrub update digest min age = 30 2024-03-04T09:31:10.723 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:31:10.723 INFO:teuthology.orchestra.run.smithi146.stdout: osd map max advance = 10 2024-03-04T09:31:10.723 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:31:10.723 INFO:teuthology.orchestra.run.smithi146.stdout: journal zero on create = true 2024-03-04T09:31:10.723 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:31:10.723 INFO:teuthology.orchestra.run.smithi146.stdout: filestore ondisk finisher threads = 3 2024-03-04T09:31:10.723 INFO:teuthology.orchestra.run.smithi146.stdout: filestore apply finisher threads = 3 2024-03-04T09:31:10.723 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:31:10.723 INFO:teuthology.orchestra.run.smithi146.stdout: bdev debug aio = true 2024-03-04T09:31:10.723 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug misdirected ops = true 2024-03-04T09:31:10.723 INFO:teuthology.orchestra.run.smithi146.stdout: bdev async discard = True 2024-03-04T09:31:10.723 INFO:teuthology.orchestra.run.smithi146.stdout: bdev enable discard = True 2024-03-04T09:31:10.723 INFO:teuthology.orchestra.run.smithi146.stdout: bluestore allocator = bitmap 2024-03-04T09:31:10.723 INFO:teuthology.orchestra.run.smithi146.stdout: bluestore block size = 96636764160 2024-03-04T09:31:10.724 INFO:teuthology.orchestra.run.smithi146.stdout: bluestore fsck on mount = True 2024-03-04T09:31:10.724 INFO:teuthology.orchestra.run.smithi146.stdout: debug bluefs = 1/20 2024-03-04T09:31:10.724 INFO:teuthology.orchestra.run.smithi146.stdout: debug bluestore = 1/20 2024-03-04T09:31:10.724 INFO:teuthology.orchestra.run.smithi146.stdout: debug ms = 1 2024-03-04T09:31:10.724 INFO:teuthology.orchestra.run.smithi146.stdout: debug osd = 20 2024-03-04T09:31:10.724 INFO:teuthology.orchestra.run.smithi146.stdout: debug rocksdb = 4/10 2024-03-04T09:31:10.724 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd backfillfull_ratio = 0.85 2024-03-04T09:31:10.724 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd full ratio = 0.9 2024-03-04T09:31:10.724 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd nearfull ratio = 0.8 2024-03-04T09:31:10.724 INFO:teuthology.orchestra.run.smithi146.stdout: osd failsafe full ratio = 0.95 2024-03-04T09:31:10.724 INFO:teuthology.orchestra.run.smithi146.stdout: osd objectstore = bluestore 2024-03-04T09:31:10.724 INFO:teuthology.orchestra.run.smithi146.stdout: osd op complaint time = 180 2024-03-04T09:31:10.724 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:31:10.724 INFO:teuthology.orchestra.run.smithi146.stdout:[mgr] 2024-03-04T09:31:10.724 INFO:teuthology.orchestra.run.smithi146.stdout: debug ms = 1 2024-03-04T09:31:10.725 INFO:teuthology.orchestra.run.smithi146.stdout: debug mgr = 20 2024-03-04T09:31:10.725 INFO:teuthology.orchestra.run.smithi146.stdout: debug mon = 20 2024-03-04T09:31:10.725 INFO:teuthology.orchestra.run.smithi146.stdout: debug auth = 20 2024-03-04T09:31:10.725 INFO:teuthology.orchestra.run.smithi146.stdout: mon reweight min pgs per osd = 4 2024-03-04T09:31:10.725 INFO:teuthology.orchestra.run.smithi146.stdout: mon reweight min bytes per osd = 10 2024-03-04T09:31:10.725 INFO:teuthology.orchestra.run.smithi146.stdout: mgr/telemetry/nag = false 2024-03-04T09:31:10.725 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:31:10.725 INFO:teuthology.orchestra.run.smithi146.stdout:[mon] 2024-03-04T09:31:10.725 INFO:teuthology.orchestra.run.smithi146.stdout: debug ms = 1 2024-03-04T09:31:10.725 INFO:teuthology.orchestra.run.smithi146.stdout: debug mon = 20 2024-03-04T09:31:10.725 INFO:teuthology.orchestra.run.smithi146.stdout: debug paxos = 20 2024-03-04T09:31:10.725 INFO:teuthology.orchestra.run.smithi146.stdout: debug auth = 20 2024-03-04T09:31:10.725 INFO:teuthology.orchestra.run.smithi146.stdout: mon data avail warn = 5 2024-03-04T09:31:10.725 INFO:teuthology.orchestra.run.smithi146.stdout: mon mgr mkfs grace = 240 2024-03-04T09:31:10.725 INFO:teuthology.orchestra.run.smithi146.stdout: mon reweight min pgs per osd = 4 2024-03-04T09:31:10.725 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd reporter subtree level = osd 2024-03-04T09:31:10.726 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd prime pg temp = true 2024-03-04T09:31:10.726 INFO:teuthology.orchestra.run.smithi146.stdout: mon reweight min bytes per osd = 10 2024-03-04T09:31:10.726 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:31:10.726 INFO:teuthology.orchestra.run.smithi146.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-04T09:31:10.726 INFO:teuthology.orchestra.run.smithi146.stdout: auth mon ticket ttl = 660 # 11m 2024-03-04T09:31:10.726 INFO:teuthology.orchestra.run.smithi146.stdout: auth service ticket ttl = 240 # 4m 2024-03-04T09:31:10.726 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:31:10.726 INFO:teuthology.orchestra.run.smithi146.stdout: # don't complain about insecure global_id in the test suite 2024-03-04T09:31:10.726 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-04T09:31:10.726 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-04T09:31:10.726 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:31:10.726 INFO:teuthology.orchestra.run.smithi146.stdout: # 1m isn't quite enough 2024-03-04T09:31:10.726 INFO:teuthology.orchestra.run.smithi146.stdout: mon_down_mkfs_grace = 2m 2024-03-04T09:31:10.726 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:31:10.726 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_filestore_osds = false 2024-03-04T09:31:10.727 INFO:teuthology.orchestra.run.smithi146.stdout: mon op complaint time = 120 2024-03-04T09:31:10.727 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:31:10.727 INFO:teuthology.orchestra.run.smithi146.stdout:[client] 2024-03-04T09:31:10.727 INFO:teuthology.orchestra.run.smithi146.stdout: rgw cache enabled = true 2024-03-04T09:31:10.727 INFO:teuthology.orchestra.run.smithi146.stdout: rgw enable ops log = true 2024-03-04T09:31:10.727 INFO:teuthology.orchestra.run.smithi146.stdout: rgw enable usage log = true 2024-03-04T09:31:10.727 INFO:teuthology.orchestra.run.smithi146.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-04T09:31:10.727 INFO:teuthology.orchestra.run.smithi146.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-04T09:31:10.727 INFO:teuthology.orchestra.run.smithi146.stdout: client mount timeout = 600 2024-03-04T09:31:10.727 INFO:teuthology.orchestra.run.smithi146.stdout: debug client = 20 2024-03-04T09:31:10.727 INFO:teuthology.orchestra.run.smithi146.stdout: debug ms = 1 2024-03-04T09:31:10.727 INFO:teuthology.orchestra.run.smithi146.stdout: pid file = "" 2024-03-04T09:31:10.727 INFO:teuthology.orchestra.run.smithi146.stdout: rados mon op timeout = 900 2024-03-04T09:31:10.727 INFO:teuthology.orchestra.run.smithi146.stdout: rados osd op timeout = 900 2024-03-04T09:31:10.728 INFO:teuthology.orchestra.run.smithi146.stdout:[mon.a] 2024-03-04T09:31:10.728 INFO:teuthology.orchestra.run.smithi146.stdout:[mon.b] 2024-03-04T09:31:10.728 INFO:teuthology.orchestra.run.smithi146.stdout:[mon.c] 2024-03-04T09:31:10.728 INFO:teuthology.orchestra.run.smithi146.stdout:[mds.a] 2024-03-04T09:31:10.728 INFO:teuthology.orchestra.run.smithi146.stdout:[mds.c] 2024-03-04T09:31:10.728 INFO:teuthology.orchestra.run.smithi146.stdout:[mds.b] 2024-03-04T09:31:10.728 INFO:teuthology.orchestra.run.smithi146.stdout:[mds.d] 2024-03-04T09:31:10.728 INFO:teuthology.orchestra.run.smithi146.stdout:[mds] 2024-03-04T09:31:10.728 INFO:teuthology.orchestra.run.smithi146.stdout: debug mds = 20 2024-03-04T09:31:10.728 INFO:teuthology.orchestra.run.smithi146.stdout: debug mds balancer = 20 2024-03-04T09:31:10.728 INFO:teuthology.orchestra.run.smithi146.stdout: debug ms = 1 2024-03-04T09:31:10.728 INFO:teuthology.orchestra.run.smithi146.stdout: mds debug frag = True 2024-03-04T09:31:10.728 INFO:teuthology.orchestra.run.smithi146.stdout: mds debug scatterstat = True 2024-03-04T09:31:10.728 INFO:teuthology.orchestra.run.smithi146.stdout: mds op complaint time = 180 2024-03-04T09:31:10.728 INFO:teuthology.orchestra.run.smithi146.stdout: mds verify scatter = True 2024-03-04T09:31:10.729 INFO:teuthology.orchestra.run.smithi146.stdout: osd op complaint time = 180 2024-03-04T09:31:10.729 INFO:teuthology.orchestra.run.smithi146.stdout: rados mon op timeout = 900 2024-03-04T09:31:10.729 INFO:teuthology.orchestra.run.smithi146.stdout: rados osd op timeout = 900 2024-03-04T09:31:10.729 INFO:teuthology.orchestra.run.smithi146.stdout: mds_max_purge_files = 6 2024-03-04T09:31:10.729 INFO:teuthology.orchestra.run.smithi146.stdout: mds_max_purge_ops = 100000000 2024-03-04T09:31:10.758 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-04T09:31:10.765 DEBUG:teuthology.orchestra.run.smithi146:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-04T09:31:10.931 INFO:teuthology.orchestra.run.smithi146.stdout:[global] 2024-03-04T09:31:10.931 INFO:teuthology.orchestra.run.smithi146.stdout: chdir = "" 2024-03-04T09:31:10.931 INFO:teuthology.orchestra.run.smithi146.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-04T09:31:10.931 INFO:teuthology.orchestra.run.smithi146.stdout: auth supported = cephx 2024-03-04T09:31:10.931 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:31:10.931 INFO:teuthology.orchestra.run.smithi146.stdout: filestore xattr use omap = true 2024-03-04T09:31:10.931 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:31:10.931 INFO:teuthology.orchestra.run.smithi146.stdout: mon clock drift allowed = 1.000 2024-03-04T09:31:10.932 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:31:10.932 INFO:teuthology.orchestra.run.smithi146.stdout: osd crush chooseleaf type = 0 2024-03-04T09:31:10.932 INFO:teuthology.orchestra.run.smithi146.stdout: auth debug = true 2024-03-04T09:31:10.932 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:31:10.932 INFO:teuthology.orchestra.run.smithi146.stdout: ms die on old message = true 2024-03-04T09:31:10.932 INFO:teuthology.orchestra.run.smithi146.stdout: ms die on bug = true 2024-03-04T09:31:10.932 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:31:10.932 INFO:teuthology.orchestra.run.smithi146.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-04T09:31:10.932 INFO:teuthology.orchestra.run.smithi146.stdout: mon pg warn max object skew = 0 2024-03-04T09:31:10.932 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:31:10.932 INFO:teuthology.orchestra.run.smithi146.stdout: # disable pg_autoscaler by default for new pools 2024-03-04T09:31:10.932 INFO:teuthology.orchestra.run.smithi146.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-04T09:31:10.932 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:31:10.932 INFO:teuthology.orchestra.run.smithi146.stdout: osd pool default size = 2 2024-03-04T09:31:10.932 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:31:10.933 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd allow primary affinity = true 2024-03-04T09:31:10.933 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd allow pg remap = true 2024-03-04T09:31:10.933 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on legacy crush tunables = false 2024-03-04T09:31:10.933 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on crush straw calc version zero = false 2024-03-04T09:31:10.933 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on no sortbitwise = false 2024-03-04T09:31:10.933 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on osd down out interval zero = false 2024-03-04T09:31:10.933 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on too few osds = false 2024-03-04T09:31:10.933 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-04T09:31:10.933 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-04T09:31:10.933 INFO:teuthology.orchestra.run.smithi146.stdout: mon_allow_pool_size_one = true 2024-03-04T09:31:10.933 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:31:10.933 INFO:teuthology.orchestra.run.smithi146.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-04T09:31:10.933 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:31:10.933 INFO:teuthology.orchestra.run.smithi146.stdout: osd default data pool replay window = 5 2024-03-04T09:31:10.933 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:31:10.933 INFO:teuthology.orchestra.run.smithi146.stdout: mon allow pool delete = true 2024-03-04T09:31:10.934 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:31:10.934 INFO:teuthology.orchestra.run.smithi146.stdout: mon cluster log file level = debug 2024-03-04T09:31:10.934 INFO:teuthology.orchestra.run.smithi146.stdout: debug asserts on shutdown = true 2024-03-04T09:31:10.934 INFO:teuthology.orchestra.run.smithi146.stdout: mon health detail to clog = false 2024-03-04T09:31:10.934 INFO:teuthology.orchestra.run.smithi146.stdout: mon host = "172.21.15.6,172.21.15.146,[v2:172.21.15.146:3301,v1:172.21.15.146:6790]" 2024-03-04T09:31:10.934 INFO:teuthology.orchestra.run.smithi146.stdout: ms die on skipped message = False 2024-03-04T09:31:10.934 INFO:teuthology.orchestra.run.smithi146.stdout: fsid = edce0578-5ae4-4402-b45a-552bdef84d7c 2024-03-04T09:31:10.934 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:31:10.934 INFO:teuthology.orchestra.run.smithi146.stdout:[osd] 2024-03-04T09:31:10.934 INFO:teuthology.orchestra.run.smithi146.stdout: osd journal size = 100 2024-03-04T09:31:10.934 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:31:10.934 INFO:teuthology.orchestra.run.smithi146.stdout: osd scrub load threshold = 5.0 2024-03-04T09:31:10.934 INFO:teuthology.orchestra.run.smithi146.stdout: osd scrub max interval = 600 2024-03-04T09:31:10.934 INFO:teuthology.orchestra.run.smithi146.stdout: osd mclock profile = high_recovery_ops 2024-03-04T09:31:10.935 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:31:10.935 INFO:teuthology.orchestra.run.smithi146.stdout: osd recover clone overlap = true 2024-03-04T09:31:10.935 INFO:teuthology.orchestra.run.smithi146.stdout: osd recovery max chunk = 1048576 2024-03-04T09:31:10.935 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:31:10.935 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug shutdown = true 2024-03-04T09:31:10.935 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug op order = true 2024-03-04T09:31:10.935 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug verify stray on activate = true 2024-03-04T09:31:10.935 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug trim objects = true 2024-03-04T09:31:10.935 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:31:10.935 INFO:teuthology.orchestra.run.smithi146.stdout: osd open classes on start = true 2024-03-04T09:31:10.935 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug pg log writeout = true 2024-03-04T09:31:10.935 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:31:10.935 INFO:teuthology.orchestra.run.smithi146.stdout: osd deep scrub update digest min age = 30 2024-03-04T09:31:10.935 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:31:10.935 INFO:teuthology.orchestra.run.smithi146.stdout: osd map max advance = 10 2024-03-04T09:31:10.936 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:31:10.936 INFO:teuthology.orchestra.run.smithi146.stdout: journal zero on create = true 2024-03-04T09:31:10.936 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:31:10.936 INFO:teuthology.orchestra.run.smithi146.stdout: filestore ondisk finisher threads = 3 2024-03-04T09:31:10.936 INFO:teuthology.orchestra.run.smithi146.stdout: filestore apply finisher threads = 3 2024-03-04T09:31:10.936 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:31:10.936 INFO:teuthology.orchestra.run.smithi146.stdout: bdev debug aio = true 2024-03-04T09:31:10.936 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug misdirected ops = true 2024-03-04T09:31:10.936 INFO:teuthology.orchestra.run.smithi146.stdout: bdev async discard = True 2024-03-04T09:31:10.936 INFO:teuthology.orchestra.run.smithi146.stdout: bdev enable discard = True 2024-03-04T09:31:10.936 INFO:teuthology.orchestra.run.smithi146.stdout: bluestore allocator = bitmap 2024-03-04T09:31:10.936 INFO:teuthology.orchestra.run.smithi146.stdout: bluestore block size = 96636764160 2024-03-04T09:31:10.937 INFO:teuthology.orchestra.run.smithi146.stdout: bluestore fsck on mount = True 2024-03-04T09:31:10.937 INFO:teuthology.orchestra.run.smithi146.stdout: debug bluefs = 1/20 2024-03-04T09:31:10.937 INFO:teuthology.orchestra.run.smithi146.stdout: debug bluestore = 1/20 2024-03-04T09:31:10.937 INFO:teuthology.orchestra.run.smithi146.stdout: debug ms = 1 2024-03-04T09:31:10.937 INFO:teuthology.orchestra.run.smithi146.stdout: debug osd = 20 2024-03-04T09:31:10.937 INFO:teuthology.orchestra.run.smithi146.stdout: debug rocksdb = 4/10 2024-03-04T09:31:10.937 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd backfillfull_ratio = 0.85 2024-03-04T09:31:10.937 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd full ratio = 0.9 2024-03-04T09:31:10.937 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd nearfull ratio = 0.8 2024-03-04T09:31:10.937 INFO:teuthology.orchestra.run.smithi146.stdout: osd failsafe full ratio = 0.95 2024-03-04T09:31:10.937 INFO:teuthology.orchestra.run.smithi146.stdout: osd objectstore = bluestore 2024-03-04T09:31:10.937 INFO:teuthology.orchestra.run.smithi146.stdout: osd op complaint time = 180 2024-03-04T09:31:10.937 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:31:10.937 INFO:teuthology.orchestra.run.smithi146.stdout:[mgr] 2024-03-04T09:31:10.938 INFO:teuthology.orchestra.run.smithi146.stdout: debug ms = 1 2024-03-04T09:31:10.938 INFO:teuthology.orchestra.run.smithi146.stdout: debug mgr = 20 2024-03-04T09:31:10.938 INFO:teuthology.orchestra.run.smithi146.stdout: debug mon = 20 2024-03-04T09:31:10.938 INFO:teuthology.orchestra.run.smithi146.stdout: debug auth = 20 2024-03-04T09:31:10.938 INFO:teuthology.orchestra.run.smithi146.stdout: mon reweight min pgs per osd = 4 2024-03-04T09:31:10.938 INFO:teuthology.orchestra.run.smithi146.stdout: mon reweight min bytes per osd = 10 2024-03-04T09:31:10.938 INFO:teuthology.orchestra.run.smithi146.stdout: mgr/telemetry/nag = false 2024-03-04T09:31:10.938 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:31:10.938 INFO:teuthology.orchestra.run.smithi146.stdout:[mon] 2024-03-04T09:31:10.938 INFO:teuthology.orchestra.run.smithi146.stdout: debug ms = 1 2024-03-04T09:31:10.938 INFO:teuthology.orchestra.run.smithi146.stdout: debug mon = 20 2024-03-04T09:31:10.938 INFO:teuthology.orchestra.run.smithi146.stdout: debug paxos = 20 2024-03-04T09:31:10.938 INFO:teuthology.orchestra.run.smithi146.stdout: debug auth = 20 2024-03-04T09:31:10.938 INFO:teuthology.orchestra.run.smithi146.stdout: mon data avail warn = 5 2024-03-04T09:31:10.939 INFO:teuthology.orchestra.run.smithi146.stdout: mon mgr mkfs grace = 240 2024-03-04T09:31:10.939 INFO:teuthology.orchestra.run.smithi146.stdout: mon reweight min pgs per osd = 4 2024-03-04T09:31:10.939 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd reporter subtree level = osd 2024-03-04T09:31:10.939 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd prime pg temp = true 2024-03-04T09:31:10.939 INFO:teuthology.orchestra.run.smithi146.stdout: mon reweight min bytes per osd = 10 2024-03-04T09:31:10.939 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:31:10.939 INFO:teuthology.orchestra.run.smithi146.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-04T09:31:10.939 INFO:teuthology.orchestra.run.smithi146.stdout: auth mon ticket ttl = 660 # 11m 2024-03-04T09:31:10.939 INFO:teuthology.orchestra.run.smithi146.stdout: auth service ticket ttl = 240 # 4m 2024-03-04T09:31:10.939 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:31:10.939 INFO:teuthology.orchestra.run.smithi146.stdout: # don't complain about insecure global_id in the test suite 2024-03-04T09:31:10.939 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-04T09:31:10.939 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-04T09:31:10.939 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:31:10.939 INFO:teuthology.orchestra.run.smithi146.stdout: # 1m isn't quite enough 2024-03-04T09:31:10.940 INFO:teuthology.orchestra.run.smithi146.stdout: mon_down_mkfs_grace = 2m 2024-03-04T09:31:10.940 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:31:10.940 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_filestore_osds = false 2024-03-04T09:31:10.940 INFO:teuthology.orchestra.run.smithi146.stdout: mon op complaint time = 120 2024-03-04T09:31:10.940 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:31:10.940 INFO:teuthology.orchestra.run.smithi146.stdout:[client] 2024-03-04T09:31:10.940 INFO:teuthology.orchestra.run.smithi146.stdout: rgw cache enabled = true 2024-03-04T09:31:10.940 INFO:teuthology.orchestra.run.smithi146.stdout: rgw enable ops log = true 2024-03-04T09:31:10.940 INFO:teuthology.orchestra.run.smithi146.stdout: rgw enable usage log = true 2024-03-04T09:31:10.940 INFO:teuthology.orchestra.run.smithi146.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-04T09:31:10.940 INFO:teuthology.orchestra.run.smithi146.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-04T09:31:10.940 INFO:teuthology.orchestra.run.smithi146.stdout: client mount timeout = 600 2024-03-04T09:31:10.940 INFO:teuthology.orchestra.run.smithi146.stdout: debug client = 20 2024-03-04T09:31:10.940 INFO:teuthology.orchestra.run.smithi146.stdout: debug ms = 1 2024-03-04T09:31:10.941 INFO:teuthology.orchestra.run.smithi146.stdout: pid file = "" 2024-03-04T09:31:10.941 INFO:teuthology.orchestra.run.smithi146.stdout: rados mon op timeout = 900 2024-03-04T09:31:10.941 INFO:teuthology.orchestra.run.smithi146.stdout: rados osd op timeout = 900 2024-03-04T09:31:10.941 INFO:teuthology.orchestra.run.smithi146.stdout:[mon.a] 2024-03-04T09:31:10.941 INFO:teuthology.orchestra.run.smithi146.stdout:[mon.b] 2024-03-04T09:31:10.941 INFO:teuthology.orchestra.run.smithi146.stdout:[mon.c] 2024-03-04T09:31:10.941 INFO:teuthology.orchestra.run.smithi146.stdout:[mds.a] 2024-03-04T09:31:10.941 INFO:teuthology.orchestra.run.smithi146.stdout:[mds.c] 2024-03-04T09:31:10.941 INFO:teuthology.orchestra.run.smithi146.stdout:[mds.b] 2024-03-04T09:31:10.941 INFO:teuthology.orchestra.run.smithi146.stdout:[mds.d] 2024-03-04T09:31:10.941 INFO:teuthology.orchestra.run.smithi146.stdout:[mds] 2024-03-04T09:31:10.941 INFO:teuthology.orchestra.run.smithi146.stdout: debug mds = 20 2024-03-04T09:31:10.941 INFO:teuthology.orchestra.run.smithi146.stdout: debug mds balancer = 20 2024-03-04T09:31:10.941 INFO:teuthology.orchestra.run.smithi146.stdout: debug ms = 1 2024-03-04T09:31:10.941 INFO:teuthology.orchestra.run.smithi146.stdout: mds debug frag = True 2024-03-04T09:31:10.941 INFO:teuthology.orchestra.run.smithi146.stdout: mds debug scatterstat = True 2024-03-04T09:31:10.942 INFO:teuthology.orchestra.run.smithi146.stdout: mds op complaint time = 180 2024-03-04T09:31:10.942 INFO:teuthology.orchestra.run.smithi146.stdout: mds verify scatter = True 2024-03-04T09:31:10.942 INFO:teuthology.orchestra.run.smithi146.stdout: osd op complaint time = 180 2024-03-04T09:31:10.942 INFO:teuthology.orchestra.run.smithi146.stdout: rados mon op timeout = 900 2024-03-04T09:31:10.942 INFO:teuthology.orchestra.run.smithi146.stdout: rados osd op timeout = 900 2024-03-04T09:31:10.942 INFO:teuthology.orchestra.run.smithi146.stdout: mds_max_purge_files = 6 2024-03-04T09:31:10.942 INFO:teuthology.orchestra.run.smithi006.stdout:[global] 2024-03-04T09:31:10.942 INFO:teuthology.orchestra.run.smithi006.stdout: chdir = "" 2024-03-04T09:31:10.942 INFO:teuthology.orchestra.run.smithi006.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-04T09:31:10.943 INFO:teuthology.orchestra.run.smithi006.stdout: auth supported = cephx 2024-03-04T09:31:10.943 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:31:10.943 INFO:teuthology.orchestra.run.smithi006.stdout: filestore xattr use omap = true 2024-03-04T09:31:10.943 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:31:10.943 INFO:teuthology.orchestra.run.smithi006.stdout: mon clock drift allowed = 1.000 2024-03-04T09:31:10.943 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:31:10.943 INFO:teuthology.orchestra.run.smithi006.stdout: osd crush chooseleaf type = 0 2024-03-04T09:31:10.943 INFO:teuthology.orchestra.run.smithi006.stdout: auth debug = true 2024-03-04T09:31:10.943 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:31:10.943 INFO:teuthology.orchestra.run.smithi006.stdout: ms die on old message = true 2024-03-04T09:31:10.943 INFO:teuthology.orchestra.run.smithi006.stdout: ms die on bug = true 2024-03-04T09:31:10.943 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:31:10.943 INFO:teuthology.orchestra.run.smithi006.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-04T09:31:10.943 INFO:teuthology.orchestra.run.smithi006.stdout: mon pg warn max object skew = 0 2024-03-04T09:31:10.944 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:31:10.944 INFO:teuthology.orchestra.run.smithi006.stdout: # disable pg_autoscaler by default for new pools 2024-03-04T09:31:10.944 INFO:teuthology.orchestra.run.smithi006.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-04T09:31:10.944 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:31:10.944 INFO:teuthology.orchestra.run.smithi006.stdout: osd pool default size = 2 2024-03-04T09:31:10.944 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:31:10.944 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd allow primary affinity = true 2024-03-04T09:31:10.944 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd allow pg remap = true 2024-03-04T09:31:10.944 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on legacy crush tunables = false 2024-03-04T09:31:10.944 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on crush straw calc version zero = false 2024-03-04T09:31:10.944 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on no sortbitwise = false 2024-03-04T09:31:10.944 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on osd down out interval zero = false 2024-03-04T09:31:10.944 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on too few osds = false 2024-03-04T09:31:10.944 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-04T09:31:10.945 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-04T09:31:10.945 INFO:teuthology.orchestra.run.smithi006.stdout: mon_allow_pool_size_one = true 2024-03-04T09:31:10.945 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:31:10.945 INFO:teuthology.orchestra.run.smithi006.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-04T09:31:10.945 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:31:10.945 INFO:teuthology.orchestra.run.smithi006.stdout: osd default data pool replay window = 5 2024-03-04T09:31:10.945 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:31:10.945 INFO:teuthology.orchestra.run.smithi006.stdout: mon allow pool delete = true 2024-03-04T09:31:10.945 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:31:10.945 INFO:teuthology.orchestra.run.smithi006.stdout: mon cluster log file level = debug 2024-03-04T09:31:10.945 INFO:teuthology.orchestra.run.smithi006.stdout: debug asserts on shutdown = true 2024-03-04T09:31:10.945 INFO:teuthology.orchestra.run.smithi006.stdout: mon health detail to clog = false 2024-03-04T09:31:10.945 INFO:teuthology.orchestra.run.smithi006.stdout: mon host = "172.21.15.6,172.21.15.146,[v2:172.21.15.146:3301,v1:172.21.15.146:6790]" 2024-03-04T09:31:10.945 INFO:teuthology.orchestra.run.smithi006.stdout: ms die on skipped message = False 2024-03-04T09:31:10.945 INFO:teuthology.orchestra.run.smithi006.stdout: fsid = edce0578-5ae4-4402-b45a-552bdef84d7c 2024-03-04T09:31:10.946 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:31:10.946 INFO:teuthology.orchestra.run.smithi006.stdout:[osd] 2024-03-04T09:31:10.946 INFO:teuthology.orchestra.run.smithi006.stdout: osd journal size = 100 2024-03-04T09:31:10.946 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:31:10.946 INFO:teuthology.orchestra.run.smithi006.stdout: osd scrub load threshold = 5.0 2024-03-04T09:31:10.946 INFO:teuthology.orchestra.run.smithi006.stdout: osd scrub max interval = 600 2024-03-04T09:31:10.946 INFO:teuthology.orchestra.run.smithi006.stdout: osd mclock profile = high_recovery_ops 2024-03-04T09:31:10.946 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:31:10.946 INFO:teuthology.orchestra.run.smithi006.stdout: osd recover clone overlap = true 2024-03-04T09:31:10.946 INFO:teuthology.orchestra.run.smithi006.stdout: osd recovery max chunk = 1048576 2024-03-04T09:31:10.946 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:31:10.946 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug shutdown = true 2024-03-04T09:31:10.946 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug op order = true 2024-03-04T09:31:10.946 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug verify stray on activate = true 2024-03-04T09:31:10.946 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug trim objects = true 2024-03-04T09:31:10.947 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:31:10.947 INFO:teuthology.orchestra.run.smithi006.stdout: osd open classes on start = true 2024-03-04T09:31:10.947 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug pg log writeout = true 2024-03-04T09:31:10.947 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:31:10.947 INFO:teuthology.orchestra.run.smithi006.stdout: osd deep scrub update digest min age = 30 2024-03-04T09:31:10.947 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:31:10.947 INFO:teuthology.orchestra.run.smithi006.stdout: osd map max advance = 10 2024-03-04T09:31:10.947 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:31:10.947 INFO:teuthology.orchestra.run.smithi006.stdout: journal zero on create = true 2024-03-04T09:31:10.947 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:31:10.947 INFO:teuthology.orchestra.run.smithi006.stdout: filestore ondisk finisher threads = 3 2024-03-04T09:31:10.947 INFO:teuthology.orchestra.run.smithi006.stdout: filestore apply finisher threads = 3 2024-03-04T09:31:10.947 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:31:10.947 INFO:teuthology.orchestra.run.smithi006.stdout: bdev debug aio = true 2024-03-04T09:31:10.948 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug misdirected ops = true 2024-03-04T09:31:10.948 INFO:teuthology.orchestra.run.smithi006.stdout: bdev async discard = True 2024-03-04T09:31:10.948 INFO:teuthology.orchestra.run.smithi006.stdout: bdev enable discard = True 2024-03-04T09:31:10.948 INFO:teuthology.orchestra.run.smithi006.stdout: bluestore allocator = bitmap 2024-03-04T09:31:10.948 INFO:teuthology.orchestra.run.smithi006.stdout: bluestore block size = 96636764160 2024-03-04T09:31:10.948 INFO:teuthology.orchestra.run.smithi006.stdout: bluestore fsck on mount = True 2024-03-04T09:31:10.948 INFO:teuthology.orchestra.run.smithi006.stdout: debug bluefs = 1/20 2024-03-04T09:31:10.948 INFO:teuthology.orchestra.run.smithi006.stdout: debug bluestore = 1/20 2024-03-04T09:31:10.948 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2024-03-04T09:31:10.948 INFO:teuthology.orchestra.run.smithi006.stdout: debug osd = 20 2024-03-04T09:31:10.948 INFO:teuthology.orchestra.run.smithi006.stdout: debug rocksdb = 4/10 2024-03-04T09:31:10.948 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd backfillfull_ratio = 0.85 2024-03-04T09:31:10.948 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd full ratio = 0.9 2024-03-04T09:31:10.948 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd nearfull ratio = 0.8 2024-03-04T09:31:10.949 INFO:teuthology.orchestra.run.smithi006.stdout: osd failsafe full ratio = 0.95 2024-03-04T09:31:10.949 INFO:teuthology.orchestra.run.smithi006.stdout: osd objectstore = bluestore 2024-03-04T09:31:10.949 INFO:teuthology.orchestra.run.smithi006.stdout: osd op complaint time = 180 2024-03-04T09:31:10.949 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:31:10.949 INFO:teuthology.orchestra.run.smithi006.stdout:[mgr] 2024-03-04T09:31:10.949 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2024-03-04T09:31:10.949 INFO:teuthology.orchestra.run.smithi006.stdout: debug mgr = 20 2024-03-04T09:31:10.949 INFO:teuthology.orchestra.run.smithi006.stdout: debug mon = 20 2024-03-04T09:31:10.949 INFO:teuthology.orchestra.run.smithi006.stdout: debug auth = 20 2024-03-04T09:31:10.949 INFO:teuthology.orchestra.run.smithi006.stdout: mon reweight min pgs per osd = 4 2024-03-04T09:31:10.949 INFO:teuthology.orchestra.run.smithi006.stdout: mon reweight min bytes per osd = 10 2024-03-04T09:31:10.949 INFO:teuthology.orchestra.run.smithi006.stdout: mgr/telemetry/nag = false 2024-03-04T09:31:10.949 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:31:10.949 INFO:teuthology.orchestra.run.smithi006.stdout:[mon] 2024-03-04T09:31:10.950 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2024-03-04T09:31:10.950 INFO:teuthology.orchestra.run.smithi006.stdout: debug mon = 20 2024-03-04T09:31:10.950 INFO:teuthology.orchestra.run.smithi006.stdout: debug paxos = 20 2024-03-04T09:31:10.950 INFO:teuthology.orchestra.run.smithi006.stdout: debug auth = 20 2024-03-04T09:31:10.950 INFO:teuthology.orchestra.run.smithi006.stdout: mon data avail warn = 5 2024-03-04T09:31:10.950 INFO:teuthology.orchestra.run.smithi006.stdout: mon mgr mkfs grace = 240 2024-03-04T09:31:10.950 INFO:teuthology.orchestra.run.smithi006.stdout: mon reweight min pgs per osd = 4 2024-03-04T09:31:10.950 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd reporter subtree level = osd 2024-03-04T09:31:10.950 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd prime pg temp = true 2024-03-04T09:31:10.950 INFO:teuthology.orchestra.run.smithi006.stdout: mon reweight min bytes per osd = 10 2024-03-04T09:31:10.950 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:31:10.950 INFO:teuthology.orchestra.run.smithi006.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-04T09:31:10.950 INFO:teuthology.orchestra.run.smithi006.stdout: auth mon ticket ttl = 660 # 11m 2024-03-04T09:31:10.950 INFO:teuthology.orchestra.run.smithi006.stdout: auth service ticket ttl = 240 # 4m 2024-03-04T09:31:10.950 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:31:10.950 INFO:teuthology.orchestra.run.smithi006.stdout: # don't complain about insecure global_id in the test suite 2024-03-04T09:31:10.951 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-04T09:31:10.951 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-04T09:31:10.951 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:31:10.951 INFO:teuthology.orchestra.run.smithi006.stdout: # 1m isn't quite enough 2024-03-04T09:31:10.951 INFO:teuthology.orchestra.run.smithi006.stdout: mon_down_mkfs_grace = 2m 2024-03-04T09:31:10.951 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:31:10.951 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_filestore_osds = false 2024-03-04T09:31:10.951 INFO:teuthology.orchestra.run.smithi006.stdout: mon op complaint time = 120 2024-03-04T09:31:10.951 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:31:10.951 INFO:teuthology.orchestra.run.smithi006.stdout:[client] 2024-03-04T09:31:10.951 INFO:teuthology.orchestra.run.smithi006.stdout: rgw cache enabled = true 2024-03-04T09:31:10.951 INFO:teuthology.orchestra.run.smithi006.stdout: rgw enable ops log = true 2024-03-04T09:31:10.951 INFO:teuthology.orchestra.run.smithi006.stdout: rgw enable usage log = true 2024-03-04T09:31:10.951 INFO:teuthology.orchestra.run.smithi006.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-04T09:31:10.951 INFO:teuthology.orchestra.run.smithi006.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-04T09:31:10.951 INFO:teuthology.orchestra.run.smithi006.stdout: client mount timeout = 600 2024-03-04T09:31:10.952 INFO:teuthology.orchestra.run.smithi006.stdout: debug client = 20 2024-03-04T09:31:10.952 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2024-03-04T09:31:10.952 INFO:teuthology.orchestra.run.smithi006.stdout: pid file = "" 2024-03-04T09:31:10.952 INFO:teuthology.orchestra.run.smithi006.stdout: rados mon op timeout = 900 2024-03-04T09:31:10.952 INFO:teuthology.orchestra.run.smithi006.stdout: rados osd op timeout = 900 2024-03-04T09:31:10.952 INFO:teuthology.orchestra.run.smithi006.stdout:[mon.a] 2024-03-04T09:31:10.952 INFO:teuthology.orchestra.run.smithi006.stdout:[mon.b] 2024-03-04T09:31:10.952 INFO:teuthology.orchestra.run.smithi006.stdout:[mon.c] 2024-03-04T09:31:10.952 INFO:teuthology.orchestra.run.smithi006.stdout:[mds.a] 2024-03-04T09:31:10.952 INFO:teuthology.orchestra.run.smithi006.stdout:[mds.c] 2024-03-04T09:31:10.952 INFO:teuthology.orchestra.run.smithi006.stdout:[mds.b] 2024-03-04T09:31:10.952 INFO:teuthology.orchestra.run.smithi006.stdout:[mds.d] 2024-03-04T09:31:10.952 INFO:teuthology.orchestra.run.smithi006.stdout:[mds] 2024-03-04T09:31:10.952 INFO:teuthology.orchestra.run.smithi006.stdout: debug mds = 20 2024-03-04T09:31:10.953 INFO:teuthology.orchestra.run.smithi006.stdout: debug mds balancer = 20 2024-03-04T09:31:10.953 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2024-03-04T09:31:10.953 INFO:teuthology.orchestra.run.smithi006.stdout: mds debug frag = True 2024-03-04T09:31:10.953 INFO:teuthology.orchestra.run.smithi006.stdout: mds debug scatterstat = True 2024-03-04T09:31:10.953 INFO:teuthology.orchestra.run.smithi006.stdout: mds op complaint time = 180 2024-03-04T09:31:10.953 INFO:teuthology.orchestra.run.smithi006.stdout: mds verify scatter = True 2024-03-04T09:31:10.953 INFO:teuthology.orchestra.run.smithi006.stdout: osd op complaint time = 180 2024-03-04T09:31:10.953 INFO:teuthology.orchestra.run.smithi006.stdout: rados mon op timeout = 900 2024-03-04T09:31:10.953 INFO:teuthology.orchestra.run.smithi006.stdout: rados osd op timeout = 900 2024-03-04T09:31:10.953 INFO:teuthology.orchestra.run.smithi006.stdout: mds_max_purge_files = 6 2024-03-04T09:31:10.989 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-04T09:31:10.996 DEBUG:teuthology.orchestra.run.smithi146:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-04T09:31:11.167 INFO:teuthology.orchestra.run.smithi146.stdout:[global] 2024-03-04T09:31:11.167 INFO:teuthology.orchestra.run.smithi146.stdout: chdir = "" 2024-03-04T09:31:11.167 INFO:teuthology.orchestra.run.smithi146.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-04T09:31:11.167 INFO:teuthology.orchestra.run.smithi146.stdout: auth supported = cephx 2024-03-04T09:31:11.167 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:31:11.167 INFO:teuthology.orchestra.run.smithi146.stdout: filestore xattr use omap = true 2024-03-04T09:31:11.167 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:31:11.167 INFO:teuthology.orchestra.run.smithi146.stdout: mon clock drift allowed = 1.000 2024-03-04T09:31:11.167 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:31:11.167 INFO:teuthology.orchestra.run.smithi146.stdout: osd crush chooseleaf type = 0 2024-03-04T09:31:11.167 INFO:teuthology.orchestra.run.smithi146.stdout: auth debug = true 2024-03-04T09:31:11.168 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:31:11.168 INFO:teuthology.orchestra.run.smithi146.stdout: ms die on old message = true 2024-03-04T09:31:11.168 INFO:teuthology.orchestra.run.smithi146.stdout: ms die on bug = true 2024-03-04T09:31:11.168 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:31:11.168 INFO:teuthology.orchestra.run.smithi146.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-04T09:31:11.168 INFO:teuthology.orchestra.run.smithi146.stdout: mon pg warn max object skew = 0 2024-03-04T09:31:11.168 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:31:11.168 INFO:teuthology.orchestra.run.smithi146.stdout: # disable pg_autoscaler by default for new pools 2024-03-04T09:31:11.168 INFO:teuthology.orchestra.run.smithi146.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-04T09:31:11.168 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:31:11.168 INFO:teuthology.orchestra.run.smithi146.stdout: osd pool default size = 2 2024-03-04T09:31:11.168 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:31:11.168 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd allow primary affinity = true 2024-03-04T09:31:11.168 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd allow pg remap = true 2024-03-04T09:31:11.169 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on legacy crush tunables = false 2024-03-04T09:31:11.169 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on crush straw calc version zero = false 2024-03-04T09:31:11.169 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on no sortbitwise = false 2024-03-04T09:31:11.169 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on osd down out interval zero = false 2024-03-04T09:31:11.169 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on too few osds = false 2024-03-04T09:31:11.169 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-04T09:31:11.169 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-04T09:31:11.169 INFO:teuthology.orchestra.run.smithi146.stdout: mon_allow_pool_size_one = true 2024-03-04T09:31:11.169 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:31:11.169 INFO:teuthology.orchestra.run.smithi146.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-04T09:31:11.169 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:31:11.169 INFO:teuthology.orchestra.run.smithi146.stdout: osd default data pool replay window = 5 2024-03-04T09:31:11.169 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:31:11.169 INFO:teuthology.orchestra.run.smithi146.stdout: mon allow pool delete = true 2024-03-04T09:31:11.169 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:31:11.170 INFO:teuthology.orchestra.run.smithi146.stdout: mon cluster log file level = debug 2024-03-04T09:31:11.170 INFO:teuthology.orchestra.run.smithi146.stdout: debug asserts on shutdown = true 2024-03-04T09:31:11.170 INFO:teuthology.orchestra.run.smithi146.stdout: mon health detail to clog = false 2024-03-04T09:31:11.170 INFO:teuthology.orchestra.run.smithi146.stdout: mon host = "172.21.15.6,172.21.15.146,[v2:172.21.15.146:3301,v1:172.21.15.146:6790]" 2024-03-04T09:31:11.170 INFO:teuthology.orchestra.run.smithi146.stdout: ms die on skipped message = False 2024-03-04T09:31:11.170 INFO:teuthology.orchestra.run.smithi146.stdout: fsid = edce0578-5ae4-4402-b45a-552bdef84d7c 2024-03-04T09:31:11.170 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:31:11.170 INFO:teuthology.orchestra.run.smithi146.stdout:[osd] 2024-03-04T09:31:11.170 INFO:teuthology.orchestra.run.smithi146.stdout: osd journal size = 100 2024-03-04T09:31:11.170 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:31:11.170 INFO:teuthology.orchestra.run.smithi146.stdout: osd scrub load threshold = 5.0 2024-03-04T09:31:11.170 INFO:teuthology.orchestra.run.smithi146.stdout: osd scrub max interval = 600 2024-03-04T09:31:11.170 INFO:teuthology.orchestra.run.smithi146.stdout: osd mclock profile = high_recovery_ops 2024-03-04T09:31:11.170 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:31:11.170 INFO:teuthology.orchestra.run.smithi146.stdout: osd recover clone overlap = true 2024-03-04T09:31:11.171 INFO:teuthology.orchestra.run.smithi146.stdout: osd recovery max chunk = 1048576 2024-03-04T09:31:11.171 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:31:11.171 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug shutdown = true 2024-03-04T09:31:11.171 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug op order = true 2024-03-04T09:31:11.171 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug verify stray on activate = true 2024-03-04T09:31:11.171 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug trim objects = true 2024-03-04T09:31:11.171 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:31:11.171 INFO:teuthology.orchestra.run.smithi146.stdout: osd open classes on start = true 2024-03-04T09:31:11.171 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug pg log writeout = true 2024-03-04T09:31:11.171 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:31:11.171 INFO:teuthology.orchestra.run.smithi146.stdout: osd deep scrub update digest min age = 30 2024-03-04T09:31:11.171 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:31:11.171 INFO:teuthology.orchestra.run.smithi146.stdout: osd map max advance = 10 2024-03-04T09:31:11.171 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:31:11.171 INFO:teuthology.orchestra.run.smithi146.stdout: journal zero on create = true 2024-03-04T09:31:11.172 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:31:11.172 INFO:teuthology.orchestra.run.smithi146.stdout: filestore ondisk finisher threads = 3 2024-03-04T09:31:11.172 INFO:teuthology.orchestra.run.smithi146.stdout: filestore apply finisher threads = 3 2024-03-04T09:31:11.172 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:31:11.172 INFO:teuthology.orchestra.run.smithi146.stdout: bdev debug aio = true 2024-03-04T09:31:11.172 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug misdirected ops = true 2024-03-04T09:31:11.172 INFO:teuthology.orchestra.run.smithi146.stdout: bdev async discard = True 2024-03-04T09:31:11.172 INFO:teuthology.orchestra.run.smithi146.stdout: bdev enable discard = True 2024-03-04T09:31:11.172 INFO:teuthology.orchestra.run.smithi146.stdout: bluestore allocator = bitmap 2024-03-04T09:31:11.172 INFO:teuthology.orchestra.run.smithi146.stdout: bluestore block size = 96636764160 2024-03-04T09:31:11.172 INFO:teuthology.orchestra.run.smithi146.stdout: bluestore fsck on mount = True 2024-03-04T09:31:11.172 INFO:teuthology.orchestra.run.smithi146.stdout: debug bluefs = 1/20 2024-03-04T09:31:11.172 INFO:teuthology.orchestra.run.smithi146.stdout: debug bluestore = 1/20 2024-03-04T09:31:11.172 INFO:teuthology.orchestra.run.smithi146.stdout: debug ms = 1 2024-03-04T09:31:11.172 INFO:teuthology.orchestra.run.smithi146.stdout: debug osd = 20 2024-03-04T09:31:11.173 INFO:teuthology.orchestra.run.smithi146.stdout: debug rocksdb = 4/10 2024-03-04T09:31:11.173 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd backfillfull_ratio = 0.85 2024-03-04T09:31:11.173 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd full ratio = 0.9 2024-03-04T09:31:11.173 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd nearfull ratio = 0.8 2024-03-04T09:31:11.173 INFO:teuthology.orchestra.run.smithi146.stdout: osd failsafe full ratio = 0.95 2024-03-04T09:31:11.173 INFO:teuthology.orchestra.run.smithi146.stdout: osd objectstore = bluestore 2024-03-04T09:31:11.173 INFO:teuthology.orchestra.run.smithi146.stdout: osd op complaint time = 180 2024-03-04T09:31:11.173 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:31:11.173 INFO:teuthology.orchestra.run.smithi146.stdout:[mgr] 2024-03-04T09:31:11.173 INFO:teuthology.orchestra.run.smithi146.stdout: debug ms = 1 2024-03-04T09:31:11.173 INFO:teuthology.orchestra.run.smithi146.stdout: debug mgr = 20 2024-03-04T09:31:11.173 INFO:teuthology.orchestra.run.smithi146.stdout: debug mon = 20 2024-03-04T09:31:11.173 INFO:teuthology.orchestra.run.smithi146.stdout: debug auth = 20 2024-03-04T09:31:11.173 INFO:teuthology.orchestra.run.smithi146.stdout: mon reweight min pgs per osd = 4 2024-03-04T09:31:11.173 INFO:teuthology.orchestra.run.smithi146.stdout: mon reweight min bytes per osd = 10 2024-03-04T09:31:11.174 INFO:teuthology.orchestra.run.smithi146.stdout: mgr/telemetry/nag = false 2024-03-04T09:31:11.174 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:31:11.174 INFO:teuthology.orchestra.run.smithi146.stdout:[mon] 2024-03-04T09:31:11.174 INFO:teuthology.orchestra.run.smithi146.stdout: debug ms = 1 2024-03-04T09:31:11.174 INFO:teuthology.orchestra.run.smithi146.stdout: debug mon = 20 2024-03-04T09:31:11.174 INFO:teuthology.orchestra.run.smithi146.stdout: debug paxos = 20 2024-03-04T09:31:11.174 INFO:teuthology.orchestra.run.smithi146.stdout: debug auth = 20 2024-03-04T09:31:11.174 INFO:teuthology.orchestra.run.smithi146.stdout: mon data avail warn = 5 2024-03-04T09:31:11.174 INFO:teuthology.orchestra.run.smithi146.stdout: mon mgr mkfs grace = 240 2024-03-04T09:31:11.174 INFO:teuthology.orchestra.run.smithi146.stdout: mon reweight min pgs per osd = 4 2024-03-04T09:31:11.174 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd reporter subtree level = osd 2024-03-04T09:31:11.174 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd prime pg temp = true 2024-03-04T09:31:11.174 INFO:teuthology.orchestra.run.smithi146.stdout: mon reweight min bytes per osd = 10 2024-03-04T09:31:11.174 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:31:11.174 INFO:teuthology.orchestra.run.smithi146.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-04T09:31:11.174 INFO:teuthology.orchestra.run.smithi146.stdout: auth mon ticket ttl = 660 # 11m 2024-03-04T09:31:11.175 INFO:teuthology.orchestra.run.smithi146.stdout: auth service ticket ttl = 240 # 4m 2024-03-04T09:31:11.175 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:31:11.175 INFO:teuthology.orchestra.run.smithi146.stdout: # don't complain about insecure global_id in the test suite 2024-03-04T09:31:11.175 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-04T09:31:11.175 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-04T09:31:11.175 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:31:11.175 INFO:teuthology.orchestra.run.smithi146.stdout: # 1m isn't quite enough 2024-03-04T09:31:11.175 INFO:teuthology.orchestra.run.smithi146.stdout: mon_down_mkfs_grace = 2m 2024-03-04T09:31:11.175 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:31:11.175 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_filestore_osds = false 2024-03-04T09:31:11.175 INFO:teuthology.orchestra.run.smithi146.stdout: mon op complaint time = 120 2024-03-04T09:31:11.175 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-04T09:31:11.175 INFO:teuthology.orchestra.run.smithi146.stdout:[client] 2024-03-04T09:31:11.175 INFO:teuthology.orchestra.run.smithi146.stdout: rgw cache enabled = true 2024-03-04T09:31:11.175 INFO:teuthology.orchestra.run.smithi146.stdout: rgw enable ops log = true 2024-03-04T09:31:11.176 INFO:teuthology.orchestra.run.smithi146.stdout: rgw enable usage log = true 2024-03-04T09:31:11.176 INFO:teuthology.orchestra.run.smithi146.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-04T09:31:11.176 INFO:teuthology.orchestra.run.smithi146.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-04T09:31:11.176 INFO:teuthology.orchestra.run.smithi146.stdout: client mount timeout = 600 2024-03-04T09:31:11.176 INFO:teuthology.orchestra.run.smithi146.stdout: debug client = 20 2024-03-04T09:31:11.176 INFO:teuthology.orchestra.run.smithi146.stdout: debug ms = 1 2024-03-04T09:31:11.176 INFO:teuthology.orchestra.run.smithi146.stdout: pid file = "" 2024-03-04T09:31:11.176 INFO:teuthology.orchestra.run.smithi146.stdout: rados mon op timeout = 900 2024-03-04T09:31:11.176 INFO:teuthology.orchestra.run.smithi146.stdout: rados osd op timeout = 900 2024-03-04T09:31:11.176 INFO:teuthology.orchestra.run.smithi146.stdout:[mon.a] 2024-03-04T09:31:11.176 INFO:teuthology.orchestra.run.smithi146.stdout:[mon.b] 2024-03-04T09:31:11.176 INFO:teuthology.orchestra.run.smithi146.stdout:[mon.c] 2024-03-04T09:31:11.176 INFO:teuthology.orchestra.run.smithi146.stdout:[mds.a] 2024-03-04T09:31:11.176 INFO:teuthology.orchestra.run.smithi146.stdout:[mds.c] 2024-03-04T09:31:11.176 INFO:teuthology.orchestra.run.smithi146.stdout:[mds.b] 2024-03-04T09:31:11.177 INFO:teuthology.orchestra.run.smithi146.stdout:[mds.d] 2024-03-04T09:31:11.177 INFO:teuthology.orchestra.run.smithi146.stdout:[mds] 2024-03-04T09:31:11.177 INFO:teuthology.orchestra.run.smithi146.stdout: debug mds = 20 2024-03-04T09:31:11.177 INFO:teuthology.orchestra.run.smithi146.stdout: debug mds balancer = 20 2024-03-04T09:31:11.177 INFO:teuthology.orchestra.run.smithi146.stdout: debug ms = 1 2024-03-04T09:31:11.177 INFO:teuthology.orchestra.run.smithi146.stdout: mds debug frag = True 2024-03-04T09:31:11.177 INFO:teuthology.orchestra.run.smithi146.stdout: mds debug scatterstat = True 2024-03-04T09:31:11.177 INFO:teuthology.orchestra.run.smithi146.stdout: mds op complaint time = 180 2024-03-04T09:31:11.177 INFO:teuthology.orchestra.run.smithi146.stdout: mds verify scatter = True 2024-03-04T09:31:11.177 INFO:teuthology.orchestra.run.smithi146.stdout: osd op complaint time = 180 2024-03-04T09:31:11.177 INFO:teuthology.orchestra.run.smithi146.stdout: rados mon op timeout = 900 2024-03-04T09:31:11.177 INFO:teuthology.orchestra.run.smithi146.stdout: rados osd op timeout = 900 2024-03-04T09:31:11.178 INFO:teuthology.orchestra.run.smithi006.stdout:[global] 2024-03-04T09:31:11.178 INFO:teuthology.orchestra.run.smithi006.stdout: chdir = "" 2024-03-04T09:31:11.178 INFO:teuthology.orchestra.run.smithi006.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-04T09:31:11.178 INFO:teuthology.orchestra.run.smithi006.stdout: auth supported = cephx 2024-03-04T09:31:11.178 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:31:11.178 INFO:teuthology.orchestra.run.smithi006.stdout: filestore xattr use omap = true 2024-03-04T09:31:11.178 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:31:11.178 INFO:teuthology.orchestra.run.smithi006.stdout: mon clock drift allowed = 1.000 2024-03-04T09:31:11.178 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:31:11.178 INFO:teuthology.orchestra.run.smithi006.stdout: osd crush chooseleaf type = 0 2024-03-04T09:31:11.178 INFO:teuthology.orchestra.run.smithi006.stdout: auth debug = true 2024-03-04T09:31:11.179 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:31:11.179 INFO:teuthology.orchestra.run.smithi006.stdout: ms die on old message = true 2024-03-04T09:31:11.179 INFO:teuthology.orchestra.run.smithi006.stdout: ms die on bug = true 2024-03-04T09:31:11.179 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:31:11.179 INFO:teuthology.orchestra.run.smithi006.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-04T09:31:11.179 INFO:teuthology.orchestra.run.smithi006.stdout: mon pg warn max object skew = 0 2024-03-04T09:31:11.179 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:31:11.179 INFO:teuthology.orchestra.run.smithi006.stdout: # disable pg_autoscaler by default for new pools 2024-03-04T09:31:11.179 INFO:teuthology.orchestra.run.smithi006.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-04T09:31:11.179 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:31:11.179 INFO:teuthology.orchestra.run.smithi006.stdout: osd pool default size = 2 2024-03-04T09:31:11.179 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:31:11.179 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd allow primary affinity = true 2024-03-04T09:31:11.179 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd allow pg remap = true 2024-03-04T09:31:11.179 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on legacy crush tunables = false 2024-03-04T09:31:11.180 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on crush straw calc version zero = false 2024-03-04T09:31:11.180 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on no sortbitwise = false 2024-03-04T09:31:11.180 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on osd down out interval zero = false 2024-03-04T09:31:11.180 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on too few osds = false 2024-03-04T09:31:11.180 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-04T09:31:11.180 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-04T09:31:11.180 INFO:teuthology.orchestra.run.smithi006.stdout: mon_allow_pool_size_one = true 2024-03-04T09:31:11.180 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:31:11.180 INFO:teuthology.orchestra.run.smithi006.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-04T09:31:11.180 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:31:11.180 INFO:teuthology.orchestra.run.smithi006.stdout: osd default data pool replay window = 5 2024-03-04T09:31:11.180 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:31:11.180 INFO:teuthology.orchestra.run.smithi006.stdout: mon allow pool delete = true 2024-03-04T09:31:11.180 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:31:11.181 INFO:teuthology.orchestra.run.smithi006.stdout: mon cluster log file level = debug 2024-03-04T09:31:11.181 INFO:teuthology.orchestra.run.smithi006.stdout: debug asserts on shutdown = true 2024-03-04T09:31:11.181 INFO:teuthology.orchestra.run.smithi006.stdout: mon health detail to clog = false 2024-03-04T09:31:11.181 INFO:teuthology.orchestra.run.smithi006.stdout: mon host = "172.21.15.6,172.21.15.146,[v2:172.21.15.146:3301,v1:172.21.15.146:6790]" 2024-03-04T09:31:11.181 INFO:teuthology.orchestra.run.smithi006.stdout: ms die on skipped message = False 2024-03-04T09:31:11.181 INFO:teuthology.orchestra.run.smithi006.stdout: fsid = edce0578-5ae4-4402-b45a-552bdef84d7c 2024-03-04T09:31:11.181 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:31:11.181 INFO:teuthology.orchestra.run.smithi006.stdout:[osd] 2024-03-04T09:31:11.181 INFO:teuthology.orchestra.run.smithi006.stdout: osd journal size = 100 2024-03-04T09:31:11.181 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:31:11.181 INFO:teuthology.orchestra.run.smithi006.stdout: osd scrub load threshold = 5.0 2024-03-04T09:31:11.181 INFO:teuthology.orchestra.run.smithi006.stdout: osd scrub max interval = 600 2024-03-04T09:31:11.181 INFO:teuthology.orchestra.run.smithi006.stdout: osd mclock profile = high_recovery_ops 2024-03-04T09:31:11.181 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:31:11.181 INFO:teuthology.orchestra.run.smithi006.stdout: osd recover clone overlap = true 2024-03-04T09:31:11.182 INFO:teuthology.orchestra.run.smithi006.stdout: osd recovery max chunk = 1048576 2024-03-04T09:31:11.182 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:31:11.182 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug shutdown = true 2024-03-04T09:31:11.182 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug op order = true 2024-03-04T09:31:11.182 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug verify stray on activate = true 2024-03-04T09:31:11.182 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug trim objects = true 2024-03-04T09:31:11.182 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:31:11.182 INFO:teuthology.orchestra.run.smithi006.stdout: osd open classes on start = true 2024-03-04T09:31:11.182 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug pg log writeout = true 2024-03-04T09:31:11.182 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:31:11.182 INFO:teuthology.orchestra.run.smithi006.stdout: osd deep scrub update digest min age = 30 2024-03-04T09:31:11.182 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:31:11.182 INFO:teuthology.orchestra.run.smithi006.stdout: osd map max advance = 10 2024-03-04T09:31:11.182 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:31:11.182 INFO:teuthology.orchestra.run.smithi006.stdout: journal zero on create = true 2024-03-04T09:31:11.183 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:31:11.183 INFO:teuthology.orchestra.run.smithi006.stdout: filestore ondisk finisher threads = 3 2024-03-04T09:31:11.183 INFO:teuthology.orchestra.run.smithi006.stdout: filestore apply finisher threads = 3 2024-03-04T09:31:11.183 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:31:11.183 INFO:teuthology.orchestra.run.smithi006.stdout: bdev debug aio = true 2024-03-04T09:31:11.183 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug misdirected ops = true 2024-03-04T09:31:11.183 INFO:teuthology.orchestra.run.smithi006.stdout: bdev async discard = True 2024-03-04T09:31:11.183 INFO:teuthology.orchestra.run.smithi006.stdout: bdev enable discard = True 2024-03-04T09:31:11.183 INFO:teuthology.orchestra.run.smithi006.stdout: bluestore allocator = bitmap 2024-03-04T09:31:11.183 INFO:teuthology.orchestra.run.smithi006.stdout: bluestore block size = 96636764160 2024-03-04T09:31:11.183 INFO:teuthology.orchestra.run.smithi006.stdout: bluestore fsck on mount = True 2024-03-04T09:31:11.183 INFO:teuthology.orchestra.run.smithi006.stdout: debug bluefs = 1/20 2024-03-04T09:31:11.183 INFO:teuthology.orchestra.run.smithi006.stdout: debug bluestore = 1/20 2024-03-04T09:31:11.183 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2024-03-04T09:31:11.183 INFO:teuthology.orchestra.run.smithi006.stdout: debug osd = 20 2024-03-04T09:31:11.184 INFO:teuthology.orchestra.run.smithi006.stdout: debug rocksdb = 4/10 2024-03-04T09:31:11.184 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd backfillfull_ratio = 0.85 2024-03-04T09:31:11.184 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd full ratio = 0.9 2024-03-04T09:31:11.184 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd nearfull ratio = 0.8 2024-03-04T09:31:11.184 INFO:teuthology.orchestra.run.smithi006.stdout: osd failsafe full ratio = 0.95 2024-03-04T09:31:11.184 INFO:teuthology.orchestra.run.smithi006.stdout: osd objectstore = bluestore 2024-03-04T09:31:11.184 INFO:teuthology.orchestra.run.smithi006.stdout: osd op complaint time = 180 2024-03-04T09:31:11.184 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:31:11.184 INFO:teuthology.orchestra.run.smithi006.stdout:[mgr] 2024-03-04T09:31:11.184 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2024-03-04T09:31:11.184 INFO:teuthology.orchestra.run.smithi006.stdout: debug mgr = 20 2024-03-04T09:31:11.184 INFO:teuthology.orchestra.run.smithi006.stdout: debug mon = 20 2024-03-04T09:31:11.184 INFO:teuthology.orchestra.run.smithi006.stdout: debug auth = 20 2024-03-04T09:31:11.184 INFO:teuthology.orchestra.run.smithi006.stdout: mon reweight min pgs per osd = 4 2024-03-04T09:31:11.184 INFO:teuthology.orchestra.run.smithi006.stdout: mon reweight min bytes per osd = 10 2024-03-04T09:31:11.185 INFO:teuthology.orchestra.run.smithi006.stdout: mgr/telemetry/nag = false 2024-03-04T09:31:11.185 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:31:11.185 INFO:teuthology.orchestra.run.smithi006.stdout:[mon] 2024-03-04T09:31:11.185 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2024-03-04T09:31:11.185 INFO:teuthology.orchestra.run.smithi006.stdout: debug mon = 20 2024-03-04T09:31:11.185 INFO:teuthology.orchestra.run.smithi006.stdout: debug paxos = 20 2024-03-04T09:31:11.185 INFO:teuthology.orchestra.run.smithi006.stdout: debug auth = 20 2024-03-04T09:31:11.185 INFO:teuthology.orchestra.run.smithi006.stdout: mon data avail warn = 5 2024-03-04T09:31:11.185 INFO:teuthology.orchestra.run.smithi006.stdout: mon mgr mkfs grace = 240 2024-03-04T09:31:11.185 INFO:teuthology.orchestra.run.smithi006.stdout: mon reweight min pgs per osd = 4 2024-03-04T09:31:11.185 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd reporter subtree level = osd 2024-03-04T09:31:11.185 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd prime pg temp = true 2024-03-04T09:31:11.185 INFO:teuthology.orchestra.run.smithi006.stdout: mon reweight min bytes per osd = 10 2024-03-04T09:31:11.185 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:31:11.185 INFO:teuthology.orchestra.run.smithi006.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-04T09:31:11.185 INFO:teuthology.orchestra.run.smithi006.stdout: auth mon ticket ttl = 660 # 11m 2024-03-04T09:31:11.186 INFO:teuthology.orchestra.run.smithi006.stdout: auth service ticket ttl = 240 # 4m 2024-03-04T09:31:11.186 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:31:11.186 INFO:teuthology.orchestra.run.smithi006.stdout: # don't complain about insecure global_id in the test suite 2024-03-04T09:31:11.186 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-04T09:31:11.186 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-04T09:31:11.186 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:31:11.186 INFO:teuthology.orchestra.run.smithi006.stdout: # 1m isn't quite enough 2024-03-04T09:31:11.186 INFO:teuthology.orchestra.run.smithi006.stdout: mon_down_mkfs_grace = 2m 2024-03-04T09:31:11.186 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:31:11.186 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_filestore_osds = false 2024-03-04T09:31:11.186 INFO:teuthology.orchestra.run.smithi006.stdout: mon op complaint time = 120 2024-03-04T09:31:11.186 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:31:11.186 INFO:teuthology.orchestra.run.smithi006.stdout:[client] 2024-03-04T09:31:11.186 INFO:teuthology.orchestra.run.smithi006.stdout: rgw cache enabled = true 2024-03-04T09:31:11.186 INFO:teuthology.orchestra.run.smithi006.stdout: rgw enable ops log = true 2024-03-04T09:31:11.187 INFO:teuthology.orchestra.run.smithi006.stdout: rgw enable usage log = true 2024-03-04T09:31:11.187 INFO:teuthology.orchestra.run.smithi006.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-04T09:31:11.187 INFO:teuthology.orchestra.run.smithi006.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-04T09:31:11.187 INFO:teuthology.orchestra.run.smithi006.stdout: client mount timeout = 600 2024-03-04T09:31:11.187 INFO:teuthology.orchestra.run.smithi006.stdout: debug client = 20 2024-03-04T09:31:11.187 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2024-03-04T09:31:11.187 INFO:teuthology.orchestra.run.smithi006.stdout: pid file = "" 2024-03-04T09:31:11.187 INFO:teuthology.orchestra.run.smithi006.stdout: rados mon op timeout = 900 2024-03-04T09:31:11.187 INFO:teuthology.orchestra.run.smithi006.stdout: rados osd op timeout = 900 2024-03-04T09:31:11.187 INFO:teuthology.orchestra.run.smithi006.stdout:[mon.a] 2024-03-04T09:31:11.187 INFO:teuthology.orchestra.run.smithi006.stdout:[mon.b] 2024-03-04T09:31:11.187 INFO:teuthology.orchestra.run.smithi006.stdout:[mon.c] 2024-03-04T09:31:11.187 INFO:teuthology.orchestra.run.smithi006.stdout:[mds.a] 2024-03-04T09:31:11.187 INFO:teuthology.orchestra.run.smithi006.stdout:[mds.c] 2024-03-04T09:31:11.187 INFO:teuthology.orchestra.run.smithi006.stdout:[mds.b] 2024-03-04T09:31:11.188 INFO:teuthology.orchestra.run.smithi006.stdout:[mds.d] 2024-03-04T09:31:11.188 INFO:teuthology.orchestra.run.smithi006.stdout:[mds] 2024-03-04T09:31:11.188 INFO:teuthology.orchestra.run.smithi006.stdout: debug mds = 20 2024-03-04T09:31:11.188 INFO:teuthology.orchestra.run.smithi006.stdout: debug mds balancer = 20 2024-03-04T09:31:11.188 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2024-03-04T09:31:11.188 INFO:teuthology.orchestra.run.smithi006.stdout: mds debug frag = True 2024-03-04T09:31:11.188 INFO:teuthology.orchestra.run.smithi006.stdout: mds debug scatterstat = True 2024-03-04T09:31:11.188 INFO:teuthology.orchestra.run.smithi006.stdout: mds op complaint time = 180 2024-03-04T09:31:11.188 INFO:teuthology.orchestra.run.smithi006.stdout: mds verify scatter = True 2024-03-04T09:31:11.188 INFO:teuthology.orchestra.run.smithi006.stdout: osd op complaint time = 180 2024-03-04T09:31:11.188 INFO:teuthology.orchestra.run.smithi006.stdout: rados mon op timeout = 900 2024-03-04T09:31:11.188 INFO:teuthology.orchestra.run.smithi006.stdout: rados osd op timeout = 900 2024-03-04T09:31:11.223 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-04T09:31:11.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.453+0000 7f3f70800640 1 Processor -- start 2024-03-04T09:31:11.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.453+0000 7f3f70800640 1 -- start start 2024-03-04T09:31:11.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.455+0000 7f3f70800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f3f680b3350 0x7f3f680d24c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:11.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.455+0000 7f3f70800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3f680b49e0 0x7f3f680d2a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:11.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.455+0000 7f3f70800640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f680d4aa0 con 0x7f3f680b2f70 2024-03-04T09:31:11.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.455+0000 7f3f70800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f680d1d60 con 0x7f3f680b3350 2024-03-04T09:31:11.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.455+0000 7f3f70800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f680b38e0 con 0x7f3f680b49e0 2024-03-04T09:31:11.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.456+0000 7f3f6d400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f3f680b3350 0x7f3f680d24c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:11.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.456+0000 7f3f6d400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f3f680b3350 0x7f3f680d24c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55000/0 (socket says 172.21.15.6:55000) 2024-03-04T09:31:11.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.456+0000 7f3f6d400640 1 -- 172.21.15.6:0/2480969129 learned_addr learned my addr 172.21.15.6:0/2480969129 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:31:11.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.456+0000 7f3f6e800640 1 --2- 172.21.15.6:0/2480969129 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3f680b49e0 0x7f3f680d2a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:11.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.456+0000 7f3f6d400640 1 -- 172.21.15.6:0/2480969129 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3f680b49e0 msgr2=0x7f3f680d2a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:11.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.456+0000 7f3f6d400640 1 --2- 172.21.15.6:0/2480969129 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3f680b49e0 0x7f3f680d2a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:11.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.456+0000 7f3f6d400640 1 -- 172.21.15.6:0/2480969129 >> v1:172.21.15.146:6789/0 conn(0x7f3f680b2f70 legacy=0x7f3f680ceaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:11.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.456+0000 7f3f6d400640 1 -- 172.21.15.6:0/2480969129 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f680d2040 con 0x7f3f680b3350 2024-03-04T09:31:11.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.457+0000 7f3f6d400640 1 --2- 172.21.15.6:0/2480969129 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3f680b3350 0x7f3f680d24c0 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7f3f58009870 tx=0x7f3f5802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c8c7bae63e6cd59 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:11.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.458+0000 7f3f6ca00640 1 -- 172.21.15.6:0/2480969129 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f58034070 con 0x7f3f680b3350 2024-03-04T09:31:11.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.458+0000 7f3f6ca00640 1 -- 172.21.15.6:0/2480969129 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f58030840 con 0x7f3f680b3350 2024-03-04T09:31:11.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.458+0000 7f3f6ca00640 1 -- 172.21.15.6:0/2480969129 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f58030c10 con 0x7f3f680b3350 2024-03-04T09:31:11.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.458+0000 7f3f70800640 1 -- 172.21.15.6:0/2480969129 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3f680b3350 msgr2=0x7f3f680d24c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:11.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.458+0000 7f3f70800640 1 --2- 172.21.15.6:0/2480969129 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3f680b3350 0x7f3f680d24c0 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7f3f58009870 tx=0x7f3f5802b980 comp rx=0 tx=0).stop 2024-03-04T09:31:11.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.459+0000 7f3f70800640 1 -- 172.21.15.6:0/2480969129 shutdown_connections 2024-03-04T09:31:11.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.459+0000 7f3f70800640 1 --2- 172.21.15.6:0/2480969129 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3f680b49e0 0x7f3f680d2a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:11.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.459+0000 7f3f70800640 1 --2- 172.21.15.6:0/2480969129 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3f680b3350 0x7f3f680d24c0 unknown :-1 s=CLOSED pgs=1202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:11.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.459+0000 7f3f70800640 1 -- 172.21.15.6:0/2480969129 >> 172.21.15.6:0/2480969129 conn(0x7f3f680d3ee0 msgr2=0x7f3f680d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:11.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.459+0000 7f3f70800640 1 -- 172.21.15.6:0/2480969129 shutdown_connections 2024-03-04T09:31:11.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.459+0000 7f3f70800640 1 -- 172.21.15.6:0/2480969129 wait complete. 2024-03-04T09:31:11.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.460+0000 7f3f70800640 1 Processor -- start 2024-03-04T09:31:11.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.461+0000 7f3f70800640 1 -- start start 2024-03-04T09:31:11.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.461+0000 7f3f70800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3f680b2f70 0x7f3f6814d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:11.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.461+0000 7f3f70800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3f680b3350 0x7f3f6815d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:11.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.461+0000 7f3f70800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3f680b49e0 0x7f3f6806c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:11.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.461+0000 7f3f70800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f680cf200 con 0x7f3f680b3350 2024-03-04T09:31:11.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.461+0000 7f3f70800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f680d4d40 con 0x7f3f680b49e0 2024-03-04T09:31:11.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.461+0000 7f3f70800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f680b4f10 con 0x7f3f680b2f70 2024-03-04T09:31:11.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.462+0000 7f3f6d400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3f680b3350 0x7f3f6815d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:11.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.462+0000 7f3f6d400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3f680b3350 0x7f3f6815d0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55004/0 (socket says 172.21.15.6:55004) 2024-03-04T09:31:11.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.462+0000 7f3f6d400640 1 -- 172.21.15.6:0/1644118338 learned_addr learned my addr 172.21.15.6:0/1644118338 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:31:11.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.462+0000 7f3f6e800640 1 --2- 172.21.15.6:0/1644118338 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3f680b49e0 0x7f3f6806c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:11.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.462+0000 7f3f6de00640 1 --2- 172.21.15.6:0/1644118338 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3f680b2f70 0x7f3f6814d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:11.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.462+0000 7f3f6d400640 1 -- 172.21.15.6:0/1644118338 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3f680b2f70 msgr2=0x7f3f6814d310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:11.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.462+0000 7f3f6d400640 1 --2- 172.21.15.6:0/1644118338 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3f680b2f70 0x7f3f6814d310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:11.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.462+0000 7f3f6d400640 1 -- 172.21.15.6:0/1644118338 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3f680b49e0 msgr2=0x7f3f6806c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:11.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.462+0000 7f3f6d400640 1 --2- 172.21.15.6:0/1644118338 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3f680b49e0 0x7f3f6806c4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:11.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.462+0000 7f3f6d400640 1 -- 172.21.15.6:0/1644118338 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f6814d850 con 0x7f3f680b3350 2024-03-04T09:31:11.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.462+0000 7f3f6e800640 1 --2- 172.21.15.6:0/1644118338 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3f680b49e0 0x7f3f6806c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:31:11.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.462+0000 7f3f6de00640 1 --2- 172.21.15.6:0/1644118338 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3f680b2f70 0x7f3f6814d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:31:11.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.463+0000 7f3f6d400640 1 --2- 172.21.15.6:0/1644118338 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3f680b3350 0x7f3f6815d0b0 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7f3f5802b9c0 tx=0x7f3f58004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:11.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.463+0000 7f3f62000640 1 -- 172.21.15.6:0/1644118338 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f58048070 con 0x7f3f680b3350 2024-03-04T09:31:11.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.463+0000 7f3f62000640 1 -- 172.21.15.6:0/1644118338 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f58030640 con 0x7f3f680b3350 2024-03-04T09:31:11.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.463+0000 7f3f62000640 1 -- 172.21.15.6:0/1644118338 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f58034040 con 0x7f3f680b3350 2024-03-04T09:31:11.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.463+0000 7f3f70800640 1 -- 172.21.15.6:0/1644118338 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f680cd6e0 con 0x7f3f680b3350 2024-03-04T09:31:11.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.463+0000 7f3f70800640 1 -- 172.21.15.6:0/1644118338 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f680cd400 con 0x7f3f680b3350 2024-03-04T09:31:11.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.465+0000 7f3f70800640 1 -- 172.21.15.6:0/1644118338 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f6804f330 con 0x7f3f680b3350 2024-03-04T09:31:11.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.465+0000 7f3f62000640 1 -- 172.21.15.6:0/1644118338 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3f5804b070 con 0x7f3f680b3350 2024-03-04T09:31:11.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.466+0000 7f3f62000640 1 --2- 172.21.15.6:0/1644118338 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f3f4c06b4d0 0x7f3f4c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:11.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.466+0000 7f3f62000640 1 -- 172.21.15.6:0/1644118338 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5115+0+0 (secure 0 0 0) 0x7f3f580abfb0 con 0x7f3f680b3350 2024-03-04T09:31:11.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.467+0000 7f3f6de00640 1 --2- 172.21.15.6:0/1644118338 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f3f4c06b4d0 0x7f3f4c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:11.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.468+0000 7f3f6de00640 1 --2- 172.21.15.6:0/1644118338 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f3f4c06b4d0 0x7f3f4c06d990 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f3f5c000f60 tx=0x7f3f5c009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:11.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.471+0000 7f3f62000640 1 -- 172.21.15.6:0/1644118338 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3f5807b070 con 0x7f3f680b3350 2024-03-04T09:31:11.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.670+0000 7f3f70800640 1 -- 172.21.15.6:0/1644118338 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f3f680d22a0 con 0x7f3f680b3350 2024-03-04T09:31:11.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.676+0000 7f3f62000640 1 -- 172.21.15.6:0/1644118338 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v29) v1 ==== 104+0+0 (secure 0 0 0) 0x7f3f58030810 con 0x7f3f680b3350 2024-03-04T09:31:11.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.678+0000 7f3f70800640 1 -- 172.21.15.6:0/1644118338 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f3f4c06b4d0 msgr2=0x7f3f4c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:11.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.678+0000 7f3f70800640 1 --2- 172.21.15.6:0/1644118338 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f3f4c06b4d0 0x7f3f4c06d990 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f3f5c000f60 tx=0x7f3f5c009040 comp rx=0 tx=0).stop 2024-03-04T09:31:11.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.678+0000 7f3f70800640 1 -- 172.21.15.6:0/1644118338 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3f680b3350 msgr2=0x7f3f6815d0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:11.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.678+0000 7f3f70800640 1 --2- 172.21.15.6:0/1644118338 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3f680b3350 0x7f3f6815d0b0 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7f3f5802b9c0 tx=0x7f3f58004730 comp rx=0 tx=0).stop 2024-03-04T09:31:11.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.679+0000 7f3f70800640 1 -- 172.21.15.6:0/1644118338 shutdown_connections 2024-03-04T09:31:11.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.679+0000 7f3f70800640 1 --2- 172.21.15.6:0/1644118338 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3f680b49e0 0x7f3f6806c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:11.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.679+0000 7f3f70800640 1 --2- 172.21.15.6:0/1644118338 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3f680b3350 0x7f3f6815d0b0 unknown :-1 s=CLOSED pgs=1203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:11.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.679+0000 7f3f70800640 1 --2- 172.21.15.6:0/1644118338 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f3f4c06b4d0 0x7f3f4c06d990 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:11.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.679+0000 7f3f70800640 1 --2- 172.21.15.6:0/1644118338 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3f680b2f70 0x7f3f6814d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:11.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.679+0000 7f3f70800640 1 -- 172.21.15.6:0/1644118338 >> 172.21.15.6:0/1644118338 conn(0x7f3f680d3ee0 msgr2=0x7f3f6806bb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:11.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.679+0000 7f3f70800640 1 -- 172.21.15.6:0/1644118338 shutdown_connections 2024-03-04T09:31:11.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.679+0000 7f3f70800640 1 -- 172.21.15.6:0/1644118338 wait complete. 2024-03-04T09:31:11.695 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_files_throttle' 2024-03-04T09:31:11.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.923+0000 7fca71600640 1 Processor -- start 2024-03-04T09:31:11.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.923+0000 7fca71600640 1 -- start start 2024-03-04T09:31:11.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.924+0000 7fca71600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fca6c145300 0x7fca6c0b49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:11.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.924+0000 7fca71600640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fca6c070f10 0x7fca6c0cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:11.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.924+0000 7fca71600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fca6c0cce90 0x7fca6c0cd930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:11.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.924+0000 7fca71600640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca6c0cd300 con 0x7fca6c0cce90 2024-03-04T09:31:11.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.924+0000 7fca71600640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca6c0714d0 con 0x7fca6c070f10 2024-03-04T09:31:11.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.924+0000 7fca71600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca6c0717b0 con 0x7fca6c145300 2024-03-04T09:31:11.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.925+0000 7fca70c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fca6c0cce90 0x7fca6c0cd930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:11.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.925+0000 7fca70c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fca6c0cce90 0x7fca6c0cd930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55016/0 (socket says 172.21.15.6:55016) 2024-03-04T09:31:11.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.925+0000 7fca70c00640 1 -- 172.21.15.6:0/1836824245 learned_addr learned my addr 172.21.15.6:0/1836824245 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:31:11.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.925+0000 7fca6b400640 1 --2- 172.21.15.6:0/1836824245 >> v2:172.21.15.146:3300/0 conn(0x7fca6c070f10 0x7fca6c0cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:11.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.925+0000 7fca6be00640 1 --2- 172.21.15.6:0/1836824245 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fca6c145300 0x7fca6c0b49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:11.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.925+0000 7fca70c00640 1 -- 172.21.15.6:0/1836824245 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fca6c145300 msgr2=0x7fca6c0b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:11.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.925+0000 7fca70c00640 1 --2- 172.21.15.6:0/1836824245 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fca6c145300 0x7fca6c0b49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:11.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.925+0000 7fca70c00640 1 -- 172.21.15.6:0/1836824245 >> v2:172.21.15.146:3300/0 conn(0x7fca6c070f10 msgr2=0x7fca6c0cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:11.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.925+0000 7fca70c00640 1 --2- 172.21.15.6:0/1836824245 >> v2:172.21.15.146:3300/0 conn(0x7fca6c070f10 0x7fca6c0cc950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:11.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.925+0000 7fca70c00640 1 -- 172.21.15.6:0/1836824245 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca6c071a90 con 0x7fca6c0cce90 2024-03-04T09:31:11.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.927+0000 7fca70c00640 1 --2- 172.21.15.6:0/1836824245 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fca6c0cce90 0x7fca6c0cd930 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7fca6c0691a0 tx=0x7fca6402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=926e3bb1bc6bb933 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:11.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.927+0000 7fca6aa00640 1 -- 172.21.15.6:0/1836824245 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca6402ca70 con 0x7fca6c0cce90 2024-03-04T09:31:11.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.927+0000 7fca6aa00640 1 -- 172.21.15.6:0/1836824245 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca6402cc10 con 0x7fca6c0cce90 2024-03-04T09:31:11.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.927+0000 7fca6aa00640 1 -- 172.21.15.6:0/1836824245 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca640343f0 con 0x7fca6c0cce90 2024-03-04T09:31:11.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.927+0000 7fca71600640 1 -- 172.21.15.6:0/1836824245 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fca6c0cce90 msgr2=0x7fca6c0cd930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:11.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.927+0000 7fca71600640 1 --2- 172.21.15.6:0/1836824245 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fca6c0cce90 0x7fca6c0cd930 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7fca6c0691a0 tx=0x7fca6402bed0 comp rx=0 tx=0).stop 2024-03-04T09:31:11.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.928+0000 7fca71600640 1 -- 172.21.15.6:0/1836824245 shutdown_connections 2024-03-04T09:31:11.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.928+0000 7fca71600640 1 --2- 172.21.15.6:0/1836824245 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fca6c0cce90 0x7fca6c0cd930 unknown :-1 s=CLOSED pgs=1204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:11.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.928+0000 7fca71600640 1 --2- 172.21.15.6:0/1836824245 >> v2:172.21.15.146:3300/0 conn(0x7fca6c070f10 0x7fca6c0cc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:11.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.928+0000 7fca71600640 1 --2- 172.21.15.6:0/1836824245 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fca6c145300 0x7fca6c0b49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:11.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.928+0000 7fca71600640 1 -- 172.21.15.6:0/1836824245 >> 172.21.15.6:0/1836824245 conn(0x7fca6c06b3d0 msgr2=0x7fca6c06b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:11.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.928+0000 7fca71600640 1 -- 172.21.15.6:0/1836824245 shutdown_connections 2024-03-04T09:31:11.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.928+0000 7fca71600640 1 -- 172.21.15.6:0/1836824245 wait complete. 2024-03-04T09:31:11.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.929+0000 7fca71600640 1 Processor -- start 2024-03-04T09:31:11.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.929+0000 7fca71600640 1 -- start start 2024-03-04T09:31:11.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.930+0000 7fca71600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fca6c070f10 0x7fca6c0a9470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:11.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.930+0000 7fca71600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fca6c0cce90 0x7fca6c15b530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:11.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.930+0000 7fca71600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fca6c145300 0x7fca6c13f460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:11.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.930+0000 7fca71600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca6c1456e0 con 0x7fca6c145300 2024-03-04T09:31:11.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.930+0000 7fca71600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca6c0b4ee0 con 0x7fca6c070f10 2024-03-04T09:31:11.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.930+0000 7fca71600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca6c0cde70 con 0x7fca6c0cce90 2024-03-04T09:31:11.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.931+0000 7fca70c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fca6c145300 0x7fca6c13f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:11.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.931+0000 7fca70c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fca6c145300 0x7fca6c13f460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55028/0 (socket says 172.21.15.6:55028) 2024-03-04T09:31:11.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.931+0000 7fca70c00640 1 -- 172.21.15.6:0/4134425111 learned_addr learned my addr 172.21.15.6:0/4134425111 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:31:11.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.931+0000 7fca70c00640 1 -- 172.21.15.6:0/4134425111 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fca6c0cce90 msgr2=0x7fca6c15b530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:11.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.931+0000 7fca70c00640 1 --2- 172.21.15.6:0/4134425111 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fca6c0cce90 0x7fca6c15b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:11.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.931+0000 7fca70c00640 1 -- 172.21.15.6:0/4134425111 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fca6c070f10 msgr2=0x7fca6c0a9470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:11.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.931+0000 7fca6be00640 1 --2- 172.21.15.6:0/4134425111 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fca6c070f10 0x7fca6c0a9470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:11.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.931+0000 7fca70c00640 1 --2- 172.21.15.6:0/4134425111 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fca6c070f10 0x7fca6c0a9470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:11.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.931+0000 7fca70c00640 1 -- 172.21.15.6:0/4134425111 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca6c0a9b70 con 0x7fca6c145300 2024-03-04T09:31:11.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.932+0000 7fca70c00640 1 --2- 172.21.15.6:0/4134425111 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fca6c145300 0x7fca6c13f460 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7fca64036040 tx=0x7fca64004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:11.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.932+0000 7fca63e00640 1 -- 172.21.15.6:0/4134425111 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca64043020 con 0x7fca6c145300 2024-03-04T09:31:11.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.932+0000 7fca63e00640 1 -- 172.21.15.6:0/4134425111 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca6403e450 con 0x7fca6c145300 2024-03-04T09:31:11.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.932+0000 7fca63e00640 1 -- 172.21.15.6:0/4134425111 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca64009510 con 0x7fca6c145300 2024-03-04T09:31:11.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.932+0000 7fca71600640 1 -- 172.21.15.6:0/4134425111 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca6c0a9e50 con 0x7fca6c145300 2024-03-04T09:31:11.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.932+0000 7fca71600640 1 -- 172.21.15.6:0/4134425111 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca6c0aa410 con 0x7fca6c145300 2024-03-04T09:31:11.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.933+0000 7fca71600640 1 -- 172.21.15.6:0/4134425111 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca6c04f330 con 0x7fca6c145300 2024-03-04T09:31:11.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.934+0000 7fca63e00640 1 -- 172.21.15.6:0/4134425111 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fca64046020 con 0x7fca6c145300 2024-03-04T09:31:11.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.936+0000 7fca63e00640 1 --2- 172.21.15.6:0/4134425111 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fca4c06b4d0 0x7fca4c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:11.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.936+0000 7fca63e00640 1 -- 172.21.15.6:0/4134425111 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5115+0+0 (secure 0 0 0) 0x7fca640acbd0 con 0x7fca6c145300 2024-03-04T09:31:11.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.936+0000 7fca6be00640 1 --2- 172.21.15.6:0/4134425111 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fca4c06b4d0 0x7fca4c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:11.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.937+0000 7fca6be00640 1 --2- 172.21.15.6:0/4134425111 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fca4c06b4d0 0x7fca4c06d990 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fca5c006fd0 tx=0x7fca5c008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:11.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:11.940+0000 7fca63e00640 1 -- 172.21.15.6:0/4134425111 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fca6404d020 con 0x7fca6c145300 2024-03-04T09:31:12.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:12.210+0000 7fca71600640 1 -- 172.21.15.6:0/4134425111 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_files_throttle"]} v 0) v1 -- 0x7fca6c1446c0 con 0x7fca6c145300 2024-03-04T09:31:12.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:12.547+0000 7fca63e00640 1 -- 172.21.15.6:0/4134425111 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_files_throttle"]}]=0 v338) v1 ==== 134+0+0 (secure 0 0 0) 0x7fca6407bc00 con 0x7fca6c145300 2024-03-04T09:31:12.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:12.550+0000 7fca71600640 1 -- 172.21.15.6:0/4134425111 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fca4c06b4d0 msgr2=0x7fca4c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:12.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:12.550+0000 7fca71600640 1 --2- 172.21.15.6:0/4134425111 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fca4c06b4d0 0x7fca4c06d990 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fca5c006fd0 tx=0x7fca5c008040 comp rx=0 tx=0).stop 2024-03-04T09:31:12.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:12.550+0000 7fca71600640 1 -- 172.21.15.6:0/4134425111 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fca6c145300 msgr2=0x7fca6c13f460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:12.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:12.550+0000 7fca71600640 1 --2- 172.21.15.6:0/4134425111 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fca6c145300 0x7fca6c13f460 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7fca64036040 tx=0x7fca64004a40 comp rx=0 tx=0).stop 2024-03-04T09:31:12.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:12.551+0000 7fca71600640 1 -- 172.21.15.6:0/4134425111 shutdown_connections 2024-03-04T09:31:12.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:12.551+0000 7fca71600640 1 --2- 172.21.15.6:0/4134425111 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fca6c145300 0x7fca6c13f460 unknown :-1 s=CLOSED pgs=1205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:12.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:12.551+0000 7fca71600640 1 --2- 172.21.15.6:0/4134425111 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fca4c06b4d0 0x7fca4c06d990 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:12.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:12.551+0000 7fca71600640 1 --2- 172.21.15.6:0/4134425111 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fca6c0cce90 0x7fca6c15b530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:12.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:12.551+0000 7fca71600640 1 --2- 172.21.15.6:0/4134425111 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fca6c070f10 0x7fca6c0a9470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:12.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:12.551+0000 7fca71600640 1 -- 172.21.15.6:0/4134425111 >> 172.21.15.6:0/4134425111 conn(0x7fca6c06b3d0 msgr2=0x7fca6c0b0910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:12.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:12.551+0000 7fca71600640 1 -- 172.21.15.6:0/4134425111 shutdown_connections 2024-03-04T09:31:12.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:12.551+0000 7fca71600640 1 -- 172.21.15.6:0/4134425111 wait complete. 2024-03-04T09:31:12.567 INFO:tasks.cephfs_test_runner:test_files_throttle (tasks.cephfs.test_strays.TestStrays) ... ok 2024-03-04T09:31:12.567 INFO:tasks.cephfs_test_runner:Starting test: test_hardlink_reintegration (tasks.cephfs.test_strays.TestStrays) That removal of primary dentry of hardlinked inode results 2024-03-04T09:31:12.567 INFO:tasks.cephfs_test_runner:test_hardlink_reintegration (tasks.cephfs.test_strays.TestStrays) 2024-03-04T09:31:12.568 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T09:31:12.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:12.792+0000 7fb769400640 1 Processor -- start 2024-03-04T09:31:12.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:12.792+0000 7fb769400640 1 -- start start 2024-03-04T09:31:12.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:12.794+0000 7fb769400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fb764070f10 0x7fb7640cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:12.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:12.794+0000 7fb769400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb7640cce90 0x7fb76415c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:12.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:12.794+0000 7fb769400640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7640714d0 con 0x7fb764145300 2024-03-04T09:31:12.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:12.794+0000 7fb769400640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7640cd300 con 0x7fb764070f10 2024-03-04T09:31:12.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:12.794+0000 7fb769400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7640717b0 con 0x7fb7640cce90 2024-03-04T09:31:12.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:12.794+0000 7fb75be00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fb764070f10 0x7fb7640cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:12.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:12.795+0000 7fb768a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb7640cce90 0x7fb76415c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:12.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:12.795+0000 7fb75be00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fb764070f10 0x7fb7640cc950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.6:60882/0 (socket says 172.21.15.6:60882) 2024-03-04T09:31:12.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:12.795+0000 7fb75be00640 1 -- 172.21.15.6:0/3744847904 learned_addr learned my addr 172.21.15.6:0/3744847904 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:31:12.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:12.795+0000 7fb768a00640 1 -- 172.21.15.6:0/3744847904 >> v2:172.21.15.146:3300/0 conn(0x7fb764070f10 msgr2=0x7fb7640cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:12.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:12.795+0000 7fb768a00640 1 --2- 172.21.15.6:0/3744847904 >> v2:172.21.15.146:3300/0 conn(0x7fb764070f10 0x7fb7640cc950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:12.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:12.795+0000 7fb768a00640 1 -- 172.21.15.6:0/3744847904 >> v1:172.21.15.146:6789/0 conn(0x7fb764145300 legacy=0x7fb7640b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:12.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:12.795+0000 7fb768a00640 1 -- 172.21.15.6:0/3744847904 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb764071a90 con 0x7fb7640cce90 2024-03-04T09:31:12.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:12.797+0000 7fb768a00640 1 --2- 172.21.15.6:0/3744847904 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb7640cce90 0x7fb76415c570 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fb75c009d90 tx=0x7fb75c02be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b8fbe94061303915 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:12.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:12.798+0000 7fb763400640 1 -- 172.21.15.6:0/3744847904 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb75c039070 con 0x7fb7640cce90 2024-03-04T09:31:12.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:12.798+0000 7fb763400640 1 -- 172.21.15.6:0/3744847904 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb75c02cbc0 con 0x7fb7640cce90 2024-03-04T09:31:12.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:12.798+0000 7fb763400640 1 -- 172.21.15.6:0/3744847904 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb75c034440 con 0x7fb7640cce90 2024-03-04T09:31:12.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:12.799+0000 7fb769400640 1 -- 172.21.15.6:0/3744847904 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb7640cce90 msgr2=0x7fb76415c570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:12.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:12.799+0000 7fb769400640 1 --2- 172.21.15.6:0/3744847904 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb7640cce90 0x7fb76415c570 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fb75c009d90 tx=0x7fb75c02be10 comp rx=0 tx=0).stop 2024-03-04T09:31:12.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:12.799+0000 7fb769400640 1 -- 172.21.15.6:0/3744847904 shutdown_connections 2024-03-04T09:31:12.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:12.799+0000 7fb769400640 1 --2- 172.21.15.6:0/3744847904 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb7640cce90 0x7fb76415c570 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:12.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:12.799+0000 7fb769400640 1 --2- 172.21.15.6:0/3744847904 >> v2:172.21.15.146:3300/0 conn(0x7fb764070f10 0x7fb7640cc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:12.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:12.799+0000 7fb769400640 1 -- 172.21.15.6:0/3744847904 >> 172.21.15.6:0/3744847904 conn(0x7fb76406b3d0 msgr2=0x7fb76406b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:12.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:12.799+0000 7fb769400640 1 -- 172.21.15.6:0/3744847904 shutdown_connections 2024-03-04T09:31:12.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:12.799+0000 7fb769400640 1 -- 172.21.15.6:0/3744847904 wait complete. 2024-03-04T09:31:12.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:12.801+0000 7fb769400640 1 Processor -- start 2024-03-04T09:31:12.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:12.802+0000 7fb769400640 1 -- start start 2024-03-04T09:31:12.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:12.803+0000 7fb769400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb764070f10 0x7fb764131fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:12.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:12.803+0000 7fb769400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb7640cce90 0x7fb7640ae220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:12.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:12.803+0000 7fb769400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb764145300 0x7fb7640ae760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:12.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:12.803+0000 7fb769400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb76415cab0 con 0x7fb764145300 2024-03-04T09:31:12.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:12.803+0000 7fb769400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7641456e0 con 0x7fb764070f10 2024-03-04T09:31:12.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:12.803+0000 7fb769400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb76415cc30 con 0x7fb7640cce90 2024-03-04T09:31:12.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:12.803+0000 7fb768a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb764145300 0x7fb7640ae760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:12.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:12.803+0000 7fb768a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb764145300 0x7fb7640ae760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55044/0 (socket says 172.21.15.6:55044) 2024-03-04T09:31:12.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:12.803+0000 7fb768a00640 1 -- 172.21.15.6:0/61143775 learned_addr learned my addr 172.21.15.6:0/61143775 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:31:12.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:12.803+0000 7fb75be00640 1 --2- 172.21.15.6:0/61143775 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb7640cce90 0x7fb7640ae220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:12.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:12.803+0000 7fb763e00640 1 --2- 172.21.15.6:0/61143775 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb764070f10 0x7fb764131fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:12.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:12.804+0000 7fb768a00640 1 -- 172.21.15.6:0/61143775 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb7640cce90 msgr2=0x7fb7640ae220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:12.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:12.804+0000 7fb768a00640 1 --2- 172.21.15.6:0/61143775 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb7640cce90 0x7fb7640ae220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:12.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:12.804+0000 7fb768a00640 1 -- 172.21.15.6:0/61143775 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb764070f10 msgr2=0x7fb764131fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:12.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:12.804+0000 7fb768a00640 1 --2- 172.21.15.6:0/61143775 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb764070f10 0x7fb764131fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:12.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:12.804+0000 7fb768a00640 1 -- 172.21.15.6:0/61143775 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7640aa440 con 0x7fb764145300 2024-03-04T09:31:12.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:12.804+0000 7fb768a00640 1 --2- 172.21.15.6:0/61143775 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb764145300 0x7fb7640ae760 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7fb75c009eb0 tx=0x7fb75c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:12.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:12.804+0000 7fb760c00640 1 -- 172.21.15.6:0/61143775 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb75c044020 con 0x7fb764145300 2024-03-04T09:31:12.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:12.804+0000 7fb760c00640 1 -- 172.21.15.6:0/61143775 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb75c03f470 con 0x7fb764145300 2024-03-04T09:31:12.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:12.804+0000 7fb769400640 1 -- 172.21.15.6:0/61143775 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7640aa720 con 0x7fb764145300 2024-03-04T09:31:12.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:12.804+0000 7fb760c00640 1 -- 172.21.15.6:0/61143775 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb75c039040 con 0x7fb764145300 2024-03-04T09:31:12.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:12.805+0000 7fb769400640 1 -- 172.21.15.6:0/61143775 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7640aace0 con 0x7fb764145300 2024-03-04T09:31:12.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:12.806+0000 7fb769400640 1 -- 172.21.15.6:0/61143775 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb764067830 con 0x7fb764145300 2024-03-04T09:31:12.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:12.807+0000 7fb760c00640 1 -- 172.21.15.6:0/61143775 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb75c047020 con 0x7fb764145300 2024-03-04T09:31:12.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:12.808+0000 7fb760c00640 1 --2- 172.21.15.6:0/61143775 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb74006b500 0x7fb74006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:12.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:12.808+0000 7fb760c00640 1 -- 172.21.15.6:0/61143775 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5115+0+0 (secure 0 0 0) 0x7fb75c0ac9e0 con 0x7fb764145300 2024-03-04T09:31:12.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:12.809+0000 7fb763e00640 1 --2- 172.21.15.6:0/61143775 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb74006b500 0x7fb74006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:12.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:12.810+0000 7fb763e00640 1 --2- 172.21.15.6:0/61143775 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb74006b500 0x7fb74006d9c0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fb754007fd0 tx=0x7fb754009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:12.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:12.813+0000 7fb760c00640 1 -- 172.21.15.6:0/61143775 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb764067830 con 0x7fb764145300 2024-03-04T09:31:13.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.015+0000 7fb769400640 1 -- 172.21.15.6:0/61143775 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb76404f330 con 0x7fb764145300 2024-03-04T09:31:13.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.017+0000 7fb760c00640 1 -- 172.21.15.6:0/61143775 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v101) v1 ==== 74+0+13230 (secure 0 0 0) 0x7fb75c04e030 con 0x7fb764145300 2024-03-04T09:31:13.015 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:31:13.015 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":101,"fsid":"edce0578-5ae4-4402-b45a-552bdef84d7c","created":"2024-03-04T09:24:54.523780+0000","modified":"2024-03-04T09:31:10.523066+0000","last_up_change":"2024-03-04T09:25:07.089049+0000","last_in_change":"2024-03-04T09:24:57.982077+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T09:25:07.935198+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a96574f-81f1-4bc6-8463-757520b84a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6825","nonce":1364449401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6827","nonce":1364449401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6831","nonce":1364449401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6829","nonce":1364449401}]},"public_addr":"172.21.15.6:6825/1364449401","cluster_addr":"172.21.15.6:6827/1364449401","heartbeat_back_addr":"172.21.15.6:6831/1364449401","heartbeat_front_addr":"172.21.15.6:6829/1364449401","state":["exists","up"]},{"osd":1,"uuid":"7d79b9c1-6b87-476a-bc36-7db5ee64cd90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6809","nonce":2494103078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6811","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6813","nonce":2494103078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6821","nonce":2494103078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6817","nonce":2494103078}]},"public_addr":"172.21.15.6:6809/2494103078","cluster_addr":"172.21.15.6:6813/2494103078","heartbeat_back_addr":"172.21.15.6:6821/2494103078","heartbeat_front_addr":"172.21.15.6:6817/2494103078","state":["exists","up"]},{"osd":2,"uuid":"492331b9-4ee9-40b6-bacd-85d44cd6c45a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6812","nonce":1500066681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6816","nonce":1500066681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6824","nonce":1500066681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6820","nonce":1500066681}]},"public_addr":"172.21.15.6:6812/1500066681","cluster_addr":"172.21.15.6:6816/1500066681","heartbeat_back_addr":"172.21.15.6:6824/1500066681","heartbeat_front_addr":"172.21.15.6:6820/1500066681","state":["exists","up"]},{"osd":3,"uuid":"21bb2efe-9ba7-4bdc-bc31-180cd1102e21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6801","nonce":1558806123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6803","nonce":1558806123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6807","nonce":1558806123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6805","nonce":1558806123}]},"public_addr":"172.21.15.6:6801/1558806123","cluster_addr":"172.21.15.6:6803/1558806123","heartbeat_back_addr":"172.21.15.6:6807/1558806123","heartbeat_front_addr":"172.21.15.6:6805/1558806123","state":["exists","up"]},{"osd":4,"uuid":"6e2c4b71-19d9-4737-83d9-558705d8644a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6801","nonce":1526135780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6803","nonce":1526135780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6807","nonce":1526135780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6805","nonce":1526135780}]},"public_addr":"172.21.15.146:6801/1526135780","cluster_addr":"172.21.15.146:6803/1526135780","heartbeat_back_addr":"172.21.15.146:6807/1526135780","heartbeat_front_addr":"172.21.15.146:6805/1526135780","state":["exists","up"]},{"osd":5,"uuid":"fb69c525-f1be-4b7d-ab32-6c29b0d6fddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6809","nonce":460262507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6811","nonce":460262507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6815","nonce":460262507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6813","nonce":460262507}]},"public_addr":"172.21.15.146:6809/460262507","cluster_addr":"172.21.15.146:6811/460262507","heartbeat_back_addr":"172.21.15.146:6815/460262507","heartbeat_front_addr":"172.21.15.146:6813/460262507","state":["exists","up"]},{"osd":6,"uuid":"c32e7fed-370d-4401-8ce0-6a4335b0d168","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6823","nonce":1392897493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6827","nonce":1392897493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6831","nonce":1392897493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6829","nonce":1392897493}]},"public_addr":"172.21.15.146:6823/1392897493","cluster_addr":"172.21.15.146:6827/1392897493","heartbeat_back_addr":"172.21.15.146:6831/1392897493","heartbeat_front_addr":"172.21.15.146:6829/1392897493","state":["exists","up"]},{"osd":7,"uuid":"75a79b44-fa5f-426d-a7d3-71f21f78835c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6817","nonce":3268323156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6819","nonce":3268323156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6825","nonce":3268323156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6822","nonce":3268323156}]},"public_addr":"172.21.15.146:6817/3268323156","cluster_addr":"172.21.15.146:6819/3268323156","heartbeat_back_addr":"172.21.15.146:6825/3268323156","heartbeat_front_addr":"172.21.15.146:6822/3268323156","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.310437+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.295850+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.294674+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.526855+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.300023+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.595160+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.346046+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.558741+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.6:6834/1732383836":"2024-03-05T09:31:07.045120+0000","172.21.15.146:6835/847815316":"2024-03-05T09:30:33.260598+0000","172.21.15.146:6834/847815316":"2024-03-05T09:30:33.260598+0000","172.21.15.6:6835/1732383836":"2024-03-05T09:31:07.045120+0000","172.21.15.6:6835/2836916196":"2024-03-05T09:30:32.267572+0000","172.21.15.6:6832/2836916196":"2024-03-05T09:30:32.267572+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T09:31:13.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.020+0000 7fb769400640 1 -- 172.21.15.6:0/61143775 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb74006b500 msgr2=0x7fb74006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:13.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.020+0000 7fb769400640 1 --2- 172.21.15.6:0/61143775 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb74006b500 0x7fb74006d9c0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fb754007fd0 tx=0x7fb754009040 comp rx=0 tx=0).stop 2024-03-04T09:31:13.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.020+0000 7fb769400640 1 -- 172.21.15.6:0/61143775 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb764145300 msgr2=0x7fb7640ae760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:13.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.020+0000 7fb769400640 1 --2- 172.21.15.6:0/61143775 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb764145300 0x7fb7640ae760 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7fb75c009eb0 tx=0x7fb75c004a40 comp rx=0 tx=0).stop 2024-03-04T09:31:13.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.020+0000 7fb769400640 1 -- 172.21.15.6:0/61143775 shutdown_connections 2024-03-04T09:31:13.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.020+0000 7fb769400640 1 --2- 172.21.15.6:0/61143775 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb764145300 0x7fb7640ae760 unknown :-1 s=CLOSED pgs=1206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:13.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.020+0000 7fb769400640 1 --2- 172.21.15.6:0/61143775 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb74006b500 0x7fb74006d9c0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:13.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.020+0000 7fb769400640 1 --2- 172.21.15.6:0/61143775 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb7640cce90 0x7fb7640ae220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:13.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.020+0000 7fb769400640 1 --2- 172.21.15.6:0/61143775 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb764070f10 0x7fb764131fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:13.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.020+0000 7fb769400640 1 -- 172.21.15.6:0/61143775 >> 172.21.15.6:0/61143775 conn(0x7fb76406b3d0 msgr2=0x7fb7640af750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:13.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.021+0000 7fb769400640 1 -- 172.21.15.6:0/61143775 shutdown_connections 2024-03-04T09:31:13.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.021+0000 7fb769400640 1 -- 172.21.15.6:0/61143775 wait complete. 2024-03-04T09:31:13.037 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T09:25:07.935198+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-04T09:31:13.037 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-04T09:31:13.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.265+0000 7ffacb200640 1 Processor -- start 2024-03-04T09:31:13.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.265+0000 7ffacb200640 1 -- start start 2024-03-04T09:31:13.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.266+0000 7ffacb200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ffac4165d50 0x7ffac40b4a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:13.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.266+0000 7ffacb200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ffac40b1b40 0x7ffac40b1fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:13.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.266+0000 7ffacb200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffac4166cb0 con 0x7ffac4162280 2024-03-04T09:31:13.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.266+0000 7ffacb200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffac40b24f0 con 0x7ffac40b1b40 2024-03-04T09:31:13.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.266+0000 7ffacb200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffac40b4fd0 con 0x7ffac4165d50 2024-03-04T09:31:13.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.267+0000 7ffaca800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ffac40b1b40 0x7ffac40b1fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:13.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.267+0000 7ffaca800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ffac40b1b40 0x7ffac40b1fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55056/0 (socket says 172.21.15.6:55056) 2024-03-04T09:31:13.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.267+0000 7ffaca800640 1 -- 172.21.15.6:0/2408110029 learned_addr learned my addr 172.21.15.6:0/2408110029 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:31:13.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.267+0000 7ffac9400640 1 --2- 172.21.15.6:0/2408110029 >> v2:172.21.15.146:3300/0 conn(0x7ffac4165d50 0x7ffac40b4a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:13.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.267+0000 7ffaca800640 1 -- 172.21.15.6:0/2408110029 >> v2:172.21.15.146:3300/0 conn(0x7ffac4165d50 msgr2=0x7ffac40b4a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:13.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.267+0000 7ffaca800640 1 --2- 172.21.15.6:0/2408110029 >> v2:172.21.15.146:3300/0 conn(0x7ffac4165d50 0x7ffac40b4a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:13.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.267+0000 7ffaca800640 1 -- 172.21.15.6:0/2408110029 >> v1:172.21.15.146:6789/0 conn(0x7ffac4162280 legacy=0x7ffac40cc430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:13.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.267+0000 7ffaca800640 1 -- 172.21.15.6:0/2408110029 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffac4166410 con 0x7ffac40b1b40 2024-03-04T09:31:13.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.268+0000 7ffaca800640 1 --2- 172.21.15.6:0/2408110029 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffac40b1b40 0x7ffac40b1fb0 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7ffac4072f30 tx=0x7ffac002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eca956cb9060fe37 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:13.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.269+0000 7ffac8a00640 1 -- 172.21.15.6:0/2408110029 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffac002ca80 con 0x7ffac40b1b40 2024-03-04T09:31:13.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.269+0000 7ffac8a00640 1 -- 172.21.15.6:0/2408110029 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffac002cc20 con 0x7ffac40b1b40 2024-03-04T09:31:13.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.269+0000 7ffac8a00640 1 -- 172.21.15.6:0/2408110029 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffac00343f0 con 0x7ffac40b1b40 2024-03-04T09:31:13.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.269+0000 7ffacb200640 1 -- 172.21.15.6:0/2408110029 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffac40b1b40 msgr2=0x7ffac40b1fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:13.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.269+0000 7ffacb200640 1 --2- 172.21.15.6:0/2408110029 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffac40b1b40 0x7ffac40b1fb0 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7ffac4072f30 tx=0x7ffac002bee0 comp rx=0 tx=0).stop 2024-03-04T09:31:13.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.269+0000 7ffacb200640 1 -- 172.21.15.6:0/2408110029 shutdown_connections 2024-03-04T09:31:13.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.269+0000 7ffacb200640 1 --2- 172.21.15.6:0/2408110029 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffac40b1b40 0x7ffac40b1fb0 unknown :-1 s=CLOSED pgs=1207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:13.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.269+0000 7ffacb200640 1 --2- 172.21.15.6:0/2408110029 >> v2:172.21.15.146:3300/0 conn(0x7ffac4165d50 0x7ffac40b4a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:13.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.269+0000 7ffacb200640 1 -- 172.21.15.6:0/2408110029 >> 172.21.15.6:0/2408110029 conn(0x7ffac406b3a0 msgr2=0x7ffac406b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:13.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.270+0000 7ffacb200640 1 -- 172.21.15.6:0/2408110029 shutdown_connections 2024-03-04T09:31:13.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.270+0000 7ffacb200640 1 -- 172.21.15.6:0/2408110029 wait complete. 2024-03-04T09:31:13.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.271+0000 7ffacb200640 1 Processor -- start 2024-03-04T09:31:13.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.271+0000 7ffacb200640 1 -- start start 2024-03-04T09:31:13.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.272+0000 7ffacb200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffac40b1b40 0x7ffac4161750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:13.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.272+0000 7ffacb200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffac4162280 0x7ffac41523a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:13.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.272+0000 7ffacb200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ffac4165d50 0x7ffac415f510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:13.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.272+0000 7ffacb200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffac41669d0 con 0x7ffac4162280 2024-03-04T09:31:13.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.272+0000 7ffacb200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffac41661b0 con 0x7ffac40b1b40 2024-03-04T09:31:13.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.272+0000 7ffacb200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffac4071560 con 0x7ffac4165d50 2024-03-04T09:31:13.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.272+0000 7ffac9400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffac4162280 0x7ffac41523a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:13.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.272+0000 7ffac9400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffac4162280 0x7ffac41523a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55064/0 (socket says 172.21.15.6:55064) 2024-03-04T09:31:13.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.272+0000 7ffac9400640 1 -- 172.21.15.6:0/585684950 learned_addr learned my addr 172.21.15.6:0/585684950 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:31:13.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.273+0000 7ffac9e00640 1 --2- 172.21.15.6:0/585684950 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffac40b1b40 0x7ffac4161750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:13.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.273+0000 7ffac9400640 1 -- 172.21.15.6:0/585684950 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ffac4165d50 msgr2=0x7ffac415f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:13.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.273+0000 7ffac9400640 1 --2- 172.21.15.6:0/585684950 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ffac4165d50 0x7ffac415f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:13.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.273+0000 7ffac9400640 1 -- 172.21.15.6:0/585684950 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffac40b1b40 msgr2=0x7ffac4161750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:13.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.273+0000 7ffac9400640 1 --2- 172.21.15.6:0/585684950 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffac40b1b40 0x7ffac4161750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:13.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.273+0000 7ffac9400640 1 -- 172.21.15.6:0/585684950 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffac41ee3f0 con 0x7ffac4162280 2024-03-04T09:31:13.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.273+0000 7ffac9400640 1 --2- 172.21.15.6:0/585684950 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffac4162280 0x7ffac41523a0 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7ffab400c820 tx=0x7ffab400cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:13.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.273+0000 7ffabe000640 1 -- 172.21.15.6:0/585684950 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffab4015070 con 0x7ffac4162280 2024-03-04T09:31:13.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.273+0000 7ffabe000640 1 -- 172.21.15.6:0/585684950 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffab4004020 con 0x7ffac4162280 2024-03-04T09:31:13.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.273+0000 7ffacb200640 1 -- 172.21.15.6:0/585684950 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffac41ee6d0 con 0x7ffac4162280 2024-03-04T09:31:13.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.273+0000 7ffabe000640 1 -- 172.21.15.6:0/585684950 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffab4004410 con 0x7ffac4162280 2024-03-04T09:31:13.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.274+0000 7ffacb200640 1 -- 172.21.15.6:0/585684950 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffac4162940 con 0x7ffac4162280 2024-03-04T09:31:13.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.274+0000 7ffacb200640 1 -- 172.21.15.6:0/585684950 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffac404f330 con 0x7ffac4162280 2024-03-04T09:31:13.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.275+0000 7ffabe000640 1 -- 172.21.15.6:0/585684950 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ffab4007830 con 0x7ffac4162280 2024-03-04T09:31:13.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.276+0000 7ffabe000640 1 --2- 172.21.15.6:0/585684950 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ffaa806b4d0 0x7ffaa806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:13.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.276+0000 7ffabe000640 1 -- 172.21.15.6:0/585684950 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5115+0+0 (secure 0 0 0) 0x7ffab4089de0 con 0x7ffac4162280 2024-03-04T09:31:13.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.277+0000 7ffac9e00640 1 --2- 172.21.15.6:0/585684950 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ffaa806b4d0 0x7ffaa806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:13.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.278+0000 7ffac9e00640 1 --2- 172.21.15.6:0/585684950 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ffaa806b4d0 0x7ffaa806d990 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7ffab80015a0 tx=0x7ffab8009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:13.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.281+0000 7ffabe000640 1 -- 172.21.15.6:0/585684950 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ffab4058e10 con 0x7ffac4162280 2024-03-04T09:31:13.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.479+0000 7ffacb200640 1 -- 172.21.15.6:0/585684950 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ffac406ce20 con 0x7ffac4162280 2024-03-04T09:31:13.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.480+0000 7ffabe000640 1 -- 172.21.15.6:0/585684950 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v101) v1 ==== 93+0+10 (secure 0 0 0) 0x7ffab4058710 con 0x7ffac4162280 2024-03-04T09:31:13.477 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-04T09:31:13.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.483+0000 7ffacb200640 1 -- 172.21.15.6:0/585684950 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ffaa806b4d0 msgr2=0x7ffaa806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:13.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.483+0000 7ffacb200640 1 --2- 172.21.15.6:0/585684950 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ffaa806b4d0 0x7ffaa806d990 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7ffab80015a0 tx=0x7ffab8009040 comp rx=0 tx=0).stop 2024-03-04T09:31:13.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.483+0000 7ffacb200640 1 -- 172.21.15.6:0/585684950 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffac4162280 msgr2=0x7ffac41523a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:13.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.483+0000 7ffacb200640 1 --2- 172.21.15.6:0/585684950 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffac4162280 0x7ffac41523a0 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7ffab400c820 tx=0x7ffab400cce0 comp rx=0 tx=0).stop 2024-03-04T09:31:13.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.483+0000 7ffacb200640 1 -- 172.21.15.6:0/585684950 shutdown_connections 2024-03-04T09:31:13.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.483+0000 7ffacb200640 1 --2- 172.21.15.6:0/585684950 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ffaa806b4d0 0x7ffaa806d990 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:13.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.483+0000 7ffacb200640 1 --2- 172.21.15.6:0/585684950 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ffac4165d50 0x7ffac415f510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:13.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.483+0000 7ffacb200640 1 --2- 172.21.15.6:0/585684950 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffac4162280 0x7ffac41523a0 unknown :-1 s=CLOSED pgs=1208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:13.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.483+0000 7ffacb200640 1 --2- 172.21.15.6:0/585684950 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffac40b1b40 0x7ffac4161750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:13.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.483+0000 7ffacb200640 1 -- 172.21.15.6:0/585684950 >> 172.21.15.6:0/585684950 conn(0x7ffac406b3a0 msgr2=0x7ffac415d560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:13.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.483+0000 7ffacb200640 1 -- 172.21.15.6:0/585684950 shutdown_connections 2024-03-04T09:31:13.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.483+0000 7ffacb200640 1 -- 172.21.15.6:0/585684950 wait complete. 2024-03-04T09:31:13.501 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_hardlink_reintegration' 2024-03-04T09:31:13.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.730+0000 7f0347000640 1 Processor -- start 2024-03-04T09:31:13.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.731+0000 7f0347000640 1 -- start start 2024-03-04T09:31:13.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.732+0000 7f0347000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f034015f360 0x7f03400b6230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:13.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.732+0000 7f0347000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f034006f0a0 con 0x7f03400b6770 2024-03-04T09:31:13.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.732+0000 7f0347000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f034006f380 con 0x7f034015df70 2024-03-04T09:31:13.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.733+0000 7f0347000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f034006fcb0 con 0x7f034015f360 2024-03-04T09:31:13.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.733+0000 7f033f400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f034015f360 0x7f03400b6230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:13.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.733+0000 7f0345000640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f03400b6770 0x7f0340161cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:48360/0 (socket says 172.21.15.6:48360) 2024-03-04T09:31:13.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.733+0000 7f033f400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f034015f360 0x7f03400b6230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55066/0 (socket says 172.21.15.6:55066) 2024-03-04T09:31:13.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.733+0000 7f0345000640 1 -- 172.21.15.6:0/68636508 learned_addr learned my addr 172.21.15.6:0/68636508 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:31:13.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.733+0000 7f033f400640 1 -- 172.21.15.6:0/68636508 >> v1:172.21.15.146:6789/0 conn(0x7f034015df70 legacy=0x7f0340070140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:13.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.733+0000 7f033f400640 1 -- 172.21.15.6:0/68636508 >> v1:172.21.15.6:6789/0 conn(0x7f03400b6770 legacy=0x7f0340161cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:13.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.733+0000 7f033f400640 1 -- 172.21.15.6:0/68636508 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f034006f660 con 0x7f034015f360 2024-03-04T09:31:13.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.733+0000 7f033ea00640 1 -- 172.21.15.6:0/68636508 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1259752327 0 0) 0x7f034006f0a0 con 0x7f03400b6770 2024-03-04T09:31:13.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.734+0000 7f033f400640 1 --2- 172.21.15.6:0/68636508 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f034015f360 0x7f03400b6230 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7f0328009d90 tx=0x7f032802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9472224f50b98ef server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:13.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.735+0000 7f033ea00640 1 -- 172.21.15.6:0/68636508 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0328039070 con 0x7f034015f360 2024-03-04T09:31:13.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.735+0000 7f033ea00640 1 -- 172.21.15.6:0/68636508 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f032802cc60 con 0x7f034015f360 2024-03-04T09:31:13.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.735+0000 7f033ea00640 1 -- 172.21.15.6:0/68636508 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0328034660 con 0x7f034015f360 2024-03-04T09:31:13.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.735+0000 7f0347000640 1 -- 172.21.15.6:0/68636508 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f034015f360 msgr2=0x7f03400b6230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:13.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.735+0000 7f0347000640 1 --2- 172.21.15.6:0/68636508 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f034015f360 0x7f03400b6230 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7f0328009d90 tx=0x7f032802beb0 comp rx=0 tx=0).stop 2024-03-04T09:31:13.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.735+0000 7f0347000640 1 -- 172.21.15.6:0/68636508 shutdown_connections 2024-03-04T09:31:13.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.735+0000 7f0347000640 1 --2- 172.21.15.6:0/68636508 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f034015f360 0x7f03400b6230 unknown :-1 s=CLOSED pgs=1210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:13.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.736+0000 7f0347000640 1 -- 172.21.15.6:0/68636508 >> 172.21.15.6:0/68636508 conn(0x7f034009aa10 msgr2=0x7f034009ae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:13.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.736+0000 7f0347000640 1 -- 172.21.15.6:0/68636508 shutdown_connections 2024-03-04T09:31:13.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.736+0000 7f0347000640 1 -- 172.21.15.6:0/68636508 wait complete. 2024-03-04T09:31:13.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.737+0000 7f0347000640 1 Processor -- start 2024-03-04T09:31:13.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.738+0000 7f0347000640 1 -- start start 2024-03-04T09:31:13.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.738+0000 7f0347000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f03400b6770 0x7f034014a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:13.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.739+0000 7f0347000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f034015df70 0x7f03400b3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:13.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.739+0000 7f0347000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f03400b3b50 0x7f034009e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:13.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.739+0000 7f0347000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0340162550 con 0x7f034015df70 2024-03-04T09:31:13.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.739+0000 7f0347000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03401623d0 con 0x7f03400b6770 2024-03-04T09:31:13.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.739+0000 7f0347000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f034006c0c0 con 0x7f03400b3b50 2024-03-04T09:31:13.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.739+0000 7f033f400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f034015df70 0x7f03400b3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:13.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.739+0000 7f033f400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f034015df70 0x7f03400b3610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55072/0 (socket says 172.21.15.6:55072) 2024-03-04T09:31:13.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.739+0000 7f033f400640 1 -- 172.21.15.6:0/2642417077 learned_addr learned my addr 172.21.15.6:0/2642417077 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:31:13.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.739+0000 7f033f400640 1 -- 172.21.15.6:0/2642417077 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f03400b3b50 msgr2=0x7f034009e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:13.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.739+0000 7f033f400640 1 --2- 172.21.15.6:0/2642417077 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f03400b3b50 0x7f034009e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:13.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.739+0000 7f033f400640 1 -- 172.21.15.6:0/2642417077 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f03400b6770 msgr2=0x7f034014a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:13.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.739+0000 7f033f400640 1 --2- 172.21.15.6:0/2642417077 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f03400b6770 0x7f034014a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:13.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.739+0000 7f033f400640 1 -- 172.21.15.6:0/2642417077 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f034009f750 con 0x7f034015df70 2024-03-04T09:31:13.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.740+0000 7f033f400640 1 --2- 172.21.15.6:0/2642417077 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f034015df70 0x7f03400b3610 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7f0328037040 tx=0x7f0328004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:13.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.740+0000 7f0337e00640 1 -- 172.21.15.6:0/2642417077 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0328044020 con 0x7f034015df70 2024-03-04T09:31:13.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.740+0000 7f0337e00640 1 -- 172.21.15.6:0/2642417077 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f032803f670 con 0x7f034015df70 2024-03-04T09:31:13.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.740+0000 7f0337e00640 1 -- 172.21.15.6:0/2642417077 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0328039040 con 0x7f034015df70 2024-03-04T09:31:13.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.740+0000 7f0347000640 1 -- 172.21.15.6:0/2642417077 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f034009fa30 con 0x7f034015df70 2024-03-04T09:31:13.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.740+0000 7f0347000640 1 -- 172.21.15.6:0/2642417077 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f034014ac80 con 0x7f034015df70 2024-03-04T09:31:13.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.741+0000 7f0347000640 1 -- 172.21.15.6:0/2642417077 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f034004f3a0 con 0x7f034015df70 2024-03-04T09:31:13.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.743+0000 7f0337e00640 1 -- 172.21.15.6:0/2642417077 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0328047020 con 0x7f034015df70 2024-03-04T09:31:13.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.743+0000 7f0337e00640 1 --2- 172.21.15.6:0/2642417077 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f031c06b520 0x7f031c06d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:13.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.744+0000 7f0337e00640 1 -- 172.21.15.6:0/2642417077 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5115+0+0 (secure 0 0 0) 0x7f03280acb90 con 0x7f034015df70 2024-03-04T09:31:13.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.744+0000 7f033fe00640 1 --2- 172.21.15.6:0/2642417077 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f031c06b520 0x7f031c06d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:13.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.745+0000 7f033fe00640 1 --2- 172.21.15.6:0/2642417077 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f031c06b520 0x7f031c06d9e0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f0330003cc0 tx=0x7f03300079b0 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:13.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:13.749+0000 7f0337e00640 1 -- 172.21.15.6:0/2642417077 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f032804e030 con 0x7f034015df70 2024-03-04T09:31:14.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:14.024+0000 7f0347000640 1 -- 172.21.15.6:0/2642417077 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_hardlink_reintegration"]} v 0) v1 -- 0x7f034009fc00 con 0x7f034015df70 2024-03-04T09:31:14.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:14.553+0000 7f0337e00640 1 -- 172.21.15.6:0/2642417077 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_hardlink_reintegration"]}]=0 v340) v1 ==== 145+0+0 (secure 0 0 0) 0x7f032807bcd0 con 0x7f034015df70 2024-03-04T09:31:14.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:14.556+0000 7f0347000640 1 -- 172.21.15.6:0/2642417077 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f031c06b520 msgr2=0x7f031c06d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:14.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:14.556+0000 7f0347000640 1 --2- 172.21.15.6:0/2642417077 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f031c06b520 0x7f031c06d9e0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f0330003cc0 tx=0x7f03300079b0 comp rx=0 tx=0).stop 2024-03-04T09:31:14.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:14.556+0000 7f0347000640 1 -- 172.21.15.6:0/2642417077 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f034015df70 msgr2=0x7f03400b3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:14.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:14.556+0000 7f0347000640 1 --2- 172.21.15.6:0/2642417077 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f034015df70 0x7f03400b3610 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7f0328037040 tx=0x7f0328004a40 comp rx=0 tx=0).stop 2024-03-04T09:31:14.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:14.556+0000 7f0347000640 1 -- 172.21.15.6:0/2642417077 shutdown_connections 2024-03-04T09:31:14.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:14.556+0000 7f0347000640 1 --2- 172.21.15.6:0/2642417077 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f031c06b520 0x7f031c06d9e0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:14.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:14.556+0000 7f0347000640 1 --2- 172.21.15.6:0/2642417077 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f03400b3b50 0x7f034009e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:14.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:14.556+0000 7f0347000640 1 --2- 172.21.15.6:0/2642417077 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f034015df70 0x7f03400b3610 unknown :-1 s=CLOSED pgs=1211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:14.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:14.556+0000 7f0347000640 1 --2- 172.21.15.6:0/2642417077 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f03400b6770 0x7f034014a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:14.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:14.556+0000 7f0347000640 1 -- 172.21.15.6:0/2642417077 >> 172.21.15.6:0/2642417077 conn(0x7f034009aa10 msgr2=0x7f034006c450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:14.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:14.557+0000 7f0347000640 1 -- 172.21.15.6:0/2642417077 shutdown_connections 2024-03-04T09:31:14.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:14.557+0000 7f0347000640 1 -- 172.21.15.6:0/2642417077 wait complete. 2024-03-04T09:31:14.576 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-04T09:31:14.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:14.809+0000 7fce05e00640 1 Processor -- start 2024-03-04T09:31:14.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:14.809+0000 7fce05e00640 1 -- start start 2024-03-04T09:31:14.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:14.811+0000 7fce05e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fce0006d8d0 0x7fce000b1b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:14.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:14.811+0000 7fce05e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce0006de40 con 0x7fce000b2080 2024-03-04T09:31:14.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:14.811+0000 7fce05e00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce0006e120 con 0x7fce00147590 2024-03-04T09:31:14.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:14.811+0000 7fce05e00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce00147970 con 0x7fce0006d8d0 2024-03-04T09:31:14.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:14.811+0000 7fce05400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fce000b2080 0x7fce000b3640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:48372/0 (socket says 172.21.15.6:48372) 2024-03-04T09:31:14.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:14.811+0000 7fce05400640 1 -- 172.21.15.6:0/3899336332 learned_addr learned my addr 172.21.15.6:0/3899336332 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:31:14.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:14.812+0000 7fcdffe00640 1 --2- 172.21.15.6:0/3899336332 >> v2:172.21.15.146:3300/0 conn(0x7fce0006d8d0 0x7fce000b1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:14.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:14.812+0000 7fcdff400640 1 -- 172.21.15.6:0/3899336332 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3920380086 0 0) 0x7fce0006de40 con 0x7fce000b2080 2024-03-04T09:31:14.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:14.812+0000 7fcdff400640 1 -- 172.21.15.6:0/3899336332 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcde4003520 con 0x7fce000b2080 2024-03-04T09:31:14.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:14.812+0000 7fcdff400640 1 -- 172.21.15.6:0/3899336332 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 413662402 0 0) 0x7fcde4003520 con 0x7fce000b2080 2024-03-04T09:31:14.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:14.812+0000 7fcdff400640 1 -- 172.21.15.6:0/3899336332 >> v2:172.21.15.146:3300/0 conn(0x7fce0006d8d0 msgr2=0x7fce000b1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:14.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:14.812+0000 7fcdff400640 1 --2- 172.21.15.6:0/3899336332 >> v2:172.21.15.146:3300/0 conn(0x7fce0006d8d0 0x7fce000b1b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:14.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:14.812+0000 7fcdff400640 1 -- 172.21.15.6:0/3899336332 >> v1:172.21.15.146:6789/0 conn(0x7fce00147590 legacy=0x7fce00071480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:14.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:14.812+0000 7fcdff400640 1 -- 172.21.15.6:0/3899336332 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce000b2e40 con 0x7fce000b2080 2024-03-04T09:31:14.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:14.812+0000 7fcdff400640 1 -- 172.21.15.6:0/3899336332 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7fcdf8001670 con 0x7fce000b2080 2024-03-04T09:31:14.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:14.812+0000 7fcdff400640 1 -- 172.21.15.6:0/3899336332 >> v1:172.21.15.6:6789/0 conn(0x7fce000b2080 legacy=0x7fce000b3640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:14.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:14.812+0000 7fcdff400640 1 --2- 172.21.15.6:0/3899336332 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcde40039b0 0x7fcde4003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:14.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:14.812+0000 7fcdff400640 1 --2- 172.21.15.6:0/3899336332 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcde4004800 0x7fcde4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:14.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:14.813+0000 7fcdff400640 1 --2- 172.21.15.6:0/3899336332 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcde40051c0 0x7fcde400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:14.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:14.813+0000 7fcdff400640 1 -- 172.21.15.6:0/3899336332 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce00147970 con 0x7fcde4004800 2024-03-04T09:31:14.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:14.813+0000 7fcdff400640 1 -- 172.21.15.6:0/3899336332 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcde400c090 con 0x7fcde40051c0 2024-03-04T09:31:14.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:14.813+0000 7fcdff400640 1 -- 172.21.15.6:0/3899336332 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcde400c290 con 0x7fcde40039b0 2024-03-04T09:31:14.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:14.813+0000 7fcdffe00640 1 --2- 172.21.15.6:0/3899336332 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcde4004800 0x7fcde4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:14.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:14.813+0000 7fce04a00640 1 --2- 172.21.15.6:0/3899336332 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcde40039b0 0x7fcde4003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:14.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:14.813+0000 7fcdffe00640 1 -- 172.21.15.6:0/3899336332 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcde40039b0 msgr2=0x7fcde4003df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:14.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:14.813+0000 7fcdffe00640 1 --2- 172.21.15.6:0/3899336332 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcde40039b0 0x7fcde4003df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:14.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:14.813+0000 7fcdffe00640 1 -- 172.21.15.6:0/3899336332 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcde40051c0 msgr2=0x7fcde400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:14.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:14.813+0000 7fce05400640 1 --2- 172.21.15.6:0/3899336332 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcde40051c0 0x7fcde400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:14.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:14.813+0000 7fcdffe00640 1 --2- 172.21.15.6:0/3899336332 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcde40051c0 0x7fcde400b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:14.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:14.813+0000 7fcdffe00640 1 -- 172.21.15.6:0/3899336332 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcde4003520 con 0x7fcde4004800 2024-03-04T09:31:14.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:14.813+0000 7fce05400640 1 -- 172.21.15.6:0/3899336332 reap_dead start 2024-03-04T09:31:14.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:14.813+0000 7fce04a00640 1 --2- 172.21.15.6:0/3899336332 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcde40039b0 0x7fcde4003df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:31:14.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:14.814+0000 7fcdffe00640 1 --2- 172.21.15.6:0/3899336332 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcde4004800 0x7fcde4004c80 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7fcdec009ac0 tx=0x7fcde4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2641a32a257852ae server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:14.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:14.815+0000 7fcdff400640 1 -- 172.21.15.6:0/3899336332 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdec039070 con 0x7fcde4004800 2024-03-04T09:31:14.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:14.815+0000 7fcdff400640 1 -- 172.21.15.6:0/3899336332 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdec02c6b0 con 0x7fcde4004800 2024-03-04T09:31:14.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:14.815+0000 7fcdff400640 1 -- 172.21.15.6:0/3899336332 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdec009690 con 0x7fcde4004800 2024-03-04T09:31:14.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:14.815+0000 7fce05e00640 1 -- 172.21.15.6:0/3899336332 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcde4004800 msgr2=0x7fcde4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:14.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:14.815+0000 7fce05e00640 1 --2- 172.21.15.6:0/3899336332 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcde4004800 0x7fcde4004c80 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7fcdec009ac0 tx=0x7fcde4005830 comp rx=0 tx=0).stop 2024-03-04T09:31:14.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:14.815+0000 7fce05e00640 1 -- 172.21.15.6:0/3899336332 shutdown_connections 2024-03-04T09:31:14.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:14.815+0000 7fce05e00640 1 --2- 172.21.15.6:0/3899336332 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcde4004800 0x7fcde4004c80 unknown :-1 s=CLOSED pgs=1213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:14.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:14.815+0000 7fce05e00640 1 -- 172.21.15.6:0/3899336332 >> 172.21.15.6:0/3899336332 conn(0x7fce0006b3a0 msgr2=0x7fce0006b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:14.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:14.815+0000 7fce05e00640 1 -- 172.21.15.6:0/3899336332 shutdown_connections 2024-03-04T09:31:14.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:14.815+0000 7fce05e00640 1 -- 172.21.15.6:0/3899336332 wait complete. 2024-03-04T09:31:14.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:14.817+0000 7fce05e00640 1 Processor -- start 2024-03-04T09:31:14.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:14.817+0000 7fce05e00640 1 -- start start 2024-03-04T09:31:14.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:14.818+0000 7fce05e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcde4004800 0x7fce0015d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:14.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:14.818+0000 7fce05e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fce0013dd00 0x7fce000b4510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:14.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:14.818+0000 7fce05e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fce0013e180 0x7fce000b4a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:14.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:14.818+0000 7fce05e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce000b24b0 con 0x7fce0013e180 2024-03-04T09:31:14.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:14.818+0000 7fce05e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce000b31b0 con 0x7fce0013dd00 2024-03-04T09:31:14.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:14.818+0000 7fce05e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce000b1b40 con 0x7fcde4004800 2024-03-04T09:31:14.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:14.819+0000 7fce05400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fce0013e180 0x7fce000b4a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:14.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:14.819+0000 7fce05400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fce0013e180 0x7fce000b4a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55084/0 (socket says 172.21.15.6:55084) 2024-03-04T09:31:14.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:14.819+0000 7fce05400640 1 -- 172.21.15.6:0/377958227 learned_addr learned my addr 172.21.15.6:0/377958227 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:31:14.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:14.819+0000 7fcdffe00640 1 --2- 172.21.15.6:0/377958227 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fce0013dd00 0x7fce000b4510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:14.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:14.819+0000 7fce04a00640 1 --2- 172.21.15.6:0/377958227 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcde4004800 0x7fce0015d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:14.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:14.819+0000 7fce05400640 1 -- 172.21.15.6:0/377958227 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcde4004800 msgr2=0x7fce0015d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:14.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:14.819+0000 7fce05400640 1 --2- 172.21.15.6:0/377958227 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcde4004800 0x7fce0015d7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:14.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:14.819+0000 7fce05400640 1 -- 172.21.15.6:0/377958227 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fce0013dd00 msgr2=0x7fce000b4510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:14.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:14.819+0000 7fce05400640 1 --2- 172.21.15.6:0/377958227 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fce0013dd00 0x7fce000b4510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:14.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:14.819+0000 7fce05400640 1 -- 172.21.15.6:0/377958227 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce00161050 con 0x7fce0013e180 2024-03-04T09:31:14.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:14.819+0000 7fce04a00640 1 --2- 172.21.15.6:0/377958227 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcde4004800 0x7fce0015d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:31:14.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:14.819+0000 7fce05400640 1 --2- 172.21.15.6:0/377958227 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fce0013e180 0x7fce000b4a50 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7fce0006ebe0 tx=0x7fcdf800d9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:14.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:14.820+0000 7fcdfcc00640 1 -- 172.21.15.6:0/377958227 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdf80074e0 con 0x7fce0013e180 2024-03-04T09:31:14.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:14.820+0000 7fcdfcc00640 1 -- 172.21.15.6:0/377958227 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdf80047e0 con 0x7fce0013e180 2024-03-04T09:31:14.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:14.820+0000 7fcdfcc00640 1 -- 172.21.15.6:0/377958227 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdf8004bb0 con 0x7fce0013e180 2024-03-04T09:31:14.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:14.820+0000 7fce05e00640 1 -- 172.21.15.6:0/377958227 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce001e6800 con 0x7fce0013e180 2024-03-04T09:31:14.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:14.820+0000 7fce05e00640 1 -- 172.21.15.6:0/377958227 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce001e6dc0 con 0x7fce0013e180 2024-03-04T09:31:14.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:14.821+0000 7fce05e00640 1 -- 172.21.15.6:0/377958227 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce0004f330 con 0x7fce0013e180 2024-03-04T09:31:14.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:14.822+0000 7fcdfcc00640 1 -- 172.21.15.6:0/377958227 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcdf801b020 con 0x7fce0013e180 2024-03-04T09:31:14.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:14.823+0000 7fcdfcc00640 1 --2- 172.21.15.6:0/377958227 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fcde006b4d0 0x7fcde006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:14.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:14.823+0000 7fcdfcc00640 1 -- 172.21.15.6:0/377958227 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5115+0+0 (secure 0 0 0) 0x7fcdf8089b50 con 0x7fce0013e180 2024-03-04T09:31:14.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:14.823+0000 7fce04a00640 1 --2- 172.21.15.6:0/377958227 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fcde006b4d0 0x7fcde006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:14.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:14.825+0000 7fce04a00640 1 --2- 172.21.15.6:0/377958227 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fcde006b4d0 0x7fcde006d990 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fce000686c0 tx=0x7fcdf0009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:14.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:14.827+0000 7fcdfcc00640 1 -- 172.21.15.6:0/377958227 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcdf800f2f0 con 0x7fce0013e180 2024-03-04T09:31:15.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:15.028+0000 7fce05e00640 1 -- 172.21.15.6:0/377958227 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fce001e6f90 con 0x7fce0013e180 2024-03-04T09:31:15.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:15.035+0000 7fcdfcc00640 1 -- 172.21.15.6:0/377958227 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v30)=0 v30) v1 ==== 122+0+0 (secure 0 0 0) 0x7fcdf8058c10 con 0x7fce0013e180 2024-03-04T09:31:15.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:15.037+0000 7fce05e00640 1 -- 172.21.15.6:0/377958227 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fcde006b4d0 msgr2=0x7fcde006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:15.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:15.037+0000 7fce05e00640 1 --2- 172.21.15.6:0/377958227 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fcde006b4d0 0x7fcde006d990 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fce000686c0 tx=0x7fcdf0009040 comp rx=0 tx=0).stop 2024-03-04T09:31:15.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:15.037+0000 7fce05e00640 1 -- 172.21.15.6:0/377958227 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fce0013e180 msgr2=0x7fce000b4a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:15.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:15.037+0000 7fce05e00640 1 --2- 172.21.15.6:0/377958227 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fce0013e180 0x7fce000b4a50 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7fce0006ebe0 tx=0x7fcdf800d9b0 comp rx=0 tx=0).stop 2024-03-04T09:31:15.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:15.038+0000 7fce05e00640 1 -- 172.21.15.6:0/377958227 shutdown_connections 2024-03-04T09:31:15.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:15.038+0000 7fce05e00640 1 --2- 172.21.15.6:0/377958227 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fce0013e180 0x7fce000b4a50 unknown :-1 s=CLOSED pgs=1214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:15.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:15.038+0000 7fce05e00640 1 --2- 172.21.15.6:0/377958227 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fce0013dd00 0x7fce000b4510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:15.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:15.038+0000 7fce05e00640 1 --2- 172.21.15.6:0/377958227 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fcde006b4d0 0x7fcde006d990 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:15.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:15.038+0000 7fce05e00640 1 --2- 172.21.15.6:0/377958227 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcde4004800 0x7fce0015d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:15.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:15.038+0000 7fce05e00640 1 -- 172.21.15.6:0/377958227 >> 172.21.15.6:0/377958227 conn(0x7fce0006b3a0 msgr2=0x7fce000ae110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:15.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:15.038+0000 7fce05e00640 1 -- 172.21.15.6:0/377958227 shutdown_connections 2024-03-04T09:31:15.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:15.038+0000 7fce05e00640 1 -- 172.21.15.6:0/377958227 wait complete. 2024-03-04T09:31:15.055 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-04T09:31:15.055 DEBUG:teuthology.orchestra.run.smithi006:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-04T09:31:15.169 DEBUG:teuthology.orchestra.run.smithi146:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-04T09:31:15.299 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-04T09:31:15.299 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-04T09:31:15.299 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T09:31:15.360 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054202,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249800k,nr_inodes=812450,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-04T09:31:15.360 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-04T09:31:15.361 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T09:31:15.416 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053429,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249180k,nr_inodes=812295,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-04T09:31:15.416 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-04T09:31:15.417 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T09:31:15.474 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054202,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249800k,nr_inodes=812450,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-04T09:31:15.474 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-04T09:31:15.474 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T09:31:15.533 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053429,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249180k,nr_inodes=812295,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-04T09:31:15.533 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-04T09:31:15.533 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:31:15.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:15.768+0000 7f3363e00640 1 Processor -- start 2024-03-04T09:31:15.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:15.769+0000 7f3363e00640 1 -- start start 2024-03-04T09:31:15.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:15.770+0000 7f3363e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f336415e0b0 0x7f33640cf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:15.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:15.770+0000 7f3363e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f33640d4a10 0x7f33640d0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:15.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:15.770+0000 7f3363e00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33640b3710 con 0x7f33640d0c80 2024-03-04T09:31:15.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:15.770+0000 7f3363e00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33640b09e0 con 0x7f33640d4a10 2024-03-04T09:31:15.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:15.770+0000 7f3363e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33640b39f0 con 0x7f336415e0b0 2024-03-04T09:31:15.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:15.771+0000 7f3362a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f336415e0b0 0x7f33640cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:15.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:15.771+0000 7f3362000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f33640d4a10 0x7f33640d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:15.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:15.771+0000 7f3362a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f336415e0b0 0x7f33640cf310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.6:43554/0 (socket says 172.21.15.6:43554) 2024-03-04T09:31:15.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:15.771+0000 7f3362a00640 1 -- 172.21.15.6:0/1541220250 learned_addr learned my addr 172.21.15.6:0/1541220250 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:31:15.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:15.771+0000 7f3362a00640 1 -- 172.21.15.6:0/1541220250 >> v2:172.21.15.146:3300/0 conn(0x7f33640d4a10 msgr2=0x7f33640d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:15.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:15.771+0000 7f3362a00640 1 --2- 172.21.15.6:0/1541220250 >> v2:172.21.15.146:3300/0 conn(0x7f33640d4a10 0x7f33640d0740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:15.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:15.771+0000 7f3362a00640 1 -- 172.21.15.6:0/1541220250 >> v1:172.21.15.146:6789/0 conn(0x7f33640d0c80 legacy=0x7f33640d1900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:15.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:15.771+0000 7f3362a00640 1 -- 172.21.15.6:0/1541220250 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33640b3cd0 con 0x7f336415e0b0 2024-03-04T09:31:15.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:15.773+0000 7f3362a00640 1 --2- 172.21.15.6:0/1541220250 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f336415e0b0 0x7f33640cf310 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f3354009870 tx=0x7f335402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cd145846b110faba server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:15.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:15.773+0000 7f3361600640 1 -- 172.21.15.6:0/1541220250 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3354034070 con 0x7f336415e0b0 2024-03-04T09:31:15.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:15.773+0000 7f3361600640 1 -- 172.21.15.6:0/1541220250 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3354030840 con 0x7f336415e0b0 2024-03-04T09:31:15.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:15.773+0000 7f3361600640 1 -- 172.21.15.6:0/1541220250 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3354030c10 con 0x7f336415e0b0 2024-03-04T09:31:15.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:15.774+0000 7f3363e00640 1 -- 172.21.15.6:0/1541220250 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f336415e0b0 msgr2=0x7f33640cf310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:15.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:15.774+0000 7f3363e00640 1 --2- 172.21.15.6:0/1541220250 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f336415e0b0 0x7f33640cf310 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f3354009870 tx=0x7f335402b980 comp rx=0 tx=0).stop 2024-03-04T09:31:15.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:15.774+0000 7f3363e00640 1 -- 172.21.15.6:0/1541220250 shutdown_connections 2024-03-04T09:31:15.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:15.774+0000 7f3363e00640 1 --2- 172.21.15.6:0/1541220250 >> v2:172.21.15.146:3300/0 conn(0x7f33640d4a10 0x7f33640d0740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:15.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:15.774+0000 7f3363e00640 1 --2- 172.21.15.6:0/1541220250 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f336415e0b0 0x7f33640cf310 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:15.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:15.774+0000 7f3363e00640 1 -- 172.21.15.6:0/1541220250 >> 172.21.15.6:0/1541220250 conn(0x7f33640d3d30 msgr2=0x7f33640d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:15.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:15.774+0000 7f3363e00640 1 -- 172.21.15.6:0/1541220250 shutdown_connections 2024-03-04T09:31:15.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:15.774+0000 7f3363e00640 1 -- 172.21.15.6:0/1541220250 wait complete. 2024-03-04T09:31:15.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:15.776+0000 7f3363e00640 1 Processor -- start 2024-03-04T09:31:15.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:15.776+0000 7f3363e00640 1 -- start start 2024-03-04T09:31:15.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:15.777+0000 7f3363e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f33640d0c80 0x7f33640cd190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:15.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:15.777+0000 7f3363e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f33640d4a10 0x7f33640b9340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:15.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:15.777+0000 7f3363e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f336415e0b0 0x7f33640b6920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:15.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:15.777+0000 7f3363e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33640cfc60 con 0x7f33640d0c80 2024-03-04T09:31:15.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:15.777+0000 7f3363e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f336415e490 con 0x7f336415e0b0 2024-03-04T09:31:15.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:15.777+0000 7f3363e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33640b54e0 con 0x7f33640d4a10 2024-03-04T09:31:15.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:15.777+0000 7f3362a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f33640d0c80 0x7f33640cd190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:15.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:15.778+0000 7f3362a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f33640d0c80 0x7f33640cd190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55092/0 (socket says 172.21.15.6:55092) 2024-03-04T09:31:15.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:15.778+0000 7f3362a00640 1 -- 172.21.15.6:0/4286989714 learned_addr learned my addr 172.21.15.6:0/4286989714 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:31:15.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:15.778+0000 7f3362000640 1 --2- 172.21.15.6:0/4286989714 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f33640d4a10 0x7f33640b9340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:15.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:15.778+0000 7f3363400640 1 --2- 172.21.15.6:0/4286989714 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f336415e0b0 0x7f33640b6920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:15.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:15.778+0000 7f3362a00640 1 -- 172.21.15.6:0/4286989714 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f33640d4a10 msgr2=0x7f33640b9340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:15.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:15.778+0000 7f3362a00640 1 --2- 172.21.15.6:0/4286989714 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f33640d4a10 0x7f33640b9340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:15.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:15.778+0000 7f3362a00640 1 -- 172.21.15.6:0/4286989714 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f336415e0b0 msgr2=0x7f33640b6920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:15.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:15.778+0000 7f3362a00640 1 --2- 172.21.15.6:0/4286989714 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f336415e0b0 0x7f33640b6920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:15.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:15.778+0000 7f3362a00640 1 -- 172.21.15.6:0/4286989714 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33640cbe50 con 0x7f33640d0c80 2024-03-04T09:31:15.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:15.778+0000 7f3362a00640 1 --2- 172.21.15.6:0/4286989714 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f33640d0c80 0x7f33640cd190 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7f3354000c00 tx=0x7f33540046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:15.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:15.779+0000 7f335aa00640 1 -- 172.21.15.6:0/4286989714 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3354034040 con 0x7f33640d0c80 2024-03-04T09:31:15.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:15.779+0000 7f335aa00640 1 -- 172.21.15.6:0/4286989714 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3354039d90 con 0x7f33640d0c80 2024-03-04T09:31:15.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:15.779+0000 7f335aa00640 1 -- 172.21.15.6:0/4286989714 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3354043630 con 0x7f33640d0c80 2024-03-04T09:31:15.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:15.779+0000 7f3363e00640 1 -- 172.21.15.6:0/4286989714 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33640cc0f0 con 0x7f33640d0c80 2024-03-04T09:31:15.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:15.779+0000 7f3363e00640 1 -- 172.21.15.6:0/4286989714 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33640cc6b0 con 0x7f33640d0c80 2024-03-04T09:31:15.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:15.780+0000 7f3363e00640 1 -- 172.21.15.6:0/4286989714 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f336404f330 con 0x7f33640d0c80 2024-03-04T09:31:15.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:15.781+0000 7f335aa00640 1 -- 172.21.15.6:0/4286989714 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f335404a060 con 0x7f33640d0c80 2024-03-04T09:31:15.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:15.782+0000 7f335aa00640 1 --2- 172.21.15.6:0/4286989714 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f334406b4d0 0x7f334406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:15.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:15.782+0000 7f335aa00640 1 -- 172.21.15.6:0/4286989714 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5115+0+0 (secure 0 0 0) 0x7f33540abfd0 con 0x7f33640d0c80 2024-03-04T09:31:15.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:15.783+0000 7f3362000640 1 --2- 172.21.15.6:0/4286989714 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f334406b4d0 0x7f334406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:15.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:15.784+0000 7f3362000640 1 --2- 172.21.15.6:0/4286989714 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f334406b4d0 0x7f334406d990 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f336415a160 tx=0x7f3350009190 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:15.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:15.787+0000 7f335aa00640 1 -- 172.21.15.6:0/4286989714 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f335403e070 con 0x7f33640d0c80 2024-03-04T09:31:16.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.032+0000 7f3363e00640 1 -- 172.21.15.6:0/4286989714 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f33640b0b70 con 0x7f33640d0c80 2024-03-04T09:31:16.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.033+0000 7f335aa00640 1 -- 172.21.15.6:0/4286989714 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) v1 ==== 94+0+3696 (secure 0 0 0) 0x7f335407af70 con 0x7f33640d0c80 2024-03-04T09:31:16.030 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:31:16.030 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":70,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5654,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/1370302030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":1370302030},{"type":"v1","addr":"172.21.15.146:6835","nonce":1370302030}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":6163,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/809822004","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":809822004},{"type":"v1","addr":"172.21.15.146:6837","nonce":809822004}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":66},{"gid":7284,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/1583413840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1583413840},{"type":"v1","addr":"172.21.15.6:6833","nonce":1583413840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":7497,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/4125448618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":4125448618},{"type":"v1","addr":"172.21.15.6:6835","nonce":4125448618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70}],"filesystems":[]} 2024-03-04T09:31:16.031 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 70 2024-03-04T09:31:16.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.036+0000 7f3363e00640 1 -- 172.21.15.6:0/4286989714 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f334406b4d0 msgr2=0x7f334406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:16.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.036+0000 7f3363e00640 1 --2- 172.21.15.6:0/4286989714 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f334406b4d0 0x7f334406d990 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f336415a160 tx=0x7f3350009190 comp rx=0 tx=0).stop 2024-03-04T09:31:16.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.036+0000 7f3363e00640 1 -- 172.21.15.6:0/4286989714 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f33640d0c80 msgr2=0x7f33640cd190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:16.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.036+0000 7f3363e00640 1 --2- 172.21.15.6:0/4286989714 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f33640d0c80 0x7f33640cd190 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7f3354000c00 tx=0x7f33540046c0 comp rx=0 tx=0).stop 2024-03-04T09:31:16.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.036+0000 7f3363e00640 1 -- 172.21.15.6:0/4286989714 shutdown_connections 2024-03-04T09:31:16.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.036+0000 7f3363e00640 1 --2- 172.21.15.6:0/4286989714 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f336415e0b0 0x7f33640b6920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:16.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.036+0000 7f3363e00640 1 --2- 172.21.15.6:0/4286989714 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f334406b4d0 0x7f334406d990 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:16.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.036+0000 7f3363e00640 1 --2- 172.21.15.6:0/4286989714 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f33640d4a10 0x7f33640b9340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:16.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.036+0000 7f3363e00640 1 --2- 172.21.15.6:0/4286989714 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f33640d0c80 0x7f33640cd190 unknown :-1 s=CLOSED pgs=1215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:16.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.036+0000 7f3363e00640 1 -- 172.21.15.6:0/4286989714 >> 172.21.15.6:0/4286989714 conn(0x7f33640d3d30 msgr2=0x7f336406d160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:16.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.036+0000 7f3363e00640 1 -- 172.21.15.6:0/4286989714 shutdown_connections 2024-03-04T09:31:16.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.036+0000 7f3363e00640 1 -- 172.21.15.6:0/4286989714 wait complete. 2024-03-04T09:31:16.052 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-04T09:31:16.052 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-04T09:31:16.052 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-04T09:31:16.052 INFO:teuthology.orchestra.run:waiting for 300 2024-03-04T09:31:16.052 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-04T09:31:16.053 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-04T09:31:16.053 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-04T09:31:16.053 INFO:teuthology.orchestra.run:waiting for 300 2024-03-04T09:31:16.053 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-04T09:31:16.053 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-04T09:31:16.053 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-04T09:31:16.053 INFO:teuthology.orchestra.run:waiting for 300 2024-03-04T09:31:16.054 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-04T09:31:16.054 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-04T09:31:16.054 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-04T09:31:16.054 INFO:teuthology.orchestra.run:waiting for 300 2024-03-04T09:31:16.069 INFO:tasks.ceph.mds.a:Stopped 2024-03-04T09:31:16.069 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-04T09:31:16.071 INFO:tasks.ceph.mds.c:Stopped 2024-03-04T09:31:16.071 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-04T09:31:16.077 INFO:tasks.ceph.mds.b:Stopped 2024-03-04T09:31:16.078 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-04T09:31:16.078 INFO:tasks.ceph.mds.d:Stopped 2024-03-04T09:31:16.079 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-04T09:31:16.116 INFO:tasks.ceph.mds.a:Started 2024-03-04T09:31:16.116 DEBUG:teuthology.parallel:result is None 2024-03-04T09:31:16.119 INFO:tasks.ceph.mds.c:Started 2024-03-04T09:31:16.119 DEBUG:teuthology.parallel:result is None 2024-03-04T09:31:16.125 INFO:tasks.ceph.mds.b:Started 2024-03-04T09:31:16.125 DEBUG:teuthology.parallel:result is None 2024-03-04T09:31:16.131 INFO:tasks.ceph.mds.d:Started 2024-03-04T09:31:16.131 DEBUG:teuthology.parallel:result is None 2024-03-04T09:31:16.131 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-04T09:31:16.273 INFO:tasks.ceph.mds.a.smithi006.stdout:starting mds.a at 2024-03-04T09:31:16.516 INFO:tasks.ceph.mds.b.smithi146.stdout:starting mds.b at 2024-03-04T09:31:16.517 INFO:tasks.ceph.mds.c.smithi006.stdout:starting mds.c at 2024-03-04T09:31:16.517 INFO:tasks.ceph.mds.d.smithi146.stdout:starting mds.d at 2024-03-04T09:31:16.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.323+0000 7f0582c00640 1 Processor -- start 2024-03-04T09:31:16.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.323+0000 7f0582c00640 1 -- start start 2024-03-04T09:31:16.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.324+0000 7f0582c00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f057c0af1a0 0x7f057c15c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:16.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.324+0000 7f0582c00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f057c15cc40 con 0x7f057c0cdc20 2024-03-04T09:31:16.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.324+0000 7f0582c00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f057c15cf20 con 0x7f057c0af580 2024-03-04T09:31:16.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.324+0000 7f0582c00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f057c15d200 con 0x7f057c0af1a0 2024-03-04T09:31:16.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.324+0000 7f0580c00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f057c0cdc20 0x7f057c0b2dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:48404/0 (socket says 172.21.15.6:48404) 2024-03-04T09:31:16.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.324+0000 7f0580c00640 1 -- 172.21.15.6:0/2397079578 learned_addr learned my addr 172.21.15.6:0/2397079578 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:31:16.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.324+0000 7f057be00640 1 --2- 172.21.15.6:0/2397079578 >> v2:172.21.15.146:3300/0 conn(0x7f057c0af1a0 0x7f057c15c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:16.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.325+0000 7f057b400640 1 -- 172.21.15.6:0/2397079578 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3217686008 0 0) 0x7f057c15cc40 con 0x7f057c0cdc20 2024-03-04T09:31:16.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.325+0000 7f057b400640 1 -- 172.21.15.6:0/2397079578 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0560003520 con 0x7f057c0cdc20 2024-03-04T09:31:16.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.325+0000 7f057b400640 1 -- 172.21.15.6:0/2397079578 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 552546028 0 0) 0x7f057c15cf20 con 0x7f057c0af580 2024-03-04T09:31:16.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.325+0000 7f057be00640 1 -- 172.21.15.6:0/2397079578 >> v1:172.21.15.146:6789/0 conn(0x7f057c0af580 legacy=0x7f057c0b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:16.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.325+0000 7f057be00640 1 -- 172.21.15.6:0/2397079578 >> v1:172.21.15.6:6789/0 conn(0x7f057c0cdc20 legacy=0x7f057c0b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:16.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.325+0000 7f057be00640 1 -- 172.21.15.6:0/2397079578 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f057c15d4e0 con 0x7f057c0af1a0 2024-03-04T09:31:16.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.326+0000 7f057be00640 1 --2- 172.21.15.6:0/2397079578 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f057c0af1a0 0x7f057c15c600 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f056c0097a0 tx=0x7f056c02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1cf222c5c5c9be94 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:16.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.327+0000 7f057b400640 1 -- 172.21.15.6:0/2397079578 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f056c035020 con 0x7f057c0af1a0 2024-03-04T09:31:16.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.327+0000 7f057b400640 1 -- 172.21.15.6:0/2397079578 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f056c031830 con 0x7f057c0af1a0 2024-03-04T09:31:16.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.327+0000 7f057b400640 1 -- 172.21.15.6:0/2397079578 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f056c031c00 con 0x7f057c0af1a0 2024-03-04T09:31:16.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.327+0000 7f0582c00640 1 -- 172.21.15.6:0/2397079578 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f057c0af1a0 msgr2=0x7f057c15c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:16.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.327+0000 7f0582c00640 1 --2- 172.21.15.6:0/2397079578 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f057c0af1a0 0x7f057c15c600 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f056c0097a0 tx=0x7f056c02b8d0 comp rx=0 tx=0).stop 2024-03-04T09:31:16.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.327+0000 7f0582c00640 1 -- 172.21.15.6:0/2397079578 shutdown_connections 2024-03-04T09:31:16.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.327+0000 7f0582c00640 1 --2- 172.21.15.6:0/2397079578 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f057c0af1a0 0x7f057c15c600 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:16.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.327+0000 7f0582c00640 1 -- 172.21.15.6:0/2397079578 >> 172.21.15.6:0/2397079578 conn(0x7f057c06afb0 msgr2=0x7f057c06b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:16.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.327+0000 7f0582c00640 1 -- 172.21.15.6:0/2397079578 shutdown_connections 2024-03-04T09:31:16.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.327+0000 7f0582c00640 1 -- 172.21.15.6:0/2397079578 wait complete. 2024-03-04T09:31:16.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.329+0000 7f0582c00640 1 Processor -- start 2024-03-04T09:31:16.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.329+0000 7f0582c00640 1 -- start start 2024-03-04T09:31:16.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.330+0000 7f0582c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f057c0af1a0 0x7f057c09ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:16.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.330+0000 7f0582c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f057c0af580 0x7f057c1f7bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:16.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.330+0000 7f0582c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f057c0cdc20 0x7f057c1f44d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:16.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.330+0000 7f0582c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f057c155e70 con 0x7f057c0af1a0 2024-03-04T09:31:16.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.330+0000 7f0582c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f057c0b3730 con 0x7f057c0af580 2024-03-04T09:31:16.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.330+0000 7f0582c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f057c155ff0 con 0x7f057c0cdc20 2024-03-04T09:31:16.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.330+0000 7f057be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f057c0af1a0 0x7f057c09ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:16.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.330+0000 7f057be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f057c0af1a0 0x7f057c09ab60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55120/0 (socket says 172.21.15.6:55120) 2024-03-04T09:31:16.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.330+0000 7f057be00640 1 -- 172.21.15.6:0/2900420706 learned_addr learned my addr 172.21.15.6:0/2900420706 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:31:16.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.331+0000 7f0580c00640 1 --2- 172.21.15.6:0/2900420706 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f057c0cdc20 0x7f057c1f44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:16.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.331+0000 7f0573400640 1 --2- 172.21.15.6:0/2900420706 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f057c0af580 0x7f057c1f7bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:16.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.331+0000 7f057be00640 1 -- 172.21.15.6:0/2900420706 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f057c0cdc20 msgr2=0x7f057c1f44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:16.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.331+0000 7f057be00640 1 --2- 172.21.15.6:0/2900420706 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f057c0cdc20 0x7f057c1f44d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:16.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.331+0000 7f057be00640 1 -- 172.21.15.6:0/2900420706 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f057c0af580 msgr2=0x7f057c1f7bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:16.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.331+0000 7f057be00640 1 --2- 172.21.15.6:0/2900420706 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f057c0af580 0x7f057c1f7bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:16.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.331+0000 7f057be00640 1 -- 172.21.15.6:0/2900420706 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f057c06e4b0 con 0x7f057c0af1a0 2024-03-04T09:31:16.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.331+0000 7f057be00640 1 --2- 172.21.15.6:0/2900420706 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f057c0af1a0 0x7f057c09ab60 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7f056c0098c0 tx=0x7f056c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:16.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.331+0000 7f0578c00640 1 -- 172.21.15.6:0/2900420706 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f056c035040 con 0x7f057c0af1a0 2024-03-04T09:31:16.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.331+0000 7f0578c00640 1 -- 172.21.15.6:0/2900420706 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f056c03a840 con 0x7f057c0af1a0 2024-03-04T09:31:16.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.331+0000 7f0578c00640 1 -- 172.21.15.6:0/2900420706 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f056c03ac10 con 0x7f057c0af1a0 2024-03-04T09:31:16.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.331+0000 7f0582c00640 1 -- 172.21.15.6:0/2900420706 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f057c06e790 con 0x7f057c0af1a0 2024-03-04T09:31:16.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.332+0000 7f0582c00640 1 -- 172.21.15.6:0/2900420706 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f057c1f3fb0 con 0x7f057c0af1a0 2024-03-04T09:31:16.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.333+0000 7f0582c00640 1 -- 172.21.15.6:0/2900420706 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f057c069510 con 0x7f057c0af1a0 2024-03-04T09:31:16.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.333+0000 7f0578c00640 1 -- 172.21.15.6:0/2900420706 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f056c038020 con 0x7f057c0af1a0 2024-03-04T09:31:16.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.334+0000 7f0578c00640 1 --2- 172.21.15.6:0/2900420706 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f055806b4d0 0x7f055806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:16.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.334+0000 7f0578c00640 1 -- 172.21.15.6:0/2900420706 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5115+0+0 (secure 0 0 0) 0x7f056c0ac260 con 0x7f057c0af1a0 2024-03-04T09:31:16.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.335+0000 7f0573400640 1 --2- 172.21.15.6:0/2900420706 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f055806b4d0 0x7f055806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:16.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.335+0000 7f0573400640 1 --2- 172.21.15.6:0/2900420706 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f055806b4d0 0x7f055806d990 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f057c0ccb00 tx=0x7f0564009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:16.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.339+0000 7f0578c00640 1 -- 172.21.15.6:0/2900420706 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f056c07b290 con 0x7f057c0af1a0 2024-03-04T09:31:16.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.547+0000 7f0582c00640 1 -- 172.21.15.6:0/2900420706 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f057c04f3a0 con 0x7f057c0af1a0 2024-03-04T09:31:16.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.583+0000 7f0578c00640 1 -- 172.21.15.6:0/2900420706 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v103) v1 ==== 97+0+0 (secure 0 0 0) 0x7f056c07ab90 con 0x7f057c0af1a0 2024-03-04T09:31:16.580 INFO:teuthology.orchestra.run.smithi006.stderr: removed all blocklist entries 2024-03-04T09:31:16.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.587+0000 7f0582c00640 1 -- 172.21.15.6:0/2900420706 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f055806b4d0 msgr2=0x7f055806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:16.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.587+0000 7f0582c00640 1 --2- 172.21.15.6:0/2900420706 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f055806b4d0 0x7f055806d990 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f057c0ccb00 tx=0x7f0564009040 comp rx=0 tx=0).stop 2024-03-04T09:31:16.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.587+0000 7f0582c00640 1 -- 172.21.15.6:0/2900420706 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f057c0af1a0 msgr2=0x7f057c09ab60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:16.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.587+0000 7f0582c00640 1 --2- 172.21.15.6:0/2900420706 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f057c0af1a0 0x7f057c09ab60 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7f056c0098c0 tx=0x7f056c0046c0 comp rx=0 tx=0).stop 2024-03-04T09:31:16.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.587+0000 7f0582c00640 1 -- 172.21.15.6:0/2900420706 shutdown_connections 2024-03-04T09:31:16.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.587+0000 7f0582c00640 1 --2- 172.21.15.6:0/2900420706 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f055806b4d0 0x7f055806d990 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:16.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.587+0000 7f0582c00640 1 --2- 172.21.15.6:0/2900420706 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f057c0cdc20 0x7f057c1f44d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:16.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.587+0000 7f0582c00640 1 --2- 172.21.15.6:0/2900420706 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f057c0af580 0x7f057c1f7bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:16.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.587+0000 7f0582c00640 1 --2- 172.21.15.6:0/2900420706 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f057c0af1a0 0x7f057c09ab60 unknown :-1 s=CLOSED pgs=1220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:16.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.587+0000 7f0582c00640 1 -- 172.21.15.6:0/2900420706 >> 172.21.15.6:0/2900420706 conn(0x7f057c06afb0 msgr2=0x7f057c0cd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:16.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.587+0000 7f0582c00640 1 -- 172.21.15.6:0/2900420706 shutdown_connections 2024-03-04T09:31:16.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.587+0000 7f0582c00640 1 -- 172.21.15.6:0/2900420706 wait complete. 2024-03-04T09:31:16.606 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-04T09:31:16.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.819+0000 7f5856200640 1 Processor -- start 2024-03-04T09:31:16.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.820+0000 7f5856200640 1 -- start start 2024-03-04T09:31:16.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.821+0000 7f5856200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f585006ef10 0x7f58500cea80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:16.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.821+0000 7f5856200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58500cf1b0 con 0x7f5850066d00 2024-03-04T09:31:16.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.821+0000 7f5856200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f585006d420 con 0x7f585006c8b0 2024-03-04T09:31:16.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.821+0000 7f5856200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f585015e470 con 0x7f585006ef10 2024-03-04T09:31:16.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.822+0000 7f584fe00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f5850066d00 0x7f58500ce370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:48412/0 (socket says 172.21.15.6:48412) 2024-03-04T09:31:16.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.822+0000 7f584fe00640 1 -- 172.21.15.6:0/666323423 learned_addr learned my addr 172.21.15.6:0/666323423 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:31:16.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.822+0000 7f5855800640 1 --2- 172.21.15.6:0/666323423 >> v2:172.21.15.6:3300/0 conn(0x7f585006ef10 0x7f58500cea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:16.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.822+0000 7f584f400640 1 -- 172.21.15.6:0/666323423 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1092631498 0 0) 0x7f58500cf1b0 con 0x7f5850066d00 2024-03-04T09:31:16.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.822+0000 7f584f400640 1 -- 172.21.15.6:0/666323423 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5834003520 con 0x7f5850066d00 2024-03-04T09:31:16.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.822+0000 7f5855800640 1 -- 172.21.15.6:0/666323423 >> v1:172.21.15.146:6789/0 conn(0x7f585006c8b0 legacy=0x7f585015ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:16.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.823+0000 7f5855800640 1 -- 172.21.15.6:0/666323423 >> v1:172.21.15.6:6789/0 conn(0x7f5850066d00 legacy=0x7f58500ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:16.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.823+0000 7f5855800640 1 -- 172.21.15.6:0/666323423 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f585015e690 con 0x7f585006ef10 2024-03-04T09:31:16.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.824+0000 7f5855800640 1 --2- 172.21.15.6:0/666323423 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f585006ef10 0x7f58500cea80 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7f58480097a0 tx=0x7f584802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f13c2b4c85907332 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:16.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.824+0000 7f584f400640 1 -- 172.21.15.6:0/666323423 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5848034070 con 0x7f585006ef10 2024-03-04T09:31:16.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.824+0000 7f584f400640 1 -- 172.21.15.6:0/666323423 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5848030610 con 0x7f585006ef10 2024-03-04T09:31:16.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.824+0000 7f584f400640 1 -- 172.21.15.6:0/666323423 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58480309c0 con 0x7f585006ef10 2024-03-04T09:31:16.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.825+0000 7f5856200640 1 -- 172.21.15.6:0/666323423 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f585006ef10 msgr2=0x7f58500cea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:16.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.825+0000 7f5856200640 1 --2- 172.21.15.6:0/666323423 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f585006ef10 0x7f58500cea80 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7f58480097a0 tx=0x7f584802b960 comp rx=0 tx=0).stop 2024-03-04T09:31:16.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.825+0000 7f5856200640 1 -- 172.21.15.6:0/666323423 shutdown_connections 2024-03-04T09:31:16.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.825+0000 7f5856200640 1 --2- 172.21.15.6:0/666323423 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f585006ef10 0x7f58500cea80 unknown :-1 s=CLOSED pgs=1222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:16.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.825+0000 7f5856200640 1 -- 172.21.15.6:0/666323423 >> 172.21.15.6:0/666323423 conn(0x7f58500ccf70 msgr2=0x7f58500cd350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:16.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.825+0000 7f5856200640 1 -- 172.21.15.6:0/666323423 shutdown_connections 2024-03-04T09:31:16.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.825+0000 7f5856200640 1 -- 172.21.15.6:0/666323423 wait complete. 2024-03-04T09:31:16.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.827+0000 7f5856200640 1 Processor -- start 2024-03-04T09:31:16.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.827+0000 7f5856200640 1 -- start start 2024-03-04T09:31:16.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.828+0000 7f5856200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5850066d00 0x7f585015bc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:16.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.828+0000 7f5856200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f585006c8b0 0x7f5850145960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:16.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.828+0000 7f5856200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f585006ef10 0x7f585015f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:16.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.828+0000 7f5856200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f585006b400 con 0x7f585006ef10 2024-03-04T09:31:16.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.828+0000 7f5856200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f585006b280 con 0x7f585006c8b0 2024-03-04T09:31:16.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.828+0000 7f5856200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f585006b580 con 0x7f5850066d00 2024-03-04T09:31:16.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.828+0000 7f5855800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f585006ef10 0x7f585015f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:16.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.828+0000 7f5855800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f585006ef10 0x7f585015f760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55138/0 (socket says 172.21.15.6:55138) 2024-03-04T09:31:16.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.828+0000 7f5855800640 1 -- 172.21.15.6:0/4222827088 learned_addr learned my addr 172.21.15.6:0/4222827088 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:31:16.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.828+0000 7f584fe00640 1 --2- 172.21.15.6:0/4222827088 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f585006c8b0 0x7f5850145960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:16.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.828+0000 7f5854e00640 1 --2- 172.21.15.6:0/4222827088 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5850066d00 0x7f585015bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:16.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.829+0000 7f5855800640 1 -- 172.21.15.6:0/4222827088 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5850066d00 msgr2=0x7f585015bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:16.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.829+0000 7f5855800640 1 --2- 172.21.15.6:0/4222827088 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5850066d00 0x7f585015bc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:16.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.829+0000 7f5855800640 1 -- 172.21.15.6:0/4222827088 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f585006c8b0 msgr2=0x7f5850145960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:16.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.829+0000 7f5855800640 1 --2- 172.21.15.6:0/4222827088 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f585006c8b0 0x7f5850145960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:16.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.829+0000 7f5855800640 1 -- 172.21.15.6:0/4222827088 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f585015fca0 con 0x7f585006ef10 2024-03-04T09:31:16.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.829+0000 7f5855800640 1 --2- 172.21.15.6:0/4222827088 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f585006ef10 0x7f585015f760 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7f58480097a0 tx=0x7f58480046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:16.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.829+0000 7f584cc00640 1 -- 172.21.15.6:0/4222827088 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5848047070 con 0x7f585006ef10 2024-03-04T09:31:16.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.829+0000 7f584cc00640 1 -- 172.21.15.6:0/4222827088 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5848030410 con 0x7f585006ef10 2024-03-04T09:31:16.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.829+0000 7f584cc00640 1 -- 172.21.15.6:0/4222827088 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5848034040 con 0x7f585006ef10 2024-03-04T09:31:16.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.829+0000 7f5856200640 1 -- 172.21.15.6:0/4222827088 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f585015ff50 con 0x7f585006ef10 2024-03-04T09:31:16.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.829+0000 7f5856200640 1 -- 172.21.15.6:0/4222827088 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5850160510 con 0x7f585006ef10 2024-03-04T09:31:16.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.831+0000 7f5856200640 1 -- 172.21.15.6:0/4222827088 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f585004f330 con 0x7f585006ef10 2024-03-04T09:31:16.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.832+0000 7f584cc00640 1 -- 172.21.15.6:0/4222827088 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f584804a070 con 0x7f585006ef10 2024-03-04T09:31:16.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.832+0000 7f584cc00640 1 --2- 172.21.15.6:0/4222827088 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f582c06b500 0x7f582c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:16.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.833+0000 7f584cc00640 1 -- 172.21.15.6:0/4222827088 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f58480ab830 con 0x7f585006ef10 2024-03-04T09:31:16.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.833+0000 7f5854e00640 1 --2- 172.21.15.6:0/4222827088 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f582c06b500 0x7f582c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:16.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.834+0000 7f5854e00640 1 --2- 172.21.15.6:0/4222827088 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f582c06b500 0x7f582c06d9c0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f5840001d40 tx=0x7f5840009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:16.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:16.837+0000 7f584cc00640 1 -- 172.21.15.6:0/4222827088 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f584807a960 con 0x7f585006ef10 2024-03-04T09:31:17.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.113+0000 7f5856200640 1 -- 172.21.15.6:0/4222827088 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f58500cdde0 con 0x7f585006ef10 2024-03-04T09:31:17.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.114+0000 7f584cc00640 1 -- 172.21.15.6:0/4222827088 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v15) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f584807a260 con 0x7f585006ef10 2024-03-04T09:31:17.111 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:31:17.111 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-04T09:31:17.111 INFO:teuthology.orchestra.run.smithi006.stdout: "auth_dump": [ 2024-03-04T09:31:17.111 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:31:17.111 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "mds.a", 2024-03-04T09:31:17.111 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQA8k+VlHCzFIBAAnKYlmCMDw45LqnSjxnZY5Q==", 2024-03-04T09:31:17.111 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:31:17.111 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-04T09:31:17.111 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow *", 2024-03-04T09:31:17.111 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow *", 2024-03-04T09:31:17.112 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-04T09:31:17.112 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:31:17.112 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:31:17.112 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:31:17.112 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "mds.b", 2024-03-04T09:31:17.112 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQA9k+VlNY7fCBAAX3z8X72/6LkQD6mwGYT69g==", 2024-03-04T09:31:17.112 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:31:17.112 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-04T09:31:17.112 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow *", 2024-03-04T09:31:17.112 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow *", 2024-03-04T09:31:17.112 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-04T09:31:17.112 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:31:17.112 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:31:17.112 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:31:17.113 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "mds.c", 2024-03-04T09:31:17.113 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQA8k+Vl0pjOMhAA7WcTw8awv4EAsrpP4yncNA==", 2024-03-04T09:31:17.113 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:31:17.113 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-04T09:31:17.113 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow *", 2024-03-04T09:31:17.113 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow *", 2024-03-04T09:31:17.113 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-04T09:31:17.113 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:31:17.113 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:31:17.113 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:31:17.113 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "mds.d", 2024-03-04T09:31:17.113 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQA9k+VlbladGRAABNgUcaqTVYuJfCqy5c/G0g==", 2024-03-04T09:31:17.113 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:31:17.114 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-04T09:31:17.114 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow *", 2024-03-04T09:31:17.114 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow *", 2024-03-04T09:31:17.114 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-04T09:31:17.114 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:31:17.114 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:31:17.114 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:31:17.114 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.0", 2024-03-04T09:31:17.114 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQBCk+VlyjMMExAAqb2FJ9n5PWehfx/6PB19ig==", 2024-03-04T09:31:17.114 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:31:17.114 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-04T09:31:17.114 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-04T09:31:17.114 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-04T09:31:17.114 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:31:17.115 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:31:17.115 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:31:17.115 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.1", 2024-03-04T09:31:17.115 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQBFk+VlXD3cLBAAGclSivcYWDslqN8MLlj7eQ==", 2024-03-04T09:31:17.115 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:31:17.115 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-04T09:31:17.115 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-04T09:31:17.115 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-04T09:31:17.115 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:31:17.115 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:31:17.115 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:31:17.115 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.2", 2024-03-04T09:31:17.115 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQBJk+Vl7d9nCxAAnb3W4EI3Yf3XY8ht3VnKEQ==", 2024-03-04T09:31:17.116 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:31:17.116 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-04T09:31:17.116 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-04T09:31:17.116 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-04T09:31:17.116 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:31:17.116 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:31:17.116 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:31:17.116 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.3", 2024-03-04T09:31:17.116 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQBMk+Vlv+Y1NBAAiRYSUqov2i2sj3iInfpHvA==", 2024-03-04T09:31:17.116 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:31:17.116 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-04T09:31:17.116 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-04T09:31:17.116 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-04T09:31:17.116 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:31:17.117 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:31:17.117 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:31:17.117 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.4", 2024-03-04T09:31:17.117 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQBUk+Vl1OlAARAAxMV4GUGS+RJ6XlrRhrVKcg==", 2024-03-04T09:31:17.117 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:31:17.117 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-04T09:31:17.117 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-04T09:31:17.117 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-04T09:31:17.117 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:31:17.117 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:31:17.117 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:31:17.117 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.5", 2024-03-04T09:31:17.117 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQBXk+Vl1lKLGhAAHlsiRr+1+OSRWhkB+weIJg==", 2024-03-04T09:31:17.117 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:31:17.117 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-04T09:31:17.118 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-04T09:31:17.118 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-04T09:31:17.118 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:31:17.118 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:31:17.118 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:31:17.118 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.6", 2024-03-04T09:31:17.118 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQBak+VlwB+GJBAA3f9KUz+zZlQuUR/ywQfkDA==", 2024-03-04T09:31:17.118 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:31:17.118 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-04T09:31:17.118 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-04T09:31:17.118 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-04T09:31:17.120 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:31:17.120 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:31:17.120 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:31:17.120 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.7", 2024-03-04T09:31:17.120 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQBdk+Vle4o8LhAAko+iH5VqqCsfWkNfKZjGVA==", 2024-03-04T09:31:17.120 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:31:17.120 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-04T09:31:17.120 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-04T09:31:17.120 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-04T09:31:17.121 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:31:17.121 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:31:17.121 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:31:17.121 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "client.0", 2024-03-04T09:31:17.121 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQA9k+Vl4L+rJxAACaDbQ4/Bybp5hAZAnJzlVw==", 2024-03-04T09:31:17.121 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:31:17.121 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-04T09:31:17.121 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow r", 2024-03-04T09:31:17.121 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-04T09:31:17.121 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:31:17.121 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:31:17.121 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:31:17.121 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "client.1", 2024-03-04T09:31:17.121 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQA+k+VlGjkOAhAAJNzB8+tQl3mqCnTgMn9EVw==", 2024-03-04T09:31:17.121 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:31:17.122 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-04T09:31:17.122 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow r", 2024-03-04T09:31:17.122 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow rw", 2024-03-04T09:31:17.122 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow rwx" 2024-03-04T09:31:17.122 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:31:17.122 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:31:17.122 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:31:17.122 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "client.2", 2024-03-04T09:31:17.122 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQA9k+VlUc8DMxAA8pGUtIh+BO6UKrrYnCqVgQ==", 2024-03-04T09:31:17.122 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:31:17.122 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-04T09:31:17.122 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow r", 2024-03-04T09:31:17.122 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow rw", 2024-03-04T09:31:17.122 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow rwx" 2024-03-04T09:31:17.122 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:31:17.123 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:31:17.123 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:31:17.123 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "client.3", 2024-03-04T09:31:17.123 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQA+k+VlzMFKDBAAGr713I2hcmGMuHMIjNknYA==", 2024-03-04T09:31:17.123 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:31:17.123 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-04T09:31:17.123 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow r", 2024-03-04T09:31:17.123 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow rw", 2024-03-04T09:31:17.123 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow rwx" 2024-03-04T09:31:17.123 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:31:17.123 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:31:17.123 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:31:17.123 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "client.admin", 2024-03-04T09:31:17.123 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQA7k+Vllb7NFhAAmnFaNFKBcEri+iJAPnG5Gg==", 2024-03-04T09:31:17.123 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:31:17.123 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow *", 2024-03-04T09:31:17.124 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow *", 2024-03-04T09:31:17.124 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow *", 2024-03-04T09:31:17.124 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-04T09:31:17.124 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:31:17.124 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:31:17.124 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:31:17.124 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "mgr.x", 2024-03-04T09:31:17.124 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQA8k+VlPFatFBAA/o3Ps/uJrcg1738HOGy/FA==", 2024-03-04T09:31:17.124 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:31:17.124 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow *", 2024-03-04T09:31:17.124 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile mgr", 2024-03-04T09:31:17.124 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-04T09:31:17.124 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:31:17.124 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:31:17.125 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:31:17.125 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "mgr.y", 2024-03-04T09:31:17.125 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQA8k+VlEA2ACBAAXofDj3gmqItVTJ1pYLqZyQ==", 2024-03-04T09:31:17.125 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:31:17.125 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow *", 2024-03-04T09:31:17.125 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile mgr", 2024-03-04T09:31:17.125 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-04T09:31:17.125 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:31:17.125 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:31:17.125 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-04T09:31:17.125 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-04T09:31:17.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.116+0000 7f5856200640 1 -- 172.21.15.6:0/4222827088 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f582c06b500 msgr2=0x7f582c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:17.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.116+0000 7f5856200640 1 --2- 172.21.15.6:0/4222827088 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f582c06b500 0x7f582c06d9c0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f5840001d40 tx=0x7f5840009040 comp rx=0 tx=0).stop 2024-03-04T09:31:17.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.116+0000 7f5856200640 1 -- 172.21.15.6:0/4222827088 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f585006ef10 msgr2=0x7f585015f760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:17.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.116+0000 7f5856200640 1 --2- 172.21.15.6:0/4222827088 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f585006ef10 0x7f585015f760 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7f58480097a0 tx=0x7f58480046c0 comp rx=0 tx=0).stop 2024-03-04T09:31:17.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.117+0000 7f5856200640 1 -- 172.21.15.6:0/4222827088 shutdown_connections 2024-03-04T09:31:17.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.117+0000 7f5856200640 1 --2- 172.21.15.6:0/4222827088 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f585006ef10 0x7f585015f760 unknown :-1 s=CLOSED pgs=1223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:17.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.117+0000 7f5856200640 1 --2- 172.21.15.6:0/4222827088 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f585006c8b0 0x7f5850145960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:17.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.117+0000 7f5856200640 1 --2- 172.21.15.6:0/4222827088 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f582c06b500 0x7f582c06d9c0 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:17.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.117+0000 7f5856200640 1 --2- 172.21.15.6:0/4222827088 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5850066d00 0x7f585015bc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:17.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.117+0000 7f5856200640 1 -- 172.21.15.6:0/4222827088 >> 172.21.15.6:0/4222827088 conn(0x7f58500ccf70 msgr2=0x7f5850161800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:17.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.117+0000 7f5856200640 1 -- 172.21.15.6:0/4222827088 shutdown_connections 2024-03-04T09:31:17.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.117+0000 7f5856200640 1 -- 172.21.15.6:0/4222827088 wait complete. 2024-03-04T09:31:17.134 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T09:31:17.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.356+0000 7f8421e00640 1 Processor -- start 2024-03-04T09:31:17.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.356+0000 7f8421e00640 1 -- start start 2024-03-04T09:31:17.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.358+0000 7f8421e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f841c0af1a0 0x7f841c15c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:17.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.358+0000 7f8421e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f841c0af580 0x7f841c0b4c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:17.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.358+0000 7f8421e00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f841c15cc40 con 0x7f841c0cdc20 2024-03-04T09:31:17.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.358+0000 7f8421e00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f841c15cf20 con 0x7f841c0af580 2024-03-04T09:31:17.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.358+0000 7f8421e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f841c15d200 con 0x7f841c0af1a0 2024-03-04T09:31:17.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.358+0000 7f841aa00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f841c0af1a0 0x7f841c15c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:17.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.358+0000 7f841aa00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f841c0af1a0 0x7f841c15c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.6:43620/0 (socket says 172.21.15.6:43620) 2024-03-04T09:31:17.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.359+0000 7f841aa00640 1 -- 172.21.15.6:0/1748268570 learned_addr learned my addr 172.21.15.6:0/1748268570 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:31:17.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.359+0000 7f841a000640 1 --2- 172.21.15.6:0/1748268570 >> v2:172.21.15.146:3300/0 conn(0x7f841c0af580 0x7f841c0b4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:17.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.359+0000 7f841aa00640 1 -- 172.21.15.6:0/1748268570 >> v2:172.21.15.146:3300/0 conn(0x7f841c0af580 msgr2=0x7f841c0b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:17.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.359+0000 7f841aa00640 1 --2- 172.21.15.6:0/1748268570 >> v2:172.21.15.146:3300/0 conn(0x7f841c0af580 0x7f841c0b4c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:17.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.359+0000 7f841aa00640 1 -- 172.21.15.6:0/1748268570 >> v1:172.21.15.146:6789/0 conn(0x7f841c0cdc20 legacy=0x7f841c0b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:17.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.359+0000 7f841aa00640 1 -- 172.21.15.6:0/1748268570 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f841c15d4e0 con 0x7f841c0af1a0 2024-03-04T09:31:17.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.361+0000 7f841aa00640 1 --2- 172.21.15.6:0/1748268570 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f841c0af1a0 0x7f841c15c600 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f840c009870 tx=0x7f840c02b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2a8c4cb697d03f32 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:17.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.362+0000 7f8419600640 1 -- 172.21.15.6:0/1748268570 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f840c035020 con 0x7f841c0af1a0 2024-03-04T09:31:17.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.362+0000 7f8419600640 1 -- 172.21.15.6:0/1748268570 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f840c031640 con 0x7f841c0af1a0 2024-03-04T09:31:17.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.362+0000 7f8419600640 1 -- 172.21.15.6:0/1748268570 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f840c02d040 con 0x7f841c0af1a0 2024-03-04T09:31:17.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.362+0000 7f8421e00640 1 -- 172.21.15.6:0/1748268570 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f841c0af1a0 msgr2=0x7f841c15c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:17.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.362+0000 7f8421e00640 1 --2- 172.21.15.6:0/1748268570 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f841c0af1a0 0x7f841c15c600 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f840c009870 tx=0x7f840c02b8f0 comp rx=0 tx=0).stop 2024-03-04T09:31:17.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.363+0000 7f8421e00640 1 -- 172.21.15.6:0/1748268570 shutdown_connections 2024-03-04T09:31:17.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.363+0000 7f8421e00640 1 --2- 172.21.15.6:0/1748268570 >> v2:172.21.15.146:3300/0 conn(0x7f841c0af580 0x7f841c0b4c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:17.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.363+0000 7f8421e00640 1 --2- 172.21.15.6:0/1748268570 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f841c0af1a0 0x7f841c15c600 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:17.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.363+0000 7f8421e00640 1 -- 172.21.15.6:0/1748268570 >> 172.21.15.6:0/1748268570 conn(0x7f841c06afb0 msgr2=0x7f841c06b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:17.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.363+0000 7f8421e00640 1 -- 172.21.15.6:0/1748268570 shutdown_connections 2024-03-04T09:31:17.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.363+0000 7f8421e00640 1 -- 172.21.15.6:0/1748268570 wait complete. 2024-03-04T09:31:17.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.364+0000 7f8421e00640 1 Processor -- start 2024-03-04T09:31:17.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.364+0000 7f8421e00640 1 -- start start 2024-03-04T09:31:17.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.365+0000 7f8421e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f841c0af1a0 0x7f841c09d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:17.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.365+0000 7f8421e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f841c0af580 0x7f841c14f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:17.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.365+0000 7f8421e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f841c0cdc20 0x7f841c09aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:17.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.365+0000 7f8421e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f841c0b3660 con 0x7f841c0af1a0 2024-03-04T09:31:17.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.365+0000 7f8421e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f841c0b34e0 con 0x7f841c0af580 2024-03-04T09:31:17.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.365+0000 7f8421e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f841c0b37e0 con 0x7f841c0cdc20 2024-03-04T09:31:17.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.366+0000 7f841aa00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f841c0af1a0 0x7f841c09d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:17.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.366+0000 7f841aa00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f841c0af1a0 0x7f841c09d980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55140/0 (socket says 172.21.15.6:55140) 2024-03-04T09:31:17.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.366+0000 7f841aa00640 1 -- 172.21.15.6:0/2699116041 learned_addr learned my addr 172.21.15.6:0/2699116041 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:31:17.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.366+0000 7f841a000640 1 --2- 172.21.15.6:0/2699116041 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f841c0af580 0x7f841c14f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:17.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.366+0000 7f841b400640 1 --2- 172.21.15.6:0/2699116041 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f841c0cdc20 0x7f841c09aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:17.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.366+0000 7f841aa00640 1 -- 172.21.15.6:0/2699116041 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f841c0cdc20 msgr2=0x7f841c09aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:17.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.366+0000 7f841aa00640 1 --2- 172.21.15.6:0/2699116041 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f841c0cdc20 0x7f841c09aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:17.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.366+0000 7f841aa00640 1 -- 172.21.15.6:0/2699116041 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f841c0af580 msgr2=0x7f841c14f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:17.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.366+0000 7f841aa00640 1 --2- 172.21.15.6:0/2699116041 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f841c0af580 0x7f841c14f7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:17.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.366+0000 7f841aa00640 1 -- 172.21.15.6:0/2699116041 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f841c09af50 con 0x7f841c0af1a0 2024-03-04T09:31:17.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.367+0000 7f841aa00640 1 --2- 172.21.15.6:0/2699116041 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f841c0af1a0 0x7f841c09d980 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7f840c000c00 tx=0x7f840c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:17.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.367+0000 7f8413400640 1 -- 172.21.15.6:0/2699116041 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f840c040070 con 0x7f841c0af1a0 2024-03-04T09:31:17.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.367+0000 7f8413400640 1 -- 172.21.15.6:0/2699116041 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f840c0399f0 con 0x7f841c0af1a0 2024-03-04T09:31:17.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.367+0000 7f8413400640 1 -- 172.21.15.6:0/2699116041 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f840c035040 con 0x7f841c0af1a0 2024-03-04T09:31:17.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.367+0000 7f8421e00640 1 -- 172.21.15.6:0/2699116041 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f841c0c99c0 con 0x7f841c0af1a0 2024-03-04T09:31:17.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.367+0000 7f8421e00640 1 -- 172.21.15.6:0/2699116041 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f841c0c9f80 con 0x7f841c0af1a0 2024-03-04T09:31:17.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.368+0000 7f8421e00640 1 -- 172.21.15.6:0/2699116041 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f841c069510 con 0x7f841c0af1a0 2024-03-04T09:31:17.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.369+0000 7f8413400640 1 -- 172.21.15.6:0/2699116041 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f840c04b060 con 0x7f841c0af1a0 2024-03-04T09:31:17.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.370+0000 7f8413400640 1 --2- 172.21.15.6:0/2699116041 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f83fc06b4d0 0x7f83fc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:17.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.370+0000 7f8413400640 1 -- 172.21.15.6:0/2699116041 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f840c0ab760 con 0x7f841c0af1a0 2024-03-04T09:31:17.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.371+0000 7f841a000640 1 --2- 172.21.15.6:0/2699116041 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f83fc06b4d0 0x7f83fc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:17.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.372+0000 7f841a000640 1 --2- 172.21.15.6:0/2699116041 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f83fc06b4d0 0x7f83fc06d990 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f841c0aec00 tx=0x7f8408008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:17.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.375+0000 7f8413400640 1 -- 172.21.15.6:0/2699116041 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f840c07a920 con 0x7f841c0af1a0 2024-03-04T09:31:17.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.574+0000 7f8421e00640 1 -- 172.21.15.6:0/2699116041 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f841c0cf230 con 0x7f841c0af1a0 2024-03-04T09:31:17.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.581+0000 7f8413400640 1 -- 172.21.15.6:0/2699116041 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v105) v1 ==== 74+0+12845 (secure 0 0 0) 0x7f840c07a220 con 0x7f841c0af1a0 2024-03-04T09:31:17.579 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:31:17.579 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":105,"fsid":"edce0578-5ae4-4402-b45a-552bdef84d7c","created":"2024-03-04T09:24:54.523780+0000","modified":"2024-03-04T09:31:16.596160+0000","last_up_change":"2024-03-04T09:25:07.089049+0000","last_in_change":"2024-03-04T09:24:57.982077+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T09:25:07.935198+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a96574f-81f1-4bc6-8463-757520b84a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6825","nonce":1364449401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6827","nonce":1364449401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6831","nonce":1364449401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6829","nonce":1364449401}]},"public_addr":"172.21.15.6:6825/1364449401","cluster_addr":"172.21.15.6:6827/1364449401","heartbeat_back_addr":"172.21.15.6:6831/1364449401","heartbeat_front_addr":"172.21.15.6:6829/1364449401","state":["exists","up"]},{"osd":1,"uuid":"7d79b9c1-6b87-476a-bc36-7db5ee64cd90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6809","nonce":2494103078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6811","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6813","nonce":2494103078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6821","nonce":2494103078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6817","nonce":2494103078}]},"public_addr":"172.21.15.6:6809/2494103078","cluster_addr":"172.21.15.6:6813/2494103078","heartbeat_back_addr":"172.21.15.6:6821/2494103078","heartbeat_front_addr":"172.21.15.6:6817/2494103078","state":["exists","up"]},{"osd":2,"uuid":"492331b9-4ee9-40b6-bacd-85d44cd6c45a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6812","nonce":1500066681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6816","nonce":1500066681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6824","nonce":1500066681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6820","nonce":1500066681}]},"public_addr":"172.21.15.6:6812/1500066681","cluster_addr":"172.21.15.6:6816/1500066681","heartbeat_back_addr":"172.21.15.6:6824/1500066681","heartbeat_front_addr":"172.21.15.6:6820/1500066681","state":["exists","up"]},{"osd":3,"uuid":"21bb2efe-9ba7-4bdc-bc31-180cd1102e21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6801","nonce":1558806123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6803","nonce":1558806123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6807","nonce":1558806123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6805","nonce":1558806123}]},"public_addr":"172.21.15.6:6801/1558806123","cluster_addr":"172.21.15.6:6803/1558806123","heartbeat_back_addr":"172.21.15.6:6807/1558806123","heartbeat_front_addr":"172.21.15.6:6805/1558806123","state":["exists","up"]},{"osd":4,"uuid":"6e2c4b71-19d9-4737-83d9-558705d8644a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6801","nonce":1526135780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6803","nonce":1526135780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6807","nonce":1526135780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6805","nonce":1526135780}]},"public_addr":"172.21.15.146:6801/1526135780","cluster_addr":"172.21.15.146:6803/1526135780","heartbeat_back_addr":"172.21.15.146:6807/1526135780","heartbeat_front_addr":"172.21.15.146:6805/1526135780","state":["exists","up"]},{"osd":5,"uuid":"fb69c525-f1be-4b7d-ab32-6c29b0d6fddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6809","nonce":460262507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6811","nonce":460262507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6815","nonce":460262507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6813","nonce":460262507}]},"public_addr":"172.21.15.146:6809/460262507","cluster_addr":"172.21.15.146:6811/460262507","heartbeat_back_addr":"172.21.15.146:6815/460262507","heartbeat_front_addr":"172.21.15.146:6813/460262507","state":["exists","up"]},{"osd":6,"uuid":"c32e7fed-370d-4401-8ce0-6a4335b0d168","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6823","nonce":1392897493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6827","nonce":1392897493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6831","nonce":1392897493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6829","nonce":1392897493}]},"public_addr":"172.21.15.146:6823/1392897493","cluster_addr":"172.21.15.146:6827/1392897493","heartbeat_back_addr":"172.21.15.146:6831/1392897493","heartbeat_front_addr":"172.21.15.146:6829/1392897493","state":["exists","up"]},{"osd":7,"uuid":"75a79b44-fa5f-426d-a7d3-71f21f78835c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6817","nonce":3268323156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6819","nonce":3268323156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6825","nonce":3268323156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6822","nonce":3268323156}]},"public_addr":"172.21.15.146:6817/3268323156","cluster_addr":"172.21.15.146:6819/3268323156","heartbeat_back_addr":"172.21.15.146:6825/3268323156","heartbeat_front_addr":"172.21.15.146:6822/3268323156","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.310437+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.295850+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.294674+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.526855+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.300023+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.595160+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.346046+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.558741+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T09:31:17.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.584+0000 7f8421e00640 1 -- 172.21.15.6:0/2699116041 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f83fc06b4d0 msgr2=0x7f83fc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:17.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.584+0000 7f8421e00640 1 --2- 172.21.15.6:0/2699116041 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f83fc06b4d0 0x7f83fc06d990 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f841c0aec00 tx=0x7f8408008040 comp rx=0 tx=0).stop 2024-03-04T09:31:17.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.584+0000 7f8421e00640 1 -- 172.21.15.6:0/2699116041 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f841c0af1a0 msgr2=0x7f841c09d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:17.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.584+0000 7f8421e00640 1 --2- 172.21.15.6:0/2699116041 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f841c0af1a0 0x7f841c09d980 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7f840c000c00 tx=0x7f840c0046c0 comp rx=0 tx=0).stop 2024-03-04T09:31:17.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.584+0000 7f8421e00640 1 -- 172.21.15.6:0/2699116041 shutdown_connections 2024-03-04T09:31:17.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.584+0000 7f8421e00640 1 --2- 172.21.15.6:0/2699116041 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f83fc06b4d0 0x7f83fc06d990 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:17.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.584+0000 7f8421e00640 1 --2- 172.21.15.6:0/2699116041 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f841c0cdc20 0x7f841c09aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:17.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.584+0000 7f8421e00640 1 --2- 172.21.15.6:0/2699116041 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f841c0af580 0x7f841c14f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:17.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.584+0000 7f8421e00640 1 --2- 172.21.15.6:0/2699116041 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f841c0af1a0 0x7f841c09d980 unknown :-1 s=CLOSED pgs=1224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:17.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.584+0000 7f8421e00640 1 -- 172.21.15.6:0/2699116041 >> 172.21.15.6:0/2699116041 conn(0x7f841c06afb0 msgr2=0x7f841c15c300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:17.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.584+0000 7f8421e00640 1 -- 172.21.15.6:0/2699116041 shutdown_connections 2024-03-04T09:31:17.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.585+0000 7f8421e00640 1 -- 172.21.15.6:0/2699116041 wait complete. 2024-03-04T09:31:17.601 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T09:25:07.935198+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-04T09:31:17.601 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-04T09:31:17.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.820+0000 7f6a3a400640 1 Processor -- start 2024-03-04T09:31:17.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.821+0000 7f6a3a400640 1 -- start start 2024-03-04T09:31:17.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.822+0000 7f6a3a400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f6a3415afa0 0x7f6a340cf3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:17.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.822+0000 7f6a3a400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f6a340cd1e0 0x7f6a340b3ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:17.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.822+0000 7f6a3a400640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a34071c60 con 0x7f6a34070ce0 2024-03-04T09:31:17.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.822+0000 7f6a3a400640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a340cfa10 con 0x7f6a3415afa0 2024-03-04T09:31:17.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.822+0000 7f6a3a400640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a34071f40 con 0x7f6a340cd1e0 2024-03-04T09:31:17.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.822+0000 7f6a33400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f6a3415afa0 0x7f6a340cf3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:17.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.822+0000 7f6a33400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f6a3415afa0 0x7f6a340cf3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55148/0 (socket says 172.21.15.6:55148) 2024-03-04T09:31:17.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.822+0000 7f6a33400640 1 -- 172.21.15.6:0/859758724 learned_addr learned my addr 172.21.15.6:0/859758724 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:31:17.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.823+0000 7f6a33400640 1 -- 172.21.15.6:0/859758724 >> v2:172.21.15.146:3300/0 conn(0x7f6a340cd1e0 msgr2=0x7f6a340b3ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:17.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.823+0000 7f6a33400640 1 --2- 172.21.15.6:0/859758724 >> v2:172.21.15.146:3300/0 conn(0x7f6a340cd1e0 0x7f6a340b3ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:17.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.823+0000 7f6a33400640 1 -- 172.21.15.6:0/859758724 >> v1:172.21.15.146:6789/0 conn(0x7f6a34070ce0 legacy=0x7f6a340ccad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:17.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.823+0000 7f6a33400640 1 -- 172.21.15.6:0/859758724 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a34072220 con 0x7f6a3415afa0 2024-03-04T09:31:17.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.824+0000 7f6a33400640 1 --2- 172.21.15.6:0/859758724 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6a3415afa0 0x7f6a340cf3a0 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7f6a34070b90 tx=0x7f6a2402b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c126737a824f3a4 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:17.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.824+0000 7f6a32a00640 1 -- 172.21.15.6:0/859758724 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a24035020 con 0x7f6a3415afa0 2024-03-04T09:31:17.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.824+0000 7f6a32a00640 1 -- 172.21.15.6:0/859758724 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a24031800 con 0x7f6a3415afa0 2024-03-04T09:31:17.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.824+0000 7f6a32a00640 1 -- 172.21.15.6:0/859758724 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a24031bd0 con 0x7f6a3415afa0 2024-03-04T09:31:17.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.825+0000 7f6a3a400640 1 -- 172.21.15.6:0/859758724 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6a3415afa0 msgr2=0x7f6a340cf3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:17.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.825+0000 7f6a3a400640 1 --2- 172.21.15.6:0/859758724 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6a3415afa0 0x7f6a340cf3a0 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7f6a34070b90 tx=0x7f6a2402b8a0 comp rx=0 tx=0).stop 2024-03-04T09:31:17.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.825+0000 7f6a3a400640 1 -- 172.21.15.6:0/859758724 shutdown_connections 2024-03-04T09:31:17.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.825+0000 7f6a3a400640 1 --2- 172.21.15.6:0/859758724 >> v2:172.21.15.146:3300/0 conn(0x7f6a340cd1e0 0x7f6a340b3ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:17.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.825+0000 7f6a3a400640 1 --2- 172.21.15.6:0/859758724 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6a3415afa0 0x7f6a340cf3a0 unknown :-1 s=CLOSED pgs=1225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:17.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.825+0000 7f6a3a400640 1 -- 172.21.15.6:0/859758724 >> 172.21.15.6:0/859758724 conn(0x7f6a3406b3d0 msgr2=0x7f6a3406b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:17.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.825+0000 7f6a3a400640 1 -- 172.21.15.6:0/859758724 shutdown_connections 2024-03-04T09:31:17.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.825+0000 7f6a3a400640 1 -- 172.21.15.6:0/859758724 wait complete. 2024-03-04T09:31:17.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.827+0000 7f6a3a400640 1 Processor -- start 2024-03-04T09:31:17.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.827+0000 7f6a3a400640 1 -- start start 2024-03-04T09:31:17.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.828+0000 7f6a3a400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6a34070ce0 0x7f6a3414e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:17.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.828+0000 7f6a3a400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6a340cd1e0 0x7f6a3409d100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:17.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.828+0000 7f6a3a400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6a3415afa0 0x7f6a3414c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:17.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.828+0000 7f6a3a400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a340b4000 con 0x7f6a34070ce0 2024-03-04T09:31:17.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.828+0000 7f6a3a400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a34071170 con 0x7f6a340cd1e0 2024-03-04T09:31:17.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.828+0000 7f6a3a400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a340b4180 con 0x7f6a3415afa0 2024-03-04T09:31:17.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.828+0000 7f6a33400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6a34070ce0 0x7f6a3414e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:17.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.828+0000 7f6a33400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6a34070ce0 0x7f6a3414e2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55160/0 (socket says 172.21.15.6:55160) 2024-03-04T09:31:17.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.828+0000 7f6a33400640 1 -- 172.21.15.6:0/2414584912 learned_addr learned my addr 172.21.15.6:0/2414584912 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:31:17.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.829+0000 7f6a33400640 1 -- 172.21.15.6:0/2414584912 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6a3415afa0 msgr2=0x7f6a3414c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:17.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.829+0000 7f6a2be00640 1 --2- 172.21.15.6:0/2414584912 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6a340cd1e0 0x7f6a3409d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:17.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.829+0000 7f6a33e00640 1 --2- 172.21.15.6:0/2414584912 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6a3415afa0 0x7f6a3414c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:17.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.829+0000 7f6a33400640 1 --2- 172.21.15.6:0/2414584912 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6a3415afa0 0x7f6a3414c1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:17.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.829+0000 7f6a33400640 1 -- 172.21.15.6:0/2414584912 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6a340cd1e0 msgr2=0x7f6a3409d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:17.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.829+0000 7f6a33400640 1 --2- 172.21.15.6:0/2414584912 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6a340cd1e0 0x7f6a3409d100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:17.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.829+0000 7f6a33400640 1 -- 172.21.15.6:0/2414584912 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a3414d820 con 0x7f6a34070ce0 2024-03-04T09:31:17.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.829+0000 7f6a33e00640 1 --2- 172.21.15.6:0/2414584912 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6a3415afa0 0x7f6a3414c1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:31:17.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.829+0000 7f6a2be00640 1 --2- 172.21.15.6:0/2414584912 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6a340cd1e0 0x7f6a3409d100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:31:17.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.830+0000 7f6a33400640 1 --2- 172.21.15.6:0/2414584912 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6a34070ce0 0x7f6a3414e2b0 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7f6a240090b0 tx=0x7f6a240046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:17.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.830+0000 7f6a2b400640 1 -- 172.21.15.6:0/2414584912 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a2403ad60 con 0x7f6a34070ce0 2024-03-04T09:31:17.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.830+0000 7f6a2b400640 1 -- 172.21.15.6:0/2414584912 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a240443f0 con 0x7f6a34070ce0 2024-03-04T09:31:17.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.830+0000 7f6a2b400640 1 -- 172.21.15.6:0/2414584912 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a240447c0 con 0x7f6a34070ce0 2024-03-04T09:31:17.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.830+0000 7f6a3a400640 1 -- 172.21.15.6:0/2414584912 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a3409c930 con 0x7f6a34070ce0 2024-03-04T09:31:17.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.830+0000 7f6a3a400640 1 -- 172.21.15.6:0/2414584912 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a3409cc10 con 0x7f6a34070ce0 2024-03-04T09:31:17.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.832+0000 7f6a3a400640 1 -- 172.21.15.6:0/2414584912 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a3404f330 con 0x7f6a34070ce0 2024-03-04T09:31:17.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.833+0000 7f6a2b400640 1 -- 172.21.15.6:0/2414584912 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6a2404b060 con 0x7f6a34070ce0 2024-03-04T09:31:17.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.834+0000 7f6a2b400640 1 --2- 172.21.15.6:0/2414584912 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f6a1006b520 0x7f6a1006d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:17.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.834+0000 7f6a2b400640 1 -- 172.21.15.6:0/2414584912 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f6a240abe10 con 0x7f6a34070ce0 2024-03-04T09:31:17.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.834+0000 7f6a2be00640 1 --2- 172.21.15.6:0/2414584912 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f6a1006b520 0x7f6a1006d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:17.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.835+0000 7f6a2be00640 1 --2- 172.21.15.6:0/2414584912 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f6a1006b520 0x7f6a1006d9e0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f6a1c007fd0 tx=0x7f6a1c009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:17.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:17.839+0000 7f6a2b400640 1 -- 172.21.15.6:0/2414584912 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6a240af030 con 0x7f6a34070ce0 2024-03-04T09:31:18.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.037+0000 7f6a3a400640 1 -- 172.21.15.6:0/2414584912 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f6a3415a130 con 0x7f6a34070ce0 2024-03-04T09:31:18.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.038+0000 7f6a2b400640 1 -- 172.21.15.6:0/2414584912 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v105) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6a2403f070 con 0x7f6a34070ce0 2024-03-04T09:31:18.036 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-04T09:31:18.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.041+0000 7f6a3a400640 1 -- 172.21.15.6:0/2414584912 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f6a1006b520 msgr2=0x7f6a1006d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:18.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.041+0000 7f6a3a400640 1 --2- 172.21.15.6:0/2414584912 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f6a1006b520 0x7f6a1006d9e0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f6a1c007fd0 tx=0x7f6a1c009040 comp rx=0 tx=0).stop 2024-03-04T09:31:18.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.041+0000 7f6a3a400640 1 -- 172.21.15.6:0/2414584912 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6a34070ce0 msgr2=0x7f6a3414e2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:18.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.041+0000 7f6a3a400640 1 --2- 172.21.15.6:0/2414584912 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6a34070ce0 0x7f6a3414e2b0 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7f6a240090b0 tx=0x7f6a240046c0 comp rx=0 tx=0).stop 2024-03-04T09:31:18.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.041+0000 7f6a3a400640 1 -- 172.21.15.6:0/2414584912 shutdown_connections 2024-03-04T09:31:18.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.041+0000 7f6a3a400640 1 --2- 172.21.15.6:0/2414584912 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f6a1006b520 0x7f6a1006d9e0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:18.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.042+0000 7f6a3a400640 1 --2- 172.21.15.6:0/2414584912 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6a3415afa0 0x7f6a3414c1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:18.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.042+0000 7f6a3a400640 1 --2- 172.21.15.6:0/2414584912 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6a340cd1e0 0x7f6a3409d100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:18.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.042+0000 7f6a3a400640 1 --2- 172.21.15.6:0/2414584912 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6a34070ce0 0x7f6a3414e2b0 unknown :-1 s=CLOSED pgs=1226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:18.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.042+0000 7f6a3a400640 1 -- 172.21.15.6:0/2414584912 >> 172.21.15.6:0/2414584912 conn(0x7f6a3406b3d0 msgr2=0x7f6a340b1150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:18.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.042+0000 7f6a3a400640 1 -- 172.21.15.6:0/2414584912 shutdown_connections 2024-03-04T09:31:18.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.042+0000 7f6a3a400640 1 -- 172.21.15.6:0/2414584912 wait complete. 2024-03-04T09:31:18.060 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-04T09:31:18.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.285+0000 7fd0f1000640 1 Processor -- start 2024-03-04T09:31:18.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.285+0000 7fd0f1000640 1 -- start start 2024-03-04T09:31:18.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.287+0000 7fd0f1000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd0ec15efc0 0x7fd0ec0cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:18.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.287+0000 7fd0f1000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd0ec0cce90 0x7fd0ec161ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:18.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.287+0000 7fd0f1000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd0ec162000 0x7fd0ec0ad880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:18.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.287+0000 7fd0f1000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0ec15f3a0 con 0x7fd0ec162000 2024-03-04T09:31:18.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.287+0000 7fd0f1000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0ec06d190 con 0x7fd0ec15efc0 2024-03-04T09:31:18.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.287+0000 7fd0f1000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0ec1627f0 con 0x7fd0ec0cce90 2024-03-04T09:31:18.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.288+0000 7fd0ebe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd0ec162000 0x7fd0ec0ad880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:18.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.288+0000 7fd0ebe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd0ec162000 0x7fd0ec0ad880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55168/0 (socket says 172.21.15.6:55168) 2024-03-04T09:31:18.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.288+0000 7fd0ebe00640 1 -- 172.21.15.6:0/3786689628 learned_addr learned my addr 172.21.15.6:0/3786689628 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:31:18.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.288+0000 7fd0eaa00640 1 --2- 172.21.15.6:0/3786689628 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd0ec0cce90 0x7fd0ec161ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:18.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.288+0000 7fd0eb400640 1 --2- 172.21.15.6:0/3786689628 >> v2:172.21.15.146:3300/0 conn(0x7fd0ec15efc0 0x7fd0ec0cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:18.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.288+0000 7fd0ebe00640 1 -- 172.21.15.6:0/3786689628 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd0ec0cce90 msgr2=0x7fd0ec161ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:18.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.288+0000 7fd0ebe00640 1 --2- 172.21.15.6:0/3786689628 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd0ec0cce90 0x7fd0ec161ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:18.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.288+0000 7fd0ebe00640 1 -- 172.21.15.6:0/3786689628 >> v2:172.21.15.146:3300/0 conn(0x7fd0ec15efc0 msgr2=0x7fd0ec0cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:18.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.288+0000 7fd0ebe00640 1 --2- 172.21.15.6:0/3786689628 >> v2:172.21.15.146:3300/0 conn(0x7fd0ec15efc0 0x7fd0ec0cc950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:18.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.288+0000 7fd0ebe00640 1 -- 172.21.15.6:0/3786689628 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0ec06c860 con 0x7fd0ec162000 2024-03-04T09:31:18.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.290+0000 7fd0ebe00640 1 --2- 172.21.15.6:0/3786689628 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd0ec162000 0x7fd0ec0ad880 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7fd0ec0cd770 tx=0x7fd0e402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=198b8d5e4bae94ac server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:18.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.290+0000 7fd0ea000640 1 -- 172.21.15.6:0/3786689628 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0e402ca80 con 0x7fd0ec162000 2024-03-04T09:31:18.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.290+0000 7fd0ea000640 1 -- 172.21.15.6:0/3786689628 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0e402cc20 con 0x7fd0ec162000 2024-03-04T09:31:18.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.290+0000 7fd0ea000640 1 -- 172.21.15.6:0/3786689628 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0e40343f0 con 0x7fd0ec162000 2024-03-04T09:31:18.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.290+0000 7fd0f1000640 1 -- 172.21.15.6:0/3786689628 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd0ec162000 msgr2=0x7fd0ec0ad880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:18.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.290+0000 7fd0f1000640 1 --2- 172.21.15.6:0/3786689628 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd0ec162000 0x7fd0ec0ad880 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7fd0ec0cd770 tx=0x7fd0e402bee0 comp rx=0 tx=0).stop 2024-03-04T09:31:18.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.291+0000 7fd0f1000640 1 -- 172.21.15.6:0/3786689628 shutdown_connections 2024-03-04T09:31:18.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.291+0000 7fd0f1000640 1 --2- 172.21.15.6:0/3786689628 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd0ec162000 0x7fd0ec0ad880 unknown :-1 s=CLOSED pgs=1227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:18.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.291+0000 7fd0f1000640 1 --2- 172.21.15.6:0/3786689628 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd0ec0cce90 0x7fd0ec161ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:18.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.291+0000 7fd0f1000640 1 --2- 172.21.15.6:0/3786689628 >> v2:172.21.15.146:3300/0 conn(0x7fd0ec15efc0 0x7fd0ec0cc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:18.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.291+0000 7fd0f1000640 1 -- 172.21.15.6:0/3786689628 >> 172.21.15.6:0/3786689628 conn(0x7fd0ec06b3d0 msgr2=0x7fd0ec06b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:18.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.291+0000 7fd0f1000640 1 -- 172.21.15.6:0/3786689628 shutdown_connections 2024-03-04T09:31:18.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.291+0000 7fd0f1000640 1 -- 172.21.15.6:0/3786689628 wait complete. 2024-03-04T09:31:18.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.292+0000 7fd0f1000640 1 Processor -- start 2024-03-04T09:31:18.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.293+0000 7fd0f1000640 1 -- start start 2024-03-04T09:31:18.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.293+0000 7fd0f1000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd0ec0cce90 0x7fd0ec13a330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:18.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.293+0000 7fd0f1000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd0ec15efc0 0x7fd0ec1f4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:18.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.293+0000 7fd0f1000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd0ec162000 0x7fd0ec0b3900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:18.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.294+0000 7fd0f1000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0ec162490 con 0x7fd0ec162000 2024-03-04T09:31:18.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.294+0000 7fd0f1000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0ec06ce20 con 0x7fd0ec0cce90 2024-03-04T09:31:18.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.294+0000 7fd0f1000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0ec06d620 con 0x7fd0ec15efc0 2024-03-04T09:31:18.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.294+0000 7fd0ebe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd0ec162000 0x7fd0ec0b3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:18.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.294+0000 7fd0ebe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd0ec162000 0x7fd0ec0b3900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55174/0 (socket says 172.21.15.6:55174) 2024-03-04T09:31:18.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.294+0000 7fd0ebe00640 1 -- 172.21.15.6:0/179208988 learned_addr learned my addr 172.21.15.6:0/179208988 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:31:18.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.294+0000 7fd0eb400640 1 --2- 172.21.15.6:0/179208988 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd0ec0cce90 0x7fd0ec13a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:18.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.294+0000 7fd0eaa00640 1 --2- 172.21.15.6:0/179208988 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd0ec15efc0 0x7fd0ec1f4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:18.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.294+0000 7fd0ebe00640 1 -- 172.21.15.6:0/179208988 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd0ec15efc0 msgr2=0x7fd0ec1f4360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:18.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.294+0000 7fd0ebe00640 1 --2- 172.21.15.6:0/179208988 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd0ec15efc0 0x7fd0ec1f4360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:18.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.294+0000 7fd0ebe00640 1 -- 172.21.15.6:0/179208988 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd0ec0cce90 msgr2=0x7fd0ec13a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:18.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.294+0000 7fd0ebe00640 1 --2- 172.21.15.6:0/179208988 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd0ec0cce90 0x7fd0ec13a330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:18.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.294+0000 7fd0ebe00640 1 -- 172.21.15.6:0/179208988 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0ec1f48a0 con 0x7fd0ec162000 2024-03-04T09:31:18.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.294+0000 7fd0eaa00640 1 --2- 172.21.15.6:0/179208988 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd0ec15efc0 0x7fd0ec1f4360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:31:18.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.295+0000 7fd0ebe00640 1 --2- 172.21.15.6:0/179208988 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd0ec162000 0x7fd0ec0b3900 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7fd0e4009f40 tx=0x7fd0e4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:18.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.295+0000 7fd0e3400640 1 -- 172.21.15.6:0/179208988 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0e4043020 con 0x7fd0ec162000 2024-03-04T09:31:18.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.295+0000 7fd0e3400640 1 -- 172.21.15.6:0/179208988 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0e403e450 con 0x7fd0ec162000 2024-03-04T09:31:18.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.295+0000 7fd0e3400640 1 -- 172.21.15.6:0/179208988 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0e4009510 con 0x7fd0ec162000 2024-03-04T09:31:18.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.295+0000 7fd0f1000640 1 -- 172.21.15.6:0/179208988 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0ec145500 con 0x7fd0ec162000 2024-03-04T09:31:18.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.295+0000 7fd0f1000640 1 -- 172.21.15.6:0/179208988 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0ec145ac0 con 0x7fd0ec162000 2024-03-04T09:31:18.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.296+0000 7fd0f1000640 1 -- 172.21.15.6:0/179208988 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0ec04f330 con 0x7fd0ec162000 2024-03-04T09:31:18.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.297+0000 7fd0e3400640 1 -- 172.21.15.6:0/179208988 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd0e4046020 con 0x7fd0ec162000 2024-03-04T09:31:18.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.298+0000 7fd0e3400640 1 --2- 172.21.15.6:0/179208988 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd0cc06b4d0 0x7fd0cc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:18.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.298+0000 7fd0e3400640 1 -- 172.21.15.6:0/179208988 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fd0e40ac6e0 con 0x7fd0ec162000 2024-03-04T09:31:18.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.299+0000 7fd0eb400640 1 --2- 172.21.15.6:0/179208988 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd0cc06b4d0 0x7fd0cc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:18.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.300+0000 7fd0eb400640 1 --2- 172.21.15.6:0/179208988 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd0cc06b4d0 0x7fd0cc06d990 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fd0dc006fd0 tx=0x7fd0dc008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:18.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.303+0000 7fd0e3400640 1 -- 172.21.15.6:0/179208988 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd0e404d030 con 0x7fd0ec162000 2024-03-04T09:31:18.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.503+0000 7fd0f1000640 1 -- 172.21.15.6:0/179208988 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fd0ec0af2f0 con 0x7fd0ec162000 2024-03-04T09:31:18.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.504+0000 7fd0e3400640 1 -- 172.21.15.6:0/179208988 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v105) v1 ==== 84+0+65 (secure 0 0 0) 0x7fd0ec1f48a0 con 0x7fd0ec162000 2024-03-04T09:31:18.501 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:31:18.501 INFO:teuthology.orchestra.run.smithi006.stdout:[ 2024-03-04T09:31:18.501 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:31:18.501 INFO:teuthology.orchestra.run.smithi006.stdout: "poolnum": 1, 2024-03-04T09:31:18.501 INFO:teuthology.orchestra.run.smithi006.stdout: "poolname": ".mgr" 2024-03-04T09:31:18.501 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:31:18.501 INFO:teuthology.orchestra.run.smithi006.stdout:] 2024-03-04T09:31:18.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.506+0000 7fd0f1000640 1 -- 172.21.15.6:0/179208988 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd0cc06b4d0 msgr2=0x7fd0cc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:18.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.506+0000 7fd0f1000640 1 --2- 172.21.15.6:0/179208988 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd0cc06b4d0 0x7fd0cc06d990 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fd0dc006fd0 tx=0x7fd0dc008040 comp rx=0 tx=0).stop 2024-03-04T09:31:18.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.506+0000 7fd0f1000640 1 -- 172.21.15.6:0/179208988 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd0ec162000 msgr2=0x7fd0ec0b3900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:18.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.506+0000 7fd0f1000640 1 --2- 172.21.15.6:0/179208988 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd0ec162000 0x7fd0ec0b3900 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7fd0e4009f40 tx=0x7fd0e4004a40 comp rx=0 tx=0).stop 2024-03-04T09:31:18.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.507+0000 7fd0f1000640 1 -- 172.21.15.6:0/179208988 shutdown_connections 2024-03-04T09:31:18.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.507+0000 7fd0f1000640 1 --2- 172.21.15.6:0/179208988 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd0ec162000 0x7fd0ec0b3900 unknown :-1 s=CLOSED pgs=1228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:18.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.507+0000 7fd0f1000640 1 --2- 172.21.15.6:0/179208988 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd0cc06b4d0 0x7fd0cc06d990 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:18.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.507+0000 7fd0f1000640 1 --2- 172.21.15.6:0/179208988 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd0ec15efc0 0x7fd0ec1f4360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:18.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.507+0000 7fd0f1000640 1 --2- 172.21.15.6:0/179208988 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd0ec0cce90 0x7fd0ec13a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:18.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.507+0000 7fd0f1000640 1 -- 172.21.15.6:0/179208988 >> 172.21.15.6:0/179208988 conn(0x7fd0ec06b3d0 msgr2=0x7fd0ec0af750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:18.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.507+0000 7fd0f1000640 1 -- 172.21.15.6:0/179208988 shutdown_connections 2024-03-04T09:31:18.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.507+0000 7fd0f1000640 1 -- 172.21.15.6:0/179208988 wait complete. 2024-03-04T09:31:18.521 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-04T09:31:18.522 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-04T09:31:18.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.745+0000 7ff04fe00640 1 Processor -- start 2024-03-04T09:31:18.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.745+0000 7ff04fe00640 1 -- start start 2024-03-04T09:31:18.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.747+0000 7ff04fe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff0500cf420 0x7ff05006daf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:18.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.747+0000 7ff04fe00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff05006e030 0x7ff05015c300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:18.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.747+0000 7ff04fe00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff050071c50 con 0x7ff050145570 2024-03-04T09:31:18.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.747+0000 7ff04fe00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0500cf880 con 0x7ff0500cf420 2024-03-04T09:31:18.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.747+0000 7ff04fe00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff05006e480 con 0x7ff05006e030 2024-03-04T09:31:18.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.747+0000 7ff04e000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff0500cf420 0x7ff05006daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:18.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.747+0000 7ff04e000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff0500cf420 0x7ff05006daf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55186/0 (socket says 172.21.15.6:55186) 2024-03-04T09:31:18.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.747+0000 7ff04e000640 1 -- 172.21.15.6:0/3999074332 learned_addr learned my addr 172.21.15.6:0/3999074332 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:31:18.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.748+0000 7ff04f400640 1 --2- 172.21.15.6:0/3999074332 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff05006e030 0x7ff05015c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:18.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.748+0000 7ff04e000640 1 -- 172.21.15.6:0/3999074332 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff05006e030 msgr2=0x7ff05015c300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:18.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.748+0000 7ff04e000640 1 --2- 172.21.15.6:0/3999074332 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff05006e030 0x7ff05015c300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:18.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.748+0000 7ff04e000640 1 -- 172.21.15.6:0/3999074332 >> v1:172.21.15.146:6789/0 conn(0x7ff050145570 legacy=0x7ff050071450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:18.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.748+0000 7ff04e000640 1 -- 172.21.15.6:0/3999074332 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff050071f30 con 0x7ff0500cf420 2024-03-04T09:31:18.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.749+0000 7ff04e000640 1 --2- 172.21.15.6:0/3999074332 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff0500cf420 0x7ff05006daf0 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7ff038009870 tx=0x7ff03802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91ebb9304de9babe server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:18.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.749+0000 7ff04d600640 1 -- 172.21.15.6:0/3999074332 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff038035020 con 0x7ff0500cf420 2024-03-04T09:31:18.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.749+0000 7ff04d600640 1 -- 172.21.15.6:0/3999074332 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff038031640 con 0x7ff0500cf420 2024-03-04T09:31:18.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.749+0000 7ff04d600640 1 -- 172.21.15.6:0/3999074332 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff03802d040 con 0x7ff0500cf420 2024-03-04T09:31:18.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.750+0000 7ff04fe00640 1 -- 172.21.15.6:0/3999074332 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff0500cf420 msgr2=0x7ff05006daf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:18.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.750+0000 7ff04fe00640 1 --2- 172.21.15.6:0/3999074332 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff0500cf420 0x7ff05006daf0 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7ff038009870 tx=0x7ff03802b8f0 comp rx=0 tx=0).stop 2024-03-04T09:31:18.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.750+0000 7ff04fe00640 1 -- 172.21.15.6:0/3999074332 shutdown_connections 2024-03-04T09:31:18.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.750+0000 7ff04fe00640 1 --2- 172.21.15.6:0/3999074332 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff05006e030 0x7ff05015c300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:18.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.750+0000 7ff04fe00640 1 --2- 172.21.15.6:0/3999074332 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff0500cf420 0x7ff05006daf0 unknown :-1 s=CLOSED pgs=1229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:18.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.750+0000 7ff04fe00640 1 -- 172.21.15.6:0/3999074332 >> 172.21.15.6:0/3999074332 conn(0x7ff0500b17c0 msgr2=0x7ff0500b1ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:18.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.750+0000 7ff04fe00640 1 -- 172.21.15.6:0/3999074332 shutdown_connections 2024-03-04T09:31:18.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.750+0000 7ff04fe00640 1 -- 172.21.15.6:0/3999074332 wait complete. 2024-03-04T09:31:18.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.752+0000 7ff04fe00640 1 Processor -- start 2024-03-04T09:31:18.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.752+0000 7ff04fe00640 1 -- start start 2024-03-04T09:31:18.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.753+0000 7ff04fe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff05006e030 0x7ff0501e6760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:18.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.753+0000 7ff04fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff0500cf420 0x7ff0501419f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:18.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.753+0000 7ff04fe00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff050145570 0x7ff0501e4e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:18.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.753+0000 7ff04fe00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0500727d0 con 0x7ff0500cf420 2024-03-04T09:31:18.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.753+0000 7ff04fe00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0500724f0 con 0x7ff05006e030 2024-03-04T09:31:18.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.753+0000 7ff04fe00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff05015c840 con 0x7ff050145570 2024-03-04T09:31:18.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.753+0000 7ff04e000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff0500cf420 0x7ff0501419f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:18.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.753+0000 7ff04e000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff0500cf420 0x7ff0501419f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55188/0 (socket says 172.21.15.6:55188) 2024-03-04T09:31:18.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.753+0000 7ff04e000640 1 -- 172.21.15.6:0/1816623666 learned_addr learned my addr 172.21.15.6:0/1816623666 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:31:18.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.753+0000 7ff04ea00640 1 --2- 172.21.15.6:0/1816623666 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff05006e030 0x7ff0501e6760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:18.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.754+0000 7ff04f400640 1 --2- 172.21.15.6:0/1816623666 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff050145570 0x7ff0501e4e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:18.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.754+0000 7ff04e000640 1 -- 172.21.15.6:0/1816623666 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff050145570 msgr2=0x7ff0501e4e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:18.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.754+0000 7ff04e000640 1 --2- 172.21.15.6:0/1816623666 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff050145570 0x7ff0501e4e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:18.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.754+0000 7ff04e000640 1 -- 172.21.15.6:0/1816623666 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff05006e030 msgr2=0x7ff0501e6760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:18.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.754+0000 7ff04e000640 1 --2- 172.21.15.6:0/1816623666 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff05006e030 0x7ff0501e6760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:18.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.754+0000 7ff04e000640 1 -- 172.21.15.6:0/1816623666 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0501e53b0 con 0x7ff0500cf420 2024-03-04T09:31:18.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.754+0000 7ff04e000640 1 --2- 172.21.15.6:0/1816623666 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff0500cf420 0x7ff0501419f0 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7ff03802b930 tx=0x7ff0380046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:18.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.754+0000 7ff046a00640 1 -- 172.21.15.6:0/1816623666 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff038040070 con 0x7ff0500cf420 2024-03-04T09:31:18.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.754+0000 7ff046a00640 1 -- 172.21.15.6:0/1816623666 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff03803e070 con 0x7ff0500cf420 2024-03-04T09:31:18.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.754+0000 7ff04fe00640 1 -- 172.21.15.6:0/1816623666 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0501e59a0 con 0x7ff0500cf420 2024-03-04T09:31:18.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.755+0000 7ff046a00640 1 -- 172.21.15.6:0/1816623666 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff038035040 con 0x7ff0500cf420 2024-03-04T09:31:18.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.755+0000 7ff04fe00640 1 -- 172.21.15.6:0/1816623666 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0501e5f60 con 0x7ff0500cf420 2024-03-04T09:31:18.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.756+0000 7ff04fe00640 1 -- 172.21.15.6:0/1816623666 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff05004f330 con 0x7ff0500cf420 2024-03-04T09:31:18.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.757+0000 7ff046a00640 1 -- 172.21.15.6:0/1816623666 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff038038020 con 0x7ff0500cf420 2024-03-04T09:31:18.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.757+0000 7ff046a00640 1 --2- 172.21.15.6:0/1816623666 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff02c06b4d0 0x7ff02c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:18.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.758+0000 7ff046a00640 1 -- 172.21.15.6:0/1816623666 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 4857+0+0 (secure 0 0 0) 0x7ff0380ab960 con 0x7ff0500cf420 2024-03-04T09:31:18.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.758+0000 7ff04ea00640 1 --2- 172.21.15.6:0/1816623666 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff02c06b4d0 0x7ff02c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:18.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.759+0000 7ff04ea00640 1 --2- 172.21.15.6:0/1816623666 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff02c06b4d0 0x7ff02c06d990 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7ff050068550 tx=0x7ff048009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:18.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.762+0000 7ff046a00640 1 -- 172.21.15.6:0/1816623666 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff0501e53b0 con 0x7ff0500cf420 2024-03-04T09:31:18.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:18.964+0000 7ff04fe00640 1 -- 172.21.15.6:0/1816623666 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7ff0500aeb20 con 0x7ff0500cf420 2024-03-04T09:31:19.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:19.602+0000 7ff046a00640 1 -- 172.21.15.6:0/1816623666 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v106) v1 ==== 138+0+0 (secure 0 0 0) 0x7ff03807aa90 con 0x7ff0500cf420 2024-03-04T09:31:19.600 INFO:teuthology.orchestra.run.smithi006.stderr:pool 'cephfs_metadata' created 2024-03-04T09:31:19.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:19.607+0000 7ff04fe00640 1 -- 172.21.15.6:0/1816623666 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff02c06b4d0 msgr2=0x7ff02c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:19.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:19.607+0000 7ff04fe00640 1 --2- 172.21.15.6:0/1816623666 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff02c06b4d0 0x7ff02c06d990 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7ff050068550 tx=0x7ff048009040 comp rx=0 tx=0).stop 2024-03-04T09:31:19.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:19.607+0000 7ff04fe00640 1 -- 172.21.15.6:0/1816623666 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff0500cf420 msgr2=0x7ff0501419f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:19.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:19.607+0000 7ff04fe00640 1 --2- 172.21.15.6:0/1816623666 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff0500cf420 0x7ff0501419f0 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7ff03802b930 tx=0x7ff0380046c0 comp rx=0 tx=0).stop 2024-03-04T09:31:19.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:19.607+0000 7ff04fe00640 1 -- 172.21.15.6:0/1816623666 shutdown_connections 2024-03-04T09:31:19.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:19.607+0000 7ff04fe00640 1 --2- 172.21.15.6:0/1816623666 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff02c06b4d0 0x7ff02c06d990 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:19.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:19.607+0000 7ff04fe00640 1 --2- 172.21.15.6:0/1816623666 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff050145570 0x7ff0501e4e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:19.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:19.607+0000 7ff04fe00640 1 --2- 172.21.15.6:0/1816623666 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff0500cf420 0x7ff0501419f0 unknown :-1 s=CLOSED pgs=1230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:19.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:19.607+0000 7ff04fe00640 1 --2- 172.21.15.6:0/1816623666 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff05006e030 0x7ff0501e6760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:19.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:19.607+0000 7ff04fe00640 1 -- 172.21.15.6:0/1816623666 >> 172.21.15.6:0/1816623666 conn(0x7ff0500b17c0 msgr2=0x7ff050144400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:19.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:19.607+0000 7ff04fe00640 1 -- 172.21.15.6:0/1816623666 shutdown_connections 2024-03-04T09:31:19.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:19.607+0000 7ff04fe00640 1 -- 172.21.15.6:0/1816623666 wait complete. 2024-03-04T09:31:19.634 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-04T09:31:19.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:19.857+0000 7fe676000640 1 Processor -- start 2024-03-04T09:31:19.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:19.858+0000 7fe676000640 1 -- start start 2024-03-04T09:31:19.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:19.859+0000 7fe676000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fe670164ce0 0x7fe6701670d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:19.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:19.859+0000 7fe676000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe670156830 con 0x7fe6700ad6f0 2024-03-04T09:31:19.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:19.859+0000 7fe676000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe670160120 con 0x7fe67015c370 2024-03-04T09:31:19.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:19.859+0000 7fe676000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe67006ce60 con 0x7fe670164ce0 2024-03-04T09:31:19.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:19.860+0000 7fe66f400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fe6700ad6f0 0x7fe67015f960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:56950/0 (socket says 172.21.15.6:56950) 2024-03-04T09:31:19.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:19.860+0000 7fe66f400640 1 -- 172.21.15.6:0/3968294872 learned_addr learned my addr 172.21.15.6:0/3968294872 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:31:19.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:19.860+0000 7fe66fe00640 1 --2- 172.21.15.6:0/3968294872 >> v2:172.21.15.6:3300/0 conn(0x7fe670164ce0 0x7fe6701670d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:19.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:19.860+0000 7fe66e000640 1 -- 172.21.15.6:0/3968294872 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2448730277 0 0) 0x7fe670156830 con 0x7fe6700ad6f0 2024-03-04T09:31:19.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:19.860+0000 7fe66e000640 1 -- 172.21.15.6:0/3968294872 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe654003520 con 0x7fe6700ad6f0 2024-03-04T09:31:19.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:19.860+0000 7fe66fe00640 1 -- 172.21.15.6:0/3968294872 >> v1:172.21.15.146:6789/0 conn(0x7fe67015c370 legacy=0x7fe6700adad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:19.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:19.860+0000 7fe66fe00640 1 -- 172.21.15.6:0/3968294872 >> v1:172.21.15.6:6789/0 conn(0x7fe6700ad6f0 legacy=0x7fe67015f960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:19.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:19.860+0000 7fe66fe00640 1 -- 172.21.15.6:0/3968294872 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe670160400 con 0x7fe670164ce0 2024-03-04T09:31:19.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:19.862+0000 7fe66fe00640 1 --2- 172.21.15.6:0/3968294872 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe670164ce0 0x7fe6701670d0 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7fe670156a00 tx=0x7fe66802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a62fb2fe727e13f server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:19.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:19.862+0000 7fe66e000640 1 -- 172.21.15.6:0/3968294872 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe668034070 con 0x7fe670164ce0 2024-03-04T09:31:19.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:19.862+0000 7fe66e000640 1 -- 172.21.15.6:0/3968294872 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe668030610 con 0x7fe670164ce0 2024-03-04T09:31:19.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:19.862+0000 7fe66e000640 1 -- 172.21.15.6:0/3968294872 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6680309c0 con 0x7fe670164ce0 2024-03-04T09:31:19.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:19.862+0000 7fe676000640 1 -- 172.21.15.6:0/3968294872 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe670164ce0 msgr2=0x7fe6701670d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:19.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:19.862+0000 7fe676000640 1 --2- 172.21.15.6:0/3968294872 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe670164ce0 0x7fe6701670d0 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7fe670156a00 tx=0x7fe66802b960 comp rx=0 tx=0).stop 2024-03-04T09:31:19.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:19.863+0000 7fe676000640 1 -- 172.21.15.6:0/3968294872 shutdown_connections 2024-03-04T09:31:19.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:19.863+0000 7fe676000640 1 --2- 172.21.15.6:0/3968294872 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe670164ce0 0x7fe6701670d0 unknown :-1 s=CLOSED pgs=1232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:19.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:19.863+0000 7fe676000640 1 -- 172.21.15.6:0/3968294872 >> 172.21.15.6:0/3968294872 conn(0x7fe67006b3d0 msgr2=0x7fe67006b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:19.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:19.863+0000 7fe676000640 1 -- 172.21.15.6:0/3968294872 shutdown_connections 2024-03-04T09:31:19.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:19.863+0000 7fe676000640 1 -- 172.21.15.6:0/3968294872 wait complete. 2024-03-04T09:31:19.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:19.865+0000 7fe676000640 1 Processor -- start 2024-03-04T09:31:19.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:19.865+0000 7fe676000640 1 -- start start 2024-03-04T09:31:19.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:19.866+0000 7fe676000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe6700ad6f0 0x7fe670150840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:19.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:19.866+0000 7fe676000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe67015c370 0x7fe6700ceed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:19.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:19.866+0000 7fe676000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe670164ce0 0x7fe670160920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:19.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:19.866+0000 7fe676000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe67006e5e0 con 0x7fe670164ce0 2024-03-04T09:31:19.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:19.866+0000 7fe676000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe67006d0b0 con 0x7fe67015c370 2024-03-04T09:31:19.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:19.866+0000 7fe676000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe67006e760 con 0x7fe6700ad6f0 2024-03-04T09:31:19.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:19.866+0000 7fe66fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe670164ce0 0x7fe670160920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:19.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:19.866+0000 7fe66fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe670164ce0 0x7fe670160920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60868/0 (socket says 172.21.15.6:60868) 2024-03-04T09:31:19.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:19.866+0000 7fe66fe00640 1 -- 172.21.15.6:0/1597692678 learned_addr learned my addr 172.21.15.6:0/1597692678 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:31:19.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:19.867+0000 7fe66ea00640 1 --2- 172.21.15.6:0/1597692678 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe67015c370 0x7fe6700ceed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:19.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:19.867+0000 7fe66f400640 1 --2- 172.21.15.6:0/1597692678 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe6700ad6f0 0x7fe670150840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:19.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:19.867+0000 7fe66fe00640 1 -- 172.21.15.6:0/1597692678 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe6700ad6f0 msgr2=0x7fe670150840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:19.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:19.867+0000 7fe66fe00640 1 --2- 172.21.15.6:0/1597692678 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe6700ad6f0 0x7fe670150840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:19.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:19.867+0000 7fe66fe00640 1 -- 172.21.15.6:0/1597692678 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe67015c370 msgr2=0x7fe6700ceed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:19.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:19.867+0000 7fe66fe00640 1 --2- 172.21.15.6:0/1597692678 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe67015c370 0x7fe6700ceed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:19.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:19.867+0000 7fe66fe00640 1 -- 172.21.15.6:0/1597692678 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe67009ac60 con 0x7fe670164ce0 2024-03-04T09:31:19.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:19.867+0000 7fe66f400640 1 --2- 172.21.15.6:0/1597692678 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe6700ad6f0 0x7fe670150840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:31:19.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:19.867+0000 7fe66ea00640 1 --2- 172.21.15.6:0/1597692678 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe67015c370 0x7fe6700ceed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:31:19.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:19.867+0000 7fe66fe00640 1 --2- 172.21.15.6:0/1597692678 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe670164ce0 0x7fe670160920 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7fe66802b9a0 tx=0x7fe6680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:19.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:19.867+0000 7fe667400640 1 -- 172.21.15.6:0/1597692678 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe668034040 con 0x7fe670164ce0 2024-03-04T09:31:19.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:19.867+0000 7fe667400640 1 -- 172.21.15.6:0/1597692678 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe668030410 con 0x7fe670164ce0 2024-03-04T09:31:19.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:19.867+0000 7fe667400640 1 -- 172.21.15.6:0/1597692678 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe668006e10 con 0x7fe670164ce0 2024-03-04T09:31:19.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:19.867+0000 7fe676000640 1 -- 172.21.15.6:0/1597692678 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe67009af40 con 0x7fe670164ce0 2024-03-04T09:31:19.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:19.868+0000 7fe676000640 1 -- 172.21.15.6:0/1597692678 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe67009b500 con 0x7fe670164ce0 2024-03-04T09:31:19.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:19.869+0000 7fe676000640 1 -- 172.21.15.6:0/1597692678 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe670068010 con 0x7fe670164ce0 2024-03-04T09:31:19.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:19.870+0000 7fe667400640 1 -- 172.21.15.6:0/1597692678 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe668037070 con 0x7fe670164ce0 2024-03-04T09:31:19.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:19.871+0000 7fe667400640 1 --2- 172.21.15.6:0/1597692678 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fe64c06b4d0 0x7fe64c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:19.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:19.871+0000 7fe667400640 1 -- 172.21.15.6:0/1597692678 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5227+0+0 (secure 0 0 0) 0x7fe6680abab0 con 0x7fe670164ce0 2024-03-04T09:31:19.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:19.871+0000 7fe66f400640 1 --2- 172.21.15.6:0/1597692678 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fe64c06b4d0 0x7fe64c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:19.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:19.872+0000 7fe66f400640 1 --2- 172.21.15.6:0/1597692678 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fe64c06b4d0 0x7fe64c06d990 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fe660001560 tx=0x7fe660009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:19.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:19.876+0000 7fe667400640 1 -- 172.21.15.6:0/1597692678 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe6680305e0 con 0x7fe670164ce0 2024-03-04T09:31:20.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:20.077+0000 7fe676000640 1 -- 172.21.15.6:0/1597692678 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fe6700cc440 con 0x7fe670164ce0 2024-03-04T09:31:20.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:20.620+0000 7fe667400640 1 -- 172.21.15.6:0/1597692678 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v107) v1 ==== 170+0+0 (secure 0 0 0) 0x7fe66807ab00 con 0x7fe670164ce0 2024-03-04T09:31:20.617 INFO:teuthology.orchestra.run.smithi006.stderr:pool 'cephfs_data' created 2024-03-04T09:31:20.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:20.623+0000 7fe676000640 1 -- 172.21.15.6:0/1597692678 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fe64c06b4d0 msgr2=0x7fe64c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:20.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:20.624+0000 7fe676000640 1 --2- 172.21.15.6:0/1597692678 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fe64c06b4d0 0x7fe64c06d990 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fe660001560 tx=0x7fe660009040 comp rx=0 tx=0).stop 2024-03-04T09:31:20.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:20.624+0000 7fe676000640 1 -- 172.21.15.6:0/1597692678 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe670164ce0 msgr2=0x7fe670160920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:20.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:20.624+0000 7fe676000640 1 --2- 172.21.15.6:0/1597692678 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe670164ce0 0x7fe670160920 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7fe66802b9a0 tx=0x7fe6680046c0 comp rx=0 tx=0).stop 2024-03-04T09:31:20.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:20.625+0000 7fe676000640 1 -- 172.21.15.6:0/1597692678 shutdown_connections 2024-03-04T09:31:20.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:20.625+0000 7fe676000640 1 --2- 172.21.15.6:0/1597692678 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe670164ce0 0x7fe670160920 unknown :-1 s=CLOSED pgs=1233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:20.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:20.625+0000 7fe676000640 1 --2- 172.21.15.6:0/1597692678 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe67015c370 0x7fe6700ceed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:20.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:20.625+0000 7fe676000640 1 --2- 172.21.15.6:0/1597692678 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fe64c06b4d0 0x7fe64c06d990 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:20.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:20.625+0000 7fe676000640 1 --2- 172.21.15.6:0/1597692678 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe6700ad6f0 0x7fe670150840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:20.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:20.625+0000 7fe676000640 1 -- 172.21.15.6:0/1597692678 >> 172.21.15.6:0/1597692678 conn(0x7fe67006b3d0 msgr2=0x7fe670151800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:20.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:20.625+0000 7fe676000640 1 -- 172.21.15.6:0/1597692678 shutdown_connections 2024-03-04T09:31:20.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:20.626+0000 7fe676000640 1 -- 172.21.15.6:0/1597692678 wait complete. 2024-03-04T09:31:20.660 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-04T09:31:20.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:20.850+0000 7fb3ce000640 1 Processor -- start 2024-03-04T09:31:20.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:20.850+0000 7fb3ce000640 1 -- start start 2024-03-04T09:31:20.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:20.852+0000 7fb3ce000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb3c815bf20 0x7fb3c80acd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:20.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:20.852+0000 7fb3ce000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fb3c8072520 0x7fb3c80cdfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:20.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:20.852+0000 7fb3ce000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3c8072df0 con 0x7fb3c8071e20 2024-03-04T09:31:20.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:20.852+0000 7fb3ce000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3c80cdab0 con 0x7fb3c8072520 2024-03-04T09:31:20.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:20.852+0000 7fb3ce000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3c80b4d70 con 0x7fb3c815bf20 2024-03-04T09:31:20.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:20.853+0000 7fb3c7400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb3c815bf20 0x7fb3c80acd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:20.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:20.853+0000 7fb3c7400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb3c815bf20 0x7fb3c80acd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.6:60504/0 (socket says 172.21.15.6:60504) 2024-03-04T09:31:20.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:20.853+0000 7fb3c7400640 1 -- 172.21.15.6:0/475205637 learned_addr learned my addr 172.21.15.6:0/475205637 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:31:20.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:20.853+0000 7fb3c6a00640 1 --2- 172.21.15.6:0/475205637 >> v2:172.21.15.146:3300/0 conn(0x7fb3c8072520 0x7fb3c80cdfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:20.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:20.853+0000 7fb3c7400640 1 -- 172.21.15.6:0/475205637 >> v2:172.21.15.146:3300/0 conn(0x7fb3c8072520 msgr2=0x7fb3c80cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:20.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:20.853+0000 7fb3c7400640 1 --2- 172.21.15.6:0/475205637 >> v2:172.21.15.146:3300/0 conn(0x7fb3c8072520 0x7fb3c80cdfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:20.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:20.853+0000 7fb3c7400640 1 -- 172.21.15.6:0/475205637 >> v1:172.21.15.146:6789/0 conn(0x7fb3c8071e20 legacy=0x7fb3c80ccad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:20.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:20.853+0000 7fb3c7400640 1 -- 172.21.15.6:0/475205637 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3c80b5050 con 0x7fb3c815bf20 2024-03-04T09:31:20.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:20.855+0000 7fb3c7400640 1 --2- 172.21.15.6:0/475205637 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb3c815bf20 0x7fb3c80acd40 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fb3b8009870 tx=0x7fb3b802b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5d0dfaab0d519e20 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:20.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:20.856+0000 7fb3c6000640 1 -- 172.21.15.6:0/475205637 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3b8034070 con 0x7fb3c815bf20 2024-03-04T09:31:20.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:20.856+0000 7fb3c6000640 1 -- 172.21.15.6:0/475205637 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3b8030840 con 0x7fb3c815bf20 2024-03-04T09:31:20.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:20.856+0000 7fb3c6000640 1 -- 172.21.15.6:0/475205637 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3b8030c10 con 0x7fb3c815bf20 2024-03-04T09:31:20.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:20.856+0000 7fb3ce000640 1 -- 172.21.15.6:0/475205637 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb3c815bf20 msgr2=0x7fb3c80acd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:20.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:20.856+0000 7fb3ce000640 1 --2- 172.21.15.6:0/475205637 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb3c815bf20 0x7fb3c80acd40 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fb3b8009870 tx=0x7fb3b802b980 comp rx=0 tx=0).stop 2024-03-04T09:31:20.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:20.856+0000 7fb3ce000640 1 -- 172.21.15.6:0/475205637 shutdown_connections 2024-03-04T09:31:20.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:20.856+0000 7fb3ce000640 1 --2- 172.21.15.6:0/475205637 >> v2:172.21.15.146:3300/0 conn(0x7fb3c8072520 0x7fb3c80cdfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:20.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:20.856+0000 7fb3ce000640 1 --2- 172.21.15.6:0/475205637 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb3c815bf20 0x7fb3c80acd40 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:20.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:20.856+0000 7fb3ce000640 1 -- 172.21.15.6:0/475205637 >> 172.21.15.6:0/475205637 conn(0x7fb3c806b3d0 msgr2=0x7fb3c806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:20.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:20.857+0000 7fb3ce000640 1 -- 172.21.15.6:0/475205637 shutdown_connections 2024-03-04T09:31:20.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:20.857+0000 7fb3ce000640 1 -- 172.21.15.6:0/475205637 wait complete. 2024-03-04T09:31:20.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:20.858+0000 7fb3ce000640 1 Processor -- start 2024-03-04T09:31:20.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:20.858+0000 7fb3ce000640 1 -- start start 2024-03-04T09:31:20.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:20.859+0000 7fb3ce000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb3c8071e20 0x7fb3c815ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:20.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:20.859+0000 7fb3ce000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb3c8072520 0x7fb3c814cc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:20.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:20.859+0000 7fb3ce000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb3c815bf20 0x7fb3c80b19f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:20.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:20.859+0000 7fb3ce000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3c80b4b10 con 0x7fb3c815bf20 2024-03-04T09:31:20.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:20.859+0000 7fb3ce000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3c80b4990 con 0x7fb3c8071e20 2024-03-04T09:31:20.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:20.859+0000 7fb3ce000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3c80cd1e0 con 0x7fb3c8072520 2024-03-04T09:31:20.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:20.860+0000 7fb3c7e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb3c815bf20 0x7fb3c80b19f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:20.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:20.860+0000 7fb3c7e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb3c815bf20 0x7fb3c80b19f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60880/0 (socket says 172.21.15.6:60880) 2024-03-04T09:31:20.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:20.860+0000 7fb3c7e00640 1 -- 172.21.15.6:0/287183328 learned_addr learned my addr 172.21.15.6:0/287183328 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:31:20.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:20.860+0000 7fb3c6a00640 1 --2- 172.21.15.6:0/287183328 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb3c8072520 0x7fb3c814cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:20.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:20.860+0000 7fb3c7400640 1 --2- 172.21.15.6:0/287183328 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb3c8071e20 0x7fb3c815ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:20.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:20.860+0000 7fb3c7e00640 1 -- 172.21.15.6:0/287183328 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb3c8072520 msgr2=0x7fb3c814cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:20.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:20.860+0000 7fb3c7e00640 1 --2- 172.21.15.6:0/287183328 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb3c8072520 0x7fb3c814cc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:20.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:20.860+0000 7fb3c7e00640 1 -- 172.21.15.6:0/287183328 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb3c8071e20 msgr2=0x7fb3c815ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:20.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:20.860+0000 7fb3c7e00640 1 --2- 172.21.15.6:0/287183328 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb3c8071e20 0x7fb3c815ccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:20.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:20.860+0000 7fb3c7e00640 1 -- 172.21.15.6:0/287183328 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3c8155f40 con 0x7fb3c815bf20 2024-03-04T09:31:20.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:20.861+0000 7fb3c7e00640 1 --2- 172.21.15.6:0/287183328 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb3c815bf20 0x7fb3c80b19f0 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7fb3c000c7a0 tx=0x7fb3c000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:20.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:20.861+0000 7fb3bf400640 1 -- 172.21.15.6:0/287183328 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3c0003b70 con 0x7fb3c815bf20 2024-03-04T09:31:20.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:20.861+0000 7fb3bf400640 1 -- 172.21.15.6:0/287183328 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3c0003d30 con 0x7fb3c815bf20 2024-03-04T09:31:20.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:20.861+0000 7fb3bf400640 1 -- 172.21.15.6:0/287183328 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3c000aa90 con 0x7fb3c815bf20 2024-03-04T09:31:20.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:20.861+0000 7fb3ce000640 1 -- 172.21.15.6:0/287183328 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3c809b7f0 con 0x7fb3c815bf20 2024-03-04T09:31:20.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:20.861+0000 7fb3ce000640 1 -- 172.21.15.6:0/287183328 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3c809bdb0 con 0x7fb3c815bf20 2024-03-04T09:31:20.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:20.862+0000 7fb3ce000640 1 -- 172.21.15.6:0/287183328 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3c804f330 con 0x7fb3c815bf20 2024-03-04T09:31:20.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:20.864+0000 7fb3bf400640 1 -- 172.21.15.6:0/287183328 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb3c000ac30 con 0x7fb3c815bf20 2024-03-04T09:31:20.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:20.865+0000 7fb3bf400640 1 --2- 172.21.15.6:0/287183328 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb3a806b4d0 0x7fb3a806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:20.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:20.865+0000 7fb3bf400640 1 -- 172.21.15.6:0/287183328 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5609+0+0 (secure 0 0 0) 0x7fb3c008a1c0 con 0x7fb3c815bf20 2024-03-04T09:31:20.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:20.866+0000 7fb3c7400640 1 --2- 172.21.15.6:0/287183328 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb3a806b4d0 0x7fb3a806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:20.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:20.867+0000 7fb3c7400640 1 --2- 172.21.15.6:0/287183328 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb3a806b4d0 0x7fb3a806d990 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fb3b8036000 tx=0x7fb3b803b040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:20.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:20.870+0000 7fb3bf400640 1 -- 172.21.15.6:0/287183328 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb3c0059000 con 0x7fb3c815bf20 2024-03-04T09:31:21.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:21.112+0000 7fb3ce000640 1 -- 172.21.15.6:0/287183328 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fb3c809bf80 con 0x7fb3c815bf20 2024-03-04T09:31:21.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:21.632+0000 7fb3bf400640 1 -- 172.21.15.6:0/287183328 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 16 and data pool 17 v73) v1 ==== 174+0+0 (secure 0 0 0) 0x7fb3c0058900 con 0x7fb3c815bf20 2024-03-04T09:31:21.632 INFO:teuthology.orchestra.run.smithi006.stderr:new fs with metadata pool 16 and data pool 17 2024-03-04T09:31:21.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:21.642+0000 7fb3ce000640 1 -- 172.21.15.6:0/287183328 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb3a806b4d0 msgr2=0x7fb3a806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:21.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:21.642+0000 7fb3ce000640 1 --2- 172.21.15.6:0/287183328 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb3a806b4d0 0x7fb3a806d990 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fb3b8036000 tx=0x7fb3b803b040 comp rx=0 tx=0).stop 2024-03-04T09:31:21.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:21.642+0000 7fb3ce000640 1 -- 172.21.15.6:0/287183328 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb3c815bf20 msgr2=0x7fb3c80b19f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:21.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:21.642+0000 7fb3ce000640 1 --2- 172.21.15.6:0/287183328 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb3c815bf20 0x7fb3c80b19f0 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7fb3c000c7a0 tx=0x7fb3c000cc60 comp rx=0 tx=0).stop 2024-03-04T09:31:21.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:21.642+0000 7fb3ce000640 1 -- 172.21.15.6:0/287183328 shutdown_connections 2024-03-04T09:31:21.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:21.642+0000 7fb3ce000640 1 --2- 172.21.15.6:0/287183328 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb3c815bf20 0x7fb3c80b19f0 unknown :-1 s=CLOSED pgs=1234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:21.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:21.642+0000 7fb3ce000640 1 --2- 172.21.15.6:0/287183328 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb3a806b4d0 0x7fb3a806d990 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:21.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:21.642+0000 7fb3ce000640 1 --2- 172.21.15.6:0/287183328 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb3c8072520 0x7fb3c814cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:21.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:21.642+0000 7fb3ce000640 1 --2- 172.21.15.6:0/287183328 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb3c8071e20 0x7fb3c815ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:21.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:21.642+0000 7fb3ce000640 1 -- 172.21.15.6:0/287183328 >> 172.21.15.6:0/287183328 conn(0x7fb3c806b3d0 msgr2=0x7fb3c80ae810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:21.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:21.643+0000 7fb3ce000640 1 -- 172.21.15.6:0/287183328 shutdown_connections 2024-03-04T09:31:21.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:21.643+0000 7fb3ce000640 1 -- 172.21.15.6:0/287183328 wait complete. 2024-03-04T09:31:21.674 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T09:31:21.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:21.858+0000 7f9cad600640 1 Processor -- start 2024-03-04T09:31:21.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:21.858+0000 7f9cad600640 1 -- start start 2024-03-04T09:31:21.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:21.860+0000 7f9cad600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9ca8070f10 0x7f9ca80cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:21.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:21.860+0000 7f9cad600640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ca80714d0 con 0x7f9ca80cce90 2024-03-04T09:31:21.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:21.860+0000 7f9cad600640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ca80717b0 con 0x7f9ca8145300 2024-03-04T09:31:21.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:21.860+0000 7f9cad600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ca80cd300 con 0x7f9ca8070f10 2024-03-04T09:31:21.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:21.860+0000 7f9cacc00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f9ca80cce90 0x7f9ca815c570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:56958/0 (socket says 172.21.15.6:56958) 2024-03-04T09:31:21.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:21.860+0000 7f9cacc00640 1 -- 172.21.15.6:0/3588930947 learned_addr learned my addr 172.21.15.6:0/3588930947 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:31:21.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:21.860+0000 7f9c9f400640 1 --2- 172.21.15.6:0/3588930947 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9ca8070f10 0x7f9ca80cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:21.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:21.860+0000 7f9ca7400640 1 -- 172.21.15.6:0/3588930947 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 245513942 0 0) 0x7f9ca80714d0 con 0x7f9ca80cce90 2024-03-04T09:31:21.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:21.861+0000 7f9ca7400640 1 -- 172.21.15.6:0/3588930947 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c8c003520 con 0x7f9ca80cce90 2024-03-04T09:31:21.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:21.861+0000 7f9ca7400640 1 -- 172.21.15.6:0/3588930947 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2023846220 0 0) 0x7f9c8c003520 con 0x7f9ca80cce90 2024-03-04T09:31:21.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:21.861+0000 7f9ca7400640 1 -- 172.21.15.6:0/3588930947 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9ca8070f10 msgr2=0x7f9ca80cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:21.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:21.861+0000 7f9ca7400640 1 --2- 172.21.15.6:0/3588930947 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9ca8070f10 0x7f9ca80cc950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:21.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:21.861+0000 7f9ca7400640 1 -- 172.21.15.6:0/3588930947 >> v1:172.21.15.146:6789/0 conn(0x7f9ca8145300 legacy=0x7f9ca80b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:21.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:21.861+0000 7f9ca7400640 1 -- 172.21.15.6:0/3588930947 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ca8071a90 con 0x7f9ca80cce90 2024-03-04T09:31:21.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:21.861+0000 7f9ca7400640 1 -- 172.21.15.6:0/3588930947 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7f9ca00019f0 con 0x7f9ca80cce90 2024-03-04T09:31:21.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:21.861+0000 7f9ca7400640 1 -- 172.21.15.6:0/3588930947 >> v1:172.21.15.6:6789/0 conn(0x7f9ca80cce90 legacy=0x7f9ca815c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:21.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:21.861+0000 7f9ca7400640 1 --2- 172.21.15.6:0/3588930947 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9c8c0039e0 0x7f9c8c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:21.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:21.861+0000 7f9ca7400640 1 --2- 172.21.15.6:0/3588930947 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9c8c004800 0x7f9c8c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:21.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:21.861+0000 7f9ca7400640 1 --2- 172.21.15.6:0/3588930947 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9ca8070f10 0x7f9c8c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:21.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:21.861+0000 7f9ca7400640 1 -- 172.21.15.6:0/3588930947 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ca80cd300 con 0x7f9c8c0039e0 2024-03-04T09:31:21.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:21.861+0000 7f9ca7400640 1 -- 172.21.15.6:0/3588930947 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c8c00ba70 con 0x7f9c8c004800 2024-03-04T09:31:21.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:21.861+0000 7f9ca7400640 1 -- 172.21.15.6:0/3588930947 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c8c00bc70 con 0x7f9ca8070f10 2024-03-04T09:31:21.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:21.861+0000 7f9ca7e00640 1 --2- 172.21.15.6:0/3588930947 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9c8c0039e0 0x7f9c8c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:21.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:21.862+0000 7f9c9f400640 1 --2- 172.21.15.6:0/3588930947 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9c8c004800 0x7f9c8c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:21.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:21.862+0000 7f9cacc00640 1 --2- 172.21.15.6:0/3588930947 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9ca8070f10 0x7f9c8c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:21.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:21.862+0000 7f9ca7e00640 1 -- 172.21.15.6:0/3588930947 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9ca8070f10 msgr2=0x7f9c8c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:21.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:21.862+0000 7f9ca7e00640 1 --2- 172.21.15.6:0/3588930947 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9ca8070f10 0x7f9c8c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:21.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:21.862+0000 7f9ca7e00640 1 -- 172.21.15.6:0/3588930947 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9c8c004800 msgr2=0x7f9c8c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:21.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:21.862+0000 7f9ca7e00640 1 --2- 172.21.15.6:0/3588930947 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9c8c004800 0x7f9c8c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:21.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:21.862+0000 7f9ca7e00640 1 -- 172.21.15.6:0/3588930947 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c8c003520 con 0x7f9c8c0039e0 2024-03-04T09:31:21.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:21.863+0000 7f9ca7e00640 1 --2- 172.21.15.6:0/3588930947 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9c8c0039e0 0x7f9c8c003e00 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7f9c98001720 tx=0x7f9c8c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4dd30682fb56414b server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:21.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:21.863+0000 7f9ca7400640 1 -- 172.21.15.6:0/3588930947 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c98001370 con 0x7f9c8c0039e0 2024-03-04T09:31:21.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:21.863+0000 7f9ca7400640 1 -- 172.21.15.6:0/3588930947 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c980052d0 con 0x7f9c8c0039e0 2024-03-04T09:31:21.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:21.863+0000 7f9ca7400640 1 -- 172.21.15.6:0/3588930947 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c98003830 con 0x7f9c8c0039e0 2024-03-04T09:31:21.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:21.864+0000 7f9cad600640 1 -- 172.21.15.6:0/3588930947 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9c8c0039e0 msgr2=0x7f9c8c003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:21.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:21.864+0000 7f9cad600640 1 --2- 172.21.15.6:0/3588930947 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9c8c0039e0 0x7f9c8c003e00 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7f9c98001720 tx=0x7f9c8c005830 comp rx=0 tx=0).stop 2024-03-04T09:31:21.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:21.864+0000 7f9cacc00640 1 -- 172.21.15.6:0/3588930947 reap_dead start 2024-03-04T09:31:21.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:21.864+0000 7f9cad600640 1 -- 172.21.15.6:0/3588930947 shutdown_connections 2024-03-04T09:31:21.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:21.864+0000 7f9cad600640 1 -- 172.21.15.6:0/3588930947 >> 172.21.15.6:0/3588930947 conn(0x7f9ca806b3d0 msgr2=0x7f9ca806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:21.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:21.864+0000 7f9cad600640 1 -- 172.21.15.6:0/3588930947 shutdown_connections 2024-03-04T09:31:21.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:21.864+0000 7f9cad600640 1 -- 172.21.15.6:0/3588930947 wait complete. 2024-03-04T09:31:21.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:21.866+0000 7f9cad600640 1 Processor -- start 2024-03-04T09:31:21.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:21.866+0000 7f9cad600640 1 -- start start 2024-03-04T09:31:21.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:21.867+0000 7f9cad600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9c8c0039e0 0x7f9ca81432c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:21.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:21.867+0000 7f9cad600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9ca81e7a00 0x7f9ca81e94e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:21.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:21.867+0000 7f9cad600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9ca81e9a20 0x7f9ca81f3b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:21.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:21.867+0000 7f9cad600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ca80cc950 con 0x7f9ca81e9a20 2024-03-04T09:31:21.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:21.867+0000 7f9cad600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ca81456e0 con 0x7f9c8c0039e0 2024-03-04T09:31:21.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:21.867+0000 7f9cad600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ca80ccad0 con 0x7f9ca81e7a00 2024-03-04T09:31:21.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:21.867+0000 7f9cacc00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9ca81e9a20 0x7f9ca81f3b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:21.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:21.867+0000 7f9cacc00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9ca81e9a20 0x7f9ca81f3b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60890/0 (socket says 172.21.15.6:60890) 2024-03-04T09:31:21.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:21.867+0000 7f9cacc00640 1 -- 172.21.15.6:0/3218687463 learned_addr learned my addr 172.21.15.6:0/3218687463 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:31:21.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:21.867+0000 7f9ca7e00640 1 --2- 172.21.15.6:0/3218687463 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9c8c0039e0 0x7f9ca81432c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:21.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:21.867+0000 7f9cacc00640 1 -- 172.21.15.6:0/3218687463 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9ca81e7a00 msgr2=0x7f9ca81e94e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:21.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:21.867+0000 7f9cacc00640 1 --2- 172.21.15.6:0/3218687463 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9ca81e7a00 0x7f9ca81e94e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:21.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:21.867+0000 7f9cacc00640 1 -- 172.21.15.6:0/3218687463 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9c8c0039e0 msgr2=0x7f9ca81432c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:21.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:21.867+0000 7f9cacc00640 1 --2- 172.21.15.6:0/3218687463 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9c8c0039e0 0x7f9ca81432c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:21.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:21.867+0000 7f9cacc00640 1 -- 172.21.15.6:0/3218687463 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ca8143800 con 0x7f9ca81e9a20 2024-03-04T09:31:21.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:21.868+0000 7f9cacc00640 1 --2- 172.21.15.6:0/3218687463 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9ca81e9a20 0x7f9ca81f3b90 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7f9ca80cefa0 tx=0x7f9ca000e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:21.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:21.868+0000 7f9ca4c00640 1 -- 172.21.15.6:0/3218687463 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ca000ec10 con 0x7f9ca81e9a20 2024-03-04T09:31:21.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:21.868+0000 7f9ca4c00640 1 -- 172.21.15.6:0/3218687463 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ca0018070 con 0x7f9ca81e9a20 2024-03-04T09:31:21.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:21.868+0000 7f9ca4c00640 1 -- 172.21.15.6:0/3218687463 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ca0013420 con 0x7f9ca81e9a20 2024-03-04T09:31:21.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:21.868+0000 7f9cad600640 1 -- 172.21.15.6:0/3218687463 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ca81e6960 con 0x7f9ca81e9a20 2024-03-04T09:31:21.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:21.868+0000 7f9cad600640 1 -- 172.21.15.6:0/3218687463 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ca81e6f20 con 0x7f9ca81e9a20 2024-03-04T09:31:21.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:21.869+0000 7f9cad600640 1 -- 172.21.15.6:0/3218687463 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ca804f330 con 0x7f9ca81e9a20 2024-03-04T09:31:21.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:21.870+0000 7f9ca4c00640 1 -- 172.21.15.6:0/3218687463 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9ca00135c0 con 0x7f9ca81e9a20 2024-03-04T09:31:21.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:21.871+0000 7f9ca4c00640 1 --2- 172.21.15.6:0/3218687463 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f9c8406b500 0x7f9c8406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:21.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:21.871+0000 7f9ca4c00640 1 -- 172.21.15.6:0/3218687463 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9ca008a4e0 con 0x7f9ca81e9a20 2024-03-04T09:31:21.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:21.872+0000 7f9ca7e00640 1 --2- 172.21.15.6:0/3218687463 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f9c8406b500 0x7f9c8406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:21.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:21.873+0000 7f9ca7e00640 1 --2- 172.21.15.6:0/3218687463 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f9c8406b500 0x7f9c8406d9c0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f9c98033000 tx=0x7f9c98009030 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:21.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:21.876+0000 7f9ca4c00640 1 -- 172.21.15.6:0/3218687463 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9ca00593f0 con 0x7f9ca81e9a20 2024-03-04T09:31:22.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.076+0000 7f9cad600640 1 -- 172.21.15.6:0/3218687463 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9ca806d630 con 0x7f9ca81e9a20 2024-03-04T09:31:22.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.078+0000 7f9ca4c00640 1 -- 172.21.15.6:0/3218687463 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v108) v1 ==== 74+0+16592 (secure 0 0 0) 0x7f9ca0058cf0 con 0x7f9ca81e9a20 2024-03-04T09:31:22.076 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:31:22.076 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":108,"fsid":"edce0578-5ae4-4402-b45a-552bdef84d7c","created":"2024-03-04T09:24:54.523780+0000","modified":"2024-03-04T09:31:21.113749+0000","last_up_change":"2024-03-04T09:25:07.089049+0000","last_in_change":"2024-03-04T09:24:57.982077+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T09:25:07.935198+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-04T09:31:19.018233+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-04T09:31:20.131544+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a96574f-81f1-4bc6-8463-757520b84a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6825","nonce":1364449401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6827","nonce":1364449401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6831","nonce":1364449401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6829","nonce":1364449401}]},"public_addr":"172.21.15.6:6825/1364449401","cluster_addr":"172.21.15.6:6827/1364449401","heartbeat_back_addr":"172.21.15.6:6831/1364449401","heartbeat_front_addr":"172.21.15.6:6829/1364449401","state":["exists","up"]},{"osd":1,"uuid":"7d79b9c1-6b87-476a-bc36-7db5ee64cd90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6809","nonce":2494103078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6811","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6813","nonce":2494103078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6821","nonce":2494103078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6817","nonce":2494103078}]},"public_addr":"172.21.15.6:6809/2494103078","cluster_addr":"172.21.15.6:6813/2494103078","heartbeat_back_addr":"172.21.15.6:6821/2494103078","heartbeat_front_addr":"172.21.15.6:6817/2494103078","state":["exists","up"]},{"osd":2,"uuid":"492331b9-4ee9-40b6-bacd-85d44cd6c45a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6812","nonce":1500066681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6816","nonce":1500066681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6824","nonce":1500066681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6820","nonce":1500066681}]},"public_addr":"172.21.15.6:6812/1500066681","cluster_addr":"172.21.15.6:6816/1500066681","heartbeat_back_addr":"172.21.15.6:6824/1500066681","heartbeat_front_addr":"172.21.15.6:6820/1500066681","state":["exists","up"]},{"osd":3,"uuid":"21bb2efe-9ba7-4bdc-bc31-180cd1102e21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6801","nonce":1558806123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6803","nonce":1558806123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6807","nonce":1558806123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6805","nonce":1558806123}]},"public_addr":"172.21.15.6:6801/1558806123","cluster_addr":"172.21.15.6:6803/1558806123","heartbeat_back_addr":"172.21.15.6:6807/1558806123","heartbeat_front_addr":"172.21.15.6:6805/1558806123","state":["exists","up"]},{"osd":4,"uuid":"6e2c4b71-19d9-4737-83d9-558705d8644a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6801","nonce":1526135780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6803","nonce":1526135780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6807","nonce":1526135780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6805","nonce":1526135780}]},"public_addr":"172.21.15.146:6801/1526135780","cluster_addr":"172.21.15.146:6803/1526135780","heartbeat_back_addr":"172.21.15.146:6807/1526135780","heartbeat_front_addr":"172.21.15.146:6805/1526135780","state":["exists","up"]},{"osd":5,"uuid":"fb69c525-f1be-4b7d-ab32-6c29b0d6fddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6809","nonce":460262507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6811","nonce":460262507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6815","nonce":460262507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6813","nonce":460262507}]},"public_addr":"172.21.15.146:6809/460262507","cluster_addr":"172.21.15.146:6811/460262507","heartbeat_back_addr":"172.21.15.146:6815/460262507","heartbeat_front_addr":"172.21.15.146:6813/460262507","state":["exists","up"]},{"osd":6,"uuid":"c32e7fed-370d-4401-8ce0-6a4335b0d168","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6823","nonce":1392897493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6827","nonce":1392897493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6831","nonce":1392897493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6829","nonce":1392897493}]},"public_addr":"172.21.15.146:6823/1392897493","cluster_addr":"172.21.15.146:6827/1392897493","heartbeat_back_addr":"172.21.15.146:6831/1392897493","heartbeat_front_addr":"172.21.15.146:6829/1392897493","state":["exists","up"]},{"osd":7,"uuid":"75a79b44-fa5f-426d-a7d3-71f21f78835c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6817","nonce":3268323156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6819","nonce":3268323156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6825","nonce":3268323156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6822","nonce":3268323156}]},"public_addr":"172.21.15.146:6817/3268323156","cluster_addr":"172.21.15.146:6819/3268323156","heartbeat_back_addr":"172.21.15.146:6825/3268323156","heartbeat_front_addr":"172.21.15.146:6822/3268323156","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.310437+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.295850+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.294674+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.526855+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.300023+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.595160+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.346046+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.558741+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T09:31:22.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.081+0000 7f9cad600640 1 -- 172.21.15.6:0/3218687463 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f9c8406b500 msgr2=0x7f9c8406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:22.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.081+0000 7f9cad600640 1 --2- 172.21.15.6:0/3218687463 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f9c8406b500 0x7f9c8406d9c0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f9c98033000 tx=0x7f9c98009030 comp rx=0 tx=0).stop 2024-03-04T09:31:22.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.081+0000 7f9cad600640 1 -- 172.21.15.6:0/3218687463 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9ca81e9a20 msgr2=0x7f9ca81f3b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:22.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.081+0000 7f9cad600640 1 --2- 172.21.15.6:0/3218687463 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9ca81e9a20 0x7f9ca81f3b90 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7f9ca80cefa0 tx=0x7f9ca000e530 comp rx=0 tx=0).stop 2024-03-04T09:31:22.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.081+0000 7f9cad600640 1 -- 172.21.15.6:0/3218687463 shutdown_connections 2024-03-04T09:31:22.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.081+0000 7f9cad600640 1 --2- 172.21.15.6:0/3218687463 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9ca81e9a20 0x7f9ca81f3b90 unknown :-1 s=CLOSED pgs=1237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:22.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.081+0000 7f9cad600640 1 --2- 172.21.15.6:0/3218687463 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f9c8406b500 0x7f9c8406d9c0 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:22.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.081+0000 7f9cad600640 1 --2- 172.21.15.6:0/3218687463 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9ca81e7a00 0x7f9ca81e94e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:22.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.081+0000 7f9cad600640 1 --2- 172.21.15.6:0/3218687463 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9c8c0039e0 0x7f9ca81432c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:22.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.081+0000 7f9cad600640 1 -- 172.21.15.6:0/3218687463 >> 172.21.15.6:0/3218687463 conn(0x7f9ca806b3d0 msgr2=0x7f9ca815ecf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:22.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.081+0000 7f9cad600640 1 -- 172.21.15.6:0/3218687463 shutdown_connections 2024-03-04T09:31:22.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.081+0000 7f9cad600640 1 -- 172.21.15.6:0/3218687463 wait complete. 2024-03-04T09:31:22.095 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T09:31:22.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.320+0000 7f7b37200640 1 Processor -- start 2024-03-04T09:31:22.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.320+0000 7f7b37200640 1 -- start start 2024-03-04T09:31:22.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.322+0000 7f7b37200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f7b380d4a10 0x7f7b380d0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:22.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.322+0000 7f7b37200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7b380d0c80 0x7f7b380b44d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:22.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.322+0000 7f7b37200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b380b3710 con 0x7f7b3815e0b0 2024-03-04T09:31:22.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.322+0000 7f7b37200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b380b09e0 con 0x7f7b380d4a10 2024-03-04T09:31:22.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.322+0000 7f7b37200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b380cfa20 con 0x7f7b380d0c80 2024-03-04T09:31:22.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.322+0000 7f7b2d400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f7b380d4a10 0x7f7b380d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:22.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.322+0000 7f7b2d400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f7b380d4a10 0x7f7b380d0740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60900/0 (socket says 172.21.15.6:60900) 2024-03-04T09:31:22.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.322+0000 7f7b2d400640 1 -- 172.21.15.6:0/1456350635 learned_addr learned my addr 172.21.15.6:0/1456350635 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:31:22.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.323+0000 7f7b36800640 1 --2- 172.21.15.6:0/1456350635 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7b380d0c80 0x7f7b380b44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:22.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.323+0000 7f7b2d400640 1 -- 172.21.15.6:0/1456350635 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7b380d0c80 msgr2=0x7f7b380b44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:22.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.323+0000 7f7b2d400640 1 --2- 172.21.15.6:0/1456350635 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7b380d0c80 0x7f7b380b44d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:22.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.323+0000 7f7b2d400640 1 -- 172.21.15.6:0/1456350635 >> v1:172.21.15.146:6789/0 conn(0x7f7b3815e0b0 legacy=0x7f7b380cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:22.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.323+0000 7f7b2d400640 1 -- 172.21.15.6:0/1456350635 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b380b39f0 con 0x7f7b380d4a10 2024-03-04T09:31:22.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.324+0000 7f7b2d400640 1 --2- 172.21.15.6:0/1456350635 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7b380d4a10 0x7f7b380d0740 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7f7b20009870 tx=0x7f7b2002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24038bc8b98ea9b2 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:22.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.324+0000 7f7b35400640 1 -- 172.21.15.6:0/1456350635 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b20035020 con 0x7f7b380d4a10 2024-03-04T09:31:22.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.324+0000 7f7b35400640 1 -- 172.21.15.6:0/1456350635 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b20031640 con 0x7f7b380d4a10 2024-03-04T09:31:22.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.324+0000 7f7b35400640 1 -- 172.21.15.6:0/1456350635 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b2002d040 con 0x7f7b380d4a10 2024-03-04T09:31:22.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.325+0000 7f7b37200640 1 -- 172.21.15.6:0/1456350635 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7b380d4a10 msgr2=0x7f7b380d0740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:22.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.325+0000 7f7b37200640 1 --2- 172.21.15.6:0/1456350635 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7b380d4a10 0x7f7b380d0740 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7f7b20009870 tx=0x7f7b2002b8f0 comp rx=0 tx=0).stop 2024-03-04T09:31:22.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.325+0000 7f7b37200640 1 -- 172.21.15.6:0/1456350635 shutdown_connections 2024-03-04T09:31:22.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.325+0000 7f7b37200640 1 --2- 172.21.15.6:0/1456350635 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7b380d0c80 0x7f7b380b44d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:22.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.325+0000 7f7b37200640 1 --2- 172.21.15.6:0/1456350635 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7b380d4a10 0x7f7b380d0740 unknown :-1 s=CLOSED pgs=1238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:22.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.325+0000 7f7b37200640 1 -- 172.21.15.6:0/1456350635 >> 172.21.15.6:0/1456350635 conn(0x7f7b380d3d30 msgr2=0x7f7b380d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:22.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.325+0000 7f7b37200640 1 -- 172.21.15.6:0/1456350635 shutdown_connections 2024-03-04T09:31:22.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.325+0000 7f7b37200640 1 -- 172.21.15.6:0/1456350635 wait complete. 2024-03-04T09:31:22.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.327+0000 7f7b37200640 1 Processor -- start 2024-03-04T09:31:22.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.327+0000 7f7b37200640 1 -- start start 2024-03-04T09:31:22.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.328+0000 7f7b37200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7b380d0c80 0x7f7b38151550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:22.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.328+0000 7f7b37200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7b380d4a10 0x7f7b3815be40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:22.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.328+0000 7f7b37200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7b3815e0b0 0x7f7b3815c380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:22.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.328+0000 7f7b37200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b3815e490 con 0x7f7b380d4a10 2024-03-04T09:31:22.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.328+0000 7f7b37200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b380b3fb0 con 0x7f7b3815e0b0 2024-03-04T09:31:22.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.328+0000 7f7b37200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b380cfc60 con 0x7f7b380d0c80 2024-03-04T09:31:22.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.328+0000 7f7b2d400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7b380d4a10 0x7f7b3815be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:22.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.328+0000 7f7b2d400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7b380d4a10 0x7f7b3815be40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60906/0 (socket says 172.21.15.6:60906) 2024-03-04T09:31:22.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.328+0000 7f7b2d400640 1 -- 172.21.15.6:0/2645835178 learned_addr learned my addr 172.21.15.6:0/2645835178 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:31:22.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.329+0000 7f7b35e00640 1 --2- 172.21.15.6:0/2645835178 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7b380d0c80 0x7f7b38151550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:22.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.329+0000 7f7b36800640 1 --2- 172.21.15.6:0/2645835178 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7b3815e0b0 0x7f7b3815c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:22.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.329+0000 7f7b2d400640 1 -- 172.21.15.6:0/2645835178 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7b380d0c80 msgr2=0x7f7b38151550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:22.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.329+0000 7f7b2d400640 1 --2- 172.21.15.6:0/2645835178 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7b380d0c80 0x7f7b38151550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:22.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.329+0000 7f7b2d400640 1 -- 172.21.15.6:0/2645835178 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7b3815e0b0 msgr2=0x7f7b3815c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:22.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.329+0000 7f7b2d400640 1 --2- 172.21.15.6:0/2645835178 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7b3815e0b0 0x7f7b3815c380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:22.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.329+0000 7f7b2d400640 1 -- 172.21.15.6:0/2645835178 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b380cc340 con 0x7f7b380d4a10 2024-03-04T09:31:22.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.329+0000 7f7b2d400640 1 --2- 172.21.15.6:0/2645835178 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7b380d4a10 0x7f7b3815be40 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7f7b20000c00 tx=0x7f7b200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:22.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.330+0000 7f7b2ea00640 1 -- 172.21.15.6:0/2645835178 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b20040070 con 0x7f7b380d4a10 2024-03-04T09:31:22.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.330+0000 7f7b2ea00640 1 -- 172.21.15.6:0/2645835178 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b200399f0 con 0x7f7b380d4a10 2024-03-04T09:31:22.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.330+0000 7f7b2ea00640 1 -- 172.21.15.6:0/2645835178 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b20035040 con 0x7f7b380d4a10 2024-03-04T09:31:22.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.330+0000 7f7b37200640 1 -- 172.21.15.6:0/2645835178 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b380b6950 con 0x7f7b380d4a10 2024-03-04T09:31:22.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.330+0000 7f7b37200640 1 -- 172.21.15.6:0/2645835178 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b380b6f10 con 0x7f7b380d4a10 2024-03-04T09:31:22.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.331+0000 7f7b37200640 1 -- 172.21.15.6:0/2645835178 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b3804f330 con 0x7f7b380d4a10 2024-03-04T09:31:22.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.332+0000 7f7b2ea00640 1 -- 172.21.15.6:0/2645835178 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7b2004b060 con 0x7f7b380d4a10 2024-03-04T09:31:22.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.333+0000 7f7b2ea00640 1 --2- 172.21.15.6:0/2645835178 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f7b1406b500 0x7f7b1406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:22.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.333+0000 7f7b2ea00640 1 -- 172.21.15.6:0/2645835178 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7b200abc20 con 0x7f7b380d4a10 2024-03-04T09:31:22.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.334+0000 7f7b35e00640 1 --2- 172.21.15.6:0/2645835178 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f7b1406b500 0x7f7b1406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:22.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.335+0000 7f7b35e00640 1 --2- 172.21.15.6:0/2645835178 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f7b1406b500 0x7f7b1406d9c0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f7b28004ea0 tx=0x7f7b28009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:22.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.338+0000 7f7b2ea00640 1 -- 172.21.15.6:0/2645835178 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7b2007aa00 con 0x7f7b380d4a10 2024-03-04T09:31:22.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.543+0000 7f7b37200640 1 -- 172.21.15.6:0/2645835178 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7b38151a90 con 0x7f7b380d4a10 2024-03-04T09:31:22.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.545+0000 7f7b2ea00640 1 -- 172.21.15.6:0/2645835178 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v108) v1 ==== 74+0+16592 (secure 0 0 0) 0x7f7b2007a300 con 0x7f7b380d4a10 2024-03-04T09:31:22.542 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:31:22.542 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":108,"fsid":"edce0578-5ae4-4402-b45a-552bdef84d7c","created":"2024-03-04T09:24:54.523780+0000","modified":"2024-03-04T09:31:21.113749+0000","last_up_change":"2024-03-04T09:25:07.089049+0000","last_in_change":"2024-03-04T09:24:57.982077+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T09:25:07.935198+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-04T09:31:19.018233+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-04T09:31:20.131544+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a96574f-81f1-4bc6-8463-757520b84a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6825","nonce":1364449401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6827","nonce":1364449401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6831","nonce":1364449401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6829","nonce":1364449401}]},"public_addr":"172.21.15.6:6825/1364449401","cluster_addr":"172.21.15.6:6827/1364449401","heartbeat_back_addr":"172.21.15.6:6831/1364449401","heartbeat_front_addr":"172.21.15.6:6829/1364449401","state":["exists","up"]},{"osd":1,"uuid":"7d79b9c1-6b87-476a-bc36-7db5ee64cd90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6809","nonce":2494103078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6811","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6813","nonce":2494103078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6821","nonce":2494103078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6817","nonce":2494103078}]},"public_addr":"172.21.15.6:6809/2494103078","cluster_addr":"172.21.15.6:6813/2494103078","heartbeat_back_addr":"172.21.15.6:6821/2494103078","heartbeat_front_addr":"172.21.15.6:6817/2494103078","state":["exists","up"]},{"osd":2,"uuid":"492331b9-4ee9-40b6-bacd-85d44cd6c45a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6812","nonce":1500066681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6816","nonce":1500066681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6824","nonce":1500066681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6820","nonce":1500066681}]},"public_addr":"172.21.15.6:6812/1500066681","cluster_addr":"172.21.15.6:6816/1500066681","heartbeat_back_addr":"172.21.15.6:6824/1500066681","heartbeat_front_addr":"172.21.15.6:6820/1500066681","state":["exists","up"]},{"osd":3,"uuid":"21bb2efe-9ba7-4bdc-bc31-180cd1102e21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6801","nonce":1558806123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6803","nonce":1558806123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6807","nonce":1558806123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6805","nonce":1558806123}]},"public_addr":"172.21.15.6:6801/1558806123","cluster_addr":"172.21.15.6:6803/1558806123","heartbeat_back_addr":"172.21.15.6:6807/1558806123","heartbeat_front_addr":"172.21.15.6:6805/1558806123","state":["exists","up"]},{"osd":4,"uuid":"6e2c4b71-19d9-4737-83d9-558705d8644a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6801","nonce":1526135780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6803","nonce":1526135780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6807","nonce":1526135780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6805","nonce":1526135780}]},"public_addr":"172.21.15.146:6801/1526135780","cluster_addr":"172.21.15.146:6803/1526135780","heartbeat_back_addr":"172.21.15.146:6807/1526135780","heartbeat_front_addr":"172.21.15.146:6805/1526135780","state":["exists","up"]},{"osd":5,"uuid":"fb69c525-f1be-4b7d-ab32-6c29b0d6fddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6809","nonce":460262507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6811","nonce":460262507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6815","nonce":460262507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6813","nonce":460262507}]},"public_addr":"172.21.15.146:6809/460262507","cluster_addr":"172.21.15.146:6811/460262507","heartbeat_back_addr":"172.21.15.146:6815/460262507","heartbeat_front_addr":"172.21.15.146:6813/460262507","state":["exists","up"]},{"osd":6,"uuid":"c32e7fed-370d-4401-8ce0-6a4335b0d168","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6823","nonce":1392897493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6827","nonce":1392897493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6831","nonce":1392897493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6829","nonce":1392897493}]},"public_addr":"172.21.15.146:6823/1392897493","cluster_addr":"172.21.15.146:6827/1392897493","heartbeat_back_addr":"172.21.15.146:6831/1392897493","heartbeat_front_addr":"172.21.15.146:6829/1392897493","state":["exists","up"]},{"osd":7,"uuid":"75a79b44-fa5f-426d-a7d3-71f21f78835c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6817","nonce":3268323156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6819","nonce":3268323156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6825","nonce":3268323156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6822","nonce":3268323156}]},"public_addr":"172.21.15.146:6817/3268323156","cluster_addr":"172.21.15.146:6819/3268323156","heartbeat_back_addr":"172.21.15.146:6825/3268323156","heartbeat_front_addr":"172.21.15.146:6822/3268323156","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.310437+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.295850+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.294674+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.526855+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.300023+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.595160+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.346046+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.558741+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T09:31:22.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.547+0000 7f7b37200640 1 -- 172.21.15.6:0/2645835178 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f7b1406b500 msgr2=0x7f7b1406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:22.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.547+0000 7f7b37200640 1 --2- 172.21.15.6:0/2645835178 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f7b1406b500 0x7f7b1406d9c0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f7b28004ea0 tx=0x7f7b28009040 comp rx=0 tx=0).stop 2024-03-04T09:31:22.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.547+0000 7f7b37200640 1 -- 172.21.15.6:0/2645835178 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7b380d4a10 msgr2=0x7f7b3815be40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:22.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.547+0000 7f7b37200640 1 --2- 172.21.15.6:0/2645835178 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7b380d4a10 0x7f7b3815be40 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7f7b20000c00 tx=0x7f7b200046c0 comp rx=0 tx=0).stop 2024-03-04T09:31:22.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.548+0000 7f7b37200640 1 -- 172.21.15.6:0/2645835178 shutdown_connections 2024-03-04T09:31:22.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.548+0000 7f7b37200640 1 --2- 172.21.15.6:0/2645835178 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7b3815e0b0 0x7f7b3815c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:22.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.548+0000 7f7b37200640 1 --2- 172.21.15.6:0/2645835178 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7b380d4a10 0x7f7b3815be40 unknown :-1 s=CLOSED pgs=1239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:22.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.548+0000 7f7b37200640 1 --2- 172.21.15.6:0/2645835178 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f7b1406b500 0x7f7b1406d9c0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:22.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.548+0000 7f7b37200640 1 --2- 172.21.15.6:0/2645835178 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7b380d0c80 0x7f7b38151550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:22.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.548+0000 7f7b37200640 1 -- 172.21.15.6:0/2645835178 >> 172.21.15.6:0/2645835178 conn(0x7f7b380d3d30 msgr2=0x7f7b380b1d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:22.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.548+0000 7f7b37200640 1 -- 172.21.15.6:0/2645835178 shutdown_connections 2024-03-04T09:31:22.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.548+0000 7f7b37200640 1 -- 172.21.15.6:0/2645835178 wait complete. 2024-03-04T09:31:22.565 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-04T09:31:22.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.790+0000 7fc0fb200640 1 Processor -- start 2024-03-04T09:31:22.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.790+0000 7fc0fb200640 1 -- start start 2024-03-04T09:31:22.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.791+0000 7fc0fb200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc0f413ed70 0x7fc0f415ce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:22.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.792+0000 7fc0fb200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0f415c350 con 0x7fc0f406dc50 2024-03-04T09:31:22.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.792+0000 7fc0fb200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0f406e6a0 con 0x7fc0f40c75b0 2024-03-04T09:31:22.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.792+0000 7fc0fb200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0f415c980 con 0x7fc0f413ed70 2024-03-04T09:31:22.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.792+0000 7fc0f3400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fc0f406dc50 0x7fc0f40c6ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:56968/0 (socket says 172.21.15.6:56968) 2024-03-04T09:31:22.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.792+0000 7fc0f3400640 1 -- 172.21.15.6:0/2032387883 learned_addr learned my addr 172.21.15.6:0/2032387883 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:31:22.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.792+0000 7fc0f3e00640 1 --2- 172.21.15.6:0/2032387883 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc0f413ed70 0x7fc0f415ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:22.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.793+0000 7fc0f2a00640 1 -- 172.21.15.6:0/2032387883 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 606694889 0 0) 0x7fc0f415c350 con 0x7fc0f406dc50 2024-03-04T09:31:22.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.793+0000 7fc0f2a00640 1 -- 172.21.15.6:0/2032387883 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0d8003520 con 0x7fc0f406dc50 2024-03-04T09:31:22.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.793+0000 7fc0f2a00640 1 -- 172.21.15.6:0/2032387883 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1146250023 0 0) 0x7fc0d8003520 con 0x7fc0f406dc50 2024-03-04T09:31:22.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.793+0000 7fc0f2a00640 1 -- 172.21.15.6:0/2032387883 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc0f413ed70 msgr2=0x7fc0f415ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:22.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.793+0000 7fc0f2a00640 1 --2- 172.21.15.6:0/2032387883 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc0f413ed70 0x7fc0f415ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:22.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.793+0000 7fc0f2a00640 1 -- 172.21.15.6:0/2032387883 >> v1:172.21.15.146:6789/0 conn(0x7fc0f40c75b0 legacy=0x7fc0f415f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:22.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.793+0000 7fc0f2a00640 1 -- 172.21.15.6:0/2032387883 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0f406e980 con 0x7fc0f406dc50 2024-03-04T09:31:22.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.793+0000 7fc0f2a00640 1 -- 172.21.15.6:0/2032387883 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7fc0e0001670 con 0x7fc0f406dc50 2024-03-04T09:31:22.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.793+0000 7fc0f2a00640 1 -- 172.21.15.6:0/2032387883 >> v1:172.21.15.6:6789/0 conn(0x7fc0f406dc50 legacy=0x7fc0f40c6ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:22.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.793+0000 7fc0f2a00640 1 --2- 172.21.15.6:0/2032387883 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc0d80039e0 0x7fc0d8003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:22.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.793+0000 7fc0f2a00640 1 --2- 172.21.15.6:0/2032387883 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc0f413ed70 0x7fc0d8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:22.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.793+0000 7fc0f2a00640 1 --2- 172.21.15.6:0/2032387883 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc0d8004d40 0x7fc0d80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:22.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.793+0000 7fc0f2a00640 1 -- 172.21.15.6:0/2032387883 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0f415c980 con 0x7fc0d8004d40 2024-03-04T09:31:22.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.793+0000 7fc0f2a00640 1 -- 172.21.15.6:0/2032387883 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0d800bb50 con 0x7fc0d80039e0 2024-03-04T09:31:22.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.793+0000 7fc0f2a00640 1 -- 172.21.15.6:0/2032387883 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0d800bd50 con 0x7fc0f413ed70 2024-03-04T09:31:22.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.793+0000 7fc0f9200640 1 --2- 172.21.15.6:0/2032387883 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc0d8004d40 0x7fc0d80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:22.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.794+0000 7fc0f9200640 1 -- 172.21.15.6:0/2032387883 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc0f413ed70 msgr2=0x7fc0d8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:22.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.794+0000 7fc0f9200640 1 --2- 172.21.15.6:0/2032387883 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc0f413ed70 0x7fc0d8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:22.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.794+0000 7fc0f9200640 1 -- 172.21.15.6:0/2032387883 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc0d80039e0 msgr2=0x7fc0d8003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:22.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.794+0000 7fc0f9200640 1 --2- 172.21.15.6:0/2032387883 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc0d80039e0 0x7fc0d8003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:22.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.794+0000 7fc0f9200640 1 -- 172.21.15.6:0/2032387883 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0d8003520 con 0x7fc0d8004d40 2024-03-04T09:31:22.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.795+0000 7fc0f9200640 1 --2- 172.21.15.6:0/2032387883 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc0d8004d40 0x7fc0d80051c0 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7fc0ec001e50 tx=0x7fc0d8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62e33dea5e605151 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:22.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.795+0000 7fc0f2a00640 1 -- 172.21.15.6:0/2032387883 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0ec001a70 con 0x7fc0d8004d40 2024-03-04T09:31:22.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.795+0000 7fc0f2a00640 1 -- 172.21.15.6:0/2032387883 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0ec003b70 con 0x7fc0d8004d40 2024-03-04T09:31:22.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.795+0000 7fc0f2a00640 1 -- 172.21.15.6:0/2032387883 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0ec0050d0 con 0x7fc0d8004d40 2024-03-04T09:31:22.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.796+0000 7fc0fb200640 1 -- 172.21.15.6:0/2032387883 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc0d8004d40 msgr2=0x7fc0d80051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:22.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.796+0000 7fc0fb200640 1 --2- 172.21.15.6:0/2032387883 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc0d8004d40 0x7fc0d80051c0 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7fc0ec001e50 tx=0x7fc0d8005830 comp rx=0 tx=0).stop 2024-03-04T09:31:22.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.796+0000 7fc0f9200640 1 -- 172.21.15.6:0/2032387883 reap_dead start 2024-03-04T09:31:22.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.796+0000 7fc0fb200640 1 -- 172.21.15.6:0/2032387883 shutdown_connections 2024-03-04T09:31:22.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.796+0000 7fc0fb200640 1 -- 172.21.15.6:0/2032387883 >> 172.21.15.6:0/2032387883 conn(0x7fc0f40b2760 msgr2=0x7fc0f40b2b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:22.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.796+0000 7fc0fb200640 1 -- 172.21.15.6:0/2032387883 shutdown_connections 2024-03-04T09:31:22.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.796+0000 7fc0fb200640 1 -- 172.21.15.6:0/2032387883 wait complete. 2024-03-04T09:31:22.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.798+0000 7fc0fb200640 1 Processor -- start 2024-03-04T09:31:22.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.798+0000 7fc0fb200640 1 -- start start 2024-03-04T09:31:22.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.799+0000 7fc0fb200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc0d8004d40 0x7fc0f40b57d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:22.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.799+0000 7fc0fb200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc0f40c7be0 0x7fc0f40ada90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:22.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.799+0000 7fc0fb200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc0f40b5d10 0x7fc0f40adfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:22.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.799+0000 7fc0fb200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0f406f370 con 0x7fc0f40c7be0 2024-03-04T09:31:22.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.799+0000 7fc0fb200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0f406f1f0 con 0x7fc0d8004d40 2024-03-04T09:31:22.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.799+0000 7fc0fb200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0f406e0b0 con 0x7fc0f40b5d10 2024-03-04T09:31:22.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.799+0000 7fc0f3400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc0f40c7be0 0x7fc0f40ada90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:22.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.799+0000 7fc0f3400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc0f40c7be0 0x7fc0f40ada90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60932/0 (socket says 172.21.15.6:60932) 2024-03-04T09:31:22.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.799+0000 7fc0f3400640 1 -- 172.21.15.6:0/3443368216 learned_addr learned my addr 172.21.15.6:0/3443368216 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:31:22.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.799+0000 7fc0f3e00640 1 --2- 172.21.15.6:0/3443368216 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc0d8004d40 0x7fc0f40b57d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:22.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.799+0000 7fc0f9200640 1 --2- 172.21.15.6:0/3443368216 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc0f40b5d10 0x7fc0f40adfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:22.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.799+0000 7fc0f3400640 1 -- 172.21.15.6:0/3443368216 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc0f40b5d10 msgr2=0x7fc0f40adfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:22.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.799+0000 7fc0f3400640 1 --2- 172.21.15.6:0/3443368216 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc0f40b5d10 0x7fc0f40adfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:22.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.799+0000 7fc0f3400640 1 -- 172.21.15.6:0/3443368216 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc0d8004d40 msgr2=0x7fc0f40b57d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:22.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.799+0000 7fc0f3400640 1 --2- 172.21.15.6:0/3443368216 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc0d8004d40 0x7fc0f40b57d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:22.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.799+0000 7fc0f3400640 1 -- 172.21.15.6:0/3443368216 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0f40caad0 con 0x7fc0f40c7be0 2024-03-04T09:31:22.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.799+0000 7fc0f9200640 1 --2- 172.21.15.6:0/3443368216 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc0f40b5d10 0x7fc0f40adfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:31:22.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.800+0000 7fc0f3400640 1 --2- 172.21.15.6:0/3443368216 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc0f40c7be0 0x7fc0f40ada90 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7fc0e000c730 tx=0x7fc0e000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:22.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.800+0000 7fc0ebe00640 1 -- 172.21.15.6:0/3443368216 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0e0018070 con 0x7fc0f40c7be0 2024-03-04T09:31:22.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.800+0000 7fc0ebe00640 1 -- 172.21.15.6:0/3443368216 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0e00072d0 con 0x7fc0f40c7be0 2024-03-04T09:31:22.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.800+0000 7fc0ebe00640 1 -- 172.21.15.6:0/3443368216 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0e00047e0 con 0x7fc0f40c7be0 2024-03-04T09:31:22.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.800+0000 7fc0fb200640 1 -- 172.21.15.6:0/3443368216 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0f40b6290 con 0x7fc0f40c7be0 2024-03-04T09:31:22.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.800+0000 7fc0fb200640 1 -- 172.21.15.6:0/3443368216 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0f40ce6e0 con 0x7fc0f40c7be0 2024-03-04T09:31:22.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.801+0000 7fc0fb200640 1 -- 172.21.15.6:0/3443368216 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0f404f330 con 0x7fc0f40c7be0 2024-03-04T09:31:22.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.802+0000 7fc0ebe00640 1 -- 172.21.15.6:0/3443368216 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc0e001b070 con 0x7fc0f40c7be0 2024-03-04T09:31:22.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.803+0000 7fc0ebe00640 1 --2- 172.21.15.6:0/3443368216 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc0d406b4d0 0x7fc0d406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:22.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.803+0000 7fc0ebe00640 1 -- 172.21.15.6:0/3443368216 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc0e0089840 con 0x7fc0f40c7be0 2024-03-04T09:31:22.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.803+0000 7fc0f3e00640 1 --2- 172.21.15.6:0/3443368216 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc0d406b4d0 0x7fc0d406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:22.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.805+0000 7fc0f3e00640 1 --2- 172.21.15.6:0/3443368216 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc0d406b4d0 0x7fc0d406d990 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fc0e4004770 tx=0x7fc0e4008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:22.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:22.807+0000 7fc0ebe00640 1 -- 172.21.15.6:0/3443368216 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc0e0058750 con 0x7fc0f40c7be0 2024-03-04T09:31:23.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:23.037+0000 7fc0fb200640 1 -- 172.21.15.6:0/3443368216 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fc0f4071da0 con 0x7fc0f40c7be0 2024-03-04T09:31:23.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:23.642+0000 7fc0ebe00640 1 -- 172.21.15.6:0/3443368216 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v76) v1 ==== 118+0+0 (secure 0 0 0) 0x7fc0e0058050 con 0x7fc0f40c7be0 2024-03-04T09:31:23.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:23.646+0000 7fc0fb200640 1 -- 172.21.15.6:0/3443368216 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc0d406b4d0 msgr2=0x7fc0d406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:23.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:23.646+0000 7fc0fb200640 1 --2- 172.21.15.6:0/3443368216 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc0d406b4d0 0x7fc0d406d990 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fc0e4004770 tx=0x7fc0e4008040 comp rx=0 tx=0).stop 2024-03-04T09:31:23.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:23.646+0000 7fc0fb200640 1 -- 172.21.15.6:0/3443368216 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc0f40c7be0 msgr2=0x7fc0f40ada90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:23.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:23.646+0000 7fc0fb200640 1 --2- 172.21.15.6:0/3443368216 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc0f40c7be0 0x7fc0f40ada90 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7fc0e000c730 tx=0x7fc0e000cbf0 comp rx=0 tx=0).stop 2024-03-04T09:31:23.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:23.646+0000 7fc0fb200640 1 -- 172.21.15.6:0/3443368216 shutdown_connections 2024-03-04T09:31:23.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:23.646+0000 7fc0fb200640 1 --2- 172.21.15.6:0/3443368216 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc0d406b4d0 0x7fc0d406d990 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:23.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:23.646+0000 7fc0fb200640 1 --2- 172.21.15.6:0/3443368216 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc0f40b5d10 0x7fc0f40adfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:23.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:23.646+0000 7fc0fb200640 1 --2- 172.21.15.6:0/3443368216 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc0f40c7be0 0x7fc0f40ada90 unknown :-1 s=CLOSED pgs=1242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:23.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:23.646+0000 7fc0fb200640 1 --2- 172.21.15.6:0/3443368216 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc0d8004d40 0x7fc0f40b57d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:23.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:23.646+0000 7fc0fb200640 1 -- 172.21.15.6:0/3443368216 >> 172.21.15.6:0/3443368216 conn(0x7fc0f40b2760 msgr2=0x7fc0f406afb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:23.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:23.647+0000 7fc0fb200640 1 -- 172.21.15.6:0/3443368216 shutdown_connections 2024-03-04T09:31:23.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:23.647+0000 7fc0fb200640 1 -- 172.21.15.6:0/3443368216 wait complete. 2024-03-04T09:31:23.663 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-04T09:31:23.664 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-04T09:31:23.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:23.878+0000 7f3cdd800640 1 Processor -- start 2024-03-04T09:31:23.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:23.878+0000 7f3cdd800640 1 -- start start 2024-03-04T09:31:23.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:23.880+0000 7f3cdd800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f3cd80b4870 0x7f3cd8070f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:23.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:23.880+0000 7f3cdd800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f3cd8071490 0x7f3cd816eb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:23.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:23.880+0000 7f3cdd800640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cd816b700 con 0x7f3cd808bb10 2024-03-04T09:31:23.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:23.880+0000 7f3cdd800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3cd816b9e0 con 0x7f3cd8071490 2024-03-04T09:31:23.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:23.880+0000 7f3cdd800640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3cd816bcc0 con 0x7f3cd80b4870 2024-03-04T09:31:23.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:23.880+0000 7f3cdce00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f3cd8071490 0x7f3cd816eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:23.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:23.880+0000 7f3cdce00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f3cd8071490 0x7f3cd816eb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60946/0 (socket says 172.21.15.6:60946) 2024-03-04T09:31:23.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:23.880+0000 7f3cdce00640 1 -- 172.21.15.6:0/121669720 learned_addr learned my addr 172.21.15.6:0/121669720 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:31:23.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:23.881+0000 7f3cd7400640 1 --2- 172.21.15.6:0/121669720 >> v2:172.21.15.146:3300/0 conn(0x7f3cd80b4870 0x7f3cd8070f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:23.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:23.881+0000 7f3cd6a00640 1 -- 172.21.15.6:0/121669720 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2447119631 0 0) 0x7f3cd816b700 con 0x7f3cd808bb10 2024-03-04T09:31:23.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:23.902+0000 7f3cd6a00640 1 -- 172.21.15.6:0/121669720 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3cbc003500 con 0x7f3cd808bb10 2024-03-04T09:31:23.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:23.903+0000 7f3cdce00640 1 -- 172.21.15.6:0/121669720 >> v2:172.21.15.146:3300/0 conn(0x7f3cd80b4870 msgr2=0x7f3cd8070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:23.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:23.903+0000 7f3cdce00640 1 --2- 172.21.15.6:0/121669720 >> v2:172.21.15.146:3300/0 conn(0x7f3cd80b4870 0x7f3cd8070f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:23.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:23.903+0000 7f3cdce00640 1 -- 172.21.15.6:0/121669720 >> v1:172.21.15.146:6789/0 conn(0x7f3cd808bb10 legacy=0x7f3cd816aee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:23.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:23.903+0000 7f3cdce00640 1 -- 172.21.15.6:0/121669720 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cd816f0d0 con 0x7f3cd8071490 2024-03-04T09:31:23.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:23.904+0000 7f3cdce00640 1 --2- 172.21.15.6:0/121669720 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3cd8071490 0x7f3cd816eb90 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7f3cd8070e60 tx=0x7f3cd002bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a9cfe03675e8aa5 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:23.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:23.904+0000 7f3cd6a00640 1 -- 172.21.15.6:0/121669720 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cd000b040 con 0x7f3cd8071490 2024-03-04T09:31:23.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:23.904+0000 7f3cd6a00640 1 -- 172.21.15.6:0/121669720 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cd0004230 con 0x7f3cd8071490 2024-03-04T09:31:23.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:23.904+0000 7f3cd6a00640 1 -- 172.21.15.6:0/121669720 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cd002c6f0 con 0x7f3cd8071490 2024-03-04T09:31:23.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:23.905+0000 7f3cdd800640 1 -- 172.21.15.6:0/121669720 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3cd8071490 msgr2=0x7f3cd816eb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:23.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:23.905+0000 7f3cdd800640 1 --2- 172.21.15.6:0/121669720 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3cd8071490 0x7f3cd816eb90 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7f3cd8070e60 tx=0x7f3cd002bd50 comp rx=0 tx=0).stop 2024-03-04T09:31:23.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:23.905+0000 7f3cdd800640 1 -- 172.21.15.6:0/121669720 shutdown_connections 2024-03-04T09:31:23.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:23.905+0000 7f3cdd800640 1 --2- 172.21.15.6:0/121669720 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3cd8071490 0x7f3cd816eb90 unknown :-1 s=CLOSED pgs=1243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:23.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:23.905+0000 7f3cdd800640 1 --2- 172.21.15.6:0/121669720 >> v2:172.21.15.146:3300/0 conn(0x7f3cd80b4870 0x7f3cd8070f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:23.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:23.905+0000 7f3cdd800640 1 -- 172.21.15.6:0/121669720 >> 172.21.15.6:0/121669720 conn(0x7f3cd806afb0 msgr2=0x7f3cd806b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:23.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:23.905+0000 7f3cdd800640 1 -- 172.21.15.6:0/121669720 shutdown_connections 2024-03-04T09:31:23.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:23.906+0000 7f3cdd800640 1 -- 172.21.15.6:0/121669720 wait complete. 2024-03-04T09:31:23.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:23.907+0000 7f3cdd800640 1 Processor -- start 2024-03-04T09:31:23.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:23.907+0000 7f3cdd800640 1 -- start start 2024-03-04T09:31:23.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:23.908+0000 7f3cdd800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3cd8071490 0x7f3cd8072340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:23.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:23.908+0000 7f3cdd800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3cd808bb10 0x7f3cd82035e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:23.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:23.908+0000 7f3cdd800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3cd80b4870 0x7f3cd8161950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:23.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:23.908+0000 7f3cdd800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cd80b1860 con 0x7f3cd808bb10 2024-03-04T09:31:23.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:23.908+0000 7f3cdd800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cd80b16e0 con 0x7f3cd8071490 2024-03-04T09:31:23.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:23.908+0000 7f3cdd800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cd80b19e0 con 0x7f3cd80b4870 2024-03-04T09:31:23.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:23.909+0000 7f3cd7400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3cd808bb10 0x7f3cd82035e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:23.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:23.909+0000 7f3cd7400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3cd808bb10 0x7f3cd82035e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60948/0 (socket says 172.21.15.6:60948) 2024-03-04T09:31:23.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:23.909+0000 7f3cd7400640 1 -- 172.21.15.6:0/866520328 learned_addr learned my addr 172.21.15.6:0/866520328 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:31:23.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:23.909+0000 7f3cd7e00640 1 --2- 172.21.15.6:0/866520328 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3cd8071490 0x7f3cd8072340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:23.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:23.909+0000 7f3cdce00640 1 --2- 172.21.15.6:0/866520328 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3cd80b4870 0x7f3cd8161950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:23.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:23.909+0000 7f3cd7400640 1 -- 172.21.15.6:0/866520328 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3cd80b4870 msgr2=0x7f3cd8161950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:23.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:23.909+0000 7f3cd7400640 1 --2- 172.21.15.6:0/866520328 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3cd80b4870 0x7f3cd8161950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:23.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:23.909+0000 7f3cd7400640 1 -- 172.21.15.6:0/866520328 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3cd8071490 msgr2=0x7f3cd8072340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:23.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:23.909+0000 7f3cd7400640 1 --2- 172.21.15.6:0/866520328 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3cd8071490 0x7f3cd8072340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:23.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:23.909+0000 7f3cd7400640 1 -- 172.21.15.6:0/866520328 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cd8202540 con 0x7f3cd808bb10 2024-03-04T09:31:23.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:23.909+0000 7f3cd7400640 1 --2- 172.21.15.6:0/866520328 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3cd808bb10 0x7f3cd82035e0 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7f3cc4007cf0 tx=0x7f3cc4013510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:23.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:23.910+0000 7f3ccfe00640 1 -- 172.21.15.6:0/866520328 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cc4013e30 con 0x7f3cd808bb10 2024-03-04T09:31:23.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:23.910+0000 7f3ccfe00640 1 -- 172.21.15.6:0/866520328 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cc400e0d0 con 0x7f3cd808bb10 2024-03-04T09:31:23.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:23.910+0000 7f3cdd800640 1 -- 172.21.15.6:0/866520328 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cd8202820 con 0x7f3cd808bb10 2024-03-04T09:31:23.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:23.910+0000 7f3ccfe00640 1 -- 172.21.15.6:0/866520328 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cc400e4a0 con 0x7f3cd808bb10 2024-03-04T09:31:23.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:23.910+0000 7f3cdd800640 1 -- 172.21.15.6:0/866520328 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cd8202de0 con 0x7f3cd808bb10 2024-03-04T09:31:23.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:23.911+0000 7f3cdd800640 1 -- 172.21.15.6:0/866520328 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cd804f330 con 0x7f3cd808bb10 2024-03-04T09:31:23.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:23.912+0000 7f3ccfe00640 1 -- 172.21.15.6:0/866520328 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3cc4020020 con 0x7f3cd808bb10 2024-03-04T09:31:23.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:23.913+0000 7f3ccfe00640 1 --2- 172.21.15.6:0/866520328 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f3cb806b500 0x7f3cb806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:23.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:23.914+0000 7f3ccfe00640 1 -- 172.21.15.6:0/866520328 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3cc408e740 con 0x7f3cd808bb10 2024-03-04T09:31:23.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:23.914+0000 7f3cd7e00640 1 --2- 172.21.15.6:0/866520328 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f3cb806b500 0x7f3cb806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:23.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:23.915+0000 7f3cd7e00640 1 --2- 172.21.15.6:0/866520328 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f3cb806b500 0x7f3cb806d9c0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f3cc8001d50 tx=0x7f3cc8009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:23.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:23.918+0000 7f3ccfe00640 1 -- 172.21.15.6:0/866520328 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3cc4015040 con 0x7f3cd808bb10 2024-03-04T09:31:24.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:24.170+0000 7f3cdd800640 1 -- 172.21.15.6:0/866520328 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f3cd806d4f0 con 0x7f3cd808bb10 2024-03-04T09:31:24.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:24.652+0000 7f3ccfe00640 1 -- 172.21.15.6:0/866520328 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v77) v1 ==== 122+0+0 (secure 0 0 0) 0x7f3cc405d5b0 con 0x7f3cd808bb10 2024-03-04T09:31:24.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:24.655+0000 7f3cdd800640 1 -- 172.21.15.6:0/866520328 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f3cb806b500 msgr2=0x7f3cb806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:24.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:24.655+0000 7f3cdd800640 1 --2- 172.21.15.6:0/866520328 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f3cb806b500 0x7f3cb806d9c0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f3cc8001d50 tx=0x7f3cc8009040 comp rx=0 tx=0).stop 2024-03-04T09:31:24.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:24.655+0000 7f3cdd800640 1 -- 172.21.15.6:0/866520328 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3cd808bb10 msgr2=0x7f3cd82035e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:24.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:24.655+0000 7f3cdd800640 1 --2- 172.21.15.6:0/866520328 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3cd808bb10 0x7f3cd82035e0 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7f3cc4007cf0 tx=0x7f3cc4013510 comp rx=0 tx=0).stop 2024-03-04T09:31:24.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:24.655+0000 7f3cdd800640 1 -- 172.21.15.6:0/866520328 shutdown_connections 2024-03-04T09:31:24.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:24.655+0000 7f3cdd800640 1 --2- 172.21.15.6:0/866520328 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f3cb806b500 0x7f3cb806d9c0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:24.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:24.655+0000 7f3cdd800640 1 --2- 172.21.15.6:0/866520328 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3cd80b4870 0x7f3cd8161950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:24.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:24.655+0000 7f3cdd800640 1 --2- 172.21.15.6:0/866520328 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3cd808bb10 0x7f3cd82035e0 unknown :-1 s=CLOSED pgs=1244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:24.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:24.655+0000 7f3cdd800640 1 --2- 172.21.15.6:0/866520328 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3cd8071490 0x7f3cd8072340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:24.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:24.655+0000 7f3cdd800640 1 -- 172.21.15.6:0/866520328 >> 172.21.15.6:0/866520328 conn(0x7f3cd806afb0 msgr2=0x7f3cd806b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:24.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:24.656+0000 7f3cdd800640 1 -- 172.21.15.6:0/866520328 shutdown_connections 2024-03-04T09:31:24.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:24.656+0000 7f3cdd800640 1 -- 172.21.15.6:0/866520328 wait complete. 2024-03-04T09:31:24.672 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-04T09:31:24.672 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:31:24.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:24.880+0000 7f8e53e00640 1 Processor -- start 2024-03-04T09:31:24.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:24.880+0000 7f8e53e00640 1 -- start start 2024-03-04T09:31:24.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:24.882+0000 7f8e53e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f8e4c15e2e0 0x7f8e4c0d0830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:24.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:24.882+0000 7f8e53e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f8e4c0d0d70 0x7f8e4c0d11d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:24.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:24.882+0000 7f8e53e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e4c0b3940 con 0x7f8e4c0d1b30 2024-03-04T09:31:24.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:24.882+0000 7f8e53e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e4c0d4e70 con 0x7f8e4c15e2e0 2024-03-04T09:31:24.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:24.882+0000 7f8e53e00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e4c0b3c40 con 0x7f8e4c0d0d70 2024-03-04T09:31:24.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:24.882+0000 7f8e51e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f8e4c0d1b30 0x7f8e4c0d1fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:56970/0 (socket says 172.21.15.6:56970) 2024-03-04T09:31:24.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:24.882+0000 7f8e51e00640 1 -- 172.21.15.6:0/2532793792 learned_addr learned my addr 172.21.15.6:0/2532793792 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:31:24.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:24.882+0000 7f8e51400640 1 --2- 172.21.15.6:0/2532793792 >> v2:172.21.15.6:3300/0 conn(0x7f8e4c15e2e0 0x7f8e4c0d0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:24.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:24.882+0000 7f8e50a00640 1 --2- 172.21.15.6:0/2532793792 >> v2:172.21.15.146:3300/0 conn(0x7f8e4c0d0d70 0x7f8e4c0d11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:24.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:24.883+0000 7f8e51400640 1 -- 172.21.15.6:0/2532793792 >> v2:172.21.15.146:3300/0 conn(0x7f8e4c0d0d70 msgr2=0x7f8e4c0d11d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:24.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:24.883+0000 7f8e51400640 1 --2- 172.21.15.6:0/2532793792 >> v2:172.21.15.146:3300/0 conn(0x7f8e4c0d0d70 0x7f8e4c0d11d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:24.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:24.883+0000 7f8e51400640 1 -- 172.21.15.6:0/2532793792 >> v1:172.21.15.6:6789/0 conn(0x7f8e4c0d1b30 legacy=0x7f8e4c0d1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:24.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:24.883+0000 7f8e51400640 1 -- 172.21.15.6:0/2532793792 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e4c0cf580 con 0x7f8e4c15e2e0 2024-03-04T09:31:24.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:24.885+0000 7f8e51400640 1 --2- 172.21.15.6:0/2532793792 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8e4c15e2e0 0x7f8e4c0d0830 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7f8e4c0d3510 tx=0x7f8e4002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f1961587a28dbd2 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:24.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:24.885+0000 7f8e4be00640 1 -- 172.21.15.6:0/2532793792 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e40035020 con 0x7f8e4c15e2e0 2024-03-04T09:31:24.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:24.885+0000 7f8e4be00640 1 -- 172.21.15.6:0/2532793792 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e40031830 con 0x7f8e4c15e2e0 2024-03-04T09:31:24.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:24.885+0000 7f8e4be00640 1 -- 172.21.15.6:0/2532793792 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e40031c00 con 0x7f8e4c15e2e0 2024-03-04T09:31:24.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:24.885+0000 7f8e53e00640 1 -- 172.21.15.6:0/2532793792 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8e4c15e2e0 msgr2=0x7f8e4c0d0830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:24.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:24.885+0000 7f8e53e00640 1 --2- 172.21.15.6:0/2532793792 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8e4c15e2e0 0x7f8e4c0d0830 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7f8e4c0d3510 tx=0x7f8e4002b8d0 comp rx=0 tx=0).stop 2024-03-04T09:31:24.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:24.886+0000 7f8e53e00640 1 -- 172.21.15.6:0/2532793792 shutdown_connections 2024-03-04T09:31:24.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:24.886+0000 7f8e53e00640 1 --2- 172.21.15.6:0/2532793792 >> v2:172.21.15.146:3300/0 conn(0x7f8e4c0d0d70 0x7f8e4c0d11d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:24.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:24.886+0000 7f8e53e00640 1 --2- 172.21.15.6:0/2532793792 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8e4c15e2e0 0x7f8e4c0d0830 unknown :-1 s=CLOSED pgs=1246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:24.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:24.886+0000 7f8e53e00640 1 -- 172.21.15.6:0/2532793792 >> 172.21.15.6:0/2532793792 conn(0x7f8e4c0d3f60 msgr2=0x7f8e4c0d4370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:24.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:24.886+0000 7f8e53e00640 1 -- 172.21.15.6:0/2532793792 shutdown_connections 2024-03-04T09:31:24.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:24.886+0000 7f8e53e00640 1 -- 172.21.15.6:0/2532793792 wait complete. 2024-03-04T09:31:24.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:24.887+0000 7f8e53e00640 1 Processor -- start 2024-03-04T09:31:24.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:24.888+0000 7f8e53e00640 1 -- start start 2024-03-04T09:31:24.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:24.889+0000 7f8e53e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8e4c0d0d70 0x7f8e4c0b9250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:24.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:24.889+0000 7f8e53e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8e4c0d1b30 0x7f8e4c14c380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:24.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:24.889+0000 7f8e53e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8e4c15e2e0 0x7f8e4c0c88e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:24.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:24.889+0000 7f8e53e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e4c0cfe20 con 0x7f8e4c15e2e0 2024-03-04T09:31:24.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:24.889+0000 7f8e53e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e4c0cfb40 con 0x7f8e4c0d1b30 2024-03-04T09:31:24.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:24.889+0000 7f8e53e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e4c0b50e0 con 0x7f8e4c0d0d70 2024-03-04T09:31:24.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:24.889+0000 7f8e51e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8e4c15e2e0 0x7f8e4c0c88e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:24.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:24.889+0000 7f8e51e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8e4c15e2e0 0x7f8e4c0c88e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60966/0 (socket says 172.21.15.6:60966) 2024-03-04T09:31:24.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:24.889+0000 7f8e51e00640 1 -- 172.21.15.6:0/3702493824 learned_addr learned my addr 172.21.15.6:0/3702493824 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:31:24.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:24.889+0000 7f8e50a00640 1 --2- 172.21.15.6:0/3702493824 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8e4c0d1b30 0x7f8e4c14c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:24.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:24.889+0000 7f8e51400640 1 --2- 172.21.15.6:0/3702493824 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8e4c0d0d70 0x7f8e4c0b9250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:24.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:24.889+0000 7f8e51e00640 1 -- 172.21.15.6:0/3702493824 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8e4c0d0d70 msgr2=0x7f8e4c0b9250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:24.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:24.889+0000 7f8e51e00640 1 --2- 172.21.15.6:0/3702493824 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8e4c0d0d70 0x7f8e4c0b9250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:24.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:24.889+0000 7f8e51e00640 1 -- 172.21.15.6:0/3702493824 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8e4c0d1b30 msgr2=0x7f8e4c14c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:24.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:24.889+0000 7f8e51e00640 1 --2- 172.21.15.6:0/3702493824 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8e4c0d1b30 0x7f8e4c14c380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:24.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:24.889+0000 7f8e51e00640 1 -- 172.21.15.6:0/3702493824 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e4c14c8c0 con 0x7f8e4c15e2e0 2024-03-04T09:31:24.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:24.889+0000 7f8e50a00640 1 --2- 172.21.15.6:0/3702493824 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8e4c0d1b30 0x7f8e4c14c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:31:24.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:24.890+0000 7f8e51e00640 1 --2- 172.21.15.6:0/3702493824 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8e4c15e2e0 0x7f8e4c0c88e0 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7f8e4400cbc0 tx=0x7f8e4400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:24.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:24.890+0000 7f8e49600640 1 -- 172.21.15.6:0/3702493824 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e44017070 con 0x7f8e4c15e2e0 2024-03-04T09:31:24.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:24.890+0000 7f8e49600640 1 -- 172.21.15.6:0/3702493824 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e4400ad40 con 0x7f8e4c15e2e0 2024-03-04T09:31:24.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:24.890+0000 7f8e49600640 1 -- 172.21.15.6:0/3702493824 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e44012600 con 0x7f8e4c15e2e0 2024-03-04T09:31:24.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:24.890+0000 7f8e53e00640 1 -- 172.21.15.6:0/3702493824 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e4c0b81b0 con 0x7f8e4c15e2e0 2024-03-04T09:31:24.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:24.890+0000 7f8e53e00640 1 -- 172.21.15.6:0/3702493824 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e4c0b8770 con 0x7f8e4c15e2e0 2024-03-04T09:31:24.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:24.891+0000 7f8e53e00640 1 -- 172.21.15.6:0/3702493824 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e4c04f3a0 con 0x7f8e4c15e2e0 2024-03-04T09:31:24.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:24.892+0000 7f8e49600640 1 -- 172.21.15.6:0/3702493824 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8e4401a070 con 0x7f8e4c15e2e0 2024-03-04T09:31:24.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:24.893+0000 7f8e49600640 1 --2- 172.21.15.6:0/3702493824 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f8e2c06b550 0x7f8e2c06da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:24.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:24.893+0000 7f8e49600640 1 -- 172.21.15.6:0/3702493824 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8e44089df0 con 0x7f8e4c15e2e0 2024-03-04T09:31:24.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:24.894+0000 7f8e51400640 1 --2- 172.21.15.6:0/3702493824 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f8e2c06b550 0x7f8e2c06da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:24.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:24.895+0000 7f8e51400640 1 --2- 172.21.15.6:0/3702493824 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f8e2c06b550 0x7f8e2c06da10 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f8e40036000 tx=0x7f8e40008030 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:24.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:24.902+0000 7f8e49600640 1 -- 172.21.15.6:0/3702493824 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8e4c14c8c0 con 0x7f8e4c15e2e0 2024-03-04T09:31:25.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.147+0000 7f8e53e00640 1 -- 172.21.15.6:0/3702493824 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8e4c15d900 con 0x7f8e4c15e2e0 2024-03-04T09:31:25.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.148+0000 7f8e49600640 1 -- 172.21.15.6:0/3702493824 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+4973 (secure 0 0 0) 0x7f8e44058690 con 0x7f8e4c15e2e0 2024-03-04T09:31:25.145 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:31:25.146 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":77,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5792,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6836/4034557380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":4034557380},{"type":"v1","addr":"172.21.15.146:6836","nonce":4034557380}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":72},{"gid":6370,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/348873623","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":348873623},{"type":"v1","addr":"172.21.15.6:6835","nonce":348873623}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":72},{"gid":6382,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/763528142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":763528142},{"type":"v1","addr":"172.21.15.146:6837","nonce":763528142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":72}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:31:21.114351+0000","modified":"2024-03-04T09:31:24.172076+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7560},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7560":{"gid":7560,"name":"a","rank":0,"incarnation":73,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2697931210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2697931210},{"type":"v1","addr":"172.21.15.6:6834","nonce":2697931210}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7560,"qdb_cluster":[7560]},"id":8}]} 2024-03-04T09:31:25.146 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 77 2024-03-04T09:31:25.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.151+0000 7f8e53e00640 1 -- 172.21.15.6:0/3702493824 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f8e2c06b550 msgr2=0x7f8e2c06da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:25.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.151+0000 7f8e53e00640 1 --2- 172.21.15.6:0/3702493824 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f8e2c06b550 0x7f8e2c06da10 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f8e40036000 tx=0x7f8e40008030 comp rx=0 tx=0).stop 2024-03-04T09:31:25.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.151+0000 7f8e53e00640 1 -- 172.21.15.6:0/3702493824 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8e4c15e2e0 msgr2=0x7f8e4c0c88e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:25.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.151+0000 7f8e53e00640 1 --2- 172.21.15.6:0/3702493824 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8e4c15e2e0 0x7f8e4c0c88e0 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7f8e4400cbc0 tx=0x7f8e4400a430 comp rx=0 tx=0).stop 2024-03-04T09:31:25.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.151+0000 7f8e53e00640 1 -- 172.21.15.6:0/3702493824 shutdown_connections 2024-03-04T09:31:25.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.151+0000 7f8e53e00640 1 --2- 172.21.15.6:0/3702493824 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8e4c15e2e0 0x7f8e4c0c88e0 unknown :-1 s=CLOSED pgs=1247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:25.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.151+0000 7f8e53e00640 1 --2- 172.21.15.6:0/3702493824 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8e4c0d1b30 0x7f8e4c14c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:25.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.151+0000 7f8e53e00640 1 --2- 172.21.15.6:0/3702493824 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f8e2c06b550 0x7f8e2c06da10 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:25.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.151+0000 7f8e53e00640 1 --2- 172.21.15.6:0/3702493824 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8e4c0d0d70 0x7f8e4c0b9250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:25.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.151+0000 7f8e53e00640 1 -- 172.21.15.6:0/3702493824 >> 172.21.15.6:0/3702493824 conn(0x7f8e4c0d3f60 msgr2=0x7f8e4c06c1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:25.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.151+0000 7f8e53e00640 1 -- 172.21.15.6:0/3702493824 shutdown_connections 2024-03-04T09:31:25.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.151+0000 7f8e53e00640 1 -- 172.21.15.6:0/3702493824 wait complete. 2024-03-04T09:31:25.167 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T09:31:25.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.377+0000 7f74c9600640 1 Processor -- start 2024-03-04T09:31:25.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.377+0000 7f74c9600640 1 -- start start 2024-03-04T09:31:25.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.379+0000 7f74c9600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f74c406afb0 0x7f74c406b410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:25.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.379+0000 7f74c9600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f74c40cf130 0x7f74c40cf5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:25.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.379+0000 7f74c9600640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74c40d0910 con 0x7f74c4144c00 2024-03-04T09:31:25.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.379+0000 7f74c9600640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74c406b950 con 0x7f74c406afb0 2024-03-04T09:31:25.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.379+0000 7f74c9600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74c40d0bf0 con 0x7f74c40cf130 2024-03-04T09:31:25.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.379+0000 7f74c3e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f74c4144c00 0x7f74c4145280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:56978/0 (socket says 172.21.15.6:56978) 2024-03-04T09:31:25.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.379+0000 7f74c3e00640 1 -- 172.21.15.6:0/3064711806 learned_addr learned my addr 172.21.15.6:0/3064711806 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:31:25.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.379+0000 7f74c2a00640 1 -- 172.21.15.6:0/3064711806 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1082880435 0 0) 0x7f74c40d0910 con 0x7f74c4144c00 2024-03-04T09:31:25.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.379+0000 7f74c8c00640 1 --2- 172.21.15.6:0/3064711806 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f74c40cf130 0x7f74c40cf5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:25.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.380+0000 7f74c2a00640 1 -- 172.21.15.6:0/3064711806 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74a8003520 con 0x7f74c4144c00 2024-03-04T09:31:25.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.380+0000 7f74c3400640 1 --2- 172.21.15.6:0/3064711806 >> v2:172.21.15.6:3300/0 conn(0x7f74c406afb0 0x7f74c406b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:25.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.380+0000 7f74c2a00640 1 -- 172.21.15.6:0/3064711806 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2750052691 0 0) 0x7f74a8003520 con 0x7f74c4144c00 2024-03-04T09:31:25.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.380+0000 7f74c2a00640 1 -- 172.21.15.6:0/3064711806 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f74c40cf130 msgr2=0x7f74c40cf5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:25.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.380+0000 7f74c2a00640 1 --2- 172.21.15.6:0/3064711806 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f74c40cf130 0x7f74c40cf5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:25.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.380+0000 7f74c2a00640 1 -- 172.21.15.6:0/3064711806 >> v2:172.21.15.6:3300/0 conn(0x7f74c406afb0 msgr2=0x7f74c406b410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:25.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.380+0000 7f74c2a00640 1 --2- 172.21.15.6:0/3064711806 >> v2:172.21.15.6:3300/0 conn(0x7f74c406afb0 0x7f74c406b410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:25.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.380+0000 7f74c2a00640 1 -- 172.21.15.6:0/3064711806 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74c40d0ed0 con 0x7f74c4144c00 2024-03-04T09:31:25.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.380+0000 7f74c2a00640 1 -- 172.21.15.6:0/3064711806 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7f74b40019f0 con 0x7f74c4144c00 2024-03-04T09:31:25.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.380+0000 7f74c2a00640 1 -- 172.21.15.6:0/3064711806 >> v1:172.21.15.6:6789/0 conn(0x7f74c4144c00 legacy=0x7f74c4145280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:25.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.380+0000 7f74c2a00640 1 --2- 172.21.15.6:0/3064711806 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f74c40cf130 0x7f74a80039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:25.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.380+0000 7f74c2a00640 1 --2- 172.21.15.6:0/3064711806 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f74a8003f80 0x7f74a8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:25.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.380+0000 7f74c2a00640 1 --2- 172.21.15.6:0/3064711806 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f74a8004d40 0x7f74a80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:25.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.380+0000 7f74c2a00640 1 -- 172.21.15.6:0/3064711806 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74c40d0bf0 con 0x7f74a8004d40 2024-03-04T09:31:25.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.380+0000 7f74c2a00640 1 -- 172.21.15.6:0/3064711806 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74a800b9f0 con 0x7f74a8003f80 2024-03-04T09:31:25.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.380+0000 7f74c2a00640 1 -- 172.21.15.6:0/3064711806 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74a800bbf0 con 0x7f74c40cf130 2024-03-04T09:31:25.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.381+0000 7f74c8c00640 1 --2- 172.21.15.6:0/3064711806 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f74a8004d40 0x7f74a80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:25.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.381+0000 7f74c3e00640 1 --2- 172.21.15.6:0/3064711806 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f74c40cf130 0x7f74a80039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:25.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.381+0000 7f74c3400640 1 --2- 172.21.15.6:0/3064711806 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f74a8003f80 0x7f74a8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:25.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.381+0000 7f74c8c00640 1 -- 172.21.15.6:0/3064711806 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f74c40cf130 msgr2=0x7f74a80039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:25.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.381+0000 7f74c8c00640 1 --2- 172.21.15.6:0/3064711806 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f74c40cf130 0x7f74a80039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:25.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.381+0000 7f74c8c00640 1 -- 172.21.15.6:0/3064711806 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f74a8003f80 msgr2=0x7f74a8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:25.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.381+0000 7f74c8c00640 1 --2- 172.21.15.6:0/3064711806 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f74a8003f80 0x7f74a8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:25.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.381+0000 7f74c8c00640 1 -- 172.21.15.6:0/3064711806 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74a8003520 con 0x7f74a8004d40 2024-03-04T09:31:25.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.381+0000 7f74c3e00640 1 --2- 172.21.15.6:0/3064711806 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f74c40cf130 0x7f74a80039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:31:25.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.382+0000 7f74c8c00640 1 --2- 172.21.15.6:0/3064711806 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f74a8004d40 0x7f74a80051c0 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7f74bc009ac0 tx=0x7f74a8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f95da51d0093634e server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:25.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.382+0000 7f74c2a00640 1 -- 172.21.15.6:0/3064711806 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74bc039070 con 0x7f74a8004d40 2024-03-04T09:31:25.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.382+0000 7f74c2a00640 1 -- 172.21.15.6:0/3064711806 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74bc02c6b0 con 0x7f74a8004d40 2024-03-04T09:31:25.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.382+0000 7f74c2a00640 1 -- 172.21.15.6:0/3064711806 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74bc009690 con 0x7f74a8004d40 2024-03-04T09:31:25.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.383+0000 7f74c9600640 1 -- 172.21.15.6:0/3064711806 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f74a8004d40 msgr2=0x7f74a80051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:25.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.383+0000 7f74c9600640 1 --2- 172.21.15.6:0/3064711806 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f74a8004d40 0x7f74a80051c0 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7f74bc009ac0 tx=0x7f74a8005830 comp rx=0 tx=0).stop 2024-03-04T09:31:25.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.383+0000 7f74c8c00640 1 -- 172.21.15.6:0/3064711806 reap_dead start 2024-03-04T09:31:25.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.383+0000 7f74c9600640 1 -- 172.21.15.6:0/3064711806 shutdown_connections 2024-03-04T09:31:25.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.383+0000 7f74c9600640 1 -- 172.21.15.6:0/3064711806 >> 172.21.15.6:0/3064711806 conn(0x7f74c406c0c0 msgr2=0x7f74c406c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:25.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.383+0000 7f74c9600640 1 -- 172.21.15.6:0/3064711806 shutdown_connections 2024-03-04T09:31:25.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.383+0000 7f74c9600640 1 -- 172.21.15.6:0/3064711806 wait complete. 2024-03-04T09:31:25.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.386+0000 7f74c9600640 1 Processor -- start 2024-03-04T09:31:25.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.386+0000 7f74c9600640 1 -- start start 2024-03-04T09:31:25.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.388+0000 7f74c9600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f74a8004d40 0x7f74c415b290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:25.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.388+0000 7f74c9600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f74c4154130 0x7f74c41545b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:25.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.388+0000 7f74c9600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f74c41370b0 0x7f74c4137530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:25.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.388+0000 7f74c9600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74c4144fe0 con 0x7f74c41370b0 2024-03-04T09:31:25.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.388+0000 7f74c9600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74c40d1490 con 0x7f74a8004d40 2024-03-04T09:31:25.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.388+0000 7f74c9600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74c4145990 con 0x7f74c4154130 2024-03-04T09:31:25.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.388+0000 7f74c8c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f74c41370b0 0x7f74c4137530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:25.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.388+0000 7f74c8c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f74c41370b0 0x7f74c4137530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:32772/0 (socket says 172.21.15.6:32772) 2024-03-04T09:31:25.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.388+0000 7f74c8c00640 1 -- 172.21.15.6:0/2258076797 learned_addr learned my addr 172.21.15.6:0/2258076797 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:31:25.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.389+0000 7f74c3e00640 1 --2- 172.21.15.6:0/2258076797 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f74a8004d40 0x7f74c415b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:25.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.389+0000 7f74c3400640 1 --2- 172.21.15.6:0/2258076797 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f74c4154130 0x7f74c41545b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:25.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.389+0000 7f74c8c00640 1 -- 172.21.15.6:0/2258076797 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f74c4154130 msgr2=0x7f74c41545b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:25.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.389+0000 7f74c8c00640 1 --2- 172.21.15.6:0/2258076797 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f74c4154130 0x7f74c41545b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:25.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.389+0000 7f74c8c00640 1 -- 172.21.15.6:0/2258076797 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f74a8004d40 msgr2=0x7f74c415b290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:25.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.389+0000 7f74c8c00640 1 --2- 172.21.15.6:0/2258076797 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f74a8004d40 0x7f74c415b290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:25.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.389+0000 7f74c8c00640 1 -- 172.21.15.6:0/2258076797 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74c40b06f0 con 0x7f74c41370b0 2024-03-04T09:31:25.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.389+0000 7f74c3400640 1 --2- 172.21.15.6:0/2258076797 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f74c4154130 0x7f74c41545b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:31:25.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.389+0000 7f74c8c00640 1 --2- 172.21.15.6:0/2258076797 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f74c41370b0 0x7f74c4137530 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7f74bc02c320 tx=0x7f74bc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:25.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.389+0000 7f74bbe00640 1 -- 172.21.15.6:0/2258076797 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74bc039070 con 0x7f74c41370b0 2024-03-04T09:31:25.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.390+0000 7f74bbe00640 1 -- 172.21.15.6:0/2258076797 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74bc043070 con 0x7f74c41370b0 2024-03-04T09:31:25.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.390+0000 7f74bbe00640 1 -- 172.21.15.6:0/2258076797 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74bc009690 con 0x7f74c41370b0 2024-03-04T09:31:25.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.390+0000 7f74c9600640 1 -- 172.21.15.6:0/2258076797 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74c4154cd0 con 0x7f74c41370b0 2024-03-04T09:31:25.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.390+0000 7f74c9600640 1 -- 172.21.15.6:0/2258076797 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74c41561a0 con 0x7f74c41370b0 2024-03-04T09:31:25.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.391+0000 7f74c9600640 1 -- 172.21.15.6:0/2258076797 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74c404f330 con 0x7f74c41370b0 2024-03-04T09:31:25.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.392+0000 7f74bbe00640 1 -- 172.21.15.6:0/2258076797 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f74bc003850 con 0x7f74c41370b0 2024-03-04T09:31:25.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.393+0000 7f74bbe00640 1 --2- 172.21.15.6:0/2258076797 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f74a006b4d0 0x7f74a006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:25.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.393+0000 7f74bbe00640 1 -- 172.21.15.6:0/2258076797 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f74bc0ab1d0 con 0x7f74c41370b0 2024-03-04T09:31:25.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.394+0000 7f74c3e00640 1 --2- 172.21.15.6:0/2258076797 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f74a006b4d0 0x7f74a006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:25.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.395+0000 7f74c3e00640 1 --2- 172.21.15.6:0/2258076797 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f74a006b4d0 0x7f74a006d990 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f74b40013d0 tx=0x7f74b4009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:25.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.398+0000 7f74bbe00640 1 -- 172.21.15.6:0/2258076797 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f74bc04c020 con 0x7f74c41370b0 2024-03-04T09:31:25.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.603+0000 7f74c9600640 1 -- 172.21.15.6:0/2258076797 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f74c40722f0 con 0x7f74c41370b0 2024-03-04T09:31:25.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.604+0000 7f74bbe00640 1 -- 172.21.15.6:0/2258076797 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v109) v1 ==== 74+0+16579 (secure 0 0 0) 0x7f74c40b06f0 con 0x7f74c41370b0 2024-03-04T09:31:25.601 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:31:25.601 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":109,"fsid":"edce0578-5ae4-4402-b45a-552bdef84d7c","created":"2024-03-04T09:24:54.523780+0000","modified":"2024-03-04T09:31:22.629333+0000","last_up_change":"2024-03-04T09:25:07.089049+0000","last_in_change":"2024-03-04T09:24:57.982077+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T09:25:07.935198+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-04T09:31:19.018233+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-04T09:31:20.131544+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a96574f-81f1-4bc6-8463-757520b84a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6825","nonce":1364449401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6827","nonce":1364449401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6831","nonce":1364449401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6829","nonce":1364449401}]},"public_addr":"172.21.15.6:6825/1364449401","cluster_addr":"172.21.15.6:6827/1364449401","heartbeat_back_addr":"172.21.15.6:6831/1364449401","heartbeat_front_addr":"172.21.15.6:6829/1364449401","state":["exists","up"]},{"osd":1,"uuid":"7d79b9c1-6b87-476a-bc36-7db5ee64cd90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6809","nonce":2494103078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6811","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6813","nonce":2494103078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6821","nonce":2494103078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6817","nonce":2494103078}]},"public_addr":"172.21.15.6:6809/2494103078","cluster_addr":"172.21.15.6:6813/2494103078","heartbeat_back_addr":"172.21.15.6:6821/2494103078","heartbeat_front_addr":"172.21.15.6:6817/2494103078","state":["exists","up"]},{"osd":2,"uuid":"492331b9-4ee9-40b6-bacd-85d44cd6c45a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6812","nonce":1500066681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6816","nonce":1500066681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6824","nonce":1500066681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6820","nonce":1500066681}]},"public_addr":"172.21.15.6:6812/1500066681","cluster_addr":"172.21.15.6:6816/1500066681","heartbeat_back_addr":"172.21.15.6:6824/1500066681","heartbeat_front_addr":"172.21.15.6:6820/1500066681","state":["exists","up"]},{"osd":3,"uuid":"21bb2efe-9ba7-4bdc-bc31-180cd1102e21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6801","nonce":1558806123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6803","nonce":1558806123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6807","nonce":1558806123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6805","nonce":1558806123}]},"public_addr":"172.21.15.6:6801/1558806123","cluster_addr":"172.21.15.6:6803/1558806123","heartbeat_back_addr":"172.21.15.6:6807/1558806123","heartbeat_front_addr":"172.21.15.6:6805/1558806123","state":["exists","up"]},{"osd":4,"uuid":"6e2c4b71-19d9-4737-83d9-558705d8644a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6801","nonce":1526135780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6803","nonce":1526135780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6807","nonce":1526135780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6805","nonce":1526135780}]},"public_addr":"172.21.15.146:6801/1526135780","cluster_addr":"172.21.15.146:6803/1526135780","heartbeat_back_addr":"172.21.15.146:6807/1526135780","heartbeat_front_addr":"172.21.15.146:6805/1526135780","state":["exists","up"]},{"osd":5,"uuid":"fb69c525-f1be-4b7d-ab32-6c29b0d6fddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6809","nonce":460262507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6811","nonce":460262507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6815","nonce":460262507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6813","nonce":460262507}]},"public_addr":"172.21.15.146:6809/460262507","cluster_addr":"172.21.15.146:6811/460262507","heartbeat_back_addr":"172.21.15.146:6815/460262507","heartbeat_front_addr":"172.21.15.146:6813/460262507","state":["exists","up"]},{"osd":6,"uuid":"c32e7fed-370d-4401-8ce0-6a4335b0d168","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6823","nonce":1392897493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6827","nonce":1392897493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6831","nonce":1392897493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6829","nonce":1392897493}]},"public_addr":"172.21.15.146:6823/1392897493","cluster_addr":"172.21.15.146:6827/1392897493","heartbeat_back_addr":"172.21.15.146:6831/1392897493","heartbeat_front_addr":"172.21.15.146:6829/1392897493","state":["exists","up"]},{"osd":7,"uuid":"75a79b44-fa5f-426d-a7d3-71f21f78835c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6817","nonce":3268323156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6819","nonce":3268323156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6825","nonce":3268323156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6822","nonce":3268323156}]},"public_addr":"172.21.15.146:6817/3268323156","cluster_addr":"172.21.15.146:6819/3268323156","heartbeat_back_addr":"172.21.15.146:6825/3268323156","heartbeat_front_addr":"172.21.15.146:6822/3268323156","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.310437+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.295850+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.294674+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.526855+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.300023+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.595160+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.346046+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.558741+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T09:31:25.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.607+0000 7f74c9600640 1 -- 172.21.15.6:0/2258076797 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f74a006b4d0 msgr2=0x7f74a006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:25.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.607+0000 7f74c9600640 1 --2- 172.21.15.6:0/2258076797 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f74a006b4d0 0x7f74a006d990 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f74b40013d0 tx=0x7f74b4009040 comp rx=0 tx=0).stop 2024-03-04T09:31:25.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.607+0000 7f74c9600640 1 -- 172.21.15.6:0/2258076797 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f74c41370b0 msgr2=0x7f74c4137530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:25.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.607+0000 7f74c9600640 1 --2- 172.21.15.6:0/2258076797 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f74c41370b0 0x7f74c4137530 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7f74bc02c320 tx=0x7f74bc004730 comp rx=0 tx=0).stop 2024-03-04T09:31:25.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.607+0000 7f74c9600640 1 -- 172.21.15.6:0/2258076797 shutdown_connections 2024-03-04T09:31:25.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.607+0000 7f74c9600640 1 --2- 172.21.15.6:0/2258076797 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f74c41370b0 0x7f74c4137530 unknown :-1 s=CLOSED pgs=1250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:25.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.607+0000 7f74c9600640 1 --2- 172.21.15.6:0/2258076797 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f74a006b4d0 0x7f74a006d990 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:25.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.607+0000 7f74c9600640 1 --2- 172.21.15.6:0/2258076797 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f74c4154130 0x7f74c41545b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:25.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.607+0000 7f74c9600640 1 --2- 172.21.15.6:0/2258076797 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f74a8004d40 0x7f74c415b290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:25.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.607+0000 7f74c9600640 1 -- 172.21.15.6:0/2258076797 >> 172.21.15.6:0/2258076797 conn(0x7f74c406c0c0 msgr2=0x7f74c415e6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:25.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.607+0000 7f74c9600640 1 -- 172.21.15.6:0/2258076797 shutdown_connections 2024-03-04T09:31:25.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.608+0000 7f74c9600640 1 -- 172.21.15.6:0/2258076797 wait complete. 2024-03-04T09:31:25.623 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-04T09:31:25.623 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T09:31:25.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.836+0000 7fab6e000640 1 Processor -- start 2024-03-04T09:31:25.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.836+0000 7fab6e000640 1 -- start start 2024-03-04T09:31:25.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.838+0000 7fab6e000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fab68161b40 0x7fab680cfd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:25.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.838+0000 7fab6e000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fab680b5060 0x7fab680d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:25.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.838+0000 7fab6e000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fab680b54d0 0x7fab680d2760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:25.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.838+0000 7fab6e000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab680cf900 con 0x7fab680b5060 2024-03-04T09:31:25.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.838+0000 7fab6e000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab680cea50 con 0x7fab68161b40 2024-03-04T09:31:25.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.838+0000 7fab6e000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab680cecf0 con 0x7fab680b54d0 2024-03-04T09:31:25.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.838+0000 7fab5fe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fab680b5060 0x7fab680d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:25.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.838+0000 7fab5fe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fab680b5060 0x7fab680d2220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:32782/0 (socket says 172.21.15.6:32782) 2024-03-04T09:31:25.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.838+0000 7fab5fe00640 1 -- 172.21.15.6:0/3731159268 learned_addr learned my addr 172.21.15.6:0/3731159268 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:31:25.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.839+0000 7fab67400640 1 --2- 172.21.15.6:0/3731159268 >> v2:172.21.15.146:3300/0 conn(0x7fab68161b40 0x7fab680cfd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:25.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.839+0000 7fab67e00640 1 --2- 172.21.15.6:0/3731159268 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fab680b54d0 0x7fab680d2760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:25.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.839+0000 7fab5fe00640 1 -- 172.21.15.6:0/3731159268 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fab680b54d0 msgr2=0x7fab680d2760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:25.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.839+0000 7fab5fe00640 1 --2- 172.21.15.6:0/3731159268 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fab680b54d0 0x7fab680d2760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:25.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.839+0000 7fab5fe00640 1 -- 172.21.15.6:0/3731159268 >> v2:172.21.15.146:3300/0 conn(0x7fab68161b40 msgr2=0x7fab680cfd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:25.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.839+0000 7fab5fe00640 1 --2- 172.21.15.6:0/3731159268 >> v2:172.21.15.146:3300/0 conn(0x7fab68161b40 0x7fab680cfd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:25.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.839+0000 7fab5fe00640 1 -- 172.21.15.6:0/3731159268 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab680cefd0 con 0x7fab680b5060 2024-03-04T09:31:25.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.840+0000 7fab5fe00640 1 --2- 172.21.15.6:0/3731159268 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fab680b5060 0x7fab680d2220 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7fab6808db00 tx=0x7fab5002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8170fb7e3b86ccc server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:25.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.840+0000 7fab66a00640 1 -- 172.21.15.6:0/3731159268 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab50035020 con 0x7fab680b5060 2024-03-04T09:31:25.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.840+0000 7fab66a00640 1 -- 172.21.15.6:0/3731159268 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab50031870 con 0x7fab680b5060 2024-03-04T09:31:25.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.840+0000 7fab66a00640 1 -- 172.21.15.6:0/3731159268 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab50031c40 con 0x7fab680b5060 2024-03-04T09:31:25.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.841+0000 7fab6e000640 1 -- 172.21.15.6:0/3731159268 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fab680b5060 msgr2=0x7fab680d2220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:25.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.841+0000 7fab6e000640 1 --2- 172.21.15.6:0/3731159268 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fab680b5060 0x7fab680d2220 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7fab6808db00 tx=0x7fab5002b910 comp rx=0 tx=0).stop 2024-03-04T09:31:25.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.841+0000 7fab6e000640 1 -- 172.21.15.6:0/3731159268 shutdown_connections 2024-03-04T09:31:25.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.841+0000 7fab6e000640 1 --2- 172.21.15.6:0/3731159268 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fab680b54d0 0x7fab680d2760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:25.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.841+0000 7fab6e000640 1 --2- 172.21.15.6:0/3731159268 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fab680b5060 0x7fab680d2220 unknown :-1 s=CLOSED pgs=1251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:25.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.841+0000 7fab6e000640 1 --2- 172.21.15.6:0/3731159268 >> v2:172.21.15.146:3300/0 conn(0x7fab68161b40 0x7fab680cfd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:25.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.841+0000 7fab6e000640 1 -- 172.21.15.6:0/3731159268 >> 172.21.15.6:0/3731159268 conn(0x7fab6809d220 msgr2=0x7fab680d3ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:25.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.841+0000 7fab6e000640 1 -- 172.21.15.6:0/3731159268 shutdown_connections 2024-03-04T09:31:25.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.841+0000 7fab6e000640 1 -- 172.21.15.6:0/3731159268 wait complete. 2024-03-04T09:31:25.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.843+0000 7fab6e000640 1 Processor -- start 2024-03-04T09:31:25.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.843+0000 7fab6e000640 1 -- start start 2024-03-04T09:31:25.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.844+0000 7fab6e000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fab680b5060 0x7fab681585d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:25.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.844+0000 7fab6e000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fab680b54d0 0x7fab680b8970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:25.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.844+0000 7fab6e000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fab68161b40 0x7fab680b8eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:25.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.844+0000 7fab6e000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab680b2ff0 con 0x7fab680b5060 2024-03-04T09:31:25.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.844+0000 7fab6e000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab680cf590 con 0x7fab680b54d0 2024-03-04T09:31:25.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.844+0000 7fab6e000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab680b3170 con 0x7fab68161b40 2024-03-04T09:31:25.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.844+0000 7fab67400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fab680b5060 0x7fab681585d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:25.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.844+0000 7fab67400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fab680b5060 0x7fab681585d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:32790/0 (socket says 172.21.15.6:32790) 2024-03-04T09:31:25.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.844+0000 7fab67400640 1 -- 172.21.15.6:0/2831404791 learned_addr learned my addr 172.21.15.6:0/2831404791 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:31:25.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.845+0000 7fab67e00640 1 --2- 172.21.15.6:0/2831404791 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fab68161b40 0x7fab680b8eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:25.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.845+0000 7fab5fe00640 1 --2- 172.21.15.6:0/2831404791 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fab680b54d0 0x7fab680b8970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:25.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.845+0000 7fab67400640 1 -- 172.21.15.6:0/2831404791 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fab68161b40 msgr2=0x7fab680b8eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:25.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.845+0000 7fab67400640 1 --2- 172.21.15.6:0/2831404791 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fab68161b40 0x7fab680b8eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:25.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.845+0000 7fab67400640 1 -- 172.21.15.6:0/2831404791 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fab680b54d0 msgr2=0x7fab680b8970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:25.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.845+0000 7fab67400640 1 --2- 172.21.15.6:0/2831404791 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fab680b54d0 0x7fab680b8970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:25.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.845+0000 7fab67400640 1 -- 172.21.15.6:0/2831404791 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab680cafc0 con 0x7fab680b5060 2024-03-04T09:31:25.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.845+0000 7fab5fe00640 1 --2- 172.21.15.6:0/2831404791 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fab680b54d0 0x7fab680b8970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:31:25.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.845+0000 7fab67e00640 1 --2- 172.21.15.6:0/2831404791 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fab68161b40 0x7fab680b8eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:31:25.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.845+0000 7fab67400640 1 --2- 172.21.15.6:0/2831404791 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fab680b5060 0x7fab681585d0 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7fab5800d800 tx=0x7fab5800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:25.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.846+0000 7fab5f400640 1 -- 172.21.15.6:0/2831404791 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab5800b6c0 con 0x7fab680b5060 2024-03-04T09:31:25.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.846+0000 7fab5f400640 1 -- 172.21.15.6:0/2831404791 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab58019070 con 0x7fab680b5060 2024-03-04T09:31:25.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.846+0000 7fab5f400640 1 -- 172.21.15.6:0/2831404791 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab5800ba80 con 0x7fab680b5060 2024-03-04T09:31:25.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.846+0000 7fab6e000640 1 -- 172.21.15.6:0/2831404791 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab680cb2a0 con 0x7fab680b5060 2024-03-04T09:31:25.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.846+0000 7fab6e000640 1 -- 172.21.15.6:0/2831404791 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab680cb860 con 0x7fab680b5060 2024-03-04T09:31:25.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.847+0000 7fab6e000640 1 -- 172.21.15.6:0/2831404791 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab6804f3a0 con 0x7fab680b5060 2024-03-04T09:31:25.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.848+0000 7fab5f400640 1 -- 172.21.15.6:0/2831404791 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fab5800bc20 con 0x7fab680b5060 2024-03-04T09:31:25.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.850+0000 7fab5f400640 1 --2- 172.21.15.6:0/2831404791 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fab4406b4d0 0x7fab4406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:25.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.850+0000 7fab5f400640 1 -- 172.21.15.6:0/2831404791 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fab5808a6c0 con 0x7fab680b5060 2024-03-04T09:31:25.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.850+0000 7fab5fe00640 1 --2- 172.21.15.6:0/2831404791 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fab4406b4d0 0x7fab4406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:25.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.851+0000 7fab5fe00640 1 --2- 172.21.15.6:0/2831404791 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fab4406b4d0 0x7fab4406d990 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fab50036000 tx=0x7fab50008030 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:25.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:25.854+0000 7fab5f400640 1 -- 172.21.15.6:0/2831404791 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fab58059530 con 0x7fab680b5060 2024-03-04T09:31:26.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.049+0000 7fab6e000640 1 -- 172.21.15.6:0/2831404791 --> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fab680d1df0 con 0x7fab4406b4d0 2024-03-04T09:31:26.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.069+0000 7fab5f400640 1 -- 172.21.15.6:0/2831404791 <== mgr.4113 v2:172.21.15.146:6832/1788613334 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252440 (secure 0 0 0) 0x7fab680d1df0 con 0x7fab4406b4d0 2024-03-04T09:31:26.069 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:31:26.071 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-03-04T09:31:26.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.075+0000 7fab6e000640 1 -- 172.21.15.6:0/2831404791 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fab4406b4d0 msgr2=0x7fab4406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:26.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.075+0000 7fab6e000640 1 --2- 172.21.15.6:0/2831404791 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fab4406b4d0 0x7fab4406d990 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fab50036000 tx=0x7fab50008030 comp rx=0 tx=0).stop 2024-03-04T09:31:26.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.075+0000 7fab6e000640 1 -- 172.21.15.6:0/2831404791 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fab680b5060 msgr2=0x7fab681585d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:26.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.075+0000 7fab6e000640 1 --2- 172.21.15.6:0/2831404791 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fab680b5060 0x7fab681585d0 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7fab5800d800 tx=0x7fab5800dcc0 comp rx=0 tx=0).stop 2024-03-04T09:31:26.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.076+0000 7fab6e000640 1 -- 172.21.15.6:0/2831404791 shutdown_connections 2024-03-04T09:31:26.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.076+0000 7fab6e000640 1 --2- 172.21.15.6:0/2831404791 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fab4406b4d0 0x7fab4406d990 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:26.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.076+0000 7fab6e000640 1 --2- 172.21.15.6:0/2831404791 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fab68161b40 0x7fab680b8eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:26.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.076+0000 7fab6e000640 1 --2- 172.21.15.6:0/2831404791 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fab680b54d0 0x7fab680b8970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:26.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.076+0000 7fab6e000640 1 --2- 172.21.15.6:0/2831404791 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fab680b5060 0x7fab681585d0 unknown :-1 s=CLOSED pgs=1252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:26.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.076+0000 7fab6e000640 1 -- 172.21.15.6:0/2831404791 >> 172.21.15.6:0/2831404791 conn(0x7fab6809d220 msgr2=0x7fab680b1a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:26.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.076+0000 7fab6e000640 1 -- 172.21.15.6:0/2831404791 shutdown_connections 2024-03-04T09:31:26.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.076+0000 7fab6e000640 1 -- 172.21.15.6:0/2831404791 wait complete. 2024-03-04T09:31:26.101 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":291,"stamp":"2024-03-04T09:31:25.992024+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":98,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":294496,"kb_used_data":11744,"kb_used_omap":12,"kb_used_meta":282611,"kb_avail":754680224,"statfs":{"total":773094113280,"available":772792549376,"internally_reserved":0,"allocated":12025856,"data_stored":5836716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":289394224},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002543"},"pg_stats":[{"pgid":"17.3f","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-04T09:31:21.664184+0000","last_change":"2024-03-04T09:31:21.656440+0000","last_active":"2024-03-04T09:31:21.664184+0000","last_peered":"2024-03-04T09:31:21.664184+0000","last_clean":"2024-03-04T09:31:21.664184+0000","last_became_active":"2024-03-04T09:31:21.656051+0000","last_became_peered":"2024-03-04T09:31:21.656051+0000","last_unstale":"2024-03-04T09:31:21.664184+0000","last_undegraded":"2024-03-04T09:31:21.664184+0000","last_fullsized":"2024-03-04T09:31:21.664184+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:32:51.022938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-04T09:31:21.658551+0000","last_change":"2024-03-04T09:31:21.654498+0000","last_active":"2024-03-04T09:31:21.658551+0000","last_peered":"2024-03-04T09:31:21.658551+0000","last_clean":"2024-03-04T09:31:21.658551+0000","last_became_active":"2024-03-04T09:31:21.653792+0000","last_became_peered":"2024-03-04T09:31:21.653792+0000","last_unstale":"2024-03-04T09:31:21.658551+0000","last_undegraded":"2024-03-04T09:31:21.658551+0000","last_fullsized":"2024-03-04T09:31:21.658551+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:01:45.712689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-04T09:31:21.666849+0000","last_change":"2024-03-04T09:31:21.665285+0000","last_active":"2024-03-04T09:31:21.666849+0000","last_peered":"2024-03-04T09:31:21.666849+0000","last_clean":"2024-03-04T09:31:21.666849+0000","last_became_active":"2024-03-04T09:31:21.664700+0000","last_became_peered":"2024-03-04T09:31:21.664700+0000","last_unstale":"2024-03-04T09:31:21.666849+0000","last_undegraded":"2024-03-04T09:31:21.666849+0000","last_fullsized":"2024-03-04T09:31:21.666849+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:01:35.042046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-04T09:31:21.664775+0000","last_change":"2024-03-04T09:31:21.663374+0000","last_active":"2024-03-04T09:31:21.664775+0000","last_peered":"2024-03-04T09:31:21.664775+0000","last_clean":"2024-03-04T09:31:21.664775+0000","last_became_active":"2024-03-04T09:31:21.663043+0000","last_became_peered":"2024-03-04T09:31:21.663043+0000","last_unstale":"2024-03-04T09:31:21.664775+0000","last_undegraded":"2024-03-04T09:31:21.664775+0000","last_fullsized":"2024-03-04T09:31:21.664775+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:47:19.591453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-04T09:31:21.653242+0000","last_change":"2024-03-04T09:31:21.650272+0000","last_active":"2024-03-04T09:31:21.653242+0000","last_peered":"2024-03-04T09:31:21.653242+0000","last_clean":"2024-03-04T09:31:21.653242+0000","last_became_active":"2024-03-04T09:31:21.649896+0000","last_became_peered":"2024-03-04T09:31:21.649896+0000","last_unstale":"2024-03-04T09:31:21.653242+0000","last_undegraded":"2024-03-04T09:31:21.653242+0000","last_fullsized":"2024-03-04T09:31:21.653242+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:23:44.996522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-04T09:31:22.645879+0000","last_change":"2024-03-04T09:31:21.646820+0000","last_active":"2024-03-04T09:31:22.645879+0000","last_peered":"2024-03-04T09:31:22.645879+0000","last_clean":"2024-03-04T09:31:22.645879+0000","last_became_active":"2024-03-04T09:31:21.646519+0000","last_became_peered":"2024-03-04T09:31:21.646519+0000","last_unstale":"2024-03-04T09:31:22.645879+0000","last_undegraded":"2024-03-04T09:31:22.645879+0000","last_fullsized":"2024-03-04T09:31:22.645879+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:36:41.886936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-04T09:31:22.659368+0000","last_change":"2024-03-04T09:31:21.638021+0000","last_active":"2024-03-04T09:31:22.659368+0000","last_peered":"2024-03-04T09:31:22.659368+0000","last_clean":"2024-03-04T09:31:22.659368+0000","last_became_active":"2024-03-04T09:31:21.637131+0000","last_became_peered":"2024-03-04T09:31:21.637131+0000","last_unstale":"2024-03-04T09:31:22.659368+0000","last_undegraded":"2024-03-04T09:31:22.659368+0000","last_fullsized":"2024-03-04T09:31:22.659368+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:48:59.896210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-04T09:31:21.659506+0000","last_change":"2024-03-04T09:31:21.656178+0000","last_active":"2024-03-04T09:31:21.659506+0000","last_peered":"2024-03-04T09:31:21.659506+0000","last_clean":"2024-03-04T09:31:21.659506+0000","last_became_active":"2024-03-04T09:31:21.655792+0000","last_became_peered":"2024-03-04T09:31:21.655792+0000","last_unstale":"2024-03-04T09:31:21.659506+0000","last_undegraded":"2024-03-04T09:31:21.659506+0000","last_fullsized":"2024-03-04T09:31:21.659506+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:15:55.025031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-04T09:31:21.664258+0000","last_change":"2024-03-04T09:31:21.660721+0000","last_active":"2024-03-04T09:31:21.664258+0000","last_peered":"2024-03-04T09:31:21.664258+0000","last_clean":"2024-03-04T09:31:21.664258+0000","last_became_active":"2024-03-04T09:31:21.660440+0000","last_became_peered":"2024-03-04T09:31:21.660440+0000","last_unstale":"2024-03-04T09:31:21.664258+0000","last_undegraded":"2024-03-04T09:31:21.664258+0000","last_fullsized":"2024-03-04T09:31:21.664258+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:11:02.962349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-04T09:31:21.669016+0000","last_change":"2024-03-04T09:31:21.660149+0000","last_active":"2024-03-04T09:31:21.669016+0000","last_peered":"2024-03-04T09:31:21.669016+0000","last_clean":"2024-03-04T09:31:21.669016+0000","last_became_active":"2024-03-04T09:31:21.659870+0000","last_became_peered":"2024-03-04T09:31:21.659870+0000","last_unstale":"2024-03-04T09:31:21.669016+0000","last_undegraded":"2024-03-04T09:31:21.669016+0000","last_fullsized":"2024-03-04T09:31:21.669016+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:33:22.518385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-04T09:31:21.657220+0000","last_change":"2024-03-04T09:31:21.650182+0000","last_active":"2024-03-04T09:31:21.657220+0000","last_peered":"2024-03-04T09:31:21.657220+0000","last_clean":"2024-03-04T09:31:21.657220+0000","last_became_active":"2024-03-04T09:31:21.649920+0000","last_became_peered":"2024-03-04T09:31:21.649920+0000","last_unstale":"2024-03-04T09:31:21.657220+0000","last_undegraded":"2024-03-04T09:31:21.657220+0000","last_fullsized":"2024-03-04T09:31:21.657220+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:15:55.025031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-04T09:31:21.664747+0000","last_change":"2024-03-04T09:31:21.662634+0000","last_active":"2024-03-04T09:31:21.664747+0000","last_peered":"2024-03-04T09:31:21.664747+0000","last_clean":"2024-03-04T09:31:21.664747+0000","last_became_active":"2024-03-04T09:31:21.648929+0000","last_became_peered":"2024-03-04T09:31:21.648929+0000","last_unstale":"2024-03-04T09:31:21.664747+0000","last_undegraded":"2024-03-04T09:31:21.664747+0000","last_fullsized":"2024-03-04T09:31:21.664747+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:35:45.458664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-04T09:31:21.659158+0000","last_change":"2024-03-04T09:31:21.655625+0000","last_active":"2024-03-04T09:31:21.659158+0000","last_peered":"2024-03-04T09:31:21.659158+0000","last_clean":"2024-03-04T09:31:21.659158+0000","last_became_active":"2024-03-04T09:31:21.655308+0000","last_became_peered":"2024-03-04T09:31:21.655308+0000","last_unstale":"2024-03-04T09:31:21.659158+0000","last_undegraded":"2024-03-04T09:31:21.659158+0000","last_fullsized":"2024-03-04T09:31:21.659158+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:21:39.986937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-04T09:31:21.663771+0000","last_change":"2024-03-04T09:31:21.661136+0000","last_active":"2024-03-04T09:31:21.663771+0000","last_peered":"2024-03-04T09:31:21.663771+0000","last_clean":"2024-03-04T09:31:21.663771+0000","last_became_active":"2024-03-04T09:31:21.660870+0000","last_became_peered":"2024-03-04T09:31:21.660870+0000","last_unstale":"2024-03-04T09:31:21.663771+0000","last_undegraded":"2024-03-04T09:31:21.663771+0000","last_fullsized":"2024-03-04T09:31:21.663771+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:31:52.099825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-04T09:31:22.645575+0000","last_change":"2024-03-04T09:31:21.644245+0000","last_active":"2024-03-04T09:31:22.645575+0000","last_peered":"2024-03-04T09:31:22.645575+0000","last_clean":"2024-03-04T09:31:22.645575+0000","last_became_active":"2024-03-04T09:31:21.643917+0000","last_became_peered":"2024-03-04T09:31:21.643917+0000","last_unstale":"2024-03-04T09:31:22.645575+0000","last_undegraded":"2024-03-04T09:31:22.645575+0000","last_fullsized":"2024-03-04T09:31:22.645575+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:18:11.958968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-04T09:31:21.663692+0000","last_change":"2024-03-04T09:31:21.651689+0000","last_active":"2024-03-04T09:31:21.663692+0000","last_peered":"2024-03-04T09:31:21.663692+0000","last_clean":"2024-03-04T09:31:21.663692+0000","last_became_active":"2024-03-04T09:31:21.648891+0000","last_became_peered":"2024-03-04T09:31:21.648891+0000","last_unstale":"2024-03-04T09:31:21.663692+0000","last_undegraded":"2024-03-04T09:31:21.663692+0000","last_fullsized":"2024-03-04T09:31:21.663692+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:02:39.356053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-04T09:31:21.657226+0000","last_change":"2024-03-04T09:31:21.650711+0000","last_active":"2024-03-04T09:31:21.657226+0000","last_peered":"2024-03-04T09:31:21.657226+0000","last_clean":"2024-03-04T09:31:21.657226+0000","last_became_active":"2024-03-04T09:31:21.650103+0000","last_became_peered":"2024-03-04T09:31:21.650103+0000","last_unstale":"2024-03-04T09:31:21.657226+0000","last_undegraded":"2024-03-04T09:31:21.657226+0000","last_fullsized":"2024-03-04T09:31:21.657226+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:13:52.946588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-04T09:31:22.647679+0000","last_change":"2024-03-04T09:31:21.647968+0000","last_active":"2024-03-04T09:31:22.647679+0000","last_peered":"2024-03-04T09:31:22.647679+0000","last_clean":"2024-03-04T09:31:22.647679+0000","last_became_active":"2024-03-04T09:31:21.647092+0000","last_became_peered":"2024-03-04T09:31:21.647092+0000","last_unstale":"2024-03-04T09:31:22.647679+0000","last_undegraded":"2024-03-04T09:31:22.647679+0000","last_fullsized":"2024-03-04T09:31:22.647679+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:27:14.606466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-04T09:31:21.661768+0000","last_change":"2024-03-04T09:31:21.658975+0000","last_active":"2024-03-04T09:31:21.661768+0000","last_peered":"2024-03-04T09:31:21.661768+0000","last_clean":"2024-03-04T09:31:21.661768+0000","last_became_active":"2024-03-04T09:31:21.658140+0000","last_became_peered":"2024-03-04T09:31:21.658140+0000","last_unstale":"2024-03-04T09:31:21.661768+0000","last_undegraded":"2024-03-04T09:31:21.661768+0000","last_fullsized":"2024-03-04T09:31:21.661768+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:31:15.039332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-04T09:31:21.657621+0000","last_change":"2024-03-04T09:31:21.653591+0000","last_active":"2024-03-04T09:31:21.657621+0000","last_peered":"2024-03-04T09:31:21.657621+0000","last_clean":"2024-03-04T09:31:21.657621+0000","last_became_active":"2024-03-04T09:31:21.652739+0000","last_became_peered":"2024-03-04T09:31:21.652739+0000","last_unstale":"2024-03-04T09:31:21.657621+0000","last_undegraded":"2024-03-04T09:31:21.657621+0000","last_fullsized":"2024-03-04T09:31:21.657621+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:08:19.417156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-04T09:31:21.645860+0000","last_change":"2024-03-04T09:31:21.642542+0000","last_active":"2024-03-04T09:31:21.645860+0000","last_peered":"2024-03-04T09:31:21.645860+0000","last_clean":"2024-03-04T09:31:21.645860+0000","last_became_active":"2024-03-04T09:31:21.641810+0000","last_became_peered":"2024-03-04T09:31:21.641810+0000","last_unstale":"2024-03-04T09:31:21.645860+0000","last_undegraded":"2024-03-04T09:31:21.645860+0000","last_fullsized":"2024-03-04T09:31:21.645860+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:20:05.757250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-04T09:31:22.647080+0000","last_change":"2024-03-04T09:31:21.646242+0000","last_active":"2024-03-04T09:31:22.647080+0000","last_peered":"2024-03-04T09:31:22.647080+0000","last_clean":"2024-03-04T09:31:22.647080+0000","last_became_active":"2024-03-04T09:31:21.645931+0000","last_became_peered":"2024-03-04T09:31:21.645931+0000","last_unstale":"2024-03-04T09:31:22.647080+0000","last_undegraded":"2024-03-04T09:31:22.647080+0000","last_fullsized":"2024-03-04T09:31:22.647080+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:01:22.488099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-04T09:31:21.646201+0000","last_change":"2024-03-04T09:31:21.643743+0000","last_active":"2024-03-04T09:31:21.646201+0000","last_peered":"2024-03-04T09:31:21.646201+0000","last_clean":"2024-03-04T09:31:21.646201+0000","last_became_active":"2024-03-04T09:31:21.643434+0000","last_became_peered":"2024-03-04T09:31:21.643434+0000","last_unstale":"2024-03-04T09:31:21.646201+0000","last_undegraded":"2024-03-04T09:31:21.646201+0000","last_fullsized":"2024-03-04T09:31:21.646201+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:00:42.083728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-04T09:31:22.646321+0000","last_change":"2024-03-04T09:31:21.657735+0000","last_active":"2024-03-04T09:31:22.646321+0000","last_peered":"2024-03-04T09:31:22.646321+0000","last_clean":"2024-03-04T09:31:22.646321+0000","last_became_active":"2024-03-04T09:31:21.657393+0000","last_became_peered":"2024-03-04T09:31:21.657393+0000","last_unstale":"2024-03-04T09:31:22.646321+0000","last_undegraded":"2024-03-04T09:31:22.646321+0000","last_fullsized":"2024-03-04T09:31:22.646321+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:31:06.794122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-04T09:31:21.658137+0000","last_change":"2024-03-04T09:31:21.653619+0000","last_active":"2024-03-04T09:31:21.658137+0000","last_peered":"2024-03-04T09:31:21.658137+0000","last_clean":"2024-03-04T09:31:21.658137+0000","last_became_active":"2024-03-04T09:31:21.649674+0000","last_became_peered":"2024-03-04T09:31:21.649674+0000","last_unstale":"2024-03-04T09:31:21.658137+0000","last_undegraded":"2024-03-04T09:31:21.658137+0000","last_fullsized":"2024-03-04T09:31:21.658137+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:31:15.039332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-04T09:31:21.668995+0000","last_change":"2024-03-04T09:31:21.659544+0000","last_active":"2024-03-04T09:31:21.668995+0000","last_peered":"2024-03-04T09:31:21.668995+0000","last_clean":"2024-03-04T09:31:21.668995+0000","last_became_active":"2024-03-04T09:31:21.659232+0000","last_became_peered":"2024-03-04T09:31:21.659232+0000","last_unstale":"2024-03-04T09:31:21.668995+0000","last_undegraded":"2024-03-04T09:31:21.668995+0000","last_fullsized":"2024-03-04T09:31:21.668995+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:02:33.790988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-04T09:31:22.646403+0000","last_change":"2024-03-04T09:31:21.669416+0000","last_active":"2024-03-04T09:31:22.646403+0000","last_peered":"2024-03-04T09:31:22.646403+0000","last_clean":"2024-03-04T09:31:22.646403+0000","last_became_active":"2024-03-04T09:31:21.669119+0000","last_became_peered":"2024-03-04T09:31:21.669119+0000","last_unstale":"2024-03-04T09:31:22.646403+0000","last_undegraded":"2024-03-04T09:31:22.646403+0000","last_fullsized":"2024-03-04T09:31:22.646403+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:49:19.511478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-04T09:31:22.647307+0000","last_change":"2024-03-04T09:31:21.644853+0000","last_active":"2024-03-04T09:31:22.647307+0000","last_peered":"2024-03-04T09:31:22.647307+0000","last_clean":"2024-03-04T09:31:22.647307+0000","last_became_active":"2024-03-04T09:31:21.644525+0000","last_became_peered":"2024-03-04T09:31:21.644525+0000","last_unstale":"2024-03-04T09:31:22.647307+0000","last_undegraded":"2024-03-04T09:31:22.647307+0000","last_fullsized":"2024-03-04T09:31:22.647307+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:53:21.199185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-04T09:31:21.655206+0000","last_change":"2024-03-04T09:31:21.648245+0000","last_active":"2024-03-04T09:31:21.655206+0000","last_peered":"2024-03-04T09:31:21.655206+0000","last_clean":"2024-03-04T09:31:21.655206+0000","last_became_active":"2024-03-04T09:31:21.647640+0000","last_became_peered":"2024-03-04T09:31:21.647640+0000","last_unstale":"2024-03-04T09:31:21.655206+0000","last_undegraded":"2024-03-04T09:31:21.655206+0000","last_fullsized":"2024-03-04T09:31:21.655206+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:38:38.988513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-04T09:31:22.659730+0000","last_change":"2024-03-04T09:31:21.641573+0000","last_active":"2024-03-04T09:31:22.659730+0000","last_peered":"2024-03-04T09:31:22.659730+0000","last_clean":"2024-03-04T09:31:22.659730+0000","last_became_active":"2024-03-04T09:31:21.641282+0000","last_became_peered":"2024-03-04T09:31:21.641282+0000","last_unstale":"2024-03-04T09:31:22.659730+0000","last_undegraded":"2024-03-04T09:31:22.659730+0000","last_fullsized":"2024-03-04T09:31:22.659730+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:04:49.856588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-04T09:31:21.668523+0000","last_change":"2024-03-04T09:31:21.663250+0000","last_active":"2024-03-04T09:31:21.668523+0000","last_peered":"2024-03-04T09:31:21.668523+0000","last_clean":"2024-03-04T09:31:21.668523+0000","last_became_active":"2024-03-04T09:31:21.662902+0000","last_became_peered":"2024-03-04T09:31:21.662902+0000","last_unstale":"2024-03-04T09:31:21.668523+0000","last_undegraded":"2024-03-04T09:31:21.668523+0000","last_fullsized":"2024-03-04T09:31:21.668523+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:28:38.909935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-04T09:31:22.644863+0000","last_change":"2024-03-04T09:31:21.640266+0000","last_active":"2024-03-04T09:31:22.644863+0000","last_peered":"2024-03-04T09:31:22.644863+0000","last_clean":"2024-03-04T09:31:22.644863+0000","last_became_active":"2024-03-04T09:31:21.639950+0000","last_became_peered":"2024-03-04T09:31:21.639950+0000","last_unstale":"2024-03-04T09:31:22.644863+0000","last_undegraded":"2024-03-04T09:31:22.644863+0000","last_fullsized":"2024-03-04T09:31:22.644863+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:01:50.307270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-04T09:31:22.646513+0000","last_change":"2024-03-04T09:31:21.641691+0000","last_active":"2024-03-04T09:31:22.646513+0000","last_peered":"2024-03-04T09:31:22.646513+0000","last_clean":"2024-03-04T09:31:22.646513+0000","last_became_active":"2024-03-04T09:31:21.640669+0000","last_became_peered":"2024-03-04T09:31:21.640669+0000","last_unstale":"2024-03-04T09:31:22.646513+0000","last_undegraded":"2024-03-04T09:31:22.646513+0000","last_fullsized":"2024-03-04T09:31:22.646513+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:48:59.896210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"108'4","reported_seq":18,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-04T09:31:22.659630+0000","last_change":"2024-03-04T09:31:20.653746+0000","last_active":"2024-03-04T09:31:22.659630+0000","last_peered":"2024-03-04T09:31:22.659630+0000","last_clean":"2024-03-04T09:31:22.659630+0000","last_became_active":"2024-03-04T09:31:20.653073+0000","last_became_peered":"2024-03-04T09:31:20.653073+0000","last_unstale":"2024-03-04T09:31:22.659630+0000","last_undegraded":"2024-03-04T09:31:22.659630+0000","last_fullsized":"2024-03-04T09:31:22.659630+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_clean_scrub_stamp":"2024-03-04T09:31:19.587017+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:53:20.852514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-04T09:31:22.648039+0000","last_change":"2024-03-04T09:31:21.651677+0000","last_active":"2024-03-04T09:31:22.648039+0000","last_peered":"2024-03-04T09:31:22.648039+0000","last_clean":"2024-03-04T09:31:22.648039+0000","last_became_active":"2024-03-04T09:31:21.651318+0000","last_became_peered":"2024-03-04T09:31:21.651318+0000","last_unstale":"2024-03-04T09:31:22.648039+0000","last_undegraded":"2024-03-04T09:31:22.648039+0000","last_fullsized":"2024-03-04T09:31:22.648039+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:01:45.712689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"108'1","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-04T09:31:21.666790+0000","last_change":"2024-03-04T09:31:20.654866+0000","last_active":"2024-03-04T09:31:21.666790+0000","last_peered":"2024-03-04T09:31:21.666790+0000","last_clean":"2024-03-04T09:31:21.666790+0000","last_became_active":"2024-03-04T09:31:20.654473+0000","last_became_peered":"2024-03-04T09:31:20.654473+0000","last_unstale":"2024-03-04T09:31:21.666790+0000","last_undegraded":"2024-03-04T09:31:21.666790+0000","last_fullsized":"2024-03-04T09:31:21.666790+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_clean_scrub_stamp":"2024-03-04T09:31:19.587017+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:38:37.971153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-04T09:31:22.646299+0000","last_change":"2024-03-04T09:31:21.641599+0000","last_active":"2024-03-04T09:31:22.646299+0000","last_peered":"2024-03-04T09:31:22.646299+0000","last_clean":"2024-03-04T09:31:22.646299+0000","last_became_active":"2024-03-04T09:31:21.641317+0000","last_became_peered":"2024-03-04T09:31:21.641317+0000","last_unstale":"2024-03-04T09:31:22.646299+0000","last_undegraded":"2024-03-04T09:31:22.646299+0000","last_fullsized":"2024-03-04T09:31:22.646299+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:03:27.258784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"108'2","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-04T09:31:21.669016+0000","last_change":"2024-03-04T09:31:20.640468+0000","last_active":"2024-03-04T09:31:21.669016+0000","last_peered":"2024-03-04T09:31:21.669016+0000","last_clean":"2024-03-04T09:31:21.669016+0000","last_became_active":"2024-03-04T09:31:20.640073+0000","last_became_peered":"2024-03-04T09:31:20.640073+0000","last_unstale":"2024-03-04T09:31:21.669016+0000","last_undegraded":"2024-03-04T09:31:21.669016+0000","last_fullsized":"2024-03-04T09:31:21.669016+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_clean_scrub_stamp":"2024-03-04T09:31:19.587017+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:57:57.417351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-04T09:31:21.661791+0000","last_change":"2024-03-04T09:31:21.659369+0000","last_active":"2024-03-04T09:31:21.661791+0000","last_peered":"2024-03-04T09:31:21.661791+0000","last_clean":"2024-03-04T09:31:21.661791+0000","last_became_active":"2024-03-04T09:31:21.659082+0000","last_became_peered":"2024-03-04T09:31:21.659082+0000","last_unstale":"2024-03-04T09:31:21.661791+0000","last_undegraded":"2024-03-04T09:31:21.661791+0000","last_fullsized":"2024-03-04T09:31:21.661791+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:01:45.712689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"108'2","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-04T09:31:21.669430+0000","last_change":"2024-03-04T09:31:20.641839+0000","last_active":"2024-03-04T09:31:21.669430+0000","last_peered":"2024-03-04T09:31:21.669430+0000","last_clean":"2024-03-04T09:31:21.669430+0000","last_became_active":"2024-03-04T09:31:20.640151+0000","last_became_peered":"2024-03-04T09:31:20.640151+0000","last_unstale":"2024-03-04T09:31:21.669430+0000","last_undegraded":"2024-03-04T09:31:21.669430+0000","last_fullsized":"2024-03-04T09:31:21.669430+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_clean_scrub_stamp":"2024-03-04T09:31:19.587017+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:27:11.829505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-04T09:31:21.661848+0000","last_change":"2024-03-04T09:31:21.650420+0000","last_active":"2024-03-04T09:31:21.661848+0000","last_peered":"2024-03-04T09:31:21.661848+0000","last_clean":"2024-03-04T09:31:21.661848+0000","last_became_active":"2024-03-04T09:31:21.648862+0000","last_became_peered":"2024-03-04T09:31:21.648862+0000","last_unstale":"2024-03-04T09:31:21.661848+0000","last_undegraded":"2024-03-04T09:31:21.661848+0000","last_fullsized":"2024-03-04T09:31:21.661848+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:55:52.210159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"108'1","reported_seq":15,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-04T09:31:22.659173+0000","last_change":"2024-03-04T09:31:20.655416+0000","last_active":"2024-03-04T09:31:22.659173+0000","last_peered":"2024-03-04T09:31:22.659173+0000","last_clean":"2024-03-04T09:31:22.659173+0000","last_became_active":"2024-03-04T09:31:20.655071+0000","last_became_peered":"2024-03-04T09:31:20.655071+0000","last_unstale":"2024-03-04T09:31:22.659173+0000","last_undegraded":"2024-03-04T09:31:22.659173+0000","last_fullsized":"2024-03-04T09:31:22.659173+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_clean_scrub_stamp":"2024-03-04T09:31:19.587017+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:26:04.483161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-04T09:31:21.655229+0000","last_change":"2024-03-04T09:31:21.649417+0000","last_active":"2024-03-04T09:31:21.655229+0000","last_peered":"2024-03-04T09:31:21.655229+0000","last_clean":"2024-03-04T09:31:21.655229+0000","last_became_active":"2024-03-04T09:31:21.649099+0000","last_became_peered":"2024-03-04T09:31:21.649099+0000","last_unstale":"2024-03-04T09:31:21.655229+0000","last_undegraded":"2024-03-04T09:31:21.655229+0000","last_fullsized":"2024-03-04T09:31:21.655229+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:21:39.986937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"108'1","reported_seq":15,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-04T09:31:22.647350+0000","last_change":"2024-03-04T09:31:20.652889+0000","last_active":"2024-03-04T09:31:22.647350+0000","last_peered":"2024-03-04T09:31:22.647350+0000","last_clean":"2024-03-04T09:31:22.647350+0000","last_became_active":"2024-03-04T09:31:20.652498+0000","last_became_peered":"2024-03-04T09:31:20.652498+0000","last_unstale":"2024-03-04T09:31:22.647350+0000","last_undegraded":"2024-03-04T09:31:22.647350+0000","last_fullsized":"2024-03-04T09:31:22.647350+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_clean_scrub_stamp":"2024-03-04T09:31:19.587017+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:26:35.978588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-04T09:31:21.663666+0000","last_change":"2024-03-04T09:31:21.645102+0000","last_active":"2024-03-04T09:31:21.663666+0000","last_peered":"2024-03-04T09:31:21.663666+0000","last_clean":"2024-03-04T09:31:21.663666+0000","last_became_active":"2024-03-04T09:31:21.644748+0000","last_became_peered":"2024-03-04T09:31:21.644748+0000","last_unstale":"2024-03-04T09:31:21.663666+0000","last_undegraded":"2024-03-04T09:31:21.663666+0000","last_fullsized":"2024-03-04T09:31:21.663666+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:11:17.789505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-04T09:31:21.643081+0000","last_change":"2024-03-04T09:31:20.651452+0000","last_active":"2024-03-04T09:31:21.643081+0000","last_peered":"2024-03-04T09:31:21.643081+0000","last_clean":"2024-03-04T09:31:21.643081+0000","last_became_active":"2024-03-04T09:31:20.651104+0000","last_became_peered":"2024-03-04T09:31:20.651104+0000","last_unstale":"2024-03-04T09:31:21.643081+0000","last_undegraded":"2024-03-04T09:31:21.643081+0000","last_fullsized":"2024-03-04T09:31:21.643081+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_clean_scrub_stamp":"2024-03-04T09:31:19.587017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:42:45.728646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-04T09:31:22.659224+0000","last_change":"2024-03-04T09:31:21.645173+0000","last_active":"2024-03-04T09:31:22.659224+0000","last_peered":"2024-03-04T09:31:22.659224+0000","last_clean":"2024-03-04T09:31:22.659224+0000","last_became_active":"2024-03-04T09:31:21.644788+0000","last_became_peered":"2024-03-04T09:31:21.644788+0000","last_unstale":"2024-03-04T09:31:22.659224+0000","last_undegraded":"2024-03-04T09:31:22.659224+0000","last_fullsized":"2024-03-04T09:31:22.659224+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:49:19.511478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-04T09:31:21.639439+0000","last_change":"2024-03-04T09:31:20.652811+0000","last_active":"2024-03-04T09:31:21.639439+0000","last_peered":"2024-03-04T09:31:21.639439+0000","last_clean":"2024-03-04T09:31:21.639439+0000","last_became_active":"2024-03-04T09:31:20.652516+0000","last_became_peered":"2024-03-04T09:31:20.652516+0000","last_unstale":"2024-03-04T09:31:21.639439+0000","last_undegraded":"2024-03-04T09:31:21.639439+0000","last_fullsized":"2024-03-04T09:31:21.639439+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_clean_scrub_stamp":"2024-03-04T09:31:19.587017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:12:47.612367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-04T09:31:22.659803+0000","last_change":"2024-03-04T09:31:21.648619+0000","last_active":"2024-03-04T09:31:22.659803+0000","last_peered":"2024-03-04T09:31:22.659803+0000","last_clean":"2024-03-04T09:31:22.659803+0000","last_became_active":"2024-03-04T09:31:21.648198+0000","last_became_peered":"2024-03-04T09:31:21.648198+0000","last_unstale":"2024-03-04T09:31:22.659803+0000","last_undegraded":"2024-03-04T09:31:22.659803+0000","last_fullsized":"2024-03-04T09:31:22.659803+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:54:05.625143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-04T09:31:21.641806+0000","last_change":"2024-03-04T09:31:20.646772+0000","last_active":"2024-03-04T09:31:21.641806+0000","last_peered":"2024-03-04T09:31:21.641806+0000","last_clean":"2024-03-04T09:31:21.641806+0000","last_became_active":"2024-03-04T09:31:20.646459+0000","last_became_peered":"2024-03-04T09:31:20.646459+0000","last_unstale":"2024-03-04T09:31:21.641806+0000","last_undegraded":"2024-03-04T09:31:21.641806+0000","last_fullsized":"2024-03-04T09:31:21.641806+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_clean_scrub_stamp":"2024-03-04T09:31:19.587017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:36:54.206546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-04T09:31:22.648070+0000","last_change":"2024-03-04T09:31:21.665335+0000","last_active":"2024-03-04T09:31:22.648070+0000","last_peered":"2024-03-04T09:31:22.648070+0000","last_clean":"2024-03-04T09:31:22.648070+0000","last_became_active":"2024-03-04T09:31:21.664400+0000","last_became_peered":"2024-03-04T09:31:21.664400+0000","last_unstale":"2024-03-04T09:31:22.648070+0000","last_undegraded":"2024-03-04T09:31:22.648070+0000","last_fullsized":"2024-03-04T09:31:22.648070+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:31:52.099825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-04T09:31:22.647325+0000","last_change":"2024-03-04T09:31:20.653051+0000","last_active":"2024-03-04T09:31:22.647325+0000","last_peered":"2024-03-04T09:31:22.647325+0000","last_clean":"2024-03-04T09:31:22.647325+0000","last_became_active":"2024-03-04T09:31:20.652751+0000","last_became_peered":"2024-03-04T09:31:20.652751+0000","last_unstale":"2024-03-04T09:31:22.647325+0000","last_undegraded":"2024-03-04T09:31:22.647325+0000","last_fullsized":"2024-03-04T09:31:22.647325+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_clean_scrub_stamp":"2024-03-04T09:31:19.587017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:39:48.758359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-04T09:31:22.647519+0000","last_change":"2024-03-04T09:31:21.649988+0000","last_active":"2024-03-04T09:31:22.647519+0000","last_peered":"2024-03-04T09:31:22.647519+0000","last_clean":"2024-03-04T09:31:22.647519+0000","last_became_active":"2024-03-04T09:31:21.649490+0000","last_became_peered":"2024-03-04T09:31:21.649490+0000","last_unstale":"2024-03-04T09:31:22.647519+0000","last_undegraded":"2024-03-04T09:31:22.647519+0000","last_fullsized":"2024-03-04T09:31:22.647519+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:48:52.528076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"108'3","reported_seq":15,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-04T09:31:21.677746+0000","last_change":"2024-03-04T09:31:20.650114+0000","last_active":"2024-03-04T09:31:21.677746+0000","last_peered":"2024-03-04T09:31:21.677746+0000","last_clean":"2024-03-04T09:31:21.677746+0000","last_became_active":"2024-03-04T09:31:20.646728+0000","last_became_peered":"2024-03-04T09:31:20.646728+0000","last_unstale":"2024-03-04T09:31:21.677746+0000","last_undegraded":"2024-03-04T09:31:21.677746+0000","last_fullsized":"2024-03-04T09:31:21.677746+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_clean_scrub_stamp":"2024-03-04T09:31:19.587017+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:56:45.010621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-04T09:31:22.659769+0000","last_change":"2024-03-04T09:31:21.648739+0000","last_active":"2024-03-04T09:31:22.659769+0000","last_peered":"2024-03-04T09:31:22.659769+0000","last_clean":"2024-03-04T09:31:22.659769+0000","last_became_active":"2024-03-04T09:31:21.647112+0000","last_became_peered":"2024-03-04T09:31:21.647112+0000","last_unstale":"2024-03-04T09:31:22.659769+0000","last_undegraded":"2024-03-04T09:31:22.659769+0000","last_fullsized":"2024-03-04T09:31:22.659769+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:50:17.143286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"108'6","reported_seq":18,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-04T09:31:21.680332+0000","last_change":"2024-03-04T09:31:20.652880+0000","last_active":"2024-03-04T09:31:21.680332+0000","last_peered":"2024-03-04T09:31:21.680332+0000","last_clean":"2024-03-04T09:31:21.680332+0000","last_became_active":"2024-03-04T09:31:20.652623+0000","last_became_peered":"2024-03-04T09:31:20.652623+0000","last_unstale":"2024-03-04T09:31:21.680332+0000","last_undegraded":"2024-03-04T09:31:21.680332+0000","last_fullsized":"2024-03-04T09:31:21.680332+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_clean_scrub_stamp":"2024-03-04T09:31:19.587017+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:04:35.028973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-04T09:31:22.647239+0000","last_change":"2024-03-04T09:31:21.653150+0000","last_active":"2024-03-04T09:31:22.647239+0000","last_peered":"2024-03-04T09:31:22.647239+0000","last_clean":"2024-03-04T09:31:22.647239+0000","last_became_active":"2024-03-04T09:31:21.652836+0000","last_became_peered":"2024-03-04T09:31:21.652836+0000","last_unstale":"2024-03-04T09:31:22.647239+0000","last_undegraded":"2024-03-04T09:31:22.647239+0000","last_fullsized":"2024-03-04T09:31:22.647239+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:19:31.048726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"108'2","reported_seq":16,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-04T09:31:22.646070+0000","last_change":"2024-03-04T09:31:20.649845+0000","last_active":"2024-03-04T09:31:22.646070+0000","last_peered":"2024-03-04T09:31:22.646070+0000","last_clean":"2024-03-04T09:31:22.646070+0000","last_became_active":"2024-03-04T09:31:20.649546+0000","last_became_peered":"2024-03-04T09:31:20.649546+0000","last_unstale":"2024-03-04T09:31:22.646070+0000","last_undegraded":"2024-03-04T09:31:22.646070+0000","last_fullsized":"2024-03-04T09:31:22.646070+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_clean_scrub_stamp":"2024-03-04T09:31:19.587017+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:39:54.323425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-04T09:31:22.647476+0000","last_change":"2024-03-04T09:31:21.648524+0000","last_active":"2024-03-04T09:31:22.647476+0000","last_peered":"2024-03-04T09:31:22.647476+0000","last_clean":"2024-03-04T09:31:22.647476+0000","last_became_active":"2024-03-04T09:31:21.648009+0000","last_became_peered":"2024-03-04T09:31:21.648009+0000","last_unstale":"2024-03-04T09:31:22.647476+0000","last_undegraded":"2024-03-04T09:31:22.647476+0000","last_fullsized":"2024-03-04T09:31:22.647476+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:12:48.629727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-04T09:31:22.647416+0000","last_change":"2024-03-04T09:31:20.658305+0000","last_active":"2024-03-04T09:31:22.647416+0000","last_peered":"2024-03-04T09:31:22.647416+0000","last_clean":"2024-03-04T09:31:22.647416+0000","last_became_active":"2024-03-04T09:31:20.657345+0000","last_became_peered":"2024-03-04T09:31:20.657345+0000","last_unstale":"2024-03-04T09:31:22.647416+0000","last_undegraded":"2024-03-04T09:31:22.647416+0000","last_fullsized":"2024-03-04T09:31:22.647416+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_clean_scrub_stamp":"2024-03-04T09:31:19.587017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:20:04.739890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-04T09:31:22.646864+0000","last_change":"2024-03-04T09:31:21.645648+0000","last_active":"2024-03-04T09:31:22.646864+0000","last_peered":"2024-03-04T09:31:22.646864+0000","last_clean":"2024-03-04T09:31:22.646864+0000","last_became_active":"2024-03-04T09:31:21.645354+0000","last_became_peered":"2024-03-04T09:31:21.645354+0000","last_unstale":"2024-03-04T09:31:22.646864+0000","last_undegraded":"2024-03-04T09:31:22.646864+0000","last_fullsized":"2024-03-04T09:31:22.646864+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:56:46.027981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"108'8","reported_seq":20,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-04T09:31:21.673338+0000","last_change":"2024-03-04T09:31:20.645637+0000","last_active":"2024-03-04T09:31:21.673338+0000","last_peered":"2024-03-04T09:31:21.673338+0000","last_clean":"2024-03-04T09:31:21.673338+0000","last_became_active":"2024-03-04T09:31:20.644362+0000","last_became_peered":"2024-03-04T09:31:20.644362+0000","last_unstale":"2024-03-04T09:31:21.673338+0000","last_undegraded":"2024-03-04T09:31:21.673338+0000","last_fullsized":"2024-03-04T09:31:21.673338+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_clean_scrub_stamp":"2024-03-04T09:31:19.587017+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:37:56.045623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-04T09:31:21.671465+0000","last_change":"2024-03-04T09:31:21.670263+0000","last_active":"2024-03-04T09:31:21.671465+0000","last_peered":"2024-03-04T09:31:21.671465+0000","last_clean":"2024-03-04T09:31:21.671465+0000","last_became_active":"2024-03-04T09:31:21.669724+0000","last_became_peered":"2024-03-04T09:31:21.669724+0000","last_unstale":"2024-03-04T09:31:21.671465+0000","last_undegraded":"2024-03-04T09:31:21.671465+0000","last_fullsized":"2024-03-04T09:31:21.671465+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:48:42.697121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"108'1","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-04T09:31:21.676644+0000","last_change":"2024-03-04T09:31:20.654522+0000","last_active":"2024-03-04T09:31:21.676644+0000","last_peered":"2024-03-04T09:31:21.676644+0000","last_clean":"2024-03-04T09:31:21.676644+0000","last_became_active":"2024-03-04T09:31:20.654074+0000","last_became_peered":"2024-03-04T09:31:20.654074+0000","last_unstale":"2024-03-04T09:31:21.676644+0000","last_undegraded":"2024-03-04T09:31:21.676644+0000","last_fullsized":"2024-03-04T09:31:21.676644+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_clean_scrub_stamp":"2024-03-04T09:31:19.587017+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:39:10.893276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-04T09:31:22.646406+0000","last_change":"2024-03-04T09:31:21.669715+0000","last_active":"2024-03-04T09:31:22.646406+0000","last_peered":"2024-03-04T09:31:22.646406+0000","last_clean":"2024-03-04T09:31:22.646406+0000","last_became_active":"2024-03-04T09:31:21.667192+0000","last_became_peered":"2024-03-04T09:31:21.667192+0000","last_unstale":"2024-03-04T09:31:22.646406+0000","last_undegraded":"2024-03-04T09:31:22.646406+0000","last_fullsized":"2024-03-04T09:31:22.646406+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:31:24.059038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"108'2","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-04T09:31:21.670428+0000","last_change":"2024-03-04T09:31:20.645260+0000","last_active":"2024-03-04T09:31:21.670428+0000","last_peered":"2024-03-04T09:31:21.670428+0000","last_clean":"2024-03-04T09:31:21.670428+0000","last_became_active":"2024-03-04T09:31:20.643856+0000","last_became_peered":"2024-03-04T09:31:20.643856+0000","last_unstale":"2024-03-04T09:31:21.670428+0000","last_undegraded":"2024-03-04T09:31:21.670428+0000","last_fullsized":"2024-03-04T09:31:21.670428+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_clean_scrub_stamp":"2024-03-04T09:31:19.587017+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:06:33.545112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-04T09:31:21.669337+0000","last_change":"2024-03-04T09:31:21.668837+0000","last_active":"2024-03-04T09:31:21.669337+0000","last_peered":"2024-03-04T09:31:21.669337+0000","last_clean":"2024-03-04T09:31:21.669337+0000","last_became_active":"2024-03-04T09:31:21.668565+0000","last_became_peered":"2024-03-04T09:31:21.668565+0000","last_unstale":"2024-03-04T09:31:21.669337+0000","last_undegraded":"2024-03-04T09:31:21.669337+0000","last_fullsized":"2024-03-04T09:31:21.669337+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:26:36.995948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"108'1","reported_seq":15,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-04T09:31:22.647874+0000","last_change":"2024-03-04T09:31:20.651140+0000","last_active":"2024-03-04T09:31:22.647874+0000","last_peered":"2024-03-04T09:31:22.647874+0000","last_clean":"2024-03-04T09:31:22.647874+0000","last_became_active":"2024-03-04T09:31:20.650838+0000","last_became_peered":"2024-03-04T09:31:20.650838+0000","last_unstale":"2024-03-04T09:31:22.647874+0000","last_undegraded":"2024-03-04T09:31:22.647874+0000","last_fullsized":"2024-03-04T09:31:22.647874+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_clean_scrub_stamp":"2024-03-04T09:31:19.587017+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:00:41.066368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-04T09:31:21.661075+0000","last_change":"2024-03-04T09:31:21.657802+0000","last_active":"2024-03-04T09:31:21.661075+0000","last_peered":"2024-03-04T09:31:21.661075+0000","last_clean":"2024-03-04T09:31:21.661075+0000","last_became_active":"2024-03-04T09:31:21.657200+0000","last_became_peered":"2024-03-04T09:31:21.657200+0000","last_unstale":"2024-03-04T09:31:21.661075+0000","last_undegraded":"2024-03-04T09:31:21.661075+0000","last_fullsized":"2024-03-04T09:31:21.661075+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:08:19.417156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-04T09:31:22.659193+0000","last_change":"2024-03-04T09:31:20.656233+0000","last_active":"2024-03-04T09:31:22.659193+0000","last_peered":"2024-03-04T09:31:22.659193+0000","last_clean":"2024-03-04T09:31:22.659193+0000","last_became_active":"2024-03-04T09:31:20.655917+0000","last_became_peered":"2024-03-04T09:31:20.655917+0000","last_unstale":"2024-03-04T09:31:22.659193+0000","last_undegraded":"2024-03-04T09:31:22.659193+0000","last_fullsized":"2024-03-04T09:31:22.659193+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_clean_scrub_stamp":"2024-03-04T09:31:19.587017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:59:07.044800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-04T09:31:21.666033+0000","last_change":"2024-03-04T09:31:21.663592+0000","last_active":"2024-03-04T09:31:21.666033+0000","last_peered":"2024-03-04T09:31:21.666033+0000","last_clean":"2024-03-04T09:31:21.666033+0000","last_became_active":"2024-03-04T09:31:21.663318+0000","last_became_peered":"2024-03-04T09:31:21.663318+0000","last_unstale":"2024-03-04T09:31:21.666033+0000","last_undegraded":"2024-03-04T09:31:21.666033+0000","last_fullsized":"2024-03-04T09:31:21.666033+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:39:49.775719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"108'2","reported_seq":16,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-04T09:31:22.646184+0000","last_change":"2024-03-04T09:31:20.635738+0000","last_active":"2024-03-04T09:31:22.646184+0000","last_peered":"2024-03-04T09:31:22.646184+0000","last_clean":"2024-03-04T09:31:22.646184+0000","last_became_active":"2024-03-04T09:31:20.635439+0000","last_became_peered":"2024-03-04T09:31:20.635439+0000","last_unstale":"2024-03-04T09:31:22.646184+0000","last_undegraded":"2024-03-04T09:31:22.646184+0000","last_fullsized":"2024-03-04T09:31:22.646184+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_clean_scrub_stamp":"2024-03-04T09:31:19.587017+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:54:52.269666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-04T09:31:21.664256+0000","last_change":"2024-03-04T09:31:21.659596+0000","last_active":"2024-03-04T09:31:21.664256+0000","last_peered":"2024-03-04T09:31:21.664256+0000","last_clean":"2024-03-04T09:31:21.664256+0000","last_became_active":"2024-03-04T09:31:21.659278+0000","last_became_peered":"2024-03-04T09:31:21.659278+0000","last_unstale":"2024-03-04T09:31:21.664256+0000","last_undegraded":"2024-03-04T09:31:21.664256+0000","last_fullsized":"2024-03-04T09:31:21.664256+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:05:31.021016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-04T09:31:21.638891+0000","last_change":"2024-03-04T09:31:20.641401+0000","last_active":"2024-03-04T09:31:21.638891+0000","last_peered":"2024-03-04T09:31:21.638891+0000","last_clean":"2024-03-04T09:31:21.638891+0000","last_became_active":"2024-03-04T09:31:20.640212+0000","last_became_peered":"2024-03-04T09:31:20.640212+0000","last_unstale":"2024-03-04T09:31:21.638891+0000","last_undegraded":"2024-03-04T09:31:21.638891+0000","last_fullsized":"2024-03-04T09:31:21.638891+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_clean_scrub_stamp":"2024-03-04T09:31:19.587017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:52:17.490227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-04T09:31:21.660986+0000","last_change":"2024-03-04T09:31:21.657200+0000","last_active":"2024-03-04T09:31:21.660986+0000","last_peered":"2024-03-04T09:31:21.660986+0000","last_clean":"2024-03-04T09:31:21.660986+0000","last_became_active":"2024-03-04T09:31:21.656756+0000","last_became_peered":"2024-03-04T09:31:21.656756+0000","last_unstale":"2024-03-04T09:31:21.660986+0000","last_undegraded":"2024-03-04T09:31:21.660986+0000","last_fullsized":"2024-03-04T09:31:21.660986+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:27:14.606466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-04T09:31:22.645620+0000","last_change":"2024-03-04T09:31:20.637485+0000","last_active":"2024-03-04T09:31:22.645620+0000","last_peered":"2024-03-04T09:31:22.645620+0000","last_clean":"2024-03-04T09:31:22.645620+0000","last_became_active":"2024-03-04T09:31:20.636835+0000","last_became_peered":"2024-03-04T09:31:20.636835+0000","last_unstale":"2024-03-04T09:31:22.645620+0000","last_undegraded":"2024-03-04T09:31:22.645620+0000","last_fullsized":"2024-03-04T09:31:22.645620+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_clean_scrub_stamp":"2024-03-04T09:31:19.587017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:05:50.693261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-04T09:31:22.647072+0000","last_change":"2024-03-04T09:31:21.641908+0000","last_active":"2024-03-04T09:31:22.647072+0000","last_peered":"2024-03-04T09:31:22.647072+0000","last_clean":"2024-03-04T09:31:22.647072+0000","last_became_active":"2024-03-04T09:31:21.641561+0000","last_became_peered":"2024-03-04T09:31:21.641561+0000","last_unstale":"2024-03-04T09:31:22.647072+0000","last_undegraded":"2024-03-04T09:31:22.647072+0000","last_fullsized":"2024-03-04T09:31:22.647072+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:04:36.046333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-04T09:31:21.638133+0000","last_change":"2024-03-04T09:31:20.657722+0000","last_active":"2024-03-04T09:31:21.638133+0000","last_peered":"2024-03-04T09:31:21.638133+0000","last_clean":"2024-03-04T09:31:21.638133+0000","last_became_active":"2024-03-04T09:31:20.657383+0000","last_became_peered":"2024-03-04T09:31:20.657383+0000","last_unstale":"2024-03-04T09:31:21.638133+0000","last_undegraded":"2024-03-04T09:31:21.638133+0000","last_fullsized":"2024-03-04T09:31:21.638133+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_clean_scrub_stamp":"2024-03-04T09:31:19.587017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:01:21.470739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-04T09:31:22.647725+0000","last_change":"2024-03-04T09:31:21.660695+0000","last_active":"2024-03-04T09:31:22.647725+0000","last_peered":"2024-03-04T09:31:22.647725+0000","last_clean":"2024-03-04T09:31:22.647725+0000","last_became_active":"2024-03-04T09:31:21.660385+0000","last_became_peered":"2024-03-04T09:31:21.660385+0000","last_unstale":"2024-03-04T09:31:22.647725+0000","last_undegraded":"2024-03-04T09:31:22.647725+0000","last_fullsized":"2024-03-04T09:31:22.647725+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:08:19.417156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"108'2","reported_seq":16,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-04T09:31:22.648148+0000","last_change":"2024-03-04T09:31:20.654509+0000","last_active":"2024-03-04T09:31:22.648148+0000","last_peered":"2024-03-04T09:31:22.648148+0000","last_clean":"2024-03-04T09:31:22.648148+0000","last_became_active":"2024-03-04T09:31:20.654023+0000","last_became_peered":"2024-03-04T09:31:20.654023+0000","last_unstale":"2024-03-04T09:31:22.648148+0000","last_undegraded":"2024-03-04T09:31:22.648148+0000","last_fullsized":"2024-03-04T09:31:22.648148+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_clean_scrub_stamp":"2024-03-04T09:31:19.587017+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:31:51.082464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-04T09:31:21.663743+0000","last_change":"2024-03-04T09:31:21.646610+0000","last_active":"2024-03-04T09:31:21.663743+0000","last_peered":"2024-03-04T09:31:21.663743+0000","last_clean":"2024-03-04T09:31:21.663743+0000","last_became_active":"2024-03-04T09:31:21.646322+0000","last_became_peered":"2024-03-04T09:31:21.646322+0000","last_unstale":"2024-03-04T09:31:21.663743+0000","last_undegraded":"2024-03-04T09:31:21.663743+0000","last_fullsized":"2024-03-04T09:31:21.663743+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:20:07.013843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"109'1","reported_seq":15,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-04T09:31:22.657460+0000","last_change":"2024-03-04T09:31:20.656170+0000","last_active":"2024-03-04T09:31:22.657460+0000","last_peered":"2024-03-04T09:31:22.657460+0000","last_clean":"2024-03-04T09:31:22.657460+0000","last_became_active":"2024-03-04T09:31:20.654568+0000","last_became_peered":"2024-03-04T09:31:20.654568+0000","last_unstale":"2024-03-04T09:31:22.657460+0000","last_undegraded":"2024-03-04T09:31:22.657460+0000","last_fullsized":"2024-03-04T09:31:22.657460+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_clean_scrub_stamp":"2024-03-04T09:31:19.587017+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:42:45.728646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-04T09:31:21.663237+0000","last_change":"2024-03-04T09:31:21.642718+0000","last_active":"2024-03-04T09:31:21.663237+0000","last_peered":"2024-03-04T09:31:21.663237+0000","last_clean":"2024-03-04T09:31:21.663237+0000","last_became_active":"2024-03-04T09:31:21.642416+0000","last_became_peered":"2024-03-04T09:31:21.642416+0000","last_unstale":"2024-03-04T09:31:21.663237+0000","last_undegraded":"2024-03-04T09:31:21.663237+0000","last_fullsized":"2024-03-04T09:31:21.663237+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:48:52.528076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-04T09:31:21.639308+0000","last_change":"2024-03-04T09:31:20.658189+0000","last_active":"2024-03-04T09:31:21.639308+0000","last_peered":"2024-03-04T09:31:21.639308+0000","last_clean":"2024-03-04T09:31:21.639308+0000","last_became_active":"2024-03-04T09:31:20.657860+0000","last_became_peered":"2024-03-04T09:31:20.657860+0000","last_unstale":"2024-03-04T09:31:21.639308+0000","last_undegraded":"2024-03-04T09:31:21.639308+0000","last_fullsized":"2024-03-04T09:31:21.639308+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_clean_scrub_stamp":"2024-03-04T09:31:19.587017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:53:20.181825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-04T09:31:22.647909+0000","last_change":"2024-03-04T09:31:21.647493+0000","last_active":"2024-03-04T09:31:22.647909+0000","last_peered":"2024-03-04T09:31:22.647909+0000","last_clean":"2024-03-04T09:31:22.647909+0000","last_became_active":"2024-03-04T09:31:21.647103+0000","last_became_peered":"2024-03-04T09:31:21.647103+0000","last_unstale":"2024-03-04T09:31:22.647909+0000","last_undegraded":"2024-03-04T09:31:22.647909+0000","last_fullsized":"2024-03-04T09:31:22.647909+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:31:15.039332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-04T09:31:22.647647+0000","last_change":"2024-03-04T09:31:20.648528+0000","last_active":"2024-03-04T09:31:22.647647+0000","last_peered":"2024-03-04T09:31:22.647647+0000","last_clean":"2024-03-04T09:31:22.647647+0000","last_became_active":"2024-03-04T09:31:20.648223+0000","last_became_peered":"2024-03-04T09:31:20.648223+0000","last_unstale":"2024-03-04T09:31:22.647647+0000","last_undegraded":"2024-03-04T09:31:22.647647+0000","last_fullsized":"2024-03-04T09:31:22.647647+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_clean_scrub_stamp":"2024-03-04T09:31:19.587017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:01:34.024686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-04T09:31:21.658370+0000","last_change":"2024-03-04T09:31:21.651580+0000","last_active":"2024-03-04T09:31:21.658370+0000","last_peered":"2024-03-04T09:31:21.658370+0000","last_clean":"2024-03-04T09:31:21.658370+0000","last_became_active":"2024-03-04T09:31:21.649396+0000","last_became_peered":"2024-03-04T09:31:21.649396+0000","last_unstale":"2024-03-04T09:31:21.658370+0000","last_undegraded":"2024-03-04T09:31:21.658370+0000","last_fullsized":"2024-03-04T09:31:21.658370+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:27:14.606466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-04T09:31:22.659126+0000","last_change":"2024-03-04T09:31:20.656370+0000","last_active":"2024-03-04T09:31:22.659126+0000","last_peered":"2024-03-04T09:31:22.659126+0000","last_clean":"2024-03-04T09:31:22.659126+0000","last_became_active":"2024-03-04T09:31:20.655442+0000","last_became_peered":"2024-03-04T09:31:20.655442+0000","last_unstale":"2024-03-04T09:31:22.659126+0000","last_undegraded":"2024-03-04T09:31:22.659126+0000","last_fullsized":"2024-03-04T09:31:22.659126+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_clean_scrub_stamp":"2024-03-04T09:31:19.587017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:41:12.969898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-04T09:31:21.655181+0000","last_change":"2024-03-04T09:31:21.647082+0000","last_active":"2024-03-04T09:31:21.655181+0000","last_peered":"2024-03-04T09:31:21.655181+0000","last_clean":"2024-03-04T09:31:21.655181+0000","last_became_active":"2024-03-04T09:31:21.646727+0000","last_became_peered":"2024-03-04T09:31:21.646727+0000","last_unstale":"2024-03-04T09:31:21.655181+0000","last_undegraded":"2024-03-04T09:31:21.655181+0000","last_fullsized":"2024-03-04T09:31:21.655181+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:49:17.194698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-04T09:31:21.642594+0000","last_change":"2024-03-04T09:31:20.650193+0000","last_active":"2024-03-04T09:31:21.642594+0000","last_peered":"2024-03-04T09:31:21.642594+0000","last_clean":"2024-03-04T09:31:21.642594+0000","last_became_active":"2024-03-04T09:31:20.649858+0000","last_became_peered":"2024-03-04T09:31:20.649858+0000","last_unstale":"2024-03-04T09:31:21.642594+0000","last_undegraded":"2024-03-04T09:31:21.642594+0000","last_fullsized":"2024-03-04T09:31:21.642594+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_clean_scrub_stamp":"2024-03-04T09:31:19.587017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:54:04.607782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-04T09:31:21.660691+0000","last_change":"2024-03-04T09:31:21.656806+0000","last_active":"2024-03-04T09:31:21.660691+0000","last_peered":"2024-03-04T09:31:21.660691+0000","last_clean":"2024-03-04T09:31:21.660691+0000","last_became_active":"2024-03-04T09:31:21.656528+0000","last_became_peered":"2024-03-04T09:31:21.656528+0000","last_unstale":"2024-03-04T09:31:21.660691+0000","last_undegraded":"2024-03-04T09:31:21.660691+0000","last_fullsized":"2024-03-04T09:31:21.660691+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:13:52.946588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-04T09:31:21.637873+0000","last_change":"2024-03-04T09:31:20.641748+0000","last_active":"2024-03-04T09:31:21.637873+0000","last_peered":"2024-03-04T09:31:21.637873+0000","last_clean":"2024-03-04T09:31:21.637873+0000","last_became_active":"2024-03-04T09:31:20.641335+0000","last_became_peered":"2024-03-04T09:31:20.641335+0000","last_unstale":"2024-03-04T09:31:21.637873+0000","last_undegraded":"2024-03-04T09:31:21.637873+0000","last_fullsized":"2024-03-04T09:31:21.637873+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_clean_scrub_stamp":"2024-03-04T09:31:19.587017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:00:31.024486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-04T09:31:21.664051+0000","last_change":"2024-03-04T09:31:21.655517+0000","last_active":"2024-03-04T09:31:21.664051+0000","last_peered":"2024-03-04T09:31:21.664051+0000","last_clean":"2024-03-04T09:31:21.664051+0000","last_became_active":"2024-03-04T09:31:21.650939+0000","last_became_peered":"2024-03-04T09:31:21.650939+0000","last_unstale":"2024-03-04T09:31:21.664051+0000","last_undegraded":"2024-03-04T09:31:21.664051+0000","last_fullsized":"2024-03-04T09:31:21.664051+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:35:05.912704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-04T09:31:21.638421+0000","last_change":"2024-03-04T09:31:20.657679+0000","last_active":"2024-03-04T09:31:21.638421+0000","last_peered":"2024-03-04T09:31:21.638421+0000","last_clean":"2024-03-04T09:31:21.638421+0000","last_became_active":"2024-03-04T09:31:20.657333+0000","last_became_peered":"2024-03-04T09:31:20.657333+0000","last_unstale":"2024-03-04T09:31:21.638421+0000","last_undegraded":"2024-03-04T09:31:21.638421+0000","last_fullsized":"2024-03-04T09:31:21.638421+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_clean_scrub_stamp":"2024-03-04T09:31:19.587017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:19:30.031366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-04T09:31:22.646703+0000","last_change":"2024-03-04T09:31:21.655803+0000","last_active":"2024-03-04T09:31:22.646703+0000","last_peered":"2024-03-04T09:31:22.646703+0000","last_clean":"2024-03-04T09:31:22.646703+0000","last_became_active":"2024-03-04T09:31:21.651394+0000","last_became_peered":"2024-03-04T09:31:21.651394+0000","last_unstale":"2024-03-04T09:31:22.646703+0000","last_undegraded":"2024-03-04T09:31:22.646703+0000","last_fullsized":"2024-03-04T09:31:22.646703+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:42:25.949190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-04T09:31:22.659559+0000","last_change":"2024-03-04T09:31:20.657927+0000","last_active":"2024-03-04T09:31:22.659559+0000","last_peered":"2024-03-04T09:31:22.659559+0000","last_clean":"2024-03-04T09:31:22.659559+0000","last_became_active":"2024-03-04T09:31:20.656107+0000","last_became_peered":"2024-03-04T09:31:20.656107+0000","last_unstale":"2024-03-04T09:31:22.659559+0000","last_undegraded":"2024-03-04T09:31:22.659559+0000","last_fullsized":"2024-03-04T09:31:22.659559+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_clean_scrub_stamp":"2024-03-04T09:31:19.587017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:07:54.338428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":333,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-04T09:31:21.648129+0000","last_change":"2024-03-04T09:25:09.316159+0000","last_active":"2024-03-04T09:31:21.648129+0000","last_peered":"2024-03-04T09:31:21.648129+0000","last_clean":"2024-03-04T09:31:21.648129+0000","last_became_active":"2024-03-04T09:25:09.313767+0000","last_became_peered":"2024-03-04T09:25:09.313767+0000","last_unstale":"2024-03-04T09:31:21.648129+0000","last_undegraded":"2024-03-04T09:31:21.648129+0000","last_fullsized":"2024-03-04T09:31:21.648129+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:08.110953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:08.110953+0000","last_clean_scrub_stamp":"2024-03-04T09:25:08.110953+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:34:21.745390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574926,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37240,"kb_used_data":1900,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334600,"statfs":{"total":96636764160,"available":96598630400,"internally_reserved":0,"allocated":1945600,"data_stored":1170804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574926,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36672,"kb_used_data":1328,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335168,"statfs":{"total":96636764160,"available":96599212032,"internally_reserved":0,"allocated":1359872,"data_stored":586190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574928,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36660,"kb_used_data":1320,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335180,"statfs":{"total":96636764160,"available":96599224320,"internally_reserved":0,"allocated":1351680,"data_stored":580424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574925,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36660,"kb_used_data":1312,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335180,"statfs":{"total":96636764160,"available":96599224320,"internally_reserved":0,"allocated":1343488,"data_stored":579820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574927,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36668,"kb_used_data":1328,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335172,"statfs":{"total":96636764160,"available":96599216128,"internally_reserved":0,"allocated":1359872,"data_stored":581502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574926,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36656,"kb_used_data":1312,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335184,"statfs":{"total":96636764160,"available":96599228416,"internally_reserved":0,"allocated":1343488,"data_stored":579820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574927,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36680,"kb_used_data":1332,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335160,"statfs":{"total":96636764160,"available":96599203840,"internally_reserved":0,"allocated":1363968,"data_stored":586244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574926,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37260,"kb_used_data":1912,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334580,"statfs":{"total":96636764160,"available":96598609920,"internally_reserved":0,"allocated":1957888,"data_stored":1171912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-04T09:31:26.105 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T09:31:26.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.399+0000 7f8343c00640 1 Processor -- start 2024-03-04T09:31:26.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.400+0000 7f8343c00640 1 -- start start 2024-03-04T09:31:26.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.402+0000 7f8343c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f833c0cf5c0 0x7f833c0cdc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:26.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.402+0000 7f8343c00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f833c071050 0x7f833c0714e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:26.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.402+0000 7f8343c00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f833c0b4860 con 0x7f833c0b3c90 2024-03-04T09:31:26.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.402+0000 7f8343c00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f833c0ce160 con 0x7f833c071050 2024-03-04T09:31:26.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.402+0000 7f8343c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833c0b4b40 con 0x7f833c0cf5c0 2024-03-04T09:31:26.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.402+0000 7f8341200640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f833c0b3c90 0x7f833c0b4070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:56982/0 (socket says 172.21.15.6:56982) 2024-03-04T09:31:26.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.402+0000 7f8341200640 1 -- 172.21.15.6:0/1525839365 learned_addr learned my addr 172.21.15.6:0/1525839365 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:31:26.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.403+0000 7f833be00640 1 --2- 172.21.15.6:0/1525839365 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f833c0cf5c0 0x7f833c0cdc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:26.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.403+0000 7f8341c00640 1 --2- 172.21.15.6:0/1525839365 >> v2:172.21.15.146:3300/0 conn(0x7f833c071050 0x7f833c0714e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:26.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.403+0000 7f833b400640 1 -- 172.21.15.6:0/1525839365 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2295083787 0 0) 0x7f833c0b4860 con 0x7f833c0b3c90 2024-03-04T09:31:26.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.403+0000 7f833b400640 1 -- 172.21.15.6:0/1525839365 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8320003520 con 0x7f833c0b3c90 2024-03-04T09:31:26.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.403+0000 7f833b400640 1 -- 172.21.15.6:0/1525839365 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3632533862 0 0) 0x7f8320003520 con 0x7f833c0b3c90 2024-03-04T09:31:26.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.403+0000 7f833b400640 1 -- 172.21.15.6:0/1525839365 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f833c0cf5c0 msgr2=0x7f833c0cdc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:26.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.403+0000 7f833b400640 1 --2- 172.21.15.6:0/1525839365 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f833c0cf5c0 0x7f833c0cdc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:26.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.403+0000 7f833b400640 1 -- 172.21.15.6:0/1525839365 >> v2:172.21.15.146:3300/0 conn(0x7f833c071050 msgr2=0x7f833c0714e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:26.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.403+0000 7f833b400640 1 --2- 172.21.15.6:0/1525839365 >> v2:172.21.15.146:3300/0 conn(0x7f833c071050 0x7f833c0714e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:26.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.403+0000 7f833b400640 1 -- 172.21.15.6:0/1525839365 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f833c0b4e20 con 0x7f833c0b3c90 2024-03-04T09:31:26.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.403+0000 7f833b400640 1 -- 172.21.15.6:0/1525839365 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7f832c001670 con 0x7f833c0b3c90 2024-03-04T09:31:26.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.403+0000 7f833b400640 1 -- 172.21.15.6:0/1525839365 >> v1:172.21.15.6:6789/0 conn(0x7f833c0b3c90 legacy=0x7f833c0b4070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:26.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.403+0000 7f833b400640 1 --2- 172.21.15.6:0/1525839365 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8320003a20 0x7f8320003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:26.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.403+0000 7f833b400640 1 --2- 172.21.15.6:0/1525839365 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8320004800 0x7f8320004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:26.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.403+0000 7f833b400640 1 --2- 172.21.15.6:0/1525839365 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f833c0cf5c0 0x7f83200051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:26.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.403+0000 7f833b400640 1 -- 172.21.15.6:0/1525839365 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833c0ce160 con 0x7f8320003a20 2024-03-04T09:31:26.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.403+0000 7f833b400640 1 -- 172.21.15.6:0/1525839365 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833c0b4b40 con 0x7f8320004800 2024-03-04T09:31:26.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.403+0000 7f833b400640 1 -- 172.21.15.6:0/1525839365 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f832000baa0 con 0x7f833c0cf5c0 2024-03-04T09:31:26.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.404+0000 7f8341200640 1 --2- 172.21.15.6:0/1525839365 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8320003a20 0x7f8320003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:26.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.404+0000 7f8341200640 1 -- 172.21.15.6:0/1525839365 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f833c0cf5c0 msgr2=0x7f83200051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:26.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.404+0000 7f8341200640 1 --2- 172.21.15.6:0/1525839365 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f833c0cf5c0 0x7f83200051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:26.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.404+0000 7f8341200640 1 -- 172.21.15.6:0/1525839365 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8320004800 msgr2=0x7f8320004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:26.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.404+0000 7f8341200640 1 --2- 172.21.15.6:0/1525839365 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8320004800 0x7f8320004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:26.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.404+0000 7f8341200640 1 -- 172.21.15.6:0/1525839365 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8320003520 con 0x7f8320003a20 2024-03-04T09:31:26.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.405+0000 7f8341200640 1 --2- 172.21.15.6:0/1525839365 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8320003a20 0x7f8320003e40 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7f832c00c160 tx=0x7f8320005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a3c471a8dd0d260 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:26.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.406+0000 7f833b400640 1 -- 172.21.15.6:0/1525839365 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f832c007c80 con 0x7f8320003a20 2024-03-04T09:31:26.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.406+0000 7f833b400640 1 -- 172.21.15.6:0/1525839365 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f832c030070 con 0x7f8320003a20 2024-03-04T09:31:26.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.406+0000 7f833b400640 1 -- 172.21.15.6:0/1525839365 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f832c005cf0 con 0x7f8320003a20 2024-03-04T09:31:26.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.406+0000 7f8343c00640 1 -- 172.21.15.6:0/1525839365 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8320003a20 msgr2=0x7f8320003e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:26.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.406+0000 7f8343c00640 1 --2- 172.21.15.6:0/1525839365 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8320003a20 0x7f8320003e40 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7f832c00c160 tx=0x7f8320005830 comp rx=0 tx=0).stop 2024-03-04T09:31:26.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.406+0000 7f8341c00640 1 -- 172.21.15.6:0/1525839365 reap_dead start 2024-03-04T09:31:26.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.406+0000 7f8343c00640 1 -- 172.21.15.6:0/1525839365 shutdown_connections 2024-03-04T09:31:26.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.406+0000 7f8343c00640 1 -- 172.21.15.6:0/1525839365 >> 172.21.15.6:0/1525839365 conn(0x7f833c06afb0 msgr2=0x7f833c06b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:26.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.406+0000 7f8343c00640 1 -- 172.21.15.6:0/1525839365 shutdown_connections 2024-03-04T09:31:26.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.406+0000 7f8343c00640 1 -- 172.21.15.6:0/1525839365 wait complete. 2024-03-04T09:31:26.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.409+0000 7f8343c00640 1 Processor -- start 2024-03-04T09:31:26.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.410+0000 7f8343c00640 1 -- start start 2024-03-04T09:31:26.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.411+0000 7f8343c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8320003a20 0x7f833c1517a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:26.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.411+0000 7f8343c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833c155560 0x7f833c1559e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:26.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.411+0000 7f8343c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f833c0b0bf0 0x7f833c0b1070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:26.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.411+0000 7f8343c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833c0cdc20 con 0x7f8320003a20 2024-03-04T09:31:26.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.411+0000 7f8343c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833c15b340 con 0x7f833c155560 2024-03-04T09:31:26.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.411+0000 7f8343c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833c0cdda0 con 0x7f833c0b0bf0 2024-03-04T09:31:26.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.411+0000 7f8341200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8320003a20 0x7f833c1517a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:26.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.412+0000 7f8341200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8320003a20 0x7f833c1517a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:32812/0 (socket says 172.21.15.6:32812) 2024-03-04T09:31:26.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.412+0000 7f8341200640 1 -- 172.21.15.6:0/1545012810 learned_addr learned my addr 172.21.15.6:0/1545012810 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:31:26.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.412+0000 7f833be00640 1 --2- 172.21.15.6:0/1545012810 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833c155560 0x7f833c1559e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:26.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.412+0000 7f8341c00640 1 --2- 172.21.15.6:0/1545012810 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f833c0b0bf0 0x7f833c0b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:26.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.412+0000 7f8341200640 1 -- 172.21.15.6:0/1545012810 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f833c0b0bf0 msgr2=0x7f833c0b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:26.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.412+0000 7f8341200640 1 --2- 172.21.15.6:0/1545012810 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f833c0b0bf0 0x7f833c0b1070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:26.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.412+0000 7f8341200640 1 -- 172.21.15.6:0/1545012810 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833c155560 msgr2=0x7f833c1559e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:26.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.412+0000 7f8341200640 1 --2- 172.21.15.6:0/1545012810 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833c155560 0x7f833c1559e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:26.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.412+0000 7f8341200640 1 -- 172.21.15.6:0/1545012810 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f833c0c92d0 con 0x7f8320003a20 2024-03-04T09:31:26.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.412+0000 7f8341200640 1 --2- 172.21.15.6:0/1545012810 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8320003a20 0x7f833c1517a0 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7f832c02c540 tx=0x7f833c070a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:26.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.413+0000 7f8338c00640 1 -- 172.21.15.6:0/1545012810 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f832c045070 con 0x7f8320003a20 2024-03-04T09:31:26.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.413+0000 7f8338c00640 1 -- 172.21.15.6:0/1545012810 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f832c004bf0 con 0x7f8320003a20 2024-03-04T09:31:26.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.413+0000 7f8338c00640 1 -- 172.21.15.6:0/1545012810 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f832c007c60 con 0x7f8320003a20 2024-03-04T09:31:26.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.413+0000 7f8343c00640 1 -- 172.21.15.6:0/1545012810 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f833c0c95b0 con 0x7f8320003a20 2024-03-04T09:31:26.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.413+0000 7f8343c00640 1 -- 172.21.15.6:0/1545012810 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f833c0c9b70 con 0x7f8320003a20 2024-03-04T09:31:26.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.414+0000 7f8343c00640 1 -- 172.21.15.6:0/1545012810 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f833c069510 con 0x7f8320003a20 2024-03-04T09:31:26.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.415+0000 7f8338c00640 1 -- 172.21.15.6:0/1545012810 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f832c007070 con 0x7f8320003a20 2024-03-04T09:31:26.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.416+0000 7f8338c00640 1 --2- 172.21.15.6:0/1545012810 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f831c06b4d0 0x7f831c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:26.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.417+0000 7f8338c00640 1 -- 172.21.15.6:0/1545012810 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f832c0ab590 con 0x7f8320003a20 2024-03-04T09:31:26.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.417+0000 7f833be00640 1 --2- 172.21.15.6:0/1545012810 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f831c06b4d0 0x7f831c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:26.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.418+0000 7f833be00640 1 --2- 172.21.15.6:0/1545012810 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f831c06b4d0 0x7f831c06d990 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f833c15a8a0 tx=0x7f8328008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:26.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.421+0000 7f8338c00640 1 -- 172.21.15.6:0/1545012810 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f832c07a360 con 0x7f8320003a20 2024-03-04T09:31:26.569 INFO:tasks.ceph.mds.a.smithi006.stderr:2024-03-04T09:31:26.572+0000 7fab75600640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-04T09:31:26.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.617+0000 7f8343c00640 1 -- 172.21.15.6:0/1545012810 --> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f833c04f3a0 con 0x7f831c06b4d0 2024-03-04T09:31:26.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.635+0000 7f8338c00640 1 -- 172.21.15.6:0/1545012810 <== mgr.4113 v2:172.21.15.146:6832/1788613334 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252440 (secure 0 0 0) 0x7f833c04f3a0 con 0x7f831c06b4d0 2024-03-04T09:31:26.635 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:31:26.636 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-03-04T09:31:26.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.640+0000 7f8343c00640 1 -- 172.21.15.6:0/1545012810 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f831c06b4d0 msgr2=0x7f831c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:26.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.640+0000 7f8343c00640 1 --2- 172.21.15.6:0/1545012810 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f831c06b4d0 0x7f831c06d990 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f833c15a8a0 tx=0x7f8328008040 comp rx=0 tx=0).stop 2024-03-04T09:31:26.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.640+0000 7f8343c00640 1 -- 172.21.15.6:0/1545012810 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8320003a20 msgr2=0x7f833c1517a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:26.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.640+0000 7f8343c00640 1 --2- 172.21.15.6:0/1545012810 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8320003a20 0x7f833c1517a0 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7f832c02c540 tx=0x7f833c070a30 comp rx=0 tx=0).stop 2024-03-04T09:31:26.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.640+0000 7f8343c00640 1 -- 172.21.15.6:0/1545012810 shutdown_connections 2024-03-04T09:31:26.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.640+0000 7f8343c00640 1 --2- 172.21.15.6:0/1545012810 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f831c06b4d0 0x7f831c06d990 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:26.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.640+0000 7f8343c00640 1 --2- 172.21.15.6:0/1545012810 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f833c0b0bf0 0x7f833c0b1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:26.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.640+0000 7f8343c00640 1 --2- 172.21.15.6:0/1545012810 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833c155560 0x7f833c1559e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:26.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.640+0000 7f8343c00640 1 --2- 172.21.15.6:0/1545012810 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8320003a20 0x7f833c1517a0 unknown :-1 s=CLOSED pgs=1255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:26.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.640+0000 7f8343c00640 1 -- 172.21.15.6:0/1545012810 >> 172.21.15.6:0/1545012810 conn(0x7f833c06afb0 msgr2=0x7f833c156ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:26.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.640+0000 7f8343c00640 1 -- 172.21.15.6:0/1545012810 shutdown_connections 2024-03-04T09:31:26.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.640+0000 7f8343c00640 1 -- 172.21.15.6:0/1545012810 wait complete. 2024-03-04T09:31:26.658 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":291,"stamp":"2024-03-04T09:31:25.992024+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":98,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":294496,"kb_used_data":11744,"kb_used_omap":12,"kb_used_meta":282611,"kb_avail":754680224,"statfs":{"total":773094113280,"available":772792549376,"internally_reserved":0,"allocated":12025856,"data_stored":5836716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":289394224},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002543"},"pg_stats":[{"pgid":"17.3f","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-04T09:31:21.664184+0000","last_change":"2024-03-04T09:31:21.656440+0000","last_active":"2024-03-04T09:31:21.664184+0000","last_peered":"2024-03-04T09:31:21.664184+0000","last_clean":"2024-03-04T09:31:21.664184+0000","last_became_active":"2024-03-04T09:31:21.656051+0000","last_became_peered":"2024-03-04T09:31:21.656051+0000","last_unstale":"2024-03-04T09:31:21.664184+0000","last_undegraded":"2024-03-04T09:31:21.664184+0000","last_fullsized":"2024-03-04T09:31:21.664184+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:32:51.022938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-04T09:31:21.658551+0000","last_change":"2024-03-04T09:31:21.654498+0000","last_active":"2024-03-04T09:31:21.658551+0000","last_peered":"2024-03-04T09:31:21.658551+0000","last_clean":"2024-03-04T09:31:21.658551+0000","last_became_active":"2024-03-04T09:31:21.653792+0000","last_became_peered":"2024-03-04T09:31:21.653792+0000","last_unstale":"2024-03-04T09:31:21.658551+0000","last_undegraded":"2024-03-04T09:31:21.658551+0000","last_fullsized":"2024-03-04T09:31:21.658551+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:01:45.712689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-04T09:31:21.666849+0000","last_change":"2024-03-04T09:31:21.665285+0000","last_active":"2024-03-04T09:31:21.666849+0000","last_peered":"2024-03-04T09:31:21.666849+0000","last_clean":"2024-03-04T09:31:21.666849+0000","last_became_active":"2024-03-04T09:31:21.664700+0000","last_became_peered":"2024-03-04T09:31:21.664700+0000","last_unstale":"2024-03-04T09:31:21.666849+0000","last_undegraded":"2024-03-04T09:31:21.666849+0000","last_fullsized":"2024-03-04T09:31:21.666849+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:01:35.042046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-04T09:31:21.664775+0000","last_change":"2024-03-04T09:31:21.663374+0000","last_active":"2024-03-04T09:31:21.664775+0000","last_peered":"2024-03-04T09:31:21.664775+0000","last_clean":"2024-03-04T09:31:21.664775+0000","last_became_active":"2024-03-04T09:31:21.663043+0000","last_became_peered":"2024-03-04T09:31:21.663043+0000","last_unstale":"2024-03-04T09:31:21.664775+0000","last_undegraded":"2024-03-04T09:31:21.664775+0000","last_fullsized":"2024-03-04T09:31:21.664775+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:47:19.591453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-04T09:31:21.653242+0000","last_change":"2024-03-04T09:31:21.650272+0000","last_active":"2024-03-04T09:31:21.653242+0000","last_peered":"2024-03-04T09:31:21.653242+0000","last_clean":"2024-03-04T09:31:21.653242+0000","last_became_active":"2024-03-04T09:31:21.649896+0000","last_became_peered":"2024-03-04T09:31:21.649896+0000","last_unstale":"2024-03-04T09:31:21.653242+0000","last_undegraded":"2024-03-04T09:31:21.653242+0000","last_fullsized":"2024-03-04T09:31:21.653242+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:23:44.996522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-04T09:31:22.645879+0000","last_change":"2024-03-04T09:31:21.646820+0000","last_active":"2024-03-04T09:31:22.645879+0000","last_peered":"2024-03-04T09:31:22.645879+0000","last_clean":"2024-03-04T09:31:22.645879+0000","last_became_active":"2024-03-04T09:31:21.646519+0000","last_became_peered":"2024-03-04T09:31:21.646519+0000","last_unstale":"2024-03-04T09:31:22.645879+0000","last_undegraded":"2024-03-04T09:31:22.645879+0000","last_fullsized":"2024-03-04T09:31:22.645879+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:36:41.886936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-04T09:31:22.659368+0000","last_change":"2024-03-04T09:31:21.638021+0000","last_active":"2024-03-04T09:31:22.659368+0000","last_peered":"2024-03-04T09:31:22.659368+0000","last_clean":"2024-03-04T09:31:22.659368+0000","last_became_active":"2024-03-04T09:31:21.637131+0000","last_became_peered":"2024-03-04T09:31:21.637131+0000","last_unstale":"2024-03-04T09:31:22.659368+0000","last_undegraded":"2024-03-04T09:31:22.659368+0000","last_fullsized":"2024-03-04T09:31:22.659368+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:48:59.896210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-04T09:31:21.659506+0000","last_change":"2024-03-04T09:31:21.656178+0000","last_active":"2024-03-04T09:31:21.659506+0000","last_peered":"2024-03-04T09:31:21.659506+0000","last_clean":"2024-03-04T09:31:21.659506+0000","last_became_active":"2024-03-04T09:31:21.655792+0000","last_became_peered":"2024-03-04T09:31:21.655792+0000","last_unstale":"2024-03-04T09:31:21.659506+0000","last_undegraded":"2024-03-04T09:31:21.659506+0000","last_fullsized":"2024-03-04T09:31:21.659506+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:15:55.025031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-04T09:31:21.664258+0000","last_change":"2024-03-04T09:31:21.660721+0000","last_active":"2024-03-04T09:31:21.664258+0000","last_peered":"2024-03-04T09:31:21.664258+0000","last_clean":"2024-03-04T09:31:21.664258+0000","last_became_active":"2024-03-04T09:31:21.660440+0000","last_became_peered":"2024-03-04T09:31:21.660440+0000","last_unstale":"2024-03-04T09:31:21.664258+0000","last_undegraded":"2024-03-04T09:31:21.664258+0000","last_fullsized":"2024-03-04T09:31:21.664258+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:11:02.962349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-04T09:31:21.669016+0000","last_change":"2024-03-04T09:31:21.660149+0000","last_active":"2024-03-04T09:31:21.669016+0000","last_peered":"2024-03-04T09:31:21.669016+0000","last_clean":"2024-03-04T09:31:21.669016+0000","last_became_active":"2024-03-04T09:31:21.659870+0000","last_became_peered":"2024-03-04T09:31:21.659870+0000","last_unstale":"2024-03-04T09:31:21.669016+0000","last_undegraded":"2024-03-04T09:31:21.669016+0000","last_fullsized":"2024-03-04T09:31:21.669016+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:33:22.518385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-04T09:31:21.657220+0000","last_change":"2024-03-04T09:31:21.650182+0000","last_active":"2024-03-04T09:31:21.657220+0000","last_peered":"2024-03-04T09:31:21.657220+0000","last_clean":"2024-03-04T09:31:21.657220+0000","last_became_active":"2024-03-04T09:31:21.649920+0000","last_became_peered":"2024-03-04T09:31:21.649920+0000","last_unstale":"2024-03-04T09:31:21.657220+0000","last_undegraded":"2024-03-04T09:31:21.657220+0000","last_fullsized":"2024-03-04T09:31:21.657220+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:15:55.025031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-04T09:31:21.664747+0000","last_change":"2024-03-04T09:31:21.662634+0000","last_active":"2024-03-04T09:31:21.664747+0000","last_peered":"2024-03-04T09:31:21.664747+0000","last_clean":"2024-03-04T09:31:21.664747+0000","last_became_active":"2024-03-04T09:31:21.648929+0000","last_became_peered":"2024-03-04T09:31:21.648929+0000","last_unstale":"2024-03-04T09:31:21.664747+0000","last_undegraded":"2024-03-04T09:31:21.664747+0000","last_fullsized":"2024-03-04T09:31:21.664747+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:35:45.458664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-04T09:31:21.659158+0000","last_change":"2024-03-04T09:31:21.655625+0000","last_active":"2024-03-04T09:31:21.659158+0000","last_peered":"2024-03-04T09:31:21.659158+0000","last_clean":"2024-03-04T09:31:21.659158+0000","last_became_active":"2024-03-04T09:31:21.655308+0000","last_became_peered":"2024-03-04T09:31:21.655308+0000","last_unstale":"2024-03-04T09:31:21.659158+0000","last_undegraded":"2024-03-04T09:31:21.659158+0000","last_fullsized":"2024-03-04T09:31:21.659158+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:21:39.986937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-04T09:31:21.663771+0000","last_change":"2024-03-04T09:31:21.661136+0000","last_active":"2024-03-04T09:31:21.663771+0000","last_peered":"2024-03-04T09:31:21.663771+0000","last_clean":"2024-03-04T09:31:21.663771+0000","last_became_active":"2024-03-04T09:31:21.660870+0000","last_became_peered":"2024-03-04T09:31:21.660870+0000","last_unstale":"2024-03-04T09:31:21.663771+0000","last_undegraded":"2024-03-04T09:31:21.663771+0000","last_fullsized":"2024-03-04T09:31:21.663771+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:31:52.099825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-04T09:31:22.645575+0000","last_change":"2024-03-04T09:31:21.644245+0000","last_active":"2024-03-04T09:31:22.645575+0000","last_peered":"2024-03-04T09:31:22.645575+0000","last_clean":"2024-03-04T09:31:22.645575+0000","last_became_active":"2024-03-04T09:31:21.643917+0000","last_became_peered":"2024-03-04T09:31:21.643917+0000","last_unstale":"2024-03-04T09:31:22.645575+0000","last_undegraded":"2024-03-04T09:31:22.645575+0000","last_fullsized":"2024-03-04T09:31:22.645575+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:18:11.958968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-04T09:31:21.663692+0000","last_change":"2024-03-04T09:31:21.651689+0000","last_active":"2024-03-04T09:31:21.663692+0000","last_peered":"2024-03-04T09:31:21.663692+0000","last_clean":"2024-03-04T09:31:21.663692+0000","last_became_active":"2024-03-04T09:31:21.648891+0000","last_became_peered":"2024-03-04T09:31:21.648891+0000","last_unstale":"2024-03-04T09:31:21.663692+0000","last_undegraded":"2024-03-04T09:31:21.663692+0000","last_fullsized":"2024-03-04T09:31:21.663692+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:02:39.356053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-04T09:31:21.657226+0000","last_change":"2024-03-04T09:31:21.650711+0000","last_active":"2024-03-04T09:31:21.657226+0000","last_peered":"2024-03-04T09:31:21.657226+0000","last_clean":"2024-03-04T09:31:21.657226+0000","last_became_active":"2024-03-04T09:31:21.650103+0000","last_became_peered":"2024-03-04T09:31:21.650103+0000","last_unstale":"2024-03-04T09:31:21.657226+0000","last_undegraded":"2024-03-04T09:31:21.657226+0000","last_fullsized":"2024-03-04T09:31:21.657226+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:13:52.946588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-04T09:31:22.647679+0000","last_change":"2024-03-04T09:31:21.647968+0000","last_active":"2024-03-04T09:31:22.647679+0000","last_peered":"2024-03-04T09:31:22.647679+0000","last_clean":"2024-03-04T09:31:22.647679+0000","last_became_active":"2024-03-04T09:31:21.647092+0000","last_became_peered":"2024-03-04T09:31:21.647092+0000","last_unstale":"2024-03-04T09:31:22.647679+0000","last_undegraded":"2024-03-04T09:31:22.647679+0000","last_fullsized":"2024-03-04T09:31:22.647679+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:27:14.606466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-04T09:31:21.661768+0000","last_change":"2024-03-04T09:31:21.658975+0000","last_active":"2024-03-04T09:31:21.661768+0000","last_peered":"2024-03-04T09:31:21.661768+0000","last_clean":"2024-03-04T09:31:21.661768+0000","last_became_active":"2024-03-04T09:31:21.658140+0000","last_became_peered":"2024-03-04T09:31:21.658140+0000","last_unstale":"2024-03-04T09:31:21.661768+0000","last_undegraded":"2024-03-04T09:31:21.661768+0000","last_fullsized":"2024-03-04T09:31:21.661768+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:31:15.039332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-04T09:31:21.657621+0000","last_change":"2024-03-04T09:31:21.653591+0000","last_active":"2024-03-04T09:31:21.657621+0000","last_peered":"2024-03-04T09:31:21.657621+0000","last_clean":"2024-03-04T09:31:21.657621+0000","last_became_active":"2024-03-04T09:31:21.652739+0000","last_became_peered":"2024-03-04T09:31:21.652739+0000","last_unstale":"2024-03-04T09:31:21.657621+0000","last_undegraded":"2024-03-04T09:31:21.657621+0000","last_fullsized":"2024-03-04T09:31:21.657621+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:08:19.417156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-04T09:31:21.645860+0000","last_change":"2024-03-04T09:31:21.642542+0000","last_active":"2024-03-04T09:31:21.645860+0000","last_peered":"2024-03-04T09:31:21.645860+0000","last_clean":"2024-03-04T09:31:21.645860+0000","last_became_active":"2024-03-04T09:31:21.641810+0000","last_became_peered":"2024-03-04T09:31:21.641810+0000","last_unstale":"2024-03-04T09:31:21.645860+0000","last_undegraded":"2024-03-04T09:31:21.645860+0000","last_fullsized":"2024-03-04T09:31:21.645860+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:20:05.757250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-04T09:31:22.647080+0000","last_change":"2024-03-04T09:31:21.646242+0000","last_active":"2024-03-04T09:31:22.647080+0000","last_peered":"2024-03-04T09:31:22.647080+0000","last_clean":"2024-03-04T09:31:22.647080+0000","last_became_active":"2024-03-04T09:31:21.645931+0000","last_became_peered":"2024-03-04T09:31:21.645931+0000","last_unstale":"2024-03-04T09:31:22.647080+0000","last_undegraded":"2024-03-04T09:31:22.647080+0000","last_fullsized":"2024-03-04T09:31:22.647080+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:01:22.488099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-04T09:31:21.646201+0000","last_change":"2024-03-04T09:31:21.643743+0000","last_active":"2024-03-04T09:31:21.646201+0000","last_peered":"2024-03-04T09:31:21.646201+0000","last_clean":"2024-03-04T09:31:21.646201+0000","last_became_active":"2024-03-04T09:31:21.643434+0000","last_became_peered":"2024-03-04T09:31:21.643434+0000","last_unstale":"2024-03-04T09:31:21.646201+0000","last_undegraded":"2024-03-04T09:31:21.646201+0000","last_fullsized":"2024-03-04T09:31:21.646201+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:00:42.083728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-04T09:31:22.646321+0000","last_change":"2024-03-04T09:31:21.657735+0000","last_active":"2024-03-04T09:31:22.646321+0000","last_peered":"2024-03-04T09:31:22.646321+0000","last_clean":"2024-03-04T09:31:22.646321+0000","last_became_active":"2024-03-04T09:31:21.657393+0000","last_became_peered":"2024-03-04T09:31:21.657393+0000","last_unstale":"2024-03-04T09:31:22.646321+0000","last_undegraded":"2024-03-04T09:31:22.646321+0000","last_fullsized":"2024-03-04T09:31:22.646321+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:31:06.794122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-04T09:31:21.658137+0000","last_change":"2024-03-04T09:31:21.653619+0000","last_active":"2024-03-04T09:31:21.658137+0000","last_peered":"2024-03-04T09:31:21.658137+0000","last_clean":"2024-03-04T09:31:21.658137+0000","last_became_active":"2024-03-04T09:31:21.649674+0000","last_became_peered":"2024-03-04T09:31:21.649674+0000","last_unstale":"2024-03-04T09:31:21.658137+0000","last_undegraded":"2024-03-04T09:31:21.658137+0000","last_fullsized":"2024-03-04T09:31:21.658137+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:31:15.039332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-04T09:31:21.668995+0000","last_change":"2024-03-04T09:31:21.659544+0000","last_active":"2024-03-04T09:31:21.668995+0000","last_peered":"2024-03-04T09:31:21.668995+0000","last_clean":"2024-03-04T09:31:21.668995+0000","last_became_active":"2024-03-04T09:31:21.659232+0000","last_became_peered":"2024-03-04T09:31:21.659232+0000","last_unstale":"2024-03-04T09:31:21.668995+0000","last_undegraded":"2024-03-04T09:31:21.668995+0000","last_fullsized":"2024-03-04T09:31:21.668995+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:02:33.790988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-04T09:31:22.646403+0000","last_change":"2024-03-04T09:31:21.669416+0000","last_active":"2024-03-04T09:31:22.646403+0000","last_peered":"2024-03-04T09:31:22.646403+0000","last_clean":"2024-03-04T09:31:22.646403+0000","last_became_active":"2024-03-04T09:31:21.669119+0000","last_became_peered":"2024-03-04T09:31:21.669119+0000","last_unstale":"2024-03-04T09:31:22.646403+0000","last_undegraded":"2024-03-04T09:31:22.646403+0000","last_fullsized":"2024-03-04T09:31:22.646403+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:49:19.511478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-04T09:31:22.647307+0000","last_change":"2024-03-04T09:31:21.644853+0000","last_active":"2024-03-04T09:31:22.647307+0000","last_peered":"2024-03-04T09:31:22.647307+0000","last_clean":"2024-03-04T09:31:22.647307+0000","last_became_active":"2024-03-04T09:31:21.644525+0000","last_became_peered":"2024-03-04T09:31:21.644525+0000","last_unstale":"2024-03-04T09:31:22.647307+0000","last_undegraded":"2024-03-04T09:31:22.647307+0000","last_fullsized":"2024-03-04T09:31:22.647307+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:53:21.199185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-04T09:31:21.655206+0000","last_change":"2024-03-04T09:31:21.648245+0000","last_active":"2024-03-04T09:31:21.655206+0000","last_peered":"2024-03-04T09:31:21.655206+0000","last_clean":"2024-03-04T09:31:21.655206+0000","last_became_active":"2024-03-04T09:31:21.647640+0000","last_became_peered":"2024-03-04T09:31:21.647640+0000","last_unstale":"2024-03-04T09:31:21.655206+0000","last_undegraded":"2024-03-04T09:31:21.655206+0000","last_fullsized":"2024-03-04T09:31:21.655206+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:38:38.988513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-04T09:31:22.659730+0000","last_change":"2024-03-04T09:31:21.641573+0000","last_active":"2024-03-04T09:31:22.659730+0000","last_peered":"2024-03-04T09:31:22.659730+0000","last_clean":"2024-03-04T09:31:22.659730+0000","last_became_active":"2024-03-04T09:31:21.641282+0000","last_became_peered":"2024-03-04T09:31:21.641282+0000","last_unstale":"2024-03-04T09:31:22.659730+0000","last_undegraded":"2024-03-04T09:31:22.659730+0000","last_fullsized":"2024-03-04T09:31:22.659730+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:04:49.856588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-04T09:31:21.668523+0000","last_change":"2024-03-04T09:31:21.663250+0000","last_active":"2024-03-04T09:31:21.668523+0000","last_peered":"2024-03-04T09:31:21.668523+0000","last_clean":"2024-03-04T09:31:21.668523+0000","last_became_active":"2024-03-04T09:31:21.662902+0000","last_became_peered":"2024-03-04T09:31:21.662902+0000","last_unstale":"2024-03-04T09:31:21.668523+0000","last_undegraded":"2024-03-04T09:31:21.668523+0000","last_fullsized":"2024-03-04T09:31:21.668523+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:28:38.909935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-04T09:31:22.644863+0000","last_change":"2024-03-04T09:31:21.640266+0000","last_active":"2024-03-04T09:31:22.644863+0000","last_peered":"2024-03-04T09:31:22.644863+0000","last_clean":"2024-03-04T09:31:22.644863+0000","last_became_active":"2024-03-04T09:31:21.639950+0000","last_became_peered":"2024-03-04T09:31:21.639950+0000","last_unstale":"2024-03-04T09:31:22.644863+0000","last_undegraded":"2024-03-04T09:31:22.644863+0000","last_fullsized":"2024-03-04T09:31:22.644863+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:01:50.307270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-04T09:31:22.646513+0000","last_change":"2024-03-04T09:31:21.641691+0000","last_active":"2024-03-04T09:31:22.646513+0000","last_peered":"2024-03-04T09:31:22.646513+0000","last_clean":"2024-03-04T09:31:22.646513+0000","last_became_active":"2024-03-04T09:31:21.640669+0000","last_became_peered":"2024-03-04T09:31:21.640669+0000","last_unstale":"2024-03-04T09:31:22.646513+0000","last_undegraded":"2024-03-04T09:31:22.646513+0000","last_fullsized":"2024-03-04T09:31:22.646513+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:48:59.896210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"108'4","reported_seq":18,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-04T09:31:22.659630+0000","last_change":"2024-03-04T09:31:20.653746+0000","last_active":"2024-03-04T09:31:22.659630+0000","last_peered":"2024-03-04T09:31:22.659630+0000","last_clean":"2024-03-04T09:31:22.659630+0000","last_became_active":"2024-03-04T09:31:20.653073+0000","last_became_peered":"2024-03-04T09:31:20.653073+0000","last_unstale":"2024-03-04T09:31:22.659630+0000","last_undegraded":"2024-03-04T09:31:22.659630+0000","last_fullsized":"2024-03-04T09:31:22.659630+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_clean_scrub_stamp":"2024-03-04T09:31:19.587017+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:53:20.852514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-04T09:31:22.648039+0000","last_change":"2024-03-04T09:31:21.651677+0000","last_active":"2024-03-04T09:31:22.648039+0000","last_peered":"2024-03-04T09:31:22.648039+0000","last_clean":"2024-03-04T09:31:22.648039+0000","last_became_active":"2024-03-04T09:31:21.651318+0000","last_became_peered":"2024-03-04T09:31:21.651318+0000","last_unstale":"2024-03-04T09:31:22.648039+0000","last_undegraded":"2024-03-04T09:31:22.648039+0000","last_fullsized":"2024-03-04T09:31:22.648039+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:01:45.712689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"108'1","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-04T09:31:21.666790+0000","last_change":"2024-03-04T09:31:20.654866+0000","last_active":"2024-03-04T09:31:21.666790+0000","last_peered":"2024-03-04T09:31:21.666790+0000","last_clean":"2024-03-04T09:31:21.666790+0000","last_became_active":"2024-03-04T09:31:20.654473+0000","last_became_peered":"2024-03-04T09:31:20.654473+0000","last_unstale":"2024-03-04T09:31:21.666790+0000","last_undegraded":"2024-03-04T09:31:21.666790+0000","last_fullsized":"2024-03-04T09:31:21.666790+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_clean_scrub_stamp":"2024-03-04T09:31:19.587017+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:38:37.971153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-04T09:31:22.646299+0000","last_change":"2024-03-04T09:31:21.641599+0000","last_active":"2024-03-04T09:31:22.646299+0000","last_peered":"2024-03-04T09:31:22.646299+0000","last_clean":"2024-03-04T09:31:22.646299+0000","last_became_active":"2024-03-04T09:31:21.641317+0000","last_became_peered":"2024-03-04T09:31:21.641317+0000","last_unstale":"2024-03-04T09:31:22.646299+0000","last_undegraded":"2024-03-04T09:31:22.646299+0000","last_fullsized":"2024-03-04T09:31:22.646299+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:03:27.258784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"108'2","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-04T09:31:21.669016+0000","last_change":"2024-03-04T09:31:20.640468+0000","last_active":"2024-03-04T09:31:21.669016+0000","last_peered":"2024-03-04T09:31:21.669016+0000","last_clean":"2024-03-04T09:31:21.669016+0000","last_became_active":"2024-03-04T09:31:20.640073+0000","last_became_peered":"2024-03-04T09:31:20.640073+0000","last_unstale":"2024-03-04T09:31:21.669016+0000","last_undegraded":"2024-03-04T09:31:21.669016+0000","last_fullsized":"2024-03-04T09:31:21.669016+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_clean_scrub_stamp":"2024-03-04T09:31:19.587017+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:57:57.417351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-04T09:31:21.661791+0000","last_change":"2024-03-04T09:31:21.659369+0000","last_active":"2024-03-04T09:31:21.661791+0000","last_peered":"2024-03-04T09:31:21.661791+0000","last_clean":"2024-03-04T09:31:21.661791+0000","last_became_active":"2024-03-04T09:31:21.659082+0000","last_became_peered":"2024-03-04T09:31:21.659082+0000","last_unstale":"2024-03-04T09:31:21.661791+0000","last_undegraded":"2024-03-04T09:31:21.661791+0000","last_fullsized":"2024-03-04T09:31:21.661791+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:01:45.712689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"108'2","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-04T09:31:21.669430+0000","last_change":"2024-03-04T09:31:20.641839+0000","last_active":"2024-03-04T09:31:21.669430+0000","last_peered":"2024-03-04T09:31:21.669430+0000","last_clean":"2024-03-04T09:31:21.669430+0000","last_became_active":"2024-03-04T09:31:20.640151+0000","last_became_peered":"2024-03-04T09:31:20.640151+0000","last_unstale":"2024-03-04T09:31:21.669430+0000","last_undegraded":"2024-03-04T09:31:21.669430+0000","last_fullsized":"2024-03-04T09:31:21.669430+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_clean_scrub_stamp":"2024-03-04T09:31:19.587017+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:27:11.829505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-04T09:31:21.661848+0000","last_change":"2024-03-04T09:31:21.650420+0000","last_active":"2024-03-04T09:31:21.661848+0000","last_peered":"2024-03-04T09:31:21.661848+0000","last_clean":"2024-03-04T09:31:21.661848+0000","last_became_active":"2024-03-04T09:31:21.648862+0000","last_became_peered":"2024-03-04T09:31:21.648862+0000","last_unstale":"2024-03-04T09:31:21.661848+0000","last_undegraded":"2024-03-04T09:31:21.661848+0000","last_fullsized":"2024-03-04T09:31:21.661848+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:55:52.210159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"108'1","reported_seq":15,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-04T09:31:22.659173+0000","last_change":"2024-03-04T09:31:20.655416+0000","last_active":"2024-03-04T09:31:22.659173+0000","last_peered":"2024-03-04T09:31:22.659173+0000","last_clean":"2024-03-04T09:31:22.659173+0000","last_became_active":"2024-03-04T09:31:20.655071+0000","last_became_peered":"2024-03-04T09:31:20.655071+0000","last_unstale":"2024-03-04T09:31:22.659173+0000","last_undegraded":"2024-03-04T09:31:22.659173+0000","last_fullsized":"2024-03-04T09:31:22.659173+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_clean_scrub_stamp":"2024-03-04T09:31:19.587017+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:26:04.483161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-04T09:31:21.655229+0000","last_change":"2024-03-04T09:31:21.649417+0000","last_active":"2024-03-04T09:31:21.655229+0000","last_peered":"2024-03-04T09:31:21.655229+0000","last_clean":"2024-03-04T09:31:21.655229+0000","last_became_active":"2024-03-04T09:31:21.649099+0000","last_became_peered":"2024-03-04T09:31:21.649099+0000","last_unstale":"2024-03-04T09:31:21.655229+0000","last_undegraded":"2024-03-04T09:31:21.655229+0000","last_fullsized":"2024-03-04T09:31:21.655229+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:21:39.986937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"108'1","reported_seq":15,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-04T09:31:22.647350+0000","last_change":"2024-03-04T09:31:20.652889+0000","last_active":"2024-03-04T09:31:22.647350+0000","last_peered":"2024-03-04T09:31:22.647350+0000","last_clean":"2024-03-04T09:31:22.647350+0000","last_became_active":"2024-03-04T09:31:20.652498+0000","last_became_peered":"2024-03-04T09:31:20.652498+0000","last_unstale":"2024-03-04T09:31:22.647350+0000","last_undegraded":"2024-03-04T09:31:22.647350+0000","last_fullsized":"2024-03-04T09:31:22.647350+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_clean_scrub_stamp":"2024-03-04T09:31:19.587017+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:26:35.978588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-04T09:31:21.663666+0000","last_change":"2024-03-04T09:31:21.645102+0000","last_active":"2024-03-04T09:31:21.663666+0000","last_peered":"2024-03-04T09:31:21.663666+0000","last_clean":"2024-03-04T09:31:21.663666+0000","last_became_active":"2024-03-04T09:31:21.644748+0000","last_became_peered":"2024-03-04T09:31:21.644748+0000","last_unstale":"2024-03-04T09:31:21.663666+0000","last_undegraded":"2024-03-04T09:31:21.663666+0000","last_fullsized":"2024-03-04T09:31:21.663666+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:11:17.789505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-04T09:31:21.643081+0000","last_change":"2024-03-04T09:31:20.651452+0000","last_active":"2024-03-04T09:31:21.643081+0000","last_peered":"2024-03-04T09:31:21.643081+0000","last_clean":"2024-03-04T09:31:21.643081+0000","last_became_active":"2024-03-04T09:31:20.651104+0000","last_became_peered":"2024-03-04T09:31:20.651104+0000","last_unstale":"2024-03-04T09:31:21.643081+0000","last_undegraded":"2024-03-04T09:31:21.643081+0000","last_fullsized":"2024-03-04T09:31:21.643081+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_clean_scrub_stamp":"2024-03-04T09:31:19.587017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:42:45.728646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-04T09:31:22.659224+0000","last_change":"2024-03-04T09:31:21.645173+0000","last_active":"2024-03-04T09:31:22.659224+0000","last_peered":"2024-03-04T09:31:22.659224+0000","last_clean":"2024-03-04T09:31:22.659224+0000","last_became_active":"2024-03-04T09:31:21.644788+0000","last_became_peered":"2024-03-04T09:31:21.644788+0000","last_unstale":"2024-03-04T09:31:22.659224+0000","last_undegraded":"2024-03-04T09:31:22.659224+0000","last_fullsized":"2024-03-04T09:31:22.659224+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:49:19.511478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-04T09:31:21.639439+0000","last_change":"2024-03-04T09:31:20.652811+0000","last_active":"2024-03-04T09:31:21.639439+0000","last_peered":"2024-03-04T09:31:21.639439+0000","last_clean":"2024-03-04T09:31:21.639439+0000","last_became_active":"2024-03-04T09:31:20.652516+0000","last_became_peered":"2024-03-04T09:31:20.652516+0000","last_unstale":"2024-03-04T09:31:21.639439+0000","last_undegraded":"2024-03-04T09:31:21.639439+0000","last_fullsized":"2024-03-04T09:31:21.639439+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_clean_scrub_stamp":"2024-03-04T09:31:19.587017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:12:47.612367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-04T09:31:22.659803+0000","last_change":"2024-03-04T09:31:21.648619+0000","last_active":"2024-03-04T09:31:22.659803+0000","last_peered":"2024-03-04T09:31:22.659803+0000","last_clean":"2024-03-04T09:31:22.659803+0000","last_became_active":"2024-03-04T09:31:21.648198+0000","last_became_peered":"2024-03-04T09:31:21.648198+0000","last_unstale":"2024-03-04T09:31:22.659803+0000","last_undegraded":"2024-03-04T09:31:22.659803+0000","last_fullsized":"2024-03-04T09:31:22.659803+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:54:05.625143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-04T09:31:21.641806+0000","last_change":"2024-03-04T09:31:20.646772+0000","last_active":"2024-03-04T09:31:21.641806+0000","last_peered":"2024-03-04T09:31:21.641806+0000","last_clean":"2024-03-04T09:31:21.641806+0000","last_became_active":"2024-03-04T09:31:20.646459+0000","last_became_peered":"2024-03-04T09:31:20.646459+0000","last_unstale":"2024-03-04T09:31:21.641806+0000","last_undegraded":"2024-03-04T09:31:21.641806+0000","last_fullsized":"2024-03-04T09:31:21.641806+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_clean_scrub_stamp":"2024-03-04T09:31:19.587017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:36:54.206546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-04T09:31:22.648070+0000","last_change":"2024-03-04T09:31:21.665335+0000","last_active":"2024-03-04T09:31:22.648070+0000","last_peered":"2024-03-04T09:31:22.648070+0000","last_clean":"2024-03-04T09:31:22.648070+0000","last_became_active":"2024-03-04T09:31:21.664400+0000","last_became_peered":"2024-03-04T09:31:21.664400+0000","last_unstale":"2024-03-04T09:31:22.648070+0000","last_undegraded":"2024-03-04T09:31:22.648070+0000","last_fullsized":"2024-03-04T09:31:22.648070+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:31:52.099825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-04T09:31:22.647325+0000","last_change":"2024-03-04T09:31:20.653051+0000","last_active":"2024-03-04T09:31:22.647325+0000","last_peered":"2024-03-04T09:31:22.647325+0000","last_clean":"2024-03-04T09:31:22.647325+0000","last_became_active":"2024-03-04T09:31:20.652751+0000","last_became_peered":"2024-03-04T09:31:20.652751+0000","last_unstale":"2024-03-04T09:31:22.647325+0000","last_undegraded":"2024-03-04T09:31:22.647325+0000","last_fullsized":"2024-03-04T09:31:22.647325+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_clean_scrub_stamp":"2024-03-04T09:31:19.587017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:39:48.758359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-04T09:31:22.647519+0000","last_change":"2024-03-04T09:31:21.649988+0000","last_active":"2024-03-04T09:31:22.647519+0000","last_peered":"2024-03-04T09:31:22.647519+0000","last_clean":"2024-03-04T09:31:22.647519+0000","last_became_active":"2024-03-04T09:31:21.649490+0000","last_became_peered":"2024-03-04T09:31:21.649490+0000","last_unstale":"2024-03-04T09:31:22.647519+0000","last_undegraded":"2024-03-04T09:31:22.647519+0000","last_fullsized":"2024-03-04T09:31:22.647519+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:48:52.528076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"108'3","reported_seq":15,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-04T09:31:21.677746+0000","last_change":"2024-03-04T09:31:20.650114+0000","last_active":"2024-03-04T09:31:21.677746+0000","last_peered":"2024-03-04T09:31:21.677746+0000","last_clean":"2024-03-04T09:31:21.677746+0000","last_became_active":"2024-03-04T09:31:20.646728+0000","last_became_peered":"2024-03-04T09:31:20.646728+0000","last_unstale":"2024-03-04T09:31:21.677746+0000","last_undegraded":"2024-03-04T09:31:21.677746+0000","last_fullsized":"2024-03-04T09:31:21.677746+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_clean_scrub_stamp":"2024-03-04T09:31:19.587017+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:56:45.010621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-04T09:31:22.659769+0000","last_change":"2024-03-04T09:31:21.648739+0000","last_active":"2024-03-04T09:31:22.659769+0000","last_peered":"2024-03-04T09:31:22.659769+0000","last_clean":"2024-03-04T09:31:22.659769+0000","last_became_active":"2024-03-04T09:31:21.647112+0000","last_became_peered":"2024-03-04T09:31:21.647112+0000","last_unstale":"2024-03-04T09:31:22.659769+0000","last_undegraded":"2024-03-04T09:31:22.659769+0000","last_fullsized":"2024-03-04T09:31:22.659769+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:50:17.143286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"108'6","reported_seq":18,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-04T09:31:21.680332+0000","last_change":"2024-03-04T09:31:20.652880+0000","last_active":"2024-03-04T09:31:21.680332+0000","last_peered":"2024-03-04T09:31:21.680332+0000","last_clean":"2024-03-04T09:31:21.680332+0000","last_became_active":"2024-03-04T09:31:20.652623+0000","last_became_peered":"2024-03-04T09:31:20.652623+0000","last_unstale":"2024-03-04T09:31:21.680332+0000","last_undegraded":"2024-03-04T09:31:21.680332+0000","last_fullsized":"2024-03-04T09:31:21.680332+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_clean_scrub_stamp":"2024-03-04T09:31:19.587017+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:04:35.028973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-04T09:31:22.647239+0000","last_change":"2024-03-04T09:31:21.653150+0000","last_active":"2024-03-04T09:31:22.647239+0000","last_peered":"2024-03-04T09:31:22.647239+0000","last_clean":"2024-03-04T09:31:22.647239+0000","last_became_active":"2024-03-04T09:31:21.652836+0000","last_became_peered":"2024-03-04T09:31:21.652836+0000","last_unstale":"2024-03-04T09:31:22.647239+0000","last_undegraded":"2024-03-04T09:31:22.647239+0000","last_fullsized":"2024-03-04T09:31:22.647239+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:19:31.048726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"108'2","reported_seq":16,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-04T09:31:22.646070+0000","last_change":"2024-03-04T09:31:20.649845+0000","last_active":"2024-03-04T09:31:22.646070+0000","last_peered":"2024-03-04T09:31:22.646070+0000","last_clean":"2024-03-04T09:31:22.646070+0000","last_became_active":"2024-03-04T09:31:20.649546+0000","last_became_peered":"2024-03-04T09:31:20.649546+0000","last_unstale":"2024-03-04T09:31:22.646070+0000","last_undegraded":"2024-03-04T09:31:22.646070+0000","last_fullsized":"2024-03-04T09:31:22.646070+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_clean_scrub_stamp":"2024-03-04T09:31:19.587017+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:39:54.323425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-04T09:31:22.647476+0000","last_change":"2024-03-04T09:31:21.648524+0000","last_active":"2024-03-04T09:31:22.647476+0000","last_peered":"2024-03-04T09:31:22.647476+0000","last_clean":"2024-03-04T09:31:22.647476+0000","last_became_active":"2024-03-04T09:31:21.648009+0000","last_became_peered":"2024-03-04T09:31:21.648009+0000","last_unstale":"2024-03-04T09:31:22.647476+0000","last_undegraded":"2024-03-04T09:31:22.647476+0000","last_fullsized":"2024-03-04T09:31:22.647476+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:12:48.629727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-04T09:31:22.647416+0000","last_change":"2024-03-04T09:31:20.658305+0000","last_active":"2024-03-04T09:31:22.647416+0000","last_peered":"2024-03-04T09:31:22.647416+0000","last_clean":"2024-03-04T09:31:22.647416+0000","last_became_active":"2024-03-04T09:31:20.657345+0000","last_became_peered":"2024-03-04T09:31:20.657345+0000","last_unstale":"2024-03-04T09:31:22.647416+0000","last_undegraded":"2024-03-04T09:31:22.647416+0000","last_fullsized":"2024-03-04T09:31:22.647416+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_clean_scrub_stamp":"2024-03-04T09:31:19.587017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:20:04.739890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-04T09:31:22.646864+0000","last_change":"2024-03-04T09:31:21.645648+0000","last_active":"2024-03-04T09:31:22.646864+0000","last_peered":"2024-03-04T09:31:22.646864+0000","last_clean":"2024-03-04T09:31:22.646864+0000","last_became_active":"2024-03-04T09:31:21.645354+0000","last_became_peered":"2024-03-04T09:31:21.645354+0000","last_unstale":"2024-03-04T09:31:22.646864+0000","last_undegraded":"2024-03-04T09:31:22.646864+0000","last_fullsized":"2024-03-04T09:31:22.646864+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:56:46.027981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"108'8","reported_seq":20,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-04T09:31:21.673338+0000","last_change":"2024-03-04T09:31:20.645637+0000","last_active":"2024-03-04T09:31:21.673338+0000","last_peered":"2024-03-04T09:31:21.673338+0000","last_clean":"2024-03-04T09:31:21.673338+0000","last_became_active":"2024-03-04T09:31:20.644362+0000","last_became_peered":"2024-03-04T09:31:20.644362+0000","last_unstale":"2024-03-04T09:31:21.673338+0000","last_undegraded":"2024-03-04T09:31:21.673338+0000","last_fullsized":"2024-03-04T09:31:21.673338+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_clean_scrub_stamp":"2024-03-04T09:31:19.587017+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:37:56.045623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-04T09:31:21.671465+0000","last_change":"2024-03-04T09:31:21.670263+0000","last_active":"2024-03-04T09:31:21.671465+0000","last_peered":"2024-03-04T09:31:21.671465+0000","last_clean":"2024-03-04T09:31:21.671465+0000","last_became_active":"2024-03-04T09:31:21.669724+0000","last_became_peered":"2024-03-04T09:31:21.669724+0000","last_unstale":"2024-03-04T09:31:21.671465+0000","last_undegraded":"2024-03-04T09:31:21.671465+0000","last_fullsized":"2024-03-04T09:31:21.671465+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:48:42.697121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"108'1","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-04T09:31:21.676644+0000","last_change":"2024-03-04T09:31:20.654522+0000","last_active":"2024-03-04T09:31:21.676644+0000","last_peered":"2024-03-04T09:31:21.676644+0000","last_clean":"2024-03-04T09:31:21.676644+0000","last_became_active":"2024-03-04T09:31:20.654074+0000","last_became_peered":"2024-03-04T09:31:20.654074+0000","last_unstale":"2024-03-04T09:31:21.676644+0000","last_undegraded":"2024-03-04T09:31:21.676644+0000","last_fullsized":"2024-03-04T09:31:21.676644+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_clean_scrub_stamp":"2024-03-04T09:31:19.587017+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:39:10.893276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-04T09:31:22.646406+0000","last_change":"2024-03-04T09:31:21.669715+0000","last_active":"2024-03-04T09:31:22.646406+0000","last_peered":"2024-03-04T09:31:22.646406+0000","last_clean":"2024-03-04T09:31:22.646406+0000","last_became_active":"2024-03-04T09:31:21.667192+0000","last_became_peered":"2024-03-04T09:31:21.667192+0000","last_unstale":"2024-03-04T09:31:22.646406+0000","last_undegraded":"2024-03-04T09:31:22.646406+0000","last_fullsized":"2024-03-04T09:31:22.646406+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:31:24.059038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"108'2","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-04T09:31:21.670428+0000","last_change":"2024-03-04T09:31:20.645260+0000","last_active":"2024-03-04T09:31:21.670428+0000","last_peered":"2024-03-04T09:31:21.670428+0000","last_clean":"2024-03-04T09:31:21.670428+0000","last_became_active":"2024-03-04T09:31:20.643856+0000","last_became_peered":"2024-03-04T09:31:20.643856+0000","last_unstale":"2024-03-04T09:31:21.670428+0000","last_undegraded":"2024-03-04T09:31:21.670428+0000","last_fullsized":"2024-03-04T09:31:21.670428+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_clean_scrub_stamp":"2024-03-04T09:31:19.587017+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:06:33.545112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-04T09:31:21.669337+0000","last_change":"2024-03-04T09:31:21.668837+0000","last_active":"2024-03-04T09:31:21.669337+0000","last_peered":"2024-03-04T09:31:21.669337+0000","last_clean":"2024-03-04T09:31:21.669337+0000","last_became_active":"2024-03-04T09:31:21.668565+0000","last_became_peered":"2024-03-04T09:31:21.668565+0000","last_unstale":"2024-03-04T09:31:21.669337+0000","last_undegraded":"2024-03-04T09:31:21.669337+0000","last_fullsized":"2024-03-04T09:31:21.669337+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:26:36.995948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"108'1","reported_seq":15,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-04T09:31:22.647874+0000","last_change":"2024-03-04T09:31:20.651140+0000","last_active":"2024-03-04T09:31:22.647874+0000","last_peered":"2024-03-04T09:31:22.647874+0000","last_clean":"2024-03-04T09:31:22.647874+0000","last_became_active":"2024-03-04T09:31:20.650838+0000","last_became_peered":"2024-03-04T09:31:20.650838+0000","last_unstale":"2024-03-04T09:31:22.647874+0000","last_undegraded":"2024-03-04T09:31:22.647874+0000","last_fullsized":"2024-03-04T09:31:22.647874+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_clean_scrub_stamp":"2024-03-04T09:31:19.587017+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:00:41.066368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-04T09:31:21.661075+0000","last_change":"2024-03-04T09:31:21.657802+0000","last_active":"2024-03-04T09:31:21.661075+0000","last_peered":"2024-03-04T09:31:21.661075+0000","last_clean":"2024-03-04T09:31:21.661075+0000","last_became_active":"2024-03-04T09:31:21.657200+0000","last_became_peered":"2024-03-04T09:31:21.657200+0000","last_unstale":"2024-03-04T09:31:21.661075+0000","last_undegraded":"2024-03-04T09:31:21.661075+0000","last_fullsized":"2024-03-04T09:31:21.661075+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:08:19.417156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-04T09:31:22.659193+0000","last_change":"2024-03-04T09:31:20.656233+0000","last_active":"2024-03-04T09:31:22.659193+0000","last_peered":"2024-03-04T09:31:22.659193+0000","last_clean":"2024-03-04T09:31:22.659193+0000","last_became_active":"2024-03-04T09:31:20.655917+0000","last_became_peered":"2024-03-04T09:31:20.655917+0000","last_unstale":"2024-03-04T09:31:22.659193+0000","last_undegraded":"2024-03-04T09:31:22.659193+0000","last_fullsized":"2024-03-04T09:31:22.659193+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_clean_scrub_stamp":"2024-03-04T09:31:19.587017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:59:07.044800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-04T09:31:21.666033+0000","last_change":"2024-03-04T09:31:21.663592+0000","last_active":"2024-03-04T09:31:21.666033+0000","last_peered":"2024-03-04T09:31:21.666033+0000","last_clean":"2024-03-04T09:31:21.666033+0000","last_became_active":"2024-03-04T09:31:21.663318+0000","last_became_peered":"2024-03-04T09:31:21.663318+0000","last_unstale":"2024-03-04T09:31:21.666033+0000","last_undegraded":"2024-03-04T09:31:21.666033+0000","last_fullsized":"2024-03-04T09:31:21.666033+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:39:49.775719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"108'2","reported_seq":16,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-04T09:31:22.646184+0000","last_change":"2024-03-04T09:31:20.635738+0000","last_active":"2024-03-04T09:31:22.646184+0000","last_peered":"2024-03-04T09:31:22.646184+0000","last_clean":"2024-03-04T09:31:22.646184+0000","last_became_active":"2024-03-04T09:31:20.635439+0000","last_became_peered":"2024-03-04T09:31:20.635439+0000","last_unstale":"2024-03-04T09:31:22.646184+0000","last_undegraded":"2024-03-04T09:31:22.646184+0000","last_fullsized":"2024-03-04T09:31:22.646184+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_clean_scrub_stamp":"2024-03-04T09:31:19.587017+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:54:52.269666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-04T09:31:21.664256+0000","last_change":"2024-03-04T09:31:21.659596+0000","last_active":"2024-03-04T09:31:21.664256+0000","last_peered":"2024-03-04T09:31:21.664256+0000","last_clean":"2024-03-04T09:31:21.664256+0000","last_became_active":"2024-03-04T09:31:21.659278+0000","last_became_peered":"2024-03-04T09:31:21.659278+0000","last_unstale":"2024-03-04T09:31:21.664256+0000","last_undegraded":"2024-03-04T09:31:21.664256+0000","last_fullsized":"2024-03-04T09:31:21.664256+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:05:31.021016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-04T09:31:21.638891+0000","last_change":"2024-03-04T09:31:20.641401+0000","last_active":"2024-03-04T09:31:21.638891+0000","last_peered":"2024-03-04T09:31:21.638891+0000","last_clean":"2024-03-04T09:31:21.638891+0000","last_became_active":"2024-03-04T09:31:20.640212+0000","last_became_peered":"2024-03-04T09:31:20.640212+0000","last_unstale":"2024-03-04T09:31:21.638891+0000","last_undegraded":"2024-03-04T09:31:21.638891+0000","last_fullsized":"2024-03-04T09:31:21.638891+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_clean_scrub_stamp":"2024-03-04T09:31:19.587017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:52:17.490227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-04T09:31:21.660986+0000","last_change":"2024-03-04T09:31:21.657200+0000","last_active":"2024-03-04T09:31:21.660986+0000","last_peered":"2024-03-04T09:31:21.660986+0000","last_clean":"2024-03-04T09:31:21.660986+0000","last_became_active":"2024-03-04T09:31:21.656756+0000","last_became_peered":"2024-03-04T09:31:21.656756+0000","last_unstale":"2024-03-04T09:31:21.660986+0000","last_undegraded":"2024-03-04T09:31:21.660986+0000","last_fullsized":"2024-03-04T09:31:21.660986+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:27:14.606466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-04T09:31:22.645620+0000","last_change":"2024-03-04T09:31:20.637485+0000","last_active":"2024-03-04T09:31:22.645620+0000","last_peered":"2024-03-04T09:31:22.645620+0000","last_clean":"2024-03-04T09:31:22.645620+0000","last_became_active":"2024-03-04T09:31:20.636835+0000","last_became_peered":"2024-03-04T09:31:20.636835+0000","last_unstale":"2024-03-04T09:31:22.645620+0000","last_undegraded":"2024-03-04T09:31:22.645620+0000","last_fullsized":"2024-03-04T09:31:22.645620+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_clean_scrub_stamp":"2024-03-04T09:31:19.587017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:05:50.693261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-04T09:31:22.647072+0000","last_change":"2024-03-04T09:31:21.641908+0000","last_active":"2024-03-04T09:31:22.647072+0000","last_peered":"2024-03-04T09:31:22.647072+0000","last_clean":"2024-03-04T09:31:22.647072+0000","last_became_active":"2024-03-04T09:31:21.641561+0000","last_became_peered":"2024-03-04T09:31:21.641561+0000","last_unstale":"2024-03-04T09:31:22.647072+0000","last_undegraded":"2024-03-04T09:31:22.647072+0000","last_fullsized":"2024-03-04T09:31:22.647072+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:04:36.046333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-04T09:31:21.638133+0000","last_change":"2024-03-04T09:31:20.657722+0000","last_active":"2024-03-04T09:31:21.638133+0000","last_peered":"2024-03-04T09:31:21.638133+0000","last_clean":"2024-03-04T09:31:21.638133+0000","last_became_active":"2024-03-04T09:31:20.657383+0000","last_became_peered":"2024-03-04T09:31:20.657383+0000","last_unstale":"2024-03-04T09:31:21.638133+0000","last_undegraded":"2024-03-04T09:31:21.638133+0000","last_fullsized":"2024-03-04T09:31:21.638133+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_clean_scrub_stamp":"2024-03-04T09:31:19.587017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:01:21.470739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-04T09:31:22.647725+0000","last_change":"2024-03-04T09:31:21.660695+0000","last_active":"2024-03-04T09:31:22.647725+0000","last_peered":"2024-03-04T09:31:22.647725+0000","last_clean":"2024-03-04T09:31:22.647725+0000","last_became_active":"2024-03-04T09:31:21.660385+0000","last_became_peered":"2024-03-04T09:31:21.660385+0000","last_unstale":"2024-03-04T09:31:22.647725+0000","last_undegraded":"2024-03-04T09:31:22.647725+0000","last_fullsized":"2024-03-04T09:31:22.647725+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:08:19.417156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"108'2","reported_seq":16,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-04T09:31:22.648148+0000","last_change":"2024-03-04T09:31:20.654509+0000","last_active":"2024-03-04T09:31:22.648148+0000","last_peered":"2024-03-04T09:31:22.648148+0000","last_clean":"2024-03-04T09:31:22.648148+0000","last_became_active":"2024-03-04T09:31:20.654023+0000","last_became_peered":"2024-03-04T09:31:20.654023+0000","last_unstale":"2024-03-04T09:31:22.648148+0000","last_undegraded":"2024-03-04T09:31:22.648148+0000","last_fullsized":"2024-03-04T09:31:22.648148+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_clean_scrub_stamp":"2024-03-04T09:31:19.587017+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:31:51.082464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-04T09:31:21.663743+0000","last_change":"2024-03-04T09:31:21.646610+0000","last_active":"2024-03-04T09:31:21.663743+0000","last_peered":"2024-03-04T09:31:21.663743+0000","last_clean":"2024-03-04T09:31:21.663743+0000","last_became_active":"2024-03-04T09:31:21.646322+0000","last_became_peered":"2024-03-04T09:31:21.646322+0000","last_unstale":"2024-03-04T09:31:21.663743+0000","last_undegraded":"2024-03-04T09:31:21.663743+0000","last_fullsized":"2024-03-04T09:31:21.663743+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:20:07.013843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"109'1","reported_seq":15,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-04T09:31:22.657460+0000","last_change":"2024-03-04T09:31:20.656170+0000","last_active":"2024-03-04T09:31:22.657460+0000","last_peered":"2024-03-04T09:31:22.657460+0000","last_clean":"2024-03-04T09:31:22.657460+0000","last_became_active":"2024-03-04T09:31:20.654568+0000","last_became_peered":"2024-03-04T09:31:20.654568+0000","last_unstale":"2024-03-04T09:31:22.657460+0000","last_undegraded":"2024-03-04T09:31:22.657460+0000","last_fullsized":"2024-03-04T09:31:22.657460+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_clean_scrub_stamp":"2024-03-04T09:31:19.587017+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:42:45.728646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-04T09:31:21.663237+0000","last_change":"2024-03-04T09:31:21.642718+0000","last_active":"2024-03-04T09:31:21.663237+0000","last_peered":"2024-03-04T09:31:21.663237+0000","last_clean":"2024-03-04T09:31:21.663237+0000","last_became_active":"2024-03-04T09:31:21.642416+0000","last_became_peered":"2024-03-04T09:31:21.642416+0000","last_unstale":"2024-03-04T09:31:21.663237+0000","last_undegraded":"2024-03-04T09:31:21.663237+0000","last_fullsized":"2024-03-04T09:31:21.663237+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:48:52.528076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-04T09:31:21.639308+0000","last_change":"2024-03-04T09:31:20.658189+0000","last_active":"2024-03-04T09:31:21.639308+0000","last_peered":"2024-03-04T09:31:21.639308+0000","last_clean":"2024-03-04T09:31:21.639308+0000","last_became_active":"2024-03-04T09:31:20.657860+0000","last_became_peered":"2024-03-04T09:31:20.657860+0000","last_unstale":"2024-03-04T09:31:21.639308+0000","last_undegraded":"2024-03-04T09:31:21.639308+0000","last_fullsized":"2024-03-04T09:31:21.639308+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_clean_scrub_stamp":"2024-03-04T09:31:19.587017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:53:20.181825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-04T09:31:22.647909+0000","last_change":"2024-03-04T09:31:21.647493+0000","last_active":"2024-03-04T09:31:22.647909+0000","last_peered":"2024-03-04T09:31:22.647909+0000","last_clean":"2024-03-04T09:31:22.647909+0000","last_became_active":"2024-03-04T09:31:21.647103+0000","last_became_peered":"2024-03-04T09:31:21.647103+0000","last_unstale":"2024-03-04T09:31:22.647909+0000","last_undegraded":"2024-03-04T09:31:22.647909+0000","last_fullsized":"2024-03-04T09:31:22.647909+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:31:15.039332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-04T09:31:22.647647+0000","last_change":"2024-03-04T09:31:20.648528+0000","last_active":"2024-03-04T09:31:22.647647+0000","last_peered":"2024-03-04T09:31:22.647647+0000","last_clean":"2024-03-04T09:31:22.647647+0000","last_became_active":"2024-03-04T09:31:20.648223+0000","last_became_peered":"2024-03-04T09:31:20.648223+0000","last_unstale":"2024-03-04T09:31:22.647647+0000","last_undegraded":"2024-03-04T09:31:22.647647+0000","last_fullsized":"2024-03-04T09:31:22.647647+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_clean_scrub_stamp":"2024-03-04T09:31:19.587017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:01:34.024686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-04T09:31:21.658370+0000","last_change":"2024-03-04T09:31:21.651580+0000","last_active":"2024-03-04T09:31:21.658370+0000","last_peered":"2024-03-04T09:31:21.658370+0000","last_clean":"2024-03-04T09:31:21.658370+0000","last_became_active":"2024-03-04T09:31:21.649396+0000","last_became_peered":"2024-03-04T09:31:21.649396+0000","last_unstale":"2024-03-04T09:31:21.658370+0000","last_undegraded":"2024-03-04T09:31:21.658370+0000","last_fullsized":"2024-03-04T09:31:21.658370+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:27:14.606466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-04T09:31:22.659126+0000","last_change":"2024-03-04T09:31:20.656370+0000","last_active":"2024-03-04T09:31:22.659126+0000","last_peered":"2024-03-04T09:31:22.659126+0000","last_clean":"2024-03-04T09:31:22.659126+0000","last_became_active":"2024-03-04T09:31:20.655442+0000","last_became_peered":"2024-03-04T09:31:20.655442+0000","last_unstale":"2024-03-04T09:31:22.659126+0000","last_undegraded":"2024-03-04T09:31:22.659126+0000","last_fullsized":"2024-03-04T09:31:22.659126+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_clean_scrub_stamp":"2024-03-04T09:31:19.587017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:41:12.969898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-04T09:31:21.655181+0000","last_change":"2024-03-04T09:31:21.647082+0000","last_active":"2024-03-04T09:31:21.655181+0000","last_peered":"2024-03-04T09:31:21.655181+0000","last_clean":"2024-03-04T09:31:21.655181+0000","last_became_active":"2024-03-04T09:31:21.646727+0000","last_became_peered":"2024-03-04T09:31:21.646727+0000","last_unstale":"2024-03-04T09:31:21.655181+0000","last_undegraded":"2024-03-04T09:31:21.655181+0000","last_fullsized":"2024-03-04T09:31:21.655181+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:49:17.194698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-04T09:31:21.642594+0000","last_change":"2024-03-04T09:31:20.650193+0000","last_active":"2024-03-04T09:31:21.642594+0000","last_peered":"2024-03-04T09:31:21.642594+0000","last_clean":"2024-03-04T09:31:21.642594+0000","last_became_active":"2024-03-04T09:31:20.649858+0000","last_became_peered":"2024-03-04T09:31:20.649858+0000","last_unstale":"2024-03-04T09:31:21.642594+0000","last_undegraded":"2024-03-04T09:31:21.642594+0000","last_fullsized":"2024-03-04T09:31:21.642594+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_clean_scrub_stamp":"2024-03-04T09:31:19.587017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:54:04.607782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-04T09:31:21.660691+0000","last_change":"2024-03-04T09:31:21.656806+0000","last_active":"2024-03-04T09:31:21.660691+0000","last_peered":"2024-03-04T09:31:21.660691+0000","last_clean":"2024-03-04T09:31:21.660691+0000","last_became_active":"2024-03-04T09:31:21.656528+0000","last_became_peered":"2024-03-04T09:31:21.656528+0000","last_unstale":"2024-03-04T09:31:21.660691+0000","last_undegraded":"2024-03-04T09:31:21.660691+0000","last_fullsized":"2024-03-04T09:31:21.660691+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:13:52.946588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-04T09:31:21.637873+0000","last_change":"2024-03-04T09:31:20.641748+0000","last_active":"2024-03-04T09:31:21.637873+0000","last_peered":"2024-03-04T09:31:21.637873+0000","last_clean":"2024-03-04T09:31:21.637873+0000","last_became_active":"2024-03-04T09:31:20.641335+0000","last_became_peered":"2024-03-04T09:31:20.641335+0000","last_unstale":"2024-03-04T09:31:21.637873+0000","last_undegraded":"2024-03-04T09:31:21.637873+0000","last_fullsized":"2024-03-04T09:31:21.637873+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_clean_scrub_stamp":"2024-03-04T09:31:19.587017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:00:31.024486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-04T09:31:21.664051+0000","last_change":"2024-03-04T09:31:21.655517+0000","last_active":"2024-03-04T09:31:21.664051+0000","last_peered":"2024-03-04T09:31:21.664051+0000","last_clean":"2024-03-04T09:31:21.664051+0000","last_became_active":"2024-03-04T09:31:21.650939+0000","last_became_peered":"2024-03-04T09:31:21.650939+0000","last_unstale":"2024-03-04T09:31:21.664051+0000","last_undegraded":"2024-03-04T09:31:21.664051+0000","last_fullsized":"2024-03-04T09:31:21.664051+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:35:05.912704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-04T09:31:21.638421+0000","last_change":"2024-03-04T09:31:20.657679+0000","last_active":"2024-03-04T09:31:21.638421+0000","last_peered":"2024-03-04T09:31:21.638421+0000","last_clean":"2024-03-04T09:31:21.638421+0000","last_became_active":"2024-03-04T09:31:20.657333+0000","last_became_peered":"2024-03-04T09:31:20.657333+0000","last_unstale":"2024-03-04T09:31:21.638421+0000","last_undegraded":"2024-03-04T09:31:21.638421+0000","last_fullsized":"2024-03-04T09:31:21.638421+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_clean_scrub_stamp":"2024-03-04T09:31:19.587017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:19:30.031366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-04T09:31:22.646703+0000","last_change":"2024-03-04T09:31:21.655803+0000","last_active":"2024-03-04T09:31:22.646703+0000","last_peered":"2024-03-04T09:31:22.646703+0000","last_clean":"2024-03-04T09:31:22.646703+0000","last_became_active":"2024-03-04T09:31:21.651394+0000","last_became_peered":"2024-03-04T09:31:21.651394+0000","last_unstale":"2024-03-04T09:31:22.646703+0000","last_undegraded":"2024-03-04T09:31:22.646703+0000","last_fullsized":"2024-03-04T09:31:22.646703+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:20.604377+0000","last_clean_scrub_stamp":"2024-03-04T09:31:20.604377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:42:25.949190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-04T09:31:22.659559+0000","last_change":"2024-03-04T09:31:20.657927+0000","last_active":"2024-03-04T09:31:22.659559+0000","last_peered":"2024-03-04T09:31:22.659559+0000","last_clean":"2024-03-04T09:31:22.659559+0000","last_became_active":"2024-03-04T09:31:20.656107+0000","last_became_peered":"2024-03-04T09:31:20.656107+0000","last_unstale":"2024-03-04T09:31:22.659559+0000","last_undegraded":"2024-03-04T09:31:22.659559+0000","last_fullsized":"2024-03-04T09:31:22.659559+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:31:19.587017+0000","last_clean_scrub_stamp":"2024-03-04T09:31:19.587017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:07:54.338428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":333,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-04T09:31:21.648129+0000","last_change":"2024-03-04T09:25:09.316159+0000","last_active":"2024-03-04T09:31:21.648129+0000","last_peered":"2024-03-04T09:31:21.648129+0000","last_clean":"2024-03-04T09:31:21.648129+0000","last_became_active":"2024-03-04T09:25:09.313767+0000","last_became_peered":"2024-03-04T09:25:09.313767+0000","last_unstale":"2024-03-04T09:31:21.648129+0000","last_undegraded":"2024-03-04T09:31:21.648129+0000","last_fullsized":"2024-03-04T09:31:21.648129+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:08.110953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:08.110953+0000","last_clean_scrub_stamp":"2024-03-04T09:25:08.110953+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:34:21.745390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574926,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37240,"kb_used_data":1900,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334600,"statfs":{"total":96636764160,"available":96598630400,"internally_reserved":0,"allocated":1945600,"data_stored":1170804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574926,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36672,"kb_used_data":1328,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335168,"statfs":{"total":96636764160,"available":96599212032,"internally_reserved":0,"allocated":1359872,"data_stored":586190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574928,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36660,"kb_used_data":1320,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335180,"statfs":{"total":96636764160,"available":96599224320,"internally_reserved":0,"allocated":1351680,"data_stored":580424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574925,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36660,"kb_used_data":1312,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335180,"statfs":{"total":96636764160,"available":96599224320,"internally_reserved":0,"allocated":1343488,"data_stored":579820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574927,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36668,"kb_used_data":1328,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335172,"statfs":{"total":96636764160,"available":96599216128,"internally_reserved":0,"allocated":1359872,"data_stored":581502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574926,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36656,"kb_used_data":1312,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335184,"statfs":{"total":96636764160,"available":96599228416,"internally_reserved":0,"allocated":1343488,"data_stored":579820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574927,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36680,"kb_used_data":1332,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335160,"statfs":{"total":96636764160,"available":96599203840,"internally_reserved":0,"allocated":1363968,"data_stored":586244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574926,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37260,"kb_used_data":1912,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334580,"statfs":{"total":96636764160,"available":96598609920,"internally_reserved":0,"allocated":1957888,"data_stored":1171912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-04T09:31:26.662 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-04T09:31:26.662 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-04T09:31:26.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.927+0000 7f3d8e400640 1 Processor -- start 2024-03-04T09:31:26.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.927+0000 7f3d8e400640 1 -- start start 2024-03-04T09:31:26.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.928+0000 7f3d8e400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f3d8815e220 0x7f3d880d4870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:26.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.929+0000 7f3d8e400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3d880d1920 0x7f3d880d1fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:26.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.929+0000 7f3d8e400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f3d880d2520 0x7f3d880d0720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:26.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.929+0000 7f3d8e400640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d880b36b0 con 0x7f3d8815e220 2024-03-04T09:31:26.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.929+0000 7f3d8e400640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d880cf4a0 con 0x7f3d880d2520 2024-03-04T09:31:26.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.929+0000 7f3d8e400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d880cf780 con 0x7f3d880d1920 2024-03-04T09:31:26.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.929+0000 7f3d8d000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f3d8815e220 0x7f3d880d4870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:26.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.929+0000 7f3d8d000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f3d8815e220 0x7f3d880d4870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:32816/0 (socket says 172.21.15.6:32816) 2024-03-04T09:31:26.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.929+0000 7f3d8d000640 1 -- 172.21.15.6:0/4075933362 learned_addr learned my addr 172.21.15.6:0/4075933362 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:31:26.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.929+0000 7f3d8da00640 1 --2- 172.21.15.6:0/4075933362 >> v2:172.21.15.146:3300/0 conn(0x7f3d880d2520 0x7f3d880d0720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:26.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.930+0000 7f3d87e00640 1 --2- 172.21.15.6:0/4075933362 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3d880d1920 0x7f3d880d1fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:26.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.930+0000 7f3d8d000640 1 -- 172.21.15.6:0/4075933362 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3d880d1920 msgr2=0x7f3d880d1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:26.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.930+0000 7f3d8d000640 1 --2- 172.21.15.6:0/4075933362 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3d880d1920 0x7f3d880d1fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:26.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.930+0000 7f3d8d000640 1 -- 172.21.15.6:0/4075933362 >> v2:172.21.15.146:3300/0 conn(0x7f3d880d2520 msgr2=0x7f3d880d0720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:26.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.930+0000 7f3d8d000640 1 --2- 172.21.15.6:0/4075933362 >> v2:172.21.15.146:3300/0 conn(0x7f3d880d2520 0x7f3d880d0720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:26.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.930+0000 7f3d8d000640 1 -- 172.21.15.6:0/4075933362 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d880cfa60 con 0x7f3d8815e220 2024-03-04T09:31:26.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.932+0000 7f3d8d000640 1 --2- 172.21.15.6:0/4075933362 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3d8815e220 0x7f3d880d4870 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7f3d8806b8d0 tx=0x7f3d7c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8716687c83f43f99 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:26.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.932+0000 7f3d87400640 1 -- 172.21.15.6:0/4075933362 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d7c02e070 con 0x7f3d8815e220 2024-03-04T09:31:26.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.932+0000 7f3d87400640 1 -- 172.21.15.6:0/4075933362 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d7c037070 con 0x7f3d8815e220 2024-03-04T09:31:26.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.932+0000 7f3d87400640 1 -- 172.21.15.6:0/4075933362 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d7c031a00 con 0x7f3d8815e220 2024-03-04T09:31:26.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.933+0000 7f3d8e400640 1 -- 172.21.15.6:0/4075933362 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3d8815e220 msgr2=0x7f3d880d4870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:26.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.933+0000 7f3d8e400640 1 --2- 172.21.15.6:0/4075933362 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3d8815e220 0x7f3d880d4870 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7f3d8806b8d0 tx=0x7f3d7c02b880 comp rx=0 tx=0).stop 2024-03-04T09:31:26.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.933+0000 7f3d8e400640 1 -- 172.21.15.6:0/4075933362 shutdown_connections 2024-03-04T09:31:26.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.933+0000 7f3d8e400640 1 --2- 172.21.15.6:0/4075933362 >> v2:172.21.15.146:3300/0 conn(0x7f3d880d2520 0x7f3d880d0720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:26.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.933+0000 7f3d8e400640 1 --2- 172.21.15.6:0/4075933362 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3d880d1920 0x7f3d880d1fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:26.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.933+0000 7f3d8e400640 1 --2- 172.21.15.6:0/4075933362 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3d8815e220 0x7f3d880d4870 unknown :-1 s=CLOSED pgs=1256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:26.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.933+0000 7f3d8e400640 1 -- 172.21.15.6:0/4075933362 >> 172.21.15.6:0/4075933362 conn(0x7f3d880d3d50 msgr2=0x7f3d880d4160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:26.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.933+0000 7f3d8e400640 1 -- 172.21.15.6:0/4075933362 shutdown_connections 2024-03-04T09:31:26.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.933+0000 7f3d8e400640 1 -- 172.21.15.6:0/4075933362 wait complete. 2024-03-04T09:31:26.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.935+0000 7f3d8e400640 1 Processor -- start 2024-03-04T09:31:26.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.935+0000 7f3d8e400640 1 -- start start 2024-03-04T09:31:26.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.936+0000 7f3d8e400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3d880d1920 0x7f3d880ca380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:26.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.936+0000 7f3d8e400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3d880d2520 0x7f3d88154db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:26.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.936+0000 7f3d8e400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3d8815e220 0x7f3d88160dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:26.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.936+0000 7f3d8e400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d880d0c60 con 0x7f3d880d1920 2024-03-04T09:31:26.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.936+0000 7f3d8e400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d880d1d80 con 0x7f3d880d2520 2024-03-04T09:31:26.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.936+0000 7f3d8e400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d880d0de0 con 0x7f3d8815e220 2024-03-04T09:31:26.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.936+0000 7f3d8d000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3d880d1920 0x7f3d880ca380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:26.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.936+0000 7f3d8d000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3d880d1920 0x7f3d880ca380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:32828/0 (socket says 172.21.15.6:32828) 2024-03-04T09:31:26.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.936+0000 7f3d8d000640 1 -- 172.21.15.6:0/4139997273 learned_addr learned my addr 172.21.15.6:0/4139997273 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:31:26.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.937+0000 7f3d87e00640 1 --2- 172.21.15.6:0/4139997273 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3d880d2520 0x7f3d88154db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:26.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.937+0000 7f3d8d000640 1 -- 172.21.15.6:0/4139997273 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3d8815e220 msgr2=0x7f3d88160dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:26.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.937+0000 7f3d8d000640 1 --2- 172.21.15.6:0/4139997273 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3d8815e220 0x7f3d88160dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:26.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.937+0000 7f3d8d000640 1 -- 172.21.15.6:0/4139997273 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3d880d2520 msgr2=0x7f3d88154db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:26.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.937+0000 7f3d8d000640 1 --2- 172.21.15.6:0/4139997273 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3d880d2520 0x7f3d88154db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:26.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.937+0000 7f3d8d000640 1 -- 172.21.15.6:0/4139997273 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d880c9060 con 0x7f3d880d1920 2024-03-04T09:31:26.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.937+0000 7f3d8d000640 1 --2- 172.21.15.6:0/4139997273 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3d880d1920 0x7f3d880ca380 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7f3d7c000c00 tx=0x7f3d7c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:26.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.937+0000 7f3d84c00640 1 -- 172.21.15.6:0/4139997273 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d7c032670 con 0x7f3d880d1920 2024-03-04T09:31:26.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.937+0000 7f3d84c00640 1 -- 172.21.15.6:0/4139997273 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d7c032850 con 0x7f3d880d1920 2024-03-04T09:31:26.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.937+0000 7f3d84c00640 1 -- 172.21.15.6:0/4139997273 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d7c032c40 con 0x7f3d880d1920 2024-03-04T09:31:26.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.937+0000 7f3d8e400640 1 -- 172.21.15.6:0/4139997273 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d880c92e0 con 0x7f3d880d1920 2024-03-04T09:31:26.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.938+0000 7f3d8e400640 1 -- 172.21.15.6:0/4139997273 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d880c98a0 con 0x7f3d880d1920 2024-03-04T09:31:26.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.939+0000 7f3d8e400640 1 -- 172.21.15.6:0/4139997273 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d8804f330 con 0x7f3d880d1920 2024-03-04T09:31:26.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.940+0000 7f3d84c00640 1 -- 172.21.15.6:0/4139997273 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3d7c04c060 con 0x7f3d880d1920 2024-03-04T09:31:26.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.941+0000 7f3d84c00640 1 --2- 172.21.15.6:0/4139997273 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f3d6806b520 0x7f3d6806d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:26.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.941+0000 7f3d84c00640 1 -- 172.21.15.6:0/4139997273 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3d7c0abb30 con 0x7f3d880d1920 2024-03-04T09:31:26.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.941+0000 7f3d87e00640 1 --2- 172.21.15.6:0/4139997273 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f3d6806b520 0x7f3d6806d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:26.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.943+0000 7f3d87e00640 1 --2- 172.21.15.6:0/4139997273 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f3d6806b520 0x7f3d6806d9e0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f3d880c75f0 tx=0x7f3d74008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:26.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:26.945+0000 7f3d84c00640 1 -- 172.21.15.6:0/4139997273 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3d880c9060 con 0x7f3d880d1920 2024-03-04T09:31:27.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:27.217+0000 7f3d8e400640 1 -- 172.21.15.6:0/4139997273 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f3d88161630 con 0x7f3d880d1920 2024-03-04T09:31:27.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:27.222+0000 7f3d84c00640 1 -- 172.21.15.6:0/4139997273 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v16) v1 ==== 190+0+147 (secure 0 0 0) 0x7f3d7c040070 con 0x7f3d880d1920 2024-03-04T09:31:27.219 INFO:teuthology.orchestra.run.smithi006.stdout:[client.0] 2024-03-04T09:31:27.219 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQA9k+Vl4L+rJxAACaDbQ4/Bybp5hAZAnJzlVw== 2024-03-04T09:31:27.219 INFO:teuthology.orchestra.run.smithi006.stdout: caps mds = "allow" 2024-03-04T09:31:27.219 INFO:teuthology.orchestra.run.smithi006.stdout: caps mon = "allow r" 2024-03-04T09:31:27.219 INFO:teuthology.orchestra.run.smithi006.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-04T09:31:27.219 INFO:teuthology.orchestra.run.smithi006.stderr:updated caps for client.0 2024-03-04T09:31:27.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:27.224+0000 7f3d8e400640 1 -- 172.21.15.6:0/4139997273 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f3d6806b520 msgr2=0x7f3d6806d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:27.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:27.224+0000 7f3d8e400640 1 --2- 172.21.15.6:0/4139997273 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f3d6806b520 0x7f3d6806d9e0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f3d880c75f0 tx=0x7f3d74008040 comp rx=0 tx=0).stop 2024-03-04T09:31:27.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:27.224+0000 7f3d8e400640 1 -- 172.21.15.6:0/4139997273 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3d880d1920 msgr2=0x7f3d880ca380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:27.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:27.224+0000 7f3d8e400640 1 --2- 172.21.15.6:0/4139997273 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3d880d1920 0x7f3d880ca380 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7f3d7c000c00 tx=0x7f3d7c0046c0 comp rx=0 tx=0).stop 2024-03-04T09:31:27.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:27.224+0000 7f3d8e400640 1 -- 172.21.15.6:0/4139997273 shutdown_connections 2024-03-04T09:31:27.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:27.224+0000 7f3d8e400640 1 --2- 172.21.15.6:0/4139997273 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f3d6806b520 0x7f3d6806d9e0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:27.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:27.224+0000 7f3d8e400640 1 --2- 172.21.15.6:0/4139997273 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3d8815e220 0x7f3d88160dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:27.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:27.224+0000 7f3d8e400640 1 --2- 172.21.15.6:0/4139997273 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3d880d2520 0x7f3d88154db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:27.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:27.224+0000 7f3d8e400640 1 --2- 172.21.15.6:0/4139997273 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3d880d1920 0x7f3d880ca380 unknown :-1 s=CLOSED pgs=1257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:27.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:27.224+0000 7f3d8e400640 1 -- 172.21.15.6:0/4139997273 >> 172.21.15.6:0/4139997273 conn(0x7f3d880d3d50 msgr2=0x7f3d880b23f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:27.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:27.224+0000 7f3d8e400640 1 -- 172.21.15.6:0/4139997273 shutdown_connections 2024-03-04T09:31:27.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:27.224+0000 7f3d8e400640 1 -- 172.21.15.6:0/4139997273 wait complete. 2024-03-04T09:31:27.241 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:31:27.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:27.457+0000 7fc83be00640 1 Processor -- start 2024-03-04T09:31:27.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:27.457+0000 7fc83be00640 1 -- start start 2024-03-04T09:31:27.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:27.459+0000 7fc83be00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fc83c0cd060 0x7fc83c1667e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:27.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:27.459+0000 7fc83be00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc83c0714d0 con 0x7fc83c070f10 2024-03-04T09:31:27.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:27.459+0000 7fc83be00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc83c0717b0 con 0x7fc83c145300 2024-03-04T09:31:27.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:27.459+0000 7fc83be00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc83c071a90 con 0x7fc83c0cd060 2024-03-04T09:31:27.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:27.459+0000 7fc83a000640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fc83c070f10 0x7fc83c0cc950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:56996/0 (socket says 172.21.15.6:56996) 2024-03-04T09:31:27.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:27.459+0000 7fc83a000640 1 -- 172.21.15.6:0/42639806 learned_addr learned my addr 172.21.15.6:0/42639806 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:31:27.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:27.460+0000 7fc83b400640 1 --2- 172.21.15.6:0/42639806 >> v2:172.21.15.146:3300/0 conn(0x7fc83c0cd060 0x7fc83c1667e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:27.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:27.460+0000 7fc839600640 1 -- 172.21.15.6:0/42639806 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1162590818 0 0) 0x7fc83c0714d0 con 0x7fc83c070f10 2024-03-04T09:31:27.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:27.460+0000 7fc839600640 1 -- 172.21.15.6:0/42639806 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc820003520 con 0x7fc83c070f10 2024-03-04T09:31:27.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:27.460+0000 7fc839600640 1 -- 172.21.15.6:0/42639806 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1647154290 0 0) 0x7fc820003520 con 0x7fc83c070f10 2024-03-04T09:31:27.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:27.460+0000 7fc839600640 1 -- 172.21.15.6:0/42639806 >> v2:172.21.15.146:3300/0 conn(0x7fc83c0cd060 msgr2=0x7fc83c1667e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:27.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:27.460+0000 7fc839600640 1 --2- 172.21.15.6:0/42639806 >> v2:172.21.15.146:3300/0 conn(0x7fc83c0cd060 0x7fc83c1667e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:27.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:27.460+0000 7fc839600640 1 -- 172.21.15.6:0/42639806 >> v1:172.21.15.146:6789/0 conn(0x7fc83c145300 legacy=0x7fc83c0b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:27.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:27.460+0000 7fc839600640 1 -- 172.21.15.6:0/42639806 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc83c071d70 con 0x7fc83c070f10 2024-03-04T09:31:27.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:27.460+0000 7fc839600640 1 -- 172.21.15.6:0/42639806 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7fc828001670 con 0x7fc83c070f10 2024-03-04T09:31:27.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:27.460+0000 7fc839600640 1 -- 172.21.15.6:0/42639806 >> v1:172.21.15.6:6789/0 conn(0x7fc83c070f10 legacy=0x7fc83c0cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:27.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:27.460+0000 7fc839600640 1 --2- 172.21.15.6:0/42639806 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc820003a00 0x7fc820003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:27.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:27.460+0000 7fc839600640 1 --2- 172.21.15.6:0/42639806 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc820004800 0x7fc820004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:27.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:27.460+0000 7fc839600640 1 --2- 172.21.15.6:0/42639806 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc820005210 0x7fc82000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:27.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:27.461+0000 7fc839600640 1 -- 172.21.15.6:0/42639806 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc83c071a90 con 0x7fc820003a00 2024-03-04T09:31:27.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:27.461+0000 7fc839600640 1 -- 172.21.15.6:0/42639806 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc82000c160 con 0x7fc820004800 2024-03-04T09:31:27.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:27.461+0000 7fc839600640 1 -- 172.21.15.6:0/42639806 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc82000c360 con 0x7fc820005210 2024-03-04T09:31:27.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:27.461+0000 7fc83aa00640 1 --2- 172.21.15.6:0/42639806 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc820003a00 0x7fc820003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:27.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:27.461+0000 7fc83a000640 1 --2- 172.21.15.6:0/42639806 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc820004800 0x7fc820004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:27.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:27.461+0000 7fc83aa00640 1 -- 172.21.15.6:0/42639806 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc820005210 msgr2=0x7fc82000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:27.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:27.461+0000 7fc83aa00640 1 --2- 172.21.15.6:0/42639806 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc820005210 0x7fc82000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:27.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:27.461+0000 7fc83aa00640 1 -- 172.21.15.6:0/42639806 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc820004800 msgr2=0x7fc820004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:27.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:27.461+0000 7fc83aa00640 1 --2- 172.21.15.6:0/42639806 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc820004800 0x7fc820004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:27.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:27.461+0000 7fc83aa00640 1 -- 172.21.15.6:0/42639806 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc820003520 con 0x7fc820003a00 2024-03-04T09:31:27.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:27.461+0000 7fc83b400640 1 -- 172.21.15.6:0/42639806 reap_dead start 2024-03-04T09:31:27.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:27.462+0000 7fc83aa00640 1 --2- 172.21.15.6:0/42639806 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc820003a00 0x7fc820003e20 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7fc82c001720 tx=0x7fc820005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6084bb6bd26fdc0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:27.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:27.463+0000 7fc839600640 1 -- 172.21.15.6:0/42639806 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc82c001370 con 0x7fc820003a00 2024-03-04T09:31:27.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:27.463+0000 7fc839600640 1 -- 172.21.15.6:0/42639806 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc82c0052d0 con 0x7fc820003a00 2024-03-04T09:31:27.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:27.463+0000 7fc839600640 1 -- 172.21.15.6:0/42639806 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc82c003830 con 0x7fc820003a00 2024-03-04T09:31:27.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:27.463+0000 7fc83be00640 1 -- 172.21.15.6:0/42639806 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc820003a00 msgr2=0x7fc820003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:27.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:27.463+0000 7fc83be00640 1 --2- 172.21.15.6:0/42639806 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc820003a00 0x7fc820003e20 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7fc82c001720 tx=0x7fc820005830 comp rx=0 tx=0).stop 2024-03-04T09:31:27.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:27.463+0000 7fc83be00640 1 -- 172.21.15.6:0/42639806 shutdown_connections 2024-03-04T09:31:27.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:27.463+0000 7fc83be00640 1 --2- 172.21.15.6:0/42639806 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc820003a00 0x7fc820003e20 unknown :-1 s=CLOSED pgs=1259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:27.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:27.463+0000 7fc83be00640 1 -- 172.21.15.6:0/42639806 >> 172.21.15.6:0/42639806 conn(0x7fc83c06b3d0 msgr2=0x7fc83c06b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:27.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:27.463+0000 7fc83be00640 1 -- 172.21.15.6:0/42639806 shutdown_connections 2024-03-04T09:31:27.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:27.463+0000 7fc83be00640 1 -- 172.21.15.6:0/42639806 wait complete. 2024-03-04T09:31:27.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:27.466+0000 7fc83be00640 1 Processor -- start 2024-03-04T09:31:27.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:27.466+0000 7fc83be00640 1 -- start start 2024-03-04T09:31:27.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:27.468+0000 7fc83be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc820003a00 0x7fc83c0af070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:27.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:27.468+0000 7fc83be00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc83c13d6e0 0x7fc83c13db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:27.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:27.468+0000 7fc83be00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc83c0a9540 0x7fc83c0a99c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:27.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:27.468+0000 7fc83be00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc83c0ccad0 con 0x7fc820003a00 2024-03-04T09:31:27.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:27.468+0000 7fc83be00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc83c0cc950 con 0x7fc83c0a9540 2024-03-04T09:31:27.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:27.468+0000 7fc83be00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc83c0ccc50 con 0x7fc83c13d6e0 2024-03-04T09:31:27.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:27.468+0000 7fc83aa00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc820003a00 0x7fc83c0af070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:27.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:27.468+0000 7fc83aa00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc820003a00 0x7fc83c0af070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:32844/0 (socket says 172.21.15.6:32844) 2024-03-04T09:31:27.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:27.468+0000 7fc83aa00640 1 -- 172.21.15.6:0/1755206641 learned_addr learned my addr 172.21.15.6:0/1755206641 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:31:27.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:27.469+0000 7fc83b400640 1 --2- 172.21.15.6:0/1755206641 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc83c0a9540 0x7fc83c0a99c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:27.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:27.469+0000 7fc83aa00640 1 -- 172.21.15.6:0/1755206641 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc83c13d6e0 msgr2=0x7fc83c13db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:27.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:27.469+0000 7fc83a000640 1 --2- 172.21.15.6:0/1755206641 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc83c13d6e0 0x7fc83c13db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:27.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:27.469+0000 7fc83aa00640 1 --2- 172.21.15.6:0/1755206641 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc83c13d6e0 0x7fc83c13db60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:27.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:27.469+0000 7fc83aa00640 1 -- 172.21.15.6:0/1755206641 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc83c0a9540 msgr2=0x7fc83c0a99c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:27.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:27.469+0000 7fc83aa00640 1 --2- 172.21.15.6:0/1755206641 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc83c0a9540 0x7fc83c0a99c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:27.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:27.469+0000 7fc83aa00640 1 -- 172.21.15.6:0/1755206641 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc83c0a9f00 con 0x7fc820003a00 2024-03-04T09:31:27.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:27.469+0000 7fc83b400640 1 --2- 172.21.15.6:0/1755206641 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc83c0a9540 0x7fc83c0a99c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:31:27.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:27.469+0000 7fc83a000640 1 --2- 172.21.15.6:0/1755206641 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc83c13d6e0 0x7fc83c13db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:31:27.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:27.469+0000 7fc83aa00640 1 --2- 172.21.15.6:0/1755206641 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc820003a00 0x7fc83c0af070 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7fc82c005040 tx=0x7fc82c001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:27.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:27.470+0000 7fc832a00640 1 -- 172.21.15.6:0/1755206641 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc82c041020 con 0x7fc820003a00 2024-03-04T09:31:27.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:27.470+0000 7fc832a00640 1 -- 172.21.15.6:0/1755206641 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc82c005480 con 0x7fc820003a00 2024-03-04T09:31:27.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:27.470+0000 7fc832a00640 1 -- 172.21.15.6:0/1755206641 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc82c007a30 con 0x7fc820003a00 2024-03-04T09:31:27.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:27.470+0000 7fc83be00640 1 -- 172.21.15.6:0/1755206641 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc83c13f5c0 con 0x7fc820003a00 2024-03-04T09:31:27.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:27.470+0000 7fc83be00640 1 -- 172.21.15.6:0/1755206641 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc83c13fb80 con 0x7fc820003a00 2024-03-04T09:31:27.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:27.471+0000 7fc83be00640 1 -- 172.21.15.6:0/1755206641 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc83c04f330 con 0x7fc820003a00 2024-03-04T09:31:27.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:27.472+0000 7fc832a00640 1 -- 172.21.15.6:0/1755206641 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc82c044020 con 0x7fc820003a00 2024-03-04T09:31:27.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:27.473+0000 7fc832a00640 1 --2- 172.21.15.6:0/1755206641 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc81c06b4d0 0x7fc81c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:27.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:27.473+0000 7fc832a00640 1 -- 172.21.15.6:0/1755206641 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc82c0abc50 con 0x7fc820003a00 2024-03-04T09:31:27.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:27.473+0000 7fc83a000640 1 --2- 172.21.15.6:0/1755206641 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc81c06b4d0 0x7fc81c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:27.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:27.475+0000 7fc83a000640 1 --2- 172.21.15.6:0/1755206641 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc81c06b4d0 0x7fc81c06d990 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fc83c073200 tx=0x7fc828000f90 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:27.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:27.478+0000 7fc832a00640 1 -- 172.21.15.6:0/1755206641 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc82c04b030 con 0x7fc820003a00 2024-03-04T09:31:27.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:27.789+0000 7fc83be00640 1 -- 172.21.15.6:0/1755206641 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc83c143bd0 con 0x7fc820003a00 2024-03-04T09:31:27.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:27.790+0000 7fc832a00640 1 -- 172.21.15.6:0/1755206641 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+4973 (secure 0 0 0) 0x7fc82c0303c0 con 0x7fc820003a00 2024-03-04T09:31:27.787 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:31:27.787 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":77,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5792,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6836/4034557380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":4034557380},{"type":"v1","addr":"172.21.15.146:6836","nonce":4034557380}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":72},{"gid":6370,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/348873623","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":348873623},{"type":"v1","addr":"172.21.15.6:6835","nonce":348873623}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":72},{"gid":6382,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/763528142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":763528142},{"type":"v1","addr":"172.21.15.146:6837","nonce":763528142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":72}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:31:21.114351+0000","modified":"2024-03-04T09:31:24.172076+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7560},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7560":{"gid":7560,"name":"a","rank":0,"incarnation":73,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2697931210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2697931210},{"type":"v1","addr":"172.21.15.6:6834","nonce":2697931210}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7560,"qdb_cluster":[7560]},"id":8}]} 2024-03-04T09:31:27.787 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 77 2024-03-04T09:31:27.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:27.793+0000 7fc83be00640 1 -- 172.21.15.6:0/1755206641 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc81c06b4d0 msgr2=0x7fc81c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:27.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:27.793+0000 7fc83be00640 1 --2- 172.21.15.6:0/1755206641 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc81c06b4d0 0x7fc81c06d990 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fc83c073200 tx=0x7fc828000f90 comp rx=0 tx=0).stop 2024-03-04T09:31:27.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:27.793+0000 7fc83be00640 1 -- 172.21.15.6:0/1755206641 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc820003a00 msgr2=0x7fc83c0af070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:27.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:27.793+0000 7fc83be00640 1 --2- 172.21.15.6:0/1755206641 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc820003a00 0x7fc83c0af070 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7fc82c005040 tx=0x7fc82c001120 comp rx=0 tx=0).stop 2024-03-04T09:31:27.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:27.793+0000 7fc83be00640 1 -- 172.21.15.6:0/1755206641 shutdown_connections 2024-03-04T09:31:27.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:27.793+0000 7fc83be00640 1 --2- 172.21.15.6:0/1755206641 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc83c0a9540 0x7fc83c0a99c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:27.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:27.793+0000 7fc83be00640 1 --2- 172.21.15.6:0/1755206641 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc81c06b4d0 0x7fc81c06d990 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:27.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:27.793+0000 7fc83be00640 1 --2- 172.21.15.6:0/1755206641 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc83c13d6e0 0x7fc83c13db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:27.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:27.793+0000 7fc83be00640 1 --2- 172.21.15.6:0/1755206641 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc820003a00 0x7fc83c0af070 unknown :-1 s=CLOSED pgs=1260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:27.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:27.793+0000 7fc83be00640 1 -- 172.21.15.6:0/1755206641 >> 172.21.15.6:0/1755206641 conn(0x7fc83c06b3d0 msgr2=0x7fc83c0cd930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:27.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:27.793+0000 7fc83be00640 1 -- 172.21.15.6:0/1755206641 shutdown_connections 2024-03-04T09:31:27.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:27.793+0000 7fc83be00640 1 -- 172.21.15.6:0/1755206641 wait complete. 2024-03-04T09:31:28.211 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 77, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:31:21.114351+0000', 'modified': '2024-03-04T09:31:24.172076+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7560}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7560': {'gid': 7560, 'name': 'a', 'rank': 0, 'incarnation': 73, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2697931210', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2697931210}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2697931210}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [17], 'metadata_pool': 16, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7560, 'qdb_cluster': [7560]} 2024-03-04T09:31:28.212 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-04T09:31:28.212 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-04T09:31:28.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.439+0000 7f8e9f200640 1 Processor -- start 2024-03-04T09:31:28.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.439+0000 7f8e9f200640 1 -- start start 2024-03-04T09:31:28.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.442+0000 7f8e9f200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f8ea0144c00 0x7f8ea0145280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:28.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.442+0000 7f8e9f200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ea00d0910 con 0x7f8ea01457c0 2024-03-04T09:31:28.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.442+0000 7f8e9f200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ea00d0bf0 con 0x7f8ea006b6c0 2024-03-04T09:31:28.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.442+0000 7f8e9f200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ea00d0ed0 con 0x7f8ea0144c00 2024-03-04T09:31:28.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.442+0000 7f8e95400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f8ea01457c0 0x7f8ea006afb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:57006/0 (socket says 172.21.15.6:57006) 2024-03-04T09:31:28.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.442+0000 7f8e95400640 1 -- 172.21.15.6:0/3725022073 learned_addr learned my addr 172.21.15.6:0/3725022073 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:31:28.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.443+0000 7f8e9d400640 1 -- 172.21.15.6:0/3725022073 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 916492390 0 0) 0x7f8ea00d0910 con 0x7f8ea01457c0 2024-03-04T09:31:28.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.443+0000 7f8e9d400640 1 -- 172.21.15.6:0/3725022073 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e84003520 con 0x7f8ea01457c0 2024-03-04T09:31:28.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.443+0000 7f8e9de00640 1 --2- 172.21.15.6:0/3725022073 >> v2:172.21.15.146:3300/0 conn(0x7f8ea0144c00 0x7f8ea0145280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:28.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.443+0000 7f8e9d400640 1 -- 172.21.15.6:0/3725022073 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2221531493 0 0) 0x7f8e84003520 con 0x7f8ea01457c0 2024-03-04T09:31:28.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.443+0000 7f8e9d400640 1 -- 172.21.15.6:0/3725022073 >> v2:172.21.15.146:3300/0 conn(0x7f8ea0144c00 msgr2=0x7f8ea0145280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:28.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.443+0000 7f8e9d400640 1 --2- 172.21.15.6:0/3725022073 >> v2:172.21.15.146:3300/0 conn(0x7f8ea0144c00 0x7f8ea0145280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:28.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.443+0000 7f8e9d400640 1 -- 172.21.15.6:0/3725022073 >> v1:172.21.15.146:6789/0 conn(0x7f8ea006b6c0 legacy=0x7f8ea00cf130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:28.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.443+0000 7f8e9d400640 1 -- 172.21.15.6:0/3725022073 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ea00d11b0 con 0x7f8ea01457c0 2024-03-04T09:31:28.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.443+0000 7f8e9d400640 1 -- 172.21.15.6:0/3725022073 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7f8e88001670 con 0x7f8ea01457c0 2024-03-04T09:31:28.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.443+0000 7f8e9d400640 1 -- 172.21.15.6:0/3725022073 >> v1:172.21.15.6:6789/0 conn(0x7f8ea01457c0 legacy=0x7f8ea006afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:28.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.443+0000 7f8e9d400640 1 --2- 172.21.15.6:0/3725022073 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8e840039b0 0x7f8e84003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:28.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.443+0000 7f8e9d400640 1 --2- 172.21.15.6:0/3725022073 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8e84004800 0x7f8e84004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:28.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.443+0000 7f8e9d400640 1 --2- 172.21.15.6:0/3725022073 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8e840051c0 0x7f8e8400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:28.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.443+0000 7f8e9d400640 1 -- 172.21.15.6:0/3725022073 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ea00d0ed0 con 0x7f8e84004800 2024-03-04T09:31:28.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.443+0000 7f8e9d400640 1 -- 172.21.15.6:0/3725022073 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e8400c110 con 0x7f8e840051c0 2024-03-04T09:31:28.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.443+0000 7f8e9d400640 1 -- 172.21.15.6:0/3725022073 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e8400c310 con 0x7f8e840039b0 2024-03-04T09:31:28.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.444+0000 7f8e95400640 1 --2- 172.21.15.6:0/3725022073 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8e84004800 0x7f8e84004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:28.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.444+0000 7f8e9de00640 1 --2- 172.21.15.6:0/3725022073 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8e840039b0 0x7f8e84003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:28.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.444+0000 7f8e9e800640 1 --2- 172.21.15.6:0/3725022073 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8e840051c0 0x7f8e8400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:28.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.444+0000 7f8e95400640 1 -- 172.21.15.6:0/3725022073 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8e840039b0 msgr2=0x7f8e84003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:28.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.444+0000 7f8e95400640 1 --2- 172.21.15.6:0/3725022073 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8e840039b0 0x7f8e84003dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:28.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.444+0000 7f8e95400640 1 -- 172.21.15.6:0/3725022073 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8e840051c0 msgr2=0x7f8e8400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:28.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.444+0000 7f8e95400640 1 --2- 172.21.15.6:0/3725022073 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8e840051c0 0x7f8e8400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:28.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.444+0000 7f8e95400640 1 -- 172.21.15.6:0/3725022073 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e84003520 con 0x7f8e84004800 2024-03-04T09:31:28.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.444+0000 7f8e9e800640 1 --2- 172.21.15.6:0/3725022073 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8e840051c0 0x7f8e8400b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:31:28.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.444+0000 7f8e9e800640 1 -- 172.21.15.6:0/3725022073 reap_dead start 2024-03-04T09:31:28.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.445+0000 7f8e95400640 1 --2- 172.21.15.6:0/3725022073 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8e84004800 0x7f8e84004c80 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7f8ea00b6030 tx=0x7f8e84005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=358029900c5504f server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:28.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.445+0000 7f8e9d400640 1 -- 172.21.15.6:0/3725022073 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e88007d30 con 0x7f8e84004800 2024-03-04T09:31:28.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.445+0000 7f8e9d400640 1 -- 172.21.15.6:0/3725022073 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e88030070 con 0x7f8e84004800 2024-03-04T09:31:28.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.445+0000 7f8e9d400640 1 -- 172.21.15.6:0/3725022073 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e88005d70 con 0x7f8e84004800 2024-03-04T09:31:28.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.446+0000 7f8e9f200640 1 -- 172.21.15.6:0/3725022073 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8e84004800 msgr2=0x7f8e84004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:28.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.446+0000 7f8e9f200640 1 --2- 172.21.15.6:0/3725022073 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8e84004800 0x7f8e84004c80 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7f8ea00b6030 tx=0x7f8e84005830 comp rx=0 tx=0).stop 2024-03-04T09:31:28.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.446+0000 7f8e9f200640 1 -- 172.21.15.6:0/3725022073 shutdown_connections 2024-03-04T09:31:28.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.446+0000 7f8e9f200640 1 --2- 172.21.15.6:0/3725022073 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8e84004800 0x7f8e84004c80 unknown :-1 s=CLOSED pgs=1262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:28.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.446+0000 7f8e9f200640 1 -- 172.21.15.6:0/3725022073 >> 172.21.15.6:0/3725022073 conn(0x7f8ea006c0c0 msgr2=0x7f8ea006c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:28.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.446+0000 7f8e9f200640 1 -- 172.21.15.6:0/3725022073 shutdown_connections 2024-03-04T09:31:28.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.446+0000 7f8e9f200640 1 -- 172.21.15.6:0/3725022073 wait complete. 2024-03-04T09:31:28.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.448+0000 7f8e9f200640 1 Processor -- start 2024-03-04T09:31:28.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.449+0000 7f8e9f200640 1 -- start start 2024-03-04T09:31:28.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.450+0000 7f8e9f200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8e84004800 0x7f8ea0155780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:28.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.450+0000 7f8e9f200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8ea013c580 0x7f8ea0155f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:28.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.450+0000 7f8e9f200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8ea013bfc0 0x7f8ea0156490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:28.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.450+0000 7f8e9f200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ea00d19b0 con 0x7f8ea013bfc0 2024-03-04T09:31:28.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.450+0000 7f8e9f200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ea0144fe0 con 0x7f8e84004800 2024-03-04T09:31:28.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.450+0000 7f8e9f200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ea006f820 con 0x7f8ea013c580 2024-03-04T09:31:28.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.451+0000 7f8e9e800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8ea013bfc0 0x7f8ea0156490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:28.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.451+0000 7f8e9e800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8ea013bfc0 0x7f8ea0156490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:32870/0 (socket says 172.21.15.6:32870) 2024-03-04T09:31:28.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.451+0000 7f8e9e800640 1 -- 172.21.15.6:0/1304136646 learned_addr learned my addr 172.21.15.6:0/1304136646 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:31:28.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.451+0000 7f8e9de00640 1 --2- 172.21.15.6:0/1304136646 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8e84004800 0x7f8ea0155780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:28.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.451+0000 7f8e95400640 1 --2- 172.21.15.6:0/1304136646 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8ea013c580 0x7f8ea0155f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:28.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.451+0000 7f8e9e800640 1 -- 172.21.15.6:0/1304136646 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8ea013c580 msgr2=0x7f8ea0155f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:28.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.451+0000 7f8e9e800640 1 --2- 172.21.15.6:0/1304136646 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8ea013c580 0x7f8ea0155f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:28.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.451+0000 7f8e9e800640 1 -- 172.21.15.6:0/1304136646 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8e84004800 msgr2=0x7f8ea0155780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:28.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.451+0000 7f8e9e800640 1 --2- 172.21.15.6:0/1304136646 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8e84004800 0x7f8ea0155780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:28.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.451+0000 7f8e9e800640 1 -- 172.21.15.6:0/1304136646 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ea015a250 con 0x7f8ea013bfc0 2024-03-04T09:31:28.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.451+0000 7f8e9e800640 1 --2- 172.21.15.6:0/1304136646 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8ea013bfc0 0x7f8ea0156490 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7f8ea0073470 tx=0x7f8e9800d9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:28.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.452+0000 7f8e96a00640 1 -- 172.21.15.6:0/1304136646 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e98018070 con 0x7f8ea013bfc0 2024-03-04T09:31:28.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.452+0000 7f8e96a00640 1 -- 172.21.15.6:0/1304136646 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e98003da0 con 0x7f8ea013bfc0 2024-03-04T09:31:28.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.452+0000 7f8e96a00640 1 -- 172.21.15.6:0/1304136646 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e980052b0 con 0x7f8ea013bfc0 2024-03-04T09:31:28.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.452+0000 7f8e9f200640 1 -- 172.21.15.6:0/1304136646 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ea015a530 con 0x7f8ea013bfc0 2024-03-04T09:31:28.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.452+0000 7f8e9f200640 1 -- 172.21.15.6:0/1304136646 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ea015aaf0 con 0x7f8ea013bfc0 2024-03-04T09:31:28.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.454+0000 7f8e96a00640 1 -- 172.21.15.6:0/1304136646 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8e9801b070 con 0x7f8ea013bfc0 2024-03-04T09:31:28.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.455+0000 7f8e96a00640 1 --2- 172.21.15.6:0/1304136646 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f8e7c06b500 0x7f8e7c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:28.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.455+0000 7f8e96a00640 1 -- 172.21.15.6:0/1304136646 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8e98089840 con 0x7f8ea013bfc0 2024-03-04T09:31:28.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.456+0000 7f8e9de00640 1 --2- 172.21.15.6:0/1304136646 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f8e7c06b500 0x7f8e7c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:28.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.457+0000 7f8e9de00640 1 --2- 172.21.15.6:0/1304136646 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f8e7c06b500 0x7f8e7c06d9c0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f8e90006fd0 tx=0x7f8e90008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:28.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.458+0000 7f8ea7fa4740 1 Processor -- start 2024-03-04T09:31:28.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.458+0000 7f8ea7fa4740 1 -- start start 2024-03-04T09:31:28.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.459+0000 7f8ea7fa4740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555fc0b4a510 0x555fc0b480d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:28.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.459+0000 7f8ea7fa4740 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x555fc0b29d50 0x555fc0b01a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:28.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.459+0000 7f8ea7fa4740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555fc0b01f80 0x555fc0b2c720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:28.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.459+0000 7f8ea7fa4740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x555fc0a66600 con 0x555fc0b01f80 2024-03-04T09:31:28.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.459+0000 7f8ea7fa4740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x555fc0a0f930 con 0x555fc0b4a510 2024-03-04T09:31:28.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.459+0000 7f8ea7fa4740 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x555fc0a626a0 con 0x555fc0b29d50 2024-03-04T09:31:28.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.460+0000 7f8e9e800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555fc0b01f80 0x555fc0b2c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:28.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.460+0000 7f8e9e800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555fc0b01f80 0x555fc0b2c720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:32884/0 (socket says 172.21.15.6:32884) 2024-03-04T09:31:28.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.460+0000 7f8e9e800640 1 -- 172.21.15.6:0/2098850590 learned_addr learned my addr 172.21.15.6:0/2098850590 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:31:28.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.460+0000 7f8e9de00640 1 --2- 172.21.15.6:0/2098850590 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555fc0b4a510 0x555fc0b480d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:28.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.460+0000 7f8e95400640 1 --2- 172.21.15.6:0/2098850590 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x555fc0b29d50 0x555fc0b01a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:28.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.460+0000 7f8e9e800640 1 -- 172.21.15.6:0/2098850590 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x555fc0b29d50 msgr2=0x555fc0b01a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:28.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.460+0000 7f8e9e800640 1 --2- 172.21.15.6:0/2098850590 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x555fc0b29d50 0x555fc0b01a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:28.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.460+0000 7f8e9e800640 1 -- 172.21.15.6:0/2098850590 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555fc0b4a510 msgr2=0x555fc0b480d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:28.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.460+0000 7f8e9e800640 1 --2- 172.21.15.6:0/2098850590 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555fc0b4a510 0x555fc0b480d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:28.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.460+0000 7f8e9e800640 1 -- 172.21.15.6:0/2098850590 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555fc077b2d0 con 0x555fc0b01f80 2024-03-04T09:31:28.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.460+0000 7f8e9de00640 1 --2- 172.21.15.6:0/2098850590 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555fc0b4a510 0x555fc0b480d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T09:31:28.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.461+0000 7f8e9e800640 1 --2- 172.21.15.6:0/2098850590 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555fc0b01f80 0x555fc0b2c720 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7f8e98087d90 tx=0x7f8e98057c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=857537428f2b5529 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:28.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.461+0000 7f8e8ea00640 1 -- 172.21.15.6:0/2098850590 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e980562c0 con 0x555fc0b01f80 2024-03-04T09:31:28.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.461+0000 7f8e8ea00640 1 -- 172.21.15.6:0/2098850590 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e98054630 con 0x555fc0b01f80 2024-03-04T09:31:28.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.461+0000 7f8e8ea00640 1 -- 172.21.15.6:0/2098850590 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e98057630 con 0x555fc0b01f80 2024-03-04T09:31:28.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.461+0000 7f8ea7fa4740 1 -- 172.21.15.6:0/2098850590 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555fc0b01f80 msgr2=0x555fc0b2c720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:28.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.461+0000 7f8ea7fa4740 1 --2- 172.21.15.6:0/2098850590 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555fc0b01f80 0x555fc0b2c720 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7f8e98087d90 tx=0x7f8e98057c30 comp rx=0 tx=0).stop 2024-03-04T09:31:28.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.462+0000 7f8ea7fa4740 1 -- 172.21.15.6:0/2098850590 shutdown_connections 2024-03-04T09:31:28.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.462+0000 7f8ea7fa4740 1 --2- 172.21.15.6:0/2098850590 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555fc0b01f80 0x555fc0b2c720 unknown :-1 s=CLOSED pgs=1264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:28.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.462+0000 7f8ea7fa4740 1 --2- 172.21.15.6:0/2098850590 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x555fc0b29d50 0x555fc0b01a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:28.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.462+0000 7f8ea7fa4740 1 --2- 172.21.15.6:0/2098850590 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555fc0b4a510 0x555fc0b480d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:28.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.462+0000 7f8ea7fa4740 1 -- 172.21.15.6:0/2098850590 >> 172.21.15.6:0/2098850590 conn(0x555fc0afeba0 msgr2=0x555fc0b47720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:28.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.462+0000 7f8ea7fa4740 1 -- 172.21.15.6:0/2098850590 shutdown_connections 2024-03-04T09:31:28.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.462+0000 7f8ea7fa4740 1 -- 172.21.15.6:0/2098850590 wait complete. 2024-03-04T09:31:28.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.463+0000 7f8ea7fa4740 1 -- start start 2024-03-04T09:31:28.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.463+0000 7f8ea7fa4740 1 Processor -- start 2024-03-04T09:31:28.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.465+0000 7f8ea7fa4740 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x555fc0b29d50 0x555fc0b26020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:28.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.465+0000 7f8ea7fa4740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555fc0b4a510 0x555fc0b41c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:28.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.465+0000 7f8ea7fa4740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555fc0afba90 0x555fc0afbf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:28.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.465+0000 7f8ea7fa4740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x555fc0b00ea0 con 0x555fc0b4a510 2024-03-04T09:31:28.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.465+0000 7f8ea7fa4740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x555fc0a22d90 con 0x555fc0afba90 2024-03-04T09:31:28.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.465+0000 7f8ea7fa4740 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x555fc0afd1d0 con 0x555fc0b29d50 2024-03-04T09:31:28.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.465+0000 7f8ea7fa4740 10 client.0 _ll_register_callbacks cb 0x555fc0a5e350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T09:31:28.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.465+0000 7f8e95400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555fc0b4a510 0x555fc0b41c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:28.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.465+0000 7f8e95400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555fc0b4a510 0x555fc0b41c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:32896/0 (socket says 172.21.15.6:32896) 2024-03-04T09:31:28.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.465+0000 7f8e95400640 1 -- 172.21.15.6:0/1589422151 learned_addr learned my addr 172.21.15.6:0/1589422151 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:31:28.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.466+0000 7f8e9de00640 1 --2- 172.21.15.6:0/1589422151 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x555fc0b29d50 0x555fc0b26020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:28.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.466+0000 7f8e95400640 1 -- 172.21.15.6:0/1589422151 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x555fc0b29d50 msgr2=0x555fc0b26020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:28.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.466+0000 7f8e9e800640 1 --2- 172.21.15.6:0/1589422151 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555fc0afba90 0x555fc0afbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:28.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.466+0000 7f8e95400640 1 --2- 172.21.15.6:0/1589422151 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x555fc0b29d50 0x555fc0b26020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:28.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.466+0000 7f8e95400640 1 -- 172.21.15.6:0/1589422151 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555fc0afba90 msgr2=0x555fc0afbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:28.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.466+0000 7f8e95400640 1 --2- 172.21.15.6:0/1589422151 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555fc0afba90 0x555fc0afbf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:28.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.466+0000 7f8e95400640 1 -- 172.21.15.6:0/1589422151 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555fc08456b0 con 0x555fc0b4a510 2024-03-04T09:31:28.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.466+0000 7f8e95400640 1 --2- 172.21.15.6:0/1589422151 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555fc0b4a510 0x555fc0b41c70 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7f8e8800c2d0 tx=0x7f8e8802e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a94d2d4f6e818255 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:28.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.466+0000 7f8e83400640 10 client.0 ms_handle_connect on v2:172.21.15.6:3300/0 2024-03-04T09:31:28.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.466+0000 7f8e83400640 1 -- 172.21.15.6:0/1589422151 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e8802cca0 con 0x555fc0b4a510 2024-03-04T09:31:28.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.466+0000 7f8e83400640 1 -- 172.21.15.6:0/1589422151 <== mon.0 v2:172.21.15.6:3300/0 2 ==== osd_map(109..109 src has 1..109) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8e880047e0 con 0x555fc0b4a510 2024-03-04T09:31:28.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.466+0000 7f8ea7fa4740 1 -- 172.21.15.6:0/1589422151 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8e98054630 con 0x555fc0b4a510 2024-03-04T09:31:28.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.467+0000 7f8e83400640 1 -- 172.21.15.6:0/1589422151 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_get_version_reply(handle=1 version=77) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8e8804c050 con 0x555fc0b4a510 2024-03-04T09:31:28.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.467+0000 7f8ea7fa4740 10 client.7701 fetch_fsmap learned FSMap version 77 2024-03-04T09:31:28.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.467+0000 7f8ea7fa4740 1 -- 172.21.15.6:0/1589422151 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,fsmap=77,monmap=2+}) v3 -- 0x555fc0b42a10 con 0x555fc0b4a510 2024-03-04T09:31:28.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.467+0000 7f8e83400640 1 -- 172.21.15.6:0/1589422151 <== mon.0 v2:172.21.15.6:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e98054630 con 0x555fc0b4a510 2024-03-04T09:31:28.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.467+0000 7f8e83400640 1 -- 172.21.15.6:0/1589422151 <== mon.0 v2:172.21.15.6:3300/0 5 ==== fsmap(e 77) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f8e8804f020 con 0x555fc0b4a510 2024-03-04T09:31:28.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.467+0000 7f8ea7fa4740 10 client.7701 fetch_fsmap finished waiting for FSMap version 77 2024-03-04T09:31:28.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.467+0000 7f8ea7fa4740 10 client.7701 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-04T09:31:28.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.467+0000 7f8ea7fa4740 20 client.7701 populate_metadata read hostname 'smithi006' 2024-03-04T09:31:28.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.467+0000 7f8ea7fa4740 1 --2- 172.21.15.6:0/1589422151 >> [v2:172.21.15.6:6832/2697931210,v1:172.21.15.6:6834/2697931210] conn(0x555fc0b44580 0x555fc0b6a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:28.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.467+0000 7f8ea7fa4740 4 client.7701 mds_command: new command op to 7560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T09:31:28.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.467+0000 7f8ea7fa4740 1 -- 172.21.15.6:0/1589422151 --> [v2:172.21.15.6:6832/2697931210,v1:172.21.15.6:6834/2697931210] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555fc0a73b40 con 0x555fc0b44580 2024-03-04T09:31:28.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.467+0000 7f8e9de00640 1 --2- 172.21.15.6:0/1589422151 >> [v2:172.21.15.6:6832/2697931210,v1:172.21.15.6:6834/2697931210] conn(0x555fc0b44580 0x555fc0b6a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:28.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.468+0000 7f8e9de00640 1 --2- 172.21.15.6:0/1589422151 >> [v2:172.21.15.6:6832/2697931210,v1:172.21.15.6:6834/2697931210] conn(0x555fc0b44580 0x555fc0b6a4c0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4f9de8b151b97013 server_cookie=58ca0ab10de39c6 in_seq=0 out_seq=0 2024-03-04T09:31:28.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.468+0000 7f8e83400640 10 client.7701 ms_handle_connect on v2:172.21.15.6:6832/2697931210 2024-03-04T09:31:28.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.469+0000 7f8e83400640 1 -- 172.21.15.6:0/1589422151 <== mds.0 v2:172.21.15.6:6832/2697931210 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x555fc0a73b40 con 0x555fc0b44580 2024-03-04T09:31:28.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.469+0000 7f8e83400640 10 client.7701 handle_command_reply: tid=0 2024-03-04T09:31:28.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.469+0000 7f8ea7fa4740 1 client.7701 shutdown 2024-03-04T09:31:28.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.469+0000 7f8ea7fa4740 1 -- 172.21.15.6:0/1589422151 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555fc0b4a510 msgr2=0x555fc0b41c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:28.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.469+0000 7f8ea7fa4740 1 --2- 172.21.15.6:0/1589422151 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555fc0b4a510 0x555fc0b41c70 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7f8e8800c2d0 tx=0x7f8e8802e070 comp rx=0 tx=0).stop 2024-03-04T09:31:28.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.470+0000 7f8ea7fa4740 1 -- 172.21.15.6:0/1589422151 shutdown_connections 2024-03-04T09:31:28.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.470+0000 7f8ea7fa4740 1 --2- 172.21.15.6:0/1589422151 >> [v2:172.21.15.6:6832/2697931210,v1:172.21.15.6:6834/2697931210] conn(0x555fc0b44580 0x555fc0b6a4c0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:28.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.470+0000 7f8ea7fa4740 1 --2- 172.21.15.6:0/1589422151 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555fc0afba90 0x555fc0afbf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:28.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.470+0000 7f8ea7fa4740 1 --2- 172.21.15.6:0/1589422151 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555fc0b4a510 0x555fc0b41c70 unknown :-1 s=CLOSED pgs=1265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:28.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.470+0000 7f8ea7fa4740 1 --2- 172.21.15.6:0/1589422151 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x555fc0b29d50 0x555fc0b26020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:28.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.470+0000 7f8ea7fa4740 1 -- 172.21.15.6:0/1589422151 >> 172.21.15.6:0/1589422151 conn(0x555fc0b01f80 msgr2=0x555fc0b47640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:28.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.470+0000 7f8e83400640 0 client.7701 ms_handle_reset on v2:172.21.15.6:6832/2697931210 2024-03-04T09:31:28.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.470+0000 7f8ea7fa4740 1 -- 172.21.15.6:0/1589422151 shutdown_connections 2024-03-04T09:31:28.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.470+0000 7f8ea7fa4740 1 -- 172.21.15.6:0/1589422151 wait complete. 2024-03-04T09:31:28.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.470+0000 7f8ea7fa4740 20 client.7701 trim_cache size 0 max 16384 2024-03-04T09:31:28.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.488+0000 7f8ea7fa4740 1 Processor -- start 2024-03-04T09:31:28.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.488+0000 7f8ea7fa4740 1 -- start start 2024-03-04T09:31:28.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.489+0000 7f8ea7fa4740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555fc0afba90 0x555fc0b305e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:28.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.489+0000 7f8ea7fa4740 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x555fc0b29d50 0x555fc0b40fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:28.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.489+0000 7f8ea7fa4740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555fc0b4a510 0x555fc0b468f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:28.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.489+0000 7f8ea7fa4740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ea01fc380 con 0x555fc0afba90 2024-03-04T09:31:28.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.489+0000 7f8ea7fa4740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ea00b50a0 con 0x555fc0b4a510 2024-03-04T09:31:28.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.489+0000 7f8ea7fa4740 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ea00b1db0 con 0x555fc0b29d50 2024-03-04T09:31:28.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.489+0000 7f8e9de00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555fc0afba90 0x555fc0b305e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:28.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.490+0000 7f8e9de00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555fc0afba90 0x555fc0b305e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:32898/0 (socket says 172.21.15.6:32898) 2024-03-04T09:31:28.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.490+0000 7f8e9de00640 1 -- 172.21.15.6:0/474680869 learned_addr learned my addr 172.21.15.6:0/474680869 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:31:28.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.490+0000 7f8e9de00640 1 -- 172.21.15.6:0/474680869 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x555fc0b29d50 msgr2=0x555fc0b40fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:28.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.490+0000 7f8e9de00640 1 --2- 172.21.15.6:0/474680869 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x555fc0b29d50 0x555fc0b40fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:28.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.490+0000 7f8e9de00640 1 -- 172.21.15.6:0/474680869 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555fc0b4a510 msgr2=0x555fc0b468f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:28.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.490+0000 7f8e9de00640 1 --2- 172.21.15.6:0/474680869 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555fc0b4a510 0x555fc0b468f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:28.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.490+0000 7f8e9de00640 1 -- 172.21.15.6:0/474680869 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555fc0b5b830 con 0x555fc0afba90 2024-03-04T09:31:28.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.490+0000 7f8e9de00640 1 --2- 172.21.15.6:0/474680869 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555fc0afba90 0x555fc0b305e0 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7f8e9000ce50 tx=0x7f8e90016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d32a7598c476b252 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:28.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.490+0000 7f8e8e000640 1 -- 172.21.15.6:0/474680869 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e9002f070 con 0x555fc0afba90 2024-03-04T09:31:28.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.490+0000 7f8e8e000640 1 -- 172.21.15.6:0/474680869 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e9001d040 con 0x555fc0afba90 2024-03-04T09:31:28.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.490+0000 7f8e8e000640 1 -- 172.21.15.6:0/474680869 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e9001c070 con 0x555fc0afba90 2024-03-04T09:31:28.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.491+0000 7f8ea7fa4740 1 -- 172.21.15.6:0/474680869 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555fc0afba90 msgr2=0x555fc0b305e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:28.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.491+0000 7f8ea7fa4740 1 --2- 172.21.15.6:0/474680869 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555fc0afba90 0x555fc0b305e0 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7f8e9000ce50 tx=0x7f8e90016000 comp rx=0 tx=0).stop 2024-03-04T09:31:28.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.491+0000 7f8ea7fa4740 1 -- 172.21.15.6:0/474680869 shutdown_connections 2024-03-04T09:31:28.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.491+0000 7f8ea7fa4740 1 --2- 172.21.15.6:0/474680869 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555fc0b4a510 0x555fc0b468f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:28.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.491+0000 7f8ea7fa4740 1 --2- 172.21.15.6:0/474680869 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x555fc0b29d50 0x555fc0b40fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:28.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.491+0000 7f8ea7fa4740 1 --2- 172.21.15.6:0/474680869 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555fc0afba90 0x555fc0b305e0 unknown :-1 s=CLOSED pgs=1266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:28.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.491+0000 7f8ea7fa4740 1 -- 172.21.15.6:0/474680869 >> 172.21.15.6:0/474680869 conn(0x555fc0b01f80 msgr2=0x555fc0b42fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:28.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.491+0000 7f8ea7fa4740 1 -- 172.21.15.6:0/474680869 shutdown_connections 2024-03-04T09:31:28.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.491+0000 7f8ea7fa4740 1 -- 172.21.15.6:0/474680869 wait complete. 2024-03-04T09:31:28.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.492+0000 7f8ea7fa4740 1 -- start start 2024-03-04T09:31:28.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.492+0000 7f8ea7fa4740 1 Processor -- start 2024-03-04T09:31:28.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.493+0000 7f8ea7fa4740 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x555fc0afba90 0x555fc0afbeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:28.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.493+0000 7f8ea7fa4740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555fc0b29d50 0x555fc0afc3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:28.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.493+0000 7f8ea7fa4740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555fc0b4a510 0x555fc0b40fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:28.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.493+0000 7f8ea7fa4740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x555fc0b421b0 con 0x555fc0b4a510 2024-03-04T09:31:28.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.493+0000 7f8ea7fa4740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ea00b23b0 con 0x555fc0b29d50 2024-03-04T09:31:28.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.493+0000 7f8ea7fa4740 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x555fc0b5bc40 con 0x555fc0afba90 2024-03-04T09:31:28.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.493+0000 7f8ea7fa4740 10 client.0 _ll_register_callbacks cb 0x7f8e840036f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T09:31:28.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.494+0000 7f8e9e800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555fc0b4a510 0x555fc0b40fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:28.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.494+0000 7f8e9e800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555fc0b4a510 0x555fc0b40fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:32904/0 (socket says 172.21.15.6:32904) 2024-03-04T09:31:28.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.494+0000 7f8e9e800640 1 -- 172.21.15.6:0/820529569 learned_addr learned my addr 172.21.15.6:0/820529569 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:31:28.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.494+0000 7f8e95400640 1 --2- 172.21.15.6:0/820529569 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555fc0b29d50 0x555fc0afc3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:28.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.494+0000 7f8e9de00640 1 --2- 172.21.15.6:0/820529569 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x555fc0afba90 0x555fc0afbeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:28.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.494+0000 7f8e9e800640 1 -- 172.21.15.6:0/820529569 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x555fc0afba90 msgr2=0x555fc0afbeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:28.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.494+0000 7f8e9e800640 1 --2- 172.21.15.6:0/820529569 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x555fc0afba90 0x555fc0afbeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:28.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.494+0000 7f8e9e800640 1 -- 172.21.15.6:0/820529569 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555fc0b29d50 msgr2=0x555fc0afc3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:28.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.494+0000 7f8e9e800640 1 --2- 172.21.15.6:0/820529569 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555fc0b29d50 0x555fc0afc3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:28.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.494+0000 7f8e9e800640 1 -- 172.21.15.6:0/820529569 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555fc0b2c140 con 0x555fc0b4a510 2024-03-04T09:31:28.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.494+0000 7f8e9e800640 1 --2- 172.21.15.6:0/820529569 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555fc0b4a510 0x555fc0b40fb0 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7f8e98002230 tx=0x7f8e98087c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25819461266330cf server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:28.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.494+0000 7f8e83400640 10 client.0 ms_handle_connect on v2:172.21.15.6:3300/0 2024-03-04T09:31:28.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.495+0000 7f8e83400640 1 -- 172.21.15.6:0/820529569 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e98057d90 con 0x555fc0b4a510 2024-03-04T09:31:28.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.495+0000 7f8e83400640 1 -- 172.21.15.6:0/820529569 <== mon.0 v2:172.21.15.6:3300/0 2 ==== osd_map(109..109 src has 1..109) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8e98057610 con 0x555fc0b4a510 2024-03-04T09:31:28.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.495+0000 7f8ea7fa4740 1 -- 172.21.15.6:0/820529569 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8e9001d040 con 0x555fc0b4a510 2024-03-04T09:31:28.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.495+0000 7f8e83400640 1 -- 172.21.15.6:0/820529569 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_get_version_reply(handle=1 version=77) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8e9807f050 con 0x555fc0b4a510 2024-03-04T09:31:28.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.495+0000 7f8ea7fa4740 10 client.7707 fetch_fsmap learned FSMap version 77 2024-03-04T09:31:28.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.495+0000 7f8ea7fa4740 1 -- 172.21.15.6:0/820529569 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,fsmap=77,monmap=2+}) v3 -- 0x555fc0b5b360 con 0x555fc0b4a510 2024-03-04T09:31:28.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.495+0000 7f8e83400640 1 -- 172.21.15.6:0/820529569 <== mon.0 v2:172.21.15.6:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e9001d040 con 0x555fc0b4a510 2024-03-04T09:31:28.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.495+0000 7f8e83400640 1 -- 172.21.15.6:0/820529569 <== mon.0 v2:172.21.15.6:3300/0 5 ==== fsmap(e 77) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f8e9805a070 con 0x555fc0b4a510 2024-03-04T09:31:28.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.495+0000 7f8ea7fa4740 10 client.7707 fetch_fsmap finished waiting for FSMap version 77 2024-03-04T09:31:28.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.495+0000 7f8ea7fa4740 10 client.7707 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-04T09:31:28.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.495+0000 7f8ea7fa4740 20 client.7707 populate_metadata read hostname 'smithi006' 2024-03-04T09:31:28.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.495+0000 7f8ea7fa4740 1 --2- 172.21.15.6:0/820529569 >> [v2:172.21.15.6:6832/2697931210,v1:172.21.15.6:6834/2697931210] conn(0x555fc0b28600 0x555fc0b289e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:28.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.495+0000 7f8ea7fa4740 4 client.7707 mds_command: new command op to 7560 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-04T09:31:28.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.495+0000 7f8ea7fa4740 1 -- 172.21.15.6:0/820529569 --> [v2:172.21.15.6:6832/2697931210,v1:172.21.15.6:6834/2697931210] -- command(tid 0: {"prefix": "status"}) v1 -- 0x555fc0a6de90 con 0x555fc0b28600 2024-03-04T09:31:28.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.496+0000 7f8e9de00640 1 --2- 172.21.15.6:0/820529569 >> [v2:172.21.15.6:6832/2697931210,v1:172.21.15.6:6834/2697931210] conn(0x555fc0b28600 0x555fc0b289e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:28.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.496+0000 7f8e9de00640 1 --2- 172.21.15.6:0/820529569 >> [v2:172.21.15.6:6832/2697931210,v1:172.21.15.6:6834/2697931210] conn(0x555fc0b28600 0x555fc0b289e0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=43160a212b8f1bf3 server_cookie=9ef15a67730986ff in_seq=0 out_seq=0 2024-03-04T09:31:28.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.496+0000 7f8e83400640 10 client.7707 ms_handle_connect on v2:172.21.15.6:6832/2697931210 2024-03-04T09:31:28.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.496+0000 7f8e83400640 1 -- 172.21.15.6:0/820529569 <== mds.0 v2:172.21.15.6:6832/2697931210 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x555fc0a6de90 con 0x555fc0b28600 2024-03-04T09:31:28.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.496+0000 7f8e83400640 10 client.7707 handle_command_reply: tid=0 2024-03-04T09:31:28.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.496+0000 7f8ea7fa4740 1 client.7707 shutdown 2024-03-04T09:31:28.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.497+0000 7f8ea7fa4740 1 -- 172.21.15.6:0/820529569 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555fc0b4a510 msgr2=0x555fc0b40fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:28.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.497+0000 7f8ea7fa4740 1 --2- 172.21.15.6:0/820529569 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555fc0b4a510 0x555fc0b40fb0 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7f8e98002230 tx=0x7f8e98087c40 comp rx=0 tx=0).stop 2024-03-04T09:31:28.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.497+0000 7f8ea7fa4740 1 -- 172.21.15.6:0/820529569 shutdown_connections 2024-03-04T09:31:28.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.497+0000 7f8ea7fa4740 1 --2- 172.21.15.6:0/820529569 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555fc0b4a510 0x555fc0b40fb0 unknown :-1 s=CLOSED pgs=1267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:28.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.497+0000 7f8ea7fa4740 1 --2- 172.21.15.6:0/820529569 >> [v2:172.21.15.6:6832/2697931210,v1:172.21.15.6:6834/2697931210] conn(0x555fc0b28600 0x555fc0b289e0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:28.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.497+0000 7f8ea7fa4740 1 --2- 172.21.15.6:0/820529569 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555fc0b29d50 0x555fc0afc3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:28.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.497+0000 7f8ea7fa4740 1 --2- 172.21.15.6:0/820529569 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x555fc0afba90 0x555fc0afbeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:28.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.497+0000 7f8ea7fa4740 1 -- 172.21.15.6:0/820529569 >> 172.21.15.6:0/820529569 conn(0x555fc0b01f80 msgr2=0x555fc0b014f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:28.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.497+0000 7f8e83400640 0 client.7707 ms_handle_reset on v2:172.21.15.6:6832/2697931210 2024-03-04T09:31:28.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.497+0000 7f8ea7fa4740 1 -- 172.21.15.6:0/820529569 shutdown_connections 2024-03-04T09:31:28.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.497+0000 7f8ea7fa4740 1 -- 172.21.15.6:0/820529569 wait complete. 2024-03-04T09:31:28.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.497+0000 7f8ea7fa4740 20 client.7707 trim_cache size 0 max 16384 2024-03-04T09:31:28.498 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-04T09:31:28.498 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_fsid": "edce0578-5ae4-4402-b45a-552bdef84d7c", 2024-03-04T09:31:28.498 INFO:teuthology.orchestra.run.smithi006.stdout: "whoami": 0, 2024-03-04T09:31:28.498 INFO:teuthology.orchestra.run.smithi006.stdout: "id": 7560, 2024-03-04T09:31:28.498 INFO:teuthology.orchestra.run.smithi006.stdout: "want_state": "up:active", 2024-03-04T09:31:28.498 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "up:active", 2024-03-04T09:31:28.498 INFO:teuthology.orchestra.run.smithi006.stdout: "fs_name": "cephfs", 2024-03-04T09:31:28.498 INFO:teuthology.orchestra.run.smithi006.stdout: "rank_uptime": 6.8647348810000004, 2024-03-04T09:31:28.498 INFO:teuthology.orchestra.run.smithi006.stdout: "mdsmap_epoch": 77, 2024-03-04T09:31:28.498 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap_epoch": 109, 2024-03-04T09:31:28.499 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap_epoch_barrier": 109, 2024-03-04T09:31:28.499 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 12.216460535 2024-03-04T09:31:28.499 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-04T09:31:28.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.498+0000 7f8e9f200640 1 -- 172.21.15.6:0/1304136646 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f8e7c06b500 msgr2=0x7f8e7c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:28.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.498+0000 7f8e9f200640 1 --2- 172.21.15.6:0/1304136646 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f8e7c06b500 0x7f8e7c06d9c0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f8e90006fd0 tx=0x7f8e90008040 comp rx=0 tx=0).stop 2024-03-04T09:31:28.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.498+0000 7f8e9f200640 1 -- 172.21.15.6:0/1304136646 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8ea013bfc0 msgr2=0x7f8ea0156490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:28.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.498+0000 7f8e9f200640 1 --2- 172.21.15.6:0/1304136646 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8ea013bfc0 0x7f8ea0156490 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7f8ea0073470 tx=0x7f8e9800d9d0 comp rx=0 tx=0).stop 2024-03-04T09:31:28.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.498+0000 7f8e9f200640 1 -- 172.21.15.6:0/1304136646 shutdown_connections 2024-03-04T09:31:28.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.498+0000 7f8e9f200640 1 --2- 172.21.15.6:0/1304136646 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8ea013bfc0 0x7f8ea0156490 unknown :-1 s=CLOSED pgs=1263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:28.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.498+0000 7f8e9f200640 1 --2- 172.21.15.6:0/1304136646 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f8e7c06b500 0x7f8e7c06d9c0 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:28.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.498+0000 7f8e9f200640 1 --2- 172.21.15.6:0/1304136646 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8ea013c580 0x7f8ea0155f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:28.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.498+0000 7f8e9f200640 1 --2- 172.21.15.6:0/1304136646 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8e84004800 0x7f8ea0155780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:28.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.498+0000 7f8e9f200640 1 -- 172.21.15.6:0/1304136646 >> 172.21.15.6:0/1304136646 conn(0x7f8ea006c0c0 msgr2=0x7f8ea0072740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:28.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.498+0000 7f8e9f200640 1 -- 172.21.15.6:0/1304136646 shutdown_connections 2024-03-04T09:31:28.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.498+0000 7f8e9f200640 1 -- 172.21.15.6:0/1304136646 wait complete. 2024-03-04T09:31:28.511 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-04T09:31:28.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.718+0000 7f985cc00640 1 Processor -- start 2024-03-04T09:31:28.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.719+0000 7f985cc00640 1 -- start start 2024-03-04T09:31:28.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.720+0000 7f985cc00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f98580af1a0 0x7f985815c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:28.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.720+0000 7f985cc00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f98580cdc20 0x7f98580b2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:28.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.720+0000 7f985cc00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f985815cc40 con 0x7f98580af580 2024-03-04T09:31:28.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.720+0000 7f985cc00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f985815cf20 con 0x7f98580af1a0 2024-03-04T09:31:28.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.720+0000 7f985cc00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f985815d200 con 0x7f98580cdc20 2024-03-04T09:31:28.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.721+0000 7f984fe00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f98580af580 0x7f98580b4c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:57008/0 (socket says 172.21.15.6:57008) 2024-03-04T09:31:28.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.721+0000 7f984fe00640 1 -- 172.21.15.6:0/3120502277 learned_addr learned my addr 172.21.15.6:0/3120502277 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:31:28.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.721+0000 7f9855e00640 1 --2- 172.21.15.6:0/3120502277 >> v2:172.21.15.6:3300/0 conn(0x7f98580af1a0 0x7f985815c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:28.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.721+0000 7f9856800640 1 --2- 172.21.15.6:0/3120502277 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f98580cdc20 0x7f98580b2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:28.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.721+0000 7f9855400640 1 -- 172.21.15.6:0/3120502277 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1274621012 0 0) 0x7f985815cc40 con 0x7f98580af580 2024-03-04T09:31:28.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.721+0000 7f9855e00640 1 -- 172.21.15.6:0/3120502277 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f98580cdc20 msgr2=0x7f98580b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:28.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.721+0000 7f9855e00640 1 --2- 172.21.15.6:0/3120502277 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f98580cdc20 0x7f98580b2dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:28.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.721+0000 7f9855e00640 1 -- 172.21.15.6:0/3120502277 >> v1:172.21.15.6:6789/0 conn(0x7f98580af580 legacy=0x7f98580b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:28.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.721+0000 7f9855e00640 1 -- 172.21.15.6:0/3120502277 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f985815d4e0 con 0x7f98580af1a0 2024-03-04T09:31:28.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.722+0000 7f9855e00640 1 --2- 172.21.15.6:0/3120502277 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f98580af1a0 0x7f985815c600 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7f98480097a0 tx=0x7f984802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74f16064521b0ac0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:28.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.723+0000 7f9855400640 1 -- 172.21.15.6:0/3120502277 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98480315a0 con 0x7f98580af1a0 2024-03-04T09:31:28.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.723+0000 7f9855400640 1 -- 172.21.15.6:0/3120502277 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9848037070 con 0x7f98580af1a0 2024-03-04T09:31:28.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.723+0000 7f9855400640 1 -- 172.21.15.6:0/3120502277 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9848031960 con 0x7f98580af1a0 2024-03-04T09:31:28.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.723+0000 7f985cc00640 1 -- 172.21.15.6:0/3120502277 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f98580af1a0 msgr2=0x7f985815c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:28.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.723+0000 7f985cc00640 1 --2- 172.21.15.6:0/3120502277 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f98580af1a0 0x7f985815c600 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7f98480097a0 tx=0x7f984802b840 comp rx=0 tx=0).stop 2024-03-04T09:31:28.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.723+0000 7f985cc00640 1 -- 172.21.15.6:0/3120502277 shutdown_connections 2024-03-04T09:31:28.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.723+0000 7f985cc00640 1 --2- 172.21.15.6:0/3120502277 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f98580cdc20 0x7f98580b2dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:28.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.724+0000 7f985cc00640 1 --2- 172.21.15.6:0/3120502277 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f98580af1a0 0x7f985815c600 unknown :-1 s=CLOSED pgs=1269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:28.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.724+0000 7f985cc00640 1 -- 172.21.15.6:0/3120502277 >> 172.21.15.6:0/3120502277 conn(0x7f985806afb0 msgr2=0x7f985806b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:28.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.724+0000 7f985cc00640 1 -- 172.21.15.6:0/3120502277 shutdown_connections 2024-03-04T09:31:28.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.724+0000 7f985cc00640 1 -- 172.21.15.6:0/3120502277 wait complete. 2024-03-04T09:31:28.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.725+0000 7f985cc00640 1 Processor -- start 2024-03-04T09:31:28.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.726+0000 7f985cc00640 1 -- start start 2024-03-04T09:31:28.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.726+0000 7f985cc00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f98580af1a0 0x7f985814a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:28.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.726+0000 7f985cc00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f98580af580 0x7f985814b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:28.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.726+0000 7f985cc00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f98580cdc20 0x7f98580ce340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:28.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.726+0000 7f985cc00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98580b36e0 con 0x7f98580cdc20 2024-03-04T09:31:28.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.726+0000 7f985cc00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98580b3560 con 0x7f98580af580 2024-03-04T09:31:28.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.726+0000 7f985cc00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9858155e70 con 0x7f98580af1a0 2024-03-04T09:31:28.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.727+0000 7f9856800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f98580cdc20 0x7f98580ce340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:28.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.727+0000 7f9856800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f98580cdc20 0x7f98580ce340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:32930/0 (socket says 172.21.15.6:32930) 2024-03-04T09:31:28.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.727+0000 7f9856800640 1 -- 172.21.15.6:0/2911249009 learned_addr learned my addr 172.21.15.6:0/2911249009 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:31:28.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.727+0000 7f9855e00640 1 --2- 172.21.15.6:0/2911249009 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f98580af1a0 0x7f985814a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:28.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.727+0000 7f984fe00640 1 --2- 172.21.15.6:0/2911249009 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f98580af580 0x7f985814b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:28.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.727+0000 7f9856800640 1 -- 172.21.15.6:0/2911249009 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f98580af1a0 msgr2=0x7f985814a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:28.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.727+0000 7f9856800640 1 --2- 172.21.15.6:0/2911249009 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f98580af1a0 0x7f985814a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:28.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.727+0000 7f9856800640 1 -- 172.21.15.6:0/2911249009 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f98580af580 msgr2=0x7f985814b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:28.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.727+0000 7f9856800640 1 --2- 172.21.15.6:0/2911249009 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f98580af580 0x7f985814b430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:28.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.727+0000 7f9856800640 1 -- 172.21.15.6:0/2911249009 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f985814ea20 con 0x7f98580cdc20 2024-03-04T09:31:28.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.727+0000 7f984fe00640 1 --2- 172.21.15.6:0/2911249009 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f98580af580 0x7f985814b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:31:28.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.727+0000 7f9855e00640 1 --2- 172.21.15.6:0/2911249009 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f98580af1a0 0x7f985814a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:31:28.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.728+0000 7f9856800640 1 --2- 172.21.15.6:0/2911249009 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f98580cdc20 0x7f98580ce340 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7f9850007e60 tx=0x7f985000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:28.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.728+0000 7f984e000640 1 -- 172.21.15.6:0/2911249009 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9850014420 con 0x7f98580cdc20 2024-03-04T09:31:28.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.728+0000 7f984e000640 1 -- 172.21.15.6:0/2911249009 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9850019070 con 0x7f98580cdc20 2024-03-04T09:31:28.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.728+0000 7f984e000640 1 -- 172.21.15.6:0/2911249009 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9850014820 con 0x7f98580cdc20 2024-03-04T09:31:28.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.728+0000 7f985cc00640 1 -- 172.21.15.6:0/2911249009 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f985814ed00 con 0x7f98580cdc20 2024-03-04T09:31:28.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.728+0000 7f985cc00640 1 -- 172.21.15.6:0/2911249009 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f985814f2c0 con 0x7f98580cdc20 2024-03-04T09:31:28.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.729+0000 7f985cc00640 1 -- 172.21.15.6:0/2911249009 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9858069510 con 0x7f98580cdc20 2024-03-04T09:31:28.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.730+0000 7f984e000640 1 -- 172.21.15.6:0/2911249009 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f98500149c0 con 0x7f98580cdc20 2024-03-04T09:31:28.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.732+0000 7f984e000640 1 --2- 172.21.15.6:0/2911249009 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f983406b500 0x7f983406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:28.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.732+0000 7f984e000640 1 -- 172.21.15.6:0/2911249009 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f985008aa20 con 0x7f98580cdc20 2024-03-04T09:31:28.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.732+0000 7f9855e00640 1 --2- 172.21.15.6:0/2911249009 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f983406b500 0x7f983406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:28.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.734+0000 7f9855e00640 1 --2- 172.21.15.6:0/2911249009 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f983406b500 0x7f983406d9c0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f9848036000 tx=0x7f9848008030 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:28.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.736+0000 7f984e000640 1 -- 172.21.15.6:0/2911249009 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9850059890 con 0x7f98580cdc20 2024-03-04T09:31:28.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.976+0000 7f985cc00640 1 -- 172.21.15.6:0/2911249009 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f9858071d80 con 0x7f98580cdc20 2024-03-04T09:31:28.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.976+0000 7f984e000640 1 -- 172.21.15.6:0/2911249009 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v77) v1 ==== 53+0+73 (secure 0 0 0) 0x7f9850059190 con 0x7f98580cdc20 2024-03-04T09:31:28.973 INFO:teuthology.orchestra.run.smithi006.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-04T09:31:28.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.979+0000 7f985cc00640 1 -- 172.21.15.6:0/2911249009 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f983406b500 msgr2=0x7f983406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:28.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.979+0000 7f985cc00640 1 --2- 172.21.15.6:0/2911249009 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f983406b500 0x7f983406d9c0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f9848036000 tx=0x7f9848008030 comp rx=0 tx=0).stop 2024-03-04T09:31:28.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.979+0000 7f985cc00640 1 -- 172.21.15.6:0/2911249009 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f98580cdc20 msgr2=0x7f98580ce340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:28.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.979+0000 7f985cc00640 1 --2- 172.21.15.6:0/2911249009 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f98580cdc20 0x7f98580ce340 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7f9850007e60 tx=0x7f985000e4b0 comp rx=0 tx=0).stop 2024-03-04T09:31:28.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.979+0000 7f985cc00640 1 -- 172.21.15.6:0/2911249009 shutdown_connections 2024-03-04T09:31:28.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.979+0000 7f985cc00640 1 --2- 172.21.15.6:0/2911249009 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f98580cdc20 0x7f98580ce340 unknown :-1 s=CLOSED pgs=1270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:28.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.979+0000 7f985cc00640 1 --2- 172.21.15.6:0/2911249009 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f98580af580 0x7f985814b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:28.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.979+0000 7f985cc00640 1 --2- 172.21.15.6:0/2911249009 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f983406b500 0x7f983406d9c0 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:28.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.979+0000 7f985cc00640 1 --2- 172.21.15.6:0/2911249009 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f98580af1a0 0x7f985814a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:28.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.979+0000 7f985cc00640 1 -- 172.21.15.6:0/2911249009 >> 172.21.15.6:0/2911249009 conn(0x7f985806afb0 msgr2=0x7f985806b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:28.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.979+0000 7f985cc00640 1 -- 172.21.15.6:0/2911249009 shutdown_connections 2024-03-04T09:31:28.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:28.979+0000 7f985cc00640 1 -- 172.21.15.6:0/2911249009 wait complete. 2024-03-04T09:31:28.995 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-04T09:31:28.995 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-04T09:31:28.995 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi006.front.sepia.ceph.com 2024-03-04T09:31:28.995 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-04T09:31:28.996 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-04T09:31:28.996 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-04T09:31:28.996 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-03-04T09:31:28.996 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-04T09:31:28.996 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-04T09:31:28.996 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:31:28.996 DEBUG:teuthology.orchestra.run.smithi006:> ip addr 2024-03-04T09:31:29.043 INFO:teuthology.orchestra.run.smithi006.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-04T09:31:29.043 INFO:teuthology.orchestra.run.smithi006.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-04T09:31:29.044 INFO:teuthology.orchestra.run.smithi006.stdout: inet 127.0.0.1/8 scope host lo 2024-03-04T09:31:29.044 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-04T09:31:29.044 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 ::1/128 scope host 2024-03-04T09:31:29.044 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-04T09:31:29.044 INFO:teuthology.orchestra.run.smithi006.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-04T09:31:29.044 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:6b:fe:e6 brd ff:ff:ff:ff:ff:ff 2024-03-04T09:31:29.044 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp7s0f0 2024-03-04T09:31:29.044 INFO:teuthology.orchestra.run.smithi006.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-04T09:31:29.044 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:6b:fe:e7 brd ff:ff:ff:ff:ff:ff 2024-03-04T09:31:29.044 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp7s0f1 2024-03-04T09:31:29.044 INFO:teuthology.orchestra.run.smithi006.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-04T09:31:29.044 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:bd:15:e4 brd ff:ff:ff:ff:ff:ff 2024-03-04T09:31:29.044 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp1s0f0 2024-03-04T09:31:29.045 INFO:teuthology.orchestra.run.smithi006.stdout: inet 172.21.15.6/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-04T09:31:29.045 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft 42245sec preferred_lft 42245sec 2024-03-04T09:31:29.045 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::ec4:7aff:febd:15e4/64 scope link noprefixroute 2024-03-04T09:31:29.045 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-04T09:31:29.045 INFO:teuthology.orchestra.run.smithi006.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-04T09:31:29.045 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:bd:15:e5 brd ff:ff:ff:ff:ff:ff 2024-03-04T09:31:29.045 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp1s0f1 2024-03-04T09:31:29.045 INFO:teuthology.orchestra.run.smithi006.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-04T09:31:29.045 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 5e:b4:59:dc:42:58 brd ff:ff:ff:ff:ff:ff 2024-03-04T09:31:29.045 INFO:teuthology.orchestra.run.smithi006.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-04T09:31:29.045 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-04T09:31:29.045 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::c6f:adff:fefe:e00d/64 scope link proto kernel_ll 2024-03-04T09:31:29.045 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-04T09:31:29.045 INFO:teuthology.orchestra.run.smithi006.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-04T09:31:29.046 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 7e:67:6d:ea:f9:ff brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-04T09:31:29.046 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::7c67:6dff:feea:f9ff/64 scope link proto kernel_ll 2024-03-04T09:31:29.046 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-04T09:31:29.046 INFO:teuthology.orchestra.run.smithi006.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-04T09:31:29.046 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 5e:b4:59:dc:42:58 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-04T09:31:29.046 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::5cb4:59ff:fedc:4258/64 scope link proto kernel_ll 2024-03-04T09:31:29.046 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-04T09:31:29.047 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:31:29.047 DEBUG:teuthology.orchestra.run.smithi006:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-04T09:31:29.177 INFO:teuthology.orchestra.run.smithi006.stdout:1 2024-03-04T09:31:29.182 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:31:29.182 DEBUG:teuthology.orchestra.run.smithi006:> route 2024-03-04T09:31:29.229 INFO:teuthology.orchestra.run.smithi006.stdout:Kernel IP routing table 2024-03-04T09:31:29.229 INFO:teuthology.orchestra.run.smithi006.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-04T09:31:29.229 INFO:teuthology.orchestra.run.smithi006.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-04T09:31:29.230 INFO:teuthology.orchestra.run.smithi006.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-04T09:31:29.230 INFO:teuthology.orchestra.run.smithi006.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-04T09:31:29.230 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:31:29.230 DEBUG:teuthology.orchestra.run.smithi006:> (cd / && exec bash -c ' 2024-03-04T09:31:29.230 DEBUG:teuthology.orchestra.run.smithi006:> set -e 2024-03-04T09:31:29.230 DEBUG:teuthology.orchestra.run.smithi006:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-04T09:31:29.230 DEBUG:teuthology.orchestra.run.smithi006:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-04T09:31:29.230 DEBUG:teuthology.orchestra.run.smithi006:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-04T09:31:29.230 DEBUG:teuthology.orchestra.run.smithi006:> ') 2024-03-04T09:31:29.418 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:31:29.419 DEBUG:teuthology.orchestra.run.smithi006:> ip netns list 2024-03-04T09:31:29.471 INFO:teuthology.orchestra.run.smithi006.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-04T09:31:29.472 INFO:teuthology.orchestra.run.smithi006.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-04T09:31:29.472 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-04T09:31:29.472 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:31:29.472 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip link set brx.0 up 2024-03-04T09:31:29.571 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-04T09:31:29.571 DEBUG:teuthology.orchestra.run.smithi006:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-04T09:31:29.619 INFO:teuthology.orchestra.run.smithi006.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-04T09:31:29.620 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-04T09:31:29.620 DEBUG:teuthology.orchestra.run.smithi006:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-04T09:31:29.675 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-03-04T09:31:29.676 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:31:29.676 DEBUG:teuthology.orchestra.run.smithi006:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-04T09:31:29.812 INFO:teuthology.orchestra.run.smithi006.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-04T09:31:29.812 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-04T09:31:29.846 INFO:teuthology.orchestra.run.smithi006.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-04T09:31:29.847 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-04T09:31:29.847 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: resolved to: "172.21.15.6:3300,172.21.15.146:3300,172.21.15.146:3301" 2024-03-04T09:31:29.847 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: trying mount with new device syntax: 0@edce0578-5ae4-4402-b45a-552bdef84d7c.cephfs=/ 2024-03-04T09:31:29.847 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.6:3300/172.21.15.146:3300/172.21.15.146:3301" will pass to kernel 2024-03-04T09:31:29.849 INFO:teuthology.orchestra.run.smithi006.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-04T09:31:29.849 INFO:teuthology.orchestra.run.smithi006.stdout: You just mounted a file system that supports labels which does not 2024-03-04T09:31:29.850 INFO:teuthology.orchestra.run.smithi006.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-04T09:31:29.850 INFO:teuthology.orchestra.run.smithi006.stdout: applications will generate AVC messages and not be allowed access to 2024-03-04T09:31:29.850 INFO:teuthology.orchestra.run.smithi006.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-04T09:31:29.854 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-04T09:31:29.854 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:31:29.854 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-04T09:31:29.933 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:31:29.933 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-04T09:31:29.995 INFO:teuthology.orchestra.run.smithi006.stdout:client5861 2024-03-04T09:31:29.995 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-04T09:31:29.995 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T09:31:30.039 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:31:30.039 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir dir_1) 2024-03-04T09:31:30.140 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-04T09:31:30.141 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T09:31:30.187 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:31:30.188 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=dir_1/file_a bs=1M conv=fdatasync count=8 seek=0) 2024-03-04T09:31:30.450 INFO:teuthology.orchestra.run.smithi006.stderr:8+0 records in 2024-03-04T09:31:30.451 INFO:teuthology.orchestra.run.smithi006.stderr:8+0 records out 2024-03-04T09:31:30.451 INFO:teuthology.orchestra.run.smithi006.stderr:8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.165169 s, 50.8 MB/s 2024-03-04T09:31:30.453 DEBUG:teuthology.orchestra.run.smithi006:> stdin-killer -- python3 -c ' 2024-03-04T09:31:30.453 DEBUG:teuthology.orchestra.run.smithi006:> import os 2024-03-04T09:31:30.453 DEBUG:teuthology.orchestra.run.smithi006:> import stat 2024-03-04T09:31:30.453 DEBUG:teuthology.orchestra.run.smithi006:> 2024-03-04T09:31:30.453 DEBUG:teuthology.orchestra.run.smithi006:> print(os.stat("/home/ubuntu/cephtest/mnt.0/dir_1/file_a").st_ino) 2024-03-04T09:31:30.453 DEBUG:teuthology.orchestra.run.smithi006:> ' 2024-03-04T09:31:30.596 INFO:teuthology.orchestra.run.smithi006.stdout:1099511627777 2024-03-04T09:31:30.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:30 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-04T09:31:30.610 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:31:30.610 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir dir_2) 2024-03-04T09:31:30.711 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:31:30.711 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest/mnt.0 && exec ln dir_1/file_a dir_2/file_b) 2024-03-04T09:31:30.768 DEBUG:teuthology.orchestra.run.smithi006:> stdin-killer -- python3 -c ' 2024-03-04T09:31:30.768 DEBUG:teuthology.orchestra.run.smithi006:> import os 2024-03-04T09:31:30.768 DEBUG:teuthology.orchestra.run.smithi006:> import stat 2024-03-04T09:31:30.768 DEBUG:teuthology.orchestra.run.smithi006:> 2024-03-04T09:31:30.768 DEBUG:teuthology.orchestra.run.smithi006:> print(os.stat("/home/ubuntu/cephtest/mnt.0/dir_2/file_b").st_ino) 2024-03-04T09:31:30.768 DEBUG:teuthology.orchestra.run.smithi006:> ' 2024-03-04T09:31:30.890 INFO:teuthology.orchestra.run.smithi006.stdout:1099511627777 2024-03-04T09:31:30.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:30 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-04T09:31:30.903 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:31:31.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.163+0000 7f7fefe00640 1 Processor -- start 2024-03-04T09:31:31.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.163+0000 7f7fefe00640 1 -- start start 2024-03-04T09:31:31.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.165+0000 7f7fefe00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f7ff00d0e50 0x7f7ff0168c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:31.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.165+0000 7f7fefe00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ff00b3710 con 0x7f7ff015e0b0 2024-03-04T09:31:31.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.165+0000 7f7fefe00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ff00b39f0 con 0x7f7ff00d4a10 2024-03-04T09:31:31.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.165+0000 7f7fefe00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ff00b09e0 con 0x7f7ff00d0e50 2024-03-04T09:31:31.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.165+0000 7f7feea00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f7ff015e0b0 0x7f7ff00cf310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:52692/0 (socket says 172.21.15.6:52692) 2024-03-04T09:31:31.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.165+0000 7f7feea00640 1 -- 172.21.15.6:0/116574469 learned_addr learned my addr 172.21.15.6:0/116574469 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:31:31.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.165+0000 7f7fed600640 1 -- 172.21.15.6:0/116574469 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3289171325 0 0) 0x7f7ff00b3710 con 0x7f7ff015e0b0 2024-03-04T09:31:31.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.165+0000 7f7fed600640 1 -- 172.21.15.6:0/116574469 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fd4003520 con 0x7f7ff015e0b0 2024-03-04T09:31:31.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.166+0000 7f7fef400640 1 --2- 172.21.15.6:0/116574469 >> v2:172.21.15.146:3300/0 conn(0x7f7ff00d0e50 0x7f7ff0168c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:31.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.166+0000 7f7fed600640 1 -- 172.21.15.6:0/116574469 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4230397480 0 0) 0x7f7fd4003520 con 0x7f7ff015e0b0 2024-03-04T09:31:31.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.166+0000 7f7fed600640 1 -- 172.21.15.6:0/116574469 >> v2:172.21.15.146:3300/0 conn(0x7f7ff00d0e50 msgr2=0x7f7ff0168c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:31.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.166+0000 7f7fed600640 1 --2- 172.21.15.6:0/116574469 >> v2:172.21.15.146:3300/0 conn(0x7f7ff00d0e50 0x7f7ff0168c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:31.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.166+0000 7f7fed600640 1 -- 172.21.15.6:0/116574469 >> v1:172.21.15.146:6789/0 conn(0x7f7ff00d4a10 legacy=0x7f7ff00d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:31.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.166+0000 7f7fed600640 1 -- 172.21.15.6:0/116574469 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ff00b3cd0 con 0x7f7ff015e0b0 2024-03-04T09:31:31.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.166+0000 7f7fed600640 1 -- 172.21.15.6:0/116574469 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7f7fe0001670 con 0x7f7ff015e0b0 2024-03-04T09:31:31.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.166+0000 7f7fed600640 1 -- 172.21.15.6:0/116574469 >> v1:172.21.15.6:6789/0 conn(0x7f7ff015e0b0 legacy=0x7f7ff00cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:31.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.166+0000 7f7fed600640 1 --2- 172.21.15.6:0/116574469 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7fd40039b0 0x7f7fd4003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:31.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.166+0000 7f7fed600640 1 --2- 172.21.15.6:0/116574469 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7fd4004800 0x7f7fd4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:31.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.166+0000 7f7fed600640 1 --2- 172.21.15.6:0/116574469 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7fd40052e0 0x7f7fd400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:31.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.166+0000 7f7fed600640 1 -- 172.21.15.6:0/116574469 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fd400c1b0 con 0x7f7fd40052e0 2024-03-04T09:31:31.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.166+0000 7f7fed600640 1 -- 172.21.15.6:0/116574469 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fd400c360 con 0x7f7fd4004800 2024-03-04T09:31:31.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.166+0000 7f7fed600640 1 -- 172.21.15.6:0/116574469 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fd400c560 con 0x7f7fd40039b0 2024-03-04T09:31:31.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.166+0000 7f7fef400640 1 --2- 172.21.15.6:0/116574469 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7fd40052e0 0x7f7fd400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:31.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.167+0000 7f7fee000640 1 --2- 172.21.15.6:0/116574469 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7fd4004800 0x7f7fd4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:31.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.167+0000 7f7fef400640 1 -- 172.21.15.6:0/116574469 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7fd40039b0 msgr2=0x7f7fd4003db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:31.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.167+0000 7f7feea00640 1 --2- 172.21.15.6:0/116574469 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7fd40039b0 0x7f7fd4003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:31.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.167+0000 7f7fef400640 1 --2- 172.21.15.6:0/116574469 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7fd40039b0 0x7f7fd4003db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:31.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.167+0000 7f7fef400640 1 -- 172.21.15.6:0/116574469 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7fd4004800 msgr2=0x7f7fd4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:31.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.167+0000 7f7fef400640 1 --2- 172.21.15.6:0/116574469 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7fd4004800 0x7f7fd4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:31.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.167+0000 7f7fef400640 1 -- 172.21.15.6:0/116574469 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fd4003520 con 0x7f7fd40052e0 2024-03-04T09:31:31.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.167+0000 7f7fee000640 1 --2- 172.21.15.6:0/116574469 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7fd4004800 0x7f7fd4004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:31:31.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.168+0000 7f7fef400640 1 -- 172.21.15.6:0/116574469 reap_dead start 2024-03-04T09:31:31.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.168+0000 7f7fef400640 1 --2- 172.21.15.6:0/116574469 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7fd40052e0 0x7f7fd400b9c0 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7f7ff00cadb0 tx=0x7f7fd4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0ead1714d1061f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:31.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.168+0000 7f7fed600640 1 -- 172.21.15.6:0/116574469 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fe8007b30 con 0x7f7fd40052e0 2024-03-04T09:31:31.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.168+0000 7f7fed600640 1 -- 172.21.15.6:0/116574469 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fe8033600 con 0x7f7fd40052e0 2024-03-04T09:31:31.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.168+0000 7f7fed600640 1 -- 172.21.15.6:0/116574469 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fe80339d0 con 0x7f7fd40052e0 2024-03-04T09:31:31.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.169+0000 7f7fefe00640 1 -- 172.21.15.6:0/116574469 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7fd40052e0 msgr2=0x7f7fd400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:31.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.169+0000 7f7fefe00640 1 --2- 172.21.15.6:0/116574469 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7fd40052e0 0x7f7fd400b9c0 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7f7ff00cadb0 tx=0x7f7fd4005830 comp rx=0 tx=0).stop 2024-03-04T09:31:31.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.169+0000 7f7fefe00640 1 -- 172.21.15.6:0/116574469 shutdown_connections 2024-03-04T09:31:31.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.169+0000 7f7fefe00640 1 --2- 172.21.15.6:0/116574469 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7fd40052e0 0x7f7fd400b9c0 unknown :-1 s=CLOSED pgs=1272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:31.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.169+0000 7f7fefe00640 1 -- 172.21.15.6:0/116574469 >> 172.21.15.6:0/116574469 conn(0x7f7ff00d3d30 msgr2=0x7f7ff00d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:31.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.169+0000 7f7fefe00640 1 -- 172.21.15.6:0/116574469 shutdown_connections 2024-03-04T09:31:31.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.169+0000 7f7fefe00640 1 -- 172.21.15.6:0/116574469 wait complete. 2024-03-04T09:31:31.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.170+0000 7f7fefe00640 1 Processor -- start 2024-03-04T09:31:31.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.171+0000 7f7fefe00640 1 -- start start 2024-03-04T09:31:31.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.171+0000 7f7fefe00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7fd40052e0 0x7f7ff0161080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:31.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.171+0000 7f7fefe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7ff015fae0 0x7f7ff00cbe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:31.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.172+0000 7f7fefe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7ff00cc3b0 0x7f7ff00a0300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:31.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.172+0000 7f7fefe00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ff00cfc60 con 0x7f7ff00cc3b0 2024-03-04T09:31:31.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.172+0000 7f7fefe00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ff015e490 con 0x7f7ff015fae0 2024-03-04T09:31:31.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.172+0000 7f7fefe00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ff00d0740 con 0x7f7fd40052e0 2024-03-04T09:31:31.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.172+0000 7f7fef400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7ff00cc3b0 0x7f7ff00a0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:31.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.172+0000 7f7fef400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7ff00cc3b0 0x7f7ff00a0300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49888/0 (socket says 172.21.15.6:49888) 2024-03-04T09:31:31.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.172+0000 7f7fef400640 1 -- 172.21.15.6:0/1401032703 learned_addr learned my addr 172.21.15.6:0/1401032703 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:31:31.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.172+0000 7f7fee000640 1 --2- 172.21.15.6:0/1401032703 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7ff015fae0 0x7f7ff00cbe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:31.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.172+0000 7f7feea00640 1 --2- 172.21.15.6:0/1401032703 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7fd40052e0 0x7f7ff0161080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:31.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.172+0000 7f7fef400640 1 -- 172.21.15.6:0/1401032703 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7fd40052e0 msgr2=0x7f7ff0161080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:31.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.172+0000 7f7fef400640 1 --2- 172.21.15.6:0/1401032703 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7fd40052e0 0x7f7ff0161080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:31.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.172+0000 7f7fef400640 1 -- 172.21.15.6:0/1401032703 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7ff015fae0 msgr2=0x7f7ff00cbe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:31.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.172+0000 7f7fef400640 1 --2- 172.21.15.6:0/1401032703 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7ff015fae0 0x7f7ff00cbe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:31.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.172+0000 7f7fef400640 1 -- 172.21.15.6:0/1401032703 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ff015ffe0 con 0x7f7ff00cc3b0 2024-03-04T09:31:31.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.172+0000 7f7feea00640 1 --2- 172.21.15.6:0/1401032703 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7fd40052e0 0x7f7ff0161080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:31:31.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.173+0000 7f7fef400640 1 --2- 172.21.15.6:0/1401032703 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7ff00cc3b0 0x7f7ff00a0300 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7f7fe802cae0 tx=0x7f7fe8004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:31.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.173+0000 7f7fe6a00640 1 -- 172.21.15.6:0/1401032703 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fe8042030 con 0x7f7ff00cc3b0 2024-03-04T09:31:31.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.173+0000 7f7fe6a00640 1 -- 172.21.15.6:0/1401032703 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fe8033400 con 0x7f7ff00cc3b0 2024-03-04T09:31:31.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.173+0000 7f7fe6a00640 1 -- 172.21.15.6:0/1401032703 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fe8007410 con 0x7f7ff00cc3b0 2024-03-04T09:31:31.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.173+0000 7f7fefe00640 1 -- 172.21.15.6:0/1401032703 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ff01602c0 con 0x7f7ff00cc3b0 2024-03-04T09:31:31.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.173+0000 7f7fefe00640 1 -- 172.21.15.6:0/1401032703 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ff0160880 con 0x7f7ff00cc3b0 2024-03-04T09:31:31.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.174+0000 7f7fefe00640 1 -- 172.21.15.6:0/1401032703 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ff00d0ad0 con 0x7f7ff00cc3b0 2024-03-04T09:31:31.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.175+0000 7f7fe6a00640 1 -- 172.21.15.6:0/1401032703 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7fe8045030 con 0x7f7ff00cc3b0 2024-03-04T09:31:31.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.176+0000 7f7fe6a00640 1 --2- 172.21.15.6:0/1401032703 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f7fd006b500 0x7f7fd006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:31.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.176+0000 7f7fe6a00640 1 -- 172.21.15.6:0/1401032703 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7fe80accc0 con 0x7f7ff00cc3b0 2024-03-04T09:31:31.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.177+0000 7f7feea00640 1 --2- 172.21.15.6:0/1401032703 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f7fd006b500 0x7f7fd006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:31.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.178+0000 7f7feea00640 1 --2- 172.21.15.6:0/1401032703 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f7fd006b500 0x7f7fd006d9c0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f7fe0001eb0 tx=0x7f7fe0009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:31.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.186+0000 7f7fe6a00640 1 -- 172.21.15.6:0/1401032703 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7ff00d0ad0 con 0x7f7ff00cc3b0 2024-03-04T09:31:31.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.430+0000 7f7fefe00640 1 -- 172.21.15.6:0/1401032703 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7ff0074070 con 0x7f7ff00cc3b0 2024-03-04T09:31:31.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.431+0000 7f7fe6a00640 1 -- 172.21.15.6:0/1401032703 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+4973 (secure 0 0 0) 0x7f7fe80b3080 con 0x7f7ff00cc3b0 2024-03-04T09:31:31.428 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:31:31.428 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":77,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5792,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6836/4034557380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":4034557380},{"type":"v1","addr":"172.21.15.146:6836","nonce":4034557380}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":72},{"gid":6370,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/348873623","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":348873623},{"type":"v1","addr":"172.21.15.6:6835","nonce":348873623}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":72},{"gid":6382,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/763528142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":763528142},{"type":"v1","addr":"172.21.15.146:6837","nonce":763528142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":72}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:31:21.114351+0000","modified":"2024-03-04T09:31:24.172076+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7560},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7560":{"gid":7560,"name":"a","rank":0,"incarnation":73,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2697931210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2697931210},{"type":"v1","addr":"172.21.15.6:6834","nonce":2697931210}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7560,"qdb_cluster":[7560]},"id":8}]} 2024-03-04T09:31:31.428 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 77 2024-03-04T09:31:31.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.434+0000 7f7fefe00640 1 -- 172.21.15.6:0/1401032703 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f7fd006b500 msgr2=0x7f7fd006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:31.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.434+0000 7f7fefe00640 1 --2- 172.21.15.6:0/1401032703 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f7fd006b500 0x7f7fd006d9c0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f7fe0001eb0 tx=0x7f7fe0009040 comp rx=0 tx=0).stop 2024-03-04T09:31:31.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.434+0000 7f7fefe00640 1 -- 172.21.15.6:0/1401032703 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7ff00cc3b0 msgr2=0x7f7ff00a0300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:31.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.434+0000 7f7fefe00640 1 --2- 172.21.15.6:0/1401032703 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7ff00cc3b0 0x7f7ff00a0300 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7f7fe802cae0 tx=0x7f7fe8004ab0 comp rx=0 tx=0).stop 2024-03-04T09:31:31.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.434+0000 7f7fefe00640 1 -- 172.21.15.6:0/1401032703 shutdown_connections 2024-03-04T09:31:31.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.434+0000 7f7fefe00640 1 --2- 172.21.15.6:0/1401032703 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7ff00cc3b0 0x7f7ff00a0300 unknown :-1 s=CLOSED pgs=1273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:31.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.434+0000 7f7fefe00640 1 --2- 172.21.15.6:0/1401032703 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7ff015fae0 0x7f7ff00cbe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:31.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.434+0000 7f7fefe00640 1 --2- 172.21.15.6:0/1401032703 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f7fd006b500 0x7f7fd006d9c0 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:31.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.434+0000 7f7fefe00640 1 --2- 172.21.15.6:0/1401032703 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7fd40052e0 0x7f7ff0161080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:31.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.434+0000 7f7fefe00640 1 -- 172.21.15.6:0/1401032703 >> 172.21.15.6:0/1401032703 conn(0x7f7ff00d3d30 msgr2=0x7f7ff015d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:31.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.434+0000 7f7fefe00640 1 -- 172.21.15.6:0/1401032703 shutdown_connections 2024-03-04T09:31:31.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.434+0000 7f7fefe00640 1 -- 172.21.15.6:0/1401032703 wait complete. 2024-03-04T09:31:31.458 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json flush journal 2024-03-04T09:31:31.715 INFO:teuthology.orchestra.run.smithi006.stdout:{"message":"","return_code":0} 2024-03-04T09:31:31.715 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-04T09:31:31.715 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 10000000001.00000000 parent 2024-03-04T09:31:31.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.809+0000 7fcf53039f40 1 Processor -- start 2024-03-04T09:31:31.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.809+0000 7fcf53039f40 1 -- start start 2024-03-04T09:31:31.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.810+0000 7fcf53039f40 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x562556c12fd0 0x562556be9540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:31.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.810+0000 7fcf53039f40 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562556c2ad80 0x562556cbbed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:31.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.810+0000 7fcf53039f40 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x562556cbd370 0x562556cbc410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:31.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.810+0000 7fcf53039f40 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x562556c2dea0 con 0x562556c2ad80 2024-03-04T09:31:31.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.810+0000 7fcf53039f40 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x562556b8cfd0 con 0x562556cbd370 2024-03-04T09:31:31.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.810+0000 7fcf53039f40 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x562556c2e020 con 0x562556c12fd0 2024-03-04T09:31:31.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.810+0000 7fcf4f000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562556c2ad80 0x562556cbbed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:31.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.810+0000 7fcf4f000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562556c2ad80 0x562556cbbed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49906/0 (socket says 172.21.15.6:49906) 2024-03-04T09:31:31.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.810+0000 7fcf4f000640 1 -- 172.21.15.6:0/2747155784 learned_addr learned my addr 172.21.15.6:0/2747155784 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:31:31.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.811+0000 7fcf4fa00640 1 --2- 172.21.15.6:0/2747155784 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x562556c12fd0 0x562556be9540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:31.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.811+0000 7fcf50400640 1 --2- 172.21.15.6:0/2747155784 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x562556cbd370 0x562556cbc410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:31.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.811+0000 7fcf4f000640 1 -- 172.21.15.6:0/2747155784 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x562556c12fd0 msgr2=0x562556be9540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:31.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.811+0000 7fcf4f000640 1 --2- 172.21.15.6:0/2747155784 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x562556c12fd0 0x562556be9540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:31.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.811+0000 7fcf4f000640 1 -- 172.21.15.6:0/2747155784 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x562556cbd370 msgr2=0x562556cbc410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:31.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.811+0000 7fcf4f000640 1 --2- 172.21.15.6:0/2747155784 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x562556cbd370 0x562556cbc410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:31.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.811+0000 7fcf4f000640 1 -- 172.21.15.6:0/2747155784 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562556c2f170 con 0x562556c2ad80 2024-03-04T09:31:31.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.811+0000 7fcf50400640 1 --2- 172.21.15.6:0/2747155784 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x562556cbd370 0x562556cbc410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:31:31.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.811+0000 7fcf4f000640 1 --2- 172.21.15.6:0/2747155784 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562556c2ad80 0x562556cbbed0 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7fcf3c00a770 tx=0x7fcf3c00a860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4831f4556e79b1de server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:31.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.812+0000 7fcf4dc00640 1 -- 172.21.15.6:0/2747155784 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf3c019070 con 0x562556c2ad80 2024-03-04T09:31:31.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.812+0000 7fcf4dc00640 1 -- 172.21.15.6:0/2747155784 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf3c007600 con 0x562556c2ad80 2024-03-04T09:31:31.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.812+0000 7fcf4dc00640 1 -- 172.21.15.6:0/2747155784 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf3c0079d0 con 0x562556c2ad80 2024-03-04T09:31:31.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.812+0000 7fcf53039f40 1 -- 172.21.15.6:0/2747155784 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562556c2ad80 msgr2=0x562556cbbed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:31.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.812+0000 7fcf53039f40 1 --2- 172.21.15.6:0/2747155784 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562556c2ad80 0x562556cbbed0 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7fcf3c00a770 tx=0x7fcf3c00a860 comp rx=0 tx=0).stop 2024-03-04T09:31:31.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.812+0000 7fcf53039f40 1 -- 172.21.15.6:0/2747155784 shutdown_connections 2024-03-04T09:31:31.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.812+0000 7fcf53039f40 1 --2- 172.21.15.6:0/2747155784 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x562556cbd370 0x562556cbc410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:31.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.812+0000 7fcf53039f40 1 --2- 172.21.15.6:0/2747155784 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562556c2ad80 0x562556cbbed0 unknown :-1 s=CLOSED pgs=1275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:31.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.812+0000 7fcf53039f40 1 --2- 172.21.15.6:0/2747155784 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x562556c12fd0 0x562556be9540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:31.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.812+0000 7fcf53039f40 1 -- 172.21.15.6:0/2747155784 >> 172.21.15.6:0/2747155784 conn(0x562556c11670 msgr2=0x562556c16800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:31.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.812+0000 7fcf53039f40 1 -- 172.21.15.6:0/2747155784 shutdown_connections 2024-03-04T09:31:31.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.812+0000 7fcf53039f40 1 -- 172.21.15.6:0/2747155784 wait complete. 2024-03-04T09:31:31.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.813+0000 7fcf53039f40 1 Processor -- start 2024-03-04T09:31:31.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.813+0000 7fcf53039f40 1 -- start start 2024-03-04T09:31:31.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.814+0000 7fcf53039f40 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562556c12fd0 0x562556cb5bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:31.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.814+0000 7fcf53039f40 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x562556c2ad80 0x562556c2b570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:31.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.814+0000 7fcf53039f40 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x562556cbd370 0x562556d56160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:31.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.814+0000 7fcf53039f40 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x562556cc7860 con 0x562556c12fd0 2024-03-04T09:31:31.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.814+0000 7fcf53039f40 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x562556cbdc40 con 0x562556c2ad80 2024-03-04T09:31:31.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.814+0000 7fcf53039f40 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x562556cb6110 con 0x562556cbd370 2024-03-04T09:31:31.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.815+0000 7fcf4fa00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562556c12fd0 0x562556cb5bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:31.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.815+0000 7fcf4fa00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562556c12fd0 0x562556cb5bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49922/0 (socket says 172.21.15.6:49922) 2024-03-04T09:31:31.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.815+0000 7fcf4fa00640 1 -- 172.21.15.6:0/428603285 learned_addr learned my addr 172.21.15.6:0/428603285 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:31:31.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.815+0000 7fcf50400640 1 --2- 172.21.15.6:0/428603285 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x562556cbd370 0x562556d56160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:31.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.815+0000 7fcf4f000640 1 --2- 172.21.15.6:0/428603285 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x562556c2ad80 0x562556c2b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:31.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.815+0000 7fcf4fa00640 1 -- 172.21.15.6:0/428603285 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x562556cbd370 msgr2=0x562556d56160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:31.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.815+0000 7fcf4fa00640 1 --2- 172.21.15.6:0/428603285 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x562556cbd370 0x562556d56160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:31.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.815+0000 7fcf4fa00640 1 -- 172.21.15.6:0/428603285 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x562556c2ad80 msgr2=0x562556c2b570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:31.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.815+0000 7fcf4fa00640 1 --2- 172.21.15.6:0/428603285 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x562556c2ad80 0x562556c2b570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:31.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.815+0000 7fcf4fa00640 1 -- 172.21.15.6:0/428603285 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562556cc7040 con 0x562556c12fd0 2024-03-04T09:31:31.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.815+0000 7fcf4fa00640 1 --2- 172.21.15.6:0/428603285 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562556c12fd0 0x562556cb5bd0 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7fcf4000dfa0 tx=0x7fcf4000b5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:31.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.816+0000 7fcf47e00640 1 -- 172.21.15.6:0/428603285 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf40011480 con 0x562556c12fd0 2024-03-04T09:31:31.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.816+0000 7fcf47e00640 1 -- 172.21.15.6:0/428603285 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf40016070 con 0x562556c12fd0 2024-03-04T09:31:31.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.816+0000 7fcf47e00640 1 -- 172.21.15.6:0/428603285 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf40011860 con 0x562556c12fd0 2024-03-04T09:31:31.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.816+0000 7fcf53039f40 1 -- 172.21.15.6:0/428603285 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x562556be8d00 con 0x562556c12fd0 2024-03-04T09:31:31.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.816+0000 7fcf53039f40 1 -- 172.21.15.6:0/428603285 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562556cb78e0 con 0x562556c12fd0 2024-03-04T09:31:31.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.818+0000 7fcf47e00640 1 -- 172.21.15.6:0/428603285 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcf40011a00 con 0x562556c12fd0 2024-03-04T09:31:31.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.820+0000 7fcf47e00640 1 --2- 172.21.15.6:0/428603285 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fcf30069ee0 0x7fcf3006c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:31.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.820+0000 7fcf47e00640 1 -- 172.21.15.6:0/428603285 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcf40088ce0 con 0x562556c12fd0 2024-03-04T09:31:31.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.820+0000 7fcf53039f40 1 --2- 172.21.15.6:0/428603285 >> [v2:172.21.15.6:6823/1364449401,v1:172.21.15.6:6825/1364449401] conn(0x562556cbe290 0x562556cbb290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:31.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.820+0000 7fcf53039f40 1 -- 172.21.15.6:0/428603285 --> [v2:172.21.15.6:6823/1364449401,v1:172.21.15.6:6825/1364449401] -- osd_op(unknown.0.0:1 17.7 17:e2e07930:::10000000001.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e109) v9 -- 0x562556cb9f00 con 0x562556cbe290 2024-03-04T09:31:31.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.820+0000 7fcf50400640 1 --2- 172.21.15.6:0/428603285 >> [v2:172.21.15.6:6823/1364449401,v1:172.21.15.6:6825/1364449401] conn(0x562556cbe290 0x562556cbb290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:31.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.820+0000 7fcf4f000640 1 --2- 172.21.15.6:0/428603285 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fcf30069ee0 0x7fcf3006c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:31.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.821+0000 7fcf50400640 1 --2- 172.21.15.6:0/428603285 >> [v2:172.21.15.6:6823/1364449401,v1:172.21.15.6:6825/1364449401] conn(0x562556cbe290 0x562556cbb290 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:31.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.821+0000 7fcf50400640 1 -- 172.21.15.6:0/428603285 <== osd.0 v2:172.21.15.6:6823/1364449401 1 ==== osd_op_reply(1 10000000001.00000000 [getxattr (93) out=93b] v0'0 uv3 ondisk = 0) v8 ==== 164+0+93 (crc 0 0 0) 0x7fcf480526e0 con 0x562556cbe290 2024-03-04T09:31:31.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.821+0000 7fcf53039f40 1 -- 172.21.15.6:0/428603285 >> [v2:172.21.15.6:6823/1364449401,v1:172.21.15.6:6825/1364449401] conn(0x562556cbe290 msgr2=0x562556cbb290 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:31.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.821+0000 7fcf53039f40 1 --2- 172.21.15.6:0/428603285 >> [v2:172.21.15.6:6823/1364449401,v1:172.21.15.6:6825/1364449401] conn(0x562556cbe290 0x562556cbb290 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:31.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.822+0000 7fcf53039f40 1 -- 172.21.15.6:0/428603285 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fcf30069ee0 msgr2=0x7fcf3006c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:31.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.822+0000 7fcf53039f40 1 --2- 172.21.15.6:0/428603285 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fcf30069ee0 0x7fcf3006c3a0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0x562556c2c000 tx=0x7fcf3c00c000 comp rx=0 tx=0).stop 2024-03-04T09:31:31.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.822+0000 7fcf53039f40 1 -- 172.21.15.6:0/428603285 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562556c12fd0 msgr2=0x562556cb5bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:31.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.822+0000 7fcf53039f40 1 --2- 172.21.15.6:0/428603285 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562556c12fd0 0x562556cb5bd0 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7fcf4000dfa0 tx=0x7fcf4000b5b0 comp rx=0 tx=0).stop 2024-03-04T09:31:31.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.822+0000 7fcf50400640 1 -- 172.21.15.6:0/428603285 reap_dead start 2024-03-04T09:31:31.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.822+0000 7fcf53039f40 1 -- 172.21.15.6:0/428603285 shutdown_connections 2024-03-04T09:31:31.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.822+0000 7fcf53039f40 1 -- 172.21.15.6:0/428603285 >> 172.21.15.6:0/428603285 conn(0x562556c11670 msgr2=0x562556c16800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:31.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.822+0000 7fcf53039f40 1 -- 172.21.15.6:0/428603285 shutdown_connections 2024-03-04T09:31:31.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:31.822+0000 7fcf53039f40 1 -- 172.21.15.6:0/428603285 wait complete. 2024-03-04T09:31:31.823 INFO:teuthology.orchestra.run.smithi006.stdout:Wfile_adir_1 2024-03-04T09:31:31.823 DEBUG:teuthology.orchestra.run.smithi006:> ceph-dencoder type inode_backtrace_t import - decode dump_json 2024-03-04T09:31:32.012 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-04T09:31:32.013 INFO:teuthology.orchestra.run.smithi006.stdout: "ino": 1099511627777, 2024-03-04T09:31:32.013 INFO:teuthology.orchestra.run.smithi006.stdout: "ancestors": [ 2024-03-04T09:31:32.013 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:31:32.013 INFO:teuthology.orchestra.run.smithi006.stdout: "dirino": 1099511627776, 2024-03-04T09:31:32.013 INFO:teuthology.orchestra.run.smithi006.stdout: "dname": "file_a", 2024-03-04T09:31:32.013 INFO:teuthology.orchestra.run.smithi006.stdout: "version": 8 2024-03-04T09:31:32.013 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:31:32.013 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:31:32.013 INFO:teuthology.orchestra.run.smithi006.stdout: "dirino": 1, 2024-03-04T09:31:32.013 INFO:teuthology.orchestra.run.smithi006.stdout: "dname": "dir_1", 2024-03-04T09:31:32.013 INFO:teuthology.orchestra.run.smithi006.stdout: "version": 15 2024-03-04T09:31:32.013 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:31:32.013 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-03-04T09:31:32.014 INFO:teuthology.orchestra.run.smithi006.stdout: "pool": 17, 2024-03-04T09:31:32.014 INFO:teuthology.orchestra.run.smithi006.stdout: "old_pools": [] 2024-03-04T09:31:32.014 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-04T09:31:32.014 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:31:32.025 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-04T09:31:32.026 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T09:31:32.080 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-04T09:31:32.080 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T09:31:32.129 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T09:31:32.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.372+0000 7f7694a00640 1 Processor -- start 2024-03-04T09:31:32.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.373+0000 7f7694a00640 1 -- start start 2024-03-04T09:31:32.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.374+0000 7f7694a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f769015e0b0 0x7f76900cf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:32.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.374+0000 7f7694a00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f76900d4a10 0x7f76900d0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:32.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.374+0000 7f7694a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76900b3710 con 0x7f76900d0c80 2024-03-04T09:31:32.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.374+0000 7f7694a00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76900b09e0 con 0x7f76900d4a10 2024-03-04T09:31:32.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.374+0000 7f7694a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76900b39f0 con 0x7f769015e0b0 2024-03-04T09:31:32.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.375+0000 7f768fe00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f76900d0c80 0x7f76900d1900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:52702/0 (socket says 172.21.15.6:52702) 2024-03-04T09:31:32.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.375+0000 7f768fe00640 1 -- 172.21.15.6:0/541500750 learned_addr learned my addr 172.21.15.6:0/541500750 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:31:32.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.375+0000 7f768f400640 1 --2- 172.21.15.6:0/541500750 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f769015e0b0 0x7f76900cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:32.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.375+0000 7f7687e00640 1 --2- 172.21.15.6:0/541500750 >> v2:172.21.15.146:3300/0 conn(0x7f76900d4a10 0x7f76900d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:32.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.375+0000 7f768ea00640 1 -- 172.21.15.6:0/541500750 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3931820527 0 0) 0x7f76900b3710 con 0x7f76900d0c80 2024-03-04T09:31:32.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.375+0000 7f768ea00640 1 -- 172.21.15.6:0/541500750 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7674003520 con 0x7f76900d0c80 2024-03-04T09:31:32.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.376+0000 7f768ea00640 1 -- 172.21.15.6:0/541500750 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4282380170 0 0) 0x7f7674003520 con 0x7f76900d0c80 2024-03-04T09:31:32.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.376+0000 7f768ea00640 1 -- 172.21.15.6:0/541500750 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f769015e0b0 msgr2=0x7f76900cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:32.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.376+0000 7f768ea00640 1 --2- 172.21.15.6:0/541500750 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f769015e0b0 0x7f76900cf310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:32.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.376+0000 7f768ea00640 1 -- 172.21.15.6:0/541500750 >> v2:172.21.15.146:3300/0 conn(0x7f76900d4a10 msgr2=0x7f76900d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:32.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.376+0000 7f768ea00640 1 --2- 172.21.15.6:0/541500750 >> v2:172.21.15.146:3300/0 conn(0x7f76900d4a10 0x7f76900d0740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:32.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.376+0000 7f768ea00640 1 -- 172.21.15.6:0/541500750 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76900b3cd0 con 0x7f76900d0c80 2024-03-04T09:31:32.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.376+0000 7f768ea00640 1 -- 172.21.15.6:0/541500750 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7f76880019f0 con 0x7f76900d0c80 2024-03-04T09:31:32.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.376+0000 7f768ea00640 1 -- 172.21.15.6:0/541500750 >> v1:172.21.15.6:6789/0 conn(0x7f76900d0c80 legacy=0x7f76900d1900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:32.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.376+0000 7f768ea00640 1 --2- 172.21.15.6:0/541500750 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f76740039b0 0x7f7674003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:32.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.376+0000 7f768ea00640 1 --2- 172.21.15.6:0/541500750 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7674004800 0x7f7674004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:32.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.376+0000 7f768ea00640 1 --2- 172.21.15.6:0/541500750 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f769015e0b0 0x7f76740051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:32.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.376+0000 7f768ea00640 1 -- 172.21.15.6:0/541500750 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76900b39f0 con 0x7f76740039b0 2024-03-04T09:31:32.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.376+0000 7f768ea00640 1 -- 172.21.15.6:0/541500750 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f767400b9c0 con 0x7f7674004800 2024-03-04T09:31:32.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.376+0000 7f768ea00640 1 -- 172.21.15.6:0/541500750 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f767400bb40 con 0x7f769015e0b0 2024-03-04T09:31:32.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.376+0000 7f768f400640 1 --2- 172.21.15.6:0/541500750 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f76740039b0 0x7f7674003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:32.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.377+0000 7f768fe00640 1 --2- 172.21.15.6:0/541500750 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f769015e0b0 0x7f76740051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:32.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.377+0000 7f7687e00640 1 --2- 172.21.15.6:0/541500750 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7674004800 0x7f7674004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:32.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.377+0000 7f768f400640 1 -- 172.21.15.6:0/541500750 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f769015e0b0 msgr2=0x7f76740051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:32.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.377+0000 7f768f400640 1 --2- 172.21.15.6:0/541500750 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f769015e0b0 0x7f76740051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:32.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.377+0000 7f768f400640 1 -- 172.21.15.6:0/541500750 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7674004800 msgr2=0x7f7674004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:32.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.377+0000 7f768f400640 1 --2- 172.21.15.6:0/541500750 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7674004800 0x7f7674004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:32.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.377+0000 7f768f400640 1 -- 172.21.15.6:0/541500750 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7674003520 con 0x7f76740039b0 2024-03-04T09:31:32.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.378+0000 7f768f400640 1 --2- 172.21.15.6:0/541500750 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f76740039b0 0x7f7674003dd0 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7f7680009ac0 tx=0x7f7674005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24e7488704f8b52d server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:32.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.378+0000 7f768ea00640 1 -- 172.21.15.6:0/541500750 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7680039070 con 0x7f76740039b0 2024-03-04T09:31:32.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.378+0000 7f768ea00640 1 -- 172.21.15.6:0/541500750 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f768002c6b0 con 0x7f76740039b0 2024-03-04T09:31:32.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.378+0000 7f768ea00640 1 -- 172.21.15.6:0/541500750 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7680009690 con 0x7f76740039b0 2024-03-04T09:31:32.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.379+0000 7f7694a00640 1 -- 172.21.15.6:0/541500750 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f76740039b0 msgr2=0x7f7674003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:32.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.379+0000 7f7694a00640 1 --2- 172.21.15.6:0/541500750 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f76740039b0 0x7f7674003dd0 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7f7680009ac0 tx=0x7f7674005830 comp rx=0 tx=0).stop 2024-03-04T09:31:32.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.379+0000 7f768fe00640 1 -- 172.21.15.6:0/541500750 reap_dead start 2024-03-04T09:31:32.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.379+0000 7f7694a00640 1 -- 172.21.15.6:0/541500750 shutdown_connections 2024-03-04T09:31:32.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.379+0000 7f7694a00640 1 -- 172.21.15.6:0/541500750 >> 172.21.15.6:0/541500750 conn(0x7f76900d3d30 msgr2=0x7f76900d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:32.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.379+0000 7f7694a00640 1 -- 172.21.15.6:0/541500750 shutdown_connections 2024-03-04T09:31:32.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.379+0000 7f7694a00640 1 -- 172.21.15.6:0/541500750 wait complete. 2024-03-04T09:31:32.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.382+0000 7f7694a00640 1 Processor -- start 2024-03-04T09:31:32.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.382+0000 7f7694a00640 1 -- start start 2024-03-04T09:31:32.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.384+0000 7f7694a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f76740039b0 0x7f7690149c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:32.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.384+0000 7f7694a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f76901f4180 0x7f76900cd440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:32.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.384+0000 7f7694a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f76901f4600 0x7f769006c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:32.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.384+0000 7f7694a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76900cfc60 con 0x7f76901f4180 2024-03-04T09:31:32.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.384+0000 7f7694a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f769015e490 con 0x7f76740039b0 2024-03-04T09:31:32.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.384+0000 7f7694a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76900b54e0 con 0x7f76901f4600 2024-03-04T09:31:32.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.384+0000 7f7687e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f76901f4180 0x7f76900cd440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:32.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.384+0000 7f7687e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f76901f4180 0x7f76900cd440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49940/0 (socket says 172.21.15.6:49940) 2024-03-04T09:31:32.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.384+0000 7f7687e00640 1 -- 172.21.15.6:0/775120469 learned_addr learned my addr 172.21.15.6:0/775120469 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:31:32.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.385+0000 7f7687e00640 1 -- 172.21.15.6:0/775120469 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f76901f4600 msgr2=0x7f769006c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:32.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.385+0000 7f7687e00640 1 --2- 172.21.15.6:0/775120469 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f76901f4600 0x7f769006c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:32.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.385+0000 7f7687e00640 1 -- 172.21.15.6:0/775120469 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f76740039b0 msgr2=0x7f7690149c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:32.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.385+0000 7f7687e00640 1 --2- 172.21.15.6:0/775120469 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f76740039b0 0x7f7690149c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:32.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.385+0000 7f7687e00640 1 -- 172.21.15.6:0/775120469 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76901f39a0 con 0x7f76901f4180 2024-03-04T09:31:32.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.385+0000 7f7687e00640 1 --2- 172.21.15.6:0/775120469 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f76901f4180 0x7f76900cd440 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7f76780108c0 tx=0x7f7678010d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:32.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.385+0000 7f7687400640 1 -- 172.21.15.6:0/775120469 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f767800f040 con 0x7f76901f4180 2024-03-04T09:31:32.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.385+0000 7f7687400640 1 -- 172.21.15.6:0/775120469 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f767800eb00 con 0x7f76901f4180 2024-03-04T09:31:32.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.385+0000 7f7687400640 1 -- 172.21.15.6:0/775120469 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f767801c400 con 0x7f76901f4180 2024-03-04T09:31:32.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.385+0000 7f7694a00640 1 -- 172.21.15.6:0/775120469 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f769014a1b0 con 0x7f76901f4180 2024-03-04T09:31:32.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.385+0000 7f7694a00640 1 -- 172.21.15.6:0/775120469 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f769014a760 con 0x7f76901f4180 2024-03-04T09:31:32.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.387+0000 7f7694a00640 1 -- 172.21.15.6:0/775120469 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f769004f330 con 0x7f76901f4180 2024-03-04T09:31:32.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.388+0000 7f7687400640 1 -- 172.21.15.6:0/775120469 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f767801c5a0 con 0x7f76901f4180 2024-03-04T09:31:32.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.388+0000 7f7687400640 1 --2- 172.21.15.6:0/775120469 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f766c06b500 0x7f766c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:32.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.388+0000 7f7687400640 1 -- 172.21.15.6:0/775120469 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f767808a5e0 con 0x7f76901f4180 2024-03-04T09:31:32.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.389+0000 7f768f400640 1 --2- 172.21.15.6:0/775120469 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f766c06b500 0x7f766c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:32.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.390+0000 7f768f400640 1 --2- 172.21.15.6:0/775120469 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f766c06b500 0x7f766c06d9c0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f7680036000 tx=0x7f768002e000 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:32.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.398+0000 7f7687400640 1 -- 172.21.15.6:0/775120469 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7678018030 con 0x7f76901f4180 2024-03-04T09:31:32.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.603+0000 7f7694a00640 1 -- 172.21.15.6:0/775120469 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f769015d690 con 0x7f76901f4180 2024-03-04T09:31:32.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.606+0000 7f7687400640 1 -- 172.21.15.6:0/775120469 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v109) v1 ==== 74+0+16579 (secure 0 0 0) 0x7f7678058ea0 con 0x7f76901f4180 2024-03-04T09:31:32.603 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:31:32.604 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":109,"fsid":"edce0578-5ae4-4402-b45a-552bdef84d7c","created":"2024-03-04T09:24:54.523780+0000","modified":"2024-03-04T09:31:22.629333+0000","last_up_change":"2024-03-04T09:25:07.089049+0000","last_in_change":"2024-03-04T09:24:57.982077+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T09:25:07.935198+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-04T09:31:19.018233+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-04T09:31:20.131544+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a96574f-81f1-4bc6-8463-757520b84a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6825","nonce":1364449401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6827","nonce":1364449401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6831","nonce":1364449401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6829","nonce":1364449401}]},"public_addr":"172.21.15.6:6825/1364449401","cluster_addr":"172.21.15.6:6827/1364449401","heartbeat_back_addr":"172.21.15.6:6831/1364449401","heartbeat_front_addr":"172.21.15.6:6829/1364449401","state":["exists","up"]},{"osd":1,"uuid":"7d79b9c1-6b87-476a-bc36-7db5ee64cd90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6809","nonce":2494103078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6811","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6813","nonce":2494103078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6821","nonce":2494103078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6817","nonce":2494103078}]},"public_addr":"172.21.15.6:6809/2494103078","cluster_addr":"172.21.15.6:6813/2494103078","heartbeat_back_addr":"172.21.15.6:6821/2494103078","heartbeat_front_addr":"172.21.15.6:6817/2494103078","state":["exists","up"]},{"osd":2,"uuid":"492331b9-4ee9-40b6-bacd-85d44cd6c45a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6812","nonce":1500066681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6816","nonce":1500066681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6824","nonce":1500066681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6820","nonce":1500066681}]},"public_addr":"172.21.15.6:6812/1500066681","cluster_addr":"172.21.15.6:6816/1500066681","heartbeat_back_addr":"172.21.15.6:6824/1500066681","heartbeat_front_addr":"172.21.15.6:6820/1500066681","state":["exists","up"]},{"osd":3,"uuid":"21bb2efe-9ba7-4bdc-bc31-180cd1102e21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6801","nonce":1558806123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6803","nonce":1558806123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6807","nonce":1558806123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6805","nonce":1558806123}]},"public_addr":"172.21.15.6:6801/1558806123","cluster_addr":"172.21.15.6:6803/1558806123","heartbeat_back_addr":"172.21.15.6:6807/1558806123","heartbeat_front_addr":"172.21.15.6:6805/1558806123","state":["exists","up"]},{"osd":4,"uuid":"6e2c4b71-19d9-4737-83d9-558705d8644a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6801","nonce":1526135780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6803","nonce":1526135780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6807","nonce":1526135780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6805","nonce":1526135780}]},"public_addr":"172.21.15.146:6801/1526135780","cluster_addr":"172.21.15.146:6803/1526135780","heartbeat_back_addr":"172.21.15.146:6807/1526135780","heartbeat_front_addr":"172.21.15.146:6805/1526135780","state":["exists","up"]},{"osd":5,"uuid":"fb69c525-f1be-4b7d-ab32-6c29b0d6fddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6809","nonce":460262507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6811","nonce":460262507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6815","nonce":460262507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6813","nonce":460262507}]},"public_addr":"172.21.15.146:6809/460262507","cluster_addr":"172.21.15.146:6811/460262507","heartbeat_back_addr":"172.21.15.146:6815/460262507","heartbeat_front_addr":"172.21.15.146:6813/460262507","state":["exists","up"]},{"osd":6,"uuid":"c32e7fed-370d-4401-8ce0-6a4335b0d168","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6823","nonce":1392897493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6827","nonce":1392897493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6831","nonce":1392897493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6829","nonce":1392897493}]},"public_addr":"172.21.15.146:6823/1392897493","cluster_addr":"172.21.15.146:6827/1392897493","heartbeat_back_addr":"172.21.15.146:6831/1392897493","heartbeat_front_addr":"172.21.15.146:6829/1392897493","state":["exists","up"]},{"osd":7,"uuid":"75a79b44-fa5f-426d-a7d3-71f21f78835c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6817","nonce":3268323156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6819","nonce":3268323156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6825","nonce":3268323156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6822","nonce":3268323156}]},"public_addr":"172.21.15.146:6817/3268323156","cluster_addr":"172.21.15.146:6819/3268323156","heartbeat_back_addr":"172.21.15.146:6825/3268323156","heartbeat_front_addr":"172.21.15.146:6822/3268323156","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.310437+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.295850+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.294674+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.526855+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.300023+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.595160+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.346046+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.558741+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T09:31:32.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.608+0000 7f7694a00640 1 -- 172.21.15.6:0/775120469 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f766c06b500 msgr2=0x7f766c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:32.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.608+0000 7f7694a00640 1 --2- 172.21.15.6:0/775120469 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f766c06b500 0x7f766c06d9c0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f7680036000 tx=0x7f768002e000 comp rx=0 tx=0).stop 2024-03-04T09:31:32.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.608+0000 7f7694a00640 1 -- 172.21.15.6:0/775120469 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f76901f4180 msgr2=0x7f76900cd440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:32.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.608+0000 7f7694a00640 1 --2- 172.21.15.6:0/775120469 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f76901f4180 0x7f76900cd440 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7f76780108c0 tx=0x7f7678010d80 comp rx=0 tx=0).stop 2024-03-04T09:31:32.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.609+0000 7f7694a00640 1 -- 172.21.15.6:0/775120469 shutdown_connections 2024-03-04T09:31:32.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.609+0000 7f7694a00640 1 --2- 172.21.15.6:0/775120469 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f766c06b500 0x7f766c06d9c0 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:32.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.609+0000 7f7694a00640 1 --2- 172.21.15.6:0/775120469 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f76901f4600 0x7f769006c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:32.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.609+0000 7f7694a00640 1 --2- 172.21.15.6:0/775120469 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f76901f4180 0x7f76900cd440 unknown :-1 s=CLOSED pgs=1279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:32.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.609+0000 7f7694a00640 1 --2- 172.21.15.6:0/775120469 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f76740039b0 0x7f7690149c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:32.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.609+0000 7f7694a00640 1 -- 172.21.15.6:0/775120469 >> 172.21.15.6:0/775120469 conn(0x7f76900d3d30 msgr2=0x7f769006b9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:32.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.609+0000 7f7694a00640 1 -- 172.21.15.6:0/775120469 shutdown_connections 2024-03-04T09:31:32.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.609+0000 7f7694a00640 1 -- 172.21.15.6:0/775120469 wait complete. 2024-03-04T09:31:32.634 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T09:25:07.935198+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-04T09:31:19.018233+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '108', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2024-03-04T09:31:20.131544+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '109', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-04T09:31:32.634 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-04T09:31:32.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.859+0000 7f849e600640 1 Processor -- start 2024-03-04T09:31:32.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.859+0000 7f849e600640 1 -- start start 2024-03-04T09:31:32.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.860+0000 7f849e600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f8498161700 0x7f84980b4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:32.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.860+0000 7f849e600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f84980d24f0 0x7f84980b1220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:32.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.860+0000 7f849e600640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84980b2c00 con 0x7f84980d4890 2024-03-04T09:31:32.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.860+0000 7f849e600640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8498161ae0 con 0x7f8498161700 2024-03-04T09:31:32.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.860+0000 7f849e600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84980b2ea0 con 0x7f84980d24f0 2024-03-04T09:31:32.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.861+0000 7f849d200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f8498161700 0x7f84980b4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:32.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.861+0000 7f8497e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f84980d4890 0x7f84980d1de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:52704/0 (socket says 172.21.15.6:52704) 2024-03-04T09:31:32.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.861+0000 7f8497e00640 1 -- 172.21.15.6:0/2989043171 learned_addr learned my addr 172.21.15.6:0/2989043171 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:31:32.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.861+0000 7f849dc00640 1 --2- 172.21.15.6:0/2989043171 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f84980d24f0 0x7f84980b1220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:32.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.861+0000 7f849d200640 1 -- 172.21.15.6:0/2989043171 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f84980d24f0 msgr2=0x7f84980b1220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:32.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.861+0000 7f849d200640 1 --2- 172.21.15.6:0/2989043171 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f84980d24f0 0x7f84980b1220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:32.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.861+0000 7f849d200640 1 -- 172.21.15.6:0/2989043171 >> v1:172.21.15.6:6789/0 conn(0x7f84980d4890 legacy=0x7f84980d1de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:32.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.862+0000 7f849d200640 1 -- 172.21.15.6:0/2989043171 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84980b3180 con 0x7f8498161700 2024-03-04T09:31:32.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.861+0000 7f8497400640 1 -- 172.21.15.6:0/2989043171 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1314034436 0 0) 0x7f84980b2c00 con 0x7f84980d4890 2024-03-04T09:31:32.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.863+0000 7f849d200640 1 --2- 172.21.15.6:0/2989043171 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8498161700 0x7f84980b4fd0 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7f848c0097a0 tx=0x7f848c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3a29a1201db7ad6 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:32.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.864+0000 7f8497400640 1 -- 172.21.15.6:0/2989043171 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f848c035020 con 0x7f8498161700 2024-03-04T09:31:32.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.864+0000 7f8497400640 1 -- 172.21.15.6:0/2989043171 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f848c031830 con 0x7f8498161700 2024-03-04T09:31:32.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.864+0000 7f8497400640 1 -- 172.21.15.6:0/2989043171 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f848c031c00 con 0x7f8498161700 2024-03-04T09:31:32.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.864+0000 7f849e600640 1 -- 172.21.15.6:0/2989043171 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8498161700 msgr2=0x7f84980b4fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:32.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.864+0000 7f849e600640 1 --2- 172.21.15.6:0/2989043171 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8498161700 0x7f84980b4fd0 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7f848c0097a0 tx=0x7f848c02b8d0 comp rx=0 tx=0).stop 2024-03-04T09:31:32.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.864+0000 7f849e600640 1 -- 172.21.15.6:0/2989043171 shutdown_connections 2024-03-04T09:31:32.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.864+0000 7f849e600640 1 --2- 172.21.15.6:0/2989043171 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f84980d24f0 0x7f84980b1220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:32.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.864+0000 7f849e600640 1 --2- 172.21.15.6:0/2989043171 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8498161700 0x7f84980b4fd0 unknown :-1 s=CLOSED pgs=1281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:32.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.865+0000 7f849e600640 1 -- 172.21.15.6:0/2989043171 >> 172.21.15.6:0/2989043171 conn(0x7f84980d3d70 msgr2=0x7f84980d4180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:32.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.865+0000 7f849e600640 1 -- 172.21.15.6:0/2989043171 shutdown_connections 2024-03-04T09:31:32.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.865+0000 7f849e600640 1 -- 172.21.15.6:0/2989043171 wait complete. 2024-03-04T09:31:32.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.868+0000 7f849e600640 1 Processor -- start 2024-03-04T09:31:32.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.868+0000 7f849e600640 1 -- start start 2024-03-04T09:31:32.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.869+0000 7f849e600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f84980d24f0 0x7f8498157360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:32.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.869+0000 7f849e600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f84980d4890 0x7f84980ccc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:32.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.870+0000 7f849e600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8498161700 0x7f84980ce710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:32.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.870+0000 7f849e600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84980b5510 con 0x7f84980d4890 2024-03-04T09:31:32.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.870+0000 7f849e600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84980b0d00 con 0x7f84980d24f0 2024-03-04T09:31:32.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.870+0000 7f849e600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84980b5690 con 0x7f8498161700 2024-03-04T09:31:32.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.870+0000 7f8497e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f84980d4890 0x7f84980ccc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:32.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.870+0000 7f8497e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f84980d4890 0x7f84980ccc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49954/0 (socket says 172.21.15.6:49954) 2024-03-04T09:31:32.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.870+0000 7f8497e00640 1 -- 172.21.15.6:0/2988669643 learned_addr learned my addr 172.21.15.6:0/2988669643 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:31:32.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.870+0000 7f849d200640 1 --2- 172.21.15.6:0/2988669643 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f84980d24f0 0x7f8498157360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:32.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.870+0000 7f849dc00640 1 --2- 172.21.15.6:0/2988669643 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8498161700 0x7f84980ce710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:32.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.870+0000 7f8497e00640 1 -- 172.21.15.6:0/2988669643 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8498161700 msgr2=0x7f84980ce710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:32.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.870+0000 7f8497e00640 1 --2- 172.21.15.6:0/2988669643 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8498161700 0x7f84980ce710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:32.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.870+0000 7f8497e00640 1 -- 172.21.15.6:0/2988669643 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f84980d24f0 msgr2=0x7f8498157360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:32.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.870+0000 7f8497e00640 1 --2- 172.21.15.6:0/2988669643 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f84980d24f0 0x7f8498157360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:32.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.870+0000 7f8497e00640 1 -- 172.21.15.6:0/2988669643 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84980cd560 con 0x7f84980d4890 2024-03-04T09:31:32.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.871+0000 7f8497e00640 1 --2- 172.21.15.6:0/2988669643 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f84980d4890 0x7f84980ccc30 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7f848800c4d0 tx=0x7f848800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:32.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.871+0000 7f8494c00640 1 -- 172.21.15.6:0/2988669643 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8488018070 con 0x7f84980d4890 2024-03-04T09:31:32.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.871+0000 7f8494c00640 1 -- 172.21.15.6:0/2988669643 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8488007500 con 0x7f84980d4890 2024-03-04T09:31:32.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.871+0000 7f849e600640 1 -- 172.21.15.6:0/2988669643 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84980c92b0 con 0x7f84980d4890 2024-03-04T09:31:32.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.871+0000 7f8494c00640 1 -- 172.21.15.6:0/2988669643 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8488004a20 con 0x7f84980d4890 2024-03-04T09:31:32.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.871+0000 7f849e600640 1 -- 172.21.15.6:0/2988669643 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84980cd170 con 0x7f84980d4890 2024-03-04T09:31:32.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.873+0000 7f849e600640 1 -- 172.21.15.6:0/2988669643 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f849804f330 con 0x7f84980d4890 2024-03-04T09:31:32.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.873+0000 7f8494c00640 1 -- 172.21.15.6:0/2988669643 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f848801b070 con 0x7f84980d4890 2024-03-04T09:31:32.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.874+0000 7f8494c00640 1 --2- 172.21.15.6:0/2988669643 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f847c06b4d0 0x7f847c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:32.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.874+0000 7f8494c00640 1 -- 172.21.15.6:0/2988669643 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f84880894b0 con 0x7f84980d4890 2024-03-04T09:31:32.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.875+0000 7f849d200640 1 --2- 172.21.15.6:0/2988669643 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f847c06b4d0 0x7f847c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:32.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.876+0000 7f849d200640 1 --2- 172.21.15.6:0/2988669643 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f847c06b4d0 0x7f847c06d990 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f848c036000 tx=0x7f848c03c040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:32.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:32.879+0000 7f8494c00640 1 -- 172.21.15.6:0/2988669643 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f84880583c0 con 0x7f84980d4890 2024-03-04T09:31:33.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.178+0000 7f849e600640 1 -- 172.21.15.6:0/2988669643 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f849806bd70 con 0x7f84980d4890 2024-03-04T09:31:33.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.179+0000 7f8494c00640 1 -- 172.21.15.6:0/2988669643 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v109) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8488057cc0 con 0x7f84980d4890 2024-03-04T09:31:33.176 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-04T09:31:33.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.181+0000 7f849e600640 1 -- 172.21.15.6:0/2988669643 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f847c06b4d0 msgr2=0x7f847c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:33.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.181+0000 7f849e600640 1 --2- 172.21.15.6:0/2988669643 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f847c06b4d0 0x7f847c06d990 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f848c036000 tx=0x7f848c03c040 comp rx=0 tx=0).stop 2024-03-04T09:31:33.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.181+0000 7f849e600640 1 -- 172.21.15.6:0/2988669643 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f84980d4890 msgr2=0x7f84980ccc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:33.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.181+0000 7f849e600640 1 --2- 172.21.15.6:0/2988669643 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f84980d4890 0x7f84980ccc30 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7f848800c4d0 tx=0x7f848800c990 comp rx=0 tx=0).stop 2024-03-04T09:31:33.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.181+0000 7f849e600640 1 -- 172.21.15.6:0/2988669643 shutdown_connections 2024-03-04T09:31:33.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.181+0000 7f849e600640 1 --2- 172.21.15.6:0/2988669643 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f847c06b4d0 0x7f847c06d990 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:33.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.181+0000 7f849e600640 1 --2- 172.21.15.6:0/2988669643 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8498161700 0x7f84980ce710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:33.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.181+0000 7f849e600640 1 --2- 172.21.15.6:0/2988669643 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f84980d4890 0x7f84980ccc30 unknown :-1 s=CLOSED pgs=1282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:33.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.181+0000 7f849e600640 1 --2- 172.21.15.6:0/2988669643 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f84980d24f0 0x7f8498157360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:33.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.181+0000 7f849e600640 1 -- 172.21.15.6:0/2988669643 >> 172.21.15.6:0/2988669643 conn(0x7f84980d3d70 msgr2=0x7f84980b4540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:33.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.181+0000 7f849e600640 1 -- 172.21.15.6:0/2988669643 shutdown_connections 2024-03-04T09:31:33.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.181+0000 7f849e600640 1 -- 172.21.15.6:0/2988669643 wait complete. 2024-03-04T09:31:33.197 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-04T09:31:33.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.412+0000 7fc6f0c00640 1 Processor -- start 2024-03-04T09:31:33.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.412+0000 7fc6f0c00640 1 -- start start 2024-03-04T09:31:33.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.416+0000 7fc6f0c00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fc6ec06ce40 0x7fc6ec06d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:33.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.416+0000 7fc6f0c00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6ec15b8d0 con 0x7fc6ec15ed90 2024-03-04T09:31:33.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.416+0000 7fc6f0c00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6ec0cdb50 con 0x7fc6ec15a4e0 2024-03-04T09:31:33.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.416+0000 7fc6f0c00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6ec0cde30 con 0x7fc6ec06ce40 2024-03-04T09:31:33.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.416+0000 7fc6e3e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fc6ec15ed90 0x7fc6ec0cebf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:52714/0 (socket says 172.21.15.6:52714) 2024-03-04T09:31:33.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.416+0000 7fc6e3e00640 1 -- 172.21.15.6:0/3883712852 learned_addr learned my addr 172.21.15.6:0/3883712852 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:31:33.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.416+0000 7fc6e9400640 1 -- 172.21.15.6:0/3883712852 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2651913077 0 0) 0x7fc6ec15b8d0 con 0x7fc6ec15ed90 2024-03-04T09:31:33.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.416+0000 7fc6e9400640 1 -- 172.21.15.6:0/3883712852 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6d0003520 con 0x7fc6ec15ed90 2024-03-04T09:31:33.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.417+0000 7fc6ea800640 1 --2- 172.21.15.6:0/3883712852 >> v2:172.21.15.146:3300/0 conn(0x7fc6ec06ce40 0x7fc6ec06d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:33.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.417+0000 7fc6e9400640 1 -- 172.21.15.6:0/3883712852 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4017139625 0 0) 0x7fc6d0003520 con 0x7fc6ec15ed90 2024-03-04T09:31:33.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.417+0000 7fc6e9400640 1 -- 172.21.15.6:0/3883712852 >> v2:172.21.15.146:3300/0 conn(0x7fc6ec06ce40 msgr2=0x7fc6ec06d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:33.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.417+0000 7fc6e9400640 1 --2- 172.21.15.6:0/3883712852 >> v2:172.21.15.146:3300/0 conn(0x7fc6ec06ce40 0x7fc6ec06d2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:33.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.417+0000 7fc6e9400640 1 -- 172.21.15.6:0/3883712852 >> v1:172.21.15.146:6789/0 conn(0x7fc6ec15a4e0 legacy=0x7fc6ec0aef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:33.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.417+0000 7fc6e9400640 1 -- 172.21.15.6:0/3883712852 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6ec0ce110 con 0x7fc6ec15ed90 2024-03-04T09:31:33.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.417+0000 7fc6e9400640 1 -- 172.21.15.6:0/3883712852 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7fc6d4001670 con 0x7fc6ec15ed90 2024-03-04T09:31:33.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.417+0000 7fc6e9400640 1 -- 172.21.15.6:0/3883712852 >> v1:172.21.15.6:6789/0 conn(0x7fc6ec15ed90 legacy=0x7fc6ec0cebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:33.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.417+0000 7fc6e9400640 1 --2- 172.21.15.6:0/3883712852 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc6d00039b0 0x7fc6d0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:33.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.417+0000 7fc6e9400640 1 --2- 172.21.15.6:0/3883712852 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc6d0004800 0x7fc6d0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:33.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.417+0000 7fc6e9400640 1 --2- 172.21.15.6:0/3883712852 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc6d00051c0 0x7fc6d000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:33.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.417+0000 7fc6e9400640 1 -- 172.21.15.6:0/3883712852 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6ec0cde30 con 0x7fc6d00039b0 2024-03-04T09:31:33.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.417+0000 7fc6e9400640 1 -- 172.21.15.6:0/3883712852 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6d000c110 con 0x7fc6d0004800 2024-03-04T09:31:33.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.417+0000 7fc6e9400640 1 -- 172.21.15.6:0/3883712852 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6d000c310 con 0x7fc6d00051c0 2024-03-04T09:31:33.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.417+0000 7fc6e9e00640 1 --2- 172.21.15.6:0/3883712852 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc6d00039b0 0x7fc6d0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:33.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.418+0000 7fc6e9e00640 1 -- 172.21.15.6:0/3883712852 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc6d00051c0 msgr2=0x7fc6d000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:33.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.418+0000 7fc6ea800640 1 --2- 172.21.15.6:0/3883712852 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc6d00051c0 0x7fc6d000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:33.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.418+0000 7fc6e3e00640 1 --2- 172.21.15.6:0/3883712852 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc6d0004800 0x7fc6d0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:33.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.418+0000 7fc6e9e00640 1 --2- 172.21.15.6:0/3883712852 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc6d00051c0 0x7fc6d000b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:33.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.418+0000 7fc6e9e00640 1 -- 172.21.15.6:0/3883712852 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc6d0004800 msgr2=0x7fc6d0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:33.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.418+0000 7fc6e9e00640 1 --2- 172.21.15.6:0/3883712852 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc6d0004800 0x7fc6d0004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:33.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.418+0000 7fc6e9e00640 1 -- 172.21.15.6:0/3883712852 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6d0003520 con 0x7fc6d00039b0 2024-03-04T09:31:33.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.418+0000 7fc6ea800640 1 -- 172.21.15.6:0/3883712852 reap_dead start 2024-03-04T09:31:33.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.419+0000 7fc6e9e00640 1 --2- 172.21.15.6:0/3883712852 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc6d00039b0 0x7fc6d0003dd0 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7fc6dc001750 tx=0x7fc6d0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c842594856e104b7 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:33.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.419+0000 7fc6e9400640 1 -- 172.21.15.6:0/3883712852 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6dc001370 con 0x7fc6d00039b0 2024-03-04T09:31:33.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.419+0000 7fc6e9400640 1 -- 172.21.15.6:0/3883712852 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6dc0052d0 con 0x7fc6d00039b0 2024-03-04T09:31:33.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.419+0000 7fc6e9400640 1 -- 172.21.15.6:0/3883712852 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6dc003830 con 0x7fc6d00039b0 2024-03-04T09:31:33.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.420+0000 7fc6f0c00640 1 -- 172.21.15.6:0/3883712852 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc6d00039b0 msgr2=0x7fc6d0003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:33.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.420+0000 7fc6f0c00640 1 --2- 172.21.15.6:0/3883712852 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc6d00039b0 0x7fc6d0003dd0 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7fc6dc001750 tx=0x7fc6d0005830 comp rx=0 tx=0).stop 2024-03-04T09:31:33.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.420+0000 7fc6f0c00640 1 -- 172.21.15.6:0/3883712852 shutdown_connections 2024-03-04T09:31:33.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.420+0000 7fc6f0c00640 1 --2- 172.21.15.6:0/3883712852 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc6d00039b0 0x7fc6d0003dd0 unknown :-1 s=CLOSED pgs=1284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:33.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.420+0000 7fc6f0c00640 1 -- 172.21.15.6:0/3883712852 >> 172.21.15.6:0/3883712852 conn(0x7fc6ec06b3d0 msgr2=0x7fc6ec06b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:33.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.420+0000 7fc6f0c00640 1 -- 172.21.15.6:0/3883712852 shutdown_connections 2024-03-04T09:31:33.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.420+0000 7fc6f0c00640 1 -- 172.21.15.6:0/3883712852 wait complete. 2024-03-04T09:31:33.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.422+0000 7fc6f0c00640 1 Processor -- start 2024-03-04T09:31:33.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.422+0000 7fc6f0c00640 1 -- start start 2024-03-04T09:31:33.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.424+0000 7fc6f0c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc6d00039b0 0x7fc6ec14d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:33.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.424+0000 7fc6f0c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc6ec09ac20 0x7fc6ec09d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:33.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.424+0000 7fc6f0c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc6ec158640 0x7fc6ec158ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:33.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.424+0000 7fc6f0c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6ec06d2f0 con 0x7fc6ec09ac20 2024-03-04T09:31:33.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.424+0000 7fc6f0c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6ec0ce6d0 con 0x7fc6d00039b0 2024-03-04T09:31:33.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.424+0000 7fc6f0c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6ec06d470 con 0x7fc6ec158640 2024-03-04T09:31:33.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.424+0000 7fc6e3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc6ec09ac20 0x7fc6ec09d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:33.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.424+0000 7fc6e3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc6ec09ac20 0x7fc6ec09d1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49976/0 (socket says 172.21.15.6:49976) 2024-03-04T09:31:33.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.424+0000 7fc6e3e00640 1 -- 172.21.15.6:0/3252088168 learned_addr learned my addr 172.21.15.6:0/3252088168 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:31:33.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.424+0000 7fc6e9e00640 1 --2- 172.21.15.6:0/3252088168 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc6d00039b0 0x7fc6ec14d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:33.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.424+0000 7fc6ea800640 1 --2- 172.21.15.6:0/3252088168 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc6ec158640 0x7fc6ec158ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:33.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.425+0000 7fc6e3e00640 1 -- 172.21.15.6:0/3252088168 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc6ec158640 msgr2=0x7fc6ec158ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:33.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.425+0000 7fc6e3e00640 1 --2- 172.21.15.6:0/3252088168 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc6ec158640 0x7fc6ec158ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:33.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.425+0000 7fc6e3e00640 1 -- 172.21.15.6:0/3252088168 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc6d00039b0 msgr2=0x7fc6ec14d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:33.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.425+0000 7fc6e3e00640 1 --2- 172.21.15.6:0/3252088168 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc6d00039b0 0x7fc6ec14d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:33.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.425+0000 7fc6e3e00640 1 -- 172.21.15.6:0/3252088168 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6ec157880 con 0x7fc6ec09ac20 2024-03-04T09:31:33.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.425+0000 7fc6e3e00640 1 --2- 172.21.15.6:0/3252088168 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc6ec09ac20 0x7fc6ec09d1b0 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7fc6d400d7d0 tx=0x7fc6d400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:33.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.425+0000 7fc6e2000640 1 -- 172.21.15.6:0/3252088168 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6d4019070 con 0x7fc6ec09ac20 2024-03-04T09:31:33.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.425+0000 7fc6e2000640 1 -- 172.21.15.6:0/3252088168 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6d40072d0 con 0x7fc6ec09ac20 2024-03-04T09:31:33.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.425+0000 7fc6f0c00640 1 -- 172.21.15.6:0/3252088168 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6ec157b60 con 0x7fc6ec09ac20 2024-03-04T09:31:33.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.425+0000 7fc6e2000640 1 -- 172.21.15.6:0/3252088168 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6d40047e0 con 0x7fc6ec09ac20 2024-03-04T09:31:33.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.425+0000 7fc6f0c00640 1 -- 172.21.15.6:0/3252088168 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6ec09d6f0 con 0x7fc6ec09ac20 2024-03-04T09:31:33.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.427+0000 7fc6f0c00640 1 -- 172.21.15.6:0/3252088168 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6ec04f330 con 0x7fc6ec09ac20 2024-03-04T09:31:33.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.428+0000 7fc6e2000640 1 -- 172.21.15.6:0/3252088168 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc6d401c070 con 0x7fc6ec09ac20 2024-03-04T09:31:33.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.429+0000 7fc6e2000640 1 --2- 172.21.15.6:0/3252088168 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc6c806b4d0 0x7fc6c806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:33.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.429+0000 7fc6e2000640 1 -- 172.21.15.6:0/3252088168 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc6d4089c70 con 0x7fc6ec09ac20 2024-03-04T09:31:33.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.430+0000 7fc6e9e00640 1 --2- 172.21.15.6:0/3252088168 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc6c806b4d0 0x7fc6c806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:33.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.431+0000 7fc6e9e00640 1 --2- 172.21.15.6:0/3252088168 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc6c806b4d0 0x7fc6c806d990 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fc6dc033000 tx=0x7fc6dc009030 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:33.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.433+0000 7fc6e2000640 1 -- 172.21.15.6:0/3252088168 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc6d4058a60 con 0x7fc6ec09ac20 2024-03-04T09:31:33.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.642+0000 7fc6f0c00640 1 -- 172.21.15.6:0/3252088168 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fc6ec156710 con 0x7fc6ec09ac20 2024-03-04T09:31:33.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.643+0000 7fc6e2000640 1 -- 172.21.15.6:0/3252088168 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v109) v1 ==== 104+0+11 (secure 0 0 0) 0x7fc6d4058360 con 0x7fc6ec09ac20 2024-03-04T09:31:33.640 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-03-04T09:31:33.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.646+0000 7fc6f0c00640 1 -- 172.21.15.6:0/3252088168 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc6c806b4d0 msgr2=0x7fc6c806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:33.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.646+0000 7fc6f0c00640 1 --2- 172.21.15.6:0/3252088168 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc6c806b4d0 0x7fc6c806d990 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fc6dc033000 tx=0x7fc6dc009030 comp rx=0 tx=0).stop 2024-03-04T09:31:33.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.646+0000 7fc6f0c00640 1 -- 172.21.15.6:0/3252088168 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc6ec09ac20 msgr2=0x7fc6ec09d1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:33.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.646+0000 7fc6f0c00640 1 --2- 172.21.15.6:0/3252088168 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc6ec09ac20 0x7fc6ec09d1b0 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7fc6d400d7d0 tx=0x7fc6d400dc90 comp rx=0 tx=0).stop 2024-03-04T09:31:33.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.646+0000 7fc6f0c00640 1 -- 172.21.15.6:0/3252088168 shutdown_connections 2024-03-04T09:31:33.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.646+0000 7fc6f0c00640 1 --2- 172.21.15.6:0/3252088168 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc6c806b4d0 0x7fc6c806d990 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:33.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.646+0000 7fc6f0c00640 1 --2- 172.21.15.6:0/3252088168 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc6ec158640 0x7fc6ec158ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:33.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.646+0000 7fc6f0c00640 1 --2- 172.21.15.6:0/3252088168 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc6ec09ac20 0x7fc6ec09d1b0 unknown :-1 s=CLOSED pgs=1285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:33.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.646+0000 7fc6f0c00640 1 --2- 172.21.15.6:0/3252088168 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc6d00039b0 0x7fc6ec14d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:33.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.646+0000 7fc6f0c00640 1 -- 172.21.15.6:0/3252088168 >> 172.21.15.6:0/3252088168 conn(0x7fc6ec06b3d0 msgr2=0x7fc6ec0cd080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:33.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.646+0000 7fc6f0c00640 1 -- 172.21.15.6:0/3252088168 shutdown_connections 2024-03-04T09:31:33.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.646+0000 7fc6f0c00640 1 -- 172.21.15.6:0/3252088168 wait complete. 2024-03-04T09:31:33.663 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-04T09:31:33.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.884+0000 7eff27e00640 1 Processor -- start 2024-03-04T09:31:33.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.884+0000 7eff27e00640 1 -- start start 2024-03-04T09:31:33.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.886+0000 7eff27e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7eff28161700 0x7eff280b4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:33.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.886+0000 7eff27e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7eff280d4890 0x7eff280d1de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:33.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.886+0000 7eff27e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff280b2c00 con 0x7eff280d2320 2024-03-04T09:31:33.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.886+0000 7eff27e00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff28161ae0 con 0x7eff280d4890 2024-03-04T09:31:33.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.886+0000 7eff27e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff280b2ea0 con 0x7eff28161700 2024-03-04T09:31:33.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.887+0000 7eff27400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7eff280d2320 0x7eff280b3600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:52730/0 (socket says 172.21.15.6:52730) 2024-03-04T09:31:33.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.887+0000 7eff27400640 1 -- 172.21.15.6:0/1415311521 learned_addr learned my addr 172.21.15.6:0/1415311521 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:31:33.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.887+0000 7eff26a00640 1 --2- 172.21.15.6:0/1415311521 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7eff28161700 0x7eff280b4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:33.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.887+0000 7eff25600640 1 -- 172.21.15.6:0/1415311521 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1769853176 0 0) 0x7eff280b2c00 con 0x7eff280d2320 2024-03-04T09:31:33.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.887+0000 7eff25600640 1 -- 172.21.15.6:0/1415311521 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff0c003520 con 0x7eff280d2320 2024-03-04T09:31:33.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.887+0000 7eff26000640 1 --2- 172.21.15.6:0/1415311521 >> v2:172.21.15.146:3300/0 conn(0x7eff280d4890 0x7eff280d1de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:33.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.888+0000 7eff25600640 1 -- 172.21.15.6:0/1415311521 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2142866621 0 0) 0x7eff0c003520 con 0x7eff280d2320 2024-03-04T09:31:33.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.888+0000 7eff25600640 1 -- 172.21.15.6:0/1415311521 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7eff28161700 msgr2=0x7eff280b4fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:33.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.888+0000 7eff25600640 1 --2- 172.21.15.6:0/1415311521 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7eff28161700 0x7eff280b4fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:33.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.888+0000 7eff25600640 1 -- 172.21.15.6:0/1415311521 >> v2:172.21.15.146:3300/0 conn(0x7eff280d4890 msgr2=0x7eff280d1de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:33.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.888+0000 7eff25600640 1 --2- 172.21.15.6:0/1415311521 >> v2:172.21.15.146:3300/0 conn(0x7eff280d4890 0x7eff280d1de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:33.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.888+0000 7eff25600640 1 -- 172.21.15.6:0/1415311521 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff280b3180 con 0x7eff280d2320 2024-03-04T09:31:33.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.888+0000 7eff25600640 1 -- 172.21.15.6:0/1415311521 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7eff200019f0 con 0x7eff280d2320 2024-03-04T09:31:33.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.888+0000 7eff25600640 1 -- 172.21.15.6:0/1415311521 >> v1:172.21.15.6:6789/0 conn(0x7eff280d2320 legacy=0x7eff280b3600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:33.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.888+0000 7eff25600640 1 --2- 172.21.15.6:0/1415311521 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7eff0c0039b0 0x7eff0c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:33.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.888+0000 7eff25600640 1 --2- 172.21.15.6:0/1415311521 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7eff28161700 0x7eff0c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:33.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.888+0000 7eff25600640 1 --2- 172.21.15.6:0/1415311521 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7eff0c004d40 0x7eff0c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:33.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.888+0000 7eff25600640 1 -- 172.21.15.6:0/1415311521 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff28161ae0 con 0x7eff0c0039b0 2024-03-04T09:31:33.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.888+0000 7eff25600640 1 -- 172.21.15.6:0/1415311521 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff280b2ea0 con 0x7eff0c004d40 2024-03-04T09:31:33.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.888+0000 7eff25600640 1 -- 172.21.15.6:0/1415311521 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff0c00ba40 con 0x7eff28161700 2024-03-04T09:31:33.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.888+0000 7eff26000640 1 --2- 172.21.15.6:0/1415311521 >> v2:172.21.15.146:3300/0 conn(0x7eff280d4890 0x7eff280d1de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T09:31:33.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.888+0000 7eff26a00640 1 --2- 172.21.15.6:0/1415311521 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7eff0c0039b0 0x7eff0c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:33.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.889+0000 7eff27400640 1 --2- 172.21.15.6:0/1415311521 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7eff0c004d40 0x7eff0c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:33.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.889+0000 7eff26000640 1 --2- 172.21.15.6:0/1415311521 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7eff28161700 0x7eff0c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:33.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.889+0000 7eff26a00640 1 -- 172.21.15.6:0/1415311521 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7eff28161700 msgr2=0x7eff0c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:33.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.889+0000 7eff26a00640 1 --2- 172.21.15.6:0/1415311521 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7eff28161700 0x7eff0c004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:33.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.889+0000 7eff26a00640 1 -- 172.21.15.6:0/1415311521 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7eff0c004d40 msgr2=0x7eff0c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:33.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.889+0000 7eff26a00640 1 --2- 172.21.15.6:0/1415311521 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7eff0c004d40 0x7eff0c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:33.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.889+0000 7eff26a00640 1 -- 172.21.15.6:0/1415311521 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff0c003520 con 0x7eff0c0039b0 2024-03-04T09:31:33.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.889+0000 7eff27400640 1 --2- 172.21.15.6:0/1415311521 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7eff0c004d40 0x7eff0c0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:31:33.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.890+0000 7eff26a00640 1 --2- 172.21.15.6:0/1415311521 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7eff0c0039b0 0x7eff0c003dd0 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7eff18009ac0 tx=0x7eff0c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35e77e5f0a95df0d server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:33.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.890+0000 7eff25600640 1 -- 172.21.15.6:0/1415311521 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff18039070 con 0x7eff0c0039b0 2024-03-04T09:31:33.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.890+0000 7eff25600640 1 -- 172.21.15.6:0/1415311521 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff1802c6b0 con 0x7eff0c0039b0 2024-03-04T09:31:33.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.890+0000 7eff25600640 1 -- 172.21.15.6:0/1415311521 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff18009690 con 0x7eff0c0039b0 2024-03-04T09:31:33.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.891+0000 7eff27e00640 1 -- 172.21.15.6:0/1415311521 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7eff0c0039b0 msgr2=0x7eff0c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:33.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.891+0000 7eff27e00640 1 --2- 172.21.15.6:0/1415311521 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7eff0c0039b0 0x7eff0c003dd0 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7eff18009ac0 tx=0x7eff0c005830 comp rx=0 tx=0).stop 2024-03-04T09:31:33.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.891+0000 7eff27400640 1 -- 172.21.15.6:0/1415311521 reap_dead start 2024-03-04T09:31:33.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.891+0000 7eff27e00640 1 -- 172.21.15.6:0/1415311521 shutdown_connections 2024-03-04T09:31:33.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.891+0000 7eff27e00640 1 -- 172.21.15.6:0/1415311521 >> 172.21.15.6:0/1415311521 conn(0x7eff280d3d70 msgr2=0x7eff280d4180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:33.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.891+0000 7eff27e00640 1 -- 172.21.15.6:0/1415311521 shutdown_connections 2024-03-04T09:31:33.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.891+0000 7eff27e00640 1 -- 172.21.15.6:0/1415311521 wait complete. 2024-03-04T09:31:33.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.893+0000 7eff27e00640 1 Processor -- start 2024-03-04T09:31:33.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.893+0000 7eff27e00640 1 -- start start 2024-03-04T09:31:33.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.894+0000 7eff27e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7eff280cd660 0x7eff281475d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:33.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.894+0000 7eff27e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7eff280cda40 0x7eff281f4d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:33.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.894+0000 7eff27e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7eff28155e60 0x7eff28156310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:33.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.894+0000 7eff27e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff280b2230 con 0x7eff280cd660 2024-03-04T09:31:33.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.894+0000 7eff27e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff280b0d00 con 0x7eff28155e60 2024-03-04T09:31:33.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.894+0000 7eff27e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff280b23b0 con 0x7eff280cda40 2024-03-04T09:31:33.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.894+0000 7eff26a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7eff280cd660 0x7eff281475d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:33.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.894+0000 7eff26a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7eff280cd660 0x7eff281475d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49996/0 (socket says 172.21.15.6:49996) 2024-03-04T09:31:33.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.894+0000 7eff26a00640 1 -- 172.21.15.6:0/2425103387 learned_addr learned my addr 172.21.15.6:0/2425103387 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:31:33.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.895+0000 7eff26000640 1 --2- 172.21.15.6:0/2425103387 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7eff280cda40 0x7eff281f4d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:33.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.895+0000 7eff26a00640 1 -- 172.21.15.6:0/2425103387 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7eff280cda40 msgr2=0x7eff281f4d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:33.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.895+0000 7eff27400640 1 --2- 172.21.15.6:0/2425103387 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7eff28155e60 0x7eff28156310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:33.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.895+0000 7eff26a00640 1 --2- 172.21.15.6:0/2425103387 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7eff280cda40 0x7eff281f4d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:33.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.895+0000 7eff26a00640 1 -- 172.21.15.6:0/2425103387 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7eff28155e60 msgr2=0x7eff28156310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:33.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.895+0000 7eff26a00640 1 --2- 172.21.15.6:0/2425103387 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7eff28155e60 0x7eff28156310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:33.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.895+0000 7eff26a00640 1 -- 172.21.15.6:0/2425103387 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff281f5250 con 0x7eff280cd660 2024-03-04T09:31:33.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.895+0000 7eff26000640 1 --2- 172.21.15.6:0/2425103387 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7eff280cda40 0x7eff281f4d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:31:33.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.895+0000 7eff26a00640 1 --2- 172.21.15.6:0/2425103387 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7eff280cd660 0x7eff281475d0 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7eff18009ac0 tx=0x7eff18004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:33.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.896+0000 7eff1ea00640 1 -- 172.21.15.6:0/2425103387 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff18044020 con 0x7eff280cd660 2024-03-04T09:31:33.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.896+0000 7eff1ea00640 1 -- 172.21.15.6:0/2425103387 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff1802dd20 con 0x7eff280cd660 2024-03-04T09:31:33.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.896+0000 7eff27e00640 1 -- 172.21.15.6:0/2425103387 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff28147c60 con 0x7eff280cd660 2024-03-04T09:31:33.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.896+0000 7eff1ea00640 1 -- 172.21.15.6:0/2425103387 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff18039040 con 0x7eff280cd660 2024-03-04T09:31:33.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.896+0000 7eff27e00640 1 -- 172.21.15.6:0/2425103387 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff28148220 con 0x7eff280cd660 2024-03-04T09:31:33.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.897+0000 7eff27e00640 1 -- 172.21.15.6:0/2425103387 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff2804f330 con 0x7eff280cd660 2024-03-04T09:31:33.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.898+0000 7eff1ea00640 1 -- 172.21.15.6:0/2425103387 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7eff18047020 con 0x7eff280cd660 2024-03-04T09:31:33.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.899+0000 7eff1ea00640 1 --2- 172.21.15.6:0/2425103387 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7eff0806b500 0x7eff0806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:33.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.900+0000 7eff1ea00640 1 -- 172.21.15.6:0/2425103387 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5709+0+0 (secure 0 0 0) 0x7eff180abda0 con 0x7eff280cd660 2024-03-04T09:31:33.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.900+0000 7eff26000640 1 --2- 172.21.15.6:0/2425103387 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7eff0806b500 0x7eff0806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:33.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.901+0000 7eff26000640 1 --2- 172.21.15.6:0/2425103387 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7eff0806b500 0x7eff0806d9c0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7eff280d17a0 tx=0x7eff14008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:33.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:33.903+0000 7eff1ea00640 1 -- 172.21.15.6:0/2425103387 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7eff1804e070 con 0x7eff280cd660 2024-03-04T09:31:34.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.102+0000 7eff27e00640 1 -- 172.21.15.6:0/2425103387 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7eff2815bc90 con 0x7eff280cd660 2024-03-04T09:31:34.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.103+0000 7eff1ea00640 1 -- 172.21.15.6:0/2425103387 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v109) v1 ==== 100+0+11 (secure 0 0 0) 0x7eff1807ad30 con 0x7eff280cd660 2024-03-04T09:31:34.100 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 64 2024-03-04T09:31:34.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.105+0000 7eff27e00640 1 -- 172.21.15.6:0/2425103387 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7eff0806b500 msgr2=0x7eff0806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:34.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.105+0000 7eff27e00640 1 --2- 172.21.15.6:0/2425103387 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7eff0806b500 0x7eff0806d9c0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7eff280d17a0 tx=0x7eff14008040 comp rx=0 tx=0).stop 2024-03-04T09:31:34.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.106+0000 7eff27e00640 1 -- 172.21.15.6:0/2425103387 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7eff280cd660 msgr2=0x7eff281475d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:34.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.106+0000 7eff27e00640 1 --2- 172.21.15.6:0/2425103387 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7eff280cd660 0x7eff281475d0 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7eff18009ac0 tx=0x7eff18004730 comp rx=0 tx=0).stop 2024-03-04T09:31:34.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.106+0000 7eff27e00640 1 -- 172.21.15.6:0/2425103387 shutdown_connections 2024-03-04T09:31:34.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.106+0000 7eff27e00640 1 --2- 172.21.15.6:0/2425103387 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7eff28155e60 0x7eff28156310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:34.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.106+0000 7eff27e00640 1 --2- 172.21.15.6:0/2425103387 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7eff0806b500 0x7eff0806d9c0 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:34.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.106+0000 7eff27e00640 1 --2- 172.21.15.6:0/2425103387 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7eff280cda40 0x7eff281f4d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:34.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.106+0000 7eff27e00640 1 --2- 172.21.15.6:0/2425103387 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7eff280cd660 0x7eff281475d0 unknown :-1 s=CLOSED pgs=1288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:34.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.106+0000 7eff27e00640 1 -- 172.21.15.6:0/2425103387 >> 172.21.15.6:0/2425103387 conn(0x7eff280d3d70 msgr2=0x7eff280b96e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:34.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.106+0000 7eff27e00640 1 -- 172.21.15.6:0/2425103387 shutdown_connections 2024-03-04T09:31:34.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.106+0000 7eff27e00640 1 -- 172.21.15.6:0/2425103387 wait complete. 2024-03-04T09:31:34.121 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-04T09:31:34.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.328+0000 7f3f84c00640 1 Processor -- start 2024-03-04T09:31:34.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.328+0000 7f3f84c00640 1 -- start start 2024-03-04T09:31:34.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.330+0000 7f3f84c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f3f80145300 0x7f3f800b49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:34.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.330+0000 7f3f84c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3f80070f10 0x7f3f800cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:34.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.330+0000 7f3f84c00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f3f800cce90 0x7f3f800cd930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:34.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.330+0000 7f3f84c00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f800cd300 con 0x7f3f80145300 2024-03-04T09:31:34.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.330+0000 7f3f84c00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f800714d0 con 0x7f3f800cce90 2024-03-04T09:31:34.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.330+0000 7f3f84c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f800717b0 con 0x7f3f80070f10 2024-03-04T09:31:34.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.330+0000 7f3f7f400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f3f80145300 0x7f3f800b49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:34.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.331+0000 7f3f7f400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f3f80145300 0x7f3f800b49a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49998/0 (socket says 172.21.15.6:49998) 2024-03-04T09:31:34.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.331+0000 7f3f7f400640 1 -- 172.21.15.6:0/1895485621 learned_addr learned my addr 172.21.15.6:0/1895485621 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:31:34.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.331+0000 7f3f7ea00640 1 --2- 172.21.15.6:0/1895485621 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3f80070f10 0x7f3f800cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:34.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.331+0000 7f3f7fe00640 1 --2- 172.21.15.6:0/1895485621 >> v2:172.21.15.146:3300/0 conn(0x7f3f800cce90 0x7f3f800cd930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:34.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.331+0000 7f3f7f400640 1 -- 172.21.15.6:0/1895485621 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3f80070f10 msgr2=0x7f3f800cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:34.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.331+0000 7f3f7f400640 1 --2- 172.21.15.6:0/1895485621 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3f80070f10 0x7f3f800cc950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:34.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.331+0000 7f3f7f400640 1 -- 172.21.15.6:0/1895485621 >> v2:172.21.15.146:3300/0 conn(0x7f3f800cce90 msgr2=0x7f3f800cd930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:34.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.331+0000 7f3f7f400640 1 --2- 172.21.15.6:0/1895485621 >> v2:172.21.15.146:3300/0 conn(0x7f3f800cce90 0x7f3f800cd930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:34.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.331+0000 7f3f7f400640 1 -- 172.21.15.6:0/1895485621 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f80071a90 con 0x7f3f80145300 2024-03-04T09:31:34.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.332+0000 7f3f7f400640 1 --2- 172.21.15.6:0/1895485621 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3f80145300 0x7f3f800b49a0 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7f3f80072420 tx=0x7f3f7002b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5519d49cd1833a server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:34.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.333+0000 7f3f7e000640 1 -- 172.21.15.6:0/1895485621 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f70035020 con 0x7f3f80145300 2024-03-04T09:31:34.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.333+0000 7f3f7e000640 1 -- 172.21.15.6:0/1895485621 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f7002d040 con 0x7f3f80145300 2024-03-04T09:31:34.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.333+0000 7f3f7e000640 1 -- 172.21.15.6:0/1895485621 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f70031a80 con 0x7f3f80145300 2024-03-04T09:31:34.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.333+0000 7f3f84c00640 1 -- 172.21.15.6:0/1895485621 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3f80145300 msgr2=0x7f3f800b49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:34.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.333+0000 7f3f84c00640 1 --2- 172.21.15.6:0/1895485621 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3f80145300 0x7f3f800b49a0 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7f3f80072420 tx=0x7f3f7002b900 comp rx=0 tx=0).stop 2024-03-04T09:31:34.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.334+0000 7f3f84c00640 1 -- 172.21.15.6:0/1895485621 shutdown_connections 2024-03-04T09:31:34.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.334+0000 7f3f84c00640 1 --2- 172.21.15.6:0/1895485621 >> v2:172.21.15.146:3300/0 conn(0x7f3f800cce90 0x7f3f800cd930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:34.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.334+0000 7f3f84c00640 1 --2- 172.21.15.6:0/1895485621 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3f80070f10 0x7f3f800cc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:34.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.334+0000 7f3f84c00640 1 --2- 172.21.15.6:0/1895485621 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3f80145300 0x7f3f800b49a0 unknown :-1 s=CLOSED pgs=1289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:34.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.334+0000 7f3f84c00640 1 -- 172.21.15.6:0/1895485621 >> 172.21.15.6:0/1895485621 conn(0x7f3f8006b3d0 msgr2=0x7f3f8006b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:34.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.334+0000 7f3f84c00640 1 -- 172.21.15.6:0/1895485621 shutdown_connections 2024-03-04T09:31:34.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.334+0000 7f3f84c00640 1 -- 172.21.15.6:0/1895485621 wait complete. 2024-03-04T09:31:34.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.336+0000 7f3f84c00640 1 Processor -- start 2024-03-04T09:31:34.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.336+0000 7f3f84c00640 1 -- start start 2024-03-04T09:31:34.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.337+0000 7f3f84c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3f80070f10 0x7f3f801eeba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:34.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.337+0000 7f3f84c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3f800cce90 0x7f3f800a9470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:34.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.337+0000 7f3f84c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3f80145300 0x7f3f801f4d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:34.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.337+0000 7f3f84c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f801456e0 con 0x7f3f80145300 2024-03-04T09:31:34.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.337+0000 7f3f84c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f800b4ee0 con 0x7f3f80070f10 2024-03-04T09:31:34.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.337+0000 7f3f84c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f800cde70 con 0x7f3f800cce90 2024-03-04T09:31:34.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.337+0000 7f3f7fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3f80145300 0x7f3f801f4d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:34.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.337+0000 7f3f7fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3f80145300 0x7f3f801f4d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:50004/0 (socket says 172.21.15.6:50004) 2024-03-04T09:31:34.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.337+0000 7f3f7fe00640 1 -- 172.21.15.6:0/3995494261 learned_addr learned my addr 172.21.15.6:0/3995494261 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:31:34.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.337+0000 7f3f7f400640 1 --2- 172.21.15.6:0/3995494261 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3f80070f10 0x7f3f801eeba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:34.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.337+0000 7f3f7ea00640 1 --2- 172.21.15.6:0/3995494261 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3f800cce90 0x7f3f800a9470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:34.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.337+0000 7f3f7fe00640 1 -- 172.21.15.6:0/3995494261 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3f800cce90 msgr2=0x7f3f800a9470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:34.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.337+0000 7f3f7fe00640 1 --2- 172.21.15.6:0/3995494261 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3f800cce90 0x7f3f800a9470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:34.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.338+0000 7f3f7fe00640 1 -- 172.21.15.6:0/3995494261 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3f80070f10 msgr2=0x7f3f801eeba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:34.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.338+0000 7f3f7fe00640 1 --2- 172.21.15.6:0/3995494261 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3f80070f10 0x7f3f801eeba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:34.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.338+0000 7f3f7fe00640 1 -- 172.21.15.6:0/3995494261 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f8013cd50 con 0x7f3f80145300 2024-03-04T09:31:34.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.338+0000 7f3f7f400640 1 --2- 172.21.15.6:0/3995494261 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3f80070f10 0x7f3f801eeba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:31:34.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.338+0000 7f3f7ea00640 1 --2- 172.21.15.6:0/3995494261 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3f800cce90 0x7f3f800a9470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:31:34.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.338+0000 7f3f7fe00640 1 --2- 172.21.15.6:0/3995494261 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3f80145300 0x7f3f801f4d10 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7f3f7800efc0 tx=0x7f3f7800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:34.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.338+0000 7f3f77400640 1 -- 172.21.15.6:0/3995494261 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f78018020 con 0x7f3f80145300 2024-03-04T09:31:34.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.338+0000 7f3f77400640 1 -- 172.21.15.6:0/3995494261 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f78014480 con 0x7f3f80145300 2024-03-04T09:31:34.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.338+0000 7f3f77400640 1 -- 172.21.15.6:0/3995494261 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f78014870 con 0x7f3f80145300 2024-03-04T09:31:34.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.338+0000 7f3f84c00640 1 -- 172.21.15.6:0/3995494261 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f8013d030 con 0x7f3f80145300 2024-03-04T09:31:34.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.338+0000 7f3f84c00640 1 -- 172.21.15.6:0/3995494261 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f8013d5f0 con 0x7f3f80145300 2024-03-04T09:31:34.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.340+0000 7f3f84c00640 1 -- 172.21.15.6:0/3995494261 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f8004f330 con 0x7f3f80145300 2024-03-04T09:31:34.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.342+0000 7f3f77400640 1 -- 172.21.15.6:0/3995494261 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3f78005d10 con 0x7f3f80145300 2024-03-04T09:31:34.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.342+0000 7f3f77400640 1 --2- 172.21.15.6:0/3995494261 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f3f6006b4d0 0x7f3f6006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:34.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.343+0000 7f3f77400640 1 -- 172.21.15.6:0/3995494261 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3f7808ad00 con 0x7f3f80145300 2024-03-04T09:31:34.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.343+0000 7f3f7f400640 1 --2- 172.21.15.6:0/3995494261 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f3f6006b4d0 0x7f3f6006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:34.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.344+0000 7f3f7f400640 1 --2- 172.21.15.6:0/3995494261 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f3f6006b4d0 0x7f3f6006d990 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f3f70036000 tx=0x7f3f70008030 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:34.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.347+0000 7f3f77400640 1 -- 172.21.15.6:0/3995494261 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3f78022070 con 0x7f3f80145300 2024-03-04T09:31:34.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.564+0000 7f3f84c00640 1 -- 172.21.15.6:0/3995494261 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f3f800b4970 con 0x7f3f80145300 2024-03-04T09:31:34.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.565+0000 7f3f77400640 1 -- 172.21.15.6:0/3995494261 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v109) v1 ==== 80+0+0 (secure 0 0 0) 0x7f3f78010040 con 0x7f3f80145300 2024-03-04T09:31:34.564 INFO:teuthology.orchestra.run.smithi006.stderr:listed 0 entries 2024-03-04T09:31:34.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.568+0000 7f3f84c00640 1 -- 172.21.15.6:0/3995494261 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f3f6006b4d0 msgr2=0x7f3f6006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:34.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.568+0000 7f3f84c00640 1 --2- 172.21.15.6:0/3995494261 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f3f6006b4d0 0x7f3f6006d990 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f3f70036000 tx=0x7f3f70008030 comp rx=0 tx=0).stop 2024-03-04T09:31:34.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.568+0000 7f3f84c00640 1 -- 172.21.15.6:0/3995494261 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3f80145300 msgr2=0x7f3f801f4d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:34.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.568+0000 7f3f84c00640 1 --2- 172.21.15.6:0/3995494261 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3f80145300 0x7f3f801f4d10 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7f3f7800efc0 tx=0x7f3f7800c490 comp rx=0 tx=0).stop 2024-03-04T09:31:34.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.568+0000 7f3f84c00640 1 -- 172.21.15.6:0/3995494261 shutdown_connections 2024-03-04T09:31:34.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.568+0000 7f3f84c00640 1 --2- 172.21.15.6:0/3995494261 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3f80145300 0x7f3f801f4d10 unknown :-1 s=CLOSED pgs=1290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:34.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.568+0000 7f3f84c00640 1 --2- 172.21.15.6:0/3995494261 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f3f6006b4d0 0x7f3f6006d990 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:34.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.568+0000 7f3f84c00640 1 --2- 172.21.15.6:0/3995494261 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3f800cce90 0x7f3f800a9470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:34.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.568+0000 7f3f84c00640 1 --2- 172.21.15.6:0/3995494261 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3f80070f10 0x7f3f801eeba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:34.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.568+0000 7f3f84c00640 1 -- 172.21.15.6:0/3995494261 >> 172.21.15.6:0/3995494261 conn(0x7f3f8006b3d0 msgr2=0x7f3f8015c790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:34.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.568+0000 7f3f84c00640 1 -- 172.21.15.6:0/3995494261 shutdown_connections 2024-03-04T09:31:34.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.568+0000 7f3f84c00640 1 -- 172.21.15.6:0/3995494261 wait complete. 2024-03-04T09:31:34.591 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-04T09:31:34.592 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:31:34.592 DEBUG:teuthology.orchestra.run.smithi006:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-04T09:31:34.747 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi006.front.sepia.ceph.com 2024-03-04T09:31:34.748 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:31:34.748 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-04T09:31:34.798 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:31:34.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.982+0000 7fcb2b600640 1 Processor -- start 2024-03-04T09:31:34.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.982+0000 7fcb2b600640 1 -- start start 2024-03-04T09:31:34.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.983+0000 7fcb2b600640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fcb240af580 0x7fcb2415c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:34.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.983+0000 7fcb2b600640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb2415cc40 con 0x7fcb240cdc20 2024-03-04T09:31:34.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.983+0000 7fcb2b600640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb2415cf20 con 0x7fcb240af1a0 2024-03-04T09:31:34.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.983+0000 7fcb2b600640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb2415d200 con 0x7fcb240af580 2024-03-04T09:31:34.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.984+0000 7fcb29600640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fcb240cdc20 0x7fcb240b2dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:52734/0 (socket says 172.21.15.6:52734) 2024-03-04T09:31:34.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.984+0000 7fcb29600640 1 -- 172.21.15.6:0/3029923280 learned_addr learned my addr 172.21.15.6:0/3029923280 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:31:34.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.984+0000 7fcb23400640 1 -- 172.21.15.6:0/3029923280 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 681645554 0 0) 0x7fcb2415cc40 con 0x7fcb240cdc20 2024-03-04T09:31:34.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.984+0000 7fcb23400640 1 -- 172.21.15.6:0/3029923280 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb08003520 con 0x7fcb240cdc20 2024-03-04T09:31:34.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.984+0000 7fcb23e00640 1 --2- 172.21.15.6:0/3029923280 >> v2:172.21.15.146:3300/0 conn(0x7fcb240af580 0x7fcb2415c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:34.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.984+0000 7fcb23400640 1 -- 172.21.15.6:0/3029923280 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2142249873 0 0) 0x7fcb08003520 con 0x7fcb240cdc20 2024-03-04T09:31:34.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.985+0000 7fcb23400640 1 -- 172.21.15.6:0/3029923280 >> v2:172.21.15.146:3300/0 conn(0x7fcb240af580 msgr2=0x7fcb2415c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:34.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.985+0000 7fcb23400640 1 --2- 172.21.15.6:0/3029923280 >> v2:172.21.15.146:3300/0 conn(0x7fcb240af580 0x7fcb2415c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:34.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.985+0000 7fcb23400640 1 -- 172.21.15.6:0/3029923280 >> v1:172.21.15.146:6789/0 conn(0x7fcb240af1a0 legacy=0x7fcb240b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:34.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.985+0000 7fcb23400640 1 -- 172.21.15.6:0/3029923280 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb2415d4e0 con 0x7fcb240cdc20 2024-03-04T09:31:34.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.985+0000 7fcb23400640 1 -- 172.21.15.6:0/3029923280 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7fcb1c001670 con 0x7fcb240cdc20 2024-03-04T09:31:34.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.985+0000 7fcb23400640 1 -- 172.21.15.6:0/3029923280 >> v1:172.21.15.6:6789/0 conn(0x7fcb240cdc20 legacy=0x7fcb240b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:34.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.985+0000 7fcb23400640 1 --2- 172.21.15.6:0/3029923280 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcb08003a70 0x7fcb08003e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:34.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.985+0000 7fcb23400640 1 --2- 172.21.15.6:0/3029923280 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcb08004800 0x7fcb08004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:34.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.985+0000 7fcb23400640 1 --2- 172.21.15.6:0/3029923280 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcb080053c0 0x7fcb0800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:34.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.985+0000 7fcb23400640 1 -- 172.21.15.6:0/3029923280 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb2415d200 con 0x7fcb08004800 2024-03-04T09:31:34.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.985+0000 7fcb23400640 1 -- 172.21.15.6:0/3029923280 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb0800c260 con 0x7fcb080053c0 2024-03-04T09:31:34.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.985+0000 7fcb23400640 1 -- 172.21.15.6:0/3029923280 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb0800c460 con 0x7fcb08003a70 2024-03-04T09:31:34.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.985+0000 7fcb23e00640 1 --2- 172.21.15.6:0/3029923280 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcb08004800 0x7fcb08004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:34.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.985+0000 7fcb28c00640 1 --2- 172.21.15.6:0/3029923280 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcb08003a70 0x7fcb08003e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:34.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.985+0000 7fcb29600640 1 --2- 172.21.15.6:0/3029923280 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcb080053c0 0x7fcb0800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:34.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.985+0000 7fcb23e00640 1 -- 172.21.15.6:0/3029923280 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcb08003a70 msgr2=0x7fcb08003e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:34.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.986+0000 7fcb23e00640 1 --2- 172.21.15.6:0/3029923280 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcb08003a70 0x7fcb08003e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:34.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.986+0000 7fcb23e00640 1 -- 172.21.15.6:0/3029923280 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcb080053c0 msgr2=0x7fcb0800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:34.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.986+0000 7fcb23e00640 1 --2- 172.21.15.6:0/3029923280 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcb080053c0 0x7fcb0800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:34.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.986+0000 7fcb23e00640 1 -- 172.21.15.6:0/3029923280 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb08003520 con 0x7fcb08004800 2024-03-04T09:31:34.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.986+0000 7fcb29600640 1 --2- 172.21.15.6:0/3029923280 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcb080053c0 0x7fcb0800b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:31:34.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.986+0000 7fcb29600640 1 -- 172.21.15.6:0/3029923280 reap_dead start 2024-03-04T09:31:34.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.987+0000 7fcb23e00640 1 --2- 172.21.15.6:0/3029923280 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcb08004800 0x7fcb08004c80 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7fcb0c007330 tx=0x7fcb08005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5e74a06afc066ed server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:34.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.987+0000 7fcb23400640 1 -- 172.21.15.6:0/3029923280 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb0c039070 con 0x7fcb08004800 2024-03-04T09:31:34.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.987+0000 7fcb23400640 1 -- 172.21.15.6:0/3029923280 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb0c02cca0 con 0x7fcb08004800 2024-03-04T09:31:34.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.987+0000 7fcb23400640 1 -- 172.21.15.6:0/3029923280 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb0c009be0 con 0x7fcb08004800 2024-03-04T09:31:34.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.988+0000 7fcb2b600640 1 -- 172.21.15.6:0/3029923280 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcb08004800 msgr2=0x7fcb08004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:34.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.988+0000 7fcb2b600640 1 --2- 172.21.15.6:0/3029923280 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcb08004800 0x7fcb08004c80 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7fcb0c007330 tx=0x7fcb08005830 comp rx=0 tx=0).stop 2024-03-04T09:31:34.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.988+0000 7fcb2b600640 1 -- 172.21.15.6:0/3029923280 shutdown_connections 2024-03-04T09:31:34.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.988+0000 7fcb2b600640 1 --2- 172.21.15.6:0/3029923280 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcb08004800 0x7fcb08004c80 unknown :-1 s=CLOSED pgs=1292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:34.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.988+0000 7fcb2b600640 1 -- 172.21.15.6:0/3029923280 >> 172.21.15.6:0/3029923280 conn(0x7fcb2406afb0 msgr2=0x7fcb2406b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:34.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.988+0000 7fcb2b600640 1 -- 172.21.15.6:0/3029923280 shutdown_connections 2024-03-04T09:31:34.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.988+0000 7fcb2b600640 1 -- 172.21.15.6:0/3029923280 wait complete. 2024-03-04T09:31:34.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.991+0000 7fcb2b600640 1 Processor -- start 2024-03-04T09:31:34.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.991+0000 7fcb2b600640 1 -- start start 2024-03-04T09:31:34.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.993+0000 7fcb2b600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcb08004800 0x7fcb24150810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:34.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.993+0000 7fcb2b600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcb241f8c20 0x7fcb240afa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:34.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.993+0000 7fcb2b600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcb240aff80 0x7fcb2406d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:34.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.993+0000 7fcb2b600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb2415c990 con 0x7fcb241f8c20 2024-03-04T09:31:34.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.993+0000 7fcb2b600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb2415c810 con 0x7fcb240aff80 2024-03-04T09:31:34.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.993+0000 7fcb2b600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb240b4c20 con 0x7fcb08004800 2024-03-04T09:31:34.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.993+0000 7fcb23e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcb241f8c20 0x7fcb240afa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:34.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.993+0000 7fcb23e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcb241f8c20 0x7fcb240afa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:50026/0 (socket says 172.21.15.6:50026) 2024-03-04T09:31:34.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.993+0000 7fcb23e00640 1 -- 172.21.15.6:0/2026829237 learned_addr learned my addr 172.21.15.6:0/2026829237 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:31:34.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.993+0000 7fcb28c00640 1 --2- 172.21.15.6:0/2026829237 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcb08004800 0x7fcb24150810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:34.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.993+0000 7fcb23e00640 1 -- 172.21.15.6:0/2026829237 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcb08004800 msgr2=0x7fcb24150810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:34.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.993+0000 7fcb23e00640 1 --2- 172.21.15.6:0/2026829237 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcb08004800 0x7fcb24150810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:34.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.993+0000 7fcb23e00640 1 -- 172.21.15.6:0/2026829237 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcb240aff80 msgr2=0x7fcb2406d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:34.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.993+0000 7fcb23e00640 1 --2- 172.21.15.6:0/2026829237 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcb240aff80 0x7fcb2406d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:34.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.993+0000 7fcb23e00640 1 -- 172.21.15.6:0/2026829237 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb2406dfd0 con 0x7fcb241f8c20 2024-03-04T09:31:34.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.994+0000 7fcb23e00640 1 --2- 172.21.15.6:0/2026829237 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcb241f8c20 0x7fcb240afa40 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7fcb0c000f80 tx=0x7fcb0c004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:34.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.994+0000 7fcb20c00640 1 -- 172.21.15.6:0/2026829237 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb0c042030 con 0x7fcb241f8c20 2024-03-04T09:31:34.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.994+0000 7fcb20c00640 1 -- 172.21.15.6:0/2026829237 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb0c03a040 con 0x7fcb241f8c20 2024-03-04T09:31:34.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.994+0000 7fcb2b600640 1 -- 172.21.15.6:0/2026829237 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb2406e2b0 con 0x7fcb241f8c20 2024-03-04T09:31:34.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.994+0000 7fcb20c00640 1 -- 172.21.15.6:0/2026829237 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb0c039070 con 0x7fcb241f8c20 2024-03-04T09:31:34.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.994+0000 7fcb2b600640 1 -- 172.21.15.6:0/2026829237 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb2406dca0 con 0x7fcb241f8c20 2024-03-04T09:31:34.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.995+0000 7fcb2b600640 1 -- 172.21.15.6:0/2026829237 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb24069510 con 0x7fcb241f8c20 2024-03-04T09:31:34.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.996+0000 7fcb20c00640 1 -- 172.21.15.6:0/2026829237 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcb0c045070 con 0x7fcb241f8c20 2024-03-04T09:31:34.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.997+0000 7fcb20c00640 1 --2- 172.21.15.6:0/2026829237 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fcb0006b500 0x7fcb0006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:34.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.997+0000 7fcb20c00640 1 -- 172.21.15.6:0/2026829237 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcb0c0abd40 con 0x7fcb241f8c20 2024-03-04T09:31:34.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.998+0000 7fcb28c00640 1 --2- 172.21.15.6:0/2026829237 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fcb0006b500 0x7fcb0006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:34.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:34.999+0000 7fcb28c00640 1 --2- 172.21.15.6:0/2026829237 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fcb0006b500 0x7fcb0006d9c0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fcb14000c00 tx=0x7fcb14009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:35.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.007+0000 7fcb20c00640 1 -- 172.21.15.6:0/2026829237 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcb0c04c070 con 0x7fcb241f8c20 2024-03-04T09:31:35.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.249+0000 7fcb2b600640 1 -- 172.21.15.6:0/2026829237 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcb2404f3a0 con 0x7fcb241f8c20 2024-03-04T09:31:35.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.250+0000 7fcb20c00640 1 -- 172.21.15.6:0/2026829237 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+4973 (secure 0 0 0) 0x7fcb0c07ac50 con 0x7fcb241f8c20 2024-03-04T09:31:35.247 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:31:35.247 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":77,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5792,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6836/4034557380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":4034557380},{"type":"v1","addr":"172.21.15.146:6836","nonce":4034557380}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":72},{"gid":6370,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/348873623","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":348873623},{"type":"v1","addr":"172.21.15.6:6835","nonce":348873623}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":72},{"gid":6382,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/763528142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":763528142},{"type":"v1","addr":"172.21.15.146:6837","nonce":763528142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":72}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:31:21.114351+0000","modified":"2024-03-04T09:31:24.172076+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7560},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7560":{"gid":7560,"name":"a","rank":0,"incarnation":73,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2697931210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2697931210},{"type":"v1","addr":"172.21.15.6:6834","nonce":2697931210}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7560,"qdb_cluster":[7560]},"id":8}]} 2024-03-04T09:31:35.247 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 77 2024-03-04T09:31:35.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.252+0000 7fcb2b600640 1 -- 172.21.15.6:0/2026829237 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fcb0006b500 msgr2=0x7fcb0006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:35.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.252+0000 7fcb2b600640 1 --2- 172.21.15.6:0/2026829237 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fcb0006b500 0x7fcb0006d9c0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fcb14000c00 tx=0x7fcb14009040 comp rx=0 tx=0).stop 2024-03-04T09:31:35.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.252+0000 7fcb2b600640 1 -- 172.21.15.6:0/2026829237 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcb241f8c20 msgr2=0x7fcb240afa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:35.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.252+0000 7fcb2b600640 1 --2- 172.21.15.6:0/2026829237 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcb241f8c20 0x7fcb240afa40 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7fcb0c000f80 tx=0x7fcb0c004ab0 comp rx=0 tx=0).stop 2024-03-04T09:31:35.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.252+0000 7fcb2b600640 1 -- 172.21.15.6:0/2026829237 shutdown_connections 2024-03-04T09:31:35.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.252+0000 7fcb2b600640 1 --2- 172.21.15.6:0/2026829237 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcb240aff80 0x7fcb2406d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:35.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.252+0000 7fcb2b600640 1 --2- 172.21.15.6:0/2026829237 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcb241f8c20 0x7fcb240afa40 unknown :-1 s=CLOSED pgs=1293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:35.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.252+0000 7fcb2b600640 1 --2- 172.21.15.6:0/2026829237 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fcb0006b500 0x7fcb0006d9c0 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:35.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.252+0000 7fcb2b600640 1 --2- 172.21.15.6:0/2026829237 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcb08004800 0x7fcb24150810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:35.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.252+0000 7fcb2b600640 1 -- 172.21.15.6:0/2026829237 >> 172.21.15.6:0/2026829237 conn(0x7fcb2406afb0 msgr2=0x7fcb240adff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:35.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.253+0000 7fcb2b600640 1 -- 172.21.15.6:0/2026829237 shutdown_connections 2024-03-04T09:31:35.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.253+0000 7fcb2b600640 1 -- 172.21.15.6:0/2026829237 wait complete. 2024-03-04T09:31:35.269 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json flush journal 2024-03-04T09:31:35.520 INFO:teuthology.orchestra.run.smithi006.stdout:{"message":"","return_code":0} 2024-03-04T09:31:35.520 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-04T09:31:35.521 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-04T09:31:35.521 INFO:teuthology.orchestra.run:waiting for 300 2024-03-04T09:31:35.521 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-04T09:31:35.521 INFO:teuthology.orchestra.run:waiting for 300 2024-03-04T09:31:35.521 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-04T09:31:35.522 INFO:teuthology.orchestra.run:waiting for 300 2024-03-04T09:31:35.522 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-04T09:31:35.522 INFO:teuthology.orchestra.run:waiting for 300 2024-03-04T09:31:35.541 INFO:tasks.ceph.mds.b:Stopped 2024-03-04T09:31:35.541 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail b 2024-03-04T09:31:35.543 INFO:tasks.ceph.mds.a:Stopped 2024-03-04T09:31:35.543 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail a 2024-03-04T09:31:35.547 INFO:tasks.ceph.mds.c:Stopped 2024-03-04T09:31:35.547 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail c 2024-03-04T09:31:35.560 INFO:tasks.ceph.mds.d:Stopped 2024-03-04T09:31:35.560 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail d 2024-03-04T09:31:35.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.734+0000 7f0fd5c00640 1 Processor -- start 2024-03-04T09:31:35.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.734+0000 7fe16b400640 1 Processor -- start 2024-03-04T09:31:35.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.735+0000 7f0fd5c00640 1 -- start start 2024-03-04T09:31:35.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.735+0000 7fe16b400640 1 -- start start 2024-03-04T09:31:35.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.735+0000 7fd305800640 1 Processor -- start 2024-03-04T09:31:35.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.735+0000 7fd305800640 1 -- start start 2024-03-04T09:31:35.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.736+0000 7fe16b400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fe1640b34e0 0x7fe16415c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:35.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.736+0000 7fe16b400640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe16415cc40 con 0x7fe1640af580 2024-03-04T09:31:35.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.736+0000 7fe16b400640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe16415cf20 con 0x7fe1640af1a0 2024-03-04T09:31:35.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.736+0000 7fe16b400640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe16415d200 con 0x7fe1640b34e0 2024-03-04T09:31:35.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.736+0000 7fe169400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fe1640b34e0 0x7fe16415c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:35.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.737+0000 7fe169400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fe1640b34e0 0x7fe16415c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:50036/0 (socket says 172.21.15.6:50036) 2024-03-04T09:31:35.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.737+0000 7fe169400640 1 -- 172.21.15.6:0/1793902137 learned_addr learned my addr 172.21.15.6:0/1793902137 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:31:35.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.737+0000 7fe163e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fe1640af580 0x7fe1640b2dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:52746/0 (socket says 172.21.15.6:52746) 2024-03-04T09:31:35.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.737+0000 7f0fd5c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f0fd015df00 0x7f0fd006c880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:35.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.737+0000 7f0fd5c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0fd00b0d30 0x7f0fd00af270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:35.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.737+0000 7f0fd5c00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fd00b6270 con 0x7f0fd00b1280 2024-03-04T09:31:35.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.737+0000 7f0fd5c00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fd00aeb00 con 0x7f0fd015df00 2024-03-04T09:31:35.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.737+0000 7f0fd5c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd00b6510 con 0x7f0fd00b0d30 2024-03-04T09:31:35.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.737+0000 7fe163400640 1 -- 172.21.15.6:0/1793902137 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1684773113 0 0) 0x7fe16415cc40 con 0x7fe1640af580 2024-03-04T09:31:35.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.737+0000 7fe163400640 1 -- 172.21.15.6:0/1793902137 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe14c003520 con 0x7fe1640af580 2024-03-04T09:31:35.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.737+0000 7fe169400640 1 -- 172.21.15.6:0/1793902137 >> v1:172.21.15.146:6789/0 conn(0x7fe1640af1a0 legacy=0x7fe1640b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:35.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.737+0000 7fe169400640 1 -- 172.21.15.6:0/1793902137 >> v1:172.21.15.6:6789/0 conn(0x7fe1640af580 legacy=0x7fe1640b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:35.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.737+0000 7fe169400640 1 -- 172.21.15.6:0/1793902137 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe16415d4e0 con 0x7fe1640b34e0 2024-03-04T09:31:35.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.737+0000 7fd305800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd30015e2e0 0x7fd3000d0830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:35.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.737+0000 7fd305800640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3000b3940 con 0x7fd3000b50e0 2024-03-04T09:31:35.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.737+0000 7fd305800640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3000cf580 con 0x7fd3000d0d70 2024-03-04T09:31:35.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.737+0000 7fd305800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3000d4e70 con 0x7fd30015e2e0 2024-03-04T09:31:35.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.738+0000 7fd2ff400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fd3000b50e0 0x7fd3000d2240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:52754/0 (socket says 172.21.15.6:52754) 2024-03-04T09:31:35.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.738+0000 7fd2ff400640 1 -- 172.21.15.6:0/1873080625 learned_addr learned my addr 172.21.15.6:0/1873080625 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:31:35.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.738+0000 7f0fcf400640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f0fd00b1280 0x7f0fd006cdc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.6:42396/0 (socket says 172.21.15.6:42396) 2024-03-04T09:31:35.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.738+0000 7f0fcf400640 1 -- 172.21.15.6:0/1211011211 learned_addr learned my addr 172.21.15.6:0/1211011211 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:31:35.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.738+0000 7f0fcfe00640 1 --2- 172.21.15.6:0/1211011211 >> v2:172.21.15.6:3300/0 conn(0x7f0fd015df00 0x7f0fd006c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:35.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.738+0000 7f0fd5200640 1 --2- 172.21.15.6:0/1211011211 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0fd00b0d30 0x7f0fd00af270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:35.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.738+0000 7fd2fea00640 1 --2- 172.21.15.6:0/1873080625 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd30015e2e0 0x7fd3000d0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:35.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.739+0000 7f0fcfe00640 1 -- 172.21.15.6:0/1211011211 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0fd00b0d30 msgr2=0x7f0fd00af270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:35.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.739+0000 7f0fcfe00640 1 --2- 172.21.15.6:0/1211011211 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0fd00b0d30 0x7f0fd00af270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:35.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.739+0000 7f0fcfe00640 1 -- 172.21.15.6:0/1211011211 >> v1:172.21.15.146:6789/0 conn(0x7f0fd00b1280 legacy=0x7f0fd006cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:35.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.739+0000 7f0fcfe00640 1 -- 172.21.15.6:0/1211011211 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fd00b67f0 con 0x7f0fd015df00 2024-03-04T09:31:35.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.739+0000 7f0fcea00640 1 -- 172.21.15.6:0/1211011211 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 802966556 0 0) 0x7f0fd00b6270 con 0x7f0fd00b1280 2024-03-04T09:31:35.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.739+0000 7fd2fd600640 1 -- 172.21.15.6:0/1873080625 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1044347200 0 0) 0x7fd3000b3940 con 0x7fd3000b50e0 2024-03-04T09:31:35.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.739+0000 7fd2fd600640 1 -- 172.21.15.6:0/1873080625 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2e4003520 con 0x7fd3000b50e0 2024-03-04T09:31:35.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.739+0000 7f0fd5200640 1 --2- 172.21.15.6:0/1211011211 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0fd00b0d30 0x7f0fd00af270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T09:31:35.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.739+0000 7fe169400640 1 --2- 172.21.15.6:0/1793902137 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe1640b34e0 0x7fe16415c600 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7fe15c009e50 tx=0x7fe15c02bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a8e0c6ddd6e4be0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:35.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.739+0000 7fd2fea00640 1 -- 172.21.15.6:0/1873080625 >> v1:172.21.15.146:6789/0 conn(0x7fd3000d0d70 legacy=0x7fd3000d1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:35.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.739+0000 7fd2fea00640 1 -- 172.21.15.6:0/1873080625 >> v1:172.21.15.6:6789/0 conn(0x7fd3000b50e0 legacy=0x7fd3000d2240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:35.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.739+0000 7fd2fea00640 1 -- 172.21.15.6:0/1873080625 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3000cf860 con 0x7fd30015e2e0 2024-03-04T09:31:35.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.740+0000 7fe163400640 1 -- 172.21.15.6:0/1793902137 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe15c02cb10 con 0x7fe1640b34e0 2024-03-04T09:31:35.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.740+0000 7fe163400640 1 -- 172.21.15.6:0/1793902137 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe15c02ccb0 con 0x7fe1640b34e0 2024-03-04T09:31:35.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.740+0000 7fe163400640 1 -- 172.21.15.6:0/1793902137 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe15c02ce60 con 0x7fe1640b34e0 2024-03-04T09:31:35.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.740+0000 7fe16b400640 1 -- 172.21.15.6:0/1793902137 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe1640b34e0 msgr2=0x7fe16415c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:35.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.740+0000 7fe16b400640 1 --2- 172.21.15.6:0/1793902137 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe1640b34e0 0x7fe16415c600 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7fe15c009e50 tx=0x7fe15c02bf70 comp rx=0 tx=0).stop 2024-03-04T09:31:35.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.740+0000 7fe16b400640 1 -- 172.21.15.6:0/1793902137 shutdown_connections 2024-03-04T09:31:35.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.740+0000 7fe16b400640 1 --2- 172.21.15.6:0/1793902137 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe1640b34e0 0x7fe16415c600 unknown :-1 s=CLOSED pgs=1296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:35.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.740+0000 7fe16b400640 1 -- 172.21.15.6:0/1793902137 >> 172.21.15.6:0/1793902137 conn(0x7fe16406afb0 msgr2=0x7fe16406b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:35.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.741+0000 7fe16b400640 1 -- 172.21.15.6:0/1793902137 shutdown_connections 2024-03-04T09:31:35.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.741+0000 7f0fcfe00640 1 --2- 172.21.15.6:0/1211011211 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0fd015df00 0x7f0fd006c880 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7f0fc00097a0 tx=0x7f0fc002b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=656afc5910102d00 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:35.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.741+0000 7fe16b400640 1 -- 172.21.15.6:0/1793902137 wait complete. 2024-03-04T09:31:35.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.741+0000 7f0fcea00640 1 -- 172.21.15.6:0/1211011211 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fc0031780 con 0x7f0fd015df00 2024-03-04T09:31:35.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.741+0000 7f0fcea00640 1 -- 172.21.15.6:0/1211011211 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fc0037070 con 0x7f0fd015df00 2024-03-04T09:31:35.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.741+0000 7fd2fea00640 1 --2- 172.21.15.6:0/1873080625 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd30015e2e0 0x7fd3000d0830 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fd2f00097d0 tx=0x7fd2f002b940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aa0ce6fe41f17b0a server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:35.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.741+0000 7f0fcea00640 1 -- 172.21.15.6:0/1211011211 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fc0031b40 con 0x7f0fd015df00 2024-03-04T09:31:35.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.741+0000 7fd2fd600640 1 -- 172.21.15.6:0/1873080625 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2f0034070 con 0x7fd30015e2e0 2024-03-04T09:31:35.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.742+0000 7fd2fd600640 1 -- 172.21.15.6:0/1873080625 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2f0030800 con 0x7fd30015e2e0 2024-03-04T09:31:35.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.742+0000 7fd2fd600640 1 -- 172.21.15.6:0/1873080625 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2f0030bb0 con 0x7fd30015e2e0 2024-03-04T09:31:35.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.742+0000 7f0fd5c00640 1 -- 172.21.15.6:0/1211011211 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0fd015df00 msgr2=0x7f0fd006c880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:35.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.742+0000 7f0fd5c00640 1 --2- 172.21.15.6:0/1211011211 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0fd015df00 0x7f0fd006c880 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7f0fc00097a0 tx=0x7f0fc002b820 comp rx=0 tx=0).stop 2024-03-04T09:31:35.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.742+0000 7f0fd5c00640 1 -- 172.21.15.6:0/1211011211 shutdown_connections 2024-03-04T09:31:35.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.742+0000 7f0fd5c00640 1 --2- 172.21.15.6:0/1211011211 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0fd00b0d30 0x7f0fd00af270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:35.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.742+0000 7f0fd5c00640 1 --2- 172.21.15.6:0/1211011211 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0fd015df00 0x7f0fd006c880 unknown :-1 s=CLOSED pgs=1297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:35.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.742+0000 7f0fd5c00640 1 -- 172.21.15.6:0/1211011211 >> 172.21.15.6:0/1211011211 conn(0x7f0fd009aa10 msgr2=0x7f0fd009ae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:35.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.742+0000 7f0fd5c00640 1 -- 172.21.15.6:0/1211011211 shutdown_connections 2024-03-04T09:31:35.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.742+0000 7f0fd5c00640 1 -- 172.21.15.6:0/1211011211 wait complete. 2024-03-04T09:31:35.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.743+0000 7fd305800640 1 -- 172.21.15.6:0/1873080625 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd30015e2e0 msgr2=0x7fd3000d0830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:35.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.743+0000 7fd305800640 1 --2- 172.21.15.6:0/1873080625 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd30015e2e0 0x7fd3000d0830 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fd2f00097d0 tx=0x7fd2f002b940 comp rx=0 tx=0).stop 2024-03-04T09:31:35.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.742+0000 7fe16b400640 1 Processor -- start 2024-03-04T09:31:35.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.743+0000 7fe16b400640 1 -- start start 2024-03-04T09:31:35.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.743+0000 7fd305800640 1 -- 172.21.15.6:0/1873080625 shutdown_connections 2024-03-04T09:31:35.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.743+0000 7fd305800640 1 --2- 172.21.15.6:0/1873080625 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd30015e2e0 0x7fd3000d0830 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:35.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.743+0000 7fd305800640 1 -- 172.21.15.6:0/1873080625 >> 172.21.15.6:0/1873080625 conn(0x7fd3000d3f60 msgr2=0x7fd3000d4370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:35.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.743+0000 7fd305800640 1 -- 172.21.15.6:0/1873080625 shutdown_connections 2024-03-04T09:31:35.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.743+0000 7fd305800640 1 -- 172.21.15.6:0/1873080625 wait complete. 2024-03-04T09:31:35.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.743+0000 7fe16b400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe1640af1a0 0x7fe1641f37c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:35.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.743+0000 7fe16b400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe1640af580 0x7fe1640afcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:35.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.743+0000 7fe16b400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe1640b34e0 0x7fe16415e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:35.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.744+0000 7fe16b400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe164155e70 con 0x7fe1640af1a0 2024-03-04T09:31:35.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.744+0000 7fe16b400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1640cde30 con 0x7fe1640b34e0 2024-03-04T09:31:35.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.744+0000 7fe16b400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe164155ff0 con 0x7fe1640af580 2024-03-04T09:31:35.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.744+0000 7fe168a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe1640af1a0 0x7fe1641f37c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:35.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.744+0000 7fe168a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe1640af1a0 0x7fe1641f37c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:50060/0 (socket says 172.21.15.6:50060) 2024-03-04T09:31:35.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.744+0000 7fe168a00640 1 -- 172.21.15.6:0/2702684177 learned_addr learned my addr 172.21.15.6:0/2702684177 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:31:35.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.744+0000 7fe163e00640 1 --2- 172.21.15.6:0/2702684177 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe1640af580 0x7fe1640afcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:35.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.744+0000 7fe169400640 1 --2- 172.21.15.6:0/2702684177 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe1640b34e0 0x7fe16415e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:35.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.744+0000 7fe168a00640 1 -- 172.21.15.6:0/2702684177 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe1640af580 msgr2=0x7fe1640afcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:35.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.744+0000 7fe168a00640 1 --2- 172.21.15.6:0/2702684177 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe1640af580 0x7fe1640afcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:35.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.744+0000 7fe168a00640 1 -- 172.21.15.6:0/2702684177 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe1640b34e0 msgr2=0x7fe16415e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:35.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.744+0000 7fe168a00640 1 --2- 172.21.15.6:0/2702684177 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe1640b34e0 0x7fe16415e410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:35.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.744+0000 7fe168a00640 1 -- 172.21.15.6:0/2702684177 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe16415f010 con 0x7fe1640af1a0 2024-03-04T09:31:35.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.744+0000 7fe169400640 1 --2- 172.21.15.6:0/2702684177 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe1640b34e0 0x7fe16415e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:31:35.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.745+0000 7f0fd5c00640 1 Processor -- start 2024-03-04T09:31:35.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.745+0000 7fe168a00640 1 --2- 172.21.15.6:0/2702684177 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe1640af1a0 0x7fe1641f37c0 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7fe15800c450 tx=0x7fe15800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:35.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.745+0000 7f0fd5c00640 1 -- start start 2024-03-04T09:31:35.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.745+0000 7fe160c00640 1 -- 172.21.15.6:0/2702684177 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1580052d0 con 0x7fe1640af1a0 2024-03-04T09:31:35.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.745+0000 7fe160c00640 1 -- 172.21.15.6:0/2702684177 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe158005470 con 0x7fe1640af1a0 2024-03-04T09:31:35.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.745+0000 7fe160c00640 1 -- 172.21.15.6:0/2702684177 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe158012870 con 0x7fe1640af1a0 2024-03-04T09:31:35.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.745+0000 7fe16b400640 1 -- 172.21.15.6:0/2702684177 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe16415d930 con 0x7fe1640af1a0 2024-03-04T09:31:35.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.745+0000 7fe16b400640 1 -- 172.21.15.6:0/2702684177 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe16415def0 con 0x7fe1640af1a0 2024-03-04T09:31:35.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.746+0000 7f0fd5c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0fd00b0d30 0x7f0fd01f4680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:35.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.746+0000 7f0fd5c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0fd00b1280 0x7f0fd015b690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:35.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.746+0000 7f0fd5c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0fd015df00 0x7f0fd0151030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:35.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.746+0000 7fd305800640 1 Processor -- start 2024-03-04T09:31:35.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.746+0000 7f0fd5c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd015e460 con 0x7f0fd015df00 2024-03-04T09:31:35.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.746+0000 7f0fd5c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd015e2e0 con 0x7f0fd00b1280 2024-03-04T09:31:35.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.746+0000 7f0fd5c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd00af7b0 con 0x7f0fd00b0d30 2024-03-04T09:31:35.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.746+0000 7fd305800640 1 -- start start 2024-03-04T09:31:35.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.746+0000 7f0fd5200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0fd015df00 0x7f0fd0151030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:35.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.747+0000 7f0fd5200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0fd015df00 0x7f0fd0151030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:50070/0 (socket says 172.21.15.6:50070) 2024-03-04T09:31:35.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.747+0000 7f0fd5200640 1 -- 172.21.15.6:0/1628311710 learned_addr learned my addr 172.21.15.6:0/1628311710 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:31:35.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.747+0000 7f0fcf400640 1 --2- 172.21.15.6:0/1628311710 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0fd00b1280 0x7f0fd015b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:35.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.747+0000 7f0fcfe00640 1 --2- 172.21.15.6:0/1628311710 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0fd00b0d30 0x7f0fd01f4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:35.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.747+0000 7fe16b400640 1 -- 172.21.15.6:0/2702684177 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe164069510 con 0x7fe1640af1a0 2024-03-04T09:31:35.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.747+0000 7f0fd5200640 1 -- 172.21.15.6:0/1628311710 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0fd00b0d30 msgr2=0x7f0fd01f4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:35.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.748+0000 7f0fd5200640 1 --2- 172.21.15.6:0/1628311710 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0fd00b0d30 0x7f0fd01f4680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:35.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.748+0000 7f0fd5200640 1 -- 172.21.15.6:0/1628311710 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0fd00b1280 msgr2=0x7f0fd015b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:35.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.748+0000 7f0fd5200640 1 --2- 172.21.15.6:0/1628311710 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0fd00b1280 0x7f0fd015b690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:35.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.748+0000 7f0fd5200640 1 -- 172.21.15.6:0/1628311710 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fd01f4bc0 con 0x7f0fd015df00 2024-03-04T09:31:35.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.748+0000 7f0fcfe00640 1 --2- 172.21.15.6:0/1628311710 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0fd00b0d30 0x7f0fd01f4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T09:31:35.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.748+0000 7fd305800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd3000b50e0 0x7fd3000b1230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:35.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.748+0000 7fd305800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd3000d0d70 0x7fd3001f7c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:35.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.748+0000 7fd305800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd30015e2e0 0x7fd3001f81c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:35.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.748+0000 7fe160c00640 1 -- 172.21.15.6:0/2702684177 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe158012a10 con 0x7fe1640af1a0 2024-03-04T09:31:35.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.748+0000 7fd305800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3000cfe20 con 0x7fd30015e2e0 2024-03-04T09:31:35.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.748+0000 7f0fd5200640 1 --2- 172.21.15.6:0/1628311710 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0fd015df00 0x7f0fd0151030 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7f0fc8007ef0 tx=0x7f0fc800d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:35.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.748+0000 7fd305800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3000b3c40 con 0x7fd3000d0d70 2024-03-04T09:31:35.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.748+0000 7fd305800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3000d2950 con 0x7fd3000b50e0 2024-03-04T09:31:35.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.749+0000 7fd2fea00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd3000b50e0 0x7fd3000b1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:35.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.749+0000 7fd2ff400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd30015e2e0 0x7fd3001f81c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:35.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.749+0000 7fd2fea00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd3000b50e0 0x7fd3000b1230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.6:57934/0 (socket says 172.21.15.6:57934) 2024-03-04T09:31:35.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.749+0000 7fd2fea00640 1 -- 172.21.15.6:0/1372647004 learned_addr learned my addr 172.21.15.6:0/1372647004 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:31:35.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.749+0000 7fd2fe000640 1 --2- 172.21.15.6:0/1372647004 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd3000d0d70 0x7fd3001f7c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:35.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.749+0000 7fd2ff400640 1 -- 172.21.15.6:0/1372647004 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd3000b50e0 msgr2=0x7fd3000b1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:35.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.749+0000 7fd2ff400640 1 --2- 172.21.15.6:0/1372647004 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd3000b50e0 0x7fd3000b1230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:35.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.749+0000 7fd2ff400640 1 -- 172.21.15.6:0/1372647004 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd3000d0d70 msgr2=0x7fd3001f7c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:35.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.749+0000 7fd2ff400640 1 --2- 172.21.15.6:0/1372647004 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd3000d0d70 0x7fd3001f7c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:35.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.749+0000 7fd2ff400640 1 -- 172.21.15.6:0/1372647004 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3001f88b0 con 0x7fd30015e2e0 2024-03-04T09:31:35.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.749+0000 7fd2fe000640 1 --2- 172.21.15.6:0/1372647004 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd3000d0d70 0x7fd3001f7c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T09:31:35.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.750+0000 7fe160c00640 1 --2- 172.21.15.6:0/2702684177 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fe14406b4d0 0x7fe14406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:35.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.750+0000 7fd2ff400640 1 --2- 172.21.15.6:0/1372647004 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd30015e2e0 0x7fd3001f81c0 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7fd2f800ca50 tx=0x7fd2f800cf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:35.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.750+0000 7fe160c00640 1 -- 172.21.15.6:0/2702684177 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe158089f60 con 0x7fe1640af1a0 2024-03-04T09:31:35.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.750+0000 7fe163e00640 1 --2- 172.21.15.6:0/2702684177 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fe14406b4d0 0x7fe14406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:35.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.751+0000 7fe163e00640 1 --2- 172.21.15.6:0/2702684177 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fe14406b4d0 0x7fe14406d990 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fe1500010e0 tx=0x7fe150009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:35.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.754+0000 7fd2f6a00640 1 -- 172.21.15.6:0/1372647004 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2f8017070 con 0x7fd30015e2e0 2024-03-04T09:31:35.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.754+0000 7fd2f6a00640 1 -- 172.21.15.6:0/1372647004 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2f800abb0 con 0x7fd30015e2e0 2024-03-04T09:31:35.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.754+0000 7fd2f6a00640 1 -- 172.21.15.6:0/1372647004 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2f8012400 con 0x7fd30015e2e0 2024-03-04T09:31:35.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.754+0000 7f0fc7e00640 1 -- 172.21.15.6:0/1628311710 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fc801a070 con 0x7f0fd015df00 2024-03-04T09:31:35.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.754+0000 7f0fc7e00640 1 -- 172.21.15.6:0/1628311710 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fc8003b90 con 0x7f0fd015df00 2024-03-04T09:31:35.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.754+0000 7f0fc7e00640 1 -- 172.21.15.6:0/1628311710 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fc8004030 con 0x7f0fd015df00 2024-03-04T09:31:35.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.755+0000 7fe160c00640 1 -- 172.21.15.6:0/2702684177 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe158058cc0 con 0x7fe1640af1a0 2024-03-04T09:31:35.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.755+0000 7f0fd5c00640 1 -- 172.21.15.6:0/1628311710 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fd014f5c0 con 0x7f0fd015df00 2024-03-04T09:31:35.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.755+0000 7f0fd5c00640 1 -- 172.21.15.6:0/1628311710 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fd014fb80 con 0x7f0fd015df00 2024-03-04T09:31:35.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.755+0000 7fd305800640 1 -- 172.21.15.6:0/1372647004 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3000c8bf0 con 0x7fd30015e2e0 2024-03-04T09:31:35.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.755+0000 7fd305800640 1 -- 172.21.15.6:0/1372647004 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3000c91b0 con 0x7fd30015e2e0 2024-03-04T09:31:35.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.757+0000 7f0fd5c00640 1 -- 172.21.15.6:0/1628311710 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fd00b48e0 con 0x7f0fd015df00 2024-03-04T09:31:35.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.757+0000 7fd305800640 1 -- 172.21.15.6:0/1372647004 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd30004f3a0 con 0x7fd30015e2e0 2024-03-04T09:31:35.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.758+0000 7f0fc7e00640 1 -- 172.21.15.6:0/1628311710 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0fc801d070 con 0x7f0fd015df00 2024-03-04T09:31:35.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.759+0000 7f0fc7e00640 1 --2- 172.21.15.6:0/1628311710 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f0fb006b520 0x7f0fb006d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:35.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.759+0000 7f0fc7e00640 1 -- 172.21.15.6:0/1628311710 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0fc808a3c0 con 0x7f0fd015df00 2024-03-04T09:31:35.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.759+0000 7fd2f6a00640 1 -- 172.21.15.6:0/1372647004 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd2f801a070 con 0x7fd30015e2e0 2024-03-04T09:31:35.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.760+0000 7fd2f6a00640 1 --2- 172.21.15.6:0/1372647004 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd2e006b520 0x7fd2e006d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:35.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.760+0000 7fd2f6a00640 1 -- 172.21.15.6:0/1372647004 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd2f8089d10 con 0x7fd30015e2e0 2024-03-04T09:31:35.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.761+0000 7f0fcfe00640 1 --2- 172.21.15.6:0/1628311710 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f0fb006b520 0x7f0fb006d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:35.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.761+0000 7fd2fea00640 1 --2- 172.21.15.6:0/1372647004 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd2e006b520 0x7fd2e006d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:35.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.763+0000 7f0fcfe00640 1 --2- 172.21.15.6:0/1628311710 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f0fb006b520 0x7f0fb006d9e0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f0fc0036000 tx=0x7f0fc003d040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:35.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.763+0000 7fd2fea00640 1 --2- 172.21.15.6:0/1372647004 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd2e006b520 0x7fd2e006d9e0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fd2f0000c00 tx=0x7fd2f0008030 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:35.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.764+0000 7f0fc7e00640 1 -- 172.21.15.6:0/1628311710 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0fd00b48e0 con 0x7f0fd015df00 2024-03-04T09:31:35.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.772+0000 7fd2f6a00640 1 -- 172.21.15.6:0/1372647004 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd2f80586b0 con 0x7fd30015e2e0 2024-03-04T09:31:35.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.793+0000 7f7a48200640 1 Processor -- start 2024-03-04T09:31:35.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.793+0000 7f7a48200640 1 -- start start 2024-03-04T09:31:35.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.795+0000 7f7a48200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f7a4015df70 0x7f7a40070140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:35.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.795+0000 7f7a48200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7a400b6230 0x7f7a400b6650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:35.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.795+0000 7f7a48200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a4006f0a0 con 0x7f7a40161cc0 2024-03-04T09:31:35.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.795+0000 7f7a48200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a4006fcb0 con 0x7f7a4015df70 2024-03-04T09:31:35.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.795+0000 7f7a48200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a4006f380 con 0x7f7a400b6230 2024-03-04T09:31:35.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.795+0000 7f7a45800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f7a4015df70 0x7f7a40070140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:35.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.796+0000 7f7a45800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f7a4015df70 0x7f7a40070140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:50074/0 (socket says 172.21.15.6:50074) 2024-03-04T09:31:35.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.796+0000 7f7a45800640 1 -- 172.21.15.6:0/2301164390 learned_addr learned my addr 172.21.15.6:0/2301164390 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:31:35.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.796+0000 7f7a45800640 1 -- 172.21.15.6:0/2301164390 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7a400b6230 msgr2=0x7f7a400b6650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:35.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.796+0000 7f7a45800640 1 --2- 172.21.15.6:0/2301164390 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7a400b6230 0x7f7a400b6650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:35.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.796+0000 7f7a45800640 1 -- 172.21.15.6:0/2301164390 >> v1:172.21.15.146:6789/0 conn(0x7f7a40161cc0 legacy=0x7f7a4006c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:35.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.796+0000 7f7a45800640 1 -- 172.21.15.6:0/2301164390 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a4006f660 con 0x7f7a4015df70 2024-03-04T09:31:35.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.798+0000 7f7a45800640 1 --2- 172.21.15.6:0/2301164390 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7a4015df70 0x7f7a40070140 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7f7a34009870 tx=0x7f7a3402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35a069d81688dabe server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:35.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.798+0000 7f7a3fe00640 1 -- 172.21.15.6:0/2301164390 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a34035020 con 0x7f7a4015df70 2024-03-04T09:31:35.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.798+0000 7f7a3fe00640 1 -- 172.21.15.6:0/2301164390 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a34031640 con 0x7f7a4015df70 2024-03-04T09:31:35.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.798+0000 7f7a3fe00640 1 -- 172.21.15.6:0/2301164390 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a3402d040 con 0x7f7a4015df70 2024-03-04T09:31:35.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.799+0000 7f7a48200640 1 -- 172.21.15.6:0/2301164390 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7a4015df70 msgr2=0x7f7a40070140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:35.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.799+0000 7f7a48200640 1 --2- 172.21.15.6:0/2301164390 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7a4015df70 0x7f7a40070140 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7f7a34009870 tx=0x7f7a3402b8f0 comp rx=0 tx=0).stop 2024-03-04T09:31:35.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.799+0000 7f7a48200640 1 -- 172.21.15.6:0/2301164390 shutdown_connections 2024-03-04T09:31:35.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.799+0000 7f7a48200640 1 --2- 172.21.15.6:0/2301164390 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7a400b6230 0x7f7a400b6650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:35.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.799+0000 7f7a48200640 1 --2- 172.21.15.6:0/2301164390 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7a4015df70 0x7f7a40070140 unknown :-1 s=CLOSED pgs=1301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:35.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.799+0000 7f7a48200640 1 -- 172.21.15.6:0/2301164390 >> 172.21.15.6:0/2301164390 conn(0x7f7a4009aa10 msgr2=0x7f7a4009ae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:35.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.799+0000 7f7a48200640 1 -- 172.21.15.6:0/2301164390 shutdown_connections 2024-03-04T09:31:35.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.799+0000 7f7a48200640 1 -- 172.21.15.6:0/2301164390 wait complete. 2024-03-04T09:31:35.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.802+0000 7f7a48200640 1 Processor -- start 2024-03-04T09:31:35.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.802+0000 7f7a48200640 1 -- start start 2024-03-04T09:31:35.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.804+0000 7f7a48200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7a400b6230 0x7f7a40143310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:35.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.804+0000 7f7a48200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7a4015df70 0x7f7a4015d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:35.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.804+0000 7f7a48200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7a40161cc0 0x7f7a4015a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:35.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.804+0000 7f7a48200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a40070800 con 0x7f7a4015df70 2024-03-04T09:31:35.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.804+0000 7f7a48200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a40070680 con 0x7f7a400b6230 2024-03-04T09:31:35.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.804+0000 7f7a48200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a4015f360 con 0x7f7a40161cc0 2024-03-04T09:31:35.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.804+0000 7f7a44e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7a4015df70 0x7f7a4015d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:35.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.804+0000 7f7a44e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7a4015df70 0x7f7a4015d0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:50084/0 (socket says 172.21.15.6:50084) 2024-03-04T09:31:35.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.804+0000 7f7a44e00640 1 -- 172.21.15.6:0/4285533982 learned_addr learned my addr 172.21.15.6:0/4285533982 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:31:35.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.805+0000 7f7a45800640 1 --2- 172.21.15.6:0/4285533982 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7a400b6230 0x7f7a40143310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:35.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.805+0000 7f7a46200640 1 --2- 172.21.15.6:0/4285533982 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7a40161cc0 0x7f7a4015a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:35.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.805+0000 7f7a44e00640 1 -- 172.21.15.6:0/4285533982 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7a40161cc0 msgr2=0x7f7a4015a070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:35.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.805+0000 7f7a44e00640 1 --2- 172.21.15.6:0/4285533982 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7a40161cc0 0x7f7a4015a070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:35.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.805+0000 7f7a44e00640 1 -- 172.21.15.6:0/4285533982 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7a400b6230 msgr2=0x7f7a40143310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:35.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.805+0000 7f7a44e00640 1 --2- 172.21.15.6:0/4285533982 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7a400b6230 0x7f7a40143310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:35.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.805+0000 7f7a44e00640 1 -- 172.21.15.6:0/4285533982 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a400b2d30 con 0x7f7a4015df70 2024-03-04T09:31:35.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.806+0000 7f7a44e00640 1 --2- 172.21.15.6:0/4285533982 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7a4015df70 0x7f7a4015d0b0 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7f7a2c00b4a0 tx=0x7f7a2c00b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:35.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.806+0000 7f7a3d600640 1 -- 172.21.15.6:0/4285533982 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a2c014070 con 0x7f7a4015df70 2024-03-04T09:31:35.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.806+0000 7f7a48200640 1 -- 172.21.15.6:0/4285533982 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a400b3010 con 0x7f7a4015df70 2024-03-04T09:31:35.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.806+0000 7f7a48200640 1 -- 172.21.15.6:0/4285533982 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a400b35d0 con 0x7f7a4015df70 2024-03-04T09:31:35.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.807+0000 7f7a3d600640 1 -- 172.21.15.6:0/4285533982 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a2c010610 con 0x7f7a4015df70 2024-03-04T09:31:35.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.807+0000 7f7a3d600640 1 -- 172.21.15.6:0/4285533982 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a2c010a00 con 0x7f7a4015df70 2024-03-04T09:31:35.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.808+0000 7f7a48200640 1 -- 172.21.15.6:0/4285533982 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a1c005290 con 0x7f7a4015df70 2024-03-04T09:31:35.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.810+0000 7f7a3d600640 1 -- 172.21.15.6:0/4285533982 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7a2c017070 con 0x7f7a4015df70 2024-03-04T09:31:35.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.811+0000 7f7a3d600640 1 --2- 172.21.15.6:0/4285533982 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f7a20073ce0 0x7f7a200761a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:35.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.811+0000 7f7a3d600640 1 -- 172.21.15.6:0/4285533982 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7a2c089c80 con 0x7f7a4015df70 2024-03-04T09:31:35.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.812+0000 7f7a45800640 1 --2- 172.21.15.6:0/4285533982 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f7a20073ce0 0x7f7a200761a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:35.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.813+0000 7f7a45800640 1 --2- 172.21.15.6:0/4285533982 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f7a20073ce0 0x7f7a200761a0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f7a34036000 tx=0x7f7a3403c040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:35.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:35.821+0000 7f7a3d600640 1 -- 172.21.15.6:0/4285533982 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7a2c058a70 con 0x7f7a4015df70 2024-03-04T09:31:36.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:36.046+0000 7f0fd5c00640 1 -- 172.21.15.6:0/1628311710 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "b"} v 0) v1 -- 0x7f0fd004f330 con 0x7f0fd015df00 2024-03-04T09:31:36.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:36.049+0000 7fe16b400640 1 -- 172.21.15.6:0/2702684177 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "c"} v 0) v1 -- 0x7fe16404f3a0 con 0x7fe1640af1a0 2024-03-04T09:31:36.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:36.068+0000 7fd305800640 1 -- 172.21.15.6:0/1372647004 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "a"} v 0) v1 -- 0x7fd3000c93f0 con 0x7fd30015e2e0 2024-03-04T09:31:36.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:36.106+0000 7f7a48200640 1 -- 172.21.15.6:0/4285533982 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "d"} v 0) v1 -- 0x7f7a1c005580 con 0x7f7a4015df70 2024-03-04T09:31:36.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:36.721+0000 7f0fc7e00640 1 -- 172.21.15.6:0/1628311710 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "b"}]=0 failed mds gid 5792 v78) v1 ==== 95+0+0 (secure 0 0 0) 0x7f0fd01f4bc0 con 0x7f0fd015df00 2024-03-04T09:31:36.718 INFO:teuthology.orchestra.run.smithi006.stderr:failed mds gid 5792 2024-03-04T09:31:36.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:36.724+0000 7f0fd5c00640 1 -- 172.21.15.6:0/1628311710 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f0fb006b520 msgr2=0x7f0fb006d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:36.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:36.724+0000 7f0fd5c00640 1 --2- 172.21.15.6:0/1628311710 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f0fb006b520 0x7f0fb006d9e0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f0fc0036000 tx=0x7f0fc003d040 comp rx=0 tx=0).stop 2024-03-04T09:31:36.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:36.724+0000 7f0fd5c00640 1 -- 172.21.15.6:0/1628311710 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0fd015df00 msgr2=0x7f0fd0151030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:36.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:36.724+0000 7f0fd5c00640 1 --2- 172.21.15.6:0/1628311710 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0fd015df00 0x7f0fd0151030 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7f0fc8007ef0 tx=0x7f0fc800d4a0 comp rx=0 tx=0).stop 2024-03-04T09:31:36.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:36.725+0000 7f0fd5c00640 1 -- 172.21.15.6:0/1628311710 shutdown_connections 2024-03-04T09:31:36.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:36.725+0000 7f0fd5c00640 1 --2- 172.21.15.6:0/1628311710 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0fd015df00 0x7f0fd0151030 unknown :-1 s=CLOSED pgs=1299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:36.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:36.725+0000 7f0fd5c00640 1 --2- 172.21.15.6:0/1628311710 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0fd00b1280 0x7f0fd015b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:36.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:36.725+0000 7f0fd5c00640 1 --2- 172.21.15.6:0/1628311710 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f0fb006b520 0x7f0fb006d9e0 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:36.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:36.725+0000 7f0fd5c00640 1 --2- 172.21.15.6:0/1628311710 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0fd00b0d30 0x7f0fd01f4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:36.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:36.725+0000 7f0fd5c00640 1 -- 172.21.15.6:0/1628311710 >> 172.21.15.6:0/1628311710 conn(0x7f0fd009aa10 msgr2=0x7f0fd00b3390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:36.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:36.725+0000 7f0fd5c00640 1 -- 172.21.15.6:0/1628311710 shutdown_connections 2024-03-04T09:31:36.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:36.725+0000 7f0fd5c00640 1 -- 172.21.15.6:0/1628311710 wait complete. 2024-03-04T09:31:36.734 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-03-04T09:31:36.737+0000 7f1603800640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-04T09:31:36.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:36.743+0000 7fe160c00640 1 -- 172.21.15.6:0/2702684177 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "c"}]=0 failed mds gid 6370 v79) v1 ==== 95+0+0 (secure 0 0 0) 0x7fe1580585c0 con 0x7fe1640af1a0 2024-03-04T09:31:36.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:36.743+0000 7fd2f6a00640 1 -- 172.21.15.6:0/1372647004 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "a"}]=0 failed mds gid 7560 v79) v1 ==== 95+0+0 (secure 0 0 0) 0x7fd2f805c750 con 0x7fd30015e2e0 2024-03-04T09:31:36.741 INFO:teuthology.orchestra.run.smithi006.stderr:failed mds gid 6370 2024-03-04T09:31:36.741 INFO:teuthology.orchestra.run.smithi006.stderr:failed mds gid 7560 2024-03-04T09:31:36.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:36.747+0000 7fe16b400640 1 -- 172.21.15.6:0/2702684177 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fe14406b4d0 msgr2=0x7fe14406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:36.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:36.747+0000 7fe16b400640 1 --2- 172.21.15.6:0/2702684177 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fe14406b4d0 0x7fe14406d990 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fe1500010e0 tx=0x7fe150009040 comp rx=0 tx=0).stop 2024-03-04T09:31:36.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:36.747+0000 7fe16b400640 1 -- 172.21.15.6:0/2702684177 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe1640af1a0 msgr2=0x7fe1641f37c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:36.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:36.747+0000 7fe16b400640 1 --2- 172.21.15.6:0/2702684177 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe1640af1a0 0x7fe1641f37c0 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7fe15800c450 tx=0x7fe15800c910 comp rx=0 tx=0).stop 2024-03-04T09:31:36.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:36.748+0000 7fe16b400640 1 -- 172.21.15.6:0/2702684177 shutdown_connections 2024-03-04T09:31:36.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:36.748+0000 7fe16b400640 1 --2- 172.21.15.6:0/2702684177 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe1640b34e0 0x7fe16415e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:36.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:36.748+0000 7fe16b400640 1 --2- 172.21.15.6:0/2702684177 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fe14406b4d0 0x7fe14406d990 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:36.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:36.748+0000 7fe16b400640 1 --2- 172.21.15.6:0/2702684177 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe1640af580 0x7fe1640afcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:36.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:36.748+0000 7fe16b400640 1 --2- 172.21.15.6:0/2702684177 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe1640af1a0 0x7fe1641f37c0 unknown :-1 s=CLOSED pgs=1298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:36.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:36.748+0000 7fe16b400640 1 -- 172.21.15.6:0/2702684177 >> 172.21.15.6:0/2702684177 conn(0x7fe16406afb0 msgr2=0x7fe164151600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:36.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:36.749+0000 7fe16b400640 1 -- 172.21.15.6:0/2702684177 shutdown_connections 2024-03-04T09:31:36.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:36.749+0000 7fe16b400640 1 -- 172.21.15.6:0/2702684177 wait complete. 2024-03-04T09:31:36.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:36.750+0000 7fd305800640 1 -- 172.21.15.6:0/1372647004 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd2e006b520 msgr2=0x7fd2e006d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:36.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:36.750+0000 7fd305800640 1 --2- 172.21.15.6:0/1372647004 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd2e006b520 0x7fd2e006d9e0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fd2f0000c00 tx=0x7fd2f0008030 comp rx=0 tx=0).stop 2024-03-04T09:31:36.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:36.750+0000 7fd305800640 1 -- 172.21.15.6:0/1372647004 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd30015e2e0 msgr2=0x7fd3001f81c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:36.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:36.750+0000 7fd305800640 1 --2- 172.21.15.6:0/1372647004 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd30015e2e0 0x7fd3001f81c0 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7fd2f800ca50 tx=0x7fd2f800cf10 comp rx=0 tx=0).stop 2024-03-04T09:31:36.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:36.750+0000 7fd305800640 1 -- 172.21.15.6:0/1372647004 shutdown_connections 2024-03-04T09:31:36.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:36.750+0000 7fd305800640 1 --2- 172.21.15.6:0/1372647004 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd30015e2e0 0x7fd3001f81c0 unknown :-1 s=CLOSED pgs=1300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:36.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:36.750+0000 7fd305800640 1 --2- 172.21.15.6:0/1372647004 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd3000d0d70 0x7fd3001f7c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:36.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:36.750+0000 7fd305800640 1 --2- 172.21.15.6:0/1372647004 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd2e006b520 0x7fd2e006d9e0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:36.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:36.750+0000 7fd305800640 1 --2- 172.21.15.6:0/1372647004 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd3000b50e0 0x7fd3000b1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:36.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:36.750+0000 7fd305800640 1 -- 172.21.15.6:0/1372647004 >> 172.21.15.6:0/1372647004 conn(0x7fd3000d3f60 msgr2=0x7fd3000cea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:36.749 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-04T09:31:36.749 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-04T09:31:36.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:36.751+0000 7fd305800640 1 -- 172.21.15.6:0/1372647004 shutdown_connections 2024-03-04T09:31:36.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:36.751+0000 7fd305800640 1 -- 172.21.15.6:0/1372647004 wait complete. 2024-03-04T09:31:36.759 INFO:tasks.ceph.mds.b:Started 2024-03-04T09:31:36.759 DEBUG:teuthology.parallel:result is None 2024-03-04T09:31:36.769 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-04T09:31:36.769 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-04T09:31:36.771 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-04T09:31:36.771 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-04T09:31:36.817 INFO:tasks.ceph.mds.c:Started 2024-03-04T09:31:36.818 DEBUG:teuthology.parallel:result is None 2024-03-04T09:31:36.824 INFO:tasks.ceph.mds.a:Started 2024-03-04T09:31:36.824 DEBUG:teuthology.parallel:result is None 2024-03-04T09:31:36.930 INFO:tasks.ceph.mds.b.smithi146.stdout:starting mds.b at 2024-03-04T09:31:36.971 INFO:tasks.ceph.mds.c.smithi006.stdout:starting mds.c at 2024-03-04T09:31:36.980 INFO:tasks.ceph.mds.a.smithi006.stdout:starting mds.a at 2024-03-04T09:31:37.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:37.747+0000 7f7a3d600640 1 -- 172.21.15.6:0/4285533982 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "d"}]=0 failed mds gid 6382 v80) v1 ==== 95+0+0 (secure 0 0 0) 0x7f7a2c058370 con 0x7f7a4015df70 2024-03-04T09:31:37.745 INFO:teuthology.orchestra.run.smithi006.stderr:failed mds gid 6382 2024-03-04T09:31:37.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:37.751+0000 7f7a48200640 1 -- 172.21.15.6:0/4285533982 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f7a20073ce0 msgr2=0x7f7a200761a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:37.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:37.751+0000 7f7a48200640 1 --2- 172.21.15.6:0/4285533982 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f7a20073ce0 0x7f7a200761a0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f7a34036000 tx=0x7f7a3403c040 comp rx=0 tx=0).stop 2024-03-04T09:31:37.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:37.751+0000 7f7a48200640 1 -- 172.21.15.6:0/4285533982 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7a4015df70 msgr2=0x7f7a4015d0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:37.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:37.751+0000 7f7a48200640 1 --2- 172.21.15.6:0/4285533982 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7a4015df70 0x7f7a4015d0b0 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7f7a2c00b4a0 tx=0x7f7a2c00b960 comp rx=0 tx=0).stop 2024-03-04T09:31:37.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:37.751+0000 7f7a48200640 1 -- 172.21.15.6:0/4285533982 shutdown_connections 2024-03-04T09:31:37.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:37.751+0000 7f7a48200640 1 --2- 172.21.15.6:0/4285533982 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f7a20073ce0 0x7f7a200761a0 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:37.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:37.751+0000 7f7a48200640 1 --2- 172.21.15.6:0/4285533982 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7a40161cc0 0x7f7a4015a070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:37.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:37.751+0000 7f7a48200640 1 --2- 172.21.15.6:0/4285533982 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7a4015df70 0x7f7a4015d0b0 unknown :-1 s=CLOSED pgs=1302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:37.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:37.751+0000 7f7a48200640 1 --2- 172.21.15.6:0/4285533982 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7a400b6230 0x7f7a40143310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:37.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:37.751+0000 7f7a48200640 1 -- 172.21.15.6:0/4285533982 >> 172.21.15.6:0/4285533982 conn(0x7f7a4009aa10 msgr2=0x7f7a4015f720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:37.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:37.751+0000 7f7a48200640 1 -- 172.21.15.6:0/4285533982 shutdown_connections 2024-03-04T09:31:37.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:37.751+0000 7f7a48200640 1 -- 172.21.15.6:0/4285533982 wait complete. 2024-03-04T09:31:37.772 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-04T09:31:37.772 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-04T09:31:37.778 INFO:tasks.ceph.mds.d:Started 2024-03-04T09:31:37.779 DEBUG:teuthology.parallel:result is None 2024-03-04T09:31:37.779 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:31:37.956 INFO:tasks.ceph.mds.d.smithi146.stdout:starting mds.d at 2024-03-04T09:31:38.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:38.023+0000 7f6815400640 1 Processor -- start 2024-03-04T09:31:38.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:38.024+0000 7f6815400640 1 -- start start 2024-03-04T09:31:38.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:38.025+0000 7f6815400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f6810161b40 0x7f68100cfd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:38.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:38.025+0000 7f6815400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f68100d2930 0x7f68100b3740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:38.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:38.025+0000 7f6815400640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68100cea50 con 0x7f68100b5060 2024-03-04T09:31:38.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:38.025+0000 7f6815400640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68100cf900 con 0x7f68100d2930 2024-03-04T09:31:38.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:38.025+0000 7f6815400640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68100cecf0 con 0x7f6810161b40 2024-03-04T09:31:38.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:38.025+0000 7f680de00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f68100b5060 0x7f68100d2220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:52786/0 (socket says 172.21.15.6:52786) 2024-03-04T09:31:38.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:38.026+0000 7f680de00640 1 -- 172.21.15.6:0/1199786448 learned_addr learned my addr 172.21.15.6:0/1199786448 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:31:38.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:38.026+0000 7f680f200640 1 --2- 172.21.15.6:0/1199786448 >> v2:172.21.15.6:3300/0 conn(0x7f68100d2930 0x7f68100b3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:38.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:38.026+0000 7f680e800640 1 --2- 172.21.15.6:0/1199786448 >> v2:172.21.15.146:3300/0 conn(0x7f6810161b40 0x7f68100cfd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:38.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:38.026+0000 7f680f200640 1 -- 172.21.15.6:0/1199786448 >> v2:172.21.15.146:3300/0 conn(0x7f6810161b40 msgr2=0x7f68100cfd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:38.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:38.026+0000 7f680f200640 1 --2- 172.21.15.6:0/1199786448 >> v2:172.21.15.146:3300/0 conn(0x7f6810161b40 0x7f68100cfd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:38.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:38.026+0000 7f680f200640 1 -- 172.21.15.6:0/1199786448 >> v1:172.21.15.6:6789/0 conn(0x7f68100b5060 legacy=0x7f68100d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:38.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:38.026+0000 7f680f200640 1 -- 172.21.15.6:0/1199786448 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68100cefd0 con 0x7f68100d2930 2024-03-04T09:31:38.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:38.027+0000 7f680f200640 1 --2- 172.21.15.6:0/1199786448 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f68100d2930 0x7f68100b3740 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7f681008db00 tx=0x7f680802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1218edbafcdbbd45 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:38.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:38.028+0000 7f680d400640 1 -- 172.21.15.6:0/1199786448 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6808035020 con 0x7f68100d2930 2024-03-04T09:31:38.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:38.028+0000 7f680d400640 1 -- 172.21.15.6:0/1199786448 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6808031830 con 0x7f68100d2930 2024-03-04T09:31:38.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:38.028+0000 7f680d400640 1 -- 172.21.15.6:0/1199786448 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6808031c00 con 0x7f68100d2930 2024-03-04T09:31:38.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:38.028+0000 7f6815400640 1 -- 172.21.15.6:0/1199786448 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f68100d2930 msgr2=0x7f68100b3740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:38.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:38.028+0000 7f6815400640 1 --2- 172.21.15.6:0/1199786448 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f68100d2930 0x7f68100b3740 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7f681008db00 tx=0x7f680802b8d0 comp rx=0 tx=0).stop 2024-03-04T09:31:38.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:38.028+0000 7f6815400640 1 -- 172.21.15.6:0/1199786448 shutdown_connections 2024-03-04T09:31:38.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:38.028+0000 7f6815400640 1 --2- 172.21.15.6:0/1199786448 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f68100d2930 0x7f68100b3740 unknown :-1 s=CLOSED pgs=1309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:38.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:38.028+0000 7f6815400640 1 --2- 172.21.15.6:0/1199786448 >> v2:172.21.15.146:3300/0 conn(0x7f6810161b40 0x7f68100cfd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:38.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:38.028+0000 7f6815400640 1 -- 172.21.15.6:0/1199786448 >> 172.21.15.6:0/1199786448 conn(0x7f681009d220 msgr2=0x7f68100d3ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:38.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:38.028+0000 7f6815400640 1 -- 172.21.15.6:0/1199786448 shutdown_connections 2024-03-04T09:31:38.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:38.028+0000 7f6815400640 1 -- 172.21.15.6:0/1199786448 wait complete. 2024-03-04T09:31:38.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:38.030+0000 7f6815400640 1 Processor -- start 2024-03-04T09:31:38.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:38.030+0000 7f6815400640 1 -- start start 2024-03-04T09:31:38.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:38.031+0000 7f6815400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f68100b5060 0x7f68100b6b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:38.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:38.031+0000 7f6815400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f68100d2930 0x7f68101557c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:38.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:38.031+0000 7f6815400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6810161b40 0x7f68100ca1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:38.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:38.031+0000 7f6815400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68100b2ff0 con 0x7f68100d2930 2024-03-04T09:31:38.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:38.031+0000 7f6815400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68100cf590 con 0x7f68100b5060 2024-03-04T09:31:38.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:38.031+0000 7f6815400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68100b3170 con 0x7f6810161b40 2024-03-04T09:31:38.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:38.031+0000 7f680de00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f68100d2930 0x7f68101557c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:38.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:38.031+0000 7f680de00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f68100d2930 0x7f68101557c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:50134/0 (socket says 172.21.15.6:50134) 2024-03-04T09:31:38.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:38.031+0000 7f680de00640 1 -- 172.21.15.6:0/2876465266 learned_addr learned my addr 172.21.15.6:0/2876465266 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:31:38.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:38.032+0000 7f680e800640 1 --2- 172.21.15.6:0/2876465266 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f68100b5060 0x7f68100b6b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:38.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:38.032+0000 7f680f200640 1 --2- 172.21.15.6:0/2876465266 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6810161b40 0x7f68100ca1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:38.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:38.032+0000 7f680de00640 1 -- 172.21.15.6:0/2876465266 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6810161b40 msgr2=0x7f68100ca1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:38.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:38.032+0000 7f680de00640 1 --2- 172.21.15.6:0/2876465266 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6810161b40 0x7f68100ca1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:38.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:38.032+0000 7f680de00640 1 -- 172.21.15.6:0/2876465266 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f68100b5060 msgr2=0x7f68100b6b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:38.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:38.032+0000 7f680de00640 1 --2- 172.21.15.6:0/2876465266 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f68100b5060 0x7f68100b6b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:38.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:38.032+0000 7f680de00640 1 -- 172.21.15.6:0/2876465266 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68101597f0 con 0x7f68100d2930 2024-03-04T09:31:38.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:38.032+0000 7f680f200640 1 --2- 172.21.15.6:0/2876465266 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6810161b40 0x7f68100ca1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:31:38.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:38.032+0000 7f680de00640 1 --2- 172.21.15.6:0/2876465266 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f68100d2930 0x7f68101557c0 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7f67f800c430 tx=0x7f67f800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:38.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:38.032+0000 7f6806a00640 1 -- 172.21.15.6:0/2876465266 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67f8017070 con 0x7f68100d2930 2024-03-04T09:31:38.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:38.033+0000 7f6806a00640 1 -- 172.21.15.6:0/2876465266 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67f8005ce0 con 0x7f68100d2930 2024-03-04T09:31:38.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:38.033+0000 7f6806a00640 1 -- 172.21.15.6:0/2876465266 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67f80072f0 con 0x7f68100d2930 2024-03-04T09:31:38.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:38.033+0000 7f6815400640 1 -- 172.21.15.6:0/2876465266 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6810159ad0 con 0x7f68100d2930 2024-03-04T09:31:38.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:38.033+0000 7f6815400640 1 -- 172.21.15.6:0/2876465266 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f681015a090 con 0x7f68100d2930 2024-03-04T09:31:38.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:38.034+0000 7f6815400640 1 -- 172.21.15.6:0/2876465266 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f681004f3a0 con 0x7f68100d2930 2024-03-04T09:31:38.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:38.035+0000 7f6806a00640 1 -- 172.21.15.6:0/2876465266 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f67f801a070 con 0x7f68100d2930 2024-03-04T09:31:38.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:38.036+0000 7f6806a00640 1 --2- 172.21.15.6:0/2876465266 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f67ec06b4d0 0x7f67ec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:38.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:38.036+0000 7f6806a00640 1 -- 172.21.15.6:0/2876465266 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f67f8089720 con 0x7f68100d2930 2024-03-04T09:31:38.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:38.037+0000 7f680e800640 1 --2- 172.21.15.6:0/2876465266 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f67ec06b4d0 0x7f67ec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:38.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:38.038+0000 7f680e800640 1 --2- 172.21.15.6:0/2876465266 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f67ec06b4d0 0x7f67ec06d990 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f68000073f0 tx=0x7f680000b040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:38.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:38.040+0000 7f6806a00640 1 -- 172.21.15.6:0/2876465266 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f67f80584a0 con 0x7f68100d2930 2024-03-04T09:31:38.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:38.283+0000 7f6815400640 1 -- 172.21.15.6:0/2876465266 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f68100d1df0 con 0x7f68100d2930 2024-03-04T09:31:38.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:38.283+0000 7f6806a00640 1 -- 172.21.15.6:0/2876465266 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) v1 ==== 94+0+4174 (secure 0 0 0) 0x7f67f8057da0 con 0x7f68100d2930 2024-03-04T09:31:38.280 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:31:38.281 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":81,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6535,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/3883122012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":3883122012},{"type":"v1","addr":"172.21.15.146:6835","nonce":3883122012}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":7812,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/679471398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":679471398},{"type":"v1","addr":"172.21.15.6:6833","nonce":679471398}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:31:21.114351+0000","modified":"2024-03-04T09:31:37.752468+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7815},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7815":{"gid":7815,"name":"a","rank":0,"incarnation":81,"state":"up:replay","state_seq":1,"addr":"172.21.15.6:6835/882667161","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":882667161},{"type":"v1","addr":"172.21.15.6:6835","nonce":882667161}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":8}]} 2024-03-04T09:31:38.281 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 81 2024-03-04T09:31:38.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:38.286+0000 7f6815400640 1 -- 172.21.15.6:0/2876465266 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f67ec06b4d0 msgr2=0x7f67ec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:38.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:38.286+0000 7f6815400640 1 --2- 172.21.15.6:0/2876465266 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f67ec06b4d0 0x7f67ec06d990 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f68000073f0 tx=0x7f680000b040 comp rx=0 tx=0).stop 2024-03-04T09:31:38.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:38.287+0000 7f6815400640 1 -- 172.21.15.6:0/2876465266 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f68100d2930 msgr2=0x7f68101557c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:38.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:38.287+0000 7f6815400640 1 --2- 172.21.15.6:0/2876465266 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f68100d2930 0x7f68101557c0 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7f67f800c430 tx=0x7f67f800c8f0 comp rx=0 tx=0).stop 2024-03-04T09:31:38.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:38.287+0000 7f6815400640 1 -- 172.21.15.6:0/2876465266 shutdown_connections 2024-03-04T09:31:38.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:38.287+0000 7f6815400640 1 --2- 172.21.15.6:0/2876465266 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f67ec06b4d0 0x7f67ec06d990 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:38.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:38.287+0000 7f6815400640 1 --2- 172.21.15.6:0/2876465266 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6810161b40 0x7f68100ca1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:38.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:38.287+0000 7f6815400640 1 --2- 172.21.15.6:0/2876465266 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f68100d2930 0x7f68101557c0 unknown :-1 s=CLOSED pgs=1310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:38.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:38.287+0000 7f6815400640 1 --2- 172.21.15.6:0/2876465266 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f68100b5060 0x7f68100b6b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:38.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:38.287+0000 7f6815400640 1 -- 172.21.15.6:0/2876465266 >> 172.21.15.6:0/2876465266 conn(0x7f681009d220 msgr2=0x7f6810160d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:38.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:38.287+0000 7f6815400640 1 -- 172.21.15.6:0/2876465266 shutdown_connections 2024-03-04T09:31:38.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:38.287+0000 7f6815400640 1 -- 172.21.15.6:0/2876465266 wait complete. 2024-03-04T09:31:38.706 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 81, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:31:21.114351+0000', 'modified': '2024-03-04T09:31:37.752468+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 112, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7815}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7815': {'gid': 7815, 'name': 'a', 'rank': 0, 'incarnation': 81, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.6:6835/882667161', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 882667161}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 882667161}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [17], 'metadata_pool': 16, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-04T09:31:38.707 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7815:up:replay 2024-03-04T09:31:39.708 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:31:39.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:39.938+0000 7f86a4200640 1 Processor -- start 2024-03-04T09:31:39.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:39.939+0000 7f86a4200640 1 -- start start 2024-03-04T09:31:39.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:39.940+0000 7f86a4200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f869c0b34e0 0x7f869c15c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:39.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:39.940+0000 7f86a4200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f869c15cc40 con 0x7f869c0af1a0 2024-03-04T09:31:39.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:39.940+0000 7f86a4200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f869c15cf20 con 0x7f869c0af580 2024-03-04T09:31:39.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:39.940+0000 7f86a4200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f869c15d200 con 0x7f869c0b34e0 2024-03-04T09:31:39.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:39.941+0000 7f86a1800640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f869c0af1a0 0x7f869c0b4c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:49984/0 (socket says 172.21.15.6:49984) 2024-03-04T09:31:39.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:39.941+0000 7f86a1800640 1 -- 172.21.15.6:0/238606671 learned_addr learned my addr 172.21.15.6:0/238606671 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:31:39.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:39.941+0000 7f86a2200640 1 --2- 172.21.15.6:0/238606671 >> v2:172.21.15.146:3300/0 conn(0x7f869c0b34e0 0x7f869c15c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:39.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:39.941+0000 7f869be00640 1 -- 172.21.15.6:0/238606671 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4050392764 0 0) 0x7f869c15cc40 con 0x7f869c0af1a0 2024-03-04T09:31:39.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:39.941+0000 7f869be00640 1 -- 172.21.15.6:0/238606671 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8680003520 con 0x7f869c0af1a0 2024-03-04T09:31:39.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:39.942+0000 7f869be00640 1 -- 172.21.15.6:0/238606671 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3277592869 0 0) 0x7f8680003520 con 0x7f869c0af1a0 2024-03-04T09:31:39.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:39.942+0000 7f869be00640 1 -- 172.21.15.6:0/238606671 >> v2:172.21.15.146:3300/0 conn(0x7f869c0b34e0 msgr2=0x7f869c15c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:39.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:39.942+0000 7f869be00640 1 --2- 172.21.15.6:0/238606671 >> v2:172.21.15.146:3300/0 conn(0x7f869c0b34e0 0x7f869c15c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:39.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:39.942+0000 7f869be00640 1 -- 172.21.15.6:0/238606671 >> v1:172.21.15.146:6789/0 conn(0x7f869c0af580 legacy=0x7f869c0b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:39.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:39.942+0000 7f869be00640 1 -- 172.21.15.6:0/238606671 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f869c15d4e0 con 0x7f869c0af1a0 2024-03-04T09:31:39.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:39.942+0000 7f869be00640 1 -- 172.21.15.6:0/238606671 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7f868c0019f0 con 0x7f869c0af1a0 2024-03-04T09:31:39.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:39.942+0000 7f869be00640 1 -- 172.21.15.6:0/238606671 >> v1:172.21.15.6:6789/0 conn(0x7f869c0af1a0 legacy=0x7f869c0b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:39.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:39.942+0000 7f869be00640 1 --2- 172.21.15.6:0/238606671 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8680003a20 0x7f8680003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:39.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:39.942+0000 7f869be00640 1 --2- 172.21.15.6:0/238606671 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8680004800 0x7f8680004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:39.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:39.942+0000 7f869be00640 1 --2- 172.21.15.6:0/238606671 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f86800052e0 0x7f868000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:39.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:39.942+0000 7f869be00640 1 -- 172.21.15.6:0/238606671 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f869c15d200 con 0x7f8680003a20 2024-03-04T09:31:39.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:39.942+0000 7f869be00640 1 -- 172.21.15.6:0/238606671 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f868000c230 con 0x7f86800052e0 2024-03-04T09:31:39.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:39.942+0000 7f869be00640 1 -- 172.21.15.6:0/238606671 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f868000c430 con 0x7f8680004800 2024-03-04T09:31:39.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:39.942+0000 7f86a1800640 1 --2- 172.21.15.6:0/238606671 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8680003a20 0x7f8680003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:39.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:39.942+0000 7f86a0e00640 1 --2- 172.21.15.6:0/238606671 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8680004800 0x7f8680004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:39.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:39.942+0000 7f86a1800640 1 -- 172.21.15.6:0/238606671 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8680004800 msgr2=0x7f8680004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:39.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:39.943+0000 7f86a2200640 1 --2- 172.21.15.6:0/238606671 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f86800052e0 0x7f868000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:39.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:39.943+0000 7f86a1800640 1 --2- 172.21.15.6:0/238606671 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8680004800 0x7f8680004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:39.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:39.943+0000 7f86a1800640 1 -- 172.21.15.6:0/238606671 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f86800052e0 msgr2=0x7f868000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:39.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:39.943+0000 7f86a1800640 1 --2- 172.21.15.6:0/238606671 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f86800052e0 0x7f868000b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:39.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:39.943+0000 7f86a1800640 1 -- 172.21.15.6:0/238606671 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8680003520 con 0x7f8680003a20 2024-03-04T09:31:39.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:39.943+0000 7f86a2200640 1 -- 172.21.15.6:0/238606671 reap_dead start 2024-03-04T09:31:39.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:39.943+0000 7f86a0e00640 1 --2- 172.21.15.6:0/238606671 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8680004800 0x7f8680004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:31:39.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:39.944+0000 7f86a1800640 1 --2- 172.21.15.6:0/238606671 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8680003a20 0x7f8680003e40 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7f868c002230 tx=0x7f8680005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25d5231600911077 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:39.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:39.944+0000 7f869be00640 1 -- 172.21.15.6:0/238606671 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f868c00a4a0 con 0x7f8680003a20 2024-03-04T09:31:39.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:39.944+0000 7f869be00640 1 -- 172.21.15.6:0/238606671 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f868c034650 con 0x7f8680003a20 2024-03-04T09:31:39.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:39.944+0000 7f869be00640 1 -- 172.21.15.6:0/238606671 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f868c034a80 con 0x7f8680003a20 2024-03-04T09:31:39.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:39.944+0000 7f86a4200640 1 -- 172.21.15.6:0/238606671 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8680003a20 msgr2=0x7f8680003e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:39.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:39.944+0000 7f86a4200640 1 --2- 172.21.15.6:0/238606671 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8680003a20 0x7f8680003e40 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7f868c002230 tx=0x7f8680005830 comp rx=0 tx=0).stop 2024-03-04T09:31:39.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:39.945+0000 7f86a4200640 1 -- 172.21.15.6:0/238606671 shutdown_connections 2024-03-04T09:31:39.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:39.945+0000 7f86a4200640 1 --2- 172.21.15.6:0/238606671 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8680003a20 0x7f8680003e40 unknown :-1 s=CLOSED pgs=1312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:39.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:39.945+0000 7f86a4200640 1 -- 172.21.15.6:0/238606671 >> 172.21.15.6:0/238606671 conn(0x7f869c06afb0 msgr2=0x7f869c06b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:39.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:39.945+0000 7f86a4200640 1 -- 172.21.15.6:0/238606671 shutdown_connections 2024-03-04T09:31:39.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:39.945+0000 7f86a4200640 1 -- 172.21.15.6:0/238606671 wait complete. 2024-03-04T09:31:39.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:39.946+0000 7f86a4200640 1 Processor -- start 2024-03-04T09:31:39.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:39.947+0000 7f86a4200640 1 -- start start 2024-03-04T09:31:39.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:39.947+0000 7f86a4200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8680003a20 0x7f869c14c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:39.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:39.947+0000 7f86a4200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f869c09aa10 0x7f869c0cb650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:39.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:39.948+0000 7f86a4200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f869c157c00 0x7f869c158080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:39.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:39.948+0000 7f86a4200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f869c15c600 con 0x7f869c157c00 2024-03-04T09:31:39.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:39.948+0000 7f86a4200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f869c0cde30 con 0x7f8680003a20 2024-03-04T09:31:39.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:39.948+0000 7f86a4200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f869c15c780 con 0x7f869c09aa10 2024-03-04T09:31:39.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:39.948+0000 7f86a2200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f869c157c00 0x7f869c158080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:39.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:39.948+0000 7f86a2200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f869c157c00 0x7f869c158080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:58364/0 (socket says 172.21.15.6:58364) 2024-03-04T09:31:39.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:39.948+0000 7f86a2200640 1 -- 172.21.15.6:0/4263200349 learned_addr learned my addr 172.21.15.6:0/4263200349 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:31:39.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:39.948+0000 7f86a0e00640 1 --2- 172.21.15.6:0/4263200349 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f869c09aa10 0x7f869c0cb650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:39.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:39.948+0000 7f86a2200640 1 -- 172.21.15.6:0/4263200349 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f869c09aa10 msgr2=0x7f869c0cb650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:39.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:39.948+0000 7f86a1800640 1 --2- 172.21.15.6:0/4263200349 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8680003a20 0x7f869c14c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:39.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:39.948+0000 7f86a2200640 1 --2- 172.21.15.6:0/4263200349 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f869c09aa10 0x7f869c0cb650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:39.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:39.948+0000 7f86a2200640 1 -- 172.21.15.6:0/4263200349 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8680003a20 msgr2=0x7f869c14c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:39.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:39.948+0000 7f86a2200640 1 --2- 172.21.15.6:0/4263200349 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8680003a20 0x7f869c14c8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:39.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:39.948+0000 7f86a2200640 1 -- 172.21.15.6:0/4263200349 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f869c0c9a60 con 0x7f869c157c00 2024-03-04T09:31:39.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:39.948+0000 7f86a0e00640 1 --2- 172.21.15.6:0/4263200349 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f869c09aa10 0x7f869c0cb650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:31:39.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:39.949+0000 7f86a2200640 1 --2- 172.21.15.6:0/4263200349 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f869c157c00 0x7f869c158080 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7f869c0b3970 tx=0x7f869400e8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:39.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:39.949+0000 7f8699600640 1 -- 172.21.15.6:0/4263200349 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f869400f040 con 0x7f869c157c00 2024-03-04T09:31:39.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:39.949+0000 7f8699600640 1 -- 172.21.15.6:0/4263200349 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8694019070 con 0x7f869c157c00 2024-03-04T09:31:39.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:39.949+0000 7f8699600640 1 -- 172.21.15.6:0/4263200349 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8694013850 con 0x7f869c157c00 2024-03-04T09:31:39.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:39.949+0000 7f86a4200640 1 -- 172.21.15.6:0/4263200349 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f869c0c9d40 con 0x7f869c157c00 2024-03-04T09:31:39.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:39.949+0000 7f86a4200640 1 -- 172.21.15.6:0/4263200349 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f869c06d520 con 0x7f869c157c00 2024-03-04T09:31:39.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:39.950+0000 7f86a4200640 1 -- 172.21.15.6:0/4263200349 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f869c069510 con 0x7f869c157c00 2024-03-04T09:31:39.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:39.951+0000 7f8699600640 1 -- 172.21.15.6:0/4263200349 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f86940139f0 con 0x7f869c157c00 2024-03-04T09:31:39.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:39.952+0000 7f8699600640 1 --2- 172.21.15.6:0/4263200349 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f867c06b500 0x7f867c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:39.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:39.952+0000 7f8699600640 1 -- 172.21.15.6:0/4263200349 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f8694089bb0 con 0x7f869c157c00 2024-03-04T09:31:39.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:39.953+0000 7f86a1800640 1 --2- 172.21.15.6:0/4263200349 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f867c06b500 0x7f867c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:39.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:39.954+0000 7f86a1800640 1 --2- 172.21.15.6:0/4263200349 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f867c06b500 0x7f867c06d9c0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f868c034000 tx=0x7f868c02f000 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:39.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:39.957+0000 7f8699600640 1 -- 172.21.15.6:0/4263200349 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8694058930 con 0x7f869c157c00 2024-03-04T09:31:40.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:40.198+0000 7f86a4200640 1 -- 172.21.15.6:0/4263200349 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f869c0b2db0 con 0x7f869c157c00 2024-03-04T09:31:40.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:40.200+0000 7f8699600640 1 -- 172.21.15.6:0/4263200349 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+4968 (secure 0 0 0) 0x7f8694058230 con 0x7f869c157c00 2024-03-04T09:31:40.198 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:31:40.198 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":83,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6535,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/3883122012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":3883122012},{"type":"v1","addr":"172.21.15.146:6835","nonce":3883122012}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":6559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/4060411493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":4060411493},{"type":"v1","addr":"172.21.15.146:6837","nonce":4060411493}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":82},{"gid":7812,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/679471398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":679471398},{"type":"v1","addr":"172.21.15.6:6833","nonce":679471398}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:31:21.114351+0000","modified":"2024-03-04T09:31:38.773029+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7815},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7815":{"gid":7815,"name":"a","rank":0,"incarnation":81,"state":"up:rejoin","state_seq":3,"addr":"172.21.15.6:6835/882667161","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":882667161},{"type":"v1","addr":"172.21.15.6:6835","nonce":882667161}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":8}]} 2024-03-04T09:31:40.198 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 83 2024-03-04T09:31:40.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:40.203+0000 7f86a4200640 1 -- 172.21.15.6:0/4263200349 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f867c06b500 msgr2=0x7f867c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:40.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:40.203+0000 7f86a4200640 1 --2- 172.21.15.6:0/4263200349 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f867c06b500 0x7f867c06d9c0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f868c034000 tx=0x7f868c02f000 comp rx=0 tx=0).stop 2024-03-04T09:31:40.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:40.203+0000 7f86a4200640 1 -- 172.21.15.6:0/4263200349 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f869c157c00 msgr2=0x7f869c158080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:40.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:40.203+0000 7f86a4200640 1 --2- 172.21.15.6:0/4263200349 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f869c157c00 0x7f869c158080 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7f869c0b3970 tx=0x7f869400e8b0 comp rx=0 tx=0).stop 2024-03-04T09:31:40.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:40.203+0000 7f86a4200640 1 -- 172.21.15.6:0/4263200349 shutdown_connections 2024-03-04T09:31:40.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:40.203+0000 7f86a4200640 1 --2- 172.21.15.6:0/4263200349 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f869c157c00 0x7f869c158080 unknown :-1 s=CLOSED pgs=1313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:40.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:40.203+0000 7f86a4200640 1 --2- 172.21.15.6:0/4263200349 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f867c06b500 0x7f867c06d9c0 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:40.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:40.203+0000 7f86a4200640 1 --2- 172.21.15.6:0/4263200349 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f869c09aa10 0x7f869c0cb650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:40.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:40.203+0000 7f86a4200640 1 --2- 172.21.15.6:0/4263200349 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8680003a20 0x7f869c14c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:40.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:40.203+0000 7f86a4200640 1 -- 172.21.15.6:0/4263200349 >> 172.21.15.6:0/4263200349 conn(0x7f869c06afb0 msgr2=0x7f869c15beb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:40.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:40.204+0000 7f86a4200640 1 -- 172.21.15.6:0/4263200349 shutdown_connections 2024-03-04T09:31:40.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:40.204+0000 7f86a4200640 1 -- 172.21.15.6:0/4263200349 wait complete. 2024-03-04T09:31:40.294 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:31:40.302 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:31:40.623 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 83, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:31:21.114351+0000', 'modified': '2024-03-04T09:31:38.773029+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 112, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7815}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7815': {'gid': 7815, 'name': 'a', 'rank': 0, 'incarnation': 81, 'state': 'up:rejoin', 'state_seq': 3, 'addr': '172.21.15.6:6835/882667161', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 882667161}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 882667161}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [17], 'metadata_pool': 16, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-04T09:31:40.624 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7815:up:rejoin 2024-03-04T09:31:41.624 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:31:41.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:41.865+0000 7f7338a00640 1 Processor -- start 2024-03-04T09:31:41.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:41.866+0000 7f7338a00640 1 -- start start 2024-03-04T09:31:41.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:41.868+0000 7f7338a00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f733415e0b0 0x7f73340cf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:41.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:41.868+0000 7f7338a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f73340d0e50 0x7f73340b44d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:41.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:41.868+0000 7f7338a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73340b3710 con 0x7f73340d4a10 2024-03-04T09:31:41.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:41.868+0000 7f7338a00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73340b09e0 con 0x7f733415e0b0 2024-03-04T09:31:41.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:41.868+0000 7f7338a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73340b39f0 con 0x7f73340d0e50 2024-03-04T09:31:41.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:41.868+0000 7f732be00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f73340d4a10 0x7f73340d0740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:49986/0 (socket says 172.21.15.6:49986) 2024-03-04T09:31:41.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:41.868+0000 7f732be00640 1 -- 172.21.15.6:0/3386247723 learned_addr learned my addr 172.21.15.6:0/3386247723 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:31:41.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:41.869+0000 7f7333200640 1 --2- 172.21.15.6:0/3386247723 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f73340d0e50 0x7f73340b44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:41.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:41.869+0000 7f7332800640 1 --2- 172.21.15.6:0/3386247723 >> v2:172.21.15.146:3300/0 conn(0x7f733415e0b0 0x7f73340cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:41.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:41.869+0000 7f7331e00640 1 -- 172.21.15.6:0/3386247723 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1053345680 0 0) 0x7f73340b3710 con 0x7f73340d4a10 2024-03-04T09:31:41.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:41.869+0000 7f7331e00640 1 -- 172.21.15.6:0/3386247723 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7318003520 con 0x7f73340d4a10 2024-03-04T09:31:41.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:41.869+0000 7f7331e00640 1 -- 172.21.15.6:0/3386247723 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3020937611 0 0) 0x7f7318003520 con 0x7f73340d4a10 2024-03-04T09:31:41.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:41.869+0000 7f7331e00640 1 -- 172.21.15.6:0/3386247723 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f73340d0e50 msgr2=0x7f73340b44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:41.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:41.869+0000 7f7331e00640 1 --2- 172.21.15.6:0/3386247723 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f73340d0e50 0x7f73340b44d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:41.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:41.869+0000 7f7331e00640 1 -- 172.21.15.6:0/3386247723 >> v2:172.21.15.146:3300/0 conn(0x7f733415e0b0 msgr2=0x7f73340cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:41.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:41.869+0000 7f7331e00640 1 --2- 172.21.15.6:0/3386247723 >> v2:172.21.15.146:3300/0 conn(0x7f733415e0b0 0x7f73340cf310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:41.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:41.869+0000 7f7331e00640 1 -- 172.21.15.6:0/3386247723 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73340b3cd0 con 0x7f73340d4a10 2024-03-04T09:31:41.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:41.869+0000 7f7331e00640 1 -- 172.21.15.6:0/3386247723 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7f731c001690 con 0x7f73340d4a10 2024-03-04T09:31:41.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:41.869+0000 7f7332800640 1 --2- 172.21.15.6:0/3386247723 >> v2:172.21.15.146:3300/0 conn(0x7f733415e0b0 0x7f73340cf310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T09:31:41.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:41.869+0000 7f7331e00640 1 -- 172.21.15.6:0/3386247723 >> v1:172.21.15.6:6789/0 conn(0x7f73340d4a10 legacy=0x7f73340d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:41.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:41.869+0000 7f7331e00640 1 --2- 172.21.15.6:0/3386247723 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f73180039b0 0x7f7318003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:41.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:41.869+0000 7f7331e00640 1 --2- 172.21.15.6:0/3386247723 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f73340d0e50 0x7f7318004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:41.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:41.869+0000 7f7331e00640 1 --2- 172.21.15.6:0/3386247723 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7318004d40 0x7f73180051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:41.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:41.869+0000 7f7331e00640 1 -- 172.21.15.6:0/3386247723 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73340b39f0 con 0x7f7318004d40 2024-03-04T09:31:41.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:41.869+0000 7f7331e00640 1 -- 172.21.15.6:0/3386247723 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f731800b9f0 con 0x7f73180039b0 2024-03-04T09:31:41.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:41.869+0000 7f7331e00640 1 -- 172.21.15.6:0/3386247723 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f731800bbf0 con 0x7f73340d0e50 2024-03-04T09:31:41.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:41.869+0000 7f7333200640 1 --2- 172.21.15.6:0/3386247723 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7318004d40 0x7f73180051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:41.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:41.869+0000 7f732be00640 1 --2- 172.21.15.6:0/3386247723 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f73340d0e50 0x7f7318004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:41.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:41.869+0000 7f7332800640 1 --2- 172.21.15.6:0/3386247723 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f73180039b0 0x7f7318003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:41.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:41.869+0000 7f7333200640 1 -- 172.21.15.6:0/3386247723 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f73340d0e50 msgr2=0x7f7318004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:41.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:41.869+0000 7f7333200640 1 --2- 172.21.15.6:0/3386247723 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f73340d0e50 0x7f7318004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:41.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:41.869+0000 7f7333200640 1 -- 172.21.15.6:0/3386247723 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f73180039b0 msgr2=0x7f7318003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:41.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:41.869+0000 7f7333200640 1 --2- 172.21.15.6:0/3386247723 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f73180039b0 0x7f7318003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:41.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:41.869+0000 7f7333200640 1 -- 172.21.15.6:0/3386247723 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7318003520 con 0x7f7318004d40 2024-03-04T09:31:41.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:41.869+0000 7f7332800640 1 --2- 172.21.15.6:0/3386247723 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f73180039b0 0x7f7318003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:31:41.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:41.870+0000 7f7333200640 1 --2- 172.21.15.6:0/3386247723 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7318004d40 0x7f73180051c0 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7f732c009f90 tx=0x7f7318005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c579bd24e0fdb519 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:41.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:41.870+0000 7f7331e00640 1 -- 172.21.15.6:0/3386247723 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f732c039070 con 0x7f7318004d40 2024-03-04T09:31:41.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:41.870+0000 7f7331e00640 1 -- 172.21.15.6:0/3386247723 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f732c02cbd0 con 0x7f7318004d40 2024-03-04T09:31:41.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:41.870+0000 7f7331e00640 1 -- 172.21.15.6:0/3386247723 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f732c009c10 con 0x7f7318004d40 2024-03-04T09:31:41.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:41.871+0000 7f7338a00640 1 -- 172.21.15.6:0/3386247723 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7318004d40 msgr2=0x7f73180051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:41.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:41.871+0000 7f7338a00640 1 --2- 172.21.15.6:0/3386247723 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7318004d40 0x7f73180051c0 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7f732c009f90 tx=0x7f7318005830 comp rx=0 tx=0).stop 2024-03-04T09:31:41.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:41.871+0000 7f7333200640 1 -- 172.21.15.6:0/3386247723 reap_dead start 2024-03-04T09:31:41.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:41.871+0000 7f7338a00640 1 -- 172.21.15.6:0/3386247723 shutdown_connections 2024-03-04T09:31:41.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:41.871+0000 7f7338a00640 1 -- 172.21.15.6:0/3386247723 >> 172.21.15.6:0/3386247723 conn(0x7f73340d3d30 msgr2=0x7f73340d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:41.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:41.871+0000 7f7338a00640 1 -- 172.21.15.6:0/3386247723 shutdown_connections 2024-03-04T09:31:41.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:41.871+0000 7f7338a00640 1 -- 172.21.15.6:0/3386247723 wait complete. 2024-03-04T09:31:41.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:41.874+0000 7f7338a00640 1 Processor -- start 2024-03-04T09:31:41.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:41.874+0000 7f7338a00640 1 -- start start 2024-03-04T09:31:41.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:41.875+0000 7f7338a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7318004d40 0x7f73341521b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:41.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:41.875+0000 7f7338a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7334159670 0x7f73340b83b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:41.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:41.875+0000 7f7338a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f73340b88f0 0x7f733415c100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:41.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:41.875+0000 7f7338a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73340cfc60 con 0x7f73340b88f0 2024-03-04T09:31:41.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:41.875+0000 7f7338a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f733415e490 con 0x7f7334159670 2024-03-04T09:31:41.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:41.875+0000 7f7338a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73340cf850 con 0x7f7318004d40 2024-03-04T09:31:41.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:41.875+0000 7f7333200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f73340b88f0 0x7f733415c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:41.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:41.876+0000 7f7333200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f73340b88f0 0x7f733415c100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:58384/0 (socket says 172.21.15.6:58384) 2024-03-04T09:31:41.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:41.876+0000 7f7333200640 1 -- 172.21.15.6:0/458615338 learned_addr learned my addr 172.21.15.6:0/458615338 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:31:41.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:41.876+0000 7f7332800640 1 --2- 172.21.15.6:0/458615338 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7318004d40 0x7f73341521b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:41.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:41.876+0000 7f732be00640 1 --2- 172.21.15.6:0/458615338 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7334159670 0x7f73340b83b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:41.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:41.876+0000 7f7333200640 1 -- 172.21.15.6:0/458615338 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7318004d40 msgr2=0x7f73341521b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:41.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:41.876+0000 7f7333200640 1 --2- 172.21.15.6:0/458615338 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7318004d40 0x7f73341521b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:41.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:41.876+0000 7f7333200640 1 -- 172.21.15.6:0/458615338 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7334159670 msgr2=0x7f73340b83b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:41.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:41.876+0000 7f7333200640 1 --2- 172.21.15.6:0/458615338 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7334159670 0x7f73340b83b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:41.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:41.876+0000 7f7333200640 1 -- 172.21.15.6:0/458615338 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73341526f0 con 0x7f73340b88f0 2024-03-04T09:31:41.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:41.876+0000 7f732be00640 1 --2- 172.21.15.6:0/458615338 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7334159670 0x7f73340b83b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:31:41.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:41.876+0000 7f7333200640 1 --2- 172.21.15.6:0/458615338 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f73340b88f0 0x7f733415c100 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7f732c037040 tx=0x7f732c009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:41.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:41.877+0000 7f732aa00640 1 -- 172.21.15.6:0/458615338 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f732c039030 con 0x7f73340b88f0 2024-03-04T09:31:41.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:41.877+0000 7f732aa00640 1 -- 172.21.15.6:0/458615338 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f732c03a040 con 0x7f73340b88f0 2024-03-04T09:31:41.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:41.877+0000 7f732aa00640 1 -- 172.21.15.6:0/458615338 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f732c009c10 con 0x7f73340b88f0 2024-03-04T09:31:41.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:41.877+0000 7f7338a00640 1 -- 172.21.15.6:0/458615338 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73340caf30 con 0x7f73340b88f0 2024-03-04T09:31:41.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:41.877+0000 7f7338a00640 1 -- 172.21.15.6:0/458615338 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73340cb4f0 con 0x7f73340b88f0 2024-03-04T09:31:41.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:41.878+0000 7f7338a00640 1 -- 172.21.15.6:0/458615338 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f733404f330 con 0x7f73340b88f0 2024-03-04T09:31:41.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:41.879+0000 7f732aa00640 1 -- 172.21.15.6:0/458615338 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f732c045020 con 0x7f73340b88f0 2024-03-04T09:31:41.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:41.881+0000 7f732aa00640 1 --2- 172.21.15.6:0/458615338 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f731006b4d0 0x7f731006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:41.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:41.881+0000 7f732aa00640 1 -- 172.21.15.6:0/458615338 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f732c0abd50 con 0x7f73340b88f0 2024-03-04T09:31:41.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:41.881+0000 7f7332800640 1 --2- 172.21.15.6:0/458615338 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f731006b4d0 0x7f731006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:41.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:41.883+0000 7f7332800640 1 --2- 172.21.15.6:0/458615338 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f731006b4d0 0x7f731006d990 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f7324006fd0 tx=0x7f7324008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:41.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:41.885+0000 7f732aa00640 1 -- 172.21.15.6:0/458615338 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f732c04c020 con 0x7f73340b88f0 2024-03-04T09:31:42.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.128+0000 7f7338a00640 1 -- 172.21.15.6:0/458615338 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7334161640 con 0x7f73340b88f0 2024-03-04T09:31:42.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.129+0000 7f732aa00640 1 -- 172.21.15.6:0/458615338 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+4975 (secure 0 0 0) 0x7f73341526f0 con 0x7f73340b88f0 2024-03-04T09:31:42.127 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:31:42.127 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":84,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6535,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/3883122012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":3883122012},{"type":"v1","addr":"172.21.15.146:6835","nonce":3883122012}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":6559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/4060411493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":4060411493},{"type":"v1","addr":"172.21.15.146:6837","nonce":4060411493}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":82},{"gid":7812,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/679471398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":679471398},{"type":"v1","addr":"172.21.15.6:6833","nonce":679471398}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:31:21.114351+0000","modified":"2024-03-04T09:31:40.781474+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7815},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7815":{"gid":7815,"name":"a","rank":0,"incarnation":81,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/882667161","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":882667161},{"type":"v1","addr":"172.21.15.6:6835","nonce":882667161}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7815,"qdb_cluster":[7815]},"id":8}]} 2024-03-04T09:31:42.127 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 84 2024-03-04T09:31:42.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.132+0000 7f7338a00640 1 -- 172.21.15.6:0/458615338 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f731006b4d0 msgr2=0x7f731006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:42.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.132+0000 7f7338a00640 1 --2- 172.21.15.6:0/458615338 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f731006b4d0 0x7f731006d990 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f7324006fd0 tx=0x7f7324008040 comp rx=0 tx=0).stop 2024-03-04T09:31:42.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.132+0000 7f7338a00640 1 -- 172.21.15.6:0/458615338 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f73340b88f0 msgr2=0x7f733415c100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:42.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.132+0000 7f7338a00640 1 --2- 172.21.15.6:0/458615338 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f73340b88f0 0x7f733415c100 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7f732c037040 tx=0x7f732c009750 comp rx=0 tx=0).stop 2024-03-04T09:31:42.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.133+0000 7f7338a00640 1 -- 172.21.15.6:0/458615338 shutdown_connections 2024-03-04T09:31:42.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.133+0000 7f7338a00640 1 --2- 172.21.15.6:0/458615338 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f73340b88f0 0x7f733415c100 unknown :-1 s=CLOSED pgs=1316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:42.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.133+0000 7f7338a00640 1 --2- 172.21.15.6:0/458615338 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7334159670 0x7f73340b83b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:42.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.133+0000 7f7338a00640 1 --2- 172.21.15.6:0/458615338 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f731006b4d0 0x7f731006d990 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:42.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.133+0000 7f7338a00640 1 --2- 172.21.15.6:0/458615338 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7318004d40 0x7f73341521b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:42.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.133+0000 7f7338a00640 1 -- 172.21.15.6:0/458615338 >> 172.21.15.6:0/458615338 conn(0x7f73340d3d30 msgr2=0x7f73340b4bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:42.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.133+0000 7f7338a00640 1 -- 172.21.15.6:0/458615338 shutdown_connections 2024-03-04T09:31:42.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.133+0000 7f7338a00640 1 -- 172.21.15.6:0/458615338 wait complete. 2024-03-04T09:31:42.551 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 84, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:31:21.114351+0000', 'modified': '2024-03-04T09:31:40.781474+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 112, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7815}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7815': {'gid': 7815, 'name': 'a', 'rank': 0, 'incarnation': 81, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/882667161', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 882667161}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 882667161}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [17], 'metadata_pool': 16, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7815, 'qdb_cluster': [7815]} 2024-03-04T09:31:42.551 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-04T09:31:42.551 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-04T09:31:42.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.801+0000 7f381b200640 1 Processor -- start 2024-03-04T09:31:42.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.801+0000 7f381b200640 1 -- start start 2024-03-04T09:31:42.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.802+0000 7f381b200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f381415e1c0 0x7f38140b0310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:42.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.803+0000 7f381b200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f381406f6b0 0x7f38140b5b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:42.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.803+0000 7f381b200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f381406afb0 0x7f381406b440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:42.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.803+0000 7f381b200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38140b6060 con 0x7f381406f6b0 2024-03-04T09:31:42.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.803+0000 7f381b200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38140b3da0 con 0x7f381406afb0 2024-03-04T09:31:42.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.803+0000 7f381b200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38140aff60 con 0x7f381415e1c0 2024-03-04T09:31:42.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.803+0000 7f3813400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f381406f6b0 0x7f38140b5b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:42.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.803+0000 7f3813400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f381406f6b0 0x7f38140b5b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:58390/0 (socket says 172.21.15.6:58390) 2024-03-04T09:31:42.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.803+0000 7f3813400640 1 -- 172.21.15.6:0/12013431 learned_addr learned my addr 172.21.15.6:0/12013431 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:31:42.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.803+0000 7f3813e00640 1 --2- 172.21.15.6:0/12013431 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f381415e1c0 0x7f38140b0310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:42.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.803+0000 7f3813400640 1 -- 172.21.15.6:0/12013431 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f381415e1c0 msgr2=0x7f38140b0310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:42.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.803+0000 7f3813400640 1 --2- 172.21.15.6:0/12013431 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f381415e1c0 0x7f38140b0310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:42.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.803+0000 7f3813400640 1 -- 172.21.15.6:0/12013431 >> v2:172.21.15.146:3300/0 conn(0x7f381406afb0 msgr2=0x7f381406b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:42.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.803+0000 7f3813400640 1 --2- 172.21.15.6:0/12013431 >> v2:172.21.15.146:3300/0 conn(0x7f381406afb0 0x7f381406b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:42.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.803+0000 7f3813400640 1 -- 172.21.15.6:0/12013431 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38140b4010 con 0x7f381406f6b0 2024-03-04T09:31:42.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.805+0000 7f3813400640 1 --2- 172.21.15.6:0/12013431 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f381406f6b0 0x7f38140b5b20 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7f3814071990 tx=0x7f380002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49a7350e5a7d13d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:42.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.805+0000 7f3812a00640 1 -- 172.21.15.6:0/12013431 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3800035020 con 0x7f381406f6b0 2024-03-04T09:31:42.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.805+0000 7f3812a00640 1 -- 172.21.15.6:0/12013431 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3800031870 con 0x7f381406f6b0 2024-03-04T09:31:42.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.805+0000 7f3812a00640 1 -- 172.21.15.6:0/12013431 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3800031c40 con 0x7f381406f6b0 2024-03-04T09:31:42.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.805+0000 7f381b200640 1 -- 172.21.15.6:0/12013431 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f381406f6b0 msgr2=0x7f38140b5b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:42.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.805+0000 7f381b200640 1 --2- 172.21.15.6:0/12013431 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f381406f6b0 0x7f38140b5b20 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7f3814071990 tx=0x7f380002b910 comp rx=0 tx=0).stop 2024-03-04T09:31:42.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.806+0000 7f381b200640 1 -- 172.21.15.6:0/12013431 shutdown_connections 2024-03-04T09:31:42.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.806+0000 7f381b200640 1 --2- 172.21.15.6:0/12013431 >> v2:172.21.15.146:3300/0 conn(0x7f381406afb0 0x7f381406b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:42.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.806+0000 7f381b200640 1 --2- 172.21.15.6:0/12013431 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f381406f6b0 0x7f38140b5b20 unknown :-1 s=CLOSED pgs=1317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:42.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.806+0000 7f381b200640 1 --2- 172.21.15.6:0/12013431 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f381415e1c0 0x7f38140b0310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:42.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.806+0000 7f381b200640 1 -- 172.21.15.6:0/12013431 >> 172.21.15.6:0/12013431 conn(0x7f38140b2a00 msgr2=0x7f38140b2e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:42.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.806+0000 7f381b200640 1 -- 172.21.15.6:0/12013431 shutdown_connections 2024-03-04T09:31:42.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.806+0000 7f381b200640 1 -- 172.21.15.6:0/12013431 wait complete. 2024-03-04T09:31:42.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.807+0000 7f381b200640 1 Processor -- start 2024-03-04T09:31:42.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.808+0000 7f381b200640 1 -- start start 2024-03-04T09:31:42.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.808+0000 7f381b200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f381406afb0 0x7f381415fe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:42.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.808+0000 7f381b200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f381406f6b0 0x7f381415e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:42.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.808+0000 7f381b200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f381415e1c0 0x7f381414f620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:42.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.808+0000 7f381b200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3814072b50 con 0x7f381406afb0 2024-03-04T09:31:42.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.808+0000 7f381b200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f381406fb30 con 0x7f381406f6b0 2024-03-04T09:31:42.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.808+0000 7f381b200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f381415fa00 con 0x7f381415e1c0 2024-03-04T09:31:42.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.809+0000 7f3813e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f381406afb0 0x7f381415fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:42.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.809+0000 7f3813e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f381406afb0 0x7f381415fe90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:58392/0 (socket says 172.21.15.6:58392) 2024-03-04T09:31:42.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.809+0000 7f3813e00640 1 -- 172.21.15.6:0/1242434422 learned_addr learned my addr 172.21.15.6:0/1242434422 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:31:42.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.809+0000 7f3813400640 1 --2- 172.21.15.6:0/1242434422 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f381406f6b0 0x7f381415e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:42.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.809+0000 7f3819200640 1 --2- 172.21.15.6:0/1242434422 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f381415e1c0 0x7f381414f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:42.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.809+0000 7f3813e00640 1 -- 172.21.15.6:0/1242434422 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f381415e1c0 msgr2=0x7f381414f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:42.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.809+0000 7f3813e00640 1 --2- 172.21.15.6:0/1242434422 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f381415e1c0 0x7f381414f620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:42.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.809+0000 7f3813e00640 1 -- 172.21.15.6:0/1242434422 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f381406f6b0 msgr2=0x7f381415e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:42.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.809+0000 7f3813e00640 1 --2- 172.21.15.6:0/1242434422 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f381406f6b0 0x7f381415e5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:42.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.809+0000 7f3813e00640 1 -- 172.21.15.6:0/1242434422 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f381415edf0 con 0x7f381406afb0 2024-03-04T09:31:42.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.810+0000 7f3813e00640 1 --2- 172.21.15.6:0/1242434422 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f381406afb0 0x7f381415fe90 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7f380400c7f0 tx=0x7f380400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:42.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.810+0000 7f380be00640 1 -- 172.21.15.6:0/1242434422 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3804015070 con 0x7f381406afb0 2024-03-04T09:31:42.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.810+0000 7f380be00640 1 -- 172.21.15.6:0/1242434422 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3804004240 con 0x7f381406afb0 2024-03-04T09:31:42.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.810+0000 7f381b200640 1 -- 172.21.15.6:0/1242434422 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f381415f0d0 con 0x7f381406afb0 2024-03-04T09:31:42.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.810+0000 7f380be00640 1 -- 172.21.15.6:0/1242434422 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3804007620 con 0x7f381406afb0 2024-03-04T09:31:42.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.810+0000 7f381b200640 1 -- 172.21.15.6:0/1242434422 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f381415f690 con 0x7f381406afb0 2024-03-04T09:31:42.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.813+0000 7f380be00640 1 -- 172.21.15.6:0/1242434422 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f38040079d0 con 0x7f381406afb0 2024-03-04T09:31:42.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.814+0000 7f380be00640 1 --2- 172.21.15.6:0/1242434422 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f37f406b500 0x7f37f406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:42.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.814+0000 7f380be00640 1 -- 172.21.15.6:0/1242434422 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f380408a2d0 con 0x7f381406afb0 2024-03-04T09:31:42.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.815+0000 7f381de54740 1 Processor -- start 2024-03-04T09:31:42.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.815+0000 7f3813400640 1 --2- 172.21.15.6:0/1242434422 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f37f406b500 0x7f37f406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:42.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.815+0000 7f381de54740 1 -- start start 2024-03-04T09:31:42.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.816+0000 7f381de54740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55a624d467b0 0x55a624d19750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:42.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.816+0000 7f381de54740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a624d46b90 0x55a624d49540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:42.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.816+0000 7f381de54740 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55a624d19c90 0x55a624d49a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:42.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.816+0000 7f381de54740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a624c83600 con 0x55a624d46b90 2024-03-04T09:31:42.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.816+0000 7f381de54740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a624c2c930 con 0x55a624d467b0 2024-03-04T09:31:42.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.816+0000 7f381de54740 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a624c7f6a0 con 0x55a624d19c90 2024-03-04T09:31:42.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.816+0000 7f3813400640 1 --2- 172.21.15.6:0/1242434422 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f37f406b500 0x7f37f406d9c0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f3800036000 tx=0x7f380003c040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:42.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.816+0000 7f3813400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a624d46b90 0x55a624d49540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:42.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.816+0000 7f3813400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a624d46b90 0x55a624d49540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:58406/0 (socket says 172.21.15.6:58406) 2024-03-04T09:31:42.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.816+0000 7f3813400640 1 -- 172.21.15.6:0/1152779699 learned_addr learned my addr 172.21.15.6:0/1152779699 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:31:42.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.817+0000 7f3813e00640 1 --2- 172.21.15.6:0/1152779699 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55a624d467b0 0x55a624d19750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:42.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.817+0000 7f3819200640 1 --2- 172.21.15.6:0/1152779699 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55a624d19c90 0x55a624d49a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:42.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.817+0000 7f3813400640 1 -- 172.21.15.6:0/1152779699 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55a624d19c90 msgr2=0x55a624d49a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:42.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.817+0000 7f3813400640 1 --2- 172.21.15.6:0/1152779699 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55a624d19c90 0x55a624d49a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:42.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.817+0000 7f3813400640 1 -- 172.21.15.6:0/1152779699 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55a624d467b0 msgr2=0x55a624d19750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:42.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.817+0000 7f3813400640 1 --2- 172.21.15.6:0/1152779699 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55a624d467b0 0x55a624d19750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:42.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.817+0000 7f3813400640 1 -- 172.21.15.6:0/1152779699 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a6249982d0 con 0x55a624d46b90 2024-03-04T09:31:42.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.817+0000 7f3813e00640 1 --2- 172.21.15.6:0/1152779699 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55a624d467b0 0x55a624d19750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T09:31:42.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.817+0000 7f3813400640 1 --2- 172.21.15.6:0/1152779699 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a624d46b90 0x55a624d49540 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7f380002bf20 tx=0x7f3800030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a5507357f46cf67 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:42.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.818+0000 7f3808c00640 1 -- 172.21.15.6:0/1152779699 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f380003aab0 con 0x55a624d46b90 2024-03-04T09:31:42.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.818+0000 7f3808c00640 1 -- 172.21.15.6:0/1152779699 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f380003ac50 con 0x55a624d46b90 2024-03-04T09:31:42.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.818+0000 7f3808c00640 1 -- 172.21.15.6:0/1152779699 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f380004e400 con 0x55a624d46b90 2024-03-04T09:31:42.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.818+0000 7f381de54740 1 -- 172.21.15.6:0/1152779699 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a624d46b90 msgr2=0x55a624d49540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:42.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.818+0000 7f381de54740 1 --2- 172.21.15.6:0/1152779699 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a624d46b90 0x55a624d49540 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7f380002bf20 tx=0x7f3800030000 comp rx=0 tx=0).stop 2024-03-04T09:31:42.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.818+0000 7f381de54740 1 -- 172.21.15.6:0/1152779699 shutdown_connections 2024-03-04T09:31:42.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.818+0000 7f381de54740 1 --2- 172.21.15.6:0/1152779699 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55a624d19c90 0x55a624d49a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:42.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.818+0000 7f381de54740 1 --2- 172.21.15.6:0/1152779699 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a624d46b90 0x55a624d49540 unknown :-1 s=CLOSED pgs=1319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:42.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.818+0000 7f381de54740 1 --2- 172.21.15.6:0/1152779699 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55a624d467b0 0x55a624d19750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:42.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.818+0000 7f381de54740 1 -- 172.21.15.6:0/1152779699 >> 172.21.15.6:0/1152779699 conn(0x55a624d1b9c0 msgr2=0x55a624d64540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:42.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.819+0000 7f381de54740 1 -- 172.21.15.6:0/1152779699 shutdown_connections 2024-03-04T09:31:42.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.819+0000 7f381de54740 1 -- 172.21.15.6:0/1152779699 wait complete. 2024-03-04T09:31:42.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.819+0000 7f381de54740 1 -- start start 2024-03-04T09:31:42.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.819+0000 7f381de54740 1 Processor -- start 2024-03-04T09:31:42.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.820+0000 7f381de54740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55a624d467b0 0x55a624d5eb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:42.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.820+0000 7f381de54740 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55a624d46b90 0x55a624d36ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:42.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.820+0000 7f381de54740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a624d346f0 0x55a624d37220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:42.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.820+0000 7f381de54740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a624d1df60 con 0x55a624d346f0 2024-03-04T09:31:42.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.820+0000 7f381de54740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a624c3fd90 con 0x55a624d467b0 2024-03-04T09:31:42.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.820+0000 7f381de54740 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a624d1d350 con 0x55a624d46b90 2024-03-04T09:31:42.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.821+0000 7f381de54740 10 client.0 _ll_register_callbacks cb 0x55a624c7b350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T09:31:42.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.821+0000 7f3819200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a624d346f0 0x55a624d37220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:42.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.821+0000 7f3819200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a624d346f0 0x55a624d37220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:58416/0 (socket says 172.21.15.6:58416) 2024-03-04T09:31:42.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.821+0000 7f3819200640 1 -- 172.21.15.6:0/273123642 learned_addr learned my addr 172.21.15.6:0/273123642 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:31:42.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.821+0000 7f3813e00640 1 --2- 172.21.15.6:0/273123642 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55a624d467b0 0x55a624d5eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:42.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.821+0000 7f3813400640 1 --2- 172.21.15.6:0/273123642 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55a624d46b90 0x55a624d36ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:42.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.821+0000 7f3819200640 1 -- 172.21.15.6:0/273123642 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55a624d46b90 msgr2=0x55a624d36ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:42.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.821+0000 7f3819200640 1 --2- 172.21.15.6:0/273123642 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55a624d46b90 0x55a624d36ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:42.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.821+0000 7f3819200640 1 -- 172.21.15.6:0/273123642 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55a624d467b0 msgr2=0x55a624d5eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:42.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.821+0000 7f3819200640 1 --2- 172.21.15.6:0/273123642 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55a624d467b0 0x55a624d5eb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:42.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.821+0000 7f3819200640 1 -- 172.21.15.6:0/273123642 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a624a626b0 con 0x55a624d346f0 2024-03-04T09:31:42.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.821+0000 7f3813400640 1 --2- 172.21.15.6:0/273123642 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55a624d46b90 0x55a624d36ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:31:42.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.822+0000 7f3819200640 1 --2- 172.21.15.6:0/273123642 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a624d346f0 0x55a624d37220 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7f380c00abc0 tx=0x7f380c0072b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5855e22bfb9145ed server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:42.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.822+0000 7f37fd600640 10 client.0 ms_handle_connect on v2:172.21.15.6:3300/0 2024-03-04T09:31:42.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.822+0000 7f37fd600640 1 -- 172.21.15.6:0/273123642 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f380c01b060 con 0x55a624d346f0 2024-03-04T09:31:42.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.822+0000 7f37fd600640 1 -- 172.21.15.6:0/273123642 <== mon.0 v2:172.21.15.6:3300/0 2 ==== osd_map(113..113 src has 1..113) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f380c016440 con 0x55a624d346f0 2024-03-04T09:31:42.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.822+0000 7f381de54740 1 -- 172.21.15.6:0/273123642 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f380003ac50 con 0x55a624d346f0 2024-03-04T09:31:42.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.822+0000 7f37fd600640 1 -- 172.21.15.6:0/273123642 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_get_version_reply(handle=1 version=84) v2 ==== 24+0+0 (secure 0 0 0) 0x7f380c021050 con 0x55a624d346f0 2024-03-04T09:31:42.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.822+0000 7f381de54740 10 client.7851 fetch_fsmap learned FSMap version 84 2024-03-04T09:31:42.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.822+0000 7f381de54740 1 -- 172.21.15.6:0/273123642 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,fsmap=84,monmap=2+}) v3 -- 0x55a624d4d030 con 0x55a624d346f0 2024-03-04T09:31:42.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.822+0000 7f37fd600640 1 -- 172.21.15.6:0/273123642 <== mon.0 v2:172.21.15.6:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f380003ac50 con 0x55a624d346f0 2024-03-04T09:31:42.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.823+0000 7f37fd600640 1 -- 172.21.15.6:0/273123642 <== mon.0 v2:172.21.15.6:3300/0 5 ==== fsmap(e 84) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f380c024020 con 0x55a624d346f0 2024-03-04T09:31:42.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.823+0000 7f381de54740 10 client.7851 fetch_fsmap finished waiting for FSMap version 84 2024-03-04T09:31:42.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.823+0000 7f381de54740 10 client.7851 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-04T09:31:42.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.823+0000 7f381de54740 20 client.7851 populate_metadata read hostname 'smithi006' 2024-03-04T09:31:42.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.823+0000 7f381de54740 1 --2- 172.21.15.6:0/273123642 >> [v2:172.21.15.6:6834/882667161,v1:172.21.15.6:6835/882667161] conn(0x55a624d359a0 0x55a624d35d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:42.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.823+0000 7f381de54740 4 client.7851 mds_command: new command op to 7815 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T09:31:42.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.823+0000 7f381de54740 1 -- 172.21.15.6:0/273123642 --> [v2:172.21.15.6:6834/882667161,v1:172.21.15.6:6835/882667161] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a624c90b40 con 0x55a624d359a0 2024-03-04T09:31:42.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.823+0000 7f3813e00640 1 --2- 172.21.15.6:0/273123642 >> [v2:172.21.15.6:6834/882667161,v1:172.21.15.6:6835/882667161] conn(0x55a624d359a0 0x55a624d35d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:42.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.823+0000 7f3813e00640 1 --2- 172.21.15.6:0/273123642 >> [v2:172.21.15.6:6834/882667161,v1:172.21.15.6:6835/882667161] conn(0x55a624d359a0 0x55a624d35d80 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=72e13d057757322c server_cookie=902162dcbf3fc89f in_seq=0 out_seq=0 2024-03-04T09:31:42.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.824+0000 7f37fd600640 10 client.7851 ms_handle_connect on v2:172.21.15.6:6834/882667161 2024-03-04T09:31:42.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.824+0000 7f37fd600640 1 -- 172.21.15.6:0/273123642 <== mds.0 v2:172.21.15.6:6834/882667161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55a624c90b40 con 0x55a624d359a0 2024-03-04T09:31:42.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.824+0000 7f37fd600640 10 client.7851 handle_command_reply: tid=0 2024-03-04T09:31:42.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.824+0000 7f381de54740 1 client.7851 shutdown 2024-03-04T09:31:42.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.825+0000 7f381de54740 1 -- 172.21.15.6:0/273123642 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a624d346f0 msgr2=0x55a624d37220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:42.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.825+0000 7f381de54740 1 --2- 172.21.15.6:0/273123642 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a624d346f0 0x55a624d37220 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7f380c00abc0 tx=0x7f380c0072b0 comp rx=0 tx=0).stop 2024-03-04T09:31:42.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.825+0000 7f381de54740 1 -- 172.21.15.6:0/273123642 shutdown_connections 2024-03-04T09:31:42.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.825+0000 7f381de54740 1 --2- 172.21.15.6:0/273123642 >> [v2:172.21.15.6:6834/882667161,v1:172.21.15.6:6835/882667161] conn(0x55a624d359a0 0x55a624d35d80 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:42.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.825+0000 7f381de54740 1 --2- 172.21.15.6:0/273123642 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a624d346f0 0x55a624d37220 unknown :-1 s=CLOSED pgs=1320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:42.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.825+0000 7f381de54740 1 --2- 172.21.15.6:0/273123642 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55a624d46b90 0x55a624d36ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:42.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.825+0000 7f381de54740 1 --2- 172.21.15.6:0/273123642 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55a624d467b0 0x55a624d5eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:42.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.825+0000 7f37fd600640 0 client.7851 ms_handle_reset on v2:172.21.15.6:6834/882667161 2024-03-04T09:31:42.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.825+0000 7f381de54740 1 -- 172.21.15.6:0/273123642 >> 172.21.15.6:0/273123642 conn(0x55a624d19c90 msgr2=0x55a624d18eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:42.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.825+0000 7f381de54740 1 -- 172.21.15.6:0/273123642 shutdown_connections 2024-03-04T09:31:42.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.825+0000 7f381de54740 1 -- 172.21.15.6:0/273123642 wait complete. 2024-03-04T09:31:42.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.825+0000 7f381de54740 20 client.7851 trim_cache size 0 max 16384 2024-03-04T09:31:42.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.844+0000 7f381de54740 1 Processor -- start 2024-03-04T09:31:42.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.844+0000 7f381de54740 1 -- start start 2024-03-04T09:31:42.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.844+0000 7f381de54740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55a624d346f0 0x55a624d64820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:42.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.845+0000 7f381de54740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a624d467b0 0x55a624d4ca40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:42.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.845+0000 7f381de54740 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55a624d46b90 0x55a624d36920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:42.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.845+0000 7f381de54740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38142003e0 con 0x55a624d467b0 2024-03-04T09:31:42.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.845+0000 7f381de54740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38141ff590 con 0x55a624d346f0 2024-03-04T09:31:42.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.845+0000 7f381de54740 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38140c69b0 con 0x55a624d46b90 2024-03-04T09:31:42.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.845+0000 7f3813400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a624d467b0 0x55a624d4ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:42.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.845+0000 7f3813400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a624d467b0 0x55a624d4ca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:58422/0 (socket says 172.21.15.6:58422) 2024-03-04T09:31:42.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.845+0000 7f3813400640 1 -- 172.21.15.6:0/3628435018 learned_addr learned my addr 172.21.15.6:0/3628435018 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:31:42.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.845+0000 7f3813e00640 1 --2- 172.21.15.6:0/3628435018 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55a624d346f0 0x55a624d64820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:42.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.845+0000 7f3819200640 1 --2- 172.21.15.6:0/3628435018 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55a624d46b90 0x55a624d36920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:42.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.845+0000 7f3813400640 1 -- 172.21.15.6:0/3628435018 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55a624d46b90 msgr2=0x55a624d36920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:42.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.845+0000 7f3813400640 1 --2- 172.21.15.6:0/3628435018 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55a624d46b90 0x55a624d36920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:42.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.845+0000 7f3813400640 1 -- 172.21.15.6:0/3628435018 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55a624d346f0 msgr2=0x55a624d64820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:42.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.845+0000 7f3813400640 1 --2- 172.21.15.6:0/3628435018 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55a624d346f0 0x55a624d64820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:42.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.845+0000 7f3813400640 1 -- 172.21.15.6:0/3628435018 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a624d1fa60 con 0x55a624d467b0 2024-03-04T09:31:42.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.845+0000 7f3813e00640 1 --2- 172.21.15.6:0/3628435018 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55a624d346f0 0x55a624d64820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:31:42.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.846+0000 7f3813400640 1 --2- 172.21.15.6:0/3628435018 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a624d467b0 0x55a624d4ca40 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7f380002b950 tx=0x7f3800009240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40343087c3eca61f server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:42.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.846+0000 7f37ffe00640 1 -- 172.21.15.6:0/3628435018 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f380005d070 con 0x55a624d467b0 2024-03-04T09:31:42.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.846+0000 7f37ffe00640 1 -- 172.21.15.6:0/3628435018 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3800058410 con 0x55a624d467b0 2024-03-04T09:31:42.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.846+0000 7f37ffe00640 1 -- 172.21.15.6:0/3628435018 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38000587c0 con 0x55a624d467b0 2024-03-04T09:31:42.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.847+0000 7f381de54740 1 -- 172.21.15.6:0/3628435018 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a624d467b0 msgr2=0x55a624d4ca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:42.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.847+0000 7f381de54740 1 --2- 172.21.15.6:0/3628435018 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a624d467b0 0x55a624d4ca40 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7f380002b950 tx=0x7f3800009240 comp rx=0 tx=0).stop 2024-03-04T09:31:42.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.847+0000 7f381de54740 1 -- 172.21.15.6:0/3628435018 shutdown_connections 2024-03-04T09:31:42.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.847+0000 7f381de54740 1 --2- 172.21.15.6:0/3628435018 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55a624d46b90 0x55a624d36920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:42.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.847+0000 7f381de54740 1 --2- 172.21.15.6:0/3628435018 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a624d467b0 0x55a624d4ca40 unknown :-1 s=CLOSED pgs=1321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:42.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.847+0000 7f381de54740 1 --2- 172.21.15.6:0/3628435018 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55a624d346f0 0x55a624d64820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:42.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.847+0000 7f381de54740 1 -- 172.21.15.6:0/3628435018 >> 172.21.15.6:0/3628435018 conn(0x55a624d19c90 msgr2=0x55a624d5fbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:42.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.847+0000 7f381de54740 1 -- 172.21.15.6:0/3628435018 shutdown_connections 2024-03-04T09:31:42.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.847+0000 7f381de54740 1 -- 172.21.15.6:0/3628435018 wait complete. 2024-03-04T09:31:42.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.848+0000 7f381de54740 1 -- start start 2024-03-04T09:31:42.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.848+0000 7f381de54740 1 Processor -- start 2024-03-04T09:31:42.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.849+0000 7f381de54740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55a624d346f0 0x55a624dd31e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:42.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.849+0000 7f381de54740 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55a624d467b0 0x55a624d65790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:42.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.849+0000 7f381de54740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a624d46b90 0x55a624d65cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:42.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.849+0000 7f381de54740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a624d5e850 con 0x55a624d46b90 2024-03-04T09:31:42.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.849+0000 7f381de54740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38141f4760 con 0x55a624d346f0 2024-03-04T09:31:42.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.849+0000 7f381de54740 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a624d5bd40 con 0x55a624d467b0 2024-03-04T09:31:42.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.849+0000 7f381de54740 10 client.0 _ll_register_callbacks cb 0x7f380c009770 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T09:31:42.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.849+0000 7f3819200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a624d46b90 0x55a624d65cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:42.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.849+0000 7f3819200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a624d46b90 0x55a624d65cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:58434/0 (socket says 172.21.15.6:58434) 2024-03-04T09:31:42.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.849+0000 7f3819200640 1 -- 172.21.15.6:0/1230111329 learned_addr learned my addr 172.21.15.6:0/1230111329 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:31:42.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.849+0000 7f3813400640 1 --2- 172.21.15.6:0/1230111329 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55a624d467b0 0x55a624d65790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:42.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.850+0000 7f3819200640 1 -- 172.21.15.6:0/1230111329 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55a624d467b0 msgr2=0x55a624d65790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:42.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.850+0000 7f3819200640 1 --2- 172.21.15.6:0/1230111329 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55a624d467b0 0x55a624d65790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:42.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.850+0000 7f3819200640 1 -- 172.21.15.6:0/1230111329 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55a624d346f0 msgr2=0x55a624dd31e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:42.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.850+0000 7f3819200640 1 --2- 172.21.15.6:0/1230111329 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55a624d346f0 0x55a624dd31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:42.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.850+0000 7f3819200640 1 -- 172.21.15.6:0/1230111329 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a624d66510 con 0x55a624d46b90 2024-03-04T09:31:42.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.850+0000 7f3813400640 1 --2- 172.21.15.6:0/1230111329 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55a624d467b0 0x55a624d65790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:31:42.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.850+0000 7f3819200640 1 --2- 172.21.15.6:0/1230111329 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a624d46b90 0x55a624d65cd0 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7f380c009a70 tx=0x7f380c004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce1f6466a576dd0c server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:42.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.850+0000 7f37fd600640 10 client.0 ms_handle_connect on v2:172.21.15.6:3300/0 2024-03-04T09:31:42.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.850+0000 7f37fd600640 1 -- 172.21.15.6:0/1230111329 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f380c0077b0 con 0x55a624d46b90 2024-03-04T09:31:42.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.850+0000 7f37fd600640 1 -- 172.21.15.6:0/1230111329 <== mon.0 v2:172.21.15.6:3300/0 2 ==== osd_map(113..113 src has 1..113) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f380c007cf0 con 0x55a624d46b90 2024-03-04T09:31:42.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.850+0000 7f381de54740 1 -- 172.21.15.6:0/1230111329 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3800058410 con 0x55a624d46b90 2024-03-04T09:31:42.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.851+0000 7f37fd600640 1 -- 172.21.15.6:0/1230111329 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_get_version_reply(handle=1 version=84) v2 ==== 24+0+0 (secure 0 0 0) 0x7f380c01b070 con 0x55a624d46b90 2024-03-04T09:31:42.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.851+0000 7f381de54740 10 client.7857 fetch_fsmap learned FSMap version 84 2024-03-04T09:31:42.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.851+0000 7f381de54740 1 -- 172.21.15.6:0/1230111329 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,fsmap=84,monmap=2+}) v3 -- 0x55a624d66f00 con 0x55a624d46b90 2024-03-04T09:31:42.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.851+0000 7f37fd600640 1 -- 172.21.15.6:0/1230111329 <== mon.0 v2:172.21.15.6:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3800058410 con 0x55a624d46b90 2024-03-04T09:31:42.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.851+0000 7f37fd600640 1 -- 172.21.15.6:0/1230111329 <== mon.0 v2:172.21.15.6:3300/0 5 ==== fsmap(e 84) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f380c016ce0 con 0x55a624d46b90 2024-03-04T09:31:42.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.851+0000 7f381de54740 10 client.7857 fetch_fsmap finished waiting for FSMap version 84 2024-03-04T09:31:42.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.851+0000 7f381de54740 10 client.7857 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-04T09:31:42.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.851+0000 7f381de54740 20 client.7857 populate_metadata read hostname 'smithi006' 2024-03-04T09:31:42.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.851+0000 7f381de54740 1 --2- 172.21.15.6:0/1230111329 >> [v2:172.21.15.6:6834/882667161,v1:172.21.15.6:6835/882667161] conn(0x55a624d87c00 0x55a624d5a170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:42.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.851+0000 7f381de54740 4 client.7857 mds_command: new command op to 7815 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-04T09:31:42.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.851+0000 7f381de54740 1 -- 172.21.15.6:0/1230111329 --> [v2:172.21.15.6:6834/882667161,v1:172.21.15.6:6835/882667161] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55a624d5f8a0 con 0x55a624d87c00 2024-03-04T09:31:42.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.851+0000 7f3813e00640 1 --2- 172.21.15.6:0/1230111329 >> [v2:172.21.15.6:6834/882667161,v1:172.21.15.6:6835/882667161] conn(0x55a624d87c00 0x55a624d5a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:42.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.852+0000 7f3813e00640 1 --2- 172.21.15.6:0/1230111329 >> [v2:172.21.15.6:6834/882667161,v1:172.21.15.6:6835/882667161] conn(0x55a624d87c00 0x55a624d5a170 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2d73e2b3f08255f4 server_cookie=884c0dad50fdd25b in_seq=0 out_seq=0 2024-03-04T09:31:42.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.852+0000 7f37fd600640 10 client.7857 ms_handle_connect on v2:172.21.15.6:6834/882667161 2024-03-04T09:31:42.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.852+0000 7f37fd600640 1 -- 172.21.15.6:0/1230111329 <== mds.0 v2:172.21.15.6:6834/882667161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x55a624d5f8a0 con 0x55a624d87c00 2024-03-04T09:31:42.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.852+0000 7f37fd600640 10 client.7857 handle_command_reply: tid=0 2024-03-04T09:31:42.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.852+0000 7f381de54740 1 client.7857 shutdown 2024-03-04T09:31:42.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.852+0000 7f381de54740 1 -- 172.21.15.6:0/1230111329 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a624d46b90 msgr2=0x55a624d65cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:42.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.852+0000 7f381de54740 1 --2- 172.21.15.6:0/1230111329 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a624d46b90 0x55a624d65cd0 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7f380c009a70 tx=0x7f380c004ab0 comp rx=0 tx=0).stop 2024-03-04T09:31:42.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.852+0000 7f381de54740 1 -- 172.21.15.6:0/1230111329 shutdown_connections 2024-03-04T09:31:42.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.852+0000 7f381de54740 1 --2- 172.21.15.6:0/1230111329 >> [v2:172.21.15.6:6834/882667161,v1:172.21.15.6:6835/882667161] conn(0x55a624d87c00 0x55a624d5a170 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:42.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.852+0000 7f381de54740 1 --2- 172.21.15.6:0/1230111329 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a624d46b90 0x55a624d65cd0 unknown :-1 s=CLOSED pgs=1322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:42.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.852+0000 7f381de54740 1 --2- 172.21.15.6:0/1230111329 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55a624d467b0 0x55a624d65790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:42.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.852+0000 7f381de54740 1 --2- 172.21.15.6:0/1230111329 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55a624d346f0 0x55a624dd31e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:42.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.852+0000 7f37fd600640 0 client.7857 ms_handle_reset on v2:172.21.15.6:6834/882667161 2024-03-04T09:31:42.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.852+0000 7f381de54740 1 -- 172.21.15.6:0/1230111329 >> 172.21.15.6:0/1230111329 conn(0x55a624d19c90 msgr2=0x55a624d645f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:42.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.853+0000 7f381de54740 1 -- 172.21.15.6:0/1230111329 shutdown_connections 2024-03-04T09:31:42.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.853+0000 7f381de54740 1 -- 172.21.15.6:0/1230111329 wait complete. 2024-03-04T09:31:42.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.853+0000 7f381de54740 20 client.7857 trim_cache size 0 max 16384 2024-03-04T09:31:42.853 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-04T09:31:42.853 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_fsid": "edce0578-5ae4-4402-b45a-552bdef84d7c", 2024-03-04T09:31:42.853 INFO:teuthology.orchestra.run.smithi006.stdout: "whoami": 0, 2024-03-04T09:31:42.853 INFO:teuthology.orchestra.run.smithi006.stdout: "id": 7815, 2024-03-04T09:31:42.853 INFO:teuthology.orchestra.run.smithi006.stdout: "want_state": "up:active", 2024-03-04T09:31:42.853 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "up:active", 2024-03-04T09:31:42.853 INFO:teuthology.orchestra.run.smithi006.stdout: "fs_name": "cephfs", 2024-03-04T09:31:42.853 INFO:teuthology.orchestra.run.smithi006.stdout: "rank_uptime": 5.0927677769999997, 2024-03-04T09:31:42.853 INFO:teuthology.orchestra.run.smithi006.stdout: "mdsmap_epoch": 84, 2024-03-04T09:31:42.854 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap_epoch": 113, 2024-03-04T09:31:42.854 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap_epoch_barrier": 113, 2024-03-04T09:31:42.854 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 5.86521852 2024-03-04T09:31:42.854 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-04T09:31:42.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.853+0000 7f381b200640 1 -- 172.21.15.6:0/1242434422 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f37f406b500 msgr2=0x7f37f406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:42.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.853+0000 7f381b200640 1 --2- 172.21.15.6:0/1242434422 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f37f406b500 0x7f37f406d9c0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f3800036000 tx=0x7f380003c040 comp rx=0 tx=0).stop 2024-03-04T09:31:42.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.853+0000 7f381b200640 1 -- 172.21.15.6:0/1242434422 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f381406afb0 msgr2=0x7f381415fe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:42.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.853+0000 7f381b200640 1 --2- 172.21.15.6:0/1242434422 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f381406afb0 0x7f381415fe90 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7f380400c7f0 tx=0x7f380400ccb0 comp rx=0 tx=0).stop 2024-03-04T09:31:42.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.853+0000 7f381b200640 1 -- 172.21.15.6:0/1242434422 shutdown_connections 2024-03-04T09:31:42.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.853+0000 7f381b200640 1 --2- 172.21.15.6:0/1242434422 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f37f406b500 0x7f37f406d9c0 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:42.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.853+0000 7f381b200640 1 --2- 172.21.15.6:0/1242434422 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f381415e1c0 0x7f381414f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:42.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.853+0000 7f381b200640 1 --2- 172.21.15.6:0/1242434422 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f381406f6b0 0x7f381415e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:42.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.853+0000 7f381b200640 1 --2- 172.21.15.6:0/1242434422 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f381406afb0 0x7f381415fe90 unknown :-1 s=CLOSED pgs=1318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:42.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.853+0000 7f381b200640 1 -- 172.21.15.6:0/1242434422 >> 172.21.15.6:0/1242434422 conn(0x7f38140b2a00 msgr2=0x7f38140b1a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:42.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.854+0000 7f381b200640 1 -- 172.21.15.6:0/1242434422 shutdown_connections 2024-03-04T09:31:42.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:42.854+0000 7f381b200640 1 -- 172.21.15.6:0/1242434422 wait complete. 2024-03-04T09:31:42.868 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-04T09:31:43.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:43.078+0000 7f4183200640 1 Processor -- start 2024-03-04T09:31:43.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:43.078+0000 7f4183200640 1 -- start start 2024-03-04T09:31:43.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:43.080+0000 7f4183200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f41840d4a10 0x7f41840d0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:43.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:43.080+0000 7f4183200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f41840d0c80 0x7f41840b44d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:43.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:43.080+0000 7f4183200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41840b3710 con 0x7f418415e0b0 2024-03-04T09:31:43.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:43.080+0000 7f4183200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41840d10f0 con 0x7f41840d4a10 2024-03-04T09:31:43.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:43.080+0000 7f4183200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41840b09e0 con 0x7f41840d0c80 2024-03-04T09:31:43.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:43.080+0000 7f417be00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f41840d4a10 0x7f41840d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:43.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:43.080+0000 7f417be00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f41840d4a10 0x7f41840d0740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:58448/0 (socket says 172.21.15.6:58448) 2024-03-04T09:31:43.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:43.080+0000 7f417be00640 1 -- 172.21.15.6:0/4128316330 learned_addr learned my addr 172.21.15.6:0/4128316330 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:31:43.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:43.080+0000 7f417be00640 1 -- 172.21.15.6:0/4128316330 >> v2:172.21.15.146:3300/0 conn(0x7f41840d0c80 msgr2=0x7f41840b44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:43.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:43.080+0000 7f417be00640 1 --2- 172.21.15.6:0/4128316330 >> v2:172.21.15.146:3300/0 conn(0x7f41840d0c80 0x7f41840b44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:43.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:43.080+0000 7f417be00640 1 -- 172.21.15.6:0/4128316330 >> v1:172.21.15.146:6789/0 conn(0x7f418415e0b0 legacy=0x7f41840cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:43.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:43.080+0000 7f417be00640 1 -- 172.21.15.6:0/4128316330 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41840b39f0 con 0x7f41840d4a10 2024-03-04T09:31:43.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:43.082+0000 7f417be00640 1 --2- 172.21.15.6:0/4128316330 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f41840d4a10 0x7f41840d0740 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7f418406b8c0 tx=0x7f416c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d674073b5d61879 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:43.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:43.082+0000 7f4181400640 1 -- 172.21.15.6:0/4128316330 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f416c035020 con 0x7f41840d4a10 2024-03-04T09:31:43.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:43.082+0000 7f4181400640 1 -- 172.21.15.6:0/4128316330 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f416c031640 con 0x7f41840d4a10 2024-03-04T09:31:43.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:43.082+0000 7f4181400640 1 -- 172.21.15.6:0/4128316330 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f416c02d040 con 0x7f41840d4a10 2024-03-04T09:31:43.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:43.083+0000 7f4183200640 1 -- 172.21.15.6:0/4128316330 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f41840d4a10 msgr2=0x7f41840d0740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:43.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:43.083+0000 7f4183200640 1 --2- 172.21.15.6:0/4128316330 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f41840d4a10 0x7f41840d0740 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7f418406b8c0 tx=0x7f416c02b8f0 comp rx=0 tx=0).stop 2024-03-04T09:31:43.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:43.083+0000 7f4183200640 1 -- 172.21.15.6:0/4128316330 shutdown_connections 2024-03-04T09:31:43.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:43.083+0000 7f4183200640 1 --2- 172.21.15.6:0/4128316330 >> v2:172.21.15.146:3300/0 conn(0x7f41840d0c80 0x7f41840b44d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:43.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:43.083+0000 7f4183200640 1 --2- 172.21.15.6:0/4128316330 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f41840d4a10 0x7f41840d0740 unknown :-1 s=CLOSED pgs=1323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:43.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:43.083+0000 7f4183200640 1 -- 172.21.15.6:0/4128316330 >> 172.21.15.6:0/4128316330 conn(0x7f41840d3d30 msgr2=0x7f41840d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:43.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:43.083+0000 7f4183200640 1 -- 172.21.15.6:0/4128316330 shutdown_connections 2024-03-04T09:31:43.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:43.083+0000 7f4183200640 1 -- 172.21.15.6:0/4128316330 wait complete. 2024-03-04T09:31:43.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:43.085+0000 7f4183200640 1 Processor -- start 2024-03-04T09:31:43.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:43.085+0000 7f4183200640 1 -- start start 2024-03-04T09:31:43.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:43.087+0000 7f4183200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f41840d0c80 0x7f4184157420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:43.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:43.087+0000 7f4183200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f41840d4a10 0x7f418415f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:43.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:43.087+0000 7f4183200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f418415e0b0 0x7f41840b8b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:43.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:43.087+0000 7f4183200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41840b3fb0 con 0x7f41840d0c80 2024-03-04T09:31:43.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:43.087+0000 7f4183200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41840cfa20 con 0x7f418415e0b0 2024-03-04T09:31:43.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:43.087+0000 7f4183200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f418415e490 con 0x7f41840d4a10 2024-03-04T09:31:43.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:43.087+0000 7f4181e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f41840d0c80 0x7f4184157420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:43.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:43.087+0000 7f4181e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f41840d0c80 0x7f4184157420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:58464/0 (socket says 172.21.15.6:58464) 2024-03-04T09:31:43.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:43.087+0000 7f4181e00640 1 -- 172.21.15.6:0/362210479 learned_addr learned my addr 172.21.15.6:0/362210479 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:31:43.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:43.088+0000 7f4181e00640 1 -- 172.21.15.6:0/362210479 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f41840d4a10 msgr2=0x7f418415f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:43.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:43.088+0000 7f4181e00640 1 --2- 172.21.15.6:0/362210479 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f41840d4a10 0x7f418415f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:43.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:43.088+0000 7f4182800640 1 --2- 172.21.15.6:0/362210479 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f418415e0b0 0x7f41840b8b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:43.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:43.088+0000 7f4181e00640 1 -- 172.21.15.6:0/362210479 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f418415e0b0 msgr2=0x7f41840b8b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:43.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:43.088+0000 7f4181e00640 1 --2- 172.21.15.6:0/362210479 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f418415e0b0 0x7f41840b8b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:43.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:43.088+0000 7f4181e00640 1 -- 172.21.15.6:0/362210479 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f418415ffd0 con 0x7f41840d0c80 2024-03-04T09:31:43.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:43.088+0000 7f4182800640 1 --2- 172.21.15.6:0/362210479 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f418415e0b0 0x7f41840b8b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:31:43.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:43.088+0000 7f4181e00640 1 --2- 172.21.15.6:0/362210479 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f41840d0c80 0x7f4184157420 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7f417400b3e0 tx=0x7f417400b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:43.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:43.088+0000 7f417a000640 1 -- 172.21.15.6:0/362210479 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4174015020 con 0x7f41840d0c80 2024-03-04T09:31:43.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:43.088+0000 7f417a000640 1 -- 172.21.15.6:0/362210479 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4174011850 con 0x7f41840d0c80 2024-03-04T09:31:43.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:43.088+0000 7f417a000640 1 -- 172.21.15.6:0/362210479 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4174011c40 con 0x7f41840d0c80 2024-03-04T09:31:43.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:43.088+0000 7f4183200640 1 -- 172.21.15.6:0/362210479 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f418415fa10 con 0x7f41840d0c80 2024-03-04T09:31:43.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:43.089+0000 7f4183200640 1 -- 172.21.15.6:0/362210479 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41840b6a80 con 0x7f41840d0c80 2024-03-04T09:31:43.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:43.090+0000 7f4183200640 1 -- 172.21.15.6:0/362210479 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41840683a0 con 0x7f41840d0c80 2024-03-04T09:31:43.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:43.091+0000 7f417a000640 1 -- 172.21.15.6:0/362210479 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4174018020 con 0x7f41840d0c80 2024-03-04T09:31:43.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:43.092+0000 7f417a000640 1 --2- 172.21.15.6:0/362210479 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f416006b500 0x7f416006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:43.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:43.092+0000 7f417a000640 1 -- 172.21.15.6:0/362210479 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f417408a170 con 0x7f41840d0c80 2024-03-04T09:31:43.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:43.093+0000 7f417be00640 1 --2- 172.21.15.6:0/362210479 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f416006b500 0x7f416006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:43.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:43.094+0000 7f417be00640 1 --2- 172.21.15.6:0/362210479 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f416006b500 0x7f416006d9c0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f416c036000 tx=0x7f416c030000 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:43.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:43.096+0000 7f417a000640 1 -- 172.21.15.6:0/362210479 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f41840683a0 con 0x7f41840d0c80 2024-03-04T09:31:43.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:43.339+0000 7f4183200640 1 -- 172.21.15.6:0/362210479 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f418404f330 con 0x7f41840d0c80 2024-03-04T09:31:43.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:43.340+0000 7f417a000640 1 -- 172.21.15.6:0/362210479 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v84) v1 ==== 53+0+73 (secure 0 0 0) 0x7f417401f020 con 0x7f41840d0c80 2024-03-04T09:31:43.336 INFO:teuthology.orchestra.run.smithi006.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-04T09:31:43.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:43.342+0000 7f4183200640 1 -- 172.21.15.6:0/362210479 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f416006b500 msgr2=0x7f416006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:43.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:43.342+0000 7f4183200640 1 --2- 172.21.15.6:0/362210479 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f416006b500 0x7f416006d9c0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f416c036000 tx=0x7f416c030000 comp rx=0 tx=0).stop 2024-03-04T09:31:43.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:43.342+0000 7f4183200640 1 -- 172.21.15.6:0/362210479 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f41840d0c80 msgr2=0x7f4184157420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:43.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:43.342+0000 7f4183200640 1 --2- 172.21.15.6:0/362210479 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f41840d0c80 0x7f4184157420 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7f417400b3e0 tx=0x7f417400b8a0 comp rx=0 tx=0).stop 2024-03-04T09:31:43.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:43.343+0000 7f4183200640 1 -- 172.21.15.6:0/362210479 shutdown_connections 2024-03-04T09:31:43.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:43.343+0000 7f4183200640 1 --2- 172.21.15.6:0/362210479 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f418415e0b0 0x7f41840b8b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:43.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:43.343+0000 7f4183200640 1 --2- 172.21.15.6:0/362210479 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f416006b500 0x7f416006d9c0 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:43.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:43.343+0000 7f4183200640 1 --2- 172.21.15.6:0/362210479 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f41840d4a10 0x7f418415f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:43.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:43.343+0000 7f4183200640 1 --2- 172.21.15.6:0/362210479 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f41840d0c80 0x7f4184157420 unknown :-1 s=CLOSED pgs=1324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:43.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:43.343+0000 7f4183200640 1 -- 172.21.15.6:0/362210479 >> 172.21.15.6:0/362210479 conn(0x7f41840d3d30 msgr2=0x7f41840b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:43.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:43.343+0000 7f4183200640 1 -- 172.21.15.6:0/362210479 shutdown_connections 2024-03-04T09:31:43.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:43.343+0000 7f4183200640 1 -- 172.21.15.6:0/362210479 wait complete. 2024-03-04T09:31:43.360 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-04T09:31:43.360 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-04T09:31:43.360 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi006.front.sepia.ceph.com 2024-03-04T09:31:43.361 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-04T09:31:43.361 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-04T09:31:43.361 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-04T09:31:43.361 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-03-04T09:31:43.361 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-04T09:31:43.361 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-04T09:31:43.361 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:31:43.361 DEBUG:teuthology.orchestra.run.smithi006:> ip addr 2024-03-04T09:31:43.412 INFO:teuthology.orchestra.run.smithi006.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-04T09:31:43.412 INFO:teuthology.orchestra.run.smithi006.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-04T09:31:43.412 INFO:teuthology.orchestra.run.smithi006.stdout: inet 127.0.0.1/8 scope host lo 2024-03-04T09:31:43.412 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-04T09:31:43.412 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 ::1/128 scope host 2024-03-04T09:31:43.412 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-04T09:31:43.412 INFO:teuthology.orchestra.run.smithi006.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-04T09:31:43.413 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:6b:fe:e6 brd ff:ff:ff:ff:ff:ff 2024-03-04T09:31:43.413 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp7s0f0 2024-03-04T09:31:43.413 INFO:teuthology.orchestra.run.smithi006.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-04T09:31:43.413 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:6b:fe:e7 brd ff:ff:ff:ff:ff:ff 2024-03-04T09:31:43.413 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp7s0f1 2024-03-04T09:31:43.413 INFO:teuthology.orchestra.run.smithi006.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-04T09:31:43.413 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:bd:15:e4 brd ff:ff:ff:ff:ff:ff 2024-03-04T09:31:43.413 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp1s0f0 2024-03-04T09:31:43.413 INFO:teuthology.orchestra.run.smithi006.stdout: inet 172.21.15.6/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-04T09:31:43.413 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft 42231sec preferred_lft 42231sec 2024-03-04T09:31:43.413 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::ec4:7aff:febd:15e4/64 scope link noprefixroute 2024-03-04T09:31:43.413 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-04T09:31:43.413 INFO:teuthology.orchestra.run.smithi006.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-04T09:31:43.413 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:bd:15:e5 brd ff:ff:ff:ff:ff:ff 2024-03-04T09:31:43.413 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp1s0f1 2024-03-04T09:31:43.414 INFO:teuthology.orchestra.run.smithi006.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-04T09:31:43.414 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 5e:b4:59:dc:42:58 brd ff:ff:ff:ff:ff:ff 2024-03-04T09:31:43.414 INFO:teuthology.orchestra.run.smithi006.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-04T09:31:43.414 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-04T09:31:43.414 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::c6f:adff:fefe:e00d/64 scope link proto kernel_ll 2024-03-04T09:31:43.414 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-04T09:31:43.414 INFO:teuthology.orchestra.run.smithi006.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-04T09:31:43.414 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 7e:67:6d:ea:f9:ff brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-04T09:31:43.414 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::7c67:6dff:feea:f9ff/64 scope link proto kernel_ll 2024-03-04T09:31:43.414 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-04T09:31:43.414 INFO:teuthology.orchestra.run.smithi006.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-04T09:31:43.414 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 5e:b4:59:dc:42:58 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-04T09:31:43.414 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::5cb4:59ff:fedc:4258/64 scope link proto kernel_ll 2024-03-04T09:31:43.414 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-04T09:31:43.415 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:31:43.416 DEBUG:teuthology.orchestra.run.smithi006:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-04T09:31:43.495 INFO:teuthology.orchestra.run.smithi006.stdout:1 2024-03-04T09:31:43.499 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:31:43.499 DEBUG:teuthology.orchestra.run.smithi006:> route 2024-03-04T09:31:43.548 INFO:teuthology.orchestra.run.smithi006.stdout:Kernel IP routing table 2024-03-04T09:31:43.549 INFO:teuthology.orchestra.run.smithi006.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-04T09:31:43.549 INFO:teuthology.orchestra.run.smithi006.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-04T09:31:43.549 INFO:teuthology.orchestra.run.smithi006.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-04T09:31:43.549 INFO:teuthology.orchestra.run.smithi006.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-04T09:31:43.549 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:31:43.549 DEBUG:teuthology.orchestra.run.smithi006:> (cd / && exec bash -c ' 2024-03-04T09:31:43.549 DEBUG:teuthology.orchestra.run.smithi006:> set -e 2024-03-04T09:31:43.549 DEBUG:teuthology.orchestra.run.smithi006:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-04T09:31:43.550 DEBUG:teuthology.orchestra.run.smithi006:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-04T09:31:43.550 DEBUG:teuthology.orchestra.run.smithi006:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-04T09:31:43.550 DEBUG:teuthology.orchestra.run.smithi006:> ') 2024-03-04T09:31:43.746 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:31:43.746 DEBUG:teuthology.orchestra.run.smithi006:> ip netns list 2024-03-04T09:31:43.760 INFO:tasks.ceph.mds.a.smithi006.stderr:2024-03-04T09:31:43.763+0000 7f03d0800640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-04T09:31:43.794 INFO:teuthology.orchestra.run.smithi006.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-04T09:31:43.795 INFO:teuthology.orchestra.run.smithi006.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-04T09:31:43.796 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-04T09:31:43.796 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:31:43.796 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip link set brx.0 up 2024-03-04T09:31:43.933 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-04T09:31:43.934 DEBUG:teuthology.orchestra.run.smithi006:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-04T09:31:43.987 INFO:teuthology.orchestra.run.smithi006.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-04T09:31:43.987 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-04T09:31:43.987 DEBUG:teuthology.orchestra.run.smithi006:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-04T09:31:44.039 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-03-04T09:31:44.039 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:31:44.039 DEBUG:teuthology.orchestra.run.smithi006:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-04T09:31:44.187 INFO:teuthology.orchestra.run.smithi006.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-04T09:31:44.187 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-04T09:31:44.218 INFO:teuthology.orchestra.run.smithi006.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-04T09:31:44.218 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-04T09:31:44.218 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: resolved to: "172.21.15.6:3300,172.21.15.146:3300,172.21.15.146:3301" 2024-03-04T09:31:44.218 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: trying mount with new device syntax: 0@edce0578-5ae4-4402-b45a-552bdef84d7c.cephfs=/ 2024-03-04T09:31:44.218 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.6:3300/172.21.15.146:3300/172.21.15.146:3301" will pass to kernel 2024-03-04T09:31:44.221 INFO:teuthology.orchestra.run.smithi006.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-04T09:31:44.221 INFO:teuthology.orchestra.run.smithi006.stdout: You just mounted a file system that supports labels which does not 2024-03-04T09:31:44.221 INFO:teuthology.orchestra.run.smithi006.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-04T09:31:44.221 INFO:teuthology.orchestra.run.smithi006.stdout: applications will generate AVC messages and not be allowed access to 2024-03-04T09:31:44.221 INFO:teuthology.orchestra.run.smithi006.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-04T09:31:44.224 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-04T09:31:44.224 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:31:44.224 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-04T09:31:44.309 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:31:44.309 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-04T09:31:44.371 INFO:teuthology.orchestra.run.smithi006.stdout:client7866 2024-03-04T09:31:44.371 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-04T09:31:44.371 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T09:31:44.422 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:31:44.422 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest/mnt.0 && exec rm -f dir_1/file_a) 2024-03-04T09:31:44.481 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:31:44.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:44.708+0000 7f45e8a00640 1 Processor -- start 2024-03-04T09:31:44.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:44.708+0000 7f45e8a00640 1 -- start start 2024-03-04T09:31:44.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:44.709+0000 7f45e8a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f45e4070f10 0x7f45e40cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:44.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:44.709+0000 7f45e8a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45e40714d0 con 0x7f45e40cce90 2024-03-04T09:31:44.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:44.709+0000 7f45e8a00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45e40717b0 con 0x7f45e4145300 2024-03-04T09:31:44.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:44.709+0000 7f45e8a00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45e40cd300 con 0x7f45e4070f10 2024-03-04T09:31:44.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:44.709+0000 7f45e2a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f45e4070f10 0x7f45e40cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:44.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:44.710+0000 7f45e2a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f45e4070f10 0x7f45e40cc950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:58480/0 (socket says 172.21.15.6:58480) 2024-03-04T09:31:44.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:44.710+0000 7f45e2a00640 1 -- 172.21.15.6:0/1136564387 learned_addr learned my addr 172.21.15.6:0/1136564387 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:31:44.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:44.710+0000 7f45e3e00640 1 --1- 172.21.15.6:0/1136564387 >> v1:172.21.15.6:6789/0 conn(0x7f45e40cce90 0x7f45e415c570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:50002/0 (socket says 172.21.15.6:50002) 2024-03-04T09:31:44.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:44.710+0000 7f45e2a00640 1 -- 172.21.15.6:0/1136564387 >> v1:172.21.15.146:6789/0 conn(0x7f45e4145300 legacy=0x7f45e40b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:44.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:44.710+0000 7f45e2a00640 1 -- 172.21.15.6:0/1136564387 >> v1:172.21.15.6:6789/0 conn(0x7f45e40cce90 legacy=0x7f45e415c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:44.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:44.710+0000 7f45e2000640 1 -- 172.21.15.6:0/1136564387 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4079868229 0 0) 0x7f45e40714d0 con 0x7f45e40cce90 2024-03-04T09:31:44.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:44.710+0000 7f45e2a00640 1 -- 172.21.15.6:0/1136564387 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45e4071a90 con 0x7f45e4070f10 2024-03-04T09:31:44.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:44.712+0000 7f45e2a00640 1 --2- 172.21.15.6:0/1136564387 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f45e4070f10 0x7f45e40cc950 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7f45d0009870 tx=0x7f45d002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e61d170ce737cd49 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:44.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:44.712+0000 7f45e2000640 1 -- 172.21.15.6:0/1136564387 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45d0035020 con 0x7f45e4070f10 2024-03-04T09:31:44.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:44.712+0000 7f45e2000640 1 -- 172.21.15.6:0/1136564387 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45d0031870 con 0x7f45e4070f10 2024-03-04T09:31:44.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:44.712+0000 7f45e2000640 1 -- 172.21.15.6:0/1136564387 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45d0031c40 con 0x7f45e4070f10 2024-03-04T09:31:44.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:44.713+0000 7f45e8a00640 1 -- 172.21.15.6:0/1136564387 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f45e4070f10 msgr2=0x7f45e40cc950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:44.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:44.713+0000 7f45e8a00640 1 --2- 172.21.15.6:0/1136564387 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f45e4070f10 0x7f45e40cc950 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7f45d0009870 tx=0x7f45d002b910 comp rx=0 tx=0).stop 2024-03-04T09:31:44.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:44.713+0000 7f45e8a00640 1 -- 172.21.15.6:0/1136564387 shutdown_connections 2024-03-04T09:31:44.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:44.713+0000 7f45e8a00640 1 --2- 172.21.15.6:0/1136564387 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f45e4070f10 0x7f45e40cc950 unknown :-1 s=CLOSED pgs=1327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:44.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:44.713+0000 7f45e8a00640 1 -- 172.21.15.6:0/1136564387 >> 172.21.15.6:0/1136564387 conn(0x7f45e406b3d0 msgr2=0x7f45e406b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:44.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:44.713+0000 7f45e8a00640 1 -- 172.21.15.6:0/1136564387 shutdown_connections 2024-03-04T09:31:44.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:44.713+0000 7f45e8a00640 1 -- 172.21.15.6:0/1136564387 wait complete. 2024-03-04T09:31:44.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:44.715+0000 7f45e8a00640 1 Processor -- start 2024-03-04T09:31:44.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:44.715+0000 7f45e8a00640 1 -- start start 2024-03-04T09:31:44.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:44.716+0000 7f45e8a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f45e4070f10 0x7f45e40cb6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:44.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:44.716+0000 7f45e8a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f45e40cce90 0x7f45e41e8b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:44.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:44.716+0000 7f45e8a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f45e4145300 0x7f45e41e90c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:44.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:44.716+0000 7f45e8a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45e415cc80 con 0x7f45e4145300 2024-03-04T09:31:44.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:44.716+0000 7f45e8a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45e41456e0 con 0x7f45e40cce90 2024-03-04T09:31:44.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:44.716+0000 7f45e8a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45e415ce00 con 0x7f45e4070f10 2024-03-04T09:31:44.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:44.716+0000 7f45e3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f45e4145300 0x7f45e41e90c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:44.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:44.716+0000 7f45e3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f45e4145300 0x7f45e41e90c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:58492/0 (socket says 172.21.15.6:58492) 2024-03-04T09:31:44.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:44.716+0000 7f45e3e00640 1 -- 172.21.15.6:0/3366651927 learned_addr learned my addr 172.21.15.6:0/3366651927 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:31:44.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:44.717+0000 7f45e3400640 1 --2- 172.21.15.6:0/3366651927 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f45e4070f10 0x7f45e40cb6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:44.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:44.717+0000 7f45e2a00640 1 --2- 172.21.15.6:0/3366651927 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f45e40cce90 0x7f45e41e8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:44.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:44.717+0000 7f45e3e00640 1 -- 172.21.15.6:0/3366651927 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f45e4070f10 msgr2=0x7f45e40cb6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:44.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:44.717+0000 7f45e3e00640 1 --2- 172.21.15.6:0/3366651927 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f45e4070f10 0x7f45e40cb6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:44.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:44.717+0000 7f45e3e00640 1 -- 172.21.15.6:0/3366651927 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f45e40cce90 msgr2=0x7f45e41e8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:44.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:44.717+0000 7f45e3e00640 1 --2- 172.21.15.6:0/3366651927 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f45e40cce90 0x7f45e41e8b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:44.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:44.717+0000 7f45e3e00640 1 -- 172.21.15.6:0/3366651927 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45e41e5d40 con 0x7f45e4145300 2024-03-04T09:31:44.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:44.717+0000 7f45e3e00640 1 --2- 172.21.15.6:0/3366651927 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f45e4145300 0x7f45e41e90c0 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7f45dc00c7a0 tx=0x7f45dc00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:44.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:44.718+0000 7f45db400640 1 -- 172.21.15.6:0/3366651927 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45dc018070 con 0x7f45e4145300 2024-03-04T09:31:44.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:44.718+0000 7f45db400640 1 -- 172.21.15.6:0/3366651927 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45dc00a640 con 0x7f45e4145300 2024-03-04T09:31:44.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:44.718+0000 7f45db400640 1 -- 172.21.15.6:0/3366651927 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45dc00aa70 con 0x7f45e4145300 2024-03-04T09:31:44.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:44.718+0000 7f45e8a00640 1 -- 172.21.15.6:0/3366651927 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45e4130340 con 0x7f45e4145300 2024-03-04T09:31:44.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:44.718+0000 7f45e8a00640 1 -- 172.21.15.6:0/3366651927 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45e4130900 con 0x7f45e4145300 2024-03-04T09:31:44.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:44.719+0000 7f45e8a00640 1 -- 172.21.15.6:0/3366651927 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45e404f330 con 0x7f45e4145300 2024-03-04T09:31:44.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:44.720+0000 7f45db400640 1 -- 172.21.15.6:0/3366651927 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f45dc00ac10 con 0x7f45e4145300 2024-03-04T09:31:44.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:44.721+0000 7f45db400640 1 --2- 172.21.15.6:0/3366651927 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f45c406b4d0 0x7f45c406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:44.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:44.721+0000 7f45db400640 1 -- 172.21.15.6:0/3366651927 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f45dc0898d0 con 0x7f45e4145300 2024-03-04T09:31:44.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:44.722+0000 7f45e3400640 1 --2- 172.21.15.6:0/3366651927 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f45c406b4d0 0x7f45c406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:44.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:44.723+0000 7f45e3400640 1 --2- 172.21.15.6:0/3366651927 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f45c406b4d0 0x7f45c406d990 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f45e4133490 tx=0x7f45d4009060 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:44.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:44.726+0000 7f45db400640 1 -- 172.21.15.6:0/3366651927 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f45dc058000 con 0x7f45e4145300 2024-03-04T09:31:45.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:45.020+0000 7f45e8a00640 1 -- 172.21.15.6:0/3366651927 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f45e40cc140 con 0x7f45e4145300 2024-03-04T09:31:45.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:45.021+0000 7f45db400640 1 -- 172.21.15.6:0/3366651927 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+4975 (secure 0 0 0) 0x7f45dc05c0a0 con 0x7f45e4145300 2024-03-04T09:31:45.018 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:31:45.018 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":84,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6535,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/3883122012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":3883122012},{"type":"v1","addr":"172.21.15.146:6835","nonce":3883122012}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":6559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/4060411493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":4060411493},{"type":"v1","addr":"172.21.15.146:6837","nonce":4060411493}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":82},{"gid":7812,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/679471398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":679471398},{"type":"v1","addr":"172.21.15.6:6833","nonce":679471398}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:31:21.114351+0000","modified":"2024-03-04T09:31:40.781474+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7815},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7815":{"gid":7815,"name":"a","rank":0,"incarnation":81,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/882667161","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":882667161},{"type":"v1","addr":"172.21.15.6:6835","nonce":882667161}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7815,"qdb_cluster":[7815]},"id":8}]} 2024-03-04T09:31:45.018 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 84 2024-03-04T09:31:45.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:45.024+0000 7f45e8a00640 1 -- 172.21.15.6:0/3366651927 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f45c406b4d0 msgr2=0x7f45c406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:45.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:45.024+0000 7f45e8a00640 1 --2- 172.21.15.6:0/3366651927 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f45c406b4d0 0x7f45c406d990 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f45e4133490 tx=0x7f45d4009060 comp rx=0 tx=0).stop 2024-03-04T09:31:45.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:45.024+0000 7f45e8a00640 1 -- 172.21.15.6:0/3366651927 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f45e4145300 msgr2=0x7f45e41e90c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:45.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:45.024+0000 7f45e8a00640 1 --2- 172.21.15.6:0/3366651927 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f45e4145300 0x7f45e41e90c0 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7f45dc00c7a0 tx=0x7f45dc00cc60 comp rx=0 tx=0).stop 2024-03-04T09:31:45.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:45.024+0000 7f45e8a00640 1 -- 172.21.15.6:0/3366651927 shutdown_connections 2024-03-04T09:31:45.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:45.024+0000 7f45e8a00640 1 --2- 172.21.15.6:0/3366651927 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f45e4145300 0x7f45e41e90c0 unknown :-1 s=CLOSED pgs=1328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:45.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:45.024+0000 7f45e8a00640 1 --2- 172.21.15.6:0/3366651927 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f45e40cce90 0x7f45e41e8b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:45.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:45.024+0000 7f45e8a00640 1 --2- 172.21.15.6:0/3366651927 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f45c406b4d0 0x7f45c406d990 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:45.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:45.024+0000 7f45e8a00640 1 --2- 172.21.15.6:0/3366651927 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f45e4070f10 0x7f45e40cb6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:45.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:45.024+0000 7f45e8a00640 1 -- 172.21.15.6:0/3366651927 >> 172.21.15.6:0/3366651927 conn(0x7f45e406b3d0 msgr2=0x7f45e40cd930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:45.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:45.024+0000 7f45e8a00640 1 -- 172.21.15.6:0/3366651927 shutdown_connections 2024-03-04T09:31:45.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:45.024+0000 7f45e8a00640 1 -- 172.21.15.6:0/3366651927 wait complete. 2024-03-04T09:31:45.039 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache num_strays 2024-03-04T09:31:45.269 INFO:teuthology.orchestra.run.smithi006.stdout:{"mds_cache":{"num_strays":1}} 2024-03-04T09:31:45.270 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "num_strays": 1 } } 2024-03-04T09:31:45.270 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:31:45.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:45.519+0000 7fbb47200640 1 Processor -- start 2024-03-04T09:31:45.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:45.520+0000 7fbb47200640 1 -- start start 2024-03-04T09:31:45.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:45.521+0000 7fbb47200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbb4815e0b0 0x7fbb480cf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:45.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:45.521+0000 7fbb47200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fbb480d0e50 0x7fbb480b44d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:45.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:45.521+0000 7fbb47200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb480b3710 con 0x7fbb480d4a10 2024-03-04T09:31:45.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:45.521+0000 7fbb47200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb480b09e0 con 0x7fbb480d0e50 2024-03-04T09:31:45.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:45.521+0000 7fbb47200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb480b39f0 con 0x7fbb4815e0b0 2024-03-04T09:31:45.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:45.522+0000 7fbb46800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fbb480d0e50 0x7fbb480b44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:45.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:45.522+0000 7fbb46800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fbb480d0e50 0x7fbb480b44d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:58508/0 (socket says 172.21.15.6:58508) 2024-03-04T09:31:45.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:45.522+0000 7fbb46800640 1 -- 172.21.15.6:0/1018944883 learned_addr learned my addr 172.21.15.6:0/1018944883 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:31:45.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:45.522+0000 7fbb3fe00640 1 --1- 172.21.15.6:0/1018944883 >> v1:172.21.15.6:6789/0 conn(0x7fbb480d4a10 0x7fbb480d0740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:50004/0 (socket says 172.21.15.6:50004) 2024-03-04T09:31:45.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:45.522+0000 7fbb45e00640 1 --2- 172.21.15.6:0/1018944883 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbb4815e0b0 0x7fbb480cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:45.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:45.522+0000 7fbb46800640 1 -- 172.21.15.6:0/1018944883 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbb4815e0b0 msgr2=0x7fbb480cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:31:45.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:45.522+0000 7fbb46800640 1 --2- 172.21.15.6:0/1018944883 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbb4815e0b0 0x7fbb480cf310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:45.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:45.522+0000 7fbb46800640 1 -- 172.21.15.6:0/1018944883 >> v1:172.21.15.6:6789/0 conn(0x7fbb480d4a10 legacy=0x7fbb480d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:45.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:45.522+0000 7fbb46800640 1 -- 172.21.15.6:0/1018944883 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb480b3cd0 con 0x7fbb480d0e50 2024-03-04T09:31:45.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:45.522+0000 7fbb45400640 1 -- 172.21.15.6:0/1018944883 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4290471928 0 0) 0x7fbb480b3710 con 0x7fbb480d4a10 2024-03-04T09:31:45.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:45.523+0000 7fbb46800640 1 --2- 172.21.15.6:0/1018944883 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbb480d0e50 0x7fbb480b44d0 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7fbb40009e50 tx=0x7fbb4002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b0a8ce75728e776 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:45.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:45.524+0000 7fbb45400640 1 -- 172.21.15.6:0/1018944883 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb4002cb00 con 0x7fbb480d0e50 2024-03-04T09:31:45.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:45.524+0000 7fbb45400640 1 -- 172.21.15.6:0/1018944883 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb4002cca0 con 0x7fbb480d0e50 2024-03-04T09:31:45.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:45.524+0000 7fbb45400640 1 -- 172.21.15.6:0/1018944883 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb40034610 con 0x7fbb480d0e50 2024-03-04T09:31:45.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:45.524+0000 7fbb47200640 1 -- 172.21.15.6:0/1018944883 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbb480d0e50 msgr2=0x7fbb480b44d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:45.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:45.524+0000 7fbb47200640 1 --2- 172.21.15.6:0/1018944883 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbb480d0e50 0x7fbb480b44d0 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7fbb40009e50 tx=0x7fbb4002bf60 comp rx=0 tx=0).stop 2024-03-04T09:31:45.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:45.524+0000 7fbb47200640 1 -- 172.21.15.6:0/1018944883 shutdown_connections 2024-03-04T09:31:45.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:45.524+0000 7fbb47200640 1 --2- 172.21.15.6:0/1018944883 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbb480d0e50 0x7fbb480b44d0 unknown :-1 s=CLOSED pgs=1330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:45.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:45.524+0000 7fbb47200640 1 --2- 172.21.15.6:0/1018944883 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbb4815e0b0 0x7fbb480cf310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:45.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:45.524+0000 7fbb47200640 1 -- 172.21.15.6:0/1018944883 >> 172.21.15.6:0/1018944883 conn(0x7fbb480d3d30 msgr2=0x7fbb480d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:45.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:45.525+0000 7fbb47200640 1 -- 172.21.15.6:0/1018944883 shutdown_connections 2024-03-04T09:31:45.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:45.525+0000 7fbb47200640 1 -- 172.21.15.6:0/1018944883 wait complete. 2024-03-04T09:31:45.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:45.526+0000 7fbb47200640 1 Processor -- start 2024-03-04T09:31:45.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:45.527+0000 7fbb47200640 1 -- start start 2024-03-04T09:31:45.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:45.528+0000 7fbb47200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbb480d0e50 0x7fbb4815b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:45.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:45.528+0000 7fbb47200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbb480d4a10 0x7fbb4814f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:45.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:45.528+0000 7fbb47200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbb4815e0b0 0x7fbb4815e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:45.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:45.528+0000 7fbb47200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb480cfc60 con 0x7fbb480d4a10 2024-03-04T09:31:45.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:45.528+0000 7fbb47200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb4815e490 con 0x7fbb480d0e50 2024-03-04T09:31:45.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:45.528+0000 7fbb47200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb480cf850 con 0x7fbb4815e0b0 2024-03-04T09:31:45.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:45.528+0000 7fbb3fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbb480d4a10 0x7fbb4814f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:45.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:45.528+0000 7fbb3fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbb480d4a10 0x7fbb4814f280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:58514/0 (socket says 172.21.15.6:58514) 2024-03-04T09:31:45.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:45.528+0000 7fbb3fe00640 1 -- 172.21.15.6:0/1280481916 learned_addr learned my addr 172.21.15.6:0/1280481916 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:31:45.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:45.529+0000 7fbb45e00640 1 --2- 172.21.15.6:0/1280481916 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbb480d0e50 0x7fbb4815b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:45.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:45.529+0000 7fbb3fe00640 1 -- 172.21.15.6:0/1280481916 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbb4815e0b0 msgr2=0x7fbb4815e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:45.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:45.529+0000 7fbb46800640 1 --2- 172.21.15.6:0/1280481916 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbb4815e0b0 0x7fbb4815e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:45.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:45.529+0000 7fbb3fe00640 1 --2- 172.21.15.6:0/1280481916 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbb4815e0b0 0x7fbb4815e610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:45.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:45.529+0000 7fbb3fe00640 1 -- 172.21.15.6:0/1280481916 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbb480d0e50 msgr2=0x7fbb4815b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:45.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:45.529+0000 7fbb3fe00640 1 --2- 172.21.15.6:0/1280481916 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbb480d0e50 0x7fbb4815b4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:45.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:45.529+0000 7fbb3fe00640 1 -- 172.21.15.6:0/1280481916 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb4815eb50 con 0x7fbb480d4a10 2024-03-04T09:31:45.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:45.529+0000 7fbb45e00640 1 --2- 172.21.15.6:0/1280481916 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbb480d0e50 0x7fbb4815b4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:31:45.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:45.529+0000 7fbb3fe00640 1 --2- 172.21.15.6:0/1280481916 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbb480d4a10 0x7fbb4814f280 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7fbb3000c4e0 tx=0x7fbb3000c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:45.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:45.529+0000 7fbb3e000640 1 -- 172.21.15.6:0/1280481916 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb30018070 con 0x7fbb480d4a10 2024-03-04T09:31:45.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:45.529+0000 7fbb3e000640 1 -- 172.21.15.6:0/1280481916 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb30007500 con 0x7fbb480d4a10 2024-03-04T09:31:45.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:45.529+0000 7fbb3e000640 1 -- 172.21.15.6:0/1280481916 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb30004a20 con 0x7fbb480d4a10 2024-03-04T09:31:45.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:45.529+0000 7fbb47200640 1 -- 172.21.15.6:0/1280481916 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb480cc4d0 con 0x7fbb480d4a10 2024-03-04T09:31:45.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:45.530+0000 7fbb47200640 1 -- 172.21.15.6:0/1280481916 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb480ccb00 con 0x7fbb480d4a10 2024-03-04T09:31:45.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:45.531+0000 7fbb47200640 1 -- 172.21.15.6:0/1280481916 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb4804f330 con 0x7fbb480d4a10 2024-03-04T09:31:45.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:45.532+0000 7fbb3e000640 1 -- 172.21.15.6:0/1280481916 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbb30004bc0 con 0x7fbb480d4a10 2024-03-04T09:31:45.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:45.533+0000 7fbb3e000640 1 --2- 172.21.15.6:0/1280481916 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fbb2406b4d0 0x7fbb2406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:31:45.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:45.533+0000 7fbb3e000640 1 -- 172.21.15.6:0/1280481916 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fbb30089610 con 0x7fbb480d4a10 2024-03-04T09:31:45.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:45.533+0000 7fbb45e00640 1 --2- 172.21.15.6:0/1280481916 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fbb2406b4d0 0x7fbb2406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:31:45.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:45.535+0000 7fbb45e00640 1 --2- 172.21.15.6:0/1280481916 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fbb2406b4d0 0x7fbb2406d990 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fbb38006fd0 tx=0x7fbb38008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:31:45.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:45.537+0000 7fbb3e000640 1 -- 172.21.15.6:0/1280481916 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbb300584e0 con 0x7fbb480d4a10 2024-03-04T09:31:45.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:45.833+0000 7fbb47200640 1 -- 172.21.15.6:0/1280481916 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbb480b3670 con 0x7fbb480d4a10 2024-03-04T09:31:45.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:45.834+0000 7fbb3e000640 1 -- 172.21.15.6:0/1280481916 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+4975 (secure 0 0 0) 0x7fbb30057de0 con 0x7fbb480d4a10 2024-03-04T09:31:45.831 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:31:45.831 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":84,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6535,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/3883122012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":3883122012},{"type":"v1","addr":"172.21.15.146:6835","nonce":3883122012}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":6559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/4060411493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":4060411493},{"type":"v1","addr":"172.21.15.146:6837","nonce":4060411493}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":82},{"gid":7812,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/679471398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":679471398},{"type":"v1","addr":"172.21.15.6:6833","nonce":679471398}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:31:21.114351+0000","modified":"2024-03-04T09:31:40.781474+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7815},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7815":{"gid":7815,"name":"a","rank":0,"incarnation":81,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/882667161","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":882667161},{"type":"v1","addr":"172.21.15.6:6835","nonce":882667161}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7815,"qdb_cluster":[7815]},"id":8}]} 2024-03-04T09:31:45.831 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 84 2024-03-04T09:31:45.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:45.836+0000 7fbb47200640 1 -- 172.21.15.6:0/1280481916 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fbb2406b4d0 msgr2=0x7fbb2406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:45.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:45.836+0000 7fbb47200640 1 --2- 172.21.15.6:0/1280481916 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fbb2406b4d0 0x7fbb2406d990 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fbb38006fd0 tx=0x7fbb38008040 comp rx=0 tx=0).stop 2024-03-04T09:31:45.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:45.836+0000 7fbb47200640 1 -- 172.21.15.6:0/1280481916 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbb480d4a10 msgr2=0x7fbb4814f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:31:45.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:45.836+0000 7fbb47200640 1 --2- 172.21.15.6:0/1280481916 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbb480d4a10 0x7fbb4814f280 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7fbb3000c4e0 tx=0x7fbb3000c9a0 comp rx=0 tx=0).stop 2024-03-04T09:31:45.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:45.836+0000 7fbb47200640 1 -- 172.21.15.6:0/1280481916 shutdown_connections 2024-03-04T09:31:45.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:45.836+0000 7fbb47200640 1 --2- 172.21.15.6:0/1280481916 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fbb2406b4d0 0x7fbb2406d990 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:45.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:45.836+0000 7fbb47200640 1 --2- 172.21.15.6:0/1280481916 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbb4815e0b0 0x7fbb4815e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:45.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:45.836+0000 7fbb47200640 1 --2- 172.21.15.6:0/1280481916 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbb480d4a10 0x7fbb4814f280 unknown :-1 s=CLOSED pgs=1331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:45.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:45.836+0000 7fbb47200640 1 --2- 172.21.15.6:0/1280481916 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbb480d0e50 0x7fbb4815b4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:31:45.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:45.836+0000 7fbb47200640 1 -- 172.21.15.6:0/1280481916 >> 172.21.15.6:0/1280481916 conn(0x7fbb480d3d30 msgr2=0x7fbb480b0b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:31:45.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:45.836+0000 7fbb47200640 1 -- 172.21.15.6:0/1280481916 shutdown_connections 2024-03-04T09:31:45.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:31:45.836+0000 7fbb47200640 1 -- 172.21.15.6:0/1280481916 wait complete. 2024-03-04T09:31:45.853 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_created 2024-03-04T09:31:46.097 INFO:teuthology.orchestra.run.smithi006.stdout:{"mds_cache":{"strays_created":1}} 2024-03-04T09:31:46.097 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_created": 1 } } 2024-03-04T09:32:10.436 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:32:10.445 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:32:16.098 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data ls 2024-03-04T09:32:16.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.203+0000 7fb51c442f40 1 Processor -- start 2024-03-04T09:32:16.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.203+0000 7fb51c442f40 1 -- start start 2024-03-04T09:32:16.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.204+0000 7fb51c442f40 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55f433ef80c0 0x55f433fb0ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:16.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.204+0000 7fb51c442f40 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f433ef84a0 0x55f433fb02a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:16.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.204+0000 7fb51c442f40 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55f433f46930 0x55f433fb2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:16.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.204+0000 7fb51c442f40 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f433ef9a40 con 0x55f433ef80c0 2024-03-04T09:32:16.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.204+0000 7fb51c442f40 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f433ea0fd0 con 0x55f433ef84a0 2024-03-04T09:32:16.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.204+0000 7fb51c442f40 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f433ef88b0 con 0x55f433f46930 2024-03-04T09:32:16.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.204+0000 7fb518e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55f433ef80c0 0x55f433fb0ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:16.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.204+0000 7fb518e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55f433ef80c0 0x55f433fb0ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:54482/0 (socket says 172.21.15.6:54482) 2024-03-04T09:32:16.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.204+0000 7fb518e00640 1 -- 172.21.15.6:0/4065113730 learned_addr learned my addr 172.21.15.6:0/4065113730 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:32:16.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.205+0000 7fb518e00640 1 -- 172.21.15.6:0/4065113730 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55f433f46930 msgr2=0x55f433fb2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:32:16.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.205+0000 7fb519800640 1 --2- 172.21.15.6:0/4065113730 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55f433f46930 0x55f433fb2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:16.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.205+0000 7fb510400640 1 --2- 172.21.15.6:0/4065113730 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f433ef84a0 0x55f433fb02a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:16.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.205+0000 7fb518e00640 1 --2- 172.21.15.6:0/4065113730 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55f433f46930 0x55f433fb2dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:16.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.205+0000 7fb518e00640 1 -- 172.21.15.6:0/4065113730 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f433ef84a0 msgr2=0x55f433fb02a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:32:16.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.205+0000 7fb518e00640 1 --2- 172.21.15.6:0/4065113730 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f433ef84a0 0x55f433fb02a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:16.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.205+0000 7fb518e00640 1 -- 172.21.15.6:0/4065113730 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f433fb39a0 con 0x55f433ef80c0 2024-03-04T09:32:16.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.205+0000 7fb518e00640 1 --2- 172.21.15.6:0/4065113730 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55f433ef80c0 0x55f433fb0ba0 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7fb50800a460 tx=0x7fb50800a920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b38b10b40fd21ef server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:16.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.205+0000 7fb513400640 1 -- 172.21.15.6:0/4065113730 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb50800b920 con 0x55f433ef80c0 2024-03-04T09:32:16.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.205+0000 7fb513400640 1 -- 172.21.15.6:0/4065113730 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb50800f070 con 0x55f433ef80c0 2024-03-04T09:32:16.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.206+0000 7fb513400640 1 -- 172.21.15.6:0/4065113730 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb50800bd20 con 0x55f433ef80c0 2024-03-04T09:32:16.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.206+0000 7fb51c442f40 1 -- 172.21.15.6:0/4065113730 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55f433ef80c0 msgr2=0x55f433fb0ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:16.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.206+0000 7fb51c442f40 1 --2- 172.21.15.6:0/4065113730 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55f433ef80c0 0x55f433fb0ba0 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7fb50800a460 tx=0x7fb50800a920 comp rx=0 tx=0).stop 2024-03-04T09:32:16.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.206+0000 7fb51c442f40 1 -- 172.21.15.6:0/4065113730 shutdown_connections 2024-03-04T09:32:16.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.206+0000 7fb51c442f40 1 --2- 172.21.15.6:0/4065113730 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55f433f46930 0x55f433fb2dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:16.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.206+0000 7fb51c442f40 1 --2- 172.21.15.6:0/4065113730 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f433ef84a0 0x55f433fb02a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:16.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.206+0000 7fb51c442f40 1 --2- 172.21.15.6:0/4065113730 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55f433ef80c0 0x55f433fb0ba0 unknown :-1 s=CLOSED pgs=1333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:16.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.206+0000 7fb51c442f40 1 -- 172.21.15.6:0/4065113730 >> 172.21.15.6:0/4065113730 conn(0x55f433f29c00 msgr2=0x55f433f29fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:32:16.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.206+0000 7fb51c442f40 1 -- 172.21.15.6:0/4065113730 shutdown_connections 2024-03-04T09:32:16.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.206+0000 7fb51c442f40 1 -- 172.21.15.6:0/4065113730 wait complete. 2024-03-04T09:32:16.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.208+0000 7fb51c442f40 1 Processor -- start 2024-03-04T09:32:16.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.208+0000 7fb51c442f40 1 -- start start 2024-03-04T09:32:16.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.209+0000 7fb51c442f40 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55f433ef80c0 0x55f433fd3c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:16.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.209+0000 7fb51c442f40 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f433ef84a0 0x55f433fcd6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:16.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.209+0000 7fb51c442f40 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55f433f46930 0x55f433fcbf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:16.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.209+0000 7fb51c442f40 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f433f2c210 con 0x55f433ef80c0 2024-03-04T09:32:16.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.209+0000 7fb51c442f40 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f433fafd90 con 0x55f433ef84a0 2024-03-04T09:32:16.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.209+0000 7fb51c442f40 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f433fb8bb0 con 0x55f433f46930 2024-03-04T09:32:16.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.210+0000 7fb518e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55f433ef80c0 0x55f433fd3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:16.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.210+0000 7fb518e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55f433ef80c0 0x55f433fd3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:54498/0 (socket says 172.21.15.6:54498) 2024-03-04T09:32:16.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.210+0000 7fb518e00640 1 -- 172.21.15.6:0/3016022178 learned_addr learned my addr 172.21.15.6:0/3016022178 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:32:16.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.210+0000 7fb519800640 1 --2- 172.21.15.6:0/3016022178 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55f433f46930 0x55f433fcbf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:16.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.210+0000 7fb518e00640 1 -- 172.21.15.6:0/3016022178 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55f433f46930 msgr2=0x55f433fcbf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:16.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.210+0000 7fb518e00640 1 --2- 172.21.15.6:0/3016022178 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55f433f46930 0x55f433fcbf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:16.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.210+0000 7fb518e00640 1 -- 172.21.15.6:0/3016022178 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f433ef84a0 msgr2=0x55f433fcd6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:16.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.210+0000 7fb510400640 1 --2- 172.21.15.6:0/3016022178 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f433ef84a0 0x55f433fcd6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:16.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.210+0000 7fb518e00640 1 --2- 172.21.15.6:0/3016022178 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f433ef84a0 0x55f433fcd6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:16.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.210+0000 7fb518e00640 1 -- 172.21.15.6:0/3016022178 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f433fb2d20 con 0x55f433ef80c0 2024-03-04T09:32:16.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.210+0000 7fb519800640 1 --2- 172.21.15.6:0/3016022178 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55f433f46930 0x55f433fcbf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:32:16.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.211+0000 7fb518e00640 1 --2- 172.21.15.6:0/3016022178 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55f433ef80c0 0x55f433fd3c30 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7fb50800a960 tx=0x7fb508002f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:16.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.211+0000 7fb512000640 1 -- 172.21.15.6:0/3016022178 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb50800ce60 con 0x55f433ef80c0 2024-03-04T09:32:16.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.211+0000 7fb512000640 1 -- 172.21.15.6:0/3016022178 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb508024070 con 0x55f433ef80c0 2024-03-04T09:32:16.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.211+0000 7fb51c442f40 1 -- 172.21.15.6:0/3016022178 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55f433fb49e0 con 0x55f433ef80c0 2024-03-04T09:32:16.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.211+0000 7fb512000640 1 -- 172.21.15.6:0/3016022178 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb508014ae0 con 0x55f433ef80c0 2024-03-04T09:32:16.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.211+0000 7fb51c442f40 1 -- 172.21.15.6:0/3016022178 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f433fb4fa0 con 0x55f433ef80c0 2024-03-04T09:32:16.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.213+0000 7fb512000640 1 -- 172.21.15.6:0/3016022178 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb508014c80 con 0x55f433ef80c0 2024-03-04T09:32:16.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.214+0000 7fb512000640 1 --2- 172.21.15.6:0/3016022178 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb4f4069ee0 0x7fb4f406c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:16.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.214+0000 7fb512000640 1 -- 172.21.15.6:0/3016022178 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5853+0+0 (secure 0 0 0) 0x7fb50808a8b0 con 0x55f433ef80c0 2024-03-04T09:32:16.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.215+0000 7fb51c442f40 1 --2- 172.21.15.6:0/3016022178 >> [v2:172.21.15.146:6816/3268323156,v1:172.21.15.146:6817/3268323156] conn(0x55f433fcc480 0x55f433fb59d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:16.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.215+0000 7fb51c442f40 1 -- 172.21.15.6:0/3016022178 --> [v2:172.21.15.146:6816/3268323156,v1:172.21.15.146:6817/3268323156] -- osd_op(unknown.0.0:1 17.0 17:00000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x55f433fb1fa0 con 0x55f433fcc480 2024-03-04T09:32:16.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.215+0000 7fb510400640 1 --2- 172.21.15.6:0/3016022178 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb4f4069ee0 0x7fb4f406c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:16.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.215+0000 7fb519800640 1 --2- 172.21.15.6:0/3016022178 >> [v2:172.21.15.146:6816/3268323156,v1:172.21.15.146:6817/3268323156] conn(0x55f433fcc480 0x55f433fb59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:16.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.216+0000 7fb510400640 1 --2- 172.21.15.6:0/3016022178 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb4f4069ee0 0x7fb4f406c3a0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fb500005400 tx=0x7fb500009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:16.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.216+0000 7fb519800640 1 --2- 172.21.15.6:0/3016022178 >> [v2:172.21.15.146:6816/3268323156,v1:172.21.15.146:6817/3268323156] conn(0x55f433fcc480 0x55f433fb59d0 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:16.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.217+0000 7fb519800640 1 -- 172.21.15.6:0/3016022178 <== osd.7 v2:172.21.15.146:6816/3268323156 1 ==== osd_op_reply(1 [pgnls start_epoch 115 out=49b] v109'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fb51405a040 con 0x55f433fcc480 2024-03-04T09:32:16.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.217+0000 7fb519800640 1 --2- 172.21.15.6:0/3016022178 >> [v2:172.21.15.6:6808/2494103078,v1:172.21.15.6:6809/2494103078] conn(0x7fb514052030 0x7fb51405e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:16.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.217+0000 7fb519800640 1 -- 172.21.15.6:0/3016022178 --> [v2:172.21.15.6:6808/2494103078,v1:172.21.15.6:6809/2494103078] -- osd_op(unknown.0.0:2 17.20 17:04000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x55f433fb1fa0 con 0x7fb514052030 2024-03-04T09:32:16.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.218+0000 7fb518e00640 1 --2- 172.21.15.6:0/3016022178 >> [v2:172.21.15.6:6808/2494103078,v1:172.21.15.6:6809/2494103078] conn(0x7fb514052030 0x7fb51405e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:16.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.218+0000 7fb518e00640 1 --2- 172.21.15.6:0/3016022178 >> [v2:172.21.15.6:6808/2494103078,v1:172.21.15.6:6809/2494103078] conn(0x7fb514052030 0x7fb51405e940 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:16.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.219+0000 7fb518e00640 1 -- 172.21.15.6:0/3016022178 <== osd.1 v2:172.21.15.6:6808/2494103078 1 ==== osd_op_reply(2 [pgnls start_epoch 115 out=49b] v109'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fb508055600 con 0x7fb514052030 2024-03-04T09:32:16.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.219+0000 7fb518e00640 1 --2- 172.21.15.6:0/3016022178 >> [v2:172.21.15.6:6800/1558806123,v1:172.21.15.6:6801/1558806123] conn(0x7fb5080591f0 0x7fb508057e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:16.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.219+0000 7fb518e00640 1 -- 172.21.15.6:0/3016022178 --> [v2:172.21.15.6:6800/1558806123,v1:172.21.15.6:6801/1558806123] -- osd_op(unknown.0.0:3 17.10 17:08000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x55f433fb1fa0 con 0x7fb5080591f0 2024-03-04T09:32:16.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.219+0000 7fb510400640 1 --2- 172.21.15.6:0/3016022178 >> [v2:172.21.15.6:6800/1558806123,v1:172.21.15.6:6801/1558806123] conn(0x7fb5080591f0 0x7fb508057e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:16.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.220+0000 7fb510400640 1 --2- 172.21.15.6:0/3016022178 >> [v2:172.21.15.6:6800/1558806123,v1:172.21.15.6:6801/1558806123] conn(0x7fb5080591f0 0x7fb508057e50 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:16.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.220+0000 7fb510400640 1 -- 172.21.15.6:0/3016022178 <== osd.3 v2:172.21.15.6:6800/1558806123 1 ==== osd_op_reply(3 [pgnls start_epoch 115 out=49b] v109'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fb500006900 con 0x7fb5080591f0 2024-03-04T09:32:16.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.220+0000 7fb510400640 1 --2- 172.21.15.6:0/3016022178 >> [v2:172.21.15.6:6823/1364449401,v1:172.21.15.6:6825/1364449401] conn(0x7fb500006b90 0x7fb5000123d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:16.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.220+0000 7fb510400640 1 -- 172.21.15.6:0/3016022178 --> [v2:172.21.15.6:6823/1364449401,v1:172.21.15.6:6825/1364449401] -- osd_op(unknown.0.0:4 17.30 17:0c000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x55f433fb1fa0 con 0x7fb500006b90 2024-03-04T09:32:16.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.221+0000 7fb519800640 1 --2- 172.21.15.6:0/3016022178 >> [v2:172.21.15.6:6823/1364449401,v1:172.21.15.6:6825/1364449401] conn(0x7fb500006b90 0x7fb5000123d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:16.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.221+0000 7fb519800640 1 --2- 172.21.15.6:0/3016022178 >> [v2:172.21.15.6:6823/1364449401,v1:172.21.15.6:6825/1364449401] conn(0x7fb500006b90 0x7fb5000123d0 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:16.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.222+0000 7fb519800640 1 -- 172.21.15.6:0/3016022178 <== osd.0 v2:172.21.15.6:6823/1364449401 1 ==== osd_op_reply(4 [pgnls start_epoch 115 out=49b] v109'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fb51405a040 con 0x7fb500006b90 2024-03-04T09:32:16.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.222+0000 7fb519800640 1 -- 172.21.15.6:0/3016022178 --> [v2:172.21.15.146:6816/3268323156,v1:172.21.15.146:6817/3268323156] -- osd_op(unknown.0.0:5 17.8 17:10000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x55f433fb1fa0 con 0x55f433fcc480 2024-03-04T09:32:16.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.223+0000 7fb519800640 1 -- 172.21.15.6:0/3016022178 <== osd.7 v2:172.21.15.146:6816/3268323156 2 ==== osd_op_reply(5 [pgnls start_epoch 115 out=49b] v109'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fb51405a040 con 0x55f433fcc480 2024-03-04T09:32:16.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.223+0000 7fb519800640 1 -- 172.21.15.6:0/3016022178 --> [v2:172.21.15.6:6800/1558806123,v1:172.21.15.6:6801/1558806123] -- osd_op(unknown.0.0:6 17.28 17:14000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x55f433fb1fa0 con 0x7fb5080591f0 2024-03-04T09:32:16.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.223+0000 7fb510400640 1 -- 172.21.15.6:0/3016022178 <== osd.3 v2:172.21.15.6:6800/1558806123 2 ==== osd_op_reply(6 [pgnls start_epoch 115 out=49b] v109'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fb500006900 con 0x7fb5080591f0 2024-03-04T09:32:16.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.223+0000 7fb510400640 1 -- 172.21.15.6:0/3016022178 --> [v2:172.21.15.6:6823/1364449401,v1:172.21.15.6:6825/1364449401] -- osd_op(unknown.0.0:7 17.18 17:18000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x55f433fb1fa0 con 0x7fb500006b90 2024-03-04T09:32:16.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.223+0000 7fb519800640 1 -- 172.21.15.6:0/3016022178 <== osd.0 v2:172.21.15.6:6823/1364449401 2 ==== osd_op_reply(7 [pgnls start_epoch 115 out=49b] v109'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fb51405a040 con 0x7fb500006b90 2024-03-04T09:32:16.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.223+0000 7fb519800640 1 -- 172.21.15.6:0/3016022178 --> [v2:172.21.15.146:6816/3268323156,v1:172.21.15.146:6817/3268323156] -- osd_op(unknown.0.0:8 17.38 17:1c000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x55f433fb1fa0 con 0x55f433fcc480 2024-03-04T09:32:16.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.224+0000 7fb519800640 1 -- 172.21.15.6:0/3016022178 <== osd.7 v2:172.21.15.146:6816/3268323156 3 ==== osd_op_reply(8 [pgnls start_epoch 115 out=49b] v109'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fb51405a040 con 0x55f433fcc480 2024-03-04T09:32:16.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.224+0000 7fb519800640 1 -- 172.21.15.6:0/3016022178 --> [v2:172.21.15.6:6823/1364449401,v1:172.21.15.6:6825/1364449401] -- osd_op(unknown.0.0:9 17.4 17:20000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x55f433fb1fa0 con 0x7fb500006b90 2024-03-04T09:32:16.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.225+0000 7fb519800640 1 -- 172.21.15.6:0/3016022178 <== osd.0 v2:172.21.15.6:6823/1364449401 3 ==== osd_op_reply(9 [pgnls start_epoch 115 out=49b] v109'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fb51405a040 con 0x7fb500006b90 2024-03-04T09:32:16.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.225+0000 7fb519800640 1 -- 172.21.15.6:0/3016022178 --> [v2:172.21.15.6:6800/1558806123,v1:172.21.15.6:6801/1558806123] -- osd_op(unknown.0.0:10 17.24 17:24000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x55f433fb1fa0 con 0x7fb5080591f0 2024-03-04T09:32:16.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.225+0000 7fb510400640 1 -- 172.21.15.6:0/3016022178 <== osd.3 v2:172.21.15.6:6800/1558806123 3 ==== osd_op_reply(10 [pgnls start_epoch 115 out=49b] v109'4 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fb500006900 con 0x7fb5080591f0 2024-03-04T09:32:16.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.225+0000 7fb510400640 1 -- 172.21.15.6:0/3016022178 --> [v2:172.21.15.6:6800/1558806123,v1:172.21.15.6:6801/1558806123] -- osd_op(unknown.0.0:11 17.14 17:28000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x55f433fb1fa0 con 0x7fb5080591f0 2024-03-04T09:32:16.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.225+0000 7fb510400640 1 -- 172.21.15.6:0/3016022178 <== osd.3 v2:172.21.15.6:6800/1558806123 4 ==== osd_op_reply(11 [pgnls start_epoch 115 out=49b] v109'11 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fb500006900 con 0x7fb5080591f0 2024-03-04T09:32:16.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.225+0000 7fb510400640 1 -- 172.21.15.6:0/3016022178 --> [v2:172.21.15.6:6823/1364449401,v1:172.21.15.6:6825/1364449401] -- osd_op(unknown.0.0:12 17.34 17:2c000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x55f433fb1fa0 con 0x7fb500006b90 2024-03-04T09:32:16.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.226+0000 7fb519800640 1 -- 172.21.15.6:0/3016022178 <== osd.0 v2:172.21.15.6:6823/1364449401 4 ==== osd_op_reply(12 [pgnls start_epoch 115 out=49b] v109'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fb51405a040 con 0x7fb500006b90 2024-03-04T09:32:16.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.226+0000 7fb519800640 1 -- 172.21.15.6:0/3016022178 --> [v2:172.21.15.146:6816/3268323156,v1:172.21.15.146:6817/3268323156] -- osd_op(unknown.0.0:13 17.c 17:30000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x55f433fb1fa0 con 0x55f433fcc480 2024-03-04T09:32:16.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.227+0000 7fb519800640 1 -- 172.21.15.6:0/3016022178 <== osd.7 v2:172.21.15.146:6816/3268323156 4 ==== osd_op_reply(13 [pgnls start_epoch 115 out=49b] v109'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fb51405a040 con 0x55f433fcc480 2024-03-04T09:32:16.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.227+0000 7fb519800640 1 --2- 172.21.15.6:0/3016022178 >> [v2:172.21.15.146:6800/1526135780,v1:172.21.15.146:6801/1526135780] conn(0x7fb514052750 0x7fb51405d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:16.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.227+0000 7fb519800640 1 -- 172.21.15.6:0/3016022178 --> [v2:172.21.15.146:6800/1526135780,v1:172.21.15.146:6801/1526135780] -- osd_op(unknown.0.0:14 17.2c 17:34000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x55f433fb1fa0 con 0x7fb514052750 2024-03-04T09:32:16.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.228+0000 7fb518e00640 1 --2- 172.21.15.6:0/3016022178 >> [v2:172.21.15.146:6800/1526135780,v1:172.21.15.146:6801/1526135780] conn(0x7fb514052750 0x7fb51405d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:16.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.229+0000 7fb518e00640 1 --2- 172.21.15.6:0/3016022178 >> [v2:172.21.15.146:6800/1526135780,v1:172.21.15.146:6801/1526135780] conn(0x7fb514052750 0x7fb51405d3c0 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:16.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.229+0000 7fb518e00640 1 -- 172.21.15.6:0/3016022178 <== osd.4 v2:172.21.15.146:6800/1526135780 1 ==== osd_op_reply(14 [pgnls start_epoch 115 out=49b] v109'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fb508055600 con 0x7fb514052750 2024-03-04T09:32:16.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.229+0000 7fb518e00640 1 -- 172.21.15.6:0/3016022178 --> [v2:172.21.15.6:6808/2494103078,v1:172.21.15.6:6809/2494103078] -- osd_op(unknown.0.0:15 17.1c 17:38000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x55f433fb1fa0 con 0x7fb514052030 2024-03-04T09:32:16.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.230+0000 7fb518e00640 1 -- 172.21.15.6:0/3016022178 <== osd.1 v2:172.21.15.6:6808/2494103078 2 ==== osd_op_reply(15 [pgnls start_epoch 115 out=49b] v109'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fb508055600 con 0x7fb514052030 2024-03-04T09:32:16.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.230+0000 7fb518e00640 1 -- 172.21.15.6:0/3016022178 --> [v2:172.21.15.6:6823/1364449401,v1:172.21.15.6:6825/1364449401] -- osd_op(unknown.0.0:16 17.3c 17:3c000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x55f433fb1fa0 con 0x7fb500006b90 2024-03-04T09:32:16.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.231+0000 7fb519800640 1 -- 172.21.15.6:0/3016022178 <== osd.0 v2:172.21.15.6:6823/1364449401 5 ==== osd_op_reply(16 [pgnls start_epoch 115 out=49b] v109'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fb51405a040 con 0x7fb500006b90 2024-03-04T09:32:16.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.231+0000 7fb519800640 1 -- 172.21.15.6:0/3016022178 --> [v2:172.21.15.146:6800/1526135780,v1:172.21.15.146:6801/1526135780] -- osd_op(unknown.0.0:17 17.2 17:40000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x55f433fb1fa0 con 0x7fb514052750 2024-03-04T09:32:16.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.231+0000 7fb518e00640 1 -- 172.21.15.6:0/3016022178 <== osd.4 v2:172.21.15.146:6800/1526135780 2 ==== osd_op_reply(17 [pgnls start_epoch 115 out=49b] v109'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fb508055600 con 0x7fb514052750 2024-03-04T09:32:16.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.231+0000 7fb518e00640 1 --2- 172.21.15.6:0/3016022178 >> [v2:172.21.15.146:6808/460262507,v1:172.21.15.146:6809/460262507] conn(0x7fb508061210 0x7fb50808a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:16.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.231+0000 7fb518e00640 1 -- 172.21.15.6:0/3016022178 --> [v2:172.21.15.146:6808/460262507,v1:172.21.15.146:6809/460262507] -- osd_op(unknown.0.0:18 17.22 17:44000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x55f433fb1fa0 con 0x7fb508061210 2024-03-04T09:32:16.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.232+0000 7fb510400640 1 --2- 172.21.15.6:0/3016022178 >> [v2:172.21.15.146:6808/460262507,v1:172.21.15.146:6809/460262507] conn(0x7fb508061210 0x7fb50808a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:16.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.233+0000 7fb510400640 1 --2- 172.21.15.6:0/3016022178 >> [v2:172.21.15.146:6808/460262507,v1:172.21.15.146:6809/460262507] conn(0x7fb508061210 0x7fb50808a180 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:16.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.234+0000 7fb510400640 1 -- 172.21.15.6:0/3016022178 <== osd.5 v2:172.21.15.146:6808/460262507 1 ==== osd_op_reply(18 [pgnls start_epoch 115 out=49b] v109'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fb500006900 con 0x7fb508061210 2024-03-04T09:32:16.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.234+0000 7fb510400640 1 -- 172.21.15.6:0/3016022178 --> [v2:172.21.15.6:6800/1558806123,v1:172.21.15.6:6801/1558806123] -- osd_op(unknown.0.0:19 17.12 17:48000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x55f433fb1fa0 con 0x7fb5080591f0 2024-03-04T09:32:16.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.234+0000 7fb510400640 1 -- 172.21.15.6:0/3016022178 <== osd.3 v2:172.21.15.6:6800/1558806123 5 ==== osd_op_reply(19 [pgnls start_epoch 115 out=49b] v109'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fb500006900 con 0x7fb5080591f0 2024-03-04T09:32:16.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.234+0000 7fb510400640 1 -- 172.21.15.6:0/3016022178 --> [v2:172.21.15.146:6816/3268323156,v1:172.21.15.146:6817/3268323156] -- osd_op(unknown.0.0:20 17.32 17:4c000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x7fb50002d040 con 0x55f433fcc480 2024-03-04T09:32:16.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.236+0000 7fb519800640 1 -- 172.21.15.6:0/3016022178 <== osd.7 v2:172.21.15.146:6816/3268323156 5 ==== osd_op_reply(20 [pgnls start_epoch 115 out=49b] v109'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fb51405a040 con 0x55f433fcc480 2024-03-04T09:32:16.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.236+0000 7fb519800640 1 -- 172.21.15.6:0/3016022178 --> [v2:172.21.15.146:6816/3268323156,v1:172.21.15.146:6817/3268323156] -- osd_op(unknown.0.0:21 17.a 17:50000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x7fb50002d040 con 0x55f433fcc480 2024-03-04T09:32:16.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.236+0000 7fb519800640 1 -- 172.21.15.6:0/3016022178 <== osd.7 v2:172.21.15.146:6816/3268323156 6 ==== osd_op_reply(21 [pgnls start_epoch 115 out=49b] v109'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fb51405a040 con 0x55f433fcc480 2024-03-04T09:32:16.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.237+0000 7fb519800640 1 -- 172.21.15.6:0/3016022178 --> [v2:172.21.15.6:6800/1558806123,v1:172.21.15.6:6801/1558806123] -- osd_op(unknown.0.0:22 17.2a 17:54000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x7fb50002d040 con 0x7fb5080591f0 2024-03-04T09:32:16.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.237+0000 7fb510400640 1 -- 172.21.15.6:0/3016022178 <== osd.3 v2:172.21.15.6:6800/1558806123 6 ==== osd_op_reply(22 [pgnls start_epoch 115 out=49b] v109'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fb500006900 con 0x7fb5080591f0 2024-03-04T09:32:16.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.237+0000 7fb510400640 1 -- 172.21.15.6:0/3016022178 --> [v2:172.21.15.6:6823/1364449401,v1:172.21.15.6:6825/1364449401] -- osd_op(unknown.0.0:23 17.1a 17:58000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x7fb50002d040 con 0x7fb500006b90 2024-03-04T09:32:16.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.238+0000 7fb519800640 1 -- 172.21.15.6:0/3016022178 <== osd.0 v2:172.21.15.6:6823/1364449401 6 ==== osd_op_reply(23 [pgnls start_epoch 115 out=49b] v109'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fb51405a040 con 0x7fb500006b90 2024-03-04T09:32:16.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.238+0000 7fb519800640 1 -- 172.21.15.6:0/3016022178 --> [v2:172.21.15.6:6808/2494103078,v1:172.21.15.6:6809/2494103078] -- osd_op(unknown.0.0:24 17.3a 17:5c000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x7fb50002d040 con 0x7fb514052030 2024-03-04T09:32:16.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.238+0000 7fb518e00640 1 -- 172.21.15.6:0/3016022178 <== osd.1 v2:172.21.15.6:6808/2494103078 3 ==== osd_op_reply(24 [pgnls start_epoch 115 out=49b] v109'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fb508055600 con 0x7fb514052030 2024-03-04T09:32:16.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.238+0000 7fb518e00640 1 --2- 172.21.15.6:0/3016022178 >> [v2:172.21.15.146:6821/1392897493,v1:172.21.15.146:6823/1392897493] conn(0x7fb508059920 0x7fb5080826a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:16.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.238+0000 7fb518e00640 1 -- 172.21.15.6:0/3016022178 --> [v2:172.21.15.146:6821/1392897493,v1:172.21.15.146:6823/1392897493] -- osd_op(unknown.0.0:25 17.6 17:60000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x7fb50002d040 con 0x7fb508059920 2024-03-04T09:32:16.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.239+0000 7fb519800640 1 --2- 172.21.15.6:0/3016022178 >> [v2:172.21.15.146:6821/1392897493,v1:172.21.15.146:6823/1392897493] conn(0x7fb508059920 0x7fb5080826a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:16.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.240+0000 7fb519800640 1 --2- 172.21.15.6:0/3016022178 >> [v2:172.21.15.146:6821/1392897493,v1:172.21.15.146:6823/1392897493] conn(0x7fb508059920 0x7fb5080826a0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:16.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.241+0000 7fb519800640 1 -- 172.21.15.6:0/3016022178 <== osd.6 v2:172.21.15.146:6821/1392897493 1 ==== osd_op_reply(25 [pgnls start_epoch 115 out=49b] v109'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fb51405a040 con 0x7fb508059920 2024-03-04T09:32:16.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.241+0000 7fb519800640 1 --2- 172.21.15.6:0/3016022178 >> [v2:172.21.15.6:6810/1500066681,v1:172.21.15.6:6812/1500066681] conn(0x7fb51407c960 0x7fb51406d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:16.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.241+0000 7fb519800640 1 -- 172.21.15.6:0/3016022178 --> [v2:172.21.15.6:6810/1500066681,v1:172.21.15.6:6812/1500066681] -- osd_op(unknown.0.0:26 17.26 17:64000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x7fb50002d040 con 0x7fb51407c960 2024-03-04T09:32:16.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.241+0000 7fb518e00640 1 --2- 172.21.15.6:0/3016022178 >> [v2:172.21.15.6:6810/1500066681,v1:172.21.15.6:6812/1500066681] conn(0x7fb51407c960 0x7fb51406d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:16.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.242+0000 7fb518e00640 1 --2- 172.21.15.6:0/3016022178 >> [v2:172.21.15.6:6810/1500066681,v1:172.21.15.6:6812/1500066681] conn(0x7fb51407c960 0x7fb51406d400 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:16.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.243+0000 7fb518e00640 1 -- 172.21.15.6:0/3016022178 <== osd.2 v2:172.21.15.6:6810/1500066681 1 ==== osd_op_reply(26 [pgnls start_epoch 115 out=49b] v109'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fb508055600 con 0x7fb51407c960 2024-03-04T09:32:16.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.243+0000 7fb518e00640 1 -- 172.21.15.6:0/3016022178 --> [v2:172.21.15.146:6808/460262507,v1:172.21.15.146:6809/460262507] -- osd_op(unknown.0.0:27 17.16 17:68000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x7fb50002d040 con 0x7fb508061210 2024-03-04T09:32:16.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.244+0000 7fb510400640 1 -- 172.21.15.6:0/3016022178 <== osd.5 v2:172.21.15.146:6808/460262507 2 ==== osd_op_reply(27 [pgnls start_epoch 115 out=49b] v109'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fb500006900 con 0x7fb508061210 2024-03-04T09:32:16.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.244+0000 7fb510400640 1 -- 172.21.15.6:0/3016022178 --> [v2:172.21.15.6:6823/1364449401,v1:172.21.15.6:6825/1364449401] -- osd_op(unknown.0.0:28 17.36 17:6c000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x7fb50002d040 con 0x7fb500006b90 2024-03-04T09:32:16.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.244+0000 7fb519800640 1 -- 172.21.15.6:0/3016022178 <== osd.0 v2:172.21.15.6:6823/1364449401 7 ==== osd_op_reply(28 [pgnls start_epoch 115 out=49b] v109'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fb51405a040 con 0x7fb500006b90 2024-03-04T09:32:16.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.244+0000 7fb519800640 1 -- 172.21.15.6:0/3016022178 --> [v2:172.21.15.6:6810/1500066681,v1:172.21.15.6:6812/1500066681] -- osd_op(unknown.0.0:29 17.e 17:70000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x7fb50002d040 con 0x7fb51407c960 2024-03-04T09:32:16.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.245+0000 7fb518e00640 1 -- 172.21.15.6:0/3016022178 <== osd.2 v2:172.21.15.6:6810/1500066681 2 ==== osd_op_reply(29 [pgnls start_epoch 115 out=49b] v109'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fb508055600 con 0x7fb51407c960 2024-03-04T09:32:16.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.245+0000 7fb518e00640 1 -- 172.21.15.6:0/3016022178 --> [v2:172.21.15.146:6821/1392897493,v1:172.21.15.146:6823/1392897493] -- osd_op(unknown.0.0:30 17.2e 17:74000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x7fb50002d040 con 0x7fb508059920 2024-03-04T09:32:16.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.246+0000 7fb519800640 1 -- 172.21.15.6:0/3016022178 <== osd.6 v2:172.21.15.146:6821/1392897493 2 ==== osd_op_reply(30 [pgnls start_epoch 115 out=49b] v109'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fb51405a040 con 0x7fb508059920 2024-03-04T09:32:16.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.246+0000 7fb519800640 1 -- 172.21.15.6:0/3016022178 --> [v2:172.21.15.6:6808/2494103078,v1:172.21.15.6:6809/2494103078] -- osd_op(unknown.0.0:31 17.1e 17:78000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x7fb50002d040 con 0x7fb514052030 2024-03-04T09:32:16.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.247+0000 7fb518e00640 1 -- 172.21.15.6:0/3016022178 <== osd.1 v2:172.21.15.6:6808/2494103078 4 ==== osd_op_reply(31 [pgnls start_epoch 115 out=49b] v109'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fb508055600 con 0x7fb514052030 2024-03-04T09:32:16.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.247+0000 7fb518e00640 1 -- 172.21.15.6:0/3016022178 --> [v2:172.21.15.146:6800/1526135780,v1:172.21.15.146:6801/1526135780] -- osd_op(unknown.0.0:32 17.3e 17:7c000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x7fb50002d040 con 0x7fb514052750 2024-03-04T09:32:16.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.247+0000 7fb518e00640 1 -- 172.21.15.6:0/3016022178 <== osd.4 v2:172.21.15.146:6800/1526135780 3 ==== osd_op_reply(32 [pgnls start_epoch 115 out=49b] v109'4 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fb508055600 con 0x7fb514052750 2024-03-04T09:32:16.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.248+0000 7fb518e00640 1 -- 172.21.15.6:0/3016022178 --> [v2:172.21.15.6:6823/1364449401,v1:172.21.15.6:6825/1364449401] -- osd_op(unknown.0.0:33 17.1 17:80000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x7fb50002d040 con 0x7fb500006b90 2024-03-04T09:32:16.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.248+0000 7fb519800640 1 -- 172.21.15.6:0/3016022178 <== osd.0 v2:172.21.15.6:6823/1364449401 8 ==== osd_op_reply(33 [pgnls start_epoch 115 out=49b] v109'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fb51405a040 con 0x7fb500006b90 2024-03-04T09:32:16.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.248+0000 7fb519800640 1 -- 172.21.15.6:0/3016022178 --> [v2:172.21.15.6:6823/1364449401,v1:172.21.15.6:6825/1364449401] -- osd_op(unknown.0.0:34 17.21 17:84000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x7fb50002d040 con 0x7fb500006b90 2024-03-04T09:32:16.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.249+0000 7fb519800640 1 -- 172.21.15.6:0/3016022178 <== osd.0 v2:172.21.15.6:6823/1364449401 9 ==== osd_op_reply(34 [pgnls start_epoch 115 out=49b] v109'12 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fb51405a040 con 0x7fb500006b90 2024-03-04T09:32:16.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.249+0000 7fb519800640 1 -- 172.21.15.6:0/3016022178 --> [v2:172.21.15.6:6800/1558806123,v1:172.21.15.6:6801/1558806123] -- osd_op(unknown.0.0:35 17.11 17:88000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x7fb50002d040 con 0x7fb5080591f0 2024-03-04T09:32:16.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.249+0000 7fb510400640 1 -- 172.21.15.6:0/3016022178 <== osd.3 v2:172.21.15.6:6800/1558806123 7 ==== osd_op_reply(35 [pgnls start_epoch 115 out=49b] v109'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fb500006900 con 0x7fb5080591f0 2024-03-04T09:32:16.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.249+0000 7fb510400640 1 -- 172.21.15.6:0/3016022178 --> [v2:172.21.15.6:6808/2494103078,v1:172.21.15.6:6809/2494103078] -- osd_op(unknown.0.0:36 17.31 17:8c000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x7fb50002d040 con 0x7fb514052030 2024-03-04T09:32:16.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.250+0000 7fb518e00640 1 -- 172.21.15.6:0/3016022178 <== osd.1 v2:172.21.15.6:6808/2494103078 5 ==== osd_op_reply(36 [pgnls start_epoch 115 out=49b] v109'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fb508055600 con 0x7fb514052030 2024-03-04T09:32:16.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.250+0000 7fb518e00640 1 -- 172.21.15.6:0/3016022178 --> [v2:172.21.15.6:6800/1558806123,v1:172.21.15.6:6801/1558806123] -- osd_op(unknown.0.0:37 17.9 17:90000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x7fb50002d040 con 0x7fb5080591f0 2024-03-04T09:32:16.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.250+0000 7fb510400640 1 -- 172.21.15.6:0/3016022178 <== osd.3 v2:172.21.15.6:6800/1558806123 8 ==== osd_op_reply(37 [pgnls start_epoch 115 out=49b] v109'2 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fb500006900 con 0x7fb5080591f0 2024-03-04T09:32:16.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.250+0000 7fb510400640 1 -- 172.21.15.6:0/3016022178 --> [v2:172.21.15.146:6800/1526135780,v1:172.21.15.146:6801/1526135780] -- osd_op(unknown.0.0:38 17.29 17:94000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x7fb50002d040 con 0x7fb514052750 2024-03-04T09:32:16.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.251+0000 7fb518e00640 1 -- 172.21.15.6:0/3016022178 <== osd.4 v2:172.21.15.146:6800/1526135780 4 ==== osd_op_reply(38 [pgnls start_epoch 115 out=49b] v109'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fb508055600 con 0x7fb514052750 2024-03-04T09:32:16.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.251+0000 7fb518e00640 1 -- 172.21.15.6:0/3016022178 --> [v2:172.21.15.146:6808/460262507,v1:172.21.15.146:6809/460262507] -- osd_op(unknown.0.0:39 17.19 17:98000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x7fb50002d040 con 0x7fb508061210 2024-03-04T09:32:16.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.252+0000 7fb510400640 1 -- 172.21.15.6:0/3016022178 <== osd.5 v2:172.21.15.146:6808/460262507 3 ==== osd_op_reply(39 [pgnls start_epoch 115 out=49b] v109'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fb500006900 con 0x7fb508061210 2024-03-04T09:32:16.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.252+0000 7fb510400640 1 -- 172.21.15.6:0/3016022178 --> [v2:172.21.15.146:6808/460262507,v1:172.21.15.146:6809/460262507] -- osd_op(unknown.0.0:40 17.39 17:9c000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x7fb50002d040 con 0x7fb508061210 2024-03-04T09:32:16.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.253+0000 7fb510400640 1 -- 172.21.15.6:0/3016022178 <== osd.5 v2:172.21.15.146:6808/460262507 4 ==== osd_op_reply(40 [pgnls start_epoch 115 out=49b] v109'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fb500006900 con 0x7fb508061210 2024-03-04T09:32:16.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.253+0000 7fb510400640 1 -- 172.21.15.6:0/3016022178 --> [v2:172.21.15.146:6821/1392897493,v1:172.21.15.146:6823/1392897493] -- osd_op(unknown.0.0:41 17.5 17:a0000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x7fb50002d040 con 0x7fb508059920 2024-03-04T09:32:16.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.254+0000 7fb519800640 1 -- 172.21.15.6:0/3016022178 <== osd.6 v2:172.21.15.146:6821/1392897493 3 ==== osd_op_reply(41 [pgnls start_epoch 115 out=49b] v109'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fb51405a040 con 0x7fb508059920 2024-03-04T09:32:16.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.254+0000 7fb519800640 1 -- 172.21.15.6:0/3016022178 --> [v2:172.21.15.6:6808/2494103078,v1:172.21.15.6:6809/2494103078] -- osd_op(unknown.0.0:42 17.25 17:a4000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x7fb50002d040 con 0x7fb514052030 2024-03-04T09:32:16.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.255+0000 7fb518e00640 1 -- 172.21.15.6:0/3016022178 <== osd.1 v2:172.21.15.6:6808/2494103078 6 ==== osd_op_reply(42 [pgnls start_epoch 115 out=49b] v109'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fb508055600 con 0x7fb514052030 2024-03-04T09:32:16.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.255+0000 7fb518e00640 1 -- 172.21.15.6:0/3016022178 --> [v2:172.21.15.146:6808/460262507,v1:172.21.15.146:6809/460262507] -- osd_op(unknown.0.0:43 17.15 17:a8000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x7fb50002d040 con 0x7fb508061210 2024-03-04T09:32:16.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.256+0000 7fb510400640 1 -- 172.21.15.6:0/3016022178 <== osd.5 v2:172.21.15.146:6808/460262507 5 ==== osd_op_reply(43 [pgnls start_epoch 115 out=49b] v109'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fb500006900 con 0x7fb508061210 2024-03-04T09:32:16.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.256+0000 7fb510400640 1 -- 172.21.15.6:0/3016022178 --> [v2:172.21.15.146:6800/1526135780,v1:172.21.15.146:6801/1526135780] -- osd_op(unknown.0.0:44 17.35 17:ac000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x7fb50002d040 con 0x7fb514052750 2024-03-04T09:32:16.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.257+0000 7fb518e00640 1 -- 172.21.15.6:0/3016022178 <== osd.4 v2:172.21.15.146:6800/1526135780 5 ==== osd_op_reply(44 [pgnls start_epoch 115 out=49b] v109'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fb508055600 con 0x7fb514052750 2024-03-04T09:32:16.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.257+0000 7fb518e00640 1 -- 172.21.15.6:0/3016022178 --> [v2:172.21.15.146:6816/3268323156,v1:172.21.15.146:6817/3268323156] -- osd_op(unknown.0.0:45 17.d 17:b0000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x7fb50002d040 con 0x55f433fcc480 2024-03-04T09:32:16.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.258+0000 7fb519800640 1 -- 172.21.15.6:0/3016022178 <== osd.7 v2:172.21.15.146:6816/3268323156 7 ==== osd_op_reply(45 [pgnls start_epoch 115 out=49b] v109'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fb51405a040 con 0x55f433fcc480 2024-03-04T09:32:16.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.258+0000 7fb519800640 1 -- 172.21.15.6:0/3016022178 --> [v2:172.21.15.146:6816/3268323156,v1:172.21.15.146:6817/3268323156] -- osd_op(unknown.0.0:46 17.2d 17:b4000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x7fb50002d040 con 0x55f433fcc480 2024-03-04T09:32:16.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.259+0000 7fb519800640 1 -- 172.21.15.6:0/3016022178 <== osd.7 v2:172.21.15.146:6816/3268323156 8 ==== osd_op_reply(46 [pgnls start_epoch 115 out=49b] v109'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fb51405a040 con 0x55f433fcc480 2024-03-04T09:32:16.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.259+0000 7fb519800640 1 -- 172.21.15.6:0/3016022178 --> [v2:172.21.15.146:6816/3268323156,v1:172.21.15.146:6817/3268323156] -- osd_op(unknown.0.0:47 17.1d 17:b8000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x7fb50002d040 con 0x55f433fcc480 2024-03-04T09:32:16.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.260+0000 7fb519800640 1 -- 172.21.15.6:0/3016022178 <== osd.7 v2:172.21.15.146:6816/3268323156 9 ==== osd_op_reply(47 [pgnls start_epoch 115 out=49b] v109'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fb51405a040 con 0x55f433fcc480 2024-03-04T09:32:16.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.260+0000 7fb519800640 1 -- 172.21.15.6:0/3016022178 --> [v2:172.21.15.146:6816/3268323156,v1:172.21.15.146:6817/3268323156] -- osd_op(unknown.0.0:48 17.3d 17:bc000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x7fb50002d040 con 0x55f433fcc480 2024-03-04T09:32:16.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.261+0000 7fb519800640 1 -- 172.21.15.6:0/3016022178 <== osd.7 v2:172.21.15.146:6816/3268323156 10 ==== osd_op_reply(48 [pgnls start_epoch 115 out=49b] v109'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fb51405a040 con 0x55f433fcc480 2024-03-04T09:32:16.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.261+0000 7fb519800640 1 -- 172.21.15.6:0/3016022178 --> [v2:172.21.15.146:6800/1526135780,v1:172.21.15.146:6801/1526135780] -- osd_op(unknown.0.0:49 17.3 17:c0000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x7fb50002d040 con 0x7fb514052750 2024-03-04T09:32:16.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.262+0000 7fb518e00640 1 -- 172.21.15.6:0/3016022178 <== osd.4 v2:172.21.15.146:6800/1526135780 6 ==== osd_op_reply(49 [pgnls start_epoch 115 out=49b] v109'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fb508055600 con 0x7fb514052750 2024-03-04T09:32:16.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.262+0000 7fb518e00640 1 -- 172.21.15.6:0/3016022178 --> [v2:172.21.15.146:6800/1526135780,v1:172.21.15.146:6801/1526135780] -- osd_op(unknown.0.0:50 17.23 17:c4000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x7fb50002d040 con 0x7fb514052750 2024-03-04T09:32:16.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.263+0000 7fb518e00640 1 -- 172.21.15.6:0/3016022178 <== osd.4 v2:172.21.15.146:6800/1526135780 7 ==== osd_op_reply(50 [pgnls start_epoch 115 out=49b] v109'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fb508055600 con 0x7fb514052750 2024-03-04T09:32:16.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.263+0000 7fb518e00640 1 -- 172.21.15.6:0/3016022178 --> [v2:172.21.15.6:6800/1558806123,v1:172.21.15.6:6801/1558806123] -- osd_op(unknown.0.0:51 17.13 17:c8000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x7fb50002d040 con 0x7fb5080591f0 2024-03-04T09:32:16.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.263+0000 7fb510400640 1 -- 172.21.15.6:0/3016022178 <== osd.3 v2:172.21.15.6:6800/1558806123 9 ==== osd_op_reply(51 [pgnls start_epoch 115 out=49b] v109'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fb500006900 con 0x7fb5080591f0 2024-03-04T09:32:16.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.263+0000 7fb510400640 1 -- 172.21.15.6:0/3016022178 --> [v2:172.21.15.146:6816/3268323156,v1:172.21.15.146:6817/3268323156] -- osd_op(unknown.0.0:52 17.33 17:cc000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x7fb50002d040 con 0x55f433fcc480 2024-03-04T09:32:16.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.264+0000 7fb519800640 1 -- 172.21.15.6:0/3016022178 <== osd.7 v2:172.21.15.146:6816/3268323156 11 ==== osd_op_reply(52 [pgnls start_epoch 115 out=49b] v109'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fb51405a040 con 0x55f433fcc480 2024-03-04T09:32:16.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.264+0000 7fb519800640 1 -- 172.21.15.6:0/3016022178 --> [v2:172.21.15.6:6823/1364449401,v1:172.21.15.6:6825/1364449401] -- osd_op(unknown.0.0:53 17.b 17:d0000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x7fb50002d040 con 0x7fb500006b90 2024-03-04T09:32:16.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.265+0000 7fb519800640 1 -- 172.21.15.6:0/3016022178 <== osd.0 v2:172.21.15.6:6823/1364449401 10 ==== osd_op_reply(53 [pgnls start_epoch 115 out=49b] v109'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fb51405a040 con 0x7fb500006b90 2024-03-04T09:32:16.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.265+0000 7fb519800640 1 -- 172.21.15.6:0/3016022178 --> [v2:172.21.15.146:6800/1526135780,v1:172.21.15.146:6801/1526135780] -- osd_op(unknown.0.0:54 17.2b 17:d4000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x7fb50002d040 con 0x7fb514052750 2024-03-04T09:32:16.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.266+0000 7fb518e00640 1 -- 172.21.15.6:0/3016022178 <== osd.4 v2:172.21.15.146:6800/1526135780 8 ==== osd_op_reply(54 [pgnls start_epoch 115 out=49b] v109'11 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fb508055600 con 0x7fb514052750 2024-03-04T09:32:16.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.266+0000 7fb518e00640 1 -- 172.21.15.6:0/3016022178 --> [v2:172.21.15.146:6800/1526135780,v1:172.21.15.146:6801/1526135780] -- osd_op(unknown.0.0:55 17.1b 17:d8000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x7fb50002d040 con 0x7fb514052750 2024-03-04T09:32:16.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.267+0000 7fb518e00640 1 -- 172.21.15.6:0/3016022178 <== osd.4 v2:172.21.15.146:6800/1526135780 9 ==== osd_op_reply(55 [pgnls start_epoch 115 out=49b] v109'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fb508055600 con 0x7fb514052750 2024-03-04T09:32:16.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.267+0000 7fb518e00640 1 -- 172.21.15.6:0/3016022178 --> [v2:172.21.15.6:6810/1500066681,v1:172.21.15.6:6812/1500066681] -- osd_op(unknown.0.0:56 17.3b 17:dc000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x7fb50002d040 con 0x7fb51407c960 2024-03-04T09:32:16.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.267+0000 7fb518e00640 1 -- 172.21.15.6:0/3016022178 <== osd.2 v2:172.21.15.6:6810/1500066681 3 ==== osd_op_reply(56 [pgnls start_epoch 115 out=49b] v109'4 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fb508055600 con 0x7fb51407c960 2024-03-04T09:32:16.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.267+0000 7fb518e00640 1 -- 172.21.15.6:0/3016022178 --> [v2:172.21.15.6:6823/1364449401,v1:172.21.15.6:6825/1364449401] -- osd_op(unknown.0.0:57 17.7 17:e0000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x7fb50002d040 con 0x7fb500006b90 2024-03-04T09:32:16.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.268+0000 7fb519800640 1 -- 172.21.15.6:0/3016022178 <== osd.0 v2:172.21.15.6:6823/1364449401 11 ==== osd_op_reply(57 [pgnls start_epoch 115 out=81b] v109'8 uv3 ondisk = 1) v8 ==== 144+0+81 (crc 0 0 0) 0x7fb51405a040 con 0x7fb500006b90 2024-03-04T09:32:16.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.268+0000 7fb519800640 1 -- 172.21.15.6:0/3016022178 --> [v2:172.21.15.146:6800/1526135780,v1:172.21.15.146:6801/1526135780] -- osd_op(unknown.0.0:58 17.27 17:e4000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x7fb50002d040 con 0x7fb514052750 2024-03-04T09:32:16.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.269+0000 7fb518e00640 1 -- 172.21.15.6:0/3016022178 <== osd.4 v2:172.21.15.146:6800/1526135780 10 ==== osd_op_reply(58 [pgnls start_epoch 115 out=49b] v109'2 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fb508055600 con 0x7fb514052750 2024-03-04T09:32:16.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.269+0000 7fb518e00640 1 -- 172.21.15.6:0/3016022178 --> [v2:172.21.15.146:6821/1392897493,v1:172.21.15.146:6823/1392897493] -- osd_op(unknown.0.0:59 17.17 17:e8000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x7fb50002d040 con 0x7fb508059920 2024-03-04T09:32:16.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.271+0000 7fb519800640 1 -- 172.21.15.6:0/3016022178 <== osd.6 v2:172.21.15.146:6821/1392897493 4 ==== osd_op_reply(59 [pgnls start_epoch 115 out=49b] v109'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fb51405a040 con 0x7fb508059920 2024-03-04T09:32:16.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.271+0000 7fb519800640 1 -- 172.21.15.6:0/3016022178 --> [v2:172.21.15.6:6823/1364449401,v1:172.21.15.6:6825/1364449401] -- osd_op(unknown.0.0:60 17.37 17:ec000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x7fb50002d040 con 0x7fb500006b90 2024-03-04T09:32:16.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.271+0000 7fb519800640 1 -- 172.21.15.6:0/3016022178 <== osd.0 v2:172.21.15.6:6823/1364449401 12 ==== osd_op_reply(60 [pgnls start_epoch 115 out=49b] v109'11 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fb51405a040 con 0x7fb500006b90 2024-03-04T09:32:16.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.271+0000 7fb519800640 1 -- 172.21.15.6:0/3016022178 --> [v2:172.21.15.6:6808/2494103078,v1:172.21.15.6:6809/2494103078] -- osd_op(unknown.0.0:61 17.f 17:f0000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x7fb50002d040 con 0x7fb514052030 2024-03-04T09:32:16.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.272+0000 7fb518e00640 1 -- 172.21.15.6:0/3016022178 <== osd.1 v2:172.21.15.6:6808/2494103078 7 ==== osd_op_reply(61 [pgnls start_epoch 115 out=49b] v109'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fb508055600 con 0x7fb514052030 2024-03-04T09:32:16.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.272+0000 7fb518e00640 1 -- 172.21.15.6:0/3016022178 --> [v2:172.21.15.146:6800/1526135780,v1:172.21.15.146:6801/1526135780] -- osd_op(unknown.0.0:62 17.2f 17:f4000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x7fb50002d040 con 0x7fb514052750 2024-03-04T09:32:16.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.273+0000 7fb518e00640 1 -- 172.21.15.6:0/3016022178 <== osd.4 v2:172.21.15.146:6800/1526135780 11 ==== osd_op_reply(62 [pgnls start_epoch 115 out=81b] v109'9 uv1 ondisk = 1) v8 ==== 144+0+81 (crc 0 0 0) 0x7fb508055600 con 0x7fb514052750 2024-03-04T09:32:16.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.273+0000 7fb518e00640 1 -- 172.21.15.6:0/3016022178 --> [v2:172.21.15.146:6821/1392897493,v1:172.21.15.146:6823/1392897493] -- osd_op(unknown.0.0:63 17.1f 17:f8000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x7fb50002d040 con 0x7fb508059920 2024-03-04T09:32:16.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.275+0000 7fb519800640 1 -- 172.21.15.6:0/3016022178 <== osd.6 v2:172.21.15.146:6821/1392897493 5 ==== osd_op_reply(63 [pgnls start_epoch 115 out=49b] v109'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fb51405a040 con 0x7fb508059920 2024-03-04T09:32:16.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.275+0000 7fb519800640 1 -- 172.21.15.6:0/3016022178 --> [v2:172.21.15.6:6823/1364449401,v1:172.21.15.6:6825/1364449401] -- osd_op(unknown.0.0:64 17.3f 17:fc000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x7fb50002d040 con 0x7fb500006b90 2024-03-04T09:32:16.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.275+0000 7fb519800640 1 -- 172.21.15.6:0/3016022178 <== osd.0 v2:172.21.15.6:6823/1364449401 13 ==== osd_op_reply(64 [pgnls start_epoch 115 out=49b] v109'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fb51405a040 con 0x7fb500006b90 2024-03-04T09:32:16.272 INFO:teuthology.orchestra.run.smithi006.stdout:10000000001.00000000 2024-03-04T09:32:16.273 INFO:teuthology.orchestra.run.smithi006.stdout:10000000001.00000001 2024-03-04T09:32:16.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.276+0000 7fb51c442f40 1 -- 172.21.15.6:0/3016022178 >> [v2:172.21.15.6:6823/1364449401,v1:172.21.15.6:6825/1364449401] conn(0x7fb500006b90 msgr2=0x7fb5000123d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:16.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.276+0000 7fb51c442f40 1 --2- 172.21.15.6:0/3016022178 >> [v2:172.21.15.6:6823/1364449401,v1:172.21.15.6:6825/1364449401] conn(0x7fb500006b90 0x7fb5000123d0 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:16.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.276+0000 7fb51c442f40 1 -- 172.21.15.6:0/3016022178 >> [v2:172.21.15.6:6808/2494103078,v1:172.21.15.6:6809/2494103078] conn(0x7fb514052030 msgr2=0x7fb51405e940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:16.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.276+0000 7fb51c442f40 1 --2- 172.21.15.6:0/3016022178 >> [v2:172.21.15.6:6808/2494103078,v1:172.21.15.6:6809/2494103078] conn(0x7fb514052030 0x7fb51405e940 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:16.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.276+0000 7fb51c442f40 1 -- 172.21.15.6:0/3016022178 >> [v2:172.21.15.6:6810/1500066681,v1:172.21.15.6:6812/1500066681] conn(0x7fb51407c960 msgr2=0x7fb51406d400 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:16.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.276+0000 7fb51c442f40 1 --2- 172.21.15.6:0/3016022178 >> [v2:172.21.15.6:6810/1500066681,v1:172.21.15.6:6812/1500066681] conn(0x7fb51407c960 0x7fb51406d400 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:16.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.276+0000 7fb51c442f40 1 -- 172.21.15.6:0/3016022178 >> [v2:172.21.15.6:6800/1558806123,v1:172.21.15.6:6801/1558806123] conn(0x7fb5080591f0 msgr2=0x7fb508057e50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:16.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.276+0000 7fb51c442f40 1 --2- 172.21.15.6:0/3016022178 >> [v2:172.21.15.6:6800/1558806123,v1:172.21.15.6:6801/1558806123] conn(0x7fb5080591f0 0x7fb508057e50 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:16.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.276+0000 7fb51c442f40 1 -- 172.21.15.6:0/3016022178 >> [v2:172.21.15.146:6800/1526135780,v1:172.21.15.146:6801/1526135780] conn(0x7fb514052750 msgr2=0x7fb51405d3c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:16.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.276+0000 7fb51c442f40 1 --2- 172.21.15.6:0/3016022178 >> [v2:172.21.15.146:6800/1526135780,v1:172.21.15.146:6801/1526135780] conn(0x7fb514052750 0x7fb51405d3c0 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:16.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.276+0000 7fb51c442f40 1 -- 172.21.15.6:0/3016022178 >> [v2:172.21.15.146:6808/460262507,v1:172.21.15.146:6809/460262507] conn(0x7fb508061210 msgr2=0x7fb50808a180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:16.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.276+0000 7fb51c442f40 1 --2- 172.21.15.6:0/3016022178 >> [v2:172.21.15.146:6808/460262507,v1:172.21.15.146:6809/460262507] conn(0x7fb508061210 0x7fb50808a180 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:16.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.276+0000 7fb519800640 1 -- 172.21.15.6:0/3016022178 reap_dead start 2024-03-04T09:32:16.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.276+0000 7fb51c442f40 1 -- 172.21.15.6:0/3016022178 >> [v2:172.21.15.146:6821/1392897493,v1:172.21.15.146:6823/1392897493] conn(0x7fb508059920 msgr2=0x7fb5080826a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:16.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.276+0000 7fb51c442f40 1 --2- 172.21.15.6:0/3016022178 >> [v2:172.21.15.146:6821/1392897493,v1:172.21.15.146:6823/1392897493] conn(0x7fb508059920 0x7fb5080826a0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:16.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.276+0000 7fb519800640 1 -- 172.21.15.6:0/3016022178 reap_dead start 2024-03-04T09:32:16.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.276+0000 7fb51c442f40 1 -- 172.21.15.6:0/3016022178 >> [v2:172.21.15.146:6816/3268323156,v1:172.21.15.146:6817/3268323156] conn(0x55f433fcc480 msgr2=0x55f433fb59d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:16.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.276+0000 7fb51c442f40 1 --2- 172.21.15.6:0/3016022178 >> [v2:172.21.15.146:6816/3268323156,v1:172.21.15.146:6817/3268323156] conn(0x55f433fcc480 0x55f433fb59d0 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:16.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.276+0000 7fb51c442f40 1 -- 172.21.15.6:0/3016022178 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb4f4069ee0 msgr2=0x7fb4f406c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:16.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.276+0000 7fb51c442f40 1 --2- 172.21.15.6:0/3016022178 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb4f4069ee0 0x7fb4f406c3a0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fb500005400 tx=0x7fb500009040 comp rx=0 tx=0).stop 2024-03-04T09:32:16.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.276+0000 7fb51c442f40 1 -- 172.21.15.6:0/3016022178 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55f433ef80c0 msgr2=0x55f433fd3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:16.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.276+0000 7fb51c442f40 1 --2- 172.21.15.6:0/3016022178 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55f433ef80c0 0x55f433fd3c30 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7fb50800a960 tx=0x7fb508002f70 comp rx=0 tx=0).stop 2024-03-04T09:32:16.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.277+0000 7fb51c442f40 1 -- 172.21.15.6:0/3016022178 shutdown_connections 2024-03-04T09:32:16.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.277+0000 7fb51c442f40 1 --2- 172.21.15.6:0/3016022178 >> [v2:172.21.15.146:6816/3268323156,v1:172.21.15.146:6817/3268323156] conn(0x55f433fcc480 0x55f433fb59d0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:16.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.277+0000 7fb51c442f40 1 --2- 172.21.15.6:0/3016022178 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb4f4069ee0 0x7fb4f406c3a0 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:16.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.277+0000 7fb51c442f40 1 --2- 172.21.15.6:0/3016022178 >> [v2:172.21.15.146:6821/1392897493,v1:172.21.15.146:6823/1392897493] conn(0x7fb508059920 0x7fb5080826a0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:16.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.277+0000 7fb51c442f40 1 --2- 172.21.15.6:0/3016022178 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55f433ef80c0 0x55f433fd3c30 unknown :-1 s=CLOSED pgs=1334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:16.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.277+0000 7fb51c442f40 1 -- 172.21.15.6:0/3016022178 >> 172.21.15.6:0/3016022178 conn(0x55f433f29c00 msgr2=0x55f433f29fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:32:16.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.277+0000 7fb51c442f40 1 -- 172.21.15.6:0/3016022178 shutdown_connections 2024-03-04T09:32:16.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.277+0000 7fb51c442f40 1 -- 172.21.15.6:0/3016022178 wait complete. 2024-03-04T09:32:16.279 DEBUG:tasks.cephfs.filesystem:All objects for ino 1099511627777 size 8388608 found 2024-03-04T09:32:16.280 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:32:16.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.497+0000 7fc293200640 1 Processor -- start 2024-03-04T09:32:16.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.497+0000 7fc293200640 1 -- start start 2024-03-04T09:32:16.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.498+0000 7fc293200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc28c0b6000 0x7fc28c162750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:16.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.499+0000 7fc293200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc28c1639f0 con 0x7fc28c0b3d70 2024-03-04T09:32:16.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.499+0000 7fc293200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc28c163430 con 0x7fc28c162c90 2024-03-04T09:32:16.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.499+0000 7fc293200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc28c163710 con 0x7fc28c0b6000 2024-03-04T09:32:16.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.499+0000 7fc28be00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fc28c0b3d70 0x7fc28c0b4150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:33504/0 (socket says 172.21.15.6:33504) 2024-03-04T09:32:16.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.499+0000 7fc28be00640 1 -- 172.21.15.6:0/3633407590 learned_addr learned my addr 172.21.15.6:0/3633407590 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:32:16.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.499+0000 7fc283400640 1 --2- 172.21.15.6:0/3633407590 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc28c0b6000 0x7fc28c162750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:16.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.499+0000 7fc28b400640 1 -- 172.21.15.6:0/3633407590 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 849286668 0 0) 0x7fc28c1639f0 con 0x7fc28c0b3d70 2024-03-04T09:32:16.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.499+0000 7fc28b400640 1 -- 172.21.15.6:0/3633407590 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc270003520 con 0x7fc28c0b3d70 2024-03-04T09:32:16.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.500+0000 7fc28b400640 1 -- 172.21.15.6:0/3633407590 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 644508134 0 0) 0x7fc270003520 con 0x7fc28c0b3d70 2024-03-04T09:32:16.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.500+0000 7fc28b400640 1 -- 172.21.15.6:0/3633407590 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc28c0b6000 msgr2=0x7fc28c162750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:32:16.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.500+0000 7fc28b400640 1 --2- 172.21.15.6:0/3633407590 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc28c0b6000 0x7fc28c162750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:16.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.500+0000 7fc28b400640 1 -- 172.21.15.6:0/3633407590 >> v1:172.21.15.146:6789/0 conn(0x7fc28c162c90 legacy=0x7fc28c0ae5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:16.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.500+0000 7fc28b400640 1 -- 172.21.15.6:0/3633407590 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc28c0addf0 con 0x7fc28c0b3d70 2024-03-04T09:32:16.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.500+0000 7fc28b400640 1 -- 172.21.15.6:0/3633407590 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7fc27c001670 con 0x7fc28c0b3d70 2024-03-04T09:32:16.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.500+0000 7fc28b400640 1 -- 172.21.15.6:0/3633407590 >> v1:172.21.15.6:6789/0 conn(0x7fc28c0b3d70 legacy=0x7fc28c0b4150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:16.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.500+0000 7fc28b400640 1 --2- 172.21.15.6:0/3633407590 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc270003a50 0x7fc270003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:16.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.500+0000 7fc28b400640 1 --2- 172.21.15.6:0/3633407590 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc28c0b6000 0x7fc270004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:16.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.500+0000 7fc28b400640 1 --2- 172.21.15.6:0/3633407590 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc270004d40 0x7fc2700051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:16.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.500+0000 7fc28b400640 1 -- 172.21.15.6:0/3633407590 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc28c163710 con 0x7fc270004d40 2024-03-04T09:32:16.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.500+0000 7fc28b400640 1 -- 172.21.15.6:0/3633407590 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc27000bb50 con 0x7fc270003a50 2024-03-04T09:32:16.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.500+0000 7fc28b400640 1 -- 172.21.15.6:0/3633407590 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc27000bd20 con 0x7fc28c0b6000 2024-03-04T09:32:16.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.500+0000 7fc291200640 1 --2- 172.21.15.6:0/3633407590 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc270004d40 0x7fc2700051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:16.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.501+0000 7fc28be00640 1 --2- 172.21.15.6:0/3633407590 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc270003a50 0x7fc270003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:16.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.501+0000 7fc283400640 1 --2- 172.21.15.6:0/3633407590 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc28c0b6000 0x7fc270004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:16.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.501+0000 7fc291200640 1 -- 172.21.15.6:0/3633407590 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc28c0b6000 msgr2=0x7fc270004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:32:16.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.501+0000 7fc291200640 1 --2- 172.21.15.6:0/3633407590 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc28c0b6000 0x7fc270004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:16.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.501+0000 7fc291200640 1 -- 172.21.15.6:0/3633407590 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc270003a50 msgr2=0x7fc270003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:32:16.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.501+0000 7fc291200640 1 --2- 172.21.15.6:0/3633407590 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc270003a50 0x7fc270003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:16.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.501+0000 7fc291200640 1 -- 172.21.15.6:0/3633407590 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc270003520 con 0x7fc270004d40 2024-03-04T09:32:16.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.501+0000 7fc28be00640 1 --2- 172.21.15.6:0/3633407590 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc270003a50 0x7fc270003e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T09:32:16.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.502+0000 7fc291200640 1 --2- 172.21.15.6:0/3633407590 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc270004d40 0x7fc2700051c0 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7fc284001e50 tx=0x7fc270005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e141413781635d4 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:16.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.502+0000 7fc28b400640 1 -- 172.21.15.6:0/3633407590 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc284001a70 con 0x7fc270004d40 2024-03-04T09:32:16.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.502+0000 7fc28b400640 1 -- 172.21.15.6:0/3633407590 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc284003b70 con 0x7fc270004d40 2024-03-04T09:32:16.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.502+0000 7fc28b400640 1 -- 172.21.15.6:0/3633407590 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2840050d0 con 0x7fc270004d40 2024-03-04T09:32:16.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.503+0000 7fc293200640 1 -- 172.21.15.6:0/3633407590 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc270004d40 msgr2=0x7fc2700051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:16.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.503+0000 7fc293200640 1 --2- 172.21.15.6:0/3633407590 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc270004d40 0x7fc2700051c0 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7fc284001e50 tx=0x7fc270005830 comp rx=0 tx=0).stop 2024-03-04T09:32:16.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.503+0000 7fc291200640 1 -- 172.21.15.6:0/3633407590 reap_dead start 2024-03-04T09:32:16.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.503+0000 7fc293200640 1 -- 172.21.15.6:0/3633407590 shutdown_connections 2024-03-04T09:32:16.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.503+0000 7fc293200640 1 -- 172.21.15.6:0/3633407590 >> 172.21.15.6:0/3633407590 conn(0x7fc28c06c2a0 msgr2=0x7fc28c06c6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:32:16.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.503+0000 7fc293200640 1 -- 172.21.15.6:0/3633407590 shutdown_connections 2024-03-04T09:32:16.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.503+0000 7fc293200640 1 -- 172.21.15.6:0/3633407590 wait complete. 2024-03-04T09:32:16.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.505+0000 7fc293200640 1 Processor -- start 2024-03-04T09:32:16.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.505+0000 7fc293200640 1 -- start start 2024-03-04T09:32:16.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.506+0000 7fc293200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc270004d40 0x7fc28c09e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:16.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.506+0000 7fc293200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc28c09aa10 0x7fc28c1fb9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:16.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.506+0000 7fc293200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc28c09ae90 0x7fc28c1fbf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:16.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.506+0000 7fc293200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc28c162750 con 0x7fc28c09ae90 2024-03-04T09:32:16.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.506+0000 7fc293200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc28c163120 con 0x7fc270004d40 2024-03-04T09:32:16.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.506+0000 7fc293200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc28c1628d0 con 0x7fc28c09aa10 2024-03-04T09:32:16.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.506+0000 7fc291200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc28c09ae90 0x7fc28c1fbf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:16.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.506+0000 7fc291200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc28c09ae90 0x7fc28c1fbf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:54514/0 (socket says 172.21.15.6:54514) 2024-03-04T09:32:16.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.506+0000 7fc291200640 1 -- 172.21.15.6:0/3619385918 learned_addr learned my addr 172.21.15.6:0/3619385918 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:32:16.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.507+0000 7fc283400640 1 --2- 172.21.15.6:0/3619385918 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc28c09aa10 0x7fc28c1fb9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:16.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.507+0000 7fc291200640 1 -- 172.21.15.6:0/3619385918 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc28c09aa10 msgr2=0x7fc28c1fb9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:16.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.507+0000 7fc28be00640 1 --2- 172.21.15.6:0/3619385918 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc270004d40 0x7fc28c09e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:16.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.507+0000 7fc291200640 1 --2- 172.21.15.6:0/3619385918 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc28c09aa10 0x7fc28c1fb9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:16.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.507+0000 7fc291200640 1 -- 172.21.15.6:0/3619385918 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc270004d40 msgr2=0x7fc28c09e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:16.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.507+0000 7fc291200640 1 --2- 172.21.15.6:0/3619385918 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc270004d40 0x7fc28c09e340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:16.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.507+0000 7fc291200640 1 -- 172.21.15.6:0/3619385918 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc28c09c500 con 0x7fc28c09ae90 2024-03-04T09:32:16.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.507+0000 7fc291200640 1 --2- 172.21.15.6:0/3619385918 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc28c09ae90 0x7fc28c1fbf30 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7fc28402c020 tx=0x7fc2840014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:16.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.507+0000 7fc288c00640 1 -- 172.21.15.6:0/3619385918 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc284007db0 con 0x7fc28c09ae90 2024-03-04T09:32:16.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.508+0000 7fc288c00640 1 -- 172.21.15.6:0/3619385918 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc284006b50 con 0x7fc28c09ae90 2024-03-04T09:32:16.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.508+0000 7fc293200640 1 -- 172.21.15.6:0/3619385918 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc28c09b460 con 0x7fc28c09ae90 2024-03-04T09:32:16.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.508+0000 7fc288c00640 1 -- 172.21.15.6:0/3619385918 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc284007030 con 0x7fc28c09ae90 2024-03-04T09:32:16.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.508+0000 7fc293200640 1 -- 172.21.15.6:0/3619385918 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc28c09ba20 con 0x7fc28c09ae90 2024-03-04T09:32:16.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.509+0000 7fc293200640 1 -- 172.21.15.6:0/3619385918 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc28c04f3a0 con 0x7fc28c09ae90 2024-03-04T09:32:16.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.510+0000 7fc288c00640 1 -- 172.21.15.6:0/3619385918 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc2840071d0 con 0x7fc28c09ae90 2024-03-04T09:32:16.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.511+0000 7fc288c00640 1 --2- 172.21.15.6:0/3619385918 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc26806b500 0x7fc26806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:16.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.512+0000 7fc288c00640 1 -- 172.21.15.6:0/3619385918 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5853+0+0 (secure 0 0 0) 0x7fc2840ac010 con 0x7fc28c09ae90 2024-03-04T09:32:16.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.512+0000 7fc28be00640 1 --2- 172.21.15.6:0/3619385918 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc26806b500 0x7fc26806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:16.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.513+0000 7fc28be00640 1 --2- 172.21.15.6:0/3619385918 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc26806b500 0x7fc26806d9c0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fc27c001eb0 tx=0x7fc27c000ed0 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:16.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.515+0000 7fc288c00640 1 -- 172.21.15.6:0/3619385918 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc284048060 con 0x7fc28c09ae90 2024-03-04T09:32:16.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.766+0000 7fc293200640 1 -- 172.21.15.6:0/3619385918 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc28c072120 con 0x7fc28c09ae90 2024-03-04T09:32:16.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.767+0000 7fc288c00640 1 -- 172.21.15.6:0/3619385918 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+4975 (secure 0 0 0) 0x7fc28c09c500 con 0x7fc28c09ae90 2024-03-04T09:32:16.763 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:32:16.764 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":84,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6535,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/3883122012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":3883122012},{"type":"v1","addr":"172.21.15.146:6835","nonce":3883122012}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":6559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/4060411493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":4060411493},{"type":"v1","addr":"172.21.15.146:6837","nonce":4060411493}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":82},{"gid":7812,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/679471398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":679471398},{"type":"v1","addr":"172.21.15.6:6833","nonce":679471398}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:31:21.114351+0000","modified":"2024-03-04T09:31:40.781474+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7815},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7815":{"gid":7815,"name":"a","rank":0,"incarnation":81,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/882667161","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":882667161},{"type":"v1","addr":"172.21.15.6:6835","nonce":882667161}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7815,"qdb_cluster":[7815]},"id":8}]} 2024-03-04T09:32:16.764 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 84 2024-03-04T09:32:16.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.769+0000 7fc293200640 1 -- 172.21.15.6:0/3619385918 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc26806b500 msgr2=0x7fc26806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:16.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.769+0000 7fc293200640 1 --2- 172.21.15.6:0/3619385918 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc26806b500 0x7fc26806d9c0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fc27c001eb0 tx=0x7fc27c000ed0 comp rx=0 tx=0).stop 2024-03-04T09:32:16.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.769+0000 7fc293200640 1 -- 172.21.15.6:0/3619385918 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc28c09ae90 msgr2=0x7fc28c1fbf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:16.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.769+0000 7fc293200640 1 --2- 172.21.15.6:0/3619385918 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc28c09ae90 0x7fc28c1fbf30 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7fc28402c020 tx=0x7fc2840014a0 comp rx=0 tx=0).stop 2024-03-04T09:32:16.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.769+0000 7fc293200640 1 -- 172.21.15.6:0/3619385918 shutdown_connections 2024-03-04T09:32:16.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.769+0000 7fc293200640 1 --2- 172.21.15.6:0/3619385918 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc28c09ae90 0x7fc28c1fbf30 unknown :-1 s=CLOSED pgs=1337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:16.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.769+0000 7fc293200640 1 --2- 172.21.15.6:0/3619385918 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc26806b500 0x7fc26806d9c0 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:16.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.769+0000 7fc293200640 1 --2- 172.21.15.6:0/3619385918 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc28c09aa10 0x7fc28c1fb9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:16.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.769+0000 7fc293200640 1 --2- 172.21.15.6:0/3619385918 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc270004d40 0x7fc28c09e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:16.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.770+0000 7fc293200640 1 -- 172.21.15.6:0/3619385918 >> 172.21.15.6:0/3619385918 conn(0x7fc28c06c2a0 msgr2=0x7fc28c073f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:32:16.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.770+0000 7fc293200640 1 -- 172.21.15.6:0/3619385918 shutdown_connections 2024-03-04T09:32:16.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:16.770+0000 7fc293200640 1 -- 172.21.15.6:0/3619385918 wait complete. 2024-03-04T09:32:16.786 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_enqueued 2024-03-04T09:32:17.054 INFO:teuthology.orchestra.run.smithi006.stdout:{"mds_cache":{"strays_enqueued":0}} 2024-03-04T09:32:17.054 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 0 } } 2024-03-04T09:32:17.055 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:32:17.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:17.277+0000 7fbe8b200640 1 Processor -- start 2024-03-04T09:32:17.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:17.277+0000 7fbe8b200640 1 -- start start 2024-03-04T09:32:17.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:17.278+0000 7fbe8b200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbe8c15f450 0x7fbe8c070990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:17.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:17.278+0000 7fbe8b200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fbe8c0d04b0 0x7fbe8c0cde80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:17.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:17.279+0000 7fbe8b200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe8c0cd0c0 con 0x7fbe8c0700c0 2024-03-04T09:32:17.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:17.279+0000 7fbe8b200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe8c0d29e0 con 0x7fbe8c0d04b0 2024-03-04T09:32:17.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:17.279+0000 7fbe8b200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe8c0cd3a0 con 0x7fbe8c15f450 2024-03-04T09:32:17.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:17.279+0000 7fbe89e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbe8c15f450 0x7fbe8c070990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:17.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:17.279+0000 7fbe89e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbe8c15f450 0x7fbe8c070990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.6:33122/0 (socket says 172.21.15.6:33122) 2024-03-04T09:32:17.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:17.279+0000 7fbe89e00640 1 -- 172.21.15.6:0/742409254 learned_addr learned my addr 172.21.15.6:0/742409254 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:32:17.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:17.280+0000 7fbe8a800640 1 --2- 172.21.15.6:0/742409254 >> v2:172.21.15.146:3300/0 conn(0x7fbe8c0d04b0 0x7fbe8c0cde80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:17.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:17.280+0000 7fbe89400640 1 -- 172.21.15.6:0/742409254 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 253789182 0 0) 0x7fbe8c0cd0c0 con 0x7fbe8c0700c0 2024-03-04T09:32:17.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:17.280+0000 7fbe89400640 1 -- 172.21.15.6:0/742409254 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe70003520 con 0x7fbe8c0700c0 2024-03-04T09:32:17.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:17.280+0000 7fbe8a800640 1 -- 172.21.15.6:0/742409254 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbe8c15f450 msgr2=0x7fbe8c070990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:32:17.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:17.280+0000 7fbe8a800640 1 --2- 172.21.15.6:0/742409254 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbe8c15f450 0x7fbe8c070990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:17.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:17.280+0000 7fbe8a800640 1 -- 172.21.15.6:0/742409254 >> v1:172.21.15.146:6789/0 conn(0x7fbe8c0700c0 legacy=0x7fbe8c0cfda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:17.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:17.280+0000 7fbe8a800640 1 -- 172.21.15.6:0/742409254 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe8c0cd680 con 0x7fbe8c0d04b0 2024-03-04T09:32:17.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:17.280+0000 7fbe89e00640 1 --2- 172.21.15.6:0/742409254 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbe8c15f450 0x7fbe8c070990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T09:32:17.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:17.282+0000 7fbe8a800640 1 --2- 172.21.15.6:0/742409254 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbe8c0d04b0 0x7fbe8c0cde80 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fbe84009d90 tx=0x7fbe8402beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4fa1ade5c0ba0d1e server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:17.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:17.282+0000 7fbe89400640 1 -- 172.21.15.6:0/742409254 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe84039070 con 0x7fbe8c0d04b0 2024-03-04T09:32:17.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:17.282+0000 7fbe89400640 1 -- 172.21.15.6:0/742409254 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe8402cc60 con 0x7fbe8c0d04b0 2024-03-04T09:32:17.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:17.282+0000 7fbe89400640 1 -- 172.21.15.6:0/742409254 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe84034660 con 0x7fbe8c0d04b0 2024-03-04T09:32:17.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:17.283+0000 7fbe8b200640 1 -- 172.21.15.6:0/742409254 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbe8c0d04b0 msgr2=0x7fbe8c0cde80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:17.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:17.283+0000 7fbe8b200640 1 --2- 172.21.15.6:0/742409254 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbe8c0d04b0 0x7fbe8c0cde80 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fbe84009d90 tx=0x7fbe8402beb0 comp rx=0 tx=0).stop 2024-03-04T09:32:17.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:17.283+0000 7fbe8b200640 1 -- 172.21.15.6:0/742409254 shutdown_connections 2024-03-04T09:32:17.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:17.283+0000 7fbe8b200640 1 --2- 172.21.15.6:0/742409254 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbe8c0d04b0 0x7fbe8c0cde80 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:17.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:17.283+0000 7fbe8b200640 1 --2- 172.21.15.6:0/742409254 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbe8c15f450 0x7fbe8c070990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:17.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:17.283+0000 7fbe8b200640 1 -- 172.21.15.6:0/742409254 >> 172.21.15.6:0/742409254 conn(0x7fbe8c0d1ae0 msgr2=0x7fbe8c0d1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:32:17.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:17.283+0000 7fbe8b200640 1 -- 172.21.15.6:0/742409254 shutdown_connections 2024-03-04T09:32:17.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:17.283+0000 7fbe8b200640 1 -- 172.21.15.6:0/742409254 wait complete. 2024-03-04T09:32:17.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:17.285+0000 7fbe8b200640 1 Processor -- start 2024-03-04T09:32:17.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:17.285+0000 7fbe8b200640 1 -- start start 2024-03-04T09:32:17.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:17.286+0000 7fbe8b200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbe8c0700c0 0x7fbe8c154030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:17.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:17.286+0000 7fbe8b200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbe8c0d04b0 0x7fbe8c0c6020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:17.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:17.286+0000 7fbe8b200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbe8c15f450 0x7fbe8c156dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:17.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:17.286+0000 7fbe8b200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe8c15f830 con 0x7fbe8c15f450 2024-03-04T09:32:17.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:17.286+0000 7fbe8b200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe8c0ae8b0 con 0x7fbe8c0d04b0 2024-03-04T09:32:17.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:17.286+0000 7fbe8b200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe8c15f9b0 con 0x7fbe8c0700c0 2024-03-04T09:32:17.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:17.287+0000 7fbe89e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbe8c0700c0 0x7fbe8c154030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:17.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:17.287+0000 7fbe83e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbe8c0d04b0 0x7fbe8c0c6020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:17.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:17.287+0000 7fbe8a800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbe8c15f450 0x7fbe8c156dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:17.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:17.287+0000 7fbe83e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbe8c0d04b0 0x7fbe8c0c6020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.6:33668/0 (socket says 172.21.15.6:33668) 2024-03-04T09:32:17.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:17.287+0000 7fbe83e00640 1 -- 172.21.15.6:0/124626243 learned_addr learned my addr 172.21.15.6:0/124626243 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:32:17.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:17.287+0000 7fbe8a800640 1 -- 172.21.15.6:0/124626243 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbe8c0700c0 msgr2=0x7fbe8c154030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:17.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:17.287+0000 7fbe8a800640 1 --2- 172.21.15.6:0/124626243 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbe8c0700c0 0x7fbe8c154030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:17.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:17.287+0000 7fbe8a800640 1 -- 172.21.15.6:0/124626243 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbe8c0d04b0 msgr2=0x7fbe8c0c6020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:17.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:17.287+0000 7fbe8a800640 1 --2- 172.21.15.6:0/124626243 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbe8c0d04b0 0x7fbe8c0c6020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:17.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:17.287+0000 7fbe8a800640 1 -- 172.21.15.6:0/124626243 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe8c0c6aa0 con 0x7fbe8c15f450 2024-03-04T09:32:17.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:17.288+0000 7fbe8a800640 1 --2- 172.21.15.6:0/124626243 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbe8c15f450 0x7fbe8c156dc0 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7fbe84033000 tx=0x7fbe84004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:17.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:17.288+0000 7fbe82000640 1 -- 172.21.15.6:0/124626243 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe84039050 con 0x7fbe8c15f450 2024-03-04T09:32:17.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:17.288+0000 7fbe82000640 1 -- 172.21.15.6:0/124626243 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe84045070 con 0x7fbe8c15f450 2024-03-04T09:32:17.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:17.288+0000 7fbe82000640 1 -- 172.21.15.6:0/124626243 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe84009450 con 0x7fbe8c15f450 2024-03-04T09:32:17.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:17.288+0000 7fbe8b200640 1 -- 172.21.15.6:0/124626243 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe8c0c6d80 con 0x7fbe8c15f450 2024-03-04T09:32:17.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:17.288+0000 7fbe8b200640 1 -- 172.21.15.6:0/124626243 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe8c154d80 con 0x7fbe8c15f450 2024-03-04T09:32:17.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:17.290+0000 7fbe8b200640 1 -- 172.21.15.6:0/124626243 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe8c0c5200 con 0x7fbe8c15f450 2024-03-04T09:32:17.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:17.291+0000 7fbe82000640 1 -- 172.21.15.6:0/124626243 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbe8403e850 con 0x7fbe8c15f450 2024-03-04T09:32:17.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:17.291+0000 7fbe82000640 1 --2- 172.21.15.6:0/124626243 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fbe6806b4d0 0x7fbe6806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:17.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:17.292+0000 7fbe82000640 1 -- 172.21.15.6:0/124626243 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5853+0+0 (secure 0 0 0) 0x7fbe840acd60 con 0x7fbe8c15f450 2024-03-04T09:32:17.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:17.292+0000 7fbe89e00640 1 --2- 172.21.15.6:0/124626243 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fbe6806b4d0 0x7fbe6806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:17.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:17.293+0000 7fbe89e00640 1 --2- 172.21.15.6:0/124626243 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fbe6806b4d0 0x7fbe6806d990 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fbe7c009630 tx=0x7fbe7c009240 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:17.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:17.301+0000 7fbe82000640 1 -- 172.21.15.6:0/124626243 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbe8c0c5200 con 0x7fbe8c15f450 2024-03-04T09:32:17.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:17.632+0000 7fbe8b200640 1 -- 172.21.15.6:0/124626243 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbe8c04f330 con 0x7fbe8c15f450 2024-03-04T09:32:17.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:17.633+0000 7fbe82000640 1 -- 172.21.15.6:0/124626243 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+4975 (secure 0 0 0) 0x7fbe8407b610 con 0x7fbe8c15f450 2024-03-04T09:32:17.630 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:32:17.630 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":84,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6535,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/3883122012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":3883122012},{"type":"v1","addr":"172.21.15.146:6835","nonce":3883122012}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":6559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/4060411493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":4060411493},{"type":"v1","addr":"172.21.15.146:6837","nonce":4060411493}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":82},{"gid":7812,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/679471398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":679471398},{"type":"v1","addr":"172.21.15.6:6833","nonce":679471398}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:31:21.114351+0000","modified":"2024-03-04T09:31:40.781474+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7815},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7815":{"gid":7815,"name":"a","rank":0,"incarnation":81,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/882667161","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":882667161},{"type":"v1","addr":"172.21.15.6:6835","nonce":882667161}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7815,"qdb_cluster":[7815]},"id":8}]} 2024-03-04T09:32:17.630 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 84 2024-03-04T09:32:17.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:17.635+0000 7fbe8b200640 1 -- 172.21.15.6:0/124626243 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fbe6806b4d0 msgr2=0x7fbe6806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:17.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:17.635+0000 7fbe8b200640 1 --2- 172.21.15.6:0/124626243 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fbe6806b4d0 0x7fbe6806d990 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fbe7c009630 tx=0x7fbe7c009240 comp rx=0 tx=0).stop 2024-03-04T09:32:17.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:17.636+0000 7fbe8b200640 1 -- 172.21.15.6:0/124626243 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbe8c15f450 msgr2=0x7fbe8c156dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:17.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:17.636+0000 7fbe8b200640 1 --2- 172.21.15.6:0/124626243 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbe8c15f450 0x7fbe8c156dc0 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7fbe84033000 tx=0x7fbe84004a40 comp rx=0 tx=0).stop 2024-03-04T09:32:17.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:17.636+0000 7fbe8b200640 1 -- 172.21.15.6:0/124626243 shutdown_connections 2024-03-04T09:32:17.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:17.636+0000 7fbe8b200640 1 --2- 172.21.15.6:0/124626243 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbe8c15f450 0x7fbe8c156dc0 unknown :-1 s=CLOSED pgs=1338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:17.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:17.636+0000 7fbe8b200640 1 --2- 172.21.15.6:0/124626243 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbe8c0d04b0 0x7fbe8c0c6020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:17.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:17.636+0000 7fbe8b200640 1 --2- 172.21.15.6:0/124626243 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fbe6806b4d0 0x7fbe6806d990 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:17.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:17.636+0000 7fbe8b200640 1 --2- 172.21.15.6:0/124626243 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbe8c0700c0 0x7fbe8c154030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:17.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:17.636+0000 7fbe8b200640 1 -- 172.21.15.6:0/124626243 >> 172.21.15.6:0/124626243 conn(0x7fbe8c0d1ae0 msgr2=0x7fbe8c15bd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:32:17.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:17.636+0000 7fbe8b200640 1 -- 172.21.15.6:0/124626243 shutdown_connections 2024-03-04T09:32:17.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:17.636+0000 7fbe8b200640 1 -- 172.21.15.6:0/124626243 wait complete. 2024-03-04T09:32:17.657 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json flush journal 2024-03-04T09:32:17.908 INFO:teuthology.orchestra.run.smithi006.stdout:{"message":"","return_code":0} 2024-03-04T09:32:17.908 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-04T09:32:17.909 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 10000000001.00000000 parent 2024-03-04T09:32:17.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:17.999+0000 7f1cf18f9f40 1 Processor -- start 2024-03-04T09:32:17.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:17.999+0000 7f1cf18f9f40 1 -- start start 2024-03-04T09:32:17.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.000+0000 7f1cf18f9f40 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555c49ed6660 0x555c49f9ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:17.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.000+0000 7f1cf18f9f40 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555c49f04540 0x555c49fa80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:17.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.000+0000 7f1cf18f9f40 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x555c49fa3830 0x555c49fa8d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:17.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.000+0000 7f1cf18f9f40 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x555c49f1ad10 con 0x555c49ed6660 2024-03-04T09:32:17.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.000+0000 7f1cf18f9f40 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x555c49e7afd0 con 0x555c49f04540 2024-03-04T09:32:17.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.000+0000 7f1cf18f9f40 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x555c49f1da90 con 0x555c49fa3830 2024-03-04T09:32:17.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.000+0000 7f1ceea00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555c49ed6660 0x555c49f9ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:17.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.000+0000 7f1ceea00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555c49ed6660 0x555c49f9ed40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:54542/0 (socket says 172.21.15.6:54542) 2024-03-04T09:32:17.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.000+0000 7f1ceea00640 1 -- 172.21.15.6:0/4054480017 learned_addr learned my addr 172.21.15.6:0/4054480017 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:32:17.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.000+0000 7f1ceea00640 1 -- 172.21.15.6:0/4054480017 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x555c49fa3830 msgr2=0x555c49fa8d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:32:17.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.000+0000 7f1ceea00640 1 --2- 172.21.15.6:0/4054480017 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x555c49fa3830 0x555c49fa8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:17.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.000+0000 7f1ceea00640 1 -- 172.21.15.6:0/4054480017 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555c49f04540 msgr2=0x555c49fa80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:32:17.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.000+0000 7f1cee000640 1 --2- 172.21.15.6:0/4054480017 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555c49f04540 0x555c49fa80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:17.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.000+0000 7f1ceea00640 1 --2- 172.21.15.6:0/4054480017 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555c49f04540 0x555c49fa80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:17.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.001+0000 7f1ceea00640 1 -- 172.21.15.6:0/4054480017 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555c49fa2ff0 con 0x555c49ed6660 2024-03-04T09:32:17.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.001+0000 7f1cee000640 1 --2- 172.21.15.6:0/4054480017 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555c49f04540 0x555c49fa80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:32:17.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.001+0000 7f1ceea00640 1 --2- 172.21.15.6:0/4054480017 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555c49ed6660 0x555c49f9ed40 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7f1ce0055040 tx=0x7f1ce004c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f74a5d5c0948e063 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:17.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.001+0000 7f1cecc00640 1 -- 172.21.15.6:0/4054480017 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ce0057040 con 0x555c49ed6660 2024-03-04T09:32:17.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.001+0000 7f1cecc00640 1 -- 172.21.15.6:0/4054480017 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ce00529f0 con 0x555c49ed6660 2024-03-04T09:32:17.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.001+0000 7f1cecc00640 1 -- 172.21.15.6:0/4054480017 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ce0052de0 con 0x555c49ed6660 2024-03-04T09:32:18.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.002+0000 7f1cf18f9f40 1 -- 172.21.15.6:0/4054480017 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555c49ed6660 msgr2=0x555c49f9ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:18.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.002+0000 7f1cf18f9f40 1 --2- 172.21.15.6:0/4054480017 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555c49ed6660 0x555c49f9ed40 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7f1ce0055040 tx=0x7f1ce004c000 comp rx=0 tx=0).stop 2024-03-04T09:32:18.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.002+0000 7f1cf18f9f40 1 -- 172.21.15.6:0/4054480017 shutdown_connections 2024-03-04T09:32:18.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.002+0000 7f1cf18f9f40 1 --2- 172.21.15.6:0/4054480017 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x555c49fa3830 0x555c49fa8d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:18.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.002+0000 7f1cf18f9f40 1 --2- 172.21.15.6:0/4054480017 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555c49f04540 0x555c49fa80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:18.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.002+0000 7f1cf18f9f40 1 --2- 172.21.15.6:0/4054480017 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555c49ed6660 0x555c49f9ed40 unknown :-1 s=CLOSED pgs=1340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:18.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.002+0000 7f1cf18f9f40 1 -- 172.21.15.6:0/4054480017 >> 172.21.15.6:0/4054480017 conn(0x555c49ed45c0 msgr2=0x555c49f039f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:32:18.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.002+0000 7f1cf18f9f40 1 -- 172.21.15.6:0/4054480017 shutdown_connections 2024-03-04T09:32:18.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.002+0000 7f1cf18f9f40 1 -- 172.21.15.6:0/4054480017 wait complete. 2024-03-04T09:32:18.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.003+0000 7f1cf18f9f40 1 Processor -- start 2024-03-04T09:32:18.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.003+0000 7f1cf18f9f40 1 -- start start 2024-03-04T09:32:18.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.004+0000 7f1cf18f9f40 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555c49ed6660 0x555c49f96f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:18.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.004+0000 7f1cf18f9f40 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555c49f04540 0x555c4a044260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:18.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.004+0000 7f1cf18f9f40 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x555c49fa3830 0x555c49f9ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:18.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.004+0000 7f1cf18f9f40 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x555c49fa7ac0 con 0x555c49ed6660 2024-03-04T09:32:18.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.004+0000 7f1cf18f9f40 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x555c49fa05c0 con 0x555c49f04540 2024-03-04T09:32:18.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.004+0000 7f1cf18f9f40 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x555c49fa0320 con 0x555c49fa3830 2024-03-04T09:32:18.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.004+0000 7f1ceea00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555c49ed6660 0x555c49f96f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:18.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.004+0000 7f1ceea00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555c49ed6660 0x555c49f96f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:54546/0 (socket says 172.21.15.6:54546) 2024-03-04T09:32:18.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.004+0000 7f1ceea00640 1 -- 172.21.15.6:0/1098104886 learned_addr learned my addr 172.21.15.6:0/1098104886 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:32:18.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.004+0000 7f1ceea00640 1 -- 172.21.15.6:0/1098104886 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x555c49fa3830 msgr2=0x555c49f9ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:18.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.004+0000 7f1ceea00640 1 --2- 172.21.15.6:0/1098104886 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x555c49fa3830 0x555c49f9ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:18.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.004+0000 7f1ceea00640 1 -- 172.21.15.6:0/1098104886 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555c49f04540 msgr2=0x555c4a044260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:18.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.004+0000 7f1ceea00640 1 --2- 172.21.15.6:0/1098104886 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555c49f04540 0x555c4a044260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:18.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.004+0000 7f1ceea00640 1 -- 172.21.15.6:0/1098104886 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555c49fa6510 con 0x555c49ed6660 2024-03-04T09:32:18.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.005+0000 7f1ceea00640 1 --2- 172.21.15.6:0/1098104886 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555c49ed6660 0x555c49f96f40 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7f1ce001cf30 tx=0x7f1ce0002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:18.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.005+0000 7f1ce7400640 1 -- 172.21.15.6:0/1098104886 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ce0057070 con 0x555c49ed6660 2024-03-04T09:32:18.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.005+0000 7f1ce7400640 1 -- 172.21.15.6:0/1098104886 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ce005d440 con 0x555c49ed6660 2024-03-04T09:32:18.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.005+0000 7f1cf18f9f40 1 -- 172.21.15.6:0/1098104886 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x555c49ed5e40 con 0x555c49ed6660 2024-03-04T09:32:18.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.005+0000 7f1ce7400640 1 -- 172.21.15.6:0/1098104886 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ce005d830 con 0x555c49ed6660 2024-03-04T09:32:18.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.005+0000 7f1cf18f9f40 1 -- 172.21.15.6:0/1098104886 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555c49f97480 con 0x555c49ed6660 2024-03-04T09:32:18.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.007+0000 7f1ce7400640 1 -- 172.21.15.6:0/1098104886 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1ce005d9d0 con 0x555c49ed6660 2024-03-04T09:32:18.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.008+0000 7f1ce7400640 1 --2- 172.21.15.6:0/1098104886 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f1cd0069eb0 0x7f1cd006c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:18.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.008+0000 7f1ce7400640 1 -- 172.21.15.6:0/1098104886 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f1ce00cc130 con 0x555c49ed6660 2024-03-04T09:32:18.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.009+0000 7f1cf18f9f40 1 --2- 172.21.15.6:0/1098104886 >> [v2:172.21.15.6:6823/1364449401,v1:172.21.15.6:6825/1364449401] conn(0x555c4a0447a0 0x555c49f9c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:18.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.009+0000 7f1cf18f9f40 1 -- 172.21.15.6:0/1098104886 --> [v2:172.21.15.6:6823/1364449401,v1:172.21.15.6:6825/1364449401] -- osd_op(unknown.0.0:1 17.7 17:e2e07930:::10000000001.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e115) v9 -- 0x555c49f9c720 con 0x555c4a0447a0 2024-03-04T09:32:18.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.009+0000 7f1cee000640 1 --2- 172.21.15.6:0/1098104886 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f1cd0069eb0 0x7f1cd006c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:18.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.009+0000 7f1cef400640 1 --2- 172.21.15.6:0/1098104886 >> [v2:172.21.15.6:6823/1364449401,v1:172.21.15.6:6825/1364449401] conn(0x555c4a0447a0 0x555c49f9c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:18.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.010+0000 7f1cef400640 1 --2- 172.21.15.6:0/1098104886 >> [v2:172.21.15.6:6823/1364449401,v1:172.21.15.6:6825/1364449401] conn(0x555c4a0447a0 0x555c49f9c1e0 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:18.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.010+0000 7f1cef400640 1 -- 172.21.15.6:0/1098104886 <== osd.0 v2:172.21.15.6:6823/1364449401 1 ==== osd_op_reply(1 10000000001.00000000 [getxattr (99) out=99b] v0'0 uv9 ondisk = 0) v8 ==== 164+0+99 (crc 0 0 0) 0x7f1ce800a3e0 con 0x555c4a0447a0 2024-03-04T09:32:18.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.010+0000 7f1cee000640 1 --2- 172.21.15.6:0/1098104886 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f1cd0069eb0 0x7f1cd006c370 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x555c49fa9b50 tx=0x7f1cdc0055e0 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:18.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.010+0000 7f1cf18f9f40 1 -- 172.21.15.6:0/1098104886 >> [v2:172.21.15.6:6823/1364449401,v1:172.21.15.6:6825/1364449401] conn(0x555c4a0447a0 msgr2=0x555c49f9c1e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:18.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.010+0000 7f1cf18f9f40 1 --2- 172.21.15.6:0/1098104886 >> [v2:172.21.15.6:6823/1364449401,v1:172.21.15.6:6825/1364449401] conn(0x555c4a0447a0 0x555c49f9c1e0 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:18.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.010+0000 7f1cf18f9f40 1 -- 172.21.15.6:0/1098104886 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f1cd0069eb0 msgr2=0x7f1cd006c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:18.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.010+0000 7f1cf18f9f40 1 --2- 172.21.15.6:0/1098104886 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f1cd0069eb0 0x7f1cd006c370 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x555c49fa9b50 tx=0x7f1cdc0055e0 comp rx=0 tx=0).stop 2024-03-04T09:32:18.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.010+0000 7f1cf18f9f40 1 -- 172.21.15.6:0/1098104886 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555c49ed6660 msgr2=0x555c49f96f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:18.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.010+0000 7f1cf18f9f40 1 --2- 172.21.15.6:0/1098104886 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555c49ed6660 0x555c49f96f40 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7f1ce001cf30 tx=0x7f1ce0002fb0 comp rx=0 tx=0).stop 2024-03-04T09:32:18.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.010+0000 7f1cef400640 1 -- 172.21.15.6:0/1098104886 reap_dead start 2024-03-04T09:32:18.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.010+0000 7f1cf18f9f40 1 -- 172.21.15.6:0/1098104886 shutdown_connections 2024-03-04T09:32:18.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.010+0000 7f1cf18f9f40 1 -- 172.21.15.6:0/1098104886 >> 172.21.15.6:0/1098104886 conn(0x555c49ed45c0 msgr2=0x555c49fa7200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:32:18.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.010+0000 7f1cf18f9f40 1 -- 172.21.15.6:0/1098104886 shutdown_connections 2024-03-04T09:32:18.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.011+0000 7f1cf18f9f40 1 -- 172.21.15.6:0/1098104886 wait complete. 2024-03-04T09:32:18.015 INFO:teuthology.orchestra.run.smithi006.stdout:] 10000000001 stray0 2024-03-04T09:32:18.015 DEBUG:teuthology.orchestra.run.smithi006:> ceph-dencoder type inode_backtrace_t import - decode dump_json 2024-03-04T09:32:18.106 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-04T09:32:18.106 INFO:teuthology.orchestra.run.smithi006.stdout: "ino": 1099511627777, 2024-03-04T09:32:18.106 INFO:teuthology.orchestra.run.smithi006.stdout: "ancestors": [ 2024-03-04T09:32:18.107 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:32:18.107 INFO:teuthology.orchestra.run.smithi006.stdout: "dirino": 1536, 2024-03-04T09:32:18.107 INFO:teuthology.orchestra.run.smithi006.stdout: "dname": "10000000001", 2024-03-04T09:32:18.107 INFO:teuthology.orchestra.run.smithi006.stdout: "version": 13 2024-03-04T09:32:18.107 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:32:18.107 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:32:18.107 INFO:teuthology.orchestra.run.smithi006.stdout: "dirino": 256, 2024-03-04T09:32:18.107 INFO:teuthology.orchestra.run.smithi006.stdout: "dname": "stray0", 2024-03-04T09:32:18.107 INFO:teuthology.orchestra.run.smithi006.stdout: "version": 5 2024-03-04T09:32:18.107 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:32:18.107 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-03-04T09:32:18.107 INFO:teuthology.orchestra.run.smithi006.stdout: "pool": 17, 2024-03-04T09:32:18.107 INFO:teuthology.orchestra.run.smithi006.stdout: "old_pools": [] 2024-03-04T09:32:18.107 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-04T09:32:18.107 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:32:18.113 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:32:18.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.374+0000 7f9a93200640 1 Processor -- start 2024-03-04T09:32:18.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.374+0000 7f9a93200640 1 -- start start 2024-03-04T09:32:18.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.376+0000 7f9a93200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9a940d0cd0 0x7f9a940b3630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:18.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.376+0000 7f9a93200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a940d2dc0 con 0x7f9a940d2960 2024-03-04T09:32:18.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.376+0000 7f9a93200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a940d18c0 con 0x7f9a9415e070 2024-03-04T09:32:18.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.376+0000 7f9a93200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a940d1ba0 con 0x7f9a940d0cd0 2024-03-04T09:32:18.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.376+0000 7f9a8be00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f9a940d2960 0x7f9a940d05c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:33520/0 (socket says 172.21.15.6:33520) 2024-03-04T09:32:18.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.376+0000 7f9a8be00640 1 -- 172.21.15.6:0/2738230547 learned_addr learned my addr 172.21.15.6:0/2738230547 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:32:18.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.377+0000 7f9a92800640 1 --2- 172.21.15.6:0/2738230547 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9a940d0cd0 0x7f9a940b3630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:18.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.377+0000 7f9a91400640 1 -- 172.21.15.6:0/2738230547 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3433826183 0 0) 0x7f9a940d2dc0 con 0x7f9a940d2960 2024-03-04T09:32:18.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.377+0000 7f9a91400640 1 -- 172.21.15.6:0/2738230547 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a78003520 con 0x7f9a940d2960 2024-03-04T09:32:18.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.377+0000 7f9a91400640 1 -- 172.21.15.6:0/2738230547 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 465786276 0 0) 0x7f9a78003520 con 0x7f9a940d2960 2024-03-04T09:32:18.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.377+0000 7f9a91400640 1 -- 172.21.15.6:0/2738230547 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9a940d0cd0 msgr2=0x7f9a940b3630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:32:18.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.377+0000 7f9a91400640 1 --2- 172.21.15.6:0/2738230547 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9a940d0cd0 0x7f9a940b3630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:18.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.377+0000 7f9a91400640 1 -- 172.21.15.6:0/2738230547 >> v1:172.21.15.146:6789/0 conn(0x7f9a9415e070 legacy=0x7f9a940cf410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:18.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.377+0000 7f9a91400640 1 -- 172.21.15.6:0/2738230547 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a940d1e80 con 0x7f9a940d2960 2024-03-04T09:32:18.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.378+0000 7f9a91400640 1 -- 172.21.15.6:0/2738230547 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7f9a7c001670 con 0x7f9a940d2960 2024-03-04T09:32:18.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.378+0000 7f9a91400640 1 -- 172.21.15.6:0/2738230547 >> v1:172.21.15.6:6789/0 conn(0x7f9a940d2960 legacy=0x7f9a940d05c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:18.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.378+0000 7f9a91400640 1 --2- 172.21.15.6:0/2738230547 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9a780039e0 0x7f9a78003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:18.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.378+0000 7f9a91400640 1 --2- 172.21.15.6:0/2738230547 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9a78004800 0x7f9a78004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:18.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.378+0000 7f9a91400640 1 --2- 172.21.15.6:0/2738230547 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9a940d0cd0 0x7f9a780051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:18.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.378+0000 7f9a91400640 1 -- 172.21.15.6:0/2738230547 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a940d1ba0 con 0x7f9a78004800 2024-03-04T09:32:18.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.378+0000 7f9a91400640 1 -- 172.21.15.6:0/2738230547 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a7800bad0 con 0x7f9a780039e0 2024-03-04T09:32:18.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.378+0000 7f9a91400640 1 -- 172.21.15.6:0/2738230547 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a7800bcd0 con 0x7f9a940d0cd0 2024-03-04T09:32:18.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.378+0000 7f9a8be00640 1 --2- 172.21.15.6:0/2738230547 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9a78004800 0x7f9a78004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:18.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.378+0000 7f9a92800640 1 --2- 172.21.15.6:0/2738230547 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9a940d0cd0 0x7f9a780051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:18.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.378+0000 7f9a91e00640 1 --2- 172.21.15.6:0/2738230547 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9a780039e0 0x7f9a78003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:18.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.378+0000 7f9a8be00640 1 -- 172.21.15.6:0/2738230547 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9a940d0cd0 msgr2=0x7f9a780051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:32:18.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.378+0000 7f9a8be00640 1 --2- 172.21.15.6:0/2738230547 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9a940d0cd0 0x7f9a780051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:18.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.378+0000 7f9a8be00640 1 -- 172.21.15.6:0/2738230547 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9a780039e0 msgr2=0x7f9a78003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:32:18.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.378+0000 7f9a8be00640 1 --2- 172.21.15.6:0/2738230547 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9a780039e0 0x7f9a78003e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:18.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.378+0000 7f9a8be00640 1 -- 172.21.15.6:0/2738230547 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a78003520 con 0x7f9a78004800 2024-03-04T09:32:18.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.380+0000 7f9a8be00640 1 --2- 172.21.15.6:0/2738230547 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9a78004800 0x7f9a78004c80 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7f9a7c007fd0 tx=0x7f9a78005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e4c11d70f1f840f server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:18.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.380+0000 7f9a91400640 1 -- 172.21.15.6:0/2738230547 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a7c007c20 con 0x7f9a78004800 2024-03-04T09:32:18.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.380+0000 7f9a91400640 1 -- 172.21.15.6:0/2738230547 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a7c030070 con 0x7f9a78004800 2024-03-04T09:32:18.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.380+0000 7f9a91400640 1 -- 172.21.15.6:0/2738230547 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a7c005d30 con 0x7f9a78004800 2024-03-04T09:32:18.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.380+0000 7f9a93200640 1 -- 172.21.15.6:0/2738230547 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9a78004800 msgr2=0x7f9a78004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:18.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.380+0000 7f9a93200640 1 --2- 172.21.15.6:0/2738230547 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9a78004800 0x7f9a78004c80 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7f9a7c007fd0 tx=0x7f9a78005830 comp rx=0 tx=0).stop 2024-03-04T09:32:18.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.380+0000 7f9a92800640 1 -- 172.21.15.6:0/2738230547 reap_dead start 2024-03-04T09:32:18.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.380+0000 7f9a93200640 1 -- 172.21.15.6:0/2738230547 shutdown_connections 2024-03-04T09:32:18.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.380+0000 7f9a93200640 1 -- 172.21.15.6:0/2738230547 >> 172.21.15.6:0/2738230547 conn(0x7f9a9408d2a0 msgr2=0x7f9a940d3cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:32:18.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.381+0000 7f9a93200640 1 -- 172.21.15.6:0/2738230547 shutdown_connections 2024-03-04T09:32:18.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.381+0000 7f9a93200640 1 -- 172.21.15.6:0/2738230547 wait complete. 2024-03-04T09:32:18.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.382+0000 7f9a93200640 1 Processor -- start 2024-03-04T09:32:18.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.383+0000 7f9a93200640 1 -- start start 2024-03-04T09:32:18.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.383+0000 7f9a93200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9a78004800 0x7f9a94158360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:18.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.383+0000 7f9a93200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9a9415e450 0x7f9a9415fa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:18.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.383+0000 7f9a93200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9a940b1480 0x7f9a940b1900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:18.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.384+0000 7f9a93200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a940d45b0 con 0x7f9a940b1480 2024-03-04T09:32:18.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.384+0000 7f9a93200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a940d2440 con 0x7f9a9415e450 2024-03-04T09:32:18.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.384+0000 7f9a93200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a940d4730 con 0x7f9a78004800 2024-03-04T09:32:18.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.384+0000 7f9a92800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9a940b1480 0x7f9a940b1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:18.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.384+0000 7f9a92800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9a940b1480 0x7f9a940b1900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:54560/0 (socket says 172.21.15.6:54560) 2024-03-04T09:32:18.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.384+0000 7f9a92800640 1 -- 172.21.15.6:0/2574846716 learned_addr learned my addr 172.21.15.6:0/2574846716 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:32:18.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.384+0000 7f9a91e00640 1 --2- 172.21.15.6:0/2574846716 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9a78004800 0x7f9a94158360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:18.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.384+0000 7f9a8be00640 1 --2- 172.21.15.6:0/2574846716 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9a9415e450 0x7f9a9415fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:18.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.384+0000 7f9a92800640 1 -- 172.21.15.6:0/2574846716 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9a78004800 msgr2=0x7f9a94158360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:18.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.384+0000 7f9a92800640 1 --2- 172.21.15.6:0/2574846716 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9a78004800 0x7f9a94158360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:18.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.384+0000 7f9a92800640 1 -- 172.21.15.6:0/2574846716 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9a9415e450 msgr2=0x7f9a9415fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:18.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.384+0000 7f9a92800640 1 --2- 172.21.15.6:0/2574846716 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9a9415e450 0x7f9a9415fa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:18.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.384+0000 7f9a92800640 1 -- 172.21.15.6:0/2574846716 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a9415ff60 con 0x7f9a940b1480 2024-03-04T09:32:18.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.384+0000 7f9a8be00640 1 --2- 172.21.15.6:0/2574846716 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9a9415e450 0x7f9a9415fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:32:18.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.385+0000 7f9a92800640 1 --2- 172.21.15.6:0/2574846716 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9a940b1480 0x7f9a940b1900 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7f9a940d3c50 tx=0x7f9a8c00f4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:18.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.385+0000 7f9a8a000640 1 -- 172.21.15.6:0/2574846716 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a8c01c070 con 0x7f9a940b1480 2024-03-04T09:32:18.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.385+0000 7f9a8a000640 1 -- 172.21.15.6:0/2574846716 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a8c017400 con 0x7f9a940b1480 2024-03-04T09:32:18.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.385+0000 7f9a8a000640 1 -- 172.21.15.6:0/2574846716 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a8c0177f0 con 0x7f9a940b1480 2024-03-04T09:32:18.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.385+0000 7f9a93200640 1 -- 172.21.15.6:0/2574846716 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a941612a0 con 0x7f9a940b1480 2024-03-04T09:32:18.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.385+0000 7f9a93200640 1 -- 172.21.15.6:0/2574846716 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a941604e0 con 0x7f9a940b1480 2024-03-04T09:32:18.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.387+0000 7f9a93200640 1 -- 172.21.15.6:0/2574846716 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a9404f330 con 0x7f9a940b1480 2024-03-04T09:32:18.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.387+0000 7f9a8a000640 1 -- 172.21.15.6:0/2574846716 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9a8c005d10 con 0x7f9a940b1480 2024-03-04T09:32:18.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.388+0000 7f9a8a000640 1 --2- 172.21.15.6:0/2574846716 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f9a7006b500 0x7f9a7006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:18.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.388+0000 7f9a8a000640 1 -- 172.21.15.6:0/2574846716 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f9a8c08a770 con 0x7f9a940b1480 2024-03-04T09:32:18.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.389+0000 7f9a91e00640 1 --2- 172.21.15.6:0/2574846716 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f9a7006b500 0x7f9a7006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:18.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.390+0000 7f9a91e00640 1 --2- 172.21.15.6:0/2574846716 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f9a7006b500 0x7f9a7006d9c0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f9a84001550 tx=0x7f9a84009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:18.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.392+0000 7f9a8a000640 1 -- 172.21.15.6:0/2574846716 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9a8c012070 con 0x7f9a940b1480 2024-03-04T09:32:18.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.633+0000 7f9a93200640 1 -- 172.21.15.6:0/2574846716 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9a940d0ab0 con 0x7f9a940b1480 2024-03-04T09:32:18.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.634+0000 7f9a8a000640 1 -- 172.21.15.6:0/2574846716 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+4975 (secure 0 0 0) 0x7f9a8c0594c0 con 0x7f9a940b1480 2024-03-04T09:32:18.631 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:32:18.631 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":84,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6535,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/3883122012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":3883122012},{"type":"v1","addr":"172.21.15.146:6835","nonce":3883122012}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":6559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/4060411493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":4060411493},{"type":"v1","addr":"172.21.15.146:6837","nonce":4060411493}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":82},{"gid":7812,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/679471398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":679471398},{"type":"v1","addr":"172.21.15.6:6833","nonce":679471398}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:31:21.114351+0000","modified":"2024-03-04T09:31:40.781474+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7815},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7815":{"gid":7815,"name":"a","rank":0,"incarnation":81,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/882667161","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":882667161},{"type":"v1","addr":"172.21.15.6:6835","nonce":882667161}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7815,"qdb_cluster":[7815]},"id":8}]} 2024-03-04T09:32:18.632 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 84 2024-03-04T09:32:18.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.637+0000 7f9a93200640 1 -- 172.21.15.6:0/2574846716 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f9a7006b500 msgr2=0x7f9a7006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:18.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.637+0000 7f9a93200640 1 --2- 172.21.15.6:0/2574846716 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f9a7006b500 0x7f9a7006d9c0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f9a84001550 tx=0x7f9a84009040 comp rx=0 tx=0).stop 2024-03-04T09:32:18.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.637+0000 7f9a93200640 1 -- 172.21.15.6:0/2574846716 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9a940b1480 msgr2=0x7f9a940b1900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:18.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.637+0000 7f9a93200640 1 --2- 172.21.15.6:0/2574846716 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9a940b1480 0x7f9a940b1900 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7f9a940d3c50 tx=0x7f9a8c00f4a0 comp rx=0 tx=0).stop 2024-03-04T09:32:18.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.637+0000 7f9a93200640 1 -- 172.21.15.6:0/2574846716 shutdown_connections 2024-03-04T09:32:18.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.637+0000 7f9a93200640 1 --2- 172.21.15.6:0/2574846716 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9a940b1480 0x7f9a940b1900 unknown :-1 s=CLOSED pgs=1344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:18.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.637+0000 7f9a93200640 1 --2- 172.21.15.6:0/2574846716 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9a9415e450 0x7f9a9415fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:18.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.637+0000 7f9a93200640 1 --2- 172.21.15.6:0/2574846716 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f9a7006b500 0x7f9a7006d9c0 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:18.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.637+0000 7f9a93200640 1 --2- 172.21.15.6:0/2574846716 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9a78004800 0x7f9a94158360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:18.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.637+0000 7f9a93200640 1 -- 172.21.15.6:0/2574846716 >> 172.21.15.6:0/2574846716 conn(0x7f9a9408d2a0 msgr2=0x7f9a940b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:32:18.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.637+0000 7f9a93200640 1 -- 172.21.15.6:0/2574846716 shutdown_connections 2024-03-04T09:32:18.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:18.637+0000 7f9a93200640 1 -- 172.21.15.6:0/2574846716 wait complete. 2024-03-04T09:32:18.653 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_reintegrated 2024-03-04T09:32:18.883 INFO:teuthology.orchestra.run.smithi006.stdout:{"mds_cache":{"strays_reintegrated":0}} 2024-03-04T09:32:18.884 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_reintegrated": 0 } } 2024-03-04T09:32:18.884 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:32:18.884 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest/mnt.0 && exec mv dir_2/file_b dir_2/file_c) 2024-03-04T09:32:18.955 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:32:19.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.172+0000 7f22e5800640 1 Processor -- start 2024-03-04T09:32:19.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.172+0000 7f22e5800640 1 -- start start 2024-03-04T09:32:19.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.174+0000 7f22e5800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f22e015e0b0 0x7f22e00cf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:19.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.174+0000 7f22e5800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f22e00d4a10 0x7f22e00d0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:19.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.174+0000 7f22e5800640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22e00b3710 con 0x7f22e00d0c80 2024-03-04T09:32:19.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.174+0000 7f22e5800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22e00d10f0 con 0x7f22e00d4a10 2024-03-04T09:32:19.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.174+0000 7f22e5800640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22e00b09e0 con 0x7f22e015e0b0 2024-03-04T09:32:19.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.174+0000 7f22df400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f22e00d4a10 0x7f22e00d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:19.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.174+0000 7f22e4e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f22e00d0c80 0x7f22e00d1900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:56232/0 (socket says 172.21.15.6:56232) 2024-03-04T09:32:19.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.174+0000 7f22e4e00640 1 -- 172.21.15.6:0/3763189330 learned_addr learned my addr 172.21.15.6:0/3763189330 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:32:19.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.175+0000 7f22dfe00640 1 --2- 172.21.15.6:0/3763189330 >> v2:172.21.15.146:3300/0 conn(0x7f22e015e0b0 0x7f22e00cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:19.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.175+0000 7f22df400640 1 -- 172.21.15.6:0/3763189330 >> v2:172.21.15.146:3300/0 conn(0x7f22e015e0b0 msgr2=0x7f22e00cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:32:19.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.175+0000 7f22df400640 1 --2- 172.21.15.6:0/3763189330 >> v2:172.21.15.146:3300/0 conn(0x7f22e015e0b0 0x7f22e00cf310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:19.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.175+0000 7f22df400640 1 -- 172.21.15.6:0/3763189330 >> v1:172.21.15.6:6789/0 conn(0x7f22e00d0c80 legacy=0x7f22e00d1900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:19.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.175+0000 7f22df400640 1 -- 172.21.15.6:0/3763189330 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22e00b39f0 con 0x7f22e00d4a10 2024-03-04T09:32:19.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.176+0000 7f22df400640 1 --2- 172.21.15.6:0/3763189330 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f22e00d4a10 0x7f22e00d0740 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7f22e00b68c0 tx=0x7f22c802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b44b9480cadc46f server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:19.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.176+0000 7f22dea00640 1 -- 172.21.15.6:0/3763189330 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22c8035020 con 0x7f22e00d4a10 2024-03-04T09:32:19.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.176+0000 7f22dea00640 1 -- 172.21.15.6:0/3763189330 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22c8031830 con 0x7f22e00d4a10 2024-03-04T09:32:19.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.176+0000 7f22dea00640 1 -- 172.21.15.6:0/3763189330 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22c8031c00 con 0x7f22e00d4a10 2024-03-04T09:32:19.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.177+0000 7f22e5800640 1 -- 172.21.15.6:0/3763189330 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f22e00d4a10 msgr2=0x7f22e00d0740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:19.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.177+0000 7f22e5800640 1 --2- 172.21.15.6:0/3763189330 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f22e00d4a10 0x7f22e00d0740 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7f22e00b68c0 tx=0x7f22c802b8d0 comp rx=0 tx=0).stop 2024-03-04T09:32:19.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.177+0000 7f22e5800640 1 -- 172.21.15.6:0/3763189330 shutdown_connections 2024-03-04T09:32:19.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.177+0000 7f22e5800640 1 --2- 172.21.15.6:0/3763189330 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f22e00d4a10 0x7f22e00d0740 unknown :-1 s=CLOSED pgs=1346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:19.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.177+0000 7f22e5800640 1 --2- 172.21.15.6:0/3763189330 >> v2:172.21.15.146:3300/0 conn(0x7f22e015e0b0 0x7f22e00cf310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:19.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.177+0000 7f22e5800640 1 -- 172.21.15.6:0/3763189330 >> 172.21.15.6:0/3763189330 conn(0x7f22e00d3d30 msgr2=0x7f22e00d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:32:19.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.177+0000 7f22e5800640 1 -- 172.21.15.6:0/3763189330 shutdown_connections 2024-03-04T09:32:19.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.177+0000 7f22e5800640 1 -- 172.21.15.6:0/3763189330 wait complete. 2024-03-04T09:32:19.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.179+0000 7f22e5800640 1 Processor -- start 2024-03-04T09:32:19.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.179+0000 7f22e5800640 1 -- start start 2024-03-04T09:32:19.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.180+0000 7f22e5800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f22e00d0c80 0x7f22e014f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:19.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.180+0000 7f22e5800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f22e00d4a10 0x7f22e00cc1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:19.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.180+0000 7f22e5800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f22e015e0b0 0x7f22e00b8af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:19.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.180+0000 7f22e5800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22e015e490 con 0x7f22e00d4a10 2024-03-04T09:32:19.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.180+0000 7f22e5800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22e00b3fb0 con 0x7f22e00d0c80 2024-03-04T09:32:19.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.180+0000 7f22e5800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22e00cfc60 con 0x7f22e015e0b0 2024-03-04T09:32:19.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.180+0000 7f22df400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f22e00d4a10 0x7f22e00cc1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:19.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.180+0000 7f22df400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f22e00d4a10 0x7f22e00cc1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43946/0 (socket says 172.21.15.6:43946) 2024-03-04T09:32:19.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.180+0000 7f22df400640 1 -- 172.21.15.6:0/1951515577 learned_addr learned my addr 172.21.15.6:0/1951515577 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:32:19.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.181+0000 7f22dfe00640 1 --2- 172.21.15.6:0/1951515577 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f22e00d0c80 0x7f22e014f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:19.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.181+0000 7f22df400640 1 -- 172.21.15.6:0/1951515577 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f22e015e0b0 msgr2=0x7f22e00b8af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:19.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.181+0000 7f22df400640 1 --2- 172.21.15.6:0/1951515577 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f22e015e0b0 0x7f22e00b8af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:19.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.181+0000 7f22df400640 1 -- 172.21.15.6:0/1951515577 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f22e00d0c80 msgr2=0x7f22e014f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:19.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.181+0000 7f22df400640 1 --2- 172.21.15.6:0/1951515577 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f22e00d0c80 0x7f22e014f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:19.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.181+0000 7f22df400640 1 -- 172.21.15.6:0/1951515577 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22e00cb150 con 0x7f22e00d4a10 2024-03-04T09:32:19.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.181+0000 7f22dfe00640 1 --2- 172.21.15.6:0/1951515577 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f22e00d0c80 0x7f22e014f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:32:19.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.181+0000 7f22df400640 1 --2- 172.21.15.6:0/1951515577 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f22e00d4a10 0x7f22e00cc1f0 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7f22c802bee0 tx=0x7f22c80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:19.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.182+0000 7f22d7e00640 1 -- 172.21.15.6:0/1951515577 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22c8041070 con 0x7f22e00d4a10 2024-03-04T09:32:19.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.182+0000 7f22d7e00640 1 -- 172.21.15.6:0/1951515577 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22c803b920 con 0x7f22e00d4a10 2024-03-04T09:32:19.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.182+0000 7f22e5800640 1 -- 172.21.15.6:0/1951515577 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22e00cb430 con 0x7f22e00d4a10 2024-03-04T09:32:19.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.182+0000 7f22d7e00640 1 -- 172.21.15.6:0/1951515577 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22c8035040 con 0x7f22e00d4a10 2024-03-04T09:32:19.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.182+0000 7f22e5800640 1 -- 172.21.15.6:0/1951515577 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22e00cb9f0 con 0x7f22e00d4a10 2024-03-04T09:32:19.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.183+0000 7f22e5800640 1 -- 172.21.15.6:0/1951515577 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22e004f330 con 0x7f22e00d4a10 2024-03-04T09:32:19.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.184+0000 7f22d7e00640 1 -- 172.21.15.6:0/1951515577 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f22c804c060 con 0x7f22e00d4a10 2024-03-04T09:32:19.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.185+0000 7f22d7e00640 1 --2- 172.21.15.6:0/1951515577 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f22bc06b4d0 0x7f22bc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:19.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.185+0000 7f22d7e00640 1 -- 172.21.15.6:0/1951515577 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f22c80abc60 con 0x7f22e00d4a10 2024-03-04T09:32:19.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.185+0000 7f22dfe00640 1 --2- 172.21.15.6:0/1951515577 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f22bc06b4d0 0x7f22bc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:19.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.187+0000 7f22dfe00640 1 --2- 172.21.15.6:0/1951515577 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f22bc06b4d0 0x7f22bc06d990 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f22d0009f50 tx=0x7f22d000b040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:19.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.189+0000 7f22d7e00640 1 -- 172.21.15.6:0/1951515577 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f22c807a9c0 con 0x7f22e00d4a10 2024-03-04T09:32:19.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.430+0000 7f22e5800640 1 -- 172.21.15.6:0/1951515577 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f22e00b49e0 con 0x7f22e00d4a10 2024-03-04T09:32:19.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.431+0000 7f22d7e00640 1 -- 172.21.15.6:0/1951515577 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+4975 (secure 0 0 0) 0x7f22c807a2c0 con 0x7f22e00d4a10 2024-03-04T09:32:19.428 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:32:19.428 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":84,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6535,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/3883122012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":3883122012},{"type":"v1","addr":"172.21.15.146:6835","nonce":3883122012}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":6559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/4060411493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":4060411493},{"type":"v1","addr":"172.21.15.146:6837","nonce":4060411493}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":82},{"gid":7812,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/679471398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":679471398},{"type":"v1","addr":"172.21.15.6:6833","nonce":679471398}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:31:21.114351+0000","modified":"2024-03-04T09:31:40.781474+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7815},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7815":{"gid":7815,"name":"a","rank":0,"incarnation":81,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/882667161","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":882667161},{"type":"v1","addr":"172.21.15.6:6835","nonce":882667161}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7815,"qdb_cluster":[7815]},"id":8}]} 2024-03-04T09:32:19.429 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 84 2024-03-04T09:32:19.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.435+0000 7f22e5800640 1 -- 172.21.15.6:0/1951515577 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f22bc06b4d0 msgr2=0x7f22bc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:19.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.435+0000 7f22e5800640 1 --2- 172.21.15.6:0/1951515577 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f22bc06b4d0 0x7f22bc06d990 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f22d0009f50 tx=0x7f22d000b040 comp rx=0 tx=0).stop 2024-03-04T09:32:19.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.435+0000 7f22e5800640 1 -- 172.21.15.6:0/1951515577 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f22e00d4a10 msgr2=0x7f22e00cc1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:19.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.435+0000 7f22e5800640 1 --2- 172.21.15.6:0/1951515577 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f22e00d4a10 0x7f22e00cc1f0 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7f22c802bee0 tx=0x7f22c80046c0 comp rx=0 tx=0).stop 2024-03-04T09:32:19.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.435+0000 7f22e5800640 1 -- 172.21.15.6:0/1951515577 shutdown_connections 2024-03-04T09:32:19.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.435+0000 7f22e5800640 1 --2- 172.21.15.6:0/1951515577 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f22bc06b4d0 0x7f22bc06d990 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:19.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.435+0000 7f22e5800640 1 --2- 172.21.15.6:0/1951515577 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f22e015e0b0 0x7f22e00b8af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:19.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.435+0000 7f22e5800640 1 --2- 172.21.15.6:0/1951515577 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f22e00d4a10 0x7f22e00cc1f0 unknown :-1 s=CLOSED pgs=1347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:19.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.435+0000 7f22e5800640 1 --2- 172.21.15.6:0/1951515577 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f22e00d0c80 0x7f22e014f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:19.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.435+0000 7f22e5800640 1 -- 172.21.15.6:0/1951515577 >> 172.21.15.6:0/1951515577 conn(0x7f22e00d3d30 msgr2=0x7f22e0073f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:32:19.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.435+0000 7f22e5800640 1 -- 172.21.15.6:0/1951515577 shutdown_connections 2024-03-04T09:32:19.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.435+0000 7f22e5800640 1 -- 172.21.15.6:0/1951515577 wait complete. 2024-03-04T09:32:19.451 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache num_strays 2024-03-04T09:32:19.680 INFO:teuthology.orchestra.run.smithi006.stdout:{"mds_cache":{"num_strays":0}} 2024-03-04T09:32:19.681 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "num_strays": 0 } } 2024-03-04T09:32:19.681 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:32:19.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.905+0000 7f88b9200640 1 Processor -- start 2024-03-04T09:32:19.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.906+0000 7f88b9200640 1 -- start start 2024-03-04T09:32:19.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.908+0000 7f88b9200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f88b415e030 0x7f88b4161d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:19.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.908+0000 7f88b9200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f88b415e410 0x7f88b40d2ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:19.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.908+0000 7f88b9200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88b40cea50 con 0x7f88b40cfd50 2024-03-04T09:32:19.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.908+0000 7f88b9200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88b40cecf0 con 0x7f88b415e410 2024-03-04T09:32:19.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.908+0000 7f88b9200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88b40cef90 con 0x7f88b415e030 2024-03-04T09:32:19.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.908+0000 7f88b2800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f88b415e410 0x7f88b40d2ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:19.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.908+0000 7f88b2800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f88b415e410 0x7f88b40d2ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43954/0 (socket says 172.21.15.6:43954) 2024-03-04T09:32:19.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.908+0000 7f88b2800640 1 -- 172.21.15.6:0/2315134861 learned_addr learned my addr 172.21.15.6:0/2315134861 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:32:19.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.909+0000 7f88b1e00640 1 --2- 172.21.15.6:0/2315134861 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f88b415e030 0x7f88b4161d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:19.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.909+0000 7f88b2800640 1 -- 172.21.15.6:0/2315134861 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f88b415e030 msgr2=0x7f88b4161d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:32:19.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.909+0000 7f88b2800640 1 --2- 172.21.15.6:0/2315134861 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f88b415e030 0x7f88b4161d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:19.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.909+0000 7f88b2800640 1 -- 172.21.15.6:0/2315134861 >> v1:172.21.15.146:6789/0 conn(0x7f88b40cfd50 legacy=0x7f88b40d2490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:32:19.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.909+0000 7f88b2800640 1 -- 172.21.15.6:0/2315134861 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88b40cf270 con 0x7f88b415e410 2024-03-04T09:32:19.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.910+0000 7f88b2800640 1 --2- 172.21.15.6:0/2315134861 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f88b415e410 0x7f88b40d2ba0 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7f88ac009e50 tx=0x7f88ac02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8295f7da7d682a4 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:19.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.910+0000 7f88b1400640 1 -- 172.21.15.6:0/2315134861 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88ac02cb00 con 0x7f88b415e410 2024-03-04T09:32:19.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.910+0000 7f88b1400640 1 -- 172.21.15.6:0/2315134861 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88ac02cca0 con 0x7f88b415e410 2024-03-04T09:32:19.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.910+0000 7f88b1400640 1 -- 172.21.15.6:0/2315134861 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88ac034610 con 0x7f88b415e410 2024-03-04T09:32:19.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.911+0000 7f88b9200640 1 -- 172.21.15.6:0/2315134861 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f88b415e410 msgr2=0x7f88b40d2ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:19.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.911+0000 7f88b9200640 1 --2- 172.21.15.6:0/2315134861 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f88b415e410 0x7f88b40d2ba0 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7f88ac009e50 tx=0x7f88ac02bf60 comp rx=0 tx=0).stop 2024-03-04T09:32:19.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.911+0000 7f88b9200640 1 -- 172.21.15.6:0/2315134861 shutdown_connections 2024-03-04T09:32:19.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.911+0000 7f88b9200640 1 --2- 172.21.15.6:0/2315134861 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f88b415e410 0x7f88b40d2ba0 unknown :-1 s=CLOSED pgs=1348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:19.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.911+0000 7f88b9200640 1 --2- 172.21.15.6:0/2315134861 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f88b415e030 0x7f88b4161d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:19.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.911+0000 7f88b9200640 1 -- 172.21.15.6:0/2315134861 >> 172.21.15.6:0/2315134861 conn(0x7f88b40d3eb0 msgr2=0x7f88b40d4290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:32:19.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.911+0000 7f88b9200640 1 -- 172.21.15.6:0/2315134861 shutdown_connections 2024-03-04T09:32:19.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.911+0000 7f88b9200640 1 -- 172.21.15.6:0/2315134861 wait complete. 2024-03-04T09:32:19.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.913+0000 7f88b9200640 1 Processor -- start 2024-03-04T09:32:19.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.913+0000 7f88b9200640 1 -- start start 2024-03-04T09:32:19.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.914+0000 7f88b9200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f88b415e030 0x7f88b40c7810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:19.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.914+0000 7f88b9200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f88b415e410 0x7f88b40d03b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:19.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.914+0000 7f88b9200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f88b4159000 0x7f88b41594b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:19.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.914+0000 7f88b9200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88b4162260 con 0x7f88b415e030 2024-03-04T09:32:19.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.914+0000 7f88b9200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88b40cf830 con 0x7f88b415e410 2024-03-04T09:32:19.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.914+0000 7f88b9200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88b41623e0 con 0x7f88b4159000 2024-03-04T09:32:19.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.914+0000 7f88b1e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f88b415e030 0x7f88b40c7810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:19.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.914+0000 7f88b1e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f88b415e030 0x7f88b40c7810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43962/0 (socket says 172.21.15.6:43962) 2024-03-04T09:32:19.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.914+0000 7f88b1e00640 1 -- 172.21.15.6:0/652926296 learned_addr learned my addr 172.21.15.6:0/652926296 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:32:19.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.915+0000 7f88b2800640 1 --2- 172.21.15.6:0/652926296 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f88b4159000 0x7f88b41594b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:19.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.915+0000 7f88a9400640 1 --2- 172.21.15.6:0/652926296 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f88b415e410 0x7f88b40d03b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:19.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.915+0000 7f88b1e00640 1 -- 172.21.15.6:0/652926296 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f88b4159000 msgr2=0x7f88b41594b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:19.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.915+0000 7f88b1e00640 1 --2- 172.21.15.6:0/652926296 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f88b4159000 0x7f88b41594b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:19.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.915+0000 7f88b1e00640 1 -- 172.21.15.6:0/652926296 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f88b415e410 msgr2=0x7f88b40d03b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:19.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.915+0000 7f88b1e00640 1 --2- 172.21.15.6:0/652926296 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f88b415e410 0x7f88b40d03b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:19.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.915+0000 7f88b1e00640 1 -- 172.21.15.6:0/652926296 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88b40cbd60 con 0x7f88b415e030 2024-03-04T09:32:19.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.915+0000 7f88b1e00640 1 --2- 172.21.15.6:0/652926296 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f88b415e030 0x7f88b40c7810 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7f88a400d800 tx=0x7f88a400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:19.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.915+0000 7f88aaa00640 1 -- 172.21.15.6:0/652926296 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88a400b6c0 con 0x7f88b415e030 2024-03-04T09:32:19.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.916+0000 7f88aaa00640 1 -- 172.21.15.6:0/652926296 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88a4019070 con 0x7f88b415e030 2024-03-04T09:32:19.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.916+0000 7f88aaa00640 1 -- 172.21.15.6:0/652926296 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88a400ba80 con 0x7f88b415e030 2024-03-04T09:32:19.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.916+0000 7f88b9200640 1 -- 172.21.15.6:0/652926296 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88b40cc370 con 0x7f88b415e030 2024-03-04T09:32:19.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.916+0000 7f88b9200640 1 -- 172.21.15.6:0/652926296 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88b40d08f0 con 0x7f88b415e030 2024-03-04T09:32:19.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.917+0000 7f88b9200640 1 -- 172.21.15.6:0/652926296 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88b404f3a0 con 0x7f88b415e030 2024-03-04T09:32:19.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.918+0000 7f88aaa00640 1 -- 172.21.15.6:0/652926296 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f88a400bc20 con 0x7f88b415e030 2024-03-04T09:32:19.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.919+0000 7f88aaa00640 1 --2- 172.21.15.6:0/652926296 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f889006b500 0x7f889006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:19.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.919+0000 7f88aaa00640 1 -- 172.21.15.6:0/652926296 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f88a408a760 con 0x7f88b415e030 2024-03-04T09:32:19.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.919+0000 7f88a9400640 1 --2- 172.21.15.6:0/652926296 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f889006b500 0x7f889006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:19.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.920+0000 7f88a9400640 1 --2- 172.21.15.6:0/652926296 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f889006b500 0x7f889006d9c0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f88b40b1900 tx=0x7f889c009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:19.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:19.924+0000 7f88aaa00640 1 -- 172.21.15.6:0/652926296 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f88a401c360 con 0x7f88b415e030 2024-03-04T09:32:20.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:20.162+0000 7f88b9200640 1 -- 172.21.15.6:0/652926296 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f88b408e030 con 0x7f88b415e030 2024-03-04T09:32:20.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:20.163+0000 7f88aaa00640 1 -- 172.21.15.6:0/652926296 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+4975 (secure 0 0 0) 0x7f88a4059540 con 0x7f88b415e030 2024-03-04T09:32:20.160 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:32:20.160 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":84,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6535,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/3883122012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":3883122012},{"type":"v1","addr":"172.21.15.146:6835","nonce":3883122012}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":6559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/4060411493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":4060411493},{"type":"v1","addr":"172.21.15.146:6837","nonce":4060411493}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":82},{"gid":7812,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/679471398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":679471398},{"type":"v1","addr":"172.21.15.6:6833","nonce":679471398}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:31:21.114351+0000","modified":"2024-03-04T09:31:40.781474+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7815},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7815":{"gid":7815,"name":"a","rank":0,"incarnation":81,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/882667161","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":882667161},{"type":"v1","addr":"172.21.15.6:6835","nonce":882667161}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7815,"qdb_cluster":[7815]},"id":8}]} 2024-03-04T09:32:20.161 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 84 2024-03-04T09:32:20.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:20.166+0000 7f88b9200640 1 -- 172.21.15.6:0/652926296 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f889006b500 msgr2=0x7f889006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:20.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:20.166+0000 7f88b9200640 1 --2- 172.21.15.6:0/652926296 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f889006b500 0x7f889006d9c0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f88b40b1900 tx=0x7f889c009040 comp rx=0 tx=0).stop 2024-03-04T09:32:20.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:20.166+0000 7f88b9200640 1 -- 172.21.15.6:0/652926296 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f88b415e030 msgr2=0x7f88b40c7810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:20.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:20.166+0000 7f88b9200640 1 --2- 172.21.15.6:0/652926296 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f88b415e030 0x7f88b40c7810 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7f88a400d800 tx=0x7f88a400dcc0 comp rx=0 tx=0).stop 2024-03-04T09:32:20.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:20.166+0000 7f88b9200640 1 -- 172.21.15.6:0/652926296 shutdown_connections 2024-03-04T09:32:20.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:20.166+0000 7f88b9200640 1 --2- 172.21.15.6:0/652926296 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f889006b500 0x7f889006d9c0 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:20.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:20.166+0000 7f88b9200640 1 --2- 172.21.15.6:0/652926296 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f88b4159000 0x7f88b41594b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:20.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:20.166+0000 7f88b9200640 1 --2- 172.21.15.6:0/652926296 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f88b415e410 0x7f88b40d03b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:20.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:20.166+0000 7f88b9200640 1 --2- 172.21.15.6:0/652926296 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f88b415e030 0x7f88b40c7810 unknown :-1 s=CLOSED pgs=1349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:20.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:20.166+0000 7f88b9200640 1 -- 172.21.15.6:0/652926296 >> 172.21.15.6:0/652926296 conn(0x7f88b40d3eb0 msgr2=0x7f88b40b2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:32:20.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:20.166+0000 7f88b9200640 1 -- 172.21.15.6:0/652926296 shutdown_connections 2024-03-04T09:32:20.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:20.166+0000 7f88b9200640 1 -- 172.21.15.6:0/652926296 wait complete. 2024-03-04T09:32:20.191 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_reintegrated 2024-03-04T09:32:20.421 INFO:teuthology.orchestra.run.smithi006.stdout:{"mds_cache":{"strays_reintegrated":1}} 2024-03-04T09:32:20.421 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_reintegrated": 1 } } 2024-03-04T09:32:20.421 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:32:20.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:20.635+0000 7f0392a00640 1 Processor -- start 2024-03-04T09:32:20.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:20.635+0000 7f0392a00640 1 -- start start 2024-03-04T09:32:20.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:20.637+0000 7f0392a00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f038c0d4a10 0x7f038c0d0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:20.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:20.637+0000 7f0392a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f038c0d0c80 0x7f038c0b44d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:20.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:20.637+0000 7f0392a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f038c0b3710 con 0x7f038c15e0b0 2024-03-04T09:32:20.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:20.637+0000 7f0392a00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f038c0b09e0 con 0x7f038c0d4a10 2024-03-04T09:32:20.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:20.637+0000 7f0392a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f038c0cfa20 con 0x7f038c0d0c80 2024-03-04T09:32:20.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:20.637+0000 7f0391600640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f038c15e0b0 0x7f038c0cf310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:56248/0 (socket says 172.21.15.6:56248) 2024-03-04T09:32:20.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:20.637+0000 7f0391600640 1 -- 172.21.15.6:0/62484569 learned_addr learned my addr 172.21.15.6:0/62484569 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:32:20.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:20.637+0000 7f0392000640 1 --2- 172.21.15.6:0/62484569 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f038c0d0c80 0x7f038c0b44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:20.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:20.638+0000 7f0390c00640 1 --2- 172.21.15.6:0/62484569 >> v2:172.21.15.146:3300/0 conn(0x7f038c0d4a10 0x7f038c0d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:20.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:20.638+0000 7f038be00640 1 -- 172.21.15.6:0/62484569 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1543705247 0 0) 0x7f038c0b3710 con 0x7f038c15e0b0 2024-03-04T09:32:20.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:20.638+0000 7f038be00640 1 -- 172.21.15.6:0/62484569 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0374003520 con 0x7f038c15e0b0 2024-03-04T09:32:20.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:20.638+0000 7f038be00640 1 -- 172.21.15.6:0/62484569 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3962449411 0 0) 0x7f0374003520 con 0x7f038c15e0b0 2024-03-04T09:32:20.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:20.638+0000 7f038be00640 1 -- 172.21.15.6:0/62484569 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f038c0d0c80 msgr2=0x7f038c0b44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:32:20.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:20.638+0000 7f038be00640 1 --2- 172.21.15.6:0/62484569 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f038c0d0c80 0x7f038c0b44d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:20.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:20.638+0000 7f038be00640 1 -- 172.21.15.6:0/62484569 >> v2:172.21.15.146:3300/0 conn(0x7f038c0d4a10 msgr2=0x7f038c0d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:32:20.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:20.638+0000 7f038be00640 1 --2- 172.21.15.6:0/62484569 >> v2:172.21.15.146:3300/0 conn(0x7f038c0d4a10 0x7f038c0d0740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:20.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:20.638+0000 7f038be00640 1 -- 172.21.15.6:0/62484569 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f038c0b39f0 con 0x7f038c15e0b0 2024-03-04T09:32:20.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:20.638+0000 7f038be00640 1 -- 172.21.15.6:0/62484569 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7f0380001670 con 0x7f038c15e0b0 2024-03-04T09:32:20.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:20.638+0000 7f038be00640 1 -- 172.21.15.6:0/62484569 >> v1:172.21.15.6:6789/0 conn(0x7f038c15e0b0 legacy=0x7f038c0cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:20.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:20.638+0000 7f038be00640 1 --2- 172.21.15.6:0/62484569 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f038c0d0c80 0x7f03740039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:20.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:20.638+0000 7f038be00640 1 --2- 172.21.15.6:0/62484569 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0374003f30 0x7f0374004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:20.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:20.638+0000 7f038be00640 1 --2- 172.21.15.6:0/62484569 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0374004d40 0x7f03740051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:20.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:20.638+0000 7f038be00640 1 -- 172.21.15.6:0/62484569 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f038c0cfa20 con 0x7f0374004d40 2024-03-04T09:32:20.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:20.638+0000 7f038be00640 1 -- 172.21.15.6:0/62484569 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f037400b9c0 con 0x7f0374003f30 2024-03-04T09:32:20.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:20.639+0000 7f038be00640 1 -- 172.21.15.6:0/62484569 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f037400bb40 con 0x7f038c0d0c80 2024-03-04T09:32:20.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:20.639+0000 7f0392000640 1 --2- 172.21.15.6:0/62484569 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0374004d40 0x7f03740051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:20.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:20.639+0000 7f0390c00640 1 --2- 172.21.15.6:0/62484569 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0374003f30 0x7f0374004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:20.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:20.639+0000 7f0391600640 1 --2- 172.21.15.6:0/62484569 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f038c0d0c80 0x7f03740039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:20.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:20.639+0000 7f0392000640 1 -- 172.21.15.6:0/62484569 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f038c0d0c80 msgr2=0x7f03740039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:32:20.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:20.639+0000 7f0392000640 1 --2- 172.21.15.6:0/62484569 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f038c0d0c80 0x7f03740039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:20.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:20.639+0000 7f0392000640 1 -- 172.21.15.6:0/62484569 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0374003f30 msgr2=0x7f0374004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:32:20.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:20.639+0000 7f0392000640 1 --2- 172.21.15.6:0/62484569 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0374003f30 0x7f0374004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:20.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:20.639+0000 7f0392000640 1 -- 172.21.15.6:0/62484569 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0374003520 con 0x7f0374004d40 2024-03-04T09:32:20.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:20.641+0000 7f0392000640 1 --2- 172.21.15.6:0/62484569 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0374004d40 0x7f03740051c0 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7f0384009fc0 tx=0x7f0374005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4709d2ee8ff94ed server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:20.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:20.641+0000 7f038be00640 1 -- 172.21.15.6:0/62484569 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0384039070 con 0x7f0374004d40 2024-03-04T09:32:20.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:20.641+0000 7f038be00640 1 -- 172.21.15.6:0/62484569 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f038402cc00 con 0x7f0374004d40 2024-03-04T09:32:20.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:20.641+0000 7f038be00640 1 -- 172.21.15.6:0/62484569 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0384009c10 con 0x7f0374004d40 2024-03-04T09:32:20.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:20.641+0000 7f0392a00640 1 -- 172.21.15.6:0/62484569 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0374004d40 msgr2=0x7f03740051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:20.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:20.641+0000 7f0392a00640 1 --2- 172.21.15.6:0/62484569 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0374004d40 0x7f03740051c0 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7f0384009fc0 tx=0x7f0374005830 comp rx=0 tx=0).stop 2024-03-04T09:32:20.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:20.641+0000 7f0392000640 1 -- 172.21.15.6:0/62484569 reap_dead start 2024-03-04T09:32:20.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:20.642+0000 7f0392a00640 1 -- 172.21.15.6:0/62484569 shutdown_connections 2024-03-04T09:32:20.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:20.642+0000 7f0392a00640 1 -- 172.21.15.6:0/62484569 >> 172.21.15.6:0/62484569 conn(0x7f038c0d3d30 msgr2=0x7f038c0d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:32:20.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:20.642+0000 7f0392a00640 1 -- 172.21.15.6:0/62484569 shutdown_connections 2024-03-04T09:32:20.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:20.642+0000 7f0392a00640 1 -- 172.21.15.6:0/62484569 wait complete. 2024-03-04T09:32:20.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:20.644+0000 7f0392a00640 1 Processor -- start 2024-03-04T09:32:20.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:20.644+0000 7f0392a00640 1 -- start start 2024-03-04T09:32:20.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:20.645+0000 7f0392a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0374004d40 0x7f038c0b7f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:20.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:20.645+0000 7f0392a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f038c0b6e70 0x7f038c160ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:20.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:20.645+0000 7f0392a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f038c0b72f0 0x7f038c146940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:20.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:20.645+0000 7f0392a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f038c15e490 con 0x7f038c0b6e70 2024-03-04T09:32:20.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:20.645+0000 7f0392a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f038c0b3fb0 con 0x7f038c0b72f0 2024-03-04T09:32:20.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:20.645+0000 7f0392a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f038c0cfc60 con 0x7f0374004d40 2024-03-04T09:32:20.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:20.645+0000 7f0390c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f038c0b6e70 0x7f038c160ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:20.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:20.645+0000 7f0390c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f038c0b6e70 0x7f038c160ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43974/0 (socket says 172.21.15.6:43974) 2024-03-04T09:32:20.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:20.645+0000 7f0390c00640 1 -- 172.21.15.6:0/2964068908 learned_addr learned my addr 172.21.15.6:0/2964068908 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:32:20.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:20.646+0000 7f0391600640 1 --2- 172.21.15.6:0/2964068908 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0374004d40 0x7f038c0b7f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:20.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:20.646+0000 7f0390c00640 1 -- 172.21.15.6:0/2964068908 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0374004d40 msgr2=0x7f038c0b7f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:20.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:20.646+0000 7f0390c00640 1 --2- 172.21.15.6:0/2964068908 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0374004d40 0x7f038c0b7f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:20.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:20.646+0000 7f0392000640 1 --2- 172.21.15.6:0/2964068908 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f038c0b72f0 0x7f038c146940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:20.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:20.646+0000 7f0390c00640 1 -- 172.21.15.6:0/2964068908 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f038c0b72f0 msgr2=0x7f038c146940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:20.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:20.646+0000 7f0390c00640 1 --2- 172.21.15.6:0/2964068908 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f038c0b72f0 0x7f038c146940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:20.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:20.646+0000 7f0390c00640 1 -- 172.21.15.6:0/2964068908 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f038c0cd9b0 con 0x7f038c0b6e70 2024-03-04T09:32:20.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:20.646+0000 7f0390c00640 1 --2- 172.21.15.6:0/2964068908 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f038c0b6e70 0x7f038c160ed0 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7f03780115e0 tx=0x7f0378011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:20.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:20.646+0000 7f0389600640 1 -- 172.21.15.6:0/2964068908 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0378004220 con 0x7f038c0b6e70 2024-03-04T09:32:20.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:20.646+0000 7f0389600640 1 -- 172.21.15.6:0/2964068908 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03780043c0 con 0x7f038c0b6e70 2024-03-04T09:32:20.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:20.646+0000 7f0392a00640 1 -- 172.21.15.6:0/2964068908 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f038c0cdc90 con 0x7f038c0b6e70 2024-03-04T09:32:20.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:20.646+0000 7f0389600640 1 -- 172.21.15.6:0/2964068908 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f037800f930 con 0x7f038c0b6e70 2024-03-04T09:32:20.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:20.647+0000 7f0392a00640 1 -- 172.21.15.6:0/2964068908 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f038c153be0 con 0x7f038c0b6e70 2024-03-04T09:32:20.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:20.648+0000 7f0392a00640 1 -- 172.21.15.6:0/2964068908 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f038c0d0740 con 0x7f038c0b6e70 2024-03-04T09:32:20.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:20.649+0000 7f0389600640 1 -- 172.21.15.6:0/2964068908 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f037800fad0 con 0x7f038c0b6e70 2024-03-04T09:32:20.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:20.650+0000 7f0389600640 1 --2- 172.21.15.6:0/2964068908 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f036c06b4d0 0x7f036c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:20.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:20.650+0000 7f0389600640 1 -- 172.21.15.6:0/2964068908 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f037808a8e0 con 0x7f038c0b6e70 2024-03-04T09:32:20.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:20.650+0000 7f0391600640 1 --2- 172.21.15.6:0/2964068908 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f036c06b4d0 0x7f036c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:20.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:20.651+0000 7f0391600640 1 --2- 172.21.15.6:0/2964068908 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f036c06b4d0 0x7f036c06d990 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f03800058b0 tx=0x7f0380001db0 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:20.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:20.659+0000 7f0389600640 1 -- 172.21.15.6:0/2964068908 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f038c0d0740 con 0x7f038c0b6e70 2024-03-04T09:32:20.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:20.938+0000 7f0392a00640 1 -- 172.21.15.6:0/2964068908 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f038c04f330 con 0x7f038c0b6e70 2024-03-04T09:32:20.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:20.938+0000 7f0389600640 1 -- 172.21.15.6:0/2964068908 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+4975 (secure 0 0 0) 0x7f0378019030 con 0x7f038c0b6e70 2024-03-04T09:32:20.935 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:32:20.936 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":84,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6535,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/3883122012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":3883122012},{"type":"v1","addr":"172.21.15.146:6835","nonce":3883122012}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":6559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/4060411493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":4060411493},{"type":"v1","addr":"172.21.15.146:6837","nonce":4060411493}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":82},{"gid":7812,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/679471398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":679471398},{"type":"v1","addr":"172.21.15.6:6833","nonce":679471398}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:31:21.114351+0000","modified":"2024-03-04T09:31:40.781474+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7815},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7815":{"gid":7815,"name":"a","rank":0,"incarnation":81,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/882667161","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":882667161},{"type":"v1","addr":"172.21.15.6:6835","nonce":882667161}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7815,"qdb_cluster":[7815]},"id":8}]} 2024-03-04T09:32:20.936 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 84 2024-03-04T09:32:20.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:20.941+0000 7f0392a00640 1 -- 172.21.15.6:0/2964068908 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f036c06b4d0 msgr2=0x7f036c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:20.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:20.941+0000 7f0392a00640 1 --2- 172.21.15.6:0/2964068908 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f036c06b4d0 0x7f036c06d990 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f03800058b0 tx=0x7f0380001db0 comp rx=0 tx=0).stop 2024-03-04T09:32:20.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:20.941+0000 7f0392a00640 1 -- 172.21.15.6:0/2964068908 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f038c0b6e70 msgr2=0x7f038c160ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:20.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:20.941+0000 7f0392a00640 1 --2- 172.21.15.6:0/2964068908 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f038c0b6e70 0x7f038c160ed0 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7f03780115e0 tx=0x7f0378011aa0 comp rx=0 tx=0).stop 2024-03-04T09:32:20.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:20.941+0000 7f0392a00640 1 -- 172.21.15.6:0/2964068908 shutdown_connections 2024-03-04T09:32:20.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:20.941+0000 7f0392a00640 1 --2- 172.21.15.6:0/2964068908 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f038c0b72f0 0x7f038c146940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:20.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:20.941+0000 7f0392a00640 1 --2- 172.21.15.6:0/2964068908 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f038c0b6e70 0x7f038c160ed0 unknown :-1 s=CLOSED pgs=1352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:20.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:20.941+0000 7f0392a00640 1 --2- 172.21.15.6:0/2964068908 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f036c06b4d0 0x7f036c06d990 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:20.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:20.941+0000 7f0392a00640 1 --2- 172.21.15.6:0/2964068908 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0374004d40 0x7f038c0b7f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:20.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:20.941+0000 7f0392a00640 1 -- 172.21.15.6:0/2964068908 >> 172.21.15.6:0/2964068908 conn(0x7f038c0d3d30 msgr2=0x7f038c0cf070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:32:20.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:20.941+0000 7f0392a00640 1 -- 172.21.15.6:0/2964068908 shutdown_connections 2024-03-04T09:32:20.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:20.942+0000 7f0392a00640 1 -- 172.21.15.6:0/2964068908 wait complete. 2024-03-04T09:32:20.957 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json flush journal 2024-03-04T09:32:21.197 INFO:teuthology.orchestra.run.smithi006.stdout:{"message":"","return_code":0} 2024-03-04T09:32:21.198 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-04T09:32:21.198 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 10000000001.00000000 parent 2024-03-04T09:32:21.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.296+0000 7f750795df40 1 Processor -- start 2024-03-04T09:32:21.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.296+0000 7f750795df40 1 -- start start 2024-03-04T09:32:21.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.297+0000 7f750795df40 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557086624ba0 0x5570866f7e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:21.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.297+0000 7f750795df40 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55708666b690 0x55708664de80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:21.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.297+0000 7f750795df40 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5570866f93d0 0x5570866237d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:21.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.297+0000 7f750795df40 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x557086624fb0 con 0x5570866f93d0 2024-03-04T09:32:21.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.297+0000 7f750795df40 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570865c8fd0 con 0x557086624ba0 2024-03-04T09:32:21.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.297+0000 7f750795df40 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x55708666bb30 con 0x55708666b690 2024-03-04T09:32:21.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.298+0000 7f7505600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5570866f93d0 0x5570866237d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:21.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.298+0000 7f7505600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5570866f93d0 0x5570866237d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43980/0 (socket says 172.21.15.6:43980) 2024-03-04T09:32:21.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.298+0000 7f7505600640 1 -- 172.21.15.6:0/1645218523 learned_addr learned my addr 172.21.15.6:0/1645218523 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:32:21.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.298+0000 7f7504200640 1 --2- 172.21.15.6:0/1645218523 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55708666b690 0x55708664de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:21.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.298+0000 7f7505600640 1 -- 172.21.15.6:0/1645218523 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55708666b690 msgr2=0x55708664de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:32:21.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.298+0000 7f7504c00640 1 --2- 172.21.15.6:0/1645218523 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557086624ba0 0x5570866f7e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:21.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.298+0000 7f7505600640 1 --2- 172.21.15.6:0/1645218523 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55708666b690 0x55708664de80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:21.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.298+0000 7f7505600640 1 -- 172.21.15.6:0/1645218523 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557086624ba0 msgr2=0x5570866f7e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:32:21.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.298+0000 7f7505600640 1 --2- 172.21.15.6:0/1645218523 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557086624ba0 0x5570866f7e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:21.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.298+0000 7f7505600640 1 -- 172.21.15.6:0/1645218523 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5570866f83b0 con 0x5570866f93d0 2024-03-04T09:32:21.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.298+0000 7f7505600640 1 --2- 172.21.15.6:0/1645218523 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5570866f93d0 0x5570866237d0 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7f74fc001660 tx=0x7f74fc001b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3bec237928a56b3 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:21.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.299+0000 7f7502e00640 1 -- 172.21.15.6:0/1645218523 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74fc003650 con 0x5570866f93d0 2024-03-04T09:32:21.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.299+0000 7f7502e00640 1 -- 172.21.15.6:0/1645218523 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74fc003830 con 0x5570866f93d0 2024-03-04T09:32:21.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.299+0000 7f7502e00640 1 -- 172.21.15.6:0/1645218523 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74fc003ca0 con 0x5570866f93d0 2024-03-04T09:32:21.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.299+0000 7f750795df40 1 -- 172.21.15.6:0/1645218523 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5570866f93d0 msgr2=0x5570866237d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:21.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.299+0000 7f750795df40 1 --2- 172.21.15.6:0/1645218523 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5570866f93d0 0x5570866237d0 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7f74fc001660 tx=0x7f74fc001b20 comp rx=0 tx=0).stop 2024-03-04T09:32:21.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.299+0000 7f750795df40 1 -- 172.21.15.6:0/1645218523 shutdown_connections 2024-03-04T09:32:21.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.299+0000 7f750795df40 1 --2- 172.21.15.6:0/1645218523 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5570866f93d0 0x5570866237d0 unknown :-1 s=CLOSED pgs=1355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:21.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.299+0000 7f750795df40 1 --2- 172.21.15.6:0/1645218523 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55708666b690 0x55708664de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:21.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.299+0000 7f750795df40 1 --2- 172.21.15.6:0/1645218523 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557086624ba0 0x5570866f7e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:21.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.299+0000 7f750795df40 1 -- 172.21.15.6:0/1645218523 >> 172.21.15.6:0/1645218523 conn(0x5570866225c0 msgr2=0x5570866f03e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:32:21.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.299+0000 7f750795df40 1 -- 172.21.15.6:0/1645218523 shutdown_connections 2024-03-04T09:32:21.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.299+0000 7f750795df40 1 -- 172.21.15.6:0/1645218523 wait complete. 2024-03-04T09:32:21.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.301+0000 7f750795df40 1 Processor -- start 2024-03-04T09:32:21.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.301+0000 7f750795df40 1 -- start start 2024-03-04T09:32:21.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.302+0000 7f750795df40 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x557086624ba0 0x5570866e78e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:21.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.302+0000 7f750795df40 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55708666b690 0x5570866e7fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:21.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.302+0000 7f750795df40 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5570866f93d0 0x5570866f0ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:21.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.302+0000 7f750795df40 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x557086666b20 con 0x557086624ba0 2024-03-04T09:32:21.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.302+0000 7f750795df40 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570866f01e0 con 0x5570866f93d0 2024-03-04T09:32:21.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.302+0000 7f750795df40 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x5570866eb200 con 0x55708666b690 2024-03-04T09:32:21.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.303+0000 7f7504c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x557086624ba0 0x5570866e78e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:21.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.303+0000 7f7504c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x557086624ba0 0x5570866e78e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43990/0 (socket says 172.21.15.6:43990) 2024-03-04T09:32:21.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.303+0000 7f7504c00640 1 -- 172.21.15.6:0/1329334802 learned_addr learned my addr 172.21.15.6:0/1329334802 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:32:21.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.303+0000 7f7504200640 1 --2- 172.21.15.6:0/1329334802 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55708666b690 0x5570866e7fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:21.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.303+0000 7f7504c00640 1 -- 172.21.15.6:0/1329334802 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55708666b690 msgr2=0x5570866e7fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:21.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.303+0000 7f7504c00640 1 --2- 172.21.15.6:0/1329334802 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55708666b690 0x5570866e7fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:21.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.303+0000 7f7504c00640 1 -- 172.21.15.6:0/1329334802 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5570866f93d0 msgr2=0x5570866f0ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:21.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.303+0000 7f7504c00640 1 --2- 172.21.15.6:0/1329334802 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5570866f93d0 0x5570866f0ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:21.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.303+0000 7f7504c00640 1 -- 172.21.15.6:0/1329334802 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5570866e8970 con 0x557086624ba0 2024-03-04T09:32:21.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.303+0000 7f7504c00640 1 --2- 172.21.15.6:0/1329334802 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x557086624ba0 0x5570866e78e0 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7f74f4055040 tx=0x7f74f4002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:21.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.304+0000 7f7501a00640 1 -- 172.21.15.6:0/1329334802 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74f4062020 con 0x557086624ba0 2024-03-04T09:32:21.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.304+0000 7f7501a00640 1 -- 172.21.15.6:0/1329334802 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74f4052b20 con 0x557086624ba0 2024-03-04T09:32:21.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.304+0000 7f750795df40 1 -- 172.21.15.6:0/1329334802 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x5570866e8c50 con 0x557086624ba0 2024-03-04T09:32:21.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.304+0000 7f7501a00640 1 -- 172.21.15.6:0/1329334802 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74f4057040 con 0x557086624ba0 2024-03-04T09:32:21.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.304+0000 7f750795df40 1 -- 172.21.15.6:0/1329334802 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5570866e9210 con 0x557086624ba0 2024-03-04T09:32:21.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.307+0000 7f7501a00640 1 -- 172.21.15.6:0/1329334802 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f74f4065020 con 0x557086624ba0 2024-03-04T09:32:21.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.308+0000 7f7501a00640 1 --2- 172.21.15.6:0/1329334802 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f74e4069eb0 0x7f74e406c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:21.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.308+0000 7f7501a00640 1 -- 172.21.15.6:0/1329334802 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f74f40cadd0 con 0x557086624ba0 2024-03-04T09:32:21.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.308+0000 7f750795df40 1 --2- 172.21.15.6:0/1329334802 >> [v2:172.21.15.6:6823/1364449401,v1:172.21.15.6:6825/1364449401] conn(0x5570866e84f0 0x5570866ed760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:21.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.308+0000 7f750795df40 1 -- 172.21.15.6:0/1329334802 --> [v2:172.21.15.6:6823/1364449401,v1:172.21.15.6:6825/1364449401] -- osd_op(unknown.0.0:1 17.7 17:e2e07930:::10000000001.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e115) v9 -- 0x55708664cb50 con 0x5570866e84f0 2024-03-04T09:32:21.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.309+0000 7f7504200640 1 --2- 172.21.15.6:0/1329334802 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f74e4069eb0 0x7f74e406c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:21.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.309+0000 7f7505600640 1 --2- 172.21.15.6:0/1329334802 >> [v2:172.21.15.6:6823/1364449401,v1:172.21.15.6:6825/1364449401] conn(0x5570866e84f0 0x5570866ed760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:21.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.310+0000 7f7505600640 1 --2- 172.21.15.6:0/1329334802 >> [v2:172.21.15.6:6823/1364449401,v1:172.21.15.6:6825/1364449401] conn(0x5570866e84f0 0x5570866ed760 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:21.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.310+0000 7f7504200640 1 --2- 172.21.15.6:0/1329334802 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f74e4069eb0 0x7f74e406c370 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f74f0005400 tx=0x7f74f0009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:21.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.310+0000 7f7505600640 1 -- 172.21.15.6:0/1329334802 <== osd.0 v2:172.21.15.6:6823/1364449401 1 ==== osd_op_reply(1 10000000001.00000000 [getxattr (93) out=93b] v0'0 uv10 ondisk = 0) v8 ==== 164+0+93 (crc 0 0 0) 0x7f74fc00f030 con 0x5570866e84f0 2024-03-04T09:32:21.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.310+0000 7f750795df40 1 -- 172.21.15.6:0/1329334802 >> [v2:172.21.15.6:6823/1364449401,v1:172.21.15.6:6825/1364449401] conn(0x5570866e84f0 msgr2=0x5570866ed760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:21.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.310+0000 7f750795df40 1 --2- 172.21.15.6:0/1329334802 >> [v2:172.21.15.6:6823/1364449401,v1:172.21.15.6:6825/1364449401] conn(0x5570866e84f0 0x5570866ed760 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:21.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.310+0000 7f750795df40 1 -- 172.21.15.6:0/1329334802 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f74e4069eb0 msgr2=0x7f74e406c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:21.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.310+0000 7f750795df40 1 --2- 172.21.15.6:0/1329334802 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f74e4069eb0 0x7f74e406c370 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f74f0005400 tx=0x7f74f0009040 comp rx=0 tx=0).stop 2024-03-04T09:32:21.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.310+0000 7f750795df40 1 -- 172.21.15.6:0/1329334802 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x557086624ba0 msgr2=0x5570866e78e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:21.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.310+0000 7f750795df40 1 --2- 172.21.15.6:0/1329334802 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x557086624ba0 0x5570866e78e0 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7f74f4055040 tx=0x7f74f4002fb0 comp rx=0 tx=0).stop 2024-03-04T09:32:21.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.310+0000 7f7505600640 1 -- 172.21.15.6:0/1329334802 reap_dead start 2024-03-04T09:32:21.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.310+0000 7f750795df40 1 -- 172.21.15.6:0/1329334802 shutdown_connections 2024-03-04T09:32:21.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.310+0000 7f750795df40 1 -- 172.21.15.6:0/1329334802 >> 172.21.15.6:0/1329334802 conn(0x5570866225c0 msgr2=0x55708664de80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:32:21.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.310+0000 7f750795df40 1 -- 172.21.15.6:0/1329334802 shutdown_connections 2024-03-04T09:32:21.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.310+0000 7f750795df40 1 -- 172.21.15.6:0/1329334802 wait complete. 2024-03-04T09:32:21.312 INFO:teuthology.orchestra.run.smithi006.stdout:W�file_cdir_2# 2024-03-04T09:32:21.312 DEBUG:teuthology.orchestra.run.smithi006:> ceph-dencoder type inode_backtrace_t import - decode dump_json 2024-03-04T09:32:21.402 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-04T09:32:21.402 INFO:teuthology.orchestra.run.smithi006.stdout: "ino": 1099511627777, 2024-03-04T09:32:21.402 INFO:teuthology.orchestra.run.smithi006.stdout: "ancestors": [ 2024-03-04T09:32:21.403 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:32:21.403 INFO:teuthology.orchestra.run.smithi006.stdout: "dirino": 1099511628278, 2024-03-04T09:32:21.403 INFO:teuthology.orchestra.run.smithi006.stdout: "dname": "file_c", 2024-03-04T09:32:21.403 INFO:teuthology.orchestra.run.smithi006.stdout: "version": 17 2024-03-04T09:32:21.403 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:32:21.403 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:32:21.403 INFO:teuthology.orchestra.run.smithi006.stdout: "dirino": 1, 2024-03-04T09:32:21.403 INFO:teuthology.orchestra.run.smithi006.stdout: "dname": "dir_2", 2024-03-04T09:32:21.403 INFO:teuthology.orchestra.run.smithi006.stdout: "version": 35 2024-03-04T09:32:21.403 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:32:21.403 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-03-04T09:32:21.403 INFO:teuthology.orchestra.run.smithi006.stdout: "pool": 17, 2024-03-04T09:32:21.403 INFO:teuthology.orchestra.run.smithi006.stdout: "old_pools": [] 2024-03-04T09:32:21.403 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-04T09:32:21.403 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:32:21.408 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:32:21.408 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest/mnt.0 && exec ln dir_2/file_c dir_2/file_d) 2024-03-04T09:32:21.463 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:32:21.463 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest/mnt.0 && exec rm -f dir_2/file_c) 2024-03-04T09:32:21.522 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:32:21.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.759+0000 7fda35e00640 1 Processor -- start 2024-03-04T09:32:21.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.759+0000 7fda35e00640 1 -- start start 2024-03-04T09:32:21.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.761+0000 7fda35e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fda300d4a10 0x7fda300d0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:21.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.761+0000 7fda35e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fda300d0c80 0x7fda300b44d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:21.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.761+0000 7fda35e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda300b3710 con 0x7fda3015e0b0 2024-03-04T09:32:21.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.761+0000 7fda35e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda300d10f0 con 0x7fda300d4a10 2024-03-04T09:32:21.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.761+0000 7fda35e00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda300b09e0 con 0x7fda300d0c80 2024-03-04T09:32:21.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.761+0000 7fda2fe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fda300d4a10 0x7fda300d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:21.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.761+0000 7fda2fe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fda300d4a10 0x7fda300d0740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:44002/0 (socket says 172.21.15.6:44002) 2024-03-04T09:32:21.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.761+0000 7fda2fe00640 1 -- 172.21.15.6:0/1287406784 learned_addr learned my addr 172.21.15.6:0/1287406784 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:32:21.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.762+0000 7fda2fe00640 1 -- 172.21.15.6:0/1287406784 >> v2:172.21.15.146:3300/0 conn(0x7fda300d0c80 msgr2=0x7fda300b44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:32:21.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.762+0000 7fda35400640 1 --2- 172.21.15.6:0/1287406784 >> v2:172.21.15.146:3300/0 conn(0x7fda300d0c80 0x7fda300b44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:21.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.762+0000 7fda2fe00640 1 --2- 172.21.15.6:0/1287406784 >> v2:172.21.15.146:3300/0 conn(0x7fda300d0c80 0x7fda300b44d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:21.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.762+0000 7fda2fe00640 1 -- 172.21.15.6:0/1287406784 >> v1:172.21.15.6:6789/0 conn(0x7fda3015e0b0 legacy=0x7fda300cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:21.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.762+0000 7fda2fe00640 1 -- 172.21.15.6:0/1287406784 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda300b39f0 con 0x7fda300d4a10 2024-03-04T09:32:21.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.763+0000 7fda2fe00640 1 --2- 172.21.15.6:0/1287406784 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fda300d4a10 0x7fda300d0740 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7fda3008d6d0 tx=0x7fda1c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d69a2accec955e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:21.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.763+0000 7fda2f400640 1 -- 172.21.15.6:0/1287406784 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda1c034070 con 0x7fda300d4a10 2024-03-04T09:32:21.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.763+0000 7fda2f400640 1 -- 172.21.15.6:0/1287406784 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda1c030860 con 0x7fda300d4a10 2024-03-04T09:32:21.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.763+0000 7fda2f400640 1 -- 172.21.15.6:0/1287406784 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda1c030c30 con 0x7fda300d4a10 2024-03-04T09:32:21.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.764+0000 7fda35e00640 1 -- 172.21.15.6:0/1287406784 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fda300d4a10 msgr2=0x7fda300d0740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:21.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.764+0000 7fda35e00640 1 --2- 172.21.15.6:0/1287406784 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fda300d4a10 0x7fda300d0740 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7fda3008d6d0 tx=0x7fda1c02b9a0 comp rx=0 tx=0).stop 2024-03-04T09:32:21.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.764+0000 7fda35e00640 1 -- 172.21.15.6:0/1287406784 shutdown_connections 2024-03-04T09:32:21.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.764+0000 7fda35e00640 1 --2- 172.21.15.6:0/1287406784 >> v2:172.21.15.146:3300/0 conn(0x7fda300d0c80 0x7fda300b44d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:21.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.764+0000 7fda35e00640 1 --2- 172.21.15.6:0/1287406784 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fda300d4a10 0x7fda300d0740 unknown :-1 s=CLOSED pgs=1358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:21.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.764+0000 7fda35e00640 1 -- 172.21.15.6:0/1287406784 >> 172.21.15.6:0/1287406784 conn(0x7fda300d3d30 msgr2=0x7fda300d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:32:21.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.764+0000 7fda35e00640 1 -- 172.21.15.6:0/1287406784 shutdown_connections 2024-03-04T09:32:21.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.764+0000 7fda35e00640 1 -- 172.21.15.6:0/1287406784 wait complete. 2024-03-04T09:32:21.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.766+0000 7fda35e00640 1 Processor -- start 2024-03-04T09:32:21.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.766+0000 7fda35e00640 1 -- start start 2024-03-04T09:32:21.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.767+0000 7fda35e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fda300d0c80 0x7fda3015e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:21.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.767+0000 7fda35e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fda300d4a10 0x7fda300b5440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:21.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.767+0000 7fda35e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fda3015e0b0 0x7fda30157e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:21.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.767+0000 7fda35e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda300b3fb0 con 0x7fda300d4a10 2024-03-04T09:32:21.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.767+0000 7fda35e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda300cfa20 con 0x7fda3015e0b0 2024-03-04T09:32:21.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.767+0000 7fda35e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda3015e490 con 0x7fda300d0c80 2024-03-04T09:32:21.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.767+0000 7fda2fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fda300d4a10 0x7fda300b5440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:21.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.767+0000 7fda2fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fda300d4a10 0x7fda300b5440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:44012/0 (socket says 172.21.15.6:44012) 2024-03-04T09:32:21.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.767+0000 7fda2fe00640 1 -- 172.21.15.6:0/3336278171 learned_addr learned my addr 172.21.15.6:0/3336278171 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:32:21.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.768+0000 7fda35400640 1 --2- 172.21.15.6:0/3336278171 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fda3015e0b0 0x7fda30157e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:21.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.768+0000 7fda34a00640 1 --2- 172.21.15.6:0/3336278171 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fda300d0c80 0x7fda3015e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:21.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.768+0000 7fda2fe00640 1 -- 172.21.15.6:0/3336278171 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fda300d0c80 msgr2=0x7fda3015e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:21.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.768+0000 7fda2fe00640 1 --2- 172.21.15.6:0/3336278171 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fda300d0c80 0x7fda3015e670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:21.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.768+0000 7fda2fe00640 1 -- 172.21.15.6:0/3336278171 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fda3015e0b0 msgr2=0x7fda30157e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:21.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.768+0000 7fda2fe00640 1 --2- 172.21.15.6:0/3336278171 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fda3015e0b0 0x7fda30157e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:21.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.768+0000 7fda2fe00640 1 -- 172.21.15.6:0/3336278171 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda3015ef30 con 0x7fda300d4a10 2024-03-04T09:32:21.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.768+0000 7fda35400640 1 --2- 172.21.15.6:0/3336278171 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fda3015e0b0 0x7fda30157e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:32:21.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.768+0000 7fda2fe00640 1 --2- 172.21.15.6:0/3336278171 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fda300d4a10 0x7fda300b5440 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7fda1c000c00 tx=0x7fda1c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:21.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.768+0000 7fda2cc00640 1 -- 172.21.15.6:0/3336278171 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda1c048070 con 0x7fda300d4a10 2024-03-04T09:32:21.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.769+0000 7fda2cc00640 1 -- 172.21.15.6:0/3336278171 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda1c039d90 con 0x7fda300d4a10 2024-03-04T09:32:21.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.769+0000 7fda2cc00640 1 -- 172.21.15.6:0/3336278171 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda1c034040 con 0x7fda300d4a10 2024-03-04T09:32:21.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.769+0000 7fda35e00640 1 -- 172.21.15.6:0/3336278171 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda3015f210 con 0x7fda300d4a10 2024-03-04T09:32:21.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.769+0000 7fda35e00640 1 -- 172.21.15.6:0/3336278171 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda3015fd00 con 0x7fda300d4a10 2024-03-04T09:32:21.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.770+0000 7fda35e00640 1 -- 172.21.15.6:0/3336278171 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda3004f330 con 0x7fda300d4a10 2024-03-04T09:32:21.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.771+0000 7fda2cc00640 1 -- 172.21.15.6:0/3336278171 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fda1c037070 con 0x7fda300d4a10 2024-03-04T09:32:21.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.772+0000 7fda2cc00640 1 --2- 172.21.15.6:0/3336278171 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fda1006b4d0 0x7fda1006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:21.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.772+0000 7fda2cc00640 1 -- 172.21.15.6:0/3336278171 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5853+0+0 (secure 0 0 0) 0x7fda1c0ac200 con 0x7fda300d4a10 2024-03-04T09:32:21.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.773+0000 7fda34a00640 1 --2- 172.21.15.6:0/3336278171 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fda1006b4d0 0x7fda1006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:21.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.774+0000 7fda34a00640 1 --2- 172.21.15.6:0/3336278171 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fda1006b4d0 0x7fda1006d990 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fda24000f60 tx=0x7fda24009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:21.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:21.776+0000 7fda2cc00640 1 -- 172.21.15.6:0/3336278171 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fda1c07afe0 con 0x7fda300d4a10 2024-03-04T09:32:22.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:22.025+0000 7fda35e00640 1 -- 172.21.15.6:0/3336278171 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fda300b6400 con 0x7fda300d4a10 2024-03-04T09:32:22.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:22.026+0000 7fda2cc00640 1 -- 172.21.15.6:0/3336278171 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+4975 (secure 0 0 0) 0x7fda1c07a8e0 con 0x7fda300d4a10 2024-03-04T09:32:22.023 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:32:22.023 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":84,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6535,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/3883122012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":3883122012},{"type":"v1","addr":"172.21.15.146:6835","nonce":3883122012}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":6559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/4060411493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":4060411493},{"type":"v1","addr":"172.21.15.146:6837","nonce":4060411493}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":82},{"gid":7812,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/679471398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":679471398},{"type":"v1","addr":"172.21.15.6:6833","nonce":679471398}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:31:21.114351+0000","modified":"2024-03-04T09:31:40.781474+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7815},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7815":{"gid":7815,"name":"a","rank":0,"incarnation":81,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/882667161","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":882667161},{"type":"v1","addr":"172.21.15.6:6835","nonce":882667161}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7815,"qdb_cluster":[7815]},"id":8}]} 2024-03-04T09:32:22.023 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 84 2024-03-04T09:32:22.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:22.029+0000 7fda35e00640 1 -- 172.21.15.6:0/3336278171 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fda1006b4d0 msgr2=0x7fda1006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:22.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:22.029+0000 7fda35e00640 1 --2- 172.21.15.6:0/3336278171 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fda1006b4d0 0x7fda1006d990 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fda24000f60 tx=0x7fda24009040 comp rx=0 tx=0).stop 2024-03-04T09:32:22.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:22.029+0000 7fda35e00640 1 -- 172.21.15.6:0/3336278171 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fda300d4a10 msgr2=0x7fda300b5440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:22.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:22.029+0000 7fda35e00640 1 --2- 172.21.15.6:0/3336278171 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fda300d4a10 0x7fda300b5440 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7fda1c000c00 tx=0x7fda1c004730 comp rx=0 tx=0).stop 2024-03-04T09:32:22.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:22.029+0000 7fda35e00640 1 -- 172.21.15.6:0/3336278171 shutdown_connections 2024-03-04T09:32:22.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:22.029+0000 7fda35e00640 1 --2- 172.21.15.6:0/3336278171 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fda3015e0b0 0x7fda30157e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:22.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:22.029+0000 7fda35e00640 1 --2- 172.21.15.6:0/3336278171 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fda300d4a10 0x7fda300b5440 unknown :-1 s=CLOSED pgs=1359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:22.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:22.029+0000 7fda35e00640 1 --2- 172.21.15.6:0/3336278171 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fda1006b4d0 0x7fda1006d990 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:22.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:22.029+0000 7fda35e00640 1 --2- 172.21.15.6:0/3336278171 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fda300d0c80 0x7fda3015e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:22.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:22.029+0000 7fda35e00640 1 -- 172.21.15.6:0/3336278171 >> 172.21.15.6:0/3336278171 conn(0x7fda300d3d30 msgr2=0x7fda300b0b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:32:22.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:22.029+0000 7fda35e00640 1 -- 172.21.15.6:0/3336278171 shutdown_connections 2024-03-04T09:32:22.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:22.029+0000 7fda35e00640 1 -- 172.21.15.6:0/3336278171 wait complete. 2024-03-04T09:32:22.044 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache num_strays 2024-03-04T09:32:22.274 INFO:teuthology.orchestra.run.smithi006.stdout:{"mds_cache":{"num_strays":0}} 2024-03-04T09:32:22.275 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "num_strays": 0 } } 2024-03-04T09:32:22.275 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:32:22.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:22.491+0000 7fa3a7e00640 1 Processor -- start 2024-03-04T09:32:22.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:22.491+0000 7fa3a7e00640 1 -- start start 2024-03-04T09:32:22.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:22.493+0000 7fa3a7e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fa3a81477e0 0x7fa3a8162420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:22.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:22.493+0000 7fa3a7e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa3a806b6c0 0x7fa3a80d0910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:22.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:22.493+0000 7fa3a7e00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3a8147bc0 con 0x7fa3a806ef20 2024-03-04T09:32:22.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:22.493+0000 7fa3a7e00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3a8161f90 con 0x7fa3a81477e0 2024-03-04T09:32:22.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:22.493+0000 7fa3a7e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3a8162960 con 0x7fa3a806b6c0 2024-03-04T09:32:22.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:22.494+0000 7fa3a7400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa3a806b6c0 0x7fa3a80d0910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:22.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:22.494+0000 7fa3a7400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa3a806b6c0 0x7fa3a80d0910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.6:36386/0 (socket says 172.21.15.6:36386) 2024-03-04T09:32:22.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:22.494+0000 7fa3a7400640 1 -- 172.21.15.6:0/1372270323 learned_addr learned my addr 172.21.15.6:0/1372270323 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:32:22.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:22.494+0000 7fa3a6a00640 1 --2- 172.21.15.6:0/1372270323 >> v2:172.21.15.146:3300/0 conn(0x7fa3a81477e0 0x7fa3a8162420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:22.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:22.495+0000 7fa3a7400640 1 -- 172.21.15.6:0/1372270323 >> v2:172.21.15.146:3300/0 conn(0x7fa3a81477e0 msgr2=0x7fa3a8162420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:32:22.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:22.495+0000 7fa3a7400640 1 --2- 172.21.15.6:0/1372270323 >> v2:172.21.15.146:3300/0 conn(0x7fa3a81477e0 0x7fa3a8162420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:22.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:22.495+0000 7fa3a7400640 1 -- 172.21.15.6:0/1372270323 >> v1:172.21.15.146:6789/0 conn(0x7fa3a806ef20 legacy=0x7fa3a806afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:22.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:22.495+0000 7fa3a7400640 1 -- 172.21.15.6:0/1372270323 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3a80b3d70 con 0x7fa3a806b6c0 2024-03-04T09:32:22.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:22.496+0000 7fa3a7400640 1 --2- 172.21.15.6:0/1372270323 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa3a806b6c0 0x7fa3a80d0910 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fa3a0009870 tx=0x7fa3a002b740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4fcd0714fbe52386 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:22.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:22.497+0000 7fa3a6000640 1 -- 172.21.15.6:0/1372270323 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3a0037070 con 0x7fa3a806b6c0 2024-03-04T09:32:22.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:22.497+0000 7fa3a6000640 1 -- 172.21.15.6:0/1372270323 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3a0030710 con 0x7fa3a806b6c0 2024-03-04T09:32:22.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:22.497+0000 7fa3a6000640 1 -- 172.21.15.6:0/1372270323 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3a0030ae0 con 0x7fa3a806b6c0 2024-03-04T09:32:22.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:22.497+0000 7fa3a7e00640 1 -- 172.21.15.6:0/1372270323 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa3a806b6c0 msgr2=0x7fa3a80d0910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:22.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:22.497+0000 7fa3a7e00640 1 --2- 172.21.15.6:0/1372270323 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa3a806b6c0 0x7fa3a80d0910 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fa3a0009870 tx=0x7fa3a002b740 comp rx=0 tx=0).stop 2024-03-04T09:32:22.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:22.498+0000 7fa3a7e00640 1 -- 172.21.15.6:0/1372270323 shutdown_connections 2024-03-04T09:32:22.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:22.498+0000 7fa3a7e00640 1 --2- 172.21.15.6:0/1372270323 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa3a806b6c0 0x7fa3a80d0910 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:22.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:22.498+0000 7fa3a7e00640 1 --2- 172.21.15.6:0/1372270323 >> v2:172.21.15.146:3300/0 conn(0x7fa3a81477e0 0x7fa3a8162420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:22.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:22.498+0000 7fa3a7e00640 1 -- 172.21.15.6:0/1372270323 >> 172.21.15.6:0/1372270323 conn(0x7fa3a806c0c0 msgr2=0x7fa3a806c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:32:22.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:22.498+0000 7fa3a7e00640 1 -- 172.21.15.6:0/1372270323 shutdown_connections 2024-03-04T09:32:22.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:22.498+0000 7fa3a7e00640 1 -- 172.21.15.6:0/1372270323 wait complete. 2024-03-04T09:32:22.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:22.499+0000 7fa3a7e00640 1 Processor -- start 2024-03-04T09:32:22.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:22.500+0000 7fa3a7e00640 1 -- start start 2024-03-04T09:32:22.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:22.500+0000 7fa3a7e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa3a806b6c0 0x7fa3a813f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:22.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:22.500+0000 7fa3a7e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa3a806ef20 0x7fa3a8160730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:22.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:22.500+0000 7fa3a7e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa3a81477e0 0x7fa3a8160c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:22.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:22.501+0000 7fa3a7e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3a8161c20 con 0x7fa3a806ef20 2024-03-04T09:32:22.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:22.501+0000 7fa3a7e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3a80b4330 con 0x7fa3a81477e0 2024-03-04T09:32:22.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:22.501+0000 7fa3a7e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3a806f8f0 con 0x7fa3a806b6c0 2024-03-04T09:32:22.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:22.501+0000 7fa39e000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa3a806ef20 0x7fa3a8160730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:22.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:22.501+0000 7fa39e000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa3a806ef20 0x7fa3a8160730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:44020/0 (socket says 172.21.15.6:44020) 2024-03-04T09:32:22.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:22.501+0000 7fa39e000640 1 -- 172.21.15.6:0/4280029603 learned_addr learned my addr 172.21.15.6:0/4280029603 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:32:22.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:22.501+0000 7fa3a6a00640 1 --2- 172.21.15.6:0/4280029603 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa3a806b6c0 0x7fa3a813f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:22.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:22.501+0000 7fa3a7400640 1 --2- 172.21.15.6:0/4280029603 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa3a81477e0 0x7fa3a8160c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:22.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:22.501+0000 7fa39e000640 1 -- 172.21.15.6:0/4280029603 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa3a806b6c0 msgr2=0x7fa3a813f2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:22.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:22.501+0000 7fa39e000640 1 --2- 172.21.15.6:0/4280029603 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa3a806b6c0 0x7fa3a813f2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:22.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:22.501+0000 7fa39e000640 1 -- 172.21.15.6:0/4280029603 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa3a81477e0 msgr2=0x7fa3a8160c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:22.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:22.501+0000 7fa39e000640 1 --2- 172.21.15.6:0/4280029603 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa3a81477e0 0x7fa3a8160c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:22.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:22.501+0000 7fa39e000640 1 -- 172.21.15.6:0/4280029603 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3a81505b0 con 0x7fa3a806ef20 2024-03-04T09:32:22.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:22.502+0000 7fa39e000640 1 --2- 172.21.15.6:0/4280029603 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa3a806ef20 0x7fa3a8160730 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7fa39000c780 tx=0x7fa39000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:22.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:22.502+0000 7fa39fe00640 1 -- 172.21.15.6:0/4280029603 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa390017070 con 0x7fa3a806ef20 2024-03-04T09:32:22.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:22.502+0000 7fa39fe00640 1 -- 172.21.15.6:0/4280029603 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa390005290 con 0x7fa3a806ef20 2024-03-04T09:32:22.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:22.502+0000 7fa39fe00640 1 -- 172.21.15.6:0/4280029603 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa39000a880 con 0x7fa3a806ef20 2024-03-04T09:32:22.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:22.502+0000 7fa3a7e00640 1 -- 172.21.15.6:0/4280029603 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3a8150890 con 0x7fa3a806ef20 2024-03-04T09:32:22.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:22.502+0000 7fa3a7e00640 1 -- 172.21.15.6:0/4280029603 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3a8150e50 con 0x7fa3a806ef20 2024-03-04T09:32:22.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:22.504+0000 7fa3a7e00640 1 -- 172.21.15.6:0/4280029603 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3a8162420 con 0x7fa3a806ef20 2024-03-04T09:32:22.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:22.506+0000 7fa39fe00640 1 -- 172.21.15.6:0/4280029603 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa39001a070 con 0x7fa3a806ef20 2024-03-04T09:32:22.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:22.506+0000 7fa39fe00640 1 --2- 172.21.15.6:0/4280029603 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa38406b4d0 0x7fa38406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:22.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:22.507+0000 7fa39fe00640 1 -- 172.21.15.6:0/4280029603 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5853+0+0 (secure 0 0 0) 0x7fa390089850 con 0x7fa3a806ef20 2024-03-04T09:32:22.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:22.507+0000 7fa3a6a00640 1 --2- 172.21.15.6:0/4280029603 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa38406b4d0 0x7fa38406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:22.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:22.508+0000 7fa3a6a00640 1 --2- 172.21.15.6:0/4280029603 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa38406b4d0 0x7fa38406d990 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fa3980073f0 tx=0x7fa39800b040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:22.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:22.511+0000 7fa39fe00640 1 -- 172.21.15.6:0/4280029603 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa3a8162420 con 0x7fa3a806ef20 2024-03-04T09:32:22.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:22.749+0000 7fa3a7e00640 1 -- 172.21.15.6:0/4280029603 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa3a815e180 con 0x7fa3a806ef20 2024-03-04T09:32:22.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:22.750+0000 7fa39fe00640 1 -- 172.21.15.6:0/4280029603 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+4975 (secure 0 0 0) 0x7fa3900585a0 con 0x7fa3a806ef20 2024-03-04T09:32:22.748 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:32:22.748 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":84,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6535,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/3883122012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":3883122012},{"type":"v1","addr":"172.21.15.146:6835","nonce":3883122012}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":6559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/4060411493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":4060411493},{"type":"v1","addr":"172.21.15.146:6837","nonce":4060411493}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":82},{"gid":7812,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/679471398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":679471398},{"type":"v1","addr":"172.21.15.6:6833","nonce":679471398}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:31:21.114351+0000","modified":"2024-03-04T09:31:40.781474+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7815},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7815":{"gid":7815,"name":"a","rank":0,"incarnation":81,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/882667161","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":882667161},{"type":"v1","addr":"172.21.15.6:6835","nonce":882667161}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7815,"qdb_cluster":[7815]},"id":8}]} 2024-03-04T09:32:22.749 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 84 2024-03-04T09:32:22.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:22.753+0000 7fa3a7e00640 1 -- 172.21.15.6:0/4280029603 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa38406b4d0 msgr2=0x7fa38406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:22.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:22.753+0000 7fa3a7e00640 1 --2- 172.21.15.6:0/4280029603 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa38406b4d0 0x7fa38406d990 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fa3980073f0 tx=0x7fa39800b040 comp rx=0 tx=0).stop 2024-03-04T09:32:22.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:22.753+0000 7fa3a7e00640 1 -- 172.21.15.6:0/4280029603 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa3a806ef20 msgr2=0x7fa3a8160730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:22.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:22.753+0000 7fa3a7e00640 1 --2- 172.21.15.6:0/4280029603 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa3a806ef20 0x7fa3a8160730 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7fa39000c780 tx=0x7fa39000cc40 comp rx=0 tx=0).stop 2024-03-04T09:32:22.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:22.754+0000 7fa3a7e00640 1 -- 172.21.15.6:0/4280029603 shutdown_connections 2024-03-04T09:32:22.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:22.754+0000 7fa3a7e00640 1 --2- 172.21.15.6:0/4280029603 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa3a81477e0 0x7fa3a8160c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:22.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:22.754+0000 7fa3a7e00640 1 --2- 172.21.15.6:0/4280029603 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa3a806ef20 0x7fa3a8160730 unknown :-1 s=CLOSED pgs=1360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:22.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:22.754+0000 7fa3a7e00640 1 --2- 172.21.15.6:0/4280029603 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa38406b4d0 0x7fa38406d990 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:22.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:22.754+0000 7fa3a7e00640 1 --2- 172.21.15.6:0/4280029603 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa3a806b6c0 0x7fa3a813f2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:22.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:22.754+0000 7fa3a7e00640 1 -- 172.21.15.6:0/4280029603 >> 172.21.15.6:0/4280029603 conn(0x7fa3a806c0c0 msgr2=0x7fa3a80b0750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:32:22.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:22.754+0000 7fa3a7e00640 1 -- 172.21.15.6:0/4280029603 shutdown_connections 2024-03-04T09:32:22.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:22.754+0000 7fa3a7e00640 1 -- 172.21.15.6:0/4280029603 wait complete. 2024-03-04T09:32:22.771 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_reintegrated 2024-03-04T09:32:23.008 INFO:teuthology.orchestra.run.smithi006.stdout:{"mds_cache":{"strays_reintegrated":2}} 2024-03-04T09:32:23.008 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_reintegrated": 2 } } 2024-03-04T09:32:23.008 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:32:23.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.224+0000 7fa005800640 1 Processor -- start 2024-03-04T09:32:23.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.225+0000 7fa005800640 1 -- start start 2024-03-04T09:32:23.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.226+0000 7fa005800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fa0000b34e0 0x7fa00015c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:23.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.226+0000 7fa005800640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa00015cc40 con 0x7fa0000af1a0 2024-03-04T09:32:23.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.227+0000 7fa005800640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa00015cf20 con 0x7fa0000af580 2024-03-04T09:32:23.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.227+0000 7fa005800640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa00015d200 con 0x7fa0000b34e0 2024-03-04T09:32:23.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.227+0000 7f9ffea00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fa0000af1a0 0x7fa0000b4c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:56282/0 (socket says 172.21.15.6:56282) 2024-03-04T09:32:23.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.227+0000 7f9ffea00640 1 -- 172.21.15.6:0/1142355622 learned_addr learned my addr 172.21.15.6:0/1142355622 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:32:23.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.227+0000 7f9ffd600640 1 -- 172.21.15.6:0/1142355622 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1044252939 0 0) 0x7fa00015cc40 con 0x7fa0000af1a0 2024-03-04T09:32:23.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.227+0000 7f9ffd600640 1 -- 172.21.15.6:0/1142355622 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9fe4003520 con 0x7fa0000af1a0 2024-03-04T09:32:23.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.227+0000 7f9fff400640 1 --2- 172.21.15.6:0/1142355622 >> v2:172.21.15.146:3300/0 conn(0x7fa0000b34e0 0x7fa00015c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:23.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.228+0000 7f9ffd600640 1 -- 172.21.15.6:0/1142355622 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 534257766 0 0) 0x7f9fe4003520 con 0x7fa0000af1a0 2024-03-04T09:32:23.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.228+0000 7f9ffd600640 1 -- 172.21.15.6:0/1142355622 >> v2:172.21.15.146:3300/0 conn(0x7fa0000b34e0 msgr2=0x7fa00015c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:32:23.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.228+0000 7f9ffd600640 1 --2- 172.21.15.6:0/1142355622 >> v2:172.21.15.146:3300/0 conn(0x7fa0000b34e0 0x7fa00015c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:23.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.228+0000 7f9ffd600640 1 -- 172.21.15.6:0/1142355622 >> v1:172.21.15.146:6789/0 conn(0x7fa0000af580 legacy=0x7fa0000b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:32:23.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.228+0000 7f9ffd600640 1 -- 172.21.15.6:0/1142355622 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa00015d4e0 con 0x7fa0000af1a0 2024-03-04T09:32:23.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.228+0000 7f9ffd600640 1 -- 172.21.15.6:0/1142355622 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7f9ff0001670 con 0x7fa0000af1a0 2024-03-04T09:32:23.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.228+0000 7f9ffd600640 1 -- 172.21.15.6:0/1142355622 >> v1:172.21.15.6:6789/0 conn(0x7fa0000af1a0 legacy=0x7fa0000b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:23.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.228+0000 7f9ffd600640 1 --2- 172.21.15.6:0/1142355622 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9fe4003a20 0x7f9fe4003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:23.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.228+0000 7f9ffd600640 1 --2- 172.21.15.6:0/1142355622 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9fe4004800 0x7f9fe4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:23.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.228+0000 7f9ffd600640 1 --2- 172.21.15.6:0/1142355622 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9fe40052e0 0x7f9fe400fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:23.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.228+0000 7f9ffd600640 1 -- 172.21.15.6:0/1142355622 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa00015d200 con 0x7f9fe40052e0 2024-03-04T09:32:23.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.228+0000 7f9ffd600640 1 -- 172.21.15.6:0/1142355622 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fe4010610 con 0x7f9fe4003a20 2024-03-04T09:32:23.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.228+0000 7f9ffd600640 1 -- 172.21.15.6:0/1142355622 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fe4010810 con 0x7f9fe4004800 2024-03-04T09:32:23.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.228+0000 7f9fff400640 1 --2- 172.21.15.6:0/1142355622 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9fe40052e0 0x7f9fe400fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:23.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.228+0000 7f9ffe000640 1 --2- 172.21.15.6:0/1142355622 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9fe4004800 0x7f9fe4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:23.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.228+0000 7f9ffea00640 1 --2- 172.21.15.6:0/1142355622 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9fe4003a20 0x7f9fe4003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:23.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.229+0000 7f9fff400640 1 -- 172.21.15.6:0/1142355622 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9fe4004800 msgr2=0x7f9fe4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:32:23.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.229+0000 7f9fff400640 1 --2- 172.21.15.6:0/1142355622 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9fe4004800 0x7f9fe4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:23.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.229+0000 7f9fff400640 1 -- 172.21.15.6:0/1142355622 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9fe4003a20 msgr2=0x7f9fe4003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:32:23.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.229+0000 7f9fff400640 1 --2- 172.21.15.6:0/1142355622 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9fe4003a20 0x7f9fe4003e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:23.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.229+0000 7f9fff400640 1 -- 172.21.15.6:0/1142355622 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fe4003520 con 0x7f9fe40052e0 2024-03-04T09:32:23.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.230+0000 7f9fff400640 1 -- 172.21.15.6:0/1142355622 reap_dead start 2024-03-04T09:32:23.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.230+0000 7f9fff400640 1 --2- 172.21.15.6:0/1142355622 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9fe40052e0 0x7f9fe400fdd0 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7f9ff8007330 tx=0x7f9fe4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df03c3ad643bcde6 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:23.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.230+0000 7f9ffd600640 1 -- 172.21.15.6:0/1142355622 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ff8039070 con 0x7f9fe40052e0 2024-03-04T09:32:23.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.230+0000 7f9ffd600640 1 -- 172.21.15.6:0/1142355622 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ff802ccd0 con 0x7f9fe40052e0 2024-03-04T09:32:23.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.230+0000 7f9ffd600640 1 -- 172.21.15.6:0/1142355622 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ff8009be0 con 0x7f9fe40052e0 2024-03-04T09:32:23.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.231+0000 7fa005800640 1 -- 172.21.15.6:0/1142355622 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9fe40052e0 msgr2=0x7f9fe400fdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:23.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.231+0000 7fa005800640 1 --2- 172.21.15.6:0/1142355622 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9fe40052e0 0x7f9fe400fdd0 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7f9ff8007330 tx=0x7f9fe4005830 comp rx=0 tx=0).stop 2024-03-04T09:32:23.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.231+0000 7fa005800640 1 -- 172.21.15.6:0/1142355622 shutdown_connections 2024-03-04T09:32:23.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.231+0000 7fa005800640 1 --2- 172.21.15.6:0/1142355622 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9fe40052e0 0x7f9fe400fdd0 unknown :-1 s=CLOSED pgs=1362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:23.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.231+0000 7fa005800640 1 -- 172.21.15.6:0/1142355622 >> 172.21.15.6:0/1142355622 conn(0x7fa00006afb0 msgr2=0x7fa00006b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:32:23.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.231+0000 7fa005800640 1 -- 172.21.15.6:0/1142355622 shutdown_connections 2024-03-04T09:32:23.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.231+0000 7fa005800640 1 -- 172.21.15.6:0/1142355622 wait complete. 2024-03-04T09:32:23.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.233+0000 7fa005800640 1 Processor -- start 2024-03-04T09:32:23.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.233+0000 7fa005800640 1 -- start start 2024-03-04T09:32:23.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.234+0000 7fa005800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9fe40052e0 0x7fa000147630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:23.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.234+0000 7fa005800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa000155ac0 0x7fa000155f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:23.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.234+0000 7fa005800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa000153a70 0x7fa000153ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:23.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.234+0000 7fa005800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa00015c600 con 0x7fa000153a70 2024-03-04T09:32:23.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.234+0000 7fa005800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0000cde30 con 0x7f9fe40052e0 2024-03-04T09:32:23.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.234+0000 7fa005800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa00015c780 con 0x7fa000155ac0 2024-03-04T09:32:23.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.234+0000 7f9fff400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa000153a70 0x7fa000153ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:23.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.234+0000 7f9fff400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa000153a70 0x7fa000153ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:44038/0 (socket says 172.21.15.6:44038) 2024-03-04T09:32:23.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.234+0000 7f9fff400640 1 -- 172.21.15.6:0/668741321 learned_addr learned my addr 172.21.15.6:0/668741321 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:32:23.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.234+0000 7f9ffea00640 1 --2- 172.21.15.6:0/668741321 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9fe40052e0 0x7fa000147630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:23.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.234+0000 7f9fff400640 1 -- 172.21.15.6:0/668741321 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa000155ac0 msgr2=0x7fa000155f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:23.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.234+0000 7f9fff400640 1 --2- 172.21.15.6:0/668741321 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa000155ac0 0x7fa000155f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:23.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.234+0000 7f9fff400640 1 -- 172.21.15.6:0/668741321 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9fe40052e0 msgr2=0x7fa000147630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:23.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.234+0000 7f9fff400640 1 --2- 172.21.15.6:0/668741321 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9fe40052e0 0x7fa000147630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:23.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.234+0000 7f9fff400640 1 -- 172.21.15.6:0/668741321 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa000147b70 con 0x7fa000153a70 2024-03-04T09:32:23.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.235+0000 7f9fff400640 1 --2- 172.21.15.6:0/668741321 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa000153a70 0x7fa000153ef0 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7f9ff8035d50 tx=0x7f9ff8004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:23.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.235+0000 7f9ff6a00640 1 -- 172.21.15.6:0/668741321 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ff8039040 con 0x7fa000153a70 2024-03-04T09:32:23.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.235+0000 7f9ff6a00640 1 -- 172.21.15.6:0/668741321 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ff8045070 con 0x7fa000153a70 2024-03-04T09:32:23.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.235+0000 7f9ff6a00640 1 -- 172.21.15.6:0/668741321 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ff8009be0 con 0x7fa000153a70 2024-03-04T09:32:23.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.235+0000 7fa005800640 1 -- 172.21.15.6:0/668741321 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0001547e0 con 0x7fa000153a70 2024-03-04T09:32:23.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.235+0000 7fa005800640 1 -- 172.21.15.6:0/668741321 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa000154da0 con 0x7fa000153a70 2024-03-04T09:32:23.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.237+0000 7fa005800640 1 -- 172.21.15.6:0/668741321 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa000156460 con 0x7fa000153a70 2024-03-04T09:32:23.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.237+0000 7f9ff6a00640 1 -- 172.21.15.6:0/668741321 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9ff803f640 con 0x7fa000153a70 2024-03-04T09:32:23.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.238+0000 7f9ff6a00640 1 --2- 172.21.15.6:0/668741321 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f9fe006b520 0x7f9fe006d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:23.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.238+0000 7f9ff6a00640 1 -- 172.21.15.6:0/668741321 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f9ff80ac620 con 0x7fa000153a70 2024-03-04T09:32:23.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.239+0000 7f9ffea00640 1 --2- 172.21.15.6:0/668741321 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f9fe006b520 0x7f9fe006d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:23.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.240+0000 7f9ffea00640 1 --2- 172.21.15.6:0/668741321 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f9fe006b520 0x7f9fe006d9e0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f9ff0000f60 tx=0x7f9ff0009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:23.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.243+0000 7f9ff6a00640 1 -- 172.21.15.6:0/668741321 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9ff807add0 con 0x7fa000153a70 2024-03-04T09:32:23.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.480+0000 7fa005800640 1 -- 172.21.15.6:0/668741321 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa000069510 con 0x7fa000153a70 2024-03-04T09:32:23.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.481+0000 7f9ff6a00640 1 -- 172.21.15.6:0/668741321 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+4975 (secure 0 0 0) 0x7f9ff807ee70 con 0x7fa000153a70 2024-03-04T09:32:23.478 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:32:23.478 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":84,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6535,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/3883122012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":3883122012},{"type":"v1","addr":"172.21.15.146:6835","nonce":3883122012}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":6559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/4060411493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":4060411493},{"type":"v1","addr":"172.21.15.146:6837","nonce":4060411493}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":82},{"gid":7812,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/679471398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":679471398},{"type":"v1","addr":"172.21.15.6:6833","nonce":679471398}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:31:21.114351+0000","modified":"2024-03-04T09:31:40.781474+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7815},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7815":{"gid":7815,"name":"a","rank":0,"incarnation":81,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/882667161","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":882667161},{"type":"v1","addr":"172.21.15.6:6835","nonce":882667161}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7815,"qdb_cluster":[7815]},"id":8}]} 2024-03-04T09:32:23.478 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 84 2024-03-04T09:32:23.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.483+0000 7fa005800640 1 -- 172.21.15.6:0/668741321 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f9fe006b520 msgr2=0x7f9fe006d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:23.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.483+0000 7fa005800640 1 --2- 172.21.15.6:0/668741321 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f9fe006b520 0x7f9fe006d9e0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f9ff0000f60 tx=0x7f9ff0009040 comp rx=0 tx=0).stop 2024-03-04T09:32:23.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.483+0000 7fa005800640 1 -- 172.21.15.6:0/668741321 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa000153a70 msgr2=0x7fa000153ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:23.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.483+0000 7fa005800640 1 --2- 172.21.15.6:0/668741321 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa000153a70 0x7fa000153ef0 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7f9ff8035d50 tx=0x7f9ff8004ab0 comp rx=0 tx=0).stop 2024-03-04T09:32:23.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.483+0000 7fa005800640 1 -- 172.21.15.6:0/668741321 shutdown_connections 2024-03-04T09:32:23.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.483+0000 7fa005800640 1 --2- 172.21.15.6:0/668741321 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa000153a70 0x7fa000153ef0 unknown :-1 s=CLOSED pgs=1363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:23.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.483+0000 7fa005800640 1 --2- 172.21.15.6:0/668741321 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f9fe006b520 0x7f9fe006d9e0 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:23.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.483+0000 7fa005800640 1 --2- 172.21.15.6:0/668741321 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa000155ac0 0x7fa000155f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:23.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.483+0000 7fa005800640 1 --2- 172.21.15.6:0/668741321 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9fe40052e0 0x7fa000147630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:23.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.483+0000 7fa005800640 1 -- 172.21.15.6:0/668741321 >> 172.21.15.6:0/668741321 conn(0x7fa00006afb0 msgr2=0x7fa0000ce850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:32:23.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.484+0000 7fa005800640 1 -- 172.21.15.6:0/668741321 shutdown_connections 2024-03-04T09:32:23.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.484+0000 7fa005800640 1 -- 172.21.15.6:0/668741321 wait complete. 2024-03-04T09:32:23.500 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json flush journal 2024-03-04T09:32:23.765 INFO:teuthology.orchestra.run.smithi006.stdout:{"message":"","return_code":0} 2024-03-04T09:32:23.766 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-04T09:32:23.766 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 10000000001.00000000 parent 2024-03-04T09:32:23.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.875+0000 7f8dc4e90f40 1 Processor -- start 2024-03-04T09:32:23.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.875+0000 7f8dc4e90f40 1 -- start start 2024-03-04T09:32:23.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.876+0000 7f8dc4e90f40 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55ef5ac31660 0x55ef5ac2d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:23.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.877+0000 7f8dc4e90f40 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55ef5ac5f540 0x55ef5ac31e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:23.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.877+0000 7f8dc4e90f40 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55ef5acfe830 0x55ef5ac30af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:23.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.877+0000 7f8dc4e90f40 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ef5abd5fd0 con 0x55ef5ac31660 2024-03-04T09:32:23.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.877+0000 7f8dc4e90f40 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ef5abd9f30 con 0x55ef5ac5f540 2024-03-04T09:32:23.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.877+0000 7f8dc4e90f40 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ef5ac75d10 con 0x55ef5acfe830 2024-03-04T09:32:23.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.877+0000 7f8dc2000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55ef5ac31660 0x55ef5ac2d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:23.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.877+0000 7f8dc2000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55ef5ac31660 0x55ef5ac2d820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:44052/0 (socket says 172.21.15.6:44052) 2024-03-04T09:32:23.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.877+0000 7f8dc2000640 1 -- 172.21.15.6:0/850076491 learned_addr learned my addr 172.21.15.6:0/850076491 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:32:23.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.877+0000 7f8dc1600640 1 --2- 172.21.15.6:0/850076491 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55ef5ac5f540 0x55ef5ac31e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:23.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.877+0000 7f8dc2000640 1 -- 172.21.15.6:0/850076491 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55ef5acfe830 msgr2=0x55ef5ac30af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:32:23.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.877+0000 7f8dc2000640 1 --2- 172.21.15.6:0/850076491 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55ef5acfe830 0x55ef5ac30af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:23.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.877+0000 7f8dc2000640 1 -- 172.21.15.6:0/850076491 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55ef5ac5f540 msgr2=0x55ef5ac31e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:32:23.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.877+0000 7f8dc2000640 1 --2- 172.21.15.6:0/850076491 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55ef5ac5f540 0x55ef5ac31e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:23.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.877+0000 7f8dc2000640 1 -- 172.21.15.6:0/850076491 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ef5ad03470 con 0x55ef5ac31660 2024-03-04T09:32:23.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.878+0000 7f8dc2000640 1 --2- 172.21.15.6:0/850076491 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55ef5ac31660 0x55ef5ac2d820 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7f8db400a460 tx=0x7f8db400a920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbfa15f44f89442f server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:23.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.878+0000 7f8dbbe00640 1 -- 172.21.15.6:0/850076491 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8db400b920 con 0x55ef5ac31660 2024-03-04T09:32:23.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.878+0000 7f8dbbe00640 1 -- 172.21.15.6:0/850076491 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8db400f070 con 0x55ef5ac31660 2024-03-04T09:32:23.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.878+0000 7f8dbbe00640 1 -- 172.21.15.6:0/850076491 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8db400bd20 con 0x55ef5ac31660 2024-03-04T09:32:23.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.879+0000 7f8dc4e90f40 1 -- 172.21.15.6:0/850076491 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55ef5ac31660 msgr2=0x55ef5ac2d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:23.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.879+0000 7f8dc4e90f40 1 --2- 172.21.15.6:0/850076491 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55ef5ac31660 0x55ef5ac2d820 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7f8db400a460 tx=0x7f8db400a920 comp rx=0 tx=0).stop 2024-03-04T09:32:23.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.879+0000 7f8dc4e90f40 1 -- 172.21.15.6:0/850076491 shutdown_connections 2024-03-04T09:32:23.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.879+0000 7f8dc4e90f40 1 --2- 172.21.15.6:0/850076491 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55ef5acfe830 0x55ef5ac30af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:23.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.879+0000 7f8dc4e90f40 1 --2- 172.21.15.6:0/850076491 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55ef5ac5f540 0x55ef5ac31e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:23.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.879+0000 7f8dc4e90f40 1 --2- 172.21.15.6:0/850076491 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55ef5ac31660 0x55ef5ac2d820 unknown :-1 s=CLOSED pgs=1366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:23.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.879+0000 7f8dc4e90f40 1 -- 172.21.15.6:0/850076491 >> 172.21.15.6:0/850076491 conn(0x55ef5ac2f5c0 msgr2=0x55ef5ad03ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:32:23.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.879+0000 7f8dc4e90f40 1 -- 172.21.15.6:0/850076491 shutdown_connections 2024-03-04T09:32:23.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.879+0000 7f8dc4e90f40 1 -- 172.21.15.6:0/850076491 wait complete. 2024-03-04T09:32:23.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.880+0000 7f8dc4e90f40 1 Processor -- start 2024-03-04T09:32:23.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.881+0000 7f8dc4e90f40 1 -- start start 2024-03-04T09:32:23.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.882+0000 7f8dc4e90f40 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55ef5ac31660 0x55ef5ac74780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:23.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.882+0000 7f8dc4e90f40 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55ef5ac5f540 0x55ef5aceb850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:23.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.882+0000 7f8dc4e90f40 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55ef5acfe830 0x55ef5ac5dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:23.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.882+0000 7f8dc4e90f40 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ef5ac2d390 con 0x55ef5ac5f540 2024-03-04T09:32:23.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.882+0000 7f8dc4e90f40 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ef5ad03710 con 0x55ef5acfe830 2024-03-04T09:32:23.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.882+0000 7f8dc4e90f40 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ef5ac74cc0 con 0x55ef5ac31660 2024-03-04T09:32:23.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.882+0000 7f8dc1600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55ef5ac5f540 0x55ef5aceb850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:23.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.882+0000 7f8dc1600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55ef5ac5f540 0x55ef5aceb850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:44068/0 (socket says 172.21.15.6:44068) 2024-03-04T09:32:23.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.882+0000 7f8dc1600640 1 -- 172.21.15.6:0/2875150603 learned_addr learned my addr 172.21.15.6:0/2875150603 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:32:23.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.883+0000 7f8dc2000640 1 --2- 172.21.15.6:0/2875150603 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55ef5ac31660 0x55ef5ac74780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:23.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.883+0000 7f8dc2a00640 1 --2- 172.21.15.6:0/2875150603 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55ef5acfe830 0x55ef5ac5dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:23.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.883+0000 7f8dc1600640 1 -- 172.21.15.6:0/2875150603 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55ef5ac31660 msgr2=0x55ef5ac74780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:23.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.883+0000 7f8dc1600640 1 --2- 172.21.15.6:0/2875150603 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55ef5ac31660 0x55ef5ac74780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:23.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.883+0000 7f8dc1600640 1 -- 172.21.15.6:0/2875150603 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55ef5acfe830 msgr2=0x55ef5ac5dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:23.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.883+0000 7f8dc1600640 1 --2- 172.21.15.6:0/2875150603 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55ef5acfe830 0x55ef5ac5dfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:23.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.883+0000 7f8dc1600640 1 -- 172.21.15.6:0/2875150603 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ef5acf2e30 con 0x55ef5ac5f540 2024-03-04T09:32:23.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.883+0000 7f8dc1600640 1 --2- 172.21.15.6:0/2875150603 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55ef5ac5f540 0x55ef5aceb850 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7f8db0001e60 tx=0x7f8db000a5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:23.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.884+0000 7f8dbaa00640 1 -- 172.21.15.6:0/2875150603 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8db000f620 con 0x55ef5ac5f540 2024-03-04T09:32:23.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.884+0000 7f8dbaa00640 1 -- 172.21.15.6:0/2875150603 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8db0006070 con 0x55ef5ac5f540 2024-03-04T09:32:23.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.884+0000 7f8dbaa00640 1 -- 172.21.15.6:0/2875150603 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8db000f9e0 con 0x55ef5ac5f540 2024-03-04T09:32:23.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.884+0000 7f8dc4e90f40 1 -- 172.21.15.6:0/2875150603 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55ef5acf3110 con 0x55ef5ac5f540 2024-03-04T09:32:23.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.884+0000 7f8dc4e90f40 1 -- 172.21.15.6:0/2875150603 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ef5acebd90 con 0x55ef5ac5f540 2024-03-04T09:32:23.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.886+0000 7f8dbaa00640 1 -- 172.21.15.6:0/2875150603 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8db000fb80 con 0x55ef5ac5f540 2024-03-04T09:32:23.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.887+0000 7f8dbaa00640 1 --2- 172.21.15.6:0/2875150603 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f8da4069eb0 0x7f8da406c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:23.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.887+0000 7f8dbaa00640 1 -- 172.21.15.6:0/2875150603 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f8db0089220 con 0x55ef5ac5f540 2024-03-04T09:32:23.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.887+0000 7f8dc4e90f40 1 --2- 172.21.15.6:0/2875150603 >> [v2:172.21.15.6:6823/1364449401,v1:172.21.15.6:6825/1364449401] conn(0x55ef5acef5b0 0x55ef5ad00400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:23.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.887+0000 7f8dc4e90f40 1 -- 172.21.15.6:0/2875150603 --> [v2:172.21.15.6:6823/1364449401,v1:172.21.15.6:6825/1364449401] -- osd_op(unknown.0.0:1 17.7 17:e2e07930:::10000000001.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e115) v9 -- 0x55ef5ad00940 con 0x55ef5acef5b0 2024-03-04T09:32:23.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.887+0000 7f8dc2a00640 1 --2- 172.21.15.6:0/2875150603 >> [v2:172.21.15.6:6823/1364449401,v1:172.21.15.6:6825/1364449401] conn(0x55ef5acef5b0 0x55ef5ad00400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:23.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.887+0000 7f8dc2000640 1 --2- 172.21.15.6:0/2875150603 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f8da4069eb0 0x7f8da406c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:23.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.888+0000 7f8dc2a00640 1 --2- 172.21.15.6:0/2875150603 >> [v2:172.21.15.6:6823/1364449401,v1:172.21.15.6:6825/1364449401] conn(0x55ef5acef5b0 0x55ef5ad00400 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:23.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.889+0000 7f8dc2a00640 1 -- 172.21.15.6:0/2875150603 <== osd.0 v2:172.21.15.6:6823/1364449401 1 ==== osd_op_reply(1 10000000001.00000000 [getxattr (93) out=93b] v0'0 uv11 ondisk = 0) v8 ==== 164+0+93 (crc 0 0 0) 0x7f8dbc0525a0 con 0x55ef5acef5b0 2024-03-04T09:32:23.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.889+0000 7f8dc2000640 1 --2- 172.21.15.6:0/2875150603 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f8da4069eb0 0x7f8da406c370 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f8db400b010 tx=0x7f8db4002000 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:23.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.889+0000 7f8dc4e90f40 1 -- 172.21.15.6:0/2875150603 >> [v2:172.21.15.6:6823/1364449401,v1:172.21.15.6:6825/1364449401] conn(0x55ef5acef5b0 msgr2=0x55ef5ad00400 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:23.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.889+0000 7f8dc4e90f40 1 --2- 172.21.15.6:0/2875150603 >> [v2:172.21.15.6:6823/1364449401,v1:172.21.15.6:6825/1364449401] conn(0x55ef5acef5b0 0x55ef5ad00400 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:23.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.889+0000 7f8dc4e90f40 1 -- 172.21.15.6:0/2875150603 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f8da4069eb0 msgr2=0x7f8da406c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:23.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.889+0000 7f8dc4e90f40 1 --2- 172.21.15.6:0/2875150603 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f8da4069eb0 0x7f8da406c370 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f8db400b010 tx=0x7f8db4002000 comp rx=0 tx=0).stop 2024-03-04T09:32:23.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.889+0000 7f8dc4e90f40 1 -- 172.21.15.6:0/2875150603 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55ef5ac5f540 msgr2=0x55ef5aceb850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:23.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.889+0000 7f8dc4e90f40 1 --2- 172.21.15.6:0/2875150603 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55ef5ac5f540 0x55ef5aceb850 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7f8db0001e60 tx=0x7f8db000a5f0 comp rx=0 tx=0).stop 2024-03-04T09:32:23.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.889+0000 7f8dc2a00640 1 -- 172.21.15.6:0/2875150603 reap_dead start 2024-03-04T09:32:23.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.889+0000 7f8dc4e90f40 1 -- 172.21.15.6:0/2875150603 shutdown_connections 2024-03-04T09:32:23.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.889+0000 7f8dc4e90f40 1 -- 172.21.15.6:0/2875150603 >> 172.21.15.6:0/2875150603 conn(0x55ef5ac2f5c0 msgr2=0x55ef5ac75290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:32:23.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.889+0000 7f8dc4e90f40 1 -- 172.21.15.6:0/2875150603 shutdown_connections 2024-03-04T09:32:23.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:23.889+0000 7f8dc4e90f40 1 -- 172.21.15.6:0/2875150603 wait complete. 2024-03-04T09:32:23.891 INFO:teuthology.orchestra.run.smithi006.stdout:W�file_ddir_2. 2024-03-04T09:32:23.891 DEBUG:teuthology.orchestra.run.smithi006:> ceph-dencoder type inode_backtrace_t import - decode dump_json 2024-03-04T09:32:23.985 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-04T09:32:23.985 INFO:teuthology.orchestra.run.smithi006.stdout: "ino": 1099511627777, 2024-03-04T09:32:23.985 INFO:teuthology.orchestra.run.smithi006.stdout: "ancestors": [ 2024-03-04T09:32:23.985 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:32:23.985 INFO:teuthology.orchestra.run.smithi006.stdout: "dirino": 1099511628278, 2024-03-04T09:32:23.985 INFO:teuthology.orchestra.run.smithi006.stdout: "dname": "file_d", 2024-03-04T09:32:23.985 INFO:teuthology.orchestra.run.smithi006.stdout: "version": 27 2024-03-04T09:32:23.985 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:32:23.986 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:32:23.986 INFO:teuthology.orchestra.run.smithi006.stdout: "dirino": 1, 2024-03-04T09:32:23.986 INFO:teuthology.orchestra.run.smithi006.stdout: "dname": "dir_2", 2024-03-04T09:32:23.986 INFO:teuthology.orchestra.run.smithi006.stdout: "version": 46 2024-03-04T09:32:23.986 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:32:23.986 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-03-04T09:32:23.986 INFO:teuthology.orchestra.run.smithi006.stdout: "pool": 17, 2024-03-04T09:32:23.986 INFO:teuthology.orchestra.run.smithi006.stdout: "old_pools": [] 2024-03-04T09:32:23.986 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-04T09:32:23.986 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:32:23.990 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:32:23.990 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest/mnt.0 && exec rm -f dir_2/file_d) 2024-03-04T09:32:24.040 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:32:24.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:24.246+0000 7f54c6e00640 1 Processor -- start 2024-03-04T09:32:24.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:24.246+0000 7f54c6e00640 1 -- start start 2024-03-04T09:32:24.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:24.248+0000 7f54c6e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f54c00ae5f0 0x7f54c00d0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:24.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:24.248+0000 7f54c6e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f54c00d1110 0x7f54c006e160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:24.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:24.248+0000 7f54c6e00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54c015b110 con 0x7f54c015ad30 2024-03-04T09:32:24.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:24.248+0000 7f54c6e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54c00d1560 con 0x7f54c00ae5f0 2024-03-04T09:32:24.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:24.248+0000 7f54c6e00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54c00aed60 con 0x7f54c00d1110 2024-03-04T09:32:24.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:24.248+0000 7f54bf400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f54c00ae5f0 0x7f54c00d0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:24.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:24.248+0000 7f54bf400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f54c00ae5f0 0x7f54c00d0bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:44070/0 (socket says 172.21.15.6:44070) 2024-03-04T09:32:24.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:24.248+0000 7f54bf400640 1 -- 172.21.15.6:0/3830887558 learned_addr learned my addr 172.21.15.6:0/3830887558 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:32:24.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:24.248+0000 7f54bf400640 1 -- 172.21.15.6:0/3830887558 >> v2:172.21.15.146:3300/0 conn(0x7f54c00d1110 msgr2=0x7f54c006e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:32:24.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:24.248+0000 7f54bf400640 1 --2- 172.21.15.6:0/3830887558 >> v2:172.21.15.146:3300/0 conn(0x7f54c00d1110 0x7f54c006e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:24.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:24.248+0000 7f54bf400640 1 -- 172.21.15.6:0/3830887558 >> v1:172.21.15.146:6789/0 conn(0x7f54c015ad30 legacy=0x7f54c00b52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:32:24.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:24.249+0000 7f54bf400640 1 -- 172.21.15.6:0/3830887558 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54c00af040 con 0x7f54c00ae5f0 2024-03-04T09:32:24.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:24.250+0000 7f54bf400640 1 --2- 172.21.15.6:0/3830887558 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54c00ae5f0 0x7f54c00d0bd0 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7f54c00b5f00 tx=0x7f54ac02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25d28f74ae46bf45 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:24.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:24.250+0000 7f54bea00640 1 -- 172.21.15.6:0/3830887558 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54ac035020 con 0x7f54c00ae5f0 2024-03-04T09:32:24.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:24.250+0000 7f54bea00640 1 -- 172.21.15.6:0/3830887558 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54ac031870 con 0x7f54c00ae5f0 2024-03-04T09:32:24.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:24.250+0000 7f54bea00640 1 -- 172.21.15.6:0/3830887558 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54ac031c40 con 0x7f54c00ae5f0 2024-03-04T09:32:24.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:24.251+0000 7f54c6e00640 1 -- 172.21.15.6:0/3830887558 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54c00ae5f0 msgr2=0x7f54c00d0bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:24.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:24.251+0000 7f54c6e00640 1 --2- 172.21.15.6:0/3830887558 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54c00ae5f0 0x7f54c00d0bd0 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7f54c00b5f00 tx=0x7f54ac02b910 comp rx=0 tx=0).stop 2024-03-04T09:32:24.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:24.251+0000 7f54c6e00640 1 -- 172.21.15.6:0/3830887558 shutdown_connections 2024-03-04T09:32:24.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:24.251+0000 7f54c6e00640 1 --2- 172.21.15.6:0/3830887558 >> v2:172.21.15.146:3300/0 conn(0x7f54c00d1110 0x7f54c006e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:24.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:24.251+0000 7f54c6e00640 1 --2- 172.21.15.6:0/3830887558 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54c00ae5f0 0x7f54c00d0bd0 unknown :-1 s=CLOSED pgs=1368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:24.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:24.251+0000 7f54c6e00640 1 -- 172.21.15.6:0/3830887558 >> 172.21.15.6:0/3830887558 conn(0x7f54c006c2a0 msgr2=0x7f54c006c6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:32:24.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:24.251+0000 7f54c6e00640 1 -- 172.21.15.6:0/3830887558 shutdown_connections 2024-03-04T09:32:24.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:24.251+0000 7f54c6e00640 1 -- 172.21.15.6:0/3830887558 wait complete. 2024-03-04T09:32:24.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:24.253+0000 7f54c6e00640 1 Processor -- start 2024-03-04T09:32:24.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:24.253+0000 7f54c6e00640 1 -- start start 2024-03-04T09:32:24.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:24.254+0000 7f54c6e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f54c00ae5f0 0x7f54c01f6eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:24.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:24.254+0000 7f54c6e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54c00d1110 0x7f54c014e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:24.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:24.254+0000 7f54c6e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f54c015ad30 0x7f54c014e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:24.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:24.254+0000 7f54c6e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54c00af8e0 con 0x7f54c00d1110 2024-03-04T09:32:24.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:24.254+0000 7f54c6e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54c00af600 con 0x7f54c00ae5f0 2024-03-04T09:32:24.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:24.254+0000 7f54c6e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54c00aea50 con 0x7f54c015ad30 2024-03-04T09:32:24.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:24.254+0000 7f54bf400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54c00d1110 0x7f54c014e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:24.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:24.254+0000 7f54bf400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54c00d1110 0x7f54c014e320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:44084/0 (socket says 172.21.15.6:44084) 2024-03-04T09:32:24.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:24.254+0000 7f54bf400640 1 -- 172.21.15.6:0/259421660 learned_addr learned my addr 172.21.15.6:0/259421660 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:32:24.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:24.255+0000 7f54c4e00640 1 --2- 172.21.15.6:0/259421660 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f54c015ad30 0x7f54c014e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:24.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:24.255+0000 7f54bfe00640 1 --2- 172.21.15.6:0/259421660 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f54c00ae5f0 0x7f54c01f6eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:24.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:24.255+0000 7f54bf400640 1 -- 172.21.15.6:0/259421660 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f54c015ad30 msgr2=0x7f54c014e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:24.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:24.255+0000 7f54bf400640 1 --2- 172.21.15.6:0/259421660 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f54c015ad30 0x7f54c014e860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:24.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:24.255+0000 7f54bf400640 1 -- 172.21.15.6:0/259421660 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f54c00ae5f0 msgr2=0x7f54c01f6eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:24.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:24.255+0000 7f54bf400640 1 --2- 172.21.15.6:0/259421660 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f54c00ae5f0 0x7f54c01f6eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:24.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:24.255+0000 7f54bf400640 1 -- 172.21.15.6:0/259421660 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54c01513a0 con 0x7f54c00d1110 2024-03-04T09:32:24.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:24.255+0000 7f54bf400640 1 --2- 172.21.15.6:0/259421660 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54c00d1110 0x7f54c014e320 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7f54ac000c00 tx=0x7f54ac0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:24.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:24.256+0000 7f54b7e00640 1 -- 172.21.15.6:0/259421660 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54ac03ad60 con 0x7f54c00d1110 2024-03-04T09:32:24.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:24.256+0000 7f54b7e00640 1 -- 172.21.15.6:0/259421660 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54ac0443f0 con 0x7f54c00d1110 2024-03-04T09:32:24.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:24.256+0000 7f54b7e00640 1 -- 172.21.15.6:0/259421660 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54ac0447e0 con 0x7f54c00d1110 2024-03-04T09:32:24.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:24.256+0000 7f54c6e00640 1 -- 172.21.15.6:0/259421660 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54c0151620 con 0x7f54c00d1110 2024-03-04T09:32:24.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:24.256+0000 7f54c6e00640 1 -- 172.21.15.6:0/259421660 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54c0151be0 con 0x7f54c00d1110 2024-03-04T09:32:24.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:24.257+0000 7f54c6e00640 1 -- 172.21.15.6:0/259421660 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54c004f3a0 con 0x7f54c00d1110 2024-03-04T09:32:24.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:24.259+0000 7f54b7e00640 1 -- 172.21.15.6:0/259421660 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f54ac04b060 con 0x7f54c00d1110 2024-03-04T09:32:24.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:24.260+0000 7f54b7e00640 1 --2- 172.21.15.6:0/259421660 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f54a006b500 0x7f54a006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:24.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:24.260+0000 7f54b7e00640 1 -- 172.21.15.6:0/259421660 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f54ac0ac980 con 0x7f54c00d1110 2024-03-04T09:32:24.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:24.261+0000 7f54bfe00640 1 --2- 172.21.15.6:0/259421660 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f54a006b500 0x7f54a006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:24.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:24.262+0000 7f54bfe00640 1 --2- 172.21.15.6:0/259421660 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f54a006b500 0x7f54a006d9c0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f54b0006fd0 tx=0x7f54b0008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:24.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:24.264+0000 7f54b7e00640 1 -- 172.21.15.6:0/259421660 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f54c01513a0 con 0x7f54c00d1110 2024-03-04T09:32:24.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:24.504+0000 7f54c6e00640 1 -- 172.21.15.6:0/259421660 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f54c006e2e0 con 0x7f54c00d1110 2024-03-04T09:32:24.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:24.505+0000 7f54b7e00640 1 -- 172.21.15.6:0/259421660 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+4975 (secure 0 0 0) 0x7f54ac03f070 con 0x7f54c00d1110 2024-03-04T09:32:24.502 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:32:24.502 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":84,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6535,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/3883122012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":3883122012},{"type":"v1","addr":"172.21.15.146:6835","nonce":3883122012}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":6559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/4060411493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":4060411493},{"type":"v1","addr":"172.21.15.146:6837","nonce":4060411493}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":82},{"gid":7812,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/679471398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":679471398},{"type":"v1","addr":"172.21.15.6:6833","nonce":679471398}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:31:21.114351+0000","modified":"2024-03-04T09:31:40.781474+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7815},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7815":{"gid":7815,"name":"a","rank":0,"incarnation":81,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/882667161","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":882667161},{"type":"v1","addr":"172.21.15.6:6835","nonce":882667161}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7815,"qdb_cluster":[7815]},"id":8}]} 2024-03-04T09:32:24.502 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 84 2024-03-04T09:32:24.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:24.507+0000 7f54c6e00640 1 -- 172.21.15.6:0/259421660 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f54a006b500 msgr2=0x7f54a006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:24.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:24.507+0000 7f54c6e00640 1 --2- 172.21.15.6:0/259421660 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f54a006b500 0x7f54a006d9c0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f54b0006fd0 tx=0x7f54b0008040 comp rx=0 tx=0).stop 2024-03-04T09:32:24.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:24.507+0000 7f54c6e00640 1 -- 172.21.15.6:0/259421660 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54c00d1110 msgr2=0x7f54c014e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:24.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:24.507+0000 7f54c6e00640 1 --2- 172.21.15.6:0/259421660 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54c00d1110 0x7f54c014e320 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7f54ac000c00 tx=0x7f54ac0046c0 comp rx=0 tx=0).stop 2024-03-04T09:32:24.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:24.508+0000 7f54c6e00640 1 -- 172.21.15.6:0/259421660 shutdown_connections 2024-03-04T09:32:24.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:24.508+0000 7f54c6e00640 1 --2- 172.21.15.6:0/259421660 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f54a006b500 0x7f54a006d9c0 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:24.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:24.508+0000 7f54c6e00640 1 --2- 172.21.15.6:0/259421660 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f54c015ad30 0x7f54c014e860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:24.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:24.508+0000 7f54c6e00640 1 --2- 172.21.15.6:0/259421660 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54c00d1110 0x7f54c014e320 unknown :-1 s=CLOSED pgs=1369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:24.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:24.508+0000 7f54c6e00640 1 --2- 172.21.15.6:0/259421660 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f54c00ae5f0 0x7f54c01f6eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:24.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:24.508+0000 7f54c6e00640 1 -- 172.21.15.6:0/259421660 >> 172.21.15.6:0/259421660 conn(0x7f54c006c2a0 msgr2=0x7f54c006ed80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:32:24.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:24.508+0000 7f54c6e00640 1 -- 172.21.15.6:0/259421660 shutdown_connections 2024-03-04T09:32:24.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:24.508+0000 7f54c6e00640 1 -- 172.21.15.6:0/259421660 wait complete. 2024-03-04T09:32:24.522 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_enqueued 2024-03-04T09:32:24.753 INFO:teuthology.orchestra.run.smithi006.stdout:{"mds_cache":{"strays_enqueued":0}} 2024-03-04T09:32:24.753 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 0 } } 2024-03-04T09:32:24.753 DEBUG:tasks.ceph_test_case:wait_until_equal: 0 != 1, waiting (timeout=60)... 2024-03-04T09:32:29.754 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:32:29.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:29.986+0000 7f5143000640 1 Processor -- start 2024-03-04T09:32:29.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:29.987+0000 7f5143000640 1 -- start start 2024-03-04T09:32:29.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:29.988+0000 7f5143000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f513c0af1a0 0x7f513c15c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:29.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:29.988+0000 7f5143000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f513c0af580 0x7f513c0b4c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:29.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:29.988+0000 7f5143000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f513c0cdc20 0x7f513c0b2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:29.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:29.988+0000 7f5143000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f513c15cc40 con 0x7f513c0cdc20 2024-03-04T09:32:29.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:29.988+0000 7f5143000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f513c15cf20 con 0x7f513c0af580 2024-03-04T09:32:29.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:29.988+0000 7f5143000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f513c15d200 con 0x7f513c0af1a0 2024-03-04T09:32:29.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:29.989+0000 7f5141000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f513c0cdc20 0x7f513c0b2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:29.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:29.989+0000 7f5141000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f513c0cdc20 0x7f513c0b2dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:44546/0 (socket says 172.21.15.6:44546) 2024-03-04T09:32:29.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:29.989+0000 7f5141000640 1 -- 172.21.15.6:0/1895158659 learned_addr learned my addr 172.21.15.6:0/1895158659 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:32:29.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:29.989+0000 7f513be00640 1 --2- 172.21.15.6:0/1895158659 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f513c0af1a0 0x7f513c15c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:29.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:29.989+0000 7f5133400640 1 --2- 172.21.15.6:0/1895158659 >> v2:172.21.15.146:3300/0 conn(0x7f513c0af580 0x7f513c0b4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:29.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:29.989+0000 7f5141000640 1 -- 172.21.15.6:0/1895158659 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f513c0af1a0 msgr2=0x7f513c15c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:32:29.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:29.989+0000 7f5141000640 1 --2- 172.21.15.6:0/1895158659 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f513c0af1a0 0x7f513c15c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:29.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:29.989+0000 7f5141000640 1 -- 172.21.15.6:0/1895158659 >> v2:172.21.15.146:3300/0 conn(0x7f513c0af580 msgr2=0x7f513c0b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:32:29.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:29.989+0000 7f5141000640 1 --2- 172.21.15.6:0/1895158659 >> v2:172.21.15.146:3300/0 conn(0x7f513c0af580 0x7f513c0b4c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:29.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:29.989+0000 7f5141000640 1 -- 172.21.15.6:0/1895158659 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f513c15d4e0 con 0x7f513c0cdc20 2024-03-04T09:32:29.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:29.991+0000 7f5141000640 1 --2- 172.21.15.6:0/1895158659 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f513c0cdc20 0x7f513c0b2dd0 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7f513c0748c0 tx=0x7f513402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=809a4c1da18f687c server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:29.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:29.991+0000 7f513b400640 1 -- 172.21.15.6:0/1895158659 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f513402c9e0 con 0x7f513c0cdc20 2024-03-04T09:32:29.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:29.991+0000 7f513b400640 1 -- 172.21.15.6:0/1895158659 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f513402cb80 con 0x7f513c0cdc20 2024-03-04T09:32:29.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:29.991+0000 7f513b400640 1 -- 172.21.15.6:0/1895158659 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51340343f0 con 0x7f513c0cdc20 2024-03-04T09:32:29.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:29.992+0000 7f5143000640 1 -- 172.21.15.6:0/1895158659 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f513c0cdc20 msgr2=0x7f513c0b2dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:29.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:29.992+0000 7f5143000640 1 --2- 172.21.15.6:0/1895158659 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f513c0cdc20 0x7f513c0b2dd0 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7f513c0748c0 tx=0x7f513402be40 comp rx=0 tx=0).stop 2024-03-04T09:32:29.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:29.992+0000 7f5143000640 1 -- 172.21.15.6:0/1895158659 shutdown_connections 2024-03-04T09:32:29.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:29.992+0000 7f5143000640 1 --2- 172.21.15.6:0/1895158659 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f513c0cdc20 0x7f513c0b2dd0 unknown :-1 s=CLOSED pgs=1370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:29.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:29.992+0000 7f5143000640 1 --2- 172.21.15.6:0/1895158659 >> v2:172.21.15.146:3300/0 conn(0x7f513c0af580 0x7f513c0b4c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:29.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:29.992+0000 7f5143000640 1 --2- 172.21.15.6:0/1895158659 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f513c0af1a0 0x7f513c15c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:29.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:29.992+0000 7f5143000640 1 -- 172.21.15.6:0/1895158659 >> 172.21.15.6:0/1895158659 conn(0x7f513c06afb0 msgr2=0x7f513c06b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:32:29.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:29.992+0000 7f5143000640 1 -- 172.21.15.6:0/1895158659 shutdown_connections 2024-03-04T09:32:29.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:29.992+0000 7f5143000640 1 -- 172.21.15.6:0/1895158659 wait complete. 2024-03-04T09:32:29.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:29.994+0000 7f5143000640 1 Processor -- start 2024-03-04T09:32:29.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:29.994+0000 7f5143000640 1 -- start start 2024-03-04T09:32:29.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:29.996+0000 7f5143000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f513c0af1a0 0x7f513c1f3230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:29.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:29.996+0000 7f5143000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f513c0af580 0x7f513c0cac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:29.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:29.996+0000 7f5143000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f513c0cdc20 0x7f513c0ccec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:29.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:29.996+0000 7f5143000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f513c0b3490 con 0x7f513c0af1a0 2024-03-04T09:32:29.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:29.996+0000 7f5143000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f513c0b3310 con 0x7f513c0cdc20 2024-03-04T09:32:29.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:29.996+0000 7f5143000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f513c0b3610 con 0x7f513c0af580 2024-03-04T09:32:29.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:29.996+0000 7f513be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f513c0af1a0 0x7f513c1f3230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:29.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:29.996+0000 7f513be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f513c0af1a0 0x7f513c1f3230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:44554/0 (socket says 172.21.15.6:44554) 2024-03-04T09:32:29.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:29.996+0000 7f513be00640 1 -- 172.21.15.6:0/3410953269 learned_addr learned my addr 172.21.15.6:0/3410953269 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:32:29.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:29.997+0000 7f5133400640 1 --2- 172.21.15.6:0/3410953269 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f513c0af580 0x7f513c0cac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:29.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:29.997+0000 7f5141000640 1 --2- 172.21.15.6:0/3410953269 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f513c0cdc20 0x7f513c0ccec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:29.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:29.997+0000 7f513be00640 1 -- 172.21.15.6:0/3410953269 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f513c0af580 msgr2=0x7f513c0cac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:29.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:29.997+0000 7f513be00640 1 --2- 172.21.15.6:0/3410953269 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f513c0af580 0x7f513c0cac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:29.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:29.997+0000 7f513be00640 1 -- 172.21.15.6:0/3410953269 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f513c0cdc20 msgr2=0x7f513c0ccec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:29.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:29.997+0000 7f513be00640 1 --2- 172.21.15.6:0/3410953269 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f513c0cdc20 0x7f513c0ccec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:29.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:29.997+0000 7f513be00640 1 -- 172.21.15.6:0/3410953269 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f513c09c510 con 0x7f513c0af1a0 2024-03-04T09:32:29.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:29.997+0000 7f513be00640 1 --2- 172.21.15.6:0/3410953269 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f513c0af1a0 0x7f513c1f3230 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7f512c00d800 tx=0x7f512c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:29.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:29.997+0000 7f5138c00640 1 -- 172.21.15.6:0/3410953269 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f512c018020 con 0x7f513c0af1a0 2024-03-04T09:32:29.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:29.997+0000 7f5138c00640 1 -- 172.21.15.6:0/3410953269 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f512c009640 con 0x7f513c0af1a0 2024-03-04T09:32:29.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:29.997+0000 7f5143000640 1 -- 172.21.15.6:0/3410953269 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f513c09c7f0 con 0x7f513c0af1a0 2024-03-04T09:32:29.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:29.998+0000 7f5138c00640 1 -- 172.21.15.6:0/3410953269 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f512c009a10 con 0x7f513c0af1a0 2024-03-04T09:32:29.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:29.998+0000 7f5143000640 1 -- 172.21.15.6:0/3410953269 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f513c09cdb0 con 0x7f513c0af1a0 2024-03-04T09:32:29.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:29.999+0000 7f5143000640 1 -- 172.21.15.6:0/3410953269 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f513c069510 con 0x7f513c0af1a0 2024-03-04T09:32:29.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:30.000+0000 7f5138c00640 1 -- 172.21.15.6:0/3410953269 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f512c01b020 con 0x7f513c0af1a0 2024-03-04T09:32:29.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:30.001+0000 7f5138c00640 1 --2- 172.21.15.6:0/3410953269 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f511806b4d0 0x7f511806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:29.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:30.001+0000 7f5138c00640 1 -- 172.21.15.6:0/3410953269 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f512c08a530 con 0x7f513c0af1a0 2024-03-04T09:32:29.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:30.001+0000 7f5133400640 1 --2- 172.21.15.6:0/3410953269 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f511806b4d0 0x7f511806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:29.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:30.003+0000 7f5133400640 1 --2- 172.21.15.6:0/3410953269 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f511806b4d0 0x7f511806d990 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f513c1f6d50 tx=0x7f5124008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:30.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:30.006+0000 7f5138c00640 1 -- 172.21.15.6:0/3410953269 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f512c022030 con 0x7f513c0af1a0 2024-03-04T09:32:30.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:30.298+0000 7f5143000640 1 -- 172.21.15.6:0/3410953269 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f513c04f3a0 con 0x7f513c0af1a0 2024-03-04T09:32:30.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:30.299+0000 7f5138c00640 1 -- 172.21.15.6:0/3410953269 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+4975 (secure 0 0 0) 0x7f512c059280 con 0x7f513c0af1a0 2024-03-04T09:32:30.296 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:32:30.296 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":84,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6535,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/3883122012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":3883122012},{"type":"v1","addr":"172.21.15.146:6835","nonce":3883122012}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":6559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/4060411493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":4060411493},{"type":"v1","addr":"172.21.15.146:6837","nonce":4060411493}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":82},{"gid":7812,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/679471398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":679471398},{"type":"v1","addr":"172.21.15.6:6833","nonce":679471398}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:31:21.114351+0000","modified":"2024-03-04T09:31:40.781474+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7815},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7815":{"gid":7815,"name":"a","rank":0,"incarnation":81,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/882667161","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":882667161},{"type":"v1","addr":"172.21.15.6:6835","nonce":882667161}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7815,"qdb_cluster":[7815]},"id":8}]} 2024-03-04T09:32:30.296 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 84 2024-03-04T09:32:30.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:30.301+0000 7f5143000640 1 -- 172.21.15.6:0/3410953269 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f511806b4d0 msgr2=0x7f511806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:30.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:30.301+0000 7f5143000640 1 --2- 172.21.15.6:0/3410953269 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f511806b4d0 0x7f511806d990 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f513c1f6d50 tx=0x7f5124008040 comp rx=0 tx=0).stop 2024-03-04T09:32:30.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:30.301+0000 7f5143000640 1 -- 172.21.15.6:0/3410953269 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f513c0af1a0 msgr2=0x7f513c1f3230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:30.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:30.301+0000 7f5143000640 1 --2- 172.21.15.6:0/3410953269 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f513c0af1a0 0x7f513c1f3230 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7f512c00d800 tx=0x7f512c00dcc0 comp rx=0 tx=0).stop 2024-03-04T09:32:30.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:30.302+0000 7f5143000640 1 -- 172.21.15.6:0/3410953269 shutdown_connections 2024-03-04T09:32:30.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:30.302+0000 7f5143000640 1 --2- 172.21.15.6:0/3410953269 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f513c0cdc20 0x7f513c0ccec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:30.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:30.302+0000 7f5143000640 1 --2- 172.21.15.6:0/3410953269 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f511806b4d0 0x7f511806d990 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:30.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:30.302+0000 7f5143000640 1 --2- 172.21.15.6:0/3410953269 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f513c0af580 0x7f513c0cac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:30.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:30.302+0000 7f5143000640 1 --2- 172.21.15.6:0/3410953269 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f513c0af1a0 0x7f513c1f3230 unknown :-1 s=CLOSED pgs=1371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:30.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:30.302+0000 7f5143000640 1 -- 172.21.15.6:0/3410953269 >> 172.21.15.6:0/3410953269 conn(0x7f513c06afb0 msgr2=0x7f513c071520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:32:30.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:30.302+0000 7f5143000640 1 -- 172.21.15.6:0/3410953269 shutdown_connections 2024-03-04T09:32:30.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:30.302+0000 7f5143000640 1 -- 172.21.15.6:0/3410953269 wait complete. 2024-03-04T09:32:30.318 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_enqueued 2024-03-04T09:32:30.550 INFO:teuthology.orchestra.run.smithi006.stdout:{"mds_cache":{"strays_enqueued":1}} 2024-03-04T09:32:30.550 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 1 } } 2024-03-04T09:32:30.550 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:32:30.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:30.808+0000 7fed11200640 1 Processor -- start 2024-03-04T09:32:30.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:30.809+0000 7fed11200640 1 -- start start 2024-03-04T09:32:30.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:30.810+0000 7fed11200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fed0c145300 0x7fed0c0b49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:30.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:30.811+0000 7fed11200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fed0c070f10 0x7fed0c0cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:30.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:30.811+0000 7fed11200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed0c0714d0 con 0x7fed0c0cce90 2024-03-04T09:32:30.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:30.811+0000 7fed11200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed0c0cd300 con 0x7fed0c145300 2024-03-04T09:32:30.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:30.811+0000 7fed11200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed0c0717b0 con 0x7fed0c070f10 2024-03-04T09:32:30.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:30.811+0000 7fed0be00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fed0c0cce90 0x7fed0c0cd930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:36852/0 (socket says 172.21.15.6:36852) 2024-03-04T09:32:30.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:30.811+0000 7fed0be00640 1 -- 172.21.15.6:0/3001862682 learned_addr learned my addr 172.21.15.6:0/3001862682 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:32:30.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:30.811+0000 7fed0aa00640 1 --2- 172.21.15.6:0/3001862682 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fed0c070f10 0x7fed0c0cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:30.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:30.811+0000 7fed0b400640 1 --2- 172.21.15.6:0/3001862682 >> v2:172.21.15.146:3300/0 conn(0x7fed0c145300 0x7fed0c0b49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:30.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:30.812+0000 7fed0a000640 1 -- 172.21.15.6:0/3001862682 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 200117727 0 0) 0x7fed0c0714d0 con 0x7fed0c0cce90 2024-03-04T09:32:30.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:30.812+0000 7fed0a000640 1 -- 172.21.15.6:0/3001862682 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fecf0003520 con 0x7fed0c0cce90 2024-03-04T09:32:30.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:30.812+0000 7fed0aa00640 1 -- 172.21.15.6:0/3001862682 >> v2:172.21.15.146:3300/0 conn(0x7fed0c145300 msgr2=0x7fed0c0b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:32:30.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:30.812+0000 7fed0a000640 1 -- 172.21.15.6:0/3001862682 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 562891589 0 0) 0x7fecf0003520 con 0x7fed0c0cce90 2024-03-04T09:32:30.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:30.812+0000 7fed0aa00640 1 --2- 172.21.15.6:0/3001862682 >> v2:172.21.15.146:3300/0 conn(0x7fed0c145300 0x7fed0c0b49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:30.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:30.812+0000 7fed0aa00640 1 -- 172.21.15.6:0/3001862682 >> v1:172.21.15.6:6789/0 conn(0x7fed0c0cce90 legacy=0x7fed0c0cd930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:30.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:30.812+0000 7fed0aa00640 1 -- 172.21.15.6:0/3001862682 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed0c071a90 con 0x7fed0c070f10 2024-03-04T09:32:30.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:30.812+0000 7fed0b400640 1 --2- 172.21.15.6:0/3001862682 >> v2:172.21.15.146:3300/0 conn(0x7fed0c145300 0x7fed0c0b49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T09:32:30.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:30.814+0000 7fed0aa00640 1 --2- 172.21.15.6:0/3001862682 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fed0c070f10 0x7fed0c0cc950 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fecf80097a0 tx=0x7fecf802b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=34809d167d90a528 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:30.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:30.814+0000 7fed0a000640 1 -- 172.21.15.6:0/3001862682 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecf8037070 con 0x7fed0c070f10 2024-03-04T09:32:30.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:30.814+0000 7fed0a000640 1 -- 172.21.15.6:0/3001862682 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecf8030510 con 0x7fed0c070f10 2024-03-04T09:32:30.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:30.814+0000 7fed0a000640 1 -- 172.21.15.6:0/3001862682 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed040019f0 con 0x7fed0c070f10 2024-03-04T09:32:30.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:30.815+0000 7fed11200640 1 -- 172.21.15.6:0/3001862682 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fed0c070f10 msgr2=0x7fed0c0cc950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:30.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:30.815+0000 7fed11200640 1 --2- 172.21.15.6:0/3001862682 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fed0c070f10 0x7fed0c0cc950 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fecf80097a0 tx=0x7fecf802b790 comp rx=0 tx=0).stop 2024-03-04T09:32:30.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:30.815+0000 7fed11200640 1 -- 172.21.15.6:0/3001862682 shutdown_connections 2024-03-04T09:32:30.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:30.815+0000 7fed11200640 1 --2- 172.21.15.6:0/3001862682 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fed0c070f10 0x7fed0c0cc950 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:30.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:30.815+0000 7fed11200640 1 --2- 172.21.15.6:0/3001862682 >> v2:172.21.15.146:3300/0 conn(0x7fed0c145300 0x7fed0c0b49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:30.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:30.815+0000 7fed11200640 1 -- 172.21.15.6:0/3001862682 >> 172.21.15.6:0/3001862682 conn(0x7fed0c06b3d0 msgr2=0x7fed0c06b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:32:30.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:30.815+0000 7fed11200640 1 -- 172.21.15.6:0/3001862682 shutdown_connections 2024-03-04T09:32:30.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:30.815+0000 7fed11200640 1 -- 172.21.15.6:0/3001862682 wait complete. 2024-03-04T09:32:30.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:30.817+0000 7fed11200640 1 Processor -- start 2024-03-04T09:32:30.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:30.818+0000 7fed11200640 1 -- start start 2024-03-04T09:32:30.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:30.819+0000 7fed11200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fed0c070f10 0x7fed0c136280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:30.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:30.819+0000 7fed11200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fed0c0cce90 0x7fed0c0ae1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:30.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:30.819+0000 7fed11200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fed0c145300 0x7fed0c0ae730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:30.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:30.819+0000 7fed11200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed0c1456e0 con 0x7fed0c145300 2024-03-04T09:32:30.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:30.819+0000 7fed11200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed0c0b4ee0 con 0x7fed0c070f10 2024-03-04T09:32:30.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:30.819+0000 7fed11200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed0c15d580 con 0x7fed0c0cce90 2024-03-04T09:32:30.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:30.819+0000 7fed0be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fed0c145300 0x7fed0c0ae730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:30.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:30.819+0000 7fed0be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fed0c145300 0x7fed0c0ae730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:44560/0 (socket says 172.21.15.6:44560) 2024-03-04T09:32:30.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:30.819+0000 7fed0be00640 1 -- 172.21.15.6:0/1468399377 learned_addr learned my addr 172.21.15.6:0/1468399377 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:32:30.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:30.819+0000 7fed0b400640 1 --2- 172.21.15.6:0/1468399377 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fed0c070f10 0x7fed0c136280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:30.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:30.820+0000 7fed0aa00640 1 --2- 172.21.15.6:0/1468399377 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fed0c0cce90 0x7fed0c0ae1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:30.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:30.820+0000 7fed0be00640 1 -- 172.21.15.6:0/1468399377 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fed0c0cce90 msgr2=0x7fed0c0ae1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:30.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:30.820+0000 7fed0be00640 1 --2- 172.21.15.6:0/1468399377 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fed0c0cce90 0x7fed0c0ae1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:30.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:30.820+0000 7fed0be00640 1 -- 172.21.15.6:0/1468399377 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fed0c070f10 msgr2=0x7fed0c136280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:30.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:30.820+0000 7fed0be00640 1 --2- 172.21.15.6:0/1468399377 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fed0c070f10 0x7fed0c136280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:30.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:30.820+0000 7fed0be00640 1 -- 172.21.15.6:0/1468399377 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed0c1f41c0 con 0x7fed0c145300 2024-03-04T09:32:30.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:30.820+0000 7fed0be00640 1 --2- 172.21.15.6:0/1468399377 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fed0c145300 0x7fed0c0ae730 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x7fed04001890 tx=0x7fed0400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:30.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:30.820+0000 7fed03400640 1 -- 172.21.15.6:0/1468399377 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed04012400 con 0x7fed0c145300 2024-03-04T09:32:30.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:30.820+0000 7fed03400640 1 -- 172.21.15.6:0/1468399377 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed040125c0 con 0x7fed0c145300 2024-03-04T09:32:30.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:30.820+0000 7fed03400640 1 -- 172.21.15.6:0/1468399377 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed040129b0 con 0x7fed0c145300 2024-03-04T09:32:30.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:30.821+0000 7fed11200640 1 -- 172.21.15.6:0/1468399377 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed0c1f44a0 con 0x7fed0c145300 2024-03-04T09:32:30.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:30.821+0000 7fed11200640 1 -- 172.21.15.6:0/1468399377 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed0c1f4a60 con 0x7fed0c145300 2024-03-04T09:32:30.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:30.822+0000 7fed11200640 1 -- 172.21.15.6:0/1468399377 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed0c04f330 con 0x7fed0c145300 2024-03-04T09:32:30.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:30.823+0000 7fed03400640 1 -- 172.21.15.6:0/1468399377 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fed04012b50 con 0x7fed0c145300 2024-03-04T09:32:30.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:30.824+0000 7fed03400640 1 --2- 172.21.15.6:0/1468399377 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fecec06b4d0 0x7fecec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:30.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:30.824+0000 7fed03400640 1 -- 172.21.15.6:0/1468399377 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5853+0+0 (secure 0 0 0) 0x7fed0408aba0 con 0x7fed0c145300 2024-03-04T09:32:30.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:30.825+0000 7fed0b400640 1 --2- 172.21.15.6:0/1468399377 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fecec06b4d0 0x7fecec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:30.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:30.826+0000 7fed0b400640 1 --2- 172.21.15.6:0/1468399377 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fecec06b4d0 0x7fecec06d990 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fecfc009560 tx=0x7fecfc009190 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:30.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:30.829+0000 7fed03400640 1 -- 172.21.15.6:0/1468399377 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fed040598f0 con 0x7fed0c145300 2024-03-04T09:32:31.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:31.135+0000 7fed11200640 1 -- 172.21.15.6:0/1468399377 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fed0c144940 con 0x7fed0c145300 2024-03-04T09:32:31.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:31.136+0000 7fed03400640 1 -- 172.21.15.6:0/1468399377 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+4975 (secure 0 0 0) 0x7fed040591f0 con 0x7fed0c145300 2024-03-04T09:32:31.133 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:32:31.133 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":84,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6535,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/3883122012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":3883122012},{"type":"v1","addr":"172.21.15.146:6835","nonce":3883122012}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":6559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/4060411493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":4060411493},{"type":"v1","addr":"172.21.15.146:6837","nonce":4060411493}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":82},{"gid":7812,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/679471398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":679471398},{"type":"v1","addr":"172.21.15.6:6833","nonce":679471398}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:31:21.114351+0000","modified":"2024-03-04T09:31:40.781474+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7815},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7815":{"gid":7815,"name":"a","rank":0,"incarnation":81,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/882667161","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":882667161},{"type":"v1","addr":"172.21.15.6:6835","nonce":882667161}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7815,"qdb_cluster":[7815]},"id":8}]} 2024-03-04T09:32:31.133 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 84 2024-03-04T09:32:31.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:31.139+0000 7fed11200640 1 -- 172.21.15.6:0/1468399377 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fecec06b4d0 msgr2=0x7fecec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:31.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:31.139+0000 7fed11200640 1 --2- 172.21.15.6:0/1468399377 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fecec06b4d0 0x7fecec06d990 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fecfc009560 tx=0x7fecfc009190 comp rx=0 tx=0).stop 2024-03-04T09:32:31.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:31.139+0000 7fed11200640 1 -- 172.21.15.6:0/1468399377 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fed0c145300 msgr2=0x7fed0c0ae730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:31.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:31.139+0000 7fed11200640 1 --2- 172.21.15.6:0/1468399377 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fed0c145300 0x7fed0c0ae730 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x7fed04001890 tx=0x7fed0400a430 comp rx=0 tx=0).stop 2024-03-04T09:32:31.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:31.139+0000 7fed11200640 1 -- 172.21.15.6:0/1468399377 shutdown_connections 2024-03-04T09:32:31.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:31.139+0000 7fed11200640 1 --2- 172.21.15.6:0/1468399377 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fed0c145300 0x7fed0c0ae730 unknown :-1 s=CLOSED pgs=1373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:31.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:31.139+0000 7fed11200640 1 --2- 172.21.15.6:0/1468399377 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fecec06b4d0 0x7fecec06d990 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:31.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:31.139+0000 7fed11200640 1 --2- 172.21.15.6:0/1468399377 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fed0c0cce90 0x7fed0c0ae1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:31.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:31.139+0000 7fed11200640 1 --2- 172.21.15.6:0/1468399377 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fed0c070f10 0x7fed0c136280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:31.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:31.139+0000 7fed11200640 1 -- 172.21.15.6:0/1468399377 >> 172.21.15.6:0/1468399377 conn(0x7fed0c06b3d0 msgr2=0x7fed0c0b17c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:32:31.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:31.139+0000 7fed11200640 1 -- 172.21.15.6:0/1468399377 shutdown_connections 2024-03-04T09:32:31.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:31.139+0000 7fed11200640 1 -- 172.21.15.6:0/1468399377 wait complete. 2024-03-04T09:32:31.155 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump purge_queue pq_executed 2024-03-04T09:32:31.394 INFO:teuthology.orchestra.run.smithi006.stdout:{"purge_queue":{"pq_executed":1}} 2024-03-04T09:32:31.394 DEBUG:tasks.cephfs.filesystem:_json_asok output { "purge_queue": { "pq_executed": 1 } } 2024-03-04T09:32:31.394 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:32:31.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:31.634+0000 7f8d28c00640 1 Processor -- start 2024-03-04T09:32:31.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:31.634+0000 7f8d28c00640 1 -- start start 2024-03-04T09:32:31.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:31.636+0000 7f8d28c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8d240d4a10 0x7f8d240d0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:31.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:31.636+0000 7f8d28c00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f8d240d0c80 0x7f8d240b44d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:31.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:31.636+0000 7f8d28c00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d240b3710 con 0x7f8d2415e0b0 2024-03-04T09:32:31.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:31.636+0000 7f8d28c00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d240b09e0 con 0x7f8d240d0c80 2024-03-04T09:32:31.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:31.636+0000 7f8d28c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d240cfa20 con 0x7f8d240d4a10 2024-03-04T09:32:31.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:31.636+0000 7f8d23400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f8d2415e0b0 0x7f8d240cf310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:36856/0 (socket says 172.21.15.6:36856) 2024-03-04T09:32:31.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:31.636+0000 7f8d23400640 1 -- 172.21.15.6:0/2252148563 learned_addr learned my addr 172.21.15.6:0/2252148563 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:32:31.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:31.636+0000 7f8d23e00640 1 --2- 172.21.15.6:0/2252148563 >> v2:172.21.15.146:3300/0 conn(0x7f8d240d0c80 0x7f8d240b44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:31.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:31.637+0000 7f8d22a00640 1 --2- 172.21.15.6:0/2252148563 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8d240d4a10 0x7f8d240d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:31.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:31.637+0000 7f8d22000640 1 -- 172.21.15.6:0/2252148563 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1155954014 0 0) 0x7f8d240b3710 con 0x7f8d2415e0b0 2024-03-04T09:32:31.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:31.637+0000 7f8d22000640 1 -- 172.21.15.6:0/2252148563 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d08003520 con 0x7f8d2415e0b0 2024-03-04T09:32:31.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:31.637+0000 7f8d22000640 1 -- 172.21.15.6:0/2252148563 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2002794154 0 0) 0x7f8d08003520 con 0x7f8d2415e0b0 2024-03-04T09:32:31.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:31.637+0000 7f8d22000640 1 -- 172.21.15.6:0/2252148563 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8d240d4a10 msgr2=0x7f8d240d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:32:31.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:31.637+0000 7f8d22000640 1 --2- 172.21.15.6:0/2252148563 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8d240d4a10 0x7f8d240d0740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:31.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:31.637+0000 7f8d22000640 1 -- 172.21.15.6:0/2252148563 >> v2:172.21.15.146:3300/0 conn(0x7f8d240d0c80 msgr2=0x7f8d240b44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:32:31.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:31.637+0000 7f8d22000640 1 --2- 172.21.15.6:0/2252148563 >> v2:172.21.15.146:3300/0 conn(0x7f8d240d0c80 0x7f8d240b44d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:31.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:31.637+0000 7f8d22000640 1 -- 172.21.15.6:0/2252148563 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d240b39f0 con 0x7f8d2415e0b0 2024-03-04T09:32:31.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:31.637+0000 7f8d22000640 1 -- 172.21.15.6:0/2252148563 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7f8d14001670 con 0x7f8d2415e0b0 2024-03-04T09:32:31.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:31.637+0000 7f8d22000640 1 -- 172.21.15.6:0/2252148563 >> v1:172.21.15.6:6789/0 conn(0x7f8d2415e0b0 legacy=0x7f8d240cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:31.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:31.637+0000 7f8d22000640 1 --2- 172.21.15.6:0/2252148563 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8d080039b0 0x7f8d08003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:31.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:31.637+0000 7f8d22000640 1 --2- 172.21.15.6:0/2252148563 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8d08004800 0x7f8d08004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:31.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:31.637+0000 7f8d22000640 1 --2- 172.21.15.6:0/2252148563 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8d240d4a10 0x7f8d080051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:31.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:31.637+0000 7f8d22000640 1 -- 172.21.15.6:0/2252148563 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d240cfa20 con 0x7f8d08004800 2024-03-04T09:32:31.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:31.637+0000 7f8d22000640 1 -- 172.21.15.6:0/2252148563 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d0800b9c0 con 0x7f8d080039b0 2024-03-04T09:32:31.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:31.637+0000 7f8d22000640 1 -- 172.21.15.6:0/2252148563 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d0800bb40 con 0x7f8d240d4a10 2024-03-04T09:32:31.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:31.638+0000 7f8d22a00640 1 --2- 172.21.15.6:0/2252148563 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8d08004800 0x7f8d08004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:31.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:31.638+0000 7f8d23e00640 1 --2- 172.21.15.6:0/2252148563 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8d240d4a10 0x7f8d080051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:31.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:31.638+0000 7f8d23400640 1 --2- 172.21.15.6:0/2252148563 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8d080039b0 0x7f8d08003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:31.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:31.638+0000 7f8d22a00640 1 -- 172.21.15.6:0/2252148563 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8d240d4a10 msgr2=0x7f8d080051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:32:31.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:31.638+0000 7f8d22a00640 1 --2- 172.21.15.6:0/2252148563 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8d240d4a10 0x7f8d080051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:31.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:31.638+0000 7f8d22a00640 1 -- 172.21.15.6:0/2252148563 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8d080039b0 msgr2=0x7f8d08003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:32:31.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:31.638+0000 7f8d22a00640 1 --2- 172.21.15.6:0/2252148563 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8d080039b0 0x7f8d08003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:31.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:31.638+0000 7f8d22a00640 1 -- 172.21.15.6:0/2252148563 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d08003520 con 0x7f8d08004800 2024-03-04T09:32:31.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:31.638+0000 7f8d23400640 1 --2- 172.21.15.6:0/2252148563 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8d080039b0 0x7f8d08003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:32:31.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:31.639+0000 7f8d22a00640 1 --2- 172.21.15.6:0/2252148563 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8d08004800 0x7f8d08004c80 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7f8d10009a90 tx=0x7f8d08005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d91704eb883a64a server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:31.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:31.640+0000 7f8d22000640 1 -- 172.21.15.6:0/2252148563 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d10039070 con 0x7f8d08004800 2024-03-04T09:32:31.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:31.640+0000 7f8d22000640 1 -- 172.21.15.6:0/2252148563 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d1002c6b0 con 0x7f8d08004800 2024-03-04T09:32:31.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:31.640+0000 7f8d22000640 1 -- 172.21.15.6:0/2252148563 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d10009690 con 0x7f8d08004800 2024-03-04T09:32:31.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:31.640+0000 7f8d28c00640 1 -- 172.21.15.6:0/2252148563 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8d08004800 msgr2=0x7f8d08004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:31.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:31.640+0000 7f8d28c00640 1 --2- 172.21.15.6:0/2252148563 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8d08004800 0x7f8d08004c80 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7f8d10009a90 tx=0x7f8d08005830 comp rx=0 tx=0).stop 2024-03-04T09:32:31.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:31.640+0000 7f8d23e00640 1 -- 172.21.15.6:0/2252148563 reap_dead start 2024-03-04T09:32:31.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:31.640+0000 7f8d28c00640 1 -- 172.21.15.6:0/2252148563 shutdown_connections 2024-03-04T09:32:31.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:31.640+0000 7f8d28c00640 1 -- 172.21.15.6:0/2252148563 >> 172.21.15.6:0/2252148563 conn(0x7f8d240d3d30 msgr2=0x7f8d240d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:32:31.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:31.641+0000 7f8d28c00640 1 -- 172.21.15.6:0/2252148563 shutdown_connections 2024-03-04T09:32:31.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:31.641+0000 7f8d28c00640 1 -- 172.21.15.6:0/2252148563 wait complete. 2024-03-04T09:32:31.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:31.642+0000 7f8d28c00640 1 Processor -- start 2024-03-04T09:32:31.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:31.642+0000 7f8d28c00640 1 -- start start 2024-03-04T09:32:31.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:31.643+0000 7f8d28c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8d08004800 0x7f8d2415fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:31.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:31.643+0000 7f8d28c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8d24152fb0 0x7f8d2415f340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:31.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:31.643+0000 7f8d28c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8d2415f880 0x7f8d2406c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:31.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:31.643+0000 7f8d28c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d2415e490 con 0x7f8d08004800 2024-03-04T09:32:31.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:31.643+0000 7f8d28c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d240b3fb0 con 0x7f8d2415f880 2024-03-04T09:32:31.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:31.643+0000 7f8d28c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d240cfc60 con 0x7f8d24152fb0 2024-03-04T09:32:31.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:31.644+0000 7f8d23400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8d08004800 0x7f8d2415fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:31.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:31.644+0000 7f8d23400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8d08004800 0x7f8d2415fd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:44576/0 (socket says 172.21.15.6:44576) 2024-03-04T09:32:31.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:31.644+0000 7f8d23400640 1 -- 172.21.15.6:0/624610961 learned_addr learned my addr 172.21.15.6:0/624610961 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:32:31.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:31.644+0000 7f8d22a00640 1 --2- 172.21.15.6:0/624610961 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8d24152fb0 0x7f8d2415f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:31.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:31.644+0000 7f8d23e00640 1 --2- 172.21.15.6:0/624610961 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8d2415f880 0x7f8d2406c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:31.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:31.644+0000 7f8d23400640 1 -- 172.21.15.6:0/624610961 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8d24152fb0 msgr2=0x7f8d2415f340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:31.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:31.644+0000 7f8d23400640 1 --2- 172.21.15.6:0/624610961 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8d24152fb0 0x7f8d2415f340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:31.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:31.644+0000 7f8d23400640 1 -- 172.21.15.6:0/624610961 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8d2415f880 msgr2=0x7f8d2406c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:31.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:31.644+0000 7f8d23400640 1 --2- 172.21.15.6:0/624610961 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8d2415f880 0x7f8d2406c9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:31.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:31.644+0000 7f8d23400640 1 -- 172.21.15.6:0/624610961 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d241606b0 con 0x7f8d08004800 2024-03-04T09:32:31.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:31.645+0000 7f8d23400640 1 --2- 172.21.15.6:0/624610961 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8d08004800 0x7f8d2415fd80 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7f8d1400d8b0 tx=0x7f8d1400dd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:31.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:31.645+0000 7f8d1b400640 1 -- 172.21.15.6:0/624610961 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d14019070 con 0x7f8d08004800 2024-03-04T09:32:31.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:31.645+0000 7f8d1b400640 1 -- 172.21.15.6:0/624610961 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d140074e0 con 0x7f8d08004800 2024-03-04T09:32:31.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:31.645+0000 7f8d1b400640 1 -- 172.21.15.6:0/624610961 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d140049e0 con 0x7f8d08004800 2024-03-04T09:32:31.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:31.645+0000 7f8d28c00640 1 -- 172.21.15.6:0/624610961 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d2415e860 con 0x7f8d08004800 2024-03-04T09:32:31.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:31.645+0000 7f8d28c00640 1 -- 172.21.15.6:0/624610961 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d2415ee20 con 0x7f8d08004800 2024-03-04T09:32:31.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:31.646+0000 7f8d28c00640 1 -- 172.21.15.6:0/624610961 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d2404f330 con 0x7f8d08004800 2024-03-04T09:32:31.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:31.648+0000 7f8d1b400640 1 -- 172.21.15.6:0/624610961 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8d1401c070 con 0x7f8d08004800 2024-03-04T09:32:31.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:31.650+0000 7f8d1b400640 1 --2- 172.21.15.6:0/624610961 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f8d0406b500 0x7f8d0406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:31.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:31.650+0000 7f8d1b400640 1 -- 172.21.15.6:0/624610961 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f8d14089d30 con 0x7f8d08004800 2024-03-04T09:32:31.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:31.651+0000 7f8d22a00640 1 --2- 172.21.15.6:0/624610961 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f8d0406b500 0x7f8d0406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:31.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:31.652+0000 7f8d22a00640 1 --2- 172.21.15.6:0/624610961 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f8d0406b500 0x7f8d0406d9c0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f8d10036000 tx=0x7f8d10030000 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:31.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:31.655+0000 7f8d1b400640 1 -- 172.21.15.6:0/624610961 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8d14058b10 con 0x7f8d08004800 2024-03-04T09:32:31.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:31.894+0000 7f8d28c00640 1 -- 172.21.15.6:0/624610961 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8d240b1720 con 0x7f8d08004800 2024-03-04T09:32:31.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:31.894+0000 7f8d1b400640 1 -- 172.21.15.6:0/624610961 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+4975 (secure 0 0 0) 0x7f8d14058410 con 0x7f8d08004800 2024-03-04T09:32:31.891 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:32:31.892 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":84,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6535,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/3883122012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":3883122012},{"type":"v1","addr":"172.21.15.146:6835","nonce":3883122012}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":6559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/4060411493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":4060411493},{"type":"v1","addr":"172.21.15.146:6837","nonce":4060411493}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":82},{"gid":7812,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/679471398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":679471398},{"type":"v1","addr":"172.21.15.6:6833","nonce":679471398}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:31:21.114351+0000","modified":"2024-03-04T09:31:40.781474+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7815},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7815":{"gid":7815,"name":"a","rank":0,"incarnation":81,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/882667161","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":882667161},{"type":"v1","addr":"172.21.15.6:6835","nonce":882667161}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7815,"qdb_cluster":[7815]},"id":8}]} 2024-03-04T09:32:31.892 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 84 2024-03-04T09:32:31.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:31.897+0000 7f8d28c00640 1 -- 172.21.15.6:0/624610961 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f8d0406b500 msgr2=0x7f8d0406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:31.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:31.897+0000 7f8d28c00640 1 --2- 172.21.15.6:0/624610961 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f8d0406b500 0x7f8d0406d9c0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f8d10036000 tx=0x7f8d10030000 comp rx=0 tx=0).stop 2024-03-04T09:32:31.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:31.897+0000 7f8d28c00640 1 -- 172.21.15.6:0/624610961 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8d08004800 msgr2=0x7f8d2415fd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:31.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:31.897+0000 7f8d28c00640 1 --2- 172.21.15.6:0/624610961 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8d08004800 0x7f8d2415fd80 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7f8d1400d8b0 tx=0x7f8d1400dd70 comp rx=0 tx=0).stop 2024-03-04T09:32:31.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:31.897+0000 7f8d28c00640 1 -- 172.21.15.6:0/624610961 shutdown_connections 2024-03-04T09:32:31.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:31.897+0000 7f8d28c00640 1 --2- 172.21.15.6:0/624610961 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8d2415f880 0x7f8d2406c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:31.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:31.897+0000 7f8d28c00640 1 --2- 172.21.15.6:0/624610961 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f8d0406b500 0x7f8d0406d9c0 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:31.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:31.897+0000 7f8d28c00640 1 --2- 172.21.15.6:0/624610961 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8d24152fb0 0x7f8d2415f340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:31.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:31.897+0000 7f8d28c00640 1 --2- 172.21.15.6:0/624610961 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8d08004800 0x7f8d2415fd80 unknown :-1 s=CLOSED pgs=1376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:31.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:31.897+0000 7f8d28c00640 1 -- 172.21.15.6:0/624610961 >> 172.21.15.6:0/624610961 conn(0x7f8d240d3d30 msgr2=0x7f8d240b1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:32:31.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:31.897+0000 7f8d28c00640 1 -- 172.21.15.6:0/624610961 shutdown_connections 2024-03-04T09:32:31.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:31.898+0000 7f8d28c00640 1 -- 172.21.15.6:0/624610961 wait complete. 2024-03-04T09:32:31.913 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache 2024-03-04T09:32:32.138 INFO:teuthology.orchestra.run.smithi006.stdout:{"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":0,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":3,"strays_enqueued":1,"strays_reintegrated":2,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0}} 2024-03-04T09:32:32.139 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 0, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 3, "strays_enqueued": 1, "strays_migrated": 0, "strays_reintegrated": 2 } } 2024-03-04T09:32:32.139 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:32:32.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.340+0000 7f3daa600640 1 Processor -- start 2024-03-04T09:32:32.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.340+0000 7f3daa600640 1 -- start start 2024-03-04T09:32:32.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.342+0000 7f3daa600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3da40b5740 0x7f3da40b5b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:32.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.342+0000 7f3daa600640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f3da409b120 0x7f3da40b1690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:32.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.342+0000 7f3daa600640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3da40b6060 con 0x7f3da415f760 2024-03-04T09:32:32.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.342+0000 7f3daa600640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3da415d9b0 con 0x7f3da409b120 2024-03-04T09:32:32.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.342+0000 7f3daa600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3da415dc90 con 0x7f3da40b5740 2024-03-04T09:32:32.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.342+0000 7f3da9c00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f3da415f760 0x7f3da415ea50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:36860/0 (socket says 172.21.15.6:36860) 2024-03-04T09:32:32.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.343+0000 7f3da9c00640 1 -- 172.21.15.6:0/2497842811 learned_addr learned my addr 172.21.15.6:0/2497842811 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:32:32.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.343+0000 7f3da9200640 1 --2- 172.21.15.6:0/2497842811 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3da40b5740 0x7f3da40b5b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:32.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.343+0000 7f3d9be00640 1 --2- 172.21.15.6:0/2497842811 >> v2:172.21.15.146:3300/0 conn(0x7f3da409b120 0x7f3da40b1690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:32.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.343+0000 7f3da3e00640 1 -- 172.21.15.6:0/2497842811 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1427977861 0 0) 0x7f3da40b6060 con 0x7f3da415f760 2024-03-04T09:32:32.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.343+0000 7f3da3e00640 1 -- 172.21.15.6:0/2497842811 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d88003520 con 0x7f3da415f760 2024-03-04T09:32:32.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.343+0000 7f3d9be00640 1 -- 172.21.15.6:0/2497842811 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3da40b5740 msgr2=0x7f3da40b5b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:32:32.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.343+0000 7f3d9be00640 1 --2- 172.21.15.6:0/2497842811 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3da40b5740 0x7f3da40b5b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:32.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.343+0000 7f3d9be00640 1 -- 172.21.15.6:0/2497842811 >> v1:172.21.15.6:6789/0 conn(0x7f3da415f760 legacy=0x7f3da415ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:32.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.343+0000 7f3d9be00640 1 -- 172.21.15.6:0/2497842811 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3da415df70 con 0x7f3da409b120 2024-03-04T09:32:32.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.345+0000 7f3d9be00640 1 --2- 172.21.15.6:0/2497842811 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3da409b120 0x7f3da40b1690 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f3d8c0097a0 tx=0x7f3d8c02b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=651ced7987e38195 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:32.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.345+0000 7f3da3e00640 1 -- 172.21.15.6:0/2497842811 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d8c031780 con 0x7f3da409b120 2024-03-04T09:32:32.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.345+0000 7f3da3e00640 1 -- 172.21.15.6:0/2497842811 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d8c037070 con 0x7f3da409b120 2024-03-04T09:32:32.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.345+0000 7f3da3e00640 1 -- 172.21.15.6:0/2497842811 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d8c031b20 con 0x7f3da409b120 2024-03-04T09:32:32.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.346+0000 7f3daa600640 1 -- 172.21.15.6:0/2497842811 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3da409b120 msgr2=0x7f3da40b1690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:32.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.346+0000 7f3daa600640 1 --2- 172.21.15.6:0/2497842811 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3da409b120 0x7f3da40b1690 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f3d8c0097a0 tx=0x7f3d8c02b820 comp rx=0 tx=0).stop 2024-03-04T09:32:32.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.346+0000 7f3daa600640 1 -- 172.21.15.6:0/2497842811 shutdown_connections 2024-03-04T09:32:32.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.346+0000 7f3daa600640 1 --2- 172.21.15.6:0/2497842811 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3da409b120 0x7f3da40b1690 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:32.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.346+0000 7f3daa600640 1 --2- 172.21.15.6:0/2497842811 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3da40b5740 0x7f3da40b5b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:32.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.346+0000 7f3daa600640 1 -- 172.21.15.6:0/2497842811 >> 172.21.15.6:0/2497842811 conn(0x7f3da40731f0 msgr2=0x7f3da409aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:32:32.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.346+0000 7f3daa600640 1 -- 172.21.15.6:0/2497842811 shutdown_connections 2024-03-04T09:32:32.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.346+0000 7f3daa600640 1 -- 172.21.15.6:0/2497842811 wait complete. 2024-03-04T09:32:32.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.350+0000 7f3daa600640 1 Processor -- start 2024-03-04T09:32:32.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.350+0000 7f3daa600640 1 -- start start 2024-03-04T09:32:32.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.352+0000 7f3daa600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3da409b120 0x7f3da4158ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:32.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.352+0000 7f3daa600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3da40b5740 0x7f3da406ff60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:32.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.352+0000 7f3daa600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3da415f760 0x7f3da40704a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:32.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.352+0000 7f3daa600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3da40b1bd0 con 0x7f3da40b5740 2024-03-04T09:32:32.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.352+0000 7f3daa600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3da415d660 con 0x7f3da409b120 2024-03-04T09:32:32.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.352+0000 7f3daa600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3da40b1d50 con 0x7f3da415f760 2024-03-04T09:32:32.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.352+0000 7f3d9be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3da40b5740 0x7f3da406ff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:32.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.353+0000 7f3d9be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3da40b5740 0x7f3da406ff60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:44584/0 (socket says 172.21.15.6:44584) 2024-03-04T09:32:32.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.353+0000 7f3d9be00640 1 -- 172.21.15.6:0/1999175465 learned_addr learned my addr 172.21.15.6:0/1999175465 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:32:32.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.353+0000 7f3da9200640 1 --2- 172.21.15.6:0/1999175465 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3da409b120 0x7f3da4158ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:32.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.353+0000 7f3da9c00640 1 --2- 172.21.15.6:0/1999175465 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3da415f760 0x7f3da40704a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:32.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.353+0000 7f3d9be00640 1 -- 172.21.15.6:0/1999175465 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3da415f760 msgr2=0x7f3da40704a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:32.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.353+0000 7f3d9be00640 1 --2- 172.21.15.6:0/1999175465 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3da415f760 0x7f3da40704a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:32.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.353+0000 7f3d9be00640 1 -- 172.21.15.6:0/1999175465 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3da409b120 msgr2=0x7f3da4158ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:32.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.353+0000 7f3d9be00640 1 --2- 172.21.15.6:0/1999175465 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3da409b120 0x7f3da4158ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:32.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.353+0000 7f3d9be00640 1 -- 172.21.15.6:0/1999175465 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3da41499b0 con 0x7f3da40b5740 2024-03-04T09:32:32.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.353+0000 7f3d9be00640 1 --2- 172.21.15.6:0/1999175465 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3da40b5740 0x7f3da406ff60 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7f3d8c034040 tx=0x7f3d8c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:32.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.354+0000 7f3da1600640 1 -- 172.21.15.6:0/1999175465 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d8c032410 con 0x7f3da40b5740 2024-03-04T09:32:32.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.354+0000 7f3da1600640 1 -- 172.21.15.6:0/1999175465 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d8c0325b0 con 0x7f3da40b5740 2024-03-04T09:32:32.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.354+0000 7f3da1600640 1 -- 172.21.15.6:0/1999175465 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d8c032960 con 0x7f3da40b5740 2024-03-04T09:32:32.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.354+0000 7f3daa600640 1 -- 172.21.15.6:0/1999175465 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3da41f4200 con 0x7f3da40b5740 2024-03-04T09:32:32.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.354+0000 7f3daa600640 1 -- 172.21.15.6:0/1999175465 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3da41f47c0 con 0x7f3da40b5740 2024-03-04T09:32:32.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.355+0000 7f3daa600640 1 -- 172.21.15.6:0/1999175465 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3da404f330 con 0x7f3da40b5740 2024-03-04T09:32:32.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.356+0000 7f3da1600640 1 -- 172.21.15.6:0/1999175465 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3d8c032b00 con 0x7f3da40b5740 2024-03-04T09:32:32.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.357+0000 7f3da1600640 1 --2- 172.21.15.6:0/1999175465 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f3d8006b4d0 0x7f3d8006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:32.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.357+0000 7f3da1600640 1 -- 172.21.15.6:0/1999175465 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f3d8c0abf80 con 0x7f3da40b5740 2024-03-04T09:32:32.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.358+0000 7f3da9200640 1 --2- 172.21.15.6:0/1999175465 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f3d8006b4d0 0x7f3d8006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:32.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.359+0000 7f3da9200640 1 --2- 172.21.15.6:0/1999175465 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f3d8006b4d0 0x7f3d8006d990 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f3d940073f0 tx=0x7f3d9400b040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:32.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.367+0000 7f3da1600640 1 -- 172.21.15.6:0/1999175465 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3d8c040070 con 0x7f3da40b5740 2024-03-04T09:32:32.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.600+0000 7f3daa600640 1 -- 172.21.15.6:0/1999175465 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3da4072f90 con 0x7f3da40b5740 2024-03-04T09:32:32.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.602+0000 7f3da1600640 1 -- 172.21.15.6:0/1999175465 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+4975 (secure 0 0 0) 0x7f3d8c02f050 con 0x7f3da40b5740 2024-03-04T09:32:32.599 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:32:32.599 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":84,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6535,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/3883122012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":3883122012},{"type":"v1","addr":"172.21.15.146:6835","nonce":3883122012}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":6559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/4060411493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":4060411493},{"type":"v1","addr":"172.21.15.146:6837","nonce":4060411493}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":82},{"gid":7812,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/679471398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":679471398},{"type":"v1","addr":"172.21.15.6:6833","nonce":679471398}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:31:21.114351+0000","modified":"2024-03-04T09:31:40.781474+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7815},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7815":{"gid":7815,"name":"a","rank":0,"incarnation":81,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/882667161","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":882667161},{"type":"v1","addr":"172.21.15.6:6835","nonce":882667161}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7815,"qdb_cluster":[7815]},"id":8}]} 2024-03-04T09:32:32.599 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 84 2024-03-04T09:32:32.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.604+0000 7f3daa600640 1 -- 172.21.15.6:0/1999175465 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f3d8006b4d0 msgr2=0x7f3d8006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:32.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.604+0000 7f3daa600640 1 --2- 172.21.15.6:0/1999175465 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f3d8006b4d0 0x7f3d8006d990 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f3d940073f0 tx=0x7f3d9400b040 comp rx=0 tx=0).stop 2024-03-04T09:32:32.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.604+0000 7f3daa600640 1 -- 172.21.15.6:0/1999175465 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3da40b5740 msgr2=0x7f3da406ff60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:32.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.604+0000 7f3daa600640 1 --2- 172.21.15.6:0/1999175465 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3da40b5740 0x7f3da406ff60 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7f3d8c034040 tx=0x7f3d8c0046c0 comp rx=0 tx=0).stop 2024-03-04T09:32:32.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.604+0000 7f3daa600640 1 -- 172.21.15.6:0/1999175465 shutdown_connections 2024-03-04T09:32:32.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.604+0000 7f3daa600640 1 --2- 172.21.15.6:0/1999175465 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f3d8006b4d0 0x7f3d8006d990 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:32.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.604+0000 7f3daa600640 1 --2- 172.21.15.6:0/1999175465 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3da415f760 0x7f3da40704a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:32.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.604+0000 7f3daa600640 1 --2- 172.21.15.6:0/1999175465 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3da40b5740 0x7f3da406ff60 unknown :-1 s=CLOSED pgs=1378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:32.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.604+0000 7f3daa600640 1 --2- 172.21.15.6:0/1999175465 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3da409b120 0x7f3da4158ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:32.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.604+0000 7f3daa600640 1 -- 172.21.15.6:0/1999175465 >> 172.21.15.6:0/1999175465 conn(0x7f3da40731f0 msgr2=0x7f3da40b1f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:32:32.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.604+0000 7f3daa600640 1 -- 172.21.15.6:0/1999175465 shutdown_connections 2024-03-04T09:32:32.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.604+0000 7f3daa600640 1 -- 172.21.15.6:0/1999175465 wait complete. 2024-03-04T09:32:32.621 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump purge_queue 2024-03-04T09:32:32.838 INFO:teuthology.orchestra.run.smithi006.stdout:{"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":2,"pq_executing":0,"pq_executing_high_water":1,"pq_executed_ops":2,"pq_executed":1,"pq_item_in_journal":0}} 2024-03-04T09:32:32.839 DEBUG:tasks.cephfs.filesystem:_json_asok output { "purge_queue": { "pq_executed": 1, "pq_executed_ops": 2, "pq_executing": 0, "pq_executing_high_water": 1, "pq_executing_ops": 0, "pq_executing_ops_high_water": 2, "pq_item_in_journal": 0 } } 2024-03-04T09:32:32.839 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data ls 2024-03-04T09:32:32.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.942+0000 7ff6448f9f40 1 Processor -- start 2024-03-04T09:32:32.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.942+0000 7ff6448f9f40 1 -- start start 2024-03-04T09:32:32.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.943+0000 7ff6448f9f40 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff6280022d0 0x55d9365ee980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:32.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.943+0000 7ff6448f9f40 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55d9366899d0 0x55d93668a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:32.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.943+0000 7ff6448f9f40 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55d936689db0 0x55d936727d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:32.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.943+0000 7ff6448f9f40 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d936566fd0 con 0x7ff6280022d0 2024-03-04T09:32:32.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.943+0000 7ff6448f9f40 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d93656af30 con 0x55d9366899d0 2024-03-04T09:32:32.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.943+0000 7ff6448f9f40 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d936609ba0 con 0x55d936689db0 2024-03-04T09:32:32.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.944+0000 7ff641a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff6280022d0 0x55d9365ee980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:32.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.944+0000 7ff641a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff6280022d0 0x55d9365ee980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:44612/0 (socket says 172.21.15.6:44612) 2024-03-04T09:32:32.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.944+0000 7ff641a00640 1 -- 172.21.15.6:0/2911277955 learned_addr learned my addr 172.21.15.6:0/2911277955 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:32:32.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.944+0000 7ff641000640 1 --2- 172.21.15.6:0/2911277955 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55d9366899d0 0x55d93668a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:32.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.944+0000 7ff641a00640 1 -- 172.21.15.6:0/2911277955 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55d936689db0 msgr2=0x55d936727d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:32:32.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.944+0000 7ff641a00640 1 --2- 172.21.15.6:0/2911277955 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55d936689db0 0x55d936727d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:32.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.944+0000 7ff641a00640 1 -- 172.21.15.6:0/2911277955 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55d9366899d0 msgr2=0x55d93668a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:32:32.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.944+0000 7ff641a00640 1 --2- 172.21.15.6:0/2911277955 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55d9366899d0 0x55d93668a8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:32.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.944+0000 7ff641a00640 1 -- 172.21.15.6:0/2911277955 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d9365bfe40 con 0x7ff6280022d0 2024-03-04T09:32:32.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.944+0000 7ff641a00640 1 --2- 172.21.15.6:0/2911277955 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff6280022d0 0x55d9365ee980 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7ff6340098b0 tx=0x7ff63404d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bea77a354ff584b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:32.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.945+0000 7ff63b400640 1 -- 172.21.15.6:0/2911277955 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff634064070 con 0x7ff6280022d0 2024-03-04T09:32:32.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.945+0000 7ff63b400640 1 -- 172.21.15.6:0/2911277955 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6340017a0 con 0x7ff6280022d0 2024-03-04T09:32:32.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.945+0000 7ff63b400640 1 -- 172.21.15.6:0/2911277955 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff63404ba00 con 0x7ff6280022d0 2024-03-04T09:32:32.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.945+0000 7ff6448f9f40 1 -- 172.21.15.6:0/2911277955 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff6280022d0 msgr2=0x55d9365ee980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:32.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.945+0000 7ff6448f9f40 1 --2- 172.21.15.6:0/2911277955 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff6280022d0 0x55d9365ee980 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7ff6340098b0 tx=0x7ff63404d000 comp rx=0 tx=0).stop 2024-03-04T09:32:32.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.945+0000 7ff6448f9f40 1 -- 172.21.15.6:0/2911277955 shutdown_connections 2024-03-04T09:32:32.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.945+0000 7ff6448f9f40 1 --2- 172.21.15.6:0/2911277955 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55d936689db0 0x55d936727d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:32.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.945+0000 7ff6448f9f40 1 --2- 172.21.15.6:0/2911277955 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55d9366899d0 0x55d93668a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:32.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.945+0000 7ff6448f9f40 1 --2- 172.21.15.6:0/2911277955 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff6280022d0 0x55d9365ee980 unknown :-1 s=CLOSED pgs=1381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:32.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.945+0000 7ff6448f9f40 1 -- 172.21.15.6:0/2911277955 >> 172.21.15.6:0/2911277955 conn(0x55d9365eb660 msgr2=0x55d9365f0a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:32:32.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.945+0000 7ff6448f9f40 1 -- 172.21.15.6:0/2911277955 shutdown_connections 2024-03-04T09:32:32.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.945+0000 7ff6448f9f40 1 -- 172.21.15.6:0/2911277955 wait complete. 2024-03-04T09:32:32.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.946+0000 7ff6448f9f40 1 Processor -- start 2024-03-04T09:32:32.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.946+0000 7ff6448f9f40 1 -- start start 2024-03-04T09:32:32.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.947+0000 7ff6448f9f40 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55d9366899d0 0x55d9366849d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:32.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.947+0000 7ff6448f9f40 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d936689db0 0x55d93667ec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:32.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.947+0000 7ff6448f9f40 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff6280022d0 0x55d93667f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:32.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.947+0000 7ff6448f9f40 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d93668a680 con 0x55d936689db0 2024-03-04T09:32:32.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.947+0000 7ff6448f9f40 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d9365bfa90 con 0x55d9366899d0 2024-03-04T09:32:32.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.947+0000 7ff6448f9f40 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d9365eeec0 con 0x7ff6280022d0 2024-03-04T09:32:32.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.947+0000 7ff641000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d936689db0 0x55d93667ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:32.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.947+0000 7ff641000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d936689db0 0x55d93667ec60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:44616/0 (socket says 172.21.15.6:44616) 2024-03-04T09:32:32.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.947+0000 7ff641000640 1 -- 172.21.15.6:0/3921300397 learned_addr learned my addr 172.21.15.6:0/3921300397 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:32:32.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.948+0000 7ff641a00640 1 --2- 172.21.15.6:0/3921300397 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55d9366899d0 0x55d9366849d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:32.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.948+0000 7ff642400640 1 --2- 172.21.15.6:0/3921300397 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff6280022d0 0x55d93667f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:32.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.948+0000 7ff641000640 1 -- 172.21.15.6:0/3921300397 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff6280022d0 msgr2=0x55d93667f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:32.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.948+0000 7ff641000640 1 --2- 172.21.15.6:0/3921300397 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff6280022d0 0x55d93667f1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:32.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.948+0000 7ff641000640 1 -- 172.21.15.6:0/3921300397 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55d9366899d0 msgr2=0x55d9366849d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:32.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.948+0000 7ff641000640 1 --2- 172.21.15.6:0/3921300397 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55d9366899d0 0x55d9366849d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:32.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.948+0000 7ff641000640 1 -- 172.21.15.6:0/3921300397 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d9366843e0 con 0x55d936689db0 2024-03-04T09:32:32.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.948+0000 7ff642400640 1 --2- 172.21.15.6:0/3921300397 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff6280022d0 0x55d93667f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:32:32.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.948+0000 7ff641000640 1 --2- 172.21.15.6:0/3921300397 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d936689db0 0x55d93667ec60 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7ff630009e10 tx=0x7ff630009f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:32.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.948+0000 7ff63a000640 1 -- 172.21.15.6:0/3921300397 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff630015400 con 0x55d936689db0 2024-03-04T09:32:32.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.948+0000 7ff63a000640 1 -- 172.21.15.6:0/3921300397 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6300155a0 con 0x55d936689db0 2024-03-04T09:32:32.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.948+0000 7ff63a000640 1 -- 172.21.15.6:0/3921300397 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff630015990 con 0x55d936689db0 2024-03-04T09:32:32.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.948+0000 7ff6448f9f40 1 -- 172.21.15.6:0/3921300397 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55d93668aa30 con 0x55d936689db0 2024-03-04T09:32:32.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.949+0000 7ff6448f9f40 1 -- 172.21.15.6:0/3921300397 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d936684f10 con 0x55d936689db0 2024-03-04T09:32:32.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.951+0000 7ff63a000640 1 -- 172.21.15.6:0/3921300397 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff630015b30 con 0x55d936689db0 2024-03-04T09:32:32.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.952+0000 7ff63a000640 1 --2- 172.21.15.6:0/3921300397 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff624069eb0 0x7ff62406c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:32.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.952+0000 7ff63a000640 1 -- 172.21.15.6:0/3921300397 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5853+0+0 (secure 0 0 0) 0x7ff630088bf0 con 0x55d936689db0 2024-03-04T09:32:32.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.953+0000 7ff6448f9f40 1 --2- 172.21.15.6:0/3921300397 >> [v2:172.21.15.146:6816/3268323156,v1:172.21.15.146:6817/3268323156] conn(0x55d93668b4f0 0x55d93668b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:32.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.953+0000 7ff6448f9f40 1 -- 172.21.15.6:0/3921300397 --> [v2:172.21.15.146:6816/3268323156,v1:172.21.15.146:6817/3268323156] -- osd_op(unknown.0.0:1 17.0 17:00000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x55d93667c4c0 con 0x55d93668b4f0 2024-03-04T09:32:32.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.953+0000 7ff641a00640 1 --2- 172.21.15.6:0/3921300397 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff624069eb0 0x7ff62406c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:32.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.953+0000 7ff642400640 1 --2- 172.21.15.6:0/3921300397 >> [v2:172.21.15.146:6816/3268323156,v1:172.21.15.146:6817/3268323156] conn(0x55d93668b4f0 0x55d93668b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:32.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.954+0000 7ff642400640 1 --2- 172.21.15.6:0/3921300397 >> [v2:172.21.15.146:6816/3268323156,v1:172.21.15.146:6817/3268323156] conn(0x55d93668b4f0 0x55d93668b8d0 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:32.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.954+0000 7ff641a00640 1 --2- 172.21.15.6:0/3921300397 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff624069eb0 0x7ff62406c370 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7ff63401c7b0 tx=0x7ff63404f000 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:32.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.955+0000 7ff642400640 1 -- 172.21.15.6:0/3921300397 <== osd.7 v2:172.21.15.146:6816/3268323156 1 ==== osd_op_reply(1 [pgnls start_epoch 115 out=49b] v109'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff63c004430 con 0x55d93668b4f0 2024-03-04T09:32:32.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.955+0000 7ff642400640 1 --2- 172.21.15.6:0/3921300397 >> [v2:172.21.15.6:6808/2494103078,v1:172.21.15.6:6809/2494103078] conn(0x55d9365eb1d0 0x7ff63c002a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:32.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.955+0000 7ff642400640 1 -- 172.21.15.6:0/3921300397 --> [v2:172.21.15.6:6808/2494103078,v1:172.21.15.6:6809/2494103078] -- osd_op(unknown.0.0:2 17.20 17:04000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x7ff63c004b70 con 0x55d9365eb1d0 2024-03-04T09:32:32.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.956+0000 7ff641a00640 1 --2- 172.21.15.6:0/3921300397 >> [v2:172.21.15.6:6808/2494103078,v1:172.21.15.6:6809/2494103078] conn(0x55d9365eb1d0 0x7ff63c002a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:32.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.956+0000 7ff641a00640 1 --2- 172.21.15.6:0/3921300397 >> [v2:172.21.15.6:6808/2494103078,v1:172.21.15.6:6809/2494103078] conn(0x55d9365eb1d0 0x7ff63c002a90 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:32.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.957+0000 7ff641a00640 1 -- 172.21.15.6:0/3921300397 <== osd.1 v2:172.21.15.6:6808/2494103078 1 ==== osd_op_reply(2 [pgnls start_epoch 115 out=49b] v109'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff634059070 con 0x55d9365eb1d0 2024-03-04T09:32:32.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.957+0000 7ff641a00640 1 --2- 172.21.15.6:0/3921300397 >> [v2:172.21.15.6:6800/1558806123,v1:172.21.15.6:6801/1558806123] conn(0x7ff63405fac0 0x7ff634053410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:32.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.957+0000 7ff641a00640 1 -- 172.21.15.6:0/3921300397 --> [v2:172.21.15.6:6800/1558806123,v1:172.21.15.6:6801/1558806123] -- osd_op(unknown.0.0:3 17.10 17:08000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x7ff63c004b70 con 0x7ff63405fac0 2024-03-04T09:32:32.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.957+0000 7ff641000640 1 --2- 172.21.15.6:0/3921300397 >> [v2:172.21.15.6:6800/1558806123,v1:172.21.15.6:6801/1558806123] conn(0x7ff63405fac0 0x7ff634053410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:32.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.958+0000 7ff641000640 1 --2- 172.21.15.6:0/3921300397 >> [v2:172.21.15.6:6800/1558806123,v1:172.21.15.6:6801/1558806123] conn(0x7ff63405fac0 0x7ff634053410 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:32.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.958+0000 7ff641000640 1 -- 172.21.15.6:0/3921300397 <== osd.3 v2:172.21.15.6:6800/1558806123 1 ==== osd_op_reply(3 [pgnls start_epoch 115 out=49b] v109'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff630013070 con 0x7ff63405fac0 2024-03-04T09:32:32.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.958+0000 7ff641000640 1 --2- 172.21.15.6:0/3921300397 >> [v2:172.21.15.6:6823/1364449401,v1:172.21.15.6:6825/1364449401] conn(0x7ff630057530 0x7ff630056190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:32.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.958+0000 7ff641000640 1 -- 172.21.15.6:0/3921300397 --> [v2:172.21.15.6:6823/1364449401,v1:172.21.15.6:6825/1364449401] -- osd_op(unknown.0.0:4 17.30 17:0c000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x7ff63c004b70 con 0x7ff630057530 2024-03-04T09:32:32.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.958+0000 7ff642400640 1 --2- 172.21.15.6:0/3921300397 >> [v2:172.21.15.6:6823/1364449401,v1:172.21.15.6:6825/1364449401] conn(0x7ff630057530 0x7ff630056190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:32.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.959+0000 7ff642400640 1 --2- 172.21.15.6:0/3921300397 >> [v2:172.21.15.6:6823/1364449401,v1:172.21.15.6:6825/1364449401] conn(0x7ff630057530 0x7ff630056190 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:32.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.959+0000 7ff642400640 1 -- 172.21.15.6:0/3921300397 <== osd.0 v2:172.21.15.6:6823/1364449401 1 ==== osd_op_reply(4 [pgnls start_epoch 115 out=49b] v109'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff63c004430 con 0x7ff630057530 2024-03-04T09:32:32.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.959+0000 7ff642400640 1 -- 172.21.15.6:0/3921300397 --> [v2:172.21.15.146:6816/3268323156,v1:172.21.15.146:6817/3268323156] -- osd_op(unknown.0.0:5 17.8 17:10000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x7ff63c004b70 con 0x55d93668b4f0 2024-03-04T09:32:32.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.960+0000 7ff642400640 1 -- 172.21.15.6:0/3921300397 <== osd.7 v2:172.21.15.146:6816/3268323156 2 ==== osd_op_reply(5 [pgnls start_epoch 115 out=49b] v109'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff63c004430 con 0x55d93668b4f0 2024-03-04T09:32:32.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.960+0000 7ff642400640 1 -- 172.21.15.6:0/3921300397 --> [v2:172.21.15.6:6800/1558806123,v1:172.21.15.6:6801/1558806123] -- osd_op(unknown.0.0:6 17.28 17:14000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x7ff63c004b70 con 0x7ff63405fac0 2024-03-04T09:32:32.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.960+0000 7ff641000640 1 -- 172.21.15.6:0/3921300397 <== osd.3 v2:172.21.15.6:6800/1558806123 2 ==== osd_op_reply(6 [pgnls start_epoch 115 out=49b] v109'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff630013070 con 0x7ff63405fac0 2024-03-04T09:32:32.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.960+0000 7ff641000640 1 -- 172.21.15.6:0/3921300397 --> [v2:172.21.15.6:6823/1364449401,v1:172.21.15.6:6825/1364449401] -- osd_op(unknown.0.0:7 17.18 17:18000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x7ff63c004b70 con 0x7ff630057530 2024-03-04T09:32:32.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.961+0000 7ff642400640 1 -- 172.21.15.6:0/3921300397 <== osd.0 v2:172.21.15.6:6823/1364449401 2 ==== osd_op_reply(7 [pgnls start_epoch 115 out=49b] v109'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff63c004430 con 0x7ff630057530 2024-03-04T09:32:32.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.961+0000 7ff642400640 1 -- 172.21.15.6:0/3921300397 --> [v2:172.21.15.146:6816/3268323156,v1:172.21.15.146:6817/3268323156] -- osd_op(unknown.0.0:8 17.38 17:1c000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x7ff63c004b70 con 0x55d93668b4f0 2024-03-04T09:32:32.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.962+0000 7ff642400640 1 -- 172.21.15.6:0/3921300397 <== osd.7 v2:172.21.15.146:6816/3268323156 3 ==== osd_op_reply(8 [pgnls start_epoch 115 out=49b] v109'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff63c004430 con 0x55d93668b4f0 2024-03-04T09:32:32.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.962+0000 7ff642400640 1 -- 172.21.15.6:0/3921300397 --> [v2:172.21.15.6:6823/1364449401,v1:172.21.15.6:6825/1364449401] -- osd_op(unknown.0.0:9 17.4 17:20000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x7ff63c004b70 con 0x7ff630057530 2024-03-04T09:32:32.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.962+0000 7ff642400640 1 -- 172.21.15.6:0/3921300397 <== osd.0 v2:172.21.15.6:6823/1364449401 3 ==== osd_op_reply(9 [pgnls start_epoch 115 out=49b] v109'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff63c004430 con 0x7ff630057530 2024-03-04T09:32:32.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.962+0000 7ff642400640 1 -- 172.21.15.6:0/3921300397 --> [v2:172.21.15.6:6800/1558806123,v1:172.21.15.6:6801/1558806123] -- osd_op(unknown.0.0:10 17.24 17:24000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x7ff63c004b70 con 0x7ff63405fac0 2024-03-04T09:32:32.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.962+0000 7ff641000640 1 -- 172.21.15.6:0/3921300397 <== osd.3 v2:172.21.15.6:6800/1558806123 3 ==== osd_op_reply(10 [pgnls start_epoch 115 out=49b] v109'4 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff630013070 con 0x7ff63405fac0 2024-03-04T09:32:32.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.962+0000 7ff641000640 1 -- 172.21.15.6:0/3921300397 --> [v2:172.21.15.6:6800/1558806123,v1:172.21.15.6:6801/1558806123] -- osd_op(unknown.0.0:11 17.14 17:28000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x7ff63c004b70 con 0x7ff63405fac0 2024-03-04T09:32:32.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.963+0000 7ff641000640 1 -- 172.21.15.6:0/3921300397 <== osd.3 v2:172.21.15.6:6800/1558806123 4 ==== osd_op_reply(11 [pgnls start_epoch 115 out=49b] v109'11 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff630013070 con 0x7ff63405fac0 2024-03-04T09:32:32.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.963+0000 7ff641000640 1 -- 172.21.15.6:0/3921300397 --> [v2:172.21.15.6:6823/1364449401,v1:172.21.15.6:6825/1364449401] -- osd_op(unknown.0.0:12 17.34 17:2c000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x7ff63c004b70 con 0x7ff630057530 2024-03-04T09:32:32.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.963+0000 7ff642400640 1 -- 172.21.15.6:0/3921300397 <== osd.0 v2:172.21.15.6:6823/1364449401 4 ==== osd_op_reply(12 [pgnls start_epoch 115 out=49b] v109'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff63c004430 con 0x7ff630057530 2024-03-04T09:32:32.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.963+0000 7ff642400640 1 -- 172.21.15.6:0/3921300397 --> [v2:172.21.15.146:6816/3268323156,v1:172.21.15.146:6817/3268323156] -- osd_op(unknown.0.0:13 17.c 17:30000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x7ff63c004b70 con 0x55d93668b4f0 2024-03-04T09:32:32.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.964+0000 7ff642400640 1 -- 172.21.15.6:0/3921300397 <== osd.7 v2:172.21.15.146:6816/3268323156 4 ==== osd_op_reply(13 [pgnls start_epoch 115 out=49b] v109'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff63c004430 con 0x55d93668b4f0 2024-03-04T09:32:32.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.964+0000 7ff642400640 1 --2- 172.21.15.6:0/3921300397 >> [v2:172.21.15.146:6800/1526135780,v1:172.21.15.146:6801/1526135780] conn(0x55d9365eca80 0x7ff63c018950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:32.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.964+0000 7ff642400640 1 -- 172.21.15.6:0/3921300397 --> [v2:172.21.15.146:6800/1526135780,v1:172.21.15.146:6801/1526135780] -- osd_op(unknown.0.0:14 17.2c 17:34000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x7ff63c004b70 con 0x55d9365eca80 2024-03-04T09:32:32.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.965+0000 7ff641a00640 1 --2- 172.21.15.6:0/3921300397 >> [v2:172.21.15.146:6800/1526135780,v1:172.21.15.146:6801/1526135780] conn(0x55d9365eca80 0x7ff63c018950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:32.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.966+0000 7ff641a00640 1 --2- 172.21.15.6:0/3921300397 >> [v2:172.21.15.146:6800/1526135780,v1:172.21.15.146:6801/1526135780] conn(0x55d9365eca80 0x7ff63c018950 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:32.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.967+0000 7ff641a00640 1 -- 172.21.15.6:0/3921300397 <== osd.4 v2:172.21.15.146:6800/1526135780 1 ==== osd_op_reply(14 [pgnls start_epoch 115 out=49b] v109'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff634059070 con 0x55d9365eca80 2024-03-04T09:32:32.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.967+0000 7ff641a00640 1 -- 172.21.15.6:0/3921300397 --> [v2:172.21.15.6:6808/2494103078,v1:172.21.15.6:6809/2494103078] -- osd_op(unknown.0.0:15 17.1c 17:38000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x7ff63c004b70 con 0x55d9365eb1d0 2024-03-04T09:32:32.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.967+0000 7ff641a00640 1 -- 172.21.15.6:0/3921300397 <== osd.1 v2:172.21.15.6:6808/2494103078 2 ==== osd_op_reply(15 [pgnls start_epoch 115 out=49b] v109'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff634059070 con 0x55d9365eb1d0 2024-03-04T09:32:32.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.967+0000 7ff641a00640 1 -- 172.21.15.6:0/3921300397 --> [v2:172.21.15.6:6823/1364449401,v1:172.21.15.6:6825/1364449401] -- osd_op(unknown.0.0:16 17.3c 17:3c000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x7ff63c004b70 con 0x7ff630057530 2024-03-04T09:32:32.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.968+0000 7ff642400640 1 -- 172.21.15.6:0/3921300397 <== osd.0 v2:172.21.15.6:6823/1364449401 5 ==== osd_op_reply(16 [pgnls start_epoch 115 out=49b] v109'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff63c004430 con 0x7ff630057530 2024-03-04T09:32:32.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.968+0000 7ff642400640 1 -- 172.21.15.6:0/3921300397 --> [v2:172.21.15.146:6800/1526135780,v1:172.21.15.146:6801/1526135780] -- osd_op(unknown.0.0:17 17.2 17:40000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x7ff63c004b70 con 0x55d9365eca80 2024-03-04T09:32:32.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.968+0000 7ff641a00640 1 -- 172.21.15.6:0/3921300397 <== osd.4 v2:172.21.15.146:6800/1526135780 2 ==== osd_op_reply(17 [pgnls start_epoch 115 out=49b] v109'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff634059070 con 0x55d9365eca80 2024-03-04T09:32:32.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.968+0000 7ff641a00640 1 --2- 172.21.15.6:0/3921300397 >> [v2:172.21.15.146:6808/460262507,v1:172.21.15.146:6809/460262507] conn(0x7ff63406b910 0x7ff6340793f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:32.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.968+0000 7ff641a00640 1 -- 172.21.15.6:0/3921300397 --> [v2:172.21.15.146:6808/460262507,v1:172.21.15.146:6809/460262507] -- osd_op(unknown.0.0:18 17.22 17:44000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x7ff63c004b70 con 0x7ff63406b910 2024-03-04T09:32:32.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.969+0000 7ff641000640 1 --2- 172.21.15.6:0/3921300397 >> [v2:172.21.15.146:6808/460262507,v1:172.21.15.146:6809/460262507] conn(0x7ff63406b910 0x7ff6340793f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:32.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.970+0000 7ff641000640 1 --2- 172.21.15.6:0/3921300397 >> [v2:172.21.15.146:6808/460262507,v1:172.21.15.146:6809/460262507] conn(0x7ff63406b910 0x7ff6340793f0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:32.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.971+0000 7ff641000640 1 -- 172.21.15.6:0/3921300397 <== osd.5 v2:172.21.15.146:6808/460262507 1 ==== osd_op_reply(18 [pgnls start_epoch 115 out=49b] v109'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff630013070 con 0x7ff63406b910 2024-03-04T09:32:32.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.971+0000 7ff641000640 1 -- 172.21.15.6:0/3921300397 --> [v2:172.21.15.6:6800/1558806123,v1:172.21.15.6:6801/1558806123] -- osd_op(unknown.0.0:19 17.12 17:48000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x7ff63c004b70 con 0x7ff63405fac0 2024-03-04T09:32:32.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.971+0000 7ff641000640 1 -- 172.21.15.6:0/3921300397 <== osd.3 v2:172.21.15.6:6800/1558806123 5 ==== osd_op_reply(19 [pgnls start_epoch 115 out=49b] v109'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff630013070 con 0x7ff63405fac0 2024-03-04T09:32:32.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.971+0000 7ff641000640 1 -- 172.21.15.6:0/3921300397 --> [v2:172.21.15.146:6816/3268323156,v1:172.21.15.146:6817/3268323156] -- osd_op(unknown.0.0:20 17.32 17:4c000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x7ff63c004b70 con 0x55d93668b4f0 2024-03-04T09:32:32.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.972+0000 7ff642400640 1 -- 172.21.15.6:0/3921300397 <== osd.7 v2:172.21.15.146:6816/3268323156 5 ==== osd_op_reply(20 [pgnls start_epoch 115 out=49b] v109'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff63c004430 con 0x55d93668b4f0 2024-03-04T09:32:32.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.972+0000 7ff642400640 1 -- 172.21.15.6:0/3921300397 --> [v2:172.21.15.146:6816/3268323156,v1:172.21.15.146:6817/3268323156] -- osd_op(unknown.0.0:21 17.a 17:50000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x7ff63c004b70 con 0x55d93668b4f0 2024-03-04T09:32:32.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.973+0000 7ff642400640 1 -- 172.21.15.6:0/3921300397 <== osd.7 v2:172.21.15.146:6816/3268323156 6 ==== osd_op_reply(21 [pgnls start_epoch 115 out=49b] v109'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff63c004430 con 0x55d93668b4f0 2024-03-04T09:32:32.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.973+0000 7ff642400640 1 -- 172.21.15.6:0/3921300397 --> [v2:172.21.15.6:6800/1558806123,v1:172.21.15.6:6801/1558806123] -- osd_op(unknown.0.0:22 17.2a 17:54000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x7ff63c004b70 con 0x7ff63405fac0 2024-03-04T09:32:32.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.974+0000 7ff641000640 1 -- 172.21.15.6:0/3921300397 <== osd.3 v2:172.21.15.6:6800/1558806123 6 ==== osd_op_reply(22 [pgnls start_epoch 115 out=49b] v109'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff630013070 con 0x7ff63405fac0 2024-03-04T09:32:32.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.974+0000 7ff641000640 1 -- 172.21.15.6:0/3921300397 --> [v2:172.21.15.6:6823/1364449401,v1:172.21.15.6:6825/1364449401] -- osd_op(unknown.0.0:23 17.1a 17:58000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x7ff63c004b70 con 0x7ff630057530 2024-03-04T09:32:32.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.974+0000 7ff642400640 1 -- 172.21.15.6:0/3921300397 <== osd.0 v2:172.21.15.6:6823/1364449401 6 ==== osd_op_reply(23 [pgnls start_epoch 115 out=49b] v109'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff63c004430 con 0x7ff630057530 2024-03-04T09:32:32.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.974+0000 7ff642400640 1 -- 172.21.15.6:0/3921300397 --> [v2:172.21.15.6:6808/2494103078,v1:172.21.15.6:6809/2494103078] -- osd_op(unknown.0.0:24 17.3a 17:5c000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x7ff63c004b70 con 0x55d9365eb1d0 2024-03-04T09:32:32.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.975+0000 7ff641a00640 1 -- 172.21.15.6:0/3921300397 <== osd.1 v2:172.21.15.6:6808/2494103078 3 ==== osd_op_reply(24 [pgnls start_epoch 115 out=49b] v109'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff634059070 con 0x55d9365eb1d0 2024-03-04T09:32:32.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.975+0000 7ff641a00640 1 --2- 172.21.15.6:0/3921300397 >> [v2:172.21.15.146:6821/1392897493,v1:172.21.15.146:6823/1392897493] conn(0x7ff63405e3e0 0x7ff63405e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:32.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.975+0000 7ff641a00640 1 -- 172.21.15.6:0/3921300397 --> [v2:172.21.15.146:6821/1392897493,v1:172.21.15.146:6823/1392897493] -- osd_op(unknown.0.0:25 17.6 17:60000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x7ff63c004b70 con 0x7ff63405e3e0 2024-03-04T09:32:32.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.976+0000 7ff642400640 1 --2- 172.21.15.6:0/3921300397 >> [v2:172.21.15.146:6821/1392897493,v1:172.21.15.146:6823/1392897493] conn(0x7ff63405e3e0 0x7ff63405e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:32.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.977+0000 7ff642400640 1 --2- 172.21.15.6:0/3921300397 >> [v2:172.21.15.146:6821/1392897493,v1:172.21.15.146:6823/1392897493] conn(0x7ff63405e3e0 0x7ff63405e840 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:32.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.978+0000 7ff642400640 1 -- 172.21.15.6:0/3921300397 <== osd.6 v2:172.21.15.146:6821/1392897493 1 ==== osd_op_reply(25 [pgnls start_epoch 115 out=49b] v109'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff63c037040 con 0x7ff63405e3e0 2024-03-04T09:32:32.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.978+0000 7ff642400640 1 --2- 172.21.15.6:0/3921300397 >> [v2:172.21.15.6:6810/1500066681,v1:172.21.15.6:6812/1500066681] conn(0x55d936695320 0x7ff63c02a440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:32.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.978+0000 7ff642400640 1 -- 172.21.15.6:0/3921300397 --> [v2:172.21.15.6:6810/1500066681,v1:172.21.15.6:6812/1500066681] -- osd_op(unknown.0.0:26 17.26 17:64000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x7ff63c004b70 con 0x55d936695320 2024-03-04T09:32:32.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.978+0000 7ff641a00640 1 --2- 172.21.15.6:0/3921300397 >> [v2:172.21.15.6:6810/1500066681,v1:172.21.15.6:6812/1500066681] conn(0x55d936695320 0x7ff63c02a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:32.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.979+0000 7ff641a00640 1 --2- 172.21.15.6:0/3921300397 >> [v2:172.21.15.6:6810/1500066681,v1:172.21.15.6:6812/1500066681] conn(0x55d936695320 0x7ff63c02a440 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:32.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.979+0000 7ff641a00640 1 -- 172.21.15.6:0/3921300397 <== osd.2 v2:172.21.15.6:6810/1500066681 1 ==== osd_op_reply(26 [pgnls start_epoch 115 out=49b] v109'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff634059070 con 0x55d936695320 2024-03-04T09:32:32.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.979+0000 7ff641a00640 1 -- 172.21.15.6:0/3921300397 --> [v2:172.21.15.146:6808/460262507,v1:172.21.15.146:6809/460262507] -- osd_op(unknown.0.0:27 17.16 17:68000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x7ff63c004b70 con 0x7ff63406b910 2024-03-04T09:32:32.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.980+0000 7ff641000640 1 -- 172.21.15.6:0/3921300397 <== osd.5 v2:172.21.15.146:6808/460262507 2 ==== osd_op_reply(27 [pgnls start_epoch 115 out=49b] v109'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff630013070 con 0x7ff63406b910 2024-03-04T09:32:32.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.980+0000 7ff641000640 1 -- 172.21.15.6:0/3921300397 --> [v2:172.21.15.6:6823/1364449401,v1:172.21.15.6:6825/1364449401] -- osd_op(unknown.0.0:28 17.36 17:6c000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x7ff63c004b70 con 0x7ff630057530 2024-03-04T09:32:32.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.981+0000 7ff642400640 1 -- 172.21.15.6:0/3921300397 <== osd.0 v2:172.21.15.6:6823/1364449401 7 ==== osd_op_reply(28 [pgnls start_epoch 115 out=49b] v109'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff63c037040 con 0x7ff630057530 2024-03-04T09:32:32.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.981+0000 7ff642400640 1 -- 172.21.15.6:0/3921300397 --> [v2:172.21.15.6:6810/1500066681,v1:172.21.15.6:6812/1500066681] -- osd_op(unknown.0.0:29 17.e 17:70000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x7ff63c004b70 con 0x55d936695320 2024-03-04T09:32:32.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.981+0000 7ff641a00640 1 -- 172.21.15.6:0/3921300397 <== osd.2 v2:172.21.15.6:6810/1500066681 2 ==== osd_op_reply(29 [pgnls start_epoch 115 out=49b] v109'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff634059070 con 0x55d936695320 2024-03-04T09:32:32.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.981+0000 7ff641a00640 1 -- 172.21.15.6:0/3921300397 --> [v2:172.21.15.146:6821/1392897493,v1:172.21.15.146:6823/1392897493] -- osd_op(unknown.0.0:30 17.2e 17:74000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x7ff63c004b70 con 0x7ff63405e3e0 2024-03-04T09:32:32.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.982+0000 7ff642400640 1 -- 172.21.15.6:0/3921300397 <== osd.6 v2:172.21.15.146:6821/1392897493 2 ==== osd_op_reply(30 [pgnls start_epoch 115 out=49b] v109'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff63c037040 con 0x7ff63405e3e0 2024-03-04T09:32:32.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.983+0000 7ff642400640 1 -- 172.21.15.6:0/3921300397 --> [v2:172.21.15.6:6808/2494103078,v1:172.21.15.6:6809/2494103078] -- osd_op(unknown.0.0:31 17.1e 17:78000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x7ff63c004b70 con 0x55d9365eb1d0 2024-03-04T09:32:32.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.983+0000 7ff641a00640 1 -- 172.21.15.6:0/3921300397 <== osd.1 v2:172.21.15.6:6808/2494103078 4 ==== osd_op_reply(31 [pgnls start_epoch 115 out=49b] v109'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff634059070 con 0x55d9365eb1d0 2024-03-04T09:32:32.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.983+0000 7ff641a00640 1 -- 172.21.15.6:0/3921300397 --> [v2:172.21.15.146:6800/1526135780,v1:172.21.15.146:6801/1526135780] -- osd_op(unknown.0.0:32 17.3e 17:7c000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x7ff63c004b70 con 0x55d9365eca80 2024-03-04T09:32:32.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.984+0000 7ff641a00640 1 -- 172.21.15.6:0/3921300397 <== osd.4 v2:172.21.15.146:6800/1526135780 3 ==== osd_op_reply(32 [pgnls start_epoch 115 out=49b] v109'4 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff634059070 con 0x55d9365eca80 2024-03-04T09:32:32.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.984+0000 7ff641a00640 1 -- 172.21.15.6:0/3921300397 --> [v2:172.21.15.6:6823/1364449401,v1:172.21.15.6:6825/1364449401] -- osd_op(unknown.0.0:33 17.1 17:80000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x7ff63c004b70 con 0x7ff630057530 2024-03-04T09:32:32.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.985+0000 7ff642400640 1 -- 172.21.15.6:0/3921300397 <== osd.0 v2:172.21.15.6:6823/1364449401 8 ==== osd_op_reply(33 [pgnls start_epoch 115 out=49b] v109'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff63c037040 con 0x7ff630057530 2024-03-04T09:32:32.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.985+0000 7ff642400640 1 -- 172.21.15.6:0/3921300397 --> [v2:172.21.15.6:6823/1364449401,v1:172.21.15.6:6825/1364449401] -- osd_op(unknown.0.0:34 17.21 17:84000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x7ff63c004b70 con 0x7ff630057530 2024-03-04T09:32:32.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.986+0000 7ff642400640 1 -- 172.21.15.6:0/3921300397 <== osd.0 v2:172.21.15.6:6823/1364449401 9 ==== osd_op_reply(34 [pgnls start_epoch 115 out=49b] v109'12 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff63c037040 con 0x7ff630057530 2024-03-04T09:32:32.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.986+0000 7ff642400640 1 -- 172.21.15.6:0/3921300397 --> [v2:172.21.15.6:6800/1558806123,v1:172.21.15.6:6801/1558806123] -- osd_op(unknown.0.0:35 17.11 17:88000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x7ff63c004b70 con 0x7ff63405fac0 2024-03-04T09:32:32.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.986+0000 7ff641000640 1 -- 172.21.15.6:0/3921300397 <== osd.3 v2:172.21.15.6:6800/1558806123 7 ==== osd_op_reply(35 [pgnls start_epoch 115 out=49b] v109'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff630013070 con 0x7ff63405fac0 2024-03-04T09:32:32.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.986+0000 7ff641000640 1 -- 172.21.15.6:0/3921300397 --> [v2:172.21.15.6:6808/2494103078,v1:172.21.15.6:6809/2494103078] -- osd_op(unknown.0.0:36 17.31 17:8c000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x7ff63c004b70 con 0x55d9365eb1d0 2024-03-04T09:32:32.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.987+0000 7ff641a00640 1 -- 172.21.15.6:0/3921300397 <== osd.1 v2:172.21.15.6:6808/2494103078 5 ==== osd_op_reply(36 [pgnls start_epoch 115 out=49b] v109'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff634059070 con 0x55d9365eb1d0 2024-03-04T09:32:32.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.987+0000 7ff641a00640 1 -- 172.21.15.6:0/3921300397 --> [v2:172.21.15.6:6800/1558806123,v1:172.21.15.6:6801/1558806123] -- osd_op(unknown.0.0:37 17.9 17:90000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x7ff63c004b70 con 0x7ff63405fac0 2024-03-04T09:32:32.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.987+0000 7ff641000640 1 -- 172.21.15.6:0/3921300397 <== osd.3 v2:172.21.15.6:6800/1558806123 8 ==== osd_op_reply(37 [pgnls start_epoch 115 out=49b] v109'2 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff630013070 con 0x7ff63405fac0 2024-03-04T09:32:32.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.987+0000 7ff641000640 1 -- 172.21.15.6:0/3921300397 --> [v2:172.21.15.146:6800/1526135780,v1:172.21.15.146:6801/1526135780] -- osd_op(unknown.0.0:38 17.29 17:94000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x7ff63c004b70 con 0x55d9365eca80 2024-03-04T09:32:32.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.988+0000 7ff641a00640 1 -- 172.21.15.6:0/3921300397 <== osd.4 v2:172.21.15.146:6800/1526135780 4 ==== osd_op_reply(38 [pgnls start_epoch 115 out=49b] v109'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff634059070 con 0x55d9365eca80 2024-03-04T09:32:32.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.988+0000 7ff641a00640 1 -- 172.21.15.6:0/3921300397 --> [v2:172.21.15.146:6808/460262507,v1:172.21.15.146:6809/460262507] -- osd_op(unknown.0.0:39 17.19 17:98000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x7ff63c004b70 con 0x7ff63406b910 2024-03-04T09:32:32.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.989+0000 7ff641000640 1 -- 172.21.15.6:0/3921300397 <== osd.5 v2:172.21.15.146:6808/460262507 3 ==== osd_op_reply(39 [pgnls start_epoch 115 out=49b] v109'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff630013070 con 0x7ff63406b910 2024-03-04T09:32:32.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.989+0000 7ff641000640 1 -- 172.21.15.6:0/3921300397 --> [v2:172.21.15.146:6808/460262507,v1:172.21.15.146:6809/460262507] -- osd_op(unknown.0.0:40 17.39 17:9c000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x7ff63c004b70 con 0x7ff63406b910 2024-03-04T09:32:32.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.990+0000 7ff641000640 1 -- 172.21.15.6:0/3921300397 <== osd.5 v2:172.21.15.146:6808/460262507 4 ==== osd_op_reply(40 [pgnls start_epoch 115 out=49b] v109'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff630013070 con 0x7ff63406b910 2024-03-04T09:32:32.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.990+0000 7ff641000640 1 -- 172.21.15.6:0/3921300397 --> [v2:172.21.15.146:6821/1392897493,v1:172.21.15.146:6823/1392897493] -- osd_op(unknown.0.0:41 17.5 17:a0000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x7ff63c004b70 con 0x7ff63405e3e0 2024-03-04T09:32:32.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.991+0000 7ff642400640 1 -- 172.21.15.6:0/3921300397 <== osd.6 v2:172.21.15.146:6821/1392897493 3 ==== osd_op_reply(41 [pgnls start_epoch 115 out=49b] v109'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff63c037040 con 0x7ff63405e3e0 2024-03-04T09:32:32.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.991+0000 7ff642400640 1 -- 172.21.15.6:0/3921300397 --> [v2:172.21.15.6:6808/2494103078,v1:172.21.15.6:6809/2494103078] -- osd_op(unknown.0.0:42 17.25 17:a4000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x7ff63c004b70 con 0x55d9365eb1d0 2024-03-04T09:32:32.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.992+0000 7ff641a00640 1 -- 172.21.15.6:0/3921300397 <== osd.1 v2:172.21.15.6:6808/2494103078 6 ==== osd_op_reply(42 [pgnls start_epoch 115 out=49b] v109'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff634059070 con 0x55d9365eb1d0 2024-03-04T09:32:32.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.992+0000 7ff641a00640 1 -- 172.21.15.6:0/3921300397 --> [v2:172.21.15.146:6808/460262507,v1:172.21.15.146:6809/460262507] -- osd_op(unknown.0.0:43 17.15 17:a8000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x7ff63c004b70 con 0x7ff63406b910 2024-03-04T09:32:32.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.993+0000 7ff641000640 1 -- 172.21.15.6:0/3921300397 <== osd.5 v2:172.21.15.146:6808/460262507 5 ==== osd_op_reply(43 [pgnls start_epoch 115 out=49b] v109'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff630013070 con 0x7ff63406b910 2024-03-04T09:32:32.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.993+0000 7ff641000640 1 -- 172.21.15.6:0/3921300397 --> [v2:172.21.15.146:6800/1526135780,v1:172.21.15.146:6801/1526135780] -- osd_op(unknown.0.0:44 17.35 17:ac000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x7ff63c004b70 con 0x55d9365eca80 2024-03-04T09:32:32.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.994+0000 7ff641a00640 1 -- 172.21.15.6:0/3921300397 <== osd.4 v2:172.21.15.146:6800/1526135780 5 ==== osd_op_reply(44 [pgnls start_epoch 115 out=49b] v109'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff634059070 con 0x55d9365eca80 2024-03-04T09:32:32.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.994+0000 7ff641a00640 1 -- 172.21.15.6:0/3921300397 --> [v2:172.21.15.146:6816/3268323156,v1:172.21.15.146:6817/3268323156] -- osd_op(unknown.0.0:45 17.d 17:b0000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x7ff63c004b70 con 0x55d93668b4f0 2024-03-04T09:32:32.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.995+0000 7ff642400640 1 -- 172.21.15.6:0/3921300397 <== osd.7 v2:172.21.15.146:6816/3268323156 7 ==== osd_op_reply(45 [pgnls start_epoch 115 out=49b] v109'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff63c037040 con 0x55d93668b4f0 2024-03-04T09:32:32.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.995+0000 7ff642400640 1 -- 172.21.15.6:0/3921300397 --> [v2:172.21.15.146:6816/3268323156,v1:172.21.15.146:6817/3268323156] -- osd_op(unknown.0.0:46 17.2d 17:b4000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x7ff63c004b70 con 0x55d93668b4f0 2024-03-04T09:32:32.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.996+0000 7ff642400640 1 -- 172.21.15.6:0/3921300397 <== osd.7 v2:172.21.15.146:6816/3268323156 8 ==== osd_op_reply(46 [pgnls start_epoch 115 out=49b] v109'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff63c037040 con 0x55d93668b4f0 2024-03-04T09:32:32.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.996+0000 7ff642400640 1 -- 172.21.15.6:0/3921300397 --> [v2:172.21.15.146:6816/3268323156,v1:172.21.15.146:6817/3268323156] -- osd_op(unknown.0.0:47 17.1d 17:b8000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x7ff63c004b70 con 0x55d93668b4f0 2024-03-04T09:32:32.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.997+0000 7ff642400640 1 -- 172.21.15.6:0/3921300397 <== osd.7 v2:172.21.15.146:6816/3268323156 9 ==== osd_op_reply(47 [pgnls start_epoch 115 out=49b] v109'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff63c037040 con 0x55d93668b4f0 2024-03-04T09:32:32.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.997+0000 7ff642400640 1 -- 172.21.15.6:0/3921300397 --> [v2:172.21.15.146:6816/3268323156,v1:172.21.15.146:6817/3268323156] -- osd_op(unknown.0.0:48 17.3d 17:bc000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x7ff63c004b70 con 0x55d93668b4f0 2024-03-04T09:32:32.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.998+0000 7ff642400640 1 -- 172.21.15.6:0/3921300397 <== osd.7 v2:172.21.15.146:6816/3268323156 10 ==== osd_op_reply(48 [pgnls start_epoch 115 out=49b] v109'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff63c04a040 con 0x55d93668b4f0 2024-03-04T09:32:32.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.998+0000 7ff642400640 1 -- 172.21.15.6:0/3921300397 --> [v2:172.21.15.146:6800/1526135780,v1:172.21.15.146:6801/1526135780] -- osd_op(unknown.0.0:49 17.3 17:c0000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x7ff63c004b70 con 0x55d9365eca80 2024-03-04T09:32:32.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.999+0000 7ff641a00640 1 -- 172.21.15.6:0/3921300397 <== osd.4 v2:172.21.15.146:6800/1526135780 6 ==== osd_op_reply(49 [pgnls start_epoch 115 out=49b] v109'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff634059070 con 0x55d9365eca80 2024-03-04T09:32:32.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:32.999+0000 7ff641a00640 1 -- 172.21.15.6:0/3921300397 --> [v2:172.21.15.146:6800/1526135780,v1:172.21.15.146:6801/1526135780] -- osd_op(unknown.0.0:50 17.23 17:c4000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x7ff63c004b70 con 0x55d9365eca80 2024-03-04T09:32:32.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.000+0000 7ff641a00640 1 -- 172.21.15.6:0/3921300397 <== osd.4 v2:172.21.15.146:6800/1526135780 7 ==== osd_op_reply(50 [pgnls start_epoch 115 out=49b] v109'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff634059070 con 0x55d9365eca80 2024-03-04T09:32:32.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.000+0000 7ff641a00640 1 -- 172.21.15.6:0/3921300397 --> [v2:172.21.15.6:6800/1558806123,v1:172.21.15.6:6801/1558806123] -- osd_op(unknown.0.0:51 17.13 17:c8000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x7ff63c004b70 con 0x7ff63405fac0 2024-03-04T09:32:32.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.001+0000 7ff641000640 1 -- 172.21.15.6:0/3921300397 <== osd.3 v2:172.21.15.6:6800/1558806123 9 ==== osd_op_reply(51 [pgnls start_epoch 115 out=49b] v109'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff630013070 con 0x7ff63405fac0 2024-03-04T09:32:32.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.001+0000 7ff641000640 1 -- 172.21.15.6:0/3921300397 --> [v2:172.21.15.146:6816/3268323156,v1:172.21.15.146:6817/3268323156] -- osd_op(unknown.0.0:52 17.33 17:cc000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x7ff63c004b70 con 0x55d93668b4f0 2024-03-04T09:32:32.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.002+0000 7ff642400640 1 -- 172.21.15.6:0/3921300397 <== osd.7 v2:172.21.15.146:6816/3268323156 11 ==== osd_op_reply(52 [pgnls start_epoch 115 out=49b] v109'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff63c04a040 con 0x55d93668b4f0 2024-03-04T09:32:32.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.002+0000 7ff642400640 1 -- 172.21.15.6:0/3921300397 --> [v2:172.21.15.6:6823/1364449401,v1:172.21.15.6:6825/1364449401] -- osd_op(unknown.0.0:53 17.b 17:d0000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x7ff63c004b70 con 0x7ff630057530 2024-03-04T09:32:32.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.003+0000 7ff642400640 1 -- 172.21.15.6:0/3921300397 <== osd.0 v2:172.21.15.6:6823/1364449401 10 ==== osd_op_reply(53 [pgnls start_epoch 115 out=49b] v109'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff63c04a040 con 0x7ff630057530 2024-03-04T09:32:32.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.003+0000 7ff642400640 1 -- 172.21.15.6:0/3921300397 --> [v2:172.21.15.146:6800/1526135780,v1:172.21.15.146:6801/1526135780] -- osd_op(unknown.0.0:54 17.2b 17:d4000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x7ff63c004b70 con 0x55d9365eca80 2024-03-04T09:32:33.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.004+0000 7ff641a00640 1 -- 172.21.15.6:0/3921300397 <== osd.4 v2:172.21.15.146:6800/1526135780 8 ==== osd_op_reply(54 [pgnls start_epoch 115 out=49b] v109'11 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff634059070 con 0x55d9365eca80 2024-03-04T09:32:33.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.004+0000 7ff641a00640 1 -- 172.21.15.6:0/3921300397 --> [v2:172.21.15.146:6800/1526135780,v1:172.21.15.146:6801/1526135780] -- osd_op(unknown.0.0:55 17.1b 17:d8000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x7ff63c004b70 con 0x55d9365eca80 2024-03-04T09:32:33.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.005+0000 7ff641a00640 1 -- 172.21.15.6:0/3921300397 <== osd.4 v2:172.21.15.146:6800/1526135780 9 ==== osd_op_reply(55 [pgnls start_epoch 115 out=49b] v109'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff634059070 con 0x55d9365eca80 2024-03-04T09:32:33.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.005+0000 7ff641a00640 1 -- 172.21.15.6:0/3921300397 --> [v2:172.21.15.6:6810/1500066681,v1:172.21.15.6:6812/1500066681] -- osd_op(unknown.0.0:56 17.3b 17:dc000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x7ff63c004b70 con 0x55d936695320 2024-03-04T09:32:33.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.006+0000 7ff641a00640 1 -- 172.21.15.6:0/3921300397 <== osd.2 v2:172.21.15.6:6810/1500066681 3 ==== osd_op_reply(56 [pgnls start_epoch 115 out=49b] v109'4 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff634059070 con 0x55d936695320 2024-03-04T09:32:33.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.006+0000 7ff641a00640 1 -- 172.21.15.6:0/3921300397 --> [v2:172.21.15.6:6823/1364449401,v1:172.21.15.6:6825/1364449401] -- osd_op(unknown.0.0:57 17.7 17:e0000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x7ff63c004b70 con 0x7ff630057530 2024-03-04T09:32:33.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.007+0000 7ff642400640 1 -- 172.21.15.6:0/3921300397 <== osd.0 v2:172.21.15.6:6823/1364449401 11 ==== osd_op_reply(57 [pgnls start_epoch 115 out=49b] v115'12 uv12 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff63c04a040 con 0x7ff630057530 2024-03-04T09:32:33.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.007+0000 7ff642400640 1 -- 172.21.15.6:0/3921300397 --> [v2:172.21.15.146:6800/1526135780,v1:172.21.15.146:6801/1526135780] -- osd_op(unknown.0.0:58 17.27 17:e4000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x7ff63c004b70 con 0x55d9365eca80 2024-03-04T09:32:33.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.008+0000 7ff641a00640 1 -- 172.21.15.6:0/3921300397 <== osd.4 v2:172.21.15.146:6800/1526135780 10 ==== osd_op_reply(58 [pgnls start_epoch 115 out=49b] v109'2 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff634059070 con 0x55d9365eca80 2024-03-04T09:32:33.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.008+0000 7ff641a00640 1 -- 172.21.15.6:0/3921300397 --> [v2:172.21.15.146:6821/1392897493,v1:172.21.15.146:6823/1392897493] -- osd_op(unknown.0.0:59 17.17 17:e8000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x7ff63c004b70 con 0x7ff63405e3e0 2024-03-04T09:32:33.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.009+0000 7ff642400640 1 -- 172.21.15.6:0/3921300397 <== osd.6 v2:172.21.15.146:6821/1392897493 4 ==== osd_op_reply(59 [pgnls start_epoch 115 out=49b] v109'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff63c04a040 con 0x7ff63405e3e0 2024-03-04T09:32:33.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.009+0000 7ff642400640 1 -- 172.21.15.6:0/3921300397 --> [v2:172.21.15.6:6823/1364449401,v1:172.21.15.6:6825/1364449401] -- osd_op(unknown.0.0:60 17.37 17:ec000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x7ff63c004b70 con 0x7ff630057530 2024-03-04T09:32:33.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.010+0000 7ff642400640 1 -- 172.21.15.6:0/3921300397 <== osd.0 v2:172.21.15.6:6823/1364449401 12 ==== osd_op_reply(60 [pgnls start_epoch 115 out=49b] v109'11 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff63c04a040 con 0x7ff630057530 2024-03-04T09:32:33.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.010+0000 7ff642400640 1 -- 172.21.15.6:0/3921300397 --> [v2:172.21.15.6:6808/2494103078,v1:172.21.15.6:6809/2494103078] -- osd_op(unknown.0.0:61 17.f 17:f0000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x7ff63c004b70 con 0x55d9365eb1d0 2024-03-04T09:32:33.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.010+0000 7ff641a00640 1 -- 172.21.15.6:0/3921300397 <== osd.1 v2:172.21.15.6:6808/2494103078 7 ==== osd_op_reply(61 [pgnls start_epoch 115 out=49b] v109'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff634059070 con 0x55d9365eb1d0 2024-03-04T09:32:33.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.010+0000 7ff641a00640 1 -- 172.21.15.6:0/3921300397 --> [v2:172.21.15.146:6800/1526135780,v1:172.21.15.146:6801/1526135780] -- osd_op(unknown.0.0:62 17.2f 17:f4000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x7ff63c004b70 con 0x55d9365eca80 2024-03-04T09:32:33.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.011+0000 7ff641a00640 1 -- 172.21.15.6:0/3921300397 <== osd.4 v2:172.21.15.146:6800/1526135780 11 ==== osd_op_reply(62 [pgnls start_epoch 115 out=49b] v115'10 uv10 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff634059070 con 0x55d9365eca80 2024-03-04T09:32:33.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.011+0000 7ff641a00640 1 -- 172.21.15.6:0/3921300397 --> [v2:172.21.15.146:6821/1392897493,v1:172.21.15.146:6823/1392897493] -- osd_op(unknown.0.0:63 17.1f 17:f8000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x7ff63c004b70 con 0x7ff63405e3e0 2024-03-04T09:32:33.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.012+0000 7ff642400640 1 -- 172.21.15.6:0/3921300397 <== osd.6 v2:172.21.15.146:6821/1392897493 5 ==== osd_op_reply(63 [pgnls start_epoch 115 out=49b] v109'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff63c04a040 con 0x7ff63405e3e0 2024-03-04T09:32:33.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.013+0000 7ff642400640 1 -- 172.21.15.6:0/3921300397 --> [v2:172.21.15.6:6823/1364449401,v1:172.21.15.6:6825/1364449401] -- osd_op(unknown.0.0:64 17.3f 17:fc000000::::head [pgnls start_epoch 115 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e115) v9 -- 0x7ff63c004b70 con 0x7ff630057530 2024-03-04T09:32:33.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.013+0000 7ff642400640 1 -- 172.21.15.6:0/3921300397 <== osd.0 v2:172.21.15.6:6823/1364449401 13 ==== osd_op_reply(64 [pgnls start_epoch 115 out=49b] v109'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff63c04a040 con 0x7ff630057530 2024-03-04T09:32:33.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.013+0000 7ff6448f9f40 1 -- 172.21.15.6:0/3921300397 >> [v2:172.21.15.6:6823/1364449401,v1:172.21.15.6:6825/1364449401] conn(0x7ff630057530 msgr2=0x7ff630056190 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:33.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.013+0000 7ff6448f9f40 1 --2- 172.21.15.6:0/3921300397 >> [v2:172.21.15.6:6823/1364449401,v1:172.21.15.6:6825/1364449401] conn(0x7ff630057530 0x7ff630056190 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:33.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.013+0000 7ff6448f9f40 1 -- 172.21.15.6:0/3921300397 >> [v2:172.21.15.6:6808/2494103078,v1:172.21.15.6:6809/2494103078] conn(0x55d9365eb1d0 msgr2=0x7ff63c002a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:33.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.013+0000 7ff6448f9f40 1 --2- 172.21.15.6:0/3921300397 >> [v2:172.21.15.6:6808/2494103078,v1:172.21.15.6:6809/2494103078] conn(0x55d9365eb1d0 0x7ff63c002a90 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:33.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.013+0000 7ff6448f9f40 1 -- 172.21.15.6:0/3921300397 >> [v2:172.21.15.6:6810/1500066681,v1:172.21.15.6:6812/1500066681] conn(0x55d936695320 msgr2=0x7ff63c02a440 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:33.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.013+0000 7ff6448f9f40 1 --2- 172.21.15.6:0/3921300397 >> [v2:172.21.15.6:6810/1500066681,v1:172.21.15.6:6812/1500066681] conn(0x55d936695320 0x7ff63c02a440 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:33.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.013+0000 7ff6448f9f40 1 -- 172.21.15.6:0/3921300397 >> [v2:172.21.15.6:6800/1558806123,v1:172.21.15.6:6801/1558806123] conn(0x7ff63405fac0 msgr2=0x7ff634053410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:33.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.013+0000 7ff6448f9f40 1 --2- 172.21.15.6:0/3921300397 >> [v2:172.21.15.6:6800/1558806123,v1:172.21.15.6:6801/1558806123] conn(0x7ff63405fac0 0x7ff634053410 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:33.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.013+0000 7ff6448f9f40 1 -- 172.21.15.6:0/3921300397 >> [v2:172.21.15.146:6800/1526135780,v1:172.21.15.146:6801/1526135780] conn(0x55d9365eca80 msgr2=0x7ff63c018950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:33.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.014+0000 7ff6448f9f40 1 --2- 172.21.15.6:0/3921300397 >> [v2:172.21.15.146:6800/1526135780,v1:172.21.15.146:6801/1526135780] conn(0x55d9365eca80 0x7ff63c018950 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:33.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.014+0000 7ff6448f9f40 1 -- 172.21.15.6:0/3921300397 >> [v2:172.21.15.146:6808/460262507,v1:172.21.15.146:6809/460262507] conn(0x7ff63406b910 msgr2=0x7ff6340793f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:33.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.014+0000 7ff6448f9f40 1 --2- 172.21.15.6:0/3921300397 >> [v2:172.21.15.146:6808/460262507,v1:172.21.15.146:6809/460262507] conn(0x7ff63406b910 0x7ff6340793f0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:33.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.014+0000 7ff6448f9f40 1 -- 172.21.15.6:0/3921300397 >> [v2:172.21.15.146:6821/1392897493,v1:172.21.15.146:6823/1392897493] conn(0x7ff63405e3e0 msgr2=0x7ff63405e840 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:33.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.014+0000 7ff6448f9f40 1 --2- 172.21.15.6:0/3921300397 >> [v2:172.21.15.146:6821/1392897493,v1:172.21.15.146:6823/1392897493] conn(0x7ff63405e3e0 0x7ff63405e840 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:33.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.014+0000 7ff6448f9f40 1 -- 172.21.15.6:0/3921300397 >> [v2:172.21.15.146:6816/3268323156,v1:172.21.15.146:6817/3268323156] conn(0x55d93668b4f0 msgr2=0x55d93668b8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:33.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.014+0000 7ff6448f9f40 1 --2- 172.21.15.6:0/3921300397 >> [v2:172.21.15.146:6816/3268323156,v1:172.21.15.146:6817/3268323156] conn(0x55d93668b4f0 0x55d93668b8d0 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:33.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.013+0000 7ff642400640 1 -- 172.21.15.6:0/3921300397 reap_dead start 2024-03-04T09:32:33.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.014+0000 7ff642400640 1 -- 172.21.15.6:0/3921300397 reap_dead start 2024-03-04T09:32:33.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.014+0000 7ff642400640 1 -- 172.21.15.6:0/3921300397 reap_dead start 2024-03-04T09:32:33.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.014+0000 7ff6448f9f40 1 -- 172.21.15.6:0/3921300397 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff624069eb0 msgr2=0x7ff62406c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:33.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.014+0000 7ff6448f9f40 1 --2- 172.21.15.6:0/3921300397 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff624069eb0 0x7ff62406c370 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7ff63401c7b0 tx=0x7ff63404f000 comp rx=0 tx=0).stop 2024-03-04T09:32:33.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.014+0000 7ff6448f9f40 1 -- 172.21.15.6:0/3921300397 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d936689db0 msgr2=0x55d93667ec60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:33.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.014+0000 7ff6448f9f40 1 --2- 172.21.15.6:0/3921300397 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d936689db0 0x55d93667ec60 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7ff630009e10 tx=0x7ff630009f00 comp rx=0 tx=0).stop 2024-03-04T09:32:33.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.014+0000 7ff642400640 1 -- 172.21.15.6:0/3921300397 reap_dead start 2024-03-04T09:32:33.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.014+0000 7ff6448f9f40 1 -- 172.21.15.6:0/3921300397 shutdown_connections 2024-03-04T09:32:33.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.014+0000 7ff6448f9f40 1 -- 172.21.15.6:0/3921300397 >> 172.21.15.6:0/3921300397 conn(0x55d9365eb660 msgr2=0x55d936608650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:32:33.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.014+0000 7ff6448f9f40 1 -- 172.21.15.6:0/3921300397 shutdown_connections 2024-03-04T09:32:33.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.014+0000 7ff6448f9f40 1 -- 172.21.15.6:0/3921300397 wait complete. 2024-03-04T09:32:33.019 DEBUG:tasks.cephfs.filesystem:All objects for ino 1099511627777 size 8388608 are absent 2024-03-04T09:32:33.019 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:32:33.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.229+0000 7f33eb200640 1 Processor -- start 2024-03-04T09:32:33.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.230+0000 7f33eb200640 1 -- start start 2024-03-04T09:32:33.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.231+0000 7f33eb200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f33ec15e0b0 0x7f33ec0cf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:33.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.231+0000 7f33eb200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f33ec0d0e50 0x7f33ec0b44d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:33.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.231+0000 7f33eb200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33ec0b3710 con 0x7f33ec0d4a10 2024-03-04T09:32:33.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.231+0000 7f33eb200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33ec0b09e0 con 0x7f33ec0d0e50 2024-03-04T09:32:33.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.231+0000 7f33eb200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33ec0b39f0 con 0x7f33ec15e0b0 2024-03-04T09:32:33.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.232+0000 7f33e3e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f33ec0d4a10 0x7f33ec0d0740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:36876/0 (socket says 172.21.15.6:36876) 2024-03-04T09:32:33.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.232+0000 7f33e3e00640 1 -- 172.21.15.6:0/1037546529 learned_addr learned my addr 172.21.15.6:0/1037546529 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:32:33.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.232+0000 7f33ea800640 1 --2- 172.21.15.6:0/1037546529 >> v2:172.21.15.6:3300/0 conn(0x7f33ec0d0e50 0x7f33ec0b44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:33.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.232+0000 7f33e9e00640 1 --2- 172.21.15.6:0/1037546529 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f33ec15e0b0 0x7f33ec0cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:33.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.232+0000 7f33e9400640 1 -- 172.21.15.6:0/1037546529 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1179337302 0 0) 0x7f33ec0b3710 con 0x7f33ec0d4a10 2024-03-04T09:32:33.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.232+0000 7f33e9400640 1 -- 172.21.15.6:0/1037546529 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33d0003520 con 0x7f33ec0d4a10 2024-03-04T09:32:33.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.232+0000 7f33ea800640 1 -- 172.21.15.6:0/1037546529 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f33ec15e0b0 msgr2=0x7f33ec0cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:32:33.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.232+0000 7f33ea800640 1 --2- 172.21.15.6:0/1037546529 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f33ec15e0b0 0x7f33ec0cf310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:33.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.232+0000 7f33ea800640 1 -- 172.21.15.6:0/1037546529 >> v1:172.21.15.6:6789/0 conn(0x7f33ec0d4a10 legacy=0x7f33ec0d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:33.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.232+0000 7f33ea800640 1 -- 172.21.15.6:0/1037546529 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33ec0b3cd0 con 0x7f33ec0d0e50 2024-03-04T09:32:33.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.234+0000 7f33ea800640 1 --2- 172.21.15.6:0/1037546529 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f33ec0d0e50 0x7f33ec0b44d0 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7f33e4009d90 tx=0x7f33e402bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3301d2d32d69a19c server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:33.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.234+0000 7f33e9400640 1 -- 172.21.15.6:0/1037546529 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33e4039070 con 0x7f33ec0d0e50 2024-03-04T09:32:33.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.234+0000 7f33e9400640 1 -- 172.21.15.6:0/1037546529 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33e402cc50 con 0x7f33ec0d0e50 2024-03-04T09:32:33.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.234+0000 7f33e9400640 1 -- 172.21.15.6:0/1037546529 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33e4034440 con 0x7f33ec0d0e50 2024-03-04T09:32:33.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.235+0000 7f33eb200640 1 -- 172.21.15.6:0/1037546529 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f33ec0d0e50 msgr2=0x7f33ec0b44d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:33.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.235+0000 7f33eb200640 1 --2- 172.21.15.6:0/1037546529 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f33ec0d0e50 0x7f33ec0b44d0 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7f33e4009d90 tx=0x7f33e402bea0 comp rx=0 tx=0).stop 2024-03-04T09:32:33.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.235+0000 7f33eb200640 1 -- 172.21.15.6:0/1037546529 shutdown_connections 2024-03-04T09:32:33.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.235+0000 7f33eb200640 1 --2- 172.21.15.6:0/1037546529 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f33ec0d0e50 0x7f33ec0b44d0 unknown :-1 s=CLOSED pgs=1384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:33.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.235+0000 7f33eb200640 1 --2- 172.21.15.6:0/1037546529 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f33ec15e0b0 0x7f33ec0cf310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:33.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.235+0000 7f33eb200640 1 -- 172.21.15.6:0/1037546529 >> 172.21.15.6:0/1037546529 conn(0x7f33ec0d3d30 msgr2=0x7f33ec0d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:32:33.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.235+0000 7f33eb200640 1 -- 172.21.15.6:0/1037546529 shutdown_connections 2024-03-04T09:32:33.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.235+0000 7f33eb200640 1 -- 172.21.15.6:0/1037546529 wait complete. 2024-03-04T09:32:33.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.238+0000 7f33eb200640 1 Processor -- start 2024-03-04T09:32:33.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.238+0000 7f33eb200640 1 -- start start 2024-03-04T09:32:33.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.239+0000 7f33eb200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f33ec0d0e50 0x7f33ec150190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:33.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.239+0000 7f33eb200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f33ec0d4a10 0x7f33ec150be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:33.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.239+0000 7f33eb200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f33ec15e0b0 0x7f33ec0b6950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:33.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.239+0000 7f33eb200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33ec0cfc60 con 0x7f33ec0d4a10 2024-03-04T09:32:33.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.239+0000 7f33eb200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33ec15e490 con 0x7f33ec15e0b0 2024-03-04T09:32:33.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.239+0000 7f33eb200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33ec0cf850 con 0x7f33ec0d0e50 2024-03-04T09:32:33.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.240+0000 7f33e3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f33ec0d4a10 0x7f33ec150be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:33.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.240+0000 7f33e3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f33ec0d4a10 0x7f33ec150be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:44632/0 (socket says 172.21.15.6:44632) 2024-03-04T09:32:33.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.240+0000 7f33e3e00640 1 -- 172.21.15.6:0/1774192454 learned_addr learned my addr 172.21.15.6:0/1774192454 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:32:33.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.240+0000 7f33e9e00640 1 --2- 172.21.15.6:0/1774192454 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f33ec0d0e50 0x7f33ec150190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:33.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.240+0000 7f33ea800640 1 --2- 172.21.15.6:0/1774192454 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f33ec15e0b0 0x7f33ec0b6950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:33.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.240+0000 7f33e3e00640 1 -- 172.21.15.6:0/1774192454 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f33ec0d0e50 msgr2=0x7f33ec150190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:33.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.240+0000 7f33e3e00640 1 --2- 172.21.15.6:0/1774192454 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f33ec0d0e50 0x7f33ec150190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:33.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.240+0000 7f33e3e00640 1 -- 172.21.15.6:0/1774192454 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f33ec15e0b0 msgr2=0x7f33ec0b6950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:33.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.240+0000 7f33e3e00640 1 --2- 172.21.15.6:0/1774192454 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f33ec15e0b0 0x7f33ec0b6950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:33.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.240+0000 7f33e3e00640 1 -- 172.21.15.6:0/1774192454 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33ec1506d0 con 0x7f33ec0d4a10 2024-03-04T09:32:33.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.241+0000 7f33e3e00640 1 --2- 172.21.15.6:0/1774192454 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f33ec0d4a10 0x7f33ec150be0 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7f33d400c400 tx=0x7f33d400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:33.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.241+0000 7f33e2000640 1 -- 172.21.15.6:0/1774192454 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33d4017070 con 0x7f33ec0d4a10 2024-03-04T09:32:33.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.241+0000 7f33e2000640 1 -- 172.21.15.6:0/1774192454 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33d4005d00 con 0x7f33ec0d4a10 2024-03-04T09:32:33.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.241+0000 7f33e2000640 1 -- 172.21.15.6:0/1774192454 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33d4007310 con 0x7f33ec0d4a10 2024-03-04T09:32:33.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.241+0000 7f33eb200640 1 -- 172.21.15.6:0/1774192454 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33ec15e610 con 0x7f33ec0d4a10 2024-03-04T09:32:33.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.241+0000 7f33eb200640 1 -- 172.21.15.6:0/1774192454 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33ec0b7660 con 0x7f33ec0d4a10 2024-03-04T09:32:33.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.242+0000 7f33eb200640 1 -- 172.21.15.6:0/1774192454 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33ec04f330 con 0x7f33ec0d4a10 2024-03-04T09:32:33.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.243+0000 7f33e2000640 1 -- 172.21.15.6:0/1774192454 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f33d401a070 con 0x7f33ec0d4a10 2024-03-04T09:32:33.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.244+0000 7f33e2000640 1 --2- 172.21.15.6:0/1774192454 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f33c806b4d0 0x7f33c806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:33.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.244+0000 7f33e2000640 1 -- 172.21.15.6:0/1774192454 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f33d4089690 con 0x7f33ec0d4a10 2024-03-04T09:32:33.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.245+0000 7f33e9e00640 1 --2- 172.21.15.6:0/1774192454 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f33c806b4d0 0x7f33c806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:33.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.246+0000 7f33e9e00640 1 --2- 172.21.15.6:0/1774192454 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f33c806b4d0 0x7f33c806d990 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f33dc009a30 tx=0x7f33dc008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:33.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.249+0000 7f33e2000640 1 -- 172.21.15.6:0/1774192454 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f33ec1506d0 con 0x7f33ec0d4a10 2024-03-04T09:32:33.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.572+0000 7f33eb200640 1 -- 172.21.15.6:0/1774192454 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f33ec0ca6e0 con 0x7f33ec0d4a10 2024-03-04T09:32:33.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.573+0000 7f33e2000640 1 -- 172.21.15.6:0/1774192454 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+4975 (secure 0 0 0) 0x7f33d4058510 con 0x7f33ec0d4a10 2024-03-04T09:32:33.570 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:32:33.570 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":84,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6535,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/3883122012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":3883122012},{"type":"v1","addr":"172.21.15.146:6835","nonce":3883122012}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":6559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/4060411493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":4060411493},{"type":"v1","addr":"172.21.15.146:6837","nonce":4060411493}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":82},{"gid":7812,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/679471398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":679471398},{"type":"v1","addr":"172.21.15.6:6833","nonce":679471398}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:31:21.114351+0000","modified":"2024-03-04T09:31:40.781474+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7815},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7815":{"gid":7815,"name":"a","rank":0,"incarnation":81,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/882667161","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":882667161},{"type":"v1","addr":"172.21.15.6:6835","nonce":882667161}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7815,"qdb_cluster":[7815]},"id":8}]} 2024-03-04T09:32:33.571 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 84 2024-03-04T09:32:33.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.576+0000 7f33eb200640 1 -- 172.21.15.6:0/1774192454 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f33c806b4d0 msgr2=0x7f33c806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:33.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.576+0000 7f33eb200640 1 --2- 172.21.15.6:0/1774192454 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f33c806b4d0 0x7f33c806d990 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f33dc009a30 tx=0x7f33dc008040 comp rx=0 tx=0).stop 2024-03-04T09:32:33.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.576+0000 7f33eb200640 1 -- 172.21.15.6:0/1774192454 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f33ec0d4a10 msgr2=0x7f33ec150be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:33.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.576+0000 7f33eb200640 1 --2- 172.21.15.6:0/1774192454 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f33ec0d4a10 0x7f33ec150be0 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7f33d400c400 tx=0x7f33d400c8c0 comp rx=0 tx=0).stop 2024-03-04T09:32:33.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.576+0000 7f33eb200640 1 -- 172.21.15.6:0/1774192454 shutdown_connections 2024-03-04T09:32:33.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.576+0000 7f33eb200640 1 --2- 172.21.15.6:0/1774192454 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f33ec15e0b0 0x7f33ec0b6950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:33.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.576+0000 7f33eb200640 1 --2- 172.21.15.6:0/1774192454 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f33ec0d4a10 0x7f33ec150be0 unknown :-1 s=CLOSED pgs=1385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:33.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.576+0000 7f33eb200640 1 --2- 172.21.15.6:0/1774192454 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f33c806b4d0 0x7f33c806d990 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:33.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.576+0000 7f33eb200640 1 --2- 172.21.15.6:0/1774192454 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f33ec0d0e50 0x7f33ec150190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:33.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.576+0000 7f33eb200640 1 -- 172.21.15.6:0/1774192454 >> 172.21.15.6:0/1774192454 conn(0x7f33ec0d3d30 msgr2=0x7f33ec0b0b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:32:33.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.576+0000 7f33eb200640 1 -- 172.21.15.6:0/1774192454 shutdown_connections 2024-03-04T09:32:33.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:33.576+0000 7f33eb200640 1 -- 172.21.15.6:0/1774192454 wait complete. 2024-03-04T09:32:33.600 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_created 2024-03-04T09:32:33.828 INFO:teuthology.orchestra.run.smithi006.stdout:{"mds_cache":{"strays_created":3}} 2024-03-04T09:32:33.828 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_created": 3 } } 2024-03-04T09:32:33.828 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:32:34.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:34.048+0000 7fdfb2600640 1 Processor -- start 2024-03-04T09:32:34.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:34.048+0000 7fdfb2600640 1 -- start start 2024-03-04T09:32:34.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:34.050+0000 7fdfb2600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fdfac161670 0x7fdfac070c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:34.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:34.050+0000 7fdfb2600640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fdfac06e240 0x7fdfac0711b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:34.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:34.050+0000 7fdfb2600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdfac06cfe0 0x7fdfac06d440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:34.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:34.050+0000 7fdfb2600640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdfac0730d0 con 0x7fdfac161670 2024-03-04T09:32:34.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:34.050+0000 7fdfb2600640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdfac0ce9a0 con 0x7fdfac06e240 2024-03-04T09:32:34.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:34.050+0000 7fdfb2600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfac0cec80 con 0x7fdfac06cfe0 2024-03-04T09:32:34.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:34.051+0000 7fdfab400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fdfac161670 0x7fdfac070c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:34.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:34.051+0000 7fdfab400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fdfac161670 0x7fdfac070c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:44638/0 (socket says 172.21.15.6:44638) 2024-03-04T09:32:34.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:34.051+0000 7fdfab400640 1 -- 172.21.15.6:0/1462821606 learned_addr learned my addr 172.21.15.6:0/1462821606 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:32:34.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:34.051+0000 7fdfa3e00640 1 --2- 172.21.15.6:0/1462821606 >> v2:172.21.15.146:3300/0 conn(0x7fdfac06e240 0x7fdfac0711b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:34.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:34.051+0000 7fdfabe00640 1 --2- 172.21.15.6:0/1462821606 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdfac06cfe0 0x7fdfac06d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:34.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:34.051+0000 7fdfab400640 1 -- 172.21.15.6:0/1462821606 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdfac06cfe0 msgr2=0x7fdfac06d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:32:34.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:34.051+0000 7fdfab400640 1 --2- 172.21.15.6:0/1462821606 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdfac06cfe0 0x7fdfac06d440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:34.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:34.051+0000 7fdfab400640 1 -- 172.21.15.6:0/1462821606 >> v2:172.21.15.146:3300/0 conn(0x7fdfac06e240 msgr2=0x7fdfac0711b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:32:34.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:34.051+0000 7fdfab400640 1 --2- 172.21.15.6:0/1462821606 >> v2:172.21.15.146:3300/0 conn(0x7fdfac06e240 0x7fdfac0711b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:34.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:34.051+0000 7fdfab400640 1 -- 172.21.15.6:0/1462821606 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfac0cef60 con 0x7fdfac161670 2024-03-04T09:32:34.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:34.054+0000 7fdfab400640 1 --2- 172.21.15.6:0/1462821606 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdfac161670 0x7fdfac070c70 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7fdfac0b2ad0 tx=0x7fdf9c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cafb6b8754b6cc5e server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:34.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:34.055+0000 7fdfaaa00640 1 -- 172.21.15.6:0/1462821606 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf9c035020 con 0x7fdfac161670 2024-03-04T09:32:34.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:34.055+0000 7fdfaaa00640 1 -- 172.21.15.6:0/1462821606 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf9c031870 con 0x7fdfac161670 2024-03-04T09:32:34.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:34.055+0000 7fdfaaa00640 1 -- 172.21.15.6:0/1462821606 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf9c031c40 con 0x7fdfac161670 2024-03-04T09:32:34.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:34.056+0000 7fdfb2600640 1 -- 172.21.15.6:0/1462821606 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdfac161670 msgr2=0x7fdfac070c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:34.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:34.056+0000 7fdfb2600640 1 --2- 172.21.15.6:0/1462821606 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdfac161670 0x7fdfac070c70 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7fdfac0b2ad0 tx=0x7fdf9c02b910 comp rx=0 tx=0).stop 2024-03-04T09:32:34.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:34.056+0000 7fdfb2600640 1 -- 172.21.15.6:0/1462821606 shutdown_connections 2024-03-04T09:32:34.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:34.056+0000 7fdfb2600640 1 --2- 172.21.15.6:0/1462821606 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdfac06cfe0 0x7fdfac06d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:34.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:34.056+0000 7fdfb2600640 1 --2- 172.21.15.6:0/1462821606 >> v2:172.21.15.146:3300/0 conn(0x7fdfac06e240 0x7fdfac0711b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:34.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:34.056+0000 7fdfb2600640 1 --2- 172.21.15.6:0/1462821606 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdfac161670 0x7fdfac070c70 unknown :-1 s=CLOSED pgs=1386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:34.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:34.056+0000 7fdfb2600640 1 -- 172.21.15.6:0/1462821606 >> 172.21.15.6:0/1462821606 conn(0x7fdfac06afb0 msgr2=0x7fdfac06b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:32:34.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:34.056+0000 7fdfb2600640 1 -- 172.21.15.6:0/1462821606 shutdown_connections 2024-03-04T09:32:34.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:34.056+0000 7fdfb2600640 1 -- 172.21.15.6:0/1462821606 wait complete. 2024-03-04T09:32:34.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:34.058+0000 7fdfb2600640 1 Processor -- start 2024-03-04T09:32:34.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:34.058+0000 7fdfb2600640 1 -- start start 2024-03-04T09:32:34.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:34.059+0000 7fdfb2600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdfac06cfe0 0x7fdfac09aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:34.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:34.060+0000 7fdfb2600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdfac06e240 0x7fdfac148000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:34.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:34.060+0000 7fdfb2600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdfac161670 0x7fdfac148540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:34.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:34.060+0000 7fdfb2600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfac0ce600 con 0x7fdfac161670 2024-03-04T09:32:34.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:34.060+0000 7fdfb2600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfac0cf520 con 0x7fdfac06e240 2024-03-04T09:32:34.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:34.060+0000 7fdfb2600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfac0ce780 con 0x7fdfac06cfe0 2024-03-04T09:32:34.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:34.060+0000 7fdfabe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdfac161670 0x7fdfac148540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:34.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:34.060+0000 7fdfabe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdfac161670 0x7fdfac148540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:44650/0 (socket says 172.21.15.6:44650) 2024-03-04T09:32:34.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:34.060+0000 7fdfabe00640 1 -- 172.21.15.6:0/2418242686 learned_addr learned my addr 172.21.15.6:0/2418242686 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:32:34.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:34.060+0000 7fdfa3e00640 1 --2- 172.21.15.6:0/2418242686 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdfac06e240 0x7fdfac148000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:34.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:34.060+0000 7fdfab400640 1 --2- 172.21.15.6:0/2418242686 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdfac06cfe0 0x7fdfac09aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:34.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:34.060+0000 7fdfabe00640 1 -- 172.21.15.6:0/2418242686 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdfac06cfe0 msgr2=0x7fdfac09aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:34.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:34.060+0000 7fdfabe00640 1 --2- 172.21.15.6:0/2418242686 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdfac06cfe0 0x7fdfac09aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:34.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:34.060+0000 7fdfabe00640 1 -- 172.21.15.6:0/2418242686 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdfac06e240 msgr2=0x7fdfac148000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:34.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:34.060+0000 7fdfabe00640 1 --2- 172.21.15.6:0/2418242686 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdfac06e240 0x7fdfac148000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:34.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:34.060+0000 7fdfabe00640 1 -- 172.21.15.6:0/2418242686 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfac15dab0 con 0x7fdfac161670 2024-03-04T09:32:34.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:34.061+0000 7fdfabe00640 1 --2- 172.21.15.6:0/2418242686 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdfac161670 0x7fdfac148540 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7fdfa400eed0 tx=0x7fdfa400b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:34.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:34.061+0000 7fdfa3400640 1 -- 172.21.15.6:0/2418242686 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfa4018020 con 0x7fdfac161670 2024-03-04T09:32:34.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:34.061+0000 7fdfa3400640 1 -- 172.21.15.6:0/2418242686 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfa4014400 con 0x7fdfac161670 2024-03-04T09:32:34.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:34.061+0000 7fdfa3400640 1 -- 172.21.15.6:0/2418242686 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfa40147f0 con 0x7fdfac161670 2024-03-04T09:32:34.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:34.061+0000 7fdfb2600640 1 -- 172.21.15.6:0/2418242686 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfac15dd90 con 0x7fdfac161670 2024-03-04T09:32:34.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:34.061+0000 7fdfb2600640 1 -- 172.21.15.6:0/2418242686 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfac15e350 con 0x7fdfac161670 2024-03-04T09:32:34.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:34.063+0000 7fdfb2600640 1 -- 172.21.15.6:0/2418242686 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfac069510 con 0x7fdfac161670 2024-03-04T09:32:34.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:34.064+0000 7fdfa3400640 1 -- 172.21.15.6:0/2418242686 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdfa4005d10 con 0x7fdfac161670 2024-03-04T09:32:34.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:34.065+0000 7fdfa3400640 1 --2- 172.21.15.6:0/2418242686 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fdf8806b4d0 0x7fdf8806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:34.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:34.065+0000 7fdfa3400640 1 -- 172.21.15.6:0/2418242686 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5853+0+0 (secure 0 0 0) 0x7fdfa408aae0 con 0x7fdfac161670 2024-03-04T09:32:34.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:34.066+0000 7fdfab400640 1 --2- 172.21.15.6:0/2418242686 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fdf8806b4d0 0x7fdf8806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:34.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:34.067+0000 7fdfab400640 1 --2- 172.21.15.6:0/2418242686 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fdf8806b4d0 0x7fdf8806d990 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fdf9c036000 tx=0x7fdf9c03c040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:34.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:34.070+0000 7fdfa3400640 1 -- 172.21.15.6:0/2418242686 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdfa4022070 con 0x7fdfac161670 2024-03-04T09:32:34.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:34.308+0000 7fdfb2600640 1 -- 172.21.15.6:0/2418242686 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdfac0b52d0 con 0x7fdfac161670 2024-03-04T09:32:34.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:34.309+0000 7fdfa3400640 1 -- 172.21.15.6:0/2418242686 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+4975 (secure 0 0 0) 0x7fdfa40597a0 con 0x7fdfac161670 2024-03-04T09:32:34.308 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:32:34.308 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":84,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6535,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/3883122012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":3883122012},{"type":"v1","addr":"172.21.15.146:6835","nonce":3883122012}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":6559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/4060411493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":4060411493},{"type":"v1","addr":"172.21.15.146:6837","nonce":4060411493}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":82},{"gid":7812,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/679471398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":679471398},{"type":"v1","addr":"172.21.15.6:6833","nonce":679471398}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:31:21.114351+0000","modified":"2024-03-04T09:31:40.781474+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7815},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7815":{"gid":7815,"name":"a","rank":0,"incarnation":81,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/882667161","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":882667161},{"type":"v1","addr":"172.21.15.6:6835","nonce":882667161}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7815,"qdb_cluster":[7815]},"id":8}]} 2024-03-04T09:32:34.309 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 84 2024-03-04T09:32:34.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:34.312+0000 7fdfb2600640 1 -- 172.21.15.6:0/2418242686 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fdf8806b4d0 msgr2=0x7fdf8806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:34.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:34.312+0000 7fdfb2600640 1 --2- 172.21.15.6:0/2418242686 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fdf8806b4d0 0x7fdf8806d990 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fdf9c036000 tx=0x7fdf9c03c040 comp rx=0 tx=0).stop 2024-03-04T09:32:34.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:34.312+0000 7fdfb2600640 1 -- 172.21.15.6:0/2418242686 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdfac161670 msgr2=0x7fdfac148540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:34.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:34.312+0000 7fdfb2600640 1 --2- 172.21.15.6:0/2418242686 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdfac161670 0x7fdfac148540 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7fdfa400eed0 tx=0x7fdfa400b480 comp rx=0 tx=0).stop 2024-03-04T09:32:34.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:34.312+0000 7fdfb2600640 1 -- 172.21.15.6:0/2418242686 shutdown_connections 2024-03-04T09:32:34.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:34.312+0000 7fdfb2600640 1 --2- 172.21.15.6:0/2418242686 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdfac161670 0x7fdfac148540 unknown :-1 s=CLOSED pgs=1387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:34.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:34.312+0000 7fdfb2600640 1 --2- 172.21.15.6:0/2418242686 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdfac06e240 0x7fdfac148000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:34.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:34.312+0000 7fdfb2600640 1 --2- 172.21.15.6:0/2418242686 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fdf8806b4d0 0x7fdf8806d990 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:34.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:34.312+0000 7fdfb2600640 1 --2- 172.21.15.6:0/2418242686 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdfac06cfe0 0x7fdfac09aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:34.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:34.312+0000 7fdfb2600640 1 -- 172.21.15.6:0/2418242686 >> 172.21.15.6:0/2418242686 conn(0x7fdfac06afb0 msgr2=0x7fdfac0b2c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:32:34.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:34.312+0000 7fdfb2600640 1 -- 172.21.15.6:0/2418242686 shutdown_connections 2024-03-04T09:32:34.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:34.312+0000 7fdfb2600640 1 -- 172.21.15.6:0/2418242686 wait complete. 2024-03-04T09:32:34.336 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_enqueued 2024-03-04T09:32:34.575 INFO:teuthology.orchestra.run.smithi006.stdout:{"mds_cache":{"strays_enqueued":1}} 2024-03-04T09:32:34.575 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 1 } } 2024-03-04T09:32:34.575 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-04T09:32:34.576 DEBUG:teuthology.orchestra.run.smithi006:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-04T09:32:34.684 DEBUG:teuthology.orchestra.run.smithi146:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-04T09:32:34.808 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-04T09:32:34.809 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-04T09:32:34.809 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T09:32:34.865 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-04T09:32:34.865 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T09:32:34.916 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T09:32:35.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.143+0000 7fbf4fa00640 1 Processor -- start 2024-03-04T09:32:35.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.143+0000 7fbf4fa00640 1 -- start start 2024-03-04T09:32:35.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.144+0000 7fbf4fa00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbf48161670 0x7fbf48070c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:35.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.144+0000 7fbf4fa00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fbf480ae0a0 0x7fbf480711b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:35.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.145+0000 7fbf4fa00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf480ce9a0 con 0x7fbf4806e240 2024-03-04T09:32:35.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.145+0000 7fbf4fa00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf480730d0 con 0x7fbf480ae0a0 2024-03-04T09:32:35.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.145+0000 7fbf4fa00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf480cec80 con 0x7fbf48161670 2024-03-04T09:32:35.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.145+0000 7fbf4da00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fbf480ae0a0 0x7fbf480711b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:35.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.145+0000 7fbf47e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fbf4806e240 0x7fbf4806cfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:36878/0 (socket says 172.21.15.6:36878) 2024-03-04T09:32:35.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.145+0000 7fbf47e00640 1 -- 172.21.15.6:0/265701236 learned_addr learned my addr 172.21.15.6:0/265701236 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:32:35.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.145+0000 7fbf4d000640 1 --2- 172.21.15.6:0/265701236 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbf48161670 0x7fbf48070c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:35.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.146+0000 7fbf4da00640 1 -- 172.21.15.6:0/265701236 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbf48161670 msgr2=0x7fbf48070c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:32:35.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.146+0000 7fbf4da00640 1 --2- 172.21.15.6:0/265701236 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbf48161670 0x7fbf48070c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:35.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.146+0000 7fbf4da00640 1 -- 172.21.15.6:0/265701236 >> v1:172.21.15.6:6789/0 conn(0x7fbf4806e240 legacy=0x7fbf4806cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:35.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.146+0000 7fbf47400640 1 -- 172.21.15.6:0/265701236 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3110338146 0 0) 0x7fbf480ce9a0 con 0x7fbf4806e240 2024-03-04T09:32:35.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.146+0000 7fbf4da00640 1 -- 172.21.15.6:0/265701236 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf480cef60 con 0x7fbf480ae0a0 2024-03-04T09:32:35.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.147+0000 7fbf4da00640 1 --2- 172.21.15.6:0/265701236 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbf480ae0a0 0x7fbf480711b0 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7fbf40009d90 tx=0x7fbf4002bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba0378e58be12ed5 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:35.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.147+0000 7fbf47400640 1 -- 172.21.15.6:0/265701236 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf40039070 con 0x7fbf480ae0a0 2024-03-04T09:32:35.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.147+0000 7fbf47400640 1 -- 172.21.15.6:0/265701236 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf4002cc50 con 0x7fbf480ae0a0 2024-03-04T09:32:35.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.147+0000 7fbf47400640 1 -- 172.21.15.6:0/265701236 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf40034440 con 0x7fbf480ae0a0 2024-03-04T09:32:35.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.148+0000 7fbf4fa00640 1 -- 172.21.15.6:0/265701236 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbf480ae0a0 msgr2=0x7fbf480711b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:35.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.148+0000 7fbf4fa00640 1 --2- 172.21.15.6:0/265701236 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbf480ae0a0 0x7fbf480711b0 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7fbf40009d90 tx=0x7fbf4002bea0 comp rx=0 tx=0).stop 2024-03-04T09:32:35.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.148+0000 7fbf4fa00640 1 -- 172.21.15.6:0/265701236 shutdown_connections 2024-03-04T09:32:35.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.148+0000 7fbf4fa00640 1 --2- 172.21.15.6:0/265701236 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbf480ae0a0 0x7fbf480711b0 unknown :-1 s=CLOSED pgs=1389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:35.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.148+0000 7fbf4fa00640 1 --2- 172.21.15.6:0/265701236 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbf48161670 0x7fbf48070c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:35.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.148+0000 7fbf4fa00640 1 -- 172.21.15.6:0/265701236 >> 172.21.15.6:0/265701236 conn(0x7fbf4806afb0 msgr2=0x7fbf4806b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:32:35.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.148+0000 7fbf4fa00640 1 -- 172.21.15.6:0/265701236 shutdown_connections 2024-03-04T09:32:35.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.148+0000 7fbf4fa00640 1 -- 172.21.15.6:0/265701236 wait complete. 2024-03-04T09:32:35.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.150+0000 7fbf4fa00640 1 Processor -- start 2024-03-04T09:32:35.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.150+0000 7fbf4fa00640 1 -- start start 2024-03-04T09:32:35.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.151+0000 7fbf4fa00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbf4806e240 0x7fbf481f3e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:35.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.151+0000 7fbf4fa00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbf480ae0a0 0x7fbf480c9150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:35.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.151+0000 7fbf4fa00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbf48161670 0x7fbf48150770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:35.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.151+0000 7fbf4fa00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf480ce600 con 0x7fbf480ae0a0 2024-03-04T09:32:35.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.151+0000 7fbf4fa00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf480cf520 con 0x7fbf4806e240 2024-03-04T09:32:35.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.151+0000 7fbf4fa00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf480ce780 con 0x7fbf48161670 2024-03-04T09:32:35.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.151+0000 7fbf47e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbf480ae0a0 0x7fbf480c9150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:35.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.151+0000 7fbf47e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbf480ae0a0 0x7fbf480c9150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:44668/0 (socket says 172.21.15.6:44668) 2024-03-04T09:32:35.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.151+0000 7fbf47e00640 1 -- 172.21.15.6:0/1125684991 learned_addr learned my addr 172.21.15.6:0/1125684991 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:32:35.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.152+0000 7fbf4da00640 1 --2- 172.21.15.6:0/1125684991 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbf48161670 0x7fbf48150770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:35.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.152+0000 7fbf4d000640 1 --2- 172.21.15.6:0/1125684991 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbf4806e240 0x7fbf481f3e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:35.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.152+0000 7fbf47e00640 1 -- 172.21.15.6:0/1125684991 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbf48161670 msgr2=0x7fbf48150770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:35.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.152+0000 7fbf47e00640 1 --2- 172.21.15.6:0/1125684991 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbf48161670 0x7fbf48150770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:35.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.152+0000 7fbf47e00640 1 -- 172.21.15.6:0/1125684991 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbf4806e240 msgr2=0x7fbf481f3e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:35.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.152+0000 7fbf47e00640 1 --2- 172.21.15.6:0/1125684991 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbf4806e240 0x7fbf481f3e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:35.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.152+0000 7fbf47e00640 1 -- 172.21.15.6:0/1125684991 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf48150cb0 con 0x7fbf480ae0a0 2024-03-04T09:32:35.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.152+0000 7fbf4da00640 1 --2- 172.21.15.6:0/1125684991 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbf48161670 0x7fbf48150770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:32:35.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.152+0000 7fbf47e00640 1 --2- 172.21.15.6:0/1125684991 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbf480ae0a0 0x7fbf480c9150 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7fbf3800c500 tx=0x7fbf3800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:35.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.152+0000 7fbf44c00640 1 -- 172.21.15.6:0/1125684991 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf38018070 con 0x7fbf480ae0a0 2024-03-04T09:32:35.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.152+0000 7fbf44c00640 1 -- 172.21.15.6:0/1125684991 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf38007580 con 0x7fbf480ae0a0 2024-03-04T09:32:35.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.152+0000 7fbf44c00640 1 -- 172.21.15.6:0/1125684991 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf38004a20 con 0x7fbf480ae0a0 2024-03-04T09:32:35.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.152+0000 7fbf4fa00640 1 -- 172.21.15.6:0/1125684991 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf481f4460 con 0x7fbf480ae0a0 2024-03-04T09:32:35.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.153+0000 7fbf4fa00640 1 -- 172.21.15.6:0/1125684991 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf481f4a20 con 0x7fbf480ae0a0 2024-03-04T09:32:35.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.154+0000 7fbf4fa00640 1 -- 172.21.15.6:0/1125684991 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf48069510 con 0x7fbf480ae0a0 2024-03-04T09:32:35.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.155+0000 7fbf44c00640 1 -- 172.21.15.6:0/1125684991 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbf38004bc0 con 0x7fbf480ae0a0 2024-03-04T09:32:35.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.156+0000 7fbf44c00640 1 --2- 172.21.15.6:0/1125684991 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fbf2c06b4d0 0x7fbf2c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:35.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.156+0000 7fbf44c00640 1 -- 172.21.15.6:0/1125684991 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5853+0+0 (secure 0 0 0) 0x7fbf38089480 con 0x7fbf480ae0a0 2024-03-04T09:32:35.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.157+0000 7fbf4d000640 1 --2- 172.21.15.6:0/1125684991 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fbf2c06b4d0 0x7fbf2c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:35.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.158+0000 7fbf4d000640 1 --2- 172.21.15.6:0/1125684991 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fbf2c06b4d0 0x7fbf2c06d990 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fbf3c006fd0 tx=0x7fbf3c008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:35.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.161+0000 7fbf44c00640 1 -- 172.21.15.6:0/1125684991 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbf38058260 con 0x7fbf480ae0a0 2024-03-04T09:32:35.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.364+0000 7fbf4fa00640 1 -- 172.21.15.6:0/1125684991 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbf480cf990 con 0x7fbf480ae0a0 2024-03-04T09:32:35.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.367+0000 7fbf44c00640 1 -- 172.21.15.6:0/1125684991 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v115) v1 ==== 74+0+16859 (secure 0 0 0) 0x7fbf38057b60 con 0x7fbf480ae0a0 2024-03-04T09:32:35.364 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:32:35.364 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":115,"fsid":"edce0578-5ae4-4402-b45a-552bdef84d7c","created":"2024-03-04T09:24:54.523780+0000","modified":"2024-03-04T09:32:03.117903+0000","last_up_change":"2024-03-04T09:25:07.089049+0000","last_in_change":"2024-03-04T09:24:57.982077+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T09:25:07.935198+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-04T09:31:19.018233+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-04T09:31:20.131544+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a96574f-81f1-4bc6-8463-757520b84a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6825","nonce":1364449401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6827","nonce":1364449401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6831","nonce":1364449401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6829","nonce":1364449401}]},"public_addr":"172.21.15.6:6825/1364449401","cluster_addr":"172.21.15.6:6827/1364449401","heartbeat_back_addr":"172.21.15.6:6831/1364449401","heartbeat_front_addr":"172.21.15.6:6829/1364449401","state":["exists","up"]},{"osd":1,"uuid":"7d79b9c1-6b87-476a-bc36-7db5ee64cd90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6809","nonce":2494103078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6811","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6813","nonce":2494103078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6821","nonce":2494103078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6817","nonce":2494103078}]},"public_addr":"172.21.15.6:6809/2494103078","cluster_addr":"172.21.15.6:6813/2494103078","heartbeat_back_addr":"172.21.15.6:6821/2494103078","heartbeat_front_addr":"172.21.15.6:6817/2494103078","state":["exists","up"]},{"osd":2,"uuid":"492331b9-4ee9-40b6-bacd-85d44cd6c45a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6812","nonce":1500066681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6816","nonce":1500066681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6824","nonce":1500066681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6820","nonce":1500066681}]},"public_addr":"172.21.15.6:6812/1500066681","cluster_addr":"172.21.15.6:6816/1500066681","heartbeat_back_addr":"172.21.15.6:6824/1500066681","heartbeat_front_addr":"172.21.15.6:6820/1500066681","state":["exists","up"]},{"osd":3,"uuid":"21bb2efe-9ba7-4bdc-bc31-180cd1102e21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6801","nonce":1558806123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6803","nonce":1558806123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6807","nonce":1558806123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6805","nonce":1558806123}]},"public_addr":"172.21.15.6:6801/1558806123","cluster_addr":"172.21.15.6:6803/1558806123","heartbeat_back_addr":"172.21.15.6:6807/1558806123","heartbeat_front_addr":"172.21.15.6:6805/1558806123","state":["exists","up"]},{"osd":4,"uuid":"6e2c4b71-19d9-4737-83d9-558705d8644a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6801","nonce":1526135780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6803","nonce":1526135780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6807","nonce":1526135780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6805","nonce":1526135780}]},"public_addr":"172.21.15.146:6801/1526135780","cluster_addr":"172.21.15.146:6803/1526135780","heartbeat_back_addr":"172.21.15.146:6807/1526135780","heartbeat_front_addr":"172.21.15.146:6805/1526135780","state":["exists","up"]},{"osd":5,"uuid":"fb69c525-f1be-4b7d-ab32-6c29b0d6fddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6809","nonce":460262507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6811","nonce":460262507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6815","nonce":460262507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6813","nonce":460262507}]},"public_addr":"172.21.15.146:6809/460262507","cluster_addr":"172.21.15.146:6811/460262507","heartbeat_back_addr":"172.21.15.146:6815/460262507","heartbeat_front_addr":"172.21.15.146:6813/460262507","state":["exists","up"]},{"osd":6,"uuid":"c32e7fed-370d-4401-8ce0-6a4335b0d168","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6823","nonce":1392897493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6827","nonce":1392897493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6831","nonce":1392897493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6829","nonce":1392897493}]},"public_addr":"172.21.15.146:6823/1392897493","cluster_addr":"172.21.15.146:6827/1392897493","heartbeat_back_addr":"172.21.15.146:6831/1392897493","heartbeat_front_addr":"172.21.15.146:6829/1392897493","state":["exists","up"]},{"osd":7,"uuid":"75a79b44-fa5f-426d-a7d3-71f21f78835c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6817","nonce":3268323156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6819","nonce":3268323156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6825","nonce":3268323156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6822","nonce":3268323156}]},"public_addr":"172.21.15.146:6817/3268323156","cluster_addr":"172.21.15.146:6819/3268323156","heartbeat_back_addr":"172.21.15.146:6825/3268323156","heartbeat_front_addr":"172.21.15.146:6822/3268323156","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.310437+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.295850+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.294674+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.526855+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.300023+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.595160+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.346046+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.558741+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"17.e","mappings":[{"from":0,"to":5}]},{"pgid":"17.26","mappings":[{"from":0,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.6:6834/2697931210":"2024-03-05T09:31:36.737386+0000","172.21.15.6:6832/2697931210":"2024-03-05T09:31:36.737386+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T09:32:35.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.370+0000 7fbf4fa00640 1 -- 172.21.15.6:0/1125684991 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fbf2c06b4d0 msgr2=0x7fbf2c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:35.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.370+0000 7fbf4fa00640 1 --2- 172.21.15.6:0/1125684991 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fbf2c06b4d0 0x7fbf2c06d990 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fbf3c006fd0 tx=0x7fbf3c008040 comp rx=0 tx=0).stop 2024-03-04T09:32:35.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.370+0000 7fbf4fa00640 1 -- 172.21.15.6:0/1125684991 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbf480ae0a0 msgr2=0x7fbf480c9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:35.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.370+0000 7fbf4fa00640 1 --2- 172.21.15.6:0/1125684991 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbf480ae0a0 0x7fbf480c9150 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7fbf3800c500 tx=0x7fbf3800c9c0 comp rx=0 tx=0).stop 2024-03-04T09:32:35.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.370+0000 7fbf4fa00640 1 -- 172.21.15.6:0/1125684991 shutdown_connections 2024-03-04T09:32:35.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.370+0000 7fbf4fa00640 1 --2- 172.21.15.6:0/1125684991 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fbf2c06b4d0 0x7fbf2c06d990 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:35.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.370+0000 7fbf4fa00640 1 --2- 172.21.15.6:0/1125684991 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbf48161670 0x7fbf48150770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:35.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.370+0000 7fbf4fa00640 1 --2- 172.21.15.6:0/1125684991 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbf480ae0a0 0x7fbf480c9150 unknown :-1 s=CLOSED pgs=1390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:35.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.370+0000 7fbf4fa00640 1 --2- 172.21.15.6:0/1125684991 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbf4806e240 0x7fbf481f3e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:35.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.370+0000 7fbf4fa00640 1 -- 172.21.15.6:0/1125684991 >> 172.21.15.6:0/1125684991 conn(0x7fbf4806afb0 msgr2=0x7fbf4806b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:32:35.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.370+0000 7fbf4fa00640 1 -- 172.21.15.6:0/1125684991 shutdown_connections 2024-03-04T09:32:35.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.370+0000 7fbf4fa00640 1 -- 172.21.15.6:0/1125684991 wait complete. 2024-03-04T09:32:35.387 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T09:25:07.935198+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-04T09:31:19.018233+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '108', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2024-03-04T09:31:20.131544+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '109', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-04T09:32:35.388 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-04T09:32:35.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.608+0000 7ff0da600640 1 Processor -- start 2024-03-04T09:32:35.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.608+0000 7ff0da600640 1 -- start start 2024-03-04T09:32:35.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.610+0000 7ff0da600640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ff0d40d4890 0x7ff0d40d1de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:35.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.610+0000 7ff0da600640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0d40b2c00 con 0x7ff0d40d2320 2024-03-04T09:32:35.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.610+0000 7ff0da600640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0d4161ae0 con 0x7ff0d4161700 2024-03-04T09:32:35.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.610+0000 7ff0da600640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0d40b56e0 con 0x7ff0d40d4890 2024-03-04T09:32:35.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.610+0000 7ff0d9c00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7ff0d40d2320 0x7ff0d40b1220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:36880/0 (socket says 172.21.15.6:36880) 2024-03-04T09:32:35.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.610+0000 7ff0d9c00640 1 -- 172.21.15.6:0/986430055 learned_addr learned my addr 172.21.15.6:0/986430055 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:32:35.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.611+0000 7ff0d3400640 1 -- 172.21.15.6:0/986430055 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3063843666 0 0) 0x7ff0d40b2c00 con 0x7ff0d40d2320 2024-03-04T09:32:35.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.611+0000 7ff0d3e00640 1 --2- 172.21.15.6:0/986430055 >> v2:172.21.15.146:3300/0 conn(0x7ff0d40d4890 0x7ff0d40d1de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:35.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.611+0000 7ff0d3400640 1 -- 172.21.15.6:0/986430055 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0bc003520 con 0x7ff0d40d2320 2024-03-04T09:32:35.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.611+0000 7ff0d3400640 1 -- 172.21.15.6:0/986430055 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 192639550 0 0) 0x7ff0bc003520 con 0x7ff0d40d2320 2024-03-04T09:32:35.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.611+0000 7ff0d3400640 1 -- 172.21.15.6:0/986430055 >> v2:172.21.15.146:3300/0 conn(0x7ff0d40d4890 msgr2=0x7ff0d40d1de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:32:35.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.611+0000 7ff0d3400640 1 --2- 172.21.15.6:0/986430055 >> v2:172.21.15.146:3300/0 conn(0x7ff0d40d4890 0x7ff0d40d1de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:35.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.611+0000 7ff0d3400640 1 -- 172.21.15.6:0/986430055 >> v1:172.21.15.146:6789/0 conn(0x7ff0d4161700 legacy=0x7ff0d40b4fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:35.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.611+0000 7ff0d3400640 1 -- 172.21.15.6:0/986430055 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0d40b2ea0 con 0x7ff0d40d2320 2024-03-04T09:32:35.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.611+0000 7ff0d3400640 1 -- 172.21.15.6:0/986430055 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7ff0cc0019f0 con 0x7ff0d40d2320 2024-03-04T09:32:35.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.611+0000 7ff0d3400640 1 -- 172.21.15.6:0/986430055 >> v1:172.21.15.6:6789/0 conn(0x7ff0d40d2320 legacy=0x7ff0d40b1220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:35.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.611+0000 7ff0d3400640 1 --2- 172.21.15.6:0/986430055 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff0bc003a00 0x7ff0bc003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:35.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.611+0000 7ff0d3400640 1 --2- 172.21.15.6:0/986430055 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff0bc004800 0x7ff0bc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:35.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.612+0000 7ff0d3400640 1 --2- 172.21.15.6:0/986430055 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff0bc005210 0x7ff0bc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:35.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.612+0000 7ff0d3400640 1 -- 172.21.15.6:0/986430055 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0d40b56e0 con 0x7ff0bc005210 2024-03-04T09:32:35.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.612+0000 7ff0d3400640 1 -- 172.21.15.6:0/986430055 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0bc00c160 con 0x7ff0bc004800 2024-03-04T09:32:35.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.612+0000 7ff0d3400640 1 -- 172.21.15.6:0/986430055 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0bc00c360 con 0x7ff0bc003a00 2024-03-04T09:32:35.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.612+0000 7ff0d9c00640 1 --2- 172.21.15.6:0/986430055 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff0bc005210 0x7ff0bc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:35.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.612+0000 7ff0d9200640 1 --2- 172.21.15.6:0/986430055 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff0bc003a00 0x7ff0bc003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:35.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.612+0000 7ff0d3e00640 1 --2- 172.21.15.6:0/986430055 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff0bc004800 0x7ff0bc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:35.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.612+0000 7ff0d9c00640 1 -- 172.21.15.6:0/986430055 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff0bc003a00 msgr2=0x7ff0bc003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:32:35.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.612+0000 7ff0d9c00640 1 --2- 172.21.15.6:0/986430055 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff0bc003a00 0x7ff0bc003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:35.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.612+0000 7ff0d9c00640 1 -- 172.21.15.6:0/986430055 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff0bc004800 msgr2=0x7ff0bc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:32:35.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.612+0000 7ff0d9c00640 1 --2- 172.21.15.6:0/986430055 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff0bc004800 0x7ff0bc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:35.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.612+0000 7ff0d9c00640 1 -- 172.21.15.6:0/986430055 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0bc003520 con 0x7ff0bc005210 2024-03-04T09:32:35.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.612+0000 7ff0d3e00640 1 --2- 172.21.15.6:0/986430055 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff0bc004800 0x7ff0bc004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:32:35.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.613+0000 7ff0d9c00640 1 -- 172.21.15.6:0/986430055 reap_dead start 2024-03-04T09:32:35.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.614+0000 7ff0d9c00640 1 --2- 172.21.15.6:0/986430055 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff0bc005210 0x7ff0bc00b9c0 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7ff0cc00a7b0 tx=0x7ff0bc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de6ebc7b767922d9 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:35.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.614+0000 7ff0d3400640 1 -- 172.21.15.6:0/986430055 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0cc00a3d0 con 0x7ff0bc005210 2024-03-04T09:32:35.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.614+0000 7ff0d3400640 1 -- 172.21.15.6:0/986430055 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0cc034650 con 0x7ff0bc005210 2024-03-04T09:32:35.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.614+0000 7ff0d3400640 1 -- 172.21.15.6:0/986430055 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0cc034a40 con 0x7ff0bc005210 2024-03-04T09:32:35.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.614+0000 7ff0da600640 1 -- 172.21.15.6:0/986430055 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff0bc005210 msgr2=0x7ff0bc00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:35.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.614+0000 7ff0da600640 1 --2- 172.21.15.6:0/986430055 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff0bc005210 0x7ff0bc00b9c0 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7ff0cc00a7b0 tx=0x7ff0bc005830 comp rx=0 tx=0).stop 2024-03-04T09:32:35.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.615+0000 7ff0da600640 1 -- 172.21.15.6:0/986430055 shutdown_connections 2024-03-04T09:32:35.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.615+0000 7ff0da600640 1 --2- 172.21.15.6:0/986430055 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff0bc005210 0x7ff0bc00b9c0 unknown :-1 s=CLOSED pgs=1392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:35.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.615+0000 7ff0da600640 1 -- 172.21.15.6:0/986430055 >> 172.21.15.6:0/986430055 conn(0x7ff0d40d3d70 msgr2=0x7ff0d40d4180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:32:35.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.615+0000 7ff0da600640 1 -- 172.21.15.6:0/986430055 shutdown_connections 2024-03-04T09:32:35.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.615+0000 7ff0da600640 1 -- 172.21.15.6:0/986430055 wait complete. 2024-03-04T09:32:35.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.616+0000 7ff0da600640 1 Processor -- start 2024-03-04T09:32:35.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.617+0000 7ff0da600640 1 -- start start 2024-03-04T09:32:35.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.617+0000 7ff0da600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff0bc005210 0x7ff0d406cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:35.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.617+0000 7ff0da600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff0d40c97e0 0x7ff0d40b8cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:35.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.618+0000 7ff0da600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff0d40b9210 0x7ff0d41f3bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:35.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.618+0000 7ff0da600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0d40b0a20 con 0x7ff0d40c97e0 2024-03-04T09:32:35.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.618+0000 7ff0da600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0d40d2750 con 0x7ff0d40b9210 2024-03-04T09:32:35.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.618+0000 7ff0da600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0d40b0d00 con 0x7ff0bc005210 2024-03-04T09:32:35.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.618+0000 7ff0d3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff0d40c97e0 0x7ff0d40b8cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:35.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.618+0000 7ff0d3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff0d40c97e0 0x7ff0d40b8cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:44682/0 (socket says 172.21.15.6:44682) 2024-03-04T09:32:35.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.618+0000 7ff0d3e00640 1 -- 172.21.15.6:0/2093981518 learned_addr learned my addr 172.21.15.6:0/2093981518 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:32:35.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.618+0000 7ff0d9200640 1 --2- 172.21.15.6:0/2093981518 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff0bc005210 0x7ff0d406cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:35.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.618+0000 7ff0d9c00640 1 --2- 172.21.15.6:0/2093981518 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff0d40b9210 0x7ff0d41f3bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:35.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.618+0000 7ff0d3e00640 1 -- 172.21.15.6:0/2093981518 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff0bc005210 msgr2=0x7ff0d406cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:35.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.618+0000 7ff0d3e00640 1 --2- 172.21.15.6:0/2093981518 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff0bc005210 0x7ff0d406cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:35.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.618+0000 7ff0d3e00640 1 -- 172.21.15.6:0/2093981518 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff0d40b9210 msgr2=0x7ff0d41f3bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:35.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.618+0000 7ff0d3e00640 1 --2- 172.21.15.6:0/2093981518 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff0d40b9210 0x7ff0d41f3bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:35.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.618+0000 7ff0d3e00640 1 -- 172.21.15.6:0/2093981518 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0d40b96a0 con 0x7ff0d40c97e0 2024-03-04T09:32:35.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.619+0000 7ff0d3e00640 1 --2- 172.21.15.6:0/2093981518 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff0d40c97e0 0x7ff0d40b8cd0 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7ff0c000e7e0 tx=0x7ff0c000eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:35.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.619+0000 7ff0d0c00640 1 -- 172.21.15.6:0/2093981518 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0c0017070 con 0x7ff0d40c97e0 2024-03-04T09:32:35.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.619+0000 7ff0d0c00640 1 -- 172.21.15.6:0/2093981518 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0c000c6d0 con 0x7ff0d40c97e0 2024-03-04T09:32:35.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.619+0000 7ff0da600640 1 -- 172.21.15.6:0/2093981518 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0d40c8a20 con 0x7ff0d40c97e0 2024-03-04T09:32:35.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.619+0000 7ff0d0c00640 1 -- 172.21.15.6:0/2093981518 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0c000f040 con 0x7ff0d40c97e0 2024-03-04T09:32:35.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.619+0000 7ff0da600640 1 -- 172.21.15.6:0/2093981518 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0d40c8fe0 con 0x7ff0d40c97e0 2024-03-04T09:32:35.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.621+0000 7ff0da600640 1 -- 172.21.15.6:0/2093981518 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0d404f330 con 0x7ff0d40c97e0 2024-03-04T09:32:35.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.622+0000 7ff0d0c00640 1 -- 172.21.15.6:0/2093981518 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff0c001a070 con 0x7ff0d40c97e0 2024-03-04T09:32:35.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.623+0000 7ff0d0c00640 1 --2- 172.21.15.6:0/2093981518 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff0b406b500 0x7ff0b406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:35.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.623+0000 7ff0d0c00640 1 -- 172.21.15.6:0/2093981518 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5853+0+0 (secure 0 0 0) 0x7ff0c0089eb0 con 0x7ff0d40c97e0 2024-03-04T09:32:35.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.624+0000 7ff0d9200640 1 --2- 172.21.15.6:0/2093981518 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff0b406b500 0x7ff0b406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:35.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.625+0000 7ff0d9200640 1 --2- 172.21.15.6:0/2093981518 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff0b406b500 0x7ff0b406d9c0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7ff0c8007fd0 tx=0x7ff0c8009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:35.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.632+0000 7ff0d0c00640 1 -- 172.21.15.6:0/2093981518 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff0c0058c90 con 0x7ff0d40c97e0 2024-03-04T09:32:35.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.870+0000 7ff0da600640 1 -- 172.21.15.6:0/2093981518 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff0d40d1dc0 con 0x7ff0d40c97e0 2024-03-04T09:32:35.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.871+0000 7ff0d0c00640 1 -- 172.21.15.6:0/2093981518 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v115) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff0c0058590 con 0x7ff0d40c97e0 2024-03-04T09:32:35.868 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-04T09:32:35.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.874+0000 7ff0da600640 1 -- 172.21.15.6:0/2093981518 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff0b406b500 msgr2=0x7ff0b406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:35.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.874+0000 7ff0da600640 1 --2- 172.21.15.6:0/2093981518 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff0b406b500 0x7ff0b406d9c0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7ff0c8007fd0 tx=0x7ff0c8009040 comp rx=0 tx=0).stop 2024-03-04T09:32:35.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.874+0000 7ff0da600640 1 -- 172.21.15.6:0/2093981518 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff0d40c97e0 msgr2=0x7ff0d40b8cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:35.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.874+0000 7ff0da600640 1 --2- 172.21.15.6:0/2093981518 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff0d40c97e0 0x7ff0d40b8cd0 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7ff0c000e7e0 tx=0x7ff0c000eca0 comp rx=0 tx=0).stop 2024-03-04T09:32:35.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.874+0000 7ff0da600640 1 -- 172.21.15.6:0/2093981518 shutdown_connections 2024-03-04T09:32:35.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.874+0000 7ff0da600640 1 --2- 172.21.15.6:0/2093981518 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff0d40b9210 0x7ff0d41f3bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:35.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.874+0000 7ff0da600640 1 --2- 172.21.15.6:0/2093981518 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff0d40c97e0 0x7ff0d40b8cd0 unknown :-1 s=CLOSED pgs=1393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:35.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.874+0000 7ff0da600640 1 --2- 172.21.15.6:0/2093981518 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff0b406b500 0x7ff0b406d9c0 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:35.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.874+0000 7ff0da600640 1 --2- 172.21.15.6:0/2093981518 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff0bc005210 0x7ff0d406cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:35.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.874+0000 7ff0da600640 1 -- 172.21.15.6:0/2093981518 >> 172.21.15.6:0/2093981518 conn(0x7ff0d40d3d70 msgr2=0x7ff0d40b4ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:32:35.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.874+0000 7ff0da600640 1 -- 172.21.15.6:0/2093981518 shutdown_connections 2024-03-04T09:32:35.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:35.874+0000 7ff0da600640 1 -- 172.21.15.6:0/2093981518 wait complete. 2024-03-04T09:32:35.891 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-04T09:32:36.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.103+0000 7f9511000640 1 Processor -- start 2024-03-04T09:32:36.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.103+0000 7f9511000640 1 -- start start 2024-03-04T09:32:36.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.104+0000 7f9511000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f950c15e2d0 0x7f950c0b5450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:36.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.104+0000 7f9511000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f950c165b60 0x7f950c167f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:36.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.104+0000 7f9511000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f950c0b5da0 con 0x7f950c06cab0 2024-03-04T09:32:36.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.105+0000 7f9511000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f950c0b6080 con 0x7f950c15e2d0 2024-03-04T09:32:36.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.105+0000 7f9511000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f950c0b6360 con 0x7f950c165b60 2024-03-04T09:32:36.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.105+0000 7f9502a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f950c15e2d0 0x7f950c0b5450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:36.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.105+0000 7f950b400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f950c06cab0 0x7f950c06ce90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:36886/0 (socket says 172.21.15.6:36886) 2024-03-04T09:32:36.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.105+0000 7f9502a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f950c15e2d0 0x7f950c0b5450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:44696/0 (socket says 172.21.15.6:44696) 2024-03-04T09:32:36.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.105+0000 7f950b400640 1 -- 172.21.15.6:0/1768735038 learned_addr learned my addr 172.21.15.6:0/1768735038 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:32:36.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.105+0000 7f950be00640 1 --2- 172.21.15.6:0/1768735038 >> v2:172.21.15.146:3300/0 conn(0x7f950c165b60 0x7f950c167f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:36.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.105+0000 7f9502a00640 1 -- 172.21.15.6:0/1768735038 >> v2:172.21.15.146:3300/0 conn(0x7f950c165b60 msgr2=0x7f950c167f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:32:36.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.105+0000 7f9502a00640 1 --2- 172.21.15.6:0/1768735038 >> v2:172.21.15.146:3300/0 conn(0x7f950c165b60 0x7f950c167f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:36.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.105+0000 7f9502a00640 1 -- 172.21.15.6:0/1768735038 >> v1:172.21.15.6:6789/0 conn(0x7f950c06cab0 legacy=0x7f950c06ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:36.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.105+0000 7f9502a00640 1 -- 172.21.15.6:0/1768735038 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f950c0b6640 con 0x7f950c15e2d0 2024-03-04T09:32:36.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.107+0000 7f9502a00640 1 --2- 172.21.15.6:0/1768735038 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f950c15e2d0 0x7f950c0b5450 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7f950c071bf0 tx=0x7f94f402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef9953cbb837233a server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:36.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.107+0000 7f950aa00640 1 -- 172.21.15.6:0/1768735038 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94f4035020 con 0x7f950c15e2d0 2024-03-04T09:32:36.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.107+0000 7f950aa00640 1 -- 172.21.15.6:0/1768735038 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94f4031830 con 0x7f950c15e2d0 2024-03-04T09:32:36.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.107+0000 7f950aa00640 1 -- 172.21.15.6:0/1768735038 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94f4031c00 con 0x7f950c15e2d0 2024-03-04T09:32:36.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.108+0000 7f9511000640 1 -- 172.21.15.6:0/1768735038 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f950c15e2d0 msgr2=0x7f950c0b5450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:36.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.108+0000 7f9511000640 1 --2- 172.21.15.6:0/1768735038 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f950c15e2d0 0x7f950c0b5450 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7f950c071bf0 tx=0x7f94f402b8d0 comp rx=0 tx=0).stop 2024-03-04T09:32:36.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.108+0000 7f9511000640 1 -- 172.21.15.6:0/1768735038 shutdown_connections 2024-03-04T09:32:36.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.108+0000 7f9511000640 1 --2- 172.21.15.6:0/1768735038 >> v2:172.21.15.146:3300/0 conn(0x7f950c165b60 0x7f950c167f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:36.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.108+0000 7f9511000640 1 --2- 172.21.15.6:0/1768735038 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f950c15e2d0 0x7f950c0b5450 unknown :-1 s=CLOSED pgs=1395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:36.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.108+0000 7f9511000640 1 -- 172.21.15.6:0/1768735038 >> 172.21.15.6:0/1768735038 conn(0x7f950c09aa10 msgr2=0x7f950c09ae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:32:36.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.108+0000 7f9511000640 1 -- 172.21.15.6:0/1768735038 shutdown_connections 2024-03-04T09:32:36.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.108+0000 7f9511000640 1 -- 172.21.15.6:0/1768735038 wait complete. 2024-03-04T09:32:36.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.110+0000 7f9511000640 1 Processor -- start 2024-03-04T09:32:36.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.110+0000 7f9511000640 1 -- start start 2024-03-04T09:32:36.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.112+0000 7f9511000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f950c06cab0 0x7f950c13e790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:36.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.112+0000 7f9511000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f950c15e2d0 0x7f950c1573a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:36.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.112+0000 7f9511000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f950c165b60 0x7f950c06ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:36.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.112+0000 7f9511000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f950c15e8b0 con 0x7f950c165b60 2024-03-04T09:32:36.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.112+0000 7f9511000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f950c15e730 con 0x7f950c15e2d0 2024-03-04T09:32:36.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.112+0000 7f9511000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f950c0b5990 con 0x7f950c06cab0 2024-03-04T09:32:36.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.112+0000 7f950be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f950c165b60 0x7f950c06ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:36.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.112+0000 7f950be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f950c165b60 0x7f950c06ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:44706/0 (socket says 172.21.15.6:44706) 2024-03-04T09:32:36.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.112+0000 7f950be00640 1 -- 172.21.15.6:0/1530650598 learned_addr learned my addr 172.21.15.6:0/1530650598 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:32:36.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.112+0000 7f9502a00640 1 --2- 172.21.15.6:0/1530650598 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f950c15e2d0 0x7f950c1573a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:36.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.112+0000 7f950b400640 1 --2- 172.21.15.6:0/1530650598 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f950c06cab0 0x7f950c13e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:36.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.113+0000 7f950be00640 1 -- 172.21.15.6:0/1530650598 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f950c06cab0 msgr2=0x7f950c13e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:36.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.113+0000 7f950be00640 1 --2- 172.21.15.6:0/1530650598 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f950c06cab0 0x7f950c13e790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:36.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.113+0000 7f950be00640 1 -- 172.21.15.6:0/1530650598 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f950c15e2d0 msgr2=0x7f950c1573a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:36.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.113+0000 7f950be00640 1 --2- 172.21.15.6:0/1530650598 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f950c15e2d0 0x7f950c1573a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:36.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.113+0000 7f950be00640 1 -- 172.21.15.6:0/1530650598 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f950c13ecd0 con 0x7f950c165b60 2024-03-04T09:32:36.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.113+0000 7f950be00640 1 --2- 172.21.15.6:0/1530650598 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f950c165b60 0x7f950c06ef90 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7f9504009fd0 tx=0x7f950400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:36.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.113+0000 7f9503e00640 1 -- 172.21.15.6:0/1530650598 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9504014420 con 0x7f950c165b60 2024-03-04T09:32:36.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.113+0000 7f9503e00640 1 -- 172.21.15.6:0/1530650598 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9504019070 con 0x7f950c165b60 2024-03-04T09:32:36.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.113+0000 7f9503e00640 1 -- 172.21.15.6:0/1530650598 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9504014800 con 0x7f950c165b60 2024-03-04T09:32:36.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.113+0000 7f9511000640 1 -- 172.21.15.6:0/1530650598 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f950c157d80 con 0x7f950c165b60 2024-03-04T09:32:36.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.114+0000 7f9511000640 1 -- 172.21.15.6:0/1530650598 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f950c158340 con 0x7f950c165b60 2024-03-04T09:32:36.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.115+0000 7f9511000640 1 -- 172.21.15.6:0/1530650598 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f950c04f330 con 0x7f950c165b60 2024-03-04T09:32:36.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.116+0000 7f9503e00640 1 -- 172.21.15.6:0/1530650598 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f95040149a0 con 0x7f950c165b60 2024-03-04T09:32:36.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.117+0000 7f9503e00640 1 --2- 172.21.15.6:0/1530650598 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f94e806b4d0 0x7f94e806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:36.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.117+0000 7f9503e00640 1 -- 172.21.15.6:0/1530650598 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f950408abd0 con 0x7f950c165b60 2024-03-04T09:32:36.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.117+0000 7f950b400640 1 --2- 172.21.15.6:0/1530650598 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f94e806b4d0 0x7f94e806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:36.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.119+0000 7f950b400640 1 --2- 172.21.15.6:0/1530650598 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f94e806b4d0 0x7f94e806d990 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f94fc000f60 tx=0x7f94fc009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:36.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.126+0000 7f9503e00640 1 -- 172.21.15.6:0/1530650598 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9504059920 con 0x7f950c165b60 2024-03-04T09:32:36.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.324+0000 7f9511000640 1 -- 172.21.15.6:0/1530650598 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f950c15d2e0 con 0x7f950c165b60 2024-03-04T09:32:36.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.325+0000 7f9503e00640 1 -- 172.21.15.6:0/1530650598 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v115) v1 ==== 104+0+11 (secure 0 0 0) 0x7f9504059220 con 0x7f950c165b60 2024-03-04T09:32:36.322 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-03-04T09:32:36.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.327+0000 7f9511000640 1 -- 172.21.15.6:0/1530650598 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f94e806b4d0 msgr2=0x7f94e806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:36.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.327+0000 7f9511000640 1 --2- 172.21.15.6:0/1530650598 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f94e806b4d0 0x7f94e806d990 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f94fc000f60 tx=0x7f94fc009040 comp rx=0 tx=0).stop 2024-03-04T09:32:36.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.327+0000 7f9511000640 1 -- 172.21.15.6:0/1530650598 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f950c165b60 msgr2=0x7f950c06ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:36.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.327+0000 7f9511000640 1 --2- 172.21.15.6:0/1530650598 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f950c165b60 0x7f950c06ef90 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7f9504009fd0 tx=0x7f950400e4b0 comp rx=0 tx=0).stop 2024-03-04T09:32:36.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.327+0000 7f9511000640 1 -- 172.21.15.6:0/1530650598 shutdown_connections 2024-03-04T09:32:36.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.327+0000 7f9511000640 1 --2- 172.21.15.6:0/1530650598 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f950c165b60 0x7f950c06ef90 unknown :-1 s=CLOSED pgs=1396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:36.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.327+0000 7f9511000640 1 --2- 172.21.15.6:0/1530650598 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f950c15e2d0 0x7f950c1573a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:36.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.327+0000 7f9511000640 1 --2- 172.21.15.6:0/1530650598 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f94e806b4d0 0x7f94e806d990 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:36.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.327+0000 7f9511000640 1 --2- 172.21.15.6:0/1530650598 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f950c06cab0 0x7f950c13e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:36.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.327+0000 7f9511000640 1 -- 172.21.15.6:0/1530650598 >> 172.21.15.6:0/1530650598 conn(0x7f950c09aa10 msgr2=0x7f950c15d600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:32:36.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.327+0000 7f9511000640 1 -- 172.21.15.6:0/1530650598 shutdown_connections 2024-03-04T09:32:36.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.327+0000 7f9511000640 1 -- 172.21.15.6:0/1530650598 wait complete. 2024-03-04T09:32:36.342 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-04T09:32:36.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.564+0000 7f2f58a00640 1 Processor -- start 2024-03-04T09:32:36.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.565+0000 7f2f58a00640 1 -- start start 2024-03-04T09:32:36.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.566+0000 7f2f58a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2f5415e0c0 0x7f2f54072270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:36.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.566+0000 7f2f58a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f2f5406b410 0x7f2f540cdd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:36.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.566+0000 7f2f58a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f540b5800 con 0x7f2f5406afb0 2024-03-04T09:32:36.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.566+0000 7f2f58a00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f5406dd30 con 0x7f2f5406b410 2024-03-04T09:32:36.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.566+0000 7f2f58a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f540b5aa0 con 0x7f2f5415e0c0 2024-03-04T09:32:36.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.567+0000 7f2f52a00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f2f5406afb0 0x7f2f540cd600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:36896/0 (socket says 172.21.15.6:36896) 2024-03-04T09:32:36.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.567+0000 7f2f52a00640 1 -- 172.21.15.6:0/2065120660 learned_addr learned my addr 172.21.15.6:0/2065120660 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:32:36.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.567+0000 7f2f53e00640 1 --2- 172.21.15.6:0/2065120660 >> v2:172.21.15.6:3300/0 conn(0x7f2f5406b410 0x7f2f540cdd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:36.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.567+0000 7f2f53400640 1 --2- 172.21.15.6:0/2065120660 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2f5415e0c0 0x7f2f54072270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:36.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.567+0000 7f2f52000640 1 -- 172.21.15.6:0/2065120660 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2416849497 0 0) 0x7f2f540b5800 con 0x7f2f5406afb0 2024-03-04T09:32:36.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.567+0000 7f2f52000640 1 -- 172.21.15.6:0/2065120660 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f38003520 con 0x7f2f5406afb0 2024-03-04T09:32:36.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.568+0000 7f2f53e00640 1 -- 172.21.15.6:0/2065120660 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2f5415e0c0 msgr2=0x7f2f54072270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:32:36.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.568+0000 7f2f53e00640 1 --2- 172.21.15.6:0/2065120660 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2f5415e0c0 0x7f2f54072270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:36.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.568+0000 7f2f53e00640 1 -- 172.21.15.6:0/2065120660 >> v1:172.21.15.6:6789/0 conn(0x7f2f5406afb0 legacy=0x7f2f540cd600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:36.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.568+0000 7f2f53e00640 1 -- 172.21.15.6:0/2065120660 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f540b5d80 con 0x7f2f5406b410 2024-03-04T09:32:36.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.569+0000 7f2f53e00640 1 --2- 172.21.15.6:0/2065120660 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2f5406b410 0x7f2f540cdd10 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7f2f4c0097a0 tx=0x7f2f4c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c862ec548baafa1 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:36.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.570+0000 7f2f52000640 1 -- 172.21.15.6:0/2065120660 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f4c035020 con 0x7f2f5406b410 2024-03-04T09:32:36.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.570+0000 7f2f52000640 1 -- 172.21.15.6:0/2065120660 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f4c031830 con 0x7f2f5406b410 2024-03-04T09:32:36.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.570+0000 7f2f52000640 1 -- 172.21.15.6:0/2065120660 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f4c031c00 con 0x7f2f5406b410 2024-03-04T09:32:36.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.570+0000 7f2f58a00640 1 -- 172.21.15.6:0/2065120660 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2f5406b410 msgr2=0x7f2f540cdd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:36.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.570+0000 7f2f58a00640 1 --2- 172.21.15.6:0/2065120660 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2f5406b410 0x7f2f540cdd10 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7f2f4c0097a0 tx=0x7f2f4c02b8d0 comp rx=0 tx=0).stop 2024-03-04T09:32:36.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.570+0000 7f2f58a00640 1 -- 172.21.15.6:0/2065120660 shutdown_connections 2024-03-04T09:32:36.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.570+0000 7f2f58a00640 1 --2- 172.21.15.6:0/2065120660 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2f5406b410 0x7f2f540cdd10 unknown :-1 s=CLOSED pgs=1398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:36.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.570+0000 7f2f58a00640 1 --2- 172.21.15.6:0/2065120660 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2f5415e0c0 0x7f2f54072270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:36.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.570+0000 7f2f58a00640 1 -- 172.21.15.6:0/2065120660 >> 172.21.15.6:0/2065120660 conn(0x7f2f540d0c00 msgr2=0x7f2f540d1010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:32:36.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.571+0000 7f2f58a00640 1 -- 172.21.15.6:0/2065120660 shutdown_connections 2024-03-04T09:32:36.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.571+0000 7f2f58a00640 1 -- 172.21.15.6:0/2065120660 wait complete. 2024-03-04T09:32:36.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.572+0000 7f2f58a00640 1 Processor -- start 2024-03-04T09:32:36.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.572+0000 7f2f58a00640 1 -- start start 2024-03-04T09:32:36.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.573+0000 7f2f58a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2f5406afb0 0x7f2f540add50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:36.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.573+0000 7f2f58a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2f5406b410 0x7f2f54160200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:36.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.573+0000 7f2f58a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2f5415e0c0 0x7f2f54148560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:36.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.573+0000 7f2f58a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f5406b880 con 0x7f2f5406afb0 2024-03-04T09:32:36.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.573+0000 7f2f58a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f540cef60 con 0x7f2f5415e0c0 2024-03-04T09:32:36.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.573+0000 7f2f58a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f540727b0 con 0x7f2f5406b410 2024-03-04T09:32:36.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.574+0000 7f2f53400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2f5406afb0 0x7f2f540add50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:36.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.574+0000 7f2f53400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2f5406afb0 0x7f2f540add50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:44732/0 (socket says 172.21.15.6:44732) 2024-03-04T09:32:36.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.574+0000 7f2f53400640 1 -- 172.21.15.6:0/3015289746 learned_addr learned my addr 172.21.15.6:0/3015289746 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:32:36.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.574+0000 7f2f52a00640 1 --2- 172.21.15.6:0/3015289746 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2f5406b410 0x7f2f54160200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:36.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.574+0000 7f2f53400640 1 -- 172.21.15.6:0/3015289746 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2f5406b410 msgr2=0x7f2f54160200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:36.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.574+0000 7f2f53e00640 1 --2- 172.21.15.6:0/3015289746 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2f5415e0c0 0x7f2f54148560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:36.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.574+0000 7f2f53400640 1 --2- 172.21.15.6:0/3015289746 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2f5406b410 0x7f2f54160200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:36.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.574+0000 7f2f53400640 1 -- 172.21.15.6:0/3015289746 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2f5415e0c0 msgr2=0x7f2f54148560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:36.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.574+0000 7f2f53400640 1 --2- 172.21.15.6:0/3015289746 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2f5415e0c0 0x7f2f54148560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:36.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.574+0000 7f2f53400640 1 -- 172.21.15.6:0/3015289746 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f540ae380 con 0x7f2f5406afb0 2024-03-04T09:32:36.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.574+0000 7f2f53e00640 1 --2- 172.21.15.6:0/3015289746 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2f5415e0c0 0x7f2f54148560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:32:36.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.574+0000 7f2f53400640 1 --2- 172.21.15.6:0/3015289746 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2f5406afb0 0x7f2f540add50 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7f2f44009d10 tx=0x7f2f4400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:36.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.575+0000 7f2f4b400640 1 -- 172.21.15.6:0/3015289746 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f44018020 con 0x7f2f5406afb0 2024-03-04T09:32:36.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.575+0000 7f2f4b400640 1 -- 172.21.15.6:0/3015289746 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f44014480 con 0x7f2f5406afb0 2024-03-04T09:32:36.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.575+0000 7f2f4b400640 1 -- 172.21.15.6:0/3015289746 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f44014830 con 0x7f2f5406afb0 2024-03-04T09:32:36.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.575+0000 7f2f58a00640 1 -- 172.21.15.6:0/3015289746 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f540ae660 con 0x7f2f5406afb0 2024-03-04T09:32:36.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.575+0000 7f2f58a00640 1 -- 172.21.15.6:0/3015289746 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f54160740 con 0x7f2f5406afb0 2024-03-04T09:32:36.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.576+0000 7f2f58a00640 1 -- 172.21.15.6:0/3015289746 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f5404f330 con 0x7f2f5406afb0 2024-03-04T09:32:36.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.577+0000 7f2f4b400640 1 -- 172.21.15.6:0/3015289746 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2f44005d10 con 0x7f2f5406afb0 2024-03-04T09:32:36.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.578+0000 7f2f4b400640 1 --2- 172.21.15.6:0/3015289746 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2f3006b4d0 0x7f2f3006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:36.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.578+0000 7f2f4b400640 1 -- 172.21.15.6:0/3015289746 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f2f4408ae10 con 0x7f2f5406afb0 2024-03-04T09:32:36.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.579+0000 7f2f52a00640 1 --2- 172.21.15.6:0/3015289746 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2f3006b4d0 0x7f2f3006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:36.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.580+0000 7f2f52a00640 1 --2- 172.21.15.6:0/3015289746 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2f3006b4d0 0x7f2f3006d990 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f2f540af210 tx=0x7f2f3c009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:36.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.583+0000 7f2f4b400640 1 -- 172.21.15.6:0/3015289746 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2f4408e030 con 0x7f2f5406afb0 2024-03-04T09:32:36.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.788+0000 7f2f58a00640 1 -- 172.21.15.6:0/3015289746 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f2f540b57c0 con 0x7f2f5406afb0 2024-03-04T09:32:36.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.789+0000 7f2f4b400640 1 -- 172.21.15.6:0/3015289746 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v115) v1 ==== 100+0+11 (secure 0 0 0) 0x7f2f44022070 con 0x7f2f5406afb0 2024-03-04T09:32:36.785 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 64 2024-03-04T09:32:36.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.791+0000 7f2f58a00640 1 -- 172.21.15.6:0/3015289746 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2f3006b4d0 msgr2=0x7f2f3006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:36.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.791+0000 7f2f58a00640 1 --2- 172.21.15.6:0/3015289746 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2f3006b4d0 0x7f2f3006d990 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f2f540af210 tx=0x7f2f3c009040 comp rx=0 tx=0).stop 2024-03-04T09:32:36.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.791+0000 7f2f58a00640 1 -- 172.21.15.6:0/3015289746 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2f5406afb0 msgr2=0x7f2f540add50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:36.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.791+0000 7f2f58a00640 1 --2- 172.21.15.6:0/3015289746 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2f5406afb0 0x7f2f540add50 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7f2f44009d10 tx=0x7f2f4400c490 comp rx=0 tx=0).stop 2024-03-04T09:32:36.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.791+0000 7f2f58a00640 1 -- 172.21.15.6:0/3015289746 shutdown_connections 2024-03-04T09:32:36.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.791+0000 7f2f58a00640 1 --2- 172.21.15.6:0/3015289746 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2f5415e0c0 0x7f2f54148560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:36.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.791+0000 7f2f58a00640 1 --2- 172.21.15.6:0/3015289746 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2f3006b4d0 0x7f2f3006d990 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:36.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.791+0000 7f2f58a00640 1 --2- 172.21.15.6:0/3015289746 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2f5406b410 0x7f2f54160200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:36.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.791+0000 7f2f58a00640 1 --2- 172.21.15.6:0/3015289746 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2f5406afb0 0x7f2f540add50 unknown :-1 s=CLOSED pgs=1399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:36.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.791+0000 7f2f58a00640 1 -- 172.21.15.6:0/3015289746 >> 172.21.15.6:0/3015289746 conn(0x7f2f540d0c00 msgr2=0x7f2f540d0fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:32:36.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.792+0000 7f2f58a00640 1 -- 172.21.15.6:0/3015289746 shutdown_connections 2024-03-04T09:32:36.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:36.792+0000 7f2f58a00640 1 -- 172.21.15.6:0/3015289746 wait complete. 2024-03-04T09:32:36.809 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-04T09:32:37.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.014+0000 7f8bc9000640 1 Processor -- start 2024-03-04T09:32:37.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.014+0000 7f8bc9000640 1 -- start start 2024-03-04T09:32:37.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.016+0000 7f8bc9000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f8bc415e1c0 0x7f8bc40b0310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:37.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.016+0000 7f8bc9000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f8bc406f6b0 0x7f8bc40b5b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:37.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.016+0000 7f8bc9000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bc40b3da0 con 0x7f8bc406afb0 2024-03-04T09:32:37.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.016+0000 7f8bc9000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8bc40b6060 con 0x7f8bc415e1c0 2024-03-04T09:32:37.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.016+0000 7f8bc9000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8bc40aff60 con 0x7f8bc406f6b0 2024-03-04T09:32:37.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.016+0000 7f8bc1e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f8bc415e1c0 0x7f8bc40b0310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:37.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.016+0000 7f8bc1e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f8bc415e1c0 0x7f8bc40b0310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:44746/0 (socket says 172.21.15.6:44746) 2024-03-04T09:32:37.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.016+0000 7f8bc1e00640 1 -- 172.21.15.6:0/1602238343 learned_addr learned my addr 172.21.15.6:0/1602238343 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:32:37.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.016+0000 7f8bc1e00640 1 -- 172.21.15.6:0/1602238343 >> v2:172.21.15.146:3300/0 conn(0x7f8bc406f6b0 msgr2=0x7f8bc40b5b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:32:37.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.016+0000 7f8bc1e00640 1 --2- 172.21.15.6:0/1602238343 >> v2:172.21.15.146:3300/0 conn(0x7f8bc406f6b0 0x7f8bc40b5b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:37.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.016+0000 7f8bc1e00640 1 -- 172.21.15.6:0/1602238343 >> v1:172.21.15.146:6789/0 conn(0x7f8bc406afb0 legacy=0x7f8bc406b3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:32:37.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.016+0000 7f8bc1e00640 1 -- 172.21.15.6:0/1602238343 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bc406fb60 con 0x7f8bc415e1c0 2024-03-04T09:32:37.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.018+0000 7f8bc1e00640 1 --2- 172.21.15.6:0/1602238343 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8bc415e1c0 0x7f8bc40b0310 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7f8bc40d1cb0 tx=0x7f8bb402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e30bb7b6a0693948 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:37.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.019+0000 7f8bc1400640 1 -- 172.21.15.6:0/1602238343 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bb4035020 con 0x7f8bc415e1c0 2024-03-04T09:32:37.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.019+0000 7f8bc1400640 1 -- 172.21.15.6:0/1602238343 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bb4031870 con 0x7f8bc415e1c0 2024-03-04T09:32:37.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.019+0000 7f8bc1400640 1 -- 172.21.15.6:0/1602238343 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bb4031c40 con 0x7f8bc415e1c0 2024-03-04T09:32:37.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.019+0000 7f8bc9000640 1 -- 172.21.15.6:0/1602238343 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8bc415e1c0 msgr2=0x7f8bc40b0310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:37.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.019+0000 7f8bc9000640 1 --2- 172.21.15.6:0/1602238343 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8bc415e1c0 0x7f8bc40b0310 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7f8bc40d1cb0 tx=0x7f8bb402b910 comp rx=0 tx=0).stop 2024-03-04T09:32:37.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.019+0000 7f8bc9000640 1 -- 172.21.15.6:0/1602238343 shutdown_connections 2024-03-04T09:32:37.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.019+0000 7f8bc9000640 1 --2- 172.21.15.6:0/1602238343 >> v2:172.21.15.146:3300/0 conn(0x7f8bc406f6b0 0x7f8bc40b5b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:37.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.019+0000 7f8bc9000640 1 --2- 172.21.15.6:0/1602238343 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8bc415e1c0 0x7f8bc40b0310 unknown :-1 s=CLOSED pgs=1400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:37.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.019+0000 7f8bc9000640 1 -- 172.21.15.6:0/1602238343 >> 172.21.15.6:0/1602238343 conn(0x7f8bc40b2a00 msgr2=0x7f8bc40b2e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:32:37.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.019+0000 7f8bc9000640 1 -- 172.21.15.6:0/1602238343 shutdown_connections 2024-03-04T09:32:37.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.020+0000 7f8bc9000640 1 -- 172.21.15.6:0/1602238343 wait complete. 2024-03-04T09:32:37.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.022+0000 7f8bc9000640 1 Processor -- start 2024-03-04T09:32:37.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.022+0000 7f8bc9000640 1 -- start start 2024-03-04T09:32:37.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.023+0000 7f8bc9000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8bc406afb0 0x7f8bc4162010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:37.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.024+0000 7f8bc9000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8bc406f6b0 0x7f8bc41f4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:37.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.024+0000 7f8bc9000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8bc415e1c0 0x7f8bc40add50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:37.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.024+0000 7f8bc9000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bc4072b50 con 0x7f8bc406afb0 2024-03-04T09:32:37.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.024+0000 7f8bc9000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bc4072870 con 0x7f8bc406f6b0 2024-03-04T09:32:37.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.024+0000 7f8bc9000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bc4156420 con 0x7f8bc415e1c0 2024-03-04T09:32:37.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.024+0000 7f8bc1e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8bc406afb0 0x7f8bc4162010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:37.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.024+0000 7f8bc1e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8bc406afb0 0x7f8bc4162010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:44748/0 (socket says 172.21.15.6:44748) 2024-03-04T09:32:37.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.024+0000 7f8bc1e00640 1 -- 172.21.15.6:0/180587525 learned_addr learned my addr 172.21.15.6:0/180587525 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:32:37.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.024+0000 7f8bc1e00640 1 -- 172.21.15.6:0/180587525 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8bc415e1c0 msgr2=0x7f8bc40add50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:37.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.024+0000 7f8bc1e00640 1 --2- 172.21.15.6:0/180587525 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8bc415e1c0 0x7f8bc40add50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:37.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.024+0000 7f8bc1e00640 1 -- 172.21.15.6:0/180587525 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8bc406f6b0 msgr2=0x7f8bc41f4580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:37.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.024+0000 7f8bc1e00640 1 --2- 172.21.15.6:0/180587525 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8bc406f6b0 0x7f8bc41f4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:37.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.024+0000 7f8bc1e00640 1 -- 172.21.15.6:0/180587525 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bc41f4ac0 con 0x7f8bc406afb0 2024-03-04T09:32:37.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.025+0000 7f8bc1e00640 1 --2- 172.21.15.6:0/180587525 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8bc406afb0 0x7f8bc4162010 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7f8bb4000c00 tx=0x7f8bb40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:37.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.025+0000 7f8bba000640 1 -- 172.21.15.6:0/180587525 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bb403ad60 con 0x7f8bc406afb0 2024-03-04T09:32:37.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.025+0000 7f8bba000640 1 -- 172.21.15.6:0/180587525 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bb40443f0 con 0x7f8bc406afb0 2024-03-04T09:32:37.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.025+0000 7f8bc9000640 1 -- 172.21.15.6:0/180587525 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bc4155550 con 0x7f8bc406afb0 2024-03-04T09:32:37.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.025+0000 7f8bba000640 1 -- 172.21.15.6:0/180587525 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bb40447e0 con 0x7f8bc406afb0 2024-03-04T09:32:37.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.025+0000 7f8bc9000640 1 -- 172.21.15.6:0/180587525 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bc4155af0 con 0x7f8bc406afb0 2024-03-04T09:32:37.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.027+0000 7f8bc9000640 1 -- 172.21.15.6:0/180587525 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bc404f3a0 con 0x7f8bc406afb0 2024-03-04T09:32:37.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.027+0000 7f8bba000640 1 -- 172.21.15.6:0/180587525 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8bb404b060 con 0x7f8bc406afb0 2024-03-04T09:32:37.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.029+0000 7f8bba000640 1 --2- 172.21.15.6:0/180587525 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f8ba006b4d0 0x7f8ba006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:37.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.029+0000 7f8bba000640 1 -- 172.21.15.6:0/180587525 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f8bb40ac4a0 con 0x7f8bc406afb0 2024-03-04T09:32:37.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.029+0000 7f8bbbe00640 1 --2- 172.21.15.6:0/180587525 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f8ba006b4d0 0x7f8ba006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:37.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.031+0000 7f8bbbe00640 1 --2- 172.21.15.6:0/180587525 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f8ba006b4d0 0x7f8ba006d990 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f8bc4158770 tx=0x7f8bac006c90 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:37.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.032+0000 7f8bba000640 1 -- 172.21.15.6:0/180587525 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8bc41f4ac0 con 0x7f8bc406afb0 2024-03-04T09:32:37.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.315+0000 7f8bc9000640 1 -- 172.21.15.6:0/180587525 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f8bc41530b0 con 0x7f8bc406afb0 2024-03-04T09:32:37.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.316+0000 7f8bba000640 1 -- 172.21.15.6:0/180587525 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v115) v1 ==== 80+0+120 (secure 0 0 0) 0x7f8bb403f070 con 0x7f8bc406afb0 2024-03-04T09:32:37.313 INFO:teuthology.orchestra.run.smithi006.stdout:172.21.15.6:6834/2697931210 2024-03-05T09:31:36.737386+0000 2024-03-04T09:32:37.313 INFO:teuthology.orchestra.run.smithi006.stdout:172.21.15.6:6832/2697931210 2024-03-05T09:31:36.737386+0000 2024-03-04T09:32:37.313 INFO:teuthology.orchestra.run.smithi006.stderr:listed 2 entries 2024-03-04T09:32:37.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.319+0000 7f8bc9000640 1 -- 172.21.15.6:0/180587525 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f8ba006b4d0 msgr2=0x7f8ba006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:37.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.319+0000 7f8bc9000640 1 --2- 172.21.15.6:0/180587525 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f8ba006b4d0 0x7f8ba006d990 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f8bc4158770 tx=0x7f8bac006c90 comp rx=0 tx=0).stop 2024-03-04T09:32:37.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.319+0000 7f8bc9000640 1 -- 172.21.15.6:0/180587525 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8bc406afb0 msgr2=0x7f8bc4162010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:37.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.319+0000 7f8bc9000640 1 --2- 172.21.15.6:0/180587525 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8bc406afb0 0x7f8bc4162010 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7f8bb4000c00 tx=0x7f8bb40046c0 comp rx=0 tx=0).stop 2024-03-04T09:32:37.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.319+0000 7f8bc9000640 1 -- 172.21.15.6:0/180587525 shutdown_connections 2024-03-04T09:32:37.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.319+0000 7f8bc9000640 1 --2- 172.21.15.6:0/180587525 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f8ba006b4d0 0x7f8ba006d990 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:37.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.319+0000 7f8bc9000640 1 --2- 172.21.15.6:0/180587525 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8bc415e1c0 0x7f8bc40add50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:37.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.319+0000 7f8bc9000640 1 --2- 172.21.15.6:0/180587525 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8bc406f6b0 0x7f8bc41f4580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:37.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.319+0000 7f8bc9000640 1 --2- 172.21.15.6:0/180587525 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8bc406afb0 0x7f8bc4162010 unknown :-1 s=CLOSED pgs=1401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:37.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.319+0000 7f8bc9000640 1 -- 172.21.15.6:0/180587525 >> 172.21.15.6:0/180587525 conn(0x7f8bc40b2a00 msgr2=0x7f8bc40b0970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:32:37.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.319+0000 7f8bc9000640 1 -- 172.21.15.6:0/180587525 shutdown_connections 2024-03-04T09:32:37.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.319+0000 7f8bc9000640 1 -- 172.21.15.6:0/180587525 wait complete. 2024-03-04T09:32:37.338 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-04T09:32:37.338 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:32:37.338 DEBUG:teuthology.orchestra.run.smithi006:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-04T09:32:37.436 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi006.front.sepia.ceph.com 2024-03-04T09:32:37.436 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:32:37.437 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-04T09:32:37.485 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-04T09:32:37.485 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T09:32:37.541 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053429,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249180k,nr_inodes=812295,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-04T09:32:37.541 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-04T09:32:37.541 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T09:32:37.594 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054202,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249800k,nr_inodes=812450,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-04T09:32:37.595 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-04T09:32:37.595 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T09:32:37.651 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053429,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249180k,nr_inodes=812295,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-04T09:32:37.651 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:32:37.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.886+0000 7f9d7b200640 1 Processor -- start 2024-03-04T09:32:37.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.886+0000 7f9d7b200640 1 -- start start 2024-03-04T09:32:37.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.888+0000 7f9d7b200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9d74145300 0x7f9d740b49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:37.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.888+0000 7f9d7b200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f9d740cd060 0x7f9d7415c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:37.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.888+0000 7f9d7b200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d740714d0 con 0x7f9d74070f10 2024-03-04T09:32:37.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.888+0000 7f9d7b200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d740717b0 con 0x7f9d740cd060 2024-03-04T09:32:37.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.888+0000 7f9d7b200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d74071a90 con 0x7f9d74145300 2024-03-04T09:32:37.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.888+0000 7f9d79400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f9d74070f10 0x7f9d740cc950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:36912/0 (socket says 172.21.15.6:36912) 2024-03-04T09:32:37.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.888+0000 7f9d79400640 1 -- 172.21.15.6:0/796354067 learned_addr learned my addr 172.21.15.6:0/796354067 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:32:37.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.889+0000 7f9d79e00640 1 --2- 172.21.15.6:0/796354067 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9d74145300 0x7f9d740b49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:37.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.889+0000 7f9d7a800640 1 --2- 172.21.15.6:0/796354067 >> v2:172.21.15.146:3300/0 conn(0x7f9d740cd060 0x7f9d7415c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:37.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.889+0000 7f9d78a00640 1 -- 172.21.15.6:0/796354067 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2073195756 0 0) 0x7f9d740714d0 con 0x7f9d74070f10 2024-03-04T09:32:37.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.889+0000 7f9d78a00640 1 -- 172.21.15.6:0/796354067 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d5c003520 con 0x7f9d74070f10 2024-03-04T09:32:37.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.889+0000 7f9d78a00640 1 -- 172.21.15.6:0/796354067 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4234982854 0 0) 0x7f9d5c003520 con 0x7f9d74070f10 2024-03-04T09:32:37.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.889+0000 7f9d78a00640 1 -- 172.21.15.6:0/796354067 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9d74145300 msgr2=0x7f9d740b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:32:37.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.889+0000 7f9d78a00640 1 --2- 172.21.15.6:0/796354067 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9d74145300 0x7f9d740b49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:37.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.889+0000 7f9d78a00640 1 -- 172.21.15.6:0/796354067 >> v2:172.21.15.146:3300/0 conn(0x7f9d740cd060 msgr2=0x7f9d7415c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:32:37.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.889+0000 7f9d78a00640 1 --2- 172.21.15.6:0/796354067 >> v2:172.21.15.146:3300/0 conn(0x7f9d740cd060 0x7f9d7415c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:37.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.889+0000 7f9d78a00640 1 -- 172.21.15.6:0/796354067 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d74071d70 con 0x7f9d74070f10 2024-03-04T09:32:37.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.889+0000 7f9d78a00640 1 -- 172.21.15.6:0/796354067 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7f9d64001670 con 0x7f9d74070f10 2024-03-04T09:32:37.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.889+0000 7f9d78a00640 1 -- 172.21.15.6:0/796354067 >> v1:172.21.15.6:6789/0 conn(0x7f9d74070f10 legacy=0x7f9d740cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:37.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.889+0000 7f9d78a00640 1 --2- 172.21.15.6:0/796354067 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9d74145300 0x7f9d5c0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:37.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.890+0000 7f9d78a00640 1 --2- 172.21.15.6:0/796354067 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9d5c003f80 0x7f9d5c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:37.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.890+0000 7f9d78a00640 1 --2- 172.21.15.6:0/796354067 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9d5c004d40 0x7f9d5c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:37.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.890+0000 7f9d78a00640 1 -- 172.21.15.6:0/796354067 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d740717b0 con 0x7f9d5c003f80 2024-03-04T09:32:37.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.890+0000 7f9d78a00640 1 -- 172.21.15.6:0/796354067 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d74071a90 con 0x7f9d5c004d40 2024-03-04T09:32:37.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.890+0000 7f9d78a00640 1 -- 172.21.15.6:0/796354067 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d5c00b9c0 con 0x7f9d74145300 2024-03-04T09:32:37.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.890+0000 7f9d79400640 1 --2- 172.21.15.6:0/796354067 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9d5c003f80 0x7f9d5c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:37.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.890+0000 7f9d79e00640 1 --2- 172.21.15.6:0/796354067 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9d74145300 0x7f9d5c0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:37.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.890+0000 7f9d7a800640 1 --2- 172.21.15.6:0/796354067 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9d5c004d40 0x7f9d5c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:37.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.890+0000 7f9d79400640 1 -- 172.21.15.6:0/796354067 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9d74145300 msgr2=0x7f9d5c0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:32:37.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.890+0000 7f9d79400640 1 --2- 172.21.15.6:0/796354067 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9d74145300 0x7f9d5c0039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:37.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.890+0000 7f9d79400640 1 -- 172.21.15.6:0/796354067 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9d5c004d40 msgr2=0x7f9d5c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:32:37.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.890+0000 7f9d79400640 1 --2- 172.21.15.6:0/796354067 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9d5c004d40 0x7f9d5c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:37.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.890+0000 7f9d79400640 1 -- 172.21.15.6:0/796354067 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d5c003520 con 0x7f9d5c003f80 2024-03-04T09:32:37.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.891+0000 7f9d79400640 1 --2- 172.21.15.6:0/796354067 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9d5c003f80 0x7f9d5c004800 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7f9d6400c190 tx=0x7f9d5c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1fc27289af062ad server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:37.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.892+0000 7f9d78a00640 1 -- 172.21.15.6:0/796354067 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d64007c80 con 0x7f9d5c003f80 2024-03-04T09:32:37.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.892+0000 7f9d78a00640 1 -- 172.21.15.6:0/796354067 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d64030070 con 0x7f9d5c003f80 2024-03-04T09:32:37.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.892+0000 7f9d78a00640 1 -- 172.21.15.6:0/796354067 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d64005d70 con 0x7f9d5c003f80 2024-03-04T09:32:37.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.892+0000 7f9d7b200640 1 -- 172.21.15.6:0/796354067 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9d5c003f80 msgr2=0x7f9d5c004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:37.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.892+0000 7f9d7b200640 1 --2- 172.21.15.6:0/796354067 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9d5c003f80 0x7f9d5c004800 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7f9d6400c190 tx=0x7f9d5c005830 comp rx=0 tx=0).stop 2024-03-04T09:32:37.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.892+0000 7f9d7a800640 1 -- 172.21.15.6:0/796354067 reap_dead start 2024-03-04T09:32:37.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.892+0000 7f9d7b200640 1 -- 172.21.15.6:0/796354067 shutdown_connections 2024-03-04T09:32:37.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.892+0000 7f9d7b200640 1 -- 172.21.15.6:0/796354067 >> 172.21.15.6:0/796354067 conn(0x7f9d7406b3d0 msgr2=0x7f9d7406b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:32:37.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.893+0000 7f9d7b200640 1 -- 172.21.15.6:0/796354067 shutdown_connections 2024-03-04T09:32:37.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.893+0000 7f9d7b200640 1 -- 172.21.15.6:0/796354067 wait complete. 2024-03-04T09:32:37.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.894+0000 7f9d7b200640 1 Processor -- start 2024-03-04T09:32:37.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.895+0000 7f9d7b200640 1 -- start start 2024-03-04T09:32:37.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.896+0000 7f9d7b200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9d5c003f80 0x7f9d74150380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:37.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.896+0000 7f9d7b200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9d7415c470 0x7f9d7415ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:37.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.896+0000 7f9d7b200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9d74140960 0x7f9d74140de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:37.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.896+0000 7f9d7b200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d740b4b20 con 0x7f9d5c003f80 2024-03-04T09:32:37.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.896+0000 7f9d7b200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d740b49a0 con 0x7f9d7415c470 2024-03-04T09:32:37.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.896+0000 7f9d7b200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d740b4ca0 con 0x7f9d74140960 2024-03-04T09:32:37.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.897+0000 7f9d79e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9d5c003f80 0x7f9d74150380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:37.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.897+0000 7f9d79e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9d5c003f80 0x7f9d74150380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:44770/0 (socket says 172.21.15.6:44770) 2024-03-04T09:32:37.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.897+0000 7f9d79e00640 1 -- 172.21.15.6:0/188168581 learned_addr learned my addr 172.21.15.6:0/188168581 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:32:37.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.897+0000 7f9d7a800640 1 --2- 172.21.15.6:0/188168581 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9d74140960 0x7f9d74140de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:37.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.897+0000 7f9d79400640 1 --2- 172.21.15.6:0/188168581 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9d7415c470 0x7f9d7415ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:37.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.897+0000 7f9d79e00640 1 -- 172.21.15.6:0/188168581 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9d74140960 msgr2=0x7f9d74140de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:37.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.897+0000 7f9d79e00640 1 --2- 172.21.15.6:0/188168581 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9d74140960 0x7f9d74140de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:37.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.897+0000 7f9d79e00640 1 -- 172.21.15.6:0/188168581 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9d7415c470 msgr2=0x7f9d7415ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:37.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.897+0000 7f9d79e00640 1 --2- 172.21.15.6:0/188168581 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9d7415c470 0x7f9d7415ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:37.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.897+0000 7f9d79e00640 1 -- 172.21.15.6:0/188168581 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d7415e2f0 con 0x7f9d5c003f80 2024-03-04T09:32:37.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.897+0000 7f9d7a800640 1 --2- 172.21.15.6:0/188168581 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9d74140960 0x7f9d74140de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:32:37.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.897+0000 7f9d79400640 1 --2- 172.21.15.6:0/188168581 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9d7415c470 0x7f9d7415ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:32:37.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.897+0000 7f9d79e00640 1 --2- 172.21.15.6:0/188168581 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9d5c003f80 0x7f9d74150380 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7f9d680119d0 tx=0x7f9d68011e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:37.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.898+0000 7f9d6e000640 1 -- 172.21.15.6:0/188168581 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d6800fb50 con 0x7f9d5c003f80 2024-03-04T09:32:37.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.898+0000 7f9d6e000640 1 -- 172.21.15.6:0/188168581 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d6800fcf0 con 0x7f9d5c003f80 2024-03-04T09:32:37.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.898+0000 7f9d7b200640 1 -- 172.21.15.6:0/188168581 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d7415cae0 con 0x7f9d5c003f80 2024-03-04T09:32:37.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.898+0000 7f9d6e000640 1 -- 172.21.15.6:0/188168581 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d6800fea0 con 0x7f9d5c003f80 2024-03-04T09:32:37.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.898+0000 7f9d7b200640 1 -- 172.21.15.6:0/188168581 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d7415d0a0 con 0x7f9d5c003f80 2024-03-04T09:32:37.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.899+0000 7f9d7b200640 1 -- 172.21.15.6:0/188168581 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d74141320 con 0x7f9d5c003f80 2024-03-04T09:32:37.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.900+0000 7f9d6e000640 1 -- 172.21.15.6:0/188168581 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9d68012040 con 0x7f9d5c003f80 2024-03-04T09:32:37.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.901+0000 7f9d6e000640 1 --2- 172.21.15.6:0/188168581 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f9d5806b470 0x7f9d5806d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:37.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.901+0000 7f9d6e000640 1 -- 172.21.15.6:0/188168581 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f9d6808a910 con 0x7f9d5c003f80 2024-03-04T09:32:37.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.902+0000 7f9d79400640 1 --2- 172.21.15.6:0/188168581 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f9d5806b470 0x7f9d5806d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:37.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.903+0000 7f9d79400640 1 --2- 172.21.15.6:0/188168581 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f9d5806b470 0x7f9d5806d930 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f9d64034000 tx=0x7f9d64007000 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:37.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:37.906+0000 7f9d6e000640 1 -- 172.21.15.6:0/188168581 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9d68019030 con 0x7f9d5c003f80 2024-03-04T09:32:38.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.149+0000 7f9d7b200640 1 -- 172.21.15.6:0/188168581 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9d7404f330 con 0x7f9d5c003f80 2024-03-04T09:32:38.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.150+0000 7f9d6e000640 1 -- 172.21.15.6:0/188168581 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+4975 (secure 0 0 0) 0x7f9d68059700 con 0x7f9d5c003f80 2024-03-04T09:32:38.147 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:32:38.147 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":84,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6535,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/3883122012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":3883122012},{"type":"v1","addr":"172.21.15.146:6835","nonce":3883122012}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":6559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/4060411493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":4060411493},{"type":"v1","addr":"172.21.15.146:6837","nonce":4060411493}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":82},{"gid":7812,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/679471398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":679471398},{"type":"v1","addr":"172.21.15.6:6833","nonce":679471398}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:31:21.114351+0000","modified":"2024-03-04T09:31:40.781474+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7815},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7815":{"gid":7815,"name":"a","rank":0,"incarnation":81,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/882667161","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":882667161},{"type":"v1","addr":"172.21.15.6:6835","nonce":882667161}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7815,"qdb_cluster":[7815]},"id":8}]} 2024-03-04T09:32:38.147 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 84 2024-03-04T09:32:38.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.153+0000 7f9d7b200640 1 -- 172.21.15.6:0/188168581 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f9d5806b470 msgr2=0x7f9d5806d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:38.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.153+0000 7f9d7b200640 1 --2- 172.21.15.6:0/188168581 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f9d5806b470 0x7f9d5806d930 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f9d64034000 tx=0x7f9d64007000 comp rx=0 tx=0).stop 2024-03-04T09:32:38.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.153+0000 7f9d7b200640 1 -- 172.21.15.6:0/188168581 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9d5c003f80 msgr2=0x7f9d74150380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:38.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.153+0000 7f9d7b200640 1 --2- 172.21.15.6:0/188168581 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9d5c003f80 0x7f9d74150380 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7f9d680119d0 tx=0x7f9d68011e90 comp rx=0 tx=0).stop 2024-03-04T09:32:38.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.153+0000 7f9d7b200640 1 -- 172.21.15.6:0/188168581 shutdown_connections 2024-03-04T09:32:38.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.153+0000 7f9d7b200640 1 --2- 172.21.15.6:0/188168581 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f9d5806b470 0x7f9d5806d930 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:38.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.153+0000 7f9d7b200640 1 --2- 172.21.15.6:0/188168581 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9d74140960 0x7f9d74140de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:38.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.153+0000 7f9d7b200640 1 --2- 172.21.15.6:0/188168581 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9d7415c470 0x7f9d7415ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:38.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.153+0000 7f9d7b200640 1 --2- 172.21.15.6:0/188168581 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9d5c003f80 0x7f9d74150380 unknown :-1 s=CLOSED pgs=1404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:38.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.153+0000 7f9d7b200640 1 -- 172.21.15.6:0/188168581 >> 172.21.15.6:0/188168581 conn(0x7f9d7406b3d0 msgr2=0x7f9d7406d2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:32:38.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.153+0000 7f9d7b200640 1 -- 172.21.15.6:0/188168581 shutdown_connections 2024-03-04T09:32:38.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.153+0000 7f9d7b200640 1 -- 172.21.15.6:0/188168581 wait complete. 2024-03-04T09:32:38.171 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T09:32:38.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.392+0000 7fb7ca600640 1 Processor -- start 2024-03-04T09:32:38.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.392+0000 7fb7ca600640 1 -- start start 2024-03-04T09:32:38.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.393+0000 7fb7ca600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb7c415bfb0 0x7fb7c40ae900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:38.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.393+0000 7fb7ca600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fb7c415cda0 0x7fb7c40b4830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:38.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.393+0000 7fb7ca600640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7c415d230 con 0x7fb7c406afb0 2024-03-04T09:32:38.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.393+0000 7fb7ca600640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7c40b3de0 con 0x7fb7c415cda0 2024-03-04T09:32:38.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.393+0000 7fb7ca600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7c40af000 con 0x7fb7c415bfb0 2024-03-04T09:32:38.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.394+0000 7fb7c3e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fb7c415cda0 0x7fb7c40b4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:38.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.394+0000 7fb7c3e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fb7c415cda0 0x7fb7c40b4830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:44782/0 (socket says 172.21.15.6:44782) 2024-03-04T09:32:38.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.394+0000 7fb7c3e00640 1 -- 172.21.15.6:0/1688040440 learned_addr learned my addr 172.21.15.6:0/1688040440 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:32:38.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.394+0000 7fb7baa00640 1 --2- 172.21.15.6:0/1688040440 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb7c415bfb0 0x7fb7c40ae900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:38.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.394+0000 7fb7c3e00640 1 -- 172.21.15.6:0/1688040440 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb7c415bfb0 msgr2=0x7fb7c40ae900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:32:38.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.394+0000 7fb7c3e00640 1 --2- 172.21.15.6:0/1688040440 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb7c415bfb0 0x7fb7c40ae900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:38.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.394+0000 7fb7c3e00640 1 -- 172.21.15.6:0/1688040440 >> v1:172.21.15.146:6789/0 conn(0x7fb7c406afb0 legacy=0x7fb7c406b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:38.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.394+0000 7fb7c3e00640 1 -- 172.21.15.6:0/1688040440 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7c40af2e0 con 0x7fb7c415cda0 2024-03-04T09:32:38.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.395+0000 7fb7c3e00640 1 --2- 172.21.15.6:0/1688040440 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb7c415cda0 0x7fb7c40b4830 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7fb7bc009870 tx=0x7fb7bc02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13169394120bba03 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:38.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.396+0000 7fb7c2a00640 1 -- 172.21.15.6:0/1688040440 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7bc034070 con 0x7fb7c415cda0 2024-03-04T09:32:38.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.396+0000 7fb7c2a00640 1 -- 172.21.15.6:0/1688040440 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7bc030840 con 0x7fb7c415cda0 2024-03-04T09:32:38.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.396+0000 7fb7c2a00640 1 -- 172.21.15.6:0/1688040440 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7bc030c10 con 0x7fb7c415cda0 2024-03-04T09:32:38.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.396+0000 7fb7ca600640 1 -- 172.21.15.6:0/1688040440 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb7c415cda0 msgr2=0x7fb7c40b4830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:38.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.396+0000 7fb7ca600640 1 --2- 172.21.15.6:0/1688040440 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb7c415cda0 0x7fb7c40b4830 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7fb7bc009870 tx=0x7fb7bc02b980 comp rx=0 tx=0).stop 2024-03-04T09:32:38.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.396+0000 7fb7ca600640 1 -- 172.21.15.6:0/1688040440 shutdown_connections 2024-03-04T09:32:38.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.396+0000 7fb7ca600640 1 --2- 172.21.15.6:0/1688040440 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb7c415cda0 0x7fb7c40b4830 unknown :-1 s=CLOSED pgs=1405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:38.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.396+0000 7fb7ca600640 1 --2- 172.21.15.6:0/1688040440 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb7c415bfb0 0x7fb7c40ae900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:38.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.396+0000 7fb7ca600640 1 -- 172.21.15.6:0/1688040440 >> 172.21.15.6:0/1688040440 conn(0x7fb7c40b2ae0 msgr2=0x7fb7c40b2ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:32:38.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.397+0000 7fb7ca600640 1 -- 172.21.15.6:0/1688040440 shutdown_connections 2024-03-04T09:32:38.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.397+0000 7fb7ca600640 1 -- 172.21.15.6:0/1688040440 wait complete. 2024-03-04T09:32:38.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.398+0000 7fb7ca600640 1 Processor -- start 2024-03-04T09:32:38.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.398+0000 7fb7ca600640 1 -- start start 2024-03-04T09:32:38.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.399+0000 7fb7ca600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb7c406afb0 0x7fb7c413a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:38.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.399+0000 7fb7ca600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb7c415bfb0 0x7fb7c415e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:38.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.399+0000 7fb7ca600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb7c415cda0 0x7fb7c415e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:38.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.399+0000 7fb7ca600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7c40b4d70 con 0x7fb7c415cda0 2024-03-04T09:32:38.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.399+0000 7fb7ca600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7c40b4310 con 0x7fb7c406afb0 2024-03-04T09:32:38.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.399+0000 7fb7ca600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7c40b4ef0 con 0x7fb7c415bfb0 2024-03-04T09:32:38.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.400+0000 7fb7c3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb7c415cda0 0x7fb7c415e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:38.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.400+0000 7fb7c3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb7c415cda0 0x7fb7c415e9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:44792/0 (socket says 172.21.15.6:44792) 2024-03-04T09:32:38.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.400+0000 7fb7c3e00640 1 -- 172.21.15.6:0/1703573266 learned_addr learned my addr 172.21.15.6:0/1703573266 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:32:38.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.400+0000 7fb7c3400640 1 --2- 172.21.15.6:0/1703573266 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb7c406afb0 0x7fb7c413a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:38.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.400+0000 7fb7baa00640 1 --2- 172.21.15.6:0/1703573266 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb7c415bfb0 0x7fb7c415e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:38.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.400+0000 7fb7c3e00640 1 -- 172.21.15.6:0/1703573266 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb7c415bfb0 msgr2=0x7fb7c415e4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:38.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.400+0000 7fb7c3e00640 1 --2- 172.21.15.6:0/1703573266 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb7c415bfb0 0x7fb7c415e4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:38.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.400+0000 7fb7c3e00640 1 -- 172.21.15.6:0/1703573266 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb7c406afb0 msgr2=0x7fb7c413a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:38.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.400+0000 7fb7c3e00640 1 --2- 172.21.15.6:0/1703573266 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb7c406afb0 0x7fb7c413a450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:38.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.400+0000 7fb7c3e00640 1 -- 172.21.15.6:0/1703573266 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7c40cd8b0 con 0x7fb7c415cda0 2024-03-04T09:32:38.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.400+0000 7fb7c3400640 1 --2- 172.21.15.6:0/1703573266 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb7c406afb0 0x7fb7c413a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T09:32:38.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.401+0000 7fb7c3e00640 1 --2- 172.21.15.6:0/1703573266 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb7c415cda0 0x7fb7c415e9e0 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7fb7bc009870 tx=0x7fb7bc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:38.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.401+0000 7fb7bbe00640 1 -- 172.21.15.6:0/1703573266 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7bc034040 con 0x7fb7c415cda0 2024-03-04T09:32:38.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.401+0000 7fb7bbe00640 1 -- 172.21.15.6:0/1703573266 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7bc039d90 con 0x7fb7c415cda0 2024-03-04T09:32:38.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.401+0000 7fb7bbe00640 1 -- 172.21.15.6:0/1703573266 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7bc043610 con 0x7fb7c415cda0 2024-03-04T09:32:38.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.401+0000 7fb7ca600640 1 -- 172.21.15.6:0/1703573266 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7c40cdb90 con 0x7fb7c415cda0 2024-03-04T09:32:38.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.401+0000 7fb7ca600640 1 -- 172.21.15.6:0/1703573266 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7c40ce150 con 0x7fb7c415cda0 2024-03-04T09:32:38.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.402+0000 7fb7ca600640 1 -- 172.21.15.6:0/1703573266 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7c404f3a0 con 0x7fb7c415cda0 2024-03-04T09:32:38.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.403+0000 7fb7bbe00640 1 -- 172.21.15.6:0/1703573266 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb7bc04a060 con 0x7fb7c415cda0 2024-03-04T09:32:38.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.404+0000 7fb7bbe00640 1 --2- 172.21.15.6:0/1703573266 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb7a006b500 0x7fb7a006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:38.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.404+0000 7fb7bbe00640 1 -- 172.21.15.6:0/1703573266 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5853+0+0 (secure 0 0 0) 0x7fb7bc0ac1e0 con 0x7fb7c415cda0 2024-03-04T09:32:38.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.405+0000 7fb7c3400640 1 --2- 172.21.15.6:0/1703573266 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb7a006b500 0x7fb7a006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:38.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.406+0000 7fb7c3400640 1 --2- 172.21.15.6:0/1703573266 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb7a006b500 0x7fb7a006d9c0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fb7b4001e00 tx=0x7fb7b4009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:38.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.409+0000 7fb7bbe00640 1 -- 172.21.15.6:0/1703573266 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb7bc03e070 con 0x7fb7c415cda0 2024-03-04T09:32:38.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.610+0000 7fb7ca600640 1 -- 172.21.15.6:0/1703573266 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb7c406b650 con 0x7fb7c415cda0 2024-03-04T09:32:38.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.612+0000 7fb7bbe00640 1 -- 172.21.15.6:0/1703573266 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v115) v1 ==== 74+0+16859 (secure 0 0 0) 0x7fb7bc07b060 con 0x7fb7c415cda0 2024-03-04T09:32:38.609 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:32:38.609 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":115,"fsid":"edce0578-5ae4-4402-b45a-552bdef84d7c","created":"2024-03-04T09:24:54.523780+0000","modified":"2024-03-04T09:32:03.117903+0000","last_up_change":"2024-03-04T09:25:07.089049+0000","last_in_change":"2024-03-04T09:24:57.982077+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T09:25:07.935198+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-04T09:31:19.018233+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-04T09:31:20.131544+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a96574f-81f1-4bc6-8463-757520b84a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6825","nonce":1364449401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6827","nonce":1364449401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6831","nonce":1364449401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6829","nonce":1364449401}]},"public_addr":"172.21.15.6:6825/1364449401","cluster_addr":"172.21.15.6:6827/1364449401","heartbeat_back_addr":"172.21.15.6:6831/1364449401","heartbeat_front_addr":"172.21.15.6:6829/1364449401","state":["exists","up"]},{"osd":1,"uuid":"7d79b9c1-6b87-476a-bc36-7db5ee64cd90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6809","nonce":2494103078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6811","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6813","nonce":2494103078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6821","nonce":2494103078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6817","nonce":2494103078}]},"public_addr":"172.21.15.6:6809/2494103078","cluster_addr":"172.21.15.6:6813/2494103078","heartbeat_back_addr":"172.21.15.6:6821/2494103078","heartbeat_front_addr":"172.21.15.6:6817/2494103078","state":["exists","up"]},{"osd":2,"uuid":"492331b9-4ee9-40b6-bacd-85d44cd6c45a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6812","nonce":1500066681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6816","nonce":1500066681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6824","nonce":1500066681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6820","nonce":1500066681}]},"public_addr":"172.21.15.6:6812/1500066681","cluster_addr":"172.21.15.6:6816/1500066681","heartbeat_back_addr":"172.21.15.6:6824/1500066681","heartbeat_front_addr":"172.21.15.6:6820/1500066681","state":["exists","up"]},{"osd":3,"uuid":"21bb2efe-9ba7-4bdc-bc31-180cd1102e21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6801","nonce":1558806123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6803","nonce":1558806123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6807","nonce":1558806123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6805","nonce":1558806123}]},"public_addr":"172.21.15.6:6801/1558806123","cluster_addr":"172.21.15.6:6803/1558806123","heartbeat_back_addr":"172.21.15.6:6807/1558806123","heartbeat_front_addr":"172.21.15.6:6805/1558806123","state":["exists","up"]},{"osd":4,"uuid":"6e2c4b71-19d9-4737-83d9-558705d8644a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6801","nonce":1526135780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6803","nonce":1526135780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6807","nonce":1526135780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6805","nonce":1526135780}]},"public_addr":"172.21.15.146:6801/1526135780","cluster_addr":"172.21.15.146:6803/1526135780","heartbeat_back_addr":"172.21.15.146:6807/1526135780","heartbeat_front_addr":"172.21.15.146:6805/1526135780","state":["exists","up"]},{"osd":5,"uuid":"fb69c525-f1be-4b7d-ab32-6c29b0d6fddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6809","nonce":460262507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6811","nonce":460262507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6815","nonce":460262507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6813","nonce":460262507}]},"public_addr":"172.21.15.146:6809/460262507","cluster_addr":"172.21.15.146:6811/460262507","heartbeat_back_addr":"172.21.15.146:6815/460262507","heartbeat_front_addr":"172.21.15.146:6813/460262507","state":["exists","up"]},{"osd":6,"uuid":"c32e7fed-370d-4401-8ce0-6a4335b0d168","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6823","nonce":1392897493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6827","nonce":1392897493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6831","nonce":1392897493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6829","nonce":1392897493}]},"public_addr":"172.21.15.146:6823/1392897493","cluster_addr":"172.21.15.146:6827/1392897493","heartbeat_back_addr":"172.21.15.146:6831/1392897493","heartbeat_front_addr":"172.21.15.146:6829/1392897493","state":["exists","up"]},{"osd":7,"uuid":"75a79b44-fa5f-426d-a7d3-71f21f78835c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6817","nonce":3268323156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6819","nonce":3268323156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6825","nonce":3268323156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6822","nonce":3268323156}]},"public_addr":"172.21.15.146:6817/3268323156","cluster_addr":"172.21.15.146:6819/3268323156","heartbeat_back_addr":"172.21.15.146:6825/3268323156","heartbeat_front_addr":"172.21.15.146:6822/3268323156","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.310437+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.295850+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.294674+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.526855+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.300023+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.595160+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.346046+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.558741+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"17.e","mappings":[{"from":0,"to":5}]},{"pgid":"17.26","mappings":[{"from":0,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.6:6834/2697931210":"2024-03-05T09:31:36.737386+0000","172.21.15.6:6832/2697931210":"2024-03-05T09:31:36.737386+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T09:32:38.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.614+0000 7fb7ca600640 1 -- 172.21.15.6:0/1703573266 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb7a006b500 msgr2=0x7fb7a006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:38.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.614+0000 7fb7ca600640 1 --2- 172.21.15.6:0/1703573266 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb7a006b500 0x7fb7a006d9c0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fb7b4001e00 tx=0x7fb7b4009040 comp rx=0 tx=0).stop 2024-03-04T09:32:38.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.614+0000 7fb7ca600640 1 -- 172.21.15.6:0/1703573266 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb7c415cda0 msgr2=0x7fb7c415e9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:38.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.614+0000 7fb7ca600640 1 --2- 172.21.15.6:0/1703573266 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb7c415cda0 0x7fb7c415e9e0 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7fb7bc009870 tx=0x7fb7bc0046c0 comp rx=0 tx=0).stop 2024-03-04T09:32:38.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.615+0000 7fb7ca600640 1 -- 172.21.15.6:0/1703573266 shutdown_connections 2024-03-04T09:32:38.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.615+0000 7fb7ca600640 1 --2- 172.21.15.6:0/1703573266 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb7c415cda0 0x7fb7c415e9e0 unknown :-1 s=CLOSED pgs=1406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:38.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.615+0000 7fb7ca600640 1 --2- 172.21.15.6:0/1703573266 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb7a006b500 0x7fb7a006d9c0 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:38.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.615+0000 7fb7ca600640 1 --2- 172.21.15.6:0/1703573266 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb7c415bfb0 0x7fb7c415e4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:38.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.615+0000 7fb7ca600640 1 --2- 172.21.15.6:0/1703573266 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb7c406afb0 0x7fb7c413a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:38.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.615+0000 7fb7ca600640 1 -- 172.21.15.6:0/1703573266 >> 172.21.15.6:0/1703573266 conn(0x7fb7c40b2ae0 msgr2=0x7fb7c40ae580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:32:38.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.615+0000 7fb7ca600640 1 -- 172.21.15.6:0/1703573266 shutdown_connections 2024-03-04T09:32:38.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.615+0000 7fb7ca600640 1 -- 172.21.15.6:0/1703573266 wait complete. 2024-03-04T09:32:38.633 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T09:25:07.935198+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-04T09:31:19.018233+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '108', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2024-03-04T09:31:20.131544+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '109', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-04T09:32:38.633 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-04T09:32:38.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.883+0000 7f27ba000640 1 Processor -- start 2024-03-04T09:32:38.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.884+0000 7f27ba000640 1 -- start start 2024-03-04T09:32:38.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.885+0000 7f27ba000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f27b415afa0 0x7f27b40cf3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:38.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.885+0000 7f27ba000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f27b4070ce0 0x7f27b40ccad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:38.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.886+0000 7f27ba000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27b4071c60 con 0x7f27b40cd010 2024-03-04T09:32:38.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.886+0000 7f27ba000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27b40cfa10 con 0x7f27b4070ce0 2024-03-04T09:32:38.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.886+0000 7f27ba000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27b4071f40 con 0x7f27b415afa0 2024-03-04T09:32:38.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.886+0000 7f27b3400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f27b415afa0 0x7f27b40cf3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:38.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.886+0000 7f27b3400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f27b415afa0 0x7f27b40cf3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.6:51500/0 (socket says 172.21.15.6:51500) 2024-03-04T09:32:38.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.886+0000 7f27b3400640 1 -- 172.21.15.6:0/2142505962 learned_addr learned my addr 172.21.15.6:0/2142505962 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:32:38.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.887+0000 7f27b2a00640 1 --2- 172.21.15.6:0/2142505962 >> v2:172.21.15.146:3300/0 conn(0x7f27b4070ce0 0x7f27b40ccad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:38.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.887+0000 7f27b3400640 1 -- 172.21.15.6:0/2142505962 >> v2:172.21.15.146:3300/0 conn(0x7f27b4070ce0 msgr2=0x7f27b40ccad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:32:38.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.887+0000 7f27b3400640 1 --2- 172.21.15.6:0/2142505962 >> v2:172.21.15.146:3300/0 conn(0x7f27b4070ce0 0x7f27b40ccad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:38.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.887+0000 7f27b3400640 1 -- 172.21.15.6:0/2142505962 >> v1:172.21.15.146:6789/0 conn(0x7f27b40cd010 legacy=0x7f27b406c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:38.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.887+0000 7f27b3400640 1 -- 172.21.15.6:0/2142505962 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27b4072220 con 0x7f27b415afa0 2024-03-04T09:32:38.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.888+0000 7f27b3400640 1 --2- 172.21.15.6:0/2142505962 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f27b415afa0 0x7f27b40cf3a0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f27a4009870 tx=0x7f27a402b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7a2464d16e1e1e97 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:38.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.889+0000 7f27b2000640 1 -- 172.21.15.6:0/2142505962 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27a4035020 con 0x7f27b415afa0 2024-03-04T09:32:38.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.889+0000 7f27b2000640 1 -- 172.21.15.6:0/2142505962 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27a4031640 con 0x7f27b415afa0 2024-03-04T09:32:38.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.889+0000 7f27b2000640 1 -- 172.21.15.6:0/2142505962 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27a402d040 con 0x7f27b415afa0 2024-03-04T09:32:38.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.890+0000 7f27ba000640 1 -- 172.21.15.6:0/2142505962 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f27b415afa0 msgr2=0x7f27b40cf3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:38.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.890+0000 7f27ba000640 1 --2- 172.21.15.6:0/2142505962 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f27b415afa0 0x7f27b40cf3a0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f27a4009870 tx=0x7f27a402b8f0 comp rx=0 tx=0).stop 2024-03-04T09:32:38.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.890+0000 7f27ba000640 1 -- 172.21.15.6:0/2142505962 shutdown_connections 2024-03-04T09:32:38.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.890+0000 7f27ba000640 1 --2- 172.21.15.6:0/2142505962 >> v2:172.21.15.146:3300/0 conn(0x7f27b4070ce0 0x7f27b40ccad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:38.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.890+0000 7f27ba000640 1 --2- 172.21.15.6:0/2142505962 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f27b415afa0 0x7f27b40cf3a0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:38.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.890+0000 7f27ba000640 1 -- 172.21.15.6:0/2142505962 >> 172.21.15.6:0/2142505962 conn(0x7f27b406b3d0 msgr2=0x7f27b406b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:32:38.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.890+0000 7f27ba000640 1 -- 172.21.15.6:0/2142505962 shutdown_connections 2024-03-04T09:32:38.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.890+0000 7f27ba000640 1 -- 172.21.15.6:0/2142505962 wait complete. 2024-03-04T09:32:38.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.892+0000 7f27ba000640 1 Processor -- start 2024-03-04T09:32:38.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.892+0000 7f27ba000640 1 -- start start 2024-03-04T09:32:38.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.893+0000 7f27ba000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f27b4070ce0 0x7f27b414c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:38.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.893+0000 7f27ba000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f27b40cd010 0x7f27b409b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:38.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.893+0000 7f27ba000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f27b415afa0 0x7f27b409b960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:38.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.893+0000 7f27ba000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27b40b4ad0 con 0x7f27b40cd010 2024-03-04T09:32:38.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.893+0000 7f27ba000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27b4071140 con 0x7f27b415afa0 2024-03-04T09:32:38.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.893+0000 7f27ba000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27b40b4c50 con 0x7f27b4070ce0 2024-03-04T09:32:38.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.893+0000 7f27b2a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f27b40cd010 0x7f27b409b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:38.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.893+0000 7f27b2a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f27b40cd010 0x7f27b409b420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:47122/0 (socket says 172.21.15.6:47122) 2024-03-04T09:32:38.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.893+0000 7f27b2a00640 1 -- 172.21.15.6:0/3076382771 learned_addr learned my addr 172.21.15.6:0/3076382771 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:32:38.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.893+0000 7f27b3400640 1 --2- 172.21.15.6:0/3076382771 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f27b4070ce0 0x7f27b414c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:38.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.894+0000 7f27b3e00640 1 --2- 172.21.15.6:0/3076382771 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f27b415afa0 0x7f27b409b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:38.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.894+0000 7f27b2a00640 1 -- 172.21.15.6:0/3076382771 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f27b4070ce0 msgr2=0x7f27b414c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:38.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.894+0000 7f27b2a00640 1 --2- 172.21.15.6:0/3076382771 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f27b4070ce0 0x7f27b414c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:38.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.894+0000 7f27b2a00640 1 -- 172.21.15.6:0/3076382771 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f27b415afa0 msgr2=0x7f27b409b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:38.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.894+0000 7f27b2a00640 1 --2- 172.21.15.6:0/3076382771 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f27b415afa0 0x7f27b409b960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:38.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.894+0000 7f27b2a00640 1 -- 172.21.15.6:0/3076382771 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27b414c960 con 0x7f27b40cd010 2024-03-04T09:32:38.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.894+0000 7f27b3e00640 1 --2- 172.21.15.6:0/3076382771 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f27b415afa0 0x7f27b409b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:32:38.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.894+0000 7f27b2a00640 1 --2- 172.21.15.6:0/3076382771 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f27b40cd010 0x7f27b409b420 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7f27a000e9b0 tx=0x7f27a000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:38.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.894+0000 7f27ab400640 1 -- 172.21.15.6:0/3076382771 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27a0017070 con 0x7f27b40cd010 2024-03-04T09:32:38.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.894+0000 7f27ab400640 1 -- 172.21.15.6:0/3076382771 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27a000c9a0 con 0x7f27b40cd010 2024-03-04T09:32:38.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.894+0000 7f27ab400640 1 -- 172.21.15.6:0/3076382771 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27a000cd90 con 0x7f27b40cd010 2024-03-04T09:32:38.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.894+0000 7f27ba000640 1 -- 172.21.15.6:0/3076382771 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27b409bf90 con 0x7f27b40cd010 2024-03-04T09:32:38.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.895+0000 7f27ba000640 1 -- 172.21.15.6:0/3076382771 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27b409c550 con 0x7f27b40cd010 2024-03-04T09:32:38.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.896+0000 7f27ba000640 1 -- 172.21.15.6:0/3076382771 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27b404f330 con 0x7f27b40cd010 2024-03-04T09:32:38.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.897+0000 7f27ab400640 1 -- 172.21.15.6:0/3076382771 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f27a001a070 con 0x7f27b40cd010 2024-03-04T09:32:38.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.898+0000 7f27ab400640 1 --2- 172.21.15.6:0/3076382771 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f279406b500 0x7f279406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:38.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.898+0000 7f27ab400640 1 -- 172.21.15.6:0/3076382771 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f27a008a280 con 0x7f27b40cd010 2024-03-04T09:32:38.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.899+0000 7f27b3400640 1 --2- 172.21.15.6:0/3076382771 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f279406b500 0x7f279406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:38.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.900+0000 7f27b3400640 1 --2- 172.21.15.6:0/3076382771 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f279406b500 0x7f279406d9c0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f27a4036000 tx=0x7f27a403c040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:38.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:38.903+0000 7f27ab400640 1 -- 172.21.15.6:0/3076382771 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f27a0059070 con 0x7f27b40cd010 2024-03-04T09:32:39.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.101+0000 7f27ba000640 1 -- 172.21.15.6:0/3076382771 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f27b415d880 con 0x7f27b40cd010 2024-03-04T09:32:39.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.102+0000 7f27ab400640 1 -- 172.21.15.6:0/3076382771 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v115) v1 ==== 93+0+10 (secure 0 0 0) 0x7f27a0058970 con 0x7f27b40cd010 2024-03-04T09:32:39.099 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-04T09:32:39.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.104+0000 7f27ba000640 1 -- 172.21.15.6:0/3076382771 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f279406b500 msgr2=0x7f279406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:39.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.104+0000 7f27ba000640 1 --2- 172.21.15.6:0/3076382771 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f279406b500 0x7f279406d9c0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f27a4036000 tx=0x7f27a403c040 comp rx=0 tx=0).stop 2024-03-04T09:32:39.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.104+0000 7f27ba000640 1 -- 172.21.15.6:0/3076382771 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f27b40cd010 msgr2=0x7f27b409b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:39.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.104+0000 7f27ba000640 1 --2- 172.21.15.6:0/3076382771 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f27b40cd010 0x7f27b409b420 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7f27a000e9b0 tx=0x7f27a000ee70 comp rx=0 tx=0).stop 2024-03-04T09:32:39.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.104+0000 7f27ba000640 1 -- 172.21.15.6:0/3076382771 shutdown_connections 2024-03-04T09:32:39.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.104+0000 7f27ba000640 1 --2- 172.21.15.6:0/3076382771 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f27b415afa0 0x7f27b409b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:39.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.104+0000 7f27ba000640 1 --2- 172.21.15.6:0/3076382771 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f27b40cd010 0x7f27b409b420 unknown :-1 s=CLOSED pgs=1407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:39.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.104+0000 7f27ba000640 1 --2- 172.21.15.6:0/3076382771 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f279406b500 0x7f279406d9c0 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:39.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.104+0000 7f27ba000640 1 --2- 172.21.15.6:0/3076382771 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f27b4070ce0 0x7f27b414c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:39.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.104+0000 7f27ba000640 1 -- 172.21.15.6:0/3076382771 >> 172.21.15.6:0/3076382771 conn(0x7f27b406b3d0 msgr2=0x7f27b406b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:32:39.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.104+0000 7f27ba000640 1 -- 172.21.15.6:0/3076382771 shutdown_connections 2024-03-04T09:32:39.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.104+0000 7f27ba000640 1 -- 172.21.15.6:0/3076382771 wait complete. 2024-03-04T09:32:39.125 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-04T09:32:39.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.325+0000 7fc622200640 1 Processor -- start 2024-03-04T09:32:39.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.325+0000 7fc622200640 1 -- start start 2024-03-04T09:32:39.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.326+0000 7fc622200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc61c0b2250 0x7fc61c168950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:39.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.326+0000 7fc622200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc61c06de40 con 0x7fc61c147590 2024-03-04T09:32:39.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.326+0000 7fc622200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc61c06e120 con 0x7fc61c06d8d0 2024-03-04T09:32:39.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.326+0000 7fc622200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc61c147970 con 0x7fc61c0b2250 2024-03-04T09:32:39.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.326+0000 7fc620e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fc61c147590 0x7fc61c071480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:35068/0 (socket says 172.21.15.6:35068) 2024-03-04T09:32:39.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.326+0000 7fc620e00640 1 -- 172.21.15.6:0/3488271592 learned_addr learned my addr 172.21.15.6:0/3488271592 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:32:39.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.327+0000 7fc61b400640 1 -- 172.21.15.6:0/3488271592 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3928469016 0 0) 0x7fc61c06de40 con 0x7fc61c147590 2024-03-04T09:32:39.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.327+0000 7fc621800640 1 --2- 172.21.15.6:0/3488271592 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc61c0b2250 0x7fc61c168950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:39.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.327+0000 7fc61b400640 1 -- 172.21.15.6:0/3488271592 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc604003520 con 0x7fc61c147590 2024-03-04T09:32:39.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.327+0000 7fc61b400640 1 -- 172.21.15.6:0/3488271592 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1515124529 0 0) 0x7fc604003520 con 0x7fc61c147590 2024-03-04T09:32:39.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.327+0000 7fc61b400640 1 -- 172.21.15.6:0/3488271592 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc61c0b2250 msgr2=0x7fc61c168950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:32:39.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.327+0000 7fc61b400640 1 --2- 172.21.15.6:0/3488271592 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc61c0b2250 0x7fc61c168950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:39.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.327+0000 7fc61b400640 1 -- 172.21.15.6:0/3488271592 >> v1:172.21.15.146:6789/0 conn(0x7fc61c06d8d0 legacy=0x7fc61c0b1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:39.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.327+0000 7fc61b400640 1 -- 172.21.15.6:0/3488271592 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc61c0b2e40 con 0x7fc61c147590 2024-03-04T09:32:39.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.327+0000 7fc61b400640 1 -- 172.21.15.6:0/3488271592 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7fc610001670 con 0x7fc61c147590 2024-03-04T09:32:39.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.327+0000 7fc61b400640 1 -- 172.21.15.6:0/3488271592 >> v1:172.21.15.6:6789/0 conn(0x7fc61c147590 legacy=0x7fc61c071480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:39.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.327+0000 7fc61b400640 1 --2- 172.21.15.6:0/3488271592 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc61c0b2250 0x7fc6040039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:39.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.327+0000 7fc61b400640 1 --2- 172.21.15.6:0/3488271592 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc604003fb0 0x7fc604004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:39.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.327+0000 7fc61b400640 1 --2- 172.21.15.6:0/3488271592 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc604004d40 0x7fc6040051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:39.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.327+0000 7fc61b400640 1 -- 172.21.15.6:0/3488271592 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc61c147970 con 0x7fc604003fb0 2024-03-04T09:32:39.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.327+0000 7fc61b400640 1 -- 172.21.15.6:0/3488271592 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc60400bb00 con 0x7fc604004d40 2024-03-04T09:32:39.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.327+0000 7fc61b400640 1 -- 172.21.15.6:0/3488271592 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc60400bd00 con 0x7fc61c0b2250 2024-03-04T09:32:39.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.328+0000 7fc61be00640 1 --2- 172.21.15.6:0/3488271592 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc604003fb0 0x7fc604004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:39.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.328+0000 7fc621800640 1 --2- 172.21.15.6:0/3488271592 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc604004d40 0x7fc6040051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:39.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.328+0000 7fc61be00640 1 -- 172.21.15.6:0/3488271592 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc61c0b2250 msgr2=0x7fc6040039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:32:39.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.328+0000 7fc620e00640 1 --2- 172.21.15.6:0/3488271592 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc61c0b2250 0x7fc6040039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:39.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.328+0000 7fc61be00640 1 --2- 172.21.15.6:0/3488271592 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc61c0b2250 0x7fc6040039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:39.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.328+0000 7fc61be00640 1 -- 172.21.15.6:0/3488271592 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc604004d40 msgr2=0x7fc6040051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:32:39.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.328+0000 7fc61be00640 1 --2- 172.21.15.6:0/3488271592 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc604004d40 0x7fc6040051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:39.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.328+0000 7fc61be00640 1 -- 172.21.15.6:0/3488271592 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc604003520 con 0x7fc604003fb0 2024-03-04T09:32:39.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.328+0000 7fc621800640 1 --2- 172.21.15.6:0/3488271592 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc604004d40 0x7fc6040051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:32:39.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.328+0000 7fc620e00640 1 --2- 172.21.15.6:0/3488271592 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc61c0b2250 0x7fc6040039e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:32:39.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.329+0000 7fc61be00640 1 --2- 172.21.15.6:0/3488271592 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc604003fb0 0x7fc604004800 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7fc608001750 tx=0x7fc604005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9ebe6ca754e2887 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:39.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.329+0000 7fc61b400640 1 -- 172.21.15.6:0/3488271592 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc608001370 con 0x7fc604003fb0 2024-03-04T09:32:39.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.330+0000 7fc61b400640 1 -- 172.21.15.6:0/3488271592 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6080052d0 con 0x7fc604003fb0 2024-03-04T09:32:39.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.330+0000 7fc61b400640 1 -- 172.21.15.6:0/3488271592 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc608003830 con 0x7fc604003fb0 2024-03-04T09:32:39.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.330+0000 7fc622200640 1 -- 172.21.15.6:0/3488271592 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc604003fb0 msgr2=0x7fc604004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:39.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.330+0000 7fc622200640 1 --2- 172.21.15.6:0/3488271592 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc604003fb0 0x7fc604004800 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7fc608001750 tx=0x7fc604005830 comp rx=0 tx=0).stop 2024-03-04T09:32:39.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.330+0000 7fc621800640 1 -- 172.21.15.6:0/3488271592 reap_dead start 2024-03-04T09:32:39.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.330+0000 7fc622200640 1 -- 172.21.15.6:0/3488271592 shutdown_connections 2024-03-04T09:32:39.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.330+0000 7fc622200640 1 -- 172.21.15.6:0/3488271592 >> 172.21.15.6:0/3488271592 conn(0x7fc61c06b3a0 msgr2=0x7fc61c06b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:32:39.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.330+0000 7fc622200640 1 -- 172.21.15.6:0/3488271592 shutdown_connections 2024-03-04T09:32:39.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.330+0000 7fc622200640 1 -- 172.21.15.6:0/3488271592 wait complete. 2024-03-04T09:32:39.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.332+0000 7fc622200640 1 Processor -- start 2024-03-04T09:32:39.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.332+0000 7fc622200640 1 -- start start 2024-03-04T09:32:39.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.333+0000 7fc622200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc604003fb0 0x7fc61c160420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:39.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.333+0000 7fc622200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc61c0cc480 0x7fc61c15fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:39.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.333+0000 7fc622200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc61c0cda90 0x7fc61c0cdf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:39.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.333+0000 7fc622200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc61c0b4650 con 0x7fc61c0cc480 2024-03-04T09:32:39.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.333+0000 7fc622200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc61c0b31b0 con 0x7fc604003fb0 2024-03-04T09:32:39.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.333+0000 7fc622200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc61c0b47d0 con 0x7fc61c0cda90 2024-03-04T09:32:39.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.333+0000 7fc61be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc61c0cc480 0x7fc61c15fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:39.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.333+0000 7fc61be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc61c0cc480 0x7fc61c15fb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:47144/0 (socket says 172.21.15.6:47144) 2024-03-04T09:32:39.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.333+0000 7fc61be00640 1 -- 172.21.15.6:0/4053813989 learned_addr learned my addr 172.21.15.6:0/4053813989 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:32:39.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.334+0000 7fc621800640 1 --2- 172.21.15.6:0/4053813989 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc61c0cda90 0x7fc61c0cdf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:39.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.334+0000 7fc61be00640 1 -- 172.21.15.6:0/4053813989 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc61c0cda90 msgr2=0x7fc61c0cdf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:39.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.334+0000 7fc61be00640 1 --2- 172.21.15.6:0/4053813989 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc61c0cda90 0x7fc61c0cdf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:39.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.334+0000 7fc61be00640 1 -- 172.21.15.6:0/4053813989 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc604003fb0 msgr2=0x7fc61c160420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:39.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.334+0000 7fc61be00640 1 --2- 172.21.15.6:0/4053813989 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc604003fb0 0x7fc61c160420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:39.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.334+0000 7fc61be00640 1 -- 172.21.15.6:0/4053813989 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc61c0cc990 con 0x7fc61c0cc480 2024-03-04T09:32:39.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.334+0000 7fc61be00640 1 --2- 172.21.15.6:0/4053813989 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc61c0cc480 0x7fc61c15fb70 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 crypto rx=0x7fc608008fd0 tx=0x7fc608001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:39.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.334+0000 7fc618c00640 1 -- 172.21.15.6:0/4053813989 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc608041070 con 0x7fc61c0cc480 2024-03-04T09:32:39.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.334+0000 7fc618c00640 1 -- 172.21.15.6:0/4053813989 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc608000ed0 con 0x7fc61c0cc480 2024-03-04T09:32:39.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.334+0000 7fc622200640 1 -- 172.21.15.6:0/4053813989 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc61c0ccc70 con 0x7fc61c0cc480 2024-03-04T09:32:39.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.334+0000 7fc618c00640 1 -- 172.21.15.6:0/4053813989 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc608007a30 con 0x7fc61c0cc480 2024-03-04T09:32:39.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.334+0000 7fc622200640 1 -- 172.21.15.6:0/4053813989 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc61c0cd230 con 0x7fc61c0cc480 2024-03-04T09:32:39.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.335+0000 7fc622200640 1 -- 172.21.15.6:0/4053813989 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc61c04f330 con 0x7fc61c0cc480 2024-03-04T09:32:39.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.336+0000 7fc618c00640 1 -- 172.21.15.6:0/4053813989 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc608044070 con 0x7fc61c0cc480 2024-03-04T09:32:39.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.337+0000 7fc618c00640 1 --2- 172.21.15.6:0/4053813989 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc5fc06b4d0 0x7fc5fc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:39.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.337+0000 7fc618c00640 1 -- 172.21.15.6:0/4053813989 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5853+0+0 (secure 0 0 0) 0x7fc6080abf30 con 0x7fc61c0cc480 2024-03-04T09:32:39.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.338+0000 7fc620e00640 1 --2- 172.21.15.6:0/4053813989 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc5fc06b4d0 0x7fc5fc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:39.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.339+0000 7fc620e00640 1 --2- 172.21.15.6:0/4053813989 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc5fc06b4d0 0x7fc5fc06d990 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fc610001eb0 tx=0x7fc610009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:39.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.342+0000 7fc618c00640 1 -- 172.21.15.6:0/4053813989 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc60807ada0 con 0x7fc61c0cc480 2024-03-04T09:32:39.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.532+0000 7fc622200640 1 -- 172.21.15.6:0/4053813989 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fc61c072730 con 0x7fc61c0cc480 2024-03-04T09:32:39.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.532+0000 7fc618c00640 1 -- 172.21.15.6:0/4053813989 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v115) v1 ==== 104+0+11 (secure 0 0 0) 0x7fc60807a6a0 con 0x7fc61c0cc480 2024-03-04T09:32:39.529 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-03-04T09:32:39.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.534+0000 7fc622200640 1 -- 172.21.15.6:0/4053813989 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc5fc06b4d0 msgr2=0x7fc5fc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:39.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.534+0000 7fc622200640 1 --2- 172.21.15.6:0/4053813989 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc5fc06b4d0 0x7fc5fc06d990 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fc610001eb0 tx=0x7fc610009040 comp rx=0 tx=0).stop 2024-03-04T09:32:39.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.534+0000 7fc622200640 1 -- 172.21.15.6:0/4053813989 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc61c0cc480 msgr2=0x7fc61c15fb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:39.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.534+0000 7fc622200640 1 --2- 172.21.15.6:0/4053813989 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc61c0cc480 0x7fc61c15fb70 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 crypto rx=0x7fc608008fd0 tx=0x7fc608001120 comp rx=0 tx=0).stop 2024-03-04T09:32:39.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.535+0000 7fc622200640 1 -- 172.21.15.6:0/4053813989 shutdown_connections 2024-03-04T09:32:39.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.535+0000 7fc622200640 1 --2- 172.21.15.6:0/4053813989 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc5fc06b4d0 0x7fc5fc06d990 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:39.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.535+0000 7fc622200640 1 --2- 172.21.15.6:0/4053813989 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc61c0cda90 0x7fc61c0cdf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:39.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.535+0000 7fc622200640 1 --2- 172.21.15.6:0/4053813989 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc61c0cc480 0x7fc61c15fb70 unknown :-1 s=CLOSED pgs=1410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:39.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.535+0000 7fc622200640 1 --2- 172.21.15.6:0/4053813989 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc604003fb0 0x7fc61c160420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:39.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.535+0000 7fc622200640 1 -- 172.21.15.6:0/4053813989 >> 172.21.15.6:0/4053813989 conn(0x7fc61c06b3a0 msgr2=0x7fc61c0afae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:32:39.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.535+0000 7fc622200640 1 -- 172.21.15.6:0/4053813989 shutdown_connections 2024-03-04T09:32:39.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.535+0000 7fc622200640 1 -- 172.21.15.6:0/4053813989 wait complete. 2024-03-04T09:32:39.549 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-04T09:32:39.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.771+0000 7f5d8ec00640 1 Processor -- start 2024-03-04T09:32:39.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.772+0000 7f5d8ec00640 1 -- start start 2024-03-04T09:32:39.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.773+0000 7f5d8ec00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f5d880d0d40 0x7f5d880b3510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:39.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.773+0000 7f5d8ec00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d880ceca0 con 0x7f5d880d4860 2024-03-04T09:32:39.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.773+0000 7f5d8ec00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d880cef80 con 0x7f5d8815dee0 2024-03-04T09:32:39.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.773+0000 7f5d8ec00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d880cf5d0 con 0x7f5d880d0d40 2024-03-04T09:32:39.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.774+0000 7f5d8ce00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f5d880d4860 0x7f5d880d0630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:35080/0 (socket says 172.21.15.6:35080) 2024-03-04T09:32:39.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.774+0000 7f5d8ce00640 1 -- 172.21.15.6:0/103157732 learned_addr learned my addr 172.21.15.6:0/103157732 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:32:39.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.774+0000 7f5d8e200640 1 --2- 172.21.15.6:0/103157732 >> v2:172.21.15.146:3300/0 conn(0x7f5d880d0d40 0x7f5d880b3510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:39.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.774+0000 7f5d87e00640 1 -- 172.21.15.6:0/103157732 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2364948742 0 0) 0x7f5d880ceca0 con 0x7f5d880d4860 2024-03-04T09:32:39.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.774+0000 7f5d87e00640 1 -- 172.21.15.6:0/103157732 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d70003520 con 0x7f5d880d4860 2024-03-04T09:32:39.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.775+0000 7f5d87e00640 1 -- 172.21.15.6:0/103157732 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 459453437 0 0) 0x7f5d70003520 con 0x7f5d880d4860 2024-03-04T09:32:39.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.775+0000 7f5d87e00640 1 -- 172.21.15.6:0/103157732 >> v2:172.21.15.146:3300/0 conn(0x7f5d880d0d40 msgr2=0x7f5d880b3510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:32:39.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.775+0000 7f5d87e00640 1 --2- 172.21.15.6:0/103157732 >> v2:172.21.15.146:3300/0 conn(0x7f5d880d0d40 0x7f5d880b3510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:39.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.775+0000 7f5d87e00640 1 -- 172.21.15.6:0/103157732 >> v1:172.21.15.146:6789/0 conn(0x7f5d8815dee0 legacy=0x7f5d880d18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:39.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.775+0000 7f5d87e00640 1 -- 172.21.15.6:0/103157732 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d880cf260 con 0x7f5d880d4860 2024-03-04T09:32:39.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.775+0000 7f5d87e00640 1 -- 172.21.15.6:0/103157732 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7f5d74001670 con 0x7f5d880d4860 2024-03-04T09:32:39.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.775+0000 7f5d87e00640 1 -- 172.21.15.6:0/103157732 >> v1:172.21.15.6:6789/0 conn(0x7f5d880d4860 legacy=0x7f5d880d0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:39.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.775+0000 7f5d87e00640 1 --2- 172.21.15.6:0/103157732 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5d70003a00 0x7f5d70003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:39.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.775+0000 7f5d87e00640 1 --2- 172.21.15.6:0/103157732 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5d70004800 0x7f5d70004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:39.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.775+0000 7f5d87e00640 1 --2- 172.21.15.6:0/103157732 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5d700051c0 0x7f5d7000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:39.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.775+0000 7f5d87e00640 1 -- 172.21.15.6:0/103157732 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d880cf5d0 con 0x7f5d700051c0 2024-03-04T09:32:39.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.775+0000 7f5d87e00640 1 -- 172.21.15.6:0/103157732 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d7000c0c0 con 0x7f5d70003a00 2024-03-04T09:32:39.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.775+0000 7f5d87e00640 1 -- 172.21.15.6:0/103157732 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d7000c2c0 con 0x7f5d70004800 2024-03-04T09:32:39.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.775+0000 7f5d8e200640 1 --2- 172.21.15.6:0/103157732 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5d700051c0 0x7f5d7000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:39.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.775+0000 7f5d8d800640 1 --2- 172.21.15.6:0/103157732 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5d70003a00 0x7f5d70003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:39.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.776+0000 7f5d8ce00640 1 --2- 172.21.15.6:0/103157732 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5d70004800 0x7f5d70004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:39.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.776+0000 7f5d8e200640 1 -- 172.21.15.6:0/103157732 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5d70004800 msgr2=0x7f5d70004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:32:39.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.776+0000 7f5d8e200640 1 --2- 172.21.15.6:0/103157732 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5d70004800 0x7f5d70004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:39.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.776+0000 7f5d8e200640 1 -- 172.21.15.6:0/103157732 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5d70003a00 msgr2=0x7f5d70003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:32:39.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.776+0000 7f5d8e200640 1 --2- 172.21.15.6:0/103157732 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5d70003a00 0x7f5d70003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:39.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.776+0000 7f5d8e200640 1 -- 172.21.15.6:0/103157732 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d70003520 con 0x7f5d700051c0 2024-03-04T09:32:39.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.777+0000 7f5d8e200640 1 -- 172.21.15.6:0/103157732 reap_dead start 2024-03-04T09:32:39.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.777+0000 7f5d8e200640 1 --2- 172.21.15.6:0/103157732 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5d700051c0 0x7f5d7000b9c0 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7f5d80009fc0 tx=0x7f5d70005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45475e890c02bc43 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:39.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.777+0000 7f5d87e00640 1 -- 172.21.15.6:0/103157732 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d80009c10 con 0x7f5d700051c0 2024-03-04T09:32:39.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.777+0000 7f5d87e00640 1 -- 172.21.15.6:0/103157732 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d8002cca0 con 0x7f5d700051c0 2024-03-04T09:32:39.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.777+0000 7f5d87e00640 1 -- 172.21.15.6:0/103157732 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d80003bd0 con 0x7f5d700051c0 2024-03-04T09:32:39.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.778+0000 7f5d8ec00640 1 -- 172.21.15.6:0/103157732 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5d700051c0 msgr2=0x7f5d7000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:39.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.778+0000 7f5d8ec00640 1 --2- 172.21.15.6:0/103157732 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5d700051c0 0x7f5d7000b9c0 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7f5d80009fc0 tx=0x7f5d70005830 comp rx=0 tx=0).stop 2024-03-04T09:32:39.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.778+0000 7f5d8ec00640 1 -- 172.21.15.6:0/103157732 shutdown_connections 2024-03-04T09:32:39.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.778+0000 7f5d8ec00640 1 --2- 172.21.15.6:0/103157732 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5d700051c0 0x7f5d7000b9c0 unknown :-1 s=CLOSED pgs=1412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:39.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.778+0000 7f5d8ec00640 1 -- 172.21.15.6:0/103157732 >> 172.21.15.6:0/103157732 conn(0x7f5d880d3cd0 msgr2=0x7f5d880d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:32:39.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.778+0000 7f5d8ec00640 1 -- 172.21.15.6:0/103157732 shutdown_connections 2024-03-04T09:32:39.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.778+0000 7f5d8ec00640 1 -- 172.21.15.6:0/103157732 wait complete. 2024-03-04T09:32:39.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.780+0000 7f5d8ec00640 1 Processor -- start 2024-03-04T09:32:39.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.780+0000 7f5d8ec00640 1 -- start start 2024-03-04T09:32:39.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.781+0000 7f5d8ec00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5d700051c0 0x7f5d881f6730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:39.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.781+0000 7f5d8ec00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5d881e6750 0x7f5d881f6c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:39.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.781+0000 7f5d8ec00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5d8814c480 0x7f5d8814c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:39.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.781+0000 7f5d8ec00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d880cfa60 con 0x7f5d8814c480 2024-03-04T09:32:39.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.781+0000 7f5d8ec00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d8815e2c0 con 0x7f5d881e6750 2024-03-04T09:32:39.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.781+0000 7f5d8ec00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d880cfbe0 con 0x7f5d700051c0 2024-03-04T09:32:39.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.781+0000 7f5d8e200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5d8814c480 0x7f5d8814c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:39.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.781+0000 7f5d8e200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5d8814c480 0x7f5d8814c900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:47170/0 (socket says 172.21.15.6:47170) 2024-03-04T09:32:39.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.781+0000 7f5d8e200640 1 -- 172.21.15.6:0/1698953683 learned_addr learned my addr 172.21.15.6:0/1698953683 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:32:39.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.782+0000 7f5d8d800640 1 --2- 172.21.15.6:0/1698953683 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5d700051c0 0x7f5d881f6730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:39.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.782+0000 7f5d8ce00640 1 --2- 172.21.15.6:0/1698953683 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5d881e6750 0x7f5d881f6c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:39.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.782+0000 7f5d8e200640 1 -- 172.21.15.6:0/1698953683 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5d700051c0 msgr2=0x7f5d881f6730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:39.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.782+0000 7f5d8e200640 1 --2- 172.21.15.6:0/1698953683 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5d700051c0 0x7f5d881f6730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:39.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.782+0000 7f5d8e200640 1 -- 172.21.15.6:0/1698953683 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5d881e6750 msgr2=0x7f5d881f6c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:39.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.782+0000 7f5d8e200640 1 --2- 172.21.15.6:0/1698953683 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5d881e6750 0x7f5d881f6c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:39.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.782+0000 7f5d8e200640 1 -- 172.21.15.6:0/1698953683 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d881617f0 con 0x7f5d8814c480 2024-03-04T09:32:39.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.782+0000 7f5d8e200640 1 --2- 172.21.15.6:0/1698953683 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5d8814c480 0x7f5d8814c900 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 crypto rx=0x7f5d8002c6d0 tx=0x7f5d80009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:39.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.783+0000 7f5d85600640 1 -- 172.21.15.6:0/1698953683 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d80003990 con 0x7f5d8814c480 2024-03-04T09:32:39.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.783+0000 7f5d85600640 1 -- 172.21.15.6:0/1698953683 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d80044070 con 0x7f5d8814c480 2024-03-04T09:32:39.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.783+0000 7f5d85600640 1 -- 172.21.15.6:0/1698953683 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d80003dd0 con 0x7f5d8814c480 2024-03-04T09:32:39.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.783+0000 7f5d8ec00640 1 -- 172.21.15.6:0/1698953683 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d8814a2b0 con 0x7f5d8814c480 2024-03-04T09:32:39.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.783+0000 7f5d8ec00640 1 -- 172.21.15.6:0/1698953683 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d8814a830 con 0x7f5d8814c480 2024-03-04T09:32:39.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.784+0000 7f5d8ec00640 1 -- 172.21.15.6:0/1698953683 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d8804f330 con 0x7f5d8814c480 2024-03-04T09:32:39.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.785+0000 7f5d85600640 1 -- 172.21.15.6:0/1698953683 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5d80005d10 con 0x7f5d8814c480 2024-03-04T09:32:39.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.786+0000 7f5d85600640 1 --2- 172.21.15.6:0/1698953683 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f5d6806b500 0x7f5d6806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:39.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.786+0000 7f5d85600640 1 -- 172.21.15.6:0/1698953683 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f5d800acc40 con 0x7f5d8814c480 2024-03-04T09:32:39.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.786+0000 7f5d8d800640 1 --2- 172.21.15.6:0/1698953683 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f5d6806b500 0x7f5d6806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:39.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.787+0000 7f5d8d800640 1 --2- 172.21.15.6:0/1698953683 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f5d6806b500 0x7f5d6806d9c0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f5d7c005410 tx=0x7f5d7c009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:39.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.791+0000 7f5d85600640 1 -- 172.21.15.6:0/1698953683 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5d8007ba20 con 0x7f5d8814c480 2024-03-04T09:32:39.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.991+0000 7f5d8ec00640 1 -- 172.21.15.6:0/1698953683 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f5d880b9140 con 0x7f5d8814c480 2024-03-04T09:32:39.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.992+0000 7f5d85600640 1 -- 172.21.15.6:0/1698953683 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v115) v1 ==== 100+0+11 (secure 0 0 0) 0x7f5d8007b320 con 0x7f5d8814c480 2024-03-04T09:32:39.989 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 64 2024-03-04T09:32:39.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.994+0000 7f5d8ec00640 1 -- 172.21.15.6:0/1698953683 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f5d6806b500 msgr2=0x7f5d6806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:39.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.994+0000 7f5d8ec00640 1 --2- 172.21.15.6:0/1698953683 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f5d6806b500 0x7f5d6806d9c0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f5d7c005410 tx=0x7f5d7c009040 comp rx=0 tx=0).stop 2024-03-04T09:32:39.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.994+0000 7f5d8ec00640 1 -- 172.21.15.6:0/1698953683 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5d8814c480 msgr2=0x7f5d8814c900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:39.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.994+0000 7f5d8ec00640 1 --2- 172.21.15.6:0/1698953683 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5d8814c480 0x7f5d8814c900 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 crypto rx=0x7f5d8002c6d0 tx=0x7f5d80009750 comp rx=0 tx=0).stop 2024-03-04T09:32:39.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.994+0000 7f5d8ec00640 1 -- 172.21.15.6:0/1698953683 shutdown_connections 2024-03-04T09:32:39.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.994+0000 7f5d8ec00640 1 --2- 172.21.15.6:0/1698953683 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5d8814c480 0x7f5d8814c900 unknown :-1 s=CLOSED pgs=1413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:39.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.994+0000 7f5d8ec00640 1 --2- 172.21.15.6:0/1698953683 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5d881e6750 0x7f5d881f6c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:39.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.994+0000 7f5d8ec00640 1 --2- 172.21.15.6:0/1698953683 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f5d6806b500 0x7f5d6806d9c0 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:39.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.994+0000 7f5d8ec00640 1 --2- 172.21.15.6:0/1698953683 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5d700051c0 0x7f5d881f6730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:39.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.994+0000 7f5d8ec00640 1 -- 172.21.15.6:0/1698953683 >> 172.21.15.6:0/1698953683 conn(0x7f5d880d3cd0 msgr2=0x7f5d8806b8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:32:39.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.994+0000 7f5d8ec00640 1 -- 172.21.15.6:0/1698953683 shutdown_connections 2024-03-04T09:32:39.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:39.994+0000 7f5d8ec00640 1 -- 172.21.15.6:0/1698953683 wait complete. 2024-03-04T09:32:40.013 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:32:40.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.240+0000 7fd66f000640 1 Processor -- start 2024-03-04T09:32:40.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.240+0000 7fd66f000640 1 -- start start 2024-03-04T09:32:40.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.242+0000 7fd66f000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd66815e2e0 0x7fd6680d0830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:40.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.242+0000 7fd66f000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd6680d0d70 0x7fd6680d11d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:40.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.242+0000 7fd66f000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd6680d1b30 0x7fd6680d1fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:40.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.242+0000 7fd66f000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6680d4e70 con 0x7fd6680d1b30 2024-03-04T09:32:40.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.242+0000 7fd66f000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6680b3c40 con 0x7fd6680d0d70 2024-03-04T09:32:40.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.242+0000 7fd66f000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6680b3940 con 0x7fd66815e2e0 2024-03-04T09:32:40.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.242+0000 7fd66d000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd6680d1b30 0x7fd6680d1fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:40.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.243+0000 7fd66d000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd6680d1b30 0x7fd6680d1fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:47176/0 (socket says 172.21.15.6:47176) 2024-03-04T09:32:40.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.243+0000 7fd66d000640 1 -- 172.21.15.6:0/76028263 learned_addr learned my addr 172.21.15.6:0/76028263 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:32:40.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.243+0000 7fd667e00640 1 --2- 172.21.15.6:0/76028263 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd66815e2e0 0x7fd6680d0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:40.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.243+0000 7fd667400640 1 --2- 172.21.15.6:0/76028263 >> v2:172.21.15.146:3300/0 conn(0x7fd6680d0d70 0x7fd6680d11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:40.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.243+0000 7fd66d000640 1 -- 172.21.15.6:0/76028263 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd66815e2e0 msgr2=0x7fd6680d0830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:32:40.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.243+0000 7fd66d000640 1 --2- 172.21.15.6:0/76028263 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd66815e2e0 0x7fd6680d0830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:40.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.243+0000 7fd66d000640 1 -- 172.21.15.6:0/76028263 >> v2:172.21.15.146:3300/0 conn(0x7fd6680d0d70 msgr2=0x7fd6680d11d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:32:40.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.243+0000 7fd66d000640 1 --2- 172.21.15.6:0/76028263 >> v2:172.21.15.146:3300/0 conn(0x7fd6680d0d70 0x7fd6680d11d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:40.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.243+0000 7fd66d000640 1 -- 172.21.15.6:0/76028263 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6680cf580 con 0x7fd6680d1b30 2024-03-04T09:32:40.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.245+0000 7fd66d000640 1 --2- 172.21.15.6:0/76028263 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd6680d1b30 0x7fd6680d1fc0 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7fd6680b6b50 tx=0x7fd66002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f0294b468d80d32 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:40.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.245+0000 7fd666a00640 1 -- 172.21.15.6:0/76028263 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd660035020 con 0x7fd6680d1b30 2024-03-04T09:32:40.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.245+0000 7fd666a00640 1 -- 172.21.15.6:0/76028263 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd660031870 con 0x7fd6680d1b30 2024-03-04T09:32:40.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.245+0000 7fd666a00640 1 -- 172.21.15.6:0/76028263 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd660031c40 con 0x7fd6680d1b30 2024-03-04T09:32:40.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.246+0000 7fd66f000640 1 -- 172.21.15.6:0/76028263 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd6680d1b30 msgr2=0x7fd6680d1fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:40.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.246+0000 7fd66f000640 1 --2- 172.21.15.6:0/76028263 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd6680d1b30 0x7fd6680d1fc0 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7fd6680b6b50 tx=0x7fd66002b910 comp rx=0 tx=0).stop 2024-03-04T09:32:40.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.246+0000 7fd66f000640 1 -- 172.21.15.6:0/76028263 shutdown_connections 2024-03-04T09:32:40.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.246+0000 7fd66f000640 1 --2- 172.21.15.6:0/76028263 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd6680d1b30 0x7fd6680d1fc0 unknown :-1 s=CLOSED pgs=1414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:40.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.246+0000 7fd66f000640 1 --2- 172.21.15.6:0/76028263 >> v2:172.21.15.146:3300/0 conn(0x7fd6680d0d70 0x7fd6680d11d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:40.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.246+0000 7fd66f000640 1 --2- 172.21.15.6:0/76028263 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd66815e2e0 0x7fd6680d0830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:40.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.246+0000 7fd66f000640 1 -- 172.21.15.6:0/76028263 >> 172.21.15.6:0/76028263 conn(0x7fd6680d3f60 msgr2=0x7fd6680d4370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:32:40.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.246+0000 7fd66f000640 1 -- 172.21.15.6:0/76028263 shutdown_connections 2024-03-04T09:32:40.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.246+0000 7fd66f000640 1 -- 172.21.15.6:0/76028263 wait complete. 2024-03-04T09:32:40.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.249+0000 7fd66f000640 1 Processor -- start 2024-03-04T09:32:40.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.249+0000 7fd66f000640 1 -- start start 2024-03-04T09:32:40.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.250+0000 7fd66f000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd6680d0d70 0x7fd6680b7e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:40.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.251+0000 7fd66f000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd6680d1b30 0x7fd668148360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:40.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.251+0000 7fd66f000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd66815e2e0 0x7fd6680b8c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:40.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.251+0000 7fd66f000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6680cfe20 con 0x7fd6680d0d70 2024-03-04T09:32:40.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.251+0000 7fd66f000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6680cfb40 con 0x7fd66815e2e0 2024-03-04T09:32:40.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.251+0000 7fd66f000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6680d2500 con 0x7fd6680d1b30 2024-03-04T09:32:40.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.251+0000 7fd667e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd6680d0d70 0x7fd6680b7e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:40.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.251+0000 7fd667e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd6680d0d70 0x7fd6680b7e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:47190/0 (socket says 172.21.15.6:47190) 2024-03-04T09:32:40.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.251+0000 7fd667e00640 1 -- 172.21.15.6:0/724076813 learned_addr learned my addr 172.21.15.6:0/724076813 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:32:40.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.251+0000 7fd667400640 1 --2- 172.21.15.6:0/724076813 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd6680d1b30 0x7fd668148360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:40.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.251+0000 7fd66d000640 1 --2- 172.21.15.6:0/724076813 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd66815e2e0 0x7fd6680b8c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:40.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.251+0000 7fd667e00640 1 -- 172.21.15.6:0/724076813 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd6680d1b30 msgr2=0x7fd668148360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:40.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.251+0000 7fd667e00640 1 --2- 172.21.15.6:0/724076813 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd6680d1b30 0x7fd668148360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:40.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.251+0000 7fd667e00640 1 -- 172.21.15.6:0/724076813 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd66815e2e0 msgr2=0x7fd6680b8c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:40.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.251+0000 7fd667e00640 1 --2- 172.21.15.6:0/724076813 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd66815e2e0 0x7fd6680b8c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:40.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.251+0000 7fd667e00640 1 -- 172.21.15.6:0/724076813 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd668160f60 con 0x7fd6680d0d70 2024-03-04T09:32:40.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.252+0000 7fd667e00640 1 --2- 172.21.15.6:0/724076813 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd6680d0d70 0x7fd6680b7e10 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7fd658009d10 tx=0x7fd65800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:40.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.252+0000 7fd65fe00640 1 -- 172.21.15.6:0/724076813 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd658018020 con 0x7fd6680d0d70 2024-03-04T09:32:40.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.252+0000 7fd65fe00640 1 -- 172.21.15.6:0/724076813 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd658014480 con 0x7fd6680d0d70 2024-03-04T09:32:40.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.252+0000 7fd65fe00640 1 -- 172.21.15.6:0/724076813 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd658014830 con 0x7fd6680d0d70 2024-03-04T09:32:40.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.252+0000 7fd66f000640 1 -- 172.21.15.6:0/724076813 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6680ca770 con 0x7fd6680d0d70 2024-03-04T09:32:40.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.252+0000 7fd66f000640 1 -- 172.21.15.6:0/724076813 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6680b86e0 con 0x7fd6680d0d70 2024-03-04T09:32:40.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.254+0000 7fd66f000640 1 -- 172.21.15.6:0/724076813 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd66804f3a0 con 0x7fd6680d0d70 2024-03-04T09:32:40.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.255+0000 7fd65fe00640 1 -- 172.21.15.6:0/724076813 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd658005d10 con 0x7fd6680d0d70 2024-03-04T09:32:40.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.256+0000 7fd65fe00640 1 --2- 172.21.15.6:0/724076813 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd64806b520 0x7fd64806d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:40.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.256+0000 7fd65fe00640 1 -- 172.21.15.6:0/724076813 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5853+0+0 (secure 0 0 0) 0x7fd65808afe0 con 0x7fd6680d0d70 2024-03-04T09:32:40.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.257+0000 7fd667400640 1 --2- 172.21.15.6:0/724076813 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd64806b520 0x7fd64806d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:40.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.258+0000 7fd667400640 1 --2- 172.21.15.6:0/724076813 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd64806b520 0x7fd64806d9e0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fd6680b59c0 tx=0x7fd654008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:40.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.261+0000 7fd65fe00640 1 -- 172.21.15.6:0/724076813 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd658022030 con 0x7fd6680d0d70 2024-03-04T09:32:40.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.505+0000 7fd66f000640 1 -- 172.21.15.6:0/724076813 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd66815d9c0 con 0x7fd6680d0d70 2024-03-04T09:32:40.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.506+0000 7fd65fe00640 1 -- 172.21.15.6:0/724076813 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+4975 (secure 0 0 0) 0x7fd658010040 con 0x7fd6680d0d70 2024-03-04T09:32:40.503 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:32:40.503 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":84,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6535,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/3883122012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":3883122012},{"type":"v1","addr":"172.21.15.146:6835","nonce":3883122012}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":6559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/4060411493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":4060411493},{"type":"v1","addr":"172.21.15.146:6837","nonce":4060411493}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":82},{"gid":7812,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/679471398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":679471398},{"type":"v1","addr":"172.21.15.6:6833","nonce":679471398}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:31:21.114351+0000","modified":"2024-03-04T09:31:40.781474+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7815},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7815":{"gid":7815,"name":"a","rank":0,"incarnation":81,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/882667161","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":882667161},{"type":"v1","addr":"172.21.15.6:6835","nonce":882667161}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7815,"qdb_cluster":[7815]},"id":8}]} 2024-03-04T09:32:40.503 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 84 2024-03-04T09:32:40.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.508+0000 7fd66f000640 1 -- 172.21.15.6:0/724076813 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd64806b520 msgr2=0x7fd64806d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:40.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.508+0000 7fd66f000640 1 --2- 172.21.15.6:0/724076813 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd64806b520 0x7fd64806d9e0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fd6680b59c0 tx=0x7fd654008040 comp rx=0 tx=0).stop 2024-03-04T09:32:40.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.508+0000 7fd66f000640 1 -- 172.21.15.6:0/724076813 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd6680d0d70 msgr2=0x7fd6680b7e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:40.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.508+0000 7fd66f000640 1 --2- 172.21.15.6:0/724076813 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd6680d0d70 0x7fd6680b7e10 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7fd658009d10 tx=0x7fd65800c490 comp rx=0 tx=0).stop 2024-03-04T09:32:40.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.508+0000 7fd66f000640 1 -- 172.21.15.6:0/724076813 shutdown_connections 2024-03-04T09:32:40.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.508+0000 7fd66f000640 1 --2- 172.21.15.6:0/724076813 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd66815e2e0 0x7fd6680b8c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:40.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.508+0000 7fd66f000640 1 --2- 172.21.15.6:0/724076813 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd64806b520 0x7fd64806d9e0 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:40.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.508+0000 7fd66f000640 1 --2- 172.21.15.6:0/724076813 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd6680d1b30 0x7fd668148360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:40.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.508+0000 7fd66f000640 1 --2- 172.21.15.6:0/724076813 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd6680d0d70 0x7fd6680b7e10 unknown :-1 s=CLOSED pgs=1415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:40.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.508+0000 7fd66f000640 1 -- 172.21.15.6:0/724076813 >> 172.21.15.6:0/724076813 conn(0x7fd6680d3f60 msgr2=0x7fd66806d3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:32:40.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.508+0000 7fd66f000640 1 -- 172.21.15.6:0/724076813 shutdown_connections 2024-03-04T09:32:40.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.509+0000 7fd66f000640 1 -- 172.21.15.6:0/724076813 wait complete. 2024-03-04T09:32:40.532 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T09:32:40.581 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:32:40.586 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:32:40.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.779+0000 7fa8ec000640 1 Processor -- start 2024-03-04T09:32:40.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.779+0000 7fa8ec000640 1 -- start start 2024-03-04T09:32:40.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.781+0000 7fa8ec000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa8e415f310 0x7fa8e40cf890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:40.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.781+0000 7fa8ec000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa8e40cfdd0 0x7fa8e40d1c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:40.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.781+0000 7fa8ec000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8e40d02d0 con 0x7fa8e40d2170 2024-03-04T09:32:40.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.781+0000 7fa8ec000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8e40d2600 con 0x7fa8e415f310 2024-03-04T09:32:40.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.781+0000 7fa8ec000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8e40d05b0 con 0x7fa8e40cfdd0 2024-03-04T09:32:40.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.782+0000 7fa8e9600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa8e415f310 0x7fa8e40cf890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:40.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.782+0000 7fa8e9600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa8e415f310 0x7fa8e40cf890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:47202/0 (socket says 172.21.15.6:47202) 2024-03-04T09:32:40.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.782+0000 7fa8e9600640 1 -- 172.21.15.6:0/2174868548 learned_addr learned my addr 172.21.15.6:0/2174868548 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:32:40.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.782+0000 7fa8e8c00640 1 --2- 172.21.15.6:0/2174868548 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa8e40cfdd0 0x7fa8e40d1c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:40.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.782+0000 7fa8e9600640 1 -- 172.21.15.6:0/2174868548 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa8e40cfdd0 msgr2=0x7fa8e40d1c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:32:40.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.782+0000 7fa8e9600640 1 --2- 172.21.15.6:0/2174868548 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa8e40cfdd0 0x7fa8e40d1c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:40.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.782+0000 7fa8e9600640 1 -- 172.21.15.6:0/2174868548 >> v1:172.21.15.6:6789/0 conn(0x7fa8e40d2170 legacy=0x7fa8e40ce6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:40.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.782+0000 7fa8e9600640 1 -- 172.21.15.6:0/2174868548 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8e40d0890 con 0x7fa8e415f310 2024-03-04T09:32:40.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.784+0000 7fa8e9600640 1 --2- 172.21.15.6:0/2174868548 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa8e415f310 0x7fa8e40cf890 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7fa8d8009870 tx=0x7fa8d802ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f11e6b77c3cd4227 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:40.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.784+0000 7fa8e3e00640 1 -- 172.21.15.6:0/2174868548 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8d802f840 con 0x7fa8e415f310 2024-03-04T09:32:40.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.784+0000 7fa8e3e00640 1 -- 172.21.15.6:0/2174868548 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8d802f9e0 con 0x7fa8e415f310 2024-03-04T09:32:40.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.784+0000 7fa8e3e00640 1 -- 172.21.15.6:0/2174868548 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8d802fdb0 con 0x7fa8e415f310 2024-03-04T09:32:40.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.784+0000 7fa8ec000640 1 -- 172.21.15.6:0/2174868548 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa8e415f310 msgr2=0x7fa8e40cf890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:40.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.784+0000 7fa8ec000640 1 --2- 172.21.15.6:0/2174868548 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa8e415f310 0x7fa8e40cf890 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7fa8d8009870 tx=0x7fa8d802ba20 comp rx=0 tx=0).stop 2024-03-04T09:32:40.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.785+0000 7fa8ec000640 1 -- 172.21.15.6:0/2174868548 shutdown_connections 2024-03-04T09:32:40.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.785+0000 7fa8ec000640 1 --2- 172.21.15.6:0/2174868548 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa8e40cfdd0 0x7fa8e40d1c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:40.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.785+0000 7fa8ec000640 1 --2- 172.21.15.6:0/2174868548 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa8e415f310 0x7fa8e40cf890 unknown :-1 s=CLOSED pgs=1417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:40.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.785+0000 7fa8ec000640 1 -- 172.21.15.6:0/2174868548 >> 172.21.15.6:0/2174868548 conn(0x7fa8e409aa10 msgr2=0x7fa8e409ae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:32:40.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.785+0000 7fa8ec000640 1 -- 172.21.15.6:0/2174868548 shutdown_connections 2024-03-04T09:32:40.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.785+0000 7fa8ec000640 1 -- 172.21.15.6:0/2174868548 wait complete. 2024-03-04T09:32:40.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.786+0000 7fa8ec000640 1 Processor -- start 2024-03-04T09:32:40.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.787+0000 7fa8ec000640 1 -- start start 2024-03-04T09:32:40.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.787+0000 7fa8ec000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa8e40cfdd0 0x7fa8e41501d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:40.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.787+0000 7fa8ec000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa8e40d2170 0x7fa8e41f44a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:40.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.787+0000 7fa8ec000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa8e415f310 0x7fa8e40afa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:40.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.788+0000 7fa8ec000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8e415f6f0 con 0x7fa8e40cfdd0 2024-03-04T09:32:40.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.788+0000 7fa8ec000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8e40cc920 con 0x7fa8e415f310 2024-03-04T09:32:40.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.788+0000 7fa8ec000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8e415f870 con 0x7fa8e40d2170 2024-03-04T09:32:40.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.788+0000 7fa8e9600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa8e40cfdd0 0x7fa8e41501d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:40.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.788+0000 7fa8e9600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa8e40cfdd0 0x7fa8e41501d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:47204/0 (socket says 172.21.15.6:47204) 2024-03-04T09:32:40.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.788+0000 7fa8e9600640 1 -- 172.21.15.6:0/879936054 learned_addr learned my addr 172.21.15.6:0/879936054 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:32:40.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.788+0000 7fa8e8c00640 1 --2- 172.21.15.6:0/879936054 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa8e40d2170 0x7fa8e41f44a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:40.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.788+0000 7fa8e9600640 1 -- 172.21.15.6:0/879936054 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa8e40d2170 msgr2=0x7fa8e41f44a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:40.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.788+0000 7fa8e9600640 1 --2- 172.21.15.6:0/879936054 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa8e40d2170 0x7fa8e41f44a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:40.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.788+0000 7fa8e9600640 1 -- 172.21.15.6:0/879936054 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa8e415f310 msgr2=0x7fa8e40afa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:40.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.788+0000 7fa8e9600640 1 --2- 172.21.15.6:0/879936054 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa8e415f310 0x7fa8e40afa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:40.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.788+0000 7fa8e9600640 1 -- 172.21.15.6:0/879936054 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8e415c910 con 0x7fa8e40cfdd0 2024-03-04T09:32:40.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.789+0000 7fa8e9600640 1 --2- 172.21.15.6:0/879936054 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa8e40cfdd0 0x7fa8e41501d0 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7fa8d8000c00 tx=0x7fa8d8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:40.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.789+0000 7fa8e1600640 1 -- 172.21.15.6:0/879936054 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8d8037c90 con 0x7fa8e40cfdd0 2024-03-04T09:32:40.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.789+0000 7fa8e1600640 1 -- 172.21.15.6:0/879936054 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8d8046070 con 0x7fa8e40cfdd0 2024-03-04T09:32:40.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.789+0000 7fa8e1600640 1 -- 172.21.15.6:0/879936054 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8d8041640 con 0x7fa8e40cfdd0 2024-03-04T09:32:40.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.789+0000 7fa8ec000640 1 -- 172.21.15.6:0/879936054 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8e41f4df0 con 0x7fa8e40cfdd0 2024-03-04T09:32:40.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.789+0000 7fa8ec000640 1 -- 172.21.15.6:0/879936054 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8e41f53b0 con 0x7fa8e40cfdd0 2024-03-04T09:32:40.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.790+0000 7fa8ec000640 1 -- 172.21.15.6:0/879936054 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8e41f5580 con 0x7fa8e40cfdd0 2024-03-04T09:32:40.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.791+0000 7fa8e1600640 1 -- 172.21.15.6:0/879936054 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa8d8040070 con 0x7fa8e40cfdd0 2024-03-04T09:32:40.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.792+0000 7fa8e1600640 1 --2- 172.21.15.6:0/879936054 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa8c406b500 0x7fa8c406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:40.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.792+0000 7fa8e1600640 1 -- 172.21.15.6:0/879936054 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5853+0+0 (secure 0 0 0) 0x7fa8d80ac8a0 con 0x7fa8e40cfdd0 2024-03-04T09:32:40.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.793+0000 7fa8e8c00640 1 --2- 172.21.15.6:0/879936054 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa8c406b500 0x7fa8c406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:40.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.794+0000 7fa8e8c00640 1 --2- 172.21.15.6:0/879936054 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa8c406b500 0x7fa8c406d9c0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fa8d0006fd0 tx=0x7fa8d0008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:40.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.797+0000 7fa8e1600640 1 -- 172.21.15.6:0/879936054 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa8d8033030 con 0x7fa8e40cfdd0 2024-03-04T09:32:40.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.992+0000 7fa8ec000640 1 -- 172.21.15.6:0/879936054 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa8e404f3a0 con 0x7fa8e40cfdd0 2024-03-04T09:32:40.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.994+0000 7fa8e1600640 1 -- 172.21.15.6:0/879936054 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v115) v1 ==== 74+0+16859 (secure 0 0 0) 0x7fa8d807b680 con 0x7fa8e40cfdd0 2024-03-04T09:32:40.991 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:32:40.991 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":115,"fsid":"edce0578-5ae4-4402-b45a-552bdef84d7c","created":"2024-03-04T09:24:54.523780+0000","modified":"2024-03-04T09:32:03.117903+0000","last_up_change":"2024-03-04T09:25:07.089049+0000","last_in_change":"2024-03-04T09:24:57.982077+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T09:25:07.935198+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-04T09:31:19.018233+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-04T09:31:20.131544+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a96574f-81f1-4bc6-8463-757520b84a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6825","nonce":1364449401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6827","nonce":1364449401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6831","nonce":1364449401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6829","nonce":1364449401}]},"public_addr":"172.21.15.6:6825/1364449401","cluster_addr":"172.21.15.6:6827/1364449401","heartbeat_back_addr":"172.21.15.6:6831/1364449401","heartbeat_front_addr":"172.21.15.6:6829/1364449401","state":["exists","up"]},{"osd":1,"uuid":"7d79b9c1-6b87-476a-bc36-7db5ee64cd90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6809","nonce":2494103078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6811","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6813","nonce":2494103078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6821","nonce":2494103078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6817","nonce":2494103078}]},"public_addr":"172.21.15.6:6809/2494103078","cluster_addr":"172.21.15.6:6813/2494103078","heartbeat_back_addr":"172.21.15.6:6821/2494103078","heartbeat_front_addr":"172.21.15.6:6817/2494103078","state":["exists","up"]},{"osd":2,"uuid":"492331b9-4ee9-40b6-bacd-85d44cd6c45a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6812","nonce":1500066681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6816","nonce":1500066681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6824","nonce":1500066681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6820","nonce":1500066681}]},"public_addr":"172.21.15.6:6812/1500066681","cluster_addr":"172.21.15.6:6816/1500066681","heartbeat_back_addr":"172.21.15.6:6824/1500066681","heartbeat_front_addr":"172.21.15.6:6820/1500066681","state":["exists","up"]},{"osd":3,"uuid":"21bb2efe-9ba7-4bdc-bc31-180cd1102e21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6801","nonce":1558806123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6803","nonce":1558806123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6807","nonce":1558806123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6805","nonce":1558806123}]},"public_addr":"172.21.15.6:6801/1558806123","cluster_addr":"172.21.15.6:6803/1558806123","heartbeat_back_addr":"172.21.15.6:6807/1558806123","heartbeat_front_addr":"172.21.15.6:6805/1558806123","state":["exists","up"]},{"osd":4,"uuid":"6e2c4b71-19d9-4737-83d9-558705d8644a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6801","nonce":1526135780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6803","nonce":1526135780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6807","nonce":1526135780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6805","nonce":1526135780}]},"public_addr":"172.21.15.146:6801/1526135780","cluster_addr":"172.21.15.146:6803/1526135780","heartbeat_back_addr":"172.21.15.146:6807/1526135780","heartbeat_front_addr":"172.21.15.146:6805/1526135780","state":["exists","up"]},{"osd":5,"uuid":"fb69c525-f1be-4b7d-ab32-6c29b0d6fddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6809","nonce":460262507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6811","nonce":460262507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6815","nonce":460262507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6813","nonce":460262507}]},"public_addr":"172.21.15.146:6809/460262507","cluster_addr":"172.21.15.146:6811/460262507","heartbeat_back_addr":"172.21.15.146:6815/460262507","heartbeat_front_addr":"172.21.15.146:6813/460262507","state":["exists","up"]},{"osd":6,"uuid":"c32e7fed-370d-4401-8ce0-6a4335b0d168","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6823","nonce":1392897493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6827","nonce":1392897493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6831","nonce":1392897493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6829","nonce":1392897493}]},"public_addr":"172.21.15.146:6823/1392897493","cluster_addr":"172.21.15.146:6827/1392897493","heartbeat_back_addr":"172.21.15.146:6831/1392897493","heartbeat_front_addr":"172.21.15.146:6829/1392897493","state":["exists","up"]},{"osd":7,"uuid":"75a79b44-fa5f-426d-a7d3-71f21f78835c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6817","nonce":3268323156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6819","nonce":3268323156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6825","nonce":3268323156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6822","nonce":3268323156}]},"public_addr":"172.21.15.146:6817/3268323156","cluster_addr":"172.21.15.146:6819/3268323156","heartbeat_back_addr":"172.21.15.146:6825/3268323156","heartbeat_front_addr":"172.21.15.146:6822/3268323156","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.310437+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.295850+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.294674+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.526855+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.300023+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.595160+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.346046+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.558741+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"17.e","mappings":[{"from":0,"to":5}]},{"pgid":"17.26","mappings":[{"from":0,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.6:6834/2697931210":"2024-03-05T09:31:36.737386+0000","172.21.15.6:6832/2697931210":"2024-03-05T09:31:36.737386+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T09:32:40.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.996+0000 7fa8ec000640 1 -- 172.21.15.6:0/879936054 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa8c406b500 msgr2=0x7fa8c406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:40.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.996+0000 7fa8ec000640 1 --2- 172.21.15.6:0/879936054 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa8c406b500 0x7fa8c406d9c0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fa8d0006fd0 tx=0x7fa8d0008040 comp rx=0 tx=0).stop 2024-03-04T09:32:40.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.996+0000 7fa8ec000640 1 -- 172.21.15.6:0/879936054 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa8e40cfdd0 msgr2=0x7fa8e41501d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:40.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.996+0000 7fa8ec000640 1 --2- 172.21.15.6:0/879936054 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa8e40cfdd0 0x7fa8e41501d0 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7fa8d8000c00 tx=0x7fa8d8004730 comp rx=0 tx=0).stop 2024-03-04T09:32:40.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.996+0000 7fa8ec000640 1 -- 172.21.15.6:0/879936054 shutdown_connections 2024-03-04T09:32:40.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.996+0000 7fa8ec000640 1 --2- 172.21.15.6:0/879936054 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa8e415f310 0x7fa8e40afa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:40.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.996+0000 7fa8ec000640 1 --2- 172.21.15.6:0/879936054 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa8c406b500 0x7fa8c406d9c0 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:40.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.996+0000 7fa8ec000640 1 --2- 172.21.15.6:0/879936054 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa8e40d2170 0x7fa8e41f44a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:40.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.996+0000 7fa8ec000640 1 --2- 172.21.15.6:0/879936054 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa8e40cfdd0 0x7fa8e41501d0 unknown :-1 s=CLOSED pgs=1418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:40.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.996+0000 7fa8ec000640 1 -- 172.21.15.6:0/879936054 >> 172.21.15.6:0/879936054 conn(0x7fa8e409aa10 msgr2=0x7fa8e40ccd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:32:40.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.997+0000 7fa8ec000640 1 -- 172.21.15.6:0/879936054 shutdown_connections 2024-03-04T09:32:40.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:40.997+0000 7fa8ec000640 1 -- 172.21.15.6:0/879936054 wait complete. 2024-03-04T09:32:41.009 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-04T09:32:41.010 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:32:41.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.207+0000 7fedcaa00640 1 Processor -- start 2024-03-04T09:32:41.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.207+0000 7fedcaa00640 1 -- start start 2024-03-04T09:32:41.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.208+0000 7fedcaa00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fedc415e0b0 0x7fedc40cf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:41.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.209+0000 7fedcaa00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedc40b3710 con 0x7fedc40d4a10 2024-03-04T09:32:41.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.209+0000 7fedcaa00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedc40b39f0 con 0x7fedc40d0e50 2024-03-04T09:32:41.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.209+0000 7fedcaa00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedc40b09e0 con 0x7fedc415e0b0 2024-03-04T09:32:41.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.209+0000 7fedc8c00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fedc40d4a10 0x7fedc40d0740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:35104/0 (socket says 172.21.15.6:35104) 2024-03-04T09:32:41.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.209+0000 7fedc8c00640 1 -- 172.21.15.6:0/722264057 learned_addr learned my addr 172.21.15.6:0/722264057 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:32:41.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.209+0000 7fedc9600640 1 --2- 172.21.15.6:0/722264057 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fedc415e0b0 0x7fedc40cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:41.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.209+0000 7fedc3e00640 1 -- 172.21.15.6:0/722264057 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 676482318 0 0) 0x7fedc40b3710 con 0x7fedc40d4a10 2024-03-04T09:32:41.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.209+0000 7fedc3e00640 1 -- 172.21.15.6:0/722264057 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fedac003520 con 0x7fedc40d4a10 2024-03-04T09:32:41.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.210+0000 7fedc3e00640 1 -- 172.21.15.6:0/722264057 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3810559960 0 0) 0x7fedac003520 con 0x7fedc40d4a10 2024-03-04T09:32:41.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.210+0000 7fedc3e00640 1 -- 172.21.15.6:0/722264057 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fedc415e0b0 msgr2=0x7fedc40cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:32:41.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.210+0000 7fedc3e00640 1 --2- 172.21.15.6:0/722264057 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fedc415e0b0 0x7fedc40cf310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:41.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.210+0000 7fedc3e00640 1 -- 172.21.15.6:0/722264057 >> v1:172.21.15.146:6789/0 conn(0x7fedc40d0e50 legacy=0x7fedc40b44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:41.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.210+0000 7fedc3e00640 1 -- 172.21.15.6:0/722264057 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedc40b3cd0 con 0x7fedc40d4a10 2024-03-04T09:32:41.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.210+0000 7fedc3e00640 1 -- 172.21.15.6:0/722264057 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7fedb0001670 con 0x7fedc40d4a10 2024-03-04T09:32:41.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.210+0000 7fedc3e00640 1 -- 172.21.15.6:0/722264057 >> v1:172.21.15.6:6789/0 conn(0x7fedc40d4a10 legacy=0x7fedc40d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:41.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.210+0000 7fedc3e00640 1 --2- 172.21.15.6:0/722264057 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fedac0039e0 0x7fedac003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:41.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.210+0000 7fedc3e00640 1 --2- 172.21.15.6:0/722264057 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fedc415e0b0 0x7fedac004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:41.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.210+0000 7fedc3e00640 1 --2- 172.21.15.6:0/722264057 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fedac004d40 0x7fedac0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:41.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.210+0000 7fedc3e00640 1 -- 172.21.15.6:0/722264057 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedac00ba70 con 0x7fedac0039e0 2024-03-04T09:32:41.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.210+0000 7fedc3e00640 1 -- 172.21.15.6:0/722264057 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedac00bc20 con 0x7fedac004d40 2024-03-04T09:32:41.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.210+0000 7fedc3e00640 1 -- 172.21.15.6:0/722264057 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedac00be20 con 0x7fedc415e0b0 2024-03-04T09:32:41.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.210+0000 7fedc9600640 1 --2- 172.21.15.6:0/722264057 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fedac0039e0 0x7fedac003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:41.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.210+0000 7fedc8c00640 1 --2- 172.21.15.6:0/722264057 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fedc415e0b0 0x7fedac004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:41.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.211+0000 7fedc9600640 1 -- 172.21.15.6:0/722264057 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fedc415e0b0 msgr2=0x7fedac004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:32:41.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.211+0000 7fedc9600640 1 --2- 172.21.15.6:0/722264057 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fedc415e0b0 0x7fedac004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:41.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.211+0000 7fedc9600640 1 -- 172.21.15.6:0/722264057 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fedac004d40 msgr2=0x7fedac0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:32:41.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.211+0000 7fedc9600640 1 --2- 172.21.15.6:0/722264057 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fedac004d40 0x7fedac0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:41.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.211+0000 7fedc9600640 1 -- 172.21.15.6:0/722264057 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedac003520 con 0x7fedac0039e0 2024-03-04T09:32:41.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.211+0000 7fedc8c00640 1 --2- 172.21.15.6:0/722264057 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fedc415e0b0 0x7fedac004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:32:41.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.212+0000 7fedc9600640 1 --2- 172.21.15.6:0/722264057 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fedac0039e0 0x7fedac003e00 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7fedb8009f90 tx=0x7fedac005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=188ddc5430c148c4 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:41.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.212+0000 7fedc3e00640 1 -- 172.21.15.6:0/722264057 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedb8039070 con 0x7fedac0039e0 2024-03-04T09:32:41.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.212+0000 7fedc3e00640 1 -- 172.21.15.6:0/722264057 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedb802cbd0 con 0x7fedac0039e0 2024-03-04T09:32:41.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.212+0000 7fedc3e00640 1 -- 172.21.15.6:0/722264057 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedb8009c10 con 0x7fedac0039e0 2024-03-04T09:32:41.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.213+0000 7fedcaa00640 1 -- 172.21.15.6:0/722264057 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fedac0039e0 msgr2=0x7fedac003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:41.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.213+0000 7fedcaa00640 1 --2- 172.21.15.6:0/722264057 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fedac0039e0 0x7fedac003e00 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7fedb8009f90 tx=0x7fedac005830 comp rx=0 tx=0).stop 2024-03-04T09:32:41.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.213+0000 7fedca000640 1 -- 172.21.15.6:0/722264057 reap_dead start 2024-03-04T09:32:41.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.213+0000 7fedcaa00640 1 -- 172.21.15.6:0/722264057 shutdown_connections 2024-03-04T09:32:41.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.213+0000 7fedcaa00640 1 -- 172.21.15.6:0/722264057 >> 172.21.15.6:0/722264057 conn(0x7fedc40d3d30 msgr2=0x7fedc40d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:32:41.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.213+0000 7fedcaa00640 1 -- 172.21.15.6:0/722264057 shutdown_connections 2024-03-04T09:32:41.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.213+0000 7fedcaa00640 1 -- 172.21.15.6:0/722264057 wait complete. 2024-03-04T09:32:41.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.215+0000 7fedcaa00640 1 Processor -- start 2024-03-04T09:32:41.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.215+0000 7fedcaa00640 1 -- start start 2024-03-04T09:32:41.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.216+0000 7fedcaa00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fedac0039e0 0x7fedc41f3130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:41.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.216+0000 7fedcaa00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fedc41532f0 0x7fedc40b6920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:41.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.216+0000 7fedcaa00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fedc41609b0 0x7fedc4160e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:41.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.216+0000 7fedcaa00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedc40cfc60 con 0x7fedac0039e0 2024-03-04T09:32:41.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.216+0000 7fedcaa00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedc415e490 con 0x7fedc41532f0 2024-03-04T09:32:41.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.216+0000 7fedcaa00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedc40cf310 con 0x7fedc41609b0 2024-03-04T09:32:41.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.216+0000 7fedc9600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fedac0039e0 0x7fedc41f3130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:41.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.216+0000 7fedc9600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fedac0039e0 0x7fedc41f3130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:47214/0 (socket says 172.21.15.6:47214) 2024-03-04T09:32:41.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.216+0000 7fedc9600640 1 -- 172.21.15.6:0/2122070761 learned_addr learned my addr 172.21.15.6:0/2122070761 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:32:41.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.216+0000 7fedc9600640 1 -- 172.21.15.6:0/2122070761 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fedc41609b0 msgr2=0x7fedc4160e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:41.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.216+0000 7fedc9600640 1 --2- 172.21.15.6:0/2122070761 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fedc41609b0 0x7fedc4160e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:41.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.216+0000 7fedc9600640 1 -- 172.21.15.6:0/2122070761 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fedc41532f0 msgr2=0x7fedc40b6920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:41.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.216+0000 7fedc9600640 1 --2- 172.21.15.6:0/2122070761 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fedc41532f0 0x7fedc40b6920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:41.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.216+0000 7fedc9600640 1 -- 172.21.15.6:0/2122070761 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedc41f3670 con 0x7fedac0039e0 2024-03-04T09:32:41.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.217+0000 7fedc9600640 1 --2- 172.21.15.6:0/2122070761 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fedac0039e0 0x7fedc41f3130 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7fedb800e230 tx=0x7fedb8009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:41.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.217+0000 7fedc1600640 1 -- 172.21.15.6:0/2122070761 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedb8039050 con 0x7fedac0039e0 2024-03-04T09:32:41.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.217+0000 7fedc1600640 1 -- 172.21.15.6:0/2122070761 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedb803a040 con 0x7fedac0039e0 2024-03-04T09:32:41.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.217+0000 7fedc1600640 1 -- 172.21.15.6:0/2122070761 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedb8009c10 con 0x7fedac0039e0 2024-03-04T09:32:41.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.217+0000 7fedcaa00640 1 -- 172.21.15.6:0/2122070761 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedc40b7170 con 0x7fedac0039e0 2024-03-04T09:32:41.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.217+0000 7fedcaa00640 1 -- 172.21.15.6:0/2122070761 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedc40b7730 con 0x7fedac0039e0 2024-03-04T09:32:41.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.218+0000 7fedcaa00640 1 -- 172.21.15.6:0/2122070761 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedc404f330 con 0x7fedac0039e0 2024-03-04T09:32:41.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.219+0000 7fedc1600640 1 -- 172.21.15.6:0/2122070761 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fedb8047020 con 0x7fedac0039e0 2024-03-04T09:32:41.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.220+0000 7fedc1600640 1 --2- 172.21.15.6:0/2122070761 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7feda406b4d0 0x7feda406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:41.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.220+0000 7fedc1600640 1 -- 172.21.15.6:0/2122070761 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5853+0+0 (secure 0 0 0) 0x7fedb80ac510 con 0x7fedac0039e0 2024-03-04T09:32:41.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.221+0000 7fedc8c00640 1 --2- 172.21.15.6:0/2122070761 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7feda406b4d0 0x7feda406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:41.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.222+0000 7fedc8c00640 1 --2- 172.21.15.6:0/2122070761 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7feda406b4d0 0x7feda406d990 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fedb00017e0 tx=0x7fedb0009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:41.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.225+0000 7fedc1600640 1 -- 172.21.15.6:0/2122070761 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fedb804e020 con 0x7fedac0039e0 2024-03-04T09:32:41.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.463+0000 7fedcaa00640 1 -- 172.21.15.6:0/2122070761 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fedc40d2b80 con 0x7fedac0039e0 2024-03-04T09:32:41.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.464+0000 7fedc1600640 1 -- 172.21.15.6:0/2122070761 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+4975 (secure 0 0 0) 0x7fedc41f3670 con 0x7fedac0039e0 2024-03-04T09:32:41.461 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:32:41.461 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":84,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6535,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/3883122012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":3883122012},{"type":"v1","addr":"172.21.15.146:6835","nonce":3883122012}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":6559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/4060411493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":4060411493},{"type":"v1","addr":"172.21.15.146:6837","nonce":4060411493}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":82},{"gid":7812,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/679471398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":679471398},{"type":"v1","addr":"172.21.15.6:6833","nonce":679471398}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:31:21.114351+0000","modified":"2024-03-04T09:31:40.781474+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7815},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7815":{"gid":7815,"name":"a","rank":0,"incarnation":81,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/882667161","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":882667161},{"type":"v1","addr":"172.21.15.6:6835","nonce":882667161}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7815,"qdb_cluster":[7815]},"id":8}]} 2024-03-04T09:32:41.462 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 84 2024-03-04T09:32:41.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.467+0000 7fedcaa00640 1 -- 172.21.15.6:0/2122070761 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7feda406b4d0 msgr2=0x7feda406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:41.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.467+0000 7fedcaa00640 1 --2- 172.21.15.6:0/2122070761 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7feda406b4d0 0x7feda406d990 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fedb00017e0 tx=0x7fedb0009040 comp rx=0 tx=0).stop 2024-03-04T09:32:41.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.467+0000 7fedcaa00640 1 -- 172.21.15.6:0/2122070761 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fedac0039e0 msgr2=0x7fedc41f3130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:41.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.467+0000 7fedcaa00640 1 --2- 172.21.15.6:0/2122070761 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fedac0039e0 0x7fedc41f3130 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7fedb800e230 tx=0x7fedb8009750 comp rx=0 tx=0).stop 2024-03-04T09:32:41.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.467+0000 7fedcaa00640 1 -- 172.21.15.6:0/2122070761 shutdown_connections 2024-03-04T09:32:41.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.467+0000 7fedcaa00640 1 --2- 172.21.15.6:0/2122070761 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7feda406b4d0 0x7feda406d990 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:41.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.467+0000 7fedcaa00640 1 --2- 172.21.15.6:0/2122070761 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fedc41609b0 0x7fedc4160e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:41.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.467+0000 7fedcaa00640 1 --2- 172.21.15.6:0/2122070761 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fedc41532f0 0x7fedc40b6920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:41.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.467+0000 7fedcaa00640 1 --2- 172.21.15.6:0/2122070761 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fedac0039e0 0x7fedc41f3130 unknown :-1 s=CLOSED pgs=1421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:41.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.467+0000 7fedcaa00640 1 -- 172.21.15.6:0/2122070761 >> 172.21.15.6:0/2122070761 conn(0x7fedc40d3d30 msgr2=0x7fedc40b61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:32:41.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.468+0000 7fedcaa00640 1 -- 172.21.15.6:0/2122070761 shutdown_connections 2024-03-04T09:32:41.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.468+0000 7fedcaa00640 1 -- 172.21.15.6:0/2122070761 wait complete. 2024-03-04T09:32:41.483 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:32:41.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.729+0000 7f1bbe600640 1 Processor -- start 2024-03-04T09:32:41.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.730+0000 7f1bbe600640 1 -- start start 2024-03-04T09:32:41.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.731+0000 7f1bbe600640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f1bb8071b60 0x7f1bb80b3f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:41.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.731+0000 7f1bbe600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f1bb80b4a70 0x7f1bb815b320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:41.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.731+0000 7f1bbe600640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bb80b31d0 con 0x7f1bb80cede0 2024-03-04T09:32:41.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.731+0000 7f1bbe600640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1bb80b4ee0 con 0x7f1bb80b4a70 2024-03-04T09:32:41.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.731+0000 7f1bbe600640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1bb80b3b00 con 0x7f1bb8071b60 2024-03-04T09:32:41.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.731+0000 7f1bbd200640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f1bb80cede0 0x7f1bb8071450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:35108/0 (socket says 172.21.15.6:35108) 2024-03-04T09:32:41.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.731+0000 7f1bbd200640 1 -- 172.21.15.6:0/2758037020 learned_addr learned my addr 172.21.15.6:0/2758037020 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:32:41.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.731+0000 7f1bbdc00640 1 --2- 172.21.15.6:0/2758037020 >> v2:172.21.15.6:3300/0 conn(0x7f1bb80b4a70 0x7f1bb815b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:41.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.732+0000 7f1bbdc00640 1 -- 172.21.15.6:0/2758037020 >> v2:172.21.15.146:3300/0 conn(0x7f1bb8071b60 msgr2=0x7f1bb80b3f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:32:41.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.732+0000 7f1bbdc00640 1 --2- 172.21.15.6:0/2758037020 >> v2:172.21.15.146:3300/0 conn(0x7f1bb8071b60 0x7f1bb80b3f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:41.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.732+0000 7f1bbdc00640 1 -- 172.21.15.6:0/2758037020 >> v1:172.21.15.6:6789/0 conn(0x7f1bb80cede0 legacy=0x7f1bb8071450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:41.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.732+0000 7f1bbdc00640 1 -- 172.21.15.6:0/2758037020 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bb80b34b0 con 0x7f1bb80b4a70 2024-03-04T09:32:41.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.733+0000 7f1bbdc00640 1 --2- 172.21.15.6:0/2758037020 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1bb80b4a70 0x7f1bb815b320 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7f1bb80711d0 tx=0x7f1bb002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5716eb52e70cdeec server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:41.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.733+0000 7f1bb7400640 1 -- 172.21.15.6:0/2758037020 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bb0039070 con 0x7f1bb80b4a70 2024-03-04T09:32:41.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.733+0000 7f1bb7400640 1 -- 172.21.15.6:0/2758037020 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bb002cbf0 con 0x7f1bb80b4a70 2024-03-04T09:32:41.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.733+0000 7f1bb7400640 1 -- 172.21.15.6:0/2758037020 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bb0034440 con 0x7f1bb80b4a70 2024-03-04T09:32:41.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.734+0000 7f1bbe600640 1 -- 172.21.15.6:0/2758037020 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1bb80b4a70 msgr2=0x7f1bb815b320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:41.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.734+0000 7f1bbe600640 1 --2- 172.21.15.6:0/2758037020 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1bb80b4a70 0x7f1bb815b320 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7f1bb80711d0 tx=0x7f1bb002be40 comp rx=0 tx=0).stop 2024-03-04T09:32:41.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.734+0000 7f1bbe600640 1 -- 172.21.15.6:0/2758037020 shutdown_connections 2024-03-04T09:32:41.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.734+0000 7f1bbe600640 1 --2- 172.21.15.6:0/2758037020 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1bb80b4a70 0x7f1bb815b320 unknown :-1 s=CLOSED pgs=1423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:41.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.734+0000 7f1bbe600640 1 --2- 172.21.15.6:0/2758037020 >> v2:172.21.15.146:3300/0 conn(0x7f1bb8071b60 0x7f1bb80b3f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:41.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.734+0000 7f1bbe600640 1 -- 172.21.15.6:0/2758037020 >> 172.21.15.6:0/2758037020 conn(0x7f1bb806b3d0 msgr2=0x7f1bb806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:32:41.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.734+0000 7f1bbe600640 1 -- 172.21.15.6:0/2758037020 shutdown_connections 2024-03-04T09:32:41.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.734+0000 7f1bbe600640 1 -- 172.21.15.6:0/2758037020 wait complete. 2024-03-04T09:32:41.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.736+0000 7f1bbe600640 1 Processor -- start 2024-03-04T09:32:41.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.736+0000 7f1bbe600640 1 -- start start 2024-03-04T09:32:41.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.737+0000 7f1bbe600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1bb8071b60 0x7f1bb8156010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:41.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.737+0000 7f1bbe600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1bb80b4a70 0x7f1bb814b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:41.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.737+0000 7f1bbe600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1bb80cede0 0x7f1bb81579f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:41.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.737+0000 7f1bbe600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bb80cf1c0 con 0x7f1bb80cede0 2024-03-04T09:32:41.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.737+0000 7f1bbe600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bb80b2f70 con 0x7f1bb8071b60 2024-03-04T09:32:41.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.737+0000 7f1bbe600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bb80cc950 con 0x7f1bb80b4a70 2024-03-04T09:32:41.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.737+0000 7f1bbdc00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1bb80cede0 0x7f1bb81579f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:41.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.737+0000 7f1bbdc00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1bb80cede0 0x7f1bb81579f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:47232/0 (socket says 172.21.15.6:47232) 2024-03-04T09:32:41.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.737+0000 7f1bbdc00640 1 -- 172.21.15.6:0/1742190697 learned_addr learned my addr 172.21.15.6:0/1742190697 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:32:41.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.738+0000 7f1bbd200640 1 --2- 172.21.15.6:0/1742190697 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1bb8071b60 0x7f1bb8156010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:41.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.738+0000 7f1bb7e00640 1 --2- 172.21.15.6:0/1742190697 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1bb80b4a70 0x7f1bb814b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:41.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.738+0000 7f1bbdc00640 1 -- 172.21.15.6:0/1742190697 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1bb80b4a70 msgr2=0x7f1bb814b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:41.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.738+0000 7f1bbdc00640 1 --2- 172.21.15.6:0/1742190697 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1bb80b4a70 0x7f1bb814b340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:41.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.738+0000 7f1bbdc00640 1 -- 172.21.15.6:0/1742190697 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1bb8071b60 msgr2=0x7f1bb8156010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:41.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.738+0000 7f1bbdc00640 1 --2- 172.21.15.6:0/1742190697 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1bb8071b60 0x7f1bb8156010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:41.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.738+0000 7f1bbdc00640 1 -- 172.21.15.6:0/1742190697 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bb8154f70 con 0x7f1bb80cede0 2024-03-04T09:32:41.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.738+0000 7f1bbdc00640 1 --2- 172.21.15.6:0/1742190697 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1bb80cede0 0x7f1bb81579f0 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7f1bb0037040 tx=0x7f1bb0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:41.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.739+0000 7f1bb4c00640 1 -- 172.21.15.6:0/1742190697 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bb0039050 con 0x7f1bb80cede0 2024-03-04T09:32:41.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.739+0000 7f1bb4c00640 1 -- 172.21.15.6:0/1742190697 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bb0045070 con 0x7f1bb80cede0 2024-03-04T09:32:41.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.739+0000 7f1bb4c00640 1 -- 172.21.15.6:0/1742190697 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bb003e810 con 0x7f1bb80cede0 2024-03-04T09:32:41.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.739+0000 7f1bbe600640 1 -- 172.21.15.6:0/1742190697 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bb8155250 con 0x7f1bb80cede0 2024-03-04T09:32:41.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.739+0000 7f1bbe600640 1 -- 172.21.15.6:0/1742190697 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bb8155af0 con 0x7f1bb80cede0 2024-03-04T09:32:41.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.740+0000 7f1bbe600640 1 -- 172.21.15.6:0/1742190697 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bb8155640 con 0x7f1bb80cede0 2024-03-04T09:32:41.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.741+0000 7f1bb4c00640 1 -- 172.21.15.6:0/1742190697 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1bb003e9b0 con 0x7f1bb80cede0 2024-03-04T09:32:41.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.742+0000 7f1bb4c00640 1 --2- 172.21.15.6:0/1742190697 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f1b9806b4d0 0x7f1b9806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:41.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.742+0000 7f1bb4c00640 1 -- 172.21.15.6:0/1742190697 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f1bb00ad250 con 0x7f1bb80cede0 2024-03-04T09:32:41.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.743+0000 7f1bbd200640 1 --2- 172.21.15.6:0/1742190697 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f1b9806b4d0 0x7f1b9806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:41.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.744+0000 7f1bbd200640 1 --2- 172.21.15.6:0/1742190697 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f1b9806b4d0 0x7f1b9806d990 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f1bac000f60 tx=0x7f1bac009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:41.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:41.751+0000 7f1bb4c00640 1 -- 172.21.15.6:0/1742190697 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1bb0032e20 con 0x7f1bb80cede0 2024-03-04T09:32:42.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.055+0000 7f1bbe600640 1 -- 172.21.15.6:0/1742190697 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1bb8155cf0 con 0x7f1bb80cede0 2024-03-04T09:32:42.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.056+0000 7f1bb4c00640 1 -- 172.21.15.6:0/1742190697 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+4975 (secure 0 0 0) 0x7f1bb8155cf0 con 0x7f1bb80cede0 2024-03-04T09:32:42.053 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:32:42.053 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":84,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6535,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/3883122012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":3883122012},{"type":"v1","addr":"172.21.15.146:6835","nonce":3883122012}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":6559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/4060411493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":4060411493},{"type":"v1","addr":"172.21.15.146:6837","nonce":4060411493}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":82},{"gid":7812,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/679471398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":679471398},{"type":"v1","addr":"172.21.15.6:6833","nonce":679471398}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:31:21.114351+0000","modified":"2024-03-04T09:31:40.781474+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7815},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7815":{"gid":7815,"name":"a","rank":0,"incarnation":81,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/882667161","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":882667161},{"type":"v1","addr":"172.21.15.6:6835","nonce":882667161}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7815,"qdb_cluster":[7815]},"id":8}]} 2024-03-04T09:32:42.053 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 84 2024-03-04T09:32:42.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.059+0000 7f1bbe600640 1 -- 172.21.15.6:0/1742190697 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f1b9806b4d0 msgr2=0x7f1b9806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:42.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.059+0000 7f1bbe600640 1 --2- 172.21.15.6:0/1742190697 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f1b9806b4d0 0x7f1b9806d990 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f1bac000f60 tx=0x7f1bac009040 comp rx=0 tx=0).stop 2024-03-04T09:32:42.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.059+0000 7f1bbe600640 1 -- 172.21.15.6:0/1742190697 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1bb80cede0 msgr2=0x7f1bb81579f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:42.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.059+0000 7f1bbe600640 1 --2- 172.21.15.6:0/1742190697 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1bb80cede0 0x7f1bb81579f0 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7f1bb0037040 tx=0x7f1bb0004a40 comp rx=0 tx=0).stop 2024-03-04T09:32:42.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.059+0000 7f1bbe600640 1 -- 172.21.15.6:0/1742190697 shutdown_connections 2024-03-04T09:32:42.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.059+0000 7f1bbe600640 1 --2- 172.21.15.6:0/1742190697 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1bb80cede0 0x7f1bb81579f0 unknown :-1 s=CLOSED pgs=1424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:42.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.059+0000 7f1bbe600640 1 --2- 172.21.15.6:0/1742190697 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f1b9806b4d0 0x7f1b9806d990 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:42.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.059+0000 7f1bbe600640 1 --2- 172.21.15.6:0/1742190697 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1bb80b4a70 0x7f1bb814b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:42.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.059+0000 7f1bbe600640 1 --2- 172.21.15.6:0/1742190697 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1bb8071b60 0x7f1bb8156010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:42.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.059+0000 7f1bbe600640 1 -- 172.21.15.6:0/1742190697 >> 172.21.15.6:0/1742190697 conn(0x7f1bb806b3d0 msgr2=0x7f1bb8154720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:32:42.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.059+0000 7f1bbe600640 1 -- 172.21.15.6:0/1742190697 shutdown_connections 2024-03-04T09:32:42.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.059+0000 7f1bbe600640 1 -- 172.21.15.6:0/1742190697 wait complete. 2024-03-04T09:32:42.075 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T09:32:42.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.291+0000 7fb291600640 1 Processor -- start 2024-03-04T09:32:42.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.291+0000 7fb291600640 1 -- start start 2024-03-04T09:32:42.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.293+0000 7fb291600640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fb28c0af580 0x7fb28c15c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:42.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.293+0000 7fb291600640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb28c15cc40 con 0x7fb28c0af1a0 2024-03-04T09:32:42.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.293+0000 7fb291600640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb28c15cf20 con 0x7fb28c0cdc20 2024-03-04T09:32:42.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.293+0000 7fb291600640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb28c15d200 con 0x7fb28c0af580 2024-03-04T09:32:42.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.293+0000 7fb28aa00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fb28c0af1a0 0x7fb28c0b4c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:35124/0 (socket says 172.21.15.6:35124) 2024-03-04T09:32:42.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.293+0000 7fb28aa00640 1 -- 172.21.15.6:0/276119517 learned_addr learned my addr 172.21.15.6:0/276119517 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:32:42.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.294+0000 7fb28a000640 1 --2- 172.21.15.6:0/276119517 >> v2:172.21.15.146:3300/0 conn(0x7fb28c0af580 0x7fb28c15c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:42.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.294+0000 7fb289600640 1 -- 172.21.15.6:0/276119517 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 19265613 0 0) 0x7fb28c15cc40 con 0x7fb28c0af1a0 2024-03-04T09:32:42.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.294+0000 7fb289600640 1 -- 172.21.15.6:0/276119517 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb270003520 con 0x7fb28c0af1a0 2024-03-04T09:32:42.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.294+0000 7fb289600640 1 -- 172.21.15.6:0/276119517 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1783197878 0 0) 0x7fb270003520 con 0x7fb28c0af1a0 2024-03-04T09:32:42.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.294+0000 7fb289600640 1 -- 172.21.15.6:0/276119517 >> v2:172.21.15.146:3300/0 conn(0x7fb28c0af580 msgr2=0x7fb28c15c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:32:42.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.294+0000 7fb289600640 1 --2- 172.21.15.6:0/276119517 >> v2:172.21.15.146:3300/0 conn(0x7fb28c0af580 0x7fb28c15c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:42.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.294+0000 7fb289600640 1 -- 172.21.15.6:0/276119517 >> v1:172.21.15.146:6789/0 conn(0x7fb28c0cdc20 legacy=0x7fb28c0b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:42.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.294+0000 7fb289600640 1 -- 172.21.15.6:0/276119517 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb28c15d4e0 con 0x7fb28c0af1a0 2024-03-04T09:32:42.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.294+0000 7fb289600640 1 -- 172.21.15.6:0/276119517 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7fb27c001670 con 0x7fb28c0af1a0 2024-03-04T09:32:42.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.294+0000 7fb289600640 1 -- 172.21.15.6:0/276119517 >> v1:172.21.15.6:6789/0 conn(0x7fb28c0af1a0 legacy=0x7fb28c0b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:42.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.294+0000 7fb289600640 1 --2- 172.21.15.6:0/276119517 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb270003a20 0x7fb270003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:42.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.294+0000 7fb289600640 1 --2- 172.21.15.6:0/276119517 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb270004800 0x7fb270004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:42.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.295+0000 7fb289600640 1 --2- 172.21.15.6:0/276119517 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb270005210 0x7fb27000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:42.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.295+0000 7fb289600640 1 -- 172.21.15.6:0/276119517 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb28c15d200 con 0x7fb270003a20 2024-03-04T09:32:42.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.295+0000 7fb289600640 1 -- 172.21.15.6:0/276119517 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb27000c140 con 0x7fb270005210 2024-03-04T09:32:42.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.295+0000 7fb289600640 1 -- 172.21.15.6:0/276119517 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb27000c340 con 0x7fb270004800 2024-03-04T09:32:42.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.295+0000 7fb28aa00640 1 --2- 172.21.15.6:0/276119517 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb270003a20 0x7fb270003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:42.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.295+0000 7fb28a000640 1 --2- 172.21.15.6:0/276119517 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb270004800 0x7fb270004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:42.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.295+0000 7fb28b400640 1 --2- 172.21.15.6:0/276119517 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb270005210 0x7fb27000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:42.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.295+0000 7fb28aa00640 1 -- 172.21.15.6:0/276119517 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb270004800 msgr2=0x7fb270004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:32:42.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.295+0000 7fb28aa00640 1 --2- 172.21.15.6:0/276119517 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb270004800 0x7fb270004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:42.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.295+0000 7fb28aa00640 1 -- 172.21.15.6:0/276119517 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb270005210 msgr2=0x7fb27000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:32:42.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.295+0000 7fb28aa00640 1 --2- 172.21.15.6:0/276119517 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb270005210 0x7fb27000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:42.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.295+0000 7fb28aa00640 1 -- 172.21.15.6:0/276119517 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb270003520 con 0x7fb270003a20 2024-03-04T09:32:42.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.295+0000 7fb28b400640 1 --2- 172.21.15.6:0/276119517 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb270005210 0x7fb27000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:32:42.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.295+0000 7fb28b400640 1 -- 172.21.15.6:0/276119517 reap_dead start 2024-03-04T09:32:42.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.296+0000 7fb28aa00640 1 --2- 172.21.15.6:0/276119517 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb270003a20 0x7fb270003e00 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7fb27c00c160 tx=0x7fb270005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=155ec2bdab0bdc7f server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:42.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.297+0000 7fb289600640 1 -- 172.21.15.6:0/276119517 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb27c007c80 con 0x7fb270003a20 2024-03-04T09:32:42.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.297+0000 7fb289600640 1 -- 172.21.15.6:0/276119517 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb27c030070 con 0x7fb270003a20 2024-03-04T09:32:42.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.297+0000 7fb289600640 1 -- 172.21.15.6:0/276119517 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb27c005d70 con 0x7fb270003a20 2024-03-04T09:32:42.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.297+0000 7fb291600640 1 -- 172.21.15.6:0/276119517 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb270003a20 msgr2=0x7fb270003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:42.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.297+0000 7fb291600640 1 --2- 172.21.15.6:0/276119517 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb270003a20 0x7fb270003e00 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7fb27c00c160 tx=0x7fb270005830 comp rx=0 tx=0).stop 2024-03-04T09:32:42.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.297+0000 7fb291600640 1 -- 172.21.15.6:0/276119517 shutdown_connections 2024-03-04T09:32:42.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.297+0000 7fb291600640 1 --2- 172.21.15.6:0/276119517 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb270003a20 0x7fb270003e00 unknown :-1 s=CLOSED pgs=1426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:42.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.297+0000 7fb291600640 1 -- 172.21.15.6:0/276119517 >> 172.21.15.6:0/276119517 conn(0x7fb28c06afb0 msgr2=0x7fb28c06b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:32:42.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.297+0000 7fb291600640 1 -- 172.21.15.6:0/276119517 shutdown_connections 2024-03-04T09:32:42.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.297+0000 7fb291600640 1 -- 172.21.15.6:0/276119517 wait complete. 2024-03-04T09:32:42.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.299+0000 7fb291600640 1 Processor -- start 2024-03-04T09:32:42.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.299+0000 7fb291600640 1 -- start start 2024-03-04T09:32:42.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.300+0000 7fb291600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb270003a20 0x7fb28c15a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:42.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.300+0000 7fb291600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb28c0cac00 0x7fb28c0cb080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:42.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.300+0000 7fb291600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb28c151300 0x7fb28c151780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:42.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.300+0000 7fb291600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb28c15c990 con 0x7fb270003a20 2024-03-04T09:32:42.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.300+0000 7fb291600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb28c15c810 con 0x7fb28c151300 2024-03-04T09:32:42.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.300+0000 7fb291600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb28c0b4c20 con 0x7fb28c0cac00 2024-03-04T09:32:42.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.301+0000 7fb28aa00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb270003a20 0x7fb28c15a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:42.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.301+0000 7fb28aa00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb270003a20 0x7fb28c15a410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:47256/0 (socket says 172.21.15.6:47256) 2024-03-04T09:32:42.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.301+0000 7fb28aa00640 1 -- 172.21.15.6:0/586353496 learned_addr learned my addr 172.21.15.6:0/586353496 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:32:42.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.301+0000 7fb28a000640 1 --2- 172.21.15.6:0/586353496 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb28c0cac00 0x7fb28c0cb080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:42.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.301+0000 7fb28b400640 1 --2- 172.21.15.6:0/586353496 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb28c151300 0x7fb28c151780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:42.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.301+0000 7fb28aa00640 1 -- 172.21.15.6:0/586353496 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb28c0cac00 msgr2=0x7fb28c0cb080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:42.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.301+0000 7fb28aa00640 1 --2- 172.21.15.6:0/586353496 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb28c0cac00 0x7fb28c0cb080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:42.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.301+0000 7fb28aa00640 1 -- 172.21.15.6:0/586353496 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb28c151300 msgr2=0x7fb28c151780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:42.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.301+0000 7fb28aa00640 1 --2- 172.21.15.6:0/586353496 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb28c151300 0x7fb28c151780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:42.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.301+0000 7fb28aa00640 1 -- 172.21.15.6:0/586353496 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb28c151cc0 con 0x7fb270003a20 2024-03-04T09:32:42.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.301+0000 7fb28aa00640 1 --2- 172.21.15.6:0/586353496 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb270003a20 0x7fb28c15a410 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 crypto rx=0x7fb27c00c280 tx=0x7fb27c02e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:42.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.302+0000 7fb282a00640 1 -- 172.21.15.6:0/586353496 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb27c046070 con 0x7fb270003a20 2024-03-04T09:32:42.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.302+0000 7fb282a00640 1 -- 172.21.15.6:0/586353496 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb27c0049e0 con 0x7fb270003a20 2024-03-04T09:32:42.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.302+0000 7fb291600640 1 -- 172.21.15.6:0/586353496 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb28c0c94c0 con 0x7fb270003a20 2024-03-04T09:32:42.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.302+0000 7fb282a00640 1 -- 172.21.15.6:0/586353496 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb27c004dd0 con 0x7fb270003a20 2024-03-04T09:32:42.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.302+0000 7fb291600640 1 -- 172.21.15.6:0/586353496 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb28c0c9a80 con 0x7fb270003a20 2024-03-04T09:32:42.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.304+0000 7fb291600640 1 -- 172.21.15.6:0/586353496 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb28c04f3a0 con 0x7fb270003a20 2024-03-04T09:32:42.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.304+0000 7fb282a00640 1 -- 172.21.15.6:0/586353496 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb27c007070 con 0x7fb270003a20 2024-03-04T09:32:42.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.306+0000 7fb282a00640 1 --2- 172.21.15.6:0/586353496 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb26c06b500 0x7fb26c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:42.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.306+0000 7fb282a00640 1 -- 172.21.15.6:0/586353496 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5853+0+0 (secure 0 0 0) 0x7fb27c0ab650 con 0x7fb270003a20 2024-03-04T09:32:42.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.306+0000 7fb28a000640 1 --2- 172.21.15.6:0/586353496 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb26c06b500 0x7fb26c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:42.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.308+0000 7fb28a000640 1 --2- 172.21.15.6:0/586353496 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb26c06b500 0x7fb26c06d9c0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fb28c15ad60 tx=0x7fb278008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:42.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.310+0000 7fb282a00640 1 -- 172.21.15.6:0/586353496 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb27c079d50 con 0x7fb270003a20 2024-03-04T09:32:42.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.509+0000 7fb291600640 1 -- 172.21.15.6:0/586353496 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb28c06bab0 con 0x7fb270003a20 2024-03-04T09:32:42.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.512+0000 7fb282a00640 1 -- 172.21.15.6:0/586353496 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v115) v1 ==== 74+0+16859 (secure 0 0 0) 0x7fb27c07ddf0 con 0x7fb270003a20 2024-03-04T09:32:42.509 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:32:42.509 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":115,"fsid":"edce0578-5ae4-4402-b45a-552bdef84d7c","created":"2024-03-04T09:24:54.523780+0000","modified":"2024-03-04T09:32:03.117903+0000","last_up_change":"2024-03-04T09:25:07.089049+0000","last_in_change":"2024-03-04T09:24:57.982077+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T09:25:07.935198+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-04T09:31:19.018233+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-04T09:31:20.131544+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a96574f-81f1-4bc6-8463-757520b84a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6825","nonce":1364449401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6827","nonce":1364449401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6831","nonce":1364449401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6829","nonce":1364449401}]},"public_addr":"172.21.15.6:6825/1364449401","cluster_addr":"172.21.15.6:6827/1364449401","heartbeat_back_addr":"172.21.15.6:6831/1364449401","heartbeat_front_addr":"172.21.15.6:6829/1364449401","state":["exists","up"]},{"osd":1,"uuid":"7d79b9c1-6b87-476a-bc36-7db5ee64cd90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6809","nonce":2494103078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6811","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6813","nonce":2494103078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6821","nonce":2494103078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6817","nonce":2494103078}]},"public_addr":"172.21.15.6:6809/2494103078","cluster_addr":"172.21.15.6:6813/2494103078","heartbeat_back_addr":"172.21.15.6:6821/2494103078","heartbeat_front_addr":"172.21.15.6:6817/2494103078","state":["exists","up"]},{"osd":2,"uuid":"492331b9-4ee9-40b6-bacd-85d44cd6c45a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6812","nonce":1500066681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6816","nonce":1500066681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6824","nonce":1500066681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6820","nonce":1500066681}]},"public_addr":"172.21.15.6:6812/1500066681","cluster_addr":"172.21.15.6:6816/1500066681","heartbeat_back_addr":"172.21.15.6:6824/1500066681","heartbeat_front_addr":"172.21.15.6:6820/1500066681","state":["exists","up"]},{"osd":3,"uuid":"21bb2efe-9ba7-4bdc-bc31-180cd1102e21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6801","nonce":1558806123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6803","nonce":1558806123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6807","nonce":1558806123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6805","nonce":1558806123}]},"public_addr":"172.21.15.6:6801/1558806123","cluster_addr":"172.21.15.6:6803/1558806123","heartbeat_back_addr":"172.21.15.6:6807/1558806123","heartbeat_front_addr":"172.21.15.6:6805/1558806123","state":["exists","up"]},{"osd":4,"uuid":"6e2c4b71-19d9-4737-83d9-558705d8644a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6801","nonce":1526135780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6803","nonce":1526135780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6807","nonce":1526135780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6805","nonce":1526135780}]},"public_addr":"172.21.15.146:6801/1526135780","cluster_addr":"172.21.15.146:6803/1526135780","heartbeat_back_addr":"172.21.15.146:6807/1526135780","heartbeat_front_addr":"172.21.15.146:6805/1526135780","state":["exists","up"]},{"osd":5,"uuid":"fb69c525-f1be-4b7d-ab32-6c29b0d6fddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6809","nonce":460262507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6811","nonce":460262507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6815","nonce":460262507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6813","nonce":460262507}]},"public_addr":"172.21.15.146:6809/460262507","cluster_addr":"172.21.15.146:6811/460262507","heartbeat_back_addr":"172.21.15.146:6815/460262507","heartbeat_front_addr":"172.21.15.146:6813/460262507","state":["exists","up"]},{"osd":6,"uuid":"c32e7fed-370d-4401-8ce0-6a4335b0d168","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6823","nonce":1392897493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6827","nonce":1392897493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6831","nonce":1392897493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6829","nonce":1392897493}]},"public_addr":"172.21.15.146:6823/1392897493","cluster_addr":"172.21.15.146:6827/1392897493","heartbeat_back_addr":"172.21.15.146:6831/1392897493","heartbeat_front_addr":"172.21.15.146:6829/1392897493","state":["exists","up"]},{"osd":7,"uuid":"75a79b44-fa5f-426d-a7d3-71f21f78835c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6817","nonce":3268323156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6819","nonce":3268323156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6825","nonce":3268323156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6822","nonce":3268323156}]},"public_addr":"172.21.15.146:6817/3268323156","cluster_addr":"172.21.15.146:6819/3268323156","heartbeat_back_addr":"172.21.15.146:6825/3268323156","heartbeat_front_addr":"172.21.15.146:6822/3268323156","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.310437+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.295850+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.294674+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.526855+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.300023+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.595160+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.346046+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.558741+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"17.e","mappings":[{"from":0,"to":5}]},{"pgid":"17.26","mappings":[{"from":0,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.6:6834/2697931210":"2024-03-05T09:31:36.737386+0000","172.21.15.6:6832/2697931210":"2024-03-05T09:31:36.737386+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T09:32:42.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.514+0000 7fb291600640 1 -- 172.21.15.6:0/586353496 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb26c06b500 msgr2=0x7fb26c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:42.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.514+0000 7fb291600640 1 --2- 172.21.15.6:0/586353496 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb26c06b500 0x7fb26c06d9c0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fb28c15ad60 tx=0x7fb278008040 comp rx=0 tx=0).stop 2024-03-04T09:32:42.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.514+0000 7fb291600640 1 -- 172.21.15.6:0/586353496 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb270003a20 msgr2=0x7fb28c15a410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:42.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.514+0000 7fb291600640 1 --2- 172.21.15.6:0/586353496 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb270003a20 0x7fb28c15a410 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 crypto rx=0x7fb27c00c280 tx=0x7fb27c02e070 comp rx=0 tx=0).stop 2024-03-04T09:32:42.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.514+0000 7fb291600640 1 -- 172.21.15.6:0/586353496 shutdown_connections 2024-03-04T09:32:42.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.514+0000 7fb291600640 1 --2- 172.21.15.6:0/586353496 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb28c151300 0x7fb28c151780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:42.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.514+0000 7fb291600640 1 --2- 172.21.15.6:0/586353496 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb26c06b500 0x7fb26c06d9c0 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:42.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.514+0000 7fb291600640 1 --2- 172.21.15.6:0/586353496 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb28c0cac00 0x7fb28c0cb080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:42.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.514+0000 7fb291600640 1 --2- 172.21.15.6:0/586353496 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb270003a20 0x7fb28c15a410 unknown :-1 s=CLOSED pgs=1427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:42.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.514+0000 7fb291600640 1 -- 172.21.15.6:0/586353496 >> 172.21.15.6:0/586353496 conn(0x7fb28c06afb0 msgr2=0x7fb28c0cc730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:32:42.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.514+0000 7fb291600640 1 -- 172.21.15.6:0/586353496 shutdown_connections 2024-03-04T09:32:42.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.514+0000 7fb291600640 1 -- 172.21.15.6:0/586353496 wait complete. 2024-03-04T09:32:42.530 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-04T09:32:42.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.744+0000 7f222ec00640 1 Processor -- start 2024-03-04T09:32:42.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.745+0000 7f222ec00640 1 -- start start 2024-03-04T09:32:42.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.747+0000 7f222ec00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f22280af1a0 0x7f222815c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:42.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.747+0000 7f222ec00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f22280cdc20 0x7f22280b2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:42.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.747+0000 7f222ec00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f222815cc40 con 0x7f22280af580 2024-03-04T09:32:42.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.747+0000 7f222ec00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f222815cf20 con 0x7f22280af1a0 2024-03-04T09:32:42.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.747+0000 7f222ec00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f222815d200 con 0x7f22280cdc20 2024-03-04T09:32:42.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.748+0000 7f2227e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f22280af1a0 0x7f222815c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:42.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.748+0000 7f221fe00640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f22280af580 0x7f22280b4c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.6:53042/0 (socket says 172.21.15.6:53042) 2024-03-04T09:32:42.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.748+0000 7f221fe00640 1 -- 172.21.15.6:0/2814235243 learned_addr learned my addr 172.21.15.6:0/2814235243 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:32:42.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.748+0000 7f222cc00640 1 --2- 172.21.15.6:0/2814235243 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f22280cdc20 0x7f22280b2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:42.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.749+0000 7f2227e00640 1 -- 172.21.15.6:0/2814235243 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f22280cdc20 msgr2=0x7f22280b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:32:42.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.749+0000 7f2227e00640 1 --2- 172.21.15.6:0/2814235243 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f22280cdc20 0x7f22280b2dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:42.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.749+0000 7f2227e00640 1 -- 172.21.15.6:0/2814235243 >> v1:172.21.15.146:6789/0 conn(0x7f22280af580 legacy=0x7f22280b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:42.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.749+0000 7f2227e00640 1 -- 172.21.15.6:0/2814235243 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f222815d4e0 con 0x7f22280af1a0 2024-03-04T09:32:42.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.750+0000 7f2227e00640 1 --2- 172.21.15.6:0/2814235243 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f22280af1a0 0x7f222815c600 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f22180097a0 tx=0x7f221802b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a9552d9a08f7b766 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:42.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.751+0000 7f2227400640 1 -- 172.21.15.6:0/2814235243 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22180315a0 con 0x7f22280af1a0 2024-03-04T09:32:42.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.751+0000 7f2227400640 1 -- 172.21.15.6:0/2814235243 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2218037070 con 0x7f22280af1a0 2024-03-04T09:32:42.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.751+0000 7f2227400640 1 -- 172.21.15.6:0/2814235243 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2218031960 con 0x7f22280af1a0 2024-03-04T09:32:42.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.751+0000 7f222ec00640 1 -- 172.21.15.6:0/2814235243 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f22280af1a0 msgr2=0x7f222815c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:42.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.751+0000 7f222ec00640 1 --2- 172.21.15.6:0/2814235243 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f22280af1a0 0x7f222815c600 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f22180097a0 tx=0x7f221802b840 comp rx=0 tx=0).stop 2024-03-04T09:32:42.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.751+0000 7f222ec00640 1 -- 172.21.15.6:0/2814235243 shutdown_connections 2024-03-04T09:32:42.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.751+0000 7f222ec00640 1 --2- 172.21.15.6:0/2814235243 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f22280cdc20 0x7f22280b2dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:42.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.751+0000 7f222ec00640 1 --2- 172.21.15.6:0/2814235243 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f22280af1a0 0x7f222815c600 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:42.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.751+0000 7f222ec00640 1 -- 172.21.15.6:0/2814235243 >> 172.21.15.6:0/2814235243 conn(0x7f222806afb0 msgr2=0x7f222806b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:32:42.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.752+0000 7f222ec00640 1 -- 172.21.15.6:0/2814235243 shutdown_connections 2024-03-04T09:32:42.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.752+0000 7f222ec00640 1 -- 172.21.15.6:0/2814235243 wait complete. 2024-03-04T09:32:42.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.754+0000 7f222ec00640 1 Processor -- start 2024-03-04T09:32:42.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.755+0000 7f222ec00640 1 -- start start 2024-03-04T09:32:42.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.756+0000 7f222ec00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f22280af1a0 0x7f222809c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:42.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.756+0000 7f222ec00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f22280af580 0x7f22280cc850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:42.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.756+0000 7f222ec00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f22280cdc20 0x7f2228159180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:42.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.756+0000 7f222ec00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22280b3720 con 0x7f22280af1a0 2024-03-04T09:32:42.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.756+0000 7f222ec00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22280b35a0 con 0x7f22280cdc20 2024-03-04T09:32:42.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.756+0000 7f222ec00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2228155e70 con 0x7f22280af580 2024-03-04T09:32:42.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.756+0000 7f2227e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f22280af1a0 0x7f222809c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:42.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.757+0000 7f2227e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f22280af1a0 0x7f222809c470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:47266/0 (socket says 172.21.15.6:47266) 2024-03-04T09:32:42.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.757+0000 7f2227e00640 1 -- 172.21.15.6:0/494770296 learned_addr learned my addr 172.21.15.6:0/494770296 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:32:42.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.757+0000 7f221fe00640 1 --2- 172.21.15.6:0/494770296 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f22280af580 0x7f22280cc850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:42.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.757+0000 7f222cc00640 1 --2- 172.21.15.6:0/494770296 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f22280cdc20 0x7f2228159180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:42.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.757+0000 7f2227e00640 1 -- 172.21.15.6:0/494770296 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f22280af580 msgr2=0x7f22280cc850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:42.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.757+0000 7f2227e00640 1 --2- 172.21.15.6:0/494770296 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f22280af580 0x7f22280cc850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:42.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.757+0000 7f2227e00640 1 -- 172.21.15.6:0/494770296 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f22280cdc20 msgr2=0x7f2228159180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:42.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.757+0000 7f2227e00640 1 --2- 172.21.15.6:0/494770296 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f22280cdc20 0x7f2228159180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:42.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.757+0000 7f2227e00640 1 -- 172.21.15.6:0/494770296 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22281562d0 con 0x7f22280af1a0 2024-03-04T09:32:42.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.757+0000 7f221fe00640 1 --2- 172.21.15.6:0/494770296 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f22280af580 0x7f22280cc850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:32:42.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.757+0000 7f222cc00640 1 --2- 172.21.15.6:0/494770296 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f22280cdc20 0x7f2228159180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:32:42.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.757+0000 7f2227e00640 1 --2- 172.21.15.6:0/494770296 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f22280af1a0 0x7f222809c470 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7f2218034040 tx=0x7f22180046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:42.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.758+0000 7f2224c00640 1 -- 172.21.15.6:0/494770296 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2218032630 con 0x7f22280af1a0 2024-03-04T09:32:42.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.758+0000 7f2224c00640 1 -- 172.21.15.6:0/494770296 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22180327d0 con 0x7f22280af1a0 2024-03-04T09:32:42.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.758+0000 7f2224c00640 1 -- 172.21.15.6:0/494770296 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2218032bc0 con 0x7f22280af1a0 2024-03-04T09:32:42.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.758+0000 7f222ec00640 1 -- 172.21.15.6:0/494770296 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22281565a0 con 0x7f22280af1a0 2024-03-04T09:32:42.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.758+0000 7f222ec00640 1 -- 172.21.15.6:0/494770296 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2228156de0 con 0x7f22280af1a0 2024-03-04T09:32:42.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.759+0000 7f222ec00640 1 -- 172.21.15.6:0/494770296 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2228069510 con 0x7f22280af1a0 2024-03-04T09:32:42.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.760+0000 7f2224c00640 1 -- 172.21.15.6:0/494770296 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f221804e430 con 0x7f22280af1a0 2024-03-04T09:32:42.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.762+0000 7f2224c00640 1 --2- 172.21.15.6:0/494770296 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f220406b4d0 0x7f220406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:42.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.762+0000 7f2224c00640 1 -- 172.21.15.6:0/494770296 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f221802f030 con 0x7f22280af1a0 2024-03-04T09:32:42.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.762+0000 7f221fe00640 1 --2- 172.21.15.6:0/494770296 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f220406b4d0 0x7f220406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:42.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.764+0000 7f221fe00640 1 --2- 172.21.15.6:0/494770296 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f220406b4d0 0x7f220406d990 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f22281f6390 tx=0x7f2210009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:42.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:42.766+0000 7f2224c00640 1 -- 172.21.15.6:0/494770296 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f22281565a0 con 0x7f22280af1a0 2024-03-04T09:32:43.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:43.005+0000 7f222ec00640 1 -- 172.21.15.6:0/494770296 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f222815f620 con 0x7f22280af1a0 2024-03-04T09:32:43.003 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-03-04T09:32:43.006+0000 7f1606a00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-04T09:32:43.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:43.009+0000 7f2224c00640 1 -- 172.21.15.6:0/494770296 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v85) v1 ==== 161+0+0 (secure 0 0 0) 0x7f2218040070 con 0x7f22280af1a0 2024-03-04T09:32:43.006 INFO:teuthology.orchestra.run.smithi006.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-04T09:32:43.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:43.012+0000 7f222ec00640 1 -- 172.21.15.6:0/494770296 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f220406b4d0 msgr2=0x7f220406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:43.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:43.012+0000 7f222ec00640 1 --2- 172.21.15.6:0/494770296 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f220406b4d0 0x7f220406d990 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f22281f6390 tx=0x7f2210009040 comp rx=0 tx=0).stop 2024-03-04T09:32:43.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:43.012+0000 7f222ec00640 1 -- 172.21.15.6:0/494770296 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f22280af1a0 msgr2=0x7f222809c470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:43.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:43.012+0000 7f222ec00640 1 --2- 172.21.15.6:0/494770296 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f22280af1a0 0x7f222809c470 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7f2218034040 tx=0x7f22180046c0 comp rx=0 tx=0).stop 2024-03-04T09:32:43.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:43.013+0000 7f222ec00640 1 -- 172.21.15.6:0/494770296 shutdown_connections 2024-03-04T09:32:43.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:43.013+0000 7f222ec00640 1 --2- 172.21.15.6:0/494770296 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f22280cdc20 0x7f2228159180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:43.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:43.013+0000 7f222ec00640 1 --2- 172.21.15.6:0/494770296 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f220406b4d0 0x7f220406d990 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:43.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:43.013+0000 7f222ec00640 1 --2- 172.21.15.6:0/494770296 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f22280af580 0x7f22280cc850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:43.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:43.013+0000 7f222ec00640 1 --2- 172.21.15.6:0/494770296 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f22280af1a0 0x7f222809c470 unknown :-1 s=CLOSED pgs=1428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:43.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:43.013+0000 7f222ec00640 1 -- 172.21.15.6:0/494770296 >> 172.21.15.6:0/494770296 conn(0x7f222806afb0 msgr2=0x7f222815ec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:32:43.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:43.013+0000 7f222ec00640 1 -- 172.21.15.6:0/494770296 shutdown_connections 2024-03-04T09:32:43.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:43.014+0000 7f222ec00640 1 -- 172.21.15.6:0/494770296 wait complete. 2024-03-04T09:32:43.028 INFO:tasks.ceph.mds.a.smithi006.stderr: -7965> 2024-03-04T09:31:43.763+0000 7f03d0800640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-04T09:32:43.035 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-04T09:32:43.089 INFO:tasks.ceph.mds.a.smithi006.stdout:starting mds.a at 2024-03-04T09:32:43.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:43.225+0000 7f7c4da00640 1 Processor -- start 2024-03-04T09:32:43.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:43.225+0000 7f7c4da00640 1 -- start start 2024-03-04T09:32:43.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:43.227+0000 7f7c4da00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f7c4815f810 0x7f7c480b1b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:43.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:43.227+0000 7f7c4da00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7c480b2080 0x7f7c480b2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:43.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:43.227+0000 7f7c4da00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c4815fe40 con 0x7f7c48145430 2024-03-04T09:32:43.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:43.227+0000 7f7c4da00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c48160120 con 0x7f7c4815f810 2024-03-04T09:32:43.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:43.227+0000 7f7c4da00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c48160400 con 0x7f7c480b2080 2024-03-04T09:32:43.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:43.227+0000 7f7c3f400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f7c4815f810 0x7f7c480b1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:43.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:43.227+0000 7f7c3f400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f7c4815f810 0x7f7c480b1b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:47292/0 (socket says 172.21.15.6:47292) 2024-03-04T09:32:43.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:43.227+0000 7f7c3f400640 1 -- 172.21.15.6:0/3800613152 learned_addr learned my addr 172.21.15.6:0/3800613152 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:32:43.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:43.228+0000 7f7c4d000640 1 --2- 172.21.15.6:0/3800613152 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7c480b2080 0x7f7c480b2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:43.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:43.228+0000 7f7c3f400640 1 -- 172.21.15.6:0/3800613152 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7c480b2080 msgr2=0x7f7c480b2fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:32:43.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:43.228+0000 7f7c3f400640 1 --2- 172.21.15.6:0/3800613152 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7c480b2080 0x7f7c480b2fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:43.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:43.228+0000 7f7c3f400640 1 -- 172.21.15.6:0/3800613152 >> v1:172.21.15.146:6789/0 conn(0x7f7c48145430 legacy=0x7f7c4815cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:32:43.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:43.228+0000 7f7c3f400640 1 -- 172.21.15.6:0/3800613152 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c481606e0 con 0x7f7c4815f810 2024-03-04T09:32:43.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:43.229+0000 7f7c3f400640 1 --2- 172.21.15.6:0/3800613152 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7c4815f810 0x7f7c480b1b40 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7f7c30009870 tx=0x7f7c3002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fceef671653326e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:43.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:43.229+0000 7f7c47400640 1 -- 172.21.15.6:0/3800613152 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c30035020 con 0x7f7c4815f810 2024-03-04T09:32:43.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:43.229+0000 7f7c47400640 1 -- 172.21.15.6:0/3800613152 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c30031640 con 0x7f7c4815f810 2024-03-04T09:32:43.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:43.229+0000 7f7c47400640 1 -- 172.21.15.6:0/3800613152 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c3002d040 con 0x7f7c4815f810 2024-03-04T09:32:43.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:43.230+0000 7f7c4da00640 1 -- 172.21.15.6:0/3800613152 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7c4815f810 msgr2=0x7f7c480b1b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:43.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:43.230+0000 7f7c4da00640 1 --2- 172.21.15.6:0/3800613152 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7c4815f810 0x7f7c480b1b40 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7f7c30009870 tx=0x7f7c3002b8f0 comp rx=0 tx=0).stop 2024-03-04T09:32:43.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:43.230+0000 7f7c4da00640 1 -- 172.21.15.6:0/3800613152 shutdown_connections 2024-03-04T09:32:43.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:43.230+0000 7f7c4da00640 1 --2- 172.21.15.6:0/3800613152 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7c480b2080 0x7f7c480b2fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:43.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:43.230+0000 7f7c4da00640 1 --2- 172.21.15.6:0/3800613152 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7c4815f810 0x7f7c480b1b40 unknown :-1 s=CLOSED pgs=1431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:43.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:43.230+0000 7f7c4da00640 1 -- 172.21.15.6:0/3800613152 >> 172.21.15.6:0/3800613152 conn(0x7f7c4806b3a0 msgr2=0x7f7c4806b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:32:43.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:43.230+0000 7f7c4da00640 1 -- 172.21.15.6:0/3800613152 shutdown_connections 2024-03-04T09:32:43.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:43.230+0000 7f7c4da00640 1 -- 172.21.15.6:0/3800613152 wait complete. 2024-03-04T09:32:43.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:43.232+0000 7f7c4da00640 1 Processor -- start 2024-03-04T09:32:43.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:43.232+0000 7f7c4da00640 1 -- start start 2024-03-04T09:32:43.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:43.234+0000 7f7c4da00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7c480b2080 0x7f7c4814fac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:43.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:43.234+0000 7f7c4da00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7c48145430 0x7f7c4813f250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:43.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:43.234+0000 7f7c4da00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7c4815f810 0x7f7c4814e120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:43.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:43.234+0000 7f7c4da00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c480cdf80 con 0x7f7c4815f810 2024-03-04T09:32:43.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:43.234+0000 7f7c4da00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c480cde00 con 0x7f7c480b2080 2024-03-04T09:32:43.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:43.234+0000 7f7c4da00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c480b3510 con 0x7f7c48145430 2024-03-04T09:32:43.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:43.234+0000 7f7c4d000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7c4815f810 0x7f7c4814e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:43.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:43.234+0000 7f7c4d000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7c4815f810 0x7f7c4814e120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:47296/0 (socket says 172.21.15.6:47296) 2024-03-04T09:32:43.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:43.234+0000 7f7c4d000640 1 -- 172.21.15.6:0/1979037101 learned_addr learned my addr 172.21.15.6:0/1979037101 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:32:43.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:43.234+0000 7f7c47e00640 1 --2- 172.21.15.6:0/1979037101 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7c480b2080 0x7f7c4814fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:43.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:43.234+0000 7f7c3f400640 1 --2- 172.21.15.6:0/1979037101 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7c48145430 0x7f7c4813f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:43.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:43.234+0000 7f7c4d000640 1 -- 172.21.15.6:0/1979037101 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7c48145430 msgr2=0x7f7c4813f250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:43.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:43.234+0000 7f7c4d000640 1 --2- 172.21.15.6:0/1979037101 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7c48145430 0x7f7c4813f250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:43.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:43.234+0000 7f7c4d000640 1 -- 172.21.15.6:0/1979037101 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7c480b2080 msgr2=0x7f7c4814fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:43.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:43.234+0000 7f7c4d000640 1 --2- 172.21.15.6:0/1979037101 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7c480b2080 0x7f7c4814fac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:43.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:43.234+0000 7f7c4d000640 1 -- 172.21.15.6:0/1979037101 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c4814ea20 con 0x7f7c4815f810 2024-03-04T09:32:43.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:43.235+0000 7f7c4d000640 1 --2- 172.21.15.6:0/1979037101 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7c4815f810 0x7f7c4814e120 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7f7c40007e70 tx=0x7f7c4000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:43.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:43.235+0000 7f7c44c00640 1 -- 172.21.15.6:0/1979037101 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c40014620 con 0x7f7c4815f810 2024-03-04T09:32:43.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:43.235+0000 7f7c44c00640 1 -- 172.21.15.6:0/1979037101 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c40019070 con 0x7f7c4815f810 2024-03-04T09:32:43.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:43.235+0000 7f7c44c00640 1 -- 172.21.15.6:0/1979037101 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c40014a20 con 0x7f7c4815f810 2024-03-04T09:32:43.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:43.235+0000 7f7c4da00640 1 -- 172.21.15.6:0/1979037101 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c4814ed00 con 0x7f7c4815f810 2024-03-04T09:32:43.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:43.235+0000 7f7c4da00640 1 -- 172.21.15.6:0/1979037101 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c4814f2c0 con 0x7f7c4815f810 2024-03-04T09:32:43.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:43.237+0000 7f7c4da00640 1 -- 172.21.15.6:0/1979037101 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c4804f330 con 0x7f7c4815f810 2024-03-04T09:32:43.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:43.237+0000 7f7c44c00640 1 -- 172.21.15.6:0/1979037101 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7c40014bc0 con 0x7f7c4815f810 2024-03-04T09:32:43.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:43.239+0000 7f7c44c00640 1 --2- 172.21.15.6:0/1979037101 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f7c2406b4d0 0x7f7c2406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:43.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:43.239+0000 7f7c44c00640 1 -- 172.21.15.6:0/1979037101 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f7c4008ac00 con 0x7f7c4815f810 2024-03-04T09:32:43.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:43.239+0000 7f7c47e00640 1 --2- 172.21.15.6:0/1979037101 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f7c2406b4d0 0x7f7c2406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:43.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:43.241+0000 7f7c47e00640 1 --2- 172.21.15.6:0/1979037101 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f7c2406b4d0 0x7f7c2406d990 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f7c38004ea0 tx=0x7f7c38009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:43.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:43.244+0000 7f7c44c00640 1 -- 172.21.15.6:0/1979037101 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7c40059980 con 0x7f7c4815f810 2024-03-04T09:32:43.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:43.480+0000 7f7c4da00640 1 -- 172.21.15.6:0/1979037101 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f7c4806e240 con 0x7f7c4815f810 2024-03-04T09:32:44.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:44.012+0000 7f7c44c00640 1 -- 172.21.15.6:0/1979037101 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v86) v1 ==== 104+0+0 (secure 0 0 0) 0x7f7c40059280 con 0x7f7c4815f810 2024-03-04T09:32:44.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:44.016+0000 7f7c4da00640 1 -- 172.21.15.6:0/1979037101 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f7c2406b4d0 msgr2=0x7f7c2406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:44.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:44.016+0000 7f7c4da00640 1 --2- 172.21.15.6:0/1979037101 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f7c2406b4d0 0x7f7c2406d990 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f7c38004ea0 tx=0x7f7c38009040 comp rx=0 tx=0).stop 2024-03-04T09:32:44.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:44.016+0000 7f7c4da00640 1 -- 172.21.15.6:0/1979037101 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7c4815f810 msgr2=0x7f7c4814e120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:44.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:44.016+0000 7f7c4da00640 1 --2- 172.21.15.6:0/1979037101 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7c4815f810 0x7f7c4814e120 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7f7c40007e70 tx=0x7f7c4000e4b0 comp rx=0 tx=0).stop 2024-03-04T09:32:44.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:44.017+0000 7f7c4da00640 1 -- 172.21.15.6:0/1979037101 shutdown_connections 2024-03-04T09:32:44.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:44.017+0000 7f7c4da00640 1 --2- 172.21.15.6:0/1979037101 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7c4815f810 0x7f7c4814e120 unknown :-1 s=CLOSED pgs=1432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:44.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:44.017+0000 7f7c4da00640 1 --2- 172.21.15.6:0/1979037101 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f7c2406b4d0 0x7f7c2406d990 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:44.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:44.017+0000 7f7c4da00640 1 --2- 172.21.15.6:0/1979037101 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7c48145430 0x7f7c4813f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:44.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:44.017+0000 7f7c4da00640 1 --2- 172.21.15.6:0/1979037101 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7c480b2080 0x7f7c4814fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:44.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:44.017+0000 7f7c4da00640 1 -- 172.21.15.6:0/1979037101 >> 172.21.15.6:0/1979037101 conn(0x7f7c4806b3a0 msgr2=0x7f7c4815e500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:32:44.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:44.017+0000 7f7c4da00640 1 -- 172.21.15.6:0/1979037101 shutdown_connections 2024-03-04T09:32:44.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:44.017+0000 7f7c4da00640 1 -- 172.21.15.6:0/1979037101 wait complete. 2024-03-04T09:32:44.034 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-04T09:32:44.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:44.261+0000 7fa38ae00640 1 Processor -- start 2024-03-04T09:32:44.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:44.262+0000 7fa38ae00640 1 -- start start 2024-03-04T09:32:44.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:44.263+0000 7fa38ae00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fa38415df10 0x7fa3840d2240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:44.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:44.263+0000 7fa38ae00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa38415e2f0 0x7fa3840d0680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:44.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:44.263+0000 7fa38ae00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3840d2a60 con 0x7fa3840d45b0 2024-03-04T09:32:44.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:44.263+0000 7fa38ae00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3840b6b80 con 0x7fa38415df10 2024-03-04T09:32:44.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:44.263+0000 7fa38ae00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3840b6e20 con 0x7fa38415e2f0 2024-03-04T09:32:44.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:44.264+0000 7fa38a400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fa3840d45b0 0x7fa3840ce950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:35136/0 (socket says 172.21.15.6:35136) 2024-03-04T09:32:44.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:44.264+0000 7fa38a400640 1 -- 172.21.15.6:0/1624455814 learned_addr learned my addr 172.21.15.6:0/1624455814 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:32:44.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:44.264+0000 7fa389a00640 1 --2- 172.21.15.6:0/1624455814 >> v2:172.21.15.146:3300/0 conn(0x7fa38415df10 0x7fa3840d2240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:44.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:44.264+0000 7fa383e00640 1 -- 172.21.15.6:0/1624455814 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2179820157 0 0) 0x7fa3840d2a60 con 0x7fa3840d45b0 2024-03-04T09:32:44.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:44.264+0000 7fa389000640 1 --2- 172.21.15.6:0/1624455814 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa38415e2f0 0x7fa3840d0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:44.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:44.264+0000 7fa383e00640 1 -- 172.21.15.6:0/1624455814 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa36c003520 con 0x7fa3840d45b0 2024-03-04T09:32:44.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:44.265+0000 7fa383e00640 1 -- 172.21.15.6:0/1624455814 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2636754616 0 0) 0x7fa36c003520 con 0x7fa3840d45b0 2024-03-04T09:32:44.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:44.265+0000 7fa383e00640 1 -- 172.21.15.6:0/1624455814 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa38415e2f0 msgr2=0x7fa3840d0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:32:44.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:44.265+0000 7fa383e00640 1 --2- 172.21.15.6:0/1624455814 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa38415e2f0 0x7fa3840d0680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:44.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:44.265+0000 7fa383e00640 1 -- 172.21.15.6:0/1624455814 >> v2:172.21.15.146:3300/0 conn(0x7fa38415df10 msgr2=0x7fa3840d2240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:32:44.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:44.265+0000 7fa383e00640 1 --2- 172.21.15.6:0/1624455814 >> v2:172.21.15.146:3300/0 conn(0x7fa38415df10 0x7fa3840d2240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:44.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:44.265+0000 7fa383e00640 1 -- 172.21.15.6:0/1624455814 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3840d2780 con 0x7fa3840d45b0 2024-03-04T09:32:44.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:44.265+0000 7fa383e00640 1 -- 172.21.15.6:0/1624455814 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7fa37c001a10 con 0x7fa3840d45b0 2024-03-04T09:32:44.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:44.265+0000 7fa383e00640 1 -- 172.21.15.6:0/1624455814 >> v1:172.21.15.6:6789/0 conn(0x7fa3840d45b0 legacy=0x7fa3840ce950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:44.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:44.265+0000 7fa383e00640 1 --2- 172.21.15.6:0/1624455814 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa36c0039b0 0x7fa36c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:44.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:44.265+0000 7fa383e00640 1 --2- 172.21.15.6:0/1624455814 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa38415e2f0 0x7fa36c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:44.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:44.265+0000 7fa383e00640 1 --2- 172.21.15.6:0/1624455814 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa36c004d40 0x7fa36c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:44.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:44.265+0000 7fa383e00640 1 -- 172.21.15.6:0/1624455814 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3840b6b80 con 0x7fa36c004d40 2024-03-04T09:32:44.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:44.265+0000 7fa383e00640 1 -- 172.21.15.6:0/1624455814 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3840b6e20 con 0x7fa36c0039b0 2024-03-04T09:32:44.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:44.265+0000 7fa383e00640 1 -- 172.21.15.6:0/1624455814 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa36c00ba90 con 0x7fa38415e2f0 2024-03-04T09:32:44.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:44.265+0000 7fa38a400640 1 --2- 172.21.15.6:0/1624455814 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa36c004d40 0x7fa36c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:44.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:44.265+0000 7fa389000640 1 --2- 172.21.15.6:0/1624455814 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa38415e2f0 0x7fa36c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:44.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:44.266+0000 7fa38a400640 1 -- 172.21.15.6:0/1624455814 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa38415e2f0 msgr2=0x7fa36c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:32:44.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:44.266+0000 7fa38a400640 1 --2- 172.21.15.6:0/1624455814 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa38415e2f0 0x7fa36c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:44.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:44.266+0000 7fa389a00640 1 --2- 172.21.15.6:0/1624455814 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa36c0039b0 0x7fa36c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:44.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:44.266+0000 7fa38a400640 1 -- 172.21.15.6:0/1624455814 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa36c0039b0 msgr2=0x7fa36c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:32:44.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:44.266+0000 7fa38a400640 1 --2- 172.21.15.6:0/1624455814 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa36c0039b0 0x7fa36c003dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:44.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:44.266+0000 7fa38a400640 1 -- 172.21.15.6:0/1624455814 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa36c003520 con 0x7fa36c004d40 2024-03-04T09:32:44.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:44.267+0000 7fa38a400640 1 --2- 172.21.15.6:0/1624455814 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa36c004d40 0x7fa36c0051c0 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7fa3840d2c30 tx=0x7fa36c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a4eac0b3cc8aeff server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:44.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:44.267+0000 7fa383e00640 1 -- 172.21.15.6:0/1624455814 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa37c00a410 con 0x7fa36c004d40 2024-03-04T09:32:44.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:44.267+0000 7fa383e00640 1 -- 172.21.15.6:0/1624455814 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa37c034630 con 0x7fa36c004d40 2024-03-04T09:32:44.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:44.267+0000 7fa383e00640 1 -- 172.21.15.6:0/1624455814 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa37c034a20 con 0x7fa36c004d40 2024-03-04T09:32:44.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:44.267+0000 7fa38ae00640 1 -- 172.21.15.6:0/1624455814 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa36c004d40 msgr2=0x7fa36c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:44.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:44.267+0000 7fa38ae00640 1 --2- 172.21.15.6:0/1624455814 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa36c004d40 0x7fa36c0051c0 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7fa3840d2c30 tx=0x7fa36c005830 comp rx=0 tx=0).stop 2024-03-04T09:32:44.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:44.267+0000 7fa38a400640 1 -- 172.21.15.6:0/1624455814 reap_dead start 2024-03-04T09:32:44.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:44.268+0000 7fa38ae00640 1 -- 172.21.15.6:0/1624455814 shutdown_connections 2024-03-04T09:32:44.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:44.268+0000 7fa38ae00640 1 -- 172.21.15.6:0/1624455814 >> 172.21.15.6:0/1624455814 conn(0x7fa38409cfe0 msgr2=0x7fa3840d3c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:32:44.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:44.268+0000 7fa38ae00640 1 -- 172.21.15.6:0/1624455814 shutdown_connections 2024-03-04T09:32:44.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:44.268+0000 7fa38ae00640 1 -- 172.21.15.6:0/1624455814 wait complete. 2024-03-04T09:32:44.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:44.270+0000 7fa38ae00640 1 Processor -- start 2024-03-04T09:32:44.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:44.270+0000 7fa38ae00640 1 -- start start 2024-03-04T09:32:44.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:44.271+0000 7fa38ae00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa36c004d40 0x7fa38414f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:44.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:44.271+0000 7fa38ae00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa38409db30 0x7fa38414e490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:44.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:44.271+0000 7fa38ae00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa38414e9d0 0x7fa38414b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:44.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:44.271+0000 7fa38ae00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3840b87b0 con 0x7fa38409db30 2024-03-04T09:32:44.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:44.271+0000 7fa38ae00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3840b8630 con 0x7fa38414e9d0 2024-03-04T09:32:44.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:44.271+0000 7fa38ae00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3840b8930 con 0x7fa36c004d40 2024-03-04T09:32:44.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:44.271+0000 7fa389000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa38409db30 0x7fa38414e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:44.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:44.271+0000 7fa389000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa38409db30 0x7fa38414e490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:47314/0 (socket says 172.21.15.6:47314) 2024-03-04T09:32:44.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:44.271+0000 7fa389000640 1 -- 172.21.15.6:0/992672185 learned_addr learned my addr 172.21.15.6:0/992672185 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:32:44.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:44.271+0000 7fa389a00640 1 --2- 172.21.15.6:0/992672185 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa36c004d40 0x7fa38414f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:44.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:44.271+0000 7fa389000640 1 -- 172.21.15.6:0/992672185 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa36c004d40 msgr2=0x7fa38414f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:44.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:44.271+0000 7fa389000640 1 --2- 172.21.15.6:0/992672185 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa36c004d40 0x7fa38414f0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:44.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:44.271+0000 7fa389000640 1 -- 172.21.15.6:0/992672185 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa38414e9d0 msgr2=0x7fa38414b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:44.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:44.271+0000 7fa389000640 1 --2- 172.21.15.6:0/992672185 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa38414e9d0 0x7fa38414b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:44.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:44.271+0000 7fa389000640 1 -- 172.21.15.6:0/992672185 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa38414ee50 con 0x7fa38409db30 2024-03-04T09:32:44.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:44.271+0000 7fa389a00640 1 --2- 172.21.15.6:0/992672185 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa36c004d40 0x7fa38414f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:32:44.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:44.272+0000 7fa389000640 1 --2- 172.21.15.6:0/992672185 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa38409db30 0x7fa38414e490 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7fa370011600 tx=0x7fa370011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:44.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:44.272+0000 7fa381600640 1 -- 172.21.15.6:0/992672185 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa370015870 con 0x7fa38409db30 2024-03-04T09:32:44.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:44.272+0000 7fa381600640 1 -- 172.21.15.6:0/992672185 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa370015a10 con 0x7fa38409db30 2024-03-04T09:32:44.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:44.272+0000 7fa381600640 1 -- 172.21.15.6:0/992672185 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa370015dc0 con 0x7fa38409db30 2024-03-04T09:32:44.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:44.272+0000 7fa38ae00640 1 -- 172.21.15.6:0/992672185 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3840ccaa0 con 0x7fa38409db30 2024-03-04T09:32:44.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:44.272+0000 7fa38ae00640 1 -- 172.21.15.6:0/992672185 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3840cd060 con 0x7fa38409db30 2024-03-04T09:32:44.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:44.273+0000 7fa38ae00640 1 -- 172.21.15.6:0/992672185 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa38404f330 con 0x7fa38409db30 2024-03-04T09:32:44.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:44.274+0000 7fa381600640 1 -- 172.21.15.6:0/992672185 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa370013430 con 0x7fa38409db30 2024-03-04T09:32:44.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:44.275+0000 7fa381600640 1 --2- 172.21.15.6:0/992672185 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa36406b500 0x7fa36406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:44.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:44.275+0000 7fa381600640 1 -- 172.21.15.6:0/992672185 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5939+0+0 (secure 0 0 0) 0x7fa37008a7b0 con 0x7fa38409db30 2024-03-04T09:32:44.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:44.276+0000 7fa389a00640 1 --2- 172.21.15.6:0/992672185 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa36406b500 0x7fa36406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:44.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:44.277+0000 7fa389a00640 1 --2- 172.21.15.6:0/992672185 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa36406b500 0x7fa36406d9c0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fa378009a60 tx=0x7fa378008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:44.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:44.280+0000 7fa381600640 1 -- 172.21.15.6:0/992672185 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa370019030 con 0x7fa38409db30 2024-03-04T09:32:44.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:44.476+0000 7fa38ae00640 1 -- 172.21.15.6:0/992672185 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fa3840cf4d0 con 0x7fa38409db30 2024-03-04T09:32:45.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:45.339+0000 7fa381600640 1 -- 172.21.15.6:0/992672185 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v117) v1 ==== 181+0+0 (secure 0 0 0) 0x7fa3700594a0 con 0x7fa38409db30 2024-03-04T09:32:45.337 INFO:teuthology.orchestra.run.smithi006.stderr:pool 'cephfs_metadata' removed 2024-03-04T09:32:45.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:45.343+0000 7fa38ae00640 1 -- 172.21.15.6:0/992672185 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa36406b500 msgr2=0x7fa36406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:45.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:45.343+0000 7fa38ae00640 1 --2- 172.21.15.6:0/992672185 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa36406b500 0x7fa36406d9c0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fa378009a60 tx=0x7fa378008040 comp rx=0 tx=0).stop 2024-03-04T09:32:45.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:45.343+0000 7fa38ae00640 1 -- 172.21.15.6:0/992672185 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa38409db30 msgr2=0x7fa38414e490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:45.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:45.343+0000 7fa38ae00640 1 --2- 172.21.15.6:0/992672185 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa38409db30 0x7fa38414e490 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7fa370011600 tx=0x7fa370011ac0 comp rx=0 tx=0).stop 2024-03-04T09:32:45.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:45.346+0000 7fa38ae00640 1 -- 172.21.15.6:0/992672185 shutdown_connections 2024-03-04T09:32:45.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:45.346+0000 7fa38ae00640 1 --2- 172.21.15.6:0/992672185 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa38414e9d0 0x7fa38414b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:45.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:45.346+0000 7fa38ae00640 1 --2- 172.21.15.6:0/992672185 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa38409db30 0x7fa38414e490 unknown :-1 s=CLOSED pgs=1435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:45.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:45.346+0000 7fa38ae00640 1 --2- 172.21.15.6:0/992672185 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa36406b500 0x7fa36406d9c0 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:45.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:45.346+0000 7fa38ae00640 1 --2- 172.21.15.6:0/992672185 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa36c004d40 0x7fa38414f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:45.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:45.346+0000 7fa38ae00640 1 -- 172.21.15.6:0/992672185 >> 172.21.15.6:0/992672185 conn(0x7fa38409cfe0 msgr2=0x7fa38406cb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:32:45.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:45.346+0000 7fa38ae00640 1 -- 172.21.15.6:0/992672185 shutdown_connections 2024-03-04T09:32:45.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:45.346+0000 7fa38ae00640 1 -- 172.21.15.6:0/992672185 wait complete. 2024-03-04T09:32:45.368 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-04T09:32:45.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:45.579+0000 7fe804a00640 1 Processor -- start 2024-03-04T09:32:45.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:45.579+0000 7fe804a00640 1 -- start start 2024-03-04T09:32:45.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:45.581+0000 7fe804a00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fe800144320 0x7fe80006daf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:45.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:45.581+0000 7fe804a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8000b3ea0 con 0x7fe80006e030 2024-03-04T09:32:45.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:45.581+0000 7fe804a00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8000b4180 con 0x7fe8000727c0 2024-03-04T09:32:45.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:45.581+0000 7fe804a00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe80006e4a0 con 0x7fe800144320 2024-03-04T09:32:45.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:45.581+0000 7fe7f5e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fe80006e030 0x7fe8000720b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:35138/0 (socket says 172.21.15.6:35138) 2024-03-04T09:32:45.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:45.581+0000 7fe7f5e00640 1 -- 172.21.15.6:0/3076862093 learned_addr learned my addr 172.21.15.6:0/3076862093 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:32:45.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:45.582+0000 7fe7fe800640 1 --2- 172.21.15.6:0/3076862093 >> v2:172.21.15.146:3300/0 conn(0x7fe800144320 0x7fe80006daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:45.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:45.582+0000 7fe7fde00640 1 -- 172.21.15.6:0/3076862093 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 611633332 0 0) 0x7fe8000b3ea0 con 0x7fe80006e030 2024-03-04T09:32:45.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:45.582+0000 7fe7fde00640 1 -- 172.21.15.6:0/3076862093 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7e4003520 con 0x7fe80006e030 2024-03-04T09:32:45.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:45.582+0000 7fe7fde00640 1 -- 172.21.15.6:0/3076862093 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3202460874 0 0) 0x7fe7e4003520 con 0x7fe80006e030 2024-03-04T09:32:45.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:45.582+0000 7fe7fde00640 1 -- 172.21.15.6:0/3076862093 >> v2:172.21.15.146:3300/0 conn(0x7fe800144320 msgr2=0x7fe80006daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:32:45.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:45.583+0000 7fe7fde00640 1 --2- 172.21.15.6:0/3076862093 >> v2:172.21.15.146:3300/0 conn(0x7fe800144320 0x7fe80006daf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:45.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:45.583+0000 7fe7fde00640 1 -- 172.21.15.6:0/3076862093 >> v1:172.21.15.146:6789/0 conn(0x7fe8000727c0 legacy=0x7fe8000ad880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:45.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:45.583+0000 7fe7fde00640 1 -- 172.21.15.6:0/3076862093 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8000b4460 con 0x7fe80006e030 2024-03-04T09:32:45.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:45.583+0000 7fe7fde00640 1 -- 172.21.15.6:0/3076862093 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7fe7e8001670 con 0x7fe80006e030 2024-03-04T09:32:45.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:45.583+0000 7fe7fde00640 1 -- 172.21.15.6:0/3076862093 >> v1:172.21.15.6:6789/0 conn(0x7fe80006e030 legacy=0x7fe8000720b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:45.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:45.583+0000 7fe7fde00640 1 --2- 172.21.15.6:0/3076862093 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe7e40039b0 0x7fe7e4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:45.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:45.583+0000 7fe7fde00640 1 --2- 172.21.15.6:0/3076862093 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe7e4004800 0x7fe7e4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:45.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:45.583+0000 7fe7fde00640 1 --2- 172.21.15.6:0/3076862093 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe7e40051c0 0x7fe7e400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:45.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:45.583+0000 7fe7fde00640 1 -- 172.21.15.6:0/3076862093 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe80006e4a0 con 0x7fe7e40039b0 2024-03-04T09:32:45.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:45.583+0000 7fe7fde00640 1 -- 172.21.15.6:0/3076862093 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7e400c060 con 0x7fe7e40051c0 2024-03-04T09:32:45.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:45.583+0000 7fe7fde00640 1 -- 172.21.15.6:0/3076862093 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7e400c260 con 0x7fe7e4004800 2024-03-04T09:32:45.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:45.583+0000 7fe7fe800640 1 --2- 172.21.15.6:0/3076862093 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe7e40039b0 0x7fe7e4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:45.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:45.583+0000 7fe7fe800640 1 -- 172.21.15.6:0/3076862093 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe7e4004800 msgr2=0x7fe7e4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:32:45.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:45.583+0000 7fe7fe800640 1 --2- 172.21.15.6:0/3076862093 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe7e4004800 0x7fe7e4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:45.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:45.583+0000 7fe7fe800640 1 -- 172.21.15.6:0/3076862093 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe7e40051c0 msgr2=0x7fe7e400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:32:45.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:45.583+0000 7fe7ff200640 1 --2- 172.21.15.6:0/3076862093 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe7e40051c0 0x7fe7e400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:45.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:45.583+0000 7fe7fe800640 1 --2- 172.21.15.6:0/3076862093 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe7e40051c0 0x7fe7e400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:45.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:45.584+0000 7fe7fe800640 1 -- 172.21.15.6:0/3076862093 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7e4003520 con 0x7fe7e40039b0 2024-03-04T09:32:45.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:45.584+0000 7fe7ff200640 1 --2- 172.21.15.6:0/3076862093 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe7e40051c0 0x7fe7e400b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:32:45.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:45.584+0000 7fe7ff200640 1 -- 172.21.15.6:0/3076862093 reap_dead start 2024-03-04T09:32:45.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:45.585+0000 7fe7fe800640 1 --2- 172.21.15.6:0/3076862093 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe7e40039b0 0x7fe7e4003dd0 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7fe7f0009ac0 tx=0x7fe7e4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=328dbbb124048b51 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:45.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:45.585+0000 7fe7fde00640 1 -- 172.21.15.6:0/3076862093 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7f0039070 con 0x7fe7e40039b0 2024-03-04T09:32:45.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:45.585+0000 7fe7fde00640 1 -- 172.21.15.6:0/3076862093 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7f002c6b0 con 0x7fe7e40039b0 2024-03-04T09:32:45.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:45.585+0000 7fe7fde00640 1 -- 172.21.15.6:0/3076862093 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7f0009690 con 0x7fe7e40039b0 2024-03-04T09:32:45.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:45.585+0000 7fe804a00640 1 -- 172.21.15.6:0/3076862093 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe7e40039b0 msgr2=0x7fe7e4003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:45.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:45.585+0000 7fe804a00640 1 --2- 172.21.15.6:0/3076862093 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe7e40039b0 0x7fe7e4003dd0 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7fe7f0009ac0 tx=0x7fe7e4005830 comp rx=0 tx=0).stop 2024-03-04T09:32:45.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:45.585+0000 7fe804a00640 1 -- 172.21.15.6:0/3076862093 shutdown_connections 2024-03-04T09:32:45.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:45.585+0000 7fe804a00640 1 --2- 172.21.15.6:0/3076862093 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe7e40039b0 0x7fe7e4003dd0 unknown :-1 s=CLOSED pgs=1437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:45.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:45.586+0000 7fe804a00640 1 -- 172.21.15.6:0/3076862093 >> 172.21.15.6:0/3076862093 conn(0x7fe8000b17c0 msgr2=0x7fe8000b1ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:32:45.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:45.586+0000 7fe804a00640 1 -- 172.21.15.6:0/3076862093 shutdown_connections 2024-03-04T09:32:45.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:45.586+0000 7fe804a00640 1 -- 172.21.15.6:0/3076862093 wait complete. 2024-03-04T09:32:45.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:45.587+0000 7fe804a00640 1 Processor -- start 2024-03-04T09:32:45.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:45.588+0000 7fe804a00640 1 -- start start 2024-03-04T09:32:45.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:45.588+0000 7fe804a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe7e40039b0 0x7fe800146c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:45.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:45.588+0000 7fe804a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe800140300 0x7fe800140780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:45.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:45.588+0000 7fe804a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe80012e4b0 0x7fe80012e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:45.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:45.588+0000 7fe804a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8000b4f40 con 0x7fe80012e4b0 2024-03-04T09:32:45.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:45.588+0000 7fe804a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8000b4a20 con 0x7fe800140300 2024-03-04T09:32:45.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:45.588+0000 7fe804a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe800144700 con 0x7fe7e40039b0 2024-03-04T09:32:45.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:45.589+0000 7fe7ff200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe80012e4b0 0x7fe80012e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:45.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:45.589+0000 7fe7ff200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe80012e4b0 0x7fe80012e960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:47330/0 (socket says 172.21.15.6:47330) 2024-03-04T09:32:45.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:45.589+0000 7fe7ff200640 1 -- 172.21.15.6:0/2124358401 learned_addr learned my addr 172.21.15.6:0/2124358401 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:32:45.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:45.589+0000 7fe7fe800640 1 --2- 172.21.15.6:0/2124358401 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe7e40039b0 0x7fe800146c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:45.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:45.589+0000 7fe7ff200640 1 -- 172.21.15.6:0/2124358401 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe7e40039b0 msgr2=0x7fe800146c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:45.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:45.589+0000 7fe7f5e00640 1 --2- 172.21.15.6:0/2124358401 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe800140300 0x7fe800140780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:45.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:45.589+0000 7fe7ff200640 1 --2- 172.21.15.6:0/2124358401 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe7e40039b0 0x7fe800146c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:45.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:45.589+0000 7fe7ff200640 1 -- 172.21.15.6:0/2124358401 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe800140300 msgr2=0x7fe800140780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:45.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:45.589+0000 7fe7ff200640 1 --2- 172.21.15.6:0/2124358401 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe800140300 0x7fe800140780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:45.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:45.589+0000 7fe7ff200640 1 -- 172.21.15.6:0/2124358401 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe80012eea0 con 0x7fe80012e4b0 2024-03-04T09:32:45.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:45.590+0000 7fe7ff200640 1 --2- 172.21.15.6:0/2124358401 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe80012e4b0 0x7fe80012e960 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7fe80006c3b0 tx=0x7fe7f800d9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:45.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:45.590+0000 7fe7f7400640 1 -- 172.21.15.6:0/2124358401 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7f8018070 con 0x7fe80012e4b0 2024-03-04T09:32:45.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:45.590+0000 7fe7f7400640 1 -- 172.21.15.6:0/2124358401 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7f8003d80 con 0x7fe80012e4b0 2024-03-04T09:32:45.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:45.590+0000 7fe7f7400640 1 -- 172.21.15.6:0/2124358401 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7f80052f0 con 0x7fe80012e4b0 2024-03-04T09:32:45.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:45.590+0000 7fe804a00640 1 -- 172.21.15.6:0/2124358401 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe800145bf0 con 0x7fe80012e4b0 2024-03-04T09:32:45.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:45.590+0000 7fe804a00640 1 -- 172.21.15.6:0/2124358401 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8001461b0 con 0x7fe80012e4b0 2024-03-04T09:32:45.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:45.591+0000 7fe804a00640 1 -- 172.21.15.6:0/2124358401 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe80004f330 con 0x7fe80012e4b0 2024-03-04T09:32:45.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:45.592+0000 7fe7f7400640 1 -- 172.21.15.6:0/2124358401 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe7f801b070 con 0x7fe80012e4b0 2024-03-04T09:32:45.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:45.593+0000 7fe7f7400640 1 --2- 172.21.15.6:0/2124358401 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fe7dc06b550 0x7fe7dc06da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:45.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:45.593+0000 7fe7f7400640 1 -- 172.21.15.6:0/2124358401 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5501+0+0 (secure 0 0 0) 0x7fe7f80897b0 con 0x7fe80012e4b0 2024-03-04T09:32:45.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:45.594+0000 7fe7fe800640 1 --2- 172.21.15.6:0/2124358401 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fe7dc06b550 0x7fe7dc06da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:45.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:45.595+0000 7fe7fe800640 1 --2- 172.21.15.6:0/2124358401 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fe7dc06b550 0x7fe7dc06da10 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fe7f0036000 tx=0x7fe7f0030000 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:45.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:45.598+0000 7fe7f7400640 1 -- 172.21.15.6:0/2124358401 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe80012eea0 con 0x7fe80012e4b0 2024-03-04T09:32:45.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:45.797+0000 7fe804a00640 1 -- 172.21.15.6:0/2124358401 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fe8000b2380 con 0x7fe80012e4b0 2024-03-04T09:32:46.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:46.351+0000 7fe7f7400640 1 -- 172.21.15.6:0/2124358401 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v118) v1 ==== 169+0+0 (secure 0 0 0) 0x7fe7f8058770 con 0x7fe80012e4b0 2024-03-04T09:32:46.348 INFO:teuthology.orchestra.run.smithi006.stderr:pool 'cephfs_data' removed 2024-03-04T09:32:46.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:46.355+0000 7fe804a00640 1 -- 172.21.15.6:0/2124358401 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fe7dc06b550 msgr2=0x7fe7dc06da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:46.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:46.355+0000 7fe804a00640 1 --2- 172.21.15.6:0/2124358401 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fe7dc06b550 0x7fe7dc06da10 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fe7f0036000 tx=0x7fe7f0030000 comp rx=0 tx=0).stop 2024-03-04T09:32:46.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:46.355+0000 7fe804a00640 1 -- 172.21.15.6:0/2124358401 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe80012e4b0 msgr2=0x7fe80012e960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:46.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:46.355+0000 7fe804a00640 1 --2- 172.21.15.6:0/2124358401 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe80012e4b0 0x7fe80012e960 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7fe80006c3b0 tx=0x7fe7f800d9f0 comp rx=0 tx=0).stop 2024-03-04T09:32:46.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:46.358+0000 7fe804a00640 1 -- 172.21.15.6:0/2124358401 shutdown_connections 2024-03-04T09:32:46.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:46.358+0000 7fe804a00640 1 --2- 172.21.15.6:0/2124358401 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe80012e4b0 0x7fe80012e960 unknown :-1 s=CLOSED pgs=1438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:46.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:46.358+0000 7fe804a00640 1 --2- 172.21.15.6:0/2124358401 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe800140300 0x7fe800140780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:46.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:46.358+0000 7fe804a00640 1 --2- 172.21.15.6:0/2124358401 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fe7dc06b550 0x7fe7dc06da10 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:46.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:46.358+0000 7fe804a00640 1 --2- 172.21.15.6:0/2124358401 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe7e40039b0 0x7fe800146c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:46.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:46.358+0000 7fe804a00640 1 -- 172.21.15.6:0/2124358401 >> 172.21.15.6:0/2124358401 conn(0x7fe8000b17c0 msgr2=0x7fe8000af950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:32:46.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:46.364+0000 7fe804a00640 1 -- 172.21.15.6:0/2124358401 shutdown_connections 2024-03-04T09:32:46.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:46.364+0000 7fe804a00640 1 -- 172.21.15.6:0/2124358401 wait complete. 2024-03-04T09:32:46.386 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-04T09:32:46.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:46.585+0000 7f8895c00640 1 Processor -- start 2024-03-04T09:32:46.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:46.586+0000 7f8895c00640 1 -- start start 2024-03-04T09:32:46.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:46.587+0000 7f8895c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f88900d2bd0 0x7f88900cea80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:46.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:46.587+0000 7f8895c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f88900c8d20 0x7f88900cefc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:46.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:46.587+0000 7f8895c00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88900d20d0 con 0x7f8890161930 2024-03-04T09:32:46.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:46.587+0000 7f8895c00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88900d4df0 con 0x7f88900c8d20 2024-03-04T09:32:46.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:46.587+0000 7f8895c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88900d1df0 con 0x7f88900d2bd0 2024-03-04T09:32:46.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:46.587+0000 7f888f400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f88900c8d20 0x7f88900cefc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:46.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:46.587+0000 7f888f400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f88900c8d20 0x7f88900cefc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:47344/0 (socket says 172.21.15.6:47344) 2024-03-04T09:32:46.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:46.587+0000 7f888f400640 1 -- 172.21.15.6:0/2037724500 learned_addr learned my addr 172.21.15.6:0/2037724500 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:32:46.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:46.587+0000 7f888e000640 1 --2- 172.21.15.6:0/2037724500 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f88900d2bd0 0x7f88900cea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:46.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:46.588+0000 7f888f400640 1 -- 172.21.15.6:0/2037724500 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f88900d2bd0 msgr2=0x7f88900cea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:32:46.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:46.588+0000 7f888f400640 1 --2- 172.21.15.6:0/2037724500 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f88900d2bd0 0x7f88900cea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:46.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:46.588+0000 7f888f400640 1 -- 172.21.15.6:0/2037724500 >> v1:172.21.15.146:6789/0 conn(0x7f8890161930 legacy=0x7f88900d24c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:46.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:46.588+0000 7f888f400640 1 -- 172.21.15.6:0/2037724500 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88900d4b10 con 0x7f88900c8d20 2024-03-04T09:32:46.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:46.589+0000 7f888f400640 1 --2- 172.21.15.6:0/2037724500 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f88900c8d20 0x7f88900cefc0 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7f8880009870 tx=0x7f888002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adb173d583947b92 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:46.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:46.589+0000 7f888d600640 1 -- 172.21.15.6:0/2037724500 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8880035020 con 0x7f88900c8d20 2024-03-04T09:32:46.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:46.589+0000 7f888d600640 1 -- 172.21.15.6:0/2037724500 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8880031640 con 0x7f88900c8d20 2024-03-04T09:32:46.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:46.589+0000 7f888d600640 1 -- 172.21.15.6:0/2037724500 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f888002d040 con 0x7f88900c8d20 2024-03-04T09:32:46.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:46.590+0000 7f8895c00640 1 -- 172.21.15.6:0/2037724500 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f88900c8d20 msgr2=0x7f88900cefc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:46.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:46.590+0000 7f8895c00640 1 --2- 172.21.15.6:0/2037724500 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f88900c8d20 0x7f88900cefc0 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7f8880009870 tx=0x7f888002b8f0 comp rx=0 tx=0).stop 2024-03-04T09:32:46.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:46.590+0000 7f8895c00640 1 -- 172.21.15.6:0/2037724500 shutdown_connections 2024-03-04T09:32:46.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:46.590+0000 7f8895c00640 1 --2- 172.21.15.6:0/2037724500 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f88900c8d20 0x7f88900cefc0 unknown :-1 s=CLOSED pgs=1439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:46.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:46.590+0000 7f8895c00640 1 --2- 172.21.15.6:0/2037724500 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f88900d2bd0 0x7f88900cea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:46.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:46.590+0000 7f8895c00640 1 -- 172.21.15.6:0/2037724500 >> 172.21.15.6:0/2037724500 conn(0x7f88900d3ee0 msgr2=0x7f88900d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:32:46.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:46.590+0000 7f8895c00640 1 -- 172.21.15.6:0/2037724500 shutdown_connections 2024-03-04T09:32:46.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:46.590+0000 7f8895c00640 1 -- 172.21.15.6:0/2037724500 wait complete. 2024-03-04T09:32:46.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:46.592+0000 7f8895c00640 1 Processor -- start 2024-03-04T09:32:46.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:46.592+0000 7f8895c00640 1 -- start start 2024-03-04T09:32:46.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:46.593+0000 7f8895c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f88900c8d20 0x7f88901514e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:46.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:46.593+0000 7f8895c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f88900d2bd0 0x7f88901544a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:46.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:46.593+0000 7f8895c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8890161930 0x7f889015d730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:46.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:46.593+0000 7f8895c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88900d0f80 con 0x7f88900d2bd0 2024-03-04T09:32:46.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:46.593+0000 7f8895c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88900b4f30 con 0x7f8890161930 2024-03-04T09:32:46.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:46.593+0000 7f8895c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88900d1100 con 0x7f88900c8d20 2024-03-04T09:32:46.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:46.593+0000 7f888e000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f88900d2bd0 0x7f88901544a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:46.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:46.593+0000 7f888e000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f88900d2bd0 0x7f88901544a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:47352/0 (socket says 172.21.15.6:47352) 2024-03-04T09:32:46.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:46.593+0000 7f888e000640 1 -- 172.21.15.6:0/2218636554 learned_addr learned my addr 172.21.15.6:0/2218636554 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:32:46.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:46.593+0000 7f888f400640 1 --2- 172.21.15.6:0/2218636554 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8890161930 0x7f889015d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:46.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:46.593+0000 7f888ea00640 1 --2- 172.21.15.6:0/2218636554 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f88900c8d20 0x7f88901514e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:46.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:46.593+0000 7f888e000640 1 -- 172.21.15.6:0/2218636554 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f88900c8d20 msgr2=0x7f88901514e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:46.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:46.593+0000 7f888e000640 1 --2- 172.21.15.6:0/2218636554 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f88900c8d20 0x7f88901514e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:46.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:46.593+0000 7f888e000640 1 -- 172.21.15.6:0/2218636554 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8890161930 msgr2=0x7f889015d730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:46.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:46.593+0000 7f888e000640 1 --2- 172.21.15.6:0/2218636554 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8890161930 0x7f889015d730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:46.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:46.593+0000 7f888e000640 1 -- 172.21.15.6:0/2218636554 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f889015bfe0 con 0x7f88900d2bd0 2024-03-04T09:32:46.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:46.593+0000 7f888f400640 1 --2- 172.21.15.6:0/2218636554 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8890161930 0x7f889015d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:32:46.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:46.594+0000 7f888e000640 1 --2- 172.21.15.6:0/2218636554 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f88900d2bd0 0x7f88901544a0 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7f887800d5f0 tx=0x7f887800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:46.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:46.594+0000 7f8886a00640 1 -- 172.21.15.6:0/2218636554 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88780125f0 con 0x7f88900d2bd0 2024-03-04T09:32:46.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:46.594+0000 7f8886a00640 1 -- 172.21.15.6:0/2218636554 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8878019070 con 0x7f88900d2bd0 2024-03-04T09:32:46.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:46.594+0000 7f8886a00640 1 -- 172.21.15.6:0/2218636554 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88780129d0 con 0x7f88900d2bd0 2024-03-04T09:32:46.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:46.594+0000 7f8895c00640 1 -- 172.21.15.6:0/2218636554 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f889015c260 con 0x7f88900d2bd0 2024-03-04T09:32:46.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:46.594+0000 7f8895c00640 1 -- 172.21.15.6:0/2218636554 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f889015c820 con 0x7f88900d2bd0 2024-03-04T09:32:46.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:46.595+0000 7f8895c00640 1 -- 172.21.15.6:0/2218636554 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f889004f330 con 0x7f88900d2bd0 2024-03-04T09:32:46.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:46.596+0000 7f8886a00640 1 -- 172.21.15.6:0/2218636554 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8878012b70 con 0x7f88900d2bd0 2024-03-04T09:32:46.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:46.597+0000 7f8886a00640 1 --2- 172.21.15.6:0/2218636554 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f886c06b4d0 0x7f886c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:46.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:46.597+0000 7f8886a00640 1 -- 172.21.15.6:0/2218636554 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f887808a360 con 0x7f88900d2bd0 2024-03-04T09:32:46.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:46.598+0000 7f888ea00640 1 --2- 172.21.15.6:0/2218636554 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f886c06b4d0 0x7f886c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:46.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:46.599+0000 7f888ea00640 1 --2- 172.21.15.6:0/2218636554 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f886c06b4d0 0x7f886c06d990 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f8888007fd0 tx=0x7f8888009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:46.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:46.602+0000 7f8886a00640 1 -- 172.21.15.6:0/2218636554 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f889015bfe0 con 0x7f88900d2bd0 2024-03-04T09:32:46.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:46.802+0000 7f8895c00640 1 -- 172.21.15.6:0/2218636554 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f88900ce3a0 con 0x7f88900d2bd0 2024-03-04T09:32:46.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:46.808+0000 7f8886a00640 1 -- 172.21.15.6:0/2218636554 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v31) v1 ==== 104+0+0 (secure 0 0 0) 0x7f8878059470 con 0x7f88900d2bd0 2024-03-04T09:32:46.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:46.810+0000 7f8895c00640 1 -- 172.21.15.6:0/2218636554 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f886c06b4d0 msgr2=0x7f886c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:46.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:46.810+0000 7f8895c00640 1 --2- 172.21.15.6:0/2218636554 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f886c06b4d0 0x7f886c06d990 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f8888007fd0 tx=0x7f8888009040 comp rx=0 tx=0).stop 2024-03-04T09:32:46.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:46.810+0000 7f8895c00640 1 -- 172.21.15.6:0/2218636554 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f88900d2bd0 msgr2=0x7f88901544a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:46.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:46.810+0000 7f8895c00640 1 --2- 172.21.15.6:0/2218636554 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f88900d2bd0 0x7f88901544a0 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7f887800d5f0 tx=0x7f887800dab0 comp rx=0 tx=0).stop 2024-03-04T09:32:46.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:46.811+0000 7f8895c00640 1 -- 172.21.15.6:0/2218636554 shutdown_connections 2024-03-04T09:32:46.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:46.811+0000 7f8895c00640 1 --2- 172.21.15.6:0/2218636554 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8890161930 0x7f889015d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:46.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:46.811+0000 7f8895c00640 1 --2- 172.21.15.6:0/2218636554 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f88900d2bd0 0x7f88901544a0 unknown :-1 s=CLOSED pgs=1440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:46.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:46.811+0000 7f8895c00640 1 --2- 172.21.15.6:0/2218636554 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f886c06b4d0 0x7f886c06d990 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:46.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:46.811+0000 7f8895c00640 1 --2- 172.21.15.6:0/2218636554 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f88900c8d20 0x7f88901514e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:46.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:46.811+0000 7f8895c00640 1 -- 172.21.15.6:0/2218636554 >> 172.21.15.6:0/2218636554 conn(0x7f88900d3ee0 msgr2=0x7f88900cfce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:32:46.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:46.811+0000 7f8895c00640 1 -- 172.21.15.6:0/2218636554 shutdown_connections 2024-03-04T09:32:46.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:46.811+0000 7f8895c00640 1 -- 172.21.15.6:0/2218636554 wait complete. 2024-03-04T09:32:46.827 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_hardlink_reintegration' 2024-03-04T09:32:47.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.052+0000 7f476f200640 1 Processor -- start 2024-03-04T09:32:47.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.052+0000 7f476f200640 1 -- start start 2024-03-04T09:32:47.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.053+0000 7f476f200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f47701552b0 0x7f477015b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:47.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.053+0000 7f476f200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f477015e810 0x7f47700cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:47.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.054+0000 7f476f200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f477015ed10 con 0x7f47700cce90 2024-03-04T09:32:47.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.054+0000 7f476f200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f477015eff0 con 0x7f47701552b0 2024-03-04T09:32:47.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.054+0000 7f476f200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f477015f2d0 con 0x7f477015e810 2024-03-04T09:32:47.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.054+0000 7f476de00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f47701552b0 0x7f477015b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:47.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.054+0000 7f476de00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f47701552b0 0x7f477015b1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:47364/0 (socket says 172.21.15.6:47364) 2024-03-04T09:32:47.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.054+0000 7f476de00640 1 -- 172.21.15.6:0/3294488023 learned_addr learned my addr 172.21.15.6:0/3294488023 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:32:47.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.054+0000 7f4767e00640 1 --2- 172.21.15.6:0/3294488023 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f477015e810 0x7f47700cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:47.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.055+0000 7f476de00640 1 -- 172.21.15.6:0/3294488023 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f477015e810 msgr2=0x7f47700cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:32:47.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.055+0000 7f476de00640 1 --2- 172.21.15.6:0/3294488023 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f477015e810 0x7f47700cc950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:47.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.055+0000 7f476de00640 1 -- 172.21.15.6:0/3294488023 >> v1:172.21.15.146:6789/0 conn(0x7f47700cce90 legacy=0x7f477006c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:32:47.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.055+0000 7f476de00640 1 -- 172.21.15.6:0/3294488023 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f477015f5b0 con 0x7f47701552b0 2024-03-04T09:32:47.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.056+0000 7f476de00640 1 --2- 172.21.15.6:0/3294488023 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f47701552b0 0x7f477015b1d0 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7f4760009870 tx=0x7f476002b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23c5f74724c37784 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:47.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.056+0000 7f476d400640 1 -- 172.21.15.6:0/3294488023 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47600317c0 con 0x7f47701552b0 2024-03-04T09:32:47.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.056+0000 7f476d400640 1 -- 172.21.15.6:0/3294488023 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4760037070 con 0x7f47701552b0 2024-03-04T09:32:47.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.056+0000 7f476d400640 1 -- 172.21.15.6:0/3294488023 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4760031b80 con 0x7f47701552b0 2024-03-04T09:32:47.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.057+0000 7f476f200640 1 -- 172.21.15.6:0/3294488023 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f47701552b0 msgr2=0x7f477015b1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:47.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.057+0000 7f476f200640 1 --2- 172.21.15.6:0/3294488023 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f47701552b0 0x7f477015b1d0 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7f4760009870 tx=0x7f476002b860 comp rx=0 tx=0).stop 2024-03-04T09:32:47.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.057+0000 7f476f200640 1 -- 172.21.15.6:0/3294488023 shutdown_connections 2024-03-04T09:32:47.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.057+0000 7f476f200640 1 --2- 172.21.15.6:0/3294488023 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f477015e810 0x7f47700cc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:47.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.057+0000 7f476f200640 1 --2- 172.21.15.6:0/3294488023 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f47701552b0 0x7f477015b1d0 unknown :-1 s=CLOSED pgs=1441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:47.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.057+0000 7f476f200640 1 -- 172.21.15.6:0/3294488023 >> 172.21.15.6:0/3294488023 conn(0x7f477006b3d0 msgr2=0x7f477006b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:32:47.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.057+0000 7f476f200640 1 -- 172.21.15.6:0/3294488023 shutdown_connections 2024-03-04T09:32:47.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.057+0000 7f476f200640 1 -- 172.21.15.6:0/3294488023 wait complete. 2024-03-04T09:32:47.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.059+0000 7f476f200640 1 Processor -- start 2024-03-04T09:32:47.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.059+0000 7f476f200640 1 -- start start 2024-03-04T09:32:47.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.060+0000 7f476f200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f47700cce90 0x7f4770141ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:47.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.060+0000 7f476f200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f47701552b0 0x7f4770167ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:47.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.060+0000 7f476f200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f477015e810 0x7f4770159cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:47.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.060+0000 7f476f200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f477006dd30 con 0x7f47700cce90 2024-03-04T09:32:47.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.060+0000 7f476f200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47700b3760 con 0x7f47701552b0 2024-03-04T09:32:47.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.060+0000 7f476f200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f477006deb0 con 0x7f477015e810 2024-03-04T09:32:47.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.060+0000 7f476de00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f47700cce90 0x7f4770141ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:47.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.060+0000 7f476de00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f47700cce90 0x7f4770141ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:47370/0 (socket says 172.21.15.6:47370) 2024-03-04T09:32:47.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.060+0000 7f476de00640 1 -- 172.21.15.6:0/3952979251 learned_addr learned my addr 172.21.15.6:0/3952979251 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:32:47.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.060+0000 7f4767e00640 1 --2- 172.21.15.6:0/3952979251 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f47701552b0 0x7f4770167ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:47.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.060+0000 7f476e800640 1 --2- 172.21.15.6:0/3952979251 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f477015e810 0x7f4770159cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:47.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.061+0000 7f476de00640 1 -- 172.21.15.6:0/3952979251 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f477015e810 msgr2=0x7f4770159cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:47.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.061+0000 7f476de00640 1 --2- 172.21.15.6:0/3952979251 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f477015e810 0x7f4770159cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:47.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.061+0000 7f476de00640 1 -- 172.21.15.6:0/3952979251 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f47701552b0 msgr2=0x7f4770167ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:47.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.061+0000 7f476de00640 1 --2- 172.21.15.6:0/3952979251 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f47701552b0 0x7f4770167ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:47.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.061+0000 7f476de00640 1 -- 172.21.15.6:0/3952979251 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47701f3f50 con 0x7f47700cce90 2024-03-04T09:32:47.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.061+0000 7f476e800640 1 --2- 172.21.15.6:0/3952979251 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f477015e810 0x7f4770159cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:32:47.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.061+0000 7f476de00640 1 --2- 172.21.15.6:0/3952979251 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f47700cce90 0x7f4770141ef0 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7f4760000c00 tx=0x7f47600046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:47.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.061+0000 7f4766000640 1 -- 172.21.15.6:0/3952979251 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4760032860 con 0x7f47700cce90 2024-03-04T09:32:47.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.061+0000 7f4766000640 1 -- 172.21.15.6:0/3952979251 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4760032a00 con 0x7f47700cce90 2024-03-04T09:32:47.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.061+0000 7f4766000640 1 -- 172.21.15.6:0/3952979251 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4760032df0 con 0x7f47700cce90 2024-03-04T09:32:47.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.061+0000 7f476f200640 1 -- 172.21.15.6:0/3952979251 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f477015d7d0 con 0x7f47700cce90 2024-03-04T09:32:47.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.061+0000 7f476f200640 1 -- 172.21.15.6:0/3952979251 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f477015dd90 con 0x7f47700cce90 2024-03-04T09:32:47.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.063+0000 7f476f200640 1 -- 172.21.15.6:0/3952979251 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47701f4120 con 0x7f47700cce90 2024-03-04T09:32:47.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.063+0000 7f4766000640 1 -- 172.21.15.6:0/3952979251 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f476004c060 con 0x7f47700cce90 2024-03-04T09:32:47.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.064+0000 7f4766000640 1 --2- 172.21.15.6:0/3952979251 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f474c06b4d0 0x7f474c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:47.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.064+0000 7f4766000640 1 -- 172.21.15.6:0/3952979251 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f47600abb40 con 0x7f47700cce90 2024-03-04T09:32:47.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.065+0000 7f4767e00640 1 --2- 172.21.15.6:0/3952979251 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f474c06b4d0 0x7f474c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:47.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.066+0000 7f4767e00640 1 --2- 172.21.15.6:0/3952979251 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f474c06b4d0 0x7f474c06d990 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f47700717f0 tx=0x7f4758008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:47.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.069+0000 7f4766000640 1 -- 172.21.15.6:0/3952979251 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4760040070 con 0x7f47700cce90 2024-03-04T09:32:47.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.341+0000 7f476f200640 1 -- 172.21.15.6:0/3952979251 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_hardlink_reintegration"]} v 0) v1 -- 0x7f4770143770 con 0x7f47700cce90 2024-03-04T09:32:47.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.354+0000 7f4766000640 1 -- 172.21.15.6:0/3952979251 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_hardlink_reintegration"]}]=0 v414) v1 ==== 142+0+0 (secure 0 0 0) 0x7f4770143770 con 0x7f47700cce90 2024-03-04T09:32:47.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.357+0000 7f476f200640 1 -- 172.21.15.6:0/3952979251 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f474c06b4d0 msgr2=0x7f474c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:47.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.357+0000 7f476f200640 1 --2- 172.21.15.6:0/3952979251 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f474c06b4d0 0x7f474c06d990 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f47700717f0 tx=0x7f4758008040 comp rx=0 tx=0).stop 2024-03-04T09:32:47.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.357+0000 7f476f200640 1 -- 172.21.15.6:0/3952979251 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f47700cce90 msgr2=0x7f4770141ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:47.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.357+0000 7f476f200640 1 --2- 172.21.15.6:0/3952979251 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f47700cce90 0x7f4770141ef0 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7f4760000c00 tx=0x7f47600046c0 comp rx=0 tx=0).stop 2024-03-04T09:32:47.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.357+0000 7f476f200640 1 -- 172.21.15.6:0/3952979251 shutdown_connections 2024-03-04T09:32:47.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.357+0000 7f476f200640 1 --2- 172.21.15.6:0/3952979251 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f474c06b4d0 0x7f474c06d990 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:47.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.357+0000 7f476f200640 1 --2- 172.21.15.6:0/3952979251 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f477015e810 0x7f4770159cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:47.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.357+0000 7f476f200640 1 --2- 172.21.15.6:0/3952979251 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f47701552b0 0x7f4770167ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:47.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.357+0000 7f476f200640 1 --2- 172.21.15.6:0/3952979251 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f47700cce90 0x7f4770141ef0 unknown :-1 s=CLOSED pgs=1442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:47.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.357+0000 7f476f200640 1 -- 172.21.15.6:0/3952979251 >> 172.21.15.6:0/3952979251 conn(0x7f477006b3d0 msgr2=0x7f4770071840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:32:47.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.358+0000 7f476f200640 1 -- 172.21.15.6:0/3952979251 shutdown_connections 2024-03-04T09:32:47.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.358+0000 7f476f200640 1 -- 172.21.15.6:0/3952979251 wait complete. 2024-03-04T09:32:47.374 INFO:tasks.cephfs_test_runner:That removal of primary dentry of hardlinked inode results ... ok 2024-03-04T09:32:47.375 INFO:tasks.cephfs_test_runner:Starting test: test_migrate_unlinked_dir (tasks.cephfs.test_strays.TestStrays) Reproduce https://tracker.ceph.com/issues/53597 2024-03-04T09:32:47.375 INFO:tasks.cephfs_test_runner:test_migrate_unlinked_dir (tasks.cephfs.test_strays.TestStrays) 2024-03-04T09:32:47.375 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T09:32:47.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.594+0000 7f4dab200640 1 Processor -- start 2024-03-04T09:32:47.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.595+0000 7f4dab200640 1 -- start start 2024-03-04T09:32:47.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.596+0000 7f4dab200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4dac0d0e50 0x7f4dac168c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:47.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.596+0000 7f4dab200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dac0b3710 con 0x7f4dac15e0b0 2024-03-04T09:32:47.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.596+0000 7f4dab200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dac0b39f0 con 0x7f4dac0d4a10 2024-03-04T09:32:47.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.596+0000 7f4dab200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4dac0b09e0 con 0x7f4dac0d0e50 2024-03-04T09:32:47.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.596+0000 7f4da9e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f4dac15e0b0 0x7f4dac0cf310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:35154/0 (socket says 172.21.15.6:35154) 2024-03-04T09:32:47.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.596+0000 7f4da9e00640 1 -- 172.21.15.6:0/1640352414 learned_addr learned my addr 172.21.15.6:0/1640352414 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:32:47.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.596+0000 7f4da8a00640 1 -- 172.21.15.6:0/1640352414 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3826805985 0 0) 0x7f4dac0b3710 con 0x7f4dac15e0b0 2024-03-04T09:32:47.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.596+0000 7f4da8a00640 1 -- 172.21.15.6:0/1640352414 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d8c003520 con 0x7f4dac15e0b0 2024-03-04T09:32:47.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.597+0000 7f4daa800640 1 --2- 172.21.15.6:0/1640352414 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4dac0d0e50 0x7f4dac168c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:47.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.597+0000 7f4da8a00640 1 -- 172.21.15.6:0/1640352414 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1292831104 0 0) 0x7f4d8c003520 con 0x7f4dac15e0b0 2024-03-04T09:32:47.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.597+0000 7f4da8a00640 1 -- 172.21.15.6:0/1640352414 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4dac0d0e50 msgr2=0x7f4dac168c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:32:47.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.597+0000 7f4da8a00640 1 --2- 172.21.15.6:0/1640352414 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4dac0d0e50 0x7f4dac168c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:47.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.597+0000 7f4da8a00640 1 -- 172.21.15.6:0/1640352414 >> v1:172.21.15.146:6789/0 conn(0x7f4dac0d4a10 legacy=0x7f4dac0d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:47.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.597+0000 7f4da8a00640 1 -- 172.21.15.6:0/1640352414 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dac0b3cd0 con 0x7f4dac15e0b0 2024-03-04T09:32:47.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.597+0000 7f4da8a00640 1 -- 172.21.15.6:0/1640352414 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7f4d9c001690 con 0x7f4dac15e0b0 2024-03-04T09:32:47.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.597+0000 7f4da8a00640 1 -- 172.21.15.6:0/1640352414 >> v1:172.21.15.6:6789/0 conn(0x7f4dac15e0b0 legacy=0x7f4dac0cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:47.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.597+0000 7f4da8a00640 1 --2- 172.21.15.6:0/1640352414 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4d8c0039e0 0x7f4d8c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:47.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.597+0000 7f4da8a00640 1 --2- 172.21.15.6:0/1640352414 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4d8c004800 0x7f4d8c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:47.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.597+0000 7f4da8a00640 1 --2- 172.21.15.6:0/1640352414 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4dac0d0e50 0x7f4d8c005250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:47.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.597+0000 7f4da8a00640 1 -- 172.21.15.6:0/1640352414 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d8c00bb50 con 0x7f4d8c0039e0 2024-03-04T09:32:47.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.597+0000 7f4da8a00640 1 -- 172.21.15.6:0/1640352414 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d8c00bd00 con 0x7f4d8c004800 2024-03-04T09:32:47.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.597+0000 7f4da8a00640 1 -- 172.21.15.6:0/1640352414 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d8c00bf00 con 0x7f4dac0d0e50 2024-03-04T09:32:47.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.597+0000 7f4da9e00640 1 --2- 172.21.15.6:0/1640352414 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4d8c0039e0 0x7f4d8c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:47.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.598+0000 7f4da9400640 1 --2- 172.21.15.6:0/1640352414 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4d8c004800 0x7f4d8c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:47.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.598+0000 7f4daa800640 1 --2- 172.21.15.6:0/1640352414 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4dac0d0e50 0x7f4d8c005250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:47.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.598+0000 7f4da9e00640 1 -- 172.21.15.6:0/1640352414 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4dac0d0e50 msgr2=0x7f4d8c005250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:32:47.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.598+0000 7f4da9e00640 1 --2- 172.21.15.6:0/1640352414 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4dac0d0e50 0x7f4d8c005250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:47.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.598+0000 7f4da9e00640 1 -- 172.21.15.6:0/1640352414 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4d8c004800 msgr2=0x7f4d8c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:32:47.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.598+0000 7f4da9e00640 1 --2- 172.21.15.6:0/1640352414 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4d8c004800 0x7f4d8c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:47.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.598+0000 7f4da9e00640 1 -- 172.21.15.6:0/1640352414 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d8c003520 con 0x7f4d8c0039e0 2024-03-04T09:32:47.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.598+0000 7f4daa800640 1 --2- 172.21.15.6:0/1640352414 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4dac0d0e50 0x7f4d8c005250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:32:47.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.599+0000 7f4da9e00640 1 --2- 172.21.15.6:0/1640352414 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4d8c0039e0 0x7f4d8c003e00 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7f4dac0d06f0 tx=0x7f4d8c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89344860ea28e752 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:47.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.599+0000 7f4da8a00640 1 -- 172.21.15.6:0/1640352414 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d9c007c80 con 0x7f4d8c0039e0 2024-03-04T09:32:47.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.599+0000 7f4da8a00640 1 -- 172.21.15.6:0/1640352414 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d9c030070 con 0x7f4d8c0039e0 2024-03-04T09:32:47.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.599+0000 7f4da8a00640 1 -- 172.21.15.6:0/1640352414 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d9c005cf0 con 0x7f4d8c0039e0 2024-03-04T09:32:47.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.600+0000 7f4dab200640 1 -- 172.21.15.6:0/1640352414 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4d8c0039e0 msgr2=0x7f4d8c003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:47.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.600+0000 7f4dab200640 1 --2- 172.21.15.6:0/1640352414 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4d8c0039e0 0x7f4d8c003e00 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7f4dac0d06f0 tx=0x7f4d8c005830 comp rx=0 tx=0).stop 2024-03-04T09:32:47.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.600+0000 7f4daa800640 1 -- 172.21.15.6:0/1640352414 reap_dead start 2024-03-04T09:32:47.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.600+0000 7f4dab200640 1 -- 172.21.15.6:0/1640352414 shutdown_connections 2024-03-04T09:32:47.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.600+0000 7f4dab200640 1 -- 172.21.15.6:0/1640352414 >> 172.21.15.6:0/1640352414 conn(0x7f4dac0d3d30 msgr2=0x7f4dac0d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:32:47.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.600+0000 7f4dab200640 1 -- 172.21.15.6:0/1640352414 shutdown_connections 2024-03-04T09:32:47.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.600+0000 7f4dab200640 1 -- 172.21.15.6:0/1640352414 wait complete. 2024-03-04T09:32:47.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.601+0000 7f4dab200640 1 Processor -- start 2024-03-04T09:32:47.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.602+0000 7f4dab200640 1 -- start start 2024-03-04T09:32:47.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.603+0000 7f4dab200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4dac1523d0 0x7f4dac1f3d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:47.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.603+0000 7f4dab200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4dac1f4280 0x7f4dac0b9310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:47.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.603+0000 7f4dab200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4dac0b9850 0x7f4dac06c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:47.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.603+0000 7f4dab200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dac15e490 con 0x7f4dac1f4280 2024-03-04T09:32:47.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.603+0000 7f4dab200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dac0cfa20 con 0x7f4dac0b9850 2024-03-04T09:32:47.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.603+0000 7f4dab200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4dac0cfc60 con 0x7f4dac1523d0 2024-03-04T09:32:47.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.603+0000 7f4da9400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4dac1f4280 0x7f4dac0b9310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:47.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.603+0000 7f4da9400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4dac1f4280 0x7f4dac0b9310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:47398/0 (socket says 172.21.15.6:47398) 2024-03-04T09:32:47.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.603+0000 7f4da9400640 1 -- 172.21.15.6:0/58099244 learned_addr learned my addr 172.21.15.6:0/58099244 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:32:47.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.603+0000 7f4da9e00640 1 --2- 172.21.15.6:0/58099244 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4dac1523d0 0x7f4dac1f3d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:47.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.603+0000 7f4daa800640 1 --2- 172.21.15.6:0/58099244 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4dac0b9850 0x7f4dac06c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:47.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.603+0000 7f4da9400640 1 -- 172.21.15.6:0/58099244 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4dac1523d0 msgr2=0x7f4dac1f3d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:47.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.603+0000 7f4da9400640 1 --2- 172.21.15.6:0/58099244 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4dac1523d0 0x7f4dac1f3d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:47.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.603+0000 7f4da9400640 1 -- 172.21.15.6:0/58099244 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4dac0b9850 msgr2=0x7f4dac06c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:47.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.603+0000 7f4da9400640 1 --2- 172.21.15.6:0/58099244 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4dac0b9850 0x7f4dac06c0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:47.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.603+0000 7f4da9400640 1 -- 172.21.15.6:0/58099244 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dac0cc4e0 con 0x7f4dac1f4280 2024-03-04T09:32:47.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.603+0000 7f4daa800640 1 --2- 172.21.15.6:0/58099244 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4dac0b9850 0x7f4dac06c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:32:47.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.604+0000 7f4da9400640 1 --2- 172.21.15.6:0/58099244 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4dac1f4280 0x7f4dac0b9310 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7f4d9400e7f0 tx=0x7f4d9400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:47.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.604+0000 7f4da2000640 1 -- 172.21.15.6:0/58099244 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d94003b80 con 0x7f4dac1f4280 2024-03-04T09:32:47.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.604+0000 7f4da2000640 1 -- 172.21.15.6:0/58099244 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d94018070 con 0x7f4dac1f4280 2024-03-04T09:32:47.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.604+0000 7f4da2000640 1 -- 172.21.15.6:0/58099244 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d940050d0 con 0x7f4dac1f4280 2024-03-04T09:32:47.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.604+0000 7f4dab200640 1 -- 172.21.15.6:0/58099244 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4dac0cc7c0 con 0x7f4dac1f4280 2024-03-04T09:32:47.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.604+0000 7f4dab200640 1 -- 172.21.15.6:0/58099244 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4dac0ccd80 con 0x7f4dac1f4280 2024-03-04T09:32:47.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.605+0000 7f4dab200640 1 -- 172.21.15.6:0/58099244 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4dac04f330 con 0x7f4dac1f4280 2024-03-04T09:32:47.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.606+0000 7f4da2000640 1 -- 172.21.15.6:0/58099244 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4d94005270 con 0x7f4dac1f4280 2024-03-04T09:32:47.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.607+0000 7f4da2000640 1 --2- 172.21.15.6:0/58099244 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f4d8806b500 0x7f4d8806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:47.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.607+0000 7f4da2000640 1 -- 172.21.15.6:0/58099244 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f4d94089f90 con 0x7f4dac1f4280 2024-03-04T09:32:47.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.608+0000 7f4da9e00640 1 --2- 172.21.15.6:0/58099244 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f4d8806b500 0x7f4d8806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:47.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.609+0000 7f4da9e00640 1 --2- 172.21.15.6:0/58099244 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f4d8806b500 0x7f4d8806d9c0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f4d9c034000 tx=0x7f4d9c009030 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:47.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.612+0000 7f4da2000640 1 -- 172.21.15.6:0/58099244 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4d94058f80 con 0x7f4dac1f4280 2024-03-04T09:32:47.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.812+0000 7f4dab200640 1 -- 172.21.15.6:0/58099244 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4dac0b55f0 con 0x7f4dac1f4280 2024-03-04T09:32:47.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.814+0000 7f4da2000640 1 -- 172.21.15.6:0/58099244 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v118) v1 ==== 74+0+13106 (secure 0 0 0) 0x7f4d9401be40 con 0x7f4dac1f4280 2024-03-04T09:32:47.811 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:32:47.811 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":118,"fsid":"edce0578-5ae4-4402-b45a-552bdef84d7c","created":"2024-03-04T09:24:54.523780+0000","modified":"2024-03-04T09:32:46.340893+0000","last_up_change":"2024-03-04T09:25:07.089049+0000","last_in_change":"2024-03-04T09:24:57.982077+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T09:25:07.935198+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a96574f-81f1-4bc6-8463-757520b84a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6825","nonce":1364449401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6827","nonce":1364449401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6831","nonce":1364449401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6829","nonce":1364449401}]},"public_addr":"172.21.15.6:6825/1364449401","cluster_addr":"172.21.15.6:6827/1364449401","heartbeat_back_addr":"172.21.15.6:6831/1364449401","heartbeat_front_addr":"172.21.15.6:6829/1364449401","state":["exists","up"]},{"osd":1,"uuid":"7d79b9c1-6b87-476a-bc36-7db5ee64cd90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6809","nonce":2494103078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6811","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6813","nonce":2494103078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6821","nonce":2494103078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6817","nonce":2494103078}]},"public_addr":"172.21.15.6:6809/2494103078","cluster_addr":"172.21.15.6:6813/2494103078","heartbeat_back_addr":"172.21.15.6:6821/2494103078","heartbeat_front_addr":"172.21.15.6:6817/2494103078","state":["exists","up"]},{"osd":2,"uuid":"492331b9-4ee9-40b6-bacd-85d44cd6c45a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6812","nonce":1500066681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6816","nonce":1500066681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6824","nonce":1500066681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6820","nonce":1500066681}]},"public_addr":"172.21.15.6:6812/1500066681","cluster_addr":"172.21.15.6:6816/1500066681","heartbeat_back_addr":"172.21.15.6:6824/1500066681","heartbeat_front_addr":"172.21.15.6:6820/1500066681","state":["exists","up"]},{"osd":3,"uuid":"21bb2efe-9ba7-4bdc-bc31-180cd1102e21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6801","nonce":1558806123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6803","nonce":1558806123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6807","nonce":1558806123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6805","nonce":1558806123}]},"public_addr":"172.21.15.6:6801/1558806123","cluster_addr":"172.21.15.6:6803/1558806123","heartbeat_back_addr":"172.21.15.6:6807/1558806123","heartbeat_front_addr":"172.21.15.6:6805/1558806123","state":["exists","up"]},{"osd":4,"uuid":"6e2c4b71-19d9-4737-83d9-558705d8644a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6801","nonce":1526135780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6803","nonce":1526135780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6807","nonce":1526135780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6805","nonce":1526135780}]},"public_addr":"172.21.15.146:6801/1526135780","cluster_addr":"172.21.15.146:6803/1526135780","heartbeat_back_addr":"172.21.15.146:6807/1526135780","heartbeat_front_addr":"172.21.15.146:6805/1526135780","state":["exists","up"]},{"osd":5,"uuid":"fb69c525-f1be-4b7d-ab32-6c29b0d6fddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6809","nonce":460262507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6811","nonce":460262507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6815","nonce":460262507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6813","nonce":460262507}]},"public_addr":"172.21.15.146:6809/460262507","cluster_addr":"172.21.15.146:6811/460262507","heartbeat_back_addr":"172.21.15.146:6815/460262507","heartbeat_front_addr":"172.21.15.146:6813/460262507","state":["exists","up"]},{"osd":6,"uuid":"c32e7fed-370d-4401-8ce0-6a4335b0d168","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6823","nonce":1392897493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6827","nonce":1392897493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6831","nonce":1392897493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6829","nonce":1392897493}]},"public_addr":"172.21.15.146:6823/1392897493","cluster_addr":"172.21.15.146:6827/1392897493","heartbeat_back_addr":"172.21.15.146:6831/1392897493","heartbeat_front_addr":"172.21.15.146:6829/1392897493","state":["exists","up"]},{"osd":7,"uuid":"75a79b44-fa5f-426d-a7d3-71f21f78835c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6817","nonce":3268323156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6819","nonce":3268323156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6825","nonce":3268323156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6822","nonce":3268323156}]},"public_addr":"172.21.15.146:6817/3268323156","cluster_addr":"172.21.15.146:6819/3268323156","heartbeat_back_addr":"172.21.15.146:6825/3268323156","heartbeat_front_addr":"172.21.15.146:6822/3268323156","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.310437+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.295850+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.294674+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.526855+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.300023+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.595160+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.346046+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.558741+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.6:6835/882667161":"2024-03-05T09:32:43.007130+0000","172.21.15.6:6834/882667161":"2024-03-05T09:32:43.007130+0000","172.21.15.6:6834/2697931210":"2024-03-05T09:31:36.737386+0000","172.21.15.6:6832/2697931210":"2024-03-05T09:31:36.737386+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T09:32:47.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.816+0000 7f4dab200640 1 -- 172.21.15.6:0/58099244 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f4d8806b500 msgr2=0x7f4d8806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:47.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.816+0000 7f4dab200640 1 --2- 172.21.15.6:0/58099244 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f4d8806b500 0x7f4d8806d9c0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f4d9c034000 tx=0x7f4d9c009030 comp rx=0 tx=0).stop 2024-03-04T09:32:47.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.816+0000 7f4dab200640 1 -- 172.21.15.6:0/58099244 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4dac1f4280 msgr2=0x7f4dac0b9310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:47.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.816+0000 7f4dab200640 1 --2- 172.21.15.6:0/58099244 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4dac1f4280 0x7f4dac0b9310 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7f4d9400e7f0 tx=0x7f4d9400ecb0 comp rx=0 tx=0).stop 2024-03-04T09:32:47.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.817+0000 7f4dab200640 1 -- 172.21.15.6:0/58099244 shutdown_connections 2024-03-04T09:32:47.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.817+0000 7f4dab200640 1 --2- 172.21.15.6:0/58099244 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4dac0b9850 0x7f4dac06c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:47.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.817+0000 7f4dab200640 1 --2- 172.21.15.6:0/58099244 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4dac1f4280 0x7f4dac0b9310 unknown :-1 s=CLOSED pgs=1445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:47.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.817+0000 7f4dab200640 1 --2- 172.21.15.6:0/58099244 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f4d8806b500 0x7f4d8806d9c0 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:47.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.817+0000 7f4dab200640 1 --2- 172.21.15.6:0/58099244 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4dac1523d0 0x7f4dac1f3d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:47.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.817+0000 7f4dab200640 1 -- 172.21.15.6:0/58099244 >> 172.21.15.6:0/58099244 conn(0x7f4dac0d3d30 msgr2=0x7f4dac0b5cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:32:47.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.817+0000 7f4dab200640 1 -- 172.21.15.6:0/58099244 shutdown_connections 2024-03-04T09:32:47.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:47.817+0000 7f4dab200640 1 -- 172.21.15.6:0/58099244 wait complete. 2024-03-04T09:32:47.833 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T09:25:07.935198+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-04T09:32:47.833 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-04T09:32:48.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.064+0000 7f626d600640 1 Processor -- start 2024-03-04T09:32:48.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.064+0000 7f626d600640 1 -- start start 2024-03-04T09:32:48.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.065+0000 7f626d600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f626815afa0 0x7f62680cf3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:48.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.065+0000 7f626d600640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f6268070ce0 0x7f62680ccad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:48.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.066+0000 7f626d600640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6268071c60 con 0x7f62680cd010 2024-03-04T09:32:48.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.066+0000 7f626d600640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62680cfa10 con 0x7f6268070ce0 2024-03-04T09:32:48.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.066+0000 7f626d600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6268071f40 con 0x7f626815afa0 2024-03-04T09:32:48.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.066+0000 7f6267400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f62680cd010 0x7f626806c980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:35156/0 (socket says 172.21.15.6:35156) 2024-03-04T09:32:48.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.066+0000 7f6267400640 1 -- 172.21.15.6:0/449322585 learned_addr learned my addr 172.21.15.6:0/449322585 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:32:48.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.066+0000 7f6266a00640 1 --2- 172.21.15.6:0/449322585 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f626815afa0 0x7f62680cf3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:48.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.066+0000 7f6266000640 1 --2- 172.21.15.6:0/449322585 >> v2:172.21.15.146:3300/0 conn(0x7f6268070ce0 0x7f62680ccad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:48.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.066+0000 7f6265600640 1 -- 172.21.15.6:0/449322585 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 35632661 0 0) 0x7f6268071c60 con 0x7f62680cd010 2024-03-04T09:32:48.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.067+0000 7f6265600640 1 -- 172.21.15.6:0/449322585 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f624c005730 con 0x7f62680cd010 2024-03-04T09:32:48.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.067+0000 7f6265600640 1 -- 172.21.15.6:0/449322585 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2866748347 0 0) 0x7f624c005730 con 0x7f62680cd010 2024-03-04T09:32:48.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.067+0000 7f6265600640 1 -- 172.21.15.6:0/449322585 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f626815afa0 msgr2=0x7f62680cf3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:32:48.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.067+0000 7f6265600640 1 --2- 172.21.15.6:0/449322585 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f626815afa0 0x7f62680cf3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:48.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.067+0000 7f6265600640 1 -- 172.21.15.6:0/449322585 >> v2:172.21.15.146:3300/0 conn(0x7f6268070ce0 msgr2=0x7f62680ccad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:32:48.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.067+0000 7f6265600640 1 --2- 172.21.15.6:0/449322585 >> v2:172.21.15.146:3300/0 conn(0x7f6268070ce0 0x7f62680ccad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:48.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.067+0000 7f6265600640 1 -- 172.21.15.6:0/449322585 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6268072220 con 0x7f62680cd010 2024-03-04T09:32:48.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.067+0000 7f6265600640 1 -- 172.21.15.6:0/449322585 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7f6260001670 con 0x7f62680cd010 2024-03-04T09:32:48.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.067+0000 7f6265600640 1 -- 172.21.15.6:0/449322585 >> v1:172.21.15.6:6789/0 conn(0x7f62680cd010 legacy=0x7f626806c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:48.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.067+0000 7f6265600640 1 --2- 172.21.15.6:0/449322585 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f624c005bc0 0x7f624c005fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:48.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.067+0000 7f6265600640 1 --2- 172.21.15.6:0/449322585 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f624c006a10 0x7f624c006e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:48.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.067+0000 7f6265600640 1 --2- 172.21.15.6:0/449322585 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f626815afa0 0x7f624c0073d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:48.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.067+0000 7f6265600640 1 -- 172.21.15.6:0/449322585 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62680cfa10 con 0x7f624c005bc0 2024-03-04T09:32:48.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.067+0000 7f6265600640 1 -- 172.21.15.6:0/449322585 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6268071f40 con 0x7f624c006a10 2024-03-04T09:32:48.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.067+0000 7f6265600640 1 -- 172.21.15.6:0/449322585 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f624c00dc20 con 0x7f626815afa0 2024-03-04T09:32:48.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.067+0000 7f6266a00640 1 --2- 172.21.15.6:0/449322585 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f624c005bc0 0x7f624c005fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:48.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.068+0000 7f6266000640 1 --2- 172.21.15.6:0/449322585 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f624c006a10 0x7f624c006e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:48.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.068+0000 7f6267400640 1 --2- 172.21.15.6:0/449322585 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f626815afa0 0x7f624c0073d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:48.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.068+0000 7f6266a00640 1 -- 172.21.15.6:0/449322585 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f626815afa0 msgr2=0x7f624c0073d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:32:48.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.068+0000 7f6266a00640 1 --2- 172.21.15.6:0/449322585 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f626815afa0 0x7f624c0073d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:48.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.068+0000 7f6266a00640 1 -- 172.21.15.6:0/449322585 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f624c006a10 msgr2=0x7f624c006e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:32:48.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.068+0000 7f6266a00640 1 --2- 172.21.15.6:0/449322585 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f624c006a10 0x7f624c006e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:48.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.068+0000 7f6266a00640 1 -- 172.21.15.6:0/449322585 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f624c005730 con 0x7f624c005bc0 2024-03-04T09:32:48.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.068+0000 7f6267400640 1 --2- 172.21.15.6:0/449322585 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f626815afa0 0x7f624c0073d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:32:48.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.068+0000 7f6266000640 1 --2- 172.21.15.6:0/449322585 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f624c006a10 0x7f624c006e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:32:48.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.069+0000 7f6266a00640 1 --2- 172.21.15.6:0/449322585 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f624c005bc0 0x7f624c005fe0 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7f6258009af0 tx=0x7f624c007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f35ff4a8741497a server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:48.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.069+0000 7f6265600640 1 -- 172.21.15.6:0/449322585 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6258039070 con 0x7f624c005bc0 2024-03-04T09:32:48.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.069+0000 7f6265600640 1 -- 172.21.15.6:0/449322585 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f625802c6b0 con 0x7f624c005bc0 2024-03-04T09:32:48.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.069+0000 7f6265600640 1 -- 172.21.15.6:0/449322585 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6258009690 con 0x7f624c005bc0 2024-03-04T09:32:48.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.070+0000 7f626d600640 1 -- 172.21.15.6:0/449322585 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f624c005bc0 msgr2=0x7f624c005fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:48.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.070+0000 7f626d600640 1 --2- 172.21.15.6:0/449322585 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f624c005bc0 0x7f624c005fe0 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7f6258009af0 tx=0x7f624c007a40 comp rx=0 tx=0).stop 2024-03-04T09:32:48.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.070+0000 7f6267400640 1 -- 172.21.15.6:0/449322585 reap_dead start 2024-03-04T09:32:48.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.070+0000 7f626d600640 1 -- 172.21.15.6:0/449322585 shutdown_connections 2024-03-04T09:32:48.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.070+0000 7f626d600640 1 -- 172.21.15.6:0/449322585 >> 172.21.15.6:0/449322585 conn(0x7f626806b3d0 msgr2=0x7f626806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:32:48.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.070+0000 7f626d600640 1 -- 172.21.15.6:0/449322585 shutdown_connections 2024-03-04T09:32:48.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.070+0000 7f626d600640 1 -- 172.21.15.6:0/449322585 wait complete. 2024-03-04T09:32:48.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.072+0000 7f626d600640 1 Processor -- start 2024-03-04T09:32:48.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.072+0000 7f626d600640 1 -- start start 2024-03-04T09:32:48.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.073+0000 7f626d600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f624c005bc0 0x7f62681518d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:48.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.073+0000 7f626d600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f62680cb490 0x7f62680cb910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:48.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.073+0000 7f626d600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f62680cbe50 0x7f626814cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:48.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.073+0000 7f626d600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62680b4ad0 con 0x7f624c005bc0 2024-03-04T09:32:48.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.073+0000 7f626d600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6268071140 con 0x7f62680cbe50 2024-03-04T09:32:48.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.073+0000 7f626d600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62680b4c50 con 0x7f62680cb490 2024-03-04T09:32:48.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.073+0000 7f6266a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f624c005bc0 0x7f62681518d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:48.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.073+0000 7f6266a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f624c005bc0 0x7f62681518d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:47418/0 (socket says 172.21.15.6:47418) 2024-03-04T09:32:48.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.073+0000 7f6266a00640 1 -- 172.21.15.6:0/3550882248 learned_addr learned my addr 172.21.15.6:0/3550882248 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:32:48.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.073+0000 7f6266000640 1 --2- 172.21.15.6:0/3550882248 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f62680cb490 0x7f62680cb910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:48.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.073+0000 7f6266a00640 1 -- 172.21.15.6:0/3550882248 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f62680cb490 msgr2=0x7f62680cb910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:48.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.073+0000 7f6266a00640 1 --2- 172.21.15.6:0/3550882248 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f62680cb490 0x7f62680cb910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:48.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.074+0000 7f6266a00640 1 -- 172.21.15.6:0/3550882248 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f62680cbe50 msgr2=0x7f626814cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:48.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.074+0000 7f6266a00640 1 --2- 172.21.15.6:0/3550882248 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f62680cbe50 0x7f626814cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:48.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.074+0000 7f6266a00640 1 -- 172.21.15.6:0/3550882248 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f626809da80 con 0x7f624c005bc0 2024-03-04T09:32:48.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.074+0000 7f6266a00640 1 --2- 172.21.15.6:0/3550882248 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f624c005bc0 0x7f62681518d0 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7f6258009af0 tx=0x7f6258004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:48.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.074+0000 7f625ea00640 1 -- 172.21.15.6:0/3550882248 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6258042020 con 0x7f624c005bc0 2024-03-04T09:32:48.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.074+0000 7f625ea00640 1 -- 172.21.15.6:0/3550882248 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f625803a040 con 0x7f624c005bc0 2024-03-04T09:32:48.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.074+0000 7f626d600640 1 -- 172.21.15.6:0/3550882248 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62680ca9f0 con 0x7f624c005bc0 2024-03-04T09:32:48.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.074+0000 7f625ea00640 1 -- 172.21.15.6:0/3550882248 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6258039070 con 0x7f624c005bc0 2024-03-04T09:32:48.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.074+0000 7f626d600640 1 -- 172.21.15.6:0/3550882248 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62680cac90 con 0x7f624c005bc0 2024-03-04T09:32:48.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.075+0000 7f626d600640 1 -- 172.21.15.6:0/3550882248 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f626804f330 con 0x7f624c005bc0 2024-03-04T09:32:48.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.076+0000 7f625ea00640 1 -- 172.21.15.6:0/3550882248 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6258045020 con 0x7f624c005bc0 2024-03-04T09:32:48.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.077+0000 7f625ea00640 1 --2- 172.21.15.6:0/3550882248 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f624806b4d0 0x7f624806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:48.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.078+0000 7f625ea00640 1 -- 172.21.15.6:0/3550882248 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f62580ab1b0 con 0x7f624c005bc0 2024-03-04T09:32:48.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.078+0000 7f6266000640 1 --2- 172.21.15.6:0/3550882248 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f624806b4d0 0x7f624806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:48.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.079+0000 7f6266000640 1 --2- 172.21.15.6:0/3550882248 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f624806b4d0 0x7f624806d990 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f62680b2590 tx=0x7f625400b040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:48.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.082+0000 7f625ea00640 1 -- 172.21.15.6:0/3550882248 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f625804c020 con 0x7f624c005bc0 2024-03-04T09:32:48.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.280+0000 7f626d600640 1 -- 172.21.15.6:0/3550882248 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f62680ae3b0 con 0x7f624c005bc0 2024-03-04T09:32:48.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.281+0000 7f625ea00640 1 -- 172.21.15.6:0/3550882248 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v118) v1 ==== 93+0+10 (secure 0 0 0) 0x7f625807a230 con 0x7f624c005bc0 2024-03-04T09:32:48.278 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-04T09:32:48.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.284+0000 7f626d600640 1 -- 172.21.15.6:0/3550882248 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f624806b4d0 msgr2=0x7f624806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:48.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.284+0000 7f626d600640 1 --2- 172.21.15.6:0/3550882248 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f624806b4d0 0x7f624806d990 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f62680b2590 tx=0x7f625400b040 comp rx=0 tx=0).stop 2024-03-04T09:32:48.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.284+0000 7f626d600640 1 -- 172.21.15.6:0/3550882248 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f624c005bc0 msgr2=0x7f62681518d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:48.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.284+0000 7f626d600640 1 --2- 172.21.15.6:0/3550882248 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f624c005bc0 0x7f62681518d0 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7f6258009af0 tx=0x7f6258004730 comp rx=0 tx=0).stop 2024-03-04T09:32:48.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.284+0000 7f626d600640 1 -- 172.21.15.6:0/3550882248 shutdown_connections 2024-03-04T09:32:48.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.284+0000 7f626d600640 1 --2- 172.21.15.6:0/3550882248 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f62680cbe50 0x7f626814cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:48.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.284+0000 7f626d600640 1 --2- 172.21.15.6:0/3550882248 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f624806b4d0 0x7f624806d990 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:48.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.284+0000 7f626d600640 1 --2- 172.21.15.6:0/3550882248 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f62680cb490 0x7f62680cb910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:48.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.284+0000 7f626d600640 1 --2- 172.21.15.6:0/3550882248 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f624c005bc0 0x7f62681518d0 unknown :-1 s=CLOSED pgs=1448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:48.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.284+0000 7f626d600640 1 -- 172.21.15.6:0/3550882248 >> 172.21.15.6:0/3550882248 conn(0x7f626806b3d0 msgr2=0x7f62680ce340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:32:48.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.284+0000 7f626d600640 1 -- 172.21.15.6:0/3550882248 shutdown_connections 2024-03-04T09:32:48.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.284+0000 7f626d600640 1 -- 172.21.15.6:0/3550882248 wait complete. 2024-03-04T09:32:48.299 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_migrate_unlinked_dir' 2024-03-04T09:32:48.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.526+0000 7f860f000640 1 Processor -- start 2024-03-04T09:32:48.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.527+0000 7f860f000640 1 -- start start 2024-03-04T09:32:48.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.529+0000 7f860f000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f8608147400 0x7f860806afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:48.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.529+0000 7f860f000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f860815ea80 0x7f860806b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:48.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.529+0000 7f860f000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8608161ae0 con 0x7f86081625a0 2024-03-04T09:32:48.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.529+0000 7f860f000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8608162110 con 0x7f8608147400 2024-03-04T09:32:48.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.529+0000 7f860f000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f860815f080 con 0x7f860815ea80 2024-03-04T09:32:48.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.529+0000 7f860dc00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f8608147400 0x7f860806afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:48.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.529+0000 7f860dc00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f8608147400 0x7f860806afb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:47432/0 (socket says 172.21.15.6:47432) 2024-03-04T09:32:48.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.529+0000 7f860dc00640 1 -- 172.21.15.6:0/1644697920 learned_addr learned my addr 172.21.15.6:0/1644697920 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:32:48.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.529+0000 7f860d200640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f86081625a0 0x7f860815e370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:35164/0 (socket says 172.21.15.6:35164) 2024-03-04T09:32:48.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.530+0000 7f860dc00640 1 -- 172.21.15.6:0/1644697920 >> v2:172.21.15.146:3300/0 conn(0x7f860815ea80 msgr2=0x7f860806b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:32:48.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.530+0000 7f860e600640 1 --2- 172.21.15.6:0/1644697920 >> v2:172.21.15.146:3300/0 conn(0x7f860815ea80 0x7f860806b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:48.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.530+0000 7f860dc00640 1 --2- 172.21.15.6:0/1644697920 >> v2:172.21.15.146:3300/0 conn(0x7f860815ea80 0x7f860806b4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:48.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.530+0000 7f860dc00640 1 -- 172.21.15.6:0/1644697920 >> v1:172.21.15.6:6789/0 conn(0x7f86081625a0 legacy=0x7f860815e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:48.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.530+0000 7f860dc00640 1 -- 172.21.15.6:0/1644697920 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8608161da0 con 0x7f8608147400 2024-03-04T09:32:48.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.530+0000 7f8607e00640 1 -- 172.21.15.6:0/1644697920 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2499176019 0 0) 0x7f8608161ae0 con 0x7f86081625a0 2024-03-04T09:32:48.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.530+0000 7f860e600640 1 --2- 172.21.15.6:0/1644697920 >> v2:172.21.15.146:3300/0 conn(0x7f860815ea80 0x7f860806b4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:32:48.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.531+0000 7f860dc00640 1 --2- 172.21.15.6:0/1644697920 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8608147400 0x7f860806afb0 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7f860806e160 tx=0x7f85fc02bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbc39d5a80142d29 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:48.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.531+0000 7f8607e00640 1 -- 172.21.15.6:0/1644697920 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85fc02cb10 con 0x7f8608147400 2024-03-04T09:32:48.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.531+0000 7f8607e00640 1 -- 172.21.15.6:0/1644697920 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85fc02ccb0 con 0x7f8608147400 2024-03-04T09:32:48.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.531+0000 7f8607e00640 1 -- 172.21.15.6:0/1644697920 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85fc02ce60 con 0x7f8608147400 2024-03-04T09:32:48.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.532+0000 7f860f000640 1 -- 172.21.15.6:0/1644697920 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8608147400 msgr2=0x7f860806afb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:48.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.532+0000 7f860f000640 1 --2- 172.21.15.6:0/1644697920 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8608147400 0x7f860806afb0 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7f860806e160 tx=0x7f85fc02bf70 comp rx=0 tx=0).stop 2024-03-04T09:32:48.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.532+0000 7f860f000640 1 -- 172.21.15.6:0/1644697920 shutdown_connections 2024-03-04T09:32:48.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.532+0000 7f860f000640 1 --2- 172.21.15.6:0/1644697920 >> v2:172.21.15.146:3300/0 conn(0x7f860815ea80 0x7f860806b4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:48.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.532+0000 7f860f000640 1 --2- 172.21.15.6:0/1644697920 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8608147400 0x7f860806afb0 unknown :-1 s=CLOSED pgs=1450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:48.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.532+0000 7f860f000640 1 -- 172.21.15.6:0/1644697920 >> 172.21.15.6:0/1644697920 conn(0x7f860806c0c0 msgr2=0x7f860806c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:32:48.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.532+0000 7f860f000640 1 -- 172.21.15.6:0/1644697920 shutdown_connections 2024-03-04T09:32:48.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.532+0000 7f860f000640 1 -- 172.21.15.6:0/1644697920 wait complete. 2024-03-04T09:32:48.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.534+0000 7f860f000640 1 Processor -- start 2024-03-04T09:32:48.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.534+0000 7f860f000640 1 -- start start 2024-03-04T09:32:48.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.535+0000 7f860f000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8608147400 0x7f86080b12d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:48.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.535+0000 7f860f000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f860815ea80 0x7f8608139de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:48.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.535+0000 7f860f000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f86081625a0 0x7f8608145040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:48.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.535+0000 7f860f000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f860806e530 con 0x7f8608147400 2024-03-04T09:32:48.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.535+0000 7f860f000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f860806e3b0 con 0x7f860815ea80 2024-03-04T09:32:48.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.535+0000 7f860f000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f860806e6b0 con 0x7f86081625a0 2024-03-04T09:32:48.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.535+0000 7f860dc00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8608147400 0x7f86080b12d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:48.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.535+0000 7f860dc00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8608147400 0x7f86080b12d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:47442/0 (socket says 172.21.15.6:47442) 2024-03-04T09:32:48.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.535+0000 7f860dc00640 1 -- 172.21.15.6:0/3099769035 learned_addr learned my addr 172.21.15.6:0/3099769035 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:32:48.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.536+0000 7f860dc00640 1 -- 172.21.15.6:0/3099769035 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f86081625a0 msgr2=0x7f8608145040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:48.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.536+0000 7f860dc00640 1 --2- 172.21.15.6:0/3099769035 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f86081625a0 0x7f8608145040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:48.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.536+0000 7f860dc00640 1 -- 172.21.15.6:0/3099769035 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f860815ea80 msgr2=0x7f8608139de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:48.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.536+0000 7f860dc00640 1 --2- 172.21.15.6:0/3099769035 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f860815ea80 0x7f8608139de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:48.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.536+0000 7f860dc00640 1 -- 172.21.15.6:0/3099769035 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86080ae600 con 0x7f8608147400 2024-03-04T09:32:48.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.536+0000 7f860dc00640 1 --2- 172.21.15.6:0/3099769035 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8608147400 0x7f86080b12d0 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7f85fc000f80 tx=0x7f85fc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:48.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.536+0000 7f8605600640 1 -- 172.21.15.6:0/3099769035 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85fc043020 con 0x7f8608147400 2024-03-04T09:32:48.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.536+0000 7f8605600640 1 -- 172.21.15.6:0/3099769035 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85fc03e450 con 0x7f8608147400 2024-03-04T09:32:48.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.536+0000 7f8605600640 1 -- 172.21.15.6:0/3099769035 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85fc02ce60 con 0x7f8608147400 2024-03-04T09:32:48.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.536+0000 7f860f000640 1 -- 172.21.15.6:0/3099769035 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86080ae8a0 con 0x7f8608147400 2024-03-04T09:32:48.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.537+0000 7f860f000640 1 -- 172.21.15.6:0/3099769035 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86080aee60 con 0x7f8608147400 2024-03-04T09:32:48.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.538+0000 7f860f000640 1 -- 172.21.15.6:0/3099769035 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86080b3850 con 0x7f8608147400 2024-03-04T09:32:48.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.538+0000 7f8605600640 1 -- 172.21.15.6:0/3099769035 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f85fc046020 con 0x7f8608147400 2024-03-04T09:32:48.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.539+0000 7f8605600640 1 --2- 172.21.15.6:0/3099769035 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f85e806b4d0 0x7f85e806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:48.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.539+0000 7f8605600640 1 -- 172.21.15.6:0/3099769035 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f85fc0ac8a0 con 0x7f8608147400 2024-03-04T09:32:48.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.540+0000 7f860d200640 1 --2- 172.21.15.6:0/3099769035 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f85e806b4d0 0x7f85e806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:48.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.541+0000 7f860d200640 1 --2- 172.21.15.6:0/3099769035 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f85e806b4d0 0x7f85e806d990 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f85f4007fd0 tx=0x7f85f4007960 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:48.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.544+0000 7f8605600640 1 -- 172.21.15.6:0/3099769035 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f86080b3850 con 0x7f8608147400 2024-03-04T09:32:48.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:48.816+0000 7f860f000640 1 -- 172.21.15.6:0/3099769035 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_migrate_unlinked_dir"]} v 0) v1 -- 0x7f860806e190 con 0x7f8608147400 2024-03-04T09:32:49.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:49.368+0000 7f8605600640 1 -- 172.21.15.6:0/3099769035 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_migrate_unlinked_dir"]}]=0 v416) v1 ==== 143+0+0 (secure 0 0 0) 0x7f85fc04d030 con 0x7f8608147400 2024-03-04T09:32:49.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:49.371+0000 7f860f000640 1 -- 172.21.15.6:0/3099769035 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f85e806b4d0 msgr2=0x7f85e806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:49.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:49.371+0000 7f860f000640 1 --2- 172.21.15.6:0/3099769035 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f85e806b4d0 0x7f85e806d990 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f85f4007fd0 tx=0x7f85f4007960 comp rx=0 tx=0).stop 2024-03-04T09:32:49.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:49.371+0000 7f860f000640 1 -- 172.21.15.6:0/3099769035 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8608147400 msgr2=0x7f86080b12d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:49.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:49.371+0000 7f860f000640 1 --2- 172.21.15.6:0/3099769035 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8608147400 0x7f86080b12d0 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7f85fc000f80 tx=0x7f85fc004a40 comp rx=0 tx=0).stop 2024-03-04T09:32:49.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:49.372+0000 7f860f000640 1 -- 172.21.15.6:0/3099769035 shutdown_connections 2024-03-04T09:32:49.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:49.372+0000 7f860f000640 1 --2- 172.21.15.6:0/3099769035 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f85e806b4d0 0x7f85e806d990 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:49.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:49.372+0000 7f860f000640 1 --2- 172.21.15.6:0/3099769035 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f86081625a0 0x7f8608145040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:49.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:49.372+0000 7f860f000640 1 --2- 172.21.15.6:0/3099769035 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f860815ea80 0x7f8608139de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:49.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:49.372+0000 7f860f000640 1 --2- 172.21.15.6:0/3099769035 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8608147400 0x7f86080b12d0 unknown :-1 s=CLOSED pgs=1451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:49.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:49.372+0000 7f860f000640 1 -- 172.21.15.6:0/3099769035 >> 172.21.15.6:0/3099769035 conn(0x7f860806c0c0 msgr2=0x7f860815f3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:32:49.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:49.372+0000 7f860f000640 1 -- 172.21.15.6:0/3099769035 shutdown_connections 2024-03-04T09:32:49.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:49.372+0000 7f860f000640 1 -- 172.21.15.6:0/3099769035 wait complete. 2024-03-04T09:32:49.388 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-04T09:32:49.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:49.621+0000 7ff94b000640 1 Processor -- start 2024-03-04T09:32:49.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:49.621+0000 7ff94b000640 1 -- start start 2024-03-04T09:32:49.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:49.622+0000 7ff94b000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff94415dee0 0x7ff9440d18a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:49.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:49.622+0000 7ff94b000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ff9440d0d40 0x7ff9440b3510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:49.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:49.622+0000 7ff94b000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9440ceca0 con 0x7ff9440d4860 2024-03-04T09:32:49.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:49.622+0000 7ff94b000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9440cf5d0 con 0x7ff9440d0d40 2024-03-04T09:32:49.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:49.622+0000 7ff94b000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9440cef80 con 0x7ff94415dee0 2024-03-04T09:32:49.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:49.623+0000 7ff949200640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7ff9440d4860 0x7ff9440d0630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:35602/0 (socket says 172.21.15.6:35602) 2024-03-04T09:32:49.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:49.623+0000 7ff949200640 1 -- 172.21.15.6:0/2916488906 learned_addr learned my addr 172.21.15.6:0/2916488906 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:32:49.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:49.623+0000 7ff949c00640 1 --2- 172.21.15.6:0/2916488906 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff94415dee0 0x7ff9440d18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:49.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:49.623+0000 7ff94a600640 1 --2- 172.21.15.6:0/2916488906 >> v2:172.21.15.146:3300/0 conn(0x7ff9440d0d40 0x7ff9440b3510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:49.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:49.623+0000 7ff943e00640 1 -- 172.21.15.6:0/2916488906 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4003431049 0 0) 0x7ff9440ceca0 con 0x7ff9440d4860 2024-03-04T09:32:49.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:49.623+0000 7ff943e00640 1 -- 172.21.15.6:0/2916488906 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff92c003520 con 0x7ff9440d4860 2024-03-04T09:32:49.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:49.624+0000 7ff943e00640 1 -- 172.21.15.6:0/2916488906 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1383522644 0 0) 0x7ff92c003520 con 0x7ff9440d4860 2024-03-04T09:32:49.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:49.624+0000 7ff943e00640 1 -- 172.21.15.6:0/2916488906 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff94415dee0 msgr2=0x7ff9440d18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:32:49.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:49.624+0000 7ff943e00640 1 --2- 172.21.15.6:0/2916488906 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff94415dee0 0x7ff9440d18a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:49.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:49.624+0000 7ff943e00640 1 -- 172.21.15.6:0/2916488906 >> v2:172.21.15.146:3300/0 conn(0x7ff9440d0d40 msgr2=0x7ff9440b3510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:32:49.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:49.624+0000 7ff943e00640 1 --2- 172.21.15.6:0/2916488906 >> v2:172.21.15.146:3300/0 conn(0x7ff9440d0d40 0x7ff9440b3510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:49.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:49.624+0000 7ff943e00640 1 -- 172.21.15.6:0/2916488906 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9440cf260 con 0x7ff9440d4860 2024-03-04T09:32:49.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:49.624+0000 7ff943e00640 1 -- 172.21.15.6:0/2916488906 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7ff934001670 con 0x7ff9440d4860 2024-03-04T09:32:49.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:49.624+0000 7ff943e00640 1 -- 172.21.15.6:0/2916488906 >> v1:172.21.15.6:6789/0 conn(0x7ff9440d4860 legacy=0x7ff9440d0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:49.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:49.624+0000 7ff943e00640 1 --2- 172.21.15.6:0/2916488906 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff92c0039b0 0x7ff92c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:49.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:49.624+0000 7ff943e00640 1 --2- 172.21.15.6:0/2916488906 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff94415dee0 0x7ff92c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:49.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:49.624+0000 7ff943e00640 1 --2- 172.21.15.6:0/2916488906 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff92c004d40 0x7ff92c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:49.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:49.624+0000 7ff943e00640 1 -- 172.21.15.6:0/2916488906 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9440cf5d0 con 0x7ff92c004d40 2024-03-04T09:32:49.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:49.624+0000 7ff943e00640 1 -- 172.21.15.6:0/2916488906 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9440cef80 con 0x7ff92c0039b0 2024-03-04T09:32:49.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:49.624+0000 7ff943e00640 1 -- 172.21.15.6:0/2916488906 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff92c00b9c0 con 0x7ff94415dee0 2024-03-04T09:32:49.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:49.624+0000 7ff94a600640 1 --2- 172.21.15.6:0/2916488906 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff92c004d40 0x7ff92c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:49.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:49.625+0000 7ff949c00640 1 --2- 172.21.15.6:0/2916488906 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff92c0039b0 0x7ff92c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:49.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:49.625+0000 7ff949200640 1 --2- 172.21.15.6:0/2916488906 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff94415dee0 0x7ff92c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:49.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:49.625+0000 7ff94a600640 1 -- 172.21.15.6:0/2916488906 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff94415dee0 msgr2=0x7ff92c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:32:49.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:49.625+0000 7ff94a600640 1 --2- 172.21.15.6:0/2916488906 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff94415dee0 0x7ff92c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:49.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:49.625+0000 7ff94a600640 1 -- 172.21.15.6:0/2916488906 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff92c0039b0 msgr2=0x7ff92c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:32:49.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:49.625+0000 7ff94a600640 1 --2- 172.21.15.6:0/2916488906 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff92c0039b0 0x7ff92c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:49.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:49.625+0000 7ff94a600640 1 -- 172.21.15.6:0/2916488906 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff92c003520 con 0x7ff92c004d40 2024-03-04T09:32:49.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:49.625+0000 7ff949c00640 1 --2- 172.21.15.6:0/2916488906 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff92c0039b0 0x7ff92c003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:32:49.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:49.625+0000 7ff949200640 1 --2- 172.21.15.6:0/2916488906 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff94415dee0 0x7ff92c004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:32:49.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:49.626+0000 7ff94a600640 1 --2- 172.21.15.6:0/2916488906 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff92c004d40 0x7ff92c0051c0 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7ff94406b2f0 tx=0x7ff92c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fefdbd8e0d5c8e63 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:49.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:49.626+0000 7ff943e00640 1 -- 172.21.15.6:0/2916488906 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff93c039070 con 0x7ff92c004d40 2024-03-04T09:32:49.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:49.626+0000 7ff943e00640 1 -- 172.21.15.6:0/2916488906 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff93c02cbd0 con 0x7ff92c004d40 2024-03-04T09:32:49.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:49.626+0000 7ff943e00640 1 -- 172.21.15.6:0/2916488906 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff93c009c10 con 0x7ff92c004d40 2024-03-04T09:32:49.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:49.627+0000 7ff94b000640 1 -- 172.21.15.6:0/2916488906 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff92c004d40 msgr2=0x7ff92c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:49.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:49.627+0000 7ff94b000640 1 --2- 172.21.15.6:0/2916488906 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff92c004d40 0x7ff92c0051c0 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7ff94406b2f0 tx=0x7ff92c005830 comp rx=0 tx=0).stop 2024-03-04T09:32:49.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:49.627+0000 7ff94a600640 1 -- 172.21.15.6:0/2916488906 reap_dead start 2024-03-04T09:32:49.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:49.627+0000 7ff94b000640 1 -- 172.21.15.6:0/2916488906 shutdown_connections 2024-03-04T09:32:49.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:49.627+0000 7ff94b000640 1 -- 172.21.15.6:0/2916488906 >> 172.21.15.6:0/2916488906 conn(0x7ff9440d3cd0 msgr2=0x7ff9440d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:32:49.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:49.627+0000 7ff94b000640 1 -- 172.21.15.6:0/2916488906 shutdown_connections 2024-03-04T09:32:49.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:49.627+0000 7ff94b000640 1 -- 172.21.15.6:0/2916488906 wait complete. 2024-03-04T09:32:49.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:49.629+0000 7ff94b000640 1 Processor -- start 2024-03-04T09:32:49.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:49.629+0000 7ff94b000640 1 -- start start 2024-03-04T09:32:49.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:49.630+0000 7ff94b000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff92c004d40 0x7ff94409d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:49.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:49.630+0000 7ff94b000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff944153980 0x7ff9440ca4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:49.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:49.630+0000 7ff94b000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff9440cd5a0 0x7ff9440a0a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:49.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:49.630+0000 7ff94b000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9440cfa60 con 0x7ff9440cd5a0 2024-03-04T09:32:49.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:49.630+0000 7ff94b000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff94415e2c0 con 0x7ff944153980 2024-03-04T09:32:49.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:49.630+0000 7ff94b000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9440cfbe0 con 0x7ff92c004d40 2024-03-04T09:32:49.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:49.630+0000 7ff94a600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff9440cd5a0 0x7ff9440a0a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:49.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:49.630+0000 7ff94a600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff9440cd5a0 0x7ff9440a0a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43296/0 (socket says 172.21.15.6:43296) 2024-03-04T09:32:49.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:49.630+0000 7ff94a600640 1 -- 172.21.15.6:0/3165280704 learned_addr learned my addr 172.21.15.6:0/3165280704 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:32:49.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:49.631+0000 7ff949c00640 1 --2- 172.21.15.6:0/3165280704 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff92c004d40 0x7ff94409d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:49.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:49.631+0000 7ff94a600640 1 -- 172.21.15.6:0/3165280704 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff92c004d40 msgr2=0x7ff94409d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:49.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:49.631+0000 7ff94a600640 1 --2- 172.21.15.6:0/3165280704 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff92c004d40 0x7ff94409d9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:49.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:49.631+0000 7ff94a600640 1 -- 172.21.15.6:0/3165280704 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff944153980 msgr2=0x7ff9440ca4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:49.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:49.631+0000 7ff949200640 1 --2- 172.21.15.6:0/3165280704 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff944153980 0x7ff9440ca4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:49.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:49.631+0000 7ff94a600640 1 --2- 172.21.15.6:0/3165280704 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff944153980 0x7ff9440ca4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:49.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:49.631+0000 7ff94a600640 1 -- 172.21.15.6:0/3165280704 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff94409e200 con 0x7ff9440cd5a0 2024-03-04T09:32:49.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:49.631+0000 7ff949c00640 1 --2- 172.21.15.6:0/3165280704 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff92c004d40 0x7ff94409d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:32:49.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:49.631+0000 7ff949200640 1 --2- 172.21.15.6:0/3165280704 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff944153980 0x7ff9440ca4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:32:49.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:49.631+0000 7ff94a600640 1 --2- 172.21.15.6:0/3165280704 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff9440cd5a0 0x7ff9440a0a60 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 crypto rx=0x7ff93c00e230 tx=0x7ff93c009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:49.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:49.631+0000 7ff941600640 1 -- 172.21.15.6:0/3165280704 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff93c039030 con 0x7ff9440cd5a0 2024-03-04T09:32:49.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:49.631+0000 7ff941600640 1 -- 172.21.15.6:0/3165280704 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff93c043070 con 0x7ff9440cd5a0 2024-03-04T09:32:49.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:49.631+0000 7ff941600640 1 -- 172.21.15.6:0/3165280704 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff93c009c10 con 0x7ff9440cd5a0 2024-03-04T09:32:49.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:49.631+0000 7ff94b000640 1 -- 172.21.15.6:0/3165280704 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff94409e4e0 con 0x7ff9440cd5a0 2024-03-04T09:32:49.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:49.632+0000 7ff94b000640 1 -- 172.21.15.6:0/3165280704 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff94409eaa0 con 0x7ff9440cd5a0 2024-03-04T09:32:49.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:49.633+0000 7ff94b000640 1 -- 172.21.15.6:0/3165280704 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff94404f330 con 0x7ff9440cd5a0 2024-03-04T09:32:49.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:49.634+0000 7ff941600640 1 -- 172.21.15.6:0/3165280704 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff93c005d10 con 0x7ff9440cd5a0 2024-03-04T09:32:49.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:49.636+0000 7ff941600640 1 --2- 172.21.15.6:0/3165280704 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff92806b4d0 0x7ff92806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:49.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:49.636+0000 7ff941600640 1 -- 172.21.15.6:0/3165280704 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5029+0+0 (secure 0 0 0) 0x7ff93c0ac110 con 0x7ff9440cd5a0 2024-03-04T09:32:49.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:49.636+0000 7ff949c00640 1 --2- 172.21.15.6:0/3165280704 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff92806b4d0 0x7ff92806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:49.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:49.638+0000 7ff949c00640 1 --2- 172.21.15.6:0/3165280704 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff92806b4d0 0x7ff92806d990 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7ff938009aa0 tx=0x7ff938008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:49.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:49.640+0000 7ff941600640 1 -- 172.21.15.6:0/3165280704 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff93c07b220 con 0x7ff9440cd5a0 2024-03-04T09:32:49.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:49.847+0000 7ff94b000640 1 -- 172.21.15.6:0/3165280704 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7ff9440d0b00 con 0x7ff9440cd5a0 2024-03-04T09:32:49.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:49.853+0000 7ff941600640 1 -- 172.21.15.6:0/3165280704 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v32)=0 v32) v1 ==== 122+0+0 (secure 0 0 0) 0x7ff93c07ab20 con 0x7ff9440cd5a0 2024-03-04T09:32:49.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:49.856+0000 7ff94b000640 1 -- 172.21.15.6:0/3165280704 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff92806b4d0 msgr2=0x7ff92806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:49.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:49.856+0000 7ff94b000640 1 --2- 172.21.15.6:0/3165280704 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff92806b4d0 0x7ff92806d990 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7ff938009aa0 tx=0x7ff938008040 comp rx=0 tx=0).stop 2024-03-04T09:32:49.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:49.856+0000 7ff94b000640 1 -- 172.21.15.6:0/3165280704 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff9440cd5a0 msgr2=0x7ff9440a0a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:49.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:49.856+0000 7ff94b000640 1 --2- 172.21.15.6:0/3165280704 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff9440cd5a0 0x7ff9440a0a60 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 crypto rx=0x7ff93c00e230 tx=0x7ff93c009750 comp rx=0 tx=0).stop 2024-03-04T09:32:49.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:49.856+0000 7ff94b000640 1 -- 172.21.15.6:0/3165280704 shutdown_connections 2024-03-04T09:32:49.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:49.856+0000 7ff94b000640 1 --2- 172.21.15.6:0/3165280704 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff9440cd5a0 0x7ff9440a0a60 unknown :-1 s=CLOSED pgs=1454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:49.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:49.856+0000 7ff94b000640 1 --2- 172.21.15.6:0/3165280704 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff944153980 0x7ff9440ca4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:49.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:49.856+0000 7ff94b000640 1 --2- 172.21.15.6:0/3165280704 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff92806b4d0 0x7ff92806d990 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:49.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:49.856+0000 7ff94b000640 1 --2- 172.21.15.6:0/3165280704 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff92c004d40 0x7ff94409d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:49.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:49.856+0000 7ff94b000640 1 -- 172.21.15.6:0/3165280704 >> 172.21.15.6:0/3165280704 conn(0x7ff9440d3cd0 msgr2=0x7ff9440cdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:32:49.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:49.857+0000 7ff94b000640 1 -- 172.21.15.6:0/3165280704 shutdown_connections 2024-03-04T09:32:49.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:49.857+0000 7ff94b000640 1 -- 172.21.15.6:0/3165280704 wait complete. 2024-03-04T09:32:49.874 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-04T09:32:49.874 DEBUG:teuthology.orchestra.run.smithi006:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-04T09:32:49.989 DEBUG:teuthology.orchestra.run.smithi146:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-04T09:32:50.108 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-04T09:32:50.109 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-04T09:32:50.109 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T09:32:50.169 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054202,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249800k,nr_inodes=812450,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-04T09:32:50.169 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-04T09:32:50.169 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T09:32:50.227 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053429,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249180k,nr_inodes=812295,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-04T09:32:50.227 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-04T09:32:50.227 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T09:32:50.286 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054202,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249800k,nr_inodes=812450,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-04T09:32:50.286 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-04T09:32:50.286 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T09:32:50.342 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053429,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249180k,nr_inodes=812295,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-04T09:32:50.342 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-04T09:32:50.343 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:32:50.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:50.572+0000 7f7a81a00640 1 Processor -- start 2024-03-04T09:32:50.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:50.572+0000 7f7a81a00640 1 -- start start 2024-03-04T09:32:50.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:50.573+0000 7f7a81a00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f7a7c070ee0 0x7f7a7c162c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:50.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:50.573+0000 7f7a81a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a7c0ce090 con 0x7f7a7c072da0 2024-03-04T09:32:50.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:50.573+0000 7f7a81a00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a7c0ce370 con 0x7f7a7c0712c0 2024-03-04T09:32:50.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:50.573+0000 7f7a81a00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a7c0ceca0 con 0x7f7a7c070ee0 2024-03-04T09:32:50.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:50.574+0000 7f7a7a000640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f7a7c072da0 0x7f7a7c06e0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:35606/0 (socket says 172.21.15.6:35606) 2024-03-04T09:32:50.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:50.574+0000 7f7a7a000640 1 -- 172.21.15.6:0/30184240 learned_addr learned my addr 172.21.15.6:0/30184240 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:32:50.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:50.574+0000 7f7a79600640 1 -- 172.21.15.6:0/30184240 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3258260439 0 0) 0x7f7a7c0ce090 con 0x7f7a7c072da0 2024-03-04T09:32:50.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:50.574+0000 7f7a79600640 1 -- 172.21.15.6:0/30184240 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a60003520 con 0x7f7a7c072da0 2024-03-04T09:32:50.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:50.574+0000 7f7a7aa00640 1 --2- 172.21.15.6:0/30184240 >> v2:172.21.15.146:3300/0 conn(0x7f7a7c070ee0 0x7f7a7c162c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:50.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:50.575+0000 7f7a79600640 1 -- 172.21.15.6:0/30184240 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3362490548 0 0) 0x7f7a60003520 con 0x7f7a7c072da0 2024-03-04T09:32:50.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:50.575+0000 7f7a79600640 1 -- 172.21.15.6:0/30184240 >> v2:172.21.15.146:3300/0 conn(0x7f7a7c070ee0 msgr2=0x7f7a7c162c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:32:50.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:50.575+0000 7f7a79600640 1 --2- 172.21.15.6:0/30184240 >> v2:172.21.15.146:3300/0 conn(0x7f7a7c070ee0 0x7f7a7c162c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:50.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:50.575+0000 7f7a79600640 1 -- 172.21.15.6:0/30184240 >> v1:172.21.15.146:6789/0 conn(0x7f7a7c0712c0 legacy=0x7f7a7c06cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:50.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:50.575+0000 7f7a79600640 1 -- 172.21.15.6:0/30184240 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a7c0ce650 con 0x7f7a7c072da0 2024-03-04T09:32:50.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:50.575+0000 7f7a79600640 1 -- 172.21.15.6:0/30184240 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7f7a64001670 con 0x7f7a7c072da0 2024-03-04T09:32:50.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:50.575+0000 7f7a79600640 1 -- 172.21.15.6:0/30184240 >> v1:172.21.15.6:6789/0 conn(0x7f7a7c072da0 legacy=0x7f7a7c06e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:50.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:50.575+0000 7f7a79600640 1 --2- 172.21.15.6:0/30184240 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7a60003a20 0x7f7a60003e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:50.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:50.575+0000 7f7a79600640 1 --2- 172.21.15.6:0/30184240 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7a60004800 0x7f7a60004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:50.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:50.575+0000 7f7a79600640 1 --2- 172.21.15.6:0/30184240 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7a600052e0 0x7f7a6000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:50.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:50.575+0000 7f7a79600640 1 -- 172.21.15.6:0/30184240 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a7c0ceca0 con 0x7f7a60004800 2024-03-04T09:32:50.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:50.575+0000 7f7a79600640 1 -- 172.21.15.6:0/30184240 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a6000c1b0 con 0x7f7a60003a20 2024-03-04T09:32:50.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:50.575+0000 7f7a79600640 1 -- 172.21.15.6:0/30184240 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a6000c3b0 con 0x7f7a600052e0 2024-03-04T09:32:50.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:50.575+0000 7f7a7a000640 1 --2- 172.21.15.6:0/30184240 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7a60004800 0x7f7a60004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:50.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:50.576+0000 7f7a7aa00640 1 --2- 172.21.15.6:0/30184240 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7a60003a20 0x7f7a60003e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:50.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:50.576+0000 7f7a7b400640 1 --2- 172.21.15.6:0/30184240 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7a600052e0 0x7f7a6000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:50.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:50.576+0000 7f7a7a000640 1 -- 172.21.15.6:0/30184240 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7a600052e0 msgr2=0x7f7a6000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:32:50.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:50.576+0000 7f7a7a000640 1 --2- 172.21.15.6:0/30184240 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7a600052e0 0x7f7a6000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:50.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:50.576+0000 7f7a7a000640 1 -- 172.21.15.6:0/30184240 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7a60003a20 msgr2=0x7f7a60003e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:32:50.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:50.576+0000 7f7a7a000640 1 --2- 172.21.15.6:0/30184240 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7a60003a20 0x7f7a60003e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:50.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:50.576+0000 7f7a7a000640 1 -- 172.21.15.6:0/30184240 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a60003520 con 0x7f7a60004800 2024-03-04T09:32:50.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:50.576+0000 7f7a7b400640 1 --2- 172.21.15.6:0/30184240 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7a600052e0 0x7f7a6000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:32:50.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:50.576+0000 7f7a7b400640 1 -- 172.21.15.6:0/30184240 reap_dead start 2024-03-04T09:32:50.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:50.577+0000 7f7a7a000640 1 --2- 172.21.15.6:0/30184240 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7a60004800 0x7f7a60004c80 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7f7a6400c210 tx=0x7f7a60005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b45f82ab03921a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:50.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:50.577+0000 7f7a79600640 1 -- 172.21.15.6:0/30184240 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a64007cf0 con 0x7f7a60004800 2024-03-04T09:32:50.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:50.577+0000 7f7a79600640 1 -- 172.21.15.6:0/30184240 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a64030070 con 0x7f7a60004800 2024-03-04T09:32:50.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:50.577+0000 7f7a79600640 1 -- 172.21.15.6:0/30184240 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a64005d70 con 0x7f7a60004800 2024-03-04T09:32:50.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:50.578+0000 7f7a81a00640 1 -- 172.21.15.6:0/30184240 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7a60004800 msgr2=0x7f7a60004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:50.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:50.578+0000 7f7a81a00640 1 --2- 172.21.15.6:0/30184240 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7a60004800 0x7f7a60004c80 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7f7a6400c210 tx=0x7f7a60005830 comp rx=0 tx=0).stop 2024-03-04T09:32:50.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:50.578+0000 7f7a81a00640 1 -- 172.21.15.6:0/30184240 shutdown_connections 2024-03-04T09:32:50.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:50.578+0000 7f7a81a00640 1 --2- 172.21.15.6:0/30184240 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7a60004800 0x7f7a60004c80 unknown :-1 s=CLOSED pgs=1456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:50.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:50.578+0000 7f7a81a00640 1 -- 172.21.15.6:0/30184240 >> 172.21.15.6:0/30184240 conn(0x7f7a7c06afb0 msgr2=0x7f7a7c06b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:32:50.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:50.578+0000 7f7a81a00640 1 -- 172.21.15.6:0/30184240 shutdown_connections 2024-03-04T09:32:50.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:50.578+0000 7f7a81a00640 1 -- 172.21.15.6:0/30184240 wait complete. 2024-03-04T09:32:50.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:50.580+0000 7f7a81a00640 1 Processor -- start 2024-03-04T09:32:50.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:50.580+0000 7f7a81a00640 1 -- start start 2024-03-04T09:32:50.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:50.581+0000 7f7a81a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7a60004800 0x7f7a7c15d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:50.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:50.581+0000 7f7a81a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7a7c14fd30 0x7f7a7c09d280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:50.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:50.581+0000 7f7a81a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7a7c15b150 0x7f7a7c15b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:50.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:50.581+0000 7f7a81a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a7c0cdda0 con 0x7f7a60004800 2024-03-04T09:32:50.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:50.581+0000 7f7a81a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a7c0cdc20 con 0x7f7a7c14fd30 2024-03-04T09:32:50.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:50.581+0000 7f7a81a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a7c162c40 con 0x7f7a7c15b150 2024-03-04T09:32:50.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:50.581+0000 7f7a7aa00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7a60004800 0x7f7a7c15d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:50.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:50.581+0000 7f7a7aa00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7a60004800 0x7f7a7c15d6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43314/0 (socket says 172.21.15.6:43314) 2024-03-04T09:32:50.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:50.581+0000 7f7a7aa00640 1 -- 172.21.15.6:0/3987405296 learned_addr learned my addr 172.21.15.6:0/3987405296 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:32:50.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:50.582+0000 7f7a7b400640 1 --2- 172.21.15.6:0/3987405296 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7a7c15b150 0x7f7a7c15b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:50.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:50.582+0000 7f7a7aa00640 1 -- 172.21.15.6:0/3987405296 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7a7c15b150 msgr2=0x7f7a7c15b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:50.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:50.582+0000 7f7a7aa00640 1 --2- 172.21.15.6:0/3987405296 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7a7c15b150 0x7f7a7c15b5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:50.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:50.582+0000 7f7a7aa00640 1 -- 172.21.15.6:0/3987405296 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7a7c14fd30 msgr2=0x7f7a7c09d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:50.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:50.582+0000 7f7a7aa00640 1 --2- 172.21.15.6:0/3987405296 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7a7c14fd30 0x7f7a7c09d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:50.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:50.582+0000 7f7a7aa00640 1 -- 172.21.15.6:0/3987405296 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a7c09c1e0 con 0x7f7a60004800 2024-03-04T09:32:50.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:50.582+0000 7f7a7aa00640 1 --2- 172.21.15.6:0/3987405296 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7a60004800 0x7f7a7c15d6b0 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 crypto rx=0x7f7a6c00f600 tx=0x7f7a6c00fac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:50.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:50.582+0000 7f7a72a00640 1 -- 172.21.15.6:0/3987405296 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a6c019020 con 0x7f7a60004800 2024-03-04T09:32:50.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:50.583+0000 7f7a72a00640 1 -- 172.21.15.6:0/3987405296 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a6c014870 con 0x7f7a60004800 2024-03-04T09:32:50.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:50.583+0000 7f7a72a00640 1 -- 172.21.15.6:0/3987405296 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a6c014c20 con 0x7f7a60004800 2024-03-04T09:32:50.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:50.583+0000 7f7a81a00640 1 -- 172.21.15.6:0/3987405296 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a7c09c4c0 con 0x7f7a60004800 2024-03-04T09:32:50.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:50.583+0000 7f7a81a00640 1 -- 172.21.15.6:0/3987405296 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a7c09ca80 con 0x7f7a60004800 2024-03-04T09:32:50.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:50.584+0000 7f7a81a00640 1 -- 172.21.15.6:0/3987405296 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a7c069510 con 0x7f7a60004800 2024-03-04T09:32:50.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:50.585+0000 7f7a72a00640 1 -- 172.21.15.6:0/3987405296 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7a6c01c020 con 0x7f7a60004800 2024-03-04T09:32:50.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:50.586+0000 7f7a72a00640 1 --2- 172.21.15.6:0/3987405296 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f7a5806b550 0x7f7a5806da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:50.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:50.586+0000 7f7a72a00640 1 -- 172.21.15.6:0/3987405296 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f7a6c08a7d0 con 0x7f7a60004800 2024-03-04T09:32:50.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:50.587+0000 7f7a7a000640 1 --2- 172.21.15.6:0/3987405296 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f7a5806b550 0x7f7a5806da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:50.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:50.588+0000 7f7a7a000640 1 --2- 172.21.15.6:0/3987405296 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f7a5806b550 0x7f7a5806da10 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f7a64034000 tx=0x7f7a6402e000 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:50.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:50.591+0000 7f7a72a00640 1 -- 172.21.15.6:0/3987405296 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7a6c023020 con 0x7f7a60004800 2024-03-04T09:32:50.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:50.828+0000 7f7a81a00640 1 -- 172.21.15.6:0/3987405296 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7a7c0b52d0 con 0x7f7a60004800 2024-03-04T09:32:50.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:50.829+0000 7f7a72a00640 1 -- 172.21.15.6:0/3987405296 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) v1 ==== 94+0+3696 (secure 0 0 0) 0x7f7a6c0598d0 con 0x7f7a60004800 2024-03-04T09:32:50.827 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:32:50.827 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":86,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6535,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/3883122012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":3883122012},{"type":"v1","addr":"172.21.15.146:6835","nonce":3883122012}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":6559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/4060411493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":4060411493},{"type":"v1","addr":"172.21.15.146:6837","nonce":4060411493}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":82},{"gid":7812,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/679471398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":679471398},{"type":"v1","addr":"172.21.15.6:6833","nonce":679471398}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":8142,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/2437307329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":2437307329},{"type":"v1","addr":"172.21.15.6:6835","nonce":2437307329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":86}],"filesystems":[]} 2024-03-04T09:32:50.827 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 86 2024-03-04T09:32:50.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:50.832+0000 7f7a81a00640 1 -- 172.21.15.6:0/3987405296 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f7a5806b550 msgr2=0x7f7a5806da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:50.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:50.832+0000 7f7a81a00640 1 --2- 172.21.15.6:0/3987405296 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f7a5806b550 0x7f7a5806da10 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f7a64034000 tx=0x7f7a6402e000 comp rx=0 tx=0).stop 2024-03-04T09:32:50.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:50.832+0000 7f7a81a00640 1 -- 172.21.15.6:0/3987405296 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7a60004800 msgr2=0x7f7a7c15d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:50.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:50.832+0000 7f7a81a00640 1 --2- 172.21.15.6:0/3987405296 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7a60004800 0x7f7a7c15d6b0 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 crypto rx=0x7f7a6c00f600 tx=0x7f7a6c00fac0 comp rx=0 tx=0).stop 2024-03-04T09:32:50.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:50.832+0000 7f7a81a00640 1 -- 172.21.15.6:0/3987405296 shutdown_connections 2024-03-04T09:32:50.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:50.832+0000 7f7a81a00640 1 --2- 172.21.15.6:0/3987405296 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f7a5806b550 0x7f7a5806da10 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:50.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:50.832+0000 7f7a81a00640 1 --2- 172.21.15.6:0/3987405296 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7a7c15b150 0x7f7a7c15b5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:50.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:50.832+0000 7f7a81a00640 1 --2- 172.21.15.6:0/3987405296 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7a7c14fd30 0x7f7a7c09d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:50.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:50.832+0000 7f7a81a00640 1 --2- 172.21.15.6:0/3987405296 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7a60004800 0x7f7a7c15d6b0 unknown :-1 s=CLOSED pgs=1457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:50.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:50.832+0000 7f7a81a00640 1 -- 172.21.15.6:0/3987405296 >> 172.21.15.6:0/3987405296 conn(0x7f7a7c06afb0 msgr2=0x7f7a7c164eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:32:50.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:50.832+0000 7f7a81a00640 1 -- 172.21.15.6:0/3987405296 shutdown_connections 2024-03-04T09:32:50.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:50.833+0000 7f7a81a00640 1 -- 172.21.15.6:0/3987405296 wait complete. 2024-03-04T09:32:50.848 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-04T09:32:50.848 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-04T09:32:50.848 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-04T09:32:50.848 INFO:teuthology.orchestra.run:waiting for 300 2024-03-04T09:32:50.848 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-04T09:32:50.848 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-04T09:32:50.849 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-04T09:32:50.849 INFO:teuthology.orchestra.run:waiting for 300 2024-03-04T09:32:50.849 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-04T09:32:50.849 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-04T09:32:50.849 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-04T09:32:50.849 INFO:teuthology.orchestra.run:waiting for 300 2024-03-04T09:32:50.849 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-04T09:32:50.850 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-04T09:32:50.850 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-04T09:32:50.850 INFO:teuthology.orchestra.run:waiting for 300 2024-03-04T09:32:50.866 INFO:tasks.ceph.mds.a:Stopped 2024-03-04T09:32:50.866 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-04T09:32:50.867 INFO:tasks.ceph.mds.c:Stopped 2024-03-04T09:32:50.867 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-04T09:32:50.870 INFO:tasks.ceph.mds.d:Stopped 2024-03-04T09:32:50.870 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-04T09:32:50.871 INFO:tasks.ceph.mds.b:Stopped 2024-03-04T09:32:50.871 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-04T09:32:50.914 INFO:tasks.ceph.mds.a:Started 2024-03-04T09:32:50.914 DEBUG:teuthology.parallel:result is None 2024-03-04T09:32:50.918 INFO:tasks.ceph.mds.d:Started 2024-03-04T09:32:50.918 DEBUG:teuthology.parallel:result is None 2024-03-04T09:32:50.919 INFO:tasks.ceph.mds.c:Started 2024-03-04T09:32:50.919 DEBUG:teuthology.parallel:result is None 2024-03-04T09:32:50.924 INFO:tasks.ceph.mds.b:Started 2024-03-04T09:32:50.924 DEBUG:teuthology.parallel:result is None 2024-03-04T09:32:50.924 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-04T09:32:51.073 INFO:tasks.ceph.mds.a.smithi006.stdout:starting mds.a at 2024-03-04T09:32:51.074 INFO:tasks.ceph.mds.c.smithi006.stdout:starting mds.c at 2024-03-04T09:32:51.088 INFO:tasks.ceph.mds.d.smithi146.stdout:starting mds.d at 2024-03-04T09:32:51.089 INFO:tasks.ceph.mds.b.smithi146.stdout:starting mds.b at 2024-03-04T09:32:51.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:51.116+0000 7f7397a00640 1 Processor -- start 2024-03-04T09:32:51.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:51.117+0000 7f7397a00640 1 -- start start 2024-03-04T09:32:51.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:51.118+0000 7f7397a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f73900af1a0 0x7f739015c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:51.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:51.118+0000 7f7397a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f739015cc40 con 0x7f73900cdc20 2024-03-04T09:32:51.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:51.118+0000 7f7397a00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f739015cf20 con 0x7f73900af580 2024-03-04T09:32:51.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:51.118+0000 7f7397a00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f739015d200 con 0x7f73900af1a0 2024-03-04T09:32:51.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:51.118+0000 7f7395000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f73900af1a0 0x7f739015c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:51.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:51.118+0000 7f7395000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f73900af1a0 0x7f739015c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43338/0 (socket says 172.21.15.6:43338) 2024-03-04T09:32:51.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:51.118+0000 7f7395000640 1 -- 172.21.15.6:0/2863934130 learned_addr learned my addr 172.21.15.6:0/2863934130 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:32:51.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:51.119+0000 7f7395000640 1 -- 172.21.15.6:0/2863934130 >> v1:172.21.15.146:6789/0 conn(0x7f73900af580 legacy=0x7f73900b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:51.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:51.119+0000 7f7395000640 1 -- 172.21.15.6:0/2863934130 >> v1:172.21.15.6:6789/0 conn(0x7f73900cdc20 legacy=0x7f73900b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:51.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:51.119+0000 7f7395000640 1 -- 172.21.15.6:0/2863934130 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f739015d4e0 con 0x7f73900af1a0 2024-03-04T09:32:51.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:51.120+0000 7f7395000640 1 --2- 172.21.15.6:0/2863934130 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f73900af1a0 0x7f739015c600 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7f7384009870 tx=0x7f738402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aff56042ed32de0c server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:51.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:51.120+0000 7f738f400640 1 -- 172.21.15.6:0/2863934130 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7384034070 con 0x7f73900af1a0 2024-03-04T09:32:51.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:51.120+0000 7f738f400640 1 -- 172.21.15.6:0/2863934130 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7384030860 con 0x7f73900af1a0 2024-03-04T09:32:51.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:51.120+0000 7f738f400640 1 -- 172.21.15.6:0/2863934130 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7384030c30 con 0x7f73900af1a0 2024-03-04T09:32:51.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:51.121+0000 7f7397a00640 1 -- 172.21.15.6:0/2863934130 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f73900af1a0 msgr2=0x7f739015c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:51.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:51.121+0000 7f7397a00640 1 --2- 172.21.15.6:0/2863934130 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f73900af1a0 0x7f739015c600 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7f7384009870 tx=0x7f738402b9a0 comp rx=0 tx=0).stop 2024-03-04T09:32:51.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:51.121+0000 7f7397a00640 1 -- 172.21.15.6:0/2863934130 shutdown_connections 2024-03-04T09:32:51.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:51.121+0000 7f7397a00640 1 --2- 172.21.15.6:0/2863934130 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f73900af1a0 0x7f739015c600 unknown :-1 s=CLOSED pgs=1464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:51.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:51.121+0000 7f7397a00640 1 -- 172.21.15.6:0/2863934130 >> 172.21.15.6:0/2863934130 conn(0x7f739006afb0 msgr2=0x7f739006b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:32:51.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:51.121+0000 7f7397a00640 1 -- 172.21.15.6:0/2863934130 shutdown_connections 2024-03-04T09:32:51.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:51.121+0000 7f7397a00640 1 -- 172.21.15.6:0/2863934130 wait complete. 2024-03-04T09:32:51.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:51.123+0000 7f7397a00640 1 Processor -- start 2024-03-04T09:32:51.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:51.123+0000 7f7397a00640 1 -- start start 2024-03-04T09:32:51.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:51.124+0000 7f7397a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f73900af1a0 0x7f739009d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:51.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:51.124+0000 7f7397a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f73900af580 0x7f73900ce300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:51.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:51.124+0000 7f7397a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f73900cdc20 0x7f739009bea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:51.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:51.124+0000 7f7397a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7390155e70 con 0x7f73900af580 2024-03-04T09:32:51.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:51.124+0000 7f7397a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73900b3770 con 0x7f73900af1a0 2024-03-04T09:32:51.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:51.124+0000 7f7397a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7390155ff0 con 0x7f73900cdc20 2024-03-04T09:32:51.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:51.124+0000 7f738fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f73900af580 0x7f73900ce300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:51.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:51.124+0000 7f738fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f73900af580 0x7f73900ce300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43350/0 (socket says 172.21.15.6:43350) 2024-03-04T09:32:51.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:51.124+0000 7f738fe00640 1 -- 172.21.15.6:0/1572695988 learned_addr learned my addr 172.21.15.6:0/1572695988 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:32:51.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:51.124+0000 7f7395000640 1 --2- 172.21.15.6:0/1572695988 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f73900af1a0 0x7f739009d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:51.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:51.124+0000 7f7395a00640 1 --2- 172.21.15.6:0/1572695988 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f73900cdc20 0x7f739009bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:51.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:51.125+0000 7f738fe00640 1 -- 172.21.15.6:0/1572695988 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f73900cdc20 msgr2=0x7f739009bea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:51.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:51.125+0000 7f738fe00640 1 --2- 172.21.15.6:0/1572695988 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f73900cdc20 0x7f739009bea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:51.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:51.125+0000 7f738fe00640 1 -- 172.21.15.6:0/1572695988 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f73900af1a0 msgr2=0x7f739009d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:51.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:51.125+0000 7f738fe00640 1 --2- 172.21.15.6:0/1572695988 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f73900af1a0 0x7f739009d8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:51.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:51.125+0000 7f738fe00640 1 -- 172.21.15.6:0/1572695988 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f739009c810 con 0x7f73900af580 2024-03-04T09:32:51.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:51.125+0000 7f738fe00640 1 --2- 172.21.15.6:0/1572695988 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f73900af580 0x7f73900ce300 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7f737c00c440 tx=0x7f737c001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:51.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:51.125+0000 7f738cc00640 1 -- 172.21.15.6:0/1572695988 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f737c0052d0 con 0x7f73900af580 2024-03-04T09:32:51.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:51.125+0000 7f738cc00640 1 -- 172.21.15.6:0/1572695988 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f737c005470 con 0x7f73900af580 2024-03-04T09:32:51.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:51.125+0000 7f738cc00640 1 -- 172.21.15.6:0/1572695988 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f737c007260 con 0x7f73900af580 2024-03-04T09:32:51.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:51.125+0000 7f7397a00640 1 -- 172.21.15.6:0/1572695988 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f739009caf0 con 0x7f73900af580 2024-03-04T09:32:51.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:51.125+0000 7f7397a00640 1 -- 172.21.15.6:0/1572695988 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f739009d0b0 con 0x7f73900af580 2024-03-04T09:32:51.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:51.126+0000 7f7397a00640 1 -- 172.21.15.6:0/1572695988 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f739004f3a0 con 0x7f73900af580 2024-03-04T09:32:51.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:51.128+0000 7f738cc00640 1 -- 172.21.15.6:0/1572695988 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f737c007400 con 0x7f73900af580 2024-03-04T09:32:51.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:51.129+0000 7f738cc00640 1 --2- 172.21.15.6:0/1572695988 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f737006b4d0 0x7f737006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:51.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:51.130+0000 7f738cc00640 1 -- 172.21.15.6:0/1572695988 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f737c089860 con 0x7f73900af580 2024-03-04T09:32:51.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:51.130+0000 7f7395000640 1 --2- 172.21.15.6:0/1572695988 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f737006b4d0 0x7f737006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:51.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:51.131+0000 7f7395000640 1 --2- 172.21.15.6:0/1572695988 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f737006b4d0 0x7f737006d990 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f7384036000 tx=0x7f7384008030 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:51.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:51.133+0000 7f738cc00640 1 -- 172.21.15.6:0/1572695988 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f737c0583a0 con 0x7f73900af580 2024-03-04T09:32:51.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:51.339+0000 7f7397a00640 1 -- 172.21.15.6:0/1572695988 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f73900b2e80 con 0x7f73900af580 2024-03-04T09:32:52.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:52.403+0000 7f738cc00640 1 -- 172.21.15.6:0/1572695988 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v123) v1 ==== 97+0+0 (secure 0 0 0) 0x7f737c05c440 con 0x7f73900af580 2024-03-04T09:32:52.401 INFO:teuthology.orchestra.run.smithi006.stderr: removed all blocklist entries 2024-03-04T09:32:52.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:52.407+0000 7f7397a00640 1 -- 172.21.15.6:0/1572695988 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f737006b4d0 msgr2=0x7f737006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:52.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:52.407+0000 7f7397a00640 1 --2- 172.21.15.6:0/1572695988 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f737006b4d0 0x7f737006d990 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f7384036000 tx=0x7f7384008030 comp rx=0 tx=0).stop 2024-03-04T09:32:52.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:52.407+0000 7f7397a00640 1 -- 172.21.15.6:0/1572695988 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f73900af580 msgr2=0x7f73900ce300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:52.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:52.407+0000 7f7397a00640 1 --2- 172.21.15.6:0/1572695988 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f73900af580 0x7f73900ce300 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7f737c00c440 tx=0x7f737c001e70 comp rx=0 tx=0).stop 2024-03-04T09:32:52.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:52.407+0000 7f7397a00640 1 -- 172.21.15.6:0/1572695988 shutdown_connections 2024-03-04T09:32:52.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:52.407+0000 7f7397a00640 1 --2- 172.21.15.6:0/1572695988 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f737006b4d0 0x7f737006d990 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:52.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:52.407+0000 7f7397a00640 1 --2- 172.21.15.6:0/1572695988 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f73900cdc20 0x7f739009bea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:52.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:52.407+0000 7f7397a00640 1 --2- 172.21.15.6:0/1572695988 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f73900af580 0x7f73900ce300 unknown :-1 s=CLOSED pgs=1465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:52.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:52.407+0000 7f7397a00640 1 --2- 172.21.15.6:0/1572695988 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f73900af1a0 0x7f739009d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:52.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:52.407+0000 7f7397a00640 1 -- 172.21.15.6:0/1572695988 >> 172.21.15.6:0/1572695988 conn(0x7f739006afb0 msgr2=0x7f7390071620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:32:52.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:52.407+0000 7f7397a00640 1 -- 172.21.15.6:0/1572695988 shutdown_connections 2024-03-04T09:32:52.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:52.407+0000 7f7397a00640 1 -- 172.21.15.6:0/1572695988 wait complete. 2024-03-04T09:32:52.425 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-04T09:32:52.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:52.635+0000 7fbefe400640 1 Processor -- start 2024-03-04T09:32:52.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:52.635+0000 7fbefe400640 1 -- start start 2024-03-04T09:32:52.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:52.637+0000 7fbefe400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fbef81552b0 0x7fbef815b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:52.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:52.637+0000 7fbefe400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbef815e810 0x7fbef80cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:52.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:52.637+0000 7fbefe400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fbef80cce90 0x7fbef806c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:52.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:52.637+0000 7fbefe400640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbef815ed10 con 0x7fbef80cce90 2024-03-04T09:32:52.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:52.637+0000 7fbefe400640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbef815eff0 con 0x7fbef81552b0 2024-03-04T09:32:52.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:52.637+0000 7fbefe400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbef815f2d0 con 0x7fbef815e810 2024-03-04T09:32:52.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:52.637+0000 7fbefda00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fbef80cce90 0x7fbef806c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:52.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:52.637+0000 7fbefda00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fbef80cce90 0x7fbef806c7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43364/0 (socket says 172.21.15.6:43364) 2024-03-04T09:32:52.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:52.637+0000 7fbefda00640 1 -- 172.21.15.6:0/1627113996 learned_addr learned my addr 172.21.15.6:0/1627113996 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:32:52.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:52.638+0000 7fbefd000640 1 --2- 172.21.15.6:0/1627113996 >> v2:172.21.15.146:3300/0 conn(0x7fbef81552b0 0x7fbef815b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:52.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:52.638+0000 7fbefda00640 1 -- 172.21.15.6:0/1627113996 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbef815e810 msgr2=0x7fbef80cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:32:52.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:52.638+0000 7fbef7e00640 1 --2- 172.21.15.6:0/1627113996 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbef815e810 0x7fbef80cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:52.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:52.638+0000 7fbefda00640 1 --2- 172.21.15.6:0/1627113996 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbef815e810 0x7fbef80cc950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:52.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:52.638+0000 7fbefda00640 1 -- 172.21.15.6:0/1627113996 >> v2:172.21.15.146:3300/0 conn(0x7fbef81552b0 msgr2=0x7fbef815b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:32:52.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:52.638+0000 7fbefda00640 1 --2- 172.21.15.6:0/1627113996 >> v2:172.21.15.146:3300/0 conn(0x7fbef81552b0 0x7fbef815b1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:52.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:52.638+0000 7fbefda00640 1 -- 172.21.15.6:0/1627113996 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbef815f5b0 con 0x7fbef80cce90 2024-03-04T09:32:52.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:52.638+0000 7fbefd000640 1 --2- 172.21.15.6:0/1627113996 >> v2:172.21.15.146:3300/0 conn(0x7fbef81552b0 0x7fbef815b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:32:52.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:52.638+0000 7fbef7e00640 1 --2- 172.21.15.6:0/1627113996 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbef815e810 0x7fbef80cc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:32:52.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:52.639+0000 7fbefda00640 1 --2- 172.21.15.6:0/1627113996 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbef80cce90 0x7fbef806c7c0 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7fbef80cd540 tx=0x7fbef002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c6ab6a1124f370d server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:52.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:52.639+0000 7fbef7400640 1 -- 172.21.15.6:0/1627113996 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbef002c9e0 con 0x7fbef80cce90 2024-03-04T09:32:52.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:52.640+0000 7fbef7400640 1 -- 172.21.15.6:0/1627113996 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbef002cb80 con 0x7fbef80cce90 2024-03-04T09:32:52.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:52.640+0000 7fbef7400640 1 -- 172.21.15.6:0/1627113996 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbef00343f0 con 0x7fbef80cce90 2024-03-04T09:32:52.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:52.640+0000 7fbefe400640 1 -- 172.21.15.6:0/1627113996 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbef80cce90 msgr2=0x7fbef806c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:52.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:52.640+0000 7fbefe400640 1 --2- 172.21.15.6:0/1627113996 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbef80cce90 0x7fbef806c7c0 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7fbef80cd540 tx=0x7fbef002be40 comp rx=0 tx=0).stop 2024-03-04T09:32:52.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:52.640+0000 7fbefe400640 1 -- 172.21.15.6:0/1627113996 shutdown_connections 2024-03-04T09:32:52.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:52.640+0000 7fbefe400640 1 --2- 172.21.15.6:0/1627113996 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbef80cce90 0x7fbef806c7c0 unknown :-1 s=CLOSED pgs=1466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:52.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:52.640+0000 7fbefe400640 1 --2- 172.21.15.6:0/1627113996 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbef815e810 0x7fbef80cc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:52.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:52.640+0000 7fbefe400640 1 --2- 172.21.15.6:0/1627113996 >> v2:172.21.15.146:3300/0 conn(0x7fbef81552b0 0x7fbef815b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:52.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:52.640+0000 7fbefe400640 1 -- 172.21.15.6:0/1627113996 >> 172.21.15.6:0/1627113996 conn(0x7fbef806b3d0 msgr2=0x7fbef806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:32:52.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:52.641+0000 7fbefe400640 1 -- 172.21.15.6:0/1627113996 shutdown_connections 2024-03-04T09:32:52.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:52.641+0000 7fbefe400640 1 -- 172.21.15.6:0/1627113996 wait complete. 2024-03-04T09:32:52.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:52.642+0000 7fbefe400640 1 Processor -- start 2024-03-04T09:32:52.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:52.643+0000 7fbefe400640 1 -- start start 2024-03-04T09:32:52.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:52.643+0000 7fbefe400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbef80cce90 0x7fbef814d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:52.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:52.644+0000 7fbefe400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbef81552b0 0x7fbef814f300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:52.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:52.644+0000 7fbefe400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbef815e810 0x7fbef80aeb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:52.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:52.644+0000 7fbefe400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbef806dd30 con 0x7fbef80cce90 2024-03-04T09:32:52.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:52.644+0000 7fbefe400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbef80b3760 con 0x7fbef81552b0 2024-03-04T09:32:52.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:52.644+0000 7fbefe400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbef806deb0 con 0x7fbef815e810 2024-03-04T09:32:52.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:52.644+0000 7fbefd000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbef80cce90 0x7fbef814d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:52.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:52.644+0000 7fbefd000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbef80cce90 0x7fbef814d840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43366/0 (socket says 172.21.15.6:43366) 2024-03-04T09:32:52.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:52.644+0000 7fbefd000640 1 -- 172.21.15.6:0/1946459316 learned_addr learned my addr 172.21.15.6:0/1946459316 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:32:52.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:52.644+0000 7fbefda00640 1 --2- 172.21.15.6:0/1946459316 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbef815e810 0x7fbef80aeb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:52.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:52.644+0000 7fbef7e00640 1 --2- 172.21.15.6:0/1946459316 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbef81552b0 0x7fbef814f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:52.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:52.645+0000 7fbefd000640 1 -- 172.21.15.6:0/1946459316 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbef815e810 msgr2=0x7fbef80aeb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:52.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:52.645+0000 7fbefd000640 1 --2- 172.21.15.6:0/1946459316 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbef815e810 0x7fbef80aeb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:52.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:52.645+0000 7fbefd000640 1 -- 172.21.15.6:0/1946459316 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbef81552b0 msgr2=0x7fbef814f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:52.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:52.645+0000 7fbefd000640 1 --2- 172.21.15.6:0/1946459316 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbef81552b0 0x7fbef814f300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:52.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:52.645+0000 7fbefd000640 1 -- 172.21.15.6:0/1946459316 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbef814e260 con 0x7fbef80cce90 2024-03-04T09:32:52.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:52.645+0000 7fbefda00640 1 --2- 172.21.15.6:0/1946459316 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbef815e810 0x7fbef80aeb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T09:32:52.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:52.645+0000 7fbefd000640 1 --2- 172.21.15.6:0/1946459316 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbef80cce90 0x7fbef814d840 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7fbee000c7f0 tx=0x7fbee000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:52.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:52.645+0000 7fbef4c00640 1 -- 172.21.15.6:0/1946459316 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbee0015070 con 0x7fbef80cce90 2024-03-04T09:32:52.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:52.645+0000 7fbef4c00640 1 -- 172.21.15.6:0/1946459316 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbee0007820 con 0x7fbef80cce90 2024-03-04T09:32:52.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:52.645+0000 7fbefe400640 1 -- 172.21.15.6:0/1946459316 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbef814e540 con 0x7fbef80cce90 2024-03-04T09:32:52.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:52.645+0000 7fbef4c00640 1 -- 172.21.15.6:0/1946459316 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbee0007c10 con 0x7fbef80cce90 2024-03-04T09:32:52.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:52.646+0000 7fbefe400640 1 -- 172.21.15.6:0/1946459316 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbef814eb00 con 0x7fbef80cce90 2024-03-04T09:32:52.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:52.647+0000 7fbefe400640 1 -- 172.21.15.6:0/1946459316 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbef804f330 con 0x7fbef80cce90 2024-03-04T09:32:52.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:52.648+0000 7fbef4c00640 1 -- 172.21.15.6:0/1946459316 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbee0018070 con 0x7fbef80cce90 2024-03-04T09:32:52.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:52.648+0000 7fbef4c00640 1 --2- 172.21.15.6:0/1946459316 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fbed406b4d0 0x7fbed406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:52.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:52.649+0000 7fbef4c00640 1 -- 172.21.15.6:0/1946459316 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fbee008a0e0 con 0x7fbef80cce90 2024-03-04T09:32:52.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:52.649+0000 7fbef7e00640 1 --2- 172.21.15.6:0/1946459316 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fbed406b4d0 0x7fbed406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:52.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:52.650+0000 7fbef7e00640 1 --2- 172.21.15.6:0/1946459316 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fbed406b4d0 0x7fbed406d990 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fbef806e860 tx=0x7fbee8008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:52.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:52.654+0000 7fbef4c00640 1 -- 172.21.15.6:0/1946459316 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbee0059190 con 0x7fbef80cce90 2024-03-04T09:32:52.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:52.926+0000 7fbefe400640 1 -- 172.21.15.6:0/1946459316 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fbef80cf4a0 con 0x7fbef80cce90 2024-03-04T09:32:52.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:52.927+0000 7fbef4c00640 1 -- 172.21.15.6:0/1946459316 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v16) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fbee0058a90 con 0x7fbef80cce90 2024-03-04T09:32:52.924 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:32:52.924 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-04T09:32:52.925 INFO:teuthology.orchestra.run.smithi006.stdout: "auth_dump": [ 2024-03-04T09:32:52.925 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:32:52.925 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "mds.a", 2024-03-04T09:32:52.925 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQA8k+VlHCzFIBAAnKYlmCMDw45LqnSjxnZY5Q==", 2024-03-04T09:32:52.925 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:32:52.925 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-04T09:32:52.925 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow *", 2024-03-04T09:32:52.925 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow *", 2024-03-04T09:32:52.925 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-04T09:32:52.925 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:32:52.925 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:32:52.925 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:32:52.926 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "mds.b", 2024-03-04T09:32:52.926 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQA9k+VlNY7fCBAAX3z8X72/6LkQD6mwGYT69g==", 2024-03-04T09:32:52.926 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:32:52.926 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-04T09:32:52.926 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow *", 2024-03-04T09:32:52.926 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow *", 2024-03-04T09:32:52.926 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-04T09:32:52.926 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:32:52.926 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:32:52.926 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:32:52.926 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "mds.c", 2024-03-04T09:32:52.926 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQA8k+Vl0pjOMhAA7WcTw8awv4EAsrpP4yncNA==", 2024-03-04T09:32:52.927 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:32:52.927 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-04T09:32:52.927 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow *", 2024-03-04T09:32:52.927 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow *", 2024-03-04T09:32:52.927 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-04T09:32:52.927 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:32:52.927 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:32:52.927 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:32:52.927 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "mds.d", 2024-03-04T09:32:52.927 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQA9k+VlbladGRAABNgUcaqTVYuJfCqy5c/G0g==", 2024-03-04T09:32:52.927 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:32:52.927 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-04T09:32:52.927 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow *", 2024-03-04T09:32:52.928 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow *", 2024-03-04T09:32:52.928 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-04T09:32:52.928 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:32:52.928 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:32:52.928 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:32:52.928 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.0", 2024-03-04T09:32:52.928 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQBCk+VlyjMMExAAqb2FJ9n5PWehfx/6PB19ig==", 2024-03-04T09:32:52.928 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:32:52.928 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-04T09:32:52.928 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-04T09:32:52.928 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-04T09:32:52.929 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:32:52.929 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:32:52.929 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:32:52.929 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.1", 2024-03-04T09:32:52.929 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQBFk+VlXD3cLBAAGclSivcYWDslqN8MLlj7eQ==", 2024-03-04T09:32:52.929 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:32:52.929 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-04T09:32:52.929 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-04T09:32:52.929 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-04T09:32:52.929 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:32:52.929 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:32:52.929 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:32:52.929 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.2", 2024-03-04T09:32:52.929 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQBJk+Vl7d9nCxAAnb3W4EI3Yf3XY8ht3VnKEQ==", 2024-03-04T09:32:52.930 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:32:52.930 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-04T09:32:52.930 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-04T09:32:52.930 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-04T09:32:52.930 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:32:52.930 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:32:52.930 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:32:52.930 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.3", 2024-03-04T09:32:52.930 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQBMk+Vlv+Y1NBAAiRYSUqov2i2sj3iInfpHvA==", 2024-03-04T09:32:52.930 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:32:52.930 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-04T09:32:52.930 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-04T09:32:52.930 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-04T09:32:52.930 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:32:52.930 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:32:52.930 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:32:52.931 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.4", 2024-03-04T09:32:52.931 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQBUk+Vl1OlAARAAxMV4GUGS+RJ6XlrRhrVKcg==", 2024-03-04T09:32:52.931 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:32:52.931 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-04T09:32:52.931 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-04T09:32:52.931 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-04T09:32:52.931 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:32:52.931 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:32:52.931 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:32:52.931 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.5", 2024-03-04T09:32:52.931 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQBXk+Vl1lKLGhAAHlsiRr+1+OSRWhkB+weIJg==", 2024-03-04T09:32:52.931 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:32:52.932 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-04T09:32:52.932 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-04T09:32:52.932 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-04T09:32:52.932 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:32:52.932 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:32:52.932 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:32:52.932 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.6", 2024-03-04T09:32:52.932 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQBak+VlwB+GJBAA3f9KUz+zZlQuUR/ywQfkDA==", 2024-03-04T09:32:52.932 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:32:52.932 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-04T09:32:52.932 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-04T09:32:52.932 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-04T09:32:52.932 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:32:52.932 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:32:52.933 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:32:52.933 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.7", 2024-03-04T09:32:52.933 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQBdk+Vle4o8LhAAko+iH5VqqCsfWkNfKZjGVA==", 2024-03-04T09:32:52.933 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:32:52.933 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-04T09:32:52.933 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-04T09:32:52.933 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-04T09:32:52.933 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:32:52.933 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:32:52.933 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:32:52.933 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "client.0", 2024-03-04T09:32:52.933 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQA9k+Vl4L+rJxAACaDbQ4/Bybp5hAZAnJzlVw==", 2024-03-04T09:32:52.933 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:32:52.933 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-04T09:32:52.933 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow r", 2024-03-04T09:32:52.933 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-04T09:32:52.934 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:32:52.934 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:32:52.934 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:32:52.934 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "client.1", 2024-03-04T09:32:52.934 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQA+k+VlGjkOAhAAJNzB8+tQl3mqCnTgMn9EVw==", 2024-03-04T09:32:52.934 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:32:52.934 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-04T09:32:52.934 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow r", 2024-03-04T09:32:52.934 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow rw", 2024-03-04T09:32:52.934 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow rwx" 2024-03-04T09:32:52.934 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:32:52.934 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:32:52.934 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:32:52.934 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "client.2", 2024-03-04T09:32:52.934 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQA9k+VlUc8DMxAA8pGUtIh+BO6UKrrYnCqVgQ==", 2024-03-04T09:32:52.934 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:32:52.934 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-04T09:32:52.935 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow r", 2024-03-04T09:32:52.935 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow rw", 2024-03-04T09:32:52.935 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow rwx" 2024-03-04T09:32:52.935 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:32:52.935 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:32:52.935 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:32:52.935 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "client.3", 2024-03-04T09:32:52.935 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQA+k+VlzMFKDBAAGr713I2hcmGMuHMIjNknYA==", 2024-03-04T09:32:52.935 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:32:52.935 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-04T09:32:52.935 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow r", 2024-03-04T09:32:52.935 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow rw", 2024-03-04T09:32:52.935 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow rwx" 2024-03-04T09:32:52.935 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:32:52.936 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:32:52.936 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:32:52.936 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "client.admin", 2024-03-04T09:32:52.936 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQA7k+Vllb7NFhAAmnFaNFKBcEri+iJAPnG5Gg==", 2024-03-04T09:32:52.936 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:32:52.936 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow *", 2024-03-04T09:32:52.936 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow *", 2024-03-04T09:32:52.936 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow *", 2024-03-04T09:32:52.936 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-04T09:32:52.936 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:32:52.936 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:32:52.936 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:32:52.936 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "mgr.x", 2024-03-04T09:32:52.936 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQA8k+VlPFatFBAA/o3Ps/uJrcg1738HOGy/FA==", 2024-03-04T09:32:52.936 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:32:52.937 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow *", 2024-03-04T09:32:52.937 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile mgr", 2024-03-04T09:32:52.937 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-04T09:32:52.937 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:32:52.937 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-04T09:32:52.937 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:32:52.937 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "mgr.y", 2024-03-04T09:32:52.937 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQA8k+VlEA2ACBAAXofDj3gmqItVTJ1pYLqZyQ==", 2024-03-04T09:32:52.937 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-04T09:32:52.937 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow *", 2024-03-04T09:32:52.937 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile mgr", 2024-03-04T09:32:52.937 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-04T09:32:52.938 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:32:52.938 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:32:52.938 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-04T09:32:52.938 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-04T09:32:52.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:52.930+0000 7fbefe400640 1 -- 172.21.15.6:0/1946459316 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fbed406b4d0 msgr2=0x7fbed406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:52.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:52.930+0000 7fbefe400640 1 --2- 172.21.15.6:0/1946459316 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fbed406b4d0 0x7fbed406d990 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fbef806e860 tx=0x7fbee8008040 comp rx=0 tx=0).stop 2024-03-04T09:32:52.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:52.930+0000 7fbefe400640 1 -- 172.21.15.6:0/1946459316 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbef80cce90 msgr2=0x7fbef814d840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:52.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:52.930+0000 7fbefe400640 1 --2- 172.21.15.6:0/1946459316 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbef80cce90 0x7fbef814d840 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7fbee000c7f0 tx=0x7fbee000ccb0 comp rx=0 tx=0).stop 2024-03-04T09:32:52.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:52.930+0000 7fbefe400640 1 -- 172.21.15.6:0/1946459316 shutdown_connections 2024-03-04T09:32:52.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:52.930+0000 7fbefe400640 1 --2- 172.21.15.6:0/1946459316 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fbed406b4d0 0x7fbed406d990 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:52.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:52.930+0000 7fbefe400640 1 --2- 172.21.15.6:0/1946459316 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbef815e810 0x7fbef80aeb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:52.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:52.930+0000 7fbefe400640 1 --2- 172.21.15.6:0/1946459316 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbef81552b0 0x7fbef814f300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:52.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:52.930+0000 7fbefe400640 1 --2- 172.21.15.6:0/1946459316 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbef80cce90 0x7fbef814d840 unknown :-1 s=CLOSED pgs=1467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:52.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:52.930+0000 7fbefe400640 1 -- 172.21.15.6:0/1946459316 >> 172.21.15.6:0/1946459316 conn(0x7fbef806b3d0 msgr2=0x7fbef80711d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:32:52.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:52.931+0000 7fbefe400640 1 -- 172.21.15.6:0/1946459316 shutdown_connections 2024-03-04T09:32:52.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:52.931+0000 7fbefe400640 1 -- 172.21.15.6:0/1946459316 wait complete. 2024-03-04T09:32:52.948 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T09:32:53.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.171+0000 7ff46e000640 1 Processor -- start 2024-03-04T09:32:53.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.171+0000 7ff46e000640 1 -- start start 2024-03-04T09:32:53.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.172+0000 7ff46e000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ff4680cd060 0x7ff46815b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:53.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.172+0000 7ff46e000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff46815e810 0x7ff4680b3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:53.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.172+0000 7ff46e000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff46815ed10 con 0x7ff4681552b0 2024-03-04T09:32:53.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.173+0000 7ff46e000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff46815eff0 con 0x7ff46815e810 2024-03-04T09:32:53.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.173+0000 7ff46e000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff46815f2d0 con 0x7ff4680cd060 2024-03-04T09:32:53.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.173+0000 7ff46d600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff46815e810 0x7ff4680b3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:53.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.173+0000 7ff46cc00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7ff4681552b0 0x7ff4680cc950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:35648/0 (socket says 172.21.15.6:35648) 2024-03-04T09:32:53.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.173+0000 7ff46cc00640 1 -- 172.21.15.6:0/3342769367 learned_addr learned my addr 172.21.15.6:0/3342769367 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:32:53.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.173+0000 7ff467e00640 1 --2- 172.21.15.6:0/3342769367 >> v2:172.21.15.146:3300/0 conn(0x7ff4680cd060 0x7ff46815b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:53.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.174+0000 7ff46d600640 1 -- 172.21.15.6:0/3342769367 >> v2:172.21.15.146:3300/0 conn(0x7ff4680cd060 msgr2=0x7ff46815b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:32:53.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.174+0000 7ff46d600640 1 --2- 172.21.15.6:0/3342769367 >> v2:172.21.15.146:3300/0 conn(0x7ff4680cd060 0x7ff46815b1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:53.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.174+0000 7ff46d600640 1 -- 172.21.15.6:0/3342769367 >> v1:172.21.15.6:6789/0 conn(0x7ff4681552b0 legacy=0x7ff4680cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:53.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.174+0000 7ff46d600640 1 -- 172.21.15.6:0/3342769367 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff46815f5b0 con 0x7ff46815e810 2024-03-04T09:32:53.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.175+0000 7ff46d600640 1 --2- 172.21.15.6:0/3342769367 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff46815e810 0x7ff4680b3cb0 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7ff4680cf1d0 tx=0x7ff46002be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d97cdc01c2b0918 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:53.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.176+0000 7ff467400640 1 -- 172.21.15.6:0/3342769367 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff460039070 con 0x7ff46815e810 2024-03-04T09:32:53.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.176+0000 7ff467400640 1 -- 172.21.15.6:0/3342769367 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff46002cbe0 con 0x7ff46815e810 2024-03-04T09:32:53.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.176+0000 7ff467400640 1 -- 172.21.15.6:0/3342769367 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff460034440 con 0x7ff46815e810 2024-03-04T09:32:53.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.176+0000 7ff46e000640 1 -- 172.21.15.6:0/3342769367 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff46815e810 msgr2=0x7ff4680b3cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:53.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.176+0000 7ff46e000640 1 --2- 172.21.15.6:0/3342769367 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff46815e810 0x7ff4680b3cb0 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7ff4680cf1d0 tx=0x7ff46002be30 comp rx=0 tx=0).stop 2024-03-04T09:32:53.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.176+0000 7ff46e000640 1 -- 172.21.15.6:0/3342769367 shutdown_connections 2024-03-04T09:32:53.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.176+0000 7ff46e000640 1 --2- 172.21.15.6:0/3342769367 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff46815e810 0x7ff4680b3cb0 unknown :-1 s=CLOSED pgs=1469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:53.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.176+0000 7ff46e000640 1 --2- 172.21.15.6:0/3342769367 >> v2:172.21.15.146:3300/0 conn(0x7ff4680cd060 0x7ff46815b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:53.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.177+0000 7ff46e000640 1 -- 172.21.15.6:0/3342769367 >> 172.21.15.6:0/3342769367 conn(0x7ff46806b3d0 msgr2=0x7ff46806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:32:53.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.177+0000 7ff46e000640 1 -- 172.21.15.6:0/3342769367 shutdown_connections 2024-03-04T09:32:53.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.177+0000 7ff46e000640 1 -- 172.21.15.6:0/3342769367 wait complete. 2024-03-04T09:32:53.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.178+0000 7ff46e000640 1 Processor -- start 2024-03-04T09:32:53.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.179+0000 7ff46e000640 1 -- start start 2024-03-04T09:32:53.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.180+0000 7ff46e000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff4680cd060 0x7ff4681503c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:53.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.180+0000 7ff46e000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff4681552b0 0x7ff46814dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:53.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.180+0000 7ff46e000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff46815e810 0x7ff46815d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:53.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.180+0000 7ff46e000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff46806dd30 con 0x7ff4681552b0 2024-03-04T09:32:53.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.180+0000 7ff46e000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4680b3760 con 0x7ff46815e810 2024-03-04T09:32:53.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.180+0000 7ff46e000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff46806deb0 con 0x7ff4680cd060 2024-03-04T09:32:53.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.180+0000 7ff467e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff4681552b0 0x7ff46814dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:53.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.180+0000 7ff467e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff4681552b0 0x7ff46814dc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43390/0 (socket says 172.21.15.6:43390) 2024-03-04T09:32:53.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.180+0000 7ff467e00640 1 -- 172.21.15.6:0/138537671 learned_addr learned my addr 172.21.15.6:0/138537671 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:32:53.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.180+0000 7ff46cc00640 1 --2- 172.21.15.6:0/138537671 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff4680cd060 0x7ff4681503c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:53.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.181+0000 7ff46d600640 1 --2- 172.21.15.6:0/138537671 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff46815e810 0x7ff46815d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:53.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.181+0000 7ff467e00640 1 -- 172.21.15.6:0/138537671 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff4680cd060 msgr2=0x7ff4681503c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:53.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.181+0000 7ff467e00640 1 --2- 172.21.15.6:0/138537671 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff4680cd060 0x7ff4681503c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:53.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.181+0000 7ff467e00640 1 -- 172.21.15.6:0/138537671 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff46815e810 msgr2=0x7ff46815d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:53.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.181+0000 7ff467e00640 1 --2- 172.21.15.6:0/138537671 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff46815e810 0x7ff46815d8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:53.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.181+0000 7ff467e00640 1 -- 172.21.15.6:0/138537671 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff46814e4f0 con 0x7ff4681552b0 2024-03-04T09:32:53.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.181+0000 7ff467e00640 1 --2- 172.21.15.6:0/138537671 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff4681552b0 0x7ff46814dc30 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 crypto rx=0x7ff45400d5f0 tx=0x7ff45400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:53.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.181+0000 7ff464c00640 1 -- 172.21.15.6:0/138537671 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4540125f0 con 0x7ff4681552b0 2024-03-04T09:32:53.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.181+0000 7ff464c00640 1 -- 172.21.15.6:0/138537671 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff454019070 con 0x7ff4681552b0 2024-03-04T09:32:53.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.181+0000 7ff464c00640 1 -- 172.21.15.6:0/138537671 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4540129d0 con 0x7ff4681552b0 2024-03-04T09:32:53.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.182+0000 7ff46e000640 1 -- 172.21.15.6:0/138537671 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff46814e7d0 con 0x7ff4681552b0 2024-03-04T09:32:53.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.182+0000 7ff46e000640 1 -- 172.21.15.6:0/138537671 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff46814ed90 con 0x7ff4681552b0 2024-03-04T09:32:53.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.183+0000 7ff46e000640 1 -- 172.21.15.6:0/138537671 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff46804f330 con 0x7ff4681552b0 2024-03-04T09:32:53.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.184+0000 7ff464c00640 1 -- 172.21.15.6:0/138537671 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff454012b70 con 0x7ff4681552b0 2024-03-04T09:32:53.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.185+0000 7ff464c00640 1 --2- 172.21.15.6:0/138537671 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff44806b500 0x7ff44806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:53.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.185+0000 7ff464c00640 1 -- 172.21.15.6:0/138537671 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 4857+0+0 (secure 0 0 0) 0x7ff45408a310 con 0x7ff4681552b0 2024-03-04T09:32:53.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.186+0000 7ff46cc00640 1 --2- 172.21.15.6:0/138537671 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff44806b500 0x7ff44806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:53.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.187+0000 7ff46cc00640 1 --2- 172.21.15.6:0/138537671 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff44806b500 0x7ff44806d9c0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7ff45c000f60 tx=0x7ff45c009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:53.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.189+0000 7ff464c00640 1 -- 172.21.15.6:0/138537671 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff454059440 con 0x7ff4681552b0 2024-03-04T09:32:53.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.389+0000 7ff46e000640 1 -- 172.21.15.6:0/138537671 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff4680ce9d0 con 0x7ff4681552b0 2024-03-04T09:32:53.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.391+0000 7ff464c00640 1 -- 172.21.15.6:0/138537671 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v123) v1 ==== 74+0+12853 (secure 0 0 0) 0x7ff454058d40 con 0x7ff4681552b0 2024-03-04T09:32:53.388 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:32:53.389 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":123,"fsid":"edce0578-5ae4-4402-b45a-552bdef84d7c","created":"2024-03-04T09:24:54.523780+0000","modified":"2024-03-04T09:32:52.400763+0000","last_up_change":"2024-03-04T09:25:07.089049+0000","last_in_change":"2024-03-04T09:24:57.982077+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T09:25:07.935198+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a96574f-81f1-4bc6-8463-757520b84a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6825","nonce":1364449401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6827","nonce":1364449401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6831","nonce":1364449401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6829","nonce":1364449401}]},"public_addr":"172.21.15.6:6825/1364449401","cluster_addr":"172.21.15.6:6827/1364449401","heartbeat_back_addr":"172.21.15.6:6831/1364449401","heartbeat_front_addr":"172.21.15.6:6829/1364449401","state":["exists","up"]},{"osd":1,"uuid":"7d79b9c1-6b87-476a-bc36-7db5ee64cd90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6809","nonce":2494103078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6811","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6813","nonce":2494103078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6821","nonce":2494103078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6817","nonce":2494103078}]},"public_addr":"172.21.15.6:6809/2494103078","cluster_addr":"172.21.15.6:6813/2494103078","heartbeat_back_addr":"172.21.15.6:6821/2494103078","heartbeat_front_addr":"172.21.15.6:6817/2494103078","state":["exists","up"]},{"osd":2,"uuid":"492331b9-4ee9-40b6-bacd-85d44cd6c45a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6812","nonce":1500066681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6816","nonce":1500066681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6824","nonce":1500066681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6820","nonce":1500066681}]},"public_addr":"172.21.15.6:6812/1500066681","cluster_addr":"172.21.15.6:6816/1500066681","heartbeat_back_addr":"172.21.15.6:6824/1500066681","heartbeat_front_addr":"172.21.15.6:6820/1500066681","state":["exists","up"]},{"osd":3,"uuid":"21bb2efe-9ba7-4bdc-bc31-180cd1102e21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6801","nonce":1558806123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6803","nonce":1558806123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6807","nonce":1558806123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6805","nonce":1558806123}]},"public_addr":"172.21.15.6:6801/1558806123","cluster_addr":"172.21.15.6:6803/1558806123","heartbeat_back_addr":"172.21.15.6:6807/1558806123","heartbeat_front_addr":"172.21.15.6:6805/1558806123","state":["exists","up"]},{"osd":4,"uuid":"6e2c4b71-19d9-4737-83d9-558705d8644a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6801","nonce":1526135780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6803","nonce":1526135780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6807","nonce":1526135780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6805","nonce":1526135780}]},"public_addr":"172.21.15.146:6801/1526135780","cluster_addr":"172.21.15.146:6803/1526135780","heartbeat_back_addr":"172.21.15.146:6807/1526135780","heartbeat_front_addr":"172.21.15.146:6805/1526135780","state":["exists","up"]},{"osd":5,"uuid":"fb69c525-f1be-4b7d-ab32-6c29b0d6fddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6809","nonce":460262507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6811","nonce":460262507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6815","nonce":460262507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6813","nonce":460262507}]},"public_addr":"172.21.15.146:6809/460262507","cluster_addr":"172.21.15.146:6811/460262507","heartbeat_back_addr":"172.21.15.146:6815/460262507","heartbeat_front_addr":"172.21.15.146:6813/460262507","state":["exists","up"]},{"osd":6,"uuid":"c32e7fed-370d-4401-8ce0-6a4335b0d168","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6823","nonce":1392897493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6827","nonce":1392897493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6831","nonce":1392897493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6829","nonce":1392897493}]},"public_addr":"172.21.15.146:6823/1392897493","cluster_addr":"172.21.15.146:6827/1392897493","heartbeat_back_addr":"172.21.15.146:6831/1392897493","heartbeat_front_addr":"172.21.15.146:6829/1392897493","state":["exists","up"]},{"osd":7,"uuid":"75a79b44-fa5f-426d-a7d3-71f21f78835c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6817","nonce":3268323156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6819","nonce":3268323156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6825","nonce":3268323156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6822","nonce":3268323156}]},"public_addr":"172.21.15.146:6817/3268323156","cluster_addr":"172.21.15.146:6819/3268323156","heartbeat_back_addr":"172.21.15.146:6825/3268323156","heartbeat_front_addr":"172.21.15.146:6822/3268323156","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.310437+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.295850+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.294674+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.526855+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.300023+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.595160+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.346046+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.558741+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T09:32:53.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.393+0000 7ff46e000640 1 -- 172.21.15.6:0/138537671 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff44806b500 msgr2=0x7ff44806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:53.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.393+0000 7ff46e000640 1 --2- 172.21.15.6:0/138537671 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff44806b500 0x7ff44806d9c0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7ff45c000f60 tx=0x7ff45c009040 comp rx=0 tx=0).stop 2024-03-04T09:32:53.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.393+0000 7ff46e000640 1 -- 172.21.15.6:0/138537671 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff4681552b0 msgr2=0x7ff46814dc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:53.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.393+0000 7ff46e000640 1 --2- 172.21.15.6:0/138537671 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff4681552b0 0x7ff46814dc30 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 crypto rx=0x7ff45400d5f0 tx=0x7ff45400dab0 comp rx=0 tx=0).stop 2024-03-04T09:32:53.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.393+0000 7ff46e000640 1 -- 172.21.15.6:0/138537671 shutdown_connections 2024-03-04T09:32:53.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.393+0000 7ff46e000640 1 --2- 172.21.15.6:0/138537671 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff46815e810 0x7ff46815d8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:53.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.393+0000 7ff46e000640 1 --2- 172.21.15.6:0/138537671 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff4681552b0 0x7ff46814dc30 unknown :-1 s=CLOSED pgs=1470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:53.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.393+0000 7ff46e000640 1 --2- 172.21.15.6:0/138537671 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff44806b500 0x7ff44806d9c0 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:53.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.393+0000 7ff46e000640 1 --2- 172.21.15.6:0/138537671 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff4680cd060 0x7ff4681503c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:53.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.394+0000 7ff46e000640 1 -- 172.21.15.6:0/138537671 >> 172.21.15.6:0/138537671 conn(0x7ff46806b3d0 msgr2=0x7ff4680b4a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:32:53.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.394+0000 7ff46e000640 1 -- 172.21.15.6:0/138537671 shutdown_connections 2024-03-04T09:32:53.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.394+0000 7ff46e000640 1 -- 172.21.15.6:0/138537671 wait complete. 2024-03-04T09:32:53.412 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T09:25:07.935198+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-04T09:32:53.412 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-04T09:32:53.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.632+0000 7f40a0200640 1 Processor -- start 2024-03-04T09:32:53.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.632+0000 7f40a0200640 1 -- start start 2024-03-04T09:32:53.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.634+0000 7f40a0200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f409806da40 0x7f409806e340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:53.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.634+0000 7f40a0200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f409806d090 con 0x7f409815e680 2024-03-04T09:32:53.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.634+0000 7f40a0200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f409815eb80 con 0x7f40980ccad0 2024-03-04T09:32:53.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.634+0000 7f40a0200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4098161c70 con 0x7f409806da40 2024-03-04T09:32:53.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.634+0000 7f409ce00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f409815e680 0x7f409806c980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:35652/0 (socket says 172.21.15.6:35652) 2024-03-04T09:32:53.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.634+0000 7f409ce00640 1 -- 172.21.15.6:0/4228973422 learned_addr learned my addr 172.21.15.6:0/4228973422 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:32:53.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.635+0000 7f409d800640 1 --2- 172.21.15.6:0/4228973422 >> v2:172.21.15.146:3300/0 conn(0x7f409806da40 0x7f409806e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:53.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.635+0000 7f4097e00640 1 -- 172.21.15.6:0/4228973422 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4094464332 0 0) 0x7f409806d090 con 0x7f409815e680 2024-03-04T09:32:53.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.635+0000 7f4097e00640 1 -- 172.21.15.6:0/4228973422 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4080003520 con 0x7f409815e680 2024-03-04T09:32:53.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.635+0000 7f4097e00640 1 -- 172.21.15.6:0/4228973422 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2600498860 0 0) 0x7f4080003520 con 0x7f409815e680 2024-03-04T09:32:53.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.635+0000 7f4097e00640 1 -- 172.21.15.6:0/4228973422 >> v2:172.21.15.146:3300/0 conn(0x7f409806da40 msgr2=0x7f409806e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:32:53.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.635+0000 7f4097e00640 1 --2- 172.21.15.6:0/4228973422 >> v2:172.21.15.146:3300/0 conn(0x7f409806da40 0x7f409806e340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:53.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.635+0000 7f4097e00640 1 -- 172.21.15.6:0/4228973422 >> v1:172.21.15.146:6789/0 conn(0x7f40980ccad0 legacy=0x7f4098162100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:53.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.635+0000 7f4097e00640 1 -- 172.21.15.6:0/4228973422 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f409815ee60 con 0x7f409815e680 2024-03-04T09:32:53.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.635+0000 7f4097e00640 1 -- 172.21.15.6:0/4228973422 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7f4084001670 con 0x7f409815e680 2024-03-04T09:32:53.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.635+0000 7f4097e00640 1 -- 172.21.15.6:0/4228973422 >> v1:172.21.15.6:6789/0 conn(0x7f409815e680 legacy=0x7f409806c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:53.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.635+0000 7f4097e00640 1 --2- 172.21.15.6:0/4228973422 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f40800039b0 0x7f4080003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:53.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.636+0000 7f4097e00640 1 --2- 172.21.15.6:0/4228973422 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4080004800 0x7f4080004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:53.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.636+0000 7f4097e00640 1 --2- 172.21.15.6:0/4228973422 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f40800051c0 0x7f408000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:53.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.636+0000 7f4097e00640 1 -- 172.21.15.6:0/4228973422 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4098161c70 con 0x7f40800039b0 2024-03-04T09:32:53.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.636+0000 7f4097e00640 1 -- 172.21.15.6:0/4228973422 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f408000c060 con 0x7f40800051c0 2024-03-04T09:32:53.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.636+0000 7f4097e00640 1 -- 172.21.15.6:0/4228973422 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f408000c260 con 0x7f4080004800 2024-03-04T09:32:53.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.636+0000 7f409d800640 1 --2- 172.21.15.6:0/4228973422 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f40800039b0 0x7f4080003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:53.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.636+0000 7f409d800640 1 -- 172.21.15.6:0/4228973422 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4080004800 msgr2=0x7f4080004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:32:53.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.636+0000 7f409d800640 1 --2- 172.21.15.6:0/4228973422 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4080004800 0x7f4080004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:53.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.636+0000 7f409d800640 1 -- 172.21.15.6:0/4228973422 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f40800051c0 msgr2=0x7f408000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:32:53.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.636+0000 7f409d800640 1 --2- 172.21.15.6:0/4228973422 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f40800051c0 0x7f408000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:53.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.636+0000 7f409d800640 1 -- 172.21.15.6:0/4228973422 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4080003520 con 0x7f40800039b0 2024-03-04T09:32:53.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.636+0000 7f409e200640 1 -- 172.21.15.6:0/4228973422 reap_dead start 2024-03-04T09:32:53.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.638+0000 7f409d800640 1 --2- 172.21.15.6:0/4228973422 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f40800039b0 0x7f4080003dd0 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7f408c009ac0 tx=0x7f4080005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fa4ace0aa12e737 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:53.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.638+0000 7f4097e00640 1 -- 172.21.15.6:0/4228973422 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f408c039070 con 0x7f40800039b0 2024-03-04T09:32:53.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.638+0000 7f4097e00640 1 -- 172.21.15.6:0/4228973422 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f408c02c6d0 con 0x7f40800039b0 2024-03-04T09:32:53.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.638+0000 7f4097e00640 1 -- 172.21.15.6:0/4228973422 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f408c009690 con 0x7f40800039b0 2024-03-04T09:32:53.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.638+0000 7f40a0200640 1 -- 172.21.15.6:0/4228973422 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f40800039b0 msgr2=0x7f4080003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:53.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.638+0000 7f40a0200640 1 --2- 172.21.15.6:0/4228973422 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f40800039b0 0x7f4080003dd0 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7f408c009ac0 tx=0x7f4080005830 comp rx=0 tx=0).stop 2024-03-04T09:32:53.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.639+0000 7f40a0200640 1 -- 172.21.15.6:0/4228973422 shutdown_connections 2024-03-04T09:32:53.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.639+0000 7f40a0200640 1 --2- 172.21.15.6:0/4228973422 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f40800039b0 0x7f4080003dd0 unknown :-1 s=CLOSED pgs=1472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:53.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.639+0000 7f40a0200640 1 -- 172.21.15.6:0/4228973422 >> 172.21.15.6:0/4228973422 conn(0x7f409806b3d0 msgr2=0x7f409806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:32:53.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.639+0000 7f40a0200640 1 -- 172.21.15.6:0/4228973422 shutdown_connections 2024-03-04T09:32:53.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.639+0000 7f40a0200640 1 -- 172.21.15.6:0/4228973422 wait complete. 2024-03-04T09:32:53.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.641+0000 7f40a0200640 1 Processor -- start 2024-03-04T09:32:53.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.641+0000 7f40a0200640 1 -- start start 2024-03-04T09:32:53.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.642+0000 7f40a0200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f40800039b0 0x7f409814b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:53.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.642+0000 7f40a0200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f40980b46f0 0x7f40980b4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:53.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.642+0000 7f40a0200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f40980cac40 0x7f40980cb0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:53.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.642+0000 7f40a0200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40980ccf80 con 0x7f40980cac40 2024-03-04T09:32:53.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.642+0000 7f40a0200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f409815f420 con 0x7f40800039b0 2024-03-04T09:32:53.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.642+0000 7f40a0200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40980cd100 con 0x7f40980b46f0 2024-03-04T09:32:53.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.643+0000 7f409e200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f40980cac40 0x7f40980cb0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:53.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.643+0000 7f409e200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f40980cac40 0x7f40980cb0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43418/0 (socket says 172.21.15.6:43418) 2024-03-04T09:32:53.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.643+0000 7f409e200640 1 -- 172.21.15.6:0/2136439280 learned_addr learned my addr 172.21.15.6:0/2136439280 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:32:53.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.643+0000 7f409d800640 1 --2- 172.21.15.6:0/2136439280 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f40800039b0 0x7f409814b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:53.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.643+0000 7f409ce00640 1 --2- 172.21.15.6:0/2136439280 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f40980b46f0 0x7f40980b4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:53.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.643+0000 7f409e200640 1 -- 172.21.15.6:0/2136439280 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f40980b46f0 msgr2=0x7f40980b4b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:53.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.643+0000 7f409e200640 1 --2- 172.21.15.6:0/2136439280 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f40980b46f0 0x7f40980b4b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:53.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.643+0000 7f409e200640 1 -- 172.21.15.6:0/2136439280 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f40800039b0 msgr2=0x7f409814b540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:53.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.643+0000 7f409e200640 1 --2- 172.21.15.6:0/2136439280 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f40800039b0 0x7f409814b540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:53.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.643+0000 7f409e200640 1 -- 172.21.15.6:0/2136439280 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f409814ba80 con 0x7f40980cac40 2024-03-04T09:32:53.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.643+0000 7f409ce00640 1 --2- 172.21.15.6:0/2136439280 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f40980b46f0 0x7f40980b4b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:32:53.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.644+0000 7f409e200640 1 --2- 172.21.15.6:0/2136439280 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f40980cac40 0x7f40980cb0c0 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7f409809e090 tx=0x7f409000c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:53.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.644+0000 7f4095600640 1 -- 172.21.15.6:0/2136439280 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4090017070 con 0x7f40980cac40 2024-03-04T09:32:53.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.644+0000 7f4095600640 1 -- 172.21.15.6:0/2136439280 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4090007070 con 0x7f40980cac40 2024-03-04T09:32:53.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.644+0000 7f4095600640 1 -- 172.21.15.6:0/2136439280 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4090007440 con 0x7f40980cac40 2024-03-04T09:32:53.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.644+0000 7f40a0200640 1 -- 172.21.15.6:0/2136439280 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40980b3a90 con 0x7f40980cac40 2024-03-04T09:32:53.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.644+0000 7f40a0200640 1 -- 172.21.15.6:0/2136439280 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40980b2cd0 con 0x7f40980cac40 2024-03-04T09:32:53.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.645+0000 7f40a0200640 1 -- 172.21.15.6:0/2136439280 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f409804f330 con 0x7f40980cac40 2024-03-04T09:32:53.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.647+0000 7f4095600640 1 -- 172.21.15.6:0/2136439280 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f409001a070 con 0x7f40980cac40 2024-03-04T09:32:53.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.647+0000 7f4095600640 1 --2- 172.21.15.6:0/2136439280 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f407806b4d0 0x7f407806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:53.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.647+0000 7f4095600640 1 -- 172.21.15.6:0/2136439280 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f4090089160 con 0x7f40980cac40 2024-03-04T09:32:53.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.648+0000 7f409d800640 1 --2- 172.21.15.6:0/2136439280 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f407806b4d0 0x7f407806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:53.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.649+0000 7f409d800640 1 --2- 172.21.15.6:0/2136439280 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f407806b4d0 0x7f407806d990 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f408c036000 tx=0x7f408c030000 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:53.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.652+0000 7f4095600640 1 -- 172.21.15.6:0/2136439280 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f409814ba80 con 0x7f40980cac40 2024-03-04T09:32:53.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.865+0000 7f40a0200640 1 -- 172.21.15.6:0/2136439280 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f409806b7b0 con 0x7f40980cac40 2024-03-04T09:32:53.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.867+0000 7f4095600640 1 -- 172.21.15.6:0/2136439280 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v123) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4090058290 con 0x7f40980cac40 2024-03-04T09:32:53.863 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-04T09:32:53.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.870+0000 7f40a0200640 1 -- 172.21.15.6:0/2136439280 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f407806b4d0 msgr2=0x7f407806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:53.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.870+0000 7f40a0200640 1 --2- 172.21.15.6:0/2136439280 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f407806b4d0 0x7f407806d990 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f408c036000 tx=0x7f408c030000 comp rx=0 tx=0).stop 2024-03-04T09:32:53.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.870+0000 7f40a0200640 1 -- 172.21.15.6:0/2136439280 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f40980cac40 msgr2=0x7f40980cb0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:53.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.870+0000 7f40a0200640 1 --2- 172.21.15.6:0/2136439280 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f40980cac40 0x7f40980cb0c0 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7f409809e090 tx=0x7f409000c960 comp rx=0 tx=0).stop 2024-03-04T09:32:53.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.870+0000 7f40a0200640 1 -- 172.21.15.6:0/2136439280 shutdown_connections 2024-03-04T09:32:53.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.870+0000 7f40a0200640 1 --2- 172.21.15.6:0/2136439280 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f40980cac40 0x7f40980cb0c0 unknown :-1 s=CLOSED pgs=1473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:53.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.870+0000 7f40a0200640 1 --2- 172.21.15.6:0/2136439280 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f407806b4d0 0x7f407806d990 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:53.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.870+0000 7f40a0200640 1 --2- 172.21.15.6:0/2136439280 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f40980b46f0 0x7f40980b4b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:53.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.870+0000 7f40a0200640 1 --2- 172.21.15.6:0/2136439280 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f40800039b0 0x7f409814b540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:53.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.870+0000 7f40a0200640 1 -- 172.21.15.6:0/2136439280 >> 172.21.15.6:0/2136439280 conn(0x7f409806b3d0 msgr2=0x7f40980ad670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:32:53.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.870+0000 7f40a0200640 1 -- 172.21.15.6:0/2136439280 shutdown_connections 2024-03-04T09:32:53.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:53.870+0000 7f40a0200640 1 -- 172.21.15.6:0/2136439280 wait complete. 2024-03-04T09:32:53.886 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-04T09:32:54.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.101+0000 7faf7b200640 1 Processor -- start 2024-03-04T09:32:54.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.102+0000 7faf7b200640 1 -- start start 2024-03-04T09:32:54.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.104+0000 7faf7b200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7faf7c145300 0x7faf7c0b49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:54.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.104+0000 7faf7b200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faf7c070f10 0x7faf7c0cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:54.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.104+0000 7faf7b200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7faf7c0cce90 0x7faf7c0cd930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:54.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.104+0000 7faf7b200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf7c0cd300 con 0x7faf7c0cce90 2024-03-04T09:32:54.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.104+0000 7faf7b200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf7c0714d0 con 0x7faf7c145300 2024-03-04T09:32:54.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.104+0000 7faf7b200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf7c0717b0 con 0x7faf7c070f10 2024-03-04T09:32:54.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.104+0000 7faf7a800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7faf7c0cce90 0x7faf7c0cd930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:54.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.104+0000 7faf7a800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7faf7c0cce90 0x7faf7c0cd930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43422/0 (socket says 172.21.15.6:43422) 2024-03-04T09:32:54.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.104+0000 7faf7a800640 1 -- 172.21.15.6:0/1737410946 learned_addr learned my addr 172.21.15.6:0/1737410946 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:32:54.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.104+0000 7faf79e00640 1 --2- 172.21.15.6:0/1737410946 >> v2:172.21.15.146:3300/0 conn(0x7faf7c145300 0x7faf7c0b49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:54.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.104+0000 7faf79400640 1 --2- 172.21.15.6:0/1737410946 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faf7c070f10 0x7faf7c0cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:54.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.104+0000 7faf7a800640 1 -- 172.21.15.6:0/1737410946 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faf7c070f10 msgr2=0x7faf7c0cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:32:54.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.104+0000 7faf7a800640 1 --2- 172.21.15.6:0/1737410946 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faf7c070f10 0x7faf7c0cc950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:54.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.104+0000 7faf7a800640 1 -- 172.21.15.6:0/1737410946 >> v2:172.21.15.146:3300/0 conn(0x7faf7c145300 msgr2=0x7faf7c0b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:32:54.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.104+0000 7faf7a800640 1 --2- 172.21.15.6:0/1737410946 >> v2:172.21.15.146:3300/0 conn(0x7faf7c145300 0x7faf7c0b49a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:54.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.104+0000 7faf7a800640 1 -- 172.21.15.6:0/1737410946 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf7c071a90 con 0x7faf7c0cce90 2024-03-04T09:32:54.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.106+0000 7faf7a800640 1 --2- 172.21.15.6:0/1737410946 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faf7c0cce90 0x7faf7c0cd930 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 crypto rx=0x7faf7c0cf990 tx=0x7faf7402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4f089a363f97c4 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:54.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.106+0000 7faf78a00640 1 -- 172.21.15.6:0/1737410946 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf7402ca70 con 0x7faf7c0cce90 2024-03-04T09:32:54.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.106+0000 7faf78a00640 1 -- 172.21.15.6:0/1737410946 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf7402cc10 con 0x7faf7c0cce90 2024-03-04T09:32:54.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.106+0000 7faf78a00640 1 -- 172.21.15.6:0/1737410946 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf740343f0 con 0x7faf7c0cce90 2024-03-04T09:32:54.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.106+0000 7faf7b200640 1 -- 172.21.15.6:0/1737410946 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faf7c0cce90 msgr2=0x7faf7c0cd930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:54.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.106+0000 7faf7b200640 1 --2- 172.21.15.6:0/1737410946 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faf7c0cce90 0x7faf7c0cd930 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 crypto rx=0x7faf7c0cf990 tx=0x7faf7402bed0 comp rx=0 tx=0).stop 2024-03-04T09:32:54.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.107+0000 7faf7b200640 1 -- 172.21.15.6:0/1737410946 shutdown_connections 2024-03-04T09:32:54.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.107+0000 7faf7b200640 1 --2- 172.21.15.6:0/1737410946 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faf7c0cce90 0x7faf7c0cd930 unknown :-1 s=CLOSED pgs=1474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:54.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.107+0000 7faf7b200640 1 --2- 172.21.15.6:0/1737410946 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faf7c070f10 0x7faf7c0cc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:54.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.107+0000 7faf7b200640 1 --2- 172.21.15.6:0/1737410946 >> v2:172.21.15.146:3300/0 conn(0x7faf7c145300 0x7faf7c0b49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:54.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.107+0000 7faf7b200640 1 -- 172.21.15.6:0/1737410946 >> 172.21.15.6:0/1737410946 conn(0x7faf7c06b3d0 msgr2=0x7faf7c06b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:32:54.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.107+0000 7faf7b200640 1 -- 172.21.15.6:0/1737410946 shutdown_connections 2024-03-04T09:32:54.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.107+0000 7faf7b200640 1 -- 172.21.15.6:0/1737410946 wait complete. 2024-03-04T09:32:54.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.108+0000 7faf7b200640 1 Processor -- start 2024-03-04T09:32:54.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.109+0000 7faf7b200640 1 -- start start 2024-03-04T09:32:54.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.110+0000 7faf7b200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faf7c070f10 0x7faf7c15b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:54.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.110+0000 7faf7b200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faf7c0cce90 0x7faf7c13f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:54.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.110+0000 7faf7b200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faf7c145300 0x7faf7c0aadb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:54.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.110+0000 7faf7b200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf7c1456e0 con 0x7faf7c0cce90 2024-03-04T09:32:54.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.110+0000 7faf7b200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf7c0b4ee0 con 0x7faf7c145300 2024-03-04T09:32:54.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.110+0000 7faf7b200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf7c0cde70 con 0x7faf7c070f10 2024-03-04T09:32:54.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.110+0000 7faf79400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faf7c0cce90 0x7faf7c13f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:54.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.110+0000 7faf79400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faf7c0cce90 0x7faf7c13f080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43438/0 (socket says 172.21.15.6:43438) 2024-03-04T09:32:54.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.110+0000 7faf79400640 1 -- 172.21.15.6:0/174921681 learned_addr learned my addr 172.21.15.6:0/174921681 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:32:54.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.110+0000 7faf79e00640 1 --2- 172.21.15.6:0/174921681 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faf7c070f10 0x7faf7c15b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:54.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.110+0000 7faf7a800640 1 --2- 172.21.15.6:0/174921681 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faf7c145300 0x7faf7c0aadb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:54.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.110+0000 7faf79400640 1 -- 172.21.15.6:0/174921681 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faf7c070f10 msgr2=0x7faf7c15b6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:54.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.110+0000 7faf79400640 1 --2- 172.21.15.6:0/174921681 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faf7c070f10 0x7faf7c15b6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:54.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.110+0000 7faf79400640 1 -- 172.21.15.6:0/174921681 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faf7c145300 msgr2=0x7faf7c0aadb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:54.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.110+0000 7faf79400640 1 --2- 172.21.15.6:0/174921681 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faf7c145300 0x7faf7c0aadb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:54.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.110+0000 7faf79400640 1 -- 172.21.15.6:0/174921681 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf7c0aa310 con 0x7faf7c0cce90 2024-03-04T09:32:54.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.111+0000 7faf79400640 1 --2- 172.21.15.6:0/174921681 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faf7c0cce90 0x7faf7c13f080 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7faf6800c7f0 tx=0x7faf6800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:54.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.111+0000 7faf72000640 1 -- 172.21.15.6:0/174921681 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf68015070 con 0x7faf7c0cce90 2024-03-04T09:32:54.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.111+0000 7faf72000640 1 -- 172.21.15.6:0/174921681 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf68004240 con 0x7faf7c0cce90 2024-03-04T09:32:54.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.111+0000 7faf72000640 1 -- 172.21.15.6:0/174921681 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf68007620 con 0x7faf7c0cce90 2024-03-04T09:32:54.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.111+0000 7faf7b200640 1 -- 172.21.15.6:0/174921681 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf7c0aa5f0 con 0x7faf7c0cce90 2024-03-04T09:32:54.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.111+0000 7faf7b200640 1 -- 172.21.15.6:0/174921681 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf7c13cf10 con 0x7faf7c0cce90 2024-03-04T09:32:54.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.113+0000 7faf7b200640 1 -- 172.21.15.6:0/174921681 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf7c04f330 con 0x7faf7c0cce90 2024-03-04T09:32:54.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.113+0000 7faf72000640 1 -- 172.21.15.6:0/174921681 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7faf680079d0 con 0x7faf7c0cce90 2024-03-04T09:32:54.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.115+0000 7faf72000640 1 --2- 172.21.15.6:0/174921681 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7faf5c06b4d0 0x7faf5c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:54.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.115+0000 7faf72000640 1 -- 172.21.15.6:0/174921681 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 4857+0+0 (secure 0 0 0) 0x7faf68089fc0 con 0x7faf7c0cce90 2024-03-04T09:32:54.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.115+0000 7faf79e00640 1 --2- 172.21.15.6:0/174921681 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7faf5c06b4d0 0x7faf5c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:54.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.117+0000 7faf79e00640 1 --2- 172.21.15.6:0/174921681 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7faf5c06b4d0 0x7faf5c06d990 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7faf6c00b740 tx=0x7faf6c008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:54.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.120+0000 7faf72000640 1 -- 172.21.15.6:0/174921681 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7faf68059180 con 0x7faf7c0cce90 2024-03-04T09:32:54.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.321+0000 7faf7b200640 1 -- 172.21.15.6:0/174921681 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7faf7c074ab0 con 0x7faf7c0cce90 2024-03-04T09:32:54.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.322+0000 7faf72000640 1 -- 172.21.15.6:0/174921681 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v123) v1 ==== 84+0+65 (secure 0 0 0) 0x7faf68058a80 con 0x7faf7c0cce90 2024-03-04T09:32:54.319 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:32:54.319 INFO:teuthology.orchestra.run.smithi006.stdout:[ 2024-03-04T09:32:54.319 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-04T09:32:54.319 INFO:teuthology.orchestra.run.smithi006.stdout: "poolnum": 1, 2024-03-04T09:32:54.319 INFO:teuthology.orchestra.run.smithi006.stdout: "poolname": ".mgr" 2024-03-04T09:32:54.319 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-04T09:32:54.320 INFO:teuthology.orchestra.run.smithi006.stdout:] 2024-03-04T09:32:54.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.325+0000 7faf7b200640 1 -- 172.21.15.6:0/174921681 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7faf5c06b4d0 msgr2=0x7faf5c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:54.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.325+0000 7faf7b200640 1 --2- 172.21.15.6:0/174921681 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7faf5c06b4d0 0x7faf5c06d990 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7faf6c00b740 tx=0x7faf6c008040 comp rx=0 tx=0).stop 2024-03-04T09:32:54.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.325+0000 7faf7b200640 1 -- 172.21.15.6:0/174921681 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faf7c0cce90 msgr2=0x7faf7c13f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:54.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.325+0000 7faf7b200640 1 --2- 172.21.15.6:0/174921681 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faf7c0cce90 0x7faf7c13f080 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7faf6800c7f0 tx=0x7faf6800ccb0 comp rx=0 tx=0).stop 2024-03-04T09:32:54.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.325+0000 7faf7b200640 1 -- 172.21.15.6:0/174921681 shutdown_connections 2024-03-04T09:32:54.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.325+0000 7faf7b200640 1 --2- 172.21.15.6:0/174921681 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faf7c145300 0x7faf7c0aadb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:54.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.325+0000 7faf7b200640 1 --2- 172.21.15.6:0/174921681 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faf7c0cce90 0x7faf7c13f080 unknown :-1 s=CLOSED pgs=1475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:54.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.325+0000 7faf7b200640 1 --2- 172.21.15.6:0/174921681 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7faf5c06b4d0 0x7faf5c06d990 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:54.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.325+0000 7faf7b200640 1 --2- 172.21.15.6:0/174921681 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faf7c070f10 0x7faf7c15b6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:54.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.325+0000 7faf7b200640 1 -- 172.21.15.6:0/174921681 >> 172.21.15.6:0/174921681 conn(0x7faf7c06b3d0 msgr2=0x7faf7c15ec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:32:54.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.325+0000 7faf7b200640 1 -- 172.21.15.6:0/174921681 shutdown_connections 2024-03-04T09:32:54.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.325+0000 7faf7b200640 1 -- 172.21.15.6:0/174921681 wait complete. 2024-03-04T09:32:54.340 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-04T09:32:54.341 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-04T09:32:54.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.573+0000 7f7b7fe00640 1 Processor -- start 2024-03-04T09:32:54.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.573+0000 7f7b7fe00640 1 -- start start 2024-03-04T09:32:54.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.575+0000 7f7b7fe00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7b80144540 0x7f7b8015f060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:54.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.575+0000 7f7b7fe00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f7b800b2ac0 0x7f7b8006daf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:54.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.575+0000 7f7b7fe00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b8015f5a0 con 0x7f7b8006e030 2024-03-04T09:32:54.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.575+0000 7f7b7fe00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b800b3ff0 con 0x7f7b800b2ac0 2024-03-04T09:32:54.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.575+0000 7f7b7fe00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b8006e460 con 0x7f7b80144540 2024-03-04T09:32:54.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.575+0000 7f7b7f400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f7b8006e030 0x7f7b80071b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:35664/0 (socket says 172.21.15.6:35664) 2024-03-04T09:32:54.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.575+0000 7f7b7f400640 1 -- 172.21.15.6:0/2167080334 learned_addr learned my addr 172.21.15.6:0/2167080334 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:32:54.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.576+0000 7f7b7ea00640 1 --2- 172.21.15.6:0/2167080334 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7b80144540 0x7f7b8015f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:54.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.576+0000 7f7b7e000640 1 --2- 172.21.15.6:0/2167080334 >> v2:172.21.15.146:3300/0 conn(0x7f7b800b2ac0 0x7f7b8006daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:54.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.576+0000 7f7b7d600640 1 -- 172.21.15.6:0/2167080334 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 744273773 0 0) 0x7f7b8015f5a0 con 0x7f7b8006e030 2024-03-04T09:32:54.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.576+0000 7f7b7d600640 1 -- 172.21.15.6:0/2167080334 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b64003520 con 0x7f7b8006e030 2024-03-04T09:32:54.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.576+0000 7f7b7d600640 1 -- 172.21.15.6:0/2167080334 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 22015301 0 0) 0x7f7b64003520 con 0x7f7b8006e030 2024-03-04T09:32:54.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.576+0000 7f7b7d600640 1 -- 172.21.15.6:0/2167080334 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7b80144540 msgr2=0x7f7b8015f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:32:54.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.576+0000 7f7b7d600640 1 --2- 172.21.15.6:0/2167080334 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7b80144540 0x7f7b8015f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:54.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.576+0000 7f7b7d600640 1 -- 172.21.15.6:0/2167080334 >> v2:172.21.15.146:3300/0 conn(0x7f7b800b2ac0 msgr2=0x7f7b8006daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:32:54.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.576+0000 7f7b7d600640 1 --2- 172.21.15.6:0/2167080334 >> v2:172.21.15.146:3300/0 conn(0x7f7b800b2ac0 0x7f7b8006daf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:54.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.576+0000 7f7b7d600640 1 -- 172.21.15.6:0/2167080334 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b800b2f50 con 0x7f7b8006e030 2024-03-04T09:32:54.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.576+0000 7f7b7d600640 1 -- 172.21.15.6:0/2167080334 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7f7b78001690 con 0x7f7b8006e030 2024-03-04T09:32:54.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.576+0000 7f7b7d600640 1 -- 172.21.15.6:0/2167080334 >> v1:172.21.15.6:6789/0 conn(0x7f7b8006e030 legacy=0x7f7b80071b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:54.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.577+0000 7f7b7d600640 1 --2- 172.21.15.6:0/2167080334 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7b80144540 0x7f7b640039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:54.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.577+0000 7f7b7d600640 1 --2- 172.21.15.6:0/2167080334 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7b64003fb0 0x7f7b64004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:54.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.577+0000 7f7b7d600640 1 --2- 172.21.15.6:0/2167080334 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7b64004d40 0x7f7b640051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:54.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.577+0000 7f7b7d600640 1 -- 172.21.15.6:0/2167080334 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b800b3ff0 con 0x7f7b64004d40 2024-03-04T09:32:54.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.577+0000 7f7b7d600640 1 -- 172.21.15.6:0/2167080334 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b8006e460 con 0x7f7b64003fb0 2024-03-04T09:32:54.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.577+0000 7f7b7d600640 1 -- 172.21.15.6:0/2167080334 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b6400dc00 con 0x7f7b80144540 2024-03-04T09:32:54.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.577+0000 7f7b7f400640 1 --2- 172.21.15.6:0/2167080334 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7b64004d40 0x7f7b640051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:54.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.577+0000 7f7b7ea00640 1 --2- 172.21.15.6:0/2167080334 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7b80144540 0x7f7b640039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:54.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.577+0000 7f7b7e000640 1 --2- 172.21.15.6:0/2167080334 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7b64003fb0 0x7f7b64004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:54.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.577+0000 7f7b7f400640 1 -- 172.21.15.6:0/2167080334 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7b80144540 msgr2=0x7f7b640039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:32:54.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.577+0000 7f7b7f400640 1 --2- 172.21.15.6:0/2167080334 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7b80144540 0x7f7b640039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:54.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.577+0000 7f7b7f400640 1 -- 172.21.15.6:0/2167080334 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7b64003fb0 msgr2=0x7f7b64004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:32:54.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.577+0000 7f7b7f400640 1 --2- 172.21.15.6:0/2167080334 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7b64003fb0 0x7f7b64004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:54.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.577+0000 7f7b7f400640 1 -- 172.21.15.6:0/2167080334 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b64003520 con 0x7f7b64004d40 2024-03-04T09:32:54.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.579+0000 7f7b7f400640 1 --2- 172.21.15.6:0/2167080334 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7b64004d40 0x7f7b640051c0 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7f7b7800c160 tx=0x7f7b64005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a43ce568a30a791 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:54.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.579+0000 7f7b7d600640 1 -- 172.21.15.6:0/2167080334 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b78007c60 con 0x7f7b64004d40 2024-03-04T09:32:54.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.579+0000 7f7b7d600640 1 -- 172.21.15.6:0/2167080334 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b78030070 con 0x7f7b64004d40 2024-03-04T09:32:54.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.579+0000 7f7b7d600640 1 -- 172.21.15.6:0/2167080334 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b78005d10 con 0x7f7b64004d40 2024-03-04T09:32:54.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.579+0000 7f7b7fe00640 1 -- 172.21.15.6:0/2167080334 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7b64004d40 msgr2=0x7f7b640051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:54.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.579+0000 7f7b7fe00640 1 --2- 172.21.15.6:0/2167080334 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7b64004d40 0x7f7b640051c0 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7f7b7800c160 tx=0x7f7b64005830 comp rx=0 tx=0).stop 2024-03-04T09:32:54.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.579+0000 7f7b7f400640 1 -- 172.21.15.6:0/2167080334 reap_dead start 2024-03-04T09:32:54.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.580+0000 7f7b7fe00640 1 -- 172.21.15.6:0/2167080334 shutdown_connections 2024-03-04T09:32:54.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.580+0000 7f7b7fe00640 1 -- 172.21.15.6:0/2167080334 >> 172.21.15.6:0/2167080334 conn(0x7f7b800b17c0 msgr2=0x7f7b800b1ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:32:54.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.580+0000 7f7b7fe00640 1 -- 172.21.15.6:0/2167080334 shutdown_connections 2024-03-04T09:32:54.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.580+0000 7f7b7fe00640 1 -- 172.21.15.6:0/2167080334 wait complete. 2024-03-04T09:32:54.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.581+0000 7f7b7fe00640 1 Processor -- start 2024-03-04T09:32:54.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.582+0000 7f7b7fe00640 1 -- start start 2024-03-04T09:32:54.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.583+0000 7f7b7fe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7b64004d40 0x7f7b80134bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:54.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.583+0000 7f7b7fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7b801356c0 0x7f7b80159e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:54.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.583+0000 7f7b7fe00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7b8013c6d0 0x7f7b8013cb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:54.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.583+0000 7f7b7fe00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b800b37f0 con 0x7f7b801356c0 2024-03-04T09:32:54.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.583+0000 7f7b7fe00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b800b3510 con 0x7f7b64004d40 2024-03-04T09:32:54.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.583+0000 7f7b7fe00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b800b3ad0 con 0x7f7b8013c6d0 2024-03-04T09:32:54.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.583+0000 7f7b7e000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7b801356c0 0x7f7b80159e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:54.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.583+0000 7f7b7e000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7b801356c0 0x7f7b80159e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43454/0 (socket says 172.21.15.6:43454) 2024-03-04T09:32:54.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.583+0000 7f7b7e000640 1 -- 172.21.15.6:0/3089234688 learned_addr learned my addr 172.21.15.6:0/3089234688 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:32:54.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.583+0000 7f7b7ea00640 1 --2- 172.21.15.6:0/3089234688 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7b64004d40 0x7f7b80134bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:54.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.583+0000 7f7b7f400640 1 --2- 172.21.15.6:0/3089234688 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7b8013c6d0 0x7f7b8013cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:54.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.583+0000 7f7b7e000640 1 -- 172.21.15.6:0/3089234688 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7b8013c6d0 msgr2=0x7f7b8013cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:54.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.584+0000 7f7b7e000640 1 --2- 172.21.15.6:0/3089234688 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7b8013c6d0 0x7f7b8013cb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:54.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.584+0000 7f7b7e000640 1 -- 172.21.15.6:0/3089234688 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7b64004d40 msgr2=0x7f7b80134bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:54.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.584+0000 7f7b7e000640 1 --2- 172.21.15.6:0/3089234688 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7b64004d40 0x7f7b80134bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:54.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.584+0000 7f7b7e000640 1 -- 172.21.15.6:0/3089234688 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b80135b40 con 0x7f7b801356c0 2024-03-04T09:32:54.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.584+0000 7f7b7e000640 1 --2- 172.21.15.6:0/3089234688 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7b801356c0 0x7f7b80159e60 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7f7b68011660 tx=0x7f7b68011b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:54.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.584+0000 7f7b76a00640 1 -- 172.21.15.6:0/3089234688 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b6800f960 con 0x7f7b801356c0 2024-03-04T09:32:54.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.584+0000 7f7b76a00640 1 -- 172.21.15.6:0/3089234688 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b6800fb00 con 0x7f7b801356c0 2024-03-04T09:32:54.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.584+0000 7f7b7fe00640 1 -- 172.21.15.6:0/3089234688 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b80135dc0 con 0x7f7b801356c0 2024-03-04T09:32:54.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.584+0000 7f7b76a00640 1 -- 172.21.15.6:0/3089234688 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b68015460 con 0x7f7b801356c0 2024-03-04T09:32:54.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.584+0000 7f7b7fe00640 1 -- 172.21.15.6:0/3089234688 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b80139550 con 0x7f7b801356c0 2024-03-04T09:32:54.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.586+0000 7f7b7fe00640 1 -- 172.21.15.6:0/3089234688 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b8004f330 con 0x7f7b801356c0 2024-03-04T09:32:54.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.586+0000 7f7b76a00640 1 -- 172.21.15.6:0/3089234688 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7b68015600 con 0x7f7b801356c0 2024-03-04T09:32:54.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.587+0000 7f7b76a00640 1 --2- 172.21.15.6:0/3089234688 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f7b5c06b500 0x7f7b5c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:54.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.587+0000 7f7b76a00640 1 -- 172.21.15.6:0/3089234688 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f7b6808a7b0 con 0x7f7b801356c0 2024-03-04T09:32:54.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.588+0000 7f7b7ea00640 1 --2- 172.21.15.6:0/3089234688 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f7b5c06b500 0x7f7b5c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:54.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.589+0000 7f7b7ea00640 1 --2- 172.21.15.6:0/3089234688 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f7b5c06b500 0x7f7b5c06d9c0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f7b70007520 tx=0x7f7b7000b040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:54.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.592+0000 7f7b76a00640 1 -- 172.21.15.6:0/3089234688 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7b80135b40 con 0x7f7b801356c0 2024-03-04T09:32:54.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:54.796+0000 7f7b7fe00640 1 -- 172.21.15.6:0/3089234688 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f7b8006e820 con 0x7f7b801356c0 2024-03-04T09:32:55.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:55.650+0000 7f7b76a00640 1 -- 172.21.15.6:0/3089234688 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v124) v1 ==== 138+0+0 (secure 0 0 0) 0x7f7b68019030 con 0x7f7b801356c0 2024-03-04T09:32:55.648 INFO:teuthology.orchestra.run.smithi006.stderr:pool 'cephfs_metadata' created 2024-03-04T09:32:55.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:55.653+0000 7f7b7fe00640 1 -- 172.21.15.6:0/3089234688 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f7b5c06b500 msgr2=0x7f7b5c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:55.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:55.653+0000 7f7b7fe00640 1 --2- 172.21.15.6:0/3089234688 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f7b5c06b500 0x7f7b5c06d9c0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f7b70007520 tx=0x7f7b7000b040 comp rx=0 tx=0).stop 2024-03-04T09:32:55.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:55.653+0000 7f7b7fe00640 1 -- 172.21.15.6:0/3089234688 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7b801356c0 msgr2=0x7f7b80159e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:55.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:55.653+0000 7f7b7fe00640 1 --2- 172.21.15.6:0/3089234688 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7b801356c0 0x7f7b80159e60 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7f7b68011660 tx=0x7f7b68011b20 comp rx=0 tx=0).stop 2024-03-04T09:32:55.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:55.653+0000 7f7b7fe00640 1 -- 172.21.15.6:0/3089234688 shutdown_connections 2024-03-04T09:32:55.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:55.653+0000 7f7b7fe00640 1 --2- 172.21.15.6:0/3089234688 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f7b5c06b500 0x7f7b5c06d9c0 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:55.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:55.653+0000 7f7b7fe00640 1 --2- 172.21.15.6:0/3089234688 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7b8013c6d0 0x7f7b8013cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:55.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:55.653+0000 7f7b7fe00640 1 --2- 172.21.15.6:0/3089234688 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7b801356c0 0x7f7b80159e60 unknown :-1 s=CLOSED pgs=1478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:55.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:55.653+0000 7f7b7fe00640 1 --2- 172.21.15.6:0/3089234688 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7b64004d40 0x7f7b80134bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:55.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:55.654+0000 7f7b7fe00640 1 -- 172.21.15.6:0/3089234688 >> 172.21.15.6:0/3089234688 conn(0x7f7b800b17c0 msgr2=0x7f7b800b4a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:32:55.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:55.654+0000 7f7b7fe00640 1 -- 172.21.15.6:0/3089234688 shutdown_connections 2024-03-04T09:32:55.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:55.654+0000 7f7b7fe00640 1 -- 172.21.15.6:0/3089234688 wait complete. 2024-03-04T09:32:55.680 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-04T09:32:55.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:55.901+0000 7feeae600640 1 Processor -- start 2024-03-04T09:32:55.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:55.901+0000 7feeae600640 1 -- start start 2024-03-04T09:32:55.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:55.903+0000 7feeae600640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7feea8072420 0x7feea815ef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:55.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:55.903+0000 7feeae600640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feea80b3c20 con 0x7feea80b4c60 2024-03-04T09:32:55.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:55.903+0000 7feeae600640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feea80b3ea0 con 0x7feea8144580 2024-03-04T09:32:55.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:55.903+0000 7feeae600640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feea80b4180 con 0x7feea8072420 2024-03-04T09:32:55.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:55.904+0000 7feea7e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7feea80b4c60 0x7feea8071d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:35668/0 (socket says 172.21.15.6:35668) 2024-03-04T09:32:55.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:55.904+0000 7feea7e00640 1 -- 172.21.15.6:0/3454283176 learned_addr learned my addr 172.21.15.6:0/3454283176 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:32:55.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:55.904+0000 7feeadc00640 1 --2- 172.21.15.6:0/3454283176 >> v2:172.21.15.146:3300/0 conn(0x7feea8072420 0x7feea815ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:55.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:55.905+0000 7feea7400640 1 -- 172.21.15.6:0/3454283176 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1115105709 0 0) 0x7fee94001110 con 0x7feea80b4c60 2024-03-04T09:32:55.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:55.905+0000 7feea7400640 1 -- 172.21.15.6:0/3454283176 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee90003520 con 0x7feea80b4c60 2024-03-04T09:32:55.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:55.905+0000 7feea7400640 1 -- 172.21.15.6:0/3454283176 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3379894221 0 0) 0x7feea80b3ea0 con 0x7feea8144580 2024-03-04T09:32:55.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:55.905+0000 7feea7400640 1 -- 172.21.15.6:0/3454283176 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee94001110 con 0x7feea8144580 2024-03-04T09:32:55.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:55.905+0000 7feeadc00640 1 -- 172.21.15.6:0/3454283176 >> v1:172.21.15.146:6789/0 conn(0x7feea8144580 legacy=0x7feea806d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:55.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:55.905+0000 7feeadc00640 1 -- 172.21.15.6:0/3454283176 >> v1:172.21.15.6:6789/0 conn(0x7feea80b4c60 legacy=0x7feea8071d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:55.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:55.905+0000 7feea7400640 1 -- 172.21.15.6:0/3454283176 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1332167725 0 0) 0x7fee90003520 con 0x7feea80b4c60 2024-03-04T09:32:55.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:55.905+0000 7feeadc00640 1 -- 172.21.15.6:0/3454283176 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feea80b4460 con 0x7feea8072420 2024-03-04T09:32:55.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:55.907+0000 7feeadc00640 1 --2- 172.21.15.6:0/3454283176 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feea8072420 0x7feea815ef70 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7feea0009d90 tx=0x7feea002bd80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cfd9ab7c77afad2e server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:55.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:55.907+0000 7feea7400640 1 -- 172.21.15.6:0/3454283176 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feea0039070 con 0x7feea8072420 2024-03-04T09:32:55.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:55.907+0000 7feea7400640 1 -- 172.21.15.6:0/3454283176 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feea002c8b0 con 0x7feea8072420 2024-03-04T09:32:55.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:55.907+0000 7feea7400640 1 -- 172.21.15.6:0/3454283176 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee94001840 con 0x7feea8072420 2024-03-04T09:32:55.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:55.908+0000 7feeae600640 1 -- 172.21.15.6:0/3454283176 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feea8072420 msgr2=0x7feea815ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:55.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:55.908+0000 7feeae600640 1 --2- 172.21.15.6:0/3454283176 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feea8072420 0x7feea815ef70 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7feea0009d90 tx=0x7feea002bd80 comp rx=0 tx=0).stop 2024-03-04T09:32:55.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:55.908+0000 7feeae600640 1 -- 172.21.15.6:0/3454283176 shutdown_connections 2024-03-04T09:32:55.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:55.908+0000 7feeae600640 1 --2- 172.21.15.6:0/3454283176 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feea8072420 0x7feea815ef70 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:55.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:55.908+0000 7feeae600640 1 -- 172.21.15.6:0/3454283176 >> 172.21.15.6:0/3454283176 conn(0x7feea806b190 msgr2=0x7feea806b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:32:55.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:55.908+0000 7feeae600640 1 -- 172.21.15.6:0/3454283176 shutdown_connections 2024-03-04T09:32:55.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:55.908+0000 7feeae600640 1 -- 172.21.15.6:0/3454283176 wait complete. 2024-03-04T09:32:55.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:55.910+0000 7feeae600640 1 Processor -- start 2024-03-04T09:32:55.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:55.910+0000 7feeae600640 1 -- start start 2024-03-04T09:32:55.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:55.911+0000 7feeae600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feea8072420 0x7feea815d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:55.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:55.911+0000 7feeae600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feea80b4c60 0x7feea81f37b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:55.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:55.911+0000 7feeae600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feea8144580 0x7feea813c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:55.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:55.911+0000 7feeae600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feea80ada00 con 0x7feea80b4c60 2024-03-04T09:32:55.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:55.911+0000 7feeae600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feea80ad880 con 0x7feea8144580 2024-03-04T09:32:55.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:55.911+0000 7feeae600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feea80adb80 con 0x7feea8072420 2024-03-04T09:32:55.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:55.911+0000 7feea7e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feea80b4c60 0x7feea81f37b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:55.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:55.911+0000 7feea7e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feea80b4c60 0x7feea81f37b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43470/0 (socket says 172.21.15.6:43470) 2024-03-04T09:32:55.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:55.911+0000 7feea7e00640 1 -- 172.21.15.6:0/2334526908 learned_addr learned my addr 172.21.15.6:0/2334526908 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:32:55.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:55.911+0000 7feead200640 1 --2- 172.21.15.6:0/2334526908 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feea8072420 0x7feea815d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:55.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:55.912+0000 7feeadc00640 1 --2- 172.21.15.6:0/2334526908 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feea8144580 0x7feea813c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:55.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:55.912+0000 7feea7e00640 1 -- 172.21.15.6:0/2334526908 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feea8072420 msgr2=0x7feea815d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:55.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:55.912+0000 7feea7e00640 1 --2- 172.21.15.6:0/2334526908 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feea8072420 0x7feea815d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:55.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:55.912+0000 7feea7e00640 1 -- 172.21.15.6:0/2334526908 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feea8144580 msgr2=0x7feea813c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:55.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:55.912+0000 7feea7e00640 1 --2- 172.21.15.6:0/2334526908 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feea8144580 0x7feea813c5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:55.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:55.912+0000 7feea7e00640 1 -- 172.21.15.6:0/2334526908 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feea815d9a0 con 0x7feea80b4c60 2024-03-04T09:32:55.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:55.912+0000 7feea7e00640 1 --2- 172.21.15.6:0/2334526908 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feea80b4c60 0x7feea81f37b0 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 crypto rx=0x7fee9400cad0 tx=0x7fee9400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:55.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:55.912+0000 7feea4c00640 1 -- 172.21.15.6:0/2334526908 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee94017070 con 0x7feea80b4c60 2024-03-04T09:32:55.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:55.912+0000 7feea4c00640 1 -- 172.21.15.6:0/2334526908 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee94005ce0 con 0x7feea80b4c60 2024-03-04T09:32:55.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:55.912+0000 7feeae600640 1 -- 172.21.15.6:0/2334526908 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feea813d080 con 0x7feea80b4c60 2024-03-04T09:32:55.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:55.912+0000 7feea4c00640 1 -- 172.21.15.6:0/2334526908 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee940072b0 con 0x7feea80b4c60 2024-03-04T09:32:55.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:55.913+0000 7feeae600640 1 -- 172.21.15.6:0/2334526908 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feea813d640 con 0x7feea80b4c60 2024-03-04T09:32:55.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:55.914+0000 7feeae600640 1 -- 172.21.15.6:0/2334526908 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feea804f330 con 0x7feea80b4c60 2024-03-04T09:32:55.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:55.916+0000 7feea4c00640 1 -- 172.21.15.6:0/2334526908 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fee9401a070 con 0x7feea80b4c60 2024-03-04T09:32:55.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:55.917+0000 7feea4c00640 1 --2- 172.21.15.6:0/2334526908 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fee8806b4d0 0x7fee8806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:55.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:55.917+0000 7feea4c00640 1 -- 172.21.15.6:0/2334526908 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 5227+0+0 (secure 0 0 0) 0x7fee940897a0 con 0x7feea80b4c60 2024-03-04T09:32:55.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:55.918+0000 7feead200640 1 --2- 172.21.15.6:0/2334526908 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fee8806b4d0 0x7fee8806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:55.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:55.919+0000 7feead200640 1 --2- 172.21.15.6:0/2334526908 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fee8806b4d0 0x7fee8806d990 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fee9c0011e0 tx=0x7fee9c009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:55.919 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:55.922+0000 7feea4c00640 1 -- 172.21.15.6:0/2334526908 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7feea80b3c20 con 0x7feea80b4c60 2024-03-04T09:32:56.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:56.125+0000 7feeae600640 1 -- 172.21.15.6:0/2334526908 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7feea81421d0 con 0x7feea80b4c60 2024-03-04T09:32:56.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:56.666+0000 7feea4c00640 1 -- 172.21.15.6:0/2334526908 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v125) v1 ==== 170+0+0 (secure 0 0 0) 0x7fee940587f0 con 0x7feea80b4c60 2024-03-04T09:32:56.664 INFO:teuthology.orchestra.run.smithi006.stderr:pool 'cephfs_data' created 2024-03-04T09:32:56.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:56.671+0000 7feeae600640 1 -- 172.21.15.6:0/2334526908 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fee8806b4d0 msgr2=0x7fee8806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:56.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:56.671+0000 7feeae600640 1 --2- 172.21.15.6:0/2334526908 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fee8806b4d0 0x7fee8806d990 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fee9c0011e0 tx=0x7fee9c009040 comp rx=0 tx=0).stop 2024-03-04T09:32:56.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:56.671+0000 7feeae600640 1 -- 172.21.15.6:0/2334526908 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feea80b4c60 msgr2=0x7feea81f37b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:56.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:56.671+0000 7feeae600640 1 --2- 172.21.15.6:0/2334526908 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feea80b4c60 0x7feea81f37b0 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 crypto rx=0x7fee9400cad0 tx=0x7fee9400cf90 comp rx=0 tx=0).stop 2024-03-04T09:32:56.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:56.672+0000 7feeae600640 1 -- 172.21.15.6:0/2334526908 shutdown_connections 2024-03-04T09:32:56.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:56.672+0000 7feeae600640 1 --2- 172.21.15.6:0/2334526908 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feea8144580 0x7feea813c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:56.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:56.672+0000 7feeae600640 1 --2- 172.21.15.6:0/2334526908 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feea80b4c60 0x7feea81f37b0 unknown :-1 s=CLOSED pgs=1480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:56.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:56.672+0000 7feeae600640 1 --2- 172.21.15.6:0/2334526908 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fee8806b4d0 0x7fee8806d990 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:56.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:56.672+0000 7feeae600640 1 --2- 172.21.15.6:0/2334526908 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feea8072420 0x7feea815d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:56.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:56.672+0000 7feeae600640 1 -- 172.21.15.6:0/2334526908 >> 172.21.15.6:0/2334526908 conn(0x7feea806b190 msgr2=0x7feea8141450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:32:56.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:56.674+0000 7feeae600640 1 -- 172.21.15.6:0/2334526908 shutdown_connections 2024-03-04T09:32:56.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:56.678+0000 7feeae600640 1 -- 172.21.15.6:0/2334526908 wait complete. 2024-03-04T09:32:56.707 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-04T09:32:56.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:56.896+0000 7f54d5600640 1 Processor -- start 2024-03-04T09:32:56.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:56.897+0000 7f54d5600640 1 -- start start 2024-03-04T09:32:56.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:56.898+0000 7f54d5600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f54d0145430 0x7f54d015f810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:56.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:56.898+0000 7f54d5600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f54d015cc90 0x7f54d00b1b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:56.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:56.898+0000 7f54d5600640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54d00b3fe0 con 0x7f54d00b2080 2024-03-04T09:32:56.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:56.898+0000 7f54d5600640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54d015fe40 con 0x7f54d0145430 2024-03-04T09:32:56.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:56.898+0000 7f54d5600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54d0160120 con 0x7f54d015cc90 2024-03-04T09:32:56.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:56.899+0000 7f54cfe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f54d0145430 0x7f54d015f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:56.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:56.899+0000 7f54d4c00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f54d00b2080 0x7f54d006c800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:35674/0 (socket says 172.21.15.6:35674) 2024-03-04T09:32:56.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:56.899+0000 7f54d4c00640 1 -- 172.21.15.6:0/4071298718 learned_addr learned my addr 172.21.15.6:0/4071298718 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:32:56.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:56.899+0000 7f54cf400640 1 --2- 172.21.15.6:0/4071298718 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f54d015cc90 0x7f54d00b1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:56.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:56.899+0000 7f54cfe00640 1 -- 172.21.15.6:0/4071298718 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f54d015cc90 msgr2=0x7f54d00b1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:32:56.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:56.899+0000 7f54cfe00640 1 --2- 172.21.15.6:0/4071298718 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f54d015cc90 0x7f54d00b1b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:56.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:56.899+0000 7f54cea00640 1 -- 172.21.15.6:0/4071298718 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1718546080 0 0) 0x7f54d00b3fe0 con 0x7f54d00b2080 2024-03-04T09:32:56.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:56.899+0000 7f54cfe00640 1 -- 172.21.15.6:0/4071298718 >> v1:172.21.15.6:6789/0 conn(0x7f54d00b2080 legacy=0x7f54d006c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:56.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:56.899+0000 7f54cfe00640 1 -- 172.21.15.6:0/4071298718 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54d0160400 con 0x7f54d0145430 2024-03-04T09:32:56.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:56.901+0000 7f54cfe00640 1 --2- 172.21.15.6:0/4071298718 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54d0145430 0x7f54d015f810 secure :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 crypto rx=0x7f54c00097a0 tx=0x7f54c002b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2758e1dbdb83983 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:56.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:56.901+0000 7f54cea00640 1 -- 172.21.15.6:0/4071298718 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54c0035020 con 0x7f54d0145430 2024-03-04T09:32:56.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:56.901+0000 7f54cea00640 1 -- 172.21.15.6:0/4071298718 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54c0031810 con 0x7f54d0145430 2024-03-04T09:32:56.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:56.901+0000 7f54cea00640 1 -- 172.21.15.6:0/4071298718 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54c0031be0 con 0x7f54d0145430 2024-03-04T09:32:56.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:56.901+0000 7f54d5600640 1 -- 172.21.15.6:0/4071298718 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54d0145430 msgr2=0x7f54d015f810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:56.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:56.901+0000 7f54d5600640 1 --2- 172.21.15.6:0/4071298718 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54d0145430 0x7f54d015f810 secure :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 crypto rx=0x7f54c00097a0 tx=0x7f54c002b8b0 comp rx=0 tx=0).stop 2024-03-04T09:32:56.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:56.902+0000 7f54d5600640 1 -- 172.21.15.6:0/4071298718 shutdown_connections 2024-03-04T09:32:56.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:56.902+0000 7f54d5600640 1 --2- 172.21.15.6:0/4071298718 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f54d015cc90 0x7f54d00b1b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:56.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:56.902+0000 7f54d5600640 1 --2- 172.21.15.6:0/4071298718 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54d0145430 0x7f54d015f810 unknown :-1 s=CLOSED pgs=1482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:56.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:56.902+0000 7f54d5600640 1 -- 172.21.15.6:0/4071298718 >> 172.21.15.6:0/4071298718 conn(0x7f54d006b3a0 msgr2=0x7f54d006b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:32:56.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:56.902+0000 7f54d5600640 1 -- 172.21.15.6:0/4071298718 shutdown_connections 2024-03-04T09:32:56.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:56.902+0000 7f54d5600640 1 -- 172.21.15.6:0/4071298718 wait complete. 2024-03-04T09:32:56.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:56.903+0000 7f54d5600640 1 Processor -- start 2024-03-04T09:32:56.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:56.903+0000 7f54d5600640 1 -- start start 2024-03-04T09:32:56.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:56.904+0000 7f54d5600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f54d00b2080 0x7f54d0135170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:56.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:56.904+0000 7f54d5600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54d0145430 0x7f54d015b710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:56.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:56.904+0000 7f54d5600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f54d015cc90 0x7f54d0142330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:56.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:56.904+0000 7f54d5600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54d0145810 con 0x7f54d0145430 2024-03-04T09:32:56.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:56.904+0000 7f54d5600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54d015d0d0 con 0x7f54d015cc90 2024-03-04T09:32:56.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:56.904+0000 7f54d5600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54d0145990 con 0x7f54d00b2080 2024-03-04T09:32:56.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:56.905+0000 7f54cf400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54d0145430 0x7f54d015b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:56.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:56.905+0000 7f54cf400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54d0145430 0x7f54d015b710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43488/0 (socket says 172.21.15.6:43488) 2024-03-04T09:32:56.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:56.905+0000 7f54cf400640 1 -- 172.21.15.6:0/3267982949 learned_addr learned my addr 172.21.15.6:0/3267982949 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:32:56.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:56.905+0000 7f54cfe00640 1 --2- 172.21.15.6:0/3267982949 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f54d00b2080 0x7f54d0135170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:56.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:56.905+0000 7f54d4c00640 1 --2- 172.21.15.6:0/3267982949 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f54d015cc90 0x7f54d0142330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:56.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:56.905+0000 7f54cf400640 1 -- 172.21.15.6:0/3267982949 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f54d00b2080 msgr2=0x7f54d0135170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:56.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:56.905+0000 7f54cf400640 1 --2- 172.21.15.6:0/3267982949 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f54d00b2080 0x7f54d0135170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:56.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:56.905+0000 7f54cf400640 1 -- 172.21.15.6:0/3267982949 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f54d015cc90 msgr2=0x7f54d0142330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:56.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:56.905+0000 7f54cf400640 1 --2- 172.21.15.6:0/3267982949 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f54d015cc90 0x7f54d0142330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:56.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:56.905+0000 7f54cf400640 1 -- 172.21.15.6:0/3267982949 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54d0142ef0 con 0x7f54d0145430 2024-03-04T09:32:56.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:56.906+0000 7f54cf400640 1 --2- 172.21.15.6:0/3267982949 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54d0145430 0x7f54d015b710 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7f54b8007e70 tx=0x7f54b800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:56.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:56.906+0000 7f54c7e00640 1 -- 172.21.15.6:0/3267982949 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54b8014620 con 0x7f54d0145430 2024-03-04T09:32:56.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:56.906+0000 7f54c7e00640 1 -- 172.21.15.6:0/3267982949 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54b8019070 con 0x7f54d0145430 2024-03-04T09:32:56.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:56.906+0000 7f54c7e00640 1 -- 172.21.15.6:0/3267982949 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54b8014a00 con 0x7f54d0145430 2024-03-04T09:32:56.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:56.906+0000 7f54d5600640 1 -- 172.21.15.6:0/3267982949 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54d01431d0 con 0x7f54d0145430 2024-03-04T09:32:56.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:56.906+0000 7f54d5600640 1 -- 172.21.15.6:0/3267982949 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54d0143790 con 0x7f54d0145430 2024-03-04T09:32:56.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:56.907+0000 7f54d5600640 1 -- 172.21.15.6:0/3267982949 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54d004f330 con 0x7f54d0145430 2024-03-04T09:32:56.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:56.908+0000 7f54c7e00640 1 -- 172.21.15.6:0/3267982949 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f54b8014ba0 con 0x7f54d0145430 2024-03-04T09:32:56.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:56.909+0000 7f54c7e00640 1 --2- 172.21.15.6:0/3267982949 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f54ac06b4d0 0x7f54ac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:56.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:56.909+0000 7f54c7e00640 1 -- 172.21.15.6:0/3267982949 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f54b808aa20 con 0x7f54d0145430 2024-03-04T09:32:56.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:56.910+0000 7f54cfe00640 1 --2- 172.21.15.6:0/3267982949 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f54ac06b4d0 0x7f54ac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:56.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:56.911+0000 7f54cfe00640 1 --2- 172.21.15.6:0/3267982949 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f54ac06b4d0 0x7f54ac06d990 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f54c0036000 tx=0x7f54c003c040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:56.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:56.914+0000 7f54c7e00640 1 -- 172.21.15.6:0/3267982949 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f54b80598f0 con 0x7f54d0145430 2024-03-04T09:32:57.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:57.157+0000 7f54d5600640 1 -- 172.21.15.6:0/3267982949 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f54d006e240 con 0x7f54d0145430 2024-03-04T09:32:57.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:57.670+0000 7f54c7e00640 1 -- 172.21.15.6:0/3267982949 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 18 and data pool 19 v88) v1 ==== 174+0+0 (secure 0 0 0) 0x7f54b80591f0 con 0x7f54d0145430 2024-03-04T09:32:57.668 INFO:teuthology.orchestra.run.smithi006.stderr:new fs with metadata pool 18 and data pool 19 2024-03-04T09:32:57.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:57.674+0000 7f54d5600640 1 -- 172.21.15.6:0/3267982949 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f54ac06b4d0 msgr2=0x7f54ac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:57.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:57.674+0000 7f54d5600640 1 --2- 172.21.15.6:0/3267982949 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f54ac06b4d0 0x7f54ac06d990 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f54c0036000 tx=0x7f54c003c040 comp rx=0 tx=0).stop 2024-03-04T09:32:57.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:57.674+0000 7f54d5600640 1 -- 172.21.15.6:0/3267982949 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54d0145430 msgr2=0x7f54d015b710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:57.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:57.674+0000 7f54d5600640 1 --2- 172.21.15.6:0/3267982949 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54d0145430 0x7f54d015b710 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7f54b8007e70 tx=0x7f54b800e4b0 comp rx=0 tx=0).stop 2024-03-04T09:32:57.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:57.675+0000 7f54d5600640 1 -- 172.21.15.6:0/3267982949 shutdown_connections 2024-03-04T09:32:57.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:57.675+0000 7f54d5600640 1 --2- 172.21.15.6:0/3267982949 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f54d015cc90 0x7f54d0142330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:57.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:57.675+0000 7f54d5600640 1 --2- 172.21.15.6:0/3267982949 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54d0145430 0x7f54d015b710 unknown :-1 s=CLOSED pgs=1483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:57.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:57.675+0000 7f54d5600640 1 --2- 172.21.15.6:0/3267982949 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f54ac06b4d0 0x7f54ac06d990 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:57.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:57.675+0000 7f54d5600640 1 --2- 172.21.15.6:0/3267982949 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f54d00b2080 0x7f54d0135170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:57.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:57.675+0000 7f54d5600640 1 -- 172.21.15.6:0/3267982949 >> 172.21.15.6:0/3267982949 conn(0x7f54d006b3a0 msgr2=0x7f54d006b780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:32:57.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:57.675+0000 7f54d5600640 1 -- 172.21.15.6:0/3267982949 shutdown_connections 2024-03-04T09:32:57.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:57.675+0000 7f54d5600640 1 -- 172.21.15.6:0/3267982949 wait complete. 2024-03-04T09:32:57.712 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T09:32:57.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:57.898+0000 7fe838600640 1 Processor -- start 2024-03-04T09:32:57.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:57.898+0000 7fe838600640 1 -- start start 2024-03-04T09:32:57.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:57.899+0000 7fe838600640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fe8300af580 0x7fe83015c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:57.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:57.899+0000 7fe838600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fe8300cdc20 0x7fe8300b2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:57.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:57.899+0000 7fe838600640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe83015cc40 con 0x7fe8300af1a0 2024-03-04T09:32:57.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:57.899+0000 7fe838600640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe83015cf20 con 0x7fe8300cdc20 2024-03-04T09:32:57.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:57.899+0000 7fe838600640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe83015d200 con 0x7fe8300af580 2024-03-04T09:32:57.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:57.900+0000 7fe836600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fe8300cdc20 0x7fe8300b2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:57.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:57.900+0000 7fe836600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fe8300cdc20 0x7fe8300b2dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43492/0 (socket says 172.21.15.6:43492) 2024-03-04T09:32:57.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:57.900+0000 7fe836600640 1 -- 172.21.15.6:0/4049444046 learned_addr learned my addr 172.21.15.6:0/4049444046 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:32:57.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:57.900+0000 7fe835200640 1 --2- 172.21.15.6:0/4049444046 >> v2:172.21.15.146:3300/0 conn(0x7fe8300af580 0x7fe83015c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:57.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:57.900+0000 7fe836600640 1 -- 172.21.15.6:0/4049444046 >> v2:172.21.15.146:3300/0 conn(0x7fe8300af580 msgr2=0x7fe83015c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:32:57.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:57.900+0000 7fe836600640 1 --2- 172.21.15.6:0/4049444046 >> v2:172.21.15.146:3300/0 conn(0x7fe8300af580 0x7fe83015c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:57.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:57.900+0000 7fe836600640 1 -- 172.21.15.6:0/4049444046 >> v1:172.21.15.146:6789/0 conn(0x7fe8300af1a0 legacy=0x7fe8300b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:32:57.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:57.901+0000 7fe836600640 1 -- 172.21.15.6:0/4049444046 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe83015d4e0 con 0x7fe8300cdc20 2024-03-04T09:32:57.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:57.902+0000 7fe836600640 1 --2- 172.21.15.6:0/4049444046 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe8300cdc20 0x7fe8300b2dd0 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7fe8300b54c0 tx=0x7fe82802be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34718934c208816a server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:57.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:57.902+0000 7fe82fe00640 1 -- 172.21.15.6:0/4049444046 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe82802c9f0 con 0x7fe8300cdc20 2024-03-04T09:32:57.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:57.902+0000 7fe82fe00640 1 -- 172.21.15.6:0/4049444046 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe82802cb90 con 0x7fe8300cdc20 2024-03-04T09:32:57.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:57.902+0000 7fe82fe00640 1 -- 172.21.15.6:0/4049444046 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8280343f0 con 0x7fe8300cdc20 2024-03-04T09:32:57.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:57.903+0000 7fe838600640 1 -- 172.21.15.6:0/4049444046 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe8300cdc20 msgr2=0x7fe8300b2dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:57.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:57.903+0000 7fe838600640 1 --2- 172.21.15.6:0/4049444046 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe8300cdc20 0x7fe8300b2dd0 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7fe8300b54c0 tx=0x7fe82802be50 comp rx=0 tx=0).stop 2024-03-04T09:32:57.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:57.903+0000 7fe838600640 1 -- 172.21.15.6:0/4049444046 shutdown_connections 2024-03-04T09:32:57.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:57.903+0000 7fe838600640 1 --2- 172.21.15.6:0/4049444046 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe8300cdc20 0x7fe8300b2dd0 unknown :-1 s=CLOSED pgs=1484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:57.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:57.903+0000 7fe838600640 1 --2- 172.21.15.6:0/4049444046 >> v2:172.21.15.146:3300/0 conn(0x7fe8300af580 0x7fe83015c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:57.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:57.903+0000 7fe838600640 1 -- 172.21.15.6:0/4049444046 >> 172.21.15.6:0/4049444046 conn(0x7fe83006afb0 msgr2=0x7fe83006b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:32:57.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:57.903+0000 7fe838600640 1 -- 172.21.15.6:0/4049444046 shutdown_connections 2024-03-04T09:32:57.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:57.903+0000 7fe838600640 1 -- 172.21.15.6:0/4049444046 wait complete. 2024-03-04T09:32:57.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:57.905+0000 7fe838600640 1 Processor -- start 2024-03-04T09:32:57.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:57.905+0000 7fe838600640 1 -- start start 2024-03-04T09:32:57.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:57.906+0000 7fe838600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe8300af1a0 0x7fe830154040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:57.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:57.906+0000 7fe838600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe8300af580 0x7fe8301525c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:57.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:57.906+0000 7fe838600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe8300cdc20 0x7fe83009b200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:57.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:57.906+0000 7fe838600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8300b36a0 con 0x7fe8300af1a0 2024-03-04T09:32:57.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:57.906+0000 7fe838600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8300b3520 con 0x7fe8300af580 2024-03-04T09:32:57.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:57.906+0000 7fe838600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8300b3820 con 0x7fe8300cdc20 2024-03-04T09:32:57.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:57.906+0000 7fe835c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe8300af1a0 0x7fe830154040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:57.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:57.906+0000 7fe835c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe8300af1a0 0x7fe830154040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43504/0 (socket says 172.21.15.6:43504) 2024-03-04T09:32:57.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:57.906+0000 7fe835c00640 1 -- 172.21.15.6:0/3912227708 learned_addr learned my addr 172.21.15.6:0/3912227708 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:32:57.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:57.906+0000 7fe835c00640 1 -- 172.21.15.6:0/3912227708 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe8300cdc20 msgr2=0x7fe83009b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:57.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:57.906+0000 7fe835c00640 1 --2- 172.21.15.6:0/3912227708 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe8300cdc20 0x7fe83009b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:57.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:57.906+0000 7fe835c00640 1 -- 172.21.15.6:0/3912227708 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe8300af580 msgr2=0x7fe8301525c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:57.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:57.906+0000 7fe835c00640 1 --2- 172.21.15.6:0/3912227708 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe8300af580 0x7fe8301525c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:57.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:57.906+0000 7fe835c00640 1 -- 172.21.15.6:0/3912227708 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe830152fa0 con 0x7fe8300af1a0 2024-03-04T09:32:57.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:57.907+0000 7fe835c00640 1 --2- 172.21.15.6:0/3912227708 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe8300af1a0 0x7fe830154040 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7fe82400c430 tx=0x7fe82400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:57.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:57.907+0000 7fe82d600640 1 -- 172.21.15.6:0/3912227708 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe824011790 con 0x7fe8300af1a0 2024-03-04T09:32:57.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:57.907+0000 7fe82d600640 1 -- 172.21.15.6:0/3912227708 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe824017070 con 0x7fe8300af1a0 2024-03-04T09:32:57.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:57.907+0000 7fe82d600640 1 -- 172.21.15.6:0/3912227708 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe824011bb0 con 0x7fe8300af1a0 2024-03-04T09:32:57.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:57.907+0000 7fe838600640 1 -- 172.21.15.6:0/3912227708 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe830153280 con 0x7fe8300af1a0 2024-03-04T09:32:57.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:57.907+0000 7fe838600640 1 -- 172.21.15.6:0/3912227708 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe830153840 con 0x7fe8300af1a0 2024-03-04T09:32:57.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:57.908+0000 7fe838600640 1 -- 172.21.15.6:0/3912227708 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe83004f3a0 con 0x7fe8300af1a0 2024-03-04T09:32:57.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:57.909+0000 7fe82d600640 1 -- 172.21.15.6:0/3912227708 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe82401c460 con 0x7fe8300af1a0 2024-03-04T09:32:57.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:57.910+0000 7fe82d600640 1 --2- 172.21.15.6:0/3912227708 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fe81006b4d0 0x7fe81006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:57.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:57.910+0000 7fe82d600640 1 -- 172.21.15.6:0/3912227708 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe82408a0f0 con 0x7fe8300af1a0 2024-03-04T09:32:57.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:57.911+0000 7fe835200640 1 --2- 172.21.15.6:0/3912227708 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fe81006b4d0 0x7fe81006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:57.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:57.912+0000 7fe835200640 1 --2- 172.21.15.6:0/3912227708 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fe81006b4d0 0x7fe81006d990 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fe8300cfb60 tx=0x7fe81c008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:57.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:57.915+0000 7fe82d600640 1 -- 172.21.15.6:0/3912227708 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe824058f60 con 0x7fe8300af1a0 2024-03-04T09:32:58.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.113+0000 7fe838600640 1 -- 172.21.15.6:0/3912227708 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe8300ce7a0 con 0x7fe8300af1a0 2024-03-04T09:32:58.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.115+0000 7fe82d600640 1 -- 172.21.15.6:0/3912227708 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v126) v1 ==== 74+0+16644 (secure 0 0 0) 0x7fe824058860 con 0x7fe8300af1a0 2024-03-04T09:32:58.112 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:32:58.113 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":126,"fsid":"edce0578-5ae4-4402-b45a-552bdef84d7c","created":"2024-03-04T09:24:54.523780+0000","modified":"2024-03-04T09:32:57.159223+0000","last_up_change":"2024-03-04T09:25:07.089049+0000","last_in_change":"2024-03-04T09:24:57.982077+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T09:25:07.935198+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-03-04T09:32:54.852054+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"126","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-03-04T09:32:56.177848+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"126","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a96574f-81f1-4bc6-8463-757520b84a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6825","nonce":1364449401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6827","nonce":1364449401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6831","nonce":1364449401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6829","nonce":1364449401}]},"public_addr":"172.21.15.6:6825/1364449401","cluster_addr":"172.21.15.6:6827/1364449401","heartbeat_back_addr":"172.21.15.6:6831/1364449401","heartbeat_front_addr":"172.21.15.6:6829/1364449401","state":["exists","up"]},{"osd":1,"uuid":"7d79b9c1-6b87-476a-bc36-7db5ee64cd90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6809","nonce":2494103078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6811","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6813","nonce":2494103078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6821","nonce":2494103078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6817","nonce":2494103078}]},"public_addr":"172.21.15.6:6809/2494103078","cluster_addr":"172.21.15.6:6813/2494103078","heartbeat_back_addr":"172.21.15.6:6821/2494103078","heartbeat_front_addr":"172.21.15.6:6817/2494103078","state":["exists","up"]},{"osd":2,"uuid":"492331b9-4ee9-40b6-bacd-85d44cd6c45a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6812","nonce":1500066681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6816","nonce":1500066681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6824","nonce":1500066681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6820","nonce":1500066681}]},"public_addr":"172.21.15.6:6812/1500066681","cluster_addr":"172.21.15.6:6816/1500066681","heartbeat_back_addr":"172.21.15.6:6824/1500066681","heartbeat_front_addr":"172.21.15.6:6820/1500066681","state":["exists","up"]},{"osd":3,"uuid":"21bb2efe-9ba7-4bdc-bc31-180cd1102e21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6801","nonce":1558806123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6803","nonce":1558806123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6807","nonce":1558806123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6805","nonce":1558806123}]},"public_addr":"172.21.15.6:6801/1558806123","cluster_addr":"172.21.15.6:6803/1558806123","heartbeat_back_addr":"172.21.15.6:6807/1558806123","heartbeat_front_addr":"172.21.15.6:6805/1558806123","state":["exists","up"]},{"osd":4,"uuid":"6e2c4b71-19d9-4737-83d9-558705d8644a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6801","nonce":1526135780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6803","nonce":1526135780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6807","nonce":1526135780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6805","nonce":1526135780}]},"public_addr":"172.21.15.146:6801/1526135780","cluster_addr":"172.21.15.146:6803/1526135780","heartbeat_back_addr":"172.21.15.146:6807/1526135780","heartbeat_front_addr":"172.21.15.146:6805/1526135780","state":["exists","up"]},{"osd":5,"uuid":"fb69c525-f1be-4b7d-ab32-6c29b0d6fddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6809","nonce":460262507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6811","nonce":460262507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6815","nonce":460262507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6813","nonce":460262507}]},"public_addr":"172.21.15.146:6809/460262507","cluster_addr":"172.21.15.146:6811/460262507","heartbeat_back_addr":"172.21.15.146:6815/460262507","heartbeat_front_addr":"172.21.15.146:6813/460262507","state":["exists","up"]},{"osd":6,"uuid":"c32e7fed-370d-4401-8ce0-6a4335b0d168","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6823","nonce":1392897493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6827","nonce":1392897493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6831","nonce":1392897493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6829","nonce":1392897493}]},"public_addr":"172.21.15.146:6823/1392897493","cluster_addr":"172.21.15.146:6827/1392897493","heartbeat_back_addr":"172.21.15.146:6831/1392897493","heartbeat_front_addr":"172.21.15.146:6829/1392897493","state":["exists","up"]},{"osd":7,"uuid":"75a79b44-fa5f-426d-a7d3-71f21f78835c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6817","nonce":3268323156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6819","nonce":3268323156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6825","nonce":3268323156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6822","nonce":3268323156}]},"public_addr":"172.21.15.146:6817/3268323156","cluster_addr":"172.21.15.146:6819/3268323156","heartbeat_back_addr":"172.21.15.146:6825/3268323156","heartbeat_front_addr":"172.21.15.146:6822/3268323156","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.310437+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.295850+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.294674+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.526855+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.300023+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.595160+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.346046+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.558741+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T09:32:58.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.118+0000 7fe838600640 1 -- 172.21.15.6:0/3912227708 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fe81006b4d0 msgr2=0x7fe81006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:58.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.118+0000 7fe838600640 1 --2- 172.21.15.6:0/3912227708 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fe81006b4d0 0x7fe81006d990 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fe8300cfb60 tx=0x7fe81c008040 comp rx=0 tx=0).stop 2024-03-04T09:32:58.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.118+0000 7fe838600640 1 -- 172.21.15.6:0/3912227708 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe8300af1a0 msgr2=0x7fe830154040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:58.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.118+0000 7fe838600640 1 --2- 172.21.15.6:0/3912227708 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe8300af1a0 0x7fe830154040 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7fe82400c430 tx=0x7fe82400c8f0 comp rx=0 tx=0).stop 2024-03-04T09:32:58.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.118+0000 7fe838600640 1 -- 172.21.15.6:0/3912227708 shutdown_connections 2024-03-04T09:32:58.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.118+0000 7fe838600640 1 --2- 172.21.15.6:0/3912227708 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fe81006b4d0 0x7fe81006d990 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:58.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.118+0000 7fe838600640 1 --2- 172.21.15.6:0/3912227708 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe8300cdc20 0x7fe83009b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:58.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.118+0000 7fe838600640 1 --2- 172.21.15.6:0/3912227708 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe8300af580 0x7fe8301525c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:58.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.118+0000 7fe838600640 1 --2- 172.21.15.6:0/3912227708 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe8300af1a0 0x7fe830154040 unknown :-1 s=CLOSED pgs=1485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:58.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.118+0000 7fe838600640 1 -- 172.21.15.6:0/3912227708 >> 172.21.15.6:0/3912227708 conn(0x7fe83006afb0 msgr2=0x7fe8300ccfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:32:58.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.118+0000 7fe838600640 1 -- 172.21.15.6:0/3912227708 shutdown_connections 2024-03-04T09:32:58.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.118+0000 7fe838600640 1 -- 172.21.15.6:0/3912227708 wait complete. 2024-03-04T09:32:58.135 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T09:32:58.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.362+0000 7fa3dec00640 1 Processor -- start 2024-03-04T09:32:58.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.362+0000 7fa3dec00640 1 -- start start 2024-03-04T09:32:58.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.363+0000 7fa3dec00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa3d81552b0 0x7fa3d815b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:58.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.364+0000 7fa3dec00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fa3d80cd060 0x7fa3d80b3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:58.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.364+0000 7fa3dec00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3d815ed10 con 0x7fa3d815e810 2024-03-04T09:32:58.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.364+0000 7fa3dec00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3d815eff0 con 0x7fa3d80cd060 2024-03-04T09:32:58.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.364+0000 7fa3dec00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3d815f2d0 con 0x7fa3d81552b0 2024-03-04T09:32:58.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.364+0000 7fa3dce00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fa3d815e810 0x7fa3d80cc950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:35684/0 (socket says 172.21.15.6:35684) 2024-03-04T09:32:58.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.364+0000 7fa3dce00640 1 -- 172.21.15.6:0/65459117 learned_addr learned my addr 172.21.15.6:0/65459117 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:32:58.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.364+0000 7fa3de200640 1 --2- 172.21.15.6:0/65459117 >> v2:172.21.15.146:3300/0 conn(0x7fa3d80cd060 0x7fa3d80b3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:58.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.364+0000 7fa3dd800640 1 --2- 172.21.15.6:0/65459117 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa3d81552b0 0x7fa3d815b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:58.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.364+0000 7fa3d7e00640 1 -- 172.21.15.6:0/65459117 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 272831950 0 0) 0x7fa3d815ed10 con 0x7fa3d815e810 2024-03-04T09:32:58.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.365+0000 7fa3d7e00640 1 -- 172.21.15.6:0/65459117 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3c0003520 con 0x7fa3d815e810 2024-03-04T09:32:58.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.365+0000 7fa3d7e00640 1 -- 172.21.15.6:0/65459117 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2878792182 0 0) 0x7fa3c0003520 con 0x7fa3d815e810 2024-03-04T09:32:58.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.365+0000 7fa3d7e00640 1 -- 172.21.15.6:0/65459117 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa3d81552b0 msgr2=0x7fa3d815b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:32:58.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.365+0000 7fa3d7e00640 1 --2- 172.21.15.6:0/65459117 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa3d81552b0 0x7fa3d815b1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:58.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.365+0000 7fa3d7e00640 1 -- 172.21.15.6:0/65459117 >> v2:172.21.15.146:3300/0 conn(0x7fa3d80cd060 msgr2=0x7fa3d80b3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:32:58.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.365+0000 7fa3d7e00640 1 --2- 172.21.15.6:0/65459117 >> v2:172.21.15.146:3300/0 conn(0x7fa3d80cd060 0x7fa3d80b3cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:58.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.365+0000 7fa3d7e00640 1 -- 172.21.15.6:0/65459117 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3d815f5b0 con 0x7fa3d815e810 2024-03-04T09:32:58.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.365+0000 7fa3d7e00640 1 -- 172.21.15.6:0/65459117 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7fa3c8001670 con 0x7fa3d815e810 2024-03-04T09:32:58.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.365+0000 7fa3d7e00640 1 -- 172.21.15.6:0/65459117 >> v1:172.21.15.6:6789/0 conn(0x7fa3d815e810 legacy=0x7fa3d80cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:58.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.365+0000 7fa3d7e00640 1 --2- 172.21.15.6:0/65459117 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa3d81552b0 0x7fa3c00039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:58.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.365+0000 7fa3d7e00640 1 --2- 172.21.15.6:0/65459117 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa3c0003f30 0x7fa3c0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:58.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.365+0000 7fa3d7e00640 1 --2- 172.21.15.6:0/65459117 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa3c0004d40 0x7fa3c00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:58.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.365+0000 7fa3d7e00640 1 -- 172.21.15.6:0/65459117 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3d815eff0 con 0x7fa3c0003f30 2024-03-04T09:32:58.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.365+0000 7fa3d7e00640 1 -- 172.21.15.6:0/65459117 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3d815f2d0 con 0x7fa3c0004d40 2024-03-04T09:32:58.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.365+0000 7fa3d7e00640 1 -- 172.21.15.6:0/65459117 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3c000ba10 con 0x7fa3d81552b0 2024-03-04T09:32:58.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.365+0000 7fa3dce00640 1 --2- 172.21.15.6:0/65459117 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa3c0003f30 0x7fa3c0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:58.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.366+0000 7fa3dd800640 1 --2- 172.21.15.6:0/65459117 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa3d81552b0 0x7fa3c00039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:58.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.366+0000 7fa3de200640 1 --2- 172.21.15.6:0/65459117 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa3c0004d40 0x7fa3c00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:58.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.366+0000 7fa3dce00640 1 -- 172.21.15.6:0/65459117 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa3d81552b0 msgr2=0x7fa3c00039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:32:58.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.366+0000 7fa3dce00640 1 --2- 172.21.15.6:0/65459117 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa3d81552b0 0x7fa3c00039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:58.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.366+0000 7fa3dce00640 1 -- 172.21.15.6:0/65459117 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa3c0004d40 msgr2=0x7fa3c00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:32:58.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.366+0000 7fa3dce00640 1 --2- 172.21.15.6:0/65459117 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa3c0004d40 0x7fa3c00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:58.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.366+0000 7fa3dce00640 1 -- 172.21.15.6:0/65459117 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3c0003520 con 0x7fa3c0003f30 2024-03-04T09:32:58.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.367+0000 7fa3dce00640 1 --2- 172.21.15.6:0/65459117 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa3c0003f30 0x7fa3c0004800 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7fa3c800c2c0 tx=0x7fa3c0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8fc23a6df237dcd server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:58.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.367+0000 7fa3d7e00640 1 -- 172.21.15.6:0/65459117 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3c8007d50 con 0x7fa3c0003f30 2024-03-04T09:32:58.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.367+0000 7fa3d7e00640 1 -- 172.21.15.6:0/65459117 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3c8030070 con 0x7fa3c0003f30 2024-03-04T09:32:58.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.367+0000 7fa3d7e00640 1 -- 172.21.15.6:0/65459117 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3c802cda0 con 0x7fa3c0003f30 2024-03-04T09:32:58.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.368+0000 7fa3dec00640 1 -- 172.21.15.6:0/65459117 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa3c0003f30 msgr2=0x7fa3c0004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:58.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.368+0000 7fa3dec00640 1 --2- 172.21.15.6:0/65459117 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa3c0003f30 0x7fa3c0004800 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7fa3c800c2c0 tx=0x7fa3c0005830 comp rx=0 tx=0).stop 2024-03-04T09:32:58.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.368+0000 7fa3de200640 1 -- 172.21.15.6:0/65459117 reap_dead start 2024-03-04T09:32:58.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.368+0000 7fa3dec00640 1 -- 172.21.15.6:0/65459117 shutdown_connections 2024-03-04T09:32:58.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.368+0000 7fa3dec00640 1 -- 172.21.15.6:0/65459117 >> 172.21.15.6:0/65459117 conn(0x7fa3d806b3d0 msgr2=0x7fa3d806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:32:58.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.368+0000 7fa3dec00640 1 -- 172.21.15.6:0/65459117 shutdown_connections 2024-03-04T09:32:58.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.368+0000 7fa3dec00640 1 -- 172.21.15.6:0/65459117 wait complete. 2024-03-04T09:32:58.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.370+0000 7fa3dec00640 1 Processor -- start 2024-03-04T09:32:58.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.370+0000 7fa3dec00640 1 -- start start 2024-03-04T09:32:58.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.372+0000 7fa3dec00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa3c0003f30 0x7fa3d80b1240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:58.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.372+0000 7fa3dec00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa3d8143f50 0x7fa3d814dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:58.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.372+0000 7fa3dec00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa3d815cf30 0x7fa3d80afd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:58.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.372+0000 7fa3dec00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3d806dd30 con 0x7fa3d8143f50 2024-03-04T09:32:58.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.372+0000 7fa3dec00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3d80b3760 con 0x7fa3c0003f30 2024-03-04T09:32:58.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.372+0000 7fa3dec00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3d806deb0 con 0x7fa3d815cf30 2024-03-04T09:32:58.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.372+0000 7fa3dce00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa3d8143f50 0x7fa3d814dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:58.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.372+0000 7fa3dce00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa3d8143f50 0x7fa3d814dbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43528/0 (socket says 172.21.15.6:43528) 2024-03-04T09:32:58.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.372+0000 7fa3dce00640 1 -- 172.21.15.6:0/1412519062 learned_addr learned my addr 172.21.15.6:0/1412519062 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:32:58.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.372+0000 7fa3dd800640 1 --2- 172.21.15.6:0/1412519062 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa3c0003f30 0x7fa3d80b1240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:58.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.372+0000 7fa3dce00640 1 -- 172.21.15.6:0/1412519062 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa3d815cf30 msgr2=0x7fa3d80afd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:58.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.372+0000 7fa3dce00640 1 --2- 172.21.15.6:0/1412519062 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa3d815cf30 0x7fa3d80afd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:58.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.372+0000 7fa3dce00640 1 -- 172.21.15.6:0/1412519062 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa3c0003f30 msgr2=0x7fa3d80b1240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:58.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.372+0000 7fa3dce00640 1 --2- 172.21.15.6:0/1412519062 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa3c0003f30 0x7fa3d80b1240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:58.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.372+0000 7fa3dce00640 1 -- 172.21.15.6:0/1412519062 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3d8144f20 con 0x7fa3d8143f50 2024-03-04T09:32:58.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.372+0000 7fa3dd800640 1 --2- 172.21.15.6:0/1412519062 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa3c0003f30 0x7fa3d80b1240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:32:58.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.373+0000 7fa3dce00640 1 --2- 172.21.15.6:0/1412519062 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa3d8143f50 0x7fa3d814dbd0 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7fa3c8001970 tx=0x7fa3c8001a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:58.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.373+0000 7fa3d5600640 1 -- 172.21.15.6:0/1412519062 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3c8001c80 con 0x7fa3d8143f50 2024-03-04T09:32:58.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.373+0000 7fa3d5600640 1 -- 172.21.15.6:0/1412519062 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3c8004bf0 con 0x7fa3d8143f50 2024-03-04T09:32:58.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.373+0000 7fa3d5600640 1 -- 172.21.15.6:0/1412519062 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3c8007070 con 0x7fa3d8143f50 2024-03-04T09:32:58.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.373+0000 7fa3dec00640 1 -- 172.21.15.6:0/1412519062 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3d81443d0 con 0x7fa3d8143f50 2024-03-04T09:32:58.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.373+0000 7fa3dec00640 1 -- 172.21.15.6:0/1412519062 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3d8144960 con 0x7fa3d8143f50 2024-03-04T09:32:58.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.374+0000 7fa3dec00640 1 -- 172.21.15.6:0/1412519062 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3d804f330 con 0x7fa3d8143f50 2024-03-04T09:32:58.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.375+0000 7fa3d5600640 1 -- 172.21.15.6:0/1412519062 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa3c8007210 con 0x7fa3d8143f50 2024-03-04T09:32:58.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.376+0000 7fa3d5600640 1 --2- 172.21.15.6:0/1412519062 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa3bc06b470 0x7fa3bc06d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:58.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.376+0000 7fa3d5600640 1 -- 172.21.15.6:0/1412519062 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa3c80ab6f0 con 0x7fa3d8143f50 2024-03-04T09:32:58.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.377+0000 7fa3dd800640 1 --2- 172.21.15.6:0/1412519062 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa3bc06b470 0x7fa3bc06d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:58.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.378+0000 7fa3dd800640 1 --2- 172.21.15.6:0/1412519062 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa3bc06b470 0x7fa3bc06d930 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fa3cc006ce0 tx=0x7fa3cc008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:58.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.381+0000 7fa3d5600640 1 -- 172.21.15.6:0/1412519062 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa3c807a560 con 0x7fa3d8143f50 2024-03-04T09:32:58.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.587+0000 7fa3dec00640 1 -- 172.21.15.6:0/1412519062 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa3d80cf4a0 con 0x7fa3d8143f50 2024-03-04T09:32:58.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.590+0000 7fa3d5600640 1 -- 172.21.15.6:0/1412519062 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v126) v1 ==== 74+0+16644 (secure 0 0 0) 0x7fa3c8079e60 con 0x7fa3d8143f50 2024-03-04T09:32:58.587 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:32:58.588 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":126,"fsid":"edce0578-5ae4-4402-b45a-552bdef84d7c","created":"2024-03-04T09:24:54.523780+0000","modified":"2024-03-04T09:32:57.159223+0000","last_up_change":"2024-03-04T09:25:07.089049+0000","last_in_change":"2024-03-04T09:24:57.982077+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T09:25:07.935198+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-03-04T09:32:54.852054+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"126","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-03-04T09:32:56.177848+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"126","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a96574f-81f1-4bc6-8463-757520b84a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6825","nonce":1364449401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6827","nonce":1364449401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6831","nonce":1364449401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6829","nonce":1364449401}]},"public_addr":"172.21.15.6:6825/1364449401","cluster_addr":"172.21.15.6:6827/1364449401","heartbeat_back_addr":"172.21.15.6:6831/1364449401","heartbeat_front_addr":"172.21.15.6:6829/1364449401","state":["exists","up"]},{"osd":1,"uuid":"7d79b9c1-6b87-476a-bc36-7db5ee64cd90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6809","nonce":2494103078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6811","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6813","nonce":2494103078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6821","nonce":2494103078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6817","nonce":2494103078}]},"public_addr":"172.21.15.6:6809/2494103078","cluster_addr":"172.21.15.6:6813/2494103078","heartbeat_back_addr":"172.21.15.6:6821/2494103078","heartbeat_front_addr":"172.21.15.6:6817/2494103078","state":["exists","up"]},{"osd":2,"uuid":"492331b9-4ee9-40b6-bacd-85d44cd6c45a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6812","nonce":1500066681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6816","nonce":1500066681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6824","nonce":1500066681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6820","nonce":1500066681}]},"public_addr":"172.21.15.6:6812/1500066681","cluster_addr":"172.21.15.6:6816/1500066681","heartbeat_back_addr":"172.21.15.6:6824/1500066681","heartbeat_front_addr":"172.21.15.6:6820/1500066681","state":["exists","up"]},{"osd":3,"uuid":"21bb2efe-9ba7-4bdc-bc31-180cd1102e21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6801","nonce":1558806123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6803","nonce":1558806123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6807","nonce":1558806123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6805","nonce":1558806123}]},"public_addr":"172.21.15.6:6801/1558806123","cluster_addr":"172.21.15.6:6803/1558806123","heartbeat_back_addr":"172.21.15.6:6807/1558806123","heartbeat_front_addr":"172.21.15.6:6805/1558806123","state":["exists","up"]},{"osd":4,"uuid":"6e2c4b71-19d9-4737-83d9-558705d8644a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6801","nonce":1526135780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6803","nonce":1526135780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6807","nonce":1526135780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6805","nonce":1526135780}]},"public_addr":"172.21.15.146:6801/1526135780","cluster_addr":"172.21.15.146:6803/1526135780","heartbeat_back_addr":"172.21.15.146:6807/1526135780","heartbeat_front_addr":"172.21.15.146:6805/1526135780","state":["exists","up"]},{"osd":5,"uuid":"fb69c525-f1be-4b7d-ab32-6c29b0d6fddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6809","nonce":460262507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6811","nonce":460262507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6815","nonce":460262507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6813","nonce":460262507}]},"public_addr":"172.21.15.146:6809/460262507","cluster_addr":"172.21.15.146:6811/460262507","heartbeat_back_addr":"172.21.15.146:6815/460262507","heartbeat_front_addr":"172.21.15.146:6813/460262507","state":["exists","up"]},{"osd":6,"uuid":"c32e7fed-370d-4401-8ce0-6a4335b0d168","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6823","nonce":1392897493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6827","nonce":1392897493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6831","nonce":1392897493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6829","nonce":1392897493}]},"public_addr":"172.21.15.146:6823/1392897493","cluster_addr":"172.21.15.146:6827/1392897493","heartbeat_back_addr":"172.21.15.146:6831/1392897493","heartbeat_front_addr":"172.21.15.146:6829/1392897493","state":["exists","up"]},{"osd":7,"uuid":"75a79b44-fa5f-426d-a7d3-71f21f78835c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6817","nonce":3268323156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6819","nonce":3268323156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6825","nonce":3268323156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6822","nonce":3268323156}]},"public_addr":"172.21.15.146:6817/3268323156","cluster_addr":"172.21.15.146:6819/3268323156","heartbeat_back_addr":"172.21.15.146:6825/3268323156","heartbeat_front_addr":"172.21.15.146:6822/3268323156","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.310437+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.295850+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.294674+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.526855+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.300023+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.595160+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.346046+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.558741+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T09:32:58.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.593+0000 7fa3dec00640 1 -- 172.21.15.6:0/1412519062 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa3bc06b470 msgr2=0x7fa3bc06d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:58.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.593+0000 7fa3dec00640 1 --2- 172.21.15.6:0/1412519062 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa3bc06b470 0x7fa3bc06d930 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fa3cc006ce0 tx=0x7fa3cc008040 comp rx=0 tx=0).stop 2024-03-04T09:32:58.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.593+0000 7fa3dec00640 1 -- 172.21.15.6:0/1412519062 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa3d8143f50 msgr2=0x7fa3d814dbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:58.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.593+0000 7fa3dec00640 1 --2- 172.21.15.6:0/1412519062 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa3d8143f50 0x7fa3d814dbd0 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7fa3c8001970 tx=0x7fa3c8001a40 comp rx=0 tx=0).stop 2024-03-04T09:32:58.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.593+0000 7fa3dec00640 1 -- 172.21.15.6:0/1412519062 shutdown_connections 2024-03-04T09:32:58.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.593+0000 7fa3dec00640 1 --2- 172.21.15.6:0/1412519062 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa3bc06b470 0x7fa3bc06d930 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:58.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.593+0000 7fa3dec00640 1 --2- 172.21.15.6:0/1412519062 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa3d815cf30 0x7fa3d80afd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:58.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.593+0000 7fa3dec00640 1 --2- 172.21.15.6:0/1412519062 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa3d8143f50 0x7fa3d814dbd0 unknown :-1 s=CLOSED pgs=1488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:58.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.593+0000 7fa3dec00640 1 --2- 172.21.15.6:0/1412519062 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa3c0003f30 0x7fa3d80b1240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:58.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.593+0000 7fa3dec00640 1 -- 172.21.15.6:0/1412519062 >> 172.21.15.6:0/1412519062 conn(0x7fa3d806b3d0 msgr2=0x7fa3d8070ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:32:58.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.593+0000 7fa3dec00640 1 -- 172.21.15.6:0/1412519062 shutdown_connections 2024-03-04T09:32:58.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.593+0000 7fa3dec00640 1 -- 172.21.15.6:0/1412519062 wait complete. 2024-03-04T09:32:58.607 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-04T09:32:58.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.830+0000 7f28ff200640 1 Processor -- start 2024-03-04T09:32:58.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.830+0000 7f28ff200640 1 -- start start 2024-03-04T09:32:58.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.831+0000 7f28ff200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f290015dee0 0x7f29000d18a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:58.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.831+0000 7f28ff200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f29000d0d40 0x7f29000b3510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:58.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.831+0000 7f28ff200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29000ceca0 con 0x7f29000d4860 2024-03-04T09:32:58.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.831+0000 7f28ff200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29000cf5d0 con 0x7f290015dee0 2024-03-04T09:32:58.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.831+0000 7f28ff200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29000cef80 con 0x7f29000d0d40 2024-03-04T09:32:58.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.832+0000 7f28fde00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f290015dee0 0x7f29000d18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:58.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.832+0000 7f28fde00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f290015dee0 0x7f29000d18a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.6:56260/0 (socket says 172.21.15.6:56260) 2024-03-04T09:32:58.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.832+0000 7f28fe800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f29000d0d40 0x7f29000b3510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:58.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.832+0000 7f28fde00640 1 -- 172.21.15.6:0/397897916 learned_addr learned my addr 172.21.15.6:0/397897916 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:32:58.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.833+0000 7f28fde00640 1 -- 172.21.15.6:0/397897916 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f29000d0d40 msgr2=0x7f29000b3510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:32:58.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.833+0000 7f28fde00640 1 --2- 172.21.15.6:0/397897916 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f29000d0d40 0x7f29000b3510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:58.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.833+0000 7f28fde00640 1 -- 172.21.15.6:0/397897916 >> v1:172.21.15.146:6789/0 conn(0x7f29000d4860 legacy=0x7f29000d0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:58.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.833+0000 7f28fde00640 1 -- 172.21.15.6:0/397897916 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29000cf260 con 0x7f290015dee0 2024-03-04T09:32:58.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.834+0000 7f28fde00640 1 --2- 172.21.15.6:0/397897916 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f290015dee0 0x7f29000d18a0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f28f0009e50 tx=0x7f28f002bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=784dbaa90cfe337f server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:58.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.835+0000 7f28fca00640 1 -- 172.21.15.6:0/397897916 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28f002cb00 con 0x7f290015dee0 2024-03-04T09:32:58.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.835+0000 7f28fca00640 1 -- 172.21.15.6:0/397897916 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28f002cca0 con 0x7f290015dee0 2024-03-04T09:32:58.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.835+0000 7f28fca00640 1 -- 172.21.15.6:0/397897916 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28f0034610 con 0x7f290015dee0 2024-03-04T09:32:58.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.835+0000 7f28ff200640 1 -- 172.21.15.6:0/397897916 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f290015dee0 msgr2=0x7f29000d18a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:58.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.835+0000 7f28ff200640 1 --2- 172.21.15.6:0/397897916 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f290015dee0 0x7f29000d18a0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f28f0009e50 tx=0x7f28f002bf60 comp rx=0 tx=0).stop 2024-03-04T09:32:58.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.836+0000 7f28ff200640 1 -- 172.21.15.6:0/397897916 shutdown_connections 2024-03-04T09:32:58.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.836+0000 7f28ff200640 1 --2- 172.21.15.6:0/397897916 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f29000d0d40 0x7f29000b3510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:58.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.836+0000 7f28ff200640 1 --2- 172.21.15.6:0/397897916 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f290015dee0 0x7f29000d18a0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:58.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.836+0000 7f28ff200640 1 -- 172.21.15.6:0/397897916 >> 172.21.15.6:0/397897916 conn(0x7f29000d3cd0 msgr2=0x7f29000d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:32:58.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.836+0000 7f28ff200640 1 -- 172.21.15.6:0/397897916 shutdown_connections 2024-03-04T09:32:58.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.836+0000 7f28ff200640 1 -- 172.21.15.6:0/397897916 wait complete. 2024-03-04T09:32:58.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.837+0000 7f28ff200640 1 Processor -- start 2024-03-04T09:32:58.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.838+0000 7f28ff200640 1 -- start start 2024-03-04T09:32:58.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.838+0000 7f28ff200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f29000d0d40 0x7f290014f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:58.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.838+0000 7f28ff200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f29000d4860 0x7f29000c9c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:58.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.838+0000 7f28ff200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f290015dee0 0x7f29001601c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:58.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.838+0000 7f28ff200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29000d1de0 con 0x7f290015dee0 2024-03-04T09:32:58.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.838+0000 7f28ff200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f290015e2c0 con 0x7f29000d4860 2024-03-04T09:32:58.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.838+0000 7f28ff200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29000d1f60 con 0x7f29000d0d40 2024-03-04T09:32:58.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.839+0000 7f28fe800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f290015dee0 0x7f29001601c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:58.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.839+0000 7f28fe800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f290015dee0 0x7f29001601c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43534/0 (socket says 172.21.15.6:43534) 2024-03-04T09:32:58.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.839+0000 7f28fe800640 1 -- 172.21.15.6:0/3191762718 learned_addr learned my addr 172.21.15.6:0/3191762718 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:32:58.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.839+0000 7f28fde00640 1 --2- 172.21.15.6:0/3191762718 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f29000d0d40 0x7f290014f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:58.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.839+0000 7f28fd400640 1 --2- 172.21.15.6:0/3191762718 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f29000d4860 0x7f29000c9c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:58.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.839+0000 7f28fe800640 1 -- 172.21.15.6:0/3191762718 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f29000d0d40 msgr2=0x7f290014f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:58.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.839+0000 7f28fe800640 1 --2- 172.21.15.6:0/3191762718 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f29000d0d40 0x7f290014f0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:58.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.839+0000 7f28fe800640 1 -- 172.21.15.6:0/3191762718 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f29000d4860 msgr2=0x7f29000c9c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:58.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.839+0000 7f28fe800640 1 --2- 172.21.15.6:0/3191762718 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f29000d4860 0x7f29000c9c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:58.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.839+0000 7f28fe800640 1 -- 172.21.15.6:0/3191762718 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29000c83b0 con 0x7f290015dee0 2024-03-04T09:32:58.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.840+0000 7f28fe800640 1 --2- 172.21.15.6:0/3191762718 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f290015dee0 0x7f29001601c0 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7f28f800e7c0 tx=0x7f28f800ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:58.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.840+0000 7f28f6000640 1 -- 172.21.15.6:0/3191762718 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28f8017070 con 0x7f290015dee0 2024-03-04T09:32:58.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.840+0000 7f28f6000640 1 -- 172.21.15.6:0/3191762718 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28f800c730 con 0x7f290015dee0 2024-03-04T09:32:58.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.840+0000 7f28ff200640 1 -- 172.21.15.6:0/3191762718 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29000c8690 con 0x7f290015dee0 2024-03-04T09:32:58.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.840+0000 7f28f6000640 1 -- 172.21.15.6:0/3191762718 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28f800cb20 con 0x7f290015dee0 2024-03-04T09:32:58.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.840+0000 7f28ff200640 1 -- 172.21.15.6:0/3191762718 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29000c8c50 con 0x7f290015dee0 2024-03-04T09:32:58.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.841+0000 7f28ff200640 1 -- 172.21.15.6:0/3191762718 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f290004f330 con 0x7f290015dee0 2024-03-04T09:32:58.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.843+0000 7f28f6000640 1 -- 172.21.15.6:0/3191762718 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f28f801a070 con 0x7f290015dee0 2024-03-04T09:32:58.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.843+0000 7f28f6000640 1 --2- 172.21.15.6:0/3191762718 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f28dc06b4d0 0x7f28dc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:58.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.844+0000 7f28f6000640 1 -- 172.21.15.6:0/3191762718 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f28f808a490 con 0x7f290015dee0 2024-03-04T09:32:58.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.844+0000 7f28fde00640 1 --2- 172.21.15.6:0/3191762718 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f28dc06b4d0 0x7f28dc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:58.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.845+0000 7f28fde00640 1 --2- 172.21.15.6:0/3191762718 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f28dc06b4d0 0x7f28dc06d990 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f28f0035000 tx=0x7f28f0030000 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:58.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:58.848+0000 7f28f6000640 1 -- 172.21.15.6:0/3191762718 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f28f8059280 con 0x7f290015dee0 2024-03-04T09:32:59.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:59.094+0000 7f28ff200640 1 -- 172.21.15.6:0/3191762718 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f290008d7d0 con 0x7f290015dee0 2024-03-04T09:32:59.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:59.706+0000 7f28f6000640 1 -- 172.21.15.6:0/3191762718 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v91) v1 ==== 118+0+0 (secure 0 0 0) 0x7f28f8058b80 con 0x7f290015dee0 2024-03-04T09:32:59.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:59.709+0000 7f28ff200640 1 -- 172.21.15.6:0/3191762718 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f28dc06b4d0 msgr2=0x7f28dc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:59.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:59.710+0000 7f28ff200640 1 --2- 172.21.15.6:0/3191762718 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f28dc06b4d0 0x7f28dc06d990 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f28f0035000 tx=0x7f28f0030000 comp rx=0 tx=0).stop 2024-03-04T09:32:59.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:59.710+0000 7f28ff200640 1 -- 172.21.15.6:0/3191762718 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f290015dee0 msgr2=0x7f29001601c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:59.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:59.710+0000 7f28ff200640 1 --2- 172.21.15.6:0/3191762718 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f290015dee0 0x7f29001601c0 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7f28f800e7c0 tx=0x7f28f800ec80 comp rx=0 tx=0).stop 2024-03-04T09:32:59.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:59.710+0000 7f28ff200640 1 -- 172.21.15.6:0/3191762718 shutdown_connections 2024-03-04T09:32:59.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:59.710+0000 7f28ff200640 1 --2- 172.21.15.6:0/3191762718 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f290015dee0 0x7f29001601c0 unknown :-1 s=CLOSED pgs=1489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:59.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:59.710+0000 7f28ff200640 1 --2- 172.21.15.6:0/3191762718 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f29000d4860 0x7f29000c9c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:59.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:59.710+0000 7f28ff200640 1 --2- 172.21.15.6:0/3191762718 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f28dc06b4d0 0x7f28dc06d990 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:59.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:59.710+0000 7f28ff200640 1 --2- 172.21.15.6:0/3191762718 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f29000d0d40 0x7f290014f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:59.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:59.710+0000 7f28ff200640 1 -- 172.21.15.6:0/3191762718 >> 172.21.15.6:0/3191762718 conn(0x7f29000d3cd0 msgr2=0x7f29000cfa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:32:59.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:59.710+0000 7f28ff200640 1 -- 172.21.15.6:0/3191762718 shutdown_connections 2024-03-04T09:32:59.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:59.710+0000 7f28ff200640 1 -- 172.21.15.6:0/3191762718 wait complete. 2024-03-04T09:32:59.735 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-04T09:32:59.736 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-04T09:32:59.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:59.959+0000 7fa121c00640 1 Processor -- start 2024-03-04T09:32:59.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:59.959+0000 7fa121c00640 1 -- start start 2024-03-04T09:32:59.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:59.961+0000 7fa121c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa11c15c3f0 0x7fa11c15c870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:59.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:59.961+0000 7fa121c00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fa11c0afcf0 0x7fa11c0b0180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:59.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:59.961+0000 7fa121c00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa11c0af250 con 0x7fa11c15c010 2024-03-04T09:32:59.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:59.961+0000 7fa121c00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa11c15cf50 con 0x7fa11c0afcf0 2024-03-04T09:32:59.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:59.961+0000 7fa121c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa11c15d230 con 0x7fa11c15c3f0 2024-03-04T09:32:59.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:59.961+0000 7fa11aa00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fa11c15c010 0x7fa11c06ee30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:58026/0 (socket says 172.21.15.6:58026) 2024-03-04T09:32:59.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:59.961+0000 7fa11aa00640 1 -- 172.21.15.6:0/1901487674 learned_addr learned my addr 172.21.15.6:0/1901487674 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:32:59.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:59.961+0000 7fa112000640 1 --2- 172.21.15.6:0/1901487674 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa11c15c3f0 0x7fa11c15c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:59.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:59.961+0000 7fa11b400640 1 --2- 172.21.15.6:0/1901487674 >> v2:172.21.15.146:3300/0 conn(0x7fa11c0afcf0 0x7fa11c0b0180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:59.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:59.961+0000 7fa11a000640 1 -- 172.21.15.6:0/1901487674 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 415747242 0 0) 0x7fa11c0af250 con 0x7fa11c15c010 2024-03-04T09:32:59.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:59.961+0000 7fa11a000640 1 -- 172.21.15.6:0/1901487674 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa100003520 con 0x7fa11c15c010 2024-03-04T09:32:59.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:59.962+0000 7fa11a000640 1 -- 172.21.15.6:0/1901487674 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1013726372 0 0) 0x7fa100003520 con 0x7fa11c15c010 2024-03-04T09:32:59.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:59.962+0000 7fa11a000640 1 -- 172.21.15.6:0/1901487674 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa11c15c3f0 msgr2=0x7fa11c15c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:32:59.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:59.962+0000 7fa11a000640 1 --2- 172.21.15.6:0/1901487674 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa11c15c3f0 0x7fa11c15c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:59.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:59.962+0000 7fa11a000640 1 -- 172.21.15.6:0/1901487674 >> v2:172.21.15.146:3300/0 conn(0x7fa11c0afcf0 msgr2=0x7fa11c0b0180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:32:59.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:59.962+0000 7fa11a000640 1 --2- 172.21.15.6:0/1901487674 >> v2:172.21.15.146:3300/0 conn(0x7fa11c0afcf0 0x7fa11c0b0180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:59.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:59.962+0000 7fa11a000640 1 -- 172.21.15.6:0/1901487674 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa11c0af4f0 con 0x7fa11c15c010 2024-03-04T09:32:59.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:59.962+0000 7fa11a000640 1 -- 172.21.15.6:0/1901487674 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7fa10c001670 con 0x7fa11c15c010 2024-03-04T09:32:59.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:59.962+0000 7fa11a000640 1 -- 172.21.15.6:0/1901487674 >> v1:172.21.15.6:6789/0 conn(0x7fa11c15c010 legacy=0x7fa11c06ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:59.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:59.962+0000 7fa11a000640 1 --2- 172.21.15.6:0/1901487674 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa11c15c3f0 0x7fa1000039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:59.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:59.962+0000 7fa11a000640 1 --2- 172.21.15.6:0/1901487674 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa100003f80 0x7fa100004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:59.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:59.962+0000 7fa11a000640 1 --2- 172.21.15.6:0/1901487674 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa100004d40 0x7fa1000051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:59.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:59.962+0000 7fa11a000640 1 -- 172.21.15.6:0/1901487674 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa11c15cf50 con 0x7fa100004d40 2024-03-04T09:32:59.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:59.962+0000 7fa11a000640 1 -- 172.21.15.6:0/1901487674 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa11c15d230 con 0x7fa100003f80 2024-03-04T09:32:59.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:59.962+0000 7fa11a000640 1 -- 172.21.15.6:0/1901487674 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa10000ba70 con 0x7fa11c15c3f0 2024-03-04T09:32:59.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:59.963+0000 7fa11b400640 1 --2- 172.21.15.6:0/1901487674 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa100004d40 0x7fa1000051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:59.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:59.963+0000 7fa11aa00640 1 --2- 172.21.15.6:0/1901487674 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa11c15c3f0 0x7fa1000039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:59.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:59.963+0000 7fa112000640 1 --2- 172.21.15.6:0/1901487674 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa100003f80 0x7fa100004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:59.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:59.963+0000 7fa11b400640 1 -- 172.21.15.6:0/1901487674 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa11c15c3f0 msgr2=0x7fa1000039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:32:59.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:59.963+0000 7fa11b400640 1 --2- 172.21.15.6:0/1901487674 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa11c15c3f0 0x7fa1000039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:59.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:59.963+0000 7fa11b400640 1 -- 172.21.15.6:0/1901487674 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa100003f80 msgr2=0x7fa100004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:32:59.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:59.963+0000 7fa11b400640 1 --2- 172.21.15.6:0/1901487674 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa100003f80 0x7fa100004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:59.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:59.963+0000 7fa11b400640 1 -- 172.21.15.6:0/1901487674 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa100003520 con 0x7fa100004d40 2024-03-04T09:32:59.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:59.964+0000 7fa11b400640 1 --2- 172.21.15.6:0/1901487674 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa100004d40 0x7fa1000051c0 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 crypto rx=0x7fa114009fc0 tx=0x7fa100005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa180cfc799bb227 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:59.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:59.965+0000 7fa11a000640 1 -- 172.21.15.6:0/1901487674 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa114039070 con 0x7fa100004d40 2024-03-04T09:32:59.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:59.965+0000 7fa11a000640 1 -- 172.21.15.6:0/1901487674 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa11402cc60 con 0x7fa100004d40 2024-03-04T09:32:59.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:59.965+0000 7fa11a000640 1 -- 172.21.15.6:0/1901487674 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa114009c10 con 0x7fa100004d40 2024-03-04T09:32:59.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:59.965+0000 7fa121c00640 1 -- 172.21.15.6:0/1901487674 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa100004d40 msgr2=0x7fa1000051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:59.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:59.965+0000 7fa121c00640 1 --2- 172.21.15.6:0/1901487674 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa100004d40 0x7fa1000051c0 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 crypto rx=0x7fa114009fc0 tx=0x7fa100005830 comp rx=0 tx=0).stop 2024-03-04T09:32:59.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:59.965+0000 7fa11b400640 1 -- 172.21.15.6:0/1901487674 reap_dead start 2024-03-04T09:32:59.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:59.965+0000 7fa121c00640 1 -- 172.21.15.6:0/1901487674 shutdown_connections 2024-03-04T09:32:59.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:59.965+0000 7fa121c00640 1 -- 172.21.15.6:0/1901487674 >> 172.21.15.6:0/1901487674 conn(0x7fa11c0732d0 msgr2=0x7fa11c09aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:32:59.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:59.966+0000 7fa121c00640 1 -- 172.21.15.6:0/1901487674 shutdown_connections 2024-03-04T09:32:59.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:59.966+0000 7fa121c00640 1 -- 172.21.15.6:0/1901487674 wait complete. 2024-03-04T09:32:59.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:59.967+0000 7fa121c00640 1 Processor -- start 2024-03-04T09:32:59.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:59.968+0000 7fa121c00640 1 -- start start 2024-03-04T09:32:59.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:59.968+0000 7fa121c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa100004d40 0x7fa11c0b43a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:59.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:59.969+0000 7fa121c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa11c1f77e0 0x7fa11c155d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:59.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:59.969+0000 7fa121c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa11c0cf740 0x7fa11c0cc500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:59.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:59.969+0000 7fa121c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa11c0d1c10 con 0x7fa11c1f77e0 2024-03-04T09:32:59.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:59.969+0000 7fa121c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa11c0d1a90 con 0x7fa100004d40 2024-03-04T09:32:59.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:59.969+0000 7fa121c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa11c15c870 con 0x7fa11c0cf740 2024-03-04T09:32:59.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:59.969+0000 7fa112000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa11c1f77e0 0x7fa11c155d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:59.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:59.969+0000 7fa112000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa11c1f77e0 0x7fa11c155d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42912/0 (socket says 172.21.15.6:42912) 2024-03-04T09:32:59.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:59.969+0000 7fa112000640 1 -- 172.21.15.6:0/110474427 learned_addr learned my addr 172.21.15.6:0/110474427 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:32:59.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:59.969+0000 7fa11aa00640 1 --2- 172.21.15.6:0/110474427 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa100004d40 0x7fa11c0b43a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:59.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:59.969+0000 7fa11b400640 1 --2- 172.21.15.6:0/110474427 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa11c0cf740 0x7fa11c0cc500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:59.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:59.969+0000 7fa112000640 1 -- 172.21.15.6:0/110474427 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa11c0cf740 msgr2=0x7fa11c0cc500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:59.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:59.969+0000 7fa112000640 1 --2- 172.21.15.6:0/110474427 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa11c0cf740 0x7fa11c0cc500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:59.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:59.969+0000 7fa112000640 1 -- 172.21.15.6:0/110474427 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa100004d40 msgr2=0x7fa11c0b43a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:32:59.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:59.969+0000 7fa112000640 1 --2- 172.21.15.6:0/110474427 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa100004d40 0x7fa11c0b43a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:32:59.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:59.969+0000 7fa112000640 1 -- 172.21.15.6:0/110474427 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa11c0ce710 con 0x7fa11c1f77e0 2024-03-04T09:32:59.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:59.970+0000 7fa112000640 1 --2- 172.21.15.6:0/110474427 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa11c1f77e0 0x7fa11c155d90 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7fa10400f5e0 tx=0x7fa10400faa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:59.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:59.970+0000 7fa113400640 1 -- 172.21.15.6:0/110474427 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa104013810 con 0x7fa11c1f77e0 2024-03-04T09:32:59.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:59.970+0000 7fa113400640 1 -- 172.21.15.6:0/110474427 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa10401a070 con 0x7fa11c1f77e0 2024-03-04T09:32:59.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:59.970+0000 7fa113400640 1 -- 172.21.15.6:0/110474427 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa104013bb0 con 0x7fa11c1f77e0 2024-03-04T09:32:59.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:59.970+0000 7fa121c00640 1 -- 172.21.15.6:0/110474427 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa11c0ce980 con 0x7fa11c1f77e0 2024-03-04T09:32:59.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:59.970+0000 7fa121c00640 1 -- 172.21.15.6:0/110474427 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa11c0cef40 con 0x7fa11c1f77e0 2024-03-04T09:32:59.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:59.971+0000 7fa121c00640 1 -- 172.21.15.6:0/110474427 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa11c04f330 con 0x7fa11c1f77e0 2024-03-04T09:32:59.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:59.972+0000 7fa113400640 1 -- 172.21.15.6:0/110474427 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa104012070 con 0x7fa11c1f77e0 2024-03-04T09:32:59.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:59.974+0000 7fa113400640 1 --2- 172.21.15.6:0/110474427 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa0f806b520 0x7fa0f806d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:32:59.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:59.974+0000 7fa113400640 1 -- 172.21.15.6:0/110474427 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa10408a180 con 0x7fa11c1f77e0 2024-03-04T09:32:59.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:59.975+0000 7fa11aa00640 1 --2- 172.21.15.6:0/110474427 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa0f806b520 0x7fa0f806d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:32:59.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:59.976+0000 7fa11aa00640 1 --2- 172.21.15.6:0/110474427 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa0f806b520 0x7fa0f806d9e0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fa10c001050 tx=0x7fa10c009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:32:59.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:32:59.978+0000 7fa113400640 1 -- 172.21.15.6:0/110474427 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa10401c310 con 0x7fa11c1f77e0 2024-03-04T09:33:00.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:00.229+0000 7fa121c00640 1 -- 172.21.15.6:0/110474427 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fa11c070930 con 0x7fa11c1f77e0 2024-03-04T09:33:00.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:00.725+0000 7fa113400640 1 -- 172.21.15.6:0/110474427 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v92) v1 ==== 122+0+0 (secure 0 0 0) 0x7fa104059070 con 0x7fa11c1f77e0 2024-03-04T09:33:00.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:00.729+0000 7fa121c00640 1 -- 172.21.15.6:0/110474427 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa0f806b520 msgr2=0x7fa0f806d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:00.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:00.729+0000 7fa121c00640 1 --2- 172.21.15.6:0/110474427 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa0f806b520 0x7fa0f806d9e0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fa10c001050 tx=0x7fa10c009040 comp rx=0 tx=0).stop 2024-03-04T09:33:00.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:00.729+0000 7fa121c00640 1 -- 172.21.15.6:0/110474427 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa11c1f77e0 msgr2=0x7fa11c155d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:00.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:00.729+0000 7fa121c00640 1 --2- 172.21.15.6:0/110474427 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa11c1f77e0 0x7fa11c155d90 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7fa10400f5e0 tx=0x7fa10400faa0 comp rx=0 tx=0).stop 2024-03-04T09:33:00.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:00.729+0000 7fa121c00640 1 -- 172.21.15.6:0/110474427 shutdown_connections 2024-03-04T09:33:00.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:00.729+0000 7fa121c00640 1 --2- 172.21.15.6:0/110474427 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa0f806b520 0x7fa0f806d9e0 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:00.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:00.729+0000 7fa121c00640 1 --2- 172.21.15.6:0/110474427 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa11c0cf740 0x7fa11c0cc500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:00.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:00.729+0000 7fa121c00640 1 --2- 172.21.15.6:0/110474427 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa11c1f77e0 0x7fa11c155d90 unknown :-1 s=CLOSED pgs=1492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:00.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:00.729+0000 7fa121c00640 1 --2- 172.21.15.6:0/110474427 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa100004d40 0x7fa11c0b43a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:00.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:00.729+0000 7fa121c00640 1 -- 172.21.15.6:0/110474427 >> 172.21.15.6:0/110474427 conn(0x7fa11c0732d0 msgr2=0x7fa11c15b220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:33:00.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:00.729+0000 7fa121c00640 1 -- 172.21.15.6:0/110474427 shutdown_connections 2024-03-04T09:33:00.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:00.730+0000 7fa121c00640 1 -- 172.21.15.6:0/110474427 wait complete. 2024-03-04T09:33:00.749 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-04T09:33:00.749 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:33:00.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:00.969+0000 7f905cc00640 1 Processor -- start 2024-03-04T09:33:00.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:00.970+0000 7f905cc00640 1 -- start start 2024-03-04T09:33:00.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:00.971+0000 7f905cc00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f905815e000 0x7f905806cb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:00.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:00.971+0000 7f905cc00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f9058161c70 0x7f90581620d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:00.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:00.971+0000 7f905cc00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f9058162610 0x7f90580b56f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:00.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:00.971+0000 7f905cc00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f905806d060 con 0x7f9058162610 2024-03-04T09:33:00.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:00.971+0000 7f905cc00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9058162aa0 con 0x7f9058161c70 2024-03-04T09:33:00.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:00.971+0000 7f905cc00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f905806d8a0 con 0x7f905815e000 2024-03-04T09:33:00.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:00.972+0000 7f9056800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f9058162610 0x7f90580b56f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:00.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:00.972+0000 7f9056800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f9058162610 0x7f90580b56f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42924/0 (socket says 172.21.15.6:42924) 2024-03-04T09:33:00.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:00.972+0000 7f9056800640 1 -- 172.21.15.6:0/2798568141 learned_addr learned my addr 172.21.15.6:0/2798568141 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:33:00.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:00.972+0000 7f9055e00640 1 --2- 172.21.15.6:0/2798568141 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f905815e000 0x7f905806cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:00.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:00.972+0000 7f9056800640 1 -- 172.21.15.6:0/2798568141 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f905815e000 msgr2=0x7f905806cb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:33:00.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:00.972+0000 7f9056800640 1 --2- 172.21.15.6:0/2798568141 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f905815e000 0x7f905806cb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:00.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:00.972+0000 7f9056800640 1 -- 172.21.15.6:0/2798568141 >> v2:172.21.15.146:3300/0 conn(0x7f9058161c70 msgr2=0x7f90581620d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:33:00.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:00.972+0000 7f9055400640 1 --2- 172.21.15.6:0/2798568141 >> v2:172.21.15.146:3300/0 conn(0x7f9058161c70 0x7f90581620d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:00.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:00.972+0000 7f9056800640 1 --2- 172.21.15.6:0/2798568141 >> v2:172.21.15.146:3300/0 conn(0x7f9058161c70 0x7f90581620d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:00.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:00.972+0000 7f9056800640 1 -- 172.21.15.6:0/2798568141 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f905806d2e0 con 0x7f9058162610 2024-03-04T09:33:00.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:00.972+0000 7f9055e00640 1 --2- 172.21.15.6:0/2798568141 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f905815e000 0x7f905806cb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:33:00.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:00.972+0000 7f9055400640 1 --2- 172.21.15.6:0/2798568141 >> v2:172.21.15.146:3300/0 conn(0x7f9058161c70 0x7f90581620d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:33:00.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:00.973+0000 7f9056800640 1 --2- 172.21.15.6:0/2798568141 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9058162610 0x7f90580b56f0 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7f90580721a0 tx=0x7f905002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc45a2b2ae6602d server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:33:00.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:00.974+0000 7f9054a00640 1 -- 172.21.15.6:0/2798568141 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f905002e070 con 0x7f9058162610 2024-03-04T09:33:00.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:00.974+0000 7f9054a00640 1 -- 172.21.15.6:0/2798568141 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9050037070 con 0x7f9058162610 2024-03-04T09:33:00.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:00.974+0000 7f9054a00640 1 -- 172.21.15.6:0/2798568141 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9050031a00 con 0x7f9058162610 2024-03-04T09:33:00.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:00.974+0000 7f905cc00640 1 -- 172.21.15.6:0/2798568141 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9058162610 msgr2=0x7f90580b56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:00.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:00.974+0000 7f905cc00640 1 --2- 172.21.15.6:0/2798568141 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9058162610 0x7f90580b56f0 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7f90580721a0 tx=0x7f905002b880 comp rx=0 tx=0).stop 2024-03-04T09:33:00.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:00.974+0000 7f905cc00640 1 -- 172.21.15.6:0/2798568141 shutdown_connections 2024-03-04T09:33:00.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:00.974+0000 7f905cc00640 1 --2- 172.21.15.6:0/2798568141 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9058162610 0x7f90580b56f0 unknown :-1 s=CLOSED pgs=1493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:00.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:00.974+0000 7f905cc00640 1 --2- 172.21.15.6:0/2798568141 >> v2:172.21.15.146:3300/0 conn(0x7f9058161c70 0x7f90581620d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:00.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:00.974+0000 7f905cc00640 1 --2- 172.21.15.6:0/2798568141 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f905815e000 0x7f905806cb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:00.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:00.974+0000 7f905cc00640 1 -- 172.21.15.6:0/2798568141 >> 172.21.15.6:0/2798568141 conn(0x7f905809aa10 msgr2=0x7f905809ae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:33:00.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:00.974+0000 7f905cc00640 1 -- 172.21.15.6:0/2798568141 shutdown_connections 2024-03-04T09:33:00.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:00.974+0000 7f905cc00640 1 -- 172.21.15.6:0/2798568141 wait complete. 2024-03-04T09:33:00.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:00.977+0000 7f905cc00640 1 Processor -- start 2024-03-04T09:33:00.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:00.977+0000 7f905cc00640 1 -- start start 2024-03-04T09:33:00.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:00.979+0000 7f905cc00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f905815e000 0x7f905815f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:00.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:00.979+0000 7f905cc00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9058161c70 0x7f905815fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:00.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:00.979+0000 7f905cc00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9058162610 0x7f9058148240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:00.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:00.979+0000 7f905cc00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f905815e3e0 con 0x7f9058162610 2024-03-04T09:33:00.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:00.979+0000 7f905cc00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90580aecf0 con 0x7f9058161c70 2024-03-04T09:33:00.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:00.979+0000 7f905cc00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90581601c0 con 0x7f905815e000 2024-03-04T09:33:00.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:00.979+0000 7f9056800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9058162610 0x7f9058148240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:00.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:00.979+0000 7f9056800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9058162610 0x7f9058148240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42938/0 (socket says 172.21.15.6:42938) 2024-03-04T09:33:00.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:00.979+0000 7f9056800640 1 -- 172.21.15.6:0/1284719675 learned_addr learned my addr 172.21.15.6:0/1284719675 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:33:00.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:00.980+0000 7f9055e00640 1 --2- 172.21.15.6:0/1284719675 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f905815e000 0x7f905815f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:00.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:00.980+0000 7f9055400640 1 --2- 172.21.15.6:0/1284719675 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9058161c70 0x7f905815fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:00.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:00.980+0000 7f9056800640 1 -- 172.21.15.6:0/1284719675 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f905815e000 msgr2=0x7f905815f740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:00.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:00.980+0000 7f9056800640 1 --2- 172.21.15.6:0/1284719675 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f905815e000 0x7f905815f740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:00.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:00.980+0000 7f9056800640 1 -- 172.21.15.6:0/1284719675 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9058161c70 msgr2=0x7f905815fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:00.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:00.980+0000 7f9056800640 1 --2- 172.21.15.6:0/1284719675 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9058161c70 0x7f905815fc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:00.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:00.980+0000 7f9056800640 1 -- 172.21.15.6:0/1284719675 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9058160430 con 0x7f9058162610 2024-03-04T09:33:00.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:00.980+0000 7f9056800640 1 --2- 172.21.15.6:0/1284719675 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9058162610 0x7f9058148240 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7f90500090b0 tx=0x7f90500046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:33:00.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:00.981+0000 7f904e000640 1 -- 172.21.15.6:0/1284719675 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9050032880 con 0x7f9058162610 2024-03-04T09:33:00.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:00.981+0000 7f904e000640 1 -- 172.21.15.6:0/1284719675 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9050032a40 con 0x7f9058162610 2024-03-04T09:33:00.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:00.981+0000 7f905cc00640 1 -- 172.21.15.6:0/1284719675 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9058160710 con 0x7f9058162610 2024-03-04T09:33:00.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:00.981+0000 7f904e000640 1 -- 172.21.15.6:0/1284719675 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9050032e30 con 0x7f9058162610 2024-03-04T09:33:00.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:00.981+0000 7f905cc00640 1 -- 172.21.15.6:0/1284719675 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9058160cd0 con 0x7f9058162610 2024-03-04T09:33:00.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:00.982+0000 7f905cc00640 1 -- 172.21.15.6:0/1284719675 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9058161010 con 0x7f9058162610 2024-03-04T09:33:00.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:00.983+0000 7f904e000640 1 -- 172.21.15.6:0/1284719675 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f905004c060 con 0x7f9058162610 2024-03-04T09:33:00.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:00.984+0000 7f904e000640 1 --2- 172.21.15.6:0/1284719675 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f903806b4d0 0x7f903806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:00.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:00.984+0000 7f904e000640 1 -- 172.21.15.6:0/1284719675 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f90500abb20 con 0x7f9058162610 2024-03-04T09:33:00.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:00.984+0000 7f9055e00640 1 --2- 172.21.15.6:0/1284719675 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f903806b4d0 0x7f903806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:00.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:00.986+0000 7f9055e00640 1 --2- 172.21.15.6:0/1284719675 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f903806b4d0 0x7f903806d990 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f9048007130 tx=0x7f90480079c0 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:33:00.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:00.988+0000 7f904e000640 1 -- 172.21.15.6:0/1284719675 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9058161010 con 0x7f9058162610 2024-03-04T09:33:01.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.228+0000 7f905cc00640 1 -- 172.21.15.6:0/1284719675 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f905804f3a0 con 0x7f9058162610 2024-03-04T09:33:01.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.229+0000 7f904e000640 1 -- 172.21.15.6:0/1284719675 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 92 v92) v1 ==== 94+0+4979 (secure 0 0 0) 0x7f9050040070 con 0x7f9058162610 2024-03-04T09:33:01.226 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:33:01.226 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":92,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6098,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/2522244217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":2522244217},{"type":"v1","addr":"172.21.15.6:6835","nonce":2522244217}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":87},{"gid":6101,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6836/3149226637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":3149226637},{"type":"v1","addr":"172.21.15.146:6836","nonce":3149226637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":87},{"gid":6796,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":87}],"filesystems":[{"mdsmap":{"epoch":92,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:33:00.230788+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8214},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":9}]} 2024-03-04T09:33:01.226 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 92 2024-03-04T09:33:01.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.231+0000 7f905cc00640 1 -- 172.21.15.6:0/1284719675 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f903806b4d0 msgr2=0x7f903806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:01.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.231+0000 7f905cc00640 1 --2- 172.21.15.6:0/1284719675 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f903806b4d0 0x7f903806d990 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f9048007130 tx=0x7f90480079c0 comp rx=0 tx=0).stop 2024-03-04T09:33:01.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.231+0000 7f905cc00640 1 -- 172.21.15.6:0/1284719675 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9058162610 msgr2=0x7f9058148240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:01.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.231+0000 7f905cc00640 1 --2- 172.21.15.6:0/1284719675 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9058162610 0x7f9058148240 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7f90500090b0 tx=0x7f90500046c0 comp rx=0 tx=0).stop 2024-03-04T09:33:01.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.232+0000 7f905cc00640 1 -- 172.21.15.6:0/1284719675 shutdown_connections 2024-03-04T09:33:01.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.232+0000 7f905cc00640 1 --2- 172.21.15.6:0/1284719675 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9058162610 0x7f9058148240 unknown :-1 s=CLOSED pgs=1494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:01.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.232+0000 7f905cc00640 1 --2- 172.21.15.6:0/1284719675 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9058161c70 0x7f905815fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:01.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.232+0000 7f905cc00640 1 --2- 172.21.15.6:0/1284719675 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f903806b4d0 0x7f903806d990 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:01.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.232+0000 7f905cc00640 1 --2- 172.21.15.6:0/1284719675 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f905815e000 0x7f905815f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:01.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.232+0000 7f905cc00640 1 -- 172.21.15.6:0/1284719675 >> 172.21.15.6:0/1284719675 conn(0x7f905809aa10 msgr2=0x7f90580b3850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:33:01.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.232+0000 7f905cc00640 1 -- 172.21.15.6:0/1284719675 shutdown_connections 2024-03-04T09:33:01.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.232+0000 7f905cc00640 1 -- 172.21.15.6:0/1284719675 wait complete. 2024-03-04T09:33:01.257 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T09:33:01.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.470+0000 7f053a600640 1 Processor -- start 2024-03-04T09:33:01.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.470+0000 7f053a600640 1 -- start start 2024-03-04T09:33:01.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.471+0000 7f053a600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f053415dec0 0x7f05340ce810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:01.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.471+0000 7f053a600640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f05340cf460 0x7f05340b1330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:01.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.471+0000 7f053a600640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05340d1120 con 0x7f053415e2a0 2024-03-04T09:33:01.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.471+0000 7f053a600640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05340b0ea0 con 0x7f053415dec0 2024-03-04T09:33:01.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.471+0000 7f053a600640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05340d1400 con 0x7f05340cf460 2024-03-04T09:33:01.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.472+0000 7f0539200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f053415dec0 0x7f05340ce810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:01.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.472+0000 7f0533e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f053415e2a0 0x7f05340ced50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:58032/0 (socket says 172.21.15.6:58032) 2024-03-04T09:33:01.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.472+0000 7f0533e00640 1 -- 172.21.15.6:0/2945295519 learned_addr learned my addr 172.21.15.6:0/2945295519 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:33:01.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.472+0000 7f0539c00640 1 --2- 172.21.15.6:0/2945295519 >> v2:172.21.15.146:3300/0 conn(0x7f05340cf460 0x7f05340b1330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:01.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.472+0000 7f0533400640 1 -- 172.21.15.6:0/2945295519 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 426573371 0 0) 0x7f05340d1120 con 0x7f053415e2a0 2024-03-04T09:33:01.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.472+0000 7f0539200640 1 -- 172.21.15.6:0/2945295519 >> v2:172.21.15.146:3300/0 conn(0x7f05340cf460 msgr2=0x7f05340b1330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:33:01.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.472+0000 7f0539200640 1 --2- 172.21.15.6:0/2945295519 >> v2:172.21.15.146:3300/0 conn(0x7f05340cf460 0x7f05340b1330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:01.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.472+0000 7f0539200640 1 -- 172.21.15.6:0/2945295519 >> v1:172.21.15.6:6789/0 conn(0x7f053415e2a0 legacy=0x7f05340ced50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:01.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.472+0000 7f0539200640 1 -- 172.21.15.6:0/2945295519 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05340b0b30 con 0x7f053415dec0 2024-03-04T09:33:01.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.474+0000 7f0539200640 1 --2- 172.21.15.6:0/2945295519 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f053415dec0 0x7f05340ce810 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7f053408c570 tx=0x7f052802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40aae262a33f20e7 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:33:01.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.475+0000 7f0533400640 1 -- 172.21.15.6:0/2945295519 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0528035020 con 0x7f053415dec0 2024-03-04T09:33:01.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.475+0000 7f0533400640 1 -- 172.21.15.6:0/2945295519 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0528031830 con 0x7f053415dec0 2024-03-04T09:33:01.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.475+0000 7f0533400640 1 -- 172.21.15.6:0/2945295519 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0528031c00 con 0x7f053415dec0 2024-03-04T09:33:01.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.475+0000 7f053a600640 1 -- 172.21.15.6:0/2945295519 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f053415dec0 msgr2=0x7f05340ce810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:01.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.475+0000 7f053a600640 1 --2- 172.21.15.6:0/2945295519 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f053415dec0 0x7f05340ce810 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7f053408c570 tx=0x7f052802b8d0 comp rx=0 tx=0).stop 2024-03-04T09:33:01.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.475+0000 7f053a600640 1 -- 172.21.15.6:0/2945295519 shutdown_connections 2024-03-04T09:33:01.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.475+0000 7f053a600640 1 --2- 172.21.15.6:0/2945295519 >> v2:172.21.15.146:3300/0 conn(0x7f05340cf460 0x7f05340b1330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:01.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.475+0000 7f053a600640 1 --2- 172.21.15.6:0/2945295519 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f053415dec0 0x7f05340ce810 unknown :-1 s=CLOSED pgs=1496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:01.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.475+0000 7f053a600640 1 -- 172.21.15.6:0/2945295519 >> 172.21.15.6:0/2945295519 conn(0x7f05340d3d40 msgr2=0x7f05340d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:33:01.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.476+0000 7f053a600640 1 -- 172.21.15.6:0/2945295519 shutdown_connections 2024-03-04T09:33:01.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.476+0000 7f053a600640 1 -- 172.21.15.6:0/2945295519 wait complete. 2024-03-04T09:33:01.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.477+0000 7f053a600640 1 Processor -- start 2024-03-04T09:33:01.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.478+0000 7f053a600640 1 -- start start 2024-03-04T09:33:01.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.479+0000 7f053a600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f05340cf460 0x7f05340ca310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:01.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.479+0000 7f053a600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f053415dec0 0x7f05340c8280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:01.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.479+0000 7f053a600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f053415e2a0 0x7f05340b9350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:01.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.479+0000 7f053a600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05340d0ec0 con 0x7f053415e2a0 2024-03-04T09:33:01.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.479+0000 7f053a600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05340d0d40 con 0x7f05340cf460 2024-03-04T09:33:01.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.479+0000 7f053a600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05340b1870 con 0x7f053415dec0 2024-03-04T09:33:01.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.479+0000 7f0539c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f053415e2a0 0x7f05340b9350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:01.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.479+0000 7f0539c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f053415e2a0 0x7f05340b9350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42954/0 (socket says 172.21.15.6:42954) 2024-03-04T09:33:01.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.479+0000 7f0539c00640 1 -- 172.21.15.6:0/2704761615 learned_addr learned my addr 172.21.15.6:0/2704761615 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:33:01.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.479+0000 7f0533e00640 1 --2- 172.21.15.6:0/2704761615 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f053415dec0 0x7f05340c8280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:01.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.479+0000 7f0539200640 1 --2- 172.21.15.6:0/2704761615 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f05340cf460 0x7f05340ca310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:01.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.480+0000 7f0539c00640 1 -- 172.21.15.6:0/2704761615 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f053415dec0 msgr2=0x7f05340c8280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:01.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.480+0000 7f0539c00640 1 --2- 172.21.15.6:0/2704761615 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f053415dec0 0x7f05340c8280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:01.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.480+0000 7f0539c00640 1 -- 172.21.15.6:0/2704761615 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f05340cf460 msgr2=0x7f05340ca310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:01.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.480+0000 7f0539c00640 1 --2- 172.21.15.6:0/2704761615 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f05340cf460 0x7f05340ca310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:01.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.480+0000 7f0539c00640 1 -- 172.21.15.6:0/2704761615 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f053415e840 con 0x7f053415e2a0 2024-03-04T09:33:01.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.480+0000 7f0533e00640 1 --2- 172.21.15.6:0/2704761615 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f053415dec0 0x7f05340c8280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:33:01.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.480+0000 7f0539c00640 1 --2- 172.21.15.6:0/2704761615 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f053415e2a0 0x7f05340b9350 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7f052c007e30 tx=0x7f052c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:33:01.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.480+0000 7f0530c00640 1 -- 172.21.15.6:0/2704761615 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f052c014620 con 0x7f053415e2a0 2024-03-04T09:33:01.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.480+0000 7f0530c00640 1 -- 172.21.15.6:0/2704761615 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f052c019070 con 0x7f053415e2a0 2024-03-04T09:33:01.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.480+0000 7f0530c00640 1 -- 172.21.15.6:0/2704761615 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f052c014a00 con 0x7f053415e2a0 2024-03-04T09:33:01.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.480+0000 7f053a600640 1 -- 172.21.15.6:0/2704761615 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05340c9460 con 0x7f053415e2a0 2024-03-04T09:33:01.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.480+0000 7f053a600640 1 -- 172.21.15.6:0/2704761615 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05340c9a20 con 0x7f053415e2a0 2024-03-04T09:33:01.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.482+0000 7f053a600640 1 -- 172.21.15.6:0/2704761615 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f053415ea10 con 0x7f053415e2a0 2024-03-04T09:33:01.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.482+0000 7f0530c00640 1 -- 172.21.15.6:0/2704761615 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f052c014ba0 con 0x7f053415e2a0 2024-03-04T09:33:01.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.483+0000 7f0530c00640 1 --2- 172.21.15.6:0/2704761615 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f051406b4d0 0x7f051406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:01.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.483+0000 7f0530c00640 1 -- 172.21.15.6:0/2704761615 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f052c08aaf0 con 0x7f053415e2a0 2024-03-04T09:33:01.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.484+0000 7f0539200640 1 --2- 172.21.15.6:0/2704761615 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f051406b4d0 0x7f051406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:01.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.485+0000 7f0539200640 1 --2- 172.21.15.6:0/2704761615 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f051406b4d0 0x7f051406d990 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f0528036000 tx=0x7f0528008030 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:33:01.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.488+0000 7f0530c00640 1 -- 172.21.15.6:0/2704761615 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f052c059960 con 0x7f053415e2a0 2024-03-04T09:33:01.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.684+0000 7f053a600640 1 -- 172.21.15.6:0/2704761615 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f053404f330 con 0x7f053415e2a0 2024-03-04T09:33:01.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.687+0000 7f0530c00640 1 -- 172.21.15.6:0/2704761615 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v127) v1 ==== 74+0+16631 (secure 0 0 0) 0x7f052c059260 con 0x7f053415e2a0 2024-03-04T09:33:01.684 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:33:01.684 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":127,"fsid":"edce0578-5ae4-4402-b45a-552bdef84d7c","created":"2024-03-04T09:24:54.523780+0000","modified":"2024-03-04T09:32:58.691545+0000","last_up_change":"2024-03-04T09:25:07.089049+0000","last_in_change":"2024-03-04T09:24:57.982077+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T09:25:07.935198+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-03-04T09:32:54.852054+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"126","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-03-04T09:32:56.177848+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"127","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a96574f-81f1-4bc6-8463-757520b84a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6825","nonce":1364449401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6827","nonce":1364449401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6831","nonce":1364449401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6829","nonce":1364449401}]},"public_addr":"172.21.15.6:6825/1364449401","cluster_addr":"172.21.15.6:6827/1364449401","heartbeat_back_addr":"172.21.15.6:6831/1364449401","heartbeat_front_addr":"172.21.15.6:6829/1364449401","state":["exists","up"]},{"osd":1,"uuid":"7d79b9c1-6b87-476a-bc36-7db5ee64cd90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6809","nonce":2494103078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6811","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6813","nonce":2494103078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6821","nonce":2494103078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6817","nonce":2494103078}]},"public_addr":"172.21.15.6:6809/2494103078","cluster_addr":"172.21.15.6:6813/2494103078","heartbeat_back_addr":"172.21.15.6:6821/2494103078","heartbeat_front_addr":"172.21.15.6:6817/2494103078","state":["exists","up"]},{"osd":2,"uuid":"492331b9-4ee9-40b6-bacd-85d44cd6c45a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6812","nonce":1500066681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6816","nonce":1500066681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6824","nonce":1500066681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6820","nonce":1500066681}]},"public_addr":"172.21.15.6:6812/1500066681","cluster_addr":"172.21.15.6:6816/1500066681","heartbeat_back_addr":"172.21.15.6:6824/1500066681","heartbeat_front_addr":"172.21.15.6:6820/1500066681","state":["exists","up"]},{"osd":3,"uuid":"21bb2efe-9ba7-4bdc-bc31-180cd1102e21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6801","nonce":1558806123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6803","nonce":1558806123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6807","nonce":1558806123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6805","nonce":1558806123}]},"public_addr":"172.21.15.6:6801/1558806123","cluster_addr":"172.21.15.6:6803/1558806123","heartbeat_back_addr":"172.21.15.6:6807/1558806123","heartbeat_front_addr":"172.21.15.6:6805/1558806123","state":["exists","up"]},{"osd":4,"uuid":"6e2c4b71-19d9-4737-83d9-558705d8644a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6801","nonce":1526135780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6803","nonce":1526135780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6807","nonce":1526135780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6805","nonce":1526135780}]},"public_addr":"172.21.15.146:6801/1526135780","cluster_addr":"172.21.15.146:6803/1526135780","heartbeat_back_addr":"172.21.15.146:6807/1526135780","heartbeat_front_addr":"172.21.15.146:6805/1526135780","state":["exists","up"]},{"osd":5,"uuid":"fb69c525-f1be-4b7d-ab32-6c29b0d6fddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6809","nonce":460262507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6811","nonce":460262507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6815","nonce":460262507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6813","nonce":460262507}]},"public_addr":"172.21.15.146:6809/460262507","cluster_addr":"172.21.15.146:6811/460262507","heartbeat_back_addr":"172.21.15.146:6815/460262507","heartbeat_front_addr":"172.21.15.146:6813/460262507","state":["exists","up"]},{"osd":6,"uuid":"c32e7fed-370d-4401-8ce0-6a4335b0d168","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6823","nonce":1392897493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6827","nonce":1392897493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6831","nonce":1392897493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6829","nonce":1392897493}]},"public_addr":"172.21.15.146:6823/1392897493","cluster_addr":"172.21.15.146:6827/1392897493","heartbeat_back_addr":"172.21.15.146:6831/1392897493","heartbeat_front_addr":"172.21.15.146:6829/1392897493","state":["exists","up"]},{"osd":7,"uuid":"75a79b44-fa5f-426d-a7d3-71f21f78835c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6817","nonce":3268323156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6819","nonce":3268323156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6825","nonce":3268323156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6822","nonce":3268323156}]},"public_addr":"172.21.15.146:6817/3268323156","cluster_addr":"172.21.15.146:6819/3268323156","heartbeat_back_addr":"172.21.15.146:6825/3268323156","heartbeat_front_addr":"172.21.15.146:6822/3268323156","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.310437+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.295850+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.294674+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.526855+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.300023+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.595160+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.346046+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.558741+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T09:33:01.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.689+0000 7f053a600640 1 -- 172.21.15.6:0/2704761615 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f051406b4d0 msgr2=0x7f051406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:01.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.689+0000 7f053a600640 1 --2- 172.21.15.6:0/2704761615 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f051406b4d0 0x7f051406d990 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f0528036000 tx=0x7f0528008030 comp rx=0 tx=0).stop 2024-03-04T09:33:01.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.689+0000 7f053a600640 1 -- 172.21.15.6:0/2704761615 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f053415e2a0 msgr2=0x7f05340b9350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:01.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.689+0000 7f053a600640 1 --2- 172.21.15.6:0/2704761615 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f053415e2a0 0x7f05340b9350 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7f052c007e30 tx=0x7f052c00e4b0 comp rx=0 tx=0).stop 2024-03-04T09:33:01.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.689+0000 7f053a600640 1 -- 172.21.15.6:0/2704761615 shutdown_connections 2024-03-04T09:33:01.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.689+0000 7f053a600640 1 --2- 172.21.15.6:0/2704761615 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f053415e2a0 0x7f05340b9350 unknown :-1 s=CLOSED pgs=1497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:01.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.689+0000 7f053a600640 1 --2- 172.21.15.6:0/2704761615 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f051406b4d0 0x7f051406d990 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:01.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.689+0000 7f053a600640 1 --2- 172.21.15.6:0/2704761615 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f053415dec0 0x7f05340c8280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:01.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.689+0000 7f053a600640 1 --2- 172.21.15.6:0/2704761615 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f05340cf460 0x7f05340ca310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:01.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.689+0000 7f053a600640 1 -- 172.21.15.6:0/2704761615 >> 172.21.15.6:0/2704761615 conn(0x7f05340d3d40 msgr2=0x7f05340d18f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:33:01.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.690+0000 7f053a600640 1 -- 172.21.15.6:0/2704761615 shutdown_connections 2024-03-04T09:33:01.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.690+0000 7f053a600640 1 -- 172.21.15.6:0/2704761615 wait complete. 2024-03-04T09:33:01.706 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-04T09:33:01.706 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T09:33:01.871 INFO:tasks.ceph.mds.a.smithi006.stderr:2024-03-04T09:33:01.874+0000 7f1df8800640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-04T09:33:01.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.905+0000 7f1c08a00640 1 Processor -- start 2024-03-04T09:33:01.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.905+0000 7f1c08a00640 1 -- start start 2024-03-04T09:33:01.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.907+0000 7f1c08a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1c0415dec0 0x7f1c040ce810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:01.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.907+0000 7f1c08a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f1c040cf460 0x7f1c040b1330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:01.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.907+0000 7f1c08a00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c040d1120 con 0x7f1c0415e2a0 2024-03-04T09:33:01.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.907+0000 7f1c08a00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c040b0ea0 con 0x7f1c040cf460 2024-03-04T09:33:01.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.907+0000 7f1c08a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c040d1400 con 0x7f1c0415dec0 2024-03-04T09:33:01.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.908+0000 7f1c03200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f1c040cf460 0x7f1c040b1330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:01.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.908+0000 7f1c03200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f1c040cf460 0x7f1c040b1330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42958/0 (socket says 172.21.15.6:42958) 2024-03-04T09:33:01.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.908+0000 7f1c03200640 1 -- 172.21.15.6:0/15195283 learned_addr learned my addr 172.21.15.6:0/15195283 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:33:01.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.908+0000 7f1c02800640 1 --2- 172.21.15.6:0/15195283 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1c0415dec0 0x7f1c040ce810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:01.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.908+0000 7f1c03200640 1 -- 172.21.15.6:0/15195283 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1c0415dec0 msgr2=0x7f1c040ce810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:33:01.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.908+0000 7f1c03200640 1 --2- 172.21.15.6:0/15195283 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1c0415dec0 0x7f1c040ce810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:01.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.908+0000 7f1c03200640 1 -- 172.21.15.6:0/15195283 >> v1:172.21.15.146:6789/0 conn(0x7f1c0415e2a0 legacy=0x7f1c040ced50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:33:01.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.908+0000 7f1c03200640 1 -- 172.21.15.6:0/15195283 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c040b0b30 con 0x7f1c040cf460 2024-03-04T09:33:01.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.908+0000 7f1c02800640 1 --2- 172.21.15.6:0/15195283 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1c0415dec0 0x7f1c040ce810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:33:01.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.909+0000 7f1c03200640 1 --2- 172.21.15.6:0/15195283 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1c040cf460 0x7f1c040b1330 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7f1bfc009e50 tx=0x7f1bfc02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6905e2a85edafd4 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:33:01.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.909+0000 7f1c01400640 1 -- 172.21.15.6:0/15195283 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bfc02ca80 con 0x7f1c040cf460 2024-03-04T09:33:01.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.909+0000 7f1c01400640 1 -- 172.21.15.6:0/15195283 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bfc02cc20 con 0x7f1c040cf460 2024-03-04T09:33:01.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.909+0000 7f1c01400640 1 -- 172.21.15.6:0/15195283 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bfc0343f0 con 0x7f1c040cf460 2024-03-04T09:33:01.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.910+0000 7f1c08a00640 1 -- 172.21.15.6:0/15195283 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1c040cf460 msgr2=0x7f1c040b1330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:01.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.910+0000 7f1c08a00640 1 --2- 172.21.15.6:0/15195283 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1c040cf460 0x7f1c040b1330 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7f1bfc009e50 tx=0x7f1bfc02bee0 comp rx=0 tx=0).stop 2024-03-04T09:33:01.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.910+0000 7f1c08a00640 1 -- 172.21.15.6:0/15195283 shutdown_connections 2024-03-04T09:33:01.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.910+0000 7f1c08a00640 1 --2- 172.21.15.6:0/15195283 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1c040cf460 0x7f1c040b1330 unknown :-1 s=CLOSED pgs=1498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:01.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.910+0000 7f1c08a00640 1 --2- 172.21.15.6:0/15195283 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1c0415dec0 0x7f1c040ce810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:01.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.910+0000 7f1c08a00640 1 -- 172.21.15.6:0/15195283 >> 172.21.15.6:0/15195283 conn(0x7f1c040d3d40 msgr2=0x7f1c040d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:33:01.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.910+0000 7f1c08a00640 1 -- 172.21.15.6:0/15195283 shutdown_connections 2024-03-04T09:33:01.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.910+0000 7f1c08a00640 1 -- 172.21.15.6:0/15195283 wait complete. 2024-03-04T09:33:01.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.912+0000 7f1c08a00640 1 Processor -- start 2024-03-04T09:33:01.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.912+0000 7f1c08a00640 1 -- start start 2024-03-04T09:33:01.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.913+0000 7f1c08a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c040cf460 0x7f1c0415a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:01.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.913+0000 7f1c08a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1c0415dec0 0x7f1c040b7850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:01.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.913+0000 7f1c08a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1c0415e2a0 0x7f1c040c9ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:01.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.913+0000 7f1c08a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c040d0ec0 con 0x7f1c0415e2a0 2024-03-04T09:33:01.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.913+0000 7f1c08a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c040d0d40 con 0x7f1c040cf460 2024-03-04T09:33:01.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.913+0000 7f1c08a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c040b1870 con 0x7f1c0415dec0 2024-03-04T09:33:01.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.914+0000 7f1c03200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1c0415e2a0 0x7f1c040c9ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:01.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.914+0000 7f1c03200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1c0415e2a0 0x7f1c040c9ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42966/0 (socket says 172.21.15.6:42966) 2024-03-04T09:33:01.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.914+0000 7f1c03200640 1 -- 172.21.15.6:0/4046268350 learned_addr learned my addr 172.21.15.6:0/4046268350 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:33:01.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.914+0000 7f1c01e00640 1 --2- 172.21.15.6:0/4046268350 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1c0415dec0 0x7f1c040b7850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:01.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.914+0000 7f1c02800640 1 --2- 172.21.15.6:0/4046268350 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c040cf460 0x7f1c0415a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:01.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.914+0000 7f1c03200640 1 -- 172.21.15.6:0/4046268350 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1c0415dec0 msgr2=0x7f1c040b7850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:01.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.914+0000 7f1c03200640 1 --2- 172.21.15.6:0/4046268350 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1c0415dec0 0x7f1c040b7850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:01.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.914+0000 7f1c03200640 1 -- 172.21.15.6:0/4046268350 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c040cf460 msgr2=0x7f1c0415a460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:01.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.914+0000 7f1c03200640 1 --2- 172.21.15.6:0/4046268350 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c040cf460 0x7f1c0415a460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:01.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.914+0000 7f1c03200640 1 -- 172.21.15.6:0/4046268350 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c040b7d90 con 0x7f1c0415e2a0 2024-03-04T09:33:01.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.915+0000 7f1c03200640 1 --2- 172.21.15.6:0/4046268350 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1c0415e2a0 0x7f1c040c9ec0 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 crypto rx=0x7f1bfc036040 tx=0x7f1bfc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:33:01.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.915+0000 7f1bfaa00640 1 -- 172.21.15.6:0/4046268350 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bfc043020 con 0x7f1c0415e2a0 2024-03-04T09:33:01.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.915+0000 7f1bfaa00640 1 -- 172.21.15.6:0/4046268350 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bfc03e450 con 0x7f1c0415e2a0 2024-03-04T09:33:01.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.915+0000 7f1bfaa00640 1 -- 172.21.15.6:0/4046268350 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bfc009510 con 0x7f1c0415e2a0 2024-03-04T09:33:01.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.915+0000 7f1c08a00640 1 -- 172.21.15.6:0/4046268350 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c040c9100 con 0x7f1c0415e2a0 2024-03-04T09:33:01.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.915+0000 7f1c08a00640 1 -- 172.21.15.6:0/4046268350 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c040c96c0 con 0x7f1c0415e2a0 2024-03-04T09:33:01.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.916+0000 7f1c08a00640 1 -- 172.21.15.6:0/4046268350 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c0404f330 con 0x7f1c0415e2a0 2024-03-04T09:33:01.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.918+0000 7f1bfaa00640 1 -- 172.21.15.6:0/4046268350 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1bfc046020 con 0x7f1c0415e2a0 2024-03-04T09:33:01.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.919+0000 7f1bfaa00640 1 --2- 172.21.15.6:0/4046268350 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f1be406b4d0 0x7f1be406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:01.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.919+0000 7f1bfaa00640 1 -- 172.21.15.6:0/4046268350 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1bfc0ac9e0 con 0x7f1c0415e2a0 2024-03-04T09:33:01.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.920+0000 7f1c02800640 1 --2- 172.21.15.6:0/4046268350 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f1be406b4d0 0x7f1be406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:01.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.921+0000 7f1c02800640 1 --2- 172.21.15.6:0/4046268350 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f1be406b4d0 0x7f1be406d990 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f1bf400a990 tx=0x7f1bf4008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:33:01.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:01.924+0000 7f1bfaa00640 1 -- 172.21.15.6:0/4046268350 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1bfc04d030 con 0x7f1c0415e2a0 2024-03-04T09:33:02.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.123+0000 7f1c08a00640 1 -- 172.21.15.6:0/4046268350 --> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1c040b1330 con 0x7f1be406b4d0 2024-03-04T09:33:02.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.143+0000 7f1bfaa00640 1 -- 172.21.15.6:0/4046268350 <== mgr.4113 v2:172.21.15.146:6832/1788613334 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252410 (secure 0 0 0) 0x7f1c040b1330 con 0x7f1be406b4d0 2024-03-04T09:33:02.142 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:33:02.146 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-03-04T09:33:02.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.151+0000 7f1c08a00640 1 -- 172.21.15.6:0/4046268350 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f1be406b4d0 msgr2=0x7f1be406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:02.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.151+0000 7f1c08a00640 1 --2- 172.21.15.6:0/4046268350 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f1be406b4d0 0x7f1be406d990 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f1bf400a990 tx=0x7f1bf4008040 comp rx=0 tx=0).stop 2024-03-04T09:33:02.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.151+0000 7f1c08a00640 1 -- 172.21.15.6:0/4046268350 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1c0415e2a0 msgr2=0x7f1c040c9ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:02.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.151+0000 7f1c08a00640 1 --2- 172.21.15.6:0/4046268350 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1c0415e2a0 0x7f1c040c9ec0 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 crypto rx=0x7f1bfc036040 tx=0x7f1bfc004a40 comp rx=0 tx=0).stop 2024-03-04T09:33:02.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.152+0000 7f1c08a00640 1 -- 172.21.15.6:0/4046268350 shutdown_connections 2024-03-04T09:33:02.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.152+0000 7f1c08a00640 1 --2- 172.21.15.6:0/4046268350 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1c0415e2a0 0x7f1c040c9ec0 unknown :-1 s=CLOSED pgs=1499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:02.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.152+0000 7f1c08a00640 1 --2- 172.21.15.6:0/4046268350 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f1be406b4d0 0x7f1be406d990 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:02.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.152+0000 7f1c08a00640 1 --2- 172.21.15.6:0/4046268350 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1c0415dec0 0x7f1c040b7850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:02.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.152+0000 7f1c08a00640 1 --2- 172.21.15.6:0/4046268350 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c040cf460 0x7f1c0415a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:02.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.152+0000 7f1c08a00640 1 -- 172.21.15.6:0/4046268350 >> 172.21.15.6:0/4046268350 conn(0x7f1c040d3d40 msgr2=0x7f1c0406c530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:33:02.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.152+0000 7f1c08a00640 1 -- 172.21.15.6:0/4046268350 shutdown_connections 2024-03-04T09:33:02.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.152+0000 7f1c08a00640 1 -- 172.21.15.6:0/4046268350 wait complete. 2024-03-04T09:33:02.167 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":357,"stamp":"2024-03-04T09:33:02.027680+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":33,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":296156,"kb_used_data":13400,"kb_used_omap":12,"kb_used_meta":282611,"kb_avail":754678564,"statfs":{"total":773094113280,"available":772790849536,"internally_reserved":0,"allocated":13721600,"data_stored":6628827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":289394224},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002258"},"pg_stats":[{"pgid":"19.3f","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.721729+0000","last_change":"2024-03-04T09:32:57.715295+0000","last_active":"2024-03-04T09:32:57.721729+0000","last_peered":"2024-03-04T09:32:57.721729+0000","last_clean":"2024-03-04T09:32:57.721729+0000","last_became_active":"2024-03-04T09:32:57.715032+0000","last_became_peered":"2024-03-04T09:32:57.715032+0000","last_unstale":"2024-03-04T09:32:57.721729+0000","last_undegraded":"2024-03-04T09:32:57.721729+0000","last_fullsized":"2024-03-04T09:32:57.721729+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:32:39.387015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3e","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.715179+0000","last_change":"2024-03-04T09:32:57.711681+0000","last_active":"2024-03-04T09:32:57.715179+0000","last_peered":"2024-03-04T09:32:57.715179+0000","last_clean":"2024-03-04T09:32:57.715179+0000","last_became_active":"2024-03-04T09:32:57.711140+0000","last_became_peered":"2024-03-04T09:32:57.711140+0000","last_unstale":"2024-03-04T09:32:57.715179+0000","last_undegraded":"2024-03-04T09:32:57.715179+0000","last_fullsized":"2024-03-04T09:32:57.715179+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:17:22.364556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3d","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.706232+0000","last_change":"2024-03-04T09:32:57.700626+0000","last_active":"2024-03-04T09:32:57.706232+0000","last_peered":"2024-03-04T09:32:57.706232+0000","last_clean":"2024-03-04T09:32:57.706232+0000","last_became_active":"2024-03-04T09:32:57.699525+0000","last_became_peered":"2024-03-04T09:32:57.699525+0000","last_unstale":"2024-03-04T09:32:57.706232+0000","last_undegraded":"2024-03-04T09:32:57.706232+0000","last_fullsized":"2024-03-04T09:32:57.706232+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:34:58.565938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.3c","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.721446+0000","last_change":"2024-03-04T09:32:57.713538+0000","last_active":"2024-03-04T09:32:57.721446+0000","last_peered":"2024-03-04T09:32:57.721446+0000","last_clean":"2024-03-04T09:32:57.721446+0000","last_became_active":"2024-03-04T09:32:57.713273+0000","last_became_peered":"2024-03-04T09:32:57.713273+0000","last_unstale":"2024-03-04T09:32:57.721446+0000","last_undegraded":"2024-03-04T09:32:57.721446+0000","last_fullsized":"2024-03-04T09:32:57.721446+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:08:36.129294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3b","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.711514+0000","last_change":"2024-03-04T09:32:57.706038+0000","last_active":"2024-03-04T09:32:57.711514+0000","last_peered":"2024-03-04T09:32:57.711514+0000","last_clean":"2024-03-04T09:32:57.711514+0000","last_became_active":"2024-03-04T09:32:57.705742+0000","last_became_peered":"2024-03-04T09:32:57.705742+0000","last_unstale":"2024-03-04T09:32:57.711514+0000","last_undegraded":"2024-03-04T09:32:57.711514+0000","last_fullsized":"2024-03-04T09:32:57.711514+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:49:58.132664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.3a","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.722955+0000","last_change":"2024-03-04T09:32:57.719201+0000","last_active":"2024-03-04T09:32:57.722955+0000","last_peered":"2024-03-04T09:32:57.722955+0000","last_clean":"2024-03-04T09:32:57.722955+0000","last_became_active":"2024-03-04T09:32:57.718866+0000","last_became_peered":"2024-03-04T09:32:57.718866+0000","last_unstale":"2024-03-04T09:32:57.722955+0000","last_undegraded":"2024-03-04T09:32:57.722955+0000","last_fullsized":"2024-03-04T09:32:57.722955+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:56:31.981113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.39","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.718264+0000","last_change":"2024-03-04T09:32:57.715088+0000","last_active":"2024-03-04T09:32:57.718264+0000","last_peered":"2024-03-04T09:32:57.718264+0000","last_clean":"2024-03-04T09:32:57.718264+0000","last_became_active":"2024-03-04T09:32:57.712371+0000","last_became_peered":"2024-03-04T09:32:57.712371+0000","last_unstale":"2024-03-04T09:32:57.718264+0000","last_undegraded":"2024-03-04T09:32:57.718264+0000","last_fullsized":"2024-03-04T09:32:57.718264+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:34:31.386306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.38","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.714668+0000","last_change":"2024-03-04T09:32:57.703017+0000","last_active":"2024-03-04T09:32:57.714668+0000","last_peered":"2024-03-04T09:32:57.714668+0000","last_clean":"2024-03-04T09:32:57.714668+0000","last_became_active":"2024-03-04T09:32:57.702395+0000","last_became_peered":"2024-03-04T09:32:57.702395+0000","last_unstale":"2024-03-04T09:32:57.714668+0000","last_undegraded":"2024-03-04T09:32:57.714668+0000","last_fullsized":"2024-03-04T09:32:57.714668+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:37:21.506217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.37","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.713108+0000","last_change":"2024-03-04T09:32:57.703315+0000","last_active":"2024-03-04T09:32:57.713108+0000","last_peered":"2024-03-04T09:32:57.713108+0000","last_clean":"2024-03-04T09:32:57.713108+0000","last_became_active":"2024-03-04T09:32:57.703001+0000","last_became_peered":"2024-03-04T09:32:57.703001+0000","last_unstale":"2024-03-04T09:32:57.713108+0000","last_undegraded":"2024-03-04T09:32:57.713108+0000","last_fullsized":"2024-03-04T09:32:57.713108+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:30:14.957488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.36","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.715157+0000","last_change":"2024-03-04T09:32:57.711439+0000","last_active":"2024-03-04T09:32:57.715157+0000","last_peered":"2024-03-04T09:32:57.715157+0000","last_clean":"2024-03-04T09:32:57.715157+0000","last_became_active":"2024-03-04T09:32:57.710733+0000","last_became_peered":"2024-03-04T09:32:57.710733+0000","last_unstale":"2024-03-04T09:32:57.715157+0000","last_undegraded":"2024-03-04T09:32:57.715157+0000","last_fullsized":"2024-03-04T09:32:57.715157+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:01:47.019520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.35","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.712514+0000","last_change":"2024-03-04T09:32:57.705947+0000","last_active":"2024-03-04T09:32:57.712514+0000","last_peered":"2024-03-04T09:32:57.712514+0000","last_clean":"2024-03-04T09:32:57.712514+0000","last_became_active":"2024-03-04T09:32:57.705648+0000","last_became_peered":"2024-03-04T09:32:57.705648+0000","last_unstale":"2024-03-04T09:32:57.712514+0000","last_undegraded":"2024-03-04T09:32:57.712514+0000","last_fullsized":"2024-03-04T09:32:57.712514+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:52:30.281316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.34","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.712136+0000","last_change":"2024-03-04T09:32:57.705170+0000","last_active":"2024-03-04T09:32:57.712136+0000","last_peered":"2024-03-04T09:32:57.712136+0000","last_clean":"2024-03-04T09:32:57.712136+0000","last_became_active":"2024-03-04T09:32:57.703175+0000","last_became_peered":"2024-03-04T09:32:57.703175+0000","last_unstale":"2024-03-04T09:32:57.712136+0000","last_undegraded":"2024-03-04T09:32:57.712136+0000","last_fullsized":"2024-03-04T09:32:57.712136+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:35:24.731959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.33","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.724986+0000","last_change":"2024-03-04T09:32:57.718409+0000","last_active":"2024-03-04T09:32:57.724986+0000","last_peered":"2024-03-04T09:32:57.724986+0000","last_clean":"2024-03-04T09:32:57.724986+0000","last_became_active":"2024-03-04T09:32:57.717404+0000","last_became_peered":"2024-03-04T09:32:57.717404+0000","last_unstale":"2024-03-04T09:32:57.724986+0000","last_undegraded":"2024-03-04T09:32:57.724986+0000","last_fullsized":"2024-03-04T09:32:57.724986+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:32:42.841675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.32","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.716694+0000","last_change":"2024-03-04T09:32:57.714535+0000","last_active":"2024-03-04T09:32:57.716694+0000","last_peered":"2024-03-04T09:32:57.716694+0000","last_clean":"2024-03-04T09:32:57.716694+0000","last_became_active":"2024-03-04T09:32:57.714251+0000","last_became_peered":"2024-03-04T09:32:57.714251+0000","last_unstale":"2024-03-04T09:32:57.716694+0000","last_undegraded":"2024-03-04T09:32:57.716694+0000","last_fullsized":"2024-03-04T09:32:57.716694+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:11:04.209303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.31","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.715040+0000","last_change":"2024-03-04T09:32:57.703832+0000","last_active":"2024-03-04T09:32:57.715040+0000","last_peered":"2024-03-04T09:32:57.715040+0000","last_clean":"2024-03-04T09:32:57.715040+0000","last_became_active":"2024-03-04T09:32:57.703555+0000","last_became_peered":"2024-03-04T09:32:57.703555+0000","last_unstale":"2024-03-04T09:32:57.715040+0000","last_undegraded":"2024-03-04T09:32:57.715040+0000","last_fullsized":"2024-03-04T09:32:57.715040+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:09:13.281054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.30","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.722113+0000","last_change":"2024-03-04T09:32:57.719691+0000","last_active":"2024-03-04T09:32:57.722113+0000","last_peered":"2024-03-04T09:32:57.722113+0000","last_clean":"2024-03-04T09:32:57.722113+0000","last_became_active":"2024-03-04T09:32:57.719398+0000","last_became_peered":"2024-03-04T09:32:57.719398+0000","last_unstale":"2024-03-04T09:32:57.722113+0000","last_undegraded":"2024-03-04T09:32:57.722113+0000","last_fullsized":"2024-03-04T09:32:57.722113+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:01:51.410127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2f","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.716157+0000","last_change":"2024-03-04T09:32:57.709718+0000","last_active":"2024-03-04T09:32:57.716157+0000","last_peered":"2024-03-04T09:32:57.716157+0000","last_clean":"2024-03-04T09:32:57.716157+0000","last_became_active":"2024-03-04T09:32:57.709375+0000","last_became_peered":"2024-03-04T09:32:57.709375+0000","last_unstale":"2024-03-04T09:32:57.716157+0000","last_undegraded":"2024-03-04T09:32:57.716157+0000","last_fullsized":"2024-03-04T09:32:57.716157+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:18:50.341054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.2e","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.713699+0000","last_change":"2024-03-04T09:32:57.710074+0000","last_active":"2024-03-04T09:32:57.713699+0000","last_peered":"2024-03-04T09:32:57.713699+0000","last_clean":"2024-03-04T09:32:57.713699+0000","last_became_active":"2024-03-04T09:32:57.709743+0000","last_became_peered":"2024-03-04T09:32:57.709743+0000","last_unstale":"2024-03-04T09:32:57.713699+0000","last_undegraded":"2024-03-04T09:32:57.713699+0000","last_fullsized":"2024-03-04T09:32:57.713699+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:24:22.403863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2d","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.718698+0000","last_change":"2024-03-04T09:32:57.713824+0000","last_active":"2024-03-04T09:32:57.718698+0000","last_peered":"2024-03-04T09:32:57.718698+0000","last_clean":"2024-03-04T09:32:57.718698+0000","last_became_active":"2024-03-04T09:32:57.712734+0000","last_became_peered":"2024-03-04T09:32:57.712734+0000","last_unstale":"2024-03-04T09:32:57.718698+0000","last_undegraded":"2024-03-04T09:32:57.718698+0000","last_fullsized":"2024-03-04T09:32:57.718698+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:37:52.957478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2c","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.705646+0000","last_change":"2024-03-04T09:32:57.703922+0000","last_active":"2024-03-04T09:32:57.705646+0000","last_peered":"2024-03-04T09:32:57.705646+0000","last_clean":"2024-03-04T09:32:57.705646+0000","last_became_active":"2024-03-04T09:32:57.702523+0000","last_became_peered":"2024-03-04T09:32:57.702523+0000","last_unstale":"2024-03-04T09:32:57.705646+0000","last_undegraded":"2024-03-04T09:32:57.705646+0000","last_fullsized":"2024-03-04T09:32:57.705646+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:02:31.190563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2b","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.719988+0000","last_change":"2024-03-04T09:32:57.716019+0000","last_active":"2024-03-04T09:32:57.719988+0000","last_peered":"2024-03-04T09:32:57.719988+0000","last_clean":"2024-03-04T09:32:57.719988+0000","last_became_active":"2024-03-04T09:32:57.715748+0000","last_became_peered":"2024-03-04T09:32:57.715748+0000","last_unstale":"2024-03-04T09:32:57.719988+0000","last_undegraded":"2024-03-04T09:32:57.719988+0000","last_fullsized":"2024-03-04T09:32:57.719988+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:01:34.747229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.2a","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.706758+0000","last_change":"2024-03-04T09:32:57.704594+0000","last_active":"2024-03-04T09:32:57.706758+0000","last_peered":"2024-03-04T09:32:57.706758+0000","last_clean":"2024-03-04T09:32:57.706758+0000","last_became_active":"2024-03-04T09:32:57.704238+0000","last_became_peered":"2024-03-04T09:32:57.704238+0000","last_unstale":"2024-03-04T09:32:57.706758+0000","last_undegraded":"2024-03-04T09:32:57.706758+0000","last_fullsized":"2024-03-04T09:32:57.706758+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:12:07.748148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.29","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.711323+0000","last_change":"2024-03-04T09:32:57.704311+0000","last_active":"2024-03-04T09:32:57.711323+0000","last_peered":"2024-03-04T09:32:57.711323+0000","last_clean":"2024-03-04T09:32:57.711323+0000","last_became_active":"2024-03-04T09:32:57.703704+0000","last_became_peered":"2024-03-04T09:32:57.703704+0000","last_unstale":"2024-03-04T09:32:57.711323+0000","last_undegraded":"2024-03-04T09:32:57.711323+0000","last_fullsized":"2024-03-04T09:32:57.711323+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:48:55.639006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.28","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.708882+0000","last_change":"2024-03-04T09:32:57.705889+0000","last_active":"2024-03-04T09:32:57.708882+0000","last_peered":"2024-03-04T09:32:57.708882+0000","last_clean":"2024-03-04T09:32:57.708882+0000","last_became_active":"2024-03-04T09:32:57.704468+0000","last_became_peered":"2024-03-04T09:32:57.704468+0000","last_unstale":"2024-03-04T09:32:57.708882+0000","last_undegraded":"2024-03-04T09:32:57.708882+0000","last_fullsized":"2024-03-04T09:32:57.708882+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:06:44.847594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.27","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.698762+0000","last_change":"2024-03-04T09:32:57.691041+0000","last_active":"2024-03-04T09:32:57.698762+0000","last_peered":"2024-03-04T09:32:57.698762+0000","last_clean":"2024-03-04T09:32:57.698762+0000","last_became_active":"2024-03-04T09:32:57.690660+0000","last_became_peered":"2024-03-04T09:32:57.690660+0000","last_unstale":"2024-03-04T09:32:57.698762+0000","last_undegraded":"2024-03-04T09:32:57.698762+0000","last_fullsized":"2024-03-04T09:32:57.698762+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:34:58.565938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.26","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.722283+0000","last_change":"2024-03-04T09:32:57.716816+0000","last_active":"2024-03-04T09:32:57.722283+0000","last_peered":"2024-03-04T09:32:57.722283+0000","last_clean":"2024-03-04T09:32:57.722283+0000","last_became_active":"2024-03-04T09:32:57.716493+0000","last_became_peered":"2024-03-04T09:32:57.716493+0000","last_unstale":"2024-03-04T09:32:57.722283+0000","last_undegraded":"2024-03-04T09:32:57.722283+0000","last_fullsized":"2024-03-04T09:32:57.722283+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:04:04.761815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.25","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.725322+0000","last_change":"2024-03-04T09:32:57.721163+0000","last_active":"2024-03-04T09:32:57.725322+0000","last_peered":"2024-03-04T09:32:57.725322+0000","last_clean":"2024-03-04T09:32:57.725322+0000","last_became_active":"2024-03-04T09:32:57.720853+0000","last_became_peered":"2024-03-04T09:32:57.720853+0000","last_unstale":"2024-03-04T09:32:57.725322+0000","last_undegraded":"2024-03-04T09:32:57.725322+0000","last_fullsized":"2024-03-04T09:32:57.725322+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:44:01.996743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.24","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.720227+0000","last_change":"2024-03-04T09:32:57.715598+0000","last_active":"2024-03-04T09:32:57.720227+0000","last_peered":"2024-03-04T09:32:57.720227+0000","last_clean":"2024-03-04T09:32:57.720227+0000","last_became_active":"2024-03-04T09:32:57.713046+0000","last_became_peered":"2024-03-04T09:32:57.713046+0000","last_unstale":"2024-03-04T09:32:57.720227+0000","last_undegraded":"2024-03-04T09:32:57.720227+0000","last_fullsized":"2024-03-04T09:32:57.720227+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:30:14.957488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.23","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.721832+0000","last_change":"2024-03-04T09:32:57.712616+0000","last_active":"2024-03-04T09:32:57.721832+0000","last_peered":"2024-03-04T09:32:57.721832+0000","last_clean":"2024-03-04T09:32:57.721832+0000","last_became_active":"2024-03-04T09:32:57.712299+0000","last_became_peered":"2024-03-04T09:32:57.712299+0000","last_unstale":"2024-03-04T09:32:57.721832+0000","last_undegraded":"2024-03-04T09:32:57.721832+0000","last_fullsized":"2024-03-04T09:32:57.721832+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:20:41.807104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.22","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.724612+0000","last_change":"2024-03-04T09:32:57.710547+0000","last_active":"2024-03-04T09:32:57.724612+0000","last_peered":"2024-03-04T09:32:57.724612+0000","last_clean":"2024-03-04T09:32:57.724612+0000","last_became_active":"2024-03-04T09:32:57.709304+0000","last_became_peered":"2024-03-04T09:32:57.709304+0000","last_unstale":"2024-03-04T09:32:57.724612+0000","last_undegraded":"2024-03-04T09:32:57.724612+0000","last_fullsized":"2024-03-04T09:32:57.724612+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:26:03.089663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.21","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.713005+0000","last_change":"2024-03-04T09:32:57.710484+0000","last_active":"2024-03-04T09:32:57.713005+0000","last_peered":"2024-03-04T09:32:57.713005+0000","last_clean":"2024-03-04T09:32:57.713005+0000","last_became_active":"2024-03-04T09:32:57.709988+0000","last_became_peered":"2024-03-04T09:32:57.709988+0000","last_unstale":"2024-03-04T09:32:57.713005+0000","last_undegraded":"2024-03-04T09:32:57.713005+0000","last_fullsized":"2024-03-04T09:32:57.713005+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:57:49.438902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.20","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.715227+0000","last_change":"2024-03-04T09:32:57.710330+0000","last_active":"2024-03-04T09:32:57.715227+0000","last_peered":"2024-03-04T09:32:57.715227+0000","last_clean":"2024-03-04T09:32:57.715227+0000","last_became_active":"2024-03-04T09:32:57.706308+0000","last_became_peered":"2024-03-04T09:32:57.706308+0000","last_unstale":"2024-03-04T09:32:57.715227+0000","last_undegraded":"2024-03-04T09:32:57.715227+0000","last_fullsized":"2024-03-04T09:32:57.715227+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:01:15.944370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1e","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.722304+0000","last_change":"2024-03-04T09:32:57.719861+0000","last_active":"2024-03-04T09:32:57.722304+0000","last_peered":"2024-03-04T09:32:57.722304+0000","last_clean":"2024-03-04T09:32:57.722304+0000","last_became_active":"2024-03-04T09:32:57.719568+0000","last_became_peered":"2024-03-04T09:32:57.719568+0000","last_unstale":"2024-03-04T09:32:57.722304+0000","last_undegraded":"2024-03-04T09:32:57.722304+0000","last_fullsized":"2024-03-04T09:32:57.722304+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:20:03.748890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1f","version":"126'4","reported_seq":16,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.725664+0000","last_change":"2024-03-04T09:32:56.708356+0000","last_active":"2024-03-04T09:32:57.725664+0000","last_peered":"2024-03-04T09:32:57.725664+0000","last_clean":"2024-03-04T09:32:57.725664+0000","last_became_active":"2024-03-04T09:32:56.708018+0000","last_became_peered":"2024-03-04T09:32:56.708018+0000","last_unstale":"2024-03-04T09:32:57.725664+0000","last_undegraded":"2024-03-04T09:32:57.725664+0000","last_fullsized":"2024-03-04T09:32:57.725664+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_clean_scrub_stamp":"2024-03-04T09:32:55.635455+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:34:30.369830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.1f","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.721154+0000","last_change":"2024-03-04T09:32:57.718404+0000","last_active":"2024-03-04T09:32:57.721154+0000","last_peered":"2024-03-04T09:32:57.721154+0000","last_clean":"2024-03-04T09:32:57.721154+0000","last_became_active":"2024-03-04T09:32:57.718124+0000","last_became_peered":"2024-03-04T09:32:57.718124+0000","last_unstale":"2024-03-04T09:32:57.721154+0000","last_undegraded":"2024-03-04T09:32:57.721154+0000","last_fullsized":"2024-03-04T09:32:57.721154+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:17:22.364556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1e","version":"126'1","reported_seq":13,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.722589+0000","last_change":"2024-03-04T09:32:56.698608+0000","last_active":"2024-03-04T09:32:57.722589+0000","last_peered":"2024-03-04T09:32:57.722589+0000","last_clean":"2024-03-04T09:32:57.722589+0000","last_became_active":"2024-03-04T09:32:56.696608+0000","last_became_peered":"2024-03-04T09:32:56.696608+0000","last_unstale":"2024-03-04T09:32:57.722589+0000","last_undegraded":"2024-03-04T09:32:57.722589+0000","last_fullsized":"2024-03-04T09:32:57.722589+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_clean_scrub_stamp":"2024-03-04T09:32:55.635455+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:34:26.054015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.1c","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.712110+0000","last_change":"2024-03-04T09:32:57.707787+0000","last_active":"2024-03-04T09:32:57.712110+0000","last_peered":"2024-03-04T09:32:57.712110+0000","last_clean":"2024-03-04T09:32:57.712110+0000","last_became_active":"2024-03-04T09:32:57.704504+0000","last_became_peered":"2024-03-04T09:32:57.704504+0000","last_unstale":"2024-03-04T09:32:57.712110+0000","last_undegraded":"2024-03-04T09:32:57.712110+0000","last_fullsized":"2024-03-04T09:32:57.712110+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:36:29.166241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1d","version":"126'2","reported_seq":14,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.723238+0000","last_change":"2024-03-04T09:32:56.691446+0000","last_active":"2024-03-04T09:32:57.723238+0000","last_peered":"2024-03-04T09:32:57.723238+0000","last_clean":"2024-03-04T09:32:57.723238+0000","last_became_active":"2024-03-04T09:32:56.691095+0000","last_became_peered":"2024-03-04T09:32:56.691095+0000","last_unstale":"2024-03-04T09:32:57.723238+0000","last_undegraded":"2024-03-04T09:32:57.723238+0000","last_fullsized":"2024-03-04T09:32:57.723238+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_clean_scrub_stamp":"2024-03-04T09:32:55.635455+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:49:40.752903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.1d","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.704200+0000","last_change":"2024-03-04T09:32:57.702688+0000","last_active":"2024-03-04T09:32:57.704200+0000","last_peered":"2024-03-04T09:32:57.704200+0000","last_clean":"2024-03-04T09:32:57.704200+0000","last_became_active":"2024-03-04T09:32:57.700913+0000","last_became_peered":"2024-03-04T09:32:57.700913+0000","last_unstale":"2024-03-04T09:32:57.704200+0000","last_undegraded":"2024-03-04T09:32:57.704200+0000","last_fullsized":"2024-03-04T09:32:57.704200+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:32:26.208985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1c","version":"126'2","reported_seq":14,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.698370+0000","last_change":"2024-03-04T09:32:56.682630+0000","last_active":"2024-03-04T09:32:57.698370+0000","last_peered":"2024-03-04T09:32:57.698370+0000","last_clean":"2024-03-04T09:32:57.698370+0000","last_became_active":"2024-03-04T09:32:56.681263+0000","last_became_peered":"2024-03-04T09:32:56.681263+0000","last_unstale":"2024-03-04T09:32:57.698370+0000","last_undegraded":"2024-03-04T09:32:57.698370+0000","last_fullsized":"2024-03-04T09:32:57.698370+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_clean_scrub_stamp":"2024-03-04T09:32:55.635455+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:04:08.822065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1a","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.722709+0000","last_change":"2024-03-04T09:32:57.716596+0000","last_active":"2024-03-04T09:32:57.722709+0000","last_peered":"2024-03-04T09:32:57.722709+0000","last_clean":"2024-03-04T09:32:57.722709+0000","last_became_active":"2024-03-04T09:32:57.715856+0000","last_became_peered":"2024-03-04T09:32:57.715856+0000","last_unstale":"2024-03-04T09:32:57.722709+0000","last_undegraded":"2024-03-04T09:32:57.722709+0000","last_fullsized":"2024-03-04T09:32:57.722709+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:04:33.104550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1b","version":"126'1","reported_seq":13,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.700303+0000","last_change":"2024-03-04T09:32:56.691411+0000","last_active":"2024-03-04T09:32:57.700303+0000","last_peered":"2024-03-04T09:32:57.700303+0000","last_clean":"2024-03-04T09:32:57.700303+0000","last_became_active":"2024-03-04T09:32:56.691109+0000","last_became_peered":"2024-03-04T09:32:56.691109+0000","last_unstale":"2024-03-04T09:32:57.700303+0000","last_undegraded":"2024-03-04T09:32:57.700303+0000","last_fullsized":"2024-03-04T09:32:57.700303+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_clean_scrub_stamp":"2024-03-04T09:32:55.635455+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:04:14.387131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1b","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.717787+0000","last_change":"2024-03-04T09:32:57.711413+0000","last_active":"2024-03-04T09:32:57.717787+0000","last_peered":"2024-03-04T09:32:57.717787+0000","last_clean":"2024-03-04T09:32:57.717787+0000","last_became_active":"2024-03-04T09:32:57.711128+0000","last_became_peered":"2024-03-04T09:32:57.711128+0000","last_unstale":"2024-03-04T09:32:57.717787+0000","last_undegraded":"2024-03-04T09:32:57.717787+0000","last_fullsized":"2024-03-04T09:32:57.717787+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:21:34.147081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.1a","version":"126'1","reported_seq":13,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.701897+0000","last_change":"2024-03-04T09:32:56.693988+0000","last_active":"2024-03-04T09:32:57.701897+0000","last_peered":"2024-03-04T09:32:57.701897+0000","last_clean":"2024-03-04T09:32:57.701897+0000","last_became_active":"2024-03-04T09:32:56.693521+0000","last_became_peered":"2024-03-04T09:32:56.693521+0000","last_unstale":"2024-03-04T09:32:57.701897+0000","last_undegraded":"2024-03-04T09:32:57.701897+0000","last_fullsized":"2024-03-04T09:32:57.701897+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_clean_scrub_stamp":"2024-03-04T09:32:55.635455+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:53:08.200757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.18","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.714647+0000","last_change":"2024-03-04T09:32:57.702755+0000","last_active":"2024-03-04T09:32:57.714647+0000","last_peered":"2024-03-04T09:32:57.714647+0000","last_clean":"2024-03-04T09:32:57.714647+0000","last_became_active":"2024-03-04T09:32:57.702038+0000","last_became_peered":"2024-03-04T09:32:57.702038+0000","last_unstale":"2024-03-04T09:32:57.714647+0000","last_undegraded":"2024-03-04T09:32:57.714647+0000","last_fullsized":"2024-03-04T09:32:57.714647+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:12:39.009902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.696328+0000","last_change":"2024-03-04T09:32:56.707512+0000","last_active":"2024-03-04T09:32:57.696328+0000","last_peered":"2024-03-04T09:32:57.696328+0000","last_clean":"2024-03-04T09:32:57.696328+0000","last_became_active":"2024-03-04T09:32:56.707061+0000","last_became_peered":"2024-03-04T09:32:56.707061+0000","last_unstale":"2024-03-04T09:32:57.696328+0000","last_undegraded":"2024-03-04T09:32:57.696328+0000","last_fullsized":"2024-03-04T09:32:57.696328+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_clean_scrub_stamp":"2024-03-04T09:32:55.635455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:01:28.738130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.702797+0000","last_change":"2024-03-04T09:32:57.699075+0000","last_active":"2024-03-04T09:32:57.702797+0000","last_peered":"2024-03-04T09:32:57.702797+0000","last_clean":"2024-03-04T09:32:57.702797+0000","last_became_active":"2024-03-04T09:32:57.698684+0000","last_became_peered":"2024-03-04T09:32:57.698684+0000","last_unstale":"2024-03-04T09:32:57.702797+0000","last_undegraded":"2024-03-04T09:32:57.702797+0000","last_fullsized":"2024-03-04T09:32:57.702797+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:35:24.731959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.677928+0000","last_change":"2024-03-04T09:32:56.706704+0000","last_active":"2024-03-04T09:32:57.677928+0000","last_peered":"2024-03-04T09:32:57.677928+0000","last_clean":"2024-03-04T09:32:57.677928+0000","last_became_active":"2024-03-04T09:32:56.706388+0000","last_became_peered":"2024-03-04T09:32:56.706388+0000","last_unstale":"2024-03-04T09:32:57.677928+0000","last_undegraded":"2024-03-04T09:32:57.677928+0000","last_fullsized":"2024-03-04T09:32:57.677928+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_clean_scrub_stamp":"2024-03-04T09:32:55.635455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:36:40.943782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.16","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.725412+0000","last_change":"2024-03-04T09:32:57.722441+0000","last_active":"2024-03-04T09:32:57.725412+0000","last_peered":"2024-03-04T09:32:57.725412+0000","last_clean":"2024-03-04T09:32:57.725412+0000","last_became_active":"2024-03-04T09:32:57.722175+0000","last_became_peered":"2024-03-04T09:32:57.722175+0000","last_unstale":"2024-03-04T09:32:57.725412+0000","last_undegraded":"2024-03-04T09:32:57.725412+0000","last_fullsized":"2024-03-04T09:32:57.725412+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:58:22.075534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.713459+0000","last_change":"2024-03-04T09:32:56.699354+0000","last_active":"2024-03-04T09:32:57.713459+0000","last_peered":"2024-03-04T09:32:57.713459+0000","last_clean":"2024-03-04T09:32:57.713459+0000","last_became_active":"2024-03-04T09:32:56.698960+0000","last_became_peered":"2024-03-04T09:32:56.698960+0000","last_unstale":"2024-03-04T09:32:57.713459+0000","last_undegraded":"2024-03-04T09:32:57.713459+0000","last_fullsized":"2024-03-04T09:32:57.713459+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_clean_scrub_stamp":"2024-03-04T09:32:55.635455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:07:06.052094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.17","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.719755+0000","last_change":"2024-03-04T09:32:57.713613+0000","last_active":"2024-03-04T09:32:57.719755+0000","last_peered":"2024-03-04T09:32:57.719755+0000","last_clean":"2024-03-04T09:32:57.719755+0000","last_became_active":"2024-03-04T09:32:57.710733+0000","last_became_peered":"2024-03-04T09:32:57.710733+0000","last_unstale":"2024-03-04T09:32:57.719755+0000","last_undegraded":"2024-03-04T09:32:57.719755+0000","last_fullsized":"2024-03-04T09:32:57.719755+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:12:39.009902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.694899+0000","last_change":"2024-03-04T09:32:56.710490+0000","last_active":"2024-03-04T09:32:57.694899+0000","last_peered":"2024-03-04T09:32:57.694899+0000","last_clean":"2024-03-04T09:32:57.694899+0000","last_became_active":"2024-03-04T09:32:56.710232+0000","last_became_peered":"2024-03-04T09:32:56.710232+0000","last_unstale":"2024-03-04T09:32:57.694899+0000","last_undegraded":"2024-03-04T09:32:57.694899+0000","last_fullsized":"2024-03-04T09:32:57.694899+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_clean_scrub_stamp":"2024-03-04T09:32:55.635455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:00:59.755527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.14","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.710612+0000","last_change":"2024-03-04T09:32:57.703977+0000","last_active":"2024-03-04T09:32:57.710612+0000","last_peered":"2024-03-04T09:32:57.710612+0000","last_clean":"2024-03-04T09:32:57.710612+0000","last_became_active":"2024-03-04T09:32:57.703018+0000","last_became_peered":"2024-03-04T09:32:57.703018+0000","last_unstale":"2024-03-04T09:32:57.710612+0000","last_undegraded":"2024-03-04T09:32:57.710612+0000","last_fullsized":"2024-03-04T09:32:57.710612+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:56:31.981113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.15","version":"126'3","reported_seq":15,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.721358+0000","last_change":"2024-03-04T09:32:56.697367+0000","last_active":"2024-03-04T09:32:57.721358+0000","last_peered":"2024-03-04T09:32:57.721358+0000","last_clean":"2024-03-04T09:32:57.721358+0000","last_became_active":"2024-03-04T09:32:56.696977+0000","last_became_peered":"2024-03-04T09:32:56.696977+0000","last_unstale":"2024-03-04T09:32:57.721358+0000","last_undegraded":"2024-03-04T09:32:57.721358+0000","last_fullsized":"2024-03-04T09:32:57.721358+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_clean_scrub_stamp":"2024-03-04T09:32:55.635455+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:36:40.943782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.15","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.712859+0000","last_change":"2024-03-04T09:32:57.709156+0000","last_active":"2024-03-04T09:32:57.712859+0000","last_peered":"2024-03-04T09:32:57.712859+0000","last_clean":"2024-03-04T09:32:57.712859+0000","last_became_active":"2024-03-04T09:32:57.708841+0000","last_became_peered":"2024-03-04T09:32:57.708841+0000","last_unstale":"2024-03-04T09:32:57.712859+0000","last_undegraded":"2024-03-04T09:32:57.712859+0000","last_fullsized":"2024-03-04T09:32:57.712859+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:21:50.465577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.14","version":"126'6","reported_seq":18,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.723698+0000","last_change":"2024-03-04T09:32:56.709561+0000","last_active":"2024-03-04T09:32:57.723698+0000","last_peered":"2024-03-04T09:32:57.723698+0000","last_clean":"2024-03-04T09:32:57.723698+0000","last_became_active":"2024-03-04T09:32:56.708663+0000","last_became_peered":"2024-03-04T09:32:56.708663+0000","last_unstale":"2024-03-04T09:32:57.723698+0000","last_undegraded":"2024-03-04T09:32:57.723698+0000","last_fullsized":"2024-03-04T09:32:57.723698+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_clean_scrub_stamp":"2024-03-04T09:32:55.635455+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:04:08.822065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.13","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.721811+0000","last_change":"2024-03-04T09:32:57.711681+0000","last_active":"2024-03-04T09:32:57.721811+0000","last_peered":"2024-03-04T09:32:57.721811+0000","last_clean":"2024-03-04T09:32:57.721811+0000","last_became_active":"2024-03-04T09:32:57.711203+0000","last_became_peered":"2024-03-04T09:32:57.711203+0000","last_unstale":"2024-03-04T09:32:57.721811+0000","last_undegraded":"2024-03-04T09:32:57.721811+0000","last_fullsized":"2024-03-04T09:32:57.721811+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:38:06.426382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.709114+0000","last_change":"2024-03-04T09:32:56.702422+0000","last_active":"2024-03-04T09:32:57.709114+0000","last_peered":"2024-03-04T09:32:57.709114+0000","last_clean":"2024-03-04T09:32:57.709114+0000","last_became_active":"2024-03-04T09:32:56.701984+0000","last_became_peered":"2024-03-04T09:32:56.701984+0000","last_unstale":"2024-03-04T09:32:57.709114+0000","last_undegraded":"2024-03-04T09:32:57.709114+0000","last_fullsized":"2024-03-04T09:32:57.709114+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_clean_scrub_stamp":"2024-03-04T09:32:55.635455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:25:11.338620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.10","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.701242+0000","last_change":"2024-03-04T09:32:57.693627+0000","last_active":"2024-03-04T09:32:57.701242+0000","last_peered":"2024-03-04T09:32:57.701242+0000","last_clean":"2024-03-04T09:32:57.701242+0000","last_became_active":"2024-03-04T09:32:57.693303+0000","last_became_peered":"2024-03-04T09:32:57.693303+0000","last_unstale":"2024-03-04T09:32:57.701242+0000","last_undegraded":"2024-03-04T09:32:57.701242+0000","last_fullsized":"2024-03-04T09:32:57.701242+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:37:21.506217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.678816+0000","last_change":"2024-03-04T09:32:56.701788+0000","last_active":"2024-03-04T09:32:57.678816+0000","last_peered":"2024-03-04T09:32:57.678816+0000","last_clean":"2024-03-04T09:32:57.678816+0000","last_became_active":"2024-03-04T09:32:56.701356+0000","last_became_peered":"2024-03-04T09:32:56.701356+0000","last_unstale":"2024-03-04T09:32:57.678816+0000","last_undegraded":"2024-03-04T09:32:57.678816+0000","last_fullsized":"2024-03-04T09:32:57.678816+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_clean_scrub_stamp":"2024-03-04T09:32:55.635455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:07:06.052094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.11","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.711346+0000","last_change":"2024-03-04T09:32:57.704622+0000","last_active":"2024-03-04T09:32:57.711346+0000","last_peered":"2024-03-04T09:32:57.711346+0000","last_clean":"2024-03-04T09:32:57.711346+0000","last_became_active":"2024-03-04T09:32:57.704058+0000","last_became_peered":"2024-03-04T09:32:57.704058+0000","last_unstale":"2024-03-04T09:32:57.711346+0000","last_undegraded":"2024-03-04T09:32:57.711346+0000","last_fullsized":"2024-03-04T09:32:57.711346+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:29:37.209382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.10","version":"126'8","reported_seq":20,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.722435+0000","last_change":"2024-03-04T09:32:56.707051+0000","last_active":"2024-03-04T09:32:57.722435+0000","last_peered":"2024-03-04T09:32:57.722435+0000","last_clean":"2024-03-04T09:32:57.722435+0000","last_became_active":"2024-03-04T09:32:56.705283+0000","last_became_peered":"2024-03-04T09:32:56.705283+0000","last_unstale":"2024-03-04T09:32:57.722435+0000","last_undegraded":"2024-03-04T09:32:57.722435+0000","last_fullsized":"2024-03-04T09:32:57.722435+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_clean_scrub_stamp":"2024-03-04T09:32:55.635455+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:07:06.052094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.e","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.721327+0000","last_change":"2024-03-04T09:32:57.715518+0000","last_active":"2024-03-04T09:32:57.721327+0000","last_peered":"2024-03-04T09:32:57.721327+0000","last_clean":"2024-03-04T09:32:57.721327+0000","last_became_active":"2024-03-04T09:32:57.714076+0000","last_became_peered":"2024-03-04T09:32:57.714076+0000","last_unstale":"2024-03-04T09:32:57.721327+0000","last_undegraded":"2024-03-04T09:32:57.721327+0000","last_fullsized":"2024-03-04T09:32:57.721327+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:01:47.019520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.f","version":"126'1","reported_seq":13,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.720863+0000","last_change":"2024-03-04T09:32:56.703103+0000","last_active":"2024-03-04T09:32:57.720863+0000","last_peered":"2024-03-04T09:32:57.720863+0000","last_clean":"2024-03-04T09:32:57.720863+0000","last_became_active":"2024-03-04T09:32:56.702790+0000","last_became_peered":"2024-03-04T09:32:56.702790+0000","last_unstale":"2024-03-04T09:32:57.720863+0000","last_undegraded":"2024-03-04T09:32:57.720863+0000","last_fullsized":"2024-03-04T09:32:57.720863+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_clean_scrub_stamp":"2024-03-04T09:32:55.635455+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:34:57.549462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.f","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.720146+0000","last_change":"2024-03-04T09:32:57.714710+0000","last_active":"2024-03-04T09:32:57.720146+0000","last_peered":"2024-03-04T09:32:57.720146+0000","last_clean":"2024-03-04T09:32:57.720146+0000","last_became_active":"2024-03-04T09:32:57.714346+0000","last_became_peered":"2024-03-04T09:32:57.714346+0000","last_unstale":"2024-03-04T09:32:57.720146+0000","last_undegraded":"2024-03-04T09:32:57.720146+0000","last_fullsized":"2024-03-04T09:32:57.720146+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:37:21.506217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.e","version":"126'2","reported_seq":14,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.723743+0000","last_change":"2024-03-04T09:32:56.707947+0000","last_active":"2024-03-04T09:32:57.723743+0000","last_peered":"2024-03-04T09:32:57.723743+0000","last_clean":"2024-03-04T09:32:57.723743+0000","last_became_active":"2024-03-04T09:32:56.707630+0000","last_became_peered":"2024-03-04T09:32:56.707630+0000","last_unstale":"2024-03-04T09:32:57.723743+0000","last_undegraded":"2024-03-04T09:32:57.723743+0000","last_fullsized":"2024-03-04T09:32:57.723743+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_clean_scrub_stamp":"2024-03-04T09:32:55.635455+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:43:43.184312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.722221+0000","last_change":"2024-03-04T09:32:57.718915+0000","last_active":"2024-03-04T09:32:57.722221+0000","last_peered":"2024-03-04T09:32:57.722221+0000","last_clean":"2024-03-04T09:32:57.722221+0000","last_became_active":"2024-03-04T09:32:57.718641+0000","last_became_peered":"2024-03-04T09:32:57.718641+0000","last_unstale":"2024-03-04T09:32:57.722221+0000","last_undegraded":"2024-03-04T09:32:57.722221+0000","last_fullsized":"2024-03-04T09:32:57.722221+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:40:10.018154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.d","version":"126'1","reported_seq":13,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.712873+0000","last_change":"2024-03-04T09:32:56.704416+0000","last_active":"2024-03-04T09:32:57.712873+0000","last_peered":"2024-03-04T09:32:57.712873+0000","last_clean":"2024-03-04T09:32:57.712873+0000","last_became_active":"2024-03-04T09:32:56.704131+0000","last_became_peered":"2024-03-04T09:32:56.704131+0000","last_unstale":"2024-03-04T09:32:57.712873+0000","last_undegraded":"2024-03-04T09:32:57.712873+0000","last_fullsized":"2024-03-04T09:32:57.712873+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_clean_scrub_stamp":"2024-03-04T09:32:55.635455+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:10:27.785793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.d","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.702852+0000","last_change":"2024-03-04T09:32:57.698996+0000","last_active":"2024-03-04T09:32:57.702852+0000","last_peered":"2024-03-04T09:32:57.702852+0000","last_clean":"2024-03-04T09:32:57.702852+0000","last_became_active":"2024-03-04T09:32:57.698685+0000","last_became_peered":"2024-03-04T09:32:57.698685+0000","last_unstale":"2024-03-04T09:32:57.702852+0000","last_undegraded":"2024-03-04T09:32:57.702852+0000","last_fullsized":"2024-03-04T09:32:57.702852+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:29:37.209382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.712842+0000","last_change":"2024-03-04T09:32:56.698084+0000","last_active":"2024-03-04T09:32:57.712842+0000","last_peered":"2024-03-04T09:32:57.712842+0000","last_clean":"2024-03-04T09:32:57.712842+0000","last_became_active":"2024-03-04T09:32:56.697794+0000","last_became_peered":"2024-03-04T09:32:56.697794+0000","last_unstale":"2024-03-04T09:32:57.712842+0000","last_undegraded":"2024-03-04T09:32:57.712842+0000","last_fullsized":"2024-03-04T09:32:57.712842+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_clean_scrub_stamp":"2024-03-04T09:32:55.635455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:04:08.822065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.a","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.724637+0000","last_change":"2024-03-04T09:32:57.716276+0000","last_active":"2024-03-04T09:32:57.724637+0000","last_peered":"2024-03-04T09:32:57.724637+0000","last_clean":"2024-03-04T09:32:57.724637+0000","last_became_active":"2024-03-04T09:32:57.715575+0000","last_became_peered":"2024-03-04T09:32:57.715575+0000","last_unstale":"2024-03-04T09:32:57.724637+0000","last_undegraded":"2024-03-04T09:32:57.724637+0000","last_fullsized":"2024-03-04T09:32:57.724637+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:40:42.783681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.b","version":"126'2","reported_seq":14,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.725662+0000","last_change":"2024-03-04T09:32:56.697279+0000","last_active":"2024-03-04T09:32:57.725662+0000","last_peered":"2024-03-04T09:32:57.725662+0000","last_clean":"2024-03-04T09:32:57.725662+0000","last_became_active":"2024-03-04T09:32:56.696722+0000","last_became_peered":"2024-03-04T09:32:56.696722+0000","last_unstale":"2024-03-04T09:32:57.725662+0000","last_undegraded":"2024-03-04T09:32:57.725662+0000","last_fullsized":"2024-03-04T09:32:57.725662+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_clean_scrub_stamp":"2024-03-04T09:32:55.635455+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:04:14.387131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.b","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.718059+0000","last_change":"2024-03-04T09:32:57.714845+0000","last_active":"2024-03-04T09:32:57.718059+0000","last_peered":"2024-03-04T09:32:57.718059+0000","last_clean":"2024-03-04T09:32:57.718059+0000","last_became_active":"2024-03-04T09:32:57.714520+0000","last_became_peered":"2024-03-04T09:32:57.714520+0000","last_unstale":"2024-03-04T09:32:57.718059+0000","last_undegraded":"2024-03-04T09:32:57.718059+0000","last_fullsized":"2024-03-04T09:32:57.718059+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:44:15.764637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.695894+0000","last_change":"2024-03-04T09:32:56.705846+0000","last_active":"2024-03-04T09:32:57.695894+0000","last_peered":"2024-03-04T09:32:57.695894+0000","last_clean":"2024-03-04T09:32:57.695894+0000","last_became_active":"2024-03-04T09:32:56.705198+0000","last_became_peered":"2024-03-04T09:32:56.705198+0000","last_unstale":"2024-03-04T09:32:57.695894+0000","last_undegraded":"2024-03-04T09:32:57.695894+0000","last_fullsized":"2024-03-04T09:32:57.695894+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_clean_scrub_stamp":"2024-03-04T09:32:55.635455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:01:33.730754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.8","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.701262+0000","last_change":"2024-03-04T09:32:57.696969+0000","last_active":"2024-03-04T09:32:57.701262+0000","last_peered":"2024-03-04T09:32:57.701262+0000","last_clean":"2024-03-04T09:32:57.701262+0000","last_became_active":"2024-03-04T09:32:57.696675+0000","last_became_peered":"2024-03-04T09:32:57.696675+0000","last_unstale":"2024-03-04T09:32:57.701262+0000","last_undegraded":"2024-03-04T09:32:57.701262+0000","last_fullsized":"2024-03-04T09:32:57.701262+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:48:55.639006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.689228+0000","last_change":"2024-03-04T09:32:56.700895+0000","last_active":"2024-03-04T09:32:57.689228+0000","last_peered":"2024-03-04T09:32:57.689228+0000","last_clean":"2024-03-04T09:32:57.689228+0000","last_became_active":"2024-03-04T09:32:56.700581+0000","last_became_peered":"2024-03-04T09:32:56.700581+0000","last_unstale":"2024-03-04T09:32:57.689228+0000","last_undegraded":"2024-03-04T09:32:57.689228+0000","last_fullsized":"2024-03-04T09:32:57.689228+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_clean_scrub_stamp":"2024-03-04T09:32:55.635455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:21:49.449101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.713025+0000","last_change":"2024-03-04T09:32:57.711121+0000","last_active":"2024-03-04T09:32:57.713025+0000","last_peered":"2024-03-04T09:32:57.713025+0000","last_clean":"2024-03-04T09:32:57.713025+0000","last_became_active":"2024-03-04T09:32:57.710405+0000","last_became_peered":"2024-03-04T09:32:57.710405+0000","last_unstale":"2024-03-04T09:32:57.713025+0000","last_undegraded":"2024-03-04T09:32:57.713025+0000","last_fullsized":"2024-03-04T09:32:57.713025+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:01:00.772002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.699340+0000","last_change":"2024-03-04T09:32:56.712214+0000","last_active":"2024-03-04T09:32:57.699340+0000","last_peered":"2024-03-04T09:32:57.699340+0000","last_clean":"2024-03-04T09:32:57.699340+0000","last_became_active":"2024-03-04T09:32:56.711412+0000","last_became_peered":"2024-03-04T09:32:56.711412+0000","last_unstale":"2024-03-04T09:32:57.699340+0000","last_undegraded":"2024-03-04T09:32:57.699340+0000","last_fullsized":"2024-03-04T09:32:57.699340+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_clean_scrub_stamp":"2024-03-04T09:32:55.635455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:42:32.193040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.6","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.721539+0000","last_change":"2024-03-04T09:32:57.718067+0000","last_active":"2024-03-04T09:32:57.721539+0000","last_peered":"2024-03-04T09:32:57.721539+0000","last_clean":"2024-03-04T09:32:57.721539+0000","last_became_active":"2024-03-04T09:32:57.716705+0000","last_became_peered":"2024-03-04T09:32:57.716705+0000","last_unstale":"2024-03-04T09:32:57.721539+0000","last_undegraded":"2024-03-04T09:32:57.721539+0000","last_fullsized":"2024-03-04T09:32:57.721539+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:51:36.030782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.7","version":"126'2","reported_seq":14,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.725065+0000","last_change":"2024-03-04T09:32:56.699236+0000","last_active":"2024-03-04T09:32:57.725065+0000","last_peered":"2024-03-04T09:32:57.725065+0000","last_clean":"2024-03-04T09:32:57.725065+0000","last_became_active":"2024-03-04T09:32:56.698957+0000","last_became_peered":"2024-03-04T09:32:56.698957+0000","last_unstale":"2024-03-04T09:32:57.725065+0000","last_undegraded":"2024-03-04T09:32:57.725065+0000","last_fullsized":"2024-03-04T09:32:57.725065+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_clean_scrub_stamp":"2024-03-04T09:32:55.635455+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:45:47.015969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.7","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.725631+0000","last_change":"2024-03-04T09:32:57.724317+0000","last_active":"2024-03-04T09:32:57.725631+0000","last_peered":"2024-03-04T09:32:57.725631+0000","last_clean":"2024-03-04T09:32:57.725631+0000","last_became_active":"2024-03-04T09:32:57.723846+0000","last_became_peered":"2024-03-04T09:32:57.723846+0000","last_unstale":"2024-03-04T09:32:57.725631+0000","last_undegraded":"2024-03-04T09:32:57.725631+0000","last_fullsized":"2024-03-04T09:32:57.725631+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:06:12.093886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.6","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.688281+0000","last_change":"2024-03-04T09:32:56.708274+0000","last_active":"2024-03-04T09:32:57.688281+0000","last_peered":"2024-03-04T09:32:57.688281+0000","last_clean":"2024-03-04T09:32:57.688281+0000","last_became_active":"2024-03-04T09:32:56.707981+0000","last_became_peered":"2024-03-04T09:32:56.707981+0000","last_unstale":"2024-03-04T09:32:57.688281+0000","last_undegraded":"2024-03-04T09:32:57.688281+0000","last_fullsized":"2024-03-04T09:32:57.688281+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_clean_scrub_stamp":"2024-03-04T09:32:55.635455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:44:14.748162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.698442+0000","last_change":"2024-03-04T09:32:57.691181+0000","last_active":"2024-03-04T09:32:57.698442+0000","last_peered":"2024-03-04T09:32:57.698442+0000","last_clean":"2024-03-04T09:32:57.698442+0000","last_became_active":"2024-03-04T09:32:57.690874+0000","last_became_peered":"2024-03-04T09:32:57.690874+0000","last_unstale":"2024-03-04T09:32:57.698442+0000","last_undegraded":"2024-03-04T09:32:57.698442+0000","last_fullsized":"2024-03-04T09:32:57.698442+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:12:39.009902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.677940+0000","last_change":"2024-03-04T09:32:56.710983+0000","last_active":"2024-03-04T09:32:57.677940+0000","last_peered":"2024-03-04T09:32:57.677940+0000","last_clean":"2024-03-04T09:32:57.677940+0000","last_became_active":"2024-03-04T09:32:56.710651+0000","last_became_peered":"2024-03-04T09:32:56.710651+0000","last_unstale":"2024-03-04T09:32:57.677940+0000","last_undegraded":"2024-03-04T09:32:57.677940+0000","last_fullsized":"2024-03-04T09:32:57.677940+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_clean_scrub_stamp":"2024-03-04T09:32:55.635455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:34:26.054015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.721780+0000","last_change":"2024-03-04T09:32:57.716019+0000","last_active":"2024-03-04T09:32:57.721780+0000","last_peered":"2024-03-04T09:32:57.721780+0000","last_clean":"2024-03-04T09:32:57.721780+0000","last_became_active":"2024-03-04T09:32:57.715731+0000","last_became_peered":"2024-03-04T09:32:57.715731+0000","last_unstale":"2024-03-04T09:32:57.721780+0000","last_undegraded":"2024-03-04T09:32:57.721780+0000","last_fullsized":"2024-03-04T09:32:57.721780+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:32:26.208985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.700183+0000","last_change":"2024-03-04T09:32:56.682683+0000","last_active":"2024-03-04T09:32:57.700183+0000","last_peered":"2024-03-04T09:32:57.700183+0000","last_clean":"2024-03-04T09:32:57.700183+0000","last_became_active":"2024-03-04T09:32:56.682313+0000","last_became_peered":"2024-03-04T09:32:56.682313+0000","last_unstale":"2024-03-04T09:32:57.700183+0000","last_undegraded":"2024-03-04T09:32:57.700183+0000","last_fullsized":"2024-03-04T09:32:57.700183+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_clean_scrub_stamp":"2024-03-04T09:32:55.635455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:15:18.079206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.711379+0000","last_change":"2024-03-04T09:32:57.704259+0000","last_active":"2024-03-04T09:32:57.711379+0000","last_peered":"2024-03-04T09:32:57.711379+0000","last_clean":"2024-03-04T09:32:57.711379+0000","last_became_active":"2024-03-04T09:32:57.703939+0000","last_became_peered":"2024-03-04T09:32:57.703939+0000","last_unstale":"2024-03-04T09:32:57.711379+0000","last_undegraded":"2024-03-04T09:32:57.711379+0000","last_fullsized":"2024-03-04T09:32:57.711379+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:33:04.056111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.677563+0000","last_change":"2024-03-04T09:32:56.703624+0000","last_active":"2024-03-04T09:32:57.677563+0000","last_peered":"2024-03-04T09:32:57.677563+0000","last_clean":"2024-03-04T09:32:57.677563+0000","last_became_active":"2024-03-04T09:32:56.703324+0000","last_became_peered":"2024-03-04T09:32:56.703324+0000","last_unstale":"2024-03-04T09:32:57.677563+0000","last_undegraded":"2024-03-04T09:32:57.677563+0000","last_fullsized":"2024-03-04T09:32:57.677563+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_clean_scrub_stamp":"2024-03-04T09:32:55.635455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:57:27.241237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.720259+0000","last_change":"2024-03-04T09:32:57.715965+0000","last_active":"2024-03-04T09:32:57.720259+0000","last_peered":"2024-03-04T09:32:57.720259+0000","last_clean":"2024-03-04T09:32:57.720259+0000","last_became_active":"2024-03-04T09:32:57.715594+0000","last_became_peered":"2024-03-04T09:32:57.715594+0000","last_unstale":"2024-03-04T09:32:57.720259+0000","last_undegraded":"2024-03-04T09:32:57.720259+0000","last_fullsized":"2024-03-04T09:32:57.720259+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:48:55.639006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.688881+0000","last_change":"2024-03-04T09:32:56.703726+0000","last_active":"2024-03-04T09:32:57.688881+0000","last_peered":"2024-03-04T09:32:57.688881+0000","last_clean":"2024-03-04T09:32:57.688881+0000","last_became_active":"2024-03-04T09:32:56.703246+0000","last_became_peered":"2024-03-04T09:32:56.703246+0000","last_unstale":"2024-03-04T09:32:57.688881+0000","last_undegraded":"2024-03-04T09:32:57.688881+0000","last_fullsized":"2024-03-04T09:32:57.688881+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_clean_scrub_stamp":"2024-03-04T09:32:55.635455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:51:35.014307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.0","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.720553+0000","last_change":"2024-03-04T09:32:57.715500+0000","last_active":"2024-03-04T09:32:57.720553+0000","last_peered":"2024-03-04T09:32:57.720553+0000","last_clean":"2024-03-04T09:32:57.720553+0000","last_became_active":"2024-03-04T09:32:57.715059+0000","last_became_peered":"2024-03-04T09:32:57.715059+0000","last_unstale":"2024-03-04T09:32:57.720553+0000","last_undegraded":"2024-03-04T09:32:57.720553+0000","last_fullsized":"2024-03-04T09:32:57.720553+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:35:45.163868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.689400+0000","last_change":"2024-03-04T09:32:56.711325+0000","last_active":"2024-03-04T09:32:57.689400+0000","last_peered":"2024-03-04T09:32:57.689400+0000","last_clean":"2024-03-04T09:32:57.689400+0000","last_became_active":"2024-03-04T09:32:56.711002+0000","last_became_peered":"2024-03-04T09:32:56.711002+0000","last_unstale":"2024-03-04T09:32:57.689400+0000","last_undegraded":"2024-03-04T09:32:57.689400+0000","last_fullsized":"2024-03-04T09:32:57.689400+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_clean_scrub_stamp":"2024-03-04T09:32:55.635455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:57:48.422427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.1","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.700903+0000","last_change":"2024-03-04T09:32:57.695150+0000","last_active":"2024-03-04T09:32:57.700903+0000","last_peered":"2024-03-04T09:32:57.700903+0000","last_clean":"2024-03-04T09:32:57.700903+0000","last_became_active":"2024-03-04T09:32:57.694800+0000","last_became_peered":"2024-03-04T09:32:57.694800+0000","last_unstale":"2024-03-04T09:32:57.700903+0000","last_undegraded":"2024-03-04T09:32:57.700903+0000","last_fullsized":"2024-03-04T09:32:57.700903+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:30:14.957488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.698689+0000","last_change":"2024-03-04T09:32:56.702914+0000","last_active":"2024-03-04T09:32:57.698689+0000","last_peered":"2024-03-04T09:32:57.698689+0000","last_clean":"2024-03-04T09:32:57.698689+0000","last_became_active":"2024-03-04T09:32:56.702511+0000","last_became_peered":"2024-03-04T09:32:56.702511+0000","last_unstale":"2024-03-04T09:32:57.698689+0000","last_undegraded":"2024-03-04T09:32:57.698689+0000","last_fullsized":"2024-03-04T09:32:57.698689+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_clean_scrub_stamp":"2024-03-04T09:32:55.635455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:21:02.657585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.717293+0000","last_change":"2024-03-04T09:32:57.714955+0000","last_active":"2024-03-04T09:32:57.717293+0000","last_peered":"2024-03-04T09:32:57.717293+0000","last_clean":"2024-03-04T09:32:57.717293+0000","last_became_active":"2024-03-04T09:32:57.713559+0000","last_became_peered":"2024-03-04T09:32:57.713559+0000","last_unstale":"2024-03-04T09:32:57.717293+0000","last_undegraded":"2024-03-04T09:32:57.717293+0000","last_fullsized":"2024-03-04T09:32:57.717293+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:53:09.217232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"126'2","reported_seq":14,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.723579+0000","last_change":"2024-03-04T09:32:56.704329+0000","last_active":"2024-03-04T09:32:57.723579+0000","last_peered":"2024-03-04T09:32:57.723579+0000","last_clean":"2024-03-04T09:32:57.723579+0000","last_became_active":"2024-03-04T09:32:56.704009+0000","last_became_peered":"2024-03-04T09:32:56.704009+0000","last_unstale":"2024-03-04T09:32:57.723579+0000","last_undegraded":"2024-03-04T09:32:57.723579+0000","last_fullsized":"2024-03-04T09:32:57.723579+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_clean_scrub_stamp":"2024-03-04T09:32:55.635455+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:21:33.130606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":366,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.681372+0000","last_change":"2024-03-04T09:25:09.316159+0000","last_active":"2024-03-04T09:32:57.681372+0000","last_peered":"2024-03-04T09:32:57.681372+0000","last_clean":"2024-03-04T09:32:57.681372+0000","last_became_active":"2024-03-04T09:25:09.313767+0000","last_became_peered":"2024-03-04T09:25:09.313767+0000","last_unstale":"2024-03-04T09:32:57.681372+0000","last_undegraded":"2024-03-04T09:32:57.681372+0000","last_fullsized":"2024-03-04T09:32:57.681372+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:08.110953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:08.110953+0000","last_clean_scrub_stamp":"2024-03-04T09:25:08.110953+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:34:21.745390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":19,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":18,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574945,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37452,"kb_used_data":2104,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334388,"statfs":{"total":96636764160,"available":96598413312,"internally_reserved":0,"allocated":2154496,"data_stored":1268816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574945,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36868,"kb_used_data":1524,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334972,"statfs":{"total":96636764160,"available":96599011328,"internally_reserved":0,"allocated":1560576,"data_stored":678448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574947,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36884,"kb_used_data":1544,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334956,"statfs":{"total":96636764160,"available":96598994944,"internally_reserved":0,"allocated":1581056,"data_stored":685335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574944,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36884,"kb_used_data":1540,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334956,"statfs":{"total":96636764160,"available":96598994944,"internally_reserved":0,"allocated":1576960,"data_stored":685293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574946,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36872,"kb_used_data":1524,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334968,"statfs":{"total":96636764160,"available":96599007232,"internally_reserved":0,"allocated":1560576,"data_stored":678448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574945,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36868,"kb_used_data":1524,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334972,"statfs":{"total":96636764160,"available":96599011328,"internally_reserved":0,"allocated":1560576,"data_stored":678448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574946,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36868,"kb_used_data":1524,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334972,"statfs":{"total":96636764160,"available":96599011328,"internally_reserved":0,"allocated":1560576,"data_stored":678448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574945,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37460,"kb_used_data":2116,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334380,"statfs":{"total":96636764160,"available":96598405120,"internally_reserved":0,"allocated":2166784,"data_stored":1275591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-04T09:33:02.171 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T09:33:02.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.429+0000 7fc08a000640 1 Processor -- start 2024-03-04T09:33:02.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.429+0000 7fc08a000640 1 -- start start 2024-03-04T09:33:02.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.431+0000 7fc08a000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fc084070f10 0x7fc0840cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:02.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.431+0000 7fc08a000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0840714d0 con 0x7fc0840cce90 2024-03-04T09:33:02.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.431+0000 7fc08a000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0840717b0 con 0x7fc084145300 2024-03-04T09:33:02.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.431+0000 7fc08a000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0840cd300 con 0x7fc084070f10 2024-03-04T09:33:02.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.431+0000 7fc089600640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fc0840cce90 0x7fc08415c570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:58046/0 (socket says 172.21.15.6:58046) 2024-03-04T09:33:02.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.431+0000 7fc089600640 1 -- 172.21.15.6:0/3228460874 learned_addr learned my addr 172.21.15.6:0/3228460874 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:33:02.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.432+0000 7fc083e00640 1 --2- 172.21.15.6:0/3228460874 >> v2:172.21.15.146:3300/0 conn(0x7fc084070f10 0x7fc0840cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:02.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.432+0000 7fc083400640 1 -- 172.21.15.6:0/3228460874 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4282931186 0 0) 0x7fc0840714d0 con 0x7fc0840cce90 2024-03-04T09:33:02.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.432+0000 7fc083400640 1 -- 172.21.15.6:0/3228460874 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc06c003520 con 0x7fc0840cce90 2024-03-04T09:33:02.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.432+0000 7fc083400640 1 -- 172.21.15.6:0/3228460874 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1910568160 0 0) 0x7fc06c003520 con 0x7fc0840cce90 2024-03-04T09:33:02.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.432+0000 7fc083400640 1 -- 172.21.15.6:0/3228460874 >> v2:172.21.15.146:3300/0 conn(0x7fc084070f10 msgr2=0x7fc0840cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:33:02.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.432+0000 7fc083400640 1 --2- 172.21.15.6:0/3228460874 >> v2:172.21.15.146:3300/0 conn(0x7fc084070f10 0x7fc0840cc950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:02.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.432+0000 7fc083400640 1 -- 172.21.15.6:0/3228460874 >> v1:172.21.15.146:6789/0 conn(0x7fc084145300 legacy=0x7fc0840b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:02.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.432+0000 7fc083400640 1 -- 172.21.15.6:0/3228460874 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc084071a90 con 0x7fc0840cce90 2024-03-04T09:33:02.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.432+0000 7fc083400640 1 -- 172.21.15.6:0/3228460874 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7fc07c0019f0 con 0x7fc0840cce90 2024-03-04T09:33:02.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.432+0000 7fc083400640 1 -- 172.21.15.6:0/3228460874 >> v1:172.21.15.6:6789/0 conn(0x7fc0840cce90 legacy=0x7fc08415c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:02.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.432+0000 7fc083400640 1 --2- 172.21.15.6:0/3228460874 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc06c0039b0 0x7fc06c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:02.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.432+0000 7fc083400640 1 --2- 172.21.15.6:0/3228460874 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc06c004800 0x7fc06c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:02.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.432+0000 7fc083400640 1 --2- 172.21.15.6:0/3228460874 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc06c0051c0 0x7fc06c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:02.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.432+0000 7fc083400640 1 -- 172.21.15.6:0/3228460874 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0840cd300 con 0x7fc06c004800 2024-03-04T09:33:02.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.432+0000 7fc083400640 1 -- 172.21.15.6:0/3228460874 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc06c00c160 con 0x7fc06c0051c0 2024-03-04T09:33:02.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.432+0000 7fc083400640 1 -- 172.21.15.6:0/3228460874 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc06c00c360 con 0x7fc06c0039b0 2024-03-04T09:33:02.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.433+0000 7fc083e00640 1 --2- 172.21.15.6:0/3228460874 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc06c004800 0x7fc06c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:02.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.433+0000 7fc089600640 1 --2- 172.21.15.6:0/3228460874 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc06c0051c0 0x7fc06c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:02.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.433+0000 7fc088c00640 1 --2- 172.21.15.6:0/3228460874 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc06c0039b0 0x7fc06c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:02.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.433+0000 7fc083e00640 1 -- 172.21.15.6:0/3228460874 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc06c0039b0 msgr2=0x7fc06c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:33:02.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.433+0000 7fc083e00640 1 --2- 172.21.15.6:0/3228460874 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc06c0039b0 0x7fc06c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:02.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.433+0000 7fc083e00640 1 -- 172.21.15.6:0/3228460874 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc06c0051c0 msgr2=0x7fc06c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:33:02.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.433+0000 7fc083e00640 1 --2- 172.21.15.6:0/3228460874 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc06c0051c0 0x7fc06c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:02.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.433+0000 7fc083e00640 1 -- 172.21.15.6:0/3228460874 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc06c003520 con 0x7fc06c004800 2024-03-04T09:33:02.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.433+0000 7fc089600640 1 -- 172.21.15.6:0/3228460874 reap_dead start 2024-03-04T09:33:02.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.435+0000 7fc083e00640 1 --2- 172.21.15.6:0/3228460874 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc06c004800 0x7fc06c004c80 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7fc070009ac0 tx=0x7fc06c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=facfb990cc04aa52 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:33:02.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.435+0000 7fc083400640 1 -- 172.21.15.6:0/3228460874 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc070039070 con 0x7fc06c004800 2024-03-04T09:33:02.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.435+0000 7fc083400640 1 -- 172.21.15.6:0/3228460874 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc07002c6b0 con 0x7fc06c004800 2024-03-04T09:33:02.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.435+0000 7fc083400640 1 -- 172.21.15.6:0/3228460874 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc070009690 con 0x7fc06c004800 2024-03-04T09:33:02.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.435+0000 7fc08a000640 1 -- 172.21.15.6:0/3228460874 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc06c004800 msgr2=0x7fc06c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:02.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.435+0000 7fc08a000640 1 --2- 172.21.15.6:0/3228460874 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc06c004800 0x7fc06c004c80 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7fc070009ac0 tx=0x7fc06c005830 comp rx=0 tx=0).stop 2024-03-04T09:33:02.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.436+0000 7fc08a000640 1 -- 172.21.15.6:0/3228460874 shutdown_connections 2024-03-04T09:33:02.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.436+0000 7fc08a000640 1 --2- 172.21.15.6:0/3228460874 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc06c004800 0x7fc06c004c80 unknown :-1 s=CLOSED pgs=1501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:02.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.436+0000 7fc08a000640 1 -- 172.21.15.6:0/3228460874 >> 172.21.15.6:0/3228460874 conn(0x7fc08406b3d0 msgr2=0x7fc08406b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:33:02.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.436+0000 7fc08a000640 1 -- 172.21.15.6:0/3228460874 shutdown_connections 2024-03-04T09:33:02.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.436+0000 7fc08a000640 1 -- 172.21.15.6:0/3228460874 wait complete. 2024-03-04T09:33:02.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.437+0000 7fc08a000640 1 Processor -- start 2024-03-04T09:33:02.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.438+0000 7fc08a000640 1 -- start start 2024-03-04T09:33:02.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.438+0000 7fc08a000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc06c004800 0x7fc0841e4170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:02.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.438+0000 7fc08a000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc08415b2a0 0x7fc084134ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:02.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.439+0000 7fc08a000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc08415da20 0x7fc08415dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:02.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.439+0000 7fc08a000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0840cc950 con 0x7fc06c004800 2024-03-04T09:33:02.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.439+0000 7fc08a000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0841456e0 con 0x7fc08415da20 2024-03-04T09:33:02.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.439+0000 7fc08a000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0840ccad0 con 0x7fc08415b2a0 2024-03-04T09:33:02.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.439+0000 7fc088c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc06c004800 0x7fc0841e4170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:02.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.439+0000 7fc088c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc06c004800 0x7fc0841e4170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42986/0 (socket says 172.21.15.6:42986) 2024-03-04T09:33:02.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.439+0000 7fc088c00640 1 -- 172.21.15.6:0/675070209 learned_addr learned my addr 172.21.15.6:0/675070209 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:33:02.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.439+0000 7fc083e00640 1 --2- 172.21.15.6:0/675070209 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc08415b2a0 0x7fc084134ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:02.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.439+0000 7fc088c00640 1 -- 172.21.15.6:0/675070209 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc08415b2a0 msgr2=0x7fc084134ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:02.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.439+0000 7fc088c00640 1 --2- 172.21.15.6:0/675070209 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc08415b2a0 0x7fc084134ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:02.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.439+0000 7fc088c00640 1 -- 172.21.15.6:0/675070209 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc08415da20 msgr2=0x7fc08415dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:02.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.439+0000 7fc088c00640 1 --2- 172.21.15.6:0/675070209 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc08415da20 0x7fc08415dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:02.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.439+0000 7fc088c00640 1 -- 172.21.15.6:0/675070209 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc084135000 con 0x7fc06c004800 2024-03-04T09:33:02.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.439+0000 7fc088c00640 1 --2- 172.21.15.6:0/675070209 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc06c004800 0x7fc0841e4170 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7fc07800e4b0 tx=0x7fc07800e970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:33:02.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.440+0000 7fc080c00640 1 -- 172.21.15.6:0/675070209 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0780052e0 con 0x7fc06c004800 2024-03-04T09:33:02.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.440+0000 7fc080c00640 1 -- 172.21.15.6:0/675070209 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc078018070 con 0x7fc06c004800 2024-03-04T09:33:02.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.440+0000 7fc080c00640 1 -- 172.21.15.6:0/675070209 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc078003810 con 0x7fc06c004800 2024-03-04T09:33:02.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.440+0000 7fc08a000640 1 -- 172.21.15.6:0/675070209 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0841e47c0 con 0x7fc06c004800 2024-03-04T09:33:02.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.440+0000 7fc08a000640 1 -- 172.21.15.6:0/675070209 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0841e4d80 con 0x7fc06c004800 2024-03-04T09:33:02.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.441+0000 7fc08a000640 1 -- 172.21.15.6:0/675070209 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc08404f330 con 0x7fc06c004800 2024-03-04T09:33:02.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.442+0000 7fc080c00640 1 -- 172.21.15.6:0/675070209 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc0780039b0 con 0x7fc06c004800 2024-03-04T09:33:02.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.443+0000 7fc080c00640 1 --2- 172.21.15.6:0/675070209 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc06406b4d0 0x7fc06406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:02.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.443+0000 7fc080c00640 1 -- 172.21.15.6:0/675070209 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc078089e60 con 0x7fc06c004800 2024-03-04T09:33:02.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.444+0000 7fc083e00640 1 --2- 172.21.15.6:0/675070209 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc06406b4d0 0x7fc06406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:02.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.445+0000 7fc083e00640 1 --2- 172.21.15.6:0/675070209 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc06406b4d0 0x7fc06406d990 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fc070036000 tx=0x7fc070030000 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:33:02.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.447+0000 7fc080c00640 1 -- 172.21.15.6:0/675070209 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc078058cd0 con 0x7fc06c004800 2024-03-04T09:33:02.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.644+0000 7fc08a000640 1 -- 172.21.15.6:0/675070209 --> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc0840af9a0 con 0x7fc06406b4d0 2024-03-04T09:33:02.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.663+0000 7fc080c00640 1 -- 172.21.15.6:0/675070209 <== mgr.4113 v2:172.21.15.146:6832/1788613334 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252410 (secure 0 0 0) 0x7fc0840af9a0 con 0x7fc06406b4d0 2024-03-04T09:33:02.665 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:33:02.666 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-03-04T09:33:02.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.668+0000 7fc08a000640 1 -- 172.21.15.6:0/675070209 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc06406b4d0 msgr2=0x7fc06406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:02.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.668+0000 7fc08a000640 1 --2- 172.21.15.6:0/675070209 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc06406b4d0 0x7fc06406d990 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fc070036000 tx=0x7fc070030000 comp rx=0 tx=0).stop 2024-03-04T09:33:02.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.668+0000 7fc08a000640 1 -- 172.21.15.6:0/675070209 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc06c004800 msgr2=0x7fc0841e4170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:02.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.668+0000 7fc08a000640 1 --2- 172.21.15.6:0/675070209 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc06c004800 0x7fc0841e4170 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7fc07800e4b0 tx=0x7fc07800e970 comp rx=0 tx=0).stop 2024-03-04T09:33:02.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.668+0000 7fc08a000640 1 -- 172.21.15.6:0/675070209 shutdown_connections 2024-03-04T09:33:02.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.668+0000 7fc08a000640 1 --2- 172.21.15.6:0/675070209 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc08415da20 0x7fc08415dea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:02.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.668+0000 7fc08a000640 1 --2- 172.21.15.6:0/675070209 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc06406b4d0 0x7fc06406d990 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:02.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.668+0000 7fc08a000640 1 --2- 172.21.15.6:0/675070209 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc08415b2a0 0x7fc084134ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:02.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.668+0000 7fc08a000640 1 --2- 172.21.15.6:0/675070209 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc06c004800 0x7fc0841e4170 unknown :-1 s=CLOSED pgs=1502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:02.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.668+0000 7fc08a000640 1 -- 172.21.15.6:0/675070209 >> 172.21.15.6:0/675070209 conn(0x7fc08406b3d0 msgr2=0x7fc0840b28b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:33:02.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.669+0000 7fc08a000640 1 -- 172.21.15.6:0/675070209 shutdown_connections 2024-03-04T09:33:02.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.669+0000 7fc08a000640 1 -- 172.21.15.6:0/675070209 wait complete. 2024-03-04T09:33:02.686 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":357,"stamp":"2024-03-04T09:33:02.027680+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":33,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":296156,"kb_used_data":13400,"kb_used_omap":12,"kb_used_meta":282611,"kb_avail":754678564,"statfs":{"total":773094113280,"available":772790849536,"internally_reserved":0,"allocated":13721600,"data_stored":6628827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":289394224},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002258"},"pg_stats":[{"pgid":"19.3f","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.721729+0000","last_change":"2024-03-04T09:32:57.715295+0000","last_active":"2024-03-04T09:32:57.721729+0000","last_peered":"2024-03-04T09:32:57.721729+0000","last_clean":"2024-03-04T09:32:57.721729+0000","last_became_active":"2024-03-04T09:32:57.715032+0000","last_became_peered":"2024-03-04T09:32:57.715032+0000","last_unstale":"2024-03-04T09:32:57.721729+0000","last_undegraded":"2024-03-04T09:32:57.721729+0000","last_fullsized":"2024-03-04T09:32:57.721729+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:32:39.387015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3e","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.715179+0000","last_change":"2024-03-04T09:32:57.711681+0000","last_active":"2024-03-04T09:32:57.715179+0000","last_peered":"2024-03-04T09:32:57.715179+0000","last_clean":"2024-03-04T09:32:57.715179+0000","last_became_active":"2024-03-04T09:32:57.711140+0000","last_became_peered":"2024-03-04T09:32:57.711140+0000","last_unstale":"2024-03-04T09:32:57.715179+0000","last_undegraded":"2024-03-04T09:32:57.715179+0000","last_fullsized":"2024-03-04T09:32:57.715179+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:17:22.364556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3d","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.706232+0000","last_change":"2024-03-04T09:32:57.700626+0000","last_active":"2024-03-04T09:32:57.706232+0000","last_peered":"2024-03-04T09:32:57.706232+0000","last_clean":"2024-03-04T09:32:57.706232+0000","last_became_active":"2024-03-04T09:32:57.699525+0000","last_became_peered":"2024-03-04T09:32:57.699525+0000","last_unstale":"2024-03-04T09:32:57.706232+0000","last_undegraded":"2024-03-04T09:32:57.706232+0000","last_fullsized":"2024-03-04T09:32:57.706232+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:34:58.565938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.3c","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.721446+0000","last_change":"2024-03-04T09:32:57.713538+0000","last_active":"2024-03-04T09:32:57.721446+0000","last_peered":"2024-03-04T09:32:57.721446+0000","last_clean":"2024-03-04T09:32:57.721446+0000","last_became_active":"2024-03-04T09:32:57.713273+0000","last_became_peered":"2024-03-04T09:32:57.713273+0000","last_unstale":"2024-03-04T09:32:57.721446+0000","last_undegraded":"2024-03-04T09:32:57.721446+0000","last_fullsized":"2024-03-04T09:32:57.721446+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:08:36.129294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3b","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.711514+0000","last_change":"2024-03-04T09:32:57.706038+0000","last_active":"2024-03-04T09:32:57.711514+0000","last_peered":"2024-03-04T09:32:57.711514+0000","last_clean":"2024-03-04T09:32:57.711514+0000","last_became_active":"2024-03-04T09:32:57.705742+0000","last_became_peered":"2024-03-04T09:32:57.705742+0000","last_unstale":"2024-03-04T09:32:57.711514+0000","last_undegraded":"2024-03-04T09:32:57.711514+0000","last_fullsized":"2024-03-04T09:32:57.711514+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:49:58.132664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.3a","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.722955+0000","last_change":"2024-03-04T09:32:57.719201+0000","last_active":"2024-03-04T09:32:57.722955+0000","last_peered":"2024-03-04T09:32:57.722955+0000","last_clean":"2024-03-04T09:32:57.722955+0000","last_became_active":"2024-03-04T09:32:57.718866+0000","last_became_peered":"2024-03-04T09:32:57.718866+0000","last_unstale":"2024-03-04T09:32:57.722955+0000","last_undegraded":"2024-03-04T09:32:57.722955+0000","last_fullsized":"2024-03-04T09:32:57.722955+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:56:31.981113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.39","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.718264+0000","last_change":"2024-03-04T09:32:57.715088+0000","last_active":"2024-03-04T09:32:57.718264+0000","last_peered":"2024-03-04T09:32:57.718264+0000","last_clean":"2024-03-04T09:32:57.718264+0000","last_became_active":"2024-03-04T09:32:57.712371+0000","last_became_peered":"2024-03-04T09:32:57.712371+0000","last_unstale":"2024-03-04T09:32:57.718264+0000","last_undegraded":"2024-03-04T09:32:57.718264+0000","last_fullsized":"2024-03-04T09:32:57.718264+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:34:31.386306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.38","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.714668+0000","last_change":"2024-03-04T09:32:57.703017+0000","last_active":"2024-03-04T09:32:57.714668+0000","last_peered":"2024-03-04T09:32:57.714668+0000","last_clean":"2024-03-04T09:32:57.714668+0000","last_became_active":"2024-03-04T09:32:57.702395+0000","last_became_peered":"2024-03-04T09:32:57.702395+0000","last_unstale":"2024-03-04T09:32:57.714668+0000","last_undegraded":"2024-03-04T09:32:57.714668+0000","last_fullsized":"2024-03-04T09:32:57.714668+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:37:21.506217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.37","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.713108+0000","last_change":"2024-03-04T09:32:57.703315+0000","last_active":"2024-03-04T09:32:57.713108+0000","last_peered":"2024-03-04T09:32:57.713108+0000","last_clean":"2024-03-04T09:32:57.713108+0000","last_became_active":"2024-03-04T09:32:57.703001+0000","last_became_peered":"2024-03-04T09:32:57.703001+0000","last_unstale":"2024-03-04T09:32:57.713108+0000","last_undegraded":"2024-03-04T09:32:57.713108+0000","last_fullsized":"2024-03-04T09:32:57.713108+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:30:14.957488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.36","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.715157+0000","last_change":"2024-03-04T09:32:57.711439+0000","last_active":"2024-03-04T09:32:57.715157+0000","last_peered":"2024-03-04T09:32:57.715157+0000","last_clean":"2024-03-04T09:32:57.715157+0000","last_became_active":"2024-03-04T09:32:57.710733+0000","last_became_peered":"2024-03-04T09:32:57.710733+0000","last_unstale":"2024-03-04T09:32:57.715157+0000","last_undegraded":"2024-03-04T09:32:57.715157+0000","last_fullsized":"2024-03-04T09:32:57.715157+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:01:47.019520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.35","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.712514+0000","last_change":"2024-03-04T09:32:57.705947+0000","last_active":"2024-03-04T09:32:57.712514+0000","last_peered":"2024-03-04T09:32:57.712514+0000","last_clean":"2024-03-04T09:32:57.712514+0000","last_became_active":"2024-03-04T09:32:57.705648+0000","last_became_peered":"2024-03-04T09:32:57.705648+0000","last_unstale":"2024-03-04T09:32:57.712514+0000","last_undegraded":"2024-03-04T09:32:57.712514+0000","last_fullsized":"2024-03-04T09:32:57.712514+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:52:30.281316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.34","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.712136+0000","last_change":"2024-03-04T09:32:57.705170+0000","last_active":"2024-03-04T09:32:57.712136+0000","last_peered":"2024-03-04T09:32:57.712136+0000","last_clean":"2024-03-04T09:32:57.712136+0000","last_became_active":"2024-03-04T09:32:57.703175+0000","last_became_peered":"2024-03-04T09:32:57.703175+0000","last_unstale":"2024-03-04T09:32:57.712136+0000","last_undegraded":"2024-03-04T09:32:57.712136+0000","last_fullsized":"2024-03-04T09:32:57.712136+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:35:24.731959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.33","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.724986+0000","last_change":"2024-03-04T09:32:57.718409+0000","last_active":"2024-03-04T09:32:57.724986+0000","last_peered":"2024-03-04T09:32:57.724986+0000","last_clean":"2024-03-04T09:32:57.724986+0000","last_became_active":"2024-03-04T09:32:57.717404+0000","last_became_peered":"2024-03-04T09:32:57.717404+0000","last_unstale":"2024-03-04T09:32:57.724986+0000","last_undegraded":"2024-03-04T09:32:57.724986+0000","last_fullsized":"2024-03-04T09:32:57.724986+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:32:42.841675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.32","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.716694+0000","last_change":"2024-03-04T09:32:57.714535+0000","last_active":"2024-03-04T09:32:57.716694+0000","last_peered":"2024-03-04T09:32:57.716694+0000","last_clean":"2024-03-04T09:32:57.716694+0000","last_became_active":"2024-03-04T09:32:57.714251+0000","last_became_peered":"2024-03-04T09:32:57.714251+0000","last_unstale":"2024-03-04T09:32:57.716694+0000","last_undegraded":"2024-03-04T09:32:57.716694+0000","last_fullsized":"2024-03-04T09:32:57.716694+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:11:04.209303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.31","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.715040+0000","last_change":"2024-03-04T09:32:57.703832+0000","last_active":"2024-03-04T09:32:57.715040+0000","last_peered":"2024-03-04T09:32:57.715040+0000","last_clean":"2024-03-04T09:32:57.715040+0000","last_became_active":"2024-03-04T09:32:57.703555+0000","last_became_peered":"2024-03-04T09:32:57.703555+0000","last_unstale":"2024-03-04T09:32:57.715040+0000","last_undegraded":"2024-03-04T09:32:57.715040+0000","last_fullsized":"2024-03-04T09:32:57.715040+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:09:13.281054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.30","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.722113+0000","last_change":"2024-03-04T09:32:57.719691+0000","last_active":"2024-03-04T09:32:57.722113+0000","last_peered":"2024-03-04T09:32:57.722113+0000","last_clean":"2024-03-04T09:32:57.722113+0000","last_became_active":"2024-03-04T09:32:57.719398+0000","last_became_peered":"2024-03-04T09:32:57.719398+0000","last_unstale":"2024-03-04T09:32:57.722113+0000","last_undegraded":"2024-03-04T09:32:57.722113+0000","last_fullsized":"2024-03-04T09:32:57.722113+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:01:51.410127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2f","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.716157+0000","last_change":"2024-03-04T09:32:57.709718+0000","last_active":"2024-03-04T09:32:57.716157+0000","last_peered":"2024-03-04T09:32:57.716157+0000","last_clean":"2024-03-04T09:32:57.716157+0000","last_became_active":"2024-03-04T09:32:57.709375+0000","last_became_peered":"2024-03-04T09:32:57.709375+0000","last_unstale":"2024-03-04T09:32:57.716157+0000","last_undegraded":"2024-03-04T09:32:57.716157+0000","last_fullsized":"2024-03-04T09:32:57.716157+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:18:50.341054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.2e","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.713699+0000","last_change":"2024-03-04T09:32:57.710074+0000","last_active":"2024-03-04T09:32:57.713699+0000","last_peered":"2024-03-04T09:32:57.713699+0000","last_clean":"2024-03-04T09:32:57.713699+0000","last_became_active":"2024-03-04T09:32:57.709743+0000","last_became_peered":"2024-03-04T09:32:57.709743+0000","last_unstale":"2024-03-04T09:32:57.713699+0000","last_undegraded":"2024-03-04T09:32:57.713699+0000","last_fullsized":"2024-03-04T09:32:57.713699+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:24:22.403863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2d","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.718698+0000","last_change":"2024-03-04T09:32:57.713824+0000","last_active":"2024-03-04T09:32:57.718698+0000","last_peered":"2024-03-04T09:32:57.718698+0000","last_clean":"2024-03-04T09:32:57.718698+0000","last_became_active":"2024-03-04T09:32:57.712734+0000","last_became_peered":"2024-03-04T09:32:57.712734+0000","last_unstale":"2024-03-04T09:32:57.718698+0000","last_undegraded":"2024-03-04T09:32:57.718698+0000","last_fullsized":"2024-03-04T09:32:57.718698+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:37:52.957478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2c","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.705646+0000","last_change":"2024-03-04T09:32:57.703922+0000","last_active":"2024-03-04T09:32:57.705646+0000","last_peered":"2024-03-04T09:32:57.705646+0000","last_clean":"2024-03-04T09:32:57.705646+0000","last_became_active":"2024-03-04T09:32:57.702523+0000","last_became_peered":"2024-03-04T09:32:57.702523+0000","last_unstale":"2024-03-04T09:32:57.705646+0000","last_undegraded":"2024-03-04T09:32:57.705646+0000","last_fullsized":"2024-03-04T09:32:57.705646+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:02:31.190563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2b","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.719988+0000","last_change":"2024-03-04T09:32:57.716019+0000","last_active":"2024-03-04T09:32:57.719988+0000","last_peered":"2024-03-04T09:32:57.719988+0000","last_clean":"2024-03-04T09:32:57.719988+0000","last_became_active":"2024-03-04T09:32:57.715748+0000","last_became_peered":"2024-03-04T09:32:57.715748+0000","last_unstale":"2024-03-04T09:32:57.719988+0000","last_undegraded":"2024-03-04T09:32:57.719988+0000","last_fullsized":"2024-03-04T09:32:57.719988+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:01:34.747229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.2a","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.706758+0000","last_change":"2024-03-04T09:32:57.704594+0000","last_active":"2024-03-04T09:32:57.706758+0000","last_peered":"2024-03-04T09:32:57.706758+0000","last_clean":"2024-03-04T09:32:57.706758+0000","last_became_active":"2024-03-04T09:32:57.704238+0000","last_became_peered":"2024-03-04T09:32:57.704238+0000","last_unstale":"2024-03-04T09:32:57.706758+0000","last_undegraded":"2024-03-04T09:32:57.706758+0000","last_fullsized":"2024-03-04T09:32:57.706758+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:12:07.748148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.29","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.711323+0000","last_change":"2024-03-04T09:32:57.704311+0000","last_active":"2024-03-04T09:32:57.711323+0000","last_peered":"2024-03-04T09:32:57.711323+0000","last_clean":"2024-03-04T09:32:57.711323+0000","last_became_active":"2024-03-04T09:32:57.703704+0000","last_became_peered":"2024-03-04T09:32:57.703704+0000","last_unstale":"2024-03-04T09:32:57.711323+0000","last_undegraded":"2024-03-04T09:32:57.711323+0000","last_fullsized":"2024-03-04T09:32:57.711323+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:48:55.639006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.28","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.708882+0000","last_change":"2024-03-04T09:32:57.705889+0000","last_active":"2024-03-04T09:32:57.708882+0000","last_peered":"2024-03-04T09:32:57.708882+0000","last_clean":"2024-03-04T09:32:57.708882+0000","last_became_active":"2024-03-04T09:32:57.704468+0000","last_became_peered":"2024-03-04T09:32:57.704468+0000","last_unstale":"2024-03-04T09:32:57.708882+0000","last_undegraded":"2024-03-04T09:32:57.708882+0000","last_fullsized":"2024-03-04T09:32:57.708882+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:06:44.847594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.27","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.698762+0000","last_change":"2024-03-04T09:32:57.691041+0000","last_active":"2024-03-04T09:32:57.698762+0000","last_peered":"2024-03-04T09:32:57.698762+0000","last_clean":"2024-03-04T09:32:57.698762+0000","last_became_active":"2024-03-04T09:32:57.690660+0000","last_became_peered":"2024-03-04T09:32:57.690660+0000","last_unstale":"2024-03-04T09:32:57.698762+0000","last_undegraded":"2024-03-04T09:32:57.698762+0000","last_fullsized":"2024-03-04T09:32:57.698762+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:34:58.565938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.26","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.722283+0000","last_change":"2024-03-04T09:32:57.716816+0000","last_active":"2024-03-04T09:32:57.722283+0000","last_peered":"2024-03-04T09:32:57.722283+0000","last_clean":"2024-03-04T09:32:57.722283+0000","last_became_active":"2024-03-04T09:32:57.716493+0000","last_became_peered":"2024-03-04T09:32:57.716493+0000","last_unstale":"2024-03-04T09:32:57.722283+0000","last_undegraded":"2024-03-04T09:32:57.722283+0000","last_fullsized":"2024-03-04T09:32:57.722283+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:04:04.761815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.25","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.725322+0000","last_change":"2024-03-04T09:32:57.721163+0000","last_active":"2024-03-04T09:32:57.725322+0000","last_peered":"2024-03-04T09:32:57.725322+0000","last_clean":"2024-03-04T09:32:57.725322+0000","last_became_active":"2024-03-04T09:32:57.720853+0000","last_became_peered":"2024-03-04T09:32:57.720853+0000","last_unstale":"2024-03-04T09:32:57.725322+0000","last_undegraded":"2024-03-04T09:32:57.725322+0000","last_fullsized":"2024-03-04T09:32:57.725322+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:44:01.996743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.24","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.720227+0000","last_change":"2024-03-04T09:32:57.715598+0000","last_active":"2024-03-04T09:32:57.720227+0000","last_peered":"2024-03-04T09:32:57.720227+0000","last_clean":"2024-03-04T09:32:57.720227+0000","last_became_active":"2024-03-04T09:32:57.713046+0000","last_became_peered":"2024-03-04T09:32:57.713046+0000","last_unstale":"2024-03-04T09:32:57.720227+0000","last_undegraded":"2024-03-04T09:32:57.720227+0000","last_fullsized":"2024-03-04T09:32:57.720227+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:30:14.957488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.23","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.721832+0000","last_change":"2024-03-04T09:32:57.712616+0000","last_active":"2024-03-04T09:32:57.721832+0000","last_peered":"2024-03-04T09:32:57.721832+0000","last_clean":"2024-03-04T09:32:57.721832+0000","last_became_active":"2024-03-04T09:32:57.712299+0000","last_became_peered":"2024-03-04T09:32:57.712299+0000","last_unstale":"2024-03-04T09:32:57.721832+0000","last_undegraded":"2024-03-04T09:32:57.721832+0000","last_fullsized":"2024-03-04T09:32:57.721832+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:20:41.807104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.22","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.724612+0000","last_change":"2024-03-04T09:32:57.710547+0000","last_active":"2024-03-04T09:32:57.724612+0000","last_peered":"2024-03-04T09:32:57.724612+0000","last_clean":"2024-03-04T09:32:57.724612+0000","last_became_active":"2024-03-04T09:32:57.709304+0000","last_became_peered":"2024-03-04T09:32:57.709304+0000","last_unstale":"2024-03-04T09:32:57.724612+0000","last_undegraded":"2024-03-04T09:32:57.724612+0000","last_fullsized":"2024-03-04T09:32:57.724612+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:26:03.089663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.21","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.713005+0000","last_change":"2024-03-04T09:32:57.710484+0000","last_active":"2024-03-04T09:32:57.713005+0000","last_peered":"2024-03-04T09:32:57.713005+0000","last_clean":"2024-03-04T09:32:57.713005+0000","last_became_active":"2024-03-04T09:32:57.709988+0000","last_became_peered":"2024-03-04T09:32:57.709988+0000","last_unstale":"2024-03-04T09:32:57.713005+0000","last_undegraded":"2024-03-04T09:32:57.713005+0000","last_fullsized":"2024-03-04T09:32:57.713005+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:57:49.438902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.20","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.715227+0000","last_change":"2024-03-04T09:32:57.710330+0000","last_active":"2024-03-04T09:32:57.715227+0000","last_peered":"2024-03-04T09:32:57.715227+0000","last_clean":"2024-03-04T09:32:57.715227+0000","last_became_active":"2024-03-04T09:32:57.706308+0000","last_became_peered":"2024-03-04T09:32:57.706308+0000","last_unstale":"2024-03-04T09:32:57.715227+0000","last_undegraded":"2024-03-04T09:32:57.715227+0000","last_fullsized":"2024-03-04T09:32:57.715227+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:01:15.944370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1e","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.722304+0000","last_change":"2024-03-04T09:32:57.719861+0000","last_active":"2024-03-04T09:32:57.722304+0000","last_peered":"2024-03-04T09:32:57.722304+0000","last_clean":"2024-03-04T09:32:57.722304+0000","last_became_active":"2024-03-04T09:32:57.719568+0000","last_became_peered":"2024-03-04T09:32:57.719568+0000","last_unstale":"2024-03-04T09:32:57.722304+0000","last_undegraded":"2024-03-04T09:32:57.722304+0000","last_fullsized":"2024-03-04T09:32:57.722304+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:20:03.748890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1f","version":"126'4","reported_seq":16,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.725664+0000","last_change":"2024-03-04T09:32:56.708356+0000","last_active":"2024-03-04T09:32:57.725664+0000","last_peered":"2024-03-04T09:32:57.725664+0000","last_clean":"2024-03-04T09:32:57.725664+0000","last_became_active":"2024-03-04T09:32:56.708018+0000","last_became_peered":"2024-03-04T09:32:56.708018+0000","last_unstale":"2024-03-04T09:32:57.725664+0000","last_undegraded":"2024-03-04T09:32:57.725664+0000","last_fullsized":"2024-03-04T09:32:57.725664+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_clean_scrub_stamp":"2024-03-04T09:32:55.635455+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:34:30.369830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.1f","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.721154+0000","last_change":"2024-03-04T09:32:57.718404+0000","last_active":"2024-03-04T09:32:57.721154+0000","last_peered":"2024-03-04T09:32:57.721154+0000","last_clean":"2024-03-04T09:32:57.721154+0000","last_became_active":"2024-03-04T09:32:57.718124+0000","last_became_peered":"2024-03-04T09:32:57.718124+0000","last_unstale":"2024-03-04T09:32:57.721154+0000","last_undegraded":"2024-03-04T09:32:57.721154+0000","last_fullsized":"2024-03-04T09:32:57.721154+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:17:22.364556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1e","version":"126'1","reported_seq":13,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.722589+0000","last_change":"2024-03-04T09:32:56.698608+0000","last_active":"2024-03-04T09:32:57.722589+0000","last_peered":"2024-03-04T09:32:57.722589+0000","last_clean":"2024-03-04T09:32:57.722589+0000","last_became_active":"2024-03-04T09:32:56.696608+0000","last_became_peered":"2024-03-04T09:32:56.696608+0000","last_unstale":"2024-03-04T09:32:57.722589+0000","last_undegraded":"2024-03-04T09:32:57.722589+0000","last_fullsized":"2024-03-04T09:32:57.722589+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_clean_scrub_stamp":"2024-03-04T09:32:55.635455+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:34:26.054015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.1c","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.712110+0000","last_change":"2024-03-04T09:32:57.707787+0000","last_active":"2024-03-04T09:32:57.712110+0000","last_peered":"2024-03-04T09:32:57.712110+0000","last_clean":"2024-03-04T09:32:57.712110+0000","last_became_active":"2024-03-04T09:32:57.704504+0000","last_became_peered":"2024-03-04T09:32:57.704504+0000","last_unstale":"2024-03-04T09:32:57.712110+0000","last_undegraded":"2024-03-04T09:32:57.712110+0000","last_fullsized":"2024-03-04T09:32:57.712110+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:36:29.166241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1d","version":"126'2","reported_seq":14,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.723238+0000","last_change":"2024-03-04T09:32:56.691446+0000","last_active":"2024-03-04T09:32:57.723238+0000","last_peered":"2024-03-04T09:32:57.723238+0000","last_clean":"2024-03-04T09:32:57.723238+0000","last_became_active":"2024-03-04T09:32:56.691095+0000","last_became_peered":"2024-03-04T09:32:56.691095+0000","last_unstale":"2024-03-04T09:32:57.723238+0000","last_undegraded":"2024-03-04T09:32:57.723238+0000","last_fullsized":"2024-03-04T09:32:57.723238+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_clean_scrub_stamp":"2024-03-04T09:32:55.635455+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:49:40.752903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.1d","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.704200+0000","last_change":"2024-03-04T09:32:57.702688+0000","last_active":"2024-03-04T09:32:57.704200+0000","last_peered":"2024-03-04T09:32:57.704200+0000","last_clean":"2024-03-04T09:32:57.704200+0000","last_became_active":"2024-03-04T09:32:57.700913+0000","last_became_peered":"2024-03-04T09:32:57.700913+0000","last_unstale":"2024-03-04T09:32:57.704200+0000","last_undegraded":"2024-03-04T09:32:57.704200+0000","last_fullsized":"2024-03-04T09:32:57.704200+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:32:26.208985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1c","version":"126'2","reported_seq":14,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.698370+0000","last_change":"2024-03-04T09:32:56.682630+0000","last_active":"2024-03-04T09:32:57.698370+0000","last_peered":"2024-03-04T09:32:57.698370+0000","last_clean":"2024-03-04T09:32:57.698370+0000","last_became_active":"2024-03-04T09:32:56.681263+0000","last_became_peered":"2024-03-04T09:32:56.681263+0000","last_unstale":"2024-03-04T09:32:57.698370+0000","last_undegraded":"2024-03-04T09:32:57.698370+0000","last_fullsized":"2024-03-04T09:32:57.698370+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_clean_scrub_stamp":"2024-03-04T09:32:55.635455+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:04:08.822065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1a","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.722709+0000","last_change":"2024-03-04T09:32:57.716596+0000","last_active":"2024-03-04T09:32:57.722709+0000","last_peered":"2024-03-04T09:32:57.722709+0000","last_clean":"2024-03-04T09:32:57.722709+0000","last_became_active":"2024-03-04T09:32:57.715856+0000","last_became_peered":"2024-03-04T09:32:57.715856+0000","last_unstale":"2024-03-04T09:32:57.722709+0000","last_undegraded":"2024-03-04T09:32:57.722709+0000","last_fullsized":"2024-03-04T09:32:57.722709+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:04:33.104550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1b","version":"126'1","reported_seq":13,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.700303+0000","last_change":"2024-03-04T09:32:56.691411+0000","last_active":"2024-03-04T09:32:57.700303+0000","last_peered":"2024-03-04T09:32:57.700303+0000","last_clean":"2024-03-04T09:32:57.700303+0000","last_became_active":"2024-03-04T09:32:56.691109+0000","last_became_peered":"2024-03-04T09:32:56.691109+0000","last_unstale":"2024-03-04T09:32:57.700303+0000","last_undegraded":"2024-03-04T09:32:57.700303+0000","last_fullsized":"2024-03-04T09:32:57.700303+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_clean_scrub_stamp":"2024-03-04T09:32:55.635455+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:04:14.387131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1b","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.717787+0000","last_change":"2024-03-04T09:32:57.711413+0000","last_active":"2024-03-04T09:32:57.717787+0000","last_peered":"2024-03-04T09:32:57.717787+0000","last_clean":"2024-03-04T09:32:57.717787+0000","last_became_active":"2024-03-04T09:32:57.711128+0000","last_became_peered":"2024-03-04T09:32:57.711128+0000","last_unstale":"2024-03-04T09:32:57.717787+0000","last_undegraded":"2024-03-04T09:32:57.717787+0000","last_fullsized":"2024-03-04T09:32:57.717787+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:21:34.147081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.1a","version":"126'1","reported_seq":13,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.701897+0000","last_change":"2024-03-04T09:32:56.693988+0000","last_active":"2024-03-04T09:32:57.701897+0000","last_peered":"2024-03-04T09:32:57.701897+0000","last_clean":"2024-03-04T09:32:57.701897+0000","last_became_active":"2024-03-04T09:32:56.693521+0000","last_became_peered":"2024-03-04T09:32:56.693521+0000","last_unstale":"2024-03-04T09:32:57.701897+0000","last_undegraded":"2024-03-04T09:32:57.701897+0000","last_fullsized":"2024-03-04T09:32:57.701897+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_clean_scrub_stamp":"2024-03-04T09:32:55.635455+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:53:08.200757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.18","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.714647+0000","last_change":"2024-03-04T09:32:57.702755+0000","last_active":"2024-03-04T09:32:57.714647+0000","last_peered":"2024-03-04T09:32:57.714647+0000","last_clean":"2024-03-04T09:32:57.714647+0000","last_became_active":"2024-03-04T09:32:57.702038+0000","last_became_peered":"2024-03-04T09:32:57.702038+0000","last_unstale":"2024-03-04T09:32:57.714647+0000","last_undegraded":"2024-03-04T09:32:57.714647+0000","last_fullsized":"2024-03-04T09:32:57.714647+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:12:39.009902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.696328+0000","last_change":"2024-03-04T09:32:56.707512+0000","last_active":"2024-03-04T09:32:57.696328+0000","last_peered":"2024-03-04T09:32:57.696328+0000","last_clean":"2024-03-04T09:32:57.696328+0000","last_became_active":"2024-03-04T09:32:56.707061+0000","last_became_peered":"2024-03-04T09:32:56.707061+0000","last_unstale":"2024-03-04T09:32:57.696328+0000","last_undegraded":"2024-03-04T09:32:57.696328+0000","last_fullsized":"2024-03-04T09:32:57.696328+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_clean_scrub_stamp":"2024-03-04T09:32:55.635455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:01:28.738130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.702797+0000","last_change":"2024-03-04T09:32:57.699075+0000","last_active":"2024-03-04T09:32:57.702797+0000","last_peered":"2024-03-04T09:32:57.702797+0000","last_clean":"2024-03-04T09:32:57.702797+0000","last_became_active":"2024-03-04T09:32:57.698684+0000","last_became_peered":"2024-03-04T09:32:57.698684+0000","last_unstale":"2024-03-04T09:32:57.702797+0000","last_undegraded":"2024-03-04T09:32:57.702797+0000","last_fullsized":"2024-03-04T09:32:57.702797+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:35:24.731959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.677928+0000","last_change":"2024-03-04T09:32:56.706704+0000","last_active":"2024-03-04T09:32:57.677928+0000","last_peered":"2024-03-04T09:32:57.677928+0000","last_clean":"2024-03-04T09:32:57.677928+0000","last_became_active":"2024-03-04T09:32:56.706388+0000","last_became_peered":"2024-03-04T09:32:56.706388+0000","last_unstale":"2024-03-04T09:32:57.677928+0000","last_undegraded":"2024-03-04T09:32:57.677928+0000","last_fullsized":"2024-03-04T09:32:57.677928+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_clean_scrub_stamp":"2024-03-04T09:32:55.635455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:36:40.943782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.16","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.725412+0000","last_change":"2024-03-04T09:32:57.722441+0000","last_active":"2024-03-04T09:32:57.725412+0000","last_peered":"2024-03-04T09:32:57.725412+0000","last_clean":"2024-03-04T09:32:57.725412+0000","last_became_active":"2024-03-04T09:32:57.722175+0000","last_became_peered":"2024-03-04T09:32:57.722175+0000","last_unstale":"2024-03-04T09:32:57.725412+0000","last_undegraded":"2024-03-04T09:32:57.725412+0000","last_fullsized":"2024-03-04T09:32:57.725412+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:58:22.075534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.713459+0000","last_change":"2024-03-04T09:32:56.699354+0000","last_active":"2024-03-04T09:32:57.713459+0000","last_peered":"2024-03-04T09:32:57.713459+0000","last_clean":"2024-03-04T09:32:57.713459+0000","last_became_active":"2024-03-04T09:32:56.698960+0000","last_became_peered":"2024-03-04T09:32:56.698960+0000","last_unstale":"2024-03-04T09:32:57.713459+0000","last_undegraded":"2024-03-04T09:32:57.713459+0000","last_fullsized":"2024-03-04T09:32:57.713459+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_clean_scrub_stamp":"2024-03-04T09:32:55.635455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:07:06.052094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.17","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.719755+0000","last_change":"2024-03-04T09:32:57.713613+0000","last_active":"2024-03-04T09:32:57.719755+0000","last_peered":"2024-03-04T09:32:57.719755+0000","last_clean":"2024-03-04T09:32:57.719755+0000","last_became_active":"2024-03-04T09:32:57.710733+0000","last_became_peered":"2024-03-04T09:32:57.710733+0000","last_unstale":"2024-03-04T09:32:57.719755+0000","last_undegraded":"2024-03-04T09:32:57.719755+0000","last_fullsized":"2024-03-04T09:32:57.719755+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:12:39.009902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.694899+0000","last_change":"2024-03-04T09:32:56.710490+0000","last_active":"2024-03-04T09:32:57.694899+0000","last_peered":"2024-03-04T09:32:57.694899+0000","last_clean":"2024-03-04T09:32:57.694899+0000","last_became_active":"2024-03-04T09:32:56.710232+0000","last_became_peered":"2024-03-04T09:32:56.710232+0000","last_unstale":"2024-03-04T09:32:57.694899+0000","last_undegraded":"2024-03-04T09:32:57.694899+0000","last_fullsized":"2024-03-04T09:32:57.694899+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_clean_scrub_stamp":"2024-03-04T09:32:55.635455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:00:59.755527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.14","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.710612+0000","last_change":"2024-03-04T09:32:57.703977+0000","last_active":"2024-03-04T09:32:57.710612+0000","last_peered":"2024-03-04T09:32:57.710612+0000","last_clean":"2024-03-04T09:32:57.710612+0000","last_became_active":"2024-03-04T09:32:57.703018+0000","last_became_peered":"2024-03-04T09:32:57.703018+0000","last_unstale":"2024-03-04T09:32:57.710612+0000","last_undegraded":"2024-03-04T09:32:57.710612+0000","last_fullsized":"2024-03-04T09:32:57.710612+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:56:31.981113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.15","version":"126'3","reported_seq":15,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.721358+0000","last_change":"2024-03-04T09:32:56.697367+0000","last_active":"2024-03-04T09:32:57.721358+0000","last_peered":"2024-03-04T09:32:57.721358+0000","last_clean":"2024-03-04T09:32:57.721358+0000","last_became_active":"2024-03-04T09:32:56.696977+0000","last_became_peered":"2024-03-04T09:32:56.696977+0000","last_unstale":"2024-03-04T09:32:57.721358+0000","last_undegraded":"2024-03-04T09:32:57.721358+0000","last_fullsized":"2024-03-04T09:32:57.721358+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_clean_scrub_stamp":"2024-03-04T09:32:55.635455+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:36:40.943782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.15","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.712859+0000","last_change":"2024-03-04T09:32:57.709156+0000","last_active":"2024-03-04T09:32:57.712859+0000","last_peered":"2024-03-04T09:32:57.712859+0000","last_clean":"2024-03-04T09:32:57.712859+0000","last_became_active":"2024-03-04T09:32:57.708841+0000","last_became_peered":"2024-03-04T09:32:57.708841+0000","last_unstale":"2024-03-04T09:32:57.712859+0000","last_undegraded":"2024-03-04T09:32:57.712859+0000","last_fullsized":"2024-03-04T09:32:57.712859+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:21:50.465577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.14","version":"126'6","reported_seq":18,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.723698+0000","last_change":"2024-03-04T09:32:56.709561+0000","last_active":"2024-03-04T09:32:57.723698+0000","last_peered":"2024-03-04T09:32:57.723698+0000","last_clean":"2024-03-04T09:32:57.723698+0000","last_became_active":"2024-03-04T09:32:56.708663+0000","last_became_peered":"2024-03-04T09:32:56.708663+0000","last_unstale":"2024-03-04T09:32:57.723698+0000","last_undegraded":"2024-03-04T09:32:57.723698+0000","last_fullsized":"2024-03-04T09:32:57.723698+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_clean_scrub_stamp":"2024-03-04T09:32:55.635455+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:04:08.822065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.13","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.721811+0000","last_change":"2024-03-04T09:32:57.711681+0000","last_active":"2024-03-04T09:32:57.721811+0000","last_peered":"2024-03-04T09:32:57.721811+0000","last_clean":"2024-03-04T09:32:57.721811+0000","last_became_active":"2024-03-04T09:32:57.711203+0000","last_became_peered":"2024-03-04T09:32:57.711203+0000","last_unstale":"2024-03-04T09:32:57.721811+0000","last_undegraded":"2024-03-04T09:32:57.721811+0000","last_fullsized":"2024-03-04T09:32:57.721811+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:38:06.426382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.709114+0000","last_change":"2024-03-04T09:32:56.702422+0000","last_active":"2024-03-04T09:32:57.709114+0000","last_peered":"2024-03-04T09:32:57.709114+0000","last_clean":"2024-03-04T09:32:57.709114+0000","last_became_active":"2024-03-04T09:32:56.701984+0000","last_became_peered":"2024-03-04T09:32:56.701984+0000","last_unstale":"2024-03-04T09:32:57.709114+0000","last_undegraded":"2024-03-04T09:32:57.709114+0000","last_fullsized":"2024-03-04T09:32:57.709114+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_clean_scrub_stamp":"2024-03-04T09:32:55.635455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:25:11.338620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.10","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.701242+0000","last_change":"2024-03-04T09:32:57.693627+0000","last_active":"2024-03-04T09:32:57.701242+0000","last_peered":"2024-03-04T09:32:57.701242+0000","last_clean":"2024-03-04T09:32:57.701242+0000","last_became_active":"2024-03-04T09:32:57.693303+0000","last_became_peered":"2024-03-04T09:32:57.693303+0000","last_unstale":"2024-03-04T09:32:57.701242+0000","last_undegraded":"2024-03-04T09:32:57.701242+0000","last_fullsized":"2024-03-04T09:32:57.701242+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:37:21.506217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.678816+0000","last_change":"2024-03-04T09:32:56.701788+0000","last_active":"2024-03-04T09:32:57.678816+0000","last_peered":"2024-03-04T09:32:57.678816+0000","last_clean":"2024-03-04T09:32:57.678816+0000","last_became_active":"2024-03-04T09:32:56.701356+0000","last_became_peered":"2024-03-04T09:32:56.701356+0000","last_unstale":"2024-03-04T09:32:57.678816+0000","last_undegraded":"2024-03-04T09:32:57.678816+0000","last_fullsized":"2024-03-04T09:32:57.678816+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_clean_scrub_stamp":"2024-03-04T09:32:55.635455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:07:06.052094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.11","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.711346+0000","last_change":"2024-03-04T09:32:57.704622+0000","last_active":"2024-03-04T09:32:57.711346+0000","last_peered":"2024-03-04T09:32:57.711346+0000","last_clean":"2024-03-04T09:32:57.711346+0000","last_became_active":"2024-03-04T09:32:57.704058+0000","last_became_peered":"2024-03-04T09:32:57.704058+0000","last_unstale":"2024-03-04T09:32:57.711346+0000","last_undegraded":"2024-03-04T09:32:57.711346+0000","last_fullsized":"2024-03-04T09:32:57.711346+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:29:37.209382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.10","version":"126'8","reported_seq":20,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.722435+0000","last_change":"2024-03-04T09:32:56.707051+0000","last_active":"2024-03-04T09:32:57.722435+0000","last_peered":"2024-03-04T09:32:57.722435+0000","last_clean":"2024-03-04T09:32:57.722435+0000","last_became_active":"2024-03-04T09:32:56.705283+0000","last_became_peered":"2024-03-04T09:32:56.705283+0000","last_unstale":"2024-03-04T09:32:57.722435+0000","last_undegraded":"2024-03-04T09:32:57.722435+0000","last_fullsized":"2024-03-04T09:32:57.722435+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_clean_scrub_stamp":"2024-03-04T09:32:55.635455+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:07:06.052094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.e","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.721327+0000","last_change":"2024-03-04T09:32:57.715518+0000","last_active":"2024-03-04T09:32:57.721327+0000","last_peered":"2024-03-04T09:32:57.721327+0000","last_clean":"2024-03-04T09:32:57.721327+0000","last_became_active":"2024-03-04T09:32:57.714076+0000","last_became_peered":"2024-03-04T09:32:57.714076+0000","last_unstale":"2024-03-04T09:32:57.721327+0000","last_undegraded":"2024-03-04T09:32:57.721327+0000","last_fullsized":"2024-03-04T09:32:57.721327+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:01:47.019520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.f","version":"126'1","reported_seq":13,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.720863+0000","last_change":"2024-03-04T09:32:56.703103+0000","last_active":"2024-03-04T09:32:57.720863+0000","last_peered":"2024-03-04T09:32:57.720863+0000","last_clean":"2024-03-04T09:32:57.720863+0000","last_became_active":"2024-03-04T09:32:56.702790+0000","last_became_peered":"2024-03-04T09:32:56.702790+0000","last_unstale":"2024-03-04T09:32:57.720863+0000","last_undegraded":"2024-03-04T09:32:57.720863+0000","last_fullsized":"2024-03-04T09:32:57.720863+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_clean_scrub_stamp":"2024-03-04T09:32:55.635455+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:34:57.549462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.f","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.720146+0000","last_change":"2024-03-04T09:32:57.714710+0000","last_active":"2024-03-04T09:32:57.720146+0000","last_peered":"2024-03-04T09:32:57.720146+0000","last_clean":"2024-03-04T09:32:57.720146+0000","last_became_active":"2024-03-04T09:32:57.714346+0000","last_became_peered":"2024-03-04T09:32:57.714346+0000","last_unstale":"2024-03-04T09:32:57.720146+0000","last_undegraded":"2024-03-04T09:32:57.720146+0000","last_fullsized":"2024-03-04T09:32:57.720146+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:37:21.506217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.e","version":"126'2","reported_seq":14,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.723743+0000","last_change":"2024-03-04T09:32:56.707947+0000","last_active":"2024-03-04T09:32:57.723743+0000","last_peered":"2024-03-04T09:32:57.723743+0000","last_clean":"2024-03-04T09:32:57.723743+0000","last_became_active":"2024-03-04T09:32:56.707630+0000","last_became_peered":"2024-03-04T09:32:56.707630+0000","last_unstale":"2024-03-04T09:32:57.723743+0000","last_undegraded":"2024-03-04T09:32:57.723743+0000","last_fullsized":"2024-03-04T09:32:57.723743+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_clean_scrub_stamp":"2024-03-04T09:32:55.635455+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:43:43.184312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.722221+0000","last_change":"2024-03-04T09:32:57.718915+0000","last_active":"2024-03-04T09:32:57.722221+0000","last_peered":"2024-03-04T09:32:57.722221+0000","last_clean":"2024-03-04T09:32:57.722221+0000","last_became_active":"2024-03-04T09:32:57.718641+0000","last_became_peered":"2024-03-04T09:32:57.718641+0000","last_unstale":"2024-03-04T09:32:57.722221+0000","last_undegraded":"2024-03-04T09:32:57.722221+0000","last_fullsized":"2024-03-04T09:32:57.722221+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:40:10.018154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.d","version":"126'1","reported_seq":13,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.712873+0000","last_change":"2024-03-04T09:32:56.704416+0000","last_active":"2024-03-04T09:32:57.712873+0000","last_peered":"2024-03-04T09:32:57.712873+0000","last_clean":"2024-03-04T09:32:57.712873+0000","last_became_active":"2024-03-04T09:32:56.704131+0000","last_became_peered":"2024-03-04T09:32:56.704131+0000","last_unstale":"2024-03-04T09:32:57.712873+0000","last_undegraded":"2024-03-04T09:32:57.712873+0000","last_fullsized":"2024-03-04T09:32:57.712873+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_clean_scrub_stamp":"2024-03-04T09:32:55.635455+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:10:27.785793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.d","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.702852+0000","last_change":"2024-03-04T09:32:57.698996+0000","last_active":"2024-03-04T09:32:57.702852+0000","last_peered":"2024-03-04T09:32:57.702852+0000","last_clean":"2024-03-04T09:32:57.702852+0000","last_became_active":"2024-03-04T09:32:57.698685+0000","last_became_peered":"2024-03-04T09:32:57.698685+0000","last_unstale":"2024-03-04T09:32:57.702852+0000","last_undegraded":"2024-03-04T09:32:57.702852+0000","last_fullsized":"2024-03-04T09:32:57.702852+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:29:37.209382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.712842+0000","last_change":"2024-03-04T09:32:56.698084+0000","last_active":"2024-03-04T09:32:57.712842+0000","last_peered":"2024-03-04T09:32:57.712842+0000","last_clean":"2024-03-04T09:32:57.712842+0000","last_became_active":"2024-03-04T09:32:56.697794+0000","last_became_peered":"2024-03-04T09:32:56.697794+0000","last_unstale":"2024-03-04T09:32:57.712842+0000","last_undegraded":"2024-03-04T09:32:57.712842+0000","last_fullsized":"2024-03-04T09:32:57.712842+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_clean_scrub_stamp":"2024-03-04T09:32:55.635455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:04:08.822065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.a","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.724637+0000","last_change":"2024-03-04T09:32:57.716276+0000","last_active":"2024-03-04T09:32:57.724637+0000","last_peered":"2024-03-04T09:32:57.724637+0000","last_clean":"2024-03-04T09:32:57.724637+0000","last_became_active":"2024-03-04T09:32:57.715575+0000","last_became_peered":"2024-03-04T09:32:57.715575+0000","last_unstale":"2024-03-04T09:32:57.724637+0000","last_undegraded":"2024-03-04T09:32:57.724637+0000","last_fullsized":"2024-03-04T09:32:57.724637+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:40:42.783681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.b","version":"126'2","reported_seq":14,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.725662+0000","last_change":"2024-03-04T09:32:56.697279+0000","last_active":"2024-03-04T09:32:57.725662+0000","last_peered":"2024-03-04T09:32:57.725662+0000","last_clean":"2024-03-04T09:32:57.725662+0000","last_became_active":"2024-03-04T09:32:56.696722+0000","last_became_peered":"2024-03-04T09:32:56.696722+0000","last_unstale":"2024-03-04T09:32:57.725662+0000","last_undegraded":"2024-03-04T09:32:57.725662+0000","last_fullsized":"2024-03-04T09:32:57.725662+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_clean_scrub_stamp":"2024-03-04T09:32:55.635455+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:04:14.387131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.b","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.718059+0000","last_change":"2024-03-04T09:32:57.714845+0000","last_active":"2024-03-04T09:32:57.718059+0000","last_peered":"2024-03-04T09:32:57.718059+0000","last_clean":"2024-03-04T09:32:57.718059+0000","last_became_active":"2024-03-04T09:32:57.714520+0000","last_became_peered":"2024-03-04T09:32:57.714520+0000","last_unstale":"2024-03-04T09:32:57.718059+0000","last_undegraded":"2024-03-04T09:32:57.718059+0000","last_fullsized":"2024-03-04T09:32:57.718059+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:44:15.764637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.695894+0000","last_change":"2024-03-04T09:32:56.705846+0000","last_active":"2024-03-04T09:32:57.695894+0000","last_peered":"2024-03-04T09:32:57.695894+0000","last_clean":"2024-03-04T09:32:57.695894+0000","last_became_active":"2024-03-04T09:32:56.705198+0000","last_became_peered":"2024-03-04T09:32:56.705198+0000","last_unstale":"2024-03-04T09:32:57.695894+0000","last_undegraded":"2024-03-04T09:32:57.695894+0000","last_fullsized":"2024-03-04T09:32:57.695894+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_clean_scrub_stamp":"2024-03-04T09:32:55.635455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:01:33.730754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.8","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.701262+0000","last_change":"2024-03-04T09:32:57.696969+0000","last_active":"2024-03-04T09:32:57.701262+0000","last_peered":"2024-03-04T09:32:57.701262+0000","last_clean":"2024-03-04T09:32:57.701262+0000","last_became_active":"2024-03-04T09:32:57.696675+0000","last_became_peered":"2024-03-04T09:32:57.696675+0000","last_unstale":"2024-03-04T09:32:57.701262+0000","last_undegraded":"2024-03-04T09:32:57.701262+0000","last_fullsized":"2024-03-04T09:32:57.701262+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:48:55.639006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.689228+0000","last_change":"2024-03-04T09:32:56.700895+0000","last_active":"2024-03-04T09:32:57.689228+0000","last_peered":"2024-03-04T09:32:57.689228+0000","last_clean":"2024-03-04T09:32:57.689228+0000","last_became_active":"2024-03-04T09:32:56.700581+0000","last_became_peered":"2024-03-04T09:32:56.700581+0000","last_unstale":"2024-03-04T09:32:57.689228+0000","last_undegraded":"2024-03-04T09:32:57.689228+0000","last_fullsized":"2024-03-04T09:32:57.689228+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_clean_scrub_stamp":"2024-03-04T09:32:55.635455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:21:49.449101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.713025+0000","last_change":"2024-03-04T09:32:57.711121+0000","last_active":"2024-03-04T09:32:57.713025+0000","last_peered":"2024-03-04T09:32:57.713025+0000","last_clean":"2024-03-04T09:32:57.713025+0000","last_became_active":"2024-03-04T09:32:57.710405+0000","last_became_peered":"2024-03-04T09:32:57.710405+0000","last_unstale":"2024-03-04T09:32:57.713025+0000","last_undegraded":"2024-03-04T09:32:57.713025+0000","last_fullsized":"2024-03-04T09:32:57.713025+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:01:00.772002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.699340+0000","last_change":"2024-03-04T09:32:56.712214+0000","last_active":"2024-03-04T09:32:57.699340+0000","last_peered":"2024-03-04T09:32:57.699340+0000","last_clean":"2024-03-04T09:32:57.699340+0000","last_became_active":"2024-03-04T09:32:56.711412+0000","last_became_peered":"2024-03-04T09:32:56.711412+0000","last_unstale":"2024-03-04T09:32:57.699340+0000","last_undegraded":"2024-03-04T09:32:57.699340+0000","last_fullsized":"2024-03-04T09:32:57.699340+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_clean_scrub_stamp":"2024-03-04T09:32:55.635455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:42:32.193040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.6","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.721539+0000","last_change":"2024-03-04T09:32:57.718067+0000","last_active":"2024-03-04T09:32:57.721539+0000","last_peered":"2024-03-04T09:32:57.721539+0000","last_clean":"2024-03-04T09:32:57.721539+0000","last_became_active":"2024-03-04T09:32:57.716705+0000","last_became_peered":"2024-03-04T09:32:57.716705+0000","last_unstale":"2024-03-04T09:32:57.721539+0000","last_undegraded":"2024-03-04T09:32:57.721539+0000","last_fullsized":"2024-03-04T09:32:57.721539+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:51:36.030782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.7","version":"126'2","reported_seq":14,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.725065+0000","last_change":"2024-03-04T09:32:56.699236+0000","last_active":"2024-03-04T09:32:57.725065+0000","last_peered":"2024-03-04T09:32:57.725065+0000","last_clean":"2024-03-04T09:32:57.725065+0000","last_became_active":"2024-03-04T09:32:56.698957+0000","last_became_peered":"2024-03-04T09:32:56.698957+0000","last_unstale":"2024-03-04T09:32:57.725065+0000","last_undegraded":"2024-03-04T09:32:57.725065+0000","last_fullsized":"2024-03-04T09:32:57.725065+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_clean_scrub_stamp":"2024-03-04T09:32:55.635455+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:45:47.015969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.7","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.725631+0000","last_change":"2024-03-04T09:32:57.724317+0000","last_active":"2024-03-04T09:32:57.725631+0000","last_peered":"2024-03-04T09:32:57.725631+0000","last_clean":"2024-03-04T09:32:57.725631+0000","last_became_active":"2024-03-04T09:32:57.723846+0000","last_became_peered":"2024-03-04T09:32:57.723846+0000","last_unstale":"2024-03-04T09:32:57.725631+0000","last_undegraded":"2024-03-04T09:32:57.725631+0000","last_fullsized":"2024-03-04T09:32:57.725631+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:06:12.093886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.6","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.688281+0000","last_change":"2024-03-04T09:32:56.708274+0000","last_active":"2024-03-04T09:32:57.688281+0000","last_peered":"2024-03-04T09:32:57.688281+0000","last_clean":"2024-03-04T09:32:57.688281+0000","last_became_active":"2024-03-04T09:32:56.707981+0000","last_became_peered":"2024-03-04T09:32:56.707981+0000","last_unstale":"2024-03-04T09:32:57.688281+0000","last_undegraded":"2024-03-04T09:32:57.688281+0000","last_fullsized":"2024-03-04T09:32:57.688281+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_clean_scrub_stamp":"2024-03-04T09:32:55.635455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:44:14.748162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.698442+0000","last_change":"2024-03-04T09:32:57.691181+0000","last_active":"2024-03-04T09:32:57.698442+0000","last_peered":"2024-03-04T09:32:57.698442+0000","last_clean":"2024-03-04T09:32:57.698442+0000","last_became_active":"2024-03-04T09:32:57.690874+0000","last_became_peered":"2024-03-04T09:32:57.690874+0000","last_unstale":"2024-03-04T09:32:57.698442+0000","last_undegraded":"2024-03-04T09:32:57.698442+0000","last_fullsized":"2024-03-04T09:32:57.698442+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:12:39.009902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.677940+0000","last_change":"2024-03-04T09:32:56.710983+0000","last_active":"2024-03-04T09:32:57.677940+0000","last_peered":"2024-03-04T09:32:57.677940+0000","last_clean":"2024-03-04T09:32:57.677940+0000","last_became_active":"2024-03-04T09:32:56.710651+0000","last_became_peered":"2024-03-04T09:32:56.710651+0000","last_unstale":"2024-03-04T09:32:57.677940+0000","last_undegraded":"2024-03-04T09:32:57.677940+0000","last_fullsized":"2024-03-04T09:32:57.677940+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_clean_scrub_stamp":"2024-03-04T09:32:55.635455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:34:26.054015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.721780+0000","last_change":"2024-03-04T09:32:57.716019+0000","last_active":"2024-03-04T09:32:57.721780+0000","last_peered":"2024-03-04T09:32:57.721780+0000","last_clean":"2024-03-04T09:32:57.721780+0000","last_became_active":"2024-03-04T09:32:57.715731+0000","last_became_peered":"2024-03-04T09:32:57.715731+0000","last_unstale":"2024-03-04T09:32:57.721780+0000","last_undegraded":"2024-03-04T09:32:57.721780+0000","last_fullsized":"2024-03-04T09:32:57.721780+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:32:26.208985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.700183+0000","last_change":"2024-03-04T09:32:56.682683+0000","last_active":"2024-03-04T09:32:57.700183+0000","last_peered":"2024-03-04T09:32:57.700183+0000","last_clean":"2024-03-04T09:32:57.700183+0000","last_became_active":"2024-03-04T09:32:56.682313+0000","last_became_peered":"2024-03-04T09:32:56.682313+0000","last_unstale":"2024-03-04T09:32:57.700183+0000","last_undegraded":"2024-03-04T09:32:57.700183+0000","last_fullsized":"2024-03-04T09:32:57.700183+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_clean_scrub_stamp":"2024-03-04T09:32:55.635455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:15:18.079206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.711379+0000","last_change":"2024-03-04T09:32:57.704259+0000","last_active":"2024-03-04T09:32:57.711379+0000","last_peered":"2024-03-04T09:32:57.711379+0000","last_clean":"2024-03-04T09:32:57.711379+0000","last_became_active":"2024-03-04T09:32:57.703939+0000","last_became_peered":"2024-03-04T09:32:57.703939+0000","last_unstale":"2024-03-04T09:32:57.711379+0000","last_undegraded":"2024-03-04T09:32:57.711379+0000","last_fullsized":"2024-03-04T09:32:57.711379+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:33:04.056111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.677563+0000","last_change":"2024-03-04T09:32:56.703624+0000","last_active":"2024-03-04T09:32:57.677563+0000","last_peered":"2024-03-04T09:32:57.677563+0000","last_clean":"2024-03-04T09:32:57.677563+0000","last_became_active":"2024-03-04T09:32:56.703324+0000","last_became_peered":"2024-03-04T09:32:56.703324+0000","last_unstale":"2024-03-04T09:32:57.677563+0000","last_undegraded":"2024-03-04T09:32:57.677563+0000","last_fullsized":"2024-03-04T09:32:57.677563+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_clean_scrub_stamp":"2024-03-04T09:32:55.635455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:57:27.241237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.720259+0000","last_change":"2024-03-04T09:32:57.715965+0000","last_active":"2024-03-04T09:32:57.720259+0000","last_peered":"2024-03-04T09:32:57.720259+0000","last_clean":"2024-03-04T09:32:57.720259+0000","last_became_active":"2024-03-04T09:32:57.715594+0000","last_became_peered":"2024-03-04T09:32:57.715594+0000","last_unstale":"2024-03-04T09:32:57.720259+0000","last_undegraded":"2024-03-04T09:32:57.720259+0000","last_fullsized":"2024-03-04T09:32:57.720259+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:48:55.639006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.688881+0000","last_change":"2024-03-04T09:32:56.703726+0000","last_active":"2024-03-04T09:32:57.688881+0000","last_peered":"2024-03-04T09:32:57.688881+0000","last_clean":"2024-03-04T09:32:57.688881+0000","last_became_active":"2024-03-04T09:32:56.703246+0000","last_became_peered":"2024-03-04T09:32:56.703246+0000","last_unstale":"2024-03-04T09:32:57.688881+0000","last_undegraded":"2024-03-04T09:32:57.688881+0000","last_fullsized":"2024-03-04T09:32:57.688881+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_clean_scrub_stamp":"2024-03-04T09:32:55.635455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:51:35.014307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.0","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.720553+0000","last_change":"2024-03-04T09:32:57.715500+0000","last_active":"2024-03-04T09:32:57.720553+0000","last_peered":"2024-03-04T09:32:57.720553+0000","last_clean":"2024-03-04T09:32:57.720553+0000","last_became_active":"2024-03-04T09:32:57.715059+0000","last_became_peered":"2024-03-04T09:32:57.715059+0000","last_unstale":"2024-03-04T09:32:57.720553+0000","last_undegraded":"2024-03-04T09:32:57.720553+0000","last_fullsized":"2024-03-04T09:32:57.720553+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:35:45.163868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.689400+0000","last_change":"2024-03-04T09:32:56.711325+0000","last_active":"2024-03-04T09:32:57.689400+0000","last_peered":"2024-03-04T09:32:57.689400+0000","last_clean":"2024-03-04T09:32:57.689400+0000","last_became_active":"2024-03-04T09:32:56.711002+0000","last_became_peered":"2024-03-04T09:32:56.711002+0000","last_unstale":"2024-03-04T09:32:57.689400+0000","last_undegraded":"2024-03-04T09:32:57.689400+0000","last_fullsized":"2024-03-04T09:32:57.689400+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_clean_scrub_stamp":"2024-03-04T09:32:55.635455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:57:48.422427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.1","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.700903+0000","last_change":"2024-03-04T09:32:57.695150+0000","last_active":"2024-03-04T09:32:57.700903+0000","last_peered":"2024-03-04T09:32:57.700903+0000","last_clean":"2024-03-04T09:32:57.700903+0000","last_became_active":"2024-03-04T09:32:57.694800+0000","last_became_peered":"2024-03-04T09:32:57.694800+0000","last_unstale":"2024-03-04T09:32:57.700903+0000","last_undegraded":"2024-03-04T09:32:57.700903+0000","last_fullsized":"2024-03-04T09:32:57.700903+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:30:14.957488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.698689+0000","last_change":"2024-03-04T09:32:56.702914+0000","last_active":"2024-03-04T09:32:57.698689+0000","last_peered":"2024-03-04T09:32:57.698689+0000","last_clean":"2024-03-04T09:32:57.698689+0000","last_became_active":"2024-03-04T09:32:56.702511+0000","last_became_peered":"2024-03-04T09:32:56.702511+0000","last_unstale":"2024-03-04T09:32:57.698689+0000","last_undegraded":"2024-03-04T09:32:57.698689+0000","last_fullsized":"2024-03-04T09:32:57.698689+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_clean_scrub_stamp":"2024-03-04T09:32:55.635455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:21:02.657585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.717293+0000","last_change":"2024-03-04T09:32:57.714955+0000","last_active":"2024-03-04T09:32:57.717293+0000","last_peered":"2024-03-04T09:32:57.717293+0000","last_clean":"2024-03-04T09:32:57.717293+0000","last_became_active":"2024-03-04T09:32:57.713559+0000","last_became_peered":"2024-03-04T09:32:57.713559+0000","last_unstale":"2024-03-04T09:32:57.717293+0000","last_undegraded":"2024-03-04T09:32:57.717293+0000","last_fullsized":"2024-03-04T09:32:57.717293+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:56.651930+0000","last_clean_scrub_stamp":"2024-03-04T09:32:56.651930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:53:09.217232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"126'2","reported_seq":14,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.723579+0000","last_change":"2024-03-04T09:32:56.704329+0000","last_active":"2024-03-04T09:32:57.723579+0000","last_peered":"2024-03-04T09:32:57.723579+0000","last_clean":"2024-03-04T09:32:57.723579+0000","last_became_active":"2024-03-04T09:32:56.704009+0000","last_became_peered":"2024-03-04T09:32:56.704009+0000","last_unstale":"2024-03-04T09:32:57.723579+0000","last_undegraded":"2024-03-04T09:32:57.723579+0000","last_fullsized":"2024-03-04T09:32:57.723579+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:32:55.635455+0000","last_clean_scrub_stamp":"2024-03-04T09:32:55.635455+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:21:33.130606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":366,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-04T09:32:57.681372+0000","last_change":"2024-03-04T09:25:09.316159+0000","last_active":"2024-03-04T09:32:57.681372+0000","last_peered":"2024-03-04T09:32:57.681372+0000","last_clean":"2024-03-04T09:32:57.681372+0000","last_became_active":"2024-03-04T09:25:09.313767+0000","last_became_peered":"2024-03-04T09:25:09.313767+0000","last_unstale":"2024-03-04T09:32:57.681372+0000","last_undegraded":"2024-03-04T09:32:57.681372+0000","last_fullsized":"2024-03-04T09:32:57.681372+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T09:25:08.110953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T09:25:08.110953+0000","last_clean_scrub_stamp":"2024-03-04T09:25:08.110953+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:34:21.745390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":19,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":18,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574945,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37452,"kb_used_data":2104,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334388,"statfs":{"total":96636764160,"available":96598413312,"internally_reserved":0,"allocated":2154496,"data_stored":1268816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574945,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36868,"kb_used_data":1524,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334972,"statfs":{"total":96636764160,"available":96599011328,"internally_reserved":0,"allocated":1560576,"data_stored":678448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574947,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36884,"kb_used_data":1544,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334956,"statfs":{"total":96636764160,"available":96598994944,"internally_reserved":0,"allocated":1581056,"data_stored":685335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574944,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36884,"kb_used_data":1540,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334956,"statfs":{"total":96636764160,"available":96598994944,"internally_reserved":0,"allocated":1576960,"data_stored":685293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574946,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36872,"kb_used_data":1524,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334968,"statfs":{"total":96636764160,"available":96599007232,"internally_reserved":0,"allocated":1560576,"data_stored":678448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574945,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36868,"kb_used_data":1524,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334972,"statfs":{"total":96636764160,"available":96599011328,"internally_reserved":0,"allocated":1560576,"data_stored":678448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574946,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36868,"kb_used_data":1524,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334972,"statfs":{"total":96636764160,"available":96599011328,"internally_reserved":0,"allocated":1560576,"data_stored":678448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574945,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37460,"kb_used_data":2116,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334380,"statfs":{"total":96636764160,"available":96598405120,"internally_reserved":0,"allocated":2166784,"data_stored":1275591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-04T09:33:02.690 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-04T09:33:02.690 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-04T09:33:02.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.988+0000 7f67d8a00640 1 Processor -- start 2024-03-04T09:33:02.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.989+0000 7f67d8a00640 1 -- start start 2024-03-04T09:33:02.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.991+0000 7f67d8a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f67d40d4870 0x7f67d40d1920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:02.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.991+0000 7f67d8a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67d40d4ca0 con 0x7f67d415e220 2024-03-04T09:33:02.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.991+0000 7f67d8a00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67d40cf4a0 con 0x7f67d40d26f0 2024-03-04T09:33:02.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.991+0000 7f67d8a00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67d40b36b0 con 0x7f67d40d4870 2024-03-04T09:33:02.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.991+0000 7f67d2800640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f67d415e220 0x7f67d40d1fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:58060/0 (socket says 172.21.15.6:58060) 2024-03-04T09:33:02.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.991+0000 7f67d2800640 1 -- 172.21.15.6:0/1438068421 learned_addr learned my addr 172.21.15.6:0/1438068421 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:33:02.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.991+0000 7f67d3200640 1 --2- 172.21.15.6:0/1438068421 >> v2:172.21.15.6:3300/0 conn(0x7f67d40d4870 0x7f67d40d1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:02.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.992+0000 7f67d1e00640 1 -- 172.21.15.6:0/1438068421 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2829387079 0 0) 0x7f67c4001110 con 0x7f67d415e220 2024-03-04T09:33:02.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.992+0000 7f67d1e00640 1 -- 172.21.15.6:0/1438068421 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67b8003520 con 0x7f67d415e220 2024-03-04T09:33:02.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.992+0000 7f67d3200640 1 -- 172.21.15.6:0/1438068421 >> v1:172.21.15.146:6789/0 conn(0x7f67d40d26f0 legacy=0x7f67d40b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:33:02.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.992+0000 7f67d3200640 1 -- 172.21.15.6:0/1438068421 >> v1:172.21.15.6:6789/0 conn(0x7f67d415e220 legacy=0x7f67d40d1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:02.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.992+0000 7f67d3200640 1 -- 172.21.15.6:0/1438068421 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67d40cf780 con 0x7f67d40d4870 2024-03-04T09:33:02.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.993+0000 7f67d3200640 1 --2- 172.21.15.6:0/1438068421 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f67d40d4870 0x7f67d40d1920 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7f67cc009d90 tx=0x7f67cc02be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec184427a9e7ae3a server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:33:02.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.993+0000 7f67d1e00640 1 -- 172.21.15.6:0/1438068421 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67cc039070 con 0x7f67d40d4870 2024-03-04T09:33:02.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.993+0000 7f67d1e00640 1 -- 172.21.15.6:0/1438068421 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67cc02cbd0 con 0x7f67d40d4870 2024-03-04T09:33:02.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.993+0000 7f67d1e00640 1 -- 172.21.15.6:0/1438068421 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67cc034440 con 0x7f67d40d4870 2024-03-04T09:33:02.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.994+0000 7f67d8a00640 1 -- 172.21.15.6:0/1438068421 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f67d40d4870 msgr2=0x7f67d40d1920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:02.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.994+0000 7f67d8a00640 1 --2- 172.21.15.6:0/1438068421 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f67d40d4870 0x7f67d40d1920 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7f67cc009d90 tx=0x7f67cc02be20 comp rx=0 tx=0).stop 2024-03-04T09:33:02.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.994+0000 7f67d8a00640 1 -- 172.21.15.6:0/1438068421 shutdown_connections 2024-03-04T09:33:02.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.994+0000 7f67d8a00640 1 --2- 172.21.15.6:0/1438068421 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f67d40d4870 0x7f67d40d1920 unknown :-1 s=CLOSED pgs=1504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:02.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.994+0000 7f67d8a00640 1 -- 172.21.15.6:0/1438068421 >> 172.21.15.6:0/1438068421 conn(0x7f67d40d3d50 msgr2=0x7f67d40d4160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:33:02.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.994+0000 7f67d8a00640 1 -- 172.21.15.6:0/1438068421 shutdown_connections 2024-03-04T09:33:02.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.994+0000 7f67d8a00640 1 -- 172.21.15.6:0/1438068421 wait complete. 2024-03-04T09:33:02.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.996+0000 7f67d8a00640 1 Processor -- start 2024-03-04T09:33:02.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.996+0000 7f67d8a00640 1 -- start start 2024-03-04T09:33:02.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.997+0000 7f67d8a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f67d40d26f0 0x7f67d40cb9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:02.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.997+0000 7f67d8a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f67d40d4870 0x7f67d40cc070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:02.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.997+0000 7f67d8a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f67d415e220 0x7f67d415c400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:02.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.997+0000 7f67d8a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67d40b4530 con 0x7f67d415e220 2024-03-04T09:33:02.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.997+0000 7f67d8a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67d40b3900 con 0x7f67d40d26f0 2024-03-04T09:33:02.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.997+0000 7f67d8a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67d40b46b0 con 0x7f67d40d4870 2024-03-04T09:33:02.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.997+0000 7f67d3200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f67d415e220 0x7f67d415c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:02.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.997+0000 7f67d3200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f67d415e220 0x7f67d415c400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43000/0 (socket says 172.21.15.6:43000) 2024-03-04T09:33:02.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.997+0000 7f67d3200640 1 -- 172.21.15.6:0/69051202 learned_addr learned my addr 172.21.15.6:0/69051202 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:33:02.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.997+0000 7f67d2800640 1 --2- 172.21.15.6:0/69051202 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f67d40d26f0 0x7f67d40cb9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:02.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.997+0000 7f67cbe00640 1 --2- 172.21.15.6:0/69051202 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f67d40d4870 0x7f67d40cc070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:02.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.998+0000 7f67d3200640 1 -- 172.21.15.6:0/69051202 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f67d40d4870 msgr2=0x7f67d40cc070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:02.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.998+0000 7f67d3200640 1 --2- 172.21.15.6:0/69051202 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f67d40d4870 0x7f67d40cc070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:02.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.998+0000 7f67d3200640 1 -- 172.21.15.6:0/69051202 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f67d40d26f0 msgr2=0x7f67d40cb9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:02.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.998+0000 7f67d3200640 1 --2- 172.21.15.6:0/69051202 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f67d40d26f0 0x7f67d40cb9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:02.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.998+0000 7f67d3200640 1 -- 172.21.15.6:0/69051202 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67d40cc5b0 con 0x7f67d415e220 2024-03-04T09:33:02.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.998+0000 7f67d3200640 1 --2- 172.21.15.6:0/69051202 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f67d415e220 0x7f67d415c400 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7f67cc009d90 tx=0x7f67cc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:33:02.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.998+0000 7f67caa00640 1 -- 172.21.15.6:0/69051202 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67cc039050 con 0x7f67d415e220 2024-03-04T09:33:02.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.998+0000 7f67caa00640 1 -- 172.21.15.6:0/69051202 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67cc045070 con 0x7f67d415e220 2024-03-04T09:33:02.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.998+0000 7f67caa00640 1 -- 172.21.15.6:0/69051202 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67cc009450 con 0x7f67d415e220 2024-03-04T09:33:02.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.998+0000 7f67d8a00640 1 -- 172.21.15.6:0/69051202 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67d40ccb10 con 0x7f67d415e220 2024-03-04T09:33:02.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:02.999+0000 7f67d8a00640 1 -- 172.21.15.6:0/69051202 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67d40cc830 con 0x7f67d415e220 2024-03-04T09:33:02.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:03.000+0000 7f67d8a00640 1 -- 172.21.15.6:0/69051202 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67d404f330 con 0x7f67d415e220 2024-03-04T09:33:02.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:03.001+0000 7f67caa00640 1 -- 172.21.15.6:0/69051202 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f67cc03e850 con 0x7f67d415e220 2024-03-04T09:33:02.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:03.001+0000 7f67caa00640 1 --2- 172.21.15.6:0/69051202 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f67b006b4d0 0x7f67b006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:02.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:03.001+0000 7f67caa00640 1 -- 172.21.15.6:0/69051202 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f67cc0acff0 con 0x7f67d415e220 2024-03-04T09:33:03.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:03.002+0000 7f67d2800640 1 --2- 172.21.15.6:0/69051202 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f67b006b4d0 0x7f67b006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:03.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:03.003+0000 7f67d2800640 1 --2- 172.21.15.6:0/69051202 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f67b006b4d0 0x7f67b006d990 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f67c4007fd0 tx=0x7f67c4009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:33:03.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:03.006+0000 7f67caa00640 1 -- 172.21.15.6:0/69051202 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f67d40cc5b0 con 0x7f67d415e220 2024-03-04T09:33:03.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:03.278+0000 7f67d8a00640 1 -- 172.21.15.6:0/69051202 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f67d40b4be0 con 0x7f67d415e220 2024-03-04T09:33:03.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:03.282+0000 7f67caa00640 1 -- 172.21.15.6:0/69051202 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v18) v1 ==== 190+0+147 (secure 0 0 0) 0x7f67cc07be00 con 0x7f67d415e220 2024-03-04T09:33:03.280 INFO:teuthology.orchestra.run.smithi006.stdout:[client.0] 2024-03-04T09:33:03.280 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQA9k+Vl4L+rJxAACaDbQ4/Bybp5hAZAnJzlVw== 2024-03-04T09:33:03.280 INFO:teuthology.orchestra.run.smithi006.stdout: caps mds = "allow" 2024-03-04T09:33:03.281 INFO:teuthology.orchestra.run.smithi006.stdout: caps mon = "allow r" 2024-03-04T09:33:03.281 INFO:teuthology.orchestra.run.smithi006.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-04T09:33:03.281 INFO:teuthology.orchestra.run.smithi006.stderr:updated caps for client.0 2024-03-04T09:33:03.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:03.285+0000 7f67d8a00640 1 -- 172.21.15.6:0/69051202 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f67b006b4d0 msgr2=0x7f67b006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:03.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:03.285+0000 7f67d8a00640 1 --2- 172.21.15.6:0/69051202 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f67b006b4d0 0x7f67b006d990 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f67c4007fd0 tx=0x7f67c4009040 comp rx=0 tx=0).stop 2024-03-04T09:33:03.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:03.285+0000 7f67d8a00640 1 -- 172.21.15.6:0/69051202 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f67d415e220 msgr2=0x7f67d415c400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:03.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:03.285+0000 7f67d8a00640 1 --2- 172.21.15.6:0/69051202 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f67d415e220 0x7f67d415c400 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7f67cc009d90 tx=0x7f67cc004a40 comp rx=0 tx=0).stop 2024-03-04T09:33:03.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:03.285+0000 7f67d8a00640 1 -- 172.21.15.6:0/69051202 shutdown_connections 2024-03-04T09:33:03.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:03.285+0000 7f67d8a00640 1 --2- 172.21.15.6:0/69051202 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f67d415e220 0x7f67d415c400 unknown :-1 s=CLOSED pgs=1505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:03.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:03.285+0000 7f67d8a00640 1 --2- 172.21.15.6:0/69051202 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f67b006b4d0 0x7f67b006d990 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:03.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:03.285+0000 7f67d8a00640 1 --2- 172.21.15.6:0/69051202 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f67d40d4870 0x7f67d40cc070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:03.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:03.285+0000 7f67d8a00640 1 --2- 172.21.15.6:0/69051202 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f67d40d26f0 0x7f67d40cb9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:03.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:03.285+0000 7f67d8a00640 1 -- 172.21.15.6:0/69051202 >> 172.21.15.6:0/69051202 conn(0x7f67d40d3d50 msgr2=0x7f67d406c180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:33:03.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:03.285+0000 7f67d8a00640 1 -- 172.21.15.6:0/69051202 shutdown_connections 2024-03-04T09:33:03.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:03.286+0000 7f67d8a00640 1 -- 172.21.15.6:0/69051202 wait complete. 2024-03-04T09:33:03.302 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:33:03.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:03.523+0000 7f2be9600640 1 Processor -- start 2024-03-04T09:33:03.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:03.524+0000 7f2be9600640 1 -- start start 2024-03-04T09:33:03.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:03.527+0000 7f2be9600640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f2be4162420 0x7f2be406afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:03.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:03.527+0000 7f2be9600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2be406b4f0 0x7f2be40d0910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:03.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:03.527+0000 7f2be9600640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2be4147bc0 con 0x7f2be41477e0 2024-03-04T09:33:03.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:03.527+0000 7f2be9600640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2be406b960 con 0x7f2be4162420 2024-03-04T09:33:03.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:03.527+0000 7f2be9600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2be4161f90 con 0x7f2be406b4f0 2024-03-04T09:33:03.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:03.527+0000 7f2be3e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f2be41477e0 0x7f2be406ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:58074/0 (socket says 172.21.15.6:58074) 2024-03-04T09:33:03.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:03.527+0000 7f2be3e00640 1 -- 172.21.15.6:0/2593335578 learned_addr learned my addr 172.21.15.6:0/2593335578 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:33:03.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:03.528+0000 7f2be2a00640 1 -- 172.21.15.6:0/2593335578 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3624150879 0 0) 0x7f2be4147bc0 con 0x7f2be41477e0 2024-03-04T09:33:03.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:03.528+0000 7f2be2a00640 1 -- 172.21.15.6:0/2593335578 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2bc8003520 con 0x7f2be41477e0 2024-03-04T09:33:03.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:03.528+0000 7f2be8c00640 1 --2- 172.21.15.6:0/2593335578 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2be406b4f0 0x7f2be40d0910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:03.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:03.528+0000 7f2be3400640 1 --2- 172.21.15.6:0/2593335578 >> v2:172.21.15.146:3300/0 conn(0x7f2be4162420 0x7f2be406afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:03.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:03.528+0000 7f2be2a00640 1 -- 172.21.15.6:0/2593335578 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2380454717 0 0) 0x7f2bc8003520 con 0x7f2be41477e0 2024-03-04T09:33:03.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:03.528+0000 7f2be2a00640 1 -- 172.21.15.6:0/2593335578 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2be406b4f0 msgr2=0x7f2be40d0910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:33:03.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:03.528+0000 7f2be2a00640 1 --2- 172.21.15.6:0/2593335578 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2be406b4f0 0x7f2be40d0910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:03.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:03.528+0000 7f2be2a00640 1 -- 172.21.15.6:0/2593335578 >> v2:172.21.15.146:3300/0 conn(0x7f2be4162420 msgr2=0x7f2be406afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:33:03.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:03.528+0000 7f2be2a00640 1 --2- 172.21.15.6:0/2593335578 >> v2:172.21.15.146:3300/0 conn(0x7f2be4162420 0x7f2be406afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:03.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:03.528+0000 7f2be2a00640 1 -- 172.21.15.6:0/2593335578 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2be40b3d70 con 0x7f2be41477e0 2024-03-04T09:33:03.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:03.528+0000 7f2be2a00640 1 -- 172.21.15.6:0/2593335578 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7f2bd4001670 con 0x7f2be41477e0 2024-03-04T09:33:03.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:03.528+0000 7f2be2a00640 1 -- 172.21.15.6:0/2593335578 >> v1:172.21.15.6:6789/0 conn(0x7f2be41477e0 legacy=0x7f2be406ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:03.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:03.528+0000 7f2be2a00640 1 --2- 172.21.15.6:0/2593335578 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2bc80039b0 0x7f2bc8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:03.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:03.528+0000 7f2be2a00640 1 --2- 172.21.15.6:0/2593335578 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2bc8004800 0x7f2bc8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:03.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:03.529+0000 7f2be2a00640 1 --2- 172.21.15.6:0/2593335578 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2be406b4f0 0x7f2bc80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:03.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:03.529+0000 7f2be2a00640 1 -- 172.21.15.6:0/2593335578 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2be406b960 con 0x7f2bc80039b0 2024-03-04T09:33:03.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:03.529+0000 7f2be2a00640 1 -- 172.21.15.6:0/2593335578 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2be4161f90 con 0x7f2bc8004800 2024-03-04T09:33:03.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:03.529+0000 7f2be2a00640 1 -- 172.21.15.6:0/2593335578 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bc800b9c0 con 0x7f2be406b4f0 2024-03-04T09:33:03.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:03.529+0000 7f2be3e00640 1 --2- 172.21.15.6:0/2593335578 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2bc80039b0 0x7f2bc8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:03.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:03.529+0000 7f2be3400640 1 --2- 172.21.15.6:0/2593335578 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2bc8004800 0x7f2bc8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:03.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:03.529+0000 7f2be8c00640 1 --2- 172.21.15.6:0/2593335578 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2be406b4f0 0x7f2bc80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:03.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:03.529+0000 7f2be3e00640 1 -- 172.21.15.6:0/2593335578 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2be406b4f0 msgr2=0x7f2bc80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:33:03.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:03.529+0000 7f2be3e00640 1 --2- 172.21.15.6:0/2593335578 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2be406b4f0 0x7f2bc80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:03.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:03.529+0000 7f2be3e00640 1 -- 172.21.15.6:0/2593335578 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2bc8004800 msgr2=0x7f2bc8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:33:03.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:03.529+0000 7f2be3e00640 1 --2- 172.21.15.6:0/2593335578 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2bc8004800 0x7f2bc8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:03.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:03.529+0000 7f2be3e00640 1 -- 172.21.15.6:0/2593335578 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bc8003520 con 0x7f2bc80039b0 2024-03-04T09:33:03.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:03.530+0000 7f2be3e00640 1 --2- 172.21.15.6:0/2593335578 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2bc80039b0 0x7f2bc8003dd0 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x7f2bd400c240 tx=0x7f2bc8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cbd63fab8e8b6b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:33:03.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:03.531+0000 7f2be2a00640 1 -- 172.21.15.6:0/2593335578 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bd4007d50 con 0x7f2bc80039b0 2024-03-04T09:33:03.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:03.531+0000 7f2be2a00640 1 -- 172.21.15.6:0/2593335578 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bd4030070 con 0x7f2bc80039b0 2024-03-04T09:33:03.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:03.531+0000 7f2be2a00640 1 -- 172.21.15.6:0/2593335578 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bd402cda0 con 0x7f2bc80039b0 2024-03-04T09:33:03.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:03.531+0000 7f2be9600640 1 -- 172.21.15.6:0/2593335578 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2bc80039b0 msgr2=0x7f2bc8003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:03.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:03.531+0000 7f2be9600640 1 --2- 172.21.15.6:0/2593335578 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2bc80039b0 0x7f2bc8003dd0 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x7f2bd400c240 tx=0x7f2bc8005830 comp rx=0 tx=0).stop 2024-03-04T09:33:03.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:03.531+0000 7f2be8c00640 1 -- 172.21.15.6:0/2593335578 reap_dead start 2024-03-04T09:33:03.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:03.532+0000 7f2be9600640 1 -- 172.21.15.6:0/2593335578 shutdown_connections 2024-03-04T09:33:03.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:03.532+0000 7f2be9600640 1 -- 172.21.15.6:0/2593335578 >> 172.21.15.6:0/2593335578 conn(0x7f2be406c0c0 msgr2=0x7f2be406c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:33:03.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:03.532+0000 7f2be9600640 1 -- 172.21.15.6:0/2593335578 shutdown_connections 2024-03-04T09:33:03.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:03.532+0000 7f2be9600640 1 -- 172.21.15.6:0/2593335578 wait complete. 2024-03-04T09:33:03.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:03.533+0000 7f2be9600640 1 Processor -- start 2024-03-04T09:33:03.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:03.533+0000 7f2be9600640 1 -- start start 2024-03-04T09:33:03.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:03.534+0000 7f2be9600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2bc80039b0 0x7f2be41f31d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:03.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:03.534+0000 7f2be9600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2be415e530 0x7f2be4160940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:03.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:03.534+0000 7f2be9600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2be415e9b0 0x7f2be4160e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:03.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:03.534+0000 7f2be9600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2be4161c20 con 0x7f2bc80039b0 2024-03-04T09:33:03.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:03.534+0000 7f2be9600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2be40b4330 con 0x7f2be415e530 2024-03-04T09:33:03.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:03.534+0000 7f2be9600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2be41628a0 con 0x7f2be415e9b0 2024-03-04T09:33:03.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:03.535+0000 7f2be3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2bc80039b0 0x7f2be41f31d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:03.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:03.535+0000 7f2be3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2bc80039b0 0x7f2be41f31d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43016/0 (socket says 172.21.15.6:43016) 2024-03-04T09:33:03.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:03.535+0000 7f2be3e00640 1 -- 172.21.15.6:0/2998012196 learned_addr learned my addr 172.21.15.6:0/2998012196 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:33:03.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:03.535+0000 7f2be3400640 1 --2- 172.21.15.6:0/2998012196 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2be415e530 0x7f2be4160940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:03.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:03.535+0000 7f2be8c00640 1 --2- 172.21.15.6:0/2998012196 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2be415e9b0 0x7f2be4160e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:03.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:03.535+0000 7f2be3e00640 1 -- 172.21.15.6:0/2998012196 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2be415e9b0 msgr2=0x7f2be4160e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:03.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:03.535+0000 7f2be3e00640 1 --2- 172.21.15.6:0/2998012196 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2be415e9b0 0x7f2be4160e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:03.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:03.535+0000 7f2be3e00640 1 -- 172.21.15.6:0/2998012196 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2be415e530 msgr2=0x7f2be4160940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:03.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:03.535+0000 7f2be3e00640 1 --2- 172.21.15.6:0/2998012196 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2be415e530 0x7f2be4160940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:03.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:03.535+0000 7f2be3e00640 1 -- 172.21.15.6:0/2998012196 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2be4160020 con 0x7f2bc80039b0 2024-03-04T09:33:03.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:03.535+0000 7f2be8c00640 1 --2- 172.21.15.6:0/2998012196 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2be415e9b0 0x7f2be4160e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:33:03.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:03.536+0000 7f2be3e00640 1 --2- 172.21.15.6:0/2998012196 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2bc80039b0 0x7f2be41f31d0 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7f2bd400c330 tx=0x7f2bd4001ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:33:03.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:03.536+0000 7f2bdbe00640 1 -- 172.21.15.6:0/2998012196 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bd4001c90 con 0x7f2bc80039b0 2024-03-04T09:33:03.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:03.536+0000 7f2bdbe00640 1 -- 172.21.15.6:0/2998012196 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bd4004bf0 con 0x7f2bc80039b0 2024-03-04T09:33:03.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:03.536+0000 7f2bdbe00640 1 -- 172.21.15.6:0/2998012196 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bd4007d50 con 0x7f2bc80039b0 2024-03-04T09:33:03.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:03.536+0000 7f2be9600640 1 -- 172.21.15.6:0/2998012196 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2be415ef80 con 0x7f2bc80039b0 2024-03-04T09:33:03.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:03.536+0000 7f2be9600640 1 -- 172.21.15.6:0/2998012196 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2be415f540 con 0x7f2bc80039b0 2024-03-04T09:33:03.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:03.537+0000 7f2be9600640 1 -- 172.21.15.6:0/2998012196 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2be404f330 con 0x7f2bc80039b0 2024-03-04T09:33:03.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:03.539+0000 7f2bdbe00640 1 -- 172.21.15.6:0/2998012196 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2bd4007070 con 0x7f2bc80039b0 2024-03-04T09:33:03.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:03.540+0000 7f2bdbe00640 1 --2- 172.21.15.6:0/2998012196 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2bc406b500 0x7f2bc406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:03.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:03.540+0000 7f2bdbe00640 1 -- 172.21.15.6:0/2998012196 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f2bd40ab570 con 0x7f2bc80039b0 2024-03-04T09:33:03.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:03.541+0000 7f2be3400640 1 --2- 172.21.15.6:0/2998012196 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2bc406b500 0x7f2bc406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:03.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:03.542+0000 7f2be3400640 1 --2- 172.21.15.6:0/2998012196 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2bc406b500 0x7f2bc406d9c0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f2be4160440 tx=0x7f2bd0008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:33:03.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:03.545+0000 7f2bdbe00640 1 -- 172.21.15.6:0/2998012196 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2bd4079ed0 con 0x7f2bc80039b0 2024-03-04T09:33:03.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:03.779+0000 7f2be9600640 1 -- 172.21.15.6:0/2998012196 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2be40724b0 con 0x7f2bc80039b0 2024-03-04T09:33:03.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:03.780+0000 7f2bdbe00640 1 -- 172.21.15.6:0/2998012196 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 92 v92) v1 ==== 94+0+4979 (secure 0 0 0) 0x7f2bd407df70 con 0x7f2bc80039b0 2024-03-04T09:33:03.778 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:33:03.778 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":92,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6098,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/2522244217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":2522244217},{"type":"v1","addr":"172.21.15.6:6835","nonce":2522244217}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":87},{"gid":6101,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6836/3149226637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":3149226637},{"type":"v1","addr":"172.21.15.146:6836","nonce":3149226637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":87},{"gid":6796,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":87}],"filesystems":[{"mdsmap":{"epoch":92,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:33:00.230788+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8214},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":9}]} 2024-03-04T09:33:03.779 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 92 2024-03-04T09:33:03.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:03.783+0000 7f2be9600640 1 -- 172.21.15.6:0/2998012196 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2bc406b500 msgr2=0x7f2bc406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:03.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:03.783+0000 7f2be9600640 1 --2- 172.21.15.6:0/2998012196 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2bc406b500 0x7f2bc406d9c0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f2be4160440 tx=0x7f2bd0008040 comp rx=0 tx=0).stop 2024-03-04T09:33:03.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:03.783+0000 7f2be9600640 1 -- 172.21.15.6:0/2998012196 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2bc80039b0 msgr2=0x7f2be41f31d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:03.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:03.783+0000 7f2be9600640 1 --2- 172.21.15.6:0/2998012196 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2bc80039b0 0x7f2be41f31d0 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7f2bd400c330 tx=0x7f2bd4001ab0 comp rx=0 tx=0).stop 2024-03-04T09:33:03.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:03.783+0000 7f2be9600640 1 -- 172.21.15.6:0/2998012196 shutdown_connections 2024-03-04T09:33:03.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:03.783+0000 7f2be9600640 1 --2- 172.21.15.6:0/2998012196 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2bc406b500 0x7f2bc406d9c0 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:03.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:03.783+0000 7f2be9600640 1 --2- 172.21.15.6:0/2998012196 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2be415e9b0 0x7f2be4160e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:03.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:03.783+0000 7f2be9600640 1 --2- 172.21.15.6:0/2998012196 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2be415e530 0x7f2be4160940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:03.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:03.783+0000 7f2be9600640 1 --2- 172.21.15.6:0/2998012196 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2bc80039b0 0x7f2be41f31d0 unknown :-1 s=CLOSED pgs=1508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:03.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:03.783+0000 7f2be9600640 1 -- 172.21.15.6:0/2998012196 >> 172.21.15.6:0/2998012196 conn(0x7f2be406c0c0 msgr2=0x7f2be40b2820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:33:03.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:03.783+0000 7f2be9600640 1 -- 172.21.15.6:0/2998012196 shutdown_connections 2024-03-04T09:33:03.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:03.783+0000 7f2be9600640 1 -- 172.21.15.6:0/2998012196 wait complete. 2024-03-04T09:33:04.200 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 92, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:32:57.159681+0000', 'modified': '2024-03-04T09:33:00.230788+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8214}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8214': {'gid': 8214, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2446627517', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2446627517}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2446627517}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8214, 'qdb_cluster': [8214]} 2024-03-04T09:33:04.200 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-04T09:33:04.200 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-04T09:33:04.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.475+0000 7fc57da00640 1 Processor -- start 2024-03-04T09:33:04.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.476+0000 7fc57da00640 1 -- start start 2024-03-04T09:33:04.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.477+0000 7fc57da00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc57815e0b0 0x7fc5780cf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:04.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.477+0000 7fc57da00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc5780d4a10 0x7fc5780d0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:04.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.477+0000 7fc57da00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5780b3710 con 0x7fc5780d0c80 2024-03-04T09:33:04.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.477+0000 7fc57da00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5780b09e0 con 0x7fc5780d4a10 2024-03-04T09:33:04.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.477+0000 7fc57da00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5780b39f0 con 0x7fc57815e0b0 2024-03-04T09:33:04.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.478+0000 7fc57d000640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fc5780d0c80 0x7fc5780d1900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:58088/0 (socket says 172.21.15.6:58088) 2024-03-04T09:33:04.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.478+0000 7fc57d000640 1 -- 172.21.15.6:0/3743605907 learned_addr learned my addr 172.21.15.6:0/3743605907 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:33:04.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.478+0000 7fc577400640 1 --2- 172.21.15.6:0/3743605907 >> v2:172.21.15.6:3300/0 conn(0x7fc5780d4a10 0x7fc5780d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:04.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.478+0000 7fc577e00640 1 --2- 172.21.15.6:0/3743605907 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc57815e0b0 0x7fc5780cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:04.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.478+0000 7fc576a00640 1 -- 172.21.15.6:0/3743605907 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3182786657 0 0) 0x7fc5780b3710 con 0x7fc5780d0c80 2024-03-04T09:33:04.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.478+0000 7fc576a00640 1 -- 172.21.15.6:0/3743605907 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc55c003520 con 0x7fc5780d0c80 2024-03-04T09:33:04.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.478+0000 7fc577400640 1 -- 172.21.15.6:0/3743605907 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc57815e0b0 msgr2=0x7fc5780cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:33:04.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.478+0000 7fc577400640 1 --2- 172.21.15.6:0/3743605907 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc57815e0b0 0x7fc5780cf310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:04.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.478+0000 7fc577400640 1 -- 172.21.15.6:0/3743605907 >> v1:172.21.15.6:6789/0 conn(0x7fc5780d0c80 legacy=0x7fc5780d1900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:04.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.478+0000 7fc577400640 1 -- 172.21.15.6:0/3743605907 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5780b3cd0 con 0x7fc5780d4a10 2024-03-04T09:33:04.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.480+0000 7fc577400640 1 --2- 172.21.15.6:0/3743605907 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc5780d4a10 0x7fc5780d0740 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x7fc5600097a0 tx=0x7fc56002b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1a4c1cd08ce9f41 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:33:04.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.480+0000 7fc576a00640 1 -- 172.21.15.6:0/3743605907 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc560035020 con 0x7fc5780d4a10 2024-03-04T09:33:04.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.480+0000 7fc576a00640 1 -- 172.21.15.6:0/3743605907 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc560031810 con 0x7fc5780d4a10 2024-03-04T09:33:04.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.480+0000 7fc576a00640 1 -- 172.21.15.6:0/3743605907 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc560031bc0 con 0x7fc5780d4a10 2024-03-04T09:33:04.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.481+0000 7fc57da00640 1 -- 172.21.15.6:0/3743605907 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc5780d4a10 msgr2=0x7fc5780d0740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:04.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.481+0000 7fc57da00640 1 --2- 172.21.15.6:0/3743605907 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc5780d4a10 0x7fc5780d0740 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x7fc5600097a0 tx=0x7fc56002b8b0 comp rx=0 tx=0).stop 2024-03-04T09:33:04.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.481+0000 7fc57da00640 1 -- 172.21.15.6:0/3743605907 shutdown_connections 2024-03-04T09:33:04.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.481+0000 7fc57da00640 1 --2- 172.21.15.6:0/3743605907 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc5780d4a10 0x7fc5780d0740 unknown :-1 s=CLOSED pgs=1510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:04.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.481+0000 7fc57da00640 1 --2- 172.21.15.6:0/3743605907 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc57815e0b0 0x7fc5780cf310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:04.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.481+0000 7fc57da00640 1 -- 172.21.15.6:0/3743605907 >> 172.21.15.6:0/3743605907 conn(0x7fc5780d3d30 msgr2=0x7fc5780d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:33:04.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.481+0000 7fc57da00640 1 -- 172.21.15.6:0/3743605907 shutdown_connections 2024-03-04T09:33:04.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.481+0000 7fc57da00640 1 -- 172.21.15.6:0/3743605907 wait complete. 2024-03-04T09:33:04.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.483+0000 7fc57da00640 1 Processor -- start 2024-03-04T09:33:04.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.483+0000 7fc57da00640 1 -- start start 2024-03-04T09:33:04.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.484+0000 7fc57da00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc5780d0c80 0x7fc5780b8ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:04.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.484+0000 7fc57da00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc5780d4a10 0x7fc5781606b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:04.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.484+0000 7fc57da00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc57815e0b0 0x7fc578160bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:04.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.484+0000 7fc57da00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5780cfc60 con 0x7fc5780d0c80 2024-03-04T09:33:04.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.484+0000 7fc57da00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc57815e490 con 0x7fc5780d4a10 2024-03-04T09:33:04.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.484+0000 7fc57da00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5780b54e0 con 0x7fc57815e0b0 2024-03-04T09:33:04.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.485+0000 7fc577e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc5780d0c80 0x7fc5780b8ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:04.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.485+0000 7fc577e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc5780d0c80 0x7fc5780b8ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43040/0 (socket says 172.21.15.6:43040) 2024-03-04T09:33:04.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.485+0000 7fc577e00640 1 -- 172.21.15.6:0/368998849 learned_addr learned my addr 172.21.15.6:0/368998849 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:33:04.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.485+0000 7fc57d000640 1 --2- 172.21.15.6:0/368998849 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc57815e0b0 0x7fc578160bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:04.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.485+0000 7fc577400640 1 --2- 172.21.15.6:0/368998849 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc5780d4a10 0x7fc5781606b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:04.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.485+0000 7fc577e00640 1 -- 172.21.15.6:0/368998849 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc57815e0b0 msgr2=0x7fc578160bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:04.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.485+0000 7fc577e00640 1 --2- 172.21.15.6:0/368998849 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc57815e0b0 0x7fc578160bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:04.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.485+0000 7fc577e00640 1 -- 172.21.15.6:0/368998849 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc5780d4a10 msgr2=0x7fc5781606b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:04.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.485+0000 7fc577e00640 1 --2- 172.21.15.6:0/368998849 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc5780d4a10 0x7fc5781606b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:04.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.485+0000 7fc577e00640 1 -- 172.21.15.6:0/368998849 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5780cbe50 con 0x7fc5780d0c80 2024-03-04T09:33:04.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.485+0000 7fc57d000640 1 --2- 172.21.15.6:0/368998849 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc57815e0b0 0x7fc578160bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:33:04.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.486+0000 7fc577e00640 1 --2- 172.21.15.6:0/368998849 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc5780d0c80 0x7fc5780b8ba0 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x7fc568007e60 tx=0x7fc56800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:33:04.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.486+0000 7fc56fe00640 1 -- 172.21.15.6:0/368998849 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc568014620 con 0x7fc5780d0c80 2024-03-04T09:33:04.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.486+0000 7fc56fe00640 1 -- 172.21.15.6:0/368998849 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc568019070 con 0x7fc5780d0c80 2024-03-04T09:33:04.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.486+0000 7fc56fe00640 1 -- 172.21.15.6:0/368998849 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc568014a20 con 0x7fc5780d0c80 2024-03-04T09:33:04.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.486+0000 7fc57da00640 1 -- 172.21.15.6:0/368998849 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5780cc130 con 0x7fc5780d0c80 2024-03-04T09:33:04.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.486+0000 7fc57da00640 1 -- 172.21.15.6:0/368998849 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5780cc6f0 con 0x7fc5780d0c80 2024-03-04T09:33:04.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.488+0000 7fc56fe00640 1 -- 172.21.15.6:0/368998849 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc568014bc0 con 0x7fc5780d0c80 2024-03-04T09:33:04.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.489+0000 7fc56fe00640 1 --2- 172.21.15.6:0/368998849 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc55406b4d0 0x7fc55406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:04.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.489+0000 7fc56fe00640 1 -- 172.21.15.6:0/368998849 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fc56808ab40 con 0x7fc5780d0c80 2024-03-04T09:33:04.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.490+0000 7fc577400640 1 --2- 172.21.15.6:0/368998849 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc55406b4d0 0x7fc55406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:04.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.490+0000 7fc581df7740 1 Processor -- start 2024-03-04T09:33:04.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.490+0000 7fc581df7740 1 -- start start 2024-03-04T09:33:04.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.491+0000 7fc581df7740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5593be5ea990 0x5593be5bd930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:04.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.491+0000 7fc581df7740 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5593be5ead70 0x5593be5ed720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:04.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.491+0000 7fc581df7740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5593be5bde70 0x5593be5edc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:04.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.491+0000 7fc581df7740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x5593be527600 con 0x5593be5ea990 2024-03-04T09:33:04.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.491+0000 7fc581df7740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x5593be4d0930 con 0x5593be5bde70 2024-03-04T09:33:04.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.491+0000 7fc581df7740 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x5593be5236a0 con 0x5593be5ead70 2024-03-04T09:33:04.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.491+0000 7fc577400640 1 --2- 172.21.15.6:0/368998849 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc55406b4d0 0x7fc55406d990 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fc560036000 tx=0x7fc560008030 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:33:04.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.491+0000 7fc577e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5593be5ea990 0x5593be5bd930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:04.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.491+0000 7fc577e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5593be5ea990 0x5593be5bd930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43054/0 (socket says 172.21.15.6:43054) 2024-03-04T09:33:04.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.491+0000 7fc577e00640 1 -- 172.21.15.6:0/1430275318 learned_addr learned my addr 172.21.15.6:0/1430275318 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:33:04.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.491+0000 7fc577e00640 1 -- 172.21.15.6:0/1430275318 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5593be5ead70 msgr2=0x5593be5ed720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:33:04.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.491+0000 7fc577e00640 1 --2- 172.21.15.6:0/1430275318 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5593be5ead70 0x5593be5ed720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:04.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.491+0000 7fc57d000640 1 --2- 172.21.15.6:0/1430275318 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5593be5bde70 0x5593be5edc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:04.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.491+0000 7fc577e00640 1 -- 172.21.15.6:0/1430275318 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5593be5bde70 msgr2=0x5593be5edc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:33:04.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.492+0000 7fc577e00640 1 --2- 172.21.15.6:0/1430275318 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5593be5bde70 0x5593be5edc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:04.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.492+0000 7fc577e00640 1 -- 172.21.15.6:0/1430275318 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5593be23c2d0 con 0x5593be5ea990 2024-03-04T09:33:04.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.492+0000 7fc577e00640 1 --2- 172.21.15.6:0/1430275318 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5593be5ea990 0x5593be5bd930 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7fc56800c040 tx=0x7fc568059a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7538ab2de22cc0b server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:33:04.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.492+0000 7fc56cc00640 1 -- 172.21.15.6:0/1430275318 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc568061980 con 0x5593be5ea990 2024-03-04T09:33:04.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.492+0000 7fc56cc00640 1 -- 172.21.15.6:0/1430275318 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc56805d170 con 0x5593be5ea990 2024-03-04T09:33:04.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.492+0000 7fc56cc00640 1 -- 172.21.15.6:0/1430275318 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5680558e0 con 0x5593be5ea990 2024-03-04T09:33:04.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.493+0000 7fc581df7740 1 -- 172.21.15.6:0/1430275318 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5593be5ea990 msgr2=0x5593be5bd930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:04.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.493+0000 7fc581df7740 1 --2- 172.21.15.6:0/1430275318 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5593be5ea990 0x5593be5bd930 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7fc56800c040 tx=0x7fc568059a30 comp rx=0 tx=0).stop 2024-03-04T09:33:04.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.493+0000 7fc581df7740 1 -- 172.21.15.6:0/1430275318 shutdown_connections 2024-03-04T09:33:04.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.493+0000 7fc581df7740 1 --2- 172.21.15.6:0/1430275318 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5593be5bde70 0x5593be5edc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:04.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.493+0000 7fc581df7740 1 --2- 172.21.15.6:0/1430275318 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5593be5ead70 0x5593be5ed720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:04.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.493+0000 7fc581df7740 1 --2- 172.21.15.6:0/1430275318 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5593be5ea990 0x5593be5bd930 unknown :-1 s=CLOSED pgs=1512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:04.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.493+0000 7fc581df7740 1 -- 172.21.15.6:0/1430275318 >> 172.21.15.6:0/1430275318 conn(0x5593be5bfba0 msgr2=0x5593be608720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:33:04.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.493+0000 7fc581df7740 1 -- 172.21.15.6:0/1430275318 shutdown_connections 2024-03-04T09:33:04.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.493+0000 7fc581df7740 1 -- 172.21.15.6:0/1430275318 wait complete. 2024-03-04T09:33:04.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.494+0000 7fc581df7740 1 -- start start 2024-03-04T09:33:04.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.494+0000 7fc581df7740 1 Processor -- start 2024-03-04T09:33:04.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.495+0000 7fc581df7740 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5593be5ea990 0x5593be5f0250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:04.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.495+0000 7fc581df7740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5593be5ead70 0x5593be5ff520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:04.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.495+0000 7fc581df7740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5593be6187f0 0x5593be5db3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:04.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.495+0000 7fc581df7740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x5593be5c2140 con 0x5593be6187f0 2024-03-04T09:33:04.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.495+0000 7fc581df7740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x5593be4e3d90 con 0x5593be5ead70 2024-03-04T09:33:04.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.495+0000 7fc581df7740 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x5593be5c1530 con 0x5593be5ea990 2024-03-04T09:33:04.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.495+0000 7fc581df7740 10 client.0 _ll_register_callbacks cb 0x5593be51f350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T09:33:04.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.495+0000 7fc57d000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5593be6187f0 0x5593be5db3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:04.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.495+0000 7fc57d000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5593be6187f0 0x5593be5db3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43064/0 (socket says 172.21.15.6:43064) 2024-03-04T09:33:04.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.495+0000 7fc57d000640 1 -- 172.21.15.6:0/3678786392 learned_addr learned my addr 172.21.15.6:0/3678786392 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:33:04.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.495+0000 7fc577400640 1 --2- 172.21.15.6:0/3678786392 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5593be5ead70 0x5593be5ff520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:04.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.496+0000 7fc577e00640 1 --2- 172.21.15.6:0/3678786392 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5593be5ea990 0x5593be5f0250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:04.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.496+0000 7fc57d000640 1 -- 172.21.15.6:0/3678786392 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5593be5ea990 msgr2=0x5593be5f0250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:33:04.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.496+0000 7fc57d000640 1 --2- 172.21.15.6:0/3678786392 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5593be5ea990 0x5593be5f0250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:04.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.496+0000 7fc57d000640 1 -- 172.21.15.6:0/3678786392 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5593be5ead70 msgr2=0x5593be5ff520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:33:04.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.496+0000 7fc57d000640 1 --2- 172.21.15.6:0/3678786392 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5593be5ead70 0x5593be5ff520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:04.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.496+0000 7fc57d000640 1 -- 172.21.15.6:0/3678786392 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5593be3066b0 con 0x5593be6187f0 2024-03-04T09:33:04.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.496+0000 7fc57d000640 1 --2- 172.21.15.6:0/3678786392 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5593be6187f0 0x5593be5db3d0 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7fc570001db0 tx=0x7fc57000e590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=278f354237a6d6f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:33:04.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.496+0000 7fc565600640 10 client.0 ms_handle_connect on v2:172.21.15.6:3300/0 2024-03-04T09:33:04.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.496+0000 7fc565600640 1 -- 172.21.15.6:0/3678786392 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc570005a50 con 0x5593be6187f0 2024-03-04T09:33:04.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.496+0000 7fc565600640 1 -- 172.21.15.6:0/3678786392 <== mon.0 v2:172.21.15.6:3300/0 2 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fc570005bf0 con 0x5593be6187f0 2024-03-04T09:33:04.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.496+0000 7fc581df7740 1 -- 172.21.15.6:0/3678786392 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc56805d170 con 0x5593be6187f0 2024-03-04T09:33:04.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.497+0000 7fc565600640 1 -- 172.21.15.6:0/3678786392 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_get_version_reply(handle=1 version=92) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc570005df0 con 0x5593be6187f0 2024-03-04T09:33:04.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.497+0000 7fc581df7740 10 client.8367 fetch_fsmap learned FSMap version 92 2024-03-04T09:33:04.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.497+0000 7fc581df7740 1 -- 172.21.15.6:0/3678786392 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,fsmap=92,monmap=2+}) v3 -- 0x5593be5efa50 con 0x5593be6187f0 2024-03-04T09:33:04.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.497+0000 7fc565600640 1 -- 172.21.15.6:0/3678786392 <== mon.0 v2:172.21.15.6:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc56805d170 con 0x5593be6187f0 2024-03-04T09:33:04.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.497+0000 7fc565600640 1 -- 172.21.15.6:0/3678786392 <== mon.0 v2:172.21.15.6:3300/0 5 ==== fsmap(e 92) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fc57001b030 con 0x5593be6187f0 2024-03-04T09:33:04.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.497+0000 7fc581df7740 10 client.8367 fetch_fsmap finished waiting for FSMap version 92 2024-03-04T09:33:04.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.497+0000 7fc581df7740 10 client.8367 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-04T09:33:04.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.497+0000 7fc581df7740 20 client.8367 populate_metadata read hostname 'smithi006' 2024-03-04T09:33:04.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.497+0000 7fc581df7740 1 --2- 172.21.15.6:0/3678786392 >> [v2:172.21.15.6:6832/2446627517,v1:172.21.15.6:6834/2446627517] conn(0x5593be600660 0x5593be6274f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:04.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.497+0000 7fc581df7740 4 client.8367 mds_command: new command op to 8214 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T09:33:04.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.497+0000 7fc581df7740 1 -- 172.21.15.6:0/3678786392 --> [v2:172.21.15.6:6832/2446627517,v1:172.21.15.6:6834/2446627517] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5593be534b40 con 0x5593be600660 2024-03-04T09:33:04.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.498+0000 7fc577e00640 1 --2- 172.21.15.6:0/3678786392 >> [v2:172.21.15.6:6832/2446627517,v1:172.21.15.6:6834/2446627517] conn(0x5593be600660 0x5593be6274f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:04.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.498+0000 7fc565600640 10 client.8367 ms_handle_connect on v2:172.21.15.6:6832/2446627517 2024-03-04T09:33:04.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.498+0000 7fc577e00640 1 --2- 172.21.15.6:0/3678786392 >> [v2:172.21.15.6:6832/2446627517,v1:172.21.15.6:6834/2446627517] conn(0x5593be600660 0x5593be6274f0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a6cd0e2ba589135c server_cookie=477a38a63452a1e6 in_seq=0 out_seq=0 2024-03-04T09:33:04.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.499+0000 7fc565600640 1 -- 172.21.15.6:0/3678786392 <== mds.0 v2:172.21.15.6:6832/2446627517 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5593be534b40 con 0x5593be600660 2024-03-04T09:33:04.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.499+0000 7fc565600640 10 client.8367 handle_command_reply: tid=0 2024-03-04T09:33:04.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.499+0000 7fc581df7740 1 client.8367 shutdown 2024-03-04T09:33:04.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.499+0000 7fc581df7740 1 -- 172.21.15.6:0/3678786392 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5593be6187f0 msgr2=0x5593be5db3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:04.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.499+0000 7fc581df7740 1 --2- 172.21.15.6:0/3678786392 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5593be6187f0 0x5593be5db3d0 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7fc570001db0 tx=0x7fc57000e590 comp rx=0 tx=0).stop 2024-03-04T09:33:04.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.499+0000 7fc581df7740 1 -- 172.21.15.6:0/3678786392 shutdown_connections 2024-03-04T09:33:04.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.499+0000 7fc581df7740 1 --2- 172.21.15.6:0/3678786392 >> [v2:172.21.15.6:6832/2446627517,v1:172.21.15.6:6834/2446627517] conn(0x5593be600660 0x5593be6274f0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:04.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.499+0000 7fc581df7740 1 --2- 172.21.15.6:0/3678786392 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5593be6187f0 0x5593be5db3d0 unknown :-1 s=CLOSED pgs=1513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:04.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.499+0000 7fc581df7740 1 --2- 172.21.15.6:0/3678786392 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5593be5ead70 0x5593be5ff520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:04.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.499+0000 7fc581df7740 1 --2- 172.21.15.6:0/3678786392 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5593be5ea990 0x5593be5f0250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:04.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.499+0000 7fc581df7740 1 -- 172.21.15.6:0/3678786392 >> 172.21.15.6:0/3678786392 conn(0x5593be5bde70 msgr2=0x5593be5ed710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:33:04.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.499+0000 7fc565600640 0 client.8367 ms_handle_reset on v2:172.21.15.6:6832/2446627517 2024-03-04T09:33:04.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.500+0000 7fc581df7740 1 -- 172.21.15.6:0/3678786392 shutdown_connections 2024-03-04T09:33:04.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.500+0000 7fc581df7740 1 -- 172.21.15.6:0/3678786392 wait complete. 2024-03-04T09:33:04.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.500+0000 7fc581df7740 20 client.8367 trim_cache size 0 max 16384 2024-03-04T09:33:04.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.518+0000 7fc581df7740 1 Processor -- start 2024-03-04T09:33:04.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.518+0000 7fc581df7740 1 -- start start 2024-03-04T09:33:04.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.519+0000 7fc581df7740 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5593be5ea990 0x5593be62c520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:04.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.519+0000 7fc581df7740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5593be5ead70 0x5593be5f0c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:04.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.519+0000 7fc581df7740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5593be6187f0 0x5593be5f11a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:04.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.519+0000 7fc581df7740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5781ff160 con 0x5593be5ead70 2024-03-04T09:33:04.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.519+0000 7fc581df7740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5781f8420 con 0x5593be6187f0 2024-03-04T09:33:04.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.519+0000 7fc581df7740 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5781feb60 con 0x5593be5ea990 2024-03-04T09:33:04.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.519+0000 7fc577400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5593be5ead70 0x5593be5f0c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:04.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.519+0000 7fc577400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5593be5ead70 0x5593be5f0c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43076/0 (socket says 172.21.15.6:43076) 2024-03-04T09:33:04.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.519+0000 7fc577400640 1 -- 172.21.15.6:0/3703607318 learned_addr learned my addr 172.21.15.6:0/3703607318 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:33:04.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.520+0000 7fc577e00640 1 --2- 172.21.15.6:0/3703607318 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5593be5ea990 0x5593be62c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:04.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.520+0000 7fc57d000640 1 --2- 172.21.15.6:0/3703607318 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5593be6187f0 0x5593be5f11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:04.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.520+0000 7fc577400640 1 -- 172.21.15.6:0/3703607318 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5593be5ea990 msgr2=0x5593be62c520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:33:04.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.520+0000 7fc577400640 1 --2- 172.21.15.6:0/3703607318 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5593be5ea990 0x5593be62c520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:04.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.520+0000 7fc577400640 1 -- 172.21.15.6:0/3703607318 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5593be6187f0 msgr2=0x5593be5f11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:33:04.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.520+0000 7fc577400640 1 --2- 172.21.15.6:0/3703607318 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5593be6187f0 0x5593be5f11a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:04.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.520+0000 7fc577400640 1 -- 172.21.15.6:0/3703607318 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5593be5eef40 con 0x5593be5ead70 2024-03-04T09:33:04.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.520+0000 7fc57d000640 1 --2- 172.21.15.6:0/3703607318 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5593be6187f0 0x5593be5f11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:33:04.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.520+0000 7fc577400640 1 --2- 172.21.15.6:0/3703607318 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5593be5ead70 0x5593be5f0c60 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7fc560009890 tx=0x7fc56002e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=600e17ad7a3209c7 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:33:04.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.521+0000 7fc567e00640 1 -- 172.21.15.6:0/3703607318 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc560053070 con 0x5593be5ead70 2024-03-04T09:33:04.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.521+0000 7fc567e00640 1 -- 172.21.15.6:0/3703607318 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc560031d40 con 0x5593be5ead70 2024-03-04T09:33:04.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.521+0000 7fc567e00640 1 -- 172.21.15.6:0/3703607318 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc560035040 con 0x5593be5ead70 2024-03-04T09:33:04.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.521+0000 7fc581df7740 1 -- 172.21.15.6:0/3703607318 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5593be5ead70 msgr2=0x5593be5f0c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:04.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.521+0000 7fc581df7740 1 --2- 172.21.15.6:0/3703607318 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5593be5ead70 0x5593be5f0c60 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7fc560009890 tx=0x7fc56002e000 comp rx=0 tx=0).stop 2024-03-04T09:33:04.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.521+0000 7fc581df7740 1 -- 172.21.15.6:0/3703607318 shutdown_connections 2024-03-04T09:33:04.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.521+0000 7fc581df7740 1 --2- 172.21.15.6:0/3703607318 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5593be6187f0 0x5593be5f11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:04.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.521+0000 7fc581df7740 1 --2- 172.21.15.6:0/3703607318 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5593be5ead70 0x5593be5f0c60 unknown :-1 s=CLOSED pgs=1514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:04.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.521+0000 7fc581df7740 1 --2- 172.21.15.6:0/3703607318 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5593be5ea990 0x5593be62c520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:04.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.521+0000 7fc581df7740 1 -- 172.21.15.6:0/3703607318 >> 172.21.15.6:0/3703607318 conn(0x5593be5bde70 msgr2=0x5593be5ff390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:33:04.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.521+0000 7fc581df7740 1 -- 172.21.15.6:0/3703607318 shutdown_connections 2024-03-04T09:33:04.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.521+0000 7fc581df7740 1 -- 172.21.15.6:0/3703607318 wait complete. 2024-03-04T09:33:04.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.522+0000 7fc581df7740 1 -- start start 2024-03-04T09:33:04.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.522+0000 7fc581df7740 1 Processor -- start 2024-03-04T09:33:04.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.523+0000 7fc581df7740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5593be5ea990 0x5593be677b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:04.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.523+0000 7fc581df7740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5593be5ead70 0x5593be5c2630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:04.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.523+0000 7fc581df7740 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5593be6187f0 0x5593be5bc7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:04.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.523+0000 7fc581df7740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x5593be5eec20 con 0x5593be5ead70 2024-03-04T09:33:04.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.523+0000 7fc581df7740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x5593be5efd30 con 0x5593be5ea990 2024-03-04T09:33:04.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.523+0000 7fc581df7740 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x5593be6025b0 con 0x5593be6187f0 2024-03-04T09:33:04.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.523+0000 7fc581df7740 10 client.0 _ll_register_callbacks cb 0x7fc570007b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T09:33:04.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.524+0000 7fc577400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5593be5ead70 0x5593be5c2630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:04.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.524+0000 7fc577400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5593be5ead70 0x5593be5c2630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43090/0 (socket says 172.21.15.6:43090) 2024-03-04T09:33:04.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.524+0000 7fc577400640 1 -- 172.21.15.6:0/3295221139 learned_addr learned my addr 172.21.15.6:0/3295221139 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:33:04.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.524+0000 7fc577e00640 1 --2- 172.21.15.6:0/3295221139 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5593be5ea990 0x5593be677b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:04.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.524+0000 7fc57d000640 1 --2- 172.21.15.6:0/3295221139 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5593be6187f0 0x5593be5bc7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:04.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.524+0000 7fc577400640 1 -- 172.21.15.6:0/3295221139 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5593be6187f0 msgr2=0x5593be5bc7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:33:04.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.524+0000 7fc577400640 1 --2- 172.21.15.6:0/3295221139 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5593be6187f0 0x5593be5bc7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:04.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.524+0000 7fc577400640 1 -- 172.21.15.6:0/3295221139 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5593be5ea990 msgr2=0x5593be677b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:33:04.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.524+0000 7fc577400640 1 --2- 172.21.15.6:0/3295221139 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5593be5ea990 0x5593be677b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:04.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.524+0000 7fc577400640 1 -- 172.21.15.6:0/3295221139 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5593be5f0690 con 0x5593be5ead70 2024-03-04T09:33:04.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.524+0000 7fc577400640 1 --2- 172.21.15.6:0/3295221139 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5593be5ead70 0x5593be5c2630 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 crypto rx=0x7fc560051040 tx=0x7fc560031000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7520d107a6b0818 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:33:04.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.524+0000 7fc565600640 10 client.0 ms_handle_connect on v2:172.21.15.6:3300/0 2024-03-04T09:33:04.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.525+0000 7fc565600640 1 -- 172.21.15.6:0/3295221139 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc560053020 con 0x5593be5ead70 2024-03-04T09:33:04.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.525+0000 7fc565600640 1 -- 172.21.15.6:0/3295221139 <== mon.0 v2:172.21.15.6:3300/0 2 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fc56004e900 con 0x5593be5ead70 2024-03-04T09:33:04.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.525+0000 7fc581df7740 1 -- 172.21.15.6:0/3295221139 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc560031d40 con 0x5593be5ead70 2024-03-04T09:33:04.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.525+0000 7fc565600640 1 -- 172.21.15.6:0/3295221139 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_get_version_reply(handle=1 version=92) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc560035070 con 0x5593be5ead70 2024-03-04T09:33:04.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.525+0000 7fc581df7740 10 client.8373 fetch_fsmap learned FSMap version 92 2024-03-04T09:33:04.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.525+0000 7fc581df7740 1 -- 172.21.15.6:0/3295221139 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,fsmap=92,monmap=2+}) v3 -- 0x5593be60c8e0 con 0x5593be5ead70 2024-03-04T09:33:04.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.525+0000 7fc565600640 1 -- 172.21.15.6:0/3295221139 <== mon.0 v2:172.21.15.6:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc560031d40 con 0x5593be5ead70 2024-03-04T09:33:04.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.525+0000 7fc565600640 1 -- 172.21.15.6:0/3295221139 <== mon.0 v2:172.21.15.6:3300/0 5 ==== fsmap(e 92) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fc56004d070 con 0x5593be5ead70 2024-03-04T09:33:04.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.525+0000 7fc581df7740 10 client.8373 fetch_fsmap finished waiting for FSMap version 92 2024-03-04T09:33:04.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.525+0000 7fc581df7740 10 client.8373 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-04T09:33:04.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.525+0000 7fc581df7740 20 client.8373 populate_metadata read hostname 'smithi006' 2024-03-04T09:33:04.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.525+0000 7fc581df7740 1 --2- 172.21.15.6:0/3295221139 >> [v2:172.21.15.6:6832/2446627517,v1:172.21.15.6:6834/2446627517] conn(0x5593be5feaf0 0x5593be616e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:04.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.526+0000 7fc581df7740 4 client.8373 mds_command: new command op to 8214 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-04T09:33:04.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.526+0000 7fc581df7740 1 -- 172.21.15.6:0/3295221139 --> [v2:172.21.15.6:6832/2446627517,v1:172.21.15.6:6834/2446627517] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5593be52ee90 con 0x5593be5feaf0 2024-03-04T09:33:04.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.526+0000 7fc577e00640 1 --2- 172.21.15.6:0/3295221139 >> [v2:172.21.15.6:6832/2446627517,v1:172.21.15.6:6834/2446627517] conn(0x5593be5feaf0 0x5593be616e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:04.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.526+0000 7fc577e00640 1 --2- 172.21.15.6:0/3295221139 >> [v2:172.21.15.6:6832/2446627517,v1:172.21.15.6:6834/2446627517] conn(0x5593be5feaf0 0x5593be616e40 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9b56954bb56f76da server_cookie=aac52223bd76c51e in_seq=0 out_seq=0 2024-03-04T09:33:04.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.526+0000 7fc565600640 10 client.8373 ms_handle_connect on v2:172.21.15.6:6832/2446627517 2024-03-04T09:33:04.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.527+0000 7fc565600640 1 -- 172.21.15.6:0/3295221139 <== mds.0 v2:172.21.15.6:6832/2446627517 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x5593be52ee90 con 0x5593be5feaf0 2024-03-04T09:33:04.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.527+0000 7fc565600640 10 client.8373 handle_command_reply: tid=0 2024-03-04T09:33:04.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.527+0000 7fc581df7740 1 client.8373 shutdown 2024-03-04T09:33:04.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.527+0000 7fc581df7740 1 -- 172.21.15.6:0/3295221139 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5593be5ead70 msgr2=0x5593be5c2630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:04.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.527+0000 7fc581df7740 1 --2- 172.21.15.6:0/3295221139 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5593be5ead70 0x5593be5c2630 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 crypto rx=0x7fc560051040 tx=0x7fc560031000 comp rx=0 tx=0).stop 2024-03-04T09:33:04.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.527+0000 7fc581df7740 1 -- 172.21.15.6:0/3295221139 shutdown_connections 2024-03-04T09:33:04.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.527+0000 7fc581df7740 1 --2- 172.21.15.6:0/3295221139 >> [v2:172.21.15.6:6832/2446627517,v1:172.21.15.6:6834/2446627517] conn(0x5593be5feaf0 0x5593be616e40 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:04.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.527+0000 7fc581df7740 1 --2- 172.21.15.6:0/3295221139 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5593be6187f0 0x5593be5bc7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:04.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.527+0000 7fc581df7740 1 --2- 172.21.15.6:0/3295221139 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5593be5ead70 0x5593be5c2630 unknown :-1 s=CLOSED pgs=1515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:04.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.527+0000 7fc581df7740 1 --2- 172.21.15.6:0/3295221139 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5593be5ea990 0x5593be677b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:04.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.527+0000 7fc565600640 0 client.8373 ms_handle_reset on v2:172.21.15.6:6832/2446627517 2024-03-04T09:33:04.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.527+0000 7fc581df7740 1 -- 172.21.15.6:0/3295221139 >> 172.21.15.6:0/3295221139 conn(0x5593be5bde70 msgr2=0x5593be5ff390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:33:04.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.527+0000 7fc581df7740 1 -- 172.21.15.6:0/3295221139 shutdown_connections 2024-03-04T09:33:04.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.527+0000 7fc581df7740 1 -- 172.21.15.6:0/3295221139 wait complete. 2024-03-04T09:33:04.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.527+0000 7fc581df7740 20 client.8373 trim_cache size 0 max 16384 2024-03-04T09:33:04.529 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-04T09:33:04.529 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_fsid": "edce0578-5ae4-4402-b45a-552bdef84d7c", 2024-03-04T09:33:04.529 INFO:teuthology.orchestra.run.smithi006.stdout: "whoami": 0, 2024-03-04T09:33:04.529 INFO:teuthology.orchestra.run.smithi006.stdout: "id": 8214, 2024-03-04T09:33:04.530 INFO:teuthology.orchestra.run.smithi006.stdout: "want_state": "up:active", 2024-03-04T09:33:04.530 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "up:active", 2024-03-04T09:33:04.530 INFO:teuthology.orchestra.run.smithi006.stdout: "fs_name": "cephfs", 2024-03-04T09:33:04.530 INFO:teuthology.orchestra.run.smithi006.stdout: "rank_uptime": 6.8547842550000002, 2024-03-04T09:33:04.530 INFO:teuthology.orchestra.run.smithi006.stdout: "mdsmap_epoch": 92, 2024-03-04T09:33:04.530 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap_epoch": 129, 2024-03-04T09:33:04.530 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap_epoch_barrier": 127, 2024-03-04T09:33:04.530 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 13.447608888 2024-03-04T09:33:04.530 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-04T09:33:04.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.528+0000 7fc57da00640 1 -- 172.21.15.6:0/368998849 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc55406b4d0 msgr2=0x7fc55406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:04.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.528+0000 7fc57da00640 1 --2- 172.21.15.6:0/368998849 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc55406b4d0 0x7fc55406d990 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fc560036000 tx=0x7fc560008030 comp rx=0 tx=0).stop 2024-03-04T09:33:04.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.528+0000 7fc57da00640 1 -- 172.21.15.6:0/368998849 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc5780d0c80 msgr2=0x7fc5780b8ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:04.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.528+0000 7fc57da00640 1 --2- 172.21.15.6:0/368998849 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc5780d0c80 0x7fc5780b8ba0 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x7fc568007e60 tx=0x7fc56800e4b0 comp rx=0 tx=0).stop 2024-03-04T09:33:04.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.528+0000 7fc57da00640 1 -- 172.21.15.6:0/368998849 shutdown_connections 2024-03-04T09:33:04.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.528+0000 7fc57da00640 1 --2- 172.21.15.6:0/368998849 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc55406b4d0 0x7fc55406d990 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:04.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.528+0000 7fc57da00640 1 --2- 172.21.15.6:0/368998849 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc57815e0b0 0x7fc578160bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:04.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.528+0000 7fc57da00640 1 --2- 172.21.15.6:0/368998849 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc5780d4a10 0x7fc5781606b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:04.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.528+0000 7fc57da00640 1 --2- 172.21.15.6:0/368998849 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc5780d0c80 0x7fc5780b8ba0 unknown :-1 s=CLOSED pgs=1511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:04.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.528+0000 7fc57da00640 1 -- 172.21.15.6:0/368998849 >> 172.21.15.6:0/368998849 conn(0x7fc5780d3d30 msgr2=0x7fc5780cf070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:33:04.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.528+0000 7fc57da00640 1 -- 172.21.15.6:0/368998849 shutdown_connections 2024-03-04T09:33:04.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.528+0000 7fc57da00640 1 -- 172.21.15.6:0/368998849 wait complete. 2024-03-04T09:33:04.545 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-04T09:33:04.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.752+0000 7f7a70000640 1 Processor -- start 2024-03-04T09:33:04.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.752+0000 7f7a70000640 1 -- start start 2024-03-04T09:33:04.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.755+0000 7f7a70000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f7a680cdc20 0x7f7a68071760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:04.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.755+0000 7f7a70000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a680b4860 con 0x7f7a680cf5c0 2024-03-04T09:33:04.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.755+0000 7f7a70000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a680b4b40 con 0x7f7a680b3c90 2024-03-04T09:33:04.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.755+0000 7f7a70000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a680b4e20 con 0x7f7a680cdc20 2024-03-04T09:33:04.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.755+0000 7f7a6cc00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f7a680cf5c0 0x7f7a68071050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:58096/0 (socket says 172.21.15.6:58096) 2024-03-04T09:33:04.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.755+0000 7f7a6cc00640 1 -- 172.21.15.6:0/2411496796 learned_addr learned my addr 172.21.15.6:0/2411496796 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:33:04.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.756+0000 7f7a6e000640 1 --2- 172.21.15.6:0/2411496796 >> v2:172.21.15.146:3300/0 conn(0x7f7a680cdc20 0x7f7a68071760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:04.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.756+0000 7f7a67e00640 1 -- 172.21.15.6:0/2411496796 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2446864116 0 0) 0x7f7a680b4860 con 0x7f7a680cf5c0 2024-03-04T09:33:04.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.756+0000 7f7a67e00640 1 -- 172.21.15.6:0/2411496796 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a50005730 con 0x7f7a680cf5c0 2024-03-04T09:33:04.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.756+0000 7f7a67e00640 1 -- 172.21.15.6:0/2411496796 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3870582688 0 0) 0x7f7a50005730 con 0x7f7a680cf5c0 2024-03-04T09:33:04.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.756+0000 7f7a67e00640 1 -- 172.21.15.6:0/2411496796 >> v2:172.21.15.146:3300/0 conn(0x7f7a680cdc20 msgr2=0x7f7a68071760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:33:04.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.756+0000 7f7a67e00640 1 --2- 172.21.15.6:0/2411496796 >> v2:172.21.15.146:3300/0 conn(0x7f7a680cdc20 0x7f7a68071760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:04.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.756+0000 7f7a67e00640 1 -- 172.21.15.6:0/2411496796 >> v1:172.21.15.146:6789/0 conn(0x7f7a680b3c90 legacy=0x7f7a680b4070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:04.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.756+0000 7f7a67e00640 1 -- 172.21.15.6:0/2411496796 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a680b5100 con 0x7f7a680cf5c0 2024-03-04T09:33:04.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.756+0000 7f7a67e00640 1 -- 172.21.15.6:0/2411496796 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7f7a54001670 con 0x7f7a680cf5c0 2024-03-04T09:33:04.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.756+0000 7f7a67e00640 1 -- 172.21.15.6:0/2411496796 >> v1:172.21.15.6:6789/0 conn(0x7f7a680cf5c0 legacy=0x7f7a68071050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:04.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.756+0000 7f7a67e00640 1 --2- 172.21.15.6:0/2411496796 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7a50005c80 0x7f7a500060a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:04.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.757+0000 7f7a67e00640 1 --2- 172.21.15.6:0/2411496796 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7a50006a10 0x7f7a50006e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:04.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.757+0000 7f7a67e00640 1 --2- 172.21.15.6:0/2411496796 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7a50007420 0x7f7a5000dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:04.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.757+0000 7f7a67e00640 1 -- 172.21.15.6:0/2411496796 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a680b4e20 con 0x7f7a50006a10 2024-03-04T09:33:04.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.757+0000 7f7a67e00640 1 -- 172.21.15.6:0/2411496796 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a5000e330 con 0x7f7a50007420 2024-03-04T09:33:04.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.757+0000 7f7a67e00640 1 -- 172.21.15.6:0/2411496796 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a5000e530 con 0x7f7a50005c80 2024-03-04T09:33:04.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.757+0000 7f7a6cc00640 1 --2- 172.21.15.6:0/2411496796 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7a50006a10 0x7f7a50006e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:04.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.757+0000 7f7a6d600640 1 --2- 172.21.15.6:0/2411496796 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7a50005c80 0x7f7a500060a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:04.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.757+0000 7f7a6e000640 1 --2- 172.21.15.6:0/2411496796 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7a50007420 0x7f7a5000dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:04.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.757+0000 7f7a6cc00640 1 -- 172.21.15.6:0/2411496796 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7a50005c80 msgr2=0x7f7a500060a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:33:04.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.757+0000 7f7a6cc00640 1 --2- 172.21.15.6:0/2411496796 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7a50005c80 0x7f7a500060a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:04.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.757+0000 7f7a6cc00640 1 -- 172.21.15.6:0/2411496796 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7a50007420 msgr2=0x7f7a5000dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:33:04.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.757+0000 7f7a6cc00640 1 --2- 172.21.15.6:0/2411496796 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7a50007420 0x7f7a5000dbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:04.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.758+0000 7f7a6e000640 1 -- 172.21.15.6:0/2411496796 reap_dead start 2024-03-04T09:33:04.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.758+0000 7f7a6cc00640 1 -- 172.21.15.6:0/2411496796 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a50005730 con 0x7f7a50006a10 2024-03-04T09:33:04.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.759+0000 7f7a6cc00640 1 --2- 172.21.15.6:0/2411496796 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7a50006a10 0x7f7a50006e90 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7f7a5400c110 tx=0x7f7a50007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d551b40ef8c524a3 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:33:04.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.761+0000 7f7a67e00640 1 -- 172.21.15.6:0/2411496796 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a54007c40 con 0x7f7a50006a10 2024-03-04T09:33:04.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.761+0000 7f7a67e00640 1 -- 172.21.15.6:0/2411496796 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a54030070 con 0x7f7a50006a10 2024-03-04T09:33:04.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.762+0000 7f7a67e00640 1 -- 172.21.15.6:0/2411496796 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a54005d70 con 0x7f7a50006a10 2024-03-04T09:33:04.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.762+0000 7f7a70000640 1 -- 172.21.15.6:0/2411496796 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7a50006a10 msgr2=0x7f7a50006e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:04.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.762+0000 7f7a70000640 1 --2- 172.21.15.6:0/2411496796 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7a50006a10 0x7f7a50006e90 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7f7a5400c110 tx=0x7f7a50007a40 comp rx=0 tx=0).stop 2024-03-04T09:33:04.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.762+0000 7f7a70000640 1 -- 172.21.15.6:0/2411496796 shutdown_connections 2024-03-04T09:33:04.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.762+0000 7f7a70000640 1 --2- 172.21.15.6:0/2411496796 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7a50006a10 0x7f7a50006e90 unknown :-1 s=CLOSED pgs=1517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:04.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.762+0000 7f7a70000640 1 -- 172.21.15.6:0/2411496796 >> 172.21.15.6:0/2411496796 conn(0x7f7a6806afb0 msgr2=0x7f7a6806b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:33:04.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.762+0000 7f7a70000640 1 -- 172.21.15.6:0/2411496796 shutdown_connections 2024-03-04T09:33:04.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.762+0000 7f7a70000640 1 -- 172.21.15.6:0/2411496796 wait complete. 2024-03-04T09:33:04.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.764+0000 7f7a70000640 1 Processor -- start 2024-03-04T09:33:04.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.764+0000 7f7a70000640 1 -- start start 2024-03-04T09:33:04.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.765+0000 7f7a70000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7a50006a10 0x7f7a68142420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:04.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.765+0000 7f7a70000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7a68142960 0x7f7a68070f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:04.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.765+0000 7f7a70000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7a68071450 0x7f7a680718d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:04.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.765+0000 7f7a70000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a680b4070 con 0x7f7a68071450 2024-03-04T09:33:04.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.765+0000 7f7a70000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a680ce060 con 0x7f7a68142960 2024-03-04T09:33:04.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.765+0000 7f7a70000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a680b41f0 con 0x7f7a50006a10 2024-03-04T09:33:04.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.765+0000 7f7a6e000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7a68071450 0x7f7a680718d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:04.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.765+0000 7f7a6e000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7a68071450 0x7f7a680718d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43106/0 (socket says 172.21.15.6:43106) 2024-03-04T09:33:04.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.765+0000 7f7a6e000640 1 -- 172.21.15.6:0/2105045900 learned_addr learned my addr 172.21.15.6:0/2105045900 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:33:04.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.766+0000 7f7a6d600640 1 --2- 172.21.15.6:0/2105045900 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7a50006a10 0x7f7a68142420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:04.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.766+0000 7f7a6cc00640 1 --2- 172.21.15.6:0/2105045900 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7a68142960 0x7f7a68070f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:04.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.766+0000 7f7a6e000640 1 -- 172.21.15.6:0/2105045900 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7a50006a10 msgr2=0x7f7a68142420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:04.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.766+0000 7f7a6e000640 1 --2- 172.21.15.6:0/2105045900 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7a50006a10 0x7f7a68142420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:04.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.766+0000 7f7a6e000640 1 -- 172.21.15.6:0/2105045900 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7a68142960 msgr2=0x7f7a68070f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:04.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.766+0000 7f7a6e000640 1 --2- 172.21.15.6:0/2105045900 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7a68142960 0x7f7a68070f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:04.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.766+0000 7f7a6e000640 1 -- 172.21.15.6:0/2105045900 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a681594d0 con 0x7f7a68071450 2024-03-04T09:33:04.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.766+0000 7f7a6e000640 1 --2- 172.21.15.6:0/2105045900 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7a68071450 0x7f7a680718d0 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7f7a680cda70 tx=0x7f7a6000f520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:33:04.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.766+0000 7f7a65600640 1 -- 172.21.15.6:0/2105045900 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a60016620 con 0x7f7a68071450 2024-03-04T09:33:04.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.766+0000 7f7a65600640 1 -- 172.21.15.6:0/2105045900 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a6001b070 con 0x7f7a68071450 2024-03-04T09:33:04.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.766+0000 7f7a70000640 1 -- 172.21.15.6:0/2105045900 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a681e6160 con 0x7f7a68071450 2024-03-04T09:33:04.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.766+0000 7f7a65600640 1 -- 172.21.15.6:0/2105045900 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a60016a20 con 0x7f7a68071450 2024-03-04T09:33:04.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.767+0000 7f7a70000640 1 -- 172.21.15.6:0/2105045900 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a681e6720 con 0x7f7a68071450 2024-03-04T09:33:04.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.768+0000 7f7a70000640 1 -- 172.21.15.6:0/2105045900 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a681e6920 con 0x7f7a68071450 2024-03-04T09:33:04.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.769+0000 7f7a65600640 1 -- 172.21.15.6:0/2105045900 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7a60016bc0 con 0x7f7a68071450 2024-03-04T09:33:04.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.770+0000 7f7a65600640 1 --2- 172.21.15.6:0/2105045900 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f7a4806b500 0x7f7a4806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:04.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.771+0000 7f7a65600640 1 -- 172.21.15.6:0/2105045900 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f7a6008b250 con 0x7f7a68071450 2024-03-04T09:33:04.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.771+0000 7f7a6d600640 1 --2- 172.21.15.6:0/2105045900 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f7a4806b500 0x7f7a4806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:04.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.772+0000 7f7a6d600640 1 --2- 172.21.15.6:0/2105045900 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f7a4806b500 0x7f7a4806d9c0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f7a5c007fd0 tx=0x7f7a5c009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:33:04.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:04.782+0000 7f7a65600640 1 -- 172.21.15.6:0/2105045900 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7a681e6920 con 0x7f7a68071450 2024-03-04T09:33:05.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:05.025+0000 7f7a70000640 1 -- 172.21.15.6:0/2105045900 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f7a6815a960 con 0x7f7a68071450 2024-03-04T09:33:05.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:05.026+0000 7f7a65600640 1 -- 172.21.15.6:0/2105045900 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v92) v1 ==== 53+0+73 (secure 0 0 0) 0x7f7a6001d310 con 0x7f7a68071450 2024-03-04T09:33:05.023 INFO:teuthology.orchestra.run.smithi006.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-04T09:33:05.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:05.028+0000 7f7a70000640 1 -- 172.21.15.6:0/2105045900 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f7a4806b500 msgr2=0x7f7a4806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:05.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:05.028+0000 7f7a70000640 1 --2- 172.21.15.6:0/2105045900 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f7a4806b500 0x7f7a4806d9c0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f7a5c007fd0 tx=0x7f7a5c009040 comp rx=0 tx=0).stop 2024-03-04T09:33:05.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:05.028+0000 7f7a70000640 1 -- 172.21.15.6:0/2105045900 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7a68071450 msgr2=0x7f7a680718d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:05.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:05.028+0000 7f7a70000640 1 --2- 172.21.15.6:0/2105045900 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7a68071450 0x7f7a680718d0 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7f7a680cda70 tx=0x7f7a6000f520 comp rx=0 tx=0).stop 2024-03-04T09:33:05.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:05.028+0000 7f7a70000640 1 -- 172.21.15.6:0/2105045900 shutdown_connections 2024-03-04T09:33:05.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:05.028+0000 7f7a70000640 1 --2- 172.21.15.6:0/2105045900 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7a68071450 0x7f7a680718d0 unknown :-1 s=CLOSED pgs=1518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:05.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:05.028+0000 7f7a70000640 1 --2- 172.21.15.6:0/2105045900 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7a68142960 0x7f7a68070f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:05.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:05.028+0000 7f7a70000640 1 --2- 172.21.15.6:0/2105045900 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f7a4806b500 0x7f7a4806d9c0 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:05.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:05.028+0000 7f7a70000640 1 --2- 172.21.15.6:0/2105045900 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7a50006a10 0x7f7a68142420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:05.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:05.028+0000 7f7a70000640 1 -- 172.21.15.6:0/2105045900 >> 172.21.15.6:0/2105045900 conn(0x7f7a6806afb0 msgr2=0x7f7a680cb680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:33:05.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:05.028+0000 7f7a70000640 1 -- 172.21.15.6:0/2105045900 shutdown_connections 2024-03-04T09:33:05.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:05.028+0000 7f7a70000640 1 -- 172.21.15.6:0/2105045900 wait complete. 2024-03-04T09:33:05.045 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-04T09:33:05.045 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-04T09:33:05.045 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi006.front.sepia.ceph.com 2024-03-04T09:33:05.046 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-04T09:33:05.046 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-04T09:33:05.046 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-04T09:33:05.046 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-03-04T09:33:05.046 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-04T09:33:05.046 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-04T09:33:05.046 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:33:05.046 DEBUG:teuthology.orchestra.run.smithi006:> ip addr 2024-03-04T09:33:05.099 INFO:teuthology.orchestra.run.smithi006.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-04T09:33:05.099 INFO:teuthology.orchestra.run.smithi006.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-04T09:33:05.099 INFO:teuthology.orchestra.run.smithi006.stdout: inet 127.0.0.1/8 scope host lo 2024-03-04T09:33:05.099 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-04T09:33:05.099 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 ::1/128 scope host 2024-03-04T09:33:05.099 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-04T09:33:05.099 INFO:teuthology.orchestra.run.smithi006.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-04T09:33:05.099 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:6b:fe:e6 brd ff:ff:ff:ff:ff:ff 2024-03-04T09:33:05.099 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp7s0f0 2024-03-04T09:33:05.099 INFO:teuthology.orchestra.run.smithi006.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-04T09:33:05.099 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:6b:fe:e7 brd ff:ff:ff:ff:ff:ff 2024-03-04T09:33:05.099 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp7s0f1 2024-03-04T09:33:05.100 INFO:teuthology.orchestra.run.smithi006.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-04T09:33:05.100 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:bd:15:e4 brd ff:ff:ff:ff:ff:ff 2024-03-04T09:33:05.100 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp1s0f0 2024-03-04T09:33:05.100 INFO:teuthology.orchestra.run.smithi006.stdout: inet 172.21.15.6/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-04T09:33:05.100 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft 42149sec preferred_lft 42149sec 2024-03-04T09:33:05.100 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::ec4:7aff:febd:15e4/64 scope link noprefixroute 2024-03-04T09:33:05.100 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-04T09:33:05.100 INFO:teuthology.orchestra.run.smithi006.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-04T09:33:05.100 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:bd:15:e5 brd ff:ff:ff:ff:ff:ff 2024-03-04T09:33:05.100 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp1s0f1 2024-03-04T09:33:05.100 INFO:teuthology.orchestra.run.smithi006.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-04T09:33:05.100 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 5e:b4:59:dc:42:58 brd ff:ff:ff:ff:ff:ff 2024-03-04T09:33:05.100 INFO:teuthology.orchestra.run.smithi006.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-04T09:33:05.100 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-04T09:33:05.101 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::c6f:adff:fefe:e00d/64 scope link proto kernel_ll 2024-03-04T09:33:05.101 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-04T09:33:05.101 INFO:teuthology.orchestra.run.smithi006.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-04T09:33:05.101 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 7e:67:6d:ea:f9:ff brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-04T09:33:05.101 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::7c67:6dff:feea:f9ff/64 scope link proto kernel_ll 2024-03-04T09:33:05.101 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-04T09:33:05.101 INFO:teuthology.orchestra.run.smithi006.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-04T09:33:05.101 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 5e:b4:59:dc:42:58 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-04T09:33:05.101 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::5cb4:59ff:fedc:4258/64 scope link proto kernel_ll 2024-03-04T09:33:05.101 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-04T09:33:05.102 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:33:05.102 DEBUG:teuthology.orchestra.run.smithi006:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-04T09:33:05.188 INFO:teuthology.orchestra.run.smithi006.stdout:1 2024-03-04T09:33:05.192 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:33:05.192 DEBUG:teuthology.orchestra.run.smithi006:> route 2024-03-04T09:33:05.244 INFO:teuthology.orchestra.run.smithi006.stdout:Kernel IP routing table 2024-03-04T09:33:05.244 INFO:teuthology.orchestra.run.smithi006.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-04T09:33:05.244 INFO:teuthology.orchestra.run.smithi006.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-04T09:33:05.244 INFO:teuthology.orchestra.run.smithi006.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-04T09:33:05.244 INFO:teuthology.orchestra.run.smithi006.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-04T09:33:05.244 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:33:05.245 DEBUG:teuthology.orchestra.run.smithi006:> (cd / && exec bash -c ' 2024-03-04T09:33:05.245 DEBUG:teuthology.orchestra.run.smithi006:> set -e 2024-03-04T09:33:05.245 DEBUG:teuthology.orchestra.run.smithi006:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-04T09:33:05.245 DEBUG:teuthology.orchestra.run.smithi006:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-04T09:33:05.245 DEBUG:teuthology.orchestra.run.smithi006:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-04T09:33:05.245 DEBUG:teuthology.orchestra.run.smithi006:> ') 2024-03-04T09:33:05.440 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:33:05.440 DEBUG:teuthology.orchestra.run.smithi006:> ip netns list 2024-03-04T09:33:05.491 INFO:teuthology.orchestra.run.smithi006.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-04T09:33:05.492 INFO:teuthology.orchestra.run.smithi006.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-04T09:33:05.492 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-04T09:33:05.492 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:33:05.492 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip link set brx.0 up 2024-03-04T09:33:05.582 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-04T09:33:05.583 DEBUG:teuthology.orchestra.run.smithi006:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-04T09:33:05.638 INFO:teuthology.orchestra.run.smithi006.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-04T09:33:05.638 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-04T09:33:05.638 DEBUG:teuthology.orchestra.run.smithi006:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-04T09:33:05.684 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-03-04T09:33:05.684 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:33:05.684 DEBUG:teuthology.orchestra.run.smithi006:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-04T09:33:05.851 INFO:teuthology.orchestra.run.smithi006.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-04T09:33:05.851 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-04T09:33:05.884 INFO:teuthology.orchestra.run.smithi006.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-04T09:33:05.884 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-04T09:33:05.884 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: resolved to: "172.21.15.6:3300,172.21.15.146:3300,172.21.15.146:3301" 2024-03-04T09:33:05.884 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: trying mount with new device syntax: 0@edce0578-5ae4-4402-b45a-552bdef84d7c.cephfs=/ 2024-03-04T09:33:05.884 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.6:3300/172.21.15.146:3300/172.21.15.146:3301" will pass to kernel 2024-03-04T09:33:05.887 INFO:teuthology.orchestra.run.smithi006.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-04T09:33:05.887 INFO:teuthology.orchestra.run.smithi006.stdout: You just mounted a file system that supports labels which does not 2024-03-04T09:33:05.887 INFO:teuthology.orchestra.run.smithi006.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-04T09:33:05.887 INFO:teuthology.orchestra.run.smithi006.stdout: applications will generate AVC messages and not be allowed access to 2024-03-04T09:33:05.887 INFO:teuthology.orchestra.run.smithi006.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-04T09:33:05.890 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-04T09:33:05.890 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:33:05.890 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-04T09:33:05.973 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:33:05.973 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-04T09:33:06.031 INFO:teuthology.orchestra.run.smithi006.stdout:client8382 2024-03-04T09:33:06.031 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-04T09:33:06.031 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T09:33:06.079 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 2 2024-03-04T09:33:06.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:06.346+0000 7f81f3e00640 1 Processor -- start 2024-03-04T09:33:06.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:06.346+0000 7f81f3e00640 1 -- start start 2024-03-04T09:33:06.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:06.347+0000 7f81f3e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f81f415dee0 0x7f81f40d18a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:06.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:06.347+0000 7f81f3e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f81f40d0d40 0x7f81f40b3510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:06.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:06.347+0000 7f81f3e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81f40ceca0 con 0x7f81f40d4860 2024-03-04T09:33:06.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:06.347+0000 7f81f3e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81f40cf5d0 con 0x7f81f415dee0 2024-03-04T09:33:06.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:06.347+0000 7f81f3e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81f40cef80 con 0x7f81f40d0d40 2024-03-04T09:33:06.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:06.348+0000 7f81ebe00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f81f40d4860 0x7f81f40d0630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:58112/0 (socket says 172.21.15.6:58112) 2024-03-04T09:33:06.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:06.348+0000 7f81ebe00640 1 -- 172.21.15.6:0/203220939 learned_addr learned my addr 172.21.15.6:0/203220939 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:33:06.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:06.348+0000 7f81f2a00640 1 --2- 172.21.15.6:0/203220939 >> v2:172.21.15.6:3300/0 conn(0x7f81f415dee0 0x7f81f40d18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:06.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:06.348+0000 7f81f3400640 1 --2- 172.21.15.6:0/203220939 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f81f40d0d40 0x7f81f40b3510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:06.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:06.348+0000 7f81f2000640 1 -- 172.21.15.6:0/203220939 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2986323396 0 0) 0x7f81f40ceca0 con 0x7f81f40d4860 2024-03-04T09:33:06.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:06.348+0000 7f81f2000640 1 -- 172.21.15.6:0/203220939 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81d8003520 con 0x7f81f40d4860 2024-03-04T09:33:06.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:06.349+0000 7f81f2a00640 1 -- 172.21.15.6:0/203220939 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f81f40d0d40 msgr2=0x7f81f40b3510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:33:06.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:06.349+0000 7f81f2a00640 1 --2- 172.21.15.6:0/203220939 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f81f40d0d40 0x7f81f40b3510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:06.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:06.349+0000 7f81f2a00640 1 -- 172.21.15.6:0/203220939 >> v1:172.21.15.6:6789/0 conn(0x7f81f40d4860 legacy=0x7f81f40d0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:06.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:06.349+0000 7f81f2a00640 1 -- 172.21.15.6:0/203220939 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81f40cf260 con 0x7f81f415dee0 2024-03-04T09:33:06.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:06.350+0000 7f81f2a00640 1 --2- 172.21.15.6:0/203220939 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f81f415dee0 0x7f81f40d18a0 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7f81e40097a0 tx=0x7f81e402b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc5d7c4ac9a31d16 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:33:06.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:06.350+0000 7f81f2000640 1 -- 172.21.15.6:0/203220939 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81e4035020 con 0x7f81f415dee0 2024-03-04T09:33:06.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:06.350+0000 7f81f2000640 1 -- 172.21.15.6:0/203220939 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81e4031810 con 0x7f81f415dee0 2024-03-04T09:33:06.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:06.350+0000 7f81f2000640 1 -- 172.21.15.6:0/203220939 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81e4031bc0 con 0x7f81f415dee0 2024-03-04T09:33:06.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:06.351+0000 7f81f3e00640 1 -- 172.21.15.6:0/203220939 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f81f415dee0 msgr2=0x7f81f40d18a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:06.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:06.351+0000 7f81f3e00640 1 --2- 172.21.15.6:0/203220939 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f81f415dee0 0x7f81f40d18a0 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7f81e40097a0 tx=0x7f81e402b8b0 comp rx=0 tx=0).stop 2024-03-04T09:33:06.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:06.351+0000 7f81f3e00640 1 -- 172.21.15.6:0/203220939 shutdown_connections 2024-03-04T09:33:06.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:06.351+0000 7f81f3e00640 1 --2- 172.21.15.6:0/203220939 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f81f40d0d40 0x7f81f40b3510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:06.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:06.351+0000 7f81f3e00640 1 --2- 172.21.15.6:0/203220939 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f81f415dee0 0x7f81f40d18a0 unknown :-1 s=CLOSED pgs=1521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:06.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:06.351+0000 7f81f3e00640 1 -- 172.21.15.6:0/203220939 >> 172.21.15.6:0/203220939 conn(0x7f81f40d3cd0 msgr2=0x7f81f40d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:33:06.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:06.351+0000 7f81f3e00640 1 -- 172.21.15.6:0/203220939 shutdown_connections 2024-03-04T09:33:06.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:06.351+0000 7f81f3e00640 1 -- 172.21.15.6:0/203220939 wait complete. 2024-03-04T09:33:06.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:06.353+0000 7f81f3e00640 1 Processor -- start 2024-03-04T09:33:06.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:06.353+0000 7f81f3e00640 1 -- start start 2024-03-04T09:33:06.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:06.354+0000 7f81f3e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f81f40d0d40 0x7f81f4158340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:06.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:06.354+0000 7f81f3e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f81f40d4860 0x7f81f40cada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:06.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:06.354+0000 7f81f3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f81f415dee0 0x7f81f415aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:06.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:06.354+0000 7f81f3e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81f40d1de0 con 0x7f81f415dee0 2024-03-04T09:33:06.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:06.354+0000 7f81f3e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81f415e2c0 con 0x7f81f40d0d40 2024-03-04T09:33:06.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:06.354+0000 7f81f3e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81f40d1f60 con 0x7f81f40d4860 2024-03-04T09:33:06.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:06.354+0000 7f81f3400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f81f415dee0 0x7f81f415aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:06.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:06.354+0000 7f81f3400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f81f415dee0 0x7f81f415aae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43122/0 (socket says 172.21.15.6:43122) 2024-03-04T09:33:06.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:06.354+0000 7f81f3400640 1 -- 172.21.15.6:0/1098218170 learned_addr learned my addr 172.21.15.6:0/1098218170 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:33:06.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:06.355+0000 7f81f2a00640 1 --2- 172.21.15.6:0/1098218170 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f81f40d0d40 0x7f81f4158340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:06.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:06.355+0000 7f81ebe00640 1 --2- 172.21.15.6:0/1098218170 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f81f40d4860 0x7f81f40cada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:06.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:06.355+0000 7f81f3400640 1 -- 172.21.15.6:0/1098218170 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f81f40d4860 msgr2=0x7f81f40cada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:06.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:06.355+0000 7f81f3400640 1 --2- 172.21.15.6:0/1098218170 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f81f40d4860 0x7f81f40cada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:06.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:06.355+0000 7f81f3400640 1 -- 172.21.15.6:0/1098218170 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f81f40d0d40 msgr2=0x7f81f4158340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:06.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:06.355+0000 7f81f3400640 1 --2- 172.21.15.6:0/1098218170 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f81f40d0d40 0x7f81f4158340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:06.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:06.355+0000 7f81f3400640 1 -- 172.21.15.6:0/1098218170 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81f40cb6a0 con 0x7f81f415dee0 2024-03-04T09:33:06.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:06.355+0000 7f81f3400640 1 --2- 172.21.15.6:0/1098218170 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f81f415dee0 0x7f81f415aae0 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7f81ec00ef90 tx=0x7f81ec00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:33:06.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:06.356+0000 7f81eaa00640 1 -- 172.21.15.6:0/1098218170 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81ec018020 con 0x7f81f415dee0 2024-03-04T09:33:06.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:06.356+0000 7f81eaa00640 1 -- 172.21.15.6:0/1098218170 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81ec014480 con 0x7f81f415dee0 2024-03-04T09:33:06.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:06.356+0000 7f81eaa00640 1 -- 172.21.15.6:0/1098218170 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81ec009e60 con 0x7f81f415dee0 2024-03-04T09:33:06.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:06.356+0000 7f81f3e00640 1 -- 172.21.15.6:0/1098218170 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81f40cb980 con 0x7f81f415dee0 2024-03-04T09:33:06.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:06.356+0000 7f81f3e00640 1 -- 172.21.15.6:0/1098218170 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81f40cc440 con 0x7f81f415dee0 2024-03-04T09:33:06.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:06.357+0000 7f81f3e00640 1 -- 172.21.15.6:0/1098218170 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81f40d32a0 con 0x7f81f415dee0 2024-03-04T09:33:06.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:06.358+0000 7f81eaa00640 1 -- 172.21.15.6:0/1098218170 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f81ec005d10 con 0x7f81f415dee0 2024-03-04T09:33:06.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:06.359+0000 7f81eaa00640 1 --2- 172.21.15.6:0/1098218170 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f81d006b4d0 0x7f81d006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:06.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:06.359+0000 7f81eaa00640 1 -- 172.21.15.6:0/1098218170 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f81ec08afe0 con 0x7f81f415dee0 2024-03-04T09:33:06.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:06.360+0000 7f81f2a00640 1 --2- 172.21.15.6:0/1098218170 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f81d006b4d0 0x7f81d006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:06.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:06.361+0000 7f81f2a00640 1 --2- 172.21.15.6:0/1098218170 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f81d006b4d0 0x7f81d006d990 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f81e4036000 tx=0x7f81e4008030 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:33:06.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:06.363+0000 7f81eaa00640 1 -- 172.21.15.6:0/1098218170 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f81f40d32a0 con 0x7f81f415dee0 2024-03-04T09:33:06.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:06.771+0000 7f81f3e00640 1 -- 172.21.15.6:0/1098218170 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"} v 0) v1 -- 0x7f81f404f330 con 0x7f81f415dee0 2024-03-04T09:33:06.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:06.783+0000 7f81eaa00640 1 -- 172.21.15.6:0/1098218170 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]=0 v93) v1 ==== 105+0+0 (secure 0 0 0) 0x7f81ec022030 con 0x7f81f415dee0 2024-03-04T09:33:06.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:06.786+0000 7f81f3e00640 1 -- 172.21.15.6:0/1098218170 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f81d006b4d0 msgr2=0x7f81d006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:06.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:06.786+0000 7f81f3e00640 1 --2- 172.21.15.6:0/1098218170 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f81d006b4d0 0x7f81d006d990 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f81e4036000 tx=0x7f81e4008030 comp rx=0 tx=0).stop 2024-03-04T09:33:06.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:06.786+0000 7f81f3e00640 1 -- 172.21.15.6:0/1098218170 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f81f415dee0 msgr2=0x7f81f415aae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:06.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:06.786+0000 7f81f3e00640 1 --2- 172.21.15.6:0/1098218170 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f81f415dee0 0x7f81f415aae0 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7f81ec00ef90 tx=0x7f81ec00c490 comp rx=0 tx=0).stop 2024-03-04T09:33:06.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:06.786+0000 7f81f3e00640 1 -- 172.21.15.6:0/1098218170 shutdown_connections 2024-03-04T09:33:06.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:06.786+0000 7f81f3e00640 1 --2- 172.21.15.6:0/1098218170 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f81f415dee0 0x7f81f415aae0 unknown :-1 s=CLOSED pgs=1522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:06.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:06.786+0000 7f81f3e00640 1 --2- 172.21.15.6:0/1098218170 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f81d006b4d0 0x7f81d006d990 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:06.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:06.786+0000 7f81f3e00640 1 --2- 172.21.15.6:0/1098218170 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f81f40d4860 0x7f81f40cada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:06.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:06.786+0000 7f81f3e00640 1 --2- 172.21.15.6:0/1098218170 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f81f40d0d40 0x7f81f4158340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:06.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:06.786+0000 7f81f3e00640 1 -- 172.21.15.6:0/1098218170 >> 172.21.15.6:0/1098218170 conn(0x7f81f40d3cd0 msgr2=0x7f81f40d21f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:33:06.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:06.787+0000 7f81f3e00640 1 -- 172.21.15.6:0/1098218170 shutdown_connections 2024-03-04T09:33:06.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:06.787+0000 7f81f3e00640 1 -- 172.21.15.6:0/1098218170 wait complete. 2024-03-04T09:33:06.811 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:33:07.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:07.008+0000 7f1230400640 1 Processor -- start 2024-03-04T09:33:07.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:07.008+0000 7f1230400640 1 -- start start 2024-03-04T09:33:07.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:07.009+0000 7f1230400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f122815df70 0x7f1228070140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:07.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:07.010+0000 7f1230400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f122815f360 0x7f12280b6650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:07.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:07.010+0000 7f1230400640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f122806f0a0 con 0x7f12280b6230 2024-03-04T09:33:07.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:07.010+0000 7f1230400640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f122806fcb0 con 0x7f122815df70 2024-03-04T09:33:07.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:07.010+0000 7f1230400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f122806f380 con 0x7f122815f360 2024-03-04T09:33:07.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:07.010+0000 7f122da00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f122815df70 0x7f1228070140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:07.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:07.010+0000 7f122d000640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f12280b6230 0x7f1228161cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:58126/0 (socket says 172.21.15.6:58126) 2024-03-04T09:33:07.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:07.010+0000 7f122d000640 1 -- 172.21.15.6:0/113620611 learned_addr learned my addr 172.21.15.6:0/113620611 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:33:07.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:07.010+0000 7f122e400640 1 --2- 172.21.15.6:0/113620611 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f122815f360 0x7f12280b6650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:07.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:07.011+0000 7f122da00640 1 -- 172.21.15.6:0/113620611 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f122815f360 msgr2=0x7f12280b6650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:33:07.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:07.011+0000 7f122da00640 1 --2- 172.21.15.6:0/113620611 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f122815f360 0x7f12280b6650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:07.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:07.011+0000 7f122da00640 1 -- 172.21.15.6:0/113620611 >> v1:172.21.15.6:6789/0 conn(0x7f12280b6230 legacy=0x7f1228161cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:07.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:07.011+0000 7f122da00640 1 -- 172.21.15.6:0/113620611 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f122806f660 con 0x7f122815df70 2024-03-04T09:33:07.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:07.011+0000 7f1227e00640 1 -- 172.21.15.6:0/113620611 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3692198559 0 0) 0x7f122806f0a0 con 0x7f12280b6230 2024-03-04T09:33:07.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:07.012+0000 7f122da00640 1 --2- 172.21.15.6:0/113620611 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f122815df70 0x7f1228070140 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7f1218009d90 tx=0x7f121802bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=846d7f48cfcb37d6 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:33:07.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:07.012+0000 7f1227e00640 1 -- 172.21.15.6:0/113620611 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1218039070 con 0x7f122815df70 2024-03-04T09:33:07.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:07.012+0000 7f1227e00640 1 -- 172.21.15.6:0/113620611 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f121802cc50 con 0x7f122815df70 2024-03-04T09:33:07.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:07.012+0000 7f1227e00640 1 -- 172.21.15.6:0/113620611 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1218034440 con 0x7f122815df70 2024-03-04T09:33:07.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:07.013+0000 7f1230400640 1 -- 172.21.15.6:0/113620611 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f122815df70 msgr2=0x7f1228070140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:07.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:07.013+0000 7f1230400640 1 --2- 172.21.15.6:0/113620611 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f122815df70 0x7f1228070140 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7f1218009d90 tx=0x7f121802bea0 comp rx=0 tx=0).stop 2024-03-04T09:33:07.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:07.013+0000 7f1230400640 1 -- 172.21.15.6:0/113620611 shutdown_connections 2024-03-04T09:33:07.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:07.013+0000 7f1230400640 1 --2- 172.21.15.6:0/113620611 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f122815f360 0x7f12280b6650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:07.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:07.013+0000 7f1230400640 1 --2- 172.21.15.6:0/113620611 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f122815df70 0x7f1228070140 unknown :-1 s=CLOSED pgs=1524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:07.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:07.013+0000 7f1230400640 1 -- 172.21.15.6:0/113620611 >> 172.21.15.6:0/113620611 conn(0x7f122809aa10 msgr2=0x7f122809ae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:33:07.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:07.013+0000 7f1230400640 1 -- 172.21.15.6:0/113620611 shutdown_connections 2024-03-04T09:33:07.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:07.013+0000 7f1230400640 1 -- 172.21.15.6:0/113620611 wait complete. 2024-03-04T09:33:07.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:07.015+0000 7f1230400640 1 Processor -- start 2024-03-04T09:33:07.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:07.015+0000 7f1230400640 1 -- start start 2024-03-04T09:33:07.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:07.016+0000 7f1230400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f12280b6230 0x7f12281527c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:07.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:07.016+0000 7f1230400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f122815df70 0x7f12281540b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:07.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:07.016+0000 7f1230400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f12281559b0 0x7f122806cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:07.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:07.016+0000 7f1230400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1228070800 con 0x7f122815df70 2024-03-04T09:33:07.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:07.016+0000 7f1230400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1228070680 con 0x7f12280b6230 2024-03-04T09:33:07.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:07.016+0000 7f1230400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12281623d0 con 0x7f12281559b0 2024-03-04T09:33:07.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:07.016+0000 7f122d000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f122815df70 0x7f12281540b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:07.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:07.016+0000 7f122d000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f122815df70 0x7f12281540b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43144/0 (socket says 172.21.15.6:43144) 2024-03-04T09:33:07.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:07.016+0000 7f122d000640 1 -- 172.21.15.6:0/3397756443 learned_addr learned my addr 172.21.15.6:0/3397756443 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:33:07.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:07.017+0000 7f122da00640 1 --2- 172.21.15.6:0/3397756443 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f12280b6230 0x7f12281527c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:07.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:07.017+0000 7f122e400640 1 --2- 172.21.15.6:0/3397756443 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f12281559b0 0x7f122806cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:07.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:07.017+0000 7f122d000640 1 -- 172.21.15.6:0/3397756443 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f12281559b0 msgr2=0x7f122806cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:07.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:07.017+0000 7f122d000640 1 --2- 172.21.15.6:0/3397756443 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f12281559b0 0x7f122806cff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:07.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:07.017+0000 7f122d000640 1 -- 172.21.15.6:0/3397756443 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f12280b6230 msgr2=0x7f12281527c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:07.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:07.017+0000 7f122d000640 1 --2- 172.21.15.6:0/3397756443 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f12280b6230 0x7f12281527c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:07.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:07.017+0000 7f122d000640 1 -- 172.21.15.6:0/3397756443 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1228154a20 con 0x7f122815df70 2024-03-04T09:33:07.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:07.017+0000 7f122d000640 1 --2- 172.21.15.6:0/3397756443 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f122815df70 0x7f12281540b0 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7f121400c500 tx=0x7f121400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:33:07.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:07.017+0000 7f1225600640 1 -- 172.21.15.6:0/3397756443 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1214018070 con 0x7f122815df70 2024-03-04T09:33:07.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:07.017+0000 7f1225600640 1 -- 172.21.15.6:0/3397756443 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1214007580 con 0x7f122815df70 2024-03-04T09:33:07.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:07.017+0000 7f1225600640 1 -- 172.21.15.6:0/3397756443 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1214004a20 con 0x7f122815df70 2024-03-04T09:33:07.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:07.017+0000 7f1230400640 1 -- 172.21.15.6:0/3397756443 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1228154d00 con 0x7f122815df70 2024-03-04T09:33:07.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:07.018+0000 7f1230400640 1 -- 172.21.15.6:0/3397756443 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12281552c0 con 0x7f122815df70 2024-03-04T09:33:07.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:07.019+0000 7f1230400640 1 -- 172.21.15.6:0/3397756443 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f122804f3a0 con 0x7f122815df70 2024-03-04T09:33:07.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:07.020+0000 7f1225600640 1 -- 172.21.15.6:0/3397756443 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1214004bc0 con 0x7f122815df70 2024-03-04T09:33:07.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:07.021+0000 7f1225600640 1 --2- 172.21.15.6:0/3397756443 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f120806b4d0 0x7f120806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:07.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:07.021+0000 7f1225600640 1 -- 172.21.15.6:0/3397756443 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f1214089480 con 0x7f122815df70 2024-03-04T09:33:07.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:07.021+0000 7f122da00640 1 --2- 172.21.15.6:0/3397756443 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f120806b4d0 0x7f120806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:07.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:07.022+0000 7f122da00640 1 --2- 172.21.15.6:0/3397756443 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f120806b4d0 0x7f120806d990 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f1218036000 tx=0x7f1218030000 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:33:07.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:07.025+0000 7f1225600640 1 -- 172.21.15.6:0/3397756443 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1214058350 con 0x7f122815df70 2024-03-04T09:33:07.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:07.264+0000 7f1230400640 1 -- 172.21.15.6:0/3397756443 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f122809b670 con 0x7f122815df70 2024-03-04T09:33:07.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:07.265+0000 7f1225600640 1 -- 172.21.15.6:0/3397756443 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+4995 (secure 0 0 0) 0x7f1214057c50 con 0x7f122815df70 2024-03-04T09:33:07.262 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:33:07.262 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":94,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6098,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/2522244217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":2522244217},{"type":"v1","addr":"172.21.15.6:6835","nonce":2522244217}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":87},{"gid":6101,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6836/3149226637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":3149226637},{"type":"v1","addr":"172.21.15.146:6836","nonce":3149226637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":87}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:33:06.787875+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:creating","state_seq":1,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":9}]} 2024-03-04T09:33:07.263 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 94 2024-03-04T09:33:07.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:07.268+0000 7f1230400640 1 -- 172.21.15.6:0/3397756443 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f120806b4d0 msgr2=0x7f120806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:07.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:07.268+0000 7f1230400640 1 --2- 172.21.15.6:0/3397756443 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f120806b4d0 0x7f120806d990 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f1218036000 tx=0x7f1218030000 comp rx=0 tx=0).stop 2024-03-04T09:33:07.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:07.268+0000 7f1230400640 1 -- 172.21.15.6:0/3397756443 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f122815df70 msgr2=0x7f12281540b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:07.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:07.268+0000 7f1230400640 1 --2- 172.21.15.6:0/3397756443 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f122815df70 0x7f12281540b0 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7f121400c500 tx=0x7f121400c9c0 comp rx=0 tx=0).stop 2024-03-04T09:33:07.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:07.268+0000 7f1230400640 1 -- 172.21.15.6:0/3397756443 shutdown_connections 2024-03-04T09:33:07.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:07.268+0000 7f1230400640 1 --2- 172.21.15.6:0/3397756443 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f120806b4d0 0x7f120806d990 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:07.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:07.268+0000 7f1230400640 1 --2- 172.21.15.6:0/3397756443 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f12281559b0 0x7f122806cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:07.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:07.268+0000 7f1230400640 1 --2- 172.21.15.6:0/3397756443 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f122815df70 0x7f12281540b0 unknown :-1 s=CLOSED pgs=1525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:07.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:07.268+0000 7f1230400640 1 --2- 172.21.15.6:0/3397756443 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f12280b6230 0x7f12281527c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:07.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:07.268+0000 7f1230400640 1 -- 172.21.15.6:0/3397756443 >> 172.21.15.6:0/3397756443 conn(0x7f122809aa10 msgr2=0x7f1228160de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:33:07.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:07.269+0000 7f1230400640 1 -- 172.21.15.6:0/3397756443 shutdown_connections 2024-03-04T09:33:07.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:07.269+0000 7f1230400640 1 -- 172.21.15.6:0/3397756443 wait complete. 2024-03-04T09:33:07.284 DEBUG:tasks.ceph_test_case:wait_until_equal: 1 != 2, waiting (timeout=30)... 2024-03-04T09:33:10.711 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:33:10.719 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:33:12.285 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:33:12.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:12.515+0000 7f321d600640 1 Processor -- start 2024-03-04T09:33:12.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:12.515+0000 7f321d600640 1 -- start start 2024-03-04T09:33:12.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:12.517+0000 7f321d600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f32180cfd90 0x7f32180d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:12.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:12.517+0000 7f321d600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f32180d2760 0x7f32180b3740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:12.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:12.517+0000 7f321d600640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32180cea50 con 0x7f3218161b40 2024-03-04T09:33:12.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:12.517+0000 7f321d600640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32180b5770 con 0x7f32180cfd90 2024-03-04T09:33:12.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:12.517+0000 7f321d600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32180cf900 con 0x7f32180d2760 2024-03-04T09:33:12.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:12.517+0000 7f320e000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f32180cfd90 0x7f32180d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:12.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:12.517+0000 7f320e000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f32180cfd90 0x7f32180d2220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36644/0 (socket says 172.21.15.6:36644) 2024-03-04T09:33:12.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:12.517+0000 7f320e000640 1 -- 172.21.15.6:0/1373976265 learned_addr learned my addr 172.21.15.6:0/1373976265 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:33:12.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:12.518+0000 7f3217400640 1 --2- 172.21.15.6:0/1373976265 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f32180d2760 0x7f32180b3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:12.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:12.518+0000 7f320e000640 1 -- 172.21.15.6:0/1373976265 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f32180d2760 msgr2=0x7f32180b3740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:33:12.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:12.518+0000 7f320e000640 1 --2- 172.21.15.6:0/1373976265 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f32180d2760 0x7f32180b3740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:12.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:12.518+0000 7f320e000640 1 -- 172.21.15.6:0/1373976265 >> v1:172.21.15.146:6789/0 conn(0x7f3218161b40 legacy=0x7f32180b5060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:33:12.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:12.518+0000 7f320e000640 1 -- 172.21.15.6:0/1373976265 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32180cecf0 con 0x7f32180cfd90 2024-03-04T09:33:12.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:12.520+0000 7f320e000640 1 --2- 172.21.15.6:0/1373976265 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f32180cfd90 0x7f32180d2220 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 crypto rx=0x7f3200009870 tx=0x7f320002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6edf9168746504 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:33:12.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:12.520+0000 7f3216000640 1 -- 172.21.15.6:0/1373976265 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3200035020 con 0x7f32180cfd90 2024-03-04T09:33:12.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:12.520+0000 7f3216000640 1 -- 172.21.15.6:0/1373976265 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3200031640 con 0x7f32180cfd90 2024-03-04T09:33:12.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:12.520+0000 7f3216000640 1 -- 172.21.15.6:0/1373976265 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f320002d040 con 0x7f32180cfd90 2024-03-04T09:33:12.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:12.521+0000 7f321d600640 1 -- 172.21.15.6:0/1373976265 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f32180cfd90 msgr2=0x7f32180d2220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:12.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:12.521+0000 7f321d600640 1 --2- 172.21.15.6:0/1373976265 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f32180cfd90 0x7f32180d2220 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 crypto rx=0x7f3200009870 tx=0x7f320002b8f0 comp rx=0 tx=0).stop 2024-03-04T09:33:12.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:12.521+0000 7f321d600640 1 -- 172.21.15.6:0/1373976265 shutdown_connections 2024-03-04T09:33:12.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:12.521+0000 7f321d600640 1 --2- 172.21.15.6:0/1373976265 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f32180d2760 0x7f32180b3740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:12.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:12.521+0000 7f321d600640 1 --2- 172.21.15.6:0/1373976265 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f32180cfd90 0x7f32180d2220 unknown :-1 s=CLOSED pgs=1526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:12.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:12.521+0000 7f321d600640 1 -- 172.21.15.6:0/1373976265 >> 172.21.15.6:0/1373976265 conn(0x7f321809d220 msgr2=0x7f32180d3ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:33:12.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:12.521+0000 7f321d600640 1 -- 172.21.15.6:0/1373976265 shutdown_connections 2024-03-04T09:33:12.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:12.521+0000 7f321d600640 1 -- 172.21.15.6:0/1373976265 wait complete. 2024-03-04T09:33:12.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:12.523+0000 7f321d600640 1 Processor -- start 2024-03-04T09:33:12.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:12.523+0000 7f321d600640 1 -- start start 2024-03-04T09:33:12.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:12.524+0000 7f321d600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f32180cfd90 0x7f32180b8eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:12.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:12.524+0000 7f321d600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f32180d2760 0x7f3218155ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:12.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:12.524+0000 7f321d600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3218161b40 0x7f32181f84a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:12.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:12.524+0000 7f321d600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32180cf590 con 0x7f32180d2760 2024-03-04T09:33:12.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:12.524+0000 7f321d600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32180cf2b0 con 0x7f32180cfd90 2024-03-04T09:33:12.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:12.524+0000 7f321d600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32180b2ff0 con 0x7f3218161b40 2024-03-04T09:33:12.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:12.524+0000 7f320e000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f32180d2760 0x7f3218155ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:12.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:12.524+0000 7f320e000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f32180d2760 0x7f3218155ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36660/0 (socket says 172.21.15.6:36660) 2024-03-04T09:33:12.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:12.524+0000 7f320e000640 1 -- 172.21.15.6:0/245706709 learned_addr learned my addr 172.21.15.6:0/245706709 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:33:12.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:12.525+0000 7f3216a00640 1 --2- 172.21.15.6:0/245706709 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f32180cfd90 0x7f32180b8eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:12.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:12.525+0000 7f3217400640 1 --2- 172.21.15.6:0/245706709 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3218161b40 0x7f32181f84a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:12.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:12.525+0000 7f320e000640 1 -- 172.21.15.6:0/245706709 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3218161b40 msgr2=0x7f32181f84a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:12.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:12.525+0000 7f320e000640 1 --2- 172.21.15.6:0/245706709 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3218161b40 0x7f32181f84a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:12.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:12.525+0000 7f320e000640 1 -- 172.21.15.6:0/245706709 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f32180cfd90 msgr2=0x7f32180b8eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:12.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:12.525+0000 7f320e000640 1 --2- 172.21.15.6:0/245706709 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f32180cfd90 0x7f32180b8eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:12.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:12.525+0000 7f320e000640 1 -- 172.21.15.6:0/245706709 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32181f89e0 con 0x7f32180d2760 2024-03-04T09:33:12.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:12.525+0000 7f3217400640 1 --2- 172.21.15.6:0/245706709 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3218161b40 0x7f32181f84a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:33:12.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:12.525+0000 7f320e000640 1 --2- 172.21.15.6:0/245706709 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f32180d2760 0x7f3218155ac0 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 crypto rx=0x7f32000097c0 tx=0x7f32000046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:33:12.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:12.526+0000 7f320f400640 1 -- 172.21.15.6:0/245706709 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3200040070 con 0x7f32180d2760 2024-03-04T09:33:12.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:12.526+0000 7f320f400640 1 -- 172.21.15.6:0/245706709 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32000399f0 con 0x7f32180d2760 2024-03-04T09:33:12.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:12.526+0000 7f320f400640 1 -- 172.21.15.6:0/245706709 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3200035040 con 0x7f32180d2760 2024-03-04T09:33:12.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:12.526+0000 7f321d600640 1 -- 172.21.15.6:0/245706709 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32180c7ba0 con 0x7f32180d2760 2024-03-04T09:33:12.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:12.526+0000 7f321d600640 1 -- 172.21.15.6:0/245706709 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32180c8160 con 0x7f32180d2760 2024-03-04T09:33:12.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:12.527+0000 7f321d600640 1 -- 172.21.15.6:0/245706709 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f321804f3a0 con 0x7f32180d2760 2024-03-04T09:33:12.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:12.528+0000 7f320f400640 1 -- 172.21.15.6:0/245706709 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f320004b060 con 0x7f32180d2760 2024-03-04T09:33:12.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:12.529+0000 7f320f400640 1 --2- 172.21.15.6:0/245706709 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f31f406b4d0 0x7f31f406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:12.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:12.530+0000 7f320f400640 1 -- 172.21.15.6:0/245706709 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f32000abc50 con 0x7f32180d2760 2024-03-04T09:33:12.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:12.530+0000 7f3216a00640 1 --2- 172.21.15.6:0/245706709 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f31f406b4d0 0x7f31f406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:12.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:12.531+0000 7f3216a00640 1 --2- 172.21.15.6:0/245706709 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f31f406b4d0 0x7f31f406d990 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f3208005520 tx=0x7f3208009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:33:12.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:12.533+0000 7f320f400640 1 -- 172.21.15.6:0/245706709 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f32181f89e0 con 0x7f32180d2760 2024-03-04T09:33:12.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:12.894+0000 7f321d600640 1 -- 172.21.15.6:0/245706709 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f32180b9620 con 0x7f32180d2760 2024-03-04T09:33:12.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:12.895+0000 7f320f400640 1 -- 172.21.15.6:0/245706709 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 95 v95) v1 ==== 94+0+4998 (secure 0 0 0) 0x7f320007aa80 con 0x7f32180d2760 2024-03-04T09:33:12.892 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:33:12.892 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":95,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6098,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/2522244217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":2522244217},{"type":"v1","addr":"172.21.15.6:6835","nonce":2522244217}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":87},{"gid":6101,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6836/3149226637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":3149226637},{"type":"v1","addr":"172.21.15.146:6836","nonce":3149226637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":87}],"filesystems":[{"mdsmap":{"epoch":95,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:33:07.791532+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:active","state_seq":5,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[6796,8214]},"id":9}]} 2024-03-04T09:33:12.893 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 95 2024-03-04T09:33:12.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:12.898+0000 7f321d600640 1 -- 172.21.15.6:0/245706709 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f31f406b4d0 msgr2=0x7f31f406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:12.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:12.898+0000 7f321d600640 1 --2- 172.21.15.6:0/245706709 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f31f406b4d0 0x7f31f406d990 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f3208005520 tx=0x7f3208009040 comp rx=0 tx=0).stop 2024-03-04T09:33:12.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:12.898+0000 7f321d600640 1 -- 172.21.15.6:0/245706709 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f32180d2760 msgr2=0x7f3218155ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:12.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:12.898+0000 7f321d600640 1 --2- 172.21.15.6:0/245706709 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f32180d2760 0x7f3218155ac0 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 crypto rx=0x7f32000097c0 tx=0x7f32000046c0 comp rx=0 tx=0).stop 2024-03-04T09:33:12.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:12.898+0000 7f321d600640 1 -- 172.21.15.6:0/245706709 shutdown_connections 2024-03-04T09:33:12.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:12.898+0000 7f321d600640 1 --2- 172.21.15.6:0/245706709 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f31f406b4d0 0x7f31f406d990 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:12.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:12.898+0000 7f321d600640 1 --2- 172.21.15.6:0/245706709 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3218161b40 0x7f32181f84a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:12.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:12.898+0000 7f321d600640 1 --2- 172.21.15.6:0/245706709 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f32180d2760 0x7f3218155ac0 unknown :-1 s=CLOSED pgs=1527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:12.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:12.898+0000 7f321d600640 1 --2- 172.21.15.6:0/245706709 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f32180cfd90 0x7f32180b8eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:12.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:12.898+0000 7f321d600640 1 -- 172.21.15.6:0/245706709 >> 172.21.15.6:0/245706709 conn(0x7f321809d220 msgr2=0x7f32180b5040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:33:12.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:12.898+0000 7f321d600640 1 -- 172.21.15.6:0/245706709 shutdown_connections 2024-03-04T09:33:12.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:12.898+0000 7f321d600640 1 -- 172.21.15.6:0/245706709 wait complete. 2024-03-04T09:33:12.914 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:33:13.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.138+0000 7fc159800640 1 Processor -- start 2024-03-04T09:33:13.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.138+0000 7fc159800640 1 -- start start 2024-03-04T09:33:13.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.140+0000 7fc159800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fc15415df00 0x7fc15406c880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:13.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.140+0000 7fc159800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc1540b0d30 0x7fc1540af270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:13.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.140+0000 7fc159800640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1540b6270 con 0x7fc1540b1280 2024-03-04T09:33:13.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.140+0000 7fc159800640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1540aeb00 con 0x7fc15415df00 2024-03-04T09:33:13.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.140+0000 7fc159800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1540b6510 con 0x7fc1540b0d30 2024-03-04T09:33:13.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.140+0000 7fc153e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fc15415df00 0x7fc15406c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:13.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.140+0000 7fc158e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc1540b0d30 0x7fc1540af270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:13.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.141+0000 7fc158e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc1540b0d30 0x7fc1540af270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.6:49008/0 (socket says 172.21.15.6:49008) 2024-03-04T09:33:13.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.141+0000 7fc153e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fc15415df00 0x7fc15406c880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.6:34134/0 (socket says 172.21.15.6:34134) 2024-03-04T09:33:13.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.141+0000 7fc158e00640 1 -- 172.21.15.6:0/3505080219 learned_addr learned my addr 172.21.15.6:0/3505080219 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:33:13.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.141+0000 7fc158e00640 1 -- 172.21.15.6:0/3505080219 >> v2:172.21.15.146:3300/0 conn(0x7fc15415df00 msgr2=0x7fc15406c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:33:13.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.141+0000 7fc158e00640 1 --2- 172.21.15.6:0/3505080219 >> v2:172.21.15.146:3300/0 conn(0x7fc15415df00 0x7fc15406c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:13.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.141+0000 7fc158e00640 1 -- 172.21.15.6:0/3505080219 >> v1:172.21.15.146:6789/0 conn(0x7fc1540b1280 legacy=0x7fc15406cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:13.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.141+0000 7fc158e00640 1 -- 172.21.15.6:0/3505080219 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1540b67f0 con 0x7fc1540b0d30 2024-03-04T09:33:13.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.143+0000 7fc158e00640 1 --2- 172.21.15.6:0/3505080219 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc1540b0d30 0x7fc1540af270 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fc14c009e50 tx=0x7fc14c02bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ad6222e5563c45e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:33:13.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.143+0000 7fc153400640 1 -- 172.21.15.6:0/3505080219 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc14c02ca70 con 0x7fc1540b0d30 2024-03-04T09:33:13.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.143+0000 7fc153400640 1 -- 172.21.15.6:0/3505080219 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc14c02cc10 con 0x7fc1540b0d30 2024-03-04T09:33:13.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.143+0000 7fc153400640 1 -- 172.21.15.6:0/3505080219 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc14c0343f0 con 0x7fc1540b0d30 2024-03-04T09:33:13.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.144+0000 7fc159800640 1 -- 172.21.15.6:0/3505080219 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc1540b0d30 msgr2=0x7fc1540af270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:13.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.144+0000 7fc159800640 1 --2- 172.21.15.6:0/3505080219 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc1540b0d30 0x7fc1540af270 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fc14c009e50 tx=0x7fc14c02bed0 comp rx=0 tx=0).stop 2024-03-04T09:33:13.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.144+0000 7fc159800640 1 -- 172.21.15.6:0/3505080219 shutdown_connections 2024-03-04T09:33:13.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.144+0000 7fc159800640 1 --2- 172.21.15.6:0/3505080219 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc1540b0d30 0x7fc1540af270 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:13.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.144+0000 7fc159800640 1 --2- 172.21.15.6:0/3505080219 >> v2:172.21.15.146:3300/0 conn(0x7fc15415df00 0x7fc15406c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:13.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.144+0000 7fc159800640 1 -- 172.21.15.6:0/3505080219 >> 172.21.15.6:0/3505080219 conn(0x7fc15409aa10 msgr2=0x7fc15409ae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:33:13.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.144+0000 7fc159800640 1 -- 172.21.15.6:0/3505080219 shutdown_connections 2024-03-04T09:33:13.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.144+0000 7fc159800640 1 -- 172.21.15.6:0/3505080219 wait complete. 2024-03-04T09:33:13.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.145+0000 7fc159800640 1 Processor -- start 2024-03-04T09:33:13.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.146+0000 7fc159800640 1 -- start start 2024-03-04T09:33:13.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.146+0000 7fc159800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc1540b0d30 0x7fc1541e83d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:13.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.146+0000 7fc159800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc1540b1280 0x7fc154154a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:13.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.146+0000 7fc159800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc15415df00 0x7fc154159f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:13.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.147+0000 7fc159800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc15415e460 con 0x7fc1540b1280 2024-03-04T09:33:13.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.147+0000 7fc159800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc15415e2e0 con 0x7fc15415df00 2024-03-04T09:33:13.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.147+0000 7fc159800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1540af7b0 con 0x7fc1540b0d30 2024-03-04T09:33:13.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.147+0000 7fc14be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc1540b1280 0x7fc154154a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:13.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.147+0000 7fc14be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc1540b1280 0x7fc154154a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36664/0 (socket says 172.21.15.6:36664) 2024-03-04T09:33:13.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.147+0000 7fc14be00640 1 -- 172.21.15.6:0/230312200 learned_addr learned my addr 172.21.15.6:0/230312200 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:33:13.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.147+0000 7fc153e00640 1 --2- 172.21.15.6:0/230312200 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc1540b0d30 0x7fc1541e83d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:13.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.147+0000 7fc158e00640 1 --2- 172.21.15.6:0/230312200 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc15415df00 0x7fc154159f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:13.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.147+0000 7fc14be00640 1 -- 172.21.15.6:0/230312200 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc1540b0d30 msgr2=0x7fc1541e83d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:13.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.147+0000 7fc14be00640 1 --2- 172.21.15.6:0/230312200 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc1540b0d30 0x7fc1541e83d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:13.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.147+0000 7fc14be00640 1 -- 172.21.15.6:0/230312200 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc15415df00 msgr2=0x7fc154159f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:13.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.147+0000 7fc14be00640 1 --2- 172.21.15.6:0/230312200 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc15415df00 0x7fc154159f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:13.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.147+0000 7fc14be00640 1 -- 172.21.15.6:0/230312200 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1541e8ab0 con 0x7fc1540b1280 2024-03-04T09:33:13.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.147+0000 7fc153e00640 1 --2- 172.21.15.6:0/230312200 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc1540b0d30 0x7fc1541e83d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T09:33:13.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.148+0000 7fc14be00640 1 --2- 172.21.15.6:0/230312200 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc1540b1280 0x7fc154154a90 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7fc13c00c400 tx=0x7fc13c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:33:13.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.148+0000 7fc150c00640 1 -- 172.21.15.6:0/230312200 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc13c0052d0 con 0x7fc1540b1280 2024-03-04T09:33:13.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.148+0000 7fc150c00640 1 -- 172.21.15.6:0/230312200 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc13c005490 con 0x7fc1540b1280 2024-03-04T09:33:13.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.148+0000 7fc150c00640 1 -- 172.21.15.6:0/230312200 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc13c007260 con 0x7fc1540b1280 2024-03-04T09:33:13.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.148+0000 7fc159800640 1 -- 172.21.15.6:0/230312200 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1541e8d90 con 0x7fc1540b1280 2024-03-04T09:33:13.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.148+0000 7fc159800640 1 -- 172.21.15.6:0/230312200 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc15409d3f0 con 0x7fc1540b1280 2024-03-04T09:33:13.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.150+0000 7fc159800640 1 -- 172.21.15.6:0/230312200 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc15404f330 con 0x7fc1540b1280 2024-03-04T09:33:13.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.151+0000 7fc150c00640 1 -- 172.21.15.6:0/230312200 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc13c007400 con 0x7fc1540b1280 2024-03-04T09:33:13.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.151+0000 7fc150c00640 1 --2- 172.21.15.6:0/230312200 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc13006b4d0 0x7fc13006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:13.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.152+0000 7fc150c00640 1 -- 172.21.15.6:0/230312200 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fc13c089790 con 0x7fc1540b1280 2024-03-04T09:33:13.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.152+0000 7fc153e00640 1 --2- 172.21.15.6:0/230312200 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc13006b4d0 0x7fc13006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:13.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.153+0000 7fc153e00640 1 --2- 172.21.15.6:0/230312200 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc13006b4d0 0x7fc13006d990 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7fc144009af0 tx=0x7fc144008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:33:13.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.161+0000 7fc150c00640 1 -- 172.21.15.6:0/230312200 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc13c0585d0 con 0x7fc1540b1280 2024-03-04T09:33:13.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.406+0000 7fc159800640 1 -- 172.21.15.6:0/230312200 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc154155d10 con 0x7fc1540b1280 2024-03-04T09:33:13.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.407+0000 7fc150c00640 1 -- 172.21.15.6:0/230312200 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 95 v95) v1 ==== 94+0+4998 (secure 0 0 0) 0x7fc13c057ed0 con 0x7fc1540b1280 2024-03-04T09:33:13.404 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:33:13.404 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":95,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6098,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/2522244217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":2522244217},{"type":"v1","addr":"172.21.15.6:6835","nonce":2522244217}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":87},{"gid":6101,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6836/3149226637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":3149226637},{"type":"v1","addr":"172.21.15.146:6836","nonce":3149226637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":87}],"filesystems":[{"mdsmap":{"epoch":95,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:33:07.791532+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:active","state_seq":5,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[6796,8214]},"id":9}]} 2024-03-04T09:33:13.404 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 95 2024-03-04T09:33:13.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.409+0000 7fc159800640 1 -- 172.21.15.6:0/230312200 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc13006b4d0 msgr2=0x7fc13006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:13.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.409+0000 7fc159800640 1 --2- 172.21.15.6:0/230312200 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc13006b4d0 0x7fc13006d990 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7fc144009af0 tx=0x7fc144008040 comp rx=0 tx=0).stop 2024-03-04T09:33:13.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.409+0000 7fc159800640 1 -- 172.21.15.6:0/230312200 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc1540b1280 msgr2=0x7fc154154a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:13.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.409+0000 7fc159800640 1 --2- 172.21.15.6:0/230312200 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc1540b1280 0x7fc154154a90 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7fc13c00c400 tx=0x7fc13c00c8c0 comp rx=0 tx=0).stop 2024-03-04T09:33:13.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.409+0000 7fc159800640 1 -- 172.21.15.6:0/230312200 shutdown_connections 2024-03-04T09:33:13.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.410+0000 7fc159800640 1 --2- 172.21.15.6:0/230312200 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc15415df00 0x7fc154159f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:13.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.410+0000 7fc159800640 1 --2- 172.21.15.6:0/230312200 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc1540b1280 0x7fc154154a90 unknown :-1 s=CLOSED pgs=1528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:13.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.410+0000 7fc159800640 1 --2- 172.21.15.6:0/230312200 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc13006b4d0 0x7fc13006d990 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:13.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.410+0000 7fc159800640 1 --2- 172.21.15.6:0/230312200 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc1540b0d30 0x7fc1541e83d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:13.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.410+0000 7fc159800640 1 -- 172.21.15.6:0/230312200 >> 172.21.15.6:0/230312200 conn(0x7fc15409aa10 msgr2=0x7fc154157840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:33:13.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.410+0000 7fc159800640 1 -- 172.21.15.6:0/230312200 shutdown_connections 2024-03-04T09:33:13.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.410+0000 7fc159800640 1 -- 172.21.15.6:0/230312200 wait complete. 2024-03-04T09:33:13.427 INFO:tasks.cephfs.test_strays:Ranks 0 and 1 are a and d 2024-03-04T09:33:13.428 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-04T09:33:13.428 INFO:teuthology.orchestra.run:waiting for 300 2024-03-04T09:33:13.448 INFO:tasks.ceph.mds.c:Stopped 2024-03-04T09:33:13.448 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail c 2024-03-04T09:33:13.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.714+0000 7fa7d1000640 1 Processor -- start 2024-03-04T09:33:13.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.714+0000 7fa7d1000640 1 -- start start 2024-03-04T09:33:13.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.715+0000 7fa7d1000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa7cc0b3c90 0x7fa7cc0cdc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:13.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.715+0000 7fa7d1000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fa7cc071050 0x7fa7cc0714e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:13.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.715+0000 7fa7d1000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7cc0b4860 con 0x7fa7cc0cf5c0 2024-03-04T09:33:13.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.715+0000 7fa7d1000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7cc0ce160 con 0x7fa7cc071050 2024-03-04T09:33:13.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.715+0000 7fa7d1000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7cc0b4b40 con 0x7fa7cc0b3c90 2024-03-04T09:33:13.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.716+0000 7fa7c9e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa7cc0b3c90 0x7fa7cc0cdc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:13.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.716+0000 7fa7ca800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fa7cc071050 0x7fa7cc0714e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:13.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.716+0000 7fa7c9e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa7cc0b3c90 0x7fa7cc0cdc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.6:49022/0 (socket says 172.21.15.6:49022) 2024-03-04T09:33:13.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.716+0000 7fa7c9e00640 1 -- 172.21.15.6:0/3036577352 learned_addr learned my addr 172.21.15.6:0/3036577352 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:33:13.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.716+0000 7fa7c9400640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fa7cc0cf5c0 0x7fa7cc0b4070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.6:46542/0 (socket says 172.21.15.6:46542) 2024-03-04T09:33:13.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.717+0000 7fa7c9e00640 1 -- 172.21.15.6:0/3036577352 >> v2:172.21.15.146:3300/0 conn(0x7fa7cc071050 msgr2=0x7fa7cc0714e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:33:13.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.717+0000 7fa7c9e00640 1 --2- 172.21.15.6:0/3036577352 >> v2:172.21.15.146:3300/0 conn(0x7fa7cc071050 0x7fa7cc0714e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:13.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.717+0000 7fa7c9e00640 1 -- 172.21.15.6:0/3036577352 >> v1:172.21.15.146:6789/0 conn(0x7fa7cc0cf5c0 legacy=0x7fa7cc0b4070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:13.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.717+0000 7fa7c9e00640 1 -- 172.21.15.6:0/3036577352 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7cc0b4e20 con 0x7fa7cc0b3c90 2024-03-04T09:33:13.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.719+0000 7fa7c9e00640 1 --2- 172.21.15.6:0/3036577352 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa7cc0b3c90 0x7fa7cc0cdc20 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fa7bc009870 tx=0x7fa7bc02ba20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e0515100f3f863df server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:33:13.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.719+0000 7fa7c8a00640 1 -- 172.21.15.6:0/3036577352 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7bc02f840 con 0x7fa7cc0b3c90 2024-03-04T09:33:13.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.719+0000 7fa7c8a00640 1 -- 172.21.15.6:0/3036577352 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7bc02f9e0 con 0x7fa7cc0b3c90 2024-03-04T09:33:13.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.719+0000 7fa7c8a00640 1 -- 172.21.15.6:0/3036577352 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7bc02fdb0 con 0x7fa7cc0b3c90 2024-03-04T09:33:13.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.720+0000 7fa7d1000640 1 -- 172.21.15.6:0/3036577352 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa7cc0b3c90 msgr2=0x7fa7cc0cdc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:13.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.720+0000 7fa7d1000640 1 --2- 172.21.15.6:0/3036577352 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa7cc0b3c90 0x7fa7cc0cdc20 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fa7bc009870 tx=0x7fa7bc02ba20 comp rx=0 tx=0).stop 2024-03-04T09:33:13.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.720+0000 7fa7d1000640 1 -- 172.21.15.6:0/3036577352 shutdown_connections 2024-03-04T09:33:13.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.720+0000 7fa7d1000640 1 --2- 172.21.15.6:0/3036577352 >> v2:172.21.15.146:3300/0 conn(0x7fa7cc071050 0x7fa7cc0714e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:13.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.720+0000 7fa7d1000640 1 --2- 172.21.15.6:0/3036577352 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa7cc0b3c90 0x7fa7cc0cdc20 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:13.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.720+0000 7fa7d1000640 1 -- 172.21.15.6:0/3036577352 >> 172.21.15.6:0/3036577352 conn(0x7fa7cc06afb0 msgr2=0x7fa7cc06b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:33:13.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.720+0000 7fa7d1000640 1 -- 172.21.15.6:0/3036577352 shutdown_connections 2024-03-04T09:33:13.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.720+0000 7fa7d1000640 1 -- 172.21.15.6:0/3036577352 wait complete. 2024-03-04T09:33:13.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.722+0000 7fa7d1000640 1 Processor -- start 2024-03-04T09:33:13.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.722+0000 7fa7d1000640 1 -- start start 2024-03-04T09:33:13.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.723+0000 7fa7d1000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa7cc071050 0x7fa7cc145bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:13.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.723+0000 7fa7d1000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa7cc0b3c90 0x7fa7cc15a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:13.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.723+0000 7fa7d1000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa7cc0cf5c0 0x7fa7cc151b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:13.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.723+0000 7fa7d1000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7cc071a20 con 0x7fa7cc0b3c90 2024-03-04T09:33:13.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.723+0000 7fa7d1000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7cc15b340 con 0x7fa7cc071050 2024-03-04T09:33:13.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.723+0000 7fa7d1000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7cc071ba0 con 0x7fa7cc0cf5c0 2024-03-04T09:33:13.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.723+0000 7fa7c9400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa7cc0b3c90 0x7fa7cc15a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:13.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.723+0000 7fa7c9400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa7cc0b3c90 0x7fa7cc15a1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36680/0 (socket says 172.21.15.6:36680) 2024-03-04T09:33:13.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.723+0000 7fa7c9400640 1 -- 172.21.15.6:0/47011823 learned_addr learned my addr 172.21.15.6:0/47011823 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:33:13.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.723+0000 7fa7c9e00640 1 --2- 172.21.15.6:0/47011823 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa7cc071050 0x7fa7cc145bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:13.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.724+0000 7fa7ca800640 1 --2- 172.21.15.6:0/47011823 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa7cc0cf5c0 0x7fa7cc151b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:13.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.724+0000 7fa7c9400640 1 -- 172.21.15.6:0/47011823 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa7cc0cf5c0 msgr2=0x7fa7cc151b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:13.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.724+0000 7fa7c9400640 1 --2- 172.21.15.6:0/47011823 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa7cc0cf5c0 0x7fa7cc151b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:13.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.724+0000 7fa7c9400640 1 -- 172.21.15.6:0/47011823 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa7cc071050 msgr2=0x7fa7cc145bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:13.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.724+0000 7fa7c9400640 1 --2- 172.21.15.6:0/47011823 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa7cc071050 0x7fa7cc145bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:13.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.724+0000 7fa7c9400640 1 -- 172.21.15.6:0/47011823 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7cc132db0 con 0x7fa7cc0b3c90 2024-03-04T09:33:13.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.724+0000 7fa7c9e00640 1 --2- 172.21.15.6:0/47011823 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa7cc071050 0x7fa7cc145bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T09:33:13.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.724+0000 7fa7c9400640 1 --2- 172.21.15.6:0/47011823 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa7cc0b3c90 0x7fa7cc15a1b0 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7fa7b800c3c0 tx=0x7fa7b800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:33:13.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.725+0000 7fa7c2000640 1 -- 172.21.15.6:0/47011823 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7b80074e0 con 0x7fa7cc0b3c90 2024-03-04T09:33:13.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.725+0000 7fa7c2000640 1 -- 172.21.15.6:0/47011823 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7b8007680 con 0x7fa7cc0b3c90 2024-03-04T09:33:13.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.725+0000 7fa7c2000640 1 -- 172.21.15.6:0/47011823 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7b8004a00 con 0x7fa7cc0b3c90 2024-03-04T09:33:13.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.725+0000 7fa7d1000640 1 -- 172.21.15.6:0/47011823 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7cc133090 con 0x7fa7cc0b3c90 2024-03-04T09:33:13.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.725+0000 7fa7d1000640 1 -- 172.21.15.6:0/47011823 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7cc133650 con 0x7fa7cc0b3c90 2024-03-04T09:33:13.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.726+0000 7fa7d1000640 1 -- 172.21.15.6:0/47011823 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7cc04f3a0 con 0x7fa7cc0b3c90 2024-03-04T09:33:13.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.727+0000 7fa7c2000640 1 -- 172.21.15.6:0/47011823 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa7b801a020 con 0x7fa7cc0b3c90 2024-03-04T09:33:13.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.728+0000 7fa7c2000640 1 --2- 172.21.15.6:0/47011823 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa7ac06b4d0 0x7fa7ac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:13.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.728+0000 7fa7c2000640 1 -- 172.21.15.6:0/47011823 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fa7b8089a60 con 0x7fa7cc0b3c90 2024-03-04T09:33:13.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.729+0000 7fa7c9e00640 1 --2- 172.21.15.6:0/47011823 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa7ac06b4d0 0x7fa7ac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:13.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.730+0000 7fa7c9e00640 1 --2- 172.21.15.6:0/47011823 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa7ac06b4d0 0x7fa7ac06d990 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7fa7bc036000 tx=0x7fa7bc008030 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:33:13.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:13.732+0000 7fa7c2000640 1 -- 172.21.15.6:0/47011823 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa7b8058800 con 0x7fa7cc0b3c90 2024-03-04T09:33:14.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:14.044+0000 7fa7d1000640 1 -- 172.21.15.6:0/47011823 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "c"} v 0) v1 -- 0x7fa7cc1505e0 con 0x7fa7cc0b3c90 2024-03-04T09:33:14.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:14.101+0000 7fa7c2000640 1 -- 172.21.15.6:0/47011823 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "c"}]=0 failed mds gid 6098 v96) v1 ==== 95+0+0 (secure 0 0 0) 0x7fa7b8058100 con 0x7fa7cc0b3c90 2024-03-04T09:33:14.098 INFO:teuthology.orchestra.run.smithi006.stderr:failed mds gid 6098 2024-03-04T09:33:14.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:14.105+0000 7fa7d1000640 1 -- 172.21.15.6:0/47011823 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa7ac06b4d0 msgr2=0x7fa7ac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:14.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:14.105+0000 7fa7d1000640 1 --2- 172.21.15.6:0/47011823 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa7ac06b4d0 0x7fa7ac06d990 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7fa7bc036000 tx=0x7fa7bc008030 comp rx=0 tx=0).stop 2024-03-04T09:33:14.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:14.105+0000 7fa7d1000640 1 -- 172.21.15.6:0/47011823 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa7cc0b3c90 msgr2=0x7fa7cc15a1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:14.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:14.105+0000 7fa7d1000640 1 --2- 172.21.15.6:0/47011823 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa7cc0b3c90 0x7fa7cc15a1b0 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7fa7b800c3c0 tx=0x7fa7b800c880 comp rx=0 tx=0).stop 2024-03-04T09:33:14.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:14.105+0000 7fa7d1000640 1 -- 172.21.15.6:0/47011823 shutdown_connections 2024-03-04T09:33:14.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:14.105+0000 7fa7d1000640 1 --2- 172.21.15.6:0/47011823 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa7ac06b4d0 0x7fa7ac06d990 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:14.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:14.105+0000 7fa7d1000640 1 --2- 172.21.15.6:0/47011823 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa7cc0cf5c0 0x7fa7cc151b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:14.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:14.105+0000 7fa7d1000640 1 --2- 172.21.15.6:0/47011823 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa7cc0b3c90 0x7fa7cc15a1b0 unknown :-1 s=CLOSED pgs=1529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:14.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:14.105+0000 7fa7d1000640 1 --2- 172.21.15.6:0/47011823 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa7cc071050 0x7fa7cc145bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:14.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:14.105+0000 7fa7d1000640 1 -- 172.21.15.6:0/47011823 >> 172.21.15.6:0/47011823 conn(0x7fa7cc06afb0 msgr2=0x7fa7cc0ce2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:33:14.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:14.105+0000 7fa7d1000640 1 -- 172.21.15.6:0/47011823 shutdown_connections 2024-03-04T09:33:14.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:14.105+0000 7fa7d1000640 1 -- 172.21.15.6:0/47011823 wait complete. 2024-03-04T09:33:14.121 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-04T09:33:14.121 INFO:teuthology.orchestra.run:waiting for 300 2024-03-04T09:33:14.144 INFO:tasks.ceph.mds.b:Stopped 2024-03-04T09:33:14.144 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail b 2024-03-04T09:33:14.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:14.379+0000 7fddbe200640 1 Processor -- start 2024-03-04T09:33:14.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:14.379+0000 7fddbe200640 1 -- start start 2024-03-04T09:33:14.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:14.381+0000 7fddbe200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fddb815e000 0x7fddb80ce440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:14.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:14.381+0000 7fddbe200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddb80cf090 con 0x7fddb8161c80 2024-03-04T09:33:14.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:14.381+0000 7fddbe200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddb80d0650 con 0x7fddb80b4450 2024-03-04T09:33:14.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:14.381+0000 7fddbe200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fddb806ef40 con 0x7fddb815e000 2024-03-04T09:33:14.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:14.381+0000 7fddb7400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fddb815e000 0x7fddb80ce440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:14.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:14.382+0000 7fddb7e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fddb8161c80 0x7fddb81620b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:58180/0 (socket says 172.21.15.6:58180) 2024-03-04T09:33:14.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:14.382+0000 7fddb7400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fddb815e000 0x7fddb80ce440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36692/0 (socket says 172.21.15.6:36692) 2024-03-04T09:33:14.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:14.382+0000 7fddb7e00640 1 -- 172.21.15.6:0/1601184696 learned_addr learned my addr 172.21.15.6:0/1601184696 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:33:14.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:14.382+0000 7fddb7400640 1 -- 172.21.15.6:0/1601184696 >> v1:172.21.15.146:6789/0 conn(0x7fddb80b4450 legacy=0x7fddb80ce980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:14.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:14.382+0000 7fddb7400640 1 -- 172.21.15.6:0/1601184696 >> v1:172.21.15.6:6789/0 conn(0x7fddb8161c80 legacy=0x7fddb81620b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:14.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:14.382+0000 7fddb7400640 1 -- 172.21.15.6:0/1601184696 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddb80d0930 con 0x7fddb815e000 2024-03-04T09:33:14.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:14.383+0000 7fddb7400640 1 --2- 172.21.15.6:0/1601184696 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fddb815e000 0x7fddb80ce440 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 crypto rx=0x7fdda80097a0 tx=0x7fdda802b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=753a8e9dd9e9d42a server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:33:14.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:14.383+0000 7fddb6000640 1 -- 172.21.15.6:0/1601184696 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdda802f810 con 0x7fddb815e000 2024-03-04T09:33:14.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:14.384+0000 7fddb6000640 1 -- 172.21.15.6:0/1601184696 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdda802f9b0 con 0x7fddb815e000 2024-03-04T09:33:14.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:14.384+0000 7fddb6000640 1 -- 172.21.15.6:0/1601184696 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdda802fd80 con 0x7fddb815e000 2024-03-04T09:33:14.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:14.384+0000 7fddbe200640 1 -- 172.21.15.6:0/1601184696 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fddb815e000 msgr2=0x7fddb80ce440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:14.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:14.384+0000 7fddbe200640 1 --2- 172.21.15.6:0/1601184696 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fddb815e000 0x7fddb80ce440 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 crypto rx=0x7fdda80097a0 tx=0x7fdda802b9f0 comp rx=0 tx=0).stop 2024-03-04T09:33:14.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:14.384+0000 7fddbe200640 1 -- 172.21.15.6:0/1601184696 shutdown_connections 2024-03-04T09:33:14.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:14.384+0000 7fddbe200640 1 --2- 172.21.15.6:0/1601184696 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fddb815e000 0x7fddb80ce440 unknown :-1 s=CLOSED pgs=1531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:14.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:14.384+0000 7fddbe200640 1 -- 172.21.15.6:0/1601184696 >> 172.21.15.6:0/1601184696 conn(0x7fddb80b38a0 msgr2=0x7fddb80b3cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:33:14.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:14.384+0000 7fddbe200640 1 -- 172.21.15.6:0/1601184696 shutdown_connections 2024-03-04T09:33:14.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:14.384+0000 7fddbe200640 1 -- 172.21.15.6:0/1601184696 wait complete. 2024-03-04T09:33:14.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:14.387+0000 7fddbe200640 1 Processor -- start 2024-03-04T09:33:14.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:14.387+0000 7fddbe200640 1 -- start start 2024-03-04T09:33:14.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:14.388+0000 7fddbe200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fddb80b4450 0x7fddb80afa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:14.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:14.388+0000 7fddbe200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fddb815e000 0x7fddb814a810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:14.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:14.389+0000 7fddbe200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fddb8161c80 0x7fddb814ad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:14.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:14.389+0000 7fddbe200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddb80d0280 con 0x7fddb815e000 2024-03-04T09:33:14.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:14.389+0000 7fddbe200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddb806f190 con 0x7fddb8161c80 2024-03-04T09:33:14.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:14.389+0000 7fddbe200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddb80d0400 con 0x7fddb80b4450 2024-03-04T09:33:14.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:14.389+0000 7fddb6a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fddb815e000 0x7fddb814a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:14.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:14.389+0000 7fddb6a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fddb815e000 0x7fddb814a810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36694/0 (socket says 172.21.15.6:36694) 2024-03-04T09:33:14.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:14.389+0000 7fddb6a00640 1 -- 172.21.15.6:0/723293371 learned_addr learned my addr 172.21.15.6:0/723293371 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:33:14.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:14.389+0000 7fddb7400640 1 --2- 172.21.15.6:0/723293371 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fddb80b4450 0x7fddb80afa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:14.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:14.389+0000 7fddb7e00640 1 --2- 172.21.15.6:0/723293371 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fddb8161c80 0x7fddb814ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:14.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:14.389+0000 7fddb6a00640 1 -- 172.21.15.6:0/723293371 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fddb80b4450 msgr2=0x7fddb80afa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:14.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:14.389+0000 7fddb6a00640 1 --2- 172.21.15.6:0/723293371 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fddb80b4450 0x7fddb80afa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:14.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:14.389+0000 7fddb6a00640 1 -- 172.21.15.6:0/723293371 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fddb8161c80 msgr2=0x7fddb814ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:14.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:14.389+0000 7fddb6a00640 1 --2- 172.21.15.6:0/723293371 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fddb8161c80 0x7fddb814ad50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:14.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:14.389+0000 7fddb6a00640 1 -- 172.21.15.6:0/723293371 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddb814d6e0 con 0x7fddb815e000 2024-03-04T09:33:14.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:14.389+0000 7fddb7400640 1 --2- 172.21.15.6:0/723293371 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fddb80b4450 0x7fddb80afa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:33:14.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:14.390+0000 7fddb6a00640 1 --2- 172.21.15.6:0/723293371 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fddb815e000 0x7fddb814a810 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7fdda000c450 tx=0x7fdda000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:33:14.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:14.390+0000 7fddaf400640 1 -- 172.21.15.6:0/723293371 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdda00052d0 con 0x7fddb815e000 2024-03-04T09:33:14.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:14.390+0000 7fddaf400640 1 -- 172.21.15.6:0/723293371 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdda0005470 con 0x7fddb815e000 2024-03-04T09:33:14.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:14.390+0000 7fddbe200640 1 -- 172.21.15.6:0/723293371 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddb814b290 con 0x7fddb815e000 2024-03-04T09:33:14.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:14.390+0000 7fddaf400640 1 -- 172.21.15.6:0/723293371 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdda0012870 con 0x7fddb815e000 2024-03-04T09:33:14.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:14.390+0000 7fddbe200640 1 -- 172.21.15.6:0/723293371 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddb814dc30 con 0x7fddb815e000 2024-03-04T09:33:14.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:14.391+0000 7fddbe200640 1 -- 172.21.15.6:0/723293371 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddb804f3a0 con 0x7fddb815e000 2024-03-04T09:33:14.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:14.392+0000 7fddaf400640 1 -- 172.21.15.6:0/723293371 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdda0012a10 con 0x7fddb815e000 2024-03-04T09:33:14.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:14.393+0000 7fddaf400640 1 --2- 172.21.15.6:0/723293371 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fdd9406b500 0x7fdd9406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:14.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:14.393+0000 7fddaf400640 1 -- 172.21.15.6:0/723293371 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fdda0089e50 con 0x7fddb815e000 2024-03-04T09:33:14.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:14.394+0000 7fddb7400640 1 --2- 172.21.15.6:0/723293371 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fdd9406b500 0x7fdd9406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:14.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:14.395+0000 7fddb7400640 1 --2- 172.21.15.6:0/723293371 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fdd9406b500 0x7fdd9406d9c0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fdda8036000 tx=0x7fdda8039040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:33:14.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:14.398+0000 7fddaf400640 1 -- 172.21.15.6:0/723293371 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdda0058c80 con 0x7fddb815e000 2024-03-04T09:33:14.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:14.675+0000 7fddbe200640 1 -- 172.21.15.6:0/723293371 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "b"} v 0) v1 -- 0x7fddb8073180 con 0x7fddb815e000 2024-03-04T09:33:15.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:15.117+0000 7fddaf400640 1 -- 172.21.15.6:0/723293371 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "b"}]=0 failed mds gid 6101 v97) v1 ==== 95+0+0 (secure 0 0 0) 0x7fdda0058580 con 0x7fddb815e000 2024-03-04T09:33:15.114 INFO:teuthology.orchestra.run.smithi006.stderr:failed mds gid 6101 2024-03-04T09:33:15.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:15.121+0000 7fddbe200640 1 -- 172.21.15.6:0/723293371 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fdd9406b500 msgr2=0x7fdd9406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:15.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:15.121+0000 7fddbe200640 1 --2- 172.21.15.6:0/723293371 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fdd9406b500 0x7fdd9406d9c0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fdda8036000 tx=0x7fdda8039040 comp rx=0 tx=0).stop 2024-03-04T09:33:15.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:15.121+0000 7fddbe200640 1 -- 172.21.15.6:0/723293371 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fddb815e000 msgr2=0x7fddb814a810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:15.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:15.121+0000 7fddbe200640 1 --2- 172.21.15.6:0/723293371 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fddb815e000 0x7fddb814a810 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7fdda000c450 tx=0x7fdda000c910 comp rx=0 tx=0).stop 2024-03-04T09:33:15.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:15.121+0000 7fddbe200640 1 -- 172.21.15.6:0/723293371 shutdown_connections 2024-03-04T09:33:15.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:15.121+0000 7fddbe200640 1 --2- 172.21.15.6:0/723293371 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fddb8161c80 0x7fddb814ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:15.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:15.121+0000 7fddbe200640 1 --2- 172.21.15.6:0/723293371 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fddb815e000 0x7fddb814a810 unknown :-1 s=CLOSED pgs=1532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:15.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:15.121+0000 7fddbe200640 1 --2- 172.21.15.6:0/723293371 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fdd9406b500 0x7fdd9406d9c0 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:15.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:15.121+0000 7fddbe200640 1 --2- 172.21.15.6:0/723293371 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fddb80b4450 0x7fddb80afa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:15.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:15.121+0000 7fddbe200640 1 -- 172.21.15.6:0/723293371 >> 172.21.15.6:0/723293371 conn(0x7fddb80b38a0 msgr2=0x7fddb80cd7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:33:15.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:15.121+0000 7fddbe200640 1 -- 172.21.15.6:0/723293371 shutdown_connections 2024-03-04T09:33:15.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:15.121+0000 7fddbe200640 1 -- 172.21.15.6:0/723293371 wait complete. 2024-03-04T09:33:15.143 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:33:15.143 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-03-04T09:33:15.143 DEBUG:teuthology.orchestra.run.smithi006:> mkdir pin 2024-03-04T09:33:15.144 DEBUG:teuthology.orchestra.run.smithi006:> touch pin/placeholder 2024-03-04T09:33:15.144 DEBUG:teuthology.orchestra.run.smithi006:> ') 2024-03-04T09:33:15.221 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:33:15.222 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.dir.pin -v 1 pin) 2024-03-04T09:33:15.278 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:33:15.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:15.542+0000 7fb6e5200640 1 Processor -- start 2024-03-04T09:33:15.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:15.542+0000 7fb6e5200640 1 -- start start 2024-03-04T09:33:15.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:15.544+0000 7fb6e5200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fb6e00cd060 0x7fb6e015b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:15.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:15.544+0000 7fb6e5200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6e015ed10 con 0x7fb6e01552b0 2024-03-04T09:33:15.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:15.544+0000 7fb6e5200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6e015eff0 con 0x7fb6e015e810 2024-03-04T09:33:15.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:15.544+0000 7fb6e5200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6e015f2d0 con 0x7fb6e00cd060 2024-03-04T09:33:15.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:15.545+0000 7fb6df400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fb6e01552b0 0x7fb6e00cc950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:58192/0 (socket says 172.21.15.6:58192) 2024-03-04T09:33:15.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:15.545+0000 7fb6df400640 1 -- 172.21.15.6:0/340195297 learned_addr learned my addr 172.21.15.6:0/340195297 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:33:15.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:15.545+0000 7fb6dea00640 1 --2- 172.21.15.6:0/340195297 >> v2:172.21.15.146:3300/0 conn(0x7fb6e00cd060 0x7fb6e015b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:15.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:15.545+0000 7fb6de000640 1 -- 172.21.15.6:0/340195297 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4245414703 0 0) 0x7fb6e015ed10 con 0x7fb6e01552b0 2024-03-04T09:33:15.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:15.545+0000 7fb6de000640 1 -- 172.21.15.6:0/340195297 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6c4003520 con 0x7fb6e01552b0 2024-03-04T09:33:15.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:15.546+0000 7fb6de000640 1 -- 172.21.15.6:0/340195297 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 139105317 0 0) 0x7fb6c4003520 con 0x7fb6e01552b0 2024-03-04T09:33:15.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:15.546+0000 7fb6de000640 1 -- 172.21.15.6:0/340195297 >> v2:172.21.15.146:3300/0 conn(0x7fb6e00cd060 msgr2=0x7fb6e015b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:33:15.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:15.546+0000 7fb6de000640 1 --2- 172.21.15.6:0/340195297 >> v2:172.21.15.146:3300/0 conn(0x7fb6e00cd060 0x7fb6e015b1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:15.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:15.546+0000 7fb6de000640 1 -- 172.21.15.6:0/340195297 >> v1:172.21.15.146:6789/0 conn(0x7fb6e015e810 legacy=0x7fb6e00b3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:15.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:15.546+0000 7fb6de000640 1 -- 172.21.15.6:0/340195297 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6e015f5b0 con 0x7fb6e01552b0 2024-03-04T09:33:15.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:15.546+0000 7fb6de000640 1 -- 172.21.15.6:0/340195297 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7fb6d0001670 con 0x7fb6e01552b0 2024-03-04T09:33:15.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:15.546+0000 7fb6de000640 1 -- 172.21.15.6:0/340195297 >> v1:172.21.15.6:6789/0 conn(0x7fb6e01552b0 legacy=0x7fb6e00cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:15.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:15.546+0000 7fb6de000640 1 --2- 172.21.15.6:0/340195297 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb6c40039b0 0x7fb6c4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:15.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:15.546+0000 7fb6de000640 1 --2- 172.21.15.6:0/340195297 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb6c4004800 0x7fb6c4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:15.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:15.546+0000 7fb6de000640 1 --2- 172.21.15.6:0/340195297 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb6c40051c0 0x7fb6c400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:15.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:15.546+0000 7fb6de000640 1 -- 172.21.15.6:0/340195297 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6e015f2d0 con 0x7fb6c4004800 2024-03-04T09:33:15.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:15.546+0000 7fb6de000640 1 -- 172.21.15.6:0/340195297 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6c400c160 con 0x7fb6c40051c0 2024-03-04T09:33:15.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:15.546+0000 7fb6de000640 1 -- 172.21.15.6:0/340195297 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6c400c360 con 0x7fb6c40039b0 2024-03-04T09:33:15.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:15.546+0000 7fb6dea00640 1 --2- 172.21.15.6:0/340195297 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb6c4004800 0x7fb6c4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:15.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:15.547+0000 7fb6df400640 1 --2- 172.21.15.6:0/340195297 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb6c40039b0 0x7fb6c4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:15.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:15.547+0000 7fb6dfe00640 1 --2- 172.21.15.6:0/340195297 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb6c40051c0 0x7fb6c400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:15.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:15.547+0000 7fb6dea00640 1 -- 172.21.15.6:0/340195297 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb6c40039b0 msgr2=0x7fb6c4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:33:15.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:15.547+0000 7fb6dea00640 1 --2- 172.21.15.6:0/340195297 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb6c40039b0 0x7fb6c4003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:15.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:15.547+0000 7fb6dea00640 1 -- 172.21.15.6:0/340195297 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb6c40051c0 msgr2=0x7fb6c400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:33:15.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:15.547+0000 7fb6dea00640 1 --2- 172.21.15.6:0/340195297 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb6c40051c0 0x7fb6c400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:15.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:15.547+0000 7fb6dea00640 1 -- 172.21.15.6:0/340195297 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6c4003520 con 0x7fb6c4004800 2024-03-04T09:33:15.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:15.547+0000 7fb6df400640 1 --2- 172.21.15.6:0/340195297 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb6c40039b0 0x7fb6c4003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:33:15.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:15.547+0000 7fb6dfe00640 1 --2- 172.21.15.6:0/340195297 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb6c40051c0 0x7fb6c400b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:33:15.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:15.547+0000 7fb6dfe00640 1 -- 172.21.15.6:0/340195297 reap_dead start 2024-03-04T09:33:15.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:15.548+0000 7fb6dea00640 1 --2- 172.21.15.6:0/340195297 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb6c4004800 0x7fb6c4004c80 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7fb6cc009ac0 tx=0x7fb6c4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5301627cb3cd893a server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:33:15.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:15.548+0000 7fb6de000640 1 -- 172.21.15.6:0/340195297 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6cc039070 con 0x7fb6c4004800 2024-03-04T09:33:15.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:15.548+0000 7fb6de000640 1 -- 172.21.15.6:0/340195297 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6cc02c6b0 con 0x7fb6c4004800 2024-03-04T09:33:15.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:15.548+0000 7fb6de000640 1 -- 172.21.15.6:0/340195297 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6cc009690 con 0x7fb6c4004800 2024-03-04T09:33:15.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:15.549+0000 7fb6e5200640 1 -- 172.21.15.6:0/340195297 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb6c4004800 msgr2=0x7fb6c4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:15.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:15.549+0000 7fb6e5200640 1 --2- 172.21.15.6:0/340195297 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb6c4004800 0x7fb6c4004c80 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7fb6cc009ac0 tx=0x7fb6c4005830 comp rx=0 tx=0).stop 2024-03-04T09:33:15.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:15.549+0000 7fb6e5200640 1 -- 172.21.15.6:0/340195297 shutdown_connections 2024-03-04T09:33:15.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:15.549+0000 7fb6e5200640 1 --2- 172.21.15.6:0/340195297 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb6c4004800 0x7fb6c4004c80 unknown :-1 s=CLOSED pgs=1534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:15.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:15.549+0000 7fb6e5200640 1 -- 172.21.15.6:0/340195297 >> 172.21.15.6:0/340195297 conn(0x7fb6e006b3d0 msgr2=0x7fb6e006b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:33:15.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:15.549+0000 7fb6e5200640 1 -- 172.21.15.6:0/340195297 shutdown_connections 2024-03-04T09:33:15.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:15.549+0000 7fb6e5200640 1 -- 172.21.15.6:0/340195297 wait complete. 2024-03-04T09:33:15.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:15.552+0000 7fb6e5200640 1 Processor -- start 2024-03-04T09:33:15.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:15.552+0000 7fb6e5200640 1 -- start start 2024-03-04T09:33:15.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:15.553+0000 7fb6e5200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb6c4004800 0x7fb6e00b07f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:15.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:15.553+0000 7fb6e5200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb6e014db10 0x7fb6e01f81e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:15.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:15.553+0000 7fb6e5200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb6e0159f20 0x7fb6e014b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:15.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:15.553+0000 7fb6e5200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6e006dd30 con 0x7fb6e0159f20 2024-03-04T09:33:15.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:15.553+0000 7fb6e5200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6e00b3760 con 0x7fb6c4004800 2024-03-04T09:33:15.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:15.553+0000 7fb6e5200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6e006deb0 con 0x7fb6e014db10 2024-03-04T09:33:15.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:15.554+0000 7fb6dfe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb6e0159f20 0x7fb6e014b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:15.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:15.554+0000 7fb6dfe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb6e0159f20 0x7fb6e014b240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36710/0 (socket says 172.21.15.6:36710) 2024-03-04T09:33:15.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:15.554+0000 7fb6dfe00640 1 -- 172.21.15.6:0/474364991 learned_addr learned my addr 172.21.15.6:0/474364991 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:33:15.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:15.554+0000 7fb6dea00640 1 --2- 172.21.15.6:0/474364991 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb6e014db10 0x7fb6e01f81e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:15.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:15.554+0000 7fb6df400640 1 --2- 172.21.15.6:0/474364991 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb6c4004800 0x7fb6e00b07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:15.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:15.554+0000 7fb6dfe00640 1 -- 172.21.15.6:0/474364991 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb6e014db10 msgr2=0x7fb6e01f81e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:15.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:15.554+0000 7fb6dfe00640 1 --2- 172.21.15.6:0/474364991 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb6e014db10 0x7fb6e01f81e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:15.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:15.554+0000 7fb6dfe00640 1 -- 172.21.15.6:0/474364991 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb6c4004800 msgr2=0x7fb6e00b07f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:15.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:15.554+0000 7fb6dfe00640 1 --2- 172.21.15.6:0/474364991 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb6c4004800 0x7fb6e00b07f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:15.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:15.554+0000 7fb6dfe00640 1 -- 172.21.15.6:0/474364991 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6e00b0e80 con 0x7fb6e0159f20 2024-03-04T09:33:15.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:15.554+0000 7fb6df400640 1 --2- 172.21.15.6:0/474364991 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb6c4004800 0x7fb6e00b07f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:33:15.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:15.554+0000 7fb6dea00640 1 --2- 172.21.15.6:0/474364991 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb6e014db10 0x7fb6e01f81e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:33:15.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:15.554+0000 7fb6dfe00640 1 --2- 172.21.15.6:0/474364991 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb6e0159f20 0x7fb6e014b240 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 crypto rx=0x7fb6e015ec40 tx=0x7fb6d800dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:33:15.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:15.555+0000 7fb6d7400640 1 -- 172.21.15.6:0/474364991 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6d8018070 con 0x7fb6e0159f20 2024-03-04T09:33:15.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:15.555+0000 7fb6d7400640 1 -- 172.21.15.6:0/474364991 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6d8003b70 con 0x7fb6e0159f20 2024-03-04T09:33:15.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:15.555+0000 7fb6d7400640 1 -- 172.21.15.6:0/474364991 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6d8005070 con 0x7fb6e0159f20 2024-03-04T09:33:15.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:15.555+0000 7fb6e5200640 1 -- 172.21.15.6:0/474364991 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6e00b1160 con 0x7fb6e0159f20 2024-03-04T09:33:15.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:15.555+0000 7fb6e5200640 1 -- 172.21.15.6:0/474364991 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6e01f8720 con 0x7fb6e0159f20 2024-03-04T09:33:15.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:15.556+0000 7fb6e5200640 1 -- 172.21.15.6:0/474364991 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6e004f330 con 0x7fb6e0159f20 2024-03-04T09:33:15.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:15.557+0000 7fb6d7400640 1 -- 172.21.15.6:0/474364991 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb6d801b070 con 0x7fb6e0159f20 2024-03-04T09:33:15.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:15.558+0000 7fb6d7400640 1 --2- 172.21.15.6:0/474364991 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb6c006b500 0x7fb6c006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:15.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:15.558+0000 7fb6d7400640 1 -- 172.21.15.6:0/474364991 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fb6d8089880 con 0x7fb6e0159f20 2024-03-04T09:33:15.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:15.559+0000 7fb6df400640 1 --2- 172.21.15.6:0/474364991 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb6c006b500 0x7fb6c006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:15.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:15.560+0000 7fb6df400640 1 --2- 172.21.15.6:0/474364991 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb6c006b500 0x7fb6c006d9c0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fb6d00017e0 tx=0x7fb6d0001470 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:33:15.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:15.567+0000 7fb6d7400640 1 -- 172.21.15.6:0/474364991 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb6d8058750 con 0x7fb6e0159f20 2024-03-04T09:33:15.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:15.807+0000 7fb6e5200640 1 -- 172.21.15.6:0/474364991 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb6e0070a40 con 0x7fb6e0159f20 2024-03-04T09:33:15.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:15.809+0000 7fb6d7400640 1 -- 172.21.15.6:0/474364991 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) v1 ==== 94+0+3417 (secure 0 0 0) 0x7fb6d80015b0 con 0x7fb6e0159f20 2024-03-04T09:33:15.806 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:33:15.806 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":97,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":95,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:33:07.791532+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:active","state_seq":5,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[6796,8214]},"id":9}]} 2024-03-04T09:33:15.806 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 97 2024-03-04T09:33:15.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:15.811+0000 7fb6e5200640 1 -- 172.21.15.6:0/474364991 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb6c006b500 msgr2=0x7fb6c006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:15.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:15.811+0000 7fb6e5200640 1 --2- 172.21.15.6:0/474364991 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb6c006b500 0x7fb6c006d9c0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fb6d00017e0 tx=0x7fb6d0001470 comp rx=0 tx=0).stop 2024-03-04T09:33:15.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:15.811+0000 7fb6e5200640 1 -- 172.21.15.6:0/474364991 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb6e0159f20 msgr2=0x7fb6e014b240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:15.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:15.811+0000 7fb6e5200640 1 --2- 172.21.15.6:0/474364991 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb6e0159f20 0x7fb6e014b240 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 crypto rx=0x7fb6e015ec40 tx=0x7fb6d800dc00 comp rx=0 tx=0).stop 2024-03-04T09:33:15.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:15.811+0000 7fb6e5200640 1 -- 172.21.15.6:0/474364991 shutdown_connections 2024-03-04T09:33:15.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:15.811+0000 7fb6e5200640 1 --2- 172.21.15.6:0/474364991 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb6e0159f20 0x7fb6e014b240 unknown :-1 s=CLOSED pgs=1535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:15.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:15.811+0000 7fb6e5200640 1 --2- 172.21.15.6:0/474364991 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb6c006b500 0x7fb6c006d9c0 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:15.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:15.811+0000 7fb6e5200640 1 --2- 172.21.15.6:0/474364991 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb6e014db10 0x7fb6e01f81e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:15.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:15.811+0000 7fb6e5200640 1 --2- 172.21.15.6:0/474364991 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb6c4004800 0x7fb6e00b07f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:15.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:15.811+0000 7fb6e5200640 1 -- 172.21.15.6:0/474364991 >> 172.21.15.6:0/474364991 conn(0x7fb6e006b3d0 msgr2=0x7fb6e0070eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:33:15.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:15.811+0000 7fb6e5200640 1 -- 172.21.15.6:0/474364991 shutdown_connections 2024-03-04T09:33:15.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:15.811+0000 7fb6e5200640 1 -- 172.21.15.6:0/474364991 wait complete. 2024-03-04T09:33:15.829 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json get subtrees 2024-03-04T09:33:16.062 INFO:teuthology.orchestra.run.smithi146.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2024-03-04T09:33:16.063 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-04T09:33:16.063 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/pin', 1)] 2024-03-04T09:33:18.064 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:33:18.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:18.290+0000 7f84ada00640 1 Processor -- start 2024-03-04T09:33:18.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:18.291+0000 7f84ada00640 1 -- start start 2024-03-04T09:33:18.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:18.292+0000 7f84ada00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f84a815e070 0x7f84a80d2960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:18.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:18.292+0000 7f84ada00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f84a80d0cd0 0x7f84a80cf870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:18.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:18.292+0000 7f84ada00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84a80d18c0 con 0x7f84a80cf410 2024-03-04T09:33:18.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:18.292+0000 7f84ada00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84a80d1ba0 con 0x7f84a815e070 2024-03-04T09:33:18.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:18.292+0000 7f84ada00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84a80d1e80 con 0x7f84a80d0cd0 2024-03-04T09:33:18.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:18.293+0000 7f84a7e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f84a815e070 0x7f84a80d2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:18.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:18.293+0000 7f849f400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f84a80cf410 0x7f84a80d05c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:58196/0 (socket says 172.21.15.6:58196) 2024-03-04T09:33:18.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:18.293+0000 7f849f400640 1 -- 172.21.15.6:0/1136028361 learned_addr learned my addr 172.21.15.6:0/1136028361 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:33:18.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:18.293+0000 7f84ad000640 1 --2- 172.21.15.6:0/1136028361 >> v2:172.21.15.146:3300/0 conn(0x7f84a80d0cd0 0x7f84a80cf870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:18.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:18.293+0000 7f84a7e00640 1 -- 172.21.15.6:0/1136028361 >> v2:172.21.15.146:3300/0 conn(0x7f84a80d0cd0 msgr2=0x7f84a80cf870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:33:18.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:18.293+0000 7f84a7e00640 1 --2- 172.21.15.6:0/1136028361 >> v2:172.21.15.146:3300/0 conn(0x7f84a80d0cd0 0x7f84a80cf870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:18.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:18.293+0000 7f84a7e00640 1 -- 172.21.15.6:0/1136028361 >> v1:172.21.15.6:6789/0 conn(0x7f84a80cf410 legacy=0x7f84a80d05c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:18.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:18.293+0000 7f84a7e00640 1 -- 172.21.15.6:0/1136028361 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84a80d2160 con 0x7f84a815e070 2024-03-04T09:33:18.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:18.293+0000 7f84a7400640 1 -- 172.21.15.6:0/1136028361 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2688924909 0 0) 0x7f84a80d18c0 con 0x7f84a80cf410 2024-03-04T09:33:18.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:18.294+0000 7f84a7e00640 1 --2- 172.21.15.6:0/1136028361 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f84a815e070 0x7f84a80d2960 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7f84a808c1b0 tx=0x7f849802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74a2989eeb1b3ad8 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:33:18.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:18.295+0000 7f84a7400640 1 -- 172.21.15.6:0/1136028361 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8498035020 con 0x7f84a815e070 2024-03-04T09:33:18.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:18.295+0000 7f84a7400640 1 -- 172.21.15.6:0/1136028361 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8498031830 con 0x7f84a815e070 2024-03-04T09:33:18.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:18.295+0000 7f84a7400640 1 -- 172.21.15.6:0/1136028361 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8498031c00 con 0x7f84a815e070 2024-03-04T09:33:18.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:18.295+0000 7f84ada00640 1 -- 172.21.15.6:0/1136028361 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f84a815e070 msgr2=0x7f84a80d2960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:18.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:18.295+0000 7f84ada00640 1 --2- 172.21.15.6:0/1136028361 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f84a815e070 0x7f84a80d2960 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7f84a808c1b0 tx=0x7f849802b8d0 comp rx=0 tx=0).stop 2024-03-04T09:33:18.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:18.296+0000 7f84ada00640 1 -- 172.21.15.6:0/1136028361 shutdown_connections 2024-03-04T09:33:18.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:18.296+0000 7f84ada00640 1 --2- 172.21.15.6:0/1136028361 >> v2:172.21.15.146:3300/0 conn(0x7f84a80d0cd0 0x7f84a80cf870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:18.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:18.296+0000 7f84ada00640 1 --2- 172.21.15.6:0/1136028361 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f84a815e070 0x7f84a80d2960 unknown :-1 s=CLOSED pgs=1537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:18.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:18.296+0000 7f84ada00640 1 -- 172.21.15.6:0/1136028361 >> 172.21.15.6:0/1136028361 conn(0x7f84a808d2a0 msgr2=0x7f84a80d3cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:33:18.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:18.296+0000 7f84ada00640 1 -- 172.21.15.6:0/1136028361 shutdown_connections 2024-03-04T09:33:18.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:18.296+0000 7f84ada00640 1 -- 172.21.15.6:0/1136028361 wait complete. 2024-03-04T09:33:18.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:18.297+0000 7f84ada00640 1 Processor -- start 2024-03-04T09:33:18.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:18.298+0000 7f84ada00640 1 -- start start 2024-03-04T09:33:18.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:18.298+0000 7f84ada00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f84a80cf410 0x7f84a8157fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:18.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:18.298+0000 7f84ada00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f84a80d0cd0 0x7f84a80b9510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:18.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:18.298+0000 7f84ada00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f84a815e070 0x7f84a80cc1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:18.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:18.298+0000 7f84ada00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84a80d4730 con 0x7f84a815e070 2024-03-04T09:33:18.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:18.298+0000 7f84ada00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84a80d45b0 con 0x7f84a80d0cd0 2024-03-04T09:33:18.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:18.298+0000 7f84ada00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84a80b3630 con 0x7f84a80cf410 2024-03-04T09:33:18.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:18.299+0000 7f84ad000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f84a815e070 0x7f84a80cc1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:18.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:18.299+0000 7f84ad000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f84a815e070 0x7f84a80cc1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36732/0 (socket says 172.21.15.6:36732) 2024-03-04T09:33:18.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:18.299+0000 7f84ad000640 1 -- 172.21.15.6:0/386648630 learned_addr learned my addr 172.21.15.6:0/386648630 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:33:18.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:18.299+0000 7f849f400640 1 --2- 172.21.15.6:0/386648630 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f84a80d0cd0 0x7f84a80b9510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:18.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:18.299+0000 7f84a7e00640 1 --2- 172.21.15.6:0/386648630 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f84a80cf410 0x7f84a8157fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:18.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:18.299+0000 7f84ad000640 1 -- 172.21.15.6:0/386648630 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f84a80cf410 msgr2=0x7f84a8157fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:18.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:18.299+0000 7f84ad000640 1 --2- 172.21.15.6:0/386648630 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f84a80cf410 0x7f84a8157fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:18.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:18.299+0000 7f84ad000640 1 -- 172.21.15.6:0/386648630 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f84a80d0cd0 msgr2=0x7f84a80b9510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:18.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:18.299+0000 7f84ad000640 1 --2- 172.21.15.6:0/386648630 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f84a80d0cd0 0x7f84a80b9510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:18.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:18.299+0000 7f84ad000640 1 -- 172.21.15.6:0/386648630 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84a80cb110 con 0x7f84a815e070 2024-03-04T09:33:18.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:18.299+0000 7f84a7e00640 1 --2- 172.21.15.6:0/386648630 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f84a80cf410 0x7f84a8157fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:33:18.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:18.300+0000 7f84ad000640 1 --2- 172.21.15.6:0/386648630 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f84a815e070 0x7f84a80cc1b0 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 crypto rx=0x7f84a0007de0 tx=0x7f84a000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:33:18.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:18.300+0000 7f84a4c00640 1 -- 172.21.15.6:0/386648630 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84a0014620 con 0x7f84a815e070 2024-03-04T09:33:18.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:18.300+0000 7f84a4c00640 1 -- 172.21.15.6:0/386648630 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84a0019070 con 0x7f84a815e070 2024-03-04T09:33:18.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:18.300+0000 7f84a4c00640 1 -- 172.21.15.6:0/386648630 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84a0014a20 con 0x7f84a815e070 2024-03-04T09:33:18.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:18.300+0000 7f84ada00640 1 -- 172.21.15.6:0/386648630 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84a80cb3f0 con 0x7f84a815e070 2024-03-04T09:33:18.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:18.300+0000 7f84ada00640 1 -- 172.21.15.6:0/386648630 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84a80cb9b0 con 0x7f84a815e070 2024-03-04T09:33:18.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:18.301+0000 7f84ada00640 1 -- 172.21.15.6:0/386648630 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84a804f330 con 0x7f84a815e070 2024-03-04T09:33:18.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:18.302+0000 7f84a4c00640 1 -- 172.21.15.6:0/386648630 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f84a0014bc0 con 0x7f84a815e070 2024-03-04T09:33:18.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:18.304+0000 7f84a4c00640 1 --2- 172.21.15.6:0/386648630 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f848406b4d0 0x7f848406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:18.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:18.304+0000 7f84a4c00640 1 -- 172.21.15.6:0/386648630 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f84a008acb0 con 0x7f84a815e070 2024-03-04T09:33:18.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:18.304+0000 7f84a7e00640 1 --2- 172.21.15.6:0/386648630 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f848406b4d0 0x7f848406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:18.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:18.306+0000 7f84a7e00640 1 --2- 172.21.15.6:0/386648630 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f848406b4d0 0x7f848406d990 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f8498036000 tx=0x7f8498008030 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:33:18.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:18.308+0000 7f84a4c00640 1 -- 172.21.15.6:0/386648630 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f84a00599c0 con 0x7f84a815e070 2024-03-04T09:33:18.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:18.568+0000 7f84ada00640 1 -- 172.21.15.6:0/386648630 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f84a80b4750 con 0x7f84a815e070 2024-03-04T09:33:18.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:18.570+0000 7f84a4c00640 1 -- 172.21.15.6:0/386648630 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) v1 ==== 94+0+3417 (secure 0 0 0) 0x7f84a00592c0 con 0x7f84a815e070 2024-03-04T09:33:18.567 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:33:18.567 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":97,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":95,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:33:07.791532+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:active","state_seq":5,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[6796,8214]},"id":9}]} 2024-03-04T09:33:18.567 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 97 2024-03-04T09:33:18.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:18.572+0000 7f84ada00640 1 -- 172.21.15.6:0/386648630 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f848406b4d0 msgr2=0x7f848406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:18.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:18.572+0000 7f84ada00640 1 --2- 172.21.15.6:0/386648630 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f848406b4d0 0x7f848406d990 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f8498036000 tx=0x7f8498008030 comp rx=0 tx=0).stop 2024-03-04T09:33:18.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:18.572+0000 7f84ada00640 1 -- 172.21.15.6:0/386648630 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f84a815e070 msgr2=0x7f84a80cc1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:18.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:18.572+0000 7f84ada00640 1 --2- 172.21.15.6:0/386648630 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f84a815e070 0x7f84a80cc1b0 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 crypto rx=0x7f84a0007de0 tx=0x7f84a000e4b0 comp rx=0 tx=0).stop 2024-03-04T09:33:18.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:18.572+0000 7f84ada00640 1 -- 172.21.15.6:0/386648630 shutdown_connections 2024-03-04T09:33:18.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:18.572+0000 7f84ada00640 1 --2- 172.21.15.6:0/386648630 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f84a815e070 0x7f84a80cc1b0 unknown :-1 s=CLOSED pgs=1538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:18.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:18.572+0000 7f84ada00640 1 --2- 172.21.15.6:0/386648630 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f84a80d0cd0 0x7f84a80b9510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:18.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:18.572+0000 7f84ada00640 1 --2- 172.21.15.6:0/386648630 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f848406b4d0 0x7f848406d990 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:18.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:18.572+0000 7f84ada00640 1 --2- 172.21.15.6:0/386648630 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f84a80cf410 0x7f84a8157fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:18.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:18.572+0000 7f84ada00640 1 -- 172.21.15.6:0/386648630 >> 172.21.15.6:0/386648630 conn(0x7f84a808d2a0 msgr2=0x7f84a80b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:33:18.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:18.573+0000 7f84ada00640 1 -- 172.21.15.6:0/386648630 shutdown_connections 2024-03-04T09:33:18.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:18.573+0000 7f84ada00640 1 -- 172.21.15.6:0/386648630 wait complete. 2024-03-04T09:33:18.589 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json get subtrees 2024-03-04T09:33:18.826 INFO:teuthology.orchestra.run.smithi146.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2024-03-04T09:33:18.826 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-04T09:33:18.826 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/pin', 1)] 2024-03-04T09:33:20.827 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:33:21.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:21.053+0000 7ff7a8600640 1 Processor -- start 2024-03-04T09:33:21.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:21.054+0000 7ff7a8600640 1 -- start start 2024-03-04T09:33:21.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:21.055+0000 7ff7a8600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff7a00d2b40 0x7ff7a00d0f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:21.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:21.055+0000 7ff7a8600640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7a00b3940 con 0x7ff7a00d1480 2024-03-04T09:33:21.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:21.055+0000 7ff7a8600640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7a00cf580 con 0x7ff7a015e2e0 2024-03-04T09:33:21.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:21.056+0000 7ff7a8600640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7a00d4e70 con 0x7ff7a00d2b40 2024-03-04T09:33:21.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:21.056+0000 7ff7a5200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff7a00d2b40 0x7ff7a00d0f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:21.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:21.056+0000 7ff7a5200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff7a00d2b40 0x7ff7a00d0f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55702/0 (socket says 172.21.15.6:55702) 2024-03-04T09:33:21.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:21.056+0000 7ff7a6600640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7ff7a00d1480 0x7ff7a00b40d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:45178/0 (socket says 172.21.15.6:45178) 2024-03-04T09:33:21.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:21.056+0000 7ff7a5200640 1 -- 172.21.15.6:0/1504764055 learned_addr learned my addr 172.21.15.6:0/1504764055 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:33:21.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:21.056+0000 7ff7a5200640 1 -- 172.21.15.6:0/1504764055 >> v1:172.21.15.146:6789/0 conn(0x7ff7a015e2e0 legacy=0x7ff7a00d0830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:33:21.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:21.056+0000 7ff7a5200640 1 -- 172.21.15.6:0/1504764055 >> v1:172.21.15.6:6789/0 conn(0x7ff7a00d1480 legacy=0x7ff7a00b40d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:21.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:21.056+0000 7ff7a5200640 1 -- 172.21.15.6:0/1504764055 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7a00cf860 con 0x7ff7a00d2b40 2024-03-04T09:33:21.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:21.058+0000 7ff7a5200640 1 --2- 172.21.15.6:0/1504764055 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff7a00d2b40 0x7ff7a00d0f40 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7ff78c009870 tx=0x7ff78c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77e47a8c12c2d3b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:33:21.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:21.058+0000 7ff79fe00640 1 -- 172.21.15.6:0/1504764055 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff78c034070 con 0x7ff7a00d2b40 2024-03-04T09:33:21.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:21.058+0000 7ff79fe00640 1 -- 172.21.15.6:0/1504764055 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff78c030860 con 0x7ff7a00d2b40 2024-03-04T09:33:21.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:21.058+0000 7ff79fe00640 1 -- 172.21.15.6:0/1504764055 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff78c030c30 con 0x7ff7a00d2b40 2024-03-04T09:33:21.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:21.059+0000 7ff7a8600640 1 -- 172.21.15.6:0/1504764055 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff7a00d2b40 msgr2=0x7ff7a00d0f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:21.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:21.059+0000 7ff7a8600640 1 --2- 172.21.15.6:0/1504764055 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff7a00d2b40 0x7ff7a00d0f40 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7ff78c009870 tx=0x7ff78c02b9a0 comp rx=0 tx=0).stop 2024-03-04T09:33:21.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:21.059+0000 7ff7a8600640 1 -- 172.21.15.6:0/1504764055 shutdown_connections 2024-03-04T09:33:21.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:21.059+0000 7ff7a8600640 1 --2- 172.21.15.6:0/1504764055 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff7a00d2b40 0x7ff7a00d0f40 unknown :-1 s=CLOSED pgs=1540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:21.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:21.059+0000 7ff7a8600640 1 -- 172.21.15.6:0/1504764055 >> 172.21.15.6:0/1504764055 conn(0x7ff7a00d3f60 msgr2=0x7ff7a00d4370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:33:21.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:21.059+0000 7ff7a8600640 1 -- 172.21.15.6:0/1504764055 shutdown_connections 2024-03-04T09:33:21.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:21.059+0000 7ff7a8600640 1 -- 172.21.15.6:0/1504764055 wait complete. 2024-03-04T09:33:21.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:21.061+0000 7ff7a8600640 1 Processor -- start 2024-03-04T09:33:21.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:21.061+0000 7ff7a8600640 1 -- start start 2024-03-04T09:33:21.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:21.062+0000 7ff7a8600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff7a00d2b40 0x7ff7a0158a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:21.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:21.062+0000 7ff7a8600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff7a015e2e0 0x7ff7a00cb030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:21.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:21.062+0000 7ff7a8600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff7a00c7a90 0x7ff7a0159870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:21.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:21.062+0000 7ff7a8600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7a00cfe20 con 0x7ff7a00d2b40 2024-03-04T09:33:21.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:21.062+0000 7ff7a8600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7a00b3c40 con 0x7ff7a015e2e0 2024-03-04T09:33:21.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:21.062+0000 7ff7a8600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7a00b47e0 con 0x7ff7a00c7a90 2024-03-04T09:33:21.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:21.062+0000 7ff7a5c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff7a00d2b40 0x7ff7a0158a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:21.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:21.062+0000 7ff7a5c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff7a00d2b40 0x7ff7a0158a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55718/0 (socket says 172.21.15.6:55718) 2024-03-04T09:33:21.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:21.062+0000 7ff7a5c00640 1 -- 172.21.15.6:0/1017797711 learned_addr learned my addr 172.21.15.6:0/1017797711 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:33:21.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:21.063+0000 7ff7a5200640 1 --2- 172.21.15.6:0/1017797711 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff7a015e2e0 0x7ff7a00cb030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:21.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:21.063+0000 7ff7a5c00640 1 -- 172.21.15.6:0/1017797711 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff7a00c7a90 msgr2=0x7ff7a0159870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:21.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:21.063+0000 7ff7a5c00640 1 --2- 172.21.15.6:0/1017797711 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff7a00c7a90 0x7ff7a0159870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:21.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:21.063+0000 7ff7a5c00640 1 -- 172.21.15.6:0/1017797711 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff7a015e2e0 msgr2=0x7ff7a00cb030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:21.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:21.063+0000 7ff7a5c00640 1 --2- 172.21.15.6:0/1017797711 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff7a015e2e0 0x7ff7a00cb030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:21.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:21.063+0000 7ff7a5c00640 1 -- 172.21.15.6:0/1017797711 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7a00b7e20 con 0x7ff7a00d2b40 2024-03-04T09:33:21.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:21.063+0000 7ff7a5c00640 1 --2- 172.21.15.6:0/1017797711 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff7a00d2b40 0x7ff7a0158a70 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 crypto rx=0x7ff79400b420 tx=0x7ff79400b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:33:21.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:21.063+0000 7ff79d600640 1 -- 172.21.15.6:0/1017797711 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff794011830 con 0x7ff7a00d2b40 2024-03-04T09:33:21.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:21.063+0000 7ff79d600640 1 -- 172.21.15.6:0/1017797711 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff794016070 con 0x7ff7a00d2b40 2024-03-04T09:33:21.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:21.063+0000 7ff79d600640 1 -- 172.21.15.6:0/1017797711 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff794011c10 con 0x7ff7a00d2b40 2024-03-04T09:33:21.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:21.063+0000 7ff7a8600640 1 -- 172.21.15.6:0/1017797711 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7a00b8100 con 0x7ff7a00d2b40 2024-03-04T09:33:21.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:21.064+0000 7ff7a8600640 1 -- 172.21.15.6:0/1017797711 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7a00b86c0 con 0x7ff7a00d2b40 2024-03-04T09:33:21.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:21.065+0000 7ff7a8600640 1 -- 172.21.15.6:0/1017797711 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7a004f3a0 con 0x7ff7a00d2b40 2024-03-04T09:33:21.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:21.066+0000 7ff79d600640 1 -- 172.21.15.6:0/1017797711 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff79401b460 con 0x7ff7a00d2b40 2024-03-04T09:33:21.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:21.066+0000 7ff79d600640 1 --2- 172.21.15.6:0/1017797711 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff78006b4d0 0x7ff78006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:21.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:21.067+0000 7ff79d600640 1 -- 172.21.15.6:0/1017797711 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7ff794089dc0 con 0x7ff7a00d2b40 2024-03-04T09:33:21.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:21.067+0000 7ff7a5200640 1 --2- 172.21.15.6:0/1017797711 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff78006b4d0 0x7ff78006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:21.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:21.068+0000 7ff7a5200640 1 --2- 172.21.15.6:0/1017797711 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff78006b4d0 0x7ff78006d990 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7ff78c036000 tx=0x7ff78c03b040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:33:21.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:21.071+0000 7ff79d600640 1 -- 172.21.15.6:0/1017797711 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff794020020 con 0x7ff7a00d2b40 2024-03-04T09:33:21.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:21.317+0000 7ff7a8600640 1 -- 172.21.15.6:0/1017797711 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff7a015d770 con 0x7ff7a00d2b40 2024-03-04T09:33:21.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:21.318+0000 7ff79d600640 1 -- 172.21.15.6:0/1017797711 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) v1 ==== 94+0+3418 (secure 0 0 0) 0x7ff794058ba0 con 0x7ff7a00d2b40 2024-03-04T09:33:21.315 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:33:21.315 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":98,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":98,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:33:21.086062+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:active","state_seq":5,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214,6796]},"id":9}]} 2024-03-04T09:33:21.315 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 98 2024-03-04T09:33:21.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:21.320+0000 7ff7a8600640 1 -- 172.21.15.6:0/1017797711 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff78006b4d0 msgr2=0x7ff78006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:21.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:21.320+0000 7ff7a8600640 1 --2- 172.21.15.6:0/1017797711 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff78006b4d0 0x7ff78006d990 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7ff78c036000 tx=0x7ff78c03b040 comp rx=0 tx=0).stop 2024-03-04T09:33:21.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:21.320+0000 7ff7a8600640 1 -- 172.21.15.6:0/1017797711 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff7a00d2b40 msgr2=0x7ff7a0158a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:21.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:21.320+0000 7ff7a8600640 1 --2- 172.21.15.6:0/1017797711 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff7a00d2b40 0x7ff7a0158a70 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 crypto rx=0x7ff79400b420 tx=0x7ff79400b8e0 comp rx=0 tx=0).stop 2024-03-04T09:33:21.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:21.320+0000 7ff7a8600640 1 -- 172.21.15.6:0/1017797711 shutdown_connections 2024-03-04T09:33:21.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:21.320+0000 7ff7a8600640 1 --2- 172.21.15.6:0/1017797711 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff78006b4d0 0x7ff78006d990 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:21.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:21.320+0000 7ff7a8600640 1 --2- 172.21.15.6:0/1017797711 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff7a00c7a90 0x7ff7a0159870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:21.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:21.320+0000 7ff7a8600640 1 --2- 172.21.15.6:0/1017797711 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff7a015e2e0 0x7ff7a00cb030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:21.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:21.320+0000 7ff7a8600640 1 --2- 172.21.15.6:0/1017797711 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff7a00d2b40 0x7ff7a0158a70 unknown :-1 s=CLOSED pgs=1541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:21.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:21.320+0000 7ff7a8600640 1 -- 172.21.15.6:0/1017797711 >> 172.21.15.6:0/1017797711 conn(0x7ff7a00d3f60 msgr2=0x7ff7a00d1d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:33:21.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:21.321+0000 7ff7a8600640 1 -- 172.21.15.6:0/1017797711 shutdown_connections 2024-03-04T09:33:21.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:21.321+0000 7ff7a8600640 1 -- 172.21.15.6:0/1017797711 wait complete. 2024-03-04T09:33:21.336 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json get subtrees 2024-03-04T09:33:21.576 INFO:teuthology.orchestra.run.smithi146.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}},{"is_auth":false,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"0","version":"8","committing_version":"0","committed_version":"0","is_rep":true,"dir_auth":"0","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1},"nref":2}},{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/pin","dirfrag":"0x10000000000","snapid_first":2,"projected_version":"3","version":"3","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"0":1}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"child":1,"frozen":0,"subtree":1,"importing":0,"replicated":1,"dirty":1},"nref":4}}] 2024-03-04T09:33:21.576 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "0", "dirfrag": "0x1", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 2, "path": "", "pins": { "child": 1, "subtree": 1 }, "projected_version": "0", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "8" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "0": 1 } }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x10000000000", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 4, "path": "/pin", "pins": { "child": 1, "dirty": 1, "frozen": 0, "importing": 0, "ptrwaiter": 0, "replicated": 1, "subtree": 1 }, "projected_version": "3", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "3" }, "distributed_ephemeral_pin": false, "export_pin": 1, "export_pin_target": 1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-04T09:33:21.576 INFO:tasks.cephfs.cephfs_test_case:[('/pin', 1)] =?= [('/pin', 1)] 2024-03-04T09:33:21.577 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-04T09:33:21.577 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T09:33:21.629 DEBUG:teuthology.orchestra.run.smithi006:> stdin-killer -- python3 -c ' 2024-03-04T09:33:21.629 DEBUG:teuthology.orchestra.run.smithi006:> import fcntl 2024-03-04T09:33:21.629 DEBUG:teuthology.orchestra.run.smithi006:> import sys 2024-03-04T09:33:21.629 DEBUG:teuthology.orchestra.run.smithi006:> import time 2024-03-04T09:33:21.629 DEBUG:teuthology.orchestra.run.smithi006:> import os 2024-03-04T09:33:21.629 DEBUG:teuthology.orchestra.run.smithi006:> 2024-03-04T09:33:21.629 DEBUG:teuthology.orchestra.run.smithi006:> fcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK) 2024-03-04T09:33:21.630 DEBUG:teuthology.orchestra.run.smithi006:> 2024-03-04T09:33:21.630 DEBUG:teuthology.orchestra.run.smithi006:> os.mkdir("/home/ubuntu/cephtest/mnt.0/pin/to-be-unlinked") 2024-03-04T09:33:21.630 DEBUG:teuthology.orchestra.run.smithi006:> fd = os.open("/home/ubuntu/cephtest/mnt.0/pin/to-be-unlinked", os.O_RDONLY) 2024-03-04T09:33:21.630 DEBUG:teuthology.orchestra.run.smithi006:> while True: 2024-03-04T09:33:21.630 DEBUG:teuthology.orchestra.run.smithi006:> print("open_dir_background: keeping dir open", file=sys.stderr) 2024-03-04T09:33:21.630 DEBUG:teuthology.orchestra.run.smithi006:> try: 2024-03-04T09:33:21.630 DEBUG:teuthology.orchestra.run.smithi006:> if os.read(0, 4096) == b"": 2024-03-04T09:33:21.630 DEBUG:teuthology.orchestra.run.smithi006:> break 2024-03-04T09:33:21.630 DEBUG:teuthology.orchestra.run.smithi006:> except BlockingIOError: 2024-03-04T09:33:21.630 DEBUG:teuthology.orchestra.run.smithi006:> pass 2024-03-04T09:33:21.630 DEBUG:teuthology.orchestra.run.smithi006:> time.sleep(2) 2024-03-04T09:33:21.630 DEBUG:teuthology.orchestra.run.smithi006:> ' 2024-03-04T09:33:21.635 DEBUG:teuthology.orchestra.run.smithi006:> stat /home/ubuntu/cephtest/mnt.0/pin/to-be-unlinked 2024-03-04T09:33:21.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T09:33:21.680 INFO:teuthology.orchestra.run.smithi006.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0/pin/to-be-unlinked': No such file or directory 2024-03-04T09:33:21.764 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-04T09:33:22.680 DEBUG:teuthology.orchestra.run.smithi006:> stat /home/ubuntu/cephtest/mnt.0/pin/to-be-unlinked 2024-03-04T09:33:22.736 INFO:teuthology.orchestra.run.smithi006.stdout: File: /home/ubuntu/cephtest/mnt.0/pin/to-be-unlinked 2024-03-04T09:33:22.736 INFO:teuthology.orchestra.run.smithi006.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-03-04T09:33:22.736 INFO:teuthology.orchestra.run.smithi006.stdout:Device: 28h/40d Inode: 2199023255552 Links: 2 2024-03-04T09:33:22.736 INFO:teuthology.orchestra.run.smithi006.stdout:Access: (0755/drwxr-xr-x) Uid: ( 1000/ ubuntu) Gid: ( 1301/ ubuntu) 2024-03-04T09:33:22.736 INFO:teuthology.orchestra.run.smithi006.stdout:Context: unconfined_u:object_r:unlabeled_t:s0 2024-03-04T09:33:22.736 INFO:teuthology.orchestra.run.smithi006.stdout:Access: 2024-03-04 09:33:21.762771913 +0000 2024-03-04T09:33:22.736 INFO:teuthology.orchestra.run.smithi006.stdout:Modify: 2024-03-04 09:33:21.762771913 +0000 2024-03-04T09:33:22.736 INFO:teuthology.orchestra.run.smithi006.stdout:Change: 2024-03-04 09:33:21.762771913 +0000 2024-03-04T09:33:22.736 INFO:teuthology.orchestra.run.smithi006.stdout: Birth: 2024-03-04 09:33:21.762771913 +0000 2024-03-04T09:33:22.737 INFO:tasks.cephfs.mount:File pin/to-be-unlinked became visible from 0 after 1s 2024-03-04T09:33:22.737 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:33:22.737 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir pin/to-be-unlinked) 2024-03-04T09:33:22.789 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json perf dump mds_cache num_strays 2024-03-04T09:33:23.023 INFO:teuthology.orchestra.run.smithi146.stdout:{"mds_cache":{"num_strays":1}} 2024-03-04T09:33:23.024 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "num_strays": 1 } } 2024-03-04T09:33:23.024 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json perf dump mds_cache strays_created 2024-03-04T09:33:23.254 INFO:teuthology.orchestra.run.smithi146.stdout:{"mds_cache":{"strays_created":1}} 2024-03-04T09:33:23.254 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_created": 1 } } 2024-03-04T09:33:23.254 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json perf dump mds_cache strays_enqueued 2024-03-04T09:33:23.470 INFO:teuthology.orchestra.run.smithi146.stdout:{"mds_cache":{"strays_enqueued":0}} 2024-03-04T09:33:23.471 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 0 } } 2024-03-04T09:33:23.471 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json flush journal 2024-03-04T09:33:23.723 INFO:teuthology.orchestra.run.smithi146.stdout:{"message":"","return_code":0} 2024-03-04T09:33:23.723 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-04T09:33:23.724 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json cache drop 2024-03-04T09:33:23.766 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-04T09:33:23.953 INFO:teuthology.orchestra.run.smithi146.stdout:{"client_recall":{"return_code":0,"message":"(0) Success","recalled":0},"flush_journal":{"return_code":0,"message":""},"trim_cache":{"trimmed":0},"cache":{"pool":{"items":378,"bytes":65362}},"duration":0.016514918999999999} 2024-03-04T09:33:23.953 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cache": { "pool": { "bytes": 65362, "items": 378 } }, "client_recall": { "message": "(0) Success", "recalled": 0, "return_code": 0 }, "duration": 0.016514919, "flush_journal": { "message": "", "return_code": 0 }, "trim_cache": { "trimmed": 0 } } 2024-03-04T09:33:23.954 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 1 2024-03-04T09:33:24.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.181+0000 7f25c9e00640 1 Processor -- start 2024-03-04T09:33:24.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.181+0000 7f25c9e00640 1 -- start start 2024-03-04T09:33:24.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.182+0000 7f25c9e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f25c4072520 0x7f25c40cdfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:24.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.182+0000 7f25c9e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f25c4071e20 0x7f25c40ce4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:24.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.182+0000 7f25c9e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25c4072df0 con 0x7f25c415bf20 2024-03-04T09:33:24.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.182+0000 7f25c9e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25c40cdab0 con 0x7f25c4071e20 2024-03-04T09:33:24.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.182+0000 7f25c9e00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25c40b4d70 con 0x7f25c4072520 2024-03-04T09:33:24.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.183+0000 7f25c3400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f25c4071e20 0x7f25c40ce4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:24.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.183+0000 7f25c2a00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f25c415bf20 0x7f25c40acd40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:45192/0 (socket says 172.21.15.6:45192) 2024-03-04T09:33:24.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.183+0000 7f25c2a00640 1 -- 172.21.15.6:0/832011781 learned_addr learned my addr 172.21.15.6:0/832011781 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:33:24.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.183+0000 7f25ba000640 1 --2- 172.21.15.6:0/832011781 >> v2:172.21.15.146:3300/0 conn(0x7f25c4072520 0x7f25c40cdfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:24.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.183+0000 7f25c2000640 1 -- 172.21.15.6:0/832011781 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3024059810 0 0) 0x7f25c4072df0 con 0x7f25c415bf20 2024-03-04T09:33:24.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.183+0000 7f25c2000640 1 -- 172.21.15.6:0/832011781 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25a8003520 con 0x7f25c415bf20 2024-03-04T09:33:24.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.183+0000 7f25c3400640 1 -- 172.21.15.6:0/832011781 >> v2:172.21.15.146:3300/0 conn(0x7f25c4072520 msgr2=0x7f25c40cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:33:24.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.183+0000 7f25c3400640 1 --2- 172.21.15.6:0/832011781 >> v2:172.21.15.146:3300/0 conn(0x7f25c4072520 0x7f25c40cdfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:24.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.183+0000 7f25c3400640 1 -- 172.21.15.6:0/832011781 >> v1:172.21.15.6:6789/0 conn(0x7f25c415bf20 legacy=0x7f25c40acd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:24.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.183+0000 7f25c3400640 1 -- 172.21.15.6:0/832011781 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25c40b5050 con 0x7f25c4071e20 2024-03-04T09:33:24.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.184+0000 7f25c3400640 1 --2- 172.21.15.6:0/832011781 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f25c4071e20 0x7f25c40ce4f0 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7f25c4070b90 tx=0x7f25bc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7cf98c65d32869d server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:33:24.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.185+0000 7f25c2000640 1 -- 172.21.15.6:0/832011781 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25bc035020 con 0x7f25c4071e20 2024-03-04T09:33:24.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.185+0000 7f25c2000640 1 -- 172.21.15.6:0/832011781 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25bc031830 con 0x7f25c4071e20 2024-03-04T09:33:24.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.185+0000 7f25c2000640 1 -- 172.21.15.6:0/832011781 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25bc031c00 con 0x7f25c4071e20 2024-03-04T09:33:24.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.185+0000 7f25c9e00640 1 -- 172.21.15.6:0/832011781 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f25c4071e20 msgr2=0x7f25c40ce4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:24.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.185+0000 7f25c9e00640 1 --2- 172.21.15.6:0/832011781 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f25c4071e20 0x7f25c40ce4f0 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7f25c4070b90 tx=0x7f25bc02b8d0 comp rx=0 tx=0).stop 2024-03-04T09:33:24.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.185+0000 7f25c9e00640 1 -- 172.21.15.6:0/832011781 shutdown_connections 2024-03-04T09:33:24.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.185+0000 7f25c9e00640 1 --2- 172.21.15.6:0/832011781 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f25c4071e20 0x7f25c40ce4f0 unknown :-1 s=CLOSED pgs=1543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:24.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.185+0000 7f25c9e00640 1 --2- 172.21.15.6:0/832011781 >> v2:172.21.15.146:3300/0 conn(0x7f25c4072520 0x7f25c40cdfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:24.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.185+0000 7f25c9e00640 1 -- 172.21.15.6:0/832011781 >> 172.21.15.6:0/832011781 conn(0x7f25c406b3d0 msgr2=0x7f25c406b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:33:24.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.185+0000 7f25c9e00640 1 -- 172.21.15.6:0/832011781 shutdown_connections 2024-03-04T09:33:24.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.185+0000 7f25c9e00640 1 -- 172.21.15.6:0/832011781 wait complete. 2024-03-04T09:33:24.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.187+0000 7f25c9e00640 1 Processor -- start 2024-03-04T09:33:24.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.187+0000 7f25c9e00640 1 -- start start 2024-03-04T09:33:24.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.188+0000 7f25c9e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f25c4071e20 0x7f25c406dac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:24.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.188+0000 7f25c9e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f25c4072520 0x7f25c4148ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:24.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.188+0000 7f25c9e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f25c415bf20 0x7f25c41490e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:24.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.188+0000 7f25c9e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25c40ad450 con 0x7f25c415bf20 2024-03-04T09:33:24.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.188+0000 7f25c9e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25c40b4990 con 0x7f25c4071e20 2024-03-04T09:33:24.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.188+0000 7f25c9e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25c406ca20 con 0x7f25c4072520 2024-03-04T09:33:24.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.189+0000 7f25c3400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f25c415bf20 0x7f25c41490e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:24.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.189+0000 7f25c3400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f25c415bf20 0x7f25c41490e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55736/0 (socket says 172.21.15.6:55736) 2024-03-04T09:33:24.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.189+0000 7f25c3400640 1 -- 172.21.15.6:0/2337376361 learned_addr learned my addr 172.21.15.6:0/2337376361 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:33:24.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.189+0000 7f25ba000640 1 --2- 172.21.15.6:0/2337376361 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f25c4072520 0x7f25c4148ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:24.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.189+0000 7f25c2a00640 1 --2- 172.21.15.6:0/2337376361 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f25c4071e20 0x7f25c406dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:24.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.189+0000 7f25c3400640 1 -- 172.21.15.6:0/2337376361 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f25c4072520 msgr2=0x7f25c4148ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:24.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.189+0000 7f25c3400640 1 --2- 172.21.15.6:0/2337376361 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f25c4072520 0x7f25c4148ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:24.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.189+0000 7f25c3400640 1 -- 172.21.15.6:0/2337376361 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f25c4071e20 msgr2=0x7f25c406dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:24.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.189+0000 7f25c3400640 1 --2- 172.21.15.6:0/2337376361 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f25c4071e20 0x7f25c406dac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:24.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.189+0000 7f25c3400640 1 -- 172.21.15.6:0/2337376361 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25c406cd00 con 0x7f25c415bf20 2024-03-04T09:33:24.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.189+0000 7f25c3400640 1 --2- 172.21.15.6:0/2337376361 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f25c415bf20 0x7f25c41490e0 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7f25bc034040 tx=0x7f25bc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:33:24.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.190+0000 7f25bbe00640 1 -- 172.21.15.6:0/2337376361 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25bc040070 con 0x7f25c415bf20 2024-03-04T09:33:24.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.190+0000 7f25bbe00640 1 -- 172.21.15.6:0/2337376361 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25bc03a850 con 0x7f25c415bf20 2024-03-04T09:33:24.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.190+0000 7f25bbe00640 1 -- 172.21.15.6:0/2337376361 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25bc035040 con 0x7f25c415bf20 2024-03-04T09:33:24.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.190+0000 7f25c9e00640 1 -- 172.21.15.6:0/2337376361 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25c406cfe0 con 0x7f25c415bf20 2024-03-04T09:33:24.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.190+0000 7f25c9e00640 1 -- 172.21.15.6:0/2337376361 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25c406d5a0 con 0x7f25c415bf20 2024-03-04T09:33:24.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.191+0000 7f25c9e00640 1 -- 172.21.15.6:0/2337376361 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25c404f330 con 0x7f25c415bf20 2024-03-04T09:33:24.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.193+0000 7f25bbe00640 1 -- 172.21.15.6:0/2337376361 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f25bc038020 con 0x7f25c415bf20 2024-03-04T09:33:24.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.194+0000 7f25bbe00640 1 --2- 172.21.15.6:0/2337376361 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f25a006b4d0 0x7f25a006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:24.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.194+0000 7f25bbe00640 1 -- 172.21.15.6:0/2337376361 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f25bc0abb40 con 0x7f25c415bf20 2024-03-04T09:33:24.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.194+0000 7f25c2a00640 1 --2- 172.21.15.6:0/2337376361 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f25a006b4d0 0x7f25a006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:24.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.196+0000 7f25c2a00640 1 --2- 172.21.15.6:0/2337376361 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f25a006b4d0 0x7f25a006d990 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f25b4005c00 tx=0x7f25b4009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:33:24.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.199+0000 7f25bbe00640 1 -- 172.21.15.6:0/2337376361 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f25bc031800 con 0x7f25c415bf20 2024-03-04T09:33:24.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.441+0000 7f25c9e00640 1 -- 172.21.15.6:0/2337376361 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"} v 0) v1 -- 0x7f25c41591d0 con 0x7f25c415bf20 2024-03-04T09:33:24.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.640+0000 7f25bbe00640 1 -- 172.21.15.6:0/2337376361 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]=0 v99) v1 ==== 105+0+0 (secure 0 0 0) 0x7f25bc07a8f0 con 0x7f25c415bf20 2024-03-04T09:33:24.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.644+0000 7f25c9e00640 1 -- 172.21.15.6:0/2337376361 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f25a006b4d0 msgr2=0x7f25a006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:24.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.644+0000 7f25c9e00640 1 --2- 172.21.15.6:0/2337376361 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f25a006b4d0 0x7f25a006d990 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f25b4005c00 tx=0x7f25b4009040 comp rx=0 tx=0).stop 2024-03-04T09:33:24.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.644+0000 7f25c9e00640 1 -- 172.21.15.6:0/2337376361 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f25c415bf20 msgr2=0x7f25c41490e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:24.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.644+0000 7f25c9e00640 1 --2- 172.21.15.6:0/2337376361 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f25c415bf20 0x7f25c41490e0 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7f25bc034040 tx=0x7f25bc0046c0 comp rx=0 tx=0).stop 2024-03-04T09:33:24.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.644+0000 7f25c9e00640 1 -- 172.21.15.6:0/2337376361 shutdown_connections 2024-03-04T09:33:24.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.644+0000 7f25c9e00640 1 --2- 172.21.15.6:0/2337376361 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f25c415bf20 0x7f25c41490e0 unknown :-1 s=CLOSED pgs=1544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:24.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.644+0000 7f25c9e00640 1 --2- 172.21.15.6:0/2337376361 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f25a006b4d0 0x7f25a006d990 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:24.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.644+0000 7f25c9e00640 1 --2- 172.21.15.6:0/2337376361 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f25c4072520 0x7f25c4148ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:24.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.644+0000 7f25c9e00640 1 --2- 172.21.15.6:0/2337376361 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f25c4071e20 0x7f25c406dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:24.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.644+0000 7f25c9e00640 1 -- 172.21.15.6:0/2337376361 >> 172.21.15.6:0/2337376361 conn(0x7f25c406b3d0 msgr2=0x7f25c40ae960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:33:24.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.645+0000 7f25c9e00640 1 -- 172.21.15.6:0/2337376361 shutdown_connections 2024-03-04T09:33:24.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.645+0000 7f25c9e00640 1 -- 172.21.15.6:0/2337376361 wait complete. 2024-03-04T09:33:24.663 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:33:24.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.893+0000 7f5a1ec00640 1 Processor -- start 2024-03-04T09:33:24.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.893+0000 7f5a1ec00640 1 -- start start 2024-03-04T09:33:24.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.895+0000 7f5a1ec00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f5a180af580 0x7f5a1815c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:24.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.895+0000 7f5a1ec00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f5a180cdc20 0x7f5a180b2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:24.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.895+0000 7f5a1ec00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a1815cc40 con 0x7f5a180af1a0 2024-03-04T09:33:24.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.895+0000 7f5a1ec00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a1815cf20 con 0x7f5a180cdc20 2024-03-04T09:33:24.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.895+0000 7f5a1ec00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a1815d200 con 0x7f5a180af580 2024-03-04T09:33:24.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.895+0000 7f5a17e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f5a180af1a0 0x7f5a180b4c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:45194/0 (socket says 172.21.15.6:45194) 2024-03-04T09:33:24.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.895+0000 7f5a17e00640 1 -- 172.21.15.6:0/4002495712 learned_addr learned my addr 172.21.15.6:0/4002495712 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:33:24.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.895+0000 7f5a1cc00640 1 --2- 172.21.15.6:0/4002495712 >> v2:172.21.15.6:3300/0 conn(0x7f5a180cdc20 0x7f5a180b2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:24.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.896+0000 7f5a17400640 1 --2- 172.21.15.6:0/4002495712 >> v2:172.21.15.146:3300/0 conn(0x7f5a180af580 0x7f5a1815c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:24.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.896+0000 7f5a1cc00640 1 -- 172.21.15.6:0/4002495712 >> v2:172.21.15.146:3300/0 conn(0x7f5a180af580 msgr2=0x7f5a1815c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:33:24.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.896+0000 7f5a1cc00640 1 --2- 172.21.15.6:0/4002495712 >> v2:172.21.15.146:3300/0 conn(0x7f5a180af580 0x7f5a1815c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:24.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.896+0000 7f5a1cc00640 1 -- 172.21.15.6:0/4002495712 >> v1:172.21.15.6:6789/0 conn(0x7f5a180af1a0 legacy=0x7f5a180b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:24.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.896+0000 7f5a16a00640 1 -- 172.21.15.6:0/4002495712 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1714153125 0 0) 0x7f5a1815cc40 con 0x7f5a180af1a0 2024-03-04T09:33:24.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.896+0000 7f5a1cc00640 1 -- 172.21.15.6:0/4002495712 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a1815d4e0 con 0x7f5a180cdc20 2024-03-04T09:33:24.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.896+0000 7f5a17400640 1 --2- 172.21.15.6:0/4002495712 >> v2:172.21.15.146:3300/0 conn(0x7f5a180af580 0x7f5a1815c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:33:24.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.897+0000 7f5a1cc00640 1 --2- 172.21.15.6:0/4002495712 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5a180cdc20 0x7f5a180b2dd0 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7f5a180b54c0 tx=0x7f5a1002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa812f316480b4e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:33:24.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.897+0000 7f5a16a00640 1 -- 172.21.15.6:0/4002495712 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a10039070 con 0x7f5a180cdc20 2024-03-04T09:33:24.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.897+0000 7f5a16a00640 1 -- 172.21.15.6:0/4002495712 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a1002cbf0 con 0x7f5a180cdc20 2024-03-04T09:33:24.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.897+0000 7f5a16a00640 1 -- 172.21.15.6:0/4002495712 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a10034440 con 0x7f5a180cdc20 2024-03-04T09:33:24.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.898+0000 7f5a1ec00640 1 -- 172.21.15.6:0/4002495712 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5a180cdc20 msgr2=0x7f5a180b2dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:24.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.898+0000 7f5a1ec00640 1 --2- 172.21.15.6:0/4002495712 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5a180cdc20 0x7f5a180b2dd0 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7f5a180b54c0 tx=0x7f5a1002be40 comp rx=0 tx=0).stop 2024-03-04T09:33:24.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.898+0000 7f5a1ec00640 1 -- 172.21.15.6:0/4002495712 shutdown_connections 2024-03-04T09:33:24.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.898+0000 7f5a1ec00640 1 --2- 172.21.15.6:0/4002495712 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5a180cdc20 0x7f5a180b2dd0 unknown :-1 s=CLOSED pgs=1546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:24.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.898+0000 7f5a1ec00640 1 --2- 172.21.15.6:0/4002495712 >> v2:172.21.15.146:3300/0 conn(0x7f5a180af580 0x7f5a1815c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:24.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.898+0000 7f5a1ec00640 1 -- 172.21.15.6:0/4002495712 >> 172.21.15.6:0/4002495712 conn(0x7f5a1806afb0 msgr2=0x7f5a1806b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:33:24.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.898+0000 7f5a1ec00640 1 -- 172.21.15.6:0/4002495712 shutdown_connections 2024-03-04T09:33:24.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.898+0000 7f5a1ec00640 1 -- 172.21.15.6:0/4002495712 wait complete. 2024-03-04T09:33:24.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.900+0000 7f5a1ec00640 1 Processor -- start 2024-03-04T09:33:24.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.900+0000 7f5a1ec00640 1 -- start start 2024-03-04T09:33:24.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.901+0000 7f5a1ec00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5a180af1a0 0x7f5a180c9e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:24.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.901+0000 7f5a1ec00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5a180af580 0x7f5a1814bc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:24.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.901+0000 7f5a1ec00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5a180cdc20 0x7f5a1809aac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:24.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.901+0000 7f5a1ec00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a180b36a0 con 0x7f5a180af580 2024-03-04T09:33:24.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.901+0000 7f5a1ec00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a180b3520 con 0x7f5a180cdc20 2024-03-04T09:33:24.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.901+0000 7f5a1ec00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a180b3820 con 0x7f5a180af1a0 2024-03-04T09:33:24.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.901+0000 7f5a17400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5a180af580 0x7f5a1814bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:24.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.901+0000 7f5a17400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5a180af580 0x7f5a1814bc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55754/0 (socket says 172.21.15.6:55754) 2024-03-04T09:33:24.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.901+0000 7f5a17400640 1 -- 172.21.15.6:0/3684908186 learned_addr learned my addr 172.21.15.6:0/3684908186 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:33:24.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.902+0000 7f5a1cc00640 1 --2- 172.21.15.6:0/3684908186 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5a180cdc20 0x7f5a1809aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:24.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.902+0000 7f5a17e00640 1 --2- 172.21.15.6:0/3684908186 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5a180af1a0 0x7f5a180c9e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:24.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.902+0000 7f5a17400640 1 -- 172.21.15.6:0/3684908186 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5a180af1a0 msgr2=0x7f5a180c9e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:24.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.902+0000 7f5a17400640 1 --2- 172.21.15.6:0/3684908186 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5a180af1a0 0x7f5a180c9e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:24.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.902+0000 7f5a17400640 1 -- 172.21.15.6:0/3684908186 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5a180cdc20 msgr2=0x7f5a1809aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:24.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.902+0000 7f5a17400640 1 --2- 172.21.15.6:0/3684908186 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5a180cdc20 0x7f5a1809aac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:24.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.902+0000 7f5a17400640 1 -- 172.21.15.6:0/3684908186 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a1809c7a0 con 0x7f5a180af580 2024-03-04T09:33:24.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.902+0000 7f5a17e00640 1 --2- 172.21.15.6:0/3684908186 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5a180af1a0 0x7f5a180c9e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:33:24.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.902+0000 7f5a17400640 1 --2- 172.21.15.6:0/3684908186 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5a180af580 0x7f5a1814bc90 secure :-1 s=READY pgs=1547 cs=0 l=1 rev1=1 crypto rx=0x7f5a0400c7f0 tx=0x7f5a0400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:33:24.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.902+0000 7f5a0fe00640 1 -- 172.21.15.6:0/3684908186 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a04015070 con 0x7f5a180af580 2024-03-04T09:33:24.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.902+0000 7f5a0fe00640 1 -- 172.21.15.6:0/3684908186 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a040077c0 con 0x7f5a180af580 2024-03-04T09:33:24.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.902+0000 7f5a0fe00640 1 -- 172.21.15.6:0/3684908186 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a04007b70 con 0x7f5a180af580 2024-03-04T09:33:24.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.903+0000 7f5a1ec00640 1 -- 172.21.15.6:0/3684908186 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a1809ca40 con 0x7f5a180af580 2024-03-04T09:33:24.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.903+0000 7f5a1ec00640 1 -- 172.21.15.6:0/3684908186 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a1809d000 con 0x7f5a180af580 2024-03-04T09:33:24.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.904+0000 7f5a1ec00640 1 -- 172.21.15.6:0/3684908186 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a18069510 con 0x7f5a180af580 2024-03-04T09:33:24.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.905+0000 7f5a0fe00640 1 -- 172.21.15.6:0/3684908186 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5a04018070 con 0x7f5a180af580 2024-03-04T09:33:24.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.906+0000 7f5a0fe00640 1 --2- 172.21.15.6:0/3684908186 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f59f806b4d0 0x7f59f806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:24.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.906+0000 7f5a0fe00640 1 -- 172.21.15.6:0/3684908186 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f5a0408a420 con 0x7f5a180af580 2024-03-04T09:33:24.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.907+0000 7f5a17e00640 1 --2- 172.21.15.6:0/3684908186 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f59f806b4d0 0x7f59f806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:24.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.908+0000 7f5a17e00640 1 --2- 172.21.15.6:0/3684908186 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f59f806b4d0 0x7f59f806d990 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f5a08000f60 tx=0x7f5a08009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:33:24.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:24.911+0000 7f5a0fe00640 1 -- 172.21.15.6:0/3684908186 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5a040591c0 con 0x7f5a180af580 2024-03-04T09:33:25.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:25.148+0000 7f5a1ec00640 1 -- 172.21.15.6:0/3684908186 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5a1804f3a0 con 0x7f5a180af580 2024-03-04T09:33:25.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:25.150+0000 7f5a0fe00640 1 -- 172.21.15.6:0/3684908186 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3415 (secure 0 0 0) 0x7f5a04058ac0 con 0x7f5a180af580 2024-03-04T09:33:25.147 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:33:25.147 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":99,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:33:24.637813+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":9}]} 2024-03-04T09:33:25.147 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 99 2024-03-04T09:33:25.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:25.152+0000 7f5a1ec00640 1 -- 172.21.15.6:0/3684908186 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f59f806b4d0 msgr2=0x7f59f806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:25.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:25.152+0000 7f5a1ec00640 1 --2- 172.21.15.6:0/3684908186 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f59f806b4d0 0x7f59f806d990 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f5a08000f60 tx=0x7f5a08009040 comp rx=0 tx=0).stop 2024-03-04T09:33:25.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:25.152+0000 7f5a1ec00640 1 -- 172.21.15.6:0/3684908186 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5a180af580 msgr2=0x7f5a1814bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:25.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:25.152+0000 7f5a1ec00640 1 --2- 172.21.15.6:0/3684908186 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5a180af580 0x7f5a1814bc90 secure :-1 s=READY pgs=1547 cs=0 l=1 rev1=1 crypto rx=0x7f5a0400c7f0 tx=0x7f5a0400ccb0 comp rx=0 tx=0).stop 2024-03-04T09:33:25.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:25.152+0000 7f5a1ec00640 1 -- 172.21.15.6:0/3684908186 shutdown_connections 2024-03-04T09:33:25.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:25.152+0000 7f5a1ec00640 1 --2- 172.21.15.6:0/3684908186 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5a180cdc20 0x7f5a1809aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:25.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:25.152+0000 7f5a1ec00640 1 --2- 172.21.15.6:0/3684908186 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5a180af580 0x7f5a1814bc90 unknown :-1 s=CLOSED pgs=1547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:25.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:25.152+0000 7f5a1ec00640 1 --2- 172.21.15.6:0/3684908186 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f59f806b4d0 0x7f59f806d990 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:25.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:25.152+0000 7f5a1ec00640 1 --2- 172.21.15.6:0/3684908186 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5a180af1a0 0x7f5a180c9e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:25.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:25.152+0000 7f5a1ec00640 1 -- 172.21.15.6:0/3684908186 >> 172.21.15.6:0/3684908186 conn(0x7f5a1806afb0 msgr2=0x7f5a1815c0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:33:25.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:25.152+0000 7f5a1ec00640 1 -- 172.21.15.6:0/3684908186 shutdown_connections 2024-03-04T09:33:25.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:25.153+0000 7f5a1ec00640 1 -- 172.21.15.6:0/3684908186 wait complete. 2024-03-04T09:33:25.371 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:32:57.159681+0000', 'modified': '2024-03-04T09:33:24.637813+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8214, 'mds_1': 6796}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6796': {'gid': 6796, 'name': 'd', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.146:6837/2458070840', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 2458070840}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 2458070840}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8214': {'gid': 8214, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2446627517', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2446627517}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2446627517}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8214, 'qdb_cluster': [8214]} 2024-03-04T09:33:25.371 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6796:up:stopping 2024-03-04T09:33:25.769 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-04T09:33:26.372 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:33:26.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:26.604+0000 7fb805a00640 1 Processor -- start 2024-03-04T09:33:26.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:26.604+0000 7fb805a00640 1 -- start start 2024-03-04T09:33:26.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:26.606+0000 7fb805a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb8000cd060 0x7fb8001667e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:26.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:26.606+0000 7fb805a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8000714d0 con 0x7fb800070f10 2024-03-04T09:33:26.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:26.606+0000 7fb805a00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8000717b0 con 0x7fb800145300 2024-03-04T09:33:26.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:26.606+0000 7fb805a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb800071a90 con 0x7fb8000cd060 2024-03-04T09:33:26.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:26.606+0000 7fb7f7400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fb800070f10 0x7fb8000cc950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:45198/0 (socket says 172.21.15.6:45198) 2024-03-04T09:33:26.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:26.606+0000 7fb7f7400640 1 -- 172.21.15.6:0/8846173 learned_addr learned my addr 172.21.15.6:0/8846173 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:33:26.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:26.607+0000 7fb805000640 1 --2- 172.21.15.6:0/8846173 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb8000cd060 0x7fb8001667e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:26.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:26.607+0000 7fb7ff400640 1 -- 172.21.15.6:0/8846173 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3668400946 0 0) 0x7fb8000714d0 con 0x7fb800070f10 2024-03-04T09:33:26.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:26.607+0000 7fb7ff400640 1 -- 172.21.15.6:0/8846173 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7e4003520 con 0x7fb800070f10 2024-03-04T09:33:26.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:26.607+0000 7fb7ff400640 1 -- 172.21.15.6:0/8846173 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1285109021 0 0) 0x7fb7e4003520 con 0x7fb800070f10 2024-03-04T09:33:26.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:26.607+0000 7fb7ff400640 1 -- 172.21.15.6:0/8846173 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb8000cd060 msgr2=0x7fb8001667e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:33:26.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:26.607+0000 7fb7ff400640 1 --2- 172.21.15.6:0/8846173 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb8000cd060 0x7fb8001667e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:26.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:26.607+0000 7fb7ff400640 1 -- 172.21.15.6:0/8846173 >> v1:172.21.15.146:6789/0 conn(0x7fb800145300 legacy=0x7fb8000b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:26.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:26.607+0000 7fb7ff400640 1 -- 172.21.15.6:0/8846173 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb800071d70 con 0x7fb800070f10 2024-03-04T09:33:26.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:26.607+0000 7fb7ff400640 1 -- 172.21.15.6:0/8846173 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7fb7e8001670 con 0x7fb800070f10 2024-03-04T09:33:26.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:26.607+0000 7fb7ff400640 1 -- 172.21.15.6:0/8846173 >> v1:172.21.15.6:6789/0 conn(0x7fb800070f10 legacy=0x7fb8000cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:26.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:26.608+0000 7fb7ff400640 1 --2- 172.21.15.6:0/8846173 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb7e40039e0 0x7fb7e4003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:26.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:26.608+0000 7fb7ff400640 1 --2- 172.21.15.6:0/8846173 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb7e4004800 0x7fb7e4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:26.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:26.608+0000 7fb7ff400640 1 --2- 172.21.15.6:0/8846173 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb8000cd060 0x7fb7e4005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:26.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:26.608+0000 7fb7ff400640 1 -- 172.21.15.6:0/8846173 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb800071a90 con 0x7fb7e4004800 2024-03-04T09:33:26.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:26.608+0000 7fb7ff400640 1 -- 172.21.15.6:0/8846173 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e400bb20 con 0x7fb7e40039e0 2024-03-04T09:33:26.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:26.608+0000 7fb7ff400640 1 -- 172.21.15.6:0/8846173 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e400bd20 con 0x7fb8000cd060 2024-03-04T09:33:26.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:26.608+0000 7fb7f7400640 1 --2- 172.21.15.6:0/8846173 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb7e4004800 0x7fb7e4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:26.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:26.608+0000 7fb7ffe00640 1 --2- 172.21.15.6:0/8846173 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb7e40039e0 0x7fb7e4003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:26.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:26.608+0000 7fb7f7400640 1 -- 172.21.15.6:0/8846173 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb8000cd060 msgr2=0x7fb7e4005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:33:26.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:26.608+0000 7fb7f7400640 1 --2- 172.21.15.6:0/8846173 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb8000cd060 0x7fb7e4005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:26.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:26.608+0000 7fb7f7400640 1 -- 172.21.15.6:0/8846173 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb7e40039e0 msgr2=0x7fb7e4003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:33:26.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:26.608+0000 7fb7f7400640 1 --2- 172.21.15.6:0/8846173 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb7e40039e0 0x7fb7e4003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:26.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:26.608+0000 7fb7f7400640 1 -- 172.21.15.6:0/8846173 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7e4003520 con 0x7fb7e4004800 2024-03-04T09:33:26.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:26.610+0000 7fb7f7400640 1 --2- 172.21.15.6:0/8846173 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb7e4004800 0x7fb7e4004c80 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 crypto rx=0x7fb7e800c240 tx=0x7fb7e4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0c868ba4788ca00 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:33:26.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:26.610+0000 7fb7ff400640 1 -- 172.21.15.6:0/8846173 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7e8007d10 con 0x7fb7e4004800 2024-03-04T09:33:26.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:26.610+0000 7fb7ff400640 1 -- 172.21.15.6:0/8846173 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7e8030070 con 0x7fb7e4004800 2024-03-04T09:33:26.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:26.610+0000 7fb7ff400640 1 -- 172.21.15.6:0/8846173 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7e802cd20 con 0x7fb7e4004800 2024-03-04T09:33:26.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:26.611+0000 7fb805a00640 1 -- 172.21.15.6:0/8846173 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb7e4004800 msgr2=0x7fb7e4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:26.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:26.611+0000 7fb805a00640 1 --2- 172.21.15.6:0/8846173 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb7e4004800 0x7fb7e4004c80 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 crypto rx=0x7fb7e800c240 tx=0x7fb7e4005830 comp rx=0 tx=0).stop 2024-03-04T09:33:26.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:26.611+0000 7fb805000640 1 -- 172.21.15.6:0/8846173 reap_dead start 2024-03-04T09:33:26.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:26.611+0000 7fb805a00640 1 -- 172.21.15.6:0/8846173 shutdown_connections 2024-03-04T09:33:26.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:26.611+0000 7fb805a00640 1 -- 172.21.15.6:0/8846173 >> 172.21.15.6:0/8846173 conn(0x7fb80006b3d0 msgr2=0x7fb80006b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:33:26.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:26.611+0000 7fb805a00640 1 -- 172.21.15.6:0/8846173 shutdown_connections 2024-03-04T09:33:26.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:26.611+0000 7fb805a00640 1 -- 172.21.15.6:0/8846173 wait complete. 2024-03-04T09:33:26.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:26.614+0000 7fb805a00640 1 Processor -- start 2024-03-04T09:33:26.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:26.614+0000 7fb805a00640 1 -- start start 2024-03-04T09:33:26.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:26.615+0000 7fb805a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb7e4004800 0x7fb80015b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:26.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:26.616+0000 7fb805a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb80006e070 0x7fb80013c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:26.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:26.616+0000 7fb805a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb800140340 0x7fb80013c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:26.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:26.616+0000 7fb805a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb80015d700 con 0x7fb800140340 2024-03-04T09:33:26.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:26.616+0000 7fb805a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb80015d580 con 0x7fb7e4004800 2024-03-04T09:33:26.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:26.616+0000 7fb805a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb80015d880 con 0x7fb80006e070 2024-03-04T09:33:26.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:26.616+0000 7fb805000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb800140340 0x7fb80013c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:26.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:26.616+0000 7fb805000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb800140340 0x7fb80013c930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55768/0 (socket says 172.21.15.6:55768) 2024-03-04T09:33:26.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:26.616+0000 7fb805000640 1 -- 172.21.15.6:0/1239074050 learned_addr learned my addr 172.21.15.6:0/1239074050 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:33:26.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:26.616+0000 7fb7ffe00640 1 --2- 172.21.15.6:0/1239074050 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb7e4004800 0x7fb80015b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:26.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:26.616+0000 7fb7f7400640 1 --2- 172.21.15.6:0/1239074050 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb80006e070 0x7fb80013c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:26.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:26.616+0000 7fb805000640 1 -- 172.21.15.6:0/1239074050 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb80006e070 msgr2=0x7fb80013c3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:26.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:26.616+0000 7fb805000640 1 --2- 172.21.15.6:0/1239074050 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb80006e070 0x7fb80013c3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:26.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:26.616+0000 7fb805000640 1 -- 172.21.15.6:0/1239074050 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb7e4004800 msgr2=0x7fb80015b0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:26.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:26.616+0000 7fb805000640 1 --2- 172.21.15.6:0/1239074050 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb7e4004800 0x7fb80015b0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:26.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:26.616+0000 7fb805000640 1 -- 172.21.15.6:0/1239074050 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb800141480 con 0x7fb800140340 2024-03-04T09:33:26.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:26.616+0000 7fb7f7400640 1 --2- 172.21.15.6:0/1239074050 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb80006e070 0x7fb80013c3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:33:26.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:26.617+0000 7fb805000640 1 --2- 172.21.15.6:0/1239074050 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb800140340 0x7fb80013c930 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7fb800072420 tx=0x7fb7f800ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:33:26.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:26.617+0000 7fb7fcc00640 1 -- 172.21.15.6:0/1239074050 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7f800c680 con 0x7fb800140340 2024-03-04T09:33:26.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:26.617+0000 7fb7fcc00640 1 -- 172.21.15.6:0/1239074050 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7f801a070 con 0x7fb800140340 2024-03-04T09:33:26.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:26.617+0000 7fb805a00640 1 -- 172.21.15.6:0/1239074050 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb80015b5f0 con 0x7fb800140340 2024-03-04T09:33:26.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:26.617+0000 7fb7fcc00640 1 -- 172.21.15.6:0/1239074050 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7f800ca60 con 0x7fb800140340 2024-03-04T09:33:26.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:26.617+0000 7fb805a00640 1 -- 172.21.15.6:0/1239074050 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb800140be0 con 0x7fb800140340 2024-03-04T09:33:26.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:26.618+0000 7fb805a00640 1 -- 172.21.15.6:0/1239074050 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb80004f330 con 0x7fb800140340 2024-03-04T09:33:26.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:26.619+0000 7fb7fcc00640 1 -- 172.21.15.6:0/1239074050 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb7f800cc00 con 0x7fb800140340 2024-03-04T09:33:26.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:26.621+0000 7fb7fcc00640 1 --2- 172.21.15.6:0/1239074050 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb7dc06b4d0 0x7fb7dc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:26.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:26.621+0000 7fb7fcc00640 1 -- 172.21.15.6:0/1239074050 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fb7f8089b20 con 0x7fb800140340 2024-03-04T09:33:26.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:26.621+0000 7fb7ffe00640 1 --2- 172.21.15.6:0/1239074050 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb7dc06b4d0 0x7fb7dc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:26.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:26.622+0000 7fb7ffe00640 1 --2- 172.21.15.6:0/1239074050 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb7dc06b4d0 0x7fb7dc06d990 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0007fd0 tx=0x7fb7f0009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:33:26.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:26.625+0000 7fb7fcc00640 1 -- 172.21.15.6:0/1239074050 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb7f8023020 con 0x7fb800140340 2024-03-04T09:33:26.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:26.927+0000 7fb805a00640 1 -- 172.21.15.6:0/1239074050 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb80006bac0 con 0x7fb800140340 2024-03-04T09:33:26.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:26.928+0000 7fb7fcc00640 1 -- 172.21.15.6:0/1239074050 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3415 (secure 0 0 0) 0x7fb7f80588c0 con 0x7fb800140340 2024-03-04T09:33:26.925 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:33:26.926 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":99,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:33:24.637813+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":9}]} 2024-03-04T09:33:26.926 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 99 2024-03-04T09:33:26.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:26.931+0000 7fb805a00640 1 -- 172.21.15.6:0/1239074050 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb7dc06b4d0 msgr2=0x7fb7dc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:26.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:26.931+0000 7fb805a00640 1 --2- 172.21.15.6:0/1239074050 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb7dc06b4d0 0x7fb7dc06d990 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0007fd0 tx=0x7fb7f0009040 comp rx=0 tx=0).stop 2024-03-04T09:33:26.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:26.931+0000 7fb805a00640 1 -- 172.21.15.6:0/1239074050 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb800140340 msgr2=0x7fb80013c930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:26.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:26.931+0000 7fb805a00640 1 --2- 172.21.15.6:0/1239074050 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb800140340 0x7fb80013c930 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7fb800072420 tx=0x7fb7f800ec00 comp rx=0 tx=0).stop 2024-03-04T09:33:26.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:26.931+0000 7fb805a00640 1 -- 172.21.15.6:0/1239074050 shutdown_connections 2024-03-04T09:33:26.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:26.931+0000 7fb805a00640 1 --2- 172.21.15.6:0/1239074050 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb800140340 0x7fb80013c930 unknown :-1 s=CLOSED pgs=1550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:26.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:26.931+0000 7fb805a00640 1 --2- 172.21.15.6:0/1239074050 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb7dc06b4d0 0x7fb7dc06d990 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:26.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:26.931+0000 7fb805a00640 1 --2- 172.21.15.6:0/1239074050 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb80006e070 0x7fb80013c3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:26.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:26.931+0000 7fb805a00640 1 --2- 172.21.15.6:0/1239074050 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb7e4004800 0x7fb80015b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:26.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:26.931+0000 7fb805a00640 1 -- 172.21.15.6:0/1239074050 >> 172.21.15.6:0/1239074050 conn(0x7fb80006b3d0 msgr2=0x7fb8000b1f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:33:26.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:26.931+0000 7fb805a00640 1 -- 172.21.15.6:0/1239074050 shutdown_connections 2024-03-04T09:33:26.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:26.931+0000 7fb805a00640 1 -- 172.21.15.6:0/1239074050 wait complete. 2024-03-04T09:33:27.147 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:32:57.159681+0000', 'modified': '2024-03-04T09:33:24.637813+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8214, 'mds_1': 6796}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6796': {'gid': 6796, 'name': 'd', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.146:6837/2458070840', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 2458070840}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 2458070840}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8214': {'gid': 8214, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2446627517', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2446627517}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2446627517}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8214, 'qdb_cluster': [8214]} 2024-03-04T09:33:27.148 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6796:up:stopping 2024-03-04T09:33:27.770 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-04T09:33:28.148 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:33:28.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:28.378+0000 7f2e27e00640 1 Processor -- start 2024-03-04T09:33:28.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:28.378+0000 7f2e27e00640 1 -- start start 2024-03-04T09:33:28.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:28.379+0000 7f2e27e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2e2815dec0 0x7f2e280ce810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:28.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:28.379+0000 7f2e27e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f2e2815e2a0 0x7f2e280ced50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:28.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:28.379+0000 7f2e27e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e280cf720 con 0x7f2e280cf290 2024-03-04T09:33:28.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:28.379+0000 7f2e27e00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e280b0ea0 con 0x7f2e2815e2a0 2024-03-04T09:33:28.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:28.379+0000 7f2e27e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e280d1120 con 0x7f2e2815dec0 2024-03-04T09:33:28.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:28.380+0000 7f2e27400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f2e280cf290 0x7f2e280d1b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:45214/0 (socket says 172.21.15.6:45214) 2024-03-04T09:33:28.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:28.380+0000 7f2e27400640 1 -- 172.21.15.6:0/110729117 learned_addr learned my addr 172.21.15.6:0/110729117 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:33:28.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:28.380+0000 7f2e1e000640 1 --2- 172.21.15.6:0/110729117 >> v2:172.21.15.146:3300/0 conn(0x7f2e2815e2a0 0x7f2e280ced50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:28.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:28.380+0000 7f2e26a00640 1 --2- 172.21.15.6:0/110729117 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2e2815dec0 0x7f2e280ce810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:28.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:28.380+0000 7f2e26000640 1 -- 172.21.15.6:0/110729117 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4104610222 0 0) 0x7f2e20001110 con 0x7f2e280cf290 2024-03-04T09:33:28.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:28.380+0000 7f2e26000640 1 -- 172.21.15.6:0/110729117 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e0c003520 con 0x7f2e280cf290 2024-03-04T09:33:28.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:28.380+0000 7f2e26000640 1 -- 172.21.15.6:0/110729117 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4281224306 0 0) 0x7f2e0c003520 con 0x7f2e280cf290 2024-03-04T09:33:28.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:28.381+0000 7f2e26000640 1 -- 172.21.15.6:0/110729117 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2e2815dec0 msgr2=0x7f2e280ce810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:33:28.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:28.381+0000 7f2e26000640 1 --2- 172.21.15.6:0/110729117 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2e2815dec0 0x7f2e280ce810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:28.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:28.381+0000 7f2e26000640 1 -- 172.21.15.6:0/110729117 >> v2:172.21.15.146:3300/0 conn(0x7f2e2815e2a0 msgr2=0x7f2e280ced50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:33:28.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:28.381+0000 7f2e26000640 1 --2- 172.21.15.6:0/110729117 >> v2:172.21.15.146:3300/0 conn(0x7f2e2815e2a0 0x7f2e280ced50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:28.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:28.381+0000 7f2e26000640 1 -- 172.21.15.6:0/110729117 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e280d1400 con 0x7f2e280cf290 2024-03-04T09:33:28.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:28.381+0000 7f2e26000640 1 -- 172.21.15.6:0/110729117 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7f2e20001840 con 0x7f2e280cf290 2024-03-04T09:33:28.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:28.381+0000 7f2e26000640 1 -- 172.21.15.6:0/110729117 >> v1:172.21.15.6:6789/0 conn(0x7f2e280cf290 legacy=0x7f2e280d1b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:28.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:28.381+0000 7f2e26000640 1 --2- 172.21.15.6:0/110729117 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2e0c0039b0 0x7f2e0c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:28.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:28.381+0000 7f2e26000640 1 --2- 172.21.15.6:0/110729117 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2e0c004800 0x7f2e0c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:28.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:28.381+0000 7f2e26000640 1 --2- 172.21.15.6:0/110729117 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2e2815dec0 0x7f2e0c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:28.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:28.381+0000 7f2e26000640 1 -- 172.21.15.6:0/110729117 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e280b0ea0 con 0x7f2e0c0039b0 2024-03-04T09:33:28.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:28.381+0000 7f2e26000640 1 -- 172.21.15.6:0/110729117 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e280d1120 con 0x7f2e0c004800 2024-03-04T09:33:28.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:28.381+0000 7f2e26000640 1 -- 172.21.15.6:0/110729117 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e0c00b9c0 con 0x7f2e2815dec0 2024-03-04T09:33:28.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:28.381+0000 7f2e26a00640 1 --2- 172.21.15.6:0/110729117 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2e0c0039b0 0x7f2e0c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:28.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:28.381+0000 7f2e26a00640 1 -- 172.21.15.6:0/110729117 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2e2815dec0 msgr2=0x7f2e0c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:33:28.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:28.381+0000 7f2e26a00640 1 --2- 172.21.15.6:0/110729117 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2e2815dec0 0x7f2e0c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:28.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:28.381+0000 7f2e26a00640 1 -- 172.21.15.6:0/110729117 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2e0c004800 msgr2=0x7f2e0c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:33:28.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:28.381+0000 7f2e26a00640 1 --2- 172.21.15.6:0/110729117 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2e0c004800 0x7f2e0c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:28.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:28.381+0000 7f2e26a00640 1 -- 172.21.15.6:0/110729117 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e0c003520 con 0x7f2e0c0039b0 2024-03-04T09:33:28.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:28.382+0000 7f2e26a00640 1 --2- 172.21.15.6:0/110729117 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2e0c0039b0 0x7f2e0c003dd0 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7f2e18009fc0 tx=0x7f2e0c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3fc0bcfa0e9f200 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:33:28.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:28.383+0000 7f2e26000640 1 -- 172.21.15.6:0/110729117 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e18039070 con 0x7f2e0c0039b0 2024-03-04T09:33:28.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:28.383+0000 7f2e26000640 1 -- 172.21.15.6:0/110729117 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e1802cc60 con 0x7f2e0c0039b0 2024-03-04T09:33:28.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:28.383+0000 7f2e26000640 1 -- 172.21.15.6:0/110729117 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e18009c10 con 0x7f2e0c0039b0 2024-03-04T09:33:28.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:28.383+0000 7f2e27e00640 1 -- 172.21.15.6:0/110729117 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2e0c0039b0 msgr2=0x7f2e0c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:28.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:28.383+0000 7f2e27e00640 1 --2- 172.21.15.6:0/110729117 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2e0c0039b0 0x7f2e0c003dd0 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7f2e18009fc0 tx=0x7f2e0c005830 comp rx=0 tx=0).stop 2024-03-04T09:33:28.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:28.383+0000 7f2e27400640 1 -- 172.21.15.6:0/110729117 reap_dead start 2024-03-04T09:33:28.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:28.383+0000 7f2e27e00640 1 -- 172.21.15.6:0/110729117 shutdown_connections 2024-03-04T09:33:28.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:28.383+0000 7f2e27e00640 1 -- 172.21.15.6:0/110729117 >> 172.21.15.6:0/110729117 conn(0x7f2e280d3d40 msgr2=0x7f2e280d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:33:28.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:28.383+0000 7f2e27e00640 1 -- 172.21.15.6:0/110729117 shutdown_connections 2024-03-04T09:33:28.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:28.383+0000 7f2e27e00640 1 -- 172.21.15.6:0/110729117 wait complete. 2024-03-04T09:33:28.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:28.385+0000 7f2e27e00640 1 Processor -- start 2024-03-04T09:33:28.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:28.385+0000 7f2e27e00640 1 -- start start 2024-03-04T09:33:28.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:28.386+0000 7f2e27e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2e0c0039b0 0x7f2e280b9330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:28.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:28.386+0000 7f2e27e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2e280b9870 0x7f2e280b6920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:28.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:28.386+0000 7f2e27e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2e2815b8a0 0x7f2e280b6e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:28.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:28.386+0000 7f2e27e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e280d0ec0 con 0x7f2e0c0039b0 2024-03-04T09:33:28.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:28.386+0000 7f2e27e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e280d0d40 con 0x7f2e2815b8a0 2024-03-04T09:33:28.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:28.386+0000 7f2e27e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e280b2340 con 0x7f2e280b9870 2024-03-04T09:33:28.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:28.386+0000 7f2e26a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2e0c0039b0 0x7f2e280b9330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:28.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:28.386+0000 7f2e26a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2e0c0039b0 0x7f2e280b9330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55790/0 (socket says 172.21.15.6:55790) 2024-03-04T09:33:28.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:28.386+0000 7f2e26a00640 1 -- 172.21.15.6:0/2897234452 learned_addr learned my addr 172.21.15.6:0/2897234452 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:33:28.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:28.387+0000 7f2e26a00640 1 -- 172.21.15.6:0/2897234452 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2e280b9870 msgr2=0x7f2e280b6920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:28.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:28.387+0000 7f2e26a00640 1 --2- 172.21.15.6:0/2897234452 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2e280b9870 0x7f2e280b6920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:28.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:28.387+0000 7f2e26a00640 1 -- 172.21.15.6:0/2897234452 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2e2815b8a0 msgr2=0x7f2e280b6e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:28.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:28.387+0000 7f2e26a00640 1 --2- 172.21.15.6:0/2897234452 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2e2815b8a0 0x7f2e280b6e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:28.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:28.387+0000 7f2e26a00640 1 -- 172.21.15.6:0/2897234452 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e280cb1f0 con 0x7f2e0c0039b0 2024-03-04T09:33:28.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:28.387+0000 7f2e26a00640 1 --2- 172.21.15.6:0/2897234452 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2e0c0039b0 0x7f2e280b9330 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7f2e1802c6d0 tx=0x7f2e18009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:33:28.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:28.387+0000 7f2e1f400640 1 -- 172.21.15.6:0/2897234452 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e18039050 con 0x7f2e0c0039b0 2024-03-04T09:33:28.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:28.387+0000 7f2e1f400640 1 -- 172.21.15.6:0/2897234452 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e1803a040 con 0x7f2e0c0039b0 2024-03-04T09:33:28.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:28.387+0000 7f2e1f400640 1 -- 172.21.15.6:0/2897234452 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e18034e40 con 0x7f2e0c0039b0 2024-03-04T09:33:28.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:28.387+0000 7f2e27e00640 1 -- 172.21.15.6:0/2897234452 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e280cb4d0 con 0x7f2e0c0039b0 2024-03-04T09:33:28.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:28.388+0000 7f2e27e00640 1 -- 172.21.15.6:0/2897234452 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e280cba90 con 0x7f2e0c0039b0 2024-03-04T09:33:28.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:28.389+0000 7f2e27e00640 1 -- 172.21.15.6:0/2897234452 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e2804f330 con 0x7f2e0c0039b0 2024-03-04T09:33:28.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:28.390+0000 7f2e1f400640 1 -- 172.21.15.6:0/2897234452 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2e18047030 con 0x7f2e0c0039b0 2024-03-04T09:33:28.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:28.391+0000 7f2e1f400640 1 --2- 172.21.15.6:0/2897234452 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2e0406b4d0 0x7f2e0406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:28.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:28.391+0000 7f2e1f400640 1 -- 172.21.15.6:0/2897234452 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f2e180ac5b0 con 0x7f2e0c0039b0 2024-03-04T09:33:28.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:28.391+0000 7f2e1e000640 1 --2- 172.21.15.6:0/2897234452 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2e0406b4d0 0x7f2e0406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:28.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:28.392+0000 7f2e1e000640 1 --2- 172.21.15.6:0/2897234452 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2e0406b4d0 0x7f2e0406d990 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f2e100093d0 tx=0x7f2e10009300 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:33:28.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:28.395+0000 7f2e1f400640 1 -- 172.21.15.6:0/2897234452 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2e1804e060 con 0x7f2e0c0039b0 2024-03-04T09:33:28.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:28.698+0000 7f2e27e00640 1 -- 172.21.15.6:0/2897234452 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2e280b5090 con 0x7f2e0c0039b0 2024-03-04T09:33:28.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:28.699+0000 7f2e1f400640 1 -- 172.21.15.6:0/2897234452 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3415 (secure 0 0 0) 0x7f2e180aae20 con 0x7f2e0c0039b0 2024-03-04T09:33:28.696 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:33:28.696 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":99,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:33:24.637813+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":9}]} 2024-03-04T09:33:28.696 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 99 2024-03-04T09:33:28.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:28.702+0000 7f2e27e00640 1 -- 172.21.15.6:0/2897234452 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2e0406b4d0 msgr2=0x7f2e0406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:28.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:28.702+0000 7f2e27e00640 1 --2- 172.21.15.6:0/2897234452 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2e0406b4d0 0x7f2e0406d990 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f2e100093d0 tx=0x7f2e10009300 comp rx=0 tx=0).stop 2024-03-04T09:33:28.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:28.702+0000 7f2e27e00640 1 -- 172.21.15.6:0/2897234452 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2e0c0039b0 msgr2=0x7f2e280b9330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:28.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:28.702+0000 7f2e27e00640 1 --2- 172.21.15.6:0/2897234452 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2e0c0039b0 0x7f2e280b9330 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7f2e1802c6d0 tx=0x7f2e18009750 comp rx=0 tx=0).stop 2024-03-04T09:33:28.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:28.702+0000 7f2e27e00640 1 -- 172.21.15.6:0/2897234452 shutdown_connections 2024-03-04T09:33:28.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:28.702+0000 7f2e27e00640 1 --2- 172.21.15.6:0/2897234452 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2e2815b8a0 0x7f2e280b6e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:28.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:28.702+0000 7f2e27e00640 1 --2- 172.21.15.6:0/2897234452 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2e0406b4d0 0x7f2e0406d990 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:28.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:28.702+0000 7f2e27e00640 1 --2- 172.21.15.6:0/2897234452 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2e280b9870 0x7f2e280b6920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:28.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:28.702+0000 7f2e27e00640 1 --2- 172.21.15.6:0/2897234452 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2e0c0039b0 0x7f2e280b9330 unknown :-1 s=CLOSED pgs=1553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:28.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:28.702+0000 7f2e27e00640 1 -- 172.21.15.6:0/2897234452 >> 172.21.15.6:0/2897234452 conn(0x7f2e280d3d40 msgr2=0x7f2e280d1f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:33:28.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:28.702+0000 7f2e27e00640 1 -- 172.21.15.6:0/2897234452 shutdown_connections 2024-03-04T09:33:28.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:28.702+0000 7f2e27e00640 1 -- 172.21.15.6:0/2897234452 wait complete. 2024-03-04T09:33:28.921 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:32:57.159681+0000', 'modified': '2024-03-04T09:33:24.637813+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8214, 'mds_1': 6796}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6796': {'gid': 6796, 'name': 'd', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.146:6837/2458070840', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 2458070840}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 2458070840}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8214': {'gid': 8214, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2446627517', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2446627517}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2446627517}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8214, 'qdb_cluster': [8214]} 2024-03-04T09:33:28.922 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6796:up:stopping 2024-03-04T09:33:29.773 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-04T09:33:29.922 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:33:30.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:30.179+0000 7f0571a00640 1 Processor -- start 2024-03-04T09:33:30.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:30.180+0000 7f0571a00640 1 -- start start 2024-03-04T09:33:30.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:30.181+0000 7f0571a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f056c15e2a0 0x7f056c0cef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:30.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:30.182+0000 7f0571a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f056c0cf460 0x7f056c0b1330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:30.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:30.182+0000 7f0571a00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f056c0d1120 con 0x7f056c15dec0 2024-03-04T09:33:30.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:30.182+0000 7f0571a00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f056c0b0ea0 con 0x7f056c0cf460 2024-03-04T09:33:30.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:30.182+0000 7f0571a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f056c0d1400 con 0x7f056c15e2a0 2024-03-04T09:33:30.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:30.182+0000 7f0571000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f056c0cf460 0x7f056c0b1330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:30.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:30.182+0000 7f0571000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f056c0cf460 0x7f056c0b1330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:48684/0 (socket says 172.21.15.6:48684) 2024-03-04T09:33:30.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:30.182+0000 7f0571000640 1 -- 172.21.15.6:0/3463794721 learned_addr learned my addr 172.21.15.6:0/3463794721 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:33:30.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:30.182+0000 7f056b400640 1 --2- 172.21.15.6:0/3463794721 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f056c15e2a0 0x7f056c0cef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:30.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:30.183+0000 7f0571000640 1 -- 172.21.15.6:0/3463794721 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f056c15e2a0 msgr2=0x7f056c0cef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:33:30.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:30.183+0000 7f0571000640 1 --2- 172.21.15.6:0/3463794721 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f056c15e2a0 0x7f056c0cef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:30.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:30.183+0000 7f0571000640 1 -- 172.21.15.6:0/3463794721 >> v1:172.21.15.146:6789/0 conn(0x7f056c15dec0 legacy=0x7f056c0ce810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:30.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:30.183+0000 7f0571000640 1 -- 172.21.15.6:0/3463794721 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f056c0b0b30 con 0x7f056c0cf460 2024-03-04T09:33:30.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:30.184+0000 7f0571000640 1 --2- 172.21.15.6:0/3463794721 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f056c0cf460 0x7f056c0b1330 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 crypto rx=0x7f0564009870 tx=0x7f056402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30aca0abe81f2af2 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:33:30.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:30.184+0000 7f056aa00640 1 -- 172.21.15.6:0/3463794721 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0564034070 con 0x7f056c0cf460 2024-03-04T09:33:30.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:30.184+0000 7f056aa00640 1 -- 172.21.15.6:0/3463794721 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0564030840 con 0x7f056c0cf460 2024-03-04T09:33:30.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:30.184+0000 7f056aa00640 1 -- 172.21.15.6:0/3463794721 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0564030c10 con 0x7f056c0cf460 2024-03-04T09:33:30.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:30.185+0000 7f0571a00640 1 -- 172.21.15.6:0/3463794721 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f056c0cf460 msgr2=0x7f056c0b1330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:30.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:30.185+0000 7f0571a00640 1 --2- 172.21.15.6:0/3463794721 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f056c0cf460 0x7f056c0b1330 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 crypto rx=0x7f0564009870 tx=0x7f056402b980 comp rx=0 tx=0).stop 2024-03-04T09:33:30.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:30.185+0000 7f0571a00640 1 -- 172.21.15.6:0/3463794721 shutdown_connections 2024-03-04T09:33:30.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:30.185+0000 7f0571a00640 1 --2- 172.21.15.6:0/3463794721 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f056c0cf460 0x7f056c0b1330 unknown :-1 s=CLOSED pgs=1554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:30.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:30.185+0000 7f0571a00640 1 --2- 172.21.15.6:0/3463794721 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f056c15e2a0 0x7f056c0cef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:30.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:30.185+0000 7f0571a00640 1 -- 172.21.15.6:0/3463794721 >> 172.21.15.6:0/3463794721 conn(0x7f056c0d3d40 msgr2=0x7f056c0d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:33:30.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:30.185+0000 7f0571a00640 1 -- 172.21.15.6:0/3463794721 shutdown_connections 2024-03-04T09:33:30.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:30.185+0000 7f0571a00640 1 -- 172.21.15.6:0/3463794721 wait complete. 2024-03-04T09:33:30.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:30.187+0000 7f0571a00640 1 Processor -- start 2024-03-04T09:33:30.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:30.187+0000 7f0571a00640 1 -- start start 2024-03-04T09:33:30.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:30.188+0000 7f0571a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f056c0cf460 0x7f056c0c8290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:30.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:30.188+0000 7f0571a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f056c15dec0 0x7f056c0cd190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:30.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:30.188+0000 7f0571a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f056c15e2a0 0x7f056c15a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:30.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:30.188+0000 7f0571a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f056c0d0ec0 con 0x7f056c15dec0 2024-03-04T09:33:30.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:30.188+0000 7f0571a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f056c0d0d40 con 0x7f056c15e2a0 2024-03-04T09:33:30.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:30.188+0000 7f0571a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f056c0b1870 con 0x7f056c0cf460 2024-03-04T09:33:30.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:30.188+0000 7f056b400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f056c15dec0 0x7f056c0cd190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:30.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:30.188+0000 7f056b400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f056c15dec0 0x7f056c0cd190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:48690/0 (socket says 172.21.15.6:48690) 2024-03-04T09:33:30.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:30.188+0000 7f056b400640 1 -- 172.21.15.6:0/1861871942 learned_addr learned my addr 172.21.15.6:0/1861871942 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:33:30.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:30.189+0000 7f056be00640 1 --2- 172.21.15.6:0/1861871942 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f056c0cf460 0x7f056c0c8290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:30.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:30.189+0000 7f0571000640 1 --2- 172.21.15.6:0/1861871942 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f056c15e2a0 0x7f056c15a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:30.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:30.189+0000 7f056b400640 1 -- 172.21.15.6:0/1861871942 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f056c0cf460 msgr2=0x7f056c0c8290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:30.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:30.189+0000 7f056b400640 1 --2- 172.21.15.6:0/1861871942 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f056c0cf460 0x7f056c0c8290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:30.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:30.189+0000 7f056b400640 1 -- 172.21.15.6:0/1861871942 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f056c15e2a0 msgr2=0x7f056c15a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:30.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:30.189+0000 7f056b400640 1 --2- 172.21.15.6:0/1861871942 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f056c15e2a0 0x7f056c15a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:30.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:30.189+0000 7f056b400640 1 -- 172.21.15.6:0/1861871942 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f056c0c8870 con 0x7f056c15dec0 2024-03-04T09:33:30.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:30.189+0000 7f056b400640 1 --2- 172.21.15.6:0/1861871942 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f056c15dec0 0x7f056c0cd190 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7f055800d5f0 tx=0x7f055800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:33:30.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:30.190+0000 7f0563e00640 1 -- 172.21.15.6:0/1861871942 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05580125f0 con 0x7f056c15dec0 2024-03-04T09:33:30.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:30.190+0000 7f0563e00640 1 -- 172.21.15.6:0/1861871942 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0558019070 con 0x7f056c15dec0 2024-03-04T09:33:30.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:30.190+0000 7f0571a00640 1 -- 172.21.15.6:0/1861871942 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f056c0c8b50 con 0x7f056c15dec0 2024-03-04T09:33:30.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:30.190+0000 7f0563e00640 1 -- 172.21.15.6:0/1861871942 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05580129d0 con 0x7f056c15dec0 2024-03-04T09:33:30.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:30.190+0000 7f0571a00640 1 -- 172.21.15.6:0/1861871942 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f056c0c93f0 con 0x7f056c15dec0 2024-03-04T09:33:30.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:30.191+0000 7f0571a00640 1 -- 172.21.15.6:0/1861871942 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f056c0c8d20 con 0x7f056c15dec0 2024-03-04T09:33:30.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:30.192+0000 7f0563e00640 1 -- 172.21.15.6:0/1861871942 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0558012b70 con 0x7f056c15dec0 2024-03-04T09:33:30.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:30.193+0000 7f0563e00640 1 --2- 172.21.15.6:0/1861871942 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f054c06b500 0x7f054c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:30.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:30.193+0000 7f0563e00640 1 -- 172.21.15.6:0/1861871942 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f055808a6b0 con 0x7f056c15dec0 2024-03-04T09:33:30.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:30.193+0000 7f056be00640 1 --2- 172.21.15.6:0/1861871942 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f054c06b500 0x7f054c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:30.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:30.195+0000 7f056be00640 1 --2- 172.21.15.6:0/1861871942 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f054c06b500 0x7f054c06d9c0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f055c0012e0 tx=0x7f055c009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:33:30.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:30.198+0000 7f0563e00640 1 -- 172.21.15.6:0/1861871942 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0558059450 con 0x7f056c15dec0 2024-03-04T09:33:30.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:30.434+0000 7f0571a00640 1 -- 172.21.15.6:0/1861871942 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f056c04f330 con 0x7f056c15dec0 2024-03-04T09:33:30.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:30.435+0000 7f0563e00640 1 -- 172.21.15.6:0/1861871942 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3415 (secure 0 0 0) 0x7f0558058d50 con 0x7f056c15dec0 2024-03-04T09:33:30.432 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:33:30.432 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":99,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:33:24.637813+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":9}]} 2024-03-04T09:33:30.433 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 99 2024-03-04T09:33:30.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:30.437+0000 7f0571a00640 1 -- 172.21.15.6:0/1861871942 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f054c06b500 msgr2=0x7f054c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:30.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:30.438+0000 7f0571a00640 1 --2- 172.21.15.6:0/1861871942 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f054c06b500 0x7f054c06d9c0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f055c0012e0 tx=0x7f055c009040 comp rx=0 tx=0).stop 2024-03-04T09:33:30.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:30.438+0000 7f0571a00640 1 -- 172.21.15.6:0/1861871942 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f056c15dec0 msgr2=0x7f056c0cd190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:30.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:30.438+0000 7f0571a00640 1 --2- 172.21.15.6:0/1861871942 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f056c15dec0 0x7f056c0cd190 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7f055800d5f0 tx=0x7f055800dab0 comp rx=0 tx=0).stop 2024-03-04T09:33:30.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:30.438+0000 7f0571a00640 1 -- 172.21.15.6:0/1861871942 shutdown_connections 2024-03-04T09:33:30.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:30.438+0000 7f0571a00640 1 --2- 172.21.15.6:0/1861871942 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f056c15e2a0 0x7f056c15a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:30.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:30.438+0000 7f0571a00640 1 --2- 172.21.15.6:0/1861871942 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f056c15dec0 0x7f056c0cd190 unknown :-1 s=CLOSED pgs=1555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:30.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:30.438+0000 7f0571a00640 1 --2- 172.21.15.6:0/1861871942 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f054c06b500 0x7f054c06d9c0 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:30.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:30.438+0000 7f0571a00640 1 --2- 172.21.15.6:0/1861871942 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f056c0cf460 0x7f056c0c8290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:30.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:30.438+0000 7f0571a00640 1 -- 172.21.15.6:0/1861871942 >> 172.21.15.6:0/1861871942 conn(0x7f056c0d3d40 msgr2=0x7f056c0d4640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:33:30.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:30.438+0000 7f0571a00640 1 -- 172.21.15.6:0/1861871942 shutdown_connections 2024-03-04T09:33:30.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:30.438+0000 7f0571a00640 1 -- 172.21.15.6:0/1861871942 wait complete. 2024-03-04T09:33:30.655 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:32:57.159681+0000', 'modified': '2024-03-04T09:33:24.637813+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8214, 'mds_1': 6796}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6796': {'gid': 6796, 'name': 'd', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.146:6837/2458070840', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 2458070840}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 2458070840}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8214': {'gid': 8214, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2446627517', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2446627517}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2446627517}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8214, 'qdb_cluster': [8214]} 2024-03-04T09:33:30.655 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6796:up:stopping 2024-03-04T09:33:31.656 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:33:31.774 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-04T09:33:31.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:31.883+0000 7f44f6600640 1 Processor -- start 2024-03-04T09:33:31.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:31.883+0000 7f44f6600640 1 -- start start 2024-03-04T09:33:31.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:31.885+0000 7f44f6600640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f44f006b6c0 0x7f44f015e370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:31.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:31.885+0000 7f44f6600640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44f0161ae0 con 0x7f44f0147400 2024-03-04T09:33:31.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:31.885+0000 7f44f6600640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44f015f080 con 0x7f44f01625a0 2024-03-04T09:33:31.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:31.885+0000 7f44f6600640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44f0162110 con 0x7f44f006b6c0 2024-03-04T09:33:31.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:31.885+0000 7f44f5200640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f44f0147400 0x7f44f006afb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:33476/0 (socket says 172.21.15.6:33476) 2024-03-04T09:33:31.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:31.885+0000 7f44f5200640 1 -- 172.21.15.6:0/2879942205 learned_addr learned my addr 172.21.15.6:0/2879942205 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:33:31.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:31.886+0000 7f44e7e00640 1 --2- 172.21.15.6:0/2879942205 >> v2:172.21.15.146:3300/0 conn(0x7f44f006b6c0 0x7f44f015e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:31.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:31.886+0000 7f44efe00640 1 -- 172.21.15.6:0/2879942205 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1358461244 0 0) 0x7f44f0161ae0 con 0x7f44f0147400 2024-03-04T09:33:31.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:31.886+0000 7f44efe00640 1 -- 172.21.15.6:0/2879942205 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44cc003520 con 0x7f44f0147400 2024-03-04T09:33:31.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:31.886+0000 7f44efe00640 1 -- 172.21.15.6:0/2879942205 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2179185323 0 0) 0x7f44cc003520 con 0x7f44f0147400 2024-03-04T09:33:31.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:31.886+0000 7f44efe00640 1 -- 172.21.15.6:0/2879942205 >> v2:172.21.15.146:3300/0 conn(0x7f44f006b6c0 msgr2=0x7f44f015e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:33:31.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:31.886+0000 7f44efe00640 1 --2- 172.21.15.6:0/2879942205 >> v2:172.21.15.146:3300/0 conn(0x7f44f006b6c0 0x7f44f015e370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:31.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:31.886+0000 7f44efe00640 1 -- 172.21.15.6:0/2879942205 >> v1:172.21.15.146:6789/0 conn(0x7f44f01625a0 legacy=0x7f44f015e8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:31.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:31.886+0000 7f44efe00640 1 -- 172.21.15.6:0/2879942205 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44f0161da0 con 0x7f44f0147400 2024-03-04T09:33:31.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:31.886+0000 7f44efe00640 1 -- 172.21.15.6:0/2879942205 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7f44d4001670 con 0x7f44f0147400 2024-03-04T09:33:31.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:31.886+0000 7f44efe00640 1 -- 172.21.15.6:0/2879942205 >> v1:172.21.15.6:6789/0 conn(0x7f44f0147400 legacy=0x7f44f006afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:31.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:31.886+0000 7f44efe00640 1 --2- 172.21.15.6:0/2879942205 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f44cc0039b0 0x7f44cc003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:31.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:31.886+0000 7f44efe00640 1 --2- 172.21.15.6:0/2879942205 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f44cc004800 0x7f44cc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:31.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:31.886+0000 7f44efe00640 1 --2- 172.21.15.6:0/2879942205 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f44cc0051c0 0x7f44cc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:31.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:31.886+0000 7f44efe00640 1 -- 172.21.15.6:0/2879942205 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44f0162110 con 0x7f44cc0051c0 2024-03-04T09:33:31.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:31.886+0000 7f44efe00640 1 -- 172.21.15.6:0/2879942205 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44cc00c090 con 0x7f44cc0039b0 2024-03-04T09:33:31.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:31.886+0000 7f44efe00640 1 -- 172.21.15.6:0/2879942205 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44cc00c290 con 0x7f44cc004800 2024-03-04T09:33:31.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:31.887+0000 7f44f5c00640 1 --2- 172.21.15.6:0/2879942205 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f44cc0051c0 0x7f44cc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:31.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:31.887+0000 7f44f5200640 1 --2- 172.21.15.6:0/2879942205 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f44cc0039b0 0x7f44cc003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:31.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:31.887+0000 7f44e7e00640 1 --2- 172.21.15.6:0/2879942205 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f44cc004800 0x7f44cc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:31.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:31.887+0000 7f44f5c00640 1 -- 172.21.15.6:0/2879942205 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f44cc004800 msgr2=0x7f44cc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:33:31.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:31.887+0000 7f44f5c00640 1 --2- 172.21.15.6:0/2879942205 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f44cc004800 0x7f44cc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:31.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:31.887+0000 7f44f5c00640 1 -- 172.21.15.6:0/2879942205 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f44cc0039b0 msgr2=0x7f44cc003df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:33:31.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:31.887+0000 7f44f5c00640 1 --2- 172.21.15.6:0/2879942205 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f44cc0039b0 0x7f44cc003df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:31.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:31.887+0000 7f44f5c00640 1 -- 172.21.15.6:0/2879942205 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44cc003520 con 0x7f44cc0051c0 2024-03-04T09:33:31.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:31.889+0000 7f44f5c00640 1 -- 172.21.15.6:0/2879942205 reap_dead start 2024-03-04T09:33:31.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:31.889+0000 7f44f5c00640 1 --2- 172.21.15.6:0/2879942205 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f44cc0051c0 0x7f44cc00b9c0 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 crypto rx=0x7f44e8001e20 tx=0x7f44cc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68a5b2b30a58f454 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:33:31.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:31.889+0000 7f44efe00640 1 -- 172.21.15.6:0/2879942205 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44e8001a70 con 0x7f44cc0051c0 2024-03-04T09:33:31.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:31.889+0000 7f44efe00640 1 -- 172.21.15.6:0/2879942205 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44e8003b70 con 0x7f44cc0051c0 2024-03-04T09:33:31.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:31.889+0000 7f44efe00640 1 -- 172.21.15.6:0/2879942205 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44e80050d0 con 0x7f44cc0051c0 2024-03-04T09:33:31.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:31.890+0000 7f44f6600640 1 -- 172.21.15.6:0/2879942205 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f44cc0051c0 msgr2=0x7f44cc00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:31.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:31.890+0000 7f44f6600640 1 --2- 172.21.15.6:0/2879942205 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f44cc0051c0 0x7f44cc00b9c0 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 crypto rx=0x7f44e8001e20 tx=0x7f44cc005830 comp rx=0 tx=0).stop 2024-03-04T09:33:31.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:31.890+0000 7f44f6600640 1 -- 172.21.15.6:0/2879942205 shutdown_connections 2024-03-04T09:33:31.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:31.890+0000 7f44f6600640 1 --2- 172.21.15.6:0/2879942205 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f44cc0051c0 0x7f44cc00b9c0 unknown :-1 s=CLOSED pgs=1557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:31.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:31.890+0000 7f44f6600640 1 -- 172.21.15.6:0/2879942205 >> 172.21.15.6:0/2879942205 conn(0x7f44f006c0c0 msgr2=0x7f44f006c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:33:31.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:31.890+0000 7f44f6600640 1 -- 172.21.15.6:0/2879942205 shutdown_connections 2024-03-04T09:33:31.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:31.890+0000 7f44f6600640 1 -- 172.21.15.6:0/2879942205 wait complete. 2024-03-04T09:33:31.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:31.892+0000 7f44f6600640 1 Processor -- start 2024-03-04T09:33:31.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:31.892+0000 7f44f6600640 1 -- start start 2024-03-04T09:33:31.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:31.893+0000 7f44f6600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f44cc0051c0 0x7f44f01f77e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:31.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:31.893+0000 7f44f6600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f44f00ae6a0 0x7f44f00aeb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:31.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:31.893+0000 7f44f6600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f44f013d820 0x7f44f013dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:31.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:31.893+0000 7f44f6600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44f006b130 con 0x7f44f013d820 2024-03-04T09:33:31.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:31.893+0000 7f44f6600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44f006afb0 con 0x7f44cc0051c0 2024-03-04T09:33:31.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:31.893+0000 7f44f6600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44f006b2b0 con 0x7f44f00ae6a0 2024-03-04T09:33:31.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:31.893+0000 7f44f5c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f44f013d820 0x7f44f013dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:31.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:31.893+0000 7f44f5c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f44f013d820 0x7f44f013dcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:48702/0 (socket says 172.21.15.6:48702) 2024-03-04T09:33:31.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:31.893+0000 7f44f5c00640 1 -- 172.21.15.6:0/153232133 learned_addr learned my addr 172.21.15.6:0/153232133 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:33:31.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:31.894+0000 7f44e7e00640 1 --2- 172.21.15.6:0/153232133 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f44f00ae6a0 0x7f44f00aeb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:31.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:31.894+0000 7f44f5200640 1 --2- 172.21.15.6:0/153232133 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f44cc0051c0 0x7f44f01f77e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:31.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:31.894+0000 7f44f5c00640 1 -- 172.21.15.6:0/153232133 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f44f00ae6a0 msgr2=0x7f44f00aeb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:31.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:31.894+0000 7f44f5c00640 1 --2- 172.21.15.6:0/153232133 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f44f00ae6a0 0x7f44f00aeb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:31.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:31.894+0000 7f44f5c00640 1 -- 172.21.15.6:0/153232133 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f44cc0051c0 msgr2=0x7f44f01f77e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:31.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:31.894+0000 7f44f5c00640 1 --2- 172.21.15.6:0/153232133 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f44cc0051c0 0x7f44f01f77e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:31.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:31.894+0000 7f44f5c00640 1 -- 172.21.15.6:0/153232133 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44f01f3ea0 con 0x7f44f013d820 2024-03-04T09:33:31.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:31.894+0000 7f44f5200640 1 --2- 172.21.15.6:0/153232133 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f44cc0051c0 0x7f44f01f77e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:33:31.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:31.894+0000 7f44f5c00640 1 --2- 172.21.15.6:0/153232133 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f44f013d820 0x7f44f013dcd0 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x7f44e8001e20 tx=0x7f44e80014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:33:31.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:31.894+0000 7f44ed600640 1 -- 172.21.15.6:0/153232133 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44e8003970 con 0x7f44f013d820 2024-03-04T09:33:31.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:31.894+0000 7f44ed600640 1 -- 172.21.15.6:0/153232133 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44e8006b50 con 0x7f44f013d820 2024-03-04T09:33:31.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:31.894+0000 7f44ed600640 1 -- 172.21.15.6:0/153232133 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44e8007020 con 0x7f44f013d820 2024-03-04T09:33:31.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:31.894+0000 7f44f6600640 1 -- 172.21.15.6:0/153232133 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44f01f4180 con 0x7f44f013d820 2024-03-04T09:33:31.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:31.895+0000 7f44f6600640 1 -- 172.21.15.6:0/153232133 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44f00ae180 con 0x7f44f013d820 2024-03-04T09:33:31.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:31.896+0000 7f44f6600640 1 -- 172.21.15.6:0/153232133 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44f01f46a0 con 0x7f44f013d820 2024-03-04T09:33:31.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:31.897+0000 7f44ed600640 1 -- 172.21.15.6:0/153232133 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f44e8049060 con 0x7f44f013d820 2024-03-04T09:33:31.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:31.898+0000 7f44ed600640 1 --2- 172.21.15.6:0/153232133 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f44c406b500 0x7f44c406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:31.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:31.898+0000 7f44ed600640 1 -- 172.21.15.6:0/153232133 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f44e80ab4e0 con 0x7f44f013d820 2024-03-04T09:33:31.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:31.899+0000 7f44f5200640 1 --2- 172.21.15.6:0/153232133 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f44c406b500 0x7f44c406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:31.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:31.900+0000 7f44f5200640 1 --2- 172.21.15.6:0/153232133 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f44c406b500 0x7f44c406d9c0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f44f00af7a0 tx=0x7f44d4000ef0 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:33:31.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:31.903+0000 7f44ed600640 1 -- 172.21.15.6:0/153232133 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f44f01f46a0 con 0x7f44f013d820 2024-03-04T09:33:32.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:32.141+0000 7f44f6600640 1 -- 172.21.15.6:0/153232133 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f44f00b5de0 con 0x7f44f013d820 2024-03-04T09:33:32.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:32.143+0000 7f44ed600640 1 -- 172.21.15.6:0/153232133 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3415 (secure 0 0 0) 0x7f44f00b5de0 con 0x7f44f013d820 2024-03-04T09:33:32.140 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:33:32.140 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":99,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:33:24.637813+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":9}]} 2024-03-04T09:33:32.140 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 99 2024-03-04T09:33:32.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:32.145+0000 7f44f6600640 1 -- 172.21.15.6:0/153232133 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f44c406b500 msgr2=0x7f44c406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:32.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:32.145+0000 7f44f6600640 1 --2- 172.21.15.6:0/153232133 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f44c406b500 0x7f44c406d9c0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f44f00af7a0 tx=0x7f44d4000ef0 comp rx=0 tx=0).stop 2024-03-04T09:33:32.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:32.145+0000 7f44f6600640 1 -- 172.21.15.6:0/153232133 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f44f013d820 msgr2=0x7f44f013dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:32.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:32.145+0000 7f44f6600640 1 --2- 172.21.15.6:0/153232133 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f44f013d820 0x7f44f013dcd0 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x7f44e8001e20 tx=0x7f44e80014a0 comp rx=0 tx=0).stop 2024-03-04T09:33:32.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:32.146+0000 7f44f6600640 1 -- 172.21.15.6:0/153232133 shutdown_connections 2024-03-04T09:33:32.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:32.146+0000 7f44f6600640 1 --2- 172.21.15.6:0/153232133 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f44f013d820 0x7f44f013dcd0 unknown :-1 s=CLOSED pgs=1558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:32.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:32.146+0000 7f44f6600640 1 --2- 172.21.15.6:0/153232133 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f44c406b500 0x7f44c406d9c0 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:32.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:32.146+0000 7f44f6600640 1 --2- 172.21.15.6:0/153232133 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f44f00ae6a0 0x7f44f00aeb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:32.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:32.146+0000 7f44f6600640 1 --2- 172.21.15.6:0/153232133 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f44cc0051c0 0x7f44f01f77e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:32.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:32.146+0000 7f44f6600640 1 -- 172.21.15.6:0/153232133 >> 172.21.15.6:0/153232133 conn(0x7f44f006c0c0 msgr2=0x7f44f006c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:33:32.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:32.146+0000 7f44f6600640 1 -- 172.21.15.6:0/153232133 shutdown_connections 2024-03-04T09:33:32.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:32.146+0000 7f44f6600640 1 -- 172.21.15.6:0/153232133 wait complete. 2024-03-04T09:33:32.364 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:32:57.159681+0000', 'modified': '2024-03-04T09:33:24.637813+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8214, 'mds_1': 6796}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6796': {'gid': 6796, 'name': 'd', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.146:6837/2458070840', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 2458070840}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 2458070840}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8214': {'gid': 8214, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2446627517', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2446627517}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2446627517}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8214, 'qdb_cluster': [8214]} 2024-03-04T09:33:32.365 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6796:up:stopping 2024-03-04T09:33:33.366 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:33:33.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:33.631+0000 7f3aed400640 1 Processor -- start 2024-03-04T09:33:33.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:33.631+0000 7f3aed400640 1 -- start start 2024-03-04T09:33:33.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:33.633+0000 7f3aed400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f3ae8071b60 0x7f3ae80b3f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:33.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:33.633+0000 7f3aed400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3ae80b4a70 0x7f3ae815b320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:33.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:33.633+0000 7f3aed400640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ae80b31d0 con 0x7f3ae80cede0 2024-03-04T09:33:33.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:33.633+0000 7f3aed400640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ae80b3b00 con 0x7f3ae8071b60 2024-03-04T09:33:33.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:33.633+0000 7f3aed400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ae80b2f70 con 0x7f3ae80b4a70 2024-03-04T09:33:33.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:33.633+0000 7f3adfe00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f3ae8071b60 0x7f3ae80b3f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:33.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:33.633+0000 7f3aeca00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3ae80b4a70 0x7f3ae815b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:33.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:33.633+0000 7f3adfe00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f3ae8071b60 0x7f3ae80b3f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.6:38578/0 (socket says 172.21.15.6:38578) 2024-03-04T09:33:33.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:33.633+0000 7f3adfe00640 1 -- 172.21.15.6:0/754191682 learned_addr learned my addr 172.21.15.6:0/754191682 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:33:33.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:33.634+0000 7f3aeca00640 1 -- 172.21.15.6:0/754191682 >> v2:172.21.15.146:3300/0 conn(0x7f3ae8071b60 msgr2=0x7f3ae80b3f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:33:33.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:33.634+0000 7f3aeca00640 1 --2- 172.21.15.6:0/754191682 >> v2:172.21.15.146:3300/0 conn(0x7f3ae8071b60 0x7f3ae80b3f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:33.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:33.634+0000 7f3aeca00640 1 -- 172.21.15.6:0/754191682 >> v1:172.21.15.146:6789/0 conn(0x7f3ae80cede0 legacy=0x7f3ae8071450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:33.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:33.634+0000 7f3aeca00640 1 -- 172.21.15.6:0/754191682 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ae80b34b0 con 0x7f3ae80b4a70 2024-03-04T09:33:33.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:33.636+0000 7f3aeca00640 1 --2- 172.21.15.6:0/754191682 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3ae80b4a70 0x7f3ae815b320 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f3ae0009d90 tx=0x7f3ae002be20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ae400bc02489656d server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:33:33.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:33.636+0000 7f3ae7400640 1 -- 172.21.15.6:0/754191682 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ae0039070 con 0x7f3ae80b4a70 2024-03-04T09:33:33.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:33.636+0000 7f3ae7400640 1 -- 172.21.15.6:0/754191682 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ae002cbd0 con 0x7f3ae80b4a70 2024-03-04T09:33:33.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:33.636+0000 7f3ae7400640 1 -- 172.21.15.6:0/754191682 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ae0034440 con 0x7f3ae80b4a70 2024-03-04T09:33:33.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:33.637+0000 7f3aed400640 1 -- 172.21.15.6:0/754191682 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3ae80b4a70 msgr2=0x7f3ae815b320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:33.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:33.637+0000 7f3aed400640 1 --2- 172.21.15.6:0/754191682 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3ae80b4a70 0x7f3ae815b320 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f3ae0009d90 tx=0x7f3ae002be20 comp rx=0 tx=0).stop 2024-03-04T09:33:33.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:33.637+0000 7f3aed400640 1 -- 172.21.15.6:0/754191682 shutdown_connections 2024-03-04T09:33:33.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:33.637+0000 7f3aed400640 1 --2- 172.21.15.6:0/754191682 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3ae80b4a70 0x7f3ae815b320 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:33.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:33.637+0000 7f3aed400640 1 --2- 172.21.15.6:0/754191682 >> v2:172.21.15.146:3300/0 conn(0x7f3ae8071b60 0x7f3ae80b3f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:33.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:33.637+0000 7f3aed400640 1 -- 172.21.15.6:0/754191682 >> 172.21.15.6:0/754191682 conn(0x7f3ae806b3d0 msgr2=0x7f3ae806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:33:33.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:33.637+0000 7f3aed400640 1 -- 172.21.15.6:0/754191682 shutdown_connections 2024-03-04T09:33:33.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:33.637+0000 7f3aed400640 1 -- 172.21.15.6:0/754191682 wait complete. 2024-03-04T09:33:33.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:33.639+0000 7f3aed400640 1 Processor -- start 2024-03-04T09:33:33.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:33.639+0000 7f3aed400640 1 -- start start 2024-03-04T09:33:33.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:33.640+0000 7f3aed400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3ae8071b60 0x7f3ae8151850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:33.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:33.640+0000 7f3aed400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3ae80b4a70 0x7f3ae81589a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:33.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:33.640+0000 7f3aed400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3ae80cede0 0x7f3ae80af750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:33.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:33.640+0000 7f3aed400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ae80cc950 con 0x7f3ae8071b60 2024-03-04T09:33:33.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:33.640+0000 7f3aed400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ae80cf1c0 con 0x7f3ae80b4a70 2024-03-04T09:33:33.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:33.640+0000 7f3aed400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ae80ccad0 con 0x7f3ae80cede0 2024-03-04T09:33:33.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:33.641+0000 7f3ae7e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3ae8071b60 0x7f3ae8151850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:33.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:33.641+0000 7f3ae7e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3ae8071b60 0x7f3ae8151850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:48718/0 (socket says 172.21.15.6:48718) 2024-03-04T09:33:33.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:33.641+0000 7f3ae7e00640 1 -- 172.21.15.6:0/2879329131 learned_addr learned my addr 172.21.15.6:0/2879329131 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:33:33.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:33.641+0000 7f3adfe00640 1 --2- 172.21.15.6:0/2879329131 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3ae80b4a70 0x7f3ae81589a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:33.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:33.641+0000 7f3aeca00640 1 --2- 172.21.15.6:0/2879329131 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3ae80cede0 0x7f3ae80af750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:33.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:33.641+0000 7f3ae7e00640 1 -- 172.21.15.6:0/2879329131 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3ae80cede0 msgr2=0x7f3ae80af750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:33.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:33.641+0000 7f3ae7e00640 1 --2- 172.21.15.6:0/2879329131 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3ae80cede0 0x7f3ae80af750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:33.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:33.641+0000 7f3ae7e00640 1 -- 172.21.15.6:0/2879329131 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3ae80b4a70 msgr2=0x7f3ae81589a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:33.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:33.641+0000 7f3ae7e00640 1 --2- 172.21.15.6:0/2879329131 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3ae80b4a70 0x7f3ae81589a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:33.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:33.641+0000 7f3ae7e00640 1 -- 172.21.15.6:0/2879329131 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ae8159310 con 0x7f3ae8071b60 2024-03-04T09:33:33.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:33.642+0000 7f3ae7e00640 1 --2- 172.21.15.6:0/2879329131 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3ae8071b60 0x7f3ae8151850 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7f3ad800c3c0 tx=0x7f3ad800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:33:33.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:33.642+0000 7f3ae4c00640 1 -- 172.21.15.6:0/2879329131 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ad80052d0 con 0x7f3ae8071b60 2024-03-04T09:33:33.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:33.642+0000 7f3ae4c00640 1 -- 172.21.15.6:0/2879329131 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ad8005490 con 0x7f3ae8071b60 2024-03-04T09:33:33.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:33.642+0000 7f3ae4c00640 1 -- 172.21.15.6:0/2879329131 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ad8007260 con 0x7f3ae8071b60 2024-03-04T09:33:33.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:33.642+0000 7f3aed400640 1 -- 172.21.15.6:0/2879329131 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ae81595f0 con 0x7f3ae8071b60 2024-03-04T09:33:33.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:33.642+0000 7f3aed400640 1 -- 172.21.15.6:0/2879329131 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ae8159bb0 con 0x7f3ae8071b60 2024-03-04T09:33:33.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:33.644+0000 7f3aed400640 1 -- 172.21.15.6:0/2879329131 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ae804f330 con 0x7f3ae8071b60 2024-03-04T09:33:33.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:33.644+0000 7f3ae4c00640 1 -- 172.21.15.6:0/2879329131 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3ad8007400 con 0x7f3ae8071b60 2024-03-04T09:33:33.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:33.645+0000 7f3ae4c00640 1 --2- 172.21.15.6:0/2879329131 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f3ac406b4d0 0x7f3ac406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:33.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:33.645+0000 7f3ae4c00640 1 -- 172.21.15.6:0/2879329131 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f3ad80897c0 con 0x7f3ae8071b60 2024-03-04T09:33:33.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:33.646+0000 7f3adfe00640 1 --2- 172.21.15.6:0/2879329131 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f3ac406b4d0 0x7f3ac406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:33.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:33.647+0000 7f3adfe00640 1 --2- 172.21.15.6:0/2879329131 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f3ac406b4d0 0x7f3ac406d990 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f3ae80b0b40 tx=0x7f3ad000b450 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:33:33.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:33.655+0000 7f3ae4c00640 1 -- 172.21.15.6:0/2879329131 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3ad8058560 con 0x7f3ae8071b60 2024-03-04T09:33:33.777 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-04T09:33:33.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:33.896+0000 7f3aed400640 1 -- 172.21.15.6:0/2879329131 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3ae806dc50 con 0x7f3ae8071b60 2024-03-04T09:33:33.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:33.898+0000 7f3ae4c00640 1 -- 172.21.15.6:0/2879329131 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3415 (secure 0 0 0) 0x7f3ad8057e60 con 0x7f3ae8071b60 2024-03-04T09:33:33.895 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:33:33.895 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":99,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:33:24.637813+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":9}]} 2024-03-04T09:33:33.895 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 99 2024-03-04T09:33:33.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:33.901+0000 7f3aed400640 1 -- 172.21.15.6:0/2879329131 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f3ac406b4d0 msgr2=0x7f3ac406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:33.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:33.901+0000 7f3aed400640 1 --2- 172.21.15.6:0/2879329131 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f3ac406b4d0 0x7f3ac406d990 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f3ae80b0b40 tx=0x7f3ad000b450 comp rx=0 tx=0).stop 2024-03-04T09:33:33.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:33.901+0000 7f3aed400640 1 -- 172.21.15.6:0/2879329131 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3ae8071b60 msgr2=0x7f3ae8151850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:33.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:33.901+0000 7f3aed400640 1 --2- 172.21.15.6:0/2879329131 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3ae8071b60 0x7f3ae8151850 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7f3ad800c3c0 tx=0x7f3ad800c880 comp rx=0 tx=0).stop 2024-03-04T09:33:33.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:33.901+0000 7f3aed400640 1 -- 172.21.15.6:0/2879329131 shutdown_connections 2024-03-04T09:33:33.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:33.901+0000 7f3aed400640 1 --2- 172.21.15.6:0/2879329131 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f3ac406b4d0 0x7f3ac406d990 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:33.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:33.901+0000 7f3aed400640 1 --2- 172.21.15.6:0/2879329131 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3ae80cede0 0x7f3ae80af750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:33.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:33.901+0000 7f3aed400640 1 --2- 172.21.15.6:0/2879329131 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3ae80b4a70 0x7f3ae81589a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:33.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:33.901+0000 7f3aed400640 1 --2- 172.21.15.6:0/2879329131 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3ae8071b60 0x7f3ae8151850 unknown :-1 s=CLOSED pgs=1559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:33.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:33.901+0000 7f3aed400640 1 -- 172.21.15.6:0/2879329131 >> 172.21.15.6:0/2879329131 conn(0x7f3ae806b3d0 msgr2=0x7f3ae8154270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:33:33.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:33.901+0000 7f3aed400640 1 -- 172.21.15.6:0/2879329131 shutdown_connections 2024-03-04T09:33:33.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:33.901+0000 7f3aed400640 1 -- 172.21.15.6:0/2879329131 wait complete. 2024-03-04T09:33:34.118 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:32:57.159681+0000', 'modified': '2024-03-04T09:33:24.637813+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8214, 'mds_1': 6796}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6796': {'gid': 6796, 'name': 'd', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.146:6837/2458070840', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 2458070840}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 2458070840}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8214': {'gid': 8214, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2446627517', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2446627517}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2446627517}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8214, 'qdb_cluster': [8214]} 2024-03-04T09:33:34.118 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6796:up:stopping 2024-03-04T09:33:35.119 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:33:35.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:35.360+0000 7f7a67e00640 1 Processor -- start 2024-03-04T09:33:35.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:35.361+0000 7f7a67e00640 1 -- start start 2024-03-04T09:33:35.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:35.362+0000 7f7a67e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f7a680b0f90 0x7f7a680cf080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:35.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:35.362+0000 7f7a67e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7a680cc730 0x7f7a680d00a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:35.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:35.362+0000 7f7a67e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a680b1370 con 0x7f7a680d05e0 2024-03-04T09:33:35.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:35.362+0000 7f7a67e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a680d0a50 con 0x7f7a680b0f90 2024-03-04T09:33:35.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:35.362+0000 7f7a67e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a680ccf80 con 0x7f7a680cc730 2024-03-04T09:33:35.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:35.362+0000 7f7a66a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f7a680b0f90 0x7f7a680cf080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:35.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:35.363+0000 7f7a66a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f7a680b0f90 0x7f7a680cf080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:48722/0 (socket says 172.21.15.6:48722) 2024-03-04T09:33:35.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:35.363+0000 7f7a66a00640 1 -- 172.21.15.6:0/806106854 learned_addr learned my addr 172.21.15.6:0/806106854 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:33:35.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:35.363+0000 7f7a66000640 1 --2- 172.21.15.6:0/806106854 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7a680cc730 0x7f7a680d00a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:35.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:35.363+0000 7f7a66a00640 1 -- 172.21.15.6:0/806106854 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7a680cc730 msgr2=0x7f7a680d00a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:33:35.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:35.363+0000 7f7a66a00640 1 --2- 172.21.15.6:0/806106854 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7a680cc730 0x7f7a680d00a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:35.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:35.363+0000 7f7a66a00640 1 -- 172.21.15.6:0/806106854 >> v1:172.21.15.6:6789/0 conn(0x7f7a680d05e0 legacy=0x7f7a6815d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:35.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:35.363+0000 7f7a66a00640 1 -- 172.21.15.6:0/806106854 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a680cd260 con 0x7f7a680b0f90 2024-03-04T09:33:35.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:35.364+0000 7f7a66a00640 1 --2- 172.21.15.6:0/806106854 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7a680b0f90 0x7f7a680cf080 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7f7a58009870 tx=0x7f7a5802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7cf4045cfcf8208 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:33:35.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:35.364+0000 7f7a65600640 1 -- 172.21.15.6:0/806106854 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a58035020 con 0x7f7a680b0f90 2024-03-04T09:33:35.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:35.365+0000 7f7a65600640 1 -- 172.21.15.6:0/806106854 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a58031640 con 0x7f7a680b0f90 2024-03-04T09:33:35.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:35.365+0000 7f7a65600640 1 -- 172.21.15.6:0/806106854 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a5802d040 con 0x7f7a680b0f90 2024-03-04T09:33:35.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:35.365+0000 7f7a67e00640 1 -- 172.21.15.6:0/806106854 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7a680b0f90 msgr2=0x7f7a680cf080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:35.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:35.365+0000 7f7a67e00640 1 --2- 172.21.15.6:0/806106854 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7a680b0f90 0x7f7a680cf080 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7f7a58009870 tx=0x7f7a5802b8f0 comp rx=0 tx=0).stop 2024-03-04T09:33:35.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:35.365+0000 7f7a67e00640 1 -- 172.21.15.6:0/806106854 shutdown_connections 2024-03-04T09:33:35.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:35.365+0000 7f7a67e00640 1 --2- 172.21.15.6:0/806106854 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7a680cc730 0x7f7a680d00a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:35.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:35.365+0000 7f7a67e00640 1 --2- 172.21.15.6:0/806106854 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7a680b0f90 0x7f7a680cf080 unknown :-1 s=CLOSED pgs=1561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:35.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:35.365+0000 7f7a67e00640 1 -- 172.21.15.6:0/806106854 >> 172.21.15.6:0/806106854 conn(0x7f7a680d1ae0 msgr2=0x7f7a680d1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:33:35.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:35.365+0000 7f7a67e00640 1 -- 172.21.15.6:0/806106854 shutdown_connections 2024-03-04T09:33:35.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:35.365+0000 7f7a67e00640 1 -- 172.21.15.6:0/806106854 wait complete. 2024-03-04T09:33:35.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:35.368+0000 7f7a67e00640 1 Processor -- start 2024-03-04T09:33:35.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:35.368+0000 7f7a67e00640 1 -- start start 2024-03-04T09:33:35.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:35.370+0000 7f7a67e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7a680b0f90 0x7f7a6815cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:35.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:35.370+0000 7f7a67e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7a680cc730 0x7f7a680c55f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:35.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:35.370+0000 7f7a67e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7a680d05e0 0x7f7a68157f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:35.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:35.370+0000 7f7a67e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a680ccb70 con 0x7f7a680d05e0 2024-03-04T09:33:35.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:35.370+0000 7f7a67e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a680cd820 con 0x7f7a680b0f90 2024-03-04T09:33:35.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:35.370+0000 7f7a67e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a680cccf0 con 0x7f7a680cc730 2024-03-04T09:33:35.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:35.370+0000 7f7a67400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7a680d05e0 0x7f7a68157f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:35.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:35.370+0000 7f7a67400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7a680d05e0 0x7f7a68157f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:48736/0 (socket says 172.21.15.6:48736) 2024-03-04T09:33:35.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:35.370+0000 7f7a67400640 1 -- 172.21.15.6:0/1254648832 learned_addr learned my addr 172.21.15.6:0/1254648832 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:33:35.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:35.371+0000 7f7a66a00640 1 --2- 172.21.15.6:0/1254648832 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7a680b0f90 0x7f7a6815cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:35.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:35.371+0000 7f7a66000640 1 --2- 172.21.15.6:0/1254648832 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7a680cc730 0x7f7a680c55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:35.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:35.371+0000 7f7a67400640 1 -- 172.21.15.6:0/1254648832 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7a680cc730 msgr2=0x7f7a680c55f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:35.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:35.371+0000 7f7a67400640 1 --2- 172.21.15.6:0/1254648832 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7a680cc730 0x7f7a680c55f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:35.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:35.371+0000 7f7a67400640 1 -- 172.21.15.6:0/1254648832 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7a680b0f90 msgr2=0x7f7a6815cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:35.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:35.371+0000 7f7a67400640 1 --2- 172.21.15.6:0/1254648832 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7a680b0f90 0x7f7a6815cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:35.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:35.371+0000 7f7a67400640 1 -- 172.21.15.6:0/1254648832 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a680c5d60 con 0x7f7a680d05e0 2024-03-04T09:33:35.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:35.371+0000 7f7a66000640 1 --2- 172.21.15.6:0/1254648832 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7a680cc730 0x7f7a680c55f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:33:35.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:35.371+0000 7f7a66a00640 1 --2- 172.21.15.6:0/1254648832 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7a680b0f90 0x7f7a6815cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:33:35.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:35.371+0000 7f7a67400640 1 --2- 172.21.15.6:0/1254648832 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7a680d05e0 0x7f7a68157f50 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7f7a6000c740 tx=0x7f7a6000cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:33:35.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:35.371+0000 7f7a5ea00640 1 -- 172.21.15.6:0/1254648832 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a60016020 con 0x7f7a680d05e0 2024-03-04T09:33:35.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:35.371+0000 7f7a5ea00640 1 -- 172.21.15.6:0/1254648832 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a6000ce50 con 0x7f7a680d05e0 2024-03-04T09:33:35.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:35.371+0000 7f7a67e00640 1 -- 172.21.15.6:0/1254648832 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a680c6040 con 0x7f7a680d05e0 2024-03-04T09:33:35.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:35.371+0000 7f7a5ea00640 1 -- 172.21.15.6:0/1254648832 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a60003970 con 0x7f7a680d05e0 2024-03-04T09:33:35.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:35.372+0000 7f7a67e00640 1 -- 172.21.15.6:0/1254648832 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a680c6600 con 0x7f7a680d05e0 2024-03-04T09:33:35.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:35.373+0000 7f7a67e00640 1 -- 172.21.15.6:0/1254648832 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a6804f330 con 0x7f7a680d05e0 2024-03-04T09:33:35.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:35.375+0000 7f7a5ea00640 1 -- 172.21.15.6:0/1254648832 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7a60003b10 con 0x7f7a680d05e0 2024-03-04T09:33:35.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:35.376+0000 7f7a5ea00640 1 --2- 172.21.15.6:0/1254648832 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f7a4806b4d0 0x7f7a4806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:35.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:35.376+0000 7f7a5ea00640 1 -- 172.21.15.6:0/1254648832 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f7a60089e90 con 0x7f7a680d05e0 2024-03-04T09:33:35.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:35.376+0000 7f7a66a00640 1 --2- 172.21.15.6:0/1254648832 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f7a4806b4d0 0x7f7a4806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:35.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:35.377+0000 7f7a66a00640 1 --2- 172.21.15.6:0/1254648832 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f7a4806b4d0 0x7f7a4806d990 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f7a58036000 tx=0x7f7a58008030 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:33:35.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:35.381+0000 7f7a5ea00640 1 -- 172.21.15.6:0/1254648832 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7a60020020 con 0x7f7a680d05e0 2024-03-04T09:33:35.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:35.624+0000 7f7a67e00640 1 -- 172.21.15.6:0/1254648832 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7a680748b0 con 0x7f7a680d05e0 2024-03-04T09:33:35.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:35.626+0000 7f7a5ea00640 1 -- 172.21.15.6:0/1254648832 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3415 (secure 0 0 0) 0x7f7a60058d60 con 0x7f7a680d05e0 2024-03-04T09:33:35.623 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:33:35.623 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":99,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:33:24.637813+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":9}]} 2024-03-04T09:33:35.623 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 99 2024-03-04T09:33:35.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:35.628+0000 7f7a67e00640 1 -- 172.21.15.6:0/1254648832 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f7a4806b4d0 msgr2=0x7f7a4806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:35.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:35.628+0000 7f7a67e00640 1 --2- 172.21.15.6:0/1254648832 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f7a4806b4d0 0x7f7a4806d990 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f7a58036000 tx=0x7f7a58008030 comp rx=0 tx=0).stop 2024-03-04T09:33:35.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:35.629+0000 7f7a67e00640 1 -- 172.21.15.6:0/1254648832 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7a680d05e0 msgr2=0x7f7a68157f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:35.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:35.629+0000 7f7a67e00640 1 --2- 172.21.15.6:0/1254648832 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7a680d05e0 0x7f7a68157f50 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7f7a6000c740 tx=0x7f7a6000cc00 comp rx=0 tx=0).stop 2024-03-04T09:33:35.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:35.629+0000 7f7a67e00640 1 -- 172.21.15.6:0/1254648832 shutdown_connections 2024-03-04T09:33:35.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:35.629+0000 7f7a67e00640 1 --2- 172.21.15.6:0/1254648832 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7a680d05e0 0x7f7a68157f50 unknown :-1 s=CLOSED pgs=1562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:35.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:35.629+0000 7f7a67e00640 1 --2- 172.21.15.6:0/1254648832 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f7a4806b4d0 0x7f7a4806d990 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:35.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:35.629+0000 7f7a67e00640 1 --2- 172.21.15.6:0/1254648832 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7a680cc730 0x7f7a680c55f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:35.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:35.629+0000 7f7a67e00640 1 --2- 172.21.15.6:0/1254648832 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7a680b0f90 0x7f7a6815cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:35.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:35.629+0000 7f7a67e00640 1 -- 172.21.15.6:0/1254648832 >> 172.21.15.6:0/1254648832 conn(0x7f7a680d1ae0 msgr2=0x7f7a6806bee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:33:35.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:35.629+0000 7f7a67e00640 1 -- 172.21.15.6:0/1254648832 shutdown_connections 2024-03-04T09:33:35.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:35.629+0000 7f7a67e00640 1 -- 172.21.15.6:0/1254648832 wait complete. 2024-03-04T09:33:35.780 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-04T09:33:35.846 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:32:57.159681+0000', 'modified': '2024-03-04T09:33:24.637813+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8214, 'mds_1': 6796}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6796': {'gid': 6796, 'name': 'd', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.146:6837/2458070840', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 2458070840}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 2458070840}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8214': {'gid': 8214, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2446627517', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2446627517}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2446627517}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8214, 'qdb_cluster': [8214]} 2024-03-04T09:33:35.847 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6796:up:stopping 2024-03-04T09:33:36.847 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:33:37.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:37.082+0000 7fb597e00640 1 Processor -- start 2024-03-04T09:33:37.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:37.083+0000 7fb597e00640 1 -- start start 2024-03-04T09:33:37.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:37.086+0000 7fb597e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fb59806b6c0 0x7fb59815e370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:37.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:37.086+0000 7fb597e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb5981625a0 0x7fb59815e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:37.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:37.086+0000 7fb597e00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb59815edf0 con 0x7fb598147400 2024-03-04T09:33:37.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:37.086+0000 7fb597e00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb598162110 con 0x7fb59806b6c0 2024-03-04T09:33:37.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:37.086+0000 7fb597e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb598161ae0 con 0x7fb5981625a0 2024-03-04T09:33:37.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:37.087+0000 7fb58fe00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fb59806b6c0 0x7fb59815e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:37.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:37.087+0000 7fb597400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb5981625a0 0x7fb59815e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:37.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:37.087+0000 7fb58fe00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fb59806b6c0 0x7fb59815e370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.6:38606/0 (socket says 172.21.15.6:38606) 2024-03-04T09:33:37.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:37.087+0000 7fb58fe00640 1 -- 172.21.15.6:0/608531404 learned_addr learned my addr 172.21.15.6:0/608531404 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:33:37.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:37.087+0000 7fb58fe00640 1 -- 172.21.15.6:0/608531404 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb5981625a0 msgr2=0x7fb59815e8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:33:37.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:37.087+0000 7fb58fe00640 1 --2- 172.21.15.6:0/608531404 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb5981625a0 0x7fb59815e8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:37.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:37.087+0000 7fb58fe00640 1 -- 172.21.15.6:0/608531404 >> v1:172.21.15.146:6789/0 conn(0x7fb598147400 legacy=0x7fb59806afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:33:37.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:37.087+0000 7fb58fe00640 1 -- 172.21.15.6:0/608531404 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb59815f080 con 0x7fb59806b6c0 2024-03-04T09:33:37.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:37.089+0000 7fb58fe00640 1 --2- 172.21.15.6:0/608531404 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb59806b6c0 0x7fb59815e370 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fb580009870 tx=0x7fb58002b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=becafa0d84a4014c server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:33:37.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:37.089+0000 7fb596000640 1 -- 172.21.15.6:0/608531404 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb580035020 con 0x7fb59806b6c0 2024-03-04T09:33:37.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:37.089+0000 7fb596000640 1 -- 172.21.15.6:0/608531404 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb580031640 con 0x7fb59806b6c0 2024-03-04T09:33:37.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:37.089+0000 7fb596000640 1 -- 172.21.15.6:0/608531404 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb58002d040 con 0x7fb59806b6c0 2024-03-04T09:33:37.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:37.090+0000 7fb597e00640 1 -- 172.21.15.6:0/608531404 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb59806b6c0 msgr2=0x7fb59815e370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:37.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:37.090+0000 7fb597e00640 1 --2- 172.21.15.6:0/608531404 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb59806b6c0 0x7fb59815e370 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fb580009870 tx=0x7fb58002b8f0 comp rx=0 tx=0).stop 2024-03-04T09:33:37.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:37.090+0000 7fb597e00640 1 -- 172.21.15.6:0/608531404 shutdown_connections 2024-03-04T09:33:37.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:37.090+0000 7fb597e00640 1 --2- 172.21.15.6:0/608531404 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb5981625a0 0x7fb59815e8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:37.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:37.090+0000 7fb597e00640 1 --2- 172.21.15.6:0/608531404 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb59806b6c0 0x7fb59815e370 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:37.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:37.090+0000 7fb597e00640 1 -- 172.21.15.6:0/608531404 >> 172.21.15.6:0/608531404 conn(0x7fb59806c0c0 msgr2=0x7fb59806c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:33:37.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:37.090+0000 7fb597e00640 1 -- 172.21.15.6:0/608531404 shutdown_connections 2024-03-04T09:33:37.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:37.090+0000 7fb597e00640 1 -- 172.21.15.6:0/608531404 wait complete. 2024-03-04T09:33:37.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:37.092+0000 7fb597e00640 1 Processor -- start 2024-03-04T09:33:37.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:37.092+0000 7fb597e00640 1 -- start start 2024-03-04T09:33:37.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:37.093+0000 7fb597e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb59806b6c0 0x7fb5980b4970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:37.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:37.093+0000 7fb597e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb598147400 0x7fb5980b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:37.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:37.093+0000 7fb597e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb5981625a0 0x7fb598141cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:37.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:37.093+0000 7fb597e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb598147960 con 0x7fb598147400 2024-03-04T09:33:37.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:37.093+0000 7fb597e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5981477e0 con 0x7fb59806b6c0 2024-03-04T09:33:37.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:37.093+0000 7fb597e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb59806e3b0 con 0x7fb5981625a0 2024-03-04T09:33:37.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:37.093+0000 7fb58fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb598147400 0x7fb5980b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:37.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:37.093+0000 7fb58fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb598147400 0x7fb5980b4eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:48744/0 (socket says 172.21.15.6:48744) 2024-03-04T09:33:37.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:37.093+0000 7fb58fe00640 1 -- 172.21.15.6:0/4127665851 learned_addr learned my addr 172.21.15.6:0/4127665851 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:33:37.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:37.094+0000 7fb597400640 1 --2- 172.21.15.6:0/4127665851 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb5981625a0 0x7fb598141cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:37.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:37.094+0000 7fb596a00640 1 --2- 172.21.15.6:0/4127665851 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb59806b6c0 0x7fb5980b4970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:37.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:37.094+0000 7fb58fe00640 1 -- 172.21.15.6:0/4127665851 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb5981625a0 msgr2=0x7fb598141cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:37.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:37.094+0000 7fb58fe00640 1 --2- 172.21.15.6:0/4127665851 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb5981625a0 0x7fb598141cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:37.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:37.094+0000 7fb58fe00640 1 -- 172.21.15.6:0/4127665851 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb59806b6c0 msgr2=0x7fb5980b4970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:37.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:37.094+0000 7fb58fe00640 1 --2- 172.21.15.6:0/4127665851 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb59806b6c0 0x7fb5980b4970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:37.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:37.094+0000 7fb58fe00640 1 -- 172.21.15.6:0/4127665851 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb598130390 con 0x7fb598147400 2024-03-04T09:33:37.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:37.094+0000 7fb58fe00640 1 --2- 172.21.15.6:0/4127665851 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb598147400 0x7fb5980b4eb0 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7fb58002b930 tx=0x7fb5800046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:33:37.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:37.095+0000 7fb58ea00640 1 -- 172.21.15.6:0/4127665851 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb580040070 con 0x7fb598147400 2024-03-04T09:33:37.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:37.095+0000 7fb58ea00640 1 -- 172.21.15.6:0/4127665851 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5800399f0 con 0x7fb598147400 2024-03-04T09:33:37.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:37.095+0000 7fb597e00640 1 -- 172.21.15.6:0/4127665851 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb598130670 con 0x7fb598147400 2024-03-04T09:33:37.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:37.095+0000 7fb58ea00640 1 -- 172.21.15.6:0/4127665851 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb580035040 con 0x7fb598147400 2024-03-04T09:33:37.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:37.095+0000 7fb597e00640 1 -- 172.21.15.6:0/4127665851 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb598130c30 con 0x7fb598147400 2024-03-04T09:33:37.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:37.096+0000 7fb597e00640 1 -- 172.21.15.6:0/4127665851 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb59815dfd0 con 0x7fb598147400 2024-03-04T09:33:37.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:37.097+0000 7fb58ea00640 1 -- 172.21.15.6:0/4127665851 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb58004b060 con 0x7fb598147400 2024-03-04T09:33:37.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:37.098+0000 7fb58ea00640 1 --2- 172.21.15.6:0/4127665851 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb57406b4d0 0x7fb57406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:37.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:37.098+0000 7fb58ea00640 1 -- 172.21.15.6:0/4127665851 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fb5800abb80 con 0x7fb598147400 2024-03-04T09:33:37.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:37.099+0000 7fb596a00640 1 --2- 172.21.15.6:0/4127665851 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb57406b4d0 0x7fb57406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:37.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:37.100+0000 7fb596a00640 1 --2- 172.21.15.6:0/4127665851 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb57406b4d0 0x7fb57406d990 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7fb588004ea0 tx=0x7fb588009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:33:37.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:37.103+0000 7fb58ea00640 1 -- 172.21.15.6:0/4127665851 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb59815dfd0 con 0x7fb598147400 2024-03-04T09:33:37.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:37.341+0000 7fb597e00640 1 -- 172.21.15.6:0/4127665851 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb59804f330 con 0x7fb598147400 2024-03-04T09:33:37.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:37.342+0000 7fb58ea00640 1 -- 172.21.15.6:0/4127665851 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3415 (secure 0 0 0) 0x7fb58007a9b0 con 0x7fb598147400 2024-03-04T09:33:37.339 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:33:37.339 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":99,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:33:24.637813+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":9}]} 2024-03-04T09:33:37.339 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 99 2024-03-04T09:33:37.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:37.344+0000 7fb597e00640 1 -- 172.21.15.6:0/4127665851 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb57406b4d0 msgr2=0x7fb57406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:37.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:37.344+0000 7fb597e00640 1 --2- 172.21.15.6:0/4127665851 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb57406b4d0 0x7fb57406d990 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7fb588004ea0 tx=0x7fb588009040 comp rx=0 tx=0).stop 2024-03-04T09:33:37.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:37.345+0000 7fb597e00640 1 -- 172.21.15.6:0/4127665851 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb598147400 msgr2=0x7fb5980b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:37.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:37.345+0000 7fb597e00640 1 --2- 172.21.15.6:0/4127665851 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb598147400 0x7fb5980b4eb0 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7fb58002b930 tx=0x7fb5800046c0 comp rx=0 tx=0).stop 2024-03-04T09:33:37.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:37.345+0000 7fb597e00640 1 -- 172.21.15.6:0/4127665851 shutdown_connections 2024-03-04T09:33:37.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:37.345+0000 7fb597e00640 1 --2- 172.21.15.6:0/4127665851 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb57406b4d0 0x7fb57406d990 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:37.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:37.345+0000 7fb597e00640 1 --2- 172.21.15.6:0/4127665851 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb5981625a0 0x7fb598141cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:37.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:37.345+0000 7fb597e00640 1 --2- 172.21.15.6:0/4127665851 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb598147400 0x7fb5980b4eb0 unknown :-1 s=CLOSED pgs=1563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:37.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:37.345+0000 7fb597e00640 1 --2- 172.21.15.6:0/4127665851 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb59806b6c0 0x7fb5980b4970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:37.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:37.345+0000 7fb597e00640 1 -- 172.21.15.6:0/4127665851 >> 172.21.15.6:0/4127665851 conn(0x7fb59806c0c0 msgr2=0x7fb59806e810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:33:37.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:37.345+0000 7fb597e00640 1 -- 172.21.15.6:0/4127665851 shutdown_connections 2024-03-04T09:33:37.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:37.345+0000 7fb597e00640 1 -- 172.21.15.6:0/4127665851 wait complete. 2024-03-04T09:33:37.561 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:32:57.159681+0000', 'modified': '2024-03-04T09:33:24.637813+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8214, 'mds_1': 6796}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6796': {'gid': 6796, 'name': 'd', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.146:6837/2458070840', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 2458070840}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 2458070840}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8214': {'gid': 8214, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2446627517', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2446627517}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2446627517}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8214, 'qdb_cluster': [8214]} 2024-03-04T09:33:37.561 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6796:up:stopping 2024-03-04T09:33:37.781 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-04T09:33:38.562 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:33:38.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:38.793+0000 7f9048e00640 1 Processor -- start 2024-03-04T09:33:38.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:38.794+0000 7f9048e00640 1 -- start start 2024-03-04T09:33:38.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:38.796+0000 7f9048e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f904415e0b0 0x7f90440cf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:38.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:38.796+0000 7f9048e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f90440d0e50 0x7f90440b44d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:38.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:38.796+0000 7f9048e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90440b3710 con 0x7f90440d4a10 2024-03-04T09:33:38.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:38.796+0000 7f9048e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90440b09e0 con 0x7f90440d0e50 2024-03-04T09:33:38.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:38.796+0000 7f9048e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90440b39f0 con 0x7f904415e0b0 2024-03-04T09:33:38.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:38.797+0000 7f9043e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f90440d0e50 0x7f90440b44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:38.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:38.797+0000 7f9043e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f90440d0e50 0x7f90440b44d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:48754/0 (socket says 172.21.15.6:48754) 2024-03-04T09:33:38.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:38.797+0000 7f9043e00640 1 -- 172.21.15.6:0/4074562401 learned_addr learned my addr 172.21.15.6:0/4074562401 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:33:38.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:38.797+0000 7f9043400640 1 --2- 172.21.15.6:0/4074562401 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f904415e0b0 0x7f90440cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:38.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:38.797+0000 7f9043e00640 1 -- 172.21.15.6:0/4074562401 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f904415e0b0 msgr2=0x7f90440cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:33:38.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:38.797+0000 7f9043e00640 1 --2- 172.21.15.6:0/4074562401 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f904415e0b0 0x7f90440cf310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:38.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:38.797+0000 7f9043e00640 1 -- 172.21.15.6:0/4074562401 >> v1:172.21.15.6:6789/0 conn(0x7f90440d4a10 legacy=0x7f90440d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:38.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:38.797+0000 7f9043e00640 1 -- 172.21.15.6:0/4074562401 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90440b3cd0 con 0x7f90440d0e50 2024-03-04T09:33:38.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:38.798+0000 7f9043e00640 1 --2- 172.21.15.6:0/4074562401 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f90440d0e50 0x7f90440b44d0 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 crypto rx=0x7f903c009e50 tx=0x7f903c02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2cc956d3f9db348 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:33:38.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:38.799+0000 7f9042000640 1 -- 172.21.15.6:0/4074562401 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f903c02cb00 con 0x7f90440d0e50 2024-03-04T09:33:38.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:38.799+0000 7f9042000640 1 -- 172.21.15.6:0/4074562401 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f903c02cca0 con 0x7f90440d0e50 2024-03-04T09:33:38.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:38.799+0000 7f9042000640 1 -- 172.21.15.6:0/4074562401 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f903c034610 con 0x7f90440d0e50 2024-03-04T09:33:38.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:38.799+0000 7f9048e00640 1 -- 172.21.15.6:0/4074562401 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f90440d0e50 msgr2=0x7f90440b44d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:38.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:38.799+0000 7f9048e00640 1 --2- 172.21.15.6:0/4074562401 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f90440d0e50 0x7f90440b44d0 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 crypto rx=0x7f903c009e50 tx=0x7f903c02bf60 comp rx=0 tx=0).stop 2024-03-04T09:33:38.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:38.799+0000 7f9048e00640 1 -- 172.21.15.6:0/4074562401 shutdown_connections 2024-03-04T09:33:38.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:38.799+0000 7f9048e00640 1 --2- 172.21.15.6:0/4074562401 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f90440d0e50 0x7f90440b44d0 unknown :-1 s=CLOSED pgs=1565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:38.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:38.799+0000 7f9048e00640 1 --2- 172.21.15.6:0/4074562401 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f904415e0b0 0x7f90440cf310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:38.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:38.799+0000 7f9048e00640 1 -- 172.21.15.6:0/4074562401 >> 172.21.15.6:0/4074562401 conn(0x7f90440d3d30 msgr2=0x7f90440d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:33:38.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:38.800+0000 7f9048e00640 1 -- 172.21.15.6:0/4074562401 shutdown_connections 2024-03-04T09:33:38.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:38.800+0000 7f9048e00640 1 -- 172.21.15.6:0/4074562401 wait complete. 2024-03-04T09:33:38.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:38.801+0000 7f9048e00640 1 Processor -- start 2024-03-04T09:33:38.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:38.801+0000 7f9048e00640 1 -- start start 2024-03-04T09:33:38.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:38.802+0000 7f9048e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f90440d0e50 0x7f90440c8630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:38.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:38.802+0000 7f9048e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f90440d4a10 0x7f90440b5220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:38.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:38.802+0000 7f9048e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f904415e0b0 0x7f904414fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:38.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:38.802+0000 7f9048e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90440cfc60 con 0x7f90440d0e50 2024-03-04T09:33:38.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:38.802+0000 7f9048e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f904415e490 con 0x7f90440d4a10 2024-03-04T09:33:38.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:38.802+0000 7f9048e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90440cf850 con 0x7f904415e0b0 2024-03-04T09:33:38.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:38.803+0000 7f9043400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f90440d0e50 0x7f90440c8630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:38.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:38.803+0000 7f9043400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f90440d0e50 0x7f90440c8630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:48756/0 (socket says 172.21.15.6:48756) 2024-03-04T09:33:38.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:38.803+0000 7f9043400640 1 -- 172.21.15.6:0/3697296878 learned_addr learned my addr 172.21.15.6:0/3697296878 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:33:38.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:38.803+0000 7f9043e00640 1 --2- 172.21.15.6:0/3697296878 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f904415e0b0 0x7f904414fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:38.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:38.803+0000 7f9042a00640 1 --2- 172.21.15.6:0/3697296878 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f90440d4a10 0x7f90440b5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:38.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:38.803+0000 7f9043400640 1 -- 172.21.15.6:0/3697296878 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f904415e0b0 msgr2=0x7f904414fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:38.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:38.803+0000 7f9043400640 1 --2- 172.21.15.6:0/3697296878 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f904415e0b0 0x7f904414fd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:38.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:38.803+0000 7f9043400640 1 -- 172.21.15.6:0/3697296878 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f90440d4a10 msgr2=0x7f90440b5220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:38.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:38.803+0000 7f9043400640 1 --2- 172.21.15.6:0/3697296878 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f90440d4a10 0x7f90440b5220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:38.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:38.803+0000 7f9043400640 1 -- 172.21.15.6:0/3697296878 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90440b6a80 con 0x7f90440d0e50 2024-03-04T09:33:38.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:38.803+0000 7f9043e00640 1 --2- 172.21.15.6:0/3697296878 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f904415e0b0 0x7f904414fd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:33:38.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:38.803+0000 7f9042a00640 1 --2- 172.21.15.6:0/3697296878 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f90440d4a10 0x7f90440b5220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:33:38.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:38.804+0000 7f9043400640 1 --2- 172.21.15.6:0/3697296878 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f90440d0e50 0x7f90440c8630 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7f903400d620 tx=0x7f903400dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:33:38.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:38.804+0000 7f903b400640 1 -- 172.21.15.6:0/3697296878 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f903400b620 con 0x7f90440d0e50 2024-03-04T09:33:38.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:38.804+0000 7f903b400640 1 -- 172.21.15.6:0/3697296878 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9034019070 con 0x7f90440d0e50 2024-03-04T09:33:38.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:38.804+0000 7f903b400640 1 -- 172.21.15.6:0/3697296878 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f903400ba00 con 0x7f90440d0e50 2024-03-04T09:33:38.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:38.804+0000 7f9048e00640 1 -- 172.21.15.6:0/3697296878 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90440b6d60 con 0x7f90440d0e50 2024-03-04T09:33:38.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:38.804+0000 7f9048e00640 1 -- 172.21.15.6:0/3697296878 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90440b7320 con 0x7f90440d0e50 2024-03-04T09:33:38.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:38.805+0000 7f9048e00640 1 -- 172.21.15.6:0/3697296878 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f904404f330 con 0x7f90440d0e50 2024-03-04T09:33:38.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:38.806+0000 7f903b400640 1 -- 172.21.15.6:0/3697296878 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f903400bba0 con 0x7f90440d0e50 2024-03-04T09:33:38.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:38.807+0000 7f903b400640 1 --2- 172.21.15.6:0/3697296878 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f902406b4d0 0x7f902406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:38.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:38.807+0000 7f903b400640 1 -- 172.21.15.6:0/3697296878 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f903408a690 con 0x7f90440d0e50 2024-03-04T09:33:38.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:38.808+0000 7f9042a00640 1 --2- 172.21.15.6:0/3697296878 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f902406b4d0 0x7f902406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:38.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:38.809+0000 7f9042a00640 1 --2- 172.21.15.6:0/3697296878 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f902406b4d0 0x7f902406d990 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f90440b9be0 tx=0x7f9030009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:33:38.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:38.811+0000 7f903b400640 1 -- 172.21.15.6:0/3697296878 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f90340594c0 con 0x7f90440d0e50 2024-03-04T09:33:39.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:39.055+0000 7f9048e00640 1 -- 172.21.15.6:0/3697296878 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f90440d3400 con 0x7f90440d0e50 2024-03-04T09:33:39.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:39.056+0000 7f903b400640 1 -- 172.21.15.6:0/3697296878 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3415 (secure 0 0 0) 0x7f9034058dc0 con 0x7f90440d0e50 2024-03-04T09:33:39.053 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:33:39.053 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":99,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:33:24.637813+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":9}]} 2024-03-04T09:33:39.053 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 99 2024-03-04T09:33:39.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:39.058+0000 7f9048e00640 1 -- 172.21.15.6:0/3697296878 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f902406b4d0 msgr2=0x7f902406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:39.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:39.058+0000 7f9048e00640 1 --2- 172.21.15.6:0/3697296878 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f902406b4d0 0x7f902406d990 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f90440b9be0 tx=0x7f9030009040 comp rx=0 tx=0).stop 2024-03-04T09:33:39.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:39.058+0000 7f9048e00640 1 -- 172.21.15.6:0/3697296878 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f90440d0e50 msgr2=0x7f90440c8630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:39.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:39.058+0000 7f9048e00640 1 --2- 172.21.15.6:0/3697296878 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f90440d0e50 0x7f90440c8630 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7f903400d620 tx=0x7f903400dae0 comp rx=0 tx=0).stop 2024-03-04T09:33:39.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:39.059+0000 7f9048e00640 1 -- 172.21.15.6:0/3697296878 shutdown_connections 2024-03-04T09:33:39.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:39.059+0000 7f9048e00640 1 --2- 172.21.15.6:0/3697296878 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f902406b4d0 0x7f902406d990 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:39.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:39.059+0000 7f9048e00640 1 --2- 172.21.15.6:0/3697296878 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f904415e0b0 0x7f904414fd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:39.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:39.059+0000 7f9048e00640 1 --2- 172.21.15.6:0/3697296878 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f90440d4a10 0x7f90440b5220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:39.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:39.059+0000 7f9048e00640 1 --2- 172.21.15.6:0/3697296878 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f90440d0e50 0x7f90440c8630 unknown :-1 s=CLOSED pgs=1566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:39.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:39.059+0000 7f9048e00640 1 -- 172.21.15.6:0/3697296878 >> 172.21.15.6:0/3697296878 conn(0x7f90440d3d30 msgr2=0x7f90440cfe70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:33:39.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:39.059+0000 7f9048e00640 1 -- 172.21.15.6:0/3697296878 shutdown_connections 2024-03-04T09:33:39.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:39.059+0000 7f9048e00640 1 -- 172.21.15.6:0/3697296878 wait complete. 2024-03-04T09:33:39.275 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:32:57.159681+0000', 'modified': '2024-03-04T09:33:24.637813+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8214, 'mds_1': 6796}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6796': {'gid': 6796, 'name': 'd', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.146:6837/2458070840', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 2458070840}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 2458070840}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8214': {'gid': 8214, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2446627517', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2446627517}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2446627517}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8214, 'qdb_cluster': [8214]} 2024-03-04T09:33:39.275 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6796:up:stopping 2024-03-04T09:33:39.783 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-04T09:33:40.277 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:33:40.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:40.509+0000 7fa56a600640 1 Processor -- start 2024-03-04T09:33:40.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:40.509+0000 7fa56a600640 1 -- start start 2024-03-04T09:33:40.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:40.511+0000 7fa56a600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa564158460 0x7fa5640d09d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:40.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:40.511+0000 7fa56a600640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fa564158840 0x7fa564161c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:40.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:40.511+0000 7fa56a600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa564162150 0x7fa5640b1310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:40.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:40.511+0000 7fa56a600640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5640cff30 con 0x7fa564158460 2024-03-04T09:33:40.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:40.511+0000 7fa56a600640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5641626d0 con 0x7fa564158840 2024-03-04T09:33:40.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:40.511+0000 7fa56a600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5641629b0 con 0x7fa564162150 2024-03-04T09:33:40.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:40.511+0000 7fa569200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa564158460 0x7fa5640d09d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:40.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:40.511+0000 7fa569200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa564158460 0x7fa5640d09d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:59164/0 (socket says 172.21.15.6:59164) 2024-03-04T09:33:40.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:40.511+0000 7fa569200640 1 -- 172.21.15.6:0/1877387696 learned_addr learned my addr 172.21.15.6:0/1877387696 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:33:40.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:40.511+0000 7fa563e00640 1 --2- 172.21.15.6:0/1877387696 >> v2:172.21.15.146:3300/0 conn(0x7fa564158840 0x7fa564161c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:40.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:40.511+0000 7fa569c00640 1 --2- 172.21.15.6:0/1877387696 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa564162150 0x7fa5640b1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:40.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:40.512+0000 7fa569200640 1 -- 172.21.15.6:0/1877387696 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa564162150 msgr2=0x7fa5640b1310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:33:40.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:40.512+0000 7fa569200640 1 --2- 172.21.15.6:0/1877387696 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa564162150 0x7fa5640b1310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:40.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:40.512+0000 7fa569200640 1 -- 172.21.15.6:0/1877387696 >> v2:172.21.15.146:3300/0 conn(0x7fa564158840 msgr2=0x7fa564161c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:33:40.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:40.512+0000 7fa569200640 1 --2- 172.21.15.6:0/1877387696 >> v2:172.21.15.146:3300/0 conn(0x7fa564158840 0x7fa564161c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:40.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:40.512+0000 7fa569200640 1 -- 172.21.15.6:0/1877387696 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5640d01d0 con 0x7fa564158460 2024-03-04T09:33:40.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:40.513+0000 7fa569200640 1 --2- 172.21.15.6:0/1877387696 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa564158460 0x7fa5640d09d0 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7fa564070f60 tx=0x7fa55802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4f5ef4e8890152f server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:33:40.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:40.513+0000 7fa563400640 1 -- 172.21.15.6:0/1877387696 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa558035020 con 0x7fa564158460 2024-03-04T09:33:40.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:40.513+0000 7fa563400640 1 -- 172.21.15.6:0/1877387696 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa558031870 con 0x7fa564158460 2024-03-04T09:33:40.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:40.513+0000 7fa563400640 1 -- 172.21.15.6:0/1877387696 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa558031c40 con 0x7fa564158460 2024-03-04T09:33:40.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:40.514+0000 7fa56a600640 1 -- 172.21.15.6:0/1877387696 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa564158460 msgr2=0x7fa5640d09d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:40.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:40.514+0000 7fa56a600640 1 --2- 172.21.15.6:0/1877387696 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa564158460 0x7fa5640d09d0 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7fa564070f60 tx=0x7fa55802b910 comp rx=0 tx=0).stop 2024-03-04T09:33:40.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:40.514+0000 7fa56a600640 1 -- 172.21.15.6:0/1877387696 shutdown_connections 2024-03-04T09:33:40.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:40.514+0000 7fa56a600640 1 --2- 172.21.15.6:0/1877387696 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa564162150 0x7fa5640b1310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:40.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:40.514+0000 7fa56a600640 1 --2- 172.21.15.6:0/1877387696 >> v2:172.21.15.146:3300/0 conn(0x7fa564158840 0x7fa564161c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:40.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:40.514+0000 7fa56a600640 1 --2- 172.21.15.6:0/1877387696 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa564158460 0x7fa5640d09d0 unknown :-1 s=CLOSED pgs=1567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:40.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:40.514+0000 7fa56a600640 1 -- 172.21.15.6:0/1877387696 >> 172.21.15.6:0/1877387696 conn(0x7fa5640b5100 msgr2=0x7fa5640b57c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:33:40.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:40.514+0000 7fa56a600640 1 -- 172.21.15.6:0/1877387696 shutdown_connections 2024-03-04T09:33:40.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:40.514+0000 7fa56a600640 1 -- 172.21.15.6:0/1877387696 wait complete. 2024-03-04T09:33:40.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:40.516+0000 7fa56a600640 1 Processor -- start 2024-03-04T09:33:40.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:40.516+0000 7fa56a600640 1 -- start start 2024-03-04T09:33:40.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:40.518+0000 7fa56a600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa564158460 0x7fa564155350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:40.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:40.518+0000 7fa56a600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa564158840 0x7fa564099530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:40.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:40.518+0000 7fa56a600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa564162150 0x7fa564099a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:40.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:40.518+0000 7fa56a600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa56415e8c0 con 0x7fa564158840 2024-03-04T09:33:40.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:40.518+0000 7fa56a600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5640d0f10 con 0x7fa564158460 2024-03-04T09:33:40.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:40.518+0000 7fa56a600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5640b1850 con 0x7fa564162150 2024-03-04T09:33:40.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:40.518+0000 7fa563e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa564158840 0x7fa564099530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:40.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:40.518+0000 7fa563e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa564158840 0x7fa564099530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:59174/0 (socket says 172.21.15.6:59174) 2024-03-04T09:33:40.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:40.518+0000 7fa563e00640 1 -- 172.21.15.6:0/3027652269 learned_addr learned my addr 172.21.15.6:0/3027652269 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:33:40.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:40.518+0000 7fa569200640 1 --2- 172.21.15.6:0/3027652269 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa564158460 0x7fa564155350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:40.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:40.518+0000 7fa569c00640 1 --2- 172.21.15.6:0/3027652269 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa564162150 0x7fa564099a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:40.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:40.518+0000 7fa563e00640 1 -- 172.21.15.6:0/3027652269 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa564162150 msgr2=0x7fa564099a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:40.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:40.518+0000 7fa563e00640 1 --2- 172.21.15.6:0/3027652269 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa564162150 0x7fa564099a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:40.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:40.518+0000 7fa563e00640 1 -- 172.21.15.6:0/3027652269 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa564158460 msgr2=0x7fa564155350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:40.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:40.518+0000 7fa563e00640 1 --2- 172.21.15.6:0/3027652269 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa564158460 0x7fa564155350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:40.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:40.518+0000 7fa563e00640 1 -- 172.21.15.6:0/3027652269 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa564142620 con 0x7fa564158840 2024-03-04T09:33:40.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:40.519+0000 7fa563e00640 1 --2- 172.21.15.6:0/3027652269 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa564158840 0x7fa564099530 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x7fa55000d830 tx=0x7fa55000dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:33:40.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:40.519+0000 7fa560c00640 1 -- 172.21.15.6:0/3027652269 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa550018020 con 0x7fa564158840 2024-03-04T09:33:40.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:40.519+0000 7fa560c00640 1 -- 172.21.15.6:0/3027652269 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5500096a0 con 0x7fa564158840 2024-03-04T09:33:40.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:40.519+0000 7fa560c00640 1 -- 172.21.15.6:0/3027652269 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa550009a70 con 0x7fa564158840 2024-03-04T09:33:40.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:40.519+0000 7fa56a600640 1 -- 172.21.15.6:0/3027652269 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa564142900 con 0x7fa564158840 2024-03-04T09:33:40.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:40.519+0000 7fa56a600640 1 -- 172.21.15.6:0/3027652269 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5640ad2b0 con 0x7fa564158840 2024-03-04T09:33:40.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:40.520+0000 7fa56a600640 1 -- 172.21.15.6:0/3027652269 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa56404f330 con 0x7fa564158840 2024-03-04T09:33:40.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:40.521+0000 7fa560c00640 1 -- 172.21.15.6:0/3027652269 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa55001b020 con 0x7fa564158840 2024-03-04T09:33:40.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:40.522+0000 7fa560c00640 1 --2- 172.21.15.6:0/3027652269 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa54406b4d0 0x7fa54406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:40.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:40.522+0000 7fa560c00640 1 -- 172.21.15.6:0/3027652269 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fa55008a690 con 0x7fa564158840 2024-03-04T09:33:40.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:40.523+0000 7fa569200640 1 --2- 172.21.15.6:0/3027652269 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa54406b4d0 0x7fa54406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:40.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:40.524+0000 7fa569200640 1 --2- 172.21.15.6:0/3027652269 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa54406b4d0 0x7fa54406d990 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fa558036000 tx=0x7fa55803c040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:33:40.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:40.531+0000 7fa560c00640 1 -- 172.21.15.6:0/3027652269 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa550022030 con 0x7fa564158840 2024-03-04T09:33:40.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:40.791+0000 7fa56a600640 1 -- 172.21.15.6:0/3027652269 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa564071570 con 0x7fa564158840 2024-03-04T09:33:40.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:40.792+0000 7fa560c00640 1 -- 172.21.15.6:0/3027652269 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3415 (secure 0 0 0) 0x7fa5500102e0 con 0x7fa564158840 2024-03-04T09:33:40.789 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:33:40.789 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":99,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:33:24.637813+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":9}]} 2024-03-04T09:33:40.789 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 99 2024-03-04T09:33:40.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:40.794+0000 7fa56a600640 1 -- 172.21.15.6:0/3027652269 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa54406b4d0 msgr2=0x7fa54406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:40.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:40.794+0000 7fa56a600640 1 --2- 172.21.15.6:0/3027652269 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa54406b4d0 0x7fa54406d990 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fa558036000 tx=0x7fa55803c040 comp rx=0 tx=0).stop 2024-03-04T09:33:40.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:40.794+0000 7fa56a600640 1 -- 172.21.15.6:0/3027652269 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa564158840 msgr2=0x7fa564099530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:40.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:40.794+0000 7fa56a600640 1 --2- 172.21.15.6:0/3027652269 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa564158840 0x7fa564099530 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x7fa55000d830 tx=0x7fa55000dcf0 comp rx=0 tx=0).stop 2024-03-04T09:33:40.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:40.794+0000 7fa56a600640 1 -- 172.21.15.6:0/3027652269 shutdown_connections 2024-03-04T09:33:40.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:40.794+0000 7fa56a600640 1 --2- 172.21.15.6:0/3027652269 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa54406b4d0 0x7fa54406d990 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:40.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:40.794+0000 7fa56a600640 1 --2- 172.21.15.6:0/3027652269 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa564162150 0x7fa564099a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:40.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:40.794+0000 7fa56a600640 1 --2- 172.21.15.6:0/3027652269 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa564158840 0x7fa564099530 unknown :-1 s=CLOSED pgs=1568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:40.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:40.794+0000 7fa56a600640 1 --2- 172.21.15.6:0/3027652269 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa564158460 0x7fa564155350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:40.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:40.794+0000 7fa56a600640 1 -- 172.21.15.6:0/3027652269 >> 172.21.15.6:0/3027652269 conn(0x7fa5640b5100 msgr2=0x7fa5641564d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:33:40.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:40.794+0000 7fa56a600640 1 -- 172.21.15.6:0/3027652269 shutdown_connections 2024-03-04T09:33:40.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:40.794+0000 7fa56a600640 1 -- 172.21.15.6:0/3027652269 wait complete. 2024-03-04T09:33:40.842 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:33:40.847 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:33:41.010 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:32:57.159681+0000', 'modified': '2024-03-04T09:33:24.637813+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8214, 'mds_1': 6796}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6796': {'gid': 6796, 'name': 'd', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.146:6837/2458070840', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 2458070840}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 2458070840}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8214': {'gid': 8214, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2446627517', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2446627517}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2446627517}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8214, 'qdb_cluster': [8214]} 2024-03-04T09:33:41.010 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6796:up:stopping 2024-03-04T09:33:41.785 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-04T09:33:42.011 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:33:42.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:42.247+0000 7f8a6fc00640 1 Processor -- start 2024-03-04T09:33:42.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:42.247+0000 7f8a6fc00640 1 -- start start 2024-03-04T09:33:42.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:42.249+0000 7f8a6fc00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f8a68161670 0x7f8a68070c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:42.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:42.249+0000 7f8a6fc00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8a6806e240 0x7f8a680711b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:42.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:42.249+0000 7f8a6fc00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a680ce9a0 con 0x7f8a6806cfe0 2024-03-04T09:33:42.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:42.249+0000 7f8a6fc00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a680730d0 con 0x7f8a68161670 2024-03-04T09:33:42.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:42.249+0000 7f8a6fc00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a680cec80 con 0x7f8a6806e240 2024-03-04T09:33:42.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:42.249+0000 7f8a6d200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f8a68161670 0x7f8a68070c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:42.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:42.249+0000 7f8a6d200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f8a68161670 0x7f8a68070c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:59190/0 (socket says 172.21.15.6:59190) 2024-03-04T09:33:42.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:42.249+0000 7f8a6d200640 1 -- 172.21.15.6:0/2390463226 learned_addr learned my addr 172.21.15.6:0/2390463226 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:33:42.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:42.250+0000 7f8a67e00640 1 --2- 172.21.15.6:0/2390463226 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8a6806e240 0x7f8a680711b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:42.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:42.250+0000 7f8a6d200640 1 -- 172.21.15.6:0/2390463226 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8a6806e240 msgr2=0x7f8a680711b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:33:42.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:42.250+0000 7f8a6d200640 1 --2- 172.21.15.6:0/2390463226 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8a6806e240 0x7f8a680711b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:42.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:42.250+0000 7f8a6d200640 1 -- 172.21.15.6:0/2390463226 >> v1:172.21.15.146:6789/0 conn(0x7f8a6806cfe0 legacy=0x7f8a6806d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:33:42.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:42.250+0000 7f8a6d200640 1 -- 172.21.15.6:0/2390463226 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a680cef60 con 0x7f8a68161670 2024-03-04T09:33:42.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:42.251+0000 7f8a6d200640 1 --2- 172.21.15.6:0/2390463226 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8a68161670 0x7f8a68070c70 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7f8a58009e50 tx=0x7f8a5802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a4a305c50320f54 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:33:42.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:42.251+0000 7f8a67400640 1 -- 172.21.15.6:0/2390463226 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a5802ca70 con 0x7f8a68161670 2024-03-04T09:33:42.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:42.251+0000 7f8a67400640 1 -- 172.21.15.6:0/2390463226 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a5802cc10 con 0x7f8a68161670 2024-03-04T09:33:42.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:42.251+0000 7f8a67400640 1 -- 172.21.15.6:0/2390463226 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a580343f0 con 0x7f8a68161670 2024-03-04T09:33:42.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:42.252+0000 7f8a6fc00640 1 -- 172.21.15.6:0/2390463226 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8a68161670 msgr2=0x7f8a68070c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:42.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:42.252+0000 7f8a6fc00640 1 --2- 172.21.15.6:0/2390463226 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8a68161670 0x7f8a68070c70 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7f8a58009e50 tx=0x7f8a5802bed0 comp rx=0 tx=0).stop 2024-03-04T09:33:42.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:42.252+0000 7f8a6fc00640 1 -- 172.21.15.6:0/2390463226 shutdown_connections 2024-03-04T09:33:42.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:42.252+0000 7f8a6fc00640 1 --2- 172.21.15.6:0/2390463226 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8a6806e240 0x7f8a680711b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:42.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:42.252+0000 7f8a6fc00640 1 --2- 172.21.15.6:0/2390463226 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8a68161670 0x7f8a68070c70 unknown :-1 s=CLOSED pgs=1569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:42.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:42.252+0000 7f8a6fc00640 1 -- 172.21.15.6:0/2390463226 >> 172.21.15.6:0/2390463226 conn(0x7f8a6806afb0 msgr2=0x7f8a6806b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:33:42.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:42.252+0000 7f8a6fc00640 1 -- 172.21.15.6:0/2390463226 shutdown_connections 2024-03-04T09:33:42.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:42.252+0000 7f8a6fc00640 1 -- 172.21.15.6:0/2390463226 wait complete. 2024-03-04T09:33:42.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:42.254+0000 7f8a6fc00640 1 Processor -- start 2024-03-04T09:33:42.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:42.254+0000 7f8a6fc00640 1 -- start start 2024-03-04T09:33:42.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:42.255+0000 7f8a6fc00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8a6806cfe0 0x7f8a68148000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:42.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:42.255+0000 7f8a6fc00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8a6806e240 0x7f8a68159e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:42.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:42.255+0000 7f8a6fc00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8a68161670 0x7f8a6814a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:42.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:42.255+0000 7f8a6fc00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a680ce600 con 0x7f8a68161670 2024-03-04T09:33:42.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:42.255+0000 7f8a6fc00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a680cf520 con 0x7f8a6806e240 2024-03-04T09:33:42.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:42.255+0000 7f8a6fc00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a680ce780 con 0x7f8a6806cfe0 2024-03-04T09:33:42.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:42.255+0000 7f8a6dc00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8a68161670 0x7f8a6814a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:42.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:42.255+0000 7f8a6dc00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8a68161670 0x7f8a6814a870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:59196/0 (socket says 172.21.15.6:59196) 2024-03-04T09:33:42.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:42.255+0000 7f8a6dc00640 1 -- 172.21.15.6:0/1070808411 learned_addr learned my addr 172.21.15.6:0/1070808411 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:33:42.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:42.256+0000 7f8a6d200640 1 --2- 172.21.15.6:0/1070808411 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8a6806cfe0 0x7f8a68148000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:42.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:42.256+0000 7f8a67e00640 1 --2- 172.21.15.6:0/1070808411 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8a6806e240 0x7f8a68159e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:42.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:42.256+0000 7f8a6dc00640 1 -- 172.21.15.6:0/1070808411 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8a6806cfe0 msgr2=0x7f8a68148000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:42.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:42.256+0000 7f8a6dc00640 1 --2- 172.21.15.6:0/1070808411 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8a6806cfe0 0x7f8a68148000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:42.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:42.256+0000 7f8a6dc00640 1 -- 172.21.15.6:0/1070808411 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8a6806e240 msgr2=0x7f8a68159e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:42.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:42.256+0000 7f8a6dc00640 1 --2- 172.21.15.6:0/1070808411 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8a6806e240 0x7f8a68159e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:42.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:42.256+0000 7f8a6dc00640 1 -- 172.21.15.6:0/1070808411 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a6815b3b0 con 0x7f8a68161670 2024-03-04T09:33:42.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:42.256+0000 7f8a6dc00640 1 --2- 172.21.15.6:0/1070808411 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8a68161670 0x7f8a6814a870 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7f8a6000c410 tx=0x7f8a6000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:33:42.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:42.256+0000 7f8a64c00640 1 -- 172.21.15.6:0/1070808411 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a60011790 con 0x7f8a68161670 2024-03-04T09:33:42.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:42.256+0000 7f8a64c00640 1 -- 172.21.15.6:0/1070808411 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a60017070 con 0x7f8a68161670 2024-03-04T09:33:42.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:42.256+0000 7f8a64c00640 1 -- 172.21.15.6:0/1070808411 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a60011b70 con 0x7f8a68161670 2024-03-04T09:33:42.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:42.256+0000 7f8a6fc00640 1 -- 172.21.15.6:0/1070808411 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a6815a830 con 0x7f8a68161670 2024-03-04T09:33:42.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:42.257+0000 7f8a6fc00640 1 -- 172.21.15.6:0/1070808411 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a6815adf0 con 0x7f8a68161670 2024-03-04T09:33:42.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:42.258+0000 7f8a6fc00640 1 -- 172.21.15.6:0/1070808411 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a68069510 con 0x7f8a68161670 2024-03-04T09:33:42.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:42.259+0000 7f8a64c00640 1 -- 172.21.15.6:0/1070808411 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8a60011d10 con 0x7f8a68161670 2024-03-04T09:33:42.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:42.259+0000 7f8a64c00640 1 --2- 172.21.15.6:0/1070808411 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f8a4406b500 0x7f8a4406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:42.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:42.260+0000 7f8a64c00640 1 -- 172.21.15.6:0/1070808411 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f8a60089e40 con 0x7f8a68161670 2024-03-04T09:33:42.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:42.260+0000 7f8a6d200640 1 --2- 172.21.15.6:0/1070808411 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f8a4406b500 0x7f8a4406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:42.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:42.261+0000 7f8a6d200640 1 --2- 172.21.15.6:0/1070808411 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f8a4406b500 0x7f8a4406d9c0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f8a58035000 tx=0x7f8a58030000 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:33:42.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:42.264+0000 7f8a64c00640 1 -- 172.21.15.6:0/1070808411 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8a60021020 con 0x7f8a68161670 2024-03-04T09:33:42.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:42.501+0000 7f8a6fc00640 1 -- 172.21.15.6:0/1070808411 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8a68067830 con 0x7f8a68161670 2024-03-04T09:33:42.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:42.502+0000 7f8a64c00640 1 -- 172.21.15.6:0/1070808411 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3415 (secure 0 0 0) 0x7f8a68067830 con 0x7f8a68161670 2024-03-04T09:33:42.499 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:33:42.499 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":99,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:33:24.637813+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":9}]} 2024-03-04T09:33:42.499 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 99 2024-03-04T09:33:42.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:42.504+0000 7f8a6fc00640 1 -- 172.21.15.6:0/1070808411 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f8a4406b500 msgr2=0x7f8a4406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:42.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:42.504+0000 7f8a6fc00640 1 --2- 172.21.15.6:0/1070808411 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f8a4406b500 0x7f8a4406d9c0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f8a58035000 tx=0x7f8a58030000 comp rx=0 tx=0).stop 2024-03-04T09:33:42.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:42.504+0000 7f8a6fc00640 1 -- 172.21.15.6:0/1070808411 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8a68161670 msgr2=0x7f8a6814a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:42.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:42.504+0000 7f8a6fc00640 1 --2- 172.21.15.6:0/1070808411 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8a68161670 0x7f8a6814a870 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7f8a6000c410 tx=0x7f8a6000c8d0 comp rx=0 tx=0).stop 2024-03-04T09:33:42.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:42.504+0000 7f8a6fc00640 1 -- 172.21.15.6:0/1070808411 shutdown_connections 2024-03-04T09:33:42.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:42.504+0000 7f8a6fc00640 1 --2- 172.21.15.6:0/1070808411 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8a68161670 0x7f8a6814a870 unknown :-1 s=CLOSED pgs=1570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:42.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:42.504+0000 7f8a6fc00640 1 --2- 172.21.15.6:0/1070808411 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8a6806e240 0x7f8a68159e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:42.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:42.504+0000 7f8a6fc00640 1 --2- 172.21.15.6:0/1070808411 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f8a4406b500 0x7f8a4406d9c0 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:42.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:42.504+0000 7f8a6fc00640 1 --2- 172.21.15.6:0/1070808411 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8a6806cfe0 0x7f8a68148000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:42.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:42.504+0000 7f8a6fc00640 1 -- 172.21.15.6:0/1070808411 >> 172.21.15.6:0/1070808411 conn(0x7f8a6806afb0 msgr2=0x7f8a680b2ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:33:42.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:42.505+0000 7f8a6fc00640 1 -- 172.21.15.6:0/1070808411 shutdown_connections 2024-03-04T09:33:42.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:42.505+0000 7f8a6fc00640 1 -- 172.21.15.6:0/1070808411 wait complete. 2024-03-04T09:33:42.722 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:32:57.159681+0000', 'modified': '2024-03-04T09:33:24.637813+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8214, 'mds_1': 6796}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6796': {'gid': 6796, 'name': 'd', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.146:6837/2458070840', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 2458070840}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 2458070840}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8214': {'gid': 8214, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2446627517', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2446627517}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2446627517}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8214, 'qdb_cluster': [8214]} 2024-03-04T09:33:42.722 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6796:up:stopping 2024-03-04T09:33:43.723 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:33:43.786 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-04T09:33:43.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:43.952+0000 7f186ca00640 1 Processor -- start 2024-03-04T09:33:43.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:43.952+0000 7f186ca00640 1 -- start start 2024-03-04T09:33:43.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:43.954+0000 7f186ca00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f1868145300 0x7f18680b49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:43.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:43.954+0000 7f186ca00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f18680cd060 0x7f186815c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:43.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:43.954+0000 7f186ca00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18680714d0 con 0x7f1868070f10 2024-03-04T09:33:43.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:43.954+0000 7f186ca00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18680717b0 con 0x7f1868145300 2024-03-04T09:33:43.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:43.954+0000 7f186ca00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1868071a90 con 0x7f18680cd060 2024-03-04T09:33:43.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:43.954+0000 7f185ea00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f1868070f10 0x7f18680cc950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:32894/0 (socket says 172.21.15.6:32894) 2024-03-04T09:33:43.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:43.954+0000 7f185ea00640 1 -- 172.21.15.6:0/1325155745 learned_addr learned my addr 172.21.15.6:0/1325155745 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:33:43.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:43.955+0000 7f1867400640 1 --2- 172.21.15.6:0/1325155745 >> v2:172.21.15.146:3300/0 conn(0x7f1868145300 0x7f18680b49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:43.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:43.955+0000 7f1867e00640 1 --2- 172.21.15.6:0/1325155745 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f18680cd060 0x7f186815c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:43.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:43.955+0000 7f1866a00640 1 -- 172.21.15.6:0/1325155745 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1501996138 0 0) 0x7f18680714d0 con 0x7f1868070f10 2024-03-04T09:33:43.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:43.955+0000 7f1866a00640 1 -- 172.21.15.6:0/1325155745 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f184c003520 con 0x7f1868070f10 2024-03-04T09:33:43.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:43.955+0000 7f1866a00640 1 -- 172.21.15.6:0/1325155745 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1377091174 0 0) 0x7f184c003520 con 0x7f1868070f10 2024-03-04T09:33:43.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:43.956+0000 7f1866a00640 1 -- 172.21.15.6:0/1325155745 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f18680cd060 msgr2=0x7f186815c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:33:43.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:43.956+0000 7f1866a00640 1 --2- 172.21.15.6:0/1325155745 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f18680cd060 0x7f186815c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:43.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:43.956+0000 7f1866a00640 1 -- 172.21.15.6:0/1325155745 >> v2:172.21.15.146:3300/0 conn(0x7f1868145300 msgr2=0x7f18680b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:33:43.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:43.956+0000 7f1866a00640 1 --2- 172.21.15.6:0/1325155745 >> v2:172.21.15.146:3300/0 conn(0x7f1868145300 0x7f18680b49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:43.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:43.956+0000 7f1866a00640 1 -- 172.21.15.6:0/1325155745 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1868071d70 con 0x7f1868070f10 2024-03-04T09:33:43.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:43.956+0000 7f1866a00640 1 -- 172.21.15.6:0/1325155745 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7f1850001670 con 0x7f1868070f10 2024-03-04T09:33:43.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:43.956+0000 7f1867400640 1 --2- 172.21.15.6:0/1325155745 >> v2:172.21.15.146:3300/0 conn(0x7f1868145300 0x7f18680b49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T09:33:43.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:43.956+0000 7f1866a00640 1 -- 172.21.15.6:0/1325155745 >> v1:172.21.15.6:6789/0 conn(0x7f1868070f10 legacy=0x7f18680cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:43.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:43.956+0000 7f1866a00640 1 --2- 172.21.15.6:0/1325155745 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f184c0039b0 0x7f184c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:43.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:43.956+0000 7f1866a00640 1 --2- 172.21.15.6:0/1325155745 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f184c004800 0x7f184c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:43.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:43.956+0000 7f1866a00640 1 --2- 172.21.15.6:0/1325155745 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f18680cd060 0x7f184c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:43.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:43.956+0000 7f1866a00640 1 -- 172.21.15.6:0/1325155745 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18680717b0 con 0x7f184c004800 2024-03-04T09:33:43.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:43.956+0000 7f1866a00640 1 -- 172.21.15.6:0/1325155745 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1868071a90 con 0x7f184c0039b0 2024-03-04T09:33:43.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:43.956+0000 7f1866a00640 1 -- 172.21.15.6:0/1325155745 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f184c00ba40 con 0x7f18680cd060 2024-03-04T09:33:43.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:43.956+0000 7f185ea00640 1 --2- 172.21.15.6:0/1325155745 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f184c004800 0x7f184c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:43.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:43.957+0000 7f1867400640 1 --2- 172.21.15.6:0/1325155745 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f184c0039b0 0x7f184c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:43.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:43.957+0000 7f185ea00640 1 -- 172.21.15.6:0/1325155745 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f18680cd060 msgr2=0x7f184c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:33:43.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:43.957+0000 7f185ea00640 1 --2- 172.21.15.6:0/1325155745 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f18680cd060 0x7f184c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:43.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:43.957+0000 7f185ea00640 1 -- 172.21.15.6:0/1325155745 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f184c0039b0 msgr2=0x7f184c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:33:43.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:43.957+0000 7f185ea00640 1 --2- 172.21.15.6:0/1325155745 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f184c0039b0 0x7f184c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:43.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:43.957+0000 7f185ea00640 1 -- 172.21.15.6:0/1325155745 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f184c003520 con 0x7f184c004800 2024-03-04T09:33:43.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:43.957+0000 7f1867400640 1 --2- 172.21.15.6:0/1325155745 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f184c0039b0 0x7f184c003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:33:43.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:43.958+0000 7f185ea00640 1 --2- 172.21.15.6:0/1325155745 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f184c004800 0x7f184c004c80 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7f185000c160 tx=0x7f184c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81f5c14d52f6981b server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:33:43.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:43.958+0000 7f1866a00640 1 -- 172.21.15.6:0/1325155745 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1850007c80 con 0x7f184c004800 2024-03-04T09:33:43.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:43.958+0000 7f1866a00640 1 -- 172.21.15.6:0/1325155745 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1850030070 con 0x7f184c004800 2024-03-04T09:33:43.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:43.958+0000 7f1866a00640 1 -- 172.21.15.6:0/1325155745 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1850005cf0 con 0x7f184c004800 2024-03-04T09:33:43.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:43.959+0000 7f186ca00640 1 -- 172.21.15.6:0/1325155745 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f184c004800 msgr2=0x7f184c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:43.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:43.959+0000 7f186ca00640 1 --2- 172.21.15.6:0/1325155745 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f184c004800 0x7f184c004c80 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7f185000c160 tx=0x7f184c005830 comp rx=0 tx=0).stop 2024-03-04T09:33:43.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:43.959+0000 7f1867e00640 1 -- 172.21.15.6:0/1325155745 reap_dead start 2024-03-04T09:33:43.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:43.959+0000 7f186ca00640 1 -- 172.21.15.6:0/1325155745 shutdown_connections 2024-03-04T09:33:43.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:43.959+0000 7f186ca00640 1 -- 172.21.15.6:0/1325155745 >> 172.21.15.6:0/1325155745 conn(0x7f186806b3d0 msgr2=0x7f186806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:33:43.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:43.959+0000 7f186ca00640 1 -- 172.21.15.6:0/1325155745 shutdown_connections 2024-03-04T09:33:43.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:43.959+0000 7f186ca00640 1 -- 172.21.15.6:0/1325155745 wait complete. 2024-03-04T09:33:43.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:43.961+0000 7f186ca00640 1 Processor -- start 2024-03-04T09:33:43.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:43.961+0000 7f186ca00640 1 -- start start 2024-03-04T09:33:43.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:43.962+0000 7f186ca00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f184c004800 0x7f18680aeb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:43.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:43.962+0000 7f186ca00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f186813bb20 0x7f186813bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:43.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:43.962+0000 7f186ca00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f186813ca50 0x7f186813ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:43.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:43.962+0000 7f186ca00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f186815c570 con 0x7f186813bb20 2024-03-04T09:33:43.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:43.962+0000 7f186ca00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18681456e0 con 0x7f184c004800 2024-03-04T09:33:43.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:43.962+0000 7f186ca00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f186815c6f0 con 0x7f186813ca50 2024-03-04T09:33:43.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:43.962+0000 7f185ea00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f186813bb20 0x7f186813bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:43.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:43.962+0000 7f185ea00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f186813bb20 0x7f186813bfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:59202/0 (socket says 172.21.15.6:59202) 2024-03-04T09:33:43.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:43.962+0000 7f185ea00640 1 -- 172.21.15.6:0/446486671 learned_addr learned my addr 172.21.15.6:0/446486671 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:33:43.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:43.962+0000 7f1867400640 1 --2- 172.21.15.6:0/446486671 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f184c004800 0x7f18680aeb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:43.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:43.962+0000 7f1867e00640 1 --2- 172.21.15.6:0/446486671 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f186813ca50 0x7f186813ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:43.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:43.963+0000 7f185ea00640 1 -- 172.21.15.6:0/446486671 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f186813ca50 msgr2=0x7f186813ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:43.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:43.963+0000 7f185ea00640 1 --2- 172.21.15.6:0/446486671 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f186813ca50 0x7f186813ced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:43.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:43.963+0000 7f185ea00640 1 -- 172.21.15.6:0/446486671 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f184c004800 msgr2=0x7f18680aeb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:43.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:43.963+0000 7f185ea00640 1 --2- 172.21.15.6:0/446486671 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f184c004800 0x7f18680aeb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:43.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:43.963+0000 7f185ea00640 1 -- 172.21.15.6:0/446486671 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18681f7e30 con 0x7f186813bb20 2024-03-04T09:33:43.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:43.963+0000 7f185ea00640 1 --2- 172.21.15.6:0/446486671 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f186813bb20 0x7f186813bfa0 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 crypto rx=0x7f185000c280 tx=0x7f18680716a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:33:43.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:43.963+0000 7f185fe00640 1 -- 172.21.15.6:0/446486671 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1850045070 con 0x7f186813bb20 2024-03-04T09:33:43.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:43.963+0000 7f185fe00640 1 -- 172.21.15.6:0/446486671 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1850004bf0 con 0x7f186813bb20 2024-03-04T09:33:43.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:43.963+0000 7f185fe00640 1 -- 172.21.15.6:0/446486671 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1850007c60 con 0x7f186813bb20 2024-03-04T09:33:43.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:43.963+0000 7f186ca00640 1 -- 172.21.15.6:0/446486671 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18681f8110 con 0x7f186813bb20 2024-03-04T09:33:43.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:43.964+0000 7f186ca00640 1 -- 172.21.15.6:0/446486671 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f186813b320 con 0x7f186813bb20 2024-03-04T09:33:43.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:43.965+0000 7f186ca00640 1 -- 172.21.15.6:0/446486671 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f186804f330 con 0x7f186813bb20 2024-03-04T09:33:43.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:43.967+0000 7f185fe00640 1 -- 172.21.15.6:0/446486671 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1850007070 con 0x7f186813bb20 2024-03-04T09:33:43.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:43.968+0000 7f185fe00640 1 --2- 172.21.15.6:0/446486671 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f184406b500 0x7f184406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:43.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:43.968+0000 7f185fe00640 1 -- 172.21.15.6:0/446486671 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f18500ab290 con 0x7f186813bb20 2024-03-04T09:33:43.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:43.969+0000 7f1867400640 1 --2- 172.21.15.6:0/446486671 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f184406b500 0x7f184406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:43.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:43.970+0000 7f1867400640 1 --2- 172.21.15.6:0/446486671 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f184406b500 0x7f184406d9c0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f1858009c30 tx=0x7f1858008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:33:43.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:43.973+0000 7f185fe00640 1 -- 172.21.15.6:0/446486671 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f185007a240 con 0x7f186813bb20 2024-03-04T09:33:44.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:44.210+0000 7f186ca00640 1 -- 172.21.15.6:0/446486671 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f18680b37a0 con 0x7f186813bb20 2024-03-04T09:33:44.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:44.210+0000 7f185fe00640 1 -- 172.21.15.6:0/446486671 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3415 (secure 0 0 0) 0x7f1850079b40 con 0x7f186813bb20 2024-03-04T09:33:44.208 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:33:44.208 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":99,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:33:24.637813+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":9}]} 2024-03-04T09:33:44.208 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 99 2024-03-04T09:33:44.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:44.213+0000 7f186ca00640 1 -- 172.21.15.6:0/446486671 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f184406b500 msgr2=0x7f184406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:44.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:44.213+0000 7f186ca00640 1 --2- 172.21.15.6:0/446486671 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f184406b500 0x7f184406d9c0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f1858009c30 tx=0x7f1858008040 comp rx=0 tx=0).stop 2024-03-04T09:33:44.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:44.213+0000 7f186ca00640 1 -- 172.21.15.6:0/446486671 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f186813bb20 msgr2=0x7f186813bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:44.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:44.213+0000 7f186ca00640 1 --2- 172.21.15.6:0/446486671 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f186813bb20 0x7f186813bfa0 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 crypto rx=0x7f185000c280 tx=0x7f18680716a0 comp rx=0 tx=0).stop 2024-03-04T09:33:44.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:44.213+0000 7f186ca00640 1 -- 172.21.15.6:0/446486671 shutdown_connections 2024-03-04T09:33:44.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:44.213+0000 7f186ca00640 1 --2- 172.21.15.6:0/446486671 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f184406b500 0x7f184406d9c0 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:44.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:44.213+0000 7f186ca00640 1 --2- 172.21.15.6:0/446486671 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f186813ca50 0x7f186813ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:44.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:44.213+0000 7f186ca00640 1 --2- 172.21.15.6:0/446486671 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f186813bb20 0x7f186813bfa0 unknown :-1 s=CLOSED pgs=1573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:44.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:44.213+0000 7f186ca00640 1 --2- 172.21.15.6:0/446486671 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f184c004800 0x7f18680aeb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:44.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:44.213+0000 7f186ca00640 1 -- 172.21.15.6:0/446486671 >> 172.21.15.6:0/446486671 conn(0x7f186806b3d0 msgr2=0x7f186815cd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:33:44.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:44.213+0000 7f186ca00640 1 -- 172.21.15.6:0/446486671 shutdown_connections 2024-03-04T09:33:44.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:44.213+0000 7f186ca00640 1 -- 172.21.15.6:0/446486671 wait complete. 2024-03-04T09:33:44.434 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:32:57.159681+0000', 'modified': '2024-03-04T09:33:24.637813+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8214, 'mds_1': 6796}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6796': {'gid': 6796, 'name': 'd', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.146:6837/2458070840', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 2458070840}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 2458070840}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8214': {'gid': 8214, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2446627517', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2446627517}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2446627517}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8214, 'qdb_cluster': [8214]} 2024-03-04T09:33:44.434 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6796:up:stopping 2024-03-04T09:33:45.434 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:33:45.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:45.662+0000 7efc26800640 1 Processor -- start 2024-03-04T09:33:45.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:45.662+0000 7efc26800640 1 -- start start 2024-03-04T09:33:45.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:45.663+0000 7efc26800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7efc20070f10 0x7efc200cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:45.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:45.664+0000 7efc26800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efc200cce90 0x7efc2015c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:45.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:45.664+0000 7efc26800640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc200714d0 con 0x7efc20145300 2024-03-04T09:33:45.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:45.664+0000 7efc26800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc200cd300 con 0x7efc20070f10 2024-03-04T09:33:45.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:45.664+0000 7efc26800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc200717b0 con 0x7efc200cce90 2024-03-04T09:33:45.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:45.664+0000 7efc24a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7efc20070f10 0x7efc200cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:45.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:45.664+0000 7efc24a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7efc20070f10 0x7efc200cc950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:59206/0 (socket says 172.21.15.6:59206) 2024-03-04T09:33:45.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:45.664+0000 7efc24a00640 1 -- 172.21.15.6:0/2876555452 learned_addr learned my addr 172.21.15.6:0/2876555452 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:33:45.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:45.664+0000 7efc25e00640 1 --2- 172.21.15.6:0/2876555452 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efc200cce90 0x7efc2015c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:45.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:45.665+0000 7efc24a00640 1 -- 172.21.15.6:0/2876555452 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efc200cce90 msgr2=0x7efc2015c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:33:45.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:45.665+0000 7efc24a00640 1 --2- 172.21.15.6:0/2876555452 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efc200cce90 0x7efc2015c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:45.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:45.665+0000 7efc24a00640 1 -- 172.21.15.6:0/2876555452 >> v1:172.21.15.146:6789/0 conn(0x7efc20145300 legacy=0x7efc200b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:33:45.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:45.665+0000 7efc24a00640 1 -- 172.21.15.6:0/2876555452 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc20071a90 con 0x7efc20070f10 2024-03-04T09:33:45.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:45.666+0000 7efc24a00640 1 --2- 172.21.15.6:0/2876555452 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efc20070f10 0x7efc200cc950 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7efc0c009870 tx=0x7efc0c02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9db1dca219535bc6 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:33:45.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:45.666+0000 7efc1fe00640 1 -- 172.21.15.6:0/2876555452 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc0c0317c0 con 0x7efc20070f10 2024-03-04T09:33:45.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:45.666+0000 7efc1fe00640 1 -- 172.21.15.6:0/2876555452 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc0c037070 con 0x7efc20070f10 2024-03-04T09:33:45.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:45.666+0000 7efc1fe00640 1 -- 172.21.15.6:0/2876555452 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc0c031b80 con 0x7efc20070f10 2024-03-04T09:33:45.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:45.667+0000 7efc26800640 1 -- 172.21.15.6:0/2876555452 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efc20070f10 msgr2=0x7efc200cc950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:45.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:45.667+0000 7efc26800640 1 --2- 172.21.15.6:0/2876555452 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efc20070f10 0x7efc200cc950 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7efc0c009870 tx=0x7efc0c02b860 comp rx=0 tx=0).stop 2024-03-04T09:33:45.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:45.667+0000 7efc26800640 1 -- 172.21.15.6:0/2876555452 shutdown_connections 2024-03-04T09:33:45.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:45.667+0000 7efc26800640 1 --2- 172.21.15.6:0/2876555452 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efc200cce90 0x7efc2015c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:45.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:45.667+0000 7efc26800640 1 --2- 172.21.15.6:0/2876555452 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efc20070f10 0x7efc200cc950 unknown :-1 s=CLOSED pgs=1574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:45.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:45.667+0000 7efc26800640 1 -- 172.21.15.6:0/2876555452 >> 172.21.15.6:0/2876555452 conn(0x7efc2006b3d0 msgr2=0x7efc2006b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:33:45.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:45.667+0000 7efc26800640 1 -- 172.21.15.6:0/2876555452 shutdown_connections 2024-03-04T09:33:45.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:45.667+0000 7efc26800640 1 -- 172.21.15.6:0/2876555452 wait complete. 2024-03-04T09:33:45.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:45.669+0000 7efc26800640 1 Processor -- start 2024-03-04T09:33:45.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:45.670+0000 7efc26800640 1 -- start start 2024-03-04T09:33:45.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:45.671+0000 7efc26800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efc20070f10 0x7efc2013c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:45.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:45.671+0000 7efc26800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efc200cce90 0x7efc2006c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:45.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:45.671+0000 7efc26800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efc20145300 0x7efc2006cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:45.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:45.671+0000 7efc26800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc2015cab0 con 0x7efc200cce90 2024-03-04T09:33:45.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:45.671+0000 7efc26800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc201456e0 con 0x7efc20070f10 2024-03-04T09:33:45.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:45.671+0000 7efc26800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc2015cc30 con 0x7efc20145300 2024-03-04T09:33:45.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:45.671+0000 7efc24a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efc200cce90 0x7efc2006c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:45.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:45.671+0000 7efc24a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efc200cce90 0x7efc2006c7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:59208/0 (socket says 172.21.15.6:59208) 2024-03-04T09:33:45.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:45.671+0000 7efc24a00640 1 -- 172.21.15.6:0/3455577784 learned_addr learned my addr 172.21.15.6:0/3455577784 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:33:45.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:45.671+0000 7efc25400640 1 --2- 172.21.15.6:0/3455577784 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efc20070f10 0x7efc2013c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:45.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:45.671+0000 7efc25e00640 1 --2- 172.21.15.6:0/3455577784 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efc20145300 0x7efc2006cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:45.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:45.672+0000 7efc24a00640 1 -- 172.21.15.6:0/3455577784 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efc20145300 msgr2=0x7efc2006cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:45.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:45.672+0000 7efc24a00640 1 --2- 172.21.15.6:0/3455577784 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efc20145300 0x7efc2006cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:45.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:45.672+0000 7efc24a00640 1 -- 172.21.15.6:0/3455577784 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efc20070f10 msgr2=0x7efc2013c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:45.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:45.672+0000 7efc24a00640 1 --2- 172.21.15.6:0/3455577784 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efc20070f10 0x7efc2013c470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:45.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:45.672+0000 7efc24a00640 1 -- 172.21.15.6:0/3455577784 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc2006deb0 con 0x7efc200cce90 2024-03-04T09:33:45.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:45.672+0000 7efc25e00640 1 --2- 172.21.15.6:0/3455577784 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efc20145300 0x7efc2006cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:33:45.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:45.672+0000 7efc24a00640 1 --2- 172.21.15.6:0/3455577784 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efc200cce90 0x7efc2006c7c0 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 crypto rx=0x7efc0c009870 tx=0x7efc0c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:33:45.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:45.672+0000 7efc1d600640 1 -- 172.21.15.6:0/3455577784 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc0c032650 con 0x7efc200cce90 2024-03-04T09:33:45.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:45.672+0000 7efc1d600640 1 -- 172.21.15.6:0/3455577784 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc0c0327f0 con 0x7efc200cce90 2024-03-04T09:33:45.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:45.672+0000 7efc1d600640 1 -- 172.21.15.6:0/3455577784 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc0c032be0 con 0x7efc200cce90 2024-03-04T09:33:45.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:45.672+0000 7efc26800640 1 -- 172.21.15.6:0/3455577784 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc2006d330 con 0x7efc200cce90 2024-03-04T09:33:45.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:45.673+0000 7efc26800640 1 -- 172.21.15.6:0/3455577784 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc2006d8f0 con 0x7efc200cce90 2024-03-04T09:33:45.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:45.674+0000 7efc26800640 1 -- 172.21.15.6:0/3455577784 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc2004f330 con 0x7efc200cce90 2024-03-04T09:33:45.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:45.675+0000 7efc1d600640 1 -- 172.21.15.6:0/3455577784 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efc0c04c060 con 0x7efc200cce90 2024-03-04T09:33:45.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:45.676+0000 7efc1d600640 1 --2- 172.21.15.6:0/3455577784 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7efc0006b4d0 0x7efc0006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:45.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:45.676+0000 7efc1d600640 1 -- 172.21.15.6:0/3455577784 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7efc0c0ab9f0 con 0x7efc200cce90 2024-03-04T09:33:45.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:45.676+0000 7efc25400640 1 --2- 172.21.15.6:0/3455577784 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7efc0006b4d0 0x7efc0006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:45.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:45.677+0000 7efc25400640 1 --2- 172.21.15.6:0/3455577784 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7efc0006b4d0 0x7efc0006d990 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7efc14006fd0 tx=0x7efc14008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:33:45.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:45.680+0000 7efc1d600640 1 -- 172.21.15.6:0/3455577784 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7efc0c040070 con 0x7efc200cce90 2024-03-04T09:33:45.789 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-04T09:33:45.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:45.932+0000 7efc26800640 1 -- 172.21.15.6:0/3455577784 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efc2015bbf0 con 0x7efc200cce90 2024-03-04T09:33:45.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:45.932+0000 7efc1d600640 1 -- 172.21.15.6:0/3455577784 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3415 (secure 0 0 0) 0x7efc0c07a830 con 0x7efc200cce90 2024-03-04T09:33:45.929 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:33:45.929 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":99,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:33:24.637813+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":9}]} 2024-03-04T09:33:45.930 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 99 2024-03-04T09:33:45.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:45.935+0000 7efc26800640 1 -- 172.21.15.6:0/3455577784 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7efc0006b4d0 msgr2=0x7efc0006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:45.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:45.935+0000 7efc26800640 1 --2- 172.21.15.6:0/3455577784 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7efc0006b4d0 0x7efc0006d990 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7efc14006fd0 tx=0x7efc14008040 comp rx=0 tx=0).stop 2024-03-04T09:33:45.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:45.935+0000 7efc26800640 1 -- 172.21.15.6:0/3455577784 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efc200cce90 msgr2=0x7efc2006c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:45.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:45.935+0000 7efc26800640 1 --2- 172.21.15.6:0/3455577784 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efc200cce90 0x7efc2006c7c0 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 crypto rx=0x7efc0c009870 tx=0x7efc0c0046c0 comp rx=0 tx=0).stop 2024-03-04T09:33:45.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:45.935+0000 7efc26800640 1 -- 172.21.15.6:0/3455577784 shutdown_connections 2024-03-04T09:33:45.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:45.935+0000 7efc26800640 1 --2- 172.21.15.6:0/3455577784 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7efc0006b4d0 0x7efc0006d990 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:45.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:45.935+0000 7efc26800640 1 --2- 172.21.15.6:0/3455577784 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efc20145300 0x7efc2006cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:45.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:45.935+0000 7efc26800640 1 --2- 172.21.15.6:0/3455577784 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efc200cce90 0x7efc2006c7c0 unknown :-1 s=CLOSED pgs=1575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:45.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:45.935+0000 7efc26800640 1 --2- 172.21.15.6:0/3455577784 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efc20070f10 0x7efc2013c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:45.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:45.935+0000 7efc26800640 1 -- 172.21.15.6:0/3455577784 >> 172.21.15.6:0/3455577784 conn(0x7efc2006b3d0 msgr2=0x7efc2015d0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:33:45.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:45.935+0000 7efc26800640 1 -- 172.21.15.6:0/3455577784 shutdown_connections 2024-03-04T09:33:45.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:45.935+0000 7efc26800640 1 -- 172.21.15.6:0/3455577784 wait complete. 2024-03-04T09:33:46.153 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:32:57.159681+0000', 'modified': '2024-03-04T09:33:24.637813+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8214, 'mds_1': 6796}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6796': {'gid': 6796, 'name': 'd', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.146:6837/2458070840', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 2458070840}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 2458070840}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8214': {'gid': 8214, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2446627517', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2446627517}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2446627517}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8214, 'qdb_cluster': [8214]} 2024-03-04T09:33:46.153 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6796:up:stopping 2024-03-04T09:33:47.154 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:33:47.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:47.376+0000 7fd8d9a00640 1 Processor -- start 2024-03-04T09:33:47.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:47.376+0000 7fd8d9a00640 1 -- start start 2024-03-04T09:33:47.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:47.378+0000 7fd8d9a00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd8d4147400 0x7fd8d406afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:47.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:47.378+0000 7fd8d9a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd8d415ea80 0x7fd8d406b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:47.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:47.378+0000 7fd8d9a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8d4161ae0 con 0x7fd8d41625a0 2024-03-04T09:33:47.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:47.378+0000 7fd8d9a00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8d4162110 con 0x7fd8d4147400 2024-03-04T09:33:47.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:47.378+0000 7fd8d9a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8d415f080 con 0x7fd8d415ea80 2024-03-04T09:33:47.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:47.378+0000 7fd8cbe00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fd8d41625a0 0x7fd8d415e370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:32902/0 (socket says 172.21.15.6:32902) 2024-03-04T09:33:47.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:47.378+0000 7fd8cbe00640 1 -- 172.21.15.6:0/2343982919 learned_addr learned my addr 172.21.15.6:0/2343982919 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:33:47.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:47.379+0000 7fd8d3e00640 1 --2- 172.21.15.6:0/2343982919 >> v2:172.21.15.146:3300/0 conn(0x7fd8d4147400 0x7fd8d406afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:47.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:47.379+0000 7fd8d9000640 1 --2- 172.21.15.6:0/2343982919 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd8d415ea80 0x7fd8d406b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:47.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:47.379+0000 7fd8d3400640 1 -- 172.21.15.6:0/2343982919 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4294533434 0 0) 0x7fd8d4161ae0 con 0x7fd8d41625a0 2024-03-04T09:33:47.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:47.379+0000 7fd8d3400640 1 -- 172.21.15.6:0/2343982919 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8b8003520 con 0x7fd8d41625a0 2024-03-04T09:33:47.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:47.379+0000 7fd8d3400640 1 -- 172.21.15.6:0/2343982919 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 657074620 0 0) 0x7fd8b8003520 con 0x7fd8d41625a0 2024-03-04T09:33:47.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:47.379+0000 7fd8d3400640 1 -- 172.21.15.6:0/2343982919 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd8d415ea80 msgr2=0x7fd8d406b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:33:47.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:47.379+0000 7fd8d3400640 1 --2- 172.21.15.6:0/2343982919 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd8d415ea80 0x7fd8d406b4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:47.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:47.379+0000 7fd8d3400640 1 -- 172.21.15.6:0/2343982919 >> v2:172.21.15.146:3300/0 conn(0x7fd8d4147400 msgr2=0x7fd8d406afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:33:47.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:47.379+0000 7fd8d3400640 1 --2- 172.21.15.6:0/2343982919 >> v2:172.21.15.146:3300/0 conn(0x7fd8d4147400 0x7fd8d406afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:47.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:47.379+0000 7fd8d3400640 1 -- 172.21.15.6:0/2343982919 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8d4161da0 con 0x7fd8d41625a0 2024-03-04T09:33:47.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:47.379+0000 7fd8d3400640 1 -- 172.21.15.6:0/2343982919 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7fd8bc001670 con 0x7fd8d41625a0 2024-03-04T09:33:47.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:47.380+0000 7fd8d3400640 1 -- 172.21.15.6:0/2343982919 >> v1:172.21.15.6:6789/0 conn(0x7fd8d41625a0 legacy=0x7fd8d415e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:47.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:47.380+0000 7fd8d3400640 1 --2- 172.21.15.6:0/2343982919 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd8d415ea80 0x7fd8b80039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:47.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:47.380+0000 7fd8d3400640 1 --2- 172.21.15.6:0/2343982919 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd8b8003f30 0x7fd8b8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:47.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:47.380+0000 7fd8d3400640 1 --2- 172.21.15.6:0/2343982919 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd8b8004d40 0x7fd8b80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:47.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:47.380+0000 7fd8d3400640 1 -- 172.21.15.6:0/2343982919 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8d4162110 con 0x7fd8b8003f30 2024-03-04T09:33:47.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:47.380+0000 7fd8d3400640 1 -- 172.21.15.6:0/2343982919 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8d415f080 con 0x7fd8b8004d40 2024-03-04T09:33:47.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:47.380+0000 7fd8d3400640 1 -- 172.21.15.6:0/2343982919 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b800b9c0 con 0x7fd8d415ea80 2024-03-04T09:33:47.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:47.380+0000 7fd8cbe00640 1 --2- 172.21.15.6:0/2343982919 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd8b8003f30 0x7fd8b8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:47.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:47.380+0000 7fd8d3e00640 1 --2- 172.21.15.6:0/2343982919 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd8d415ea80 0x7fd8b80039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:47.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:47.380+0000 7fd8d9000640 1 --2- 172.21.15.6:0/2343982919 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd8b8004d40 0x7fd8b80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:47.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:47.380+0000 7fd8cbe00640 1 -- 172.21.15.6:0/2343982919 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd8d415ea80 msgr2=0x7fd8b80039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:33:47.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:47.380+0000 7fd8cbe00640 1 --2- 172.21.15.6:0/2343982919 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd8d415ea80 0x7fd8b80039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:47.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:47.380+0000 7fd8cbe00640 1 -- 172.21.15.6:0/2343982919 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd8b8004d40 msgr2=0x7fd8b80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:33:47.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:47.380+0000 7fd8cbe00640 1 --2- 172.21.15.6:0/2343982919 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd8b8004d40 0x7fd8b80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:47.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:47.381+0000 7fd8cbe00640 1 -- 172.21.15.6:0/2343982919 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8b8003520 con 0x7fd8b8003f30 2024-03-04T09:33:47.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:47.381+0000 7fd8d3e00640 1 --2- 172.21.15.6:0/2343982919 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd8d415ea80 0x7fd8b80039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:33:47.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:47.381+0000 7fd8d9000640 1 --2- 172.21.15.6:0/2343982919 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd8b8004d40 0x7fd8b80051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:33:47.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:47.382+0000 7fd8cbe00640 1 --2- 172.21.15.6:0/2343982919 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd8b8003f30 0x7fd8b8004800 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7fd8bc00c210 tx=0x7fd8b8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac952999376a0f71 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:33:47.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:47.382+0000 7fd8d3400640 1 -- 172.21.15.6:0/2343982919 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8bc007d30 con 0x7fd8b8003f30 2024-03-04T09:33:47.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:47.382+0000 7fd8d3400640 1 -- 172.21.15.6:0/2343982919 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8bc030070 con 0x7fd8b8003f30 2024-03-04T09:33:47.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:47.382+0000 7fd8d3400640 1 -- 172.21.15.6:0/2343982919 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8bc005d70 con 0x7fd8b8003f30 2024-03-04T09:33:47.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:47.383+0000 7fd8d9a00640 1 -- 172.21.15.6:0/2343982919 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd8b8003f30 msgr2=0x7fd8b8004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:47.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:47.383+0000 7fd8d9a00640 1 --2- 172.21.15.6:0/2343982919 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd8b8003f30 0x7fd8b8004800 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7fd8bc00c210 tx=0x7fd8b8005830 comp rx=0 tx=0).stop 2024-03-04T09:33:47.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:47.383+0000 7fd8d9000640 1 -- 172.21.15.6:0/2343982919 reap_dead start 2024-03-04T09:33:47.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:47.383+0000 7fd8d9a00640 1 -- 172.21.15.6:0/2343982919 shutdown_connections 2024-03-04T09:33:47.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:47.383+0000 7fd8d9a00640 1 -- 172.21.15.6:0/2343982919 >> 172.21.15.6:0/2343982919 conn(0x7fd8d406c0c0 msgr2=0x7fd8d406c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:33:47.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:47.383+0000 7fd8d9a00640 1 -- 172.21.15.6:0/2343982919 shutdown_connections 2024-03-04T09:33:47.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:47.383+0000 7fd8d9a00640 1 -- 172.21.15.6:0/2343982919 wait complete. 2024-03-04T09:33:47.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:47.385+0000 7fd8d9a00640 1 Processor -- start 2024-03-04T09:33:47.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:47.385+0000 7fd8d9a00640 1 -- start start 2024-03-04T09:33:47.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:47.386+0000 7fd8d9a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd8b8003f30 0x7fd8d4145600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:47.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:47.386+0000 7fd8d9a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd8d4157450 0x7fd8d41578d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:47.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:47.386+0000 7fd8d9a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd8d41e7d20 0x7fd8d41e81a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:47.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:47.386+0000 7fd8d9a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8d406b670 con 0x7fd8d4157450 2024-03-04T09:33:47.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:47.386+0000 7fd8d9a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8d406b4f0 con 0x7fd8b8003f30 2024-03-04T09:33:47.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:47.386+0000 7fd8d9a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8d406b7f0 con 0x7fd8d41e7d20 2024-03-04T09:33:47.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:47.386+0000 7fd8cbe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd8d4157450 0x7fd8d41578d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:47.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:47.386+0000 7fd8cbe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd8d4157450 0x7fd8d41578d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:59226/0 (socket says 172.21.15.6:59226) 2024-03-04T09:33:47.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:47.386+0000 7fd8cbe00640 1 -- 172.21.15.6:0/1218851066 learned_addr learned my addr 172.21.15.6:0/1218851066 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:33:47.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:47.387+0000 7fd8d3e00640 1 --2- 172.21.15.6:0/1218851066 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd8b8003f30 0x7fd8d4145600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:47.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:47.387+0000 7fd8d9000640 1 --2- 172.21.15.6:0/1218851066 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd8d41e7d20 0x7fd8d41e81a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:47.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:47.387+0000 7fd8cbe00640 1 -- 172.21.15.6:0/1218851066 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd8d41e7d20 msgr2=0x7fd8d41e81a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:47.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:47.387+0000 7fd8cbe00640 1 --2- 172.21.15.6:0/1218851066 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd8d41e7d20 0x7fd8d41e81a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:47.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:47.387+0000 7fd8cbe00640 1 -- 172.21.15.6:0/1218851066 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd8b8003f30 msgr2=0x7fd8d4145600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:47.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:47.387+0000 7fd8cbe00640 1 --2- 172.21.15.6:0/1218851066 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd8b8003f30 0x7fd8d4145600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:47.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:47.387+0000 7fd8cbe00640 1 -- 172.21.15.6:0/1218851066 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8d40ae110 con 0x7fd8d4157450 2024-03-04T09:33:47.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:47.387+0000 7fd8d9000640 1 --2- 172.21.15.6:0/1218851066 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd8d41e7d20 0x7fd8d41e81a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:33:47.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:47.387+0000 7fd8cbe00640 1 --2- 172.21.15.6:0/1218851066 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd8d4157450 0x7fd8d41578d0 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7fd8bc00c330 tx=0x7fd8d4161f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:33:47.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:47.387+0000 7fd8d0c00640 1 -- 172.21.15.6:0/1218851066 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8bc001c70 con 0x7fd8d4157450 2024-03-04T09:33:47.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:47.387+0000 7fd8d0c00640 1 -- 172.21.15.6:0/1218851066 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8bc004bf0 con 0x7fd8d4157450 2024-03-04T09:33:47.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:47.387+0000 7fd8d0c00640 1 -- 172.21.15.6:0/1218851066 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8bc007d30 con 0x7fd8d4157450 2024-03-04T09:33:47.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:47.387+0000 7fd8d9a00640 1 -- 172.21.15.6:0/1218851066 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8d40ae3f0 con 0x7fd8d4157450 2024-03-04T09:33:47.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:47.388+0000 7fd8d9a00640 1 -- 172.21.15.6:0/1218851066 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8d40ae9b0 con 0x7fd8d4157450 2024-03-04T09:33:47.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:47.389+0000 7fd8d9a00640 1 -- 172.21.15.6:0/1218851066 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8d41e86e0 con 0x7fd8d4157450 2024-03-04T09:33:47.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:47.390+0000 7fd8d0c00640 1 -- 172.21.15.6:0/1218851066 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd8bc047020 con 0x7fd8d4157450 2024-03-04T09:33:47.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:47.390+0000 7fd8d0c00640 1 --2- 172.21.15.6:0/1218851066 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd8b006b4d0 0x7fd8b006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:47.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:47.391+0000 7fd8d0c00640 1 -- 172.21.15.6:0/1218851066 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fd8bc0ab420 con 0x7fd8d4157450 2024-03-04T09:33:47.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:47.391+0000 7fd8d3e00640 1 --2- 172.21.15.6:0/1218851066 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd8b006b4d0 0x7fd8b006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:47.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:47.392+0000 7fd8d3e00640 1 --2- 172.21.15.6:0/1218851066 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd8b006b4d0 0x7fd8b006d990 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fd8c40099c0 tx=0x7fd8c4008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:33:47.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:47.400+0000 7fd8d0c00640 1 -- 172.21.15.6:0/1218851066 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd8bc07a250 con 0x7fd8d4157450 2024-03-04T09:33:47.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:47.639+0000 7fd8d9a00640 1 -- 172.21.15.6:0/1218851066 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd8d404f330 con 0x7fd8d4157450 2024-03-04T09:33:47.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:47.640+0000 7fd8d0c00640 1 -- 172.21.15.6:0/1218851066 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3415 (secure 0 0 0) 0x7fd8bc079b50 con 0x7fd8d4157450 2024-03-04T09:33:47.637 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:33:47.637 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":99,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:33:24.637813+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":9}]} 2024-03-04T09:33:47.637 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 99 2024-03-04T09:33:47.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:47.643+0000 7fd8d9a00640 1 -- 172.21.15.6:0/1218851066 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd8b006b4d0 msgr2=0x7fd8b006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:47.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:47.643+0000 7fd8d9a00640 1 --2- 172.21.15.6:0/1218851066 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd8b006b4d0 0x7fd8b006d990 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fd8c40099c0 tx=0x7fd8c4008040 comp rx=0 tx=0).stop 2024-03-04T09:33:47.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:47.643+0000 7fd8d9a00640 1 -- 172.21.15.6:0/1218851066 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd8d4157450 msgr2=0x7fd8d41578d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:47.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:47.643+0000 7fd8d9a00640 1 --2- 172.21.15.6:0/1218851066 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd8d4157450 0x7fd8d41578d0 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7fd8bc00c330 tx=0x7fd8d4161f70 comp rx=0 tx=0).stop 2024-03-04T09:33:47.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:47.643+0000 7fd8d9a00640 1 -- 172.21.15.6:0/1218851066 shutdown_connections 2024-03-04T09:33:47.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:47.643+0000 7fd8d9a00640 1 --2- 172.21.15.6:0/1218851066 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd8b006b4d0 0x7fd8b006d990 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:47.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:47.643+0000 7fd8d9a00640 1 --2- 172.21.15.6:0/1218851066 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd8d41e7d20 0x7fd8d41e81a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:47.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:47.643+0000 7fd8d9a00640 1 --2- 172.21.15.6:0/1218851066 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd8d4157450 0x7fd8d41578d0 unknown :-1 s=CLOSED pgs=1578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:47.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:47.643+0000 7fd8d9a00640 1 --2- 172.21.15.6:0/1218851066 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd8b8003f30 0x7fd8d4145600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:47.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:47.643+0000 7fd8d9a00640 1 -- 172.21.15.6:0/1218851066 >> 172.21.15.6:0/1218851066 conn(0x7fd8d406c0c0 msgr2=0x7fd8d40b16a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:33:47.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:47.643+0000 7fd8d9a00640 1 -- 172.21.15.6:0/1218851066 shutdown_connections 2024-03-04T09:33:47.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:47.643+0000 7fd8d9a00640 1 -- 172.21.15.6:0/1218851066 wait complete. 2024-03-04T09:33:47.791 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-04T09:33:47.861 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:32:57.159681+0000', 'modified': '2024-03-04T09:33:24.637813+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8214, 'mds_1': 6796}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6796': {'gid': 6796, 'name': 'd', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.146:6837/2458070840', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 2458070840}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 2458070840}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8214': {'gid': 8214, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2446627517', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2446627517}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2446627517}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8214, 'qdb_cluster': [8214]} 2024-03-04T09:33:47.861 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6796:up:stopping 2024-03-04T09:33:48.862 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:33:49.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:49.087+0000 7f9486000640 1 Processor -- start 2024-03-04T09:33:49.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:49.087+0000 7f9486000640 1 -- start start 2024-03-04T09:33:49.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:49.089+0000 7f9486000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f94800af1a0 0x7f948015c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:49.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:49.089+0000 7f9486000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f94800af580 0x7f94800b4c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:49.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:49.089+0000 7f9486000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f94800cdc20 0x7f94800b2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:49.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:49.089+0000 7f9486000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f948015cc40 con 0x7f94800cdc20 2024-03-04T09:33:49.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:49.089+0000 7f9486000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f948015cf20 con 0x7f94800af580 2024-03-04T09:33:49.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:49.089+0000 7f9486000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f948015d200 con 0x7f94800af1a0 2024-03-04T09:33:49.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:49.089+0000 7f947fe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f94800cdc20 0x7f94800b2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:49.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:49.089+0000 7f947fe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f94800cdc20 0x7f94800b2dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57138/0 (socket says 172.21.15.6:57138) 2024-03-04T09:33:49.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:49.090+0000 7f947fe00640 1 -- 172.21.15.6:0/2778077471 learned_addr learned my addr 172.21.15.6:0/2778077471 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:33:49.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:49.090+0000 7f9477e00640 1 --2- 172.21.15.6:0/2778077471 >> v2:172.21.15.146:3300/0 conn(0x7f94800af580 0x7f94800b4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:49.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:49.090+0000 7f947f400640 1 --2- 172.21.15.6:0/2778077471 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f94800af1a0 0x7f948015c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:49.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:49.090+0000 7f947fe00640 1 -- 172.21.15.6:0/2778077471 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f94800af1a0 msgr2=0x7f948015c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:33:49.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:49.090+0000 7f947fe00640 1 --2- 172.21.15.6:0/2778077471 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f94800af1a0 0x7f948015c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:49.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:49.090+0000 7f947fe00640 1 -- 172.21.15.6:0/2778077471 >> v2:172.21.15.146:3300/0 conn(0x7f94800af580 msgr2=0x7f94800b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:33:49.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:49.090+0000 7f947fe00640 1 --2- 172.21.15.6:0/2778077471 >> v2:172.21.15.146:3300/0 conn(0x7f94800af580 0x7f94800b4c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:49.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:49.090+0000 7f947fe00640 1 -- 172.21.15.6:0/2778077471 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f948015d4e0 con 0x7f94800cdc20 2024-03-04T09:33:49.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:49.091+0000 7f947fe00640 1 --2- 172.21.15.6:0/2778077471 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f94800cdc20 0x7f94800b2dd0 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 crypto rx=0x7f94800748c0 tx=0x7f947802be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=acd4494216dcb18c server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:33:49.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:49.092+0000 7f947ea00640 1 -- 172.21.15.6:0/2778077471 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f947802c9f0 con 0x7f94800cdc20 2024-03-04T09:33:49.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:49.092+0000 7f947ea00640 1 -- 172.21.15.6:0/2778077471 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f947802cb90 con 0x7f94800cdc20 2024-03-04T09:33:49.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:49.092+0000 7f947ea00640 1 -- 172.21.15.6:0/2778077471 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94780343f0 con 0x7f94800cdc20 2024-03-04T09:33:49.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:49.092+0000 7f9486000640 1 -- 172.21.15.6:0/2778077471 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f94800cdc20 msgr2=0x7f94800b2dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:49.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:49.092+0000 7f9486000640 1 --2- 172.21.15.6:0/2778077471 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f94800cdc20 0x7f94800b2dd0 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 crypto rx=0x7f94800748c0 tx=0x7f947802be50 comp rx=0 tx=0).stop 2024-03-04T09:33:49.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:49.092+0000 7f9486000640 1 -- 172.21.15.6:0/2778077471 shutdown_connections 2024-03-04T09:33:49.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:49.092+0000 7f9486000640 1 --2- 172.21.15.6:0/2778077471 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f94800cdc20 0x7f94800b2dd0 unknown :-1 s=CLOSED pgs=1579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:49.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:49.092+0000 7f9486000640 1 --2- 172.21.15.6:0/2778077471 >> v2:172.21.15.146:3300/0 conn(0x7f94800af580 0x7f94800b4c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:49.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:49.092+0000 7f9486000640 1 --2- 172.21.15.6:0/2778077471 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f94800af1a0 0x7f948015c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:49.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:49.092+0000 7f9486000640 1 -- 172.21.15.6:0/2778077471 >> 172.21.15.6:0/2778077471 conn(0x7f948006afb0 msgr2=0x7f948006b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:33:49.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:49.093+0000 7f9486000640 1 -- 172.21.15.6:0/2778077471 shutdown_connections 2024-03-04T09:33:49.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:49.093+0000 7f9486000640 1 -- 172.21.15.6:0/2778077471 wait complete. 2024-03-04T09:33:49.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:49.094+0000 7f9486000640 1 Processor -- start 2024-03-04T09:33:49.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:49.094+0000 7f9486000640 1 -- start start 2024-03-04T09:33:49.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:49.095+0000 7f9486000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f94800af1a0 0x7f94801f3230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:49.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:49.095+0000 7f9486000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f94800af580 0x7f94800cac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:49.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:49.095+0000 7f9486000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f94800cdc20 0x7f94800ccec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:49.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:49.095+0000 7f9486000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94800b3490 con 0x7f94800cdc20 2024-03-04T09:33:49.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:49.095+0000 7f9486000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94800b3310 con 0x7f94800af1a0 2024-03-04T09:33:49.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:49.095+0000 7f9486000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94800b3610 con 0x7f94800af580 2024-03-04T09:33:49.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:49.096+0000 7f947fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f94800cdc20 0x7f94800ccec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:49.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:49.096+0000 7f947fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f94800cdc20 0x7f94800ccec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57140/0 (socket says 172.21.15.6:57140) 2024-03-04T09:33:49.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:49.096+0000 7f947fe00640 1 -- 172.21.15.6:0/624707538 learned_addr learned my addr 172.21.15.6:0/624707538 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:33:49.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:49.096+0000 7f9477e00640 1 --2- 172.21.15.6:0/624707538 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f94800af580 0x7f94800cac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:49.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:49.096+0000 7f947f400640 1 --2- 172.21.15.6:0/624707538 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f94800af1a0 0x7f94801f3230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:49.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:49.096+0000 7f947fe00640 1 -- 172.21.15.6:0/624707538 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f94800af580 msgr2=0x7f94800cac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:49.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:49.096+0000 7f947fe00640 1 --2- 172.21.15.6:0/624707538 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f94800af580 0x7f94800cac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:49.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:49.096+0000 7f947fe00640 1 -- 172.21.15.6:0/624707538 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f94800af1a0 msgr2=0x7f94801f3230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:49.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:49.096+0000 7f947fe00640 1 --2- 172.21.15.6:0/624707538 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f94800af1a0 0x7f94801f3230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:49.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:49.096+0000 7f947fe00640 1 -- 172.21.15.6:0/624707538 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f948009c510 con 0x7f94800cdc20 2024-03-04T09:33:49.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:49.096+0000 7f947f400640 1 --2- 172.21.15.6:0/624707538 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f94800af1a0 0x7f94801f3230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:33:49.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:49.096+0000 7f947fe00640 1 --2- 172.21.15.6:0/624707538 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f94800cdc20 0x7f94800ccec0 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 crypto rx=0x7f9478036040 tx=0x7f9478004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:33:49.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:49.097+0000 7f9477400640 1 -- 172.21.15.6:0/624707538 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9478044020 con 0x7f94800cdc20 2024-03-04T09:33:49.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:49.097+0000 7f9477400640 1 -- 172.21.15.6:0/624707538 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9478040690 con 0x7f94800cdc20 2024-03-04T09:33:49.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:49.097+0000 7f9486000640 1 -- 172.21.15.6:0/624707538 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f948009c7f0 con 0x7f94800cdc20 2024-03-04T09:33:49.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:49.097+0000 7f9477400640 1 -- 172.21.15.6:0/624707538 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9478009510 con 0x7f94800cdc20 2024-03-04T09:33:49.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:49.097+0000 7f9486000640 1 -- 172.21.15.6:0/624707538 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f948009cdb0 con 0x7f94800cdc20 2024-03-04T09:33:49.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:49.098+0000 7f9486000640 1 -- 172.21.15.6:0/624707538 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9480069510 con 0x7f94800cdc20 2024-03-04T09:33:49.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:49.099+0000 7f9477400640 1 -- 172.21.15.6:0/624707538 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9478005d10 con 0x7f94800cdc20 2024-03-04T09:33:49.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:49.100+0000 7f9477400640 1 --2- 172.21.15.6:0/624707538 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f945c06b4d0 0x7f945c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:49.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:49.100+0000 7f9477400640 1 -- 172.21.15.6:0/624707538 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f94780ad170 con 0x7f94800cdc20 2024-03-04T09:33:49.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:49.101+0000 7f947f400640 1 --2- 172.21.15.6:0/624707538 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f945c06b4d0 0x7f945c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:49.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:49.102+0000 7f947f400640 1 --2- 172.21.15.6:0/624707538 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f945c06b4d0 0x7f945c06d990 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f9470006fd0 tx=0x7f9470008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:33:49.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:49.104+0000 7f9477400640 1 -- 172.21.15.6:0/624707538 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f947807c030 con 0x7f94800cdc20 2024-03-04T09:33:49.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:49.390+0000 7f9486000640 1 -- 172.21.15.6:0/624707538 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f948004f3a0 con 0x7f94800cdc20 2024-03-04T09:33:49.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:49.391+0000 7f9477400640 1 -- 172.21.15.6:0/624707538 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3415 (secure 0 0 0) 0x7f947807b930 con 0x7f94800cdc20 2024-03-04T09:33:49.388 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:33:49.389 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":99,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:33:24.637813+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":9}]} 2024-03-04T09:33:49.389 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 99 2024-03-04T09:33:49.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:49.394+0000 7f9486000640 1 -- 172.21.15.6:0/624707538 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f945c06b4d0 msgr2=0x7f945c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:49.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:49.394+0000 7f9486000640 1 --2- 172.21.15.6:0/624707538 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f945c06b4d0 0x7f945c06d990 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f9470006fd0 tx=0x7f9470008040 comp rx=0 tx=0).stop 2024-03-04T09:33:49.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:49.394+0000 7f9486000640 1 -- 172.21.15.6:0/624707538 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f94800cdc20 msgr2=0x7f94800ccec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:49.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:49.394+0000 7f9486000640 1 --2- 172.21.15.6:0/624707538 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f94800cdc20 0x7f94800ccec0 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 crypto rx=0x7f9478036040 tx=0x7f9478004a40 comp rx=0 tx=0).stop 2024-03-04T09:33:49.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:49.394+0000 7f9486000640 1 -- 172.21.15.6:0/624707538 shutdown_connections 2024-03-04T09:33:49.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:49.394+0000 7f9486000640 1 --2- 172.21.15.6:0/624707538 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f94800cdc20 0x7f94800ccec0 unknown :-1 s=CLOSED pgs=1580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:49.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:49.394+0000 7f9486000640 1 --2- 172.21.15.6:0/624707538 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f945c06b4d0 0x7f945c06d990 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:49.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:49.394+0000 7f9486000640 1 --2- 172.21.15.6:0/624707538 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f94800af580 0x7f94800cac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:49.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:49.394+0000 7f9486000640 1 --2- 172.21.15.6:0/624707538 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f94800af1a0 0x7f94801f3230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:49.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:49.394+0000 7f9486000640 1 -- 172.21.15.6:0/624707538 >> 172.21.15.6:0/624707538 conn(0x7f948006afb0 msgr2=0x7f9480071520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:33:49.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:49.394+0000 7f9486000640 1 -- 172.21.15.6:0/624707538 shutdown_connections 2024-03-04T09:33:49.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:49.394+0000 7f9486000640 1 -- 172.21.15.6:0/624707538 wait complete. 2024-03-04T09:33:49.609 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:32:57.159681+0000', 'modified': '2024-03-04T09:33:24.637813+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8214, 'mds_1': 6796}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6796': {'gid': 6796, 'name': 'd', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.146:6837/2458070840', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 2458070840}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 2458070840}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8214': {'gid': 8214, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2446627517', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2446627517}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2446627517}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8214, 'qdb_cluster': [8214]} 2024-03-04T09:33:49.609 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6796:up:stopping 2024-03-04T09:33:49.793 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-04T09:33:50.610 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:33:50.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:50.875+0000 7f7d5ca00640 1 Processor -- start 2024-03-04T09:33:50.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:50.875+0000 7f7d5ca00640 1 -- start start 2024-03-04T09:33:50.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:50.877+0000 7f7d5ca00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f7d58145300 0x7f7d580b49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:50.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:50.877+0000 7f7d5ca00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7d58070f10 0x7f7d580cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:50.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:50.877+0000 7f7d5ca00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f7d580cce90 0x7f7d580cd930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:50.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:50.877+0000 7f7d5ca00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d580cd300 con 0x7f7d580cce90 2024-03-04T09:33:50.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:50.877+0000 7f7d5ca00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d580714d0 con 0x7f7d58145300 2024-03-04T09:33:50.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:50.877+0000 7f7d5ca00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d580717b0 con 0x7f7d58070f10 2024-03-04T09:33:50.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:50.877+0000 7f7d57e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f7d580cce90 0x7f7d580cd930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:50.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:50.877+0000 7f7d57e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f7d580cce90 0x7f7d580cd930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57152/0 (socket says 172.21.15.6:57152) 2024-03-04T09:33:50.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:50.877+0000 7f7d57e00640 1 -- 172.21.15.6:0/4049426128 learned_addr learned my addr 172.21.15.6:0/4049426128 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:33:50.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:50.878+0000 7f7d56a00640 1 --2- 172.21.15.6:0/4049426128 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7d58070f10 0x7f7d580cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:50.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:50.878+0000 7f7d57400640 1 --2- 172.21.15.6:0/4049426128 >> v2:172.21.15.146:3300/0 conn(0x7f7d58145300 0x7f7d580b49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:50.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:50.878+0000 7f7d57e00640 1 -- 172.21.15.6:0/4049426128 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7d58070f10 msgr2=0x7f7d580cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:33:50.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:50.878+0000 7f7d57e00640 1 --2- 172.21.15.6:0/4049426128 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7d58070f10 0x7f7d580cc950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:50.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:50.878+0000 7f7d57e00640 1 -- 172.21.15.6:0/4049426128 >> v2:172.21.15.146:3300/0 conn(0x7f7d58145300 msgr2=0x7f7d580b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:33:50.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:50.878+0000 7f7d57e00640 1 --2- 172.21.15.6:0/4049426128 >> v2:172.21.15.146:3300/0 conn(0x7f7d58145300 0x7f7d580b49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:50.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:50.878+0000 7f7d57e00640 1 -- 172.21.15.6:0/4049426128 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d58071a90 con 0x7f7d580cce90 2024-03-04T09:33:50.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:50.879+0000 7f7d57e00640 1 --2- 172.21.15.6:0/4049426128 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7d580cce90 0x7f7d580cd930 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 crypto rx=0x7f7d580cf990 tx=0x7f7d5002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfc660c7b2e1c7d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:33:50.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:50.879+0000 7f7d56000640 1 -- 172.21.15.6:0/4049426128 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d5002ca80 con 0x7f7d580cce90 2024-03-04T09:33:50.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:50.880+0000 7f7d56000640 1 -- 172.21.15.6:0/4049426128 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d5002cc20 con 0x7f7d580cce90 2024-03-04T09:33:50.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:50.880+0000 7f7d56000640 1 -- 172.21.15.6:0/4049426128 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d500343f0 con 0x7f7d580cce90 2024-03-04T09:33:50.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:50.880+0000 7f7d5ca00640 1 -- 172.21.15.6:0/4049426128 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7d580cce90 msgr2=0x7f7d580cd930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:50.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:50.880+0000 7f7d5ca00640 1 --2- 172.21.15.6:0/4049426128 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7d580cce90 0x7f7d580cd930 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 crypto rx=0x7f7d580cf990 tx=0x7f7d5002bee0 comp rx=0 tx=0).stop 2024-03-04T09:33:50.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:50.880+0000 7f7d5ca00640 1 -- 172.21.15.6:0/4049426128 shutdown_connections 2024-03-04T09:33:50.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:50.880+0000 7f7d5ca00640 1 --2- 172.21.15.6:0/4049426128 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7d580cce90 0x7f7d580cd930 unknown :-1 s=CLOSED pgs=1581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:50.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:50.880+0000 7f7d5ca00640 1 --2- 172.21.15.6:0/4049426128 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7d58070f10 0x7f7d580cc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:50.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:50.880+0000 7f7d5ca00640 1 --2- 172.21.15.6:0/4049426128 >> v2:172.21.15.146:3300/0 conn(0x7f7d58145300 0x7f7d580b49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:50.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:50.880+0000 7f7d5ca00640 1 -- 172.21.15.6:0/4049426128 >> 172.21.15.6:0/4049426128 conn(0x7f7d5806b3d0 msgr2=0x7f7d5806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:33:50.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:50.880+0000 7f7d5ca00640 1 -- 172.21.15.6:0/4049426128 shutdown_connections 2024-03-04T09:33:50.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:50.880+0000 7f7d5ca00640 1 -- 172.21.15.6:0/4049426128 wait complete. 2024-03-04T09:33:50.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:50.882+0000 7f7d5ca00640 1 Processor -- start 2024-03-04T09:33:50.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:50.882+0000 7f7d5ca00640 1 -- start start 2024-03-04T09:33:50.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:50.883+0000 7f7d5ca00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7d58070f10 0x7f7d5815b150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:50.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:50.883+0000 7f7d5ca00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7d580cce90 0x7f7d580c6f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:50.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:50.883+0000 7f7d5ca00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7d58145300 0x7f7d580ca8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:50.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:50.883+0000 7f7d5ca00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d581456e0 con 0x7f7d58145300 2024-03-04T09:33:50.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:50.883+0000 7f7d5ca00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d580b4ee0 con 0x7f7d580cce90 2024-03-04T09:33:50.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:50.883+0000 7f7d5ca00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d580cde70 con 0x7f7d58070f10 2024-03-04T09:33:50.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:50.883+0000 7f7d57e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7d58145300 0x7f7d580ca8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:50.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:50.884+0000 7f7d57e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7d58145300 0x7f7d580ca8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57158/0 (socket says 172.21.15.6:57158) 2024-03-04T09:33:50.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:50.884+0000 7f7d57e00640 1 -- 172.21.15.6:0/3904868365 learned_addr learned my addr 172.21.15.6:0/3904868365 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:33:50.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:50.884+0000 7f7d57400640 1 --2- 172.21.15.6:0/3904868365 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7d58070f10 0x7f7d5815b150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:50.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:50.884+0000 7f7d56a00640 1 --2- 172.21.15.6:0/3904868365 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7d580cce90 0x7f7d580c6f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:50.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:50.884+0000 7f7d57e00640 1 -- 172.21.15.6:0/3904868365 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7d58070f10 msgr2=0x7f7d5815b150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:50.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:50.884+0000 7f7d57e00640 1 --2- 172.21.15.6:0/3904868365 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7d58070f10 0x7f7d5815b150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:50.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:50.884+0000 7f7d57e00640 1 -- 172.21.15.6:0/3904868365 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7d580cce90 msgr2=0x7f7d580c6f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:50.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:50.884+0000 7f7d57e00640 1 --2- 172.21.15.6:0/3904868365 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7d580cce90 0x7f7d580c6f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:50.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:50.884+0000 7f7d57e00640 1 -- 172.21.15.6:0/3904868365 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d580c8430 con 0x7f7d58145300 2024-03-04T09:33:50.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:50.884+0000 7f7d56a00640 1 --2- 172.21.15.6:0/3904868365 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7d580cce90 0x7f7d580c6f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:33:50.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:50.884+0000 7f7d57e00640 1 --2- 172.21.15.6:0/3904868365 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7d58145300 0x7f7d580ca8b0 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 crypto rx=0x7f7d50036040 tx=0x7f7d50004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:33:50.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:50.885+0000 7f7d4f400640 1 -- 172.21.15.6:0/3904868365 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d50043020 con 0x7f7d58145300 2024-03-04T09:33:50.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:50.885+0000 7f7d4f400640 1 -- 172.21.15.6:0/3904868365 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d5003e450 con 0x7f7d58145300 2024-03-04T09:33:50.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:50.885+0000 7f7d4f400640 1 -- 172.21.15.6:0/3904868365 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d50009510 con 0x7f7d58145300 2024-03-04T09:33:50.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:50.885+0000 7f7d5ca00640 1 -- 172.21.15.6:0/3904868365 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d580c8710 con 0x7f7d58145300 2024-03-04T09:33:50.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:50.885+0000 7f7d5ca00640 1 -- 172.21.15.6:0/3904868365 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d580c8cd0 con 0x7f7d58145300 2024-03-04T09:33:50.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:50.886+0000 7f7d5ca00640 1 -- 172.21.15.6:0/3904868365 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d5804f330 con 0x7f7d58145300 2024-03-04T09:33:50.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:50.888+0000 7f7d4f400640 1 -- 172.21.15.6:0/3904868365 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7d50046020 con 0x7f7d58145300 2024-03-04T09:33:50.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:50.889+0000 7f7d4f400640 1 --2- 172.21.15.6:0/3904868365 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f7d3806b4d0 0x7f7d3806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:50.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:50.889+0000 7f7d4f400640 1 -- 172.21.15.6:0/3904868365 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f7d500acaa0 con 0x7f7d58145300 2024-03-04T09:33:50.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:50.890+0000 7f7d57400640 1 --2- 172.21.15.6:0/3904868365 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f7d3806b4d0 0x7f7d3806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:50.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:50.891+0000 7f7d57400640 1 --2- 172.21.15.6:0/3904868365 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f7d3806b4d0 0x7f7d3806d990 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f7d48006fd0 tx=0x7f7d48008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:33:50.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:50.894+0000 7f7d4f400640 1 -- 172.21.15.6:0/3904868365 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7d5004d030 con 0x7f7d58145300 2024-03-04T09:33:51.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:51.131+0000 7f7d5ca00640 1 -- 172.21.15.6:0/3904868365 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7d580cd930 con 0x7f7d58145300 2024-03-04T09:33:51.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:51.132+0000 7f7d4f400640 1 -- 172.21.15.6:0/3904868365 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3415 (secure 0 0 0) 0x7f7d5007b8d0 con 0x7f7d58145300 2024-03-04T09:33:51.129 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:33:51.130 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":99,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:33:24.637813+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":9}]} 2024-03-04T09:33:51.130 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 99 2024-03-04T09:33:51.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:51.135+0000 7f7d5ca00640 1 -- 172.21.15.6:0/3904868365 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f7d3806b4d0 msgr2=0x7f7d3806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:51.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:51.135+0000 7f7d5ca00640 1 --2- 172.21.15.6:0/3904868365 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f7d3806b4d0 0x7f7d3806d990 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f7d48006fd0 tx=0x7f7d48008040 comp rx=0 tx=0).stop 2024-03-04T09:33:51.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:51.135+0000 7f7d5ca00640 1 -- 172.21.15.6:0/3904868365 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7d58145300 msgr2=0x7f7d580ca8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:51.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:51.135+0000 7f7d5ca00640 1 --2- 172.21.15.6:0/3904868365 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7d58145300 0x7f7d580ca8b0 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 crypto rx=0x7f7d50036040 tx=0x7f7d50004a40 comp rx=0 tx=0).stop 2024-03-04T09:33:51.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:51.135+0000 7f7d5ca00640 1 -- 172.21.15.6:0/3904868365 shutdown_connections 2024-03-04T09:33:51.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:51.135+0000 7f7d5ca00640 1 --2- 172.21.15.6:0/3904868365 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7d58145300 0x7f7d580ca8b0 unknown :-1 s=CLOSED pgs=1582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:51.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:51.135+0000 7f7d5ca00640 1 --2- 172.21.15.6:0/3904868365 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7d580cce90 0x7f7d580c6f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:51.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:51.135+0000 7f7d5ca00640 1 --2- 172.21.15.6:0/3904868365 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f7d3806b4d0 0x7f7d3806d990 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:51.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:51.135+0000 7f7d5ca00640 1 --2- 172.21.15.6:0/3904868365 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7d58070f10 0x7f7d5815b150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:51.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:51.135+0000 7f7d5ca00640 1 -- 172.21.15.6:0/3904868365 >> 172.21.15.6:0/3904868365 conn(0x7f7d5806b3d0 msgr2=0x7f7d580b2fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:33:51.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:51.135+0000 7f7d5ca00640 1 -- 172.21.15.6:0/3904868365 shutdown_connections 2024-03-04T09:33:51.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:51.135+0000 7f7d5ca00640 1 -- 172.21.15.6:0/3904868365 wait complete. 2024-03-04T09:33:51.356 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:32:57.159681+0000', 'modified': '2024-03-04T09:33:24.637813+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8214, 'mds_1': 6796}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6796': {'gid': 6796, 'name': 'd', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.146:6837/2458070840', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 2458070840}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 2458070840}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8214': {'gid': 8214, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2446627517', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2446627517}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2446627517}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8214, 'qdb_cluster': [8214]} 2024-03-04T09:33:51.356 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6796:up:stopping 2024-03-04T09:33:51.795 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-04T09:33:52.357 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:33:52.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:52.592+0000 7f2573200640 1 Processor -- start 2024-03-04T09:33:52.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:52.592+0000 7f2573200640 1 -- start start 2024-03-04T09:33:52.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:52.594+0000 7f2573200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f2574145300 0x7f25740b49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:52.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:52.594+0000 7f2573200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f25740cd060 0x7f257415c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:52.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:52.594+0000 7f2573200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25740714d0 con 0x7f2574070f10 2024-03-04T09:33:52.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:52.594+0000 7f2573200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25740717b0 con 0x7f2574145300 2024-03-04T09:33:52.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:52.594+0000 7f2573200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2574071a90 con 0x7f25740cd060 2024-03-04T09:33:52.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:52.594+0000 7f2571e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f2574145300 0x7f25740b49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:52.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:52.594+0000 7f256be00640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f2574070f10 0x7f25740cc950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.6:39522/0 (socket says 172.21.15.6:39522) 2024-03-04T09:33:52.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:52.594+0000 7f256be00640 1 -- 172.21.15.6:0/2447309159 learned_addr learned my addr 172.21.15.6:0/2447309159 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:33:52.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:52.594+0000 7f2572800640 1 --2- 172.21.15.6:0/2447309159 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f25740cd060 0x7f257415c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:52.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:52.595+0000 7f2571400640 1 -- 172.21.15.6:0/2447309159 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4069377658 0 0) 0x7f25740714d0 con 0x7f2574070f10 2024-03-04T09:33:52.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:52.595+0000 7f2571400640 1 -- 172.21.15.6:0/2447309159 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2558003520 con 0x7f2574070f10 2024-03-04T09:33:52.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:52.595+0000 7f2571e00640 1 -- 172.21.15.6:0/2447309159 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f25740cd060 msgr2=0x7f257415c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:33:52.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:52.595+0000 7f2571e00640 1 --2- 172.21.15.6:0/2447309159 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f25740cd060 0x7f257415c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:52.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:52.595+0000 7f2571e00640 1 -- 172.21.15.6:0/2447309159 >> v1:172.21.15.146:6789/0 conn(0x7f2574070f10 legacy=0x7f25740cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:52.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:52.595+0000 7f2571e00640 1 -- 172.21.15.6:0/2447309159 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2574071d70 con 0x7f2574145300 2024-03-04T09:33:52.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:52.595+0000 7f2572800640 1 --2- 172.21.15.6:0/2447309159 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f25740cd060 0x7f257415c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T09:33:52.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:52.597+0000 7f2571e00640 1 --2- 172.21.15.6:0/2447309159 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2574145300 0x7f25740b49a0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f25640097a0 tx=0x7f256402b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b23375c55826df92 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:33:52.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:52.597+0000 7f2571400640 1 -- 172.21.15.6:0/2447309159 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2564031780 con 0x7f2574145300 2024-03-04T09:33:52.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:52.597+0000 7f2571400640 1 -- 172.21.15.6:0/2447309159 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2564037070 con 0x7f2574145300 2024-03-04T09:33:52.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:52.597+0000 7f2571400640 1 -- 172.21.15.6:0/2447309159 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2564031b20 con 0x7f2574145300 2024-03-04T09:33:52.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:52.598+0000 7f2573200640 1 -- 172.21.15.6:0/2447309159 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2574145300 msgr2=0x7f25740b49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:52.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:52.598+0000 7f2573200640 1 --2- 172.21.15.6:0/2447309159 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2574145300 0x7f25740b49a0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f25640097a0 tx=0x7f256402b820 comp rx=0 tx=0).stop 2024-03-04T09:33:52.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:52.599+0000 7f2573200640 1 -- 172.21.15.6:0/2447309159 shutdown_connections 2024-03-04T09:33:52.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:52.599+0000 7f2573200640 1 --2- 172.21.15.6:0/2447309159 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f25740cd060 0x7f257415c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:52.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:52.599+0000 7f2573200640 1 --2- 172.21.15.6:0/2447309159 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2574145300 0x7f25740b49a0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:52.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:52.599+0000 7f2573200640 1 -- 172.21.15.6:0/2447309159 >> 172.21.15.6:0/2447309159 conn(0x7f257406b3d0 msgr2=0x7f257406b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:33:52.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:52.599+0000 7f2573200640 1 -- 172.21.15.6:0/2447309159 shutdown_connections 2024-03-04T09:33:52.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:52.599+0000 7f2573200640 1 -- 172.21.15.6:0/2447309159 wait complete. 2024-03-04T09:33:52.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:52.603+0000 7f2573200640 1 Processor -- start 2024-03-04T09:33:52.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:52.603+0000 7f2573200640 1 -- start start 2024-03-04T09:33:52.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:52.606+0000 7f2573200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2574070f10 0x7f25740ae8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:52.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:52.606+0000 7f2573200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f25740cd060 0x7f257415b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:52.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:52.606+0000 7f2573200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2574145300 0x7f257415b700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:52.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:52.606+0000 7f2573200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f257415cab0 con 0x7f25740cd060 2024-03-04T09:33:52.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:52.606+0000 7f2573200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25741456e0 con 0x7f2574145300 2024-03-04T09:33:52.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:52.606+0000 7f2573200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f257415cc30 con 0x7f2574070f10 2024-03-04T09:33:52.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:52.606+0000 7f256be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f25740cd060 0x7f257415b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:52.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:52.606+0000 7f256be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f25740cd060 0x7f257415b1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57160/0 (socket says 172.21.15.6:57160) 2024-03-04T09:33:52.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:52.606+0000 7f256be00640 1 -- 172.21.15.6:0/39892282 learned_addr learned my addr 172.21.15.6:0/39892282 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:33:52.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:52.606+0000 7f2571e00640 1 --2- 172.21.15.6:0/39892282 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2574070f10 0x7f25740ae8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:52.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:52.606+0000 7f2572800640 1 --2- 172.21.15.6:0/39892282 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2574145300 0x7f257415b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:52.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:52.606+0000 7f256be00640 1 -- 172.21.15.6:0/39892282 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2574070f10 msgr2=0x7f25740ae8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:52.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:52.606+0000 7f256be00640 1 --2- 172.21.15.6:0/39892282 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2574070f10 0x7f25740ae8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:52.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:52.607+0000 7f256be00640 1 -- 172.21.15.6:0/39892282 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2574145300 msgr2=0x7f257415b700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:52.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:52.607+0000 7f256be00640 1 --2- 172.21.15.6:0/39892282 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2574145300 0x7f257415b700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:52.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:52.607+0000 7f256be00640 1 -- 172.21.15.6:0/39892282 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25740aade0 con 0x7f25740cd060 2024-03-04T09:33:52.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:52.607+0000 7f2572800640 1 --2- 172.21.15.6:0/39892282 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2574145300 0x7f257415b700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:33:52.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:52.607+0000 7f256be00640 1 --2- 172.21.15.6:0/39892282 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f25740cd060 0x7f257415b1c0 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 crypto rx=0x7f255c00c430 tx=0x7f255c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:33:52.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:52.607+0000 7f256a000640 1 -- 172.21.15.6:0/39892282 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f255c017070 con 0x7f25740cd060 2024-03-04T09:33:52.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:52.607+0000 7f256a000640 1 -- 172.21.15.6:0/39892282 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f255c005ce0 con 0x7f25740cd060 2024-03-04T09:33:52.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:52.607+0000 7f256a000640 1 -- 172.21.15.6:0/39892282 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f255c0072f0 con 0x7f25740cd060 2024-03-04T09:33:52.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:52.607+0000 7f2573200640 1 -- 172.21.15.6:0/39892282 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25740aab00 con 0x7f25740cd060 2024-03-04T09:33:52.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:52.607+0000 7f2573200640 1 -- 172.21.15.6:0/39892282 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25740ac110 con 0x7f25740cd060 2024-03-04T09:33:52.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:52.608+0000 7f2573200640 1 -- 172.21.15.6:0/39892282 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f257404f330 con 0x7f25740cd060 2024-03-04T09:33:52.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:52.610+0000 7f256a000640 1 -- 172.21.15.6:0/39892282 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f255c01a070 con 0x7f25740cd060 2024-03-04T09:33:52.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:52.611+0000 7f256a000640 1 --2- 172.21.15.6:0/39892282 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f255006b4d0 0x7f255006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:52.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:52.612+0000 7f256a000640 1 -- 172.21.15.6:0/39892282 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f255c089840 con 0x7f25740cd060 2024-03-04T09:33:52.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:52.612+0000 7f2571e00640 1 --2- 172.21.15.6:0/39892282 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f255006b4d0 0x7f255006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:52.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:52.613+0000 7f2571e00640 1 --2- 172.21.15.6:0/39892282 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f255006b4d0 0x7f255006d990 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f2564036000 tx=0x7f2564008030 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:33:52.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:52.615+0000 7f256a000640 1 -- 172.21.15.6:0/39892282 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f255c058560 con 0x7f25740cd060 2024-03-04T09:33:52.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:52.873+0000 7f2573200640 1 -- 172.21.15.6:0/39892282 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f25741448b0 con 0x7f25740cd060 2024-03-04T09:33:52.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:52.874+0000 7f256a000640 1 -- 172.21.15.6:0/39892282 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3415 (secure 0 0 0) 0x7f255c057e60 con 0x7f25740cd060 2024-03-04T09:33:52.872 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:33:52.872 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":99,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:33:24.637813+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":9}]} 2024-03-04T09:33:52.872 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 99 2024-03-04T09:33:52.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:52.877+0000 7f2573200640 1 -- 172.21.15.6:0/39892282 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f255006b4d0 msgr2=0x7f255006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:52.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:52.877+0000 7f2573200640 1 --2- 172.21.15.6:0/39892282 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f255006b4d0 0x7f255006d990 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f2564036000 tx=0x7f2564008030 comp rx=0 tx=0).stop 2024-03-04T09:33:52.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:52.877+0000 7f2573200640 1 -- 172.21.15.6:0/39892282 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f25740cd060 msgr2=0x7f257415b1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:52.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:52.877+0000 7f2573200640 1 --2- 172.21.15.6:0/39892282 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f25740cd060 0x7f257415b1c0 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 crypto rx=0x7f255c00c430 tx=0x7f255c00c8f0 comp rx=0 tx=0).stop 2024-03-04T09:33:52.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:52.877+0000 7f2573200640 1 -- 172.21.15.6:0/39892282 shutdown_connections 2024-03-04T09:33:52.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:52.877+0000 7f2573200640 1 --2- 172.21.15.6:0/39892282 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2574145300 0x7f257415b700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:52.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:52.877+0000 7f2573200640 1 --2- 172.21.15.6:0/39892282 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f25740cd060 0x7f257415b1c0 unknown :-1 s=CLOSED pgs=1583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:52.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:52.877+0000 7f2573200640 1 --2- 172.21.15.6:0/39892282 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f255006b4d0 0x7f255006d990 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:52.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:52.877+0000 7f2573200640 1 --2- 172.21.15.6:0/39892282 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2574070f10 0x7f25740ae8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:52.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:52.877+0000 7f2573200640 1 -- 172.21.15.6:0/39892282 >> 172.21.15.6:0/39892282 conn(0x7f257406b3d0 msgr2=0x7f25740b28b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:33:52.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:52.878+0000 7f2573200640 1 -- 172.21.15.6:0/39892282 shutdown_connections 2024-03-04T09:33:52.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:52.878+0000 7f2573200640 1 -- 172.21.15.6:0/39892282 wait complete. 2024-03-04T09:33:53.095 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:32:57.159681+0000', 'modified': '2024-03-04T09:33:24.637813+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8214, 'mds_1': 6796}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6796': {'gid': 6796, 'name': 'd', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.146:6837/2458070840', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 2458070840}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 2458070840}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8214': {'gid': 8214, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2446627517', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2446627517}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2446627517}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8214, 'qdb_cluster': [8214]} 2024-03-04T09:33:53.096 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6796:up:stopping 2024-03-04T09:33:53.797 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-04T09:33:54.096 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:33:54.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:54.342+0000 7f9089200640 1 Processor -- start 2024-03-04T09:33:54.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:54.342+0000 7f9089200640 1 -- start start 2024-03-04T09:33:54.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:54.344+0000 7f9089200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f908415e070 0x7f90840d2960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:54.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:54.344+0000 7f9089200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90840d18c0 con 0x7f90840d0cd0 2024-03-04T09:33:54.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:54.344+0000 7f9089200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90840d1ba0 con 0x7f90840cf410 2024-03-04T09:33:54.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:54.344+0000 7f9089200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90840d1e80 con 0x7f908415e070 2024-03-04T09:33:54.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:54.344+0000 7f9083e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f90840d0cd0 0x7f90840b3630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:42696/0 (socket says 172.21.15.6:42696) 2024-03-04T09:33:54.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:54.344+0000 7f9083e00640 1 -- 172.21.15.6:0/3296144968 learned_addr learned my addr 172.21.15.6:0/3296144968 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:33:54.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:54.345+0000 7f9083400640 1 --2- 172.21.15.6:0/3296144968 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f908415e070 0x7f90840d2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:54.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:54.345+0000 7f9082000640 1 -- 172.21.15.6:0/3296144968 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2654354818 0 0) 0x7f90840d18c0 con 0x7f90840d0cd0 2024-03-04T09:33:54.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:54.345+0000 7f9082000640 1 -- 172.21.15.6:0/3296144968 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9068003520 con 0x7f90840d0cd0 2024-03-04T09:33:54.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:54.345+0000 7f9082000640 1 -- 172.21.15.6:0/3296144968 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2909343102 0 0) 0x7f9068003520 con 0x7f90840d0cd0 2024-03-04T09:33:54.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:54.345+0000 7f9082000640 1 -- 172.21.15.6:0/3296144968 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f908415e070 msgr2=0x7f90840d2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:33:54.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:54.345+0000 7f9082000640 1 --2- 172.21.15.6:0/3296144968 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f908415e070 0x7f90840d2960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:54.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:54.345+0000 7f9082000640 1 -- 172.21.15.6:0/3296144968 >> v1:172.21.15.146:6789/0 conn(0x7f90840cf410 legacy=0x7f90840d05c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:54.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:54.345+0000 7f9082000640 1 -- 172.21.15.6:0/3296144968 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90840d2160 con 0x7f90840d0cd0 2024-03-04T09:33:54.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:54.345+0000 7f9082000640 1 -- 172.21.15.6:0/3296144968 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7f907c0019f0 con 0x7f90840d0cd0 2024-03-04T09:33:54.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:54.345+0000 7f9082000640 1 -- 172.21.15.6:0/3296144968 >> v1:172.21.15.6:6789/0 conn(0x7f90840d0cd0 legacy=0x7f90840b3630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:54.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:54.345+0000 7f9082000640 1 --2- 172.21.15.6:0/3296144968 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f90680039e0 0x7f9068003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:54.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:54.345+0000 7f9082000640 1 --2- 172.21.15.6:0/3296144968 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f908415e070 0x7f9068004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:54.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:54.345+0000 7f9082000640 1 --2- 172.21.15.6:0/3296144968 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9068004d40 0x7f90680051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:54.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:54.345+0000 7f9082000640 1 -- 172.21.15.6:0/3296144968 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90840d1e80 con 0x7f90680039e0 2024-03-04T09:33:54.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:54.345+0000 7f9082000640 1 -- 172.21.15.6:0/3296144968 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f906800ba70 con 0x7f9068004d40 2024-03-04T09:33:54.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:54.345+0000 7f9082000640 1 -- 172.21.15.6:0/3296144968 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f906800bc70 con 0x7f908415e070 2024-03-04T09:33:54.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:54.346+0000 7f9083400640 1 --2- 172.21.15.6:0/3296144968 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f90680039e0 0x7f9068003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:54.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:54.346+0000 7f9082a00640 1 --2- 172.21.15.6:0/3296144968 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f908415e070 0x7f9068004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:54.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:54.346+0000 7f9083e00640 1 --2- 172.21.15.6:0/3296144968 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9068004d40 0x7f90680051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:54.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:54.346+0000 7f9083400640 1 -- 172.21.15.6:0/3296144968 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f908415e070 msgr2=0x7f9068004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:33:54.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:54.346+0000 7f9083400640 1 --2- 172.21.15.6:0/3296144968 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f908415e070 0x7f9068004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:54.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:54.346+0000 7f9083400640 1 -- 172.21.15.6:0/3296144968 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9068004d40 msgr2=0x7f90680051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:33:54.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:54.346+0000 7f9083400640 1 --2- 172.21.15.6:0/3296144968 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9068004d40 0x7f90680051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:54.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:54.346+0000 7f9083400640 1 -- 172.21.15.6:0/3296144968 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9068003520 con 0x7f90680039e0 2024-03-04T09:33:54.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:54.346+0000 7f9082a00640 1 --2- 172.21.15.6:0/3296144968 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f908415e070 0x7f9068004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:33:54.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:54.347+0000 7f9083400640 1 --2- 172.21.15.6:0/3296144968 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f90680039e0 0x7f9068003e00 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x7f9074009a90 tx=0x7f9068005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f559210a07af78b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:33:54.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:54.348+0000 7f9082000640 1 -- 172.21.15.6:0/3296144968 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9074039070 con 0x7f90680039e0 2024-03-04T09:33:54.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:54.348+0000 7f9082000640 1 -- 172.21.15.6:0/3296144968 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f907402c680 con 0x7f90680039e0 2024-03-04T09:33:54.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:54.348+0000 7f9082000640 1 -- 172.21.15.6:0/3296144968 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9074009690 con 0x7f90680039e0 2024-03-04T09:33:54.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:54.349+0000 7f9089200640 1 -- 172.21.15.6:0/3296144968 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f90680039e0 msgr2=0x7f9068003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:54.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:54.349+0000 7f9089200640 1 --2- 172.21.15.6:0/3296144968 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f90680039e0 0x7f9068003e00 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x7f9074009a90 tx=0x7f9068005830 comp rx=0 tx=0).stop 2024-03-04T09:33:54.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:54.349+0000 7f9083e00640 1 -- 172.21.15.6:0/3296144968 reap_dead start 2024-03-04T09:33:54.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:54.349+0000 7f9089200640 1 -- 172.21.15.6:0/3296144968 shutdown_connections 2024-03-04T09:33:54.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:54.349+0000 7f9089200640 1 -- 172.21.15.6:0/3296144968 >> 172.21.15.6:0/3296144968 conn(0x7f908408d2a0 msgr2=0x7f90840d3cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:33:54.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:54.349+0000 7f9089200640 1 -- 172.21.15.6:0/3296144968 shutdown_connections 2024-03-04T09:33:54.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:54.349+0000 7f9089200640 1 -- 172.21.15.6:0/3296144968 wait complete. 2024-03-04T09:33:54.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:54.351+0000 7f9089200640 1 Processor -- start 2024-03-04T09:33:54.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:54.351+0000 7f9089200640 1 -- start start 2024-03-04T09:33:54.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:54.353+0000 7f9089200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f90680039e0 0x7f90840ca2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:54.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:54.353+0000 7f9089200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9084153fa0 0x7f9084154400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:54.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:54.353+0000 7f9089200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f90840b1fa0 0x7f90840b2420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:54.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:54.353+0000 7f9089200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90840d4730 con 0x7f90840b1fa0 2024-03-04T09:33:54.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:54.353+0000 7f9089200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90840d45b0 con 0x7f90680039e0 2024-03-04T09:33:54.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:54.353+0000 7f9089200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90840cf870 con 0x7f9084153fa0 2024-03-04T09:33:54.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:54.353+0000 7f9083e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f90840b1fa0 0x7f90840b2420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:54.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:54.353+0000 7f9083e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f90840b1fa0 0x7f90840b2420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57176/0 (socket says 172.21.15.6:57176) 2024-03-04T09:33:54.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:54.353+0000 7f9083e00640 1 -- 172.21.15.6:0/4227757450 learned_addr learned my addr 172.21.15.6:0/4227757450 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:33:54.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:54.353+0000 7f9082a00640 1 --2- 172.21.15.6:0/4227757450 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9084153fa0 0x7f9084154400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:54.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:54.354+0000 7f9083400640 1 --2- 172.21.15.6:0/4227757450 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f90680039e0 0x7f90840ca2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:54.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:54.354+0000 7f9083e00640 1 -- 172.21.15.6:0/4227757450 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9084153fa0 msgr2=0x7f9084154400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:54.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:54.354+0000 7f9083e00640 1 --2- 172.21.15.6:0/4227757450 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9084153fa0 0x7f9084154400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:54.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:54.354+0000 7f9083e00640 1 -- 172.21.15.6:0/4227757450 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f90680039e0 msgr2=0x7f90840ca2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:54.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:54.354+0000 7f9083e00640 1 --2- 172.21.15.6:0/4227757450 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f90680039e0 0x7f90840ca2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:54.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:54.354+0000 7f9083e00640 1 -- 172.21.15.6:0/4227757450 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90841552c0 con 0x7f90840b1fa0 2024-03-04T09:33:54.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:54.354+0000 7f9083e00640 1 --2- 172.21.15.6:0/4227757450 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f90840b1fa0 0x7f90840b2420 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 crypto rx=0x7f908408d270 tx=0x7f907c00b0d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:33:54.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:54.354+0000 7f907b400640 1 -- 172.21.15.6:0/4227757450 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f907c00b870 con 0x7f90840b1fa0 2024-03-04T09:33:54.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:54.354+0000 7f907b400640 1 -- 172.21.15.6:0/4227757450 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f907c00ba30 con 0x7f90840b1fa0 2024-03-04T09:33:54.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:54.354+0000 7f9089200640 1 -- 172.21.15.6:0/4227757450 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90840ca800 con 0x7f90840b1fa0 2024-03-04T09:33:54.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:54.354+0000 7f907b400640 1 -- 172.21.15.6:0/4227757450 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f907c00be20 con 0x7f90840b1fa0 2024-03-04T09:33:54.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:54.355+0000 7f9089200640 1 -- 172.21.15.6:0/4227757450 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9084154d00 con 0x7f90840b1fa0 2024-03-04T09:33:54.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:54.356+0000 7f9089200640 1 -- 172.21.15.6:0/4227757450 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90840b2960 con 0x7f90840b1fa0 2024-03-04T09:33:54.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:54.357+0000 7f907b400640 1 -- 172.21.15.6:0/4227757450 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f907c01b020 con 0x7f90840b1fa0 2024-03-04T09:33:54.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:54.357+0000 7f907b400640 1 --2- 172.21.15.6:0/4227757450 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f906406b500 0x7f906406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:54.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:54.358+0000 7f907b400640 1 -- 172.21.15.6:0/4227757450 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f907c08a300 con 0x7f90840b1fa0 2024-03-04T09:33:54.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:54.358+0000 7f9083400640 1 --2- 172.21.15.6:0/4227757450 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f906406b500 0x7f906406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:54.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:54.359+0000 7f9083400640 1 --2- 172.21.15.6:0/4227757450 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f906406b500 0x7f906406d9c0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f9074036000 tx=0x7f9074030000 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:33:54.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:54.366+0000 7f907b400640 1 -- 172.21.15.6:0/4227757450 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f907c058a50 con 0x7f90840b1fa0 2024-03-04T09:33:54.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:54.603+0000 7f9089200640 1 -- 172.21.15.6:0/4227757450 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f90840c75c0 con 0x7f90840b1fa0 2024-03-04T09:33:54.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:54.604+0000 7f907b400640 1 -- 172.21.15.6:0/4227757450 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3415 (secure 0 0 0) 0x7f907c05caf0 con 0x7f90840b1fa0 2024-03-04T09:33:54.602 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:33:54.602 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":99,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:33:24.637813+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":9}]} 2024-03-04T09:33:54.602 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 99 2024-03-04T09:33:54.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:54.607+0000 7f9089200640 1 -- 172.21.15.6:0/4227757450 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f906406b500 msgr2=0x7f906406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:54.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:54.607+0000 7f9089200640 1 --2- 172.21.15.6:0/4227757450 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f906406b500 0x7f906406d9c0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f9074036000 tx=0x7f9074030000 comp rx=0 tx=0).stop 2024-03-04T09:33:54.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:54.607+0000 7f9089200640 1 -- 172.21.15.6:0/4227757450 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f90840b1fa0 msgr2=0x7f90840b2420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:54.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:54.607+0000 7f9089200640 1 --2- 172.21.15.6:0/4227757450 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f90840b1fa0 0x7f90840b2420 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 crypto rx=0x7f908408d270 tx=0x7f907c00b0d0 comp rx=0 tx=0).stop 2024-03-04T09:33:54.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:54.607+0000 7f9089200640 1 -- 172.21.15.6:0/4227757450 shutdown_connections 2024-03-04T09:33:54.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:54.607+0000 7f9089200640 1 --2- 172.21.15.6:0/4227757450 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f90840b1fa0 0x7f90840b2420 unknown :-1 s=CLOSED pgs=1586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:54.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:54.607+0000 7f9089200640 1 --2- 172.21.15.6:0/4227757450 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f906406b500 0x7f906406d9c0 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:54.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:54.607+0000 7f9089200640 1 --2- 172.21.15.6:0/4227757450 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9084153fa0 0x7f9084154400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:54.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:54.607+0000 7f9089200640 1 --2- 172.21.15.6:0/4227757450 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f90680039e0 0x7f90840ca2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:54.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:54.607+0000 7f9089200640 1 -- 172.21.15.6:0/4227757450 >> 172.21.15.6:0/4227757450 conn(0x7f908408d2a0 msgr2=0x7f9084160460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:33:54.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:54.607+0000 7f9089200640 1 -- 172.21.15.6:0/4227757450 shutdown_connections 2024-03-04T09:33:54.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:54.608+0000 7f9089200640 1 -- 172.21.15.6:0/4227757450 wait complete. 2024-03-04T09:33:54.825 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:32:57.159681+0000', 'modified': '2024-03-04T09:33:24.637813+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8214, 'mds_1': 6796}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6796': {'gid': 6796, 'name': 'd', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.146:6837/2458070840', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 2458070840}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 2458070840}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8214': {'gid': 8214, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2446627517', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2446627517}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2446627517}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8214, 'qdb_cluster': [8214]} 2024-03-04T09:33:54.825 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6796:up:stopping 2024-03-04T09:33:55.800 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-04T09:33:55.826 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:33:56.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:56.059+0000 7f6655600640 1 Processor -- start 2024-03-04T09:33:56.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:56.060+0000 7f6655600640 1 -- start start 2024-03-04T09:33:56.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:56.061+0000 7f6655600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f665015e2e0 0x7f66500d0830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:56.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:56.061+0000 7f6655600640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f66500d0d70 0x7f66500d11d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:56.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:56.061+0000 7f6655600640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66500b3940 con 0x7f66500d1b30 2024-03-04T09:33:56.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:56.061+0000 7f6655600640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66500d4e70 con 0x7f665015e2e0 2024-03-04T09:33:56.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:56.061+0000 7f6655600640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66500b3c40 con 0x7f66500d0d70 2024-03-04T09:33:56.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:56.062+0000 7f664ea00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f665015e2e0 0x7f66500d0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:56.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:56.062+0000 7f664ea00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f665015e2e0 0x7f66500d0830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57182/0 (socket says 172.21.15.6:57182) 2024-03-04T09:33:56.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:56.062+0000 7f664ea00640 1 -- 172.21.15.6:0/3560293155 learned_addr learned my addr 172.21.15.6:0/3560293155 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:33:56.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:56.062+0000 7f6647e00640 1 --2- 172.21.15.6:0/3560293155 >> v2:172.21.15.146:3300/0 conn(0x7f66500d0d70 0x7f66500d11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:56.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:56.062+0000 7f664ea00640 1 -- 172.21.15.6:0/3560293155 >> v2:172.21.15.146:3300/0 conn(0x7f66500d0d70 msgr2=0x7f66500d11d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:33:56.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:56.062+0000 7f664ea00640 1 --2- 172.21.15.6:0/3560293155 >> v2:172.21.15.146:3300/0 conn(0x7f66500d0d70 0x7f66500d11d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:56.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:56.062+0000 7f664ea00640 1 -- 172.21.15.6:0/3560293155 >> v1:172.21.15.146:6789/0 conn(0x7f66500d1b30 legacy=0x7f66500d1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:33:56.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:56.062+0000 7f664ea00640 1 -- 172.21.15.6:0/3560293155 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66500cf580 con 0x7f665015e2e0 2024-03-04T09:33:56.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:56.063+0000 7f664ea00640 1 --2- 172.21.15.6:0/3560293155 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f665015e2e0 0x7f66500d0830 secure :-1 s=READY pgs=1587 cs=0 l=1 rev1=1 crypto rx=0x7f66500d3510 tx=0x7f664002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30550122f827f1fe server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:33:56.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:56.064+0000 7f664e000640 1 -- 172.21.15.6:0/3560293155 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6640035020 con 0x7f665015e2e0 2024-03-04T09:33:56.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:56.064+0000 7f664e000640 1 -- 172.21.15.6:0/3560293155 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6640031870 con 0x7f665015e2e0 2024-03-04T09:33:56.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:56.064+0000 7f664e000640 1 -- 172.21.15.6:0/3560293155 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6640031c40 con 0x7f665015e2e0 2024-03-04T09:33:56.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:56.064+0000 7f6655600640 1 -- 172.21.15.6:0/3560293155 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f665015e2e0 msgr2=0x7f66500d0830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:56.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:56.064+0000 7f6655600640 1 --2- 172.21.15.6:0/3560293155 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f665015e2e0 0x7f66500d0830 secure :-1 s=READY pgs=1587 cs=0 l=1 rev1=1 crypto rx=0x7f66500d3510 tx=0x7f664002b910 comp rx=0 tx=0).stop 2024-03-04T09:33:56.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:56.064+0000 7f6655600640 1 -- 172.21.15.6:0/3560293155 shutdown_connections 2024-03-04T09:33:56.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:56.064+0000 7f6655600640 1 --2- 172.21.15.6:0/3560293155 >> v2:172.21.15.146:3300/0 conn(0x7f66500d0d70 0x7f66500d11d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:56.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:56.064+0000 7f6655600640 1 --2- 172.21.15.6:0/3560293155 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f665015e2e0 0x7f66500d0830 unknown :-1 s=CLOSED pgs=1587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:56.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:56.064+0000 7f6655600640 1 -- 172.21.15.6:0/3560293155 >> 172.21.15.6:0/3560293155 conn(0x7f66500d3f60 msgr2=0x7f66500d4370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:33:56.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:56.065+0000 7f6655600640 1 -- 172.21.15.6:0/3560293155 shutdown_connections 2024-03-04T09:33:56.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:56.065+0000 7f6655600640 1 -- 172.21.15.6:0/3560293155 wait complete. 2024-03-04T09:33:56.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:56.066+0000 7f6655600640 1 Processor -- start 2024-03-04T09:33:56.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:56.067+0000 7f6655600640 1 -- start start 2024-03-04T09:33:56.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:56.067+0000 7f6655600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f66500d0d70 0x7f66500c8fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:56.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:56.067+0000 7f6655600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f66500d1b30 0x7f66500b2840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:56.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:56.067+0000 7f6655600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f665015e2e0 0x7f66500b6df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:56.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:56.067+0000 7f6655600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66500cfe20 con 0x7f66500d1b30 2024-03-04T09:33:56.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:56.067+0000 7f6655600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66500cfb40 con 0x7f665015e2e0 2024-03-04T09:33:56.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:56.067+0000 7f6655600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66500b50e0 con 0x7f66500d0d70 2024-03-04T09:33:56.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:56.068+0000 7f6647e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f66500d1b30 0x7f66500b2840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:56.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:56.068+0000 7f6647e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f66500d1b30 0x7f66500b2840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57186/0 (socket says 172.21.15.6:57186) 2024-03-04T09:33:56.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:56.068+0000 7f6647e00640 1 -- 172.21.15.6:0/436603017 learned_addr learned my addr 172.21.15.6:0/436603017 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:33:56.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:56.068+0000 7f664f400640 1 --2- 172.21.15.6:0/436603017 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f665015e2e0 0x7f66500b6df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:56.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:56.068+0000 7f664ea00640 1 --2- 172.21.15.6:0/436603017 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f66500d0d70 0x7f66500c8fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:56.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:56.068+0000 7f6647e00640 1 -- 172.21.15.6:0/436603017 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f66500d0d70 msgr2=0x7f66500c8fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:56.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:56.068+0000 7f6647e00640 1 --2- 172.21.15.6:0/436603017 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f66500d0d70 0x7f66500c8fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:56.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:56.068+0000 7f6647e00640 1 -- 172.21.15.6:0/436603017 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f665015e2e0 msgr2=0x7f66500b6df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:56.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:56.068+0000 7f6647e00640 1 --2- 172.21.15.6:0/436603017 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f665015e2e0 0x7f66500b6df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:56.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:56.068+0000 7f6647e00640 1 -- 172.21.15.6:0/436603017 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66500b2d80 con 0x7f66500d1b30 2024-03-04T09:33:56.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:56.069+0000 7f6647e00640 1 --2- 172.21.15.6:0/436603017 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f66500d1b30 0x7f66500b2840 secure :-1 s=READY pgs=1588 cs=0 l=1 rev1=1 crypto rx=0x7f663800d800 tx=0x7f663800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:33:56.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:56.069+0000 7f6646a00640 1 -- 172.21.15.6:0/436603017 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f663800b660 con 0x7f66500d1b30 2024-03-04T09:33:56.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:56.069+0000 7f6646a00640 1 -- 172.21.15.6:0/436603017 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6638019070 con 0x7f66500d1b30 2024-03-04T09:33:56.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:56.069+0000 7f6646a00640 1 -- 172.21.15.6:0/436603017 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f663800ba20 con 0x7f66500d1b30 2024-03-04T09:33:56.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:56.069+0000 7f6655600640 1 -- 172.21.15.6:0/436603017 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66501f3f10 con 0x7f66500d1b30 2024-03-04T09:33:56.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:56.069+0000 7f6655600640 1 -- 172.21.15.6:0/436603017 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66501f44d0 con 0x7f66500d1b30 2024-03-04T09:33:56.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:56.070+0000 7f6655600640 1 -- 172.21.15.6:0/436603017 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f665004f3a0 con 0x7f66500d1b30 2024-03-04T09:33:56.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:56.072+0000 7f6646a00640 1 -- 172.21.15.6:0/436603017 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f663800bbc0 con 0x7f66500d1b30 2024-03-04T09:33:56.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:56.073+0000 7f6646a00640 1 --2- 172.21.15.6:0/436603017 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f662c06b4d0 0x7f662c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:56.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:56.073+0000 7f6646a00640 1 -- 172.21.15.6:0/436603017 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f663808a6e0 con 0x7f66500d1b30 2024-03-04T09:33:56.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:56.074+0000 7f664ea00640 1 --2- 172.21.15.6:0/436603017 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f662c06b4d0 0x7f662c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:56.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:56.075+0000 7f664ea00640 1 --2- 172.21.15.6:0/436603017 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f662c06b4d0 0x7f662c06d990 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f6640036000 tx=0x7f6640008030 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:33:56.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:56.083+0000 7f6646a00640 1 -- 172.21.15.6:0/436603017 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f66500b2d80 con 0x7f66500d1b30 2024-03-04T09:33:56.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:56.321+0000 7f6655600640 1 -- 172.21.15.6:0/436603017 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f66500d2870 con 0x7f66500d1b30 2024-03-04T09:33:56.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:56.322+0000 7f6646a00640 1 -- 172.21.15.6:0/436603017 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3465 (secure 0 0 0) 0x7f663801c360 con 0x7f66500d1b30 2024-03-04T09:33:56.319 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:33:56.319 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:33:54.652601+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"laggy_since":"2024-03-04T09:33:54.652601+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":9}]} 2024-03-04T09:33:56.319 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 100 2024-03-04T09:33:56.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:56.324+0000 7f6655600640 1 -- 172.21.15.6:0/436603017 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f662c06b4d0 msgr2=0x7f662c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:56.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:56.324+0000 7f6655600640 1 --2- 172.21.15.6:0/436603017 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f662c06b4d0 0x7f662c06d990 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f6640036000 tx=0x7f6640008030 comp rx=0 tx=0).stop 2024-03-04T09:33:56.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:56.324+0000 7f6655600640 1 -- 172.21.15.6:0/436603017 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f66500d1b30 msgr2=0x7f66500b2840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:56.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:56.324+0000 7f6655600640 1 --2- 172.21.15.6:0/436603017 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f66500d1b30 0x7f66500b2840 secure :-1 s=READY pgs=1588 cs=0 l=1 rev1=1 crypto rx=0x7f663800d800 tx=0x7f663800dcc0 comp rx=0 tx=0).stop 2024-03-04T09:33:56.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:56.324+0000 7f6655600640 1 -- 172.21.15.6:0/436603017 shutdown_connections 2024-03-04T09:33:56.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:56.324+0000 7f6655600640 1 --2- 172.21.15.6:0/436603017 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f665015e2e0 0x7f66500b6df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:56.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:56.324+0000 7f6655600640 1 --2- 172.21.15.6:0/436603017 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f66500d1b30 0x7f66500b2840 unknown :-1 s=CLOSED pgs=1588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:56.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:56.324+0000 7f6655600640 1 --2- 172.21.15.6:0/436603017 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f662c06b4d0 0x7f662c06d990 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:56.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:56.324+0000 7f6655600640 1 --2- 172.21.15.6:0/436603017 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f66500d0d70 0x7f66500c8fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:56.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:56.324+0000 7f6655600640 1 -- 172.21.15.6:0/436603017 >> 172.21.15.6:0/436603017 conn(0x7f66500d3f60 msgr2=0x7f665006d060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:33:56.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:56.324+0000 7f6655600640 1 -- 172.21.15.6:0/436603017 shutdown_connections 2024-03-04T09:33:56.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:56.324+0000 7f6655600640 1 -- 172.21.15.6:0/436603017 wait complete. 2024-03-04T09:33:56.539 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:32:57.159681+0000', 'modified': '2024-03-04T09:33:54.652601+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8214, 'mds_1': 6796}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6796': {'gid': 6796, 'name': 'd', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.146:6837/2458070840', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 2458070840}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 2458070840}]}, 'join_fscid': -1, 'laggy_since': '2024-03-04T09:33:54.652601+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8214': {'gid': 8214, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2446627517', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2446627517}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2446627517}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8214, 'qdb_cluster': [8214]} 2024-03-04T09:33:56.539 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6796:up:stopping 2024-03-04T09:33:57.540 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:33:57.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:57.760+0000 7ffa6fe00640 1 Processor -- start 2024-03-04T09:33:57.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:57.761+0000 7ffa6fe00640 1 -- start start 2024-03-04T09:33:57.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:57.762+0000 7ffa6fe00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ffa7006c690 0x7ffa7015c740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:57.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:57.762+0000 7ffa6fe00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa700afb40 con 0x7ffa7015c360 2024-03-04T09:33:57.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:57.762+0000 7ffa6fe00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa700afe20 con 0x7ffa7006ed90 2024-03-04T09:33:57.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:57.762+0000 7ffa6fe00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa7006d310 con 0x7ffa7006c690 2024-03-04T09:33:57.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:57.763+0000 7ffa6ea00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7ffa7015c360 0x7ffa700ce370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:42708/0 (socket says 172.21.15.6:42708) 2024-03-04T09:33:57.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:57.763+0000 7ffa6ea00640 1 -- 172.21.15.6:0/1830032619 learned_addr learned my addr 172.21.15.6:0/1830032619 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:33:57.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:57.763+0000 7ffa6f400640 1 --2- 172.21.15.6:0/1830032619 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ffa7006c690 0x7ffa7015c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:57.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:57.763+0000 7ffa6d600640 1 -- 172.21.15.6:0/1830032619 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3028313041 0 0) 0x7ffa700afb40 con 0x7ffa7015c360 2024-03-04T09:33:57.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:57.763+0000 7ffa6d600640 1 -- 172.21.15.6:0/1830032619 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa54003520 con 0x7ffa7015c360 2024-03-04T09:33:57.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:57.764+0000 7ffa6d600640 1 -- 172.21.15.6:0/1830032619 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 913851640 0 0) 0x7ffa54003520 con 0x7ffa7015c360 2024-03-04T09:33:57.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:57.764+0000 7ffa6d600640 1 -- 172.21.15.6:0/1830032619 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ffa7006c690 msgr2=0x7ffa7015c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:33:57.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:57.764+0000 7ffa6d600640 1 --2- 172.21.15.6:0/1830032619 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ffa7006c690 0x7ffa7015c740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:57.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:57.764+0000 7ffa6d600640 1 -- 172.21.15.6:0/1830032619 >> v1:172.21.15.146:6789/0 conn(0x7ffa7006ed90 legacy=0x7ffa7006bf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:57.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:57.764+0000 7ffa6d600640 1 -- 172.21.15.6:0/1830032619 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa7006d560 con 0x7ffa7015c360 2024-03-04T09:33:57.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:57.764+0000 7ffa6d600640 1 -- 172.21.15.6:0/1830032619 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7ffa58001670 con 0x7ffa7015c360 2024-03-04T09:33:57.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:57.764+0000 7ffa6d600640 1 -- 172.21.15.6:0/1830032619 >> v1:172.21.15.6:6789/0 conn(0x7ffa7015c360 legacy=0x7ffa700ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:57.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:57.764+0000 7ffa6d600640 1 --2- 172.21.15.6:0/1830032619 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ffa7006c690 0x7ffa540039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:57.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:57.764+0000 7ffa6d600640 1 --2- 172.21.15.6:0/1830032619 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffa54003f60 0x7ffa54004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:57.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:57.764+0000 7ffa6d600640 1 --2- 172.21.15.6:0/1830032619 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffa54004d40 0x7ffa540051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:57.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:57.764+0000 7ffa6d600640 1 -- 172.21.15.6:0/1830032619 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa7006d310 con 0x7ffa54003f60 2024-03-04T09:33:57.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:57.764+0000 7ffa6d600640 1 -- 172.21.15.6:0/1830032619 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa5400ff60 con 0x7ffa54004d40 2024-03-04T09:33:57.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:57.764+0000 7ffa6d600640 1 -- 172.21.15.6:0/1830032619 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa54010160 con 0x7ffa7006c690 2024-03-04T09:33:57.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:57.764+0000 7ffa6e000640 1 --2- 172.21.15.6:0/1830032619 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffa54003f60 0x7ffa54004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:57.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:57.765+0000 7ffa6f400640 1 --2- 172.21.15.6:0/1830032619 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffa54004d40 0x7ffa540051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:57.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:57.765+0000 7ffa6ea00640 1 --2- 172.21.15.6:0/1830032619 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ffa7006c690 0x7ffa540039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:57.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:57.765+0000 7ffa6e000640 1 -- 172.21.15.6:0/1830032619 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ffa7006c690 msgr2=0x7ffa540039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:33:57.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:57.765+0000 7ffa6e000640 1 --2- 172.21.15.6:0/1830032619 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ffa7006c690 0x7ffa540039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:57.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:57.765+0000 7ffa6e000640 1 -- 172.21.15.6:0/1830032619 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffa54004d40 msgr2=0x7ffa540051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:33:57.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:57.765+0000 7ffa6e000640 1 --2- 172.21.15.6:0/1830032619 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffa54004d40 0x7ffa540051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:57.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:57.765+0000 7ffa6e000640 1 -- 172.21.15.6:0/1830032619 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa54003520 con 0x7ffa54003f60 2024-03-04T09:33:57.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:57.766+0000 7ffa6e000640 1 --2- 172.21.15.6:0/1830032619 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffa54003f60 0x7ffa54004800 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 crypto rx=0x7ffa60001750 tx=0x7ffa54005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa224b1eda5e66ac server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:33:57.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:57.766+0000 7ffa6d600640 1 -- 172.21.15.6:0/1830032619 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa60001370 con 0x7ffa54003f60 2024-03-04T09:33:57.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:57.767+0000 7ffa6d600640 1 -- 172.21.15.6:0/1830032619 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa600052d0 con 0x7ffa54003f60 2024-03-04T09:33:57.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:57.767+0000 7ffa6d600640 1 -- 172.21.15.6:0/1830032619 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa60003830 con 0x7ffa54003f60 2024-03-04T09:33:57.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:57.767+0000 7ffa6fe00640 1 -- 172.21.15.6:0/1830032619 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffa54003f60 msgr2=0x7ffa54004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:57.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:57.767+0000 7ffa6fe00640 1 --2- 172.21.15.6:0/1830032619 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffa54003f60 0x7ffa54004800 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 crypto rx=0x7ffa60001750 tx=0x7ffa54005830 comp rx=0 tx=0).stop 2024-03-04T09:33:57.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:57.767+0000 7ffa6f400640 1 -- 172.21.15.6:0/1830032619 reap_dead start 2024-03-04T09:33:57.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:57.767+0000 7ffa6fe00640 1 -- 172.21.15.6:0/1830032619 shutdown_connections 2024-03-04T09:33:57.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:57.767+0000 7ffa6fe00640 1 -- 172.21.15.6:0/1830032619 >> 172.21.15.6:0/1830032619 conn(0x7ffa700ccf70 msgr2=0x7ffa700cd350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:33:57.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:57.767+0000 7ffa6fe00640 1 -- 172.21.15.6:0/1830032619 shutdown_connections 2024-03-04T09:33:57.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:57.767+0000 7ffa6fe00640 1 -- 172.21.15.6:0/1830032619 wait complete. 2024-03-04T09:33:57.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:57.770+0000 7ffa6fe00640 1 Processor -- start 2024-03-04T09:33:57.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:57.770+0000 7ffa6fe00640 1 -- start start 2024-03-04T09:33:57.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:57.771+0000 7ffa6fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffa54003f60 0x7ffa700a9470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:57.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:57.771+0000 7ffa6fe00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ffa70155c70 0x7ffa70149d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:57.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:57.771+0000 7ffa6fe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffa70158d30 0x7ffa700b4290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:57.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:57.771+0000 7ffa6fe00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa700cea80 con 0x7ffa54003f60 2024-03-04T09:33:57.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:57.771+0000 7ffa6fe00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa7006d840 con 0x7ffa70158d30 2024-03-04T09:33:57.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:57.771+0000 7ffa6fe00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa7006f210 con 0x7ffa70155c70 2024-03-04T09:33:57.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:57.771+0000 7ffa6ea00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffa54003f60 0x7ffa700a9470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:57.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:57.771+0000 7ffa6ea00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffa54003f60 0x7ffa700a9470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57216/0 (socket says 172.21.15.6:57216) 2024-03-04T09:33:57.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:57.771+0000 7ffa6ea00640 1 -- 172.21.15.6:0/3044440818 learned_addr learned my addr 172.21.15.6:0/3044440818 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:33:57.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:57.771+0000 7ffa6ea00640 1 -- 172.21.15.6:0/3044440818 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ffa70155c70 msgr2=0x7ffa70149d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:57.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:57.771+0000 7ffa6ea00640 1 --2- 172.21.15.6:0/3044440818 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ffa70155c70 0x7ffa70149d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:57.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:57.771+0000 7ffa6ea00640 1 -- 172.21.15.6:0/3044440818 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffa70158d30 msgr2=0x7ffa700b4290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:57.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:57.771+0000 7ffa6ea00640 1 --2- 172.21.15.6:0/3044440818 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffa70158d30 0x7ffa700b4290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:57.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:57.771+0000 7ffa6ea00640 1 -- 172.21.15.6:0/3044440818 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa701564f0 con 0x7ffa54003f60 2024-03-04T09:33:57.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:57.772+0000 7ffa6ea00640 1 --2- 172.21.15.6:0/3044440818 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffa54003f60 0x7ffa700a9470 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7ffa5800e860 tx=0x7ffa5800ec60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:33:57.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:57.772+0000 7ffa66a00640 1 -- 172.21.15.6:0/3044440818 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa5800c850 con 0x7ffa54003f60 2024-03-04T09:33:57.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:57.772+0000 7ffa66a00640 1 -- 172.21.15.6:0/3044440818 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa58018070 con 0x7ffa54003f60 2024-03-04T09:33:57.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:57.772+0000 7ffa6fe00640 1 -- 172.21.15.6:0/3044440818 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa700a9b70 con 0x7ffa54003f60 2024-03-04T09:33:57.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:57.772+0000 7ffa66a00640 1 -- 172.21.15.6:0/3044440818 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa5800cc30 con 0x7ffa54003f60 2024-03-04T09:33:57.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:57.772+0000 7ffa6fe00640 1 -- 172.21.15.6:0/3044440818 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa70158250 con 0x7ffa54003f60 2024-03-04T09:33:57.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:57.774+0000 7ffa6fe00640 1 -- 172.21.15.6:0/3044440818 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa7004f330 con 0x7ffa54003f60 2024-03-04T09:33:57.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:57.775+0000 7ffa66a00640 1 -- 172.21.15.6:0/3044440818 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ffa5801d460 con 0x7ffa54003f60 2024-03-04T09:33:57.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:57.775+0000 7ffa66a00640 1 --2- 172.21.15.6:0/3044440818 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ffa4c06b4d0 0x7ffa4c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:57.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:57.776+0000 7ffa66a00640 1 -- 172.21.15.6:0/3044440818 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7ffa5808a160 con 0x7ffa54003f60 2024-03-04T09:33:57.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:57.776+0000 7ffa6e000640 1 --2- 172.21.15.6:0/3044440818 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ffa4c06b4d0 0x7ffa4c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:57.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:57.777+0000 7ffa6e000640 1 --2- 172.21.15.6:0/3044440818 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ffa4c06b4d0 0x7ffa4c06d990 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7ffa60001750 tx=0x7ffa6002f000 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:33:57.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:57.781+0000 7ffa66a00640 1 -- 172.21.15.6:0/3044440818 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ffa701564f0 con 0x7ffa54003f60 2024-03-04T09:33:57.801 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-04T09:33:58.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:58.021+0000 7ffa6fe00640 1 -- 172.21.15.6:0/3044440818 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffa7006caf0 con 0x7ffa54003f60 2024-03-04T09:33:58.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:58.022+0000 7ffa66a00640 1 -- 172.21.15.6:0/3044440818 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3465 (secure 0 0 0) 0x7ffa58058f00 con 0x7ffa54003f60 2024-03-04T09:33:58.020 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:33:58.020 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:33:54.652601+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"laggy_since":"2024-03-04T09:33:54.652601+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":9}]} 2024-03-04T09:33:58.020 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 100 2024-03-04T09:33:58.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:58.025+0000 7ffa6fe00640 1 -- 172.21.15.6:0/3044440818 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ffa4c06b4d0 msgr2=0x7ffa4c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:58.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:58.025+0000 7ffa6fe00640 1 --2- 172.21.15.6:0/3044440818 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ffa4c06b4d0 0x7ffa4c06d990 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7ffa60001750 tx=0x7ffa6002f000 comp rx=0 tx=0).stop 2024-03-04T09:33:58.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:58.025+0000 7ffa6fe00640 1 -- 172.21.15.6:0/3044440818 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffa54003f60 msgr2=0x7ffa700a9470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:58.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:58.025+0000 7ffa6fe00640 1 --2- 172.21.15.6:0/3044440818 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffa54003f60 0x7ffa700a9470 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7ffa5800e860 tx=0x7ffa5800ec60 comp rx=0 tx=0).stop 2024-03-04T09:33:58.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:58.025+0000 7ffa6fe00640 1 -- 172.21.15.6:0/3044440818 shutdown_connections 2024-03-04T09:33:58.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:58.025+0000 7ffa6fe00640 1 --2- 172.21.15.6:0/3044440818 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffa70158d30 0x7ffa700b4290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:58.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:58.025+0000 7ffa6fe00640 1 --2- 172.21.15.6:0/3044440818 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ffa4c06b4d0 0x7ffa4c06d990 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:58.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:58.025+0000 7ffa6fe00640 1 --2- 172.21.15.6:0/3044440818 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ffa70155c70 0x7ffa70149d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:58.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:58.025+0000 7ffa6fe00640 1 --2- 172.21.15.6:0/3044440818 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffa54003f60 0x7ffa700a9470 unknown :-1 s=CLOSED pgs=1591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:58.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:58.025+0000 7ffa6fe00640 1 -- 172.21.15.6:0/3044440818 >> 172.21.15.6:0/3044440818 conn(0x7ffa700ccf70 msgr2=0x7ffa700717c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:33:58.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:58.025+0000 7ffa6fe00640 1 -- 172.21.15.6:0/3044440818 shutdown_connections 2024-03-04T09:33:58.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:58.026+0000 7ffa6fe00640 1 -- 172.21.15.6:0/3044440818 wait complete. 2024-03-04T09:33:58.242 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:32:57.159681+0000', 'modified': '2024-03-04T09:33:54.652601+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8214, 'mds_1': 6796}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6796': {'gid': 6796, 'name': 'd', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.146:6837/2458070840', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 2458070840}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 2458070840}]}, 'join_fscid': -1, 'laggy_since': '2024-03-04T09:33:54.652601+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8214': {'gid': 8214, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2446627517', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2446627517}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2446627517}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8214, 'qdb_cluster': [8214]} 2024-03-04T09:33:58.242 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6796:up:stopping 2024-03-04T09:33:59.243 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:33:59.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:59.482+0000 7f7014c00640 1 Processor -- start 2024-03-04T09:33:59.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:59.482+0000 7f7014c00640 1 -- start start 2024-03-04T09:33:59.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:59.484+0000 7f7014c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f701015e0b0 0x7f70100cf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:59.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:59.484+0000 7f7014c00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70100b3710 con 0x7f70100d4a10 2024-03-04T09:33:59.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:59.484+0000 7f7014c00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70100b39f0 con 0x7f70100d0e50 2024-03-04T09:33:59.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:59.484+0000 7f7014c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70100b09e0 con 0x7f701015e0b0 2024-03-04T09:33:59.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:59.484+0000 7f7007e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f70100d4a10 0x7f70100d0740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:55010/0 (socket says 172.21.15.6:55010) 2024-03-04T09:33:59.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:59.484+0000 7f7007e00640 1 -- 172.21.15.6:0/1451936605 learned_addr learned my addr 172.21.15.6:0/1451936605 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:33:59.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:59.485+0000 7f700f400640 1 --2- 172.21.15.6:0/1451936605 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f701015e0b0 0x7f70100cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:59.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:59.485+0000 7f700ea00640 1 -- 172.21.15.6:0/1451936605 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2272522821 0 0) 0x7f70100b3710 con 0x7f70100d4a10 2024-03-04T09:33:59.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:59.485+0000 7f700ea00640 1 -- 172.21.15.6:0/1451936605 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ff4003520 con 0x7f70100d4a10 2024-03-04T09:33:59.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:59.485+0000 7f700ea00640 1 -- 172.21.15.6:0/1451936605 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 606100580 0 0) 0x7f6ff4003520 con 0x7f70100d4a10 2024-03-04T09:33:59.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:59.485+0000 7f700ea00640 1 -- 172.21.15.6:0/1451936605 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f701015e0b0 msgr2=0x7f70100cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:33:59.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:59.485+0000 7f700ea00640 1 --2- 172.21.15.6:0/1451936605 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f701015e0b0 0x7f70100cf310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:59.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:59.485+0000 7f700ea00640 1 -- 172.21.15.6:0/1451936605 >> v1:172.21.15.146:6789/0 conn(0x7f70100d0e50 legacy=0x7f70100b44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:59.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:59.485+0000 7f700ea00640 1 -- 172.21.15.6:0/1451936605 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70100b3cd0 con 0x7f70100d4a10 2024-03-04T09:33:59.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:59.485+0000 7f700ea00640 1 -- 172.21.15.6:0/1451936605 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7f6ff8001670 con 0x7f70100d4a10 2024-03-04T09:33:59.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:59.485+0000 7f700ea00640 1 -- 172.21.15.6:0/1451936605 >> v1:172.21.15.6:6789/0 conn(0x7f70100d4a10 legacy=0x7f70100d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:59.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:59.485+0000 7f700ea00640 1 --2- 172.21.15.6:0/1451936605 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6ff40039e0 0x7f6ff4003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:59.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:59.486+0000 7f700ea00640 1 --2- 172.21.15.6:0/1451936605 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6ff4004800 0x7f6ff4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:59.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:59.486+0000 7f700ea00640 1 --2- 172.21.15.6:0/1451936605 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f701015e0b0 0x7f6ff4005250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:59.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:59.486+0000 7f700ea00640 1 -- 172.21.15.6:0/1451936605 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff400bb50 con 0x7f6ff40039e0 2024-03-04T09:33:59.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:59.486+0000 7f700ea00640 1 -- 172.21.15.6:0/1451936605 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff400bcd0 con 0x7f6ff4004800 2024-03-04T09:33:59.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:59.486+0000 7f700ea00640 1 -- 172.21.15.6:0/1451936605 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff400bed0 con 0x7f701015e0b0 2024-03-04T09:33:59.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:59.486+0000 7f700f400640 1 --2- 172.21.15.6:0/1451936605 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6ff40039e0 0x7f6ff4003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:59.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:59.486+0000 7f7007e00640 1 --2- 172.21.15.6:0/1451936605 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6ff4004800 0x7f6ff4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:59.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:59.486+0000 7f700fe00640 1 --2- 172.21.15.6:0/1451936605 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f701015e0b0 0x7f6ff4005250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:59.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:59.486+0000 7f700f400640 1 -- 172.21.15.6:0/1451936605 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f701015e0b0 msgr2=0x7f6ff4005250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:33:59.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:59.486+0000 7f700f400640 1 --2- 172.21.15.6:0/1451936605 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f701015e0b0 0x7f6ff4005250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:59.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:59.486+0000 7f700f400640 1 -- 172.21.15.6:0/1451936605 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6ff4004800 msgr2=0x7f6ff4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:33:59.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:59.486+0000 7f700f400640 1 --2- 172.21.15.6:0/1451936605 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6ff4004800 0x7f6ff4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:59.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:59.486+0000 7f700f400640 1 -- 172.21.15.6:0/1451936605 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ff4003520 con 0x7f6ff40039e0 2024-03-04T09:33:59.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:59.486+0000 7f7007e00640 1 --2- 172.21.15.6:0/1451936605 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6ff4004800 0x7f6ff4004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:33:59.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:59.487+0000 7f700f400640 1 --2- 172.21.15.6:0/1451936605 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6ff40039e0 0x7f6ff4003e00 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 crypto rx=0x7f700000ba30 tx=0x7f6ff4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40dffbbe5e924810 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:33:59.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:59.488+0000 7f700ea00640 1 -- 172.21.15.6:0/1451936605 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7000036020 con 0x7f6ff40039e0 2024-03-04T09:33:59.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:59.488+0000 7f700ea00640 1 -- 172.21.15.6:0/1451936605 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7000032640 con 0x7f6ff40039e0 2024-03-04T09:33:59.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:59.488+0000 7f700ea00640 1 -- 172.21.15.6:0/1451936605 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f700000b600 con 0x7f6ff40039e0 2024-03-04T09:33:59.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:59.488+0000 7f7014c00640 1 -- 172.21.15.6:0/1451936605 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6ff40039e0 msgr2=0x7f6ff4003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:59.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:59.488+0000 7f7014c00640 1 --2- 172.21.15.6:0/1451936605 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6ff40039e0 0x7f6ff4003e00 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 crypto rx=0x7f700000ba30 tx=0x7f6ff4005830 comp rx=0 tx=0).stop 2024-03-04T09:33:59.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:59.488+0000 7f700fe00640 1 -- 172.21.15.6:0/1451936605 reap_dead start 2024-03-04T09:33:59.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:59.489+0000 7f7014c00640 1 -- 172.21.15.6:0/1451936605 shutdown_connections 2024-03-04T09:33:59.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:59.489+0000 7f7014c00640 1 -- 172.21.15.6:0/1451936605 >> 172.21.15.6:0/1451936605 conn(0x7f70100d3d30 msgr2=0x7f70100d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:33:59.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:59.489+0000 7f7014c00640 1 -- 172.21.15.6:0/1451936605 shutdown_connections 2024-03-04T09:33:59.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:59.489+0000 7f7014c00640 1 -- 172.21.15.6:0/1451936605 wait complete. 2024-03-04T09:33:59.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:59.490+0000 7f7014c00640 1 Processor -- start 2024-03-04T09:33:59.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:59.491+0000 7f7014c00640 1 -- start start 2024-03-04T09:33:59.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:59.492+0000 7f7014c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6ff40039e0 0x7f701015b790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:59.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:59.492+0000 7f7014c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f70100cb880 0x7f70101f3bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:59.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:59.492+0000 7f7014c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f701015feb0 0x7f701006c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:59.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:59.492+0000 7f7014c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70100cfc60 con 0x7f70100cb880 2024-03-04T09:33:59.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:59.492+0000 7f7014c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f701015e490 con 0x7f6ff40039e0 2024-03-04T09:33:59.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:59.492+0000 7f7014c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70100cf310 con 0x7f701015feb0 2024-03-04T09:33:59.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:59.492+0000 7f7007e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f70100cb880 0x7f70101f3bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:59.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:59.492+0000 7f7007e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f70100cb880 0x7f70101f3bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55504/0 (socket says 172.21.15.6:55504) 2024-03-04T09:33:59.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:59.492+0000 7f7007e00640 1 -- 172.21.15.6:0/3590022439 learned_addr learned my addr 172.21.15.6:0/3590022439 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:33:59.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:59.492+0000 7f700f400640 1 --2- 172.21.15.6:0/3590022439 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6ff40039e0 0x7f701015b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:59.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:59.492+0000 7f700fe00640 1 --2- 172.21.15.6:0/3590022439 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f701015feb0 0x7f701006c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:59.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:59.492+0000 7f7007e00640 1 -- 172.21.15.6:0/3590022439 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f701015feb0 msgr2=0x7f701006c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:59.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:59.492+0000 7f7007e00640 1 --2- 172.21.15.6:0/3590022439 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f701015feb0 0x7f701006c9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:59.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:59.492+0000 7f7007e00640 1 -- 172.21.15.6:0/3590022439 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6ff40039e0 msgr2=0x7f701015b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:59.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:59.492+0000 7f7007e00640 1 --2- 172.21.15.6:0/3590022439 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6ff40039e0 0x7f701015b790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:59.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:59.493+0000 7f7007e00640 1 -- 172.21.15.6:0/3590022439 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7010160620 con 0x7f70100cb880 2024-03-04T09:33:59.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:59.493+0000 7f7007e00640 1 --2- 172.21.15.6:0/3590022439 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f70100cb880 0x7f70101f3bc0 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x7f6ff800d820 tx=0x7f6ff800dce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:33:59.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:59.493+0000 7f7007400640 1 -- 172.21.15.6:0/3590022439 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ff8019070 con 0x7f70100cb880 2024-03-04T09:33:59.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:59.493+0000 7f7007400640 1 -- 172.21.15.6:0/3590022439 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ff80074e0 con 0x7f70100cb880 2024-03-04T09:33:59.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:59.493+0000 7f7007400640 1 -- 172.21.15.6:0/3590022439 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ff80049e0 con 0x7f70100cb880 2024-03-04T09:33:59.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:59.493+0000 7f7014c00640 1 -- 172.21.15.6:0/3590022439 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7010160900 con 0x7f70100cb880 2024-03-04T09:33:59.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:59.493+0000 7f7014c00640 1 -- 172.21.15.6:0/3590022439 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7010160ec0 con 0x7f70100cb880 2024-03-04T09:33:59.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:59.495+0000 7f7014c00640 1 -- 172.21.15.6:0/3590022439 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f701004f330 con 0x7f70100cb880 2024-03-04T09:33:59.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:59.495+0000 7f7007400640 1 -- 172.21.15.6:0/3590022439 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6ff8004b80 con 0x7f70100cb880 2024-03-04T09:33:59.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:59.497+0000 7f7007400640 1 --2- 172.21.15.6:0/3590022439 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f6fec06b4d0 0x7f6fec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:33:59.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:59.497+0000 7f7007400640 1 -- 172.21.15.6:0/3590022439 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f6ff8089cf0 con 0x7f70100cb880 2024-03-04T09:33:59.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:59.498+0000 7f700f400640 1 --2- 172.21.15.6:0/3590022439 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f6fec06b4d0 0x7f6fec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:33:59.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:59.499+0000 7f700f400640 1 --2- 172.21.15.6:0/3590022439 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f6fec06b4d0 0x7f6fec06d990 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f7000037000 tx=0x7f700003d040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:33:59.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:59.502+0000 7f7007400640 1 -- 172.21.15.6:0/3590022439 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6ff8058b50 con 0x7f70100cb880 2024-03-04T09:33:59.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:59.739+0000 7f7014c00640 1 -- 172.21.15.6:0/3590022439 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f70100ce7c0 con 0x7f70100cb880 2024-03-04T09:33:59.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:59.740+0000 7f7007400640 1 -- 172.21.15.6:0/3590022439 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3465 (secure 0 0 0) 0x7f6ff8058450 con 0x7f70100cb880 2024-03-04T09:33:59.738 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:33:59.738 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:33:54.652601+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"laggy_since":"2024-03-04T09:33:54.652601+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":9}]} 2024-03-04T09:33:59.738 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 100 2024-03-04T09:33:59.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:59.742+0000 7f7014c00640 1 -- 172.21.15.6:0/3590022439 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f6fec06b4d0 msgr2=0x7f6fec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:59.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:59.742+0000 7f7014c00640 1 --2- 172.21.15.6:0/3590022439 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f6fec06b4d0 0x7f6fec06d990 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f7000037000 tx=0x7f700003d040 comp rx=0 tx=0).stop 2024-03-04T09:33:59.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:59.742+0000 7f7014c00640 1 -- 172.21.15.6:0/3590022439 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f70100cb880 msgr2=0x7f70101f3bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:33:59.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:59.742+0000 7f7014c00640 1 --2- 172.21.15.6:0/3590022439 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f70100cb880 0x7f70101f3bc0 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x7f6ff800d820 tx=0x7f6ff800dce0 comp rx=0 tx=0).stop 2024-03-04T09:33:59.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:59.743+0000 7f7014c00640 1 -- 172.21.15.6:0/3590022439 shutdown_connections 2024-03-04T09:33:59.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:59.743+0000 7f7014c00640 1 --2- 172.21.15.6:0/3590022439 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f6fec06b4d0 0x7f6fec06d990 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:59.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:59.743+0000 7f7014c00640 1 --2- 172.21.15.6:0/3590022439 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f701015feb0 0x7f701006c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:59.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:59.743+0000 7f7014c00640 1 --2- 172.21.15.6:0/3590022439 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f70100cb880 0x7f70101f3bc0 unknown :-1 s=CLOSED pgs=1594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:59.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:59.743+0000 7f7014c00640 1 --2- 172.21.15.6:0/3590022439 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6ff40039e0 0x7f701015b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:33:59.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:59.743+0000 7f7014c00640 1 -- 172.21.15.6:0/3590022439 >> 172.21.15.6:0/3590022439 conn(0x7f70100d3d30 msgr2=0x7f70100b09e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:33:59.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:59.743+0000 7f7014c00640 1 -- 172.21.15.6:0/3590022439 shutdown_connections 2024-03-04T09:33:59.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:33:59.743+0000 7f7014c00640 1 -- 172.21.15.6:0/3590022439 wait complete. 2024-03-04T09:33:59.803 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-04T09:33:59.961 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:32:57.159681+0000', 'modified': '2024-03-04T09:33:54.652601+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8214, 'mds_1': 6796}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6796': {'gid': 6796, 'name': 'd', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.146:6837/2458070840', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 2458070840}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 2458070840}]}, 'join_fscid': -1, 'laggy_since': '2024-03-04T09:33:54.652601+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8214': {'gid': 8214, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2446627517', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2446627517}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2446627517}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8214, 'qdb_cluster': [8214]} 2024-03-04T09:33:59.961 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6796:up:stopping 2024-03-04T09:34:00.962 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:34:01.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:01.188+0000 7f77ece00640 1 Processor -- start 2024-03-04T09:34:01.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:01.188+0000 7f77ece00640 1 -- start start 2024-03-04T09:34:01.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:01.190+0000 7f77ece00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f77e8162420 0x7f77e806afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:01.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:01.190+0000 7f77ece00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f77e806b4f0 0x7f77e80d0910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:01.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:01.190+0000 7f77ece00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77e8147bc0 con 0x7f77e81477e0 2024-03-04T09:34:01.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:01.190+0000 7f77ece00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77e8161f90 con 0x7f77e806b4f0 2024-03-04T09:34:01.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:01.190+0000 7f77ece00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77e806b920 con 0x7f77e8162420 2024-03-04T09:34:01.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:01.190+0000 7f77e7e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f77e806b4f0 0x7f77e80d0910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:01.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:01.190+0000 7f77e7400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f77e81477e0 0x7f77e806ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:55022/0 (socket says 172.21.15.6:55022) 2024-03-04T09:34:01.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:01.190+0000 7f77e7e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f77e806b4f0 0x7f77e80d0910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55518/0 (socket says 172.21.15.6:55518) 2024-03-04T09:34:01.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:01.190+0000 7f77e7400640 1 -- 172.21.15.6:0/331754809 learned_addr learned my addr 172.21.15.6:0/331754809 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:34:01.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:01.191+0000 7f77e6a00640 1 --2- 172.21.15.6:0/331754809 >> v2:172.21.15.146:3300/0 conn(0x7f77e8162420 0x7f77e806afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:01.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:01.191+0000 7f77e7e00640 1 -- 172.21.15.6:0/331754809 >> v2:172.21.15.146:3300/0 conn(0x7f77e8162420 msgr2=0x7f77e806afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:34:01.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:01.191+0000 7f77e7e00640 1 --2- 172.21.15.6:0/331754809 >> v2:172.21.15.146:3300/0 conn(0x7f77e8162420 0x7f77e806afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:01.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:01.191+0000 7f77e7e00640 1 -- 172.21.15.6:0/331754809 >> v1:172.21.15.6:6789/0 conn(0x7f77e81477e0 legacy=0x7f77e806ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:01.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:01.191+0000 7f77e7e00640 1 -- 172.21.15.6:0/331754809 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77e80b3d70 con 0x7f77e806b4f0 2024-03-04T09:34:01.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:01.191+0000 7f77e6000640 1 -- 172.21.15.6:0/331754809 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 74671342 0 0) 0x7f77e8147bc0 con 0x7f77e81477e0 2024-03-04T09:34:01.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:01.192+0000 7f77e7e00640 1 --2- 172.21.15.6:0/331754809 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f77e806b4f0 0x7f77e80d0910 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7f77e80b6030 tx=0x7f77e002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b4ec533b3b107f server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:34:01.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:01.192+0000 7f77e6000640 1 -- 172.21.15.6:0/331754809 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77e0039070 con 0x7f77e806b4f0 2024-03-04T09:34:01.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:01.192+0000 7f77e6000640 1 -- 172.21.15.6:0/331754809 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77e002cbf0 con 0x7f77e806b4f0 2024-03-04T09:34:01.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:01.193+0000 7f77e6000640 1 -- 172.21.15.6:0/331754809 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77e0034440 con 0x7f77e806b4f0 2024-03-04T09:34:01.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:01.193+0000 7f77ece00640 1 -- 172.21.15.6:0/331754809 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f77e806b4f0 msgr2=0x7f77e80d0910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:01.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:01.193+0000 7f77ece00640 1 --2- 172.21.15.6:0/331754809 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f77e806b4f0 0x7f77e80d0910 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7f77e80b6030 tx=0x7f77e002be40 comp rx=0 tx=0).stop 2024-03-04T09:34:01.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:01.193+0000 7f77ece00640 1 -- 172.21.15.6:0/331754809 shutdown_connections 2024-03-04T09:34:01.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:01.193+0000 7f77ece00640 1 --2- 172.21.15.6:0/331754809 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f77e806b4f0 0x7f77e80d0910 unknown :-1 s=CLOSED pgs=1596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:01.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:01.193+0000 7f77ece00640 1 --2- 172.21.15.6:0/331754809 >> v2:172.21.15.146:3300/0 conn(0x7f77e8162420 0x7f77e806afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:01.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:01.193+0000 7f77ece00640 1 -- 172.21.15.6:0/331754809 >> 172.21.15.6:0/331754809 conn(0x7f77e806c0c0 msgr2=0x7f77e806c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:34:01.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:01.193+0000 7f77ece00640 1 -- 172.21.15.6:0/331754809 shutdown_connections 2024-03-04T09:34:01.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:01.193+0000 7f77ece00640 1 -- 172.21.15.6:0/331754809 wait complete. 2024-03-04T09:34:01.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:01.196+0000 7f77ece00640 1 Processor -- start 2024-03-04T09:34:01.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:01.196+0000 7f77ece00640 1 -- start start 2024-03-04T09:34:01.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:01.198+0000 7f77ece00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f77e806b4f0 0x7f77e813e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:01.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:01.198+0000 7f77ece00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f77e81477e0 0x7f77e8160690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:01.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:01.198+0000 7f77ece00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f77e8162420 0x7f77e8140900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:01.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:01.198+0000 7f77ece00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77e8161c20 con 0x7f77e806b4f0 2024-03-04T09:34:01.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:01.198+0000 7f77ece00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77e80b4330 con 0x7f77e81477e0 2024-03-04T09:34:01.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:01.198+0000 7f77ece00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77e81628a0 con 0x7f77e8162420 2024-03-04T09:34:01.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:01.198+0000 7f77e7400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f77e806b4f0 0x7f77e813e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:01.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:01.198+0000 7f77e7400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f77e806b4f0 0x7f77e813e900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55534/0 (socket says 172.21.15.6:55534) 2024-03-04T09:34:01.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:01.198+0000 7f77e7400640 1 -- 172.21.15.6:0/1644375910 learned_addr learned my addr 172.21.15.6:0/1644375910 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:34:01.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:01.199+0000 7f77e7400640 1 -- 172.21.15.6:0/1644375910 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f77e8162420 msgr2=0x7f77e8140900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:01.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:01.199+0000 7f77e7400640 1 --2- 172.21.15.6:0/1644375910 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f77e8162420 0x7f77e8140900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:01.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:01.199+0000 7f77e6a00640 1 --2- 172.21.15.6:0/1644375910 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f77e81477e0 0x7f77e8160690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:01.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:01.199+0000 7f77e7400640 1 -- 172.21.15.6:0/1644375910 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f77e81477e0 msgr2=0x7f77e8160690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:01.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:01.199+0000 7f77e7400640 1 --2- 172.21.15.6:0/1644375910 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f77e81477e0 0x7f77e8160690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:01.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:01.199+0000 7f77e7400640 1 -- 172.21.15.6:0/1644375910 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77e815f010 con 0x7f77e806b4f0 2024-03-04T09:34:01.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:01.199+0000 7f77e7400640 1 --2- 172.21.15.6:0/1644375910 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f77e806b4f0 0x7f77e813e900 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7f77d800c400 tx=0x7f77d800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:34:01.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:01.199+0000 7f77df400640 1 -- 172.21.15.6:0/1644375910 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77d8017070 con 0x7f77e806b4f0 2024-03-04T09:34:01.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:01.199+0000 7f77df400640 1 -- 172.21.15.6:0/1644375910 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77d8005ce0 con 0x7f77e806b4f0 2024-03-04T09:34:01.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:01.199+0000 7f77ece00640 1 -- 172.21.15.6:0/1644375910 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77e813ee40 con 0x7f77e806b4f0 2024-03-04T09:34:01.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:01.199+0000 7f77df400640 1 -- 172.21.15.6:0/1644375910 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77d8007310 con 0x7f77e806b4f0 2024-03-04T09:34:01.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:01.199+0000 7f77ece00640 1 -- 172.21.15.6:0/1644375910 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77e815f590 con 0x7f77e806b4f0 2024-03-04T09:34:01.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:01.201+0000 7f77ece00640 1 -- 172.21.15.6:0/1644375910 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77e804f330 con 0x7f77e806b4f0 2024-03-04T09:34:01.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:01.202+0000 7f77df400640 1 -- 172.21.15.6:0/1644375910 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f77d801a070 con 0x7f77e806b4f0 2024-03-04T09:34:01.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:01.202+0000 7f77df400640 1 --2- 172.21.15.6:0/1644375910 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f77c806b4d0 0x7f77c806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:01.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:01.203+0000 7f77df400640 1 -- 172.21.15.6:0/1644375910 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f77d8089680 con 0x7f77e806b4f0 2024-03-04T09:34:01.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:01.203+0000 7f77e6a00640 1 --2- 172.21.15.6:0/1644375910 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f77c806b4d0 0x7f77c806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:01.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:01.204+0000 7f77e6a00640 1 --2- 172.21.15.6:0/1644375910 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f77c806b4d0 0x7f77c806d990 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f77e80cd710 tx=0x7f77d4008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:34:01.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:01.212+0000 7f77df400640 1 -- 172.21.15.6:0/1644375910 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f77d80584c0 con 0x7f77e806b4f0 2024-03-04T09:34:01.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:01.451+0000 7f77ece00640 1 -- 172.21.15.6:0/1644375910 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f77e80724b0 con 0x7f77e806b4f0 2024-03-04T09:34:01.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:01.452+0000 7f77df400640 1 -- 172.21.15.6:0/1644375910 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3465 (secure 0 0 0) 0x7f77d8057dc0 con 0x7f77e806b4f0 2024-03-04T09:34:01.449 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:34:01.449 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:33:54.652601+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"laggy_since":"2024-03-04T09:33:54.652601+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":9}]} 2024-03-04T09:34:01.449 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 100 2024-03-04T09:34:01.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:01.454+0000 7f77ece00640 1 -- 172.21.15.6:0/1644375910 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f77c806b4d0 msgr2=0x7f77c806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:01.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:01.454+0000 7f77ece00640 1 --2- 172.21.15.6:0/1644375910 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f77c806b4d0 0x7f77c806d990 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f77e80cd710 tx=0x7f77d4008040 comp rx=0 tx=0).stop 2024-03-04T09:34:01.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:01.454+0000 7f77ece00640 1 -- 172.21.15.6:0/1644375910 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f77e806b4f0 msgr2=0x7f77e813e900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:01.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:01.454+0000 7f77ece00640 1 --2- 172.21.15.6:0/1644375910 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f77e806b4f0 0x7f77e813e900 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7f77d800c400 tx=0x7f77d800c8c0 comp rx=0 tx=0).stop 2024-03-04T09:34:01.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:01.454+0000 7f77ece00640 1 -- 172.21.15.6:0/1644375910 shutdown_connections 2024-03-04T09:34:01.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:01.454+0000 7f77ece00640 1 --2- 172.21.15.6:0/1644375910 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f77c806b4d0 0x7f77c806d990 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:01.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:01.454+0000 7f77ece00640 1 --2- 172.21.15.6:0/1644375910 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f77e8162420 0x7f77e8140900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:01.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:01.454+0000 7f77ece00640 1 --2- 172.21.15.6:0/1644375910 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f77e81477e0 0x7f77e8160690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:01.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:01.454+0000 7f77ece00640 1 --2- 172.21.15.6:0/1644375910 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f77e806b4f0 0x7f77e813e900 unknown :-1 s=CLOSED pgs=1597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:01.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:01.454+0000 7f77ece00640 1 -- 172.21.15.6:0/1644375910 >> 172.21.15.6:0/1644375910 conn(0x7f77e806c0c0 msgr2=0x7f77e806ef20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:34:01.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:01.455+0000 7f77ece00640 1 -- 172.21.15.6:0/1644375910 shutdown_connections 2024-03-04T09:34:01.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:01.455+0000 7f77ece00640 1 -- 172.21.15.6:0/1644375910 wait complete. 2024-03-04T09:34:01.673 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:32:57.159681+0000', 'modified': '2024-03-04T09:33:54.652601+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8214, 'mds_1': 6796}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6796': {'gid': 6796, 'name': 'd', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.146:6837/2458070840', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 2458070840}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 2458070840}]}, 'join_fscid': -1, 'laggy_since': '2024-03-04T09:33:54.652601+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8214': {'gid': 8214, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2446627517', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2446627517}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2446627517}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8214, 'qdb_cluster': [8214]} 2024-03-04T09:34:01.674 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6796:up:stopping 2024-03-04T09:34:01.806 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-04T09:34:02.675 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:34:02.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:02.903+0000 7f7171000640 1 Processor -- start 2024-03-04T09:34:02.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:02.903+0000 7f7171000640 1 -- start start 2024-03-04T09:34:02.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:02.904+0000 7f7171000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f716c0af1a0 0x7f716c15c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:02.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:02.904+0000 7f7171000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f716c0cdc20 0x7f716c0b2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:02.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:02.904+0000 7f7171000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f716c15cc40 con 0x7f716c0af580 2024-03-04T09:34:02.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:02.904+0000 7f7171000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f716c15cf20 con 0x7f716c0cdc20 2024-03-04T09:34:02.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:02.905+0000 7f7171000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f716c15d200 con 0x7f716c0af1a0 2024-03-04T09:34:02.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:02.905+0000 7f716a800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f716c0cdc20 0x7f716c0b2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:02.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:02.905+0000 7f716a800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f716c0cdc20 0x7f716c0b2dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55546/0 (socket says 172.21.15.6:55546) 2024-03-04T09:34:02.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:02.905+0000 7f716a800640 1 -- 172.21.15.6:0/975154857 learned_addr learned my addr 172.21.15.6:0/975154857 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:34:02.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:02.905+0000 7f7169e00640 1 --2- 172.21.15.6:0/975154857 >> v2:172.21.15.146:3300/0 conn(0x7f716c0af1a0 0x7f716c15c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:02.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:02.906+0000 7f716a800640 1 -- 172.21.15.6:0/975154857 >> v2:172.21.15.146:3300/0 conn(0x7f716c0af1a0 msgr2=0x7f716c15c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:34:02.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:02.906+0000 7f716a800640 1 --2- 172.21.15.6:0/975154857 >> v2:172.21.15.146:3300/0 conn(0x7f716c0af1a0 0x7f716c15c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:02.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:02.906+0000 7f716a800640 1 -- 172.21.15.6:0/975154857 >> v1:172.21.15.146:6789/0 conn(0x7f716c0af580 legacy=0x7f716c0b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:02.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:02.906+0000 7f716a800640 1 -- 172.21.15.6:0/975154857 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f716c15d4e0 con 0x7f716c0cdc20 2024-03-04T09:34:02.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:02.907+0000 7f716a800640 1 --2- 172.21.15.6:0/975154857 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f716c0cdc20 0x7f716c0b2dd0 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 crypto rx=0x7f716c074960 tx=0x7f716402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c9fbf9e253beebe server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:34:02.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:02.907+0000 7f7168a00640 1 -- 172.21.15.6:0/975154857 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7164035020 con 0x7f716c0cdc20 2024-03-04T09:34:02.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:02.907+0000 7f7168a00640 1 -- 172.21.15.6:0/975154857 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7164031870 con 0x7f716c0cdc20 2024-03-04T09:34:02.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:02.907+0000 7f7168a00640 1 -- 172.21.15.6:0/975154857 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7164031c40 con 0x7f716c0cdc20 2024-03-04T09:34:02.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:02.908+0000 7f7171000640 1 -- 172.21.15.6:0/975154857 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f716c0cdc20 msgr2=0x7f716c0b2dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:02.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:02.908+0000 7f7171000640 1 --2- 172.21.15.6:0/975154857 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f716c0cdc20 0x7f716c0b2dd0 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 crypto rx=0x7f716c074960 tx=0x7f716402b910 comp rx=0 tx=0).stop 2024-03-04T09:34:02.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:02.908+0000 7f7171000640 1 -- 172.21.15.6:0/975154857 shutdown_connections 2024-03-04T09:34:02.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:02.908+0000 7f7171000640 1 --2- 172.21.15.6:0/975154857 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f716c0cdc20 0x7f716c0b2dd0 unknown :-1 s=CLOSED pgs=1598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:02.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:02.908+0000 7f7171000640 1 --2- 172.21.15.6:0/975154857 >> v2:172.21.15.146:3300/0 conn(0x7f716c0af1a0 0x7f716c15c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:02.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:02.908+0000 7f7171000640 1 -- 172.21.15.6:0/975154857 >> 172.21.15.6:0/975154857 conn(0x7f716c06afb0 msgr2=0x7f716c06b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:34:02.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:02.908+0000 7f7171000640 1 -- 172.21.15.6:0/975154857 shutdown_connections 2024-03-04T09:34:02.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:02.908+0000 7f7171000640 1 -- 172.21.15.6:0/975154857 wait complete. 2024-03-04T09:34:02.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:02.911+0000 7f7171000640 1 Processor -- start 2024-03-04T09:34:02.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:02.911+0000 7f7171000640 1 -- start start 2024-03-04T09:34:02.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:02.913+0000 7f7171000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f716c0af1a0 0x7f716c0b0ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:02.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:02.913+0000 7f7171000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f716c0af580 0x7f716c14c040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:02.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:02.913+0000 7f7171000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f716c0cdc20 0x7f716c0c9150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:02.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:02.913+0000 7f7171000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f716c0b36a0 con 0x7f716c0af1a0 2024-03-04T09:34:02.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:02.913+0000 7f7171000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f716c0b3520 con 0x7f716c0af580 2024-03-04T09:34:02.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:02.913+0000 7f7171000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f716c0b3820 con 0x7f716c0cdc20 2024-03-04T09:34:02.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:02.913+0000 7f7169e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f716c0af1a0 0x7f716c0b0ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:02.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:02.913+0000 7f7169e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f716c0af1a0 0x7f716c0b0ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55554/0 (socket says 172.21.15.6:55554) 2024-03-04T09:34:02.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:02.913+0000 7f7169e00640 1 -- 172.21.15.6:0/3440006834 learned_addr learned my addr 172.21.15.6:0/3440006834 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:34:02.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:02.914+0000 7f716a800640 1 --2- 172.21.15.6:0/3440006834 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f716c0cdc20 0x7f716c0c9150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:02.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:02.914+0000 7f7169e00640 1 -- 172.21.15.6:0/3440006834 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f716c0cdc20 msgr2=0x7f716c0c9150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:02.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:02.914+0000 7f7169400640 1 --2- 172.21.15.6:0/3440006834 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f716c0af580 0x7f716c14c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:02.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:02.914+0000 7f7169e00640 1 --2- 172.21.15.6:0/3440006834 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f716c0cdc20 0x7f716c0c9150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:02.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:02.914+0000 7f7169e00640 1 -- 172.21.15.6:0/3440006834 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f716c0af580 msgr2=0x7f716c14c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:02.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:02.914+0000 7f7169e00640 1 --2- 172.21.15.6:0/3440006834 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f716c0af580 0x7f716c14c040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:02.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:02.914+0000 7f7169e00640 1 -- 172.21.15.6:0/3440006834 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f716c0c98c0 con 0x7f716c0af1a0 2024-03-04T09:34:02.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:02.914+0000 7f7169e00640 1 --2- 172.21.15.6:0/3440006834 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f716c0af1a0 0x7f716c0b0ea0 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 crypto rx=0x7f715c007f20 tx=0x7f715c00e570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:34:02.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:02.914+0000 7f7162000640 1 -- 172.21.15.6:0/3440006834 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f715c014620 con 0x7f716c0af1a0 2024-03-04T09:34:02.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:02.915+0000 7f7162000640 1 -- 172.21.15.6:0/3440006834 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f715c019070 con 0x7f716c0af1a0 2024-03-04T09:34:02.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:02.915+0000 7f7162000640 1 -- 172.21.15.6:0/3440006834 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f715c014a00 con 0x7f716c0af1a0 2024-03-04T09:34:02.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:02.915+0000 7f7171000640 1 -- 172.21.15.6:0/3440006834 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f716c0c9ba0 con 0x7f716c0af1a0 2024-03-04T09:34:02.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:02.915+0000 7f7171000640 1 -- 172.21.15.6:0/3440006834 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f716c0ca160 con 0x7f716c0af1a0 2024-03-04T09:34:02.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:02.916+0000 7f7171000640 1 -- 172.21.15.6:0/3440006834 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f716c069510 con 0x7f716c0af1a0 2024-03-04T09:34:02.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:02.918+0000 7f7162000640 1 -- 172.21.15.6:0/3440006834 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f715c014ba0 con 0x7f716c0af1a0 2024-03-04T09:34:02.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:02.919+0000 7f7162000640 1 --2- 172.21.15.6:0/3440006834 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f714806b4d0 0x7f714806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:02.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:02.919+0000 7f7162000640 1 -- 172.21.15.6:0/3440006834 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f715c08ab50 con 0x7f716c0af1a0 2024-03-04T09:34:02.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:02.919+0000 7f7169400640 1 --2- 172.21.15.6:0/3440006834 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f714806b4d0 0x7f714806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:02.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:02.921+0000 7f7169400640 1 --2- 172.21.15.6:0/3440006834 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f714806b4d0 0x7f714806d990 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f716c072bb0 tx=0x7f7154009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:34:02.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:02.924+0000 7f7162000640 1 -- 172.21.15.6:0/3440006834 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f715c059990 con 0x7f716c0af1a0 2024-03-04T09:34:03.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:03.161+0000 7f7171000640 1 -- 172.21.15.6:0/3440006834 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f716c04f3a0 con 0x7f716c0af1a0 2024-03-04T09:34:03.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:03.163+0000 7f7162000640 1 -- 172.21.15.6:0/3440006834 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3465 (secure 0 0 0) 0x7f715c059290 con 0x7f716c0af1a0 2024-03-04T09:34:03.160 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:34:03.160 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:33:54.652601+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"laggy_since":"2024-03-04T09:33:54.652601+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":9}]} 2024-03-04T09:34:03.160 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 100 2024-03-04T09:34:03.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:03.165+0000 7f7171000640 1 -- 172.21.15.6:0/3440006834 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f714806b4d0 msgr2=0x7f714806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:03.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:03.165+0000 7f7171000640 1 --2- 172.21.15.6:0/3440006834 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f714806b4d0 0x7f714806d990 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f716c072bb0 tx=0x7f7154009040 comp rx=0 tx=0).stop 2024-03-04T09:34:03.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:03.166+0000 7f7171000640 1 -- 172.21.15.6:0/3440006834 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f716c0af1a0 msgr2=0x7f716c0b0ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:03.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:03.166+0000 7f7171000640 1 --2- 172.21.15.6:0/3440006834 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f716c0af1a0 0x7f716c0b0ea0 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 crypto rx=0x7f715c007f20 tx=0x7f715c00e570 comp rx=0 tx=0).stop 2024-03-04T09:34:03.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:03.166+0000 7f7171000640 1 -- 172.21.15.6:0/3440006834 shutdown_connections 2024-03-04T09:34:03.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:03.166+0000 7f7171000640 1 --2- 172.21.15.6:0/3440006834 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f714806b4d0 0x7f714806d990 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:03.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:03.166+0000 7f7171000640 1 --2- 172.21.15.6:0/3440006834 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f716c0cdc20 0x7f716c0c9150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:03.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:03.166+0000 7f7171000640 1 --2- 172.21.15.6:0/3440006834 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f716c0af580 0x7f716c14c040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:03.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:03.166+0000 7f7171000640 1 --2- 172.21.15.6:0/3440006834 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f716c0af1a0 0x7f716c0b0ea0 unknown :-1 s=CLOSED pgs=1599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:03.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:03.166+0000 7f7171000640 1 -- 172.21.15.6:0/3440006834 >> 172.21.15.6:0/3440006834 conn(0x7f716c06afb0 msgr2=0x7f716c15c060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:34:03.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:03.166+0000 7f7171000640 1 -- 172.21.15.6:0/3440006834 shutdown_connections 2024-03-04T09:34:03.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:03.166+0000 7f7171000640 1 -- 172.21.15.6:0/3440006834 wait complete. 2024-03-04T09:34:03.388 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:32:57.159681+0000', 'modified': '2024-03-04T09:33:54.652601+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8214, 'mds_1': 6796}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6796': {'gid': 6796, 'name': 'd', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.146:6837/2458070840', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 2458070840}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 2458070840}]}, 'join_fscid': -1, 'laggy_since': '2024-03-04T09:33:54.652601+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8214': {'gid': 8214, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2446627517', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2446627517}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2446627517}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8214, 'qdb_cluster': [8214]} 2024-03-04T09:34:03.388 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6796:up:stopping 2024-03-04T09:34:03.808 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-04T09:34:04.389 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:34:04.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:04.616+0000 7efcc6600640 1 Processor -- start 2024-03-04T09:34:04.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:04.616+0000 7efcc6600640 1 -- start start 2024-03-04T09:34:04.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:04.618+0000 7efcc6600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efcc0147400 0x7efcc006afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:04.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:04.618+0000 7efcc6600640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcc0161ae0 con 0x7efcc01625a0 2024-03-04T09:34:04.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:04.618+0000 7efcc6600640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcc015f080 con 0x7efcc015ea80 2024-03-04T09:34:04.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:04.618+0000 7efcc6600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcc0162110 con 0x7efcc0147400 2024-03-04T09:34:04.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:04.618+0000 7efcbfe00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7efcc01625a0 0x7efcc015e370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:55026/0 (socket says 172.21.15.6:55026) 2024-03-04T09:34:04.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:04.618+0000 7efcbfe00640 1 -- 172.21.15.6:0/2730455514 learned_addr learned my addr 172.21.15.6:0/2730455514 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:34:04.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:04.619+0000 7efcc5200640 1 --2- 172.21.15.6:0/2730455514 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efcc0147400 0x7efcc006afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:04.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:04.619+0000 7efcbf400640 1 -- 172.21.15.6:0/2730455514 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3276105301 0 0) 0x7efcc0161ae0 con 0x7efcc01625a0 2024-03-04T09:34:04.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:04.619+0000 7efcbf400640 1 -- 172.21.15.6:0/2730455514 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efca8003520 con 0x7efcc01625a0 2024-03-04T09:34:04.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:04.619+0000 7efcbf400640 1 -- 172.21.15.6:0/2730455514 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2081547994 0 0) 0x7efca8003520 con 0x7efcc01625a0 2024-03-04T09:34:04.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:04.619+0000 7efcbf400640 1 -- 172.21.15.6:0/2730455514 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efcc0147400 msgr2=0x7efcc006afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:34:04.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:04.619+0000 7efcbf400640 1 --2- 172.21.15.6:0/2730455514 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efcc0147400 0x7efcc006afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:04.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:04.619+0000 7efcbf400640 1 -- 172.21.15.6:0/2730455514 >> v1:172.21.15.146:6789/0 conn(0x7efcc015ea80 legacy=0x7efcc006e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:04.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:04.620+0000 7efcbf400640 1 -- 172.21.15.6:0/2730455514 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcc0161da0 con 0x7efcc01625a0 2024-03-04T09:34:04.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:04.620+0000 7efcbf400640 1 -- 172.21.15.6:0/2730455514 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7efcac001670 con 0x7efcc01625a0 2024-03-04T09:34:04.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:04.620+0000 7efcbf400640 1 -- 172.21.15.6:0/2730455514 >> v1:172.21.15.6:6789/0 conn(0x7efcc01625a0 legacy=0x7efcc015e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:04.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:04.620+0000 7efcbf400640 1 --2- 172.21.15.6:0/2730455514 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efca80039e0 0x7efca8003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:04.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:04.620+0000 7efcbf400640 1 --2- 172.21.15.6:0/2730455514 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efca8004800 0x7efca8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:04.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:04.620+0000 7efcbf400640 1 --2- 172.21.15.6:0/2730455514 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efcc0147400 0x7efca80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:04.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:04.620+0000 7efcbf400640 1 -- 172.21.15.6:0/2730455514 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcc0162110 con 0x7efca80039e0 2024-03-04T09:34:04.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:04.620+0000 7efcbf400640 1 -- 172.21.15.6:0/2730455514 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efca800ba70 con 0x7efca8004800 2024-03-04T09:34:04.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:04.620+0000 7efcbf400640 1 -- 172.21.15.6:0/2730455514 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efca800bc70 con 0x7efcc0147400 2024-03-04T09:34:04.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:04.620+0000 7efcc5200640 1 --2- 172.21.15.6:0/2730455514 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efca80039e0 0x7efca8003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:04.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:04.620+0000 7efcc5200640 1 -- 172.21.15.6:0/2730455514 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efcc0147400 msgr2=0x7efca80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:34:04.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:04.620+0000 7efcc5200640 1 --2- 172.21.15.6:0/2730455514 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efcc0147400 0x7efca80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:04.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:04.620+0000 7efcc5200640 1 -- 172.21.15.6:0/2730455514 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efca8004800 msgr2=0x7efca8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:34:04.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:04.620+0000 7efcc5200640 1 --2- 172.21.15.6:0/2730455514 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efca8004800 0x7efca8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:04.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:04.620+0000 7efcc5200640 1 -- 172.21.15.6:0/2730455514 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efca8003520 con 0x7efca80039e0 2024-03-04T09:34:04.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:04.622+0000 7efcc5200640 1 --2- 172.21.15.6:0/2730455514 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efca80039e0 0x7efca8003e00 secure :-1 s=READY pgs=1601 cs=0 l=1 rev1=1 crypto rx=0x7efcb4009a90 tx=0x7efca8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c43eb14cceec640 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:34:04.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:04.622+0000 7efcbf400640 1 -- 172.21.15.6:0/2730455514 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcb4039070 con 0x7efca80039e0 2024-03-04T09:34:04.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:04.622+0000 7efcbf400640 1 -- 172.21.15.6:0/2730455514 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcb402c6b0 con 0x7efca80039e0 2024-03-04T09:34:04.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:04.622+0000 7efcbf400640 1 -- 172.21.15.6:0/2730455514 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcb4009690 con 0x7efca80039e0 2024-03-04T09:34:04.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:04.623+0000 7efcc6600640 1 -- 172.21.15.6:0/2730455514 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efca80039e0 msgr2=0x7efca8003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:04.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:04.623+0000 7efcc6600640 1 --2- 172.21.15.6:0/2730455514 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efca80039e0 0x7efca8003e00 secure :-1 s=READY pgs=1601 cs=0 l=1 rev1=1 crypto rx=0x7efcb4009a90 tx=0x7efca8005830 comp rx=0 tx=0).stop 2024-03-04T09:34:04.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:04.623+0000 7efcc5c00640 1 -- 172.21.15.6:0/2730455514 reap_dead start 2024-03-04T09:34:04.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:04.623+0000 7efcc6600640 1 -- 172.21.15.6:0/2730455514 shutdown_connections 2024-03-04T09:34:04.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:04.623+0000 7efcc6600640 1 -- 172.21.15.6:0/2730455514 >> 172.21.15.6:0/2730455514 conn(0x7efcc006c0c0 msgr2=0x7efcc006c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:34:04.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:04.623+0000 7efcc6600640 1 -- 172.21.15.6:0/2730455514 shutdown_connections 2024-03-04T09:34:04.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:04.623+0000 7efcc6600640 1 -- 172.21.15.6:0/2730455514 wait complete. 2024-03-04T09:34:04.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:04.625+0000 7efcc6600640 1 Processor -- start 2024-03-04T09:34:04.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:04.625+0000 7efcc6600640 1 -- start start 2024-03-04T09:34:04.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:04.626+0000 7efcc6600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efca80039e0 0x7efcc0144150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:04.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:04.626+0000 7efcc6600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efcc00af6c0 0x7efcc00afb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:04.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:04.626+0000 7efcc6600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efcc00adfe0 0x7efcc00ae440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:04.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:04.626+0000 7efcc6600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcc006ec40 con 0x7efca80039e0 2024-03-04T09:34:04.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:04.626+0000 7efcc6600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcc006eac0 con 0x7efcc00af6c0 2024-03-04T09:34:04.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:04.626+0000 7efcc6600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcc006edc0 con 0x7efcc00adfe0 2024-03-04T09:34:04.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:04.626+0000 7efcc5200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efca80039e0 0x7efcc0144150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:04.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:04.626+0000 7efcc5200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efca80039e0 0x7efcc0144150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55576/0 (socket says 172.21.15.6:55576) 2024-03-04T09:34:04.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:04.626+0000 7efcc5200640 1 -- 172.21.15.6:0/2326834004 learned_addr learned my addr 172.21.15.6:0/2326834004 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:34:04.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:04.626+0000 7efcc5200640 1 -- 172.21.15.6:0/2326834004 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efcc00adfe0 msgr2=0x7efcc00ae440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:04.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:04.626+0000 7efcc5200640 1 --2- 172.21.15.6:0/2326834004 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efcc00adfe0 0x7efcc00ae440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:04.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:04.626+0000 7efcc5200640 1 -- 172.21.15.6:0/2326834004 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efcc00af6c0 msgr2=0x7efcc00afb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:04.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:04.626+0000 7efcc5200640 1 --2- 172.21.15.6:0/2326834004 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efcc00af6c0 0x7efcc00afb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:04.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:04.626+0000 7efcc5200640 1 -- 172.21.15.6:0/2326834004 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcc013c390 con 0x7efca80039e0 2024-03-04T09:34:04.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:04.627+0000 7efcc5200640 1 --2- 172.21.15.6:0/2326834004 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efca80039e0 0x7efcc0144150 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x7efcb4009a90 tx=0x7efcb4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:34:04.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:04.627+0000 7efcbcc00640 1 -- 172.21.15.6:0/2326834004 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcb4039050 con 0x7efca80039e0 2024-03-04T09:34:04.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:04.627+0000 7efcbcc00640 1 -- 172.21.15.6:0/2326834004 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcb4045070 con 0x7efca80039e0 2024-03-04T09:34:04.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:04.627+0000 7efcbcc00640 1 -- 172.21.15.6:0/2326834004 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcb4009690 con 0x7efca80039e0 2024-03-04T09:34:04.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:04.627+0000 7efcc6600640 1 -- 172.21.15.6:0/2326834004 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcc013c670 con 0x7efca80039e0 2024-03-04T09:34:04.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:04.627+0000 7efcc6600640 1 -- 172.21.15.6:0/2326834004 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcc013cc30 con 0x7efca80039e0 2024-03-04T09:34:04.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:04.628+0000 7efcc6600640 1 -- 172.21.15.6:0/2326834004 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcc006c9d0 con 0x7efca80039e0 2024-03-04T09:34:04.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:04.630+0000 7efcbcc00640 1 -- 172.21.15.6:0/2326834004 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efcb4003a60 con 0x7efca80039e0 2024-03-04T09:34:04.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:04.630+0000 7efcbcc00640 1 --2- 172.21.15.6:0/2326834004 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7efca006b4d0 0x7efca006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:04.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:04.630+0000 7efcbcc00640 1 -- 172.21.15.6:0/2326834004 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7efcb40ac200 con 0x7efca80039e0 2024-03-04T09:34:04.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:04.631+0000 7efcbfe00640 1 --2- 172.21.15.6:0/2326834004 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7efca006b4d0 0x7efca006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:04.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:04.632+0000 7efcbfe00640 1 --2- 172.21.15.6:0/2326834004 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7efca006b4d0 0x7efca006d990 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7efcac000f60 tx=0x7efcac001560 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:34:04.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:04.635+0000 7efcbcc00640 1 -- 172.21.15.6:0/2326834004 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7efcc006c9d0 con 0x7efca80039e0 2024-03-04T09:34:04.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:04.877+0000 7efcc6600640 1 -- 172.21.15.6:0/2326834004 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efcc006e190 con 0x7efca80039e0 2024-03-04T09:34:04.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:04.878+0000 7efcbcc00640 1 -- 172.21.15.6:0/2326834004 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3465 (secure 0 0 0) 0x7efcb404f020 con 0x7efca80039e0 2024-03-04T09:34:04.875 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:34:04.876 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:33:54.652601+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"laggy_since":"2024-03-04T09:33:54.652601+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":9}]} 2024-03-04T09:34:04.876 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 100 2024-03-04T09:34:04.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:04.881+0000 7efcc6600640 1 -- 172.21.15.6:0/2326834004 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7efca006b4d0 msgr2=0x7efca006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:04.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:04.881+0000 7efcc6600640 1 --2- 172.21.15.6:0/2326834004 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7efca006b4d0 0x7efca006d990 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7efcac000f60 tx=0x7efcac001560 comp rx=0 tx=0).stop 2024-03-04T09:34:04.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:04.881+0000 7efcc6600640 1 -- 172.21.15.6:0/2326834004 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efca80039e0 msgr2=0x7efcc0144150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:04.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:04.881+0000 7efcc6600640 1 --2- 172.21.15.6:0/2326834004 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efca80039e0 0x7efcc0144150 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x7efcb4009a90 tx=0x7efcb4004730 comp rx=0 tx=0).stop 2024-03-04T09:34:04.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:04.881+0000 7efcc6600640 1 -- 172.21.15.6:0/2326834004 shutdown_connections 2024-03-04T09:34:04.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:04.881+0000 7efcc6600640 1 --2- 172.21.15.6:0/2326834004 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7efca006b4d0 0x7efca006d990 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:04.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:04.881+0000 7efcc6600640 1 --2- 172.21.15.6:0/2326834004 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efcc00adfe0 0x7efcc00ae440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:04.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:04.881+0000 7efcc6600640 1 --2- 172.21.15.6:0/2326834004 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efcc00af6c0 0x7efcc00afb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:04.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:04.881+0000 7efcc6600640 1 --2- 172.21.15.6:0/2326834004 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efca80039e0 0x7efcc0144150 unknown :-1 s=CLOSED pgs=1602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:04.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:04.881+0000 7efcc6600640 1 -- 172.21.15.6:0/2326834004 >> 172.21.15.6:0/2326834004 conn(0x7efcc006c0c0 msgr2=0x7efcc00b07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:34:04.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:04.881+0000 7efcc6600640 1 -- 172.21.15.6:0/2326834004 shutdown_connections 2024-03-04T09:34:04.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:04.881+0000 7efcc6600640 1 -- 172.21.15.6:0/2326834004 wait complete. 2024-03-04T09:34:05.101 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:32:57.159681+0000', 'modified': '2024-03-04T09:33:54.652601+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8214, 'mds_1': 6796}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6796': {'gid': 6796, 'name': 'd', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.146:6837/2458070840', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 2458070840}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 2458070840}]}, 'join_fscid': -1, 'laggy_since': '2024-03-04T09:33:54.652601+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8214': {'gid': 8214, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2446627517', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2446627517}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2446627517}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8214, 'qdb_cluster': [8214]} 2024-03-04T09:34:05.101 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6796:up:stopping 2024-03-04T09:34:05.810 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-04T09:34:06.102 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:34:06.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:06.392+0000 7fbd9a200640 1 Processor -- start 2024-03-04T09:34:06.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:06.392+0000 7fbd9a200640 1 -- start start 2024-03-04T09:34:06.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:06.394+0000 7fbd9a200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fbd94145300 0x7fbd940b49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:06.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:06.394+0000 7fbd9a200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fbd94070f10 0x7fbd940cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:06.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:06.394+0000 7fbd9a200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbd940cce90 0x7fbd940cd930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:06.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:06.394+0000 7fbd9a200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd940cd300 con 0x7fbd94145300 2024-03-04T09:34:06.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:06.394+0000 7fbd9a200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd940714d0 con 0x7fbd94070f10 2024-03-04T09:34:06.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:06.394+0000 7fbd9a200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd940717b0 con 0x7fbd940cce90 2024-03-04T09:34:06.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:06.394+0000 7fbd98e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fbd94145300 0x7fbd940b49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:06.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:06.394+0000 7fbd98e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fbd94145300 0x7fbd940b49a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55590/0 (socket says 172.21.15.6:55590) 2024-03-04T09:34:06.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:06.394+0000 7fbd98e00640 1 -- 172.21.15.6:0/681580188 learned_addr learned my addr 172.21.15.6:0/681580188 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:34:06.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:06.395+0000 7fbd99800640 1 --2- 172.21.15.6:0/681580188 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbd940cce90 0x7fbd940cd930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:06.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:06.395+0000 7fbd93e00640 1 --2- 172.21.15.6:0/681580188 >> v2:172.21.15.146:3300/0 conn(0x7fbd94070f10 0x7fbd940cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:06.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:06.395+0000 7fbd98e00640 1 -- 172.21.15.6:0/681580188 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbd940cce90 msgr2=0x7fbd940cd930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:34:06.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:06.395+0000 7fbd98e00640 1 --2- 172.21.15.6:0/681580188 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbd940cce90 0x7fbd940cd930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:06.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:06.395+0000 7fbd98e00640 1 -- 172.21.15.6:0/681580188 >> v2:172.21.15.146:3300/0 conn(0x7fbd94070f10 msgr2=0x7fbd940cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:34:06.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:06.395+0000 7fbd98e00640 1 --2- 172.21.15.6:0/681580188 >> v2:172.21.15.146:3300/0 conn(0x7fbd94070f10 0x7fbd940cc950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:06.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:06.395+0000 7fbd98e00640 1 -- 172.21.15.6:0/681580188 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd94071a90 con 0x7fbd94145300 2024-03-04T09:34:06.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:06.395+0000 7fbd93e00640 1 --2- 172.21.15.6:0/681580188 >> v2:172.21.15.146:3300/0 conn(0x7fbd94070f10 0x7fbd940cc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:34:06.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:06.395+0000 7fbd99800640 1 --2- 172.21.15.6:0/681580188 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbd940cce90 0x7fbd940cd930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:34:06.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:06.396+0000 7fbd98e00640 1 --2- 172.21.15.6:0/681580188 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbd94145300 0x7fbd940b49a0 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 crypto rx=0x7fbd940b5150 tx=0x7fbd8802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90bfa8416a828774 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:34:06.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:06.396+0000 7fbd93400640 1 -- 172.21.15.6:0/681580188 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd8802e070 con 0x7fbd94145300 2024-03-04T09:34:06.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:06.396+0000 7fbd93400640 1 -- 172.21.15.6:0/681580188 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd88037070 con 0x7fbd94145300 2024-03-04T09:34:06.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:06.396+0000 7fbd93400640 1 -- 172.21.15.6:0/681580188 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd88031a00 con 0x7fbd94145300 2024-03-04T09:34:06.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:06.397+0000 7fbd9a200640 1 -- 172.21.15.6:0/681580188 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbd94145300 msgr2=0x7fbd940b49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:06.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:06.397+0000 7fbd9a200640 1 --2- 172.21.15.6:0/681580188 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbd94145300 0x7fbd940b49a0 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 crypto rx=0x7fbd940b5150 tx=0x7fbd8802b880 comp rx=0 tx=0).stop 2024-03-04T09:34:06.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:06.397+0000 7fbd9a200640 1 -- 172.21.15.6:0/681580188 shutdown_connections 2024-03-04T09:34:06.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:06.397+0000 7fbd9a200640 1 --2- 172.21.15.6:0/681580188 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbd940cce90 0x7fbd940cd930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:06.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:06.397+0000 7fbd9a200640 1 --2- 172.21.15.6:0/681580188 >> v2:172.21.15.146:3300/0 conn(0x7fbd94070f10 0x7fbd940cc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:06.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:06.397+0000 7fbd9a200640 1 --2- 172.21.15.6:0/681580188 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbd94145300 0x7fbd940b49a0 unknown :-1 s=CLOSED pgs=1603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:06.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:06.397+0000 7fbd9a200640 1 -- 172.21.15.6:0/681580188 >> 172.21.15.6:0/681580188 conn(0x7fbd9406b3d0 msgr2=0x7fbd9406b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:34:06.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:06.397+0000 7fbd9a200640 1 -- 172.21.15.6:0/681580188 shutdown_connections 2024-03-04T09:34:06.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:06.397+0000 7fbd9a200640 1 -- 172.21.15.6:0/681580188 wait complete. 2024-03-04T09:34:06.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:06.399+0000 7fbd9a200640 1 Processor -- start 2024-03-04T09:34:06.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:06.399+0000 7fbd9a200640 1 -- start start 2024-03-04T09:34:06.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:06.400+0000 7fbd9a200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbd94070f10 0x7fbd9406d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:06.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:06.400+0000 7fbd9a200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbd940cce90 0x7fbd940c4a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:06.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:06.400+0000 7fbd9a200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbd94145300 0x7fbd9415b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:06.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:06.400+0000 7fbd9a200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd941456e0 con 0x7fbd94145300 2024-03-04T09:34:06.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:06.400+0000 7fbd9a200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd940b4ee0 con 0x7fbd94070f10 2024-03-04T09:34:06.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:06.400+0000 7fbd9a200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd940cde70 con 0x7fbd940cce90 2024-03-04T09:34:06.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:06.400+0000 7fbd99800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbd94145300 0x7fbd9415b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:06.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:06.400+0000 7fbd99800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbd94145300 0x7fbd9415b430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55600/0 (socket says 172.21.15.6:55600) 2024-03-04T09:34:06.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:06.400+0000 7fbd99800640 1 -- 172.21.15.6:0/46942685 learned_addr learned my addr 172.21.15.6:0/46942685 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:34:06.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:06.400+0000 7fbd98e00640 1 --2- 172.21.15.6:0/46942685 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbd94070f10 0x7fbd9406d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:06.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:06.400+0000 7fbd93e00640 1 --2- 172.21.15.6:0/46942685 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbd940cce90 0x7fbd940c4a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:06.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:06.401+0000 7fbd99800640 1 -- 172.21.15.6:0/46942685 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbd940cce90 msgr2=0x7fbd940c4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:06.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:06.401+0000 7fbd99800640 1 --2- 172.21.15.6:0/46942685 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbd940cce90 0x7fbd940c4a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:06.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:06.401+0000 7fbd99800640 1 -- 172.21.15.6:0/46942685 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbd94070f10 msgr2=0x7fbd9406d720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:06.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:06.401+0000 7fbd99800640 1 --2- 172.21.15.6:0/46942685 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbd94070f10 0x7fbd9406d720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:06.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:06.401+0000 7fbd99800640 1 -- 172.21.15.6:0/46942685 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd9413eb90 con 0x7fbd94145300 2024-03-04T09:34:06.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:06.401+0000 7fbd93e00640 1 --2- 172.21.15.6:0/46942685 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbd940cce90 0x7fbd940c4a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:34:06.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:06.401+0000 7fbd99800640 1 --2- 172.21.15.6:0/46942685 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbd94145300 0x7fbd9415b430 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7fbd8c00cef0 tx=0x7fbd8c00a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:34:06.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:06.401+0000 7fbd90c00640 1 -- 172.21.15.6:0/46942685 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd8c01a070 con 0x7fbd94145300 2024-03-04T09:34:06.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:06.401+0000 7fbd90c00640 1 -- 172.21.15.6:0/46942685 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd8c00d860 con 0x7fbd94145300 2024-03-04T09:34:06.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:06.401+0000 7fbd90c00640 1 -- 172.21.15.6:0/46942685 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd8c00dc50 con 0x7fbd94145300 2024-03-04T09:34:06.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:06.401+0000 7fbd9a200640 1 -- 172.21.15.6:0/46942685 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd9406c960 con 0x7fbd94145300 2024-03-04T09:34:06.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:06.401+0000 7fbd9a200640 1 -- 172.21.15.6:0/46942685 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd9406cf20 con 0x7fbd94145300 2024-03-04T09:34:06.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:06.403+0000 7fbd9a200640 1 -- 172.21.15.6:0/46942685 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd9404f330 con 0x7fbd94145300 2024-03-04T09:34:06.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:06.404+0000 7fbd90c00640 1 -- 172.21.15.6:0/46942685 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbd8c01d070 con 0x7fbd94145300 2024-03-04T09:34:06.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:06.405+0000 7fbd90c00640 1 --2- 172.21.15.6:0/46942685 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fbd7406b4d0 0x7fbd7406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:06.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:06.405+0000 7fbd90c00640 1 -- 172.21.15.6:0/46942685 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fbd8c08ab10 con 0x7fbd94145300 2024-03-04T09:34:06.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:06.405+0000 7fbd98e00640 1 --2- 172.21.15.6:0/46942685 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fbd7406b4d0 0x7fbd7406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:06.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:06.406+0000 7fbd98e00640 1 --2- 172.21.15.6:0/46942685 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fbd7406b4d0 0x7fbd7406d990 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fbd88036000 tx=0x7fbd8803d040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:34:06.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:06.409+0000 7fbd90c00640 1 -- 172.21.15.6:0/46942685 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbd9413eb90 con 0x7fbd94145300 2024-03-04T09:34:06.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:06.648+0000 7fbd9a200640 1 -- 172.21.15.6:0/46942685 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbd940cebf0 con 0x7fbd94145300 2024-03-04T09:34:06.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:06.649+0000 7fbd90c00640 1 -- 172.21.15.6:0/46942685 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3465 (secure 0 0 0) 0x7fbd8c059970 con 0x7fbd94145300 2024-03-04T09:34:06.647 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:34:06.647 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:33:54.652601+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"laggy_since":"2024-03-04T09:33:54.652601+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":9}]} 2024-03-04T09:34:06.647 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 100 2024-03-04T09:34:06.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:06.651+0000 7fbd9a200640 1 -- 172.21.15.6:0/46942685 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fbd7406b4d0 msgr2=0x7fbd7406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:06.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:06.651+0000 7fbd9a200640 1 --2- 172.21.15.6:0/46942685 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fbd7406b4d0 0x7fbd7406d990 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fbd88036000 tx=0x7fbd8803d040 comp rx=0 tx=0).stop 2024-03-04T09:34:06.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:06.651+0000 7fbd9a200640 1 -- 172.21.15.6:0/46942685 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbd94145300 msgr2=0x7fbd9415b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:06.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:06.651+0000 7fbd9a200640 1 --2- 172.21.15.6:0/46942685 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbd94145300 0x7fbd9415b430 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7fbd8c00cef0 tx=0x7fbd8c00a710 comp rx=0 tx=0).stop 2024-03-04T09:34:06.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:06.652+0000 7fbd9a200640 1 -- 172.21.15.6:0/46942685 shutdown_connections 2024-03-04T09:34:06.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:06.652+0000 7fbd9a200640 1 --2- 172.21.15.6:0/46942685 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbd94145300 0x7fbd9415b430 unknown :-1 s=CLOSED pgs=1604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:06.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:06.652+0000 7fbd9a200640 1 --2- 172.21.15.6:0/46942685 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fbd7406b4d0 0x7fbd7406d990 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:06.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:06.652+0000 7fbd9a200640 1 --2- 172.21.15.6:0/46942685 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbd940cce90 0x7fbd940c4a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:06.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:06.652+0000 7fbd9a200640 1 --2- 172.21.15.6:0/46942685 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbd94070f10 0x7fbd9406d720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:06.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:06.652+0000 7fbd9a200640 1 -- 172.21.15.6:0/46942685 >> 172.21.15.6:0/46942685 conn(0x7fbd9406b3d0 msgr2=0x7fbd940af750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:34:06.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:06.652+0000 7fbd9a200640 1 -- 172.21.15.6:0/46942685 shutdown_connections 2024-03-04T09:34:06.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:06.652+0000 7fbd9a200640 1 -- 172.21.15.6:0/46942685 wait complete. 2024-03-04T09:34:06.870 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:32:57.159681+0000', 'modified': '2024-03-04T09:33:54.652601+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8214, 'mds_1': 6796}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6796': {'gid': 6796, 'name': 'd', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.146:6837/2458070840', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 2458070840}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 2458070840}]}, 'join_fscid': -1, 'laggy_since': '2024-03-04T09:33:54.652601+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8214': {'gid': 8214, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2446627517', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2446627517}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2446627517}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8214, 'qdb_cluster': [8214]} 2024-03-04T09:34:06.870 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6796:up:stopping 2024-03-04T09:34:07.812 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-04T09:34:07.871 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:34:08.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:08.102+0000 7fa227200640 1 Processor -- start 2024-03-04T09:34:08.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:08.102+0000 7fa227200640 1 -- start start 2024-03-04T09:34:08.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:08.103+0000 7fa227200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa2200d0e50 0x7fa220168c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:08.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:08.103+0000 7fa227200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2200b3710 con 0x7fa22015e0b0 2024-03-04T09:34:08.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:08.103+0000 7fa227200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2200b39f0 con 0x7fa2200d4a10 2024-03-04T09:34:08.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:08.103+0000 7fa227200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2200b09e0 con 0x7fa2200d0e50 2024-03-04T09:34:08.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:08.104+0000 7fa225e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fa22015e0b0 0x7fa2200cf310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:55032/0 (socket says 172.21.15.6:55032) 2024-03-04T09:34:08.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:08.104+0000 7fa225e00640 1 -- 172.21.15.6:0/3726331006 learned_addr learned my addr 172.21.15.6:0/3726331006 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:34:08.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:08.104+0000 7fa226800640 1 --2- 172.21.15.6:0/3726331006 >> v2:172.21.15.6:3300/0 conn(0x7fa2200d0e50 0x7fa220168c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:08.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:08.104+0000 7fa224a00640 1 -- 172.21.15.6:0/3726331006 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1367030012 0 0) 0x7fa2200b3710 con 0x7fa22015e0b0 2024-03-04T09:34:08.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:08.104+0000 7fa226800640 1 -- 172.21.15.6:0/3726331006 >> v1:172.21.15.146:6789/0 conn(0x7fa2200d4a10 legacy=0x7fa2200d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:34:08.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:08.104+0000 7fa226800640 1 -- 172.21.15.6:0/3726331006 >> v1:172.21.15.6:6789/0 conn(0x7fa22015e0b0 legacy=0x7fa2200cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:08.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:08.104+0000 7fa226800640 1 -- 172.21.15.6:0/3726331006 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2200b3cd0 con 0x7fa2200d0e50 2024-03-04T09:34:08.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:08.106+0000 7fa226800640 1 --2- 172.21.15.6:0/3726331006 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa2200d0e50 0x7fa220168c50 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7fa21c0097a0 tx=0x7fa21c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f61e1aee8c88de6 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:34:08.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:08.107+0000 7fa224a00640 1 -- 172.21.15.6:0/3726331006 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa21c035020 con 0x7fa2200d0e50 2024-03-04T09:34:08.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:08.107+0000 7fa224a00640 1 -- 172.21.15.6:0/3726331006 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa21c031830 con 0x7fa2200d0e50 2024-03-04T09:34:08.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:08.107+0000 7fa224a00640 1 -- 172.21.15.6:0/3726331006 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa21c031c00 con 0x7fa2200d0e50 2024-03-04T09:34:08.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:08.107+0000 7fa227200640 1 -- 172.21.15.6:0/3726331006 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa2200d0e50 msgr2=0x7fa220168c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:08.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:08.107+0000 7fa227200640 1 --2- 172.21.15.6:0/3726331006 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa2200d0e50 0x7fa220168c50 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7fa21c0097a0 tx=0x7fa21c02b8d0 comp rx=0 tx=0).stop 2024-03-04T09:34:08.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:08.108+0000 7fa227200640 1 -- 172.21.15.6:0/3726331006 shutdown_connections 2024-03-04T09:34:08.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:08.108+0000 7fa227200640 1 --2- 172.21.15.6:0/3726331006 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa2200d0e50 0x7fa220168c50 unknown :-1 s=CLOSED pgs=1606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:08.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:08.108+0000 7fa227200640 1 -- 172.21.15.6:0/3726331006 >> 172.21.15.6:0/3726331006 conn(0x7fa2200d3d30 msgr2=0x7fa2200d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:34:08.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:08.108+0000 7fa227200640 1 -- 172.21.15.6:0/3726331006 shutdown_connections 2024-03-04T09:34:08.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:08.108+0000 7fa227200640 1 -- 172.21.15.6:0/3726331006 wait complete. 2024-03-04T09:34:08.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:08.110+0000 7fa227200640 1 Processor -- start 2024-03-04T09:34:08.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:08.110+0000 7fa227200640 1 -- start start 2024-03-04T09:34:08.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:08.111+0000 7fa227200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa2200d0e50 0x7fa2200a16b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:08.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:08.111+0000 7fa227200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa2200d4a10 0x7fa22009e850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:08.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:08.111+0000 7fa227200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa22015e0b0 0x7fa2200b8960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:08.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:08.111+0000 7fa227200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa22015e490 con 0x7fa2200d0e50 2024-03-04T09:34:08.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:08.111+0000 7fa227200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2200cfa20 con 0x7fa22015e0b0 2024-03-04T09:34:08.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:08.111+0000 7fa227200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2200cfc60 con 0x7fa2200d4a10 2024-03-04T09:34:08.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:08.111+0000 7fa225e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa2200d0e50 0x7fa2200a16b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:08.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:08.111+0000 7fa225e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa2200d0e50 0x7fa2200a16b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55612/0 (socket says 172.21.15.6:55612) 2024-03-04T09:34:08.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:08.111+0000 7fa225e00640 1 -- 172.21.15.6:0/128618090 learned_addr learned my addr 172.21.15.6:0/128618090 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:34:08.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:08.112+0000 7fa226800640 1 --2- 172.21.15.6:0/128618090 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa22015e0b0 0x7fa2200b8960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:08.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:08.112+0000 7fa225400640 1 --2- 172.21.15.6:0/128618090 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa2200d4a10 0x7fa22009e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:08.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:08.112+0000 7fa225e00640 1 -- 172.21.15.6:0/128618090 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa2200d4a10 msgr2=0x7fa22009e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:08.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:08.112+0000 7fa225e00640 1 --2- 172.21.15.6:0/128618090 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa2200d4a10 0x7fa22009e850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:08.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:08.112+0000 7fa225e00640 1 -- 172.21.15.6:0/128618090 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa22015e0b0 msgr2=0x7fa2200b8960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:08.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:08.112+0000 7fa225e00640 1 --2- 172.21.15.6:0/128618090 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa22015e0b0 0x7fa2200b8960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:08.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:08.112+0000 7fa225e00640 1 -- 172.21.15.6:0/128618090 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2201614e0 con 0x7fa2200d0e50 2024-03-04T09:34:08.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:08.112+0000 7fa225e00640 1 --2- 172.21.15.6:0/128618090 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa2200d0e50 0x7fa2200a16b0 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7fa21800c7c0 tx=0x7fa21800cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:34:08.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:08.112+0000 7fa216000640 1 -- 172.21.15.6:0/128618090 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa218018070 con 0x7fa2200d0e50 2024-03-04T09:34:08.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:08.112+0000 7fa216000640 1 -- 172.21.15.6:0/128618090 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa21800a640 con 0x7fa2200d0e50 2024-03-04T09:34:08.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:08.113+0000 7fa216000640 1 -- 172.21.15.6:0/128618090 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa21800aa30 con 0x7fa2200d0e50 2024-03-04T09:34:08.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:08.113+0000 7fa227200640 1 -- 172.21.15.6:0/128618090 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2200cb520 con 0x7fa2200d0e50 2024-03-04T09:34:08.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:08.113+0000 7fa227200640 1 -- 172.21.15.6:0/128618090 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa220161140 con 0x7fa2200d0e50 2024-03-04T09:34:08.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:08.114+0000 7fa227200640 1 -- 172.21.15.6:0/128618090 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa22004f330 con 0x7fa2200d0e50 2024-03-04T09:34:08.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:08.115+0000 7fa216000640 1 -- 172.21.15.6:0/128618090 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa21801b070 con 0x7fa2200d0e50 2024-03-04T09:34:08.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:08.116+0000 7fa216000640 1 --2- 172.21.15.6:0/128618090 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa20406b4d0 0x7fa20406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:08.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:08.116+0000 7fa216000640 1 -- 172.21.15.6:0/128618090 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fa218089800 con 0x7fa2200d0e50 2024-03-04T09:34:08.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:08.116+0000 7fa225400640 1 --2- 172.21.15.6:0/128618090 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa20406b4d0 0x7fa20406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:08.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:08.118+0000 7fa225400640 1 --2- 172.21.15.6:0/128618090 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa20406b4d0 0x7fa20406d990 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7fa2200ca3b0 tx=0x7fa210009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:34:08.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:08.121+0000 7fa216000640 1 -- 172.21.15.6:0/128618090 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa2180586d0 con 0x7fa2200d0e50 2024-03-04T09:34:08.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:08.359+0000 7fa227200640 1 -- 172.21.15.6:0/128618090 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa220074180 con 0x7fa2200d0e50 2024-03-04T09:34:08.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:08.360+0000 7fa216000640 1 -- 172.21.15.6:0/128618090 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3465 (secure 0 0 0) 0x7fa218057fd0 con 0x7fa2200d0e50 2024-03-04T09:34:08.357 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:34:08.357 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:33:54.652601+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"laggy_since":"2024-03-04T09:33:54.652601+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":9}]} 2024-03-04T09:34:08.357 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 100 2024-03-04T09:34:08.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:08.362+0000 7fa227200640 1 -- 172.21.15.6:0/128618090 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa20406b4d0 msgr2=0x7fa20406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:08.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:08.362+0000 7fa227200640 1 --2- 172.21.15.6:0/128618090 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa20406b4d0 0x7fa20406d990 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7fa2200ca3b0 tx=0x7fa210009040 comp rx=0 tx=0).stop 2024-03-04T09:34:08.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:08.362+0000 7fa227200640 1 -- 172.21.15.6:0/128618090 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa2200d0e50 msgr2=0x7fa2200a16b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:08.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:08.362+0000 7fa227200640 1 --2- 172.21.15.6:0/128618090 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa2200d0e50 0x7fa2200a16b0 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7fa21800c7c0 tx=0x7fa21800cc80 comp rx=0 tx=0).stop 2024-03-04T09:34:08.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:08.363+0000 7fa227200640 1 -- 172.21.15.6:0/128618090 shutdown_connections 2024-03-04T09:34:08.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:08.363+0000 7fa227200640 1 --2- 172.21.15.6:0/128618090 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa22015e0b0 0x7fa2200b8960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:08.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:08.363+0000 7fa227200640 1 --2- 172.21.15.6:0/128618090 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa20406b4d0 0x7fa20406d990 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:08.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:08.363+0000 7fa227200640 1 --2- 172.21.15.6:0/128618090 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa2200d4a10 0x7fa22009e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:08.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:08.363+0000 7fa227200640 1 --2- 172.21.15.6:0/128618090 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa2200d0e50 0x7fa2200a16b0 unknown :-1 s=CLOSED pgs=1607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:08.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:08.363+0000 7fa227200640 1 -- 172.21.15.6:0/128618090 >> 172.21.15.6:0/128618090 conn(0x7fa2200d3d30 msgr2=0x7fa2200b5ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:34:08.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:08.363+0000 7fa227200640 1 -- 172.21.15.6:0/128618090 shutdown_connections 2024-03-04T09:34:08.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:08.363+0000 7fa227200640 1 -- 172.21.15.6:0/128618090 wait complete. 2024-03-04T09:34:08.580 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:32:57.159681+0000', 'modified': '2024-03-04T09:33:54.652601+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8214, 'mds_1': 6796}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6796': {'gid': 6796, 'name': 'd', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.146:6837/2458070840', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 2458070840}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 2458070840}]}, 'join_fscid': -1, 'laggy_since': '2024-03-04T09:33:54.652601+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8214': {'gid': 8214, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2446627517', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2446627517}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2446627517}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8214, 'qdb_cluster': [8214]} 2024-03-04T09:34:08.580 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6796:up:stopping 2024-03-04T09:34:09.581 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:34:09.814 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-04T09:34:09.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:09.857+0000 7f75cce00640 1 Processor -- start 2024-03-04T09:34:09.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:09.857+0000 7f75cce00640 1 -- start start 2024-03-04T09:34:09.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:09.858+0000 7f75cce00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f75c8145300 0x7f75c80b49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:09.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:09.858+0000 7f75cce00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f75c8070f10 0x7f75c80cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:09.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:09.858+0000 7f75cce00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75c80714d0 con 0x7f75c80cce90 2024-03-04T09:34:09.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:09.858+0000 7f75cce00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75c80cd300 con 0x7f75c8070f10 2024-03-04T09:34:09.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:09.858+0000 7f75cce00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75c80717b0 con 0x7f75c8145300 2024-03-04T09:34:09.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:09.858+0000 7f75c6a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f75c8070f10 0x7f75c80cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:09.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:09.859+0000 7f75c6a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f75c8070f10 0x7f75c80cc950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:52548/0 (socket says 172.21.15.6:52548) 2024-03-04T09:34:09.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:09.859+0000 7f75c6a00640 1 -- 172.21.15.6:0/1846325935 learned_addr learned my addr 172.21.15.6:0/1846325935 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:34:09.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:09.859+0000 7f75c6a00640 1 -- 172.21.15.6:0/1846325935 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f75c8145300 msgr2=0x7f75c80b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:34:09.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:09.859+0000 7f75c6a00640 1 --2- 172.21.15.6:0/1846325935 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f75c8145300 0x7f75c80b49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:09.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:09.859+0000 7f75c6a00640 1 -- 172.21.15.6:0/1846325935 >> v1:172.21.15.146:6789/0 conn(0x7f75c80cce90 legacy=0x7f75c80cd930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:34:09.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:09.859+0000 7f75c6a00640 1 -- 172.21.15.6:0/1846325935 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75c8071a90 con 0x7f75c8070f10 2024-03-04T09:34:09.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:09.860+0000 7f75c6a00640 1 --2- 172.21.15.6:0/1846325935 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f75c8070f10 0x7f75c80cc950 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7f75b4009870 tx=0x7f75b402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c29a9c47dead1b45 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:34:09.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:09.860+0000 7f75c6000640 1 -- 172.21.15.6:0/1846325935 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75b4035020 con 0x7f75c8070f10 2024-03-04T09:34:09.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:09.860+0000 7f75c6000640 1 -- 172.21.15.6:0/1846325935 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75b4031640 con 0x7f75c8070f10 2024-03-04T09:34:09.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:09.860+0000 7f75c6000640 1 -- 172.21.15.6:0/1846325935 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75b402d040 con 0x7f75c8070f10 2024-03-04T09:34:09.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:09.861+0000 7f75cce00640 1 -- 172.21.15.6:0/1846325935 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f75c8070f10 msgr2=0x7f75c80cc950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:09.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:09.861+0000 7f75cce00640 1 --2- 172.21.15.6:0/1846325935 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f75c8070f10 0x7f75c80cc950 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7f75b4009870 tx=0x7f75b402b8f0 comp rx=0 tx=0).stop 2024-03-04T09:34:09.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:09.861+0000 7f75cce00640 1 -- 172.21.15.6:0/1846325935 shutdown_connections 2024-03-04T09:34:09.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:09.861+0000 7f75cce00640 1 --2- 172.21.15.6:0/1846325935 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f75c8070f10 0x7f75c80cc950 unknown :-1 s=CLOSED pgs=1608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:09.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:09.861+0000 7f75cce00640 1 --2- 172.21.15.6:0/1846325935 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f75c8145300 0x7f75c80b49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:09.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:09.861+0000 7f75cce00640 1 -- 172.21.15.6:0/1846325935 >> 172.21.15.6:0/1846325935 conn(0x7f75c806b3d0 msgr2=0x7f75c806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:34:09.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:09.861+0000 7f75cce00640 1 -- 172.21.15.6:0/1846325935 shutdown_connections 2024-03-04T09:34:09.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:09.861+0000 7f75cce00640 1 -- 172.21.15.6:0/1846325935 wait complete. 2024-03-04T09:34:09.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:09.863+0000 7f75cce00640 1 Processor -- start 2024-03-04T09:34:09.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:09.863+0000 7f75cce00640 1 -- start start 2024-03-04T09:34:09.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:09.865+0000 7f75cce00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f75c8070f10 0x7f75c80ae240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:09.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:09.865+0000 7f75cce00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f75c80cce90 0x7f75c813c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:09.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:09.865+0000 7f75cce00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f75c8145300 0x7f75c80a9b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:09.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:09.865+0000 7f75cce00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75c81456e0 con 0x7f75c8070f10 2024-03-04T09:34:09.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:09.865+0000 7f75cce00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75c80b4ee0 con 0x7f75c8145300 2024-03-04T09:34:09.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:09.865+0000 7f75cce00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75c815d580 con 0x7f75c80cce90 2024-03-04T09:34:09.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:09.865+0000 7f75c7400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f75c8070f10 0x7f75c80ae240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:09.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:09.865+0000 7f75c7400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f75c8070f10 0x7f75c80ae240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:52552/0 (socket says 172.21.15.6:52552) 2024-03-04T09:34:09.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:09.865+0000 7f75c7400640 1 -- 172.21.15.6:0/3133369873 learned_addr learned my addr 172.21.15.6:0/3133369873 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:34:09.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:09.866+0000 7f75c6a00640 1 --2- 172.21.15.6:0/3133369873 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f75c80cce90 0x7f75c813c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:09.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:09.866+0000 7f75c7e00640 1 --2- 172.21.15.6:0/3133369873 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f75c8145300 0x7f75c80a9b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:09.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:09.866+0000 7f75c7400640 1 -- 172.21.15.6:0/3133369873 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f75c80cce90 msgr2=0x7f75c813c370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:09.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:09.866+0000 7f75c7400640 1 --2- 172.21.15.6:0/3133369873 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f75c80cce90 0x7f75c813c370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:09.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:09.866+0000 7f75c7400640 1 -- 172.21.15.6:0/3133369873 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f75c8145300 msgr2=0x7f75c80a9b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:09.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:09.866+0000 7f75c7400640 1 --2- 172.21.15.6:0/3133369873 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f75c8145300 0x7f75c80a9b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:09.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:09.866+0000 7f75c7400640 1 -- 172.21.15.6:0/3133369873 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75c80aec90 con 0x7f75c8070f10 2024-03-04T09:34:09.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:09.866+0000 7f75c7e00640 1 --2- 172.21.15.6:0/3133369873 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f75c8145300 0x7f75c80a9b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:34:09.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:09.866+0000 7f75c7400640 1 --2- 172.21.15.6:0/3133369873 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f75c8070f10 0x7f75c80ae240 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 crypto rx=0x7f75b800b410 tx=0x7f75b800b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:34:09.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:09.867+0000 7f75bf400640 1 -- 172.21.15.6:0/3133369873 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75b8015020 con 0x7f75c8070f10 2024-03-04T09:34:09.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:09.867+0000 7f75bf400640 1 -- 172.21.15.6:0/3133369873 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75b8011850 con 0x7f75c8070f10 2024-03-04T09:34:09.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:09.867+0000 7f75cce00640 1 -- 172.21.15.6:0/3133369873 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75c80aef70 con 0x7f75c8070f10 2024-03-04T09:34:09.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:09.867+0000 7f75bf400640 1 -- 172.21.15.6:0/3133369873 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75b8011c40 con 0x7f75c8070f10 2024-03-04T09:34:09.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:09.867+0000 7f75cce00640 1 -- 172.21.15.6:0/3133369873 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75c80a9610 con 0x7f75c8070f10 2024-03-04T09:34:09.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:09.868+0000 7f75cce00640 1 -- 172.21.15.6:0/3133369873 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75c804f330 con 0x7f75c8070f10 2024-03-04T09:34:09.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:09.869+0000 7f75bf400640 1 -- 172.21.15.6:0/3133369873 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f75b8018020 con 0x7f75c8070f10 2024-03-04T09:34:09.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:09.870+0000 7f75bf400640 1 --2- 172.21.15.6:0/3133369873 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f75a806b4d0 0x7f75a806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:09.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:09.870+0000 7f75bf400640 1 -- 172.21.15.6:0/3133369873 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f75b808a280 con 0x7f75c8070f10 2024-03-04T09:34:09.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:09.871+0000 7f75c6a00640 1 --2- 172.21.15.6:0/3133369873 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f75a806b4d0 0x7f75a806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:09.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:09.872+0000 7f75c6a00640 1 --2- 172.21.15.6:0/3133369873 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f75a806b4d0 0x7f75a806d990 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f75b4036000 tx=0x7f75b403c040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:34:09.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:09.875+0000 7f75bf400640 1 -- 172.21.15.6:0/3133369873 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f75b801f020 con 0x7f75c8070f10 2024-03-04T09:34:10.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:10.129+0000 7f75cce00640 1 -- 172.21.15.6:0/3133369873 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f75c81448f0 con 0x7f75c8070f10 2024-03-04T09:34:10.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:10.130+0000 7f75bf400640 1 -- 172.21.15.6:0/3133369873 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3465 (secure 0 0 0) 0x7f75b80172c0 con 0x7f75c8070f10 2024-03-04T09:34:10.127 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:34:10.127 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:33:54.652601+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"laggy_since":"2024-03-04T09:33:54.652601+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":9}]} 2024-03-04T09:34:10.128 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 100 2024-03-04T09:34:10.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:10.133+0000 7f75cce00640 1 -- 172.21.15.6:0/3133369873 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f75a806b4d0 msgr2=0x7f75a806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:10.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:10.133+0000 7f75cce00640 1 --2- 172.21.15.6:0/3133369873 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f75a806b4d0 0x7f75a806d990 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f75b4036000 tx=0x7f75b403c040 comp rx=0 tx=0).stop 2024-03-04T09:34:10.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:10.134+0000 7f75cce00640 1 -- 172.21.15.6:0/3133369873 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f75c8070f10 msgr2=0x7f75c80ae240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:10.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:10.134+0000 7f75cce00640 1 --2- 172.21.15.6:0/3133369873 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f75c8070f10 0x7f75c80ae240 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 crypto rx=0x7f75b800b410 tx=0x7f75b800b8d0 comp rx=0 tx=0).stop 2024-03-04T09:34:10.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:10.134+0000 7f75cce00640 1 -- 172.21.15.6:0/3133369873 shutdown_connections 2024-03-04T09:34:10.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:10.134+0000 7f75cce00640 1 --2- 172.21.15.6:0/3133369873 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f75c8145300 0x7f75c80a9b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:10.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:10.134+0000 7f75cce00640 1 --2- 172.21.15.6:0/3133369873 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f75a806b4d0 0x7f75a806d990 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:10.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:10.134+0000 7f75cce00640 1 --2- 172.21.15.6:0/3133369873 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f75c80cce90 0x7f75c813c370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:10.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:10.134+0000 7f75cce00640 1 --2- 172.21.15.6:0/3133369873 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f75c8070f10 0x7f75c80ae240 unknown :-1 s=CLOSED pgs=1609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:10.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:10.134+0000 7f75cce00640 1 -- 172.21.15.6:0/3133369873 >> 172.21.15.6:0/3133369873 conn(0x7f75c806b3d0 msgr2=0x7f75c815e160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:34:10.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:10.134+0000 7f75cce00640 1 -- 172.21.15.6:0/3133369873 shutdown_connections 2024-03-04T09:34:10.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:10.134+0000 7f75cce00640 1 -- 172.21.15.6:0/3133369873 wait complete. 2024-03-04T09:34:10.351 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:32:57.159681+0000', 'modified': '2024-03-04T09:33:54.652601+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8214, 'mds_1': 6796}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6796': {'gid': 6796, 'name': 'd', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.146:6837/2458070840', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 2458070840}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 2458070840}]}, 'join_fscid': -1, 'laggy_since': '2024-03-04T09:33:54.652601+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8214': {'gid': 8214, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2446627517', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2446627517}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2446627517}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8214, 'qdb_cluster': [8214]} 2024-03-04T09:34:10.351 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6796:up:stopping 2024-03-04T09:34:10.980 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:34:10.988 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:34:11.352 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:34:11.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:11.577+0000 7f8a1da00640 1 Processor -- start 2024-03-04T09:34:11.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:11.578+0000 7f8a1da00640 1 -- start start 2024-03-04T09:34:11.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:11.579+0000 7f8a1da00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8a180afb80 0x7f8a180cf5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:11.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:11.579+0000 7f8a1da00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f8a1815b6a0 0x7f8a1815bb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:11.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:11.579+0000 7f8a1da00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a180cfd50 con 0x7f8a1815b2c0 2024-03-04T09:34:11.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:11.579+0000 7f8a1da00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a180cfb20 con 0x7f8a1815b6a0 2024-03-04T09:34:11.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:11.579+0000 7f8a1da00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a180b3520 con 0x7f8a180afb80 2024-03-04T09:34:11.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:11.579+0000 7f8a17400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f8a1815b6a0 0x7f8a1815bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:11.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:11.579+0000 7f8a17400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f8a1815b6a0 0x7f8a1815bb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:52556/0 (socket says 172.21.15.6:52556) 2024-03-04T09:34:11.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:11.579+0000 7f8a17400640 1 -- 172.21.15.6:0/2516325181 learned_addr learned my addr 172.21.15.6:0/2516325181 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:34:11.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:11.579+0000 7f8a16a00640 1 --1- 172.21.15.6:0/2516325181 >> v1:172.21.15.6:6789/0 conn(0x7f8a1815b2c0 0x7f8a1806d540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:55502/0 (socket says 172.21.15.6:55502) 2024-03-04T09:34:11.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:11.580+0000 7f8a17400640 1 -- 172.21.15.6:0/2516325181 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8a180afb80 msgr2=0x7f8a180cf5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:34:11.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:11.580+0000 7f8a17400640 1 --2- 172.21.15.6:0/2516325181 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8a180afb80 0x7f8a180cf5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:11.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:11.580+0000 7f8a17400640 1 -- 172.21.15.6:0/2516325181 >> v1:172.21.15.6:6789/0 conn(0x7f8a1815b2c0 legacy=0x7f8a1806d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:11.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:11.580+0000 7f8a17400640 1 -- 172.21.15.6:0/2516325181 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a180b3770 con 0x7f8a1815b6a0 2024-03-04T09:34:11.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:11.581+0000 7f8a17400640 1 --2- 172.21.15.6:0/2516325181 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8a1815b6a0 0x7f8a1815bb30 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 crypto rx=0x7f8a10009e80 tx=0x7f8a1002bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19db08710ea29fd6 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:34:11.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:11.581+0000 7f8a15600640 1 -- 172.21.15.6:0/2516325181 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a1002caf0 con 0x7f8a1815b6a0 2024-03-04T09:34:11.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:11.581+0000 7f8a15600640 1 -- 172.21.15.6:0/2516325181 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a1002cc90 con 0x7f8a1815b6a0 2024-03-04T09:34:11.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:11.581+0000 7f8a15600640 1 -- 172.21.15.6:0/2516325181 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a10034610 con 0x7f8a1815b6a0 2024-03-04T09:34:11.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:11.582+0000 7f8a1da00640 1 -- 172.21.15.6:0/2516325181 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8a1815b6a0 msgr2=0x7f8a1815bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:11.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:11.582+0000 7f8a1da00640 1 --2- 172.21.15.6:0/2516325181 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8a1815b6a0 0x7f8a1815bb30 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 crypto rx=0x7f8a10009e80 tx=0x7f8a1002bf50 comp rx=0 tx=0).stop 2024-03-04T09:34:11.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:11.582+0000 7f8a1da00640 1 -- 172.21.15.6:0/2516325181 shutdown_connections 2024-03-04T09:34:11.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:11.582+0000 7f8a1da00640 1 --2- 172.21.15.6:0/2516325181 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8a1815b6a0 0x7f8a1815bb30 unknown :-1 s=CLOSED pgs=1611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:11.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:11.582+0000 7f8a1da00640 1 --2- 172.21.15.6:0/2516325181 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8a180afb80 0x7f8a180cf5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:11.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:11.582+0000 7f8a1da00640 1 -- 172.21.15.6:0/2516325181 >> 172.21.15.6:0/2516325181 conn(0x7f8a180b2640 msgr2=0x7f8a180b2a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:34:11.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:11.582+0000 7f8a1da00640 1 -- 172.21.15.6:0/2516325181 shutdown_connections 2024-03-04T09:34:11.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:11.582+0000 7f8a1da00640 1 -- 172.21.15.6:0/2516325181 wait complete. 2024-03-04T09:34:11.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:11.584+0000 7f8a1da00640 1 Processor -- start 2024-03-04T09:34:11.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:11.584+0000 7f8a1da00640 1 -- start start 2024-03-04T09:34:11.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:11.585+0000 7f8a1da00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8a180afb80 0x7f8a1814ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:11.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:11.585+0000 7f8a1da00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8a1815b2c0 0x7f8a181450c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:11.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:11.585+0000 7f8a1da00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8a1815b6a0 0x7f8a1815c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:11.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:11.585+0000 7f8a1da00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a1815c070 con 0x7f8a180afb80 2024-03-04T09:34:11.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:11.585+0000 7f8a1da00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a180b3d30 con 0x7f8a1815b6a0 2024-03-04T09:34:11.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:11.585+0000 7f8a1da00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a18155610 con 0x7f8a1815b2c0 2024-03-04T09:34:11.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:11.585+0000 7f8a16a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8a180afb80 0x7f8a1814ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:11.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:11.585+0000 7f8a16a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8a180afb80 0x7f8a1814ca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:52564/0 (socket says 172.21.15.6:52564) 2024-03-04T09:34:11.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:11.585+0000 7f8a16a00640 1 -- 172.21.15.6:0/4167291185 learned_addr learned my addr 172.21.15.6:0/4167291185 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:34:11.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:11.586+0000 7f8a17400640 1 --2- 172.21.15.6:0/4167291185 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8a1815b6a0 0x7f8a1815c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:11.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:11.586+0000 7f8a16000640 1 --2- 172.21.15.6:0/4167291185 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8a1815b2c0 0x7f8a181450c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:11.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:11.586+0000 7f8a16a00640 1 -- 172.21.15.6:0/4167291185 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8a1815b2c0 msgr2=0x7f8a181450c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:11.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:11.586+0000 7f8a16a00640 1 --2- 172.21.15.6:0/4167291185 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8a1815b2c0 0x7f8a181450c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:11.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:11.586+0000 7f8a16a00640 1 -- 172.21.15.6:0/4167291185 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8a1815b6a0 msgr2=0x7f8a1815c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:11.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:11.586+0000 7f8a16a00640 1 --2- 172.21.15.6:0/4167291185 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8a1815b6a0 0x7f8a1815c700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:11.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:11.586+0000 7f8a16a00640 1 -- 172.21.15.6:0/4167291185 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a18149570 con 0x7f8a180afb80 2024-03-04T09:34:11.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:11.586+0000 7f8a17400640 1 --2- 172.21.15.6:0/4167291185 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8a1815b6a0 0x7f8a1815c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:34:11.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:11.586+0000 7f8a16a00640 1 --2- 172.21.15.6:0/4167291185 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8a180afb80 0x7f8a1814ca40 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 crypto rx=0x7f8a0800c4d0 tx=0x7f8a0800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:34:11.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:11.586+0000 7f8a0ea00640 1 -- 172.21.15.6:0/4167291185 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a08017070 con 0x7f8a180afb80 2024-03-04T09:34:11.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:11.586+0000 7f8a0ea00640 1 -- 172.21.15.6:0/4167291185 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a080064e0 con 0x7f8a180afb80 2024-03-04T09:34:11.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:11.586+0000 7f8a0ea00640 1 -- 172.21.15.6:0/4167291185 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a080049e0 con 0x7f8a180afb80 2024-03-04T09:34:11.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:11.586+0000 7f8a1da00640 1 -- 172.21.15.6:0/4167291185 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a1815cf50 con 0x7f8a180afb80 2024-03-04T09:34:11.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:11.587+0000 7f8a1da00640 1 -- 172.21.15.6:0/4167291185 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a1815d510 con 0x7f8a180afb80 2024-03-04T09:34:11.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:11.588+0000 7f8a1da00640 1 -- 172.21.15.6:0/4167291185 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a1804f3a0 con 0x7f8a180afb80 2024-03-04T09:34:11.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:11.589+0000 7f8a0ea00640 1 -- 172.21.15.6:0/4167291185 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8a08004b80 con 0x7f8a180afb80 2024-03-04T09:34:11.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:11.590+0000 7f8a0ea00640 1 --2- 172.21.15.6:0/4167291185 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f89f806b4d0 0x7f89f806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:11.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:11.590+0000 7f8a0ea00640 1 -- 172.21.15.6:0/4167291185 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f8a08089500 con 0x7f8a180afb80 2024-03-04T09:34:11.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:11.591+0000 7f8a16000640 1 --2- 172.21.15.6:0/4167291185 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f89f806b4d0 0x7f89f806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:11.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:11.592+0000 7f8a16000640 1 --2- 172.21.15.6:0/4167291185 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f89f806b4d0 0x7f89f806d990 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f8a1815c4f0 tx=0x7f8a04008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:34:11.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:11.598+0000 7f8a0ea00640 1 -- 172.21.15.6:0/4167291185 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8a080582a0 con 0x7f8a180afb80 2024-03-04T09:34:11.816 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-04T09:34:11.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:11.836+0000 7f8a1da00640 1 -- 172.21.15.6:0/4167291185 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8a18149740 con 0x7f8a180afb80 2024-03-04T09:34:11.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:11.837+0000 7f8a0ea00640 1 -- 172.21.15.6:0/4167291185 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3465 (secure 0 0 0) 0x7f8a08057ba0 con 0x7f8a180afb80 2024-03-04T09:34:11.835 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:34:11.835 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:33:54.652601+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"laggy_since":"2024-03-04T09:33:54.652601+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":9}]} 2024-03-04T09:34:11.835 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 100 2024-03-04T09:34:11.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:11.840+0000 7f8a1da00640 1 -- 172.21.15.6:0/4167291185 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f89f806b4d0 msgr2=0x7f89f806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:11.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:11.840+0000 7f8a1da00640 1 --2- 172.21.15.6:0/4167291185 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f89f806b4d0 0x7f89f806d990 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f8a1815c4f0 tx=0x7f8a04008040 comp rx=0 tx=0).stop 2024-03-04T09:34:11.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:11.840+0000 7f8a1da00640 1 -- 172.21.15.6:0/4167291185 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8a180afb80 msgr2=0x7f8a1814ca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:11.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:11.840+0000 7f8a1da00640 1 --2- 172.21.15.6:0/4167291185 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8a180afb80 0x7f8a1814ca40 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 crypto rx=0x7f8a0800c4d0 tx=0x7f8a0800c990 comp rx=0 tx=0).stop 2024-03-04T09:34:11.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:11.841+0000 7f8a1da00640 1 -- 172.21.15.6:0/4167291185 shutdown_connections 2024-03-04T09:34:11.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:11.841+0000 7f8a1da00640 1 --2- 172.21.15.6:0/4167291185 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8a1815b6a0 0x7f8a1815c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:11.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:11.841+0000 7f8a1da00640 1 --2- 172.21.15.6:0/4167291185 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f89f806b4d0 0x7f89f806d990 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:11.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:11.841+0000 7f8a1da00640 1 --2- 172.21.15.6:0/4167291185 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8a1815b2c0 0x7f8a181450c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:11.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:11.841+0000 7f8a1da00640 1 --2- 172.21.15.6:0/4167291185 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8a180afb80 0x7f8a1814ca40 unknown :-1 s=CLOSED pgs=1612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:11.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:11.841+0000 7f8a1da00640 1 -- 172.21.15.6:0/4167291185 >> 172.21.15.6:0/4167291185 conn(0x7f8a180b2640 msgr2=0x7f8a180b2a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:34:11.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:11.841+0000 7f8a1da00640 1 -- 172.21.15.6:0/4167291185 shutdown_connections 2024-03-04T09:34:11.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:11.841+0000 7f8a1da00640 1 -- 172.21.15.6:0/4167291185 wait complete. 2024-03-04T09:34:12.061 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:32:57.159681+0000', 'modified': '2024-03-04T09:33:54.652601+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8214, 'mds_1': 6796}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6796': {'gid': 6796, 'name': 'd', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.146:6837/2458070840', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 2458070840}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 2458070840}]}, 'join_fscid': -1, 'laggy_since': '2024-03-04T09:33:54.652601+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8214': {'gid': 8214, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2446627517', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2446627517}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2446627517}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8214, 'qdb_cluster': [8214]} 2024-03-04T09:34:12.061 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6796:up:stopping 2024-03-04T09:34:13.061 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:34:13.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:13.299+0000 7ff570e00640 1 Processor -- start 2024-03-04T09:34:13.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:13.299+0000 7ff570e00640 1 -- start start 2024-03-04T09:34:13.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:13.301+0000 7ff570e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff56c0cd060 0x7ff56c161ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:13.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:13.301+0000 7ff570e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff56c1627f0 con 0x7ff56c162000 2024-03-04T09:34:13.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:13.301+0000 7ff570e00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff56c06c860 con 0x7ff56c15efc0 2024-03-04T09:34:13.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:13.301+0000 7ff570e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff56c15f3a0 con 0x7ff56c0cd060 2024-03-04T09:34:13.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:13.301+0000 7ff56be00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7ff56c162000 0x7ff56c070c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:55504/0 (socket says 172.21.15.6:55504) 2024-03-04T09:34:13.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:13.301+0000 7ff56be00640 1 -- 172.21.15.6:0/2486728016 learned_addr learned my addr 172.21.15.6:0/2486728016 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:34:13.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:13.302+0000 7ff563e00640 1 --2- 172.21.15.6:0/2486728016 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff56c0cd060 0x7ff56c161ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:13.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:13.302+0000 7ff56aa00640 1 -- 172.21.15.6:0/2486728016 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 851001857 0 0) 0x7ff56c1627f0 con 0x7ff56c162000 2024-03-04T09:34:13.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:13.302+0000 7ff56aa00640 1 -- 172.21.15.6:0/2486728016 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff550003520 con 0x7ff56c162000 2024-03-04T09:34:13.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:13.302+0000 7ff56aa00640 1 -- 172.21.15.6:0/2486728016 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 153401001 0 0) 0x7ff56c06c860 con 0x7ff56c15efc0 2024-03-04T09:34:13.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:13.302+0000 7ff56aa00640 1 -- 172.21.15.6:0/2486728016 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff56c1627f0 con 0x7ff56c15efc0 2024-03-04T09:34:13.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:13.302+0000 7ff56aa00640 1 -- 172.21.15.6:0/2486728016 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1313130515 0 0) 0x7ff550003520 con 0x7ff56c162000 2024-03-04T09:34:13.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:13.302+0000 7ff56aa00640 1 -- 172.21.15.6:0/2486728016 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff56c0cd060 msgr2=0x7ff56c161ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:34:13.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:13.302+0000 7ff56aa00640 1 --2- 172.21.15.6:0/2486728016 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff56c0cd060 0x7ff56c161ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:13.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:13.302+0000 7ff56aa00640 1 -- 172.21.15.6:0/2486728016 >> v1:172.21.15.146:6789/0 conn(0x7ff56c15efc0 legacy=0x7ff56c0cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:13.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:13.302+0000 7ff56aa00640 1 -- 172.21.15.6:0/2486728016 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff56c06cb40 con 0x7ff56c162000 2024-03-04T09:34:13.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:13.303+0000 7ff56aa00640 1 -- 172.21.15.6:0/2486728016 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7ff5640019f0 con 0x7ff56c162000 2024-03-04T09:34:13.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:13.303+0000 7ff56aa00640 1 -- 172.21.15.6:0/2486728016 >> v1:172.21.15.6:6789/0 conn(0x7ff56c162000 legacy=0x7ff56c070c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:13.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:13.303+0000 7ff56aa00640 1 --2- 172.21.15.6:0/2486728016 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff550003b70 0x7ff550003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:13.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:13.303+0000 7ff56aa00640 1 --2- 172.21.15.6:0/2486728016 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff550004800 0x7ff550004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:13.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:13.303+0000 7ff56aa00640 1 --2- 172.21.15.6:0/2486728016 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff56c0cd060 0x7ff5500051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:13.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:13.303+0000 7ff56aa00640 1 -- 172.21.15.6:0/2486728016 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff56c15f3a0 con 0x7ff550003b70 2024-03-04T09:34:13.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:13.303+0000 7ff56aa00640 1 -- 172.21.15.6:0/2486728016 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff55000ba70 con 0x7ff550004800 2024-03-04T09:34:13.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:13.303+0000 7ff56aa00640 1 -- 172.21.15.6:0/2486728016 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff55000bc70 con 0x7ff56c0cd060 2024-03-04T09:34:13.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:13.303+0000 7ff56b400640 1 --2- 172.21.15.6:0/2486728016 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff550003b70 0x7ff550003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:13.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:13.303+0000 7ff56b400640 1 -- 172.21.15.6:0/2486728016 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff56c0cd060 msgr2=0x7ff5500051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:34:13.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:13.303+0000 7ff56b400640 1 --2- 172.21.15.6:0/2486728016 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff56c0cd060 0x7ff5500051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:13.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:13.303+0000 7ff56b400640 1 -- 172.21.15.6:0/2486728016 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff550004800 msgr2=0x7ff550004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:34:13.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:13.303+0000 7ff56b400640 1 --2- 172.21.15.6:0/2486728016 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff550004800 0x7ff550004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:13.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:13.303+0000 7ff56b400640 1 -- 172.21.15.6:0/2486728016 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff550003520 con 0x7ff550003b70 2024-03-04T09:34:13.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:13.304+0000 7ff56b400640 1 --2- 172.21.15.6:0/2486728016 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff550003b70 0x7ff550003f70 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 crypto rx=0x7ff55c0018d0 tx=0x7ff550005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6841750ba20ad5b server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:34:13.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:13.305+0000 7ff56aa00640 1 -- 172.21.15.6:0/2486728016 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff55c037070 con 0x7ff550003b70 2024-03-04T09:34:13.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:13.305+0000 7ff56aa00640 1 -- 172.21.15.6:0/2486728016 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff55c0052d0 con 0x7ff550003b70 2024-03-04T09:34:13.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:13.305+0000 7ff56aa00640 1 -- 172.21.15.6:0/2486728016 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff55c0014f0 con 0x7ff550003b70 2024-03-04T09:34:13.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:13.305+0000 7ff570e00640 1 -- 172.21.15.6:0/2486728016 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff550003b70 msgr2=0x7ff550003f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:13.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:13.305+0000 7ff570e00640 1 --2- 172.21.15.6:0/2486728016 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff550003b70 0x7ff550003f70 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 crypto rx=0x7ff55c0018d0 tx=0x7ff550005830 comp rx=0 tx=0).stop 2024-03-04T09:34:13.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:13.305+0000 7ff56be00640 1 -- 172.21.15.6:0/2486728016 reap_dead start 2024-03-04T09:34:13.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:13.305+0000 7ff570e00640 1 -- 172.21.15.6:0/2486728016 shutdown_connections 2024-03-04T09:34:13.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:13.305+0000 7ff570e00640 1 -- 172.21.15.6:0/2486728016 >> 172.21.15.6:0/2486728016 conn(0x7ff56c06b3d0 msgr2=0x7ff56c06b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:34:13.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:13.306+0000 7ff570e00640 1 -- 172.21.15.6:0/2486728016 shutdown_connections 2024-03-04T09:34:13.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:13.306+0000 7ff570e00640 1 -- 172.21.15.6:0/2486728016 wait complete. 2024-03-04T09:34:13.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:13.307+0000 7ff570e00640 1 Processor -- start 2024-03-04T09:34:13.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:13.307+0000 7ff570e00640 1 -- start start 2024-03-04T09:34:13.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:13.308+0000 7ff570e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff550003b70 0x7ff56c1f3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:13.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:13.308+0000 7ff570e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff56c0af750 0x7ff56c0ce1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:13.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:13.308+0000 7ff570e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff56c0afbd0 0x7ff56c0ce730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:13.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:13.308+0000 7ff570e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff56c1624b0 con 0x7ff550003b70 2024-03-04T09:34:13.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:13.308+0000 7ff570e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff56c06d190 con 0x7ff56c0afbd0 2024-03-04T09:34:13.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:13.308+0000 7ff570e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff56c06d620 con 0x7ff56c0af750 2024-03-04T09:34:13.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:13.309+0000 7ff56b400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff550003b70 0x7ff56c1f3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:13.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:13.309+0000 7ff56b400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff550003b70 0x7ff56c1f3610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:52582/0 (socket says 172.21.15.6:52582) 2024-03-04T09:34:13.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:13.309+0000 7ff56b400640 1 -- 172.21.15.6:0/3047977077 learned_addr learned my addr 172.21.15.6:0/3047977077 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:34:13.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:13.309+0000 7ff56be00640 1 --2- 172.21.15.6:0/3047977077 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff56c0afbd0 0x7ff56c0ce730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:13.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:13.309+0000 7ff563e00640 1 --2- 172.21.15.6:0/3047977077 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff56c0af750 0x7ff56c0ce1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:13.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:13.309+0000 7ff56b400640 1 -- 172.21.15.6:0/3047977077 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff56c0af750 msgr2=0x7ff56c0ce1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:13.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:13.309+0000 7ff56b400640 1 --2- 172.21.15.6:0/3047977077 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff56c0af750 0x7ff56c0ce1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:13.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:13.309+0000 7ff56b400640 1 -- 172.21.15.6:0/3047977077 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff56c0afbd0 msgr2=0x7ff56c0ce730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:13.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:13.309+0000 7ff56b400640 1 --2- 172.21.15.6:0/3047977077 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff56c0afbd0 0x7ff56c0ce730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:13.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:13.309+0000 7ff56b400640 1 -- 172.21.15.6:0/3047977077 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff56c1f3f70 con 0x7ff550003b70 2024-03-04T09:34:13.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:13.309+0000 7ff56be00640 1 --2- 172.21.15.6:0/3047977077 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff56c0afbd0 0x7ff56c0ce730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:34:13.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:13.309+0000 7ff563e00640 1 --2- 172.21.15.6:0/3047977077 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff56c0af750 0x7ff56c0ce1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:34:13.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:13.309+0000 7ff56b400640 1 --2- 172.21.15.6:0/3047977077 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff550003b70 0x7ff56c1f3610 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 crypto rx=0x7ff55c0018d0 tx=0x7ff55c0012d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:34:13.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:13.310+0000 7ff563400640 1 -- 172.21.15.6:0/3047977077 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff55c042070 con 0x7ff550003b70 2024-03-04T09:34:13.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:13.310+0000 7ff563400640 1 -- 172.21.15.6:0/3047977077 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff55c005480 con 0x7ff550003b70 2024-03-04T09:34:13.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:13.310+0000 7ff563400640 1 -- 172.21.15.6:0/3047977077 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff55c037040 con 0x7ff550003b70 2024-03-04T09:34:13.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:13.310+0000 7ff570e00640 1 -- 172.21.15.6:0/3047977077 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff56c0b0130 con 0x7ff550003b70 2024-03-04T09:34:13.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:13.310+0000 7ff570e00640 1 -- 172.21.15.6:0/3047977077 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff56c0b06f0 con 0x7ff550003b70 2024-03-04T09:34:13.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:13.311+0000 7ff570e00640 1 -- 172.21.15.6:0/3047977077 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff56c04f330 con 0x7ff550003b70 2024-03-04T09:34:13.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:13.312+0000 7ff563400640 1 -- 172.21.15.6:0/3047977077 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff55c045070 con 0x7ff550003b70 2024-03-04T09:34:13.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:13.313+0000 7ff563400640 1 --2- 172.21.15.6:0/3047977077 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff54806b500 0x7ff54806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:13.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:13.313+0000 7ff563400640 1 -- 172.21.15.6:0/3047977077 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7ff55c0abd30 con 0x7ff550003b70 2024-03-04T09:34:13.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:13.313+0000 7ff563e00640 1 --2- 172.21.15.6:0/3047977077 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff54806b500 0x7ff54806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:13.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:13.315+0000 7ff563e00640 1 --2- 172.21.15.6:0/3047977077 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff54806b500 0x7ff54806d9c0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7ff56c1f7330 tx=0x7ff554008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:34:13.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:13.317+0000 7ff563400640 1 -- 172.21.15.6:0/3047977077 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff55c07a5a0 con 0x7ff550003b70 2024-03-04T09:34:13.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:13.556+0000 7ff570e00640 1 -- 172.21.15.6:0/3047977077 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff56c0af2f0 con 0x7ff550003b70 2024-03-04T09:34:13.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:13.557+0000 7ff563400640 1 -- 172.21.15.6:0/3047977077 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3465 (secure 0 0 0) 0x7ff55c07e640 con 0x7ff550003b70 2024-03-04T09:34:13.554 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:34:13.554 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:33:54.652601+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"laggy_since":"2024-03-04T09:33:54.652601+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":9}]} 2024-03-04T09:34:13.555 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 100 2024-03-04T09:34:13.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:13.560+0000 7ff570e00640 1 -- 172.21.15.6:0/3047977077 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff54806b500 msgr2=0x7ff54806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:13.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:13.560+0000 7ff570e00640 1 --2- 172.21.15.6:0/3047977077 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff54806b500 0x7ff54806d9c0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7ff56c1f7330 tx=0x7ff554008040 comp rx=0 tx=0).stop 2024-03-04T09:34:13.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:13.560+0000 7ff570e00640 1 -- 172.21.15.6:0/3047977077 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff550003b70 msgr2=0x7ff56c1f3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:13.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:13.560+0000 7ff570e00640 1 --2- 172.21.15.6:0/3047977077 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff550003b70 0x7ff56c1f3610 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 crypto rx=0x7ff55c0018d0 tx=0x7ff55c0012d0 comp rx=0 tx=0).stop 2024-03-04T09:34:13.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:13.560+0000 7ff570e00640 1 -- 172.21.15.6:0/3047977077 shutdown_connections 2024-03-04T09:34:13.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:13.560+0000 7ff570e00640 1 --2- 172.21.15.6:0/3047977077 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff56c0afbd0 0x7ff56c0ce730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:13.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:13.560+0000 7ff570e00640 1 --2- 172.21.15.6:0/3047977077 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff54806b500 0x7ff54806d9c0 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:13.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:13.560+0000 7ff570e00640 1 --2- 172.21.15.6:0/3047977077 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff56c0af750 0x7ff56c0ce1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:13.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:13.560+0000 7ff570e00640 1 --2- 172.21.15.6:0/3047977077 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff550003b70 0x7ff56c1f3610 unknown :-1 s=CLOSED pgs=1615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:13.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:13.560+0000 7ff570e00640 1 -- 172.21.15.6:0/3047977077 >> 172.21.15.6:0/3047977077 conn(0x7ff56c06b3d0 msgr2=0x7ff56c1467e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:34:13.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:13.560+0000 7ff570e00640 1 -- 172.21.15.6:0/3047977077 shutdown_connections 2024-03-04T09:34:13.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:13.560+0000 7ff570e00640 1 -- 172.21.15.6:0/3047977077 wait complete. 2024-03-04T09:34:13.780 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:32:57.159681+0000', 'modified': '2024-03-04T09:33:54.652601+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8214, 'mds_1': 6796}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6796': {'gid': 6796, 'name': 'd', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.146:6837/2458070840', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 2458070840}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 2458070840}]}, 'join_fscid': -1, 'laggy_since': '2024-03-04T09:33:54.652601+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8214': {'gid': 8214, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2446627517', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2446627517}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2446627517}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8214, 'qdb_cluster': [8214]} 2024-03-04T09:34:13.780 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6796:up:stopping 2024-03-04T09:34:13.818 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-04T09:34:14.781 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:34:15.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:15.014+0000 7f19a6a00640 1 Processor -- start 2024-03-04T09:34:15.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:15.015+0000 7f19a6a00640 1 -- start start 2024-03-04T09:34:15.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:15.016+0000 7f19a6a00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f19a00d2960 0x7f19a00d05c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:15.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:15.016+0000 7f19a6a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f19a00d0b00 0x7f19a00b3630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:15.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:15.016+0000 7f19a6a00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19a00d2dc0 con 0x7f19a015e070 2024-03-04T09:34:15.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:15.016+0000 7f19a6a00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19a00d18c0 con 0x7f19a00d2960 2024-03-04T09:34:15.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:15.016+0000 7f19a6a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19a00d1ba0 con 0x7f19a00d0b00 2024-03-04T09:34:15.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:15.017+0000 7f19a6000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f19a00d0b00 0x7f19a00b3630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:15.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:15.017+0000 7f19a4c00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f19a00d2960 0x7f19a00d05c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:15.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:15.017+0000 7f19a6000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f19a00d0b00 0x7f19a00b3630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.6:47886/0 (socket says 172.21.15.6:47886) 2024-03-04T09:34:15.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:15.017+0000 7f19a6000640 1 -- 172.21.15.6:0/2964195401 learned_addr learned my addr 172.21.15.6:0/2964195401 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:34:15.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:15.018+0000 7f19a6000640 1 -- 172.21.15.6:0/2964195401 >> v2:172.21.15.146:3300/0 conn(0x7f19a00d2960 msgr2=0x7f19a00d05c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:34:15.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:15.018+0000 7f19a6000640 1 --2- 172.21.15.6:0/2964195401 >> v2:172.21.15.146:3300/0 conn(0x7f19a00d2960 0x7f19a00d05c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:15.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:15.018+0000 7f19a6000640 1 -- 172.21.15.6:0/2964195401 >> v1:172.21.15.146:6789/0 conn(0x7f19a015e070 legacy=0x7f19a00cf410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:15.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:15.018+0000 7f19a6000640 1 -- 172.21.15.6:0/2964195401 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19a00d1e80 con 0x7f19a00d0b00 2024-03-04T09:34:15.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:15.018+0000 7f19a4c00640 1 --2- 172.21.15.6:0/2964195401 >> v2:172.21.15.146:3300/0 conn(0x7f19a00d2960 0x7f19a00d05c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T09:34:15.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:15.019+0000 7f19a6000640 1 --2- 172.21.15.6:0/2964195401 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f19a00d0b00 0x7f19a00b3630 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f1998009870 tx=0x7f199802b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b55cbdbf61ea9f14 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:34:15.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:15.019+0000 7f199fe00640 1 -- 172.21.15.6:0/2964195401 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1998034070 con 0x7f19a00d0b00 2024-03-04T09:34:15.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:15.019+0000 7f199fe00640 1 -- 172.21.15.6:0/2964195401 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1998030840 con 0x7f19a00d0b00 2024-03-04T09:34:15.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:15.019+0000 7f199fe00640 1 -- 172.21.15.6:0/2964195401 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1998030c10 con 0x7f19a00d0b00 2024-03-04T09:34:15.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:15.020+0000 7f19a6a00640 1 -- 172.21.15.6:0/2964195401 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f19a00d0b00 msgr2=0x7f19a00b3630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:15.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:15.020+0000 7f19a6a00640 1 --2- 172.21.15.6:0/2964195401 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f19a00d0b00 0x7f19a00b3630 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f1998009870 tx=0x7f199802b980 comp rx=0 tx=0).stop 2024-03-04T09:34:15.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:15.020+0000 7f19a6a00640 1 -- 172.21.15.6:0/2964195401 shutdown_connections 2024-03-04T09:34:15.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:15.020+0000 7f19a6a00640 1 --2- 172.21.15.6:0/2964195401 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f19a00d0b00 0x7f19a00b3630 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:15.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:15.020+0000 7f19a6a00640 1 --2- 172.21.15.6:0/2964195401 >> v2:172.21.15.146:3300/0 conn(0x7f19a00d2960 0x7f19a00d05c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:15.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:15.020+0000 7f19a6a00640 1 -- 172.21.15.6:0/2964195401 >> 172.21.15.6:0/2964195401 conn(0x7f19a008d2a0 msgr2=0x7f19a00d3cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:34:15.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:15.020+0000 7f19a6a00640 1 -- 172.21.15.6:0/2964195401 shutdown_connections 2024-03-04T09:34:15.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:15.020+0000 7f19a6a00640 1 -- 172.21.15.6:0/2964195401 wait complete. 2024-03-04T09:34:15.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:15.022+0000 7f19a6a00640 1 Processor -- start 2024-03-04T09:34:15.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:15.022+0000 7f19a6a00640 1 -- start start 2024-03-04T09:34:15.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:15.023+0000 7f19a6a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f19a00d0b00 0x7f19a00c77e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:15.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:15.023+0000 7f19a6a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f19a00d2960 0x7f19a0160d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:15.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:15.023+0000 7f19a6a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f19a015e070 0x7f19a01f3dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:15.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:15.023+0000 7f19a6a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19a00d45b0 con 0x7f19a015e070 2024-03-04T09:34:15.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:15.023+0000 7f19a6a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19a00d2440 con 0x7f19a00d0b00 2024-03-04T09:34:15.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:15.023+0000 7f19a6a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19a00d4730 con 0x7f19a00d2960 2024-03-04T09:34:15.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:15.023+0000 7f19a6000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f19a015e070 0x7f19a01f3dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:15.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:15.023+0000 7f19a6000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f19a015e070 0x7f19a01f3dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:52598/0 (socket says 172.21.15.6:52598) 2024-03-04T09:34:15.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:15.023+0000 7f19a6000640 1 -- 172.21.15.6:0/4245141685 learned_addr learned my addr 172.21.15.6:0/4245141685 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:34:15.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:15.024+0000 7f19a4c00640 1 --2- 172.21.15.6:0/4245141685 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f19a00d2960 0x7f19a0160d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:15.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:15.024+0000 7f19a5600640 1 --2- 172.21.15.6:0/4245141685 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f19a00d0b00 0x7f19a00c77e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:15.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:15.024+0000 7f19a6000640 1 -- 172.21.15.6:0/4245141685 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f19a00d2960 msgr2=0x7f19a0160d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:15.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:15.024+0000 7f19a6000640 1 --2- 172.21.15.6:0/4245141685 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f19a00d2960 0x7f19a0160d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:15.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:15.024+0000 7f19a6000640 1 -- 172.21.15.6:0/4245141685 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f19a00d0b00 msgr2=0x7f19a00c77e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:15.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:15.024+0000 7f19a6000640 1 --2- 172.21.15.6:0/4245141685 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f19a00d0b00 0x7f19a00c77e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:15.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:15.024+0000 7f19a6000640 1 -- 172.21.15.6:0/4245141685 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19a00c8830 con 0x7f19a015e070 2024-03-04T09:34:15.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:15.024+0000 7f19a5600640 1 --2- 172.21.15.6:0/4245141685 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f19a00d0b00 0x7f19a00c77e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:34:15.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:15.024+0000 7f19a4c00640 1 --2- 172.21.15.6:0/4245141685 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f19a00d2960 0x7f19a0160d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:34:15.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:15.024+0000 7f19a6000640 1 --2- 172.21.15.6:0/4245141685 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f19a015e070 0x7f19a01f3dd0 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7f1998009870 tx=0x7f1998004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:34:15.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:15.024+0000 7f199d600640 1 -- 172.21.15.6:0/4245141685 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1998048070 con 0x7f19a015e070 2024-03-04T09:34:15.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:15.024+0000 7f199d600640 1 -- 172.21.15.6:0/4245141685 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1998030640 con 0x7f19a015e070 2024-03-04T09:34:15.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:15.024+0000 7f199d600640 1 -- 172.21.15.6:0/4245141685 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1998034040 con 0x7f19a015e070 2024-03-04T09:34:15.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:15.024+0000 7f19a6a00640 1 -- 172.21.15.6:0/4245141685 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19a00c8b10 con 0x7f19a015e070 2024-03-04T09:34:15.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:15.025+0000 7f19a6a00640 1 -- 172.21.15.6:0/4245141685 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19a015b190 con 0x7f19a015e070 2024-03-04T09:34:15.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:15.026+0000 7f19a6a00640 1 -- 172.21.15.6:0/4245141685 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19a004f330 con 0x7f19a015e070 2024-03-04T09:34:15.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:15.027+0000 7f199d600640 1 -- 172.21.15.6:0/4245141685 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1998037070 con 0x7f19a015e070 2024-03-04T09:34:15.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:15.028+0000 7f199d600640 1 --2- 172.21.15.6:0/4245141685 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f198006b4d0 0x7f198006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:15.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:15.028+0000 7f199d600640 1 -- 172.21.15.6:0/4245141685 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f19980abf80 con 0x7f19a015e070 2024-03-04T09:34:15.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:15.028+0000 7f19a5600640 1 --2- 172.21.15.6:0/4245141685 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f198006b4d0 0x7f198006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:15.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:15.029+0000 7f19a5600640 1 --2- 172.21.15.6:0/4245141685 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f198006b4d0 0x7f198006d990 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f1994001cd0 tx=0x7f1994009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:34:15.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:15.034+0000 7f199d600640 1 -- 172.21.15.6:0/4245141685 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f199807aed0 con 0x7f19a015e070 2024-03-04T09:34:15.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:15.276+0000 7f19a6a00640 1 -- 172.21.15.6:0/4245141685 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f19a00b3390 con 0x7f19a015e070 2024-03-04T09:34:15.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:15.277+0000 7f199d600640 1 -- 172.21.15.6:0/4245141685 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3465 (secure 0 0 0) 0x7f199807a7d0 con 0x7f19a015e070 2024-03-04T09:34:15.274 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:34:15.274 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:33:54.652601+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"laggy_since":"2024-03-04T09:33:54.652601+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":9}]} 2024-03-04T09:34:15.275 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 100 2024-03-04T09:34:15.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:15.279+0000 7f19a6a00640 1 -- 172.21.15.6:0/4245141685 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f198006b4d0 msgr2=0x7f198006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:15.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:15.280+0000 7f19a6a00640 1 --2- 172.21.15.6:0/4245141685 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f198006b4d0 0x7f198006d990 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f1994001cd0 tx=0x7f1994009040 comp rx=0 tx=0).stop 2024-03-04T09:34:15.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:15.280+0000 7f19a6a00640 1 -- 172.21.15.6:0/4245141685 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f19a015e070 msgr2=0x7f19a01f3dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:15.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:15.280+0000 7f19a6a00640 1 --2- 172.21.15.6:0/4245141685 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f19a015e070 0x7f19a01f3dd0 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7f1998009870 tx=0x7f1998004730 comp rx=0 tx=0).stop 2024-03-04T09:34:15.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:15.280+0000 7f19a6a00640 1 -- 172.21.15.6:0/4245141685 shutdown_connections 2024-03-04T09:34:15.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:15.280+0000 7f19a6a00640 1 --2- 172.21.15.6:0/4245141685 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f19a015e070 0x7f19a01f3dd0 unknown :-1 s=CLOSED pgs=1616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:15.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:15.280+0000 7f19a6a00640 1 --2- 172.21.15.6:0/4245141685 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f198006b4d0 0x7f198006d990 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:15.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:15.280+0000 7f19a6a00640 1 --2- 172.21.15.6:0/4245141685 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f19a00d2960 0x7f19a0160d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:15.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:15.280+0000 7f19a6a00640 1 --2- 172.21.15.6:0/4245141685 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f19a00d0b00 0x7f19a00c77e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:15.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:15.280+0000 7f19a6a00640 1 -- 172.21.15.6:0/4245141685 >> 172.21.15.6:0/4245141685 conn(0x7f19a008d2a0 msgr2=0x7f19a006c9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:34:15.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:15.280+0000 7f19a6a00640 1 -- 172.21.15.6:0/4245141685 shutdown_connections 2024-03-04T09:34:15.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:15.280+0000 7f19a6a00640 1 -- 172.21.15.6:0/4245141685 wait complete. 2024-03-04T09:34:15.496 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:32:57.159681+0000', 'modified': '2024-03-04T09:33:54.652601+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8214, 'mds_1': 6796}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6796': {'gid': 6796, 'name': 'd', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.146:6837/2458070840', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 2458070840}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 2458070840}]}, 'join_fscid': -1, 'laggy_since': '2024-03-04T09:33:54.652601+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8214': {'gid': 8214, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2446627517', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2446627517}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2446627517}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8214, 'qdb_cluster': [8214]} 2024-03-04T09:34:15.497 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6796:up:stopping 2024-03-04T09:34:15.819 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-04T09:34:16.498 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:34:16.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:16.730+0000 7f5a9dc00640 1 Processor -- start 2024-03-04T09:34:16.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:16.730+0000 7f5a9dc00640 1 -- start start 2024-03-04T09:34:16.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:16.732+0000 7f5a9dc00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f5a981477e0 0x7f5a98162420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:16.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:16.732+0000 7f5a9dc00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f5a9806ef20 0x7f5a9806afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:16.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:16.732+0000 7f5a9dc00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5a9806b4f0 0x7f5a980d0910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:16.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:16.732+0000 7f5a9dc00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a9806b960 con 0x7f5a9806ef20 2024-03-04T09:34:16.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:16.732+0000 7f5a9dc00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a98161f90 con 0x7f5a981477e0 2024-03-04T09:34:16.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:16.732+0000 7f5a9dc00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a98162960 con 0x7f5a9806b4f0 2024-03-04T09:34:16.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:16.732+0000 7f5a8fe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f5a9806ef20 0x7f5a9806afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:16.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:16.732+0000 7f5a8fe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f5a9806ef20 0x7f5a9806afb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:52608/0 (socket says 172.21.15.6:52608) 2024-03-04T09:34:16.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:16.732+0000 7f5a8fe00640 1 -- 172.21.15.6:0/2768383435 learned_addr learned my addr 172.21.15.6:0/2768383435 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:34:16.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:16.733+0000 7f5a97e00640 1 --2- 172.21.15.6:0/2768383435 >> v2:172.21.15.146:3300/0 conn(0x7f5a981477e0 0x7f5a98162420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:16.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:16.733+0000 7f5a9d200640 1 --2- 172.21.15.6:0/2768383435 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5a9806b4f0 0x7f5a980d0910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:16.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:16.733+0000 7f5a8fe00640 1 -- 172.21.15.6:0/2768383435 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5a9806b4f0 msgr2=0x7f5a980d0910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:34:16.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:16.733+0000 7f5a8fe00640 1 --2- 172.21.15.6:0/2768383435 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5a9806b4f0 0x7f5a980d0910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:16.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:16.733+0000 7f5a8fe00640 1 -- 172.21.15.6:0/2768383435 >> v2:172.21.15.146:3300/0 conn(0x7f5a981477e0 msgr2=0x7f5a98162420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:34:16.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:16.733+0000 7f5a8fe00640 1 --2- 172.21.15.6:0/2768383435 >> v2:172.21.15.146:3300/0 conn(0x7f5a981477e0 0x7f5a98162420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:16.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:16.733+0000 7f5a8fe00640 1 -- 172.21.15.6:0/2768383435 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a98147bc0 con 0x7f5a9806ef20 2024-03-04T09:34:16.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:16.735+0000 7f5a8fe00640 1 --2- 172.21.15.6:0/2768383435 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5a9806ef20 0x7f5a9806afb0 secure :-1 s=READY pgs=1617 cs=0 l=1 rev1=1 crypto rx=0x7f5a9806c050 tx=0x7f5a8002b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc1a773eaaaf2779 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:34:16.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:16.736+0000 7f5a97400640 1 -- 172.21.15.6:0/2768383435 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a800317c0 con 0x7f5a9806ef20 2024-03-04T09:34:16.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:16.736+0000 7f5a97400640 1 -- 172.21.15.6:0/2768383435 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a80037070 con 0x7f5a9806ef20 2024-03-04T09:34:16.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:16.736+0000 7f5a97400640 1 -- 172.21.15.6:0/2768383435 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a80031b80 con 0x7f5a9806ef20 2024-03-04T09:34:16.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:16.736+0000 7f5a9dc00640 1 -- 172.21.15.6:0/2768383435 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5a9806ef20 msgr2=0x7f5a9806afb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:16.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:16.736+0000 7f5a9dc00640 1 --2- 172.21.15.6:0/2768383435 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5a9806ef20 0x7f5a9806afb0 secure :-1 s=READY pgs=1617 cs=0 l=1 rev1=1 crypto rx=0x7f5a9806c050 tx=0x7f5a8002b860 comp rx=0 tx=0).stop 2024-03-04T09:34:16.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:16.737+0000 7f5a9dc00640 1 -- 172.21.15.6:0/2768383435 shutdown_connections 2024-03-04T09:34:16.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:16.737+0000 7f5a9dc00640 1 --2- 172.21.15.6:0/2768383435 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5a9806b4f0 0x7f5a980d0910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:16.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:16.737+0000 7f5a9dc00640 1 --2- 172.21.15.6:0/2768383435 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5a9806ef20 0x7f5a9806afb0 unknown :-1 s=CLOSED pgs=1617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:16.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:16.737+0000 7f5a9dc00640 1 --2- 172.21.15.6:0/2768383435 >> v2:172.21.15.146:3300/0 conn(0x7f5a981477e0 0x7f5a98162420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:16.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:16.737+0000 7f5a9dc00640 1 -- 172.21.15.6:0/2768383435 >> 172.21.15.6:0/2768383435 conn(0x7f5a9806c0c0 msgr2=0x7f5a9806c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:34:16.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:16.737+0000 7f5a9dc00640 1 -- 172.21.15.6:0/2768383435 shutdown_connections 2024-03-04T09:34:16.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:16.737+0000 7f5a9dc00640 1 -- 172.21.15.6:0/2768383435 wait complete. 2024-03-04T09:34:16.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:16.738+0000 7f5a9dc00640 1 Processor -- start 2024-03-04T09:34:16.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:16.739+0000 7f5a9dc00640 1 -- start start 2024-03-04T09:34:16.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:16.739+0000 7f5a9dc00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5a9806b4f0 0x7f5a9815d130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:16.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:16.740+0000 7f5a9dc00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5a9806ef20 0x7f5a9815e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:16.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:16.740+0000 7f5a9dc00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5a981477e0 0x7f5a98160970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:16.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:16.740+0000 7f5a9dc00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a980b4330 con 0x7f5a9806ef20 2024-03-04T09:34:16.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:16.740+0000 7f5a9dc00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a980b4050 con 0x7f5a9806b4f0 2024-03-04T09:34:16.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:16.740+0000 7f5a9dc00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a98161c20 con 0x7f5a981477e0 2024-03-04T09:34:16.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:16.740+0000 7f5a8fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5a9806ef20 0x7f5a9815e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:16.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:16.740+0000 7f5a8fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5a9806ef20 0x7f5a9815e530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:52618/0 (socket says 172.21.15.6:52618) 2024-03-04T09:34:16.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:16.740+0000 7f5a8fe00640 1 -- 172.21.15.6:0/2256810498 learned_addr learned my addr 172.21.15.6:0/2256810498 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:34:16.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:16.740+0000 7f5a97e00640 1 --2- 172.21.15.6:0/2256810498 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5a9806b4f0 0x7f5a9815d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:16.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:16.740+0000 7f5a9d200640 1 --2- 172.21.15.6:0/2256810498 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5a981477e0 0x7f5a98160970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:16.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:16.740+0000 7f5a8fe00640 1 -- 172.21.15.6:0/2256810498 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5a981477e0 msgr2=0x7f5a98160970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:16.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:16.740+0000 7f5a8fe00640 1 --2- 172.21.15.6:0/2256810498 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5a981477e0 0x7f5a98160970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:16.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:16.740+0000 7f5a8fe00640 1 -- 172.21.15.6:0/2256810498 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5a9806b4f0 msgr2=0x7f5a9815d130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:16.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:16.740+0000 7f5a8fe00640 1 --2- 172.21.15.6:0/2256810498 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5a9806b4f0 0x7f5a9815d130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:16.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:16.740+0000 7f5a8fe00640 1 -- 172.21.15.6:0/2256810498 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a98160090 con 0x7f5a9806ef20 2024-03-04T09:34:16.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:16.741+0000 7f5a8fe00640 1 --2- 172.21.15.6:0/2256810498 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5a9806ef20 0x7f5a9815e530 secure :-1 s=READY pgs=1618 cs=0 l=1 rev1=1 crypto rx=0x7f5a80000c00 tx=0x7f5a800046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:34:16.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:16.741+0000 7f5a94c00640 1 -- 172.21.15.6:0/2256810498 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a80032650 con 0x7f5a9806ef20 2024-03-04T09:34:16.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:16.741+0000 7f5a94c00640 1 -- 172.21.15.6:0/2256810498 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a80032810 con 0x7f5a9806ef20 2024-03-04T09:34:16.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:16.741+0000 7f5a94c00640 1 -- 172.21.15.6:0/2256810498 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a80032c00 con 0x7f5a9806ef20 2024-03-04T09:34:16.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:16.741+0000 7f5a9dc00640 1 -- 172.21.15.6:0/2256810498 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a9815eff0 con 0x7f5a9806ef20 2024-03-04T09:34:16.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:16.741+0000 7f5a9dc00640 1 -- 172.21.15.6:0/2256810498 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a9815f5b0 con 0x7f5a9806ef20 2024-03-04T09:34:16.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:16.743+0000 7f5a9dc00640 1 -- 172.21.15.6:0/2256810498 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a98067830 con 0x7f5a9806ef20 2024-03-04T09:34:16.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:16.744+0000 7f5a94c00640 1 -- 172.21.15.6:0/2256810498 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5a8004c060 con 0x7f5a9806ef20 2024-03-04T09:34:16.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:16.745+0000 7f5a94c00640 1 --2- 172.21.15.6:0/2256810498 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f5a7406b500 0x7f5a7406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:16.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:16.745+0000 7f5a94c00640 1 -- 172.21.15.6:0/2256810498 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f5a800abd70 con 0x7f5a9806ef20 2024-03-04T09:34:16.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:16.745+0000 7f5a97e00640 1 --2- 172.21.15.6:0/2256810498 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f5a7406b500 0x7f5a7406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:16.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:16.746+0000 7f5a97e00640 1 --2- 172.21.15.6:0/2256810498 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f5a7406b500 0x7f5a7406d9c0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f5a88006fd0 tx=0x7f5a88008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:34:16.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:16.754+0000 7f5a94c00640 1 -- 172.21.15.6:0/2256810498 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5a98067830 con 0x7f5a9806ef20 2024-03-04T09:34:16.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:16.991+0000 7f5a9dc00640 1 -- 172.21.15.6:0/2256810498 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5a9804f330 con 0x7f5a9806ef20 2024-03-04T09:34:16.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:16.991+0000 7f5a94c00640 1 -- 172.21.15.6:0/2256810498 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3465 (secure 0 0 0) 0x7f5a98160090 con 0x7f5a9806ef20 2024-03-04T09:34:16.988 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:34:16.988 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:33:54.652601+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"laggy_since":"2024-03-04T09:33:54.652601+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":9}]} 2024-03-04T09:34:16.989 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 100 2024-03-04T09:34:16.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:16.994+0000 7f5a9dc00640 1 -- 172.21.15.6:0/2256810498 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f5a7406b500 msgr2=0x7f5a7406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:16.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:16.994+0000 7f5a9dc00640 1 --2- 172.21.15.6:0/2256810498 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f5a7406b500 0x7f5a7406d9c0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f5a88006fd0 tx=0x7f5a88008040 comp rx=0 tx=0).stop 2024-03-04T09:34:16.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:16.994+0000 7f5a9dc00640 1 -- 172.21.15.6:0/2256810498 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5a9806ef20 msgr2=0x7f5a9815e530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:16.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:16.994+0000 7f5a9dc00640 1 --2- 172.21.15.6:0/2256810498 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5a9806ef20 0x7f5a9815e530 secure :-1 s=READY pgs=1618 cs=0 l=1 rev1=1 crypto rx=0x7f5a80000c00 tx=0x7f5a800046c0 comp rx=0 tx=0).stop 2024-03-04T09:34:16.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:16.994+0000 7f5a9dc00640 1 -- 172.21.15.6:0/2256810498 shutdown_connections 2024-03-04T09:34:16.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:16.994+0000 7f5a9dc00640 1 --2- 172.21.15.6:0/2256810498 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f5a7406b500 0x7f5a7406d9c0 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:16.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:16.994+0000 7f5a9dc00640 1 --2- 172.21.15.6:0/2256810498 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5a981477e0 0x7f5a98160970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:16.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:16.994+0000 7f5a9dc00640 1 --2- 172.21.15.6:0/2256810498 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5a9806ef20 0x7f5a9815e530 unknown :-1 s=CLOSED pgs=1618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:16.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:16.994+0000 7f5a9dc00640 1 --2- 172.21.15.6:0/2256810498 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5a9806b4f0 0x7f5a9815d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:16.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:16.994+0000 7f5a9dc00640 1 -- 172.21.15.6:0/2256810498 >> 172.21.15.6:0/2256810498 conn(0x7f5a9806c0c0 msgr2=0x7f5a9806c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:34:16.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:16.994+0000 7f5a9dc00640 1 -- 172.21.15.6:0/2256810498 shutdown_connections 2024-03-04T09:34:16.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:16.994+0000 7f5a9dc00640 1 -- 172.21.15.6:0/2256810498 wait complete. 2024-03-04T09:34:17.212 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:32:57.159681+0000', 'modified': '2024-03-04T09:33:54.652601+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8214, 'mds_1': 6796}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6796': {'gid': 6796, 'name': 'd', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.146:6837/2458070840', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 2458070840}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 2458070840}]}, 'join_fscid': -1, 'laggy_since': '2024-03-04T09:33:54.652601+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8214': {'gid': 8214, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2446627517', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2446627517}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2446627517}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8214, 'qdb_cluster': [8214]} 2024-03-04T09:34:17.212 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6796:up:stopping 2024-03-04T09:34:17.822 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-04T09:34:18.213 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:34:18.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:18.440+0000 7f7a94a00640 1 Processor -- start 2024-03-04T09:34:18.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:18.440+0000 7f7a94a00640 1 -- start start 2024-03-04T09:34:18.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:18.441+0000 7f7a94a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f7a9015e0b0 0x7f7a900cf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:18.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:18.441+0000 7f7a94a00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f7a900d4a10 0x7f7a900d0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:18.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:18.441+0000 7f7a94a00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a900b3710 con 0x7f7a900d0c80 2024-03-04T09:34:18.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:18.441+0000 7f7a94a00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a900d10f0 con 0x7f7a9015e0b0 2024-03-04T09:34:18.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:18.441+0000 7f7a94a00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a900b09e0 con 0x7f7a900d4a10 2024-03-04T09:34:18.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:18.442+0000 7f7a8f400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f7a9015e0b0 0x7f7a900cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:18.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:18.442+0000 7f7a8f400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f7a9015e0b0 0x7f7a900cf310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:52620/0 (socket says 172.21.15.6:52620) 2024-03-04T09:34:18.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:18.442+0000 7f7a8f400640 1 -- 172.21.15.6:0/3630692219 learned_addr learned my addr 172.21.15.6:0/3630692219 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:34:18.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:18.442+0000 7f7a87e00640 1 --2- 172.21.15.6:0/3630692219 >> v2:172.21.15.146:3300/0 conn(0x7f7a900d4a10 0x7f7a900d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:18.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:18.443+0000 7f7a8f400640 1 -- 172.21.15.6:0/3630692219 >> v2:172.21.15.146:3300/0 conn(0x7f7a900d4a10 msgr2=0x7f7a900d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:34:18.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:18.443+0000 7f7a8f400640 1 --2- 172.21.15.6:0/3630692219 >> v2:172.21.15.146:3300/0 conn(0x7f7a900d4a10 0x7f7a900d0740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:18.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:18.443+0000 7f7a8f400640 1 -- 172.21.15.6:0/3630692219 >> v1:172.21.15.146:6789/0 conn(0x7f7a900d0c80 legacy=0x7f7a900d1900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:34:18.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:18.443+0000 7f7a8f400640 1 -- 172.21.15.6:0/3630692219 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a900b39f0 con 0x7f7a9015e0b0 2024-03-04T09:34:18.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:18.443+0000 7f7a87e00640 1 --2- 172.21.15.6:0/3630692219 >> v2:172.21.15.146:3300/0 conn(0x7f7a900d4a10 0x7f7a900d0740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T09:34:18.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:18.444+0000 7f7a8f400640 1 --2- 172.21.15.6:0/3630692219 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7a9015e0b0 0x7f7a900cf310 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7f7a900b68c0 tx=0x7f7a8002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cc0f40c4ca9b2bb server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:34:18.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:18.444+0000 7f7a8ea00640 1 -- 172.21.15.6:0/3630692219 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a80035020 con 0x7f7a9015e0b0 2024-03-04T09:34:18.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:18.444+0000 7f7a8ea00640 1 -- 172.21.15.6:0/3630692219 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a80031830 con 0x7f7a9015e0b0 2024-03-04T09:34:18.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:18.444+0000 7f7a8ea00640 1 -- 172.21.15.6:0/3630692219 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a80031c00 con 0x7f7a9015e0b0 2024-03-04T09:34:18.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:18.445+0000 7f7a94a00640 1 -- 172.21.15.6:0/3630692219 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7a9015e0b0 msgr2=0x7f7a900cf310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:18.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:18.445+0000 7f7a94a00640 1 --2- 172.21.15.6:0/3630692219 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7a9015e0b0 0x7f7a900cf310 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7f7a900b68c0 tx=0x7f7a8002b8d0 comp rx=0 tx=0).stop 2024-03-04T09:34:18.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:18.445+0000 7f7a94a00640 1 -- 172.21.15.6:0/3630692219 shutdown_connections 2024-03-04T09:34:18.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:18.445+0000 7f7a94a00640 1 --2- 172.21.15.6:0/3630692219 >> v2:172.21.15.146:3300/0 conn(0x7f7a900d4a10 0x7f7a900d0740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:18.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:18.445+0000 7f7a94a00640 1 --2- 172.21.15.6:0/3630692219 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7a9015e0b0 0x7f7a900cf310 unknown :-1 s=CLOSED pgs=1619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:18.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:18.445+0000 7f7a94a00640 1 -- 172.21.15.6:0/3630692219 >> 172.21.15.6:0/3630692219 conn(0x7f7a900d3d30 msgr2=0x7f7a900d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:34:18.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:18.445+0000 7f7a94a00640 1 -- 172.21.15.6:0/3630692219 shutdown_connections 2024-03-04T09:34:18.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:18.445+0000 7f7a94a00640 1 -- 172.21.15.6:0/3630692219 wait complete. 2024-03-04T09:34:18.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:18.447+0000 7f7a94a00640 1 Processor -- start 2024-03-04T09:34:18.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:18.447+0000 7f7a94a00640 1 -- start start 2024-03-04T09:34:18.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:18.448+0000 7f7a94a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7a900d0c80 0x7f7a900cd730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:18.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:18.448+0000 7f7a94a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7a900d4a10 0x7f7a900b2520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:18.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:18.448+0000 7f7a94a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7a9015e0b0 0x7f7a90157020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:18.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:18.448+0000 7f7a94a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a9015e490 con 0x7f7a900d0c80 2024-03-04T09:34:18.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:18.448+0000 7f7a94a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a900b3fb0 con 0x7f7a9015e0b0 2024-03-04T09:34:18.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:18.448+0000 7f7a94a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a900cfc60 con 0x7f7a900d4a10 2024-03-04T09:34:18.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:18.448+0000 7f7a8f400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7a900d0c80 0x7f7a900cd730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:18.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:18.449+0000 7f7a8f400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7a900d0c80 0x7f7a900cd730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:52636/0 (socket says 172.21.15.6:52636) 2024-03-04T09:34:18.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:18.449+0000 7f7a8f400640 1 -- 172.21.15.6:0/1324646504 learned_addr learned my addr 172.21.15.6:0/1324646504 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:34:18.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:18.449+0000 7f7a8fe00640 1 --2- 172.21.15.6:0/1324646504 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7a9015e0b0 0x7f7a90157020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:18.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:18.449+0000 7f7a8f400640 1 -- 172.21.15.6:0/1324646504 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7a900d4a10 msgr2=0x7f7a900b2520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:18.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:18.449+0000 7f7a8f400640 1 --2- 172.21.15.6:0/1324646504 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7a900d4a10 0x7f7a900b2520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:18.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:18.449+0000 7f7a8f400640 1 -- 172.21.15.6:0/1324646504 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7a9015e0b0 msgr2=0x7f7a90157020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:18.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:18.449+0000 7f7a8f400640 1 --2- 172.21.15.6:0/1324646504 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7a9015e0b0 0x7f7a90157020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:18.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:18.449+0000 7f7a8f400640 1 -- 172.21.15.6:0/1324646504 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a900cbe50 con 0x7f7a900d0c80 2024-03-04T09:34:18.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:18.449+0000 7f7a8f400640 1 --2- 172.21.15.6:0/1324646504 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7a900d0c80 0x7f7a900cd730 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7f7a800097f0 tx=0x7f7a800046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:34:18.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:18.450+0000 7f7a87400640 1 -- 172.21.15.6:0/1324646504 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a8003a8b0 con 0x7f7a900d0c80 2024-03-04T09:34:18.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:18.450+0000 7f7a87400640 1 -- 172.21.15.6:0/1324646504 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a8003aa50 con 0x7f7a900d0c80 2024-03-04T09:34:18.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:18.450+0000 7f7a87400640 1 -- 172.21.15.6:0/1324646504 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a8003ae40 con 0x7f7a900d0c80 2024-03-04T09:34:18.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:18.450+0000 7f7a94a00640 1 -- 172.21.15.6:0/1324646504 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a900cc130 con 0x7f7a900d0c80 2024-03-04T09:34:18.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:18.450+0000 7f7a94a00640 1 -- 172.21.15.6:0/1324646504 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a900cc6f0 con 0x7f7a900d0c80 2024-03-04T09:34:18.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:18.451+0000 7f7a94a00640 1 -- 172.21.15.6:0/1324646504 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a9004f330 con 0x7f7a900d0c80 2024-03-04T09:34:18.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:18.452+0000 7f7a87400640 1 -- 172.21.15.6:0/1324646504 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7a8004b060 con 0x7f7a900d0c80 2024-03-04T09:34:18.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:18.453+0000 7f7a87400640 1 --2- 172.21.15.6:0/1324646504 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f7a6c06b500 0x7f7a6c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:18.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:18.453+0000 7f7a87400640 1 -- 172.21.15.6:0/1324646504 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f7a800abde0 con 0x7f7a900d0c80 2024-03-04T09:34:18.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:18.454+0000 7f7a87e00640 1 --2- 172.21.15.6:0/1324646504 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f7a6c06b500 0x7f7a6c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:18.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:18.455+0000 7f7a87e00640 1 --2- 172.21.15.6:0/1324646504 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f7a6c06b500 0x7f7a6c06d9c0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f7a78006e80 tx=0x7f7a78008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:34:18.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:18.457+0000 7f7a87400640 1 -- 172.21.15.6:0/1324646504 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7a8003f070 con 0x7f7a900d0c80 2024-03-04T09:34:18.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:18.828+0000 7f7a94a00640 1 -- 172.21.15.6:0/1324646504 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7a9006d4f0 con 0x7f7a900d0c80 2024-03-04T09:34:18.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:18.829+0000 7f7a87400640 1 -- 172.21.15.6:0/1324646504 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3465 (secure 0 0 0) 0x7f7a80031800 con 0x7f7a900d0c80 2024-03-04T09:34:18.826 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:34:18.826 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:33:54.652601+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"laggy_since":"2024-03-04T09:33:54.652601+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":9}]} 2024-03-04T09:34:18.827 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 100 2024-03-04T09:34:18.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:18.833+0000 7f7a94a00640 1 -- 172.21.15.6:0/1324646504 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f7a6c06b500 msgr2=0x7f7a6c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:18.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:18.833+0000 7f7a94a00640 1 --2- 172.21.15.6:0/1324646504 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f7a6c06b500 0x7f7a6c06d9c0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f7a78006e80 tx=0x7f7a78008040 comp rx=0 tx=0).stop 2024-03-04T09:34:18.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:18.833+0000 7f7a94a00640 1 -- 172.21.15.6:0/1324646504 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7a900d0c80 msgr2=0x7f7a900cd730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:18.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:18.833+0000 7f7a94a00640 1 --2- 172.21.15.6:0/1324646504 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7a900d0c80 0x7f7a900cd730 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7f7a800097f0 tx=0x7f7a800046c0 comp rx=0 tx=0).stop 2024-03-04T09:34:18.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:18.833+0000 7f7a94a00640 1 -- 172.21.15.6:0/1324646504 shutdown_connections 2024-03-04T09:34:18.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:18.833+0000 7f7a94a00640 1 --2- 172.21.15.6:0/1324646504 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7a9015e0b0 0x7f7a90157020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:18.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:18.833+0000 7f7a94a00640 1 --2- 172.21.15.6:0/1324646504 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f7a6c06b500 0x7f7a6c06d9c0 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:18.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:18.833+0000 7f7a94a00640 1 --2- 172.21.15.6:0/1324646504 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7a900d4a10 0x7f7a900b2520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:18.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:18.833+0000 7f7a94a00640 1 --2- 172.21.15.6:0/1324646504 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7a900d0c80 0x7f7a900cd730 unknown :-1 s=CLOSED pgs=1620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:18.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:18.833+0000 7f7a94a00640 1 -- 172.21.15.6:0/1324646504 >> 172.21.15.6:0/1324646504 conn(0x7f7a900d3d30 msgr2=0x7f7a900ce7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:34:18.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:18.833+0000 7f7a94a00640 1 -- 172.21.15.6:0/1324646504 shutdown_connections 2024-03-04T09:34:18.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:18.833+0000 7f7a94a00640 1 -- 172.21.15.6:0/1324646504 wait complete. 2024-03-04T09:34:19.059 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:32:57.159681+0000', 'modified': '2024-03-04T09:33:54.652601+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8214, 'mds_1': 6796}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6796': {'gid': 6796, 'name': 'd', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.146:6837/2458070840', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 2458070840}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 2458070840}]}, 'join_fscid': -1, 'laggy_since': '2024-03-04T09:33:54.652601+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8214': {'gid': 8214, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2446627517', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2446627517}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2446627517}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8214, 'qdb_cluster': [8214]} 2024-03-04T09:34:19.060 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6796:up:stopping 2024-03-04T09:34:19.824 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-04T09:34:20.061 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:34:20.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:20.284+0000 7f55fca00640 1 Processor -- start 2024-03-04T09:34:20.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:20.284+0000 7f55fca00640 1 -- start start 2024-03-04T09:34:20.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:20.285+0000 7f55fca00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f55f815e0b0 0x7f55f80cf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:20.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:20.285+0000 7f55fca00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f55f80d0e50 0x7f55f80b44d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:20.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:20.286+0000 7f55fca00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55f80b3710 con 0x7f55f80d4a10 2024-03-04T09:34:20.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:20.286+0000 7f55fca00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55f80b09e0 con 0x7f55f80d0e50 2024-03-04T09:34:20.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:20.286+0000 7f55fca00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55f80b39f0 con 0x7f55f815e0b0 2024-03-04T09:34:20.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:20.286+0000 7f55f7200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f55f80d0e50 0x7f55f80b44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:20.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:20.286+0000 7f55f7200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f55f80d0e50 0x7f55f80b44d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51592/0 (socket says 172.21.15.6:51592) 2024-03-04T09:34:20.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:20.286+0000 7f55f7200640 1 -- 172.21.15.6:0/612809258 learned_addr learned my addr 172.21.15.6:0/612809258 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:34:20.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:20.286+0000 7f55f6800640 1 --2- 172.21.15.6:0/612809258 >> v2:172.21.15.146:3300/0 conn(0x7f55f815e0b0 0x7f55f80cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:20.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:20.286+0000 7f55f7200640 1 -- 172.21.15.6:0/612809258 >> v2:172.21.15.146:3300/0 conn(0x7f55f815e0b0 msgr2=0x7f55f80cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:34:20.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:20.286+0000 7f55f7200640 1 --2- 172.21.15.6:0/612809258 >> v2:172.21.15.146:3300/0 conn(0x7f55f815e0b0 0x7f55f80cf310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:20.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:20.286+0000 7f55f7200640 1 -- 172.21.15.6:0/612809258 >> v1:172.21.15.146:6789/0 conn(0x7f55f80d4a10 legacy=0x7f55f80d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:34:20.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:20.287+0000 7f55f7200640 1 -- 172.21.15.6:0/612809258 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55f80b3cd0 con 0x7f55f80d0e50 2024-03-04T09:34:20.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:20.288+0000 7f55f7200640 1 --2- 172.21.15.6:0/612809258 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f55f80d0e50 0x7f55f80b44d0 secure :-1 s=READY pgs=1621 cs=0 l=1 rev1=1 crypto rx=0x7f55f80b68c0 tx=0x7f55f002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad733b611ab48c25 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:34:20.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:20.288+0000 7f55f5400640 1 -- 172.21.15.6:0/612809258 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55f002ca80 con 0x7f55f80d0e50 2024-03-04T09:34:20.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:20.288+0000 7f55f5400640 1 -- 172.21.15.6:0/612809258 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55f002cc20 con 0x7f55f80d0e50 2024-03-04T09:34:20.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:20.288+0000 7f55f5400640 1 -- 172.21.15.6:0/612809258 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55f00343f0 con 0x7f55f80d0e50 2024-03-04T09:34:20.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:20.289+0000 7f55fca00640 1 -- 172.21.15.6:0/612809258 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f55f80d0e50 msgr2=0x7f55f80b44d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:20.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:20.289+0000 7f55fca00640 1 --2- 172.21.15.6:0/612809258 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f55f80d0e50 0x7f55f80b44d0 secure :-1 s=READY pgs=1621 cs=0 l=1 rev1=1 crypto rx=0x7f55f80b68c0 tx=0x7f55f002bee0 comp rx=0 tx=0).stop 2024-03-04T09:34:20.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:20.289+0000 7f55fca00640 1 -- 172.21.15.6:0/612809258 shutdown_connections 2024-03-04T09:34:20.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:20.289+0000 7f55fca00640 1 --2- 172.21.15.6:0/612809258 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f55f80d0e50 0x7f55f80b44d0 unknown :-1 s=CLOSED pgs=1621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:20.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:20.289+0000 7f55fca00640 1 --2- 172.21.15.6:0/612809258 >> v2:172.21.15.146:3300/0 conn(0x7f55f815e0b0 0x7f55f80cf310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:20.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:20.289+0000 7f55fca00640 1 -- 172.21.15.6:0/612809258 >> 172.21.15.6:0/612809258 conn(0x7f55f80d3d30 msgr2=0x7f55f80d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:34:20.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:20.289+0000 7f55fca00640 1 -- 172.21.15.6:0/612809258 shutdown_connections 2024-03-04T09:34:20.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:20.289+0000 7f55fca00640 1 -- 172.21.15.6:0/612809258 wait complete. 2024-03-04T09:34:20.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:20.291+0000 7f55fca00640 1 Processor -- start 2024-03-04T09:34:20.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:20.291+0000 7f55fca00640 1 -- start start 2024-03-04T09:34:20.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:20.292+0000 7f55fca00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f55f80d0e50 0x7f55f80cb7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:20.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:20.292+0000 7f55fca00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f55f80d4a10 0x7f55f815a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:20.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:20.292+0000 7f55fca00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f55f815e0b0 0x7f55f80cd740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:20.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:20.292+0000 7f55fca00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55f80cfc60 con 0x7f55f80d0e50 2024-03-04T09:34:20.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:20.292+0000 7f55fca00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55f815e490 con 0x7f55f80d4a10 2024-03-04T09:34:20.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:20.292+0000 7f55fca00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55f80cf850 con 0x7f55f815e0b0 2024-03-04T09:34:20.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:20.292+0000 7f55f6800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f55f80d0e50 0x7f55f80cb7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:20.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:20.292+0000 7f55f6800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f55f80d0e50 0x7f55f80cb7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51606/0 (socket says 172.21.15.6:51606) 2024-03-04T09:34:20.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:20.292+0000 7f55f6800640 1 -- 172.21.15.6:0/2843222405 learned_addr learned my addr 172.21.15.6:0/2843222405 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:34:20.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:20.292+0000 7f55f6800640 1 -- 172.21.15.6:0/2843222405 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f55f815e0b0 msgr2=0x7f55f80cd740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:20.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:20.292+0000 7f55f6800640 1 --2- 172.21.15.6:0/2843222405 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f55f815e0b0 0x7f55f80cd740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:20.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:20.292+0000 7f55f6800640 1 -- 172.21.15.6:0/2843222405 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f55f80d4a10 msgr2=0x7f55f815a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:20.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:20.292+0000 7f55f6800640 1 --2- 172.21.15.6:0/2843222405 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f55f80d4a10 0x7f55f815a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:20.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:20.292+0000 7f55f6800640 1 -- 172.21.15.6:0/2843222405 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55f815ac50 con 0x7f55f80d0e50 2024-03-04T09:34:20.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:20.293+0000 7f55f6800640 1 --2- 172.21.15.6:0/2843222405 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f55f80d0e50 0x7f55f80cb7e0 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7f55e800d5f0 tx=0x7f55e800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:34:20.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:20.293+0000 7f55eea00640 1 -- 172.21.15.6:0/2843222405 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55e80125f0 con 0x7f55f80d0e50 2024-03-04T09:34:20.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:20.293+0000 7f55eea00640 1 -- 172.21.15.6:0/2843222405 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55e8019070 con 0x7f55f80d0e50 2024-03-04T09:34:20.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:20.293+0000 7f55eea00640 1 -- 172.21.15.6:0/2843222405 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55e80129b0 con 0x7f55f80d0e50 2024-03-04T09:34:20.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:20.293+0000 7f55fca00640 1 -- 172.21.15.6:0/2843222405 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55f80cbdd0 con 0x7f55f80d0e50 2024-03-04T09:34:20.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:20.293+0000 7f55fca00640 1 -- 172.21.15.6:0/2843222405 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55f80cc390 con 0x7f55f80d0e50 2024-03-04T09:34:20.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:20.295+0000 7f55fca00640 1 -- 172.21.15.6:0/2843222405 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55f804f330 con 0x7f55f80d0e50 2024-03-04T09:34:20.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:20.295+0000 7f55eea00640 1 -- 172.21.15.6:0/2843222405 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f55e8012b50 con 0x7f55f80d0e50 2024-03-04T09:34:20.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:20.296+0000 7f55eea00640 1 --2- 172.21.15.6:0/2843222405 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f55d806b4d0 0x7f55d806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:20.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:20.296+0000 7f55eea00640 1 -- 172.21.15.6:0/2843222405 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f55e808a660 con 0x7f55f80d0e50 2024-03-04T09:34:20.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:20.297+0000 7f55f5e00640 1 --2- 172.21.15.6:0/2843222405 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f55d806b4d0 0x7f55d806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:20.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:20.298+0000 7f55f5e00640 1 --2- 172.21.15.6:0/2843222405 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f55d806b4d0 0x7f55d806d990 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f55f80ca1a0 tx=0x7f55e40079b0 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:34:20.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:20.301+0000 7f55eea00640 1 -- 172.21.15.6:0/2843222405 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f55e80594a0 con 0x7f55f80d0e50 2024-03-04T09:34:20.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:20.538+0000 7f55fca00640 1 -- 172.21.15.6:0/2843222405 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f55f80ce8e0 con 0x7f55f80d0e50 2024-03-04T09:34:20.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:20.539+0000 7f55eea00640 1 -- 172.21.15.6:0/2843222405 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3465 (secure 0 0 0) 0x7f55e8058da0 con 0x7f55f80d0e50 2024-03-04T09:34:20.536 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:34:20.537 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:33:54.652601+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"laggy_since":"2024-03-04T09:33:54.652601+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":9}]} 2024-03-04T09:34:20.537 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 100 2024-03-04T09:34:20.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:20.542+0000 7f55fca00640 1 -- 172.21.15.6:0/2843222405 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f55d806b4d0 msgr2=0x7f55d806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:20.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:20.542+0000 7f55fca00640 1 --2- 172.21.15.6:0/2843222405 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f55d806b4d0 0x7f55d806d990 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f55f80ca1a0 tx=0x7f55e40079b0 comp rx=0 tx=0).stop 2024-03-04T09:34:20.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:20.542+0000 7f55fca00640 1 -- 172.21.15.6:0/2843222405 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f55f80d0e50 msgr2=0x7f55f80cb7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:20.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:20.542+0000 7f55fca00640 1 --2- 172.21.15.6:0/2843222405 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f55f80d0e50 0x7f55f80cb7e0 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7f55e800d5f0 tx=0x7f55e800dab0 comp rx=0 tx=0).stop 2024-03-04T09:34:20.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:20.542+0000 7f55fca00640 1 -- 172.21.15.6:0/2843222405 shutdown_connections 2024-03-04T09:34:20.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:20.542+0000 7f55fca00640 1 --2- 172.21.15.6:0/2843222405 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f55d806b4d0 0x7f55d806d990 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:20.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:20.542+0000 7f55fca00640 1 --2- 172.21.15.6:0/2843222405 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f55f815e0b0 0x7f55f80cd740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:20.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:20.542+0000 7f55fca00640 1 --2- 172.21.15.6:0/2843222405 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f55f80d4a10 0x7f55f815a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:20.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:20.542+0000 7f55fca00640 1 --2- 172.21.15.6:0/2843222405 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f55f80d0e50 0x7f55f80cb7e0 unknown :-1 s=CLOSED pgs=1622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:20.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:20.542+0000 7f55fca00640 1 -- 172.21.15.6:0/2843222405 >> 172.21.15.6:0/2843222405 conn(0x7f55f80d3d30 msgr2=0x7f55f80b21a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:34:20.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:20.542+0000 7f55fca00640 1 -- 172.21.15.6:0/2843222405 shutdown_connections 2024-03-04T09:34:20.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:20.542+0000 7f55fca00640 1 -- 172.21.15.6:0/2843222405 wait complete. 2024-03-04T09:34:20.760 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:32:57.159681+0000', 'modified': '2024-03-04T09:33:54.652601+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8214, 'mds_1': 6796}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6796': {'gid': 6796, 'name': 'd', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.146:6837/2458070840', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 2458070840}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 2458070840}]}, 'join_fscid': -1, 'laggy_since': '2024-03-04T09:33:54.652601+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8214': {'gid': 8214, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2446627517', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2446627517}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2446627517}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8214, 'qdb_cluster': [8214]} 2024-03-04T09:34:20.760 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6796:up:stopping 2024-03-04T09:34:21.761 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:34:21.826 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-04T09:34:22.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:22.006+0000 7fb610a00640 1 Processor -- start 2024-03-04T09:34:22.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:22.006+0000 7fb610a00640 1 -- start start 2024-03-04T09:34:22.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:22.007+0000 7fb610a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb60c145300 0x7fb60c0b49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:22.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:22.007+0000 7fb610a00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fb60c070f10 0x7fb60c0cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:22.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:22.008+0000 7fb610a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fb60c0cce90 0x7fb60c0cd930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:22.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:22.008+0000 7fb610a00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb60c0cd300 con 0x7fb60c0cce90 2024-03-04T09:34:22.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:22.008+0000 7fb610a00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb60c0714d0 con 0x7fb60c070f10 2024-03-04T09:34:22.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:22.008+0000 7fb610a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb60c0717b0 con 0x7fb60c145300 2024-03-04T09:34:22.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:22.008+0000 7fb60b200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fb60c0cce90 0x7fb60c0cd930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:22.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:22.008+0000 7fb60b200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fb60c0cce90 0x7fb60c0cd930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51612/0 (socket says 172.21.15.6:51612) 2024-03-04T09:34:22.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:22.008+0000 7fb60b200640 1 -- 172.21.15.6:0/819286244 learned_addr learned my addr 172.21.15.6:0/819286244 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:34:22.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:22.008+0000 7fb60a800640 1 --2- 172.21.15.6:0/819286244 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb60c145300 0x7fb60c0b49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:22.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:22.008+0000 7fb60b200640 1 -- 172.21.15.6:0/819286244 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb60c145300 msgr2=0x7fb60c0b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:34:22.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:22.008+0000 7fb603e00640 1 --2- 172.21.15.6:0/819286244 >> v2:172.21.15.146:3300/0 conn(0x7fb60c070f10 0x7fb60c0cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:22.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:22.008+0000 7fb60b200640 1 --2- 172.21.15.6:0/819286244 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb60c145300 0x7fb60c0b49a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:22.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:22.008+0000 7fb60b200640 1 -- 172.21.15.6:0/819286244 >> v2:172.21.15.146:3300/0 conn(0x7fb60c070f10 msgr2=0x7fb60c0cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:34:22.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:22.008+0000 7fb60b200640 1 --2- 172.21.15.6:0/819286244 >> v2:172.21.15.146:3300/0 conn(0x7fb60c070f10 0x7fb60c0cc950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:22.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:22.008+0000 7fb60b200640 1 -- 172.21.15.6:0/819286244 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb60c071a90 con 0x7fb60c0cce90 2024-03-04T09:34:22.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:22.010+0000 7fb60b200640 1 --2- 172.21.15.6:0/819286244 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb60c0cce90 0x7fb60c0cd930 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7fb60c0691a0 tx=0x7fb60402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a66a4e3980c4c103 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:34:22.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:22.010+0000 7fb609e00640 1 -- 172.21.15.6:0/819286244 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb60402ca80 con 0x7fb60c0cce90 2024-03-04T09:34:22.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:22.010+0000 7fb609e00640 1 -- 172.21.15.6:0/819286244 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb60402cc20 con 0x7fb60c0cce90 2024-03-04T09:34:22.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:22.010+0000 7fb609e00640 1 -- 172.21.15.6:0/819286244 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6040343f0 con 0x7fb60c0cce90 2024-03-04T09:34:22.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:22.011+0000 7fb610a00640 1 -- 172.21.15.6:0/819286244 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb60c0cce90 msgr2=0x7fb60c0cd930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:22.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:22.011+0000 7fb610a00640 1 --2- 172.21.15.6:0/819286244 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb60c0cce90 0x7fb60c0cd930 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7fb60c0691a0 tx=0x7fb60402bee0 comp rx=0 tx=0).stop 2024-03-04T09:34:22.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:22.011+0000 7fb610a00640 1 -- 172.21.15.6:0/819286244 shutdown_connections 2024-03-04T09:34:22.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:22.011+0000 7fb610a00640 1 --2- 172.21.15.6:0/819286244 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb60c0cce90 0x7fb60c0cd930 unknown :-1 s=CLOSED pgs=1623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:22.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:22.011+0000 7fb610a00640 1 --2- 172.21.15.6:0/819286244 >> v2:172.21.15.146:3300/0 conn(0x7fb60c070f10 0x7fb60c0cc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:22.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:22.011+0000 7fb610a00640 1 --2- 172.21.15.6:0/819286244 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb60c145300 0x7fb60c0b49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:22.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:22.011+0000 7fb610a00640 1 -- 172.21.15.6:0/819286244 >> 172.21.15.6:0/819286244 conn(0x7fb60c06b3d0 msgr2=0x7fb60c06b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:34:22.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:22.011+0000 7fb610a00640 1 -- 172.21.15.6:0/819286244 shutdown_connections 2024-03-04T09:34:22.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:22.011+0000 7fb610a00640 1 -- 172.21.15.6:0/819286244 wait complete. 2024-03-04T09:34:22.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:22.013+0000 7fb610a00640 1 Processor -- start 2024-03-04T09:34:22.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:22.013+0000 7fb610a00640 1 -- start start 2024-03-04T09:34:22.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:22.015+0000 7fb610a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb60c070f10 0x7fb60c1f4140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:22.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:22.015+0000 7fb610a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb60c0cce90 0x7fb60c0aeb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:22.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:22.015+0000 7fb610a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb60c145300 0x7fb60c13d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:22.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:22.015+0000 7fb610a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb60c1456e0 con 0x7fb60c070f10 2024-03-04T09:34:22.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:22.015+0000 7fb610a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb60c0b4ee0 con 0x7fb60c145300 2024-03-04T09:34:22.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:22.015+0000 7fb610a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb60c0cde70 con 0x7fb60c0cce90 2024-03-04T09:34:22.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:22.015+0000 7fb60a800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb60c070f10 0x7fb60c1f4140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:22.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:22.015+0000 7fb60a800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb60c070f10 0x7fb60c1f4140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51620/0 (socket says 172.21.15.6:51620) 2024-03-04T09:34:22.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:22.015+0000 7fb60a800640 1 -- 172.21.15.6:0/2862392044 learned_addr learned my addr 172.21.15.6:0/2862392044 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:34:22.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:22.016+0000 7fb60b200640 1 --2- 172.21.15.6:0/2862392044 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb60c145300 0x7fb60c13d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:22.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:22.016+0000 7fb603e00640 1 --2- 172.21.15.6:0/2862392044 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb60c0cce90 0x7fb60c0aeb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:22.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:22.016+0000 7fb60a800640 1 -- 172.21.15.6:0/2862392044 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb60c0cce90 msgr2=0x7fb60c0aeb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:22.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:22.016+0000 7fb60a800640 1 --2- 172.21.15.6:0/2862392044 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb60c0cce90 0x7fb60c0aeb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:22.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:22.016+0000 7fb60a800640 1 -- 172.21.15.6:0/2862392044 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb60c145300 msgr2=0x7fb60c13d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:22.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:22.016+0000 7fb60a800640 1 --2- 172.21.15.6:0/2862392044 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb60c145300 0x7fb60c13d7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:22.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:22.016+0000 7fb60a800640 1 -- 172.21.15.6:0/2862392044 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb60c1f4680 con 0x7fb60c070f10 2024-03-04T09:34:22.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:22.016+0000 7fb60b200640 1 --2- 172.21.15.6:0/2862392044 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb60c145300 0x7fb60c13d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:34:22.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:22.016+0000 7fb603e00640 1 --2- 172.21.15.6:0/2862392044 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb60c0cce90 0x7fb60c0aeb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:34:22.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:22.016+0000 7fb60a800640 1 --2- 172.21.15.6:0/2862392044 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb60c070f10 0x7fb60c1f4140 secure :-1 s=READY pgs=1624 cs=0 l=1 rev1=1 crypto rx=0x7fb5fc00c7f0 tx=0x7fb5fc00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:34:22.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:22.017+0000 7fb602a00640 1 -- 172.21.15.6:0/2862392044 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5fc015070 con 0x7fb60c070f10 2024-03-04T09:34:22.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:22.017+0000 7fb602a00640 1 -- 172.21.15.6:0/2862392044 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5fc004220 con 0x7fb60c070f10 2024-03-04T09:34:22.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:22.017+0000 7fb602a00640 1 -- 172.21.15.6:0/2862392044 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5fc007620 con 0x7fb60c070f10 2024-03-04T09:34:22.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:22.017+0000 7fb610a00640 1 -- 172.21.15.6:0/2862392044 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb60c0adad0 con 0x7fb60c070f10 2024-03-04T09:34:22.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:22.017+0000 7fb610a00640 1 -- 172.21.15.6:0/2862392044 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb60c0ae090 con 0x7fb60c070f10 2024-03-04T09:34:22.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:22.018+0000 7fb610a00640 1 -- 172.21.15.6:0/2862392044 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb60c04f330 con 0x7fb60c070f10 2024-03-04T09:34:22.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:22.019+0000 7fb602a00640 1 -- 172.21.15.6:0/2862392044 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb5fc0079d0 con 0x7fb60c070f10 2024-03-04T09:34:22.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:22.020+0000 7fb602a00640 1 --2- 172.21.15.6:0/2862392044 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb5e806b4d0 0x7fb5e806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:22.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:22.021+0000 7fb602a00640 1 -- 172.21.15.6:0/2862392044 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fb5fc08a4d0 con 0x7fb60c070f10 2024-03-04T09:34:22.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:22.021+0000 7fb603e00640 1 --2- 172.21.15.6:0/2862392044 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb5e806b4d0 0x7fb5e806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:22.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:22.022+0000 7fb603e00640 1 --2- 172.21.15.6:0/2862392044 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb5e806b4d0 0x7fb5e806d990 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7fb60c15f430 tx=0x7fb5f4008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:34:22.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:22.025+0000 7fb602a00640 1 -- 172.21.15.6:0/2862392044 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb60c1f4680 con 0x7fb60c070f10 2024-03-04T09:34:22.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:22.263+0000 7fb610a00640 1 -- 172.21.15.6:0/2862392044 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb60c0b2a50 con 0x7fb60c070f10 2024-03-04T09:34:22.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:22.264+0000 7fb602a00640 1 -- 172.21.15.6:0/2862392044 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3465 (secure 0 0 0) 0x7fb5fc059310 con 0x7fb60c070f10 2024-03-04T09:34:22.261 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:34:22.261 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:33:54.652601+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"laggy_since":"2024-03-04T09:33:54.652601+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":9}]} 2024-03-04T09:34:22.261 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 100 2024-03-04T09:34:22.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:22.266+0000 7fb610a00640 1 -- 172.21.15.6:0/2862392044 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb5e806b4d0 msgr2=0x7fb5e806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:22.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:22.266+0000 7fb610a00640 1 --2- 172.21.15.6:0/2862392044 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb5e806b4d0 0x7fb5e806d990 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7fb60c15f430 tx=0x7fb5f4008040 comp rx=0 tx=0).stop 2024-03-04T09:34:22.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:22.266+0000 7fb610a00640 1 -- 172.21.15.6:0/2862392044 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb60c070f10 msgr2=0x7fb60c1f4140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:22.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:22.266+0000 7fb610a00640 1 --2- 172.21.15.6:0/2862392044 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb60c070f10 0x7fb60c1f4140 secure :-1 s=READY pgs=1624 cs=0 l=1 rev1=1 crypto rx=0x7fb5fc00c7f0 tx=0x7fb5fc00ccb0 comp rx=0 tx=0).stop 2024-03-04T09:34:22.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:22.266+0000 7fb610a00640 1 -- 172.21.15.6:0/2862392044 shutdown_connections 2024-03-04T09:34:22.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:22.266+0000 7fb610a00640 1 --2- 172.21.15.6:0/2862392044 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb60c145300 0x7fb60c13d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:22.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:22.266+0000 7fb610a00640 1 --2- 172.21.15.6:0/2862392044 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb5e806b4d0 0x7fb5e806d990 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:22.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:22.266+0000 7fb610a00640 1 --2- 172.21.15.6:0/2862392044 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb60c0cce90 0x7fb60c0aeb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:22.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:22.266+0000 7fb610a00640 1 --2- 172.21.15.6:0/2862392044 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb60c070f10 0x7fb60c1f4140 unknown :-1 s=CLOSED pgs=1624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:22.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:22.266+0000 7fb610a00640 1 -- 172.21.15.6:0/2862392044 >> 172.21.15.6:0/2862392044 conn(0x7fb60c06b3d0 msgr2=0x7fb60c072620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:34:22.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:22.267+0000 7fb610a00640 1 -- 172.21.15.6:0/2862392044 shutdown_connections 2024-03-04T09:34:22.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:22.267+0000 7fb610a00640 1 -- 172.21.15.6:0/2862392044 wait complete. 2024-03-04T09:34:22.486 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:32:57.159681+0000', 'modified': '2024-03-04T09:33:54.652601+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8214, 'mds_1': 6796}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6796': {'gid': 6796, 'name': 'd', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.146:6837/2458070840', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 2458070840}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 2458070840}]}, 'join_fscid': -1, 'laggy_since': '2024-03-04T09:33:54.652601+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8214': {'gid': 8214, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2446627517', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2446627517}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2446627517}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8214, 'qdb_cluster': [8214]} 2024-03-04T09:34:22.487 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6796:up:stopping 2024-03-04T09:34:23.487 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:34:23.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:23.718+0000 7fa253800640 1 Processor -- start 2024-03-04T09:34:23.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:23.719+0000 7fa253800640 1 -- start start 2024-03-04T09:34:23.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:23.720+0000 7fa253800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa24c15e200 0x7fa24c0b3de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:23.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:23.720+0000 7fa253800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa24c0afdf0 0x7fa24c0af140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:23.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:23.720+0000 7fa253800640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa24c06bec0 con 0x7fa24c06afb0 2024-03-04T09:34:23.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:23.720+0000 7fa253800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa24c0b0670 con 0x7fa24c15e200 2024-03-04T09:34:23.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:23.720+0000 7fa253800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa24c0ae380 con 0x7fa24c0afdf0 2024-03-04T09:34:23.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:23.720+0000 7fa250e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa24c15e200 0x7fa24c0b3de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:23.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:23.720+0000 7fa250e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa24c15e200 0x7fa24c0b3de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51628/0 (socket says 172.21.15.6:51628) 2024-03-04T09:34:23.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:23.720+0000 7fa250e00640 1 -- 172.21.15.6:0/3036082033 learned_addr learned my addr 172.21.15.6:0/3036082033 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:34:23.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:23.721+0000 7fa250e00640 1 -- 172.21.15.6:0/3036082033 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa24c0afdf0 msgr2=0x7fa24c0af140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:34:23.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:23.721+0000 7fa24be00640 1 --2- 172.21.15.6:0/3036082033 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa24c0afdf0 0x7fa24c0af140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:23.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:23.721+0000 7fa250e00640 1 --2- 172.21.15.6:0/3036082033 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa24c0afdf0 0x7fa24c0af140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:23.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:23.721+0000 7fa250e00640 1 -- 172.21.15.6:0/3036082033 >> v1:172.21.15.146:6789/0 conn(0x7fa24c06afb0 legacy=0x7fa24c06dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:34:23.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:23.721+0000 7fa250e00640 1 -- 172.21.15.6:0/3036082033 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa24c0ae660 con 0x7fa24c15e200 2024-03-04T09:34:23.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:23.721+0000 7fa24be00640 1 --2- 172.21.15.6:0/3036082033 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa24c0afdf0 0x7fa24c0af140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:34:23.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:23.722+0000 7fa250e00640 1 --2- 172.21.15.6:0/3036082033 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa24c15e200 0x7fa24c0b3de0 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7fa24c071cc0 tx=0x7fa23c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27d0cb305a54f803 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:34:23.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:23.722+0000 7fa24b400640 1 -- 172.21.15.6:0/3036082033 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa23c035020 con 0x7fa24c15e200 2024-03-04T09:34:23.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:23.722+0000 7fa24b400640 1 -- 172.21.15.6:0/3036082033 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa23c031640 con 0x7fa24c15e200 2024-03-04T09:34:23.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:23.722+0000 7fa24b400640 1 -- 172.21.15.6:0/3036082033 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa23c02d040 con 0x7fa24c15e200 2024-03-04T09:34:23.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:23.723+0000 7fa253800640 1 -- 172.21.15.6:0/3036082033 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa24c15e200 msgr2=0x7fa24c0b3de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:23.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:23.723+0000 7fa253800640 1 --2- 172.21.15.6:0/3036082033 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa24c15e200 0x7fa24c0b3de0 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7fa24c071cc0 tx=0x7fa23c02b8f0 comp rx=0 tx=0).stop 2024-03-04T09:34:23.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:23.723+0000 7fa253800640 1 -- 172.21.15.6:0/3036082033 shutdown_connections 2024-03-04T09:34:23.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:23.723+0000 7fa253800640 1 --2- 172.21.15.6:0/3036082033 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa24c0afdf0 0x7fa24c0af140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:23.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:23.723+0000 7fa253800640 1 --2- 172.21.15.6:0/3036082033 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa24c15e200 0x7fa24c0b3de0 unknown :-1 s=CLOSED pgs=1625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:23.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:23.723+0000 7fa253800640 1 -- 172.21.15.6:0/3036082033 >> 172.21.15.6:0/3036082033 conn(0x7fa24c0b2ae0 msgr2=0x7fa24c0b2ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:34:23.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:23.723+0000 7fa253800640 1 -- 172.21.15.6:0/3036082033 shutdown_connections 2024-03-04T09:34:23.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:23.723+0000 7fa253800640 1 -- 172.21.15.6:0/3036082033 wait complete. 2024-03-04T09:34:23.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:23.725+0000 7fa253800640 1 Processor -- start 2024-03-04T09:34:23.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:23.725+0000 7fa253800640 1 -- start start 2024-03-04T09:34:23.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:23.727+0000 7fa253800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa24c06afb0 0x7fa24c14c630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:23.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:23.727+0000 7fa253800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa24c0afdf0 0x7fa24c15c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:23.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:23.727+0000 7fa253800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa24c15e200 0x7fa24c15c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:23.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:23.727+0000 7fa253800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa24c0af680 con 0x7fa24c15e200 2024-03-04T09:34:23.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:23.727+0000 7fa253800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa24c0aec20 con 0x7fa24c06afb0 2024-03-04T09:34:23.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:23.727+0000 7fa253800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa24c06e2c0 con 0x7fa24c0afdf0 2024-03-04T09:34:23.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:23.727+0000 7fa251800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa24c15e200 0x7fa24c15c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:23.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:23.727+0000 7fa251800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa24c15e200 0x7fa24c15c8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51634/0 (socket says 172.21.15.6:51634) 2024-03-04T09:34:23.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:23.727+0000 7fa251800640 1 -- 172.21.15.6:0/2527678445 learned_addr learned my addr 172.21.15.6:0/2527678445 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:34:23.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:23.728+0000 7fa24be00640 1 --2- 172.21.15.6:0/2527678445 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa24c0afdf0 0x7fa24c15c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:23.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:23.728+0000 7fa250e00640 1 --2- 172.21.15.6:0/2527678445 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa24c06afb0 0x7fa24c14c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:23.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:23.728+0000 7fa251800640 1 -- 172.21.15.6:0/2527678445 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa24c0afdf0 msgr2=0x7fa24c15c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:23.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:23.728+0000 7fa251800640 1 --2- 172.21.15.6:0/2527678445 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa24c0afdf0 0x7fa24c15c3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:23.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:23.728+0000 7fa251800640 1 -- 172.21.15.6:0/2527678445 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa24c06afb0 msgr2=0x7fa24c14c630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:23.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:23.728+0000 7fa251800640 1 --2- 172.21.15.6:0/2527678445 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa24c06afb0 0x7fa24c14c630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:23.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:23.728+0000 7fa251800640 1 -- 172.21.15.6:0/2527678445 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa24c1f5760 con 0x7fa24c15e200 2024-03-04T09:34:23.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:23.728+0000 7fa251800640 1 --2- 172.21.15.6:0/2527678445 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa24c15e200 0x7fa24c15c8e0 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x7fa24400c700 tx=0x7fa24400cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:34:23.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:23.728+0000 7fa248c00640 1 -- 172.21.15.6:0/2527678445 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa244016020 con 0x7fa24c15e200 2024-03-04T09:34:23.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:23.728+0000 7fa248c00640 1 -- 172.21.15.6:0/2527678445 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa24400aa50 con 0x7fa24c15e200 2024-03-04T09:34:23.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:23.728+0000 7fa248c00640 1 -- 172.21.15.6:0/2527678445 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa24400ae40 con 0x7fa24c15e200 2024-03-04T09:34:23.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:23.728+0000 7fa253800640 1 -- 172.21.15.6:0/2527678445 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa24c1f5a40 con 0x7fa24c15e200 2024-03-04T09:34:23.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:23.729+0000 7fa253800640 1 -- 172.21.15.6:0/2527678445 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa24c1f6000 con 0x7fa24c15e200 2024-03-04T09:34:23.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:23.730+0000 7fa253800640 1 -- 172.21.15.6:0/2527678445 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa24c04f3a0 con 0x7fa24c15e200 2024-03-04T09:34:23.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:23.731+0000 7fa248c00640 1 -- 172.21.15.6:0/2527678445 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa244019020 con 0x7fa24c15e200 2024-03-04T09:34:23.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:23.732+0000 7fa248c00640 1 --2- 172.21.15.6:0/2527678445 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa22c06b4d0 0x7fa22c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:23.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:23.732+0000 7fa248c00640 1 -- 172.21.15.6:0/2527678445 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fa24408a5e0 con 0x7fa24c15e200 2024-03-04T09:34:23.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:23.733+0000 7fa250e00640 1 --2- 172.21.15.6:0/2527678445 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa22c06b4d0 0x7fa22c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:23.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:23.734+0000 7fa250e00640 1 --2- 172.21.15.6:0/2527678445 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa22c06b4d0 0x7fa22c06d990 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fa23c036000 tx=0x7fa23c03c040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:34:23.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:23.737+0000 7fa248c00640 1 -- 172.21.15.6:0/2527678445 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa244020020 con 0x7fa24c15e200 2024-03-04T09:34:23.828 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-04T09:34:23.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:23.992+0000 7fa253800640 1 -- 172.21.15.6:0/2527678445 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa24c073050 con 0x7fa24c15e200 2024-03-04T09:34:23.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:23.993+0000 7fa248c00640 1 -- 172.21.15.6:0/2527678445 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3465 (secure 0 0 0) 0x7fa244059380 con 0x7fa24c15e200 2024-03-04T09:34:23.991 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:34:23.991 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:33:54.652601+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"laggy_since":"2024-03-04T09:33:54.652601+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":9}]} 2024-03-04T09:34:23.991 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 100 2024-03-04T09:34:23.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:23.996+0000 7fa253800640 1 -- 172.21.15.6:0/2527678445 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa22c06b4d0 msgr2=0x7fa22c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:23.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:23.996+0000 7fa253800640 1 --2- 172.21.15.6:0/2527678445 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa22c06b4d0 0x7fa22c06d990 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fa23c036000 tx=0x7fa23c03c040 comp rx=0 tx=0).stop 2024-03-04T09:34:23.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:23.996+0000 7fa253800640 1 -- 172.21.15.6:0/2527678445 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa24c15e200 msgr2=0x7fa24c15c8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:23.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:23.996+0000 7fa253800640 1 --2- 172.21.15.6:0/2527678445 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa24c15e200 0x7fa24c15c8e0 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x7fa24400c700 tx=0x7fa24400cbc0 comp rx=0 tx=0).stop 2024-03-04T09:34:23.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:23.996+0000 7fa253800640 1 -- 172.21.15.6:0/2527678445 shutdown_connections 2024-03-04T09:34:23.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:23.996+0000 7fa253800640 1 --2- 172.21.15.6:0/2527678445 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa24c15e200 0x7fa24c15c8e0 unknown :-1 s=CLOSED pgs=1626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:23.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:23.996+0000 7fa253800640 1 --2- 172.21.15.6:0/2527678445 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa22c06b4d0 0x7fa22c06d990 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:23.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:23.996+0000 7fa253800640 1 --2- 172.21.15.6:0/2527678445 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa24c0afdf0 0x7fa24c15c3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:23.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:23.996+0000 7fa253800640 1 --2- 172.21.15.6:0/2527678445 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa24c06afb0 0x7fa24c14c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:23.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:23.996+0000 7fa253800640 1 -- 172.21.15.6:0/2527678445 >> 172.21.15.6:0/2527678445 conn(0x7fa24c0b2ae0 msgr2=0x7fa24c1605f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:34:23.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:23.997+0000 7fa253800640 1 -- 172.21.15.6:0/2527678445 shutdown_connections 2024-03-04T09:34:23.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:23.997+0000 7fa253800640 1 -- 172.21.15.6:0/2527678445 wait complete. 2024-03-04T09:34:24.213 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:32:57.159681+0000', 'modified': '2024-03-04T09:33:54.652601+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8214, 'mds_1': 6796}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6796': {'gid': 6796, 'name': 'd', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.146:6837/2458070840', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 2458070840}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 2458070840}]}, 'join_fscid': -1, 'laggy_since': '2024-03-04T09:33:54.652601+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8214': {'gid': 8214, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2446627517', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2446627517}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2446627517}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8214, 'qdb_cluster': [8214]} 2024-03-04T09:34:24.213 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6796:up:stopping 2024-03-04T09:34:25.214 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:34:25.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:25.437+0000 7f446d400640 1 Processor -- start 2024-03-04T09:34:25.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:25.437+0000 7f446d400640 1 -- start start 2024-03-04T09:34:25.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:25.439+0000 7f446d400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f44680af1a0 0x7f446815c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:25.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:25.439+0000 7f446d400640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f446815cc40 con 0x7f44680af580 2024-03-04T09:34:25.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:25.439+0000 7f446d400640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f446815cf20 con 0x7f44680cdc20 2024-03-04T09:34:25.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:25.439+0000 7f446d400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f446815d200 con 0x7f44680af1a0 2024-03-04T09:34:25.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:25.439+0000 7f4465e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f44680af580 0x7f44680b4c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:36340/0 (socket says 172.21.15.6:36340) 2024-03-04T09:34:25.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:25.439+0000 7f4465e00640 1 -- 172.21.15.6:0/2470442949 learned_addr learned my addr 172.21.15.6:0/2470442949 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:34:25.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:25.440+0000 7f4465400640 1 -- 172.21.15.6:0/2470442949 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1048643915 0 0) 0x7f446815cc40 con 0x7f44680af580 2024-03-04T09:34:25.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:25.440+0000 7f4466800640 1 --2- 172.21.15.6:0/2470442949 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f44680af1a0 0x7f446815c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:25.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:25.440+0000 7f4465400640 1 -- 172.21.15.6:0/2470442949 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f444c003520 con 0x7f44680af580 2024-03-04T09:34:25.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:25.440+0000 7f4465400640 1 -- 172.21.15.6:0/2470442949 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4219867626 0 0) 0x7f444c003520 con 0x7f44680af580 2024-03-04T09:34:25.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:25.440+0000 7f4465400640 1 -- 172.21.15.6:0/2470442949 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f44680af1a0 msgr2=0x7f446815c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:34:25.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:25.440+0000 7f4465400640 1 --2- 172.21.15.6:0/2470442949 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f44680af1a0 0x7f446815c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:25.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:25.440+0000 7f4465400640 1 -- 172.21.15.6:0/2470442949 >> v1:172.21.15.146:6789/0 conn(0x7f44680cdc20 legacy=0x7f44680b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:25.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:25.440+0000 7f4465400640 1 -- 172.21.15.6:0/2470442949 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f446815d4e0 con 0x7f44680af580 2024-03-04T09:34:25.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:25.440+0000 7f4465400640 1 -- 172.21.15.6:0/2470442949 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7f4450001670 con 0x7f44680af580 2024-03-04T09:34:25.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:25.440+0000 7f4465400640 1 -- 172.21.15.6:0/2470442949 >> v1:172.21.15.6:6789/0 conn(0x7f44680af580 legacy=0x7f44680b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:25.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:25.440+0000 7f4465400640 1 --2- 172.21.15.6:0/2470442949 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f444c003a50 0x7f444c003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:25.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:25.440+0000 7f4465400640 1 --2- 172.21.15.6:0/2470442949 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f444c004800 0x7f444c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:25.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:25.440+0000 7f4465400640 1 --2- 172.21.15.6:0/2470442949 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f44680af1a0 0x7f444c005270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:25.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:25.440+0000 7f4465400640 1 -- 172.21.15.6:0/2470442949 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f446815d200 con 0x7f444c003a50 2024-03-04T09:34:25.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:25.440+0000 7f4465400640 1 -- 172.21.15.6:0/2470442949 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f444c00bb80 con 0x7f444c004800 2024-03-04T09:34:25.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:25.440+0000 7f4465400640 1 -- 172.21.15.6:0/2470442949 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f444c00bd80 con 0x7f44680af1a0 2024-03-04T09:34:25.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:25.440+0000 7f4466800640 1 --2- 172.21.15.6:0/2470442949 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f444c003a50 0x7f444c003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:25.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:25.441+0000 7f4465e00640 1 --2- 172.21.15.6:0/2470442949 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f444c004800 0x7f444c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:25.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:25.441+0000 7f4467200640 1 --2- 172.21.15.6:0/2470442949 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f44680af1a0 0x7f444c005270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:25.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:25.441+0000 7f4466800640 1 -- 172.21.15.6:0/2470442949 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f44680af1a0 msgr2=0x7f444c005270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:34:25.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:25.441+0000 7f4466800640 1 --2- 172.21.15.6:0/2470442949 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f44680af1a0 0x7f444c005270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:25.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:25.441+0000 7f4466800640 1 -- 172.21.15.6:0/2470442949 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f444c004800 msgr2=0x7f444c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:34:25.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:25.441+0000 7f4466800640 1 --2- 172.21.15.6:0/2470442949 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f444c004800 0x7f444c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:25.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:25.441+0000 7f4466800640 1 -- 172.21.15.6:0/2470442949 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f444c003520 con 0x7f444c003a50 2024-03-04T09:34:25.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:25.441+0000 7f4467200640 1 --2- 172.21.15.6:0/2470442949 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f44680af1a0 0x7f444c005270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:34:25.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:25.441+0000 7f4465e00640 1 --2- 172.21.15.6:0/2470442949 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f444c004800 0x7f444c004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:34:25.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:25.442+0000 7f4466800640 1 --2- 172.21.15.6:0/2470442949 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f444c003a50 0x7f444c003e70 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7f4458009f90 tx=0x7f444c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a091b05cf75f0221 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:34:25.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:25.442+0000 7f4465400640 1 -- 172.21.15.6:0/2470442949 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4458039070 con 0x7f444c003a50 2024-03-04T09:34:25.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:25.442+0000 7f4465400640 1 -- 172.21.15.6:0/2470442949 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f445802cbd0 con 0x7f444c003a50 2024-03-04T09:34:25.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:25.443+0000 7f4465400640 1 -- 172.21.15.6:0/2470442949 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4458009c10 con 0x7f444c003a50 2024-03-04T09:34:25.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:25.443+0000 7f446d400640 1 -- 172.21.15.6:0/2470442949 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f444c003a50 msgr2=0x7f444c003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:25.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:25.443+0000 7f446d400640 1 --2- 172.21.15.6:0/2470442949 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f444c003a50 0x7f444c003e70 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7f4458009f90 tx=0x7f444c005830 comp rx=0 tx=0).stop 2024-03-04T09:34:25.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:25.443+0000 7f4467200640 1 -- 172.21.15.6:0/2470442949 reap_dead start 2024-03-04T09:34:25.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:25.443+0000 7f446d400640 1 -- 172.21.15.6:0/2470442949 shutdown_connections 2024-03-04T09:34:25.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:25.443+0000 7f446d400640 1 -- 172.21.15.6:0/2470442949 >> 172.21.15.6:0/2470442949 conn(0x7f446806afb0 msgr2=0x7f446806b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:34:25.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:25.443+0000 7f446d400640 1 -- 172.21.15.6:0/2470442949 shutdown_connections 2024-03-04T09:34:25.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:25.443+0000 7f446d400640 1 -- 172.21.15.6:0/2470442949 wait complete. 2024-03-04T09:34:25.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:25.445+0000 7f446d400640 1 Processor -- start 2024-03-04T09:34:25.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:25.445+0000 7f446d400640 1 -- start start 2024-03-04T09:34:25.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:25.446+0000 7f446d400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f444c003a50 0x7f446815a080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:25.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:25.446+0000 7f446d400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f446806d560 0x7f446806de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:25.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:25.446+0000 7f446d400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f446815da70 0x7f44681491f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:25.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:25.446+0000 7f446d400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f446815c600 con 0x7f444c003a50 2024-03-04T09:34:25.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:25.446+0000 7f446d400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44680b36f0 con 0x7f446806d560 2024-03-04T09:34:25.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:25.446+0000 7f446d400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f446815c780 con 0x7f446815da70 2024-03-04T09:34:25.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:25.446+0000 7f4466800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f444c003a50 0x7f446815a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:25.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:25.446+0000 7f4466800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f444c003a50 0x7f446815a080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51644/0 (socket says 172.21.15.6:51644) 2024-03-04T09:34:25.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:25.446+0000 7f4466800640 1 -- 172.21.15.6:0/379625873 learned_addr learned my addr 172.21.15.6:0/379625873 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:34:25.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:25.447+0000 7f4465e00640 1 --2- 172.21.15.6:0/379625873 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f446806d560 0x7f446806de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:25.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:25.447+0000 7f4466800640 1 -- 172.21.15.6:0/379625873 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f446815da70 msgr2=0x7f44681491f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:25.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:25.447+0000 7f4466800640 1 --2- 172.21.15.6:0/379625873 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f446815da70 0x7f44681491f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:25.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:25.447+0000 7f4466800640 1 -- 172.21.15.6:0/379625873 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f446806d560 msgr2=0x7f446806de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:25.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:25.447+0000 7f4466800640 1 --2- 172.21.15.6:0/379625873 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f446806d560 0x7f446806de80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:25.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:25.447+0000 7f4466800640 1 -- 172.21.15.6:0/379625873 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f446806e3c0 con 0x7f444c003a50 2024-03-04T09:34:25.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:25.447+0000 7f4465e00640 1 --2- 172.21.15.6:0/379625873 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f446806d560 0x7f446806de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:34:25.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:25.447+0000 7f4466800640 1 --2- 172.21.15.6:0/379625873 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f444c003a50 0x7f446815a080 secure :-1 s=READY pgs=1629 cs=0 l=1 rev1=1 crypto rx=0x7f4458037040 tx=0x7f4458009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:34:25.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:25.448+0000 7f445ea00640 1 -- 172.21.15.6:0/379625873 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4458039050 con 0x7f444c003a50 2024-03-04T09:34:25.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:25.448+0000 7f445ea00640 1 -- 172.21.15.6:0/379625873 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4458045070 con 0x7f444c003a50 2024-03-04T09:34:25.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:25.448+0000 7f445ea00640 1 -- 172.21.15.6:0/379625873 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4458009c10 con 0x7f444c003a50 2024-03-04T09:34:25.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:25.448+0000 7f446d400640 1 -- 172.21.15.6:0/379625873 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f446806e690 con 0x7f444c003a50 2024-03-04T09:34:25.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:25.448+0000 7f446d400640 1 -- 172.21.15.6:0/379625873 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f446806cd60 con 0x7f444c003a50 2024-03-04T09:34:25.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:25.449+0000 7f446d400640 1 -- 172.21.15.6:0/379625873 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4468069510 con 0x7f444c003a50 2024-03-04T09:34:25.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:25.450+0000 7f445ea00640 1 -- 172.21.15.6:0/379625873 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4458005d10 con 0x7f444c003a50 2024-03-04T09:34:25.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:25.451+0000 7f445ea00640 1 --2- 172.21.15.6:0/379625873 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f444406b500 0x7f444406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:25.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:25.451+0000 7f445ea00640 1 -- 172.21.15.6:0/379625873 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f44580ac890 con 0x7f444c003a50 2024-03-04T09:34:25.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:25.452+0000 7f4465e00640 1 --2- 172.21.15.6:0/379625873 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f444406b500 0x7f444406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:25.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:25.453+0000 7f4465e00640 1 --2- 172.21.15.6:0/379625873 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f444406b500 0x7f444406d9c0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f44500017e0 tx=0x7f4450001c50 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:34:25.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:25.456+0000 7f445ea00640 1 -- 172.21.15.6:0/379625873 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f445807b760 con 0x7f444c003a50 2024-03-04T09:34:25.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:25.697+0000 7f446d400640 1 -- 172.21.15.6:0/379625873 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f44680aea40 con 0x7f444c003a50 2024-03-04T09:34:25.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:25.698+0000 7f445ea00640 1 -- 172.21.15.6:0/379625873 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3465 (secure 0 0 0) 0x7f445807b060 con 0x7f444c003a50 2024-03-04T09:34:25.695 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:34:25.695 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:33:54.652601+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"laggy_since":"2024-03-04T09:33:54.652601+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":9}]} 2024-03-04T09:34:25.695 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 100 2024-03-04T09:34:25.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:25.700+0000 7f446d400640 1 -- 172.21.15.6:0/379625873 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f444406b500 msgr2=0x7f444406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:25.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:25.700+0000 7f446d400640 1 --2- 172.21.15.6:0/379625873 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f444406b500 0x7f444406d9c0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f44500017e0 tx=0x7f4450001c50 comp rx=0 tx=0).stop 2024-03-04T09:34:25.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:25.701+0000 7f446d400640 1 -- 172.21.15.6:0/379625873 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f444c003a50 msgr2=0x7f446815a080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:25.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:25.701+0000 7f446d400640 1 --2- 172.21.15.6:0/379625873 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f444c003a50 0x7f446815a080 secure :-1 s=READY pgs=1629 cs=0 l=1 rev1=1 crypto rx=0x7f4458037040 tx=0x7f4458009750 comp rx=0 tx=0).stop 2024-03-04T09:34:25.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:25.701+0000 7f446d400640 1 -- 172.21.15.6:0/379625873 shutdown_connections 2024-03-04T09:34:25.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:25.701+0000 7f446d400640 1 --2- 172.21.15.6:0/379625873 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f444406b500 0x7f444406d9c0 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:25.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:25.701+0000 7f446d400640 1 --2- 172.21.15.6:0/379625873 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f446815da70 0x7f44681491f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:25.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:25.701+0000 7f446d400640 1 --2- 172.21.15.6:0/379625873 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f446806d560 0x7f446806de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:25.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:25.701+0000 7f446d400640 1 --2- 172.21.15.6:0/379625873 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f444c003a50 0x7f446815a080 unknown :-1 s=CLOSED pgs=1629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:25.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:25.701+0000 7f446d400640 1 -- 172.21.15.6:0/379625873 >> 172.21.15.6:0/379625873 conn(0x7f446806afb0 msgr2=0x7f446806b630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:34:25.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:25.701+0000 7f446d400640 1 -- 172.21.15.6:0/379625873 shutdown_connections 2024-03-04T09:34:25.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:25.701+0000 7f446d400640 1 -- 172.21.15.6:0/379625873 wait complete. 2024-03-04T09:34:25.830 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-04T09:34:25.922 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:32:57.159681+0000', 'modified': '2024-03-04T09:33:54.652601+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8214, 'mds_1': 6796}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6796': {'gid': 6796, 'name': 'd', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.146:6837/2458070840', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 2458070840}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 2458070840}]}, 'join_fscid': -1, 'laggy_since': '2024-03-04T09:33:54.652601+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8214': {'gid': 8214, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2446627517', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2446627517}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2446627517}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8214, 'qdb_cluster': [8214]} 2024-03-04T09:34:25.922 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6796:up:stopping 2024-03-04T09:34:26.923 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:34:27.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:27.154+0000 7f7c3f200640 1 Processor -- start 2024-03-04T09:34:27.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:27.154+0000 7f7c3f200640 1 -- start start 2024-03-04T09:34:27.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:27.156+0000 7f7c3f200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f7c400d4a10 0x7f7c400d0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:27.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:27.156+0000 7f7c3f200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c400b3710 con 0x7f7c400d0c80 2024-03-04T09:34:27.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:27.156+0000 7f7c3f200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c400b39f0 con 0x7f7c4015e0b0 2024-03-04T09:34:27.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:27.156+0000 7f7c3f200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c400d10f0 con 0x7f7c400d4a10 2024-03-04T09:34:27.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:27.156+0000 7f7c3e800640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f7c400d0c80 0x7f7c400b44d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:36344/0 (socket says 172.21.15.6:36344) 2024-03-04T09:34:27.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:27.156+0000 7f7c3e800640 1 -- 172.21.15.6:0/2665425184 learned_addr learned my addr 172.21.15.6:0/2665425184 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:34:27.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:27.157+0000 7f7c35400640 1 --2- 172.21.15.6:0/2665425184 >> v2:172.21.15.146:3300/0 conn(0x7f7c400d4a10 0x7f7c400d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:27.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:27.157+0000 7f7c3d400640 1 -- 172.21.15.6:0/2665425184 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3841834657 0 0) 0x7f7c400b3710 con 0x7f7c400d0c80 2024-03-04T09:34:27.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:27.157+0000 7f7c3d400640 1 -- 172.21.15.6:0/2665425184 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c24003520 con 0x7f7c400d0c80 2024-03-04T09:34:27.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:27.157+0000 7f7c3d400640 1 -- 172.21.15.6:0/2665425184 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 804850345 0 0) 0x7f7c24003520 con 0x7f7c400d0c80 2024-03-04T09:34:27.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:27.157+0000 7f7c3d400640 1 -- 172.21.15.6:0/2665425184 >> v2:172.21.15.146:3300/0 conn(0x7f7c400d4a10 msgr2=0x7f7c400d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:34:27.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:27.157+0000 7f7c3d400640 1 --2- 172.21.15.6:0/2665425184 >> v2:172.21.15.146:3300/0 conn(0x7f7c400d4a10 0x7f7c400d0740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:27.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:27.157+0000 7f7c3d400640 1 -- 172.21.15.6:0/2665425184 >> v1:172.21.15.146:6789/0 conn(0x7f7c4015e0b0 legacy=0x7f7c400cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:27.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:27.157+0000 7f7c3d400640 1 -- 172.21.15.6:0/2665425184 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c400b3cd0 con 0x7f7c400d0c80 2024-03-04T09:34:27.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:27.157+0000 7f7c3d400640 1 -- 172.21.15.6:0/2665425184 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7f7c38001690 con 0x7f7c400d0c80 2024-03-04T09:34:27.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:27.157+0000 7f7c3d400640 1 -- 172.21.15.6:0/2665425184 >> v1:172.21.15.6:6789/0 conn(0x7f7c400d0c80 legacy=0x7f7c400b44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:27.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:27.157+0000 7f7c3d400640 1 --2- 172.21.15.6:0/2665425184 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7c24003a00 0x7f7c24003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:27.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:27.157+0000 7f7c3d400640 1 --2- 172.21.15.6:0/2665425184 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7c24004800 0x7f7c24004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:27.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:27.157+0000 7f7c3d400640 1 --2- 172.21.15.6:0/2665425184 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7c240051c0 0x7f7c2400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:27.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:27.158+0000 7f7c3d400640 1 -- 172.21.15.6:0/2665425184 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c400d10f0 con 0x7f7c24004800 2024-03-04T09:34:27.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:27.158+0000 7f7c3d400640 1 -- 172.21.15.6:0/2665425184 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c2400c0c0 con 0x7f7c24003a00 2024-03-04T09:34:27.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:27.158+0000 7f7c3d400640 1 -- 172.21.15.6:0/2665425184 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c2400c2c0 con 0x7f7c240051c0 2024-03-04T09:34:27.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:27.158+0000 7f7c35400640 1 --2- 172.21.15.6:0/2665425184 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7c24004800 0x7f7c24004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:27.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:27.158+0000 7f7c3de00640 1 --2- 172.21.15.6:0/2665425184 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7c24003a00 0x7f7c24003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:27.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:27.158+0000 7f7c3e800640 1 --2- 172.21.15.6:0/2665425184 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7c240051c0 0x7f7c2400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:27.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:27.158+0000 7f7c35400640 1 -- 172.21.15.6:0/2665425184 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7c240051c0 msgr2=0x7f7c2400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:34:27.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:27.158+0000 7f7c35400640 1 --2- 172.21.15.6:0/2665425184 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7c240051c0 0x7f7c2400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:27.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:27.158+0000 7f7c35400640 1 -- 172.21.15.6:0/2665425184 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7c24003a00 msgr2=0x7f7c24003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:34:27.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:27.158+0000 7f7c35400640 1 --2- 172.21.15.6:0/2665425184 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7c24003a00 0x7f7c24003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:27.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:27.158+0000 7f7c35400640 1 -- 172.21.15.6:0/2665425184 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c24003520 con 0x7f7c24004800 2024-03-04T09:34:27.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:27.158+0000 7f7c3e800640 1 --2- 172.21.15.6:0/2665425184 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7c240051c0 0x7f7c2400b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:34:27.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:27.158+0000 7f7c3de00640 1 --2- 172.21.15.6:0/2665425184 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7c24003a00 0x7f7c24003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T09:34:27.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:27.158+0000 7f7c3e800640 1 -- 172.21.15.6:0/2665425184 reap_dead start 2024-03-04T09:34:27.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:27.159+0000 7f7c35400640 1 --2- 172.21.15.6:0/2665425184 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7c24004800 0x7f7c24004c80 secure :-1 s=READY pgs=1631 cs=0 l=1 rev1=1 crypto rx=0x7f7c28009a90 tx=0x7f7c24005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c142f9eba343174d server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:34:27.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:27.160+0000 7f7c3d400640 1 -- 172.21.15.6:0/2665425184 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c28039070 con 0x7f7c24004800 2024-03-04T09:34:27.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:27.160+0000 7f7c3d400640 1 -- 172.21.15.6:0/2665425184 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c2802c680 con 0x7f7c24004800 2024-03-04T09:34:27.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:27.160+0000 7f7c3d400640 1 -- 172.21.15.6:0/2665425184 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c28009690 con 0x7f7c24004800 2024-03-04T09:34:27.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:27.160+0000 7f7c3f200640 1 -- 172.21.15.6:0/2665425184 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7c24004800 msgr2=0x7f7c24004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:27.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:27.160+0000 7f7c3f200640 1 --2- 172.21.15.6:0/2665425184 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7c24004800 0x7f7c24004c80 secure :-1 s=READY pgs=1631 cs=0 l=1 rev1=1 crypto rx=0x7f7c28009a90 tx=0x7f7c24005830 comp rx=0 tx=0).stop 2024-03-04T09:34:27.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:27.160+0000 7f7c3f200640 1 -- 172.21.15.6:0/2665425184 shutdown_connections 2024-03-04T09:34:27.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:27.160+0000 7f7c3f200640 1 --2- 172.21.15.6:0/2665425184 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7c24004800 0x7f7c24004c80 unknown :-1 s=CLOSED pgs=1631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:27.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:27.160+0000 7f7c3f200640 1 -- 172.21.15.6:0/2665425184 >> 172.21.15.6:0/2665425184 conn(0x7f7c400d3d30 msgr2=0x7f7c400d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:34:27.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:27.161+0000 7f7c3f200640 1 -- 172.21.15.6:0/2665425184 shutdown_connections 2024-03-04T09:34:27.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:27.161+0000 7f7c3f200640 1 -- 172.21.15.6:0/2665425184 wait complete. 2024-03-04T09:34:27.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:27.162+0000 7f7c3f200640 1 Processor -- start 2024-03-04T09:34:27.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:27.163+0000 7f7c3f200640 1 -- start start 2024-03-04T09:34:27.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:27.163+0000 7f7c3f200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7c24004800 0x7f7c400cae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:27.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:27.163+0000 7f7c3f200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7c40153220 0x7f7c40160e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:27.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:27.163+0000 7f7c3f200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7c4014fd80 0x7f7c40150200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:27.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:27.163+0000 7f7c3f200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c4015e490 con 0x7f7c4014fd80 2024-03-04T09:34:27.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:27.163+0000 7f7c3f200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c400cfc60 con 0x7f7c40153220 2024-03-04T09:34:27.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:27.164+0000 7f7c3f200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c400d0740 con 0x7f7c24004800 2024-03-04T09:34:27.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:27.164+0000 7f7c3e800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7c4014fd80 0x7f7c40150200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:27.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:27.164+0000 7f7c3e800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7c4014fd80 0x7f7c40150200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51664/0 (socket says 172.21.15.6:51664) 2024-03-04T09:34:27.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:27.164+0000 7f7c3e800640 1 -- 172.21.15.6:0/3503441721 learned_addr learned my addr 172.21.15.6:0/3503441721 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:34:27.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:27.164+0000 7f7c3de00640 1 --2- 172.21.15.6:0/3503441721 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7c24004800 0x7f7c400cae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:27.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:27.164+0000 7f7c35400640 1 --2- 172.21.15.6:0/3503441721 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7c40153220 0x7f7c40160e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:27.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:27.164+0000 7f7c3e800640 1 -- 172.21.15.6:0/3503441721 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7c24004800 msgr2=0x7f7c400cae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:27.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:27.164+0000 7f7c3e800640 1 --2- 172.21.15.6:0/3503441721 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7c24004800 0x7f7c400cae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:27.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:27.164+0000 7f7c3e800640 1 -- 172.21.15.6:0/3503441721 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7c40153220 msgr2=0x7f7c40160e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:27.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:27.164+0000 7f7c3e800640 1 --2- 172.21.15.6:0/3503441721 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7c40153220 0x7f7c40160e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:27.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:27.164+0000 7f7c3e800640 1 -- 172.21.15.6:0/3503441721 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c400cb6e0 con 0x7f7c4014fd80 2024-03-04T09:34:27.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:27.165+0000 7f7c3e800640 1 --2- 172.21.15.6:0/3503441721 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7c4014fd80 0x7f7c40150200 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 crypto rx=0x7f7c400d3c80 tx=0x7f7c3800d9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:34:27.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:27.165+0000 7f7c36a00640 1 -- 172.21.15.6:0/3503441721 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c38019070 con 0x7f7c4014fd80 2024-03-04T09:34:27.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:27.165+0000 7f7c36a00640 1 -- 172.21.15.6:0/3503441721 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c38007460 con 0x7f7c4014fd80 2024-03-04T09:34:27.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:27.165+0000 7f7c36a00640 1 -- 172.21.15.6:0/3503441721 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c380047e0 con 0x7f7c4014fd80 2024-03-04T09:34:27.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:27.165+0000 7f7c3f200640 1 -- 172.21.15.6:0/3503441721 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c400cb9c0 con 0x7f7c4014fd80 2024-03-04T09:34:27.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:27.165+0000 7f7c3f200640 1 -- 172.21.15.6:0/3503441721 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c400cbf80 con 0x7f7c4014fd80 2024-03-04T09:34:27.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:27.166+0000 7f7c3f200640 1 -- 172.21.15.6:0/3503441721 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c4004f330 con 0x7f7c4014fd80 2024-03-04T09:34:27.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:27.169+0000 7f7c36a00640 1 -- 172.21.15.6:0/3503441721 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7c380123d0 con 0x7f7c4014fd80 2024-03-04T09:34:27.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:27.169+0000 7f7c36a00640 1 --2- 172.21.15.6:0/3503441721 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f7c1c06b500 0x7f7c1c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:27.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:27.170+0000 7f7c36a00640 1 -- 172.21.15.6:0/3503441721 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f7c380895d0 con 0x7f7c4014fd80 2024-03-04T09:34:27.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:27.170+0000 7f7c3de00640 1 --2- 172.21.15.6:0/3503441721 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f7c1c06b500 0x7f7c1c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:27.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:27.171+0000 7f7c3de00640 1 --2- 172.21.15.6:0/3503441721 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f7c1c06b500 0x7f7c1c06d9c0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f7c30007fd0 tx=0x7f7c30009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:34:27.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:27.174+0000 7f7c36a00640 1 -- 172.21.15.6:0/3503441721 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7c38058370 con 0x7f7c4014fd80 2024-03-04T09:34:27.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:27.413+0000 7f7c3f200640 1 -- 172.21.15.6:0/3503441721 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7c400b3180 con 0x7f7c4014fd80 2024-03-04T09:34:27.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:27.414+0000 7f7c36a00640 1 -- 172.21.15.6:0/3503441721 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3465 (secure 0 0 0) 0x7f7c38057c70 con 0x7f7c4014fd80 2024-03-04T09:34:27.412 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:34:27.412 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:33:54.652601+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"laggy_since":"2024-03-04T09:33:54.652601+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":9}]} 2024-03-04T09:34:27.412 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 100 2024-03-04T09:34:27.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:27.416+0000 7f7c3f200640 1 -- 172.21.15.6:0/3503441721 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f7c1c06b500 msgr2=0x7f7c1c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:27.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:27.416+0000 7f7c3f200640 1 --2- 172.21.15.6:0/3503441721 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f7c1c06b500 0x7f7c1c06d9c0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f7c30007fd0 tx=0x7f7c30009040 comp rx=0 tx=0).stop 2024-03-04T09:34:27.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:27.416+0000 7f7c3f200640 1 -- 172.21.15.6:0/3503441721 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7c4014fd80 msgr2=0x7f7c40150200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:27.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:27.416+0000 7f7c3f200640 1 --2- 172.21.15.6:0/3503441721 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7c4014fd80 0x7f7c40150200 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 crypto rx=0x7f7c400d3c80 tx=0x7f7c3800d9d0 comp rx=0 tx=0).stop 2024-03-04T09:34:27.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:27.417+0000 7f7c3f200640 1 -- 172.21.15.6:0/3503441721 shutdown_connections 2024-03-04T09:34:27.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:27.417+0000 7f7c3f200640 1 --2- 172.21.15.6:0/3503441721 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7c4014fd80 0x7f7c40150200 unknown :-1 s=CLOSED pgs=1632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:27.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:27.417+0000 7f7c3f200640 1 --2- 172.21.15.6:0/3503441721 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7c40153220 0x7f7c40160e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:27.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:27.417+0000 7f7c3f200640 1 --2- 172.21.15.6:0/3503441721 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f7c1c06b500 0x7f7c1c06d9c0 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:27.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:27.417+0000 7f7c3f200640 1 --2- 172.21.15.6:0/3503441721 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7c24004800 0x7f7c400cae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:27.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:27.417+0000 7f7c3f200640 1 -- 172.21.15.6:0/3503441721 >> 172.21.15.6:0/3503441721 conn(0x7f7c400d3d30 msgr2=0x7f7c400cfde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:34:27.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:27.417+0000 7f7c3f200640 1 -- 172.21.15.6:0/3503441721 shutdown_connections 2024-03-04T09:34:27.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:27.417+0000 7f7c3f200640 1 -- 172.21.15.6:0/3503441721 wait complete. 2024-03-04T09:34:27.634 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:32:57.159681+0000', 'modified': '2024-03-04T09:33:54.652601+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8214, 'mds_1': 6796}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6796': {'gid': 6796, 'name': 'd', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.146:6837/2458070840', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 2458070840}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 2458070840}]}, 'join_fscid': -1, 'laggy_since': '2024-03-04T09:33:54.652601+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8214': {'gid': 8214, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2446627517', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2446627517}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2446627517}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8214, 'qdb_cluster': [8214]} 2024-03-04T09:34:27.634 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6796:up:stopping 2024-03-04T09:34:27.832 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-04T09:34:28.636 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:34:28.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:28.862+0000 7f3b9a800640 1 Processor -- start 2024-03-04T09:34:28.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:28.863+0000 7f3b9a800640 1 -- start start 2024-03-04T09:34:28.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:28.864+0000 7f3b9a800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f3b940d0e50 0x7f3b94168c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:28.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:28.864+0000 7f3b9a800640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b940b3710 con 0x7f3b940d4a10 2024-03-04T09:34:28.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:28.864+0000 7f3b9a800640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b940b39f0 con 0x7f3b9415e0b0 2024-03-04T09:34:28.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:28.864+0000 7f3b9a800640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b940b09e0 con 0x7f3b940d0e50 2024-03-04T09:34:28.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:28.864+0000 7f3b98a00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f3b940d4a10 0x7f3b940d0740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:56346/0 (socket says 172.21.15.6:56346) 2024-03-04T09:34:28.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:28.864+0000 7f3b98a00640 1 -- 172.21.15.6:0/1112428974 learned_addr learned my addr 172.21.15.6:0/1112428974 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:34:28.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:28.865+0000 7f3b99e00640 1 --2- 172.21.15.6:0/1112428974 >> v2:172.21.15.146:3300/0 conn(0x7f3b940d0e50 0x7f3b94168c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:28.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:28.865+0000 7f3b93e00640 1 -- 172.21.15.6:0/1112428974 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2331155882 0 0) 0x7f3b940b3710 con 0x7f3b940d4a10 2024-03-04T09:34:28.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:28.865+0000 7f3b93e00640 1 -- 172.21.15.6:0/1112428974 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b7c003520 con 0x7f3b940d4a10 2024-03-04T09:34:28.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:28.865+0000 7f3b93e00640 1 -- 172.21.15.6:0/1112428974 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1354573817 0 0) 0x7f3b7c003520 con 0x7f3b940d4a10 2024-03-04T09:34:28.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:28.865+0000 7f3b93e00640 1 -- 172.21.15.6:0/1112428974 >> v2:172.21.15.146:3300/0 conn(0x7f3b940d0e50 msgr2=0x7f3b94168c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:34:28.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:28.865+0000 7f3b93e00640 1 --2- 172.21.15.6:0/1112428974 >> v2:172.21.15.146:3300/0 conn(0x7f3b940d0e50 0x7f3b94168c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:28.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:28.865+0000 7f3b93e00640 1 -- 172.21.15.6:0/1112428974 >> v1:172.21.15.146:6789/0 conn(0x7f3b9415e0b0 legacy=0x7f3b940cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:28.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:28.865+0000 7f3b93e00640 1 -- 172.21.15.6:0/1112428974 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b940b3cd0 con 0x7f3b940d4a10 2024-03-04T09:34:28.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:28.865+0000 7f3b93e00640 1 -- 172.21.15.6:0/1112428974 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7f3b80001670 con 0x7f3b940d4a10 2024-03-04T09:34:28.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:28.866+0000 7f3b93e00640 1 -- 172.21.15.6:0/1112428974 >> v1:172.21.15.6:6789/0 conn(0x7f3b940d4a10 legacy=0x7f3b940d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:28.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:28.866+0000 7f3b93e00640 1 --2- 172.21.15.6:0/1112428974 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3b7c003a00 0x7f3b7c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:28.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:28.866+0000 7f3b93e00640 1 --2- 172.21.15.6:0/1112428974 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3b7c004800 0x7f3b7c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:28.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:28.866+0000 7f3b93e00640 1 --2- 172.21.15.6:0/1112428974 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3b7c0051c0 0x7f3b7c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:28.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:28.866+0000 7f3b93e00640 1 -- 172.21.15.6:0/1112428974 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b7c00c090 con 0x7f3b7c0051c0 2024-03-04T09:34:28.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:28.866+0000 7f3b93e00640 1 -- 172.21.15.6:0/1112428974 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b7c00c240 con 0x7f3b7c004800 2024-03-04T09:34:28.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:28.866+0000 7f3b93e00640 1 -- 172.21.15.6:0/1112428974 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b7c00c440 con 0x7f3b7c003a00 2024-03-04T09:34:28.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:28.866+0000 7f3b99e00640 1 --2- 172.21.15.6:0/1112428974 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3b7c0051c0 0x7f3b7c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:28.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:28.866+0000 7f3b99400640 1 --2- 172.21.15.6:0/1112428974 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3b7c003a00 0x7f3b7c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:28.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:28.866+0000 7f3b98a00640 1 --2- 172.21.15.6:0/1112428974 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3b7c004800 0x7f3b7c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:28.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:28.866+0000 7f3b99e00640 1 -- 172.21.15.6:0/1112428974 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3b7c003a00 msgr2=0x7f3b7c003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:34:28.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:28.866+0000 7f3b99e00640 1 --2- 172.21.15.6:0/1112428974 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3b7c003a00 0x7f3b7c003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:28.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:28.866+0000 7f3b99e00640 1 -- 172.21.15.6:0/1112428974 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3b7c004800 msgr2=0x7f3b7c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:34:28.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:28.866+0000 7f3b99e00640 1 --2- 172.21.15.6:0/1112428974 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3b7c004800 0x7f3b7c004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:28.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:28.866+0000 7f3b99e00640 1 -- 172.21.15.6:0/1112428974 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b7c003520 con 0x7f3b7c0051c0 2024-03-04T09:34:28.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:28.866+0000 7f3b99400640 1 --2- 172.21.15.6:0/1112428974 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3b7c003a00 0x7f3b7c003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:34:28.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:28.867+0000 7f3b99e00640 1 -- 172.21.15.6:0/1112428974 reap_dead start 2024-03-04T09:34:28.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:28.868+0000 7f3b99e00640 1 --2- 172.21.15.6:0/1112428974 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3b7c0051c0 0x7f3b7c00b9c0 secure :-1 s=READY pgs=1634 cs=0 l=1 rev1=1 crypto rx=0x7f3b8c009f90 tx=0x7f3b7c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bec627d08baa464b server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:34:28.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:28.868+0000 7f3b93e00640 1 -- 172.21.15.6:0/1112428974 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b8c009c10 con 0x7f3b7c0051c0 2024-03-04T09:34:28.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:28.868+0000 7f3b93e00640 1 -- 172.21.15.6:0/1112428974 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b8c02cc20 con 0x7f3b7c0051c0 2024-03-04T09:34:28.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:28.868+0000 7f3b93e00640 1 -- 172.21.15.6:0/1112428974 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b8c003990 con 0x7f3b7c0051c0 2024-03-04T09:34:28.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:28.868+0000 7f3b9a800640 1 -- 172.21.15.6:0/1112428974 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3b7c0051c0 msgr2=0x7f3b7c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:28.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:28.868+0000 7f3b9a800640 1 --2- 172.21.15.6:0/1112428974 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3b7c0051c0 0x7f3b7c00b9c0 secure :-1 s=READY pgs=1634 cs=0 l=1 rev1=1 crypto rx=0x7f3b8c009f90 tx=0x7f3b7c005830 comp rx=0 tx=0).stop 2024-03-04T09:34:28.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:28.868+0000 7f3b9a800640 1 -- 172.21.15.6:0/1112428974 shutdown_connections 2024-03-04T09:34:28.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:28.868+0000 7f3b9a800640 1 --2- 172.21.15.6:0/1112428974 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3b7c0051c0 0x7f3b7c00b9c0 unknown :-1 s=CLOSED pgs=1634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:28.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:28.868+0000 7f3b9a800640 1 -- 172.21.15.6:0/1112428974 >> 172.21.15.6:0/1112428974 conn(0x7f3b940d3d30 msgr2=0x7f3b940d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:34:28.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:28.869+0000 7f3b9a800640 1 -- 172.21.15.6:0/1112428974 shutdown_connections 2024-03-04T09:34:28.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:28.869+0000 7f3b9a800640 1 -- 172.21.15.6:0/1112428974 wait complete. 2024-03-04T09:34:28.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:28.870+0000 7f3b9a800640 1 Processor -- start 2024-03-04T09:34:28.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:28.871+0000 7f3b9a800640 1 -- start start 2024-03-04T09:34:28.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:28.871+0000 7f3b9a800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3b7c0051c0 0x7f3b940cbe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:28.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:28.871+0000 7f3b9a800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3b940cc650 0x7f3b941f3c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:28.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:28.872+0000 7f3b9a800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3b941f41c0 0x7f3b941f4650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:28.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:28.872+0000 7f3b9a800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b940cfc60 con 0x7f3b940cc650 2024-03-04T09:34:28.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:28.872+0000 7f3b9a800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b9415e490 con 0x7f3b941f41c0 2024-03-04T09:34:28.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:28.872+0000 7f3b9a800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b940d0740 con 0x7f3b7c0051c0 2024-03-04T09:34:28.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:28.872+0000 7f3b98a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3b940cc650 0x7f3b941f3c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:28.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:28.872+0000 7f3b98a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3b940cc650 0x7f3b941f3c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51886/0 (socket says 172.21.15.6:51886) 2024-03-04T09:34:28.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:28.872+0000 7f3b98a00640 1 -- 172.21.15.6:0/3832674867 learned_addr learned my addr 172.21.15.6:0/3832674867 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:34:28.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:28.872+0000 7f3b99e00640 1 --2- 172.21.15.6:0/3832674867 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3b941f41c0 0x7f3b941f4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:28.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:28.872+0000 7f3b99400640 1 --2- 172.21.15.6:0/3832674867 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3b7c0051c0 0x7f3b940cbe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:28.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:28.872+0000 7f3b98a00640 1 -- 172.21.15.6:0/3832674867 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3b7c0051c0 msgr2=0x7f3b940cbe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:28.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:28.872+0000 7f3b98a00640 1 --2- 172.21.15.6:0/3832674867 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3b7c0051c0 0x7f3b940cbe40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:28.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:28.872+0000 7f3b98a00640 1 -- 172.21.15.6:0/3832674867 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3b941f41c0 msgr2=0x7f3b941f4650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:28.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:28.872+0000 7f3b98a00640 1 --2- 172.21.15.6:0/3832674867 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3b941f41c0 0x7f3b941f4650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:28.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:28.872+0000 7f3b98a00640 1 -- 172.21.15.6:0/3832674867 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b940ca640 con 0x7f3b940cc650 2024-03-04T09:34:28.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:28.872+0000 7f3b99e00640 1 --2- 172.21.15.6:0/3832674867 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3b941f41c0 0x7f3b941f4650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:34:28.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:28.873+0000 7f3b98a00640 1 --2- 172.21.15.6:0/3832674867 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3b940cc650 0x7f3b941f3c80 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7f3b8000d770 tx=0x7f3b8000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:34:28.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:28.873+0000 7f3b91600640 1 -- 172.21.15.6:0/3832674867 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b80019070 con 0x7f3b940cc650 2024-03-04T09:34:28.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:28.873+0000 7f3b91600640 1 -- 172.21.15.6:0/3832674867 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b800072d0 con 0x7f3b940cc650 2024-03-04T09:34:28.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:28.873+0000 7f3b9a800640 1 -- 172.21.15.6:0/3832674867 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b940ccad0 con 0x7f3b940cc650 2024-03-04T09:34:28.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:28.873+0000 7f3b91600640 1 -- 172.21.15.6:0/3832674867 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b800047e0 con 0x7f3b940cc650 2024-03-04T09:34:28.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:28.873+0000 7f3b9a800640 1 -- 172.21.15.6:0/3832674867 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b940ccd80 con 0x7f3b940cc650 2024-03-04T09:34:28.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:28.875+0000 7f3b9a800640 1 -- 172.21.15.6:0/3832674867 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b9404f330 con 0x7f3b940cc650 2024-03-04T09:34:28.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:28.877+0000 7f3b91600640 1 -- 172.21.15.6:0/3832674867 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3b8001c070 con 0x7f3b940cc650 2024-03-04T09:34:28.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:28.877+0000 7f3b91600640 1 --2- 172.21.15.6:0/3832674867 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f3b7406b500 0x7f3b7406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:28.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:28.877+0000 7f3b91600640 1 -- 172.21.15.6:0/3832674867 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f3b80089b30 con 0x7f3b940cc650 2024-03-04T09:34:28.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:28.878+0000 7f3b99400640 1 --2- 172.21.15.6:0/3832674867 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f3b7406b500 0x7f3b7406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:28.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:28.879+0000 7f3b99400640 1 --2- 172.21.15.6:0/3832674867 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f3b7406b500 0x7f3b7406d9c0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f3b88000f60 tx=0x7f3b88009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:34:28.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:28.882+0000 7f3b91600640 1 -- 172.21.15.6:0/3832674867 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3b80058960 con 0x7f3b940cc650 2024-03-04T09:34:29.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:29.123+0000 7f3b9a800640 1 -- 172.21.15.6:0/3832674867 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3b9415c850 con 0x7f3b940cc650 2024-03-04T09:34:29.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:29.124+0000 7f3b91600640 1 -- 172.21.15.6:0/3832674867 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3465 (secure 0 0 0) 0x7f3b80058260 con 0x7f3b940cc650 2024-03-04T09:34:29.122 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:34:29.122 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:33:54.652601+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"laggy_since":"2024-03-04T09:33:54.652601+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":9}]} 2024-03-04T09:34:29.122 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 100 2024-03-04T09:34:29.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:29.127+0000 7f3b9a800640 1 -- 172.21.15.6:0/3832674867 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f3b7406b500 msgr2=0x7f3b7406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:29.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:29.127+0000 7f3b9a800640 1 --2- 172.21.15.6:0/3832674867 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f3b7406b500 0x7f3b7406d9c0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f3b88000f60 tx=0x7f3b88009040 comp rx=0 tx=0).stop 2024-03-04T09:34:29.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:29.127+0000 7f3b9a800640 1 -- 172.21.15.6:0/3832674867 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3b940cc650 msgr2=0x7f3b941f3c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:29.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:29.127+0000 7f3b9a800640 1 --2- 172.21.15.6:0/3832674867 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3b940cc650 0x7f3b941f3c80 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7f3b8000d770 tx=0x7f3b8000dc30 comp rx=0 tx=0).stop 2024-03-04T09:34:29.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:29.127+0000 7f3b9a800640 1 -- 172.21.15.6:0/3832674867 shutdown_connections 2024-03-04T09:34:29.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:29.127+0000 7f3b9a800640 1 --2- 172.21.15.6:0/3832674867 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3b941f41c0 0x7f3b941f4650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:29.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:29.127+0000 7f3b9a800640 1 --2- 172.21.15.6:0/3832674867 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3b940cc650 0x7f3b941f3c80 unknown :-1 s=CLOSED pgs=1635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:29.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:29.127+0000 7f3b9a800640 1 --2- 172.21.15.6:0/3832674867 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f3b7406b500 0x7f3b7406d9c0 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:29.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:29.127+0000 7f3b9a800640 1 --2- 172.21.15.6:0/3832674867 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3b7c0051c0 0x7f3b940cbe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:29.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:29.127+0000 7f3b9a800640 1 -- 172.21.15.6:0/3832674867 >> 172.21.15.6:0/3832674867 conn(0x7f3b940d3d30 msgr2=0x7f3b940cf070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:34:29.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:29.127+0000 7f3b9a800640 1 -- 172.21.15.6:0/3832674867 shutdown_connections 2024-03-04T09:34:29.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:29.127+0000 7f3b9a800640 1 -- 172.21.15.6:0/3832674867 wait complete. 2024-03-04T09:34:29.346 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:32:57.159681+0000', 'modified': '2024-03-04T09:33:54.652601+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8214, 'mds_1': 6796}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6796': {'gid': 6796, 'name': 'd', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.146:6837/2458070840', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 2458070840}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 2458070840}]}, 'join_fscid': -1, 'laggy_since': '2024-03-04T09:33:54.652601+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8214': {'gid': 8214, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2446627517', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2446627517}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2446627517}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8214, 'qdb_cluster': [8214]} 2024-03-04T09:34:29.346 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6796:up:stopping 2024-03-04T09:34:29.835 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-04T09:34:30.347 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:34:30.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:30.571+0000 7fe19cc00640 1 Processor -- start 2024-03-04T09:34:30.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:30.571+0000 7fe19cc00640 1 -- start start 2024-03-04T09:34:30.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:30.573+0000 7fe19cc00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fe1980cd060 0x7fe198161ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:30.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:30.573+0000 7fe19cc00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fe198162000 0x7fe198070c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:30.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:30.573+0000 7fe19cc00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1981627f0 con 0x7fe19815efc0 2024-03-04T09:34:30.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:30.573+0000 7fe19cc00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe19815f3a0 con 0x7fe1980cd060 2024-03-04T09:34:30.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:30.573+0000 7fe19cc00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe19806d190 con 0x7fe198162000 2024-03-04T09:34:30.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:30.573+0000 7fe18fe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fe1980cd060 0x7fe198161ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:30.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:30.573+0000 7fe18fe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fe1980cd060 0x7fe198161ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51888/0 (socket says 172.21.15.6:51888) 2024-03-04T09:34:30.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:30.573+0000 7fe18fe00640 1 -- 172.21.15.6:0/143612692 learned_addr learned my addr 172.21.15.6:0/143612692 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:34:30.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:30.573+0000 7fe197e00640 1 --2- 172.21.15.6:0/143612692 >> v2:172.21.15.146:3300/0 conn(0x7fe198162000 0x7fe198070c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:30.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:30.574+0000 7fe18fe00640 1 -- 172.21.15.6:0/143612692 >> v2:172.21.15.146:3300/0 conn(0x7fe198162000 msgr2=0x7fe198070c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:34:30.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:30.574+0000 7fe18fe00640 1 --2- 172.21.15.6:0/143612692 >> v2:172.21.15.146:3300/0 conn(0x7fe198162000 0x7fe198070c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:30.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:30.574+0000 7fe18fe00640 1 -- 172.21.15.6:0/143612692 >> v1:172.21.15.146:6789/0 conn(0x7fe19815efc0 legacy=0x7fe1980cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:34:30.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:30.574+0000 7fe18fe00640 1 -- 172.21.15.6:0/143612692 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe19806c860 con 0x7fe1980cd060 2024-03-04T09:34:30.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:30.575+0000 7fe18fe00640 1 --2- 172.21.15.6:0/143612692 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe1980cd060 0x7fe198161ac0 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 crypto rx=0x7fe1980cf9e0 tx=0x7fe18002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1783881d1ad9ecf server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:34:30.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:30.575+0000 7fe196a00640 1 -- 172.21.15.6:0/143612692 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe180035020 con 0x7fe1980cd060 2024-03-04T09:34:30.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:30.576+0000 7fe196a00640 1 -- 172.21.15.6:0/143612692 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe180031870 con 0x7fe1980cd060 2024-03-04T09:34:30.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:30.576+0000 7fe196a00640 1 -- 172.21.15.6:0/143612692 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe180031c40 con 0x7fe1980cd060 2024-03-04T09:34:30.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:30.576+0000 7fe19cc00640 1 -- 172.21.15.6:0/143612692 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe1980cd060 msgr2=0x7fe198161ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:30.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:30.576+0000 7fe19cc00640 1 --2- 172.21.15.6:0/143612692 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe1980cd060 0x7fe198161ac0 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 crypto rx=0x7fe1980cf9e0 tx=0x7fe18002b910 comp rx=0 tx=0).stop 2024-03-04T09:34:30.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:30.576+0000 7fe19cc00640 1 -- 172.21.15.6:0/143612692 shutdown_connections 2024-03-04T09:34:30.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:30.576+0000 7fe19cc00640 1 --2- 172.21.15.6:0/143612692 >> v2:172.21.15.146:3300/0 conn(0x7fe198162000 0x7fe198070c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:30.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:30.576+0000 7fe19cc00640 1 --2- 172.21.15.6:0/143612692 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe1980cd060 0x7fe198161ac0 unknown :-1 s=CLOSED pgs=1636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:30.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:30.576+0000 7fe19cc00640 1 -- 172.21.15.6:0/143612692 >> 172.21.15.6:0/143612692 conn(0x7fe19806b3d0 msgr2=0x7fe19806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:34:30.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:30.577+0000 7fe19cc00640 1 -- 172.21.15.6:0/143612692 shutdown_connections 2024-03-04T09:34:30.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:30.577+0000 7fe19cc00640 1 -- 172.21.15.6:0/143612692 wait complete. 2024-03-04T09:34:30.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:30.578+0000 7fe19cc00640 1 Processor -- start 2024-03-04T09:34:30.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:30.578+0000 7fe19cc00640 1 -- start start 2024-03-04T09:34:30.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:30.579+0000 7fe19cc00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe1980cd060 0x7fe1981415c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:30.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:30.579+0000 7fe19cc00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe19815efc0 0x7fe1981f8690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:30.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:30.579+0000 7fe19cc00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe198162000 0x7fe198146530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:30.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:30.580+0000 7fe19cc00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe198162470 con 0x7fe19815efc0 2024-03-04T09:34:30.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:30.580+0000 7fe19cc00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe19806ce20 con 0x7fe1980cd060 2024-03-04T09:34:30.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:30.580+0000 7fe19cc00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe19806d620 con 0x7fe198162000 2024-03-04T09:34:30.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:30.580+0000 7fe18fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe19815efc0 0x7fe1981f8690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:30.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:30.580+0000 7fe18fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe19815efc0 0x7fe1981f8690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51904/0 (socket says 172.21.15.6:51904) 2024-03-04T09:34:30.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:30.580+0000 7fe18fe00640 1 -- 172.21.15.6:0/1881891765 learned_addr learned my addr 172.21.15.6:0/1881891765 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:34:30.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:30.580+0000 7fe197400640 1 --2- 172.21.15.6:0/1881891765 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe1980cd060 0x7fe1981415c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:30.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:30.580+0000 7fe197e00640 1 --2- 172.21.15.6:0/1881891765 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe198162000 0x7fe198146530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:30.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:30.580+0000 7fe18fe00640 1 -- 172.21.15.6:0/1881891765 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe198162000 msgr2=0x7fe198146530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:30.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:30.580+0000 7fe18fe00640 1 --2- 172.21.15.6:0/1881891765 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe198162000 0x7fe198146530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:30.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:30.580+0000 7fe18fe00640 1 -- 172.21.15.6:0/1881891765 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe1980cd060 msgr2=0x7fe1981415c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:30.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:30.580+0000 7fe18fe00640 1 --2- 172.21.15.6:0/1881891765 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe1980cd060 0x7fe1981415c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:30.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:30.580+0000 7fe18fe00640 1 -- 172.21.15.6:0/1881891765 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe198145490 con 0x7fe19815efc0 2024-03-04T09:34:30.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:30.581+0000 7fe18fe00640 1 --2- 172.21.15.6:0/1881891765 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe19815efc0 0x7fe1981f8690 secure :-1 s=READY pgs=1637 cs=0 l=1 rev1=1 crypto rx=0x7fe180000c00 tx=0x7fe1800046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:34:30.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:30.581+0000 7fe18f400640 1 -- 172.21.15.6:0/1881891765 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe180040070 con 0x7fe19815efc0 2024-03-04T09:34:30.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:30.581+0000 7fe18f400640 1 -- 172.21.15.6:0/1881891765 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe180030d00 con 0x7fe19815efc0 2024-03-04T09:34:30.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:30.581+0000 7fe19cc00640 1 -- 172.21.15.6:0/1881891765 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe198145770 con 0x7fe19815efc0 2024-03-04T09:34:30.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:30.581+0000 7fe18f400640 1 -- 172.21.15.6:0/1881891765 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe180035040 con 0x7fe19815efc0 2024-03-04T09:34:30.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:30.581+0000 7fe19cc00640 1 -- 172.21.15.6:0/1881891765 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe198145d30 con 0x7fe19815efc0 2024-03-04T09:34:30.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:30.583+0000 7fe19cc00640 1 -- 172.21.15.6:0/1881891765 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe19804f330 con 0x7fe19815efc0 2024-03-04T09:34:30.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:30.584+0000 7fe18f400640 1 -- 172.21.15.6:0/1881891765 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe18004b060 con 0x7fe19815efc0 2024-03-04T09:34:30.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:30.584+0000 7fe18f400640 1 --2- 172.21.15.6:0/1881891765 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fe17406b4d0 0x7fe17406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:30.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:30.585+0000 7fe18f400640 1 -- 172.21.15.6:0/1881891765 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fe1800ac000 con 0x7fe19815efc0 2024-03-04T09:34:30.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:30.585+0000 7fe197400640 1 --2- 172.21.15.6:0/1881891765 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fe17406b4d0 0x7fe17406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:30.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:30.586+0000 7fe197400640 1 --2- 172.21.15.6:0/1881891765 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fe17406b4d0 0x7fe17406d990 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fe188004ea0 tx=0x7fe188009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:34:30.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:30.589+0000 7fe18f400640 1 -- 172.21.15.6:0/1881891765 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe18007ae40 con 0x7fe19815efc0 2024-03-04T09:34:30.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:30.827+0000 7fe19cc00640 1 -- 172.21.15.6:0/1881891765 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe198161ac0 con 0x7fe19815efc0 2024-03-04T09:34:30.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:30.829+0000 7fe18f400640 1 -- 172.21.15.6:0/1881891765 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3465 (secure 0 0 0) 0x7fe18007a740 con 0x7fe19815efc0 2024-03-04T09:34:30.826 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:34:30.826 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:33:54.652601+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"laggy_since":"2024-03-04T09:33:54.652601+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":9}]} 2024-03-04T09:34:30.826 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 100 2024-03-04T09:34:30.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:30.832+0000 7fe19cc00640 1 -- 172.21.15.6:0/1881891765 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fe17406b4d0 msgr2=0x7fe17406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:30.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:30.832+0000 7fe19cc00640 1 --2- 172.21.15.6:0/1881891765 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fe17406b4d0 0x7fe17406d990 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fe188004ea0 tx=0x7fe188009040 comp rx=0 tx=0).stop 2024-03-04T09:34:30.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:30.832+0000 7fe19cc00640 1 -- 172.21.15.6:0/1881891765 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe19815efc0 msgr2=0x7fe1981f8690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:30.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:30.832+0000 7fe19cc00640 1 --2- 172.21.15.6:0/1881891765 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe19815efc0 0x7fe1981f8690 secure :-1 s=READY pgs=1637 cs=0 l=1 rev1=1 crypto rx=0x7fe180000c00 tx=0x7fe1800046c0 comp rx=0 tx=0).stop 2024-03-04T09:34:30.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:30.832+0000 7fe19cc00640 1 -- 172.21.15.6:0/1881891765 shutdown_connections 2024-03-04T09:34:30.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:30.832+0000 7fe19cc00640 1 --2- 172.21.15.6:0/1881891765 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fe17406b4d0 0x7fe17406d990 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:30.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:30.832+0000 7fe19cc00640 1 --2- 172.21.15.6:0/1881891765 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe198162000 0x7fe198146530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:30.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:30.832+0000 7fe19cc00640 1 --2- 172.21.15.6:0/1881891765 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe19815efc0 0x7fe1981f8690 unknown :-1 s=CLOSED pgs=1637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:30.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:30.832+0000 7fe19cc00640 1 --2- 172.21.15.6:0/1881891765 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe1980cd060 0x7fe1981415c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:30.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:30.832+0000 7fe19cc00640 1 -- 172.21.15.6:0/1881891765 >> 172.21.15.6:0/1881891765 conn(0x7fe19806b3d0 msgr2=0x7fe1980affc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:34:30.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:30.832+0000 7fe19cc00640 1 -- 172.21.15.6:0/1881891765 shutdown_connections 2024-03-04T09:34:30.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:30.832+0000 7fe19cc00640 1 -- 172.21.15.6:0/1881891765 wait complete. 2024-03-04T09:34:31.050 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:32:57.159681+0000', 'modified': '2024-03-04T09:33:54.652601+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8214, 'mds_1': 6796}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6796': {'gid': 6796, 'name': 'd', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.146:6837/2458070840', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 2458070840}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 2458070840}]}, 'join_fscid': -1, 'laggy_since': '2024-03-04T09:33:54.652601+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8214': {'gid': 8214, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2446627517', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2446627517}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2446627517}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8214, 'qdb_cluster': [8214]} 2024-03-04T09:34:31.050 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6796:up:stopping 2024-03-04T09:34:31.837 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-04T09:34:32.051 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:34:32.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:32.268+0000 7f5c56200640 1 Processor -- start 2024-03-04T09:34:32.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:32.268+0000 7f5c56200640 1 -- start start 2024-03-04T09:34:32.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:32.271+0000 7f5c56200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f5c500d2b40 0x7f5c500d0f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:32.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:32.271+0000 7f5c56200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f5c500d1480 0x7f5c500b40d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:32.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:32.271+0000 7f5c56200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c500b3940 con 0x7f5c5015e2e0 2024-03-04T09:34:32.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:32.271+0000 7f5c56200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c500d4e70 con 0x7f5c500d1480 2024-03-04T09:34:32.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:32.271+0000 7f5c56200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c500b3c40 con 0x7f5c500d2b40 2024-03-04T09:34:32.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:32.272+0000 7f5c4fe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f5c500d1480 0x7f5c500b40d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:32.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:32.272+0000 7f5c4fe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f5c500d1480 0x7f5c500b40d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51908/0 (socket says 172.21.15.6:51908) 2024-03-04T09:34:32.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:32.272+0000 7f5c4fe00640 1 -- 172.21.15.6:0/3482372444 learned_addr learned my addr 172.21.15.6:0/3482372444 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:34:32.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:32.272+0000 7f5c4ea00640 1 --2- 172.21.15.6:0/3482372444 >> v2:172.21.15.146:3300/0 conn(0x7f5c500d2b40 0x7f5c500d0f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:32.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:32.272+0000 7f5c4fe00640 1 -- 172.21.15.6:0/3482372444 >> v2:172.21.15.146:3300/0 conn(0x7f5c500d2b40 msgr2=0x7f5c500d0f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:34:32.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:32.272+0000 7f5c4fe00640 1 --2- 172.21.15.6:0/3482372444 >> v2:172.21.15.146:3300/0 conn(0x7f5c500d2b40 0x7f5c500d0f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:32.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:32.272+0000 7f5c4fe00640 1 -- 172.21.15.6:0/3482372444 >> v1:172.21.15.146:6789/0 conn(0x7f5c5015e2e0 legacy=0x7f5c500d0830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:32.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:32.272+0000 7f5c4fe00640 1 -- 172.21.15.6:0/3482372444 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c500cf580 con 0x7f5c500d1480 2024-03-04T09:34:32.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:32.273+0000 7f5c4fe00640 1 --2- 172.21.15.6:0/3482372444 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5c500d1480 0x7f5c500b40d0 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7f5c500d3510 tx=0x7f5c4802bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=458b6bbdf270acdb server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:34:32.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:32.274+0000 7f5c4e000640 1 -- 172.21.15.6:0/3482372444 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c4802cb10 con 0x7f5c500d1480 2024-03-04T09:34:32.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:32.274+0000 7f5c4e000640 1 -- 172.21.15.6:0/3482372444 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c4802ccb0 con 0x7f5c500d1480 2024-03-04T09:34:32.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:32.274+0000 7f5c4e000640 1 -- 172.21.15.6:0/3482372444 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c4802ce60 con 0x7f5c500d1480 2024-03-04T09:34:32.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:32.274+0000 7f5c56200640 1 -- 172.21.15.6:0/3482372444 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5c500d1480 msgr2=0x7f5c500b40d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:32.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:32.274+0000 7f5c56200640 1 --2- 172.21.15.6:0/3482372444 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5c500d1480 0x7f5c500b40d0 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7f5c500d3510 tx=0x7f5c4802bf70 comp rx=0 tx=0).stop 2024-03-04T09:34:32.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:32.274+0000 7f5c56200640 1 -- 172.21.15.6:0/3482372444 shutdown_connections 2024-03-04T09:34:32.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:32.274+0000 7f5c56200640 1 --2- 172.21.15.6:0/3482372444 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5c500d1480 0x7f5c500b40d0 unknown :-1 s=CLOSED pgs=1638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:32.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:32.274+0000 7f5c56200640 1 --2- 172.21.15.6:0/3482372444 >> v2:172.21.15.146:3300/0 conn(0x7f5c500d2b40 0x7f5c500d0f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:32.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:32.274+0000 7f5c56200640 1 -- 172.21.15.6:0/3482372444 >> 172.21.15.6:0/3482372444 conn(0x7f5c500d3f60 msgr2=0x7f5c500d4370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:34:32.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:32.275+0000 7f5c56200640 1 -- 172.21.15.6:0/3482372444 shutdown_connections 2024-03-04T09:34:32.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:32.275+0000 7f5c56200640 1 -- 172.21.15.6:0/3482372444 wait complete. 2024-03-04T09:34:32.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:32.276+0000 7f5c56200640 1 Processor -- start 2024-03-04T09:34:32.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:32.277+0000 7f5c56200640 1 -- start start 2024-03-04T09:34:32.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:32.278+0000 7f5c56200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5c500d2b40 0x7f5c500cb030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:32.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:32.278+0000 7f5c56200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5c5015e2e0 0x7f5c500c9d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:32.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:32.278+0000 7f5c56200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5c50153610 0x7f5c5015c160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:32.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:32.278+0000 7f5c56200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c500cfe20 con 0x7f5c5015e2e0 2024-03-04T09:34:32.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:32.278+0000 7f5c56200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c500cfb40 con 0x7f5c500d2b40 2024-03-04T09:34:32.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:32.278+0000 7f5c56200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c500b4610 con 0x7f5c50153610 2024-03-04T09:34:32.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:32.278+0000 7f5c4ea00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5c5015e2e0 0x7f5c500c9d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:32.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:32.278+0000 7f5c4ea00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5c5015e2e0 0x7f5c500c9d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51912/0 (socket says 172.21.15.6:51912) 2024-03-04T09:34:32.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:32.278+0000 7f5c4ea00640 1 -- 172.21.15.6:0/1040373029 learned_addr learned my addr 172.21.15.6:0/1040373029 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:34:32.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:32.278+0000 7f5c4f400640 1 --2- 172.21.15.6:0/1040373029 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5c500d2b40 0x7f5c500cb030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:32.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:32.278+0000 7f5c4fe00640 1 --2- 172.21.15.6:0/1040373029 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5c50153610 0x7f5c5015c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:32.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:32.278+0000 7f5c4ea00640 1 -- 172.21.15.6:0/1040373029 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5c50153610 msgr2=0x7f5c5015c160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:32.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:32.279+0000 7f5c4ea00640 1 --2- 172.21.15.6:0/1040373029 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5c50153610 0x7f5c5015c160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:32.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:32.279+0000 7f5c4ea00640 1 -- 172.21.15.6:0/1040373029 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5c500d2b40 msgr2=0x7f5c500cb030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:32.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:32.279+0000 7f5c4ea00640 1 --2- 172.21.15.6:0/1040373029 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5c500d2b40 0x7f5c500cb030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:32.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:32.279+0000 7f5c4ea00640 1 -- 172.21.15.6:0/1040373029 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c500cb6a0 con 0x7f5c5015e2e0 2024-03-04T09:34:32.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:32.279+0000 7f5c4fe00640 1 --2- 172.21.15.6:0/1040373029 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5c50153610 0x7f5c5015c160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:34:32.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:32.279+0000 7f5c4ea00640 1 --2- 172.21.15.6:0/1040373029 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5c5015e2e0 0x7f5c500c9d60 secure :-1 s=READY pgs=1639 cs=0 l=1 rev1=1 crypto rx=0x7f5c3c00d7d0 tx=0x7f5c3c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:34:32.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:32.279+0000 7f5c47400640 1 -- 172.21.15.6:0/1040373029 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c3c00b6c0 con 0x7f5c5015e2e0 2024-03-04T09:34:32.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:32.279+0000 7f5c47400640 1 -- 172.21.15.6:0/1040373029 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c3c019070 con 0x7f5c5015e2e0 2024-03-04T09:34:32.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:32.279+0000 7f5c47400640 1 -- 172.21.15.6:0/1040373029 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c3c00bac0 con 0x7f5c5015e2e0 2024-03-04T09:34:32.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:32.279+0000 7f5c56200640 1 -- 172.21.15.6:0/1040373029 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c500cb980 con 0x7f5c5015e2e0 2024-03-04T09:34:32.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:32.280+0000 7f5c56200640 1 -- 172.21.15.6:0/1040373029 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c500cbf40 con 0x7f5c5015e2e0 2024-03-04T09:34:32.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:32.281+0000 7f5c56200640 1 -- 172.21.15.6:0/1040373029 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c5008d610 con 0x7f5c5015e2e0 2024-03-04T09:34:32.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:32.282+0000 7f5c47400640 1 -- 172.21.15.6:0/1040373029 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5c3c00bc60 con 0x7f5c5015e2e0 2024-03-04T09:34:32.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:32.283+0000 7f5c47400640 1 --2- 172.21.15.6:0/1040373029 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f5c3006b500 0x7f5c3006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:32.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:32.283+0000 7f5c47400640 1 -- 172.21.15.6:0/1040373029 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f5c3c08a760 con 0x7f5c5015e2e0 2024-03-04T09:34:32.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:32.284+0000 7f5c4f400640 1 --2- 172.21.15.6:0/1040373029 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f5c3006b500 0x7f5c3006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:32.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:32.285+0000 7f5c4f400640 1 --2- 172.21.15.6:0/1040373029 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f5c3006b500 0x7f5c3006d9c0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f5c500c8c80 tx=0x7f5c40009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:34:32.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:32.288+0000 7f5c47400640 1 -- 172.21.15.6:0/1040373029 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5c5008d610 con 0x7f5c5015e2e0 2024-03-04T09:34:32.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:32.612+0000 7f5c56200640 1 -- 172.21.15.6:0/1040373029 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5c5004f3a0 con 0x7f5c5015e2e0 2024-03-04T09:34:32.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:32.613+0000 7f5c47400640 1 -- 172.21.15.6:0/1040373029 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3465 (secure 0 0 0) 0x7f5c3c059590 con 0x7f5c5015e2e0 2024-03-04T09:34:32.610 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:34:32.610 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:33:54.652601+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"laggy_since":"2024-03-04T09:33:54.652601+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":9}]} 2024-03-04T09:34:32.610 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 100 2024-03-04T09:34:32.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:32.615+0000 7f5c56200640 1 -- 172.21.15.6:0/1040373029 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f5c3006b500 msgr2=0x7f5c3006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:32.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:32.615+0000 7f5c56200640 1 --2- 172.21.15.6:0/1040373029 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f5c3006b500 0x7f5c3006d9c0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f5c500c8c80 tx=0x7f5c40009040 comp rx=0 tx=0).stop 2024-03-04T09:34:32.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:32.615+0000 7f5c56200640 1 -- 172.21.15.6:0/1040373029 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5c5015e2e0 msgr2=0x7f5c500c9d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:32.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:32.615+0000 7f5c56200640 1 --2- 172.21.15.6:0/1040373029 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5c5015e2e0 0x7f5c500c9d60 secure :-1 s=READY pgs=1639 cs=0 l=1 rev1=1 crypto rx=0x7f5c3c00d7d0 tx=0x7f5c3c00dc90 comp rx=0 tx=0).stop 2024-03-04T09:34:32.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:32.615+0000 7f5c56200640 1 -- 172.21.15.6:0/1040373029 shutdown_connections 2024-03-04T09:34:32.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:32.615+0000 7f5c56200640 1 --2- 172.21.15.6:0/1040373029 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f5c3006b500 0x7f5c3006d9c0 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:32.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:32.615+0000 7f5c56200640 1 --2- 172.21.15.6:0/1040373029 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5c50153610 0x7f5c5015c160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:32.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:32.615+0000 7f5c56200640 1 --2- 172.21.15.6:0/1040373029 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5c5015e2e0 0x7f5c500c9d60 unknown :-1 s=CLOSED pgs=1639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:32.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:32.615+0000 7f5c56200640 1 --2- 172.21.15.6:0/1040373029 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5c500d2b40 0x7f5c500cb030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:32.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:32.615+0000 7f5c56200640 1 -- 172.21.15.6:0/1040373029 >> 172.21.15.6:0/1040373029 conn(0x7f5c500d3f60 msgr2=0x7f5c5006bb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:34:32.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:32.615+0000 7f5c56200640 1 -- 172.21.15.6:0/1040373029 shutdown_connections 2024-03-04T09:34:32.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:32.615+0000 7f5c56200640 1 -- 172.21.15.6:0/1040373029 wait complete. 2024-03-04T09:34:32.830 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:32:57.159681+0000', 'modified': '2024-03-04T09:33:54.652601+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8214, 'mds_1': 6796}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6796': {'gid': 6796, 'name': 'd', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.146:6837/2458070840', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 2458070840}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 2458070840}]}, 'join_fscid': -1, 'laggy_since': '2024-03-04T09:33:54.652601+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8214': {'gid': 8214, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2446627517', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2446627517}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2446627517}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8214, 'qdb_cluster': [8214]} 2024-03-04T09:34:32.830 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6796:up:stopping 2024-03-04T09:34:33.831 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:34:33.840 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-04T09:34:34.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:34.063+0000 7f01ce800640 1 Processor -- start 2024-03-04T09:34:34.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:34.064+0000 7f01ce800640 1 -- start start 2024-03-04T09:34:34.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:34.067+0000 7f01ce800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f01c815a550 0x7f01c80aefe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:34.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:34.067+0000 7f01ce800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f01c815a930 0x7f01c80ce4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:34.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:34.067+0000 7f01ce800640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01c8071640 con 0x7f01c80ce9f0 2024-03-04T09:34:34.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:34.067+0000 7f01ce800640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01c8071920 con 0x7f01c815a550 2024-03-04T09:34:34.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:34.067+0000 7f01ce800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01c8071c00 con 0x7f01c815a930 2024-03-04T09:34:34.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:34.068+0000 7f01c7400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f01c815a550 0x7f01c80aefe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:34.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:34.068+0000 7f01bfe00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f01c815a930 0x7f01c80ce4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:34.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:34.068+0000 7f01c7400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f01c815a550 0x7f01c80aefe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.6:47718/0 (socket says 172.21.15.6:47718) 2024-03-04T09:34:34.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:34.068+0000 7f01c7400640 1 -- 172.21.15.6:0/1594403983 learned_addr learned my addr 172.21.15.6:0/1594403983 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:34:34.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:34.069+0000 7f01c7400640 1 -- 172.21.15.6:0/1594403983 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f01c815a930 msgr2=0x7f01c80ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:34:34.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:34.069+0000 7f01c7400640 1 --2- 172.21.15.6:0/1594403983 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f01c815a930 0x7f01c80ce4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:34.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:34.069+0000 7f01c7400640 1 -- 172.21.15.6:0/1594403983 >> v1:172.21.15.146:6789/0 conn(0x7f01c80ce9f0 legacy=0x7f01c80b32f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:34.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:34.069+0000 7f01c7400640 1 -- 172.21.15.6:0/1594403983 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01c8071ee0 con 0x7f01c815a550 2024-03-04T09:34:34.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:34.070+0000 7f01c7400640 1 --2- 172.21.15.6:0/1594403983 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f01c815a550 0x7f01c80aefe0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f01b8009870 tx=0x7f01b802dc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1a7d1ac42cdca720 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:34:34.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:34.071+0000 7f01c6a00640 1 -- 172.21.15.6:0/1594403983 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01b803b070 con 0x7f01c815a550 2024-03-04T09:34:34.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:34.071+0000 7f01c6a00640 1 -- 172.21.15.6:0/1594403983 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01b802e9e0 con 0x7f01c815a550 2024-03-04T09:34:34.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:34.071+0000 7f01c6a00640 1 -- 172.21.15.6:0/1594403983 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01b802edb0 con 0x7f01c815a550 2024-03-04T09:34:34.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:34.072+0000 7f01ce800640 1 -- 172.21.15.6:0/1594403983 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f01c815a550 msgr2=0x7f01c80aefe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:34.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:34.072+0000 7f01ce800640 1 --2- 172.21.15.6:0/1594403983 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f01c815a550 0x7f01c80aefe0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f01b8009870 tx=0x7f01b802dc30 comp rx=0 tx=0).stop 2024-03-04T09:34:34.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:34.072+0000 7f01ce800640 1 -- 172.21.15.6:0/1594403983 shutdown_connections 2024-03-04T09:34:34.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:34.072+0000 7f01ce800640 1 --2- 172.21.15.6:0/1594403983 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f01c815a930 0x7f01c80ce4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:34.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:34.072+0000 7f01ce800640 1 --2- 172.21.15.6:0/1594403983 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f01c815a550 0x7f01c80aefe0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:34.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:34.072+0000 7f01ce800640 1 -- 172.21.15.6:0/1594403983 >> 172.21.15.6:0/1594403983 conn(0x7f01c806afb0 msgr2=0x7f01c806b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:34:34.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:34.072+0000 7f01ce800640 1 -- 172.21.15.6:0/1594403983 shutdown_connections 2024-03-04T09:34:34.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:34.072+0000 7f01ce800640 1 -- 172.21.15.6:0/1594403983 wait complete. 2024-03-04T09:34:34.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:34.076+0000 7f01ce800640 1 Processor -- start 2024-03-04T09:34:34.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:34.076+0000 7f01ce800640 1 -- start start 2024-03-04T09:34:34.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:34.079+0000 7f01ce800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f01c80ce9f0 0x7f01c8148c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:34.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:34.079+0000 7f01ce800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f01c815a550 0x7f01c814b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:34.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:34.079+0000 7f01ce800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f01c815a930 0x7f01c809aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:34.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:34.079+0000 7f01ce800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01c80b52b0 con 0x7f01c80ce9f0 2024-03-04T09:34:34.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:34.079+0000 7f01ce800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01c80b5130 con 0x7f01c815a550 2024-03-04T09:34:34.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:34.079+0000 7f01ce800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01c80ade50 con 0x7f01c815a930 2024-03-04T09:34:34.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:34.079+0000 7f01c7400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f01c80ce9f0 0x7f01c8148c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:34.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:34.079+0000 7f01c7400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f01c80ce9f0 0x7f01c8148c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51922/0 (socket says 172.21.15.6:51922) 2024-03-04T09:34:34.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:34.079+0000 7f01c7400640 1 -- 172.21.15.6:0/3662071167 learned_addr learned my addr 172.21.15.6:0/3662071167 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:34:34.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:34.080+0000 7f01bfe00640 1 --2- 172.21.15.6:0/3662071167 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f01c815a550 0x7f01c814b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:34.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:34.080+0000 7f01c7e00640 1 --2- 172.21.15.6:0/3662071167 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f01c815a930 0x7f01c809aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:34.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:34.080+0000 7f01c7400640 1 -- 172.21.15.6:0/3662071167 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f01c815a930 msgr2=0x7f01c809aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:34.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:34.080+0000 7f01c7400640 1 --2- 172.21.15.6:0/3662071167 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f01c815a930 0x7f01c809aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:34.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:34.080+0000 7f01c7400640 1 -- 172.21.15.6:0/3662071167 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f01c815a550 msgr2=0x7f01c814b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:34.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:34.080+0000 7f01c7400640 1 --2- 172.21.15.6:0/3662071167 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f01c815a550 0x7f01c814b480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:34.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:34.080+0000 7f01c7400640 1 -- 172.21.15.6:0/3662071167 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01c814be40 con 0x7f01c80ce9f0 2024-03-04T09:34:34.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:34.080+0000 7f01c7400640 1 --2- 172.21.15.6:0/3662071167 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f01c80ce9f0 0x7f01c8148c40 secure :-1 s=READY pgs=1640 cs=0 l=1 rev1=1 crypto rx=0x7f01b802dc70 tx=0x7f01b80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:34:34.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:34.080+0000 7f01bf400640 1 -- 172.21.15.6:0/3662071167 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01b8046070 con 0x7f01c80ce9f0 2024-03-04T09:34:34.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:34.081+0000 7f01bf400640 1 -- 172.21.15.6:0/3662071167 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01b8036d00 con 0x7f01c80ce9f0 2024-03-04T09:34:34.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:34.081+0000 7f01bf400640 1 -- 172.21.15.6:0/3662071167 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01b803b070 con 0x7f01c80ce9f0 2024-03-04T09:34:34.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:34.081+0000 7f01ce800640 1 -- 172.21.15.6:0/3662071167 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01c814c120 con 0x7f01c80ce9f0 2024-03-04T09:34:34.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:34.081+0000 7f01ce800640 1 -- 172.21.15.6:0/3662071167 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01c81493f0 con 0x7f01c80ce9f0 2024-03-04T09:34:34.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:34.082+0000 7f01ce800640 1 -- 172.21.15.6:0/3662071167 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01c8069510 con 0x7f01c80ce9f0 2024-03-04T09:34:34.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:34.083+0000 7f01bf400640 1 -- 172.21.15.6:0/3662071167 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f01b803f070 con 0x7f01c80ce9f0 2024-03-04T09:34:34.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:34.084+0000 7f01bf400640 1 --2- 172.21.15.6:0/3662071167 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f01a406b4d0 0x7f01a406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:34.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:34.084+0000 7f01bf400640 1 -- 172.21.15.6:0/3662071167 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f01b80ab6c0 con 0x7f01c80ce9f0 2024-03-04T09:34:34.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:34.085+0000 7f01bfe00640 1 --2- 172.21.15.6:0/3662071167 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f01a406b4d0 0x7f01a406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:34.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:34.086+0000 7f01bfe00640 1 --2- 172.21.15.6:0/3662071167 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f01a406b4d0 0x7f01a406d990 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f01b0009a30 tx=0x7f01b0008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:34:34.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:34.089+0000 7f01bf400640 1 -- 172.21.15.6:0/3662071167 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f01b807a590 con 0x7f01c80ce9f0 2024-03-04T09:34:34.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:34.330+0000 7f01ce800640 1 -- 172.21.15.6:0/3662071167 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f01c804f3a0 con 0x7f01c80ce9f0 2024-03-04T09:34:34.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:34.331+0000 7f01bf400640 1 -- 172.21.15.6:0/3662071167 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3465 (secure 0 0 0) 0x7f01b8079e90 con 0x7f01c80ce9f0 2024-03-04T09:34:34.329 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:34:34.329 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:33:54.652601+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"laggy_since":"2024-03-04T09:33:54.652601+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":9}]} 2024-03-04T09:34:34.329 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 100 2024-03-04T09:34:34.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:34.334+0000 7f01ce800640 1 -- 172.21.15.6:0/3662071167 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f01a406b4d0 msgr2=0x7f01a406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:34.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:34.334+0000 7f01ce800640 1 --2- 172.21.15.6:0/3662071167 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f01a406b4d0 0x7f01a406d990 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f01b0009a30 tx=0x7f01b0008040 comp rx=0 tx=0).stop 2024-03-04T09:34:34.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:34.334+0000 7f01ce800640 1 -- 172.21.15.6:0/3662071167 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f01c80ce9f0 msgr2=0x7f01c8148c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:34.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:34.334+0000 7f01ce800640 1 --2- 172.21.15.6:0/3662071167 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f01c80ce9f0 0x7f01c8148c40 secure :-1 s=READY pgs=1640 cs=0 l=1 rev1=1 crypto rx=0x7f01b802dc70 tx=0x7f01b80046c0 comp rx=0 tx=0).stop 2024-03-04T09:34:34.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:34.334+0000 7f01ce800640 1 -- 172.21.15.6:0/3662071167 shutdown_connections 2024-03-04T09:34:34.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:34.334+0000 7f01ce800640 1 --2- 172.21.15.6:0/3662071167 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f01a406b4d0 0x7f01a406d990 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:34.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:34.334+0000 7f01ce800640 1 --2- 172.21.15.6:0/3662071167 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f01c815a930 0x7f01c809aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:34.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:34.334+0000 7f01ce800640 1 --2- 172.21.15.6:0/3662071167 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f01c815a550 0x7f01c814b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:34.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:34.334+0000 7f01ce800640 1 --2- 172.21.15.6:0/3662071167 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f01c80ce9f0 0x7f01c8148c40 unknown :-1 s=CLOSED pgs=1640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:34.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:34.334+0000 7f01ce800640 1 -- 172.21.15.6:0/3662071167 >> 172.21.15.6:0/3662071167 conn(0x7f01c806afb0 msgr2=0x7f01c80cc030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:34:34.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:34.334+0000 7f01ce800640 1 -- 172.21.15.6:0/3662071167 shutdown_connections 2024-03-04T09:34:34.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:34.335+0000 7f01ce800640 1 -- 172.21.15.6:0/3662071167 wait complete. 2024-03-04T09:34:34.551 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:32:57.159681+0000', 'modified': '2024-03-04T09:33:54.652601+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8214, 'mds_1': 6796}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6796': {'gid': 6796, 'name': 'd', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.146:6837/2458070840', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 2458070840}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 2458070840}]}, 'join_fscid': -1, 'laggy_since': '2024-03-04T09:33:54.652601+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8214': {'gid': 8214, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2446627517', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2446627517}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2446627517}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8214, 'qdb_cluster': [8214]} 2024-03-04T09:34:34.551 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6796:up:stopping 2024-03-04T09:34:35.552 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:34:35.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:35.824+0000 7fd3c7200640 1 Processor -- start 2024-03-04T09:34:35.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:35.825+0000 7fd3c7200640 1 -- start start 2024-03-04T09:34:35.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:35.826+0000 7fd3c7200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd3c80d04b0 0x7fd3c8070990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:35.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:35.826+0000 7fd3c7200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3c80cd0c0 con 0x7fd3c80700c0 2024-03-04T09:34:35.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:35.826+0000 7fd3c7200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3c80cd3a0 con 0x7fd3c815f450 2024-03-04T09:34:35.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:35.826+0000 7fd3c7200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3c80d29e0 con 0x7fd3c80d04b0 2024-03-04T09:34:35.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:35.827+0000 7fd3bfe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd3c80d04b0 0x7fd3c8070990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:35.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:35.827+0000 7fd3bfe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd3c80d04b0 0x7fd3c8070990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51932/0 (socket says 172.21.15.6:51932) 2024-03-04T09:34:35.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:35.827+0000 7fd3bfe00640 1 -- 172.21.15.6:0/13081631 learned_addr learned my addr 172.21.15.6:0/13081631 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:34:35.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:35.827+0000 7fd3bfe00640 1 -- 172.21.15.6:0/13081631 >> v1:172.21.15.146:6789/0 conn(0x7fd3c815f450 legacy=0x7fd3c80cfda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:35.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:35.827+0000 7fd3bfe00640 1 -- 172.21.15.6:0/13081631 >> v1:172.21.15.6:6789/0 conn(0x7fd3c80700c0 legacy=0x7fd3c80cde80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:35.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:35.827+0000 7fd3bfe00640 1 -- 172.21.15.6:0/13081631 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3c80cd680 con 0x7fd3c80d04b0 2024-03-04T09:34:35.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:35.828+0000 7fd3bfe00640 1 --2- 172.21.15.6:0/13081631 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd3c80d04b0 0x7fd3c8070990 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7fd3b0009870 tx=0x7fd3b002ba40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c32afbafe93ae7e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:34:35.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:35.829+0000 7fd3c5400640 1 -- 172.21.15.6:0/13081631 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3b002f860 con 0x7fd3c80d04b0 2024-03-04T09:34:35.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:35.829+0000 7fd3c5400640 1 -- 172.21.15.6:0/13081631 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3b002fa00 con 0x7fd3c80d04b0 2024-03-04T09:34:35.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:35.829+0000 7fd3c5400640 1 -- 172.21.15.6:0/13081631 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3b002fdd0 con 0x7fd3c80d04b0 2024-03-04T09:34:35.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:35.829+0000 7fd3c7200640 1 -- 172.21.15.6:0/13081631 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd3c80d04b0 msgr2=0x7fd3c8070990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:35.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:35.829+0000 7fd3c7200640 1 --2- 172.21.15.6:0/13081631 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd3c80d04b0 0x7fd3c8070990 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7fd3b0009870 tx=0x7fd3b002ba40 comp rx=0 tx=0).stop 2024-03-04T09:34:35.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:35.829+0000 7fd3c7200640 1 -- 172.21.15.6:0/13081631 shutdown_connections 2024-03-04T09:34:35.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:35.829+0000 7fd3c7200640 1 --2- 172.21.15.6:0/13081631 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd3c80d04b0 0x7fd3c8070990 unknown :-1 s=CLOSED pgs=1642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:35.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:35.829+0000 7fd3c7200640 1 -- 172.21.15.6:0/13081631 >> 172.21.15.6:0/13081631 conn(0x7fd3c80d1ae0 msgr2=0x7fd3c80d1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:34:35.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:35.829+0000 7fd3c7200640 1 -- 172.21.15.6:0/13081631 shutdown_connections 2024-03-04T09:34:35.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:35.830+0000 7fd3c7200640 1 -- 172.21.15.6:0/13081631 wait complete. 2024-03-04T09:34:35.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:35.831+0000 7fd3c7200640 1 Processor -- start 2024-03-04T09:34:35.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:35.831+0000 7fd3c7200640 1 -- start start 2024-03-04T09:34:35.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:35.832+0000 7fd3c7200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd3c80700c0 0x7fd3c809c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:35.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:35.832+0000 7fd3c7200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd3c80d04b0 0x7fd3c80cb160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:35.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:35.832+0000 7fd3c7200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd3c815f450 0x7fd3c815b130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:35.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:35.832+0000 7fd3c7200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3c815f830 con 0x7fd3c80700c0 2024-03-04T09:34:35.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:35.832+0000 7fd3c7200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3c80ae8b0 con 0x7fd3c80d04b0 2024-03-04T09:34:35.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:35.832+0000 7fd3c7200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3c815f9b0 con 0x7fd3c815f450 2024-03-04T09:34:35.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:35.832+0000 7fd3c5e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd3c80700c0 0x7fd3c809c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:35.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:35.833+0000 7fd3c5e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd3c80700c0 0x7fd3c809c3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51936/0 (socket says 172.21.15.6:51936) 2024-03-04T09:34:35.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:35.833+0000 7fd3c5e00640 1 -- 172.21.15.6:0/490839442 learned_addr learned my addr 172.21.15.6:0/490839442 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:34:35.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:35.833+0000 7fd3bfe00640 1 --2- 172.21.15.6:0/490839442 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd3c80d04b0 0x7fd3c80cb160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:35.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:35.833+0000 7fd3c6800640 1 --2- 172.21.15.6:0/490839442 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd3c815f450 0x7fd3c815b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:35.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:35.833+0000 7fd3c5e00640 1 -- 172.21.15.6:0/490839442 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd3c815f450 msgr2=0x7fd3c815b130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:35.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:35.833+0000 7fd3c5e00640 1 --2- 172.21.15.6:0/490839442 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd3c815f450 0x7fd3c815b130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:35.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:35.833+0000 7fd3c5e00640 1 -- 172.21.15.6:0/490839442 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd3c80d04b0 msgr2=0x7fd3c80cb160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:35.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:35.833+0000 7fd3c5e00640 1 --2- 172.21.15.6:0/490839442 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd3c80d04b0 0x7fd3c80cb160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:35.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:35.833+0000 7fd3c5e00640 1 -- 172.21.15.6:0/490839442 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3c815b670 con 0x7fd3c80700c0 2024-03-04T09:34:35.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:35.833+0000 7fd3c6800640 1 --2- 172.21.15.6:0/490839442 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd3c815f450 0x7fd3c815b130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:34:35.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:35.833+0000 7fd3c5e00640 1 --2- 172.21.15.6:0/490839442 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd3c80700c0 0x7fd3c809c3f0 secure :-1 s=READY pgs=1643 cs=0 l=1 rev1=1 crypto rx=0x7fd3b800c430 tx=0x7fd3b800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:34:35.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:35.834+0000 7fd3be000640 1 -- 172.21.15.6:0/490839442 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3b8017070 con 0x7fd3c80700c0 2024-03-04T09:34:35.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:35.834+0000 7fd3be000640 1 -- 172.21.15.6:0/490839442 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3b80052d0 con 0x7fd3c80700c0 2024-03-04T09:34:35.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:35.834+0000 7fd3be000640 1 -- 172.21.15.6:0/490839442 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3b8003830 con 0x7fd3c80700c0 2024-03-04T09:34:35.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:35.834+0000 7fd3c7200640 1 -- 172.21.15.6:0/490839442 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3c809cb40 con 0x7fd3c80700c0 2024-03-04T09:34:35.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:35.834+0000 7fd3c7200640 1 -- 172.21.15.6:0/490839442 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3c809d100 con 0x7fd3c80700c0 2024-03-04T09:34:35.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:35.835+0000 7fd3c7200640 1 -- 172.21.15.6:0/490839442 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3c80d1190 con 0x7fd3c80700c0 2024-03-04T09:34:35.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:35.836+0000 7fd3be000640 1 -- 172.21.15.6:0/490839442 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd3b801a070 con 0x7fd3c80700c0 2024-03-04T09:34:35.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:35.838+0000 7fd3be000640 1 --2- 172.21.15.6:0/490839442 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd3a406b4d0 0x7fd3a406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:35.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:35.838+0000 7fd3be000640 1 -- 172.21.15.6:0/490839442 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fd3b80895c0 con 0x7fd3c80700c0 2024-03-04T09:34:35.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:35.838+0000 7fd3bfe00640 1 --2- 172.21.15.6:0/490839442 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd3a406b4d0 0x7fd3a406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:35.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:35.839+0000 7fd3bfe00640 1 --2- 172.21.15.6:0/490839442 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd3a406b4d0 0x7fd3a406d990 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fd3b0036000 tx=0x7fd3b0008030 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:34:35.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:35.842+0000 7fd3be000640 1 -- 172.21.15.6:0/490839442 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd3c80d1190 con 0x7fd3c80700c0 2024-03-04T09:34:35.840 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-04T09:34:36.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:36.127+0000 7fd3c7200640 1 -- 172.21.15.6:0/490839442 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd3c804f330 con 0x7fd3c80700c0 2024-03-04T09:34:36.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:36.128+0000 7fd3be000640 1 -- 172.21.15.6:0/490839442 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3465 (secure 0 0 0) 0x7fd3b8058400 con 0x7fd3c80700c0 2024-03-04T09:34:36.125 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:34:36.125 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:33:54.652601+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"laggy_since":"2024-03-04T09:33:54.652601+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":9}]} 2024-03-04T09:34:36.125 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 100 2024-03-04T09:34:36.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:36.130+0000 7fd3c7200640 1 -- 172.21.15.6:0/490839442 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd3a406b4d0 msgr2=0x7fd3a406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:36.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:36.130+0000 7fd3c7200640 1 --2- 172.21.15.6:0/490839442 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd3a406b4d0 0x7fd3a406d990 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fd3b0036000 tx=0x7fd3b0008030 comp rx=0 tx=0).stop 2024-03-04T09:34:36.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:36.130+0000 7fd3c7200640 1 -- 172.21.15.6:0/490839442 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd3c80700c0 msgr2=0x7fd3c809c3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:36.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:36.130+0000 7fd3c7200640 1 --2- 172.21.15.6:0/490839442 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd3c80700c0 0x7fd3c809c3f0 secure :-1 s=READY pgs=1643 cs=0 l=1 rev1=1 crypto rx=0x7fd3b800c430 tx=0x7fd3b800c8f0 comp rx=0 tx=0).stop 2024-03-04T09:34:36.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:36.131+0000 7fd3c7200640 1 -- 172.21.15.6:0/490839442 shutdown_connections 2024-03-04T09:34:36.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:36.131+0000 7fd3c7200640 1 --2- 172.21.15.6:0/490839442 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd3a406b4d0 0x7fd3a406d990 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:36.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:36.131+0000 7fd3c7200640 1 --2- 172.21.15.6:0/490839442 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd3c815f450 0x7fd3c815b130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:36.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:36.131+0000 7fd3c7200640 1 --2- 172.21.15.6:0/490839442 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd3c80d04b0 0x7fd3c80cb160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:36.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:36.131+0000 7fd3c7200640 1 --2- 172.21.15.6:0/490839442 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd3c80700c0 0x7fd3c809c3f0 unknown :-1 s=CLOSED pgs=1643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:36.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:36.131+0000 7fd3c7200640 1 -- 172.21.15.6:0/490839442 >> 172.21.15.6:0/490839442 conn(0x7fd3c80d1ae0 msgr2=0x7fd3c80cdf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:34:36.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:36.131+0000 7fd3c7200640 1 -- 172.21.15.6:0/490839442 shutdown_connections 2024-03-04T09:34:36.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:36.131+0000 7fd3c7200640 1 -- 172.21.15.6:0/490839442 wait complete. 2024-03-04T09:34:36.348 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:32:57.159681+0000', 'modified': '2024-03-04T09:33:54.652601+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8214, 'mds_1': 6796}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6796': {'gid': 6796, 'name': 'd', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.146:6837/2458070840', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 2458070840}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 2458070840}]}, 'join_fscid': -1, 'laggy_since': '2024-03-04T09:33:54.652601+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8214': {'gid': 8214, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2446627517', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2446627517}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2446627517}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8214, 'qdb_cluster': [8214]} 2024-03-04T09:34:36.349 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6796:up:stopping 2024-03-04T09:34:37.350 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:34:37.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:37.587+0000 7ff71ec00640 1 Processor -- start 2024-03-04T09:34:37.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:37.587+0000 7ff71ec00640 1 -- start start 2024-03-04T09:34:37.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:37.589+0000 7ff71ec00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff71815e070 0x7ff7180d2960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:37.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:37.589+0000 7ff71ec00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ff7180d0cd0 0x7ff7180cf870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:37.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:37.589+0000 7ff71ec00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7180d18c0 con 0x7ff7180cf410 2024-03-04T09:34:37.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:37.589+0000 7ff71ec00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7180d1ba0 con 0x7ff71815e070 2024-03-04T09:34:37.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:37.589+0000 7ff71ec00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7180d1e80 con 0x7ff7180d0cd0 2024-03-04T09:34:37.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:37.589+0000 7ff71d800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff71815e070 0x7ff7180d2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:37.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:37.589+0000 7ff71d800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff71815e070 0x7ff7180d2960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51938/0 (socket says 172.21.15.6:51938) 2024-03-04T09:34:37.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:37.589+0000 7ff71d800640 1 -- 172.21.15.6:0/2972985446 learned_addr learned my addr 172.21.15.6:0/2972985446 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:34:37.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:37.590+0000 7ff71e200640 1 --2- 172.21.15.6:0/2972985446 >> v2:172.21.15.146:3300/0 conn(0x7ff7180d0cd0 0x7ff7180cf870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:37.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:37.590+0000 7ff71d800640 1 -- 172.21.15.6:0/2972985446 >> v2:172.21.15.146:3300/0 conn(0x7ff7180d0cd0 msgr2=0x7ff7180cf870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:34:37.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:37.590+0000 7ff71d800640 1 --2- 172.21.15.6:0/2972985446 >> v2:172.21.15.146:3300/0 conn(0x7ff7180d0cd0 0x7ff7180cf870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:37.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:37.590+0000 7ff71d800640 1 -- 172.21.15.6:0/2972985446 >> v1:172.21.15.146:6789/0 conn(0x7ff7180cf410 legacy=0x7ff7180d05c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:34:37.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:37.590+0000 7ff71d800640 1 -- 172.21.15.6:0/2972985446 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7180d2160 con 0x7ff71815e070 2024-03-04T09:34:37.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:37.591+0000 7ff71d800640 1 --2- 172.21.15.6:0/2972985446 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff71815e070 0x7ff7180d2960 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 crypto rx=0x7ff71808c1b0 tx=0x7ff70c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1eeeeee1354ed311 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:34:37.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:37.591+0000 7ff717e00640 1 -- 172.21.15.6:0/2972985446 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff70c035020 con 0x7ff71815e070 2024-03-04T09:34:37.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:37.592+0000 7ff717e00640 1 -- 172.21.15.6:0/2972985446 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff70c031870 con 0x7ff71815e070 2024-03-04T09:34:37.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:37.592+0000 7ff717e00640 1 -- 172.21.15.6:0/2972985446 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff70c031c40 con 0x7ff71815e070 2024-03-04T09:34:37.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:37.592+0000 7ff71ec00640 1 -- 172.21.15.6:0/2972985446 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff71815e070 msgr2=0x7ff7180d2960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:37.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:37.592+0000 7ff71ec00640 1 --2- 172.21.15.6:0/2972985446 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff71815e070 0x7ff7180d2960 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 crypto rx=0x7ff71808c1b0 tx=0x7ff70c02b910 comp rx=0 tx=0).stop 2024-03-04T09:34:37.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:37.592+0000 7ff71ec00640 1 -- 172.21.15.6:0/2972985446 shutdown_connections 2024-03-04T09:34:37.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:37.592+0000 7ff71ec00640 1 --2- 172.21.15.6:0/2972985446 >> v2:172.21.15.146:3300/0 conn(0x7ff7180d0cd0 0x7ff7180cf870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:37.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:37.592+0000 7ff71ec00640 1 --2- 172.21.15.6:0/2972985446 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff71815e070 0x7ff7180d2960 unknown :-1 s=CLOSED pgs=1644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:37.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:37.592+0000 7ff71ec00640 1 -- 172.21.15.6:0/2972985446 >> 172.21.15.6:0/2972985446 conn(0x7ff71808d2a0 msgr2=0x7ff7180d3cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:34:37.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:37.592+0000 7ff71ec00640 1 -- 172.21.15.6:0/2972985446 shutdown_connections 2024-03-04T09:34:37.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:37.592+0000 7ff71ec00640 1 -- 172.21.15.6:0/2972985446 wait complete. 2024-03-04T09:34:37.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:37.594+0000 7ff71ec00640 1 Processor -- start 2024-03-04T09:34:37.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:37.594+0000 7ff71ec00640 1 -- start start 2024-03-04T09:34:37.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:37.595+0000 7ff71ec00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff7180cf410 0x7ff7180cc0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:37.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:37.595+0000 7ff71ec00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff7180d0cd0 0x7ff7181f78f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:37.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:37.595+0000 7ff71ec00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff71815e070 0x7ff7181f7e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:37.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:37.595+0000 7ff71ec00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7180d4730 con 0x7ff7180d0cd0 2024-03-04T09:34:37.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:37.595+0000 7ff71ec00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7180d45b0 con 0x7ff7180cf410 2024-03-04T09:34:37.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:37.595+0000 7ff71ec00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7180b3630 con 0x7ff71815e070 2024-03-04T09:34:37.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:37.595+0000 7ff71ce00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff7180d0cd0 0x7ff7181f78f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:37.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:37.595+0000 7ff71ce00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff7180d0cd0 0x7ff7181f78f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51952/0 (socket says 172.21.15.6:51952) 2024-03-04T09:34:37.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:37.595+0000 7ff71ce00640 1 -- 172.21.15.6:0/1013386216 learned_addr learned my addr 172.21.15.6:0/1013386216 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:34:37.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:37.596+0000 7ff71d800640 1 --2- 172.21.15.6:0/1013386216 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff7180cf410 0x7ff7180cc0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:37.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:37.596+0000 7ff71e200640 1 --2- 172.21.15.6:0/1013386216 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff71815e070 0x7ff7181f7e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:37.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:37.596+0000 7ff71ce00640 1 -- 172.21.15.6:0/1013386216 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff71815e070 msgr2=0x7ff7181f7e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:37.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:37.596+0000 7ff71ce00640 1 --2- 172.21.15.6:0/1013386216 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff71815e070 0x7ff7181f7e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:37.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:37.596+0000 7ff71ce00640 1 -- 172.21.15.6:0/1013386216 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff7180cf410 msgr2=0x7ff7180cc0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:37.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:37.596+0000 7ff71ce00640 1 --2- 172.21.15.6:0/1013386216 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff7180cf410 0x7ff7180cc0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:37.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:37.596+0000 7ff71ce00640 1 -- 172.21.15.6:0/1013386216 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7180cc9c0 con 0x7ff7180d0cd0 2024-03-04T09:34:37.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:37.596+0000 7ff71d800640 1 --2- 172.21.15.6:0/1013386216 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff7180cf410 0x7ff7180cc0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:34:37.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:37.596+0000 7ff71ce00640 1 --2- 172.21.15.6:0/1013386216 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff7180d0cd0 0x7ff7181f78f0 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 crypto rx=0x7ff70400c440 tx=0x7ff70400c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:34:37.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:37.596+0000 7ff715600640 1 -- 172.21.15.6:0/1013386216 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff704017070 con 0x7ff7180d0cd0 2024-03-04T09:34:37.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:37.596+0000 7ff715600640 1 -- 172.21.15.6:0/1013386216 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7040035f0 con 0x7ff7180d0cd0 2024-03-04T09:34:37.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:37.596+0000 7ff715600640 1 -- 172.21.15.6:0/1013386216 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7040039e0 con 0x7ff7180d0cd0 2024-03-04T09:34:37.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:37.597+0000 7ff71ec00640 1 -- 172.21.15.6:0/1013386216 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7180ccca0 con 0x7ff7180d0cd0 2024-03-04T09:34:37.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:37.597+0000 7ff71ec00640 1 -- 172.21.15.6:0/1013386216 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7181e6c40 con 0x7ff7180d0cd0 2024-03-04T09:34:37.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:37.598+0000 7ff71ec00640 1 -- 172.21.15.6:0/1013386216 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7181e6a30 con 0x7ff7180d0cd0 2024-03-04T09:34:37.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:37.599+0000 7ff715600640 1 -- 172.21.15.6:0/1013386216 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff704003b80 con 0x7ff7180d0cd0 2024-03-04T09:34:37.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:37.600+0000 7ff715600640 1 --2- 172.21.15.6:0/1013386216 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff6f806b4d0 0x7ff6f806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:37.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:37.600+0000 7ff715600640 1 -- 172.21.15.6:0/1013386216 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7ff7040895a0 con 0x7ff7180d0cd0 2024-03-04T09:34:37.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:37.600+0000 7ff71d800640 1 --2- 172.21.15.6:0/1013386216 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff6f806b4d0 0x7ff6f806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:37.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:37.602+0000 7ff71d800640 1 --2- 172.21.15.6:0/1013386216 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff6f806b4d0 0x7ff6f806d990 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7ff70c036000 tx=0x7ff70c008030 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:34:37.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:37.604+0000 7ff715600640 1 -- 172.21.15.6:0/1013386216 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff704058340 con 0x7ff7180d0cd0 2024-03-04T09:34:37.843 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-04T09:34:37.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:37.846+0000 7ff71ec00640 1 -- 172.21.15.6:0/1013386216 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff7181e6e10 con 0x7ff7180d0cd0 2024-03-04T09:34:37.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:37.847+0000 7ff715600640 1 -- 172.21.15.6:0/1013386216 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3465 (secure 0 0 0) 0x7ff7040010c0 con 0x7ff7180d0cd0 2024-03-04T09:34:37.844 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:34:37.844 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:33:54.652601+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"laggy_since":"2024-03-04T09:33:54.652601+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":9}]} 2024-03-04T09:34:37.845 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 100 2024-03-04T09:34:37.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:37.850+0000 7ff71ec00640 1 -- 172.21.15.6:0/1013386216 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff6f806b4d0 msgr2=0x7ff6f806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:37.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:37.850+0000 7ff71ec00640 1 --2- 172.21.15.6:0/1013386216 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff6f806b4d0 0x7ff6f806d990 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7ff70c036000 tx=0x7ff70c008030 comp rx=0 tx=0).stop 2024-03-04T09:34:37.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:37.850+0000 7ff71ec00640 1 -- 172.21.15.6:0/1013386216 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff7180d0cd0 msgr2=0x7ff7181f78f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:37.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:37.850+0000 7ff71ec00640 1 --2- 172.21.15.6:0/1013386216 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff7180d0cd0 0x7ff7181f78f0 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 crypto rx=0x7ff70400c440 tx=0x7ff70400c900 comp rx=0 tx=0).stop 2024-03-04T09:34:37.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:37.850+0000 7ff71ec00640 1 -- 172.21.15.6:0/1013386216 shutdown_connections 2024-03-04T09:34:37.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:37.850+0000 7ff71ec00640 1 --2- 172.21.15.6:0/1013386216 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff6f806b4d0 0x7ff6f806d990 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:37.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:37.850+0000 7ff71ec00640 1 --2- 172.21.15.6:0/1013386216 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff71815e070 0x7ff7181f7e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:37.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:37.850+0000 7ff71ec00640 1 --2- 172.21.15.6:0/1013386216 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff7180d0cd0 0x7ff7181f78f0 unknown :-1 s=CLOSED pgs=1645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:37.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:37.850+0000 7ff71ec00640 1 --2- 172.21.15.6:0/1013386216 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff7180cf410 0x7ff7180cc0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:37.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:37.850+0000 7ff71ec00640 1 -- 172.21.15.6:0/1013386216 >> 172.21.15.6:0/1013386216 conn(0x7ff71808d2a0 msgr2=0x7ff7180d2860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:34:37.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:37.850+0000 7ff71ec00640 1 -- 172.21.15.6:0/1013386216 shutdown_connections 2024-03-04T09:34:37.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:37.850+0000 7ff71ec00640 1 -- 172.21.15.6:0/1013386216 wait complete. 2024-03-04T09:34:38.068 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:32:57.159681+0000', 'modified': '2024-03-04T09:33:54.652601+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8214, 'mds_1': 6796}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6796': {'gid': 6796, 'name': 'd', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.146:6837/2458070840', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 2458070840}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 2458070840}]}, 'join_fscid': -1, 'laggy_since': '2024-03-04T09:33:54.652601+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8214': {'gid': 8214, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2446627517', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2446627517}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2446627517}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8214, 'qdb_cluster': [8214]} 2024-03-04T09:34:38.068 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6796:up:stopping 2024-03-04T09:34:39.069 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:34:39.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:39.288+0000 7f1fa2400640 1 Processor -- start 2024-03-04T09:34:39.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:39.288+0000 7f1fa2400640 1 -- start start 2024-03-04T09:34:39.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:39.290+0000 7f1fa2400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1f9c0cdaa0 0x7f1f9c0cfc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:39.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:39.290+0000 7f1fa2400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f1f9c06e190 0x7f1f9c0cde80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:39.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:39.290+0000 7f1fa2400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f1f9c15c010 0x7f1f9c15c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:39.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:39.290+0000 7f1fa2400640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f9c06e5f0 con 0x7f1f9c06e190 2024-03-04T09:34:39.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:39.290+0000 7f1fa2400640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f9c0ce3c0 con 0x7f1f9c15c010 2024-03-04T09:34:39.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:39.290+0000 7f1fa2400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f9c06e860 con 0x7f1f9c0cdaa0 2024-03-04T09:34:39.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:39.290+0000 7f1f9aa00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f1f9c06e190 0x7f1f9c0cde80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:39.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:39.290+0000 7f1f9aa00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f1f9c06e190 0x7f1f9c0cde80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:56948/0 (socket says 172.21.15.6:56948) 2024-03-04T09:34:39.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:39.290+0000 7f1f9aa00640 1 -- 172.21.15.6:0/641732779 learned_addr learned my addr 172.21.15.6:0/641732779 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:34:39.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:39.291+0000 7f1f9be00640 1 --2- 172.21.15.6:0/641732779 >> v2:172.21.15.146:3300/0 conn(0x7f1f9c15c010 0x7f1f9c15c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:39.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:39.291+0000 7f1f9b400640 1 --2- 172.21.15.6:0/641732779 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1f9c0cdaa0 0x7f1f9c0cfc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:39.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:39.291+0000 7f1f9aa00640 1 -- 172.21.15.6:0/641732779 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1f9c0cdaa0 msgr2=0x7f1f9c0cfc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:34:39.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:39.291+0000 7f1f9aa00640 1 --2- 172.21.15.6:0/641732779 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1f9c0cdaa0 0x7f1f9c0cfc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:39.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:39.291+0000 7f1f9aa00640 1 -- 172.21.15.6:0/641732779 >> v2:172.21.15.146:3300/0 conn(0x7f1f9c15c010 msgr2=0x7f1f9c15c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:34:39.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:39.291+0000 7f1f9aa00640 1 --2- 172.21.15.6:0/641732779 >> v2:172.21.15.146:3300/0 conn(0x7f1f9c15c010 0x7f1f9c15c4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:39.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:39.291+0000 7f1f9aa00640 1 -- 172.21.15.6:0/641732779 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f9c06b150 con 0x7f1f9c06e190 2024-03-04T09:34:39.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:39.292+0000 7f1f9aa00640 1 --2- 172.21.15.6:0/641732779 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1f9c06e190 0x7f1f9c0cde80 secure :-1 s=READY pgs=1646 cs=0 l=1 rev1=1 crypto rx=0x7f1f9c06d540 tx=0x7f1f8402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f69fe4e71c768b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:34:39.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:39.293+0000 7f1f9a000640 1 -- 172.21.15.6:0/641732779 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f84035020 con 0x7f1f9c06e190 2024-03-04T09:34:39.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:39.293+0000 7f1f9a000640 1 -- 172.21.15.6:0/641732779 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f84031870 con 0x7f1f9c06e190 2024-03-04T09:34:39.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:39.293+0000 7f1f9a000640 1 -- 172.21.15.6:0/641732779 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f84031c40 con 0x7f1f9c06e190 2024-03-04T09:34:39.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:39.293+0000 7f1fa2400640 1 -- 172.21.15.6:0/641732779 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1f9c06e190 msgr2=0x7f1f9c0cde80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:39.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:39.293+0000 7f1fa2400640 1 --2- 172.21.15.6:0/641732779 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1f9c06e190 0x7f1f9c0cde80 secure :-1 s=READY pgs=1646 cs=0 l=1 rev1=1 crypto rx=0x7f1f9c06d540 tx=0x7f1f8402b910 comp rx=0 tx=0).stop 2024-03-04T09:34:39.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:39.293+0000 7f1fa2400640 1 -- 172.21.15.6:0/641732779 shutdown_connections 2024-03-04T09:34:39.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:39.293+0000 7f1fa2400640 1 --2- 172.21.15.6:0/641732779 >> v2:172.21.15.146:3300/0 conn(0x7f1f9c15c010 0x7f1f9c15c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:39.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:39.293+0000 7f1fa2400640 1 --2- 172.21.15.6:0/641732779 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1f9c06e190 0x7f1f9c0cde80 unknown :-1 s=CLOSED pgs=1646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:39.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:39.293+0000 7f1fa2400640 1 --2- 172.21.15.6:0/641732779 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1f9c0cdaa0 0x7f1f9c0cfc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:39.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:39.293+0000 7f1fa2400640 1 -- 172.21.15.6:0/641732779 >> 172.21.15.6:0/641732779 conn(0x7f1f9c0b2640 msgr2=0x7f1f9c0b2a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:34:39.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:39.294+0000 7f1fa2400640 1 -- 172.21.15.6:0/641732779 shutdown_connections 2024-03-04T09:34:39.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:39.294+0000 7f1fa2400640 1 -- 172.21.15.6:0/641732779 wait complete. 2024-03-04T09:34:39.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:39.295+0000 7f1fa2400640 1 Processor -- start 2024-03-04T09:34:39.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:39.296+0000 7f1fa2400640 1 -- start start 2024-03-04T09:34:39.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:39.297+0000 7f1fa2400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1f9c06e190 0x7f1f9c1f76d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:39.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:39.297+0000 7f1fa2400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1f9c0cdaa0 0x7f1f9c0ad9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:39.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:39.297+0000 7f1fa2400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1f9c15c010 0x7f1f9c0adf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:39.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:39.297+0000 7f1fa2400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f9c0afc50 con 0x7f1f9c0cdaa0 2024-03-04T09:34:39.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:39.297+0000 7f1fa2400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f9c06b710 con 0x7f1f9c15c010 2024-03-04T09:34:39.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:39.297+0000 7f1fa2400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f9c06bc30 con 0x7f1f9c06e190 2024-03-04T09:34:39.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:39.297+0000 7f1f9aa00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1f9c0cdaa0 0x7f1f9c0ad9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:39.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:39.297+0000 7f1f9aa00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1f9c0cdaa0 0x7f1f9c0ad9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:56960/0 (socket says 172.21.15.6:56960) 2024-03-04T09:34:39.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:39.297+0000 7f1f9aa00640 1 -- 172.21.15.6:0/3663128474 learned_addr learned my addr 172.21.15.6:0/3663128474 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:34:39.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:39.297+0000 7f1f9be00640 1 --2- 172.21.15.6:0/3663128474 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1f9c15c010 0x7f1f9c0adf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:39.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:39.297+0000 7f1f9b400640 1 --2- 172.21.15.6:0/3663128474 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1f9c06e190 0x7f1f9c1f76d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:39.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:39.298+0000 7f1f9aa00640 1 -- 172.21.15.6:0/3663128474 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1f9c06e190 msgr2=0x7f1f9c1f76d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:39.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:39.298+0000 7f1f9aa00640 1 --2- 172.21.15.6:0/3663128474 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1f9c06e190 0x7f1f9c1f76d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:39.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:39.298+0000 7f1f9aa00640 1 -- 172.21.15.6:0/3663128474 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1f9c15c010 msgr2=0x7f1f9c0adf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:39.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:39.298+0000 7f1f9aa00640 1 --2- 172.21.15.6:0/3663128474 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1f9c15c010 0x7f1f9c0adf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:39.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:39.298+0000 7f1f9aa00640 1 -- 172.21.15.6:0/3663128474 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f9c1f7e40 con 0x7f1f9c0cdaa0 2024-03-04T09:34:39.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:39.298+0000 7f1f9b400640 1 --2- 172.21.15.6:0/3663128474 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1f9c06e190 0x7f1f9c1f76d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T09:34:39.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:39.298+0000 7f1f9be00640 1 --2- 172.21.15.6:0/3663128474 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1f9c15c010 0x7f1f9c0adf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T09:34:39.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:39.298+0000 7f1f9aa00640 1 --2- 172.21.15.6:0/3663128474 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1f9c0cdaa0 0x7f1f9c0ad9f0 secure :-1 s=READY pgs=1647 cs=0 l=1 rev1=1 crypto rx=0x7f1f8402b950 tx=0x7f1f840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:34:39.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:39.299+0000 7f1f93400640 1 -- 172.21.15.6:0/3663128474 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f8403ad60 con 0x7f1f9c0cdaa0 2024-03-04T09:34:39.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:39.299+0000 7f1f93400640 1 -- 172.21.15.6:0/3663128474 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f840443f0 con 0x7f1f9c0cdaa0 2024-03-04T09:34:39.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:39.299+0000 7f1f93400640 1 -- 172.21.15.6:0/3663128474 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f840447e0 con 0x7f1f9c0cdaa0 2024-03-04T09:34:39.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:39.299+0000 7f1fa2400640 1 -- 172.21.15.6:0/3663128474 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f9c1f8120 con 0x7f1f9c0cdaa0 2024-03-04T09:34:39.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:39.299+0000 7f1fa2400640 1 -- 172.21.15.6:0/3663128474 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f9c15f0c0 con 0x7f1f9c0cdaa0 2024-03-04T09:34:39.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:39.300+0000 7f1fa2400640 1 -- 172.21.15.6:0/3663128474 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f9c04f3a0 con 0x7f1f9c0cdaa0 2024-03-04T09:34:39.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:39.302+0000 7f1f93400640 1 -- 172.21.15.6:0/3663128474 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1f84038020 con 0x7f1f9c0cdaa0 2024-03-04T09:34:39.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:39.303+0000 7f1f93400640 1 --2- 172.21.15.6:0/3663128474 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f1f7806b4d0 0x7f1f7806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:39.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:39.303+0000 7f1f93400640 1 -- 172.21.15.6:0/3663128474 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f1f840ac370 con 0x7f1f9c0cdaa0 2024-03-04T09:34:39.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:39.304+0000 7f1f9b400640 1 --2- 172.21.15.6:0/3663128474 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f1f7806b4d0 0x7f1f7806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:39.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:39.305+0000 7f1f9b400640 1 --2- 172.21.15.6:0/3663128474 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f1f7806b4d0 0x7f1f7806d990 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f1f8c006fd0 tx=0x7f1f8c008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:34:39.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:39.308+0000 7f1f93400640 1 -- 172.21.15.6:0/3663128474 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1f8403f070 con 0x7f1f9c0cdaa0 2024-03-04T09:34:39.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:39.545+0000 7f1fa2400640 1 -- 172.21.15.6:0/3663128474 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1f9c067370 con 0x7f1f9c0cdaa0 2024-03-04T09:34:39.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:39.547+0000 7f1f93400640 1 -- 172.21.15.6:0/3663128474 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3465 (secure 0 0 0) 0x7f1f8407b240 con 0x7f1f9c0cdaa0 2024-03-04T09:34:39.544 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:34:39.544 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:33:54.652601+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"laggy_since":"2024-03-04T09:33:54.652601+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":9}]} 2024-03-04T09:34:39.544 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 100 2024-03-04T09:34:39.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:39.549+0000 7f1fa2400640 1 -- 172.21.15.6:0/3663128474 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f1f7806b4d0 msgr2=0x7f1f7806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:39.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:39.549+0000 7f1fa2400640 1 --2- 172.21.15.6:0/3663128474 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f1f7806b4d0 0x7f1f7806d990 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f1f8c006fd0 tx=0x7f1f8c008040 comp rx=0 tx=0).stop 2024-03-04T09:34:39.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:39.549+0000 7f1fa2400640 1 -- 172.21.15.6:0/3663128474 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1f9c0cdaa0 msgr2=0x7f1f9c0ad9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:39.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:39.549+0000 7f1fa2400640 1 --2- 172.21.15.6:0/3663128474 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1f9c0cdaa0 0x7f1f9c0ad9f0 secure :-1 s=READY pgs=1647 cs=0 l=1 rev1=1 crypto rx=0x7f1f8402b950 tx=0x7f1f840046c0 comp rx=0 tx=0).stop 2024-03-04T09:34:39.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:39.549+0000 7f1fa2400640 1 -- 172.21.15.6:0/3663128474 shutdown_connections 2024-03-04T09:34:39.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:39.549+0000 7f1fa2400640 1 --2- 172.21.15.6:0/3663128474 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1f9c15c010 0x7f1f9c0adf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:39.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:39.549+0000 7f1fa2400640 1 --2- 172.21.15.6:0/3663128474 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1f9c0cdaa0 0x7f1f9c0ad9f0 unknown :-1 s=CLOSED pgs=1647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:39.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:39.549+0000 7f1fa2400640 1 --2- 172.21.15.6:0/3663128474 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f1f7806b4d0 0x7f1f7806d990 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:39.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:39.549+0000 7f1fa2400640 1 --2- 172.21.15.6:0/3663128474 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1f9c06e190 0x7f1f9c1f76d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:39.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:39.549+0000 7f1fa2400640 1 -- 172.21.15.6:0/3663128474 >> 172.21.15.6:0/3663128474 conn(0x7f1f9c0b2640 msgr2=0x7f1f9c0b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:34:39.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:39.550+0000 7f1fa2400640 1 -- 172.21.15.6:0/3663128474 shutdown_connections 2024-03-04T09:34:39.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:39.550+0000 7f1fa2400640 1 -- 172.21.15.6:0/3663128474 wait complete. 2024-03-04T09:34:39.769 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:32:57.159681+0000', 'modified': '2024-03-04T09:33:54.652601+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8214, 'mds_1': 6796}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6796': {'gid': 6796, 'name': 'd', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.146:6837/2458070840', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 2458070840}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 2458070840}]}, 'join_fscid': -1, 'laggy_since': '2024-03-04T09:33:54.652601+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8214': {'gid': 8214, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2446627517', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2446627517}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2446627517}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8214, 'qdb_cluster': [8214]} 2024-03-04T09:34:39.769 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6796:up:stopping 2024-03-04T09:34:39.845 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-04T09:34:40.770 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:34:40.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:41.001+0000 7fd4b1000640 1 Processor -- start 2024-03-04T09:34:40.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:41.002+0000 7fd4b1000640 1 -- start start 2024-03-04T09:34:41.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:41.003+0000 7fd4b1000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd4ac15e0b0 0x7fd4ac0cf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:41.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:41.003+0000 7fd4b1000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd4ac0d4a10 0x7fd4ac0d0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:41.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:41.003+0000 7fd4b1000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd4ac0d0c80 0x7fd4ac0d1900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:41.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:41.003+0000 7fd4b1000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4ac0b09e0 con 0x7fd4ac0d0c80 2024-03-04T09:34:41.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:41.003+0000 7fd4b1000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4ac0b3710 con 0x7fd4ac15e0b0 2024-03-04T09:34:41.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:41.003+0000 7fd4b1000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4ac0b39f0 con 0x7fd4ac0d4a10 2024-03-04T09:34:41.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:41.003+0000 7fd4abe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd4ac0d0c80 0x7fd4ac0d1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:41.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:41.003+0000 7fd4abe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd4ac0d0c80 0x7fd4ac0d1900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:56966/0 (socket says 172.21.15.6:56966) 2024-03-04T09:34:41.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:41.003+0000 7fd4abe00640 1 -- 172.21.15.6:0/2558352740 learned_addr learned my addr 172.21.15.6:0/2558352740 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:34:41.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:41.004+0000 7fd4aaa00640 1 --2- 172.21.15.6:0/2558352740 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd4ac0d4a10 0x7fd4ac0d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:41.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:41.004+0000 7fd4ab400640 1 --2- 172.21.15.6:0/2558352740 >> v2:172.21.15.146:3300/0 conn(0x7fd4ac15e0b0 0x7fd4ac0cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:41.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:41.004+0000 7fd4abe00640 1 -- 172.21.15.6:0/2558352740 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd4ac0d4a10 msgr2=0x7fd4ac0d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:34:41.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:41.004+0000 7fd4abe00640 1 --2- 172.21.15.6:0/2558352740 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd4ac0d4a10 0x7fd4ac0d0740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:41.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:41.004+0000 7fd4abe00640 1 -- 172.21.15.6:0/2558352740 >> v2:172.21.15.146:3300/0 conn(0x7fd4ac15e0b0 msgr2=0x7fd4ac0cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:34:41.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:41.004+0000 7fd4abe00640 1 --2- 172.21.15.6:0/2558352740 >> v2:172.21.15.146:3300/0 conn(0x7fd4ac15e0b0 0x7fd4ac0cf310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:41.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:41.004+0000 7fd4abe00640 1 -- 172.21.15.6:0/2558352740 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4ac0b3cd0 con 0x7fd4ac0d0c80 2024-03-04T09:34:41.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:41.005+0000 7fd4abe00640 1 --2- 172.21.15.6:0/2558352740 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd4ac0d0c80 0x7fd4ac0d1900 secure :-1 s=READY pgs=1648 cs=0 l=1 rev1=1 crypto rx=0x7fd4ac0b68c0 tx=0x7fd4a402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5c2de6282241bba server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:34:41.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:41.006+0000 7fd4aa000640 1 -- 172.21.15.6:0/2558352740 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4a402ca70 con 0x7fd4ac0d0c80 2024-03-04T09:34:41.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:41.006+0000 7fd4aa000640 1 -- 172.21.15.6:0/2558352740 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4a402cc10 con 0x7fd4ac0d0c80 2024-03-04T09:34:41.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:41.006+0000 7fd4aa000640 1 -- 172.21.15.6:0/2558352740 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4a40343f0 con 0x7fd4ac0d0c80 2024-03-04T09:34:41.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:41.007+0000 7fd4b1000640 1 -- 172.21.15.6:0/2558352740 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd4ac0d0c80 msgr2=0x7fd4ac0d1900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:41.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:41.007+0000 7fd4b1000640 1 --2- 172.21.15.6:0/2558352740 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd4ac0d0c80 0x7fd4ac0d1900 secure :-1 s=READY pgs=1648 cs=0 l=1 rev1=1 crypto rx=0x7fd4ac0b68c0 tx=0x7fd4a402bed0 comp rx=0 tx=0).stop 2024-03-04T09:34:41.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:41.007+0000 7fd4b1000640 1 -- 172.21.15.6:0/2558352740 shutdown_connections 2024-03-04T09:34:41.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:41.007+0000 7fd4b1000640 1 --2- 172.21.15.6:0/2558352740 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd4ac0d0c80 0x7fd4ac0d1900 unknown :-1 s=CLOSED pgs=1648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:41.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:41.007+0000 7fd4b1000640 1 --2- 172.21.15.6:0/2558352740 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd4ac0d4a10 0x7fd4ac0d0740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:41.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:41.007+0000 7fd4b1000640 1 --2- 172.21.15.6:0/2558352740 >> v2:172.21.15.146:3300/0 conn(0x7fd4ac15e0b0 0x7fd4ac0cf310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:41.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:41.007+0000 7fd4b1000640 1 -- 172.21.15.6:0/2558352740 >> 172.21.15.6:0/2558352740 conn(0x7fd4ac0d3d30 msgr2=0x7fd4ac0d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:34:41.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:41.007+0000 7fd4b1000640 1 -- 172.21.15.6:0/2558352740 shutdown_connections 2024-03-04T09:34:41.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:41.007+0000 7fd4b1000640 1 -- 172.21.15.6:0/2558352740 wait complete. 2024-03-04T09:34:41.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:41.009+0000 7fd4b1000640 1 Processor -- start 2024-03-04T09:34:41.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:41.009+0000 7fd4b1000640 1 -- start start 2024-03-04T09:34:41.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:41.010+0000 7fd4b1000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd4ac0d0c80 0x7fd4ac09f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:41.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:41.010+0000 7fd4b1000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd4ac0d4a10 0x7fd4ac15d110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:41.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:41.010+0000 7fd4b1000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd4ac15e0b0 0x7fd4ac0b8b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:41.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:41.010+0000 7fd4b1000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4ac0cfc60 con 0x7fd4ac0d4a10 2024-03-04T09:34:41.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:41.010+0000 7fd4b1000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4ac15e490 con 0x7fd4ac0d0c80 2024-03-04T09:34:41.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:41.010+0000 7fd4b1000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4ac0cf850 con 0x7fd4ac15e0b0 2024-03-04T09:34:41.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:41.010+0000 7fd4aaa00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd4ac0d4a10 0x7fd4ac15d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:41.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:41.010+0000 7fd4aaa00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd4ac0d4a10 0x7fd4ac15d110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:56970/0 (socket says 172.21.15.6:56970) 2024-03-04T09:34:41.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:41.010+0000 7fd4aaa00640 1 -- 172.21.15.6:0/4050761842 learned_addr learned my addr 172.21.15.6:0/4050761842 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:34:41.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:41.010+0000 7fd4ab400640 1 --2- 172.21.15.6:0/4050761842 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd4ac0d0c80 0x7fd4ac09f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:41.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:41.010+0000 7fd4abe00640 1 --2- 172.21.15.6:0/4050761842 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd4ac15e0b0 0x7fd4ac0b8b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:41.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:41.011+0000 7fd4aaa00640 1 -- 172.21.15.6:0/4050761842 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd4ac15e0b0 msgr2=0x7fd4ac0b8b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:41.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:41.011+0000 7fd4aaa00640 1 --2- 172.21.15.6:0/4050761842 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd4ac15e0b0 0x7fd4ac0b8b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:41.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:41.011+0000 7fd4aaa00640 1 -- 172.21.15.6:0/4050761842 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd4ac0d0c80 msgr2=0x7fd4ac09f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:41.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:41.011+0000 7fd4aaa00640 1 --2- 172.21.15.6:0/4050761842 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd4ac0d0c80 0x7fd4ac09f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:41.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:41.011+0000 7fd4aaa00640 1 -- 172.21.15.6:0/4050761842 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4ac09f570 con 0x7fd4ac0d4a10 2024-03-04T09:34:41.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:41.011+0000 7fd4aaa00640 1 --2- 172.21.15.6:0/4050761842 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd4ac0d4a10 0x7fd4ac15d110 secure :-1 s=READY pgs=1649 cs=0 l=1 rev1=1 crypto rx=0x7fd49800d7d0 tx=0x7fd49800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:34:41.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:41.011+0000 7fd4a3400640 1 -- 172.21.15.6:0/4050761842 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd49800b6c0 con 0x7fd4ac0d4a10 2024-03-04T09:34:41.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:41.011+0000 7fd4a3400640 1 -- 172.21.15.6:0/4050761842 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd498019070 con 0x7fd4ac0d4a10 2024-03-04T09:34:41.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:41.011+0000 7fd4a3400640 1 -- 172.21.15.6:0/4050761842 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd49800ba80 con 0x7fd4ac0d4a10 2024-03-04T09:34:41.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:41.011+0000 7fd4b1000640 1 -- 172.21.15.6:0/4050761842 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4ac0cafa0 con 0x7fd4ac0d4a10 2024-03-04T09:34:41.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:41.012+0000 7fd4b1000640 1 -- 172.21.15.6:0/4050761842 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4ac0cb560 con 0x7fd4ac0d4a10 2024-03-04T09:34:41.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:41.013+0000 7fd4b1000640 1 -- 172.21.15.6:0/4050761842 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4ac04f330 con 0x7fd4ac0d4a10 2024-03-04T09:34:41.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:41.014+0000 7fd4a3400640 1 -- 172.21.15.6:0/4050761842 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd49800bc20 con 0x7fd4ac0d4a10 2024-03-04T09:34:41.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:41.015+0000 7fd4a3400640 1 --2- 172.21.15.6:0/4050761842 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd48c06b4d0 0x7fd48c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:41.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:41.015+0000 7fd4a3400640 1 -- 172.21.15.6:0/4050761842 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fd49808a720 con 0x7fd4ac0d4a10 2024-03-04T09:34:41.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:41.015+0000 7fd4ab400640 1 --2- 172.21.15.6:0/4050761842 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd48c06b4d0 0x7fd48c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:41.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:41.016+0000 7fd4ab400640 1 --2- 172.21.15.6:0/4050761842 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd48c06b4d0 0x7fd48c06d990 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7fd49c006fd0 tx=0x7fd49c008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:34:41.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:41.024+0000 7fd4a3400640 1 -- 172.21.15.6:0/4050761842 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd4980594c0 con 0x7fd4ac0d4a10 2024-03-04T09:34:41.128 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:34:41.136 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:34:41.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:41.268+0000 7fd4b1000640 1 -- 172.21.15.6:0/4050761842 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd4ac06d570 con 0x7fd4ac0d4a10 2024-03-04T09:34:41.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:41.269+0000 7fd4a3400640 1 -- 172.21.15.6:0/4050761842 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3465 (secure 0 0 0) 0x7fd498058dc0 con 0x7fd4ac0d4a10 2024-03-04T09:34:41.267 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:34:41.267 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:33:54.652601+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"laggy_since":"2024-03-04T09:33:54.652601+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":9}]} 2024-03-04T09:34:41.267 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 100 2024-03-04T09:34:41.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:41.272+0000 7fd4b1000640 1 -- 172.21.15.6:0/4050761842 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd48c06b4d0 msgr2=0x7fd48c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:41.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:41.272+0000 7fd4b1000640 1 --2- 172.21.15.6:0/4050761842 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd48c06b4d0 0x7fd48c06d990 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7fd49c006fd0 tx=0x7fd49c008040 comp rx=0 tx=0).stop 2024-03-04T09:34:41.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:41.272+0000 7fd4b1000640 1 -- 172.21.15.6:0/4050761842 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd4ac0d4a10 msgr2=0x7fd4ac15d110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:41.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:41.272+0000 7fd4b1000640 1 --2- 172.21.15.6:0/4050761842 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd4ac0d4a10 0x7fd4ac15d110 secure :-1 s=READY pgs=1649 cs=0 l=1 rev1=1 crypto rx=0x7fd49800d7d0 tx=0x7fd49800dc90 comp rx=0 tx=0).stop 2024-03-04T09:34:41.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:41.272+0000 7fd4b1000640 1 -- 172.21.15.6:0/4050761842 shutdown_connections 2024-03-04T09:34:41.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:41.272+0000 7fd4b1000640 1 --2- 172.21.15.6:0/4050761842 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd48c06b4d0 0x7fd48c06d990 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:41.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:41.272+0000 7fd4b1000640 1 --2- 172.21.15.6:0/4050761842 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd4ac15e0b0 0x7fd4ac0b8b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:41.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:41.272+0000 7fd4b1000640 1 --2- 172.21.15.6:0/4050761842 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd4ac0d4a10 0x7fd4ac15d110 unknown :-1 s=CLOSED pgs=1649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:41.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:41.272+0000 7fd4b1000640 1 --2- 172.21.15.6:0/4050761842 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd4ac0d0c80 0x7fd4ac09f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:41.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:41.272+0000 7fd4b1000640 1 -- 172.21.15.6:0/4050761842 >> 172.21.15.6:0/4050761842 conn(0x7fd4ac0d3d30 msgr2=0x7fd4ac160620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:34:41.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:41.272+0000 7fd4b1000640 1 -- 172.21.15.6:0/4050761842 shutdown_connections 2024-03-04T09:34:41.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:41.272+0000 7fd4b1000640 1 -- 172.21.15.6:0/4050761842 wait complete. 2024-03-04T09:34:41.490 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:32:57.159681+0000', 'modified': '2024-03-04T09:33:54.652601+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8214, 'mds_1': 6796}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6796': {'gid': 6796, 'name': 'd', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.146:6837/2458070840', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 2458070840}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 2458070840}]}, 'join_fscid': -1, 'laggy_since': '2024-03-04T09:33:54.652601+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8214': {'gid': 8214, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2446627517', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2446627517}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2446627517}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8214, 'qdb_cluster': [8214]} 2024-03-04T09:34:41.490 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6796:up:stopping 2024-03-04T09:34:41.847 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-04T09:34:42.491 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:34:42.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:42.720+0000 7f2afa400640 1 Processor -- start 2024-03-04T09:34:42.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:42.720+0000 7f2afa400640 1 -- start start 2024-03-04T09:34:42.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:42.722+0000 7f2afa400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f2af40d0e50 0x7f2af4168c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:42.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:42.722+0000 7f2afa400640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2af40b3710 con 0x7f2af40d4a10 2024-03-04T09:34:42.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:42.722+0000 7f2afa400640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2af40b39f0 con 0x7f2af415e0b0 2024-03-04T09:34:42.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:42.722+0000 7f2afa400640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2af40b09e0 con 0x7f2af40d0e50 2024-03-04T09:34:42.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:42.722+0000 7f2af9a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f2af40d0e50 0x7f2af4168c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:42.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:42.723+0000 7f2af9a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f2af40d0e50 0x7f2af4168c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:56972/0 (socket says 172.21.15.6:56972) 2024-03-04T09:34:42.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:42.723+0000 7f2af9a00640 1 -- 172.21.15.6:0/110194914 learned_addr learned my addr 172.21.15.6:0/110194914 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:34:42.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:42.723+0000 7f2af9a00640 1 -- 172.21.15.6:0/110194914 >> v1:172.21.15.146:6789/0 conn(0x7f2af415e0b0 legacy=0x7f2af40cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:42.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:42.723+0000 7f2af9a00640 1 -- 172.21.15.6:0/110194914 >> v1:172.21.15.6:6789/0 conn(0x7f2af40d4a10 legacy=0x7f2af40d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:42.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:42.723+0000 7f2af9a00640 1 -- 172.21.15.6:0/110194914 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2af40b3cd0 con 0x7f2af40d0e50 2024-03-04T09:34:42.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:42.723+0000 7f2af3400640 1 -- 172.21.15.6:0/110194914 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 741576644 0 0) 0x7f2af40b3710 con 0x7f2af40d4a10 2024-03-04T09:34:42.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:42.724+0000 7f2af9a00640 1 --2- 172.21.15.6:0/110194914 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2af40d0e50 0x7f2af4168c50 secure :-1 s=READY pgs=1651 cs=0 l=1 rev1=1 crypto rx=0x7f2aec009e50 tx=0x7f2aec02c010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f08a4ae0cc4e90e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:34:42.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:42.725+0000 7f2af3400640 1 -- 172.21.15.6:0/110194914 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2aec02cbb0 con 0x7f2af40d0e50 2024-03-04T09:34:42.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:42.725+0000 7f2af3400640 1 -- 172.21.15.6:0/110194914 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2aec02cd50 con 0x7f2af40d0e50 2024-03-04T09:34:42.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:42.725+0000 7f2af3400640 1 -- 172.21.15.6:0/110194914 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2aec034610 con 0x7f2af40d0e50 2024-03-04T09:34:42.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:42.725+0000 7f2afa400640 1 -- 172.21.15.6:0/110194914 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2af40d0e50 msgr2=0x7f2af4168c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:42.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:42.725+0000 7f2afa400640 1 --2- 172.21.15.6:0/110194914 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2af40d0e50 0x7f2af4168c50 secure :-1 s=READY pgs=1651 cs=0 l=1 rev1=1 crypto rx=0x7f2aec009e50 tx=0x7f2aec02c010 comp rx=0 tx=0).stop 2024-03-04T09:34:42.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:42.725+0000 7f2afa400640 1 -- 172.21.15.6:0/110194914 shutdown_connections 2024-03-04T09:34:42.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:42.725+0000 7f2afa400640 1 --2- 172.21.15.6:0/110194914 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2af40d0e50 0x7f2af4168c50 unknown :-1 s=CLOSED pgs=1651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:42.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:42.725+0000 7f2afa400640 1 -- 172.21.15.6:0/110194914 >> 172.21.15.6:0/110194914 conn(0x7f2af40d3d30 msgr2=0x7f2af40d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:34:42.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:42.725+0000 7f2afa400640 1 -- 172.21.15.6:0/110194914 shutdown_connections 2024-03-04T09:34:42.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:42.725+0000 7f2afa400640 1 -- 172.21.15.6:0/110194914 wait complete. 2024-03-04T09:34:42.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:42.727+0000 7f2afa400640 1 Processor -- start 2024-03-04T09:34:42.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:42.727+0000 7f2afa400640 1 -- start start 2024-03-04T09:34:42.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:42.728+0000 7f2afa400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2af40d0e50 0x7f2af4152140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:42.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:42.728+0000 7f2afa400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2af40d4a10 0x7f2af40b53a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:42.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:42.728+0000 7f2afa400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2af415e0b0 0x7f2af415ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:42.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:42.728+0000 7f2afa400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2af415e490 con 0x7f2af40d4a10 2024-03-04T09:34:42.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:42.728+0000 7f2afa400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2af40cfa20 con 0x7f2af40d0e50 2024-03-04T09:34:42.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:42.728+0000 7f2afa400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2af40cfc60 con 0x7f2af415e0b0 2024-03-04T09:34:42.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:42.729+0000 7f2af3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2af40d4a10 0x7f2af40b53a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:42.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:42.729+0000 7f2af3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2af40d4a10 0x7f2af40b53a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:56982/0 (socket says 172.21.15.6:56982) 2024-03-04T09:34:42.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:42.729+0000 7f2af3e00640 1 -- 172.21.15.6:0/2117447520 learned_addr learned my addr 172.21.15.6:0/2117447520 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:34:42.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:42.729+0000 7f2af9000640 1 --2- 172.21.15.6:0/2117447520 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2af40d0e50 0x7f2af4152140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:42.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:42.729+0000 7f2af9a00640 1 --2- 172.21.15.6:0/2117447520 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2af415e0b0 0x7f2af415ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:42.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:42.729+0000 7f2af3e00640 1 -- 172.21.15.6:0/2117447520 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2af415e0b0 msgr2=0x7f2af415ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:42.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:42.729+0000 7f2af3e00640 1 --2- 172.21.15.6:0/2117447520 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2af415e0b0 0x7f2af415ed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:42.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:42.729+0000 7f2af3e00640 1 -- 172.21.15.6:0/2117447520 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2af40d0e50 msgr2=0x7f2af4152140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:42.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:42.729+0000 7f2af3e00640 1 --2- 172.21.15.6:0/2117447520 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2af40d0e50 0x7f2af4152140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:42.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:42.729+0000 7f2af3e00640 1 -- 172.21.15.6:0/2117447520 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2af40b5ea0 con 0x7f2af40d4a10 2024-03-04T09:34:42.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:42.729+0000 7f2af3e00640 1 --2- 172.21.15.6:0/2117447520 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2af40d4a10 0x7f2af40b53a0 secure :-1 s=READY pgs=1652 cs=0 l=1 rev1=1 crypto rx=0x7f2ae000c460 tx=0x7f2ae000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:34:42.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:42.730+0000 7f2af0c00640 1 -- 172.21.15.6:0/2117447520 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ae0005480 con 0x7f2af40d4a10 2024-03-04T09:34:42.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:42.730+0000 7f2af0c00640 1 -- 172.21.15.6:0/2117447520 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ae00035f0 con 0x7f2af40d4a10 2024-03-04T09:34:42.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:42.730+0000 7f2af0c00640 1 -- 172.21.15.6:0/2117447520 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ae00039e0 con 0x7f2af40d4a10 2024-03-04T09:34:42.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:42.730+0000 7f2afa400640 1 -- 172.21.15.6:0/2117447520 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2af40b6180 con 0x7f2af40d4a10 2024-03-04T09:34:42.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:42.730+0000 7f2afa400640 1 -- 172.21.15.6:0/2117447520 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2af40b7cf0 con 0x7f2af40d4a10 2024-03-04T09:34:42.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:42.731+0000 7f2afa400640 1 -- 172.21.15.6:0/2117447520 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2af404f330 con 0x7f2af40d4a10 2024-03-04T09:34:42.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:42.732+0000 7f2af0c00640 1 -- 172.21.15.6:0/2117447520 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2ae0003b80 con 0x7f2af40d4a10 2024-03-04T09:34:42.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:42.733+0000 7f2af0c00640 1 --2- 172.21.15.6:0/2117447520 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2ad406b670 0x7f2ad406db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:42.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:42.733+0000 7f2af0c00640 1 -- 172.21.15.6:0/2117447520 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f2ae0089f00 con 0x7f2af40d4a10 2024-03-04T09:34:42.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:42.734+0000 7f2af9000640 1 --2- 172.21.15.6:0/2117447520 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2ad406b670 0x7f2ad406db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:42.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:42.735+0000 7f2af9000640 1 --2- 172.21.15.6:0/2117447520 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2ad406b670 0x7f2ad406db30 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f2ae8000f60 tx=0x7f2ae8009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:34:42.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:42.737+0000 7f2af0c00640 1 -- 172.21.15.6:0/2117447520 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2ae0058d30 con 0x7f2af40d4a10 2024-03-04T09:34:43.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:43.063+0000 7f2afa400640 1 -- 172.21.15.6:0/2117447520 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2af40caa30 con 0x7f2af40d4a10 2024-03-04T09:34:43.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:43.064+0000 7f2af0c00640 1 -- 172.21.15.6:0/2117447520 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3465 (secure 0 0 0) 0x7f2ae0058630 con 0x7f2af40d4a10 2024-03-04T09:34:43.061 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:34:43.061 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:33:54.652601+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"laggy_since":"2024-03-04T09:33:54.652601+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":9}]} 2024-03-04T09:34:43.061 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 100 2024-03-04T09:34:43.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:43.067+0000 7f2afa400640 1 -- 172.21.15.6:0/2117447520 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2ad406b670 msgr2=0x7f2ad406db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:43.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:43.067+0000 7f2afa400640 1 --2- 172.21.15.6:0/2117447520 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2ad406b670 0x7f2ad406db30 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f2ae8000f60 tx=0x7f2ae8009040 comp rx=0 tx=0).stop 2024-03-04T09:34:43.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:43.067+0000 7f2afa400640 1 -- 172.21.15.6:0/2117447520 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2af40d4a10 msgr2=0x7f2af40b53a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:43.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:43.067+0000 7f2afa400640 1 --2- 172.21.15.6:0/2117447520 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2af40d4a10 0x7f2af40b53a0 secure :-1 s=READY pgs=1652 cs=0 l=1 rev1=1 crypto rx=0x7f2ae000c460 tx=0x7f2ae000c920 comp rx=0 tx=0).stop 2024-03-04T09:34:43.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:43.067+0000 7f2afa400640 1 -- 172.21.15.6:0/2117447520 shutdown_connections 2024-03-04T09:34:43.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:43.067+0000 7f2afa400640 1 --2- 172.21.15.6:0/2117447520 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2ad406b670 0x7f2ad406db30 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:43.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:43.067+0000 7f2afa400640 1 --2- 172.21.15.6:0/2117447520 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2af415e0b0 0x7f2af415ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:43.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:43.067+0000 7f2afa400640 1 --2- 172.21.15.6:0/2117447520 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2af40d4a10 0x7f2af40b53a0 unknown :-1 s=CLOSED pgs=1652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:43.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:43.067+0000 7f2afa400640 1 --2- 172.21.15.6:0/2117447520 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2af40d0e50 0x7f2af4152140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:43.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:43.067+0000 7f2afa400640 1 -- 172.21.15.6:0/2117447520 >> 172.21.15.6:0/2117447520 conn(0x7f2af40d3d30 msgr2=0x7f2af40cce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:34:43.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:43.067+0000 7f2afa400640 1 -- 172.21.15.6:0/2117447520 shutdown_connections 2024-03-04T09:34:43.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:43.067+0000 7f2afa400640 1 -- 172.21.15.6:0/2117447520 wait complete. 2024-03-04T09:34:43.284 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:32:57.159681+0000', 'modified': '2024-03-04T09:33:54.652601+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8214, 'mds_1': 6796}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6796': {'gid': 6796, 'name': 'd', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.146:6837/2458070840', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 2458070840}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 2458070840}]}, 'join_fscid': -1, 'laggy_since': '2024-03-04T09:33:54.652601+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8214': {'gid': 8214, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2446627517', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2446627517}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2446627517}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8214, 'qdb_cluster': [8214]} 2024-03-04T09:34:43.284 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6796:up:stopping 2024-03-04T09:34:43.848 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-04T09:34:44.285 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:34:44.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:44.513+0000 7f31d1800640 1 Processor -- start 2024-03-04T09:34:44.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:44.513+0000 7f31d1800640 1 -- start start 2024-03-04T09:34:44.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:44.515+0000 7f31d1800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f31cc071380 0x7f31cc0ad090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:44.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:44.515+0000 7f31d1800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f31cc06e240 0x7f31cc0ad5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:44.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:44.515+0000 7f31d1800640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31cc0ce9a0 con 0x7f31cc161670 2024-03-04T09:34:44.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:44.515+0000 7f31d1800640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31cc0730d0 con 0x7f31cc071380 2024-03-04T09:34:44.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:44.515+0000 7f31d1800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31cc0cec80 con 0x7f31cc06e240 2024-03-04T09:34:44.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:44.516+0000 7f31ca000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f31cc071380 0x7f31cc0ad090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:44.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:44.516+0000 7f31cb400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f31cc06e240 0x7f31cc0ad5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:44.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:44.516+0000 7f31ca000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f31cc071380 0x7f31cc0ad090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.6:45594/0 (socket says 172.21.15.6:45594) 2024-03-04T09:34:44.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:44.516+0000 7f31ca000640 1 -- 172.21.15.6:0/250397481 learned_addr learned my addr 172.21.15.6:0/250397481 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:34:44.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:44.517+0000 7f31cb400640 1 -- 172.21.15.6:0/250397481 >> v2:172.21.15.146:3300/0 conn(0x7f31cc071380 msgr2=0x7f31cc0ad090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:34:44.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:44.517+0000 7f31cb400640 1 --2- 172.21.15.6:0/250397481 >> v2:172.21.15.146:3300/0 conn(0x7f31cc071380 0x7f31cc0ad090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:44.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:44.517+0000 7f31cb400640 1 -- 172.21.15.6:0/250397481 >> v1:172.21.15.146:6789/0 conn(0x7f31cc161670 legacy=0x7f31cc070c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:44.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:44.517+0000 7f31cb400640 1 -- 172.21.15.6:0/250397481 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31cc0cef60 con 0x7f31cc06e240 2024-03-04T09:34:44.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:44.517+0000 7f31ca000640 1 --2- 172.21.15.6:0/250397481 >> v2:172.21.15.146:3300/0 conn(0x7f31cc071380 0x7f31cc0ad090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T09:34:44.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:44.518+0000 7f31cb400640 1 --2- 172.21.15.6:0/250397481 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f31cc06e240 0x7f31cc0ad5d0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f31c4009d90 tx=0x7f31c402be20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8a3b5a4b545ed403 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:34:44.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:44.518+0000 7f31c9600640 1 -- 172.21.15.6:0/250397481 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31c4039070 con 0x7f31cc06e240 2024-03-04T09:34:44.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:44.518+0000 7f31c9600640 1 -- 172.21.15.6:0/250397481 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31c402cbd0 con 0x7f31cc06e240 2024-03-04T09:34:44.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:44.518+0000 7f31c9600640 1 -- 172.21.15.6:0/250397481 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31c4034440 con 0x7f31cc06e240 2024-03-04T09:34:44.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:44.519+0000 7f31d1800640 1 -- 172.21.15.6:0/250397481 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f31cc06e240 msgr2=0x7f31cc0ad5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:44.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:44.519+0000 7f31d1800640 1 --2- 172.21.15.6:0/250397481 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f31cc06e240 0x7f31cc0ad5d0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f31c4009d90 tx=0x7f31c402be20 comp rx=0 tx=0).stop 2024-03-04T09:34:44.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:44.519+0000 7f31d1800640 1 -- 172.21.15.6:0/250397481 shutdown_connections 2024-03-04T09:34:44.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:44.519+0000 7f31d1800640 1 --2- 172.21.15.6:0/250397481 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f31cc06e240 0x7f31cc0ad5d0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:44.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:44.519+0000 7f31d1800640 1 --2- 172.21.15.6:0/250397481 >> v2:172.21.15.146:3300/0 conn(0x7f31cc071380 0x7f31cc0ad090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:44.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:44.519+0000 7f31d1800640 1 -- 172.21.15.6:0/250397481 >> 172.21.15.6:0/250397481 conn(0x7f31cc06afb0 msgr2=0x7f31cc06b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:34:44.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:44.519+0000 7f31d1800640 1 -- 172.21.15.6:0/250397481 shutdown_connections 2024-03-04T09:34:44.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:44.519+0000 7f31d1800640 1 -- 172.21.15.6:0/250397481 wait complete. 2024-03-04T09:34:44.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:44.521+0000 7f31d1800640 1 Processor -- start 2024-03-04T09:34:44.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:44.521+0000 7f31d1800640 1 -- start start 2024-03-04T09:34:44.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:44.522+0000 7f31d1800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f31cc06e240 0x7f31cc09c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:44.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:44.522+0000 7f31d1800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f31cc071380 0x7f31cc0b0c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:44.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:44.522+0000 7f31d1800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f31cc161670 0x7f31cc06d250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:44.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:44.522+0000 7f31d1800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31cc0ce600 con 0x7f31cc161670 2024-03-04T09:34:44.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:44.522+0000 7f31d1800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31cc0cf520 con 0x7f31cc071380 2024-03-04T09:34:44.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:44.522+0000 7f31d1800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31cc0ce780 con 0x7f31cc06e240 2024-03-04T09:34:44.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:44.523+0000 7f31cb400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f31cc161670 0x7f31cc06d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:44.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:44.523+0000 7f31cb400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f31cc161670 0x7f31cc06d250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:56988/0 (socket says 172.21.15.6:56988) 2024-03-04T09:34:44.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:44.523+0000 7f31cb400640 1 -- 172.21.15.6:0/3761911374 learned_addr learned my addr 172.21.15.6:0/3761911374 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:34:44.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:44.523+0000 7f31ca000640 1 --2- 172.21.15.6:0/3761911374 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f31cc071380 0x7f31cc0b0c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:44.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:44.523+0000 7f31caa00640 1 --2- 172.21.15.6:0/3761911374 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f31cc06e240 0x7f31cc09c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:44.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:44.523+0000 7f31cb400640 1 -- 172.21.15.6:0/3761911374 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f31cc06e240 msgr2=0x7f31cc09c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:44.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:44.523+0000 7f31cb400640 1 --2- 172.21.15.6:0/3761911374 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f31cc06e240 0x7f31cc09c5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:44.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:44.523+0000 7f31cb400640 1 -- 172.21.15.6:0/3761911374 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f31cc071380 msgr2=0x7f31cc0b0c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:44.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:44.523+0000 7f31cb400640 1 --2- 172.21.15.6:0/3761911374 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f31cc071380 0x7f31cc0b0c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:44.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:44.523+0000 7f31cb400640 1 -- 172.21.15.6:0/3761911374 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31cc0b1150 con 0x7f31cc161670 2024-03-04T09:34:44.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:44.523+0000 7f31caa00640 1 --2- 172.21.15.6:0/3761911374 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f31cc06e240 0x7f31cc09c5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T09:34:44.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:44.523+0000 7f31ca000640 1 --2- 172.21.15.6:0/3761911374 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f31cc071380 0x7f31cc0b0c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T09:34:44.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:44.524+0000 7f31cb400640 1 --2- 172.21.15.6:0/3761911374 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f31cc161670 0x7f31cc06d250 secure :-1 s=READY pgs=1653 cs=0 l=1 rev1=1 crypto rx=0x7f31c4009eb0 tx=0x7f31c4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:34:44.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:44.524+0000 7f31c2a00640 1 -- 172.21.15.6:0/3761911374 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31c4039050 con 0x7f31cc161670 2024-03-04T09:34:44.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:44.524+0000 7f31c2a00640 1 -- 172.21.15.6:0/3761911374 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31c4045070 con 0x7f31cc161670 2024-03-04T09:34:44.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:44.524+0000 7f31d1800640 1 -- 172.21.15.6:0/3761911374 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31cc09cd30 con 0x7f31cc161670 2024-03-04T09:34:44.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:44.524+0000 7f31c2a00640 1 -- 172.21.15.6:0/3761911374 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31c4009450 con 0x7f31cc161670 2024-03-04T09:34:44.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:44.524+0000 7f31d1800640 1 -- 172.21.15.6:0/3761911374 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31cc09d2f0 con 0x7f31cc161670 2024-03-04T09:34:44.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:44.525+0000 7f31d1800640 1 -- 172.21.15.6:0/3761911374 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31cc069510 con 0x7f31cc161670 2024-03-04T09:34:44.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:44.526+0000 7f31c2a00640 1 -- 172.21.15.6:0/3761911374 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f31c403e850 con 0x7f31cc161670 2024-03-04T09:34:44.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:44.527+0000 7f31c2a00640 1 --2- 172.21.15.6:0/3761911374 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f31ac06b4d0 0x7f31ac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:44.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:44.527+0000 7f31c2a00640 1 -- 172.21.15.6:0/3761911374 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f31c40ad2e0 con 0x7f31cc161670 2024-03-04T09:34:44.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:44.528+0000 7f31caa00640 1 --2- 172.21.15.6:0/3761911374 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f31ac06b4d0 0x7f31ac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:44.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:44.529+0000 7f31caa00640 1 --2- 172.21.15.6:0/3761911374 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f31ac06b4d0 0x7f31ac06d990 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f31cc06dd10 tx=0x7f31bc009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:34:44.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:44.536+0000 7f31c2a00640 1 -- 172.21.15.6:0/3761911374 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f31c407c120 con 0x7f31cc161670 2024-03-04T09:34:44.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:44.826+0000 7f31d1800640 1 -- 172.21.15.6:0/3761911374 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f31cc04f3a0 con 0x7f31cc161670 2024-03-04T09:34:44.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:44.827+0000 7f31c2a00640 1 -- 172.21.15.6:0/3761911374 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3465 (secure 0 0 0) 0x7f31c407ba20 con 0x7f31cc161670 2024-03-04T09:34:44.824 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:34:44.824 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:33:54.652601+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"laggy_since":"2024-03-04T09:33:54.652601+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":9}]} 2024-03-04T09:34:44.825 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 100 2024-03-04T09:34:44.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:44.829+0000 7f31d1800640 1 -- 172.21.15.6:0/3761911374 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f31ac06b4d0 msgr2=0x7f31ac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:44.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:44.829+0000 7f31d1800640 1 --2- 172.21.15.6:0/3761911374 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f31ac06b4d0 0x7f31ac06d990 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f31cc06dd10 tx=0x7f31bc009040 comp rx=0 tx=0).stop 2024-03-04T09:34:44.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:44.830+0000 7f31d1800640 1 -- 172.21.15.6:0/3761911374 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f31cc161670 msgr2=0x7f31cc06d250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:44.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:44.830+0000 7f31d1800640 1 --2- 172.21.15.6:0/3761911374 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f31cc161670 0x7f31cc06d250 secure :-1 s=READY pgs=1653 cs=0 l=1 rev1=1 crypto rx=0x7f31c4009eb0 tx=0x7f31c4004a40 comp rx=0 tx=0).stop 2024-03-04T09:34:44.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:44.830+0000 7f31d1800640 1 -- 172.21.15.6:0/3761911374 shutdown_connections 2024-03-04T09:34:44.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:44.830+0000 7f31d1800640 1 --2- 172.21.15.6:0/3761911374 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f31cc161670 0x7f31cc06d250 unknown :-1 s=CLOSED pgs=1653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:44.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:44.830+0000 7f31d1800640 1 --2- 172.21.15.6:0/3761911374 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f31cc071380 0x7f31cc0b0c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:44.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:44.830+0000 7f31d1800640 1 --2- 172.21.15.6:0/3761911374 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f31ac06b4d0 0x7f31ac06d990 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:44.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:44.830+0000 7f31d1800640 1 --2- 172.21.15.6:0/3761911374 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f31cc06e240 0x7f31cc09c5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:44.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:44.830+0000 7f31d1800640 1 -- 172.21.15.6:0/3761911374 >> 172.21.15.6:0/3761911374 conn(0x7f31cc06afb0 msgr2=0x7f31cc070c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:34:44.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:44.830+0000 7f31d1800640 1 -- 172.21.15.6:0/3761911374 shutdown_connections 2024-03-04T09:34:44.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:44.830+0000 7f31d1800640 1 -- 172.21.15.6:0/3761911374 wait complete. 2024-03-04T09:34:45.047 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:32:57.159681+0000', 'modified': '2024-03-04T09:33:54.652601+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8214, 'mds_1': 6796}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6796': {'gid': 6796, 'name': 'd', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.146:6837/2458070840', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 2458070840}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 2458070840}]}, 'join_fscid': -1, 'laggy_since': '2024-03-04T09:33:54.652601+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8214': {'gid': 8214, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2446627517', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2446627517}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2446627517}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8214, 'qdb_cluster': [8214]} 2024-03-04T09:34:45.047 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6796:up:stopping 2024-03-04T09:34:45.850 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-04T09:34:46.049 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:34:46.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:46.278+0000 7f39d7000640 1 Processor -- start 2024-03-04T09:34:46.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:46.278+0000 7f39d7000640 1 -- start start 2024-03-04T09:34:46.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:46.281+0000 7f39d7000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f39d00af580 0x7f39d015c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:46.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:46.281+0000 7f39d7000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f39d00cdc20 0x7f39d00b2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:46.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:46.281+0000 7f39d7000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39d015cc40 con 0x7f39d00af1a0 2024-03-04T09:34:46.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:46.281+0000 7f39d7000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39d015cf20 con 0x7f39d00cdc20 2024-03-04T09:34:46.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:46.281+0000 7f39d7000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39d015d200 con 0x7f39d00af580 2024-03-04T09:34:46.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:46.281+0000 7f39cfe00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f39d00af1a0 0x7f39d00b4c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:46922/0 (socket says 172.21.15.6:46922) 2024-03-04T09:34:46.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:46.281+0000 7f39cfe00640 1 -- 172.21.15.6:0/1267583052 learned_addr learned my addr 172.21.15.6:0/1267583052 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:34:46.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:46.282+0000 7f39d5000640 1 --2- 172.21.15.6:0/1267583052 >> v2:172.21.15.146:3300/0 conn(0x7f39d00cdc20 0x7f39d00b2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:46.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:46.282+0000 7f39c7e00640 1 --2- 172.21.15.6:0/1267583052 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f39d00af580 0x7f39d015c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:46.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:46.282+0000 7f39cf400640 1 -- 172.21.15.6:0/1267583052 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1496802462 0 0) 0x7f39d015cc40 con 0x7f39d00af1a0 2024-03-04T09:34:46.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:46.282+0000 7f39cf400640 1 -- 172.21.15.6:0/1267583052 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39b4003520 con 0x7f39d00af1a0 2024-03-04T09:34:46.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:46.282+0000 7f39cf400640 1 -- 172.21.15.6:0/1267583052 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1955624776 0 0) 0x7f39b4003520 con 0x7f39d00af1a0 2024-03-04T09:34:46.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:46.282+0000 7f39cf400640 1 -- 172.21.15.6:0/1267583052 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f39d00af580 msgr2=0x7f39d015c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:34:46.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:46.282+0000 7f39cf400640 1 --2- 172.21.15.6:0/1267583052 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f39d00af580 0x7f39d015c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:46.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:46.282+0000 7f39cf400640 1 -- 172.21.15.6:0/1267583052 >> v2:172.21.15.146:3300/0 conn(0x7f39d00cdc20 msgr2=0x7f39d00b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:34:46.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:46.282+0000 7f39cf400640 1 --2- 172.21.15.6:0/1267583052 >> v2:172.21.15.146:3300/0 conn(0x7f39d00cdc20 0x7f39d00b2dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:46.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:46.282+0000 7f39cf400640 1 -- 172.21.15.6:0/1267583052 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39d015d4e0 con 0x7f39d00af1a0 2024-03-04T09:34:46.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:46.282+0000 7f39cf400640 1 -- 172.21.15.6:0/1267583052 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7f39c0001670 con 0x7f39d00af1a0 2024-03-04T09:34:46.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:46.282+0000 7f39cf400640 1 -- 172.21.15.6:0/1267583052 >> v1:172.21.15.6:6789/0 conn(0x7f39d00af1a0 legacy=0x7f39d00b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:46.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:46.282+0000 7f39cf400640 1 --2- 172.21.15.6:0/1267583052 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f39b4003a20 0x7f39b4003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:46.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:46.282+0000 7f39cf400640 1 --2- 172.21.15.6:0/1267583052 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f39b4004800 0x7f39b4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:46.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:46.282+0000 7f39cf400640 1 --2- 172.21.15.6:0/1267583052 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f39d00af580 0x7f39b40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:46.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:46.282+0000 7f39cf400640 1 -- 172.21.15.6:0/1267583052 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39d015cf20 con 0x7f39b4004800 2024-03-04T09:34:46.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:46.282+0000 7f39cf400640 1 -- 172.21.15.6:0/1267583052 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39d015d200 con 0x7f39b4003a20 2024-03-04T09:34:46.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:46.282+0000 7f39cf400640 1 -- 172.21.15.6:0/1267583052 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39b400baa0 con 0x7f39d00af580 2024-03-04T09:34:46.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:46.283+0000 7f39c7e00640 1 --2- 172.21.15.6:0/1267583052 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f39b4004800 0x7f39b4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:46.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:46.283+0000 7f39cfe00640 1 --2- 172.21.15.6:0/1267583052 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f39b4003a20 0x7f39b4003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:46.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:46.283+0000 7f39d5000640 1 --2- 172.21.15.6:0/1267583052 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f39d00af580 0x7f39b40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:46.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:46.283+0000 7f39c7e00640 1 -- 172.21.15.6:0/1267583052 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f39d00af580 msgr2=0x7f39b40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:34:46.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:46.283+0000 7f39c7e00640 1 --2- 172.21.15.6:0/1267583052 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f39d00af580 0x7f39b40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:46.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:46.283+0000 7f39c7e00640 1 -- 172.21.15.6:0/1267583052 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f39b4003a20 msgr2=0x7f39b4003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:34:46.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:46.283+0000 7f39c7e00640 1 --2- 172.21.15.6:0/1267583052 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f39b4003a20 0x7f39b4003e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:46.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:46.283+0000 7f39c7e00640 1 -- 172.21.15.6:0/1267583052 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39b4003520 con 0x7f39b4004800 2024-03-04T09:34:46.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:46.285+0000 7f39c7e00640 1 --2- 172.21.15.6:0/1267583052 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f39b4004800 0x7f39b4004c80 secure :-1 s=READY pgs=1655 cs=0 l=1 rev1=1 crypto rx=0x7f39b8009a90 tx=0x7f39b4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fae1047eaa5c1d30 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:34:46.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:46.285+0000 7f39cf400640 1 -- 172.21.15.6:0/1267583052 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39b8039070 con 0x7f39b4004800 2024-03-04T09:34:46.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:46.285+0000 7f39cf400640 1 -- 172.21.15.6:0/1267583052 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39b802c680 con 0x7f39b4004800 2024-03-04T09:34:46.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:46.285+0000 7f39cf400640 1 -- 172.21.15.6:0/1267583052 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39b8009690 con 0x7f39b4004800 2024-03-04T09:34:46.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:46.286+0000 7f39d7000640 1 -- 172.21.15.6:0/1267583052 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f39b4004800 msgr2=0x7f39b4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:46.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:46.286+0000 7f39d7000640 1 --2- 172.21.15.6:0/1267583052 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f39b4004800 0x7f39b4004c80 secure :-1 s=READY pgs=1655 cs=0 l=1 rev1=1 crypto rx=0x7f39b8009a90 tx=0x7f39b4005830 comp rx=0 tx=0).stop 2024-03-04T09:34:46.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:46.286+0000 7f39d5000640 1 -- 172.21.15.6:0/1267583052 reap_dead start 2024-03-04T09:34:46.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:46.286+0000 7f39d7000640 1 -- 172.21.15.6:0/1267583052 shutdown_connections 2024-03-04T09:34:46.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:46.286+0000 7f39d7000640 1 -- 172.21.15.6:0/1267583052 >> 172.21.15.6:0/1267583052 conn(0x7f39d006afb0 msgr2=0x7f39d006b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:34:46.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:46.286+0000 7f39d7000640 1 -- 172.21.15.6:0/1267583052 shutdown_connections 2024-03-04T09:34:46.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:46.286+0000 7f39d7000640 1 -- 172.21.15.6:0/1267583052 wait complete. 2024-03-04T09:34:46.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:46.289+0000 7f39d7000640 1 Processor -- start 2024-03-04T09:34:46.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:46.289+0000 7f39d7000640 1 -- start start 2024-03-04T09:34:46.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:46.291+0000 7f39d7000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f39b4004800 0x7f39d014efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:46.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:46.291+0000 7f39d7000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f39d009aaa0 0x7f39d009b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:46.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:46.291+0000 7f39d7000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f39d00cca40 0x7f39d00ccef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:46.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:46.291+0000 7f39d7000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39d015c990 con 0x7f39d00cca40 2024-03-04T09:34:46.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:46.291+0000 7f39d7000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39d015c810 con 0x7f39d009aaa0 2024-03-04T09:34:46.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:46.291+0000 7f39d7000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39d00b3310 con 0x7f39b4004800 2024-03-04T09:34:46.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:46.291+0000 7f39d5000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f39d00cca40 0x7f39d00ccef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:46.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:46.291+0000 7f39d5000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f39d00cca40 0x7f39d00ccef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57000/0 (socket says 172.21.15.6:57000) 2024-03-04T09:34:46.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:46.291+0000 7f39d5000640 1 -- 172.21.15.6:0/270983034 learned_addr learned my addr 172.21.15.6:0/270983034 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:34:46.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:46.292+0000 7f39cfe00640 1 --2- 172.21.15.6:0/270983034 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f39b4004800 0x7f39d014efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:46.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:46.292+0000 7f39c7e00640 1 --2- 172.21.15.6:0/270983034 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f39d009aaa0 0x7f39d009b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:46.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:46.292+0000 7f39d5000640 1 -- 172.21.15.6:0/270983034 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f39b4004800 msgr2=0x7f39d014efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:46.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:46.292+0000 7f39d5000640 1 --2- 172.21.15.6:0/270983034 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f39b4004800 0x7f39d014efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:46.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:46.292+0000 7f39d5000640 1 -- 172.21.15.6:0/270983034 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f39d009aaa0 msgr2=0x7f39d009b3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:46.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:46.292+0000 7f39d5000640 1 --2- 172.21.15.6:0/270983034 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f39d009aaa0 0x7f39d009b3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:46.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:46.292+0000 7f39d5000640 1 -- 172.21.15.6:0/270983034 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39d009d1c0 con 0x7f39d00cca40 2024-03-04T09:34:46.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:46.292+0000 7f39cfe00640 1 --2- 172.21.15.6:0/270983034 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f39b4004800 0x7f39d014efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:34:46.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:46.292+0000 7f39d5000640 1 --2- 172.21.15.6:0/270983034 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f39d00cca40 0x7f39d00ccef0 secure :-1 s=READY pgs=1656 cs=0 l=1 rev1=1 crypto rx=0x7f39d00728b0 tx=0x7f39c80115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:34:46.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:46.292+0000 7f39ccc00640 1 -- 172.21.15.6:0/270983034 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39c8011e10 con 0x7f39d00cca40 2024-03-04T09:34:46.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:46.292+0000 7f39ccc00640 1 -- 172.21.15.6:0/270983034 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39c801b070 con 0x7f39d00cca40 2024-03-04T09:34:46.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:46.292+0000 7f39ccc00640 1 -- 172.21.15.6:0/270983034 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39c8003990 con 0x7f39d00cca40 2024-03-04T09:34:46.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:46.292+0000 7f39d7000640 1 -- 172.21.15.6:0/270983034 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39d009d4a0 con 0x7f39d00cca40 2024-03-04T09:34:46.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:46.293+0000 7f39d7000640 1 -- 172.21.15.6:0/270983034 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39d009da60 con 0x7f39d00cca40 2024-03-04T09:34:46.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:46.294+0000 7f39d7000640 1 -- 172.21.15.6:0/270983034 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39d0069510 con 0x7f39d00cca40 2024-03-04T09:34:46.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:46.295+0000 7f39ccc00640 1 -- 172.21.15.6:0/270983034 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f39c8003b50 con 0x7f39d00cca40 2024-03-04T09:34:46.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:46.296+0000 7f39ccc00640 1 --2- 172.21.15.6:0/270983034 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f39ac06b4d0 0x7f39ac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:46.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:46.296+0000 7f39ccc00640 1 -- 172.21.15.6:0/270983034 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f39c808aba0 con 0x7f39d00cca40 2024-03-04T09:34:46.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:46.297+0000 7f39cfe00640 1 --2- 172.21.15.6:0/270983034 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f39ac06b4d0 0x7f39ac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:46.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:46.298+0000 7f39cfe00640 1 --2- 172.21.15.6:0/270983034 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f39ac06b4d0 0x7f39ac06d990 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f39c0001880 tx=0x7f39c0001010 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:34:46.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:46.301+0000 7f39ccc00640 1 -- 172.21.15.6:0/270983034 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f39c80599d0 con 0x7f39d00cca40 2024-03-04T09:34:46.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:46.538+0000 7f39d7000640 1 -- 172.21.15.6:0/270983034 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f39d004f3a0 con 0x7f39d00cca40 2024-03-04T09:34:46.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:46.540+0000 7f39ccc00640 1 -- 172.21.15.6:0/270983034 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3465 (secure 0 0 0) 0x7f39c80592d0 con 0x7f39d00cca40 2024-03-04T09:34:46.537 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:34:46.537 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:33:54.652601+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"laggy_since":"2024-03-04T09:33:54.652601+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":9}]} 2024-03-04T09:34:46.537 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 100 2024-03-04T09:34:46.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:46.542+0000 7f39d7000640 1 -- 172.21.15.6:0/270983034 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f39ac06b4d0 msgr2=0x7f39ac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:46.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:46.542+0000 7f39d7000640 1 --2- 172.21.15.6:0/270983034 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f39ac06b4d0 0x7f39ac06d990 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f39c0001880 tx=0x7f39c0001010 comp rx=0 tx=0).stop 2024-03-04T09:34:46.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:46.542+0000 7f39d7000640 1 -- 172.21.15.6:0/270983034 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f39d00cca40 msgr2=0x7f39d00ccef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:46.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:46.542+0000 7f39d7000640 1 --2- 172.21.15.6:0/270983034 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f39d00cca40 0x7f39d00ccef0 secure :-1 s=READY pgs=1656 cs=0 l=1 rev1=1 crypto rx=0x7f39d00728b0 tx=0x7f39c80115a0 comp rx=0 tx=0).stop 2024-03-04T09:34:46.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:46.542+0000 7f39d7000640 1 -- 172.21.15.6:0/270983034 shutdown_connections 2024-03-04T09:34:46.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:46.542+0000 7f39d7000640 1 --2- 172.21.15.6:0/270983034 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f39d00cca40 0x7f39d00ccef0 unknown :-1 s=CLOSED pgs=1656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:46.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:46.542+0000 7f39d7000640 1 --2- 172.21.15.6:0/270983034 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f39d009aaa0 0x7f39d009b3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:46.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:46.542+0000 7f39d7000640 1 --2- 172.21.15.6:0/270983034 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f39ac06b4d0 0x7f39ac06d990 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:46.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:46.542+0000 7f39d7000640 1 --2- 172.21.15.6:0/270983034 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f39b4004800 0x7f39d014efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:46.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:46.542+0000 7f39d7000640 1 -- 172.21.15.6:0/270983034 >> 172.21.15.6:0/270983034 conn(0x7f39d006afb0 msgr2=0x7f39d00cbf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:34:46.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:46.543+0000 7f39d7000640 1 -- 172.21.15.6:0/270983034 shutdown_connections 2024-03-04T09:34:46.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:46.543+0000 7f39d7000640 1 -- 172.21.15.6:0/270983034 wait complete. 2024-03-04T09:34:46.761 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:32:57.159681+0000', 'modified': '2024-03-04T09:33:54.652601+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8214, 'mds_1': 6796}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6796': {'gid': 6796, 'name': 'd', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.146:6837/2458070840', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 2458070840}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 2458070840}]}, 'join_fscid': -1, 'laggy_since': '2024-03-04T09:33:54.652601+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8214': {'gid': 8214, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2446627517', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2446627517}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2446627517}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8214, 'qdb_cluster': [8214]} 2024-03-04T09:34:46.761 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6796:up:stopping 2024-03-04T09:34:47.762 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:34:47.852 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-04T09:34:47.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:47.991+0000 7f8667200640 1 Processor -- start 2024-03-04T09:34:47.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:47.991+0000 7f8667200640 1 -- start start 2024-03-04T09:34:47.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:47.992+0000 7f8667200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f866815e0b0 0x7f86680cf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:47.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:47.992+0000 7f8667200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f86680d4a10 0x7f86680d0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:47.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:47.992+0000 7f8667200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86680b3710 con 0x7f86680d0c80 2024-03-04T09:34:47.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:47.993+0000 7f8667200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86680b09e0 con 0x7f866815e0b0 2024-03-04T09:34:47.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:47.993+0000 7f8667200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86680b39f0 con 0x7f86680d4a10 2024-03-04T09:34:47.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:47.993+0000 7f8666800640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f86680d0c80 0x7f86680d1900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:46932/0 (socket says 172.21.15.6:46932) 2024-03-04T09:34:47.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:47.993+0000 7f8666800640 1 -- 172.21.15.6:0/3993538387 learned_addr learned my addr 172.21.15.6:0/3993538387 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:34:47.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:47.993+0000 7f8665e00640 1 --2- 172.21.15.6:0/3993538387 >> v2:172.21.15.146:3300/0 conn(0x7f866815e0b0 0x7f86680cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:47.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:47.993+0000 7f865fe00640 1 --2- 172.21.15.6:0/3993538387 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f86680d4a10 0x7f86680d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:47.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:47.993+0000 7f8665400640 1 -- 172.21.15.6:0/3993538387 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3868746172 0 0) 0x7f86680b3710 con 0x7f86680d0c80 2024-03-04T09:34:47.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:47.994+0000 7f8665400640 1 -- 172.21.15.6:0/3993538387 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f864c003520 con 0x7f86680d0c80 2024-03-04T09:34:47.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:47.994+0000 7f8665400640 1 -- 172.21.15.6:0/3993538387 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2856676582 0 0) 0x7f864c003520 con 0x7f86680d0c80 2024-03-04T09:34:47.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:47.994+0000 7f8665e00640 1 -- 172.21.15.6:0/3993538387 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f86680d4a10 msgr2=0x7f86680d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:34:47.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:47.994+0000 7f8665e00640 1 --2- 172.21.15.6:0/3993538387 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f86680d4a10 0x7f86680d0740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:47.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:47.994+0000 7f8665e00640 1 -- 172.21.15.6:0/3993538387 >> v1:172.21.15.6:6789/0 conn(0x7f86680d0c80 legacy=0x7f86680d1900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:47.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:47.994+0000 7f8665e00640 1 -- 172.21.15.6:0/3993538387 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86680b3cd0 con 0x7f866815e0b0 2024-03-04T09:34:47.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:47.996+0000 7f8665e00640 1 --2- 172.21.15.6:0/3993538387 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f866815e0b0 0x7f86680cf310 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f86580097a0 tx=0x7f865802b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fd50340943d3bd38 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:34:47.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:47.996+0000 7f8665400640 1 -- 172.21.15.6:0/3993538387 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8658037070 con 0x7f866815e0b0 2024-03-04T09:34:47.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:47.996+0000 7f8665400640 1 -- 172.21.15.6:0/3993538387 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8658030510 con 0x7f866815e0b0 2024-03-04T09:34:47.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:47.996+0000 7f8665400640 1 -- 172.21.15.6:0/3993538387 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86600019f0 con 0x7f866815e0b0 2024-03-04T09:34:47.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:47.997+0000 7f8667200640 1 -- 172.21.15.6:0/3993538387 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f866815e0b0 msgr2=0x7f86680cf310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:47.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:47.997+0000 7f8667200640 1 --2- 172.21.15.6:0/3993538387 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f866815e0b0 0x7f86680cf310 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f86580097a0 tx=0x7f865802b790 comp rx=0 tx=0).stop 2024-03-04T09:34:47.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:47.997+0000 7f8667200640 1 -- 172.21.15.6:0/3993538387 shutdown_connections 2024-03-04T09:34:47.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:47.997+0000 7f8667200640 1 --2- 172.21.15.6:0/3993538387 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f86680d4a10 0x7f86680d0740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:47.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:47.997+0000 7f8667200640 1 --2- 172.21.15.6:0/3993538387 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f866815e0b0 0x7f86680cf310 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:47.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:47.997+0000 7f8667200640 1 -- 172.21.15.6:0/3993538387 >> 172.21.15.6:0/3993538387 conn(0x7f86680d3d30 msgr2=0x7f86680d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:34:47.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:47.997+0000 7f8667200640 1 -- 172.21.15.6:0/3993538387 shutdown_connections 2024-03-04T09:34:47.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:47.997+0000 7f8667200640 1 -- 172.21.15.6:0/3993538387 wait complete. 2024-03-04T09:34:47.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:48.000+0000 7f8667200640 1 Processor -- start 2024-03-04T09:34:47.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:48.000+0000 7f8667200640 1 -- start start 2024-03-04T09:34:48.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:48.002+0000 7f8667200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f86680d0c80 0x7f866815b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:48.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:48.002+0000 7f8667200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f86680d4a10 0x7f86681f8130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:48.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:48.002+0000 7f8667200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f866815e0b0 0x7f86680b7360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:48.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:48.002+0000 7f8667200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86680cfc60 con 0x7f86680d4a10 2024-03-04T09:34:48.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:48.002+0000 7f8667200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f866815e490 con 0x7f86680d0c80 2024-03-04T09:34:48.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:48.002+0000 7f8667200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86680b54e0 con 0x7f866815e0b0 2024-03-04T09:34:48.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:48.002+0000 7f865fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f86680d4a10 0x7f86681f8130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:48.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:48.002+0000 7f865fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f86680d4a10 0x7f86681f8130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57014/0 (socket says 172.21.15.6:57014) 2024-03-04T09:34:48.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:48.002+0000 7f865fe00640 1 -- 172.21.15.6:0/2878814215 learned_addr learned my addr 172.21.15.6:0/2878814215 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:34:48.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:48.003+0000 7f8665e00640 1 --2- 172.21.15.6:0/2878814215 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f86680d0c80 0x7f866815b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:48.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:48.003+0000 7f8666800640 1 --2- 172.21.15.6:0/2878814215 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f866815e0b0 0x7f86680b7360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:48.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:48.003+0000 7f865fe00640 1 -- 172.21.15.6:0/2878814215 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f866815e0b0 msgr2=0x7f86680b7360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:48.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:48.003+0000 7f865fe00640 1 --2- 172.21.15.6:0/2878814215 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f866815e0b0 0x7f86680b7360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:48.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:48.003+0000 7f865fe00640 1 -- 172.21.15.6:0/2878814215 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f86680d0c80 msgr2=0x7f866815b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:48.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:48.003+0000 7f865fe00640 1 --2- 172.21.15.6:0/2878814215 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f86680d0c80 0x7f866815b3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:48.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:48.003+0000 7f865fe00640 1 -- 172.21.15.6:0/2878814215 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86680b7e60 con 0x7f86680d4a10 2024-03-04T09:34:48.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:48.003+0000 7f8665e00640 1 --2- 172.21.15.6:0/2878814215 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f86680d0c80 0x7f866815b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T09:34:48.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:48.003+0000 7f865fe00640 1 --2- 172.21.15.6:0/2878814215 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f86680d4a10 0x7f86681f8130 secure :-1 s=READY pgs=1658 cs=0 l=1 rev1=1 crypto rx=0x7f865000e980 tx=0x7f865000ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:34:48.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:48.004+0000 7f865e000640 1 -- 172.21.15.6:0/2878814215 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8650017070 con 0x7f86680d4a10 2024-03-04T09:34:48.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:48.004+0000 7f865e000640 1 -- 172.21.15.6:0/2878814215 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f865000ca00 con 0x7f86680d4a10 2024-03-04T09:34:48.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:48.004+0000 7f865e000640 1 -- 172.21.15.6:0/2878814215 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f865000cdd0 con 0x7f86680d4a10 2024-03-04T09:34:48.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:48.004+0000 7f8667200640 1 -- 172.21.15.6:0/2878814215 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86680b8140 con 0x7f86680d4a10 2024-03-04T09:34:48.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:48.004+0000 7f8667200640 1 -- 172.21.15.6:0/2878814215 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86680b8700 con 0x7f86680d4a10 2024-03-04T09:34:48.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:48.005+0000 7f8667200640 1 -- 172.21.15.6:0/2878814215 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f866804f330 con 0x7f86680d4a10 2024-03-04T09:34:48.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:48.006+0000 7f865e000640 1 -- 172.21.15.6:0/2878814215 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f865001a070 con 0x7f86680d4a10 2024-03-04T09:34:48.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:48.007+0000 7f865e000640 1 --2- 172.21.15.6:0/2878814215 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f864406b4d0 0x7f864406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:48.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:48.007+0000 7f865e000640 1 -- 172.21.15.6:0/2878814215 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f865008a690 con 0x7f86680d4a10 2024-03-04T09:34:48.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:48.008+0000 7f8665e00640 1 --2- 172.21.15.6:0/2878814215 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f864406b4d0 0x7f864406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:48.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:48.009+0000 7f8665e00640 1 --2- 172.21.15.6:0/2878814215 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f864406b4d0 0x7f864406d990 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f8658034000 tx=0x7f865802f000 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:34:48.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:48.011+0000 7f865e000640 1 -- 172.21.15.6:0/2878814215 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8650059430 con 0x7f86680d4a10 2024-03-04T09:34:48.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:48.255+0000 7f8667200640 1 -- 172.21.15.6:0/2878814215 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f86680b3440 con 0x7f86680d4a10 2024-03-04T09:34:48.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:48.256+0000 7f865e000640 1 -- 172.21.15.6:0/2878814215 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3465 (secure 0 0 0) 0x7f8650058d30 con 0x7f86680d4a10 2024-03-04T09:34:48.253 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:34:48.253 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:33:54.652601+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"laggy_since":"2024-03-04T09:33:54.652601+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":9}]} 2024-03-04T09:34:48.253 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 100 2024-03-04T09:34:48.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:48.258+0000 7f8667200640 1 -- 172.21.15.6:0/2878814215 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f864406b4d0 msgr2=0x7f864406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:48.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:48.258+0000 7f8667200640 1 --2- 172.21.15.6:0/2878814215 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f864406b4d0 0x7f864406d990 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f8658034000 tx=0x7f865802f000 comp rx=0 tx=0).stop 2024-03-04T09:34:48.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:48.258+0000 7f8667200640 1 -- 172.21.15.6:0/2878814215 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f86680d4a10 msgr2=0x7f86681f8130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:48.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:48.258+0000 7f8667200640 1 --2- 172.21.15.6:0/2878814215 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f86680d4a10 0x7f86681f8130 secure :-1 s=READY pgs=1658 cs=0 l=1 rev1=1 crypto rx=0x7f865000e980 tx=0x7f865000ee40 comp rx=0 tx=0).stop 2024-03-04T09:34:48.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:48.258+0000 7f8667200640 1 -- 172.21.15.6:0/2878814215 shutdown_connections 2024-03-04T09:34:48.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:48.258+0000 7f8667200640 1 --2- 172.21.15.6:0/2878814215 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f864406b4d0 0x7f864406d990 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:48.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:48.258+0000 7f8667200640 1 --2- 172.21.15.6:0/2878814215 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f866815e0b0 0x7f86680b7360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:48.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:48.258+0000 7f8667200640 1 --2- 172.21.15.6:0/2878814215 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f86680d4a10 0x7f86681f8130 unknown :-1 s=CLOSED pgs=1658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:48.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:48.258+0000 7f8667200640 1 --2- 172.21.15.6:0/2878814215 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f86680d0c80 0x7f866815b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:48.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:48.258+0000 7f8667200640 1 -- 172.21.15.6:0/2878814215 >> 172.21.15.6:0/2878814215 conn(0x7f86680d3d30 msgr2=0x7f86680cf070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:34:48.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:48.259+0000 7f8667200640 1 -- 172.21.15.6:0/2878814215 shutdown_connections 2024-03-04T09:34:48.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:48.259+0000 7f8667200640 1 -- 172.21.15.6:0/2878814215 wait complete. 2024-03-04T09:34:48.475 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:32:57.159681+0000', 'modified': '2024-03-04T09:33:54.652601+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8214, 'mds_1': 6796}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6796': {'gid': 6796, 'name': 'd', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.146:6837/2458070840', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 2458070840}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 2458070840}]}, 'join_fscid': -1, 'laggy_since': '2024-03-04T09:33:54.652601+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8214': {'gid': 8214, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2446627517', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2446627517}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2446627517}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8214, 'qdb_cluster': [8214]} 2024-03-04T09:34:48.475 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6796:up:stopping 2024-03-04T09:34:49.476 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:34:49.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:49.699+0000 7fe9bdc00640 1 Processor -- start 2024-03-04T09:34:49.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:49.699+0000 7fe9bdc00640 1 -- start start 2024-03-04T09:34:49.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:49.700+0000 7fe9bdc00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fe9b80cd060 0x7fe9b8161ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:49.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:49.700+0000 7fe9bdc00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9b81627f0 con 0x7fe9b815efc0 2024-03-04T09:34:49.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:49.700+0000 7fe9bdc00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9b806c860 con 0x7fe9b8162000 2024-03-04T09:34:49.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:49.700+0000 7fe9bdc00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9b815f3a0 con 0x7fe9b80cd060 2024-03-04T09:34:49.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:49.701+0000 7fe9b7e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fe9b815efc0 0x7fe9b80cc950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:45964/0 (socket says 172.21.15.6:45964) 2024-03-04T09:34:49.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:49.701+0000 7fe9b7e00640 1 -- 172.21.15.6:0/691682467 learned_addr learned my addr 172.21.15.6:0/691682467 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:34:49.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:49.701+0000 7fe9af400640 1 --2- 172.21.15.6:0/691682467 >> v2:172.21.15.146:3300/0 conn(0x7fe9b80cd060 0x7fe9b8161ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:49.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:49.701+0000 7fe9b7400640 1 -- 172.21.15.6:0/691682467 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 520682669 0 0) 0x7fe9b81627f0 con 0x7fe9b815efc0 2024-03-04T09:34:49.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:49.701+0000 7fe9b7400640 1 -- 172.21.15.6:0/691682467 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe998003520 con 0x7fe9b815efc0 2024-03-04T09:34:49.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:49.702+0000 7fe9b7400640 1 -- 172.21.15.6:0/691682467 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2694985445 0 0) 0x7fe998003520 con 0x7fe9b815efc0 2024-03-04T09:34:49.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:49.702+0000 7fe9b7400640 1 -- 172.21.15.6:0/691682467 >> v2:172.21.15.146:3300/0 conn(0x7fe9b80cd060 msgr2=0x7fe9b8161ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:34:49.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:49.702+0000 7fe9b7400640 1 --2- 172.21.15.6:0/691682467 >> v2:172.21.15.146:3300/0 conn(0x7fe9b80cd060 0x7fe9b8161ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:49.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:49.702+0000 7fe9b7400640 1 -- 172.21.15.6:0/691682467 >> v1:172.21.15.146:6789/0 conn(0x7fe9b8162000 legacy=0x7fe9b8070c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:49.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:49.702+0000 7fe9b7400640 1 -- 172.21.15.6:0/691682467 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9b806cb40 con 0x7fe9b815efc0 2024-03-04T09:34:49.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:49.702+0000 7fe9b7400640 1 -- 172.21.15.6:0/691682467 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7fe9a8001670 con 0x7fe9b815efc0 2024-03-04T09:34:49.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:49.702+0000 7fe9b7400640 1 -- 172.21.15.6:0/691682467 >> v1:172.21.15.6:6789/0 conn(0x7fe9b815efc0 legacy=0x7fe9b80cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:49.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:49.702+0000 7fe9b7400640 1 --2- 172.21.15.6:0/691682467 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe998003a00 0x7fe998003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:49.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:49.702+0000 7fe9b7400640 1 --2- 172.21.15.6:0/691682467 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe998004800 0x7fe998004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:49.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:49.702+0000 7fe9b7400640 1 --2- 172.21.15.6:0/691682467 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe9980051c0 0x7fe99800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:49.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:49.702+0000 7fe9b7400640 1 -- 172.21.15.6:0/691682467 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9b815f3a0 con 0x7fe998004800 2024-03-04T09:34:49.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:49.702+0000 7fe9b7400640 1 -- 172.21.15.6:0/691682467 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe99800c0c0 con 0x7fe9980051c0 2024-03-04T09:34:49.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:49.702+0000 7fe9b7400640 1 -- 172.21.15.6:0/691682467 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe99800c2c0 con 0x7fe998003a00 2024-03-04T09:34:49.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:49.702+0000 7fe9af400640 1 --2- 172.21.15.6:0/691682467 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe998004800 0x7fe998004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:49.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:49.703+0000 7fe9b7e00640 1 --2- 172.21.15.6:0/691682467 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe998003a00 0x7fe998003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:49.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:49.703+0000 7fe9bd200640 1 --2- 172.21.15.6:0/691682467 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe9980051c0 0x7fe99800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:49.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:49.703+0000 7fe9af400640 1 -- 172.21.15.6:0/691682467 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe998003a00 msgr2=0x7fe998003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:34:49.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:49.703+0000 7fe9af400640 1 --2- 172.21.15.6:0/691682467 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe998003a00 0x7fe998003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:49.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:49.703+0000 7fe9af400640 1 -- 172.21.15.6:0/691682467 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe9980051c0 msgr2=0x7fe99800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:34:49.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:49.703+0000 7fe9af400640 1 --2- 172.21.15.6:0/691682467 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe9980051c0 0x7fe99800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:49.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:49.703+0000 7fe9af400640 1 -- 172.21.15.6:0/691682467 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe998003520 con 0x7fe998004800 2024-03-04T09:34:49.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:49.703+0000 7fe9b7e00640 1 --2- 172.21.15.6:0/691682467 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe998003a00 0x7fe998003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:34:49.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:49.703+0000 7fe9bd200640 1 --2- 172.21.15.6:0/691682467 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe9980051c0 0x7fe99800b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:34:49.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:49.703+0000 7fe9bd200640 1 -- 172.21.15.6:0/691682467 reap_dead start 2024-03-04T09:34:49.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:49.704+0000 7fe9af400640 1 --2- 172.21.15.6:0/691682467 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe998004800 0x7fe998004c80 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7fe9a0009ac0 tx=0x7fe998005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea1001a7597cf325 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:34:49.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:49.705+0000 7fe9b7400640 1 -- 172.21.15.6:0/691682467 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9a0039070 con 0x7fe998004800 2024-03-04T09:34:49.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:49.705+0000 7fe9b7400640 1 -- 172.21.15.6:0/691682467 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9a002c6b0 con 0x7fe998004800 2024-03-04T09:34:49.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:49.705+0000 7fe9b7400640 1 -- 172.21.15.6:0/691682467 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9a0009690 con 0x7fe998004800 2024-03-04T09:34:49.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:49.705+0000 7fe9bdc00640 1 -- 172.21.15.6:0/691682467 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe998004800 msgr2=0x7fe998004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:49.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:49.705+0000 7fe9bdc00640 1 --2- 172.21.15.6:0/691682467 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe998004800 0x7fe998004c80 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7fe9a0009ac0 tx=0x7fe998005830 comp rx=0 tx=0).stop 2024-03-04T09:34:49.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:49.706+0000 7fe9bdc00640 1 -- 172.21.15.6:0/691682467 shutdown_connections 2024-03-04T09:34:49.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:49.706+0000 7fe9bdc00640 1 --2- 172.21.15.6:0/691682467 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe998004800 0x7fe998004c80 unknown :-1 s=CLOSED pgs=1660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:49.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:49.706+0000 7fe9bdc00640 1 -- 172.21.15.6:0/691682467 >> 172.21.15.6:0/691682467 conn(0x7fe9b806b3d0 msgr2=0x7fe9b806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:34:49.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:49.706+0000 7fe9bdc00640 1 -- 172.21.15.6:0/691682467 shutdown_connections 2024-03-04T09:34:49.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:49.706+0000 7fe9bdc00640 1 -- 172.21.15.6:0/691682467 wait complete. 2024-03-04T09:34:49.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:49.708+0000 7fe9bdc00640 1 Processor -- start 2024-03-04T09:34:49.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:49.708+0000 7fe9bdc00640 1 -- start start 2024-03-04T09:34:49.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:49.709+0000 7fe9bdc00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe998004800 0x7fe9b8157890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:49.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:49.709+0000 7fe9bdc00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe9b81606a0 0x7fe9b8160b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:49.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:49.709+0000 7fe9bdc00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe9b8159410 0x7fe9b81598c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:49.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:49.709+0000 7fe9bdc00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9b8162470 con 0x7fe998004800 2024-03-04T09:34:49.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:49.709+0000 7fe9bdc00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9b806d190 con 0x7fe9b81606a0 2024-03-04T09:34:49.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:49.709+0000 7fe9bdc00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9b806d620 con 0x7fe9b8159410 2024-03-04T09:34:49.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:49.709+0000 7fe9b7e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe998004800 0x7fe9b8157890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:49.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:49.709+0000 7fe9b7e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe998004800 0x7fe9b8157890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:40384/0 (socket says 172.21.15.6:40384) 2024-03-04T09:34:49.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:49.709+0000 7fe9b7e00640 1 -- 172.21.15.6:0/1098054027 learned_addr learned my addr 172.21.15.6:0/1098054027 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:34:49.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:49.710+0000 7fe9b7e00640 1 -- 172.21.15.6:0/1098054027 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe9b8159410 msgr2=0x7fe9b81598c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:49.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:49.710+0000 7fe9bd200640 1 --2- 172.21.15.6:0/1098054027 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe9b8159410 0x7fe9b81598c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:49.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:49.710+0000 7fe9af400640 1 --2- 172.21.15.6:0/1098054027 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe9b81606a0 0x7fe9b8160b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:49.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:49.710+0000 7fe9b7e00640 1 --2- 172.21.15.6:0/1098054027 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe9b8159410 0x7fe9b81598c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:49.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:49.710+0000 7fe9b7e00640 1 -- 172.21.15.6:0/1098054027 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe9b81606a0 msgr2=0x7fe9b8160b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:49.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:49.710+0000 7fe9b7e00640 1 --2- 172.21.15.6:0/1098054027 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe9b81606a0 0x7fe9b8160b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:49.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:49.710+0000 7fe9b7e00640 1 -- 172.21.15.6:0/1098054027 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9b815a6a0 con 0x7fe998004800 2024-03-04T09:34:49.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:49.710+0000 7fe9af400640 1 --2- 172.21.15.6:0/1098054027 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe9b81606a0 0x7fe9b8160b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:34:49.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:49.710+0000 7fe9b7e00640 1 --2- 172.21.15.6:0/1098054027 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe998004800 0x7fe9b8157890 secure :-1 s=READY pgs=1661 cs=0 l=1 rev1=1 crypto rx=0x7fe9a800d840 tx=0x7fe9a800dd00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:34:49.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:49.710+0000 7fe9b4c00640 1 -- 172.21.15.6:0/1098054027 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9a8019070 con 0x7fe998004800 2024-03-04T09:34:49.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:49.711+0000 7fe9b4c00640 1 -- 172.21.15.6:0/1098054027 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9a80074e0 con 0x7fe998004800 2024-03-04T09:34:49.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:49.711+0000 7fe9b4c00640 1 -- 172.21.15.6:0/1098054027 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9a8004a00 con 0x7fe998004800 2024-03-04T09:34:49.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:49.711+0000 7fe9bdc00640 1 -- 172.21.15.6:0/1098054027 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9b815bb10 con 0x7fe998004800 2024-03-04T09:34:49.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:49.711+0000 7fe9bdc00640 1 -- 172.21.15.6:0/1098054027 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9b815a3c0 con 0x7fe998004800 2024-03-04T09:34:49.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:49.712+0000 7fe9bdc00640 1 -- 172.21.15.6:0/1098054027 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9b8161060 con 0x7fe998004800 2024-03-04T09:34:49.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:49.713+0000 7fe9b4c00640 1 -- 172.21.15.6:0/1098054027 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe9a8004ba0 con 0x7fe998004800 2024-03-04T09:34:49.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:49.714+0000 7fe9b4c00640 1 --2- 172.21.15.6:0/1098054027 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fe99406b4d0 0x7fe99406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:49.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:49.714+0000 7fe9b4c00640 1 -- 172.21.15.6:0/1098054027 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fe9a8089e40 con 0x7fe998004800 2024-03-04T09:34:49.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:49.715+0000 7fe9af400640 1 --2- 172.21.15.6:0/1098054027 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fe99406b4d0 0x7fe99406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:49.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:49.717+0000 7fe9af400640 1 --2- 172.21.15.6:0/1098054027 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fe99406b4d0 0x7fe99406d990 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fe9a0036000 tx=0x7fe9a0030000 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:34:49.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:49.721+0000 7fe9b4c00640 1 -- 172.21.15.6:0/1098054027 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe9a8058b60 con 0x7fe998004800 2024-03-04T09:34:49.853 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-04T09:34:49.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:49.960+0000 7fe9bdc00640 1 -- 172.21.15.6:0/1098054027 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe9b804f330 con 0x7fe998004800 2024-03-04T09:34:49.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:49.961+0000 7fe9b4c00640 1 -- 172.21.15.6:0/1098054027 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3465 (secure 0 0 0) 0x7fe9a8006e20 con 0x7fe998004800 2024-03-04T09:34:49.958 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:34:49.959 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:33:54.652601+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"laggy_since":"2024-03-04T09:33:54.652601+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":9}]} 2024-03-04T09:34:49.959 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 100 2024-03-04T09:34:49.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:49.964+0000 7fe9bdc00640 1 -- 172.21.15.6:0/1098054027 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fe99406b4d0 msgr2=0x7fe99406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:49.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:49.964+0000 7fe9bdc00640 1 --2- 172.21.15.6:0/1098054027 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fe99406b4d0 0x7fe99406d990 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fe9a0036000 tx=0x7fe9a0030000 comp rx=0 tx=0).stop 2024-03-04T09:34:49.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:49.964+0000 7fe9bdc00640 1 -- 172.21.15.6:0/1098054027 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe998004800 msgr2=0x7fe9b8157890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:49.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:49.964+0000 7fe9bdc00640 1 --2- 172.21.15.6:0/1098054027 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe998004800 0x7fe9b8157890 secure :-1 s=READY pgs=1661 cs=0 l=1 rev1=1 crypto rx=0x7fe9a800d840 tx=0x7fe9a800dd00 comp rx=0 tx=0).stop 2024-03-04T09:34:49.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:49.964+0000 7fe9bdc00640 1 -- 172.21.15.6:0/1098054027 shutdown_connections 2024-03-04T09:34:49.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:49.964+0000 7fe9bdc00640 1 --2- 172.21.15.6:0/1098054027 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fe99406b4d0 0x7fe99406d990 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:49.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:49.964+0000 7fe9bdc00640 1 --2- 172.21.15.6:0/1098054027 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe9b8159410 0x7fe9b81598c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:49.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:49.964+0000 7fe9bdc00640 1 --2- 172.21.15.6:0/1098054027 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe9b81606a0 0x7fe9b8160b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:49.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:49.964+0000 7fe9bdc00640 1 --2- 172.21.15.6:0/1098054027 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe998004800 0x7fe9b8157890 unknown :-1 s=CLOSED pgs=1661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:49.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:49.964+0000 7fe9bdc00640 1 -- 172.21.15.6:0/1098054027 >> 172.21.15.6:0/1098054027 conn(0x7fe9b806b3d0 msgr2=0x7fe9b80cdd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:34:49.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:49.964+0000 7fe9bdc00640 1 -- 172.21.15.6:0/1098054027 shutdown_connections 2024-03-04T09:34:49.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:49.964+0000 7fe9bdc00640 1 -- 172.21.15.6:0/1098054027 wait complete. 2024-03-04T09:34:50.181 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:32:57.159681+0000', 'modified': '2024-03-04T09:33:54.652601+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8214, 'mds_1': 6796}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6796': {'gid': 6796, 'name': 'd', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.146:6837/2458070840', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 2458070840}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 2458070840}]}, 'join_fscid': -1, 'laggy_since': '2024-03-04T09:33:54.652601+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8214': {'gid': 8214, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2446627517', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2446627517}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2446627517}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8214, 'qdb_cluster': [8214]} 2024-03-04T09:34:50.182 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6796:up:stopping 2024-03-04T09:34:51.182 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:34:51.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:51.412+0000 7fb1f2a00640 1 Processor -- start 2024-03-04T09:34:51.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:51.413+0000 7fb1f2a00640 1 -- start start 2024-03-04T09:34:51.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:51.414+0000 7fb1f2a00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fb1ec0af1a0 0x7fb1ec15c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:51.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:51.414+0000 7fb1f2a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fb1ec0cdc20 0x7fb1ec0b2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:51.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:51.414+0000 7fb1f2a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1ec15cc40 con 0x7fb1ec0af580 2024-03-04T09:34:51.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:51.414+0000 7fb1f2a00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1ec15cf20 con 0x7fb1ec0cdc20 2024-03-04T09:34:51.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:51.414+0000 7fb1f2a00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1ec15d200 con 0x7fb1ec0af1a0 2024-03-04T09:34:51.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:51.415+0000 7fb1f0a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fb1ec0cdc20 0x7fb1ec0b2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:51.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:51.415+0000 7fb1f0a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fb1ec0cdc20 0x7fb1ec0b2dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:40392/0 (socket says 172.21.15.6:40392) 2024-03-04T09:34:51.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:51.415+0000 7fb1f0a00640 1 -- 172.21.15.6:0/2905344127 learned_addr learned my addr 172.21.15.6:0/2905344127 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:34:51.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:51.415+0000 7fb1ebe00640 1 --2- 172.21.15.6:0/2905344127 >> v2:172.21.15.146:3300/0 conn(0x7fb1ec0af1a0 0x7fb1ec15c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:51.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:51.415+0000 7fb1f0a00640 1 -- 172.21.15.6:0/2905344127 >> v2:172.21.15.146:3300/0 conn(0x7fb1ec0af1a0 msgr2=0x7fb1ec15c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:34:51.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:51.415+0000 7fb1f0a00640 1 --2- 172.21.15.6:0/2905344127 >> v2:172.21.15.146:3300/0 conn(0x7fb1ec0af1a0 0x7fb1ec15c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:51.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:51.415+0000 7fb1f0a00640 1 -- 172.21.15.6:0/2905344127 >> v1:172.21.15.6:6789/0 conn(0x7fb1ec0af580 legacy=0x7fb1ec0b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:51.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:51.415+0000 7fb1f0a00640 1 -- 172.21.15.6:0/2905344127 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1ec15d4e0 con 0x7fb1ec0cdc20 2024-03-04T09:34:51.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:51.416+0000 7fb1f0a00640 1 --2- 172.21.15.6:0/2905344127 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb1ec0cdc20 0x7fb1ec0b2dd0 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 crypto rx=0x7fb1ec074960 tx=0x7fb1e402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d44c7b1c8123f8a server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:34:51.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:51.417+0000 7fb1eb400640 1 -- 172.21.15.6:0/2905344127 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1e402ca80 con 0x7fb1ec0cdc20 2024-03-04T09:34:51.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:51.417+0000 7fb1eb400640 1 -- 172.21.15.6:0/2905344127 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1e402cc20 con 0x7fb1ec0cdc20 2024-03-04T09:34:51.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:51.417+0000 7fb1eb400640 1 -- 172.21.15.6:0/2905344127 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1e40343f0 con 0x7fb1ec0cdc20 2024-03-04T09:34:51.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:51.417+0000 7fb1f2a00640 1 -- 172.21.15.6:0/2905344127 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb1ec0cdc20 msgr2=0x7fb1ec0b2dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:51.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:51.417+0000 7fb1f2a00640 1 --2- 172.21.15.6:0/2905344127 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb1ec0cdc20 0x7fb1ec0b2dd0 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 crypto rx=0x7fb1ec074960 tx=0x7fb1e402bee0 comp rx=0 tx=0).stop 2024-03-04T09:34:51.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:51.418+0000 7fb1f2a00640 1 -- 172.21.15.6:0/2905344127 shutdown_connections 2024-03-04T09:34:51.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:51.418+0000 7fb1f2a00640 1 --2- 172.21.15.6:0/2905344127 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb1ec0cdc20 0x7fb1ec0b2dd0 unknown :-1 s=CLOSED pgs=1663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:51.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:51.418+0000 7fb1f2a00640 1 --2- 172.21.15.6:0/2905344127 >> v2:172.21.15.146:3300/0 conn(0x7fb1ec0af1a0 0x7fb1ec15c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:51.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:51.418+0000 7fb1f2a00640 1 -- 172.21.15.6:0/2905344127 >> 172.21.15.6:0/2905344127 conn(0x7fb1ec06afb0 msgr2=0x7fb1ec06b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:34:51.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:51.418+0000 7fb1f2a00640 1 -- 172.21.15.6:0/2905344127 shutdown_connections 2024-03-04T09:34:51.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:51.418+0000 7fb1f2a00640 1 -- 172.21.15.6:0/2905344127 wait complete. 2024-03-04T09:34:51.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:51.419+0000 7fb1f2a00640 1 Processor -- start 2024-03-04T09:34:51.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:51.420+0000 7fb1f2a00640 1 -- start start 2024-03-04T09:34:51.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:51.420+0000 7fb1f2a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb1ec0af1a0 0x7fb1ec152bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:51.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:51.420+0000 7fb1f2a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb1ec0af580 0x7fb1ec14c270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:51.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:51.421+0000 7fb1f2a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb1ec0cdc20 0x7fb1ec0b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:51.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:51.421+0000 7fb1f2a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1ec0b36a0 con 0x7fb1ec0af580 2024-03-04T09:34:51.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:51.421+0000 7fb1f2a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1ec0b3520 con 0x7fb1ec0cdc20 2024-03-04T09:34:51.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:51.421+0000 7fb1f2a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1ec0b3820 con 0x7fb1ec0af1a0 2024-03-04T09:34:51.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:51.421+0000 7fb1e3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb1ec0af580 0x7fb1ec14c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:51.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:51.421+0000 7fb1e3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb1ec0af580 0x7fb1ec14c270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:40398/0 (socket says 172.21.15.6:40398) 2024-03-04T09:34:51.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:51.421+0000 7fb1e3e00640 1 -- 172.21.15.6:0/3560682274 learned_addr learned my addr 172.21.15.6:0/3560682274 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:34:51.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:51.421+0000 7fb1ebe00640 1 --2- 172.21.15.6:0/3560682274 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb1ec0af1a0 0x7fb1ec152bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:51.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:51.421+0000 7fb1f0a00640 1 --2- 172.21.15.6:0/3560682274 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb1ec0cdc20 0x7fb1ec0b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:51.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:51.421+0000 7fb1e3e00640 1 -- 172.21.15.6:0/3560682274 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb1ec0af1a0 msgr2=0x7fb1ec152bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:51.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:51.421+0000 7fb1e3e00640 1 --2- 172.21.15.6:0/3560682274 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb1ec0af1a0 0x7fb1ec152bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:51.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:51.421+0000 7fb1e3e00640 1 -- 172.21.15.6:0/3560682274 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb1ec0cdc20 msgr2=0x7fb1ec0b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:51.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:51.421+0000 7fb1e3e00640 1 --2- 172.21.15.6:0/3560682274 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb1ec0cdc20 0x7fb1ec0b0cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:51.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:51.421+0000 7fb1e3e00640 1 -- 172.21.15.6:0/3560682274 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1ec1530f0 con 0x7fb1ec0af580 2024-03-04T09:34:51.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:51.422+0000 7fb1e3e00640 1 --2- 172.21.15.6:0/3560682274 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb1ec0af580 0x7fb1ec14c270 secure :-1 s=READY pgs=1664 cs=0 l=1 rev1=1 crypto rx=0x7fb1d400c470 tx=0x7fb1d4001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:34:51.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:51.422+0000 7fb1e8c00640 1 -- 172.21.15.6:0/3560682274 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1d40052d0 con 0x7fb1ec0af580 2024-03-04T09:34:51.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:51.422+0000 7fb1e8c00640 1 -- 172.21.15.6:0/3560682274 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1d4005470 con 0x7fb1ec0af580 2024-03-04T09:34:51.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:51.422+0000 7fb1e8c00640 1 -- 172.21.15.6:0/3560682274 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1d4007260 con 0x7fb1ec0af580 2024-03-04T09:34:51.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:51.422+0000 7fb1f2a00640 1 -- 172.21.15.6:0/3560682274 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1ec09ad40 con 0x7fb1ec0af580 2024-03-04T09:34:51.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:51.422+0000 7fb1f2a00640 1 -- 172.21.15.6:0/3560682274 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1ec09b300 con 0x7fb1ec0af580 2024-03-04T09:34:51.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:51.423+0000 7fb1f2a00640 1 -- 172.21.15.6:0/3560682274 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1ec069510 con 0x7fb1ec0af580 2024-03-04T09:34:51.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:51.424+0000 7fb1e8c00640 1 -- 172.21.15.6:0/3560682274 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb1d4007400 con 0x7fb1ec0af580 2024-03-04T09:34:51.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:51.426+0000 7fb1e8c00640 1 --2- 172.21.15.6:0/3560682274 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb1c806b500 0x7fb1c806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:51.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:51.426+0000 7fb1e8c00640 1 -- 172.21.15.6:0/3560682274 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fb1d40897f0 con 0x7fb1ec0af580 2024-03-04T09:34:51.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:51.427+0000 7fb1ebe00640 1 --2- 172.21.15.6:0/3560682274 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb1c806b500 0x7fb1c806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:51.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:51.428+0000 7fb1ebe00640 1 --2- 172.21.15.6:0/3560682274 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb1c806b500 0x7fb1c806d9c0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fb1dc006fd0 tx=0x7fb1dc008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:34:51.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:51.431+0000 7fb1e8c00640 1 -- 172.21.15.6:0/3560682274 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb1ec1530f0 con 0x7fb1ec0af580 2024-03-04T09:34:51.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:51.667+0000 7fb1f2a00640 1 -- 172.21.15.6:0/3560682274 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb1ec04f3a0 con 0x7fb1ec0af580 2024-03-04T09:34:51.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:51.668+0000 7fb1e8c00640 1 -- 172.21.15.6:0/3560682274 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3465 (secure 0 0 0) 0x7fb1d4058590 con 0x7fb1ec0af580 2024-03-04T09:34:51.665 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:34:51.665 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:33:54.652601+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"laggy_since":"2024-03-04T09:33:54.652601+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":9}]} 2024-03-04T09:34:51.665 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 100 2024-03-04T09:34:51.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:51.671+0000 7fb1f2a00640 1 -- 172.21.15.6:0/3560682274 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb1c806b500 msgr2=0x7fb1c806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:51.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:51.671+0000 7fb1f2a00640 1 --2- 172.21.15.6:0/3560682274 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb1c806b500 0x7fb1c806d9c0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fb1dc006fd0 tx=0x7fb1dc008040 comp rx=0 tx=0).stop 2024-03-04T09:34:51.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:51.671+0000 7fb1f2a00640 1 -- 172.21.15.6:0/3560682274 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb1ec0af580 msgr2=0x7fb1ec14c270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:51.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:51.671+0000 7fb1f2a00640 1 --2- 172.21.15.6:0/3560682274 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb1ec0af580 0x7fb1ec14c270 secure :-1 s=READY pgs=1664 cs=0 l=1 rev1=1 crypto rx=0x7fb1d400c470 tx=0x7fb1d4001e70 comp rx=0 tx=0).stop 2024-03-04T09:34:51.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:51.671+0000 7fb1f2a00640 1 -- 172.21.15.6:0/3560682274 shutdown_connections 2024-03-04T09:34:51.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:51.671+0000 7fb1f2a00640 1 --2- 172.21.15.6:0/3560682274 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb1ec0cdc20 0x7fb1ec0b0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:51.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:51.671+0000 7fb1f2a00640 1 --2- 172.21.15.6:0/3560682274 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb1ec0af580 0x7fb1ec14c270 unknown :-1 s=CLOSED pgs=1664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:51.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:51.671+0000 7fb1f2a00640 1 --2- 172.21.15.6:0/3560682274 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb1c806b500 0x7fb1c806d9c0 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:51.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:51.671+0000 7fb1f2a00640 1 --2- 172.21.15.6:0/3560682274 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb1ec0af1a0 0x7fb1ec152bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:51.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:51.671+0000 7fb1f2a00640 1 -- 172.21.15.6:0/3560682274 >> 172.21.15.6:0/3560682274 conn(0x7fb1ec06afb0 msgr2=0x7fb1ec15c0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:34:51.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:51.671+0000 7fb1f2a00640 1 -- 172.21.15.6:0/3560682274 shutdown_connections 2024-03-04T09:34:51.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:51.671+0000 7fb1f2a00640 1 -- 172.21.15.6:0/3560682274 wait complete. 2024-03-04T09:34:51.856 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-04T09:34:51.887 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:32:57.159681+0000', 'modified': '2024-03-04T09:33:54.652601+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8214, 'mds_1': 6796}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6796': {'gid': 6796, 'name': 'd', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.146:6837/2458070840', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 2458070840}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 2458070840}]}, 'join_fscid': -1, 'laggy_since': '2024-03-04T09:33:54.652601+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8214': {'gid': 8214, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2446627517', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2446627517}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2446627517}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8214, 'qdb_cluster': [8214]} 2024-03-04T09:34:51.887 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6796:up:stopping 2024-03-04T09:34:52.889 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:34:53.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:53.110+0000 7fed6be00640 1 Processor -- start 2024-03-04T09:34:53.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:53.110+0000 7fed6be00640 1 -- start start 2024-03-04T09:34:53.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:53.111+0000 7fed6be00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fed6c070f10 0x7fed6c0cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:53.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:53.111+0000 7fed6be00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fed6c0cce90 0x7fed6c15c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:53.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:53.111+0000 7fed6be00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed6c0714d0 con 0x7fed6c145300 2024-03-04T09:34:53.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:53.111+0000 7fed6be00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed6c0cd300 con 0x7fed6c070f10 2024-03-04T09:34:53.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:53.111+0000 7fed6be00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed6c0717b0 con 0x7fed6c0cce90 2024-03-04T09:34:53.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:53.111+0000 7fed6a000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fed6c070f10 0x7fed6c0cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:53.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:53.112+0000 7fed6a000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fed6c070f10 0x7fed6c0cc950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:40402/0 (socket says 172.21.15.6:40402) 2024-03-04T09:34:53.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:53.112+0000 7fed6a000640 1 -- 172.21.15.6:0/3053136282 learned_addr learned my addr 172.21.15.6:0/3053136282 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:34:53.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:53.112+0000 7fed6b400640 1 --2- 172.21.15.6:0/3053136282 >> v2:172.21.15.146:3300/0 conn(0x7fed6c0cce90 0x7fed6c15c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:53.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:53.112+0000 7fed6a000640 1 -- 172.21.15.6:0/3053136282 >> v2:172.21.15.146:3300/0 conn(0x7fed6c0cce90 msgr2=0x7fed6c15c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:34:53.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:53.112+0000 7fed6a000640 1 --2- 172.21.15.6:0/3053136282 >> v2:172.21.15.146:3300/0 conn(0x7fed6c0cce90 0x7fed6c15c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:53.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:53.112+0000 7fed6a000640 1 -- 172.21.15.6:0/3053136282 >> v1:172.21.15.146:6789/0 conn(0x7fed6c145300 legacy=0x7fed6c0b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:34:53.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:53.112+0000 7fed6a000640 1 -- 172.21.15.6:0/3053136282 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed6c071a90 con 0x7fed6c070f10 2024-03-04T09:34:53.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:53.113+0000 7fed6a000640 1 --2- 172.21.15.6:0/3053136282 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fed6c070f10 0x7fed6c0cc950 secure :-1 s=READY pgs=1665 cs=0 l=1 rev1=1 crypto rx=0x7fed6c0cc500 tx=0x7fed5802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e46a75e3811d3ab2 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:34:53.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:53.114+0000 7fed69600640 1 -- 172.21.15.6:0/3053136282 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed5802e070 con 0x7fed6c070f10 2024-03-04T09:34:53.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:53.114+0000 7fed69600640 1 -- 172.21.15.6:0/3053136282 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed58037070 con 0x7fed6c070f10 2024-03-04T09:34:53.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:53.114+0000 7fed69600640 1 -- 172.21.15.6:0/3053136282 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed58031a00 con 0x7fed6c070f10 2024-03-04T09:34:53.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:53.114+0000 7fed6be00640 1 -- 172.21.15.6:0/3053136282 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fed6c070f10 msgr2=0x7fed6c0cc950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:53.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:53.114+0000 7fed6be00640 1 --2- 172.21.15.6:0/3053136282 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fed6c070f10 0x7fed6c0cc950 secure :-1 s=READY pgs=1665 cs=0 l=1 rev1=1 crypto rx=0x7fed6c0cc500 tx=0x7fed5802b880 comp rx=0 tx=0).stop 2024-03-04T09:34:53.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:53.114+0000 7fed6be00640 1 -- 172.21.15.6:0/3053136282 shutdown_connections 2024-03-04T09:34:53.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:53.114+0000 7fed6be00640 1 --2- 172.21.15.6:0/3053136282 >> v2:172.21.15.146:3300/0 conn(0x7fed6c0cce90 0x7fed6c15c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:53.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:53.114+0000 7fed6be00640 1 --2- 172.21.15.6:0/3053136282 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fed6c070f10 0x7fed6c0cc950 unknown :-1 s=CLOSED pgs=1665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:53.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:53.114+0000 7fed6be00640 1 -- 172.21.15.6:0/3053136282 >> 172.21.15.6:0/3053136282 conn(0x7fed6c06b3d0 msgr2=0x7fed6c06b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:34:53.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:53.114+0000 7fed6be00640 1 -- 172.21.15.6:0/3053136282 shutdown_connections 2024-03-04T09:34:53.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:53.114+0000 7fed6be00640 1 -- 172.21.15.6:0/3053136282 wait complete. 2024-03-04T09:34:53.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:53.116+0000 7fed6be00640 1 Processor -- start 2024-03-04T09:34:53.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:53.116+0000 7fed6be00640 1 -- start start 2024-03-04T09:34:53.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:53.117+0000 7fed6be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fed6c070f10 0x7fed6c1e9840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:53.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:53.117+0000 7fed6be00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fed6c0cce90 0x7fed6c15ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:53.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:53.117+0000 7fed6be00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fed6c145300 0x7fed6c15b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:53.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:53.117+0000 7fed6be00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed6c15cab0 con 0x7fed6c070f10 2024-03-04T09:34:53.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:53.117+0000 7fed6be00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed6c1456e0 con 0x7fed6c0cce90 2024-03-04T09:34:53.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:53.117+0000 7fed6be00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed6c15cc30 con 0x7fed6c145300 2024-03-04T09:34:53.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:53.118+0000 7fed6aa00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fed6c070f10 0x7fed6c1e9840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:53.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:53.118+0000 7fed6aa00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fed6c070f10 0x7fed6c1e9840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:40408/0 (socket says 172.21.15.6:40408) 2024-03-04T09:34:53.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:53.118+0000 7fed6aa00640 1 -- 172.21.15.6:0/4292158311 learned_addr learned my addr 172.21.15.6:0/4292158311 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:34:53.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:53.118+0000 7fed6a000640 1 --2- 172.21.15.6:0/4292158311 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fed6c0cce90 0x7fed6c15ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:53.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:53.118+0000 7fed6b400640 1 --2- 172.21.15.6:0/4292158311 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fed6c145300 0x7fed6c15b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:53.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:53.118+0000 7fed6aa00640 1 -- 172.21.15.6:0/4292158311 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fed6c145300 msgr2=0x7fed6c15b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:53.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:53.118+0000 7fed6aa00640 1 --2- 172.21.15.6:0/4292158311 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fed6c145300 0x7fed6c15b3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:53.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:53.118+0000 7fed6aa00640 1 -- 172.21.15.6:0/4292158311 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fed6c0cce90 msgr2=0x7fed6c15ae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:53.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:53.118+0000 7fed6aa00640 1 --2- 172.21.15.6:0/4292158311 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fed6c0cce90 0x7fed6c15ae90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:53.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:53.118+0000 7fed6aa00640 1 -- 172.21.15.6:0/4292158311 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed6c1e6e10 con 0x7fed6c070f10 2024-03-04T09:34:53.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:53.118+0000 7fed6aa00640 1 --2- 172.21.15.6:0/4292158311 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fed6c070f10 0x7fed6c1e9840 secure :-1 s=READY pgs=1666 cs=0 l=1 rev1=1 crypto rx=0x7fed5c00ba80 tx=0x7fed5c00bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:34:53.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:53.119+0000 7fed62a00640 1 -- 172.21.15.6:0/4292158311 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed5c019070 con 0x7fed6c070f10 2024-03-04T09:34:53.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:53.119+0000 7fed62a00640 1 -- 172.21.15.6:0/4292158311 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed5c00cc90 con 0x7fed6c070f10 2024-03-04T09:34:53.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:53.119+0000 7fed62a00640 1 -- 172.21.15.6:0/4292158311 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed5c014680 con 0x7fed6c070f10 2024-03-04T09:34:53.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:53.119+0000 7fed6be00640 1 -- 172.21.15.6:0/4292158311 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed6c1e70f0 con 0x7fed6c070f10 2024-03-04T09:34:53.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:53.119+0000 7fed6be00640 1 -- 172.21.15.6:0/4292158311 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed6c1e76b0 con 0x7fed6c070f10 2024-03-04T09:34:53.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:53.120+0000 7fed6be00640 1 -- 172.21.15.6:0/4292158311 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed6c0b2b10 con 0x7fed6c070f10 2024-03-04T09:34:53.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:53.121+0000 7fed62a00640 1 -- 172.21.15.6:0/4292158311 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fed5c01c070 con 0x7fed6c070f10 2024-03-04T09:34:53.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:53.122+0000 7fed62a00640 1 --2- 172.21.15.6:0/4292158311 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fed4c06b4d0 0x7fed4c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:53.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:53.122+0000 7fed62a00640 1 -- 172.21.15.6:0/4292158311 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fed5c08a120 con 0x7fed6c070f10 2024-03-04T09:34:53.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:53.123+0000 7fed6a000640 1 --2- 172.21.15.6:0/4292158311 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fed4c06b4d0 0x7fed4c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:53.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:53.124+0000 7fed6a000640 1 --2- 172.21.15.6:0/4292158311 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fed4c06b4d0 0x7fed4c06d990 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7fed58036000 tx=0x7fed5803d040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:34:53.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:53.127+0000 7fed62a00640 1 -- 172.21.15.6:0/4292158311 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fed6c0b2b10 con 0x7fed6c070f10 2024-03-04T09:34:53.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:53.367+0000 7fed6be00640 1 -- 172.21.15.6:0/4292158311 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fed6c04f330 con 0x7fed6c070f10 2024-03-04T09:34:53.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:53.368+0000 7fed62a00640 1 -- 172.21.15.6:0/4292158311 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3465 (secure 0 0 0) 0x7fed5c058ec0 con 0x7fed6c070f10 2024-03-04T09:34:53.365 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:34:53.365 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:33:54.652601+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"laggy_since":"2024-03-04T09:33:54.652601+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":9}]} 2024-03-04T09:34:53.365 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 100 2024-03-04T09:34:53.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:53.370+0000 7fed6be00640 1 -- 172.21.15.6:0/4292158311 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fed4c06b4d0 msgr2=0x7fed4c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:53.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:53.371+0000 7fed6be00640 1 --2- 172.21.15.6:0/4292158311 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fed4c06b4d0 0x7fed4c06d990 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7fed58036000 tx=0x7fed5803d040 comp rx=0 tx=0).stop 2024-03-04T09:34:53.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:53.371+0000 7fed6be00640 1 -- 172.21.15.6:0/4292158311 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fed6c070f10 msgr2=0x7fed6c1e9840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:53.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:53.371+0000 7fed6be00640 1 --2- 172.21.15.6:0/4292158311 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fed6c070f10 0x7fed6c1e9840 secure :-1 s=READY pgs=1666 cs=0 l=1 rev1=1 crypto rx=0x7fed5c00ba80 tx=0x7fed5c00bf40 comp rx=0 tx=0).stop 2024-03-04T09:34:53.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:53.371+0000 7fed6be00640 1 -- 172.21.15.6:0/4292158311 shutdown_connections 2024-03-04T09:34:53.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:53.371+0000 7fed6be00640 1 --2- 172.21.15.6:0/4292158311 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fed4c06b4d0 0x7fed4c06d990 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:53.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:53.371+0000 7fed6be00640 1 --2- 172.21.15.6:0/4292158311 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fed6c145300 0x7fed6c15b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:53.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:53.371+0000 7fed6be00640 1 --2- 172.21.15.6:0/4292158311 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fed6c0cce90 0x7fed6c15ae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:53.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:53.371+0000 7fed6be00640 1 --2- 172.21.15.6:0/4292158311 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fed6c070f10 0x7fed6c1e9840 unknown :-1 s=CLOSED pgs=1666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:53.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:53.371+0000 7fed6be00640 1 -- 172.21.15.6:0/4292158311 >> 172.21.15.6:0/4292158311 conn(0x7fed6c06b3d0 msgr2=0x7fed6c0cd930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:34:53.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:53.371+0000 7fed6be00640 1 -- 172.21.15.6:0/4292158311 shutdown_connections 2024-03-04T09:34:53.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:53.371+0000 7fed6be00640 1 -- 172.21.15.6:0/4292158311 wait complete. 2024-03-04T09:34:53.589 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:32:57.159681+0000', 'modified': '2024-03-04T09:33:54.652601+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8214, 'mds_1': 6796}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6796': {'gid': 6796, 'name': 'd', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.146:6837/2458070840', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 2458070840}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 2458070840}]}, 'join_fscid': -1, 'laggy_since': '2024-03-04T09:33:54.652601+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8214': {'gid': 8214, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2446627517', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2446627517}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2446627517}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8214, 'qdb_cluster': [8214]} 2024-03-04T09:34:53.589 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6796:up:stopping 2024-03-04T09:34:53.858 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-04T09:34:54.590 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:34:54.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:54.813+0000 7f2ed8a00640 1 Processor -- start 2024-03-04T09:34:54.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:54.813+0000 7f2ed8a00640 1 -- start start 2024-03-04T09:34:54.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:54.815+0000 7f2ed8a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f2ed40d0e50 0x7f2ed4168c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:54.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:54.815+0000 7f2ed8a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ed40b3710 con 0x7f2ed40d4a10 2024-03-04T09:34:54.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:54.815+0000 7f2ed8a00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ed40b39f0 con 0x7f2ed415e0b0 2024-03-04T09:34:54.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:54.815+0000 7f2ed8a00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ed40b09e0 con 0x7f2ed40d0e50 2024-03-04T09:34:54.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:54.815+0000 7f2ed3200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f2ed40d0e50 0x7f2ed4168c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:54.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:54.815+0000 7f2ed3200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f2ed40d0e50 0x7f2ed4168c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:40422/0 (socket says 172.21.15.6:40422) 2024-03-04T09:34:54.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:54.815+0000 7f2ed3200640 1 -- 172.21.15.6:0/3884790272 learned_addr learned my addr 172.21.15.6:0/3884790272 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:34:54.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:54.815+0000 7f2ec9e00640 1 --1- 172.21.15.6:0/3884790272 >> v1:172.21.15.6:6789/0 conn(0x7f2ed40d4a10 0x7f2ed40d0740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:45976/0 (socket says 172.21.15.6:45976) 2024-03-04T09:34:54.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:54.816+0000 7f2ed3200640 1 -- 172.21.15.6:0/3884790272 >> v1:172.21.15.146:6789/0 conn(0x7f2ed415e0b0 legacy=0x7f2ed40cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:34:54.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:54.816+0000 7f2ed3200640 1 -- 172.21.15.6:0/3884790272 >> v1:172.21.15.6:6789/0 conn(0x7f2ed40d4a10 legacy=0x7f2ed40d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:54.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:54.816+0000 7f2ed3200640 1 -- 172.21.15.6:0/3884790272 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ed40b3cd0 con 0x7f2ed40d0e50 2024-03-04T09:34:54.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:54.817+0000 7f2ed3200640 1 --2- 172.21.15.6:0/3884790272 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2ed40d0e50 0x7f2ed4168c50 secure :-1 s=READY pgs=1668 cs=0 l=1 rev1=1 crypto rx=0x7f2ecc009e50 tx=0x7f2ecc02bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35bb8f9782b2c2b5 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:34:54.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:54.817+0000 7f2ed1e00640 1 -- 172.21.15.6:0/3884790272 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ecc02cb10 con 0x7f2ed40d0e50 2024-03-04T09:34:54.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:54.817+0000 7f2ed1e00640 1 -- 172.21.15.6:0/3884790272 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ecc02ccb0 con 0x7f2ed40d0e50 2024-03-04T09:34:54.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:54.817+0000 7f2ed1e00640 1 -- 172.21.15.6:0/3884790272 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ecc02ce60 con 0x7f2ed40d0e50 2024-03-04T09:34:54.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:54.818+0000 7f2ed8a00640 1 -- 172.21.15.6:0/3884790272 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2ed40d0e50 msgr2=0x7f2ed4168c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:54.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:54.818+0000 7f2ed8a00640 1 --2- 172.21.15.6:0/3884790272 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2ed40d0e50 0x7f2ed4168c50 secure :-1 s=READY pgs=1668 cs=0 l=1 rev1=1 crypto rx=0x7f2ecc009e50 tx=0x7f2ecc02bf70 comp rx=0 tx=0).stop 2024-03-04T09:34:54.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:54.818+0000 7f2ed8a00640 1 -- 172.21.15.6:0/3884790272 shutdown_connections 2024-03-04T09:34:54.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:54.818+0000 7f2ed8a00640 1 --2- 172.21.15.6:0/3884790272 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2ed40d0e50 0x7f2ed4168c50 unknown :-1 s=CLOSED pgs=1668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:54.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:54.818+0000 7f2ed8a00640 1 -- 172.21.15.6:0/3884790272 >> 172.21.15.6:0/3884790272 conn(0x7f2ed40d3d30 msgr2=0x7f2ed40d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:34:54.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:54.818+0000 7f2ed8a00640 1 -- 172.21.15.6:0/3884790272 shutdown_connections 2024-03-04T09:34:54.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:54.818+0000 7f2ed8a00640 1 -- 172.21.15.6:0/3884790272 wait complete. 2024-03-04T09:34:54.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:54.820+0000 7f2ed8a00640 1 Processor -- start 2024-03-04T09:34:54.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:54.820+0000 7f2ed8a00640 1 -- start start 2024-03-04T09:34:54.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:54.821+0000 7f2ed8a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2ed40d0e50 0x7f2ed409d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:54.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:54.821+0000 7f2ed8a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2ed40d4a10 0x7f2ed415c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:54.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:54.821+0000 7f2ed8a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2ed415e0b0 0x7f2ed409f430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:54.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:54.821+0000 7f2ed8a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ed415e490 con 0x7f2ed40d4a10 2024-03-04T09:34:54.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:54.821+0000 7f2ed8a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ed40cfa20 con 0x7f2ed415e0b0 2024-03-04T09:34:54.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:54.821+0000 7f2ed8a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ed40cfc60 con 0x7f2ed40d0e50 2024-03-04T09:34:54.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:54.821+0000 7f2ec9e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2ed40d4a10 0x7f2ed415c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:54.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:54.821+0000 7f2ec9e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2ed40d4a10 0x7f2ed415c7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:40428/0 (socket says 172.21.15.6:40428) 2024-03-04T09:34:54.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:54.821+0000 7f2ec9e00640 1 -- 172.21.15.6:0/598438918 learned_addr learned my addr 172.21.15.6:0/598438918 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:34:54.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:54.822+0000 7f2ed2800640 1 --2- 172.21.15.6:0/598438918 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2ed40d0e50 0x7f2ed409d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:54.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:54.822+0000 7f2ed3200640 1 --2- 172.21.15.6:0/598438918 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2ed415e0b0 0x7f2ed409f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:54.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:54.822+0000 7f2ec9e00640 1 -- 172.21.15.6:0/598438918 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2ed40d0e50 msgr2=0x7f2ed409d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:54.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:54.822+0000 7f2ec9e00640 1 --2- 172.21.15.6:0/598438918 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2ed40d0e50 0x7f2ed409d9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:54.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:54.822+0000 7f2ec9e00640 1 -- 172.21.15.6:0/598438918 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2ed415e0b0 msgr2=0x7f2ed409f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:54.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:54.822+0000 7f2ec9e00640 1 --2- 172.21.15.6:0/598438918 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2ed415e0b0 0x7f2ed409f430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:54.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:54.822+0000 7f2ec9e00640 1 -- 172.21.15.6:0/598438918 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ed40b7af0 con 0x7f2ed40d4a10 2024-03-04T09:34:54.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:54.822+0000 7f2ec9e00640 1 --2- 172.21.15.6:0/598438918 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2ed40d4a10 0x7f2ed415c7a0 secure :-1 s=READY pgs=1669 cs=0 l=1 rev1=1 crypto rx=0x7f2ebc00c440 tx=0x7f2ebc001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:34:54.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:54.822+0000 7f2ecb400640 1 -- 172.21.15.6:0/598438918 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ebc0074e0 con 0x7f2ed40d4a10 2024-03-04T09:34:54.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:54.822+0000 7f2ecb400640 1 -- 172.21.15.6:0/598438918 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ebc007680 con 0x7f2ed40d4a10 2024-03-04T09:34:54.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:54.822+0000 7f2ecb400640 1 -- 172.21.15.6:0/598438918 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ebc004a20 con 0x7f2ed40d4a10 2024-03-04T09:34:54.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:54.822+0000 7f2ed8a00640 1 -- 172.21.15.6:0/598438918 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ed40b7dd0 con 0x7f2ed40d4a10 2024-03-04T09:34:54.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:54.823+0000 7f2ed8a00640 1 -- 172.21.15.6:0/598438918 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ed40b8390 con 0x7f2ed40d4a10 2024-03-04T09:34:54.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:54.824+0000 7f2ed8a00640 1 -- 172.21.15.6:0/598438918 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ed404f330 con 0x7f2ed40d4a10 2024-03-04T09:34:54.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:54.825+0000 7f2ecb400640 1 -- 172.21.15.6:0/598438918 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2ebc01a020 con 0x7f2ed40d4a10 2024-03-04T09:34:54.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:54.825+0000 7f2ecb400640 1 --2- 172.21.15.6:0/598438918 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2eb006b4d0 0x7f2eb006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:54.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:54.826+0000 7f2ecb400640 1 -- 172.21.15.6:0/598438918 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f2ebc089a60 con 0x7f2ed40d4a10 2024-03-04T09:34:54.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:54.826+0000 7f2ed2800640 1 --2- 172.21.15.6:0/598438918 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2eb006b4d0 0x7f2eb006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:54.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:54.827+0000 7f2ed2800640 1 --2- 172.21.15.6:0/598438918 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2eb006b4d0 0x7f2eb006d990 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f2ec4006fd0 tx=0x7f2ec4008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:34:54.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:54.831+0000 7f2ecb400640 1 -- 172.21.15.6:0/598438918 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2ebc010300 con 0x7f2ed40d4a10 2024-03-04T09:34:55.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:55.125+0000 7f2ed8a00640 1 -- 172.21.15.6:0/598438918 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2ed40b0e50 con 0x7f2ed40d4a10 2024-03-04T09:34:55.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:55.125+0000 7f2ecb400640 1 -- 172.21.15.6:0/598438918 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3465 (secure 0 0 0) 0x7f2ebc058890 con 0x7f2ed40d4a10 2024-03-04T09:34:55.123 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:34:55.123 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:33:54.652601+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"laggy_since":"2024-03-04T09:33:54.652601+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":9}]} 2024-03-04T09:34:55.123 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 100 2024-03-04T09:34:55.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:55.128+0000 7f2ed8a00640 1 -- 172.21.15.6:0/598438918 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2eb006b4d0 msgr2=0x7f2eb006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:55.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:55.128+0000 7f2ed8a00640 1 --2- 172.21.15.6:0/598438918 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2eb006b4d0 0x7f2eb006d990 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f2ec4006fd0 tx=0x7f2ec4008040 comp rx=0 tx=0).stop 2024-03-04T09:34:55.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:55.128+0000 7f2ed8a00640 1 -- 172.21.15.6:0/598438918 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2ed40d4a10 msgr2=0x7f2ed415c7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:55.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:55.128+0000 7f2ed8a00640 1 --2- 172.21.15.6:0/598438918 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2ed40d4a10 0x7f2ed415c7a0 secure :-1 s=READY pgs=1669 cs=0 l=1 rev1=1 crypto rx=0x7f2ebc00c440 tx=0x7f2ebc001e70 comp rx=0 tx=0).stop 2024-03-04T09:34:55.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:55.128+0000 7f2ed8a00640 1 -- 172.21.15.6:0/598438918 shutdown_connections 2024-03-04T09:34:55.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:55.128+0000 7f2ed8a00640 1 --2- 172.21.15.6:0/598438918 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2ed415e0b0 0x7f2ed409f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:55.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:55.128+0000 7f2ed8a00640 1 --2- 172.21.15.6:0/598438918 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2ed40d4a10 0x7f2ed415c7a0 unknown :-1 s=CLOSED pgs=1669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:55.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:55.128+0000 7f2ed8a00640 1 --2- 172.21.15.6:0/598438918 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2eb006b4d0 0x7f2eb006d990 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:55.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:55.128+0000 7f2ed8a00640 1 --2- 172.21.15.6:0/598438918 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2ed40d0e50 0x7f2ed409d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:55.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:55.128+0000 7f2ed8a00640 1 -- 172.21.15.6:0/598438918 >> 172.21.15.6:0/598438918 conn(0x7f2ed40d3d30 msgr2=0x7f2ed40b1150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:34:55.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:55.128+0000 7f2ed8a00640 1 -- 172.21.15.6:0/598438918 shutdown_connections 2024-03-04T09:34:55.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:55.128+0000 7f2ed8a00640 1 -- 172.21.15.6:0/598438918 wait complete. 2024-03-04T09:34:55.344 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:32:57.159681+0000', 'modified': '2024-03-04T09:33:54.652601+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8214, 'mds_1': 6796}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6796': {'gid': 6796, 'name': 'd', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.146:6837/2458070840', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 2458070840}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 2458070840}]}, 'join_fscid': -1, 'laggy_since': '2024-03-04T09:33:54.652601+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8214': {'gid': 8214, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2446627517', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2446627517}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2446627517}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8214, 'qdb_cluster': [8214]} 2024-03-04T09:34:55.344 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6796:up:stopping 2024-03-04T09:34:55.858 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-04T09:34:56.345 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:34:56.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:56.574+0000 7f3e1ec00640 1 Processor -- start 2024-03-04T09:34:56.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:56.574+0000 7f3e1ec00640 1 -- start start 2024-03-04T09:34:56.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:56.576+0000 7f3e1ec00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3e180b34e0 0x7f3e1815c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:56.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:56.576+0000 7f3e1ec00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e1815cc40 con 0x7f3e180af1a0 2024-03-04T09:34:56.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:56.576+0000 7f3e1ec00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e1815cf20 con 0x7f3e180af580 2024-03-04T09:34:56.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:56.576+0000 7f3e1ec00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e1815d200 con 0x7f3e180b34e0 2024-03-04T09:34:56.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:56.576+0000 7f3e17e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f3e180af1a0 0x7f3e180b4c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:45990/0 (socket says 172.21.15.6:45990) 2024-03-04T09:34:56.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:56.576+0000 7f3e17e00640 1 -- 172.21.15.6:0/1584066667 learned_addr learned my addr 172.21.15.6:0/1584066667 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:34:56.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:56.577+0000 7f3e1cc00640 1 --2- 172.21.15.6:0/1584066667 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3e180b34e0 0x7f3e1815c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:56.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:56.577+0000 7f3e16a00640 1 -- 172.21.15.6:0/1584066667 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3325343869 0 0) 0x7f3e1815cc40 con 0x7f3e180af1a0 2024-03-04T09:34:56.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:56.577+0000 7f3e16a00640 1 -- 172.21.15.6:0/1584066667 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3dfc003520 con 0x7f3e180af1a0 2024-03-04T09:34:56.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:56.577+0000 7f3e16a00640 1 -- 172.21.15.6:0/1584066667 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4092353629 0 0) 0x7f3dfc003520 con 0x7f3e180af1a0 2024-03-04T09:34:56.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:56.577+0000 7f3e16a00640 1 -- 172.21.15.6:0/1584066667 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3e180b34e0 msgr2=0x7f3e1815c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:34:56.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:56.577+0000 7f3e16a00640 1 --2- 172.21.15.6:0/1584066667 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3e180b34e0 0x7f3e1815c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:56.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:56.577+0000 7f3e16a00640 1 -- 172.21.15.6:0/1584066667 >> v1:172.21.15.146:6789/0 conn(0x7f3e180af580 legacy=0x7f3e180b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:56.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:56.577+0000 7f3e16a00640 1 -- 172.21.15.6:0/1584066667 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e1815d4e0 con 0x7f3e180af1a0 2024-03-04T09:34:56.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:56.577+0000 7f3e16a00640 1 -- 172.21.15.6:0/1584066667 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7f3e080019f0 con 0x7f3e180af1a0 2024-03-04T09:34:56.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:56.577+0000 7f3e16a00640 1 -- 172.21.15.6:0/1584066667 >> v1:172.21.15.6:6789/0 conn(0x7f3e180af1a0 legacy=0x7f3e180b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:56.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:56.577+0000 7f3e16a00640 1 --2- 172.21.15.6:0/1584066667 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3dfc003a50 0x7f3dfc003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:56.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:56.578+0000 7f3e16a00640 1 --2- 172.21.15.6:0/1584066667 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3e180b34e0 0x7f3dfc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:56.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:56.578+0000 7f3e16a00640 1 --2- 172.21.15.6:0/1584066667 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3dfc004d40 0x7f3dfc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:56.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:56.578+0000 7f3e16a00640 1 -- 172.21.15.6:0/1584066667 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e1815d200 con 0x7f3dfc003a50 2024-03-04T09:34:56.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:56.578+0000 7f3e16a00640 1 -- 172.21.15.6:0/1584066667 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dfc00bbd0 con 0x7f3dfc004d40 2024-03-04T09:34:56.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:56.578+0000 7f3e16a00640 1 -- 172.21.15.6:0/1584066667 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3dfc00bdd0 con 0x7f3e180b34e0 2024-03-04T09:34:56.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:56.578+0000 7f3e17e00640 1 --2- 172.21.15.6:0/1584066667 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3dfc003a50 0x7f3dfc003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:56.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:56.578+0000 7f3e1cc00640 1 --2- 172.21.15.6:0/1584066667 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3dfc004d40 0x7f3dfc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:56.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:56.578+0000 7f3e17400640 1 --2- 172.21.15.6:0/1584066667 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3e180b34e0 0x7f3dfc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:56.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:56.578+0000 7f3e17e00640 1 -- 172.21.15.6:0/1584066667 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3e180b34e0 msgr2=0x7f3dfc004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:34:56.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:56.578+0000 7f3e17e00640 1 --2- 172.21.15.6:0/1584066667 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3e180b34e0 0x7f3dfc004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:56.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:56.578+0000 7f3e17e00640 1 -- 172.21.15.6:0/1584066667 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3dfc004d40 msgr2=0x7f3dfc0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:34:56.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:56.578+0000 7f3e17e00640 1 --2- 172.21.15.6:0/1584066667 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3dfc004d40 0x7f3dfc0051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:56.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:56.578+0000 7f3e17e00640 1 -- 172.21.15.6:0/1584066667 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dfc003520 con 0x7f3dfc003a50 2024-03-04T09:34:56.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:56.578+0000 7f3e17400640 1 --2- 172.21.15.6:0/1584066667 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3e180b34e0 0x7f3dfc004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:34:56.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:56.580+0000 7f3e17e00640 1 --2- 172.21.15.6:0/1584066667 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3dfc003a50 0x7f3dfc003e70 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7f3e0800a7b0 tx=0x7f3dfc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e634737c6245941c server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:34:56.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:56.580+0000 7f3e16a00640 1 -- 172.21.15.6:0/1584066667 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e0800a3d0 con 0x7f3dfc003a50 2024-03-04T09:34:56.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:56.580+0000 7f3e16a00640 1 -- 172.21.15.6:0/1584066667 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e08034630 con 0x7f3dfc003a50 2024-03-04T09:34:56.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:56.580+0000 7f3e16a00640 1 -- 172.21.15.6:0/1584066667 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e08034a60 con 0x7f3dfc003a50 2024-03-04T09:34:56.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:56.581+0000 7f3e1ec00640 1 -- 172.21.15.6:0/1584066667 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3dfc003a50 msgr2=0x7f3dfc003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:56.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:56.581+0000 7f3e1ec00640 1 --2- 172.21.15.6:0/1584066667 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3dfc003a50 0x7f3dfc003e70 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7f3e0800a7b0 tx=0x7f3dfc005830 comp rx=0 tx=0).stop 2024-03-04T09:34:56.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:56.581+0000 7f3e1cc00640 1 -- 172.21.15.6:0/1584066667 reap_dead start 2024-03-04T09:34:56.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:56.581+0000 7f3e1ec00640 1 -- 172.21.15.6:0/1584066667 shutdown_connections 2024-03-04T09:34:56.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:56.581+0000 7f3e1ec00640 1 -- 172.21.15.6:0/1584066667 >> 172.21.15.6:0/1584066667 conn(0x7f3e1806afb0 msgr2=0x7f3e1806b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:34:56.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:56.581+0000 7f3e1ec00640 1 -- 172.21.15.6:0/1584066667 shutdown_connections 2024-03-04T09:34:56.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:56.581+0000 7f3e1ec00640 1 -- 172.21.15.6:0/1584066667 wait complete. 2024-03-04T09:34:56.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:56.584+0000 7f3e1ec00640 1 Processor -- start 2024-03-04T09:34:56.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:56.584+0000 7f3e1ec00640 1 -- start start 2024-03-04T09:34:56.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:56.586+0000 7f3e1ec00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3dfc003a50 0x7f3e18154900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:56.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:56.586+0000 7f3e1ec00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3e1806e6a0 0x7f3e1814c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:56.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:56.586+0000 7f3e1ec00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3e180c9150 0x7f3e1806cb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:56.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:56.586+0000 7f3e1ec00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e1815c600 con 0x7f3e180c9150 2024-03-04T09:34:56.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:56.586+0000 7f3e1ec00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e180cde70 con 0x7f3e1806e6a0 2024-03-04T09:34:56.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:56.586+0000 7f3e1ec00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e1815c780 con 0x7f3dfc003a50 2024-03-04T09:34:56.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:56.586+0000 7f3e1cc00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3e180c9150 0x7f3e1806cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:56.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:56.586+0000 7f3e1cc00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3e180c9150 0x7f3e1806cb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:40456/0 (socket says 172.21.15.6:40456) 2024-03-04T09:34:56.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:56.586+0000 7f3e1cc00640 1 -- 172.21.15.6:0/4058066393 learned_addr learned my addr 172.21.15.6:0/4058066393 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:34:56.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:56.587+0000 7f3e17e00640 1 --2- 172.21.15.6:0/4058066393 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3dfc003a50 0x7f3e18154900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:56.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:56.587+0000 7f3e17400640 1 --2- 172.21.15.6:0/4058066393 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3e1806e6a0 0x7f3e1814c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:56.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:56.587+0000 7f3e1cc00640 1 -- 172.21.15.6:0/4058066393 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3dfc003a50 msgr2=0x7f3e18154900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:56.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:56.587+0000 7f3e1cc00640 1 --2- 172.21.15.6:0/4058066393 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3dfc003a50 0x7f3e18154900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:56.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:56.587+0000 7f3e1cc00640 1 -- 172.21.15.6:0/4058066393 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3e1806e6a0 msgr2=0x7f3e1814c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:56.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:56.587+0000 7f3e1cc00640 1 --2- 172.21.15.6:0/4058066393 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3e1806e6a0 0x7f3e1814c8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:56.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:56.587+0000 7f3e1cc00640 1 -- 172.21.15.6:0/4058066393 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e18155320 con 0x7f3e180c9150 2024-03-04T09:34:56.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:56.587+0000 7f3e17400640 1 --2- 172.21.15.6:0/4058066393 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3e1806e6a0 0x7f3e1814c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:34:56.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:56.587+0000 7f3e1cc00640 1 --2- 172.21.15.6:0/4058066393 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3e180c9150 0x7f3e1806cb10 secure :-1 s=READY pgs=1672 cs=0 l=1 rev1=1 crypto rx=0x7f3e1809e100 tx=0x7f3e10010c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:34:56.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:56.587+0000 7f3e0fe00640 1 -- 172.21.15.6:0/4058066393 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e1000f040 con 0x7f3e180c9150 2024-03-04T09:34:56.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:56.587+0000 7f3e0fe00640 1 -- 172.21.15.6:0/4058066393 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e1001c070 con 0x7f3e180c9150 2024-03-04T09:34:56.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:56.587+0000 7f3e1ec00640 1 -- 172.21.15.6:0/4058066393 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e18155600 con 0x7f3e180c9150 2024-03-04T09:34:56.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:56.588+0000 7f3e0fe00640 1 -- 172.21.15.6:0/4058066393 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e1000eae0 con 0x7f3e180c9150 2024-03-04T09:34:56.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:56.588+0000 7f3e1ec00640 1 -- 172.21.15.6:0/4058066393 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e1815b2c0 con 0x7f3e180c9150 2024-03-04T09:34:56.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:56.589+0000 7f3e1ec00640 1 -- 172.21.15.6:0/4058066393 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e18069510 con 0x7f3e180c9150 2024-03-04T09:34:56.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:56.590+0000 7f3e0fe00640 1 -- 172.21.15.6:0/4058066393 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3e1000ec80 con 0x7f3e180c9150 2024-03-04T09:34:56.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:56.591+0000 7f3e0fe00640 1 --2- 172.21.15.6:0/4058066393 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f3df406b4d0 0x7f3df406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:56.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:56.591+0000 7f3e0fe00640 1 -- 172.21.15.6:0/4058066393 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f3e10089d70 con 0x7f3e180c9150 2024-03-04T09:34:56.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:56.592+0000 7f3e17e00640 1 --2- 172.21.15.6:0/4058066393 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f3df406b4d0 0x7f3df406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:56.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:56.593+0000 7f3e17e00640 1 --2- 172.21.15.6:0/4058066393 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f3df406b4d0 0x7f3df406d990 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f3e0800a780 tx=0x7f3e0802f000 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:34:56.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:56.595+0000 7f3e0fe00640 1 -- 172.21.15.6:0/4058066393 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3e10026020 con 0x7f3e180c9150 2024-03-04T09:34:56.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:56.837+0000 7f3e1ec00640 1 -- 172.21.15.6:0/4058066393 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3e1804f3a0 con 0x7f3e180c9150 2024-03-04T09:34:56.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:56.838+0000 7f3e0fe00640 1 -- 172.21.15.6:0/4058066393 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3465 (secure 0 0 0) 0x7f3e10058b10 con 0x7f3e180c9150 2024-03-04T09:34:56.953 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:34:56.953 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:33:54.652601+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"laggy_since":"2024-03-04T09:33:54.652601+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":9}]} 2024-03-04T09:34:56.955 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 100 2024-03-04T09:34:56.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:56.842+0000 7f3e1ec00640 1 -- 172.21.15.6:0/4058066393 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f3df406b4d0 msgr2=0x7f3df406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:56.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:56.842+0000 7f3e1ec00640 1 --2- 172.21.15.6:0/4058066393 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f3df406b4d0 0x7f3df406d990 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f3e0800a780 tx=0x7f3e0802f000 comp rx=0 tx=0).stop 2024-03-04T09:34:56.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:56.842+0000 7f3e1ec00640 1 -- 172.21.15.6:0/4058066393 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3e180c9150 msgr2=0x7f3e1806cb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:56.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:56.842+0000 7f3e1ec00640 1 --2- 172.21.15.6:0/4058066393 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3e180c9150 0x7f3e1806cb10 secure :-1 s=READY pgs=1672 cs=0 l=1 rev1=1 crypto rx=0x7f3e1809e100 tx=0x7f3e10010c20 comp rx=0 tx=0).stop 2024-03-04T09:34:56.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:56.842+0000 7f3e1ec00640 1 -- 172.21.15.6:0/4058066393 shutdown_connections 2024-03-04T09:34:56.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:56.842+0000 7f3e1ec00640 1 --2- 172.21.15.6:0/4058066393 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3e180c9150 0x7f3e1806cb10 unknown :-1 s=CLOSED pgs=1672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:56.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:56.842+0000 7f3e1ec00640 1 --2- 172.21.15.6:0/4058066393 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3e1806e6a0 0x7f3e1814c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:56.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:56.842+0000 7f3e1ec00640 1 --2- 172.21.15.6:0/4058066393 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f3df406b4d0 0x7f3df406d990 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:56.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:56.842+0000 7f3e1ec00640 1 --2- 172.21.15.6:0/4058066393 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3dfc003a50 0x7f3e18154900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:56.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:56.842+0000 7f3e1ec00640 1 -- 172.21.15.6:0/4058066393 >> 172.21.15.6:0/4058066393 conn(0x7f3e1806afb0 msgr2=0x7f3e1815fd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:34:56.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:56.842+0000 7f3e1ec00640 1 -- 172.21.15.6:0/4058066393 shutdown_connections 2024-03-04T09:34:56.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:56.842+0000 7f3e1ec00640 1 -- 172.21.15.6:0/4058066393 wait complete. 2024-03-04T09:34:57.158 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:32:57.159681+0000', 'modified': '2024-03-04T09:33:54.652601+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8214, 'mds_1': 6796}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6796': {'gid': 6796, 'name': 'd', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.146:6837/2458070840', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 2458070840}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 2458070840}]}, 'join_fscid': -1, 'laggy_since': '2024-03-04T09:33:54.652601+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8214': {'gid': 8214, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2446627517', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2446627517}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2446627517}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8214, 'qdb_cluster': [8214]} 2024-03-04T09:34:57.159 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6796:up:stopping 2024-03-04T09:34:57.860 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-04T09:34:58.159 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:34:58.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:58.389+0000 7fab5c800640 1 Processor -- start 2024-03-04T09:34:58.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:58.389+0000 7fab5c800640 1 -- start start 2024-03-04T09:34:58.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:58.390+0000 7fab5c800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fab540af1a0 0x7fab5415c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:58.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:58.391+0000 7fab5c800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fab540af580 0x7fab540b4c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:58.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:58.391+0000 7fab5c800640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab5415cc40 con 0x7fab540cdc20 2024-03-04T09:34:58.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:58.391+0000 7fab5c800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab5415cf20 con 0x7fab540af580 2024-03-04T09:34:58.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:58.391+0000 7fab5c800640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab5415d200 con 0x7fab540af1a0 2024-03-04T09:34:58.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:58.391+0000 7fab59400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fab540af580 0x7fab540b4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:58.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:58.391+0000 7fab59400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fab540af580 0x7fab540b4c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:40458/0 (socket says 172.21.15.6:40458) 2024-03-04T09:34:58.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:58.391+0000 7fab59400640 1 -- 172.21.15.6:0/3675841504 learned_addr learned my addr 172.21.15.6:0/3675841504 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:34:58.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:58.391+0000 7fab59e00640 1 --2- 172.21.15.6:0/3675841504 >> v2:172.21.15.146:3300/0 conn(0x7fab540af1a0 0x7fab5415c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:58.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:58.392+0000 7fab59400640 1 -- 172.21.15.6:0/3675841504 >> v2:172.21.15.146:3300/0 conn(0x7fab540af1a0 msgr2=0x7fab5415c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:34:58.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:58.392+0000 7fab59400640 1 --2- 172.21.15.6:0/3675841504 >> v2:172.21.15.146:3300/0 conn(0x7fab540af1a0 0x7fab5415c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:58.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:58.392+0000 7fab59400640 1 -- 172.21.15.6:0/3675841504 >> v1:172.21.15.146:6789/0 conn(0x7fab540cdc20 legacy=0x7fab540b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:34:58.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:58.392+0000 7fab59400640 1 -- 172.21.15.6:0/3675841504 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab5415d4e0 con 0x7fab540af580 2024-03-04T09:34:58.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:58.393+0000 7fab59400640 1 --2- 172.21.15.6:0/3675841504 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fab540af580 0x7fab540b4c20 secure :-1 s=READY pgs=1673 cs=0 l=1 rev1=1 crypto rx=0x7fab54074960 tx=0x7fab4402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f485a26f90c3cfaf server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:34:58.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:58.393+0000 7fab58a00640 1 -- 172.21.15.6:0/3675841504 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab4402e070 con 0x7fab540af580 2024-03-04T09:34:58.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:58.393+0000 7fab58a00640 1 -- 172.21.15.6:0/3675841504 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab44037070 con 0x7fab540af580 2024-03-04T09:34:58.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:58.393+0000 7fab58a00640 1 -- 172.21.15.6:0/3675841504 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab44031a00 con 0x7fab540af580 2024-03-04T09:34:58.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:58.394+0000 7fab5c800640 1 -- 172.21.15.6:0/3675841504 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fab540af580 msgr2=0x7fab540b4c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:58.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:58.394+0000 7fab5c800640 1 --2- 172.21.15.6:0/3675841504 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fab540af580 0x7fab540b4c20 secure :-1 s=READY pgs=1673 cs=0 l=1 rev1=1 crypto rx=0x7fab54074960 tx=0x7fab4402b880 comp rx=0 tx=0).stop 2024-03-04T09:34:58.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:58.394+0000 7fab5c800640 1 -- 172.21.15.6:0/3675841504 shutdown_connections 2024-03-04T09:34:58.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:58.394+0000 7fab5c800640 1 --2- 172.21.15.6:0/3675841504 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fab540af580 0x7fab540b4c20 unknown :-1 s=CLOSED pgs=1673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:58.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:58.394+0000 7fab5c800640 1 --2- 172.21.15.6:0/3675841504 >> v2:172.21.15.146:3300/0 conn(0x7fab540af1a0 0x7fab5415c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:58.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:58.394+0000 7fab5c800640 1 -- 172.21.15.6:0/3675841504 >> 172.21.15.6:0/3675841504 conn(0x7fab5406afb0 msgr2=0x7fab5406b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:34:58.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:58.394+0000 7fab5c800640 1 -- 172.21.15.6:0/3675841504 shutdown_connections 2024-03-04T09:34:58.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:58.394+0000 7fab5c800640 1 -- 172.21.15.6:0/3675841504 wait complete. 2024-03-04T09:34:58.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:58.396+0000 7fab5c800640 1 Processor -- start 2024-03-04T09:34:58.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:58.396+0000 7fab5c800640 1 -- start start 2024-03-04T09:34:58.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:58.397+0000 7fab5c800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fab540af1a0 0x7fab540cb930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:58.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:58.397+0000 7fab5c800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fab540af580 0x7fab54159630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:58.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:58.397+0000 7fab5c800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fab540cdc20 0x7fab541f41f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:58.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:58.397+0000 7fab5c800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab540b3660 con 0x7fab540af1a0 2024-03-04T09:34:58.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:58.397+0000 7fab5c800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab540b34e0 con 0x7fab540cdc20 2024-03-04T09:34:58.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:58.397+0000 7fab5c800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab540b37e0 con 0x7fab540af580 2024-03-04T09:34:58.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:58.397+0000 7fab59e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fab540af1a0 0x7fab540cb930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:58.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:58.397+0000 7fab59e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fab540af1a0 0x7fab540cb930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:40474/0 (socket says 172.21.15.6:40474) 2024-03-04T09:34:58.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:58.397+0000 7fab59e00640 1 -- 172.21.15.6:0/1505342136 learned_addr learned my addr 172.21.15.6:0/1505342136 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:34:58.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:58.397+0000 7fab59e00640 1 -- 172.21.15.6:0/1505342136 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fab540af580 msgr2=0x7fab54159630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:58.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:58.397+0000 7fab59e00640 1 --2- 172.21.15.6:0/1505342136 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fab540af580 0x7fab54159630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:58.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:58.397+0000 7fab59e00640 1 -- 172.21.15.6:0/1505342136 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fab540cdc20 msgr2=0x7fab541f41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:58.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:58.397+0000 7fab59e00640 1 --2- 172.21.15.6:0/1505342136 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fab540cdc20 0x7fab541f41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:58.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:58.397+0000 7fab59e00640 1 -- 172.21.15.6:0/1505342136 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab5406d920 con 0x7fab540af1a0 2024-03-04T09:34:58.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:58.398+0000 7fab59e00640 1 --2- 172.21.15.6:0/1505342136 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fab540af1a0 0x7fab540cb930 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x7fab4c007ef0 tx=0x7fab4c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:34:58.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:58.398+0000 7fab4a000640 1 -- 172.21.15.6:0/1505342136 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab4c014620 con 0x7fab540af1a0 2024-03-04T09:34:58.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:58.398+0000 7fab4a000640 1 -- 172.21.15.6:0/1505342136 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab4c019070 con 0x7fab540af1a0 2024-03-04T09:34:58.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:58.398+0000 7fab4a000640 1 -- 172.21.15.6:0/1505342136 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab4c014a00 con 0x7fab540af1a0 2024-03-04T09:34:58.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:58.398+0000 7fab5c800640 1 -- 172.21.15.6:0/1505342136 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab5406d080 con 0x7fab540af1a0 2024-03-04T09:34:58.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:58.398+0000 7fab5c800640 1 -- 172.21.15.6:0/1505342136 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab5406d640 con 0x7fab540af1a0 2024-03-04T09:34:58.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:58.400+0000 7fab5c800640 1 -- 172.21.15.6:0/1505342136 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab5404f3a0 con 0x7fab540af1a0 2024-03-04T09:34:58.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:58.401+0000 7fab4a000640 1 -- 172.21.15.6:0/1505342136 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fab4c014ba0 con 0x7fab540af1a0 2024-03-04T09:34:58.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:58.401+0000 7fab4a000640 1 --2- 172.21.15.6:0/1505342136 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fab3806b4d0 0x7fab3806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:34:58.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:58.402+0000 7fab4a000640 1 -- 172.21.15.6:0/1505342136 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fab4c08ac20 con 0x7fab540af1a0 2024-03-04T09:34:58.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:58.402+0000 7fab59400640 1 --2- 172.21.15.6:0/1505342136 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fab3806b4d0 0x7fab3806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:34:58.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:58.403+0000 7fab59400640 1 --2- 172.21.15.6:0/1505342136 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fab3806b4d0 0x7fab3806d990 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7fab44036000 tx=0x7fab44030000 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:34:58.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:58.410+0000 7fab4a000640 1 -- 172.21.15.6:0/1505342136 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fab4c0599c0 con 0x7fab540af1a0 2024-03-04T09:34:58.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:58.716+0000 7fab5c800640 1 -- 172.21.15.6:0/1505342136 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fab54159b70 con 0x7fab540af1a0 2024-03-04T09:34:58.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:58.717+0000 7fab4a000640 1 -- 172.21.15.6:0/1505342136 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3465 (secure 0 0 0) 0x7fab4c0592c0 con 0x7fab540af1a0 2024-03-04T09:34:58.714 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:34:58.714 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:33:54.652601+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"laggy_since":"2024-03-04T09:33:54.652601+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":9}]} 2024-03-04T09:34:58.714 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 100 2024-03-04T09:34:58.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:58.719+0000 7fab5c800640 1 -- 172.21.15.6:0/1505342136 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fab3806b4d0 msgr2=0x7fab3806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:58.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:58.719+0000 7fab5c800640 1 --2- 172.21.15.6:0/1505342136 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fab3806b4d0 0x7fab3806d990 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7fab44036000 tx=0x7fab44030000 comp rx=0 tx=0).stop 2024-03-04T09:34:58.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:58.719+0000 7fab5c800640 1 -- 172.21.15.6:0/1505342136 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fab540af1a0 msgr2=0x7fab540cb930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:34:58.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:58.720+0000 7fab5c800640 1 --2- 172.21.15.6:0/1505342136 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fab540af1a0 0x7fab540cb930 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x7fab4c007ef0 tx=0x7fab4c00e4b0 comp rx=0 tx=0).stop 2024-03-04T09:34:58.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:58.720+0000 7fab5c800640 1 -- 172.21.15.6:0/1505342136 shutdown_connections 2024-03-04T09:34:58.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:58.720+0000 7fab5c800640 1 --2- 172.21.15.6:0/1505342136 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fab540cdc20 0x7fab541f41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:58.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:58.720+0000 7fab5c800640 1 --2- 172.21.15.6:0/1505342136 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fab3806b4d0 0x7fab3806d990 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:58.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:58.720+0000 7fab5c800640 1 --2- 172.21.15.6:0/1505342136 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fab540af580 0x7fab54159630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:58.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:58.720+0000 7fab5c800640 1 --2- 172.21.15.6:0/1505342136 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fab540af1a0 0x7fab540cb930 unknown :-1 s=CLOSED pgs=1674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:34:58.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:58.720+0000 7fab5c800640 1 -- 172.21.15.6:0/1505342136 >> 172.21.15.6:0/1505342136 conn(0x7fab5406afb0 msgr2=0x7fab540ae5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:34:58.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:58.720+0000 7fab5c800640 1 -- 172.21.15.6:0/1505342136 shutdown_connections 2024-03-04T09:34:58.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:34:58.720+0000 7fab5c800640 1 -- 172.21.15.6:0/1505342136 wait complete. 2024-03-04T09:34:58.938 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:32:57.159681+0000', 'modified': '2024-03-04T09:33:54.652601+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8214, 'mds_1': 6796}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6796': {'gid': 6796, 'name': 'd', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.146:6837/2458070840', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 2458070840}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 2458070840}]}, 'join_fscid': -1, 'laggy_since': '2024-03-04T09:33:54.652601+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8214': {'gid': 8214, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2446627517', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2446627517}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2446627517}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8214, 'qdb_cluster': [8214]} 2024-03-04T09:34:58.939 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6796:up:stopping 2024-03-04T09:34:59.861 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-04T09:34:59.940 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:35:00.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:00.171+0000 7fbc9ba00640 1 Processor -- start 2024-03-04T09:35:00.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:00.171+0000 7fbc9ba00640 1 -- start start 2024-03-04T09:35:00.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:00.173+0000 7fbc9ba00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fbc940cdaa0 0x7fbc940cfc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:00.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:00.173+0000 7fbc9ba00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc9406e5f0 con 0x7fbc9415c010 2024-03-04T09:35:00.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:00.173+0000 7fbc9ba00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc9406e860 con 0x7fbc9406e190 2024-03-04T09:35:00.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:00.173+0000 7fbc9ba00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc9406b150 con 0x7fbc940cdaa0 2024-03-04T09:35:00.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:00.173+0000 7fbc99000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fbc940cdaa0 0x7fbc940cfc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:00.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:00.173+0000 7fbc99a00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fbc9415c010 0x7fbc9406bc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:46266/0 (socket says 172.21.15.6:46266) 2024-03-04T09:35:00.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:00.173+0000 7fbc99a00640 1 -- 172.21.15.6:0/3468486722 learned_addr learned my addr 172.21.15.6:0/3468486722 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:35:00.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:00.174+0000 7fbc99000640 1 -- 172.21.15.6:0/3468486722 >> v1:172.21.15.146:6789/0 conn(0x7fbc9406e190 legacy=0x7fbc940cde80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:35:00.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:00.174+0000 7fbc99000640 1 -- 172.21.15.6:0/3468486722 >> v1:172.21.15.6:6789/0 conn(0x7fbc9415c010 legacy=0x7fbc9406bc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:00.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:00.174+0000 7fbc99000640 1 -- 172.21.15.6:0/3468486722 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc9406b430 con 0x7fbc940cdaa0 2024-03-04T09:35:00.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:00.175+0000 7fbc99000640 1 --2- 172.21.15.6:0/3468486722 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbc940cdaa0 0x7fbc940cfc40 secure :-1 s=READY pgs=1676 cs=0 l=1 rev1=1 crypto rx=0x7fbc840097a0 tx=0x7fbc8402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d09da7defaa0f0c server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:35:00.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:00.175+0000 7fbc93e00640 1 -- 172.21.15.6:0/3468486722 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc84035020 con 0x7fbc940cdaa0 2024-03-04T09:35:00.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:00.176+0000 7fbc93e00640 1 -- 172.21.15.6:0/3468486722 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc84031830 con 0x7fbc940cdaa0 2024-03-04T09:35:00.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:00.176+0000 7fbc93e00640 1 -- 172.21.15.6:0/3468486722 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc84031c00 con 0x7fbc940cdaa0 2024-03-04T09:35:00.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:00.176+0000 7fbc9ba00640 1 -- 172.21.15.6:0/3468486722 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbc940cdaa0 msgr2=0x7fbc940cfc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:00.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:00.176+0000 7fbc9ba00640 1 --2- 172.21.15.6:0/3468486722 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbc940cdaa0 0x7fbc940cfc40 secure :-1 s=READY pgs=1676 cs=0 l=1 rev1=1 crypto rx=0x7fbc840097a0 tx=0x7fbc8402b8d0 comp rx=0 tx=0).stop 2024-03-04T09:35:00.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:00.176+0000 7fbc9ba00640 1 -- 172.21.15.6:0/3468486722 shutdown_connections 2024-03-04T09:35:00.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:00.176+0000 7fbc9ba00640 1 --2- 172.21.15.6:0/3468486722 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbc940cdaa0 0x7fbc940cfc40 unknown :-1 s=CLOSED pgs=1676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:00.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:00.176+0000 7fbc9ba00640 1 -- 172.21.15.6:0/3468486722 >> 172.21.15.6:0/3468486722 conn(0x7fbc940b2640 msgr2=0x7fbc940b2a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:35:00.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:00.177+0000 7fbc9ba00640 1 -- 172.21.15.6:0/3468486722 shutdown_connections 2024-03-04T09:35:00.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:00.177+0000 7fbc9ba00640 1 -- 172.21.15.6:0/3468486722 wait complete. 2024-03-04T09:35:00.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:00.178+0000 7fbc9ba00640 1 Processor -- start 2024-03-04T09:35:00.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:00.178+0000 7fbc9ba00640 1 -- start start 2024-03-04T09:35:00.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:00.179+0000 7fbc9ba00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbc9406e190 0x7fbc940adfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:00.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:00.179+0000 7fbc9ba00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbc940cdaa0 0x7fbc940a9600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:00.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:00.179+0000 7fbc9ba00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbc9415c010 0x7fbc9414a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:00.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:00.179+0000 7fbc9ba00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc9415c420 con 0x7fbc940cdaa0 2024-03-04T09:35:00.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:00.179+0000 7fbc9ba00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc940afc50 con 0x7fbc9415c010 2024-03-04T09:35:00.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:00.179+0000 7fbc9ba00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc9415c5a0 con 0x7fbc9406e190 2024-03-04T09:35:00.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:00.180+0000 7fbc8be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbc940cdaa0 0x7fbc940a9600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:00.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:00.180+0000 7fbc8be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbc940cdaa0 0x7fbc940a9600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42462/0 (socket says 172.21.15.6:42462) 2024-03-04T09:35:00.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:00.180+0000 7fbc8be00640 1 -- 172.21.15.6:0/2278828193 learned_addr learned my addr 172.21.15.6:0/2278828193 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:35:00.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:00.180+0000 7fbc99000640 1 --2- 172.21.15.6:0/2278828193 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbc9406e190 0x7fbc940adfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:00.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:00.180+0000 7fbc99a00640 1 --2- 172.21.15.6:0/2278828193 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbc9415c010 0x7fbc9414a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:00.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:00.180+0000 7fbc8be00640 1 -- 172.21.15.6:0/2278828193 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbc9406e190 msgr2=0x7fbc940adfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:00.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:00.180+0000 7fbc8be00640 1 --2- 172.21.15.6:0/2278828193 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbc9406e190 0x7fbc940adfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:00.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:00.180+0000 7fbc8be00640 1 -- 172.21.15.6:0/2278828193 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbc9415c010 msgr2=0x7fbc9414a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:00.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:00.180+0000 7fbc8be00640 1 --2- 172.21.15.6:0/2278828193 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbc9415c010 0x7fbc9414a0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:00.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:00.180+0000 7fbc8be00640 1 -- 172.21.15.6:0/2278828193 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc940aca70 con 0x7fbc940cdaa0 2024-03-04T09:35:00.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:00.181+0000 7fbc8be00640 1 --2- 172.21.15.6:0/2278828193 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbc940cdaa0 0x7fbc940a9600 secure :-1 s=READY pgs=1677 cs=0 l=1 rev1=1 crypto rx=0x7fbc8000c430 tx=0x7fbc8000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:35:00.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:00.181+0000 7fbc91600640 1 -- 172.21.15.6:0/2278828193 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc80011790 con 0x7fbc940cdaa0 2024-03-04T09:35:00.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:00.181+0000 7fbc91600640 1 -- 172.21.15.6:0/2278828193 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc80017070 con 0x7fbc940cdaa0 2024-03-04T09:35:00.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:00.181+0000 7fbc91600640 1 -- 172.21.15.6:0/2278828193 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc80011bb0 con 0x7fbc940cdaa0 2024-03-04T09:35:00.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:00.181+0000 7fbc9ba00640 1 -- 172.21.15.6:0/2278828193 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc940a9d00 con 0x7fbc940cdaa0 2024-03-04T09:35:00.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:00.181+0000 7fbc9ba00640 1 -- 172.21.15.6:0/2278828193 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc940aa2c0 con 0x7fbc940cdaa0 2024-03-04T09:35:00.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:00.182+0000 7fbc9ba00640 1 -- 172.21.15.6:0/2278828193 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc9404f3a0 con 0x7fbc940cdaa0 2024-03-04T09:35:00.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:00.183+0000 7fbc91600640 1 -- 172.21.15.6:0/2278828193 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbc8001c460 con 0x7fbc940cdaa0 2024-03-04T09:35:00.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:00.184+0000 7fbc91600640 1 --2- 172.21.15.6:0/2278828193 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fbc7406b500 0x7fbc7406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:00.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:00.184+0000 7fbc91600640 1 -- 172.21.15.6:0/2278828193 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fbc8008a100 con 0x7fbc940cdaa0 2024-03-04T09:35:00.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:00.185+0000 7fbc99000640 1 --2- 172.21.15.6:0/2278828193 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fbc7406b500 0x7fbc7406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:00.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:00.186+0000 7fbc99000640 1 --2- 172.21.15.6:0/2278828193 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fbc7406b500 0x7fbc7406d9c0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fbc84036000 tx=0x7fbc8403c040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:35:00.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:00.189+0000 7fbc91600640 1 -- 172.21.15.6:0/2278828193 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbc940aca70 con 0x7fbc940cdaa0 2024-03-04T09:35:00.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:00.432+0000 7fbc9ba00640 1 -- 172.21.15.6:0/2278828193 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbc94150be0 con 0x7fbc940cdaa0 2024-03-04T09:35:00.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:00.433+0000 7fbc91600640 1 -- 172.21.15.6:0/2278828193 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3465 (secure 0 0 0) 0x7fbc80058ea0 con 0x7fbc940cdaa0 2024-03-04T09:35:00.430 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:35:00.430 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:33:54.652601+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"laggy_since":"2024-03-04T09:33:54.652601+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":9}]} 2024-03-04T09:35:00.430 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 100 2024-03-04T09:35:00.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:00.435+0000 7fbc9ba00640 1 -- 172.21.15.6:0/2278828193 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fbc7406b500 msgr2=0x7fbc7406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:00.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:00.435+0000 7fbc9ba00640 1 --2- 172.21.15.6:0/2278828193 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fbc7406b500 0x7fbc7406d9c0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fbc84036000 tx=0x7fbc8403c040 comp rx=0 tx=0).stop 2024-03-04T09:35:00.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:00.435+0000 7fbc9ba00640 1 -- 172.21.15.6:0/2278828193 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbc940cdaa0 msgr2=0x7fbc940a9600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:00.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:00.435+0000 7fbc9ba00640 1 --2- 172.21.15.6:0/2278828193 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbc940cdaa0 0x7fbc940a9600 secure :-1 s=READY pgs=1677 cs=0 l=1 rev1=1 crypto rx=0x7fbc8000c430 tx=0x7fbc8000c8f0 comp rx=0 tx=0).stop 2024-03-04T09:35:00.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:00.435+0000 7fbc9ba00640 1 -- 172.21.15.6:0/2278828193 shutdown_connections 2024-03-04T09:35:00.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:00.435+0000 7fbc9ba00640 1 --2- 172.21.15.6:0/2278828193 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbc9415c010 0x7fbc9414a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:00.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:00.435+0000 7fbc9ba00640 1 --2- 172.21.15.6:0/2278828193 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbc940cdaa0 0x7fbc940a9600 unknown :-1 s=CLOSED pgs=1677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:00.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:00.435+0000 7fbc9ba00640 1 --2- 172.21.15.6:0/2278828193 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fbc7406b500 0x7fbc7406d9c0 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:00.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:00.435+0000 7fbc9ba00640 1 --2- 172.21.15.6:0/2278828193 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbc9406e190 0x7fbc940adfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:00.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:00.435+0000 7fbc9ba00640 1 -- 172.21.15.6:0/2278828193 >> 172.21.15.6:0/2278828193 conn(0x7fbc940b2640 msgr2=0x7fbc9406d780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:35:00.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:00.436+0000 7fbc9ba00640 1 -- 172.21.15.6:0/2278828193 shutdown_connections 2024-03-04T09:35:00.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:00.436+0000 7fbc9ba00640 1 -- 172.21.15.6:0/2278828193 wait complete. 2024-03-04T09:35:00.652 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:32:57.159681+0000', 'modified': '2024-03-04T09:33:54.652601+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8214, 'mds_1': 6796}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6796': {'gid': 6796, 'name': 'd', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.146:6837/2458070840', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 2458070840}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 2458070840}]}, 'join_fscid': -1, 'laggy_since': '2024-03-04T09:33:54.652601+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8214': {'gid': 8214, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2446627517', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2446627517}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2446627517}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8214, 'qdb_cluster': [8214]} 2024-03-04T09:35:00.652 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6796:up:stopping 2024-03-04T09:35:01.653 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:35:01.862 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-04T09:35:01.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:01.875+0000 7ff46fe00640 1 Processor -- start 2024-03-04T09:35:01.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:01.876+0000 7ff46fe00640 1 -- start start 2024-03-04T09:35:01.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:01.878+0000 7ff46fe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff4700b2340 0x7ff470168cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:01.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:01.878+0000 7ff46fe00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4700d1120 con 0x7ff47015e2a0 2024-03-04T09:35:01.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:01.879+0000 7ff46fe00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4700d1400 con 0x7ff47015dec0 2024-03-04T09:35:01.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:01.879+0000 7ff46fe00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4700b0ea0 con 0x7ff4700b2340 2024-03-04T09:35:01.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:01.879+0000 7ff46f400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff4700b2340 0x7ff470168cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:01.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:01.879+0000 7ff467e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7ff47015e2a0 0x7ff4700cef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:46268/0 (socket says 172.21.15.6:46268) 2024-03-04T09:35:01.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:01.879+0000 7ff467e00640 1 -- 172.21.15.6:0/210823298 learned_addr learned my addr 172.21.15.6:0/210823298 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:35:01.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:01.879+0000 7ff46f400640 1 -- 172.21.15.6:0/210823298 >> v1:172.21.15.146:6789/0 conn(0x7ff47015dec0 legacy=0x7ff4700ce810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:35:01.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:01.879+0000 7ff46f400640 1 -- 172.21.15.6:0/210823298 >> v1:172.21.15.6:6789/0 conn(0x7ff47015e2a0 legacy=0x7ff4700cef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:01.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:01.879+0000 7ff46f400640 1 -- 172.21.15.6:0/210823298 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4700b0b30 con 0x7ff4700b2340 2024-03-04T09:35:01.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:01.879+0000 7ff46e000640 1 -- 172.21.15.6:0/210823298 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 519080674 0 0) 0x7ff4700d1120 con 0x7ff47015e2a0 2024-03-04T09:35:01.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:01.881+0000 7ff46f400640 1 --2- 172.21.15.6:0/210823298 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff4700b2340 0x7ff470168cd0 secure :-1 s=READY pgs=1679 cs=0 l=1 rev1=1 crypto rx=0x7ff468009d90 tx=0x7ff46802bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a26683e36779b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:35:01.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:01.881+0000 7ff46e000640 1 -- 172.21.15.6:0/210823298 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff468039070 con 0x7ff4700b2340 2024-03-04T09:35:01.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:01.881+0000 7ff46e000640 1 -- 172.21.15.6:0/210823298 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff46802cd00 con 0x7ff4700b2340 2024-03-04T09:35:01.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:01.881+0000 7ff46e000640 1 -- 172.21.15.6:0/210823298 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff468034660 con 0x7ff4700b2340 2024-03-04T09:35:01.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:01.881+0000 7ff46fe00640 1 -- 172.21.15.6:0/210823298 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff4700b2340 msgr2=0x7ff470168cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:01.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:01.881+0000 7ff46fe00640 1 --2- 172.21.15.6:0/210823298 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff4700b2340 0x7ff470168cd0 secure :-1 s=READY pgs=1679 cs=0 l=1 rev1=1 crypto rx=0x7ff468009d90 tx=0x7ff46802bf50 comp rx=0 tx=0).stop 2024-03-04T09:35:01.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:01.882+0000 7ff46fe00640 1 -- 172.21.15.6:0/210823298 shutdown_connections 2024-03-04T09:35:01.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:01.882+0000 7ff46fe00640 1 --2- 172.21.15.6:0/210823298 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff4700b2340 0x7ff470168cd0 unknown :-1 s=CLOSED pgs=1679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:01.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:01.882+0000 7ff46fe00640 1 -- 172.21.15.6:0/210823298 >> 172.21.15.6:0/210823298 conn(0x7ff4700d3d40 msgr2=0x7ff4700d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:35:01.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:01.882+0000 7ff46fe00640 1 -- 172.21.15.6:0/210823298 shutdown_connections 2024-03-04T09:35:01.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:01.882+0000 7ff46fe00640 1 -- 172.21.15.6:0/210823298 wait complete. 2024-03-04T09:35:01.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:01.883+0000 7ff46fe00640 1 Processor -- start 2024-03-04T09:35:01.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:01.884+0000 7ff46fe00640 1 -- start start 2024-03-04T09:35:01.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:01.884+0000 7ff46fe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff4700b2340 0x7ff47015e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:01.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:01.884+0000 7ff46fe00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff47015dec0 0x7ff470150a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:01.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:01.884+0000 7ff46fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff47015e2a0 0x7ff47014cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:01.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:01.885+0000 7ff46fe00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4700d0d40 con 0x7ff47015e2a0 2024-03-04T09:35:01.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:01.885+0000 7ff46fe00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4700cf630 con 0x7ff4700b2340 2024-03-04T09:35:01.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:01.885+0000 7ff46fe00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4700d0ec0 con 0x7ff47015dec0 2024-03-04T09:35:01.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:01.885+0000 7ff46f400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff47015e2a0 0x7ff47014cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:01.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:01.885+0000 7ff46f400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff47015e2a0 0x7ff47014cd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42482/0 (socket says 172.21.15.6:42482) 2024-03-04T09:35:01.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:01.885+0000 7ff46f400640 1 -- 172.21.15.6:0/3714953711 learned_addr learned my addr 172.21.15.6:0/3714953711 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:35:01.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:01.885+0000 7ff467e00640 1 --2- 172.21.15.6:0/3714953711 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff47015dec0 0x7ff470150a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:01.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:01.885+0000 7ff46ea00640 1 --2- 172.21.15.6:0/3714953711 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff4700b2340 0x7ff47015e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:01.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:01.885+0000 7ff46f400640 1 -- 172.21.15.6:0/3714953711 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff47015dec0 msgr2=0x7ff470150a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:01.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:01.885+0000 7ff46f400640 1 --2- 172.21.15.6:0/3714953711 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff47015dec0 0x7ff470150a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:01.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:01.885+0000 7ff46f400640 1 -- 172.21.15.6:0/3714953711 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff4700b2340 msgr2=0x7ff47015e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:01.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:01.885+0000 7ff46f400640 1 --2- 172.21.15.6:0/3714953711 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff4700b2340 0x7ff47015e770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:01.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:01.885+0000 7ff46f400640 1 -- 172.21.15.6:0/3714953711 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff47015ecb0 con 0x7ff47015e2a0 2024-03-04T09:35:01.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:01.886+0000 7ff46f400640 1 --2- 172.21.15.6:0/3714953711 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff47015e2a0 0x7ff47014cd40 secure :-1 s=READY pgs=1680 cs=0 l=1 rev1=1 crypto rx=0x7ff468037040 tx=0x7ff468004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:35:01.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:01.886+0000 7ff467400640 1 -- 172.21.15.6:0/3714953711 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff468044020 con 0x7ff47015e2a0 2024-03-04T09:35:01.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:01.886+0000 7ff467400640 1 -- 172.21.15.6:0/3714953711 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff46803a040 con 0x7ff47015e2a0 2024-03-04T09:35:01.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:01.886+0000 7ff467400640 1 -- 172.21.15.6:0/3714953711 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff468039050 con 0x7ff47015e2a0 2024-03-04T09:35:01.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:01.886+0000 7ff46fe00640 1 -- 172.21.15.6:0/3714953711 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff470160600 con 0x7ff47015e2a0 2024-03-04T09:35:01.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:01.886+0000 7ff46fe00640 1 -- 172.21.15.6:0/3714953711 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff470160bc0 con 0x7ff47015e2a0 2024-03-04T09:35:01.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:01.888+0000 7ff46fe00640 1 -- 172.21.15.6:0/3714953711 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff47004f330 con 0x7ff47015e2a0 2024-03-04T09:35:01.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:01.888+0000 7ff467400640 1 -- 172.21.15.6:0/3714953711 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff46802cb80 con 0x7ff47015e2a0 2024-03-04T09:35:01.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:01.890+0000 7ff467400640 1 --2- 172.21.15.6:0/3714953711 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff44c06b4d0 0x7ff44c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:01.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:01.890+0000 7ff467400640 1 -- 172.21.15.6:0/3714953711 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7ff4680ac980 con 0x7ff47015e2a0 2024-03-04T09:35:01.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:01.890+0000 7ff46ea00640 1 --2- 172.21.15.6:0/3714953711 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff44c06b4d0 0x7ff44c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:01.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:01.892+0000 7ff46ea00640 1 --2- 172.21.15.6:0/3714953711 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff44c06b4d0 0x7ff44c06d990 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7ff460000f60 tx=0x7ff460009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:35:01.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:01.893+0000 7ff467400640 1 -- 172.21.15.6:0/3714953711 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff46804e060 con 0x7ff47015e2a0 2024-03-04T09:35:02.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:02.195+0000 7ff46fe00640 1 -- 172.21.15.6:0/3714953711 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff4700d2640 con 0x7ff47015e2a0 2024-03-04T09:35:02.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:02.196+0000 7ff467400640 1 -- 172.21.15.6:0/3714953711 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3465 (secure 0 0 0) 0x7ff46807b850 con 0x7ff47015e2a0 2024-03-04T09:35:02.193 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:35:02.193 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:33:54.652601+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"laggy_since":"2024-03-04T09:33:54.652601+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":9}]} 2024-03-04T09:35:02.193 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 100 2024-03-04T09:35:02.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:02.198+0000 7ff46fe00640 1 -- 172.21.15.6:0/3714953711 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff44c06b4d0 msgr2=0x7ff44c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:02.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:02.198+0000 7ff46fe00640 1 --2- 172.21.15.6:0/3714953711 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff44c06b4d0 0x7ff44c06d990 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7ff460000f60 tx=0x7ff460009040 comp rx=0 tx=0).stop 2024-03-04T09:35:02.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:02.198+0000 7ff46fe00640 1 -- 172.21.15.6:0/3714953711 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff47015e2a0 msgr2=0x7ff47014cd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:02.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:02.198+0000 7ff46fe00640 1 --2- 172.21.15.6:0/3714953711 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff47015e2a0 0x7ff47014cd40 secure :-1 s=READY pgs=1680 cs=0 l=1 rev1=1 crypto rx=0x7ff468037040 tx=0x7ff468004a40 comp rx=0 tx=0).stop 2024-03-04T09:35:02.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:02.199+0000 7ff46fe00640 1 -- 172.21.15.6:0/3714953711 shutdown_connections 2024-03-04T09:35:02.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:02.199+0000 7ff46fe00640 1 --2- 172.21.15.6:0/3714953711 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff47015e2a0 0x7ff47014cd40 unknown :-1 s=CLOSED pgs=1680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:02.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:02.199+0000 7ff46fe00640 1 --2- 172.21.15.6:0/3714953711 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff44c06b4d0 0x7ff44c06d990 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:02.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:02.199+0000 7ff46fe00640 1 --2- 172.21.15.6:0/3714953711 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff47015dec0 0x7ff470150a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:02.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:02.199+0000 7ff46fe00640 1 --2- 172.21.15.6:0/3714953711 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff4700b2340 0x7ff47015e770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:02.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:02.199+0000 7ff46fe00640 1 -- 172.21.15.6:0/3714953711 >> 172.21.15.6:0/3714953711 conn(0x7ff4700d3d40 msgr2=0x7ff47015d260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:35:02.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:02.199+0000 7ff46fe00640 1 -- 172.21.15.6:0/3714953711 shutdown_connections 2024-03-04T09:35:02.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:02.199+0000 7ff46fe00640 1 -- 172.21.15.6:0/3714953711 wait complete. 2024-03-04T09:35:02.416 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:32:57.159681+0000', 'modified': '2024-03-04T09:33:54.652601+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8214, 'mds_1': 6796}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6796': {'gid': 6796, 'name': 'd', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.146:6837/2458070840', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 2458070840}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 2458070840}]}, 'join_fscid': -1, 'laggy_since': '2024-03-04T09:33:54.652601+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8214': {'gid': 8214, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2446627517', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2446627517}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2446627517}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8214, 'qdb_cluster': [8214]} 2024-03-04T09:35:02.417 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6796:up:stopping 2024-03-04T09:35:03.417 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:35:03.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:03.644+0000 7f31c6c00640 1 Processor -- start 2024-03-04T09:35:03.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:03.644+0000 7f31c6c00640 1 -- start start 2024-03-04T09:35:03.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:03.645+0000 7f31c6c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f31c00d2960 0x7f31c00d05c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:03.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:03.645+0000 7f31c6c00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f31c00d0b00 0x7f31c00b3630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:03.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:03.645+0000 7f31c6c00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31c00d2dc0 con 0x7f31c015e070 2024-03-04T09:35:03.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:03.645+0000 7f31c6c00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31c00d0f70 con 0x7f31c00d2960 2024-03-04T09:35:03.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:03.645+0000 7f31c6c00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31c00d18c0 con 0x7f31c00d0b00 2024-03-04T09:35:03.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:03.646+0000 7f31c4e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f31c00d2960 0x7f31c00d05c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:03.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:03.646+0000 7f31c6200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f31c00d0b00 0x7f31c00b3630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:03.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:03.646+0000 7f31c6200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f31c00d0b00 0x7f31c00b3630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.6:49168/0 (socket says 172.21.15.6:49168) 2024-03-04T09:35:03.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:03.646+0000 7f31c6200640 1 -- 172.21.15.6:0/1215600098 learned_addr learned my addr 172.21.15.6:0/1215600098 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:35:03.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:03.647+0000 7f31c4e00640 1 -- 172.21.15.6:0/1215600098 >> v2:172.21.15.146:3300/0 conn(0x7f31c00d0b00 msgr2=0x7f31c00b3630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:35:03.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:03.647+0000 7f31c4e00640 1 --2- 172.21.15.6:0/1215600098 >> v2:172.21.15.146:3300/0 conn(0x7f31c00d0b00 0x7f31c00b3630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:03.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:03.647+0000 7f31c4e00640 1 -- 172.21.15.6:0/1215600098 >> v1:172.21.15.146:6789/0 conn(0x7f31c015e070 legacy=0x7f31c00cf410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:03.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:03.647+0000 7f31c4e00640 1 -- 172.21.15.6:0/1215600098 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31c00d1ba0 con 0x7f31c00d2960 2024-03-04T09:35:03.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:03.648+0000 7f31c4e00640 1 --2- 172.21.15.6:0/1215600098 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f31c00d2960 0x7f31c00d05c0 secure :-1 s=READY pgs=1681 cs=0 l=1 rev1=1 crypto rx=0x7f31c008c1b0 tx=0x7f31ac02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db46b1fa8f861352 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:35:03.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:03.648+0000 7f31bfe00640 1 -- 172.21.15.6:0/1215600098 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31ac035020 con 0x7f31c00d2960 2024-03-04T09:35:03.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:03.648+0000 7f31bfe00640 1 -- 172.21.15.6:0/1215600098 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31ac031830 con 0x7f31c00d2960 2024-03-04T09:35:03.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:03.648+0000 7f31bfe00640 1 -- 172.21.15.6:0/1215600098 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31ac031c00 con 0x7f31c00d2960 2024-03-04T09:35:03.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:03.649+0000 7f31c6c00640 1 -- 172.21.15.6:0/1215600098 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f31c00d2960 msgr2=0x7f31c00d05c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:03.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:03.649+0000 7f31c6c00640 1 --2- 172.21.15.6:0/1215600098 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f31c00d2960 0x7f31c00d05c0 secure :-1 s=READY pgs=1681 cs=0 l=1 rev1=1 crypto rx=0x7f31c008c1b0 tx=0x7f31ac02b8d0 comp rx=0 tx=0).stop 2024-03-04T09:35:03.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:03.649+0000 7f31c6c00640 1 -- 172.21.15.6:0/1215600098 shutdown_connections 2024-03-04T09:35:03.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:03.649+0000 7f31c6c00640 1 --2- 172.21.15.6:0/1215600098 >> v2:172.21.15.146:3300/0 conn(0x7f31c00d0b00 0x7f31c00b3630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:03.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:03.649+0000 7f31c6c00640 1 --2- 172.21.15.6:0/1215600098 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f31c00d2960 0x7f31c00d05c0 unknown :-1 s=CLOSED pgs=1681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:03.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:03.649+0000 7f31c6c00640 1 -- 172.21.15.6:0/1215600098 >> 172.21.15.6:0/1215600098 conn(0x7f31c008d2a0 msgr2=0x7f31c00d3cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:35:03.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:03.649+0000 7f31c6c00640 1 -- 172.21.15.6:0/1215600098 shutdown_connections 2024-03-04T09:35:03.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:03.649+0000 7f31c6c00640 1 -- 172.21.15.6:0/1215600098 wait complete. 2024-03-04T09:35:03.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:03.651+0000 7f31c6c00640 1 Processor -- start 2024-03-04T09:35:03.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:03.651+0000 7f31c6c00640 1 -- start start 2024-03-04T09:35:03.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:03.652+0000 7f31c6c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f31c00d0b00 0x7f31c0155070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:03.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:03.652+0000 7f31c6c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f31c00d2960 0x7f31c01608d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:03.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:03.652+0000 7f31c6c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f31c015e070 0x7f31c00b6910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:03.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:03.652+0000 7f31c6c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31c00d2440 con 0x7f31c015e070 2024-03-04T09:35:03.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:03.652+0000 7f31c6c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31c00d2160 con 0x7f31c00d0b00 2024-03-04T09:35:03.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:03.652+0000 7f31c6c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31c00d45b0 con 0x7f31c00d2960 2024-03-04T09:35:03.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:03.652+0000 7f31c6200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f31c015e070 0x7f31c00b6910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:03.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:03.652+0000 7f31c6200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f31c015e070 0x7f31c00b6910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42498/0 (socket says 172.21.15.6:42498) 2024-03-04T09:35:03.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:03.652+0000 7f31c6200640 1 -- 172.21.15.6:0/329747713 learned_addr learned my addr 172.21.15.6:0/329747713 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:35:03.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:03.653+0000 7f31c5800640 1 --2- 172.21.15.6:0/329747713 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f31c00d0b00 0x7f31c0155070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:03.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:03.653+0000 7f31c4e00640 1 --2- 172.21.15.6:0/329747713 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f31c00d2960 0x7f31c01608d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:03.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:03.653+0000 7f31c6200640 1 -- 172.21.15.6:0/329747713 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f31c00d2960 msgr2=0x7f31c01608d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:03.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:03.653+0000 7f31c6200640 1 --2- 172.21.15.6:0/329747713 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f31c00d2960 0x7f31c01608d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:03.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:03.653+0000 7f31c6200640 1 -- 172.21.15.6:0/329747713 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f31c00d0b00 msgr2=0x7f31c0155070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:03.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:03.653+0000 7f31c6200640 1 --2- 172.21.15.6:0/329747713 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f31c00d0b00 0x7f31c0155070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:03.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:03.653+0000 7f31c6200640 1 -- 172.21.15.6:0/329747713 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31c015f020 con 0x7f31c015e070 2024-03-04T09:35:03.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:03.653+0000 7f31c6200640 1 --2- 172.21.15.6:0/329747713 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f31c015e070 0x7f31c00b6910 secure :-1 s=READY pgs=1682 cs=0 l=1 rev1=1 crypto rx=0x7f31b800efc0 tx=0x7f31b800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:35:03.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:03.653+0000 7f31bd600640 1 -- 172.21.15.6:0/329747713 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31b800cd20 con 0x7f31c015e070 2024-03-04T09:35:03.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:03.653+0000 7f31bd600640 1 -- 172.21.15.6:0/329747713 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31b8003990 con 0x7f31c015e070 2024-03-04T09:35:03.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:03.653+0000 7f31bd600640 1 -- 172.21.15.6:0/329747713 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31b8003d80 con 0x7f31c015e070 2024-03-04T09:35:03.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:03.654+0000 7f31c6c00640 1 -- 172.21.15.6:0/329747713 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31c015f300 con 0x7f31c015e070 2024-03-04T09:35:03.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:03.654+0000 7f31c6c00640 1 -- 172.21.15.6:0/329747713 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31c015f8c0 con 0x7f31c015e070 2024-03-04T09:35:03.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:03.655+0000 7f31c6c00640 1 -- 172.21.15.6:0/329747713 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31c004f330 con 0x7f31c015e070 2024-03-04T09:35:03.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:03.656+0000 7f31bd600640 1 -- 172.21.15.6:0/329747713 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f31b8005d10 con 0x7f31c015e070 2024-03-04T09:35:03.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:03.657+0000 7f31bd600640 1 --2- 172.21.15.6:0/329747713 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f31a006b4d0 0x7f31a006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:03.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:03.657+0000 7f31bd600640 1 -- 172.21.15.6:0/329747713 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f31b808a7e0 con 0x7f31c015e070 2024-03-04T09:35:03.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:03.657+0000 7f31c5800640 1 --2- 172.21.15.6:0/329747713 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f31a006b4d0 0x7f31a006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:03.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:03.659+0000 7f31c5800640 1 --2- 172.21.15.6:0/329747713 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f31a006b4d0 0x7f31a006d990 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f31b4000f60 tx=0x7f31b4009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:35:03.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:03.662+0000 7f31bd600640 1 -- 172.21.15.6:0/329747713 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f31b8022020 con 0x7f31c015e070 2024-03-04T09:35:03.867 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-04T09:35:03.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:03.901+0000 7f31c6c00640 1 -- 172.21.15.6:0/329747713 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f31c00caa20 con 0x7f31c015e070 2024-03-04T09:35:03.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:03.902+0000 7f31bd600640 1 -- 172.21.15.6:0/329747713 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3465 (secure 0 0 0) 0x7f31b8010040 con 0x7f31c015e070 2024-03-04T09:35:03.899 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:35:03.899 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:33:54.652601+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"laggy_since":"2024-03-04T09:33:54.652601+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":9}]} 2024-03-04T09:35:03.899 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 100 2024-03-04T09:35:03.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:03.904+0000 7f31c6c00640 1 -- 172.21.15.6:0/329747713 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f31a006b4d0 msgr2=0x7f31a006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:03.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:03.904+0000 7f31c6c00640 1 --2- 172.21.15.6:0/329747713 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f31a006b4d0 0x7f31a006d990 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f31b4000f60 tx=0x7f31b4009040 comp rx=0 tx=0).stop 2024-03-04T09:35:03.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:03.905+0000 7f31c6c00640 1 -- 172.21.15.6:0/329747713 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f31c015e070 msgr2=0x7f31c00b6910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:03.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:03.905+0000 7f31c6c00640 1 --2- 172.21.15.6:0/329747713 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f31c015e070 0x7f31c00b6910 secure :-1 s=READY pgs=1682 cs=0 l=1 rev1=1 crypto rx=0x7f31b800efc0 tx=0x7f31b800c490 comp rx=0 tx=0).stop 2024-03-04T09:35:03.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:03.905+0000 7f31c6c00640 1 -- 172.21.15.6:0/329747713 shutdown_connections 2024-03-04T09:35:03.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:03.905+0000 7f31c6c00640 1 --2- 172.21.15.6:0/329747713 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f31c015e070 0x7f31c00b6910 unknown :-1 s=CLOSED pgs=1682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:03.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:03.905+0000 7f31c6c00640 1 --2- 172.21.15.6:0/329747713 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f31a006b4d0 0x7f31a006d990 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:03.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:03.905+0000 7f31c6c00640 1 --2- 172.21.15.6:0/329747713 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f31c00d2960 0x7f31c01608d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:03.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:03.905+0000 7f31c6c00640 1 --2- 172.21.15.6:0/329747713 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f31c00d0b00 0x7f31c0155070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:03.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:03.905+0000 7f31c6c00640 1 -- 172.21.15.6:0/329747713 >> 172.21.15.6:0/329747713 conn(0x7f31c008d2a0 msgr2=0x7f31c015d180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:35:03.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:03.905+0000 7f31c6c00640 1 -- 172.21.15.6:0/329747713 shutdown_connections 2024-03-04T09:35:03.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:03.905+0000 7f31c6c00640 1 -- 172.21.15.6:0/329747713 wait complete. 2024-03-04T09:35:04.122 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:32:57.159681+0000', 'modified': '2024-03-04T09:33:54.652601+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8214, 'mds_1': 6796}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6796': {'gid': 6796, 'name': 'd', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.146:6837/2458070840', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 2458070840}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 2458070840}]}, 'join_fscid': -1, 'laggy_since': '2024-03-04T09:33:54.652601+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8214': {'gid': 8214, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2446627517', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2446627517}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2446627517}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8214, 'qdb_cluster': [8214]} 2024-03-04T09:35:04.123 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6796:up:stopping 2024-03-04T09:35:05.123 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:35:05.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:05.351+0000 7f2fd6400640 1 Processor -- start 2024-03-04T09:35:05.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:05.351+0000 7f2fd6400640 1 -- start start 2024-03-04T09:35:05.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:05.353+0000 7f2fd6400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f2fd00af580 0x7f2fd015c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:05.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:05.353+0000 7f2fd6400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f2fd00cdc20 0x7f2fd00b2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:05.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:05.353+0000 7f2fd6400640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fd015cc40 con 0x7f2fd00af1a0 2024-03-04T09:35:05.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:05.353+0000 7f2fd6400640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fd015cf20 con 0x7f2fd00af580 2024-03-04T09:35:05.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:05.353+0000 7f2fd6400640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fd015d200 con 0x7f2fd00cdc20 2024-03-04T09:35:05.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:05.353+0000 7f2fcea00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f2fd00af580 0x7f2fd015c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:05.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:05.353+0000 7f2fcea00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f2fd00af580 0x7f2fd015c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42504/0 (socket says 172.21.15.6:42504) 2024-03-04T09:35:05.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:05.353+0000 7f2fcea00640 1 -- 172.21.15.6:0/1655658659 learned_addr learned my addr 172.21.15.6:0/1655658659 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:35:05.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:05.353+0000 7f2fcfe00640 1 --2- 172.21.15.6:0/1655658659 >> v2:172.21.15.146:3300/0 conn(0x7f2fd00cdc20 0x7f2fd00b2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:05.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:05.354+0000 7f2fcea00640 1 -- 172.21.15.6:0/1655658659 >> v2:172.21.15.146:3300/0 conn(0x7f2fd00cdc20 msgr2=0x7f2fd00b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:35:05.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:05.354+0000 7f2fcea00640 1 --2- 172.21.15.6:0/1655658659 >> v2:172.21.15.146:3300/0 conn(0x7f2fd00cdc20 0x7f2fd00b2dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:05.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:05.354+0000 7f2fcea00640 1 -- 172.21.15.6:0/1655658659 >> v1:172.21.15.146:6789/0 conn(0x7f2fd00af1a0 legacy=0x7f2fd00b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:35:05.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:05.354+0000 7f2fcea00640 1 -- 172.21.15.6:0/1655658659 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fd015d4e0 con 0x7f2fd00af580 2024-03-04T09:35:05.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:05.355+0000 7f2fcea00640 1 --2- 172.21.15.6:0/1655658659 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2fd00af580 0x7f2fd015c600 secure :-1 s=READY pgs=1683 cs=0 l=1 rev1=1 crypto rx=0x7f2fd006c0c0 tx=0x7f2fbc02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=359422aeca924dd8 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:35:05.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:05.356+0000 7f2fce000640 1 -- 172.21.15.6:0/1655658659 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fbc02e070 con 0x7f2fd00af580 2024-03-04T09:35:05.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:05.356+0000 7f2fce000640 1 -- 172.21.15.6:0/1655658659 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fbc037070 con 0x7f2fd00af580 2024-03-04T09:35:05.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:05.356+0000 7f2fce000640 1 -- 172.21.15.6:0/1655658659 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fbc031a00 con 0x7f2fd00af580 2024-03-04T09:35:05.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:05.356+0000 7f2fd6400640 1 -- 172.21.15.6:0/1655658659 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2fd00af580 msgr2=0x7f2fd015c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:05.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:05.356+0000 7f2fd6400640 1 --2- 172.21.15.6:0/1655658659 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2fd00af580 0x7f2fd015c600 secure :-1 s=READY pgs=1683 cs=0 l=1 rev1=1 crypto rx=0x7f2fd006c0c0 tx=0x7f2fbc02b880 comp rx=0 tx=0).stop 2024-03-04T09:35:05.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:05.356+0000 7f2fd6400640 1 -- 172.21.15.6:0/1655658659 shutdown_connections 2024-03-04T09:35:05.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:05.356+0000 7f2fd6400640 1 --2- 172.21.15.6:0/1655658659 >> v2:172.21.15.146:3300/0 conn(0x7f2fd00cdc20 0x7f2fd00b2dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:05.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:05.356+0000 7f2fd6400640 1 --2- 172.21.15.6:0/1655658659 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2fd00af580 0x7f2fd015c600 unknown :-1 s=CLOSED pgs=1683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:05.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:05.356+0000 7f2fd6400640 1 -- 172.21.15.6:0/1655658659 >> 172.21.15.6:0/1655658659 conn(0x7f2fd006afb0 msgr2=0x7f2fd006b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:35:05.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:05.357+0000 7f2fd6400640 1 -- 172.21.15.6:0/1655658659 shutdown_connections 2024-03-04T09:35:05.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:05.357+0000 7f2fd6400640 1 -- 172.21.15.6:0/1655658659 wait complete. 2024-03-04T09:35:05.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:05.359+0000 7f2fd6400640 1 Processor -- start 2024-03-04T09:35:05.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:05.359+0000 7f2fd6400640 1 -- start start 2024-03-04T09:35:05.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:05.360+0000 7f2fd6400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2fd00af1a0 0x7f2fd009aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:05.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:05.360+0000 7f2fd6400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2fd00af580 0x7f2fd014e750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:05.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:05.360+0000 7f2fd6400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2fd00cdc20 0x7f2fd0071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:05.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:05.360+0000 7f2fd6400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fd00b36a0 con 0x7f2fd00af1a0 2024-03-04T09:35:05.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:05.360+0000 7f2fd6400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fd00b3520 con 0x7f2fd00af580 2024-03-04T09:35:05.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:05.360+0000 7f2fd6400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fd00b3820 con 0x7f2fd00cdc20 2024-03-04T09:35:05.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:05.360+0000 7f2fcf400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2fd00af1a0 0x7f2fd009aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:05.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:05.361+0000 7f2fcf400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2fd00af1a0 0x7f2fd009aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42516/0 (socket says 172.21.15.6:42516) 2024-03-04T09:35:05.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:05.361+0000 7f2fcf400640 1 -- 172.21.15.6:0/2994076245 learned_addr learned my addr 172.21.15.6:0/2994076245 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:35:05.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:05.361+0000 7f2fcf400640 1 -- 172.21.15.6:0/2994076245 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2fd00cdc20 msgr2=0x7f2fd0071ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:05.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:05.361+0000 7f2fcfe00640 1 --2- 172.21.15.6:0/2994076245 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2fd00cdc20 0x7f2fd0071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:05.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:05.361+0000 7f2fcea00640 1 --2- 172.21.15.6:0/2994076245 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2fd00af580 0x7f2fd014e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:05.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:05.361+0000 7f2fcf400640 1 --2- 172.21.15.6:0/2994076245 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2fd00cdc20 0x7f2fd0071ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:05.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:05.361+0000 7f2fcf400640 1 -- 172.21.15.6:0/2994076245 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2fd00af580 msgr2=0x7f2fd014e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:05.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:05.361+0000 7f2fcf400640 1 --2- 172.21.15.6:0/2994076245 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2fd00af580 0x7f2fd014e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:05.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:05.361+0000 7f2fcf400640 1 -- 172.21.15.6:0/2994076245 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fd014f160 con 0x7f2fd00af1a0 2024-03-04T09:35:05.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:05.361+0000 7f2fcea00640 1 --2- 172.21.15.6:0/2994076245 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2fd00af580 0x7f2fd014e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:35:05.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:05.361+0000 7f2fcf400640 1 --2- 172.21.15.6:0/2994076245 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2fd00af1a0 0x7f2fd009aa10 secure :-1 s=READY pgs=1684 cs=0 l=1 rev1=1 crypto rx=0x7f2fc000c430 tx=0x7f2fc000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:35:05.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:05.362+0000 7f2fc7400640 1 -- 172.21.15.6:0/2994076245 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fc0011790 con 0x7f2fd00af1a0 2024-03-04T09:35:05.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:05.362+0000 7f2fc7400640 1 -- 172.21.15.6:0/2994076245 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fc0017070 con 0x7f2fd00af1a0 2024-03-04T09:35:05.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:05.362+0000 7f2fc7400640 1 -- 172.21.15.6:0/2994076245 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fc0011bb0 con 0x7f2fd00af1a0 2024-03-04T09:35:05.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:05.362+0000 7f2fd6400640 1 -- 172.21.15.6:0/2994076245 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fd014f440 con 0x7f2fd00af1a0 2024-03-04T09:35:05.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:05.362+0000 7f2fd6400640 1 -- 172.21.15.6:0/2994076245 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fd009af50 con 0x7f2fd00af1a0 2024-03-04T09:35:05.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:05.363+0000 7f2fd6400640 1 -- 172.21.15.6:0/2994076245 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fd004f3a0 con 0x7f2fd00af1a0 2024-03-04T09:35:05.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:05.364+0000 7f2fc7400640 1 -- 172.21.15.6:0/2994076245 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2fc001c460 con 0x7f2fd00af1a0 2024-03-04T09:35:05.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:05.366+0000 7f2fc7400640 1 --2- 172.21.15.6:0/2994076245 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2fb006b4d0 0x7f2fb006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:05.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:05.366+0000 7f2fc7400640 1 -- 172.21.15.6:0/2994076245 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f2fc008a260 con 0x7f2fd00af1a0 2024-03-04T09:35:05.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:05.366+0000 7f2fcea00640 1 --2- 172.21.15.6:0/2994076245 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2fb006b4d0 0x7f2fb006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:05.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:05.368+0000 7f2fcea00640 1 --2- 172.21.15.6:0/2994076245 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2fb006b4d0 0x7f2fb006d990 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f2fbc036000 tx=0x7f2fbc008030 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:35:05.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:05.370+0000 7f2fc7400640 1 -- 172.21.15.6:0/2994076245 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2fc0058f70 con 0x7f2fd00af1a0 2024-03-04T09:35:05.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:05.614+0000 7f2fd6400640 1 -- 172.21.15.6:0/2994076245 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2fd00cd4c0 con 0x7f2fd00af1a0 2024-03-04T09:35:05.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:05.615+0000 7f2fc7400640 1 -- 172.21.15.6:0/2994076245 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3465 (secure 0 0 0) 0x7f2fc0058870 con 0x7f2fd00af1a0 2024-03-04T09:35:05.612 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:35:05.613 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:33:54.652601+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"laggy_since":"2024-03-04T09:33:54.652601+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":9}]} 2024-03-04T09:35:05.613 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 100 2024-03-04T09:35:05.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:05.618+0000 7f2fd6400640 1 -- 172.21.15.6:0/2994076245 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2fb006b4d0 msgr2=0x7f2fb006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:05.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:05.618+0000 7f2fd6400640 1 --2- 172.21.15.6:0/2994076245 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2fb006b4d0 0x7f2fb006d990 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f2fbc036000 tx=0x7f2fbc008030 comp rx=0 tx=0).stop 2024-03-04T09:35:05.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:05.618+0000 7f2fd6400640 1 -- 172.21.15.6:0/2994076245 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2fd00af1a0 msgr2=0x7f2fd009aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:05.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:05.618+0000 7f2fd6400640 1 --2- 172.21.15.6:0/2994076245 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2fd00af1a0 0x7f2fd009aa10 secure :-1 s=READY pgs=1684 cs=0 l=1 rev1=1 crypto rx=0x7f2fc000c430 tx=0x7f2fc000c8f0 comp rx=0 tx=0).stop 2024-03-04T09:35:05.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:05.618+0000 7f2fd6400640 1 -- 172.21.15.6:0/2994076245 shutdown_connections 2024-03-04T09:35:05.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:05.618+0000 7f2fd6400640 1 --2- 172.21.15.6:0/2994076245 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2fb006b4d0 0x7f2fb006d990 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:05.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:05.618+0000 7f2fd6400640 1 --2- 172.21.15.6:0/2994076245 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2fd00cdc20 0x7f2fd0071ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:05.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:05.618+0000 7f2fd6400640 1 --2- 172.21.15.6:0/2994076245 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2fd00af580 0x7f2fd014e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:05.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:05.618+0000 7f2fd6400640 1 --2- 172.21.15.6:0/2994076245 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2fd00af1a0 0x7f2fd009aa10 unknown :-1 s=CLOSED pgs=1684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:05.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:05.618+0000 7f2fd6400640 1 -- 172.21.15.6:0/2994076245 >> 172.21.15.6:0/2994076245 conn(0x7f2fd006afb0 msgr2=0x7f2fd00ae5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:35:05.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:05.618+0000 7f2fd6400640 1 -- 172.21.15.6:0/2994076245 shutdown_connections 2024-03-04T09:35:05.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:05.618+0000 7f2fd6400640 1 -- 172.21.15.6:0/2994076245 wait complete. 2024-03-04T09:35:05.834 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:32:57.159681+0000', 'modified': '2024-03-04T09:33:54.652601+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8214, 'mds_1': 6796}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6796': {'gid': 6796, 'name': 'd', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.146:6837/2458070840', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 2458070840}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 2458070840}]}, 'join_fscid': -1, 'laggy_since': '2024-03-04T09:33:54.652601+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8214': {'gid': 8214, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2446627517', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2446627517}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2446627517}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8214, 'qdb_cluster': [8214]} 2024-03-04T09:35:05.834 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6796:up:stopping 2024-03-04T09:35:05.866 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-04T09:35:06.835 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:35:07.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:07.063+0000 7fe1fbc00640 1 Processor -- start 2024-03-04T09:35:07.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:07.063+0000 7fe1fbc00640 1 -- start start 2024-03-04T09:35:07.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:07.066+0000 7fe1fbc00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe1f40b34e0 0x7fe1f415c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:07.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:07.066+0000 7fe1fbc00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1f415cc40 con 0x7fe1f40af580 2024-03-04T09:35:07.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:07.066+0000 7fe1fbc00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1f415cf20 con 0x7fe1f40af1a0 2024-03-04T09:35:07.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:07.066+0000 7fe1fbc00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1f415d200 con 0x7fe1f40b34e0 2024-03-04T09:35:07.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:07.066+0000 7fe1f3e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fe1f40af580 0x7fe1f40b2dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:46280/0 (socket says 172.21.15.6:46280) 2024-03-04T09:35:07.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:07.066+0000 7fe1f3e00640 1 -- 172.21.15.6:0/3981626732 learned_addr learned my addr 172.21.15.6:0/3981626732 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:35:07.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:07.067+0000 7fe1f3400640 1 -- 172.21.15.6:0/3981626732 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3768666959 0 0) 0x7fe1f415cc40 con 0x7fe1f40af580 2024-03-04T09:35:07.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:07.067+0000 7fe1f9c00640 1 --2- 172.21.15.6:0/3981626732 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe1f40b34e0 0x7fe1f415c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:07.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:07.067+0000 7fe1f3400640 1 -- 172.21.15.6:0/3981626732 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1d8003520 con 0x7fe1f40af580 2024-03-04T09:35:07.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:07.067+0000 7fe1f3400640 1 -- 172.21.15.6:0/3981626732 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3193423746 0 0) 0x7fe1d8003520 con 0x7fe1f40af580 2024-03-04T09:35:07.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:07.067+0000 7fe1f3400640 1 -- 172.21.15.6:0/3981626732 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe1f40b34e0 msgr2=0x7fe1f415c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:35:07.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:07.067+0000 7fe1f3400640 1 --2- 172.21.15.6:0/3981626732 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe1f40b34e0 0x7fe1f415c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:07.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:07.067+0000 7fe1f3400640 1 -- 172.21.15.6:0/3981626732 >> v1:172.21.15.146:6789/0 conn(0x7fe1f40af1a0 legacy=0x7fe1f40b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:07.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:07.067+0000 7fe1f3400640 1 -- 172.21.15.6:0/3981626732 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1f415d4e0 con 0x7fe1f40af580 2024-03-04T09:35:07.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:07.067+0000 7fe1f3400640 1 -- 172.21.15.6:0/3981626732 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7fe1dc001670 con 0x7fe1f40af580 2024-03-04T09:35:07.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:07.067+0000 7fe1f3400640 1 -- 172.21.15.6:0/3981626732 >> v1:172.21.15.6:6789/0 conn(0x7fe1f40af580 legacy=0x7fe1f40b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:07.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:07.068+0000 7fe1f3400640 1 --2- 172.21.15.6:0/3981626732 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe1f40b34e0 0x7fe1d8003a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:07.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:07.068+0000 7fe1f3400640 1 --2- 172.21.15.6:0/3981626732 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe1d80040d0 0x7fe1d8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:07.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:07.068+0000 7fe1f3400640 1 --2- 172.21.15.6:0/3981626732 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe1d8004db0 0x7fe1d8005230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:07.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:07.068+0000 7fe1f3400640 1 -- 172.21.15.6:0/3981626732 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1f415d200 con 0x7fe1d8004db0 2024-03-04T09:35:07.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:07.068+0000 7fe1f3400640 1 -- 172.21.15.6:0/3981626732 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1d800bcc0 con 0x7fe1d80040d0 2024-03-04T09:35:07.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:07.068+0000 7fe1f3400640 1 -- 172.21.15.6:0/3981626732 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1d800bec0 con 0x7fe1f40b34e0 2024-03-04T09:35:07.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:07.068+0000 7fe1f9c00640 1 --2- 172.21.15.6:0/3981626732 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe1d8004db0 0x7fe1d8005230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:07.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:07.068+0000 7fe1f9200640 1 --2- 172.21.15.6:0/3981626732 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe1f40b34e0 0x7fe1d8003a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:07.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:07.068+0000 7fe1f3e00640 1 --2- 172.21.15.6:0/3981626732 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe1d80040d0 0x7fe1d8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:07.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:07.068+0000 7fe1f9c00640 1 -- 172.21.15.6:0/3981626732 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe1f40b34e0 msgr2=0x7fe1d8003a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:35:07.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:07.068+0000 7fe1f9c00640 1 --2- 172.21.15.6:0/3981626732 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe1f40b34e0 0x7fe1d8003a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:07.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:07.068+0000 7fe1f9c00640 1 -- 172.21.15.6:0/3981626732 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe1d80040d0 msgr2=0x7fe1d8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:35:07.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:07.068+0000 7fe1f9c00640 1 --2- 172.21.15.6:0/3981626732 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe1d80040d0 0x7fe1d8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:07.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:07.068+0000 7fe1f9c00640 1 -- 172.21.15.6:0/3981626732 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1d8003520 con 0x7fe1d8004db0 2024-03-04T09:35:07.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:07.070+0000 7fe1f9c00640 1 --2- 172.21.15.6:0/3981626732 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe1d8004db0 0x7fe1d8005230 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 crypto rx=0x7fe1ec00ba10 tx=0x7fe1d8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aaab4e35fe8cf9dd server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:35:07.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:07.070+0000 7fe1f3400640 1 -- 172.21.15.6:0/3981626732 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1ec00b610 con 0x7fe1d8004db0 2024-03-04T09:35:07.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:07.070+0000 7fe1f3400640 1 -- 172.21.15.6:0/3981626732 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1ec038070 con 0x7fe1d8004db0 2024-03-04T09:35:07.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:07.070+0000 7fe1f3400640 1 -- 172.21.15.6:0/3981626732 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1ec033830 con 0x7fe1d8004db0 2024-03-04T09:35:07.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:07.070+0000 7fe1fbc00640 1 -- 172.21.15.6:0/3981626732 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe1d8004db0 msgr2=0x7fe1d8005230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:07.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:07.070+0000 7fe1fbc00640 1 --2- 172.21.15.6:0/3981626732 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe1d8004db0 0x7fe1d8005230 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 crypto rx=0x7fe1ec00ba10 tx=0x7fe1d8005830 comp rx=0 tx=0).stop 2024-03-04T09:35:07.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:07.070+0000 7fe1f9c00640 1 -- 172.21.15.6:0/3981626732 reap_dead start 2024-03-04T09:35:07.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:07.071+0000 7fe1fbc00640 1 -- 172.21.15.6:0/3981626732 shutdown_connections 2024-03-04T09:35:07.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:07.071+0000 7fe1fbc00640 1 -- 172.21.15.6:0/3981626732 >> 172.21.15.6:0/3981626732 conn(0x7fe1f406afb0 msgr2=0x7fe1f406b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:35:07.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:07.071+0000 7fe1fbc00640 1 -- 172.21.15.6:0/3981626732 shutdown_connections 2024-03-04T09:35:07.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:07.071+0000 7fe1fbc00640 1 -- 172.21.15.6:0/3981626732 wait complete. 2024-03-04T09:35:07.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:07.073+0000 7fe1fbc00640 1 Processor -- start 2024-03-04T09:35:07.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:07.073+0000 7fe1fbc00640 1 -- start start 2024-03-04T09:35:07.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:07.074+0000 7fe1fbc00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe1d8004db0 0x7fe1f41f47c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:07.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:07.074+0000 7fe1fbc00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe1f41502b0 0x7fe1f4158be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:07.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:07.074+0000 7fe1fbc00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe1f4150730 0x7fe1f4159120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:07.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:07.074+0000 7fe1fbc00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1f415c600 con 0x7fe1f4150730 2024-03-04T09:35:07.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:07.074+0000 7fe1fbc00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1f40cde30 con 0x7fe1d8004db0 2024-03-04T09:35:07.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:07.074+0000 7fe1fbc00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1f415c780 con 0x7fe1f41502b0 2024-03-04T09:35:07.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:07.074+0000 7fe1f9c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe1f4150730 0x7fe1f4159120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:07.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:07.074+0000 7fe1f9c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe1f4150730 0x7fe1f4159120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42540/0 (socket says 172.21.15.6:42540) 2024-03-04T09:35:07.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:07.074+0000 7fe1f9c00640 1 -- 172.21.15.6:0/2966874656 learned_addr learned my addr 172.21.15.6:0/2966874656 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:35:07.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:07.074+0000 7fe1f9200640 1 --2- 172.21.15.6:0/2966874656 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe1d8004db0 0x7fe1f41f47c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:07.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:07.074+0000 7fe1f3e00640 1 --2- 172.21.15.6:0/2966874656 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe1f41502b0 0x7fe1f4158be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:07.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:07.075+0000 7fe1f9c00640 1 -- 172.21.15.6:0/2966874656 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe1f41502b0 msgr2=0x7fe1f4158be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:07.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:07.075+0000 7fe1f9c00640 1 --2- 172.21.15.6:0/2966874656 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe1f41502b0 0x7fe1f4158be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:07.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:07.075+0000 7fe1f9c00640 1 -- 172.21.15.6:0/2966874656 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe1d8004db0 msgr2=0x7fe1f41f47c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:07.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:07.075+0000 7fe1f9c00640 1 --2- 172.21.15.6:0/2966874656 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe1d8004db0 0x7fe1f41f47c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:07.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:07.075+0000 7fe1f9c00640 1 -- 172.21.15.6:0/2966874656 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1f41f4d00 con 0x7fe1f4150730 2024-03-04T09:35:07.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:07.075+0000 7fe1f9c00640 1 --2- 172.21.15.6:0/2966874656 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe1f4150730 0x7fe1f4159120 secure :-1 s=READY pgs=1687 cs=0 l=1 rev1=1 crypto rx=0x7fe1ec00ba70 tx=0x7fe1ec004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:35:07.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:07.075+0000 7fe1f0c00640 1 -- 172.21.15.6:0/2966874656 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1ec033df0 con 0x7fe1f4150730 2024-03-04T09:35:07.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:07.075+0000 7fe1f0c00640 1 -- 172.21.15.6:0/2966874656 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1ec040070 con 0x7fe1f4150730 2024-03-04T09:35:07.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:07.075+0000 7fe1f0c00640 1 -- 172.21.15.6:0/2966874656 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1ec047630 con 0x7fe1f4150730 2024-03-04T09:35:07.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:07.075+0000 7fe1fbc00640 1 -- 172.21.15.6:0/2966874656 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1f4157990 con 0x7fe1f4150730 2024-03-04T09:35:07.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:07.076+0000 7fe1fbc00640 1 -- 172.21.15.6:0/2966874656 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1f4157f50 con 0x7fe1f4150730 2024-03-04T09:35:07.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:07.077+0000 7fe1fbc00640 1 -- 172.21.15.6:0/2966874656 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1f404f3a0 con 0x7fe1f4150730 2024-03-04T09:35:07.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:07.078+0000 7fe1f0c00640 1 -- 172.21.15.6:0/2966874656 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe1ec0477f0 con 0x7fe1f4150730 2024-03-04T09:35:07.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:07.079+0000 7fe1f0c00640 1 --2- 172.21.15.6:0/2966874656 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fe1d006b4d0 0x7fe1d006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:07.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:07.079+0000 7fe1f0c00640 1 -- 172.21.15.6:0/2966874656 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fe1ec0ac0c0 con 0x7fe1f4150730 2024-03-04T09:35:07.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:07.080+0000 7fe1f9200640 1 --2- 172.21.15.6:0/2966874656 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fe1d006b4d0 0x7fe1d006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:07.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:07.081+0000 7fe1f9200640 1 --2- 172.21.15.6:0/2966874656 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fe1d006b4d0 0x7fe1d006d990 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fe1f40b19d0 tx=0x7fe1e4009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:35:07.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:07.084+0000 7fe1f0c00640 1 -- 172.21.15.6:0/2966874656 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe1ec04c020 con 0x7fe1f4150730 2024-03-04T09:35:07.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:07.319+0000 7fe1fbc00640 1 -- 172.21.15.6:0/2966874656 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe1f4074130 con 0x7fe1f4150730 2024-03-04T09:35:07.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:07.320+0000 7fe1f0c00640 1 -- 172.21.15.6:0/2966874656 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3465 (secure 0 0 0) 0x7fe1f41f4d00 con 0x7fe1f4150730 2024-03-04T09:35:07.318 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:35:07.318 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:33:54.652601+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"laggy_since":"2024-03-04T09:33:54.652601+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":9}]} 2024-03-04T09:35:07.318 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 100 2024-03-04T09:35:07.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:07.322+0000 7fe1fbc00640 1 -- 172.21.15.6:0/2966874656 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fe1d006b4d0 msgr2=0x7fe1d006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:07.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:07.322+0000 7fe1fbc00640 1 --2- 172.21.15.6:0/2966874656 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fe1d006b4d0 0x7fe1d006d990 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fe1f40b19d0 tx=0x7fe1e4009040 comp rx=0 tx=0).stop 2024-03-04T09:35:07.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:07.322+0000 7fe1fbc00640 1 -- 172.21.15.6:0/2966874656 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe1f4150730 msgr2=0x7fe1f4159120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:07.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:07.322+0000 7fe1fbc00640 1 --2- 172.21.15.6:0/2966874656 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe1f4150730 0x7fe1f4159120 secure :-1 s=READY pgs=1687 cs=0 l=1 rev1=1 crypto rx=0x7fe1ec00ba70 tx=0x7fe1ec004730 comp rx=0 tx=0).stop 2024-03-04T09:35:07.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:07.323+0000 7fe1fbc00640 1 -- 172.21.15.6:0/2966874656 shutdown_connections 2024-03-04T09:35:07.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:07.323+0000 7fe1fbc00640 1 --2- 172.21.15.6:0/2966874656 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe1f4150730 0x7fe1f4159120 unknown :-1 s=CLOSED pgs=1687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:07.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:07.323+0000 7fe1fbc00640 1 --2- 172.21.15.6:0/2966874656 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fe1d006b4d0 0x7fe1d006d990 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:07.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:07.323+0000 7fe1fbc00640 1 --2- 172.21.15.6:0/2966874656 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe1f41502b0 0x7fe1f4158be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:07.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:07.323+0000 7fe1fbc00640 1 --2- 172.21.15.6:0/2966874656 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe1d8004db0 0x7fe1f41f47c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:07.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:07.323+0000 7fe1fbc00640 1 -- 172.21.15.6:0/2966874656 >> 172.21.15.6:0/2966874656 conn(0x7fe1f406afb0 msgr2=0x7fe1f40b2db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:35:07.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:07.323+0000 7fe1fbc00640 1 -- 172.21.15.6:0/2966874656 shutdown_connections 2024-03-04T09:35:07.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:07.323+0000 7fe1fbc00640 1 -- 172.21.15.6:0/2966874656 wait complete. 2024-03-04T09:35:07.540 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:32:57.159681+0000', 'modified': '2024-03-04T09:33:54.652601+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8214, 'mds_1': 6796}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6796': {'gid': 6796, 'name': 'd', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.146:6837/2458070840', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 2458070840}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 2458070840}]}, 'join_fscid': -1, 'laggy_since': '2024-03-04T09:33:54.652601+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8214': {'gid': 8214, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2446627517', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2446627517}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2446627517}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8214, 'qdb_cluster': [8214]} 2024-03-04T09:35:07.540 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6796:up:stopping 2024-03-04T09:35:07.869 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-04T09:35:08.541 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:35:08.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:08.777+0000 7fa2d2000640 1 Processor -- start 2024-03-04T09:35:08.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:08.777+0000 7fa2d2000640 1 -- start start 2024-03-04T09:35:08.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:08.778+0000 7fa2d2000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fa2cc0af1a0 0x7fa2cc15c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:08.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:08.778+0000 7fa2d2000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa2cc0cdc20 0x7fa2cc0b2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:08.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:08.779+0000 7fa2d2000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2cc15cc40 con 0x7fa2cc0af580 2024-03-04T09:35:08.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:08.779+0000 7fa2d2000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2cc15cf20 con 0x7fa2cc0cdc20 2024-03-04T09:35:08.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:08.779+0000 7fa2d2000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2cc15d200 con 0x7fa2cc0af1a0 2024-03-04T09:35:08.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:08.779+0000 7fa2cbe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa2cc0cdc20 0x7fa2cc0b2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:08.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:08.779+0000 7fa2cbe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa2cc0cdc20 0x7fa2cc0b2dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42556/0 (socket says 172.21.15.6:42556) 2024-03-04T09:35:08.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:08.779+0000 7fa2cbe00640 1 -- 172.21.15.6:0/740443475 learned_addr learned my addr 172.21.15.6:0/740443475 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:35:08.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:08.779+0000 7fa2cb400640 1 --2- 172.21.15.6:0/740443475 >> v2:172.21.15.146:3300/0 conn(0x7fa2cc0af1a0 0x7fa2cc15c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:08.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:08.780+0000 7fa2cbe00640 1 -- 172.21.15.6:0/740443475 >> v2:172.21.15.146:3300/0 conn(0x7fa2cc0af1a0 msgr2=0x7fa2cc15c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:35:08.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:08.780+0000 7fa2cbe00640 1 --2- 172.21.15.6:0/740443475 >> v2:172.21.15.146:3300/0 conn(0x7fa2cc0af1a0 0x7fa2cc15c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:08.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:08.780+0000 7fa2cbe00640 1 -- 172.21.15.6:0/740443475 >> v1:172.21.15.146:6789/0 conn(0x7fa2cc0af580 legacy=0x7fa2cc0b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:35:08.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:08.780+0000 7fa2cbe00640 1 -- 172.21.15.6:0/740443475 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2cc15d4e0 con 0x7fa2cc0cdc20 2024-03-04T09:35:08.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:08.781+0000 7fa2cbe00640 1 --2- 172.21.15.6:0/740443475 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa2cc0cdc20 0x7fa2cc0b2dd0 secure :-1 s=READY pgs=1688 cs=0 l=1 rev1=1 crypto rx=0x7fa2cc074960 tx=0x7fa2c402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85b4ef143ca6ac79 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:35:08.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:08.781+0000 7fa2caa00640 1 -- 172.21.15.6:0/740443475 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2c402e070 con 0x7fa2cc0cdc20 2024-03-04T09:35:08.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:08.781+0000 7fa2caa00640 1 -- 172.21.15.6:0/740443475 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2c4037070 con 0x7fa2cc0cdc20 2024-03-04T09:35:08.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:08.781+0000 7fa2caa00640 1 -- 172.21.15.6:0/740443475 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2c4031a00 con 0x7fa2cc0cdc20 2024-03-04T09:35:08.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:08.781+0000 7fa2d2000640 1 -- 172.21.15.6:0/740443475 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa2cc0cdc20 msgr2=0x7fa2cc0b2dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:08.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:08.781+0000 7fa2d2000640 1 --2- 172.21.15.6:0/740443475 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa2cc0cdc20 0x7fa2cc0b2dd0 secure :-1 s=READY pgs=1688 cs=0 l=1 rev1=1 crypto rx=0x7fa2cc074960 tx=0x7fa2c402b880 comp rx=0 tx=0).stop 2024-03-04T09:35:08.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:08.782+0000 7fa2d2000640 1 -- 172.21.15.6:0/740443475 shutdown_connections 2024-03-04T09:35:08.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:08.782+0000 7fa2d2000640 1 --2- 172.21.15.6:0/740443475 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa2cc0cdc20 0x7fa2cc0b2dd0 unknown :-1 s=CLOSED pgs=1688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:08.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:08.782+0000 7fa2d2000640 1 --2- 172.21.15.6:0/740443475 >> v2:172.21.15.146:3300/0 conn(0x7fa2cc0af1a0 0x7fa2cc15c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:08.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:08.782+0000 7fa2d2000640 1 -- 172.21.15.6:0/740443475 >> 172.21.15.6:0/740443475 conn(0x7fa2cc06afb0 msgr2=0x7fa2cc06b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:35:08.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:08.782+0000 7fa2d2000640 1 -- 172.21.15.6:0/740443475 shutdown_connections 2024-03-04T09:35:08.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:08.782+0000 7fa2d2000640 1 -- 172.21.15.6:0/740443475 wait complete. 2024-03-04T09:35:08.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:08.783+0000 7fa2d2000640 1 Processor -- start 2024-03-04T09:35:08.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:08.784+0000 7fa2d2000640 1 -- start start 2024-03-04T09:35:08.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:08.784+0000 7fa2d2000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa2cc0af1a0 0x7fa2cc1f46b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:08.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:08.784+0000 7fa2d2000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa2cc0af580 0x7fa2cc1f1790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:08.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:08.784+0000 7fa2d2000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa2cc0cdc20 0x7fa2cc09b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:08.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:08.785+0000 7fa2d2000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2cc0b36a0 con 0x7fa2cc0af1a0 2024-03-04T09:35:08.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:08.785+0000 7fa2d2000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2cc0b3520 con 0x7fa2cc0cdc20 2024-03-04T09:35:08.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:08.785+0000 7fa2d2000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2cc0b3820 con 0x7fa2cc0af580 2024-03-04T09:35:08.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:08.785+0000 7fa2cb400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa2cc0af1a0 0x7fa2cc1f46b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:08.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:08.785+0000 7fa2cb400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa2cc0af1a0 0x7fa2cc1f46b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42572/0 (socket says 172.21.15.6:42572) 2024-03-04T09:35:08.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:08.785+0000 7fa2cb400640 1 -- 172.21.15.6:0/1967332375 learned_addr learned my addr 172.21.15.6:0/1967332375 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:35:08.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:08.785+0000 7fa2cbe00640 1 --2- 172.21.15.6:0/1967332375 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa2cc0cdc20 0x7fa2cc09b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:08.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:08.785+0000 7fa2cb400640 1 -- 172.21.15.6:0/1967332375 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa2cc0af580 msgr2=0x7fa2cc1f1790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:08.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:08.785+0000 7fa2c2a00640 1 --2- 172.21.15.6:0/1967332375 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa2cc0af580 0x7fa2cc1f1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:08.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:08.785+0000 7fa2cb400640 1 --2- 172.21.15.6:0/1967332375 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa2cc0af580 0x7fa2cc1f1790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:08.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:08.785+0000 7fa2cb400640 1 -- 172.21.15.6:0/1967332375 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa2cc0cdc20 msgr2=0x7fa2cc09b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:08.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:08.785+0000 7fa2cb400640 1 --2- 172.21.15.6:0/1967332375 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa2cc0cdc20 0x7fa2cc09b420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:08.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:08.785+0000 7fa2cb400640 1 -- 172.21.15.6:0/1967332375 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2cc1f1cd0 con 0x7fa2cc0af1a0 2024-03-04T09:35:08.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:08.785+0000 7fa2cbe00640 1 --2- 172.21.15.6:0/1967332375 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa2cc0cdc20 0x7fa2cc09b420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:35:08.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:08.786+0000 7fa2cb400640 1 --2- 172.21.15.6:0/1967332375 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa2cc0af1a0 0x7fa2cc1f46b0 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 crypto rx=0x7fa2bc007ef0 tx=0x7fa2bc00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:35:08.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:08.786+0000 7fa2c3e00640 1 -- 172.21.15.6:0/1967332375 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2bc014620 con 0x7fa2cc0af1a0 2024-03-04T09:35:08.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:08.786+0000 7fa2c3e00640 1 -- 172.21.15.6:0/1967332375 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2bc019070 con 0x7fa2cc0af1a0 2024-03-04T09:35:08.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:08.786+0000 7fa2c3e00640 1 -- 172.21.15.6:0/1967332375 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2bc014a00 con 0x7fa2cc0af1a0 2024-03-04T09:35:08.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:08.786+0000 7fa2d2000640 1 -- 172.21.15.6:0/1967332375 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2cc1584f0 con 0x7fa2cc0af1a0 2024-03-04T09:35:08.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:08.786+0000 7fa2d2000640 1 -- 172.21.15.6:0/1967332375 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2cc158a90 con 0x7fa2cc0af1a0 2024-03-04T09:35:08.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:08.787+0000 7fa2d2000640 1 -- 172.21.15.6:0/1967332375 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2cc069510 con 0x7fa2cc0af1a0 2024-03-04T09:35:08.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:08.788+0000 7fa2c3e00640 1 -- 172.21.15.6:0/1967332375 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa2bc014ba0 con 0x7fa2cc0af1a0 2024-03-04T09:35:08.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:08.789+0000 7fa2c3e00640 1 --2- 172.21.15.6:0/1967332375 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa2a806b4d0 0x7fa2a806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:08.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:08.789+0000 7fa2c3e00640 1 -- 172.21.15.6:0/1967332375 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fa2bc08ab90 con 0x7fa2cc0af1a0 2024-03-04T09:35:08.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:08.790+0000 7fa2c2a00640 1 --2- 172.21.15.6:0/1967332375 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa2a806b4d0 0x7fa2a806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:08.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:08.791+0000 7fa2c2a00640 1 --2- 172.21.15.6:0/1967332375 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa2a806b4d0 0x7fa2a806d990 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fa2cc0aebf0 tx=0x7fa2b4009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:35:08.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:08.793+0000 7fa2c3e00640 1 -- 172.21.15.6:0/1967332375 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa2bc0599c0 con 0x7fa2cc0af1a0 2024-03-04T09:35:09.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:09.035+0000 7fa2d2000640 1 -- 172.21.15.6:0/1967332375 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa2cc04f3a0 con 0x7fa2cc0af1a0 2024-03-04T09:35:09.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:09.036+0000 7fa2c3e00640 1 -- 172.21.15.6:0/1967332375 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3465 (secure 0 0 0) 0x7fa2bc0592c0 con 0x7fa2cc0af1a0 2024-03-04T09:35:09.034 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:35:09.034 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:33:54.652601+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"laggy_since":"2024-03-04T09:33:54.652601+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":9}]} 2024-03-04T09:35:09.034 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 100 2024-03-04T09:35:09.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:09.039+0000 7fa2d2000640 1 -- 172.21.15.6:0/1967332375 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa2a806b4d0 msgr2=0x7fa2a806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:09.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:09.039+0000 7fa2d2000640 1 --2- 172.21.15.6:0/1967332375 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa2a806b4d0 0x7fa2a806d990 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fa2cc0aebf0 tx=0x7fa2b4009040 comp rx=0 tx=0).stop 2024-03-04T09:35:09.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:09.039+0000 7fa2d2000640 1 -- 172.21.15.6:0/1967332375 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa2cc0af1a0 msgr2=0x7fa2cc1f46b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:09.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:09.039+0000 7fa2d2000640 1 --2- 172.21.15.6:0/1967332375 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa2cc0af1a0 0x7fa2cc1f46b0 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 crypto rx=0x7fa2bc007ef0 tx=0x7fa2bc00e4b0 comp rx=0 tx=0).stop 2024-03-04T09:35:09.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:09.039+0000 7fa2d2000640 1 -- 172.21.15.6:0/1967332375 shutdown_connections 2024-03-04T09:35:09.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:09.039+0000 7fa2d2000640 1 --2- 172.21.15.6:0/1967332375 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa2cc0cdc20 0x7fa2cc09b420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:09.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:09.039+0000 7fa2d2000640 1 --2- 172.21.15.6:0/1967332375 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa2a806b4d0 0x7fa2a806d990 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:09.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:09.039+0000 7fa2d2000640 1 --2- 172.21.15.6:0/1967332375 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa2cc0af580 0x7fa2cc1f1790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:09.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:09.039+0000 7fa2d2000640 1 --2- 172.21.15.6:0/1967332375 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa2cc0af1a0 0x7fa2cc1f46b0 unknown :-1 s=CLOSED pgs=1689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:09.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:09.039+0000 7fa2d2000640 1 -- 172.21.15.6:0/1967332375 >> 172.21.15.6:0/1967332375 conn(0x7fa2cc06afb0 msgr2=0x7fa2cc15c080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:35:09.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:09.039+0000 7fa2d2000640 1 -- 172.21.15.6:0/1967332375 shutdown_connections 2024-03-04T09:35:09.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:09.039+0000 7fa2d2000640 1 -- 172.21.15.6:0/1967332375 wait complete. 2024-03-04T09:35:09.257 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:32:57.159681+0000', 'modified': '2024-03-04T09:33:54.652601+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8214, 'mds_1': 6796}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6796': {'gid': 6796, 'name': 'd', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.146:6837/2458070840', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 2458070840}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 2458070840}]}, 'join_fscid': -1, 'laggy_since': '2024-03-04T09:33:54.652601+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8214': {'gid': 8214, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2446627517', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2446627517}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2446627517}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8214, 'qdb_cluster': [8214]} 2024-03-04T09:35:09.258 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6796:up:stopping 2024-03-04T09:35:09.871 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-04T09:35:10.259 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:35:10.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:10.482+0000 7fe316000640 1 Processor -- start 2024-03-04T09:35:10.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:10.482+0000 7fe316000640 1 -- start start 2024-03-04T09:35:10.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:10.483+0000 7fe316000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fe310072da0 0x7fe310162c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:10.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:10.483+0000 7fe316000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fe3100712c0 0x7fe31006e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:10.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:10.483+0000 7fe316000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3100ce090 con 0x7fe310070ee0 2024-03-04T09:35:10.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:10.483+0000 7fe316000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3100ceca0 con 0x7fe310072da0 2024-03-04T09:35:10.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:10.483+0000 7fe316000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3100ce370 con 0x7fe3100712c0 2024-03-04T09:35:10.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:10.484+0000 7fe30ea00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fe310072da0 0x7fe310162c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:10.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:10.484+0000 7fe30f400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fe310070ee0 0x7fe31006e0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:39190/0 (socket says 172.21.15.6:39190) 2024-03-04T09:35:10.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:10.484+0000 7fe30f400640 1 -- 172.21.15.6:0/2932834598 learned_addr learned my addr 172.21.15.6:0/2932834598 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:35:10.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:10.484+0000 7fe30fe00640 1 --2- 172.21.15.6:0/2932834598 >> v2:172.21.15.146:3300/0 conn(0x7fe3100712c0 0x7fe31006e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:10.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:10.484+0000 7fe30ea00640 1 -- 172.21.15.6:0/2932834598 >> v2:172.21.15.146:3300/0 conn(0x7fe3100712c0 msgr2=0x7fe31006e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:35:10.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:10.484+0000 7fe30ea00640 1 --2- 172.21.15.6:0/2932834598 >> v2:172.21.15.146:3300/0 conn(0x7fe3100712c0 0x7fe31006e7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:10.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:10.484+0000 7fe30ea00640 1 -- 172.21.15.6:0/2932834598 >> v1:172.21.15.6:6789/0 conn(0x7fe310070ee0 legacy=0x7fe31006e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:10.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:10.484+0000 7fe30ea00640 1 -- 172.21.15.6:0/2932834598 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3100ce650 con 0x7fe310072da0 2024-03-04T09:35:10.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:10.484+0000 7fe30fe00640 1 --2- 172.21.15.6:0/2932834598 >> v2:172.21.15.146:3300/0 conn(0x7fe3100712c0 0x7fe31006e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:35:10.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:10.486+0000 7fe30ea00640 1 --2- 172.21.15.6:0/2932834598 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe310072da0 0x7fe310162c40 secure :-1 s=READY pgs=1691 cs=0 l=1 rev1=1 crypto rx=0x7fe31006f070 tx=0x7fe2f802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97313370618dd6a6 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:35:10.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:10.486+0000 7fe30e000640 1 -- 172.21.15.6:0/2932834598 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2f8035020 con 0x7fe310072da0 2024-03-04T09:35:10.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:10.486+0000 7fe30e000640 1 -- 172.21.15.6:0/2932834598 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2f8031830 con 0x7fe310072da0 2024-03-04T09:35:10.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:10.486+0000 7fe30e000640 1 -- 172.21.15.6:0/2932834598 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2f8031c00 con 0x7fe310072da0 2024-03-04T09:35:10.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:10.487+0000 7fe316000640 1 -- 172.21.15.6:0/2932834598 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe310072da0 msgr2=0x7fe310162c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:10.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:10.487+0000 7fe316000640 1 --2- 172.21.15.6:0/2932834598 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe310072da0 0x7fe310162c40 secure :-1 s=READY pgs=1691 cs=0 l=1 rev1=1 crypto rx=0x7fe31006f070 tx=0x7fe2f802b8d0 comp rx=0 tx=0).stop 2024-03-04T09:35:10.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:10.487+0000 7fe316000640 1 -- 172.21.15.6:0/2932834598 shutdown_connections 2024-03-04T09:35:10.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:10.487+0000 7fe316000640 1 --2- 172.21.15.6:0/2932834598 >> v2:172.21.15.146:3300/0 conn(0x7fe3100712c0 0x7fe31006e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:10.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:10.487+0000 7fe316000640 1 --2- 172.21.15.6:0/2932834598 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe310072da0 0x7fe310162c40 unknown :-1 s=CLOSED pgs=1691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:10.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:10.487+0000 7fe316000640 1 -- 172.21.15.6:0/2932834598 >> 172.21.15.6:0/2932834598 conn(0x7fe31006afb0 msgr2=0x7fe31006b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:35:10.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:10.487+0000 7fe316000640 1 -- 172.21.15.6:0/2932834598 shutdown_connections 2024-03-04T09:35:10.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:10.487+0000 7fe316000640 1 -- 172.21.15.6:0/2932834598 wait complete. 2024-03-04T09:35:10.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:10.490+0000 7fe316000640 1 Processor -- start 2024-03-04T09:35:10.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:10.490+0000 7fe316000640 1 -- start start 2024-03-04T09:35:10.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:10.492+0000 7fe316000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe310070ee0 0x7fe3100b2b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:10.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:10.492+0000 7fe316000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe3100712c0 0x7fe3100ade00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:10.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:10.492+0000 7fe316000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe310072da0 0x7fe3100ae340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:10.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:10.492+0000 7fe316000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3100cdda0 con 0x7fe310072da0 2024-03-04T09:35:10.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:10.492+0000 7fe316000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3100cdc20 con 0x7fe310070ee0 2024-03-04T09:35:10.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:10.492+0000 7fe316000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3100cf130 con 0x7fe3100712c0 2024-03-04T09:35:10.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:10.492+0000 7fe30fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe310072da0 0x7fe3100ae340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:10.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:10.492+0000 7fe30fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe310072da0 0x7fe3100ae340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45250/0 (socket says 172.21.15.6:45250) 2024-03-04T09:35:10.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:10.492+0000 7fe30fe00640 1 -- 172.21.15.6:0/3839821550 learned_addr learned my addr 172.21.15.6:0/3839821550 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:35:10.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:10.493+0000 7fe30f400640 1 --2- 172.21.15.6:0/3839821550 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe310070ee0 0x7fe3100b2b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:10.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:10.493+0000 7fe30ea00640 1 --2- 172.21.15.6:0/3839821550 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe3100712c0 0x7fe3100ade00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:10.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:10.493+0000 7fe30fe00640 1 -- 172.21.15.6:0/3839821550 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe3100712c0 msgr2=0x7fe3100ade00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:10.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:10.493+0000 7fe30fe00640 1 --2- 172.21.15.6:0/3839821550 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe3100712c0 0x7fe3100ade00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:10.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:10.493+0000 7fe30fe00640 1 -- 172.21.15.6:0/3839821550 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe310070ee0 msgr2=0x7fe3100b2b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:10.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:10.493+0000 7fe30fe00640 1 --2- 172.21.15.6:0/3839821550 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe310070ee0 0x7fe3100b2b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:10.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:10.493+0000 7fe30fe00640 1 -- 172.21.15.6:0/3839821550 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3101f9430 con 0x7fe310072da0 2024-03-04T09:35:10.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:10.493+0000 7fe30fe00640 1 --2- 172.21.15.6:0/3839821550 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe310072da0 0x7fe3100ae340 secure :-1 s=READY pgs=1692 cs=0 l=1 rev1=1 crypto rx=0x7fe30800cef0 tx=0x7fe30800a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:35:10.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:10.494+0000 7fe307400640 1 -- 172.21.15.6:0/3839821550 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe30801a070 con 0x7fe310072da0 2024-03-04T09:35:10.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:10.494+0000 7fe307400640 1 -- 172.21.15.6:0/3839821550 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe30800d860 con 0x7fe310072da0 2024-03-04T09:35:10.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:10.494+0000 7fe307400640 1 -- 172.21.15.6:0/3839821550 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe30800dc50 con 0x7fe310072da0 2024-03-04T09:35:10.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:10.494+0000 7fe316000640 1 -- 172.21.15.6:0/3839821550 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3101f9710 con 0x7fe310072da0 2024-03-04T09:35:10.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:10.494+0000 7fe316000640 1 -- 172.21.15.6:0/3839821550 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3101f9cd0 con 0x7fe310072da0 2024-03-04T09:35:10.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:10.495+0000 7fe316000640 1 -- 172.21.15.6:0/3839821550 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe31004f3a0 con 0x7fe310072da0 2024-03-04T09:35:10.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:10.496+0000 7fe307400640 1 -- 172.21.15.6:0/3839821550 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe30801d070 con 0x7fe310072da0 2024-03-04T09:35:10.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:10.497+0000 7fe307400640 1 --2- 172.21.15.6:0/3839821550 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fe2ec06b4d0 0x7fe2ec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:10.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:10.497+0000 7fe307400640 1 -- 172.21.15.6:0/3839821550 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fe30808ab60 con 0x7fe310072da0 2024-03-04T09:35:10.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:10.497+0000 7fe30f400640 1 --2- 172.21.15.6:0/3839821550 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fe2ec06b4d0 0x7fe2ec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:10.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:10.499+0000 7fe30f400640 1 --2- 172.21.15.6:0/3839821550 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fe2ec06b4d0 0x7fe2ec06d990 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fe300000f60 tx=0x7fe300009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:35:10.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:10.502+0000 7fe307400640 1 -- 172.21.15.6:0/3839821550 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe3080599c0 con 0x7fe310072da0 2024-03-04T09:35:10.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:10.832+0000 7fe316000640 1 -- 172.21.15.6:0/3839821550 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe3100b52d0 con 0x7fe310072da0 2024-03-04T09:35:10.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:10.834+0000 7fe307400640 1 -- 172.21.15.6:0/3839821550 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3465 (secure 0 0 0) 0x7fe3080592c0 con 0x7fe310072da0 2024-03-04T09:35:10.831 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:35:10.831 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:33:54.652601+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"laggy_since":"2024-03-04T09:33:54.652601+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":9}]} 2024-03-04T09:35:10.831 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 100 2024-03-04T09:35:10.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:10.837+0000 7fe316000640 1 -- 172.21.15.6:0/3839821550 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fe2ec06b4d0 msgr2=0x7fe2ec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:10.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:10.837+0000 7fe316000640 1 --2- 172.21.15.6:0/3839821550 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fe2ec06b4d0 0x7fe2ec06d990 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fe300000f60 tx=0x7fe300009040 comp rx=0 tx=0).stop 2024-03-04T09:35:10.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:10.837+0000 7fe316000640 1 -- 172.21.15.6:0/3839821550 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe310072da0 msgr2=0x7fe3100ae340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:10.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:10.837+0000 7fe316000640 1 --2- 172.21.15.6:0/3839821550 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe310072da0 0x7fe3100ae340 secure :-1 s=READY pgs=1692 cs=0 l=1 rev1=1 crypto rx=0x7fe30800cef0 tx=0x7fe30800a710 comp rx=0 tx=0).stop 2024-03-04T09:35:10.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:10.837+0000 7fe316000640 1 -- 172.21.15.6:0/3839821550 shutdown_connections 2024-03-04T09:35:10.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:10.837+0000 7fe316000640 1 --2- 172.21.15.6:0/3839821550 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe310072da0 0x7fe3100ae340 unknown :-1 s=CLOSED pgs=1692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:10.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:10.837+0000 7fe316000640 1 --2- 172.21.15.6:0/3839821550 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fe2ec06b4d0 0x7fe2ec06d990 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:10.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:10.837+0000 7fe316000640 1 --2- 172.21.15.6:0/3839821550 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe3100712c0 0x7fe3100ade00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:10.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:10.837+0000 7fe316000640 1 --2- 172.21.15.6:0/3839821550 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe310070ee0 0x7fe3100b2b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:10.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:10.837+0000 7fe316000640 1 -- 172.21.15.6:0/3839821550 >> 172.21.15.6:0/3839821550 conn(0x7fe31006afb0 msgr2=0x7fe31006b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:35:10.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:10.837+0000 7fe316000640 1 -- 172.21.15.6:0/3839821550 shutdown_connections 2024-03-04T09:35:10.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:10.837+0000 7fe316000640 1 -- 172.21.15.6:0/3839821550 wait complete. 2024-03-04T09:35:11.059 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:32:57.159681+0000', 'modified': '2024-03-04T09:33:54.652601+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8214, 'mds_1': 6796}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6796': {'gid': 6796, 'name': 'd', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.146:6837/2458070840', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 2458070840}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 2458070840}]}, 'join_fscid': -1, 'laggy_since': '2024-03-04T09:33:54.652601+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8214': {'gid': 8214, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2446627517', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2446627517}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2446627517}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8214, 'qdb_cluster': [8214]} 2024-03-04T09:35:11.060 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6796:up:stopping 2024-03-04T09:35:11.265 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:35:11.273 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:35:11.873 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-04T09:35:12.061 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:35:12.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:12.293+0000 7f1aab200640 1 Processor -- start 2024-03-04T09:35:12.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:12.293+0000 7f1aab200640 1 -- start start 2024-03-04T09:35:12.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:12.296+0000 7f1aab200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f1aa415e0b0 0x7f1aa40cf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:12.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:12.296+0000 7f1aab200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f1aa40d4a10 0x7f1aa40d0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:12.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:12.296+0000 7f1aab200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1aa40d0c80 0x7f1aa40d1900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:12.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:12.296+0000 7f1aab200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1aa40b09e0 con 0x7f1aa40d4a10 2024-03-04T09:35:12.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:12.296+0000 7f1aab200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1aa40b3710 con 0x7f1aa415e0b0 2024-03-04T09:35:12.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:12.296+0000 7f1aab200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1aa40b39f0 con 0x7f1aa40d0c80 2024-03-04T09:35:12.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:12.296+0000 7f1aa9400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f1aa40d4a10 0x7f1aa40d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:12.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:12.297+0000 7f1aa9400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f1aa40d4a10 0x7f1aa40d0740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45258/0 (socket says 172.21.15.6:45258) 2024-03-04T09:35:12.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:12.297+0000 7f1aa9400640 1 -- 172.21.15.6:0/2420676194 learned_addr learned my addr 172.21.15.6:0/2420676194 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:35:12.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:12.297+0000 7f1aa9e00640 1 --2- 172.21.15.6:0/2420676194 >> v2:172.21.15.146:3300/0 conn(0x7f1aa415e0b0 0x7f1aa40cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:12.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:12.297+0000 7f1aaa800640 1 --2- 172.21.15.6:0/2420676194 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1aa40d0c80 0x7f1aa40d1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:12.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:12.297+0000 7f1aa9400640 1 -- 172.21.15.6:0/2420676194 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1aa40d0c80 msgr2=0x7f1aa40d1900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:35:12.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:12.297+0000 7f1aa9400640 1 --2- 172.21.15.6:0/2420676194 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1aa40d0c80 0x7f1aa40d1900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:12.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:12.297+0000 7f1aa9400640 1 -- 172.21.15.6:0/2420676194 >> v2:172.21.15.146:3300/0 conn(0x7f1aa415e0b0 msgr2=0x7f1aa40cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:35:12.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:12.297+0000 7f1aa9400640 1 --2- 172.21.15.6:0/2420676194 >> v2:172.21.15.146:3300/0 conn(0x7f1aa415e0b0 0x7f1aa40cf310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:12.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:12.297+0000 7f1aa9400640 1 -- 172.21.15.6:0/2420676194 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1aa40b3cd0 con 0x7f1aa40d4a10 2024-03-04T09:35:12.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:12.298+0000 7f1aa9400640 1 --2- 172.21.15.6:0/2420676194 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1aa40d4a10 0x7f1aa40d0740 secure :-1 s=READY pgs=1693 cs=0 l=1 rev1=1 crypto rx=0x7f1aa40b68c0 tx=0x7f1a9402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b92b642a3acf7eea server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:35:12.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:12.299+0000 7f1aa8a00640 1 -- 172.21.15.6:0/2420676194 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a94035020 con 0x7f1aa40d4a10 2024-03-04T09:35:12.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:12.299+0000 7f1aa8a00640 1 -- 172.21.15.6:0/2420676194 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a94031870 con 0x7f1aa40d4a10 2024-03-04T09:35:12.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:12.299+0000 7f1aa8a00640 1 -- 172.21.15.6:0/2420676194 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a94031c40 con 0x7f1aa40d4a10 2024-03-04T09:35:12.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:12.299+0000 7f1aab200640 1 -- 172.21.15.6:0/2420676194 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1aa40d4a10 msgr2=0x7f1aa40d0740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:12.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:12.299+0000 7f1aab200640 1 --2- 172.21.15.6:0/2420676194 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1aa40d4a10 0x7f1aa40d0740 secure :-1 s=READY pgs=1693 cs=0 l=1 rev1=1 crypto rx=0x7f1aa40b68c0 tx=0x7f1a9402b910 comp rx=0 tx=0).stop 2024-03-04T09:35:12.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:12.299+0000 7f1aab200640 1 -- 172.21.15.6:0/2420676194 shutdown_connections 2024-03-04T09:35:12.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:12.299+0000 7f1aab200640 1 --2- 172.21.15.6:0/2420676194 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1aa40d0c80 0x7f1aa40d1900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:12.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:12.299+0000 7f1aab200640 1 --2- 172.21.15.6:0/2420676194 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1aa40d4a10 0x7f1aa40d0740 unknown :-1 s=CLOSED pgs=1693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:12.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:12.299+0000 7f1aab200640 1 --2- 172.21.15.6:0/2420676194 >> v2:172.21.15.146:3300/0 conn(0x7f1aa415e0b0 0x7f1aa40cf310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:12.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:12.299+0000 7f1aab200640 1 -- 172.21.15.6:0/2420676194 >> 172.21.15.6:0/2420676194 conn(0x7f1aa40d3d30 msgr2=0x7f1aa40d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:35:12.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:12.300+0000 7f1aab200640 1 -- 172.21.15.6:0/2420676194 shutdown_connections 2024-03-04T09:35:12.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:12.300+0000 7f1aab200640 1 -- 172.21.15.6:0/2420676194 wait complete. 2024-03-04T09:35:12.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:12.302+0000 7f1aab200640 1 Processor -- start 2024-03-04T09:35:12.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:12.303+0000 7f1aab200640 1 -- start start 2024-03-04T09:35:12.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:12.304+0000 7f1aab200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1aa40d0c80 0x7f1aa414f160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:12.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:12.304+0000 7f1aab200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1aa40d4a10 0x7f1aa415e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:12.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:12.304+0000 7f1aab200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1aa415e0b0 0x7f1aa40b4640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:12.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:12.304+0000 7f1aab200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1aa40cfc60 con 0x7f1aa415e0b0 2024-03-04T09:35:12.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:12.304+0000 7f1aab200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1aa415e490 con 0x7f1aa40d4a10 2024-03-04T09:35:12.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:12.304+0000 7f1aab200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1aa40cf850 con 0x7f1aa40d0c80 2024-03-04T09:35:12.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:12.305+0000 7f1aaa800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1aa415e0b0 0x7f1aa40b4640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:12.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:12.305+0000 7f1aaa800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1aa415e0b0 0x7f1aa40b4640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45272/0 (socket says 172.21.15.6:45272) 2024-03-04T09:35:12.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:12.305+0000 7f1aaa800640 1 -- 172.21.15.6:0/2021133310 learned_addr learned my addr 172.21.15.6:0/2021133310 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:35:12.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:12.305+0000 7f1aa9e00640 1 --2- 172.21.15.6:0/2021133310 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1aa40d0c80 0x7f1aa414f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:12.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:12.305+0000 7f1aa9400640 1 --2- 172.21.15.6:0/2021133310 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1aa40d4a10 0x7f1aa415e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:12.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:12.305+0000 7f1aaa800640 1 -- 172.21.15.6:0/2021133310 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1aa40d0c80 msgr2=0x7f1aa414f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:12.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:12.305+0000 7f1aaa800640 1 --2- 172.21.15.6:0/2021133310 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1aa40d0c80 0x7f1aa414f160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:12.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:12.305+0000 7f1aaa800640 1 -- 172.21.15.6:0/2021133310 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1aa40d4a10 msgr2=0x7f1aa415e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:12.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:12.305+0000 7f1aaa800640 1 --2- 172.21.15.6:0/2021133310 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1aa40d4a10 0x7f1aa415e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:12.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:12.305+0000 7f1aaa800640 1 -- 172.21.15.6:0/2021133310 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1aa414f6a0 con 0x7f1aa415e0b0 2024-03-04T09:35:12.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:12.306+0000 7f1aaa800640 1 --2- 172.21.15.6:0/2021133310 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1aa415e0b0 0x7f1aa40b4640 secure :-1 s=READY pgs=1694 cs=0 l=1 rev1=1 crypto rx=0x7f1aa0009dd0 tx=0x7f1aa000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:35:12.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:12.306+0000 7f1a9e000640 1 -- 172.21.15.6:0/2021133310 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1aa0018020 con 0x7f1aa415e0b0 2024-03-04T09:35:12.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:12.306+0000 7f1a9e000640 1 -- 172.21.15.6:0/2021133310 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1aa0014480 con 0x7f1aa415e0b0 2024-03-04T09:35:12.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:12.306+0000 7f1a9e000640 1 -- 172.21.15.6:0/2021133310 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1aa0014870 con 0x7f1aa415e0b0 2024-03-04T09:35:12.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:12.306+0000 7f1aab200640 1 -- 172.21.15.6:0/2021133310 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1aa415f0d0 con 0x7f1aa415e0b0 2024-03-04T09:35:12.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:12.306+0000 7f1aab200640 1 -- 172.21.15.6:0/2021133310 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1aa415fa00 con 0x7f1aa415e0b0 2024-03-04T09:35:12.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:12.307+0000 7f1aab200640 1 -- 172.21.15.6:0/2021133310 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1aa404f330 con 0x7f1aa415e0b0 2024-03-04T09:35:12.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:12.308+0000 7f1a9e000640 1 -- 172.21.15.6:0/2021133310 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1aa0005d10 con 0x7f1aa415e0b0 2024-03-04T09:35:12.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:12.309+0000 7f1a9e000640 1 --2- 172.21.15.6:0/2021133310 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f1a8806b4d0 0x7f1a8806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:12.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:12.309+0000 7f1a9e000640 1 -- 172.21.15.6:0/2021133310 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f1aa008af80 con 0x7f1aa415e0b0 2024-03-04T09:35:12.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:12.310+0000 7f1aa9e00640 1 --2- 172.21.15.6:0/2021133310 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f1a8806b4d0 0x7f1a8806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:12.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:12.311+0000 7f1aa9e00640 1 --2- 172.21.15.6:0/2021133310 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f1a8806b4d0 0x7f1a8806d990 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f1a98009a20 tx=0x7f1a98009190 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:35:12.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:12.314+0000 7f1a9e000640 1 -- 172.21.15.6:0/2021133310 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1aa0022030 con 0x7f1aa415e0b0 2024-03-04T09:35:12.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:12.559+0000 7f1aab200640 1 -- 172.21.15.6:0/2021133310 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1aa40cfff0 con 0x7f1aa415e0b0 2024-03-04T09:35:12.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:12.560+0000 7f1a9e000640 1 -- 172.21.15.6:0/2021133310 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3465 (secure 0 0 0) 0x7f1aa0010040 con 0x7f1aa415e0b0 2024-03-04T09:35:12.557 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:35:12.558 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:33:54.652601+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"laggy_since":"2024-03-04T09:33:54.652601+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":9}]} 2024-03-04T09:35:12.558 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 100 2024-03-04T09:35:12.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:12.563+0000 7f1aab200640 1 -- 172.21.15.6:0/2021133310 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f1a8806b4d0 msgr2=0x7f1a8806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:12.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:12.563+0000 7f1aab200640 1 --2- 172.21.15.6:0/2021133310 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f1a8806b4d0 0x7f1a8806d990 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f1a98009a20 tx=0x7f1a98009190 comp rx=0 tx=0).stop 2024-03-04T09:35:12.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:12.563+0000 7f1aab200640 1 -- 172.21.15.6:0/2021133310 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1aa415e0b0 msgr2=0x7f1aa40b4640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:12.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:12.563+0000 7f1aab200640 1 --2- 172.21.15.6:0/2021133310 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1aa415e0b0 0x7f1aa40b4640 secure :-1 s=READY pgs=1694 cs=0 l=1 rev1=1 crypto rx=0x7f1aa0009dd0 tx=0x7f1aa000c490 comp rx=0 tx=0).stop 2024-03-04T09:35:12.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:12.563+0000 7f1aab200640 1 -- 172.21.15.6:0/2021133310 shutdown_connections 2024-03-04T09:35:12.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:12.563+0000 7f1aab200640 1 --2- 172.21.15.6:0/2021133310 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1aa415e0b0 0x7f1aa40b4640 unknown :-1 s=CLOSED pgs=1694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:12.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:12.563+0000 7f1aab200640 1 --2- 172.21.15.6:0/2021133310 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1aa40d4a10 0x7f1aa415e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:12.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:12.563+0000 7f1aab200640 1 --2- 172.21.15.6:0/2021133310 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f1a8806b4d0 0x7f1a8806d990 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:12.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:12.563+0000 7f1aab200640 1 --2- 172.21.15.6:0/2021133310 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1aa40d0c80 0x7f1aa414f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:12.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:12.563+0000 7f1aab200640 1 -- 172.21.15.6:0/2021133310 >> 172.21.15.6:0/2021133310 conn(0x7f1aa40d3d30 msgr2=0x7f1aa40ca600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:35:12.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:12.564+0000 7f1aab200640 1 -- 172.21.15.6:0/2021133310 shutdown_connections 2024-03-04T09:35:12.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:12.564+0000 7f1aab200640 1 -- 172.21.15.6:0/2021133310 wait complete. 2024-03-04T09:35:12.782 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:32:57.159681+0000', 'modified': '2024-03-04T09:33:54.652601+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8214, 'mds_1': 6796}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6796': {'gid': 6796, 'name': 'd', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.146:6837/2458070840', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 2458070840}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 2458070840}]}, 'join_fscid': -1, 'laggy_since': '2024-03-04T09:33:54.652601+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8214': {'gid': 8214, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2446627517', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2446627517}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2446627517}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8214, 'qdb_cluster': [8214]} 2024-03-04T09:35:12.783 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6796:up:stopping 2024-03-04T09:35:13.784 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:35:13.874 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-04T09:35:14.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:14.022+0000 7f0b18c00640 1 Processor -- start 2024-03-04T09:35:14.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:14.022+0000 7f0b18c00640 1 -- start start 2024-03-04T09:35:14.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:14.024+0000 7f0b18c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f0b140cf5c0 0x7f0b140cdc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:14.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:14.025+0000 7f0b18c00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f0b14071050 0x7f0b14071490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:14.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:14.025+0000 7f0b18c00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b140b4860 con 0x7f0b140b3c90 2024-03-04T09:35:14.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:14.025+0000 7f0b18c00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b140ce160 con 0x7f0b140cf5c0 2024-03-04T09:35:14.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:14.025+0000 7f0b18c00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b140b4b40 con 0x7f0b14071050 2024-03-04T09:35:14.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:14.025+0000 7f0b11400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f0b140cf5c0 0x7f0b140cdc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:14.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:14.025+0000 7f0b11400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f0b140cf5c0 0x7f0b140cdc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45286/0 (socket says 172.21.15.6:45286) 2024-03-04T09:35:14.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:14.025+0000 7f0b11e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f0b140b3c90 0x7f0b140b4070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:39192/0 (socket says 172.21.15.6:39192) 2024-03-04T09:35:14.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:14.025+0000 7f0b11400640 1 -- 172.21.15.6:0/2300385982 learned_addr learned my addr 172.21.15.6:0/2300385982 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:35:14.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:14.025+0000 7f0b12800640 1 --2- 172.21.15.6:0/2300385982 >> v2:172.21.15.146:3300/0 conn(0x7f0b14071050 0x7f0b14071490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:14.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:14.025+0000 7f0b11400640 1 -- 172.21.15.6:0/2300385982 >> v2:172.21.15.146:3300/0 conn(0x7f0b14071050 msgr2=0x7f0b14071490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:35:14.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:14.025+0000 7f0b11400640 1 --2- 172.21.15.6:0/2300385982 >> v2:172.21.15.146:3300/0 conn(0x7f0b14071050 0x7f0b14071490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:14.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:14.026+0000 7f0b11400640 1 -- 172.21.15.6:0/2300385982 >> v1:172.21.15.6:6789/0 conn(0x7f0b140b3c90 legacy=0x7f0b140b4070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:14.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:14.025+0000 7f0b10a00640 1 -- 172.21.15.6:0/2300385982 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1527369889 0 0) 0x7f0b140b4860 con 0x7f0b140b3c90 2024-03-04T09:35:14.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:14.026+0000 7f0b11400640 1 -- 172.21.15.6:0/2300385982 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b140b4e20 con 0x7f0b140cf5c0 2024-03-04T09:35:14.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:14.026+0000 7f0b12800640 1 --2- 172.21.15.6:0/2300385982 >> v2:172.21.15.146:3300/0 conn(0x7f0b14071050 0x7f0b14071490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:35:14.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:14.027+0000 7f0b11400640 1 --2- 172.21.15.6:0/2300385982 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0b140cf5c0 0x7f0b140cdc20 secure :-1 s=READY pgs=1696 cs=0 l=1 rev1=1 crypto rx=0x7f0b14074860 tx=0x7f0afc02e200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b2002da18b8bb44 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:35:14.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:14.027+0000 7f0b10a00640 1 -- 172.21.15.6:0/2300385982 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0afc03b070 con 0x7f0b140cf5c0 2024-03-04T09:35:14.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:14.027+0000 7f0b10a00640 1 -- 172.21.15.6:0/2300385982 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0afc036440 con 0x7f0b140cf5c0 2024-03-04T09:35:14.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:14.027+0000 7f0b10a00640 1 -- 172.21.15.6:0/2300385982 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0afc036810 con 0x7f0b140cf5c0 2024-03-04T09:35:14.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:14.028+0000 7f0b18c00640 1 -- 172.21.15.6:0/2300385982 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0b140cf5c0 msgr2=0x7f0b140cdc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:14.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:14.028+0000 7f0b18c00640 1 --2- 172.21.15.6:0/2300385982 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0b140cf5c0 0x7f0b140cdc20 secure :-1 s=READY pgs=1696 cs=0 l=1 rev1=1 crypto rx=0x7f0b14074860 tx=0x7f0afc02e200 comp rx=0 tx=0).stop 2024-03-04T09:35:14.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:14.028+0000 7f0b18c00640 1 -- 172.21.15.6:0/2300385982 shutdown_connections 2024-03-04T09:35:14.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:14.028+0000 7f0b18c00640 1 --2- 172.21.15.6:0/2300385982 >> v2:172.21.15.146:3300/0 conn(0x7f0b14071050 0x7f0b14071490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:14.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:14.028+0000 7f0b18c00640 1 --2- 172.21.15.6:0/2300385982 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0b140cf5c0 0x7f0b140cdc20 unknown :-1 s=CLOSED pgs=1696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:14.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:14.028+0000 7f0b18c00640 1 -- 172.21.15.6:0/2300385982 >> 172.21.15.6:0/2300385982 conn(0x7f0b1406afb0 msgr2=0x7f0b1406b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:35:14.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:14.028+0000 7f0b18c00640 1 -- 172.21.15.6:0/2300385982 shutdown_connections 2024-03-04T09:35:14.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:14.028+0000 7f0b18c00640 1 -- 172.21.15.6:0/2300385982 wait complete. 2024-03-04T09:35:14.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:14.030+0000 7f0b18c00640 1 Processor -- start 2024-03-04T09:35:14.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:14.030+0000 7f0b18c00640 1 -- start start 2024-03-04T09:35:14.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:14.031+0000 7f0b18c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0b14071050 0x7f0b140cca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:14.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:14.031+0000 7f0b18c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0b140b3c90 0x7f0b1414b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:14.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:14.031+0000 7f0b18c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0b140cf5c0 0x7f0b1414b960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:14.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:14.031+0000 7f0b18c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b140719d0 con 0x7f0b140b3c90 2024-03-04T09:35:14.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:14.031+0000 7f0b18c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b1415b340 con 0x7f0b140cf5c0 2024-03-04T09:35:14.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:14.031+0000 7f0b18c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b14071b50 con 0x7f0b14071050 2024-03-04T09:35:14.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:14.031+0000 7f0b11400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0b140b3c90 0x7f0b1414b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:14.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:14.031+0000 7f0b11400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0b140b3c90 0x7f0b1414b420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45300/0 (socket says 172.21.15.6:45300) 2024-03-04T09:35:14.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:14.031+0000 7f0b11400640 1 -- 172.21.15.6:0/3941130582 learned_addr learned my addr 172.21.15.6:0/3941130582 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:35:14.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:14.032+0000 7f0b11400640 1 -- 172.21.15.6:0/3941130582 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0b14071050 msgr2=0x7f0b140cca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:14.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:14.032+0000 7f0b11400640 1 --2- 172.21.15.6:0/3941130582 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0b14071050 0x7f0b140cca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:14.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:14.032+0000 7f0b11400640 1 -- 172.21.15.6:0/3941130582 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0b140cf5c0 msgr2=0x7f0b1414b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:14.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:14.032+0000 7f0b11400640 1 --2- 172.21.15.6:0/3941130582 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0b140cf5c0 0x7f0b1414b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:14.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:14.032+0000 7f0b11400640 1 -- 172.21.15.6:0/3941130582 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b141e0420 con 0x7f0b140b3c90 2024-03-04T09:35:14.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:14.032+0000 7f0b11400640 1 --2- 172.21.15.6:0/3941130582 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0b140b3c90 0x7f0b1414b420 secure :-1 s=READY pgs=1697 cs=0 l=1 rev1=1 crypto rx=0x7f0afc02e240 tx=0x7f0afc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:35:14.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:14.032+0000 7f0b0a000640 1 -- 172.21.15.6:0/3941130582 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0afc03b030 con 0x7f0b140b3c90 2024-03-04T09:35:14.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:14.032+0000 7f0b0a000640 1 -- 172.21.15.6:0/3941130582 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0afc03d070 con 0x7f0b140b3c90 2024-03-04T09:35:14.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:14.032+0000 7f0b18c00640 1 -- 172.21.15.6:0/3941130582 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b141e0700 con 0x7f0b140b3c90 2024-03-04T09:35:14.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:14.032+0000 7f0b0a000640 1 -- 172.21.15.6:0/3941130582 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0afc034bc0 con 0x7f0b140b3c90 2024-03-04T09:35:14.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:14.032+0000 7f0b18c00640 1 -- 172.21.15.6:0/3941130582 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b140ccf40 con 0x7f0b140b3c90 2024-03-04T09:35:14.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:14.034+0000 7f0b18c00640 1 -- 172.21.15.6:0/3941130582 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b14069510 con 0x7f0b140b3c90 2024-03-04T09:35:14.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:14.035+0000 7f0b0a000640 1 -- 172.21.15.6:0/3941130582 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0afc048060 con 0x7f0b140b3c90 2024-03-04T09:35:14.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:14.036+0000 7f0b0a000640 1 --2- 172.21.15.6:0/3941130582 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f0af006b500 0x7f0af006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:14.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:14.036+0000 7f0b0a000640 1 -- 172.21.15.6:0/3941130582 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f0afc0abf90 con 0x7f0b140b3c90 2024-03-04T09:35:14.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:14.037+0000 7f0b11e00640 1 --2- 172.21.15.6:0/3941130582 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f0af006b500 0x7f0af006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:14.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:14.038+0000 7f0b11e00640 1 --2- 172.21.15.6:0/3941130582 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f0af006b500 0x7f0af006d9c0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f0b04007fd0 tx=0x7f0b04007920 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:35:14.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:14.040+0000 7f0b0a000640 1 -- 172.21.15.6:0/3941130582 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0afc07adc0 con 0x7f0b140b3c90 2024-03-04T09:35:14.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:14.284+0000 7f0b18c00640 1 -- 172.21.15.6:0/3941130582 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0b1404f3a0 con 0x7f0b140b3c90 2024-03-04T09:35:14.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:14.285+0000 7f0b0a000640 1 -- 172.21.15.6:0/3941130582 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3465 (secure 0 0 0) 0x7f0afc07a6c0 con 0x7f0b140b3c90 2024-03-04T09:35:14.283 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:35:14.283 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:33:54.652601+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"laggy_since":"2024-03-04T09:33:54.652601+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":9}]} 2024-03-04T09:35:14.283 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 100 2024-03-04T09:35:14.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:14.288+0000 7f0b18c00640 1 -- 172.21.15.6:0/3941130582 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f0af006b500 msgr2=0x7f0af006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:14.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:14.288+0000 7f0b18c00640 1 --2- 172.21.15.6:0/3941130582 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f0af006b500 0x7f0af006d9c0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f0b04007fd0 tx=0x7f0b04007920 comp rx=0 tx=0).stop 2024-03-04T09:35:14.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:14.288+0000 7f0b18c00640 1 -- 172.21.15.6:0/3941130582 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0b140b3c90 msgr2=0x7f0b1414b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:14.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:14.288+0000 7f0b18c00640 1 --2- 172.21.15.6:0/3941130582 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0b140b3c90 0x7f0b1414b420 secure :-1 s=READY pgs=1697 cs=0 l=1 rev1=1 crypto rx=0x7f0afc02e240 tx=0x7f0afc0046c0 comp rx=0 tx=0).stop 2024-03-04T09:35:14.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:14.288+0000 7f0b18c00640 1 -- 172.21.15.6:0/3941130582 shutdown_connections 2024-03-04T09:35:14.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:14.288+0000 7f0b18c00640 1 --2- 172.21.15.6:0/3941130582 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0b140cf5c0 0x7f0b1414b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:14.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:14.288+0000 7f0b18c00640 1 --2- 172.21.15.6:0/3941130582 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0b140b3c90 0x7f0b1414b420 unknown :-1 s=CLOSED pgs=1697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:14.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:14.288+0000 7f0b18c00640 1 --2- 172.21.15.6:0/3941130582 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f0af006b500 0x7f0af006d9c0 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:14.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:14.288+0000 7f0b18c00640 1 --2- 172.21.15.6:0/3941130582 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0b14071050 0x7f0b140cca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:14.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:14.288+0000 7f0b18c00640 1 -- 172.21.15.6:0/3941130582 >> 172.21.15.6:0/3941130582 conn(0x7f0b1406afb0 msgr2=0x7f0b1406d8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:35:14.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:14.288+0000 7f0b18c00640 1 -- 172.21.15.6:0/3941130582 shutdown_connections 2024-03-04T09:35:14.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:14.288+0000 7f0b18c00640 1 -- 172.21.15.6:0/3941130582 wait complete. 2024-03-04T09:35:14.506 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:32:57.159681+0000', 'modified': '2024-03-04T09:33:54.652601+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8214, 'mds_1': 6796}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6796': {'gid': 6796, 'name': 'd', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.146:6837/2458070840', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 2458070840}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 2458070840}]}, 'join_fscid': -1, 'laggy_since': '2024-03-04T09:33:54.652601+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8214': {'gid': 8214, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2446627517', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2446627517}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2446627517}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8214, 'qdb_cluster': [8214]} 2024-03-04T09:35:14.506 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6796:up:stopping 2024-03-04T09:35:15.507 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:35:15.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:15.750+0000 7fa7d3200640 1 Processor -- start 2024-03-04T09:35:15.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:15.750+0000 7fa7d3200640 1 -- start start 2024-03-04T09:35:15.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:15.752+0000 7fa7d3200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fa7d415e0b0 0x7fa7d40cf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:15.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:15.752+0000 7fa7d3200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa7d40d4a10 0x7fa7d40d0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:15.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:15.752+0000 7fa7d3200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7d40b3710 con 0x7fa7d40d0c80 2024-03-04T09:35:15.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:15.752+0000 7fa7d3200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7d40b09e0 con 0x7fa7d415e0b0 2024-03-04T09:35:15.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:15.752+0000 7fa7d3200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7d40b39f0 con 0x7fa7d40d4a10 2024-03-04T09:35:15.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:15.753+0000 7fa7d1e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fa7d415e0b0 0x7fa7d40cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:15.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:15.753+0000 7fa7c9400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa7d40d4a10 0x7fa7d40d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:15.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:15.753+0000 7fa7d1e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fa7d415e0b0 0x7fa7d40cf310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.6:53162/0 (socket says 172.21.15.6:53162) 2024-03-04T09:35:15.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:15.753+0000 7fa7d1e00640 1 -- 172.21.15.6:0/2270091711 learned_addr learned my addr 172.21.15.6:0/2270091711 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:35:15.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:15.753+0000 7fa7c9400640 1 -- 172.21.15.6:0/2270091711 >> v2:172.21.15.146:3300/0 conn(0x7fa7d415e0b0 msgr2=0x7fa7d40cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:35:15.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:15.753+0000 7fa7c9400640 1 --2- 172.21.15.6:0/2270091711 >> v2:172.21.15.146:3300/0 conn(0x7fa7d415e0b0 0x7fa7d40cf310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:15.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:15.753+0000 7fa7c9400640 1 -- 172.21.15.6:0/2270091711 >> v1:172.21.15.146:6789/0 conn(0x7fa7d40d0c80 legacy=0x7fa7d40d1900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:15.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:15.754+0000 7fa7c9400640 1 -- 172.21.15.6:0/2270091711 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7d40b3cd0 con 0x7fa7d40d4a10 2024-03-04T09:35:15.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:15.755+0000 7fa7c9400640 1 --2- 172.21.15.6:0/2270091711 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa7d40d4a10 0x7fa7d40d0740 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fa7bc0097a0 tx=0x7fa7bc02b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2899609738c3dd97 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:35:15.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:15.755+0000 7fa7d1400640 1 -- 172.21.15.6:0/2270091711 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7bc035020 con 0x7fa7d40d4a10 2024-03-04T09:35:15.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:15.756+0000 7fa7d1400640 1 -- 172.21.15.6:0/2270091711 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7bc031810 con 0x7fa7d40d4a10 2024-03-04T09:35:15.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:15.756+0000 7fa7d1400640 1 -- 172.21.15.6:0/2270091711 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7bc031be0 con 0x7fa7d40d4a10 2024-03-04T09:35:15.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:15.756+0000 7fa7d3200640 1 -- 172.21.15.6:0/2270091711 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa7d40d4a10 msgr2=0x7fa7d40d0740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:15.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:15.756+0000 7fa7d3200640 1 --2- 172.21.15.6:0/2270091711 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa7d40d4a10 0x7fa7d40d0740 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fa7bc0097a0 tx=0x7fa7bc02b8b0 comp rx=0 tx=0).stop 2024-03-04T09:35:15.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:15.756+0000 7fa7d3200640 1 -- 172.21.15.6:0/2270091711 shutdown_connections 2024-03-04T09:35:15.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:15.756+0000 7fa7d3200640 1 --2- 172.21.15.6:0/2270091711 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa7d40d4a10 0x7fa7d40d0740 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:15.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:15.756+0000 7fa7d3200640 1 --2- 172.21.15.6:0/2270091711 >> v2:172.21.15.146:3300/0 conn(0x7fa7d415e0b0 0x7fa7d40cf310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:15.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:15.756+0000 7fa7d3200640 1 -- 172.21.15.6:0/2270091711 >> 172.21.15.6:0/2270091711 conn(0x7fa7d40d3d30 msgr2=0x7fa7d40d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:35:15.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:15.756+0000 7fa7d3200640 1 -- 172.21.15.6:0/2270091711 shutdown_connections 2024-03-04T09:35:15.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:15.756+0000 7fa7d3200640 1 -- 172.21.15.6:0/2270091711 wait complete. 2024-03-04T09:35:15.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:15.758+0000 7fa7d3200640 1 Processor -- start 2024-03-04T09:35:15.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:15.758+0000 7fa7d3200640 1 -- start start 2024-03-04T09:35:15.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:15.759+0000 7fa7d3200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa7d40d0c80 0x7fa7d40b0ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:15.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:15.760+0000 7fa7d3200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa7d40d4a10 0x7fa7d40cb160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:15.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:15.760+0000 7fa7d3200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa7d415e0b0 0x7fa7d4156e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:15.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:15.760+0000 7fa7d3200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7d40cfc60 con 0x7fa7d40d0c80 2024-03-04T09:35:15.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:15.760+0000 7fa7d3200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7d415e490 con 0x7fa7d40d4a10 2024-03-04T09:35:15.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:15.760+0000 7fa7d3200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7d40b54e0 con 0x7fa7d415e0b0 2024-03-04T09:35:15.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:15.760+0000 7fa7d1e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa7d40d0c80 0x7fa7d40b0ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:15.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:15.760+0000 7fa7d1e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa7d40d0c80 0x7fa7d40b0ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45314/0 (socket says 172.21.15.6:45314) 2024-03-04T09:35:15.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:15.760+0000 7fa7d1e00640 1 -- 172.21.15.6:0/2986657367 learned_addr learned my addr 172.21.15.6:0/2986657367 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:35:15.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:15.760+0000 7fa7d1e00640 1 -- 172.21.15.6:0/2986657367 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa7d415e0b0 msgr2=0x7fa7d4156e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:15.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:15.760+0000 7fa7d1e00640 1 --2- 172.21.15.6:0/2986657367 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa7d415e0b0 0x7fa7d4156e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:15.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:15.760+0000 7fa7d1e00640 1 -- 172.21.15.6:0/2986657367 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa7d40d4a10 msgr2=0x7fa7d40cb160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:15.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:15.760+0000 7fa7d1e00640 1 --2- 172.21.15.6:0/2986657367 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa7d40d4a10 0x7fa7d40cb160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:15.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:15.760+0000 7fa7d1e00640 1 -- 172.21.15.6:0/2986657367 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7d40b1e70 con 0x7fa7d40d0c80 2024-03-04T09:35:15.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:15.761+0000 7fa7d1e00640 1 --2- 172.21.15.6:0/2986657367 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa7d40d0c80 0x7fa7d40b0ee0 secure :-1 s=READY pgs=1698 cs=0 l=1 rev1=1 crypto rx=0x7fa7c4009cf0 tx=0x7fa7c400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:35:15.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:15.761+0000 7fa7caa00640 1 -- 172.21.15.6:0/2986657367 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7c400cda0 con 0x7fa7d40d0c80 2024-03-04T09:35:15.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:15.761+0000 7fa7caa00640 1 -- 172.21.15.6:0/2986657367 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7c4019070 con 0x7fa7d40d0c80 2024-03-04T09:35:15.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:15.761+0000 7fa7caa00640 1 -- 172.21.15.6:0/2986657367 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7c40146a0 con 0x7fa7d40d0c80 2024-03-04T09:35:15.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:15.761+0000 7fa7d3200640 1 -- 172.21.15.6:0/2986657367 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7d40cd3a0 con 0x7fa7d40d0c80 2024-03-04T09:35:15.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:15.761+0000 7fa7d3200640 1 -- 172.21.15.6:0/2986657367 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7d40cd960 con 0x7fa7d40d0c80 2024-03-04T09:35:15.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:15.762+0000 7fa7d3200640 1 -- 172.21.15.6:0/2986657367 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7d404f330 con 0x7fa7d40d0c80 2024-03-04T09:35:15.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:15.763+0000 7fa7caa00640 1 -- 172.21.15.6:0/2986657367 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa7c4005d10 con 0x7fa7d40d0c80 2024-03-04T09:35:15.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:15.764+0000 7fa7caa00640 1 --2- 172.21.15.6:0/2986657367 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa7b006b4d0 0x7fa7b006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:15.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:15.764+0000 7fa7caa00640 1 -- 172.21.15.6:0/2986657367 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fa7c408aa20 con 0x7fa7d40d0c80 2024-03-04T09:35:15.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:15.765+0000 7fa7c9400640 1 --2- 172.21.15.6:0/2986657367 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa7b006b4d0 0x7fa7b006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:15.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:15.766+0000 7fa7c9400640 1 --2- 172.21.15.6:0/2986657367 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa7b006b4d0 0x7fa7b006d990 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fa7bc036000 tx=0x7fa7bc030000 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:35:15.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:15.769+0000 7fa7caa00640 1 -- 172.21.15.6:0/2986657367 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa7c4023020 con 0x7fa7d40d0c80 2024-03-04T09:35:15.877 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-04T09:35:16.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:16.032+0000 7fa7d3200640 1 -- 172.21.15.6:0/2986657367 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa7d40b9b90 con 0x7fa7d40d0c80 2024-03-04T09:35:16.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:16.034+0000 7fa7caa00640 1 -- 172.21.15.6:0/2986657367 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3465 (secure 0 0 0) 0x7fa7c4010040 con 0x7fa7d40d0c80 2024-03-04T09:35:16.031 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:35:16.031 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:33:54.652601+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"laggy_since":"2024-03-04T09:33:54.652601+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":9}]} 2024-03-04T09:35:16.031 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 100 2024-03-04T09:35:16.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:16.036+0000 7fa7d3200640 1 -- 172.21.15.6:0/2986657367 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa7b006b4d0 msgr2=0x7fa7b006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:16.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:16.036+0000 7fa7d3200640 1 --2- 172.21.15.6:0/2986657367 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa7b006b4d0 0x7fa7b006d990 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fa7bc036000 tx=0x7fa7bc030000 comp rx=0 tx=0).stop 2024-03-04T09:35:16.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:16.036+0000 7fa7d3200640 1 -- 172.21.15.6:0/2986657367 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa7d40d0c80 msgr2=0x7fa7d40b0ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:16.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:16.036+0000 7fa7d3200640 1 --2- 172.21.15.6:0/2986657367 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa7d40d0c80 0x7fa7d40b0ee0 secure :-1 s=READY pgs=1698 cs=0 l=1 rev1=1 crypto rx=0x7fa7c4009cf0 tx=0x7fa7c400c490 comp rx=0 tx=0).stop 2024-03-04T09:35:16.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:16.036+0000 7fa7d3200640 1 -- 172.21.15.6:0/2986657367 shutdown_connections 2024-03-04T09:35:16.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:16.036+0000 7fa7d3200640 1 --2- 172.21.15.6:0/2986657367 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa7b006b4d0 0x7fa7b006d990 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:16.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:16.036+0000 7fa7d3200640 1 --2- 172.21.15.6:0/2986657367 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa7d415e0b0 0x7fa7d4156e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:16.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:16.036+0000 7fa7d3200640 1 --2- 172.21.15.6:0/2986657367 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa7d40d4a10 0x7fa7d40cb160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:16.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:16.036+0000 7fa7d3200640 1 --2- 172.21.15.6:0/2986657367 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa7d40d0c80 0x7fa7d40b0ee0 unknown :-1 s=CLOSED pgs=1698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:16.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:16.036+0000 7fa7d3200640 1 -- 172.21.15.6:0/2986657367 >> 172.21.15.6:0/2986657367 conn(0x7fa7d40d3d30 msgr2=0x7fa7d40d18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:35:16.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:16.036+0000 7fa7d3200640 1 -- 172.21.15.6:0/2986657367 shutdown_connections 2024-03-04T09:35:16.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:16.036+0000 7fa7d3200640 1 -- 172.21.15.6:0/2986657367 wait complete. 2024-03-04T09:35:16.254 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:32:57.159681+0000', 'modified': '2024-03-04T09:33:54.652601+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8214, 'mds_1': 6796}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6796': {'gid': 6796, 'name': 'd', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.146:6837/2458070840', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 2458070840}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 2458070840}]}, 'join_fscid': -1, 'laggy_since': '2024-03-04T09:33:54.652601+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8214': {'gid': 8214, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2446627517', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2446627517}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2446627517}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8214, 'qdb_cluster': [8214]} 2024-03-04T09:35:16.254 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6796:up:stopping 2024-03-04T09:35:17.255 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:35:17.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:17.484+0000 7faa7be00640 1 Processor -- start 2024-03-04T09:35:17.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:17.485+0000 7faa7be00640 1 -- start start 2024-03-04T09:35:17.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:17.486+0000 7faa7be00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7faa7c1518e0 0x7faa7c06efc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:17.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:17.486+0000 7faa7be00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa7c15b350 con 0x7faa7c06e7e0 2024-03-04T09:35:17.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:17.486+0000 7faa7be00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa7c15b630 con 0x7faa7c151cc0 2024-03-04T09:35:17.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:17.486+0000 7faa7be00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa7c15b910 con 0x7faa7c1518e0 2024-03-04T09:35:17.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:17.487+0000 7faa7aa00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7faa7c1518e0 0x7faa7c06efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:17.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:17.487+0000 7faa7b400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7faa7c06e7e0 0x7faa7c0ce510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:39198/0 (socket says 172.21.15.6:39198) 2024-03-04T09:35:17.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:17.487+0000 7faa7b400640 1 -- 172.21.15.6:0/2848064313 learned_addr learned my addr 172.21.15.6:0/2848064313 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:35:17.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:17.487+0000 7faa7aa00640 1 -- 172.21.15.6:0/2848064313 >> v1:172.21.15.146:6789/0 conn(0x7faa7c151cc0 legacy=0x7faa7c06e0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:35:17.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:17.487+0000 7faa7aa00640 1 -- 172.21.15.6:0/2848064313 >> v1:172.21.15.6:6789/0 conn(0x7faa7c06e7e0 legacy=0x7faa7c0ce510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:17.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:17.487+0000 7faa7aa00640 1 -- 172.21.15.6:0/2848064313 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa7c15bbf0 con 0x7faa7c1518e0 2024-03-04T09:35:17.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:17.487+0000 7faa79600640 1 -- 172.21.15.6:0/2848064313 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2774955410 0 0) 0x7faa7c15b350 con 0x7faa7c06e7e0 2024-03-04T09:35:17.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:17.489+0000 7faa7aa00640 1 --2- 172.21.15.6:0/2848064313 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faa7c1518e0 0x7faa7c06efc0 secure :-1 s=READY pgs=1700 cs=0 l=1 rev1=1 crypto rx=0x7faa640097a0 tx=0x7faa6402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f22cb8d0e390377 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:35:17.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:17.489+0000 7faa79600640 1 -- 172.21.15.6:0/2848064313 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa64035020 con 0x7faa7c1518e0 2024-03-04T09:35:17.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:17.489+0000 7faa79600640 1 -- 172.21.15.6:0/2848064313 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa64031830 con 0x7faa7c1518e0 2024-03-04T09:35:17.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:17.489+0000 7faa79600640 1 -- 172.21.15.6:0/2848064313 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa64031c00 con 0x7faa7c1518e0 2024-03-04T09:35:17.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:17.489+0000 7faa7be00640 1 -- 172.21.15.6:0/2848064313 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faa7c1518e0 msgr2=0x7faa7c06efc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:17.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:17.489+0000 7faa7be00640 1 --2- 172.21.15.6:0/2848064313 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faa7c1518e0 0x7faa7c06efc0 secure :-1 s=READY pgs=1700 cs=0 l=1 rev1=1 crypto rx=0x7faa640097a0 tx=0x7faa6402b8d0 comp rx=0 tx=0).stop 2024-03-04T09:35:17.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:17.489+0000 7faa7be00640 1 -- 172.21.15.6:0/2848064313 shutdown_connections 2024-03-04T09:35:17.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:17.489+0000 7faa7be00640 1 --2- 172.21.15.6:0/2848064313 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faa7c1518e0 0x7faa7c06efc0 unknown :-1 s=CLOSED pgs=1700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:17.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:17.490+0000 7faa7be00640 1 -- 172.21.15.6:0/2848064313 >> 172.21.15.6:0/2848064313 conn(0x7faa7c0afb00 msgr2=0x7faa7c072df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:35:17.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:17.490+0000 7faa7be00640 1 -- 172.21.15.6:0/2848064313 shutdown_connections 2024-03-04T09:35:17.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:17.490+0000 7faa7be00640 1 -- 172.21.15.6:0/2848064313 wait complete. 2024-03-04T09:35:17.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:17.491+0000 7faa7be00640 1 Processor -- start 2024-03-04T09:35:17.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:17.492+0000 7faa7be00640 1 -- start start 2024-03-04T09:35:17.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:17.492+0000 7faa7be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faa7c06e7e0 0x7faa7c1563c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:17.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:17.492+0000 7faa7be00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faa7c1518e0 0x7faa7c1597d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:17.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:17.493+0000 7faa7be00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faa7c151cc0 0x7faa7c140960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:17.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:17.493+0000 7faa7be00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa7c06f500 con 0x7faa7c06e7e0 2024-03-04T09:35:17.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:17.493+0000 7faa7be00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa7c15b040 con 0x7faa7c1518e0 2024-03-04T09:35:17.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:17.493+0000 7faa7be00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa7c06f680 con 0x7faa7c151cc0 2024-03-04T09:35:17.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:17.493+0000 7faa7aa00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faa7c06e7e0 0x7faa7c1563c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:17.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:17.493+0000 7faa7aa00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faa7c06e7e0 0x7faa7c1563c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45340/0 (socket says 172.21.15.6:45340) 2024-03-04T09:35:17.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:17.493+0000 7faa7aa00640 1 -- 172.21.15.6:0/1170126089 learned_addr learned my addr 172.21.15.6:0/1170126089 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:35:17.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:17.493+0000 7faa7a000640 1 --2- 172.21.15.6:0/1170126089 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faa7c1518e0 0x7faa7c1597d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:17.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:17.493+0000 7faa7b400640 1 --2- 172.21.15.6:0/1170126089 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faa7c151cc0 0x7faa7c140960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:17.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:17.493+0000 7faa7aa00640 1 -- 172.21.15.6:0/1170126089 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faa7c151cc0 msgr2=0x7faa7c140960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:17.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:17.493+0000 7faa7aa00640 1 --2- 172.21.15.6:0/1170126089 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faa7c151cc0 0x7faa7c140960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:17.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:17.493+0000 7faa7aa00640 1 -- 172.21.15.6:0/1170126089 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faa7c1518e0 msgr2=0x7faa7c1597d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:17.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:17.493+0000 7faa7aa00640 1 --2- 172.21.15.6:0/1170126089 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faa7c1518e0 0x7faa7c1597d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:17.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:17.493+0000 7faa7aa00640 1 -- 172.21.15.6:0/1170126089 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa7c159fb0 con 0x7faa7c06e7e0 2024-03-04T09:35:17.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:17.494+0000 7faa7aa00640 1 --2- 172.21.15.6:0/1170126089 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faa7c06e7e0 0x7faa7c1563c0 secure :-1 s=READY pgs=1701 cs=0 l=1 rev1=1 crypto rx=0x7faa640097a0 tx=0x7faa640046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:35:17.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:17.494+0000 7faa72a00640 1 -- 172.21.15.6:0/1170126089 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa64004230 con 0x7faa7c06e7e0 2024-03-04T09:35:17.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:17.494+0000 7faa72a00640 1 -- 172.21.15.6:0/1170126089 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa640043d0 con 0x7faa7c06e7e0 2024-03-04T09:35:17.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:17.494+0000 7faa72a00640 1 -- 172.21.15.6:0/1170126089 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa6403cad0 con 0x7faa7c06e7e0 2024-03-04T09:35:17.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:17.494+0000 7faa7be00640 1 -- 172.21.15.6:0/1170126089 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa7c159d10 con 0x7faa7c06e7e0 2024-03-04T09:35:17.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:17.494+0000 7faa7be00640 1 -- 172.21.15.6:0/1170126089 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa7c1416a0 con 0x7faa7c06e7e0 2024-03-04T09:35:17.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:17.496+0000 7faa7be00640 1 -- 172.21.15.6:0/1170126089 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa7c04f330 con 0x7faa7c06e7e0 2024-03-04T09:35:17.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:17.496+0000 7faa72a00640 1 -- 172.21.15.6:0/1170126089 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7faa64038020 con 0x7faa7c06e7e0 2024-03-04T09:35:17.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:17.497+0000 7faa72a00640 1 --2- 172.21.15.6:0/1170126089 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7faa5806b4d0 0x7faa5806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:17.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:17.497+0000 7faa72a00640 1 -- 172.21.15.6:0/1170126089 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7faa640ac460 con 0x7faa7c06e7e0 2024-03-04T09:35:17.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:17.498+0000 7faa7a000640 1 --2- 172.21.15.6:0/1170126089 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7faa5806b4d0 0x7faa5806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:17.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:17.499+0000 7faa7a000640 1 --2- 172.21.15.6:0/1170126089 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7faa5806b4d0 0x7faa5806d990 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7faa6c006fd0 tx=0x7faa6c008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:35:17.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:17.502+0000 7faa72a00640 1 -- 172.21.15.6:0/1170126089 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7faa64041070 con 0x7faa7c06e7e0 2024-03-04T09:35:17.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:17.748+0000 7faa7be00640 1 -- 172.21.15.6:0/1170126089 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faa7c0b5e70 con 0x7faa7c06e7e0 2024-03-04T09:35:17.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:17.748+0000 7faa72a00640 1 -- 172.21.15.6:0/1170126089 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3465 (secure 0 0 0) 0x7faa6407b330 con 0x7faa7c06e7e0 2024-03-04T09:35:17.746 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:35:17.746 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:33:54.652601+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"laggy_since":"2024-03-04T09:33:54.652601+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":9}]} 2024-03-04T09:35:17.746 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 100 2024-03-04T09:35:17.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:17.751+0000 7faa7be00640 1 -- 172.21.15.6:0/1170126089 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7faa5806b4d0 msgr2=0x7faa5806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:17.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:17.751+0000 7faa7be00640 1 --2- 172.21.15.6:0/1170126089 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7faa5806b4d0 0x7faa5806d990 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7faa6c006fd0 tx=0x7faa6c008040 comp rx=0 tx=0).stop 2024-03-04T09:35:17.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:17.751+0000 7faa7be00640 1 -- 172.21.15.6:0/1170126089 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faa7c06e7e0 msgr2=0x7faa7c1563c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:17.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:17.751+0000 7faa7be00640 1 --2- 172.21.15.6:0/1170126089 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faa7c06e7e0 0x7faa7c1563c0 secure :-1 s=READY pgs=1701 cs=0 l=1 rev1=1 crypto rx=0x7faa640097a0 tx=0x7faa640046c0 comp rx=0 tx=0).stop 2024-03-04T09:35:17.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:17.751+0000 7faa7be00640 1 -- 172.21.15.6:0/1170126089 shutdown_connections 2024-03-04T09:35:17.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:17.751+0000 7faa7be00640 1 --2- 172.21.15.6:0/1170126089 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7faa5806b4d0 0x7faa5806d990 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:17.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:17.751+0000 7faa7be00640 1 --2- 172.21.15.6:0/1170126089 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faa7c151cc0 0x7faa7c140960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:17.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:17.751+0000 7faa7be00640 1 --2- 172.21.15.6:0/1170126089 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faa7c1518e0 0x7faa7c1597d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:17.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:17.751+0000 7faa7be00640 1 --2- 172.21.15.6:0/1170126089 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faa7c06e7e0 0x7faa7c1563c0 unknown :-1 s=CLOSED pgs=1701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:17.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:17.751+0000 7faa7be00640 1 -- 172.21.15.6:0/1170126089 >> 172.21.15.6:0/1170126089 conn(0x7faa7c0afb00 msgr2=0x7faa7c15c240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:35:17.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:17.752+0000 7faa7be00640 1 -- 172.21.15.6:0/1170126089 shutdown_connections 2024-03-04T09:35:17.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:17.752+0000 7faa7be00640 1 -- 172.21.15.6:0/1170126089 wait complete. 2024-03-04T09:35:17.879 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-04T09:35:17.971 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:32:57.159681+0000', 'modified': '2024-03-04T09:33:54.652601+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8214, 'mds_1': 6796}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6796': {'gid': 6796, 'name': 'd', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.146:6837/2458070840', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 2458070840}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 2458070840}]}, 'join_fscid': -1, 'laggy_since': '2024-03-04T09:33:54.652601+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8214': {'gid': 8214, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2446627517', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2446627517}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2446627517}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8214, 'qdb_cluster': [8214]} 2024-03-04T09:35:17.971 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6796:up:stopping 2024-03-04T09:35:18.972 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:35:19.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:19.203+0000 7f8f98a00640 1 Processor -- start 2024-03-04T09:35:19.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:19.204+0000 7f8f98a00640 1 -- start start 2024-03-04T09:35:19.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:19.205+0000 7f8f98a00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f8f940cd060 0x7f8f94161ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:19.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:19.205+0000 7f8f98a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f8f94162000 0x7f8f94070c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:19.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:19.205+0000 7f8f98a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f941627f0 con 0x7f8f9415efc0 2024-03-04T09:35:19.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:19.205+0000 7f8f98a00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f9415f3a0 con 0x7f8f94162000 2024-03-04T09:35:19.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:19.205+0000 7f8f98a00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f9406d190 con 0x7f8f940cd060 2024-03-04T09:35:19.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:19.206+0000 7f8f93400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f8f9415efc0 0x7f8f940cc950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:47196/0 (socket says 172.21.15.6:47196) 2024-03-04T09:35:19.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:19.206+0000 7f8f93400640 1 -- 172.21.15.6:0/67020373 learned_addr learned my addr 172.21.15.6:0/67020373 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:35:19.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:19.206+0000 7f8f93e00640 1 --2- 172.21.15.6:0/67020373 >> v2:172.21.15.6:3300/0 conn(0x7f8f94162000 0x7f8f94070c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:19.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:19.206+0000 7f8f8be00640 1 --2- 172.21.15.6:0/67020373 >> v2:172.21.15.146:3300/0 conn(0x7f8f940cd060 0x7f8f94161ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:19.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:19.206+0000 7f8f92a00640 1 -- 172.21.15.6:0/67020373 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 747330959 0 0) 0x7f8f941627f0 con 0x7f8f9415efc0 2024-03-04T09:35:19.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:19.206+0000 7f8f92a00640 1 -- 172.21.15.6:0/67020373 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f78003520 con 0x7f8f9415efc0 2024-03-04T09:35:19.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:19.207+0000 7f8f93e00640 1 -- 172.21.15.6:0/67020373 >> v2:172.21.15.146:3300/0 conn(0x7f8f940cd060 msgr2=0x7f8f94161ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:35:19.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:19.207+0000 7f8f93e00640 1 --2- 172.21.15.6:0/67020373 >> v2:172.21.15.146:3300/0 conn(0x7f8f940cd060 0x7f8f94161ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:19.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:19.207+0000 7f8f93e00640 1 -- 172.21.15.6:0/67020373 >> v1:172.21.15.6:6789/0 conn(0x7f8f9415efc0 legacy=0x7f8f940cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:19.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:19.207+0000 7f8f93e00640 1 -- 172.21.15.6:0/67020373 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f9406c860 con 0x7f8f94162000 2024-03-04T09:35:19.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:19.208+0000 7f8f93e00640 1 --2- 172.21.15.6:0/67020373 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8f94162000 0x7f8f94070c20 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 crypto rx=0x7f8f940b51d0 tx=0x7f8f8c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d763408e8cbd021d server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:35:19.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:19.208+0000 7f8f92a00640 1 -- 172.21.15.6:0/67020373 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f8c039070 con 0x7f8f94162000 2024-03-04T09:35:19.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:19.208+0000 7f8f92a00640 1 -- 172.21.15.6:0/67020373 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f8c02cc80 con 0x7f8f94162000 2024-03-04T09:35:19.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:19.208+0000 7f8f92a00640 1 -- 172.21.15.6:0/67020373 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f8c034640 con 0x7f8f94162000 2024-03-04T09:35:19.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:19.209+0000 7f8f98a00640 1 -- 172.21.15.6:0/67020373 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8f94162000 msgr2=0x7f8f94070c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:19.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:19.209+0000 7f8f98a00640 1 --2- 172.21.15.6:0/67020373 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8f94162000 0x7f8f94070c20 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 crypto rx=0x7f8f940b51d0 tx=0x7f8f8c02bed0 comp rx=0 tx=0).stop 2024-03-04T09:35:19.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:19.209+0000 7f8f98a00640 1 -- 172.21.15.6:0/67020373 shutdown_connections 2024-03-04T09:35:19.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:19.209+0000 7f8f98a00640 1 --2- 172.21.15.6:0/67020373 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8f94162000 0x7f8f94070c20 unknown :-1 s=CLOSED pgs=1703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:19.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:19.209+0000 7f8f98a00640 1 --2- 172.21.15.6:0/67020373 >> v2:172.21.15.146:3300/0 conn(0x7f8f940cd060 0x7f8f94161ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:19.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:19.209+0000 7f8f98a00640 1 -- 172.21.15.6:0/67020373 >> 172.21.15.6:0/67020373 conn(0x7f8f9406b3d0 msgr2=0x7f8f9406b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:35:19.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:19.209+0000 7f8f98a00640 1 -- 172.21.15.6:0/67020373 shutdown_connections 2024-03-04T09:35:19.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:19.209+0000 7f8f98a00640 1 -- 172.21.15.6:0/67020373 wait complete. 2024-03-04T09:35:19.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:19.211+0000 7f8f98a00640 1 Processor -- start 2024-03-04T09:35:19.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:19.211+0000 7f8f98a00640 1 -- start start 2024-03-04T09:35:19.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:19.212+0000 7f8f98a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8f940cd060 0x7f8f940cee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:19.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:19.212+0000 7f8f98a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8f9415efc0 0x7f8f941570d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:19.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:19.212+0000 7f8f98a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8f94162000 0x7f8f94157610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:19.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:19.212+0000 7f8f98a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f94162470 con 0x7f8f9415efc0 2024-03-04T09:35:19.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:19.212+0000 7f8f98a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f9406ce20 con 0x7f8f940cd060 2024-03-04T09:35:19.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:19.212+0000 7f8f98a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f9406d620 con 0x7f8f94162000 2024-03-04T09:35:19.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:19.212+0000 7f8f8be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8f9415efc0 0x7f8f941570d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:19.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:19.212+0000 7f8f8be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8f9415efc0 0x7f8f941570d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:35300/0 (socket says 172.21.15.6:35300) 2024-03-04T09:35:19.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:19.212+0000 7f8f8be00640 1 -- 172.21.15.6:0/1527778102 learned_addr learned my addr 172.21.15.6:0/1527778102 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:35:19.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:19.212+0000 7f8f8be00640 1 -- 172.21.15.6:0/1527778102 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8f94162000 msgr2=0x7f8f94157610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:19.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:19.212+0000 7f8f8be00640 1 --2- 172.21.15.6:0/1527778102 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8f94162000 0x7f8f94157610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:19.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:19.212+0000 7f8f8be00640 1 -- 172.21.15.6:0/1527778102 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8f940cd060 msgr2=0x7f8f940cee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:19.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:19.212+0000 7f8f8be00640 1 --2- 172.21.15.6:0/1527778102 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8f940cd060 0x7f8f940cee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:19.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:19.212+0000 7f8f8be00640 1 -- 172.21.15.6:0/1527778102 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f9415cb40 con 0x7f8f9415efc0 2024-03-04T09:35:19.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:19.213+0000 7f8f8be00640 1 --2- 172.21.15.6:0/1527778102 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8f9415efc0 0x7f8f941570d0 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 crypto rx=0x7f8f7c00e9b0 tx=0x7f8f7c00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:35:19.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:19.213+0000 7f8f8b400640 1 -- 172.21.15.6:0/1527778102 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f7c017070 con 0x7f8f9415efc0 2024-03-04T09:35:19.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:19.213+0000 7f8f8b400640 1 -- 172.21.15.6:0/1527778102 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f7c00c9a0 con 0x7f8f9415efc0 2024-03-04T09:35:19.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:19.213+0000 7f8f8b400640 1 -- 172.21.15.6:0/1527778102 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f7c00cd90 con 0x7f8f9415efc0 2024-03-04T09:35:19.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:19.213+0000 7f8f98a00640 1 -- 172.21.15.6:0/1527778102 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f9415ce20 con 0x7f8f9415efc0 2024-03-04T09:35:19.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:19.213+0000 7f8f98a00640 1 -- 172.21.15.6:0/1527778102 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f9415d3e0 con 0x7f8f9415efc0 2024-03-04T09:35:19.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:19.215+0000 7f8f98a00640 1 -- 172.21.15.6:0/1527778102 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f9404f330 con 0x7f8f9415efc0 2024-03-04T09:35:19.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:19.216+0000 7f8f8b400640 1 -- 172.21.15.6:0/1527778102 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8f7c01a070 con 0x7f8f9415efc0 2024-03-04T09:35:19.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:19.217+0000 7f8f8b400640 1 --2- 172.21.15.6:0/1527778102 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f8f7006b4d0 0x7f8f7006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:19.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:19.217+0000 7f8f8b400640 1 -- 172.21.15.6:0/1527778102 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f8f7c08a670 con 0x7f8f9415efc0 2024-03-04T09:35:19.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:19.218+0000 7f8f93400640 1 --2- 172.21.15.6:0/1527778102 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f8f7006b4d0 0x7f8f7006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:19.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:19.219+0000 7f8f93400640 1 --2- 172.21.15.6:0/1527778102 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f8f7006b4d0 0x7f8f7006d990 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f8f84000f60 tx=0x7f8f840078f0 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:35:19.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:19.221+0000 7f8f8b400640 1 -- 172.21.15.6:0/1527778102 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8f7c059410 con 0x7f8f9415efc0 2024-03-04T09:35:19.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:19.509+0000 7f8f98a00640 1 -- 172.21.15.6:0/1527778102 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8f940ae7a0 con 0x7f8f9415efc0 2024-03-04T09:35:19.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:19.510+0000 7f8f8b400640 1 -- 172.21.15.6:0/1527778102 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3465 (secure 0 0 0) 0x7f8f7c058d10 con 0x7f8f9415efc0 2024-03-04T09:35:19.507 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:35:19.508 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:33:54.652601+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"laggy_since":"2024-03-04T09:33:54.652601+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":9}]} 2024-03-04T09:35:19.508 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 100 2024-03-04T09:35:19.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:19.513+0000 7f8f98a00640 1 -- 172.21.15.6:0/1527778102 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f8f7006b4d0 msgr2=0x7f8f7006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:19.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:19.513+0000 7f8f98a00640 1 --2- 172.21.15.6:0/1527778102 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f8f7006b4d0 0x7f8f7006d990 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f8f84000f60 tx=0x7f8f840078f0 comp rx=0 tx=0).stop 2024-03-04T09:35:19.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:19.513+0000 7f8f98a00640 1 -- 172.21.15.6:0/1527778102 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8f9415efc0 msgr2=0x7f8f941570d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:19.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:19.513+0000 7f8f98a00640 1 --2- 172.21.15.6:0/1527778102 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8f9415efc0 0x7f8f941570d0 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 crypto rx=0x7f8f7c00e9b0 tx=0x7f8f7c00ee70 comp rx=0 tx=0).stop 2024-03-04T09:35:19.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:19.513+0000 7f8f98a00640 1 -- 172.21.15.6:0/1527778102 shutdown_connections 2024-03-04T09:35:19.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:19.513+0000 7f8f98a00640 1 --2- 172.21.15.6:0/1527778102 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f8f7006b4d0 0x7f8f7006d990 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:19.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:19.513+0000 7f8f98a00640 1 --2- 172.21.15.6:0/1527778102 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8f94162000 0x7f8f94157610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:19.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:19.513+0000 7f8f98a00640 1 --2- 172.21.15.6:0/1527778102 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8f9415efc0 0x7f8f941570d0 unknown :-1 s=CLOSED pgs=1704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:19.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:19.513+0000 7f8f98a00640 1 --2- 172.21.15.6:0/1527778102 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8f940cd060 0x7f8f940cee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:19.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:19.513+0000 7f8f98a00640 1 -- 172.21.15.6:0/1527778102 >> 172.21.15.6:0/1527778102 conn(0x7f8f9406b3d0 msgr2=0x7f8f94072070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:35:19.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:19.513+0000 7f8f98a00640 1 -- 172.21.15.6:0/1527778102 shutdown_connections 2024-03-04T09:35:19.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:19.513+0000 7f8f98a00640 1 -- 172.21.15.6:0/1527778102 wait complete. 2024-03-04T09:35:19.733 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:32:57.159681+0000', 'modified': '2024-03-04T09:33:54.652601+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8214, 'mds_1': 6796}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6796': {'gid': 6796, 'name': 'd', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.146:6837/2458070840', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 2458070840}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 2458070840}]}, 'join_fscid': -1, 'laggy_since': '2024-03-04T09:33:54.652601+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8214': {'gid': 8214, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2446627517', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2446627517}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2446627517}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8214, 'qdb_cluster': [8214]} 2024-03-04T09:35:19.734 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6796:up:stopping 2024-03-04T09:35:19.881 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-04T09:35:20.735 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:35:20.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:20.964+0000 7fc95da00640 1 Processor -- start 2024-03-04T09:35:20.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:20.964+0000 7fc95da00640 1 -- start start 2024-03-04T09:35:20.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:20.966+0000 7fc95da00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc958145300 0x7fc9580b49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:20.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:20.966+0000 7fc95da00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc958070f10 0x7fc9580cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:20.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:20.966+0000 7fc95da00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9580714d0 con 0x7fc9580cce90 2024-03-04T09:35:20.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:20.966+0000 7fc95da00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9580cd300 con 0x7fc958145300 2024-03-04T09:35:20.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:20.966+0000 7fc95da00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9580717b0 con 0x7fc958070f10 2024-03-04T09:35:20.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:20.966+0000 7fc957e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc958145300 0x7fc9580b49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:20.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:20.966+0000 7fc95d000640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fc9580cce90 0x7fc9580cd930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:47200/0 (socket says 172.21.15.6:47200) 2024-03-04T09:35:20.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:20.966+0000 7fc95d000640 1 -- 172.21.15.6:0/3861560414 learned_addr learned my addr 172.21.15.6:0/3861560414 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:35:20.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:20.967+0000 7fc957400640 1 --2- 172.21.15.6:0/3861560414 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc958070f10 0x7fc9580cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:20.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:20.967+0000 7fc957e00640 1 -- 172.21.15.6:0/3861560414 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc958070f10 msgr2=0x7fc9580cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:35:20.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:20.967+0000 7fc957e00640 1 --2- 172.21.15.6:0/3861560414 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc958070f10 0x7fc9580cc950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:20.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:20.967+0000 7fc957e00640 1 -- 172.21.15.6:0/3861560414 >> v1:172.21.15.6:6789/0 conn(0x7fc9580cce90 legacy=0x7fc9580cd930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:20.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:20.967+0000 7fc957e00640 1 -- 172.21.15.6:0/3861560414 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc958071a90 con 0x7fc958145300 2024-03-04T09:35:20.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:20.967+0000 7fc956a00640 1 -- 172.21.15.6:0/3861560414 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 645277053 0 0) 0x7fc9580714d0 con 0x7fc9580cce90 2024-03-04T09:35:20.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:20.968+0000 7fc957e00640 1 --2- 172.21.15.6:0/3861560414 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc958145300 0x7fc9580b49a0 secure :-1 s=READY pgs=1706 cs=0 l=1 rev1=1 crypto rx=0x7fc948009d90 tx=0x7fc94802bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89d471eb0db687da server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:35:20.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:20.968+0000 7fc956a00640 1 -- 172.21.15.6:0/3861560414 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc948039070 con 0x7fc958145300 2024-03-04T09:35:20.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:20.968+0000 7fc956a00640 1 -- 172.21.15.6:0/3861560414 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc94802cc50 con 0x7fc958145300 2024-03-04T09:35:20.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:20.968+0000 7fc956a00640 1 -- 172.21.15.6:0/3861560414 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc948034440 con 0x7fc958145300 2024-03-04T09:35:20.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:20.969+0000 7fc95da00640 1 -- 172.21.15.6:0/3861560414 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc958145300 msgr2=0x7fc9580b49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:20.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:20.969+0000 7fc95da00640 1 --2- 172.21.15.6:0/3861560414 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc958145300 0x7fc9580b49a0 secure :-1 s=READY pgs=1706 cs=0 l=1 rev1=1 crypto rx=0x7fc948009d90 tx=0x7fc94802bea0 comp rx=0 tx=0).stop 2024-03-04T09:35:20.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:20.969+0000 7fc95da00640 1 -- 172.21.15.6:0/3861560414 shutdown_connections 2024-03-04T09:35:20.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:20.969+0000 7fc95da00640 1 --2- 172.21.15.6:0/3861560414 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc958070f10 0x7fc9580cc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:20.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:20.969+0000 7fc95da00640 1 --2- 172.21.15.6:0/3861560414 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc958145300 0x7fc9580b49a0 unknown :-1 s=CLOSED pgs=1706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:20.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:20.969+0000 7fc95da00640 1 -- 172.21.15.6:0/3861560414 >> 172.21.15.6:0/3861560414 conn(0x7fc95806b3d0 msgr2=0x7fc95806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:35:20.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:20.969+0000 7fc95da00640 1 -- 172.21.15.6:0/3861560414 shutdown_connections 2024-03-04T09:35:20.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:20.969+0000 7fc95da00640 1 -- 172.21.15.6:0/3861560414 wait complete. 2024-03-04T09:35:20.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:20.971+0000 7fc95da00640 1 Processor -- start 2024-03-04T09:35:20.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:20.971+0000 7fc95da00640 1 -- start start 2024-03-04T09:35:20.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:20.972+0000 7fc95da00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc958070f10 0x7fc95815b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:20.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:20.972+0000 7fc95da00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc9580cce90 0x7fc95815e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:20.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:20.972+0000 7fc95da00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc958145300 0x7fc95815eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:20.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:20.972+0000 7fc95da00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9581456e0 con 0x7fc9580cce90 2024-03-04T09:35:20.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:20.972+0000 7fc95da00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9580b4ee0 con 0x7fc958070f10 2024-03-04T09:35:20.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:20.972+0000 7fc95da00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc95815d580 con 0x7fc958145300 2024-03-04T09:35:20.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:20.972+0000 7fc957400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc9580cce90 0x7fc95815e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:20.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:20.972+0000 7fc957400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc9580cce90 0x7fc95815e650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:35324/0 (socket says 172.21.15.6:35324) 2024-03-04T09:35:20.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:20.972+0000 7fc957400640 1 -- 172.21.15.6:0/3712552631 learned_addr learned my addr 172.21.15.6:0/3712552631 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:35:20.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:20.972+0000 7fc957400640 1 -- 172.21.15.6:0/3712552631 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc958145300 msgr2=0x7fc95815eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:20.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:20.972+0000 7fc957e00640 1 --2- 172.21.15.6:0/3712552631 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc958070f10 0x7fc95815b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:20.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:20.972+0000 7fc95d000640 1 --2- 172.21.15.6:0/3712552631 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc958145300 0x7fc95815eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:20.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:20.972+0000 7fc957400640 1 --2- 172.21.15.6:0/3712552631 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc958145300 0x7fc95815eb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:20.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:20.973+0000 7fc957400640 1 -- 172.21.15.6:0/3712552631 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc958070f10 msgr2=0x7fc95815b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:20.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:20.973+0000 7fc957400640 1 --2- 172.21.15.6:0/3712552631 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc958070f10 0x7fc95815b4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:20.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:20.973+0000 7fc957400640 1 -- 172.21.15.6:0/3712552631 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9580af360 con 0x7fc9580cce90 2024-03-04T09:35:20.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:20.973+0000 7fc957e00640 1 --2- 172.21.15.6:0/3712552631 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc958070f10 0x7fc95815b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:35:20.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:20.973+0000 7fc957400640 1 --2- 172.21.15.6:0/3712552631 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc9580cce90 0x7fc95815e650 secure :-1 s=READY pgs=1707 cs=0 l=1 rev1=1 crypto rx=0x7fc94000d800 tx=0x7fc94000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:35:20.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:20.973+0000 7fc94fe00640 1 -- 172.21.15.6:0/3712552631 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc94000b6c0 con 0x7fc9580cce90 2024-03-04T09:35:20.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:20.973+0000 7fc94fe00640 1 -- 172.21.15.6:0/3712552631 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc940019070 con 0x7fc9580cce90 2024-03-04T09:35:20.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:20.973+0000 7fc94fe00640 1 -- 172.21.15.6:0/3712552631 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc94000ba80 con 0x7fc9580cce90 2024-03-04T09:35:20.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:20.973+0000 7fc95da00640 1 -- 172.21.15.6:0/3712552631 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9580aeac0 con 0x7fc9580cce90 2024-03-04T09:35:20.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:20.973+0000 7fc95da00640 1 -- 172.21.15.6:0/3712552631 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9580af080 con 0x7fc9580cce90 2024-03-04T09:35:20.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:20.974+0000 7fc95da00640 1 -- 172.21.15.6:0/3712552631 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc95804f330 con 0x7fc9580cce90 2024-03-04T09:35:20.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:20.976+0000 7fc94fe00640 1 -- 172.21.15.6:0/3712552631 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc94000bc20 con 0x7fc9580cce90 2024-03-04T09:35:20.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:20.976+0000 7fc94fe00640 1 --2- 172.21.15.6:0/3712552631 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc93406b4d0 0x7fc93406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:20.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:20.977+0000 7fc94fe00640 1 -- 172.21.15.6:0/3712552631 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fc94008a790 con 0x7fc9580cce90 2024-03-04T09:35:20.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:20.977+0000 7fc957e00640 1 --2- 172.21.15.6:0/3712552631 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc93406b4d0 0x7fc93406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:20.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:20.978+0000 7fc957e00640 1 --2- 172.21.15.6:0/3712552631 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc93406b4d0 0x7fc93406d990 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fc948036000 tx=0x7fc948030000 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:35:20.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:20.982+0000 7fc94fe00640 1 -- 172.21.15.6:0/3712552631 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc940059530 con 0x7fc9580cce90 2024-03-04T09:35:21.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:21.377+0000 7fc95da00640 1 -- 172.21.15.6:0/3712552631 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc958144a20 con 0x7fc9580cce90 2024-03-04T09:35:21.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:21.378+0000 7fc94fe00640 1 -- 172.21.15.6:0/3712552631 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3465 (secure 0 0 0) 0x7fc940058e30 con 0x7fc9580cce90 2024-03-04T09:35:21.375 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:35:21.375 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:33:54.652601+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"laggy_since":"2024-03-04T09:33:54.652601+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":9}]} 2024-03-04T09:35:21.376 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 100 2024-03-04T09:35:21.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:21.380+0000 7fc95da00640 1 -- 172.21.15.6:0/3712552631 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc93406b4d0 msgr2=0x7fc93406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:21.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:21.381+0000 7fc95da00640 1 --2- 172.21.15.6:0/3712552631 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc93406b4d0 0x7fc93406d990 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fc948036000 tx=0x7fc948030000 comp rx=0 tx=0).stop 2024-03-04T09:35:21.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:21.381+0000 7fc95da00640 1 -- 172.21.15.6:0/3712552631 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc9580cce90 msgr2=0x7fc95815e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:21.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:21.381+0000 7fc95da00640 1 --2- 172.21.15.6:0/3712552631 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc9580cce90 0x7fc95815e650 secure :-1 s=READY pgs=1707 cs=0 l=1 rev1=1 crypto rx=0x7fc94000d800 tx=0x7fc94000dcc0 comp rx=0 tx=0).stop 2024-03-04T09:35:21.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:21.381+0000 7fc95da00640 1 -- 172.21.15.6:0/3712552631 shutdown_connections 2024-03-04T09:35:21.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:21.381+0000 7fc95da00640 1 --2- 172.21.15.6:0/3712552631 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc93406b4d0 0x7fc93406d990 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:21.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:21.381+0000 7fc95da00640 1 --2- 172.21.15.6:0/3712552631 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc958145300 0x7fc95815eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:21.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:21.381+0000 7fc95da00640 1 --2- 172.21.15.6:0/3712552631 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc9580cce90 0x7fc95815e650 unknown :-1 s=CLOSED pgs=1707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:21.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:21.381+0000 7fc95da00640 1 --2- 172.21.15.6:0/3712552631 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc958070f10 0x7fc95815b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:21.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:21.381+0000 7fc95da00640 1 -- 172.21.15.6:0/3712552631 >> 172.21.15.6:0/3712552631 conn(0x7fc95806b3d0 msgr2=0x7fc95815dc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:35:21.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:21.381+0000 7fc95da00640 1 -- 172.21.15.6:0/3712552631 shutdown_connections 2024-03-04T09:35:21.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:21.381+0000 7fc95da00640 1 -- 172.21.15.6:0/3712552631 wait complete. 2024-03-04T09:35:21.597 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:32:57.159681+0000', 'modified': '2024-03-04T09:33:54.652601+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8214, 'mds_1': 6796}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6796': {'gid': 6796, 'name': 'd', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.146:6837/2458070840', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 2458070840}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 2458070840}]}, 'join_fscid': -1, 'laggy_since': '2024-03-04T09:33:54.652601+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8214': {'gid': 8214, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2446627517', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2446627517}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2446627517}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8214, 'qdb_cluster': [8214]} 2024-03-04T09:35:21.597 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6796:up:stopping 2024-03-04T09:35:21.883 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-04T09:35:22.598 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:35:22.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:22.824+0000 7f96f7e00640 1 Processor -- start 2024-03-04T09:35:22.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:22.824+0000 7f96f7e00640 1 -- start start 2024-03-04T09:35:22.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:22.826+0000 7f96f7e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f96f815ded0 0x7f96f806c490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:22.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:22.826+0000 7f96f7e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f96f80b61b0 0x7f96f806cab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:22.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:22.826+0000 7f96f7e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f96f806cff0 0x7f96f81619e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:22.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:22.826+0000 7f96f7e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96f815e2b0 con 0x7f96f806cff0 2024-03-04T09:35:22.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:22.826+0000 7f96f7e00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96f80b6970 con 0x7f96f80b61b0 2024-03-04T09:35:22.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:22.826+0000 7f96f7e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96f80b6c50 con 0x7f96f815ded0 2024-03-04T09:35:22.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:22.827+0000 7f96f7400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f96f806cff0 0x7f96f81619e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:22.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:22.827+0000 7f96f7400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f96f806cff0 0x7f96f81619e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:35336/0 (socket says 172.21.15.6:35336) 2024-03-04T09:35:22.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:22.827+0000 7f96f7400640 1 -- 172.21.15.6:0/3676188907 learned_addr learned my addr 172.21.15.6:0/3676188907 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:35:22.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:22.827+0000 7f96efe00640 1 --2- 172.21.15.6:0/3676188907 >> v2:172.21.15.146:3300/0 conn(0x7f96f80b61b0 0x7f96f806cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:22.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:22.827+0000 7f96f7400640 1 -- 172.21.15.6:0/3676188907 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f96f815ded0 msgr2=0x7f96f806c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:35:22.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:22.827+0000 7f96f6a00640 1 --2- 172.21.15.6:0/3676188907 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f96f815ded0 0x7f96f806c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:22.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:22.827+0000 7f96f7400640 1 --2- 172.21.15.6:0/3676188907 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f96f815ded0 0x7f96f806c490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:22.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:22.827+0000 7f96f7400640 1 -- 172.21.15.6:0/3676188907 >> v2:172.21.15.146:3300/0 conn(0x7f96f80b61b0 msgr2=0x7f96f806cab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:35:22.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:22.827+0000 7f96f7400640 1 --2- 172.21.15.6:0/3676188907 >> v2:172.21.15.146:3300/0 conn(0x7f96f80b61b0 0x7f96f806cab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:22.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:22.827+0000 7f96f7400640 1 -- 172.21.15.6:0/3676188907 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96f80aeb30 con 0x7f96f806cff0 2024-03-04T09:35:22.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:22.827+0000 7f96efe00640 1 --2- 172.21.15.6:0/3676188907 >> v2:172.21.15.146:3300/0 conn(0x7f96f80b61b0 0x7f96f806cab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:35:22.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:22.827+0000 7f96f6a00640 1 --2- 172.21.15.6:0/3676188907 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f96f815ded0 0x7f96f806c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:35:22.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:22.828+0000 7f96f7400640 1 --2- 172.21.15.6:0/3676188907 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f96f806cff0 0x7f96f81619e0 secure :-1 s=READY pgs=1708 cs=0 l=1 rev1=1 crypto rx=0x7f96f8071720 tx=0x7f96f002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d487800945b349bd server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:35:22.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:22.828+0000 7f96f6000640 1 -- 172.21.15.6:0/3676188907 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96f002ca80 con 0x7f96f806cff0 2024-03-04T09:35:22.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:22.829+0000 7f96f6000640 1 -- 172.21.15.6:0/3676188907 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96f002cc20 con 0x7f96f806cff0 2024-03-04T09:35:22.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:22.829+0000 7f96f6000640 1 -- 172.21.15.6:0/3676188907 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96f00343f0 con 0x7f96f806cff0 2024-03-04T09:35:22.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:22.829+0000 7f96f7e00640 1 -- 172.21.15.6:0/3676188907 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f96f806cff0 msgr2=0x7f96f81619e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:22.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:22.829+0000 7f96f7e00640 1 --2- 172.21.15.6:0/3676188907 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f96f806cff0 0x7f96f81619e0 secure :-1 s=READY pgs=1708 cs=0 l=1 rev1=1 crypto rx=0x7f96f8071720 tx=0x7f96f002bee0 comp rx=0 tx=0).stop 2024-03-04T09:35:22.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:22.829+0000 7f96f7e00640 1 -- 172.21.15.6:0/3676188907 shutdown_connections 2024-03-04T09:35:22.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:22.829+0000 7f96f7e00640 1 --2- 172.21.15.6:0/3676188907 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f96f806cff0 0x7f96f81619e0 unknown :-1 s=CLOSED pgs=1708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:22.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:22.829+0000 7f96f7e00640 1 --2- 172.21.15.6:0/3676188907 >> v2:172.21.15.146:3300/0 conn(0x7f96f80b61b0 0x7f96f806cab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:22.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:22.829+0000 7f96f7e00640 1 --2- 172.21.15.6:0/3676188907 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f96f815ded0 0x7f96f806c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:22.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:22.829+0000 7f96f7e00640 1 -- 172.21.15.6:0/3676188907 >> 172.21.15.6:0/3676188907 conn(0x7f96f80731f0 msgr2=0x7f96f809aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:35:22.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:22.829+0000 7f96f7e00640 1 -- 172.21.15.6:0/3676188907 shutdown_connections 2024-03-04T09:35:22.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:22.829+0000 7f96f7e00640 1 -- 172.21.15.6:0/3676188907 wait complete. 2024-03-04T09:35:22.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:22.831+0000 7f96f7e00640 1 Processor -- start 2024-03-04T09:35:22.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:22.831+0000 7f96f7e00640 1 -- start start 2024-03-04T09:35:22.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:22.832+0000 7f96f7e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f96f806cff0 0x7f96f8155900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:22.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:22.832+0000 7f96f7e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f96f80b61b0 0x7f96f8158740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:22.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:22.832+0000 7f96f7e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f96f815ded0 0x7f96f8159610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:22.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:22.832+0000 7f96f7e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96f8161f20 con 0x7f96f815ded0 2024-03-04T09:35:22.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:22.832+0000 7f96f7e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96f80b6610 con 0x7f96f80b61b0 2024-03-04T09:35:22.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:22.832+0000 7f96f7e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96f81620a0 con 0x7f96f806cff0 2024-03-04T09:35:22.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:22.833+0000 7f96f7400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f96f815ded0 0x7f96f8159610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:22.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:22.833+0000 7f96f7400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f96f815ded0 0x7f96f8159610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:35340/0 (socket says 172.21.15.6:35340) 2024-03-04T09:35:22.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:22.833+0000 7f96f7400640 1 -- 172.21.15.6:0/931700571 learned_addr learned my addr 172.21.15.6:0/931700571 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:35:22.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:22.833+0000 7f96f6a00640 1 --2- 172.21.15.6:0/931700571 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f96f806cff0 0x7f96f8155900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:22.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:22.833+0000 7f96efe00640 1 --2- 172.21.15.6:0/931700571 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f96f80b61b0 0x7f96f8158740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:22.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:22.833+0000 7f96f7400640 1 -- 172.21.15.6:0/931700571 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f96f806cff0 msgr2=0x7f96f8155900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:22.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:22.833+0000 7f96f7400640 1 --2- 172.21.15.6:0/931700571 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f96f806cff0 0x7f96f8155900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:22.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:22.833+0000 7f96f7400640 1 -- 172.21.15.6:0/931700571 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f96f80b61b0 msgr2=0x7f96f8158740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:22.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:22.833+0000 7f96f7400640 1 --2- 172.21.15.6:0/931700571 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f96f80b61b0 0x7f96f8158740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:22.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:22.833+0000 7f96f7400640 1 -- 172.21.15.6:0/931700571 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96f815adb0 con 0x7f96f815ded0 2024-03-04T09:35:22.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:22.833+0000 7f96efe00640 1 --2- 172.21.15.6:0/931700571 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f96f80b61b0 0x7f96f8158740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T09:35:22.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:22.834+0000 7f96f7400640 1 --2- 172.21.15.6:0/931700571 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f96f815ded0 0x7f96f8159610 secure :-1 s=READY pgs=1709 cs=0 l=1 rev1=1 crypto rx=0x7f96f0036040 tx=0x7f96f0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:35:22.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:22.834+0000 7f96eea00640 1 -- 172.21.15.6:0/931700571 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96f0043020 con 0x7f96f815ded0 2024-03-04T09:35:22.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:22.834+0000 7f96eea00640 1 -- 172.21.15.6:0/931700571 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96f003e450 con 0x7f96f815ded0 2024-03-04T09:35:22.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:22.834+0000 7f96eea00640 1 -- 172.21.15.6:0/931700571 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96f0009510 con 0x7f96f815ded0 2024-03-04T09:35:22.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:22.834+0000 7f96f7e00640 1 -- 172.21.15.6:0/931700571 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96f815b090 con 0x7f96f815ded0 2024-03-04T09:35:22.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:22.835+0000 7f96f7e00640 1 -- 172.21.15.6:0/931700571 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96f815b650 con 0x7f96f815ded0 2024-03-04T09:35:22.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:22.836+0000 7f96f7e00640 1 -- 172.21.15.6:0/931700571 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96f804f330 con 0x7f96f815ded0 2024-03-04T09:35:22.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:22.837+0000 7f96eea00640 1 -- 172.21.15.6:0/931700571 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f96f0046020 con 0x7f96f815ded0 2024-03-04T09:35:22.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:22.837+0000 7f96eea00640 1 --2- 172.21.15.6:0/931700571 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f96d406b4d0 0x7f96d406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:22.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:22.838+0000 7f96eea00640 1 -- 172.21.15.6:0/931700571 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f96f00acad0 con 0x7f96f815ded0 2024-03-04T09:35:22.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:22.838+0000 7f96f6a00640 1 --2- 172.21.15.6:0/931700571 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f96d406b4d0 0x7f96d406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:22.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:22.839+0000 7f96f6a00640 1 --2- 172.21.15.6:0/931700571 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f96d406b4d0 0x7f96d406d990 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f96e800aac0 tx=0x7f96e8008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:35:22.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:22.843+0000 7f96eea00640 1 -- 172.21.15.6:0/931700571 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f96f004d030 con 0x7f96f815ded0 2024-03-04T09:35:23.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:23.105+0000 7f96f7e00640 1 -- 172.21.15.6:0/931700571 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f96f815d6d0 con 0x7f96f815ded0 2024-03-04T09:35:23.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:23.106+0000 7f96eea00640 1 -- 172.21.15.6:0/931700571 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3465 (secure 0 0 0) 0x7f96f007b900 con 0x7f96f815ded0 2024-03-04T09:35:23.103 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:35:23.103 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:33:54.652601+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"laggy_since":"2024-03-04T09:33:54.652601+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":9}]} 2024-03-04T09:35:23.104 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 100 2024-03-04T09:35:23.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:23.109+0000 7f96f7e00640 1 -- 172.21.15.6:0/931700571 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f96d406b4d0 msgr2=0x7f96d406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:23.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:23.109+0000 7f96f7e00640 1 --2- 172.21.15.6:0/931700571 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f96d406b4d0 0x7f96d406d990 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f96e800aac0 tx=0x7f96e8008040 comp rx=0 tx=0).stop 2024-03-04T09:35:23.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:23.109+0000 7f96f7e00640 1 -- 172.21.15.6:0/931700571 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f96f815ded0 msgr2=0x7f96f8159610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:23.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:23.109+0000 7f96f7e00640 1 --2- 172.21.15.6:0/931700571 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f96f815ded0 0x7f96f8159610 secure :-1 s=READY pgs=1709 cs=0 l=1 rev1=1 crypto rx=0x7f96f0036040 tx=0x7f96f0004a40 comp rx=0 tx=0).stop 2024-03-04T09:35:23.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:23.109+0000 7f96f7e00640 1 -- 172.21.15.6:0/931700571 shutdown_connections 2024-03-04T09:35:23.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:23.109+0000 7f96f7e00640 1 --2- 172.21.15.6:0/931700571 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f96f815ded0 0x7f96f8159610 unknown :-1 s=CLOSED pgs=1709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:23.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:23.109+0000 7f96f7e00640 1 --2- 172.21.15.6:0/931700571 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f96f80b61b0 0x7f96f8158740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:23.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:23.109+0000 7f96f7e00640 1 --2- 172.21.15.6:0/931700571 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f96d406b4d0 0x7f96d406d990 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:23.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:23.109+0000 7f96f7e00640 1 --2- 172.21.15.6:0/931700571 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f96f806cff0 0x7f96f8155900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:23.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:23.109+0000 7f96f7e00640 1 -- 172.21.15.6:0/931700571 >> 172.21.15.6:0/931700571 conn(0x7f96f80731f0 msgr2=0x7f96f815e8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:35:23.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:23.109+0000 7f96f7e00640 1 -- 172.21.15.6:0/931700571 shutdown_connections 2024-03-04T09:35:23.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:23.109+0000 7f96f7e00640 1 -- 172.21.15.6:0/931700571 wait complete. 2024-03-04T09:35:23.326 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:32:57.159681+0000', 'modified': '2024-03-04T09:33:54.652601+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8214, 'mds_1': 6796}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6796': {'gid': 6796, 'name': 'd', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.146:6837/2458070840', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 2458070840}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 2458070840}]}, 'join_fscid': -1, 'laggy_since': '2024-03-04T09:33:54.652601+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8214': {'gid': 8214, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2446627517', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2446627517}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2446627517}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8214, 'qdb_cluster': [8214]} 2024-03-04T09:35:23.326 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6796:up:stopping 2024-03-04T09:35:23.885 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-04T09:35:24.327 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:35:24.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:24.553+0000 7f5158a00640 1 Processor -- start 2024-03-04T09:35:24.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:24.554+0000 7f5158a00640 1 -- start start 2024-03-04T09:35:24.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:24.557+0000 7f5158a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f515415a550 0x7f51540aefe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:24.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:24.557+0000 7f5158a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5154071640 con 0x7f51540ade50 2024-03-04T09:35:24.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:24.557+0000 7f5158a00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5154071920 con 0x7f515415a930 2024-03-04T09:35:24.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:24.557+0000 7f5158a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5154071c00 con 0x7f515415a550 2024-03-04T09:35:24.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:24.557+0000 7f5152800640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f51540ade50 0x7f51540ce330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:47208/0 (socket says 172.21.15.6:47208) 2024-03-04T09:35:24.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:24.557+0000 7f5152800640 1 -- 172.21.15.6:0/3363953758 learned_addr learned my addr 172.21.15.6:0/3363953758 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:35:24.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:24.557+0000 7f5151e00640 1 --2- 172.21.15.6:0/3363953758 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f515415a550 0x7f51540aefe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:24.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:24.558+0000 7f5151400640 1 -- 172.21.15.6:0/3363953758 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2920747929 0 0) 0x7f5154071640 con 0x7f51540ade50 2024-03-04T09:35:24.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:24.558+0000 7f5151400640 1 -- 172.21.15.6:0/3363953758 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5138003520 con 0x7f51540ade50 2024-03-04T09:35:24.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:24.558+0000 7f5151400640 1 -- 172.21.15.6:0/3363953758 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 482409920 0 0) 0x7f5138003520 con 0x7f51540ade50 2024-03-04T09:35:24.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:24.558+0000 7f5151400640 1 -- 172.21.15.6:0/3363953758 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f515415a550 msgr2=0x7f51540aefe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:35:24.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:24.558+0000 7f5151400640 1 --2- 172.21.15.6:0/3363953758 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f515415a550 0x7f51540aefe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:24.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:24.558+0000 7f5151400640 1 -- 172.21.15.6:0/3363953758 >> v1:172.21.15.146:6789/0 conn(0x7f515415a930 legacy=0x7f51540cdc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:24.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:24.558+0000 7f5151400640 1 -- 172.21.15.6:0/3363953758 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5154071ee0 con 0x7f51540ade50 2024-03-04T09:35:24.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:24.558+0000 7f5151400640 1 -- 172.21.15.6:0/3363953758 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7f514c001670 con 0x7f51540ade50 2024-03-04T09:35:24.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:24.558+0000 7f5151400640 1 -- 172.21.15.6:0/3363953758 >> v1:172.21.15.6:6789/0 conn(0x7f51540ade50 legacy=0x7f51540ce330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:24.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:24.558+0000 7f5151400640 1 --2- 172.21.15.6:0/3363953758 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f515415a550 0x7f5138003a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:24.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:24.558+0000 7f5151400640 1 --2- 172.21.15.6:0/3363953758 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5138004020 0x7f5138004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:24.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:24.558+0000 7f5151400640 1 --2- 172.21.15.6:0/3363953758 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5138004d40 0x7f51380051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:24.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:24.558+0000 7f5151400640 1 -- 172.21.15.6:0/3363953758 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5154071c00 con 0x7f5138004d40 2024-03-04T09:35:24.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:24.558+0000 7f5151400640 1 -- 172.21.15.6:0/3363953758 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f513800bc10 con 0x7f5138004020 2024-03-04T09:35:24.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:24.558+0000 7f5151400640 1 -- 172.21.15.6:0/3363953758 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f513800e020 con 0x7f515415a550 2024-03-04T09:35:24.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:24.559+0000 7f5152800640 1 --2- 172.21.15.6:0/3363953758 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5138004d40 0x7f51380051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:24.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:24.559+0000 7f5151e00640 1 --2- 172.21.15.6:0/3363953758 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f515415a550 0x7f5138003a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:24.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:24.559+0000 7f5149400640 1 --2- 172.21.15.6:0/3363953758 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5138004020 0x7f5138004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:24.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:24.559+0000 7f5152800640 1 -- 172.21.15.6:0/3363953758 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f515415a550 msgr2=0x7f5138003a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:35:24.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:24.559+0000 7f5152800640 1 --2- 172.21.15.6:0/3363953758 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f515415a550 0x7f5138003a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:24.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:24.559+0000 7f5152800640 1 -- 172.21.15.6:0/3363953758 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5138004020 msgr2=0x7f5138004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:35:24.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:24.559+0000 7f5152800640 1 --2- 172.21.15.6:0/3363953758 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5138004020 0x7f5138004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:24.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:24.559+0000 7f5152800640 1 -- 172.21.15.6:0/3363953758 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5138003520 con 0x7f5138004d40 2024-03-04T09:35:24.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:24.560+0000 7f5152800640 1 --2- 172.21.15.6:0/3363953758 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5138004d40 0x7f51380051c0 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 crypto rx=0x7f514c007fd0 tx=0x7f5138005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1df66e6343b933a6 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:35:24.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:24.561+0000 7f5151400640 1 -- 172.21.15.6:0/3363953758 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f514c007c20 con 0x7f5138004d40 2024-03-04T09:35:24.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:24.561+0000 7f5151400640 1 -- 172.21.15.6:0/3363953758 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f514c030070 con 0x7f5138004d40 2024-03-04T09:35:24.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:24.561+0000 7f5151400640 1 -- 172.21.15.6:0/3363953758 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f514c005d70 con 0x7f5138004d40 2024-03-04T09:35:24.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:24.561+0000 7f5158a00640 1 -- 172.21.15.6:0/3363953758 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5138004d40 msgr2=0x7f51380051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:24.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:24.561+0000 7f5158a00640 1 --2- 172.21.15.6:0/3363953758 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5138004d40 0x7f51380051c0 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 crypto rx=0x7f514c007fd0 tx=0x7f5138005830 comp rx=0 tx=0).stop 2024-03-04T09:35:24.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:24.561+0000 7f5152800640 1 -- 172.21.15.6:0/3363953758 reap_dead start 2024-03-04T09:35:24.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:24.561+0000 7f5158a00640 1 -- 172.21.15.6:0/3363953758 shutdown_connections 2024-03-04T09:35:24.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:24.561+0000 7f5158a00640 1 -- 172.21.15.6:0/3363953758 >> 172.21.15.6:0/3363953758 conn(0x7f515406afb0 msgr2=0x7f515406b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:35:24.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:24.562+0000 7f5158a00640 1 -- 172.21.15.6:0/3363953758 shutdown_connections 2024-03-04T09:35:24.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:24.562+0000 7f5158a00640 1 -- 172.21.15.6:0/3363953758 wait complete. 2024-03-04T09:35:24.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:24.563+0000 7f5158a00640 1 Processor -- start 2024-03-04T09:35:24.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:24.563+0000 7f5158a00640 1 -- start start 2024-03-04T09:35:24.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:24.564+0000 7f5158a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5138004d40 0x7f5154150420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:24.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:24.564+0000 7f5158a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5154150dc0 0x7f515415c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:24.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:24.564+0000 7f5158a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f515415cd30 0x7f51540cb160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:24.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:24.564+0000 7f5158a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51540cea40 con 0x7f515415cd30 2024-03-04T09:35:24.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:24.564+0000 7f5158a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51540b5340 con 0x7f5138004d40 2024-03-04T09:35:24.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:24.564+0000 7f5158a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51540cebc0 con 0x7f5154150dc0 2024-03-04T09:35:24.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:24.565+0000 7f5152800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f515415cd30 0x7f51540cb160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:24.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:24.565+0000 7f5152800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f515415cd30 0x7f51540cb160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:35360/0 (socket says 172.21.15.6:35360) 2024-03-04T09:35:24.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:24.565+0000 7f5152800640 1 -- 172.21.15.6:0/1779626155 learned_addr learned my addr 172.21.15.6:0/1779626155 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:35:24.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:24.565+0000 7f5151e00640 1 --2- 172.21.15.6:0/1779626155 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5138004d40 0x7f5154150420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:24.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:24.565+0000 7f5149400640 1 --2- 172.21.15.6:0/1779626155 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5154150dc0 0x7f515415c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:24.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:24.565+0000 7f5152800640 1 -- 172.21.15.6:0/1779626155 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5154150dc0 msgr2=0x7f515415c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:24.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:24.565+0000 7f5152800640 1 --2- 172.21.15.6:0/1779626155 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5154150dc0 0x7f515415c7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:24.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:24.565+0000 7f5152800640 1 -- 172.21.15.6:0/1779626155 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5138004d40 msgr2=0x7f5154150420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:24.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:24.565+0000 7f5152800640 1 --2- 172.21.15.6:0/1779626155 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5138004d40 0x7f5154150420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:24.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:24.565+0000 7f5152800640 1 -- 172.21.15.6:0/1779626155 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f515409ac60 con 0x7f515415cd30 2024-03-04T09:35:24.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:24.566+0000 7f5152800640 1 --2- 172.21.15.6:0/1779626155 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f515415cd30 0x7f51540cb160 secure :-1 s=READY pgs=1712 cs=0 l=1 rev1=1 crypto rx=0x7f514c02c430 tx=0x7f5154071af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:35:24.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:24.566+0000 7f514aa00640 1 -- 172.21.15.6:0/1779626155 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f514c001b60 con 0x7f515415cd30 2024-03-04T09:35:24.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:24.566+0000 7f514aa00640 1 -- 172.21.15.6:0/1779626155 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f514c000ed0 con 0x7f515415cd30 2024-03-04T09:35:24.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:24.566+0000 7f514aa00640 1 -- 172.21.15.6:0/1779626155 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f514c02cdd0 con 0x7f515415cd30 2024-03-04T09:35:24.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:24.566+0000 7f5158a00640 1 -- 172.21.15.6:0/1779626155 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f515409af40 con 0x7f515415cd30 2024-03-04T09:35:24.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:24.566+0000 7f5158a00640 1 -- 172.21.15.6:0/1779626155 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f515409b500 con 0x7f515415cd30 2024-03-04T09:35:24.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:24.567+0000 7f5158a00640 1 -- 172.21.15.6:0/1779626155 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5154069510 con 0x7f515415cd30 2024-03-04T09:35:24.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:24.568+0000 7f514aa00640 1 -- 172.21.15.6:0/1779626155 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f514c007070 con 0x7f515415cd30 2024-03-04T09:35:24.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:24.569+0000 7f514aa00640 1 --2- 172.21.15.6:0/1779626155 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f513006b4d0 0x7f513006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:24.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:24.569+0000 7f514aa00640 1 -- 172.21.15.6:0/1779626155 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f514c0aafe0 con 0x7f515415cd30 2024-03-04T09:35:24.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:24.570+0000 7f5151e00640 1 --2- 172.21.15.6:0/1779626155 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f513006b4d0 0x7f513006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:24.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:24.571+0000 7f5151e00640 1 --2- 172.21.15.6:0/1779626155 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f513006b4d0 0x7f513006d990 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f5144007520 tx=0x7f514400b040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:35:24.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:24.574+0000 7f514aa00640 1 -- 172.21.15.6:0/1779626155 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f514c079e10 con 0x7f515415cd30 2024-03-04T09:35:24.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:24.813+0000 7f5158a00640 1 -- 172.21.15.6:0/1779626155 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f515404f3a0 con 0x7f515415cd30 2024-03-04T09:35:24.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:24.814+0000 7f514aa00640 1 -- 172.21.15.6:0/1779626155 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3465 (secure 0 0 0) 0x7f514c079710 con 0x7f515415cd30 2024-03-04T09:35:24.811 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:35:24.811 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:33:54.652601+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"laggy_since":"2024-03-04T09:33:54.652601+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":9}]} 2024-03-04T09:35:24.812 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 100 2024-03-04T09:35:24.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:24.817+0000 7f5158a00640 1 -- 172.21.15.6:0/1779626155 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f513006b4d0 msgr2=0x7f513006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:24.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:24.817+0000 7f5158a00640 1 --2- 172.21.15.6:0/1779626155 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f513006b4d0 0x7f513006d990 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f5144007520 tx=0x7f514400b040 comp rx=0 tx=0).stop 2024-03-04T09:35:24.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:24.817+0000 7f5158a00640 1 -- 172.21.15.6:0/1779626155 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f515415cd30 msgr2=0x7f51540cb160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:24.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:24.817+0000 7f5158a00640 1 --2- 172.21.15.6:0/1779626155 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f515415cd30 0x7f51540cb160 secure :-1 s=READY pgs=1712 cs=0 l=1 rev1=1 crypto rx=0x7f514c02c430 tx=0x7f5154071af0 comp rx=0 tx=0).stop 2024-03-04T09:35:24.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:24.817+0000 7f5158a00640 1 -- 172.21.15.6:0/1779626155 shutdown_connections 2024-03-04T09:35:24.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:24.817+0000 7f5158a00640 1 --2- 172.21.15.6:0/1779626155 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f515415cd30 0x7f51540cb160 unknown :-1 s=CLOSED pgs=1712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:24.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:24.817+0000 7f5158a00640 1 --2- 172.21.15.6:0/1779626155 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f513006b4d0 0x7f513006d990 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:24.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:24.817+0000 7f5158a00640 1 --2- 172.21.15.6:0/1779626155 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5154150dc0 0x7f515415c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:24.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:24.817+0000 7f5158a00640 1 --2- 172.21.15.6:0/1779626155 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5138004d40 0x7f5154150420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:24.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:24.817+0000 7f5158a00640 1 -- 172.21.15.6:0/1779626155 >> 172.21.15.6:0/1779626155 conn(0x7f515406afb0 msgr2=0x7f515406b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:35:24.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:24.817+0000 7f5158a00640 1 -- 172.21.15.6:0/1779626155 shutdown_connections 2024-03-04T09:35:24.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:24.817+0000 7f5158a00640 1 -- 172.21.15.6:0/1779626155 wait complete. 2024-03-04T09:35:25.034 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:32:57.159681+0000', 'modified': '2024-03-04T09:33:54.652601+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8214, 'mds_1': 6796}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6796': {'gid': 6796, 'name': 'd', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.146:6837/2458070840', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 2458070840}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 2458070840}]}, 'join_fscid': -1, 'laggy_since': '2024-03-04T09:33:54.652601+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8214': {'gid': 8214, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2446627517', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2446627517}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2446627517}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8214, 'qdb_cluster': [8214]} 2024-03-04T09:35:25.034 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6796:up:stopping 2024-03-04T09:35:25.887 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-04T09:35:26.035 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:35:26.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:26.270+0000 7f2432600640 1 Processor -- start 2024-03-04T09:35:26.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:26.270+0000 7f2432600640 1 -- start start 2024-03-04T09:35:26.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:26.272+0000 7f2432600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f242c15e070 0x7f242c0d2960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:26.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:26.272+0000 7f2432600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f242c0cf410 0x7f242c0cf870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:26.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:26.272+0000 7f2432600640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f242c0d18c0 con 0x7f242c0d05c0 2024-03-04T09:35:26.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:26.272+0000 7f2432600640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f242c0d1ba0 con 0x7f242c0cf410 2024-03-04T09:35:26.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:26.272+0000 7f2432600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f242c0d1e80 con 0x7f242c15e070 2024-03-04T09:35:26.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:26.272+0000 7f242be00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f242c0cf410 0x7f242c0cf870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:26.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:26.272+0000 7f242be00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f242c0cf410 0x7f242c0cf870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:35372/0 (socket says 172.21.15.6:35372) 2024-03-04T09:35:26.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:26.272+0000 7f242be00640 1 -- 172.21.15.6:0/2615984082 learned_addr learned my addr 172.21.15.6:0/2615984082 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:35:26.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:26.272+0000 7f2431c00640 1 --1- 172.21.15.6:0/2615984082 >> v1:172.21.15.6:6789/0 conn(0x7f242c0d05c0 0x7f242c0b3630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:47220/0 (socket says 172.21.15.6:47220) 2024-03-04T09:35:26.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:26.272+0000 7f2431200640 1 --2- 172.21.15.6:0/2615984082 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f242c15e070 0x7f242c0d2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:26.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:26.273+0000 7f242be00640 1 -- 172.21.15.6:0/2615984082 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f242c15e070 msgr2=0x7f242c0d2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:35:26.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:26.273+0000 7f242be00640 1 --2- 172.21.15.6:0/2615984082 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f242c15e070 0x7f242c0d2960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:26.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:26.273+0000 7f242be00640 1 -- 172.21.15.6:0/2615984082 >> v1:172.21.15.6:6789/0 conn(0x7f242c0d05c0 legacy=0x7f242c0b3630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:26.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:26.273+0000 7f242be00640 1 -- 172.21.15.6:0/2615984082 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f242c0d2160 con 0x7f242c0cf410 2024-03-04T09:35:26.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:26.273+0000 7f2431200640 1 --2- 172.21.15.6:0/2615984082 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f242c15e070 0x7f242c0d2960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:35:26.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:26.274+0000 7f242be00640 1 --2- 172.21.15.6:0/2615984082 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f242c0cf410 0x7f242c0cf870 secure :-1 s=READY pgs=1714 cs=0 l=1 rev1=1 crypto rx=0x7f2418009870 tx=0x7f241802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=555da2129701af02 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:35:26.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:26.274+0000 7f242b400640 1 -- 172.21.15.6:0/2615984082 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2418034070 con 0x7f242c0cf410 2024-03-04T09:35:26.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:26.274+0000 7f242b400640 1 -- 172.21.15.6:0/2615984082 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2418030840 con 0x7f242c0cf410 2024-03-04T09:35:26.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:26.274+0000 7f242b400640 1 -- 172.21.15.6:0/2615984082 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2418030c10 con 0x7f242c0cf410 2024-03-04T09:35:26.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:26.275+0000 7f2432600640 1 -- 172.21.15.6:0/2615984082 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f242c0cf410 msgr2=0x7f242c0cf870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:26.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:26.275+0000 7f2432600640 1 --2- 172.21.15.6:0/2615984082 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f242c0cf410 0x7f242c0cf870 secure :-1 s=READY pgs=1714 cs=0 l=1 rev1=1 crypto rx=0x7f2418009870 tx=0x7f241802b980 comp rx=0 tx=0).stop 2024-03-04T09:35:26.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:26.275+0000 7f2432600640 1 -- 172.21.15.6:0/2615984082 shutdown_connections 2024-03-04T09:35:26.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:26.275+0000 7f2432600640 1 --2- 172.21.15.6:0/2615984082 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f242c0cf410 0x7f242c0cf870 unknown :-1 s=CLOSED pgs=1714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:26.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:26.275+0000 7f2432600640 1 --2- 172.21.15.6:0/2615984082 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f242c15e070 0x7f242c0d2960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:26.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:26.275+0000 7f2432600640 1 -- 172.21.15.6:0/2615984082 >> 172.21.15.6:0/2615984082 conn(0x7f242c08d2a0 msgr2=0x7f242c0d3cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:35:26.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:26.275+0000 7f2432600640 1 -- 172.21.15.6:0/2615984082 shutdown_connections 2024-03-04T09:35:26.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:26.275+0000 7f2432600640 1 -- 172.21.15.6:0/2615984082 wait complete. 2024-03-04T09:35:26.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:26.276+0000 7f2432600640 1 Processor -- start 2024-03-04T09:35:26.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:26.277+0000 7f2432600640 1 -- start start 2024-03-04T09:35:26.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:26.277+0000 7f2432600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f242c0cf410 0x7f242c154f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:26.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:26.278+0000 7f2432600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f242c0d05c0 0x7f242c0cd1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:26.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:26.278+0000 7f2432600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f242c15e070 0x7f242c0cd6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:26.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:26.278+0000 7f2432600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f242c0d4730 con 0x7f242c0d05c0 2024-03-04T09:35:26.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:26.278+0000 7f2432600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f242c0d45b0 con 0x7f242c0cf410 2024-03-04T09:35:26.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:26.278+0000 7f2432600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f242c0d0a50 con 0x7f242c15e070 2024-03-04T09:35:26.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:26.278+0000 7f242be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f242c0d05c0 0x7f242c0cd1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:26.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:26.278+0000 7f242be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f242c0d05c0 0x7f242c0cd1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:35376/0 (socket says 172.21.15.6:35376) 2024-03-04T09:35:26.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:26.278+0000 7f242be00640 1 -- 172.21.15.6:0/1684782256 learned_addr learned my addr 172.21.15.6:0/1684782256 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:35:26.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:26.278+0000 7f2431c00640 1 --2- 172.21.15.6:0/1684782256 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f242c15e070 0x7f242c0cd6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:26.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:26.278+0000 7f2431200640 1 --2- 172.21.15.6:0/1684782256 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f242c0cf410 0x7f242c154f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:26.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:26.278+0000 7f242be00640 1 -- 172.21.15.6:0/1684782256 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f242c15e070 msgr2=0x7f242c0cd6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:26.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:26.278+0000 7f242be00640 1 --2- 172.21.15.6:0/1684782256 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f242c15e070 0x7f242c0cd6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:26.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:26.278+0000 7f242be00640 1 -- 172.21.15.6:0/1684782256 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f242c0cf410 msgr2=0x7f242c154f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:26.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:26.278+0000 7f242be00640 1 --2- 172.21.15.6:0/1684782256 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f242c0cf410 0x7f242c154f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:26.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:26.279+0000 7f242be00640 1 -- 172.21.15.6:0/1684782256 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f242c0cae60 con 0x7f242c0d05c0 2024-03-04T09:35:26.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:26.279+0000 7f242be00640 1 --2- 172.21.15.6:0/1684782256 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f242c0d05c0 0x7f242c0cd1b0 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 crypto rx=0x7f2418000c00 tx=0x7f2418004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:35:26.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:26.279+0000 7f2428c00640 1 -- 172.21.15.6:0/1684782256 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2418048070 con 0x7f242c0d05c0 2024-03-04T09:35:26.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:26.279+0000 7f2428c00640 1 -- 172.21.15.6:0/1684782256 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2418030640 con 0x7f242c0d05c0 2024-03-04T09:35:26.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:26.279+0000 7f2428c00640 1 -- 172.21.15.6:0/1684782256 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2418034040 con 0x7f242c0d05c0 2024-03-04T09:35:26.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:26.279+0000 7f2432600640 1 -- 172.21.15.6:0/1684782256 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f242c160960 con 0x7f242c0d05c0 2024-03-04T09:35:26.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:26.279+0000 7f2432600640 1 -- 172.21.15.6:0/1684782256 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f242c155440 con 0x7f242c0d05c0 2024-03-04T09:35:26.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:26.281+0000 7f2432600640 1 -- 172.21.15.6:0/1684782256 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f242c04f330 con 0x7f242c0d05c0 2024-03-04T09:35:26.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:26.282+0000 7f2428c00640 1 -- 172.21.15.6:0/1684782256 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2418037070 con 0x7f242c0d05c0 2024-03-04T09:35:26.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:26.282+0000 7f2428c00640 1 --2- 172.21.15.6:0/1684782256 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f240c06b4d0 0x7f240c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:26.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:26.283+0000 7f2428c00640 1 -- 172.21.15.6:0/1684782256 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f24180ac280 con 0x7f242c0d05c0 2024-03-04T09:35:26.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:26.283+0000 7f2431200640 1 --2- 172.21.15.6:0/1684782256 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f240c06b4d0 0x7f240c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:26.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:26.284+0000 7f2431200640 1 --2- 172.21.15.6:0/1684782256 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f240c06b4d0 0x7f240c06d990 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f241c007510 tx=0x7f241c0074a0 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:35:26.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:26.288+0000 7f2428c00640 1 -- 172.21.15.6:0/1684782256 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f241807b140 con 0x7f242c0d05c0 2024-03-04T09:35:26.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:26.549+0000 7f2432600640 1 -- 172.21.15.6:0/1684782256 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f242c0d1370 con 0x7f242c0d05c0 2024-03-04T09:35:26.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:26.550+0000 7f2428c00640 1 -- 172.21.15.6:0/1684782256 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3464 (secure 0 0 0) 0x7f241807aa40 con 0x7f242c0d05c0 2024-03-04T09:35:26.547 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:35:26.547 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:35:26.088222+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"laggy_since":"2024-03-04T09:33:54.652601+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":9}]} 2024-03-04T09:35:26.547 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 101 2024-03-04T09:35:26.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:26.552+0000 7f2432600640 1 -- 172.21.15.6:0/1684782256 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f240c06b4d0 msgr2=0x7f240c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:26.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:26.552+0000 7f2432600640 1 --2- 172.21.15.6:0/1684782256 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f240c06b4d0 0x7f240c06d990 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f241c007510 tx=0x7f241c0074a0 comp rx=0 tx=0).stop 2024-03-04T09:35:26.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:26.552+0000 7f2432600640 1 -- 172.21.15.6:0/1684782256 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f242c0d05c0 msgr2=0x7f242c0cd1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:26.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:26.552+0000 7f2432600640 1 --2- 172.21.15.6:0/1684782256 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f242c0d05c0 0x7f242c0cd1b0 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 crypto rx=0x7f2418000c00 tx=0x7f2418004730 comp rx=0 tx=0).stop 2024-03-04T09:35:26.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:26.553+0000 7f2432600640 1 -- 172.21.15.6:0/1684782256 shutdown_connections 2024-03-04T09:35:26.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:26.553+0000 7f2432600640 1 --2- 172.21.15.6:0/1684782256 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f240c06b4d0 0x7f240c06d990 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:26.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:26.553+0000 7f2432600640 1 --2- 172.21.15.6:0/1684782256 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f242c15e070 0x7f242c0cd6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:26.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:26.553+0000 7f2432600640 1 --2- 172.21.15.6:0/1684782256 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f242c0d05c0 0x7f242c0cd1b0 unknown :-1 s=CLOSED pgs=1715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:26.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:26.553+0000 7f2432600640 1 --2- 172.21.15.6:0/1684782256 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f242c0cf410 0x7f242c154f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:26.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:26.553+0000 7f2432600640 1 -- 172.21.15.6:0/1684782256 >> 172.21.15.6:0/1684782256 conn(0x7f242c08d2a0 msgr2=0x7f242c0d2860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:35:26.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:26.553+0000 7f2432600640 1 -- 172.21.15.6:0/1684782256 shutdown_connections 2024-03-04T09:35:26.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:26.553+0000 7f2432600640 1 -- 172.21.15.6:0/1684782256 wait complete. 2024-03-04T09:35:26.773 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:32:57.159681+0000', 'modified': '2024-03-04T09:35:26.088222+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8214, 'mds_1': 6796}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6796': {'gid': 6796, 'name': 'd', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.146:6837/2458070840', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 2458070840}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 2458070840}]}, 'join_fscid': -1, 'laggy_since': '2024-03-04T09:33:54.652601+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8214': {'gid': 8214, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2446627517', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2446627517}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2446627517}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8214, 'qdb_cluster': [8214]} 2024-03-04T09:35:26.773 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6796:up:stopping 2024-03-04T09:35:27.774 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:35:27.889 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-04T09:35:28.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:28.040+0000 7f3133000640 1 Processor -- start 2024-03-04T09:35:28.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:28.040+0000 7f3133000640 1 -- start start 2024-03-04T09:35:28.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:28.041+0000 7f3133000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f312c15e0b0 0x7f312c0cf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:28.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:28.041+0000 7f3133000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f312c0d4a10 0x7f312c0d0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:28.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:28.041+0000 7f3133000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f312c0b3710 con 0x7f312c0d0c80 2024-03-04T09:35:28.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:28.041+0000 7f3133000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f312c0d10f0 con 0x7f312c15e0b0 2024-03-04T09:35:28.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:28.041+0000 7f3133000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f312c0b09e0 con 0x7f312c0d4a10 2024-03-04T09:35:28.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:28.042+0000 7f3131c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f312c15e0b0 0x7f312c0cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:28.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:28.042+0000 7f3131c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f312c15e0b0 0x7f312c0cf310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:35388/0 (socket says 172.21.15.6:35388) 2024-03-04T09:35:28.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:28.042+0000 7f3131c00640 1 -- 172.21.15.6:0/2329183991 learned_addr learned my addr 172.21.15.6:0/2329183991 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:35:28.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:28.042+0000 7f3131200640 1 --2- 172.21.15.6:0/2329183991 >> v2:172.21.15.146:3300/0 conn(0x7f312c0d4a10 0x7f312c0d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:28.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:28.042+0000 7f3131c00640 1 -- 172.21.15.6:0/2329183991 >> v2:172.21.15.146:3300/0 conn(0x7f312c0d4a10 msgr2=0x7f312c0d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:35:28.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:28.042+0000 7f3131c00640 1 --2- 172.21.15.6:0/2329183991 >> v2:172.21.15.146:3300/0 conn(0x7f312c0d4a10 0x7f312c0d0740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:28.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:28.042+0000 7f3131c00640 1 -- 172.21.15.6:0/2329183991 >> v1:172.21.15.6:6789/0 conn(0x7f312c0d0c80 legacy=0x7f312c0d1900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:28.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:28.042+0000 7f3131c00640 1 -- 172.21.15.6:0/2329183991 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f312c0b39f0 con 0x7f312c15e0b0 2024-03-04T09:35:28.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:28.042+0000 7f312be00640 1 -- 172.21.15.6:0/2329183991 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2040944295 0 0) 0x7f312c0b3710 con 0x7f312c0d0c80 2024-03-04T09:35:28.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:28.044+0000 7f3131c00640 1 --2- 172.21.15.6:0/2329183991 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f312c15e0b0 0x7f312c0cf310 secure :-1 s=READY pgs=1717 cs=0 l=1 rev1=1 crypto rx=0x7f312c06b8c0 tx=0x7f312002bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a2c8efa5e0ce701 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:35:28.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:28.044+0000 7f312be00640 1 -- 172.21.15.6:0/2329183991 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f312002cb20 con 0x7f312c15e0b0 2024-03-04T09:35:28.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:28.044+0000 7f312be00640 1 -- 172.21.15.6:0/2329183991 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f312002ccc0 con 0x7f312c15e0b0 2024-03-04T09:35:28.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:28.044+0000 7f312be00640 1 -- 172.21.15.6:0/2329183991 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3120034610 con 0x7f312c15e0b0 2024-03-04T09:35:28.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:28.045+0000 7f3133000640 1 -- 172.21.15.6:0/2329183991 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f312c15e0b0 msgr2=0x7f312c0cf310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:28.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:28.045+0000 7f3133000640 1 --2- 172.21.15.6:0/2329183991 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f312c15e0b0 0x7f312c0cf310 secure :-1 s=READY pgs=1717 cs=0 l=1 rev1=1 crypto rx=0x7f312c06b8c0 tx=0x7f312002bf80 comp rx=0 tx=0).stop 2024-03-04T09:35:28.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:28.045+0000 7f3133000640 1 -- 172.21.15.6:0/2329183991 shutdown_connections 2024-03-04T09:35:28.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:28.045+0000 7f3133000640 1 --2- 172.21.15.6:0/2329183991 >> v2:172.21.15.146:3300/0 conn(0x7f312c0d4a10 0x7f312c0d0740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:28.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:28.045+0000 7f3133000640 1 --2- 172.21.15.6:0/2329183991 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f312c15e0b0 0x7f312c0cf310 unknown :-1 s=CLOSED pgs=1717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:28.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:28.045+0000 7f3133000640 1 -- 172.21.15.6:0/2329183991 >> 172.21.15.6:0/2329183991 conn(0x7f312c0d3d30 msgr2=0x7f312c0d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:35:28.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:28.045+0000 7f3133000640 1 -- 172.21.15.6:0/2329183991 shutdown_connections 2024-03-04T09:35:28.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:28.045+0000 7f3133000640 1 -- 172.21.15.6:0/2329183991 wait complete. 2024-03-04T09:35:28.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:28.047+0000 7f3133000640 1 Processor -- start 2024-03-04T09:35:28.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:28.047+0000 7f3133000640 1 -- start start 2024-03-04T09:35:28.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:28.048+0000 7f3133000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f312c0d0c80 0x7f312c15f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:28.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:28.048+0000 7f3133000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f312c0d4a10 0x7f312c0b6950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:28.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:28.048+0000 7f3133000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f312c15e0b0 0x7f312c153150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:28.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:28.048+0000 7f3133000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f312c15e490 con 0x7f312c0d4a10 2024-03-04T09:35:28.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:28.048+0000 7f3133000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f312c0b3fb0 con 0x7f312c0d0c80 2024-03-04T09:35:28.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:28.048+0000 7f3133000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f312c0cfc60 con 0x7f312c15e0b0 2024-03-04T09:35:28.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:28.048+0000 7f3131200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f312c0d4a10 0x7f312c0b6950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:28.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:28.048+0000 7f3131200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f312c0d4a10 0x7f312c0b6950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:35392/0 (socket says 172.21.15.6:35392) 2024-03-04T09:35:28.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:28.048+0000 7f3131200640 1 -- 172.21.15.6:0/3659854187 learned_addr learned my addr 172.21.15.6:0/3659854187 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:35:28.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:28.048+0000 7f3131c00640 1 --2- 172.21.15.6:0/3659854187 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f312c0d0c80 0x7f312c15f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:28.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:28.048+0000 7f3131200640 1 -- 172.21.15.6:0/3659854187 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f312c15e0b0 msgr2=0x7f312c153150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:28.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:28.048+0000 7f3131200640 1 --2- 172.21.15.6:0/3659854187 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f312c15e0b0 0x7f312c153150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:28.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:28.048+0000 7f3131200640 1 -- 172.21.15.6:0/3659854187 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f312c0d0c80 msgr2=0x7f312c15f2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:28.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:28.048+0000 7f3131200640 1 --2- 172.21.15.6:0/3659854187 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f312c0d0c80 0x7f312c15f2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:28.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:28.048+0000 7f3131200640 1 -- 172.21.15.6:0/3659854187 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f312c0b70a0 con 0x7f312c0d4a10 2024-03-04T09:35:28.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:28.049+0000 7f3131200640 1 --2- 172.21.15.6:0/3659854187 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f312c0d4a10 0x7f312c0b6950 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7f311800d7d0 tx=0x7f311800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:35:28.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:28.049+0000 7f3129600640 1 -- 172.21.15.6:0/3659854187 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f311800b6c0 con 0x7f312c0d4a10 2024-03-04T09:35:28.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:28.049+0000 7f3129600640 1 -- 172.21.15.6:0/3659854187 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3118019070 con 0x7f312c0d4a10 2024-03-04T09:35:28.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:28.049+0000 7f3129600640 1 -- 172.21.15.6:0/3659854187 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f311800ba80 con 0x7f312c0d4a10 2024-03-04T09:35:28.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:28.049+0000 7f3133000640 1 -- 172.21.15.6:0/3659854187 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f312c0b7380 con 0x7f312c0d4a10 2024-03-04T09:35:28.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:28.049+0000 7f3133000640 1 -- 172.21.15.6:0/3659854187 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f312c0b7940 con 0x7f312c0d4a10 2024-03-04T09:35:28.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:28.051+0000 7f3133000640 1 -- 172.21.15.6:0/3659854187 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f312c04f330 con 0x7f312c0d4a10 2024-03-04T09:35:28.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:28.051+0000 7f3129600640 1 -- 172.21.15.6:0/3659854187 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f311800bc20 con 0x7f312c0d4a10 2024-03-04T09:35:28.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:28.052+0000 7f3129600640 1 --2- 172.21.15.6:0/3659854187 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f310c06b500 0x7f310c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:28.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:28.052+0000 7f3129600640 1 -- 172.21.15.6:0/3659854187 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f311808a6c0 con 0x7f312c0d4a10 2024-03-04T09:35:28.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:28.053+0000 7f3131c00640 1 --2- 172.21.15.6:0/3659854187 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f310c06b500 0x7f310c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:28.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:28.054+0000 7f3131c00640 1 --2- 172.21.15.6:0/3659854187 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f310c06b500 0x7f310c06d9c0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f3120035000 tx=0x7f3120030000 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:35:28.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:28.057+0000 7f3129600640 1 -- 172.21.15.6:0/3659854187 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f31180594f0 con 0x7f312c0d4a10 2024-03-04T09:35:28.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:28.298+0000 7f3133000640 1 -- 172.21.15.6:0/3659854187 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f312c0b9b30 con 0x7f312c0d4a10 2024-03-04T09:35:28.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:28.299+0000 7f3129600640 1 -- 172.21.15.6:0/3659854187 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3464 (secure 0 0 0) 0x7f3118058df0 con 0x7f312c0d4a10 2024-03-04T09:35:28.296 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:35:28.296 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:35:26.088222+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"laggy_since":"2024-03-04T09:33:54.652601+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":9}]} 2024-03-04T09:35:28.296 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 101 2024-03-04T09:35:28.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:28.301+0000 7f3133000640 1 -- 172.21.15.6:0/3659854187 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f310c06b500 msgr2=0x7f310c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:28.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:28.301+0000 7f3133000640 1 --2- 172.21.15.6:0/3659854187 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f310c06b500 0x7f310c06d9c0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f3120035000 tx=0x7f3120030000 comp rx=0 tx=0).stop 2024-03-04T09:35:28.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:28.301+0000 7f3133000640 1 -- 172.21.15.6:0/3659854187 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f312c0d4a10 msgr2=0x7f312c0b6950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:28.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:28.301+0000 7f3133000640 1 --2- 172.21.15.6:0/3659854187 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f312c0d4a10 0x7f312c0b6950 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7f311800d7d0 tx=0x7f311800dc90 comp rx=0 tx=0).stop 2024-03-04T09:35:28.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:28.301+0000 7f3133000640 1 -- 172.21.15.6:0/3659854187 shutdown_connections 2024-03-04T09:35:28.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:28.301+0000 7f3133000640 1 --2- 172.21.15.6:0/3659854187 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f310c06b500 0x7f310c06d9c0 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:28.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:28.301+0000 7f3133000640 1 --2- 172.21.15.6:0/3659854187 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f312c15e0b0 0x7f312c153150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:28.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:28.301+0000 7f3133000640 1 --2- 172.21.15.6:0/3659854187 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f312c0d4a10 0x7f312c0b6950 unknown :-1 s=CLOSED pgs=1718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:28.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:28.301+0000 7f3133000640 1 --2- 172.21.15.6:0/3659854187 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f312c0d0c80 0x7f312c15f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:28.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:28.301+0000 7f3133000640 1 -- 172.21.15.6:0/3659854187 >> 172.21.15.6:0/3659854187 conn(0x7f312c0d3d30 msgr2=0x7f312c15d2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:35:28.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:28.301+0000 7f3133000640 1 -- 172.21.15.6:0/3659854187 shutdown_connections 2024-03-04T09:35:28.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:28.301+0000 7f3133000640 1 -- 172.21.15.6:0/3659854187 wait complete. 2024-03-04T09:35:28.517 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:32:57.159681+0000', 'modified': '2024-03-04T09:35:26.088222+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8214, 'mds_1': 6796}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6796': {'gid': 6796, 'name': 'd', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.146:6837/2458070840', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 2458070840}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 2458070840}]}, 'join_fscid': -1, 'laggy_since': '2024-03-04T09:33:54.652601+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8214': {'gid': 8214, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2446627517', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2446627517}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2446627517}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8214, 'qdb_cluster': [8214]} 2024-03-04T09:35:28.518 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6796:up:stopping 2024-03-04T09:35:29.519 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:35:29.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:29.747+0000 7fbe24600640 1 Processor -- start 2024-03-04T09:35:29.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:29.747+0000 7fbe24600640 1 -- start start 2024-03-04T09:35:29.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:29.748+0000 7fbe24600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fbe1c0af1a0 0x7fbe1c15c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:29.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:29.748+0000 7fbe24600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbe1c0af580 0x7fbe1c0b4c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:29.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:29.749+0000 7fbe24600640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fbe1c0cdc20 0x7fbe1c0b2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:29.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:29.749+0000 7fbe24600640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe1c15cc40 con 0x7fbe1c0af1a0 2024-03-04T09:35:29.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:29.749+0000 7fbe24600640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe1c15cf20 con 0x7fbe1c0cdc20 2024-03-04T09:35:29.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:29.749+0000 7fbe24600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe1c15d200 con 0x7fbe1c0af580 2024-03-04T09:35:29.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:29.749+0000 7fbe21c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fbe1c0af1a0 0x7fbe1c15c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:29.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:29.749+0000 7fbe21c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fbe1c0af1a0 0x7fbe1c15c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:39352/0 (socket says 172.21.15.6:39352) 2024-03-04T09:35:29.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:29.749+0000 7fbe21c00640 1 -- 172.21.15.6:0/3527659264 learned_addr learned my addr 172.21.15.6:0/3527659264 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:35:29.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:29.749+0000 7fbe22600640 1 --2- 172.21.15.6:0/3527659264 >> v2:172.21.15.146:3300/0 conn(0x7fbe1c0cdc20 0x7fbe1c0b2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:29.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:29.749+0000 7fbe21200640 1 --2- 172.21.15.6:0/3527659264 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbe1c0af580 0x7fbe1c0b4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:29.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:29.750+0000 7fbe21c00640 1 -- 172.21.15.6:0/3527659264 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbe1c0af580 msgr2=0x7fbe1c0b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:35:29.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:29.750+0000 7fbe21c00640 1 --2- 172.21.15.6:0/3527659264 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbe1c0af580 0x7fbe1c0b4c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:29.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:29.750+0000 7fbe21c00640 1 -- 172.21.15.6:0/3527659264 >> v2:172.21.15.146:3300/0 conn(0x7fbe1c0cdc20 msgr2=0x7fbe1c0b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:35:29.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:29.750+0000 7fbe21c00640 1 --2- 172.21.15.6:0/3527659264 >> v2:172.21.15.146:3300/0 conn(0x7fbe1c0cdc20 0x7fbe1c0b2dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:29.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:29.750+0000 7fbe21c00640 1 -- 172.21.15.6:0/3527659264 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe1c15d4e0 con 0x7fbe1c0af1a0 2024-03-04T09:35:29.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:29.750+0000 7fbe21200640 1 --2- 172.21.15.6:0/3527659264 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbe1c0af580 0x7fbe1c0b4c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:35:29.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:29.751+0000 7fbe21c00640 1 --2- 172.21.15.6:0/3527659264 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbe1c0af1a0 0x7fbe1c15c600 secure :-1 s=READY pgs=1719 cs=0 l=1 rev1=1 crypto rx=0x7fbe1c0cda30 tx=0x7fbe1002be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c2b160ff0c0583c server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:35:29.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:29.751+0000 7fbe1be00640 1 -- 172.21.15.6:0/3527659264 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe1002c9f0 con 0x7fbe1c0af1a0 2024-03-04T09:35:29.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:29.751+0000 7fbe1be00640 1 -- 172.21.15.6:0/3527659264 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe1002cb90 con 0x7fbe1c0af1a0 2024-03-04T09:35:29.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:29.751+0000 7fbe1be00640 1 -- 172.21.15.6:0/3527659264 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe100343f0 con 0x7fbe1c0af1a0 2024-03-04T09:35:29.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:29.752+0000 7fbe24600640 1 -- 172.21.15.6:0/3527659264 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbe1c0af1a0 msgr2=0x7fbe1c15c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:29.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:29.752+0000 7fbe24600640 1 --2- 172.21.15.6:0/3527659264 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbe1c0af1a0 0x7fbe1c15c600 secure :-1 s=READY pgs=1719 cs=0 l=1 rev1=1 crypto rx=0x7fbe1c0cda30 tx=0x7fbe1002be50 comp rx=0 tx=0).stop 2024-03-04T09:35:29.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:29.752+0000 7fbe24600640 1 -- 172.21.15.6:0/3527659264 shutdown_connections 2024-03-04T09:35:29.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:29.752+0000 7fbe24600640 1 --2- 172.21.15.6:0/3527659264 >> v2:172.21.15.146:3300/0 conn(0x7fbe1c0cdc20 0x7fbe1c0b2dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:29.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:29.752+0000 7fbe24600640 1 --2- 172.21.15.6:0/3527659264 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbe1c0af580 0x7fbe1c0b4c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:29.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:29.752+0000 7fbe24600640 1 --2- 172.21.15.6:0/3527659264 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbe1c0af1a0 0x7fbe1c15c600 unknown :-1 s=CLOSED pgs=1719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:29.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:29.752+0000 7fbe24600640 1 -- 172.21.15.6:0/3527659264 >> 172.21.15.6:0/3527659264 conn(0x7fbe1c06afb0 msgr2=0x7fbe1c06b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:35:29.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:29.752+0000 7fbe24600640 1 -- 172.21.15.6:0/3527659264 shutdown_connections 2024-03-04T09:35:29.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:29.752+0000 7fbe24600640 1 -- 172.21.15.6:0/3527659264 wait complete. 2024-03-04T09:35:29.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:29.754+0000 7fbe24600640 1 Processor -- start 2024-03-04T09:35:29.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:29.754+0000 7fbe24600640 1 -- start start 2024-03-04T09:35:29.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:29.755+0000 7fbe24600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbe1c0af1a0 0x7fbe1c06e730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:29.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:29.755+0000 7fbe24600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbe1c0af580 0x7fbe1c09cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:29.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:29.755+0000 7fbe24600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbe1c0cdc20 0x7fbe1c09d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:29.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:29.755+0000 7fbe24600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe1c0b3490 con 0x7fbe1c0cdc20 2024-03-04T09:35:29.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:29.755+0000 7fbe24600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe1c0b3310 con 0x7fbe1c0af580 2024-03-04T09:35:29.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:29.755+0000 7fbe24600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe1c0b3610 con 0x7fbe1c0af1a0 2024-03-04T09:35:29.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:29.755+0000 7fbe22600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbe1c0cdc20 0x7fbe1c09d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:29.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:29.755+0000 7fbe22600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbe1c0cdc20 0x7fbe1c09d260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:39368/0 (socket says 172.21.15.6:39368) 2024-03-04T09:35:29.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:29.755+0000 7fbe22600640 1 -- 172.21.15.6:0/3702629474 learned_addr learned my addr 172.21.15.6:0/3702629474 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:35:29.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:29.756+0000 7fbe21c00640 1 --2- 172.21.15.6:0/3702629474 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbe1c0af1a0 0x7fbe1c06e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:29.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:29.756+0000 7fbe21200640 1 --2- 172.21.15.6:0/3702629474 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbe1c0af580 0x7fbe1c09cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:29.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:29.756+0000 7fbe22600640 1 -- 172.21.15.6:0/3702629474 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbe1c0af1a0 msgr2=0x7fbe1c06e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:29.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:29.756+0000 7fbe22600640 1 --2- 172.21.15.6:0/3702629474 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbe1c0af1a0 0x7fbe1c06e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:29.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:29.756+0000 7fbe22600640 1 -- 172.21.15.6:0/3702629474 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbe1c0af580 msgr2=0x7fbe1c09cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:29.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:29.756+0000 7fbe22600640 1 --2- 172.21.15.6:0/3702629474 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbe1c0af580 0x7fbe1c09cd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:29.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:29.756+0000 7fbe22600640 1 -- 172.21.15.6:0/3702629474 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe1c09d890 con 0x7fbe1c0cdc20 2024-03-04T09:35:29.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:29.756+0000 7fbe21200640 1 --2- 172.21.15.6:0/3702629474 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbe1c0af580 0x7fbe1c09cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:35:29.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:29.756+0000 7fbe21c00640 1 --2- 172.21.15.6:0/3702629474 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbe1c0af1a0 0x7fbe1c06e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T09:35:29.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:29.756+0000 7fbe22600640 1 --2- 172.21.15.6:0/3702629474 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbe1c0cdc20 0x7fbe1c09d260 secure :-1 s=READY pgs=1720 cs=0 l=1 rev1=1 crypto rx=0x7fbe1400d5d0 tx=0x7fbe1400da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:35:29.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:29.757+0000 7fbe19600640 1 -- 172.21.15.6:0/3702629474 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe140125f0 con 0x7fbe1c0cdc20 2024-03-04T09:35:29.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:29.757+0000 7fbe19600640 1 -- 172.21.15.6:0/3702629474 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe14019070 con 0x7fbe1c0cdc20 2024-03-04T09:35:29.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:29.757+0000 7fbe24600640 1 -- 172.21.15.6:0/3702629474 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe1c1f3c70 con 0x7fbe1c0cdc20 2024-03-04T09:35:29.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:29.757+0000 7fbe19600640 1 -- 172.21.15.6:0/3702629474 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe140129f0 con 0x7fbe1c0cdc20 2024-03-04T09:35:29.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:29.757+0000 7fbe24600640 1 -- 172.21.15.6:0/3702629474 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe1c1f4230 con 0x7fbe1c0cdc20 2024-03-04T09:35:29.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:29.758+0000 7fbe24600640 1 -- 172.21.15.6:0/3702629474 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe1c04f3a0 con 0x7fbe1c0cdc20 2024-03-04T09:35:29.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:29.759+0000 7fbe19600640 1 -- 172.21.15.6:0/3702629474 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbe14012b90 con 0x7fbe1c0cdc20 2024-03-04T09:35:29.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:29.760+0000 7fbe19600640 1 --2- 172.21.15.6:0/3702629474 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fbdfc06b4d0 0x7fbdfc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:29.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:29.760+0000 7fbe19600640 1 -- 172.21.15.6:0/3702629474 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fbe1408a6a0 con 0x7fbe1c0cdc20 2024-03-04T09:35:29.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:29.761+0000 7fbe21c00640 1 --2- 172.21.15.6:0/3702629474 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fbdfc06b4d0 0x7fbdfc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:29.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:29.762+0000 7fbe21c00640 1 --2- 172.21.15.6:0/3702629474 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fbdfc06b4d0 0x7fbdfc06d990 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fbe10035000 tx=0x7fbe10030000 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:35:29.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:29.765+0000 7fbe19600640 1 -- 172.21.15.6:0/3702629474 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbe14059450 con 0x7fbe1c0cdc20 2024-03-04T09:35:29.891 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-04T09:35:30.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:30.004+0000 7fbe24600640 1 -- 172.21.15.6:0/3702629474 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbe1c0ae940 con 0x7fbe1c0cdc20 2024-03-04T09:35:30.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:30.005+0000 7fbe19600640 1 -- 172.21.15.6:0/3702629474 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3464 (secure 0 0 0) 0x7fbe14058d50 con 0x7fbe1c0cdc20 2024-03-04T09:35:30.002 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:35:30.002 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:35:26.088222+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"laggy_since":"2024-03-04T09:33:54.652601+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":9}]} 2024-03-04T09:35:30.002 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 101 2024-03-04T09:35:30.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:30.008+0000 7fbe24600640 1 -- 172.21.15.6:0/3702629474 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fbdfc06b4d0 msgr2=0x7fbdfc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:30.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:30.008+0000 7fbe24600640 1 --2- 172.21.15.6:0/3702629474 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fbdfc06b4d0 0x7fbdfc06d990 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fbe10035000 tx=0x7fbe10030000 comp rx=0 tx=0).stop 2024-03-04T09:35:30.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:30.008+0000 7fbe24600640 1 -- 172.21.15.6:0/3702629474 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbe1c0cdc20 msgr2=0x7fbe1c09d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:30.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:30.008+0000 7fbe24600640 1 --2- 172.21.15.6:0/3702629474 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbe1c0cdc20 0x7fbe1c09d260 secure :-1 s=READY pgs=1720 cs=0 l=1 rev1=1 crypto rx=0x7fbe1400d5d0 tx=0x7fbe1400da90 comp rx=0 tx=0).stop 2024-03-04T09:35:30.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:30.008+0000 7fbe24600640 1 -- 172.21.15.6:0/3702629474 shutdown_connections 2024-03-04T09:35:30.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:30.008+0000 7fbe24600640 1 --2- 172.21.15.6:0/3702629474 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbe1c0cdc20 0x7fbe1c09d260 unknown :-1 s=CLOSED pgs=1720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:30.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:30.008+0000 7fbe24600640 1 --2- 172.21.15.6:0/3702629474 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbe1c0af580 0x7fbe1c09cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:30.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:30.008+0000 7fbe24600640 1 --2- 172.21.15.6:0/3702629474 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fbdfc06b4d0 0x7fbdfc06d990 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:30.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:30.008+0000 7fbe24600640 1 --2- 172.21.15.6:0/3702629474 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbe1c0af1a0 0x7fbe1c06e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:30.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:30.008+0000 7fbe24600640 1 -- 172.21.15.6:0/3702629474 >> 172.21.15.6:0/3702629474 conn(0x7fbe1c06afb0 msgr2=0x7fbe1c06b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:35:30.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:30.008+0000 7fbe24600640 1 -- 172.21.15.6:0/3702629474 shutdown_connections 2024-03-04T09:35:30.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:30.008+0000 7fbe24600640 1 -- 172.21.15.6:0/3702629474 wait complete. 2024-03-04T09:35:30.225 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:32:57.159681+0000', 'modified': '2024-03-04T09:35:26.088222+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8214, 'mds_1': 6796}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6796': {'gid': 6796, 'name': 'd', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.146:6837/2458070840', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 2458070840}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 2458070840}]}, 'join_fscid': -1, 'laggy_since': '2024-03-04T09:33:54.652601+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8214': {'gid': 8214, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2446627517', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2446627517}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2446627517}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8214, 'qdb_cluster': [8214]} 2024-03-04T09:35:30.225 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6796:up:stopping 2024-03-04T09:35:31.226 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:35:31.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:31.450+0000 7fde08a00640 1 Processor -- start 2024-03-04T09:35:31.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:31.451+0000 7fde08a00640 1 -- start start 2024-03-04T09:35:31.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:31.452+0000 7fde08a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde0406e710 0x7fde0406afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:31.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:31.452+0000 7fde08a00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fde040ad880 0x7fde0406b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:31.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:31.452+0000 7fde08a00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde0406fb90 con 0x7fde04147870 2024-03-04T09:35:31.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:31.452+0000 7fde08a00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde04147c50 con 0x7fde040ad880 2024-03-04T09:35:31.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:31.452+0000 7fde08a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde040b3b60 con 0x7fde0406e710 2024-03-04T09:35:31.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:31.453+0000 7fddf9e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde0406e710 0x7fde0406afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:31.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:31.453+0000 7fde03200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fde040ad880 0x7fde0406b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:31.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:31.453+0000 7fde03200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fde040ad880 0x7fde0406b4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.6:33254/0 (socket says 172.21.15.6:33254) 2024-03-04T09:35:31.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:31.453+0000 7fde03200640 1 -- 172.21.15.6:0/3762796111 learned_addr learned my addr 172.21.15.6:0/3762796111 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:35:31.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:31.454+0000 7fde03200640 1 -- 172.21.15.6:0/3762796111 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde0406e710 msgr2=0x7fde0406afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:35:31.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:31.454+0000 7fde03200640 1 --2- 172.21.15.6:0/3762796111 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde0406e710 0x7fde0406afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:31.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:31.454+0000 7fde03200640 1 -- 172.21.15.6:0/3762796111 >> v1:172.21.15.146:6789/0 conn(0x7fde04147870 legacy=0x7fde040b5860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:31.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:31.454+0000 7fde03200640 1 -- 172.21.15.6:0/3762796111 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde0406ebf0 con 0x7fde040ad880 2024-03-04T09:35:31.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:31.455+0000 7fde03200640 1 --2- 172.21.15.6:0/3762796111 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fde040ad880 0x7fde0406b4f0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fddfc009e50 tx=0x7fddfc02bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=95684c4c8f339869 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:35:31.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:31.456+0000 7fde01e00640 1 -- 172.21.15.6:0/3762796111 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddfc02cb00 con 0x7fde040ad880 2024-03-04T09:35:31.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:31.456+0000 7fde01e00640 1 -- 172.21.15.6:0/3762796111 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddfc02cca0 con 0x7fde040ad880 2024-03-04T09:35:31.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:31.456+0000 7fde01e00640 1 -- 172.21.15.6:0/3762796111 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddfc034610 con 0x7fde040ad880 2024-03-04T09:35:31.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:31.456+0000 7fde08a00640 1 -- 172.21.15.6:0/3762796111 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fde040ad880 msgr2=0x7fde0406b4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:31.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:31.456+0000 7fde08a00640 1 --2- 172.21.15.6:0/3762796111 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fde040ad880 0x7fde0406b4f0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fddfc009e50 tx=0x7fddfc02bf60 comp rx=0 tx=0).stop 2024-03-04T09:35:31.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:31.457+0000 7fde08a00640 1 -- 172.21.15.6:0/3762796111 shutdown_connections 2024-03-04T09:35:31.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:31.457+0000 7fde08a00640 1 --2- 172.21.15.6:0/3762796111 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fde040ad880 0x7fde0406b4f0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:31.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:31.457+0000 7fde08a00640 1 --2- 172.21.15.6:0/3762796111 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde0406e710 0x7fde0406afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:31.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:31.457+0000 7fde08a00640 1 -- 172.21.15.6:0/3762796111 >> 172.21.15.6:0/3762796111 conn(0x7fde0406c0c0 msgr2=0x7fde0406c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:35:31.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:31.457+0000 7fde08a00640 1 -- 172.21.15.6:0/3762796111 shutdown_connections 2024-03-04T09:35:31.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:31.457+0000 7fde08a00640 1 -- 172.21.15.6:0/3762796111 wait complete. 2024-03-04T09:35:31.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:31.459+0000 7fde08a00640 1 Processor -- start 2024-03-04T09:35:31.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:31.459+0000 7fde08a00640 1 -- start start 2024-03-04T09:35:31.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:31.460+0000 7fde08a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde0406e710 0x7fde0415d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:31.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:31.460+0000 7fde08a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fde040ad880 0x7fde041e9710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:31.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:31.460+0000 7fde08a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fde04147870 0x7fde04144760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:31.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:31.460+0000 7fde08a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde040b4370 con 0x7fde040ad880 2024-03-04T09:35:31.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:31.460+0000 7fde08a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde040b4090 con 0x7fde04147870 2024-03-04T09:35:31.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:31.460+0000 7fde08a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde040d0c60 con 0x7fde0406e710 2024-03-04T09:35:31.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:31.460+0000 7fddf9e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fde040ad880 0x7fde041e9710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:31.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:31.460+0000 7fddf9e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fde040ad880 0x7fde041e9710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:39370/0 (socket says 172.21.15.6:39370) 2024-03-04T09:35:31.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:31.460+0000 7fddf9e00640 1 -- 172.21.15.6:0/1635374155 learned_addr learned my addr 172.21.15.6:0/1635374155 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:35:31.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:31.460+0000 7fde02800640 1 --2- 172.21.15.6:0/1635374155 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde0406e710 0x7fde0415d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:31.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:31.461+0000 7fde03200640 1 --2- 172.21.15.6:0/1635374155 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fde04147870 0x7fde04144760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:31.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:31.461+0000 7fddf9e00640 1 -- 172.21.15.6:0/1635374155 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde0406e710 msgr2=0x7fde0415d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:31.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:31.461+0000 7fddf9e00640 1 --2- 172.21.15.6:0/1635374155 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde0406e710 0x7fde0415d920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:31.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:31.461+0000 7fddf9e00640 1 -- 172.21.15.6:0/1635374155 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fde04147870 msgr2=0x7fde04144760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:31.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:31.461+0000 7fddf9e00640 1 --2- 172.21.15.6:0/1635374155 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fde04147870 0x7fde04144760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:31.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:31.461+0000 7fddf9e00640 1 -- 172.21.15.6:0/1635374155 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde041e7b80 con 0x7fde040ad880 2024-03-04T09:35:31.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:31.461+0000 7fddf9e00640 1 --2- 172.21.15.6:0/1635374155 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fde040ad880 0x7fde041e9710 secure :-1 s=READY pgs=1721 cs=0 l=1 rev1=1 crypto rx=0x7fddec00e9b0 tx=0x7fddec00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:35:31.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:31.461+0000 7fddfb400640 1 -- 172.21.15.6:0/1635374155 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddec017070 con 0x7fde040ad880 2024-03-04T09:35:31.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:31.461+0000 7fddfb400640 1 -- 172.21.15.6:0/1635374155 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddec00c9a0 con 0x7fde040ad880 2024-03-04T09:35:31.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:31.461+0000 7fde08a00640 1 -- 172.21.15.6:0/1635374155 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde041e7e60 con 0x7fde040ad880 2024-03-04T09:35:31.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:31.462+0000 7fddfb400640 1 -- 172.21.15.6:0/1635374155 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddec00cd90 con 0x7fde040ad880 2024-03-04T09:35:31.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:31.462+0000 7fde08a00640 1 -- 172.21.15.6:0/1635374155 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde041e8420 con 0x7fde040ad880 2024-03-04T09:35:31.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:31.463+0000 7fde08a00640 1 -- 172.21.15.6:0/1635374155 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde0404f330 con 0x7fde040ad880 2024-03-04T09:35:31.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:31.464+0000 7fddfb400640 1 -- 172.21.15.6:0/1635374155 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fddec01a070 con 0x7fde040ad880 2024-03-04T09:35:31.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:31.465+0000 7fddfb400640 1 --2- 172.21.15.6:0/1635374155 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fdde006b500 0x7fdde006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:31.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:31.465+0000 7fddfb400640 1 -- 172.21.15.6:0/1635374155 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fddec08a1a0 con 0x7fde040ad880 2024-03-04T09:35:31.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:31.466+0000 7fde02800640 1 --2- 172.21.15.6:0/1635374155 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fdde006b500 0x7fdde006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:31.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:31.467+0000 7fde02800640 1 --2- 172.21.15.6:0/1635374155 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fdde006b500 0x7fdde006d9c0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7fddf4007fd0 tx=0x7fddf4009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:35:31.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:31.470+0000 7fddfb400640 1 -- 172.21.15.6:0/1635374155 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fddec059070 con 0x7fde040ad880 2024-03-04T09:35:31.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:31.713+0000 7fde08a00640 1 -- 172.21.15.6:0/1635374155 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fde040724f0 con 0x7fde040ad880 2024-03-04T09:35:31.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:31.714+0000 7fddfb400640 1 -- 172.21.15.6:0/1635374155 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3464 (secure 0 0 0) 0x7fddec058970 con 0x7fde040ad880 2024-03-04T09:35:31.711 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:35:31.711 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:35:26.088222+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"laggy_since":"2024-03-04T09:33:54.652601+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":9}]} 2024-03-04T09:35:31.712 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 101 2024-03-04T09:35:31.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:31.717+0000 7fde08a00640 1 -- 172.21.15.6:0/1635374155 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fdde006b500 msgr2=0x7fdde006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:31.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:31.717+0000 7fde08a00640 1 --2- 172.21.15.6:0/1635374155 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fdde006b500 0x7fdde006d9c0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7fddf4007fd0 tx=0x7fddf4009040 comp rx=0 tx=0).stop 2024-03-04T09:35:31.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:31.717+0000 7fde08a00640 1 -- 172.21.15.6:0/1635374155 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fde040ad880 msgr2=0x7fde041e9710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:31.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:31.717+0000 7fde08a00640 1 --2- 172.21.15.6:0/1635374155 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fde040ad880 0x7fde041e9710 secure :-1 s=READY pgs=1721 cs=0 l=1 rev1=1 crypto rx=0x7fddec00e9b0 tx=0x7fddec00ee70 comp rx=0 tx=0).stop 2024-03-04T09:35:31.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:31.717+0000 7fde08a00640 1 -- 172.21.15.6:0/1635374155 shutdown_connections 2024-03-04T09:35:31.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:31.717+0000 7fde08a00640 1 --2- 172.21.15.6:0/1635374155 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fde04147870 0x7fde04144760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:31.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:31.717+0000 7fde08a00640 1 --2- 172.21.15.6:0/1635374155 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fde040ad880 0x7fde041e9710 unknown :-1 s=CLOSED pgs=1721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:31.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:31.717+0000 7fde08a00640 1 --2- 172.21.15.6:0/1635374155 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fdde006b500 0x7fdde006d9c0 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:31.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:31.717+0000 7fde08a00640 1 --2- 172.21.15.6:0/1635374155 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde0406e710 0x7fde0415d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:31.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:31.717+0000 7fde08a00640 1 -- 172.21.15.6:0/1635374155 >> 172.21.15.6:0/1635374155 conn(0x7fde0406c0c0 msgr2=0x7fde040d1120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:35:31.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:31.717+0000 7fde08a00640 1 -- 172.21.15.6:0/1635374155 shutdown_connections 2024-03-04T09:35:31.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:31.717+0000 7fde08a00640 1 -- 172.21.15.6:0/1635374155 wait complete. 2024-03-04T09:35:31.892 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-04T09:35:31.942 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:32:57.159681+0000', 'modified': '2024-03-04T09:35:26.088222+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8214, 'mds_1': 6796}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6796': {'gid': 6796, 'name': 'd', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.146:6837/2458070840', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 2458070840}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 2458070840}]}, 'join_fscid': -1, 'laggy_since': '2024-03-04T09:33:54.652601+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8214': {'gid': 8214, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2446627517', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2446627517}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2446627517}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8214, 'qdb_cluster': [8214]} 2024-03-04T09:35:31.942 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6796:up:stopping 2024-03-04T09:35:32.943 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:35:33.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:33.182+0000 7f03c5c00640 1 Processor -- start 2024-03-04T09:35:33.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:33.182+0000 7f03c5c00640 1 -- start start 2024-03-04T09:35:33.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:33.184+0000 7f03c5c00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f03c0144c00 0x7f03c0145280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:33.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:33.184+0000 7f03c5c00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03c00d0910 con 0x7f03c006b6c0 2024-03-04T09:35:33.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:33.184+0000 7f03c5c00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03c00d0bf0 con 0x7f03c01457c0 2024-03-04T09:35:33.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:33.184+0000 7f03c5c00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03c00d0ed0 con 0x7f03c0144c00 2024-03-04T09:35:33.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:33.184+0000 7f03c5200640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f03c006b6c0 0x7f03c00cf130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:54496/0 (socket says 172.21.15.6:54496) 2024-03-04T09:35:33.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:33.184+0000 7f03c5200640 1 -- 172.21.15.6:0/3509260968 learned_addr learned my addr 172.21.15.6:0/3509260968 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:35:33.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:33.184+0000 7f03bfe00640 1 --2- 172.21.15.6:0/3509260968 >> v2:172.21.15.146:3300/0 conn(0x7f03c0144c00 0x7f03c0145280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:33.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:33.185+0000 7f03bea00640 1 -- 172.21.15.6:0/3509260968 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3196577827 0 0) 0x7f03c00d0910 con 0x7f03c006b6c0 2024-03-04T09:35:33.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:33.185+0000 7f03bea00640 1 -- 172.21.15.6:0/3509260968 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03a4003520 con 0x7f03c006b6c0 2024-03-04T09:35:33.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:33.185+0000 7f03bea00640 1 -- 172.21.15.6:0/3509260968 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2710908659 0 0) 0x7f03a4003520 con 0x7f03c006b6c0 2024-03-04T09:35:33.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:33.185+0000 7f03bea00640 1 -- 172.21.15.6:0/3509260968 >> v2:172.21.15.146:3300/0 conn(0x7f03c0144c00 msgr2=0x7f03c0145280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:35:33.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:33.185+0000 7f03bea00640 1 --2- 172.21.15.6:0/3509260968 >> v2:172.21.15.146:3300/0 conn(0x7f03c0144c00 0x7f03c0145280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:33.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:33.185+0000 7f03bea00640 1 -- 172.21.15.6:0/3509260968 >> v1:172.21.15.146:6789/0 conn(0x7f03c01457c0 legacy=0x7f03c006afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:33.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:33.185+0000 7f03bea00640 1 -- 172.21.15.6:0/3509260968 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03c00d11b0 con 0x7f03c006b6c0 2024-03-04T09:35:33.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:33.185+0000 7f03bea00640 1 -- 172.21.15.6:0/3509260968 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7f03b80019f0 con 0x7f03c006b6c0 2024-03-04T09:35:33.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:33.185+0000 7f03bea00640 1 -- 172.21.15.6:0/3509260968 >> v1:172.21.15.6:6789/0 conn(0x7f03c006b6c0 legacy=0x7f03c00cf130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:33.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:33.185+0000 7f03bea00640 1 --2- 172.21.15.6:0/3509260968 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f03a40039b0 0x7f03a4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:33.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:33.185+0000 7f03bea00640 1 --2- 172.21.15.6:0/3509260968 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f03a4004800 0x7f03a4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:33.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:33.185+0000 7f03bea00640 1 --2- 172.21.15.6:0/3509260968 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f03a40051c0 0x7f03a400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:33.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:33.185+0000 7f03bea00640 1 -- 172.21.15.6:0/3509260968 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03c00d0ed0 con 0x7f03a40051c0 2024-03-04T09:35:33.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:33.185+0000 7f03bea00640 1 -- 172.21.15.6:0/3509260968 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03a400c060 con 0x7f03a40039b0 2024-03-04T09:35:33.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:33.185+0000 7f03bea00640 1 -- 172.21.15.6:0/3509260968 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03a400c260 con 0x7f03a4004800 2024-03-04T09:35:33.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:33.186+0000 7f03c5200640 1 --2- 172.21.15.6:0/3509260968 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f03a40051c0 0x7f03a400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:33.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:33.186+0000 7f03bfe00640 1 --2- 172.21.15.6:0/3509260968 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f03a40039b0 0x7f03a4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:33.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:33.186+0000 7f03bf400640 1 --2- 172.21.15.6:0/3509260968 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f03a4004800 0x7f03a4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:33.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:33.186+0000 7f03c5200640 1 -- 172.21.15.6:0/3509260968 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f03a4004800 msgr2=0x7f03a4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:35:33.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:33.186+0000 7f03c5200640 1 --2- 172.21.15.6:0/3509260968 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f03a4004800 0x7f03a4004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:33.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:33.186+0000 7f03c5200640 1 -- 172.21.15.6:0/3509260968 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f03a40039b0 msgr2=0x7f03a4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:35:33.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:33.186+0000 7f03c5200640 1 --2- 172.21.15.6:0/3509260968 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f03a40039b0 0x7f03a4003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:33.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:33.186+0000 7f03c5200640 1 -- 172.21.15.6:0/3509260968 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03a4003520 con 0x7f03a40051c0 2024-03-04T09:35:33.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:33.187+0000 7f03c5200640 1 -- 172.21.15.6:0/3509260968 reap_dead start 2024-03-04T09:35:33.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:33.187+0000 7f03c5200640 1 --2- 172.21.15.6:0/3509260968 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f03a40051c0 0x7f03a400b9c0 secure :-1 s=READY pgs=1723 cs=0 l=1 rev1=1 crypto rx=0x7f03c0073470 tx=0x7f03a4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f83f556c2b22ac8 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:35:33.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:33.188+0000 7f03bea00640 1 -- 172.21.15.6:0/3509260968 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03b800a4a0 con 0x7f03a40051c0 2024-03-04T09:35:33.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:33.188+0000 7f03bea00640 1 -- 172.21.15.6:0/3509260968 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03b8034650 con 0x7f03a40051c0 2024-03-04T09:35:33.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:33.188+0000 7f03bea00640 1 -- 172.21.15.6:0/3509260968 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03b8034a40 con 0x7f03a40051c0 2024-03-04T09:35:33.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:33.188+0000 7f03c5c00640 1 -- 172.21.15.6:0/3509260968 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f03a40051c0 msgr2=0x7f03a400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:33.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:33.188+0000 7f03c5c00640 1 --2- 172.21.15.6:0/3509260968 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f03a40051c0 0x7f03a400b9c0 secure :-1 s=READY pgs=1723 cs=0 l=1 rev1=1 crypto rx=0x7f03c0073470 tx=0x7f03a4005830 comp rx=0 tx=0).stop 2024-03-04T09:35:33.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:33.188+0000 7f03c5c00640 1 -- 172.21.15.6:0/3509260968 shutdown_connections 2024-03-04T09:35:33.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:33.188+0000 7f03c5c00640 1 --2- 172.21.15.6:0/3509260968 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f03a40051c0 0x7f03a400b9c0 unknown :-1 s=CLOSED pgs=1723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:33.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:33.188+0000 7f03c5c00640 1 -- 172.21.15.6:0/3509260968 >> 172.21.15.6:0/3509260968 conn(0x7f03c006c0c0 msgr2=0x7f03c006c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:35:33.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:33.188+0000 7f03c5c00640 1 -- 172.21.15.6:0/3509260968 shutdown_connections 2024-03-04T09:35:33.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:33.188+0000 7f03c5c00640 1 -- 172.21.15.6:0/3509260968 wait complete. 2024-03-04T09:35:33.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:33.190+0000 7f03c5c00640 1 Processor -- start 2024-03-04T09:35:33.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:33.190+0000 7f03c5c00640 1 -- start start 2024-03-04T09:35:33.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:33.191+0000 7f03c5c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f03a40051c0 0x7f03c0154a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:33.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:33.191+0000 7f03c5c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f03c0154f50 0x7f03c0155a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:33.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:33.191+0000 7f03c5c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f03c0140440 0x7f03c0155f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:33.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:33.191+0000 7f03c5c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03c00d19b0 con 0x7f03a40051c0 2024-03-04T09:35:33.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:33.191+0000 7f03c5c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03c0144fe0 con 0x7f03c0140440 2024-03-04T09:35:33.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:33.191+0000 7f03c5c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03c006f820 con 0x7f03c0154f50 2024-03-04T09:35:33.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:33.191+0000 7f03bfe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f03a40051c0 0x7f03c0154a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:33.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:33.192+0000 7f03bfe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f03a40051c0 0x7f03c0154a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:39392/0 (socket says 172.21.15.6:39392) 2024-03-04T09:35:33.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:33.192+0000 7f03bfe00640 1 -- 172.21.15.6:0/2320217883 learned_addr learned my addr 172.21.15.6:0/2320217883 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:35:33.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:33.192+0000 7f03c5200640 1 --2- 172.21.15.6:0/2320217883 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f03c0140440 0x7f03c0155f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:33.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:33.192+0000 7f03bf400640 1 --2- 172.21.15.6:0/2320217883 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f03c0154f50 0x7f03c0155a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:33.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:33.192+0000 7f03bfe00640 1 -- 172.21.15.6:0/2320217883 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f03c0154f50 msgr2=0x7f03c0155a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:33.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:33.192+0000 7f03bfe00640 1 --2- 172.21.15.6:0/2320217883 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f03c0154f50 0x7f03c0155a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:33.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:33.192+0000 7f03bfe00640 1 -- 172.21.15.6:0/2320217883 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f03c0140440 msgr2=0x7f03c0155f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:33.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:33.192+0000 7f03bfe00640 1 --2- 172.21.15.6:0/2320217883 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f03c0140440 0x7f03c0155f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:33.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:33.192+0000 7f03bfe00640 1 -- 172.21.15.6:0/2320217883 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03c00b0b40 con 0x7f03a40051c0 2024-03-04T09:35:33.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:33.192+0000 7f03bfe00640 1 --2- 172.21.15.6:0/2320217883 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f03a40051c0 0x7f03c0154a10 secure :-1 s=READY pgs=1724 cs=0 l=1 rev1=1 crypto rx=0x7f03b0011540 tx=0x7f03b0011a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:35:33.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:33.193+0000 7f03b7e00640 1 -- 172.21.15.6:0/2320217883 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03b0015930 con 0x7f03a40051c0 2024-03-04T09:35:33.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:33.193+0000 7f03b7e00640 1 -- 172.21.15.6:0/2320217883 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03b0015ad0 con 0x7f03a40051c0 2024-03-04T09:35:33.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:33.193+0000 7f03b7e00640 1 -- 172.21.15.6:0/2320217883 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03b000f460 con 0x7f03a40051c0 2024-03-04T09:35:33.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:33.193+0000 7f03c5c00640 1 -- 172.21.15.6:0/2320217883 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03c00b0e20 con 0x7f03a40051c0 2024-03-04T09:35:33.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:33.193+0000 7f03c5c00640 1 -- 172.21.15.6:0/2320217883 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03c00b13e0 con 0x7f03a40051c0 2024-03-04T09:35:33.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:33.194+0000 7f03c5c00640 1 -- 172.21.15.6:0/2320217883 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03c004f330 con 0x7f03a40051c0 2024-03-04T09:35:33.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:33.196+0000 7f03b7e00640 1 -- 172.21.15.6:0/2320217883 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f03b000f600 con 0x7f03a40051c0 2024-03-04T09:35:33.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:33.197+0000 7f03b7e00640 1 --2- 172.21.15.6:0/2320217883 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f03a006b4d0 0x7f03a006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:33.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:33.197+0000 7f03b7e00640 1 -- 172.21.15.6:0/2320217883 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f03b008ac50 con 0x7f03a40051c0 2024-03-04T09:35:33.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:33.198+0000 7f03bf400640 1 --2- 172.21.15.6:0/2320217883 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f03a006b4d0 0x7f03a006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:33.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:33.199+0000 7f03bf400640 1 --2- 172.21.15.6:0/2320217883 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f03a006b4d0 0x7f03a006d990 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f03c0155330 tx=0x7f03ac009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:35:33.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:33.202+0000 7f03b7e00640 1 -- 172.21.15.6:0/2320217883 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f03b0019030 con 0x7f03a40051c0 2024-03-04T09:35:33.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:33.441+0000 7f03c5c00640 1 -- 172.21.15.6:0/2320217883 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f03c00b4130 con 0x7f03a40051c0 2024-03-04T09:35:33.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:33.442+0000 7f03b7e00640 1 -- 172.21.15.6:0/2320217883 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3464 (secure 0 0 0) 0x7f03b0059960 con 0x7f03a40051c0 2024-03-04T09:35:33.439 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:35:33.439 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:35:26.088222+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"laggy_since":"2024-03-04T09:33:54.652601+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":9}]} 2024-03-04T09:35:33.440 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 101 2024-03-04T09:35:33.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:33.445+0000 7f03c5c00640 1 -- 172.21.15.6:0/2320217883 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f03a006b4d0 msgr2=0x7f03a006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:33.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:33.445+0000 7f03c5c00640 1 --2- 172.21.15.6:0/2320217883 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f03a006b4d0 0x7f03a006d990 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f03c0155330 tx=0x7f03ac009040 comp rx=0 tx=0).stop 2024-03-04T09:35:33.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:33.445+0000 7f03c5c00640 1 -- 172.21.15.6:0/2320217883 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f03a40051c0 msgr2=0x7f03c0154a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:33.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:33.445+0000 7f03c5c00640 1 --2- 172.21.15.6:0/2320217883 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f03a40051c0 0x7f03c0154a10 secure :-1 s=READY pgs=1724 cs=0 l=1 rev1=1 crypto rx=0x7f03b0011540 tx=0x7f03b0011a00 comp rx=0 tx=0).stop 2024-03-04T09:35:33.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:33.445+0000 7f03c5c00640 1 -- 172.21.15.6:0/2320217883 shutdown_connections 2024-03-04T09:35:33.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:33.445+0000 7f03c5c00640 1 --2- 172.21.15.6:0/2320217883 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f03c0140440 0x7f03c0155f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:33.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:33.445+0000 7f03c5c00640 1 --2- 172.21.15.6:0/2320217883 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f03a006b4d0 0x7f03a006d990 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:33.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:33.445+0000 7f03c5c00640 1 --2- 172.21.15.6:0/2320217883 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f03c0154f50 0x7f03c0155a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:33.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:33.445+0000 7f03c5c00640 1 --2- 172.21.15.6:0/2320217883 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f03a40051c0 0x7f03c0154a10 unknown :-1 s=CLOSED pgs=1724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:33.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:33.445+0000 7f03c5c00640 1 -- 172.21.15.6:0/2320217883 >> 172.21.15.6:0/2320217883 conn(0x7f03c006c0c0 msgr2=0x7f03c006c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:35:33.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:33.445+0000 7f03c5c00640 1 -- 172.21.15.6:0/2320217883 shutdown_connections 2024-03-04T09:35:33.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:33.446+0000 7f03c5c00640 1 -- 172.21.15.6:0/2320217883 wait complete. 2024-03-04T09:35:33.664 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:32:57.159681+0000', 'modified': '2024-03-04T09:35:26.088222+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8214, 'mds_1': 6796}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6796': {'gid': 6796, 'name': 'd', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.146:6837/2458070840', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 2458070840}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 2458070840}]}, 'join_fscid': -1, 'laggy_since': '2024-03-04T09:33:54.652601+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8214': {'gid': 8214, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2446627517', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2446627517}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2446627517}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8214, 'qdb_cluster': [8214]} 2024-03-04T09:35:33.664 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6796:up:stopping 2024-03-04T09:35:33.895 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-04T09:35:34.665 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:35:34.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:34.899+0000 7f51f4e00640 1 Processor -- start 2024-03-04T09:35:34.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:34.899+0000 7f51f4e00640 1 -- start start 2024-03-04T09:35:34.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:34.901+0000 7f51f4e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f51f00b3c90 0x7f51f00cdc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:34.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:34.901+0000 7f51f4e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f51f00cf5c0 0x7f51f00b4070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:34.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:34.901+0000 7f51f4e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51f00b45b0 con 0x7f51f006cb10 2024-03-04T09:35:34.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:34.901+0000 7f51f4e00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51f00ce160 con 0x7f51f00b3c90 2024-03-04T09:35:34.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:34.901+0000 7f51f4e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51f00b4860 con 0x7f51f00cf5c0 2024-03-04T09:35:34.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:34.901+0000 7f51ee800640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f51f006cb10 0x7f51f006cfa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:54506/0 (socket says 172.21.15.6:54506) 2024-03-04T09:35:34.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:34.901+0000 7f51ee800640 1 -- 172.21.15.6:0/2651426153 learned_addr learned my addr 172.21.15.6:0/2651426153 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:35:34.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:34.902+0000 7f51ede00640 1 --2- 172.21.15.6:0/2651426153 >> v2:172.21.15.146:3300/0 conn(0x7f51f00b3c90 0x7f51f00cdc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:34.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:34.902+0000 7f51ed400640 1 --2- 172.21.15.6:0/2651426153 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f51f00cf5c0 0x7f51f00b4070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:34.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:34.902+0000 7f51eca00640 1 -- 172.21.15.6:0/2651426153 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2559553129 0 0) 0x7f51f00b45b0 con 0x7f51f006cb10 2024-03-04T09:35:34.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:34.902+0000 7f51eca00640 1 -- 172.21.15.6:0/2651426153 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51d4003520 con 0x7f51f006cb10 2024-03-04T09:35:34.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:34.902+0000 7f51eca00640 1 -- 172.21.15.6:0/2651426153 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2628776719 0 0) 0x7f51d4003520 con 0x7f51f006cb10 2024-03-04T09:35:34.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:34.902+0000 7f51eca00640 1 -- 172.21.15.6:0/2651426153 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f51f00cf5c0 msgr2=0x7f51f00b4070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:35:34.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:34.902+0000 7f51eca00640 1 --2- 172.21.15.6:0/2651426153 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f51f00cf5c0 0x7f51f00b4070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:34.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:34.902+0000 7f51eca00640 1 -- 172.21.15.6:0/2651426153 >> v2:172.21.15.146:3300/0 conn(0x7f51f00b3c90 msgr2=0x7f51f00cdc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:35:34.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:34.902+0000 7f51eca00640 1 --2- 172.21.15.6:0/2651426153 >> v2:172.21.15.146:3300/0 conn(0x7f51f00b3c90 0x7f51f00cdc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:34.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:34.903+0000 7f51eca00640 1 -- 172.21.15.6:0/2651426153 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51f00b4b40 con 0x7f51f006cb10 2024-03-04T09:35:34.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:34.903+0000 7f51eca00640 1 -- 172.21.15.6:0/2651426153 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7f51e80019f0 con 0x7f51f006cb10 2024-03-04T09:35:34.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:34.903+0000 7f51eca00640 1 -- 172.21.15.6:0/2651426153 >> v1:172.21.15.6:6789/0 conn(0x7f51f006cb10 legacy=0x7f51f006cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:34.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:34.903+0000 7f51eca00640 1 --2- 172.21.15.6:0/2651426153 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f51f00cf5c0 0x7f51d4003a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:34.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:34.903+0000 7f51eca00640 1 --2- 172.21.15.6:0/2651426153 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f51d4003fa0 0x7f51d4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:34.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:34.903+0000 7f51eca00640 1 --2- 172.21.15.6:0/2651426153 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f51d4004d40 0x7f51d40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:34.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:34.903+0000 7f51eca00640 1 -- 172.21.15.6:0/2651426153 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51f00ce160 con 0x7f51d4003fa0 2024-03-04T09:35:34.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:34.903+0000 7f51eca00640 1 -- 172.21.15.6:0/2651426153 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51f00b4860 con 0x7f51d4004d40 2024-03-04T09:35:34.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:34.903+0000 7f51eca00640 1 -- 172.21.15.6:0/2651426153 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51d400dc80 con 0x7f51f00cf5c0 2024-03-04T09:35:34.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:34.903+0000 7f51ed400640 1 --2- 172.21.15.6:0/2651426153 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f51d4003fa0 0x7f51d4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:34.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:34.903+0000 7f51ede00640 1 --2- 172.21.15.6:0/2651426153 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f51f00cf5c0 0x7f51d4003a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:34.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:34.904+0000 7f51ee800640 1 --2- 172.21.15.6:0/2651426153 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f51d4004d40 0x7f51d40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:34.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:34.904+0000 7f51ed400640 1 -- 172.21.15.6:0/2651426153 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f51f00cf5c0 msgr2=0x7f51d4003a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:35:34.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:34.904+0000 7f51ed400640 1 --2- 172.21.15.6:0/2651426153 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f51f00cf5c0 0x7f51d4003a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:34.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:34.904+0000 7f51ed400640 1 -- 172.21.15.6:0/2651426153 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f51d4004d40 msgr2=0x7f51d40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:35:34.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:34.904+0000 7f51ed400640 1 --2- 172.21.15.6:0/2651426153 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f51d4004d40 0x7f51d40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:34.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:34.904+0000 7f51ed400640 1 -- 172.21.15.6:0/2651426153 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51d4003520 con 0x7f51d4003fa0 2024-03-04T09:35:34.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:34.904+0000 7f51ee800640 1 --2- 172.21.15.6:0/2651426153 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f51d4004d40 0x7f51d40051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:35:34.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:34.905+0000 7f51ed400640 1 --2- 172.21.15.6:0/2651426153 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f51d4003fa0 0x7f51d4004800 secure :-1 s=READY pgs=1726 cs=0 l=1 rev1=1 crypto rx=0x7f51dc009ac0 tx=0x7f51d4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec12cb94db9fdc5c server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:35:34.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:34.906+0000 7f51eca00640 1 -- 172.21.15.6:0/2651426153 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51dc039070 con 0x7f51d4003fa0 2024-03-04T09:35:34.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:34.906+0000 7f51eca00640 1 -- 172.21.15.6:0/2651426153 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51dc02c6b0 con 0x7f51d4003fa0 2024-03-04T09:35:34.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:34.906+0000 7f51eca00640 1 -- 172.21.15.6:0/2651426153 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51dc009690 con 0x7f51d4003fa0 2024-03-04T09:35:34.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:34.906+0000 7f51f4e00640 1 -- 172.21.15.6:0/2651426153 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f51d4003fa0 msgr2=0x7f51d4004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:34.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:34.906+0000 7f51f4e00640 1 --2- 172.21.15.6:0/2651426153 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f51d4003fa0 0x7f51d4004800 secure :-1 s=READY pgs=1726 cs=0 l=1 rev1=1 crypto rx=0x7f51dc009ac0 tx=0x7f51d4005830 comp rx=0 tx=0).stop 2024-03-04T09:35:34.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:34.906+0000 7f51ee800640 1 -- 172.21.15.6:0/2651426153 reap_dead start 2024-03-04T09:35:34.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:34.906+0000 7f51f4e00640 1 -- 172.21.15.6:0/2651426153 shutdown_connections 2024-03-04T09:35:34.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:34.906+0000 7f51f4e00640 1 -- 172.21.15.6:0/2651426153 >> 172.21.15.6:0/2651426153 conn(0x7f51f006afb0 msgr2=0x7f51f006b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:35:34.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:34.907+0000 7f51f4e00640 1 -- 172.21.15.6:0/2651426153 shutdown_connections 2024-03-04T09:35:34.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:34.907+0000 7f51f4e00640 1 -- 172.21.15.6:0/2651426153 wait complete. 2024-03-04T09:35:34.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:34.908+0000 7f51f4e00640 1 Processor -- start 2024-03-04T09:35:34.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:34.909+0000 7f51f4e00640 1 -- start start 2024-03-04T09:35:34.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:34.910+0000 7f51f4e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f51d4003fa0 0x7f51f0150530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:34.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:34.910+0000 7f51f4e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f51f014f880 0x7f51f01529b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:34.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:34.910+0000 7f51f4e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f51f014fd00 0x7f51f01548b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:34.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:34.910+0000 7f51f4e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51f015b340 con 0x7f51f014f880 2024-03-04T09:35:34.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:34.910+0000 7f51f4e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51f00b5100 con 0x7f51d4003fa0 2024-03-04T09:35:34.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:34.910+0000 7f51f4e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51f006d6b0 con 0x7f51f014fd00 2024-03-04T09:35:34.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:34.910+0000 7f51ed400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f51f014f880 0x7f51f01529b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:34.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:34.910+0000 7f51ed400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f51f014f880 0x7f51f01529b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:39416/0 (socket says 172.21.15.6:39416) 2024-03-04T09:35:34.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:34.910+0000 7f51ed400640 1 -- 172.21.15.6:0/2892846230 learned_addr learned my addr 172.21.15.6:0/2892846230 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:35:34.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:34.910+0000 7f51ede00640 1 --2- 172.21.15.6:0/2892846230 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f51d4003fa0 0x7f51f0150530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:34.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:34.910+0000 7f51ee800640 1 --2- 172.21.15.6:0/2892846230 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f51f014fd00 0x7f51f01548b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:34.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:34.911+0000 7f51ed400640 1 -- 172.21.15.6:0/2892846230 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f51f014fd00 msgr2=0x7f51f01548b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:34.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:34.911+0000 7f51ed400640 1 --2- 172.21.15.6:0/2892846230 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f51f014fd00 0x7f51f01548b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:34.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:34.911+0000 7f51ed400640 1 -- 172.21.15.6:0/2892846230 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f51d4003fa0 msgr2=0x7f51f0150530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:34.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:34.911+0000 7f51ed400640 1 --2- 172.21.15.6:0/2892846230 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f51d4003fa0 0x7f51f0150530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:34.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:34.911+0000 7f51ed400640 1 -- 172.21.15.6:0/2892846230 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51f0151bf0 con 0x7f51f014f880 2024-03-04T09:35:34.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:34.911+0000 7f51ee800640 1 --2- 172.21.15.6:0/2892846230 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f51f014fd00 0x7f51f01548b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:35:34.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:34.911+0000 7f51ed400640 1 --2- 172.21.15.6:0/2892846230 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f51f014f880 0x7f51f01529b0 secure :-1 s=READY pgs=1727 cs=0 l=1 rev1=1 crypto rx=0x7f51dc000c00 tx=0x7f51dc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:35:34.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:34.911+0000 7f51e6000640 1 -- 172.21.15.6:0/2892846230 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51dc042020 con 0x7f51f014f880 2024-03-04T09:35:34.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:34.911+0000 7f51e6000640 1 -- 172.21.15.6:0/2892846230 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51dc03a040 con 0x7f51f014f880 2024-03-04T09:35:34.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:34.911+0000 7f51e6000640 1 -- 172.21.15.6:0/2892846230 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51dc039070 con 0x7f51f014f880 2024-03-04T09:35:34.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:34.911+0000 7f51f4e00640 1 -- 172.21.15.6:0/2892846230 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51f0151ed0 con 0x7f51f014f880 2024-03-04T09:35:34.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:34.912+0000 7f51f4e00640 1 -- 172.21.15.6:0/2892846230 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51f0152490 con 0x7f51f014f880 2024-03-04T09:35:34.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:34.913+0000 7f51f4e00640 1 -- 172.21.15.6:0/2892846230 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51f0069510 con 0x7f51f014f880 2024-03-04T09:35:34.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:34.914+0000 7f51e6000640 1 -- 172.21.15.6:0/2892846230 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f51dc045020 con 0x7f51f014f880 2024-03-04T09:35:34.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:34.915+0000 7f51e6000640 1 --2- 172.21.15.6:0/2892846230 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f51d006b4d0 0x7f51d006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:34.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:34.916+0000 7f51e6000640 1 -- 172.21.15.6:0/2892846230 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f51dc0ab680 con 0x7f51f014f880 2024-03-04T09:35:34.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:34.916+0000 7f51ede00640 1 --2- 172.21.15.6:0/2892846230 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f51d006b4d0 0x7f51d006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:34.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:34.917+0000 7f51ede00640 1 --2- 172.21.15.6:0/2892846230 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f51d006b4d0 0x7f51d006d990 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f51f00b1030 tx=0x7f51e0008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:35:34.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:34.920+0000 7f51e6000640 1 -- 172.21.15.6:0/2892846230 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f51dc04c070 con 0x7f51f014f880 2024-03-04T09:35:35.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:35.158+0000 7f51f4e00640 1 -- 172.21.15.6:0/2892846230 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f51f004f3a0 con 0x7f51f014f880 2024-03-04T09:35:35.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:35.158+0000 7f51e6000640 1 -- 172.21.15.6:0/2892846230 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3464 (secure 0 0 0) 0x7f51dc07a4c0 con 0x7f51f014f880 2024-03-04T09:35:35.155 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:35:35.156 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:35:26.088222+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"laggy_since":"2024-03-04T09:33:54.652601+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":9}]} 2024-03-04T09:35:35.156 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 101 2024-03-04T09:35:35.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:35.161+0000 7f51f4e00640 1 -- 172.21.15.6:0/2892846230 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f51d006b4d0 msgr2=0x7f51d006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:35.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:35.161+0000 7f51f4e00640 1 --2- 172.21.15.6:0/2892846230 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f51d006b4d0 0x7f51d006d990 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f51f00b1030 tx=0x7f51e0008040 comp rx=0 tx=0).stop 2024-03-04T09:35:35.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:35.161+0000 7f51f4e00640 1 -- 172.21.15.6:0/2892846230 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f51f014f880 msgr2=0x7f51f01529b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:35.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:35.161+0000 7f51f4e00640 1 --2- 172.21.15.6:0/2892846230 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f51f014f880 0x7f51f01529b0 secure :-1 s=READY pgs=1727 cs=0 l=1 rev1=1 crypto rx=0x7f51dc000c00 tx=0x7f51dc004730 comp rx=0 tx=0).stop 2024-03-04T09:35:35.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:35.161+0000 7f51f4e00640 1 -- 172.21.15.6:0/2892846230 shutdown_connections 2024-03-04T09:35:35.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:35.161+0000 7f51f4e00640 1 --2- 172.21.15.6:0/2892846230 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f51d006b4d0 0x7f51d006d990 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:35.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:35.161+0000 7f51f4e00640 1 --2- 172.21.15.6:0/2892846230 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f51f014fd00 0x7f51f01548b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:35.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:35.161+0000 7f51f4e00640 1 --2- 172.21.15.6:0/2892846230 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f51f014f880 0x7f51f01529b0 unknown :-1 s=CLOSED pgs=1727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:35.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:35.161+0000 7f51f4e00640 1 --2- 172.21.15.6:0/2892846230 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f51d4003fa0 0x7f51f0150530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:35.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:35.161+0000 7f51f4e00640 1 -- 172.21.15.6:0/2892846230 >> 172.21.15.6:0/2892846230 conn(0x7f51f006afb0 msgr2=0x7f51f0155990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:35:35.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:35.161+0000 7f51f4e00640 1 -- 172.21.15.6:0/2892846230 shutdown_connections 2024-03-04T09:35:35.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:35.161+0000 7f51f4e00640 1 -- 172.21.15.6:0/2892846230 wait complete. 2024-03-04T09:35:35.376 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:32:57.159681+0000', 'modified': '2024-03-04T09:35:26.088222+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8214, 'mds_1': 6796}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6796': {'gid': 6796, 'name': 'd', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.146:6837/2458070840', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 2458070840}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 2458070840}]}, 'join_fscid': -1, 'laggy_since': '2024-03-04T09:33:54.652601+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8214': {'gid': 8214, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2446627517', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2446627517}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2446627517}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8214, 'qdb_cluster': [8214]} 2024-03-04T09:35:35.376 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6796:up:stopping 2024-03-04T09:35:35.897 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-04T09:35:36.376 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:35:36.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:36.605+0000 7f96f5a00640 1 Processor -- start 2024-03-04T09:35:36.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:36.606+0000 7f96f5a00640 1 -- start start 2024-03-04T09:35:36.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:36.607+0000 7f96f5a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f96f0161b40 0x7f96f00cfd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:36.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:36.607+0000 7f96f5a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f96f00b5060 0x7f96f00d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:36.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:36.607+0000 7f96f5a00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f96f00b54a0 0x7f96f00d2760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:36.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:36.607+0000 7f96f5a00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96f00cf900 con 0x7f96f0161b40 2024-03-04T09:35:36.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:36.607+0000 7f96f5a00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96f00cea50 con 0x7f96f00b54a0 2024-03-04T09:35:36.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:36.607+0000 7f96f5a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96f00cecf0 con 0x7f96f00b5060 2024-03-04T09:35:36.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:36.607+0000 7f96eea00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f96f0161b40 0x7f96f00cfd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:36.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:36.608+0000 7f96eea00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f96f0161b40 0x7f96f00cfd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:39424/0 (socket says 172.21.15.6:39424) 2024-03-04T09:35:36.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:36.608+0000 7f96eea00640 1 -- 172.21.15.6:0/2911590340 learned_addr learned my addr 172.21.15.6:0/2911590340 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:35:36.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:36.608+0000 7f96ee000640 1 --2- 172.21.15.6:0/2911590340 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f96f00b5060 0x7f96f00d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:36.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:36.608+0000 7f96ef400640 1 --2- 172.21.15.6:0/2911590340 >> v2:172.21.15.146:3300/0 conn(0x7f96f00b54a0 0x7f96f00d2760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:36.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:36.608+0000 7f96eea00640 1 -- 172.21.15.6:0/2911590340 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f96f00b5060 msgr2=0x7f96f00d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:35:36.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:36.608+0000 7f96eea00640 1 --2- 172.21.15.6:0/2911590340 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f96f00b5060 0x7f96f00d2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:36.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:36.608+0000 7f96eea00640 1 -- 172.21.15.6:0/2911590340 >> v2:172.21.15.146:3300/0 conn(0x7f96f00b54a0 msgr2=0x7f96f00d2760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:35:36.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:36.608+0000 7f96eea00640 1 --2- 172.21.15.6:0/2911590340 >> v2:172.21.15.146:3300/0 conn(0x7f96f00b54a0 0x7f96f00d2760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:36.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:36.608+0000 7f96eea00640 1 -- 172.21.15.6:0/2911590340 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96f00cefd0 con 0x7f96f0161b40 2024-03-04T09:35:36.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:36.608+0000 7f96ef400640 1 --2- 172.21.15.6:0/2911590340 >> v2:172.21.15.146:3300/0 conn(0x7f96f00b54a0 0x7f96f00d2760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:35:36.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:36.608+0000 7f96ee000640 1 --2- 172.21.15.6:0/2911590340 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f96f00b5060 0x7f96f00d2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:35:36.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:36.609+0000 7f96eea00640 1 --2- 172.21.15.6:0/2911590340 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f96f0161b40 0x7f96f00cfd90 secure :-1 s=READY pgs=1728 cs=0 l=1 rev1=1 crypto rx=0x7f96f006b7f0 tx=0x7f96e002b920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4aa21d9a882cb77 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:35:36.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:36.610+0000 7f96ed600640 1 -- 172.21.15.6:0/2911590340 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96e0035020 con 0x7f96f0161b40 2024-03-04T09:35:36.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:36.610+0000 7f96ed600640 1 -- 172.21.15.6:0/2911590340 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96e0031880 con 0x7f96f0161b40 2024-03-04T09:35:36.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:36.610+0000 7f96ed600640 1 -- 172.21.15.6:0/2911590340 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96e0031c50 con 0x7f96f0161b40 2024-03-04T09:35:36.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:36.610+0000 7f96f5a00640 1 -- 172.21.15.6:0/2911590340 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f96f0161b40 msgr2=0x7f96f00cfd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:36.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:36.610+0000 7f96f5a00640 1 --2- 172.21.15.6:0/2911590340 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f96f0161b40 0x7f96f00cfd90 secure :-1 s=READY pgs=1728 cs=0 l=1 rev1=1 crypto rx=0x7f96f006b7f0 tx=0x7f96e002b920 comp rx=0 tx=0).stop 2024-03-04T09:35:36.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:36.610+0000 7f96f5a00640 1 -- 172.21.15.6:0/2911590340 shutdown_connections 2024-03-04T09:35:36.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:36.610+0000 7f96f5a00640 1 --2- 172.21.15.6:0/2911590340 >> v2:172.21.15.146:3300/0 conn(0x7f96f00b54a0 0x7f96f00d2760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:36.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:36.610+0000 7f96f5a00640 1 --2- 172.21.15.6:0/2911590340 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f96f00b5060 0x7f96f00d2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:36.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:36.610+0000 7f96f5a00640 1 --2- 172.21.15.6:0/2911590340 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f96f0161b40 0x7f96f00cfd90 unknown :-1 s=CLOSED pgs=1728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:36.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:36.610+0000 7f96f5a00640 1 -- 172.21.15.6:0/2911590340 >> 172.21.15.6:0/2911590340 conn(0x7f96f009d220 msgr2=0x7f96f00d3ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:35:36.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:36.610+0000 7f96f5a00640 1 -- 172.21.15.6:0/2911590340 shutdown_connections 2024-03-04T09:35:36.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:36.611+0000 7f96f5a00640 1 -- 172.21.15.6:0/2911590340 wait complete. 2024-03-04T09:35:36.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:36.612+0000 7f96f5a00640 1 Processor -- start 2024-03-04T09:35:36.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:36.612+0000 7f96f5a00640 1 -- start start 2024-03-04T09:35:36.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:36.613+0000 7f96f5a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f96f00b5060 0x7f96f01f4c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:36.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:36.613+0000 7f96f5a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f96f00b54a0 0x7f96f015ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:36.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:36.613+0000 7f96f5a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f96f0161b40 0x7f96f00cd6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:36.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:36.613+0000 7f96f5a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96f00b2ff0 con 0x7f96f00b54a0 2024-03-04T09:35:36.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:36.613+0000 7f96f5a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96f00cf590 con 0x7f96f00b5060 2024-03-04T09:35:36.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:36.613+0000 7f96f5a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96f00b3170 con 0x7f96f0161b40 2024-03-04T09:35:36.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:36.614+0000 7f96ee000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f96f00b54a0 0x7f96f015ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:36.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:36.614+0000 7f96ee000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f96f00b54a0 0x7f96f015ad40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:39430/0 (socket says 172.21.15.6:39430) 2024-03-04T09:35:36.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:36.614+0000 7f96ee000640 1 -- 172.21.15.6:0/3482287565 learned_addr learned my addr 172.21.15.6:0/3482287565 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:35:36.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:36.614+0000 7f96eea00640 1 --2- 172.21.15.6:0/3482287565 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f96f00b5060 0x7f96f01f4c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:36.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:36.614+0000 7f96ef400640 1 --2- 172.21.15.6:0/3482287565 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f96f0161b40 0x7f96f00cd6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:36.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:36.614+0000 7f96ee000640 1 -- 172.21.15.6:0/3482287565 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f96f0161b40 msgr2=0x7f96f00cd6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:36.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:36.614+0000 7f96ee000640 1 --2- 172.21.15.6:0/3482287565 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f96f0161b40 0x7f96f00cd6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:36.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:36.614+0000 7f96ee000640 1 -- 172.21.15.6:0/3482287565 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f96f00b5060 msgr2=0x7f96f01f4c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:36.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:36.614+0000 7f96ee000640 1 --2- 172.21.15.6:0/3482287565 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f96f00b5060 0x7f96f01f4c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:36.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:36.614+0000 7f96ee000640 1 -- 172.21.15.6:0/3482287565 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96f00b7fc0 con 0x7f96f00b54a0 2024-03-04T09:35:36.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:36.615+0000 7f96ee000640 1 --2- 172.21.15.6:0/3482287565 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f96f00b54a0 0x7f96f015ad40 secure :-1 s=READY pgs=1729 cs=0 l=1 rev1=1 crypto rx=0x7f96dc00c7f0 tx=0x7f96dc00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:35:36.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:36.615+0000 7f96e6a00640 1 -- 172.21.15.6:0/3482287565 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96dc015070 con 0x7f96f00b54a0 2024-03-04T09:35:36.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:36.615+0000 7f96e6a00640 1 -- 172.21.15.6:0/3482287565 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96dc007820 con 0x7f96f00b54a0 2024-03-04T09:35:36.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:36.615+0000 7f96f5a00640 1 -- 172.21.15.6:0/3482287565 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96f00b8300 con 0x7f96f00b54a0 2024-03-04T09:35:36.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:36.615+0000 7f96e6a00640 1 -- 172.21.15.6:0/3482287565 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96dc007c10 con 0x7f96f00b54a0 2024-03-04T09:35:36.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:36.615+0000 7f96f5a00640 1 -- 172.21.15.6:0/3482287565 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96f01f5170 con 0x7f96f00b54a0 2024-03-04T09:35:36.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:36.616+0000 7f96f5a00640 1 -- 172.21.15.6:0/3482287565 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96f004f3a0 con 0x7f96f00b54a0 2024-03-04T09:35:36.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:36.617+0000 7f96e6a00640 1 -- 172.21.15.6:0/3482287565 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f96dc018070 con 0x7f96f00b54a0 2024-03-04T09:35:36.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:36.619+0000 7f96e6a00640 1 --2- 172.21.15.6:0/3482287565 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f96d006b4d0 0x7f96d006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:36.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:36.619+0000 7f96e6a00640 1 -- 172.21.15.6:0/3482287565 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f96dc08a340 con 0x7f96f00b54a0 2024-03-04T09:35:36.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:36.620+0000 7f96eea00640 1 --2- 172.21.15.6:0/3482287565 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f96d006b4d0 0x7f96d006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:36.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:36.621+0000 7f96eea00640 1 --2- 172.21.15.6:0/3482287565 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f96d006b4d0 0x7f96d006d990 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f96e0036000 tx=0x7f96e003c040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:35:36.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:36.623+0000 7f96e6a00640 1 -- 172.21.15.6:0/3482287565 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f96f00b8300 con 0x7f96f00b54a0 2024-03-04T09:35:36.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:36.874+0000 7f96f5a00640 1 -- 172.21.15.6:0/3482287565 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f96f01f4390 con 0x7f96f00b54a0 2024-03-04T09:35:36.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:36.875+0000 7f96e6a00640 1 -- 172.21.15.6:0/3482287565 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3464 (secure 0 0 0) 0x7f96dc059190 con 0x7f96f00b54a0 2024-03-04T09:35:36.872 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:35:36.872 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:35:26.088222+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"laggy_since":"2024-03-04T09:33:54.652601+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":9}]} 2024-03-04T09:35:36.872 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 101 2024-03-04T09:35:36.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:36.877+0000 7f96f5a00640 1 -- 172.21.15.6:0/3482287565 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f96d006b4d0 msgr2=0x7f96d006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:36.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:36.877+0000 7f96f5a00640 1 --2- 172.21.15.6:0/3482287565 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f96d006b4d0 0x7f96d006d990 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f96e0036000 tx=0x7f96e003c040 comp rx=0 tx=0).stop 2024-03-04T09:35:36.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:36.877+0000 7f96f5a00640 1 -- 172.21.15.6:0/3482287565 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f96f00b54a0 msgr2=0x7f96f015ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:36.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:36.877+0000 7f96f5a00640 1 --2- 172.21.15.6:0/3482287565 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f96f00b54a0 0x7f96f015ad40 secure :-1 s=READY pgs=1729 cs=0 l=1 rev1=1 crypto rx=0x7f96dc00c7f0 tx=0x7f96dc00ccb0 comp rx=0 tx=0).stop 2024-03-04T09:35:36.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:36.877+0000 7f96f5a00640 1 -- 172.21.15.6:0/3482287565 shutdown_connections 2024-03-04T09:35:36.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:36.877+0000 7f96f5a00640 1 --2- 172.21.15.6:0/3482287565 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f96d006b4d0 0x7f96d006d990 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:36.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:36.877+0000 7f96f5a00640 1 --2- 172.21.15.6:0/3482287565 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f96f0161b40 0x7f96f00cd6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:36.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:36.877+0000 7f96f5a00640 1 --2- 172.21.15.6:0/3482287565 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f96f00b54a0 0x7f96f015ad40 unknown :-1 s=CLOSED pgs=1729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:36.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:36.877+0000 7f96f5a00640 1 --2- 172.21.15.6:0/3482287565 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f96f00b5060 0x7f96f01f4c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:36.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:36.877+0000 7f96f5a00640 1 -- 172.21.15.6:0/3482287565 >> 172.21.15.6:0/3482287565 conn(0x7f96f009d220 msgr2=0x7f96f00d0570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:35:36.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:36.878+0000 7f96f5a00640 1 -- 172.21.15.6:0/3482287565 shutdown_connections 2024-03-04T09:35:36.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:36.878+0000 7f96f5a00640 1 -- 172.21.15.6:0/3482287565 wait complete. 2024-03-04T09:35:37.096 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:32:57.159681+0000', 'modified': '2024-03-04T09:35:26.088222+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8214, 'mds_1': 6796}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6796': {'gid': 6796, 'name': 'd', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.146:6837/2458070840', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 2458070840}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 2458070840}]}, 'join_fscid': -1, 'laggy_since': '2024-03-04T09:33:54.652601+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8214': {'gid': 8214, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2446627517', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2446627517}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2446627517}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8214, 'qdb_cluster': [8214]} 2024-03-04T09:35:37.096 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6796:up:stopping 2024-03-04T09:35:37.899 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-04T09:35:38.097 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:35:38.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:38.319+0000 7f5f1f200640 1 Processor -- start 2024-03-04T09:35:38.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:38.319+0000 7f5f1f200640 1 -- start start 2024-03-04T09:35:38.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:38.321+0000 7f5f1f200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5f2015e0b0 0x7f5f200cf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:38.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:38.321+0000 7f5f1f200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f5f200d4a10 0x7f5f200d0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:38.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:38.321+0000 7f5f1f200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f5f200d0c80 0x7f5f200d1900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:38.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:38.321+0000 7f5f1f200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f200b09e0 con 0x7f5f200d0c80 2024-03-04T09:35:38.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:38.321+0000 7f5f1f200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f200b3710 con 0x7f5f200d4a10 2024-03-04T09:35:38.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:38.321+0000 7f5f1f200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f200b39f0 con 0x7f5f2015e0b0 2024-03-04T09:35:38.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:38.321+0000 7f5f1e800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f5f200d0c80 0x7f5f200d1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:38.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:38.321+0000 7f5f1e800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f5f200d0c80 0x7f5f200d1900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:39440/0 (socket says 172.21.15.6:39440) 2024-03-04T09:35:38.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:38.321+0000 7f5f1e800640 1 -- 172.21.15.6:0/2003299452 learned_addr learned my addr 172.21.15.6:0/2003299452 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:35:38.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:38.321+0000 7f5f1de00640 1 --2- 172.21.15.6:0/2003299452 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5f2015e0b0 0x7f5f200cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:38.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:38.322+0000 7f5f1d400640 1 --2- 172.21.15.6:0/2003299452 >> v2:172.21.15.146:3300/0 conn(0x7f5f200d4a10 0x7f5f200d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:38.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:38.322+0000 7f5f1e800640 1 -- 172.21.15.6:0/2003299452 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5f2015e0b0 msgr2=0x7f5f200cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:35:38.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:38.322+0000 7f5f1e800640 1 --2- 172.21.15.6:0/2003299452 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5f2015e0b0 0x7f5f200cf310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:38.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:38.322+0000 7f5f1e800640 1 -- 172.21.15.6:0/2003299452 >> v2:172.21.15.146:3300/0 conn(0x7f5f200d4a10 msgr2=0x7f5f200d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:35:38.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:38.322+0000 7f5f1e800640 1 --2- 172.21.15.6:0/2003299452 >> v2:172.21.15.146:3300/0 conn(0x7f5f200d4a10 0x7f5f200d0740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:38.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:38.322+0000 7f5f1e800640 1 -- 172.21.15.6:0/2003299452 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f200b3cd0 con 0x7f5f200d0c80 2024-03-04T09:35:38.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:38.322+0000 7f5f1d400640 1 --2- 172.21.15.6:0/2003299452 >> v2:172.21.15.146:3300/0 conn(0x7f5f200d4a10 0x7f5f200d0740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:35:38.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:38.323+0000 7f5f1e800640 1 --2- 172.21.15.6:0/2003299452 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5f200d0c80 0x7f5f200d1900 secure :-1 s=READY pgs=1730 cs=0 l=1 rev1=1 crypto rx=0x7f5f200b6920 tx=0x7f5f1802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=606db25481f87515 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:35:38.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:38.323+0000 7f5f1ca00640 1 -- 172.21.15.6:0/2003299452 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f1802ca80 con 0x7f5f200d0c80 2024-03-04T09:35:38.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:38.323+0000 7f5f1ca00640 1 -- 172.21.15.6:0/2003299452 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f1802cc20 con 0x7f5f200d0c80 2024-03-04T09:35:38.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:38.323+0000 7f5f1ca00640 1 -- 172.21.15.6:0/2003299452 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f180343f0 con 0x7f5f200d0c80 2024-03-04T09:35:38.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:38.324+0000 7f5f1f200640 1 -- 172.21.15.6:0/2003299452 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5f200d0c80 msgr2=0x7f5f200d1900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:38.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:38.324+0000 7f5f1f200640 1 --2- 172.21.15.6:0/2003299452 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5f200d0c80 0x7f5f200d1900 secure :-1 s=READY pgs=1730 cs=0 l=1 rev1=1 crypto rx=0x7f5f200b6920 tx=0x7f5f1802bee0 comp rx=0 tx=0).stop 2024-03-04T09:35:38.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:38.324+0000 7f5f1f200640 1 -- 172.21.15.6:0/2003299452 shutdown_connections 2024-03-04T09:35:38.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:38.324+0000 7f5f1f200640 1 --2- 172.21.15.6:0/2003299452 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5f200d0c80 0x7f5f200d1900 unknown :-1 s=CLOSED pgs=1730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:38.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:38.324+0000 7f5f1f200640 1 --2- 172.21.15.6:0/2003299452 >> v2:172.21.15.146:3300/0 conn(0x7f5f200d4a10 0x7f5f200d0740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:38.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:38.324+0000 7f5f1f200640 1 --2- 172.21.15.6:0/2003299452 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5f2015e0b0 0x7f5f200cf310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:38.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:38.324+0000 7f5f1f200640 1 -- 172.21.15.6:0/2003299452 >> 172.21.15.6:0/2003299452 conn(0x7f5f200d3d30 msgr2=0x7f5f200d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:35:38.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:38.324+0000 7f5f1f200640 1 -- 172.21.15.6:0/2003299452 shutdown_connections 2024-03-04T09:35:38.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:38.324+0000 7f5f1f200640 1 -- 172.21.15.6:0/2003299452 wait complete. 2024-03-04T09:35:38.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:38.326+0000 7f5f1f200640 1 Processor -- start 2024-03-04T09:35:38.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:38.326+0000 7f5f1f200640 1 -- start start 2024-03-04T09:35:38.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:38.327+0000 7f5f1f200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5f200d0c80 0x7f5f2015ad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:38.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:38.327+0000 7f5f1f200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5f200d4a10 0x7f5f200b8ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:38.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:38.327+0000 7f5f1f200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5f2015e0b0 0x7f5f20160620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:38.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:38.327+0000 7f5f1f200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f200cfc60 con 0x7f5f200d4a10 2024-03-04T09:35:38.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:38.327+0000 7f5f1f200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f2015e490 con 0x7f5f200d0c80 2024-03-04T09:35:38.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:38.327+0000 7f5f1f200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f200cf850 con 0x7f5f2015e0b0 2024-03-04T09:35:38.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:38.328+0000 7f5f1d400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5f200d4a10 0x7f5f200b8ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:38.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:38.328+0000 7f5f1d400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5f200d4a10 0x7f5f200b8ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:39444/0 (socket says 172.21.15.6:39444) 2024-03-04T09:35:38.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:38.328+0000 7f5f1d400640 1 -- 172.21.15.6:0/3467708355 learned_addr learned my addr 172.21.15.6:0/3467708355 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:35:38.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:38.328+0000 7f5f1de00640 1 --2- 172.21.15.6:0/3467708355 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5f200d0c80 0x7f5f2015ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:38.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:38.328+0000 7f5f1d400640 1 -- 172.21.15.6:0/3467708355 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5f2015e0b0 msgr2=0x7f5f20160620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:38.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:38.328+0000 7f5f1d400640 1 --2- 172.21.15.6:0/3467708355 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5f2015e0b0 0x7f5f20160620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:38.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:38.328+0000 7f5f1d400640 1 -- 172.21.15.6:0/3467708355 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5f200d0c80 msgr2=0x7f5f2015ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:38.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:38.328+0000 7f5f1d400640 1 --2- 172.21.15.6:0/3467708355 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5f200d0c80 0x7f5f2015ad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:38.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:38.328+0000 7f5f1d400640 1 -- 172.21.15.6:0/3467708355 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f200cb3d0 con 0x7f5f200d4a10 2024-03-04T09:35:38.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:38.329+0000 7f5f1d400640 1 --2- 172.21.15.6:0/3467708355 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5f200d4a10 0x7f5f200b8ef0 secure :-1 s=READY pgs=1731 cs=0 l=1 rev1=1 crypto rx=0x7f5f0c00c820 tx=0x7f5f0c00cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:35:38.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:38.329+0000 7f5f16000640 1 -- 172.21.15.6:0/3467708355 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f0c015070 con 0x7f5f200d4a10 2024-03-04T09:35:38.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:38.329+0000 7f5f16000640 1 -- 172.21.15.6:0/3467708355 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f0c007a30 con 0x7f5f200d4a10 2024-03-04T09:35:38.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:38.329+0000 7f5f16000640 1 -- 172.21.15.6:0/3467708355 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f0c007e00 con 0x7f5f200d4a10 2024-03-04T09:35:38.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:38.329+0000 7f5f1f200640 1 -- 172.21.15.6:0/3467708355 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f200cb6b0 con 0x7f5f200d4a10 2024-03-04T09:35:38.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:38.329+0000 7f5f1f200640 1 -- 172.21.15.6:0/3467708355 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f200cbc70 con 0x7f5f200d4a10 2024-03-04T09:35:38.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:38.331+0000 7f5f1f200640 1 -- 172.21.15.6:0/3467708355 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f2004f330 con 0x7f5f200d4a10 2024-03-04T09:35:38.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:38.331+0000 7f5f16000640 1 -- 172.21.15.6:0/3467708355 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5f0c018070 con 0x7f5f200d4a10 2024-03-04T09:35:38.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:38.333+0000 7f5f16000640 1 --2- 172.21.15.6:0/3467708355 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f5f0006b4d0 0x7f5f0006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:38.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:38.333+0000 7f5f16000640 1 -- 172.21.15.6:0/3467708355 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f5f0c08a230 con 0x7f5f200d4a10 2024-03-04T09:35:38.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:38.334+0000 7f5f1de00640 1 --2- 172.21.15.6:0/3467708355 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f5f0006b4d0 0x7f5f0006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:38.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:38.335+0000 7f5f1de00640 1 --2- 172.21.15.6:0/3467708355 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f5f0006b4d0 0x7f5f0006d990 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f5f10006fd0 tx=0x7f5f10008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:35:38.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:38.337+0000 7f5f16000640 1 -- 172.21.15.6:0/3467708355 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5f0c059060 con 0x7f5f200d4a10 2024-03-04T09:35:38.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:38.636+0000 7f5f1f200640 1 -- 172.21.15.6:0/3467708355 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5f200b5860 con 0x7f5f200d4a10 2024-03-04T09:35:38.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:38.637+0000 7f5f16000640 1 -- 172.21.15.6:0/3467708355 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3464 (secure 0 0 0) 0x7f5f0c058960 con 0x7f5f200d4a10 2024-03-04T09:35:38.635 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:35:38.635 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:35:26.088222+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"laggy_since":"2024-03-04T09:33:54.652601+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":9}]} 2024-03-04T09:35:38.635 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 101 2024-03-04T09:35:38.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:38.641+0000 7f5f1f200640 1 -- 172.21.15.6:0/3467708355 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f5f0006b4d0 msgr2=0x7f5f0006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:38.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:38.641+0000 7f5f1f200640 1 --2- 172.21.15.6:0/3467708355 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f5f0006b4d0 0x7f5f0006d990 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f5f10006fd0 tx=0x7f5f10008040 comp rx=0 tx=0).stop 2024-03-04T09:35:38.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:38.641+0000 7f5f1f200640 1 -- 172.21.15.6:0/3467708355 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5f200d4a10 msgr2=0x7f5f200b8ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:38.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:38.641+0000 7f5f1f200640 1 --2- 172.21.15.6:0/3467708355 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5f200d4a10 0x7f5f200b8ef0 secure :-1 s=READY pgs=1731 cs=0 l=1 rev1=1 crypto rx=0x7f5f0c00c820 tx=0x7f5f0c00cce0 comp rx=0 tx=0).stop 2024-03-04T09:35:38.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:38.641+0000 7f5f1f200640 1 -- 172.21.15.6:0/3467708355 shutdown_connections 2024-03-04T09:35:38.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:38.641+0000 7f5f1f200640 1 --2- 172.21.15.6:0/3467708355 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f5f0006b4d0 0x7f5f0006d990 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:38.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:38.641+0000 7f5f1f200640 1 --2- 172.21.15.6:0/3467708355 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5f2015e0b0 0x7f5f20160620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:38.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:38.641+0000 7f5f1f200640 1 --2- 172.21.15.6:0/3467708355 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5f200d4a10 0x7f5f200b8ef0 unknown :-1 s=CLOSED pgs=1731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:38.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:38.641+0000 7f5f1f200640 1 --2- 172.21.15.6:0/3467708355 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5f200d0c80 0x7f5f2015ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:38.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:38.641+0000 7f5f1f200640 1 -- 172.21.15.6:0/3467708355 >> 172.21.15.6:0/3467708355 conn(0x7f5f200d3d30 msgr2=0x7f5f200b0b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:35:38.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:38.641+0000 7f5f1f200640 1 -- 172.21.15.6:0/3467708355 shutdown_connections 2024-03-04T09:35:38.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:38.641+0000 7f5f1f200640 1 -- 172.21.15.6:0/3467708355 wait complete. 2024-03-04T09:35:38.861 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:32:57.159681+0000', 'modified': '2024-03-04T09:35:26.088222+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8214, 'mds_1': 6796}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6796': {'gid': 6796, 'name': 'd', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.146:6837/2458070840', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 2458070840}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 2458070840}]}, 'join_fscid': -1, 'laggy_since': '2024-03-04T09:33:54.652601+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8214': {'gid': 8214, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2446627517', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2446627517}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2446627517}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8214, 'qdb_cluster': [8214]} 2024-03-04T09:35:38.862 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6796:up:stopping 2024-03-04T09:35:39.863 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:35:39.900 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-04T09:35:40.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:40.083+0000 7fb74d400640 1 Processor -- start 2024-03-04T09:35:40.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:40.083+0000 7fb74d400640 1 -- start start 2024-03-04T09:35:40.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:40.085+0000 7fb74d400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fb748144c00 0x7fb748145280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:40.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:40.085+0000 7fb74d400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fb7481457c0 0x7fb74806afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:40.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:40.085+0000 7fb74d400640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7480d0910 con 0x7fb74806b4f0 2024-03-04T09:35:40.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:40.085+0000 7fb74d400640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7480d0bf0 con 0x7fb7481457c0 2024-03-04T09:35:40.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:40.085+0000 7fb74d400640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7480d0ed0 con 0x7fb748144c00 2024-03-04T09:35:40.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:40.086+0000 7fb74ca00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fb74806b4f0 0x7fb74815f990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:47222/0 (socket says 172.21.15.6:47222) 2024-03-04T09:35:40.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:40.086+0000 7fb74ca00640 1 -- 172.21.15.6:0/2461392261 learned_addr learned my addr 172.21.15.6:0/2461392261 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:35:40.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:40.086+0000 7fb747400640 1 --2- 172.21.15.6:0/2461392261 >> v2:172.21.15.6:3300/0 conn(0x7fb7481457c0 0x7fb74806afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:40.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:40.086+0000 7fb747e00640 1 --2- 172.21.15.6:0/2461392261 >> v2:172.21.15.146:3300/0 conn(0x7fb748144c00 0x7fb748145280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:40.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:40.086+0000 7fb747400640 1 -- 172.21.15.6:0/2461392261 >> v2:172.21.15.146:3300/0 conn(0x7fb748144c00 msgr2=0x7fb748145280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:35:40.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:40.086+0000 7fb747400640 1 --2- 172.21.15.6:0/2461392261 >> v2:172.21.15.146:3300/0 conn(0x7fb748144c00 0x7fb748145280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:40.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:40.086+0000 7fb747400640 1 -- 172.21.15.6:0/2461392261 >> v1:172.21.15.6:6789/0 conn(0x7fb74806b4f0 legacy=0x7fb74815f990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:40.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:40.086+0000 7fb746a00640 1 -- 172.21.15.6:0/2461392261 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2231269003 0 0) 0x7fb7480d0910 con 0x7fb74806b4f0 2024-03-04T09:35:40.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:40.086+0000 7fb747400640 1 -- 172.21.15.6:0/2461392261 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7480d11b0 con 0x7fb7481457c0 2024-03-04T09:35:40.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:40.088+0000 7fb747400640 1 --2- 172.21.15.6:0/2461392261 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb7481457c0 0x7fb74806afb0 secure :-1 s=READY pgs=1733 cs=0 l=1 rev1=1 crypto rx=0x7fb74806bcf0 tx=0x7fb73802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b5a942b02029c34 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:35:40.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:40.089+0000 7fb746a00640 1 -- 172.21.15.6:0/2461392261 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb738035020 con 0x7fb7481457c0 2024-03-04T09:35:40.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:40.089+0000 7fb746a00640 1 -- 172.21.15.6:0/2461392261 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb738031830 con 0x7fb7481457c0 2024-03-04T09:35:40.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:40.089+0000 7fb746a00640 1 -- 172.21.15.6:0/2461392261 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb738031c00 con 0x7fb7481457c0 2024-03-04T09:35:40.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:40.089+0000 7fb74d400640 1 -- 172.21.15.6:0/2461392261 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb7481457c0 msgr2=0x7fb74806afb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:40.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:40.089+0000 7fb74d400640 1 --2- 172.21.15.6:0/2461392261 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb7481457c0 0x7fb74806afb0 secure :-1 s=READY pgs=1733 cs=0 l=1 rev1=1 crypto rx=0x7fb74806bcf0 tx=0x7fb73802b8d0 comp rx=0 tx=0).stop 2024-03-04T09:35:40.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:40.089+0000 7fb74d400640 1 -- 172.21.15.6:0/2461392261 shutdown_connections 2024-03-04T09:35:40.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:40.089+0000 7fb74d400640 1 --2- 172.21.15.6:0/2461392261 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb7481457c0 0x7fb74806afb0 unknown :-1 s=CLOSED pgs=1733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:40.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:40.089+0000 7fb74d400640 1 --2- 172.21.15.6:0/2461392261 >> v2:172.21.15.146:3300/0 conn(0x7fb748144c00 0x7fb748145280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:40.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:40.089+0000 7fb74d400640 1 -- 172.21.15.6:0/2461392261 >> 172.21.15.6:0/2461392261 conn(0x7fb74806c0c0 msgr2=0x7fb74806c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:35:40.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:40.090+0000 7fb74d400640 1 -- 172.21.15.6:0/2461392261 shutdown_connections 2024-03-04T09:35:40.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:40.090+0000 7fb74d400640 1 -- 172.21.15.6:0/2461392261 wait complete. 2024-03-04T09:35:40.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:40.091+0000 7fb74d400640 1 Processor -- start 2024-03-04T09:35:40.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:40.091+0000 7fb74d400640 1 -- start start 2024-03-04T09:35:40.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:40.092+0000 7fb74d400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb74806b4f0 0x7fb748140890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:40.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:40.092+0000 7fb74d400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb748144c00 0x7fb7480b16c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:40.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:40.092+0000 7fb74d400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb7481457c0 0x7fb7480b3110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:40.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:40.092+0000 7fb74d400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7480d19b0 con 0x7fb74806b4f0 2024-03-04T09:35:40.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:40.092+0000 7fb74d400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb748144fe0 con 0x7fb748144c00 2024-03-04T09:35:40.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:40.092+0000 7fb74d400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7480d0140 con 0x7fb7481457c0 2024-03-04T09:35:40.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:40.092+0000 7fb747e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb74806b4f0 0x7fb748140890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:40.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:40.093+0000 7fb747e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb74806b4f0 0x7fb748140890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49664/0 (socket says 172.21.15.6:49664) 2024-03-04T09:35:40.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:40.093+0000 7fb747e00640 1 -- 172.21.15.6:0/3183346598 learned_addr learned my addr 172.21.15.6:0/3183346598 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:35:40.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:40.093+0000 7fb747400640 1 --2- 172.21.15.6:0/3183346598 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb748144c00 0x7fb7480b16c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:40.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:40.093+0000 7fb747e00640 1 -- 172.21.15.6:0/3183346598 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb7481457c0 msgr2=0x7fb7480b3110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:40.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:40.093+0000 7fb747e00640 1 --2- 172.21.15.6:0/3183346598 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb7481457c0 0x7fb7480b3110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:40.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:40.093+0000 7fb747e00640 1 -- 172.21.15.6:0/3183346598 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb748144c00 msgr2=0x7fb7480b16c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:40.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:40.093+0000 7fb747e00640 1 --2- 172.21.15.6:0/3183346598 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb748144c00 0x7fb7480b16c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:40.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:40.093+0000 7fb747e00640 1 -- 172.21.15.6:0/3183346598 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb748158970 con 0x7fb74806b4f0 2024-03-04T09:35:40.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:40.093+0000 7fb747400640 1 --2- 172.21.15.6:0/3183346598 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb748144c00 0x7fb7480b16c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:35:40.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:40.093+0000 7fb747e00640 1 --2- 172.21.15.6:0/3183346598 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb74806b4f0 0x7fb748140890 secure :-1 s=READY pgs=1734 cs=0 l=1 rev1=1 crypto rx=0x7fb73000e9e0 tx=0x7fb73000eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:35:40.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:40.094+0000 7fb73fe00640 1 -- 172.21.15.6:0/3183346598 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb730017070 con 0x7fb74806b4f0 2024-03-04T09:35:40.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:40.094+0000 7fb73fe00640 1 -- 172.21.15.6:0/3183346598 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb73000c9d0 con 0x7fb74806b4f0 2024-03-04T09:35:40.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:40.094+0000 7fb73fe00640 1 -- 172.21.15.6:0/3183346598 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb73000cdc0 con 0x7fb74806b4f0 2024-03-04T09:35:40.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:40.094+0000 7fb74d400640 1 -- 172.21.15.6:0/3183346598 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7480b0900 con 0x7fb74806b4f0 2024-03-04T09:35:40.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:40.094+0000 7fb74d400640 1 -- 172.21.15.6:0/3183346598 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7480b0ec0 con 0x7fb74806b4f0 2024-03-04T09:35:40.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:40.095+0000 7fb74d400640 1 -- 172.21.15.6:0/3183346598 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb74804f330 con 0x7fb74806b4f0 2024-03-04T09:35:40.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:40.096+0000 7fb73fe00640 1 -- 172.21.15.6:0/3183346598 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb73001a070 con 0x7fb74806b4f0 2024-03-04T09:35:40.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:40.097+0000 7fb73fe00640 1 --2- 172.21.15.6:0/3183346598 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb72406b4d0 0x7fb72406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:40.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:40.097+0000 7fb73fe00640 1 -- 172.21.15.6:0/3183346598 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fb73008a5f0 con 0x7fb74806b4f0 2024-03-04T09:35:40.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:40.098+0000 7fb747400640 1 --2- 172.21.15.6:0/3183346598 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb72406b4d0 0x7fb72406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:40.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:40.099+0000 7fb747400640 1 --2- 172.21.15.6:0/3183346598 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb72406b4d0 0x7fb72406d990 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7fb738036000 tx=0x7fb738008030 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:35:40.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:40.102+0000 7fb73fe00640 1 -- 172.21.15.6:0/3183346598 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb7300594c0 con 0x7fb74806b4f0 2024-03-04T09:35:40.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:40.345+0000 7fb74d400640 1 -- 172.21.15.6:0/3183346598 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb7480722f0 con 0x7fb74806b4f0 2024-03-04T09:35:40.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:40.346+0000 7fb73fe00640 1 -- 172.21.15.6:0/3183346598 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3464 (secure 0 0 0) 0x7fb730058dc0 con 0x7fb74806b4f0 2024-03-04T09:35:40.343 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:35:40.343 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:35:26.088222+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"laggy_since":"2024-03-04T09:33:54.652601+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":9}]} 2024-03-04T09:35:40.343 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 101 2024-03-04T09:35:40.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:40.349+0000 7fb74d400640 1 -- 172.21.15.6:0/3183346598 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb72406b4d0 msgr2=0x7fb72406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:40.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:40.349+0000 7fb74d400640 1 --2- 172.21.15.6:0/3183346598 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb72406b4d0 0x7fb72406d990 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7fb738036000 tx=0x7fb738008030 comp rx=0 tx=0).stop 2024-03-04T09:35:40.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:40.349+0000 7fb74d400640 1 -- 172.21.15.6:0/3183346598 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb74806b4f0 msgr2=0x7fb748140890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:40.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:40.349+0000 7fb74d400640 1 --2- 172.21.15.6:0/3183346598 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb74806b4f0 0x7fb748140890 secure :-1 s=READY pgs=1734 cs=0 l=1 rev1=1 crypto rx=0x7fb73000e9e0 tx=0x7fb73000eea0 comp rx=0 tx=0).stop 2024-03-04T09:35:40.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:40.349+0000 7fb74d400640 1 -- 172.21.15.6:0/3183346598 shutdown_connections 2024-03-04T09:35:40.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:40.349+0000 7fb74d400640 1 --2- 172.21.15.6:0/3183346598 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb72406b4d0 0x7fb72406d990 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:40.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:40.349+0000 7fb74d400640 1 --2- 172.21.15.6:0/3183346598 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb7481457c0 0x7fb7480b3110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:40.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:40.349+0000 7fb74d400640 1 --2- 172.21.15.6:0/3183346598 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb748144c00 0x7fb7480b16c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:40.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:40.349+0000 7fb74d400640 1 --2- 172.21.15.6:0/3183346598 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb74806b4f0 0x7fb748140890 unknown :-1 s=CLOSED pgs=1734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:40.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:40.349+0000 7fb74d400640 1 -- 172.21.15.6:0/3183346598 >> 172.21.15.6:0/3183346598 conn(0x7fb74806c0c0 msgr2=0x7fb74815ece0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:35:40.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:40.349+0000 7fb74d400640 1 -- 172.21.15.6:0/3183346598 shutdown_connections 2024-03-04T09:35:40.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:40.349+0000 7fb74d400640 1 -- 172.21.15.6:0/3183346598 wait complete. 2024-03-04T09:35:40.566 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:32:57.159681+0000', 'modified': '2024-03-04T09:35:26.088222+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8214, 'mds_1': 6796}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6796': {'gid': 6796, 'name': 'd', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.146:6837/2458070840', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 2458070840}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 2458070840}]}, 'join_fscid': -1, 'laggy_since': '2024-03-04T09:33:54.652601+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8214': {'gid': 8214, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2446627517', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2446627517}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2446627517}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8214, 'qdb_cluster': [8214]} 2024-03-04T09:35:40.566 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6796:up:stopping 2024-03-04T09:35:41.436 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:35:41.445 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:35:41.567 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:35:41.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:41.793+0000 7fc57a400640 1 Processor -- start 2024-03-04T09:35:41.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:41.793+0000 7fc57a400640 1 -- start start 2024-03-04T09:35:41.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:41.795+0000 7fc57a400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc57415efc0 0x7fc5740cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:41.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:41.795+0000 7fc57a400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc5740cce90 0x7fc574161ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:41.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:41.795+0000 7fc57a400640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5741627f0 con 0x7fc574162000 2024-03-04T09:35:41.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:41.795+0000 7fc57a400640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc57415f3a0 con 0x7fc57415efc0 2024-03-04T09:35:41.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:41.795+0000 7fc57a400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc57406d190 con 0x7fc5740cce90 2024-03-04T09:35:41.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:41.795+0000 7fc579000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc57415efc0 0x7fc5740cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:41.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:41.795+0000 7fc579000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc57415efc0 0x7fc5740cc950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49670/0 (socket says 172.21.15.6:49670) 2024-03-04T09:35:41.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:41.795+0000 7fc579000640 1 -- 172.21.15.6:0/852608864 learned_addr learned my addr 172.21.15.6:0/852608864 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:35:41.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:41.796+0000 7fc573e00640 1 --2- 172.21.15.6:0/852608864 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc5740cce90 0x7fc574161ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:41.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:41.796+0000 7fc579000640 1 -- 172.21.15.6:0/852608864 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc5740cce90 msgr2=0x7fc574161ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:35:41.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:41.796+0000 7fc579000640 1 --2- 172.21.15.6:0/852608864 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc5740cce90 0x7fc574161ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:41.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:41.796+0000 7fc579000640 1 -- 172.21.15.6:0/852608864 >> v1:172.21.15.146:6789/0 conn(0x7fc574162000 legacy=0x7fc5740ad880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:35:41.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:41.796+0000 7fc579000640 1 -- 172.21.15.6:0/852608864 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc57406c860 con 0x7fc57415efc0 2024-03-04T09:35:41.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:41.796+0000 7fc573e00640 1 --2- 172.21.15.6:0/852608864 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc5740cce90 0x7fc574161ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:35:41.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:41.797+0000 7fc579000640 1 --2- 172.21.15.6:0/852608864 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc57415efc0 0x7fc5740cc950 secure :-1 s=READY pgs=1735 cs=0 l=1 rev1=1 crypto rx=0x7fc568009870 tx=0x7fc56802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebe2cd813ebcf36 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:35:41.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:41.797+0000 7fc573400640 1 -- 172.21.15.6:0/852608864 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc568035020 con 0x7fc57415efc0 2024-03-04T09:35:41.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:41.797+0000 7fc573400640 1 -- 172.21.15.6:0/852608864 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc568031640 con 0x7fc57415efc0 2024-03-04T09:35:41.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:41.797+0000 7fc573400640 1 -- 172.21.15.6:0/852608864 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc56802d040 con 0x7fc57415efc0 2024-03-04T09:35:41.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:41.798+0000 7fc57a400640 1 -- 172.21.15.6:0/852608864 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc57415efc0 msgr2=0x7fc5740cc950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:41.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:41.798+0000 7fc57a400640 1 --2- 172.21.15.6:0/852608864 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc57415efc0 0x7fc5740cc950 secure :-1 s=READY pgs=1735 cs=0 l=1 rev1=1 crypto rx=0x7fc568009870 tx=0x7fc56802b8f0 comp rx=0 tx=0).stop 2024-03-04T09:35:41.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:41.798+0000 7fc57a400640 1 -- 172.21.15.6:0/852608864 shutdown_connections 2024-03-04T09:35:41.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:41.798+0000 7fc57a400640 1 --2- 172.21.15.6:0/852608864 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc5740cce90 0x7fc574161ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:41.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:41.798+0000 7fc57a400640 1 --2- 172.21.15.6:0/852608864 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc57415efc0 0x7fc5740cc950 unknown :-1 s=CLOSED pgs=1735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:41.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:41.798+0000 7fc57a400640 1 -- 172.21.15.6:0/852608864 >> 172.21.15.6:0/852608864 conn(0x7fc57406b3d0 msgr2=0x7fc57406b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:35:41.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:41.798+0000 7fc57a400640 1 -- 172.21.15.6:0/852608864 shutdown_connections 2024-03-04T09:35:41.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:41.798+0000 7fc57a400640 1 -- 172.21.15.6:0/852608864 wait complete. 2024-03-04T09:35:41.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:41.800+0000 7fc57a400640 1 Processor -- start 2024-03-04T09:35:41.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:41.800+0000 7fc57a400640 1 -- start start 2024-03-04T09:35:41.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:41.801+0000 7fc57a400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc5740cce90 0x7fc5741f4830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:41.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:41.801+0000 7fc57a400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc57415efc0 0x7fc5740b4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:41.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:41.801+0000 7fc57a400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc574162000 0x7fc5741577a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:41.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:41.801+0000 7fc57a400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc574071c30 con 0x7fc5740cce90 2024-03-04T09:35:41.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:41.801+0000 7fc57a400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc57406ce20 con 0x7fc574162000 2024-03-04T09:35:41.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:41.801+0000 7fc57a400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc574162490 con 0x7fc57415efc0 2024-03-04T09:35:41.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:41.801+0000 7fc579000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc5740cce90 0x7fc5741f4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:41.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:41.801+0000 7fc579000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc5740cce90 0x7fc5741f4830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49684/0 (socket says 172.21.15.6:49684) 2024-03-04T09:35:41.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:41.801+0000 7fc579000640 1 -- 172.21.15.6:0/1360336292 learned_addr learned my addr 172.21.15.6:0/1360336292 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:35:41.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:41.802+0000 7fc573e00640 1 --2- 172.21.15.6:0/1360336292 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc57415efc0 0x7fc5740b4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:41.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:41.802+0000 7fc579000640 1 -- 172.21.15.6:0/1360336292 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc57415efc0 msgr2=0x7fc5740b4090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:41.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:41.802+0000 7fc579000640 1 --2- 172.21.15.6:0/1360336292 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc57415efc0 0x7fc5740b4090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:41.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:41.802+0000 7fc579000640 1 -- 172.21.15.6:0/1360336292 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc574162000 msgr2=0x7fc5741577a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:41.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:41.802+0000 7fc579000640 1 --2- 172.21.15.6:0/1360336292 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc574162000 0x7fc5741577a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:41.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:41.802+0000 7fc579000640 1 -- 172.21.15.6:0/1360336292 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc57415c270 con 0x7fc5740cce90 2024-03-04T09:35:41.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:41.802+0000 7fc579000640 1 --2- 172.21.15.6:0/1360336292 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc5740cce90 0x7fc5741f4830 secure :-1 s=READY pgs=1736 cs=0 l=1 rev1=1 crypto rx=0x7fc568000c00 tx=0x7fc5680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:35:41.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:41.802+0000 7fc570c00640 1 -- 172.21.15.6:0/1360336292 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc56803ac60 con 0x7fc5740cce90 2024-03-04T09:35:41.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:41.802+0000 7fc570c00640 1 -- 172.21.15.6:0/1360336292 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc56803ae20 con 0x7fc5740cce90 2024-03-04T09:35:41.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:41.802+0000 7fc570c00640 1 -- 172.21.15.6:0/1360336292 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc568044630 con 0x7fc5740cce90 2024-03-04T09:35:41.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:41.802+0000 7fc57a400640 1 -- 172.21.15.6:0/1360336292 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc57415c510 con 0x7fc5740cce90 2024-03-04T09:35:41.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:41.803+0000 7fc57a400640 1 -- 172.21.15.6:0/1360336292 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc57415cad0 con 0x7fc5740cce90 2024-03-04T09:35:41.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:41.804+0000 7fc57a400640 1 -- 172.21.15.6:0/1360336292 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc57404f330 con 0x7fc5740cce90 2024-03-04T09:35:41.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:41.805+0000 7fc570c00640 1 -- 172.21.15.6:0/1360336292 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc568038020 con 0x7fc5740cce90 2024-03-04T09:35:41.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:41.806+0000 7fc570c00640 1 --2- 172.21.15.6:0/1360336292 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc55406b4d0 0x7fc55406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:41.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:41.806+0000 7fc570c00640 1 -- 172.21.15.6:0/1360336292 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fc5680ac0e0 con 0x7fc5740cce90 2024-03-04T09:35:41.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:41.806+0000 7fc573e00640 1 --2- 172.21.15.6:0/1360336292 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc55406b4d0 0x7fc55406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:41.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:41.808+0000 7fc573e00640 1 --2- 172.21.15.6:0/1360336292 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc55406b4d0 0x7fc55406d990 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7fc57415da60 tx=0x7fc560008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:35:41.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:41.815+0000 7fc570c00640 1 -- 172.21.15.6:0/1360336292 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc56803f070 con 0x7fc5740cce90 2024-03-04T09:35:41.903 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-04T09:35:42.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:42.055+0000 7fc57a400640 1 -- 172.21.15.6:0/1360336292 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc5740b25b0 con 0x7fc5740cce90 2024-03-04T09:35:42.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:42.056+0000 7fc570c00640 1 -- 172.21.15.6:0/1360336292 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3464 (secure 0 0 0) 0x7fc56807afa0 con 0x7fc5740cce90 2024-03-04T09:35:42.053 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:35:42.053 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:35:26.088222+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"laggy_since":"2024-03-04T09:33:54.652601+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":9}]} 2024-03-04T09:35:42.053 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 101 2024-03-04T09:35:42.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:42.058+0000 7fc57a400640 1 -- 172.21.15.6:0/1360336292 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc55406b4d0 msgr2=0x7fc55406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:42.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:42.058+0000 7fc57a400640 1 --2- 172.21.15.6:0/1360336292 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc55406b4d0 0x7fc55406d990 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7fc57415da60 tx=0x7fc560008040 comp rx=0 tx=0).stop 2024-03-04T09:35:42.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:42.058+0000 7fc57a400640 1 -- 172.21.15.6:0/1360336292 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc5740cce90 msgr2=0x7fc5741f4830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:42.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:42.058+0000 7fc57a400640 1 --2- 172.21.15.6:0/1360336292 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc5740cce90 0x7fc5741f4830 secure :-1 s=READY pgs=1736 cs=0 l=1 rev1=1 crypto rx=0x7fc568000c00 tx=0x7fc5680046c0 comp rx=0 tx=0).stop 2024-03-04T09:35:42.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:42.058+0000 7fc57a400640 1 -- 172.21.15.6:0/1360336292 shutdown_connections 2024-03-04T09:35:42.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:42.058+0000 7fc57a400640 1 --2- 172.21.15.6:0/1360336292 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc574162000 0x7fc5741577a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:42.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:42.058+0000 7fc57a400640 1 --2- 172.21.15.6:0/1360336292 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc55406b4d0 0x7fc55406d990 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:42.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:42.058+0000 7fc57a400640 1 --2- 172.21.15.6:0/1360336292 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc57415efc0 0x7fc5740b4090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:42.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:42.058+0000 7fc57a400640 1 --2- 172.21.15.6:0/1360336292 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc5740cce90 0x7fc5741f4830 unknown :-1 s=CLOSED pgs=1736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:42.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:42.058+0000 7fc57a400640 1 -- 172.21.15.6:0/1360336292 >> 172.21.15.6:0/1360336292 conn(0x7fc57406b3d0 msgr2=0x7fc5740af750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:35:42.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:42.059+0000 7fc57a400640 1 -- 172.21.15.6:0/1360336292 shutdown_connections 2024-03-04T09:35:42.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:42.059+0000 7fc57a400640 1 -- 172.21.15.6:0/1360336292 wait complete. 2024-03-04T09:35:42.283 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:32:57.159681+0000', 'modified': '2024-03-04T09:35:26.088222+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8214, 'mds_1': 6796}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6796': {'gid': 6796, 'name': 'd', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.146:6837/2458070840', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 2458070840}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 2458070840}]}, 'join_fscid': -1, 'laggy_since': '2024-03-04T09:33:54.652601+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8214': {'gid': 8214, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2446627517', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2446627517}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2446627517}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8214, 'qdb_cluster': [8214]} 2024-03-04T09:35:42.284 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6796:up:stopping 2024-03-04T09:35:43.285 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:35:43.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:43.516+0000 7f9d03a00640 1 Processor -- start 2024-03-04T09:35:43.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:43.517+0000 7f9d03a00640 1 -- start start 2024-03-04T09:35:43.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:43.518+0000 7f9d03a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f9cfc161b40 0x7f9cfc0cfd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:43.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:43.518+0000 7f9d03a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9cfc0d2930 0x7f9cfc0b3740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:43.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:43.518+0000 7f9d03a00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cfc0cea50 con 0x7f9cfc0b5060 2024-03-04T09:35:43.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:43.518+0000 7f9d03a00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9cfc0cf900 con 0x7f9cfc161b40 2024-03-04T09:35:43.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:43.518+0000 7f9d03a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cfc0cecf0 con 0x7f9cfc0d2930 2024-03-04T09:35:43.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:43.518+0000 7f9d01000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f9cfc161b40 0x7f9cfc0cfd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:43.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:43.519+0000 7f9d01000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f9cfc161b40 0x7f9cfc0cfd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49700/0 (socket says 172.21.15.6:49700) 2024-03-04T09:35:43.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:43.519+0000 7f9d01000640 1 -- 172.21.15.6:0/1670132219 learned_addr learned my addr 172.21.15.6:0/1670132219 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:35:43.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:43.519+0000 7f9d01a00640 1 --2- 172.21.15.6:0/1670132219 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9cfc0d2930 0x7f9cfc0b3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:43.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:43.519+0000 7f9d01000640 1 -- 172.21.15.6:0/1670132219 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9cfc0d2930 msgr2=0x7f9cfc0b3740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:35:43.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:43.519+0000 7f9d01000640 1 --2- 172.21.15.6:0/1670132219 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9cfc0d2930 0x7f9cfc0b3740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:43.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:43.519+0000 7f9d01000640 1 -- 172.21.15.6:0/1670132219 >> v1:172.21.15.146:6789/0 conn(0x7f9cfc0b5060 legacy=0x7f9cfc0d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:35:43.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:43.519+0000 7f9d01000640 1 -- 172.21.15.6:0/1670132219 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cfc0cefd0 con 0x7f9cfc161b40 2024-03-04T09:35:43.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:43.519+0000 7f9d01a00640 1 --2- 172.21.15.6:0/1670132219 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9cfc0d2930 0x7f9cfc0b3740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:35:43.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:43.521+0000 7f9d01000640 1 --2- 172.21.15.6:0/1670132219 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9cfc161b40 0x7f9cfc0cfd90 secure :-1 s=READY pgs=1737 cs=0 l=1 rev1=1 crypto rx=0x7f9cf0009870 tx=0x7f9cf002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=207e879523f514b3 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:35:43.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:43.521+0000 7f9cfb400640 1 -- 172.21.15.6:0/1670132219 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cf0035020 con 0x7f9cfc161b40 2024-03-04T09:35:43.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:43.521+0000 7f9cfb400640 1 -- 172.21.15.6:0/1670132219 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cf0031640 con 0x7f9cfc161b40 2024-03-04T09:35:43.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:43.521+0000 7f9cfb400640 1 -- 172.21.15.6:0/1670132219 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cf002d040 con 0x7f9cfc161b40 2024-03-04T09:35:43.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:43.522+0000 7f9d03a00640 1 -- 172.21.15.6:0/1670132219 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9cfc161b40 msgr2=0x7f9cfc0cfd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:43.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:43.522+0000 7f9d03a00640 1 --2- 172.21.15.6:0/1670132219 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9cfc161b40 0x7f9cfc0cfd90 secure :-1 s=READY pgs=1737 cs=0 l=1 rev1=1 crypto rx=0x7f9cf0009870 tx=0x7f9cf002b8f0 comp rx=0 tx=0).stop 2024-03-04T09:35:43.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:43.522+0000 7f9d03a00640 1 -- 172.21.15.6:0/1670132219 shutdown_connections 2024-03-04T09:35:43.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:43.522+0000 7f9d03a00640 1 --2- 172.21.15.6:0/1670132219 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9cfc0d2930 0x7f9cfc0b3740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:43.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:43.522+0000 7f9d03a00640 1 --2- 172.21.15.6:0/1670132219 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9cfc161b40 0x7f9cfc0cfd90 unknown :-1 s=CLOSED pgs=1737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:43.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:43.522+0000 7f9d03a00640 1 -- 172.21.15.6:0/1670132219 >> 172.21.15.6:0/1670132219 conn(0x7f9cfc09d220 msgr2=0x7f9cfc0d3ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:35:43.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:43.522+0000 7f9d03a00640 1 -- 172.21.15.6:0/1670132219 shutdown_connections 2024-03-04T09:35:43.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:43.522+0000 7f9d03a00640 1 -- 172.21.15.6:0/1670132219 wait complete. 2024-03-04T09:35:43.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:43.524+0000 7f9d03a00640 1 Processor -- start 2024-03-04T09:35:43.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:43.524+0000 7f9d03a00640 1 -- start start 2024-03-04T09:35:43.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:43.525+0000 7f9d03a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9cfc0b5060 0x7f9cfc15b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:43.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:43.525+0000 7f9d03a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9cfc0d2930 0x7f9cfc1f73e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:43.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:43.525+0000 7f9d03a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9cfc161b40 0x7f9cfc15cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:43.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:43.525+0000 7f9d03a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cfc0b2ff0 con 0x7f9cfc0d2930 2024-03-04T09:35:43.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:43.525+0000 7f9d03a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cfc0cf590 con 0x7f9cfc0b5060 2024-03-04T09:35:43.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:43.525+0000 7f9d03a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cfc0b3170 con 0x7f9cfc161b40 2024-03-04T09:35:43.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:43.525+0000 7f9cfbe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9cfc0d2930 0x7f9cfc1f73e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:43.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:43.525+0000 7f9cfbe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9cfc0d2930 0x7f9cfc1f73e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49712/0 (socket says 172.21.15.6:49712) 2024-03-04T09:35:43.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:43.525+0000 7f9cfbe00640 1 -- 172.21.15.6:0/288068294 learned_addr learned my addr 172.21.15.6:0/288068294 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:35:43.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:43.526+0000 7f9d01a00640 1 --2- 172.21.15.6:0/288068294 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9cfc161b40 0x7f9cfc15cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:43.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:43.526+0000 7f9d01000640 1 --2- 172.21.15.6:0/288068294 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9cfc0b5060 0x7f9cfc15b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:43.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:43.526+0000 7f9cfbe00640 1 -- 172.21.15.6:0/288068294 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9cfc161b40 msgr2=0x7f9cfc15cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:43.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:43.526+0000 7f9cfbe00640 1 --2- 172.21.15.6:0/288068294 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9cfc161b40 0x7f9cfc15cd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:43.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:43.526+0000 7f9cfbe00640 1 -- 172.21.15.6:0/288068294 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9cfc0b5060 msgr2=0x7f9cfc15b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:43.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:43.526+0000 7f9cfbe00640 1 --2- 172.21.15.6:0/288068294 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9cfc0b5060 0x7f9cfc15b450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:43.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:43.526+0000 7f9cfbe00640 1 -- 172.21.15.6:0/288068294 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cfc1f7ea0 con 0x7f9cfc0d2930 2024-03-04T09:35:43.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:43.526+0000 7f9d01000640 1 --2- 172.21.15.6:0/288068294 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9cfc0b5060 0x7f9cfc15b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:35:43.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:43.526+0000 7f9d01a00640 1 --2- 172.21.15.6:0/288068294 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9cfc161b40 0x7f9cfc15cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:35:43.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:43.526+0000 7f9cfbe00640 1 --2- 172.21.15.6:0/288068294 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9cfc0d2930 0x7f9cfc1f73e0 secure :-1 s=READY pgs=1738 cs=0 l=1 rev1=1 crypto rx=0x7f9ce800c430 tx=0x7f9ce800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:35:43.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:43.526+0000 7f9cf8c00640 1 -- 172.21.15.6:0/288068294 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ce8011790 con 0x7f9cfc0d2930 2024-03-04T09:35:43.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:43.527+0000 7f9cf8c00640 1 -- 172.21.15.6:0/288068294 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ce8017070 con 0x7f9cfc0d2930 2024-03-04T09:35:43.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:43.527+0000 7f9cf8c00640 1 -- 172.21.15.6:0/288068294 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ce8011b70 con 0x7f9cfc0d2930 2024-03-04T09:35:43.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:43.527+0000 7f9d03a00640 1 -- 172.21.15.6:0/288068294 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cfc1f8180 con 0x7f9cfc0d2930 2024-03-04T09:35:43.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:43.527+0000 7f9d03a00640 1 -- 172.21.15.6:0/288068294 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cfc0cc960 con 0x7f9cfc0d2930 2024-03-04T09:35:43.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:43.528+0000 7f9d03a00640 1 -- 172.21.15.6:0/288068294 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cfc0ccb60 con 0x7f9cfc0d2930 2024-03-04T09:35:43.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:43.529+0000 7f9cf8c00640 1 -- 172.21.15.6:0/288068294 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9ce8011d10 con 0x7f9cfc0d2930 2024-03-04T09:35:43.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:43.530+0000 7f9cf8c00640 1 --2- 172.21.15.6:0/288068294 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f9cdc06b500 0x7f9cdc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:43.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:43.530+0000 7f9cf8c00640 1 -- 172.21.15.6:0/288068294 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f9ce808a120 con 0x7f9cfc0d2930 2024-03-04T09:35:43.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:43.531+0000 7f9d01000640 1 --2- 172.21.15.6:0/288068294 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f9cdc06b500 0x7f9cdc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:43.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:43.532+0000 7f9d01000640 1 --2- 172.21.15.6:0/288068294 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f9cdc06b500 0x7f9cdc06d9c0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f9cf0036000 tx=0x7f9cf0008030 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:35:43.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:43.535+0000 7f9cf8c00640 1 -- 172.21.15.6:0/288068294 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9cfc0ccb60 con 0x7f9cfc0d2930 2024-03-04T09:35:43.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:43.806+0000 7f9d03a00640 1 -- 172.21.15.6:0/288068294 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9cfc0d1df0 con 0x7f9cfc0d2930 2024-03-04T09:35:43.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:43.807+0000 7f9cf8c00640 1 -- 172.21.15.6:0/288068294 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3464 (secure 0 0 0) 0x7f9ce8058ec0 con 0x7f9cfc0d2930 2024-03-04T09:35:43.804 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:35:43.807 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:35:26.088222+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"laggy_since":"2024-03-04T09:33:54.652601+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":9}]} 2024-03-04T09:35:43.808 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 101 2024-03-04T09:35:43.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:43.809+0000 7f9d03a00640 1 -- 172.21.15.6:0/288068294 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f9cdc06b500 msgr2=0x7f9cdc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:43.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:43.809+0000 7f9d03a00640 1 --2- 172.21.15.6:0/288068294 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f9cdc06b500 0x7f9cdc06d9c0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f9cf0036000 tx=0x7f9cf0008030 comp rx=0 tx=0).stop 2024-03-04T09:35:43.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:43.809+0000 7f9d03a00640 1 -- 172.21.15.6:0/288068294 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9cfc0d2930 msgr2=0x7f9cfc1f73e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:43.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:43.809+0000 7f9d03a00640 1 --2- 172.21.15.6:0/288068294 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9cfc0d2930 0x7f9cfc1f73e0 secure :-1 s=READY pgs=1738 cs=0 l=1 rev1=1 crypto rx=0x7f9ce800c430 tx=0x7f9ce800c8f0 comp rx=0 tx=0).stop 2024-03-04T09:35:43.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:43.809+0000 7f9d03a00640 1 -- 172.21.15.6:0/288068294 shutdown_connections 2024-03-04T09:35:43.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:43.809+0000 7f9d03a00640 1 --2- 172.21.15.6:0/288068294 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f9cdc06b500 0x7f9cdc06d9c0 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:43.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:43.809+0000 7f9d03a00640 1 --2- 172.21.15.6:0/288068294 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9cfc161b40 0x7f9cfc15cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:43.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:43.809+0000 7f9d03a00640 1 --2- 172.21.15.6:0/288068294 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9cfc0d2930 0x7f9cfc1f73e0 unknown :-1 s=CLOSED pgs=1738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:43.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:43.810+0000 7f9d03a00640 1 --2- 172.21.15.6:0/288068294 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9cfc0b5060 0x7f9cfc15b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:43.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:43.810+0000 7f9d03a00640 1 -- 172.21.15.6:0/288068294 >> 172.21.15.6:0/288068294 conn(0x7f9cfc09d220 msgr2=0x7f9cfc0d3d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:35:43.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:43.810+0000 7f9d03a00640 1 -- 172.21.15.6:0/288068294 shutdown_connections 2024-03-04T09:35:43.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:43.810+0000 7f9d03a00640 1 -- 172.21.15.6:0/288068294 wait complete. 2024-03-04T09:35:43.904 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-04T09:35:44.028 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:32:57.159681+0000', 'modified': '2024-03-04T09:35:26.088222+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8214, 'mds_1': 6796}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6796': {'gid': 6796, 'name': 'd', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.146:6837/2458070840', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 2458070840}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 2458070840}]}, 'join_fscid': -1, 'laggy_since': '2024-03-04T09:33:54.652601+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8214': {'gid': 8214, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2446627517', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2446627517}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2446627517}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8214, 'qdb_cluster': [8214]} 2024-03-04T09:35:44.029 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6796:up:stopping 2024-03-04T09:35:45.030 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:35:45.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:45.265+0000 7faa8e400640 1 Processor -- start 2024-03-04T09:35:45.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:45.265+0000 7faa8e400640 1 -- start start 2024-03-04T09:35:45.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:45.267+0000 7faa8e400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faa8815e070 0x7faa880d2960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:45.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:45.267+0000 7faa8e400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7faa880cf410 0x7faa880cf870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:45.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:45.267+0000 7faa8e400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7faa880d05c0 0x7faa880d0a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:45.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:45.267+0000 7faa8e400640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa880d18c0 con 0x7faa880cf410 2024-03-04T09:35:45.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:45.267+0000 7faa8e400640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa880d1ba0 con 0x7faa880d05c0 2024-03-04T09:35:45.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:45.267+0000 7faa8e400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa880d1e80 con 0x7faa8815e070 2024-03-04T09:35:45.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:45.267+0000 7faa87e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7faa880cf410 0x7faa880cf870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:45.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:45.267+0000 7faa87e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7faa880cf410 0x7faa880cf870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49722/0 (socket says 172.21.15.6:49722) 2024-03-04T09:35:45.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:45.267+0000 7faa87e00640 1 -- 172.21.15.6:0/259324261 learned_addr learned my addr 172.21.15.6:0/259324261 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:35:45.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:45.268+0000 7faa8da00640 1 --2- 172.21.15.6:0/259324261 >> v2:172.21.15.146:3300/0 conn(0x7faa880d05c0 0x7faa880d0a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:45.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:45.268+0000 7faa8d000640 1 --2- 172.21.15.6:0/259324261 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faa8815e070 0x7faa880d2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:45.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:45.268+0000 7faa87e00640 1 -- 172.21.15.6:0/259324261 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faa8815e070 msgr2=0x7faa880d2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:35:45.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:45.268+0000 7faa87e00640 1 --2- 172.21.15.6:0/259324261 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faa8815e070 0x7faa880d2960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:45.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:45.268+0000 7faa87e00640 1 -- 172.21.15.6:0/259324261 >> v2:172.21.15.146:3300/0 conn(0x7faa880d05c0 msgr2=0x7faa880d0a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:35:45.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:45.268+0000 7faa87e00640 1 --2- 172.21.15.6:0/259324261 >> v2:172.21.15.146:3300/0 conn(0x7faa880d05c0 0x7faa880d0a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:45.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:45.268+0000 7faa87e00640 1 -- 172.21.15.6:0/259324261 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa880d2160 con 0x7faa880cf410 2024-03-04T09:35:45.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:45.269+0000 7faa87e00640 1 --2- 172.21.15.6:0/259324261 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faa880cf410 0x7faa880cf870 secure :-1 s=READY pgs=1739 cs=0 l=1 rev1=1 crypto rx=0x7faa880d3bc0 tx=0x7faa7002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d91477c0394d67a server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:35:45.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:45.269+0000 7faa87400640 1 -- 172.21.15.6:0/259324261 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa70035020 con 0x7faa880cf410 2024-03-04T09:35:45.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:45.270+0000 7faa87400640 1 -- 172.21.15.6:0/259324261 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa70031870 con 0x7faa880cf410 2024-03-04T09:35:45.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:45.270+0000 7faa87400640 1 -- 172.21.15.6:0/259324261 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa70031c40 con 0x7faa880cf410 2024-03-04T09:35:45.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:45.270+0000 7faa8e400640 1 -- 172.21.15.6:0/259324261 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faa880cf410 msgr2=0x7faa880cf870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:45.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:45.270+0000 7faa8e400640 1 --2- 172.21.15.6:0/259324261 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faa880cf410 0x7faa880cf870 secure :-1 s=READY pgs=1739 cs=0 l=1 rev1=1 crypto rx=0x7faa880d3bc0 tx=0x7faa7002b910 comp rx=0 tx=0).stop 2024-03-04T09:35:45.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:45.270+0000 7faa8e400640 1 -- 172.21.15.6:0/259324261 shutdown_connections 2024-03-04T09:35:45.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:45.270+0000 7faa8e400640 1 --2- 172.21.15.6:0/259324261 >> v2:172.21.15.146:3300/0 conn(0x7faa880d05c0 0x7faa880d0a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:45.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:45.270+0000 7faa8e400640 1 --2- 172.21.15.6:0/259324261 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faa880cf410 0x7faa880cf870 unknown :-1 s=CLOSED pgs=1739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:45.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:45.270+0000 7faa8e400640 1 --2- 172.21.15.6:0/259324261 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faa8815e070 0x7faa880d2960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:45.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:45.270+0000 7faa8e400640 1 -- 172.21.15.6:0/259324261 >> 172.21.15.6:0/259324261 conn(0x7faa8808d2a0 msgr2=0x7faa880d3cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:35:45.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:45.271+0000 7faa8e400640 1 -- 172.21.15.6:0/259324261 shutdown_connections 2024-03-04T09:35:45.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:45.271+0000 7faa8e400640 1 -- 172.21.15.6:0/259324261 wait complete. 2024-03-04T09:35:45.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:45.272+0000 7faa8e400640 1 Processor -- start 2024-03-04T09:35:45.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:45.272+0000 7faa8e400640 1 -- start start 2024-03-04T09:35:45.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:45.273+0000 7faa8e400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faa880cf410 0x7faa88154060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:45.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:45.273+0000 7faa8e400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faa880d05c0 0x7faa880cd700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:45.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:45.273+0000 7faa8e400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faa8815e070 0x7faa880c91b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:45.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:45.273+0000 7faa8e400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa880d4730 con 0x7faa880cf410 2024-03-04T09:35:45.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:45.273+0000 7faa8e400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa880d45b0 con 0x7faa8815e070 2024-03-04T09:35:45.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:45.273+0000 7faa8e400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa880b3630 con 0x7faa880d05c0 2024-03-04T09:35:45.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:45.274+0000 7faa8d000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faa880cf410 0x7faa88154060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:45.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:45.274+0000 7faa8d000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faa880cf410 0x7faa88154060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49734/0 (socket says 172.21.15.6:49734) 2024-03-04T09:35:45.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:45.274+0000 7faa8d000640 1 -- 172.21.15.6:0/2641062725 learned_addr learned my addr 172.21.15.6:0/2641062725 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:35:45.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:45.274+0000 7faa87e00640 1 --2- 172.21.15.6:0/2641062725 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faa880d05c0 0x7faa880cd700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:45.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:45.274+0000 7faa8d000640 1 -- 172.21.15.6:0/2641062725 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faa880d05c0 msgr2=0x7faa880cd700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:45.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:45.274+0000 7faa8d000640 1 --2- 172.21.15.6:0/2641062725 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faa880d05c0 0x7faa880cd700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:45.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:45.274+0000 7faa8d000640 1 -- 172.21.15.6:0/2641062725 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faa8815e070 msgr2=0x7faa880c91b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:45.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:45.274+0000 7faa8da00640 1 --2- 172.21.15.6:0/2641062725 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faa8815e070 0x7faa880c91b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:45.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:45.274+0000 7faa8d000640 1 --2- 172.21.15.6:0/2641062725 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faa8815e070 0x7faa880c91b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:45.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:45.274+0000 7faa8d000640 1 -- 172.21.15.6:0/2641062725 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa880caf80 con 0x7faa880cf410 2024-03-04T09:35:45.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:45.274+0000 7faa87e00640 1 --2- 172.21.15.6:0/2641062725 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faa880d05c0 0x7faa880cd700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:35:45.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:45.275+0000 7faa8d000640 1 --2- 172.21.15.6:0/2641062725 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faa880cf410 0x7faa88154060 secure :-1 s=READY pgs=1740 cs=0 l=1 rev1=1 crypto rx=0x7faa7800d5f0 tx=0x7faa7800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:35:45.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:45.275+0000 7faa84c00640 1 -- 172.21.15.6:0/2641062725 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa780125f0 con 0x7faa880cf410 2024-03-04T09:35:45.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:45.275+0000 7faa84c00640 1 -- 172.21.15.6:0/2641062725 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa78019070 con 0x7faa880cf410 2024-03-04T09:35:45.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:45.275+0000 7faa84c00640 1 -- 172.21.15.6:0/2641062725 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa780129d0 con 0x7faa880cf410 2024-03-04T09:35:45.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:45.275+0000 7faa8e400640 1 -- 172.21.15.6:0/2641062725 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa8815d580 con 0x7faa880cf410 2024-03-04T09:35:45.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:45.275+0000 7faa8e400640 1 -- 172.21.15.6:0/2641062725 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa880ca4a0 con 0x7faa880cf410 2024-03-04T09:35:45.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:45.276+0000 7faa8e400640 1 -- 172.21.15.6:0/2641062725 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa8804f330 con 0x7faa880cf410 2024-03-04T09:35:45.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:45.277+0000 7faa84c00640 1 -- 172.21.15.6:0/2641062725 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7faa78012b70 con 0x7faa880cf410 2024-03-04T09:35:45.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:45.279+0000 7faa84c00640 1 --2- 172.21.15.6:0/2641062725 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7faa6406b500 0x7faa6406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:45.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:45.279+0000 7faa84c00640 1 -- 172.21.15.6:0/2641062725 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7faa7808a6a0 con 0x7faa880cf410 2024-03-04T09:35:45.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:45.279+0000 7faa87e00640 1 --2- 172.21.15.6:0/2641062725 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7faa6406b500 0x7faa6406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:45.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:45.281+0000 7faa87e00640 1 --2- 172.21.15.6:0/2641062725 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7faa6406b500 0x7faa6406d9c0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7faa70036000 tx=0x7faa7003c040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:35:45.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:45.284+0000 7faa84c00640 1 -- 172.21.15.6:0/2641062725 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7faa78059440 con 0x7faa880cf410 2024-03-04T09:35:45.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:45.544+0000 7faa8e400640 1 -- 172.21.15.6:0/2641062725 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faa880c75c0 con 0x7faa880cf410 2024-03-04T09:35:45.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:45.544+0000 7faa84c00640 1 -- 172.21.15.6:0/2641062725 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3464 (secure 0 0 0) 0x7faa78058d40 con 0x7faa880cf410 2024-03-04T09:35:45.541 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:35:45.542 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:35:26.088222+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"laggy_since":"2024-03-04T09:33:54.652601+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":9}]} 2024-03-04T09:35:45.542 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 101 2024-03-04T09:35:45.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:45.547+0000 7faa8e400640 1 -- 172.21.15.6:0/2641062725 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7faa6406b500 msgr2=0x7faa6406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:45.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:45.547+0000 7faa8e400640 1 --2- 172.21.15.6:0/2641062725 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7faa6406b500 0x7faa6406d9c0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7faa70036000 tx=0x7faa7003c040 comp rx=0 tx=0).stop 2024-03-04T09:35:45.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:45.547+0000 7faa8e400640 1 -- 172.21.15.6:0/2641062725 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faa880cf410 msgr2=0x7faa88154060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:45.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:45.547+0000 7faa8e400640 1 --2- 172.21.15.6:0/2641062725 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faa880cf410 0x7faa88154060 secure :-1 s=READY pgs=1740 cs=0 l=1 rev1=1 crypto rx=0x7faa7800d5f0 tx=0x7faa7800dab0 comp rx=0 tx=0).stop 2024-03-04T09:35:45.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:45.547+0000 7faa8e400640 1 -- 172.21.15.6:0/2641062725 shutdown_connections 2024-03-04T09:35:45.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:45.547+0000 7faa8e400640 1 --2- 172.21.15.6:0/2641062725 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faa8815e070 0x7faa880c91b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:45.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:45.547+0000 7faa8e400640 1 --2- 172.21.15.6:0/2641062725 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7faa6406b500 0x7faa6406d9c0 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:45.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:45.547+0000 7faa8e400640 1 --2- 172.21.15.6:0/2641062725 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faa880d05c0 0x7faa880cd700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:45.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:45.547+0000 7faa8e400640 1 --2- 172.21.15.6:0/2641062725 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faa880cf410 0x7faa88154060 unknown :-1 s=CLOSED pgs=1740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:45.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:45.547+0000 7faa8e400640 1 -- 172.21.15.6:0/2641062725 >> 172.21.15.6:0/2641062725 conn(0x7faa8808d2a0 msgr2=0x7faa880d2960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:35:45.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:45.547+0000 7faa8e400640 1 -- 172.21.15.6:0/2641062725 shutdown_connections 2024-03-04T09:35:45.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:45.547+0000 7faa8e400640 1 -- 172.21.15.6:0/2641062725 wait complete. 2024-03-04T09:35:45.768 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:32:57.159681+0000', 'modified': '2024-03-04T09:35:26.088222+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8214, 'mds_1': 6796}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6796': {'gid': 6796, 'name': 'd', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.146:6837/2458070840', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 2458070840}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 2458070840}]}, 'join_fscid': -1, 'laggy_since': '2024-03-04T09:33:54.652601+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8214': {'gid': 8214, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2446627517', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2446627517}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2446627517}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8214, 'qdb_cluster': [8214]} 2024-03-04T09:35:45.768 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6796:up:stopping 2024-03-04T09:35:45.906 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-04T09:35:46.769 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:35:46.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:46.994+0000 7f3843200640 1 Processor -- start 2024-03-04T09:35:46.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:46.994+0000 7f3843200640 1 -- start start 2024-03-04T09:35:46.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:46.995+0000 7f3843200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f38440d0e50 0x7f3844168c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:46.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:46.995+0000 7f3843200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38440b3710 con 0x7f38440d4a10 2024-03-04T09:35:46.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:46.995+0000 7f3843200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38440b39f0 con 0x7f384415e0b0 2024-03-04T09:35:46.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:46.995+0000 7f3843200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38440b09e0 con 0x7f38440d0e50 2024-03-04T09:35:46.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:46.995+0000 7f3841400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f38440d4a10 0x7f38440d0740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:47228/0 (socket says 172.21.15.6:47228) 2024-03-04T09:35:46.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:46.996+0000 7f3841400640 1 -- 172.21.15.6:0/3236971558 learned_addr learned my addr 172.21.15.6:0/3236971558 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:35:46.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:46.996+0000 7f3840a00640 1 -- 172.21.15.6:0/3236971558 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1062453979 0 0) 0x7f38440b3710 con 0x7f38440d4a10 2024-03-04T09:35:46.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:46.996+0000 7f3840a00640 1 -- 172.21.15.6:0/3236971558 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3828003520 con 0x7f38440d4a10 2024-03-04T09:35:46.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:46.996+0000 7f3842800640 1 --2- 172.21.15.6:0/3236971558 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f38440d0e50 0x7f3844168c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:46.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:46.996+0000 7f3840a00640 1 -- 172.21.15.6:0/3236971558 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 449874490 0 0) 0x7f3828003520 con 0x7f38440d4a10 2024-03-04T09:35:46.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:46.996+0000 7f3840a00640 1 -- 172.21.15.6:0/3236971558 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f38440d0e50 msgr2=0x7f3844168c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:35:46.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:46.996+0000 7f3840a00640 1 --2- 172.21.15.6:0/3236971558 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f38440d0e50 0x7f3844168c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:46.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:46.996+0000 7f3840a00640 1 -- 172.21.15.6:0/3236971558 >> v1:172.21.15.146:6789/0 conn(0x7f384415e0b0 legacy=0x7f38440cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:46.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:46.996+0000 7f3840a00640 1 -- 172.21.15.6:0/3236971558 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38440b3cd0 con 0x7f38440d4a10 2024-03-04T09:35:46.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:46.996+0000 7f3840a00640 1 -- 172.21.15.6:0/3236971558 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7f382c001690 con 0x7f38440d4a10 2024-03-04T09:35:46.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:46.996+0000 7f3840a00640 1 -- 172.21.15.6:0/3236971558 >> v1:172.21.15.6:6789/0 conn(0x7f38440d4a10 legacy=0x7f38440d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:46.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:46.996+0000 7f3840a00640 1 --2- 172.21.15.6:0/3236971558 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f38280039e0 0x7f3828003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:46.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:46.997+0000 7f3840a00640 1 --2- 172.21.15.6:0/3236971558 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3828004800 0x7f3828004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:46.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:46.997+0000 7f3840a00640 1 --2- 172.21.15.6:0/3236971558 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f38440d0e50 0x7f3828005250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:46.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:46.997+0000 7f3840a00640 1 -- 172.21.15.6:0/3236971558 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f382800bb50 con 0x7f3828004800 2024-03-04T09:35:46.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:46.997+0000 7f3840a00640 1 -- 172.21.15.6:0/3236971558 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f382800bd00 con 0x7f38280039e0 2024-03-04T09:35:46.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:46.997+0000 7f3840a00640 1 -- 172.21.15.6:0/3236971558 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f382800bf00 con 0x7f38440d0e50 2024-03-04T09:35:46.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:46.997+0000 7f3841400640 1 --2- 172.21.15.6:0/3236971558 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3828004800 0x7f3828004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:46.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:46.997+0000 7f3841e00640 1 --2- 172.21.15.6:0/3236971558 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f38280039e0 0x7f3828003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:46.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:46.997+0000 7f3841400640 1 -- 172.21.15.6:0/3236971558 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f38440d0e50 msgr2=0x7f3828005250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:35:46.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:46.997+0000 7f3841400640 1 --2- 172.21.15.6:0/3236971558 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f38440d0e50 0x7f3828005250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:46.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:46.997+0000 7f3841400640 1 -- 172.21.15.6:0/3236971558 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f38280039e0 msgr2=0x7f3828003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:35:46.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:46.997+0000 7f3841400640 1 --2- 172.21.15.6:0/3236971558 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f38280039e0 0x7f3828003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:46.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:46.997+0000 7f3841400640 1 -- 172.21.15.6:0/3236971558 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3828003520 con 0x7f3828004800 2024-03-04T09:35:46.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:46.997+0000 7f3841e00640 1 --2- 172.21.15.6:0/3236971558 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f38280039e0 0x7f3828003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:35:46.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:46.998+0000 7f3841400640 1 --2- 172.21.15.6:0/3236971558 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3828004800 0x7f3828004c80 secure :-1 s=READY pgs=1742 cs=0 l=1 rev1=1 crypto rx=0x7f38440d4e30 tx=0x7f3828005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46860980799ccf95 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:35:46.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:46.999+0000 7f3840a00640 1 -- 172.21.15.6:0/3236971558 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f382c007c80 con 0x7f3828004800 2024-03-04T09:35:46.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:46.999+0000 7f3840a00640 1 -- 172.21.15.6:0/3236971558 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f382c030070 con 0x7f3828004800 2024-03-04T09:35:46.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:46.999+0000 7f3840a00640 1 -- 172.21.15.6:0/3236971558 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f382c005cd0 con 0x7f3828004800 2024-03-04T09:35:46.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:46.999+0000 7f3843200640 1 -- 172.21.15.6:0/3236971558 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3828004800 msgr2=0x7f3828004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:46.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:46.999+0000 7f3843200640 1 --2- 172.21.15.6:0/3236971558 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3828004800 0x7f3828004c80 secure :-1 s=READY pgs=1742 cs=0 l=1 rev1=1 crypto rx=0x7f38440d4e30 tx=0x7f3828005830 comp rx=0 tx=0).stop 2024-03-04T09:35:46.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:46.999+0000 7f3842800640 1 -- 172.21.15.6:0/3236971558 reap_dead start 2024-03-04T09:35:46.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:46.999+0000 7f3843200640 1 -- 172.21.15.6:0/3236971558 shutdown_connections 2024-03-04T09:35:46.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:46.999+0000 7f3843200640 1 -- 172.21.15.6:0/3236971558 >> 172.21.15.6:0/3236971558 conn(0x7f38440d3d30 msgr2=0x7f38440d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:35:46.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:46.999+0000 7f3843200640 1 -- 172.21.15.6:0/3236971558 shutdown_connections 2024-03-04T09:35:46.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:46.999+0000 7f3843200640 1 -- 172.21.15.6:0/3236971558 wait complete. 2024-03-04T09:35:46.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:47.002+0000 7f3843200640 1 Processor -- start 2024-03-04T09:35:46.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:47.002+0000 7f3843200640 1 -- start start 2024-03-04T09:35:47.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:47.004+0000 7f3843200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3828004800 0x7f3844150390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:47.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:47.004+0000 7f3843200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3844160ee0 0x7f384415f850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:47.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:47.004+0000 7f3843200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3844161360 0x7f38440cce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:47.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:47.004+0000 7f3843200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f384415e490 con 0x7f3828004800 2024-03-04T09:35:47.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:47.004+0000 7f3843200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38440cfa20 con 0x7f3844160ee0 2024-03-04T09:35:47.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:47.004+0000 7f3843200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38440cfc60 con 0x7f3844161360 2024-03-04T09:35:47.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:47.004+0000 7f3841e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3828004800 0x7f3844150390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:47.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:47.005+0000 7f3841e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3828004800 0x7f3844150390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49752/0 (socket says 172.21.15.6:49752) 2024-03-04T09:35:47.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:47.005+0000 7f3841e00640 1 -- 172.21.15.6:0/3237413294 learned_addr learned my addr 172.21.15.6:0/3237413294 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:35:47.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:47.005+0000 7f3841400640 1 --2- 172.21.15.6:0/3237413294 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3844160ee0 0x7f384415f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:47.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:47.005+0000 7f3842800640 1 --2- 172.21.15.6:0/3237413294 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3844161360 0x7f38440cce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:47.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:47.005+0000 7f3841e00640 1 -- 172.21.15.6:0/3237413294 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3844161360 msgr2=0x7f38440cce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:47.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:47.005+0000 7f3841e00640 1 --2- 172.21.15.6:0/3237413294 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3844161360 0x7f38440cce50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:47.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:47.005+0000 7f3841e00640 1 -- 172.21.15.6:0/3237413294 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3844160ee0 msgr2=0x7f384415f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:47.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:47.005+0000 7f3841e00640 1 --2- 172.21.15.6:0/3237413294 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3844160ee0 0x7f384415f850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:47.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:47.005+0000 7f3841e00640 1 -- 172.21.15.6:0/3237413294 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f384415e7b0 con 0x7f3828004800 2024-03-04T09:35:47.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:47.005+0000 7f3842800640 1 --2- 172.21.15.6:0/3237413294 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3844161360 0x7f38440cce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:35:47.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:47.005+0000 7f3841e00640 1 --2- 172.21.15.6:0/3237413294 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3828004800 0x7f3844150390 secure :-1 s=READY pgs=1743 cs=0 l=1 rev1=1 crypto rx=0x7f383400db40 tx=0x7f38340020c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:35:47.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:47.006+0000 7f383a000640 1 -- 172.21.15.6:0/3237413294 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3834003ba0 con 0x7f3828004800 2024-03-04T09:35:47.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:47.006+0000 7f383a000640 1 -- 172.21.15.6:0/3237413294 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3834003d40 con 0x7f3828004800 2024-03-04T09:35:47.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:47.006+0000 7f383a000640 1 -- 172.21.15.6:0/3237413294 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3834006d70 con 0x7f3828004800 2024-03-04T09:35:47.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:47.006+0000 7f3843200640 1 -- 172.21.15.6:0/3237413294 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f384415ea90 con 0x7f3828004800 2024-03-04T09:35:47.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:47.006+0000 7f3843200640 1 -- 172.21.15.6:0/3237413294 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f384415f050 con 0x7f3828004800 2024-03-04T09:35:47.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:47.007+0000 7f3843200640 1 -- 172.21.15.6:0/3237413294 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f384404f330 con 0x7f3828004800 2024-03-04T09:35:47.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:47.008+0000 7f383a000640 1 -- 172.21.15.6:0/3237413294 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f383401a030 con 0x7f3828004800 2024-03-04T09:35:47.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:47.009+0000 7f383a000640 1 --2- 172.21.15.6:0/3237413294 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f382006b500 0x7f382006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:47.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:47.010+0000 7f383a000640 1 -- 172.21.15.6:0/3237413294 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f3834089ca0 con 0x7f3828004800 2024-03-04T09:35:47.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:47.010+0000 7f3841400640 1 --2- 172.21.15.6:0/3237413294 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f382006b500 0x7f382006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:47.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:47.011+0000 7f3841400640 1 --2- 172.21.15.6:0/3237413294 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f382006b500 0x7f382006d9c0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f382c034000 tx=0x7f382c009030 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:35:47.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:47.014+0000 7f383a000640 1 -- 172.21.15.6:0/3237413294 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3834021020 con 0x7f3828004800 2024-03-04T09:35:47.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:47.261+0000 7f3843200640 1 -- 172.21.15.6:0/3237413294 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f38440b09e0 con 0x7f3828004800 2024-03-04T09:35:47.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:47.262+0000 7f383a000640 1 -- 172.21.15.6:0/3237413294 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3464 (secure 0 0 0) 0x7f3834058ad0 con 0x7f3828004800 2024-03-04T09:35:47.260 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:35:47.260 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:35:26.088222+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"laggy_since":"2024-03-04T09:33:54.652601+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":9}]} 2024-03-04T09:35:47.260 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 101 2024-03-04T09:35:47.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:47.265+0000 7f3843200640 1 -- 172.21.15.6:0/3237413294 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f382006b500 msgr2=0x7f382006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:47.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:47.265+0000 7f3843200640 1 --2- 172.21.15.6:0/3237413294 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f382006b500 0x7f382006d9c0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f382c034000 tx=0x7f382c009030 comp rx=0 tx=0).stop 2024-03-04T09:35:47.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:47.265+0000 7f3843200640 1 -- 172.21.15.6:0/3237413294 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3828004800 msgr2=0x7f3844150390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:47.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:47.265+0000 7f3843200640 1 --2- 172.21.15.6:0/3237413294 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3828004800 0x7f3844150390 secure :-1 s=READY pgs=1743 cs=0 l=1 rev1=1 crypto rx=0x7f383400db40 tx=0x7f38340020c0 comp rx=0 tx=0).stop 2024-03-04T09:35:47.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:47.266+0000 7f3843200640 1 -- 172.21.15.6:0/3237413294 shutdown_connections 2024-03-04T09:35:47.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:47.266+0000 7f3843200640 1 --2- 172.21.15.6:0/3237413294 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f382006b500 0x7f382006d9c0 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:47.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:47.266+0000 7f3843200640 1 --2- 172.21.15.6:0/3237413294 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3844161360 0x7f38440cce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:47.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:47.266+0000 7f3843200640 1 --2- 172.21.15.6:0/3237413294 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3844160ee0 0x7f384415f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:47.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:47.266+0000 7f3843200640 1 --2- 172.21.15.6:0/3237413294 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3828004800 0x7f3844150390 unknown :-1 s=CLOSED pgs=1743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:47.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:47.266+0000 7f3843200640 1 -- 172.21.15.6:0/3237413294 >> 172.21.15.6:0/3237413294 conn(0x7f38440d3d30 msgr2=0x7f38440d0740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:35:47.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:47.266+0000 7f3843200640 1 -- 172.21.15.6:0/3237413294 shutdown_connections 2024-03-04T09:35:47.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:47.266+0000 7f3843200640 1 -- 172.21.15.6:0/3237413294 wait complete. 2024-03-04T09:35:47.486 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:32:57.159681+0000', 'modified': '2024-03-04T09:35:26.088222+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8214, 'mds_1': 6796}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6796': {'gid': 6796, 'name': 'd', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.146:6837/2458070840', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 2458070840}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 2458070840}]}, 'join_fscid': -1, 'laggy_since': '2024-03-04T09:33:54.652601+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8214': {'gid': 8214, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2446627517', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2446627517}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2446627517}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8214, 'qdb_cluster': [8214]} 2024-03-04T09:35:47.486 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6796:up:stopping 2024-03-04T09:35:47.908 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-04T09:35:48.486 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:35:48.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:48.715+0000 7fb1d3200640 1 Processor -- start 2024-03-04T09:35:48.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:48.715+0000 7fb1d3200640 1 -- start start 2024-03-04T09:35:48.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:48.716+0000 7fb1d3200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb1cc0d4a10 0x7fb1cc0d0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:48.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:48.716+0000 7fb1d3200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1cc0b3710 con 0x7fb1cc0d0c80 2024-03-04T09:35:48.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:48.717+0000 7fb1d3200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1cc0b39f0 con 0x7fb1cc15e0b0 2024-03-04T09:35:48.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:48.717+0000 7fb1d3200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1cc0b09e0 con 0x7fb1cc0d4a10 2024-03-04T09:35:48.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:48.717+0000 7fb1d2800640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fb1cc0d0c80 0x7fb1cc0b44d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:47238/0 (socket says 172.21.15.6:47238) 2024-03-04T09:35:48.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:48.717+0000 7fb1d2800640 1 -- 172.21.15.6:0/61498642 learned_addr learned my addr 172.21.15.6:0/61498642 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:35:48.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:48.717+0000 7fb1d1400640 1 --2- 172.21.15.6:0/61498642 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb1cc0d4a10 0x7fb1cc0d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:48.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:48.717+0000 7fb1d0a00640 1 -- 172.21.15.6:0/61498642 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1430009196 0 0) 0x7fb1cc0b3710 con 0x7fb1cc0d0c80 2024-03-04T09:35:48.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:48.717+0000 7fb1d0a00640 1 -- 172.21.15.6:0/61498642 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1b4003520 con 0x7fb1cc0d0c80 2024-03-04T09:35:48.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:48.718+0000 7fb1d0a00640 1 -- 172.21.15.6:0/61498642 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2263664416 0 0) 0x7fb1b4003520 con 0x7fb1cc0d0c80 2024-03-04T09:35:48.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:48.718+0000 7fb1d0a00640 1 -- 172.21.15.6:0/61498642 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb1cc0d4a10 msgr2=0x7fb1cc0d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:35:48.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:48.718+0000 7fb1d0a00640 1 --2- 172.21.15.6:0/61498642 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb1cc0d4a10 0x7fb1cc0d0740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:48.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:48.718+0000 7fb1d0a00640 1 -- 172.21.15.6:0/61498642 >> v1:172.21.15.146:6789/0 conn(0x7fb1cc15e0b0 legacy=0x7fb1cc0cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:48.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:48.718+0000 7fb1d0a00640 1 -- 172.21.15.6:0/61498642 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1cc0b3cd0 con 0x7fb1cc0d0c80 2024-03-04T09:35:48.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:48.718+0000 7fb1d0a00640 1 -- 172.21.15.6:0/61498642 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7fb1c80019f0 con 0x7fb1cc0d0c80 2024-03-04T09:35:48.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:48.718+0000 7fb1d0a00640 1 -- 172.21.15.6:0/61498642 >> v1:172.21.15.6:6789/0 conn(0x7fb1cc0d0c80 legacy=0x7fb1cc0b44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:48.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:48.718+0000 7fb1d0a00640 1 --2- 172.21.15.6:0/61498642 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb1cc0d4a10 0x7fb1b40039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:48.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:48.718+0000 7fb1d0a00640 1 --2- 172.21.15.6:0/61498642 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb1b4003fb0 0x7fb1b4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:48.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:48.718+0000 7fb1d0a00640 1 --2- 172.21.15.6:0/61498642 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb1b4004d40 0x7fb1b40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:48.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:48.718+0000 7fb1d0a00640 1 -- 172.21.15.6:0/61498642 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1b400bad0 con 0x7fb1b4003fb0 2024-03-04T09:35:48.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:48.718+0000 7fb1d0a00640 1 -- 172.21.15.6:0/61498642 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1b400bc80 con 0x7fb1b4004d40 2024-03-04T09:35:48.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:48.718+0000 7fb1d0a00640 1 -- 172.21.15.6:0/61498642 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1b400be80 con 0x7fb1cc0d4a10 2024-03-04T09:35:48.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:48.718+0000 7fb1d1400640 1 --2- 172.21.15.6:0/61498642 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb1b4003fb0 0x7fb1b4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:48.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:48.719+0000 7fb1d2800640 1 --2- 172.21.15.6:0/61498642 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb1b4004d40 0x7fb1b40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:48.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:48.719+0000 7fb1d1e00640 1 --2- 172.21.15.6:0/61498642 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb1cc0d4a10 0x7fb1b40039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:48.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:48.719+0000 7fb1d1400640 1 -- 172.21.15.6:0/61498642 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb1cc0d4a10 msgr2=0x7fb1b40039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:35:48.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:48.719+0000 7fb1d1400640 1 --2- 172.21.15.6:0/61498642 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb1cc0d4a10 0x7fb1b40039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:48.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:48.719+0000 7fb1d1400640 1 -- 172.21.15.6:0/61498642 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb1b4004d40 msgr2=0x7fb1b40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:35:48.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:48.719+0000 7fb1d1400640 1 --2- 172.21.15.6:0/61498642 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb1b4004d40 0x7fb1b40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:48.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:48.719+0000 7fb1d1400640 1 -- 172.21.15.6:0/61498642 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1b4003520 con 0x7fb1b4003fb0 2024-03-04T09:35:48.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:48.720+0000 7fb1d1400640 1 --2- 172.21.15.6:0/61498642 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb1b4003fb0 0x7fb1b4004800 secure :-1 s=READY pgs=1745 cs=0 l=1 rev1=1 crypto rx=0x7fb1bc009a90 tx=0x7fb1b4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b43855b0ec2ccfb server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:35:48.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:48.720+0000 7fb1d0a00640 1 -- 172.21.15.6:0/61498642 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1bc039070 con 0x7fb1b4003fb0 2024-03-04T09:35:48.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:48.720+0000 7fb1d0a00640 1 -- 172.21.15.6:0/61498642 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1bc02c5c0 con 0x7fb1b4003fb0 2024-03-04T09:35:48.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:48.720+0000 7fb1d0a00640 1 -- 172.21.15.6:0/61498642 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1bc009690 con 0x7fb1b4003fb0 2024-03-04T09:35:48.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:48.721+0000 7fb1d3200640 1 -- 172.21.15.6:0/61498642 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb1b4003fb0 msgr2=0x7fb1b4004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:48.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:48.721+0000 7fb1d3200640 1 --2- 172.21.15.6:0/61498642 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb1b4003fb0 0x7fb1b4004800 secure :-1 s=READY pgs=1745 cs=0 l=1 rev1=1 crypto rx=0x7fb1bc009a90 tx=0x7fb1b4005830 comp rx=0 tx=0).stop 2024-03-04T09:35:48.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:48.721+0000 7fb1d2800640 1 -- 172.21.15.6:0/61498642 reap_dead start 2024-03-04T09:35:48.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:48.721+0000 7fb1d3200640 1 -- 172.21.15.6:0/61498642 shutdown_connections 2024-03-04T09:35:48.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:48.721+0000 7fb1d3200640 1 -- 172.21.15.6:0/61498642 >> 172.21.15.6:0/61498642 conn(0x7fb1cc0d3d30 msgr2=0x7fb1cc0d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:35:48.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:48.721+0000 7fb1d3200640 1 -- 172.21.15.6:0/61498642 shutdown_connections 2024-03-04T09:35:48.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:48.721+0000 7fb1d3200640 1 -- 172.21.15.6:0/61498642 wait complete. 2024-03-04T09:35:48.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:48.724+0000 7fb1d3200640 1 Processor -- start 2024-03-04T09:35:48.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:48.724+0000 7fb1d3200640 1 -- start start 2024-03-04T09:35:48.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:48.726+0000 7fb1d3200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb1b4003fb0 0x7fb1cc14f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:48.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:48.726+0000 7fb1d3200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb1cc06c9f0 0x7fb1cc06ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:48.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:48.726+0000 7fb1d3200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb1cc0cbb60 0x7fb1cc0cbfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:48.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:48.726+0000 7fb1d3200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1cc15e490 con 0x7fb1cc0cbb60 2024-03-04T09:35:48.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:48.726+0000 7fb1d3200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1cc0cfa20 con 0x7fb1cc06c9f0 2024-03-04T09:35:48.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:48.726+0000 7fb1d3200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1cc0cfc60 con 0x7fb1b4003fb0 2024-03-04T09:35:48.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:48.726+0000 7fb1d2800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb1cc0cbb60 0x7fb1cc0cbfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:48.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:48.726+0000 7fb1d2800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb1cc0cbb60 0x7fb1cc0cbfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49764/0 (socket says 172.21.15.6:49764) 2024-03-04T09:35:48.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:48.726+0000 7fb1d2800640 1 -- 172.21.15.6:0/912552555 learned_addr learned my addr 172.21.15.6:0/912552555 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:35:48.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:48.727+0000 7fb1d1400640 1 --2- 172.21.15.6:0/912552555 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb1cc06c9f0 0x7fb1cc06ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:48.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:48.727+0000 7fb1d1e00640 1 --2- 172.21.15.6:0/912552555 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb1b4003fb0 0x7fb1cc14f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:48.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:48.727+0000 7fb1d2800640 1 -- 172.21.15.6:0/912552555 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb1b4003fb0 msgr2=0x7fb1cc14f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:48.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:48.727+0000 7fb1d2800640 1 --2- 172.21.15.6:0/912552555 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb1b4003fb0 0x7fb1cc14f050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:48.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:48.727+0000 7fb1d2800640 1 -- 172.21.15.6:0/912552555 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb1cc06c9f0 msgr2=0x7fb1cc06ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:48.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:48.727+0000 7fb1d2800640 1 --2- 172.21.15.6:0/912552555 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb1cc06c9f0 0x7fb1cc06ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:48.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:48.727+0000 7fb1d2800640 1 -- 172.21.15.6:0/912552555 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1cc14f590 con 0x7fb1cc0cbb60 2024-03-04T09:35:48.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:48.727+0000 7fb1d1400640 1 --2- 172.21.15.6:0/912552555 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb1cc06c9f0 0x7fb1cc06ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:35:48.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:48.727+0000 7fb1d2800640 1 --2- 172.21.15.6:0/912552555 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb1cc0cbb60 0x7fb1cc0cbfe0 secure :-1 s=READY pgs=1746 cs=0 l=1 rev1=1 crypto rx=0x7fb1cc0d3c80 tx=0x7fb1c800e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:35:48.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:48.727+0000 7fb1c6000640 1 -- 172.21.15.6:0/912552555 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1c800ec10 con 0x7fb1cc0cbb60 2024-03-04T09:35:48.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:48.727+0000 7fb1c6000640 1 -- 172.21.15.6:0/912552555 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1c8018070 con 0x7fb1cc0cbb60 2024-03-04T09:35:48.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:48.727+0000 7fb1d3200640 1 -- 172.21.15.6:0/912552555 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1cc160810 con 0x7fb1cc0cbb60 2024-03-04T09:35:48.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:48.727+0000 7fb1c6000640 1 -- 172.21.15.6:0/912552555 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1c8013420 con 0x7fb1cc0cbb60 2024-03-04T09:35:48.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:48.728+0000 7fb1d3200640 1 -- 172.21.15.6:0/912552555 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1cc160dd0 con 0x7fb1cc0cbb60 2024-03-04T09:35:48.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:48.729+0000 7fb1d3200640 1 -- 172.21.15.6:0/912552555 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1cc06d3b0 con 0x7fb1cc0cbb60 2024-03-04T09:35:48.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:48.730+0000 7fb1c6000640 1 -- 172.21.15.6:0/912552555 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb1c80135c0 con 0x7fb1cc0cbb60 2024-03-04T09:35:48.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:48.732+0000 7fb1c6000640 1 --2- 172.21.15.6:0/912552555 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb1b006b4d0 0x7fb1b006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:48.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:48.732+0000 7fb1c6000640 1 -- 172.21.15.6:0/912552555 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fb1c808a610 con 0x7fb1cc0cbb60 2024-03-04T09:35:48.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:48.732+0000 7fb1d1e00640 1 --2- 172.21.15.6:0/912552555 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb1b006b4d0 0x7fb1b006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:48.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:48.734+0000 7fb1d1e00640 1 --2- 172.21.15.6:0/912552555 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb1b006b4d0 0x7fb1b006d990 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fb1c0001550 tx=0x7fb1c0009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:35:48.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:48.736+0000 7fb1c6000640 1 -- 172.21.15.6:0/912552555 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb1c8059440 con 0x7fb1cc0cbb60 2024-03-04T09:35:48.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:48.979+0000 7fb1d3200640 1 -- 172.21.15.6:0/912552555 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb1cc04f330 con 0x7fb1cc0cbb60 2024-03-04T09:35:48.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:48.980+0000 7fb1c6000640 1 -- 172.21.15.6:0/912552555 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3464 (secure 0 0 0) 0x7fb1c8058d40 con 0x7fb1cc0cbb60 2024-03-04T09:35:48.977 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:35:48.978 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:35:26.088222+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"laggy_since":"2024-03-04T09:33:54.652601+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":9}]} 2024-03-04T09:35:48.978 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 101 2024-03-04T09:35:48.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:48.983+0000 7fb1d3200640 1 -- 172.21.15.6:0/912552555 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb1b006b4d0 msgr2=0x7fb1b006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:48.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:48.983+0000 7fb1d3200640 1 --2- 172.21.15.6:0/912552555 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb1b006b4d0 0x7fb1b006d990 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fb1c0001550 tx=0x7fb1c0009040 comp rx=0 tx=0).stop 2024-03-04T09:35:48.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:48.983+0000 7fb1d3200640 1 -- 172.21.15.6:0/912552555 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb1cc0cbb60 msgr2=0x7fb1cc0cbfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:48.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:48.983+0000 7fb1d3200640 1 --2- 172.21.15.6:0/912552555 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb1cc0cbb60 0x7fb1cc0cbfe0 secure :-1 s=READY pgs=1746 cs=0 l=1 rev1=1 crypto rx=0x7fb1cc0d3c80 tx=0x7fb1c800e530 comp rx=0 tx=0).stop 2024-03-04T09:35:48.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:48.984+0000 7fb1d3200640 1 -- 172.21.15.6:0/912552555 shutdown_connections 2024-03-04T09:35:48.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:48.984+0000 7fb1d3200640 1 --2- 172.21.15.6:0/912552555 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb1cc0cbb60 0x7fb1cc0cbfe0 unknown :-1 s=CLOSED pgs=1746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:48.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:48.984+0000 7fb1d3200640 1 --2- 172.21.15.6:0/912552555 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb1cc06c9f0 0x7fb1cc06ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:48.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:48.984+0000 7fb1d3200640 1 --2- 172.21.15.6:0/912552555 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb1b006b4d0 0x7fb1b006d990 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:48.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:48.984+0000 7fb1d3200640 1 --2- 172.21.15.6:0/912552555 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb1b4003fb0 0x7fb1cc14f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:48.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:48.984+0000 7fb1d3200640 1 -- 172.21.15.6:0/912552555 >> 172.21.15.6:0/912552555 conn(0x7fb1cc0d3d30 msgr2=0x7fb1cc0b8960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:35:48.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:48.984+0000 7fb1d3200640 1 -- 172.21.15.6:0/912552555 shutdown_connections 2024-03-04T09:35:48.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:48.984+0000 7fb1d3200640 1 -- 172.21.15.6:0/912552555 wait complete. 2024-03-04T09:35:49.203 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:32:57.159681+0000', 'modified': '2024-03-04T09:35:26.088222+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8214, 'mds_1': 6796}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6796': {'gid': 6796, 'name': 'd', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.146:6837/2458070840', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 2458070840}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 2458070840}]}, 'join_fscid': -1, 'laggy_since': '2024-03-04T09:33:54.652601+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8214': {'gid': 8214, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2446627517', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2446627517}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2446627517}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8214, 'qdb_cluster': [8214]} 2024-03-04T09:35:49.203 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6796:up:stopping 2024-03-04T09:35:49.910 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-04T09:35:50.204 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:35:50.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:50.433+0000 7f9a28c00640 1 Processor -- start 2024-03-04T09:35:50.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:50.433+0000 7f9a28c00640 1 -- start start 2024-03-04T09:35:50.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:50.434+0000 7f9a28c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9a24145300 0x7f9a240b49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:50.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:50.434+0000 7f9a28c00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a240714d0 con 0x7f9a24070f10 2024-03-04T09:35:50.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:50.434+0000 7f9a28c00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a240717b0 con 0x7f9a240cd060 2024-03-04T09:35:50.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:50.434+0000 7f9a28c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a24071a90 con 0x7f9a24145300 2024-03-04T09:35:50.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:50.435+0000 7f9a22a00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f9a24070f10 0x7f9a240cc950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:32774/0 (socket says 172.21.15.6:32774) 2024-03-04T09:35:50.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:50.435+0000 7f9a22a00640 1 -- 172.21.15.6:0/2428590075 learned_addr learned my addr 172.21.15.6:0/2428590075 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:35:50.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:50.435+0000 7f9a23400640 1 --2- 172.21.15.6:0/2428590075 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9a24145300 0x7f9a240b49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:50.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:50.435+0000 7f9a22000640 1 -- 172.21.15.6:0/2428590075 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3999773967 0 0) 0x7f9a240714d0 con 0x7f9a24070f10 2024-03-04T09:35:50.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:50.435+0000 7f9a22000640 1 -- 172.21.15.6:0/2428590075 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a08003520 con 0x7f9a24070f10 2024-03-04T09:35:50.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:50.436+0000 7f9a22000640 1 -- 172.21.15.6:0/2428590075 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 380973763 0 0) 0x7f9a08003520 con 0x7f9a24070f10 2024-03-04T09:35:50.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:50.436+0000 7f9a22000640 1 -- 172.21.15.6:0/2428590075 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9a24145300 msgr2=0x7f9a240b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:35:50.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:50.436+0000 7f9a22000640 1 --2- 172.21.15.6:0/2428590075 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9a24145300 0x7f9a240b49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:50.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:50.436+0000 7f9a22000640 1 -- 172.21.15.6:0/2428590075 >> v1:172.21.15.146:6789/0 conn(0x7f9a240cd060 legacy=0x7f9a2415c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:50.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:50.436+0000 7f9a22000640 1 -- 172.21.15.6:0/2428590075 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a24071d70 con 0x7f9a24070f10 2024-03-04T09:35:50.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:50.436+0000 7f9a22000640 1 -- 172.21.15.6:0/2428590075 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7f9a10001670 con 0x7f9a24070f10 2024-03-04T09:35:50.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:50.436+0000 7f9a22000640 1 -- 172.21.15.6:0/2428590075 >> v1:172.21.15.6:6789/0 conn(0x7f9a24070f10 legacy=0x7f9a240cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:50.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:50.436+0000 7f9a22000640 1 --2- 172.21.15.6:0/2428590075 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9a080039e0 0x7f9a08003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:50.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:50.436+0000 7f9a22000640 1 --2- 172.21.15.6:0/2428590075 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9a24145300 0x7f9a08004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:50.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:50.436+0000 7f9a22000640 1 --2- 172.21.15.6:0/2428590075 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9a08004d40 0x7f9a080051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:50.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:50.436+0000 7f9a22000640 1 -- 172.21.15.6:0/2428590075 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a24071a90 con 0x7f9a080039e0 2024-03-04T09:35:50.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:50.436+0000 7f9a22000640 1 -- 172.21.15.6:0/2428590075 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a0800ba70 con 0x7f9a08004d40 2024-03-04T09:35:50.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:50.436+0000 7f9a22000640 1 -- 172.21.15.6:0/2428590075 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a0800bc70 con 0x7f9a24145300 2024-03-04T09:35:50.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:50.436+0000 7f9a23400640 1 --2- 172.21.15.6:0/2428590075 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9a080039e0 0x7f9a08003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:50.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:50.436+0000 7f9a22a00640 1 --2- 172.21.15.6:0/2428590075 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9a24145300 0x7f9a08004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:50.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:50.436+0000 7f9a23e00640 1 --2- 172.21.15.6:0/2428590075 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9a08004d40 0x7f9a080051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:50.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:50.437+0000 7f9a23400640 1 -- 172.21.15.6:0/2428590075 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9a24145300 msgr2=0x7f9a08004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:35:50.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:50.437+0000 7f9a23400640 1 --2- 172.21.15.6:0/2428590075 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9a24145300 0x7f9a08004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:50.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:50.437+0000 7f9a23400640 1 -- 172.21.15.6:0/2428590075 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9a08004d40 msgr2=0x7f9a080051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:35:50.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:50.437+0000 7f9a23400640 1 --2- 172.21.15.6:0/2428590075 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9a08004d40 0x7f9a080051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:50.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:50.437+0000 7f9a23400640 1 -- 172.21.15.6:0/2428590075 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a08003520 con 0x7f9a080039e0 2024-03-04T09:35:50.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:50.438+0000 7f9a23400640 1 --2- 172.21.15.6:0/2428590075 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9a080039e0 0x7f9a08003e00 secure :-1 s=READY pgs=1748 cs=0 l=1 rev1=1 crypto rx=0x7f9a14009a90 tx=0x7f9a08005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d234869da44d14 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:35:50.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:50.438+0000 7f9a22000640 1 -- 172.21.15.6:0/2428590075 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a14039070 con 0x7f9a080039e0 2024-03-04T09:35:50.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:50.439+0000 7f9a22000640 1 -- 172.21.15.6:0/2428590075 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a1402c680 con 0x7f9a080039e0 2024-03-04T09:35:50.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:50.439+0000 7f9a22000640 1 -- 172.21.15.6:0/2428590075 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a14009690 con 0x7f9a080039e0 2024-03-04T09:35:50.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:50.439+0000 7f9a28c00640 1 -- 172.21.15.6:0/2428590075 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9a080039e0 msgr2=0x7f9a08003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:50.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:50.439+0000 7f9a28c00640 1 --2- 172.21.15.6:0/2428590075 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9a080039e0 0x7f9a08003e00 secure :-1 s=READY pgs=1748 cs=0 l=1 rev1=1 crypto rx=0x7f9a14009a90 tx=0x7f9a08005830 comp rx=0 tx=0).stop 2024-03-04T09:35:50.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:50.439+0000 7f9a23e00640 1 -- 172.21.15.6:0/2428590075 reap_dead start 2024-03-04T09:35:50.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:50.439+0000 7f9a28c00640 1 -- 172.21.15.6:0/2428590075 shutdown_connections 2024-03-04T09:35:50.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:50.439+0000 7f9a28c00640 1 -- 172.21.15.6:0/2428590075 >> 172.21.15.6:0/2428590075 conn(0x7f9a2406b3d0 msgr2=0x7f9a2406b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:35:50.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:50.439+0000 7f9a28c00640 1 -- 172.21.15.6:0/2428590075 shutdown_connections 2024-03-04T09:35:50.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:50.439+0000 7f9a28c00640 1 -- 172.21.15.6:0/2428590075 wait complete. 2024-03-04T09:35:50.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:50.441+0000 7f9a28c00640 1 Processor -- start 2024-03-04T09:35:50.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:50.441+0000 7f9a28c00640 1 -- start start 2024-03-04T09:35:50.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:50.442+0000 7f9a28c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9a080039e0 0x7f9a2413e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:50.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:50.442+0000 7f9a28c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9a2413f370 0x7f9a240abd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:50.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:50.442+0000 7f9a28c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9a2413edb0 0x7f9a240ac2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:50.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:50.442+0000 7f9a28c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a240b4b20 con 0x7f9a2413f370 2024-03-04T09:35:50.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:50.442+0000 7f9a28c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a240b49a0 con 0x7f9a2413edb0 2024-03-04T09:35:50.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:50.442+0000 7f9a28c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a240b4ca0 con 0x7f9a080039e0 2024-03-04T09:35:50.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:50.442+0000 7f9a22a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9a2413f370 0x7f9a240abd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:50.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:50.443+0000 7f9a22a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9a2413f370 0x7f9a240abd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:46642/0 (socket says 172.21.15.6:46642) 2024-03-04T09:35:50.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:50.443+0000 7f9a22a00640 1 -- 172.21.15.6:0/3267526090 learned_addr learned my addr 172.21.15.6:0/3267526090 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:35:50.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:50.443+0000 7f9a23400640 1 --2- 172.21.15.6:0/3267526090 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9a080039e0 0x7f9a2413e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:50.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:50.443+0000 7f9a23e00640 1 --2- 172.21.15.6:0/3267526090 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9a2413edb0 0x7f9a240ac2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:50.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:50.443+0000 7f9a22a00640 1 -- 172.21.15.6:0/3267526090 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9a080039e0 msgr2=0x7f9a2413e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:50.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:50.443+0000 7f9a22a00640 1 --2- 172.21.15.6:0/3267526090 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9a080039e0 0x7f9a2413e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:50.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:50.443+0000 7f9a22a00640 1 -- 172.21.15.6:0/3267526090 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9a2413edb0 msgr2=0x7f9a240ac2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:50.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:50.443+0000 7f9a22a00640 1 --2- 172.21.15.6:0/3267526090 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9a2413edb0 0x7f9a240ac2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:50.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:50.443+0000 7f9a22a00640 1 -- 172.21.15.6:0/3267526090 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a2413eb60 con 0x7f9a2413f370 2024-03-04T09:35:50.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:50.443+0000 7f9a23e00640 1 --2- 172.21.15.6:0/3267526090 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9a2413edb0 0x7f9a240ac2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:35:50.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:50.443+0000 7f9a22a00640 1 --2- 172.21.15.6:0/3267526090 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9a2413f370 0x7f9a240abd90 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 crypto rx=0x7f9a1000e870 tx=0x7f9a1000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:35:50.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:50.444+0000 7f9a1b400640 1 -- 172.21.15.6:0/3267526090 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a1000c8d0 con 0x7f9a2413f370 2024-03-04T09:35:50.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:50.444+0000 7f9a1b400640 1 -- 172.21.15.6:0/3267526090 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a10018070 con 0x7f9a2413f370 2024-03-04T09:35:50.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:50.444+0000 7f9a28c00640 1 -- 172.21.15.6:0/3267526090 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a240aeca0 con 0x7f9a2413f370 2024-03-04T09:35:50.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:50.444+0000 7f9a1b400640 1 -- 172.21.15.6:0/3267526090 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a1000ccd0 con 0x7f9a2413f370 2024-03-04T09:35:50.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:50.444+0000 7f9a28c00640 1 -- 172.21.15.6:0/3267526090 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a240af260 con 0x7f9a2413f370 2024-03-04T09:35:50.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:50.445+0000 7f9a28c00640 1 -- 172.21.15.6:0/3267526090 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a2404f330 con 0x7f9a2413f370 2024-03-04T09:35:50.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:50.447+0000 7f9a1b400640 1 -- 172.21.15.6:0/3267526090 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9a10013420 con 0x7f9a2413f370 2024-03-04T09:35:50.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:50.448+0000 7f9a1b400640 1 --2- 172.21.15.6:0/3267526090 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f9a0406b500 0x7f9a0406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:50.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:50.448+0000 7f9a1b400640 1 -- 172.21.15.6:0/3267526090 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f9a10089e90 con 0x7f9a2413f370 2024-03-04T09:35:50.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:50.449+0000 7f9a23400640 1 --2- 172.21.15.6:0/3267526090 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f9a0406b500 0x7f9a0406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:50.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:50.450+0000 7f9a23400640 1 --2- 172.21.15.6:0/3267526090 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f9a0406b500 0x7f9a0406d9c0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f9a14036000 tx=0x7f9a14030000 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:35:50.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:50.454+0000 7f9a1b400640 1 -- 172.21.15.6:0/3267526090 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9a10058cc0 con 0x7f9a2413f370 2024-03-04T09:35:50.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:50.698+0000 7f9a28c00640 1 -- 172.21.15.6:0/3267526090 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9a240af460 con 0x7f9a2413f370 2024-03-04T09:35:50.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:50.699+0000 7f9a1b400640 1 -- 172.21.15.6:0/3267526090 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3464 (secure 0 0 0) 0x7f9a1008b070 con 0x7f9a2413f370 2024-03-04T09:35:50.696 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:35:50.696 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:35:26.088222+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"laggy_since":"2024-03-04T09:33:54.652601+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":9}]} 2024-03-04T09:35:50.696 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 101 2024-03-04T09:35:50.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:50.701+0000 7f9a28c00640 1 -- 172.21.15.6:0/3267526090 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f9a0406b500 msgr2=0x7f9a0406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:50.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:50.701+0000 7f9a28c00640 1 --2- 172.21.15.6:0/3267526090 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f9a0406b500 0x7f9a0406d9c0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f9a14036000 tx=0x7f9a14030000 comp rx=0 tx=0).stop 2024-03-04T09:35:50.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:50.701+0000 7f9a28c00640 1 -- 172.21.15.6:0/3267526090 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9a2413f370 msgr2=0x7f9a240abd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:50.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:50.701+0000 7f9a28c00640 1 --2- 172.21.15.6:0/3267526090 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9a2413f370 0x7f9a240abd90 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 crypto rx=0x7f9a1000e870 tx=0x7f9a1000ed30 comp rx=0 tx=0).stop 2024-03-04T09:35:50.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:50.701+0000 7f9a28c00640 1 -- 172.21.15.6:0/3267526090 shutdown_connections 2024-03-04T09:35:50.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:50.701+0000 7f9a28c00640 1 --2- 172.21.15.6:0/3267526090 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9a2413edb0 0x7f9a240ac2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:50.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:50.701+0000 7f9a28c00640 1 --2- 172.21.15.6:0/3267526090 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9a2413f370 0x7f9a240abd90 unknown :-1 s=CLOSED pgs=1749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:50.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:50.701+0000 7f9a28c00640 1 --2- 172.21.15.6:0/3267526090 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f9a0406b500 0x7f9a0406d9c0 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:50.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:50.701+0000 7f9a28c00640 1 --2- 172.21.15.6:0/3267526090 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9a080039e0 0x7f9a2413e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:50.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:50.701+0000 7f9a28c00640 1 -- 172.21.15.6:0/3267526090 >> 172.21.15.6:0/3267526090 conn(0x7f9a2406b3d0 msgr2=0x7f9a240b28b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:35:50.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:50.702+0000 7f9a28c00640 1 -- 172.21.15.6:0/3267526090 shutdown_connections 2024-03-04T09:35:50.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:50.702+0000 7f9a28c00640 1 -- 172.21.15.6:0/3267526090 wait complete. 2024-03-04T09:35:50.921 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:32:57.159681+0000', 'modified': '2024-03-04T09:35:26.088222+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8214, 'mds_1': 6796}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6796': {'gid': 6796, 'name': 'd', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.146:6837/2458070840', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 2458070840}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 2458070840}]}, 'join_fscid': -1, 'laggy_since': '2024-03-04T09:33:54.652601+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8214': {'gid': 8214, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2446627517', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2446627517}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2446627517}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8214, 'qdb_cluster': [8214]} 2024-03-04T09:35:50.921 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6796:up:stopping 2024-03-04T09:35:51.912 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-04T09:35:51.923 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:35:52.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:52.153+0000 7f3497e00640 1 Processor -- start 2024-03-04T09:35:52.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:52.153+0000 7f3497e00640 1 -- start start 2024-03-04T09:35:52.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:52.155+0000 7f3497e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f34980d4a10 0x7f34980d0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:52.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:52.155+0000 7f3497e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34980b3710 con 0x7f349815e0b0 2024-03-04T09:35:52.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:52.155+0000 7f3497e00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34980b39f0 con 0x7f34980d0c80 2024-03-04T09:35:52.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:52.155+0000 7f3497e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34980b09e0 con 0x7f34980d4a10 2024-03-04T09:35:52.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:52.155+0000 7f3496a00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f349815e0b0 0x7f34980cf310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:32788/0 (socket says 172.21.15.6:32788) 2024-03-04T09:35:52.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:52.155+0000 7f3496a00640 1 -- 172.21.15.6:0/240719228 learned_addr learned my addr 172.21.15.6:0/240719228 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:35:52.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:52.155+0000 7f3496000640 1 -- 172.21.15.6:0/240719228 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3130359400 0 0) 0x7f34980b3710 con 0x7f349815e0b0 2024-03-04T09:35:52.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:52.155+0000 7f348e000640 1 --2- 172.21.15.6:0/240719228 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f34980d4a10 0x7f34980d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:52.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:52.155+0000 7f3496000640 1 -- 172.21.15.6:0/240719228 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f347c003520 con 0x7f349815e0b0 2024-03-04T09:35:52.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:52.156+0000 7f3496000640 1 -- 172.21.15.6:0/240719228 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3212298723 0 0) 0x7f347c003520 con 0x7f349815e0b0 2024-03-04T09:35:52.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:52.156+0000 7f3496000640 1 -- 172.21.15.6:0/240719228 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f34980d4a10 msgr2=0x7f34980d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:35:52.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:52.156+0000 7f3496000640 1 --2- 172.21.15.6:0/240719228 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f34980d4a10 0x7f34980d0740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:52.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:52.156+0000 7f3496000640 1 -- 172.21.15.6:0/240719228 >> v1:172.21.15.146:6789/0 conn(0x7f34980d0c80 legacy=0x7f34980b44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:52.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:52.156+0000 7f3496000640 1 -- 172.21.15.6:0/240719228 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34980b3cd0 con 0x7f349815e0b0 2024-03-04T09:35:52.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:52.156+0000 7f3496000640 1 -- 172.21.15.6:0/240719228 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7f34880019f0 con 0x7f349815e0b0 2024-03-04T09:35:52.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:52.156+0000 7f3496000640 1 -- 172.21.15.6:0/240719228 >> v1:172.21.15.6:6789/0 conn(0x7f349815e0b0 legacy=0x7f34980cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:52.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:52.156+0000 7f3496000640 1 --2- 172.21.15.6:0/240719228 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f347c0039e0 0x7f347c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:52.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:52.156+0000 7f3496000640 1 --2- 172.21.15.6:0/240719228 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f347c004800 0x7f347c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:52.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:52.156+0000 7f3496000640 1 --2- 172.21.15.6:0/240719228 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f34980d4a10 0x7f347c005250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:52.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:52.156+0000 7f3496000640 1 -- 172.21.15.6:0/240719228 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f347c00bb50 con 0x7f347c0039e0 2024-03-04T09:35:52.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:52.156+0000 7f3496000640 1 -- 172.21.15.6:0/240719228 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f347c00bcd0 con 0x7f347c004800 2024-03-04T09:35:52.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:52.156+0000 7f3496000640 1 -- 172.21.15.6:0/240719228 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f347c00bed0 con 0x7f34980d4a10 2024-03-04T09:35:52.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:52.156+0000 7f3496a00640 1 --2- 172.21.15.6:0/240719228 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f347c0039e0 0x7f347c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:52.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:52.157+0000 7f348e000640 1 --2- 172.21.15.6:0/240719228 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f347c004800 0x7f347c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:52.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:52.157+0000 7f3497400640 1 --2- 172.21.15.6:0/240719228 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f34980d4a10 0x7f347c005250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:52.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:52.157+0000 7f3496a00640 1 -- 172.21.15.6:0/240719228 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f34980d4a10 msgr2=0x7f347c005250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:35:52.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:52.157+0000 7f3496a00640 1 --2- 172.21.15.6:0/240719228 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f34980d4a10 0x7f347c005250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:52.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:52.157+0000 7f3496a00640 1 -- 172.21.15.6:0/240719228 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f347c004800 msgr2=0x7f347c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:35:52.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:52.157+0000 7f3496a00640 1 --2- 172.21.15.6:0/240719228 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f347c004800 0x7f347c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:52.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:52.157+0000 7f3496a00640 1 -- 172.21.15.6:0/240719228 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f347c003520 con 0x7f347c0039e0 2024-03-04T09:35:52.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:52.157+0000 7f348e000640 1 --2- 172.21.15.6:0/240719228 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f347c004800 0x7f347c004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:35:52.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:52.157+0000 7f3497400640 1 --2- 172.21.15.6:0/240719228 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f34980d4a10 0x7f347c005250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:35:52.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:52.158+0000 7f3496a00640 1 --2- 172.21.15.6:0/240719228 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f347c0039e0 0x7f347c003e00 secure :-1 s=READY pgs=1751 cs=0 l=1 rev1=1 crypto rx=0x7f348800a7b0 tx=0x7f347c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c19e7a5a906f04c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:35:52.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:52.158+0000 7f3496000640 1 -- 172.21.15.6:0/240719228 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f348800a3d0 con 0x7f347c0039e0 2024-03-04T09:35:52.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:52.158+0000 7f3496000640 1 -- 172.21.15.6:0/240719228 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3488034630 con 0x7f347c0039e0 2024-03-04T09:35:52.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:52.158+0000 7f3496000640 1 -- 172.21.15.6:0/240719228 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3488034a60 con 0x7f347c0039e0 2024-03-04T09:35:52.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:52.159+0000 7f3497e00640 1 -- 172.21.15.6:0/240719228 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f347c0039e0 msgr2=0x7f347c003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:52.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:52.159+0000 7f3497e00640 1 --2- 172.21.15.6:0/240719228 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f347c0039e0 0x7f347c003e00 secure :-1 s=READY pgs=1751 cs=0 l=1 rev1=1 crypto rx=0x7f348800a7b0 tx=0x7f347c005830 comp rx=0 tx=0).stop 2024-03-04T09:35:52.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:52.159+0000 7f3497400640 1 -- 172.21.15.6:0/240719228 reap_dead start 2024-03-04T09:35:52.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:52.159+0000 7f3497e00640 1 -- 172.21.15.6:0/240719228 shutdown_connections 2024-03-04T09:35:52.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:52.159+0000 7f3497e00640 1 -- 172.21.15.6:0/240719228 >> 172.21.15.6:0/240719228 conn(0x7f34980d3d30 msgr2=0x7f34980d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:35:52.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:52.159+0000 7f3497e00640 1 -- 172.21.15.6:0/240719228 shutdown_connections 2024-03-04T09:35:52.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:52.159+0000 7f3497e00640 1 -- 172.21.15.6:0/240719228 wait complete. 2024-03-04T09:35:52.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:52.161+0000 7f3497e00640 1 Processor -- start 2024-03-04T09:35:52.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:52.161+0000 7f3497e00640 1 -- start start 2024-03-04T09:35:52.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:52.162+0000 7f3497e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f347c0039e0 0x7f34980b6950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:52.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:52.162+0000 7f3497e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3498152340 0x7f34980b58a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:52.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:52.162+0000 7f3497e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f34980b5de0 0x7f34980cc470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:52.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:52.162+0000 7f3497e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f349815e490 con 0x7f34980b5de0 2024-03-04T09:35:52.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:52.162+0000 7f3497e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34980cfa20 con 0x7f3498152340 2024-03-04T09:35:52.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:52.162+0000 7f3497e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34980cfc60 con 0x7f347c0039e0 2024-03-04T09:35:52.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:52.162+0000 7f3497400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f34980b5de0 0x7f34980cc470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:52.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:52.162+0000 7f3497400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f34980b5de0 0x7f34980cc470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:46662/0 (socket says 172.21.15.6:46662) 2024-03-04T09:35:52.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:52.162+0000 7f3497400640 1 -- 172.21.15.6:0/2330293292 learned_addr learned my addr 172.21.15.6:0/2330293292 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:35:52.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:52.163+0000 7f348e000640 1 --2- 172.21.15.6:0/2330293292 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3498152340 0x7f34980b58a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:52.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:52.163+0000 7f3496a00640 1 --2- 172.21.15.6:0/2330293292 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f347c0039e0 0x7f34980b6950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:52.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:52.163+0000 7f3497400640 1 -- 172.21.15.6:0/2330293292 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f347c0039e0 msgr2=0x7f34980b6950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:52.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:52.163+0000 7f3497400640 1 --2- 172.21.15.6:0/2330293292 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f347c0039e0 0x7f34980b6950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:52.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:52.163+0000 7f3497400640 1 -- 172.21.15.6:0/2330293292 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3498152340 msgr2=0x7f34980b58a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:52.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:52.163+0000 7f3497400640 1 --2- 172.21.15.6:0/2330293292 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3498152340 0x7f34980b58a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:52.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:52.163+0000 7f3497400640 1 -- 172.21.15.6:0/2330293292 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34980cb1a0 con 0x7f34980b5de0 2024-03-04T09:35:52.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:52.163+0000 7f348e000640 1 --2- 172.21.15.6:0/2330293292 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3498152340 0x7f34980b58a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:35:52.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:52.163+0000 7f3497400640 1 --2- 172.21.15.6:0/2330293292 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f34980b5de0 0x7f34980cc470 secure :-1 s=READY pgs=1752 cs=0 l=1 rev1=1 crypto rx=0x7f349808e2b0 tx=0x7f349000d890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:35:52.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:52.163+0000 7f348f400640 1 -- 172.21.15.6:0/2330293292 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3490005480 con 0x7f34980b5de0 2024-03-04T09:35:52.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:52.163+0000 7f348f400640 1 -- 172.21.15.6:0/2330293292 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3490007030 con 0x7f34980b5de0 2024-03-04T09:35:52.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:52.163+0000 7f348f400640 1 -- 172.21.15.6:0/2330293292 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3490007440 con 0x7f34980b5de0 2024-03-04T09:35:52.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:52.164+0000 7f3497e00640 1 -- 172.21.15.6:0/2330293292 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34980cb480 con 0x7f34980b5de0 2024-03-04T09:35:52.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:52.164+0000 7f3497e00640 1 -- 172.21.15.6:0/2330293292 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34980cba40 con 0x7f34980b5de0 2024-03-04T09:35:52.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:52.165+0000 7f3497e00640 1 -- 172.21.15.6:0/2330293292 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34980b6260 con 0x7f34980b5de0 2024-03-04T09:35:52.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:52.166+0000 7f348f400640 1 -- 172.21.15.6:0/2330293292 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f349001b020 con 0x7f34980b5de0 2024-03-04T09:35:52.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:52.167+0000 7f348f400640 1 --2- 172.21.15.6:0/2330293292 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f347406b500 0x7f347406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:52.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:52.167+0000 7f348f400640 1 -- 172.21.15.6:0/2330293292 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f3490089b80 con 0x7f34980b5de0 2024-03-04T09:35:52.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:52.167+0000 7f3496a00640 1 --2- 172.21.15.6:0/2330293292 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f347406b500 0x7f347406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:52.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:52.168+0000 7f3496a00640 1 --2- 172.21.15.6:0/2330293292 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f347406b500 0x7f347406d9c0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f3488034000 tx=0x7f348802f000 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:35:52.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:52.172+0000 7f348f400640 1 -- 172.21.15.6:0/2330293292 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3490058920 con 0x7f34980b5de0 2024-03-04T09:35:52.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:52.409+0000 7f3497e00640 1 -- 172.21.15.6:0/2330293292 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f349804f330 con 0x7f34980b5de0 2024-03-04T09:35:52.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:52.410+0000 7f348f400640 1 -- 172.21.15.6:0/2330293292 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3464 (secure 0 0 0) 0x7f3490058220 con 0x7f34980b5de0 2024-03-04T09:35:52.408 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:35:52.408 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:35:26.088222+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"laggy_since":"2024-03-04T09:33:54.652601+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":9}]} 2024-03-04T09:35:52.408 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 101 2024-03-04T09:35:52.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:52.412+0000 7f3497e00640 1 -- 172.21.15.6:0/2330293292 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f347406b500 msgr2=0x7f347406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:52.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:52.412+0000 7f3497e00640 1 --2- 172.21.15.6:0/2330293292 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f347406b500 0x7f347406d9c0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f3488034000 tx=0x7f348802f000 comp rx=0 tx=0).stop 2024-03-04T09:35:52.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:52.412+0000 7f3497e00640 1 -- 172.21.15.6:0/2330293292 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f34980b5de0 msgr2=0x7f34980cc470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:52.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:52.412+0000 7f3497e00640 1 --2- 172.21.15.6:0/2330293292 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f34980b5de0 0x7f34980cc470 secure :-1 s=READY pgs=1752 cs=0 l=1 rev1=1 crypto rx=0x7f349808e2b0 tx=0x7f349000d890 comp rx=0 tx=0).stop 2024-03-04T09:35:52.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:52.412+0000 7f3497e00640 1 -- 172.21.15.6:0/2330293292 shutdown_connections 2024-03-04T09:35:52.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:52.412+0000 7f3497e00640 1 --2- 172.21.15.6:0/2330293292 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f34980b5de0 0x7f34980cc470 unknown :-1 s=CLOSED pgs=1752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:52.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:52.412+0000 7f3497e00640 1 --2- 172.21.15.6:0/2330293292 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3498152340 0x7f34980b58a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:52.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:52.412+0000 7f3497e00640 1 --2- 172.21.15.6:0/2330293292 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f347406b500 0x7f347406d9c0 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:52.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:52.412+0000 7f3497e00640 1 --2- 172.21.15.6:0/2330293292 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f347c0039e0 0x7f34980b6950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:52.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:52.412+0000 7f3497e00640 1 -- 172.21.15.6:0/2330293292 >> 172.21.15.6:0/2330293292 conn(0x7f34980d3d30 msgr2=0x7f3498160620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:35:52.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:52.412+0000 7f3497e00640 1 -- 172.21.15.6:0/2330293292 shutdown_connections 2024-03-04T09:35:52.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:52.412+0000 7f3497e00640 1 -- 172.21.15.6:0/2330293292 wait complete. 2024-03-04T09:35:52.630 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:32:57.159681+0000', 'modified': '2024-03-04T09:35:26.088222+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8214, 'mds_1': 6796}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6796': {'gid': 6796, 'name': 'd', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.146:6837/2458070840', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 2458070840}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 2458070840}]}, 'join_fscid': -1, 'laggy_since': '2024-03-04T09:33:54.652601+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8214': {'gid': 8214, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2446627517', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2446627517}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2446627517}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8214, 'qdb_cluster': [8214]} 2024-03-04T09:35:52.630 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6796:up:stopping 2024-03-04T09:35:53.632 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:35:53.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:53.851+0000 7f165ac00640 1 Processor -- start 2024-03-04T09:35:53.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:53.851+0000 7f165ac00640 1 -- start start 2024-03-04T09:35:53.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:53.853+0000 7f165ac00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f165415e070 0x7f16540d2960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:53.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:53.853+0000 7f165ac00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f16540cf410 0x7f16540cf870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:53.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:53.853+0000 7f165ac00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f16540d05c0 0x7f16540d0a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:53.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:53.853+0000 7f165ac00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16540d18c0 con 0x7f165415e070 2024-03-04T09:35:53.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:53.853+0000 7f165ac00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16540d1ba0 con 0x7f16540cf410 2024-03-04T09:35:53.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:53.853+0000 7f165ac00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16540d1e80 con 0x7f16540d05c0 2024-03-04T09:35:53.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:53.853+0000 7f1659800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f165415e070 0x7f16540d2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:53.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:53.853+0000 7f1659800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f165415e070 0x7f16540d2960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:46670/0 (socket says 172.21.15.6:46670) 2024-03-04T09:35:53.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:53.853+0000 7f1659800640 1 -- 172.21.15.6:0/4209723074 learned_addr learned my addr 172.21.15.6:0/4209723074 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:35:53.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:53.854+0000 7f1658e00640 1 --2- 172.21.15.6:0/4209723074 >> v2:172.21.15.146:3300/0 conn(0x7f16540cf410 0x7f16540cf870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:53.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:53.854+0000 7f165a200640 1 --2- 172.21.15.6:0/4209723074 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f16540d05c0 0x7f16540d0a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:53.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:53.854+0000 7f1659800640 1 -- 172.21.15.6:0/4209723074 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f16540d05c0 msgr2=0x7f16540d0a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:35:53.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:53.854+0000 7f1659800640 1 --2- 172.21.15.6:0/4209723074 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f16540d05c0 0x7f16540d0a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:53.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:53.854+0000 7f1659800640 1 -- 172.21.15.6:0/4209723074 >> v2:172.21.15.146:3300/0 conn(0x7f16540cf410 msgr2=0x7f16540cf870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:35:53.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:53.854+0000 7f1659800640 1 --2- 172.21.15.6:0/4209723074 >> v2:172.21.15.146:3300/0 conn(0x7f16540cf410 0x7f16540cf870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:53.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:53.854+0000 7f1659800640 1 -- 172.21.15.6:0/4209723074 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16540d2160 con 0x7f165415e070 2024-03-04T09:35:53.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:53.854+0000 7f165a200640 1 --2- 172.21.15.6:0/4209723074 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f16540d05c0 0x7f16540d0a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:35:53.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:53.854+0000 7f1659800640 1 --2- 172.21.15.6:0/4209723074 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f165415e070 0x7f16540d2960 secure :-1 s=READY pgs=1753 cs=0 l=1 rev1=1 crypto rx=0x7f16540b6880 tx=0x7f164802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d79349ee2f6bd80f server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:35:53.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:53.854+0000 7f1653e00640 1 -- 172.21.15.6:0/4209723074 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f164802ca80 con 0x7f165415e070 2024-03-04T09:35:53.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:53.854+0000 7f1653e00640 1 -- 172.21.15.6:0/4209723074 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f164802cc20 con 0x7f165415e070 2024-03-04T09:35:53.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:53.854+0000 7f1653e00640 1 -- 172.21.15.6:0/4209723074 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16480343f0 con 0x7f165415e070 2024-03-04T09:35:53.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:53.855+0000 7f165ac00640 1 -- 172.21.15.6:0/4209723074 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f165415e070 msgr2=0x7f16540d2960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:53.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:53.855+0000 7f165ac00640 1 --2- 172.21.15.6:0/4209723074 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f165415e070 0x7f16540d2960 secure :-1 s=READY pgs=1753 cs=0 l=1 rev1=1 crypto rx=0x7f16540b6880 tx=0x7f164802bee0 comp rx=0 tx=0).stop 2024-03-04T09:35:53.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:53.855+0000 7f165ac00640 1 -- 172.21.15.6:0/4209723074 shutdown_connections 2024-03-04T09:35:53.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:53.855+0000 7f165ac00640 1 --2- 172.21.15.6:0/4209723074 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f16540d05c0 0x7f16540d0a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:53.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:53.855+0000 7f165ac00640 1 --2- 172.21.15.6:0/4209723074 >> v2:172.21.15.146:3300/0 conn(0x7f16540cf410 0x7f16540cf870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:53.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:53.855+0000 7f165ac00640 1 --2- 172.21.15.6:0/4209723074 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f165415e070 0x7f16540d2960 unknown :-1 s=CLOSED pgs=1753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:53.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:53.855+0000 7f165ac00640 1 -- 172.21.15.6:0/4209723074 >> 172.21.15.6:0/4209723074 conn(0x7f165408d2a0 msgr2=0x7f16540d3cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:35:53.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:53.855+0000 7f165ac00640 1 -- 172.21.15.6:0/4209723074 shutdown_connections 2024-03-04T09:35:53.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:53.855+0000 7f165ac00640 1 -- 172.21.15.6:0/4209723074 wait complete. 2024-03-04T09:35:53.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:53.857+0000 7f165ac00640 1 Processor -- start 2024-03-04T09:35:53.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:53.857+0000 7f165ac00640 1 -- start start 2024-03-04T09:35:53.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:53.858+0000 7f165ac00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f16540cf410 0x7f165414e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:53.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:53.858+0000 7f165ac00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f16540d05c0 0x7f16540b7100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:53.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:53.858+0000 7f165ac00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f165415e070 0x7f1654153830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:53.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:53.858+0000 7f165ac00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16540d4730 con 0x7f16540cf410 2024-03-04T09:35:53.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:53.858+0000 7f165ac00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16540d45b0 con 0x7f16540d05c0 2024-03-04T09:35:53.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:53.858+0000 7f165ac00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16540b3630 con 0x7f165415e070 2024-03-04T09:35:53.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:53.858+0000 7f1659800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f16540cf410 0x7f165414e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:53.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:53.858+0000 7f1659800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f16540cf410 0x7f165414e320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:46684/0 (socket says 172.21.15.6:46684) 2024-03-04T09:35:53.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:53.858+0000 7f1659800640 1 -- 172.21.15.6:0/2893854202 learned_addr learned my addr 172.21.15.6:0/2893854202 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:35:53.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:53.859+0000 7f1659800640 1 -- 172.21.15.6:0/2893854202 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f165415e070 msgr2=0x7f1654153830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:53.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:53.859+0000 7f165a200640 1 --2- 172.21.15.6:0/2893854202 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f165415e070 0x7f1654153830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:53.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:53.859+0000 7f1658e00640 1 --2- 172.21.15.6:0/2893854202 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f16540d05c0 0x7f16540b7100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:53.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:53.859+0000 7f1659800640 1 --2- 172.21.15.6:0/2893854202 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f165415e070 0x7f1654153830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:53.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:53.859+0000 7f1659800640 1 -- 172.21.15.6:0/2893854202 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f16540d05c0 msgr2=0x7f16540b7100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:53.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:53.859+0000 7f1659800640 1 --2- 172.21.15.6:0/2893854202 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f16540d05c0 0x7f16540b7100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:53.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:53.859+0000 7f1659800640 1 -- 172.21.15.6:0/2893854202 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f165414e000 con 0x7f16540cf410 2024-03-04T09:35:53.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:53.859+0000 7f1659800640 1 --2- 172.21.15.6:0/2893854202 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f16540cf410 0x7f165414e320 secure :-1 s=READY pgs=1754 cs=0 l=1 rev1=1 crypto rx=0x7f1648036040 tx=0x7f1648004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:35:53.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:53.860+0000 7f1651600640 1 -- 172.21.15.6:0/2893854202 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1648043020 con 0x7f16540cf410 2024-03-04T09:35:53.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:53.860+0000 7f1651600640 1 -- 172.21.15.6:0/2893854202 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f164803e450 con 0x7f16540cf410 2024-03-04T09:35:53.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:53.860+0000 7f1651600640 1 -- 172.21.15.6:0/2893854202 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1648009510 con 0x7f16540cf410 2024-03-04T09:35:53.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:53.860+0000 7f165ac00640 1 -- 172.21.15.6:0/2893854202 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16540c7910 con 0x7f16540cf410 2024-03-04T09:35:53.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:53.860+0000 7f165ac00640 1 -- 172.21.15.6:0/2893854202 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16540c7ed0 con 0x7f16540cf410 2024-03-04T09:35:53.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:53.861+0000 7f165ac00640 1 -- 172.21.15.6:0/2893854202 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f165404f330 con 0x7f16540cf410 2024-03-04T09:35:53.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:53.863+0000 7f1651600640 1 -- 172.21.15.6:0/2893854202 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1648046020 con 0x7f16540cf410 2024-03-04T09:35:53.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:53.864+0000 7f1651600640 1 --2- 172.21.15.6:0/2893854202 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f163406b4d0 0x7f163406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:53.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:53.864+0000 7f1651600640 1 -- 172.21.15.6:0/2893854202 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f16480acb40 con 0x7f16540cf410 2024-03-04T09:35:53.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:53.865+0000 7f1658e00640 1 --2- 172.21.15.6:0/2893854202 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f163406b4d0 0x7f163406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:53.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:53.866+0000 7f1658e00640 1 --2- 172.21.15.6:0/2893854202 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f163406b4d0 0x7f163406d990 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f16540b8e90 tx=0x7f1640008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:35:53.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:53.870+0000 7f1651600640 1 -- 172.21.15.6:0/2893854202 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f164804d030 con 0x7f16540cf410 2024-03-04T09:35:53.913 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-04T09:35:54.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:54.109+0000 7f165ac00640 1 -- 172.21.15.6:0/2893854202 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f16540c75c0 con 0x7f16540cf410 2024-03-04T09:35:54.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:54.110+0000 7f1651600640 1 -- 172.21.15.6:0/2893854202 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3464 (secure 0 0 0) 0x7f164807ba00 con 0x7f16540cf410 2024-03-04T09:35:54.107 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:35:54.107 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:35:26.088222+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"laggy_since":"2024-03-04T09:33:54.652601+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":9}]} 2024-03-04T09:35:54.108 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 101 2024-03-04T09:35:54.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:54.113+0000 7f165ac00640 1 -- 172.21.15.6:0/2893854202 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f163406b4d0 msgr2=0x7f163406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:54.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:54.113+0000 7f165ac00640 1 --2- 172.21.15.6:0/2893854202 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f163406b4d0 0x7f163406d990 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f16540b8e90 tx=0x7f1640008040 comp rx=0 tx=0).stop 2024-03-04T09:35:54.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:54.113+0000 7f165ac00640 1 -- 172.21.15.6:0/2893854202 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f16540cf410 msgr2=0x7f165414e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:54.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:54.113+0000 7f165ac00640 1 --2- 172.21.15.6:0/2893854202 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f16540cf410 0x7f165414e320 secure :-1 s=READY pgs=1754 cs=0 l=1 rev1=1 crypto rx=0x7f1648036040 tx=0x7f1648004a40 comp rx=0 tx=0).stop 2024-03-04T09:35:54.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:54.113+0000 7f165ac00640 1 -- 172.21.15.6:0/2893854202 shutdown_connections 2024-03-04T09:35:54.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:54.113+0000 7f165ac00640 1 --2- 172.21.15.6:0/2893854202 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f163406b4d0 0x7f163406d990 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:54.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:54.113+0000 7f165ac00640 1 --2- 172.21.15.6:0/2893854202 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f165415e070 0x7f1654153830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:54.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:54.113+0000 7f165ac00640 1 --2- 172.21.15.6:0/2893854202 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f16540d05c0 0x7f16540b7100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:54.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:54.113+0000 7f165ac00640 1 --2- 172.21.15.6:0/2893854202 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f16540cf410 0x7f165414e320 unknown :-1 s=CLOSED pgs=1754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:54.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:54.113+0000 7f165ac00640 1 -- 172.21.15.6:0/2893854202 >> 172.21.15.6:0/2893854202 conn(0x7f165408d2a0 msgr2=0x7f16540d3e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:35:54.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:54.113+0000 7f165ac00640 1 -- 172.21.15.6:0/2893854202 shutdown_connections 2024-03-04T09:35:54.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:54.113+0000 7f165ac00640 1 -- 172.21.15.6:0/2893854202 wait complete. 2024-03-04T09:35:54.340 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:32:57.159681+0000', 'modified': '2024-03-04T09:35:26.088222+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8214, 'mds_1': 6796}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6796': {'gid': 6796, 'name': 'd', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.146:6837/2458070840', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 2458070840}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 2458070840}]}, 'join_fscid': -1, 'laggy_since': '2024-03-04T09:33:54.652601+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8214': {'gid': 8214, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2446627517', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2446627517}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2446627517}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8214, 'qdb_cluster': [8214]} 2024-03-04T09:35:54.340 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6796:up:stopping 2024-03-04T09:35:55.342 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:35:55.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:55.566+0000 7f0a09800640 1 Processor -- start 2024-03-04T09:35:55.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:55.567+0000 7f0a09800640 1 -- start start 2024-03-04T09:35:55.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:55.568+0000 7f0a09800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f0a040cd060 0x7f0a041667e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:55.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:55.568+0000 7f0a09800640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a040714d0 con 0x7f0a04145300 2024-03-04T09:35:55.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:55.568+0000 7f0a09800640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a040717b0 con 0x7f0a04070f10 2024-03-04T09:35:55.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:55.568+0000 7f0a09800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a04071a90 con 0x7f0a040cd060 2024-03-04T09:35:55.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:55.569+0000 7f0a03e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f0a04145300 0x7f0a040b49a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:32792/0 (socket says 172.21.15.6:32792) 2024-03-04T09:35:55.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:55.569+0000 7f0a03e00640 1 -- 172.21.15.6:0/3112953250 learned_addr learned my addr 172.21.15.6:0/3112953250 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:35:55.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:55.569+0000 7f0a08e00640 1 --2- 172.21.15.6:0/3112953250 >> v2:172.21.15.6:3300/0 conn(0x7f0a040cd060 0x7f0a041667e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:55.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:55.569+0000 7f0a03400640 1 -- 172.21.15.6:0/3112953250 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 46038078 0 0) 0x7f0a040714d0 con 0x7f0a04145300 2024-03-04T09:35:55.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:55.569+0000 7f0a03400640 1 -- 172.21.15.6:0/3112953250 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09e8003520 con 0x7f0a04145300 2024-03-04T09:35:55.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:55.569+0000 7f0a08e00640 1 -- 172.21.15.6:0/3112953250 >> v1:172.21.15.146:6789/0 conn(0x7f0a04070f10 legacy=0x7f0a040cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:55.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:55.569+0000 7f0a08e00640 1 -- 172.21.15.6:0/3112953250 >> v1:172.21.15.6:6789/0 conn(0x7f0a04145300 legacy=0x7f0a040b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:55.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:55.569+0000 7f0a08e00640 1 -- 172.21.15.6:0/3112953250 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a04071d70 con 0x7f0a040cd060 2024-03-04T09:35:55.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:55.571+0000 7f0a08e00640 1 --2- 172.21.15.6:0/3112953250 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0a040cd060 0x7f0a041667e0 secure :-1 s=READY pgs=1756 cs=0 l=1 rev1=1 crypto rx=0x7f09fc009d90 tx=0x7f09fc02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=480dac7653358c9d server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:35:55.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:55.571+0000 7f0a03400640 1 -- 172.21.15.6:0/3112953250 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09fc039070 con 0x7f0a040cd060 2024-03-04T09:35:55.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:55.571+0000 7f0a03400640 1 -- 172.21.15.6:0/3112953250 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09fc02cc60 con 0x7f0a040cd060 2024-03-04T09:35:55.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:55.571+0000 7f0a03400640 1 -- 172.21.15.6:0/3112953250 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09fc034640 con 0x7f0a040cd060 2024-03-04T09:35:55.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:55.572+0000 7f0a09800640 1 -- 172.21.15.6:0/3112953250 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0a040cd060 msgr2=0x7f0a041667e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:55.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:55.572+0000 7f0a09800640 1 --2- 172.21.15.6:0/3112953250 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0a040cd060 0x7f0a041667e0 secure :-1 s=READY pgs=1756 cs=0 l=1 rev1=1 crypto rx=0x7f09fc009d90 tx=0x7f09fc02beb0 comp rx=0 tx=0).stop 2024-03-04T09:35:55.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:55.572+0000 7f0a09800640 1 -- 172.21.15.6:0/3112953250 shutdown_connections 2024-03-04T09:35:55.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:55.572+0000 7f0a09800640 1 --2- 172.21.15.6:0/3112953250 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0a040cd060 0x7f0a041667e0 unknown :-1 s=CLOSED pgs=1756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:55.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:55.572+0000 7f0a09800640 1 -- 172.21.15.6:0/3112953250 >> 172.21.15.6:0/3112953250 conn(0x7f0a0406b3d0 msgr2=0x7f0a0406b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:35:55.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:55.572+0000 7f0a09800640 1 -- 172.21.15.6:0/3112953250 shutdown_connections 2024-03-04T09:35:55.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:55.572+0000 7f0a09800640 1 -- 172.21.15.6:0/3112953250 wait complete. 2024-03-04T09:35:55.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:55.574+0000 7f0a09800640 1 Processor -- start 2024-03-04T09:35:55.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:55.574+0000 7f0a09800640 1 -- start start 2024-03-04T09:35:55.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:55.575+0000 7f0a09800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0a04070f10 0x7f0a0406e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:55.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:55.575+0000 7f0a09800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0a040cd060 0x7f0a0413d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:55.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:55.575+0000 7f0a09800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0a04145300 0x7f0a0413b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:55.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:55.575+0000 7f0a09800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a0415d700 con 0x7f0a040cd060 2024-03-04T09:35:55.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:55.575+0000 7f0a09800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a0415d580 con 0x7f0a04145300 2024-03-04T09:35:55.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:55.575+0000 7f0a09800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a0415d880 con 0x7f0a04070f10 2024-03-04T09:35:55.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:55.575+0000 7f09fbe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0a040cd060 0x7f0a0413d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:55.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:55.575+0000 7f09fbe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0a040cd060 0x7f0a0413d8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:46700/0 (socket says 172.21.15.6:46700) 2024-03-04T09:35:55.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:55.575+0000 7f09fbe00640 1 -- 172.21.15.6:0/2480533807 learned_addr learned my addr 172.21.15.6:0/2480533807 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:35:55.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:55.576+0000 7f0a03e00640 1 --2- 172.21.15.6:0/2480533807 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0a04070f10 0x7f0a0406e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:55.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:55.576+0000 7f09fbe00640 1 -- 172.21.15.6:0/2480533807 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0a04070f10 msgr2=0x7f0a0406e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:55.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:55.576+0000 7f0a08e00640 1 --2- 172.21.15.6:0/2480533807 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0a04145300 0x7f0a0413b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:55.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:55.576+0000 7f09fbe00640 1 --2- 172.21.15.6:0/2480533807 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0a04070f10 0x7f0a0406e3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:55.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:55.576+0000 7f09fbe00640 1 -- 172.21.15.6:0/2480533807 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0a04145300 msgr2=0x7f0a0413b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:55.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:55.576+0000 7f09fbe00640 1 --2- 172.21.15.6:0/2480533807 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0a04145300 0x7f0a0413b660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:55.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:55.576+0000 7f09fbe00640 1 -- 172.21.15.6:0/2480533807 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a040ae020 con 0x7f0a040cd060 2024-03-04T09:35:55.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:55.576+0000 7f09fbe00640 1 --2- 172.21.15.6:0/2480533807 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0a040cd060 0x7f0a0413d8a0 secure :-1 s=READY pgs=1757 cs=0 l=1 rev1=1 crypto rx=0x7f09ec00c400 tx=0x7f09ec00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:35:55.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:55.576+0000 7f0a00c00640 1 -- 172.21.15.6:0/2480533807 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09ec0052d0 con 0x7f0a040cd060 2024-03-04T09:35:55.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:55.576+0000 7f0a00c00640 1 -- 172.21.15.6:0/2480533807 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09ec005490 con 0x7f0a040cd060 2024-03-04T09:35:55.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:55.576+0000 7f0a00c00640 1 -- 172.21.15.6:0/2480533807 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09ec007260 con 0x7f0a040cd060 2024-03-04T09:35:55.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:55.576+0000 7f0a09800640 1 -- 172.21.15.6:0/2480533807 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a040ae300 con 0x7f0a040cd060 2024-03-04T09:35:55.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:55.576+0000 7f0a09800640 1 -- 172.21.15.6:0/2480533807 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a0406c940 con 0x7f0a040cd060 2024-03-04T09:35:55.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:55.578+0000 7f0a09800640 1 -- 172.21.15.6:0/2480533807 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a040ae820 con 0x7f0a040cd060 2024-03-04T09:35:55.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:55.579+0000 7f0a00c00640 1 -- 172.21.15.6:0/2480533807 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f09ec007400 con 0x7f0a040cd060 2024-03-04T09:35:55.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:55.580+0000 7f0a00c00640 1 --2- 172.21.15.6:0/2480533807 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f09e006b4d0 0x7f09e006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:55.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:55.580+0000 7f0a00c00640 1 -- 172.21.15.6:0/2480533807 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f09ec0897a0 con 0x7f0a040cd060 2024-03-04T09:35:55.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:55.581+0000 7f0a03e00640 1 --2- 172.21.15.6:0/2480533807 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f09e006b4d0 0x7f09e006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:55.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:55.582+0000 7f0a03e00640 1 --2- 172.21.15.6:0/2480533807 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f09e006b4d0 0x7f09e006d990 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f09f40011e0 tx=0x7f09f4009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:35:55.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:55.585+0000 7f0a00c00640 1 -- 172.21.15.6:0/2480533807 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f09ec0585d0 con 0x7f0a040cd060 2024-03-04T09:35:55.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:55.852+0000 7f0a09800640 1 -- 172.21.15.6:0/2480533807 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0a0406cb40 con 0x7f0a040cd060 2024-03-04T09:35:55.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:55.853+0000 7f0a00c00640 1 -- 172.21.15.6:0/2480533807 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3464 (secure 0 0 0) 0x7f09ec057ed0 con 0x7f0a040cd060 2024-03-04T09:35:55.851 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:35:55.851 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:35:26.088222+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"laggy_since":"2024-03-04T09:33:54.652601+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":9}]} 2024-03-04T09:35:55.851 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 101 2024-03-04T09:35:55.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:55.856+0000 7f0a09800640 1 -- 172.21.15.6:0/2480533807 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f09e006b4d0 msgr2=0x7f09e006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:55.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:55.856+0000 7f0a09800640 1 --2- 172.21.15.6:0/2480533807 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f09e006b4d0 0x7f09e006d990 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f09f40011e0 tx=0x7f09f4009040 comp rx=0 tx=0).stop 2024-03-04T09:35:55.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:55.856+0000 7f0a09800640 1 -- 172.21.15.6:0/2480533807 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0a040cd060 msgr2=0x7f0a0413d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:55.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:55.856+0000 7f0a09800640 1 --2- 172.21.15.6:0/2480533807 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0a040cd060 0x7f0a0413d8a0 secure :-1 s=READY pgs=1757 cs=0 l=1 rev1=1 crypto rx=0x7f09ec00c400 tx=0x7f09ec00c8c0 comp rx=0 tx=0).stop 2024-03-04T09:35:55.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:55.856+0000 7f0a09800640 1 -- 172.21.15.6:0/2480533807 shutdown_connections 2024-03-04T09:35:55.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:55.856+0000 7f0a09800640 1 --2- 172.21.15.6:0/2480533807 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0a04145300 0x7f0a0413b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:55.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:55.856+0000 7f0a09800640 1 --2- 172.21.15.6:0/2480533807 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0a040cd060 0x7f0a0413d8a0 unknown :-1 s=CLOSED pgs=1757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:55.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:55.856+0000 7f0a09800640 1 --2- 172.21.15.6:0/2480533807 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f09e006b4d0 0x7f09e006d990 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:55.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:55.856+0000 7f0a09800640 1 --2- 172.21.15.6:0/2480533807 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0a04070f10 0x7f0a0406e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:55.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:55.856+0000 7f0a09800640 1 -- 172.21.15.6:0/2480533807 >> 172.21.15.6:0/2480533807 conn(0x7f0a0406b3d0 msgr2=0x7f0a040cbd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:35:55.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:55.856+0000 7f0a09800640 1 -- 172.21.15.6:0/2480533807 shutdown_connections 2024-03-04T09:35:55.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:55.857+0000 7f0a09800640 1 -- 172.21.15.6:0/2480533807 wait complete. 2024-03-04T09:35:55.916 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-04T09:35:56.071 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:32:57.159681+0000', 'modified': '2024-03-04T09:35:26.088222+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8214, 'mds_1': 6796}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6796': {'gid': 6796, 'name': 'd', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.146:6837/2458070840', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 2458070840}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 2458070840}]}, 'join_fscid': -1, 'laggy_since': '2024-03-04T09:33:54.652601+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8214': {'gid': 8214, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2446627517', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2446627517}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2446627517}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8214, 'qdb_cluster': [8214]} 2024-03-04T09:35:56.071 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6796:up:stopping 2024-03-04T09:35:57.073 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:35:57.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:57.299+0000 7f7a35e00640 1 Processor -- start 2024-03-04T09:35:57.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:57.299+0000 7f7a35e00640 1 -- start start 2024-03-04T09:35:57.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:57.301+0000 7f7a35e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f7a30161b40 0x7f7a300cfd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:57.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:57.301+0000 7f7a35e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f7a300b5060 0x7f7a300d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:57.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:57.301+0000 7f7a35e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7a300b54d0 0x7f7a300d2760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:57.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:57.301+0000 7f7a35e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a300cf900 con 0x7f7a30161b40 2024-03-04T09:35:57.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:57.301+0000 7f7a35e00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a300cea50 con 0x7f7a300b5060 2024-03-04T09:35:57.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:57.301+0000 7f7a35e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a300cecf0 con 0x7f7a300b54d0 2024-03-04T09:35:57.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:57.301+0000 7f7a2ea00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f7a30161b40 0x7f7a300cfd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:57.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:57.302+0000 7f7a2ea00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f7a30161b40 0x7f7a300cfd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:46710/0 (socket says 172.21.15.6:46710) 2024-03-04T09:35:57.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:57.302+0000 7f7a2ea00640 1 -- 172.21.15.6:0/471864545 learned_addr learned my addr 172.21.15.6:0/471864545 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:35:57.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:57.302+0000 7f7a2e000640 1 --2- 172.21.15.6:0/471864545 >> v2:172.21.15.146:3300/0 conn(0x7f7a300b5060 0x7f7a300d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:57.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:57.302+0000 7f7a2f400640 1 --2- 172.21.15.6:0/471864545 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7a300b54d0 0x7f7a300d2760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:57.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:57.302+0000 7f7a2ea00640 1 -- 172.21.15.6:0/471864545 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7a300b54d0 msgr2=0x7f7a300d2760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:35:57.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:57.302+0000 7f7a2ea00640 1 --2- 172.21.15.6:0/471864545 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7a300b54d0 0x7f7a300d2760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:57.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:57.302+0000 7f7a2ea00640 1 -- 172.21.15.6:0/471864545 >> v2:172.21.15.146:3300/0 conn(0x7f7a300b5060 msgr2=0x7f7a300d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:35:57.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:57.302+0000 7f7a2ea00640 1 --2- 172.21.15.6:0/471864545 >> v2:172.21.15.146:3300/0 conn(0x7f7a300b5060 0x7f7a300d2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:57.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:57.302+0000 7f7a2ea00640 1 -- 172.21.15.6:0/471864545 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a300cefd0 con 0x7f7a30161b40 2024-03-04T09:35:57.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:57.302+0000 7f7a2f400640 1 --2- 172.21.15.6:0/471864545 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7a300b54d0 0x7f7a300d2760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:35:57.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:57.303+0000 7f7a2ea00640 1 --2- 172.21.15.6:0/471864545 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7a30161b40 0x7f7a300cfd90 secure :-1 s=READY pgs=1758 cs=0 l=1 rev1=1 crypto rx=0x7f7a300b9fa0 tx=0x7f7a1802b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec9669655b6d2ca8 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:35:57.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:57.304+0000 7f7a2d600640 1 -- 172.21.15.6:0/471864545 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a18035020 con 0x7f7a30161b40 2024-03-04T09:35:57.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:57.304+0000 7f7a2d600640 1 -- 172.21.15.6:0/471864545 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a1802d040 con 0x7f7a30161b40 2024-03-04T09:35:57.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:57.304+0000 7f7a2d600640 1 -- 172.21.15.6:0/471864545 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a18031a80 con 0x7f7a30161b40 2024-03-04T09:35:57.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:57.305+0000 7f7a35e00640 1 -- 172.21.15.6:0/471864545 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7a30161b40 msgr2=0x7f7a300cfd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:57.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:57.305+0000 7f7a35e00640 1 --2- 172.21.15.6:0/471864545 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7a30161b40 0x7f7a300cfd90 secure :-1 s=READY pgs=1758 cs=0 l=1 rev1=1 crypto rx=0x7f7a300b9fa0 tx=0x7f7a1802b900 comp rx=0 tx=0).stop 2024-03-04T09:35:57.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:57.305+0000 7f7a35e00640 1 -- 172.21.15.6:0/471864545 shutdown_connections 2024-03-04T09:35:57.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:57.305+0000 7f7a35e00640 1 --2- 172.21.15.6:0/471864545 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7a300b54d0 0x7f7a300d2760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:57.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:57.305+0000 7f7a35e00640 1 --2- 172.21.15.6:0/471864545 >> v2:172.21.15.146:3300/0 conn(0x7f7a300b5060 0x7f7a300d2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:57.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:57.305+0000 7f7a35e00640 1 --2- 172.21.15.6:0/471864545 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7a30161b40 0x7f7a300cfd90 unknown :-1 s=CLOSED pgs=1758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:57.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:57.305+0000 7f7a35e00640 1 -- 172.21.15.6:0/471864545 >> 172.21.15.6:0/471864545 conn(0x7f7a3009d220 msgr2=0x7f7a300d3ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:35:57.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:57.305+0000 7f7a35e00640 1 -- 172.21.15.6:0/471864545 shutdown_connections 2024-03-04T09:35:57.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:57.305+0000 7f7a35e00640 1 -- 172.21.15.6:0/471864545 wait complete. 2024-03-04T09:35:57.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:57.306+0000 7f7a35e00640 1 Processor -- start 2024-03-04T09:35:57.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:57.307+0000 7f7a35e00640 1 -- start start 2024-03-04T09:35:57.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:57.308+0000 7f7a35e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7a300b5060 0x7f7a300b7f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:57.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:57.308+0000 7f7a35e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7a300b54d0 0x7f7a30150ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:57.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:57.308+0000 7f7a35e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7a30161b40 0x7f7a3015fa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:57.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:57.308+0000 7f7a35e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a300b2ff0 con 0x7f7a300b5060 2024-03-04T09:35:57.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:57.308+0000 7f7a35e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a300cf590 con 0x7f7a300b54d0 2024-03-04T09:35:57.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:57.308+0000 7f7a35e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a300b3170 con 0x7f7a30161b40 2024-03-04T09:35:57.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:57.308+0000 7f7a2ea00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7a300b5060 0x7f7a300b7f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:57.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:57.308+0000 7f7a2ea00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7a300b5060 0x7f7a300b7f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:46714/0 (socket says 172.21.15.6:46714) 2024-03-04T09:35:57.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:57.308+0000 7f7a2ea00640 1 -- 172.21.15.6:0/50478373 learned_addr learned my addr 172.21.15.6:0/50478373 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:35:57.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:57.308+0000 7f7a2ea00640 1 -- 172.21.15.6:0/50478373 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7a30161b40 msgr2=0x7f7a3015fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:57.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:57.308+0000 7f7a2ea00640 1 --2- 172.21.15.6:0/50478373 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7a30161b40 0x7f7a3015fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:57.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:57.308+0000 7f7a2ea00640 1 -- 172.21.15.6:0/50478373 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7a300b54d0 msgr2=0x7f7a30150ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:57.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:57.308+0000 7f7a2ea00640 1 --2- 172.21.15.6:0/50478373 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7a300b54d0 0x7f7a30150ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:57.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:57.308+0000 7f7a2ea00640 1 -- 172.21.15.6:0/50478373 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a300b6c40 con 0x7f7a300b5060 2024-03-04T09:35:57.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:57.309+0000 7f7a2ea00640 1 --2- 172.21.15.6:0/50478373 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7a300b5060 0x7f7a300b7f10 secure :-1 s=READY pgs=1759 cs=0 l=1 rev1=1 crypto rx=0x7f7a18000c00 tx=0x7f7a180046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:35:57.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:57.309+0000 7f7a27400640 1 -- 172.21.15.6:0/50478373 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a18035040 con 0x7f7a300b5060 2024-03-04T09:35:57.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:57.309+0000 7f7a27400640 1 -- 172.21.15.6:0/50478373 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a18039bf0 con 0x7f7a300b5060 2024-03-04T09:35:57.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:57.309+0000 7f7a27400640 1 -- 172.21.15.6:0/50478373 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a18045480 con 0x7f7a300b5060 2024-03-04T09:35:57.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:57.309+0000 7f7a35e00640 1 -- 172.21.15.6:0/50478373 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a300b6ee0 con 0x7f7a300b5060 2024-03-04T09:35:57.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:57.309+0000 7f7a35e00640 1 -- 172.21.15.6:0/50478373 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a300b7780 con 0x7f7a300b5060 2024-03-04T09:35:57.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:57.310+0000 7f7a35e00640 1 -- 172.21.15.6:0/50478373 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a3004f3a0 con 0x7f7a300b5060 2024-03-04T09:35:57.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:57.311+0000 7f7a27400640 1 -- 172.21.15.6:0/50478373 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7a18038020 con 0x7f7a300b5060 2024-03-04T09:35:57.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:57.312+0000 7f7a27400640 1 --2- 172.21.15.6:0/50478373 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f7a0c06b4d0 0x7f7a0c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:57.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:57.312+0000 7f7a27400640 1 -- 172.21.15.6:0/50478373 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f7a180ac450 con 0x7f7a300b5060 2024-03-04T09:35:57.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:57.313+0000 7f7a2e000640 1 --2- 172.21.15.6:0/50478373 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f7a0c06b4d0 0x7f7a0c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:57.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:57.314+0000 7f7a2e000640 1 --2- 172.21.15.6:0/50478373 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f7a0c06b4d0 0x7f7a0c06d990 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f7a30160650 tx=0x7f7a20009450 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:35:57.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:57.317+0000 7f7a27400640 1 -- 172.21.15.6:0/50478373 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7a1807b280 con 0x7f7a300b5060 2024-03-04T09:35:57.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:57.665+0000 7f7a35e00640 1 -- 172.21.15.6:0/50478373 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7a300b4540 con 0x7f7a300b5060 2024-03-04T09:35:57.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:57.666+0000 7f7a27400640 1 -- 172.21.15.6:0/50478373 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3464 (secure 0 0 0) 0x7f7a1807ab80 con 0x7f7a300b5060 2024-03-04T09:35:57.663 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:35:57.663 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:35:26.088222+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"laggy_since":"2024-03-04T09:33:54.652601+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":9}]} 2024-03-04T09:35:57.663 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 101 2024-03-04T09:35:57.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:57.669+0000 7f7a35e00640 1 -- 172.21.15.6:0/50478373 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f7a0c06b4d0 msgr2=0x7f7a0c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:57.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:57.669+0000 7f7a35e00640 1 --2- 172.21.15.6:0/50478373 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f7a0c06b4d0 0x7f7a0c06d990 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f7a30160650 tx=0x7f7a20009450 comp rx=0 tx=0).stop 2024-03-04T09:35:57.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:57.669+0000 7f7a35e00640 1 -- 172.21.15.6:0/50478373 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7a300b5060 msgr2=0x7f7a300b7f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:57.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:57.669+0000 7f7a35e00640 1 --2- 172.21.15.6:0/50478373 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7a300b5060 0x7f7a300b7f10 secure :-1 s=READY pgs=1759 cs=0 l=1 rev1=1 crypto rx=0x7f7a18000c00 tx=0x7f7a180046c0 comp rx=0 tx=0).stop 2024-03-04T09:35:57.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:57.669+0000 7f7a35e00640 1 -- 172.21.15.6:0/50478373 shutdown_connections 2024-03-04T09:35:57.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:57.669+0000 7f7a35e00640 1 --2- 172.21.15.6:0/50478373 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f7a0c06b4d0 0x7f7a0c06d990 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:57.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:57.669+0000 7f7a35e00640 1 --2- 172.21.15.6:0/50478373 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7a30161b40 0x7f7a3015fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:57.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:57.669+0000 7f7a35e00640 1 --2- 172.21.15.6:0/50478373 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7a300b54d0 0x7f7a30150ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:57.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:57.669+0000 7f7a35e00640 1 --2- 172.21.15.6:0/50478373 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7a300b5060 0x7f7a300b7f10 unknown :-1 s=CLOSED pgs=1759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:57.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:57.669+0000 7f7a35e00640 1 -- 172.21.15.6:0/50478373 >> 172.21.15.6:0/50478373 conn(0x7f7a3009d220 msgr2=0x7f7a300d0570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:35:57.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:57.669+0000 7f7a35e00640 1 -- 172.21.15.6:0/50478373 shutdown_connections 2024-03-04T09:35:57.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:57.669+0000 7f7a35e00640 1 -- 172.21.15.6:0/50478373 wait complete. 2024-03-04T09:35:57.886 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:32:57.159681+0000', 'modified': '2024-03-04T09:35:26.088222+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8214, 'mds_1': 6796}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6796': {'gid': 6796, 'name': 'd', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.146:6837/2458070840', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 2458070840}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 2458070840}]}, 'join_fscid': -1, 'laggy_since': '2024-03-04T09:33:54.652601+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8214': {'gid': 8214, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2446627517', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2446627517}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2446627517}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8214, 'qdb_cluster': [8214]} 2024-03-04T09:35:57.886 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6796:up:stopping 2024-03-04T09:35:57.918 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-04T09:35:58.886 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:35:59.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:59.113+0000 7f7aa2c00640 1 Processor -- start 2024-03-04T09:35:59.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:59.114+0000 7f7aa2c00640 1 -- start start 2024-03-04T09:35:59.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:59.115+0000 7f7aa2c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f7a9c06b6c0 0x7f7a9c0d0910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:59.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:59.115+0000 7f7aa2c00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a9c147bc0 con 0x7f7a9c162420 2024-03-04T09:35:59.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:59.115+0000 7f7aa2c00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a9c0b3d70 con 0x7f7a9c1477e0 2024-03-04T09:35:59.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:59.115+0000 7f7aa2c00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a9c161f90 con 0x7f7a9c06b6c0 2024-03-04T09:35:59.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:59.116+0000 7f7aa2200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f7a9c06b6c0 0x7f7a9c0d0910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:59.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:59.116+0000 7f7aa0e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f7a9c162420 0x7f7a9c06afb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:45398/0 (socket says 172.21.15.6:45398) 2024-03-04T09:35:59.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:59.116+0000 7f7aa2200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f7a9c06b6c0 0x7f7a9c0d0910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:46712/0 (socket says 172.21.15.6:46712) 2024-03-04T09:35:59.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:59.116+0000 7f7aa0e00640 1 -- 172.21.15.6:0/1014075555 learned_addr learned my addr 172.21.15.6:0/1014075555 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:35:59.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:59.116+0000 7f7aa2200640 1 -- 172.21.15.6:0/1014075555 >> v1:172.21.15.146:6789/0 conn(0x7f7a9c1477e0 legacy=0x7f7a9c06ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:35:59.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:59.116+0000 7f7aa2200640 1 -- 172.21.15.6:0/1014075555 >> v1:172.21.15.6:6789/0 conn(0x7f7a9c162420 legacy=0x7f7a9c06afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:59.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:59.116+0000 7f7aa2200640 1 -- 172.21.15.6:0/1014075555 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a9c0b4050 con 0x7f7a9c06b6c0 2024-03-04T09:35:59.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:59.116+0000 7f7a9be00640 1 -- 172.21.15.6:0/1014075555 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 360866887 0 0) 0x7f7a9c147bc0 con 0x7f7a9c162420 2024-03-04T09:35:59.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:59.118+0000 7f7aa2200640 1 --2- 172.21.15.6:0/1014075555 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7a9c06b6c0 0x7f7a9c0d0910 secure :-1 s=READY pgs=1761 cs=0 l=1 rev1=1 crypto rx=0x7f7a94009d90 tx=0x7f7a9402be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d2692b70a3c6a3a server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:35:59.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:59.118+0000 7f7a9be00640 1 -- 172.21.15.6:0/1014075555 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a94039070 con 0x7f7a9c06b6c0 2024-03-04T09:35:59.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:59.118+0000 7f7a9be00640 1 -- 172.21.15.6:0/1014075555 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a9402cbd0 con 0x7f7a9c06b6c0 2024-03-04T09:35:59.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:59.118+0000 7f7a9be00640 1 -- 172.21.15.6:0/1014075555 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a94034440 con 0x7f7a9c06b6c0 2024-03-04T09:35:59.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:59.118+0000 7f7aa2c00640 1 -- 172.21.15.6:0/1014075555 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7a9c06b6c0 msgr2=0x7f7a9c0d0910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:59.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:59.118+0000 7f7aa2c00640 1 --2- 172.21.15.6:0/1014075555 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7a9c06b6c0 0x7f7a9c0d0910 secure :-1 s=READY pgs=1761 cs=0 l=1 rev1=1 crypto rx=0x7f7a94009d90 tx=0x7f7a9402be20 comp rx=0 tx=0).stop 2024-03-04T09:35:59.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:59.119+0000 7f7aa2c00640 1 -- 172.21.15.6:0/1014075555 shutdown_connections 2024-03-04T09:35:59.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:59.119+0000 7f7aa2c00640 1 --2- 172.21.15.6:0/1014075555 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7a9c06b6c0 0x7f7a9c0d0910 unknown :-1 s=CLOSED pgs=1761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:59.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:59.119+0000 7f7aa2c00640 1 -- 172.21.15.6:0/1014075555 >> 172.21.15.6:0/1014075555 conn(0x7f7a9c06c0c0 msgr2=0x7f7a9c06c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:35:59.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:59.119+0000 7f7aa2c00640 1 -- 172.21.15.6:0/1014075555 shutdown_connections 2024-03-04T09:35:59.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:59.119+0000 7f7aa2c00640 1 -- 172.21.15.6:0/1014075555 wait complete. 2024-03-04T09:35:59.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:59.120+0000 7f7aa2c00640 1 Processor -- start 2024-03-04T09:35:59.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:59.120+0000 7f7aa2c00640 1 -- start start 2024-03-04T09:35:59.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:59.121+0000 7f7aa2c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7a9c06b6c0 0x7f7a9c15d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:59.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:59.121+0000 7f7aa2c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7a9c1477e0 0x7f7a9c0b2820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:59.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:59.121+0000 7f7aa2c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7a9c162420 0x7f7a9c0b2d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:59.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:59.121+0000 7f7aa2c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a9c1628a0 con 0x7f7a9c162420 2024-03-04T09:35:59.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:59.121+0000 7f7aa2c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a9c161c20 con 0x7f7a9c1477e0 2024-03-04T09:35:59.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:59.121+0000 7f7aa2c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a9c06f8f0 con 0x7f7a9c06b6c0 2024-03-04T09:35:59.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:59.122+0000 7f7aa2200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7a9c162420 0x7f7a9c0b2d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:59.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:59.122+0000 7f7aa2200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7a9c162420 0x7f7a9c0b2d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:46716/0 (socket says 172.21.15.6:46716) 2024-03-04T09:35:59.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:59.122+0000 7f7aa2200640 1 -- 172.21.15.6:0/1905693382 learned_addr learned my addr 172.21.15.6:0/1905693382 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:35:59.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:59.122+0000 7f7aa1800640 1 --2- 172.21.15.6:0/1905693382 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7a9c06b6c0 0x7f7a9c15d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:59.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:59.122+0000 7f7aa0e00640 1 --2- 172.21.15.6:0/1905693382 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7a9c1477e0 0x7f7a9c0b2820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:59.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:59.122+0000 7f7aa2200640 1 -- 172.21.15.6:0/1905693382 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7a9c06b6c0 msgr2=0x7f7a9c15d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:59.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:59.122+0000 7f7aa2200640 1 --2- 172.21.15.6:0/1905693382 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7a9c06b6c0 0x7f7a9c15d7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:59.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:59.122+0000 7f7aa2200640 1 -- 172.21.15.6:0/1905693382 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7a9c1477e0 msgr2=0x7f7a9c0b2820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:59.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:59.122+0000 7f7aa2200640 1 --2- 172.21.15.6:0/1905693382 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7a9c1477e0 0x7f7a9c0b2820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:59.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:59.122+0000 7f7aa2200640 1 -- 172.21.15.6:0/1905693382 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a9c0cd7e0 con 0x7f7a9c162420 2024-03-04T09:35:59.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:59.123+0000 7f7aa2200640 1 --2- 172.21.15.6:0/1905693382 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7a9c162420 0x7f7a9c0b2d60 secure :-1 s=READY pgs=1762 cs=0 l=1 rev1=1 crypto rx=0x7f7a94037040 tx=0x7f7a94004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:35:59.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:59.123+0000 7f7a99600640 1 -- 172.21.15.6:0/1905693382 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a94044030 con 0x7f7a9c162420 2024-03-04T09:35:59.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:59.123+0000 7f7a99600640 1 -- 172.21.15.6:0/1905693382 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a9402c9d0 con 0x7f7a9c162420 2024-03-04T09:35:59.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:59.123+0000 7f7a99600640 1 -- 172.21.15.6:0/1905693382 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a94039040 con 0x7f7a9c162420 2024-03-04T09:35:59.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:59.123+0000 7f7aa2c00640 1 -- 172.21.15.6:0/1905693382 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a9c0cdac0 con 0x7f7a9c162420 2024-03-04T09:35:59.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:59.123+0000 7f7aa2c00640 1 -- 172.21.15.6:0/1905693382 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a9c0ce080 con 0x7f7a9c162420 2024-03-04T09:35:59.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:59.125+0000 7f7aa2c00640 1 -- 172.21.15.6:0/1905693382 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a9c04f330 con 0x7f7a9c162420 2024-03-04T09:35:59.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:59.125+0000 7f7a99600640 1 -- 172.21.15.6:0/1905693382 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7a94047070 con 0x7f7a9c162420 2024-03-04T09:35:59.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:59.127+0000 7f7a99600640 1 --2- 172.21.15.6:0/1905693382 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f7a7c06b4d0 0x7f7a7c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:35:59.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:59.127+0000 7f7a99600640 1 -- 172.21.15.6:0/1905693382 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f7a940acc40 con 0x7f7a9c162420 2024-03-04T09:35:59.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:59.128+0000 7f7aa1800640 1 --2- 172.21.15.6:0/1905693382 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f7a7c06b4d0 0x7f7a7c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:35:59.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:59.129+0000 7f7aa1800640 1 --2- 172.21.15.6:0/1905693382 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f7a7c06b4d0 0x7f7a7c06d990 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f7a90004ea0 tx=0x7f7a90009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:35:59.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:59.131+0000 7f7a99600640 1 -- 172.21.15.6:0/1905693382 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7a9404e070 con 0x7f7a9c162420 2024-03-04T09:35:59.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:59.370+0000 7f7aa2c00640 1 -- 172.21.15.6:0/1905693382 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7a9c0724b0 con 0x7f7a9c162420 2024-03-04T09:35:59.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:59.371+0000 7f7a99600640 1 -- 172.21.15.6:0/1905693382 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3464 (secure 0 0 0) 0x7f7a9407b9e0 con 0x7f7a9c162420 2024-03-04T09:35:59.368 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:35:59.368 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:35:26.088222+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"laggy_since":"2024-03-04T09:33:54.652601+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":9}]} 2024-03-04T09:35:59.369 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 101 2024-03-04T09:35:59.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:59.374+0000 7f7aa2c00640 1 -- 172.21.15.6:0/1905693382 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f7a7c06b4d0 msgr2=0x7f7a7c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:59.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:59.374+0000 7f7aa2c00640 1 --2- 172.21.15.6:0/1905693382 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f7a7c06b4d0 0x7f7a7c06d990 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f7a90004ea0 tx=0x7f7a90009040 comp rx=0 tx=0).stop 2024-03-04T09:35:59.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:59.374+0000 7f7aa2c00640 1 -- 172.21.15.6:0/1905693382 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7a9c162420 msgr2=0x7f7a9c0b2d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:35:59.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:59.374+0000 7f7aa2c00640 1 --2- 172.21.15.6:0/1905693382 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7a9c162420 0x7f7a9c0b2d60 secure :-1 s=READY pgs=1762 cs=0 l=1 rev1=1 crypto rx=0x7f7a94037040 tx=0x7f7a94004a40 comp rx=0 tx=0).stop 2024-03-04T09:35:59.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:59.374+0000 7f7aa2c00640 1 -- 172.21.15.6:0/1905693382 shutdown_connections 2024-03-04T09:35:59.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:59.374+0000 7f7aa2c00640 1 --2- 172.21.15.6:0/1905693382 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7a9c162420 0x7f7a9c0b2d60 unknown :-1 s=CLOSED pgs=1762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:59.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:59.374+0000 7f7aa2c00640 1 --2- 172.21.15.6:0/1905693382 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7a9c1477e0 0x7f7a9c0b2820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:59.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:59.374+0000 7f7aa2c00640 1 --2- 172.21.15.6:0/1905693382 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f7a7c06b4d0 0x7f7a7c06d990 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:59.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:59.374+0000 7f7aa2c00640 1 --2- 172.21.15.6:0/1905693382 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7a9c06b6c0 0x7f7a9c15d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:35:59.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:59.374+0000 7f7aa2c00640 1 -- 172.21.15.6:0/1905693382 >> 172.21.15.6:0/1905693382 conn(0x7f7a9c06c0c0 msgr2=0x7f7a9c146aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:35:59.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:59.374+0000 7f7aa2c00640 1 -- 172.21.15.6:0/1905693382 shutdown_connections 2024-03-04T09:35:59.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:35:59.374+0000 7f7aa2c00640 1 -- 172.21.15.6:0/1905693382 wait complete. 2024-03-04T09:35:59.593 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:32:57.159681+0000', 'modified': '2024-03-04T09:35:26.088222+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8214, 'mds_1': 6796}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6796': {'gid': 6796, 'name': 'd', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.146:6837/2458070840', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 2458070840}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 2458070840}]}, 'join_fscid': -1, 'laggy_since': '2024-03-04T09:33:54.652601+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8214': {'gid': 8214, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2446627517', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2446627517}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2446627517}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8214, 'qdb_cluster': [8214]} 2024-03-04T09:35:59.593 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6796:up:stopping 2024-03-04T09:35:59.920 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-04T09:36:00.594 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:36:00.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:00.828+0000 7febcf200640 1 Processor -- start 2024-03-04T09:36:00.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:00.828+0000 7febcf200640 1 -- start start 2024-03-04T09:36:00.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:00.829+0000 7febcf200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7febd00d2960 0x7febd00d05c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:00.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:00.829+0000 7febcf200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7febd00d0b00 0x7febd00b3630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:00.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:00.829+0000 7febcf200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febd00d2dc0 con 0x7febd015e070 2024-03-04T09:36:00.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:00.829+0000 7febcf200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febd00d0f70 con 0x7febd00d0b00 2024-03-04T09:36:00.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:00.829+0000 7febcf200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febd00d18c0 con 0x7febd00d2960 2024-03-04T09:36:00.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:00.830+0000 7febcde00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7febd015e070 0x7febd00cf410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:45406/0 (socket says 172.21.15.6:45406) 2024-03-04T09:36:00.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:00.830+0000 7febcde00640 1 -- 172.21.15.6:0/849810119 learned_addr learned my addr 172.21.15.6:0/849810119 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:36:00.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:00.830+0000 7febce800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7febd00d0b00 0x7febd00b3630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:00.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:00.830+0000 7febcd400640 1 --2- 172.21.15.6:0/849810119 >> v2:172.21.15.146:3300/0 conn(0x7febd00d2960 0x7febd00d05c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:00.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:00.830+0000 7febcca00640 1 -- 172.21.15.6:0/849810119 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2474476048 0 0) 0x7febd00d2dc0 con 0x7febd015e070 2024-03-04T09:36:00.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:00.830+0000 7febcca00640 1 -- 172.21.15.6:0/849810119 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febb4003520 con 0x7febd015e070 2024-03-04T09:36:00.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:00.830+0000 7febce800640 1 -- 172.21.15.6:0/849810119 >> v2:172.21.15.146:3300/0 conn(0x7febd00d2960 msgr2=0x7febd00d05c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:36:00.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:00.830+0000 7febce800640 1 --2- 172.21.15.6:0/849810119 >> v2:172.21.15.146:3300/0 conn(0x7febd00d2960 0x7febd00d05c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:00.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:00.830+0000 7febce800640 1 -- 172.21.15.6:0/849810119 >> v1:172.21.15.6:6789/0 conn(0x7febd015e070 legacy=0x7febd00cf410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:00.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:00.830+0000 7febce800640 1 -- 172.21.15.6:0/849810119 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febd00d1ba0 con 0x7febd00d0b00 2024-03-04T09:36:00.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:00.831+0000 7febce800640 1 --2- 172.21.15.6:0/849810119 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7febd00d0b00 0x7febd00b3630 secure :-1 s=READY pgs=1764 cs=0 l=1 rev1=1 crypto rx=0x7febd00b6880 tx=0x7febc802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db980b8ac7dee8eb server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:36:00.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:00.832+0000 7febcca00640 1 -- 172.21.15.6:0/849810119 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febc8039070 con 0x7febd00d0b00 2024-03-04T09:36:00.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:00.832+0000 7febcca00640 1 -- 172.21.15.6:0/849810119 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febc802cc80 con 0x7febd00d0b00 2024-03-04T09:36:00.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:00.832+0000 7febcca00640 1 -- 172.21.15.6:0/849810119 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febc8034640 con 0x7febd00d0b00 2024-03-04T09:36:00.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:00.832+0000 7febcf200640 1 -- 172.21.15.6:0/849810119 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7febd00d0b00 msgr2=0x7febd00b3630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:00.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:00.832+0000 7febcf200640 1 --2- 172.21.15.6:0/849810119 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7febd00d0b00 0x7febd00b3630 secure :-1 s=READY pgs=1764 cs=0 l=1 rev1=1 crypto rx=0x7febd00b6880 tx=0x7febc802bed0 comp rx=0 tx=0).stop 2024-03-04T09:36:00.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:00.832+0000 7febcf200640 1 -- 172.21.15.6:0/849810119 shutdown_connections 2024-03-04T09:36:00.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:00.833+0000 7febcf200640 1 --2- 172.21.15.6:0/849810119 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7febd00d0b00 0x7febd00b3630 unknown :-1 s=CLOSED pgs=1764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:00.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:00.833+0000 7febcf200640 1 --2- 172.21.15.6:0/849810119 >> v2:172.21.15.146:3300/0 conn(0x7febd00d2960 0x7febd00d05c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:00.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:00.833+0000 7febcf200640 1 -- 172.21.15.6:0/849810119 >> 172.21.15.6:0/849810119 conn(0x7febd008d2a0 msgr2=0x7febd00d3cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:36:00.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:00.833+0000 7febcf200640 1 -- 172.21.15.6:0/849810119 shutdown_connections 2024-03-04T09:36:00.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:00.833+0000 7febcf200640 1 -- 172.21.15.6:0/849810119 wait complete. 2024-03-04T09:36:00.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:00.834+0000 7febcf200640 1 Processor -- start 2024-03-04T09:36:00.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:00.834+0000 7febcf200640 1 -- start start 2024-03-04T09:36:00.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:00.835+0000 7febcf200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7febd00d0b00 0x7febd00b4c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:00.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:00.835+0000 7febcf200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7febd00d2960 0x7febd01f41d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:00.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:00.835+0000 7febcf200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7febd015e070 0x7febd014f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:00.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:00.835+0000 7febcf200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febd00d2440 con 0x7febd00d0b00 2024-03-04T09:36:00.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:00.835+0000 7febcf200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febd00d2160 con 0x7febd015e070 2024-03-04T09:36:00.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:00.835+0000 7febcf200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febd00d45b0 con 0x7febd00d2960 2024-03-04T09:36:00.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:00.836+0000 7febcde00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7febd00d0b00 0x7febd00b4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:00.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:00.836+0000 7febcde00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7febd00d0b00 0x7febd00b4c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:46732/0 (socket says 172.21.15.6:46732) 2024-03-04T09:36:00.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:00.836+0000 7febcde00640 1 -- 172.21.15.6:0/2774466787 learned_addr learned my addr 172.21.15.6:0/2774466787 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:36:00.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:00.836+0000 7febcde00640 1 -- 172.21.15.6:0/2774466787 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7febd00d2960 msgr2=0x7febd01f41d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:00.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:00.836+0000 7febcd400640 1 --2- 172.21.15.6:0/2774466787 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7febd00d2960 0x7febd01f41d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:00.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:00.836+0000 7febcde00640 1 --2- 172.21.15.6:0/2774466787 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7febd00d2960 0x7febd01f41d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:00.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:00.836+0000 7febce800640 1 --2- 172.21.15.6:0/2774466787 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7febd015e070 0x7febd014f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:00.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:00.836+0000 7febcde00640 1 -- 172.21.15.6:0/2774466787 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7febd015e070 msgr2=0x7febd014f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:00.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:00.836+0000 7febcde00640 1 --2- 172.21.15.6:0/2774466787 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7febd015e070 0x7febd014f010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:00.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:00.836+0000 7febcde00640 1 -- 172.21.15.6:0/2774466787 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febd01f4710 con 0x7febd00d0b00 2024-03-04T09:36:00.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:00.837+0000 7febcde00640 1 --2- 172.21.15.6:0/2774466787 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7febd00d0b00 0x7febd00b4c10 secure :-1 s=READY pgs=1765 cs=0 l=1 rev1=1 crypto rx=0x7febc000c390 tx=0x7febc000c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:36:00.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:00.837+0000 7febc6000640 1 -- 172.21.15.6:0/2774466787 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febc0017070 con 0x7febd00d0b00 2024-03-04T09:36:00.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:00.837+0000 7febc6000640 1 -- 172.21.15.6:0/2774466787 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febc0005ce0 con 0x7febd00d0b00 2024-03-04T09:36:00.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:00.837+0000 7febcf200640 1 -- 172.21.15.6:0/2774466787 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febd00b3b70 con 0x7febd00d0b00 2024-03-04T09:36:00.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:00.837+0000 7febc6000640 1 -- 172.21.15.6:0/2774466787 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febc00072f0 con 0x7febd00d0b00 2024-03-04T09:36:00.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:00.837+0000 7febcf200640 1 -- 172.21.15.6:0/2774466787 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febd00b4130 con 0x7febd00d0b00 2024-03-04T09:36:00.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:00.838+0000 7febcf200640 1 -- 172.21.15.6:0/2774466787 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febd004f330 con 0x7febd00d0b00 2024-03-04T09:36:00.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:00.839+0000 7febc6000640 1 -- 172.21.15.6:0/2774466787 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7febc001a070 con 0x7febd00d0b00 2024-03-04T09:36:00.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:00.840+0000 7febc6000640 1 --2- 172.21.15.6:0/2774466787 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7febb006b4d0 0x7febb006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:00.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:00.840+0000 7febc6000640 1 -- 172.21.15.6:0/2774466787 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7febc0089700 con 0x7febd00d0b00 2024-03-04T09:36:00.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:00.841+0000 7febcd400640 1 --2- 172.21.15.6:0/2774466787 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7febb006b4d0 0x7febb006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:00.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:00.842+0000 7febcd400640 1 --2- 172.21.15.6:0/2774466787 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7febb006b4d0 0x7febb006d990 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7febd01f1340 tx=0x7febbc008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:36:00.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:00.845+0000 7febc6000640 1 -- 172.21.15.6:0/2774466787 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7febd01f4710 con 0x7febd00d0b00 2024-03-04T09:36:01.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:01.135+0000 7febcf200640 1 -- 172.21.15.6:0/2774466787 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7febd00d1370 con 0x7febd00d0b00 2024-03-04T09:36:01.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:01.136+0000 7febc6000640 1 -- 172.21.15.6:0/2774466787 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3464 (secure 0 0 0) 0x7febc00584a0 con 0x7febd00d0b00 2024-03-04T09:36:01.133 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:36:01.134 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:35:26.088222+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"laggy_since":"2024-03-04T09:33:54.652601+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":9}]} 2024-03-04T09:36:01.134 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 101 2024-03-04T09:36:01.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:01.139+0000 7febcf200640 1 -- 172.21.15.6:0/2774466787 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7febb006b4d0 msgr2=0x7febb006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:01.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:01.139+0000 7febcf200640 1 --2- 172.21.15.6:0/2774466787 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7febb006b4d0 0x7febb006d990 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7febd01f1340 tx=0x7febbc008040 comp rx=0 tx=0).stop 2024-03-04T09:36:01.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:01.139+0000 7febcf200640 1 -- 172.21.15.6:0/2774466787 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7febd00d0b00 msgr2=0x7febd00b4c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:01.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:01.139+0000 7febcf200640 1 --2- 172.21.15.6:0/2774466787 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7febd00d0b00 0x7febd00b4c10 secure :-1 s=READY pgs=1765 cs=0 l=1 rev1=1 crypto rx=0x7febc000c390 tx=0x7febc000c850 comp rx=0 tx=0).stop 2024-03-04T09:36:01.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:01.139+0000 7febcf200640 1 -- 172.21.15.6:0/2774466787 shutdown_connections 2024-03-04T09:36:01.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:01.139+0000 7febcf200640 1 --2- 172.21.15.6:0/2774466787 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7febd015e070 0x7febd014f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:01.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:01.139+0000 7febcf200640 1 --2- 172.21.15.6:0/2774466787 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7febb006b4d0 0x7febb006d990 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:01.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:01.139+0000 7febcf200640 1 --2- 172.21.15.6:0/2774466787 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7febd00d2960 0x7febd01f41d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:01.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:01.139+0000 7febcf200640 1 --2- 172.21.15.6:0/2774466787 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7febd00d0b00 0x7febd00b4c10 unknown :-1 s=CLOSED pgs=1765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:01.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:01.139+0000 7febcf200640 1 -- 172.21.15.6:0/2774466787 >> 172.21.15.6:0/2774466787 conn(0x7febd008d2a0 msgr2=0x7febd006cb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:36:01.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:01.139+0000 7febcf200640 1 -- 172.21.15.6:0/2774466787 shutdown_connections 2024-03-04T09:36:01.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:01.139+0000 7febcf200640 1 -- 172.21.15.6:0/2774466787 wait complete. 2024-03-04T09:36:01.355 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:32:57.159681+0000', 'modified': '2024-03-04T09:35:26.088222+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8214, 'mds_1': 6796}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6796': {'gid': 6796, 'name': 'd', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.146:6837/2458070840', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 2458070840}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 2458070840}]}, 'join_fscid': -1, 'laggy_since': '2024-03-04T09:33:54.652601+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8214': {'gid': 8214, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2446627517', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2446627517}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2446627517}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8214, 'qdb_cluster': [8214]} 2024-03-04T09:36:01.355 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6796:up:stopping 2024-03-04T09:36:01.922 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-04T09:36:02.356 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:36:02.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:02.588+0000 7f5ef7200640 1 Processor -- start 2024-03-04T09:36:02.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:02.589+0000 7f5ef7200640 1 -- start start 2024-03-04T09:36:02.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:02.590+0000 7f5ef7200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f5ef80b2340 0x7f5ef8168cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:02.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:02.590+0000 7f5ef7200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ef80d1120 con 0x7f5ef815dec0 2024-03-04T09:36:02.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:02.590+0000 7f5ef7200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ef80d1400 con 0x7f5ef815e2a0 2024-03-04T09:36:02.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:02.590+0000 7f5ef7200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ef80b0ea0 con 0x7f5ef80b2340 2024-03-04T09:36:02.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:02.590+0000 7f5ef5e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f5ef815dec0 0x7f5ef80ce810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:45414/0 (socket says 172.21.15.6:45414) 2024-03-04T09:36:02.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:02.591+0000 7f5ef5e00640 1 -- 172.21.15.6:0/857080527 learned_addr learned my addr 172.21.15.6:0/857080527 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:36:02.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:02.591+0000 7f5ef6800640 1 --2- 172.21.15.6:0/857080527 >> v2:172.21.15.146:3300/0 conn(0x7f5ef80b2340 0x7f5ef8168cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:02.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:02.591+0000 7f5ef4a00640 1 -- 172.21.15.6:0/857080527 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2716502349 0 0) 0x7f5ef80d1120 con 0x7f5ef815dec0 2024-03-04T09:36:02.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:02.591+0000 7f5ef4a00640 1 -- 172.21.15.6:0/857080527 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5edc003520 con 0x7f5ef815dec0 2024-03-04T09:36:02.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:02.591+0000 7f5ef4a00640 1 -- 172.21.15.6:0/857080527 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1788569850 0 0) 0x7f5edc003520 con 0x7f5ef815dec0 2024-03-04T09:36:02.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:02.591+0000 7f5ef4a00640 1 -- 172.21.15.6:0/857080527 >> v2:172.21.15.146:3300/0 conn(0x7f5ef80b2340 msgr2=0x7f5ef8168cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:36:02.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:02.591+0000 7f5ef4a00640 1 --2- 172.21.15.6:0/857080527 >> v2:172.21.15.146:3300/0 conn(0x7f5ef80b2340 0x7f5ef8168cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:02.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:02.591+0000 7f5ef4a00640 1 -- 172.21.15.6:0/857080527 >> v1:172.21.15.146:6789/0 conn(0x7f5ef815e2a0 legacy=0x7f5ef80cef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:02.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:02.591+0000 7f5ef4a00640 1 -- 172.21.15.6:0/857080527 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ef80b0b30 con 0x7f5ef815dec0 2024-03-04T09:36:02.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:02.591+0000 7f5ef4a00640 1 -- 172.21.15.6:0/857080527 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7f5ee80019f0 con 0x7f5ef815dec0 2024-03-04T09:36:02.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:02.592+0000 7f5ef4a00640 1 -- 172.21.15.6:0/857080527 >> v1:172.21.15.6:6789/0 conn(0x7f5ef815dec0 legacy=0x7f5ef80ce810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:02.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:02.592+0000 7f5ef4a00640 1 --2- 172.21.15.6:0/857080527 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5edc003a00 0x7f5edc003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:02.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:02.592+0000 7f5ef4a00640 1 --2- 172.21.15.6:0/857080527 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5edc004800 0x7f5edc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:02.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:02.592+0000 7f5ef4a00640 1 --2- 172.21.15.6:0/857080527 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5edc0051c0 0x7f5edc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:02.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:02.592+0000 7f5ef4a00640 1 -- 172.21.15.6:0/857080527 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ef80b0ea0 con 0x7f5edc003a00 2024-03-04T09:36:02.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:02.592+0000 7f5ef4a00640 1 -- 172.21.15.6:0/857080527 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5edc00c0c0 con 0x7f5edc004800 2024-03-04T09:36:02.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:02.592+0000 7f5ef4a00640 1 -- 172.21.15.6:0/857080527 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5edc00c2c0 con 0x7f5edc0051c0 2024-03-04T09:36:02.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:02.592+0000 7f5ef5e00640 1 --2- 172.21.15.6:0/857080527 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5edc003a00 0x7f5edc003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:02.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:02.592+0000 7f5ef5400640 1 --2- 172.21.15.6:0/857080527 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5edc004800 0x7f5edc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:02.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:02.592+0000 7f5ef5e00640 1 -- 172.21.15.6:0/857080527 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5edc0051c0 msgr2=0x7f5edc00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:36:02.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:02.592+0000 7f5ef5e00640 1 --2- 172.21.15.6:0/857080527 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5edc0051c0 0x7f5edc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:02.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:02.592+0000 7f5ef5e00640 1 -- 172.21.15.6:0/857080527 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5edc004800 msgr2=0x7f5edc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:36:02.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:02.592+0000 7f5ef5e00640 1 --2- 172.21.15.6:0/857080527 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5edc004800 0x7f5edc004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:02.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:02.592+0000 7f5ef6800640 1 -- 172.21.15.6:0/857080527 reap_dead start 2024-03-04T09:36:02.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:02.592+0000 7f5ef5e00640 1 -- 172.21.15.6:0/857080527 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5edc003520 con 0x7f5edc003a00 2024-03-04T09:36:02.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:02.593+0000 7f5ef5e00640 1 --2- 172.21.15.6:0/857080527 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5edc003a00 0x7f5edc003e20 secure :-1 s=READY pgs=1767 cs=0 l=1 rev1=1 crypto rx=0x7f5ee800a7a0 tx=0x7f5edc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cd89e200f8a7144 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:36:02.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:02.594+0000 7f5ef4a00640 1 -- 172.21.15.6:0/857080527 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ee800a3f0 con 0x7f5edc003a00 2024-03-04T09:36:02.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:02.594+0000 7f5ef4a00640 1 -- 172.21.15.6:0/857080527 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ee8034630 con 0x7f5edc003a00 2024-03-04T09:36:02.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:02.594+0000 7f5ef4a00640 1 -- 172.21.15.6:0/857080527 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ee8034a20 con 0x7f5edc003a00 2024-03-04T09:36:02.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:02.594+0000 7f5ef7200640 1 -- 172.21.15.6:0/857080527 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5edc003a00 msgr2=0x7f5edc003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:02.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:02.594+0000 7f5ef7200640 1 --2- 172.21.15.6:0/857080527 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5edc003a00 0x7f5edc003e20 secure :-1 s=READY pgs=1767 cs=0 l=1 rev1=1 crypto rx=0x7f5ee800a7a0 tx=0x7f5edc005830 comp rx=0 tx=0).stop 2024-03-04T09:36:02.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:02.594+0000 7f5ef7200640 1 -- 172.21.15.6:0/857080527 shutdown_connections 2024-03-04T09:36:02.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:02.594+0000 7f5ef7200640 1 --2- 172.21.15.6:0/857080527 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5edc003a00 0x7f5edc003e20 unknown :-1 s=CLOSED pgs=1767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:02.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:02.594+0000 7f5ef7200640 1 -- 172.21.15.6:0/857080527 >> 172.21.15.6:0/857080527 conn(0x7f5ef80d3d40 msgr2=0x7f5ef80d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:36:02.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:02.595+0000 7f5ef7200640 1 -- 172.21.15.6:0/857080527 shutdown_connections 2024-03-04T09:36:02.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:02.595+0000 7f5ef7200640 1 -- 172.21.15.6:0/857080527 wait complete. 2024-03-04T09:36:02.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:02.596+0000 7f5ef7200640 1 Processor -- start 2024-03-04T09:36:02.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:02.597+0000 7f5ef7200640 1 -- start start 2024-03-04T09:36:02.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:02.597+0000 7f5ef7200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5edc003a00 0x7f5ef815c360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:02.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:02.597+0000 7f5ef7200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5ef81f4c60 0x7f5ef80c8280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:02.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:02.598+0000 7f5ef7200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5ef81f50e0 0x7f5ef80c87c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:02.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:02.598+0000 7f5ef7200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ef80b14b0 con 0x7f5ef81f4c60 2024-03-04T09:36:02.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:02.598+0000 7f5ef7200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ef80b1330 con 0x7f5edc003a00 2024-03-04T09:36:02.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:02.598+0000 7f5ef7200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ef80b1630 con 0x7f5ef81f50e0 2024-03-04T09:36:02.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:02.598+0000 7f5ef5400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5ef81f4c60 0x7f5ef80c8280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:02.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:02.598+0000 7f5ef5400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5ef81f4c60 0x7f5ef80c8280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:46754/0 (socket says 172.21.15.6:46754) 2024-03-04T09:36:02.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:02.598+0000 7f5ef5400640 1 -- 172.21.15.6:0/4037667519 learned_addr learned my addr 172.21.15.6:0/4037667519 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:36:02.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:02.598+0000 7f5ef6800640 1 --2- 172.21.15.6:0/4037667519 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5ef81f50e0 0x7f5ef80c87c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:02.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:02.598+0000 7f5ef5e00640 1 --2- 172.21.15.6:0/4037667519 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5edc003a00 0x7f5ef815c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:02.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:02.598+0000 7f5ef5400640 1 -- 172.21.15.6:0/4037667519 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5ef81f50e0 msgr2=0x7f5ef80c87c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:02.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:02.598+0000 7f5ef5400640 1 --2- 172.21.15.6:0/4037667519 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5ef81f50e0 0x7f5ef80c87c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:02.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:02.598+0000 7f5ef5400640 1 -- 172.21.15.6:0/4037667519 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5edc003a00 msgr2=0x7f5ef815c360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:02.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:02.598+0000 7f5ef5400640 1 --2- 172.21.15.6:0/4037667519 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5edc003a00 0x7f5ef815c360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:02.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:02.598+0000 7f5ef5400640 1 -- 172.21.15.6:0/4037667519 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ef80b6a00 con 0x7f5ef81f4c60 2024-03-04T09:36:02.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:02.598+0000 7f5ef6800640 1 --2- 172.21.15.6:0/4037667519 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5ef81f50e0 0x7f5ef80c87c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:36:02.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:02.599+0000 7f5ef5400640 1 --2- 172.21.15.6:0/4037667519 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5ef81f4c60 0x7f5ef80c8280 secure :-1 s=READY pgs=1768 cs=0 l=1 rev1=1 crypto rx=0x7f5ee000d470 tx=0x7f5ee000d930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:36:02.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:02.599+0000 7f5eee000640 1 -- 172.21.15.6:0/4037667519 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ee0005480 con 0x7f5ef81f4c60 2024-03-04T09:36:02.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:02.599+0000 7f5eee000640 1 -- 172.21.15.6:0/4037667519 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ee0017070 con 0x7f5ef81f4c60 2024-03-04T09:36:02.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:02.599+0000 7f5eee000640 1 -- 172.21.15.6:0/4037667519 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ee0003810 con 0x7f5ef81f4c60 2024-03-04T09:36:02.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:02.599+0000 7f5ef7200640 1 -- 172.21.15.6:0/4037667519 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ef80b9300 con 0x7f5ef81f4c60 2024-03-04T09:36:02.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:02.599+0000 7f5ef7200640 1 -- 172.21.15.6:0/4037667519 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ef80b6f80 con 0x7f5ef81f4c60 2024-03-04T09:36:02.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:02.600+0000 7f5ef7200640 1 -- 172.21.15.6:0/4037667519 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ef804f330 con 0x7f5ef81f4c60 2024-03-04T09:36:02.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:02.601+0000 7f5eee000640 1 -- 172.21.15.6:0/4037667519 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5ee00039b0 con 0x7f5ef81f4c60 2024-03-04T09:36:02.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:02.602+0000 7f5eee000640 1 --2- 172.21.15.6:0/4037667519 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f5ed406b4d0 0x7f5ed406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:02.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:02.602+0000 7f5eee000640 1 -- 172.21.15.6:0/4037667519 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f5ee0089f80 con 0x7f5ef81f4c60 2024-03-04T09:36:02.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:02.603+0000 7f5ef5e00640 1 --2- 172.21.15.6:0/4037667519 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f5ed406b4d0 0x7f5ed406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:02.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:02.604+0000 7f5ef5e00640 1 --2- 172.21.15.6:0/4037667519 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f5ed406b4d0 0x7f5ed406d990 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f5ee8034000 tx=0x7f5ee8036040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:36:02.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:02.607+0000 7f5eee000640 1 -- 172.21.15.6:0/4037667519 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5ee0058db0 con 0x7f5ef81f4c60 2024-03-04T09:36:02.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:02.854+0000 7f5ef7200640 1 -- 172.21.15.6:0/4037667519 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5ef80d02f0 con 0x7f5ef81f4c60 2024-03-04T09:36:02.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:02.855+0000 7f5eee000640 1 -- 172.21.15.6:0/4037667519 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3464 (secure 0 0 0) 0x7f5ee00586b0 con 0x7f5ef81f4c60 2024-03-04T09:36:02.852 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:36:02.852 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:35:26.088222+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"laggy_since":"2024-03-04T09:33:54.652601+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":9}]} 2024-03-04T09:36:02.852 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 101 2024-03-04T09:36:02.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:02.858+0000 7f5ef7200640 1 -- 172.21.15.6:0/4037667519 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f5ed406b4d0 msgr2=0x7f5ed406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:02.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:02.858+0000 7f5ef7200640 1 --2- 172.21.15.6:0/4037667519 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f5ed406b4d0 0x7f5ed406d990 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f5ee8034000 tx=0x7f5ee8036040 comp rx=0 tx=0).stop 2024-03-04T09:36:02.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:02.858+0000 7f5ef7200640 1 -- 172.21.15.6:0/4037667519 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5ef81f4c60 msgr2=0x7f5ef80c8280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:02.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:02.858+0000 7f5ef7200640 1 --2- 172.21.15.6:0/4037667519 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5ef81f4c60 0x7f5ef80c8280 secure :-1 s=READY pgs=1768 cs=0 l=1 rev1=1 crypto rx=0x7f5ee000d470 tx=0x7f5ee000d930 comp rx=0 tx=0).stop 2024-03-04T09:36:02.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:02.858+0000 7f5ef7200640 1 -- 172.21.15.6:0/4037667519 shutdown_connections 2024-03-04T09:36:02.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:02.858+0000 7f5ef7200640 1 --2- 172.21.15.6:0/4037667519 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f5ed406b4d0 0x7f5ed406d990 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:02.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:02.858+0000 7f5ef7200640 1 --2- 172.21.15.6:0/4037667519 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5ef81f50e0 0x7f5ef80c87c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:02.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:02.858+0000 7f5ef7200640 1 --2- 172.21.15.6:0/4037667519 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5ef81f4c60 0x7f5ef80c8280 unknown :-1 s=CLOSED pgs=1768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:02.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:02.858+0000 7f5ef7200640 1 --2- 172.21.15.6:0/4037667519 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5edc003a00 0x7f5ef815c360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:02.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:02.858+0000 7f5ef7200640 1 -- 172.21.15.6:0/4037667519 >> 172.21.15.6:0/4037667519 conn(0x7f5ef80d3d40 msgr2=0x7f5ef80cf260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:36:02.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:02.858+0000 7f5ef7200640 1 -- 172.21.15.6:0/4037667519 shutdown_connections 2024-03-04T09:36:02.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:02.858+0000 7f5ef7200640 1 -- 172.21.15.6:0/4037667519 wait complete. 2024-03-04T09:36:03.076 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:32:57.159681+0000', 'modified': '2024-03-04T09:35:26.088222+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8214, 'mds_1': 6796}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6796': {'gid': 6796, 'name': 'd', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.146:6837/2458070840', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 2458070840}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 2458070840}]}, 'join_fscid': -1, 'laggy_since': '2024-03-04T09:33:54.652601+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8214': {'gid': 8214, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2446627517', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2446627517}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2446627517}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8214, 'qdb_cluster': [8214]} 2024-03-04T09:36:03.077 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6796:up:stopping 2024-03-04T09:36:03.924 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-04T09:36:04.078 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:36:04.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:04.307+0000 7f5891400640 1 Processor -- start 2024-03-04T09:36:04.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:04.307+0000 7f5891400640 1 -- start start 2024-03-04T09:36:04.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:04.308+0000 7f5891400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f588c070f10 0x7f588c0cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:04.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:04.308+0000 7f5891400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f588c0cce90 0x7f588c15c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:04.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:04.308+0000 7f5891400640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f588c0714d0 con 0x7f588c145300 2024-03-04T09:36:04.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:04.308+0000 7f5891400640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f588c0cd300 con 0x7f588c0cce90 2024-03-04T09:36:04.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:04.308+0000 7f5891400640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f588c0717b0 con 0x7f588c070f10 2024-03-04T09:36:04.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:04.308+0000 7f5890a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f588c0cce90 0x7f588c15c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:04.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:04.309+0000 7f5890a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f588c0cce90 0x7f588c15c570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:46768/0 (socket says 172.21.15.6:46768) 2024-03-04T09:36:04.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:04.309+0000 7f5890a00640 1 -- 172.21.15.6:0/3740846343 learned_addr learned my addr 172.21.15.6:0/3740846343 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:36:04.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:04.309+0000 7f588b400640 1 --2- 172.21.15.6:0/3740846343 >> v2:172.21.15.146:3300/0 conn(0x7f588c070f10 0x7f588c0cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:04.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:04.309+0000 7f5890a00640 1 -- 172.21.15.6:0/3740846343 >> v2:172.21.15.146:3300/0 conn(0x7f588c070f10 msgr2=0x7f588c0cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:36:04.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:04.309+0000 7f5890a00640 1 --2- 172.21.15.6:0/3740846343 >> v2:172.21.15.146:3300/0 conn(0x7f588c070f10 0x7f588c0cc950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:04.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:04.309+0000 7f5890a00640 1 -- 172.21.15.6:0/3740846343 >> v1:172.21.15.146:6789/0 conn(0x7f588c145300 legacy=0x7f588c0b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:36:04.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:04.309+0000 7f5890a00640 1 -- 172.21.15.6:0/3740846343 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f588c071a90 con 0x7f588c0cce90 2024-03-04T09:36:04.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:04.310+0000 7f5890a00640 1 --2- 172.21.15.6:0/3740846343 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f588c0cce90 0x7f588c15c570 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 crypto rx=0x7f588c0b5150 tx=0x7f588402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=875cb813130144e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:36:04.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:04.311+0000 7f588aa00640 1 -- 172.21.15.6:0/3740846343 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f588402e070 con 0x7f588c0cce90 2024-03-04T09:36:04.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:04.311+0000 7f588aa00640 1 -- 172.21.15.6:0/3740846343 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5884037070 con 0x7f588c0cce90 2024-03-04T09:36:04.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:04.311+0000 7f588aa00640 1 -- 172.21.15.6:0/3740846343 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5884031a00 con 0x7f588c0cce90 2024-03-04T09:36:04.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:04.311+0000 7f5891400640 1 -- 172.21.15.6:0/3740846343 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f588c0cce90 msgr2=0x7f588c15c570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:04.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:04.311+0000 7f5891400640 1 --2- 172.21.15.6:0/3740846343 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f588c0cce90 0x7f588c15c570 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 crypto rx=0x7f588c0b5150 tx=0x7f588402b880 comp rx=0 tx=0).stop 2024-03-04T09:36:04.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:04.312+0000 7f5891400640 1 -- 172.21.15.6:0/3740846343 shutdown_connections 2024-03-04T09:36:04.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:04.312+0000 7f5891400640 1 --2- 172.21.15.6:0/3740846343 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f588c0cce90 0x7f588c15c570 unknown :-1 s=CLOSED pgs=1769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:04.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:04.312+0000 7f5891400640 1 --2- 172.21.15.6:0/3740846343 >> v2:172.21.15.146:3300/0 conn(0x7f588c070f10 0x7f588c0cc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:04.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:04.312+0000 7f5891400640 1 -- 172.21.15.6:0/3740846343 >> 172.21.15.6:0/3740846343 conn(0x7f588c06b3d0 msgr2=0x7f588c06b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:36:04.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:04.312+0000 7f5891400640 1 -- 172.21.15.6:0/3740846343 shutdown_connections 2024-03-04T09:36:04.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:04.312+0000 7f5891400640 1 -- 172.21.15.6:0/3740846343 wait complete. 2024-03-04T09:36:04.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:04.314+0000 7f5891400640 1 Processor -- start 2024-03-04T09:36:04.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:04.314+0000 7f5891400640 1 -- start start 2024-03-04T09:36:04.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:04.316+0000 7f5891400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f588c070f10 0x7f588c1f38f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:04.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:04.316+0000 7f5891400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f588c0cce90 0x7f588c143a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:04.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:04.316+0000 7f5891400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f588c145300 0x7f588c1f6e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:04.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:04.316+0000 7f5891400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f588c15cab0 con 0x7f588c0cce90 2024-03-04T09:36:04.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:04.316+0000 7f5891400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f588c1456e0 con 0x7f588c145300 2024-03-04T09:36:04.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:04.316+0000 7f5891400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f588c15cc30 con 0x7f588c070f10 2024-03-04T09:36:04.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:04.316+0000 7f588b400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f588c0cce90 0x7f588c143a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:04.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:04.316+0000 7f588b400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f588c0cce90 0x7f588c143a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:46778/0 (socket says 172.21.15.6:46778) 2024-03-04T09:36:04.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:04.316+0000 7f588b400640 1 -- 172.21.15.6:0/2089433379 learned_addr learned my addr 172.21.15.6:0/2089433379 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:36:04.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:04.316+0000 7f5890a00640 1 --2- 172.21.15.6:0/2089433379 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f588c145300 0x7f588c1f6e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:04.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:04.316+0000 7f588be00640 1 --2- 172.21.15.6:0/2089433379 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f588c070f10 0x7f588c1f38f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:04.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:04.317+0000 7f588b400640 1 -- 172.21.15.6:0/2089433379 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f588c070f10 msgr2=0x7f588c1f38f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:04.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:04.317+0000 7f588b400640 1 --2- 172.21.15.6:0/2089433379 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f588c070f10 0x7f588c1f38f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:04.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:04.317+0000 7f588b400640 1 -- 172.21.15.6:0/2089433379 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f588c145300 msgr2=0x7f588c1f6e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:04.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:04.317+0000 7f588b400640 1 --2- 172.21.15.6:0/2089433379 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f588c145300 0x7f588c1f6e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:04.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:04.317+0000 7f588b400640 1 -- 172.21.15.6:0/2089433379 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f588c0c4c60 con 0x7f588c0cce90 2024-03-04T09:36:04.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:04.317+0000 7f588b400640 1 --2- 172.21.15.6:0/2089433379 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f588c0cce90 0x7f588c143a70 secure :-1 s=READY pgs=1770 cs=0 l=1 rev1=1 crypto rx=0x7f587400d800 tx=0x7f587400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:36:04.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:04.317+0000 7f5883e00640 1 -- 172.21.15.6:0/2089433379 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f587400b6c0 con 0x7f588c0cce90 2024-03-04T09:36:04.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:04.317+0000 7f5883e00640 1 -- 172.21.15.6:0/2089433379 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5874019070 con 0x7f588c0cce90 2024-03-04T09:36:04.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:04.317+0000 7f5891400640 1 -- 172.21.15.6:0/2089433379 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f588c0c4f40 con 0x7f588c0cce90 2024-03-04T09:36:04.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:04.317+0000 7f5883e00640 1 -- 172.21.15.6:0/2089433379 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f587400ba80 con 0x7f588c0cce90 2024-03-04T09:36:04.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:04.318+0000 7f5891400640 1 -- 172.21.15.6:0/2089433379 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f588c1e79b0 con 0x7f588c0cce90 2024-03-04T09:36:04.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:04.319+0000 7f5891400640 1 -- 172.21.15.6:0/2089433379 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f588c04f330 con 0x7f588c0cce90 2024-03-04T09:36:04.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:04.320+0000 7f5883e00640 1 -- 172.21.15.6:0/2089433379 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f587400bc20 con 0x7f588c0cce90 2024-03-04T09:36:04.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:04.321+0000 7f5883e00640 1 --2- 172.21.15.6:0/2089433379 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f586806b500 0x7f586806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:04.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:04.321+0000 7f5883e00640 1 -- 172.21.15.6:0/2089433379 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f587408a740 con 0x7f588c0cce90 2024-03-04T09:36:04.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:04.322+0000 7f588be00640 1 --2- 172.21.15.6:0/2089433379 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f586806b500 0x7f586806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:04.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:04.323+0000 7f588be00640 1 --2- 172.21.15.6:0/2089433379 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f586806b500 0x7f586806d9c0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f587c009f20 tx=0x7f587c0094d0 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:36:04.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:04.329+0000 7f5883e00640 1 -- 172.21.15.6:0/2089433379 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f587401c360 con 0x7f588c0cce90 2024-03-04T09:36:04.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:04.581+0000 7f5891400640 1 -- 172.21.15.6:0/2089433379 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f588c072720 con 0x7f588c0cce90 2024-03-04T09:36:04.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:04.582+0000 7f5883e00640 1 -- 172.21.15.6:0/2089433379 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3464 (secure 0 0 0) 0x7f58740594e0 con 0x7f588c0cce90 2024-03-04T09:36:04.579 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:36:04.579 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:35:26.088222+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"laggy_since":"2024-03-04T09:33:54.652601+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":9}]} 2024-03-04T09:36:04.580 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 101 2024-03-04T09:36:04.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:04.585+0000 7f5891400640 1 -- 172.21.15.6:0/2089433379 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f586806b500 msgr2=0x7f586806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:04.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:04.585+0000 7f5891400640 1 --2- 172.21.15.6:0/2089433379 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f586806b500 0x7f586806d9c0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f587c009f20 tx=0x7f587c0094d0 comp rx=0 tx=0).stop 2024-03-04T09:36:04.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:04.585+0000 7f5891400640 1 -- 172.21.15.6:0/2089433379 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f588c0cce90 msgr2=0x7f588c143a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:04.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:04.585+0000 7f5891400640 1 --2- 172.21.15.6:0/2089433379 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f588c0cce90 0x7f588c143a70 secure :-1 s=READY pgs=1770 cs=0 l=1 rev1=1 crypto rx=0x7f587400d800 tx=0x7f587400dcc0 comp rx=0 tx=0).stop 2024-03-04T09:36:04.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:04.585+0000 7f5891400640 1 -- 172.21.15.6:0/2089433379 shutdown_connections 2024-03-04T09:36:04.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:04.585+0000 7f5891400640 1 --2- 172.21.15.6:0/2089433379 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f588c145300 0x7f588c1f6e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:04.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:04.585+0000 7f5891400640 1 --2- 172.21.15.6:0/2089433379 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f588c0cce90 0x7f588c143a70 unknown :-1 s=CLOSED pgs=1770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:04.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:04.585+0000 7f5891400640 1 --2- 172.21.15.6:0/2089433379 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f586806b500 0x7f586806d9c0 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:04.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:04.585+0000 7f5891400640 1 --2- 172.21.15.6:0/2089433379 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f588c070f10 0x7f588c1f38f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:04.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:04.585+0000 7f5891400640 1 -- 172.21.15.6:0/2089433379 >> 172.21.15.6:0/2089433379 conn(0x7f588c06b3d0 msgr2=0x7f588c0af750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:36:04.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:04.585+0000 7f5891400640 1 -- 172.21.15.6:0/2089433379 shutdown_connections 2024-03-04T09:36:04.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:04.585+0000 7f5891400640 1 -- 172.21.15.6:0/2089433379 wait complete. 2024-03-04T09:36:04.804 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:32:57.159681+0000', 'modified': '2024-03-04T09:35:26.088222+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8214, 'mds_1': 6796}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6796': {'gid': 6796, 'name': 'd', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.146:6837/2458070840', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 2458070840}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 2458070840}]}, 'join_fscid': -1, 'laggy_since': '2024-03-04T09:33:54.652601+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8214': {'gid': 8214, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2446627517', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2446627517}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2446627517}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8214, 'qdb_cluster': [8214]} 2024-03-04T09:36:04.804 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6796:up:stopping 2024-03-04T09:36:05.805 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:36:05.926 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-04T09:36:06.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:06.032+0000 7f0b35e00640 1 Processor -- start 2024-03-04T09:36:06.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:06.033+0000 7f0b35e00640 1 -- start start 2024-03-04T09:36:06.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:06.034+0000 7f0b35e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0b30072da0 0x7f0b30162c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:06.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:06.034+0000 7f0b35e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f0b300712c0 0x7f0b3006e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:06.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:06.034+0000 7f0b35e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b300ce090 con 0x7f0b30070ee0 2024-03-04T09:36:06.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:06.034+0000 7f0b35e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b300ceca0 con 0x7f0b300712c0 2024-03-04T09:36:06.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:06.034+0000 7f0b35e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b300ce370 con 0x7f0b30072da0 2024-03-04T09:36:06.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:06.035+0000 7f0b2ea00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f0b30070ee0 0x7f0b3006e0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:45420/0 (socket says 172.21.15.6:45420) 2024-03-04T09:36:06.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:06.035+0000 7f0b2ea00640 1 -- 172.21.15.6:0/3997238982 learned_addr learned my addr 172.21.15.6:0/3997238982 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:36:06.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:06.035+0000 7f0b2f400640 1 --2- 172.21.15.6:0/3997238982 >> v2:172.21.15.6:3300/0 conn(0x7f0b300712c0 0x7f0b3006e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:06.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:06.035+0000 7f0b27e00640 1 --2- 172.21.15.6:0/3997238982 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0b30072da0 0x7f0b30162c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:06.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:06.035+0000 7f0b2e000640 1 -- 172.21.15.6:0/3997238982 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 9994845 0 0) 0x7f0b300ce090 con 0x7f0b30070ee0 2024-03-04T09:36:06.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:06.035+0000 7f0b2e000640 1 -- 172.21.15.6:0/3997238982 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b14003520 con 0x7f0b30070ee0 2024-03-04T09:36:06.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:06.036+0000 7f0b2e000640 1 -- 172.21.15.6:0/3997238982 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1429594736 0 0) 0x7f0b14003520 con 0x7f0b30070ee0 2024-03-04T09:36:06.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:06.036+0000 7f0b2e000640 1 -- 172.21.15.6:0/3997238982 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0b30072da0 msgr2=0x7f0b30162c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:36:06.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:06.036+0000 7f0b2e000640 1 --2- 172.21.15.6:0/3997238982 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0b30072da0 0x7f0b30162c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:06.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:06.036+0000 7f0b2e000640 1 -- 172.21.15.6:0/3997238982 >> v2:172.21.15.6:3300/0 conn(0x7f0b300712c0 msgr2=0x7f0b3006e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:36:06.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:06.036+0000 7f0b2e000640 1 --2- 172.21.15.6:0/3997238982 >> v2:172.21.15.6:3300/0 conn(0x7f0b300712c0 0x7f0b3006e7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:06.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:06.036+0000 7f0b2e000640 1 -- 172.21.15.6:0/3997238982 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b300ce650 con 0x7f0b30070ee0 2024-03-04T09:36:06.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:06.036+0000 7f0b2e000640 1 -- 172.21.15.6:0/3997238982 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7f0b20001670 con 0x7f0b30070ee0 2024-03-04T09:36:06.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:06.036+0000 7f0b2e000640 1 -- 172.21.15.6:0/3997238982 >> v1:172.21.15.6:6789/0 conn(0x7f0b30070ee0 legacy=0x7f0b3006e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:06.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:06.036+0000 7f0b2e000640 1 --2- 172.21.15.6:0/3997238982 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0b14003a20 0x7f0b14003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:06.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:06.036+0000 7f0b2e000640 1 --2- 172.21.15.6:0/3997238982 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0b14004800 0x7f0b14004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:06.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:06.036+0000 7f0b2e000640 1 --2- 172.21.15.6:0/3997238982 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0b30072da0 0x7f0b140051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:06.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:06.036+0000 7f0b2e000640 1 -- 172.21.15.6:0/3997238982 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b300ceca0 con 0x7f0b14004800 2024-03-04T09:36:06.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:06.036+0000 7f0b2e000640 1 -- 172.21.15.6:0/3997238982 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b300ce370 con 0x7f0b14003a20 2024-03-04T09:36:06.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:06.036+0000 7f0b2e000640 1 -- 172.21.15.6:0/3997238982 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b1400b9c0 con 0x7f0b30072da0 2024-03-04T09:36:06.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:06.036+0000 7f0b2f400640 1 --2- 172.21.15.6:0/3997238982 >> v2:172.21.15.6:3300/0 conn(0x7f0b300712c0 0x7f0b3006e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T09:36:06.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:06.036+0000 7f0b27e00640 1 --2- 172.21.15.6:0/3997238982 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0b14004800 0x7f0b14004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:06.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:06.037+0000 7f0b2ea00640 1 --2- 172.21.15.6:0/3997238982 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0b14003a20 0x7f0b14003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:06.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:06.037+0000 7f0b2f400640 1 --2- 172.21.15.6:0/3997238982 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0b30072da0 0x7f0b140051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:06.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:06.037+0000 7f0b27e00640 1 -- 172.21.15.6:0/3997238982 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0b30072da0 msgr2=0x7f0b140051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:36:06.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:06.037+0000 7f0b27e00640 1 --2- 172.21.15.6:0/3997238982 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0b30072da0 0x7f0b140051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:06.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:06.037+0000 7f0b27e00640 1 -- 172.21.15.6:0/3997238982 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0b14003a20 msgr2=0x7f0b14003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:36:06.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:06.037+0000 7f0b27e00640 1 --2- 172.21.15.6:0/3997238982 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0b14003a20 0x7f0b14003e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:06.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:06.037+0000 7f0b27e00640 1 -- 172.21.15.6:0/3997238982 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b14003520 con 0x7f0b14004800 2024-03-04T09:36:06.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:06.038+0000 7f0b27e00640 1 --2- 172.21.15.6:0/3997238982 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0b14004800 0x7f0b14004c80 secure :-1 s=READY pgs=1772 cs=0 l=1 rev1=1 crypto rx=0x7f0b18009af0 tx=0x7f0b14005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23a65873ed5953c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:36:06.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:06.039+0000 7f0b2e000640 1 -- 172.21.15.6:0/3997238982 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b18039070 con 0x7f0b14004800 2024-03-04T09:36:06.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:06.039+0000 7f0b2e000640 1 -- 172.21.15.6:0/3997238982 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b1802c6b0 con 0x7f0b14004800 2024-03-04T09:36:06.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:06.039+0000 7f0b2e000640 1 -- 172.21.15.6:0/3997238982 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b18009690 con 0x7f0b14004800 2024-03-04T09:36:06.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:06.039+0000 7f0b35e00640 1 -- 172.21.15.6:0/3997238982 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0b14004800 msgr2=0x7f0b14004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:06.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:06.039+0000 7f0b35e00640 1 --2- 172.21.15.6:0/3997238982 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0b14004800 0x7f0b14004c80 secure :-1 s=READY pgs=1772 cs=0 l=1 rev1=1 crypto rx=0x7f0b18009af0 tx=0x7f0b14005830 comp rx=0 tx=0).stop 2024-03-04T09:36:06.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:06.039+0000 7f0b2f400640 1 -- 172.21.15.6:0/3997238982 reap_dead start 2024-03-04T09:36:06.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:06.039+0000 7f0b35e00640 1 -- 172.21.15.6:0/3997238982 shutdown_connections 2024-03-04T09:36:06.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:06.039+0000 7f0b35e00640 1 -- 172.21.15.6:0/3997238982 >> 172.21.15.6:0/3997238982 conn(0x7f0b3006afb0 msgr2=0x7f0b3006b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:36:06.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:06.039+0000 7f0b35e00640 1 -- 172.21.15.6:0/3997238982 shutdown_connections 2024-03-04T09:36:06.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:06.039+0000 7f0b35e00640 1 -- 172.21.15.6:0/3997238982 wait complete. 2024-03-04T09:36:06.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:06.041+0000 7f0b35e00640 1 Processor -- start 2024-03-04T09:36:06.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:06.041+0000 7f0b35e00640 1 -- start start 2024-03-04T09:36:06.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:06.042+0000 7f0b35e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0b14004800 0x7f0b3014ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:06.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:06.042+0000 7f0b35e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0b300afac0 0x7f0b300aff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:06.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:06.042+0000 7f0b35e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0b3009c4e0 0x7f0b3009c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:06.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:06.042+0000 7f0b35e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b300cdda0 con 0x7f0b300afac0 2024-03-04T09:36:06.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:06.042+0000 7f0b35e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b300cdc20 con 0x7f0b14004800 2024-03-04T09:36:06.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:06.042+0000 7f0b35e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b30162c40 con 0x7f0b3009c4e0 2024-03-04T09:36:06.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:06.042+0000 7f0b27e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0b300afac0 0x7f0b300aff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:06.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:06.043+0000 7f0b27e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0b300afac0 0x7f0b300aff40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:46804/0 (socket says 172.21.15.6:46804) 2024-03-04T09:36:06.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:06.043+0000 7f0b27e00640 1 -- 172.21.15.6:0/693991053 learned_addr learned my addr 172.21.15.6:0/693991053 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:36:06.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:06.043+0000 7f0b2ea00640 1 --2- 172.21.15.6:0/693991053 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0b14004800 0x7f0b3014ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:06.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:06.043+0000 7f0b2f400640 1 --2- 172.21.15.6:0/693991053 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0b3009c4e0 0x7f0b3009c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:06.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:06.043+0000 7f0b27e00640 1 -- 172.21.15.6:0/693991053 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0b3009c4e0 msgr2=0x7f0b3009c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:06.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:06.043+0000 7f0b27e00640 1 --2- 172.21.15.6:0/693991053 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0b3009c4e0 0x7f0b3009c960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:06.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:06.043+0000 7f0b27e00640 1 -- 172.21.15.6:0/693991053 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0b14004800 msgr2=0x7f0b3014ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:06.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:06.043+0000 7f0b27e00640 1 --2- 172.21.15.6:0/693991053 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0b14004800 0x7f0b3014ef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:06.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:06.043+0000 7f0b27e00640 1 -- 172.21.15.6:0/693991053 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b300ad130 con 0x7f0b300afac0 2024-03-04T09:36:06.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:06.044+0000 7f0b27e00640 1 --2- 172.21.15.6:0/693991053 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0b300afac0 0x7f0b300aff40 secure :-1 s=READY pgs=1773 cs=0 l=1 rev1=1 crypto rx=0x7f0b18009af0 tx=0x7f0b18004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:36:06.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:06.044+0000 7f0b27400640 1 -- 172.21.15.6:0/693991053 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b18042020 con 0x7f0b300afac0 2024-03-04T09:36:06.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:06.044+0000 7f0b27400640 1 -- 172.21.15.6:0/693991053 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b1803a040 con 0x7f0b300afac0 2024-03-04T09:36:06.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:06.044+0000 7f0b27400640 1 -- 172.21.15.6:0/693991053 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b18039070 con 0x7f0b300afac0 2024-03-04T09:36:06.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:06.044+0000 7f0b35e00640 1 -- 172.21.15.6:0/693991053 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b3014f440 con 0x7f0b300afac0 2024-03-04T09:36:06.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:06.044+0000 7f0b35e00640 1 -- 172.21.15.6:0/693991053 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b3009d250 con 0x7f0b300afac0 2024-03-04T09:36:06.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:06.045+0000 7f0b35e00640 1 -- 172.21.15.6:0/693991053 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b3004f3a0 con 0x7f0b300afac0 2024-03-04T09:36:06.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:06.047+0000 7f0b27400640 1 -- 172.21.15.6:0/693991053 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0b18045020 con 0x7f0b300afac0 2024-03-04T09:36:06.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:06.048+0000 7f0b27400640 1 --2- 172.21.15.6:0/693991053 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f0b0c06b4d0 0x7f0b0c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:06.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:06.048+0000 7f0b27400640 1 -- 172.21.15.6:0/693991053 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f0b180ab600 con 0x7f0b300afac0 2024-03-04T09:36:06.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:06.049+0000 7f0b2ea00640 1 --2- 172.21.15.6:0/693991053 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f0b0c06b4d0 0x7f0b0c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:06.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:06.050+0000 7f0b2ea00640 1 --2- 172.21.15.6:0/693991053 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f0b0c06b4d0 0x7f0b0c06d990 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f0b20007fd0 tx=0x7f0b20009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:36:06.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:06.058+0000 7f0b27400640 1 -- 172.21.15.6:0/693991053 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0b1804c070 con 0x7f0b300afac0 2024-03-04T09:36:06.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:06.301+0000 7f0b35e00640 1 -- 172.21.15.6:0/693991053 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0b300cc600 con 0x7f0b300afac0 2024-03-04T09:36:06.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:06.303+0000 7f0b27400640 1 -- 172.21.15.6:0/693991053 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3464 (secure 0 0 0) 0x7f0b3014f440 con 0x7f0b300afac0 2024-03-04T09:36:06.300 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:36:06.300 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:35:26.088222+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"laggy_since":"2024-03-04T09:33:54.652601+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":9}]} 2024-03-04T09:36:06.300 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 101 2024-03-04T09:36:06.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:06.306+0000 7f0b35e00640 1 -- 172.21.15.6:0/693991053 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f0b0c06b4d0 msgr2=0x7f0b0c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:06.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:06.306+0000 7f0b35e00640 1 --2- 172.21.15.6:0/693991053 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f0b0c06b4d0 0x7f0b0c06d990 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f0b20007fd0 tx=0x7f0b20009040 comp rx=0 tx=0).stop 2024-03-04T09:36:06.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:06.306+0000 7f0b35e00640 1 -- 172.21.15.6:0/693991053 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0b300afac0 msgr2=0x7f0b300aff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:06.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:06.306+0000 7f0b35e00640 1 --2- 172.21.15.6:0/693991053 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0b300afac0 0x7f0b300aff40 secure :-1 s=READY pgs=1773 cs=0 l=1 rev1=1 crypto rx=0x7f0b18009af0 tx=0x7f0b18004730 comp rx=0 tx=0).stop 2024-03-04T09:36:06.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:06.306+0000 7f0b35e00640 1 -- 172.21.15.6:0/693991053 shutdown_connections 2024-03-04T09:36:06.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:06.306+0000 7f0b35e00640 1 --2- 172.21.15.6:0/693991053 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f0b0c06b4d0 0x7f0b0c06d990 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:06.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:06.306+0000 7f0b35e00640 1 --2- 172.21.15.6:0/693991053 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0b3009c4e0 0x7f0b3009c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:06.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:06.306+0000 7f0b35e00640 1 --2- 172.21.15.6:0/693991053 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0b300afac0 0x7f0b300aff40 unknown :-1 s=CLOSED pgs=1773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:06.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:06.306+0000 7f0b35e00640 1 --2- 172.21.15.6:0/693991053 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0b14004800 0x7f0b3014ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:06.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:06.306+0000 7f0b35e00640 1 -- 172.21.15.6:0/693991053 >> 172.21.15.6:0/693991053 conn(0x7f0b3006afb0 msgr2=0x7f0b30166030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:36:06.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:06.306+0000 7f0b35e00640 1 -- 172.21.15.6:0/693991053 shutdown_connections 2024-03-04T09:36:06.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:06.306+0000 7f0b35e00640 1 -- 172.21.15.6:0/693991053 wait complete. 2024-03-04T09:36:06.523 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:32:57.159681+0000', 'modified': '2024-03-04T09:35:26.088222+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8214, 'mds_1': 6796}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6796': {'gid': 6796, 'name': 'd', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.146:6837/2458070840', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 2458070840}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 2458070840}]}, 'join_fscid': -1, 'laggy_since': '2024-03-04T09:33:54.652601+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8214': {'gid': 8214, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2446627517', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2446627517}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2446627517}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8214, 'qdb_cluster': [8214]} 2024-03-04T09:36:06.523 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6796:up:stopping 2024-03-04T09:36:07.524 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:36:07.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:07.760+0000 7fa732800640 1 Processor -- start 2024-03-04T09:36:07.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:07.761+0000 7fa732800640 1 -- start start 2024-03-04T09:36:07.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:07.764+0000 7fa732800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa72c0cd770 0x7fa72c0cdb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:07.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:07.764+0000 7fa732800640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa72c0ce090 con 0x7fa72c15f810 2024-03-04T09:36:07.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:07.764+0000 7fa732800640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa72c15c340 con 0x7fa72c15bb20 2024-03-04T09:36:07.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:07.764+0000 7fa732800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa72c15eae0 con 0x7fa72c0cd770 2024-03-04T09:36:07.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:07.764+0000 7fa723e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fa72c15f810 0x7fa72c15b410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:45428/0 (socket says 172.21.15.6:45428) 2024-03-04T09:36:07.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:07.764+0000 7fa723e00640 1 -- 172.21.15.6:0/1023619007 learned_addr learned my addr 172.21.15.6:0/1023619007 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:36:07.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:07.764+0000 7fa72b400640 1 --2- 172.21.15.6:0/1023619007 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa72c0cd770 0x7fa72c0cdb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:07.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:07.764+0000 7fa72aa00640 1 -- 172.21.15.6:0/1023619007 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 522352303 0 0) 0x7fa72c0ce090 con 0x7fa72c15f810 2024-03-04T09:36:07.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:07.765+0000 7fa72aa00640 1 -- 172.21.15.6:0/1023619007 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa710003520 con 0x7fa72c15f810 2024-03-04T09:36:07.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:07.765+0000 7fa72aa00640 1 -- 172.21.15.6:0/1023619007 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1448838144 0 0) 0x7fa710003520 con 0x7fa72c15f810 2024-03-04T09:36:07.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:07.765+0000 7fa72aa00640 1 -- 172.21.15.6:0/1023619007 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa72c0cd770 msgr2=0x7fa72c0cdb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:36:07.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:07.765+0000 7fa72aa00640 1 --2- 172.21.15.6:0/1023619007 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa72c0cd770 0x7fa72c0cdb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:07.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:07.765+0000 7fa72aa00640 1 -- 172.21.15.6:0/1023619007 >> v1:172.21.15.146:6789/0 conn(0x7fa72c15bb20 legacy=0x7fa72c0cf070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:07.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:07.765+0000 7fa72aa00640 1 -- 172.21.15.6:0/1023619007 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa72c15ed30 con 0x7fa72c15f810 2024-03-04T09:36:07.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:07.765+0000 7fa72aa00640 1 -- 172.21.15.6:0/1023619007 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7fa714001670 con 0x7fa72c15f810 2024-03-04T09:36:07.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:07.765+0000 7fa72aa00640 1 -- 172.21.15.6:0/1023619007 >> v1:172.21.15.6:6789/0 conn(0x7fa72c15f810 legacy=0x7fa72c15b410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:07.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:07.765+0000 7fa72aa00640 1 --2- 172.21.15.6:0/1023619007 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa710003a50 0x7fa710003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:07.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:07.765+0000 7fa72aa00640 1 --2- 172.21.15.6:0/1023619007 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa710004800 0x7fa710004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:07.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:07.765+0000 7fa72aa00640 1 --2- 172.21.15.6:0/1023619007 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa72c0cd770 0x7fa710005220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:07.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:07.765+0000 7fa72aa00640 1 -- 172.21.15.6:0/1023619007 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa72c15eae0 con 0x7fa710004800 2024-03-04T09:36:07.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:07.765+0000 7fa72aa00640 1 -- 172.21.15.6:0/1023619007 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa71000bb80 con 0x7fa710003a50 2024-03-04T09:36:07.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:07.765+0000 7fa72aa00640 1 -- 172.21.15.6:0/1023619007 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa71000bd80 con 0x7fa72c0cd770 2024-03-04T09:36:07.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:07.765+0000 7fa723e00640 1 --2- 172.21.15.6:0/1023619007 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa710004800 0x7fa710004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:07.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:07.766+0000 7fa72b400640 1 --2- 172.21.15.6:0/1023619007 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa710003a50 0x7fa710003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:07.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:07.766+0000 7fa72be00640 1 --2- 172.21.15.6:0/1023619007 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa72c0cd770 0x7fa710005220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:07.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:07.766+0000 7fa723e00640 1 -- 172.21.15.6:0/1023619007 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa72c0cd770 msgr2=0x7fa710005220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:36:07.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:07.766+0000 7fa723e00640 1 --2- 172.21.15.6:0/1023619007 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa72c0cd770 0x7fa710005220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:07.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:07.766+0000 7fa723e00640 1 -- 172.21.15.6:0/1023619007 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa710003a50 msgr2=0x7fa710003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:36:07.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:07.766+0000 7fa723e00640 1 --2- 172.21.15.6:0/1023619007 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa710003a50 0x7fa710003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:07.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:07.766+0000 7fa723e00640 1 -- 172.21.15.6:0/1023619007 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa710003520 con 0x7fa710004800 2024-03-04T09:36:07.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:07.768+0000 7fa723e00640 1 --2- 172.21.15.6:0/1023619007 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa710004800 0x7fa710004c80 secure :-1 s=READY pgs=1775 cs=0 l=1 rev1=1 crypto rx=0x7fa71400c0e0 tx=0x7fa710005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42c4f6a997295495 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:36:07.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:07.768+0000 7fa72aa00640 1 -- 172.21.15.6:0/1023619007 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa714007c20 con 0x7fa710004800 2024-03-04T09:36:07.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:07.768+0000 7fa72aa00640 1 -- 172.21.15.6:0/1023619007 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa714030070 con 0x7fa710004800 2024-03-04T09:36:07.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:07.768+0000 7fa72aa00640 1 -- 172.21.15.6:0/1023619007 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa714005d30 con 0x7fa710004800 2024-03-04T09:36:07.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:07.769+0000 7fa732800640 1 -- 172.21.15.6:0/1023619007 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa710004800 msgr2=0x7fa710004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:07.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:07.769+0000 7fa732800640 1 --2- 172.21.15.6:0/1023619007 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa710004800 0x7fa710004c80 secure :-1 s=READY pgs=1775 cs=0 l=1 rev1=1 crypto rx=0x7fa71400c0e0 tx=0x7fa710005830 comp rx=0 tx=0).stop 2024-03-04T09:36:07.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:07.769+0000 7fa72be00640 1 -- 172.21.15.6:0/1023619007 reap_dead start 2024-03-04T09:36:07.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:07.769+0000 7fa732800640 1 -- 172.21.15.6:0/1023619007 shutdown_connections 2024-03-04T09:36:07.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:07.769+0000 7fa732800640 1 -- 172.21.15.6:0/1023619007 >> 172.21.15.6:0/1023619007 conn(0x7fa72c0d0d80 msgr2=0x7fa72c0d1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:36:07.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:07.769+0000 7fa732800640 1 -- 172.21.15.6:0/1023619007 shutdown_connections 2024-03-04T09:36:07.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:07.769+0000 7fa732800640 1 -- 172.21.15.6:0/1023619007 wait complete. 2024-03-04T09:36:07.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:07.772+0000 7fa732800640 1 Processor -- start 2024-03-04T09:36:07.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:07.772+0000 7fa732800640 1 -- start start 2024-03-04T09:36:07.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:07.774+0000 7fa732800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa710004800 0x7fa72c150790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:07.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:07.774+0000 7fa732800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa72c153110 0x7fa72c149b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:07.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:07.774+0000 7fa732800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa72c15db90 0x7fa72c15e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:07.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:07.774+0000 7fa732800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa72c070630 con 0x7fa72c15db90 2024-03-04T09:36:07.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:07.774+0000 7fa732800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa72c15bfb0 con 0x7fa710004800 2024-03-04T09:36:07.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:07.774+0000 7fa732800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa72c0707b0 con 0x7fa72c153110 2024-03-04T09:36:07.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:07.774+0000 7fa72be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa72c15db90 0x7fa72c15e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:07.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:07.774+0000 7fa72be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa72c15db90 0x7fa72c15e020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:46824/0 (socket says 172.21.15.6:46824) 2024-03-04T09:36:07.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:07.774+0000 7fa72be00640 1 -- 172.21.15.6:0/2314953723 learned_addr learned my addr 172.21.15.6:0/2314953723 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:36:07.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:07.775+0000 7fa72b400640 1 --2- 172.21.15.6:0/2314953723 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa710004800 0x7fa72c150790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:07.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:07.775+0000 7fa723e00640 1 --2- 172.21.15.6:0/2314953723 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa72c153110 0x7fa72c149b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:07.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:07.775+0000 7fa72be00640 1 -- 172.21.15.6:0/2314953723 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa72c153110 msgr2=0x7fa72c149b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:07.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:07.775+0000 7fa72be00640 1 --2- 172.21.15.6:0/2314953723 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa72c153110 0x7fa72c149b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:07.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:07.775+0000 7fa72be00640 1 -- 172.21.15.6:0/2314953723 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa710004800 msgr2=0x7fa72c150790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:07.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:07.775+0000 7fa72be00640 1 --2- 172.21.15.6:0/2314953723 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa710004800 0x7fa72c150790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:07.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:07.775+0000 7fa72be00640 1 -- 172.21.15.6:0/2314953723 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa72c150fe0 con 0x7fa72c15db90 2024-03-04T09:36:07.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:07.775+0000 7fa723e00640 1 --2- 172.21.15.6:0/2314953723 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa72c153110 0x7fa72c149b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T09:36:07.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:07.775+0000 7fa72be00640 1 --2- 172.21.15.6:0/2314953723 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa72c15db90 0x7fa72c15e020 secure :-1 s=READY pgs=1776 cs=0 l=1 rev1=1 crypto rx=0x7fa72c071d50 tx=0x7fa72400ec10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:36:07.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:07.776+0000 7fa723400640 1 -- 172.21.15.6:0/2314953723 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa724003970 con 0x7fa72c15db90 2024-03-04T09:36:07.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:07.776+0000 7fa723400640 1 -- 172.21.15.6:0/2314953723 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa724018070 con 0x7fa72c15db90 2024-03-04T09:36:07.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:07.776+0000 7fa723400640 1 -- 172.21.15.6:0/2314953723 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa724003d50 con 0x7fa72c15db90 2024-03-04T09:36:07.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:07.776+0000 7fa732800640 1 -- 172.21.15.6:0/2314953723 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa72c1512c0 con 0x7fa72c15db90 2024-03-04T09:36:07.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:07.776+0000 7fa732800640 1 -- 172.21.15.6:0/2314953723 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa72c151880 con 0x7fa72c15db90 2024-03-04T09:36:07.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:07.777+0000 7fa732800640 1 -- 172.21.15.6:0/2314953723 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa72c0b53e0 con 0x7fa72c15db90 2024-03-04T09:36:07.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:07.779+0000 7fa723400640 1 -- 172.21.15.6:0/2314953723 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa7240050d0 con 0x7fa72c15db90 2024-03-04T09:36:07.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:07.780+0000 7fa723400640 1 --2- 172.21.15.6:0/2314953723 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa70806b4d0 0x7fa70806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:07.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:07.780+0000 7fa723400640 1 -- 172.21.15.6:0/2314953723 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fa724089f50 con 0x7fa72c15db90 2024-03-04T09:36:07.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:07.781+0000 7fa72b400640 1 --2- 172.21.15.6:0/2314953723 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa70806b4d0 0x7fa70806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:07.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:07.782+0000 7fa72b400640 1 --2- 172.21.15.6:0/2314953723 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa70806b4d0 0x7fa70806d990 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fa71c009ac0 tx=0x7fa71c008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:36:07.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:07.785+0000 7fa723400640 1 -- 172.21.15.6:0/2314953723 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa72c0b53e0 con 0x7fa72c15db90 2024-03-04T09:36:07.928 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-04T09:36:08.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:08.111+0000 7fa732800640 1 -- 172.21.15.6:0/2314953723 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa72c04f3a0 con 0x7fa72c15db90 2024-03-04T09:36:08.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:08.112+0000 7fa723400640 1 -- 172.21.15.6:0/2314953723 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3464 (secure 0 0 0) 0x7fa724058d00 con 0x7fa72c15db90 2024-03-04T09:36:08.109 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:36:08.109 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:35:26.088222+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"laggy_since":"2024-03-04T09:33:54.652601+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":9}]} 2024-03-04T09:36:08.109 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 101 2024-03-04T09:36:08.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:08.114+0000 7fa732800640 1 -- 172.21.15.6:0/2314953723 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa70806b4d0 msgr2=0x7fa70806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:08.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:08.114+0000 7fa732800640 1 --2- 172.21.15.6:0/2314953723 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa70806b4d0 0x7fa70806d990 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fa71c009ac0 tx=0x7fa71c008040 comp rx=0 tx=0).stop 2024-03-04T09:36:08.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:08.114+0000 7fa732800640 1 -- 172.21.15.6:0/2314953723 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa72c15db90 msgr2=0x7fa72c15e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:08.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:08.114+0000 7fa732800640 1 --2- 172.21.15.6:0/2314953723 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa72c15db90 0x7fa72c15e020 secure :-1 s=READY pgs=1776 cs=0 l=1 rev1=1 crypto rx=0x7fa72c071d50 tx=0x7fa72400ec10 comp rx=0 tx=0).stop 2024-03-04T09:36:08.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:08.115+0000 7fa732800640 1 -- 172.21.15.6:0/2314953723 shutdown_connections 2024-03-04T09:36:08.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:08.115+0000 7fa732800640 1 --2- 172.21.15.6:0/2314953723 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa72c15db90 0x7fa72c15e020 unknown :-1 s=CLOSED pgs=1776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:08.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:08.115+0000 7fa732800640 1 --2- 172.21.15.6:0/2314953723 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa70806b4d0 0x7fa70806d990 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:08.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:08.115+0000 7fa732800640 1 --2- 172.21.15.6:0/2314953723 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa72c153110 0x7fa72c149b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:08.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:08.115+0000 7fa732800640 1 --2- 172.21.15.6:0/2314953723 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa710004800 0x7fa72c150790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:08.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:08.115+0000 7fa732800640 1 -- 172.21.15.6:0/2314953723 >> 172.21.15.6:0/2314953723 conn(0x7fa72c0d0d80 msgr2=0x7fa72c15c620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:36:08.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:08.115+0000 7fa732800640 1 -- 172.21.15.6:0/2314953723 shutdown_connections 2024-03-04T09:36:08.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:08.115+0000 7fa732800640 1 -- 172.21.15.6:0/2314953723 wait complete. 2024-03-04T09:36:08.339 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:32:57.159681+0000', 'modified': '2024-03-04T09:35:26.088222+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8214, 'mds_1': 6796}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6796': {'gid': 6796, 'name': 'd', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.146:6837/2458070840', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 2458070840}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 2458070840}]}, 'join_fscid': -1, 'laggy_since': '2024-03-04T09:33:54.652601+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8214': {'gid': 8214, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2446627517', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2446627517}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2446627517}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8214, 'qdb_cluster': [8214]} 2024-03-04T09:36:08.339 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6796:up:stopping 2024-03-04T09:36:09.340 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:36:09.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:09.574+0000 7effc4a00640 1 Processor -- start 2024-03-04T09:36:09.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:09.575+0000 7effc4a00640 1 -- start start 2024-03-04T09:36:09.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:09.576+0000 7effc4a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7effc015e2a0 0x7effc00cef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:09.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:09.576+0000 7effc4a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effc00d1120 con 0x7effc015dec0 2024-03-04T09:36:09.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:09.576+0000 7effc4a00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effc00d1400 con 0x7effc00cf460 2024-03-04T09:36:09.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:09.576+0000 7effc4a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effc00b0ea0 con 0x7effc015e2a0 2024-03-04T09:36:09.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:09.576+0000 7effbf400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7effc015dec0 0x7effc00ce810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:53454/0 (socket says 172.21.15.6:53454) 2024-03-04T09:36:09.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:09.576+0000 7effbf400640 1 -- 172.21.15.6:0/643024171 learned_addr learned my addr 172.21.15.6:0/643024171 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:36:09.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:09.577+0000 7effb7e00640 1 --2- 172.21.15.6:0/643024171 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7effc015e2a0 0x7effc00cef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:09.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:09.577+0000 7effbea00640 1 -- 172.21.15.6:0/643024171 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1098203623 0 0) 0x7effc00d1120 con 0x7effc015dec0 2024-03-04T09:36:09.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:09.577+0000 7effbea00640 1 -- 172.21.15.6:0/643024171 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7effa4003520 con 0x7effc015dec0 2024-03-04T09:36:09.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:09.577+0000 7effbea00640 1 -- 172.21.15.6:0/643024171 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1946313988 0 0) 0x7effc00d1400 con 0x7effc00cf460 2024-03-04T09:36:09.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:09.577+0000 7effbea00640 1 -- 172.21.15.6:0/643024171 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7effc00d1120 con 0x7effc00cf460 2024-03-04T09:36:09.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:09.577+0000 7effbea00640 1 -- 172.21.15.6:0/643024171 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3036461909 0 0) 0x7effa4003520 con 0x7effc015dec0 2024-03-04T09:36:09.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:09.577+0000 7effbea00640 1 -- 172.21.15.6:0/643024171 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7effc015e2a0 msgr2=0x7effc00cef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:36:09.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:09.577+0000 7effbea00640 1 --2- 172.21.15.6:0/643024171 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7effc015e2a0 0x7effc00cef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:09.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:09.577+0000 7effbea00640 1 -- 172.21.15.6:0/643024171 >> v1:172.21.15.146:6789/0 conn(0x7effc00cf460 legacy=0x7effc00b1330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:09.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:09.578+0000 7effbea00640 1 -- 172.21.15.6:0/643024171 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effc00b0b30 con 0x7effc015dec0 2024-03-04T09:36:09.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:09.578+0000 7effbea00640 1 -- 172.21.15.6:0/643024171 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7effb0001670 con 0x7effc015dec0 2024-03-04T09:36:09.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:09.578+0000 7effbea00640 1 -- 172.21.15.6:0/643024171 >> v1:172.21.15.6:6789/0 conn(0x7effc015dec0 legacy=0x7effc00ce810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:09.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:09.578+0000 7effbea00640 1 --2- 172.21.15.6:0/643024171 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7effa4003b70 0x7effa4003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:09.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:09.578+0000 7effbea00640 1 --2- 172.21.15.6:0/643024171 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7effc015e2a0 0x7effa4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:09.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:09.578+0000 7effbea00640 1 --2- 172.21.15.6:0/643024171 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7effa4004d40 0x7effa40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:09.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:09.578+0000 7effbea00640 1 -- 172.21.15.6:0/643024171 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effc00b0ea0 con 0x7effa4003b70 2024-03-04T09:36:09.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:09.578+0000 7effbea00640 1 -- 172.21.15.6:0/643024171 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effa400ba70 con 0x7effa4004d40 2024-03-04T09:36:09.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:09.578+0000 7effbea00640 1 -- 172.21.15.6:0/643024171 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effa400bc70 con 0x7effc015e2a0 2024-03-04T09:36:09.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:09.578+0000 7effbf400640 1 --2- 172.21.15.6:0/643024171 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7effa4003b70 0x7effa4003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:09.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:09.578+0000 7effb7e00640 1 --2- 172.21.15.6:0/643024171 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7effc015e2a0 0x7effa4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:09.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:09.578+0000 7effbfe00640 1 --2- 172.21.15.6:0/643024171 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7effa4004d40 0x7effa40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:09.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:09.578+0000 7effbf400640 1 -- 172.21.15.6:0/643024171 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7effc015e2a0 msgr2=0x7effa4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:36:09.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:09.578+0000 7effbf400640 1 --2- 172.21.15.6:0/643024171 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7effc015e2a0 0x7effa4004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:09.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:09.578+0000 7effbf400640 1 -- 172.21.15.6:0/643024171 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7effa4004d40 msgr2=0x7effa40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:36:09.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:09.579+0000 7effbf400640 1 --2- 172.21.15.6:0/643024171 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7effa4004d40 0x7effa40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:09.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:09.579+0000 7effbf400640 1 -- 172.21.15.6:0/643024171 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effa4003520 con 0x7effa4003b70 2024-03-04T09:36:09.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:09.579+0000 7effbfe00640 1 --2- 172.21.15.6:0/643024171 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7effa4004d40 0x7effa40051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:36:09.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:09.580+0000 7effbf400640 1 --2- 172.21.15.6:0/643024171 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7effa4003b70 0x7effa4003f70 secure :-1 s=READY pgs=1778 cs=0 l=1 rev1=1 crypto rx=0x7effb000c0e0 tx=0x7effa4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7599fbc04a580805 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:36:09.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:09.580+0000 7effbea00640 1 -- 172.21.15.6:0/643024171 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effb0007c40 con 0x7effa4003b70 2024-03-04T09:36:09.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:09.580+0000 7effbea00640 1 -- 172.21.15.6:0/643024171 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effb0030070 con 0x7effa4003b70 2024-03-04T09:36:09.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:09.580+0000 7effbea00640 1 -- 172.21.15.6:0/643024171 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effb0005d50 con 0x7effa4003b70 2024-03-04T09:36:09.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:09.581+0000 7effc4a00640 1 -- 172.21.15.6:0/643024171 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7effa4003b70 msgr2=0x7effa4003f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:09.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:09.581+0000 7effc4a00640 1 --2- 172.21.15.6:0/643024171 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7effa4003b70 0x7effa4003f70 secure :-1 s=READY pgs=1778 cs=0 l=1 rev1=1 crypto rx=0x7effb000c0e0 tx=0x7effa4005830 comp rx=0 tx=0).stop 2024-03-04T09:36:09.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:09.581+0000 7effbfe00640 1 -- 172.21.15.6:0/643024171 reap_dead start 2024-03-04T09:36:09.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:09.581+0000 7effc4a00640 1 -- 172.21.15.6:0/643024171 shutdown_connections 2024-03-04T09:36:09.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:09.581+0000 7effc4a00640 1 -- 172.21.15.6:0/643024171 >> 172.21.15.6:0/643024171 conn(0x7effc00d3d40 msgr2=0x7effc00d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:36:09.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:09.581+0000 7effc4a00640 1 -- 172.21.15.6:0/643024171 shutdown_connections 2024-03-04T09:36:09.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:09.581+0000 7effc4a00640 1 -- 172.21.15.6:0/643024171 wait complete. 2024-03-04T09:36:09.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:09.583+0000 7effc4a00640 1 Processor -- start 2024-03-04T09:36:09.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:09.583+0000 7effc4a00640 1 -- start start 2024-03-04T09:36:09.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:09.584+0000 7effc4a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7effa4003b70 0x7effc015b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:09.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:09.584+0000 7effc4a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7effc00b8e60 0x7effc00b72b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:09.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:09.584+0000 7effc4a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7effc00b35d0 0x7effc00b3a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:09.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:09.584+0000 7effc4a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effc00d0ec0 con 0x7effa4003b70 2024-03-04T09:36:09.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:09.584+0000 7effc4a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effc00d0d40 con 0x7effc00b8e60 2024-03-04T09:36:09.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:09.584+0000 7effc4a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effc00cef20 con 0x7effc00b35d0 2024-03-04T09:36:09.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:09.584+0000 7effbf400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7effa4003b70 0x7effc015b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:09.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:09.584+0000 7effbf400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7effa4003b70 0x7effc015b670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:54202/0 (socket says 172.21.15.6:54202) 2024-03-04T09:36:09.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:09.584+0000 7effbf400640 1 -- 172.21.15.6:0/1533687684 learned_addr learned my addr 172.21.15.6:0/1533687684 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:36:09.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:09.585+0000 7effbfe00640 1 --2- 172.21.15.6:0/1533687684 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7effc00b35d0 0x7effc00b3a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:09.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:09.585+0000 7effb7e00640 1 --2- 172.21.15.6:0/1533687684 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7effc00b8e60 0x7effc00b72b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:09.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:09.585+0000 7effbf400640 1 -- 172.21.15.6:0/1533687684 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7effc00b35d0 msgr2=0x7effc00b3a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:09.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:09.585+0000 7effbf400640 1 --2- 172.21.15.6:0/1533687684 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7effc00b35d0 0x7effc00b3a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:09.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:09.585+0000 7effbf400640 1 -- 172.21.15.6:0/1533687684 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7effc00b8e60 msgr2=0x7effc00b72b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:09.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:09.585+0000 7effbf400640 1 --2- 172.21.15.6:0/1533687684 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7effc00b8e60 0x7effc00b72b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:09.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:09.585+0000 7effbf400640 1 -- 172.21.15.6:0/1533687684 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effc015bbb0 con 0x7effa4003b70 2024-03-04T09:36:09.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:09.585+0000 7effbfe00640 1 --2- 172.21.15.6:0/1533687684 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7effc00b35d0 0x7effc00b3a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:36:09.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:09.585+0000 7effb7e00640 1 --2- 172.21.15.6:0/1533687684 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7effc00b8e60 0x7effc00b72b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:36:09.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:09.585+0000 7effbf400640 1 --2- 172.21.15.6:0/1533687684 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7effa4003b70 0x7effc015b670 secure :-1 s=READY pgs=1779 cs=0 l=1 rev1=1 crypto rx=0x7effb000c200 tx=0x7effb0001970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:36:09.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:09.585+0000 7effb7400640 1 -- 172.21.15.6:0/1533687684 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effb0045070 con 0x7effa4003b70 2024-03-04T09:36:09.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:09.585+0000 7effb7400640 1 -- 172.21.15.6:0/1533687684 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effb0004bf0 con 0x7effa4003b70 2024-03-04T09:36:09.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:09.585+0000 7effb7400640 1 -- 172.21.15.6:0/1533687684 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effb0007c40 con 0x7effa4003b70 2024-03-04T09:36:09.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:09.585+0000 7effc4a00640 1 -- 172.21.15.6:0/1533687684 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effc00b77f0 con 0x7effa4003b70 2024-03-04T09:36:09.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:09.586+0000 7effc4a00640 1 -- 172.21.15.6:0/1533687684 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effc00b8570 con 0x7effa4003b70 2024-03-04T09:36:09.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:09.587+0000 7effc4a00640 1 -- 172.21.15.6:0/1533687684 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effc00d3100 con 0x7effa4003b70 2024-03-04T09:36:09.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:09.588+0000 7effb7400640 1 -- 172.21.15.6:0/1533687684 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7effb0007070 con 0x7effa4003b70 2024-03-04T09:36:09.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:09.589+0000 7effb7400640 1 --2- 172.21.15.6:0/1533687684 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7eff9c06b500 0x7eff9c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:09.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:09.590+0000 7effb7400640 1 -- 172.21.15.6:0/1533687684 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7effb00ab280 con 0x7effa4003b70 2024-03-04T09:36:09.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:09.590+0000 7effb7e00640 1 --2- 172.21.15.6:0/1533687684 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7eff9c06b500 0x7eff9c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:09.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:09.591+0000 7effb7e00640 1 --2- 172.21.15.6:0/1533687684 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7eff9c06b500 0x7eff9c06d9c0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7effc00b9260 tx=0x7effa8008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:36:09.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:09.593+0000 7effb7400640 1 -- 172.21.15.6:0/1533687684 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7effc00d3100 con 0x7effa4003b70 2024-03-04T09:36:09.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:09.855+0000 7effc4a00640 1 -- 172.21.15.6:0/1533687684 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7effc004f330 con 0x7effa4003b70 2024-03-04T09:36:09.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:09.856+0000 7effb7400640 1 -- 172.21.15.6:0/1533687684 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3464 (secure 0 0 0) 0x7effb0079b50 con 0x7effa4003b70 2024-03-04T09:36:09.853 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:36:09.853 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:35:26.088222+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"laggy_since":"2024-03-04T09:33:54.652601+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":9}]} 2024-03-04T09:36:09.853 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 101 2024-03-04T09:36:09.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:09.858+0000 7effc4a00640 1 -- 172.21.15.6:0/1533687684 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7eff9c06b500 msgr2=0x7eff9c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:09.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:09.858+0000 7effc4a00640 1 --2- 172.21.15.6:0/1533687684 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7eff9c06b500 0x7eff9c06d9c0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7effc00b9260 tx=0x7effa8008040 comp rx=0 tx=0).stop 2024-03-04T09:36:09.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:09.858+0000 7effc4a00640 1 -- 172.21.15.6:0/1533687684 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7effa4003b70 msgr2=0x7effc015b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:09.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:09.858+0000 7effc4a00640 1 --2- 172.21.15.6:0/1533687684 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7effa4003b70 0x7effc015b670 secure :-1 s=READY pgs=1779 cs=0 l=1 rev1=1 crypto rx=0x7effb000c200 tx=0x7effb0001970 comp rx=0 tx=0).stop 2024-03-04T09:36:09.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:09.859+0000 7effc4a00640 1 -- 172.21.15.6:0/1533687684 shutdown_connections 2024-03-04T09:36:09.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:09.859+0000 7effc4a00640 1 --2- 172.21.15.6:0/1533687684 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7eff9c06b500 0x7eff9c06d9c0 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:09.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:09.859+0000 7effc4a00640 1 --2- 172.21.15.6:0/1533687684 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7effc00b35d0 0x7effc00b3a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:09.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:09.859+0000 7effc4a00640 1 --2- 172.21.15.6:0/1533687684 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7effc00b8e60 0x7effc00b72b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:09.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:09.859+0000 7effc4a00640 1 --2- 172.21.15.6:0/1533687684 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7effa4003b70 0x7effc015b670 unknown :-1 s=CLOSED pgs=1779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:09.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:09.859+0000 7effc4a00640 1 -- 172.21.15.6:0/1533687684 >> 172.21.15.6:0/1533687684 conn(0x7effc00d3d40 msgr2=0x7effc00b20f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:36:09.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:09.859+0000 7effc4a00640 1 -- 172.21.15.6:0/1533687684 shutdown_connections 2024-03-04T09:36:09.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:09.859+0000 7effc4a00640 1 -- 172.21.15.6:0/1533687684 wait complete. 2024-03-04T09:36:09.931 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-04T09:36:10.078 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:32:57.159681+0000', 'modified': '2024-03-04T09:35:26.088222+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8214, 'mds_1': 6796}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6796': {'gid': 6796, 'name': 'd', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.146:6837/2458070840', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 2458070840}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 2458070840}]}, 'join_fscid': -1, 'laggy_since': '2024-03-04T09:33:54.652601+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8214': {'gid': 8214, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2446627517', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2446627517}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2446627517}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8214, 'qdb_cluster': [8214]} 2024-03-04T09:36:10.078 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6796:up:stopping 2024-03-04T09:36:11.079 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:36:11.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:11.368+0000 7f1c80200640 1 Processor -- start 2024-03-04T09:36:11.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:11.369+0000 7f1c80200640 1 -- start start 2024-03-04T09:36:11.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:11.371+0000 7f1c80200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f1c780b3c90 0x7f1c780cdc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:11.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:11.371+0000 7f1c80200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1c780cf5c0 0x7f1c780b4070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:11.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:11.371+0000 7f1c80200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c780b45b0 con 0x7f1c7806cb10 2024-03-04T09:36:11.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:11.371+0000 7f1c80200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c780ce160 con 0x7f1c780b3c90 2024-03-04T09:36:11.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:11.371+0000 7f1c80200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c780b4860 con 0x7f1c780cf5c0 2024-03-04T09:36:11.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:11.372+0000 7f1c7d800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f1c780b3c90 0x7f1c780cdc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:11.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:11.372+0000 7f1c7ce00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1c780cf5c0 0x7f1c780b4070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:11.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:11.372+0000 7f1c7d800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f1c780b3c90 0x7f1c780cdc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.6:56334/0 (socket says 172.21.15.6:56334) 2024-03-04T09:36:11.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:11.372+0000 7f1c7d800640 1 -- 172.21.15.6:0/1024213842 learned_addr learned my addr 172.21.15.6:0/1024213842 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:36:11.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:11.373+0000 7f1c7d800640 1 -- 172.21.15.6:0/1024213842 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1c780cf5c0 msgr2=0x7f1c780b4070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:36:11.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:11.373+0000 7f1c7d800640 1 --2- 172.21.15.6:0/1024213842 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1c780cf5c0 0x7f1c780b4070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:11.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:11.373+0000 7f1c7d800640 1 -- 172.21.15.6:0/1024213842 >> v1:172.21.15.146:6789/0 conn(0x7f1c7806cb10 legacy=0x7f1c7806cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:11.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:11.373+0000 7f1c7d800640 1 -- 172.21.15.6:0/1024213842 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c780b4b40 con 0x7f1c780b3c90 2024-03-04T09:36:11.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:11.374+0000 7f1c7d800640 1 --2- 172.21.15.6:0/1024213842 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c780b3c90 0x7f1c780cdc20 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f1c6c009870 tx=0x7f1c6c02ba10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5d67df0a272a3ef2 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:36:11.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:11.374+0000 7f1c77e00640 1 -- 172.21.15.6:0/1024213842 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c6c02f830 con 0x7f1c780b3c90 2024-03-04T09:36:11.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:11.374+0000 7f1c77e00640 1 -- 172.21.15.6:0/1024213842 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c6c02f9d0 con 0x7f1c780b3c90 2024-03-04T09:36:11.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:11.374+0000 7f1c77e00640 1 -- 172.21.15.6:0/1024213842 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c6c02fda0 con 0x7f1c780b3c90 2024-03-04T09:36:11.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:11.375+0000 7f1c80200640 1 -- 172.21.15.6:0/1024213842 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c780b3c90 msgr2=0x7f1c780cdc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:11.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:11.375+0000 7f1c80200640 1 --2- 172.21.15.6:0/1024213842 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c780b3c90 0x7f1c780cdc20 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f1c6c009870 tx=0x7f1c6c02ba10 comp rx=0 tx=0).stop 2024-03-04T09:36:11.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:11.375+0000 7f1c80200640 1 -- 172.21.15.6:0/1024213842 shutdown_connections 2024-03-04T09:36:11.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:11.375+0000 7f1c80200640 1 --2- 172.21.15.6:0/1024213842 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1c780cf5c0 0x7f1c780b4070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:11.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:11.375+0000 7f1c80200640 1 --2- 172.21.15.6:0/1024213842 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c780b3c90 0x7f1c780cdc20 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:11.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:11.375+0000 7f1c80200640 1 -- 172.21.15.6:0/1024213842 >> 172.21.15.6:0/1024213842 conn(0x7f1c7806afb0 msgr2=0x7f1c7806b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:36:11.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:11.375+0000 7f1c80200640 1 -- 172.21.15.6:0/1024213842 shutdown_connections 2024-03-04T09:36:11.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:11.375+0000 7f1c80200640 1 -- 172.21.15.6:0/1024213842 wait complete. 2024-03-04T09:36:11.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:11.378+0000 7f1c80200640 1 Processor -- start 2024-03-04T09:36:11.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:11.378+0000 7f1c80200640 1 -- start start 2024-03-04T09:36:11.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:11.379+0000 7f1c80200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1c7806cb10 0x7f1c7809ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:11.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:11.379+0000 7f1c80200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1c780b3c90 0x7f1c781f7c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:11.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:11.379+0000 7f1c80200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c780cf5c0 0x7f1c7809cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:11.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:11.379+0000 7f1c80200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c7815b340 con 0x7f1c7806cb10 2024-03-04T09:36:11.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:11.379+0000 7f1c80200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c780b5100 con 0x7f1c780cf5c0 2024-03-04T09:36:11.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:11.379+0000 7f1c80200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c7806d6b0 con 0x7f1c780b3c90 2024-03-04T09:36:11.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:11.380+0000 7f1c7d800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1c7806cb10 0x7f1c7809ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:11.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:11.380+0000 7f1c7d800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1c7806cb10 0x7f1c7809ab60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:54214/0 (socket says 172.21.15.6:54214) 2024-03-04T09:36:11.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:11.380+0000 7f1c7d800640 1 -- 172.21.15.6:0/1079214354 learned_addr learned my addr 172.21.15.6:0/1079214354 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:36:11.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:11.380+0000 7f1c7e200640 1 --2- 172.21.15.6:0/1079214354 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c780cf5c0 0x7f1c7809cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:11.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:11.380+0000 7f1c7ce00640 1 --2- 172.21.15.6:0/1079214354 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1c780b3c90 0x7f1c781f7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:11.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:11.380+0000 7f1c7d800640 1 -- 172.21.15.6:0/1079214354 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1c780b3c90 msgr2=0x7f1c781f7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:11.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:11.380+0000 7f1c7d800640 1 --2- 172.21.15.6:0/1079214354 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1c780b3c90 0x7f1c781f7c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:11.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:11.380+0000 7f1c7d800640 1 -- 172.21.15.6:0/1079214354 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c780cf5c0 msgr2=0x7f1c7809cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:11.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:11.380+0000 7f1c7d800640 1 --2- 172.21.15.6:0/1079214354 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c780cf5c0 0x7f1c7809cd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:11.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:11.380+0000 7f1c7d800640 1 -- 172.21.15.6:0/1079214354 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c781f8160 con 0x7f1c7806cb10 2024-03-04T09:36:11.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:11.381+0000 7f1c7d800640 1 --2- 172.21.15.6:0/1079214354 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1c7806cb10 0x7f1c7809ab60 secure :-1 s=READY pgs=1780 cs=0 l=1 rev1=1 crypto rx=0x7f1c6c000c00 tx=0x7f1c6c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:36:11.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:11.381+0000 7f1c75600640 1 -- 172.21.15.6:0/1079214354 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c6c037c40 con 0x7f1c7806cb10 2024-03-04T09:36:11.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:11.381+0000 7f1c75600640 1 -- 172.21.15.6:0/1079214354 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c6c046070 con 0x7f1c7806cb10 2024-03-04T09:36:11.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:11.381+0000 7f1c80200640 1 -- 172.21.15.6:0/1079214354 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c781f3d30 con 0x7f1c7806cb10 2024-03-04T09:36:11.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:11.381+0000 7f1c75600640 1 -- 172.21.15.6:0/1079214354 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c6c041400 con 0x7f1c7806cb10 2024-03-04T09:36:11.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:11.381+0000 7f1c80200640 1 -- 172.21.15.6:0/1079214354 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c781f42f0 con 0x7f1c7806cb10 2024-03-04T09:36:11.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:11.383+0000 7f1c80200640 1 -- 172.21.15.6:0/1079214354 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c78069510 con 0x7f1c7806cb10 2024-03-04T09:36:11.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:11.383+0000 7f1c75600640 1 -- 172.21.15.6:0/1079214354 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1c6c040070 con 0x7f1c7806cb10 2024-03-04T09:36:11.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:11.385+0000 7f1c75600640 1 --2- 172.21.15.6:0/1079214354 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f1c5806b4d0 0x7f1c5806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:11.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:11.385+0000 7f1c75600640 1 -- 172.21.15.6:0/1079214354 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f1c6c0ac8e0 con 0x7f1c7806cb10 2024-03-04T09:36:11.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:11.386+0000 7f1c7ce00640 1 --2- 172.21.15.6:0/1079214354 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f1c5806b4d0 0x7f1c5806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:11.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:11.387+0000 7f1c7ce00640 1 --2- 172.21.15.6:0/1079214354 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f1c5806b4d0 0x7f1c5806d990 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f1c781517e0 tx=0x7f1c64008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:36:11.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:11.389+0000 7f1c75600640 1 -- 172.21.15.6:0/1079214354 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1c781f8160 con 0x7f1c7806cb10 2024-03-04T09:36:11.580 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:36:11.588 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:36:11.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:11.720+0000 7f1c80200640 1 -- 172.21.15.6:0/1079214354 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1c780b1580 con 0x7f1c7806cb10 2024-03-04T09:36:11.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:11.720+0000 7f1c75600640 1 -- 172.21.15.6:0/1079214354 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3464 (secure 0 0 0) 0x7f1c6c033030 con 0x7f1c7806cb10 2024-03-04T09:36:11.717 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:36:11.718 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:35:26.088222+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"laggy_since":"2024-03-04T09:33:54.652601+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":9}]} 2024-03-04T09:36:11.718 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 101 2024-03-04T09:36:11.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:11.724+0000 7f1c80200640 1 -- 172.21.15.6:0/1079214354 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f1c5806b4d0 msgr2=0x7f1c5806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:11.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:11.724+0000 7f1c80200640 1 --2- 172.21.15.6:0/1079214354 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f1c5806b4d0 0x7f1c5806d990 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f1c781517e0 tx=0x7f1c64008040 comp rx=0 tx=0).stop 2024-03-04T09:36:11.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:11.724+0000 7f1c80200640 1 -- 172.21.15.6:0/1079214354 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1c7806cb10 msgr2=0x7f1c7809ab60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:11.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:11.724+0000 7f1c80200640 1 --2- 172.21.15.6:0/1079214354 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1c7806cb10 0x7f1c7809ab60 secure :-1 s=READY pgs=1780 cs=0 l=1 rev1=1 crypto rx=0x7f1c6c000c00 tx=0x7f1c6c004730 comp rx=0 tx=0).stop 2024-03-04T09:36:11.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:11.724+0000 7f1c80200640 1 -- 172.21.15.6:0/1079214354 shutdown_connections 2024-03-04T09:36:11.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:11.724+0000 7f1c80200640 1 --2- 172.21.15.6:0/1079214354 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c780cf5c0 0x7f1c7809cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:11.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:11.724+0000 7f1c80200640 1 --2- 172.21.15.6:0/1079214354 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f1c5806b4d0 0x7f1c5806d990 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:11.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:11.724+0000 7f1c80200640 1 --2- 172.21.15.6:0/1079214354 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1c780b3c90 0x7f1c781f7c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:11.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:11.724+0000 7f1c80200640 1 --2- 172.21.15.6:0/1079214354 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1c7806cb10 0x7f1c7809ab60 unknown :-1 s=CLOSED pgs=1780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:11.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:11.724+0000 7f1c80200640 1 -- 172.21.15.6:0/1079214354 >> 172.21.15.6:0/1079214354 conn(0x7f1c7806afb0 msgr2=0x7f1c78158880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:36:11.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:11.724+0000 7f1c80200640 1 -- 172.21.15.6:0/1079214354 shutdown_connections 2024-03-04T09:36:11.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:11.724+0000 7f1c80200640 1 -- 172.21.15.6:0/1079214354 wait complete. 2024-03-04T09:36:11.933 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-04T09:36:11.943 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:32:57.159681+0000', 'modified': '2024-03-04T09:35:26.088222+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8214, 'mds_1': 6796}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6796': {'gid': 6796, 'name': 'd', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.146:6837/2458070840', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 2458070840}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 2458070840}]}, 'join_fscid': -1, 'laggy_since': '2024-03-04T09:33:54.652601+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8214': {'gid': 8214, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2446627517', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2446627517}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2446627517}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8214, 'qdb_cluster': [8214]} 2024-03-04T09:36:11.944 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6796:up:stopping 2024-03-04T09:36:12.944 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:36:13.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:13.162+0000 7fb279400640 1 Processor -- start 2024-03-04T09:36:13.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:13.162+0000 7fb279400640 1 -- start start 2024-03-04T09:36:13.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:13.163+0000 7fb279400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb2740d4a10 0x7fb2740d0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:13.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:13.163+0000 7fb279400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fb2740d0c80 0x7fb2740b44d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:13.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:13.163+0000 7fb279400640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2740b3710 con 0x7fb27415e0b0 2024-03-04T09:36:13.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:13.164+0000 7fb279400640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2740b09e0 con 0x7fb2740d0c80 2024-03-04T09:36:13.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:13.164+0000 7fb279400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2740cfa20 con 0x7fb2740d4a10 2024-03-04T09:36:13.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:13.164+0000 7fb273e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fb27415e0b0 0x7fb2740cf310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:53468/0 (socket says 172.21.15.6:53468) 2024-03-04T09:36:13.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:13.164+0000 7fb273e00640 1 -- 172.21.15.6:0/1875681417 learned_addr learned my addr 172.21.15.6:0/1875681417 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:36:13.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:13.164+0000 7fb273400640 1 --2- 172.21.15.6:0/1875681417 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb2740d4a10 0x7fb2740d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:13.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:13.164+0000 7fb278a00640 1 --2- 172.21.15.6:0/1875681417 >> v2:172.21.15.146:3300/0 conn(0x7fb2740d0c80 0x7fb2740b44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:13.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:13.164+0000 7fb272a00640 1 -- 172.21.15.6:0/1875681417 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3526343871 0 0) 0x7fb2740b3710 con 0x7fb27415e0b0 2024-03-04T09:36:13.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:13.164+0000 7fb272a00640 1 -- 172.21.15.6:0/1875681417 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb258003520 con 0x7fb27415e0b0 2024-03-04T09:36:13.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:13.165+0000 7fb272a00640 1 -- 172.21.15.6:0/1875681417 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4089244318 0 0) 0x7fb258003520 con 0x7fb27415e0b0 2024-03-04T09:36:13.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:13.165+0000 7fb272a00640 1 -- 172.21.15.6:0/1875681417 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb2740d4a10 msgr2=0x7fb2740d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:36:13.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:13.165+0000 7fb272a00640 1 --2- 172.21.15.6:0/1875681417 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb2740d4a10 0x7fb2740d0740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:13.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:13.165+0000 7fb272a00640 1 -- 172.21.15.6:0/1875681417 >> v2:172.21.15.146:3300/0 conn(0x7fb2740d0c80 msgr2=0x7fb2740b44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:36:13.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:13.165+0000 7fb272a00640 1 --2- 172.21.15.6:0/1875681417 >> v2:172.21.15.146:3300/0 conn(0x7fb2740d0c80 0x7fb2740b44d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:13.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:13.165+0000 7fb272a00640 1 -- 172.21.15.6:0/1875681417 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2740b39f0 con 0x7fb27415e0b0 2024-03-04T09:36:13.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:13.165+0000 7fb272a00640 1 -- 172.21.15.6:0/1875681417 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7fb264001670 con 0x7fb27415e0b0 2024-03-04T09:36:13.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:13.165+0000 7fb272a00640 1 -- 172.21.15.6:0/1875681417 >> v1:172.21.15.6:6789/0 conn(0x7fb27415e0b0 legacy=0x7fb2740cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:13.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:13.165+0000 7fb272a00640 1 --2- 172.21.15.6:0/1875681417 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb2580039b0 0x7fb258003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:13.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:13.165+0000 7fb272a00640 1 --2- 172.21.15.6:0/1875681417 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb258004800 0x7fb258004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:13.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:13.165+0000 7fb272a00640 1 --2- 172.21.15.6:0/1875681417 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb2740d4a10 0x7fb2580051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:13.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:13.165+0000 7fb272a00640 1 -- 172.21.15.6:0/1875681417 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2740cfa20 con 0x7fb258004800 2024-03-04T09:36:13.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:13.165+0000 7fb272a00640 1 -- 172.21.15.6:0/1875681417 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb25800b9c0 con 0x7fb2580039b0 2024-03-04T09:36:13.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:13.165+0000 7fb272a00640 1 -- 172.21.15.6:0/1875681417 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb25800bb40 con 0x7fb2740d4a10 2024-03-04T09:36:13.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:13.165+0000 7fb273400640 1 --2- 172.21.15.6:0/1875681417 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb258004800 0x7fb258004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:13.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:13.166+0000 7fb273e00640 1 --2- 172.21.15.6:0/1875681417 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb2580039b0 0x7fb258003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:13.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:13.166+0000 7fb273400640 1 -- 172.21.15.6:0/1875681417 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb2740d4a10 msgr2=0x7fb2580051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:36:13.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:13.166+0000 7fb273400640 1 --2- 172.21.15.6:0/1875681417 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb2740d4a10 0x7fb2580051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:13.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:13.166+0000 7fb273400640 1 -- 172.21.15.6:0/1875681417 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb2580039b0 msgr2=0x7fb258003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:36:13.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:13.166+0000 7fb273400640 1 --2- 172.21.15.6:0/1875681417 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb2580039b0 0x7fb258003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:13.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:13.166+0000 7fb273400640 1 -- 172.21.15.6:0/1875681417 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb258003520 con 0x7fb258004800 2024-03-04T09:36:13.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:13.166+0000 7fb273e00640 1 --2- 172.21.15.6:0/1875681417 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb2580039b0 0x7fb258003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:36:13.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:13.167+0000 7fb273400640 1 --2- 172.21.15.6:0/1875681417 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb258004800 0x7fb258004c80 secure :-1 s=READY pgs=1782 cs=0 l=1 rev1=1 crypto rx=0x7fb260009a90 tx=0x7fb258005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43c81d207ec15905 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:36:13.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:13.167+0000 7fb272a00640 1 -- 172.21.15.6:0/1875681417 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb260039070 con 0x7fb258004800 2024-03-04T09:36:13.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:13.167+0000 7fb272a00640 1 -- 172.21.15.6:0/1875681417 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb26002c6b0 con 0x7fb258004800 2024-03-04T09:36:13.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:13.167+0000 7fb272a00640 1 -- 172.21.15.6:0/1875681417 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb260009690 con 0x7fb258004800 2024-03-04T09:36:13.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:13.168+0000 7fb279400640 1 -- 172.21.15.6:0/1875681417 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb258004800 msgr2=0x7fb258004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:13.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:13.168+0000 7fb279400640 1 --2- 172.21.15.6:0/1875681417 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb258004800 0x7fb258004c80 secure :-1 s=READY pgs=1782 cs=0 l=1 rev1=1 crypto rx=0x7fb260009a90 tx=0x7fb258005830 comp rx=0 tx=0).stop 2024-03-04T09:36:13.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:13.168+0000 7fb278a00640 1 -- 172.21.15.6:0/1875681417 reap_dead start 2024-03-04T09:36:13.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:13.168+0000 7fb279400640 1 -- 172.21.15.6:0/1875681417 shutdown_connections 2024-03-04T09:36:13.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:13.168+0000 7fb279400640 1 -- 172.21.15.6:0/1875681417 >> 172.21.15.6:0/1875681417 conn(0x7fb2740d3d30 msgr2=0x7fb2740d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:36:13.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:13.168+0000 7fb279400640 1 -- 172.21.15.6:0/1875681417 shutdown_connections 2024-03-04T09:36:13.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:13.168+0000 7fb279400640 1 -- 172.21.15.6:0/1875681417 wait complete. 2024-03-04T09:36:13.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:13.170+0000 7fb279400640 1 Processor -- start 2024-03-04T09:36:13.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:13.170+0000 7fb279400640 1 -- start start 2024-03-04T09:36:13.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:13.171+0000 7fb279400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb258004800 0x7fb27415f660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:13.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:13.171+0000 7fb279400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb2741f4090 0x7fb2740cd380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:13.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:13.171+0000 7fb279400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb274147b60 0x7fb2740cd8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:13.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:13.171+0000 7fb279400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb27415e490 con 0x7fb274147b60 2024-03-04T09:36:13.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:13.171+0000 7fb279400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2740b3fb0 con 0x7fb2741f4090 2024-03-04T09:36:13.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:13.171+0000 7fb279400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2740cfc60 con 0x7fb258004800 2024-03-04T09:36:13.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:13.171+0000 7fb278a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb274147b60 0x7fb2740cd8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:13.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:13.171+0000 7fb278a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb274147b60 0x7fb2740cd8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:54238/0 (socket says 172.21.15.6:54238) 2024-03-04T09:36:13.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:13.171+0000 7fb278a00640 1 -- 172.21.15.6:0/733777238 learned_addr learned my addr 172.21.15.6:0/733777238 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:36:13.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:13.171+0000 7fb273400640 1 --2- 172.21.15.6:0/733777238 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb2741f4090 0x7fb2740cd380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:13.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:13.171+0000 7fb273e00640 1 --2- 172.21.15.6:0/733777238 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb258004800 0x7fb27415f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:13.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:13.172+0000 7fb278a00640 1 -- 172.21.15.6:0/733777238 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb258004800 msgr2=0x7fb27415f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:13.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:13.172+0000 7fb278a00640 1 --2- 172.21.15.6:0/733777238 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb258004800 0x7fb27415f660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:13.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:13.172+0000 7fb278a00640 1 -- 172.21.15.6:0/733777238 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb2741f4090 msgr2=0x7fb2740cd380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:13.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:13.172+0000 7fb278a00640 1 --2- 172.21.15.6:0/733777238 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb2741f4090 0x7fb2740cd380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:13.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:13.172+0000 7fb278a00640 1 -- 172.21.15.6:0/733777238 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb27415ff60 con 0x7fb274147b60 2024-03-04T09:36:13.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:13.172+0000 7fb278a00640 1 --2- 172.21.15.6:0/733777238 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb274147b60 0x7fb2740cd8c0 secure :-1 s=READY pgs=1783 cs=0 l=1 rev1=1 crypto rx=0x7fb2740d3c80 tx=0x7fb26c00b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:36:13.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:13.172+0000 7fb26be00640 1 -- 172.21.15.6:0/733777238 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb26c00bae0 con 0x7fb274147b60 2024-03-04T09:36:13.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:13.172+0000 7fb26be00640 1 -- 172.21.15.6:0/733777238 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb26c01a070 con 0x7fb274147b60 2024-03-04T09:36:13.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:13.172+0000 7fb279400640 1 -- 172.21.15.6:0/733777238 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb274160240 con 0x7fb274147b60 2024-03-04T09:36:13.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:13.172+0000 7fb26be00640 1 -- 172.21.15.6:0/733777238 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb26c014400 con 0x7fb274147b60 2024-03-04T09:36:13.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:13.173+0000 7fb279400640 1 -- 172.21.15.6:0/733777238 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb274160800 con 0x7fb274147b60 2024-03-04T09:36:13.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:13.174+0000 7fb279400640 1 -- 172.21.15.6:0/733777238 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb27404f330 con 0x7fb274147b60 2024-03-04T09:36:13.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:13.175+0000 7fb26be00640 1 -- 172.21.15.6:0/733777238 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb26c005d10 con 0x7fb274147b60 2024-03-04T09:36:13.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:13.176+0000 7fb26be00640 1 --2- 172.21.15.6:0/733777238 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb25406b500 0x7fb25406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:13.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:13.176+0000 7fb26be00640 1 -- 172.21.15.6:0/733777238 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fb26c08a550 con 0x7fb274147b60 2024-03-04T09:36:13.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:13.176+0000 7fb273e00640 1 --2- 172.21.15.6:0/733777238 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb25406b500 0x7fb25406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:13.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:13.178+0000 7fb273e00640 1 --2- 172.21.15.6:0/733777238 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb25406b500 0x7fb25406d9c0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7fb264001eb0 tx=0x7fb264009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:36:13.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:13.180+0000 7fb26be00640 1 -- 172.21.15.6:0/733777238 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb26c024020 con 0x7fb274147b60 2024-03-04T09:36:13.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:13.420+0000 7fb279400640 1 -- 172.21.15.6:0/733777238 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb274161640 con 0x7fb274147b60 2024-03-04T09:36:13.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:13.421+0000 7fb26be00640 1 -- 172.21.15.6:0/733777238 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3464 (secure 0 0 0) 0x7fb26c059260 con 0x7fb274147b60 2024-03-04T09:36:13.418 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:36:13.418 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:35:26.088222+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"laggy_since":"2024-03-04T09:33:54.652601+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":9}]} 2024-03-04T09:36:13.418 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 101 2024-03-04T09:36:13.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:13.423+0000 7fb279400640 1 -- 172.21.15.6:0/733777238 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb25406b500 msgr2=0x7fb25406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:13.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:13.423+0000 7fb279400640 1 --2- 172.21.15.6:0/733777238 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb25406b500 0x7fb25406d9c0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7fb264001eb0 tx=0x7fb264009040 comp rx=0 tx=0).stop 2024-03-04T09:36:13.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:13.423+0000 7fb279400640 1 -- 172.21.15.6:0/733777238 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb274147b60 msgr2=0x7fb2740cd8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:13.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:13.423+0000 7fb279400640 1 --2- 172.21.15.6:0/733777238 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb274147b60 0x7fb2740cd8c0 secure :-1 s=READY pgs=1783 cs=0 l=1 rev1=1 crypto rx=0x7fb2740d3c80 tx=0x7fb26c00b480 comp rx=0 tx=0).stop 2024-03-04T09:36:13.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:13.423+0000 7fb279400640 1 -- 172.21.15.6:0/733777238 shutdown_connections 2024-03-04T09:36:13.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:13.423+0000 7fb279400640 1 --2- 172.21.15.6:0/733777238 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb274147b60 0x7fb2740cd8c0 unknown :-1 s=CLOSED pgs=1783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:13.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:13.423+0000 7fb279400640 1 --2- 172.21.15.6:0/733777238 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb2741f4090 0x7fb2740cd380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:13.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:13.423+0000 7fb279400640 1 --2- 172.21.15.6:0/733777238 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb25406b500 0x7fb25406d9c0 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:13.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:13.423+0000 7fb279400640 1 --2- 172.21.15.6:0/733777238 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb258004800 0x7fb27415f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:13.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:13.423+0000 7fb279400640 1 -- 172.21.15.6:0/733777238 >> 172.21.15.6:0/733777238 conn(0x7fb2740d3d30 msgr2=0x7fb2740cf070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:36:13.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:13.424+0000 7fb279400640 1 -- 172.21.15.6:0/733777238 shutdown_connections 2024-03-04T09:36:13.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:13.424+0000 7fb279400640 1 -- 172.21.15.6:0/733777238 wait complete. 2024-03-04T09:36:13.640 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:32:57.159681+0000', 'modified': '2024-03-04T09:35:26.088222+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8214, 'mds_1': 6796}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6796': {'gid': 6796, 'name': 'd', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.146:6837/2458070840', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 2458070840}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 2458070840}]}, 'join_fscid': -1, 'laggy_since': '2024-03-04T09:33:54.652601+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8214': {'gid': 8214, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2446627517', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2446627517}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2446627517}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8214, 'qdb_cluster': [8214]} 2024-03-04T09:36:13.640 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6796:up:stopping 2024-03-04T09:36:13.935 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-04T09:36:14.642 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:36:14.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:14.878+0000 7fa71a600640 1 Processor -- start 2024-03-04T09:36:14.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:14.878+0000 7fa71a600640 1 -- start start 2024-03-04T09:36:14.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:14.879+0000 7fa71a600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa71415e070 0x7fa7140d2960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:14.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:14.879+0000 7fa71a600640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fa7140d0cd0 0x7fa7140cf870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:14.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:14.879+0000 7fa71a600640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7140d18c0 con 0x7fa7140cf410 2024-03-04T09:36:14.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:14.879+0000 7fa71a600640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7140d1ba0 con 0x7fa7140d0cd0 2024-03-04T09:36:14.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:14.879+0000 7fa71a600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7140d1e80 con 0x7fa71415e070 2024-03-04T09:36:14.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:14.880+0000 7fa719200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa71415e070 0x7fa7140d2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:14.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:14.880+0000 7fa719c00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fa7140d0cd0 0x7fa7140cf870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:14.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:14.880+0000 7fa719200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa71415e070 0x7fa7140d2960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.6:36854/0 (socket says 172.21.15.6:36854) 2024-03-04T09:36:14.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:14.880+0000 7fa719c00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fa7140d0cd0 0x7fa7140cf870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.6:56382/0 (socket says 172.21.15.6:56382) 2024-03-04T09:36:14.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:14.880+0000 7fa719200640 1 -- 172.21.15.6:0/642212742 learned_addr learned my addr 172.21.15.6:0/642212742 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:36:14.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:14.881+0000 7fa719200640 1 -- 172.21.15.6:0/642212742 >> v2:172.21.15.146:3300/0 conn(0x7fa7140d0cd0 msgr2=0x7fa7140cf870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:36:14.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:14.881+0000 7fa719200640 1 --2- 172.21.15.6:0/642212742 >> v2:172.21.15.146:3300/0 conn(0x7fa7140d0cd0 0x7fa7140cf870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:14.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:14.881+0000 7fa719200640 1 -- 172.21.15.6:0/642212742 >> v1:172.21.15.146:6789/0 conn(0x7fa7140cf410 legacy=0x7fa7140d05c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:14.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:14.881+0000 7fa719200640 1 -- 172.21.15.6:0/642212742 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7140d2160 con 0x7fa71415e070 2024-03-04T09:36:14.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:14.882+0000 7fa719200640 1 --2- 172.21.15.6:0/642212742 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa71415e070 0x7fa7140d2960 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fa704009870 tx=0x7fa70402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cca152cab402f06c server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:36:14.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:14.883+0000 7fa713400640 1 -- 172.21.15.6:0/642212742 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa704034070 con 0x7fa71415e070 2024-03-04T09:36:14.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:14.883+0000 7fa713400640 1 -- 172.21.15.6:0/642212742 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa704030840 con 0x7fa71415e070 2024-03-04T09:36:14.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:14.883+0000 7fa713400640 1 -- 172.21.15.6:0/642212742 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa704030c10 con 0x7fa71415e070 2024-03-04T09:36:14.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:14.883+0000 7fa71a600640 1 -- 172.21.15.6:0/642212742 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa71415e070 msgr2=0x7fa7140d2960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:14.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:14.883+0000 7fa71a600640 1 --2- 172.21.15.6:0/642212742 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa71415e070 0x7fa7140d2960 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fa704009870 tx=0x7fa70402b980 comp rx=0 tx=0).stop 2024-03-04T09:36:14.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:14.884+0000 7fa71a600640 1 -- 172.21.15.6:0/642212742 shutdown_connections 2024-03-04T09:36:14.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:14.884+0000 7fa71a600640 1 --2- 172.21.15.6:0/642212742 >> v2:172.21.15.146:3300/0 conn(0x7fa7140d0cd0 0x7fa7140cf870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:14.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:14.884+0000 7fa71a600640 1 --2- 172.21.15.6:0/642212742 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa71415e070 0x7fa7140d2960 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:14.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:14.884+0000 7fa71a600640 1 -- 172.21.15.6:0/642212742 >> 172.21.15.6:0/642212742 conn(0x7fa71408d2a0 msgr2=0x7fa7140d3cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:36:14.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:14.884+0000 7fa71a600640 1 -- 172.21.15.6:0/642212742 shutdown_connections 2024-03-04T09:36:14.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:14.884+0000 7fa71a600640 1 -- 172.21.15.6:0/642212742 wait complete. 2024-03-04T09:36:14.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:14.885+0000 7fa71a600640 1 Processor -- start 2024-03-04T09:36:14.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:14.886+0000 7fa71a600640 1 -- start start 2024-03-04T09:36:14.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:14.887+0000 7fa71a600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa7140cf410 0x7fa714160200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:14.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:14.887+0000 7fa71a600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa7140d0cd0 0x7fa7140c77e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:14.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:14.887+0000 7fa71a600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa71415e070 0x7fa7140c9320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:14.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:14.887+0000 7fa71a600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7140d4730 con 0x7fa7140d0cd0 2024-03-04T09:36:14.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:14.887+0000 7fa71a600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7140d45b0 con 0x7fa71415e070 2024-03-04T09:36:14.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:14.887+0000 7fa71a600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7140b3630 con 0x7fa7140cf410 2024-03-04T09:36:14.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:14.887+0000 7fa713e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa7140d0cd0 0x7fa7140c77e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:14.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:14.887+0000 7fa713e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa7140d0cd0 0x7fa7140c77e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:54252/0 (socket says 172.21.15.6:54252) 2024-03-04T09:36:14.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:14.887+0000 7fa713e00640 1 -- 172.21.15.6:0/698909526 learned_addr learned my addr 172.21.15.6:0/698909526 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:36:14.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:14.887+0000 7fa719200640 1 --2- 172.21.15.6:0/698909526 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa7140cf410 0x7fa714160200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:14.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:14.887+0000 7fa719c00640 1 --2- 172.21.15.6:0/698909526 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa71415e070 0x7fa7140c9320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:14.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:14.888+0000 7fa713e00640 1 -- 172.21.15.6:0/698909526 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa7140cf410 msgr2=0x7fa714160200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:14.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:14.888+0000 7fa713e00640 1 --2- 172.21.15.6:0/698909526 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa7140cf410 0x7fa714160200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:14.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:14.888+0000 7fa713e00640 1 -- 172.21.15.6:0/698909526 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa71415e070 msgr2=0x7fa7140c9320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:14.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:14.888+0000 7fa713e00640 1 --2- 172.21.15.6:0/698909526 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa71415e070 0x7fa7140c9320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:14.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:14.888+0000 7fa713e00640 1 -- 172.21.15.6:0/698909526 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7140ca160 con 0x7fa7140d0cd0 2024-03-04T09:36:14.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:14.888+0000 7fa713e00640 1 --2- 172.21.15.6:0/698909526 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa7140d0cd0 0x7fa7140c77e0 secure :-1 s=READY pgs=1784 cs=0 l=1 rev1=1 crypto rx=0x7fa70000c7c0 tx=0x7fa70000cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:36:14.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:14.888+0000 7fa710c00640 1 -- 172.21.15.6:0/698909526 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa700016020 con 0x7fa7140d0cd0 2024-03-04T09:36:14.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:14.888+0000 7fa710c00640 1 -- 172.21.15.6:0/698909526 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa700006d50 con 0x7fa7140d0cd0 2024-03-04T09:36:14.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:14.888+0000 7fa710c00640 1 -- 172.21.15.6:0/698909526 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa700003bb0 con 0x7fa7140d0cd0 2024-03-04T09:36:14.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:14.888+0000 7fa71a600640 1 -- 172.21.15.6:0/698909526 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7140ca440 con 0x7fa7140d0cd0 2024-03-04T09:36:14.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:14.888+0000 7fa71a600640 1 -- 172.21.15.6:0/698909526 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7140caa00 con 0x7fa7140d0cd0 2024-03-04T09:36:14.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:14.890+0000 7fa71a600640 1 -- 172.21.15.6:0/698909526 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa71404f330 con 0x7fa7140d0cd0 2024-03-04T09:36:14.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:14.891+0000 7fa710c00640 1 -- 172.21.15.6:0/698909526 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa70000a850 con 0x7fa7140d0cd0 2024-03-04T09:36:14.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:14.892+0000 7fa710c00640 1 --2- 172.21.15.6:0/698909526 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa6f406b4d0 0x7fa6f406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:14.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:14.892+0000 7fa710c00640 1 -- 172.21.15.6:0/698909526 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fa700089ff0 con 0x7fa7140d0cd0 2024-03-04T09:36:14.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:14.892+0000 7fa719200640 1 --2- 172.21.15.6:0/698909526 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa6f406b4d0 0x7fa6f406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:14.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:14.893+0000 7fa719200640 1 --2- 172.21.15.6:0/698909526 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa6f406b4d0 0x7fa6f406d990 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7fa704036000 tx=0x7fa70403b040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:36:14.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:14.896+0000 7fa710c00640 1 -- 172.21.15.6:0/698909526 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa700020020 con 0x7fa7140d0cd0 2024-03-04T09:36:15.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:15.143+0000 7fa71a600640 1 -- 172.21.15.6:0/698909526 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa71415d720 con 0x7fa7140d0cd0 2024-03-04T09:36:15.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:15.144+0000 7fa710c00640 1 -- 172.21.15.6:0/698909526 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3464 (secure 0 0 0) 0x7fa700019310 con 0x7fa7140d0cd0 2024-03-04T09:36:15.141 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:36:15.141 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:35:26.088222+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"laggy_since":"2024-03-04T09:33:54.652601+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":9}]} 2024-03-04T09:36:15.141 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 101 2024-03-04T09:36:15.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:15.146+0000 7fa71a600640 1 -- 172.21.15.6:0/698909526 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa6f406b4d0 msgr2=0x7fa6f406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:15.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:15.146+0000 7fa71a600640 1 --2- 172.21.15.6:0/698909526 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa6f406b4d0 0x7fa6f406d990 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7fa704036000 tx=0x7fa70403b040 comp rx=0 tx=0).stop 2024-03-04T09:36:15.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:15.146+0000 7fa71a600640 1 -- 172.21.15.6:0/698909526 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa7140d0cd0 msgr2=0x7fa7140c77e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:15.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:15.146+0000 7fa71a600640 1 --2- 172.21.15.6:0/698909526 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa7140d0cd0 0x7fa7140c77e0 secure :-1 s=READY pgs=1784 cs=0 l=1 rev1=1 crypto rx=0x7fa70000c7c0 tx=0x7fa70000cc80 comp rx=0 tx=0).stop 2024-03-04T09:36:15.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:15.146+0000 7fa71a600640 1 -- 172.21.15.6:0/698909526 shutdown_connections 2024-03-04T09:36:15.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:15.146+0000 7fa71a600640 1 --2- 172.21.15.6:0/698909526 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa71415e070 0x7fa7140c9320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:15.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:15.146+0000 7fa71a600640 1 --2- 172.21.15.6:0/698909526 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa7140d0cd0 0x7fa7140c77e0 unknown :-1 s=CLOSED pgs=1784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:15.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:15.146+0000 7fa71a600640 1 --2- 172.21.15.6:0/698909526 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa6f406b4d0 0x7fa6f406d990 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:15.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:15.146+0000 7fa71a600640 1 --2- 172.21.15.6:0/698909526 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa7140cf410 0x7fa714160200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:15.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:15.146+0000 7fa71a600640 1 -- 172.21.15.6:0/698909526 >> 172.21.15.6:0/698909526 conn(0x7fa71408d2a0 msgr2=0x7fa7140ce8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:36:15.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:15.146+0000 7fa71a600640 1 -- 172.21.15.6:0/698909526 shutdown_connections 2024-03-04T09:36:15.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:15.147+0000 7fa71a600640 1 -- 172.21.15.6:0/698909526 wait complete. 2024-03-04T09:36:15.365 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:32:57.159681+0000', 'modified': '2024-03-04T09:35:26.088222+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8214, 'mds_1': 6796}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6796': {'gid': 6796, 'name': 'd', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.146:6837/2458070840', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 2458070840}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 2458070840}]}, 'join_fscid': -1, 'laggy_since': '2024-03-04T09:33:54.652601+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8214': {'gid': 8214, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2446627517', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2446627517}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2446627517}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8214, 'qdb_cluster': [8214]} 2024-03-04T09:36:15.365 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6796:up:stopping 2024-03-04T09:36:15.937 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-04T09:36:16.366 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:36:16.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:16.595+0000 7f17e0800640 1 Processor -- start 2024-03-04T09:36:16.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:16.595+0000 7f17e0800640 1 -- start start 2024-03-04T09:36:16.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:16.597+0000 7f17e0800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f17d80af1a0 0x7f17d815c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:16.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:16.597+0000 7f17e0800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f17d80af580 0x7f17d80b4c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:16.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:16.597+0000 7f17e0800640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17d815cc40 con 0x7f17d80cdc20 2024-03-04T09:36:16.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:16.597+0000 7f17e0800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17d815cf20 con 0x7f17d80af580 2024-03-04T09:36:16.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:16.597+0000 7f17e0800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17d815d200 con 0x7f17d80af1a0 2024-03-04T09:36:16.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:16.597+0000 7f17dd400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f17d80af580 0x7f17d80b4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:16.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:16.597+0000 7f17dd400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f17d80af580 0x7f17d80b4c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:54268/0 (socket says 172.21.15.6:54268) 2024-03-04T09:36:16.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:16.597+0000 7f17dd400640 1 -- 172.21.15.6:0/514545605 learned_addr learned my addr 172.21.15.6:0/514545605 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:36:16.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:16.598+0000 7f17dde00640 1 --2- 172.21.15.6:0/514545605 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f17d80af1a0 0x7f17d815c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:16.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:16.598+0000 7f17dd400640 1 -- 172.21.15.6:0/514545605 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f17d80af1a0 msgr2=0x7f17d815c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:36:16.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:16.598+0000 7f17dd400640 1 --2- 172.21.15.6:0/514545605 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f17d80af1a0 0x7f17d815c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:16.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:16.598+0000 7f17dd400640 1 -- 172.21.15.6:0/514545605 >> v1:172.21.15.146:6789/0 conn(0x7f17d80cdc20 legacy=0x7f17d80b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:16.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:16.598+0000 7f17dd400640 1 -- 172.21.15.6:0/514545605 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17d815d4e0 con 0x7f17d80af580 2024-03-04T09:36:16.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:16.599+0000 7f17dd400640 1 --2- 172.21.15.6:0/514545605 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f17d80af580 0x7f17d80b4c20 secure :-1 s=READY pgs=1785 cs=0 l=1 rev1=1 crypto rx=0x7f17c8009870 tx=0x7f17c802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0a5a16ed17f3979 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:36:16.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:16.600+0000 7f17dca00640 1 -- 172.21.15.6:0/514545605 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17c8035020 con 0x7f17d80af580 2024-03-04T09:36:16.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:16.600+0000 7f17dca00640 1 -- 172.21.15.6:0/514545605 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17c8031640 con 0x7f17d80af580 2024-03-04T09:36:16.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:16.600+0000 7f17dca00640 1 -- 172.21.15.6:0/514545605 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17c802d040 con 0x7f17d80af580 2024-03-04T09:36:16.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:16.600+0000 7f17e0800640 1 -- 172.21.15.6:0/514545605 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f17d80af580 msgr2=0x7f17d80b4c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:16.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:16.600+0000 7f17e0800640 1 --2- 172.21.15.6:0/514545605 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f17d80af580 0x7f17d80b4c20 secure :-1 s=READY pgs=1785 cs=0 l=1 rev1=1 crypto rx=0x7f17c8009870 tx=0x7f17c802b8f0 comp rx=0 tx=0).stop 2024-03-04T09:36:16.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:16.601+0000 7f17e0800640 1 -- 172.21.15.6:0/514545605 shutdown_connections 2024-03-04T09:36:16.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:16.601+0000 7f17e0800640 1 --2- 172.21.15.6:0/514545605 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f17d80af580 0x7f17d80b4c20 unknown :-1 s=CLOSED pgs=1785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:16.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:16.601+0000 7f17e0800640 1 --2- 172.21.15.6:0/514545605 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f17d80af1a0 0x7f17d815c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:16.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:16.601+0000 7f17e0800640 1 -- 172.21.15.6:0/514545605 >> 172.21.15.6:0/514545605 conn(0x7f17d806afb0 msgr2=0x7f17d806b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:36:16.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:16.601+0000 7f17e0800640 1 -- 172.21.15.6:0/514545605 shutdown_connections 2024-03-04T09:36:16.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:16.601+0000 7f17e0800640 1 -- 172.21.15.6:0/514545605 wait complete. 2024-03-04T09:36:16.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:16.603+0000 7f17e0800640 1 Processor -- start 2024-03-04T09:36:16.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:16.604+0000 7f17e0800640 1 -- start start 2024-03-04T09:36:16.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:16.605+0000 7f17e0800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f17d80af1a0 0x7f17d8158c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:16.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:16.605+0000 7f17e0800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f17d80af580 0x7f17d81f3120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:16.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:16.605+0000 7f17e0800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f17d80cdc20 0x7f17d8153ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:16.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:16.605+0000 7f17e0800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17d80b3660 con 0x7f17d80af1a0 2024-03-04T09:36:16.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:16.605+0000 7f17e0800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17d80b34e0 con 0x7f17d80af580 2024-03-04T09:36:16.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:16.605+0000 7f17e0800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17d80b37e0 con 0x7f17d80cdc20 2024-03-04T09:36:16.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:16.606+0000 7f17dde00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f17d80af1a0 0x7f17d8158c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:16.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:16.606+0000 7f17dde00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f17d80af1a0 0x7f17d8158c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:54282/0 (socket says 172.21.15.6:54282) 2024-03-04T09:36:16.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:16.606+0000 7f17dde00640 1 -- 172.21.15.6:0/2349368767 learned_addr learned my addr 172.21.15.6:0/2349368767 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:36:16.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:16.606+0000 7f17dd400640 1 --2- 172.21.15.6:0/2349368767 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f17d80af580 0x7f17d81f3120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:16.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:16.606+0000 7f17de800640 1 --2- 172.21.15.6:0/2349368767 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f17d80cdc20 0x7f17d8153ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:16.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:16.606+0000 7f17dde00640 1 -- 172.21.15.6:0/2349368767 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f17d80cdc20 msgr2=0x7f17d8153ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:16.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:16.606+0000 7f17dde00640 1 --2- 172.21.15.6:0/2349368767 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f17d80cdc20 0x7f17d8153ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:16.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:16.606+0000 7f17dde00640 1 -- 172.21.15.6:0/2349368767 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f17d80af580 msgr2=0x7f17d81f3120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:16.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:16.606+0000 7f17dde00640 1 --2- 172.21.15.6:0/2349368767 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f17d80af580 0x7f17d81f3120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:16.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:16.606+0000 7f17dde00640 1 -- 172.21.15.6:0/2349368767 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17d8153550 con 0x7f17d80af1a0 2024-03-04T09:36:16.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:16.607+0000 7f17dde00640 1 --2- 172.21.15.6:0/2349368767 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f17d80af1a0 0x7f17d8158c40 secure :-1 s=READY pgs=1786 cs=0 l=1 rev1=1 crypto rx=0x7f17cc00d800 tx=0x7f17cc00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:36:16.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:16.607+0000 7f17d2000640 1 -- 172.21.15.6:0/2349368767 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17cc00b6c0 con 0x7f17d80af1a0 2024-03-04T09:36:16.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:16.607+0000 7f17d2000640 1 -- 172.21.15.6:0/2349368767 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17cc019070 con 0x7f17d80af1a0 2024-03-04T09:36:16.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:16.607+0000 7f17d2000640 1 -- 172.21.15.6:0/2349368767 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17cc00ba80 con 0x7f17d80af1a0 2024-03-04T09:36:16.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:16.607+0000 7f17e0800640 1 -- 172.21.15.6:0/2349368767 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17d8159490 con 0x7f17d80af1a0 2024-03-04T09:36:16.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:16.607+0000 7f17e0800640 1 -- 172.21.15.6:0/2349368767 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17d8159180 con 0x7f17d80af1a0 2024-03-04T09:36:16.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:16.608+0000 7f17e0800640 1 -- 172.21.15.6:0/2349368767 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17d8069510 con 0x7f17d80af1a0 2024-03-04T09:36:16.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:16.610+0000 7f17d2000640 1 -- 172.21.15.6:0/2349368767 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f17cc00bc20 con 0x7f17d80af1a0 2024-03-04T09:36:16.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:16.611+0000 7f17d2000640 1 --2- 172.21.15.6:0/2349368767 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f17bc06b4d0 0x7f17bc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:16.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:16.611+0000 7f17d2000640 1 -- 172.21.15.6:0/2349368767 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f17cc08a720 con 0x7f17d80af1a0 2024-03-04T09:36:16.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:16.612+0000 7f17dd400640 1 --2- 172.21.15.6:0/2349368767 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f17bc06b4d0 0x7f17bc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:16.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:16.613+0000 7f17dd400640 1 --2- 172.21.15.6:0/2349368767 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f17bc06b4d0 0x7f17bc06d990 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f17c8036000 tx=0x7f17c803c040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:36:16.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:16.615+0000 7f17d2000640 1 -- 172.21.15.6:0/2349368767 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f17cc059560 con 0x7f17d80af1a0 2024-03-04T09:36:16.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:16.861+0000 7f17e0800640 1 -- 172.21.15.6:0/2349368767 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f17d804f3a0 con 0x7f17d80af1a0 2024-03-04T09:36:16.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:16.861+0000 7f17d2000640 1 -- 172.21.15.6:0/2349368767 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3464 (secure 0 0 0) 0x7f17cc058e60 con 0x7f17d80af1a0 2024-03-04T09:36:16.858 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:36:16.858 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:35:26.088222+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"laggy_since":"2024-03-04T09:33:54.652601+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":9}]} 2024-03-04T09:36:16.859 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 101 2024-03-04T09:36:16.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:16.864+0000 7f17e0800640 1 -- 172.21.15.6:0/2349368767 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f17bc06b4d0 msgr2=0x7f17bc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:16.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:16.864+0000 7f17e0800640 1 --2- 172.21.15.6:0/2349368767 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f17bc06b4d0 0x7f17bc06d990 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f17c8036000 tx=0x7f17c803c040 comp rx=0 tx=0).stop 2024-03-04T09:36:16.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:16.864+0000 7f17e0800640 1 -- 172.21.15.6:0/2349368767 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f17d80af1a0 msgr2=0x7f17d8158c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:16.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:16.864+0000 7f17e0800640 1 --2- 172.21.15.6:0/2349368767 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f17d80af1a0 0x7f17d8158c40 secure :-1 s=READY pgs=1786 cs=0 l=1 rev1=1 crypto rx=0x7f17cc00d800 tx=0x7f17cc00dcc0 comp rx=0 tx=0).stop 2024-03-04T09:36:16.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:16.864+0000 7f17e0800640 1 -- 172.21.15.6:0/2349368767 shutdown_connections 2024-03-04T09:36:16.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:16.864+0000 7f17e0800640 1 --2- 172.21.15.6:0/2349368767 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f17bc06b4d0 0x7f17bc06d990 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:16.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:16.864+0000 7f17e0800640 1 --2- 172.21.15.6:0/2349368767 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f17d80cdc20 0x7f17d8153ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:16.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:16.864+0000 7f17e0800640 1 --2- 172.21.15.6:0/2349368767 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f17d80af580 0x7f17d81f3120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:16.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:16.864+0000 7f17e0800640 1 --2- 172.21.15.6:0/2349368767 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f17d80af1a0 0x7f17d8158c40 unknown :-1 s=CLOSED pgs=1786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:16.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:16.864+0000 7f17e0800640 1 -- 172.21.15.6:0/2349368767 >> 172.21.15.6:0/2349368767 conn(0x7f17d806afb0 msgr2=0x7f17d80b2ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:36:16.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:16.865+0000 7f17e0800640 1 -- 172.21.15.6:0/2349368767 shutdown_connections 2024-03-04T09:36:16.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:16.865+0000 7f17e0800640 1 -- 172.21.15.6:0/2349368767 wait complete. 2024-03-04T09:36:17.085 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:32:57.159681+0000', 'modified': '2024-03-04T09:35:26.088222+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8214, 'mds_1': 6796}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6796': {'gid': 6796, 'name': 'd', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.146:6837/2458070840', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 2458070840}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 2458070840}]}, 'join_fscid': -1, 'laggy_since': '2024-03-04T09:33:54.652601+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8214': {'gid': 8214, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2446627517', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2446627517}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2446627517}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8214, 'qdb_cluster': [8214]} 2024-03-04T09:36:17.085 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6796:up:stopping 2024-03-04T09:36:17.939 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-04T09:36:18.086 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:36:18.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:18.309+0000 7f9eb2800640 1 Processor -- start 2024-03-04T09:36:18.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:18.310+0000 7f9eb2800640 1 -- start start 2024-03-04T09:36:18.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:18.311+0000 7f9eb2800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9eac070f10 0x7f9eac0cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:18.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:18.311+0000 7f9eb2800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f9eac0cce90 0x7f9eac15c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:18.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:18.311+0000 7f9eb2800640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9eac0714d0 con 0x7f9eac145300 2024-03-04T09:36:18.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:18.311+0000 7f9eb2800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9eac0cd300 con 0x7f9eac0cce90 2024-03-04T09:36:18.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:18.311+0000 7f9eb2800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9eac0717b0 con 0x7f9eac070f10 2024-03-04T09:36:18.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:18.312+0000 7f9eb1e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f9eac0cce90 0x7f9eac15c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:18.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:18.312+0000 7f9eb1e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f9eac0cce90 0x7f9eac15c570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:54286/0 (socket says 172.21.15.6:54286) 2024-03-04T09:36:18.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:18.312+0000 7f9eb1e00640 1 -- 172.21.15.6:0/3083672256 learned_addr learned my addr 172.21.15.6:0/3083672256 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:36:18.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:18.312+0000 7f9eb0a00640 1 --2- 172.21.15.6:0/3083672256 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9eac070f10 0x7f9eac0cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:18.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:18.312+0000 7f9eb1e00640 1 -- 172.21.15.6:0/3083672256 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9eac070f10 msgr2=0x7f9eac0cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:36:18.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:18.312+0000 7f9eb1e00640 1 --2- 172.21.15.6:0/3083672256 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9eac070f10 0x7f9eac0cc950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:18.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:18.312+0000 7f9eb1e00640 1 -- 172.21.15.6:0/3083672256 >> v1:172.21.15.146:6789/0 conn(0x7f9eac145300 legacy=0x7f9eac0b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:18.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:18.312+0000 7f9eb1e00640 1 -- 172.21.15.6:0/3083672256 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9eac071a90 con 0x7f9eac0cce90 2024-03-04T09:36:18.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:18.314+0000 7f9eb1e00640 1 --2- 172.21.15.6:0/3083672256 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9eac0cce90 0x7f9eac15c570 secure :-1 s=READY pgs=1787 cs=0 l=1 rev1=1 crypto rx=0x7f9ea4009e50 tx=0x7f9ea402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21fc5af7757f568e server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:36:18.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:18.314+0000 7f9eabe00640 1 -- 172.21.15.6:0/3083672256 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ea402cb00 con 0x7f9eac0cce90 2024-03-04T09:36:18.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:18.314+0000 7f9eabe00640 1 -- 172.21.15.6:0/3083672256 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ea402cca0 con 0x7f9eac0cce90 2024-03-04T09:36:18.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:18.314+0000 7f9eabe00640 1 -- 172.21.15.6:0/3083672256 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ea4034610 con 0x7f9eac0cce90 2024-03-04T09:36:18.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:18.315+0000 7f9eb2800640 1 -- 172.21.15.6:0/3083672256 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9eac0cce90 msgr2=0x7f9eac15c570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:18.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:18.315+0000 7f9eb2800640 1 --2- 172.21.15.6:0/3083672256 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9eac0cce90 0x7f9eac15c570 secure :-1 s=READY pgs=1787 cs=0 l=1 rev1=1 crypto rx=0x7f9ea4009e50 tx=0x7f9ea402bf60 comp rx=0 tx=0).stop 2024-03-04T09:36:18.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:18.315+0000 7f9eb2800640 1 -- 172.21.15.6:0/3083672256 shutdown_connections 2024-03-04T09:36:18.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:18.315+0000 7f9eb2800640 1 --2- 172.21.15.6:0/3083672256 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9eac0cce90 0x7f9eac15c570 unknown :-1 s=CLOSED pgs=1787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:18.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:18.315+0000 7f9eb2800640 1 --2- 172.21.15.6:0/3083672256 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9eac070f10 0x7f9eac0cc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:18.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:18.315+0000 7f9eb2800640 1 -- 172.21.15.6:0/3083672256 >> 172.21.15.6:0/3083672256 conn(0x7f9eac06b3d0 msgr2=0x7f9eac06b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:36:18.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:18.315+0000 7f9eb2800640 1 -- 172.21.15.6:0/3083672256 shutdown_connections 2024-03-04T09:36:18.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:18.315+0000 7f9eb2800640 1 -- 172.21.15.6:0/3083672256 wait complete. 2024-03-04T09:36:18.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:18.317+0000 7f9eb2800640 1 Processor -- start 2024-03-04T09:36:18.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:18.317+0000 7f9eb2800640 1 -- start start 2024-03-04T09:36:18.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:18.318+0000 7f9eb2800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9eac070f10 0x7f9eac150d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:18.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:18.318+0000 7f9eb2800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9eac0cce90 0x7f9eac0ad880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:18.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:18.318+0000 7f9eb2800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9eac145300 0x7f9eac141690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:18.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:18.318+0000 7f9eb2800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9eac15cab0 con 0x7f9eac0cce90 2024-03-04T09:36:18.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:18.318+0000 7f9eb2800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9eac1456e0 con 0x7f9eac070f10 2024-03-04T09:36:18.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:18.318+0000 7f9eb2800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9eac15cc30 con 0x7f9eac145300 2024-03-04T09:36:18.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:18.318+0000 7f9eb0a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9eac0cce90 0x7f9eac0ad880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:18.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:18.318+0000 7f9eb0a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9eac0cce90 0x7f9eac0ad880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:54298/0 (socket says 172.21.15.6:54298) 2024-03-04T09:36:18.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:18.318+0000 7f9eb0a00640 1 -- 172.21.15.6:0/4257841092 learned_addr learned my addr 172.21.15.6:0/4257841092 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:36:18.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:18.319+0000 7f9eb1e00640 1 --2- 172.21.15.6:0/4257841092 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9eac145300 0x7f9eac141690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:18.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:18.319+0000 7f9eb1400640 1 --2- 172.21.15.6:0/4257841092 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9eac070f10 0x7f9eac150d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:18.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:18.319+0000 7f9eb0a00640 1 -- 172.21.15.6:0/4257841092 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9eac145300 msgr2=0x7f9eac141690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:18.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:18.319+0000 7f9eb0a00640 1 --2- 172.21.15.6:0/4257841092 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9eac145300 0x7f9eac141690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:18.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:18.319+0000 7f9eb0a00640 1 -- 172.21.15.6:0/4257841092 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9eac070f10 msgr2=0x7f9eac150d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:18.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:18.319+0000 7f9eb0a00640 1 --2- 172.21.15.6:0/4257841092 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9eac070f10 0x7f9eac150d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:18.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:18.319+0000 7f9eb0a00640 1 -- 172.21.15.6:0/4257841092 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9eac0ae180 con 0x7f9eac0cce90 2024-03-04T09:36:18.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:18.319+0000 7f9eb0a00640 1 --2- 172.21.15.6:0/4257841092 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9eac0cce90 0x7f9eac0ad880 secure :-1 s=READY pgs=1788 cs=0 l=1 rev1=1 crypto rx=0x7f9e9800d830 tx=0x7f9e9800dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:36:18.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:18.319+0000 7f9ea9600640 1 -- 172.21.15.6:0/4257841092 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e9800b6c0 con 0x7f9eac0cce90 2024-03-04T09:36:18.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:18.319+0000 7f9ea9600640 1 -- 172.21.15.6:0/4257841092 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e98019070 con 0x7f9eac0cce90 2024-03-04T09:36:18.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:18.319+0000 7f9ea9600640 1 -- 172.21.15.6:0/4257841092 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e9800bac0 con 0x7f9eac0cce90 2024-03-04T09:36:18.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:18.319+0000 7f9eb2800640 1 -- 172.21.15.6:0/4257841092 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9eac0ae460 con 0x7f9eac0cce90 2024-03-04T09:36:18.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:18.320+0000 7f9eb2800640 1 -- 172.21.15.6:0/4257841092 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9eac1512a0 con 0x7f9eac0cce90 2024-03-04T09:36:18.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:18.321+0000 7f9eb2800640 1 -- 172.21.15.6:0/4257841092 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9eac04f330 con 0x7f9eac0cce90 2024-03-04T09:36:18.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:18.322+0000 7f9ea9600640 1 -- 172.21.15.6:0/4257841092 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9e9801e460 con 0x7f9eac0cce90 2024-03-04T09:36:18.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:18.323+0000 7f9ea9600640 1 --2- 172.21.15.6:0/4257841092 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f9e8c06b4d0 0x7f9e8c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:18.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:18.323+0000 7f9ea9600640 1 -- 172.21.15.6:0/4257841092 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f9e9808a9d0 con 0x7f9eac0cce90 2024-03-04T09:36:18.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:18.323+0000 7f9eb1400640 1 --2- 172.21.15.6:0/4257841092 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f9e8c06b4d0 0x7f9e8c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:18.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:18.325+0000 7f9eb1400640 1 --2- 172.21.15.6:0/4257841092 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f9e8c06b4d0 0x7f9e8c06d990 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f9e9c001500 tx=0x7f9e9c009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:36:18.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:18.327+0000 7f9ea9600640 1 -- 172.21.15.6:0/4257841092 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9e98059770 con 0x7f9eac0cce90 2024-03-04T09:36:18.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:18.568+0000 7f9eb2800640 1 -- 172.21.15.6:0/4257841092 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9eac074ab0 con 0x7f9eac0cce90 2024-03-04T09:36:18.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:18.569+0000 7f9ea9600640 1 -- 172.21.15.6:0/4257841092 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3464 (secure 0 0 0) 0x7f9e98059070 con 0x7f9eac0cce90 2024-03-04T09:36:18.566 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:36:18.566 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:35:26.088222+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"laggy_since":"2024-03-04T09:33:54.652601+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":9}]} 2024-03-04T09:36:18.567 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 101 2024-03-04T09:36:18.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:18.572+0000 7f9eb2800640 1 -- 172.21.15.6:0/4257841092 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f9e8c06b4d0 msgr2=0x7f9e8c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:18.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:18.572+0000 7f9eb2800640 1 --2- 172.21.15.6:0/4257841092 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f9e8c06b4d0 0x7f9e8c06d990 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f9e9c001500 tx=0x7f9e9c009040 comp rx=0 tx=0).stop 2024-03-04T09:36:18.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:18.572+0000 7f9eb2800640 1 -- 172.21.15.6:0/4257841092 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9eac0cce90 msgr2=0x7f9eac0ad880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:18.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:18.572+0000 7f9eb2800640 1 --2- 172.21.15.6:0/4257841092 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9eac0cce90 0x7f9eac0ad880 secure :-1 s=READY pgs=1788 cs=0 l=1 rev1=1 crypto rx=0x7f9e9800d830 tx=0x7f9e9800dcf0 comp rx=0 tx=0).stop 2024-03-04T09:36:18.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:18.572+0000 7f9eb2800640 1 -- 172.21.15.6:0/4257841092 shutdown_connections 2024-03-04T09:36:18.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:18.572+0000 7f9eb2800640 1 --2- 172.21.15.6:0/4257841092 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f9e8c06b4d0 0x7f9e8c06d990 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:18.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:18.572+0000 7f9eb2800640 1 --2- 172.21.15.6:0/4257841092 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9eac145300 0x7f9eac141690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:18.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:18.572+0000 7f9eb2800640 1 --2- 172.21.15.6:0/4257841092 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9eac0cce90 0x7f9eac0ad880 unknown :-1 s=CLOSED pgs=1788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:18.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:18.572+0000 7f9eb2800640 1 --2- 172.21.15.6:0/4257841092 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9eac070f10 0x7f9eac150d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:18.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:18.572+0000 7f9eb2800640 1 -- 172.21.15.6:0/4257841092 >> 172.21.15.6:0/4257841092 conn(0x7f9eac06b3d0 msgr2=0x7f9eac0724b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:36:18.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:18.572+0000 7f9eb2800640 1 -- 172.21.15.6:0/4257841092 shutdown_connections 2024-03-04T09:36:18.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:18.572+0000 7f9eb2800640 1 -- 172.21.15.6:0/4257841092 wait complete. 2024-03-04T09:36:18.787 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:32:57.159681+0000', 'modified': '2024-03-04T09:35:26.088222+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8214, 'mds_1': 6796}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6796': {'gid': 6796, 'name': 'd', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.146:6837/2458070840', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 2458070840}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 2458070840}]}, 'join_fscid': -1, 'laggy_since': '2024-03-04T09:33:54.652601+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8214': {'gid': 8214, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2446627517', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2446627517}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2446627517}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8214, 'qdb_cluster': [8214]} 2024-03-04T09:36:18.788 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6796:up:stopping 2024-03-04T09:36:19.789 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:36:19.941 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-04T09:36:20.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:20.015+0000 7fa1fcc00640 1 Processor -- start 2024-03-04T09:36:20.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:20.015+0000 7fa1fcc00640 1 -- start start 2024-03-04T09:36:20.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:20.017+0000 7fa1fcc00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa1f815e2f0 0x7fa1f806afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:20.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:20.017+0000 7fa1fcc00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1f80b5290 con 0x7fa1f806b4f0 2024-03-04T09:36:20.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:20.017+0000 7fa1fcc00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1f80b6010 con 0x7fa1f8070460 2024-03-04T09:36:20.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:20.017+0000 7fa1fcc00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1f815e6d0 con 0x7fa1f815e2f0 2024-03-04T09:36:20.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:20.018+0000 7fa1f5e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa1f815e2f0 0x7fa1f806afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:20.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:20.018+0000 7fa1ed400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fa1f806b4f0 0x7fa1f806fd50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:47328/0 (socket says 172.21.15.6:47328) 2024-03-04T09:36:20.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:20.018+0000 7fa1ed400640 1 -- 172.21.15.6:0/963280017 learned_addr learned my addr 172.21.15.6:0/963280017 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:36:20.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:20.018+0000 7fa1f5e00640 1 -- 172.21.15.6:0/963280017 >> v1:172.21.15.146:6789/0 conn(0x7fa1f8070460 legacy=0x7fa1f806db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:36:20.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:20.018+0000 7fa1f5e00640 1 -- 172.21.15.6:0/963280017 >> v1:172.21.15.6:6789/0 conn(0x7fa1f806b4f0 legacy=0x7fa1f806fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:20.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:20.018+0000 7fa1f5e00640 1 -- 172.21.15.6:0/963280017 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1f806b9c0 con 0x7fa1f815e2f0 2024-03-04T09:36:20.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:20.018+0000 7fa1f5400640 1 -- 172.21.15.6:0/963280017 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3939017781 0 0) 0x7fa1f80b5290 con 0x7fa1f806b4f0 2024-03-04T09:36:20.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:20.020+0000 7fa1f5e00640 1 --2- 172.21.15.6:0/963280017 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa1f815e2f0 0x7fa1f806afb0 secure :-1 s=READY pgs=1790 cs=0 l=1 rev1=1 crypto rx=0x7fa1e80097a0 tx=0x7fa1e802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86e78ec9fd02fa50 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:36:20.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:20.021+0000 7fa1f5400640 1 -- 172.21.15.6:0/963280017 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1e8035020 con 0x7fa1f815e2f0 2024-03-04T09:36:20.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:20.021+0000 7fa1f5400640 1 -- 172.21.15.6:0/963280017 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1e8031830 con 0x7fa1f815e2f0 2024-03-04T09:36:20.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:20.021+0000 7fa1f5400640 1 -- 172.21.15.6:0/963280017 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1e8031c00 con 0x7fa1f815e2f0 2024-03-04T09:36:20.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:20.021+0000 7fa1fcc00640 1 -- 172.21.15.6:0/963280017 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa1f815e2f0 msgr2=0x7fa1f806afb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:20.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:20.021+0000 7fa1fcc00640 1 --2- 172.21.15.6:0/963280017 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa1f815e2f0 0x7fa1f806afb0 secure :-1 s=READY pgs=1790 cs=0 l=1 rev1=1 crypto rx=0x7fa1e80097a0 tx=0x7fa1e802b8d0 comp rx=0 tx=0).stop 2024-03-04T09:36:20.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:20.021+0000 7fa1fcc00640 1 -- 172.21.15.6:0/963280017 shutdown_connections 2024-03-04T09:36:20.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:20.021+0000 7fa1fcc00640 1 --2- 172.21.15.6:0/963280017 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa1f815e2f0 0x7fa1f806afb0 unknown :-1 s=CLOSED pgs=1790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:20.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:20.021+0000 7fa1fcc00640 1 -- 172.21.15.6:0/963280017 >> 172.21.15.6:0/963280017 conn(0x7fa1f80d0d80 msgr2=0x7fa1f80d1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:36:20.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:20.021+0000 7fa1fcc00640 1 -- 172.21.15.6:0/963280017 shutdown_connections 2024-03-04T09:36:20.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:20.021+0000 7fa1fcc00640 1 -- 172.21.15.6:0/963280017 wait complete. 2024-03-04T09:36:20.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:20.024+0000 7fa1fcc00640 1 Processor -- start 2024-03-04T09:36:20.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:20.024+0000 7fa1fcc00640 1 -- start start 2024-03-04T09:36:20.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:20.025+0000 7fa1fcc00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa1f806b4f0 0x7fa1f814a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:20.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:20.025+0000 7fa1fcc00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa1f8070460 0x7fa1f815b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:20.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:20.025+0000 7fa1fcc00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa1f815e2f0 0x7fa1f815bd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:20.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:20.026+0000 7fa1fcc00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1f80b5af0 con 0x7fa1f806b4f0 2024-03-04T09:36:20.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:20.026+0000 7fa1fcc00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1f80b5810 con 0x7fa1f815e2f0 2024-03-04T09:36:20.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:20.026+0000 7fa1fcc00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1f8072650 con 0x7fa1f8070460 2024-03-04T09:36:20.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:20.026+0000 7fa1f5e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa1f806b4f0 0x7fa1f814a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:20.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:20.026+0000 7fa1f5e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa1f806b4f0 0x7fa1f814a7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36022/0 (socket says 172.21.15.6:36022) 2024-03-04T09:36:20.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:20.026+0000 7fa1f5e00640 1 -- 172.21.15.6:0/2261877673 learned_addr learned my addr 172.21.15.6:0/2261877673 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:36:20.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:20.026+0000 7fa1f5e00640 1 -- 172.21.15.6:0/2261877673 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa1f8070460 msgr2=0x7fa1f815b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:20.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:20.026+0000 7fa1f5e00640 1 --2- 172.21.15.6:0/2261877673 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa1f8070460 0x7fa1f815b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:20.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:20.026+0000 7fa1f5e00640 1 -- 172.21.15.6:0/2261877673 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa1f815e2f0 msgr2=0x7fa1f815bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:20.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:20.026+0000 7fa1f6800640 1 --2- 172.21.15.6:0/2261877673 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa1f815e2f0 0x7fa1f815bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:20.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:20.026+0000 7fa1f5e00640 1 --2- 172.21.15.6:0/2261877673 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa1f815e2f0 0x7fa1f815bd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:20.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:20.026+0000 7fa1f5e00640 1 -- 172.21.15.6:0/2261877673 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1f815f7d0 con 0x7fa1f806b4f0 2024-03-04T09:36:20.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:20.027+0000 7fa1f5e00640 1 --2- 172.21.15.6:0/2261877673 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa1f806b4f0 0x7fa1f814a7a0 secure :-1 s=READY pgs=1791 cs=0 l=1 rev1=1 crypto rx=0x7fa1e80098c0 tx=0x7fa1e80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:36:20.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:20.027+0000 7fa1eea00640 1 -- 172.21.15.6:0/2261877673 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1e8004230 con 0x7fa1f806b4f0 2024-03-04T09:36:20.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:20.027+0000 7fa1eea00640 1 -- 172.21.15.6:0/2261877673 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1e80043d0 con 0x7fa1f806b4f0 2024-03-04T09:36:20.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:20.027+0000 7fa1fcc00640 1 -- 172.21.15.6:0/2261877673 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1f815fab0 con 0x7fa1f806b4f0 2024-03-04T09:36:20.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:20.027+0000 7fa1eea00640 1 -- 172.21.15.6:0/2261877673 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1e803cad0 con 0x7fa1f806b4f0 2024-03-04T09:36:20.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:20.027+0000 7fa1fcc00640 1 -- 172.21.15.6:0/2261877673 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1f8160070 con 0x7fa1f806b4f0 2024-03-04T09:36:20.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:20.029+0000 7fa1fcc00640 1 -- 172.21.15.6:0/2261877673 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1f806afb0 con 0x7fa1f806b4f0 2024-03-04T09:36:20.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:20.029+0000 7fa1eea00640 1 -- 172.21.15.6:0/2261877673 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa1e8038020 con 0x7fa1f806b4f0 2024-03-04T09:36:20.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:20.030+0000 7fa1eea00640 1 --2- 172.21.15.6:0/2261877673 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa1d406b520 0x7fa1d406d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:20.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:20.030+0000 7fa1eea00640 1 -- 172.21.15.6:0/2261877673 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fa1e80ac370 con 0x7fa1f806b4f0 2024-03-04T09:36:20.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:20.031+0000 7fa1ed400640 1 --2- 172.21.15.6:0/2261877673 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa1d406b520 0x7fa1d406d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:20.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:20.032+0000 7fa1ed400640 1 --2- 172.21.15.6:0/2261877673 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa1d406b520 0x7fa1d406d9e0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7fa1e0007fd0 tx=0x7fa1e0007960 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:36:20.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:20.035+0000 7fa1eea00640 1 -- 172.21.15.6:0/2261877673 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa1f806afb0 con 0x7fa1f806b4f0 2024-03-04T09:36:20.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:20.272+0000 7fa1fcc00640 1 -- 172.21.15.6:0/2261877673 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa1f804f3a0 con 0x7fa1f806b4f0 2024-03-04T09:36:20.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:20.274+0000 7fa1eea00640 1 -- 172.21.15.6:0/2261877673 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3464 (secure 0 0 0) 0x7fa1e8041070 con 0x7fa1f806b4f0 2024-03-04T09:36:20.271 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:36:20.271 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:35:26.088222+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"laggy_since":"2024-03-04T09:33:54.652601+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":9}]} 2024-03-04T09:36:20.271 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 101 2024-03-04T09:36:20.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:20.276+0000 7fa1fcc00640 1 -- 172.21.15.6:0/2261877673 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa1d406b520 msgr2=0x7fa1d406d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:20.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:20.276+0000 7fa1fcc00640 1 --2- 172.21.15.6:0/2261877673 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa1d406b520 0x7fa1d406d9e0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7fa1e0007fd0 tx=0x7fa1e0007960 comp rx=0 tx=0).stop 2024-03-04T09:36:20.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:20.276+0000 7fa1fcc00640 1 -- 172.21.15.6:0/2261877673 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa1f806b4f0 msgr2=0x7fa1f814a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:20.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:20.276+0000 7fa1fcc00640 1 --2- 172.21.15.6:0/2261877673 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa1f806b4f0 0x7fa1f814a7a0 secure :-1 s=READY pgs=1791 cs=0 l=1 rev1=1 crypto rx=0x7fa1e80098c0 tx=0x7fa1e80046c0 comp rx=0 tx=0).stop 2024-03-04T09:36:20.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:20.276+0000 7fa1fcc00640 1 -- 172.21.15.6:0/2261877673 shutdown_connections 2024-03-04T09:36:20.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:20.276+0000 7fa1fcc00640 1 --2- 172.21.15.6:0/2261877673 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa1f815e2f0 0x7fa1f815bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:20.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:20.276+0000 7fa1fcc00640 1 --2- 172.21.15.6:0/2261877673 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa1d406b520 0x7fa1d406d9e0 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:20.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:20.276+0000 7fa1fcc00640 1 --2- 172.21.15.6:0/2261877673 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa1f8070460 0x7fa1f815b850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:20.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:20.276+0000 7fa1fcc00640 1 --2- 172.21.15.6:0/2261877673 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa1f806b4f0 0x7fa1f814a7a0 unknown :-1 s=CLOSED pgs=1791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:20.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:20.276+0000 7fa1fcc00640 1 -- 172.21.15.6:0/2261877673 >> 172.21.15.6:0/2261877673 conn(0x7fa1f80d0d80 msgr2=0x7fa1f80afd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:36:20.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:20.276+0000 7fa1fcc00640 1 -- 172.21.15.6:0/2261877673 shutdown_connections 2024-03-04T09:36:20.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:20.276+0000 7fa1fcc00640 1 -- 172.21.15.6:0/2261877673 wait complete. 2024-03-04T09:36:20.492 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:32:57.159681+0000', 'modified': '2024-03-04T09:35:26.088222+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8214, 'mds_1': 6796}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6796': {'gid': 6796, 'name': 'd', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.146:6837/2458070840', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 2458070840}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 2458070840}]}, 'join_fscid': -1, 'laggy_since': '2024-03-04T09:33:54.652601+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8214': {'gid': 8214, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2446627517', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2446627517}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2446627517}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8214, 'qdb_cluster': [8214]} 2024-03-04T09:36:20.493 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6796:up:stopping 2024-03-04T09:36:21.493 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:36:21.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:21.715+0000 7f486e200640 1 Processor -- start 2024-03-04T09:36:21.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:21.715+0000 7f486e200640 1 -- start start 2024-03-04T09:36:21.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:21.717+0000 7f486e200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f48680b34e0 0x7f486815c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:21.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:21.717+0000 7f486e200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f486815cc40 con 0x7f48680af1a0 2024-03-04T09:36:21.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:21.717+0000 7f486e200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f486815cf20 con 0x7f48680af580 2024-03-04T09:36:21.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:21.717+0000 7f486e200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f486815d200 con 0x7f48680b34e0 2024-03-04T09:36:21.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:21.718+0000 7f4867400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f48680af1a0 0x7f48680b4c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:47330/0 (socket says 172.21.15.6:47330) 2024-03-04T09:36:21.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:21.718+0000 7f4867400640 1 -- 172.21.15.6:0/3314464421 learned_addr learned my addr 172.21.15.6:0/3314464421 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:36:21.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:21.718+0000 7f4867e00640 1 --2- 172.21.15.6:0/3314464421 >> v2:172.21.15.146:3300/0 conn(0x7f48680b34e0 0x7f486815c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:21.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:21.718+0000 7f4866a00640 1 -- 172.21.15.6:0/3314464421 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1556810189 0 0) 0x7f486815cc40 con 0x7f48680af1a0 2024-03-04T09:36:21.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:21.718+0000 7f4866a00640 1 -- 172.21.15.6:0/3314464421 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f484c003520 con 0x7f48680af1a0 2024-03-04T09:36:21.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:21.718+0000 7f4866a00640 1 -- 172.21.15.6:0/3314464421 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1565629457 0 0) 0x7f484c003520 con 0x7f48680af1a0 2024-03-04T09:36:21.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:21.719+0000 7f4866a00640 1 -- 172.21.15.6:0/3314464421 >> v2:172.21.15.146:3300/0 conn(0x7f48680b34e0 msgr2=0x7f486815c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:36:21.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:21.719+0000 7f4866a00640 1 --2- 172.21.15.6:0/3314464421 >> v2:172.21.15.146:3300/0 conn(0x7f48680b34e0 0x7f486815c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:21.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:21.719+0000 7f4866a00640 1 -- 172.21.15.6:0/3314464421 >> v1:172.21.15.146:6789/0 conn(0x7f48680af580 legacy=0x7f48680b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:21.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:21.719+0000 7f4866a00640 1 -- 172.21.15.6:0/3314464421 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f486815d4e0 con 0x7f48680af1a0 2024-03-04T09:36:21.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:21.719+0000 7f4866a00640 1 -- 172.21.15.6:0/3314464421 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7f4858001670 con 0x7f48680af1a0 2024-03-04T09:36:21.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:21.719+0000 7f4866a00640 1 -- 172.21.15.6:0/3314464421 >> v1:172.21.15.6:6789/0 conn(0x7f48680af1a0 legacy=0x7f48680b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:21.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:21.719+0000 7f4866a00640 1 --2- 172.21.15.6:0/3314464421 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f484c003a70 0x7f484c003e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:21.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:21.719+0000 7f4866a00640 1 --2- 172.21.15.6:0/3314464421 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f484c004800 0x7f484c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:21.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:21.719+0000 7f4866a00640 1 --2- 172.21.15.6:0/3314464421 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f484c0052e0 0x7f484c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:21.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:21.719+0000 7f4866a00640 1 -- 172.21.15.6:0/3314464421 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f486815d200 con 0x7f484c004800 2024-03-04T09:36:21.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:21.719+0000 7f4866a00640 1 -- 172.21.15.6:0/3314464421 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f484c00c1b0 con 0x7f484c0052e0 2024-03-04T09:36:21.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:21.719+0000 7f4866a00640 1 -- 172.21.15.6:0/3314464421 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f484c00c3b0 con 0x7f484c003a70 2024-03-04T09:36:21.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:21.719+0000 7f485fe00640 1 --2- 172.21.15.6:0/3314464421 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f484c004800 0x7f484c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:21.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:21.720+0000 7f4867400640 1 --2- 172.21.15.6:0/3314464421 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f484c003a70 0x7f484c003e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:21.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:21.720+0000 7f4867e00640 1 --2- 172.21.15.6:0/3314464421 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f484c0052e0 0x7f484c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:21.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:21.720+0000 7f485fe00640 1 -- 172.21.15.6:0/3314464421 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f484c003a70 msgr2=0x7f484c003e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:36:21.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:21.720+0000 7f485fe00640 1 --2- 172.21.15.6:0/3314464421 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f484c003a70 0x7f484c003e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:21.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:21.720+0000 7f485fe00640 1 -- 172.21.15.6:0/3314464421 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f484c0052e0 msgr2=0x7f484c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:36:21.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:21.720+0000 7f485fe00640 1 --2- 172.21.15.6:0/3314464421 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f484c0052e0 0x7f484c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:21.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:21.720+0000 7f485fe00640 1 -- 172.21.15.6:0/3314464421 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f484c003520 con 0x7f484c004800 2024-03-04T09:36:21.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:21.720+0000 7f4867400640 1 --2- 172.21.15.6:0/3314464421 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f484c003a70 0x7f484c003e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:36:21.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:21.720+0000 7f4867e00640 1 --2- 172.21.15.6:0/3314464421 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f484c0052e0 0x7f484c00b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:36:21.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:21.720+0000 7f4867e00640 1 -- 172.21.15.6:0/3314464421 reap_dead start 2024-03-04T09:36:21.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:21.721+0000 7f485fe00640 1 --2- 172.21.15.6:0/3314464421 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f484c004800 0x7f484c004c80 secure :-1 s=READY pgs=1793 cs=0 l=1 rev1=1 crypto rx=0x7f4850001720 tx=0x7f484c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f616027465a221bc server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:36:21.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:21.721+0000 7f4866a00640 1 -- 172.21.15.6:0/3314464421 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4850001370 con 0x7f484c004800 2024-03-04T09:36:21.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:21.721+0000 7f4866a00640 1 -- 172.21.15.6:0/3314464421 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48500052d0 con 0x7f484c004800 2024-03-04T09:36:21.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:21.721+0000 7f4866a00640 1 -- 172.21.15.6:0/3314464421 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4850003830 con 0x7f484c004800 2024-03-04T09:36:21.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:21.722+0000 7f486e200640 1 -- 172.21.15.6:0/3314464421 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f484c004800 msgr2=0x7f484c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:21.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:21.722+0000 7f486e200640 1 --2- 172.21.15.6:0/3314464421 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f484c004800 0x7f484c004c80 secure :-1 s=READY pgs=1793 cs=0 l=1 rev1=1 crypto rx=0x7f4850001720 tx=0x7f484c005830 comp rx=0 tx=0).stop 2024-03-04T09:36:21.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:21.722+0000 7f486e200640 1 -- 172.21.15.6:0/3314464421 shutdown_connections 2024-03-04T09:36:21.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:21.722+0000 7f486e200640 1 --2- 172.21.15.6:0/3314464421 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f484c004800 0x7f484c004c80 unknown :-1 s=CLOSED pgs=1793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:21.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:21.722+0000 7f486e200640 1 -- 172.21.15.6:0/3314464421 >> 172.21.15.6:0/3314464421 conn(0x7f486806afb0 msgr2=0x7f486806b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:36:21.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:21.722+0000 7f486e200640 1 -- 172.21.15.6:0/3314464421 shutdown_connections 2024-03-04T09:36:21.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:21.722+0000 7f486e200640 1 -- 172.21.15.6:0/3314464421 wait complete. 2024-03-04T09:36:21.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:21.724+0000 7f486e200640 1 Processor -- start 2024-03-04T09:36:21.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:21.724+0000 7f486e200640 1 -- start start 2024-03-04T09:36:21.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:21.726+0000 7f486e200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f484c004800 0x7f4868153830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:21.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:21.726+0000 7f486e200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4868156c30 0x7f48680c9fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:21.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:21.726+0000 7f486e200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f48680c9150 0x7f48680ca4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:21.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:21.726+0000 7f486e200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f486815c600 con 0x7f4868156c30 2024-03-04T09:36:21.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:21.726+0000 7f486e200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48680cde30 con 0x7f48680c9150 2024-03-04T09:36:21.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:21.726+0000 7f486e200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f486815c780 con 0x7f484c004800 2024-03-04T09:36:21.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:21.726+0000 7f485fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4868156c30 0x7f48680c9fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:21.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:21.726+0000 7f485fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4868156c30 0x7f48680c9fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36046/0 (socket says 172.21.15.6:36046) 2024-03-04T09:36:21.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:21.726+0000 7f485fe00640 1 -- 172.21.15.6:0/1431116157 learned_addr learned my addr 172.21.15.6:0/1431116157 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:36:21.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:21.726+0000 7f4867400640 1 --2- 172.21.15.6:0/1431116157 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f484c004800 0x7f4868153830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:21.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:21.726+0000 7f4867e00640 1 --2- 172.21.15.6:0/1431116157 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f48680c9150 0x7f48680ca4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:21.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:21.727+0000 7f485fe00640 1 -- 172.21.15.6:0/1431116157 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f484c004800 msgr2=0x7f4868153830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:21.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:21.727+0000 7f485fe00640 1 --2- 172.21.15.6:0/1431116157 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f484c004800 0x7f4868153830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:21.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:21.727+0000 7f485fe00640 1 -- 172.21.15.6:0/1431116157 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f48680c9150 msgr2=0x7f48680ca4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:21.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:21.727+0000 7f485fe00640 1 --2- 172.21.15.6:0/1431116157 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f48680c9150 0x7f48680ca4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:21.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:21.727+0000 7f485fe00640 1 -- 172.21.15.6:0/1431116157 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f486806df40 con 0x7f4868156c30 2024-03-04T09:36:21.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:21.727+0000 7f485fe00640 1 --2- 172.21.15.6:0/1431116157 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4868156c30 0x7f48680c9fb0 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 crypto rx=0x7f485002bc10 tx=0x7f4850001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:36:21.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:21.727+0000 7f485f400640 1 -- 172.21.15.6:0/1431116157 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4850041070 con 0x7f4868156c30 2024-03-04T09:36:21.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:21.727+0000 7f485f400640 1 -- 172.21.15.6:0/1431116157 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4850005480 con 0x7f4868156c30 2024-03-04T09:36:21.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:21.727+0000 7f485f400640 1 -- 172.21.15.6:0/1431116157 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4850007a30 con 0x7f4868156c30 2024-03-04T09:36:21.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:21.727+0000 7f486e200640 1 -- 172.21.15.6:0/1431116157 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f486806e220 con 0x7f4868156c30 2024-03-04T09:36:21.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:21.728+0000 7f486e200640 1 -- 172.21.15.6:0/1431116157 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f486806e7e0 con 0x7f4868156c30 2024-03-04T09:36:21.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:21.729+0000 7f486e200640 1 -- 172.21.15.6:0/1431116157 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4868069510 con 0x7f4868156c30 2024-03-04T09:36:21.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:21.730+0000 7f485f400640 1 -- 172.21.15.6:0/1431116157 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4850044070 con 0x7f4868156c30 2024-03-04T09:36:21.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:21.731+0000 7f485f400640 1 --2- 172.21.15.6:0/1431116157 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f484406b4d0 0x7f484406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:21.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:21.731+0000 7f485f400640 1 -- 172.21.15.6:0/1431116157 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f48500abf20 con 0x7f4868156c30 2024-03-04T09:36:21.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:21.731+0000 7f4867400640 1 --2- 172.21.15.6:0/1431116157 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f484406b4d0 0x7f484406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:21.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:21.732+0000 7f4867400640 1 --2- 172.21.15.6:0/1431116157 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f484406b4d0 0x7f484406d990 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f4858000c00 tx=0x7f4858000ed0 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:36:21.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:21.735+0000 7f485f400640 1 -- 172.21.15.6:0/1431116157 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f485007ac30 con 0x7f4868156c30 2024-03-04T09:36:21.943 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-04T09:36:21.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:21.972+0000 7f486e200640 1 -- 172.21.15.6:0/1431116157 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f48680b3000 con 0x7f4868156c30 2024-03-04T09:36:21.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:21.973+0000 7f485f400640 1 -- 172.21.15.6:0/1431116157 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3464 (secure 0 0 0) 0x7f485007a530 con 0x7f4868156c30 2024-03-04T09:36:21.970 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:36:21.970 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:35:26.088222+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"laggy_since":"2024-03-04T09:33:54.652601+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":9}]} 2024-03-04T09:36:21.971 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 101 2024-03-04T09:36:21.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:21.976+0000 7f486e200640 1 -- 172.21.15.6:0/1431116157 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f484406b4d0 msgr2=0x7f484406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:21.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:21.976+0000 7f486e200640 1 --2- 172.21.15.6:0/1431116157 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f484406b4d0 0x7f484406d990 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f4858000c00 tx=0x7f4858000ed0 comp rx=0 tx=0).stop 2024-03-04T09:36:21.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:21.976+0000 7f486e200640 1 -- 172.21.15.6:0/1431116157 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4868156c30 msgr2=0x7f48680c9fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:21.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:21.976+0000 7f486e200640 1 --2- 172.21.15.6:0/1431116157 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4868156c30 0x7f48680c9fb0 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 crypto rx=0x7f485002bc10 tx=0x7f4850001120 comp rx=0 tx=0).stop 2024-03-04T09:36:21.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:21.976+0000 7f486e200640 1 -- 172.21.15.6:0/1431116157 shutdown_connections 2024-03-04T09:36:21.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:21.976+0000 7f486e200640 1 --2- 172.21.15.6:0/1431116157 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f48680c9150 0x7f48680ca4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:21.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:21.976+0000 7f486e200640 1 --2- 172.21.15.6:0/1431116157 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4868156c30 0x7f48680c9fb0 unknown :-1 s=CLOSED pgs=1794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:21.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:21.976+0000 7f486e200640 1 --2- 172.21.15.6:0/1431116157 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f484406b4d0 0x7f484406d990 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:21.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:21.976+0000 7f486e200640 1 --2- 172.21.15.6:0/1431116157 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f484c004800 0x7f4868153830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:21.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:21.976+0000 7f486e200640 1 -- 172.21.15.6:0/1431116157 >> 172.21.15.6:0/1431116157 conn(0x7f486806afb0 msgr2=0x7f486815d8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:36:21.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:21.976+0000 7f486e200640 1 -- 172.21.15.6:0/1431116157 shutdown_connections 2024-03-04T09:36:21.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:21.976+0000 7f486e200640 1 -- 172.21.15.6:0/1431116157 wait complete. 2024-03-04T09:36:22.194 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:32:57.159681+0000', 'modified': '2024-03-04T09:35:26.088222+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8214, 'mds_1': 6796}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6796': {'gid': 6796, 'name': 'd', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.146:6837/2458070840', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 2458070840}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 2458070840}]}, 'join_fscid': -1, 'laggy_since': '2024-03-04T09:33:54.652601+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8214': {'gid': 8214, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2446627517', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2446627517}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2446627517}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8214, 'qdb_cluster': [8214]} 2024-03-04T09:36:22.194 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6796:up:stopping 2024-03-04T09:36:23.195 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:36:23.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:23.417+0000 7f8609e00640 1 Processor -- start 2024-03-04T09:36:23.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:23.417+0000 7f8609e00640 1 -- start start 2024-03-04T09:36:23.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:23.419+0000 7f8609e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f86040b0450 0x7f86040b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:23.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:23.419+0000 7f8609e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86040b5d60 con 0x7f8604159a50 2024-03-04T09:36:23.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:23.419+0000 7f8609e00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86040b6040 con 0x7f86040b52a0 2024-03-04T09:36:23.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:23.419+0000 7f8609e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86040af540 con 0x7f86040b0450 2024-03-04T09:36:23.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:23.419+0000 7f8609400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f86040b0450 0x7f86040b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:23.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:23.419+0000 7f8608a00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f8604159a50 0x7f860406ecc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:47346/0 (socket says 172.21.15.6:47346) 2024-03-04T09:36:23.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:23.419+0000 7f8608a00640 1 -- 172.21.15.6:0/2008776392 learned_addr learned my addr 172.21.15.6:0/2008776392 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:36:23.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:23.420+0000 7f8609400640 1 -- 172.21.15.6:0/2008776392 >> v1:172.21.15.146:6789/0 conn(0x7f86040b52a0 legacy=0x7f86040afd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:23.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:23.420+0000 7f8609400640 1 -- 172.21.15.6:0/2008776392 >> v1:172.21.15.6:6789/0 conn(0x7f8604159a50 legacy=0x7f860406ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:23.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:23.420+0000 7f8609400640 1 -- 172.21.15.6:0/2008776392 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86040af820 con 0x7f86040b0450 2024-03-04T09:36:23.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:23.420+0000 7f8603e00640 1 -- 172.21.15.6:0/2008776392 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 805881887 0 0) 0x7f86040b5d60 con 0x7f8604159a50 2024-03-04T09:36:23.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:23.421+0000 7f8609400640 1 --2- 172.21.15.6:0/2008776392 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f86040b0450 0x7f86040b5700 secure :-1 s=READY pgs=1796 cs=0 l=1 rev1=1 crypto rx=0x7f85fc009d90 tx=0x7f85fc02bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee455c47de494b5b server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:36:23.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:23.422+0000 7f8603e00640 1 -- 172.21.15.6:0/2008776392 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85fc039070 con 0x7f86040b0450 2024-03-04T09:36:23.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:23.422+0000 7f8603e00640 1 -- 172.21.15.6:0/2008776392 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85fc02cd00 con 0x7f86040b0450 2024-03-04T09:36:23.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:23.422+0000 7f8603e00640 1 -- 172.21.15.6:0/2008776392 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85fc034660 con 0x7f86040b0450 2024-03-04T09:36:23.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:23.422+0000 7f8609e00640 1 -- 172.21.15.6:0/2008776392 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f86040b0450 msgr2=0x7f86040b5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:23.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:23.422+0000 7f8609e00640 1 --2- 172.21.15.6:0/2008776392 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f86040b0450 0x7f86040b5700 secure :-1 s=READY pgs=1796 cs=0 l=1 rev1=1 crypto rx=0x7f85fc009d90 tx=0x7f85fc02bf50 comp rx=0 tx=0).stop 2024-03-04T09:36:23.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:23.422+0000 7f8609e00640 1 -- 172.21.15.6:0/2008776392 shutdown_connections 2024-03-04T09:36:23.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:23.422+0000 7f8609e00640 1 --2- 172.21.15.6:0/2008776392 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f86040b0450 0x7f86040b5700 unknown :-1 s=CLOSED pgs=1796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:23.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:23.422+0000 7f8609e00640 1 -- 172.21.15.6:0/2008776392 >> 172.21.15.6:0/2008776392 conn(0x7f860409aa10 msgr2=0x7f860409ae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:36:23.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:23.423+0000 7f8609e00640 1 -- 172.21.15.6:0/2008776392 shutdown_connections 2024-03-04T09:36:23.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:23.423+0000 7f8609e00640 1 -- 172.21.15.6:0/2008776392 wait complete. 2024-03-04T09:36:23.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:23.424+0000 7f8609e00640 1 Processor -- start 2024-03-04T09:36:23.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:23.424+0000 7f8609e00640 1 -- start start 2024-03-04T09:36:23.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:23.425+0000 7f8609e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f86040b0450 0x7f86040d1f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:23.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:23.425+0000 7f8609e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f86040b52a0 0x7f8604154170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:23.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:23.425+0000 7f8609e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8604159a50 0x7f86040cbcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:23.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:23.425+0000 7f8609e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86040b0a80 con 0x7f86040b52a0 2024-03-04T09:36:23.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:23.425+0000 7f8609e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f860415c650 con 0x7f86040b0450 2024-03-04T09:36:23.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:23.425+0000 7f8609e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86040b0c00 con 0x7f8604159a50 2024-03-04T09:36:23.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:23.426+0000 7f85fbe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f86040b52a0 0x7f8604154170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:23.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:23.426+0000 7f85fbe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f86040b52a0 0x7f8604154170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36054/0 (socket says 172.21.15.6:36054) 2024-03-04T09:36:23.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:23.426+0000 7f85fbe00640 1 -- 172.21.15.6:0/2971528363 learned_addr learned my addr 172.21.15.6:0/2971528363 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:36:23.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:23.426+0000 7f85fbe00640 1 -- 172.21.15.6:0/2971528363 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8604159a50 msgr2=0x7f86040cbcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:23.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:23.426+0000 7f85fbe00640 1 --2- 172.21.15.6:0/2971528363 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8604159a50 0x7f86040cbcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:23.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:23.426+0000 7f85fbe00640 1 -- 172.21.15.6:0/2971528363 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f86040b0450 msgr2=0x7f86040d1f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:23.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:23.426+0000 7f85fbe00640 1 --2- 172.21.15.6:0/2971528363 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f86040b0450 0x7f86040d1f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:23.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:23.426+0000 7f85fbe00640 1 -- 172.21.15.6:0/2971528363 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f860415b9e0 con 0x7f86040b52a0 2024-03-04T09:36:23.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:23.427+0000 7f85fbe00640 1 --2- 172.21.15.6:0/2971528363 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f86040b52a0 0x7f8604154170 secure :-1 s=READY pgs=1797 cs=0 l=1 rev1=1 crypto rx=0x7f85ec00c480 tx=0x7f85ec00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:36:23.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:23.427+0000 7f8601600640 1 -- 172.21.15.6:0/2971528363 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85ec0052d0 con 0x7f86040b52a0 2024-03-04T09:36:23.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:23.427+0000 7f8601600640 1 -- 172.21.15.6:0/2971528363 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85ec005470 con 0x7f86040b52a0 2024-03-04T09:36:23.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:23.427+0000 7f8601600640 1 -- 172.21.15.6:0/2971528363 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85ec012870 con 0x7f86040b52a0 2024-03-04T09:36:23.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:23.427+0000 7f8609e00640 1 -- 172.21.15.6:0/2971528363 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f860415bcc0 con 0x7f86040b52a0 2024-03-04T09:36:23.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:23.427+0000 7f8609e00640 1 -- 172.21.15.6:0/2971528363 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86040b30a0 con 0x7f86040b52a0 2024-03-04T09:36:23.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:23.428+0000 7f8609e00640 1 -- 172.21.15.6:0/2971528363 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f860404f330 con 0x7f86040b52a0 2024-03-04T09:36:23.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:23.429+0000 7f8601600640 1 -- 172.21.15.6:0/2971528363 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f85ec012a10 con 0x7f86040b52a0 2024-03-04T09:36:23.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:23.430+0000 7f8601600640 1 --2- 172.21.15.6:0/2971528363 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f85e006b4d0 0x7f85e006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:23.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:23.430+0000 7f8601600640 1 -- 172.21.15.6:0/2971528363 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f85ec089ee0 con 0x7f86040b52a0 2024-03-04T09:36:23.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:23.431+0000 7f8608a00640 1 --2- 172.21.15.6:0/2971528363 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f85e006b4d0 0x7f85e006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:23.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:23.432+0000 7f8608a00640 1 --2- 172.21.15.6:0/2971528363 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f85e006b4d0 0x7f85e006d990 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f85f4000c00 tx=0x7f85f40079c0 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:36:23.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:23.435+0000 7f8601600640 1 -- 172.21.15.6:0/2971528363 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f85ec058d10 con 0x7f86040b52a0 2024-03-04T09:36:23.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:23.679+0000 7f8609e00640 1 -- 172.21.15.6:0/2971528363 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f860415f480 con 0x7f86040b52a0 2024-03-04T09:36:23.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:23.680+0000 7f8601600640 1 -- 172.21.15.6:0/2971528363 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3464 (secure 0 0 0) 0x7f85ec058610 con 0x7f86040b52a0 2024-03-04T09:36:23.677 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:36:23.677 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:35:26.088222+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"laggy_since":"2024-03-04T09:33:54.652601+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":9}]} 2024-03-04T09:36:23.677 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 101 2024-03-04T09:36:23.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:23.683+0000 7f8609e00640 1 -- 172.21.15.6:0/2971528363 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f85e006b4d0 msgr2=0x7f85e006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:23.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:23.683+0000 7f8609e00640 1 --2- 172.21.15.6:0/2971528363 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f85e006b4d0 0x7f85e006d990 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f85f4000c00 tx=0x7f85f40079c0 comp rx=0 tx=0).stop 2024-03-04T09:36:23.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:23.683+0000 7f8609e00640 1 -- 172.21.15.6:0/2971528363 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f86040b52a0 msgr2=0x7f8604154170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:23.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:23.683+0000 7f8609e00640 1 --2- 172.21.15.6:0/2971528363 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f86040b52a0 0x7f8604154170 secure :-1 s=READY pgs=1797 cs=0 l=1 rev1=1 crypto rx=0x7f85ec00c480 tx=0x7f85ec00c940 comp rx=0 tx=0).stop 2024-03-04T09:36:23.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:23.683+0000 7f8609e00640 1 -- 172.21.15.6:0/2971528363 shutdown_connections 2024-03-04T09:36:23.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:23.683+0000 7f8609e00640 1 --2- 172.21.15.6:0/2971528363 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f85e006b4d0 0x7f85e006d990 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:23.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:23.683+0000 7f8609e00640 1 --2- 172.21.15.6:0/2971528363 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8604159a50 0x7f86040cbcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:23.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:23.683+0000 7f8609e00640 1 --2- 172.21.15.6:0/2971528363 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f86040b52a0 0x7f8604154170 unknown :-1 s=CLOSED pgs=1797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:23.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:23.683+0000 7f8609e00640 1 --2- 172.21.15.6:0/2971528363 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f86040b0450 0x7f86040d1f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:23.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:23.683+0000 7f8609e00640 1 -- 172.21.15.6:0/2971528363 >> 172.21.15.6:0/2971528363 conn(0x7f860409aa10 msgr2=0x7f860406ecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:36:23.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:23.683+0000 7f8609e00640 1 -- 172.21.15.6:0/2971528363 shutdown_connections 2024-03-04T09:36:23.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:23.683+0000 7f8609e00640 1 -- 172.21.15.6:0/2971528363 wait complete. 2024-03-04T09:36:23.903 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:32:57.159681+0000', 'modified': '2024-03-04T09:35:26.088222+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8214, 'mds_1': 6796}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6796': {'gid': 6796, 'name': 'd', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.146:6837/2458070840', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 2458070840}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 2458070840}]}, 'join_fscid': -1, 'laggy_since': '2024-03-04T09:33:54.652601+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8214': {'gid': 8214, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2446627517', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2446627517}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2446627517}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8214, 'qdb_cluster': [8214]} 2024-03-04T09:36:23.903 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6796:up:stopping 2024-03-04T09:36:23.945 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-04T09:36:24.904 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:36:25.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:25.129+0000 7fdcdba00640 1 Processor -- start 2024-03-04T09:36:25.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:25.130+0000 7fdcdba00640 1 -- start start 2024-03-04T09:36:25.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:25.131+0000 7fdcdba00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdcd40af1a0 0x7fdcd415c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:25.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:25.131+0000 7fdcdba00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fdcd40cdc20 0x7fdcd40b2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:25.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:25.131+0000 7fdcdba00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcd415cc40 con 0x7fdcd40af580 2024-03-04T09:36:25.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:25.132+0000 7fdcdba00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdcd415cf20 con 0x7fdcd40cdc20 2024-03-04T09:36:25.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:25.132+0000 7fdcdba00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcd415d200 con 0x7fdcd40af1a0 2024-03-04T09:36:25.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:25.132+0000 7fdcd9a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fdcd40cdc20 0x7fdcd40b2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:25.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:25.132+0000 7fdcd9a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fdcd40cdc20 0x7fdcd40b2dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36068/0 (socket says 172.21.15.6:36068) 2024-03-04T09:36:25.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:25.132+0000 7fdcd9a00640 1 -- 172.21.15.6:0/3329686828 learned_addr learned my addr 172.21.15.6:0/3329686828 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:36:25.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:25.132+0000 7fdcd9000640 1 --2- 172.21.15.6:0/3329686828 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdcd40af1a0 0x7fdcd415c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:25.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:25.132+0000 7fdcd9a00640 1 -- 172.21.15.6:0/3329686828 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdcd40af1a0 msgr2=0x7fdcd415c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:36:25.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:25.132+0000 7fdcd9a00640 1 --2- 172.21.15.6:0/3329686828 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdcd40af1a0 0x7fdcd415c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:25.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:25.133+0000 7fdcd9a00640 1 -- 172.21.15.6:0/3329686828 >> v1:172.21.15.146:6789/0 conn(0x7fdcd40af580 legacy=0x7fdcd40b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:36:25.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:25.133+0000 7fdcd9a00640 1 -- 172.21.15.6:0/3329686828 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcd415d4e0 con 0x7fdcd40cdc20 2024-03-04T09:36:25.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:25.134+0000 7fdcd9a00640 1 --2- 172.21.15.6:0/3329686828 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdcd40cdc20 0x7fdcd40b2dd0 secure :-1 s=READY pgs=1798 cs=0 l=1 rev1=1 crypto rx=0x7fdccc009e50 tx=0x7fdccc02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6abaec128115798 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:36:25.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:25.134+0000 7fdcd3400640 1 -- 172.21.15.6:0/3329686828 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdccc02c9e0 con 0x7fdcd40cdc20 2024-03-04T09:36:25.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:25.134+0000 7fdcd3400640 1 -- 172.21.15.6:0/3329686828 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdccc02cb80 con 0x7fdcd40cdc20 2024-03-04T09:36:25.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:25.134+0000 7fdcd3400640 1 -- 172.21.15.6:0/3329686828 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdccc0343f0 con 0x7fdcd40cdc20 2024-03-04T09:36:25.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:25.135+0000 7fdcdba00640 1 -- 172.21.15.6:0/3329686828 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdcd40cdc20 msgr2=0x7fdcd40b2dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:25.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:25.135+0000 7fdcdba00640 1 --2- 172.21.15.6:0/3329686828 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdcd40cdc20 0x7fdcd40b2dd0 secure :-1 s=READY pgs=1798 cs=0 l=1 rev1=1 crypto rx=0x7fdccc009e50 tx=0x7fdccc02be40 comp rx=0 tx=0).stop 2024-03-04T09:36:25.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:25.135+0000 7fdcdba00640 1 -- 172.21.15.6:0/3329686828 shutdown_connections 2024-03-04T09:36:25.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:25.135+0000 7fdcdba00640 1 --2- 172.21.15.6:0/3329686828 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdcd40cdc20 0x7fdcd40b2dd0 unknown :-1 s=CLOSED pgs=1798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:25.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:25.135+0000 7fdcdba00640 1 --2- 172.21.15.6:0/3329686828 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdcd40af1a0 0x7fdcd415c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:25.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:25.135+0000 7fdcdba00640 1 -- 172.21.15.6:0/3329686828 >> 172.21.15.6:0/3329686828 conn(0x7fdcd406afb0 msgr2=0x7fdcd406b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:36:25.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:25.135+0000 7fdcdba00640 1 -- 172.21.15.6:0/3329686828 shutdown_connections 2024-03-04T09:36:25.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:25.135+0000 7fdcdba00640 1 -- 172.21.15.6:0/3329686828 wait complete. 2024-03-04T09:36:25.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:25.137+0000 7fdcdba00640 1 Processor -- start 2024-03-04T09:36:25.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:25.137+0000 7fdcdba00640 1 -- start start 2024-03-04T09:36:25.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:25.138+0000 7fdcdba00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdcd40af1a0 0x7fdcd415dd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:25.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:25.138+0000 7fdcdba00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdcd40af580 0x7fdcd406dc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:25.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:25.138+0000 7fdcdba00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdcd40cdc20 0x7fdcd41f70c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:25.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:25.138+0000 7fdcdba00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcd40b36a0 con 0x7fdcd40af580 2024-03-04T09:36:25.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:25.138+0000 7fdcdba00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcd40b3520 con 0x7fdcd40cdc20 2024-03-04T09:36:25.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:25.138+0000 7fdcdba00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcd40b3820 con 0x7fdcd40af1a0 2024-03-04T09:36:25.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:25.138+0000 7fdcd3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdcd40af580 0x7fdcd406dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:25.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:25.138+0000 7fdcd3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdcd40af580 0x7fdcd406dc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36080/0 (socket says 172.21.15.6:36080) 2024-03-04T09:36:25.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:25.138+0000 7fdcd3e00640 1 -- 172.21.15.6:0/2742278355 learned_addr learned my addr 172.21.15.6:0/2742278355 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:36:25.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:25.138+0000 7fdcd9000640 1 --2- 172.21.15.6:0/2742278355 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdcd40af1a0 0x7fdcd415dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:25.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:25.138+0000 7fdcd9a00640 1 --2- 172.21.15.6:0/2742278355 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdcd40cdc20 0x7fdcd41f70c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:25.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:25.138+0000 7fdcd3e00640 1 -- 172.21.15.6:0/2742278355 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdcd40af1a0 msgr2=0x7fdcd415dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:25.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:25.138+0000 7fdcd3e00640 1 --2- 172.21.15.6:0/2742278355 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdcd40af1a0 0x7fdcd415dd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:25.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:25.138+0000 7fdcd3e00640 1 -- 172.21.15.6:0/2742278355 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdcd40cdc20 msgr2=0x7fdcd41f70c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:25.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:25.138+0000 7fdcd3e00640 1 --2- 172.21.15.6:0/2742278355 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdcd40cdc20 0x7fdcd41f70c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:25.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:25.138+0000 7fdcd3e00640 1 -- 172.21.15.6:0/2742278355 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcd415f650 con 0x7fdcd40af580 2024-03-04T09:36:25.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:25.139+0000 7fdcd3e00640 1 --2- 172.21.15.6:0/2742278355 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdcd40af580 0x7fdcd406dc80 secure :-1 s=READY pgs=1799 cs=0 l=1 rev1=1 crypto rx=0x7fdcc000c460 tx=0x7fdcc000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:36:25.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:25.139+0000 7fdcd0c00640 1 -- 172.21.15.6:0/2742278355 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcc0016020 con 0x7fdcd40af580 2024-03-04T09:36:25.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:25.139+0000 7fdcd0c00640 1 -- 172.21.15.6:0/2742278355 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcc0012870 con 0x7fdcd40af580 2024-03-04T09:36:25.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:25.139+0000 7fdcdba00640 1 -- 172.21.15.6:0/2742278355 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcd415e5b0 con 0x7fdcd40af580 2024-03-04T09:36:25.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:25.139+0000 7fdcd0c00640 1 -- 172.21.15.6:0/2742278355 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcc0012c60 con 0x7fdcd40af580 2024-03-04T09:36:25.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:25.139+0000 7fdcdba00640 1 -- 172.21.15.6:0/2742278355 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcd415ee50 con 0x7fdcd40af580 2024-03-04T09:36:25.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:25.140+0000 7fdcdba00640 1 -- 172.21.15.6:0/2742278355 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcd4069510 con 0x7fdcd40af580 2024-03-04T09:36:25.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:25.141+0000 7fdcd0c00640 1 -- 172.21.15.6:0/2742278355 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdcc0019020 con 0x7fdcd40af580 2024-03-04T09:36:25.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:25.142+0000 7fdcd0c00640 1 --2- 172.21.15.6:0/2742278355 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fdcb406b4d0 0x7fdcb406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:25.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:25.142+0000 7fdcd0c00640 1 -- 172.21.15.6:0/2742278355 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fdcc0089ed0 con 0x7fdcd40af580 2024-03-04T09:36:25.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:25.143+0000 7fdcd9000640 1 --2- 172.21.15.6:0/2742278355 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fdcb406b4d0 0x7fdcb406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:25.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:25.144+0000 7fdcd9000640 1 --2- 172.21.15.6:0/2742278355 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fdcb406b4d0 0x7fdcb406d990 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7fdcc8006fd0 tx=0x7fdcc8008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:36:25.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:25.148+0000 7fdcd0c00640 1 -- 172.21.15.6:0/2742278355 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdcc0020030 con 0x7fdcd40af580 2024-03-04T09:36:25.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:25.383+0000 7fdcdba00640 1 -- 172.21.15.6:0/2742278355 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdcd40cf350 con 0x7fdcd40af580 2024-03-04T09:36:25.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:25.385+0000 7fdcd0c00640 1 -- 172.21.15.6:0/2742278355 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3464 (secure 0 0 0) 0x7fdcc0058da0 con 0x7fdcd40af580 2024-03-04T09:36:25.382 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:36:25.382 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:35:26.088222+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"laggy_since":"2024-03-04T09:33:54.652601+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":9}]} 2024-03-04T09:36:25.382 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 101 2024-03-04T09:36:25.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:25.387+0000 7fdcdba00640 1 -- 172.21.15.6:0/2742278355 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fdcb406b4d0 msgr2=0x7fdcb406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:25.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:25.387+0000 7fdcdba00640 1 --2- 172.21.15.6:0/2742278355 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fdcb406b4d0 0x7fdcb406d990 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7fdcc8006fd0 tx=0x7fdcc8008040 comp rx=0 tx=0).stop 2024-03-04T09:36:25.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:25.387+0000 7fdcdba00640 1 -- 172.21.15.6:0/2742278355 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdcd40af580 msgr2=0x7fdcd406dc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:25.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:25.387+0000 7fdcdba00640 1 --2- 172.21.15.6:0/2742278355 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdcd40af580 0x7fdcd406dc80 secure :-1 s=READY pgs=1799 cs=0 l=1 rev1=1 crypto rx=0x7fdcc000c460 tx=0x7fdcc000c920 comp rx=0 tx=0).stop 2024-03-04T09:36:25.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:25.387+0000 7fdcdba00640 1 -- 172.21.15.6:0/2742278355 shutdown_connections 2024-03-04T09:36:25.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:25.387+0000 7fdcdba00640 1 --2- 172.21.15.6:0/2742278355 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdcd40cdc20 0x7fdcd41f70c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:25.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:25.387+0000 7fdcdba00640 1 --2- 172.21.15.6:0/2742278355 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdcd40af580 0x7fdcd406dc80 unknown :-1 s=CLOSED pgs=1799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:25.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:25.387+0000 7fdcdba00640 1 --2- 172.21.15.6:0/2742278355 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fdcb406b4d0 0x7fdcb406d990 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:25.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:25.387+0000 7fdcdba00640 1 --2- 172.21.15.6:0/2742278355 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdcd40af1a0 0x7fdcd415dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:25.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:25.387+0000 7fdcdba00640 1 -- 172.21.15.6:0/2742278355 >> 172.21.15.6:0/2742278355 conn(0x7fdcd406afb0 msgr2=0x7fdcd40cc4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:36:25.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:25.387+0000 7fdcdba00640 1 -- 172.21.15.6:0/2742278355 shutdown_connections 2024-03-04T09:36:25.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:25.388+0000 7fdcdba00640 1 -- 172.21.15.6:0/2742278355 wait complete. 2024-03-04T09:36:25.606 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:32:57.159681+0000', 'modified': '2024-03-04T09:35:26.088222+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8214, 'mds_1': 6796}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6796': {'gid': 6796, 'name': 'd', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.146:6837/2458070840', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 2458070840}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 2458070840}]}, 'join_fscid': -1, 'laggy_since': '2024-03-04T09:33:54.652601+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8214': {'gid': 8214, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2446627517', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2446627517}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2446627517}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8214, 'qdb_cluster': [8214]} 2024-03-04T09:36:25.606 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6796:up:stopping 2024-03-04T09:36:25.947 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-04T09:36:26.606 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:36:26.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:26.823+0000 7eff3d000640 1 Processor -- start 2024-03-04T09:36:26.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:26.824+0000 7eff3d000640 1 -- start start 2024-03-04T09:36:26.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:26.825+0000 7eff3d000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7eff380af1a0 0x7eff3815c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:26.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:26.825+0000 7eff3d000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7eff380cdc20 0x7eff380b2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:26.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:26.825+0000 7eff3d000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff3815cc40 con 0x7eff380af580 2024-03-04T09:36:26.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:26.825+0000 7eff3d000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff3815cf20 con 0x7eff380af1a0 2024-03-04T09:36:26.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:26.825+0000 7eff3d000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff3815d200 con 0x7eff380cdc20 2024-03-04T09:36:26.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:26.825+0000 7eff35e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7eff380af1a0 0x7eff3815c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:26.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:26.825+0000 7eff35e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7eff380af1a0 0x7eff3815c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36092/0 (socket says 172.21.15.6:36092) 2024-03-04T09:36:26.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:26.825+0000 7eff35e00640 1 -- 172.21.15.6:0/1866201289 learned_addr learned my addr 172.21.15.6:0/1866201289 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:36:26.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:26.826+0000 7eff36800640 1 --2- 172.21.15.6:0/1866201289 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7eff380cdc20 0x7eff380b2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:26.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:26.826+0000 7eff35e00640 1 -- 172.21.15.6:0/1866201289 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7eff380cdc20 msgr2=0x7eff380b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:36:26.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:26.826+0000 7eff35e00640 1 --2- 172.21.15.6:0/1866201289 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7eff380cdc20 0x7eff380b2dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:26.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:26.826+0000 7eff35e00640 1 -- 172.21.15.6:0/1866201289 >> v1:172.21.15.146:6789/0 conn(0x7eff380af580 legacy=0x7eff380b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:36:26.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:26.826+0000 7eff35e00640 1 -- 172.21.15.6:0/1866201289 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff3815d4e0 con 0x7eff380af1a0 2024-03-04T09:36:26.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:26.826+0000 7eff36800640 1 --2- 172.21.15.6:0/1866201289 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7eff380cdc20 0x7eff380b2dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:36:26.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:26.827+0000 7eff35e00640 1 --2- 172.21.15.6:0/1866201289 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7eff380af1a0 0x7eff3815c600 secure :-1 s=READY pgs=1800 cs=0 l=1 rev1=1 crypto rx=0x7eff28009870 tx=0x7eff2802b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9272b9270a2513 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:36:26.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:26.827+0000 7eff35400640 1 -- 172.21.15.6:0/1866201289 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff280317c0 con 0x7eff380af1a0 2024-03-04T09:36:26.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:26.827+0000 7eff35400640 1 -- 172.21.15.6:0/1866201289 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff28037070 con 0x7eff380af1a0 2024-03-04T09:36:26.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:26.827+0000 7eff35400640 1 -- 172.21.15.6:0/1866201289 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff28031b80 con 0x7eff380af1a0 2024-03-04T09:36:26.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:26.828+0000 7eff3d000640 1 -- 172.21.15.6:0/1866201289 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7eff380af1a0 msgr2=0x7eff3815c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:26.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:26.828+0000 7eff3d000640 1 --2- 172.21.15.6:0/1866201289 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7eff380af1a0 0x7eff3815c600 secure :-1 s=READY pgs=1800 cs=0 l=1 rev1=1 crypto rx=0x7eff28009870 tx=0x7eff2802b860 comp rx=0 tx=0).stop 2024-03-04T09:36:26.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:26.828+0000 7eff3d000640 1 -- 172.21.15.6:0/1866201289 shutdown_connections 2024-03-04T09:36:26.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:26.828+0000 7eff3d000640 1 --2- 172.21.15.6:0/1866201289 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7eff380cdc20 0x7eff380b2dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:26.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:26.828+0000 7eff3d000640 1 --2- 172.21.15.6:0/1866201289 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7eff380af1a0 0x7eff3815c600 unknown :-1 s=CLOSED pgs=1800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:26.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:26.828+0000 7eff3d000640 1 -- 172.21.15.6:0/1866201289 >> 172.21.15.6:0/1866201289 conn(0x7eff3806afb0 msgr2=0x7eff3806b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:36:26.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:26.828+0000 7eff3d000640 1 -- 172.21.15.6:0/1866201289 shutdown_connections 2024-03-04T09:36:26.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:26.828+0000 7eff3d000640 1 -- 172.21.15.6:0/1866201289 wait complete. 2024-03-04T09:36:26.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:26.830+0000 7eff3d000640 1 Processor -- start 2024-03-04T09:36:26.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:26.830+0000 7eff3d000640 1 -- start start 2024-03-04T09:36:26.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:26.831+0000 7eff3d000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7eff380af1a0 0x7eff3809b590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:26.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:26.831+0000 7eff3d000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7eff380af580 0x7eff381464a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:26.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:26.831+0000 7eff3d000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7eff380cdc20 0x7eff38152670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:26.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:26.831+0000 7eff3d000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff380b3720 con 0x7eff380af580 2024-03-04T09:36:26.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:26.831+0000 7eff3d000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff380b35a0 con 0x7eff380cdc20 2024-03-04T09:36:26.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:26.831+0000 7eff3d000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff38155e70 con 0x7eff380af1a0 2024-03-04T09:36:26.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:26.831+0000 7eff2fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7eff380af580 0x7eff381464a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:26.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:26.831+0000 7eff2fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7eff380af580 0x7eff381464a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36094/0 (socket says 172.21.15.6:36094) 2024-03-04T09:36:26.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:26.831+0000 7eff2fe00640 1 -- 172.21.15.6:0/2123464804 learned_addr learned my addr 172.21.15.6:0/2123464804 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:36:26.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:26.832+0000 7eff35e00640 1 --2- 172.21.15.6:0/2123464804 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7eff380af1a0 0x7eff3809b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:26.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:26.832+0000 7eff36800640 1 --2- 172.21.15.6:0/2123464804 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7eff380cdc20 0x7eff38152670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:26.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:26.832+0000 7eff2fe00640 1 -- 172.21.15.6:0/2123464804 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7eff380af1a0 msgr2=0x7eff3809b590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:26.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:26.832+0000 7eff2fe00640 1 --2- 172.21.15.6:0/2123464804 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7eff380af1a0 0x7eff3809b590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:26.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:26.832+0000 7eff2fe00640 1 -- 172.21.15.6:0/2123464804 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7eff380cdc20 msgr2=0x7eff38152670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:26.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:26.832+0000 7eff2fe00640 1 --2- 172.21.15.6:0/2123464804 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7eff380cdc20 0x7eff38152670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:26.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:26.832+0000 7eff2fe00640 1 -- 172.21.15.6:0/2123464804 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff3809bad0 con 0x7eff380af580 2024-03-04T09:36:26.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:26.832+0000 7eff2fe00640 1 --2- 172.21.15.6:0/2123464804 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7eff380af580 0x7eff381464a0 secure :-1 s=READY pgs=1801 cs=0 l=1 rev1=1 crypto rx=0x7eff2000c400 tx=0x7eff2000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:36:26.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:26.833+0000 7eff2e000640 1 -- 172.21.15.6:0/2123464804 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff20011790 con 0x7eff380af580 2024-03-04T09:36:26.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:26.833+0000 7eff2e000640 1 -- 172.21.15.6:0/2123464804 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff20017070 con 0x7eff380af580 2024-03-04T09:36:26.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:26.833+0000 7eff3d000640 1 -- 172.21.15.6:0/2123464804 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff38153130 con 0x7eff380af580 2024-03-04T09:36:26.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:26.833+0000 7eff2e000640 1 -- 172.21.15.6:0/2123464804 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff20011bb0 con 0x7eff380af580 2024-03-04T09:36:26.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:26.833+0000 7eff3d000640 1 -- 172.21.15.6:0/2123464804 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff3809aab0 con 0x7eff380af580 2024-03-04T09:36:26.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:26.834+0000 7eff3d000640 1 -- 172.21.15.6:0/2123464804 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff3804f3a0 con 0x7eff380af580 2024-03-04T09:36:26.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:26.835+0000 7eff2e000640 1 -- 172.21.15.6:0/2123464804 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7eff2001c460 con 0x7eff380af580 2024-03-04T09:36:26.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:26.836+0000 7eff2e000640 1 --2- 172.21.15.6:0/2123464804 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7eff1406b4d0 0x7eff1406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:26.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:26.836+0000 7eff2e000640 1 -- 172.21.15.6:0/2123464804 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7eff2008a130 con 0x7eff380af580 2024-03-04T09:36:26.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:26.837+0000 7eff35e00640 1 --2- 172.21.15.6:0/2123464804 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7eff1406b4d0 0x7eff1406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:26.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:26.838+0000 7eff35e00640 1 --2- 172.21.15.6:0/2123464804 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7eff1406b4d0 0x7eff1406d990 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7eff28036000 tx=0x7eff2803d040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:36:26.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:26.840+0000 7eff2e000640 1 -- 172.21.15.6:0/2123464804 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7eff20058ed0 con 0x7eff380af580 2024-03-04T09:36:27.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:27.245+0000 7eff3d000640 1 -- 172.21.15.6:0/2123464804 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7eff380ae770 con 0x7eff380af580 2024-03-04T09:36:27.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:27.246+0000 7eff2e000640 1 -- 172.21.15.6:0/2123464804 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3464 (secure 0 0 0) 0x7eff200587d0 con 0x7eff380af580 2024-03-04T09:36:27.243 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:36:27.243 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:35:26.088222+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"laggy_since":"2024-03-04T09:33:54.652601+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":9}]} 2024-03-04T09:36:27.243 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 101 2024-03-04T09:36:27.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:27.248+0000 7eff3d000640 1 -- 172.21.15.6:0/2123464804 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7eff1406b4d0 msgr2=0x7eff1406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:27.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:27.248+0000 7eff3d000640 1 --2- 172.21.15.6:0/2123464804 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7eff1406b4d0 0x7eff1406d990 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7eff28036000 tx=0x7eff2803d040 comp rx=0 tx=0).stop 2024-03-04T09:36:27.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:27.248+0000 7eff3d000640 1 -- 172.21.15.6:0/2123464804 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7eff380af580 msgr2=0x7eff381464a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:27.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:27.248+0000 7eff3d000640 1 --2- 172.21.15.6:0/2123464804 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7eff380af580 0x7eff381464a0 secure :-1 s=READY pgs=1801 cs=0 l=1 rev1=1 crypto rx=0x7eff2000c400 tx=0x7eff2000c8c0 comp rx=0 tx=0).stop 2024-03-04T09:36:27.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:27.248+0000 7eff3d000640 1 -- 172.21.15.6:0/2123464804 shutdown_connections 2024-03-04T09:36:27.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:27.248+0000 7eff3d000640 1 --2- 172.21.15.6:0/2123464804 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7eff380cdc20 0x7eff38152670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:27.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:27.248+0000 7eff3d000640 1 --2- 172.21.15.6:0/2123464804 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7eff380af580 0x7eff381464a0 unknown :-1 s=CLOSED pgs=1801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:27.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:27.248+0000 7eff3d000640 1 --2- 172.21.15.6:0/2123464804 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7eff1406b4d0 0x7eff1406d990 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:27.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:27.248+0000 7eff3d000640 1 --2- 172.21.15.6:0/2123464804 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7eff380af1a0 0x7eff3809b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:27.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:27.248+0000 7eff3d000640 1 -- 172.21.15.6:0/2123464804 >> 172.21.15.6:0/2123464804 conn(0x7eff3806afb0 msgr2=0x7eff3806b420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:36:27.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:27.248+0000 7eff3d000640 1 -- 172.21.15.6:0/2123464804 shutdown_connections 2024-03-04T09:36:27.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:27.249+0000 7eff3d000640 1 -- 172.21.15.6:0/2123464804 wait complete. 2024-03-04T09:36:27.466 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:32:57.159681+0000', 'modified': '2024-03-04T09:35:26.088222+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8214, 'mds_1': 6796}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6796': {'gid': 6796, 'name': 'd', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.146:6837/2458070840', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 2458070840}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 2458070840}]}, 'join_fscid': -1, 'laggy_since': '2024-03-04T09:33:54.652601+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8214': {'gid': 8214, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2446627517', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2446627517}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2446627517}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8214, 'qdb_cluster': [8214]} 2024-03-04T09:36:27.466 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6796:up:stopping 2024-03-04T09:36:27.949 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-04T09:36:28.467 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:36:28.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:28.698+0000 7f56d6600640 1 Processor -- start 2024-03-04T09:36:28.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:28.699+0000 7f56d6600640 1 -- start start 2024-03-04T09:36:28.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:28.700+0000 7f56d6600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f56d00b4650 0x7f56d00b4ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:28.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:28.700+0000 7f56d6600640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f56d015c4a0 0x7f56d015c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:28.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:28.700+0000 7f56d6600640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56d006e4b0 con 0x7f56d0151b30 2024-03-04T09:36:28.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:28.700+0000 7f56d6600640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56d00b5000 con 0x7f56d00b4650 2024-03-04T09:36:28.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:28.700+0000 7f56d6600640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56d006e790 con 0x7f56d015c4a0 2024-03-04T09:36:28.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:28.700+0000 7f56cfe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f56d00b4650 0x7f56d00b4ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:28.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:28.700+0000 7f56d5200640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f56d0151b30 0x7f56d006f270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:47350/0 (socket says 172.21.15.6:47350) 2024-03-04T09:36:28.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:28.700+0000 7f56d5200640 1 -- 172.21.15.6:0/2959648763 learned_addr learned my addr 172.21.15.6:0/2959648763 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:36:28.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:28.701+0000 7f56d5c00640 1 --2- 172.21.15.6:0/2959648763 >> v2:172.21.15.146:3300/0 conn(0x7f56d015c4a0 0x7f56d015c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:28.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:28.701+0000 7f56cfe00640 1 -- 172.21.15.6:0/2959648763 >> v2:172.21.15.146:3300/0 conn(0x7f56d015c4a0 msgr2=0x7f56d015c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:36:28.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:28.701+0000 7f56cfe00640 1 --2- 172.21.15.6:0/2959648763 >> v2:172.21.15.146:3300/0 conn(0x7f56d015c4a0 0x7f56d015c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:28.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:28.701+0000 7f56cfe00640 1 -- 172.21.15.6:0/2959648763 >> v1:172.21.15.6:6789/0 conn(0x7f56d0151b30 legacy=0x7f56d006f270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:28.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:28.701+0000 7f56cfe00640 1 -- 172.21.15.6:0/2959648763 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56d006ea70 con 0x7f56d00b4650 2024-03-04T09:36:28.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:28.702+0000 7f56cfe00640 1 --2- 172.21.15.6:0/2959648763 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f56d00b4650 0x7f56d00b4ac0 secure :-1 s=READY pgs=1803 cs=0 l=1 rev1=1 crypto rx=0x7f56d006e0c0 tx=0x7f56bc02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3692de416e307eb6 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:36:28.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:28.702+0000 7f56cf400640 1 -- 172.21.15.6:0/2959648763 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56bc0315a0 con 0x7f56d00b4650 2024-03-04T09:36:28.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:28.702+0000 7f56cf400640 1 -- 172.21.15.6:0/2959648763 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56bc037070 con 0x7f56d00b4650 2024-03-04T09:36:28.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:28.702+0000 7f56cf400640 1 -- 172.21.15.6:0/2959648763 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56bc031960 con 0x7f56d00b4650 2024-03-04T09:36:28.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:28.703+0000 7f56d6600640 1 -- 172.21.15.6:0/2959648763 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f56d00b4650 msgr2=0x7f56d00b4ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:28.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:28.703+0000 7f56d6600640 1 --2- 172.21.15.6:0/2959648763 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f56d00b4650 0x7f56d00b4ac0 secure :-1 s=READY pgs=1803 cs=0 l=1 rev1=1 crypto rx=0x7f56d006e0c0 tx=0x7f56bc02b840 comp rx=0 tx=0).stop 2024-03-04T09:36:28.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:28.703+0000 7f56d6600640 1 -- 172.21.15.6:0/2959648763 shutdown_connections 2024-03-04T09:36:28.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:28.703+0000 7f56d6600640 1 --2- 172.21.15.6:0/2959648763 >> v2:172.21.15.146:3300/0 conn(0x7f56d015c4a0 0x7f56d015c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:28.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:28.703+0000 7f56d6600640 1 --2- 172.21.15.6:0/2959648763 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f56d00b4650 0x7f56d00b4ac0 unknown :-1 s=CLOSED pgs=1803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:28.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:28.703+0000 7f56d6600640 1 -- 172.21.15.6:0/2959648763 >> 172.21.15.6:0/2959648763 conn(0x7f56d006c0c0 msgr2=0x7f56d006c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:36:28.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:28.703+0000 7f56d6600640 1 -- 172.21.15.6:0/2959648763 shutdown_connections 2024-03-04T09:36:28.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:28.703+0000 7f56d6600640 1 -- 172.21.15.6:0/2959648763 wait complete. 2024-03-04T09:36:28.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:28.705+0000 7f56d6600640 1 Processor -- start 2024-03-04T09:36:28.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:28.705+0000 7f56d6600640 1 -- start start 2024-03-04T09:36:28.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:28.706+0000 7f56d6600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f56d00b4650 0x7f56d00cf2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:28.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:28.706+0000 7f56d6600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f56d0151b30 0x7f56d014c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:28.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:28.706+0000 7f56d6600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f56d015c4a0 0x7f56d0147140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:28.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:28.706+0000 7f56d6600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56d015ce30 con 0x7f56d00b4650 2024-03-04T09:36:28.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:28.706+0000 7f56d6600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56d0151f10 con 0x7f56d015c4a0 2024-03-04T09:36:28.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:28.706+0000 7f56d6600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56d015e8b0 con 0x7f56d0151b30 2024-03-04T09:36:28.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:28.706+0000 7f56d5200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f56d00b4650 0x7f56d00cf2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:28.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:28.706+0000 7f56d5200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f56d00b4650 0x7f56d00cf2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36112/0 (socket says 172.21.15.6:36112) 2024-03-04T09:36:28.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:28.706+0000 7f56d5200640 1 -- 172.21.15.6:0/1373392873 learned_addr learned my addr 172.21.15.6:0/1373392873 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:36:28.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:28.707+0000 7f56d5c00640 1 --2- 172.21.15.6:0/1373392873 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f56d015c4a0 0x7f56d0147140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:28.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:28.707+0000 7f56cfe00640 1 --2- 172.21.15.6:0/1373392873 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f56d0151b30 0x7f56d014c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:28.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:28.707+0000 7f56d5200640 1 -- 172.21.15.6:0/1373392873 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f56d0151b30 msgr2=0x7f56d014c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:28.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:28.707+0000 7f56d5200640 1 --2- 172.21.15.6:0/1373392873 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f56d0151b30 0x7f56d014c990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:28.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:28.707+0000 7f56d5200640 1 -- 172.21.15.6:0/1373392873 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f56d015c4a0 msgr2=0x7f56d0147140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:28.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:28.707+0000 7f56d5200640 1 --2- 172.21.15.6:0/1373392873 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f56d015c4a0 0x7f56d0147140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:28.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:28.707+0000 7f56d5200640 1 -- 172.21.15.6:0/1373392873 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56d0152f10 con 0x7f56d00b4650 2024-03-04T09:36:28.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:28.707+0000 7f56cfe00640 1 --2- 172.21.15.6:0/1373392873 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f56d0151b30 0x7f56d014c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:36:28.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:28.707+0000 7f56d5200640 1 --2- 172.21.15.6:0/1373392873 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f56d00b4650 0x7f56d00cf2e0 secure :-1 s=READY pgs=1804 cs=0 l=1 rev1=1 crypto rx=0x7f56c400c400 tx=0x7f56c400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:36:28.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:28.708+0000 7f56ccc00640 1 -- 172.21.15.6:0/1373392873 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56c4017070 con 0x7f56d00b4650 2024-03-04T09:36:28.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:28.708+0000 7f56ccc00640 1 -- 172.21.15.6:0/1373392873 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56c4005ce0 con 0x7f56d00b4650 2024-03-04T09:36:28.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:28.708+0000 7f56ccc00640 1 -- 172.21.15.6:0/1373392873 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56c40072f0 con 0x7f56d00b4650 2024-03-04T09:36:28.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:28.708+0000 7f56d6600640 1 -- 172.21.15.6:0/1373392873 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56d00ce800 con 0x7f56d00b4650 2024-03-04T09:36:28.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:28.708+0000 7f56d6600640 1 -- 172.21.15.6:0/1373392873 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56d00cedc0 con 0x7f56d00b4650 2024-03-04T09:36:28.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:28.709+0000 7f56d6600640 1 -- 172.21.15.6:0/1373392873 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56d004f330 con 0x7f56d00b4650 2024-03-04T09:36:28.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:28.710+0000 7f56ccc00640 1 -- 172.21.15.6:0/1373392873 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f56c401a070 con 0x7f56d00b4650 2024-03-04T09:36:28.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:28.711+0000 7f56ccc00640 1 --2- 172.21.15.6:0/1373392873 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f56b006b500 0x7f56b006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:28.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:28.711+0000 7f56ccc00640 1 -- 172.21.15.6:0/1373392873 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f56c40895a0 con 0x7f56d00b4650 2024-03-04T09:36:28.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:28.711+0000 7f56cfe00640 1 --2- 172.21.15.6:0/1373392873 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f56b006b500 0x7f56b006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:28.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:28.713+0000 7f56cfe00640 1 --2- 172.21.15.6:0/1373392873 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f56b006b500 0x7f56b006d9c0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f56bc036000 tx=0x7f56bc008030 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:36:28.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:28.719+0000 7f56ccc00640 1 -- 172.21.15.6:0/1373392873 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f56c40583d0 con 0x7f56d00b4650 2024-03-04T09:36:28.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:28.961+0000 7f56d6600640 1 -- 172.21.15.6:0/1373392873 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f56d006fb40 con 0x7f56d00b4650 2024-03-04T09:36:28.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:28.962+0000 7f56ccc00640 1 -- 172.21.15.6:0/1373392873 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3464 (secure 0 0 0) 0x7f56c405e0d0 con 0x7f56d00b4650 2024-03-04T09:36:28.959 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:36:28.959 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:35:26.088222+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"laggy_since":"2024-03-04T09:33:54.652601+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":9}]} 2024-03-04T09:36:28.959 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 101 2024-03-04T09:36:28.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:28.965+0000 7f56d6600640 1 -- 172.21.15.6:0/1373392873 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f56b006b500 msgr2=0x7f56b006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:28.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:28.965+0000 7f56d6600640 1 --2- 172.21.15.6:0/1373392873 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f56b006b500 0x7f56b006d9c0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f56bc036000 tx=0x7f56bc008030 comp rx=0 tx=0).stop 2024-03-04T09:36:28.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:28.965+0000 7f56d6600640 1 -- 172.21.15.6:0/1373392873 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f56d00b4650 msgr2=0x7f56d00cf2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:28.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:28.965+0000 7f56d6600640 1 --2- 172.21.15.6:0/1373392873 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f56d00b4650 0x7f56d00cf2e0 secure :-1 s=READY pgs=1804 cs=0 l=1 rev1=1 crypto rx=0x7f56c400c400 tx=0x7f56c400c8c0 comp rx=0 tx=0).stop 2024-03-04T09:36:28.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:28.965+0000 7f56d6600640 1 -- 172.21.15.6:0/1373392873 shutdown_connections 2024-03-04T09:36:28.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:28.965+0000 7f56d6600640 1 --2- 172.21.15.6:0/1373392873 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f56d015c4a0 0x7f56d0147140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:28.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:28.965+0000 7f56d6600640 1 --2- 172.21.15.6:0/1373392873 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f56b006b500 0x7f56b006d9c0 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:28.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:28.965+0000 7f56d6600640 1 --2- 172.21.15.6:0/1373392873 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f56d0151b30 0x7f56d014c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:28.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:28.965+0000 7f56d6600640 1 --2- 172.21.15.6:0/1373392873 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f56d00b4650 0x7f56d00cf2e0 unknown :-1 s=CLOSED pgs=1804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:28.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:28.965+0000 7f56d6600640 1 -- 172.21.15.6:0/1373392873 >> 172.21.15.6:0/1373392873 conn(0x7f56d006c0c0 msgr2=0x7f56d0073f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:36:28.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:28.966+0000 7f56d6600640 1 -- 172.21.15.6:0/1373392873 shutdown_connections 2024-03-04T09:36:28.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:28.966+0000 7f56d6600640 1 -- 172.21.15.6:0/1373392873 wait complete. 2024-03-04T09:36:29.191 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:32:57.159681+0000', 'modified': '2024-03-04T09:35:26.088222+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8214, 'mds_1': 6796}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6796': {'gid': 6796, 'name': 'd', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.146:6837/2458070840', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 2458070840}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 2458070840}]}, 'join_fscid': -1, 'laggy_since': '2024-03-04T09:33:54.652601+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8214': {'gid': 8214, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2446627517', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2446627517}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2446627517}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8214, 'qdb_cluster': [8214]} 2024-03-04T09:36:29.192 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6796:up:stopping 2024-03-04T09:36:29.951 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-04T09:36:30.193 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:36:30.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:30.423+0000 7fa79be00640 1 Processor -- start 2024-03-04T09:36:30.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:30.423+0000 7fa79be00640 1 -- start start 2024-03-04T09:36:30.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:30.425+0000 7fa79be00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa79c06afb0 0x7fa79c06b410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:30.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:30.425+0000 7fa79be00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa79c0cf130 0x7fa79c0cf5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:30.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:30.425+0000 7fa79be00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa79c0d0910 con 0x7fa79c144c00 2024-03-04T09:36:30.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:30.425+0000 7fa79be00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa79c06b950 con 0x7fa79c0cf130 2024-03-04T09:36:30.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:30.425+0000 7fa79be00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa79c0d0bf0 con 0x7fa79c06afb0 2024-03-04T09:36:30.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:30.425+0000 7fa79b400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa79c0cf130 0x7fa79c0cf5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:30.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:30.425+0000 7fa79b400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa79c0cf130 0x7fa79c0cf5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:58486/0 (socket says 172.21.15.6:58486) 2024-03-04T09:36:30.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:30.425+0000 7fa79b400640 1 -- 172.21.15.6:0/678082417 learned_addr learned my addr 172.21.15.6:0/678082417 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:36:30.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:30.425+0000 7fa79a000640 1 --2- 172.21.15.6:0/678082417 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa79c06afb0 0x7fa79c06b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:30.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:30.426+0000 7fa79b400640 1 -- 172.21.15.6:0/678082417 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa79c06afb0 msgr2=0x7fa79c06b410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:36:30.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:30.426+0000 7fa79b400640 1 --2- 172.21.15.6:0/678082417 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa79c06afb0 0x7fa79c06b410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:30.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:30.426+0000 7fa79b400640 1 -- 172.21.15.6:0/678082417 >> v1:172.21.15.146:6789/0 conn(0x7fa79c144c00 legacy=0x7fa79c145280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:36:30.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:30.426+0000 7fa79b400640 1 -- 172.21.15.6:0/678082417 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa79c0d0ed0 con 0x7fa79c0cf130 2024-03-04T09:36:30.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:30.426+0000 7fa79a000640 1 --2- 172.21.15.6:0/678082417 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa79c06afb0 0x7fa79c06b410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:36:30.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:30.427+0000 7fa79b400640 1 --2- 172.21.15.6:0/678082417 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa79c0cf130 0x7fa79c0cf5b0 secure :-1 s=READY pgs=1805 cs=0 l=1 rev1=1 crypto rx=0x7fa794009e50 tx=0x7fa79402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db80bccf53d83420 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:36:30.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:30.427+0000 7fa799600640 1 -- 172.21.15.6:0/678082417 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa79402ca80 con 0x7fa79c0cf130 2024-03-04T09:36:30.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:30.427+0000 7fa799600640 1 -- 172.21.15.6:0/678082417 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa79402cc20 con 0x7fa79c0cf130 2024-03-04T09:36:30.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:30.427+0000 7fa799600640 1 -- 172.21.15.6:0/678082417 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7940343f0 con 0x7fa79c0cf130 2024-03-04T09:36:30.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:30.428+0000 7fa79be00640 1 -- 172.21.15.6:0/678082417 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa79c0cf130 msgr2=0x7fa79c0cf5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:30.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:30.428+0000 7fa79be00640 1 --2- 172.21.15.6:0/678082417 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa79c0cf130 0x7fa79c0cf5b0 secure :-1 s=READY pgs=1805 cs=0 l=1 rev1=1 crypto rx=0x7fa794009e50 tx=0x7fa79402bee0 comp rx=0 tx=0).stop 2024-03-04T09:36:30.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:30.428+0000 7fa79be00640 1 -- 172.21.15.6:0/678082417 shutdown_connections 2024-03-04T09:36:30.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:30.428+0000 7fa79be00640 1 --2- 172.21.15.6:0/678082417 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa79c0cf130 0x7fa79c0cf5b0 unknown :-1 s=CLOSED pgs=1805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:30.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:30.428+0000 7fa79be00640 1 --2- 172.21.15.6:0/678082417 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa79c06afb0 0x7fa79c06b410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:30.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:30.428+0000 7fa79be00640 1 -- 172.21.15.6:0/678082417 >> 172.21.15.6:0/678082417 conn(0x7fa79c06c0c0 msgr2=0x7fa79c06c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:36:30.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:30.428+0000 7fa79be00640 1 -- 172.21.15.6:0/678082417 shutdown_connections 2024-03-04T09:36:30.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:30.428+0000 7fa79be00640 1 -- 172.21.15.6:0/678082417 wait complete. 2024-03-04T09:36:30.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:30.430+0000 7fa79be00640 1 Processor -- start 2024-03-04T09:36:30.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:30.430+0000 7fa79be00640 1 -- start start 2024-03-04T09:36:30.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:30.431+0000 7fa79be00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa79c06afb0 0x7fa79c15b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:30.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:30.431+0000 7fa79be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa79c0cf130 0x7fa79c0b4da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:30.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:30.431+0000 7fa79be00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa79c144c00 0x7fa79c0afaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:30.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:30.431+0000 7fa79be00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa79c144fe0 con 0x7fa79c0cf130 2024-03-04T09:36:30.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:30.431+0000 7fa79be00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa79c0d1490 con 0x7fa79c06afb0 2024-03-04T09:36:30.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:30.431+0000 7fa79be00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa79c145990 con 0x7fa79c144c00 2024-03-04T09:36:30.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:30.431+0000 7fa79a000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa79c0cf130 0x7fa79c0b4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:30.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:30.431+0000 7fa79a000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa79c0cf130 0x7fa79c0b4da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:58498/0 (socket says 172.21.15.6:58498) 2024-03-04T09:36:30.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:30.431+0000 7fa79a000640 1 -- 172.21.15.6:0/3995207219 learned_addr learned my addr 172.21.15.6:0/3995207219 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:36:30.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:30.431+0000 7fa79aa00640 1 --2- 172.21.15.6:0/3995207219 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa79c06afb0 0x7fa79c15b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:30.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:30.431+0000 7fa79b400640 1 --2- 172.21.15.6:0/3995207219 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa79c144c00 0x7fa79c0afaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:30.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:30.432+0000 7fa79a000640 1 -- 172.21.15.6:0/3995207219 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa79c144c00 msgr2=0x7fa79c0afaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:30.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:30.432+0000 7fa79a000640 1 --2- 172.21.15.6:0/3995207219 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa79c144c00 0x7fa79c0afaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:30.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:30.432+0000 7fa79a000640 1 -- 172.21.15.6:0/3995207219 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa79c06afb0 msgr2=0x7fa79c15b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:30.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:30.432+0000 7fa79a000640 1 --2- 172.21.15.6:0/3995207219 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa79c06afb0 0x7fa79c15b660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:30.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:30.432+0000 7fa79a000640 1 -- 172.21.15.6:0/3995207219 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa79c0b3d00 con 0x7fa79c0cf130 2024-03-04T09:36:30.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:30.432+0000 7fa79a000640 1 --2- 172.21.15.6:0/3995207219 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa79c0cf130 0x7fa79c0b4da0 secure :-1 s=READY pgs=1806 cs=0 l=1 rev1=1 crypto rx=0x7fa78800c820 tx=0x7fa78800cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:36:30.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:30.432+0000 7fa792a00640 1 -- 172.21.15.6:0/3995207219 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa788015070 con 0x7fa79c0cf130 2024-03-04T09:36:30.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:30.432+0000 7fa792a00640 1 -- 172.21.15.6:0/3995207219 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7880079d0 con 0x7fa79c0cf130 2024-03-04T09:36:30.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:30.432+0000 7fa792a00640 1 -- 172.21.15.6:0/3995207219 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa788007dc0 con 0x7fa79c0cf130 2024-03-04T09:36:30.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:30.432+0000 7fa79be00640 1 -- 172.21.15.6:0/3995207219 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa79c0b3fe0 con 0x7fa79c0cf130 2024-03-04T09:36:30.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:30.433+0000 7fa79be00640 1 -- 172.21.15.6:0/3995207219 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa79c0b45a0 con 0x7fa79c0cf130 2024-03-04T09:36:30.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:30.434+0000 7fa79be00640 1 -- 172.21.15.6:0/3995207219 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa79c04f330 con 0x7fa79c0cf130 2024-03-04T09:36:30.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:30.435+0000 7fa792a00640 1 -- 172.21.15.6:0/3995207219 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa788018070 con 0x7fa79c0cf130 2024-03-04T09:36:30.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:30.435+0000 7fa792a00640 1 --2- 172.21.15.6:0/3995207219 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa77c06b4d0 0x7fa77c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:30.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:30.436+0000 7fa792a00640 1 -- 172.21.15.6:0/3995207219 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fa788089dd0 con 0x7fa79c0cf130 2024-03-04T09:36:30.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:30.436+0000 7fa79aa00640 1 --2- 172.21.15.6:0/3995207219 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa77c06b4d0 0x7fa77c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:30.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:30.437+0000 7fa79aa00640 1 --2- 172.21.15.6:0/3995207219 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa77c06b4d0 0x7fa77c06d990 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7fa78c004ea0 tx=0x7fa78c009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:36:30.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:30.440+0000 7fa792a00640 1 -- 172.21.15.6:0/3995207219 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa788058c00 con 0x7fa79c0cf130 2024-03-04T09:36:30.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:30.678+0000 7fa79be00640 1 -- 172.21.15.6:0/3995207219 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa79c06c6b0 con 0x7fa79c0cf130 2024-03-04T09:36:30.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:30.679+0000 7fa792a00640 1 -- 172.21.15.6:0/3995207219 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3464 (secure 0 0 0) 0x7fa788058500 con 0x7fa79c0cf130 2024-03-04T09:36:30.676 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:36:30.676 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:35:26.088222+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"laggy_since":"2024-03-04T09:33:54.652601+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":9}]} 2024-03-04T09:36:30.676 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 101 2024-03-04T09:36:30.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:30.682+0000 7fa79be00640 1 -- 172.21.15.6:0/3995207219 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa77c06b4d0 msgr2=0x7fa77c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:30.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:30.682+0000 7fa79be00640 1 --2- 172.21.15.6:0/3995207219 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa77c06b4d0 0x7fa77c06d990 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7fa78c004ea0 tx=0x7fa78c009040 comp rx=0 tx=0).stop 2024-03-04T09:36:30.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:30.682+0000 7fa79be00640 1 -- 172.21.15.6:0/3995207219 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa79c0cf130 msgr2=0x7fa79c0b4da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:30.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:30.682+0000 7fa79be00640 1 --2- 172.21.15.6:0/3995207219 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa79c0cf130 0x7fa79c0b4da0 secure :-1 s=READY pgs=1806 cs=0 l=1 rev1=1 crypto rx=0x7fa78800c820 tx=0x7fa78800cce0 comp rx=0 tx=0).stop 2024-03-04T09:36:30.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:30.682+0000 7fa79be00640 1 -- 172.21.15.6:0/3995207219 shutdown_connections 2024-03-04T09:36:30.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:30.682+0000 7fa79be00640 1 --2- 172.21.15.6:0/3995207219 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fa77c06b4d0 0x7fa77c06d990 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:30.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:30.682+0000 7fa79be00640 1 --2- 172.21.15.6:0/3995207219 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa79c144c00 0x7fa79c0afaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:30.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:30.682+0000 7fa79be00640 1 --2- 172.21.15.6:0/3995207219 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa79c0cf130 0x7fa79c0b4da0 unknown :-1 s=CLOSED pgs=1806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:30.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:30.682+0000 7fa79be00640 1 --2- 172.21.15.6:0/3995207219 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa79c06afb0 0x7fa79c15b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:30.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:30.682+0000 7fa79be00640 1 -- 172.21.15.6:0/3995207219 >> 172.21.15.6:0/3995207219 conn(0x7fa79c06c0c0 msgr2=0x7fa79c0cdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:36:30.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:30.682+0000 7fa79be00640 1 -- 172.21.15.6:0/3995207219 shutdown_connections 2024-03-04T09:36:30.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:30.682+0000 7fa79be00640 1 -- 172.21.15.6:0/3995207219 wait complete. 2024-03-04T09:36:30.901 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:32:57.159681+0000', 'modified': '2024-03-04T09:35:26.088222+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8214, 'mds_1': 6796}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6796': {'gid': 6796, 'name': 'd', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.146:6837/2458070840', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 2458070840}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 2458070840}]}, 'join_fscid': -1, 'laggy_since': '2024-03-04T09:33:54.652601+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8214': {'gid': 8214, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2446627517', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2446627517}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2446627517}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8214, 'qdb_cluster': [8214]} 2024-03-04T09:36:30.901 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6796:up:stopping 2024-03-04T09:36:31.902 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:36:31.953 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-04T09:36:32.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:32.128+0000 7f6e6b200640 1 Processor -- start 2024-03-04T09:36:32.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:32.128+0000 7f6e6b200640 1 -- start start 2024-03-04T09:36:32.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:32.131+0000 7f6e6b200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6e6415e0b0 0x7f6e640cf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:32.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:32.131+0000 7f6e6b200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f6e640d4a10 0x7f6e640d0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:32.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:32.131+0000 7f6e6b200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e640b3710 con 0x7f6e640d0c80 2024-03-04T09:36:32.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:32.131+0000 7f6e6b200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e640b09e0 con 0x7f6e640d4a10 2024-03-04T09:36:32.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:32.131+0000 7f6e6b200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e640b39f0 con 0x7f6e6415e0b0 2024-03-04T09:36:32.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:32.131+0000 7f6e69400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f6e640d4a10 0x7f6e640d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:32.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:32.131+0000 7f6e69400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f6e640d4a10 0x7f6e640d0740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:58502/0 (socket says 172.21.15.6:58502) 2024-03-04T09:36:32.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:32.131+0000 7f6e69400640 1 -- 172.21.15.6:0/2047923757 learned_addr learned my addr 172.21.15.6:0/2047923757 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:36:32.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:32.132+0000 7f6e69e00640 1 --2- 172.21.15.6:0/2047923757 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6e6415e0b0 0x7f6e640cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:32.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:32.132+0000 7f6e69400640 1 -- 172.21.15.6:0/2047923757 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6e6415e0b0 msgr2=0x7f6e640cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:36:32.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:32.132+0000 7f6e69400640 1 --2- 172.21.15.6:0/2047923757 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6e6415e0b0 0x7f6e640cf310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:32.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:32.132+0000 7f6e69400640 1 -- 172.21.15.6:0/2047923757 >> v1:172.21.15.6:6789/0 conn(0x7f6e640d0c80 legacy=0x7f6e640d1900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:32.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:32.132+0000 7f6e69400640 1 -- 172.21.15.6:0/2047923757 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e640b3cd0 con 0x7f6e640d4a10 2024-03-04T09:36:32.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:32.132+0000 7f6e68a00640 1 -- 172.21.15.6:0/2047923757 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3969372913 0 0) 0x7f6e640b3710 con 0x7f6e640d0c80 2024-03-04T09:36:32.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:32.133+0000 7f6e69400640 1 --2- 172.21.15.6:0/2047923757 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6e640d4a10 0x7f6e640d0740 secure :-1 s=READY pgs=1808 cs=0 l=1 rev1=1 crypto rx=0x7f6e54009870 tx=0x7f6e5402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dca26d49e4e3e832 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:36:32.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:32.134+0000 7f6e68a00640 1 -- 172.21.15.6:0/2047923757 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e54034070 con 0x7f6e640d4a10 2024-03-04T09:36:32.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:32.134+0000 7f6e68a00640 1 -- 172.21.15.6:0/2047923757 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e54030840 con 0x7f6e640d4a10 2024-03-04T09:36:32.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:32.134+0000 7f6e68a00640 1 -- 172.21.15.6:0/2047923757 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e54030c10 con 0x7f6e640d4a10 2024-03-04T09:36:32.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:32.134+0000 7f6e6b200640 1 -- 172.21.15.6:0/2047923757 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6e640d4a10 msgr2=0x7f6e640d0740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:32.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:32.134+0000 7f6e6b200640 1 --2- 172.21.15.6:0/2047923757 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6e640d4a10 0x7f6e640d0740 secure :-1 s=READY pgs=1808 cs=0 l=1 rev1=1 crypto rx=0x7f6e54009870 tx=0x7f6e5402b980 comp rx=0 tx=0).stop 2024-03-04T09:36:32.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:32.134+0000 7f6e6b200640 1 -- 172.21.15.6:0/2047923757 shutdown_connections 2024-03-04T09:36:32.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:32.134+0000 7f6e6b200640 1 --2- 172.21.15.6:0/2047923757 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6e640d4a10 0x7f6e640d0740 unknown :-1 s=CLOSED pgs=1808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:32.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:32.134+0000 7f6e6b200640 1 --2- 172.21.15.6:0/2047923757 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6e6415e0b0 0x7f6e640cf310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:32.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:32.134+0000 7f6e6b200640 1 -- 172.21.15.6:0/2047923757 >> 172.21.15.6:0/2047923757 conn(0x7f6e640d3d30 msgr2=0x7f6e640d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:36:32.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:32.134+0000 7f6e6b200640 1 -- 172.21.15.6:0/2047923757 shutdown_connections 2024-03-04T09:36:32.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:32.135+0000 7f6e6b200640 1 -- 172.21.15.6:0/2047923757 wait complete. 2024-03-04T09:36:32.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:32.136+0000 7f6e6b200640 1 Processor -- start 2024-03-04T09:36:32.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:32.136+0000 7f6e6b200640 1 -- start start 2024-03-04T09:36:32.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:32.137+0000 7f6e6b200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6e640d0c80 0x7f6e6415b960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:32.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:32.137+0000 7f6e6b200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6e640d4a10 0x7f6e6415bea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:32.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:32.137+0000 7f6e6b200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6e6415e0b0 0x7f6e6406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:32.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:32.137+0000 7f6e6b200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e640cfc60 con 0x7f6e6415e0b0 2024-03-04T09:36:32.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:32.137+0000 7f6e6b200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e6415e490 con 0x7f6e640d4a10 2024-03-04T09:36:32.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:32.137+0000 7f6e6b200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e640b54e0 con 0x7f6e640d0c80 2024-03-04T09:36:32.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:32.138+0000 7f6e6a800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6e6415e0b0 0x7f6e6406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:32.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:32.138+0000 7f6e6a800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6e6415e0b0 0x7f6e6406b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:58506/0 (socket says 172.21.15.6:58506) 2024-03-04T09:36:32.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:32.138+0000 7f6e6a800640 1 -- 172.21.15.6:0/3565624781 learned_addr learned my addr 172.21.15.6:0/3565624781 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:36:32.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:32.138+0000 7f6e69e00640 1 --2- 172.21.15.6:0/3565624781 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6e640d0c80 0x7f6e6415b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:32.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:32.138+0000 7f6e69400640 1 --2- 172.21.15.6:0/3565624781 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6e640d4a10 0x7f6e6415bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:32.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:32.138+0000 7f6e6a800640 1 -- 172.21.15.6:0/3565624781 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6e640d0c80 msgr2=0x7f6e6415b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:32.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:32.138+0000 7f6e6a800640 1 --2- 172.21.15.6:0/3565624781 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6e640d0c80 0x7f6e6415b960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:32.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:32.138+0000 7f6e6a800640 1 -- 172.21.15.6:0/3565624781 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6e640d4a10 msgr2=0x7f6e6415bea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:32.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:32.138+0000 7f6e6a800640 1 --2- 172.21.15.6:0/3565624781 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6e640d4a10 0x7f6e6415bea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:32.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:32.138+0000 7f6e6a800640 1 -- 172.21.15.6:0/3565624781 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e64160720 con 0x7f6e6415e0b0 2024-03-04T09:36:32.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:32.139+0000 7f6e6a800640 1 --2- 172.21.15.6:0/3565624781 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6e6415e0b0 0x7f6e6406b9a0 secure :-1 s=READY pgs=1809 cs=0 l=1 rev1=1 crypto rx=0x7f6e6000c790 tx=0x7f6e6000cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:36:32.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:32.139+0000 7f6e5e000640 1 -- 172.21.15.6:0/3565624781 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e60003da0 con 0x7f6e6415e0b0 2024-03-04T09:36:32.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:32.139+0000 7f6e5e000640 1 -- 172.21.15.6:0/3565624781 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e60005070 con 0x7f6e6415e0b0 2024-03-04T09:36:32.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:32.139+0000 7f6e5e000640 1 -- 172.21.15.6:0/3565624781 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e60005460 con 0x7f6e6415e0b0 2024-03-04T09:36:32.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:32.139+0000 7f6e6b200640 1 -- 172.21.15.6:0/3565624781 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e641f3970 con 0x7f6e6415e0b0 2024-03-04T09:36:32.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:32.139+0000 7f6e6b200640 1 -- 172.21.15.6:0/3565624781 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e641f3f30 con 0x7f6e6415e0b0 2024-03-04T09:36:32.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:32.141+0000 7f6e6b200640 1 -- 172.21.15.6:0/3565624781 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e6404f330 con 0x7f6e6415e0b0 2024-03-04T09:36:32.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:32.141+0000 7f6e5e000640 1 -- 172.21.15.6:0/3565624781 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6e60005600 con 0x7f6e6415e0b0 2024-03-04T09:36:32.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:32.143+0000 7f6e5e000640 1 --2- 172.21.15.6:0/3565624781 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f6e4806b4d0 0x7f6e4806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:32.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:32.143+0000 7f6e5e000640 1 -- 172.21.15.6:0/3565624781 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f6e6008a0b0 con 0x7f6e6415e0b0 2024-03-04T09:36:32.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:32.143+0000 7f6e69e00640 1 --2- 172.21.15.6:0/3565624781 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f6e4806b4d0 0x7f6e4806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:32.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:32.144+0000 7f6e69e00640 1 --2- 172.21.15.6:0/3565624781 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f6e4806b4d0 0x7f6e4806d990 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f6e58006fd0 tx=0x7f6e58008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:36:32.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:32.147+0000 7f6e5e000640 1 -- 172.21.15.6:0/3565624781 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6e60058f80 con 0x7f6e6415e0b0 2024-03-04T09:36:32.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:32.410+0000 7f6e6b200640 1 -- 172.21.15.6:0/3565624781 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6e64074920 con 0x7f6e6415e0b0 2024-03-04T09:36:32.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:32.411+0000 7f6e5e000640 1 -- 172.21.15.6:0/3565624781 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3464 (secure 0 0 0) 0x7f6e60058880 con 0x7f6e6415e0b0 2024-03-04T09:36:32.408 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:36:32.408 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:35:26.088222+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"laggy_since":"2024-03-04T09:33:54.652601+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":9}]} 2024-03-04T09:36:32.408 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 101 2024-03-04T09:36:32.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:32.413+0000 7f6e6b200640 1 -- 172.21.15.6:0/3565624781 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f6e4806b4d0 msgr2=0x7f6e4806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:32.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:32.413+0000 7f6e6b200640 1 --2- 172.21.15.6:0/3565624781 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f6e4806b4d0 0x7f6e4806d990 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f6e58006fd0 tx=0x7f6e58008040 comp rx=0 tx=0).stop 2024-03-04T09:36:32.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:32.413+0000 7f6e6b200640 1 -- 172.21.15.6:0/3565624781 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6e6415e0b0 msgr2=0x7f6e6406b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:32.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:32.413+0000 7f6e6b200640 1 --2- 172.21.15.6:0/3565624781 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6e6415e0b0 0x7f6e6406b9a0 secure :-1 s=READY pgs=1809 cs=0 l=1 rev1=1 crypto rx=0x7f6e6000c790 tx=0x7f6e6000cc50 comp rx=0 tx=0).stop 2024-03-04T09:36:32.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:32.413+0000 7f6e6b200640 1 -- 172.21.15.6:0/3565624781 shutdown_connections 2024-03-04T09:36:32.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:32.413+0000 7f6e6b200640 1 --2- 172.21.15.6:0/3565624781 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6e6415e0b0 0x7f6e6406b9a0 unknown :-1 s=CLOSED pgs=1809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:32.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:32.413+0000 7f6e6b200640 1 --2- 172.21.15.6:0/3565624781 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6e640d4a10 0x7f6e6415bea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:32.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:32.413+0000 7f6e6b200640 1 --2- 172.21.15.6:0/3565624781 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f6e4806b4d0 0x7f6e4806d990 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:32.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:32.413+0000 7f6e6b200640 1 --2- 172.21.15.6:0/3565624781 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6e640d0c80 0x7f6e6415b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:32.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:32.413+0000 7f6e6b200640 1 -- 172.21.15.6:0/3565624781 >> 172.21.15.6:0/3565624781 conn(0x7f6e640d3d30 msgr2=0x7f6e640cf070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:36:32.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:32.413+0000 7f6e6b200640 1 -- 172.21.15.6:0/3565624781 shutdown_connections 2024-03-04T09:36:32.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:32.413+0000 7f6e6b200640 1 -- 172.21.15.6:0/3565624781 wait complete. 2024-03-04T09:36:32.633 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:32:57.159681+0000', 'modified': '2024-03-04T09:35:26.088222+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8214, 'mds_1': 6796}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6796': {'gid': 6796, 'name': 'd', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.146:6837/2458070840', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 2458070840}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 2458070840}]}, 'join_fscid': -1, 'laggy_since': '2024-03-04T09:33:54.652601+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8214': {'gid': 8214, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2446627517', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2446627517}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2446627517}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8214, 'qdb_cluster': [8214]} 2024-03-04T09:36:32.633 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6796:up:stopping 2024-03-04T09:36:33.634 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:36:33.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:33.867+0000 7fb8a9200640 1 Processor -- start 2024-03-04T09:36:33.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:33.868+0000 7fb8a9200640 1 -- start start 2024-03-04T09:36:33.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:33.870+0000 7fb8a9200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fb8a415a930 0x7fb8a40ce4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:33.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:33.870+0000 7fb8a9200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fb8a40ce9f0 0x7fb8a40ace40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:33.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:33.870+0000 7fb8a9200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8a4071640 con 0x7fb8a415a550 2024-03-04T09:36:33.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:33.870+0000 7fb8a9200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8a40af6f0 con 0x7fb8a40ce9f0 2024-03-04T09:36:33.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:33.870+0000 7fb8a9200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8a40cee30 con 0x7fb8a415a930 2024-03-04T09:36:33.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:33.870+0000 7fb8a2800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fb8a40ce9f0 0x7fb8a40ace40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:33.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:33.870+0000 7fb8a2800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fb8a40ce9f0 0x7fb8a40ace40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:58516/0 (socket says 172.21.15.6:58516) 2024-03-04T09:36:33.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:33.870+0000 7fb8a2800640 1 -- 172.21.15.6:0/2399099387 learned_addr learned my addr 172.21.15.6:0/2399099387 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:36:33.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:33.871+0000 7fb8a1400640 1 --2- 172.21.15.6:0/2399099387 >> v2:172.21.15.146:3300/0 conn(0x7fb8a415a930 0x7fb8a40ce4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:33.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:33.871+0000 7fb8a2800640 1 -- 172.21.15.6:0/2399099387 >> v2:172.21.15.146:3300/0 conn(0x7fb8a415a930 msgr2=0x7fb8a40ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:36:33.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:33.871+0000 7fb8a2800640 1 --2- 172.21.15.6:0/2399099387 >> v2:172.21.15.146:3300/0 conn(0x7fb8a415a930 0x7fb8a40ce4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:33.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:33.871+0000 7fb8a2800640 1 -- 172.21.15.6:0/2399099387 >> v1:172.21.15.146:6789/0 conn(0x7fb8a415a550 legacy=0x7fb8a40aefe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:36:33.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:33.871+0000 7fb8a2800640 1 -- 172.21.15.6:0/2399099387 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8a4071920 con 0x7fb8a40ce9f0 2024-03-04T09:36:33.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:33.872+0000 7fb8a2800640 1 --2- 172.21.15.6:0/2399099387 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb8a40ce9f0 0x7fb8a40ace40 secure :-1 s=READY pgs=1810 cs=0 l=1 rev1=1 crypto rx=0x7fb8a4074890 tx=0x7fb89c02e180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9b4c0238c947007 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:36:33.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:33.872+0000 7fb8a0a00640 1 -- 172.21.15.6:0/2399099387 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb89c034070 con 0x7fb8a40ce9f0 2024-03-04T09:36:33.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:33.872+0000 7fb8a0a00640 1 -- 172.21.15.6:0/2399099387 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb89c038400 con 0x7fb8a40ce9f0 2024-03-04T09:36:33.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:33.872+0000 7fb8a0a00640 1 -- 172.21.15.6:0/2399099387 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb89c0387d0 con 0x7fb8a40ce9f0 2024-03-04T09:36:33.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:33.873+0000 7fb8a9200640 1 -- 172.21.15.6:0/2399099387 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb8a40ce9f0 msgr2=0x7fb8a40ace40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:33.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:33.873+0000 7fb8a9200640 1 --2- 172.21.15.6:0/2399099387 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb8a40ce9f0 0x7fb8a40ace40 secure :-1 s=READY pgs=1810 cs=0 l=1 rev1=1 crypto rx=0x7fb8a4074890 tx=0x7fb89c02e180 comp rx=0 tx=0).stop 2024-03-04T09:36:33.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:33.873+0000 7fb8a9200640 1 -- 172.21.15.6:0/2399099387 shutdown_connections 2024-03-04T09:36:33.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:33.873+0000 7fb8a9200640 1 --2- 172.21.15.6:0/2399099387 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb8a40ce9f0 0x7fb8a40ace40 unknown :-1 s=CLOSED pgs=1810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:33.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:33.873+0000 7fb8a9200640 1 --2- 172.21.15.6:0/2399099387 >> v2:172.21.15.146:3300/0 conn(0x7fb8a415a930 0x7fb8a40ce4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:33.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:33.873+0000 7fb8a9200640 1 -- 172.21.15.6:0/2399099387 >> 172.21.15.6:0/2399099387 conn(0x7fb8a406afb0 msgr2=0x7fb8a406b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:36:33.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:33.873+0000 7fb8a9200640 1 -- 172.21.15.6:0/2399099387 shutdown_connections 2024-03-04T09:36:33.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:33.873+0000 7fb8a9200640 1 -- 172.21.15.6:0/2399099387 wait complete. 2024-03-04T09:36:33.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:33.875+0000 7fb8a9200640 1 Processor -- start 2024-03-04T09:36:33.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:33.875+0000 7fb8a9200640 1 -- start start 2024-03-04T09:36:33.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:33.876+0000 7fb8a9200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb8a40ce9f0 0x7fb8a414c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:33.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:33.876+0000 7fb8a9200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb8a415a550 0x7fb8a41e4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:33.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:33.876+0000 7fb8a9200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb8a415a930 0x7fb8a41e4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:33.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:33.876+0000 7fb8a9200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8a40b5130 con 0x7fb8a415a550 2024-03-04T09:36:33.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:33.876+0000 7fb8a9200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8a4071ee0 con 0x7fb8a40ce9f0 2024-03-04T09:36:33.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:33.876+0000 7fb8a9200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8a40b52b0 con 0x7fb8a415a930 2024-03-04T09:36:33.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:33.876+0000 7fb8a1400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb8a415a550 0x7fb8a41e4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:33.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:33.876+0000 7fb8a1400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb8a415a550 0x7fb8a41e4960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:58530/0 (socket says 172.21.15.6:58530) 2024-03-04T09:36:33.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:33.876+0000 7fb8a1400640 1 -- 172.21.15.6:0/1628672607 learned_addr learned my addr 172.21.15.6:0/1628672607 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:36:33.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:33.877+0000 7fb8a1e00640 1 --2- 172.21.15.6:0/1628672607 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb8a40ce9f0 0x7fb8a414c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:33.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:33.877+0000 7fb8a2800640 1 --2- 172.21.15.6:0/1628672607 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb8a415a930 0x7fb8a41e4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:33.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:33.877+0000 7fb8a1400640 1 -- 172.21.15.6:0/1628672607 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb8a415a930 msgr2=0x7fb8a41e4ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:33.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:33.877+0000 7fb8a1400640 1 --2- 172.21.15.6:0/1628672607 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb8a415a930 0x7fb8a41e4ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:33.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:33.877+0000 7fb8a1400640 1 -- 172.21.15.6:0/1628672607 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb8a40ce9f0 msgr2=0x7fb8a414c6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:33.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:33.877+0000 7fb8a1400640 1 --2- 172.21.15.6:0/1628672607 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb8a40ce9f0 0x7fb8a414c6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:33.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:33.877+0000 7fb8a1400640 1 -- 172.21.15.6:0/1628672607 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8a41599c0 con 0x7fb8a415a550 2024-03-04T09:36:33.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:33.877+0000 7fb8a1400640 1 --2- 172.21.15.6:0/1628672607 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb8a415a550 0x7fb8a41e4960 secure :-1 s=READY pgs=1811 cs=0 l=1 rev1=1 crypto rx=0x7fb89000d800 tx=0x7fb89000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:36:33.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:33.877+0000 7fb89a000640 1 -- 172.21.15.6:0/1628672607 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb89000b6c0 con 0x7fb8a415a550 2024-03-04T09:36:33.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:33.877+0000 7fb89a000640 1 -- 172.21.15.6:0/1628672607 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb890019070 con 0x7fb8a415a550 2024-03-04T09:36:33.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:33.877+0000 7fb89a000640 1 -- 172.21.15.6:0/1628672607 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb89000ba80 con 0x7fb8a415a550 2024-03-04T09:36:33.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:33.877+0000 7fb8a9200640 1 -- 172.21.15.6:0/1628672607 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8a414cc00 con 0x7fb8a415a550 2024-03-04T09:36:33.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:33.878+0000 7fb8a9200640 1 -- 172.21.15.6:0/1628672607 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8a409b8d0 con 0x7fb8a415a550 2024-03-04T09:36:33.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:33.879+0000 7fb8a9200640 1 -- 172.21.15.6:0/1628672607 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8a4069510 con 0x7fb8a415a550 2024-03-04T09:36:33.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:33.880+0000 7fb89a000640 1 -- 172.21.15.6:0/1628672607 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb89000bc20 con 0x7fb8a415a550 2024-03-04T09:36:33.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:33.881+0000 7fb89a000640 1 --2- 172.21.15.6:0/1628672607 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb88406b4d0 0x7fb88406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:33.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:33.881+0000 7fb89a000640 1 -- 172.21.15.6:0/1628672607 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fb89008a700 con 0x7fb8a415a550 2024-03-04T09:36:33.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:33.881+0000 7fb8a1e00640 1 --2- 172.21.15.6:0/1628672607 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb88406b4d0 0x7fb88406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:33.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:33.882+0000 7fb8a1e00640 1 --2- 172.21.15.6:0/1628672607 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb88406b4d0 0x7fb88406d990 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fb894004ea0 tx=0x7fb894009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:36:33.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:33.885+0000 7fb89a000640 1 -- 172.21.15.6:0/1628672607 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb8a414cc00 con 0x7fb8a415a550 2024-03-04T09:36:33.955 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-04T09:36:34.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:34.154+0000 7fb8a9200640 1 -- 172.21.15.6:0/1628672607 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb8a404f3a0 con 0x7fb8a415a550 2024-03-04T09:36:34.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:34.155+0000 7fb89a000640 1 -- 172.21.15.6:0/1628672607 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3464 (secure 0 0 0) 0x7fb8a41599c0 con 0x7fb8a415a550 2024-03-04T09:36:34.153 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:36:34.153 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:35:26.088222+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"laggy_since":"2024-03-04T09:33:54.652601+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":9}]} 2024-03-04T09:36:34.153 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 101 2024-03-04T09:36:34.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:34.159+0000 7fb8a9200640 1 -- 172.21.15.6:0/1628672607 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb88406b4d0 msgr2=0x7fb88406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:34.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:34.159+0000 7fb8a9200640 1 --2- 172.21.15.6:0/1628672607 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb88406b4d0 0x7fb88406d990 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fb894004ea0 tx=0x7fb894009040 comp rx=0 tx=0).stop 2024-03-04T09:36:34.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:34.159+0000 7fb8a9200640 1 -- 172.21.15.6:0/1628672607 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb8a415a550 msgr2=0x7fb8a41e4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:34.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:34.159+0000 7fb8a9200640 1 --2- 172.21.15.6:0/1628672607 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb8a415a550 0x7fb8a41e4960 secure :-1 s=READY pgs=1811 cs=0 l=1 rev1=1 crypto rx=0x7fb89000d800 tx=0x7fb89000dcc0 comp rx=0 tx=0).stop 2024-03-04T09:36:34.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:34.159+0000 7fb8a9200640 1 -- 172.21.15.6:0/1628672607 shutdown_connections 2024-03-04T09:36:34.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:34.159+0000 7fb8a9200640 1 --2- 172.21.15.6:0/1628672607 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb88406b4d0 0x7fb88406d990 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:34.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:34.159+0000 7fb8a9200640 1 --2- 172.21.15.6:0/1628672607 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb8a415a930 0x7fb8a41e4ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:34.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:34.159+0000 7fb8a9200640 1 --2- 172.21.15.6:0/1628672607 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb8a415a550 0x7fb8a41e4960 unknown :-1 s=CLOSED pgs=1811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:34.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:34.159+0000 7fb8a9200640 1 --2- 172.21.15.6:0/1628672607 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb8a40ce9f0 0x7fb8a414c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:34.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:34.159+0000 7fb8a9200640 1 -- 172.21.15.6:0/1628672607 >> 172.21.15.6:0/1628672607 conn(0x7fb8a406afb0 msgr2=0x7fb8a40cdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:36:34.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:34.159+0000 7fb8a9200640 1 -- 172.21.15.6:0/1628672607 shutdown_connections 2024-03-04T09:36:34.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:34.159+0000 7fb8a9200640 1 -- 172.21.15.6:0/1628672607 wait complete. 2024-03-04T09:36:34.376 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:32:57.159681+0000', 'modified': '2024-03-04T09:35:26.088222+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8214, 'mds_1': 6796}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6796': {'gid': 6796, 'name': 'd', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.146:6837/2458070840', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 2458070840}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 2458070840}]}, 'join_fscid': -1, 'laggy_since': '2024-03-04T09:33:54.652601+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8214': {'gid': 8214, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2446627517', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2446627517}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2446627517}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8214, 'qdb_cluster': [8214]} 2024-03-04T09:36:34.376 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6796:up:stopping 2024-03-04T09:36:35.377 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:36:35.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:35.602+0000 7f25f2c00640 1 Processor -- start 2024-03-04T09:36:35.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:35.603+0000 7f25f2c00640 1 -- start start 2024-03-04T09:36:35.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:35.604+0000 7f25f2c00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f25ec161b40 0x7f25ec0cfd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:35.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:35.604+0000 7f25f2c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f25ec0d2930 0x7f25ec0b3740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:35.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:35.604+0000 7f25f2c00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25ec0cea50 con 0x7f25ec0b5060 2024-03-04T09:36:35.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:35.604+0000 7f25f2c00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25ec0cf900 con 0x7f25ec161b40 2024-03-04T09:36:35.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:35.604+0000 7f25f2c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25ec0cecf0 con 0x7f25ec0d2930 2024-03-04T09:36:35.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:35.605+0000 7f25e3e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f25ec0b5060 0x7f25ec0d2220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:55164/0 (socket says 172.21.15.6:55164) 2024-03-04T09:36:35.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:35.605+0000 7f25e3e00640 1 -- 172.21.15.6:0/2630825007 learned_addr learned my addr 172.21.15.6:0/2630825007 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:36:35.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:35.605+0000 7f25ebe00640 1 --2- 172.21.15.6:0/2630825007 >> v2:172.21.15.146:3300/0 conn(0x7f25ec161b40 0x7f25ec0cfd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:35.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:35.605+0000 7f25f0c00640 1 --2- 172.21.15.6:0/2630825007 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f25ec0d2930 0x7f25ec0b3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:35.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:35.605+0000 7f25eb400640 1 -- 172.21.15.6:0/2630825007 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4156888261 0 0) 0x7f25ec0cea50 con 0x7f25ec0b5060 2024-03-04T09:36:35.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:35.605+0000 7f25eb400640 1 -- 172.21.15.6:0/2630825007 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25d0003520 con 0x7f25ec0b5060 2024-03-04T09:36:35.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:35.605+0000 7f25eb400640 1 -- 172.21.15.6:0/2630825007 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2625678915 0 0) 0x7f25d0003520 con 0x7f25ec0b5060 2024-03-04T09:36:35.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:35.605+0000 7f25eb400640 1 -- 172.21.15.6:0/2630825007 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f25ec0d2930 msgr2=0x7f25ec0b3740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:36:35.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:35.606+0000 7f25eb400640 1 --2- 172.21.15.6:0/2630825007 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f25ec0d2930 0x7f25ec0b3740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:35.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:35.606+0000 7f25eb400640 1 -- 172.21.15.6:0/2630825007 >> v2:172.21.15.146:3300/0 conn(0x7f25ec161b40 msgr2=0x7f25ec0cfd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:36:35.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:35.606+0000 7f25eb400640 1 --2- 172.21.15.6:0/2630825007 >> v2:172.21.15.146:3300/0 conn(0x7f25ec161b40 0x7f25ec0cfd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:35.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:35.606+0000 7f25eb400640 1 -- 172.21.15.6:0/2630825007 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25ec0cefd0 con 0x7f25ec0b5060 2024-03-04T09:36:35.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:35.606+0000 7f25eb400640 1 -- 172.21.15.6:0/2630825007 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7f25d4001670 con 0x7f25ec0b5060 2024-03-04T09:36:35.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:35.606+0000 7f25eb400640 1 -- 172.21.15.6:0/2630825007 >> v1:172.21.15.6:6789/0 conn(0x7f25ec0b5060 legacy=0x7f25ec0d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:35.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:35.606+0000 7f25eb400640 1 --2- 172.21.15.6:0/2630825007 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f25d0003a20 0x7f25d0003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:35.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:35.606+0000 7f25eb400640 1 --2- 172.21.15.6:0/2630825007 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f25d0004800 0x7f25d0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:35.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:35.606+0000 7f25eb400640 1 --2- 172.21.15.6:0/2630825007 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f25ec0d2930 0x7f25d00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:35.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:35.606+0000 7f25eb400640 1 -- 172.21.15.6:0/2630825007 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25ec0cf900 con 0x7f25d0004800 2024-03-04T09:36:35.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:35.606+0000 7f25eb400640 1 -- 172.21.15.6:0/2630825007 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25ec0cecf0 con 0x7f25d0003a20 2024-03-04T09:36:35.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:35.606+0000 7f25eb400640 1 -- 172.21.15.6:0/2630825007 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25d000b9c0 con 0x7f25ec0d2930 2024-03-04T09:36:35.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:35.606+0000 7f25e3e00640 1 --2- 172.21.15.6:0/2630825007 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f25d0004800 0x7f25d0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:35.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:35.606+0000 7f25ebe00640 1 --2- 172.21.15.6:0/2630825007 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f25d0003a20 0x7f25d0003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:35.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:35.606+0000 7f25e3e00640 1 -- 172.21.15.6:0/2630825007 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f25ec0d2930 msgr2=0x7f25d00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:36:35.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:35.606+0000 7f25f0c00640 1 --2- 172.21.15.6:0/2630825007 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f25ec0d2930 0x7f25d00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:35.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:35.606+0000 7f25e3e00640 1 --2- 172.21.15.6:0/2630825007 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f25ec0d2930 0x7f25d00051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:35.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:35.606+0000 7f25e3e00640 1 -- 172.21.15.6:0/2630825007 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f25d0003a20 msgr2=0x7f25d0003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:36:35.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:35.606+0000 7f25e3e00640 1 --2- 172.21.15.6:0/2630825007 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f25d0003a20 0x7f25d0003e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:35.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:35.606+0000 7f25e3e00640 1 -- 172.21.15.6:0/2630825007 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25d0003520 con 0x7f25d0004800 2024-03-04T09:36:35.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:35.606+0000 7f25ebe00640 1 --2- 172.21.15.6:0/2630825007 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f25d0003a20 0x7f25d0003e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:36:35.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:35.607+0000 7f25e3e00640 1 --2- 172.21.15.6:0/2630825007 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f25d0004800 0x7f25d0004c80 secure :-1 s=READY pgs=1813 cs=0 l=1 rev1=1 crypto rx=0x7f25d400c240 tx=0x7f25d0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=184c7ff7ab994044 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:36:35.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:35.608+0000 7f25eb400640 1 -- 172.21.15.6:0/2630825007 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25d4007d50 con 0x7f25d0004800 2024-03-04T09:36:35.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:35.608+0000 7f25eb400640 1 -- 172.21.15.6:0/2630825007 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25d4030070 con 0x7f25d0004800 2024-03-04T09:36:35.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:35.608+0000 7f25eb400640 1 -- 172.21.15.6:0/2630825007 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25d402cd80 con 0x7f25d0004800 2024-03-04T09:36:35.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:35.608+0000 7f25f2c00640 1 -- 172.21.15.6:0/2630825007 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f25d0004800 msgr2=0x7f25d0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:35.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:35.608+0000 7f25f2c00640 1 --2- 172.21.15.6:0/2630825007 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f25d0004800 0x7f25d0004c80 secure :-1 s=READY pgs=1813 cs=0 l=1 rev1=1 crypto rx=0x7f25d400c240 tx=0x7f25d0005830 comp rx=0 tx=0).stop 2024-03-04T09:36:35.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:35.608+0000 7f25f0c00640 1 -- 172.21.15.6:0/2630825007 reap_dead start 2024-03-04T09:36:35.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:35.608+0000 7f25f2c00640 1 -- 172.21.15.6:0/2630825007 shutdown_connections 2024-03-04T09:36:35.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:35.608+0000 7f25f2c00640 1 -- 172.21.15.6:0/2630825007 >> 172.21.15.6:0/2630825007 conn(0x7f25ec09d220 msgr2=0x7f25ec0d3ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:36:35.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:35.609+0000 7f25f2c00640 1 -- 172.21.15.6:0/2630825007 shutdown_connections 2024-03-04T09:36:35.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:35.609+0000 7f25f2c00640 1 -- 172.21.15.6:0/2630825007 wait complete. 2024-03-04T09:36:35.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:35.611+0000 7f25f2c00640 1 Processor -- start 2024-03-04T09:36:35.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:35.612+0000 7f25f2c00640 1 -- start start 2024-03-04T09:36:35.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:35.613+0000 7f25f2c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f25d0004800 0x7f25ec15e790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:35.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:35.613+0000 7f25f2c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f25ec0cb6e0 0x7f25ec15ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:35.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:35.613+0000 7f25f2c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f25ec14d520 0x7f25ec15f210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:35.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:35.613+0000 7f25f2c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25ec0b2ff0 con 0x7f25ec14d520 2024-03-04T09:36:35.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:35.613+0000 7f25f2c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25ec0cf590 con 0x7f25ec0cb6e0 2024-03-04T09:36:35.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:35.613+0000 7f25f2c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25ec0b3170 con 0x7f25d0004800 2024-03-04T09:36:35.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:35.614+0000 7f25f0c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f25ec14d520 0x7f25ec15f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:35.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:35.614+0000 7f25f0c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f25ec14d520 0x7f25ec15f210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:58546/0 (socket says 172.21.15.6:58546) 2024-03-04T09:36:35.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:35.614+0000 7f25f0c00640 1 -- 172.21.15.6:0/220615709 learned_addr learned my addr 172.21.15.6:0/220615709 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:36:35.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:35.614+0000 7f25e3e00640 1 --2- 172.21.15.6:0/220615709 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f25ec0cb6e0 0x7f25ec15ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:35.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:35.614+0000 7f25ebe00640 1 --2- 172.21.15.6:0/220615709 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f25d0004800 0x7f25ec15e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:35.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:35.614+0000 7f25f0c00640 1 -- 172.21.15.6:0/220615709 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f25d0004800 msgr2=0x7f25ec15e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:35.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:35.614+0000 7f25f0c00640 1 --2- 172.21.15.6:0/220615709 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f25d0004800 0x7f25ec15e790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:35.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:35.614+0000 7f25f0c00640 1 -- 172.21.15.6:0/220615709 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f25ec0cb6e0 msgr2=0x7f25ec15ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:35.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:35.614+0000 7f25f0c00640 1 --2- 172.21.15.6:0/220615709 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f25ec0cb6e0 0x7f25ec15ecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:35.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:35.614+0000 7f25f0c00640 1 -- 172.21.15.6:0/220615709 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25ec0b5970 con 0x7f25ec14d520 2024-03-04T09:36:35.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:35.614+0000 7f25e3e00640 1 --2- 172.21.15.6:0/220615709 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f25ec0cb6e0 0x7f25ec15ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T09:36:35.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:35.615+0000 7f25f0c00640 1 --2- 172.21.15.6:0/220615709 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f25ec14d520 0x7f25ec15f210 secure :-1 s=READY pgs=1814 cs=0 l=1 rev1=1 crypto rx=0x7f25ec06e1e0 tx=0x7f25e400f4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:36:35.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:35.615+0000 7f25e8c00640 1 -- 172.21.15.6:0/220615709 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25e401c070 con 0x7f25ec14d520 2024-03-04T09:36:35.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:35.615+0000 7f25e8c00640 1 -- 172.21.15.6:0/220615709 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25e4017400 con 0x7f25ec14d520 2024-03-04T09:36:35.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:35.615+0000 7f25f2c00640 1 -- 172.21.15.6:0/220615709 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25ec0b5c50 con 0x7f25ec14d520 2024-03-04T09:36:35.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:35.615+0000 7f25e8c00640 1 -- 172.21.15.6:0/220615709 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25e40177f0 con 0x7f25ec14d520 2024-03-04T09:36:35.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:35.615+0000 7f25f2c00640 1 -- 172.21.15.6:0/220615709 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25ec0ce0e0 con 0x7f25ec14d520 2024-03-04T09:36:35.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:35.616+0000 7f25f2c00640 1 -- 172.21.15.6:0/220615709 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25ec04f3a0 con 0x7f25ec14d520 2024-03-04T09:36:35.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:35.617+0000 7f25e8c00640 1 -- 172.21.15.6:0/220615709 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f25e401f070 con 0x7f25ec14d520 2024-03-04T09:36:35.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:35.618+0000 7f25e8c00640 1 --2- 172.21.15.6:0/220615709 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f25c806b500 0x7f25c806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:35.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:35.618+0000 7f25e8c00640 1 -- 172.21.15.6:0/220615709 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f25e408a180 con 0x7f25ec14d520 2024-03-04T09:36:35.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:35.619+0000 7f25ebe00640 1 --2- 172.21.15.6:0/220615709 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f25c806b500 0x7f25c806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:35.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:35.620+0000 7f25ebe00640 1 --2- 172.21.15.6:0/220615709 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f25c806b500 0x7f25c806d9c0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f25dc009b20 tx=0x7f25dc008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:36:35.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:35.623+0000 7f25e8c00640 1 -- 172.21.15.6:0/220615709 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f25e4012070 con 0x7f25ec14d520 2024-03-04T09:36:35.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:35.951+0000 7f25f2c00640 1 -- 172.21.15.6:0/220615709 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f25ec0d1df0 con 0x7f25ec14d520 2024-03-04T09:36:35.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:35.952+0000 7f25e8c00640 1 -- 172.21.15.6:0/220615709 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3464 (secure 0 0 0) 0x7f25e4058a00 con 0x7f25ec14d520 2024-03-04T09:36:35.950 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:36:35.950 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:35:26.088222+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"laggy_since":"2024-03-04T09:33:54.652601+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":9}]} 2024-03-04T09:36:35.950 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 101 2024-03-04T09:36:35.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:35.954+0000 7f25f2c00640 1 -- 172.21.15.6:0/220615709 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f25c806b500 msgr2=0x7f25c806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:35.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:35.955+0000 7f25f2c00640 1 --2- 172.21.15.6:0/220615709 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f25c806b500 0x7f25c806d9c0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f25dc009b20 tx=0x7f25dc008040 comp rx=0 tx=0).stop 2024-03-04T09:36:35.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:35.955+0000 7f25f2c00640 1 -- 172.21.15.6:0/220615709 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f25ec14d520 msgr2=0x7f25ec15f210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:35.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:35.955+0000 7f25f2c00640 1 --2- 172.21.15.6:0/220615709 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f25ec14d520 0x7f25ec15f210 secure :-1 s=READY pgs=1814 cs=0 l=1 rev1=1 crypto rx=0x7f25ec06e1e0 tx=0x7f25e400f4a0 comp rx=0 tx=0).stop 2024-03-04T09:36:35.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:35.955+0000 7f25f2c00640 1 -- 172.21.15.6:0/220615709 shutdown_connections 2024-03-04T09:36:35.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:35.955+0000 7f25f2c00640 1 --2- 172.21.15.6:0/220615709 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f25ec14d520 0x7f25ec15f210 unknown :-1 s=CLOSED pgs=1814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:35.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:35.955+0000 7f25f2c00640 1 --2- 172.21.15.6:0/220615709 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f25ec0cb6e0 0x7f25ec15ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:35.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:35.955+0000 7f25f2c00640 1 --2- 172.21.15.6:0/220615709 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f25c806b500 0x7f25c806d9c0 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:35.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:35.955+0000 7f25f2c00640 1 --2- 172.21.15.6:0/220615709 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f25d0004800 0x7f25ec15e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:35.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:35.955+0000 7f25f2c00640 1 -- 172.21.15.6:0/220615709 >> 172.21.15.6:0/220615709 conn(0x7f25ec09d220 msgr2=0x7f25ec06bb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:36:35.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:35.955+0000 7f25f2c00640 1 -- 172.21.15.6:0/220615709 shutdown_connections 2024-03-04T09:36:35.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:35.955+0000 7f25f2c00640 1 -- 172.21.15.6:0/220615709 wait complete. 2024-03-04T09:36:35.956 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-04T09:36:36.174 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:32:57.159681+0000', 'modified': '2024-03-04T09:35:26.088222+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8214, 'mds_1': 6796}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6796': {'gid': 6796, 'name': 'd', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.146:6837/2458070840', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 2458070840}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 2458070840}]}, 'join_fscid': -1, 'laggy_since': '2024-03-04T09:33:54.652601+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8214': {'gid': 8214, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2446627517', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2446627517}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2446627517}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8214, 'qdb_cluster': [8214]} 2024-03-04T09:36:36.174 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6796:up:stopping 2024-03-04T09:36:37.175 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:36:37.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:37.410+0000 7ff2cda00640 1 Processor -- start 2024-03-04T09:36:37.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:37.410+0000 7ff2cda00640 1 -- start start 2024-03-04T09:36:37.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:37.412+0000 7ff2cda00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ff2c80b2340 0x7ff2c8168cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:37.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:37.412+0000 7ff2cda00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2c80d1120 con 0x7ff2c815dec0 2024-03-04T09:36:37.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:37.412+0000 7ff2cda00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2c80d1400 con 0x7ff2c815e2a0 2024-03-04T09:36:37.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:37.412+0000 7ff2cda00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2c80b0ea0 con 0x7ff2c80b2340 2024-03-04T09:36:37.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:37.412+0000 7ff2c7e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7ff2c815dec0 0x7ff2c80ce810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:55180/0 (socket says 172.21.15.6:55180) 2024-03-04T09:36:37.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:37.412+0000 7ff2c7e00640 1 -- 172.21.15.6:0/3431169232 learned_addr learned my addr 172.21.15.6:0/3431169232 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:36:37.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:37.413+0000 7ff2cd000640 1 --2- 172.21.15.6:0/3431169232 >> v2:172.21.15.146:3300/0 conn(0x7ff2c80b2340 0x7ff2c8168cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:37.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:37.413+0000 7ff2c6a00640 1 -- 172.21.15.6:0/3431169232 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1607359917 0 0) 0x7ff2c80d1120 con 0x7ff2c815dec0 2024-03-04T09:36:37.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:37.413+0000 7ff2c6a00640 1 -- 172.21.15.6:0/3431169232 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2ac003520 con 0x7ff2c815dec0 2024-03-04T09:36:37.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:37.413+0000 7ff2c6a00640 1 -- 172.21.15.6:0/3431169232 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 636906010 0 0) 0x7ff2ac003520 con 0x7ff2c815dec0 2024-03-04T09:36:37.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:37.413+0000 7ff2c6a00640 1 -- 172.21.15.6:0/3431169232 >> v2:172.21.15.146:3300/0 conn(0x7ff2c80b2340 msgr2=0x7ff2c8168cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:36:37.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:37.413+0000 7ff2c6a00640 1 --2- 172.21.15.6:0/3431169232 >> v2:172.21.15.146:3300/0 conn(0x7ff2c80b2340 0x7ff2c8168cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:37.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:37.413+0000 7ff2c6a00640 1 -- 172.21.15.6:0/3431169232 >> v1:172.21.15.146:6789/0 conn(0x7ff2c815e2a0 legacy=0x7ff2c80cef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:37.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:37.413+0000 7ff2c6a00640 1 -- 172.21.15.6:0/3431169232 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2c80b0b30 con 0x7ff2c815dec0 2024-03-04T09:36:37.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:37.413+0000 7ff2c6a00640 1 -- 172.21.15.6:0/3431169232 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7ff2b8001670 con 0x7ff2c815dec0 2024-03-04T09:36:37.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:37.413+0000 7ff2c6a00640 1 -- 172.21.15.6:0/3431169232 >> v1:172.21.15.6:6789/0 conn(0x7ff2c815dec0 legacy=0x7ff2c80ce810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:37.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:37.413+0000 7ff2c6a00640 1 --2- 172.21.15.6:0/3431169232 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff2ac003a00 0x7ff2ac003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:37.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:37.413+0000 7ff2c6a00640 1 --2- 172.21.15.6:0/3431169232 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff2ac004800 0x7ff2ac004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:37.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:37.414+0000 7ff2c6a00640 1 --2- 172.21.15.6:0/3431169232 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff2ac0051c0 0x7ff2ac00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:37.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:37.414+0000 7ff2c6a00640 1 -- 172.21.15.6:0/3431169232 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2c80b0ea0 con 0x7ff2ac004800 2024-03-04T09:36:37.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:37.414+0000 7ff2c6a00640 1 -- 172.21.15.6:0/3431169232 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2ac00c0c0 con 0x7ff2ac003a00 2024-03-04T09:36:37.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:37.414+0000 7ff2c6a00640 1 -- 172.21.15.6:0/3431169232 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2ac00c2c0 con 0x7ff2ac0051c0 2024-03-04T09:36:37.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:37.414+0000 7ff2c7400640 1 --2- 172.21.15.6:0/3431169232 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff2ac004800 0x7ff2ac004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:37.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:37.414+0000 7ff2c7e00640 1 --2- 172.21.15.6:0/3431169232 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff2ac003a00 0x7ff2ac003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:37.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:37.414+0000 7ff2c7400640 1 -- 172.21.15.6:0/3431169232 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff2ac0051c0 msgr2=0x7ff2ac00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:36:37.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:37.414+0000 7ff2cd000640 1 --2- 172.21.15.6:0/3431169232 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff2ac0051c0 0x7ff2ac00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:37.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:37.414+0000 7ff2c7400640 1 --2- 172.21.15.6:0/3431169232 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff2ac0051c0 0x7ff2ac00b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:37.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:37.414+0000 7ff2c7400640 1 -- 172.21.15.6:0/3431169232 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff2ac003a00 msgr2=0x7ff2ac003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:36:37.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:37.414+0000 7ff2c7400640 1 --2- 172.21.15.6:0/3431169232 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff2ac003a00 0x7ff2ac003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:37.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:37.414+0000 7ff2c7400640 1 -- 172.21.15.6:0/3431169232 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2ac003520 con 0x7ff2ac004800 2024-03-04T09:36:37.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:37.414+0000 7ff2cd000640 1 -- 172.21.15.6:0/3431169232 reap_dead start 2024-03-04T09:36:37.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:37.415+0000 7ff2c7400640 1 --2- 172.21.15.6:0/3431169232 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff2ac004800 0x7ff2ac004c80 secure :-1 s=READY pgs=1816 cs=0 l=1 rev1=1 crypto rx=0x7ff2b4001750 tx=0x7ff2ac005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eac9804cfaea403a server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:36:37.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:37.416+0000 7ff2c6a00640 1 -- 172.21.15.6:0/3431169232 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2b4001370 con 0x7ff2ac004800 2024-03-04T09:36:37.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:37.416+0000 7ff2c6a00640 1 -- 172.21.15.6:0/3431169232 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2b40052d0 con 0x7ff2ac004800 2024-03-04T09:36:37.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:37.416+0000 7ff2c6a00640 1 -- 172.21.15.6:0/3431169232 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2b4003830 con 0x7ff2ac004800 2024-03-04T09:36:37.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:37.416+0000 7ff2cda00640 1 -- 172.21.15.6:0/3431169232 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff2ac004800 msgr2=0x7ff2ac004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:37.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:37.416+0000 7ff2cda00640 1 --2- 172.21.15.6:0/3431169232 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff2ac004800 0x7ff2ac004c80 secure :-1 s=READY pgs=1816 cs=0 l=1 rev1=1 crypto rx=0x7ff2b4001750 tx=0x7ff2ac005830 comp rx=0 tx=0).stop 2024-03-04T09:36:37.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:37.416+0000 7ff2cda00640 1 -- 172.21.15.6:0/3431169232 shutdown_connections 2024-03-04T09:36:37.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:37.416+0000 7ff2cda00640 1 --2- 172.21.15.6:0/3431169232 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff2ac004800 0x7ff2ac004c80 unknown :-1 s=CLOSED pgs=1816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:37.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:37.416+0000 7ff2cda00640 1 -- 172.21.15.6:0/3431169232 >> 172.21.15.6:0/3431169232 conn(0x7ff2c80d3d40 msgr2=0x7ff2c80d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:36:37.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:37.416+0000 7ff2cda00640 1 -- 172.21.15.6:0/3431169232 shutdown_connections 2024-03-04T09:36:37.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:37.417+0000 7ff2cda00640 1 -- 172.21.15.6:0/3431169232 wait complete. 2024-03-04T09:36:37.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:37.418+0000 7ff2cda00640 1 Processor -- start 2024-03-04T09:36:37.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:37.418+0000 7ff2cda00640 1 -- start start 2024-03-04T09:36:37.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:37.419+0000 7ff2cda00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff2ac004800 0x7ff2c81f3a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:37.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:37.419+0000 7ff2cda00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff2c814d6d0 0x7ff2c806c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:37.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:37.419+0000 7ff2cda00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff2c814db50 0x7ff2c806cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:37.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:37.419+0000 7ff2cda00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2c80b14b0 con 0x7ff2c814d6d0 2024-03-04T09:36:37.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:37.419+0000 7ff2cda00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2c80b1330 con 0x7ff2ac004800 2024-03-04T09:36:37.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:37.419+0000 7ff2cda00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2c80b1630 con 0x7ff2c814db50 2024-03-04T09:36:37.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:37.420+0000 7ff2c7400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff2c814d6d0 0x7ff2c806c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:37.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:37.420+0000 7ff2c7400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff2c814d6d0 0x7ff2c806c9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:58564/0 (socket says 172.21.15.6:58564) 2024-03-04T09:36:37.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:37.420+0000 7ff2c7400640 1 -- 172.21.15.6:0/2208951995 learned_addr learned my addr 172.21.15.6:0/2208951995 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:36:37.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:37.420+0000 7ff2c7e00640 1 --2- 172.21.15.6:0/2208951995 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff2ac004800 0x7ff2c81f3a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:37.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:37.420+0000 7ff2cd000640 1 --2- 172.21.15.6:0/2208951995 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff2c814db50 0x7ff2c806cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:37.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:37.420+0000 7ff2c7400640 1 -- 172.21.15.6:0/2208951995 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff2c814db50 msgr2=0x7ff2c806cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:37.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:37.420+0000 7ff2c7400640 1 --2- 172.21.15.6:0/2208951995 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff2c814db50 0x7ff2c806cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:37.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:37.420+0000 7ff2c7400640 1 -- 172.21.15.6:0/2208951995 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff2ac004800 msgr2=0x7ff2c81f3a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:37.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:37.420+0000 7ff2c7400640 1 --2- 172.21.15.6:0/2208951995 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff2ac004800 0x7ff2c81f3a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:37.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:37.420+0000 7ff2c7400640 1 -- 172.21.15.6:0/2208951995 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2c81f3f80 con 0x7ff2c814d6d0 2024-03-04T09:36:37.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:37.420+0000 7ff2c7e00640 1 --2- 172.21.15.6:0/2208951995 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff2ac004800 0x7ff2c81f3a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:36:37.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:37.420+0000 7ff2cd000640 1 --2- 172.21.15.6:0/2208951995 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff2c814db50 0x7ff2c806cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:36:37.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:37.421+0000 7ff2c7400640 1 --2- 172.21.15.6:0/2208951995 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff2c814d6d0 0x7ff2c806c9b0 secure :-1 s=READY pgs=1817 cs=0 l=1 rev1=1 crypto rx=0x7ff2b4005040 tx=0x7ff2b4001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:36:37.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:37.421+0000 7ff2bfe00640 1 -- 172.21.15.6:0/2208951995 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2b4041020 con 0x7ff2c814d6d0 2024-03-04T09:36:37.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:37.421+0000 7ff2bfe00640 1 -- 172.21.15.6:0/2208951995 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2b4005480 con 0x7ff2c814d6d0 2024-03-04T09:36:37.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:37.421+0000 7ff2cda00640 1 -- 172.21.15.6:0/2208951995 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2c81f4220 con 0x7ff2c814d6d0 2024-03-04T09:36:37.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:37.421+0000 7ff2bfe00640 1 -- 172.21.15.6:0/2208951995 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2b4001370 con 0x7ff2c814d6d0 2024-03-04T09:36:37.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:37.421+0000 7ff2cda00640 1 -- 172.21.15.6:0/2208951995 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2c81f47e0 con 0x7ff2c814d6d0 2024-03-04T09:36:37.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:37.423+0000 7ff2cda00640 1 -- 172.21.15.6:0/2208951995 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2c804f330 con 0x7ff2c814d6d0 2024-03-04T09:36:37.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:37.423+0000 7ff2bfe00640 1 -- 172.21.15.6:0/2208951995 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff2b4044020 con 0x7ff2c814d6d0 2024-03-04T09:36:37.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:37.424+0000 7ff2bfe00640 1 --2- 172.21.15.6:0/2208951995 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff2a806b4d0 0x7ff2a806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:37.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:37.424+0000 7ff2bfe00640 1 -- 172.21.15.6:0/2208951995 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7ff2b40abec0 con 0x7ff2c814d6d0 2024-03-04T09:36:37.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:37.425+0000 7ff2c7e00640 1 --2- 172.21.15.6:0/2208951995 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff2a806b4d0 0x7ff2a806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:37.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:37.426+0000 7ff2c7e00640 1 --2- 172.21.15.6:0/2208951995 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff2a806b4d0 0x7ff2a806d990 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7ff2c80cfae0 tx=0x7ff2b8000f50 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:36:37.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:37.429+0000 7ff2bfe00640 1 -- 172.21.15.6:0/2208951995 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff2b404b030 con 0x7ff2c814d6d0 2024-03-04T09:36:37.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:37.841+0000 7ff2cda00640 1 -- 172.21.15.6:0/2208951995 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff2c80b1db0 con 0x7ff2c814d6d0 2024-03-04T09:36:37.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:37.842+0000 7ff2bfe00640 1 -- 172.21.15.6:0/2208951995 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3464 (secure 0 0 0) 0x7ff2b40303c0 con 0x7ff2c814d6d0 2024-03-04T09:36:37.839 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:36:37.840 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:35:26.088222+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"laggy_since":"2024-03-04T09:33:54.652601+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":9}]} 2024-03-04T09:36:37.840 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 101 2024-03-04T09:36:37.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:37.845+0000 7ff2cda00640 1 -- 172.21.15.6:0/2208951995 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff2a806b4d0 msgr2=0x7ff2a806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:37.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:37.845+0000 7ff2cda00640 1 --2- 172.21.15.6:0/2208951995 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff2a806b4d0 0x7ff2a806d990 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7ff2c80cfae0 tx=0x7ff2b8000f50 comp rx=0 tx=0).stop 2024-03-04T09:36:37.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:37.845+0000 7ff2cda00640 1 -- 172.21.15.6:0/2208951995 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff2c814d6d0 msgr2=0x7ff2c806c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:37.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:37.845+0000 7ff2cda00640 1 --2- 172.21.15.6:0/2208951995 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff2c814d6d0 0x7ff2c806c9b0 secure :-1 s=READY pgs=1817 cs=0 l=1 rev1=1 crypto rx=0x7ff2b4005040 tx=0x7ff2b4001120 comp rx=0 tx=0).stop 2024-03-04T09:36:37.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:37.846+0000 7ff2cda00640 1 -- 172.21.15.6:0/2208951995 shutdown_connections 2024-03-04T09:36:37.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:37.846+0000 7ff2cda00640 1 --2- 172.21.15.6:0/2208951995 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff2a806b4d0 0x7ff2a806d990 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:37.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:37.846+0000 7ff2cda00640 1 --2- 172.21.15.6:0/2208951995 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff2c814db50 0x7ff2c806cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:37.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:37.846+0000 7ff2cda00640 1 --2- 172.21.15.6:0/2208951995 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff2c814d6d0 0x7ff2c806c9b0 unknown :-1 s=CLOSED pgs=1817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:37.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:37.846+0000 7ff2cda00640 1 --2- 172.21.15.6:0/2208951995 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff2ac004800 0x7ff2c81f3a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:37.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:37.846+0000 7ff2cda00640 1 -- 172.21.15.6:0/2208951995 >> 172.21.15.6:0/2208951995 conn(0x7ff2c80d3d40 msgr2=0x7ff2c80cffa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:36:37.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:37.846+0000 7ff2cda00640 1 -- 172.21.15.6:0/2208951995 shutdown_connections 2024-03-04T09:36:37.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:37.846+0000 7ff2cda00640 1 -- 172.21.15.6:0/2208951995 wait complete. 2024-03-04T09:36:37.959 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-04T09:36:38.065 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:32:57.159681+0000', 'modified': '2024-03-04T09:35:26.088222+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8214, 'mds_1': 6796}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6796': {'gid': 6796, 'name': 'd', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.146:6837/2458070840', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 2458070840}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 2458070840}]}, 'join_fscid': -1, 'laggy_since': '2024-03-04T09:33:54.652601+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8214': {'gid': 8214, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2446627517', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2446627517}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2446627517}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8214, 'qdb_cluster': [8214]} 2024-03-04T09:36:38.065 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6796:up:stopping 2024-03-04T09:36:39.066 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:36:39.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:39.297+0000 7ff25b200640 1 Processor -- start 2024-03-04T09:36:39.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:39.298+0000 7ff25b200640 1 -- start start 2024-03-04T09:36:39.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:39.299+0000 7ff25b200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff25c144c00 0x7ff25c145280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:39.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:39.299+0000 7ff25b200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ff25c06b6c0 0x7ff25c0cf130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:39.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:39.299+0000 7ff25b200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff25c0d0910 con 0x7ff25c1457c0 2024-03-04T09:36:39.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:39.299+0000 7ff25b200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff25c0d0bf0 con 0x7ff25c06b6c0 2024-03-04T09:36:39.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:39.299+0000 7ff25b200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff25c0d0ed0 con 0x7ff25c144c00 2024-03-04T09:36:39.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:39.300+0000 7ff259400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7ff25c1457c0 0x7ff25c06afb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:53572/0 (socket says 172.21.15.6:53572) 2024-03-04T09:36:39.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:39.300+0000 7ff259400640 1 -- 172.21.15.6:0/288147675 learned_addr learned my addr 172.21.15.6:0/288147675 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:36:39.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:39.300+0000 7ff25a800640 1 --2- 172.21.15.6:0/288147675 >> v2:172.21.15.146:3300/0 conn(0x7ff25c06b6c0 0x7ff25c0cf130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:39.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:39.300+0000 7ff259e00640 1 --2- 172.21.15.6:0/288147675 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff25c144c00 0x7ff25c145280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:39.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:39.300+0000 7ff258a00640 1 -- 172.21.15.6:0/288147675 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3289286481 0 0) 0x7ff25c0d0910 con 0x7ff25c1457c0 2024-03-04T09:36:39.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:39.300+0000 7ff258a00640 1 -- 172.21.15.6:0/288147675 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff240003520 con 0x7ff25c1457c0 2024-03-04T09:36:39.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:39.301+0000 7ff258a00640 1 -- 172.21.15.6:0/288147675 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1259808651 0 0) 0x7ff240003520 con 0x7ff25c1457c0 2024-03-04T09:36:39.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:39.301+0000 7ff258a00640 1 -- 172.21.15.6:0/288147675 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff25c144c00 msgr2=0x7ff25c145280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:36:39.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:39.301+0000 7ff258a00640 1 --2- 172.21.15.6:0/288147675 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff25c144c00 0x7ff25c145280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:39.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:39.301+0000 7ff258a00640 1 -- 172.21.15.6:0/288147675 >> v2:172.21.15.146:3300/0 conn(0x7ff25c06b6c0 msgr2=0x7ff25c0cf130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:36:39.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:39.301+0000 7ff258a00640 1 --2- 172.21.15.6:0/288147675 >> v2:172.21.15.146:3300/0 conn(0x7ff25c06b6c0 0x7ff25c0cf130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:39.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:39.301+0000 7ff258a00640 1 -- 172.21.15.6:0/288147675 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff25c0d11b0 con 0x7ff25c1457c0 2024-03-04T09:36:39.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:39.301+0000 7ff258a00640 1 -- 172.21.15.6:0/288147675 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7ff244001670 con 0x7ff25c1457c0 2024-03-04T09:36:39.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:39.301+0000 7ff258a00640 1 -- 172.21.15.6:0/288147675 >> v1:172.21.15.6:6789/0 conn(0x7ff25c1457c0 legacy=0x7ff25c06afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:39.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:39.301+0000 7ff258a00640 1 --2- 172.21.15.6:0/288147675 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff2400039b0 0x7ff240003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:39.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:39.301+0000 7ff258a00640 1 --2- 172.21.15.6:0/288147675 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff25c144c00 0x7ff240004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:39.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:39.301+0000 7ff258a00640 1 --2- 172.21.15.6:0/288147675 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff240004d40 0x7ff2400051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:39.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:39.301+0000 7ff258a00640 1 -- 172.21.15.6:0/288147675 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff25c0d0bf0 con 0x7ff2400039b0 2024-03-04T09:36:39.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:39.301+0000 7ff258a00640 1 -- 172.21.15.6:0/288147675 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff25c0d0ed0 con 0x7ff240004d40 2024-03-04T09:36:39.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:39.301+0000 7ff258a00640 1 -- 172.21.15.6:0/288147675 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff24000b9c0 con 0x7ff25c144c00 2024-03-04T09:36:39.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:39.301+0000 7ff259e00640 1 --2- 172.21.15.6:0/288147675 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff2400039b0 0x7ff240003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:39.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:39.302+0000 7ff25a800640 1 --2- 172.21.15.6:0/288147675 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff240004d40 0x7ff2400051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:39.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:39.302+0000 7ff259400640 1 --2- 172.21.15.6:0/288147675 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff25c144c00 0x7ff240004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:39.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:39.302+0000 7ff259e00640 1 -- 172.21.15.6:0/288147675 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff25c144c00 msgr2=0x7ff240004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:36:39.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:39.302+0000 7ff259e00640 1 --2- 172.21.15.6:0/288147675 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff25c144c00 0x7ff240004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:39.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:39.302+0000 7ff259e00640 1 -- 172.21.15.6:0/288147675 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff240004d40 msgr2=0x7ff2400051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:36:39.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:39.302+0000 7ff259e00640 1 --2- 172.21.15.6:0/288147675 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff240004d40 0x7ff2400051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:39.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:39.302+0000 7ff259e00640 1 -- 172.21.15.6:0/288147675 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff240003520 con 0x7ff2400039b0 2024-03-04T09:36:39.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:39.303+0000 7ff259e00640 1 --2- 172.21.15.6:0/288147675 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff2400039b0 0x7ff240003dd0 secure :-1 s=READY pgs=1819 cs=0 l=1 rev1=1 crypto rx=0x7ff24c009fc0 tx=0x7ff240005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efb6c74352d0ec09 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:36:39.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:39.303+0000 7ff258a00640 1 -- 172.21.15.6:0/288147675 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff24c039070 con 0x7ff2400039b0 2024-03-04T09:36:39.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:39.303+0000 7ff258a00640 1 -- 172.21.15.6:0/288147675 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff24c02cc20 con 0x7ff2400039b0 2024-03-04T09:36:39.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:39.303+0000 7ff258a00640 1 -- 172.21.15.6:0/288147675 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff24c009c10 con 0x7ff2400039b0 2024-03-04T09:36:39.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:39.304+0000 7ff25b200640 1 -- 172.21.15.6:0/288147675 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff2400039b0 msgr2=0x7ff240003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:39.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:39.304+0000 7ff25b200640 1 --2- 172.21.15.6:0/288147675 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff2400039b0 0x7ff240003dd0 secure :-1 s=READY pgs=1819 cs=0 l=1 rev1=1 crypto rx=0x7ff24c009fc0 tx=0x7ff240005830 comp rx=0 tx=0).stop 2024-03-04T09:36:39.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:39.304+0000 7ff25a800640 1 -- 172.21.15.6:0/288147675 reap_dead start 2024-03-04T09:36:39.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:39.304+0000 7ff25b200640 1 -- 172.21.15.6:0/288147675 shutdown_connections 2024-03-04T09:36:39.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:39.304+0000 7ff25b200640 1 -- 172.21.15.6:0/288147675 >> 172.21.15.6:0/288147675 conn(0x7ff25c06c0c0 msgr2=0x7ff25c06c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:36:39.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:39.304+0000 7ff25b200640 1 -- 172.21.15.6:0/288147675 shutdown_connections 2024-03-04T09:36:39.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:39.304+0000 7ff25b200640 1 -- 172.21.15.6:0/288147675 wait complete. 2024-03-04T09:36:39.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:39.306+0000 7ff25b200640 1 Processor -- start 2024-03-04T09:36:39.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:39.306+0000 7ff25b200640 1 -- start start 2024-03-04T09:36:39.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:39.307+0000 7ff25b200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff2400039b0 0x7ff25c1f4810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:39.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:39.307+0000 7ff25b200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff25c0ad880 0x7ff25c1edc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:39.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:39.307+0000 7ff25b200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff25c06e980 0x7ff25c06ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:39.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:39.307+0000 7ff25b200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff25c0d19b0 con 0x7ff2400039b0 2024-03-04T09:36:39.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:39.307+0000 7ff25b200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff25c144fe0 con 0x7ff25c06e980 2024-03-04T09:36:39.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:39.307+0000 7ff25b200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff25c06f820 con 0x7ff25c0ad880 2024-03-04T09:36:39.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:39.307+0000 7ff259e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff2400039b0 0x7ff25c1f4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:39.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:39.307+0000 7ff259e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff2400039b0 0x7ff25c1f4810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55562/0 (socket says 172.21.15.6:55562) 2024-03-04T09:36:39.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:39.307+0000 7ff259e00640 1 -- 172.21.15.6:0/1526764466 learned_addr learned my addr 172.21.15.6:0/1526764466 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:36:39.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:39.307+0000 7ff259e00640 1 -- 172.21.15.6:0/1526764466 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff25c0ad880 msgr2=0x7ff25c1edc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:39.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:39.308+0000 7ff259e00640 1 --2- 172.21.15.6:0/1526764466 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff25c0ad880 0x7ff25c1edc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:39.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:39.308+0000 7ff25a800640 1 --2- 172.21.15.6:0/1526764466 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff25c06e980 0x7ff25c06ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:39.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:39.308+0000 7ff259e00640 1 -- 172.21.15.6:0/1526764466 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff25c06e980 msgr2=0x7ff25c06ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:39.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:39.308+0000 7ff259e00640 1 --2- 172.21.15.6:0/1526764466 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff25c06e980 0x7ff25c06ee30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:39.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:39.308+0000 7ff259e00640 1 -- 172.21.15.6:0/1526764466 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff25c1f4d50 con 0x7ff2400039b0 2024-03-04T09:36:39.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:39.308+0000 7ff259e00640 1 --2- 172.21.15.6:0/1526764466 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff2400039b0 0x7ff25c1f4810 secure :-1 s=READY pgs=1820 cs=0 l=1 rev1=1 crypto rx=0x7ff24c00e230 tx=0x7ff24c009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:36:39.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:39.308+0000 7ff252000640 1 -- 172.21.15.6:0/1526764466 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff24c039050 con 0x7ff2400039b0 2024-03-04T09:36:39.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:39.308+0000 7ff252000640 1 -- 172.21.15.6:0/1526764466 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff24c03a040 con 0x7ff2400039b0 2024-03-04T09:36:39.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:39.308+0000 7ff252000640 1 -- 172.21.15.6:0/1526764466 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff24c003bb0 con 0x7ff2400039b0 2024-03-04T09:36:39.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:39.308+0000 7ff25b200640 1 -- 172.21.15.6:0/1526764466 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff25c0ade20 con 0x7ff2400039b0 2024-03-04T09:36:39.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:39.309+0000 7ff25b200640 1 -- 172.21.15.6:0/1526764466 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff25c0ae3e0 con 0x7ff2400039b0 2024-03-04T09:36:39.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:39.310+0000 7ff25b200640 1 -- 172.21.15.6:0/1526764466 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff25c04f330 con 0x7ff2400039b0 2024-03-04T09:36:39.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:39.311+0000 7ff252000640 1 -- 172.21.15.6:0/1526764466 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff24c004030 con 0x7ff2400039b0 2024-03-04T09:36:39.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:39.311+0000 7ff252000640 1 --2- 172.21.15.6:0/1526764466 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff23806b520 0x7ff23806d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:39.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:39.312+0000 7ff252000640 1 -- 172.21.15.6:0/1526764466 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7ff24c0acb40 con 0x7ff2400039b0 2024-03-04T09:36:39.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:39.312+0000 7ff259400640 1 --2- 172.21.15.6:0/1526764466 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff23806b520 0x7ff23806d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:39.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:39.313+0000 7ff259400640 1 --2- 172.21.15.6:0/1526764466 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff23806b520 0x7ff23806d9e0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7ff244001eb0 tx=0x7ff244009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:36:39.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:39.316+0000 7ff252000640 1 -- 172.21.15.6:0/1526764466 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff25c1f4d50 con 0x7ff2400039b0 2024-03-04T09:36:39.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:39.696+0000 7ff25b200640 1 -- 172.21.15.6:0/1526764466 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff25c0722f0 con 0x7ff2400039b0 2024-03-04T09:36:39.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:39.697+0000 7ff252000640 1 -- 172.21.15.6:0/1526764466 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3464 (secure 0 0 0) 0x7ff24c07b960 con 0x7ff2400039b0 2024-03-04T09:36:39.694 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:36:39.694 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:35:26.088222+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"laggy_since":"2024-03-04T09:33:54.652601+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":9}]} 2024-03-04T09:36:39.695 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 101 2024-03-04T09:36:39.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:39.700+0000 7ff25b200640 1 -- 172.21.15.6:0/1526764466 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff23806b520 msgr2=0x7ff23806d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:39.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:39.700+0000 7ff25b200640 1 --2- 172.21.15.6:0/1526764466 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff23806b520 0x7ff23806d9e0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7ff244001eb0 tx=0x7ff244009040 comp rx=0 tx=0).stop 2024-03-04T09:36:39.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:39.700+0000 7ff25b200640 1 -- 172.21.15.6:0/1526764466 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff2400039b0 msgr2=0x7ff25c1f4810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:39.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:39.700+0000 7ff25b200640 1 --2- 172.21.15.6:0/1526764466 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff2400039b0 0x7ff25c1f4810 secure :-1 s=READY pgs=1820 cs=0 l=1 rev1=1 crypto rx=0x7ff24c00e230 tx=0x7ff24c009750 comp rx=0 tx=0).stop 2024-03-04T09:36:39.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:39.700+0000 7ff25b200640 1 -- 172.21.15.6:0/1526764466 shutdown_connections 2024-03-04T09:36:39.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:39.700+0000 7ff25b200640 1 --2- 172.21.15.6:0/1526764466 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff25c06e980 0x7ff25c06ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:39.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:39.700+0000 7ff25b200640 1 --2- 172.21.15.6:0/1526764466 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7ff23806b520 0x7ff23806d9e0 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:39.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:39.700+0000 7ff25b200640 1 --2- 172.21.15.6:0/1526764466 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff25c0ad880 0x7ff25c1edc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:39.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:39.700+0000 7ff25b200640 1 --2- 172.21.15.6:0/1526764466 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff2400039b0 0x7ff25c1f4810 unknown :-1 s=CLOSED pgs=1820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:39.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:39.700+0000 7ff25b200640 1 -- 172.21.15.6:0/1526764466 >> 172.21.15.6:0/1526764466 conn(0x7ff25c06c0c0 msgr2=0x7ff25c0b06f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:36:39.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:39.700+0000 7ff25b200640 1 -- 172.21.15.6:0/1526764466 shutdown_connections 2024-03-04T09:36:39.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:39.700+0000 7ff25b200640 1 -- 172.21.15.6:0/1526764466 wait complete. 2024-03-04T09:36:39.917 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:32:57.159681+0000', 'modified': '2024-03-04T09:35:26.088222+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8214, 'mds_1': 6796}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6796': {'gid': 6796, 'name': 'd', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.146:6837/2458070840', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 2458070840}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 2458070840}]}, 'join_fscid': -1, 'laggy_since': '2024-03-04T09:33:54.652601+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8214': {'gid': 8214, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2446627517', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2446627517}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2446627517}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8214, 'qdb_cluster': [8214]} 2024-03-04T09:36:39.918 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6796:up:stopping 2024-03-04T09:36:39.961 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-04T09:36:40.919 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:36:41.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:41.145+0000 7f1f29200640 1 Processor -- start 2024-03-04T09:36:41.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:41.146+0000 7f1f29200640 1 -- start start 2024-03-04T09:36:41.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:41.147+0000 7f1f29200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f1f2415dec0 0x7f1f240ce810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:41.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:41.147+0000 7f1f29200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f1f2415e2a0 0x7f1f240ced50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:41.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:41.147+0000 7f1f29200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1f240cf290 0x7f1f240d1b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:41.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:41.147+0000 7f1f29200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f240b0ea0 con 0x7f1f2415dec0 2024-03-04T09:36:41.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:41.147+0000 7f1f29200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f240cf720 con 0x7f1f2415e2a0 2024-03-04T09:36:41.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:41.147+0000 7f1f29200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f240d1120 con 0x7f1f240cf290 2024-03-04T09:36:41.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:41.147+0000 7f1f23400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f1f2415dec0 0x7f1f240ce810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:41.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:41.147+0000 7f1f23400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f1f2415dec0 0x7f1f240ce810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55564/0 (socket says 172.21.15.6:55564) 2024-03-04T09:36:41.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:41.147+0000 7f1f23400640 1 -- 172.21.15.6:0/315015104 learned_addr learned my addr 172.21.15.6:0/315015104 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:36:41.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:41.148+0000 7f1f1be00640 1 --2- 172.21.15.6:0/315015104 >> v2:172.21.15.146:3300/0 conn(0x7f1f2415e2a0 0x7f1f240ced50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:41.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:41.148+0000 7f1f23e00640 1 --2- 172.21.15.6:0/315015104 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1f240cf290 0x7f1f240d1b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:41.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:41.148+0000 7f1f23400640 1 -- 172.21.15.6:0/315015104 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1f240cf290 msgr2=0x7f1f240d1b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:36:41.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:41.148+0000 7f1f23400640 1 --2- 172.21.15.6:0/315015104 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1f240cf290 0x7f1f240d1b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:41.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:41.148+0000 7f1f23400640 1 -- 172.21.15.6:0/315015104 >> v2:172.21.15.146:3300/0 conn(0x7f1f2415e2a0 msgr2=0x7f1f240ced50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:36:41.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:41.148+0000 7f1f23400640 1 --2- 172.21.15.6:0/315015104 >> v2:172.21.15.146:3300/0 conn(0x7f1f2415e2a0 0x7f1f240ced50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:41.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:41.148+0000 7f1f23400640 1 -- 172.21.15.6:0/315015104 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f240d1400 con 0x7f1f2415dec0 2024-03-04T09:36:41.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:41.148+0000 7f1f1be00640 1 --2- 172.21.15.6:0/315015104 >> v2:172.21.15.146:3300/0 conn(0x7f1f2415e2a0 0x7f1f240ced50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T09:36:41.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:41.149+0000 7f1f23400640 1 --2- 172.21.15.6:0/315015104 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1f2415dec0 0x7f1f240ce810 secure :-1 s=READY pgs=1821 cs=0 l=1 rev1=1 crypto rx=0x7f1f240b5800 tx=0x7f1f1402b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=117130be76a9e61c server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:36:41.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:41.150+0000 7f1f22a00640 1 -- 172.21.15.6:0/315015104 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f14035020 con 0x7f1f2415dec0 2024-03-04T09:36:41.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:41.150+0000 7f1f22a00640 1 -- 172.21.15.6:0/315015104 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f1402d040 con 0x7f1f2415dec0 2024-03-04T09:36:41.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:41.150+0000 7f1f22a00640 1 -- 172.21.15.6:0/315015104 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f14031a80 con 0x7f1f2415dec0 2024-03-04T09:36:41.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:41.150+0000 7f1f29200640 1 -- 172.21.15.6:0/315015104 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1f2415dec0 msgr2=0x7f1f240ce810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:41.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:41.150+0000 7f1f29200640 1 --2- 172.21.15.6:0/315015104 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1f2415dec0 0x7f1f240ce810 secure :-1 s=READY pgs=1821 cs=0 l=1 rev1=1 crypto rx=0x7f1f240b5800 tx=0x7f1f1402b900 comp rx=0 tx=0).stop 2024-03-04T09:36:41.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:41.151+0000 7f1f29200640 1 -- 172.21.15.6:0/315015104 shutdown_connections 2024-03-04T09:36:41.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:41.151+0000 7f1f29200640 1 --2- 172.21.15.6:0/315015104 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1f240cf290 0x7f1f240d1b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:41.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:41.151+0000 7f1f29200640 1 --2- 172.21.15.6:0/315015104 >> v2:172.21.15.146:3300/0 conn(0x7f1f2415e2a0 0x7f1f240ced50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:41.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:41.151+0000 7f1f29200640 1 --2- 172.21.15.6:0/315015104 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1f2415dec0 0x7f1f240ce810 unknown :-1 s=CLOSED pgs=1821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:41.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:41.151+0000 7f1f29200640 1 -- 172.21.15.6:0/315015104 >> 172.21.15.6:0/315015104 conn(0x7f1f240d3d40 msgr2=0x7f1f240d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:36:41.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:41.151+0000 7f1f29200640 1 -- 172.21.15.6:0/315015104 shutdown_connections 2024-03-04T09:36:41.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:41.151+0000 7f1f29200640 1 -- 172.21.15.6:0/315015104 wait complete. 2024-03-04T09:36:41.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:41.153+0000 7f1f29200640 1 Processor -- start 2024-03-04T09:36:41.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:41.153+0000 7f1f29200640 1 -- start start 2024-03-04T09:36:41.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:41.155+0000 7f1f29200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1f240cf290 0x7f1f2415c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:41.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:41.155+0000 7f1f29200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1f2415dec0 0x7f1f240cd6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:41.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:41.155+0000 7f1f29200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1f2415e2a0 0x7f1f240cdbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:41.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:41.155+0000 7f1f29200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f240d0ec0 con 0x7f1f2415dec0 2024-03-04T09:36:41.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:41.155+0000 7f1f29200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f240d0d40 con 0x7f1f240cf290 2024-03-04T09:36:41.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:41.155+0000 7f1f29200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f240d2060 con 0x7f1f2415e2a0 2024-03-04T09:36:41.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:41.155+0000 7f1f1be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1f2415dec0 0x7f1f240cd6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:41.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:41.155+0000 7f1f1be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1f2415dec0 0x7f1f240cd6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55578/0 (socket says 172.21.15.6:55578) 2024-03-04T09:36:41.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:41.155+0000 7f1f1be00640 1 -- 172.21.15.6:0/2859058184 learned_addr learned my addr 172.21.15.6:0/2859058184 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:36:41.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:41.156+0000 7f1f23400640 1 --2- 172.21.15.6:0/2859058184 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1f240cf290 0x7f1f2415c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:41.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:41.156+0000 7f1f23e00640 1 --2- 172.21.15.6:0/2859058184 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1f2415e2a0 0x7f1f240cdbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:41.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:41.156+0000 7f1f1be00640 1 -- 172.21.15.6:0/2859058184 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1f2415e2a0 msgr2=0x7f1f240cdbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:41.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:41.156+0000 7f1f1be00640 1 --2- 172.21.15.6:0/2859058184 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1f2415e2a0 0x7f1f240cdbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:41.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:41.156+0000 7f1f1be00640 1 -- 172.21.15.6:0/2859058184 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1f240cf290 msgr2=0x7f1f2415c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:41.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:41.156+0000 7f1f1be00640 1 --2- 172.21.15.6:0/2859058184 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1f240cf290 0x7f1f2415c480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:41.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:41.156+0000 7f1f1be00640 1 -- 172.21.15.6:0/2859058184 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f240b5e10 con 0x7f1f2415dec0 2024-03-04T09:36:41.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:41.156+0000 7f1f23e00640 1 --2- 172.21.15.6:0/2859058184 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1f2415e2a0 0x7f1f240cdbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:36:41.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:41.156+0000 7f1f1be00640 1 --2- 172.21.15.6:0/2859058184 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1f2415dec0 0x7f1f240cd6a0 secure :-1 s=READY pgs=1822 cs=0 l=1 rev1=1 crypto rx=0x7f1f0c00d7d0 tx=0x7f1f0c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:36:41.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:41.156+0000 7f1f1b400640 1 -- 172.21.15.6:0/2859058184 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f0c004030 con 0x7f1f2415dec0 2024-03-04T09:36:41.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:41.156+0000 7f1f1b400640 1 -- 172.21.15.6:0/2859058184 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f0c0041d0 con 0x7f1f2415dec0 2024-03-04T09:36:41.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:41.156+0000 7f1f1b400640 1 -- 172.21.15.6:0/2859058184 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f0c00b6c0 con 0x7f1f2415dec0 2024-03-04T09:36:41.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:41.156+0000 7f1f29200640 1 -- 172.21.15.6:0/2859058184 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f240b3190 con 0x7f1f2415dec0 2024-03-04T09:36:41.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:41.157+0000 7f1f29200640 1 -- 172.21.15.6:0/2859058184 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f240ccbd0 con 0x7f1f2415dec0 2024-03-04T09:36:41.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:41.158+0000 7f1f29200640 1 -- 172.21.15.6:0/2859058184 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f2404f330 con 0x7f1f2415dec0 2024-03-04T09:36:41.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:41.159+0000 7f1f1b400640 1 -- 172.21.15.6:0/2859058184 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1f0c00b860 con 0x7f1f2415dec0 2024-03-04T09:36:41.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:41.160+0000 7f1f1b400640 1 --2- 172.21.15.6:0/2859058184 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f1f0006b4d0 0x7f1f0006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:41.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:41.160+0000 7f1f1b400640 1 -- 172.21.15.6:0/2859058184 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f1f0c08a440 con 0x7f1f2415dec0 2024-03-04T09:36:41.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:41.160+0000 7f1f23400640 1 --2- 172.21.15.6:0/2859058184 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f1f0006b4d0 0x7f1f0006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:41.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:41.162+0000 7f1f23400640 1 --2- 172.21.15.6:0/2859058184 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f1f0006b4d0 0x7f1f0006d990 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f1f14036000 tx=0x7f1f1403c040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:36:41.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:41.164+0000 7f1f1b400640 1 -- 172.21.15.6:0/2859058184 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1f240b5e10 con 0x7f1f2415dec0 2024-03-04T09:36:41.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:41.404+0000 7f1f29200640 1 -- 172.21.15.6:0/2859058184 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1f24072ab0 con 0x7f1f2415dec0 2024-03-04T09:36:41.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:41.405+0000 7f1f1b400640 1 -- 172.21.15.6:0/2859058184 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3464 (secure 0 0 0) 0x7f1f0c059270 con 0x7f1f2415dec0 2024-03-04T09:36:41.402 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:36:41.402 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:35:26.088222+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"laggy_since":"2024-03-04T09:33:54.652601+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":9}]} 2024-03-04T09:36:41.402 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 101 2024-03-04T09:36:41.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:41.408+0000 7f1f29200640 1 -- 172.21.15.6:0/2859058184 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f1f0006b4d0 msgr2=0x7f1f0006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:41.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:41.408+0000 7f1f29200640 1 --2- 172.21.15.6:0/2859058184 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f1f0006b4d0 0x7f1f0006d990 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f1f14036000 tx=0x7f1f1403c040 comp rx=0 tx=0).stop 2024-03-04T09:36:41.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:41.408+0000 7f1f29200640 1 -- 172.21.15.6:0/2859058184 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1f2415dec0 msgr2=0x7f1f240cd6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:41.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:41.408+0000 7f1f29200640 1 --2- 172.21.15.6:0/2859058184 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1f2415dec0 0x7f1f240cd6a0 secure :-1 s=READY pgs=1822 cs=0 l=1 rev1=1 crypto rx=0x7f1f0c00d7d0 tx=0x7f1f0c00dc90 comp rx=0 tx=0).stop 2024-03-04T09:36:41.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:41.408+0000 7f1f29200640 1 -- 172.21.15.6:0/2859058184 shutdown_connections 2024-03-04T09:36:41.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:41.408+0000 7f1f29200640 1 --2- 172.21.15.6:0/2859058184 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f1f0006b4d0 0x7f1f0006d990 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:41.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:41.408+0000 7f1f29200640 1 --2- 172.21.15.6:0/2859058184 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1f2415e2a0 0x7f1f240cdbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:41.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:41.408+0000 7f1f29200640 1 --2- 172.21.15.6:0/2859058184 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1f2415dec0 0x7f1f240cd6a0 unknown :-1 s=CLOSED pgs=1822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:41.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:41.408+0000 7f1f29200640 1 --2- 172.21.15.6:0/2859058184 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1f240cf290 0x7f1f2415c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:41.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:41.408+0000 7f1f29200640 1 -- 172.21.15.6:0/2859058184 >> 172.21.15.6:0/2859058184 conn(0x7f1f240d3d40 msgr2=0x7f1f2415e7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:36:41.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:41.408+0000 7f1f29200640 1 -- 172.21.15.6:0/2859058184 shutdown_connections 2024-03-04T09:36:41.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:41.408+0000 7f1f29200640 1 -- 172.21.15.6:0/2859058184 wait complete. 2024-03-04T09:36:41.624 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:32:57.159681+0000', 'modified': '2024-03-04T09:35:26.088222+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8214, 'mds_1': 6796}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6796': {'gid': 6796, 'name': 'd', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.146:6837/2458070840', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 2458070840}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 2458070840}]}, 'join_fscid': -1, 'laggy_since': '2024-03-04T09:33:54.652601+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8214': {'gid': 8214, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2446627517', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2446627517}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2446627517}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8214, 'qdb_cluster': [8214]} 2024-03-04T09:36:41.624 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6796:up:stopping 2024-03-04T09:36:41.742 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:36:41.749 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:36:41.963 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-04T09:36:42.625 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:36:42.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:42.849+0000 7fefbb200640 1 Processor -- start 2024-03-04T09:36:42.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:42.850+0000 7fefbb200640 1 -- start start 2024-03-04T09:36:42.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:42.851+0000 7fefbb200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fefb415d870 0x7fefb4066d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:42.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:42.851+0000 7fefbb200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fefb406bfc0 0x7fefb406c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:42.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:42.851+0000 7fefbb200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefb415dc50 con 0x7fefb406ea30 2024-03-04T09:36:42.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:42.851+0000 7fefbb200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fefb406c960 con 0x7fefb415d870 2024-03-04T09:36:42.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:42.851+0000 7fefbb200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fefb40b3650 con 0x7fefb406bfc0 2024-03-04T09:36:42.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:42.851+0000 7fefb9200640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fefb406ea30 0x7fefb406ee80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:53588/0 (socket says 172.21.15.6:53588) 2024-03-04T09:36:42.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:42.851+0000 7fefb9200640 1 -- 172.21.15.6:0/2229187720 learned_addr learned my addr 172.21.15.6:0/2229187720 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:36:42.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:42.851+0000 7fefb3e00640 1 --2- 172.21.15.6:0/2229187720 >> v2:172.21.15.6:3300/0 conn(0x7fefb415d870 0x7fefb4066d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:42.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:42.852+0000 7fefab400640 1 --2- 172.21.15.6:0/2229187720 >> v2:172.21.15.146:3300/0 conn(0x7fefb406bfc0 0x7fefb406c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:42.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:42.852+0000 7fefb3400640 1 -- 172.21.15.6:0/2229187720 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2394059969 0 0) 0x7fefb415dc50 con 0x7fefb406ea30 2024-03-04T09:36:42.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:42.852+0000 7fefb3400640 1 -- 172.21.15.6:0/2229187720 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef98005730 con 0x7fefb406ea30 2024-03-04T09:36:42.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:42.852+0000 7fefb3e00640 1 -- 172.21.15.6:0/2229187720 >> v2:172.21.15.146:3300/0 conn(0x7fefb406bfc0 msgr2=0x7fefb406c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:36:42.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:42.852+0000 7fefb3e00640 1 --2- 172.21.15.6:0/2229187720 >> v2:172.21.15.146:3300/0 conn(0x7fefb406bfc0 0x7fefb406c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:42.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:42.852+0000 7fefb3e00640 1 -- 172.21.15.6:0/2229187720 >> v1:172.21.15.6:6789/0 conn(0x7fefb406ea30 legacy=0x7fefb406ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:42.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:42.852+0000 7fefb3e00640 1 -- 172.21.15.6:0/2229187720 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefb40b3930 con 0x7fefb415d870 2024-03-04T09:36:42.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:42.852+0000 7fefab400640 1 --2- 172.21.15.6:0/2229187720 >> v2:172.21.15.146:3300/0 conn(0x7fefb406bfc0 0x7fefb406c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T09:36:42.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:42.853+0000 7fefb3e00640 1 --2- 172.21.15.6:0/2229187720 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fefb415d870 0x7fefb4066d00 secure :-1 s=READY pgs=1824 cs=0 l=1 rev1=1 crypto rx=0x7fefb4072970 tx=0x7fefa402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fec0ca6c4b6e8bc server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:36:42.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:42.854+0000 7fefb3400640 1 -- 172.21.15.6:0/2229187720 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefa40315a0 con 0x7fefb415d870 2024-03-04T09:36:42.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:42.854+0000 7fefb3400640 1 -- 172.21.15.6:0/2229187720 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefa4037070 con 0x7fefb415d870 2024-03-04T09:36:42.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:42.854+0000 7fefb3400640 1 -- 172.21.15.6:0/2229187720 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefa4031960 con 0x7fefb415d870 2024-03-04T09:36:42.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:42.855+0000 7fefbb200640 1 -- 172.21.15.6:0/2229187720 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fefb415d870 msgr2=0x7fefb4066d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:42.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:42.855+0000 7fefbb200640 1 --2- 172.21.15.6:0/2229187720 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fefb415d870 0x7fefb4066d00 secure :-1 s=READY pgs=1824 cs=0 l=1 rev1=1 crypto rx=0x7fefb4072970 tx=0x7fefa402b840 comp rx=0 tx=0).stop 2024-03-04T09:36:42.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:42.855+0000 7fefbb200640 1 -- 172.21.15.6:0/2229187720 shutdown_connections 2024-03-04T09:36:42.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:42.855+0000 7fefbb200640 1 --2- 172.21.15.6:0/2229187720 >> v2:172.21.15.146:3300/0 conn(0x7fefb406bfc0 0x7fefb406c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:42.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:42.855+0000 7fefbb200640 1 --2- 172.21.15.6:0/2229187720 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fefb415d870 0x7fefb4066d00 unknown :-1 s=CLOSED pgs=1824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:42.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:42.855+0000 7fefbb200640 1 -- 172.21.15.6:0/2229187720 >> 172.21.15.6:0/2229187720 conn(0x7fefb40714d0 msgr2=0x7fefb40718b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:36:42.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:42.855+0000 7fefbb200640 1 -- 172.21.15.6:0/2229187720 shutdown_connections 2024-03-04T09:36:42.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:42.855+0000 7fefbb200640 1 -- 172.21.15.6:0/2229187720 wait complete. 2024-03-04T09:36:42.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:42.857+0000 7fefbb200640 1 Processor -- start 2024-03-04T09:36:42.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:42.857+0000 7fefbb200640 1 -- start start 2024-03-04T09:36:42.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:42.858+0000 7fefbb200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fefb406bfc0 0x7fefb41e5b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:42.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:42.858+0000 7fefbb200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fefb406ea30 0x7fefb415ba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:42.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:42.858+0000 7fefbb200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fefb415d870 0x7fefb41f46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:42.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:42.858+0000 7fefbb200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefb40b41d0 con 0x7fefb415d870 2024-03-04T09:36:42.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:42.858+0000 7fefbb200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefb40b3ef0 con 0x7fefb406ea30 2024-03-04T09:36:42.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:42.858+0000 7fefbb200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefb40ce320 con 0x7fefb406bfc0 2024-03-04T09:36:42.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:42.858+0000 7fefb9200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fefb415d870 0x7fefb41f46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:42.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:42.858+0000 7fefb9200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fefb415d870 0x7fefb41f46a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55596/0 (socket says 172.21.15.6:55596) 2024-03-04T09:36:42.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:42.858+0000 7fefb9200640 1 -- 172.21.15.6:0/3281701938 learned_addr learned my addr 172.21.15.6:0/3281701938 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:36:42.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:42.859+0000 7fefb3e00640 1 --2- 172.21.15.6:0/3281701938 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fefb406bfc0 0x7fefb41e5b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:42.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:42.859+0000 7fefab400640 1 --2- 172.21.15.6:0/3281701938 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fefb406ea30 0x7fefb415ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:42.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:42.859+0000 7fefb9200640 1 -- 172.21.15.6:0/3281701938 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fefb406bfc0 msgr2=0x7fefb41e5b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:42.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:42.859+0000 7fefb9200640 1 --2- 172.21.15.6:0/3281701938 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fefb406bfc0 0x7fefb41e5b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:42.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:42.859+0000 7fefb9200640 1 -- 172.21.15.6:0/3281701938 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fefb406ea30 msgr2=0x7fefb415ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:42.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:42.859+0000 7fefb9200640 1 --2- 172.21.15.6:0/3281701938 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fefb406ea30 0x7fefb415ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:42.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:42.859+0000 7fefb9200640 1 -- 172.21.15.6:0/3281701938 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefb41f7390 con 0x7fefb415d870 2024-03-04T09:36:42.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:42.859+0000 7fefab400640 1 --2- 172.21.15.6:0/3281701938 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fefb406ea30 0x7fefb415ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:36:42.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:42.859+0000 7fefb9200640 1 --2- 172.21.15.6:0/3281701938 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fefb415d870 0x7fefb41f46a0 secure :-1 s=READY pgs=1825 cs=0 l=1 rev1=1 crypto rx=0x7fefac00caa0 tx=0x7fefac00cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:36:42.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:42.859+0000 7fefb0c00640 1 -- 172.21.15.6:0/3281701938 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefac017070 con 0x7fefb415d870 2024-03-04T09:36:42.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:42.859+0000 7fefb0c00640 1 -- 172.21.15.6:0/3281701938 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefac00ac00 con 0x7fefb415d870 2024-03-04T09:36:42.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:42.859+0000 7fefbb200640 1 -- 172.21.15.6:0/3281701938 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefb41f7670 con 0x7fefb415d870 2024-03-04T09:36:42.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:42.860+0000 7fefb0c00640 1 -- 172.21.15.6:0/3281701938 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefac012400 con 0x7fefb415d870 2024-03-04T09:36:42.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:42.860+0000 7fefbb200640 1 -- 172.21.15.6:0/3281701938 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefb41f7c30 con 0x7fefb415d870 2024-03-04T09:36:42.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:42.861+0000 7fefbb200640 1 -- 172.21.15.6:0/3281701938 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefb404f3a0 con 0x7fefb415d870 2024-03-04T09:36:42.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:42.862+0000 7fefb0c00640 1 -- 172.21.15.6:0/3281701938 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fefac01a070 con 0x7fefb415d870 2024-03-04T09:36:42.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:42.863+0000 7fefb0c00640 1 --2- 172.21.15.6:0/3281701938 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fef9006b4d0 0x7fef9006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:42.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:42.863+0000 7fefb0c00640 1 -- 172.21.15.6:0/3281701938 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fefac08a150 con 0x7fefb415d870 2024-03-04T09:36:42.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:42.863+0000 7fefb3e00640 1 --2- 172.21.15.6:0/3281701938 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fef9006b4d0 0x7fef9006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:42.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:42.865+0000 7fefb3e00640 1 --2- 172.21.15.6:0/3281701938 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fef9006b4d0 0x7fef9006d990 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fefa4036000 tx=0x7fefa4008030 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:36:42.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:42.867+0000 7fefb0c00640 1 -- 172.21.15.6:0/3281701938 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fefac058e60 con 0x7fefb415d870 2024-03-04T09:36:43.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:43.108+0000 7fefbb200640 1 -- 172.21.15.6:0/3281701938 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fefb41e62b0 con 0x7fefb415d870 2024-03-04T09:36:43.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:43.109+0000 7fefb0c00640 1 -- 172.21.15.6:0/3281701938 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3464 (secure 0 0 0) 0x7fefac058760 con 0x7fefb415d870 2024-03-04T09:36:43.106 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:36:43.106 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:35:26.088222+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"laggy_since":"2024-03-04T09:33:54.652601+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":9}]} 2024-03-04T09:36:43.107 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 101 2024-03-04T09:36:43.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:43.112+0000 7fefbb200640 1 -- 172.21.15.6:0/3281701938 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fef9006b4d0 msgr2=0x7fef9006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:43.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:43.112+0000 7fefbb200640 1 --2- 172.21.15.6:0/3281701938 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fef9006b4d0 0x7fef9006d990 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fefa4036000 tx=0x7fefa4008030 comp rx=0 tx=0).stop 2024-03-04T09:36:43.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:43.112+0000 7fefbb200640 1 -- 172.21.15.6:0/3281701938 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fefb415d870 msgr2=0x7fefb41f46a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:43.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:43.112+0000 7fefbb200640 1 --2- 172.21.15.6:0/3281701938 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fefb415d870 0x7fefb41f46a0 secure :-1 s=READY pgs=1825 cs=0 l=1 rev1=1 crypto rx=0x7fefac00caa0 tx=0x7fefac00cf60 comp rx=0 tx=0).stop 2024-03-04T09:36:43.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:43.112+0000 7fefbb200640 1 -- 172.21.15.6:0/3281701938 shutdown_connections 2024-03-04T09:36:43.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:43.112+0000 7fefbb200640 1 --2- 172.21.15.6:0/3281701938 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fefb415d870 0x7fefb41f46a0 unknown :-1 s=CLOSED pgs=1825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:43.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:43.112+0000 7fefbb200640 1 --2- 172.21.15.6:0/3281701938 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fefb406ea30 0x7fefb415ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:43.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:43.112+0000 7fefbb200640 1 --2- 172.21.15.6:0/3281701938 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fef9006b4d0 0x7fef9006d990 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:43.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:43.112+0000 7fefbb200640 1 --2- 172.21.15.6:0/3281701938 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fefb406bfc0 0x7fefb41e5b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:43.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:43.112+0000 7fefbb200640 1 -- 172.21.15.6:0/3281701938 >> 172.21.15.6:0/3281701938 conn(0x7fefb40714d0 msgr2=0x7fefb4159300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:36:43.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:43.112+0000 7fefbb200640 1 -- 172.21.15.6:0/3281701938 shutdown_connections 2024-03-04T09:36:43.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:43.112+0000 7fefbb200640 1 -- 172.21.15.6:0/3281701938 wait complete. 2024-03-04T09:36:43.329 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:32:57.159681+0000', 'modified': '2024-03-04T09:35:26.088222+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8214, 'mds_1': 6796}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6796': {'gid': 6796, 'name': 'd', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.146:6837/2458070840', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 2458070840}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 2458070840}]}, 'join_fscid': -1, 'laggy_since': '2024-03-04T09:33:54.652601+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8214': {'gid': 8214, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2446627517', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2446627517}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2446627517}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8214, 'qdb_cluster': [8214]} 2024-03-04T09:36:43.329 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6796:up:stopping 2024-03-04T09:36:43.965 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-04T09:36:44.330 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:36:44.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:44.559+0000 7fc029000640 1 Processor -- start 2024-03-04T09:36:44.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:44.559+0000 7fc029000640 1 -- start start 2024-03-04T09:36:44.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:44.561+0000 7fc029000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc024070f10 0x7fc0240cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:44.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:44.561+0000 7fc029000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc0240cce90 0x7fc02415c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:44.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:44.561+0000 7fc029000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0240714d0 con 0x7fc024145300 2024-03-04T09:36:44.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:44.561+0000 7fc029000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0240cd300 con 0x7fc024070f10 2024-03-04T09:36:44.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:44.561+0000 7fc029000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0240717b0 con 0x7fc0240cce90 2024-03-04T09:36:44.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:44.561+0000 7fc022a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc024070f10 0x7fc0240cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:44.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:44.561+0000 7fc022a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc024070f10 0x7fc0240cc950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55606/0 (socket says 172.21.15.6:55606) 2024-03-04T09:36:44.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:44.561+0000 7fc022a00640 1 -- 172.21.15.6:0/470052656 learned_addr learned my addr 172.21.15.6:0/470052656 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:36:44.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:44.562+0000 7fc023e00640 1 --2- 172.21.15.6:0/470052656 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc0240cce90 0x7fc02415c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:44.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:44.562+0000 7fc022a00640 1 -- 172.21.15.6:0/470052656 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc0240cce90 msgr2=0x7fc02415c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:36:44.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:44.562+0000 7fc022a00640 1 --2- 172.21.15.6:0/470052656 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc0240cce90 0x7fc02415c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:44.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:44.562+0000 7fc022a00640 1 -- 172.21.15.6:0/470052656 >> v1:172.21.15.146:6789/0 conn(0x7fc024145300 legacy=0x7fc0240b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:44.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:44.562+0000 7fc022a00640 1 -- 172.21.15.6:0/470052656 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc024071a90 con 0x7fc024070f10 2024-03-04T09:36:44.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:44.564+0000 7fc022a00640 1 --2- 172.21.15.6:0/470052656 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc024070f10 0x7fc0240cc950 secure :-1 s=READY pgs=1826 cs=0 l=1 rev1=1 crypto rx=0x7fc010009870 tx=0x7fc01002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62fb6e764635fde2 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:36:44.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:44.564+0000 7fc022000640 1 -- 172.21.15.6:0/470052656 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc010035020 con 0x7fc024070f10 2024-03-04T09:36:44.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:44.564+0000 7fc022000640 1 -- 172.21.15.6:0/470052656 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc010031640 con 0x7fc024070f10 2024-03-04T09:36:44.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:44.564+0000 7fc022000640 1 -- 172.21.15.6:0/470052656 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc01002d040 con 0x7fc024070f10 2024-03-04T09:36:44.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:44.565+0000 7fc029000640 1 -- 172.21.15.6:0/470052656 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc024070f10 msgr2=0x7fc0240cc950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:44.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:44.565+0000 7fc029000640 1 --2- 172.21.15.6:0/470052656 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc024070f10 0x7fc0240cc950 secure :-1 s=READY pgs=1826 cs=0 l=1 rev1=1 crypto rx=0x7fc010009870 tx=0x7fc01002b8f0 comp rx=0 tx=0).stop 2024-03-04T09:36:44.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:44.565+0000 7fc029000640 1 -- 172.21.15.6:0/470052656 shutdown_connections 2024-03-04T09:36:44.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:44.565+0000 7fc029000640 1 --2- 172.21.15.6:0/470052656 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc0240cce90 0x7fc02415c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:44.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:44.565+0000 7fc029000640 1 --2- 172.21.15.6:0/470052656 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc024070f10 0x7fc0240cc950 unknown :-1 s=CLOSED pgs=1826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:44.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:44.565+0000 7fc029000640 1 -- 172.21.15.6:0/470052656 >> 172.21.15.6:0/470052656 conn(0x7fc02406b3d0 msgr2=0x7fc02406b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:36:44.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:44.565+0000 7fc029000640 1 -- 172.21.15.6:0/470052656 shutdown_connections 2024-03-04T09:36:44.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:44.565+0000 7fc029000640 1 -- 172.21.15.6:0/470052656 wait complete. 2024-03-04T09:36:44.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:44.567+0000 7fc029000640 1 Processor -- start 2024-03-04T09:36:44.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:44.567+0000 7fc029000640 1 -- start start 2024-03-04T09:36:44.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:44.568+0000 7fc029000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc024070f10 0x7fc02413e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:44.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:44.568+0000 7fc029000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc0240cce90 0x7fc0240b02a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:44.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:44.568+0000 7fc029000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc024145300 0x7fc0240b07e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:44.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:44.568+0000 7fc029000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc02415cab0 con 0x7fc024070f10 2024-03-04T09:36:44.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:44.568+0000 7fc029000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0241456e0 con 0x7fc024145300 2024-03-04T09:36:44.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:44.568+0000 7fc029000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc02415cc30 con 0x7fc0240cce90 2024-03-04T09:36:44.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:44.568+0000 7fc023400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc024070f10 0x7fc02413e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:44.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:44.569+0000 7fc023400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc024070f10 0x7fc02413e4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55614/0 (socket says 172.21.15.6:55614) 2024-03-04T09:36:44.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:44.569+0000 7fc023400640 1 -- 172.21.15.6:0/3929692779 learned_addr learned my addr 172.21.15.6:0/3929692779 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:36:44.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:44.569+0000 7fc023e00640 1 --2- 172.21.15.6:0/3929692779 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc024145300 0x7fc0240b07e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:44.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:44.569+0000 7fc022a00640 1 --2- 172.21.15.6:0/3929692779 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc0240cce90 0x7fc0240b02a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:44.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:44.569+0000 7fc023400640 1 -- 172.21.15.6:0/3929692779 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc0240cce90 msgr2=0x7fc0240b02a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:44.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:44.569+0000 7fc023400640 1 --2- 172.21.15.6:0/3929692779 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc0240cce90 0x7fc0240b02a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:44.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:44.569+0000 7fc023400640 1 -- 172.21.15.6:0/3929692779 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc024145300 msgr2=0x7fc0240b07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:44.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:44.569+0000 7fc023400640 1 --2- 172.21.15.6:0/3929692779 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc024145300 0x7fc0240b07e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:44.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:44.569+0000 7fc023400640 1 -- 172.21.15.6:0/3929692779 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0241f5080 con 0x7fc024070f10 2024-03-04T09:36:44.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:44.569+0000 7fc023e00640 1 --2- 172.21.15.6:0/3929692779 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc024145300 0x7fc0240b07e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:36:44.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:44.569+0000 7fc023400640 1 --2- 172.21.15.6:0/3929692779 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc024070f10 0x7fc02413e4c0 secure :-1 s=READY pgs=1827 cs=0 l=1 rev1=1 crypto rx=0x7fc01400c3f0 tx=0x7fc01400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:36:44.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:44.570+0000 7fc01b400640 1 -- 172.21.15.6:0/3929692779 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0140052d0 con 0x7fc024070f10 2024-03-04T09:36:44.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:44.570+0000 7fc01b400640 1 -- 172.21.15.6:0/3929692779 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc014005490 con 0x7fc024070f10 2024-03-04T09:36:44.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:44.570+0000 7fc01b400640 1 -- 172.21.15.6:0/3929692779 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc014007260 con 0x7fc024070f10 2024-03-04T09:36:44.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:44.570+0000 7fc029000640 1 -- 172.21.15.6:0/3929692779 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0241f5360 con 0x7fc024070f10 2024-03-04T09:36:44.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:44.570+0000 7fc029000640 1 -- 172.21.15.6:0/3929692779 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0241f5920 con 0x7fc024070f10 2024-03-04T09:36:44.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:44.571+0000 7fc029000640 1 -- 172.21.15.6:0/3929692779 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc02404f330 con 0x7fc024070f10 2024-03-04T09:36:44.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:44.573+0000 7fc01b400640 1 -- 172.21.15.6:0/3929692779 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc014007400 con 0x7fc024070f10 2024-03-04T09:36:44.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:44.574+0000 7fc01b400640 1 --2- 172.21.15.6:0/3929692779 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc00406b4d0 0x7fc00406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:44.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:44.574+0000 7fc01b400640 1 -- 172.21.15.6:0/3929692779 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fc014089780 con 0x7fc024070f10 2024-03-04T09:36:44.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:44.575+0000 7fc022a00640 1 --2- 172.21.15.6:0/3929692779 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc00406b4d0 0x7fc00406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:44.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:44.576+0000 7fc022a00640 1 --2- 172.21.15.6:0/3929692779 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc00406b4d0 0x7fc00406d990 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7fc010036000 tx=0x7fc010008030 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:36:44.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:44.583+0000 7fc01b400640 1 -- 172.21.15.6:0/3929692779 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc014058520 con 0x7fc024070f10 2024-03-04T09:36:44.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:44.822+0000 7fc029000640 1 -- 172.21.15.6:0/3929692779 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc02413ca70 con 0x7fc024070f10 2024-03-04T09:36:44.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:44.823+0000 7fc01b400640 1 -- 172.21.15.6:0/3929692779 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3464 (secure 0 0 0) 0x7fc014057e20 con 0x7fc024070f10 2024-03-04T09:36:44.820 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:36:44.820 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:35:26.088222+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"laggy_since":"2024-03-04T09:33:54.652601+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":9}]} 2024-03-04T09:36:44.820 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 101 2024-03-04T09:36:44.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:44.825+0000 7fc029000640 1 -- 172.21.15.6:0/3929692779 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc00406b4d0 msgr2=0x7fc00406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:44.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:44.825+0000 7fc029000640 1 --2- 172.21.15.6:0/3929692779 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc00406b4d0 0x7fc00406d990 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7fc010036000 tx=0x7fc010008030 comp rx=0 tx=0).stop 2024-03-04T09:36:44.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:44.825+0000 7fc029000640 1 -- 172.21.15.6:0/3929692779 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc024070f10 msgr2=0x7fc02413e4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:44.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:44.825+0000 7fc029000640 1 --2- 172.21.15.6:0/3929692779 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc024070f10 0x7fc02413e4c0 secure :-1 s=READY pgs=1827 cs=0 l=1 rev1=1 crypto rx=0x7fc01400c3f0 tx=0x7fc01400c8b0 comp rx=0 tx=0).stop 2024-03-04T09:36:44.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:44.825+0000 7fc029000640 1 -- 172.21.15.6:0/3929692779 shutdown_connections 2024-03-04T09:36:44.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:44.826+0000 7fc029000640 1 --2- 172.21.15.6:0/3929692779 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc024145300 0x7fc0240b07e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:44.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:44.826+0000 7fc029000640 1 --2- 172.21.15.6:0/3929692779 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc00406b4d0 0x7fc00406d990 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:44.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:44.826+0000 7fc029000640 1 --2- 172.21.15.6:0/3929692779 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc0240cce90 0x7fc0240b02a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:44.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:44.826+0000 7fc029000640 1 --2- 172.21.15.6:0/3929692779 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc024070f10 0x7fc02413e4c0 unknown :-1 s=CLOSED pgs=1827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:44.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:44.826+0000 7fc029000640 1 -- 172.21.15.6:0/3929692779 >> 172.21.15.6:0/3929692779 conn(0x7fc02406b3d0 msgr2=0x7fc02415d0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:36:44.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:44.826+0000 7fc029000640 1 -- 172.21.15.6:0/3929692779 shutdown_connections 2024-03-04T09:36:44.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:44.826+0000 7fc029000640 1 -- 172.21.15.6:0/3929692779 wait complete. 2024-03-04T09:36:45.045 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:32:57.159681+0000', 'modified': '2024-03-04T09:35:26.088222+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8214, 'mds_1': 6796}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6796': {'gid': 6796, 'name': 'd', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.146:6837/2458070840', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 2458070840}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 2458070840}]}, 'join_fscid': -1, 'laggy_since': '2024-03-04T09:33:54.652601+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8214': {'gid': 8214, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2446627517', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2446627517}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2446627517}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8214, 'qdb_cluster': [8214]} 2024-03-04T09:36:45.045 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6796:up:stopping 2024-03-04T09:36:45.967 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-04T09:36:46.046 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:36:46.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:46.282+0000 7fad68200640 1 Processor -- start 2024-03-04T09:36:46.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:46.283+0000 7fad68200640 1 -- start start 2024-03-04T09:36:46.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:46.284+0000 7fad68200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fad600af1a0 0x7fad6015c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:46.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:46.284+0000 7fad68200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fad600af580 0x7fad600b4c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:46.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:46.284+0000 7fad68200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fad600cdc20 0x7fad600b2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:46.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:46.284+0000 7fad68200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad6015cc40 con 0x7fad600af580 2024-03-04T09:36:46.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:46.284+0000 7fad68200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad6015cf20 con 0x7fad600af1a0 2024-03-04T09:36:46.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:46.284+0000 7fad68200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad6015d200 con 0x7fad600cdc20 2024-03-04T09:36:46.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:46.285+0000 7fad64e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fad600af580 0x7fad600b4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:46.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:46.285+0000 7fad64e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fad600af580 0x7fad600b4c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55620/0 (socket says 172.21.15.6:55620) 2024-03-04T09:36:46.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:46.285+0000 7fad64e00640 1 -- 172.21.15.6:0/2920465772 learned_addr learned my addr 172.21.15.6:0/2920465772 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:36:46.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:46.285+0000 7fad66200640 1 --2- 172.21.15.6:0/2920465772 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fad600cdc20 0x7fad600b2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:46.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:46.285+0000 7fad65800640 1 --2- 172.21.15.6:0/2920465772 >> v2:172.21.15.146:3300/0 conn(0x7fad600af1a0 0x7fad6015c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:46.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:46.285+0000 7fad64e00640 1 -- 172.21.15.6:0/2920465772 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fad600cdc20 msgr2=0x7fad600b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:36:46.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:46.285+0000 7fad64e00640 1 --2- 172.21.15.6:0/2920465772 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fad600cdc20 0x7fad600b2dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:46.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:46.285+0000 7fad64e00640 1 -- 172.21.15.6:0/2920465772 >> v2:172.21.15.146:3300/0 conn(0x7fad600af1a0 msgr2=0x7fad6015c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:36:46.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:46.285+0000 7fad64e00640 1 --2- 172.21.15.6:0/2920465772 >> v2:172.21.15.146:3300/0 conn(0x7fad600af1a0 0x7fad6015c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:46.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:46.285+0000 7fad64e00640 1 -- 172.21.15.6:0/2920465772 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad6015d4e0 con 0x7fad600af580 2024-03-04T09:36:46.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:46.287+0000 7fad64e00640 1 --2- 172.21.15.6:0/2920465772 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fad600af580 0x7fad600b4c20 secure :-1 s=READY pgs=1828 cs=0 l=1 rev1=1 crypto rx=0x7fad60074960 tx=0x7fad4c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a162d971bbdaa20a server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:36:46.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:46.287+0000 7fad5fe00640 1 -- 172.21.15.6:0/2920465772 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad4c02e070 con 0x7fad600af580 2024-03-04T09:36:46.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:46.287+0000 7fad5fe00640 1 -- 172.21.15.6:0/2920465772 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad4c037070 con 0x7fad600af580 2024-03-04T09:36:46.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:46.287+0000 7fad5fe00640 1 -- 172.21.15.6:0/2920465772 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad4c031a00 con 0x7fad600af580 2024-03-04T09:36:46.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:46.288+0000 7fad68200640 1 -- 172.21.15.6:0/2920465772 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fad600af580 msgr2=0x7fad600b4c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:46.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:46.288+0000 7fad68200640 1 --2- 172.21.15.6:0/2920465772 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fad600af580 0x7fad600b4c20 secure :-1 s=READY pgs=1828 cs=0 l=1 rev1=1 crypto rx=0x7fad60074960 tx=0x7fad4c02b880 comp rx=0 tx=0).stop 2024-03-04T09:36:46.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:46.288+0000 7fad68200640 1 -- 172.21.15.6:0/2920465772 shutdown_connections 2024-03-04T09:36:46.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:46.288+0000 7fad68200640 1 --2- 172.21.15.6:0/2920465772 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fad600cdc20 0x7fad600b2dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:46.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:46.288+0000 7fad68200640 1 --2- 172.21.15.6:0/2920465772 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fad600af580 0x7fad600b4c20 unknown :-1 s=CLOSED pgs=1828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:46.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:46.288+0000 7fad68200640 1 --2- 172.21.15.6:0/2920465772 >> v2:172.21.15.146:3300/0 conn(0x7fad600af1a0 0x7fad6015c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:46.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:46.288+0000 7fad68200640 1 -- 172.21.15.6:0/2920465772 >> 172.21.15.6:0/2920465772 conn(0x7fad6006afb0 msgr2=0x7fad6006b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:36:46.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:46.288+0000 7fad68200640 1 -- 172.21.15.6:0/2920465772 shutdown_connections 2024-03-04T09:36:46.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:46.288+0000 7fad68200640 1 -- 172.21.15.6:0/2920465772 wait complete. 2024-03-04T09:36:46.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:46.290+0000 7fad68200640 1 Processor -- start 2024-03-04T09:36:46.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:46.290+0000 7fad68200640 1 -- start start 2024-03-04T09:36:46.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:46.291+0000 7fad68200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fad600af1a0 0x7fad60147600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:46.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:46.291+0000 7fad68200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fad600af580 0x7fad60157d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:46.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:46.291+0000 7fad68200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fad600cdc20 0x7fad6015e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:46.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:46.291+0000 7fad68200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad600b3490 con 0x7fad600af580 2024-03-04T09:36:46.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:46.291+0000 7fad68200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad600b3310 con 0x7fad600af1a0 2024-03-04T09:36:46.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:46.291+0000 7fad68200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad600b3610 con 0x7fad600cdc20 2024-03-04T09:36:46.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:46.292+0000 7fad64e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fad600af580 0x7fad60157d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:46.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:46.292+0000 7fad64e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fad600af580 0x7fad60157d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55634/0 (socket says 172.21.15.6:55634) 2024-03-04T09:36:46.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:46.292+0000 7fad64e00640 1 -- 172.21.15.6:0/596026819 learned_addr learned my addr 172.21.15.6:0/596026819 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:36:46.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:46.292+0000 7fad66200640 1 --2- 172.21.15.6:0/596026819 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fad600cdc20 0x7fad6015e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:46.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:46.292+0000 7fad65800640 1 --2- 172.21.15.6:0/596026819 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fad600af1a0 0x7fad60147600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:46.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:46.292+0000 7fad64e00640 1 -- 172.21.15.6:0/596026819 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fad600cdc20 msgr2=0x7fad6015e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:46.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:46.292+0000 7fad64e00640 1 --2- 172.21.15.6:0/596026819 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fad600cdc20 0x7fad6015e8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:46.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:46.292+0000 7fad64e00640 1 -- 172.21.15.6:0/596026819 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fad600af1a0 msgr2=0x7fad60147600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:46.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:46.292+0000 7fad64e00640 1 --2- 172.21.15.6:0/596026819 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fad600af1a0 0x7fad60147600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:46.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:46.292+0000 7fad64e00640 1 -- 172.21.15.6:0/596026819 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad601f2af0 con 0x7fad600af580 2024-03-04T09:36:46.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:46.293+0000 7fad64e00640 1 --2- 172.21.15.6:0/596026819 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fad600af580 0x7fad60157d20 secure :-1 s=READY pgs=1829 cs=0 l=1 rev1=1 crypto rx=0x7fad4c000c00 tx=0x7fad4c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:36:46.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:46.293+0000 7fad5d600640 1 -- 172.21.15.6:0/596026819 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad4c032670 con 0x7fad600af580 2024-03-04T09:36:46.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:46.293+0000 7fad5d600640 1 -- 172.21.15.6:0/596026819 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad4c032850 con 0x7fad600af580 2024-03-04T09:36:46.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:46.293+0000 7fad5d600640 1 -- 172.21.15.6:0/596026819 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad4c032c40 con 0x7fad600af580 2024-03-04T09:36:46.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:46.293+0000 7fad68200640 1 -- 172.21.15.6:0/596026819 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad60157240 con 0x7fad600af580 2024-03-04T09:36:46.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:46.293+0000 7fad68200640 1 -- 172.21.15.6:0/596026819 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad60157800 con 0x7fad600af580 2024-03-04T09:36:46.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:46.295+0000 7fad68200640 1 -- 172.21.15.6:0/596026819 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad60069510 con 0x7fad600af580 2024-03-04T09:36:46.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:46.296+0000 7fad5d600640 1 -- 172.21.15.6:0/596026819 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fad4c04c060 con 0x7fad600af580 2024-03-04T09:36:46.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:46.296+0000 7fad5d600640 1 --2- 172.21.15.6:0/596026819 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fad4006b4d0 0x7fad4006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:46.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:46.296+0000 7fad5d600640 1 -- 172.21.15.6:0/596026819 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fad4c0abc90 con 0x7fad600af580 2024-03-04T09:36:46.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:46.297+0000 7fad65800640 1 --2- 172.21.15.6:0/596026819 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fad4006b4d0 0x7fad4006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:46.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:46.298+0000 7fad65800640 1 --2- 172.21.15.6:0/596026819 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fad4006b4d0 0x7fad4006d990 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7fad54006fd0 tx=0x7fad54008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:36:46.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:46.301+0000 7fad5d600640 1 -- 172.21.15.6:0/596026819 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fad4c040070 con 0x7fad600af580 2024-03-04T09:36:46.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:46.618+0000 7fad68200640 1 -- 172.21.15.6:0/596026819 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fad6004f3a0 con 0x7fad600af580 2024-03-04T09:36:46.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:46.619+0000 7fad5d600640 1 -- 172.21.15.6:0/596026819 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3464 (secure 0 0 0) 0x7fad4c07aad0 con 0x7fad600af580 2024-03-04T09:36:46.616 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:36:46.616 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:35:26.088222+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"laggy_since":"2024-03-04T09:33:54.652601+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":9}]} 2024-03-04T09:36:46.617 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 101 2024-03-04T09:36:46.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:46.621+0000 7fad68200640 1 -- 172.21.15.6:0/596026819 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fad4006b4d0 msgr2=0x7fad4006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:46.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:46.621+0000 7fad68200640 1 --2- 172.21.15.6:0/596026819 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fad4006b4d0 0x7fad4006d990 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7fad54006fd0 tx=0x7fad54008040 comp rx=0 tx=0).stop 2024-03-04T09:36:46.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:46.621+0000 7fad68200640 1 -- 172.21.15.6:0/596026819 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fad600af580 msgr2=0x7fad60157d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:46.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:46.621+0000 7fad68200640 1 --2- 172.21.15.6:0/596026819 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fad600af580 0x7fad60157d20 secure :-1 s=READY pgs=1829 cs=0 l=1 rev1=1 crypto rx=0x7fad4c000c00 tx=0x7fad4c0046c0 comp rx=0 tx=0).stop 2024-03-04T09:36:46.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:46.622+0000 7fad68200640 1 -- 172.21.15.6:0/596026819 shutdown_connections 2024-03-04T09:36:46.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:46.622+0000 7fad68200640 1 --2- 172.21.15.6:0/596026819 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fad4006b4d0 0x7fad4006d990 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:46.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:46.622+0000 7fad68200640 1 --2- 172.21.15.6:0/596026819 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fad600cdc20 0x7fad6015e8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:46.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:46.622+0000 7fad68200640 1 --2- 172.21.15.6:0/596026819 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fad600af580 0x7fad60157d20 unknown :-1 s=CLOSED pgs=1829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:46.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:46.622+0000 7fad68200640 1 --2- 172.21.15.6:0/596026819 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fad600af1a0 0x7fad60147600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:46.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:46.622+0000 7fad68200640 1 -- 172.21.15.6:0/596026819 >> 172.21.15.6:0/596026819 conn(0x7fad6006afb0 msgr2=0x7fad600cc020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:36:46.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:46.622+0000 7fad68200640 1 -- 172.21.15.6:0/596026819 shutdown_connections 2024-03-04T09:36:46.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:46.622+0000 7fad68200640 1 -- 172.21.15.6:0/596026819 wait complete. 2024-03-04T09:36:46.840 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:32:57.159681+0000', 'modified': '2024-03-04T09:35:26.088222+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8214, 'mds_1': 6796}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6796': {'gid': 6796, 'name': 'd', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.146:6837/2458070840', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 2458070840}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 2458070840}]}, 'join_fscid': -1, 'laggy_since': '2024-03-04T09:33:54.652601+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8214': {'gid': 8214, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2446627517', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2446627517}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2446627517}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8214, 'qdb_cluster': [8214]} 2024-03-04T09:36:46.840 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6796:up:stopping 2024-03-04T09:36:47.841 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:36:47.969 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-04T09:36:48.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:48.075+0000 7f6e9f200640 1 Processor -- start 2024-03-04T09:36:48.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:48.076+0000 7f6e9f200640 1 -- start start 2024-03-04T09:36:48.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:48.078+0000 7f6e9f200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f6ea006c8b0 0x7f6ea006ef10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:48.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:48.079+0000 7f6e9f200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f6ea0066d00 0x7f6ea015ee90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:48.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:48.079+0000 7f6e9f200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ea00cf1b0 con 0x7f6ea00ce370 2024-03-04T09:36:48.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:48.079+0000 7f6e9f200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ea015e470 con 0x7f6ea006c8b0 2024-03-04T09:36:48.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:48.079+0000 7f6e9f200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ea015ea00 con 0x7f6ea0066d00 2024-03-04T09:36:48.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:48.079+0000 7f6e9de00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f6ea006c8b0 0x7f6ea006ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:48.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:48.079+0000 7f6e9de00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f6ea006c8b0 0x7f6ea006ef10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55644/0 (socket says 172.21.15.6:55644) 2024-03-04T09:36:48.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:48.079+0000 7f6e9de00640 1 -- 172.21.15.6:0/3034444487 learned_addr learned my addr 172.21.15.6:0/3034444487 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:36:48.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:48.079+0000 7f6e9d400640 1 --2- 172.21.15.6:0/3034444487 >> v2:172.21.15.146:3300/0 conn(0x7f6ea0066d00 0x7f6ea015ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:48.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:48.080+0000 7f6e9de00640 1 -- 172.21.15.6:0/3034444487 >> v2:172.21.15.146:3300/0 conn(0x7f6ea0066d00 msgr2=0x7f6ea015ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:36:48.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:48.080+0000 7f6e9de00640 1 --2- 172.21.15.6:0/3034444487 >> v2:172.21.15.146:3300/0 conn(0x7f6ea0066d00 0x7f6ea015ee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:48.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:48.080+0000 7f6e9de00640 1 -- 172.21.15.6:0/3034444487 >> v1:172.21.15.146:6789/0 conn(0x7f6ea00ce370 legacy=0x7f6ea00ce780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:36:48.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:48.080+0000 7f6e9de00640 1 -- 172.21.15.6:0/3034444487 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ea015f3d0 con 0x7f6ea006c8b0 2024-03-04T09:36:48.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:48.080+0000 7f6e9d400640 1 --2- 172.21.15.6:0/3034444487 >> v2:172.21.15.146:3300/0 conn(0x7f6ea0066d00 0x7f6ea015ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:36:48.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:48.081+0000 7f6e9de00640 1 --2- 172.21.15.6:0/3034444487 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6ea006c8b0 0x7f6ea006ef10 secure :-1 s=READY pgs=1830 cs=0 l=1 rev1=1 crypto rx=0x7f6ea0072d20 tx=0x7f6e9002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bc95d3ff2ef0f6e server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:36:48.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:48.081+0000 7f6e9ca00640 1 -- 172.21.15.6:0/3034444487 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e9002ca80 con 0x7f6ea006c8b0 2024-03-04T09:36:48.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:48.081+0000 7f6e9ca00640 1 -- 172.21.15.6:0/3034444487 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e9002cc20 con 0x7f6ea006c8b0 2024-03-04T09:36:48.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:48.081+0000 7f6e9ca00640 1 -- 172.21.15.6:0/3034444487 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e900343f0 con 0x7f6ea006c8b0 2024-03-04T09:36:48.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:48.082+0000 7f6e9f200640 1 -- 172.21.15.6:0/3034444487 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6ea006c8b0 msgr2=0x7f6ea006ef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:48.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:48.082+0000 7f6e9f200640 1 --2- 172.21.15.6:0/3034444487 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6ea006c8b0 0x7f6ea006ef10 secure :-1 s=READY pgs=1830 cs=0 l=1 rev1=1 crypto rx=0x7f6ea0072d20 tx=0x7f6e9002bee0 comp rx=0 tx=0).stop 2024-03-04T09:36:48.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:48.082+0000 7f6e9f200640 1 -- 172.21.15.6:0/3034444487 shutdown_connections 2024-03-04T09:36:48.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:48.082+0000 7f6e9f200640 1 --2- 172.21.15.6:0/3034444487 >> v2:172.21.15.146:3300/0 conn(0x7f6ea0066d00 0x7f6ea015ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:48.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:48.082+0000 7f6e9f200640 1 --2- 172.21.15.6:0/3034444487 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6ea006c8b0 0x7f6ea006ef10 unknown :-1 s=CLOSED pgs=1830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:48.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:48.082+0000 7f6e9f200640 1 -- 172.21.15.6:0/3034444487 >> 172.21.15.6:0/3034444487 conn(0x7f6ea00ccf70 msgr2=0x7f6ea00cd350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:36:48.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:48.082+0000 7f6e9f200640 1 -- 172.21.15.6:0/3034444487 shutdown_connections 2024-03-04T09:36:48.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:48.082+0000 7f6e9f200640 1 -- 172.21.15.6:0/3034444487 wait complete. 2024-03-04T09:36:48.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:48.084+0000 7f6e9f200640 1 Processor -- start 2024-03-04T09:36:48.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:48.085+0000 7f6e9f200640 1 -- start start 2024-03-04T09:36:48.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:48.086+0000 7f6e9f200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6ea0066d00 0x7f6ea00b3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:48.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:48.086+0000 7f6e9f200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6ea006c8b0 0x7f6ea0153560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:48.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:48.086+0000 7f6e9f200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6ea00ce370 0x7f6ea01595c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:48.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:48.086+0000 7f6e9f200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ea00cee90 con 0x7f6ea00ce370 2024-03-04T09:36:48.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:48.086+0000 7f6e9f200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ea015e690 con 0x7f6ea006c8b0 2024-03-04T09:36:48.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:48.086+0000 7f6e9f200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ea006afb0 con 0x7f6ea0066d00 2024-03-04T09:36:48.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:48.086+0000 7f6e9e800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6ea00ce370 0x7f6ea01595c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:48.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:48.086+0000 7f6e9e800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6ea00ce370 0x7f6ea01595c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55660/0 (socket says 172.21.15.6:55660) 2024-03-04T09:36:48.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:48.086+0000 7f6e9e800640 1 -- 172.21.15.6:0/1608347969 learned_addr learned my addr 172.21.15.6:0/1608347969 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:36:48.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:48.086+0000 7f6e9e800640 1 -- 172.21.15.6:0/1608347969 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6ea0066d00 msgr2=0x7f6ea00b3750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:48.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:48.087+0000 7f6e9e800640 1 --2- 172.21.15.6:0/1608347969 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6ea0066d00 0x7f6ea00b3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:48.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:48.087+0000 7f6e9e800640 1 -- 172.21.15.6:0/1608347969 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6ea006c8b0 msgr2=0x7f6ea0153560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:48.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:48.087+0000 7f6e9e800640 1 --2- 172.21.15.6:0/1608347969 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6ea006c8b0 0x7f6ea0153560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:48.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:48.087+0000 7f6e9e800640 1 -- 172.21.15.6:0/1608347969 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ea00b3d30 con 0x7f6ea00ce370 2024-03-04T09:36:48.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:48.087+0000 7f6e9e800640 1 --2- 172.21.15.6:0/1608347969 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6ea00ce370 0x7f6ea01595c0 secure :-1 s=READY pgs=1831 cs=0 l=1 rev1=1 crypto rx=0x7f6e9800b440 tx=0x7f6e9800b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:36:48.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:48.087+0000 7f6e96000640 1 -- 172.21.15.6:0/1608347969 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e98015020 con 0x7f6ea00ce370 2024-03-04T09:36:48.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:48.087+0000 7f6e96000640 1 -- 172.21.15.6:0/1608347969 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e98011830 con 0x7f6ea00ce370 2024-03-04T09:36:48.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:48.087+0000 7f6e9f200640 1 -- 172.21.15.6:0/1608347969 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ea00b4010 con 0x7f6ea00ce370 2024-03-04T09:36:48.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:48.087+0000 7f6e96000640 1 -- 172.21.15.6:0/1608347969 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e98011c20 con 0x7f6ea00ce370 2024-03-04T09:36:48.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:48.087+0000 7f6e9f200640 1 -- 172.21.15.6:0/1608347969 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ea00b45d0 con 0x7f6ea00ce370 2024-03-04T09:36:48.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:48.089+0000 7f6e9f200640 1 -- 172.21.15.6:0/1608347969 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ea004f330 con 0x7f6ea00ce370 2024-03-04T09:36:48.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:48.090+0000 7f6e96000640 1 -- 172.21.15.6:0/1608347969 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6e98018020 con 0x7f6ea00ce370 2024-03-04T09:36:48.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:48.091+0000 7f6e96000640 1 --2- 172.21.15.6:0/1608347969 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f6e8006b500 0x7f6e8006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:48.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:48.091+0000 7f6e96000640 1 -- 172.21.15.6:0/1608347969 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f6e98089d90 con 0x7f6ea00ce370 2024-03-04T09:36:48.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:48.092+0000 7f6e9de00640 1 --2- 172.21.15.6:0/1608347969 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f6e8006b500 0x7f6e8006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:48.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:48.093+0000 7f6e9de00640 1 --2- 172.21.15.6:0/1608347969 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f6e8006b500 0x7f6e8006d9c0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f6e90035000 tx=0x7f6e9002e000 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:36:48.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:48.096+0000 7f6e96000640 1 -- 172.21.15.6:0/1608347969 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6e9801f030 con 0x7f6ea00ce370 2024-03-04T09:36:48.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:48.368+0000 7f6e9f200640 1 -- 172.21.15.6:0/1608347969 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6ea00cdde0 con 0x7f6ea00ce370 2024-03-04T09:36:48.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:48.369+0000 7f6e96000640 1 -- 172.21.15.6:0/1608347969 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3464 (secure 0 0 0) 0x7f6e980172c0 con 0x7f6ea00ce370 2024-03-04T09:36:48.366 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:36:48.366 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:35:26.088222+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"laggy_since":"2024-03-04T09:33:54.652601+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":9}]} 2024-03-04T09:36:48.366 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 101 2024-03-04T09:36:48.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:48.372+0000 7f6e9f200640 1 -- 172.21.15.6:0/1608347969 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f6e8006b500 msgr2=0x7f6e8006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:48.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:48.372+0000 7f6e9f200640 1 --2- 172.21.15.6:0/1608347969 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f6e8006b500 0x7f6e8006d9c0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f6e90035000 tx=0x7f6e9002e000 comp rx=0 tx=0).stop 2024-03-04T09:36:48.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:48.372+0000 7f6e9f200640 1 -- 172.21.15.6:0/1608347969 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6ea00ce370 msgr2=0x7f6ea01595c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:48.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:48.372+0000 7f6e9f200640 1 --2- 172.21.15.6:0/1608347969 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6ea00ce370 0x7f6ea01595c0 secure :-1 s=READY pgs=1831 cs=0 l=1 rev1=1 crypto rx=0x7f6e9800b440 tx=0x7f6e9800b900 comp rx=0 tx=0).stop 2024-03-04T09:36:48.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:48.372+0000 7f6e9f200640 1 -- 172.21.15.6:0/1608347969 shutdown_connections 2024-03-04T09:36:48.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:48.372+0000 7f6e9f200640 1 --2- 172.21.15.6:0/1608347969 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6ea00ce370 0x7f6ea01595c0 unknown :-1 s=CLOSED pgs=1831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:48.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:48.372+0000 7f6e9f200640 1 --2- 172.21.15.6:0/1608347969 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6ea006c8b0 0x7f6ea0153560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:48.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:48.372+0000 7f6e9f200640 1 --2- 172.21.15.6:0/1608347969 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f6e8006b500 0x7f6e8006d9c0 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:48.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:48.372+0000 7f6e9f200640 1 --2- 172.21.15.6:0/1608347969 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6ea0066d00 0x7f6ea00b3750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:48.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:48.372+0000 7f6e9f200640 1 -- 172.21.15.6:0/1608347969 >> 172.21.15.6:0/1608347969 conn(0x7f6ea00ccf70 msgr2=0x7f6ea0160530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:36:48.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:48.372+0000 7f6e9f200640 1 -- 172.21.15.6:0/1608347969 shutdown_connections 2024-03-04T09:36:48.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:48.372+0000 7f6e9f200640 1 -- 172.21.15.6:0/1608347969 wait complete. 2024-03-04T09:36:48.589 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:32:57.159681+0000', 'modified': '2024-03-04T09:35:26.088222+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8214, 'mds_1': 6796}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6796': {'gid': 6796, 'name': 'd', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.146:6837/2458070840', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 2458070840}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 2458070840}]}, 'join_fscid': -1, 'laggy_since': '2024-03-04T09:33:54.652601+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8214': {'gid': 8214, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2446627517', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2446627517}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2446627517}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8214, 'qdb_cluster': [8214]} 2024-03-04T09:36:48.589 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6796:up:stopping 2024-03-04T09:36:49.590 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:36:49.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:49.824+0000 7fb716a00640 1 Processor -- start 2024-03-04T09:36:49.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:49.824+0000 7fb716a00640 1 -- start start 2024-03-04T09:36:49.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:49.826+0000 7fb716a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fb7100af580 0x7fb71015c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:49.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:49.826+0000 7fb716a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb71015cc40 con 0x7fb7100af1a0 2024-03-04T09:36:49.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:49.826+0000 7fb716a00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb71015cf20 con 0x7fb7100cdc20 2024-03-04T09:36:49.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:49.826+0000 7fb716a00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb71015d200 con 0x7fb7100af580 2024-03-04T09:36:49.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:49.826+0000 7fb70f400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fb7100af580 0x7fb71015c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:49.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:49.826+0000 7fb70fe00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fb7100af1a0 0x7fb7100b4c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:60284/0 (socket says 172.21.15.6:60284) 2024-03-04T09:36:49.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:49.826+0000 7fb70fe00640 1 -- 172.21.15.6:0/3043500801 learned_addr learned my addr 172.21.15.6:0/3043500801 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:36:49.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:49.827+0000 7fb70f400640 1 -- 172.21.15.6:0/3043500801 >> v1:172.21.15.146:6789/0 conn(0x7fb7100cdc20 legacy=0x7fb7100b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:36:49.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:49.827+0000 7fb70f400640 1 -- 172.21.15.6:0/3043500801 >> v1:172.21.15.6:6789/0 conn(0x7fb7100af1a0 legacy=0x7fb7100b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:49.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:49.827+0000 7fb70f400640 1 -- 172.21.15.6:0/3043500801 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb71015d4e0 con 0x7fb7100af580 2024-03-04T09:36:49.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:49.828+0000 7fb70f400640 1 --2- 172.21.15.6:0/3043500801 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb7100af580 0x7fb71015c600 secure :-1 s=READY pgs=1833 cs=0 l=1 rev1=1 crypto rx=0x7fb6fc0097a0 tx=0x7fb6fc02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf86dec360fd40a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:36:49.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:49.828+0000 7fb70ea00640 1 -- 172.21.15.6:0/3043500801 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6fc0315a0 con 0x7fb7100af580 2024-03-04T09:36:49.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:49.828+0000 7fb70ea00640 1 -- 172.21.15.6:0/3043500801 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6fc037070 con 0x7fb7100af580 2024-03-04T09:36:49.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:49.828+0000 7fb70ea00640 1 -- 172.21.15.6:0/3043500801 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6fc031960 con 0x7fb7100af580 2024-03-04T09:36:49.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:49.829+0000 7fb716a00640 1 -- 172.21.15.6:0/3043500801 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb7100af580 msgr2=0x7fb71015c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:49.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:49.829+0000 7fb716a00640 1 --2- 172.21.15.6:0/3043500801 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb7100af580 0x7fb71015c600 secure :-1 s=READY pgs=1833 cs=0 l=1 rev1=1 crypto rx=0x7fb6fc0097a0 tx=0x7fb6fc02b840 comp rx=0 tx=0).stop 2024-03-04T09:36:49.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:49.829+0000 7fb716a00640 1 -- 172.21.15.6:0/3043500801 shutdown_connections 2024-03-04T09:36:49.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:49.829+0000 7fb716a00640 1 --2- 172.21.15.6:0/3043500801 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb7100af580 0x7fb71015c600 unknown :-1 s=CLOSED pgs=1833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:49.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:49.829+0000 7fb716a00640 1 -- 172.21.15.6:0/3043500801 >> 172.21.15.6:0/3043500801 conn(0x7fb71006afb0 msgr2=0x7fb71006b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:36:49.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:49.829+0000 7fb716a00640 1 -- 172.21.15.6:0/3043500801 shutdown_connections 2024-03-04T09:36:49.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:49.829+0000 7fb716a00640 1 -- 172.21.15.6:0/3043500801 wait complete. 2024-03-04T09:36:49.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:49.831+0000 7fb716a00640 1 Processor -- start 2024-03-04T09:36:49.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:49.831+0000 7fb716a00640 1 -- start start 2024-03-04T09:36:49.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:49.833+0000 7fb716a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb7100af1a0 0x7fb71009d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:49.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:49.833+0000 7fb716a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb7100af580 0x7fb710147650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:49.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:49.833+0000 7fb716a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb7100cdc20 0x7fb71015b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:49.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:49.833+0000 7fb716a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb710155e70 con 0x7fb7100cdc20 2024-03-04T09:36:49.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:49.833+0000 7fb716a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7100b3770 con 0x7fb7100af580 2024-03-04T09:36:49.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:49.833+0000 7fb716a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb710155ff0 con 0x7fb7100af1a0 2024-03-04T09:36:49.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:49.833+0000 7fb714a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb7100cdc20 0x7fb71015b6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:49.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:49.833+0000 7fb714a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb7100cdc20 0x7fb71015b6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41834/0 (socket says 172.21.15.6:41834) 2024-03-04T09:36:49.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:49.833+0000 7fb714a00640 1 -- 172.21.15.6:0/3831955841 learned_addr learned my addr 172.21.15.6:0/3831955841 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:36:49.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:49.833+0000 7fb70fe00640 1 --2- 172.21.15.6:0/3831955841 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb7100af1a0 0x7fb71009d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:49.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:49.833+0000 7fb70f400640 1 --2- 172.21.15.6:0/3831955841 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb7100af580 0x7fb710147650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:49.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:49.834+0000 7fb714a00640 1 -- 172.21.15.6:0/3831955841 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb7100af1a0 msgr2=0x7fb71009d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:49.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:49.834+0000 7fb714a00640 1 --2- 172.21.15.6:0/3831955841 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb7100af1a0 0x7fb71009d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:49.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:49.834+0000 7fb714a00640 1 -- 172.21.15.6:0/3831955841 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb7100af580 msgr2=0x7fb710147650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:49.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:49.834+0000 7fb714a00640 1 --2- 172.21.15.6:0/3831955841 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb7100af580 0x7fb710147650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:49.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:49.834+0000 7fb714a00640 1 -- 172.21.15.6:0/3831955841 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb710147b90 con 0x7fb7100cdc20 2024-03-04T09:36:49.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:49.834+0000 7fb714a00640 1 --2- 172.21.15.6:0/3831955841 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb7100cdc20 0x7fb71015b6e0 secure :-1 s=READY pgs=1834 cs=0 l=1 rev1=1 crypto rx=0x7fb70800b970 tx=0x7fb70800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:36:49.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:49.834+0000 7fb707e00640 1 -- 172.21.15.6:0/3831955841 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb708019070 con 0x7fb7100cdc20 2024-03-04T09:36:49.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:49.834+0000 7fb707e00640 1 -- 172.21.15.6:0/3831955841 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb70800cb40 con 0x7fb7100cdc20 2024-03-04T09:36:49.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:49.834+0000 7fb716a00640 1 -- 172.21.15.6:0/3831955841 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb71009c340 con 0x7fb7100cdc20 2024-03-04T09:36:49.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:49.834+0000 7fb707e00640 1 -- 172.21.15.6:0/3831955841 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb708014440 con 0x7fb7100cdc20 2024-03-04T09:36:49.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:49.835+0000 7fb716a00640 1 -- 172.21.15.6:0/3831955841 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb71009c900 con 0x7fb7100cdc20 2024-03-04T09:36:49.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:49.836+0000 7fb716a00640 1 -- 172.21.15.6:0/3831955841 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb71004f3a0 con 0x7fb7100cdc20 2024-03-04T09:36:49.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:49.837+0000 7fb707e00640 1 -- 172.21.15.6:0/3831955841 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb70801c070 con 0x7fb7100cdc20 2024-03-04T09:36:49.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:49.838+0000 7fb707e00640 1 --2- 172.21.15.6:0/3831955841 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb6f006b4d0 0x7fb6f006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:49.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:49.838+0000 7fb707e00640 1 -- 172.21.15.6:0/3831955841 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fb708089d90 con 0x7fb7100cdc20 2024-03-04T09:36:49.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:49.839+0000 7fb70fe00640 1 --2- 172.21.15.6:0/3831955841 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb6f006b4d0 0x7fb6f006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:49.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:49.840+0000 7fb70fe00640 1 --2- 172.21.15.6:0/3831955841 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb6f006b4d0 0x7fb6f006d990 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fb700000f60 tx=0x7fb700009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:36:49.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:49.851+0000 7fb707e00640 1 -- 172.21.15.6:0/3831955841 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb708058c60 con 0x7fb7100cdc20 2024-03-04T09:36:49.971 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-04T09:36:50.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:50.088+0000 7fb716a00640 1 -- 172.21.15.6:0/3831955841 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb71009bff0 con 0x7fb7100cdc20 2024-03-04T09:36:50.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:50.089+0000 7fb707e00640 1 -- 172.21.15.6:0/3831955841 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3464 (secure 0 0 0) 0x7fb708058560 con 0x7fb7100cdc20 2024-03-04T09:36:50.086 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:36:50.086 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:35:26.088222+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"laggy_since":"2024-03-04T09:33:54.652601+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":9}]} 2024-03-04T09:36:50.087 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 101 2024-03-04T09:36:50.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:50.092+0000 7fb716a00640 1 -- 172.21.15.6:0/3831955841 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb6f006b4d0 msgr2=0x7fb6f006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:50.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:50.092+0000 7fb716a00640 1 --2- 172.21.15.6:0/3831955841 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb6f006b4d0 0x7fb6f006d990 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fb700000f60 tx=0x7fb700009040 comp rx=0 tx=0).stop 2024-03-04T09:36:50.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:50.092+0000 7fb716a00640 1 -- 172.21.15.6:0/3831955841 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb7100cdc20 msgr2=0x7fb71015b6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:50.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:50.092+0000 7fb716a00640 1 --2- 172.21.15.6:0/3831955841 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb7100cdc20 0x7fb71015b6e0 secure :-1 s=READY pgs=1834 cs=0 l=1 rev1=1 crypto rx=0x7fb70800b970 tx=0x7fb70800be30 comp rx=0 tx=0).stop 2024-03-04T09:36:50.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:50.092+0000 7fb716a00640 1 -- 172.21.15.6:0/3831955841 shutdown_connections 2024-03-04T09:36:50.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:50.092+0000 7fb716a00640 1 --2- 172.21.15.6:0/3831955841 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb7100cdc20 0x7fb71015b6e0 unknown :-1 s=CLOSED pgs=1834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:50.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:50.092+0000 7fb716a00640 1 --2- 172.21.15.6:0/3831955841 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb7100af580 0x7fb710147650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:50.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:50.092+0000 7fb716a00640 1 --2- 172.21.15.6:0/3831955841 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fb6f006b4d0 0x7fb6f006d990 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:50.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:50.092+0000 7fb716a00640 1 --2- 172.21.15.6:0/3831955841 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb7100af1a0 0x7fb71009d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:50.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:50.092+0000 7fb716a00640 1 -- 172.21.15.6:0/3831955841 >> 172.21.15.6:0/3831955841 conn(0x7fb71006afb0 msgr2=0x7fb71006b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:36:50.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:50.093+0000 7fb716a00640 1 -- 172.21.15.6:0/3831955841 shutdown_connections 2024-03-04T09:36:50.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:50.093+0000 7fb716a00640 1 -- 172.21.15.6:0/3831955841 wait complete. 2024-03-04T09:36:50.311 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:32:57.159681+0000', 'modified': '2024-03-04T09:35:26.088222+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8214, 'mds_1': 6796}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6796': {'gid': 6796, 'name': 'd', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.146:6837/2458070840', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 2458070840}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 2458070840}]}, 'join_fscid': -1, 'laggy_since': '2024-03-04T09:33:54.652601+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8214': {'gid': 8214, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2446627517', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2446627517}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2446627517}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8214, 'qdb_cluster': [8214]} 2024-03-04T09:36:50.312 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6796:up:stopping 2024-03-04T09:36:51.313 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:36:51.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:51.542+0000 7fc380e00640 1 Processor -- start 2024-03-04T09:36:51.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:51.542+0000 7fc380e00640 1 -- start start 2024-03-04T09:36:51.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:51.543+0000 7fc380e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc37c0d2960 0x7fc37c0d05c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:51.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:51.543+0000 7fc380e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc37c0d2dc0 con 0x7fc37c0d0b00 2024-03-04T09:36:51.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:51.543+0000 7fc380e00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc37c0d18c0 con 0x7fc37c15e070 2024-03-04T09:36:51.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:51.543+0000 7fc380e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc37c0d0f70 con 0x7fc37c0d2960 2024-03-04T09:36:51.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:51.544+0000 7fc37aa00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc37c0d2960 0x7fc37c0d05c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:51.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:51.544+0000 7fc37aa00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc37c0d2960 0x7fc37c0d05c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41846/0 (socket says 172.21.15.6:41846) 2024-03-04T09:36:51.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:51.544+0000 7fc37aa00640 1 -- 172.21.15.6:0/2512498665 learned_addr learned my addr 172.21.15.6:0/2512498665 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:36:51.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:51.544+0000 7fc37be00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fc37c0d0b00 0x7fc37c0b3630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:60298/0 (socket says 172.21.15.6:60298) 2024-03-04T09:36:51.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:51.544+0000 7fc37aa00640 1 -- 172.21.15.6:0/2512498665 >> v1:172.21.15.146:6789/0 conn(0x7fc37c15e070 legacy=0x7fc37c0cf410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:36:51.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:51.544+0000 7fc37aa00640 1 -- 172.21.15.6:0/2512498665 >> v1:172.21.15.6:6789/0 conn(0x7fc37c0d0b00 legacy=0x7fc37c0b3630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:51.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:51.544+0000 7fc37aa00640 1 -- 172.21.15.6:0/2512498665 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc37c0d1ba0 con 0x7fc37c0d2960 2024-03-04T09:36:51.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:51.545+0000 7fc37aa00640 1 --2- 172.21.15.6:0/2512498665 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc37c0d2960 0x7fc37c0d05c0 secure :-1 s=READY pgs=1836 cs=0 l=1 rev1=1 crypto rx=0x7fc364009e50 tx=0x7fc36402bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efc28753597ea0b5 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:36:51.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:51.546+0000 7fc37a000640 1 -- 172.21.15.6:0/2512498665 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc36402cb10 con 0x7fc37c0d2960 2024-03-04T09:36:51.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:51.546+0000 7fc37a000640 1 -- 172.21.15.6:0/2512498665 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc36402ccb0 con 0x7fc37c0d2960 2024-03-04T09:36:51.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:51.546+0000 7fc37a000640 1 -- 172.21.15.6:0/2512498665 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc36402ce60 con 0x7fc37c0d2960 2024-03-04T09:36:51.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:51.546+0000 7fc380e00640 1 -- 172.21.15.6:0/2512498665 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc37c0d2960 msgr2=0x7fc37c0d05c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:51.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:51.546+0000 7fc380e00640 1 --2- 172.21.15.6:0/2512498665 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc37c0d2960 0x7fc37c0d05c0 secure :-1 s=READY pgs=1836 cs=0 l=1 rev1=1 crypto rx=0x7fc364009e50 tx=0x7fc36402bf70 comp rx=0 tx=0).stop 2024-03-04T09:36:51.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:51.546+0000 7fc380e00640 1 -- 172.21.15.6:0/2512498665 shutdown_connections 2024-03-04T09:36:51.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:51.546+0000 7fc380e00640 1 --2- 172.21.15.6:0/2512498665 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc37c0d2960 0x7fc37c0d05c0 unknown :-1 s=CLOSED pgs=1836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:51.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:51.546+0000 7fc380e00640 1 -- 172.21.15.6:0/2512498665 >> 172.21.15.6:0/2512498665 conn(0x7fc37c08d2a0 msgr2=0x7fc37c0d3cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:36:51.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:51.547+0000 7fc380e00640 1 -- 172.21.15.6:0/2512498665 shutdown_connections 2024-03-04T09:36:51.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:51.547+0000 7fc380e00640 1 -- 172.21.15.6:0/2512498665 wait complete. 2024-03-04T09:36:51.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:51.548+0000 7fc380e00640 1 Processor -- start 2024-03-04T09:36:51.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:51.548+0000 7fc380e00640 1 -- start start 2024-03-04T09:36:51.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:51.549+0000 7fc380e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc37c0d0b00 0x7fc37c0c9c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:51.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:51.549+0000 7fc380e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc37c0d2960 0x7fc37c1f4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:51.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:51.549+0000 7fc380e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc37c15e070 0x7fc37c15e450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:51.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:51.549+0000 7fc380e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc37c0d2440 con 0x7fc37c0d0b00 2024-03-04T09:36:51.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:51.549+0000 7fc380e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc37c0d2160 con 0x7fc37c0d2960 2024-03-04T09:36:51.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:51.549+0000 7fc380e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc37c0d45b0 con 0x7fc37c15e070 2024-03-04T09:36:51.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:51.550+0000 7fc37b400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc37c0d0b00 0x7fc37c0c9c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:51.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:51.550+0000 7fc37b400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc37c0d0b00 0x7fc37c0c9c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41862/0 (socket says 172.21.15.6:41862) 2024-03-04T09:36:51.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:51.550+0000 7fc37b400640 1 -- 172.21.15.6:0/1510628781 learned_addr learned my addr 172.21.15.6:0/1510628781 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:36:51.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:51.550+0000 7fc37aa00640 1 --2- 172.21.15.6:0/1510628781 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc37c0d2960 0x7fc37c1f4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:51.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:51.550+0000 7fc37b400640 1 -- 172.21.15.6:0/1510628781 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc37c15e070 msgr2=0x7fc37c15e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:51.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:51.550+0000 7fc37b400640 1 --2- 172.21.15.6:0/1510628781 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc37c15e070 0x7fc37c15e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:51.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:51.550+0000 7fc37b400640 1 -- 172.21.15.6:0/1510628781 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc37c0d2960 msgr2=0x7fc37c1f4ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:51.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:51.550+0000 7fc37b400640 1 --2- 172.21.15.6:0/1510628781 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc37c0d2960 0x7fc37c1f4ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:51.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:51.550+0000 7fc37b400640 1 -- 172.21.15.6:0/1510628781 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc37c0ca170 con 0x7fc37c0d0b00 2024-03-04T09:36:51.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:51.550+0000 7fc37aa00640 1 --2- 172.21.15.6:0/1510628781 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc37c0d2960 0x7fc37c1f4ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:36:51.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:51.550+0000 7fc37b400640 1 --2- 172.21.15.6:0/1510628781 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc37c0d0b00 0x7fc37c0c9c30 secure :-1 s=READY pgs=1837 cs=0 l=1 rev1=1 crypto rx=0x7fc36c00b3f0 tx=0x7fc36c00b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:36:51.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:51.551+0000 7fc373400640 1 -- 172.21.15.6:0/1510628781 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc36c015020 con 0x7fc37c0d0b00 2024-03-04T09:36:51.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:51.551+0000 7fc373400640 1 -- 172.21.15.6:0/1510628781 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc36c011830 con 0x7fc37c0d0b00 2024-03-04T09:36:51.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:51.551+0000 7fc373400640 1 -- 172.21.15.6:0/1510628781 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc36c011c20 con 0x7fc37c0d0b00 2024-03-04T09:36:51.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:51.551+0000 7fc380e00640 1 -- 172.21.15.6:0/1510628781 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc37c1f84b0 con 0x7fc37c0d0b00 2024-03-04T09:36:51.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:51.551+0000 7fc380e00640 1 -- 172.21.15.6:0/1510628781 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc37c1f43f0 con 0x7fc37c0d0b00 2024-03-04T09:36:51.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:51.552+0000 7fc380e00640 1 -- 172.21.15.6:0/1510628781 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc37c1f4a00 con 0x7fc37c0d0b00 2024-03-04T09:36:51.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:51.553+0000 7fc373400640 1 -- 172.21.15.6:0/1510628781 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc36c018020 con 0x7fc37c0d0b00 2024-03-04T09:36:51.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:51.554+0000 7fc373400640 1 --2- 172.21.15.6:0/1510628781 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc35806b4d0 0x7fc35806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:51.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:51.554+0000 7fc373400640 1 -- 172.21.15.6:0/1510628781 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fc36c089db0 con 0x7fc37c0d0b00 2024-03-04T09:36:51.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:51.555+0000 7fc37aa00640 1 --2- 172.21.15.6:0/1510628781 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc35806b4d0 0x7fc35806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:51.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:51.556+0000 7fc37aa00640 1 --2- 172.21.15.6:0/1510628781 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc35806b4d0 0x7fc35806d990 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fc364035000 tx=0x7fc364030000 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:36:51.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:51.564+0000 7fc373400640 1 -- 172.21.15.6:0/1510628781 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc36c01f030 con 0x7fc37c0d0b00 2024-03-04T09:36:51.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:51.805+0000 7fc380e00640 1 -- 172.21.15.6:0/1510628781 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc37c0ce8c0 con 0x7fc37c0d0b00 2024-03-04T09:36:51.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:51.806+0000 7fc373400640 1 -- 172.21.15.6:0/1510628781 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3464 (secure 0 0 0) 0x7fc37c0ce8c0 con 0x7fc37c0d0b00 2024-03-04T09:36:51.804 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:36:51.804 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:35:26.088222+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"laggy_since":"2024-03-04T09:33:54.652601+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":9}]} 2024-03-04T09:36:51.805 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 101 2024-03-04T09:36:51.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:51.809+0000 7fc380e00640 1 -- 172.21.15.6:0/1510628781 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc35806b4d0 msgr2=0x7fc35806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:51.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:51.809+0000 7fc380e00640 1 --2- 172.21.15.6:0/1510628781 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc35806b4d0 0x7fc35806d990 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fc364035000 tx=0x7fc364030000 comp rx=0 tx=0).stop 2024-03-04T09:36:51.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:51.809+0000 7fc380e00640 1 -- 172.21.15.6:0/1510628781 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc37c0d0b00 msgr2=0x7fc37c0c9c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:51.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:51.809+0000 7fc380e00640 1 --2- 172.21.15.6:0/1510628781 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc37c0d0b00 0x7fc37c0c9c30 secure :-1 s=READY pgs=1837 cs=0 l=1 rev1=1 crypto rx=0x7fc36c00b3f0 tx=0x7fc36c00b8b0 comp rx=0 tx=0).stop 2024-03-04T09:36:51.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:51.809+0000 7fc380e00640 1 -- 172.21.15.6:0/1510628781 shutdown_connections 2024-03-04T09:36:51.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:51.809+0000 7fc380e00640 1 --2- 172.21.15.6:0/1510628781 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc35806b4d0 0x7fc35806d990 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:51.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:51.809+0000 7fc380e00640 1 --2- 172.21.15.6:0/1510628781 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc37c15e070 0x7fc37c15e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:51.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:51.809+0000 7fc380e00640 1 --2- 172.21.15.6:0/1510628781 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc37c0d2960 0x7fc37c1f4ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:51.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:51.809+0000 7fc380e00640 1 --2- 172.21.15.6:0/1510628781 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc37c0d0b00 0x7fc37c0c9c30 unknown :-1 s=CLOSED pgs=1837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:51.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:51.809+0000 7fc380e00640 1 -- 172.21.15.6:0/1510628781 >> 172.21.15.6:0/1510628781 conn(0x7fc37c08d2a0 msgr2=0x7fc37c06cb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:36:51.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:51.809+0000 7fc380e00640 1 -- 172.21.15.6:0/1510628781 shutdown_connections 2024-03-04T09:36:51.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:51.809+0000 7fc380e00640 1 -- 172.21.15.6:0/1510628781 wait complete. 2024-03-04T09:36:51.973 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-04T09:36:52.036 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:32:57.159681+0000', 'modified': '2024-03-04T09:35:26.088222+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8214, 'mds_1': 6796}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6796': {'gid': 6796, 'name': 'd', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.146:6837/2458070840', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 2458070840}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 2458070840}]}, 'join_fscid': -1, 'laggy_since': '2024-03-04T09:33:54.652601+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8214': {'gid': 8214, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2446627517', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2446627517}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2446627517}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8214, 'qdb_cluster': [8214]} 2024-03-04T09:36:52.036 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6796:up:stopping 2024-03-04T09:36:53.037 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T09:36:53.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:53.252+0000 7f2edf000640 1 Processor -- start 2024-03-04T09:36:53.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:53.252+0000 7f2edf000640 1 -- start start 2024-03-04T09:36:53.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:53.254+0000 7f2edf000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2ed806de20 0x7f2ed80d1c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:53.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:53.254+0000 7f2edf000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f2ed80d2170 0x7f2ed80cfd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:53.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:53.254+0000 7f2edf000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ed806c920 con 0x7f2ed815e080 2024-03-04T09:36:53.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:53.254+0000 7f2edf000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ed806d530 con 0x7f2ed80d2170 2024-03-04T09:36:53.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:53.254+0000 7f2edf000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ed806cc00 con 0x7f2ed806de20 2024-03-04T09:36:53.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:53.255+0000 7f2edd000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f2ed80d2170 0x7f2ed80cfd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:53.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:53.255+0000 7f2ecfe00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2ed806de20 0x7f2ed80d1c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:53.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:53.255+0000 7f2ecfe00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2ed806de20 0x7f2ed80d1c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.6:35938/0 (socket says 172.21.15.6:35938) 2024-03-04T09:36:53.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:53.255+0000 7f2ecfe00640 1 -- 172.21.15.6:0/1222988889 learned_addr learned my addr 172.21.15.6:0/1222988889 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:36:53.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:53.256+0000 7f2edd000640 1 -- 172.21.15.6:0/1222988889 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2ed806de20 msgr2=0x7f2ed80d1c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:36:53.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:53.256+0000 7f2edd000640 1 --2- 172.21.15.6:0/1222988889 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2ed806de20 0x7f2ed80d1c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:53.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:53.256+0000 7f2edd000640 1 -- 172.21.15.6:0/1222988889 >> v1:172.21.15.146:6789/0 conn(0x7f2ed815e080 legacy=0x7f2ed806c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:53.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:53.256+0000 7f2edd000640 1 -- 172.21.15.6:0/1222988889 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ed806cee0 con 0x7f2ed80d2170 2024-03-04T09:36:53.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:53.256+0000 7f2ecfe00640 1 --2- 172.21.15.6:0/1222988889 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2ed806de20 0x7f2ed80d1c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T09:36:53.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:53.257+0000 7f2edd000640 1 --2- 172.21.15.6:0/1222988889 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2ed80d2170 0x7f2ed80cfd50 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f2ed0009d90 tx=0x7f2ed002be20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=db762dfc69b4f217 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:36:53.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:53.257+0000 7f2ed7400640 1 -- 172.21.15.6:0/1222988889 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ed0039070 con 0x7f2ed80d2170 2024-03-04T09:36:53.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:53.257+0000 7f2ed7400640 1 -- 172.21.15.6:0/1222988889 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ed002cbd0 con 0x7f2ed80d2170 2024-03-04T09:36:53.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:53.257+0000 7f2ed7400640 1 -- 172.21.15.6:0/1222988889 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ed0034440 con 0x7f2ed80d2170 2024-03-04T09:36:53.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:53.258+0000 7f2edf000640 1 -- 172.21.15.6:0/1222988889 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2ed80d2170 msgr2=0x7f2ed80cfd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:53.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:53.258+0000 7f2edf000640 1 --2- 172.21.15.6:0/1222988889 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2ed80d2170 0x7f2ed80cfd50 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f2ed0009d90 tx=0x7f2ed002be20 comp rx=0 tx=0).stop 2024-03-04T09:36:53.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:53.258+0000 7f2edf000640 1 -- 172.21.15.6:0/1222988889 shutdown_connections 2024-03-04T09:36:53.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:53.258+0000 7f2edf000640 1 --2- 172.21.15.6:0/1222988889 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2ed80d2170 0x7f2ed80cfd50 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:53.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:53.258+0000 7f2edf000640 1 --2- 172.21.15.6:0/1222988889 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2ed806de20 0x7f2ed80d1c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:53.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:53.258+0000 7f2edf000640 1 -- 172.21.15.6:0/1222988889 >> 172.21.15.6:0/1222988889 conn(0x7f2ed809aa10 msgr2=0x7f2ed809ae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:36:53.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:53.258+0000 7f2edf000640 1 -- 172.21.15.6:0/1222988889 shutdown_connections 2024-03-04T09:36:53.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:53.258+0000 7f2edf000640 1 -- 172.21.15.6:0/1222988889 wait complete. 2024-03-04T09:36:53.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:53.260+0000 7f2edf000640 1 Processor -- start 2024-03-04T09:36:53.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:53.260+0000 7f2edf000640 1 -- start start 2024-03-04T09:36:53.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:53.261+0000 7f2edf000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2ed806de20 0x7f2ed80c57f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:53.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:53.261+0000 7f2edf000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2ed80d2170 0x7f2ed81499d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:53.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:53.261+0000 7f2edf000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2ed815e080 0x7f2ed80b7440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:53.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:53.261+0000 7f2edf000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ed80d0290 con 0x7f2ed80d2170 2024-03-04T09:36:53.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:53.261+0000 7f2edf000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ed815e460 con 0x7f2ed815e080 2024-03-04T09:36:53.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:53.261+0000 7f2edf000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ed80d0410 con 0x7f2ed806de20 2024-03-04T09:36:53.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:53.261+0000 7f2ecfe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2ed80d2170 0x7f2ed81499d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:53.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:53.261+0000 7f2ecfe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2ed80d2170 0x7f2ed81499d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41874/0 (socket says 172.21.15.6:41874) 2024-03-04T09:36:53.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:53.261+0000 7f2ecfe00640 1 -- 172.21.15.6:0/1364335183 learned_addr learned my addr 172.21.15.6:0/1364335183 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:36:53.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:53.262+0000 7f2ed7e00640 1 --2- 172.21.15.6:0/1364335183 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2ed806de20 0x7f2ed80c57f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:53.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:53.262+0000 7f2edd000640 1 --2- 172.21.15.6:0/1364335183 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2ed815e080 0x7f2ed80b7440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:53.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:53.262+0000 7f2ecfe00640 1 -- 172.21.15.6:0/1364335183 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2ed806de20 msgr2=0x7f2ed80c57f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:53.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:53.262+0000 7f2ecfe00640 1 --2- 172.21.15.6:0/1364335183 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2ed806de20 0x7f2ed80c57f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:53.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:53.262+0000 7f2ecfe00640 1 -- 172.21.15.6:0/1364335183 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2ed815e080 msgr2=0x7f2ed80b7440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:53.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:53.262+0000 7f2ecfe00640 1 --2- 172.21.15.6:0/1364335183 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2ed815e080 0x7f2ed80b7440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:53.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:53.262+0000 7f2ecfe00640 1 -- 172.21.15.6:0/1364335183 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ed8149f10 con 0x7f2ed80d2170 2024-03-04T09:36:53.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:53.262+0000 7f2ecfe00640 1 --2- 172.21.15.6:0/1364335183 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2ed80d2170 0x7f2ed81499d0 secure :-1 s=READY pgs=1838 cs=0 l=1 rev1=1 crypto rx=0x7f2ec000e9d0 tx=0x7f2ec000ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:36:53.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:53.262+0000 7f2ed4c00640 1 -- 172.21.15.6:0/1364335183 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ec0017070 con 0x7f2ed80d2170 2024-03-04T09:36:53.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:53.262+0000 7f2ed4c00640 1 -- 172.21.15.6:0/1364335183 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ec000ca60 con 0x7f2ed80d2170 2024-03-04T09:36:53.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:53.262+0000 7f2edf000640 1 -- 172.21.15.6:0/1364335183 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ed814a370 con 0x7f2ed80d2170 2024-03-04T09:36:53.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:53.262+0000 7f2ed4c00640 1 -- 172.21.15.6:0/1364335183 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ec000ce10 con 0x7f2ed80d2170 2024-03-04T09:36:53.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:53.263+0000 7f2edf000640 1 -- 172.21.15.6:0/1364335183 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ed814a930 con 0x7f2ed80d2170 2024-03-04T09:36:53.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:53.264+0000 7f2edf000640 1 -- 172.21.15.6:0/1364335183 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ed804f3a0 con 0x7f2ed80d2170 2024-03-04T09:36:53.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:53.265+0000 7f2ed4c00640 1 -- 172.21.15.6:0/1364335183 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2ec001a070 con 0x7f2ed80d2170 2024-03-04T09:36:53.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:53.266+0000 7f2ed4c00640 1 --2- 172.21.15.6:0/1364335183 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2eb406b4d0 0x7f2eb406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:53.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:53.266+0000 7f2ed4c00640 1 -- 172.21.15.6:0/1364335183 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f2ec008a530 con 0x7f2ed80d2170 2024-03-04T09:36:53.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:53.267+0000 7f2ed7e00640 1 --2- 172.21.15.6:0/1364335183 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2eb406b4d0 0x7f2eb406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:53.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:53.268+0000 7f2ed7e00640 1 --2- 172.21.15.6:0/1364335183 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2eb406b4d0 0x7f2eb406d990 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f2ec8007fd0 tx=0x7f2ec8009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:36:53.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:53.271+0000 7f2ed4c00640 1 -- 172.21.15.6:0/1364335183 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2ec0059360 con 0x7f2ed80d2170 2024-03-04T09:36:53.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:53.508+0000 7f2edf000640 1 -- 172.21.15.6:0/1364335183 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2ed80b21d0 con 0x7f2ed80d2170 2024-03-04T09:36:53.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:53.509+0000 7f2ed4c00640 1 -- 172.21.15.6:0/1364335183 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3464 (secure 0 0 0) 0x7f2ec0058c60 con 0x7f2ed80d2170 2024-03-04T09:36:53.506 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:36:53.506 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T09:32:57.159681+0000","modified":"2024-03-04T09:35:26.088222+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8214,"mds_1":6796},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6796":{"gid":6796,"name":"d","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.146:6837/2458070840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2458070840},{"type":"v1","addr":"172.21.15.146:6837","nonce":2458070840}]},"join_fscid":-1,"laggy_since":"2024-03-04T09:33:54.652601+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2446627517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2446627517},{"type":"v1","addr":"172.21.15.6:6834","nonce":2446627517}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":9}]} 2024-03-04T09:36:53.506 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 101 2024-03-04T09:36:53.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:53.511+0000 7f2edf000640 1 -- 172.21.15.6:0/1364335183 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2eb406b4d0 msgr2=0x7f2eb406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:53.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:53.511+0000 7f2edf000640 1 --2- 172.21.15.6:0/1364335183 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2eb406b4d0 0x7f2eb406d990 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f2ec8007fd0 tx=0x7f2ec8009040 comp rx=0 tx=0).stop 2024-03-04T09:36:53.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:53.511+0000 7f2edf000640 1 -- 172.21.15.6:0/1364335183 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2ed80d2170 msgr2=0x7f2ed81499d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:53.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:53.511+0000 7f2edf000640 1 --2- 172.21.15.6:0/1364335183 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2ed80d2170 0x7f2ed81499d0 secure :-1 s=READY pgs=1838 cs=0 l=1 rev1=1 crypto rx=0x7f2ec000e9d0 tx=0x7f2ec000ee90 comp rx=0 tx=0).stop 2024-03-04T09:36:53.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:53.512+0000 7f2edf000640 1 -- 172.21.15.6:0/1364335183 shutdown_connections 2024-03-04T09:36:53.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:53.512+0000 7f2edf000640 1 --2- 172.21.15.6:0/1364335183 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2ed815e080 0x7f2ed80b7440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:53.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:53.512+0000 7f2edf000640 1 --2- 172.21.15.6:0/1364335183 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2ed80d2170 0x7f2ed81499d0 unknown :-1 s=CLOSED pgs=1838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:53.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:53.512+0000 7f2edf000640 1 --2- 172.21.15.6:0/1364335183 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f2eb406b4d0 0x7f2eb406d990 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:53.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:53.512+0000 7f2edf000640 1 --2- 172.21.15.6:0/1364335183 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2ed806de20 0x7f2ed80c57f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:53.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:53.512+0000 7f2edf000640 1 -- 172.21.15.6:0/1364335183 >> 172.21.15.6:0/1364335183 conn(0x7f2ed809aa10 msgr2=0x7f2ed80aeb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:36:53.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:53.512+0000 7f2edf000640 1 -- 172.21.15.6:0/1364335183 shutdown_connections 2024-03-04T09:36:53.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:53.512+0000 7f2edf000640 1 -- 172.21.15.6:0/1364335183 wait complete. 2024-03-04T09:36:53.726 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T09:32:57.159681+0000', 'modified': '2024-03-04T09:35:26.088222+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8214, 'mds_1': 6796}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6796': {'gid': 6796, 'name': 'd', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.146:6837/2458070840', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 2458070840}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 2458070840}]}, 'join_fscid': -1, 'laggy_since': '2024-03-04T09:33:54.652601+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8214': {'gid': 8214, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2446627517', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2446627517}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2446627517}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8214, 'qdb_cluster': [8214]} 2024-03-04T09:36:53.726 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6796:up:stopping 2024-03-04T09:36:53.975 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-04T09:36:54.728 DEBUG:tasks.cephfs.filesystem:status = { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 9, "epoch": 101, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 9, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-04T09:32:57.159681+0000", "damaged": [], "data_pools": [ 19 ], "enabled": true, "epoch": 101, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_6796": { "addr": "172.21.15.146:6837/2458070840", "addrs": { "addrvec": [ { "addr": "172.21.15.146:6835", "nonce": 2458070840, "type": "v2" }, { "addr": "172.21.15.146:6837", "nonce": 2458070840, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6796, "incarnation": 94, "join_fscid": -1, "laggy_since": "2024-03-04T09:33:54.652601+0000", "name": "d", "rank": 1, "state": "up:stopping", "state_seq": 5 }, "gid_8214": { "addr": "172.21.15.6:6834/2446627517", "addrs": { "addrvec": [ { "addr": "172.21.15.6:6832", "nonce": 2446627517, "type": "v2" }, { "addr": "172.21.15.6:6834", "nonce": 2446627517, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 8214, "incarnation": 88, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 3 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 18, "modified": "2024-03-04T09:35:26.088222+0000", "qdb_cluster": [ 8214 ], "qdb_leader": 8214, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8214, "mds_1": 6796 } } } ], "standbys": [] } 2024-03-04T09:36:54.728 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-04T09:36:54.729 DEBUG:teuthology.orchestra.run.smithi006:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-04T09:36:54.850 DEBUG:teuthology.orchestra.run.smithi146:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-04T09:36:55.090 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-04T09:36:55.090 INFO:tasks.cephfs.mount:Terminating background process 2024-03-04T09:36:55.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:55 stdin-killer INFO: expiration expected; waiting 5 seconds for command to complete 2024-03-04T09:36:55.977 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-04T09:36:55.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:55 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-04T09:36:56.007 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-04T09:36:56.007 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T09:36:56.110 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-04T09:36:56.110 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T09:36:56.169 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T09:36:56.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.396+0000 7fd745e00640 1 Processor -- start 2024-03-04T09:36:56.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.396+0000 7fd745e00640 1 -- start start 2024-03-04T09:36:56.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.398+0000 7fd745e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd7400cdc20 0x7fd740071760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:56.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.398+0000 7fd745e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7400b4860 con 0x7fd7400b3c90 2024-03-04T09:36:56.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.398+0000 7fd745e00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7400b4b40 con 0x7fd7400cf5c0 2024-03-04T09:36:56.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.398+0000 7fd745e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7400b4e20 con 0x7fd7400cdc20 2024-03-04T09:36:56.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.398+0000 7fd73ea00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fd7400b3c90 0x7fd7400b4070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:60308/0 (socket says 172.21.15.6:60308) 2024-03-04T09:36:56.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.399+0000 7fd73ea00640 1 -- 172.21.15.6:0/2625134517 learned_addr learned my addr 172.21.15.6:0/2625134517 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:36:56.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.399+0000 7fd73f400640 1 --2- 172.21.15.6:0/2625134517 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd7400cdc20 0x7fd740071760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:56.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.399+0000 7fd73e000640 1 -- 172.21.15.6:0/2625134517 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1425718923 0 0) 0x7fd7400b4860 con 0x7fd7400b3c90 2024-03-04T09:36:56.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.399+0000 7fd73e000640 1 -- 172.21.15.6:0/2625134517 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd724003520 con 0x7fd7400b3c90 2024-03-04T09:36:56.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.400+0000 7fd73e000640 1 -- 172.21.15.6:0/2625134517 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2664116468 0 0) 0x7fd724003520 con 0x7fd7400b3c90 2024-03-04T09:36:56.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.400+0000 7fd73e000640 1 -- 172.21.15.6:0/2625134517 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd7400cdc20 msgr2=0x7fd740071760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:36:56.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.400+0000 7fd73e000640 1 --2- 172.21.15.6:0/2625134517 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd7400cdc20 0x7fd740071760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:56.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.400+0000 7fd73e000640 1 -- 172.21.15.6:0/2625134517 >> v1:172.21.15.146:6789/0 conn(0x7fd7400cf5c0 legacy=0x7fd740071050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:56.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.400+0000 7fd73e000640 1 -- 172.21.15.6:0/2625134517 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7400b5100 con 0x7fd7400b3c90 2024-03-04T09:36:56.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.400+0000 7fd73e000640 1 -- 172.21.15.6:0/2625134517 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7fd730001670 con 0x7fd7400b3c90 2024-03-04T09:36:56.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.400+0000 7fd73e000640 1 -- 172.21.15.6:0/2625134517 >> v1:172.21.15.6:6789/0 conn(0x7fd7400b3c90 legacy=0x7fd7400b4070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:56.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.400+0000 7fd73e000640 1 --2- 172.21.15.6:0/2625134517 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd724003a50 0x7fd724003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:56.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.400+0000 7fd73e000640 1 --2- 172.21.15.6:0/2625134517 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd724004800 0x7fd724004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:56.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.400+0000 7fd73e000640 1 --2- 172.21.15.6:0/2625134517 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd7400cdc20 0x7fd724005220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:56.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.400+0000 7fd73e000640 1 -- 172.21.15.6:0/2625134517 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7400b4e20 con 0x7fd724004800 2024-03-04T09:36:56.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.400+0000 7fd73e000640 1 -- 172.21.15.6:0/2625134517 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd72400de50 con 0x7fd724003a50 2024-03-04T09:36:56.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.400+0000 7fd73e000640 1 -- 172.21.15.6:0/2625134517 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd72400e050 con 0x7fd7400cdc20 2024-03-04T09:36:56.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.400+0000 7fd737e00640 1 --2- 172.21.15.6:0/2625134517 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd724004800 0x7fd724004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:56.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.400+0000 7fd73ea00640 1 --2- 172.21.15.6:0/2625134517 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd724003a50 0x7fd724003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:56.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.401+0000 7fd73f400640 1 --2- 172.21.15.6:0/2625134517 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd7400cdc20 0x7fd724005220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:56.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.401+0000 7fd737e00640 1 -- 172.21.15.6:0/2625134517 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd7400cdc20 msgr2=0x7fd724005220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:36:56.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.401+0000 7fd737e00640 1 --2- 172.21.15.6:0/2625134517 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd7400cdc20 0x7fd724005220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:56.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.401+0000 7fd737e00640 1 -- 172.21.15.6:0/2625134517 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd724003a50 msgr2=0x7fd724003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:36:56.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.401+0000 7fd737e00640 1 --2- 172.21.15.6:0/2625134517 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd724003a50 0x7fd724003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:56.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.401+0000 7fd737e00640 1 -- 172.21.15.6:0/2625134517 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd724003520 con 0x7fd724004800 2024-03-04T09:36:56.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.402+0000 7fd737e00640 1 --2- 172.21.15.6:0/2625134517 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd724004800 0x7fd724004c80 secure :-1 s=READY pgs=1840 cs=0 l=1 rev1=1 crypto rx=0x7fd728001750 tx=0x7fd724005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa3905ebe2863604 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:36:56.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.402+0000 7fd73e000640 1 -- 172.21.15.6:0/2625134517 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd728001370 con 0x7fd724004800 2024-03-04T09:36:56.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.402+0000 7fd73e000640 1 -- 172.21.15.6:0/2625134517 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7280052d0 con 0x7fd724004800 2024-03-04T09:36:56.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.402+0000 7fd73e000640 1 -- 172.21.15.6:0/2625134517 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd728003830 con 0x7fd724004800 2024-03-04T09:36:56.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.403+0000 7fd745e00640 1 -- 172.21.15.6:0/2625134517 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd724004800 msgr2=0x7fd724004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:56.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.403+0000 7fd745e00640 1 --2- 172.21.15.6:0/2625134517 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd724004800 0x7fd724004c80 secure :-1 s=READY pgs=1840 cs=0 l=1 rev1=1 crypto rx=0x7fd728001750 tx=0x7fd724005830 comp rx=0 tx=0).stop 2024-03-04T09:36:56.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.403+0000 7fd73f400640 1 -- 172.21.15.6:0/2625134517 reap_dead start 2024-03-04T09:36:56.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.403+0000 7fd745e00640 1 -- 172.21.15.6:0/2625134517 shutdown_connections 2024-03-04T09:36:56.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.403+0000 7fd745e00640 1 -- 172.21.15.6:0/2625134517 >> 172.21.15.6:0/2625134517 conn(0x7fd74006afb0 msgr2=0x7fd74006b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:36:56.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.403+0000 7fd745e00640 1 -- 172.21.15.6:0/2625134517 shutdown_connections 2024-03-04T09:36:56.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.403+0000 7fd745e00640 1 -- 172.21.15.6:0/2625134517 wait complete. 2024-03-04T09:36:56.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.405+0000 7fd745e00640 1 Processor -- start 2024-03-04T09:36:56.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.405+0000 7fd745e00640 1 -- start start 2024-03-04T09:36:56.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.406+0000 7fd745e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd724004800 0x7fd7401572a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:56.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.406+0000 7fd745e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd7401e0ca0 0x7fd7401577e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:56.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.406+0000 7fd745e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd7401473b0 0x7fd740147860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:56.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.406+0000 7fd745e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd740071760 con 0x7fd7401473b0 2024-03-04T09:36:56.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.406+0000 7fd745e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7400ce060 con 0x7fd7401e0ca0 2024-03-04T09:36:56.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.406+0000 7fd745e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7400718e0 con 0x7fd724004800 2024-03-04T09:36:56.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.406+0000 7fd73f400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd7401473b0 0x7fd740147860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:56.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.406+0000 7fd73f400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd7401473b0 0x7fd740147860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41902/0 (socket says 172.21.15.6:41902) 2024-03-04T09:36:56.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.406+0000 7fd73f400640 1 -- 172.21.15.6:0/405891805 learned_addr learned my addr 172.21.15.6:0/405891805 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:36:56.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.406+0000 7fd73ea00640 1 --2- 172.21.15.6:0/405891805 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd724004800 0x7fd7401572a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:56.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.406+0000 7fd737e00640 1 --2- 172.21.15.6:0/405891805 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd7401e0ca0 0x7fd7401577e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:56.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.406+0000 7fd73f400640 1 -- 172.21.15.6:0/405891805 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd724004800 msgr2=0x7fd7401572a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:56.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.406+0000 7fd73f400640 1 --2- 172.21.15.6:0/405891805 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd724004800 0x7fd7401572a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:56.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.406+0000 7fd73f400640 1 -- 172.21.15.6:0/405891805 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd7401e0ca0 msgr2=0x7fd7401577e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:56.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.406+0000 7fd73f400640 1 --2- 172.21.15.6:0/405891805 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd7401e0ca0 0x7fd7401577e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:56.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.406+0000 7fd73f400640 1 -- 172.21.15.6:0/405891805 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7401e1120 con 0x7fd7401473b0 2024-03-04T09:36:56.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.406+0000 7fd73ea00640 1 --2- 172.21.15.6:0/405891805 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd724004800 0x7fd7401572a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:36:56.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.407+0000 7fd73f400640 1 --2- 172.21.15.6:0/405891805 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd7401473b0 0x7fd740147860 secure :-1 s=READY pgs=1841 cs=0 l=1 rev1=1 crypto rx=0x7fd7400cf9a0 tx=0x7fd7380115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:36:56.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.407+0000 7fd737400640 1 -- 172.21.15.6:0/405891805 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd738003990 con 0x7fd7401473b0 2024-03-04T09:36:56.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.407+0000 7fd737400640 1 -- 172.21.15.6:0/405891805 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd73801b070 con 0x7fd7401473b0 2024-03-04T09:36:56.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.407+0000 7fd737400640 1 -- 172.21.15.6:0/405891805 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd738003b70 con 0x7fd7401473b0 2024-03-04T09:36:56.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.407+0000 7fd745e00640 1 -- 172.21.15.6:0/405891805 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7401f0de0 con 0x7fd7401473b0 2024-03-04T09:36:56.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.407+0000 7fd745e00640 1 -- 172.21.15.6:0/405891805 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7401f13a0 con 0x7fd7401473b0 2024-03-04T09:36:56.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.408+0000 7fd745e00640 1 -- 172.21.15.6:0/405891805 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd740069510 con 0x7fd7401473b0 2024-03-04T09:36:56.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.411+0000 7fd737400640 1 -- 172.21.15.6:0/405891805 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd738005d10 con 0x7fd7401473b0 2024-03-04T09:36:56.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.411+0000 7fd737400640 1 --2- 172.21.15.6:0/405891805 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd71c06b500 0x7fd71c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:56.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.411+0000 7fd737400640 1 -- 172.21.15.6:0/405891805 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fd73808a7f0 con 0x7fd7401473b0 2024-03-04T09:36:56.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.412+0000 7fd73ea00640 1 --2- 172.21.15.6:0/405891805 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd71c06b500 0x7fd71c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:56.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.413+0000 7fd73ea00640 1 --2- 172.21.15.6:0/405891805 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd71c06b500 0x7fd71c06d9c0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fd730001eb0 tx=0x7fd730000f30 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:36:56.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.417+0000 7fd737400640 1 -- 172.21.15.6:0/405891805 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd738025020 con 0x7fd7401473b0 2024-03-04T09:36:56.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.618+0000 7fd745e00640 1 -- 172.21.15.6:0/405891805 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd74004f3a0 con 0x7fd7401473b0 2024-03-04T09:36:56.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.622+0000 7fd737400640 1 -- 172.21.15.6:0/405891805 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v131) v1 ==== 74+0+16673 (secure 0 0 0) 0x7fd7380596a0 con 0x7fd7401473b0 2024-03-04T09:36:56.619 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-04T09:36:56.619 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":131,"fsid":"edce0578-5ae4-4402-b45a-552bdef84d7c","created":"2024-03-04T09:24:54.523780+0000","modified":"2024-03-04T09:33:14.677348+0000","last_up_change":"2024-03-04T09:25:07.089049+0000","last_in_change":"2024-03-04T09:24:57.982077+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T09:25:07.935198+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-03-04T09:32:54.852054+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"126","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-03-04T09:32:56.177848+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"127","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a96574f-81f1-4bc6-8463-757520b84a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6825","nonce":1364449401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6827","nonce":1364449401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6831","nonce":1364449401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1364449401},{"type":"v1","addr":"172.21.15.6:6829","nonce":1364449401}]},"public_addr":"172.21.15.6:6825/1364449401","cluster_addr":"172.21.15.6:6827/1364449401","heartbeat_back_addr":"172.21.15.6:6831/1364449401","heartbeat_front_addr":"172.21.15.6:6829/1364449401","state":["exists","up"]},{"osd":1,"uuid":"7d79b9c1-6b87-476a-bc36-7db5ee64cd90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6809","nonce":2494103078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6811","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6813","nonce":2494103078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6821","nonce":2494103078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2494103078},{"type":"v1","addr":"172.21.15.6:6817","nonce":2494103078}]},"public_addr":"172.21.15.6:6809/2494103078","cluster_addr":"172.21.15.6:6813/2494103078","heartbeat_back_addr":"172.21.15.6:6821/2494103078","heartbeat_front_addr":"172.21.15.6:6817/2494103078","state":["exists","up"]},{"osd":2,"uuid":"492331b9-4ee9-40b6-bacd-85d44cd6c45a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6812","nonce":1500066681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6816","nonce":1500066681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6824","nonce":1500066681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1500066681},{"type":"v1","addr":"172.21.15.6:6820","nonce":1500066681}]},"public_addr":"172.21.15.6:6812/1500066681","cluster_addr":"172.21.15.6:6816/1500066681","heartbeat_back_addr":"172.21.15.6:6824/1500066681","heartbeat_front_addr":"172.21.15.6:6820/1500066681","state":["exists","up"]},{"osd":3,"uuid":"21bb2efe-9ba7-4bdc-bc31-180cd1102e21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6801","nonce":1558806123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6803","nonce":1558806123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6807","nonce":1558806123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":1558806123},{"type":"v1","addr":"172.21.15.6:6805","nonce":1558806123}]},"public_addr":"172.21.15.6:6801/1558806123","cluster_addr":"172.21.15.6:6803/1558806123","heartbeat_back_addr":"172.21.15.6:6807/1558806123","heartbeat_front_addr":"172.21.15.6:6805/1558806123","state":["exists","up"]},{"osd":4,"uuid":"6e2c4b71-19d9-4737-83d9-558705d8644a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6801","nonce":1526135780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6803","nonce":1526135780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6807","nonce":1526135780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":1526135780},{"type":"v1","addr":"172.21.15.146:6805","nonce":1526135780}]},"public_addr":"172.21.15.146:6801/1526135780","cluster_addr":"172.21.15.146:6803/1526135780","heartbeat_back_addr":"172.21.15.146:6807/1526135780","heartbeat_front_addr":"172.21.15.146:6805/1526135780","state":["exists","up"]},{"osd":5,"uuid":"fb69c525-f1be-4b7d-ab32-6c29b0d6fddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6809","nonce":460262507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6811","nonce":460262507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6815","nonce":460262507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":460262507},{"type":"v1","addr":"172.21.15.146:6813","nonce":460262507}]},"public_addr":"172.21.15.146:6809/460262507","cluster_addr":"172.21.15.146:6811/460262507","heartbeat_back_addr":"172.21.15.146:6815/460262507","heartbeat_front_addr":"172.21.15.146:6813/460262507","state":["exists","up"]},{"osd":6,"uuid":"c32e7fed-370d-4401-8ce0-6a4335b0d168","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6823","nonce":1392897493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6827","nonce":1392897493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6831","nonce":1392897493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":1392897493},{"type":"v1","addr":"172.21.15.146:6829","nonce":1392897493}]},"public_addr":"172.21.15.146:6823/1392897493","cluster_addr":"172.21.15.146:6827/1392897493","heartbeat_back_addr":"172.21.15.146:6831/1392897493","heartbeat_front_addr":"172.21.15.146:6829/1392897493","state":["exists","up"]},{"osd":7,"uuid":"75a79b44-fa5f-426d-a7d3-71f21f78835c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6817","nonce":3268323156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6819","nonce":3268323156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6825","nonce":3268323156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3268323156},{"type":"v1","addr":"172.21.15.146:6822","nonce":3268323156}]},"public_addr":"172.21.15.146:6817/3268323156","cluster_addr":"172.21.15.146:6819/3268323156","heartbeat_back_addr":"172.21.15.146:6825/3268323156","heartbeat_front_addr":"172.21.15.146:6822/3268323156","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.310437+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.295850+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.294674+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.526855+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.300023+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.595160+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.346046+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T09:25:03.558741+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"19.1f","mappings":[{"from":2,"to":0}]},{"pgid":"19.25","mappings":[{"from":2,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T09:36:56.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.624+0000 7fd745e00640 1 -- 172.21.15.6:0/405891805 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd71c06b500 msgr2=0x7fd71c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:56.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.624+0000 7fd745e00640 1 --2- 172.21.15.6:0/405891805 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd71c06b500 0x7fd71c06d9c0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fd730001eb0 tx=0x7fd730000f30 comp rx=0 tx=0).stop 2024-03-04T09:36:56.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.624+0000 7fd745e00640 1 -- 172.21.15.6:0/405891805 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd7401473b0 msgr2=0x7fd740147860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:56.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.624+0000 7fd745e00640 1 --2- 172.21.15.6:0/405891805 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd7401473b0 0x7fd740147860 secure :-1 s=READY pgs=1841 cs=0 l=1 rev1=1 crypto rx=0x7fd7400cf9a0 tx=0x7fd7380115a0 comp rx=0 tx=0).stop 2024-03-04T09:36:56.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.624+0000 7fd745e00640 1 -- 172.21.15.6:0/405891805 shutdown_connections 2024-03-04T09:36:56.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.624+0000 7fd745e00640 1 --2- 172.21.15.6:0/405891805 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd7401473b0 0x7fd740147860 unknown :-1 s=CLOSED pgs=1841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:56.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.624+0000 7fd745e00640 1 --2- 172.21.15.6:0/405891805 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd7401e0ca0 0x7fd7401577e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:56.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.624+0000 7fd745e00640 1 --2- 172.21.15.6:0/405891805 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fd71c06b500 0x7fd71c06d9c0 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:56.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.624+0000 7fd745e00640 1 --2- 172.21.15.6:0/405891805 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd724004800 0x7fd7401572a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:56.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.624+0000 7fd745e00640 1 -- 172.21.15.6:0/405891805 >> 172.21.15.6:0/405891805 conn(0x7fd74006afb0 msgr2=0x7fd740072150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:36:56.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.625+0000 7fd745e00640 1 -- 172.21.15.6:0/405891805 shutdown_connections 2024-03-04T09:36:56.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.625+0000 7fd745e00640 1 -- 172.21.15.6:0/405891805 wait complete. 2024-03-04T09:36:56.641 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T09:25:07.935198+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 18, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-04T09:32:54.852054+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '126', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 19, 'pool_name': 'cephfs_data', 'create_time': '2024-03-04T09:32:56.177848+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '127', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-04T09:36:56.642 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-04T09:36:56.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.863+0000 7f792de00640 1 Processor -- start 2024-03-04T09:36:56.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.863+0000 7f792de00640 1 -- start start 2024-03-04T09:36:56.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.865+0000 7f792de00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f79280af930 0x7f79280cfa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:56.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.865+0000 7f792de00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f792815bef0 0x7f792815c380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:56.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.865+0000 7f792de00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79280b3770 con 0x7f792806f540 2024-03-04T09:36:56.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.865+0000 7f792de00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79280b3520 con 0x7f79280af930 2024-03-04T09:36:56.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.865+0000 7f792de00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79280b3a50 con 0x7f792815bef0 2024-03-04T09:36:56.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.865+0000 7f792ca00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f792806f540 0x7f792806f920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:60318/0 (socket says 172.21.15.6:60318) 2024-03-04T09:36:56.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.865+0000 7f792ca00640 1 -- 172.21.15.6:0/158786907 learned_addr learned my addr 172.21.15.6:0/158786907 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:36:56.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.865+0000 7f792d400640 1 --2- 172.21.15.6:0/158786907 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f792815bef0 0x7f792815c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:56.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.865+0000 7f7927400640 1 -- 172.21.15.6:0/158786907 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3615561568 0 0) 0x7f79280b3770 con 0x7f792806f540 2024-03-04T09:36:56.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.865+0000 7f7927400640 1 -- 172.21.15.6:0/158786907 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7910003520 con 0x7f792806f540 2024-03-04T09:36:56.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.865+0000 7f7927e00640 1 --2- 172.21.15.6:0/158786907 >> v2:172.21.15.146:3300/0 conn(0x7f79280af930 0x7f79280cfa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:56.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.866+0000 7f7927400640 1 -- 172.21.15.6:0/158786907 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2884340366 0 0) 0x7f7910003520 con 0x7f792806f540 2024-03-04T09:36:56.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.866+0000 7f7927400640 1 -- 172.21.15.6:0/158786907 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f792815bef0 msgr2=0x7f792815c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:36:56.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.866+0000 7f7927400640 1 --2- 172.21.15.6:0/158786907 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f792815bef0 0x7f792815c380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:56.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.866+0000 7f7927400640 1 -- 172.21.15.6:0/158786907 >> v2:172.21.15.146:3300/0 conn(0x7f79280af930 msgr2=0x7f79280cfa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:36:56.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.866+0000 7f7927400640 1 --2- 172.21.15.6:0/158786907 >> v2:172.21.15.146:3300/0 conn(0x7f79280af930 0x7f79280cfa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:56.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.866+0000 7f7927400640 1 -- 172.21.15.6:0/158786907 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79280b3d30 con 0x7f792806f540 2024-03-04T09:36:56.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.866+0000 7f7927400640 1 -- 172.21.15.6:0/158786907 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 774119140 0 0) 0x7f791c001670 con 0x7f792806f540 2024-03-04T09:36:56.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.866+0000 7f7927400640 1 -- 172.21.15.6:0/158786907 >> v1:172.21.15.6:6789/0 conn(0x7f792806f540 legacy=0x7f792806f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:56.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.866+0000 7f7927400640 1 --2- 172.21.15.6:0/158786907 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f792815bef0 0x7f79100039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:56.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.866+0000 7f7927400640 1 --2- 172.21.15.6:0/158786907 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7910003f80 0x7f7910004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:56.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.866+0000 7f7927400640 1 --2- 172.21.15.6:0/158786907 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7910004d40 0x7f79100051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:56.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.866+0000 7f7927400640 1 -- 172.21.15.6:0/158786907 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79280b3520 con 0x7f7910004d40 2024-03-04T09:36:56.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.866+0000 7f7927400640 1 -- 172.21.15.6:0/158786907 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79280b3a50 con 0x7f7910003f80 2024-03-04T09:36:56.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.866+0000 7f7927400640 1 -- 172.21.15.6:0/158786907 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f791000b9c0 con 0x7f792815bef0 2024-03-04T09:36:56.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.866+0000 7f792d400640 1 --2- 172.21.15.6:0/158786907 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7910004d40 0x7f79100051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:56.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.867+0000 7f7927e00640 1 --2- 172.21.15.6:0/158786907 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7910003f80 0x7f7910004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:56.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.867+0000 7f792ca00640 1 --2- 172.21.15.6:0/158786907 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f792815bef0 0x7f79100039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:56.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.867+0000 7f792d400640 1 -- 172.21.15.6:0/158786907 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f792815bef0 msgr2=0x7f79100039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:36:56.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.867+0000 7f792d400640 1 --2- 172.21.15.6:0/158786907 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f792815bef0 0x7f79100039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:56.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.867+0000 7f792d400640 1 -- 172.21.15.6:0/158786907 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7910003f80 msgr2=0x7f7910004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:36:56.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.867+0000 7f792d400640 1 --2- 172.21.15.6:0/158786907 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7910003f80 0x7f7910004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:56.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.867+0000 7f792d400640 1 -- 172.21.15.6:0/158786907 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7910003520 con 0x7f7910004d40 2024-03-04T09:36:56.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.867+0000 7f792ca00640 1 --2- 172.21.15.6:0/158786907 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f792815bef0 0x7f79100039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T09:36:56.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.868+0000 7f792d400640 1 --2- 172.21.15.6:0/158786907 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7910004d40 0x7f79100051c0 secure :-1 s=READY pgs=1843 cs=0 l=1 rev1=1 crypto rx=0x7f792000e170 tx=0x7f7910005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d301cd836ab31b71 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:36:56.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.868+0000 7f7927400640 1 -- 172.21.15.6:0/158786907 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7920039070 con 0x7f7910004d40 2024-03-04T09:36:56.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.868+0000 7f7927400640 1 -- 172.21.15.6:0/158786907 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f792002cbf0 con 0x7f7910004d40 2024-03-04T09:36:56.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.868+0000 7f7927400640 1 -- 172.21.15.6:0/158786907 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7920009c10 con 0x7f7910004d40 2024-03-04T09:36:56.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.869+0000 7f792de00640 1 -- 172.21.15.6:0/158786907 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7910004d40 msgr2=0x7f79100051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:56.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.869+0000 7f792de00640 1 --2- 172.21.15.6:0/158786907 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7910004d40 0x7f79100051c0 secure :-1 s=READY pgs=1843 cs=0 l=1 rev1=1 crypto rx=0x7f792000e170 tx=0x7f7910005830 comp rx=0 tx=0).stop 2024-03-04T09:36:56.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.869+0000 7f792d400640 1 -- 172.21.15.6:0/158786907 reap_dead start 2024-03-04T09:36:56.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.869+0000 7f792de00640 1 -- 172.21.15.6:0/158786907 shutdown_connections 2024-03-04T09:36:56.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.869+0000 7f792de00640 1 -- 172.21.15.6:0/158786907 >> 172.21.15.6:0/158786907 conn(0x7f79280723d0 msgr2=0x7f792806afb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:36:56.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.869+0000 7f792de00640 1 -- 172.21.15.6:0/158786907 shutdown_connections 2024-03-04T09:36:56.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.869+0000 7f792de00640 1 -- 172.21.15.6:0/158786907 wait complete. 2024-03-04T09:36:56.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.871+0000 7f792de00640 1 Processor -- start 2024-03-04T09:36:56.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.871+0000 7f792de00640 1 -- start start 2024-03-04T09:36:56.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.872+0000 7f792de00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7910004d40 0x7f792815a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:56.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.872+0000 7f792de00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f79281494a0 0x7f79281e2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:56.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.872+0000 7f792de00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f792806dde0 0x7f792806e260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:56.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.872+0000 7f792de00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f792815d2b0 con 0x7f79281494a0 2024-03-04T09:36:56.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.872+0000 7f792de00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79280cffd0 con 0x7f7910004d40 2024-03-04T09:36:56.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.872+0000 7f792de00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f792815d430 con 0x7f792806dde0 2024-03-04T09:36:56.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.872+0000 7f7927e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f79281494a0 0x7f79281e2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:56.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.873+0000 7f7927e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f79281494a0 0x7f79281e2cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41918/0 (socket says 172.21.15.6:41918) 2024-03-04T09:36:56.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.873+0000 7f7927e00640 1 -- 172.21.15.6:0/4262127767 learned_addr learned my addr 172.21.15.6:0/4262127767 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:36:56.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.873+0000 7f792ca00640 1 --2- 172.21.15.6:0/4262127767 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7910004d40 0x7f792815a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:56.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.873+0000 7f7927e00640 1 -- 172.21.15.6:0/4262127767 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f792806dde0 msgr2=0x7f792806e260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:56.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.873+0000 7f792d400640 1 --2- 172.21.15.6:0/4262127767 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f792806dde0 0x7f792806e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:56.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.873+0000 7f7927e00640 1 --2- 172.21.15.6:0/4262127767 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f792806dde0 0x7f792806e260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:56.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.873+0000 7f7927e00640 1 -- 172.21.15.6:0/4262127767 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7910004d40 msgr2=0x7f792815a060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:56.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.873+0000 7f7927e00640 1 --2- 172.21.15.6:0/4262127767 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7910004d40 0x7f792815a060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:56.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.873+0000 7f7927e00640 1 -- 172.21.15.6:0/4262127767 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f792815ad60 con 0x7f79281494a0 2024-03-04T09:36:56.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.873+0000 7f792ca00640 1 --2- 172.21.15.6:0/4262127767 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7910004d40 0x7f792815a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T09:36:56.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.874+0000 7f7927e00640 1 --2- 172.21.15.6:0/4262127767 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f79281494a0 0x7f79281e2cf0 secure :-1 s=READY pgs=1844 cs=0 l=1 rev1=1 crypto rx=0x7f79140116d0 tx=0x7f7914011b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:36:56.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.874+0000 7f7924c00640 1 -- 172.21.15.6:0/4262127767 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f791400f9c0 con 0x7f79281494a0 2024-03-04T09:36:56.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.874+0000 7f7924c00640 1 -- 172.21.15.6:0/4262127767 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f791400fb60 con 0x7f79281494a0 2024-03-04T09:36:56.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.874+0000 7f7924c00640 1 -- 172.21.15.6:0/4262127767 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7914015460 con 0x7f79281494a0 2024-03-04T09:36:56.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.874+0000 7f792de00640 1 -- 172.21.15.6:0/4262127767 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79281e24e0 con 0x7f79281494a0 2024-03-04T09:36:56.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.874+0000 7f792de00640 1 -- 172.21.15.6:0/4262127767 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f792815a840 con 0x7f79281494a0 2024-03-04T09:36:56.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.875+0000 7f792de00640 1 -- 172.21.15.6:0/4262127767 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79281e2a00 con 0x7f79281494a0 2024-03-04T09:36:56.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.877+0000 7f7924c00640 1 -- 172.21.15.6:0/4262127767 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7914015600 con 0x7f79281494a0 2024-03-04T09:36:56.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.878+0000 7f7924c00640 1 --2- 172.21.15.6:0/4262127767 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f790806b4d0 0x7f790806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:56.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.878+0000 7f7924c00640 1 -- 172.21.15.6:0/4262127767 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f791408a940 con 0x7f79281494a0 2024-03-04T09:36:56.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.879+0000 7f792ca00640 1 --2- 172.21.15.6:0/4262127767 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f790806b4d0 0x7f790806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:56.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.880+0000 7f792ca00640 1 --2- 172.21.15.6:0/4262127767 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f790806b4d0 0x7f790806d990 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f791c000c00 tx=0x7f791c009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:36:56.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:56.884+0000 7f7924c00640 1 -- 172.21.15.6:0/4262127767 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7914019030 con 0x7f79281494a0 2024-03-04T09:36:57.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.082+0000 7f792de00640 1 -- 172.21.15.6:0/4262127767 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f792804f330 con 0x7f79281494a0 2024-03-04T09:36:57.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.083+0000 7f7924c00640 1 -- 172.21.15.6:0/4262127767 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v131) v1 ==== 93+0+10 (secure 0 0 0) 0x7f791408a070 con 0x7f79281494a0 2024-03-04T09:36:57.080 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-04T09:36:57.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.085+0000 7f792de00640 1 -- 172.21.15.6:0/4262127767 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f790806b4d0 msgr2=0x7f790806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:57.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.085+0000 7f792de00640 1 --2- 172.21.15.6:0/4262127767 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f790806b4d0 0x7f790806d990 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f791c000c00 tx=0x7f791c009040 comp rx=0 tx=0).stop 2024-03-04T09:36:57.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.085+0000 7f792de00640 1 -- 172.21.15.6:0/4262127767 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f79281494a0 msgr2=0x7f79281e2cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:57.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.085+0000 7f792de00640 1 --2- 172.21.15.6:0/4262127767 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f79281494a0 0x7f79281e2cf0 secure :-1 s=READY pgs=1844 cs=0 l=1 rev1=1 crypto rx=0x7f79140116d0 tx=0x7f7914011b90 comp rx=0 tx=0).stop 2024-03-04T09:36:57.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.086+0000 7f792de00640 1 -- 172.21.15.6:0/4262127767 shutdown_connections 2024-03-04T09:36:57.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.086+0000 7f792de00640 1 --2- 172.21.15.6:0/4262127767 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f790806b4d0 0x7f790806d990 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:57.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.086+0000 7f792de00640 1 --2- 172.21.15.6:0/4262127767 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f792806dde0 0x7f792806e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:57.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.086+0000 7f792de00640 1 --2- 172.21.15.6:0/4262127767 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f79281494a0 0x7f79281e2cf0 unknown :-1 s=CLOSED pgs=1844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:57.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.086+0000 7f792de00640 1 --2- 172.21.15.6:0/4262127767 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7910004d40 0x7f792815a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:57.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.086+0000 7f792de00640 1 -- 172.21.15.6:0/4262127767 >> 172.21.15.6:0/4262127767 conn(0x7f79280723d0 msgr2=0x7f79280b5170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:36:57.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.086+0000 7f792de00640 1 -- 172.21.15.6:0/4262127767 shutdown_connections 2024-03-04T09:36:57.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.086+0000 7f792de00640 1 -- 172.21.15.6:0/4262127767 wait complete. 2024-03-04T09:36:57.103 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-04T09:36:57.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.328+0000 7f8377600640 1 Processor -- start 2024-03-04T09:36:57.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.328+0000 7f8377600640 1 -- start start 2024-03-04T09:36:57.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.329+0000 7f8377600640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f83700d4930 0x7f837015dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:57.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.329+0000 7f8377600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f83700d0bb0 0x7f837015e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:57.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.329+0000 7f8377600640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83700cf780 con 0x7f83700d07d0 2024-03-04T09:36:57.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.329+0000 7f8377600640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83700b3770 con 0x7f83700d4930 2024-03-04T09:36:57.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.329+0000 7f8377600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83700cfa60 con 0x7f83700d0bb0 2024-03-04T09:36:57.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.330+0000 7f8374c00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f83700d07d0 0x7f83700b4f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:60332/0 (socket says 172.21.15.6:60332) 2024-03-04T09:36:57.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.330+0000 7f8374c00640 1 -- 172.21.15.6:0/3477078094 learned_addr learned my addr 172.21.15.6:0/3477078094 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:36:57.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.330+0000 7f8375600640 1 --2- 172.21.15.6:0/3477078094 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f83700d0bb0 0x7f837015e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:57.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.330+0000 7f836fe00640 1 --2- 172.21.15.6:0/3477078094 >> v2:172.21.15.146:3300/0 conn(0x7f83700d4930 0x7f837015dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:57.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.331+0000 7f836f400640 1 -- 172.21.15.6:0/3477078094 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3255413065 0 0) 0x7f83700cf780 con 0x7f83700d07d0 2024-03-04T09:36:57.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.331+0000 7f836f400640 1 -- 172.21.15.6:0/3477078094 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8358003520 con 0x7f83700d07d0 2024-03-04T09:36:57.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.331+0000 7f836fe00640 1 -- 172.21.15.6:0/3477078094 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f83700d0bb0 msgr2=0x7f837015e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:36:57.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.331+0000 7f836fe00640 1 --2- 172.21.15.6:0/3477078094 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f83700d0bb0 0x7f837015e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:57.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.331+0000 7f836fe00640 1 -- 172.21.15.6:0/3477078094 >> v1:172.21.15.6:6789/0 conn(0x7f83700d07d0 legacy=0x7f83700b4f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:57.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.331+0000 7f836fe00640 1 -- 172.21.15.6:0/3477078094 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83700cfd40 con 0x7f83700d4930 2024-03-04T09:36:57.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.332+0000 7f836fe00640 1 --2- 172.21.15.6:0/3477078094 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83700d4930 0x7f837015dfb0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f835c0097a0 tx=0x7f835c02b940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9fef5ccdb7b402a server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:36:57.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.333+0000 7f836f400640 1 -- 172.21.15.6:0/3477078094 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f835c034070 con 0x7f83700d4930 2024-03-04T09:36:57.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.333+0000 7f836f400640 1 -- 172.21.15.6:0/3477078094 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f835c030800 con 0x7f83700d4930 2024-03-04T09:36:57.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.333+0000 7f836f400640 1 -- 172.21.15.6:0/3477078094 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f835c030bd0 con 0x7f83700d4930 2024-03-04T09:36:57.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.334+0000 7f8377600640 1 -- 172.21.15.6:0/3477078094 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83700d4930 msgr2=0x7f837015dfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:57.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.334+0000 7f8377600640 1 --2- 172.21.15.6:0/3477078094 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83700d4930 0x7f837015dfb0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f835c0097a0 tx=0x7f835c02b940 comp rx=0 tx=0).stop 2024-03-04T09:36:57.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.334+0000 7f8377600640 1 -- 172.21.15.6:0/3477078094 shutdown_connections 2024-03-04T09:36:57.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.334+0000 7f8377600640 1 --2- 172.21.15.6:0/3477078094 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f83700d0bb0 0x7f837015e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:57.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.334+0000 7f8377600640 1 --2- 172.21.15.6:0/3477078094 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83700d4930 0x7f837015dfb0 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:57.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.334+0000 7f8377600640 1 -- 172.21.15.6:0/3477078094 >> 172.21.15.6:0/3477078094 conn(0x7f83700d3e10 msgr2=0x7f83700d4220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:36:57.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.334+0000 7f8377600640 1 -- 172.21.15.6:0/3477078094 shutdown_connections 2024-03-04T09:36:57.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.334+0000 7f8377600640 1 -- 172.21.15.6:0/3477078094 wait complete. 2024-03-04T09:36:57.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.336+0000 7f8377600640 1 Processor -- start 2024-03-04T09:36:57.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.336+0000 7f8377600640 1 -- start start 2024-03-04T09:36:57.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.337+0000 7f8377600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f83700d07d0 0x7f837015c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:57.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.337+0000 7f8377600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83700d0bb0 0x7f83701f4070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:57.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.337+0000 7f8377600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f83700d4930 0x7f837015fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:57.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.337+0000 7f8377600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83700b3fd0 con 0x7f83700d07d0 2024-03-04T09:36:57.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.337+0000 7f8377600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83700b3e50 con 0x7f83700d0bb0 2024-03-04T09:36:57.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.337+0000 7f8377600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83700b4150 con 0x7f83700d4930 2024-03-04T09:36:57.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.337+0000 7f8374c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f83700d07d0 0x7f837015c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:57.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.337+0000 7f8374c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f83700d07d0 0x7f837015c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41920/0 (socket says 172.21.15.6:41920) 2024-03-04T09:36:57.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.337+0000 7f8374c00640 1 -- 172.21.15.6:0/434872934 learned_addr learned my addr 172.21.15.6:0/434872934 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:36:57.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.337+0000 7f8374c00640 1 -- 172.21.15.6:0/434872934 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f83700d4930 msgr2=0x7f837015fdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:57.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.337+0000 7f8375600640 1 --2- 172.21.15.6:0/434872934 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f83700d4930 0x7f837015fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:57.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.338+0000 7f8374c00640 1 --2- 172.21.15.6:0/434872934 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f83700d4930 0x7f837015fdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:57.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.338+0000 7f8374c00640 1 -- 172.21.15.6:0/434872934 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83700d0bb0 msgr2=0x7f83701f4070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:57.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.338+0000 7f836fe00640 1 --2- 172.21.15.6:0/434872934 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83700d0bb0 0x7f83701f4070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:57.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.338+0000 7f8374c00640 1 --2- 172.21.15.6:0/434872934 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83700d0bb0 0x7f83701f4070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:57.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.338+0000 7f8374c00640 1 -- 172.21.15.6:0/434872934 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83700cc230 con 0x7f83700d07d0 2024-03-04T09:36:57.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.338+0000 7f8374c00640 1 --2- 172.21.15.6:0/434872934 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f83700d07d0 0x7f837015c2c0 secure :-1 s=READY pgs=1846 cs=0 l=1 rev1=1 crypto rx=0x7f836400c430 tx=0x7f836400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:36:57.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.338+0000 7f836cc00640 1 -- 172.21.15.6:0/434872934 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8364017070 con 0x7f83700d07d0 2024-03-04T09:36:57.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.338+0000 7f836cc00640 1 -- 172.21.15.6:0/434872934 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8364005ce0 con 0x7f83700d07d0 2024-03-04T09:36:57.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.338+0000 7f8377600640 1 -- 172.21.15.6:0/434872934 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83700cc510 con 0x7f83700d07d0 2024-03-04T09:36:57.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.338+0000 7f836cc00640 1 -- 172.21.15.6:0/434872934 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83640072f0 con 0x7f83700d07d0 2024-03-04T09:36:57.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.338+0000 7f8377600640 1 -- 172.21.15.6:0/434872934 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83700ccad0 con 0x7f83700d07d0 2024-03-04T09:36:57.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.340+0000 7f8377600640 1 -- 172.21.15.6:0/434872934 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f837004f330 con 0x7f83700d07d0 2024-03-04T09:36:57.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.342+0000 7f836cc00640 1 -- 172.21.15.6:0/434872934 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f836401a070 con 0x7f83700d07d0 2024-03-04T09:36:57.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.342+0000 7f836cc00640 1 --2- 172.21.15.6:0/434872934 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f835006b500 0x7f835006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:57.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.343+0000 7f836cc00640 1 -- 172.21.15.6:0/434872934 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f8364089780 con 0x7f83700d07d0 2024-03-04T09:36:57.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.343+0000 7f836fe00640 1 --2- 172.21.15.6:0/434872934 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f835006b500 0x7f835006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:57.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.344+0000 7f836fe00640 1 --2- 172.21.15.6:0/434872934 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f835006b500 0x7f835006d9c0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f835c036000 tx=0x7f835c008030 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:36:57.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.347+0000 7f836cc00640 1 -- 172.21.15.6:0/434872934 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8364058520 con 0x7f83700d07d0 2024-03-04T09:36:57.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.546+0000 7f8377600640 1 -- 172.21.15.6:0/434872934 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f8370161490 con 0x7f83700d07d0 2024-03-04T09:36:57.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.547+0000 7f836cc00640 1 -- 172.21.15.6:0/434872934 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v131) v1 ==== 104+0+11 (secure 0 0 0) 0x7f8364057e20 con 0x7f83700d07d0 2024-03-04T09:36:57.544 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-03-04T09:36:57.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.550+0000 7f8377600640 1 -- 172.21.15.6:0/434872934 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f835006b500 msgr2=0x7f835006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:57.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.550+0000 7f8377600640 1 --2- 172.21.15.6:0/434872934 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f835006b500 0x7f835006d9c0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f835c036000 tx=0x7f835c008030 comp rx=0 tx=0).stop 2024-03-04T09:36:57.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.550+0000 7f8377600640 1 -- 172.21.15.6:0/434872934 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f83700d07d0 msgr2=0x7f837015c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:57.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.550+0000 7f8377600640 1 --2- 172.21.15.6:0/434872934 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f83700d07d0 0x7f837015c2c0 secure :-1 s=READY pgs=1846 cs=0 l=1 rev1=1 crypto rx=0x7f836400c430 tx=0x7f836400c8f0 comp rx=0 tx=0).stop 2024-03-04T09:36:57.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.550+0000 7f8377600640 1 -- 172.21.15.6:0/434872934 shutdown_connections 2024-03-04T09:36:57.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.550+0000 7f8377600640 1 --2- 172.21.15.6:0/434872934 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f835006b500 0x7f835006d9c0 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:57.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.550+0000 7f8377600640 1 --2- 172.21.15.6:0/434872934 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f83700d4930 0x7f837015fdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:57.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.550+0000 7f8377600640 1 --2- 172.21.15.6:0/434872934 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83700d0bb0 0x7f83701f4070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:57.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.550+0000 7f8377600640 1 --2- 172.21.15.6:0/434872934 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f83700d07d0 0x7f837015c2c0 unknown :-1 s=CLOSED pgs=1846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:57.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.550+0000 7f8377600640 1 -- 172.21.15.6:0/434872934 >> 172.21.15.6:0/434872934 conn(0x7f83700d3e10 msgr2=0x7f83700d41f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:36:57.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.550+0000 7f8377600640 1 -- 172.21.15.6:0/434872934 shutdown_connections 2024-03-04T09:36:57.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.550+0000 7f8377600640 1 -- 172.21.15.6:0/434872934 wait complete. 2024-03-04T09:36:57.574 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-04T09:36:57.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.843+0000 7f4add400640 1 Processor -- start 2024-03-04T09:36:57.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.844+0000 7f4add400640 1 -- start start 2024-03-04T09:36:57.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.845+0000 7f4add400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f4ad8071fb0 0x7f4ad80d0520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:57.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.845+0000 7f4add400640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ad80cf760 con 0x7f4ad8072d10 2024-03-04T09:36:57.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.845+0000 7f4add400640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ad80cfa40 con 0x7f4ad806afb0 2024-03-04T09:36:57.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.845+0000 7f4add400640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ad80cf510 con 0x7f4ad8071fb0 2024-03-04T09:36:57.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.846+0000 7f4ad7400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f4ad8071fb0 0x7f4ad80d0520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:57.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.846+0000 7f4adca00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f4ad8072d10 0x7f4ad806dd30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:60334/0 (socket says 172.21.15.6:60334) 2024-03-04T09:36:57.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.846+0000 7f4adca00640 1 -- 172.21.15.6:0/4071491023 learned_addr learned my addr 172.21.15.6:0/4071491023 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-04T09:36:57.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.846+0000 7f4ad7400640 1 -- 172.21.15.6:0/4071491023 >> v1:172.21.15.146:6789/0 conn(0x7f4ad806afb0 legacy=0x7f4ad806b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:36:57.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.846+0000 7f4ad7400640 1 -- 172.21.15.6:0/4071491023 >> v1:172.21.15.6:6789/0 conn(0x7f4ad8072d10 legacy=0x7f4ad806dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:57.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.846+0000 7f4ad7400640 1 -- 172.21.15.6:0/4071491023 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ad80cfd20 con 0x7f4ad8071fb0 2024-03-04T09:36:57.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.846+0000 7f4ad6a00640 1 -- 172.21.15.6:0/4071491023 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1377044304 0 0) 0x7f4ad80cf760 con 0x7f4ad8072d10 2024-03-04T09:36:57.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.847+0000 7f4ad7400640 1 --2- 172.21.15.6:0/4071491023 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4ad8071fb0 0x7f4ad80d0520 secure :-1 s=READY pgs=1848 cs=0 l=1 rev1=1 crypto rx=0x7f4ac00097a0 tx=0x7f4ac002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=135f78d0746bf34b server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:36:57.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.848+0000 7f4ad6a00640 1 -- 172.21.15.6:0/4071491023 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ac0035020 con 0x7f4ad8071fb0 2024-03-04T09:36:57.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.848+0000 7f4ad6a00640 1 -- 172.21.15.6:0/4071491023 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ac0031830 con 0x7f4ad8071fb0 2024-03-04T09:36:57.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.848+0000 7f4ad6a00640 1 -- 172.21.15.6:0/4071491023 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ac0031c00 con 0x7f4ad8071fb0 2024-03-04T09:36:57.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.848+0000 7f4add400640 1 -- 172.21.15.6:0/4071491023 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4ad8071fb0 msgr2=0x7f4ad80d0520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:57.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.848+0000 7f4add400640 1 --2- 172.21.15.6:0/4071491023 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4ad8071fb0 0x7f4ad80d0520 secure :-1 s=READY pgs=1848 cs=0 l=1 rev1=1 crypto rx=0x7f4ac00097a0 tx=0x7f4ac002b8d0 comp rx=0 tx=0).stop 2024-03-04T09:36:57.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.848+0000 7f4add400640 1 -- 172.21.15.6:0/4071491023 shutdown_connections 2024-03-04T09:36:57.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.848+0000 7f4add400640 1 --2- 172.21.15.6:0/4071491023 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4ad8071fb0 0x7f4ad80d0520 unknown :-1 s=CLOSED pgs=1848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:57.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.848+0000 7f4add400640 1 -- 172.21.15.6:0/4071491023 >> 172.21.15.6:0/4071491023 conn(0x7f4ad80d0c00 msgr2=0x7f4ad80d1010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:36:57.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.848+0000 7f4add400640 1 -- 172.21.15.6:0/4071491023 shutdown_connections 2024-03-04T09:36:57.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.848+0000 7f4add400640 1 -- 172.21.15.6:0/4071491023 wait complete. 2024-03-04T09:36:57.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.850+0000 7f4add400640 1 Processor -- start 2024-03-04T09:36:57.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.850+0000 7f4add400640 1 -- start start 2024-03-04T09:36:57.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.851+0000 7f4add400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4ad806afb0 0x7f4ad815bd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:57.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.851+0000 7f4add400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4ad8071fb0 0x7f4ad81f8fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:57.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.851+0000 7f4add400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4ad8072d10 0x7f4ad8163370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:57.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.851+0000 7f4add400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ad8162780 con 0x7f4ad8072d10 2024-03-04T09:36:57.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.851+0000 7f4add400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ad8162600 con 0x7f4ad8071fb0 2024-03-04T09:36:57.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.851+0000 7f4add400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ad8072980 con 0x7f4ad806afb0 2024-03-04T09:36:57.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.851+0000 7f4adca00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4ad8072d10 0x7f4ad8163370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:57.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.851+0000 7f4adca00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4ad8072d10 0x7f4ad8163370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41938/0 (socket says 172.21.15.6:41938) 2024-03-04T09:36:57.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.851+0000 7f4adca00640 1 -- 172.21.15.6:0/2059143019 learned_addr learned my addr 172.21.15.6:0/2059143019 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:36:57.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.852+0000 7f4ad7400640 1 --2- 172.21.15.6:0/2059143019 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4ad8071fb0 0x7f4ad81f8fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:57.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.852+0000 7f4ad7e00640 1 --2- 172.21.15.6:0/2059143019 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4ad806afb0 0x7f4ad815bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:57.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.852+0000 7f4adca00640 1 -- 172.21.15.6:0/2059143019 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4ad806afb0 msgr2=0x7f4ad815bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:57.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.852+0000 7f4adca00640 1 --2- 172.21.15.6:0/2059143019 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4ad806afb0 0x7f4ad815bd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:57.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.852+0000 7f4adca00640 1 -- 172.21.15.6:0/2059143019 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4ad8071fb0 msgr2=0x7f4ad81f8fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:57.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.852+0000 7f4adca00640 1 --2- 172.21.15.6:0/2059143019 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4ad8071fb0 0x7f4ad81f8fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:57.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.852+0000 7f4adca00640 1 -- 172.21.15.6:0/2059143019 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ad80adfa0 con 0x7f4ad8072d10 2024-03-04T09:36:57.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.852+0000 7f4adca00640 1 --2- 172.21.15.6:0/2059143019 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4ad8072d10 0x7f4ad8163370 secure :-1 s=READY pgs=1849 cs=0 l=1 rev1=1 crypto rx=0x7f4ad000c700 tx=0x7f4ad000cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:36:57.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.852+0000 7f4acfe00640 1 -- 172.21.15.6:0/2059143019 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ad0018070 con 0x7f4ad8072d10 2024-03-04T09:36:57.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.852+0000 7f4acfe00640 1 -- 172.21.15.6:0/2059143019 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ad000ce10 con 0x7f4ad8072d10 2024-03-04T09:36:57.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.852+0000 7f4acfe00640 1 -- 172.21.15.6:0/2059143019 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ad000a860 con 0x7f4ad8072d10 2024-03-04T09:36:57.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.853+0000 7f4add400640 1 -- 172.21.15.6:0/2059143019 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ad80ae280 con 0x7f4ad8072d10 2024-03-04T09:36:57.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.853+0000 7f4add400640 1 -- 172.21.15.6:0/2059143019 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ad80ae840 con 0x7f4ad8072d10 2024-03-04T09:36:57.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.854+0000 7f4add400640 1 -- 172.21.15.6:0/2059143019 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ad804f330 con 0x7f4ad8072d10 2024-03-04T09:36:57.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.855+0000 7f4acfe00640 1 -- 172.21.15.6:0/2059143019 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4ad000aa00 con 0x7f4ad8072d10 2024-03-04T09:36:57.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.856+0000 7f4acfe00640 1 --2- 172.21.15.6:0/2059143019 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f4ab406b4d0 0x7f4ab406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:57.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.856+0000 7f4acfe00640 1 -- 172.21.15.6:0/2059143019 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f4ad0089850 con 0x7f4ad8072d10 2024-03-04T09:36:57.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.857+0000 7f4ad7e00640 1 --2- 172.21.15.6:0/2059143019 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f4ab406b4d0 0x7f4ab406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:57.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.858+0000 7f4ad7e00640 1 --2- 172.21.15.6:0/2059143019 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f4ab406b4d0 0x7f4ab406d990 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f4ad8072920 tx=0x7f4ac8009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:36:57.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:57.861+0000 7f4acfe00640 1 -- 172.21.15.6:0/2059143019 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4ad0058680 con 0x7f4ad8072d10 2024-03-04T09:36:58.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:58.061+0000 7f4add400640 1 -- 172.21.15.6:0/2059143019 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f4ad8070760 con 0x7f4ad8072d10 2024-03-04T09:36:58.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:58.062+0000 7f4acfe00640 1 -- 172.21.15.6:0/2059143019 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v131) v1 ==== 100+0+11 (secure 0 0 0) 0x7f4ad0057f80 con 0x7f4ad8072d10 2024-03-04T09:36:58.059 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 64 2024-03-04T09:36:58.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:58.066+0000 7f4add400640 1 -- 172.21.15.6:0/2059143019 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f4ab406b4d0 msgr2=0x7f4ab406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:58.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:58.066+0000 7f4add400640 1 --2- 172.21.15.6:0/2059143019 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f4ab406b4d0 0x7f4ab406d990 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f4ad8072920 tx=0x7f4ac8009040 comp rx=0 tx=0).stop 2024-03-04T09:36:58.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:58.066+0000 7f4add400640 1 -- 172.21.15.6:0/2059143019 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4ad8072d10 msgr2=0x7f4ad8163370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:58.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:58.066+0000 7f4add400640 1 --2- 172.21.15.6:0/2059143019 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4ad8072d10 0x7f4ad8163370 secure :-1 s=READY pgs=1849 cs=0 l=1 rev1=1 crypto rx=0x7f4ad000c700 tx=0x7f4ad000cbc0 comp rx=0 tx=0).stop 2024-03-04T09:36:58.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:58.066+0000 7f4add400640 1 -- 172.21.15.6:0/2059143019 shutdown_connections 2024-03-04T09:36:58.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:58.066+0000 7f4add400640 1 --2- 172.21.15.6:0/2059143019 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4ad8072d10 0x7f4ad8163370 unknown :-1 s=CLOSED pgs=1849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:58.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:58.066+0000 7f4add400640 1 --2- 172.21.15.6:0/2059143019 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4ad8071fb0 0x7f4ad81f8fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:58.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:58.066+0000 7f4add400640 1 --2- 172.21.15.6:0/2059143019 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7f4ab406b4d0 0x7f4ab406d990 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:58.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:58.066+0000 7f4add400640 1 --2- 172.21.15.6:0/2059143019 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4ad806afb0 0x7f4ad815bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:58.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:58.066+0000 7f4add400640 1 -- 172.21.15.6:0/2059143019 >> 172.21.15.6:0/2059143019 conn(0x7f4ad80d0c00 msgr2=0x7f4ad806c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:36:58.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:58.066+0000 7f4add400640 1 -- 172.21.15.6:0/2059143019 shutdown_connections 2024-03-04T09:36:58.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:58.066+0000 7f4add400640 1 -- 172.21.15.6:0/2059143019 wait complete. 2024-03-04T09:36:58.083 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-04T09:36:58.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:58.298+0000 7fc205a00640 1 Processor -- start 2024-03-04T09:36:58.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:58.299+0000 7fc205a00640 1 -- start start 2024-03-04T09:36:58.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:58.300+0000 7fc205a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc200070f10 0x7fc2000cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:58.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:58.300+0000 7fc205a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2000714d0 con 0x7fc200145300 2024-03-04T09:36:58.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:58.300+0000 7fc205a00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2000717b0 con 0x7fc2000cce90 2024-03-04T09:36:58.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:58.300+0000 7fc205a00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2000cd300 con 0x7fc200070f10 2024-03-04T09:36:58.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:58.300+0000 7fc1f7e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc200070f10 0x7fc2000cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:58.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:58.301+0000 7fc1f7e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc200070f10 0x7fc2000cc950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41952/0 (socket says 172.21.15.6:41952) 2024-03-04T09:36:58.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:58.301+0000 7fc1f7e00640 1 -- 172.21.15.6:0/351617491 learned_addr learned my addr 172.21.15.6:0/351617491 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:36:58.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:58.301+0000 7fc1f7e00640 1 -- 172.21.15.6:0/351617491 >> v1:172.21.15.146:6789/0 conn(0x7fc2000cce90 legacy=0x7fc20015c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T09:36:58.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:58.301+0000 7fc1f7e00640 1 -- 172.21.15.6:0/351617491 >> v1:172.21.15.6:6789/0 conn(0x7fc200145300 legacy=0x7fc2000b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:58.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:58.301+0000 7fc1f7e00640 1 -- 172.21.15.6:0/351617491 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc200071a90 con 0x7fc200070f10 2024-03-04T09:36:58.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:58.302+0000 7fc1f7e00640 1 --2- 172.21.15.6:0/351617491 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc200070f10 0x7fc2000cc950 secure :-1 s=READY pgs=1851 cs=0 l=1 rev1=1 crypto rx=0x7fc1e8009870 tx=0x7fc1e802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1e54855a04d5c78 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:36:58.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:58.303+0000 7fc1ff400640 1 -- 172.21.15.6:0/351617491 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1e8035020 con 0x7fc200070f10 2024-03-04T09:36:58.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:58.303+0000 7fc1ff400640 1 -- 172.21.15.6:0/351617491 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1e8031870 con 0x7fc200070f10 2024-03-04T09:36:58.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:58.303+0000 7fc1ff400640 1 -- 172.21.15.6:0/351617491 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1e8031c40 con 0x7fc200070f10 2024-03-04T09:36:58.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:58.303+0000 7fc205a00640 1 -- 172.21.15.6:0/351617491 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc200070f10 msgr2=0x7fc2000cc950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:58.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:58.303+0000 7fc205a00640 1 --2- 172.21.15.6:0/351617491 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc200070f10 0x7fc2000cc950 secure :-1 s=READY pgs=1851 cs=0 l=1 rev1=1 crypto rx=0x7fc1e8009870 tx=0x7fc1e802b910 comp rx=0 tx=0).stop 2024-03-04T09:36:58.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:58.303+0000 7fc205a00640 1 -- 172.21.15.6:0/351617491 shutdown_connections 2024-03-04T09:36:58.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:58.303+0000 7fc205a00640 1 --2- 172.21.15.6:0/351617491 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc200070f10 0x7fc2000cc950 unknown :-1 s=CLOSED pgs=1851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:58.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:58.303+0000 7fc205a00640 1 -- 172.21.15.6:0/351617491 >> 172.21.15.6:0/351617491 conn(0x7fc20006b3d0 msgr2=0x7fc20006b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:36:58.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:58.303+0000 7fc205a00640 1 -- 172.21.15.6:0/351617491 shutdown_connections 2024-03-04T09:36:58.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:58.303+0000 7fc205a00640 1 -- 172.21.15.6:0/351617491 wait complete. 2024-03-04T09:36:58.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:58.305+0000 7fc205a00640 1 Processor -- start 2024-03-04T09:36:58.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:58.305+0000 7fc205a00640 1 -- start start 2024-03-04T09:36:58.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:58.306+0000 7fc205a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc200070f10 0x7fc20006da80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:58.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:58.306+0000 7fc205a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc2000cce90 0x7fc200139660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:58.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:58.306+0000 7fc205a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc200145300 0x7fc200137cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:58.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:58.306+0000 7fc205a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc20015cc80 con 0x7fc200070f10 2024-03-04T09:36:58.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:58.306+0000 7fc205a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2001456e0 con 0x7fc2000cce90 2024-03-04T09:36:58.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:58.306+0000 7fc205a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc20015ce00 con 0x7fc200145300 2024-03-04T09:36:58.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:58.306+0000 7fc1ffe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc200070f10 0x7fc20006da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:58.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:58.307+0000 7fc1ffe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc200070f10 0x7fc20006da80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41964/0 (socket says 172.21.15.6:41964) 2024-03-04T09:36:58.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:58.307+0000 7fc1ffe00640 1 -- 172.21.15.6:0/516112910 learned_addr learned my addr 172.21.15.6:0/516112910 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-04T09:36:58.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:58.307+0000 7fc1f7e00640 1 --2- 172.21.15.6:0/516112910 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc2000cce90 0x7fc200139660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:58.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:58.307+0000 7fc205000640 1 --2- 172.21.15.6:0/516112910 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc200145300 0x7fc200137cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:58.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:58.307+0000 7fc1ffe00640 1 -- 172.21.15.6:0/516112910 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc200145300 msgr2=0x7fc200137cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:58.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:58.307+0000 7fc1ffe00640 1 --2- 172.21.15.6:0/516112910 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc200145300 0x7fc200137cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:58.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:58.307+0000 7fc1ffe00640 1 -- 172.21.15.6:0/516112910 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc2000cce90 msgr2=0x7fc200139660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:58.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:58.307+0000 7fc1ffe00640 1 --2- 172.21.15.6:0/516112910 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc2000cce90 0x7fc200139660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:58.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:58.307+0000 7fc1ffe00640 1 -- 172.21.15.6:0/516112910 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc20015b590 con 0x7fc200070f10 2024-03-04T09:36:58.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:58.307+0000 7fc1f7e00640 1 --2- 172.21.15.6:0/516112910 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc2000cce90 0x7fc200139660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T09:36:58.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:58.308+0000 7fc1ffe00640 1 --2- 172.21.15.6:0/516112910 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc200070f10 0x7fc20006da80 secure :-1 s=READY pgs=1852 cs=0 l=1 rev1=1 crypto rx=0x7fc1f000c3c0 tx=0x7fc1f000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:36:58.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:58.308+0000 7fc1fcc00640 1 -- 172.21.15.6:0/516112910 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1f00052d0 con 0x7fc200070f10 2024-03-04T09:36:58.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:58.308+0000 7fc1fcc00640 1 -- 172.21.15.6:0/516112910 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1f0005490 con 0x7fc200070f10 2024-03-04T09:36:58.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:58.308+0000 7fc1fcc00640 1 -- 172.21.15.6:0/516112910 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1f0007260 con 0x7fc200070f10 2024-03-04T09:36:58.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:58.308+0000 7fc205a00640 1 -- 172.21.15.6:0/516112910 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2000ad600 con 0x7fc200070f10 2024-03-04T09:36:58.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:58.308+0000 7fc205a00640 1 -- 172.21.15.6:0/516112910 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc20015afd0 con 0x7fc200070f10 2024-03-04T09:36:58.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:58.309+0000 7fc205a00640 1 -- 172.21.15.6:0/516112910 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2000ad7d0 con 0x7fc200070f10 2024-03-04T09:36:58.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:58.310+0000 7fc1fcc00640 1 -- 172.21.15.6:0/516112910 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc1f0007400 con 0x7fc200070f10 2024-03-04T09:36:58.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:58.311+0000 7fc1fcc00640 1 --2- 172.21.15.6:0/516112910 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc1dc06b4d0 0x7fc1dc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T09:36:58.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:58.311+0000 7fc1fcc00640 1 -- 172.21.15.6:0/516112910 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fc1f0089720 con 0x7fc200070f10 2024-03-04T09:36:58.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:58.312+0000 7fc1f7e00640 1 --2- 172.21.15.6:0/516112910 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc1dc06b4d0 0x7fc1dc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T09:36:58.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:58.313+0000 7fc1f7e00640 1 --2- 172.21.15.6:0/516112910 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc1dc06b4d0 0x7fc1dc06d990 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7fc1e8036000 tx=0x7fc1e803c040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T09:36:58.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:58.320+0000 7fc1fcc00640 1 -- 172.21.15.6:0/516112910 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc1f0058560 con 0x7fc200070f10 2024-03-04T09:36:58.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:58.531+0000 7fc205a00640 1 -- 172.21.15.6:0/516112910 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fc2000b3830 con 0x7fc200070f10 2024-03-04T09:36:58.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:58.532+0000 7fc1fcc00640 1 -- 172.21.15.6:0/516112910 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v131) v1 ==== 80+0+0 (secure 0 0 0) 0x7fc1f0057e60 con 0x7fc200070f10 2024-03-04T09:36:58.529 INFO:teuthology.orchestra.run.smithi006.stderr:listed 0 entries 2024-03-04T09:36:58.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:58.534+0000 7fc205a00640 1 -- 172.21.15.6:0/516112910 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc1dc06b4d0 msgr2=0x7fc1dc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:58.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:58.534+0000 7fc205a00640 1 --2- 172.21.15.6:0/516112910 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc1dc06b4d0 0x7fc1dc06d990 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7fc1e8036000 tx=0x7fc1e803c040 comp rx=0 tx=0).stop 2024-03-04T09:36:58.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:58.535+0000 7fc205a00640 1 -- 172.21.15.6:0/516112910 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc200070f10 msgr2=0x7fc20006da80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T09:36:58.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:58.535+0000 7fc205a00640 1 --2- 172.21.15.6:0/516112910 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc200070f10 0x7fc20006da80 secure :-1 s=READY pgs=1852 cs=0 l=1 rev1=1 crypto rx=0x7fc1f000c3c0 tx=0x7fc1f000c880 comp rx=0 tx=0).stop 2024-03-04T09:36:58.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:58.535+0000 7fc205a00640 1 -- 172.21.15.6:0/516112910 shutdown_connections 2024-03-04T09:36:58.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:58.535+0000 7fc205a00640 1 --2- 172.21.15.6:0/516112910 >> [v2:172.21.15.146:6832/1788613334,v1:172.21.15.146:6833/1788613334] conn(0x7fc1dc06b4d0 0x7fc1dc06d990 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:58.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:58.535+0000 7fc205a00640 1 --2- 172.21.15.6:0/516112910 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc200145300 0x7fc200137cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:58.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:58.535+0000 7fc205a00640 1 --2- 172.21.15.6:0/516112910 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc2000cce90 0x7fc200139660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:58.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:58.535+0000 7fc205a00640 1 --2- 172.21.15.6:0/516112910 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc200070f10 0x7fc20006da80 unknown :-1 s=CLOSED pgs=1852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T09:36:58.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:58.535+0000 7fc205a00640 1 -- 172.21.15.6:0/516112910 >> 172.21.15.6:0/516112910 conn(0x7fc20006b3d0 msgr2=0x7fc20015dd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T09:36:58.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:58.535+0000 7fc205a00640 1 -- 172.21.15.6:0/516112910 shutdown_connections 2024-03-04T09:36:58.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-04T09:36:58.535+0000 7fc205a00640 1 -- 172.21.15.6:0/516112910 wait complete. 2024-03-04T09:36:58.550 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-04T09:36:58.551 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T09:36:58.551 DEBUG:teuthology.orchestra.run.smithi006:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-04T09:37:11.881 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:37:11.893 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:37:42.023 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:37:42.032 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:38:12.162 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:38:12.170 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:38:42.309 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:38:42.318 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:39:12.450 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:39:12.458 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:39:42.590 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:39:42.599 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:40:12.740 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:40:12.753 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:40:42.915 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:40:42.923 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:41:13.053 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:41:13.062 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:41:43.192 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:41:43.200 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:41:58.558 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 745, in recv_stderr out = self.in_stderr_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/buffered_pipe.py", line 154, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 109, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 275, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1374, in _read return self.channel.recv_stderr(size) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 747, in recv_stderr raise socket.timeout() socket.timeout 2024-03-04T09:41:58.627 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 697, in recv out = self.in_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/buffered_pipe.py", line 154, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 109, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 275, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1359, in _read return self.channel.recv(size) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 699, in recv raise socket.timeout() socket.timeout 2024-03-04T09:42:13.339 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:42:13.348 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:42:43.477 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:42:43.486 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:43:13.660 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:43:13.668 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:43:43.810 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:43:43.819 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:44:13.951 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:44:13.957 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:44:44.103 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:44:44.112 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:45:14.236 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:45:14.244 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:45:44.394 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:45:44.403 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:46:14.606 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:46:14.614 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:46:44.756 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:46:44.765 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:47:14.901 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:47:15.245 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:47:45.404 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:47:45.417 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:48:15.600 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:48:15.608 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:48:45.748 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:48:45.757 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:49:15.892 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:49:15.900 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:49:46.041 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:49:46.049 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:50:16.196 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:50:16.205 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:50:46.352 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:50:46.361 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:51:16.498 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:51:16.507 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:51:46.657 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:51:46.666 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:52:16.826 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:52:16.834 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:52:46.962 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:52:46.970 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:53:17.117 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:53:17.126 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:53:47.274 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:53:47.282 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:54:17.438 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:54:17.447 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:54:47.585 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:54:47.593 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:55:17.725 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:55:17.733 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:55:47.880 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:55:47.889 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:56:18.021 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:56:18.030 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:56:48.155 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:56:48.166 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:57:18.310 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:57:18.318 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:57:48.465 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:57:48.475 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:58:18.604 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:58:18.613 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:58:48.758 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:58:48.767 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:59:18.906 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:59:18.915 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:59:49.051 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T09:59:49.059 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:00:19.194 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:00:19.203 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:00:49.339 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:00:49.349 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:01:19.492 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:01:19.500 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:01:49.648 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:01:49.657 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:02:19.828 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:02:19.837 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:02:49.975 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:02:49.984 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:03:20.116 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:03:20.126 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:03:50.277 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:03:50.285 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:04:20.414 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:04:20.422 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:04:50.561 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:04:50.572 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:05:20.724 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:05:20.733 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:05:50.871 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:05:50.881 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:06:21.011 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:06:21.020 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:06:51.166 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:06:51.174 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:07:21.319 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:07:21.328 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:07:51.475 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:07:51.487 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:08:21.646 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:08:21.654 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:08:51.785 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:08:51.793 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:09:21.932 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:09:21.941 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:09:52.086 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:09:52.095 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:10:22.234 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:10:22.243 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:10:52.388 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:10:52.395 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:11:22.561 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:11:22.569 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:11:52.698 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:11:52.714 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:12:22.850 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:12:22.859 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:12:53.010 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:12:53.021 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:13:23.207 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:13:23.216 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:13:53.412 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:13:53.420 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:14:23.550 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:14:23.558 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:14:53.788 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:14:53.800 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:15:23.940 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:15:23.949 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:15:54.140 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:15:54.149 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:16:24.344 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:16:24.353 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:16:54.524 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:16:54.532 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:17:24.730 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:17:24.740 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:17:54.885 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:17:54.893 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:18:25.106 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:18:25.117 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:18:55.295 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:18:55.303 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:19:25.432 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:19:25.444 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:19:55.729 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:19:55.742 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:20:25.882 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:20:25.900 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:20:56.290 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:20:56.299 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:21:26.496 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:21:26.512 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:21:56.640 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:21:56.649 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:22:26.796 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:22:26.804 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:22:56.954 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:22:56.963 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:23:27.155 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:23:27.164 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:23:57.293 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:23:57.302 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:24:27.493 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:24:27.502 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:24:57.654 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:24:57.662 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:25:27.850 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:25:27.859 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:25:58.035 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:25:58.043 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:26:28.177 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:26:28.186 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:26:58.457 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:26:58.465 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:27:28.644 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:27:28.654 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:27:58.849 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:27:58.860 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:28:29.063 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:28:29.072 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:28:59.284 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:28:59.293 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:29:29.488 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:29:29.497 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:29:59.711 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:29:59.719 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:30:29.931 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:30:29.944 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:31:00.086 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:31:00.098 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:31:30.325 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:31:30.334 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:32:00.529 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:32:00.538 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:32:30.744 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:32:30.753 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:32:54.998 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:32:54.994+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:32:54.995356+0000) 2024-03-04T10:32:55.997 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:32:55.994+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:32:55.995614+0000) 2024-03-04T10:32:56.998 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:32:56.995+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:32:56.995906+0000) 2024-03-04T10:32:57.998 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:32:57.995+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:32:57.996116+0000) 2024-03-04T10:32:58.998 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:32:58.995+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:32:58.996378+0000) 2024-03-04T10:32:59.998 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:32:59.996+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:32:59.996617+0000) 2024-03-04T10:33:00.948 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:33:00.960 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:33:00.998 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:33:00.995+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:33:00.996800+0000) 2024-03-04T10:33:01.999 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:33:01.996+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:33:01.997045+0000) 2024-03-04T10:33:02.999 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:33:02.996+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:33:02.997318+0000) 2024-03-04T10:33:03.999 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:33:03.997+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:33:03.997558+0000) 2024-03-04T10:33:05.000 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:33:04.997+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:33:04.997797+0000) 2024-03-04T10:33:06.000 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:33:05.997+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:33:05.998099+0000) 2024-03-04T10:33:07.000 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:33:06.997+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:33:06.998349+0000) 2024-03-04T10:33:08.000 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:33:07.998+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:33:07.998592+0000) 2024-03-04T10:33:09.000 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:33:08.998+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:33:08.998880+0000) 2024-03-04T10:33:10.001 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:33:09.998+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:33:09.999134+0000) 2024-03-04T10:33:11.001 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:33:10.999+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:33:10.999382+0000) 2024-03-04T10:33:12.001 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:33:11.998+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:33:11.999631+0000) 2024-03-04T10:33:13.002 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:33:12.999+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:33:12.999950+0000) 2024-03-04T10:33:14.002 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:33:13.999+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:33:14.000192+0000) 2024-03-04T10:33:15.002 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:33:14.999+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:33:15.000428+0000) 2024-03-04T10:33:16.002 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:33:15.999+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:33:16.000673+0000) 2024-03-04T10:33:17.003 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:33:17.000+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:33:17.000954+0000) 2024-03-04T10:33:18.003 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:33:18.000+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:33:18.001211+0000) 2024-03-04T10:33:19.003 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:33:19.001+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:33:19.001464+0000) 2024-03-04T10:33:20.003 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:33:20.000+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:33:20.001731+0000) 2024-03-04T10:33:21.004 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:33:21.001+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:33:21.001975+0000) 2024-03-04T10:33:22.004 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:33:22.002+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:33:22.002218+0000) 2024-03-04T10:33:23.004 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:33:23.002+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:33:23.002469+0000) 2024-03-04T10:33:24.005 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:33:24.002+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:33:24.002716+0000) 2024-03-04T10:33:25.005 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:33:25.002+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:33:25.002981+0000) 2024-03-04T10:33:26.005 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:33:26.003+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:33:26.003232+0000) 2024-03-04T10:33:27.005 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:33:27.002+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:33:27.003490+0000) 2024-03-04T10:33:28.005 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:33:28.001+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:33:28.003691+0000) 2024-03-04T10:33:29.006 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:33:29.002+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:33:29.003970+0000) 2024-03-04T10:33:30.006 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:33:30.003+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:33:30.004236+0000) 2024-03-04T10:33:31.006 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:33:31.003+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:33:31.004499+0000) 2024-03-04T10:33:31.089 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:33:31.098 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:33:32.006 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:33:32.003+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:33:32.004767+0000) 2024-03-04T10:33:33.007 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:33:33.003+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:33:33.004977+0000) 2024-03-04T10:33:34.007 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:33:34.004+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:33:34.005238+0000) 2024-03-04T10:33:35.007 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:33:35.004+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:33:35.005486+0000) 2024-03-04T10:33:36.007 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:33:36.004+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:33:36.005750+0000) 2024-03-04T10:33:37.008 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:33:37.005+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:33:37.006006+0000) 2024-03-04T10:33:38.008 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:33:38.005+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:33:38.006215+0000) 2024-03-04T10:33:39.008 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:33:39.005+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:33:39.006456+0000) 2024-03-04T10:33:40.008 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:33:40.005+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:33:40.006687+0000) 2024-03-04T10:33:41.009 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:33:41.006+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:33:41.006928+0000) 2024-03-04T10:33:42.009 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:33:42.006+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:33:42.007156+0000) 2024-03-04T10:33:43.009 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:33:43.006+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:33:43.007402+0000) 2024-03-04T10:33:44.009 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:33:44.006+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:33:44.007680+0000) 2024-03-04T10:33:45.010 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:33:45.007+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:33:45.007955+0000) 2024-03-04T10:33:46.010 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:33:46.007+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:33:46.008167+0000) 2024-03-04T10:33:47.010 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:33:47.007+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:33:47.008407+0000) 2024-03-04T10:33:48.010 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:33:48.007+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:33:48.008609+0000) 2024-03-04T10:33:49.011 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:33:49.008+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:33:49.008929+0000) 2024-03-04T10:33:50.011 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:33:50.008+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:33:50.009166+0000) 2024-03-04T10:33:51.011 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:33:51.008+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:33:51.009409+0000) 2024-03-04T10:33:52.011 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:33:52.008+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:33:52.009652+0000) 2024-03-04T10:33:53.012 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:33:53.009+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:33:53.009893+0000) 2024-03-04T10:33:54.012 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:33:54.009+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:33:54.010179+0000) 2024-03-04T10:33:55.012 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:33:55.009+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:33:55.010427+0000) 2024-03-04T10:33:56.012 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:33:56.009+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:33:56.010677+0000) 2024-03-04T10:33:57.013 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:33:57.010+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:33:57.010958+0000) 2024-03-04T10:33:58.013 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:33:58.010+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:33:58.011187+0000) 2024-03-04T10:33:59.013 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:33:59.010+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:33:59.011475+0000) 2024-03-04T10:34:00.013 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:34:00.011+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:34:00.011717+0000) 2024-03-04T10:34:01.014 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:34:01.011+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:34:01.011966+0000) 2024-03-04T10:34:01.315 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:34:01.323 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:34:02.014 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:34:02.011+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:34:02.012218+0000) 2024-03-04T10:34:03.014 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:34:03.011+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:34:03.012466+0000) 2024-03-04T10:34:04.015 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:34:04.012+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:34:04.012717+0000) 2024-03-04T10:34:05.015 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:34:05.012+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:34:05.012978+0000) 2024-03-04T10:34:06.015 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:34:06.012+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:34:06.013226+0000) 2024-03-04T10:34:07.015 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:34:07.012+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:34:07.013422+0000) 2024-03-04T10:34:08.015 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:34:08.012+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:34:08.013613+0000) 2024-03-04T10:34:09.016 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:34:09.013+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:34:09.013913+0000) 2024-03-04T10:34:10.016 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:34:10.013+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:34:10.014155+0000) 2024-03-04T10:34:11.016 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:34:11.013+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:34:11.014398+0000) 2024-03-04T10:34:12.016 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:34:12.014+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:34:12.014627+0000) 2024-03-04T10:34:13.017 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:34:13.014+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:34:13.014863+0000) 2024-03-04T10:34:14.017 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:34:14.014+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:34:14.015101+0000) 2024-03-04T10:34:15.017 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:34:15.014+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:34:15.015343+0000) 2024-03-04T10:34:16.017 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:34:16.015+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:34:16.015595+0000) 2024-03-04T10:34:17.018 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:34:17.015+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:34:17.015863+0000) 2024-03-04T10:34:18.018 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:34:18.015+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:34:18.016059+0000) 2024-03-04T10:34:19.018 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:34:19.016+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:34:19.016298+0000) 2024-03-04T10:34:20.018 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:34:20.016+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:34:20.016535+0000) 2024-03-04T10:34:21.019 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:34:21.016+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:34:21.016772+0000) 2024-03-04T10:34:22.019 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:34:22.016+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:34:22.016996+0000) 2024-03-04T10:34:23.019 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:34:23.016+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:34:23.017230+0000) 2024-03-04T10:34:24.019 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:34:24.017+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:34:24.017469+0000) 2024-03-04T10:34:25.019 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:34:25.017+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:34:25.017711+0000) 2024-03-04T10:34:26.020 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:34:26.017+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:34:26.017963+0000) 2024-03-04T10:34:27.020 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:34:27.018+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:34:27.018199+0000) 2024-03-04T10:34:28.020 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:34:28.017+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:34:28.018397+0000) 2024-03-04T10:34:29.020 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:34:29.017+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:34:29.018654+0000) 2024-03-04T10:34:30.021 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:34:30.017+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:34:30.018945+0000) 2024-03-04T10:34:31.021 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:34:31.018+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:34:31.019190+0000) 2024-03-04T10:34:31.546 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:34:31.557 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:34:32.022 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:34:32.018+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:34:32.019440+0000) 2024-03-04T10:34:33.022 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:34:33.017+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:34:33.019645+0000) 2024-03-04T10:34:34.022 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:34:34.018+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:34:34.019934+0000) 2024-03-04T10:34:35.022 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:34:35.019+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:34:35.020175+0000) 2024-03-04T10:34:36.022 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:34:36.019+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:34:36.020378+0000) 2024-03-04T10:34:37.022 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:34:37.019+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:34:37.020624+0000) 2024-03-04T10:34:38.023 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:34:38.019+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:34:38.020936+0000) 2024-03-04T10:34:39.023 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:34:39.020+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:34:39.021184+0000) 2024-03-04T10:34:40.023 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:34:40.020+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:34:40.021429+0000) 2024-03-04T10:34:41.024 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:34:41.020+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:34:41.021670+0000) 2024-03-04T10:34:42.024 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:34:42.021+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:34:42.021967+0000) 2024-03-04T10:34:43.024 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:34:43.021+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:34:43.022217+0000) 2024-03-04T10:34:44.024 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:34:44.021+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:34:44.022463+0000) 2024-03-04T10:34:45.024 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:34:45.021+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:34:45.022701+0000) 2024-03-04T10:34:46.025 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:34:46.022+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:34:46.022955+0000) 2024-03-04T10:34:47.025 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:34:47.022+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:34:47.023193+0000) 2024-03-04T10:34:48.025 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:34:48.022+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:34:48.023431+0000) 2024-03-04T10:34:49.025 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:34:49.022+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:34:49.023684+0000) 2024-03-04T10:34:50.026 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:34:50.023+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:34:50.023952+0000) 2024-03-04T10:34:51.026 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:34:51.023+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:34:51.024196+0000) 2024-03-04T10:34:52.026 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:34:52.023+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:34:52.024441+0000) 2024-03-04T10:34:53.026 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:34:53.023+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:34:53.024726+0000) 2024-03-04T10:34:54.027 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:34:54.024+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:34:54.024989+0000) 2024-03-04T10:34:55.027 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:34:55.024+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:34:55.025235+0000) 2024-03-04T10:34:56.027 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:34:56.024+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:34:56.025499+0000) 2024-03-04T10:34:57.029 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:34:57.025+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:34:57.025731+0000) 2024-03-04T10:34:58.028 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:34:58.025+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:34:58.025981+0000) 2024-03-04T10:34:59.028 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:34:59.025+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:34:59.026218+0000) 2024-03-04T10:35:00.028 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:35:00.025+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:35:00.026524+0000) 2024-03-04T10:35:01.031 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:35:01.025+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:35:01.026761+0000) 2024-03-04T10:35:01.759 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:35:01.769 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:35:02.030 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:35:02.026+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:35:02.027013+0000) 2024-03-04T10:35:03.029 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:35:03.026+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:35:03.027260+0000) 2024-03-04T10:35:04.030 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:35:04.026+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:35:04.027519+0000) 2024-03-04T10:35:05.030 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:35:05.027+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:35:05.027765+0000) 2024-03-04T10:35:06.030 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:35:06.027+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:35:06.028036+0000) 2024-03-04T10:35:07.030 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:35:07.027+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:35:07.028278+0000) 2024-03-04T10:35:08.030 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:35:08.028+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:35:08.028516+0000) 2024-03-04T10:35:09.030 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:35:09.028+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:35:09.028755+0000) 2024-03-04T10:35:10.031 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:35:10.028+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:35:10.028986+0000) 2024-03-04T10:35:11.031 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:35:11.028+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:35:11.029231+0000) 2024-03-04T10:35:12.031 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:35:12.028+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:35:12.029467+0000) 2024-03-04T10:35:13.031 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:35:13.029+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:35:13.029721+0000) 2024-03-04T10:35:14.032 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:35:14.029+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:35:14.029974+0000) 2024-03-04T10:35:15.032 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:35:15.029+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:35:15.030174+0000) 2024-03-04T10:35:16.032 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:35:16.030+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:35:16.030428+0000) 2024-03-04T10:35:17.033 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:35:17.030+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:35:17.030673+0000) 2024-03-04T10:35:18.033 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:35:18.030+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:35:18.030957+0000) 2024-03-04T10:35:19.033 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:35:19.030+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:35:19.031220+0000) 2024-03-04T10:35:20.033 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:35:20.031+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:35:20.031455+0000) 2024-03-04T10:35:21.033 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:35:21.031+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:35:21.031695+0000) 2024-03-04T10:35:22.034 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:35:22.031+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:35:22.031942+0000) 2024-03-04T10:35:23.034 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:35:23.031+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:35:23.032140+0000) 2024-03-04T10:35:24.034 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:35:24.031+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:35:24.032374+0000) 2024-03-04T10:35:25.034 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:35:25.031+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:35:25.032609+0000) 2024-03-04T10:35:26.035 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:35:26.032+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:35:26.032914+0000) 2024-03-04T10:35:27.035 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:35:27.032+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:35:27.033163+0000) 2024-03-04T10:35:28.035 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:35:28.033+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:35:28.033412+0000) 2024-03-04T10:35:29.035 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:35:29.033+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:35:29.033668+0000) 2024-03-04T10:35:30.036 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:35:30.033+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:35:30.033932+0000) 2024-03-04T10:35:31.036 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:35:31.034+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:35:31.034211+0000) 2024-03-04T10:35:31.972 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:35:31.980 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:35:32.036 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:35:32.032+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:35:32.034379+0000) 2024-03-04T10:35:33.036 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:35:33.033+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:35:33.034606+0000) 2024-03-04T10:35:34.037 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:35:34.033+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:35:34.034911+0000) 2024-03-04T10:35:35.037 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:35:35.034+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:35:35.035150+0000) 2024-03-04T10:35:36.037 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:35:36.034+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:35:36.035398+0000) 2024-03-04T10:35:37.037 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:35:37.033+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:35:37.035595+0000) 2024-03-04T10:35:38.038 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:35:38.034+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:35:38.035900+0000) 2024-03-04T10:35:39.038 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:35:39.035+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:35:39.036148+0000) 2024-03-04T10:35:40.038 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:35:40.035+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:35:40.036393+0000) 2024-03-04T10:35:41.038 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:35:41.035+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:35:41.036633+0000) 2024-03-04T10:35:42.039 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:35:42.035+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:35:42.036922+0000) 2024-03-04T10:35:43.040 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:35:43.036+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:35:43.037168+0000) 2024-03-04T10:35:44.039 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:35:44.036+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:35:44.037413+0000) 2024-03-04T10:35:45.040 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:35:45.036+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:35:45.037673+0000) 2024-03-04T10:35:46.040 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:35:46.037+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:35:46.037949+0000) 2024-03-04T10:35:47.040 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:35:47.037+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:35:47.038190+0000) 2024-03-04T10:35:48.040 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:35:48.037+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:35:48.038426+0000) 2024-03-04T10:35:49.043 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:35:49.037+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:35:49.038664+0000) 2024-03-04T10:35:50.041 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:35:50.038+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:35:50.038948+0000) 2024-03-04T10:35:51.041 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:35:51.038+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:35:51.039216+0000) 2024-03-04T10:35:52.041 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:35:52.038+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:35:52.039394+0000) 2024-03-04T10:35:53.041 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:35:53.038+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:35:53.039667+0000) 2024-03-04T10:35:54.042 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:35:54.039+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:35:54.039960+0000) 2024-03-04T10:35:55.042 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:35:55.039+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:35:55.040207+0000) 2024-03-04T10:35:56.042 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:35:56.039+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:35:56.040464+0000) 2024-03-04T10:35:57.042 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:35:57.039+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:35:57.040704+0000) 2024-03-04T10:35:58.043 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:35:58.040+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:35:58.040954+0000) 2024-03-04T10:35:59.043 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:35:59.040+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:35:59.041219+0000) 2024-03-04T10:36:00.043 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:36:00.040+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:36:00.041493+0000) 2024-03-04T10:36:01.044 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:36:01.041+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:36:01.041765+0000) 2024-03-04T10:36:02.044 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:36:02.041+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:36:02.042009+0000) 2024-03-04T10:36:02.171 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:36:02.179 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:36:03.044 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:36:03.041+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:36:03.042258+0000) 2024-03-04T10:36:04.044 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:36:04.041+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:36:04.042509+0000) 2024-03-04T10:36:05.045 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:36:05.042+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:36:05.042738+0000) 2024-03-04T10:36:06.045 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:36:06.042+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:36:06.042949+0000) 2024-03-04T10:36:07.045 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:36:07.042+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:36:07.043201+0000) 2024-03-04T10:36:08.045 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:36:08.042+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:36:08.043446+0000) 2024-03-04T10:36:09.046 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:36:09.043+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:36:09.043696+0000) 2024-03-04T10:36:10.046 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:36:10.043+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:36:10.043947+0000) 2024-03-04T10:36:11.046 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:36:11.043+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:36:11.044186+0000) 2024-03-04T10:36:12.046 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:36:12.043+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:36:12.044426+0000) 2024-03-04T10:36:13.047 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:36:13.044+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:36:13.044680+0000) 2024-03-04T10:36:14.047 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:36:14.044+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:36:14.044965+0000) 2024-03-04T10:36:15.047 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:36:15.044+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:36:15.045206+0000) 2024-03-04T10:36:16.047 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:36:16.045+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:36:16.045451+0000) 2024-03-04T10:36:17.047 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:36:17.044+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:36:17.045641+0000) 2024-03-04T10:36:18.048 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:36:18.045+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:36:18.045941+0000) 2024-03-04T10:36:19.048 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:36:19.045+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:36:19.046187+0000) 2024-03-04T10:36:20.048 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:36:20.045+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:36:20.046383+0000) 2024-03-04T10:36:21.048 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:36:21.046+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:36:21.046624+0000) 2024-03-04T10:36:22.049 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:36:22.046+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:36:22.046929+0000) 2024-03-04T10:36:23.049 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:36:23.046+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:36:23.047179+0000) 2024-03-04T10:36:24.049 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:36:24.047+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:36:24.047460+0000) 2024-03-04T10:36:25.050 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:36:25.046+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:36:25.047708+0000) 2024-03-04T10:36:26.050 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:36:26.047+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:36:26.047964+0000) 2024-03-04T10:36:27.050 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:36:27.047+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:36:27.048174+0000) 2024-03-04T10:36:28.050 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:36:28.048+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:36:28.048425+0000) 2024-03-04T10:36:29.050 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:36:29.047+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:36:29.048669+0000) 2024-03-04T10:36:30.051 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:36:30.048+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:36:30.048948+0000) 2024-03-04T10:36:31.051 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:36:31.048+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:36:31.049212+0000) 2024-03-04T10:36:32.051 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:36:32.049+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:36:32.049459+0000) 2024-03-04T10:36:32.383 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:36:32.395 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:36:33.052 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:36:33.049+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:36:33.049702+0000) 2024-03-04T10:36:34.052 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:36:34.048+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:36:34.049950+0000) 2024-03-04T10:36:35.052 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:36:35.049+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:36:35.050188+0000) 2024-03-04T10:36:36.052 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:36:36.049+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:36:36.050439+0000) 2024-03-04T10:36:37.052 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:36:37.049+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:36:37.050675+0000) 2024-03-04T10:36:38.053 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:36:38.049+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:36:38.050967+0000) 2024-03-04T10:36:39.053 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:36:39.050+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:36:39.051211+0000) 2024-03-04T10:36:40.053 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:36:40.049+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:36:40.051401+0000) 2024-03-04T10:36:41.053 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:36:41.050+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:36:41.051647+0000) 2024-03-04T10:36:42.054 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:36:42.050+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:36:42.051932+0000) 2024-03-04T10:36:43.054 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:36:43.051+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:36:43.052181+0000) 2024-03-04T10:36:44.054 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:36:44.051+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:36:44.052424+0000) 2024-03-04T10:36:45.054 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:36:45.051+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:36:45.052664+0000) 2024-03-04T10:36:46.055 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:36:46.052+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:36:46.052953+0000) 2024-03-04T10:36:47.055 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:36:47.052+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:36:47.053198+0000) 2024-03-04T10:36:48.055 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:36:48.052+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:36:48.053449+0000) 2024-03-04T10:36:49.055 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:36:49.052+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:36:49.053695+0000) 2024-03-04T10:36:50.056 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:36:50.053+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:36:50.053946+0000) 2024-03-04T10:36:51.056 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:36:51.053+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:36:51.054190+0000) 2024-03-04T10:36:52.056 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:36:52.053+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:36:52.054452+0000) 2024-03-04T10:36:53.057 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:36:53.053+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:36:53.054703+0000) 2024-03-04T10:36:54.057 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:36:54.054+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:36:54.054950+0000) 2024-03-04T10:36:55.057 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:36:55.054+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:36:55.055190+0000) 2024-03-04T10:36:56.057 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:36:56.054+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:36:56.055469+0000) 2024-03-04T10:36:57.058 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:36:57.054+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:36:57.055717+0000) 2024-03-04T10:36:58.058 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:36:58.055+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:36:58.055963+0000) 2024-03-04T10:36:59.058 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:36:59.055+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:36:59.056205+0000) 2024-03-04T10:37:00.058 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:37:00.055+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:37:00.056429+0000) 2024-03-04T10:37:01.059 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:37:01.055+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:37:01.056668+0000) 2024-03-04T10:37:02.059 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:37:02.056+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:37:02.056957+0000) 2024-03-04T10:37:02.597 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:37:02.606 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:37:03.059 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:37:03.056+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:37:03.057178+0000) 2024-03-04T10:37:04.059 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:37:04.056+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:37:04.057456+0000) 2024-03-04T10:37:05.059 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:37:05.056+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:37:05.057718+0000) 2024-03-04T10:37:06.060 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:37:06.057+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:37:06.057988+0000) 2024-03-04T10:37:07.060 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:37:07.057+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:37:07.058266+0000) 2024-03-04T10:37:08.060 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:37:08.057+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:37:08.058567+0000) 2024-03-04T10:37:09.061 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:37:09.058+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:37:09.058874+0000) 2024-03-04T10:37:10.061 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:37:10.058+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:37:10.059074+0000) 2024-03-04T10:37:11.061 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:37:11.058+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:37:11.059314+0000) 2024-03-04T10:37:12.061 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:37:12.059+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:37:12.059549+0000) 2024-03-04T10:37:13.062 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:37:13.059+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:37:13.059876+0000) 2024-03-04T10:37:14.062 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:37:14.059+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:37:14.060068+0000) 2024-03-04T10:37:15.062 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:37:15.059+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:37:15.060312+0000) 2024-03-04T10:37:16.063 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:37:16.060+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:37:16.060569+0000) 2024-03-04T10:37:17.063 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:37:17.060+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:37:17.060817+0000) 2024-03-04T10:37:18.063 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:37:18.060+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:37:18.061113+0000) 2024-03-04T10:37:19.063 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:37:19.060+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:37:19.061364+0000) 2024-03-04T10:37:20.064 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:37:20.061+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:37:20.061622+0000) 2024-03-04T10:37:21.064 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:37:21.061+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:37:21.061929+0000) 2024-03-04T10:37:22.065 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:37:22.061+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:37:22.062175+0000) 2024-03-04T10:37:23.064 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:37:23.062+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:37:23.062446+0000) 2024-03-04T10:37:24.065 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:37:24.061+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:37:24.062637+0000) 2024-03-04T10:37:25.065 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:37:25.062+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:37:25.062936+0000) 2024-03-04T10:37:26.065 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:37:26.062+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:37:26.063186+0000) 2024-03-04T10:37:27.065 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:37:27.063+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:37:27.063448+0000) 2024-03-04T10:37:28.066 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:37:28.062+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:37:28.063696+0000) 2024-03-04T10:37:29.066 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:37:29.063+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:37:29.063942+0000) 2024-03-04T10:37:30.066 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:37:30.063+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:37:30.064209+0000) 2024-03-04T10:37:31.066 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:37:31.063+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:37:31.064458+0000) 2024-03-04T10:37:32.067 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:37:32.064+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:37:32.064687+0000) 2024-03-04T10:37:32.745 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:37:32.753 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:37:33.067 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:37:33.064+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:37:33.064940+0000) 2024-03-04T10:37:34.067 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:37:34.065+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:37:34.065198+0000) 2024-03-04T10:37:35.067 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:37:35.065+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:37:35.065459+0000) 2024-03-04T10:37:36.068 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:37:36.065+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:37:36.065686+0000) 2024-03-04T10:37:37.068 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:37:37.064+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:37:37.065930+0000) 2024-03-04T10:37:38.068 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:37:38.065+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:37:38.066180+0000) 2024-03-04T10:37:39.068 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:37:39.065+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:37:39.066420+0000) 2024-03-04T10:37:40.068 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:37:40.065+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:37:40.066659+0000) 2024-03-04T10:37:41.069 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:37:41.065+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:37:41.066947+0000) 2024-03-04T10:37:42.069 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:37:42.066+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:37:42.067234+0000) 2024-03-04T10:37:43.069 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:37:43.066+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:37:43.067494+0000) 2024-03-04T10:37:44.070 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:37:44.066+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:37:44.067746+0000) 2024-03-04T10:37:45.070 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:37:45.066+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:37:45.067951+0000) 2024-03-04T10:37:46.070 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:37:46.067+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:37:46.068206+0000) 2024-03-04T10:37:47.070 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:37:47.067+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:37:47.068445+0000) 2024-03-04T10:37:48.071 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:37:48.067+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:37:48.068698+0000) 2024-03-04T10:37:49.071 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:37:49.068+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:37:49.068971+0000) 2024-03-04T10:37:50.071 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:37:50.068+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:37:50.069208+0000) 2024-03-04T10:37:51.071 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:37:51.068+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:37:51.069453+0000) 2024-03-04T10:37:52.072 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:37:52.068+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:37:52.069711+0000) 2024-03-04T10:37:53.072 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:37:53.069+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:37:53.069968+0000) 2024-03-04T10:37:54.072 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:37:54.069+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:37:54.070219+0000) 2024-03-04T10:37:55.073 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:37:55.069+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:37:55.070480+0000) 2024-03-04T10:37:56.073 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:37:56.069+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:37:56.070786+0000) 2024-03-04T10:37:57.073 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:37:57.070+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:37:57.071035+0000) 2024-03-04T10:37:58.073 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:37:58.070+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:37:58.071282+0000) 2024-03-04T10:37:59.073 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:37:59.070+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:37:59.071484+0000) 2024-03-04T10:38:00.074 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:38:00.070+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:38:00.071750+0000) 2024-03-04T10:38:01.074 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:38:01.071+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:38:01.072034+0000) 2024-03-04T10:38:02.074 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:38:02.071+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:38:02.072324+0000) 2024-03-04T10:38:02.903 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:38:03.093 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:38:03.071+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:38:03.072613+0000) 2024-03-04T10:38:03.100 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:38:04.075 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:38:04.072+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:38:04.072927+0000) 2024-03-04T10:38:05.075 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:38:05.072+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:38:05.073167+0000) 2024-03-04T10:38:06.075 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:38:06.072+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:38:06.073414+0000) 2024-03-04T10:38:07.076 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:38:07.073+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:38:07.073653+0000) 2024-03-04T10:38:08.076 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:38:08.073+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:38:08.073950+0000) 2024-03-04T10:38:09.076 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:38:09.073+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:38:09.074202+0000) 2024-03-04T10:38:10.076 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:38:10.073+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:38:10.074454+0000) 2024-03-04T10:38:11.077 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:38:11.074+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:38:11.074701+0000) 2024-03-04T10:38:12.077 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:38:12.074+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:38:12.074941+0000) 2024-03-04T10:38:13.077 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:38:13.074+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:38:13.075183+0000) 2024-03-04T10:38:14.077 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:38:14.074+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:38:14.075436+0000) 2024-03-04T10:38:15.078 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:38:15.075+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:38:15.075684+0000) 2024-03-04T10:38:16.078 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:38:16.075+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:38:16.075934+0000) 2024-03-04T10:38:17.078 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:38:17.075+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:38:17.076183+0000) 2024-03-04T10:38:18.078 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:38:18.075+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:38:18.076446+0000) 2024-03-04T10:38:19.079 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:38:19.076+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:38:19.076720+0000) 2024-03-04T10:38:20.079 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:38:20.076+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:38:20.076960+0000) 2024-03-04T10:38:21.079 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:38:21.076+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:38:21.077203+0000) 2024-03-04T10:38:22.079 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:38:22.077+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:38:22.077450+0000) 2024-03-04T10:38:23.079 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:38:23.077+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:38:23.077690+0000) 2024-03-04T10:38:24.080 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:38:24.077+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:38:24.077938+0000) 2024-03-04T10:38:25.080 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:38:25.077+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:38:25.078214+0000) 2024-03-04T10:38:26.080 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:38:26.078+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:38:26.078466+0000) 2024-03-04T10:38:27.081 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:38:27.078+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:38:27.078703+0000) 2024-03-04T10:38:28.081 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:38:28.078+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:38:28.078942+0000) 2024-03-04T10:38:29.081 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:38:29.078+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:38:29.079212+0000) 2024-03-04T10:38:30.081 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:38:30.079+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:38:30.079473+0000) 2024-03-04T10:38:31.082 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:38:31.079+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:38:31.079718+0000) 2024-03-04T10:38:32.082 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:38:32.079+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:38:32.079960+0000) 2024-03-04T10:38:33.082 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:38:33.079+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:38:33.080227+0000) 2024-03-04T10:38:33.231 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:38:33.239 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:38:34.082 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:38:34.080+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:38:34.080502+0000) 2024-03-04T10:38:35.083 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:38:35.080+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:38:35.080764+0000) 2024-03-04T10:38:36.083 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:38:36.080+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:38:36.081009+0000) 2024-03-04T10:38:37.083 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:38:37.081+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:38:37.081255+0000) 2024-03-04T10:38:38.083 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:38:38.081+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:38:38.081499+0000) 2024-03-04T10:38:39.084 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:38:39.080+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:38:39.081807+0000) 2024-03-04T10:38:40.084 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:38:40.080+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:38:40.082090+0000) 2024-03-04T10:38:41.084 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:38:41.081+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:38:41.082348+0000) 2024-03-04T10:38:42.084 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:38:42.081+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:38:42.082593+0000) 2024-03-04T10:38:43.085 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:38:43.081+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:38:43.082873+0000) 2024-03-04T10:38:44.085 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:38:44.082+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:38:44.083137+0000) 2024-03-04T10:38:45.085 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:38:45.082+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:38:45.083362+0000) 2024-03-04T10:38:46.085 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:38:46.082+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:38:46.083609+0000) 2024-03-04T10:38:47.086 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:38:47.082+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:38:47.083904+0000) 2024-03-04T10:38:48.086 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:38:48.083+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:38:48.084153+0000) 2024-03-04T10:38:49.086 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:38:49.083+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:38:49.084445+0000) 2024-03-04T10:38:50.087 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:38:50.083+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:38:50.084721+0000) 2024-03-04T10:38:51.087 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:38:51.084+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:38:51.084967+0000) 2024-03-04T10:38:52.087 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:38:52.084+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:38:52.085201+0000) 2024-03-04T10:38:53.087 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:38:53.084+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:38:53.085447+0000) 2024-03-04T10:38:54.088 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:38:54.084+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:38:54.085695+0000) 2024-03-04T10:38:55.088 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:38:55.085+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:38:55.085939+0000) 2024-03-04T10:38:56.088 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:38:56.085+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:38:56.086189+0000) 2024-03-04T10:38:57.088 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:38:57.085+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:38:57.086432+0000) 2024-03-04T10:38:58.089 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:38:58.085+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:38:58.086701+0000) 2024-03-04T10:38:59.089 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:38:59.086+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:38:59.086997+0000) 2024-03-04T10:39:00.089 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:39:00.086+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:39:00.087262+0000) 2024-03-04T10:39:01.090 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:39:01.086+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:39:01.087527+0000) 2024-03-04T10:39:02.090 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:39:02.087+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:39:02.087776+0000) 2024-03-04T10:39:03.090 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:39:03.087+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:39:03.088036+0000) 2024-03-04T10:39:03.450 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:39:03.458 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:39:04.090 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:39:04.087+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:39:04.088335+0000) 2024-03-04T10:39:05.091 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:39:05.087+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:39:05.088601+0000) 2024-03-04T10:39:06.091 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:39:06.088+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:39:06.088922+0000) 2024-03-04T10:39:07.091 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:39:07.088+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:39:07.089165+0000) 2024-03-04T10:39:08.091 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:39:08.088+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:39:08.089424+0000) 2024-03-04T10:39:09.092 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:39:09.089+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:39:09.089679+0000) 2024-03-04T10:39:10.092 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:39:10.089+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:39:10.089959+0000) 2024-03-04T10:39:11.092 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:39:11.089+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:39:11.090219+0000) 2024-03-04T10:39:12.093 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:39:12.089+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:39:12.090474+0000) 2024-03-04T10:39:13.093 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:39:13.090+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:39:13.090733+0000) 2024-03-04T10:39:14.093 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:39:14.090+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:39:14.090993+0000) 2024-03-04T10:39:15.093 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:39:15.090+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:39:15.091246+0000) 2024-03-04T10:39:16.094 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:39:16.090+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:39:16.091614+0000) 2024-03-04T10:39:17.094 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:39:17.091+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:39:17.091930+0000) 2024-03-04T10:39:18.094 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:39:18.091+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:39:18.092170+0000) 2024-03-04T10:39:19.094 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:39:19.091+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:39:19.092407+0000) 2024-03-04T10:39:20.096 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:39:20.092+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:39:20.092706+0000) 2024-03-04T10:39:21.095 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:39:21.092+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:39:21.092998+0000) 2024-03-04T10:39:22.095 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:39:22.092+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:39:22.093303+0000) 2024-03-04T10:39:23.096 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:39:23.092+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:39:23.093615+0000) 2024-03-04T10:39:24.096 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:39:24.093+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:39:24.093931+0000) 2024-03-04T10:39:25.096 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:39:25.093+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:39:25.094252+0000) 2024-03-04T10:39:26.097 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:39:26.094+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:39:26.094593+0000) 2024-03-04T10:39:27.097 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:39:27.094+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:39:27.094931+0000) 2024-03-04T10:39:28.097 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:39:28.094+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:39:28.095225+0000) 2024-03-04T10:39:29.097 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:39:29.095+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:39:29.095525+0000) 2024-03-04T10:39:30.098 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:39:30.095+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:39:30.095784+0000) 2024-03-04T10:39:31.098 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:39:31.095+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:39:31.096032+0000) 2024-03-04T10:39:32.098 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:39:32.095+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:39:32.096270+0000) 2024-03-04T10:39:33.098 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:39:33.096+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:39:33.096553+0000) 2024-03-04T10:39:33.672 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:39:33.680 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:39:34.099 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:39:34.096+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:39:34.096895+0000) 2024-03-04T10:39:35.099 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:39:35.096+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:39:35.097189+0000) 2024-03-04T10:39:36.099 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:39:36.097+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:39:36.097450+0000) 2024-03-04T10:39:37.100 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:39:37.097+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:39:37.097759+0000) 2024-03-04T10:39:38.100 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:39:38.097+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:39:38.098029+0000) 2024-03-04T10:39:39.100 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:39:39.098+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:39:39.098277+0000) 2024-03-04T10:39:40.101 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:39:40.098+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:39:40.098521+0000) 2024-03-04T10:39:41.101 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:39:41.098+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:39:41.098764+0000) 2024-03-04T10:39:42.101 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:39:42.097+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:39:42.099001+0000) 2024-03-04T10:39:43.101 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:39:43.098+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:39:43.099255+0000) 2024-03-04T10:39:44.102 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:39:44.097+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:39:44.099498+0000) 2024-03-04T10:39:45.102 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:39:45.097+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:39:45.099751+0000) 2024-03-04T10:39:46.102 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:39:46.098+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:39:46.099998+0000) 2024-03-04T10:39:47.102 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:39:47.099+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:39:47.100235+0000) 2024-03-04T10:39:48.102 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:39:48.099+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:39:48.100478+0000) 2024-03-04T10:39:49.103 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:39:49.099+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:39:49.100720+0000) 2024-03-04T10:39:50.103 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:39:50.099+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:39:50.100983+0000) 2024-03-04T10:39:51.103 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:39:51.100+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:39:51.101233+0000) 2024-03-04T10:39:52.103 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:39:52.100+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:39:52.101489+0000) 2024-03-04T10:39:53.104 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:39:53.100+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:39:53.101779+0000) 2024-03-04T10:39:54.104 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:39:54.101+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:39:54.102148+0000) 2024-03-04T10:39:55.104 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:39:55.101+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:39:55.102473+0000) 2024-03-04T10:39:56.105 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:39:56.101+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:39:56.102788+0000) 2024-03-04T10:39:57.105 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:39:57.102+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:39:57.103079+0000) 2024-03-04T10:39:58.106 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:39:58.102+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:39:58.103379+0000) 2024-03-04T10:39:59.106 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:39:59.102+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:39:59.103637+0000) 2024-03-04T10:40:00.106 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:40:00.103+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:40:00.103957+0000) 2024-03-04T10:40:01.106 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:40:01.103+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:40:01.104256+0000) 2024-03-04T10:40:02.107 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:40:02.103+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:40:02.104557+0000) 2024-03-04T10:40:03.107 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:40:03.104+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:40:03.104875+0000) 2024-03-04T10:40:03.850 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:40:03.858 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:40:04.107 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:40:04.104+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:40:04.105184+0000) 2024-03-04T10:40:05.107 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:40:05.104+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:40:05.105435+0000) 2024-03-04T10:40:06.108 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:40:06.104+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:40:06.105740+0000) 2024-03-04T10:40:07.108 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:40:07.105+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:40:07.106024+0000) 2024-03-04T10:40:08.108 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:40:08.105+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:40:08.106282+0000) 2024-03-04T10:40:09.109 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:40:09.105+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:40:09.106552+0000) 2024-03-04T10:40:10.109 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:40:10.106+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:40:10.106814+0000) 2024-03-04T10:40:11.109 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:40:11.106+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:40:11.107136+0000) 2024-03-04T10:40:12.111 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:40:12.106+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:40:12.107409+0000) 2024-03-04T10:40:13.110 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:40:13.106+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:40:13.107696+0000) 2024-03-04T10:40:14.110 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:40:14.107+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:40:14.107955+0000) 2024-03-04T10:40:15.110 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:40:15.107+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:40:15.108228+0000) 2024-03-04T10:40:16.110 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:40:16.107+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:40:16.108445+0000) 2024-03-04T10:40:17.111 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:40:17.108+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:40:17.108694+0000) 2024-03-04T10:40:18.111 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:40:18.108+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:40:18.108950+0000) 2024-03-04T10:40:19.112 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:40:19.108+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:40:19.109212+0000) 2024-03-04T10:40:20.112 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:40:20.108+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:40:20.109490+0000) 2024-03-04T10:40:21.112 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:40:21.109+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:40:21.109780+0000) 2024-03-04T10:40:22.112 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:40:22.109+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:40:22.110078+0000) 2024-03-04T10:40:23.112 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:40:23.109+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:40:23.110330+0000) 2024-03-04T10:40:24.113 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:40:24.110+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:40:24.110585+0000) 2024-03-04T10:40:25.113 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:40:25.110+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:40:25.110794+0000) 2024-03-04T10:40:26.113 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:40:26.110+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:40:26.111077+0000) 2024-03-04T10:40:27.113 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:40:27.110+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:40:27.111334+0000) 2024-03-04T10:40:28.114 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:40:28.110+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:40:28.111546+0000) 2024-03-04T10:40:29.114 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:40:29.111+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:40:29.111859+0000) 2024-03-04T10:40:30.114 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:40:30.111+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:40:30.112123+0000) 2024-03-04T10:40:31.114 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:40:31.112+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:40:31.112426+0000) 2024-03-04T10:40:32.115 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:40:32.112+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:40:32.112717+0000) 2024-03-04T10:40:33.115 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:40:33.112+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:40:33.113015+0000) 2024-03-04T10:40:34.065 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:40:34.073 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:40:34.116 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:40:34.112+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:40:34.113207+0000) 2024-03-04T10:40:35.115 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:40:35.113+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:40:35.113436+0000) 2024-03-04T10:40:36.116 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:40:36.113+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:40:36.113695+0000) 2024-03-04T10:40:37.116 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:40:37.113+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:40:37.113924+0000) 2024-03-04T10:40:38.116 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:40:38.113+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:40:38.114165+0000) 2024-03-04T10:40:39.116 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:40:39.114+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:40:39.114412+0000) 2024-03-04T10:40:40.117 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:40:40.114+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:40:40.114656+0000) 2024-03-04T10:40:41.117 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:40:41.114+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:40:41.114965+0000) 2024-03-04T10:40:42.117 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:40:42.115+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:40:42.115216+0000) 2024-03-04T10:40:43.118 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:40:43.114+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:40:43.115484+0000) 2024-03-04T10:40:44.118 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:40:44.115+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:40:44.115760+0000) 2024-03-04T10:40:45.118 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:40:45.114+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:40:45.116010+0000) 2024-03-04T10:40:46.118 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:40:46.115+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:40:46.116265+0000) 2024-03-04T10:40:47.119 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:40:47.115+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:40:47.116520+0000) 2024-03-04T10:40:48.119 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:40:48.115+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:40:48.116765+0000) 2024-03-04T10:40:49.119 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:40:49.115+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:40:49.117029+0000) 2024-03-04T10:40:50.119 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:40:50.116+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:40:50.117313+0000) 2024-03-04T10:40:51.120 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:40:51.115+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:40:51.117552+0000) 2024-03-04T10:40:52.120 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:40:52.116+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:40:52.117789+0000) 2024-03-04T10:40:53.120 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:40:53.117+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:40:53.118059+0000) 2024-03-04T10:40:54.120 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:40:54.117+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:40:54.118309+0000) 2024-03-04T10:40:55.121 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:40:55.117+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:40:55.118550+0000) 2024-03-04T10:40:56.121 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:40:56.118+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:40:56.118811+0000) 2024-03-04T10:40:57.121 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:40:57.118+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:40:57.119107+0000) 2024-03-04T10:40:58.121 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:40:58.118+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:40:58.119357+0000) 2024-03-04T10:40:59.122 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:40:59.118+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:40:59.119611+0000) 2024-03-04T10:41:00.122 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:41:00.119+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:41:00.119900+0000) 2024-03-04T10:41:01.122 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:41:01.119+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:41:01.120159+0000) 2024-03-04T10:41:02.123 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:41:02.119+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:41:02.120400+0000) 2024-03-04T10:41:03.123 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:41:03.119+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:41:03.120650+0000) 2024-03-04T10:41:04.123 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:41:04.120+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:41:04.120940+0000) 2024-03-04T10:41:04.211 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:41:04.220 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:41:05.123 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:41:05.120+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:41:05.121183+0000) 2024-03-04T10:41:06.123 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:41:06.120+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:41:06.121459+0000) 2024-03-04T10:41:07.124 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:41:07.120+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:41:07.121699+0000) 2024-03-04T10:41:08.124 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:41:08.121+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:41:08.121941+0000) 2024-03-04T10:41:09.124 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:41:09.121+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:41:09.122192+0000) 2024-03-04T10:41:10.125 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:41:10.121+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:41:10.122445+0000) 2024-03-04T10:41:11.125 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:41:11.122+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:41:11.122701+0000) 2024-03-04T10:41:12.125 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:41:12.122+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:41:12.122963+0000) 2024-03-04T10:41:13.125 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:41:13.122+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:41:13.123213+0000) 2024-03-04T10:41:14.126 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:41:14.122+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:41:14.123453+0000) 2024-03-04T10:41:15.126 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:41:15.122+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:41:15.123653+0000) 2024-03-04T10:41:16.126 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:41:16.123+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:41:16.123953+0000) 2024-03-04T10:41:17.126 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:41:17.123+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:41:17.124183+0000) 2024-03-04T10:41:18.127 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:41:18.123+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:41:18.124423+0000) 2024-03-04T10:41:19.127 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:41:19.124+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:41:19.124703+0000) 2024-03-04T10:41:20.127 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:41:20.124+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:41:20.124970+0000) 2024-03-04T10:41:21.127 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:41:21.124+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:41:21.125215+0000) 2024-03-04T10:41:22.127 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:41:22.124+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:41:22.125461+0000) 2024-03-04T10:41:23.128 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:41:23.125+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:41:23.125704+0000) 2024-03-04T10:41:24.128 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:41:24.125+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:41:24.125961+0000) 2024-03-04T10:41:25.128 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:41:25.125+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:41:25.126168+0000) 2024-03-04T10:41:26.128 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:41:26.125+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:41:26.126421+0000) 2024-03-04T10:41:27.129 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:41:27.126+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:41:27.126666+0000) 2024-03-04T10:41:28.129 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:41:28.126+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:41:28.126955+0000) 2024-03-04T10:41:29.129 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:41:29.126+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:41:29.127210+0000) 2024-03-04T10:41:30.130 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:41:30.127+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:41:30.127458+0000) 2024-03-04T10:41:31.130 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:41:31.126+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:41:31.127718+0000) 2024-03-04T10:41:32.130 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:41:32.127+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:41:32.127958+0000) 2024-03-04T10:41:33.130 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:41:33.127+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:41:33.128192+0000) 2024-03-04T10:41:34.131 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:41:34.128+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:41:34.128443+0000) 2024-03-04T10:41:34.425 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:41:34.433 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:41:35.131 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:41:35.128+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:41:35.128698+0000) 2024-03-04T10:41:36.131 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:41:36.128+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:41:36.128953+0000) 2024-03-04T10:41:37.131 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:41:37.128+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:41:37.129194+0000) 2024-03-04T10:41:38.132 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:41:38.129+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:41:38.129443+0000) 2024-03-04T10:41:39.132 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:41:39.129+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:41:39.129706+0000) 2024-03-04T10:41:40.132 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:41:40.129+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:41:40.129950+0000) 2024-03-04T10:41:41.132 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:41:41.129+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:41:41.130193+0000) 2024-03-04T10:41:42.133 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:41:42.130+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:41:42.130470+0000) 2024-03-04T10:41:43.133 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:41:43.130+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:41:43.130764+0000) 2024-03-04T10:41:44.133 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:41:44.130+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:41:44.131031+0000) 2024-03-04T10:41:45.133 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:41:45.131+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:41:45.131283+0000) 2024-03-04T10:41:46.134 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:41:46.131+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:41:46.131579+0000) 2024-03-04T10:41:47.134 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:41:47.131+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:41:47.131926+0000) 2024-03-04T10:41:48.134 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:41:48.131+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:41:48.132240+0000) 2024-03-04T10:41:49.135 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:41:49.131+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:41:49.132550+0000) 2024-03-04T10:41:50.135 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:41:50.131+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:41:50.132896+0000) 2024-03-04T10:41:51.135 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:41:51.132+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:41:51.133154+0000) 2024-03-04T10:41:52.135 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:41:52.131+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:41:52.133338+0000) 2024-03-04T10:41:53.136 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:41:53.132+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:41:53.133586+0000) 2024-03-04T10:41:54.136 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:41:54.132+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:41:54.133892+0000) 2024-03-04T10:41:55.136 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:41:55.133+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:41:55.134158+0000) 2024-03-04T10:41:56.137 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:41:56.133+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:41:56.134474+0000) 2024-03-04T10:41:57.137 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:41:57.133+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:41:57.134723+0000) 2024-03-04T10:41:58.137 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:41:58.134+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:41:58.134965+0000) 2024-03-04T10:41:59.137 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:41:59.134+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:41:59.135161+0000) 2024-03-04T10:42:00.138 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:42:00.134+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:42:00.135403+0000) 2024-03-04T10:42:01.138 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:42:01.134+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:42:01.135651+0000) 2024-03-04T10:42:02.138 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:42:02.135+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:42:02.135936+0000) 2024-03-04T10:42:03.138 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:42:03.135+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:42:03.136206+0000) 2024-03-04T10:42:04.139 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:42:04.135+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:42:04.136511+0000) 2024-03-04T10:42:04.637 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:42:04.645 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:42:05.139 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:42:05.135+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:42:05.136796+0000) 2024-03-04T10:42:06.139 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:42:06.136+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:42:06.137161+0000) 2024-03-04T10:42:07.140 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:42:07.136+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:42:07.137441+0000) 2024-03-04T10:42:08.140 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:42:08.136+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:42:08.137726+0000) 2024-03-04T10:42:09.140 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:42:09.137+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:42:09.138065+0000) 2024-03-04T10:42:10.141 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:42:10.137+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:42:10.138377+0000) 2024-03-04T10:42:11.141 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:42:11.137+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:42:11.138649+0000) 2024-03-04T10:42:12.141 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:42:12.138+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:42:12.138939+0000) 2024-03-04T10:42:13.141 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:42:13.138+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:42:13.139176+0000) 2024-03-04T10:42:14.141 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:42:14.138+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:42:14.139430+0000) 2024-03-04T10:42:15.142 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:42:15.139+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:42:15.139700+0000) 2024-03-04T10:42:16.142 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:42:16.139+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:42:16.139966+0000) 2024-03-04T10:42:17.143 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:42:17.139+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:42:17.140213+0000) 2024-03-04T10:42:18.143 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:42:18.139+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:42:18.140461+0000) 2024-03-04T10:42:19.143 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:42:19.140+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:42:19.140716+0000) 2024-03-04T10:42:20.143 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:42:20.140+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:42:20.140970+0000) 2024-03-04T10:42:21.143 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:42:21.140+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:42:21.141218+0000) 2024-03-04T10:42:22.144 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:42:22.140+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:42:22.141462+0000) 2024-03-04T10:42:23.144 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:42:23.141+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:42:23.141699+0000) 2024-03-04T10:42:24.144 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:42:24.141+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:42:24.141946+0000) 2024-03-04T10:42:25.144 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:42:25.141+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:42:25.142197+0000) 2024-03-04T10:42:26.145 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:42:26.141+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:42:26.142410+0000) 2024-03-04T10:42:27.145 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:42:27.142+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:42:27.142649+0000) 2024-03-04T10:42:28.145 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:42:28.142+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:42:28.142938+0000) 2024-03-04T10:42:29.145 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:42:29.142+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:42:29.143181+0000) 2024-03-04T10:42:30.146 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:42:30.142+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:42:30.143429+0000) 2024-03-04T10:42:31.146 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:42:31.143+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:42:31.143714+0000) 2024-03-04T10:42:32.146 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:42:32.143+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:42:32.143966+0000) 2024-03-04T10:42:33.146 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:42:33.143+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:42:33.144159+0000) 2024-03-04T10:42:34.147 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:42:34.144+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:42:34.144417+0000) 2024-03-04T10:42:34.860 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:42:34.868 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:42:35.147 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:42:35.144+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:42:35.144697+0000) 2024-03-04T10:42:36.147 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:42:36.144+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:42:36.144945+0000) 2024-03-04T10:42:37.147 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:42:37.144+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:42:37.145178+0000) 2024-03-04T10:42:38.148 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:42:38.145+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:42:38.145424+0000) 2024-03-04T10:42:39.148 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:42:39.145+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:42:39.145716+0000) 2024-03-04T10:42:40.149 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:42:40.145+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:42:40.145968+0000) 2024-03-04T10:42:41.148 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:42:41.145+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:42:41.146149+0000) 2024-03-04T10:42:42.148 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:42:42.145+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:42:42.146401+0000) 2024-03-04T10:42:43.149 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:42:43.146+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:42:43.146647+0000) 2024-03-04T10:42:44.149 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:42:44.146+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:42:44.146965+0000) 2024-03-04T10:42:45.149 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:42:45.147+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:42:45.147216+0000) 2024-03-04T10:42:46.149 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:42:46.146+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:42:46.147409+0000) 2024-03-04T10:42:47.150 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:42:47.147+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:42:47.147661+0000) 2024-03-04T10:42:48.150 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:42:48.147+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:42:48.147958+0000) 2024-03-04T10:42:49.150 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:42:49.148+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:42:49.148210+0000) 2024-03-04T10:42:50.151 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:42:50.147+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:42:50.148501+0000) 2024-03-04T10:42:51.151 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:42:51.147+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:42:51.148746+0000) 2024-03-04T10:42:52.151 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:42:52.147+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:42:52.148991+0000) 2024-03-04T10:42:53.151 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:42:53.147+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:42:53.149184+0000) 2024-03-04T10:42:54.152 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:42:54.148+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:42:54.149428+0000) 2024-03-04T10:42:55.152 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:42:55.147+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:42:55.149702+0000) 2024-03-04T10:42:56.152 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:42:56.148+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:42:56.149957+0000) 2024-03-04T10:42:57.152 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:42:57.149+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:42:57.150190+0000) 2024-03-04T10:42:58.153 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:42:58.149+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:42:58.150487+0000) 2024-03-04T10:42:59.153 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:42:59.149+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:42:59.150783+0000) 2024-03-04T10:43:00.153 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:43:00.149+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:43:00.151077+0000) 2024-03-04T10:43:01.154 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:43:01.150+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:43:01.151359+0000) 2024-03-04T10:43:02.154 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:43:02.150+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:43:02.151610+0000) 2024-03-04T10:43:03.154 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:43:03.151+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:43:03.151916+0000) 2024-03-04T10:43:04.154 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:43:04.151+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:43:04.152158+0000) 2024-03-04T10:43:05.114 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:43:05.125 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:43:05.154 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:43:05.151+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:43:05.152358+0000) 2024-03-04T10:43:06.155 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:43:06.151+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:43:06.152611+0000) 2024-03-04T10:43:07.155 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:43:07.152+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:43:07.152896+0000) 2024-03-04T10:43:08.155 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:43:08.152+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:43:08.153142+0000) 2024-03-04T10:43:09.156 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:43:09.152+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:43:09.153402+0000) 2024-03-04T10:43:10.156 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:43:10.152+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:43:10.153654+0000) 2024-03-04T10:43:11.156 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:43:11.153+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:43:11.153943+0000) 2024-03-04T10:43:12.156 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:43:12.153+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:43:12.154192+0000) 2024-03-04T10:43:13.157 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:43:13.153+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:43:13.154436+0000) 2024-03-04T10:43:14.157 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:43:14.153+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:43:14.154684+0000) 2024-03-04T10:43:15.157 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:43:15.154+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:43:15.154957+0000) 2024-03-04T10:43:16.157 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:43:16.154+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:43:16.155247+0000) 2024-03-04T10:43:17.158 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:43:17.154+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:43:17.155474+0000) 2024-03-04T10:43:18.158 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:43:18.155+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:43:18.155715+0000) 2024-03-04T10:43:19.158 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:43:19.155+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:43:19.155959+0000) 2024-03-04T10:43:20.158 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:43:20.155+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:43:20.156215+0000) 2024-03-04T10:43:21.159 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:43:21.155+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:43:21.156455+0000) 2024-03-04T10:43:22.159 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:43:22.156+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:43:22.156735+0000) 2024-03-04T10:43:23.159 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:43:23.156+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:43:23.157007+0000) 2024-03-04T10:43:24.159 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:43:24.156+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:43:24.157243+0000) 2024-03-04T10:43:25.160 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:43:25.156+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:43:25.157496+0000) 2024-03-04T10:43:26.161 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:43:26.157+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:43:26.157745+0000) 2024-03-04T10:43:27.161 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:43:27.157+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:43:27.158077+0000) 2024-03-04T10:43:28.160 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:43:28.157+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:43:28.158262+0000) 2024-03-04T10:43:29.161 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:43:29.158+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:43:29.158494+0000) 2024-03-04T10:43:30.161 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:43:30.158+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:43:30.158745+0000) 2024-03-04T10:43:31.161 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:43:31.158+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:43:31.158979+0000) 2024-03-04T10:43:32.161 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:43:32.158+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:43:32.159231+0000) 2024-03-04T10:43:33.162 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:43:33.159+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:43:33.159474+0000) 2024-03-04T10:43:34.162 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:43:34.159+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:43:34.159726+0000) 2024-03-04T10:43:35.162 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:43:35.159+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:43:35.159975+0000) 2024-03-04T10:43:35.286 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:43:35.294 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:43:36.162 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:43:36.159+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:43:36.160230+0000) 2024-03-04T10:43:37.163 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:43:37.160+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:43:37.160467+0000) 2024-03-04T10:43:38.163 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:43:38.160+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:43:38.160661+0000) 2024-03-04T10:43:39.163 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:43:39.160+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:43:39.160933+0000) 2024-03-04T10:43:40.163 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:43:40.160+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:43:40.161213+0000) 2024-03-04T10:43:41.164 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:43:41.160+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:43:41.161438+0000) 2024-03-04T10:43:42.164 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:43:42.160+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:43:42.161696+0000) 2024-03-04T10:43:43.164 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:43:43.161+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:43:43.161931+0000) 2024-03-04T10:43:44.165 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:43:44.161+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:43:44.162182+0000) 2024-03-04T10:43:45.165 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:43:45.161+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:43:45.162380+0000) 2024-03-04T10:43:46.165 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:43:46.161+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:43:46.162656+0000) 2024-03-04T10:43:47.165 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:43:47.162+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:43:47.162935+0000) 2024-03-04T10:43:48.165 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:43:48.162+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:43:48.163183+0000) 2024-03-04T10:43:49.166 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:43:49.163+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:43:49.163411+0000) 2024-03-04T10:43:50.166 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:43:50.163+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:43:50.163664+0000) 2024-03-04T10:43:51.166 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:43:51.163+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:43:51.163941+0000) 2024-03-04T10:43:52.166 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:43:52.164+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:43:52.164182+0000) 2024-03-04T10:43:53.167 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:43:53.163+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:43:53.164430+0000) 2024-03-04T10:43:54.167 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:43:54.163+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:43:54.164676+0000) 2024-03-04T10:43:55.167 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:43:55.163+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:43:55.164925+0000) 2024-03-04T10:43:56.167 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:43:56.164+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:43:56.165184+0000) 2024-03-04T10:43:57.168 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:43:57.164+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:43:57.165418+0000) 2024-03-04T10:43:58.168 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:43:58.164+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:43:58.165663+0000) 2024-03-04T10:43:59.168 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:43:59.164+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:43:59.165940+0000) 2024-03-04T10:44:00.168 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:44:00.164+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:44:00.166136+0000) 2024-03-04T10:44:01.169 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:44:01.165+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:44:01.166331+0000) 2024-03-04T10:44:02.169 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:44:02.164+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:44:02.166522+0000) 2024-03-04T10:44:03.169 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:44:03.165+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:44:03.166703+0000) 2024-03-04T10:44:04.169 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:44:04.166+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:44:04.166996+0000) 2024-03-04T10:44:05.169 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:44:05.166+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:44:05.167291+0000) 2024-03-04T10:44:05.508 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:44:05.516 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:44:06.170 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:44:06.166+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:44:06.167594+0000) 2024-03-04T10:44:07.170 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:44:07.167+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:44:07.167938+0000) 2024-03-04T10:44:08.170 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:44:08.167+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:44:08.168238+0000) 2024-03-04T10:44:09.171 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:44:09.167+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:44:09.168465+0000) 2024-03-04T10:44:10.171 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:44:10.167+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:44:10.168758+0000) 2024-03-04T10:44:11.171 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:44:11.168+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:44:11.169047+0000) 2024-03-04T10:44:12.172 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:44:12.168+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:44:12.169359+0000) 2024-03-04T10:44:13.172 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:44:13.168+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:44:13.169630+0000) 2024-03-04T10:44:14.172 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:44:14.169+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:44:14.169944+0000) 2024-03-04T10:44:15.172 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:44:15.169+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:44:15.170201+0000) 2024-03-04T10:44:16.173 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:44:16.169+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:44:16.170462+0000) 2024-03-04T10:44:17.173 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:44:17.169+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:44:17.170721+0000) 2024-03-04T10:44:18.173 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:44:18.170+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:44:18.171013+0000) 2024-03-04T10:44:19.174 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:44:19.170+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:44:19.171318+0000) 2024-03-04T10:44:20.174 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:44:20.170+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:44:20.171605+0000) 2024-03-04T10:44:21.174 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:44:21.171+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:44:21.171918+0000) 2024-03-04T10:44:22.174 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:44:22.171+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:44:22.172223+0000) 2024-03-04T10:44:23.175 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:44:23.171+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:44:23.172512+0000) 2024-03-04T10:44:24.175 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:44:24.172+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:44:24.172807+0000) 2024-03-04T10:44:25.175 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:44:25.172+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:44:25.173161+0000) 2024-03-04T10:44:26.176 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:44:26.172+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:44:26.173466+0000) 2024-03-04T10:44:27.176 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:44:27.173+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:44:27.173734+0000) 2024-03-04T10:44:28.176 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:44:28.173+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:44:28.174022+0000) 2024-03-04T10:44:29.176 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:44:29.173+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:44:29.174280+0000) 2024-03-04T10:44:30.177 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:44:30.174+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:44:30.174584+0000) 2024-03-04T10:44:31.177 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:44:31.174+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:44:31.174921+0000) 2024-03-04T10:44:32.177 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:44:32.174+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:44:32.175207+0000) 2024-03-04T10:44:33.178 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:44:33.174+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:44:33.175501+0000) 2024-03-04T10:44:34.178 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:44:34.175+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:44:34.175763+0000) 2024-03-04T10:44:35.178 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:44:35.175+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:44:35.176002+0000) 2024-03-04T10:44:35.712 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:44:35.721 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:44:36.178 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:44:36.175+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:44:36.176266+0000) 2024-03-04T10:44:37.179 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:44:37.175+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:44:37.176443+0000) 2024-03-04T10:44:38.179 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:44:38.176+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:44:38.176696+0000) 2024-03-04T10:44:39.179 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:44:39.176+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:44:39.176930+0000) 2024-03-04T10:44:40.179 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:44:40.176+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:44:40.177174+0000) 2024-03-04T10:44:41.180 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:44:41.177+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:44:41.177418+0000) 2024-03-04T10:44:42.180 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:44:42.176+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:44:42.177617+0000) 2024-03-04T10:44:43.180 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:44:43.177+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:44:43.177923+0000) 2024-03-04T10:44:44.180 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:44:44.177+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:44:44.178166+0000) 2024-03-04T10:44:45.181 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:44:45.178+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:44:45.178409+0000) 2024-03-04T10:44:46.181 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:44:46.178+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:44:46.178670+0000) 2024-03-04T10:44:47.181 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:44:47.178+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:44:47.178946+0000) 2024-03-04T10:44:48.181 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:44:48.178+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:44:48.179189+0000) 2024-03-04T10:44:49.182 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:44:49.179+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:44:49.179419+0000) 2024-03-04T10:44:50.182 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:44:50.179+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:44:50.179609+0000) 2024-03-04T10:44:51.182 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:44:51.179+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:44:51.179896+0000) 2024-03-04T10:44:52.182 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:44:52.179+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:44:52.180140+0000) 2024-03-04T10:44:53.183 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:44:53.180+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:44:53.180376+0000) 2024-03-04T10:44:54.183 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:44:54.180+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:44:54.180617+0000) 2024-03-04T10:44:55.183 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:44:55.180+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:44:55.180905+0000) 2024-03-04T10:44:56.184 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:44:56.180+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:44:56.181161+0000) 2024-03-04T10:44:57.184 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:44:57.180+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:44:57.181395+0000) 2024-03-04T10:44:58.184 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:44:58.180+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:44:58.181639+0000) 2024-03-04T10:44:59.184 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:44:59.180+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:44:59.181923+0000) 2024-03-04T10:45:00.184 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:45:00.181+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:45:00.182173+0000) 2024-03-04T10:45:01.185 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:45:01.180+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:45:01.182433+0000) 2024-03-04T10:45:02.185 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:45:02.181+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:45:02.182675+0000) 2024-03-04T10:45:03.186 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:45:03.181+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:45:03.182947+0000) 2024-03-04T10:45:04.185 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:45:04.182+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:45:04.183195+0000) 2024-03-04T10:45:05.186 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:45:05.182+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:45:05.183440+0000) 2024-03-04T10:45:05.918 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:45:05.928 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:45:06.186 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:45:06.182+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:45:06.183699+0000) 2024-03-04T10:45:07.186 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:45:07.183+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:45:07.183932+0000) 2024-03-04T10:45:08.186 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:45:08.183+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:45:08.184178+0000) 2024-03-04T10:45:09.187 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:45:09.183+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:45:09.184426+0000) 2024-03-04T10:45:10.187 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:45:10.183+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:45:10.184670+0000) 2024-03-04T10:45:11.187 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:45:11.184+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:45:11.184951+0000) 2024-03-04T10:45:12.187 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:45:12.184+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:45:12.185215+0000) 2024-03-04T10:45:13.188 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:45:13.184+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:45:13.185478+0000) 2024-03-04T10:45:14.188 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:45:14.184+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:45:14.185725+0000) 2024-03-04T10:45:15.188 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:45:15.185+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:45:15.185968+0000) 2024-03-04T10:45:16.189 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:45:16.185+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:45:16.186252+0000) 2024-03-04T10:45:17.189 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:45:17.185+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:45:17.186479+0000) 2024-03-04T10:45:18.189 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:45:18.185+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:45:18.186729+0000) 2024-03-04T10:45:19.189 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:45:19.186+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:45:19.186982+0000) 2024-03-04T10:45:20.190 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:45:20.186+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:45:20.187226+0000) 2024-03-04T10:45:21.190 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:45:21.186+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:45:21.187469+0000) 2024-03-04T10:45:22.190 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:45:22.187+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:45:22.187714+0000) 2024-03-04T10:45:23.190 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:45:23.187+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:45:23.187981+0000) 2024-03-04T10:45:24.191 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:45:24.187+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:45:24.188234+0000) 2024-03-04T10:45:25.191 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:45:25.187+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:45:25.188482+0000) 2024-03-04T10:45:26.191 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:45:26.187+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:45:26.188753+0000) 2024-03-04T10:45:27.191 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:45:27.188+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:45:27.188986+0000) 2024-03-04T10:45:28.191 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:45:28.188+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:45:28.189227+0000) 2024-03-04T10:45:29.192 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:45:29.188+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:45:29.189460+0000) 2024-03-04T10:45:30.192 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:45:30.189+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:45:30.189709+0000) 2024-03-04T10:45:31.192 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:45:31.189+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:45:31.189951+0000) 2024-03-04T10:45:32.192 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:45:32.189+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:45:32.190192+0000) 2024-03-04T10:45:33.193 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:45:33.189+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:45:33.190376+0000) 2024-03-04T10:45:34.193 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:45:34.190+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:45:34.190621+0000) 2024-03-04T10:45:35.193 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:45:35.190+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:45:35.190921+0000) 2024-03-04T10:45:36.127 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:45:36.135 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:45:36.193 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:45:36.190+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:45:36.191128+0000) 2024-03-04T10:45:37.194 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:45:37.190+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:45:37.191387+0000) 2024-03-04T10:45:38.194 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:45:38.191+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:45:38.191639+0000) 2024-03-04T10:45:39.194 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:45:39.191+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:45:39.191926+0000) 2024-03-04T10:45:40.194 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:45:40.191+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:45:40.192174+0000) 2024-03-04T10:45:41.195 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:45:41.192+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:45:41.192418+0000) 2024-03-04T10:45:42.195 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:45:42.192+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:45:42.192682+0000) 2024-03-04T10:45:43.195 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:45:43.192+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:45:43.192941+0000) 2024-03-04T10:45:44.196 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:45:44.192+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:45:44.193191+0000) 2024-03-04T10:45:45.196 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:45:45.193+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:45:45.193443+0000) 2024-03-04T10:45:46.196 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:45:46.193+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:45:46.193702+0000) 2024-03-04T10:45:47.196 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:45:47.193+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:45:47.193919+0000) 2024-03-04T10:45:48.197 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:45:48.193+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:45:48.194180+0000) 2024-03-04T10:45:49.197 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:45:49.194+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:45:49.194421+0000) 2024-03-04T10:45:50.197 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:45:50.194+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:45:50.194693+0000) 2024-03-04T10:45:51.201 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:45:51.194+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:45:51.194952+0000) 2024-03-04T10:45:52.198 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:45:52.194+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:45:52.195216+0000) 2024-03-04T10:45:53.198 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:45:53.195+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:45:53.195471+0000) 2024-03-04T10:45:54.198 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:45:54.195+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:45:54.195764+0000) 2024-03-04T10:45:55.198 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:45:55.195+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:45:55.196022+0000) 2024-03-04T10:45:56.198 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:45:56.195+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:45:56.196252+0000) 2024-03-04T10:45:57.199 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:45:57.196+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:45:57.196517+0000) 2024-03-04T10:45:58.199 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:45:58.195+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:45:58.196780+0000) 2024-03-04T10:45:59.199 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:45:59.195+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:45:59.197068+0000) 2024-03-04T10:46:00.200 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:46:00.196+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:46:00.197332+0000) 2024-03-04T10:46:01.200 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:46:01.196+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:46:01.197598+0000) 2024-03-04T10:46:02.200 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:46:02.196+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:46:02.197928+0000) 2024-03-04T10:46:03.200 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:46:03.197+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:46:03.198188+0000) 2024-03-04T10:46:04.201 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:46:04.197+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:46:04.198454+0000) 2024-03-04T10:46:05.201 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:46:05.197+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:46:05.198672+0000) 2024-03-04T10:46:06.201 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:46:06.197+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:46:06.198970+0000) 2024-03-04T10:46:06.328 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:46:06.336 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:46:07.202 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:46:07.198+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:46:07.199230+0000) 2024-03-04T10:46:08.202 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:46:08.198+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:46:08.199493+0000) 2024-03-04T10:46:09.202 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:46:09.198+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:46:09.199740+0000) 2024-03-04T10:46:10.202 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:46:10.199+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:46:10.200013+0000) 2024-03-04T10:46:11.203 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:46:11.199+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:46:11.200293+0000) 2024-03-04T10:46:12.203 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:46:12.199+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:46:12.200575+0000) 2024-03-04T10:46:13.205 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:46:13.200+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:46:13.200867+0000) 2024-03-04T10:46:14.204 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:46:14.200+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:46:14.201166+0000) 2024-03-04T10:46:15.204 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:46:15.200+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:46:15.201441+0000) 2024-03-04T10:46:16.204 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:46:16.200+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:46:16.201714+0000) 2024-03-04T10:46:17.204 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:46:17.201+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:46:17.202010+0000) 2024-03-04T10:46:18.205 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:46:18.201+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:46:18.202274+0000) 2024-03-04T10:46:19.205 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:46:19.201+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:46:19.202518+0000) 2024-03-04T10:46:20.205 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:46:20.202+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:46:20.202796+0000) 2024-03-04T10:46:21.205 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:46:21.202+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:46:21.203094+0000) 2024-03-04T10:46:22.206 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:46:22.202+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:46:22.203354+0000) 2024-03-04T10:46:23.206 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:46:23.202+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:46:23.203609+0000) 2024-03-04T10:46:24.206 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:46:24.203+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:46:24.203957+0000) 2024-03-04T10:46:25.207 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:46:25.203+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:46:25.204224+0000) 2024-03-04T10:46:26.207 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:46:26.203+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:46:26.204507+0000) 2024-03-04T10:46:27.207 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:46:27.204+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:46:27.204753+0000) 2024-03-04T10:46:28.208 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:46:28.204+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:46:28.205017+0000) 2024-03-04T10:46:29.208 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:46:29.204+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:46:29.205259+0000) 2024-03-04T10:46:30.208 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:46:30.204+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:46:30.205565+0000) 2024-03-04T10:46:31.208 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:46:31.205+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:46:31.205861+0000) 2024-03-04T10:46:32.209 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:46:32.205+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:46:32.206161+0000) 2024-03-04T10:46:33.209 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:46:33.205+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:46:33.206417+0000) 2024-03-04T10:46:34.209 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:46:34.206+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:46:34.206677+0000) 2024-03-04T10:46:35.209 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:46:35.206+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:46:35.206945+0000) 2024-03-04T10:46:36.210 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:46:36.206+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:46:36.207217+0000) 2024-03-04T10:46:36.599 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:46:36.607 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:46:37.210 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:46:37.206+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:46:37.207465+0000) 2024-03-04T10:46:38.210 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:46:38.207+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:46:38.207710+0000) 2024-03-04T10:46:39.210 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:46:39.207+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:46:39.207966+0000) 2024-03-04T10:46:40.211 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:46:40.207+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:46:40.208241+0000) 2024-03-04T10:46:41.211 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:46:41.208+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:46:41.208485+0000) 2024-03-04T10:46:42.211 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:46:42.207+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:46:42.208760+0000) 2024-03-04T10:46:43.212 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:46:43.208+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:46:43.208997+0000) 2024-03-04T10:46:44.212 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:46:44.208+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:46:44.209238+0000) 2024-03-04T10:46:45.212 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:46:45.209+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:46:45.209489+0000) 2024-03-04T10:46:46.212 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:46:46.209+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:46:46.209746+0000) 2024-03-04T10:46:47.212 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:46:47.209+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:46:47.209977+0000) 2024-03-04T10:46:48.213 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:46:48.209+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:46:48.210226+0000) 2024-03-04T10:46:49.213 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:46:49.210+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:46:49.210463+0000) 2024-03-04T10:46:50.213 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:46:50.210+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:46:50.210666+0000) 2024-03-04T10:46:51.213 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:46:51.210+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:46:51.210953+0000) 2024-03-04T10:46:52.213 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:46:52.210+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:46:52.211204+0000) 2024-03-04T10:46:53.214 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:46:53.210+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:46:53.211458+0000) 2024-03-04T10:46:54.215 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:46:54.211+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:46:54.211707+0000) 2024-03-04T10:46:55.214 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:46:55.211+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:46:55.211961+0000) 2024-03-04T10:46:56.215 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:46:56.212+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:46:56.212219+0000) 2024-03-04T10:46:57.215 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:46:57.211+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:46:57.212395+0000) 2024-03-04T10:46:58.215 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:46:58.212+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:46:58.212646+0000) 2024-03-04T10:46:59.215 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:46:59.212+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:46:59.212916+0000) 2024-03-04T10:47:00.216 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:47:00.213+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:47:00.213187+0000) 2024-03-04T10:47:01.216 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:47:01.212+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:47:01.213430+0000) 2024-03-04T10:47:02.216 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:47:02.212+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:47:02.213672+0000) 2024-03-04T10:47:03.216 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:47:03.212+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:47:03.213948+0000) 2024-03-04T10:47:04.216 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:47:04.213+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:47:04.214191+0000) 2024-03-04T10:47:05.217 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:47:05.213+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:47:05.214516+0000) 2024-03-04T10:47:06.217 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:47:06.213+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:47:06.214792+0000) 2024-03-04T10:47:06.812 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:47:06.820 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:47:07.217 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:47:07.214+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:47:07.215080+0000) 2024-03-04T10:47:08.218 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:47:08.214+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:47:08.215327+0000) 2024-03-04T10:47:09.218 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:47:09.213+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:47:09.215571+0000) 2024-03-04T10:47:10.218 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:47:10.214+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:47:10.215882+0000) 2024-03-04T10:47:11.219 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:47:11.215+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:47:11.216133+0000) 2024-03-04T10:47:12.219 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:47:12.215+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:47:12.216330+0000) 2024-03-04T10:47:13.219 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:47:13.215+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:47:13.216569+0000) 2024-03-04T10:47:14.219 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:47:14.216+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:47:14.216816+0000) 2024-03-04T10:47:15.219 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:47:15.216+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:47:15.217082+0000) 2024-03-04T10:47:16.220 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:47:16.216+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:47:16.217344+0000) 2024-03-04T10:47:17.220 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:47:17.216+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:47:17.217581+0000) 2024-03-04T10:47:18.220 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:47:18.217+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:47:18.217853+0000) 2024-03-04T10:47:19.221 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:47:19.217+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:47:19.218118+0000) 2024-03-04T10:47:20.221 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:47:20.217+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:47:20.218365+0000) 2024-03-04T10:47:21.221 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:47:21.217+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:47:21.218641+0000) 2024-03-04T10:47:22.221 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:47:22.218+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:47:22.218942+0000) 2024-03-04T10:47:23.222 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:47:23.218+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:47:23.219170+0000) 2024-03-04T10:47:24.222 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:47:24.218+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:47:24.219422+0000) 2024-03-04T10:47:25.222 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:47:25.218+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:47:25.219652+0000) 2024-03-04T10:47:26.222 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:47:26.219+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:47:26.219949+0000) 2024-03-04T10:47:27.223 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:47:27.219+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:47:27.220197+0000) 2024-03-04T10:47:28.223 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:47:28.219+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:47:28.220443+0000) 2024-03-04T10:47:29.223 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:47:29.220+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:47:29.220677+0000) 2024-03-04T10:47:30.223 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:47:30.220+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:47:30.220949+0000) 2024-03-04T10:47:31.224 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:47:31.220+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:47:31.221195+0000) 2024-03-04T10:47:32.224 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:47:32.220+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:47:32.221444+0000) 2024-03-04T10:47:33.224 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:47:33.221+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:47:33.221677+0000) 2024-03-04T10:47:34.224 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:47:34.221+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:47:34.221965+0000) 2024-03-04T10:47:35.224 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:47:35.221+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:47:35.222195+0000) 2024-03-04T10:47:36.225 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:47:36.221+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:47:36.222455+0000) 2024-03-04T10:47:36.996 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:47:37.004 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:47:37.225 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:47:37.221+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:47:37.222705+0000) 2024-03-04T10:47:38.225 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:47:38.222+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:47:38.222976+0000) 2024-03-04T10:47:39.226 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:47:39.222+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:47:39.223236+0000) 2024-03-04T10:47:40.226 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:47:40.222+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:47:40.223476+0000) 2024-03-04T10:47:41.226 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:47:41.222+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:47:41.223721+0000) 2024-03-04T10:47:42.226 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:47:42.223+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:47:42.223990+0000) 2024-03-04T10:47:43.227 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:47:43.223+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:47:43.224230+0000) 2024-03-04T10:47:44.227 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:47:44.224+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:47:44.224478+0000) 2024-03-04T10:47:45.227 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:47:45.224+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:47:45.224712+0000) 2024-03-04T10:47:46.227 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:47:46.224+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:47:46.224995+0000) 2024-03-04T10:47:47.228 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:47:47.224+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:47:47.225230+0000) 2024-03-04T10:47:48.228 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:47:48.225+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:47:48.225481+0000) 2024-03-04T10:47:49.228 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:47:49.224+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:47:49.225705+0000) 2024-03-04T10:47:50.228 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:47:50.225+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:47:50.225946+0000) 2024-03-04T10:47:51.229 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:47:51.225+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:47:51.226194+0000) 2024-03-04T10:47:52.229 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:47:52.226+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:47:52.226440+0000) 2024-03-04T10:47:53.229 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:47:53.226+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:47:53.226684+0000) 2024-03-04T10:47:54.229 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:47:54.226+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:47:54.226954+0000) 2024-03-04T10:47:55.230 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:47:55.226+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:47:55.227190+0000) 2024-03-04T10:47:56.230 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:47:56.227+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:47:56.227451+0000) 2024-03-04T10:47:57.230 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:47:57.227+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:47:57.227681+0000) 2024-03-04T10:47:58.230 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:47:58.227+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:47:58.227933+0000) 2024-03-04T10:47:59.231 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:47:59.228+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:47:59.228230+0000) 2024-03-04T10:48:00.231 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:48:00.228+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:48:00.228478+0000) 2024-03-04T10:48:01.231 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:48:01.228+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:48:01.228736+0000) 2024-03-04T10:48:02.231 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:48:02.228+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:48:02.228980+0000) 2024-03-04T10:48:03.231 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:48:03.229+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:48:03.229217+0000) 2024-03-04T10:48:04.232 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:48:04.228+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:48:04.229465+0000) 2024-03-04T10:48:05.232 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:48:05.228+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:48:05.229702+0000) 2024-03-04T10:48:06.232 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:48:06.228+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:48:06.229964+0000) 2024-03-04T10:48:07.198 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:48:07.206 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:48:07.232 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:48:07.228+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:48:07.230166+0000) 2024-03-04T10:48:08.233 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:48:08.228+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:48:08.230369+0000) 2024-03-04T10:48:09.233 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:48:09.229+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:48:09.230623+0000) 2024-03-04T10:48:10.233 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:48:10.229+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:48:10.230919+0000) 2024-03-04T10:48:11.233 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:48:11.230+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:48:11.231173+0000) 2024-03-04T10:48:12.234 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:48:12.230+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:48:12.231381+0000) 2024-03-04T10:48:13.234 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:48:13.230+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:48:13.231614+0000) 2024-03-04T10:48:14.234 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:48:14.230+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:48:14.231890+0000) 2024-03-04T10:48:15.234 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:48:15.231+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:48:15.232130+0000) 2024-03-04T10:48:16.234 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:48:16.231+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:48:16.232331+0000) 2024-03-04T10:48:17.235 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:48:17.231+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:48:17.232571+0000) 2024-03-04T10:48:18.235 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:48:18.232+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:48:18.232853+0000) 2024-03-04T10:48:19.235 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:48:19.232+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:48:19.233088+0000) 2024-03-04T10:48:20.235 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:48:20.232+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:48:20.233332+0000) 2024-03-04T10:48:21.235 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:48:21.232+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:48:21.233532+0000) 2024-03-04T10:48:22.236 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:48:22.232+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:48:22.233771+0000) 2024-03-04T10:48:23.236 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:48:23.233+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:48:23.233976+0000) 2024-03-04T10:48:24.236 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:48:24.233+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:48:24.234213+0000) 2024-03-04T10:48:25.236 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:48:25.233+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:48:25.234485+0000) 2024-03-04T10:48:26.237 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:48:26.233+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:48:26.234758+0000) 2024-03-04T10:48:27.237 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:48:27.234+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:48:27.235015+0000) 2024-03-04T10:48:28.237 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:48:28.234+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:48:28.235309+0000) 2024-03-04T10:48:29.238 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:48:29.234+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:48:29.235584+0000) 2024-03-04T10:48:30.238 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:48:30.235+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:48:30.235901+0000) 2024-03-04T10:48:31.238 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:48:31.235+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:48:31.236205+0000) 2024-03-04T10:48:32.238 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:48:32.235+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:48:32.236469+0000) 2024-03-04T10:48:33.239 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:48:33.236+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:48:33.236729+0000) 2024-03-04T10:48:34.239 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:48:34.236+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:48:34.236994+0000) 2024-03-04T10:48:35.239 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:48:35.236+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:48:35.237251+0000) 2024-03-04T10:48:36.239 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:48:36.236+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:48:36.237530+0000) 2024-03-04T10:48:37.240 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:48:37.237+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:48:37.237789+0000) 2024-03-04T10:48:37.334 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:48:37.342 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:48:38.240 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:48:38.237+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:48:38.238093+0000) 2024-03-04T10:48:39.240 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:48:39.237+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:48:39.238362+0000) 2024-03-04T10:48:40.240 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:48:40.238+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:48:40.238608+0000) 2024-03-04T10:48:41.240 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:48:41.238+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:48:41.238922+0000) 2024-03-04T10:48:42.241 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:48:42.238+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:48:42.239188+0000) 2024-03-04T10:48:43.241 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:48:43.238+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:48:43.239440+0000) 2024-03-04T10:48:44.241 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:48:44.239+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:48:44.239684+0000) 2024-03-04T10:48:45.242 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:48:45.239+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:48:45.239959+0000) 2024-03-04T10:48:46.242 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:48:46.239+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:48:46.240215+0000) 2024-03-04T10:48:47.242 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:48:47.240+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:48:47.240536+0000) 2024-03-04T10:48:48.242 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:48:48.240+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:48:48.240803+0000) 2024-03-04T10:48:49.243 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:48:49.240+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:48:49.241117+0000) 2024-03-04T10:48:50.243 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:48:50.240+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:48:50.241382+0000) 2024-03-04T10:48:51.243 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:48:51.241+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:48:51.241640+0000) 2024-03-04T10:48:52.243 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:48:52.241+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:48:52.241955+0000) 2024-03-04T10:48:53.244 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:48:53.241+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:48:53.242218+0000) 2024-03-04T10:48:54.244 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:48:54.242+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:48:54.242473+0000) 2024-03-04T10:48:55.244 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:48:55.242+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:48:55.242729+0000) 2024-03-04T10:48:56.244 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:48:56.242+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:48:56.242999+0000) 2024-03-04T10:48:57.245 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:48:57.242+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:48:57.243264+0000) 2024-03-04T10:48:58.245 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:48:58.243+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:48:58.243532+0000) 2024-03-04T10:48:59.245 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:48:59.243+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:48:59.243787+0000) 2024-03-04T10:49:00.245 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:49:00.243+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:49:00.244083+0000) 2024-03-04T10:49:01.246 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:49:01.244+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:49:01.244342+0000) 2024-03-04T10:49:02.246 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:49:02.244+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:49:02.244601+0000) 2024-03-04T10:49:03.246 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:49:03.244+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:49:03.244922+0000) 2024-03-04T10:49:04.247 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:49:04.245+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:49:04.245182+0000) 2024-03-04T10:49:05.247 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:49:05.245+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:49:05.245444+0000) 2024-03-04T10:49:06.247 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:49:06.245+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:49:06.245734+0000) 2024-03-04T10:49:07.247 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:49:07.244+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:49:07.246022+0000) 2024-03-04T10:49:07.547 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:49:07.556 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:49:08.247 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:49:08.245+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:49:08.246280+0000) 2024-03-04T10:49:09.248 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:49:09.244+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:49:09.246560+0000) 2024-03-04T10:49:10.248 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:49:10.244+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:49:10.246753+0000) 2024-03-04T10:49:11.248 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:49:11.245+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:49:11.247012+0000) 2024-03-04T10:49:12.248 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:49:12.246+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:49:12.247274+0000) 2024-03-04T10:49:13.249 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:49:13.245+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:49:13.247558+0000) 2024-03-04T10:49:14.249 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:49:14.246+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:49:14.247862+0000) 2024-03-04T10:49:15.249 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:49:15.247+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:49:15.248126+0000) 2024-03-04T10:49:16.250 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:49:16.247+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:49:16.248396+0000) 2024-03-04T10:49:17.250 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:49:17.247+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:49:17.248657+0000) 2024-03-04T10:49:18.250 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:49:18.248+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:49:18.248951+0000) 2024-03-04T10:49:19.250 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:49:19.248+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:49:19.249211+0000) 2024-03-04T10:49:20.251 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:49:20.248+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:49:20.249467+0000) 2024-03-04T10:49:21.251 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:49:21.248+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:49:21.249726+0000) 2024-03-04T10:49:22.252 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:49:22.249+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:49:22.249986+0000) 2024-03-04T10:49:23.251 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:49:23.249+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:49:23.250248+0000) 2024-03-04T10:49:24.252 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:49:24.249+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:49:24.250459+0000) 2024-03-04T10:49:25.252 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:49:25.249+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:49:25.250728+0000) 2024-03-04T10:49:26.252 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:49:26.250+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:49:26.251003+0000) 2024-03-04T10:49:27.252 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:49:27.250+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:49:27.251263+0000) 2024-03-04T10:49:28.253 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:49:28.250+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:49:28.251522+0000) 2024-03-04T10:49:29.253 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:49:29.251+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:49:29.251782+0000) 2024-03-04T10:49:30.253 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:49:30.251+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:49:30.252071+0000) 2024-03-04T10:49:31.253 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:49:31.251+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:49:31.252331+0000) 2024-03-04T10:49:32.253 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:49:32.251+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:49:32.252593+0000) 2024-03-04T10:49:33.254 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:49:33.252+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:49:33.252896+0000) 2024-03-04T10:49:34.254 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:49:34.252+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:49:34.253154+0000) 2024-03-04T10:49:35.254 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:49:35.252+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:49:35.253447+0000) 2024-03-04T10:49:36.255 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:49:36.253+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:49:36.253719+0000) 2024-03-04T10:49:37.255 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:49:37.253+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:49:37.253967+0000) 2024-03-04T10:49:37.735 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:49:37.743 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:49:38.255 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:49:38.253+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:49:38.254237+0000) 2024-03-04T10:49:39.255 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:49:39.253+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:49:39.254500+0000) 2024-03-04T10:49:40.256 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:49:40.254+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:49:40.254757+0000) 2024-03-04T10:49:41.256 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:49:41.254+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:49:41.255016+0000) 2024-03-04T10:49:42.256 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:49:42.254+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:49:42.255271+0000) 2024-03-04T10:49:43.256 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:49:43.254+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:49:43.255531+0000) 2024-03-04T10:49:44.257 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:49:44.255+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:49:44.255785+0000) 2024-03-04T10:49:45.257 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:49:45.255+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:49:45.256091+0000) 2024-03-04T10:49:46.257 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:49:46.255+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:49:46.256360+0000) 2024-03-04T10:49:47.257 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:49:47.255+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:49:47.256627+0000) 2024-03-04T10:49:48.258 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:49:48.256+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:49:48.256937+0000) 2024-03-04T10:49:49.258 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:49:49.256+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:49:49.257211+0000) 2024-03-04T10:49:50.258 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:49:50.257+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:49:50.257470+0000) 2024-03-04T10:49:51.259 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:49:51.256+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:49:51.257751+0000) 2024-03-04T10:49:52.259 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:49:52.257+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:49:52.258000+0000) 2024-03-04T10:49:53.259 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:49:53.257+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:49:53.258259+0000) 2024-03-04T10:49:54.259 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:49:54.258+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:49:54.258519+0000) 2024-03-04T10:49:55.259 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:49:55.258+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:49:55.258781+0000) 2024-03-04T10:49:56.260 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:49:56.258+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:49:56.259117+0000) 2024-03-04T10:49:57.260 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:49:57.259+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:49:57.259379+0000) 2024-03-04T10:49:58.260 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:49:58.259+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:49:58.259636+0000) 2024-03-04T10:49:59.260 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:49:59.259+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:49:59.259949+0000) 2024-03-04T10:50:00.261 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:50:00.259+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:50:00.260211+0000) 2024-03-04T10:50:01.261 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:50:01.259+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:50:01.260474+0000) 2024-03-04T10:50:02.261 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:50:02.260+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:50:02.260735+0000) 2024-03-04T10:50:03.262 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:50:03.260+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:50:03.261059+0000) 2024-03-04T10:50:04.262 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:50:04.261+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:50:04.261323+0000) 2024-03-04T10:50:05.262 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:50:05.261+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:50:05.261575+0000) 2024-03-04T10:50:06.262 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:50:06.261+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:50:06.261884+0000) 2024-03-04T10:50:07.263 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:50:07.262+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:50:07.262160+0000) 2024-03-04T10:50:07.943 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:50:07.951 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:50:08.263 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:50:08.262+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:50:08.262461+0000) 2024-03-04T10:50:09.263 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:50:09.262+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:50:09.262726+0000) 2024-03-04T10:50:10.263 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:50:10.261+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:50:10.262995+0000) 2024-03-04T10:50:11.264 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:50:11.261+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:50:11.263208+0000) 2024-03-04T10:50:12.264 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:50:12.261+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:50:12.263475+0000) 2024-03-04T10:50:13.264 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:50:13.262+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:50:13.263743+0000) 2024-03-04T10:50:14.264 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:50:14.262+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:50:14.263992+0000) 2024-03-04T10:50:15.265 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:50:15.262+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:50:15.264247+0000) 2024-03-04T10:50:16.265 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:50:16.262+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:50:16.264515+0000) 2024-03-04T10:50:17.265 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:50:17.262+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:50:17.264719+0000) 2024-03-04T10:50:18.265 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:50:18.263+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:50:18.264981+0000) 2024-03-04T10:50:19.266 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:50:19.263+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:50:19.265216+0000) 2024-03-04T10:50:20.266 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:50:20.263+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:50:20.265435+0000) 2024-03-04T10:50:21.266 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:50:21.264+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:50:21.265734+0000) 2024-03-04T10:50:22.266 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:50:22.265+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:50:22.266029+0000) 2024-03-04T10:50:23.267 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:50:23.265+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:50:23.266330+0000) 2024-03-04T10:50:24.267 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:50:24.265+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:50:24.266620+0000) 2024-03-04T10:50:25.267 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:50:25.266+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:50:25.266947+0000) 2024-03-04T10:50:26.268 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:50:26.266+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:50:26.267263+0000) 2024-03-04T10:50:27.268 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:50:27.266+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:50:27.267559+0000) 2024-03-04T10:50:28.268 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:50:28.267+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:50:28.267883+0000) 2024-03-04T10:50:29.269 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:50:29.267+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:50:29.268182+0000) 2024-03-04T10:50:30.269 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:50:30.267+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:50:30.268467+0000) 2024-03-04T10:50:31.269 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:50:31.267+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:50:31.268762+0000) 2024-03-04T10:50:32.269 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:50:32.268+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:50:32.269051+0000) 2024-03-04T10:50:33.270 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:50:33.268+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:50:33.269310+0000) 2024-03-04T10:50:34.270 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:50:34.268+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:50:34.269494+0000) 2024-03-04T10:50:35.270 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:50:35.268+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:50:35.269751+0000) 2024-03-04T10:50:36.270 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:50:36.269+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:50:36.270020+0000) 2024-03-04T10:50:37.270 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:50:37.269+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:50:37.270276+0000) 2024-03-04T10:50:38.095 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:50:38.107 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:50:38.271 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:50:38.269+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:50:38.270558+0000) 2024-03-04T10:50:39.271 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:50:39.270+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:50:39.270929+0000) 2024-03-04T10:50:40.271 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:50:40.270+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:50:40.271189+0000) 2024-03-04T10:50:41.272 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:50:41.270+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:50:41.271452+0000) 2024-03-04T10:50:42.272 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:50:42.271+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:50:42.271709+0000) 2024-03-04T10:50:43.272 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:50:43.271+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:50:43.271967+0000) 2024-03-04T10:50:44.272 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:50:44.271+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:50:44.272219+0000) 2024-03-04T10:50:45.272 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:50:45.271+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:50:45.272475+0000) 2024-03-04T10:50:46.273 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:50:46.272+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:50:46.272748+0000) 2024-03-04T10:50:47.273 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:50:47.272+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:50:47.273025+0000) 2024-03-04T10:50:48.273 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:50:48.272+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:50:48.273274+0000) 2024-03-04T10:50:49.274 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:50:49.273+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:50:49.273538+0000) 2024-03-04T10:50:50.274 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:50:50.273+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:50:50.273805+0000) 2024-03-04T10:50:51.274 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:50:51.273+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:50:51.274124+0000) 2024-03-04T10:50:52.275 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:50:52.273+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:50:52.274382+0000) 2024-03-04T10:50:53.275 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:50:53.274+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:50:53.274634+0000) 2024-03-04T10:50:54.275 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:50:54.274+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:50:54.274934+0000) 2024-03-04T10:50:55.275 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:50:55.274+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:50:55.275191+0000) 2024-03-04T10:50:56.275 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:50:56.275+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:50:56.275470+0000) 2024-03-04T10:50:57.276 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:50:57.275+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:50:57.275724+0000) 2024-03-04T10:50:58.277 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:50:58.275+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:50:58.275999+0000) 2024-03-04T10:50:59.276 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:50:59.275+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:50:59.276274+0000) 2024-03-04T10:51:00.276 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:51:00.276+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:51:00.276531+0000) 2024-03-04T10:51:01.277 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:51:01.276+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:51:01.276864+0000) 2024-03-04T10:51:02.277 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:51:02.276+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:51:02.277124+0000) 2024-03-04T10:51:03.277 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:51:03.277+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:51:03.277378+0000) 2024-03-04T10:51:04.277 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:51:04.276+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:51:04.277648+0000) 2024-03-04T10:51:05.278 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:51:05.277+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:51:05.277945+0000) 2024-03-04T10:51:06.278 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:51:06.277+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:51:06.278222+0000) 2024-03-04T10:51:07.278 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:51:07.278+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:51:07.278458+0000) 2024-03-04T10:51:08.278 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:51:08.278+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:51:08.278698+0000) 2024-03-04T10:51:08.308 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:51:08.316 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:51:09.279 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:51:09.278+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:51:09.278943+0000) 2024-03-04T10:51:10.279 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:51:10.279+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:51:10.279175+0000) 2024-03-04T10:51:11.279 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:51:11.278+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:51:11.279379+0000) 2024-03-04T10:51:12.280 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:51:12.279+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:51:12.279643+0000) 2024-03-04T10:51:13.280 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:51:13.278+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:51:13.279926+0000) 2024-03-04T10:51:14.280 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:51:14.279+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:51:14.280165+0000) 2024-03-04T10:51:15.280 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:51:15.279+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:51:15.280401+0000) 2024-03-04T10:51:16.280 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:51:16.279+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:51:16.280605+0000) 2024-03-04T10:51:17.281 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:51:17.279+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:51:17.280877+0000) 2024-03-04T10:51:18.281 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:51:18.280+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:51:18.281131+0000) 2024-03-04T10:51:19.281 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:51:19.280+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:51:19.281367+0000) 2024-03-04T10:51:20.281 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:51:20.279+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:51:20.281622+0000) 2024-03-04T10:51:21.282 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:51:21.280+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:51:21.281952+0000) 2024-03-04T10:51:22.282 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:51:22.281+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:51:22.282186+0000) 2024-03-04T10:51:23.282 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:51:23.281+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:51:23.282430+0000) 2024-03-04T10:51:24.282 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:51:24.281+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:51:24.282670+0000) 2024-03-04T10:51:25.283 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:51:25.282+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:51:25.282948+0000) 2024-03-04T10:51:26.283 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:51:26.282+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:51:26.283204+0000) 2024-03-04T10:51:27.283 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:51:27.282+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:51:27.283444+0000) 2024-03-04T10:51:28.283 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:51:28.282+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:51:28.283706+0000) 2024-03-04T10:51:29.283 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:51:29.283+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:51:29.283978+0000) 2024-03-04T10:51:30.284 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:51:30.283+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:51:30.284175+0000) 2024-03-04T10:51:31.284 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:51:31.283+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:51:31.284430+0000) 2024-03-04T10:51:32.284 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:51:32.283+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:51:32.284673+0000) 2024-03-04T10:51:33.284 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:51:33.284+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:51:33.284954+0000) 2024-03-04T10:51:34.285 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:51:34.284+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:51:34.285190+0000) 2024-03-04T10:51:35.285 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:51:35.284+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:51:35.285427+0000) 2024-03-04T10:51:36.285 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:51:36.284+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:51:36.285673+0000) 2024-03-04T10:51:37.285 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:51:37.285+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:51:37.286003+0000) 2024-03-04T10:51:38.286 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:51:38.285+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:51:38.286250+0000) 2024-03-04T10:51:38.513 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:51:38.521 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:51:39.286 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:51:39.285+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:51:39.286492+0000) 2024-03-04T10:51:40.286 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:51:40.286+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:51:40.286735+0000) 2024-03-04T10:51:41.286 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:51:41.286+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:51:41.286983+0000) 2024-03-04T10:51:42.287 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:51:42.286+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:51:42.287220+0000) 2024-03-04T10:51:43.287 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:51:43.286+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:51:43.287458+0000) 2024-03-04T10:51:44.287 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:51:44.287+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:51:44.287698+0000) 2024-03-04T10:51:45.287 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:51:45.287+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:51:45.287946+0000) 2024-03-04T10:51:46.288 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:51:46.287+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:51:46.288203+0000) 2024-03-04T10:51:47.289 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:51:47.287+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:51:47.288440+0000) 2024-03-04T10:51:48.288 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:51:48.288+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:51:48.288693+0000) 2024-03-04T10:51:49.288 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:51:49.288+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:51:49.288929+0000) 2024-03-04T10:51:50.290 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:51:50.288+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:51:50.289168+0000) 2024-03-04T10:51:51.289 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:51:51.288+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:51:51.289432+0000) 2024-03-04T10:51:52.289 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:51:52.289+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:51:52.289682+0000) 2024-03-04T10:51:53.289 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:51:53.289+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:51:53.289947+0000) 2024-03-04T10:51:54.290 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:51:54.289+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:51:54.290204+0000) 2024-03-04T10:51:55.290 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:51:55.290+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:51:55.290463+0000) 2024-03-04T10:51:56.290 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:51:56.290+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:51:56.290721+0000) 2024-03-04T10:51:57.290 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:51:57.290+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:51:57.290956+0000) 2024-03-04T10:51:58.290 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:51:58.290+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:51:58.291148+0000) 2024-03-04T10:51:59.291 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:51:59.291+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:51:59.291409+0000) 2024-03-04T10:52:00.291 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:52:00.291+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:52:00.291659+0000) 2024-03-04T10:52:01.291 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:52:01.291+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:52:01.291952+0000) 2024-03-04T10:52:02.292 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:52:02.291+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:52:02.292186+0000) 2024-03-04T10:52:03.292 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:52:03.292+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:52:03.292424+0000) 2024-03-04T10:52:04.292 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:52:04.292+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:52:04.292667+0000) 2024-03-04T10:52:05.292 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:52:05.292+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:52:05.292935+0000) 2024-03-04T10:52:06.292 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:52:06.292+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:52:06.293188+0000) 2024-03-04T10:52:07.293 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:52:07.293+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:52:07.293431+0000) 2024-03-04T10:52:08.293 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:52:08.293+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:52:08.293670+0000) 2024-03-04T10:52:08.717 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:52:08.725 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:52:09.293 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:52:09.293+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:52:09.293945+0000) 2024-03-04T10:52:10.293 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:52:10.294+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:52:10.294233+0000) 2024-03-04T10:52:11.294 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:52:11.294+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:52:11.294487+0000) 2024-03-04T10:52:12.294 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:52:12.294+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:52:12.294689+0000) 2024-03-04T10:52:13.294 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:52:13.294+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:52:13.294927+0000) 2024-03-04T10:52:14.294 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:52:14.295+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:52:14.295167+0000) 2024-03-04T10:52:15.295 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:52:15.294+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:52:15.295407+0000) 2024-03-04T10:52:16.295 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:52:16.294+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:52:16.295650+0000) 2024-03-04T10:52:17.295 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:52:17.294+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:52:17.295933+0000) 2024-03-04T10:52:18.296 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:52:18.294+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:52:18.296191+0000) 2024-03-04T10:52:19.295 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:52:19.295+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:52:19.296447+0000) 2024-03-04T10:52:20.296 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:52:20.295+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:52:20.296682+0000) 2024-03-04T10:52:21.296 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:52:21.295+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:52:21.296938+0000) 2024-03-04T10:52:22.296 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:52:22.296+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:52:22.297173+0000) 2024-03-04T10:52:23.296 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:52:23.296+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:52:23.297422+0000) 2024-03-04T10:52:24.297 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:52:24.296+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:52:24.297663+0000) 2024-03-04T10:52:25.297 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:52:25.296+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:52:25.297985+0000) 2024-03-04T10:52:26.297 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:52:26.297+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:52:26.298274+0000) 2024-03-04T10:52:27.297 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:52:27.297+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:52:27.298522+0000) 2024-03-04T10:52:28.298 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:52:28.297+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:52:28.298717+0000) 2024-03-04T10:52:29.298 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:52:29.298+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:52:29.298973+0000) 2024-03-04T10:52:30.298 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:52:30.298+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:52:30.299209+0000) 2024-03-04T10:52:31.298 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:52:31.298+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:52:31.299449+0000) 2024-03-04T10:52:32.299 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:52:32.298+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:52:32.299684+0000) 2024-03-04T10:52:33.299 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:52:33.299+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:52:33.299930+0000) 2024-03-04T10:52:34.299 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:52:34.299+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:52:34.300168+0000) 2024-03-04T10:52:35.299 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:52:35.299+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:52:35.300406+0000) 2024-03-04T10:52:36.300 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:52:36.299+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:52:36.300662+0000) 2024-03-04T10:52:37.300 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:52:37.300+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:52:37.300929+0000) 2024-03-04T10:52:38.300 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:52:38.300+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:52:38.301184+0000) 2024-03-04T10:52:38.924 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:52:38.936 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:52:39.300 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:52:39.300+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:52:39.301430+0000) 2024-03-04T10:52:40.301 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:52:40.300+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:52:40.301669+0000) 2024-03-04T10:52:41.301 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:52:41.301+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:52:41.301933+0000) 2024-03-04T10:52:42.301 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:52:42.301+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:52:42.302170+0000) 2024-03-04T10:52:43.301 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:52:43.301+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:52:43.302426+0000) 2024-03-04T10:52:44.304 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:52:44.301+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:52:44.302695+0000) 2024-03-04T10:52:45.302 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:52:45.302+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:52:45.302933+0000) 2024-03-04T10:52:46.302 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:52:46.302+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:52:46.303184+0000) 2024-03-04T10:52:47.302 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:52:47.302+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:52:47.303421+0000) 2024-03-04T10:52:48.302 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:52:48.302+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:52:48.303661+0000) 2024-03-04T10:52:49.303 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:52:49.303+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:52:49.303933+0000) 2024-03-04T10:52:50.303 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:52:50.303+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:52:50.304209+0000) 2024-03-04T10:52:51.303 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:52:51.303+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:52:51.304448+0000) 2024-03-04T10:52:52.303 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:52:52.304+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:52:52.304694+0000) 2024-03-04T10:52:53.304 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:52:53.304+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:52:53.304913+0000) 2024-03-04T10:52:54.304 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:52:54.304+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:52:54.305176+0000) 2024-03-04T10:52:55.304 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:52:55.304+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:52:55.305411+0000) 2024-03-04T10:52:56.305 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:52:56.304+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:52:56.305674+0000) 2024-03-04T10:52:57.305 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:52:57.305+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:52:57.305947+0000) 2024-03-04T10:52:58.305 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:52:58.305+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:52:58.306212+0000) 2024-03-04T10:52:59.305 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:52:59.306+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:52:59.306476+0000) 2024-03-04T10:53:00.305 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:53:00.305+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:53:00.306727+0000) 2024-03-04T10:53:01.306 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:53:01.306+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:53:01.306965+0000) 2024-03-04T10:53:02.306 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:53:02.306+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:53:02.307211+0000) 2024-03-04T10:53:03.306 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:53:03.307+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:53:03.307452+0000) 2024-03-04T10:53:04.306 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:53:04.307+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:53:04.307693+0000) 2024-03-04T10:53:05.307 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:53:05.307+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:53:05.307930+0000) 2024-03-04T10:53:06.307 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:53:06.307+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:53:06.308180+0000) 2024-03-04T10:53:07.307 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:53:07.308+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:53:07.308423+0000) 2024-03-04T10:53:08.307 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:53:08.308+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:53:08.308700+0000) 2024-03-04T10:53:09.068 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:53:09.076 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:53:09.307 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:53:09.308+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:53:09.309023+0000) 2024-03-04T10:53:10.308 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:53:10.309+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:53:10.309267+0000) 2024-03-04T10:53:11.308 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:53:11.309+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:53:11.309505+0000) 2024-03-04T10:53:12.308 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:53:12.308+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:53:12.309744+0000) 2024-03-04T10:53:13.309 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:53:13.309+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:53:13.310050+0000) 2024-03-04T10:53:14.309 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:53:14.310+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:53:14.310288+0000) 2024-03-04T10:53:15.309 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:53:15.310+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:53:15.310525+0000) 2024-03-04T10:53:16.309 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:53:16.310+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:53:16.310774+0000) 2024-03-04T10:53:17.309 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:53:17.310+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:53:17.311006+0000) 2024-03-04T10:53:18.310 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:53:18.310+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:53:18.311250+0000) 2024-03-04T10:53:19.310 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:53:19.310+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:53:19.311487+0000) 2024-03-04T10:53:20.310 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:53:20.310+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:53:20.311723+0000) 2024-03-04T10:53:21.310 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:53:21.310+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:53:21.311998+0000) 2024-03-04T10:53:22.311 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:53:22.311+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:53:22.312238+0000) 2024-03-04T10:53:23.311 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:53:23.311+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:53:23.312481+0000) 2024-03-04T10:53:24.311 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:53:24.311+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:53:24.312720+0000) 2024-03-04T10:53:25.311 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:53:25.311+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:53:25.312934+0000) 2024-03-04T10:53:26.312 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:53:26.312+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:53:26.313185+0000) 2024-03-04T10:53:27.312 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:53:27.312+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:53:27.313423+0000) 2024-03-04T10:53:28.312 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:53:28.312+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:53:28.313653+0000) 2024-03-04T10:53:29.312 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:53:29.312+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:53:29.313948+0000) 2024-03-04T10:53:30.312 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:53:30.313+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:53:30.314209+0000) 2024-03-04T10:53:31.313 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:53:31.313+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:53:31.314447+0000) 2024-03-04T10:53:32.313 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:53:32.313+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:53:32.314695+0000) 2024-03-04T10:53:33.313 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:53:33.314+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:53:33.314941+0000) 2024-03-04T10:53:34.314 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:53:34.314+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:53:34.315178+0000) 2024-03-04T10:53:35.314 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:53:35.314+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:53:35.315417+0000) 2024-03-04T10:53:36.314 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:53:36.314+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:53:36.315667+0000) 2024-03-04T10:53:37.314 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:53:37.315+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:53:37.315920+0000) 2024-03-04T10:53:38.314 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:53:38.315+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:53:38.316162+0000) 2024-03-04T10:53:39.299 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:53:39.308 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:53:39.316 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:53:39.315+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:53:39.316390+0000) 2024-03-04T10:53:40.315 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:53:40.315+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:53:40.316583+0000) 2024-03-04T10:53:41.315 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:53:41.316+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:53:41.316862+0000) 2024-03-04T10:53:42.315 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:53:42.316+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:53:42.317105+0000) 2024-03-04T10:53:43.316 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:53:43.316+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:53:43.317353+0000) 2024-03-04T10:53:44.316 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:53:44.316+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:53:44.317583+0000) 2024-03-04T10:53:45.316 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:53:45.317+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:53:45.317863+0000) 2024-03-04T10:53:46.316 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:53:46.317+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:53:46.318130+0000) 2024-03-04T10:53:47.317 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:53:47.317+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:53:47.318365+0000) 2024-03-04T10:53:48.317 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:53:48.317+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:53:48.318610+0000) 2024-03-04T10:53:49.317 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:53:49.318+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:53:49.318894+0000) 2024-03-04T10:53:50.317 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:53:50.318+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:53:50.319126+0000) 2024-03-04T10:53:51.318 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:53:51.318+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:53:51.319374+0000) 2024-03-04T10:53:52.318 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:53:52.319+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:53:52.319609+0000) 2024-03-04T10:53:53.318 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:53:53.319+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:53:53.319908+0000) 2024-03-04T10:53:54.318 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:53:54.319+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:53:54.320148+0000) 2024-03-04T10:53:55.319 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:53:55.319+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:53:55.320396+0000) 2024-03-04T10:53:56.319 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:53:56.320+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:53:56.320658+0000) 2024-03-04T10:53:57.319 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:53:57.320+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:53:57.320933+0000) 2024-03-04T10:53:58.319 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:53:58.320+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:53:58.321180+0000) 2024-03-04T10:53:59.320 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:53:59.320+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:53:59.321455+0000) 2024-03-04T10:54:00.320 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:54:00.321+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:54:00.321720+0000) 2024-03-04T10:54:01.320 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:54:01.321+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:54:01.321967+0000) 2024-03-04T10:54:02.321 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:54:02.321+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:54:02.322200+0000) 2024-03-04T10:54:03.320 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:54:03.321+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:54:03.322393+0000) 2024-03-04T10:54:04.321 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:54:04.322+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:54:04.322644+0000) 2024-03-04T10:54:05.321 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:54:05.322+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:54:05.322933+0000) 2024-03-04T10:54:06.321 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:54:06.322+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:54:06.323188+0000) 2024-03-04T10:54:07.322 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:54:07.323+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:54:07.323481+0000) 2024-03-04T10:54:08.323 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:54:08.323+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:54:08.323729+0000) 2024-03-04T10:54:09.322 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:54:09.323+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:54:09.323970+0000) 2024-03-04T10:54:09.451 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:54:09.459 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:54:10.322 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:54:10.323+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:54:10.324218+0000) 2024-03-04T10:54:11.322 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:54:11.324+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:54:11.324460+0000) 2024-03-04T10:54:12.323 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:54:12.324+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:54:12.324701+0000) 2024-03-04T10:54:13.325 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:54:13.324+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:54:13.324954+0000) 2024-03-04T10:54:14.323 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:54:14.324+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:54:14.325196+0000) 2024-03-04T10:54:15.323 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:54:15.325+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:54:15.325439+0000) 2024-03-04T10:54:16.324 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:54:16.325+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:54:16.325688+0000) 2024-03-04T10:54:17.324 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:54:17.325+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:54:17.325931+0000) 2024-03-04T10:54:18.324 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:54:18.326+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:54:18.326180+0000) 2024-03-04T10:54:19.324 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:54:19.326+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:54:19.326421+0000) 2024-03-04T10:54:20.325 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:54:20.326+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:54:20.326664+0000) 2024-03-04T10:54:21.325 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:54:21.325+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:54:21.326947+0000) 2024-03-04T10:54:22.325 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:54:22.325+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:54:22.327141+0000) 2024-03-04T10:54:23.325 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:54:23.326+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:54:23.327407+0000) 2024-03-04T10:54:24.326 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:54:24.326+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:54:24.327650+0000) 2024-03-04T10:54:25.326 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:54:25.326+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:54:25.327938+0000) 2024-03-04T10:54:26.326 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:54:26.327+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:54:26.328189+0000) 2024-03-04T10:54:27.326 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:54:27.327+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:54:27.328429+0000) 2024-03-04T10:54:28.327 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:54:28.327+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:54:28.328676+0000) 2024-03-04T10:54:29.327 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:54:29.327+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:54:29.328937+0000) 2024-03-04T10:54:30.327 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:54:30.328+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:54:30.329201+0000) 2024-03-04T10:54:31.327 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:54:31.328+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:54:31.329450+0000) 2024-03-04T10:54:32.328 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:54:32.328+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:54:32.329697+0000) 2024-03-04T10:54:33.328 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:54:33.329+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:54:33.329946+0000) 2024-03-04T10:54:34.328 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:54:34.329+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:54:34.330147+0000) 2024-03-04T10:54:35.328 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:54:35.329+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:54:35.330346+0000) 2024-03-04T10:54:36.328 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:54:36.329+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:54:36.330624+0000) 2024-03-04T10:54:37.329 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:54:37.330+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:54:37.330917+0000) 2024-03-04T10:54:38.329 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:54:38.330+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:54:38.331167+0000) 2024-03-04T10:54:39.329 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:54:39.330+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:54:39.331409+0000) 2024-03-04T10:54:39.665 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:54:39.673 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:54:40.330 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:54:40.330+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:54:40.331654+0000) 2024-03-04T10:54:41.330 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:54:41.331+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:54:41.331944+0000) 2024-03-04T10:54:42.330 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:54:42.331+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:54:42.332178+0000) 2024-03-04T10:54:43.330 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:54:43.331+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:54:43.332429+0000) 2024-03-04T10:54:44.331 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:54:44.331+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:54:44.332669+0000) 2024-03-04T10:54:45.331 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:54:45.332+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:54:45.332970+0000) 2024-03-04T10:54:46.331 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:54:46.332+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:54:46.333219+0000) 2024-03-04T10:54:47.331 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:54:47.332+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:54:47.333460+0000) 2024-03-04T10:54:48.332 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:54:48.333+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:54:48.333708+0000) 2024-03-04T10:54:49.332 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:54:49.333+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:54:49.333956+0000) 2024-03-04T10:54:50.332 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:54:50.333+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:54:50.334196+0000) 2024-03-04T10:54:51.332 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:54:51.333+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:54:51.334439+0000) 2024-03-04T10:54:52.332 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:54:52.333+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:54:52.334668+0000) 2024-03-04T10:54:53.333 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:54:53.334+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:54:53.334942+0000) 2024-03-04T10:54:54.333 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:54:54.334+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:54:54.335170+0000) 2024-03-04T10:54:55.333 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:54:55.334+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:54:55.335411+0000) 2024-03-04T10:54:56.333 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:54:56.334+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:54:56.335648+0000) 2024-03-04T10:54:57.334 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:54:57.335+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:54:57.335937+0000) 2024-03-04T10:54:58.334 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:54:58.335+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:54:58.336192+0000) 2024-03-04T10:54:59.334 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:54:59.335+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:54:59.336472+0000) 2024-03-04T10:55:00.334 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:55:00.335+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:55:00.336712+0000) 2024-03-04T10:55:01.335 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:55:01.336+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:55:01.336966+0000) 2024-03-04T10:55:02.335 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:55:02.336+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:55:02.337206+0000) 2024-03-04T10:55:03.335 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:55:03.336+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:55:03.337455+0000) 2024-03-04T10:55:04.335 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:55:04.337+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:55:04.337705+0000) 2024-03-04T10:55:05.336 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:55:05.337+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:55:05.337949+0000) 2024-03-04T10:55:06.336 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:55:06.337+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:55:06.338202+0000) 2024-03-04T10:55:07.336 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:55:07.337+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:55:07.338441+0000) 2024-03-04T10:55:08.336 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:55:08.338+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:55:08.338688+0000) 2024-03-04T10:55:09.337 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:55:09.338+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:55:09.338931+0000) 2024-03-04T10:55:09.870 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:55:09.879 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:55:10.337 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:55:10.338+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:55:10.339177+0000) 2024-03-04T10:55:11.337 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:55:11.338+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:55:11.339455+0000) 2024-03-04T10:55:12.337 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:55:12.339+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:55:12.339693+0000) 2024-03-04T10:55:13.337 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:55:13.339+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:55:13.339963+0000) 2024-03-04T10:55:14.338 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:55:14.339+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:55:14.340210+0000) 2024-03-04T10:55:15.338 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:55:15.339+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:55:15.340475+0000) 2024-03-04T10:55:16.338 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:55:16.340+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:55:16.340732+0000) 2024-03-04T10:55:17.339 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:55:17.340+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:55:17.340984+0000) 2024-03-04T10:55:18.339 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:55:18.340+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:55:18.341188+0000) 2024-03-04T10:55:19.339 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:55:19.341+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:55:19.341444+0000) 2024-03-04T10:55:20.339 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:55:20.341+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:55:20.341683+0000) 2024-03-04T10:55:21.339 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:55:21.341+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:55:21.341960+0000) 2024-03-04T10:55:22.340 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:55:22.342+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:55:22.342209+0000) 2024-03-04T10:55:23.340 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:55:23.341+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:55:23.342475+0000) 2024-03-04T10:55:24.340 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:55:24.340+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:55:24.342710+0000) 2024-03-04T10:55:25.341 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:55:25.341+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:55:25.342956+0000) 2024-03-04T10:55:26.341 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:55:26.341+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:55:26.343199+0000) 2024-03-04T10:55:27.341 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:55:27.341+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:55:27.343466+0000) 2024-03-04T10:55:28.341 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:55:28.341+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:55:28.343712+0000) 2024-03-04T10:55:29.341 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:55:29.342+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:55:29.343963+0000) 2024-03-04T10:55:30.343 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:55:30.343+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:55:30.344236+0000) 2024-03-04T10:55:31.342 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:55:31.342+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:55:31.344484+0000) 2024-03-04T10:55:32.342 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:55:32.342+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:55:32.344723+0000) 2024-03-04T10:55:33.342 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:55:33.343+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:55:33.344964+0000) 2024-03-04T10:55:34.343 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:55:34.343+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:55:34.345146+0000) 2024-03-04T10:55:35.343 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:55:35.344+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:55:35.345423+0000) 2024-03-04T10:55:36.343 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:55:36.344+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:55:36.345673+0000) 2024-03-04T10:55:37.343 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:55:37.345+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:55:37.345951+0000) 2024-03-04T10:55:38.344 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:55:38.345+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:55:38.346198+0000) 2024-03-04T10:55:39.344 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:55:39.345+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:55:39.346442+0000) 2024-03-04T10:55:40.065 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:55:40.073 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:55:40.344 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:55:40.345+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:55:40.346690+0000) 2024-03-04T10:55:41.344 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:55:41.346+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:55:41.346948+0000) 2024-03-04T10:55:42.345 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:55:42.346+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:55:42.347185+0000) 2024-03-04T10:55:43.345 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:55:43.346+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:55:43.347427+0000) 2024-03-04T10:55:44.345 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:55:44.346+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:55:44.347665+0000) 2024-03-04T10:55:45.345 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:55:45.347+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:55:45.347932+0000) 2024-03-04T10:55:46.346 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:55:46.347+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:55:46.348179+0000) 2024-03-04T10:55:47.346 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:55:47.347+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:55:47.348416+0000) 2024-03-04T10:55:48.346 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:55:48.347+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:55:48.348676+0000) 2024-03-04T10:55:49.346 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:55:49.348+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:55:49.348950+0000) 2024-03-04T10:55:50.346 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:55:50.348+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:55:50.349192+0000) 2024-03-04T10:55:51.347 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:55:51.348+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:55:51.349426+0000) 2024-03-04T10:55:52.347 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:55:52.348+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:55:52.349663+0000) 2024-03-04T10:55:53.347 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:55:53.349+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:55:53.349950+0000) 2024-03-04T10:55:54.348 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:55:54.349+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:55:54.350197+0000) 2024-03-04T10:55:55.348 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:55:55.349+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:55:55.350438+0000) 2024-03-04T10:55:56.348 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:55:56.350+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:55:56.350682+0000) 2024-03-04T10:55:57.348 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:55:57.350+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:55:57.350946+0000) 2024-03-04T10:55:58.348 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:55:58.350+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:55:58.351188+0000) 2024-03-04T10:55:59.349 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:55:59.350+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:55:59.351518+0000) 2024-03-04T10:56:00.349 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:56:00.351+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:56:00.351775+0000) 2024-03-04T10:56:01.349 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:56:01.351+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:56:01.352019+0000) 2024-03-04T10:56:02.349 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:56:02.351+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:56:02.352253+0000) 2024-03-04T10:56:03.350 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:56:03.352+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:56:03.352505+0000) 2024-03-04T10:56:04.350 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:56:04.352+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:56:04.352770+0000) 2024-03-04T10:56:05.350 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:56:05.352+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:56:05.353005+0000) 2024-03-04T10:56:06.350 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:56:06.352+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:56:06.353215+0000) 2024-03-04T10:56:07.351 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:56:07.353+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:56:07.353468+0000) 2024-03-04T10:56:08.351 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:56:08.353+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:56:08.353717+0000) 2024-03-04T10:56:09.351 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:56:09.353+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:56:09.353995+0000) 2024-03-04T10:56:10.208 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:56:10.217 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:56:10.351 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:56:10.353+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:56:10.354266+0000) 2024-03-04T10:56:11.352 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:56:11.354+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:56:11.354507+0000) 2024-03-04T10:56:12.352 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:56:12.353+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:56:12.354689+0000) 2024-03-04T10:56:13.352 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:56:13.354+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:56:13.354974+0000) 2024-03-04T10:56:14.352 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:56:14.354+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:56:14.355211+0000) 2024-03-04T10:56:15.353 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:56:15.355+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:56:15.355450+0000) 2024-03-04T10:56:16.353 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:56:16.355+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:56:16.355703+0000) 2024-03-04T10:56:17.353 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:56:17.355+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:56:17.355962+0000) 2024-03-04T10:56:18.353 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:56:18.355+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:56:18.356201+0000) 2024-03-04T10:56:19.353 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:56:19.356+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:56:19.356438+0000) 2024-03-04T10:56:20.354 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:56:20.356+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:56:20.356676+0000) 2024-03-04T10:56:21.354 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:56:21.356+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:56:21.356952+0000) 2024-03-04T10:56:22.354 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:56:22.357+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:56:22.357186+0000) 2024-03-04T10:56:23.354 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:56:23.357+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:56:23.357437+0000) 2024-03-04T10:56:24.355 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:56:24.357+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:56:24.357672+0000) 2024-03-04T10:56:25.355 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:56:25.357+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:56:25.357952+0000) 2024-03-04T10:56:26.355 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:56:26.358+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:56:26.358200+0000) 2024-03-04T10:56:27.355 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:56:27.357+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:56:27.358439+0000) 2024-03-04T10:56:28.356 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:56:28.357+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:56:28.358687+0000) 2024-03-04T10:56:29.356 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:56:29.357+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:56:29.358927+0000) 2024-03-04T10:56:30.356 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:56:30.358+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:56:30.359185+0000) 2024-03-04T10:56:31.356 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:56:31.358+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:56:31.359447+0000) 2024-03-04T10:56:32.357 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:56:32.358+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:56:32.359689+0000) 2024-03-04T10:56:33.357 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:56:33.358+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:56:33.359939+0000) 2024-03-04T10:56:34.357 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:56:34.358+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:56:34.360178+0000) 2024-03-04T10:56:35.357 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:56:35.359+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:56:35.360416+0000) 2024-03-04T10:56:36.358 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:56:36.359+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:56:36.360673+0000) 2024-03-04T10:56:37.358 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:56:37.359+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:56:37.360936+0000) 2024-03-04T10:56:38.358 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:56:38.360+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:56:38.361136+0000) 2024-03-04T10:56:39.358 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:56:39.360+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:56:39.361318+0000) 2024-03-04T10:56:40.358 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:56:40.359 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:56:40.360+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:56:40.361493+0000) 2024-03-04T10:56:40.366 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:56:41.359 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:56:41.360+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:56:41.361733+0000) 2024-03-04T10:56:42.359 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:56:42.361+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:56:42.361991+0000) 2024-03-04T10:56:43.359 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:56:43.361+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:56:43.362247+0000) 2024-03-04T10:56:44.360 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:56:44.361+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:56:44.362483+0000) 2024-03-04T10:56:45.360 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:56:45.361+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:56:45.362718+0000) 2024-03-04T10:56:46.360 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:56:46.362+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:56:46.362983+0000) 2024-03-04T10:56:47.360 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:56:47.362+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:56:47.363217+0000) 2024-03-04T10:56:48.360 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:56:48.362+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:56:48.363465+0000) 2024-03-04T10:56:49.361 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:56:49.362+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:56:49.363702+0000) 2024-03-04T10:56:50.361 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:56:50.363+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:56:50.363950+0000) 2024-03-04T10:56:51.364 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:56:51.363+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:56:51.364203+0000) 2024-03-04T10:56:52.362 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:56:52.363+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:56:52.364524+0000) 2024-03-04T10:56:53.362 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:56:53.364+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:56:53.364792+0000) 2024-03-04T10:56:54.362 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:56:54.364+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:56:54.365088+0000) 2024-03-04T10:56:55.362 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:56:55.364+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:56:55.365373+0000) 2024-03-04T10:56:56.363 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:56:56.364+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:56:56.365646+0000) 2024-03-04T10:56:57.363 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:56:57.365+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:56:57.365929+0000) 2024-03-04T10:56:58.363 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:56:58.365+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:56:58.366194+0000) 2024-03-04T10:56:59.363 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:56:59.365+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:56:59.366453+0000) 2024-03-04T10:57:00.364 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:57:00.366+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:57:00.366743+0000) 2024-03-04T10:57:01.364 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:57:01.366+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:57:01.367031+0000) 2024-03-04T10:57:02.364 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:57:02.366+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:57:02.367285+0000) 2024-03-04T10:57:03.364 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:57:03.367+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:57:03.367553+0000) 2024-03-04T10:57:04.365 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:57:04.367+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:57:04.367817+0000) 2024-03-04T10:57:05.365 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:57:05.367+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:57:05.368110+0000) 2024-03-04T10:57:06.365 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:57:06.367+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:57:06.368375+0000) 2024-03-04T10:57:07.365 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:57:07.368+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:57:07.368635+0000) 2024-03-04T10:57:08.366 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:57:08.368+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:57:08.368953+0000) 2024-03-04T10:57:09.366 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:57:09.368+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:57:09.369171+0000) 2024-03-04T10:57:10.366 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:57:10.369+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:57:10.369427+0000) 2024-03-04T10:57:10.621 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:57:10.633 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:57:11.366 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:57:11.369+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:57:11.369686+0000) 2024-03-04T10:57:12.367 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:57:12.369+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:57:12.369976+0000) 2024-03-04T10:57:13.367 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:57:13.369+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:57:13.370247+0000) 2024-03-04T10:57:14.367 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:57:14.370+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:57:14.370508+0000) 2024-03-04T10:57:15.367 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:57:15.370+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:57:15.370761+0000) 2024-03-04T10:57:16.368 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:57:16.370+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:57:16.371033+0000) 2024-03-04T10:57:17.368 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:57:17.371+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:57:17.371310+0000) 2024-03-04T10:57:18.368 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:57:18.371+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:57:18.371581+0000) 2024-03-04T10:57:19.369 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:57:19.371+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:57:19.371855+0000) 2024-03-04T10:57:20.369 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:57:20.371+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:57:20.372115+0000) 2024-03-04T10:57:21.369 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:57:21.372+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:57:21.372363+0000) 2024-03-04T10:57:22.369 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:57:22.372+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:57:22.372613+0000) 2024-03-04T10:57:23.370 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:57:23.372+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:57:23.372943+0000) 2024-03-04T10:57:24.370 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:57:24.372+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:57:24.373142+0000) 2024-03-04T10:57:25.370 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:57:25.373+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:57:25.373381+0000) 2024-03-04T10:57:26.370 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:57:26.372+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:57:26.373625+0000) 2024-03-04T10:57:27.371 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:57:27.373+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:57:27.373950+0000) 2024-03-04T10:57:28.371 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:57:28.372+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:57:28.374198+0000) 2024-03-04T10:57:29.371 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:57:29.373+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:57:29.374457+0000) 2024-03-04T10:57:30.371 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:57:30.372+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:57:30.374711+0000) 2024-03-04T10:57:31.372 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:57:31.373+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:57:31.374947+0000) 2024-03-04T10:57:32.372 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:57:32.374+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:57:32.375186+0000) 2024-03-04T10:57:33.372 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:57:33.374+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:57:33.375451+0000) 2024-03-04T10:57:34.372 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:57:34.373+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:57:34.375683+0000) 2024-03-04T10:57:35.373 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:57:35.374+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:57:35.375961+0000) 2024-03-04T10:57:36.376 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:57:36.375+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:57:36.376207+0000) 2024-03-04T10:57:37.373 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:57:37.375+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:57:37.376442+0000) 2024-03-04T10:57:38.373 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:57:38.375+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:57:38.376704+0000) 2024-03-04T10:57:39.374 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:57:39.376+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:57:39.376943+0000) 2024-03-04T10:57:40.374 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:57:40.376+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:57:40.377177+0000) 2024-03-04T10:57:40.769 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:57:40.777 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:57:41.374 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:57:41.376+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:57:41.377415+0000) 2024-03-04T10:57:42.374 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:57:42.376+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:57:42.377649+0000) 2024-03-04T10:57:43.376 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:57:43.377+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:57:43.377933+0000) 2024-03-04T10:57:44.375 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:57:44.377+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:57:44.378167+0000) 2024-03-04T10:57:45.375 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:57:45.377+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:57:45.378463+0000) 2024-03-04T10:57:46.375 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:57:46.377+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:57:46.378727+0000) 2024-03-04T10:57:47.376 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:57:47.378+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:57:47.378961+0000) 2024-03-04T10:57:48.376 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:57:48.378+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:57:48.379194+0000) 2024-03-04T10:57:49.376 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:57:49.378+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:57:49.379429+0000) 2024-03-04T10:57:50.376 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:57:50.378+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:57:50.379660+0000) 2024-03-04T10:57:51.377 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:57:51.379+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:57:51.379936+0000) 2024-03-04T10:57:52.377 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:57:52.379+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:57:52.380137+0000) 2024-03-04T10:57:53.377 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:57:53.379+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:57:53.380374+0000) 2024-03-04T10:57:54.377 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:57:54.379+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:57:54.380607+0000) 2024-03-04T10:57:55.377 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:57:55.380+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:57:55.380879+0000) 2024-03-04T10:57:56.378 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:57:56.380+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:57:56.381076+0000) 2024-03-04T10:57:57.378 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:57:57.380+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:57:57.381308+0000) 2024-03-04T10:57:58.378 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:57:58.380+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:57:58.381541+0000) 2024-03-04T10:57:59.378 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:57:59.381+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:57:59.381806+0000) 2024-03-04T10:58:00.379 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:58:00.381+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:58:00.382103+0000) 2024-03-04T10:58:01.379 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:58:01.381+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:58:01.382288+0000) 2024-03-04T10:58:02.379 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:58:02.381+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:58:02.382529+0000) 2024-03-04T10:58:03.379 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:58:03.382+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:58:03.382765+0000) 2024-03-04T10:58:04.379 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:58:04.382+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:58:04.382999+0000) 2024-03-04T10:58:05.380 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:58:05.382+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:58:05.383234+0000) 2024-03-04T10:58:06.380 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:58:06.382+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:58:06.383430+0000) 2024-03-04T10:58:07.380 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:58:07.383+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:58:07.383668+0000) 2024-03-04T10:58:08.380 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:58:08.383+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:58:08.383932+0000) 2024-03-04T10:58:09.381 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:58:09.383+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:58:09.384168+0000) 2024-03-04T10:58:10.381 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:58:10.383+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:58:10.384402+0000) 2024-03-04T10:58:10.982 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:58:10.990 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:58:11.381 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:58:11.384+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:58:11.384647+0000) 2024-03-04T10:58:12.383 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:58:12.384+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:58:12.384914+0000) 2024-03-04T10:58:13.381 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:58:13.384+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:58:13.385152+0000) 2024-03-04T10:58:14.382 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:58:14.385+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:58:14.385396+0000) 2024-03-04T10:58:15.382 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:58:15.385+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:58:15.385641+0000) 2024-03-04T10:58:16.382 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:58:16.385+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:58:16.385926+0000) 2024-03-04T10:58:17.382 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:58:17.385+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:58:17.386165+0000) 2024-03-04T10:58:18.383 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:58:18.386+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:58:18.386400+0000) 2024-03-04T10:58:19.383 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:58:19.386+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:58:19.386673+0000) 2024-03-04T10:58:20.383 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:58:20.386+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:58:20.386939+0000) 2024-03-04T10:58:21.384 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:58:21.386+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:58:21.387199+0000) 2024-03-04T10:58:22.384 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:58:22.387+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:58:22.387431+0000) 2024-03-04T10:58:23.384 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:58:23.387+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:58:23.387675+0000) 2024-03-04T10:58:24.384 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:58:24.387+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:58:24.387944+0000) 2024-03-04T10:58:25.385 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:58:25.387+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:58:25.388180+0000) 2024-03-04T10:58:26.385 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:58:26.388+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:58:26.388424+0000) 2024-03-04T10:58:27.385 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:58:27.388+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:58:27.388660+0000) 2024-03-04T10:58:28.385 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:58:28.388+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:58:28.388919+0000) 2024-03-04T10:58:29.386 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:58:29.388+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:58:29.389113+0000) 2024-03-04T10:58:30.386 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:58:30.387+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:58:30.389300+0000) 2024-03-04T10:58:31.386 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:58:31.387+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:58:31.389480+0000) 2024-03-04T10:58:32.386 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:58:32.388+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:58:32.389691+0000) 2024-03-04T10:58:33.387 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:58:33.388+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:58:33.390073+0000) 2024-03-04T10:58:34.387 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:58:34.389+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:58:34.390344+0000) 2024-03-04T10:58:35.387 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:58:35.389+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:58:35.390633+0000) 2024-03-04T10:58:36.387 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:58:36.389+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:58:36.390928+0000) 2024-03-04T10:58:37.388 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:58:37.390+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:58:37.391208+0000) 2024-03-04T10:58:38.388 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:58:38.390+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:58:38.391443+0000) 2024-03-04T10:58:39.388 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:58:39.390+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:58:39.391682+0000) 2024-03-04T10:58:40.388 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:58:40.390+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:58:40.391938+0000) 2024-03-04T10:58:41.138 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:58:41.146 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:58:41.388 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:58:41.391+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:58:41.392188+0000) 2024-03-04T10:58:42.389 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:58:42.391+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:58:42.392372+0000) 2024-03-04T10:58:43.389 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:58:43.391+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:58:43.392665+0000) 2024-03-04T10:58:44.389 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:58:44.392+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:58:44.392940+0000) 2024-03-04T10:58:45.390 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:58:45.392+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:58:45.393196+0000) 2024-03-04T10:58:46.390 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:58:46.392+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:58:46.393472+0000) 2024-03-04T10:58:47.390 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:58:47.392+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:58:47.393657+0000) 2024-03-04T10:58:48.390 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:58:48.393+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:58:48.393923+0000) 2024-03-04T10:58:49.391 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:58:49.393+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:58:49.394166+0000) 2024-03-04T10:58:50.391 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:58:50.393+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:58:50.394396+0000) 2024-03-04T10:58:51.392 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:58:51.393+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:58:51.394638+0000) 2024-03-04T10:58:52.391 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:58:52.394+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:58:52.394922+0000) 2024-03-04T10:58:53.391 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:58:53.394+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:58:53.395117+0000) 2024-03-04T10:58:54.392 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:58:54.394+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:58:54.395366+0000) 2024-03-04T10:58:55.392 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:58:55.394+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:58:55.395628+0000) 2024-03-04T10:58:56.392 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:58:56.395+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:58:56.395921+0000) 2024-03-04T10:58:57.392 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:58:57.395+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:58:57.396164+0000) 2024-03-04T10:58:58.393 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:58:58.395+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:58:58.396423+0000) 2024-03-04T10:58:59.393 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:58:59.396+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:58:59.396687+0000) 2024-03-04T10:59:00.393 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:59:00.396+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:59:00.396976+0000) 2024-03-04T10:59:01.393 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:59:01.396+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:59:01.397177+0000) 2024-03-04T10:59:02.394 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:59:02.396+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:59:02.397425+0000) 2024-03-04T10:59:03.394 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:59:03.397+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:59:03.397693+0000) 2024-03-04T10:59:04.394 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:59:04.397+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:59:04.397944+0000) 2024-03-04T10:59:05.394 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:59:05.397+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:59:05.398205+0000) 2024-03-04T10:59:06.395 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:59:06.397+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:59:06.398484+0000) 2024-03-04T10:59:07.395 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:59:07.398+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:59:07.398736+0000) 2024-03-04T10:59:08.395 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:59:08.398+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:59:08.399011+0000) 2024-03-04T10:59:09.395 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:59:09.398+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:59:09.399267+0000) 2024-03-04T10:59:10.396 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:59:10.399+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:59:10.399516+0000) 2024-03-04T10:59:11.334 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:59:11.342 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:59:11.396 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:59:11.398+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:59:11.399726+0000) 2024-03-04T10:59:12.396 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:59:12.399+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:59:12.399982+0000) 2024-03-04T10:59:13.397 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:59:13.399+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:59:13.400279+0000) 2024-03-04T10:59:14.397 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:59:14.400+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:59:14.400568+0000) 2024-03-04T10:59:15.397 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:59:15.400+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:59:15.400914+0000) 2024-03-04T10:59:16.397 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:59:16.400+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:59:16.401210+0000) 2024-03-04T10:59:17.398 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:59:17.401+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:59:17.401491+0000) 2024-03-04T10:59:18.398 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:59:18.401+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:59:18.401778+0000) 2024-03-04T10:59:19.398 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:59:19.401+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:59:19.402115+0000) 2024-03-04T10:59:20.398 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:59:20.402+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:59:20.402403+0000) 2024-03-04T10:59:21.399 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:59:21.402+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:59:21.402686+0000) 2024-03-04T10:59:22.399 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:59:22.402+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:59:22.402975+0000) 2024-03-04T10:59:23.399 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:59:23.402+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:59:23.403265+0000) 2024-03-04T10:59:24.400 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:59:24.403+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:59:24.403560+0000) 2024-03-04T10:59:25.400 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:59:25.403+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:59:25.403901+0000) 2024-03-04T10:59:26.400 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:59:26.403+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:59:26.404202+0000) 2024-03-04T10:59:27.401 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:59:27.404+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:59:27.404581+0000) 2024-03-04T10:59:28.401 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:59:28.404+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:59:28.404925+0000) 2024-03-04T10:59:29.401 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:59:29.405+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:59:29.405237+0000) 2024-03-04T10:59:30.402 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:59:30.405+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:59:30.405528+0000) 2024-03-04T10:59:31.402 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:59:31.405+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:59:31.405811+0000) 2024-03-04T10:59:32.402 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:59:32.405+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:59:32.406153+0000) 2024-03-04T10:59:33.403 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:59:33.405+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:59:33.406448+0000) 2024-03-04T10:59:34.403 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:59:34.405+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:59:34.406733+0000) 2024-03-04T10:59:35.403 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:59:35.405+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:59:35.407020+0000) 2024-03-04T10:59:36.403 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:59:36.406+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:59:36.407315+0000) 2024-03-04T10:59:37.404 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:59:37.406+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:59:37.407593+0000) 2024-03-04T10:59:38.404 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:59:38.406+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:59:38.407907+0000) 2024-03-04T10:59:39.404 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:59:39.407+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:59:39.408199+0000) 2024-03-04T10:59:40.404 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:59:40.407+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:59:40.408489+0000) 2024-03-04T10:59:41.405 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:59:41.407+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:59:41.408740+0000) 2024-03-04T10:59:41.530 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:59:41.538 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:59:42.405 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:59:42.408+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:59:42.409058+0000) 2024-03-04T10:59:43.405 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:59:43.408+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:59:43.409315+0000) 2024-03-04T10:59:44.406 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:59:44.408+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:59:44.409593+0000) 2024-03-04T10:59:45.406 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:59:45.409+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:59:45.409932+0000) 2024-03-04T10:59:46.406 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:59:46.409+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:59:46.410250+0000) 2024-03-04T10:59:47.407 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:59:47.409+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:59:47.410519+0000) 2024-03-04T10:59:48.407 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:59:48.409+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:59:48.410755+0000) 2024-03-04T10:59:49.407 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:59:49.410+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:59:49.410998+0000) 2024-03-04T10:59:50.407 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:59:50.410+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:59:50.411230+0000) 2024-03-04T10:59:51.407 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:59:51.410+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:59:51.411499+0000) 2024-03-04T10:59:52.408 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:59:52.410+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:59:52.411748+0000) 2024-03-04T10:59:53.408 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:59:53.411+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:59:53.412017+0000) 2024-03-04T10:59:54.408 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:59:54.411+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:59:54.412273+0000) 2024-03-04T10:59:55.409 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:59:55.411+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:59:55.412521+0000) 2024-03-04T10:59:56.409 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:59:56.412+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:59:56.412789+0000) 2024-03-04T10:59:57.409 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:59:57.412+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:59:57.413031+0000) 2024-03-04T10:59:58.409 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:59:58.412+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:59:58.413285+0000) 2024-03-04T10:59:59.409 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T10:59:59.412+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T09:59:59.413531+0000) 2024-03-04T11:00:00.410 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:00:00.413+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:00:00.413765+0000) 2024-03-04T11:00:01.410 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:00:01.413+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:00:01.414038+0000) 2024-03-04T11:00:02.410 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:00:02.413+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:00:02.414335+0000) 2024-03-04T11:00:03.411 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:00:03.413+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:00:03.414626+0000) 2024-03-04T11:00:04.411 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:00:04.414+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:00:04.414961+0000) 2024-03-04T11:00:05.411 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:00:05.414+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:00:05.415249+0000) 2024-03-04T11:00:06.411 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:00:06.414+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:00:06.415511+0000) 2024-03-04T11:00:07.412 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:00:07.415+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:00:07.415751+0000) 2024-03-04T11:00:08.412 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:00:08.415+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:00:08.415986+0000) 2024-03-04T11:00:09.412 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:00:09.415+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:00:09.416235+0000) 2024-03-04T11:00:10.412 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:00:10.415+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:00:10.416478+0000) 2024-03-04T11:00:11.413 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:00:11.416+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:00:11.416717+0000) 2024-03-04T11:00:11.718 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:00:11.726 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:00:12.413 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:00:12.416+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:00:12.416947+0000) 2024-03-04T11:00:13.413 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:00:13.416+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:00:13.417194+0000) 2024-03-04T11:00:14.413 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:00:14.416+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:00:14.417465+0000) 2024-03-04T11:00:15.414 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:00:15.417+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:00:15.417708+0000) 2024-03-04T11:00:16.414 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:00:16.417+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:00:16.417957+0000) 2024-03-04T11:00:17.414 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:00:17.417+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:00:17.418201+0000) 2024-03-04T11:00:18.414 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:00:18.417+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:00:18.418443+0000) 2024-03-04T11:00:19.415 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:00:19.418+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:00:19.418689+0000) 2024-03-04T11:00:20.415 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:00:20.418+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:00:20.418930+0000) 2024-03-04T11:00:21.415 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:00:21.418+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:00:21.419171+0000) 2024-03-04T11:00:22.415 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:00:22.418+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:00:22.419397+0000) 2024-03-04T11:00:23.416 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:00:23.418+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:00:23.419663+0000) 2024-03-04T11:00:24.416 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:00:24.419+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:00:24.419947+0000) 2024-03-04T11:00:25.416 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:00:25.419+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:00:25.420250+0000) 2024-03-04T11:00:26.417 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:00:26.419+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:00:26.420582+0000) 2024-03-04T11:00:27.417 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:00:27.420+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:00:27.420810+0000) 2024-03-04T11:00:28.417 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:00:28.420+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:00:28.421083+0000) 2024-03-04T11:00:29.417 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:00:29.421+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:00:29.421323+0000) 2024-03-04T11:00:30.417 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:00:30.421+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:00:30.421555+0000) 2024-03-04T11:00:31.418 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:00:31.421+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:00:31.421812+0000) 2024-03-04T11:00:32.418 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:00:32.421+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:00:32.422036+0000) 2024-03-04T11:00:33.418 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:00:33.422+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:00:33.422286+0000) 2024-03-04T11:00:34.418 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:00:34.421+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:00:34.422521+0000) 2024-03-04T11:00:35.418 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:00:35.421+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:00:35.422704+0000) 2024-03-04T11:00:36.419 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:00:36.421+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:00:36.422946+0000) 2024-03-04T11:00:37.419 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:00:37.422+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:00:37.423184+0000) 2024-03-04T11:00:38.419 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:00:38.422+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:00:38.423428+0000) 2024-03-04T11:00:39.419 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:00:39.422+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:00:39.423670+0000) 2024-03-04T11:00:40.420 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:00:40.422+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:00:40.423934+0000) 2024-03-04T11:00:41.424 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:00:41.423+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:00:41.424181+0000) 2024-03-04T11:00:41.907 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:00:41.918 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:00:42.420 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:00:42.423+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:00:42.424414+0000) 2024-03-04T11:00:43.420 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:00:43.423+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:00:43.424671+0000) 2024-03-04T11:00:44.421 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:00:44.423+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:00:44.424939+0000) 2024-03-04T11:00:45.421 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:00:45.424+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:00:45.425197+0000) 2024-03-04T11:00:46.421 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:00:46.424+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:00:46.425439+0000) 2024-03-04T11:00:47.421 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:00:47.424+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:00:47.425676+0000) 2024-03-04T11:00:48.422 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:00:48.425+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:00:48.425933+0000) 2024-03-04T11:00:49.422 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:00:49.425+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:00:49.426178+0000) 2024-03-04T11:00:50.422 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:00:50.425+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:00:50.426418+0000) 2024-03-04T11:00:51.422 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:00:51.425+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:00:51.426655+0000) 2024-03-04T11:00:52.422 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:00:52.426+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:00:52.426921+0000) 2024-03-04T11:00:53.423 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:00:53.426+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:00:53.427188+0000) 2024-03-04T11:00:54.423 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:00:54.426+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:00:54.427442+0000) 2024-03-04T11:00:55.423 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:00:55.426+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:00:55.427688+0000) 2024-03-04T11:00:56.424 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:00:56.427+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:00:56.427979+0000) 2024-03-04T11:00:57.424 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:00:57.427+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:00:57.428224+0000) 2024-03-04T11:00:58.424 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:00:58.427+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:00:58.428477+0000) 2024-03-04T11:00:59.424 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:00:59.428+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:00:59.428739+0000) 2024-03-04T11:01:00.425 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:01:00.428+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:01:00.428991+0000) 2024-03-04T11:01:01.425 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:01:01.428+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:01:01.429254+0000) 2024-03-04T11:01:02.425 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:01:02.428+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:01:02.429505+0000) 2024-03-04T11:01:03.425 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:01:03.429+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:01:03.429771+0000) 2024-03-04T11:01:04.426 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:01:04.429+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:01:04.430045+0000) 2024-03-04T11:01:05.426 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:01:05.429+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:01:05.430307+0000) 2024-03-04T11:01:06.426 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:01:06.429+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:01:06.430571+0000) 2024-03-04T11:01:07.427 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:01:07.430+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:01:07.430910+0000) 2024-03-04T11:01:08.427 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:01:08.430+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:01:08.431167+0000) 2024-03-04T11:01:09.427 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:01:09.430+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:01:09.431440+0000) 2024-03-04T11:01:10.427 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:01:10.431+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:01:10.431689+0000) 2024-03-04T11:01:11.428 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:01:11.431+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:01:11.431939+0000) 2024-03-04T11:01:12.095 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:01:12.104 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:01:12.428 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:01:12.431+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:01:12.432128+0000) 2024-03-04T11:01:13.428 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:01:13.431+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:01:13.432336+0000) 2024-03-04T11:01:14.428 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:01:14.431+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:01:14.432526+0000) 2024-03-04T11:01:15.428 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:01:15.431+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:01:15.432788+0000) 2024-03-04T11:01:16.429 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:01:16.432+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:01:16.433078+0000) 2024-03-04T11:01:17.429 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:01:17.432+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:01:17.433331+0000) 2024-03-04T11:01:18.429 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:01:18.433+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:01:18.433593+0000) 2024-03-04T11:01:19.429 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:01:19.433+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:01:19.433954+0000) 2024-03-04T11:01:20.430 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:01:20.433+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:01:20.434134+0000) 2024-03-04T11:01:21.430 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:01:21.434+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:01:21.434376+0000) 2024-03-04T11:01:22.430 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:01:22.434+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:01:22.434615+0000) 2024-03-04T11:01:23.431 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:01:23.434+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:01:23.434961+0000) 2024-03-04T11:01:24.431 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:01:24.434+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:01:24.435255+0000) 2024-03-04T11:01:25.431 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:01:25.435+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:01:25.435501+0000) 2024-03-04T11:01:26.431 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:01:26.435+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:01:26.435744+0000) 2024-03-04T11:01:27.432 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:01:27.435+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:01:27.435978+0000) 2024-03-04T11:01:28.432 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:01:28.435+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:01:28.436224+0000) 2024-03-04T11:01:29.432 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:01:29.435+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:01:29.436430+0000) 2024-03-04T11:01:30.432 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:01:30.436+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:01:30.436665+0000) 2024-03-04T11:01:31.432 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:01:31.436+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:01:31.436958+0000) 2024-03-04T11:01:32.433 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:01:32.437+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:01:32.437189+0000) 2024-03-04T11:01:33.433 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:01:33.437+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:01:33.437432+0000) 2024-03-04T11:01:34.433 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:01:34.436+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:01:34.437686+0000) 2024-03-04T11:01:35.433 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:01:35.436+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:01:35.437980+0000) 2024-03-04T11:01:36.435 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:01:36.436+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:01:36.438213+0000) 2024-03-04T11:01:37.434 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:01:37.437+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:01:37.438447+0000) 2024-03-04T11:01:38.434 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:01:38.437+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:01:38.438711+0000) 2024-03-04T11:01:39.434 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:01:39.437+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:01:39.438976+0000) 2024-03-04T11:01:40.435 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:01:40.438+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:01:40.439211+0000) 2024-03-04T11:01:41.435 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:01:41.438+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:01:41.439455+0000) 2024-03-04T11:01:42.242 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:01:42.250 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:01:42.435 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:01:42.438+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:01:42.439693+0000) 2024-03-04T11:01:43.435 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:01:43.438+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:01:43.439949+0000) 2024-03-04T11:01:44.436 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:01:44.439+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:01:44.440186+0000) 2024-03-04T11:01:45.436 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:01:45.439+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:01:45.440433+0000) 2024-03-04T11:01:46.436 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:01:46.439+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:01:46.440677+0000) 2024-03-04T11:01:47.436 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:01:47.440+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:01:47.440953+0000) 2024-03-04T11:01:48.437 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:01:48.440+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:01:48.441189+0000) 2024-03-04T11:01:49.437 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:01:49.440+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:01:49.441439+0000) 2024-03-04T11:01:50.437 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:01:50.440+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:01:50.441673+0000) 2024-03-04T11:01:51.437 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:01:51.441+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:01:51.441970+0000) 2024-03-04T11:01:52.438 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:01:52.441+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:01:52.442224+0000) 2024-03-04T11:01:53.438 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:01:53.441+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:01:53.442494+0000) 2024-03-04T11:01:54.439 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:01:54.441+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:01:54.442728+0000) 2024-03-04T11:01:55.438 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:01:55.442+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:01:55.442971+0000) 2024-03-04T11:01:56.439 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:01:56.442+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:01:56.443215+0000) 2024-03-04T11:01:57.439 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:01:57.442+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:01:57.443463+0000) 2024-03-04T11:01:58.439 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:01:58.442+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:01:58.443714+0000) 2024-03-04T11:01:59.439 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:01:59.443+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:01:59.443962+0000) 2024-03-04T11:02:00.440 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:02:00.443+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:02:00.444158+0000) 2024-03-04T11:02:01.440 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:02:01.443+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:02:01.444356+0000) 2024-03-04T11:02:02.440 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:02:02.443+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:02:02.444531+0000) 2024-03-04T11:02:03.440 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:02:03.444+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:02:03.444782+0000) 2024-03-04T11:02:04.440 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:02:04.444+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:02:04.445019+0000) 2024-03-04T11:02:05.441 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:02:05.444+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:02:05.445262+0000) 2024-03-04T11:02:06.441 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:02:06.444+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:02:06.445510+0000) 2024-03-04T11:02:07.441 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:02:07.444+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:02:07.445768+0000) 2024-03-04T11:02:08.441 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:02:08.445+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:02:08.446001+0000) 2024-03-04T11:02:09.442 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:02:09.445+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:02:09.446205+0000) 2024-03-04T11:02:10.442 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:02:10.445+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:02:10.446440+0000) 2024-03-04T11:02:11.442 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:02:11.446+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:02:11.446680+0000) 2024-03-04T11:02:12.387 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:02:12.395 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:02:12.442 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:02:12.446+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:02:12.446895+0000) 2024-03-04T11:02:13.442 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:02:13.446+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:02:13.447147+0000) 2024-03-04T11:02:14.443 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:02:14.446+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:02:14.447407+0000) 2024-03-04T11:02:15.443 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:02:15.447+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:02:15.447666+0000) 2024-03-04T11:02:16.443 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:02:16.447+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:02:16.447937+0000) 2024-03-04T11:02:17.444 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:02:17.447+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:02:17.448198+0000) 2024-03-04T11:02:18.444 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:02:18.447+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:02:18.448395+0000) 2024-03-04T11:02:19.444 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:02:19.448+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:02:19.448664+0000) 2024-03-04T11:02:20.444 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:02:20.448+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:02:20.448953+0000) 2024-03-04T11:02:21.445 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:02:21.448+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:02:21.449217+0000) 2024-03-04T11:02:22.445 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:02:22.448+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:02:22.449456+0000) 2024-03-04T11:02:23.445 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:02:23.449+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:02:23.449735+0000) 2024-03-04T11:02:24.445 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:02:24.449+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:02:24.449995+0000) 2024-03-04T11:02:25.446 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:02:25.449+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:02:25.450262+0000) 2024-03-04T11:02:26.446 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:02:26.450+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:02:26.450531+0000) 2024-03-04T11:02:27.446 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:02:27.450+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:02:27.450793+0000) 2024-03-04T11:02:28.446 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:02:28.450+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:02:28.451089+0000) 2024-03-04T11:02:29.447 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:02:29.451+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:02:29.451350+0000) 2024-03-04T11:02:30.447 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:02:30.451+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:02:30.451604+0000) 2024-03-04T11:02:31.447 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:02:31.451+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:02:31.451924+0000) 2024-03-04T11:02:32.447 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:02:32.451+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:02:32.452129+0000) 2024-03-04T11:02:33.448 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:02:33.451+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:02:33.452418+0000) 2024-03-04T11:02:34.448 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:02:34.452+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:02:34.452672+0000) 2024-03-04T11:02:35.448 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:02:35.452+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:02:35.452976+0000) 2024-03-04T11:02:36.449 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:02:36.452+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:02:36.453260+0000) 2024-03-04T11:02:37.449 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:02:37.452+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:02:37.453502+0000) 2024-03-04T11:02:38.449 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:02:38.451+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:02:38.453704+0000) 2024-03-04T11:02:39.449 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:02:39.452+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:02:39.454001+0000) 2024-03-04T11:02:40.450 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:02:40.452+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:02:40.454248+0000) 2024-03-04T11:02:41.450 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:02:41.452+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:02:41.454502+0000) 2024-03-04T11:02:42.450 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:02:42.453+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:02:42.454783+0000) 2024-03-04T11:02:42.593 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:02:42.604 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:02:43.450 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:02:43.454+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:02:43.455090+0000) 2024-03-04T11:02:44.451 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:02:44.454+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:02:44.455324+0000) 2024-03-04T11:02:45.451 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:02:45.454+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:02:45.455597+0000) 2024-03-04T11:02:46.451 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:02:46.454+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:02:46.455902+0000) 2024-03-04T11:02:47.451 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:02:47.455+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:02:47.456191+0000) 2024-03-04T11:02:48.452 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:02:48.455+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:02:48.456438+0000) 2024-03-04T11:02:49.452 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:02:49.455+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:02:49.456679+0000) 2024-03-04T11:02:50.452 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:02:50.456+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:02:50.456948+0000) 2024-03-04T11:02:51.452 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:02:51.456+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:02:51.457183+0000) 2024-03-04T11:02:52.453 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:02:52.456+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:02:52.457464+0000) 2024-03-04T11:02:53.453 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:02:53.456+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:02:53.457739+0000) 2024-03-04T11:02:54.453 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:02:54.457+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:02:54.457990+0000) 2024-03-04T11:02:55.454 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:02:55.457+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:02:55.458329+0000) 2024-03-04T11:02:56.454 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:02:56.457+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:02:56.458609+0000) 2024-03-04T11:02:57.454 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:02:57.458+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:02:57.458915+0000) 2024-03-04T11:02:58.454 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:02:58.458+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:02:58.459165+0000) 2024-03-04T11:02:59.455 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:02:59.458+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:02:59.459419+0000) 2024-03-04T11:03:00.455 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:03:00.458+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:03:00.459670+0000) 2024-03-04T11:03:01.455 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:03:01.459+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:03:01.459929+0000) 2024-03-04T11:03:02.455 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:03:02.459+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:03:02.460183+0000) 2024-03-04T11:03:03.456 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:03:03.459+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:03:03.460442+0000) 2024-03-04T11:03:04.456 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:03:04.460+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:03:04.460690+0000) 2024-03-04T11:03:05.456 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:03:05.460+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:03:05.460943+0000) 2024-03-04T11:03:06.456 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:03:06.460+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:03:06.461210+0000) 2024-03-04T11:03:07.457 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:03:07.460+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:03:07.461462+0000) 2024-03-04T11:03:08.457 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:03:08.461+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:03:08.461747+0000) 2024-03-04T11:03:09.457 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:03:09.461+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:03:09.462005+0000) 2024-03-04T11:03:10.457 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:03:10.461+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:03:10.462254+0000) 2024-03-04T11:03:11.458 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:03:11.461+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:03:11.462511+0000) 2024-03-04T11:03:12.458 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:03:12.461+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:03:12.462751+0000) 2024-03-04T11:03:12.799 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:03:12.807 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:03:13.458 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:03:13.462+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:03:13.462996+0000) 2024-03-04T11:03:14.459 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:03:14.462+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:03:14.463235+0000) 2024-03-04T11:03:15.459 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:03:15.462+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:03:15.463480+0000) 2024-03-04T11:03:16.459 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:03:16.463+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:03:16.463733+0000) 2024-03-04T11:03:17.459 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:03:17.463+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:03:17.463971+0000) 2024-03-04T11:03:18.459 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:03:18.463+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:03:18.464163+0000) 2024-03-04T11:03:19.460 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:03:19.463+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:03:19.464403+0000) 2024-03-04T11:03:20.460 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:03:20.464+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:03:20.464640+0000) 2024-03-04T11:03:21.460 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:03:21.464+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:03:21.464937+0000) 2024-03-04T11:03:22.460 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:03:22.464+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:03:22.465187+0000) 2024-03-04T11:03:23.461 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:03:23.465+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:03:23.465429+0000) 2024-03-04T11:03:24.461 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:03:24.465+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:03:24.465666+0000) 2024-03-04T11:03:25.461 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:03:25.465+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:03:25.465935+0000) 2024-03-04T11:03:26.461 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:03:26.465+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:03:26.466183+0000) 2024-03-04T11:03:27.461 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:03:27.466+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:03:27.466422+0000) 2024-03-04T11:03:28.462 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:03:28.466+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:03:28.466657+0000) 2024-03-04T11:03:29.462 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:03:29.466+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:03:29.466934+0000) 2024-03-04T11:03:30.462 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:03:30.466+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:03:30.467168+0000) 2024-03-04T11:03:31.463 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:03:31.466+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:03:31.467401+0000) 2024-03-04T11:03:32.463 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:03:32.467+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:03:32.467641+0000) 2024-03-04T11:03:33.463 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:03:33.467+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:03:33.467942+0000) 2024-03-04T11:03:34.463 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:03:34.468+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:03:34.468191+0000) 2024-03-04T11:03:35.464 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:03:35.468+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:03:35.468443+0000) 2024-03-04T11:03:36.464 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:03:36.468+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:03:36.468693+0000) 2024-03-04T11:03:37.464 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:03:37.468+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:03:37.468929+0000) 2024-03-04T11:03:38.464 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:03:38.468+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:03:38.469189+0000) 2024-03-04T11:03:39.465 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:03:39.468+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:03:39.469434+0000) 2024-03-04T11:03:40.465 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:03:40.468+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:03:40.469671+0000) 2024-03-04T11:03:41.465 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:03:41.468+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:03:41.469940+0000) 2024-03-04T11:03:42.465 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:03:42.469+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:03:42.470196+0000) 2024-03-04T11:03:43.027 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:03:43.035 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:03:43.465 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:03:43.469+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:03:43.470462+0000) 2024-03-04T11:03:44.466 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:03:44.469+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:03:44.470714+0000) 2024-03-04T11:03:45.466 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:03:45.469+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:03:45.470966+0000) 2024-03-04T11:03:46.466 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:03:46.469+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:03:46.471174+0000) 2024-03-04T11:03:47.466 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:03:47.470+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:03:47.471425+0000) 2024-03-04T11:03:48.467 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:03:48.470+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:03:48.471673+0000) 2024-03-04T11:03:49.467 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:03:49.471+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:03:49.471928+0000) 2024-03-04T11:03:50.467 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:03:50.471+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:03:50.472166+0000) 2024-03-04T11:03:51.467 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:03:51.471+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:03:51.472437+0000) 2024-03-04T11:03:52.468 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:03:52.471+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:03:52.472711+0000) 2024-03-04T11:03:53.468 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:03:53.472+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:03:53.473025+0000) 2024-03-04T11:03:54.468 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:03:54.472+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:03:54.473279+0000) 2024-03-04T11:03:55.468 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:03:55.472+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:03:55.473533+0000) 2024-03-04T11:03:56.469 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:03:56.472+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:03:56.473797+0000) 2024-03-04T11:03:57.469 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:03:57.473+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:03:57.474100+0000) 2024-03-04T11:03:58.469 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:03:58.473+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:03:58.474357+0000) 2024-03-04T11:03:59.470 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:03:59.473+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:03:59.474623+0000) 2024-03-04T11:04:00.470 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:04:00.474+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:04:00.474924+0000) 2024-03-04T11:04:01.470 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:04:01.474+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:04:01.475196+0000) 2024-03-04T11:04:02.470 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:04:02.474+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:04:02.475451+0000) 2024-03-04T11:04:03.471 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:04:03.474+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:04:03.475708+0000) 2024-03-04T11:04:04.471 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:04:04.475+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:04:04.475963+0000) 2024-03-04T11:04:05.471 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:04:05.475+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:04:05.476224+0000) 2024-03-04T11:04:06.471 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:04:06.475+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:04:06.476510+0000) 2024-03-04T11:04:07.472 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:04:07.476+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:04:07.476757+0000) 2024-03-04T11:04:08.472 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:04:08.476+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:04:08.477010+0000) 2024-03-04T11:04:09.472 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:04:09.476+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:04:09.477296+0000) 2024-03-04T11:04:10.472 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:04:10.476+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:04:10.477548+0000) 2024-03-04T11:04:11.473 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:04:11.477+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:04:11.477790+0000) 2024-03-04T11:04:12.473 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:04:12.477+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:04:12.478023+0000) 2024-03-04T11:04:13.165 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:04:13.173 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:04:13.473 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:04:13.477+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:04:13.478268+0000) 2024-03-04T11:04:14.473 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:04:14.477+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:04:14.478458+0000) 2024-03-04T11:04:15.474 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:04:15.478+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:04:15.478706+0000) 2024-03-04T11:04:16.474 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:04:16.478+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:04:16.478950+0000) 2024-03-04T11:04:17.474 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:04:17.478+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:04:17.479190+0000) 2024-03-04T11:04:18.474 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:04:18.478+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:04:18.479424+0000) 2024-03-04T11:04:19.475 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:04:19.479+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:04:19.479672+0000) 2024-03-04T11:04:20.475 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:04:20.479+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:04:20.479952+0000) 2024-03-04T11:04:21.475 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:04:21.479+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:04:21.480212+0000) 2024-03-04T11:04:22.475 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:04:22.480+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:04:22.480447+0000) 2024-03-04T11:04:23.476 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:04:23.480+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:04:23.480689+0000) 2024-03-04T11:04:24.476 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:04:24.480+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:04:24.480931+0000) 2024-03-04T11:04:25.476 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:04:25.480+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:04:25.481299+0000) 2024-03-04T11:04:26.477 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:04:26.481+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:04:26.481565+0000) 2024-03-04T11:04:27.477 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:04:27.481+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:04:27.481859+0000) 2024-03-04T11:04:28.477 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:04:28.481+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:04:28.482100+0000) 2024-03-04T11:04:29.477 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:04:29.482+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:04:29.482347+0000) 2024-03-04T11:04:30.478 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:04:30.482+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:04:30.482582+0000) 2024-03-04T11:04:31.478 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:04:31.482+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:04:31.482863+0000) 2024-03-04T11:04:32.478 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:04:32.482+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:04:32.483103+0000) 2024-03-04T11:04:33.478 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:04:33.483+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:04:33.483352+0000) 2024-03-04T11:04:34.479 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:04:34.483+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:04:34.483579+0000) 2024-03-04T11:04:35.479 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:04:35.483+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:04:35.483767+0000) 2024-03-04T11:04:36.479 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:04:36.483+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:04:36.484019+0000) 2024-03-04T11:04:37.479 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:04:37.484+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:04:37.484255+0000) 2024-03-04T11:04:38.480 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:04:38.484+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:04:38.484492+0000) 2024-03-04T11:04:39.480 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:04:39.483+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:04:39.484737+0000) 2024-03-04T11:04:40.480 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:04:40.483+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:04:40.484976+0000) 2024-03-04T11:04:41.480 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:04:41.484+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:04:41.485221+0000) 2024-03-04T11:04:42.480 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:04:42.484+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:04:42.485453+0000) 2024-03-04T11:04:43.378 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:04:43.386 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:04:43.480 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:04:43.483+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:04:43.485629+0000) 2024-03-04T11:04:44.481 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:04:44.484+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:04:44.485911+0000) 2024-03-04T11:04:45.481 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:04:45.485+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:04:45.486156+0000) 2024-03-04T11:04:46.481 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:04:46.485+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:04:46.486391+0000) 2024-03-04T11:04:47.481 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:04:47.485+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:04:47.486626+0000) 2024-03-04T11:04:48.482 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:04:48.485+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:04:48.486907+0000) 2024-03-04T11:04:49.482 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:04:49.486+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:04:49.487180+0000) 2024-03-04T11:04:50.482 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:04:50.486+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:04:50.487490+0000) 2024-03-04T11:04:51.483 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:04:51.486+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:04:51.487793+0000) 2024-03-04T11:04:52.483 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:04:52.487+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:04:52.488085+0000) 2024-03-04T11:04:53.483 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:04:53.487+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:04:53.488331+0000) 2024-03-04T11:04:54.483 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:04:54.487+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:04:54.488578+0000) 2024-03-04T11:04:55.484 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:04:55.488+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:04:55.488866+0000) 2024-03-04T11:04:56.484 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:04:56.488+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:04:56.489118+0000) 2024-03-04T11:04:57.484 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:04:57.488+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:04:57.489356+0000) 2024-03-04T11:04:58.485 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:04:58.488+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:04:58.489589+0000) 2024-03-04T11:04:59.485 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:04:59.489+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:04:59.489884+0000) 2024-03-04T11:05:00.485 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:05:00.489+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:05:00.490121+0000) 2024-03-04T11:05:01.486 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:05:01.489+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:05:01.490364+0000) 2024-03-04T11:05:02.485 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:05:02.489+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:05:02.490597+0000) 2024-03-04T11:05:03.486 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:05:03.490+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:05:03.490884+0000) 2024-03-04T11:05:04.486 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:05:04.490+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:05:04.491123+0000) 2024-03-04T11:05:05.486 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:05:05.490+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:05:05.491367+0000) 2024-03-04T11:05:06.486 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:05:06.490+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:05:06.491609+0000) 2024-03-04T11:05:07.487 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:05:07.491+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:05:07.491878+0000) 2024-03-04T11:05:08.487 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:05:08.491+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:05:08.492115+0000) 2024-03-04T11:05:09.487 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:05:09.491+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:05:09.492366+0000) 2024-03-04T11:05:10.487 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:05:10.491+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:05:10.492601+0000) 2024-03-04T11:05:11.488 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:05:11.492+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:05:11.492889+0000) 2024-03-04T11:05:12.488 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:05:12.492+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:05:12.493171+0000) 2024-03-04T11:05:13.488 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:05:13.492+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:05:13.493426+0000) 2024-03-04T11:05:13.524 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:05:13.532 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:05:14.488 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:05:14.493+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:05:14.493661+0000) 2024-03-04T11:05:15.489 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:05:15.493+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:05:15.493953+0000) 2024-03-04T11:05:16.489 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:05:16.493+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:05:16.494202+0000) 2024-03-04T11:05:17.489 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:05:17.493+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:05:17.494385+0000) 2024-03-04T11:05:18.489 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:05:18.494+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:05:18.494621+0000) 2024-03-04T11:05:19.490 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:05:19.494+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:05:19.494913+0000) 2024-03-04T11:05:20.490 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:05:20.494+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:05:20.495167+0000) 2024-03-04T11:05:21.490 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:05:21.494+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:05:21.495436+0000) 2024-03-04T11:05:22.490 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:05:22.495+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:05:22.495669+0000) 2024-03-04T11:05:23.491 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:05:23.495+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:05:23.495967+0000) 2024-03-04T11:05:24.491 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:05:24.495+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:05:24.496205+0000) 2024-03-04T11:05:25.491 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:05:25.496+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:05:25.496467+0000) 2024-03-04T11:05:26.491 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:05:26.496+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:05:26.496730+0000) 2024-03-04T11:05:27.492 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:05:27.496+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:05:27.496979+0000) 2024-03-04T11:05:28.492 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:05:28.496+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:05:28.497179+0000) 2024-03-04T11:05:29.492 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:05:29.497+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:05:29.497442+0000) 2024-03-04T11:05:30.492 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:05:30.496+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:05:30.497648+0000) 2024-03-04T11:05:31.493 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:05:31.497+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:05:31.497932+0000) 2024-03-04T11:05:32.493 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:05:32.497+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:05:32.498170+0000) 2024-03-04T11:05:33.493 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:05:33.498+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:05:33.498411+0000) 2024-03-04T11:05:34.493 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:05:34.498+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:05:34.498646+0000) 2024-03-04T11:05:35.494 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:05:35.498+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:05:35.498962+0000) 2024-03-04T11:05:36.494 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:05:36.499+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:05:36.499210+0000) 2024-03-04T11:05:37.494 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:05:37.499+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:05:37.499446+0000) 2024-03-04T11:05:38.494 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:05:38.499+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:05:38.499696+0000) 2024-03-04T11:05:39.495 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:05:39.499+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:05:39.499992+0000) 2024-03-04T11:05:40.495 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:05:40.499+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:05:40.500242+0000) 2024-03-04T11:05:41.495 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:05:41.499+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:05:41.500496+0000) 2024-03-04T11:05:42.495 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:05:42.499+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:05:42.500737+0000) 2024-03-04T11:05:43.496 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:05:43.499+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:05:43.500978+0000) 2024-03-04T11:05:43.754 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:05:43.762 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:05:44.496 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:05:44.500+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:05:44.501221+0000) 2024-03-04T11:05:45.496 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:05:45.500+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:05:45.501525+0000) 2024-03-04T11:05:46.496 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:05:46.500+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:05:46.501773+0000) 2024-03-04T11:05:47.497 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:05:47.500+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:05:47.502009+0000) 2024-03-04T11:05:48.497 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:05:48.501+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:05:48.502250+0000) 2024-03-04T11:05:49.497 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:05:49.501+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:05:49.502501+0000) 2024-03-04T11:05:50.498 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:05:50.501+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:05:50.502761+0000) 2024-03-04T11:05:51.498 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:05:51.502+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:05:51.503031+0000) 2024-03-04T11:05:52.498 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:05:52.502+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:05:52.503268+0000) 2024-03-04T11:05:53.498 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:05:53.502+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:05:53.503520+0000) 2024-03-04T11:05:54.499 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:05:54.502+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:05:54.503780+0000) 2024-03-04T11:05:55.499 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:05:55.503+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:05:55.504093+0000) 2024-03-04T11:05:56.499 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:05:56.503+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:05:56.504364+0000) 2024-03-04T11:05:57.499 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:05:57.503+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:05:57.504654+0000) 2024-03-04T11:05:58.499 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:05:58.504+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:05:58.504955+0000) 2024-03-04T11:05:59.500 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:05:59.504+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:05:59.505226+0000) 2024-03-04T11:06:00.500 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:06:00.504+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:06:00.505483+0000) 2024-03-04T11:06:01.500 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:06:01.504+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:06:01.505765+0000) 2024-03-04T11:06:02.501 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:06:02.505+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:06:02.506013+0000) 2024-03-04T11:06:03.501 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:06:03.505+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:06:03.506271+0000) 2024-03-04T11:06:04.501 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:06:04.505+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:06:04.506534+0000) 2024-03-04T11:06:05.501 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:06:05.505+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:06:05.506800+0000) 2024-03-04T11:06:06.502 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:06:06.506+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:06:06.507099+0000) 2024-03-04T11:06:07.502 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:06:07.506+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:06:07.507305+0000) 2024-03-04T11:06:08.502 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:06:08.506+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:06:08.507499+0000) 2024-03-04T11:06:09.502 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:06:09.507+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:06:09.507794+0000) 2024-03-04T11:06:10.503 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:06:10.507+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:06:10.508089+0000) 2024-03-04T11:06:11.503 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:06:11.507+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:06:11.508376+0000) 2024-03-04T11:06:12.504 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:06:12.507+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:06:12.508613+0000) 2024-03-04T11:06:13.503 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:06:13.508+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:06:13.508907+0000) 2024-03-04T11:06:13.967 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:06:13.975 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:06:14.504 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:06:14.508+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:06:14.509150+0000) 2024-03-04T11:06:15.504 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:06:15.508+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:06:15.509409+0000) 2024-03-04T11:06:16.504 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:06:16.508+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:06:16.509618+0000) 2024-03-04T11:06:17.504 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:06:17.509+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:06:17.509899+0000) 2024-03-04T11:06:18.505 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:06:18.509+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:06:18.510197+0000) 2024-03-04T11:06:19.505 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:06:19.509+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:06:19.510444+0000) 2024-03-04T11:06:20.505 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:06:20.510+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:06:20.510701+0000) 2024-03-04T11:06:21.505 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:06:21.510+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:06:21.510958+0000) 2024-03-04T11:06:22.506 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:06:22.510+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:06:22.511202+0000) 2024-03-04T11:06:23.506 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:06:23.511+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:06:23.511461+0000) 2024-03-04T11:06:24.506 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:06:24.510+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:06:24.511716+0000) 2024-03-04T11:06:25.507 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:06:25.511+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:06:25.511990+0000) 2024-03-04T11:06:26.507 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:06:26.511+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:06:26.512243+0000) 2024-03-04T11:06:27.507 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:06:27.512+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:06:27.512513+0000) 2024-03-04T11:06:28.507 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:06:28.511+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:06:28.512770+0000) 2024-03-04T11:06:29.507 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:06:29.512+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:06:29.513007+0000) 2024-03-04T11:06:30.508 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:06:30.512+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:06:30.513244+0000) 2024-03-04T11:06:31.508 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:06:31.513+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:06:31.513482+0000) 2024-03-04T11:06:32.508 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:06:32.513+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:06:32.513718+0000) 2024-03-04T11:06:33.509 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:06:33.513+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:06:33.514068+0000) 2024-03-04T11:06:34.509 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:06:34.514+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:06:34.514326+0000) 2024-03-04T11:06:35.509 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:06:35.513+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:06:35.514600+0000) 2024-03-04T11:06:36.509 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:06:36.514+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:06:36.514886+0000) 2024-03-04T11:06:37.510 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:06:37.514+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:06:37.515124+0000) 2024-03-04T11:06:38.510 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:06:38.515+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:06:38.515359+0000) 2024-03-04T11:06:39.510 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:06:39.515+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:06:39.515597+0000) 2024-03-04T11:06:40.511 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:06:40.515+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:06:40.515881+0000) 2024-03-04T11:06:41.511 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:06:41.515+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:06:41.516126+0000) 2024-03-04T11:06:42.511 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:06:42.516+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:06:42.516360+0000) 2024-03-04T11:06:43.511 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:06:43.516+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:06:43.516599+0000) 2024-03-04T11:06:44.172 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:06:44.181 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:06:44.511 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:06:44.515+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:06:44.516900+0000) 2024-03-04T11:06:45.512 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:06:45.516+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:06:45.517153+0000) 2024-03-04T11:06:46.512 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:06:46.516+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:06:46.517404+0000) 2024-03-04T11:06:47.512 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:06:47.516+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:06:47.517640+0000) 2024-03-04T11:06:48.512 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:06:48.516+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:06:48.517949+0000) 2024-03-04T11:06:49.513 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:06:49.516+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:06:49.518144+0000) 2024-03-04T11:06:50.513 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:06:50.517+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:06:50.518392+0000) 2024-03-04T11:06:51.513 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:06:51.517+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:06:51.518631+0000) 2024-03-04T11:06:52.513 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:06:52.517+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:06:52.518907+0000) 2024-03-04T11:06:53.514 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:06:53.518+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:06:53.519145+0000) 2024-03-04T11:06:54.514 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:06:54.518+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:06:54.519382+0000) 2024-03-04T11:06:55.514 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:06:55.518+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:06:55.519635+0000) 2024-03-04T11:06:56.514 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:06:56.519+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:06:56.519944+0000) 2024-03-04T11:06:57.515 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:06:57.519+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:06:57.520177+0000) 2024-03-04T11:06:58.515 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:06:58.519+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:06:58.520412+0000) 2024-03-04T11:06:59.515 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:06:59.519+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:06:59.520647+0000) 2024-03-04T11:07:00.515 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:07:00.520+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:07:00.520947+0000) 2024-03-04T11:07:01.516 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:07:01.520+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:07:01.521193+0000) 2024-03-04T11:07:02.516 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:07:02.520+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:07:02.521426+0000) 2024-03-04T11:07:03.516 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:07:03.520+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:07:03.521666+0000) 2024-03-04T11:07:04.516 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:07:04.521+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:07:04.521927+0000) 2024-03-04T11:07:05.517 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:07:05.521+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:07:05.522175+0000) 2024-03-04T11:07:06.517 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:07:06.521+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:07:06.522437+0000) 2024-03-04T11:07:07.517 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:07:07.521+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:07:07.522673+0000) 2024-03-04T11:07:08.518 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:07:08.522+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:07:08.522943+0000) 2024-03-04T11:07:09.518 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:07:09.522+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:07:09.523178+0000) 2024-03-04T11:07:10.518 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:07:10.522+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:07:10.523456+0000) 2024-03-04T11:07:11.518 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:07:11.523+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:07:11.523695+0000) 2024-03-04T11:07:12.519 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:07:12.523+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:07:12.523935+0000) 2024-03-04T11:07:13.519 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:07:13.523+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:07:13.524175+0000) 2024-03-04T11:07:14.404 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:07:14.415 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:07:14.553 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:07:14.523+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:07:14.524440+0000) 2024-03-04T11:07:15.519 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:07:15.524+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:07:15.524713+0000) 2024-03-04T11:07:16.519 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:07:16.524+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:07:16.524956+0000) 2024-03-04T11:07:17.520 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:07:17.524+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:07:17.525188+0000) 2024-03-04T11:07:18.520 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:07:18.524+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:07:18.525440+0000) 2024-03-04T11:07:19.520 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:07:19.525+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:07:19.525697+0000) 2024-03-04T11:07:20.520 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:07:20.525+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:07:20.525936+0000) 2024-03-04T11:07:21.521 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:07:21.525+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:07:21.526180+0000) 2024-03-04T11:07:22.521 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:07:22.525+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:07:22.526411+0000) 2024-03-04T11:07:23.521 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:07:23.526+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:07:23.526648+0000) 2024-03-04T11:07:24.521 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:07:24.526+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:07:24.526934+0000) 2024-03-04T11:07:25.521 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:07:25.526+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:07:25.527129+0000) 2024-03-04T11:07:26.522 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:07:26.526+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:07:26.527388+0000) 2024-03-04T11:07:27.523 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:07:27.527+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:07:27.527626+0000) 2024-03-04T11:07:28.522 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:07:28.527+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:07:28.527916+0000) 2024-03-04T11:07:29.523 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:07:29.527+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:07:29.528156+0000) 2024-03-04T11:07:30.523 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:07:30.527+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:07:30.528406+0000) 2024-03-04T11:07:31.523 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:07:31.528+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:07:31.528652+0000) 2024-03-04T11:07:32.523 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:07:32.528+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:07:32.528971+0000) 2024-03-04T11:07:33.524 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:07:33.528+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:07:33.529209+0000) 2024-03-04T11:07:34.524 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:07:34.529+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:07:34.529449+0000) 2024-03-04T11:07:35.524 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:07:35.529+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:07:35.529694+0000) 2024-03-04T11:07:36.524 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:07:36.529+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:07:36.529974+0000) 2024-03-04T11:07:37.525 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:07:37.529+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:07:37.530231+0000) 2024-03-04T11:07:38.525 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:07:38.529+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:07:38.530448+0000) 2024-03-04T11:07:39.525 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:07:39.529+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:07:39.530691+0000) 2024-03-04T11:07:40.525 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:07:40.530+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:07:40.530929+0000) 2024-03-04T11:07:41.526 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:07:41.530+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:07:41.531167+0000) 2024-03-04T11:07:42.526 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:07:42.531+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:07:42.531396+0000) 2024-03-04T11:07:43.526 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:07:43.531+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:07:43.531632+0000) 2024-03-04T11:07:44.526 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:07:44.531+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:07:44.531911+0000) 2024-03-04T11:07:44.615 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:07:44.623 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:07:45.526 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:07:45.532+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:07:45.532153+0000) 2024-03-04T11:07:46.527 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:07:46.531+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:07:46.532401+0000) 2024-03-04T11:07:47.527 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:07:47.531+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:07:47.532771+0000) 2024-03-04T11:07:48.527 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:07:48.531+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:07:48.533046+0000) 2024-03-04T11:07:49.528 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:07:49.532+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:07:49.533300+0000) 2024-03-04T11:07:50.528 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:07:50.532+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:07:50.533533+0000) 2024-03-04T11:07:51.528 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:07:51.532+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:07:51.533780+0000) 2024-03-04T11:07:52.528 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:07:52.533+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:07:52.534011+0000) 2024-03-04T11:07:53.528 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:07:53.533+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:07:53.534249+0000) 2024-03-04T11:07:54.529 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:07:54.533+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:07:54.534487+0000) 2024-03-04T11:07:55.529 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:07:55.533+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:07:55.534727+0000) 2024-03-04T11:07:56.529 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:07:56.534+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:07:56.534971+0000) 2024-03-04T11:07:57.529 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:07:57.534+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:07:57.535226+0000) 2024-03-04T11:07:58.530 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:07:58.534+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:07:58.535408+0000) 2024-03-04T11:07:59.532 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:07:59.534+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:07:59.535663+0000) 2024-03-04T11:08:00.530 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:08:00.535+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:08:00.535932+0000) 2024-03-04T11:08:01.531 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:08:01.535+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:08:01.536190+0000) 2024-03-04T11:08:02.531 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:08:02.535+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:08:02.536425+0000) 2024-03-04T11:08:03.531 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:08:03.535+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:08:03.536664+0000) 2024-03-04T11:08:04.531 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:08:04.536+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:08:04.536942+0000) 2024-03-04T11:08:05.532 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:08:05.536+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:08:05.537183+0000) 2024-03-04T11:08:06.532 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:08:06.536+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:08:06.537432+0000) 2024-03-04T11:08:07.532 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:08:07.536+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:08:07.537670+0000) 2024-03-04T11:08:08.532 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:08:08.537+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:08:08.537958+0000) 2024-03-04T11:08:09.532 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:08:09.537+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:08:09.538203+0000) 2024-03-04T11:08:10.533 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:08:10.537+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:08:10.538436+0000) 2024-03-04T11:08:11.533 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:08:11.537+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:08:11.538693+0000) 2024-03-04T11:08:12.533 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:08:12.538+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:08:12.538943+0000) 2024-03-04T11:08:13.533 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:08:13.538+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:08:13.539185+0000) 2024-03-04T11:08:14.534 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:08:14.538+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:08:14.539415+0000) 2024-03-04T11:08:14.757 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:08:14.765 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:08:15.534 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:08:15.539+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:08:15.539727+0000) 2024-03-04T11:08:16.534 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:08:16.539+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:08:16.539971+0000) 2024-03-04T11:08:17.534 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:08:17.539+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:08:17.540213+0000) 2024-03-04T11:08:18.535 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:08:18.539+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:08:18.540475+0000) 2024-03-04T11:08:19.535 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:08:19.540+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:08:19.540711+0000) 2024-03-04T11:08:20.535 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:08:20.540+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:08:20.540943+0000) 2024-03-04T11:08:21.535 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:08:21.540+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:08:21.541178+0000) 2024-03-04T11:08:22.536 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:08:22.540+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:08:22.541366+0000) 2024-03-04T11:08:23.536 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:08:23.541+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:08:23.541602+0000) 2024-03-04T11:08:24.536 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:08:24.541+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:08:24.541791+0000) 2024-03-04T11:08:25.537 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:08:25.541+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:08:25.542077+0000) 2024-03-04T11:08:26.537 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:08:26.541+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:08:26.542318+0000) 2024-03-04T11:08:27.537 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:08:27.541+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:08:27.542494+0000) 2024-03-04T11:08:28.537 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:08:28.542+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:08:28.542725+0000) 2024-03-04T11:08:29.537 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:08:29.542+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:08:29.542989+0000) 2024-03-04T11:08:30.538 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:08:30.542+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:08:30.543223+0000) 2024-03-04T11:08:31.538 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:08:31.543+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:08:31.543464+0000) 2024-03-04T11:08:32.538 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:08:32.543+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:08:32.543702+0000) 2024-03-04T11:08:33.538 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:08:33.543+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:08:33.544000+0000) 2024-03-04T11:08:34.538 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:08:34.543+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:08:34.544236+0000) 2024-03-04T11:08:35.539 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:08:35.544+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:08:35.544482+0000) 2024-03-04T11:08:36.539 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:08:36.544+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:08:36.544729+0000) 2024-03-04T11:08:37.539 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:08:37.544+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:08:37.544961+0000) 2024-03-04T11:08:38.539 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:08:38.544+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:08:38.545200+0000) 2024-03-04T11:08:39.540 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:08:39.544+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:08:39.545389+0000) 2024-03-04T11:08:40.540 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:08:40.545+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:08:40.545620+0000) 2024-03-04T11:08:41.540 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:08:41.545+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:08:41.545907+0000) 2024-03-04T11:08:42.540 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:08:42.545+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:08:42.546124+0000) 2024-03-04T11:08:43.541 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:08:43.545+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:08:43.546381+0000) 2024-03-04T11:08:44.541 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:08:44.546+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:08:44.546616+0000) 2024-03-04T11:08:44.993 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:08:45.001 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:08:45.541 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:08:45.546+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:08:45.546883+0000) 2024-03-04T11:08:46.541 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:08:46.546+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:08:46.547162+0000) 2024-03-04T11:08:47.542 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:08:47.547+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:08:47.547398+0000) 2024-03-04T11:08:48.542 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:08:48.546+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:08:48.547632+0000) 2024-03-04T11:08:49.542 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:08:49.546+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:08:49.547922+0000) 2024-03-04T11:08:50.542 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:08:50.547+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:08:50.548155+0000) 2024-03-04T11:08:51.542 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:08:51.546+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:08:51.548338+0000) 2024-03-04T11:08:52.543 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:08:52.547+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:08:52.548575+0000) 2024-03-04T11:08:53.543 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:08:53.547+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:08:53.548811+0000) 2024-03-04T11:08:54.543 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:08:54.548+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:08:54.549082+0000) 2024-03-04T11:08:55.543 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:08:55.548+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:08:55.549322+0000) 2024-03-04T11:08:56.544 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:08:56.548+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:08:56.549566+0000) 2024-03-04T11:08:57.544 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:08:57.548+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:08:57.549803+0000) 2024-03-04T11:08:58.544 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:08:58.549+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:08:58.550077+0000) 2024-03-04T11:08:59.544 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:08:59.549+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:08:59.550338+0000) 2024-03-04T11:09:00.545 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:09:00.549+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:09:00.550573+0000) 2024-03-04T11:09:01.545 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:09:01.550+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:09:01.550817+0000) 2024-03-04T11:09:02.545 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:09:02.550+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:09:02.551091+0000) 2024-03-04T11:09:03.545 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:09:03.550+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:09:03.551336+0000) 2024-03-04T11:09:04.546 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:09:04.550+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:09:04.551569+0000) 2024-03-04T11:09:05.546 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:09:05.551+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:09:05.551814+0000) 2024-03-04T11:09:06.546 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:09:06.551+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:09:06.552104+0000) 2024-03-04T11:09:07.547 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:09:07.551+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:09:07.552344+0000) 2024-03-04T11:09:08.547 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:09:08.551+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:09:08.552581+0000) 2024-03-04T11:09:09.547 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:09:09.552+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:09:09.552894+0000) 2024-03-04T11:09:10.547 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:09:10.552+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:09:10.553073+0000) 2024-03-04T11:09:11.547 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:09:11.552+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:09:11.553272+0000) 2024-03-04T11:09:12.547 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:09:12.552+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:09:12.553497+0000) 2024-03-04T11:09:13.548 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:09:13.553+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:09:13.553792+0000) 2024-03-04T11:09:14.549 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:09:14.553+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:09:14.554080+0000) 2024-03-04T11:09:15.156 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:09:15.165 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:09:15.548 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:09:15.553+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:09:15.554329+0000) 2024-03-04T11:09:16.549 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:09:16.553+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:09:16.554534+0000) 2024-03-04T11:09:17.549 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:09:17.553+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:09:17.554805+0000) 2024-03-04T11:09:18.549 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:09:18.554+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:09:18.555089+0000) 2024-03-04T11:09:19.549 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:09:19.554+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:09:19.555321+0000) 2024-03-04T11:09:20.550 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:09:20.554+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:09:20.555577+0000) 2024-03-04T11:09:21.550 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:09:21.555+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:09:21.555864+0000) 2024-03-04T11:09:22.550 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:09:22.555+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:09:22.556104+0000) 2024-03-04T11:09:23.550 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:09:23.555+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:09:23.556347+0000) 2024-03-04T11:09:24.551 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:09:24.556+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:09:24.556587+0000) 2024-03-04T11:09:25.551 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:09:25.556+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:09:25.556870+0000) 2024-03-04T11:09:26.551 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:09:26.556+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:09:26.557120+0000) 2024-03-04T11:09:27.551 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:09:27.556+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:09:27.557362+0000) 2024-03-04T11:09:28.552 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:09:28.557+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:09:28.557607+0000) 2024-03-04T11:09:29.552 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:09:29.557+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:09:29.557895+0000) 2024-03-04T11:09:30.552 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:09:30.557+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:09:30.558139+0000) 2024-03-04T11:09:31.552 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:09:31.557+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:09:31.558386+0000) 2024-03-04T11:09:32.553 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:09:32.558+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:09:32.558628+0000) 2024-03-04T11:09:33.553 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:09:33.558+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:09:33.558923+0000) 2024-03-04T11:09:34.553 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:09:34.558+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:09:34.559152+0000) 2024-03-04T11:09:35.553 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:09:35.559+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:09:35.559402+0000) 2024-03-04T11:09:36.554 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:09:36.559+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:09:36.559653+0000) 2024-03-04T11:09:37.554 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:09:37.559+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:09:37.559947+0000) 2024-03-04T11:09:38.554 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:09:38.559+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:09:38.560194+0000) 2024-03-04T11:09:39.554 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:09:39.559+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:09:39.560388+0000) 2024-03-04T11:09:40.555 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:09:40.560+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:09:40.560629+0000) 2024-03-04T11:09:41.555 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:09:41.560+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:09:41.560926+0000) 2024-03-04T11:09:42.555 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:09:42.560+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:09:42.561180+0000) 2024-03-04T11:09:43.555 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:09:43.561+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:09:43.561431+0000) 2024-03-04T11:09:44.556 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:09:44.561+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:09:44.561676+0000) 2024-03-04T11:09:45.328 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:09:45.336 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:09:45.556 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:09:45.561+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:09:45.561972+0000) 2024-03-04T11:09:46.556 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:09:46.562+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:09:46.562222+0000) 2024-03-04T11:09:47.556 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:09:47.562+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:09:47.562478+0000) 2024-03-04T11:09:48.557 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:09:48.561+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:09:48.562715+0000) 2024-03-04T11:09:49.557 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:09:49.562+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:09:49.562959+0000) 2024-03-04T11:09:50.557 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:09:50.563+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:09:50.563222+0000) 2024-03-04T11:09:51.557 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:09:51.562+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:09:51.563499+0000) 2024-03-04T11:09:52.558 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:09:52.562+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:09:52.563742+0000) 2024-03-04T11:09:53.558 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:09:53.562+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:09:53.563987+0000) 2024-03-04T11:09:54.558 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:09:54.563+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:09:54.564228+0000) 2024-03-04T11:09:55.559 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:09:55.562+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:09:55.564427+0000) 2024-03-04T11:09:56.559 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:09:56.562+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:09:56.564678+0000) 2024-03-04T11:09:57.559 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:09:57.563+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:09:57.564928+0000) 2024-03-04T11:09:58.559 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:09:58.563+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:09:58.565160+0000) 2024-03-04T11:09:59.559 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:09:59.563+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:09:59.565367+0000) 2024-03-04T11:10:00.560 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:10:00.564+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:10:00.565606+0000) 2024-03-04T11:10:01.560 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:10:01.564+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:10:01.565902+0000) 2024-03-04T11:10:02.560 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:10:02.565+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:10:02.566136+0000) 2024-03-04T11:10:03.560 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:10:03.565+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:10:03.566377+0000) 2024-03-04T11:10:04.561 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:10:04.565+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:10:04.566609+0000) 2024-03-04T11:10:05.561 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:10:05.566+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:10:05.566811+0000) 2024-03-04T11:10:06.563 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:10:06.566+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:10:06.567098+0000) 2024-03-04T11:10:07.561 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:10:07.566+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:10:07.567333+0000) 2024-03-04T11:10:08.562 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:10:08.566+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:10:08.567594+0000) 2024-03-04T11:10:09.562 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:10:09.567+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:10:09.567868+0000) 2024-03-04T11:10:10.562 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:10:10.567+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:10:10.568161+0000) 2024-03-04T11:10:11.562 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:10:11.567+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:10:11.568442+0000) 2024-03-04T11:10:12.563 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:10:12.567+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:10:12.568744+0000) 2024-03-04T11:10:13.563 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:10:13.568+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:10:13.568979+0000) 2024-03-04T11:10:14.563 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:10:14.568+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:10:14.569267+0000) 2024-03-04T11:10:15.490 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:10:15.498 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:10:15.563 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:10:15.568+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:10:15.569485+0000) 2024-03-04T11:10:16.564 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:10:16.568+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:10:16.569752+0000) 2024-03-04T11:10:17.564 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:10:17.569+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:10:17.570001+0000) 2024-03-04T11:10:18.564 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:10:18.569+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:10:18.570269+0000) 2024-03-04T11:10:19.564 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:10:19.569+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:10:19.570516+0000) 2024-03-04T11:10:20.565 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:10:20.570+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:10:20.570776+0000) 2024-03-04T11:10:21.565 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:10:21.570+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:10:21.571043+0000) 2024-03-04T11:10:22.565 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:10:22.570+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:10:22.571234+0000) 2024-03-04T11:10:23.565 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:10:23.570+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:10:23.571474+0000) 2024-03-04T11:10:24.566 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:10:24.571+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:10:24.571707+0000) 2024-03-04T11:10:25.566 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:10:25.571+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:10:25.571956+0000) 2024-03-04T11:10:26.566 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:10:26.571+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:10:26.572200+0000) 2024-03-04T11:10:27.566 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:10:27.571+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:10:27.572434+0000) 2024-03-04T11:10:28.567 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:10:28.572+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:10:28.572680+0000) 2024-03-04T11:10:29.567 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:10:29.572+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:10:29.572927+0000) 2024-03-04T11:10:30.567 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:10:30.572+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:10:30.573179+0000) 2024-03-04T11:10:31.567 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:10:31.572+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:10:31.573380+0000) 2024-03-04T11:10:32.568 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:10:32.573+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:10:32.573612+0000) 2024-03-04T11:10:33.568 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:10:33.573+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:10:33.573906+0000) 2024-03-04T11:10:34.568 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:10:34.573+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:10:34.574141+0000) 2024-03-04T11:10:35.568 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:10:35.573+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:10:35.574389+0000) 2024-03-04T11:10:36.569 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:10:36.574+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:10:36.574637+0000) 2024-03-04T11:10:37.569 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:10:37.574+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:10:37.574903+0000) 2024-03-04T11:10:38.569 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:10:38.574+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:10:38.575136+0000) 2024-03-04T11:10:39.569 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:10:39.575+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:10:39.575386+0000) 2024-03-04T11:10:40.570 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:10:40.575+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:10:40.575631+0000) 2024-03-04T11:10:41.570 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:10:41.575+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:10:41.575918+0000) 2024-03-04T11:10:42.570 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:10:42.575+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:10:42.576150+0000) 2024-03-04T11:10:43.570 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:10:43.576+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:10:43.576395+0000) 2024-03-04T11:10:44.571 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:10:44.576+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:10:44.576633+0000) 2024-03-04T11:10:45.571 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:10:45.576+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:10:45.576931+0000) 2024-03-04T11:10:45.704 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:10:45.713 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:10:46.572 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:10:46.576+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:10:46.577177+0000) 2024-03-04T11:10:47.571 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:10:47.577+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:10:47.577411+0000) 2024-03-04T11:10:48.572 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:10:48.577+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:10:48.577653+0000) 2024-03-04T11:10:49.572 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:10:49.577+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:10:49.577935+0000) 2024-03-04T11:10:50.572 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:10:50.577+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:10:50.578168+0000) 2024-03-04T11:10:51.572 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:10:51.578+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:10:51.578438+0000) 2024-03-04T11:10:52.573 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:10:52.577+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:10:52.578672+0000) 2024-03-04T11:10:53.573 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:10:53.578+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:10:53.578948+0000) 2024-03-04T11:10:54.573 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:10:54.578+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:10:54.579181+0000) 2024-03-04T11:10:55.573 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:10:55.578+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:10:55.579422+0000) 2024-03-04T11:10:56.574 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:10:56.578+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:10:56.579670+0000) 2024-03-04T11:10:57.574 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:10:57.578+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:10:57.579940+0000) 2024-03-04T11:10:58.574 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:10:58.579+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:10:58.580171+0000) 2024-03-04T11:10:59.574 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:10:59.579+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:10:59.580414+0000) 2024-03-04T11:11:00.575 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:11:00.579+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:11:00.580654+0000) 2024-03-04T11:11:01.575 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:11:01.579+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:11:01.580947+0000) 2024-03-04T11:11:02.575 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:11:02.579+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:11:02.581311+0000) 2024-03-04T11:11:03.575 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:11:03.580+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:11:03.581571+0000) 2024-03-04T11:11:04.576 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:11:04.581+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:11:04.581858+0000) 2024-03-04T11:11:05.576 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:11:05.581+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:11:05.582177+0000) 2024-03-04T11:11:06.576 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:11:06.581+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:11:06.582440+0000) 2024-03-04T11:11:07.576 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:11:07.581+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:11:07.582696+0000) 2024-03-04T11:11:08.577 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:11:08.582+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:11:08.582944+0000) 2024-03-04T11:11:09.577 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:11:09.582+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:11:09.583194+0000) 2024-03-04T11:11:10.577 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:11:10.582+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:11:10.583443+0000) 2024-03-04T11:11:11.578 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:11:11.582+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:11:11.583702+0000) 2024-03-04T11:11:12.578 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:11:12.583+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:11:12.583985+0000) 2024-03-04T11:11:13.578 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:11:13.583+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:11:13.584192+0000) 2024-03-04T11:11:14.578 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:11:14.583+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:11:14.584384+0000) 2024-03-04T11:11:15.579 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:11:15.583+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:11:15.584652+0000) 2024-03-04T11:11:15.918 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:11:15.926 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:11:16.579 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:11:16.584+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:11:16.584933+0000) 2024-03-04T11:11:17.579 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:11:17.584+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:11:17.585185+0000) 2024-03-04T11:11:18.579 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:11:18.584+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:11:18.585434+0000) 2024-03-04T11:11:19.580 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:11:19.584+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:11:19.585686+0000) 2024-03-04T11:11:20.580 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:11:20.585+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:11:20.585960+0000) 2024-03-04T11:11:21.580 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:11:21.585+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:11:21.586218+0000) 2024-03-04T11:11:22.580 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:11:22.585+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:11:22.586482+0000) 2024-03-04T11:11:23.581 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:11:23.586+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:11:23.586737+0000) 2024-03-04T11:11:24.581 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:11:24.586+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:11:24.587005+0000) 2024-03-04T11:11:25.581 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:11:25.586+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:11:25.587257+0000) 2024-03-04T11:11:26.583 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:11:26.586+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:11:26.587529+0000) 2024-03-04T11:11:27.582 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:11:27.587+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:11:27.587876+0000) 2024-03-04T11:11:28.582 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:11:28.587+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:11:28.588176+0000) 2024-03-04T11:11:29.582 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:11:29.587+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:11:29.588484+0000) 2024-03-04T11:11:30.583 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:11:30.588+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:11:30.588774+0000) 2024-03-04T11:11:31.583 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:11:31.588+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:11:31.589081+0000) 2024-03-04T11:11:32.583 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:11:32.588+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:11:32.589349+0000) 2024-03-04T11:11:33.583 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:11:33.589+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:11:33.589621+0000) 2024-03-04T11:11:34.584 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:11:34.589+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:11:34.589909+0000) 2024-03-04T11:11:35.584 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:11:35.589+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:11:35.590146+0000) 2024-03-04T11:11:36.584 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:11:36.589+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:11:36.590347+0000) 2024-03-04T11:11:37.584 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:11:37.590+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:11:37.590589+0000) 2024-03-04T11:11:38.585 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:11:38.590+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:11:38.590856+0000) 2024-03-04T11:11:39.585 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:11:39.590+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:11:39.591090+0000) 2024-03-04T11:11:40.585 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:11:40.590+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:11:40.591322+0000) 2024-03-04T11:11:41.585 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:11:41.591+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:11:41.591581+0000) 2024-03-04T11:11:42.586 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:11:42.591+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:11:42.591887+0000) 2024-03-04T11:11:43.586 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:11:43.591+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:11:43.592134+0000) 2024-03-04T11:11:44.586 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:11:44.592+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:11:44.592373+0000) 2024-03-04T11:11:45.586 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:11:45.592+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:11:45.592621+0000) 2024-03-04T11:11:46.090 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:11:46.098 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:11:46.587 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:11:46.592+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:11:46.592864+0000) 2024-03-04T11:11:47.587 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:11:47.592+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:11:47.593103+0000) 2024-03-04T11:11:48.587 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:11:48.593+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:11:48.593371+0000) 2024-03-04T11:11:49.587 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:11:49.593+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:11:49.593637+0000) 2024-03-04T11:11:50.588 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:11:50.593+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:11:50.593922+0000) 2024-03-04T11:11:51.588 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:11:51.593+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:11:51.594201+0000) 2024-03-04T11:11:52.589 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:11:52.594+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:11:52.594456+0000) 2024-03-04T11:11:53.588 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:11:53.594+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:11:53.594759+0000) 2024-03-04T11:11:54.589 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:11:54.594+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:11:54.594992+0000) 2024-03-04T11:11:55.589 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:11:55.595+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:11:55.595231+0000) 2024-03-04T11:11:56.589 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:11:56.594+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:11:56.595477+0000) 2024-03-04T11:11:57.589 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:11:57.594+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:11:57.595716+0000) 2024-03-04T11:11:58.590 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:11:58.594+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:11:58.595953+0000) 2024-03-04T11:11:59.590 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:11:59.595+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:11:59.596193+0000) 2024-03-04T11:12:00.590 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:12:00.595+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:12:00.596427+0000) 2024-03-04T11:12:01.590 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:12:01.595+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:12:01.596680+0000) 2024-03-04T11:12:02.591 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:12:02.595+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:12:02.596953+0000) 2024-03-04T11:12:03.591 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:12:03.596+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:12:03.597192+0000) 2024-03-04T11:12:04.591 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:12:04.596+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:12:04.597444+0000) 2024-03-04T11:12:05.591 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:12:05.596+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:12:05.597699+0000) 2024-03-04T11:12:06.592 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:12:06.597+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:12:06.597958+0000) 2024-03-04T11:12:07.592 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:12:07.597+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:12:07.598207+0000) 2024-03-04T11:12:08.592 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:12:08.597+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:12:08.598459+0000) 2024-03-04T11:12:09.592 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:12:09.597+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:12:09.598715+0000) 2024-03-04T11:12:10.592 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:12:10.598+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:12:10.598964+0000) 2024-03-04T11:12:11.593 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:12:11.598+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:12:11.599225+0000) 2024-03-04T11:12:12.593 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:12:12.598+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:12:12.599472+0000) 2024-03-04T11:12:13.593 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:12:13.598+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:12:13.599727+0000) 2024-03-04T11:12:14.594 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:12:14.599+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:12:14.599975+0000) 2024-03-04T11:12:15.594 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:12:15.599+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:12:15.600223+0000) 2024-03-04T11:12:16.295 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:12:16.308 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:12:16.594 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:12:16.599+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:12:16.600490+0000) 2024-03-04T11:12:17.594 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:12:17.599+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:12:17.600740+0000) 2024-03-04T11:12:18.595 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:12:18.600+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:12:18.601006+0000) 2024-03-04T11:12:19.595 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:12:19.600+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:12:19.601212+0000) 2024-03-04T11:12:20.595 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:12:20.600+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:12:20.601462+0000) 2024-03-04T11:12:21.595 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:12:21.600+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:12:21.601715+0000) 2024-03-04T11:12:22.596 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:12:22.601+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:12:22.601965+0000) 2024-03-04T11:12:23.596 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:12:23.601+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:12:23.602176+0000) 2024-03-04T11:12:24.596 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:12:24.601+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:12:24.602470+0000) 2024-03-04T11:12:25.596 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:12:25.602+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:12:25.602732+0000) 2024-03-04T11:12:26.597 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:12:26.602+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:12:26.602979+0000) 2024-03-04T11:12:27.597 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:12:27.602+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:12:27.603219+0000) 2024-03-04T11:12:28.597 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:12:28.602+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:12:28.603548+0000) 2024-03-04T11:12:29.597 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:12:29.603+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:12:29.603806+0000) 2024-03-04T11:12:30.598 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:12:30.603+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:12:30.604106+0000) 2024-03-04T11:12:31.598 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:12:31.603+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:12:31.604348+0000) 2024-03-04T11:12:32.598 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:12:32.603+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:12:32.604527+0000) 2024-03-04T11:12:33.598 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:12:33.604+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:12:33.604782+0000) 2024-03-04T11:12:34.599 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:12:34.604+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:12:34.605072+0000) 2024-03-04T11:12:35.599 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:12:35.604+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:12:35.605327+0000) 2024-03-04T11:12:36.599 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:12:36.605+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:12:36.605595+0000) 2024-03-04T11:12:37.600 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:12:37.605+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:12:37.605879+0000) 2024-03-04T11:12:38.600 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:12:38.605+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:12:38.606114+0000) 2024-03-04T11:12:39.600 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:12:39.605+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:12:39.606354+0000) 2024-03-04T11:12:40.600 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:12:40.605+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:12:40.606620+0000) 2024-03-04T11:12:41.600 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:12:41.606+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:12:41.606915+0000) 2024-03-04T11:12:42.601 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:12:42.606+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:12:42.607140+0000) 2024-03-04T11:12:43.601 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:12:43.606+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:12:43.607353+0000) 2024-03-04T11:12:44.601 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:12:44.607+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:12:44.607584+0000) 2024-03-04T11:12:45.601 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:12:45.607+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:12:45.607878+0000) 2024-03-04T11:12:46.499 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:12:46.511 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:12:46.603 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:12:46.607+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:12:46.608161+0000) 2024-03-04T11:12:47.602 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:12:47.608+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:12:47.608434+0000) 2024-03-04T11:12:48.602 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:12:48.608+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:12:48.608694+0000) 2024-03-04T11:12:49.602 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:12:49.608+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:12:49.608933+0000) 2024-03-04T11:12:50.603 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:12:50.608+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:12:50.609166+0000) 2024-03-04T11:12:51.603 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:12:51.609+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:12:51.609412+0000) 2024-03-04T11:12:52.604 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:12:52.609+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:12:52.609698+0000) 2024-03-04T11:12:53.603 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:12:53.609+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:12:53.609937+0000) 2024-03-04T11:12:54.604 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:12:54.609+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:12:54.610169+0000) 2024-03-04T11:12:55.604 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:12:55.609+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:12:55.610354+0000) 2024-03-04T11:12:56.604 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:12:56.609+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:12:56.610623+0000) 2024-03-04T11:12:57.604 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:12:57.610+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:12:57.610808+0000) 2024-03-04T11:12:58.605 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:12:58.610+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:12:58.611033+0000) 2024-03-04T11:12:59.605 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:12:59.610+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:12:59.611284+0000) 2024-03-04T11:13:00.605 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:13:00.610+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:13:00.611599+0000) 2024-03-04T11:13:01.605 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:13:01.610+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:13:01.611926+0000) 2024-03-04T11:13:02.606 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:13:02.611+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:13:02.612244+0000) 2024-03-04T11:13:03.606 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:13:03.610+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:13:03.612574+0000) 2024-03-04T11:13:04.606 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:13:04.611+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:13:04.612912+0000) 2024-03-04T11:13:05.607 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:13:05.612+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:13:05.613204+0000) 2024-03-04T11:13:06.607 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:13:06.612+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:13:06.613511+0000) 2024-03-04T11:13:07.607 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:13:07.612+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:13:07.613883+0000) 2024-03-04T11:13:08.608 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:13:08.613+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:13:08.614205+0000) 2024-03-04T11:13:09.608 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:13:09.613+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:13:09.614509+0000) 2024-03-04T11:13:10.608 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:13:10.613+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:13:10.614793+0000) 2024-03-04T11:13:11.609 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:13:11.614+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:13:11.615137+0000) 2024-03-04T11:13:12.609 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:13:12.614+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:13:12.615426+0000) 2024-03-04T11:13:13.609 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:13:13.614+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:13:13.615710+0000) 2024-03-04T11:13:14.610 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:13:14.615+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:13:14.616030+0000) 2024-03-04T11:13:15.610 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:13:15.615+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:13:15.616336+0000) 2024-03-04T11:13:16.610 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:13:16.615+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:13:16.616639+0000) 2024-03-04T11:13:16.721 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:13:16.729 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:13:17.611 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:13:17.616+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:13:17.616963+0000) 2024-03-04T11:13:18.611 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:13:18.616+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:13:18.617212+0000) 2024-03-04T11:13:19.611 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:13:19.616+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:13:19.617452+0000) 2024-03-04T11:13:20.611 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:13:20.616+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:13:20.617691+0000) 2024-03-04T11:13:21.611 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:13:21.617+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:13:21.617947+0000) 2024-03-04T11:13:22.612 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:13:22.617+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:13:22.618190+0000) 2024-03-04T11:13:23.612 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:13:23.617+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:13:23.618456+0000) 2024-03-04T11:13:24.612 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:13:24.617+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:13:24.618703+0000) 2024-03-04T11:13:25.612 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:13:25.618+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:13:25.618969+0000) 2024-03-04T11:13:26.613 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:13:26.618+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:13:26.619216+0000) 2024-03-04T11:13:27.613 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:13:27.618+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:13:27.619458+0000) 2024-03-04T11:13:28.613 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:13:28.618+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:13:28.619646+0000) 2024-03-04T11:13:29.613 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:13:29.619+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:13:29.619930+0000) 2024-03-04T11:13:30.614 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:13:30.619+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:13:30.620172+0000) 2024-03-04T11:13:31.614 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:13:31.619+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:13:31.620418+0000) 2024-03-04T11:13:32.614 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:13:32.620+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:13:32.620654+0000) 2024-03-04T11:13:33.614 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:13:33.620+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:13:33.620929+0000) 2024-03-04T11:13:34.615 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:13:34.620+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:13:34.621125+0000) 2024-03-04T11:13:35.615 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:13:35.620+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:13:35.621384+0000) 2024-03-04T11:13:36.615 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:13:36.620+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:13:36.621576+0000) 2024-03-04T11:13:37.615 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:13:37.621+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:13:37.621864+0000) 2024-03-04T11:13:38.616 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:13:38.621+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:13:38.622106+0000) 2024-03-04T11:13:39.616 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:13:39.621+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:13:39.622353+0000) 2024-03-04T11:13:40.616 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:13:40.622+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:13:40.622590+0000) 2024-03-04T11:13:41.616 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:13:41.622+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:13:41.622883+0000) 2024-03-04T11:13:42.617 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:13:42.622+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:13:42.623075+0000) 2024-03-04T11:13:43.617 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:13:43.622+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:13:43.623319+0000) 2024-03-04T11:13:44.617 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:13:44.623+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:13:44.623577+0000) 2024-03-04T11:13:45.617 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:13:45.623+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:13:45.623884+0000) 2024-03-04T11:13:46.618 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:13:46.623+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:13:46.624155+0000) 2024-03-04T11:13:46.927 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:13:46.939 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:13:47.618 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:13:47.623+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:13:47.624397+0000) 2024-03-04T11:13:48.618 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:13:48.623+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:13:48.624654+0000) 2024-03-04T11:13:49.618 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:13:49.624+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:13:49.624931+0000) 2024-03-04T11:13:50.619 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:13:50.624+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:13:50.625184+0000) 2024-03-04T11:13:51.619 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:13:51.625+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:13:51.625431+0000) 2024-03-04T11:13:52.619 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:13:52.625+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:13:52.625674+0000) 2024-03-04T11:13:53.619 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:13:53.625+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:13:53.625948+0000) 2024-03-04T11:13:54.620 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:13:54.625+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:13:54.626189+0000) 2024-03-04T11:13:55.620 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:13:55.626+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:13:55.626466+0000) 2024-03-04T11:13:56.620 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:13:56.626+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:13:56.626721+0000) 2024-03-04T11:13:57.620 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:13:57.626+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:13:57.626963+0000) 2024-03-04T11:13:58.621 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:13:58.626+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:13:58.627161+0000) 2024-03-04T11:13:59.621 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:13:59.627+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:13:59.627409+0000) 2024-03-04T11:14:00.621 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:14:00.627+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:14:00.627648+0000) 2024-03-04T11:14:01.621 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:14:01.627+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:14:01.627938+0000) 2024-03-04T11:14:02.622 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:14:02.627+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:14:02.628178+0000) 2024-03-04T11:14:03.622 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:14:03.626+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:14:03.628377+0000) 2024-03-04T11:14:04.622 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:14:04.626+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:14:04.628591+0000) 2024-03-04T11:14:05.622 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:14:05.627+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:14:05.628860+0000) 2024-03-04T11:14:06.623 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:14:06.627+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:14:06.629104+0000) 2024-03-04T11:14:07.623 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:14:07.627+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:14:07.629350+0000) 2024-03-04T11:14:08.623 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:14:08.628+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:14:08.629643+0000) 2024-03-04T11:14:09.623 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:14:09.628+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:14:09.629976+0000) 2024-03-04T11:14:10.624 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:14:10.628+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:14:10.630205+0000) 2024-03-04T11:14:11.624 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:14:11.628+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:14:11.630388+0000) 2024-03-04T11:14:12.624 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:14:12.629+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:14:12.630624+0000) 2024-03-04T11:14:13.624 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:14:13.630+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:14:13.630913+0000) 2024-03-04T11:14:14.625 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:14:14.630+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:14:14.631166+0000) 2024-03-04T11:14:15.625 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:14:15.630+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:14:15.631422+0000) 2024-03-04T11:14:16.625 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:14:16.630+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:14:16.631666+0000) 2024-03-04T11:14:17.258 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:14:17.266 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:14:17.625 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:14:17.631+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:14:17.631941+0000) 2024-03-04T11:14:18.625 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:14:18.631+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:14:18.632174+0000) 2024-03-04T11:14:19.626 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:14:19.631+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:14:19.632423+0000) 2024-03-04T11:14:20.626 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:14:20.631+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:14:20.632671+0000) 2024-03-04T11:14:21.626 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:14:21.632+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:14:21.632969+0000) 2024-03-04T11:14:22.627 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:14:22.632+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:14:22.633231+0000) 2024-03-04T11:14:23.627 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:14:23.632+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:14:23.633533+0000) 2024-03-04T11:14:24.627 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:14:24.633+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:14:24.633763+0000) 2024-03-04T11:14:25.627 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:14:25.633+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:14:25.634010+0000) 2024-03-04T11:14:26.628 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:14:26.633+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:14:26.634254+0000) 2024-03-04T11:14:27.628 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:14:27.633+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:14:27.634508+0000) 2024-03-04T11:14:28.628 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:14:28.634+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:14:28.634759+0000) 2024-03-04T11:14:29.628 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:14:29.634+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:14:29.635014+0000) 2024-03-04T11:14:30.629 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:14:30.634+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:14:30.635260+0000) 2024-03-04T11:14:31.629 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:14:31.634+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:14:31.635539+0000) 2024-03-04T11:14:32.629 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:14:32.634+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:14:32.635766+0000) 2024-03-04T11:14:33.629 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:14:33.635+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:14:33.636011+0000) 2024-03-04T11:14:34.630 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:14:34.635+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:14:34.636246+0000) 2024-03-04T11:14:35.630 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:14:35.635+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:14:35.636484+0000) 2024-03-04T11:14:36.630 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:14:36.635+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:14:36.636761+0000) 2024-03-04T11:14:37.630 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:14:37.636+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:14:37.636995+0000) 2024-03-04T11:14:38.631 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:14:38.636+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:14:38.637245+0000) 2024-03-04T11:14:39.631 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:14:39.636+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:14:39.637497+0000) 2024-03-04T11:14:40.631 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:14:40.637+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:14:40.637749+0000) 2024-03-04T11:14:41.631 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:14:41.637+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:14:41.638017+0000) 2024-03-04T11:14:42.632 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:14:42.637+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:14:42.638270+0000) 2024-03-04T11:14:43.632 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:14:43.638+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:14:43.638525+0000) 2024-03-04T11:14:44.632 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:14:44.638+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:14:44.638793+0000) 2024-03-04T11:14:45.632 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:14:45.638+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:14:45.639082+0000) 2024-03-04T11:14:46.633 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:14:46.638+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:14:46.639346+0000) 2024-03-04T11:14:47.437 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:14:47.445 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:14:47.633 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:14:47.639+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:14:47.639619+0000) 2024-03-04T11:14:48.633 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:14:48.639+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:14:48.639911+0000) 2024-03-04T11:14:49.633 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:14:49.639+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:14:49.640170+0000) 2024-03-04T11:14:50.634 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:14:50.640+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:14:50.640418+0000) 2024-03-04T11:14:51.634 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:14:51.640+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:14:51.640682+0000) 2024-03-04T11:14:52.634 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:14:52.640+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:14:52.640939+0000) 2024-03-04T11:14:53.634 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:14:53.640+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:14:53.641205+0000) 2024-03-04T11:14:54.635 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:14:54.641+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:14:54.641451+0000) 2024-03-04T11:14:55.635 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:14:55.641+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:14:55.641690+0000) 2024-03-04T11:14:56.635 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:14:56.641+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:14:56.641981+0000) 2024-03-04T11:14:57.636 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:14:57.641+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:14:57.642221+0000) 2024-03-04T11:14:58.636 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:14:58.642+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:14:58.642463+0000) 2024-03-04T11:14:59.636 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:14:59.642+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:14:59.642704+0000) 2024-03-04T11:15:00.637 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:15:00.642+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:15:00.642945+0000) 2024-03-04T11:15:01.636 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:15:01.642+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:15:01.643149+0000) 2024-03-04T11:15:02.637 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:15:02.643+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:15:02.643386+0000) 2024-03-04T11:15:03.637 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:15:03.643+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:15:03.643637+0000) 2024-03-04T11:15:04.637 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:15:04.642+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:15:04.643925+0000) 2024-03-04T11:15:05.637 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:15:05.643+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:15:05.644169+0000) 2024-03-04T11:15:06.638 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:15:06.643+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:15:06.644432+0000) 2024-03-04T11:15:07.638 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:15:07.643+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:15:07.644677+0000) 2024-03-04T11:15:08.638 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:15:08.643+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:15:08.644943+0000) 2024-03-04T11:15:09.638 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:15:09.644+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:15:09.645184+0000) 2024-03-04T11:15:10.639 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:15:10.644+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:15:10.645420+0000) 2024-03-04T11:15:11.639 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:15:11.644+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:15:11.645663+0000) 2024-03-04T11:15:12.639 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:15:12.644+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:15:12.645941+0000) 2024-03-04T11:15:13.639 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:15:13.645+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:15:13.646216+0000) 2024-03-04T11:15:14.640 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:15:14.645+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:15:14.646446+0000) 2024-03-04T11:15:15.640 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:15:15.645+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:15:15.646711+0000) 2024-03-04T11:15:16.640 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:15:16.646+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:15:16.646978+0000) 2024-03-04T11:15:17.583 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:15:17.592 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:15:17.640 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:15:17.646+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:15:17.647156+0000) 2024-03-04T11:15:18.641 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:15:18.646+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:15:18.647349+0000) 2024-03-04T11:15:19.641 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:15:19.646+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:15:19.647589+0000) 2024-03-04T11:15:20.641 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:15:20.647+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:15:20.647897+0000) 2024-03-04T11:15:21.641 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:15:21.647+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:15:21.648202+0000) 2024-03-04T11:15:22.642 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:15:22.647+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:15:22.648517+0000) 2024-03-04T11:15:23.642 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:15:23.647+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:15:23.648802+0000) 2024-03-04T11:15:24.642 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:15:24.648+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:15:24.649088+0000) 2024-03-04T11:15:25.643 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:15:25.648+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:15:25.649386+0000) 2024-03-04T11:15:26.643 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:15:26.648+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:15:26.649688+0000) 2024-03-04T11:15:27.643 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:15:27.649+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:15:27.649990+0000) 2024-03-04T11:15:28.644 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:15:28.649+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:15:28.650272+0000) 2024-03-04T11:15:29.644 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:15:29.649+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:15:29.650574+0000) 2024-03-04T11:15:30.644 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:15:30.650+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:15:30.650907+0000) 2024-03-04T11:15:31.645 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:15:31.650+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:15:31.651228+0000) 2024-03-04T11:15:32.645 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:15:32.650+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:15:32.651509+0000) 2024-03-04T11:15:33.645 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:15:33.650+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:15:33.651732+0000) 2024-03-04T11:15:34.645 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:15:34.651+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:15:34.651978+0000) 2024-03-04T11:15:35.645 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:15:35.651+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:15:35.652218+0000) 2024-03-04T11:15:36.646 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:15:36.651+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:15:36.652479+0000) 2024-03-04T11:15:37.646 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:15:37.651+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:15:37.652728+0000) 2024-03-04T11:15:38.646 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:15:38.652+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:15:38.652972+0000) 2024-03-04T11:15:39.646 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:15:39.652+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:15:39.653233+0000) 2024-03-04T11:15:40.647 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:15:40.652+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:15:40.653468+0000) 2024-03-04T11:15:41.647 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:15:41.653+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:15:41.653760+0000) 2024-03-04T11:15:42.647 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:15:42.653+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:15:42.654075+0000) 2024-03-04T11:15:43.648 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:15:43.653+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:15:43.654367+0000) 2024-03-04T11:15:44.648 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:15:44.654+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:15:44.654658+0000) 2024-03-04T11:15:45.648 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:15:45.654+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:15:45.654985+0000) 2024-03-04T11:15:46.651 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:15:46.654+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:15:46.655289+0000) 2024-03-04T11:15:47.649 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:15:47.654+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:15:47.655534+0000) 2024-03-04T11:15:47.845 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:15:47.853 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:15:48.649 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:15:48.655+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:15:48.655870+0000) 2024-03-04T11:15:49.649 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:15:49.655+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:15:49.656164+0000) 2024-03-04T11:15:50.650 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:15:50.656+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:15:50.656448+0000) 2024-03-04T11:15:51.650 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:15:51.655+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:15:51.656756+0000) 2024-03-04T11:15:52.650 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:15:52.656+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:15:52.657040+0000) 2024-03-04T11:15:53.650 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:15:53.656+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:15:53.657325+0000) 2024-03-04T11:15:54.651 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:15:54.657+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:15:54.657607+0000) 2024-03-04T11:15:55.651 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:15:55.657+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:15:55.657925+0000) 2024-03-04T11:15:56.651 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:15:56.657+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:15:56.658181+0000) 2024-03-04T11:15:57.652 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:15:57.657+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:15:57.658417+0000) 2024-03-04T11:15:58.652 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:15:58.658+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:15:58.658666+0000) 2024-03-04T11:15:59.652 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:15:59.658+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:15:59.658923+0000) 2024-03-04T11:16:00.652 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:16:00.658+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:16:00.659155+0000) 2024-03-04T11:16:01.653 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:16:01.659+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:16:01.659413+0000) 2024-03-04T11:16:02.653 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:16:02.659+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:16:02.659677+0000) 2024-03-04T11:16:03.653 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:16:03.659+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:16:03.659939+0000) 2024-03-04T11:16:04.653 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:16:04.660+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:16:04.660184+0000) 2024-03-04T11:16:05.654 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:16:05.659+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:16:05.660375+0000) 2024-03-04T11:16:06.654 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:16:06.660+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:16:06.660615+0000) 2024-03-04T11:16:07.654 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:16:07.659+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:16:07.660921+0000) 2024-03-04T11:16:08.654 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:16:08.660+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:16:08.661157+0000) 2024-03-04T11:16:09.654 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:16:09.659+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:16:09.661353+0000) 2024-03-04T11:16:10.655 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:16:10.660+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:16:10.661606+0000) 2024-03-04T11:16:11.655 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:16:11.660+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:16:11.661908+0000) 2024-03-04T11:16:12.655 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:16:12.661+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:16:12.662150+0000) 2024-03-04T11:16:13.656 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:16:13.661+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:16:13.662387+0000) 2024-03-04T11:16:14.656 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:16:14.661+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:16:14.662618+0000) 2024-03-04T11:16:15.656 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:16:15.661+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:16:15.662945+0000) 2024-03-04T11:16:16.656 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:16:16.662+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:16:16.663221+0000) 2024-03-04T11:16:17.657 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:16:17.662+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:16:17.663488+0000) 2024-03-04T11:16:18.044 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:16:18.052 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:16:18.657 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:16:18.662+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:16:18.663736+0000) 2024-03-04T11:16:19.657 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:16:19.663+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:16:19.664025+0000) 2024-03-04T11:16:20.658 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:16:20.663+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:16:20.664282+0000) 2024-03-04T11:16:21.658 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:16:21.663+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:16:21.664483+0000) 2024-03-04T11:16:22.658 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:16:22.663+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:16:22.664739+0000) 2024-03-04T11:16:23.658 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:16:23.664+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:16:23.664983+0000) 2024-03-04T11:16:24.658 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:16:24.664+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:16:24.665217+0000) 2024-03-04T11:16:25.659 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:16:25.664+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:16:25.665461+0000) 2024-03-04T11:16:26.659 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:16:26.664+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:16:26.665705+0000) 2024-03-04T11:16:27.659 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:16:27.665+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:16:27.665955+0000) 2024-03-04T11:16:28.659 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:16:28.665+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:16:28.666186+0000) 2024-03-04T11:16:29.659 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:16:29.665+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:16:29.666427+0000) 2024-03-04T11:16:30.660 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:16:30.665+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:16:30.666663+0000) 2024-03-04T11:16:31.660 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:16:31.666+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:16:31.666933+0000) 2024-03-04T11:16:32.660 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:16:32.666+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:16:32.667163+0000) 2024-03-04T11:16:33.661 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:16:33.666+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:16:33.667400+0000) 2024-03-04T11:16:34.661 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:16:34.666+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:16:34.667635+0000) 2024-03-04T11:16:35.661 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:16:35.667+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:16:35.667913+0000) 2024-03-04T11:16:36.661 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:16:36.667+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:16:36.668159+0000) 2024-03-04T11:16:37.661 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:16:37.667+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:16:37.668391+0000) 2024-03-04T11:16:38.662 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:16:38.668+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:16:38.668624+0000) 2024-03-04T11:16:39.662 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:16:39.668+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:16:39.668932+0000) 2024-03-04T11:16:40.662 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:16:40.668+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:16:40.669186+0000) 2024-03-04T11:16:41.663 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:16:41.668+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:16:41.669451+0000) 2024-03-04T11:16:42.663 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:16:42.669+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:16:42.669689+0000) 2024-03-04T11:16:43.663 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:16:43.669+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:16:43.669933+0000) 2024-03-04T11:16:44.663 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:16:44.669+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:16:44.670185+0000) 2024-03-04T11:16:45.663 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:16:45.669+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:16:45.670420+0000) 2024-03-04T11:16:46.664 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:16:46.670+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:16:46.670667+0000) 2024-03-04T11:16:47.664 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:16:47.670+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:16:47.670946+0000) 2024-03-04T11:16:48.241 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:16:48.249 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:16:48.664 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:16:48.670+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:16:48.671182+0000) 2024-03-04T11:16:49.665 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:16:49.670+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:16:49.671419+0000) 2024-03-04T11:16:50.665 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:16:50.670+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:16:50.671678+0000) 2024-03-04T11:16:51.665 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:16:51.671+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:16:51.671953+0000) 2024-03-04T11:16:52.665 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:16:52.671+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:16:52.672181+0000) 2024-03-04T11:16:53.665 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:16:53.672+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:16:53.672457+0000) 2024-03-04T11:16:54.666 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:16:54.672+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:16:54.672688+0000) 2024-03-04T11:16:55.666 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:16:55.672+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:16:55.672930+0000) 2024-03-04T11:16:56.666 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:16:56.672+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:16:56.673191+0000) 2024-03-04T11:16:57.667 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:16:57.673+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:16:57.673457+0000) 2024-03-04T11:16:58.667 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:16:58.673+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:16:58.673704+0000) 2024-03-04T11:16:59.667 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:16:59.673+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:16:59.673948+0000) 2024-03-04T11:17:00.667 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:17:00.673+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:17:00.674188+0000) 2024-03-04T11:17:01.668 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:17:01.673+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:17:01.674448+0000) 2024-03-04T11:17:02.668 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:17:02.674+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:17:02.674688+0000) 2024-03-04T11:17:03.668 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:17:03.674+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:17:03.674956+0000) 2024-03-04T11:17:04.668 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:17:04.674+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:17:04.675186+0000) 2024-03-04T11:17:05.669 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:17:05.675+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:17:05.675430+0000) 2024-03-04T11:17:06.669 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:17:06.675+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:17:06.675677+0000) 2024-03-04T11:17:07.669 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:17:07.675+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:17:07.675954+0000) 2024-03-04T11:17:08.669 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:17:08.676+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:17:08.676202+0000) 2024-03-04T11:17:09.669 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:17:09.676+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:17:09.676448+0000) 2024-03-04T11:17:10.670 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:17:10.675+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:17:10.676706+0000) 2024-03-04T11:17:11.670 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:17:11.675+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:17:11.676984+0000) 2024-03-04T11:17:12.670 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:17:12.675+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:17:12.677178+0000) 2024-03-04T11:17:13.670 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:17:13.676+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:17:13.677492+0000) 2024-03-04T11:17:14.671 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:17:14.676+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:17:14.677732+0000) 2024-03-04T11:17:15.671 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:17:15.676+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:17:15.677976+0000) 2024-03-04T11:17:16.671 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:17:16.677+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:17:16.678226+0000) 2024-03-04T11:17:17.671 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:17:17.677+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:17:17.678471+0000) 2024-03-04T11:17:18.413 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:17:18.421 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:17:18.672 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:17:18.677+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:17:18.678744+0000) 2024-03-04T11:17:19.672 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:17:19.677+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:17:19.678948+0000) 2024-03-04T11:17:20.672 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:17:20.677+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:17:20.679223+0000) 2024-03-04T11:17:21.673 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:17:21.678+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:17:21.679471+0000) 2024-03-04T11:17:22.673 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:17:22.678+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:17:22.679706+0000) 2024-03-04T11:17:23.673 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:17:23.679+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:17:23.679960+0000) 2024-03-04T11:17:24.673 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:17:24.679+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:17:24.680158+0000) 2024-03-04T11:17:25.673 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:17:25.679+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:17:25.680401+0000) 2024-03-04T11:17:26.674 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:17:26.679+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:17:26.680646+0000) 2024-03-04T11:17:27.674 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:17:27.680+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:17:27.680932+0000) 2024-03-04T11:17:28.674 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:17:28.680+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:17:28.681171+0000) 2024-03-04T11:17:29.674 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:17:29.680+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:17:29.681372+0000) 2024-03-04T11:17:30.675 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:17:30.680+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:17:30.681607+0000) 2024-03-04T11:17:31.675 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:17:31.681+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:17:31.681878+0000) 2024-03-04T11:17:32.675 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:17:32.681+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:17:32.682077+0000) 2024-03-04T11:17:33.675 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:17:33.681+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:17:33.682287+0000) 2024-03-04T11:17:34.676 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:17:34.681+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:17:34.682483+0000) 2024-03-04T11:17:35.676 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:17:35.682+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:17:35.682741+0000) 2024-03-04T11:17:36.676 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:17:36.682+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:17:36.683002+0000) 2024-03-04T11:17:37.676 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:17:37.682+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:17:37.683254+0000) 2024-03-04T11:17:38.677 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:17:38.682+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:17:38.683530+0000) 2024-03-04T11:17:39.677 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:17:39.683+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:17:39.683768+0000) 2024-03-04T11:17:40.677 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:17:40.683+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:17:40.684033+0000) 2024-03-04T11:17:41.677 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:17:41.683+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:17:41.684288+0000) 2024-03-04T11:17:42.678 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:17:42.683+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:17:42.684518+0000) 2024-03-04T11:17:43.678 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:17:43.683+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:17:43.684756+0000) 2024-03-04T11:17:44.678 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:17:44.684+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:17:44.684995+0000) 2024-03-04T11:17:45.678 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:17:45.684+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:17:45.685249+0000) 2024-03-04T11:17:46.678 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:17:46.684+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:17:46.685512+0000) 2024-03-04T11:17:47.679 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:17:47.685+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:17:47.685767+0000) 2024-03-04T11:17:48.644 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:17:48.652 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:17:48.679 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:17:48.685+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:17:48.685992+0000) 2024-03-04T11:17:49.679 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:17:49.685+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:17:49.686249+0000) 2024-03-04T11:17:50.679 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:17:50.686+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:17:50.686495+0000) 2024-03-04T11:17:51.680 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:17:51.686+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:17:51.686752+0000) 2024-03-04T11:17:52.680 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:17:52.686+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:17:52.687000+0000) 2024-03-04T11:17:53.680 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:17:53.686+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:17:53.687257+0000) 2024-03-04T11:17:54.680 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:17:54.687+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:17:54.687511+0000) 2024-03-04T11:17:55.681 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:17:55.687+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:17:55.687767+0000) 2024-03-04T11:17:56.681 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:17:56.687+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:17:56.688048+0000) 2024-03-04T11:17:57.681 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:17:57.687+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:17:57.688301+0000) 2024-03-04T11:17:58.682 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:17:58.688+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:17:58.688552+0000) 2024-03-04T11:17:59.682 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:17:59.688+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:17:59.688806+0000) 2024-03-04T11:18:00.682 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:18:00.688+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:18:00.689097+0000) 2024-03-04T11:18:01.682 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:18:01.689+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:18:01.689408+0000) 2024-03-04T11:18:02.683 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:18:02.689+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:18:02.689655+0000) 2024-03-04T11:18:03.683 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:18:03.689+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:18:03.689961+0000) 2024-03-04T11:18:04.683 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:18:04.689+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:18:04.690218+0000) 2024-03-04T11:18:05.683 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:18:05.690+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:18:05.690472+0000) 2024-03-04T11:18:06.684 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:18:06.690+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:18:06.690729+0000) 2024-03-04T11:18:07.684 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:18:07.690+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:18:07.690954+0000) 2024-03-04T11:18:08.687 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:18:08.691+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:18:08.691209+0000) 2024-03-04T11:18:09.684 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:18:09.691+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:18:09.691466+0000) 2024-03-04T11:18:10.685 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:18:10.691+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:18:10.691736+0000) 2024-03-04T11:18:11.685 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:18:11.691+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:18:11.692009+0000) 2024-03-04T11:18:12.685 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:18:12.692+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:18:12.692264+0000) 2024-03-04T11:18:13.685 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:18:13.691+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:18:13.692517+0000) 2024-03-04T11:18:14.686 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:18:14.691+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:18:14.692769+0000) 2024-03-04T11:18:15.686 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:18:15.691+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:18:15.693031+0000) 2024-03-04T11:18:16.687 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:18:16.692+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:18:16.693295+0000) 2024-03-04T11:18:17.686 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:18:17.692+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:18:17.693535+0000) 2024-03-04T11:18:18.687 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:18:18.692+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:18:18.693770+0000) 2024-03-04T11:18:18.940 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:18:18.948 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:18:19.687 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:18:19.692+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:18:19.694022+0000) 2024-03-04T11:18:20.687 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:18:20.693+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:18:20.694278+0000) 2024-03-04T11:18:21.687 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:18:21.693+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:18:21.694535+0000) 2024-03-04T11:18:22.688 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:18:22.693+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:18:22.694795+0000) 2024-03-04T11:18:23.688 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:18:23.694+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:18:23.695097+0000) 2024-03-04T11:18:24.688 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:18:24.694+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:18:24.695291+0000) 2024-03-04T11:18:25.689 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:18:25.694+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:18:25.695542+0000) 2024-03-04T11:18:26.689 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:18:26.694+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:18:26.695801+0000) 2024-03-04T11:18:27.689 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:18:27.695+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:18:27.696098+0000) 2024-03-04T11:18:28.689 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:18:28.695+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:18:28.696353+0000) 2024-03-04T11:18:29.689 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:18:29.695+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:18:29.696601+0000) 2024-03-04T11:18:30.690 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:18:30.696+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:18:30.696886+0000) 2024-03-04T11:18:31.690 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:18:31.696+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:18:31.697140+0000) 2024-03-04T11:18:32.690 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:18:32.696+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:18:32.697393+0000) 2024-03-04T11:18:33.691 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:18:33.696+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:18:33.697676+0000) 2024-03-04T11:18:34.691 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:18:34.697+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:18:34.697926+0000) 2024-03-04T11:18:35.691 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:18:35.697+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:18:35.698185+0000) 2024-03-04T11:18:36.691 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:18:36.697+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:18:36.698442+0000) 2024-03-04T11:18:37.691 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:18:37.697+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:18:37.698677+0000) 2024-03-04T11:18:38.692 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:18:38.698+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:18:38.698952+0000) 2024-03-04T11:18:39.692 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:18:39.698+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:18:39.699223+0000) 2024-03-04T11:18:40.692 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:18:40.698+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:18:40.699473+0000) 2024-03-04T11:18:41.693 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:18:41.698+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:18:41.699664+0000) 2024-03-04T11:18:42.693 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:18:42.699+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:18:42.699891+0000) 2024-03-04T11:18:43.693 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:18:43.699+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:18:43.700134+0000) 2024-03-04T11:18:44.694 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:18:44.699+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:18:44.700371+0000) 2024-03-04T11:18:45.693 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:18:45.700+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:18:45.700610+0000) 2024-03-04T11:18:46.694 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:18:46.700+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:18:46.700888+0000) 2024-03-04T11:18:47.694 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:18:47.700+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:18:47.701126+0000) 2024-03-04T11:18:48.694 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:18:48.700+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:18:48.701359+0000) 2024-03-04T11:18:49.100 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:18:49.108 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:18:49.694 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:18:49.700+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:18:49.701602+0000) 2024-03-04T11:18:50.695 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:18:50.701+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:18:50.701868+0000) 2024-03-04T11:18:51.695 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:18:51.701+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:18:51.702109+0000) 2024-03-04T11:18:52.695 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:18:52.701+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:18:52.702339+0000) 2024-03-04T11:18:53.695 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:18:53.702+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:18:53.702580+0000) 2024-03-04T11:18:54.696 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:18:54.702+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:18:54.702812+0000) 2024-03-04T11:18:55.696 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:18:55.702+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:18:55.703142+0000) 2024-03-04T11:18:56.696 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:18:56.702+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:18:56.703382+0000) 2024-03-04T11:18:57.696 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:18:57.703+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:18:57.703643+0000) 2024-03-04T11:18:58.697 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:18:58.703+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:18:58.703920+0000) 2024-03-04T11:18:59.697 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:18:59.703+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:18:59.704164+0000) 2024-03-04T11:19:00.697 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:19:00.704+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:19:00.704399+0000) 2024-03-04T11:19:01.697 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:19:01.704+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:19:01.704633+0000) 2024-03-04T11:19:02.698 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:19:02.704+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:19:02.704915+0000) 2024-03-04T11:19:03.698 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:19:03.704+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:19:03.705160+0000) 2024-03-04T11:19:04.698 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:19:04.705+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:19:04.705397+0000) 2024-03-04T11:19:05.699 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:19:05.705+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:19:05.705654+0000) 2024-03-04T11:19:06.699 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:19:06.705+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:19:06.705953+0000) 2024-03-04T11:19:07.699 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:19:07.705+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:19:07.706209+0000) 2024-03-04T11:19:08.699 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:19:08.706+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:19:08.706462+0000) 2024-03-04T11:19:09.700 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:19:09.705+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:19:09.706722+0000) 2024-03-04T11:19:10.700 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:19:10.706+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:19:10.706968+0000) 2024-03-04T11:19:11.700 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:19:11.707+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:19:11.707249+0000) 2024-03-04T11:19:12.700 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:19:12.707+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:19:12.707507+0000) 2024-03-04T11:19:13.701 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:19:13.707+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:19:13.707762+0000) 2024-03-04T11:19:14.701 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:19:14.707+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:19:14.708018+0000) 2024-03-04T11:19:15.701 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:19:15.707+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:19:15.708275+0000) 2024-03-04T11:19:16.701 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:19:16.707+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:19:16.708537+0000) 2024-03-04T11:19:17.702 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:19:17.707+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:19:17.708783+0000) 2024-03-04T11:19:18.702 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:19:18.708+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:19:18.709072+0000) 2024-03-04T11:19:19.239 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:19:19.247 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:19:19.703 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:19:19.708+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:19:19.709346+0000) 2024-03-04T11:19:20.703 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:19:20.708+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:19:20.709601+0000) 2024-03-04T11:19:21.703 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:19:21.708+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:19:21.709892+0000) 2024-03-04T11:19:22.703 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:19:22.709+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:19:22.710155+0000) 2024-03-04T11:19:23.703 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:19:23.709+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:19:23.710418+0000) 2024-03-04T11:19:24.704 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:19:24.709+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:19:24.710673+0000) 2024-03-04T11:19:25.704 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:19:25.709+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:19:25.710904+0000) 2024-03-04T11:19:26.704 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:19:26.710+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:19:26.711166+0000) 2024-03-04T11:19:27.704 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:19:27.710+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:19:27.711438+0000) 2024-03-04T11:19:28.704 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:19:28.710+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:19:28.711690+0000) 2024-03-04T11:19:29.705 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:19:29.711+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:19:29.711945+0000) 2024-03-04T11:19:30.705 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:19:30.711+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:19:30.712149+0000) 2024-03-04T11:19:31.705 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:19:31.711+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:19:31.712405+0000) 2024-03-04T11:19:32.705 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:19:32.711+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:19:32.712611+0000) 2024-03-04T11:19:33.706 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:19:33.712+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:19:33.712920+0000) 2024-03-04T11:19:34.706 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:19:34.712+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:19:34.713171+0000) 2024-03-04T11:19:35.706 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:19:35.712+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:19:35.713423+0000) 2024-03-04T11:19:36.707 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:19:36.712+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:19:36.713682+0000) 2024-03-04T11:19:37.707 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:19:37.713+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:19:37.713951+0000) 2024-03-04T11:19:38.707 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:19:38.713+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:19:38.714202+0000) 2024-03-04T11:19:39.707 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:19:39.713+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:19:39.714478+0000) 2024-03-04T11:19:40.707 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:19:40.714+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:19:40.714744+0000) 2024-03-04T11:19:41.708 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:19:41.714+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:19:41.714997+0000) 2024-03-04T11:19:42.708 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:19:42.714+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:19:42.715259+0000) 2024-03-04T11:19:43.708 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:19:43.714+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:19:43.715548+0000) 2024-03-04T11:19:44.709 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:19:44.715+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:19:44.715796+0000) 2024-03-04T11:19:45.709 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:19:45.715+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:19:45.716094+0000) 2024-03-04T11:19:46.709 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:19:46.715+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:19:46.716339+0000) 2024-03-04T11:19:47.709 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:19:47.715+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:19:47.716595+0000) 2024-03-04T11:19:48.710 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:19:48.716+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:19:48.716889+0000) 2024-03-04T11:19:49.472 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:19:49.480 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:19:49.710 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:19:49.716+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:19:49.717166+0000) 2024-03-04T11:19:50.710 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:19:50.716+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:19:50.717400+0000) 2024-03-04T11:19:51.710 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:19:51.717+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:19:51.717660+0000) 2024-03-04T11:19:52.711 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:19:52.717+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:19:52.717943+0000) 2024-03-04T11:19:53.711 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:19:53.717+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:19:53.718186+0000) 2024-03-04T11:19:54.711 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:19:54.717+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:19:54.718423+0000) 2024-03-04T11:19:55.712 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:19:55.718+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:19:55.718673+0000) 2024-03-04T11:19:56.712 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:19:56.718+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:19:56.718954+0000) 2024-03-04T11:19:57.712 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:19:57.718+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:19:57.719144+0000) 2024-03-04T11:19:58.712 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:19:58.718+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:19:58.719384+0000) 2024-03-04T11:19:59.712 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:19:59.719+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:19:59.719619+0000) 2024-03-04T11:20:00.713 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:20:00.718+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:20:00.719798+0000) 2024-03-04T11:20:01.713 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:20:01.719+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:20:01.720080+0000) 2024-03-04T11:20:02.713 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:20:02.719+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:20:02.720272+0000) 2024-03-04T11:20:03.713 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:20:03.720+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:20:03.720515+0000) 2024-03-04T11:20:04.713 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:20:04.720+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:20:04.720748+0000) 2024-03-04T11:20:05.714 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:20:05.720+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:20:05.720983+0000) 2024-03-04T11:20:06.714 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:20:06.720+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:20:06.721228+0000) 2024-03-04T11:20:07.714 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:20:07.721+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:20:07.721472+0000) 2024-03-04T11:20:08.715 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:20:08.721+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:20:08.721707+0000) 2024-03-04T11:20:09.715 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:20:09.721+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:20:09.721966+0000) 2024-03-04T11:20:10.715 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:20:10.721+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:20:10.722221+0000) 2024-03-04T11:20:11.715 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:20:11.722+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:20:11.722467+0000) 2024-03-04T11:20:12.716 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:20:12.721+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:20:12.722691+0000) 2024-03-04T11:20:13.716 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:20:13.722+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:20:13.722938+0000) 2024-03-04T11:20:14.716 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:20:14.723+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:20:14.723199+0000) 2024-03-04T11:20:15.716 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:20:15.723+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:20:15.723443+0000) 2024-03-04T11:20:16.716 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:20:16.723+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:20:16.723686+0000) 2024-03-04T11:20:17.717 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:20:17.723+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:20:17.723920+0000) 2024-03-04T11:20:18.717 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:20:18.723+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:20:18.724167+0000) 2024-03-04T11:20:19.635 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:20:19.644 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:20:19.717 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:20:19.722+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:20:19.724368+0000) 2024-03-04T11:20:20.717 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:20:20.723+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:20:20.724601+0000) 2024-03-04T11:20:21.718 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:20:21.723+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:20:21.724895+0000) 2024-03-04T11:20:22.718 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:20:22.723+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:20:22.725099+0000) 2024-03-04T11:20:23.718 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:20:23.724+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:20:23.725362+0000) 2024-03-04T11:20:24.718 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:20:24.724+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:20:24.725610+0000) 2024-03-04T11:20:25.719 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:20:25.724+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:20:25.725913+0000) 2024-03-04T11:20:26.719 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:20:26.725+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:20:26.726212+0000) 2024-03-04T11:20:27.719 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:20:27.725+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:20:27.726457+0000) 2024-03-04T11:20:28.719 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:20:28.725+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:20:28.726699+0000) 2024-03-04T11:20:29.720 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:20:29.726+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:20:29.726938+0000) 2024-03-04T11:20:30.720 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:20:30.726+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:20:30.727193+0000) 2024-03-04T11:20:31.720 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:20:31.726+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:20:31.727433+0000) 2024-03-04T11:20:32.720 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:20:32.726+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:20:32.727699+0000) 2024-03-04T11:20:33.721 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:20:33.727+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:20:33.727949+0000) 2024-03-04T11:20:34.721 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:20:34.727+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:20:34.728189+0000) 2024-03-04T11:20:35.721 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:20:35.727+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:20:35.728425+0000) 2024-03-04T11:20:36.721 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:20:36.727+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:20:36.728668+0000) 2024-03-04T11:20:37.722 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:20:37.728+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:20:37.728938+0000) 2024-03-04T11:20:38.722 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:20:38.728+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:20:38.729187+0000) 2024-03-04T11:20:39.722 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:20:39.728+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:20:39.729455+0000) 2024-03-04T11:20:40.722 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:20:40.728+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:20:40.729636+0000) 2024-03-04T11:20:41.723 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:20:41.729+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:20:41.729934+0000) 2024-03-04T11:20:42.723 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:20:42.729+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:20:42.730174+0000) 2024-03-04T11:20:43.723 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:20:43.729+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:20:43.730363+0000) 2024-03-04T11:20:44.723 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:20:44.729+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:20:44.730619+0000) 2024-03-04T11:20:45.723 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:20:45.730+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:20:45.730891+0000) 2024-03-04T11:20:46.724 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:20:46.730+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:20:46.731132+0000) 2024-03-04T11:20:47.724 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:20:47.730+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:20:47.731378+0000) 2024-03-04T11:20:48.724 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:20:48.730+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:20:48.731579+0000) 2024-03-04T11:20:49.724 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:20:49.731+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:20:49.731857+0000) 2024-03-04T11:20:49.840 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:20:49.849 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:20:50.725 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:20:50.731+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:20:50.732103+0000) 2024-03-04T11:20:51.725 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:20:51.731+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:20:51.732352+0000) 2024-03-04T11:20:52.725 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:20:52.732+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:20:52.732605+0000) 2024-03-04T11:20:53.725 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:20:53.732+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:20:53.732878+0000) 2024-03-04T11:20:54.726 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:20:54.732+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:20:54.733116+0000) 2024-03-04T11:20:55.726 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:20:55.732+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:20:55.733357+0000) 2024-03-04T11:20:56.726 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:20:56.733+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:20:56.733603+0000) 2024-03-04T11:20:57.726 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:20:57.733+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:20:57.733881+0000) 2024-03-04T11:20:58.727 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:20:58.733+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:20:58.734132+0000) 2024-03-04T11:20:59.727 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:20:59.733+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:20:59.734410+0000) 2024-03-04T11:21:00.727 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:21:00.734+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:21:00.734653+0000) 2024-03-04T11:21:01.728 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:21:01.734+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:21:01.734939+0000) 2024-03-04T11:21:02.728 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:21:02.734+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:21:02.735171+0000) 2024-03-04T11:21:03.728 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:21:03.735+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:21:03.735426+0000) 2024-03-04T11:21:04.728 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:21:04.735+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:21:04.735667+0000) 2024-03-04T11:21:05.729 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:21:05.735+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:21:05.735947+0000) 2024-03-04T11:21:06.729 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:21:06.735+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:21:06.736196+0000) 2024-03-04T11:21:07.729 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:21:07.736+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:21:07.736437+0000) 2024-03-04T11:21:08.729 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:21:08.735+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:21:08.736632+0000) 2024-03-04T11:21:09.729 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:21:09.736+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:21:09.736926+0000) 2024-03-04T11:21:10.730 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:21:10.736+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:21:10.737179+0000) 2024-03-04T11:21:11.730 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:21:11.737+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:21:11.737441+0000) 2024-03-04T11:21:12.730 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:21:12.737+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:21:12.737682+0000) 2024-03-04T11:21:13.731 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:21:13.737+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:21:13.737950+0000) 2024-03-04T11:21:14.731 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:21:14.737+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:21:14.738148+0000) 2024-03-04T11:21:15.731 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:21:15.738+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:21:15.738388+0000) 2024-03-04T11:21:16.731 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:21:16.737+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:21:16.738649+0000) 2024-03-04T11:21:17.732 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:21:17.738+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:21:17.738928+0000) 2024-03-04T11:21:18.732 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:21:18.738+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:21:18.739117+0000) 2024-03-04T11:21:19.732 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:21:19.738+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:21:19.739356+0000) 2024-03-04T11:21:20.053 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:21:20.063 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:21:20.732 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:21:20.739+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:21:20.739599+0000) 2024-03-04T11:21:21.732 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:21:21.737+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:21:21.739802+0000) 2024-03-04T11:21:22.733 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:21:22.738+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:21:22.740081+0000) 2024-03-04T11:21:23.733 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:21:23.738+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:21:23.740327+0000) 2024-03-04T11:21:24.733 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:21:24.739+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:21:24.740547+0000) 2024-03-04T11:21:25.733 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:21:25.739+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:21:25.740798+0000) 2024-03-04T11:21:26.734 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:21:26.740+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:21:26.741124+0000) 2024-03-04T11:21:27.734 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:21:27.740+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:21:27.741383+0000) 2024-03-04T11:21:28.734 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:21:28.739+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:21:28.741647+0000) 2024-03-04T11:21:29.734 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:21:29.740+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:21:29.741944+0000) 2024-03-04T11:21:30.735 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:21:30.741+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:21:30.742181+0000) 2024-03-04T11:21:31.735 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:21:31.741+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:21:31.742422+0000) 2024-03-04T11:21:32.735 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:21:32.741+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:21:32.742661+0000) 2024-03-04T11:21:33.735 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:21:33.742+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:21:33.742939+0000) 2024-03-04T11:21:34.736 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:21:34.742+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:21:34.743189+0000) 2024-03-04T11:21:35.736 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:21:35.742+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:21:35.743523+0000) 2024-03-04T11:21:36.736 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:21:36.742+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:21:36.743782+0000) 2024-03-04T11:21:37.737 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:21:37.743+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:21:37.744023+0000) 2024-03-04T11:21:38.737 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:21:38.743+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:21:38.744264+0000) 2024-03-04T11:21:39.737 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:21:39.743+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:21:39.744515+0000) 2024-03-04T11:21:40.737 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:21:40.743+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:21:40.744775+0000) 2024-03-04T11:21:41.737 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:21:41.744+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:21:41.745042+0000) 2024-03-04T11:21:42.738 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:21:42.744+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:21:42.745276+0000) 2024-03-04T11:21:43.738 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:21:43.744+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:21:43.745532+0000) 2024-03-04T11:21:44.738 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:21:44.745+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:21:44.745783+0000) 2024-03-04T11:21:45.739 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:21:45.745+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:21:45.746022+0000) 2024-03-04T11:21:46.739 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:21:46.745+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:21:46.746227+0000) 2024-03-04T11:21:47.739 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:21:47.745+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:21:47.746475+0000) 2024-03-04T11:21:48.739 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:21:48.745+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:21:48.746734+0000) 2024-03-04T11:21:49.740 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:21:49.746+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:21:49.746975+0000) 2024-03-04T11:21:50.250 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:21:50.258 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:21:50.740 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:21:50.746+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:21:50.747219+0000) 2024-03-04T11:21:51.740 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:21:51.746+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:21:51.747466+0000) 2024-03-04T11:21:52.740 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:21:52.747+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:21:52.747705+0000) 2024-03-04T11:21:53.740 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:21:53.747+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:21:53.747956+0000) 2024-03-04T11:21:54.741 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:21:54.747+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:21:54.748189+0000) 2024-03-04T11:21:55.741 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:21:55.747+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:21:55.748431+0000) 2024-03-04T11:21:56.741 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:21:56.748+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:21:56.748681+0000) 2024-03-04T11:21:57.741 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:21:57.748+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:21:57.748958+0000) 2024-03-04T11:21:58.742 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:21:58.748+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:21:58.749201+0000) 2024-03-04T11:21:59.742 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:21:59.748+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:21:59.749481+0000) 2024-03-04T11:22:00.742 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:22:00.748+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:22:00.749726+0000) 2024-03-04T11:22:01.742 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:22:01.749+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:22:01.749982+0000) 2024-03-04T11:22:02.743 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:22:02.749+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:22:02.750200+0000) 2024-03-04T11:22:03.743 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:22:03.749+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:22:03.750443+0000) 2024-03-04T11:22:04.743 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:22:04.750+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:22:04.750679+0000) 2024-03-04T11:22:05.743 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:22:05.750+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:22:05.750926+0000) 2024-03-04T11:22:06.744 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:22:06.750+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:22:06.751213+0000) 2024-03-04T11:22:07.744 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:22:07.751+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:22:07.751456+0000) 2024-03-04T11:22:08.744 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:22:08.750+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:22:08.751793+0000) 2024-03-04T11:22:09.744 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:22:09.751+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:22:09.752106+0000) 2024-03-04T11:22:10.744 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:22:10.751+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:22:10.752334+0000) 2024-03-04T11:22:11.745 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:22:11.752+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:22:11.752582+0000) 2024-03-04T11:22:12.745 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:22:12.752+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:22:12.752817+0000) 2024-03-04T11:22:13.745 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:22:13.752+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:22:13.753102+0000) 2024-03-04T11:22:14.745 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:22:14.753+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:22:14.753335+0000) 2024-03-04T11:22:15.746 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:22:15.753+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:22:15.753573+0000) 2024-03-04T11:22:16.746 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:22:16.753+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:22:16.753814+0000) 2024-03-04T11:22:17.746 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:22:17.753+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:22:17.754095+0000) 2024-03-04T11:22:18.746 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:22:18.754+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:22:18.754328+0000) 2024-03-04T11:22:19.747 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:22:19.754+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:22:19.754572+0000) 2024-03-04T11:22:20.445 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:22:20.453 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:22:20.747 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:22:20.753+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:22:20.754763+0000) 2024-03-04T11:22:21.747 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:22:21.754+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:22:21.755021+0000) 2024-03-04T11:22:22.747 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:22:22.754+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:22:22.755243+0000) 2024-03-04T11:22:23.747 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:22:23.755+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:22:23.755510+0000) 2024-03-04T11:22:24.748 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:22:24.754+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:22:24.755745+0000) 2024-03-04T11:22:25.748 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:22:25.754+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:22:25.755990+0000) 2024-03-04T11:22:26.748 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:22:26.755+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:22:26.756234+0000) 2024-03-04T11:22:27.748 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:22:27.755+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:22:27.756470+0000) 2024-03-04T11:22:28.748 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:22:28.755+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:22:28.756703+0000) 2024-03-04T11:22:29.749 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:22:29.755+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:22:29.756944+0000) 2024-03-04T11:22:30.749 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:22:30.756+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:22:30.757183+0000) 2024-03-04T11:22:31.753 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:22:31.756+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:22:31.757428+0000) 2024-03-04T11:22:32.750 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:22:32.755+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:22:32.757642+0000) 2024-03-04T11:22:33.750 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:22:33.756+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:22:33.757991+0000) 2024-03-04T11:22:34.750 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:22:34.757+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:22:34.758224+0000) 2024-03-04T11:22:35.750 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:22:35.757+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:22:35.758461+0000) 2024-03-04T11:22:36.750 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:22:36.757+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:22:36.758702+0000) 2024-03-04T11:22:37.750 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:22:37.758+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:22:37.758943+0000) 2024-03-04T11:22:38.751 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:22:38.758+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:22:38.759175+0000) 2024-03-04T11:22:39.751 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:22:39.758+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:22:39.759418+0000) 2024-03-04T11:22:40.751 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:22:40.758+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:22:40.759601+0000) 2024-03-04T11:22:41.751 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:22:41.759+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:22:41.759904+0000) 2024-03-04T11:22:42.752 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:22:42.759+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:22:42.760137+0000) 2024-03-04T11:22:43.752 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:22:43.759+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:22:43.760378+0000) 2024-03-04T11:22:44.752 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:22:44.759+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:22:44.760617+0000) 2024-03-04T11:22:45.752 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:22:45.760+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:22:45.760931+0000) 2024-03-04T11:22:46.753 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:22:46.760+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:22:46.761181+0000) 2024-03-04T11:22:47.753 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:22:47.760+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:22:47.761421+0000) 2024-03-04T11:22:48.753 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:22:48.760+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:22:48.761661+0000) 2024-03-04T11:22:49.753 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:22:49.761+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:22:49.761942+0000) 2024-03-04T11:22:50.608 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:22:50.616 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:22:50.753 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:22:50.761+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:22:50.762226+0000) 2024-03-04T11:22:51.754 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:22:51.761+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:22:51.762472+0000) 2024-03-04T11:22:52.754 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:22:52.762+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:22:52.762709+0000) 2024-03-04T11:22:53.754 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:22:53.762+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:22:53.762954+0000) 2024-03-04T11:22:54.754 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:22:54.762+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:22:54.763136+0000) 2024-03-04T11:22:55.754 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:22:55.762+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:22:55.763384+0000) 2024-03-04T11:22:56.755 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:22:56.763+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:22:56.763628+0000) 2024-03-04T11:22:57.755 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:22:57.763+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:22:57.763927+0000) 2024-03-04T11:22:58.755 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:22:58.763+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:22:58.764176+0000) 2024-03-04T11:22:59.756 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:22:59.763+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:22:59.764434+0000) 2024-03-04T11:23:00.756 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:23:00.764+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:23:00.764719+0000) 2024-03-04T11:23:01.756 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:23:01.764+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:23:01.764974+0000) 2024-03-04T11:23:02.756 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:23:02.764+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:23:02.765179+0000) 2024-03-04T11:23:03.756 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:23:03.764+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:23:03.765439+0000) 2024-03-04T11:23:04.757 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:23:04.764+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:23:04.765639+0000) 2024-03-04T11:23:05.757 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:23:05.765+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:23:05.765934+0000) 2024-03-04T11:23:06.757 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:23:06.765+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:23:06.766198+0000) 2024-03-04T11:23:07.758 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:23:07.766+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:23:07.766450+0000) 2024-03-04T11:23:08.758 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:23:08.766+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:23:08.766699+0000) 2024-03-04T11:23:09.758 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:23:09.766+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:23:09.766952+0000) 2024-03-04T11:23:10.758 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:23:10.766+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:23:10.767221+0000) 2024-03-04T11:23:11.758 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:23:11.767+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:23:11.767506+0000) 2024-03-04T11:23:12.758 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:23:12.767+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:23:12.767754+0000) 2024-03-04T11:23:13.759 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:23:13.767+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:23:13.768014+0000) 2024-03-04T11:23:14.759 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:23:14.767+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:23:14.768262+0000) 2024-03-04T11:23:15.759 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:23:15.768+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:23:15.768513+0000) 2024-03-04T11:23:16.759 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:23:16.768+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:23:16.768779+0000) 2024-03-04T11:23:17.760 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:23:17.768+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:23:17.769035+0000) 2024-03-04T11:23:18.760 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:23:18.769+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:23:18.769282+0000) 2024-03-04T11:23:19.760 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:23:19.769+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:23:19.769542+0000) 2024-03-04T11:23:20.760 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:23:20.769+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:23:20.769792+0000) 2024-03-04T11:23:20.788 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:23:20.796 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:23:21.761 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:23:21.769+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:23:21.770149+0000) 2024-03-04T11:23:22.761 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:23:22.770+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:23:22.770399+0000) 2024-03-04T11:23:23.761 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:23:23.770+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:23:23.770655+0000) 2024-03-04T11:23:24.763 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:23:24.770+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:23:24.770940+0000) 2024-03-04T11:23:25.762 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:23:25.771+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:23:25.771176+0000) 2024-03-04T11:23:26.762 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:23:26.770+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:23:26.771419+0000) 2024-03-04T11:23:27.762 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:23:27.770+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:23:27.771661+0000) 2024-03-04T11:23:28.762 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:23:28.770+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:23:28.771934+0000) 2024-03-04T11:23:29.763 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:23:29.771+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:23:29.772172+0000) 2024-03-04T11:23:30.763 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:23:30.771+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:23:30.772406+0000) 2024-03-04T11:23:31.763 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:23:31.771+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:23:31.772649+0000) 2024-03-04T11:23:32.763 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:23:32.771+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:23:32.772927+0000) 2024-03-04T11:23:33.764 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:23:33.772+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:23:33.773170+0000) 2024-03-04T11:23:34.764 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:23:34.772+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:23:34.773348+0000) 2024-03-04T11:23:35.764 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:23:35.772+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:23:35.773627+0000) 2024-03-04T11:23:36.764 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:23:36.772+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:23:36.773908+0000) 2024-03-04T11:23:37.764 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:23:37.773+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:23:37.774139+0000) 2024-03-04T11:23:38.765 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:23:38.773+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:23:38.774374+0000) 2024-03-04T11:23:39.765 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:23:39.773+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:23:39.774633+0000) 2024-03-04T11:23:40.765 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:23:40.774+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:23:40.774934+0000) 2024-03-04T11:23:41.765 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:23:41.774+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:23:41.775183+0000) 2024-03-04T11:23:42.766 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:23:42.774+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:23:42.775430+0000) 2024-03-04T11:23:43.766 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:23:43.774+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:23:43.775709+0000) 2024-03-04T11:23:44.766 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:23:44.775+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:23:44.775982+0000) 2024-03-04T11:23:45.766 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:23:45.775+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:23:45.776230+0000) 2024-03-04T11:23:46.767 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:23:46.775+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:23:46.776523+0000) 2024-03-04T11:23:47.767 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:23:47.775+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:23:47.776765+0000) 2024-03-04T11:23:48.767 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:23:48.776+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:23:48.777003+0000) 2024-03-04T11:23:49.768 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:23:49.776+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:23:49.777244+0000) 2024-03-04T11:23:50.768 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:23:50.776+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:23:50.777476+0000) 2024-03-04T11:23:50.993 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:23:51.001 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:23:51.768 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:23:51.776+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:23:51.777730+0000) 2024-03-04T11:23:52.768 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:23:52.777+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:23:52.777997+0000) 2024-03-04T11:23:53.768 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:23:53.777+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:23:53.778251+0000) 2024-03-04T11:23:54.768 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:23:54.777+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:23:54.778517+0000) 2024-03-04T11:23:55.769 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:23:55.778+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:23:55.778769+0000) 2024-03-04T11:23:56.769 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:23:56.778+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:23:56.779079+0000) 2024-03-04T11:23:57.769 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:23:57.778+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:23:57.779327+0000) 2024-03-04T11:23:58.769 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:23:58.778+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:23:58.779581+0000) 2024-03-04T11:23:59.770 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:23:59.779+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:23:59.779895+0000) 2024-03-04T11:24:00.770 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:24:00.779+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:24:00.780085+0000) 2024-03-04T11:24:01.770 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:24:01.779+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:24:01.780285+0000) 2024-03-04T11:24:02.770 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:24:02.779+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:24:02.780532+0000) 2024-03-04T11:24:03.771 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:24:03.780+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:24:03.780793+0000) 2024-03-04T11:24:04.771 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:24:04.780+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:24:04.781131+0000) 2024-03-04T11:24:05.771 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:24:05.780+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:24:05.781380+0000) 2024-03-04T11:24:06.771 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:24:06.781+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:24:06.781640+0000) 2024-03-04T11:24:07.771 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:24:07.781+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:24:07.781922+0000) 2024-03-04T11:24:08.772 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:24:08.781+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:24:08.782184+0000) 2024-03-04T11:24:09.772 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:24:09.781+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:24:09.782488+0000) 2024-03-04T11:24:10.773 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:24:10.782+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:24:10.782763+0000) 2024-03-04T11:24:11.772 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:24:11.782+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:24:11.783020+0000) 2024-03-04T11:24:12.773 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:24:12.782+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:24:12.783268+0000) 2024-03-04T11:24:13.773 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:24:13.783+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:24:13.783533+0000) 2024-03-04T11:24:14.773 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:24:14.783+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:24:14.783781+0000) 2024-03-04T11:24:15.774 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:24:15.783+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:24:15.784078+0000) 2024-03-04T11:24:16.774 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:24:16.784+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:24:16.784340+0000) 2024-03-04T11:24:17.774 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:24:17.784+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:24:17.784594+0000) 2024-03-04T11:24:18.774 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:24:18.784+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:24:18.784885+0000) 2024-03-04T11:24:19.774 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:24:19.784+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:24:19.785150+0000) 2024-03-04T11:24:20.775 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:24:20.785+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:24:20.785399+0000) 2024-03-04T11:24:21.155 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:24:21.167 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:24:21.775 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:24:21.784+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:24:21.785652+0000) 2024-03-04T11:24:22.775 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:24:22.785+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:24:22.785934+0000) 2024-03-04T11:24:23.775 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:24:23.785+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:24:23.786196+0000) 2024-03-04T11:24:24.776 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:24:24.786+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:24:24.786453+0000) 2024-03-04T11:24:25.776 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:24:25.786+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:24:25.786732+0000) 2024-03-04T11:24:26.776 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:24:26.786+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:24:26.787000+0000) 2024-03-04T11:24:27.777 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:24:27.787+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:24:27.787247+0000) 2024-03-04T11:24:28.777 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:24:28.786+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:24:28.787440+0000) 2024-03-04T11:24:29.777 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:24:29.786+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:24:29.787697+0000) 2024-03-04T11:24:30.777 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:24:30.786+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:24:30.787924+0000) 2024-03-04T11:24:31.777 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:24:31.787+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:24:31.788177+0000) 2024-03-04T11:24:32.778 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:24:32.786+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:24:32.788443+0000) 2024-03-04T11:24:33.778 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:24:33.787+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:24:33.788701+0000) 2024-03-04T11:24:34.778 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:24:34.787+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:24:34.788957+0000) 2024-03-04T11:24:35.778 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:24:35.788+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:24:35.789212+0000) 2024-03-04T11:24:36.779 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:24:36.788+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:24:36.789480+0000) 2024-03-04T11:24:37.779 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:24:37.787+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:24:37.789722+0000) 2024-03-04T11:24:38.779 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:24:38.789+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:24:38.789978+0000) 2024-03-04T11:24:39.779 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:24:39.789+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:24:39.790254+0000) 2024-03-04T11:24:40.779 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:24:40.789+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:24:40.790521+0000) 2024-03-04T11:24:41.780 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:24:41.789+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:24:41.790770+0000) 2024-03-04T11:24:42.780 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:24:42.790+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:24:42.791032+0000) 2024-03-04T11:24:43.780 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:24:43.790+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:24:43.791303+0000) 2024-03-04T11:24:44.781 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:24:44.790+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:24:44.791555+0000) 2024-03-04T11:24:45.781 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:24:45.791+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:24:45.791810+0000) 2024-03-04T11:24:46.781 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:24:46.791+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:24:46.792132+0000) 2024-03-04T11:24:47.781 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:24:47.791+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:24:47.792384+0000) 2024-03-04T11:24:48.781 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:24:48.791+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:24:48.792639+0000) 2024-03-04T11:24:49.782 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:24:49.792+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:24:49.792933+0000) 2024-03-04T11:24:50.782 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:24:50.792+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:24:50.793183+0000) 2024-03-04T11:24:51.376 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:24:51.386 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:24:51.782 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:24:51.792+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:24:51.793383+0000) 2024-03-04T11:24:52.782 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:24:52.792+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:24:52.793655+0000) 2024-03-04T11:24:53.783 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:24:53.793+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:24:53.793956+0000) 2024-03-04T11:24:54.783 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:24:54.793+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:24:54.794242+0000) 2024-03-04T11:24:55.783 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:24:55.793+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:24:55.794498+0000) 2024-03-04T11:24:56.783 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:24:56.794+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:24:56.794764+0000) 2024-03-04T11:24:57.784 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:24:57.794+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:24:57.795024+0000) 2024-03-04T11:24:58.784 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:24:58.794+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:24:58.795282+0000) 2024-03-04T11:24:59.784 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:24:59.794+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:24:59.795530+0000) 2024-03-04T11:25:00.784 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:25:00.795+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:25:00.795770+0000) 2024-03-04T11:25:01.784 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:25:01.795+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:25:01.796013+0000) 2024-03-04T11:25:02.785 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:25:02.795+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:25:02.796238+0000) 2024-03-04T11:25:03.785 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:25:03.795+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:25:03.796529+0000) 2024-03-04T11:25:04.785 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:25:04.796+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:25:04.796779+0000) 2024-03-04T11:25:05.785 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:25:05.796+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:25:05.797020+0000) 2024-03-04T11:25:06.786 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:25:06.796+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:25:06.797269+0000) 2024-03-04T11:25:07.786 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:25:07.796+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:25:07.797493+0000) 2024-03-04T11:25:08.786 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:25:08.796+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:25:08.797738+0000) 2024-03-04T11:25:09.786 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:25:09.797+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:25:09.798033+0000) 2024-03-04T11:25:10.787 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:25:10.797+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:25:10.798274+0000) 2024-03-04T11:25:11.787 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:25:11.797+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:25:11.798494+0000) 2024-03-04T11:25:12.787 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:25:12.797+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:25:12.798755+0000) 2024-03-04T11:25:13.787 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:25:13.798+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:25:13.798971+0000) 2024-03-04T11:25:14.788 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:25:14.798+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:25:14.799230+0000) 2024-03-04T11:25:15.788 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:25:15.799+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:25:15.799483+0000) 2024-03-04T11:25:16.788 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:25:16.799+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:25:16.799743+0000) 2024-03-04T11:25:17.788 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:25:17.799+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:25:17.799991+0000) 2024-03-04T11:25:18.788 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:25:18.799+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:25:18.800238+0000) 2024-03-04T11:25:19.789 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:25:19.800+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:25:19.800535+0000) 2024-03-04T11:25:20.789 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:25:20.800+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:25:20.800792+0000) 2024-03-04T11:25:21.580 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:25:21.589 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:25:21.789 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:25:21.800+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:25:21.801012+0000) 2024-03-04T11:25:22.789 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:25:22.800+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:25:22.801241+0000) 2024-03-04T11:25:23.790 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:25:23.801+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:25:23.801523+0000) 2024-03-04T11:25:24.790 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:25:24.801+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:25:24.801782+0000) 2024-03-04T11:25:25.790 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:25:25.801+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:25:25.802073+0000) 2024-03-04T11:25:26.790 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:25:26.802+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:25:26.802331+0000) 2024-03-04T11:25:27.791 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:25:27.802+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:25:27.802580+0000) 2024-03-04T11:25:28.791 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:25:28.802+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:25:28.802882+0000) 2024-03-04T11:25:29.791 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:25:29.802+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:25:29.803149+0000) 2024-03-04T11:25:30.791 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:25:30.803+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:25:30.803399+0000) 2024-03-04T11:25:31.792 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:25:31.803+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:25:31.803651+0000) 2024-03-04T11:25:32.792 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:25:32.802+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:25:32.803918+0000) 2024-03-04T11:25:33.792 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:25:33.803+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:25:33.804156+0000) 2024-03-04T11:25:34.792 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:25:34.803+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:25:34.804401+0000) 2024-03-04T11:25:35.793 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:25:35.803+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:25:35.804647+0000) 2024-03-04T11:25:36.793 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:25:36.803+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:25:36.804925+0000) 2024-03-04T11:25:37.793 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:25:37.804+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:25:37.805179+0000) 2024-03-04T11:25:38.794 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:25:38.804+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:25:38.805430+0000) 2024-03-04T11:25:39.794 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:25:39.804+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:25:39.805683+0000) 2024-03-04T11:25:40.794 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:25:40.804+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:25:40.805953+0000) 2024-03-04T11:25:41.794 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:25:41.805+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:25:41.806197+0000) 2024-03-04T11:25:42.794 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:25:42.805+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:25:42.806427+0000) 2024-03-04T11:25:43.795 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:25:43.805+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:25:43.806692+0000) 2024-03-04T11:25:44.795 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:25:44.806+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:25:44.806936+0000) 2024-03-04T11:25:45.795 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:25:45.806+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:25:45.807130+0000) 2024-03-04T11:25:46.795 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:25:46.806+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:25:46.807373+0000) 2024-03-04T11:25:47.795 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:25:47.806+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:25:47.807624+0000) 2024-03-04T11:25:48.796 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:25:48.807+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:25:48.807916+0000) 2024-03-04T11:25:49.796 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:25:49.807+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:25:49.808165+0000) 2024-03-04T11:25:50.796 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:25:50.807+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:25:50.808397+0000) 2024-03-04T11:25:51.793 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:25:51.796 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:25:51.807+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:25:51.808640+0000) 2024-03-04T11:25:51.802 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:25:52.797 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:25:52.808+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:25:52.808925+0000) 2024-03-04T11:25:53.797 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:25:53.808+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:25:53.809170+0000) 2024-03-04T11:25:54.797 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:25:54.808+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:25:54.809431+0000) 2024-03-04T11:25:55.797 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:25:55.808+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:25:55.809688+0000) 2024-03-04T11:25:56.798 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:25:56.809+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:25:56.810001+0000) 2024-03-04T11:25:57.798 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:25:57.809+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:25:57.810257+0000) 2024-03-04T11:25:58.798 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:25:58.809+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:25:58.810496+0000) 2024-03-04T11:25:59.798 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:25:59.810+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:25:59.810742+0000) 2024-03-04T11:26:00.798 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:26:00.810+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:26:00.810987+0000) 2024-03-04T11:26:01.799 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:26:01.810+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:26:01.811231+0000) 2024-03-04T11:26:02.799 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:26:02.810+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:26:02.811473+0000) 2024-03-04T11:26:03.799 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:26:03.811+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:26:03.811717+0000) 2024-03-04T11:26:04.799 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:26:04.811+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:26:04.812000+0000) 2024-03-04T11:26:05.800 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:26:05.811+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:26:05.812254+0000) 2024-03-04T11:26:06.800 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:26:06.811+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:26:06.812494+0000) 2024-03-04T11:26:07.800 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:26:07.811+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:26:07.812733+0000) 2024-03-04T11:26:08.800 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:26:08.812+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:26:08.813010+0000) 2024-03-04T11:26:09.801 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:26:09.812+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:26:09.813239+0000) 2024-03-04T11:26:10.801 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:26:10.812+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:26:10.813469+0000) 2024-03-04T11:26:11.801 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:26:11.813+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:26:11.813760+0000) 2024-03-04T11:26:12.801 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:26:12.813+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:26:12.814045+0000) 2024-03-04T11:26:13.802 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:26:13.813+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:26:13.814347+0000) 2024-03-04T11:26:14.802 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:26:14.814+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:26:14.814631+0000) 2024-03-04T11:26:15.802 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:26:15.814+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:26:15.814963+0000) 2024-03-04T11:26:16.802 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:26:16.814+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:26:16.815272+0000) 2024-03-04T11:26:17.803 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:26:17.814+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:26:17.815563+0000) 2024-03-04T11:26:18.803 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:26:18.815+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:26:18.815902+0000) 2024-03-04T11:26:19.803 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:26:19.815+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:26:19.816184+0000) 2024-03-04T11:26:20.804 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:26:20.816+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:26:20.816480+0000) 2024-03-04T11:26:21.804 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:26:21.816+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:26:21.816774+0000) 2024-03-04T11:26:21.931 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:26:21.939 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:26:22.804 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:26:22.816+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:26:22.817059+0000) 2024-03-04T11:26:23.804 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:26:23.817+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:26:23.817343+0000) 2024-03-04T11:26:24.805 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:26:24.817+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:26:24.817632+0000) 2024-03-04T11:26:25.805 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:26:25.817+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:26:25.817975+0000) 2024-03-04T11:26:26.805 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:26:26.817+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:26:26.818267+0000) 2024-03-04T11:26:27.805 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:26:27.817+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:26:27.818491+0000) 2024-03-04T11:26:28.806 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:26:28.818+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:26:28.818778+0000) 2024-03-04T11:26:29.806 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:26:29.818+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:26:29.819069+0000) 2024-03-04T11:26:30.807 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:26:30.819+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:26:30.819356+0000) 2024-03-04T11:26:31.807 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:26:31.819+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:26:31.819632+0000) 2024-03-04T11:26:32.807 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:26:32.819+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:26:32.819955+0000) 2024-03-04T11:26:33.807 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:26:33.820+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:26:33.820260+0000) 2024-03-04T11:26:34.808 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:26:34.819+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:26:34.820566+0000) 2024-03-04T11:26:35.808 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:26:35.819+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:26:35.820807+0000) 2024-03-04T11:26:36.808 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:26:36.820+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:26:36.821130+0000) 2024-03-04T11:26:37.808 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:26:37.820+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:26:37.821408+0000) 2024-03-04T11:26:38.809 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:26:38.819+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:26:38.821728+0000) 2024-03-04T11:26:39.809 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:26:39.820+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:26:39.822071+0000) 2024-03-04T11:26:40.809 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:26:40.821+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:26:40.822347+0000) 2024-03-04T11:26:41.810 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:26:41.821+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:26:41.822640+0000) 2024-03-04T11:26:42.810 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:26:42.821+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:26:42.822962+0000) 2024-03-04T11:26:43.810 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:26:43.822+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:26:43.823249+0000) 2024-03-04T11:26:44.810 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:26:44.822+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:26:44.823537+0000) 2024-03-04T11:26:45.811 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:26:45.823+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:26:45.823872+0000) 2024-03-04T11:26:46.811 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:26:46.823+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:26:46.824200+0000) 2024-03-04T11:26:47.811 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:26:47.823+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:26:47.824490+0000) 2024-03-04T11:26:48.811 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:26:48.823+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:26:48.824704+0000) 2024-03-04T11:26:49.812 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:26:49.824+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:26:49.824950+0000) 2024-03-04T11:26:50.812 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:26:50.824+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:26:50.825181+0000) 2024-03-04T11:26:51.812 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:26:51.824+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:26:51.825476+0000) 2024-03-04T11:26:52.127 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:26:52.136 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:26:52.812 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:26:52.824+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:26:52.825697+0000) 2024-03-04T11:26:53.813 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:26:53.825+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:26:53.825994+0000) 2024-03-04T11:26:54.813 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:26:54.825+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:26:54.826282+0000) 2024-03-04T11:26:55.813 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:26:55.825+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:26:55.826652+0000) 2024-03-04T11:26:56.814 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:26:56.826+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:26:56.826976+0000) 2024-03-04T11:26:57.814 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:26:57.826+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:26:57.827267+0000) 2024-03-04T11:26:58.814 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:26:58.826+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:26:58.827513+0000) 2024-03-04T11:26:59.814 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:26:59.826+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:26:59.827757+0000) 2024-03-04T11:27:00.815 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:27:00.827+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:27:00.828015+0000) 2024-03-04T11:27:01.815 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:27:01.827+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:27:01.828289+0000) 2024-03-04T11:27:02.815 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:27:02.827+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:27:02.828523+0000) 2024-03-04T11:27:03.815 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:27:03.828+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:27:03.828770+0000) 2024-03-04T11:27:04.816 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:27:04.828+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:27:04.829009+0000) 2024-03-04T11:27:05.816 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:27:05.828+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:27:05.829258+0000) 2024-03-04T11:27:06.816 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:27:06.828+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:27:06.829519+0000) 2024-03-04T11:27:07.816 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:27:07.829+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:27:07.829756+0000) 2024-03-04T11:27:08.816 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:27:08.829+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:27:08.830017+0000) 2024-03-04T11:27:09.817 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:27:09.829+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:27:09.830256+0000) 2024-03-04T11:27:10.817 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:27:10.829+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:27:10.830489+0000) 2024-03-04T11:27:11.817 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:27:11.830+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:27:11.830739+0000) 2024-03-04T11:27:12.817 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:27:12.830+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:27:12.830979+0000) 2024-03-04T11:27:13.818 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:27:13.830+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:27:13.831221+0000) 2024-03-04T11:27:14.818 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:27:14.830+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:27:14.831483+0000) 2024-03-04T11:27:15.818 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:27:15.831+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:27:15.831724+0000) 2024-03-04T11:27:16.818 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:27:16.831+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:27:16.831971+0000) 2024-03-04T11:27:17.818 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:27:17.831+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:27:17.832206+0000) 2024-03-04T11:27:18.819 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:27:18.832+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:27:18.832440+0000) 2024-03-04T11:27:19.819 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:27:19.832+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:27:19.832687+0000) 2024-03-04T11:27:20.819 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:27:20.832+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:27:20.832953+0000) 2024-03-04T11:27:21.819 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:27:21.832+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:27:21.833209+0000) 2024-03-04T11:27:22.333 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:27:22.341 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:27:22.820 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:27:22.833+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:27:22.833448+0000) 2024-03-04T11:27:23.820 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:27:23.833+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:27:23.833688+0000) 2024-03-04T11:27:24.820 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:27:24.833+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:27:24.833941+0000) 2024-03-04T11:27:25.820 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:27:25.833+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:27:25.834188+0000) 2024-03-04T11:27:26.820 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:27:26.833+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:27:26.834395+0000) 2024-03-04T11:27:27.821 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:27:27.834+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:27:27.834651+0000) 2024-03-04T11:27:28.821 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:27:28.834+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:27:28.834934+0000) 2024-03-04T11:27:29.821 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:27:29.834+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:27:29.835175+0000) 2024-03-04T11:27:30.822 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:27:30.835+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:27:30.835444+0000) 2024-03-04T11:27:31.822 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:27:31.835+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:27:31.835708+0000) 2024-03-04T11:27:32.822 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:27:32.835+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:27:32.835941+0000) 2024-03-04T11:27:33.822 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:27:33.836+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:27:33.836180+0000) 2024-03-04T11:27:34.822 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:27:34.836+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:27:34.836417+0000) 2024-03-04T11:27:35.823 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:27:35.835+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:27:35.836599+0000) 2024-03-04T11:27:36.824 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:27:36.836+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:27:36.836885+0000) 2024-03-04T11:27:37.823 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:27:37.836+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:27:37.837132+0000) 2024-03-04T11:27:38.823 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:27:38.836+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:27:38.837365+0000) 2024-03-04T11:27:39.823 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:27:39.836+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:27:39.837601+0000) 2024-03-04T11:27:40.824 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:27:40.836+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:27:40.837872+0000) 2024-03-04T11:27:41.824 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:27:41.837+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:27:41.838114+0000) 2024-03-04T11:27:42.824 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:27:42.837+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:27:42.838346+0000) 2024-03-04T11:27:43.825 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:27:43.837+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:27:43.838596+0000) 2024-03-04T11:27:44.825 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:27:44.837+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:27:44.838908+0000) 2024-03-04T11:27:45.825 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:27:45.838+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:27:45.839155+0000) 2024-03-04T11:27:46.825 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:27:46.838+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:27:46.839398+0000) 2024-03-04T11:27:47.826 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:27:47.838+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:27:47.839637+0000) 2024-03-04T11:27:48.828 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:27:48.839+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:27:48.839964+0000) 2024-03-04T11:27:49.826 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:27:49.839+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:27:49.840205+0000) 2024-03-04T11:27:50.826 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:27:50.839+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:27:50.840437+0000) 2024-03-04T11:27:51.827 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:27:51.839+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:27:51.840686+0000) 2024-03-04T11:27:52.512 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:27:52.520 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:27:52.827 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:27:52.840+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:27:52.840927+0000) 2024-03-04T11:27:53.827 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:27:53.840+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:27:53.841170+0000) 2024-03-04T11:27:54.827 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:27:54.840+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:27:54.841404+0000) 2024-03-04T11:27:55.828 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:27:55.840+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:27:55.841648+0000) 2024-03-04T11:27:56.828 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:27:56.841+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:27:56.841933+0000) 2024-03-04T11:27:57.828 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:27:57.841+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:27:57.842171+0000) 2024-03-04T11:27:58.828 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:27:58.841+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:27:58.842403+0000) 2024-03-04T11:27:59.828 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:27:59.841+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:27:59.842649+0000) 2024-03-04T11:28:00.829 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:28:00.842+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:28:00.842964+0000) 2024-03-04T11:28:01.829 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:28:01.842+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:28:01.843172+0000) 2024-03-04T11:28:02.829 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:28:02.842+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:28:02.843408+0000) 2024-03-04T11:28:03.829 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:28:03.842+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:28:03.843680+0000) 2024-03-04T11:28:04.830 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:28:04.843+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:28:04.843960+0000) 2024-03-04T11:28:05.830 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:28:05.843+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:28:05.844199+0000) 2024-03-04T11:28:06.830 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:28:06.843+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:28:06.844447+0000) 2024-03-04T11:28:07.830 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:28:07.844+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:28:07.844682+0000) 2024-03-04T11:28:08.830 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:28:08.844+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:28:08.844946+0000) 2024-03-04T11:28:09.831 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:28:09.844+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:28:09.845189+0000) 2024-03-04T11:28:10.831 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:28:10.844+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:28:10.845424+0000) 2024-03-04T11:28:11.831 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:28:11.845+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:28:11.845669+0000) 2024-03-04T11:28:12.831 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:28:12.845+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:28:12.845931+0000) 2024-03-04T11:28:13.832 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:28:13.845+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:28:13.846123+0000) 2024-03-04T11:28:14.832 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:28:14.845+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:28:14.846403+0000) 2024-03-04T11:28:15.832 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:28:15.845+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:28:15.846667+0000) 2024-03-04T11:28:16.832 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:28:16.846+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:28:16.846950+0000) 2024-03-04T11:28:17.833 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:28:17.846+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:28:17.847196+0000) 2024-03-04T11:28:18.833 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:28:18.846+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:28:18.847434+0000) 2024-03-04T11:28:19.833 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:28:19.847+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:28:19.847681+0000) 2024-03-04T11:28:20.833 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:28:20.847+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:28:20.847940+0000) 2024-03-04T11:28:21.834 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:28:21.847+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:28:21.848172+0000) 2024-03-04T11:28:22.659 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:28:22.670 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:28:22.834 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:28:22.848+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:28:22.848417+0000) 2024-03-04T11:28:23.834 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:28:23.848+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:28:23.848657+0000) 2024-03-04T11:28:24.834 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:28:24.848+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:28:24.848937+0000) 2024-03-04T11:28:25.834 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:28:25.848+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:28:25.849187+0000) 2024-03-04T11:28:26.835 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:28:26.849+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:28:26.849439+0000) 2024-03-04T11:28:27.835 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:28:27.848+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:28:27.849692+0000) 2024-03-04T11:28:28.835 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:28:28.849+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:28:28.849933+0000) 2024-03-04T11:28:29.835 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:28:29.849+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:28:29.850180+0000) 2024-03-04T11:28:30.836 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:28:30.850+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:28:30.850421+0000) 2024-03-04T11:28:31.836 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:28:31.850+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:28:31.850619+0000) 2024-03-04T11:28:32.836 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:28:32.850+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:28:32.850938+0000) 2024-03-04T11:28:33.836 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:28:33.850+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:28:33.851197+0000) 2024-03-04T11:28:34.837 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:28:34.851+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:28:34.851436+0000) 2024-03-04T11:28:35.837 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:28:35.851+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:28:35.851700+0000) 2024-03-04T11:28:36.837 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:28:36.851+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:28:36.851943+0000) 2024-03-04T11:28:37.837 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:28:37.852+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:28:37.852182+0000) 2024-03-04T11:28:38.838 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:28:38.852+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:28:38.852415+0000) 2024-03-04T11:28:39.838 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:28:39.851+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:28:39.852660+0000) 2024-03-04T11:28:40.838 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:28:40.851+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:28:40.852940+0000) 2024-03-04T11:28:41.838 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:28:41.852+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:28:41.853227+0000) 2024-03-04T11:28:42.839 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:28:42.852+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:28:42.853468+0000) 2024-03-04T11:28:43.839 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:28:43.851+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:28:43.853723+0000) 2024-03-04T11:28:44.839 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:28:44.852+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:28:44.853967+0000) 2024-03-04T11:28:45.839 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:28:45.853+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:28:45.854206+0000) 2024-03-04T11:28:46.840 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:28:46.853+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:28:46.854469+0000) 2024-03-04T11:28:47.840 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:28:47.853+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:28:47.854707+0000) 2024-03-04T11:28:48.840 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:28:48.853+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:28:48.854961+0000) 2024-03-04T11:28:49.840 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:28:49.854+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:28:49.855203+0000) 2024-03-04T11:28:50.840 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:28:50.854+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:28:50.855438+0000) 2024-03-04T11:28:51.841 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:28:51.854+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:28:51.855713+0000) 2024-03-04T11:28:52.841 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:28:52.855+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:28:52.855949+0000) 2024-03-04T11:28:52.855 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:28:52.867 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:28:53.842 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:28:53.855+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:28:53.856193+0000) 2024-03-04T11:28:54.841 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:28:54.855+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:28:54.856467+0000) 2024-03-04T11:28:55.842 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:28:55.855+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:28:55.856707+0000) 2024-03-04T11:28:56.842 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:28:56.856+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:28:56.856953+0000) 2024-03-04T11:28:57.842 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:28:57.856+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:28:57.857191+0000) 2024-03-04T11:28:58.842 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:28:58.856+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:28:58.857425+0000) 2024-03-04T11:28:59.843 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:28:59.856+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:28:59.857716+0000) 2024-03-04T11:29:00.843 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:29:00.857+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:29:00.857941+0000) 2024-03-04T11:29:01.843 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:29:01.857+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:29:01.858198+0000) 2024-03-04T11:29:02.843 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:29:02.857+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:29:02.858434+0000) 2024-03-04T11:29:03.844 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:29:03.857+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:29:03.858663+0000) 2024-03-04T11:29:04.844 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:29:04.858+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:29:04.858932+0000) 2024-03-04T11:29:05.844 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:29:05.858+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:29:05.859155+0000) 2024-03-04T11:29:06.844 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:29:06.858+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:29:06.859402+0000) 2024-03-04T11:29:07.845 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:29:07.858+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:29:07.859631+0000) 2024-03-04T11:29:08.845 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:29:08.859+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:29:08.859930+0000) 2024-03-04T11:29:09.845 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:29:09.859+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:29:09.860140+0000) 2024-03-04T11:29:10.845 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:29:10.859+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:29:10.860376+0000) 2024-03-04T11:29:11.845 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:29:11.859+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:29:11.860611+0000) 2024-03-04T11:29:12.846 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:29:12.860+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:29:12.860912+0000) 2024-03-04T11:29:13.846 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:29:13.860+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:29:13.861154+0000) 2024-03-04T11:29:14.846 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:29:14.860+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:29:14.861414+0000) 2024-03-04T11:29:15.846 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:29:15.861+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:29:15.861677+0000) 2024-03-04T11:29:16.847 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:29:16.861+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:29:16.861947+0000) 2024-03-04T11:29:17.847 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:29:17.861+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:29:17.862144+0000) 2024-03-04T11:29:18.848 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:29:18.861+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:29:18.862378+0000) 2024-03-04T11:29:19.847 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:29:19.862+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:29:19.862625+0000) 2024-03-04T11:29:20.848 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:29:20.862+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:29:20.862927+0000) 2024-03-04T11:29:21.848 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:29:21.862+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:29:21.863168+0000) 2024-03-04T11:29:22.848 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:29:22.862+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:29:22.863401+0000) 2024-03-04T11:29:23.000 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:29:23.008 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:29:23.848 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:29:23.863+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:29:23.863641+0000) 2024-03-04T11:29:24.848 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:29:24.863+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:29:24.863935+0000) 2024-03-04T11:29:25.849 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:29:25.863+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:29:25.864174+0000) 2024-03-04T11:29:26.849 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:29:26.864+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:29:26.864437+0000) 2024-03-04T11:29:27.849 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:29:27.864+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:29:27.864670+0000) 2024-03-04T11:29:28.849 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:29:28.864+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:29:28.864962+0000) 2024-03-04T11:29:29.850 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:29:29.864+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:29:29.865274+0000) 2024-03-04T11:29:30.850 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:29:30.865+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:29:30.865520+0000) 2024-03-04T11:29:31.850 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:29:31.865+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:29:31.865777+0000) 2024-03-04T11:29:32.850 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:29:32.865+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:29:32.866032+0000) 2024-03-04T11:29:33.851 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:29:33.866+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:29:33.866272+0000) 2024-03-04T11:29:34.851 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:29:34.866+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:29:34.866514+0000) 2024-03-04T11:29:35.851 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:29:35.866+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:29:35.866763+0000) 2024-03-04T11:29:36.851 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:29:36.866+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:29:36.867012+0000) 2024-03-04T11:29:37.852 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:29:37.866+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:29:37.867191+0000) 2024-03-04T11:29:38.852 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:29:38.866+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:29:38.867378+0000) 2024-03-04T11:29:39.852 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:29:39.867+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:29:39.867625+0000) 2024-03-04T11:29:40.852 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:29:40.867+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:29:40.867942+0000) 2024-03-04T11:29:41.853 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:29:41.868+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:29:41.868193+0000) 2024-03-04T11:29:42.853 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:29:42.867+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:29:42.868432+0000) 2024-03-04T11:29:43.853 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:29:43.866+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:29:43.868687+0000) 2024-03-04T11:29:44.853 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:29:44.867+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:29:44.868922+0000) 2024-03-04T11:29:45.853 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:29:45.868+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:29:45.869166+0000) 2024-03-04T11:29:46.854 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:29:46.867+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:29:46.869362+0000) 2024-03-04T11:29:47.854 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:29:47.868+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:29:47.869592+0000) 2024-03-04T11:29:48.854 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:29:48.868+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:29:48.869917+0000) 2024-03-04T11:29:49.854 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:29:49.869+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:29:49.870153+0000) 2024-03-04T11:29:50.855 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:29:50.869+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:29:50.870392+0000) 2024-03-04T11:29:51.855 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:29:51.869+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:29:51.870631+0000) 2024-03-04T11:29:52.855 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:29:52.869+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:29:52.870916+0000) 2024-03-04T11:29:53.231 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:29:53.239 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:29:53.855 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:29:53.870+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:29:53.871158+0000) 2024-03-04T11:29:54.856 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:29:54.870+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:29:54.871391+0000) 2024-03-04T11:29:55.856 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:29:55.870+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:29:55.871637+0000) 2024-03-04T11:29:56.856 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:29:56.871+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:29:56.871938+0000) 2024-03-04T11:29:57.856 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:29:57.871+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:29:57.872132+0000) 2024-03-04T11:29:58.856 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:29:58.871+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:29:58.872371+0000) 2024-03-04T11:29:59.857 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:29:59.871+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:29:59.872606+0000) 2024-03-04T11:30:00.857 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:30:00.872+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:30:00.872928+0000) 2024-03-04T11:30:01.857 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:30:01.872+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:30:01.873241+0000) 2024-03-04T11:30:02.858 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:30:02.872+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:30:02.873548+0000) 2024-03-04T11:30:03.858 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:30:03.872+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:30:03.873776+0000) 2024-03-04T11:30:04.858 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:30:04.873+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:30:04.874057+0000) 2024-03-04T11:30:05.858 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:30:05.873+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:30:05.874355+0000) 2024-03-04T11:30:06.859 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:30:06.873+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:30:06.874647+0000) 2024-03-04T11:30:07.859 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:30:07.874+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:30:07.874964+0000) 2024-03-04T11:30:08.859 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:30:08.874+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:30:08.875247+0000) 2024-03-04T11:30:09.860 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:30:09.874+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:30:09.875543+0000) 2024-03-04T11:30:10.860 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:30:10.875+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:30:10.875868+0000) 2024-03-04T11:30:11.860 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:30:11.875+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:30:11.876155+0000) 2024-03-04T11:30:12.860 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:30:12.875+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:30:12.876430+0000) 2024-03-04T11:30:13.864 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:30:13.876+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:30:13.876721+0000) 2024-03-04T11:30:14.861 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:30:14.876+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:30:14.877016+0000) 2024-03-04T11:30:15.861 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:30:15.876+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:30:15.877309+0000) 2024-03-04T11:30:16.862 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:30:16.877+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:30:16.877617+0000) 2024-03-04T11:30:17.862 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:30:17.877+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:30:17.877955+0000) 2024-03-04T11:30:18.862 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:30:18.877+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:30:18.878240+0000) 2024-03-04T11:30:19.862 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:30:19.877+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:30:19.878530+0000) 2024-03-04T11:30:20.863 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:30:20.878+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:30:20.878861+0000) 2024-03-04T11:30:21.863 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:30:21.878+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:30:21.879163+0000) 2024-03-04T11:30:22.863 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:30:22.878+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:30:22.879453+0000) 2024-03-04T11:30:23.461 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:30:23.469 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:30:23.864 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:30:23.879+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:30:23.879776+0000) 2024-03-04T11:30:24.864 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:30:24.879+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:30:24.880016+0000) 2024-03-04T11:30:25.864 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:30:25.879+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:30:25.880251+0000) 2024-03-04T11:30:26.864 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:30:26.879+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:30:26.880455+0000) 2024-03-04T11:30:27.864 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:30:27.880+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:30:27.880718+0000) 2024-03-04T11:30:28.865 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:30:28.880+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:30:28.880979+0000) 2024-03-04T11:30:29.865 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:30:29.880+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:30:29.881213+0000) 2024-03-04T11:30:30.865 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:30:30.881+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:30:30.881469+0000) 2024-03-04T11:30:31.865 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:30:31.881+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:30:31.881710+0000) 2024-03-04T11:30:32.866 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:30:32.881+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:30:32.882007+0000) 2024-03-04T11:30:33.866 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:30:33.881+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:30:33.882264+0000) 2024-03-04T11:30:34.866 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:30:34.882+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:30:34.882516+0000) 2024-03-04T11:30:35.866 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:30:35.882+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:30:35.882784+0000) 2024-03-04T11:30:36.867 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:30:36.882+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:30:36.883121+0000) 2024-03-04T11:30:37.867 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:30:37.882+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:30:37.883379+0000) 2024-03-04T11:30:38.867 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:30:38.883+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:30:38.883631+0000) 2024-03-04T11:30:39.868 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:30:39.883+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:30:39.883935+0000) 2024-03-04T11:30:40.868 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:30:40.883+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:30:40.884167+0000) 2024-03-04T11:30:41.868 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:30:41.884+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:30:41.884405+0000) 2024-03-04T11:30:42.868 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:30:42.884+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:30:42.884670+0000) 2024-03-04T11:30:43.868 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:30:43.884+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:30:43.884944+0000) 2024-03-04T11:30:44.869 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:30:44.885+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:30:44.885181+0000) 2024-03-04T11:30:45.869 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:30:45.884+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:30:45.885439+0000) 2024-03-04T11:30:46.870 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:30:46.883+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:30:46.885712+0000) 2024-03-04T11:30:47.870 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:30:47.884+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:30:47.885945+0000) 2024-03-04T11:30:48.870 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:30:48.885+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:30:48.886171+0000) 2024-03-04T11:30:49.870 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:30:49.885+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:30:49.886410+0000) 2024-03-04T11:30:50.870 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:30:50.884+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:30:50.886661+0000) 2024-03-04T11:30:51.871 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:30:51.885+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:30:51.886976+0000) 2024-03-04T11:30:52.871 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:30:52.886+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:30:52.887180+0000) 2024-03-04T11:30:53.674 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:30:53.682 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:30:53.871 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:30:53.886+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:30:53.887436+0000) 2024-03-04T11:30:54.871 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:30:54.886+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:30:54.887675+0000) 2024-03-04T11:30:55.871 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:30:55.886+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:30:55.887956+0000) 2024-03-04T11:30:56.872 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:30:56.887+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:30:56.888204+0000) 2024-03-04T11:30:57.872 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:30:57.887+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:30:57.888449+0000) 2024-03-04T11:30:58.872 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:30:58.887+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:30:58.888701+0000) 2024-03-04T11:30:59.872 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:30:59.888+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:30:59.888941+0000) 2024-03-04T11:31:00.873 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:31:00.888+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:31:00.889176+0000) 2024-03-04T11:31:01.873 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:31:01.888+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:31:01.889415+0000) 2024-03-04T11:31:02.873 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:31:02.888+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:31:02.889647+0000) 2024-03-04T11:31:03.873 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:31:03.889+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:31:03.889937+0000) 2024-03-04T11:31:04.873 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:31:04.889+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:31:04.890170+0000) 2024-03-04T11:31:05.874 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:31:05.889+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:31:05.890447+0000) 2024-03-04T11:31:06.874 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:31:06.889+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:31:06.890707+0000) 2024-03-04T11:31:07.874 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:31:07.890+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:31:07.890939+0000) 2024-03-04T11:31:08.874 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:31:08.890+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:31:08.891177+0000) 2024-03-04T11:31:09.875 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:31:09.890+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:31:09.891409+0000) 2024-03-04T11:31:10.875 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:31:10.890+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:31:10.891632+0000) 2024-03-04T11:31:11.875 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:31:11.891+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:31:11.891932+0000) 2024-03-04T11:31:12.875 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:31:12.891+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:31:12.892186+0000) 2024-03-04T11:31:13.876 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:31:13.891+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:31:13.892426+0000) 2024-03-04T11:31:14.876 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:31:14.892+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:31:14.892662+0000) 2024-03-04T11:31:15.876 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:31:15.892+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:31:15.892934+0000) 2024-03-04T11:31:16.877 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:31:16.892+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:31:16.893179+0000) 2024-03-04T11:31:17.877 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:31:17.892+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:31:17.893447+0000) 2024-03-04T11:31:18.877 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:31:18.893+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:31:18.893685+0000) 2024-03-04T11:31:19.877 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:31:19.893+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:31:19.893922+0000) 2024-03-04T11:31:20.877 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:31:20.893+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:31:20.894151+0000) 2024-03-04T11:31:21.878 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:31:21.893+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:31:21.894444+0000) 2024-03-04T11:31:22.878 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:31:22.894+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:31:22.894697+0000) 2024-03-04T11:31:23.878 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:31:23.894+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:31:23.894980+0000) 2024-03-04T11:31:23.895 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:31:23.903 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:31:24.879 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:31:24.894+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:31:24.895214+0000) 2024-03-04T11:31:25.879 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:31:25.894+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:31:25.895453+0000) 2024-03-04T11:31:26.879 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:31:26.895+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:31:26.895696+0000) 2024-03-04T11:31:27.879 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:31:27.895+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:31:27.895962+0000) 2024-03-04T11:31:28.879 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:31:28.895+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:31:28.896231+0000) 2024-03-04T11:31:29.880 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:31:29.896+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:31:29.896472+0000) 2024-03-04T11:31:30.880 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:31:30.895+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:31:30.896698+0000) 2024-03-04T11:31:31.880 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:31:31.896+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:31:31.896939+0000) 2024-03-04T11:31:32.880 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:31:32.896+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:31:32.897175+0000) 2024-03-04T11:31:33.880 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:31:33.896+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:31:33.897389+0000) 2024-03-04T11:31:34.881 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:31:34.897+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:31:34.897621+0000) 2024-03-04T11:31:35.881 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:31:35.897+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:31:35.897918+0000) 2024-03-04T11:31:36.881 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:31:36.897+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:31:36.898163+0000) 2024-03-04T11:31:37.881 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:31:37.898+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:31:37.898405+0000) 2024-03-04T11:31:38.882 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:31:38.898+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:31:38.898644+0000) 2024-03-04T11:31:39.882 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:31:39.898+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:31:39.898932+0000) 2024-03-04T11:31:40.882 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:31:40.898+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:31:40.899163+0000) 2024-03-04T11:31:41.882 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:31:41.899+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:31:41.899403+0000) 2024-03-04T11:31:42.883 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:31:42.899+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:31:42.899639+0000) 2024-03-04T11:31:43.883 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:31:43.899+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:31:43.899863+0000) 2024-03-04T11:31:44.883 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:31:44.899+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:31:44.900059+0000) 2024-03-04T11:31:45.883 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:31:45.900+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:31:45.900316+0000) 2024-03-04T11:31:46.884 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:31:46.900+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:31:46.900557+0000) 2024-03-04T11:31:47.884 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:31:47.900+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:31:47.900791+0000) 2024-03-04T11:31:48.884 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:31:48.899+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:31:48.900952+0000) 2024-03-04T11:31:49.884 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:31:49.900+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:31:49.901185+0000) 2024-03-04T11:31:50.884 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:31:50.900+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:31:50.901417+0000) 2024-03-04T11:31:51.885 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:31:51.899+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:31:51.901668+0000) 2024-03-04T11:31:52.885 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:31:52.900+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:31:52.901950+0000) 2024-03-04T11:31:53.885 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:31:53.901+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:31:53.902191+0000) 2024-03-04T11:31:54.033 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:31:54.043 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:31:54.885 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:31:54.901+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:31:54.902429+0000) 2024-03-04T11:31:55.885 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:31:55.901+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:31:55.902671+0000) 2024-03-04T11:31:56.886 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:31:56.901+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:31:56.902943+0000) 2024-03-04T11:31:57.886 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:31:57.902+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:31:57.903182+0000) 2024-03-04T11:31:58.886 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:31:58.902+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:31:58.903439+0000) 2024-03-04T11:31:59.887 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:31:59.902+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:31:59.903691+0000) 2024-03-04T11:32:00.887 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:32:00.903+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:32:00.903925+0000) 2024-03-04T11:32:01.887 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:32:01.903+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:32:01.904167+0000) 2024-03-04T11:32:02.887 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:32:02.903+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:32:02.904404+0000) 2024-03-04T11:32:03.888 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:32:03.903+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:32:03.904640+0000) 2024-03-04T11:32:04.888 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:32:04.904+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:32:04.904920+0000) 2024-03-04T11:32:05.888 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:32:05.904+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:32:05.905164+0000) 2024-03-04T11:32:06.888 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:32:06.904+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:32:06.905411+0000) 2024-03-04T11:32:07.888 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:32:07.904+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:32:07.905644+0000) 2024-03-04T11:32:08.889 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:32:08.905+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:32:08.905930+0000) 2024-03-04T11:32:09.889 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:32:09.905+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:32:09.906170+0000) 2024-03-04T11:32:10.889 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:32:10.905+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:32:10.906432+0000) 2024-03-04T11:32:11.889 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:32:11.905+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:32:11.906673+0000) 2024-03-04T11:32:12.890 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:32:12.906+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:32:12.906941+0000) 2024-03-04T11:32:13.890 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:32:13.906+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:32:13.907186+0000) 2024-03-04T11:32:14.890 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:32:14.906+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:32:14.907428+0000) 2024-03-04T11:32:15.891 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:32:15.906+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:32:15.907736+0000) 2024-03-04T11:32:16.891 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:32:16.907+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:32:16.907999+0000) 2024-03-04T11:32:17.891 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:32:17.907+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:32:17.908232+0000) 2024-03-04T11:32:18.891 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:32:18.907+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:32:18.908465+0000) 2024-03-04T11:32:19.891 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:32:19.908+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:32:19.908700+0000) 2024-03-04T11:32:20.891 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:32:20.908+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:32:20.908937+0000) 2024-03-04T11:32:21.892 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:32:21.908+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:32:21.909176+0000) 2024-03-04T11:32:22.892 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:32:22.908+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:32:22.909413+0000) 2024-03-04T11:32:23.892 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:32:23.909+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:32:23.909655+0000) 2024-03-04T11:32:24.271 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:32:24.279 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:32:24.892 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:32:24.909+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:32:24.909920+0000) 2024-03-04T11:32:25.893 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:32:25.909+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:32:25.910161+0000) 2024-03-04T11:32:26.893 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:32:26.909+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:32:26.910403+0000) 2024-03-04T11:32:27.893 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:32:27.909+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:32:27.910629+0000) 2024-03-04T11:32:28.894 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:32:28.910+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:32:28.910918+0000) 2024-03-04T11:32:29.894 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:32:29.910+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:32:29.911101+0000) 2024-03-04T11:32:30.894 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:32:30.910+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:32:30.911292+0000) 2024-03-04T11:32:31.894 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:32:31.911+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:32:31.911533+0000) 2024-03-04T11:32:32.894 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:32:32.910+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:32:32.911780+0000) 2024-03-04T11:32:33.895 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:32:33.911+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:32:33.912016+0000) 2024-03-04T11:32:34.895 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:32:34.911+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:32:34.912248+0000) 2024-03-04T11:32:35.895 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:32:35.912+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:32:35.912489+0000) 2024-03-04T11:32:36.895 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:32:36.912+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:32:36.912742+0000) 2024-03-04T11:32:37.895 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:32:37.912+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:32:37.913004+0000) 2024-03-04T11:32:38.896 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:32:38.912+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:32:38.913192+0000) 2024-03-04T11:32:39.896 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:32:39.913+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:32:39.913427+0000) 2024-03-04T11:32:40.897 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:32:40.913+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:32:40.913662+0000) 2024-03-04T11:32:41.897 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:32:41.913+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:32:41.913936+0000) 2024-03-04T11:32:42.897 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:32:42.913+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:32:42.914211+0000) 2024-03-04T11:32:43.897 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:32:43.913+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:32:43.914435+0000) 2024-03-04T11:32:44.897 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:32:44.913+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:32:44.914658+0000) 2024-03-04T11:32:45.897 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:32:45.914+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:32:45.914972+0000) 2024-03-04T11:32:46.898 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:32:46.915+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:32:46.915277+0000) 2024-03-04T11:32:47.898 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:32:47.915+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:32:47.915561+0000) 2024-03-04T11:32:48.898 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:32:48.915+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:32:48.915896+0000) 2024-03-04T11:32:49.899 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:32:49.916+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:32:49.916186+0000) 2024-03-04T11:32:50.899 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:32:50.915+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:32:50.916437+0000) 2024-03-04T11:32:51.899 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:32:51.915+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:32:51.916677+0000) 2024-03-04T11:32:52.899 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:32:52.915+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:32:52.916953+0000) 2024-03-04T11:32:53.900 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:32:53.916+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:32:53.917193+0000) 2024-03-04T11:32:54.415 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:32:54.423 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:32:54.900 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:32:54.916+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:32:54.917430+0000) 2024-03-04T11:32:55.900 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:32:55.916+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:32:55.917663+0000) 2024-03-04T11:32:56.900 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:32:56.916+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:32:56.917944+0000) 2024-03-04T11:32:57.901 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:32:57.917+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:32:57.918194+0000) 2024-03-04T11:32:58.901 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:32:58.917+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:32:58.918436+0000) 2024-03-04T11:32:59.901 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:32:59.916+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:32:59.918676+0000) 2024-03-04T11:33:00.901 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:33:00.917+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:33:00.918934+0000) 2024-03-04T11:33:01.901 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:33:01.918+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:33:01.919172+0000) 2024-03-04T11:33:02.902 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:33:02.918+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:33:02.919439+0000) 2024-03-04T11:33:03.902 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:33:03.918+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:33:03.919677+0000) 2024-03-04T11:33:04.902 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:33:04.919+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:33:04.919947+0000) 2024-03-04T11:33:05.902 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:33:05.919+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:33:05.920185+0000) 2024-03-04T11:33:06.903 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:33:06.919+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:33:06.920430+0000) 2024-03-04T11:33:07.903 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:33:07.919+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:33:07.920667+0000) 2024-03-04T11:33:08.903 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:33:08.920+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:33:08.920985+0000) 2024-03-04T11:33:09.903 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:33:09.920+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:33:09.921236+0000) 2024-03-04T11:33:10.904 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:33:10.920+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:33:10.921472+0000) 2024-03-04T11:33:11.904 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:33:11.920+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:33:11.921735+0000) 2024-03-04T11:33:12.904 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:33:12.921+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:33:12.921972+0000) 2024-03-04T11:33:13.904 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:33:13.921+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:33:13.922173+0000) 2024-03-04T11:33:14.905 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:33:14.921+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:33:14.922433+0000) 2024-03-04T11:33:15.905 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:33:15.921+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:33:15.922679+0000) 2024-03-04T11:33:16.905 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:33:16.922+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:33:16.922925+0000) 2024-03-04T11:33:17.905 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:33:17.922+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:33:17.923163+0000) 2024-03-04T11:33:18.906 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:33:18.922+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:33:18.923349+0000) 2024-03-04T11:33:19.906 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:33:19.922+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:33:19.923583+0000) 2024-03-04T11:33:20.907 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:33:20.923+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:33:20.923931+0000) 2024-03-04T11:33:21.906 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:33:21.923+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:33:21.924186+0000) 2024-03-04T11:33:22.906 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:33:22.923+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:33:22.924418+0000) 2024-03-04T11:33:23.907 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:33:23.924+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:33:23.924660+0000) 2024-03-04T11:33:24.613 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:33:24.621 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:33:24.907 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:33:24.924+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:33:24.924975+0000) 2024-03-04T11:33:25.907 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:33:25.924+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:33:25.925201+0000) 2024-03-04T11:33:26.908 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:33:26.924+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:33:26.925498+0000) 2024-03-04T11:33:27.908 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:33:27.924+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:33:27.925725+0000) 2024-03-04T11:33:28.908 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:33:28.925+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:33:28.926034+0000) 2024-03-04T11:33:29.908 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:33:29.925+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:33:29.926319+0000) 2024-03-04T11:33:30.909 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:33:30.926+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:33:30.926621+0000) 2024-03-04T11:33:31.909 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:33:31.926+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:33:31.926950+0000) 2024-03-04T11:33:32.909 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:33:32.926+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:33:32.927183+0000) 2024-03-04T11:33:33.909 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:33:33.926+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:33:33.927421+0000) 2024-03-04T11:33:34.910 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:33:34.927+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:33:34.927660+0000) 2024-03-04T11:33:35.910 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:33:35.927+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:33:35.927943+0000) 2024-03-04T11:33:36.910 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:33:36.927+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:33:36.928189+0000) 2024-03-04T11:33:37.910 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:33:37.928+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:33:37.928421+0000) 2024-03-04T11:33:38.911 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:33:38.928+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:33:38.928659+0000) 2024-03-04T11:33:39.911 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:33:39.928+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:33:39.928934+0000) 2024-03-04T11:33:40.911 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:33:40.928+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:33:40.929189+0000) 2024-03-04T11:33:41.911 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:33:41.929+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:33:41.929428+0000) 2024-03-04T11:33:42.912 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:33:42.928+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:33:42.929675+0000) 2024-03-04T11:33:43.912 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:33:43.929+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:33:43.929943+0000) 2024-03-04T11:33:44.912 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:33:44.929+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:33:44.930179+0000) 2024-03-04T11:33:45.912 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:33:45.930+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:33:45.930425+0000) 2024-03-04T11:33:46.913 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:33:46.930+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:33:46.930670+0000) 2024-03-04T11:33:47.913 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:33:47.930+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:33:47.930938+0000) 2024-03-04T11:33:48.913 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:33:48.930+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:33:48.931167+0000) 2024-03-04T11:33:49.913 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:33:49.931+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:33:49.931410+0000) 2024-03-04T11:33:50.913 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:33:50.931+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:33:50.931650+0000) 2024-03-04T11:33:51.914 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:33:51.931+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:33:51.931945+0000) 2024-03-04T11:33:52.914 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:33:52.931+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:33:52.932217+0000) 2024-03-04T11:33:53.914 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:33:53.931+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:33:53.932514+0000) 2024-03-04T11:33:54.751 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:33:54.759 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:33:54.914 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:33:54.930+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:33:54.932803+0000) 2024-03-04T11:33:55.915 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:33:55.931+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:33:55.933095+0000) 2024-03-04T11:33:56.915 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:33:56.931+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:33:56.933350+0000) 2024-03-04T11:33:57.915 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:33:57.931+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:33:57.933578+0000) 2024-03-04T11:33:58.916 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:33:58.932+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:33:58.933933+0000) 2024-03-04T11:33:59.916 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:33:59.932+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:33:59.934216+0000) 2024-03-04T11:34:00.916 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:34:00.932+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:34:00.934456+0000) 2024-03-04T11:34:01.916 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:34:01.932+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:34:01.934759+0000) 2024-03-04T11:34:02.917 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:34:02.934+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:34:02.935040+0000) 2024-03-04T11:34:03.917 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:34:03.933+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:34:03.935262+0000) 2024-03-04T11:34:04.917 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:34:04.934+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:34:04.935500+0000) 2024-03-04T11:34:05.917 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:34:05.934+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:34:05.935787+0000) 2024-03-04T11:34:06.918 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:34:06.935+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:34:06.936130+0000) 2024-03-04T11:34:07.918 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:34:07.935+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:34:07.936422+0000) 2024-03-04T11:34:08.918 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:34:08.935+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:34:08.936715+0000) 2024-03-04T11:34:09.919 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:34:09.936+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:34:09.937013+0000) 2024-03-04T11:34:10.919 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:34:10.936+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:34:10.937235+0000) 2024-03-04T11:34:11.919 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:34:11.936+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:34:11.937542+0000) 2024-03-04T11:34:12.920 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:34:12.937+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:34:12.937878+0000) 2024-03-04T11:34:13.920 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:34:13.937+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:34:13.938158+0000) 2024-03-04T11:34:14.920 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:34:14.937+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:34:14.938453+0000) 2024-03-04T11:34:15.920 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:34:15.937+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:34:15.938734+0000) 2024-03-04T11:34:16.921 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:34:16.938+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:34:16.939041+0000) 2024-03-04T11:34:17.921 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:34:17.938+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:34:17.939317+0000) 2024-03-04T11:34:18.921 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:34:18.938+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:34:18.939607+0000) 2024-03-04T11:34:19.921 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:34:19.939+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:34:19.939946+0000) 2024-03-04T11:34:20.922 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:34:20.939+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:34:20.940238+0000) 2024-03-04T11:34:21.922 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:34:21.939+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:34:21.940527+0000) 2024-03-04T11:34:22.922 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:34:22.940+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:34:22.940815+0000) 2024-03-04T11:34:23.923 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:34:23.940+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:34:23.941155+0000) 2024-03-04T11:34:24.923 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:34:24.940+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:34:24.941467+0000) 2024-03-04T11:34:24.948 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:34:24.956 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:34:25.923 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:34:25.940+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:34:25.941782+0000) 2024-03-04T11:34:26.924 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:34:26.941+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:34:26.942125+0000) 2024-03-04T11:34:27.924 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:34:27.941+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:34:27.942430+0000) 2024-03-04T11:34:28.924 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:34:28.942+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:34:28.942681+0000) 2024-03-04T11:34:29.924 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:34:29.942+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:34:29.942925+0000) 2024-03-04T11:34:30.929 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:34:30.942+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:34:30.943159+0000) 2024-03-04T11:34:31.925 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:34:31.942+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:34:31.943411+0000) 2024-03-04T11:34:32.925 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:34:32.943+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:34:32.943645+0000) 2024-03-04T11:34:33.925 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:34:33.943+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:34:33.943928+0000) 2024-03-04T11:34:34.926 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:34:34.943+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:34:34.944197+0000) 2024-03-04T11:34:35.926 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:34:35.943+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:34:35.944434+0000) 2024-03-04T11:34:36.926 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:34:36.944+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:34:36.944685+0000) 2024-03-04T11:34:37.926 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:34:37.944+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:34:37.944923+0000) 2024-03-04T11:34:38.926 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:34:38.944+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:34:38.945164+0000) 2024-03-04T11:34:39.928 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:34:39.945+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:34:39.945410+0000) 2024-03-04T11:34:40.927 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:34:40.944+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:34:40.945637+0000) 2024-03-04T11:34:41.927 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:34:41.945+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:34:41.945925+0000) 2024-03-04T11:34:42.927 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:34:42.945+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:34:42.946181+0000) 2024-03-04T11:34:43.928 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:34:43.946+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:34:43.946429+0000) 2024-03-04T11:34:44.928 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:34:44.946+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:34:44.946682+0000) 2024-03-04T11:34:45.928 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:34:45.946+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:34:45.946957+0000) 2024-03-04T11:34:46.928 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:34:46.946+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:34:46.947224+0000) 2024-03-04T11:34:47.929 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:34:47.947+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:34:47.947475+0000) 2024-03-04T11:34:48.929 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:34:48.947+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:34:48.947725+0000) 2024-03-04T11:34:49.929 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:34:49.947+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:34:49.948035+0000) 2024-03-04T11:34:50.930 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:34:50.948+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:34:50.948291+0000) 2024-03-04T11:34:51.930 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:34:51.948+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:34:51.948563+0000) 2024-03-04T11:34:52.930 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:34:52.948+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:34:52.948893+0000) 2024-03-04T11:34:53.930 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:34:53.948+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:34:53.949157+0000) 2024-03-04T11:34:54.931 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:34:54.949+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:34:54.949390+0000) 2024-03-04T11:34:55.098 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:34:55.106 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:34:55.931 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:34:55.949+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:34:55.949628+0000) 2024-03-04T11:34:56.931 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:34:56.948+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:34:56.949927+0000) 2024-03-04T11:34:57.931 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:34:57.949+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:34:57.950167+0000) 2024-03-04T11:34:58.932 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:34:58.949+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:34:58.950408+0000) 2024-03-04T11:34:59.932 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:34:59.949+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:34:59.950652+0000) 2024-03-04T11:35:00.932 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:35:00.949+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:35:00.950933+0000) 2024-03-04T11:35:01.932 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:35:01.950+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:35:01.951180+0000) 2024-03-04T11:35:02.933 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:35:02.950+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:35:02.951413+0000) 2024-03-04T11:35:03.933 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:35:03.950+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:35:03.951656+0000) 2024-03-04T11:35:04.933 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:35:04.950+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:35:04.951926+0000) 2024-03-04T11:35:05.935 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:35:05.951+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:35:05.952170+0000) 2024-03-04T11:35:06.934 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:35:06.951+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:35:06.952415+0000) 2024-03-04T11:35:07.934 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:35:07.951+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:35:07.952649+0000) 2024-03-04T11:35:08.934 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:35:08.952+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:35:08.952929+0000) 2024-03-04T11:35:09.934 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:35:09.952+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:35:09.953117+0000) 2024-03-04T11:35:10.934 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:35:10.952+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:35:10.953306+0000) 2024-03-04T11:35:11.935 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:35:11.952+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:35:11.953571+0000) 2024-03-04T11:35:12.935 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:35:12.953+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:35:12.953866+0000) 2024-03-04T11:35:13.936 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:35:13.953+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:35:13.954102+0000) 2024-03-04T11:35:14.935 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:35:14.953+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:35:14.954300+0000) 2024-03-04T11:35:15.936 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:35:15.953+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:35:15.954537+0000) 2024-03-04T11:35:16.936 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:35:16.953+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:35:16.954812+0000) 2024-03-04T11:35:17.936 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:35:17.954+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:35:17.955092+0000) 2024-03-04T11:35:18.936 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:35:18.954+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:35:18.955297+0000) 2024-03-04T11:35:19.937 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:35:19.954+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:35:19.955539+0000) 2024-03-04T11:35:20.937 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:35:20.954+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:35:20.955729+0000) 2024-03-04T11:35:21.937 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:35:21.955+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:35:21.955971+0000) 2024-03-04T11:35:22.937 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:35:22.955+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:35:22.956225+0000) 2024-03-04T11:35:23.937 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:35:23.955+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:35:23.956483+0000) 2024-03-04T11:35:24.938 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:35:24.955+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:35:24.956714+0000) 2024-03-04T11:35:25.300 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:35:25.308 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:35:25.938 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:35:25.956+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:35:25.956961+0000) 2024-03-04T11:35:26.938 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:35:26.956+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:35:26.957168+0000) 2024-03-04T11:35:27.938 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:35:27.956+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:35:27.957410+0000) 2024-03-04T11:35:28.939 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:35:28.956+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:35:28.957644+0000) 2024-03-04T11:35:29.939 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:35:29.957+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:35:29.957937+0000) 2024-03-04T11:35:30.939 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:35:30.957+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:35:30.958157+0000) 2024-03-04T11:35:31.939 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:35:31.957+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:35:31.958416+0000) 2024-03-04T11:35:32.940 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:35:32.957+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:35:32.958666+0000) 2024-03-04T11:35:33.940 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:35:33.958+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:35:33.958949+0000) 2024-03-04T11:35:34.940 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:35:34.958+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:35:34.959160+0000) 2024-03-04T11:35:35.940 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:35:35.958+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:35:35.959438+0000) 2024-03-04T11:35:36.941 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:35:36.959+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:35:36.959701+0000) 2024-03-04T11:35:37.941 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:35:37.959+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:35:37.959964+0000) 2024-03-04T11:35:38.941 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:35:38.959+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:35:38.960213+0000) 2024-03-04T11:35:39.941 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:35:39.960+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:35:39.960500+0000) 2024-03-04T11:35:40.942 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:35:40.959+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:35:40.960769+0000) 2024-03-04T11:35:41.943 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:35:41.960+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:35:41.961029+0000) 2024-03-04T11:35:42.943 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:35:42.960+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:35:42.961285+0000) 2024-03-04T11:35:43.942 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:35:43.961+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:35:43.961538+0000) 2024-03-04T11:35:44.943 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:35:44.960+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:35:44.961778+0000) 2024-03-04T11:35:45.943 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:35:45.961+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:35:45.961984+0000) 2024-03-04T11:35:46.943 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:35:46.961+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:35:46.962264+0000) 2024-03-04T11:35:47.943 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:35:47.962+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:35:47.962544+0000) 2024-03-04T11:35:48.944 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:35:48.962+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:35:48.962801+0000) 2024-03-04T11:35:49.944 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:35:49.962+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:35:49.963101+0000) 2024-03-04T11:35:50.944 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:35:50.962+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:35:50.963303+0000) 2024-03-04T11:35:51.944 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:35:51.962+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:35:51.963546+0000) 2024-03-04T11:35:52.944 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:35:52.963+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:35:52.963798+0000) 2024-03-04T11:35:53.945 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:35:53.963+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:35:53.964121+0000) 2024-03-04T11:35:54.945 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:35:54.964+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:35:54.964393+0000) 2024-03-04T11:35:55.512 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:35:55.520 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:35:55.945 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:35:55.964+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:35:55.964652+0000) 2024-03-04T11:35:56.946 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:35:56.964+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:35:56.964946+0000) 2024-03-04T11:35:57.946 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:35:57.965+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:35:57.965198+0000) 2024-03-04T11:35:58.948 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:35:58.964+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:35:58.965446+0000) 2024-03-04T11:35:59.946 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:35:59.964+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:35:59.965705+0000) 2024-03-04T11:36:00.947 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:36:00.964+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:36:00.965929+0000) 2024-03-04T11:36:01.947 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:36:01.965+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:36:01.966192+0000) 2024-03-04T11:36:02.947 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:36:02.965+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:36:02.966440+0000) 2024-03-04T11:36:03.947 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:36:03.965+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:36:03.966693+0000) 2024-03-04T11:36:04.948 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:36:04.965+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:36:04.966943+0000) 2024-03-04T11:36:05.948 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:36:05.966+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:36:05.967195+0000) 2024-03-04T11:36:06.948 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:36:06.966+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:36:06.967455+0000) 2024-03-04T11:36:07.948 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:36:07.966+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:36:07.967721+0000) 2024-03-04T11:36:08.949 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:36:08.967+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:36:08.967982+0000) 2024-03-04T11:36:09.949 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:36:09.967+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:36:09.968246+0000) 2024-03-04T11:36:10.949 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:36:10.967+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:36:10.968500+0000) 2024-03-04T11:36:11.949 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:36:11.967+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:36:11.968778+0000) 2024-03-04T11:36:12.950 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:36:12.968+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:36:12.969081+0000) 2024-03-04T11:36:13.950 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:36:13.968+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:36:13.969339+0000) 2024-03-04T11:36:14.950 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:36:14.968+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:36:14.969616+0000) 2024-03-04T11:36:15.950 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:36:15.969+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:36:15.969939+0000) 2024-03-04T11:36:16.951 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:36:16.969+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:36:16.970239+0000) 2024-03-04T11:36:17.951 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:36:17.969+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:36:17.970526+0000) 2024-03-04T11:36:18.951 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:36:18.970+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:36:18.970888+0000) 2024-03-04T11:36:19.952 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:36:19.970+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:36:19.971191+0000) 2024-03-04T11:36:20.952 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:36:20.970+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:36:20.971492+0000) 2024-03-04T11:36:21.953 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:36:21.971+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:36:21.971796+0000) 2024-03-04T11:36:22.953 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:36:22.971+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:36:22.972127+0000) 2024-03-04T11:36:23.953 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:36:23.971+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:36:23.972421+0000) 2024-03-04T11:36:24.955 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:36:24.971+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:36:24.972733+0000) 2024-03-04T11:36:25.708 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:36:25.716 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:36:25.953 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:36:25.972+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:36:25.973051+0000) 2024-03-04T11:36:26.954 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:36:26.972+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:36:26.973318+0000) 2024-03-04T11:36:27.954 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:36:27.972+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:36:27.973553+0000) 2024-03-04T11:36:28.954 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:36:28.973+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:36:28.973854+0000) 2024-03-04T11:36:29.955 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:36:29.973+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:36:29.974096+0000) 2024-03-04T11:36:30.955 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:36:30.973+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:36:30.974346+0000) 2024-03-04T11:36:31.955 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:36:31.974+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:36:31.974605+0000) 2024-03-04T11:36:32.955 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:36:32.974+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:36:32.974893+0000) 2024-03-04T11:36:33.956 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:36:33.974+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:36:33.975148+0000) 2024-03-04T11:36:34.956 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:36:34.974+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:36:34.975351+0000) 2024-03-04T11:36:35.956 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:36:35.974+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:36:35.975631+0000) 2024-03-04T11:36:36.956 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:36:36.975+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:36:36.975930+0000) 2024-03-04T11:36:37.956 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:36:37.975+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:36:37.976182+0000) 2024-03-04T11:36:38.957 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:36:38.975+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:36:38.976429+0000) 2024-03-04T11:36:39.957 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:36:39.976+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:36:39.976685+0000) 2024-03-04T11:36:40.957 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:36:40.976+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:36:40.976934+0000) 2024-03-04T11:36:41.958 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:36:41.976+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:36:41.977261+0000) 2024-03-04T11:36:42.958 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:36:42.977+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:36:42.977516+0000) 2024-03-04T11:36:43.958 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:36:43.977+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:36:43.977771+0000) 2024-03-04T11:36:44.958 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:36:44.977+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:36:44.978028+0000) 2024-03-04T11:36:45.959 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:36:45.977+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:36:45.978276+0000) 2024-03-04T11:36:46.959 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:36:46.978+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:36:46.978517+0000) 2024-03-04T11:36:47.959 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:36:47.978+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:36:47.978753+0000) 2024-03-04T11:36:48.959 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:36:48.978+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:36:48.978987+0000) 2024-03-04T11:36:49.959 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:36:49.978+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:36:49.979178+0000) 2024-03-04T11:36:50.960 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:36:50.979+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:36:50.979436+0000) 2024-03-04T11:36:51.960 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:36:51.979+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:36:51.979680+0000) 2024-03-04T11:36:52.960 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:36:52.979+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:36:52.979945+0000) 2024-03-04T11:36:53.961 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:36:53.979+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:36:53.980180+0000) 2024-03-04T11:36:54.961 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:36:54.979+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:36:54.980409+0000) 2024-03-04T11:36:55.961 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:36:55.979+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:36:55.980630+0000) 2024-03-04T11:36:56.131 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:36:56.143 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:36:56.961 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:36:56.980+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:36:56.980919+0000) 2024-03-04T11:36:57.961 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:36:57.980+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:36:57.981160+0000) 2024-03-04T11:36:58.964 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:36:58.981+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:36:58.981393+0000) 2024-03-04T11:36:59.962 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:36:59.980+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:36:59.981640+0000) 2024-03-04T11:37:00.962 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:37:00.981+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:37:00.981917+0000) 2024-03-04T11:37:01.962 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:37:01.981+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:37:01.982154+0000) 2024-03-04T11:37:02.963 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:37:02.981+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:37:02.982384+0000) 2024-03-04T11:37:03.963 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:37:03.981+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:37:03.982647+0000) 2024-03-04T11:37:04.963 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:37:04.981+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:37:04.982917+0000) 2024-03-04T11:37:05.963 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:37:05.982+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:37:05.983175+0000) 2024-03-04T11:37:06.964 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:37:06.982+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:37:06.983436+0000) 2024-03-04T11:37:07.964 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:37:07.982+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:37:07.983696+0000) 2024-03-04T11:37:08.964 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:37:08.982+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:37:08.983936+0000) 2024-03-04T11:37:09.964 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:37:09.983+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:37:09.984187+0000) 2024-03-04T11:37:10.965 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:37:10.983+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:37:10.984435+0000) 2024-03-04T11:37:11.965 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:37:11.983+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:37:11.984695+0000) 2024-03-04T11:37:12.965 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:37:12.984+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:37:12.984942+0000) 2024-03-04T11:37:13.965 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:37:13.984+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:37:13.985200+0000) 2024-03-04T11:37:14.965 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:37:14.984+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:37:14.985466+0000) 2024-03-04T11:37:15.966 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:37:15.984+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:37:15.985718+0000) 2024-03-04T11:37:16.966 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:37:16.985+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:37:16.985982+0000) 2024-03-04T11:37:17.966 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:37:17.985+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:37:17.986233+0000) 2024-03-04T11:37:18.966 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:37:18.985+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:37:18.986423+0000) 2024-03-04T11:37:19.967 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:37:19.985+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:37:19.986693+0000) 2024-03-04T11:37:20.967 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:37:20.986+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:37:20.986962+0000) 2024-03-04T11:37:21.967 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:37:21.986+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:37:21.987214+0000) 2024-03-04T11:37:22.967 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:37:22.986+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:37:22.987463+0000) 2024-03-04T11:37:23.968 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:37:23.986+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:37:23.987656+0000) 2024-03-04T11:37:24.968 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:37:24.987+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:37:24.987946+0000) 2024-03-04T11:37:25.968 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:37:25.987+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:37:25.988196+0000) 2024-03-04T11:37:26.336 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:37:26.344 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:37:26.968 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:37:26.987+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:37:26.988455+0000) 2024-03-04T11:37:27.973 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:37:27.988+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:37:27.988705+0000) 2024-03-04T11:37:28.969 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:37:28.988+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:37:28.988955+0000) 2024-03-04T11:37:29.969 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:37:29.988+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:37:29.989263+0000) 2024-03-04T11:37:30.969 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:37:30.988+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:37:30.989509+0000) 2024-03-04T11:37:31.970 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:37:31.989+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:37:31.989751+0000) 2024-03-04T11:37:32.970 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:37:32.989+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:37:32.989982+0000) 2024-03-04T11:37:33.970 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:37:33.989+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:37:33.990216+0000) 2024-03-04T11:37:34.970 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:37:34.989+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:37:34.990450+0000) 2024-03-04T11:37:35.971 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:37:35.990+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:37:35.990685+0000) 2024-03-04T11:37:36.971 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:37:36.990+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:37:36.990931+0000) 2024-03-04T11:37:37.971 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:37:37.990+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:37:37.991185+0000) 2024-03-04T11:37:38.971 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:37:38.990+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:37:38.991415+0000) 2024-03-04T11:37:39.972 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:37:39.991+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:37:39.991657+0000) 2024-03-04T11:37:40.972 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:37:40.991+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:37:40.991926+0000) 2024-03-04T11:37:41.972 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:37:41.991+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:37:41.992159+0000) 2024-03-04T11:37:42.972 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:37:42.991+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:37:42.992399+0000) 2024-03-04T11:37:43.973 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:37:43.992+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:37:43.992642+0000) 2024-03-04T11:37:44.973 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:37:44.992+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:37:44.992908+0000) 2024-03-04T11:37:45.973 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:37:45.992+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:37:45.993147+0000) 2024-03-04T11:37:46.973 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:37:46.993+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:37:46.993395+0000) 2024-03-04T11:37:47.973 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:37:47.993+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:37:47.993640+0000) 2024-03-04T11:37:48.974 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:37:48.993+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:37:48.993911+0000) 2024-03-04T11:37:49.974 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:37:49.993+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:37:49.994176+0000) 2024-03-04T11:37:50.974 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:37:50.994+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:37:50.994452+0000) 2024-03-04T11:37:51.975 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:37:51.993+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:37:51.994688+0000) 2024-03-04T11:37:52.975 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:37:52.994+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:37:52.994919+0000) 2024-03-04T11:37:53.975 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:37:53.994+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:37:53.995167+0000) 2024-03-04T11:37:54.975 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:37:54.995+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:37:54.995401+0000) 2024-03-04T11:37:55.976 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:37:55.995+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:37:55.995639+0000) 2024-03-04T11:37:56.549 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:37:56.557 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:37:56.976 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:37:56.995+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:37:56.995921+0000) 2024-03-04T11:37:57.976 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:37:57.995+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:37:57.996161+0000) 2024-03-04T11:37:58.976 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:37:58.996+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:37:58.996393+0000) 2024-03-04T11:37:59.976 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:37:59.995+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:37:59.996624+0000) 2024-03-04T11:38:00.977 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:38:00.996+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:38:00.996914+0000) 2024-03-04T11:38:01.977 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:38:01.997+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:38:01.997160+0000) 2024-03-04T11:38:02.977 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:38:02.996+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:38:02.997424+0000) 2024-03-04T11:38:03.977 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:38:03.996+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:38:03.997649+0000) 2024-03-04T11:38:04.978 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:38:04.996+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:38:04.997976+0000) 2024-03-04T11:38:05.978 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:38:05.997+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:38:05.998268+0000) 2024-03-04T11:38:06.978 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:38:06.997+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:38:06.998593+0000) 2024-03-04T11:38:07.979 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:38:07.997+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:38:07.998922+0000) 2024-03-04T11:38:08.979 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:38:08.997+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:38:08.999205+0000) 2024-03-04T11:38:09.979 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:38:09.998+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:38:09.999498+0000) 2024-03-04T11:38:10.980 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:38:10.998+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:38:10.999790+0000) 2024-03-04T11:38:11.980 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:38:11.999+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:38:12.000110+0000) 2024-03-04T11:38:12.980 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:38:12.999+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:38:13.000400+0000) 2024-03-04T11:38:13.980 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:38:13.999+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:38:14.000701+0000) 2024-03-04T11:38:14.981 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:38:14.999+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:38:15.000982+0000) 2024-03-04T11:38:15.981 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:38:16.000+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:38:16.001261+0000) 2024-03-04T11:38:16.981 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:38:17.000+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:38:17.001560+0000) 2024-03-04T11:38:17.982 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:38:18.001+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:38:18.001921+0000) 2024-03-04T11:38:18.982 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:38:19.001+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:38:19.002233+0000) 2024-03-04T11:38:19.982 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:38:20.001+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:38:20.002525+0000) 2024-03-04T11:38:20.982 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:38:21.001+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:38:21.002745+0000) 2024-03-04T11:38:21.983 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:38:22.002+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:38:22.003050+0000) 2024-03-04T11:38:22.983 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:38:23.002+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:38:23.003341+0000) 2024-03-04T11:38:23.983 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:38:24.002+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:38:24.003634+0000) 2024-03-04T11:38:24.984 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:38:25.003+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:38:25.003969+0000) 2024-03-04T11:38:25.984 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:38:26.003+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:38:26.004265+0000) 2024-03-04T11:38:26.686 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:38:26.698 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:38:26.984 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:38:27.003+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:38:27.004565+0000) 2024-03-04T11:38:27.985 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:38:28.004+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:38:28.004917+0000) 2024-03-04T11:38:28.985 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:38:29.004+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:38:29.005272+0000) 2024-03-04T11:38:29.985 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:38:30.004+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:38:30.005556+0000) 2024-03-04T11:38:30.986 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:38:31.005+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:38:31.005876+0000) 2024-03-04T11:38:31.986 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:38:32.005+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:38:32.006171+0000) 2024-03-04T11:38:32.986 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:38:33.005+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:38:33.006463+0000) 2024-03-04T11:38:33.986 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:38:34.006+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:38:34.006749+0000) 2024-03-04T11:38:34.987 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:38:35.006+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:38:35.007035+0000) 2024-03-04T11:38:35.987 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:38:36.006+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:38:36.007307+0000) 2024-03-04T11:38:36.987 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:38:37.006+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:38:37.007617+0000) 2024-03-04T11:38:37.988 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:38:38.007+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:38:38.007975+0000) 2024-03-04T11:38:38.988 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:38:39.007+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:38:39.008262+0000) 2024-03-04T11:38:39.988 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:38:40.007+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:38:40.008560+0000) 2024-03-04T11:38:40.988 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:38:41.008+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:38:41.008895+0000) 2024-03-04T11:38:41.989 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:38:42.008+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:38:42.009201+0000) 2024-03-04T11:38:42.989 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:38:43.008+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:38:43.009520+0000) 2024-03-04T11:38:43.990 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:38:44.009+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:38:44.009865+0000) 2024-03-04T11:38:44.990 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:38:45.009+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:38:45.010145+0000) 2024-03-04T11:38:45.990 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:38:46.009+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:38:46.010379+0000) 2024-03-04T11:38:46.990 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:38:47.010+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:38:47.010641+0000) 2024-03-04T11:38:47.990 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:38:48.010+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:38:48.010929+0000) 2024-03-04T11:38:48.991 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:38:49.010+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:38:49.011123+0000) 2024-03-04T11:38:49.991 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:38:50.010+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:38:50.011363+0000) 2024-03-04T11:38:50.991 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:38:51.011+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:38:51.011601+0000) 2024-03-04T11:38:51.991 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:38:52.011+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:38:52.011883+0000) 2024-03-04T11:38:52.992 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:38:53.011+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:38:53.012126+0000) 2024-03-04T11:38:53.992 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:38:54.012+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:38:54.012366+0000) 2024-03-04T11:38:54.992 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:38:55.012+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:38:55.012603+0000) 2024-03-04T11:38:55.992 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:38:56.012+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:38:56.012871+0000) 2024-03-04T11:38:56.917 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:38:56.926 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:38:56.992 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:38:57.012+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:38:57.013104+0000) 2024-03-04T11:38:57.993 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:38:58.013+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:38:58.013360+0000) 2024-03-04T11:38:58.995 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:38:59.013+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:38:59.013596+0000) 2024-03-04T11:38:59.993 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:39:00.012+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:39:00.013789+0000) 2024-03-04T11:39:00.993 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:39:01.013+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:39:01.014027+0000) 2024-03-04T11:39:01.994 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:39:02.014+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:39:02.014269+0000) 2024-03-04T11:39:02.994 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:39:03.013+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:39:03.014507+0000) 2024-03-04T11:39:03.994 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:39:04.014+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:39:04.014747+0000) 2024-03-04T11:39:04.994 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:39:05.014+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:39:05.014988+0000) 2024-03-04T11:39:05.995 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:39:06.015+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:39:06.015236+0000) 2024-03-04T11:39:06.995 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:39:07.014+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:39:07.015490+0000) 2024-03-04T11:39:07.995 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:39:08.014+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:39:08.015734+0000) 2024-03-04T11:39:08.995 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:39:09.014+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:39:09.016007+0000) 2024-03-04T11:39:09.996 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:39:10.014+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:39:10.016234+0000) 2024-03-04T11:39:10.996 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:39:11.015+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:39:11.016471+0000) 2024-03-04T11:39:11.996 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:39:12.015+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:39:12.016754+0000) 2024-03-04T11:39:12.996 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:39:13.016+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:39:13.017025+0000) 2024-03-04T11:39:13.997 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:39:14.016+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:39:14.017285+0000) 2024-03-04T11:39:14.997 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:39:15.016+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:39:15.017536+0000) 2024-03-04T11:39:15.997 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:39:16.016+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:39:16.017776+0000) 2024-03-04T11:39:16.997 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:39:17.017+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:39:17.018023+0000) 2024-03-04T11:39:17.998 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:39:18.017+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:39:18.018266+0000) 2024-03-04T11:39:18.998 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:39:19.017+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:39:19.018532+0000) 2024-03-04T11:39:19.998 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:39:20.017+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:39:20.018771+0000) 2024-03-04T11:39:20.998 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:39:21.018+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:39:21.019004+0000) 2024-03-04T11:39:21.999 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:39:22.018+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:39:22.019263+0000) 2024-03-04T11:39:22.999 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:39:23.018+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:39:23.019521+0000) 2024-03-04T11:39:23.999 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:39:24.018+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:39:24.019702+0000) 2024-03-04T11:39:24.999 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:39:25.019+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:39:25.019940+0000) 2024-03-04T11:39:25.999 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:39:26.019+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:39:26.020203+0000) 2024-03-04T11:39:27.000 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:39:27.019+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:39:27.020473+0000) 2024-03-04T11:39:27.103 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:39:27.111 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:39:28.000 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:39:28.019+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:39:28.020709+0000) 2024-03-04T11:39:29.000 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:39:29.020+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:39:29.020952+0000) 2024-03-04T11:39:30.000 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:39:30.020+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:39:30.021183+0000) 2024-03-04T11:39:31.000 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:39:31.020+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:39:31.021423+0000) 2024-03-04T11:39:32.001 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:39:32.020+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:39:32.021651+0000) 2024-03-04T11:39:33.001 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:39:33.021+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:39:33.021948+0000) 2024-03-04T11:39:34.001 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:39:34.021+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:39:34.022186+0000) 2024-03-04T11:39:35.002 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:39:35.021+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:39:35.022423+0000) 2024-03-04T11:39:36.002 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:39:36.022+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:39:36.022663+0000) 2024-03-04T11:39:37.002 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:39:37.022+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:39:37.022950+0000) 2024-03-04T11:39:38.002 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:39:38.022+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:39:38.023154+0000) 2024-03-04T11:39:39.003 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:39:39.022+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:39:39.023396+0000) 2024-03-04T11:39:40.003 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:39:40.023+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:39:40.023634+0000) 2024-03-04T11:39:41.003 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:39:41.023+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:39:41.023940+0000) 2024-03-04T11:39:42.003 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:39:42.023+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:39:42.024230+0000) 2024-03-04T11:39:43.004 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:39:43.023+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:39:43.024547+0000) 2024-03-04T11:39:44.004 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:39:44.024+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:39:44.024919+0000) 2024-03-04T11:39:45.004 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:39:45.024+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:39:45.025212+0000) 2024-03-04T11:39:46.005 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:39:46.024+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:39:46.025460+0000) 2024-03-04T11:39:47.005 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:39:47.025+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:39:47.025767+0000) 2024-03-04T11:39:48.005 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:39:48.025+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:39:48.026057+0000) 2024-03-04T11:39:49.005 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:39:49.025+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:39:49.026332+0000) 2024-03-04T11:39:50.006 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:39:50.025+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:39:50.026641+0000) 2024-03-04T11:39:51.006 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:39:51.026+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:39:51.026967+0000) 2024-03-04T11:39:52.006 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:39:52.026+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:39:52.027259+0000) 2024-03-04T11:39:53.007 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:39:53.027+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:39:53.027549+0000) 2024-03-04T11:39:54.007 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:39:54.027+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:39:54.027873+0000) 2024-03-04T11:39:55.007 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:39:55.027+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:39:55.028163+0000) 2024-03-04T11:39:56.007 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:39:56.028+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:39:56.028445+0000) 2024-03-04T11:39:57.008 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:39:57.028+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:39:57.028750+0000) 2024-03-04T11:39:57.343 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:39:57.352 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:39:58.008 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:39:58.028+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:39:58.029052+0000) 2024-03-04T11:39:59.008 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:39:59.029+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:39:59.029347+0000) 2024-03-04T11:40:00.009 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:40:00.029+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:40:00.029641+0000) 2024-03-04T11:40:01.009 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:40:01.029+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:40:01.029970+0000) 2024-03-04T11:40:02.009 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:40:02.029+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:40:02.030260+0000) 2024-03-04T11:40:03.009 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:40:03.030+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:40:03.030545+0000) 2024-03-04T11:40:04.010 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:40:04.030+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:40:04.030892+0000) 2024-03-04T11:40:05.010 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:40:05.030+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:40:05.031195+0000) 2024-03-04T11:40:06.011 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:40:06.030+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:40:06.031480+0000) 2024-03-04T11:40:07.011 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:40:07.031+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:40:07.031782+0000) 2024-03-04T11:40:08.011 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:40:08.031+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:40:08.032107+0000) 2024-03-04T11:40:09.011 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:40:09.032+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:40:09.032410+0000) 2024-03-04T11:40:10.012 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:40:10.031+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:40:10.032741+0000) 2024-03-04T11:40:11.012 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:40:11.031+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:40:11.033018+0000) 2024-03-04T11:40:12.012 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:40:12.032+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:40:12.033317+0000) 2024-03-04T11:40:13.013 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:40:13.032+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:40:13.033600+0000) 2024-03-04T11:40:14.013 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:40:14.032+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:40:14.033922+0000) 2024-03-04T11:40:15.015 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:40:15.033+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:40:15.034212+0000) 2024-03-04T11:40:16.013 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:40:16.033+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:40:16.034501+0000) 2024-03-04T11:40:17.014 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:40:17.033+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:40:17.034848+0000) 2024-03-04T11:40:18.014 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:40:18.034+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:40:18.035152+0000) 2024-03-04T11:40:19.014 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:40:19.034+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:40:19.035442+0000) 2024-03-04T11:40:20.015 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:40:20.034+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:40:20.035735+0000) 2024-03-04T11:40:21.015 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:40:21.035+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:40:21.035979+0000) 2024-03-04T11:40:22.015 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:40:22.035+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:40:22.036212+0000) 2024-03-04T11:40:23.015 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:40:23.035+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:40:23.036446+0000) 2024-03-04T11:40:24.016 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:40:24.035+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:40:24.036683+0000) 2024-03-04T11:40:25.016 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:40:25.036+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:40:25.036948+0000) 2024-03-04T11:40:26.016 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:40:26.036+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:40:26.037189+0000) 2024-03-04T11:40:27.017 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:40:27.036+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:40:27.037450+0000) 2024-03-04T11:40:27.572 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:40:27.580 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:40:28.017 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:40:28.036+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:40:28.037699+0000) 2024-03-04T11:40:29.017 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:40:29.037+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:40:29.037935+0000) 2024-03-04T11:40:30.017 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:40:30.037+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:40:30.038170+0000) 2024-03-04T11:40:31.017 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:40:31.037+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:40:31.038404+0000) 2024-03-04T11:40:32.018 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:40:32.037+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:40:32.038643+0000) 2024-03-04T11:40:33.018 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:40:33.038+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:40:33.038932+0000) 2024-03-04T11:40:34.018 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:40:34.038+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:40:34.039112+0000) 2024-03-04T11:40:35.018 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:40:35.038+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:40:35.039344+0000) 2024-03-04T11:40:36.018 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:40:36.038+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:40:36.039586+0000) 2024-03-04T11:40:37.019 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:40:37.039+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:40:37.039889+0000) 2024-03-04T11:40:38.019 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:40:38.039+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:40:38.040144+0000) 2024-03-04T11:40:39.019 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:40:39.039+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:40:39.040402+0000) 2024-03-04T11:40:40.019 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:40:40.039+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:40:40.040643+0000) 2024-03-04T11:40:41.020 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:40:41.040+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:40:41.040914+0000) 2024-03-04T11:40:42.020 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:40:42.040+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:40:42.041161+0000) 2024-03-04T11:40:43.020 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:40:43.040+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:40:43.041335+0000) 2024-03-04T11:40:44.020 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:40:44.041+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:40:44.041573+0000) 2024-03-04T11:40:45.020 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:40:45.041+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:40:45.041808+0000) 2024-03-04T11:40:46.021 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:40:46.041+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:40:46.042085+0000) 2024-03-04T11:40:47.021 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:40:47.041+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:40:47.042333+0000) 2024-03-04T11:40:48.021 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:40:48.041+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:40:48.042572+0000) 2024-03-04T11:40:49.021 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:40:49.042+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:40:49.042805+0000) 2024-03-04T11:40:50.022 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:40:50.042+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:40:50.043071+0000) 2024-03-04T11:40:51.022 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:40:51.042+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:40:51.043307+0000) 2024-03-04T11:40:52.022 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:40:52.043+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:40:52.043546+0000) 2024-03-04T11:40:53.022 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:40:53.043+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:40:53.043781+0000) 2024-03-04T11:40:54.022 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:40:54.043+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:40:54.043983+0000) 2024-03-04T11:40:55.023 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:40:55.043+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:40:55.044207+0000) 2024-03-04T11:40:56.023 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:40:56.044+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:40:56.044449+0000) 2024-03-04T11:40:57.023 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:40:57.044+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:40:57.044695+0000) 2024-03-04T11:40:57.785 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:40:57.794 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:40:58.023 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:40:58.044+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:40:58.044946+0000) 2024-03-04T11:40:59.024 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:40:59.044+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:40:59.045178+0000) 2024-03-04T11:41:00.024 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:41:00.045+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:41:00.045414+0000) 2024-03-04T11:41:01.024 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:41:01.045+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:41:01.045662+0000) 2024-03-04T11:41:02.024 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:41:02.045+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:41:02.045942+0000) 2024-03-04T11:41:03.025 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:41:03.045+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:41:03.046194+0000) 2024-03-04T11:41:04.025 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:41:04.046+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:41:04.046435+0000) 2024-03-04T11:41:05.025 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:41:05.046+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:41:05.046666+0000) 2024-03-04T11:41:06.026 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:41:06.046+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:41:06.046939+0000) 2024-03-04T11:41:07.026 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:41:07.047+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:41:07.047202+0000) 2024-03-04T11:41:08.026 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:41:08.047+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:41:08.047462+0000) 2024-03-04T11:41:09.026 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:41:09.047+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:41:09.047719+0000) 2024-03-04T11:41:10.026 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:41:10.047+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:41:10.047973+0000) 2024-03-04T11:41:11.027 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:41:11.048+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:41:11.048217+0000) 2024-03-04T11:41:12.027 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:41:12.047+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:41:12.048451+0000) 2024-03-04T11:41:13.027 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:41:13.047+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:41:13.048690+0000) 2024-03-04T11:41:14.027 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:41:14.047+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:41:14.048942+0000) 2024-03-04T11:41:15.028 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:41:15.048+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:41:15.049178+0000) 2024-03-04T11:41:16.028 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:41:16.048+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:41:16.049421+0000) 2024-03-04T11:41:17.028 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:41:17.048+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:41:17.049672+0000) 2024-03-04T11:41:18.028 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:41:18.048+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:41:18.049946+0000) 2024-03-04T11:41:19.029 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:41:19.049+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:41:19.050185+0000) 2024-03-04T11:41:20.029 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:41:20.049+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:41:20.050423+0000) 2024-03-04T11:41:21.029 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:41:21.049+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:41:21.050656+0000) 2024-03-04T11:41:22.029 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:41:22.049+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:41:22.050923+0000) 2024-03-04T11:41:23.030 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:41:23.050+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:41:23.051163+0000) 2024-03-04T11:41:24.030 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:41:24.050+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:41:24.051415+0000) 2024-03-04T11:41:25.030 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:41:25.050+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:41:25.051664+0000) 2024-03-04T11:41:26.030 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:41:26.051+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:41:26.051947+0000) 2024-03-04T11:41:27.031 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:41:27.051+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:41:27.052205+0000) 2024-03-04T11:41:27.955 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:41:27.967 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:41:28.031 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:41:28.051+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:41:28.052391+0000) 2024-03-04T11:41:29.031 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:41:29.051+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:41:29.052642+0000) 2024-03-04T11:41:30.031 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:41:30.052+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:41:30.052911+0000) 2024-03-04T11:41:31.032 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:41:31.052+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:41:31.053147+0000) 2024-03-04T11:41:32.032 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:41:32.052+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:41:32.053403+0000) 2024-03-04T11:41:33.032 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:41:33.052+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:41:33.053637+0000) 2024-03-04T11:41:34.033 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:41:34.053+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:41:34.053924+0000) 2024-03-04T11:41:35.033 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:41:35.053+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:41:35.054157+0000) 2024-03-04T11:41:36.033 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:41:36.053+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:41:36.054397+0000) 2024-03-04T11:41:37.033 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:41:37.053+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:41:37.054647+0000) 2024-03-04T11:41:38.033 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:41:38.054+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:41:38.054942+0000) 2024-03-04T11:41:39.034 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:41:39.054+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:41:39.055194+0000) 2024-03-04T11:41:40.034 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:41:40.054+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:41:40.055431+0000) 2024-03-04T11:41:41.034 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:41:41.054+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:41:41.055687+0000) 2024-03-04T11:41:42.034 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:41:42.055+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:41:42.055928+0000) 2024-03-04T11:41:43.035 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:41:43.055+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:41:43.056166+0000) 2024-03-04T11:41:44.035 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:41:44.055+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:41:44.056406+0000) 2024-03-04T11:41:45.035 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:41:45.055+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:41:45.056630+0000) 2024-03-04T11:41:46.035 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:41:46.056+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:41:46.056916+0000) 2024-03-04T11:41:47.036 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:41:47.056+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:41:47.057167+0000) 2024-03-04T11:41:48.036 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:41:48.056+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:41:48.057406+0000) 2024-03-04T11:41:49.036 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:41:49.057+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:41:49.057640+0000) 2024-03-04T11:41:50.036 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:41:50.057+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:41:50.057913+0000) 2024-03-04T11:41:51.037 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:41:51.057+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:41:51.058155+0000) 2024-03-04T11:41:52.037 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:41:52.057+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:41:52.058391+0000) 2024-03-04T11:41:53.037 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:41:53.058+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:41:53.058630+0000) 2024-03-04T11:41:54.037 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:41:54.058+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:41:54.058949+0000) 2024-03-04T11:41:55.037 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:41:55.058+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:41:55.059183+0000) 2024-03-04T11:41:56.038 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:41:56.059+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:41:56.059422+0000) 2024-03-04T11:41:57.038 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:41:57.058+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:41:57.059677+0000) 2024-03-04T11:41:58.038 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:41:58.059+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:41:58.059955+0000) 2024-03-04T11:41:58.095 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:41:58.103 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:41:59.039 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:41:59.059+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:41:59.060198+0000) 2024-03-04T11:42:00.038 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:42:00.059+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:42:00.060390+0000) 2024-03-04T11:42:01.039 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:42:01.059+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:42:01.060581+0000) 2024-03-04T11:42:02.039 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:42:02.060+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:42:02.060940+0000) 2024-03-04T11:42:03.039 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:42:03.060+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:42:03.061167+0000) 2024-03-04T11:42:04.040 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:42:04.061+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:42:04.061410+0000) 2024-03-04T11:42:05.040 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:42:05.061+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:42:05.061649+0000) 2024-03-04T11:42:06.040 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:42:06.061+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:42:06.061917+0000) 2024-03-04T11:42:07.040 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:42:07.061+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:42:07.062127+0000) 2024-03-04T11:42:08.041 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:42:08.062+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:42:08.062388+0000) 2024-03-04T11:42:09.041 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:42:09.061+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:42:09.062643+0000) 2024-03-04T11:42:10.041 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:42:10.062+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:42:10.062925+0000) 2024-03-04T11:42:11.041 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:42:11.062+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:42:11.063166+0000) 2024-03-04T11:42:12.041 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:42:12.063+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:42:12.063405+0000) 2024-03-04T11:42:13.042 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:42:13.063+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:42:13.063669+0000) 2024-03-04T11:42:14.042 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:42:14.063+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:42:14.063931+0000) 2024-03-04T11:42:15.042 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:42:15.063+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:42:15.064188+0000) 2024-03-04T11:42:16.043 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:42:16.063+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:42:16.064441+0000) 2024-03-04T11:42:17.043 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:42:17.062+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:42:17.064720+0000) 2024-03-04T11:42:18.043 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:42:18.063+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:42:18.064983+0000) 2024-03-04T11:42:19.044 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:42:19.064+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:42:19.065262+0000) 2024-03-04T11:42:20.044 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:42:20.064+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:42:20.065537+0000) 2024-03-04T11:42:21.044 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:42:21.064+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:42:21.065789+0000) 2024-03-04T11:42:22.044 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:42:22.064+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:42:22.066037+0000) 2024-03-04T11:42:23.044 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:42:23.065+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:42:23.066288+0000) 2024-03-04T11:42:24.045 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:42:24.065+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:42:24.066567+0000) 2024-03-04T11:42:25.045 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:42:25.065+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:42:25.066858+0000) 2024-03-04T11:42:26.045 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:42:26.066+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:42:26.067151+0000) 2024-03-04T11:42:27.045 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:42:27.066+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:42:27.067407+0000) 2024-03-04T11:42:28.046 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:42:28.066+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:42:28.067645+0000) 2024-03-04T11:42:28.232 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:42:28.240 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:42:29.046 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:42:29.067+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:42:29.067953+0000) 2024-03-04T11:42:30.046 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:42:30.067+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:42:30.068202+0000) 2024-03-04T11:42:31.047 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:42:31.067+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:42:31.068469+0000) 2024-03-04T11:42:32.047 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:42:32.067+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:42:32.068727+0000) 2024-03-04T11:42:33.047 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:42:33.068+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:42:33.068994+0000) 2024-03-04T11:42:34.047 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:42:34.068+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:42:34.069233+0000) 2024-03-04T11:42:35.047 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:42:35.068+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:42:35.069470+0000) 2024-03-04T11:42:36.048 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:42:36.068+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:42:36.069705+0000) 2024-03-04T11:42:37.048 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:42:37.069+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:42:37.069968+0000) 2024-03-04T11:42:38.048 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:42:38.069+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:42:38.070226+0000) 2024-03-04T11:42:39.049 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:42:39.069+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:42:39.070461+0000) 2024-03-04T11:42:40.049 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:42:40.069+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:42:40.070704+0000) 2024-03-04T11:42:41.049 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:42:41.070+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:42:41.070940+0000) 2024-03-04T11:42:42.049 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:42:42.070+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:42:42.071181+0000) 2024-03-04T11:42:43.049 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:42:43.070+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:42:43.071361+0000) 2024-03-04T11:42:44.050 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:42:44.070+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:42:44.071617+0000) 2024-03-04T11:42:45.050 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:42:45.071+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:42:45.071913+0000) 2024-03-04T11:42:46.050 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:42:46.071+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:42:46.072170+0000) 2024-03-04T11:42:47.050 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:42:47.071+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:42:47.072429+0000) 2024-03-04T11:42:48.051 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:42:48.072+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:42:48.072690+0000) 2024-03-04T11:42:49.051 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:42:49.072+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:42:49.072951+0000) 2024-03-04T11:42:50.051 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:42:50.072+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:42:50.073262+0000) 2024-03-04T11:42:51.051 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:42:51.072+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:42:51.073519+0000) 2024-03-04T11:42:52.052 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:42:52.073+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:42:52.073773+0000) 2024-03-04T11:42:53.052 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:42:53.073+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:42:53.074033+0000) 2024-03-04T11:42:54.052 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:42:54.073+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:42:54.074285+0000) 2024-03-04T11:42:55.053 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:42:55.074+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:42:55.074535+0000) 2024-03-04T11:42:56.053 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:42:56.074+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:42:56.074790+0000) 2024-03-04T11:42:57.053 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:42:57.074+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:42:57.075101+0000) 2024-03-04T11:42:58.053 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:42:58.074+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:42:58.075320+0000) 2024-03-04T11:42:58.471 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:42:58.480 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:42:59.053 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:42:59.074+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:42:59.075591+0000) 2024-03-04T11:43:00.054 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:43:00.074+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:43:00.075788+0000) 2024-03-04T11:43:01.054 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:43:01.075+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:43:01.076079+0000) 2024-03-04T11:43:02.054 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:43:02.075+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:43:02.076277+0000) 2024-03-04T11:43:03.054 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:43:03.076+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:43:03.076535+0000) 2024-03-04T11:43:04.055 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:43:04.076+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:43:04.076807+0000) 2024-03-04T11:43:05.055 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:43:05.076+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:43:05.077100+0000) 2024-03-04T11:43:06.055 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:43:06.077+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:43:06.077355+0000) 2024-03-04T11:43:07.056 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:43:07.077+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:43:07.077637+0000) 2024-03-04T11:43:08.056 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:43:08.077+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:43:08.077954+0000) 2024-03-04T11:43:09.056 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:43:09.077+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:43:09.078212+0000) 2024-03-04T11:43:10.056 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:43:10.078+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:43:10.078466+0000) 2024-03-04T11:43:11.057 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:43:11.078+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:43:11.078718+0000) 2024-03-04T11:43:12.057 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:43:12.078+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:43:12.079013+0000) 2024-03-04T11:43:13.057 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:43:13.079+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:43:13.079274+0000) 2024-03-04T11:43:14.057 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:43:14.079+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:43:14.079528+0000) 2024-03-04T11:43:15.058 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:43:15.079+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:43:15.079816+0000) 2024-03-04T11:43:16.058 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:43:16.079+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:43:16.080113+0000) 2024-03-04T11:43:17.058 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:43:17.079+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:43:17.080379+0000) 2024-03-04T11:43:18.058 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:43:18.079+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:43:18.080626+0000) 2024-03-04T11:43:19.059 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:43:19.079+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:43:19.080921+0000) 2024-03-04T11:43:20.059 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:43:20.079+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:43:20.081172+0000) 2024-03-04T11:43:21.059 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:43:21.079+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:43:21.081411+0000) 2024-03-04T11:43:22.059 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:43:22.080+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:43:22.081667+0000) 2024-03-04T11:43:23.060 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:43:23.080+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:43:23.081966+0000) 2024-03-04T11:43:24.060 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:43:24.081+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:43:24.082234+0000) 2024-03-04T11:43:25.060 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:43:25.081+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:43:25.082483+0000) 2024-03-04T11:43:26.061 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:43:26.081+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:43:26.082753+0000) 2024-03-04T11:43:27.061 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:43:27.082+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:43:27.082979+0000) 2024-03-04T11:43:28.061 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:43:28.082+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:43:28.083224+0000) 2024-03-04T11:43:28.694 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:43:28.702 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:43:29.061 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:43:29.082+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:43:29.083481+0000) 2024-03-04T11:43:30.061 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:43:30.082+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:43:30.083724+0000) 2024-03-04T11:43:31.062 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:43:31.083+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:43:31.083960+0000) 2024-03-04T11:43:32.062 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:43:32.083+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:43:32.084196+0000) 2024-03-04T11:43:33.062 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:43:33.083+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:43:33.084435+0000) 2024-03-04T11:43:34.062 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:43:34.083+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:43:34.084675+0000) 2024-03-04T11:43:35.063 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:43:35.084+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:43:35.084939+0000) 2024-03-04T11:43:36.063 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:43:36.084+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:43:36.085198+0000) 2024-03-04T11:43:37.063 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:43:37.084+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:43:37.085469+0000) 2024-03-04T11:43:38.063 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:43:38.084+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:43:38.085714+0000) 2024-03-04T11:43:39.064 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:43:39.085+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:43:39.085962+0000) 2024-03-04T11:43:40.064 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:43:40.085+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:43:40.086201+0000) 2024-03-04T11:43:41.064 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:43:41.085+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:43:41.086391+0000) 2024-03-04T11:43:42.064 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:43:42.085+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:43:42.086573+0000) 2024-03-04T11:43:43.065 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:43:43.085+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:43:43.086773+0000) 2024-03-04T11:43:44.065 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:43:44.086+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:43:44.087015+0000) 2024-03-04T11:43:45.065 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:43:45.086+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:43:45.087256+0000) 2024-03-04T11:43:46.065 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:43:46.086+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:43:46.087492+0000) 2024-03-04T11:43:47.066 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:43:47.087+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:43:47.087740+0000) 2024-03-04T11:43:48.066 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:43:48.087+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:43:48.087985+0000) 2024-03-04T11:43:49.066 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:43:49.087+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:43:49.088286+0000) 2024-03-04T11:43:50.066 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:43:50.087+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:43:50.088530+0000) 2024-03-04T11:43:51.067 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:43:51.088+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:43:51.088767+0000) 2024-03-04T11:43:52.067 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:43:52.088+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:43:52.089006+0000) 2024-03-04T11:43:53.067 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:43:53.088+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:43:53.089253+0000) 2024-03-04T11:43:54.067 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:43:54.088+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:43:54.089498+0000) 2024-03-04T11:43:55.067 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:43:55.088+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:43:55.089727+0000) 2024-03-04T11:43:56.068 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:43:56.089+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:43:56.089967+0000) 2024-03-04T11:43:57.068 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:43:57.089+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:43:57.090209+0000) 2024-03-04T11:43:58.068 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:43:58.089+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:43:58.090451+0000) 2024-03-04T11:43:58.831 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:43:58.840 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:43:59.068 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:43:59.089+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:43:59.090721+0000) 2024-03-04T11:44:00.068 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:44:00.090+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:44:00.090972+0000) 2024-03-04T11:44:01.069 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:44:01.090+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:44:01.091206+0000) 2024-03-04T11:44:02.069 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:44:02.091+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:44:02.091466+0000) 2024-03-04T11:44:03.069 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:44:03.091+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:44:03.091710+0000) 2024-03-04T11:44:04.070 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:44:04.091+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:44:04.091971+0000) 2024-03-04T11:44:05.070 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:44:05.091+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:44:05.092224+0000) 2024-03-04T11:44:06.070 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:44:06.092+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:44:06.092483+0000) 2024-03-04T11:44:07.070 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:44:07.092+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:44:07.092765+0000) 2024-03-04T11:44:08.071 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:44:08.092+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:44:08.093014+0000) 2024-03-04T11:44:09.071 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:44:09.092+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:44:09.093270+0000) 2024-03-04T11:44:10.071 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:44:10.092+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:44:10.093535+0000) 2024-03-04T11:44:11.071 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:44:11.093+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:44:11.093793+0000) 2024-03-04T11:44:12.072 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:44:12.093+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:44:12.094072+0000) 2024-03-04T11:44:13.072 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:44:13.094+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:44:13.094308+0000) 2024-03-04T11:44:14.072 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:44:14.093+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:44:14.094569+0000) 2024-03-04T11:44:15.072 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:44:15.094+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:44:15.094809+0000) 2024-03-04T11:44:16.073 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:44:16.094+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:44:16.095102+0000) 2024-03-04T11:44:17.073 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:44:17.094+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:44:17.095291+0000) 2024-03-04T11:44:18.073 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:44:18.094+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:44:18.095535+0000) 2024-03-04T11:44:19.073 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:44:19.095+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:44:19.095790+0000) 2024-03-04T11:44:20.073 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:44:20.094+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:44:20.096032+0000) 2024-03-04T11:44:21.074 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:44:21.095+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:44:21.096267+0000) 2024-03-04T11:44:22.074 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:44:22.094+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:44:22.096445+0000) 2024-03-04T11:44:23.074 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:44:23.095+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:44:23.096682+0000) 2024-03-04T11:44:24.074 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:44:24.095+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:44:24.096964+0000) 2024-03-04T11:44:25.075 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:44:25.096+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:44:25.097243+0000) 2024-03-04T11:44:26.075 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:44:26.095+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:44:26.097464+0000) 2024-03-04T11:44:27.075 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:44:27.095+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:44:27.097783+0000) 2024-03-04T11:44:28.075 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:44:28.097+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:44:28.098096+0000) 2024-03-04T11:44:29.045 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:44:29.059 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:44:29.076 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:44:29.097+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:44:29.098388+0000) 2024-03-04T11:44:30.076 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:44:30.097+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:44:30.098681+0000) 2024-03-04T11:44:31.076 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:44:31.098+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:44:31.098964+0000) 2024-03-04T11:44:32.077 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:44:32.098+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:44:32.099325+0000) 2024-03-04T11:44:33.077 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:44:33.098+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:44:33.099606+0000) 2024-03-04T11:44:34.077 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:44:34.099+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:44:34.099897+0000) 2024-03-04T11:44:35.078 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:44:35.099+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:44:35.100082+0000) 2024-03-04T11:44:36.078 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:44:36.099+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:44:36.100344+0000) 2024-03-04T11:44:37.078 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:44:37.099+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:44:37.100609+0000) 2024-03-04T11:44:38.078 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:44:38.100+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:44:38.100883+0000) 2024-03-04T11:44:39.078 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:44:39.100+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:44:39.101128+0000) 2024-03-04T11:44:40.079 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:44:40.100+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:44:40.101369+0000) 2024-03-04T11:44:41.079 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:44:41.100+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:44:41.101604+0000) 2024-03-04T11:44:42.079 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:44:42.101+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:44:42.101907+0000) 2024-03-04T11:44:43.079 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:44:43.101+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:44:43.102158+0000) 2024-03-04T11:44:44.080 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:44:44.101+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:44:44.102411+0000) 2024-03-04T11:44:45.080 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:44:45.101+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:44:45.102661+0000) 2024-03-04T11:44:46.080 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:44:46.102+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:44:46.102964+0000) 2024-03-04T11:44:47.081 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:44:47.102+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:44:47.103250+0000) 2024-03-04T11:44:48.081 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:44:48.102+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:44:48.103510+0000) 2024-03-04T11:44:49.081 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:44:49.103+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:44:49.103766+0000) 2024-03-04T11:44:50.082 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:44:50.103+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:44:50.104137+0000) 2024-03-04T11:44:51.082 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:44:51.103+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:44:51.104438+0000) 2024-03-04T11:44:52.082 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:44:52.104+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:44:52.104721+0000) 2024-03-04T11:44:53.082 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:44:53.104+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:44:53.105010+0000) 2024-03-04T11:44:54.083 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:44:54.104+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:44:54.105302+0000) 2024-03-04T11:44:55.083 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:44:55.104+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:44:55.105582+0000) 2024-03-04T11:44:56.083 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:44:56.105+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:44:56.105899+0000) 2024-03-04T11:44:57.083 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:44:57.105+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:44:57.106203+0000) 2024-03-04T11:44:58.084 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:44:58.105+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:44:58.106469+0000) 2024-03-04T11:44:59.084 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:44:59.106+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:44:59.106724+0000) 2024-03-04T11:44:59.199 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:44:59.208 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:45:00.084 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:45:00.106+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:45:00.106965+0000) 2024-03-04T11:45:01.084 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:45:01.106+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:45:01.107199+0000) 2024-03-04T11:45:02.085 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:45:02.106+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:45:02.107439+0000) 2024-03-04T11:45:03.085 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:45:03.107+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:45:03.107692+0000) 2024-03-04T11:45:04.085 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:45:04.107+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:45:04.107953+0000) 2024-03-04T11:45:05.085 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:45:05.107+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:45:05.108126+0000) 2024-03-04T11:45:06.086 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:45:06.107+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:45:06.108431+0000) 2024-03-04T11:45:07.086 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:45:07.108+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:45:07.108718+0000) 2024-03-04T11:45:08.086 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:45:08.108+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:45:08.108949+0000) 2024-03-04T11:45:09.087 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:45:09.108+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:45:09.109189+0000) 2024-03-04T11:45:10.087 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:45:10.109+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:45:10.109428+0000) 2024-03-04T11:45:11.087 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:45:11.108+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:45:11.109679+0000) 2024-03-04T11:45:12.087 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:45:12.109+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:45:12.109945+0000) 2024-03-04T11:45:13.087 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:45:13.109+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:45:13.110170+0000) 2024-03-04T11:45:14.088 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:45:14.109+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:45:14.110479+0000) 2024-03-04T11:45:15.088 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:45:15.110+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:45:15.110801+0000) 2024-03-04T11:45:16.088 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:45:16.110+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:45:16.111080+0000) 2024-03-04T11:45:17.088 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:45:17.111+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:45:17.111320+0000) 2024-03-04T11:45:18.089 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:45:18.111+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:45:18.111552+0000) 2024-03-04T11:45:19.089 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:45:19.111+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:45:19.111850+0000) 2024-03-04T11:45:20.089 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:45:20.111+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:45:20.112169+0000) 2024-03-04T11:45:21.089 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:45:21.112+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:45:21.112401+0000) 2024-03-04T11:45:22.090 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:45:22.110+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:45:22.112585+0000) 2024-03-04T11:45:23.090 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:45:23.111+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:45:23.112818+0000) 2024-03-04T11:45:24.090 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:45:24.112+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:45:24.113119+0000) 2024-03-04T11:45:25.090 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:45:25.112+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:45:25.113354+0000) 2024-03-04T11:45:26.091 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:45:26.112+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:45:26.113595+0000) 2024-03-04T11:45:27.091 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:45:27.112+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:45:27.113864+0000) 2024-03-04T11:45:28.091 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:45:28.113+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:45:28.114108+0000) 2024-03-04T11:45:29.091 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:45:29.113+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:45:29.114342+0000) 2024-03-04T11:45:29.421 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:45:29.433 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:45:30.092 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:45:30.112+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:45:30.114535+0000) 2024-03-04T11:45:31.092 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:45:31.113+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:45:31.114786+0000) 2024-03-04T11:45:32.092 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:45:32.114+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:45:32.114990+0000) 2024-03-04T11:45:33.092 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:45:33.114+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:45:33.115167+0000) 2024-03-04T11:45:34.092 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:45:34.114+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:45:34.115428+0000) 2024-03-04T11:45:35.093 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:45:35.114+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:45:35.115679+0000) 2024-03-04T11:45:36.093 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:45:36.115+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:45:36.115926+0000) 2024-03-04T11:45:37.093 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:45:37.115+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:45:37.116171+0000) 2024-03-04T11:45:38.093 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:45:38.115+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:45:38.116345+0000) 2024-03-04T11:45:39.094 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:45:39.115+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:45:39.116578+0000) 2024-03-04T11:45:40.094 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:45:40.116+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:45:40.116857+0000) 2024-03-04T11:45:41.094 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:45:41.116+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:45:41.117094+0000) 2024-03-04T11:45:42.094 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:45:42.116+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:45:42.117329+0000) 2024-03-04T11:45:43.094 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:45:43.116+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:45:43.117567+0000) 2024-03-04T11:45:44.095 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:45:44.117+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:45:44.117810+0000) 2024-03-04T11:45:45.095 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:45:45.117+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:45:45.118090+0000) 2024-03-04T11:45:46.095 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:45:46.117+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:45:46.118351+0000) 2024-03-04T11:45:47.096 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:45:47.117+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:45:47.118597+0000) 2024-03-04T11:45:48.096 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:45:48.118+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:45:48.118871+0000) 2024-03-04T11:45:49.096 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:45:49.118+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:45:49.119058+0000) 2024-03-04T11:45:50.096 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:45:50.118+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:45:50.119301+0000) 2024-03-04T11:45:51.097 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:45:51.118+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:45:51.119557+0000) 2024-03-04T11:45:52.097 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:45:52.118+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:45:52.119788+0000) 2024-03-04T11:45:53.097 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:45:53.119+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:45:53.120031+0000) 2024-03-04T11:45:54.097 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:45:54.119+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:45:54.120274+0000) 2024-03-04T11:45:55.100 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:45:55.119+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:45:55.120509+0000) 2024-03-04T11:45:56.098 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:45:56.120+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:45:56.120748+0000) 2024-03-04T11:45:57.098 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:45:57.120+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:45:57.121000+0000) 2024-03-04T11:45:58.098 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:45:58.120+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:45:58.121242+0000) 2024-03-04T11:45:59.098 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:45:59.120+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:45:59.121478+0000) 2024-03-04T11:45:59.617 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:45:59.625 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:46:00.099 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:46:00.120+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:46:00.121739+0000) 2024-03-04T11:46:01.099 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:46:01.121+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:46:01.121969+0000) 2024-03-04T11:46:02.099 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:46:02.121+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:46:02.122204+0000) 2024-03-04T11:46:03.099 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:46:03.121+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:46:03.122439+0000) 2024-03-04T11:46:04.100 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:46:04.122+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:46:04.122683+0000) 2024-03-04T11:46:05.100 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:46:05.122+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:46:05.122936+0000) 2024-03-04T11:46:06.100 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:46:06.122+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:46:06.123172+0000) 2024-03-04T11:46:07.100 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:46:07.123+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:46:07.123414+0000) 2024-03-04T11:46:08.101 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:46:08.123+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:46:08.123651+0000) 2024-03-04T11:46:09.101 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:46:09.123+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:46:09.123962+0000) 2024-03-04T11:46:10.101 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:46:10.123+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:46:10.124263+0000) 2024-03-04T11:46:11.101 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:46:11.123+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:46:11.124544+0000) 2024-03-04T11:46:12.102 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:46:12.124+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:46:12.124859+0000) 2024-03-04T11:46:13.102 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:46:13.124+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:46:13.125143+0000) 2024-03-04T11:46:14.102 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:46:14.124+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:46:14.125376+0000) 2024-03-04T11:46:15.103 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:46:15.125+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:46:15.125658+0000) 2024-03-04T11:46:16.103 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:46:16.125+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:46:16.125977+0000) 2024-03-04T11:46:17.104 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:46:17.125+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:46:17.126239+0000) 2024-03-04T11:46:18.103 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:46:18.125+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:46:18.126524+0000) 2024-03-04T11:46:19.104 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:46:19.126+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:46:19.126762+0000) 2024-03-04T11:46:20.104 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:46:20.126+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:46:20.126997+0000) 2024-03-04T11:46:21.105 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:46:21.127+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:46:21.127231+0000) 2024-03-04T11:46:22.104 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:46:22.127+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:46:22.127463+0000) 2024-03-04T11:46:23.104 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:46:23.127+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:46:23.127694+0000) 2024-03-04T11:46:24.105 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:46:24.127+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:46:24.127919+0000) 2024-03-04T11:46:25.105 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:46:25.126+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:46:25.128272+0000) 2024-03-04T11:46:26.105 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:46:26.126+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:46:26.128560+0000) 2024-03-04T11:46:27.106 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:46:27.127+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:46:27.128810+0000) 2024-03-04T11:46:28.106 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:46:28.128+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:46:28.129086+0000) 2024-03-04T11:46:29.106 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:46:29.128+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:46:29.129327+0000) 2024-03-04T11:46:29.798 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:46:29.806 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:46:30.106 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:46:30.128+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:46:30.129569+0000) 2024-03-04T11:46:31.107 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:46:31.128+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:46:31.129806+0000) 2024-03-04T11:46:32.107 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:46:32.129+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:46:32.130074+0000) 2024-03-04T11:46:33.107 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:46:33.128+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:46:33.130285+0000) 2024-03-04T11:46:34.107 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:46:34.128+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:46:34.130565+0000) 2024-03-04T11:46:35.108 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:46:35.129+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:46:35.130874+0000) 2024-03-04T11:46:36.108 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:46:36.130+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:46:36.131129+0000) 2024-03-04T11:46:37.108 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:46:37.130+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:46:37.131397+0000) 2024-03-04T11:46:38.109 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:46:38.130+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:46:38.131656+0000) 2024-03-04T11:46:39.350 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:46:39.131+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:46:39.131960+0000) 2024-03-04T11:46:40.109 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:46:40.131+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:46:40.132216+0000) 2024-03-04T11:46:41.109 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:46:41.131+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:46:41.132424+0000) 2024-03-04T11:46:42.109 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:46:42.131+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:46:42.132618+0000) 2024-03-04T11:46:43.110 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:46:43.132+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:46:43.132917+0000) 2024-03-04T11:46:44.110 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:46:44.132+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:46:44.133181+0000) 2024-03-04T11:46:45.110 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:46:45.132+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:46:45.133430+0000) 2024-03-04T11:46:46.110 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:46:46.132+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:46:46.133686+0000) 2024-03-04T11:46:47.111 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:46:47.133+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:46:47.133966+0000) 2024-03-04T11:46:48.111 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:46:48.133+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:46:48.134168+0000) 2024-03-04T11:46:49.111 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:46:49.133+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:46:49.134378+0000) 2024-03-04T11:46:50.111 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:46:50.133+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:46:50.134612+0000) 2024-03-04T11:46:51.111 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:46:51.134+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:46:51.134910+0000) 2024-03-04T11:46:52.112 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:46:52.134+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:46:52.135162+0000) 2024-03-04T11:46:53.112 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:46:53.134+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:46:53.135414+0000) 2024-03-04T11:46:54.112 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:46:54.135+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:46:54.135667+0000) 2024-03-04T11:46:55.112 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:46:55.135+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:46:55.135924+0000) 2024-03-04T11:46:56.113 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:46:56.135+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:46:56.136202+0000) 2024-03-04T11:46:57.113 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:46:57.135+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:46:57.136421+0000) 2024-03-04T11:46:58.113 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:46:58.135+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:46:58.136691+0000) 2024-03-04T11:46:59.114 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:46:59.136+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:46:59.136942+0000) 2024-03-04T11:47:00.002 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:47:00.010 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:47:00.114 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:47:00.136+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:47:00.137128+0000) 2024-03-04T11:47:01.114 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:47:01.136+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:47:01.137350+0000) 2024-03-04T11:47:02.114 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:47:02.137+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:47:02.137644+0000) 2024-03-04T11:47:03.115 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:47:03.137+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:47:03.137990+0000) 2024-03-04T11:47:04.115 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:47:04.137+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:47:04.138289+0000) 2024-03-04T11:47:05.115 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:47:05.138+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:47:05.138582+0000) 2024-03-04T11:47:06.115 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:47:06.138+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:47:06.138918+0000) 2024-03-04T11:47:07.116 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:47:07.138+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:47:07.139221+0000) 2024-03-04T11:47:08.116 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:47:08.139+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:47:08.139503+0000) 2024-03-04T11:47:09.116 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:47:09.139+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:47:09.139800+0000) 2024-03-04T11:47:10.117 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:47:10.139+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:47:10.140149+0000) 2024-03-04T11:47:11.117 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:47:11.140+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:47:11.140428+0000) 2024-03-04T11:47:12.117 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:47:12.140+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:47:12.140785+0000) 2024-03-04T11:47:13.117 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:47:13.140+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:47:13.141048+0000) 2024-03-04T11:47:14.118 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:47:14.140+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:47:14.141348+0000) 2024-03-04T11:47:15.118 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:47:15.141+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:47:15.141637+0000) 2024-03-04T11:47:16.119 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:47:16.141+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:47:16.141963+0000) 2024-03-04T11:47:17.119 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:47:17.141+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:47:17.142265+0000) 2024-03-04T11:47:18.119 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:47:18.142+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:47:18.142552+0000) 2024-03-04T11:47:19.119 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:47:19.142+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:47:19.142784+0000) 2024-03-04T11:47:20.120 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:47:20.142+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:47:20.143121+0000) 2024-03-04T11:47:21.120 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:47:21.143+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:47:21.143410+0000) 2024-03-04T11:47:22.120 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:47:22.143+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:47:22.143707+0000) 2024-03-04T11:47:23.120 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:47:23.143+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:47:23.143956+0000) 2024-03-04T11:47:24.121 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:47:24.143+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:47:24.144196+0000) 2024-03-04T11:47:25.121 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:47:25.144+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:47:25.144435+0000) 2024-03-04T11:47:26.121 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:47:26.143+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:47:26.144682+0000) 2024-03-04T11:47:27.121 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:47:27.143+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:47:27.144932+0000) 2024-03-04T11:47:28.122 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:47:28.144+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:47:28.145171+0000) 2024-03-04T11:47:29.122 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:47:29.144+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:47:29.145413+0000) 2024-03-04T11:47:30.122 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:47:30.144+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:47:30.145672+0000) 2024-03-04T11:47:30.190 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:47:30.198 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:47:31.122 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:47:31.144+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:47:31.145929+0000) 2024-03-04T11:47:32.123 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:47:32.145+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:47:32.146171+0000) 2024-03-04T11:47:33.123 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:47:33.145+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:47:33.146408+0000) 2024-03-04T11:47:34.123 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:47:34.145+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:47:34.146613+0000) 2024-03-04T11:47:35.123 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:47:35.145+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:47:35.146921+0000) 2024-03-04T11:47:36.124 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:47:36.146+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:47:36.147162+0000) 2024-03-04T11:47:37.124 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:47:37.146+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:47:37.147409+0000) 2024-03-04T11:47:38.124 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:47:38.146+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:47:38.147632+0000) 2024-03-04T11:47:39.124 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:47:39.147+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:47:39.147926+0000) 2024-03-04T11:47:40.125 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:47:40.147+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:47:40.148121+0000) 2024-03-04T11:47:41.125 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:47:41.147+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:47:41.148357+0000) 2024-03-04T11:47:42.125 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:47:42.147+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:47:42.148597+0000) 2024-03-04T11:47:43.125 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:47:43.148+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:47:43.148870+0000) 2024-03-04T11:47:44.126 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:47:44.148+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:47:44.149133+0000) 2024-03-04T11:47:45.126 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:47:45.148+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:47:45.149327+0000) 2024-03-04T11:47:46.126 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:47:46.148+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:47:46.149585+0000) 2024-03-04T11:47:47.126 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:47:47.149+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:47:47.149865+0000) 2024-03-04T11:47:48.126 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:47:48.149+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:47:48.150101+0000) 2024-03-04T11:47:49.127 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:47:49.149+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:47:49.150359+0000) 2024-03-04T11:47:50.127 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:47:50.149+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:47:50.150600+0000) 2024-03-04T11:47:51.127 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:47:51.150+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:47:51.150865+0000) 2024-03-04T11:47:52.127 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:47:52.150+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:47:52.151109+0000) 2024-03-04T11:47:53.128 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:47:53.150+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:47:53.151347+0000) 2024-03-04T11:47:54.128 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:47:54.150+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:47:54.151607+0000) 2024-03-04T11:47:55.128 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:47:55.151+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:47:55.151903+0000) 2024-03-04T11:47:56.128 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:47:56.151+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:47:56.152154+0000) 2024-03-04T11:47:57.129 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:47:57.151+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:47:57.152407+0000) 2024-03-04T11:47:58.129 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:47:58.151+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:47:58.152605+0000) 2024-03-04T11:47:59.129 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:47:59.152+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:47:59.152883+0000) 2024-03-04T11:48:00.130 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:48:00.152+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:48:00.153145+0000) 2024-03-04T11:48:00.337 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:48:00.345 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:48:01.130 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:48:01.152+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:48:01.153399+0000) 2024-03-04T11:48:02.130 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:48:02.153+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:48:02.153640+0000) 2024-03-04T11:48:03.130 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:48:03.153+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:48:03.153898+0000) 2024-03-04T11:48:04.130 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:48:04.153+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:48:04.154139+0000) 2024-03-04T11:48:05.131 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:48:05.153+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:48:05.154382+0000) 2024-03-04T11:48:06.131 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:48:06.153+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:48:06.154634+0000) 2024-03-04T11:48:07.131 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:48:07.154+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:48:07.154925+0000) 2024-03-04T11:48:08.131 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:48:08.154+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:48:08.155105+0000) 2024-03-04T11:48:09.132 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:48:09.154+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:48:09.155298+0000) 2024-03-04T11:48:10.132 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:48:10.155+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:48:10.155598+0000) 2024-03-04T11:48:11.132 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:48:11.155+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:48:11.155870+0000) 2024-03-04T11:48:12.132 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:48:12.155+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:48:12.156143+0000) 2024-03-04T11:48:13.133 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:48:13.155+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:48:13.156398+0000) 2024-03-04T11:48:14.133 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:48:14.156+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:48:14.156637+0000) 2024-03-04T11:48:15.133 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:48:15.156+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:48:15.156928+0000) 2024-03-04T11:48:16.133 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:48:16.156+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:48:16.157170+0000) 2024-03-04T11:48:17.134 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:48:17.157+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:48:17.157416+0000) 2024-03-04T11:48:18.134 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:48:18.156+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:48:18.157645+0000) 2024-03-04T11:48:19.134 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:48:19.157+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:48:19.157933+0000) 2024-03-04T11:48:20.134 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:48:20.157+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:48:20.158178+0000) 2024-03-04T11:48:21.135 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:48:21.158+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:48:21.158423+0000) 2024-03-04T11:48:22.135 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:48:22.158+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:48:22.158665+0000) 2024-03-04T11:48:23.135 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:48:23.158+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:48:23.158947+0000) 2024-03-04T11:48:24.135 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:48:24.158+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:48:24.159206+0000) 2024-03-04T11:48:25.136 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:48:25.158+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:48:25.159469+0000) 2024-03-04T11:48:26.136 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:48:26.158+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:48:26.159711+0000) 2024-03-04T11:48:27.136 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:48:27.159+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:48:27.159953+0000) 2024-03-04T11:48:28.136 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:48:28.159+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:48:28.160172+0000) 2024-03-04T11:48:29.137 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:48:29.160+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:48:29.160557+0000) 2024-03-04T11:48:30.137 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:48:30.159+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:48:30.160817+0000) 2024-03-04T11:48:30.534 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:48:30.542 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:48:31.137 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:48:31.160+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:48:31.161112+0000) 2024-03-04T11:48:32.137 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:48:32.160+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:48:32.161353+0000) 2024-03-04T11:48:33.138 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:48:33.159+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:48:33.161584+0000) 2024-03-04T11:48:34.138 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:48:34.160+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:48:34.161917+0000) 2024-03-04T11:48:35.138 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:48:35.161+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:48:35.162173+0000) 2024-03-04T11:48:36.138 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:48:36.161+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:48:36.162411+0000) 2024-03-04T11:48:37.139 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:48:37.161+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:48:37.162656+0000) 2024-03-04T11:48:38.139 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:48:38.161+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:48:38.162918+0000) 2024-03-04T11:48:39.139 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:48:39.162+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:48:39.163170+0000) 2024-03-04T11:48:40.139 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:48:40.162+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:48:40.163450+0000) 2024-03-04T11:48:41.140 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:48:41.162+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:48:41.163707+0000) 2024-03-04T11:48:42.140 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:48:42.163+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:48:42.163973+0000) 2024-03-04T11:48:43.140 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:48:43.163+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:48:43.164196+0000) 2024-03-04T11:48:44.141 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:48:44.163+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:48:44.164461+0000) 2024-03-04T11:48:45.141 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:48:45.163+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:48:45.164705+0000) 2024-03-04T11:48:46.141 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:48:46.164+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:48:46.164944+0000) 2024-03-04T11:48:47.141 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:48:47.164+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:48:47.165184+0000) 2024-03-04T11:48:48.141 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:48:48.164+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:48:48.165371+0000) 2024-03-04T11:48:49.142 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:48:49.164+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:48:49.165597+0000) 2024-03-04T11:48:50.142 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:48:50.165+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:48:50.165913+0000) 2024-03-04T11:48:51.142 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:48:51.165+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:48:51.166145+0000) 2024-03-04T11:48:52.142 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:48:52.165+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:48:52.166387+0000) 2024-03-04T11:48:53.143 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:48:53.165+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:48:53.166642+0000) 2024-03-04T11:48:54.143 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:48:54.166+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:48:54.166929+0000) 2024-03-04T11:48:55.143 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:48:55.166+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:48:55.167169+0000) 2024-03-04T11:48:56.143 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:48:56.166+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:48:56.167407+0000) 2024-03-04T11:48:57.144 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:48:57.166+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:48:57.167656+0000) 2024-03-04T11:48:58.144 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:48:58.167+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:48:58.167993+0000) 2024-03-04T11:48:59.144 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:48:59.167+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:48:59.168247+0000) 2024-03-04T11:49:00.145 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:49:00.167+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:49:00.168515+0000) 2024-03-04T11:49:00.706 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:49:00.715 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:49:01.145 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:49:01.168+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:49:01.168755+0000) 2024-03-04T11:49:02.145 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:49:02.168+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:49:02.168994+0000) 2024-03-04T11:49:03.145 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:49:03.168+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:49:03.169229+0000) 2024-03-04T11:49:04.145 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:49:04.168+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:49:04.169469+0000) 2024-03-04T11:49:05.146 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:49:05.169+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:49:05.169718+0000) 2024-03-04T11:49:06.146 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:49:06.169+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:49:06.169960+0000) 2024-03-04T11:49:07.146 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:49:07.169+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:49:07.170231+0000) 2024-03-04T11:49:08.146 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:49:08.169+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:49:08.170419+0000) 2024-03-04T11:49:09.147 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:49:09.170+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:49:09.170654+0000) 2024-03-04T11:49:10.147 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:49:10.170+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:49:10.170926+0000) 2024-03-04T11:49:11.147 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:49:11.170+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:49:11.171175+0000) 2024-03-04T11:49:12.147 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:49:12.170+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:49:12.171422+0000) 2024-03-04T11:49:13.148 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:49:13.171+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:49:13.171689+0000) 2024-03-04T11:49:14.148 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:49:14.171+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:49:14.171937+0000) 2024-03-04T11:49:15.148 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:49:15.171+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:49:15.172175+0000) 2024-03-04T11:49:16.148 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:49:16.172+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:49:16.172417+0000) 2024-03-04T11:49:17.149 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:49:17.172+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:49:17.172659+0000) 2024-03-04T11:49:18.149 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:49:18.172+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:49:18.172937+0000) 2024-03-04T11:49:19.149 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:49:19.172+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:49:19.173188+0000) 2024-03-04T11:49:20.149 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:49:20.173+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:49:20.173439+0000) 2024-03-04T11:49:21.150 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:49:21.173+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:49:21.173622+0000) 2024-03-04T11:49:22.150 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:49:22.172+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:49:22.173797+0000) 2024-03-04T11:49:23.150 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:49:23.173+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:49:23.174006+0000) 2024-03-04T11:49:24.150 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:49:24.173+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:49:24.174244+0000) 2024-03-04T11:49:25.150 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:49:25.173+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:49:25.174446+0000) 2024-03-04T11:49:26.151 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:49:26.174+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:49:26.174691+0000) 2024-03-04T11:49:27.151 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:49:27.174+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:49:27.174935+0000) 2024-03-04T11:49:28.152 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:49:28.174+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:49:28.175159+0000) 2024-03-04T11:49:29.151 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:49:29.175+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:49:29.175396+0000) 2024-03-04T11:49:30.151 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:49:30.175+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:49:30.175639+0000) 2024-03-04T11:49:30.853 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:49:30.861 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:49:31.152 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:49:31.175+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:49:31.175943+0000) 2024-03-04T11:49:32.152 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:49:32.175+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:49:32.176204+0000) 2024-03-04T11:49:33.152 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:49:33.175+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:49:33.176429+0000) 2024-03-04T11:49:34.152 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:49:34.175+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:49:34.176664+0000) 2024-03-04T11:49:35.153 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:49:35.175+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:49:35.176935+0000) 2024-03-04T11:49:36.153 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:49:36.176+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:49:36.177211+0000) 2024-03-04T11:49:37.153 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:49:37.175+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:49:37.177472+0000) 2024-03-04T11:49:38.154 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:49:38.176+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:49:38.177703+0000) 2024-03-04T11:49:39.154 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:49:39.176+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:49:39.177950+0000) 2024-03-04T11:49:40.154 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:49:40.177+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:49:40.178189+0000) 2024-03-04T11:49:41.154 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:49:41.177+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:49:41.178426+0000) 2024-03-04T11:49:42.155 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:49:42.177+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:49:42.178667+0000) 2024-03-04T11:49:43.155 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:49:43.178+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:49:43.178967+0000) 2024-03-04T11:49:44.155 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:49:44.178+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:49:44.179220+0000) 2024-03-04T11:49:45.155 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:49:45.178+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:49:45.179474+0000) 2024-03-04T11:49:46.155 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:49:46.178+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:49:46.179667+0000) 2024-03-04T11:49:47.156 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:49:47.179+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:49:47.179928+0000) 2024-03-04T11:49:48.156 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:49:48.179+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:49:48.180133+0000) 2024-03-04T11:49:49.156 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:49:49.179+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:49:49.180395+0000) 2024-03-04T11:49:50.156 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:49:50.179+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:49:50.180645+0000) 2024-03-04T11:49:51.157 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:49:51.180+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:49:51.180879+0000) 2024-03-04T11:49:52.157 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:49:52.180+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:49:52.181191+0000) 2024-03-04T11:49:53.157 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:49:53.180+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:49:53.181432+0000) 2024-03-04T11:49:54.158 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:49:54.180+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:49:54.181688+0000) 2024-03-04T11:49:55.158 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:49:55.181+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:49:55.181926+0000) 2024-03-04T11:49:56.158 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:49:56.181+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:49:56.182170+0000) 2024-03-04T11:49:57.158 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:49:57.181+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:49:57.182413+0000) 2024-03-04T11:49:58.159 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:49:58.181+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:49:58.182666+0000) 2024-03-04T11:49:59.159 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:49:59.182+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:49:59.182931+0000) 2024-03-04T11:50:00.159 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:50:00.182+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:50:00.183171+0000) 2024-03-04T11:50:01.065 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:50:01.073 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:50:01.159 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:50:01.182+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:50:01.183336+0000) 2024-03-04T11:50:02.159 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:50:02.182+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:50:02.183514+0000) 2024-03-04T11:50:03.159 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:50:03.182+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:50:03.183698+0000) 2024-03-04T11:50:04.160 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:50:04.183+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:50:04.183939+0000) 2024-03-04T11:50:05.160 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:50:05.183+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:50:05.184176+0000) 2024-03-04T11:50:06.160 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:50:06.183+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:50:06.184416+0000) 2024-03-04T11:50:07.160 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:50:07.184+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:50:07.184688+0000) 2024-03-04T11:50:08.161 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:50:08.184+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:50:08.184929+0000) 2024-03-04T11:50:09.161 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:50:09.184+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:50:09.185183+0000) 2024-03-04T11:50:10.161 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:50:10.184+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:50:10.185416+0000) 2024-03-04T11:50:11.161 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:50:11.185+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:50:11.185650+0000) 2024-03-04T11:50:12.162 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:50:12.185+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:50:12.185925+0000) 2024-03-04T11:50:13.162 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:50:13.185+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:50:13.186115+0000) 2024-03-04T11:50:14.162 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:50:14.185+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:50:14.186352+0000) 2024-03-04T11:50:15.162 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:50:15.185+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:50:15.186529+0000) 2024-03-04T11:50:16.162 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:50:16.185+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:50:16.186716+0000) 2024-03-04T11:50:17.163 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:50:17.186+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:50:17.186959+0000) 2024-03-04T11:50:18.163 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:50:18.186+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:50:18.187151+0000) 2024-03-04T11:50:19.163 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:50:19.187+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:50:19.187392+0000) 2024-03-04T11:50:20.163 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:50:20.187+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:50:20.187633+0000) 2024-03-04T11:50:21.163 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:50:21.187+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:50:21.187901+0000) 2024-03-04T11:50:22.164 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:50:22.187+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:50:22.188143+0000) 2024-03-04T11:50:23.164 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:50:23.188+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:50:23.188397+0000) 2024-03-04T11:50:24.164 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:50:24.187+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:50:24.188590+0000) 2024-03-04T11:50:25.164 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:50:25.188+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:50:25.188855+0000) 2024-03-04T11:50:26.165 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:50:26.188+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:50:26.189098+0000) 2024-03-04T11:50:27.165 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:50:27.189+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:50:27.189340+0000) 2024-03-04T11:50:28.165 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:50:28.188+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:50:28.189611+0000) 2024-03-04T11:50:29.165 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:50:29.189+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:50:29.189909+0000) 2024-03-04T11:50:30.166 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:50:30.189+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:50:30.190146+0000) 2024-03-04T11:50:31.166 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:50:31.190+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:50:31.190379+0000) 2024-03-04T11:50:31.213 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:50:31.221 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:50:32.166 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:50:32.190+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:50:32.190622+0000) 2024-03-04T11:50:33.166 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:50:33.190+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:50:33.190902+0000) 2024-03-04T11:50:34.167 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:50:34.190+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:50:34.191086+0000) 2024-03-04T11:50:35.167 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:50:35.190+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:50:35.191351+0000) 2024-03-04T11:50:36.167 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:50:36.189+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:50:36.191604+0000) 2024-03-04T11:50:37.167 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:50:37.190+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:50:37.191894+0000) 2024-03-04T11:50:38.168 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:50:38.191+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:50:38.192131+0000) 2024-03-04T11:50:39.168 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:50:39.191+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:50:39.192371+0000) 2024-03-04T11:50:40.168 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:50:40.191+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:50:40.192659+0000) 2024-03-04T11:50:41.169 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:50:41.191+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:50:41.192937+0000) 2024-03-04T11:50:42.169 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:50:42.192+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:50:42.193181+0000) 2024-03-04T11:50:43.169 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:50:43.192+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:50:43.193420+0000) 2024-03-04T11:50:44.169 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:50:44.192+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:50:44.193657+0000) 2024-03-04T11:50:45.169 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:50:45.192+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:50:45.193938+0000) 2024-03-04T11:50:46.170 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:50:46.193+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:50:46.194176+0000) 2024-03-04T11:50:47.170 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:50:47.193+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:50:47.194439+0000) 2024-03-04T11:50:48.170 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:50:48.193+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:50:48.194683+0000) 2024-03-04T11:50:49.170 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:50:49.194+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:50:49.194936+0000) 2024-03-04T11:50:50.171 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:50:50.194+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:50:50.195180+0000) 2024-03-04T11:50:51.171 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:50:51.194+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:50:51.195419+0000) 2024-03-04T11:50:52.171 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:50:52.194+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:50:52.195655+0000) 2024-03-04T11:50:53.171 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:50:53.195+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:50:53.195924+0000) 2024-03-04T11:50:54.172 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:50:54.195+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:50:54.196164+0000) 2024-03-04T11:50:55.172 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:50:55.195+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:50:55.196399+0000) 2024-03-04T11:50:56.172 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:50:56.195+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:50:56.196637+0000) 2024-03-04T11:50:57.172 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:50:57.196+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:50:57.196935+0000) 2024-03-04T11:50:58.173 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:50:58.196+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:50:58.197128+0000) 2024-03-04T11:50:59.173 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:50:59.196+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:50:59.197368+0000) 2024-03-04T11:51:00.173 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:51:00.196+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:51:00.197607+0000) 2024-03-04T11:51:01.173 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:51:01.197+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:51:01.197891+0000) 2024-03-04T11:51:01.424 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:51:01.432 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:51:02.173 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:51:02.197+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:51:02.198131+0000) 2024-03-04T11:51:03.174 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:51:03.197+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:51:03.198394+0000) 2024-03-04T11:51:04.174 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:51:04.197+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:51:04.198633+0000) 2024-03-04T11:51:05.174 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:51:05.198+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:51:05.198925+0000) 2024-03-04T11:51:06.175 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:51:06.198+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:51:06.199167+0000) 2024-03-04T11:51:07.175 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:51:07.198+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:51:07.199412+0000) 2024-03-04T11:51:08.175 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:51:08.198+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:51:08.199647+0000) 2024-03-04T11:51:09.175 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:51:09.199+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:51:09.199920+0000) 2024-03-04T11:51:10.176 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:51:10.199+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:51:10.200164+0000) 2024-03-04T11:51:11.176 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:51:11.199+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:51:11.200399+0000) 2024-03-04T11:51:12.176 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:51:12.200+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:51:12.200668+0000) 2024-03-04T11:51:13.176 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:51:13.200+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:51:13.200954+0000) 2024-03-04T11:51:14.177 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:51:14.200+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:51:14.201187+0000) 2024-03-04T11:51:15.177 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:51:15.200+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:51:15.201427+0000) 2024-03-04T11:51:16.177 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:51:16.201+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:51:16.201673+0000) 2024-03-04T11:51:17.177 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:51:17.201+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:51:17.201951+0000) 2024-03-04T11:51:18.177 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:51:18.201+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:51:18.202183+0000) 2024-03-04T11:51:19.178 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:51:19.202+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:51:19.202417+0000) 2024-03-04T11:51:20.178 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:51:20.201+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:51:20.202676+0000) 2024-03-04T11:51:21.178 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:51:21.202+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:51:21.202944+0000) 2024-03-04T11:51:22.179 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:51:22.202+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:51:22.203186+0000) 2024-03-04T11:51:23.179 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:51:23.203+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:51:23.203420+0000) 2024-03-04T11:51:24.179 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:51:24.203+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:51:24.203663+0000) 2024-03-04T11:51:25.179 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:51:25.203+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:51:25.203970+0000) 2024-03-04T11:51:26.179 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:51:26.203+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:51:26.204209+0000) 2024-03-04T11:51:27.180 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:51:27.203+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:51:27.204469+0000) 2024-03-04T11:51:28.180 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:51:28.203+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:51:28.204724+0000) 2024-03-04T11:51:29.180 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:51:29.204+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:51:29.204961+0000) 2024-03-04T11:51:30.180 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:51:30.204+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:51:30.205205+0000) 2024-03-04T11:51:31.181 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:51:31.205+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:51:31.205442+0000) 2024-03-04T11:51:31.629 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:51:31.638 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:51:32.181 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:51:32.204+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:51:32.205625+0000) 2024-03-04T11:51:33.187 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:51:33.205+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:51:33.205930+0000) 2024-03-04T11:51:34.181 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:51:34.206+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:51:34.206192+0000) 2024-03-04T11:51:35.182 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:51:35.206+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:51:35.206426+0000) 2024-03-04T11:51:36.182 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:51:36.206+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:51:36.206656+0000) 2024-03-04T11:51:37.182 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:51:37.205+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:51:37.206933+0000) 2024-03-04T11:51:38.182 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:51:38.206+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:51:38.207173+0000) 2024-03-04T11:51:39.183 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:51:39.205+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:51:39.207423+0000) 2024-03-04T11:51:40.183 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:51:40.206+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:51:40.207659+0000) 2024-03-04T11:51:41.183 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:51:41.206+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:51:41.207930+0000) 2024-03-04T11:51:42.183 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:51:42.207+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:51:42.208170+0000) 2024-03-04T11:51:43.184 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:51:43.207+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:51:43.208407+0000) 2024-03-04T11:51:44.184 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:51:44.207+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:51:44.208649+0000) 2024-03-04T11:51:45.184 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:51:45.207+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:51:45.208910+0000) 2024-03-04T11:51:46.184 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:51:46.208+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:51:46.209164+0000) 2024-03-04T11:51:47.185 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:51:47.208+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:51:47.209446+0000) 2024-03-04T11:51:48.185 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:51:48.208+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:51:48.209698+0000) 2024-03-04T11:51:49.185 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:51:49.209+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:51:49.209931+0000) 2024-03-04T11:51:50.185 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:51:50.209+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:51:50.210175+0000) 2024-03-04T11:51:51.186 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:51:51.209+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:51:51.210426+0000) 2024-03-04T11:51:52.186 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:51:52.209+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:51:52.210684+0000) 2024-03-04T11:51:53.186 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:51:53.210+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:51:53.210927+0000) 2024-03-04T11:51:54.186 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:51:54.210+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:51:54.211163+0000) 2024-03-04T11:51:55.187 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:51:55.210+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:51:55.211342+0000) 2024-03-04T11:51:56.187 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:51:56.210+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:51:56.211584+0000) 2024-03-04T11:51:57.187 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:51:57.211+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:51:57.211865+0000) 2024-03-04T11:51:58.187 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:51:58.211+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:51:58.212109+0000) 2024-03-04T11:51:59.187 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:51:59.211+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:51:59.212339+0000) 2024-03-04T11:52:00.188 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:52:00.211+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:52:00.212581+0000) 2024-03-04T11:52:01.188 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:52:01.212+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:52:01.212850+0000) 2024-03-04T11:52:01.833 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:52:01.842 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:52:02.188 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:52:02.212+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:52:02.213088+0000) 2024-03-04T11:52:03.188 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:52:03.212+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:52:03.213325+0000) 2024-03-04T11:52:04.192 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:52:04.212+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:52:04.213580+0000) 2024-03-04T11:52:05.189 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:52:05.213+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:52:05.213874+0000) 2024-03-04T11:52:06.189 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:52:06.213+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:52:06.214114+0000) 2024-03-04T11:52:07.190 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:52:07.213+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:52:07.214355+0000) 2024-03-04T11:52:08.190 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:52:08.214+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:52:08.214617+0000) 2024-03-04T11:52:09.190 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:52:09.214+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:52:09.214919+0000) 2024-03-04T11:52:10.190 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:52:10.214+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:52:10.215159+0000) 2024-03-04T11:52:11.191 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:52:11.214+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:52:11.215394+0000) 2024-03-04T11:52:12.191 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:52:12.214+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:52:12.215645+0000) 2024-03-04T11:52:13.191 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:52:13.215+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:52:13.215914+0000) 2024-03-04T11:52:14.191 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:52:14.215+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:52:14.216151+0000) 2024-03-04T11:52:15.191 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:52:15.215+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:52:15.216388+0000) 2024-03-04T11:52:16.192 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:52:16.216+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:52:16.216633+0000) 2024-03-04T11:52:17.192 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:52:17.216+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:52:17.216940+0000) 2024-03-04T11:52:18.192 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:52:18.216+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:52:18.217179+0000) 2024-03-04T11:52:19.193 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:52:19.216+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:52:19.217412+0000) 2024-03-04T11:52:20.193 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:52:20.217+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:52:20.217655+0000) 2024-03-04T11:52:21.193 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:52:21.217+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:52:21.217937+0000) 2024-03-04T11:52:22.193 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:52:22.217+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:52:22.218174+0000) 2024-03-04T11:52:23.193 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:52:23.217+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:52:23.218404+0000) 2024-03-04T11:52:24.194 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:52:24.218+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:52:24.218645+0000) 2024-03-04T11:52:25.194 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:52:25.218+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:52:25.218860+0000) 2024-03-04T11:52:26.194 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:52:26.218+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:52:26.219109+0000) 2024-03-04T11:52:27.194 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:52:27.219+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:52:27.219361+0000) 2024-03-04T11:52:28.195 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:52:28.219+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:52:28.219601+0000) 2024-03-04T11:52:29.195 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:52:29.219+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:52:29.219903+0000) 2024-03-04T11:52:30.195 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:52:30.219+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:52:30.220154+0000) 2024-03-04T11:52:31.196 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:52:31.220+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:52:31.220409+0000) 2024-03-04T11:52:31.981 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:52:31.989 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:52:32.196 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:52:32.220+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:52:32.220703+0000) 2024-03-04T11:52:33.196 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:52:33.220+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:52:33.220948+0000) 2024-03-04T11:52:34.196 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:52:34.221+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:52:34.221210+0000) 2024-03-04T11:52:35.197 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:52:35.221+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:52:35.221482+0000) 2024-03-04T11:52:36.197 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:52:36.221+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:52:36.221722+0000) 2024-03-04T11:52:37.197 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:52:37.221+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:52:37.221980+0000) 2024-03-04T11:52:38.197 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:52:38.221+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:52:38.222210+0000) 2024-03-04T11:52:39.197 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:52:39.221+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:52:39.222393+0000) 2024-03-04T11:52:40.198 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:52:40.221+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:52:40.222638+0000) 2024-03-04T11:52:41.199 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:52:41.221+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:52:41.222928+0000) 2024-03-04T11:52:42.198 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:52:42.222+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:52:42.223163+0000) 2024-03-04T11:52:43.198 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:52:43.222+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:52:43.223400+0000) 2024-03-04T11:52:44.199 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:52:44.222+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:52:44.223644+0000) 2024-03-04T11:52:45.199 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:52:45.222+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:52:45.223925+0000) 2024-03-04T11:52:46.199 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:52:46.223+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:52:46.224169+0000) 2024-03-04T11:52:47.199 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:52:47.223+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:52:47.224417+0000) 2024-03-04T11:52:48.200 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:52:48.223+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:52:48.224654+0000) 2024-03-04T11:52:49.200 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:52:49.223+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:52:49.224916+0000) 2024-03-04T11:52:50.200 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:52:50.224+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:52:50.225156+0000) 2024-03-04T11:52:51.200 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:52:51.224+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:52:51.225396+0000) 2024-03-04T11:52:52.201 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:52:52.224+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:52:52.225631+0000) 2024-03-04T11:52:53.201 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:52:53.225+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:52:53.225921+0000) 2024-03-04T11:52:54.201 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:52:54.225+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:52:54.226156+0000) 2024-03-04T11:52:55.201 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:52:55.225+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:52:55.226394+0000) 2024-03-04T11:52:56.202 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:52:56.225+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:52:56.226638+0000) 2024-03-04T11:52:57.202 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:52:57.226+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:52:57.226940+0000) 2024-03-04T11:52:58.202 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:52:58.226+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:52:58.227186+0000) 2024-03-04T11:52:59.202 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:52:59.226+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:52:59.227424+0000) 2024-03-04T11:53:00.203 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:53:00.226+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:53:00.227675+0000) 2024-03-04T11:53:01.203 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:53:01.227+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:53:01.227940+0000) 2024-03-04T11:53:02.129 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:53:02.141 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:53:02.203 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:53:02.227+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:53:02.228116+0000) 2024-03-04T11:53:03.203 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:53:03.227+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:53:03.228336+0000) 2024-03-04T11:53:04.203 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:53:04.227+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:53:04.228594+0000) 2024-03-04T11:53:05.204 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:53:05.228+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:53:05.228915+0000) 2024-03-04T11:53:06.204 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:53:06.228+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:53:06.229110+0000) 2024-03-04T11:53:07.204 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:53:07.228+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:53:07.229356+0000) 2024-03-04T11:53:08.205 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:53:08.228+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:53:08.229593+0000) 2024-03-04T11:53:09.205 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:53:09.229+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:53:09.229887+0000) 2024-03-04T11:53:10.205 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:53:10.229+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:53:10.230083+0000) 2024-03-04T11:53:11.205 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:53:11.229+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:53:11.230313+0000) 2024-03-04T11:53:12.205 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:53:12.229+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:53:12.230578+0000) 2024-03-04T11:53:13.206 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:53:13.230+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:53:13.230817+0000) 2024-03-04T11:53:14.206 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:53:14.230+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:53:14.231107+0000) 2024-03-04T11:53:15.206 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:53:15.230+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:53:15.231389+0000) 2024-03-04T11:53:16.206 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:53:16.231+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:53:16.231630+0000) 2024-03-04T11:53:17.207 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:53:17.231+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:53:17.231917+0000) 2024-03-04T11:53:18.207 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:53:18.231+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:53:18.232152+0000) 2024-03-04T11:53:19.207 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:53:19.231+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:53:19.232389+0000) 2024-03-04T11:53:20.207 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:53:20.231+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:53:20.232599+0000) 2024-03-04T11:53:21.208 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:53:21.232+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:53:21.232879+0000) 2024-03-04T11:53:22.208 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:53:22.232+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:53:22.233120+0000) 2024-03-04T11:53:23.208 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:53:23.232+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:53:23.233355+0000) 2024-03-04T11:53:24.209 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:53:24.233+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:53:24.233597+0000) 2024-03-04T11:53:25.209 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:53:25.233+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:53:25.233864+0000) 2024-03-04T11:53:26.209 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:53:26.233+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:53:26.234108+0000) 2024-03-04T11:53:27.209 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:53:27.233+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:53:27.234307+0000) 2024-03-04T11:53:28.209 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:53:28.234+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:53:28.234550+0000) 2024-03-04T11:53:29.210 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:53:29.234+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:53:29.234787+0000) 2024-03-04T11:53:30.210 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:53:30.234+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:53:30.235112+0000) 2024-03-04T11:53:31.210 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:53:31.235+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:53:31.235408+0000) 2024-03-04T11:53:32.210 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:53:32.235+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:53:32.235699+0000) 2024-03-04T11:53:32.331 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:53:32.339 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:53:33.211 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:53:33.235+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:53:33.235981+0000) 2024-03-04T11:53:34.211 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:53:34.236+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:53:34.236294+0000) 2024-03-04T11:53:35.211 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:53:35.235+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:53:35.236527+0000) 2024-03-04T11:53:36.212 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:53:36.236+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:53:36.236877+0000) 2024-03-04T11:53:37.212 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:53:37.236+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:53:37.237205+0000) 2024-03-04T11:53:38.212 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:53:38.237+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:53:38.237502+0000) 2024-03-04T11:53:39.213 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:53:39.237+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:53:39.237792+0000) 2024-03-04T11:53:40.213 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:53:40.237+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:53:40.238155+0000) 2024-03-04T11:53:41.213 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:53:41.238+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:53:41.238450+0000) 2024-03-04T11:53:42.214 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:53:42.237+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:53:42.238742+0000) 2024-03-04T11:53:43.214 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:53:43.237+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:53:43.239030+0000) 2024-03-04T11:53:44.214 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:53:44.237+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:53:44.239278+0000) 2024-03-04T11:53:45.214 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:53:45.238+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:53:45.239589+0000) 2024-03-04T11:53:46.215 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:53:46.238+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:53:46.239928+0000) 2024-03-04T11:53:47.215 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:53:47.239+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:53:47.240235+0000) 2024-03-04T11:53:48.215 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:53:48.239+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:53:48.240535+0000) 2024-03-04T11:53:49.216 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:53:49.239+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:53:49.240877+0000) 2024-03-04T11:53:50.216 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:53:50.240+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:53:50.241184+0000) 2024-03-04T11:53:51.216 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:53:51.240+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:53:51.241464+0000) 2024-03-04T11:53:52.216 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:53:52.240+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:53:52.241748+0000) 2024-03-04T11:53:53.217 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:53:53.241+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:53:53.242041+0000) 2024-03-04T11:53:54.217 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:53:54.241+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:53:54.242316+0000) 2024-03-04T11:53:55.217 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:53:55.241+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:53:55.242603+0000) 2024-03-04T11:53:56.218 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:53:56.242+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:53:56.242976+0000) 2024-03-04T11:53:57.218 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:53:57.242+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:53:57.243276+0000) 2024-03-04T11:53:58.218 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:53:58.242+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:53:58.243569+0000) 2024-03-04T11:53:59.218 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:53:59.243+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:53:59.243919+0000) 2024-03-04T11:54:00.219 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:54:00.243+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:54:00.244198+0000) 2024-03-04T11:54:01.219 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:54:01.243+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:54:01.244459+0000) 2024-03-04T11:54:02.219 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:54:02.243+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:54:02.244703+0000) 2024-03-04T11:54:02.544 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:54:02.552 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:54:03.220 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:54:03.244+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:54:03.244939+0000) 2024-03-04T11:54:04.220 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:54:04.244+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:54:04.245172+0000) 2024-03-04T11:54:05.220 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:54:05.244+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:54:05.245426+0000) 2024-03-04T11:54:06.220 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:54:06.244+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:54:06.245681+0000) 2024-03-04T11:54:07.220 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:54:07.245+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:54:07.245955+0000) 2024-03-04T11:54:08.221 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:54:08.245+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:54:08.246195+0000) 2024-03-04T11:54:09.221 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:54:09.245+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:54:09.246464+0000) 2024-03-04T11:54:10.221 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:54:10.246+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:54:10.246701+0000) 2024-03-04T11:54:11.222 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:54:11.246+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:54:11.246939+0000) 2024-03-04T11:54:12.222 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:54:12.246+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:54:12.247177+0000) 2024-03-04T11:54:13.222 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:54:13.246+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:54:13.247411+0000) 2024-03-04T11:54:14.222 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:54:14.246+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:54:14.247667+0000) 2024-03-04T11:54:15.223 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:54:15.247+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:54:15.247932+0000) 2024-03-04T11:54:16.223 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:54:16.247+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:54:16.248171+0000) 2024-03-04T11:54:17.223 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:54:17.247+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:54:17.248401+0000) 2024-03-04T11:54:18.223 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:54:18.248+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:54:18.248724+0000) 2024-03-04T11:54:19.224 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:54:19.248+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:54:19.249015+0000) 2024-03-04T11:54:20.224 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:54:20.248+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:54:20.249287+0000) 2024-03-04T11:54:21.224 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:54:21.249+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:54:21.249539+0000) 2024-03-04T11:54:22.224 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:54:22.249+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:54:22.249799+0000) 2024-03-04T11:54:23.225 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:54:23.249+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:54:23.250101+0000) 2024-03-04T11:54:24.225 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:54:24.249+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:54:24.250380+0000) 2024-03-04T11:54:25.225 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:54:25.250+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:54:25.250575+0000) 2024-03-04T11:54:26.226 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:54:26.250+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:54:26.250888+0000) 2024-03-04T11:54:27.226 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:54:27.250+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:54:27.251149+0000) 2024-03-04T11:54:28.226 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:54:28.251+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:54:28.251407+0000) 2024-03-04T11:54:29.226 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:54:29.250+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:54:29.251627+0000) 2024-03-04T11:54:30.226 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:54:30.251+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:54:30.251903+0000) 2024-03-04T11:54:31.227 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:54:31.251+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:54:31.252143+0000) 2024-03-04T11:54:32.227 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:54:32.252+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:54:32.252385+0000) 2024-03-04T11:54:32.699 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:54:32.707 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:54:33.227 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:54:33.251+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:54:33.252640+0000) 2024-03-04T11:54:34.227 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:54:34.252+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:54:34.252929+0000) 2024-03-04T11:54:35.228 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:54:35.252+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:54:35.253163+0000) 2024-03-04T11:54:36.228 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:54:36.253+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:54:36.253397+0000) 2024-03-04T11:54:37.228 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:54:37.253+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:54:37.253644+0000) 2024-03-04T11:54:38.228 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:54:38.253+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:54:38.253940+0000) 2024-03-04T11:54:39.229 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:54:39.253+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:54:39.254173+0000) 2024-03-04T11:54:40.229 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:54:40.254+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:54:40.254414+0000) 2024-03-04T11:54:41.229 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:54:41.254+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:54:41.254650+0000) 2024-03-04T11:54:42.229 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:54:42.254+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:54:42.254927+0000) 2024-03-04T11:54:43.230 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:54:43.255+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:54:43.255167+0000) 2024-03-04T11:54:44.230 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:54:44.254+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:54:44.255407+0000) 2024-03-04T11:54:45.230 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:54:45.254+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:54:45.255642+0000) 2024-03-04T11:54:46.230 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:54:46.254+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:54:46.255930+0000) 2024-03-04T11:54:47.231 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:54:47.255+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:54:47.256178+0000) 2024-03-04T11:54:48.231 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:54:48.255+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:54:48.256441+0000) 2024-03-04T11:54:49.231 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:54:49.255+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:54:49.256695+0000) 2024-03-04T11:54:50.231 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:54:50.255+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:54:50.256936+0000) 2024-03-04T11:54:51.232 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:54:51.256+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:54:51.257168+0000) 2024-03-04T11:54:52.232 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:54:52.256+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:54:52.257414+0000) 2024-03-04T11:54:53.232 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:54:53.255+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:54:53.257676+0000) 2024-03-04T11:54:54.232 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:54:54.256+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:54:54.257945+0000) 2024-03-04T11:54:55.233 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:54:55.257+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:54:55.258179+0000) 2024-03-04T11:54:56.233 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:54:56.257+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:54:56.258363+0000) 2024-03-04T11:54:57.233 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:54:57.257+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:54:57.258614+0000) 2024-03-04T11:54:58.233 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:54:58.258+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:54:58.258890+0000) 2024-03-04T11:54:59.233 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:54:59.258+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:54:59.259116+0000) 2024-03-04T11:55:00.234 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:55:00.258+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:55:00.259353+0000) 2024-03-04T11:55:01.234 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:55:01.258+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:55:01.259612+0000) 2024-03-04T11:55:02.234 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:55:02.259+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:55:02.259886+0000) 2024-03-04T11:55:02.912 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:55:02.920 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:55:03.234 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:55:03.259+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:55:03.260134+0000) 2024-03-04T11:55:04.235 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:55:04.259+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:55:04.260374+0000) 2024-03-04T11:55:05.235 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:55:05.259+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:55:05.260608+0000) 2024-03-04T11:55:06.235 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:55:06.260+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:55:06.260934+0000) 2024-03-04T11:55:07.235 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:55:07.260+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:55:07.261132+0000) 2024-03-04T11:55:08.236 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:55:08.260+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:55:08.261325+0000) 2024-03-04T11:55:09.236 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:55:09.260+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:55:09.261538+0000) 2024-03-04T11:55:10.236 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:55:10.260+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:55:10.261721+0000) 2024-03-04T11:55:11.236 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:55:11.261+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:55:11.261938+0000) 2024-03-04T11:55:12.236 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:55:12.261+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:55:12.262176+0000) 2024-03-04T11:55:13.237 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:55:13.261+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:55:13.262379+0000) 2024-03-04T11:55:14.239 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:55:14.261+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:55:14.262622+0000) 2024-03-04T11:55:15.237 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:55:15.262+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:55:15.262897+0000) 2024-03-04T11:55:16.237 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:55:16.262+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:55:16.263076+0000) 2024-03-04T11:55:17.237 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:55:17.262+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:55:17.263311+0000) 2024-03-04T11:55:18.238 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:55:18.262+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:55:18.263554+0000) 2024-03-04T11:55:19.238 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:55:19.263+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:55:19.263790+0000) 2024-03-04T11:55:20.238 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:55:20.263+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:55:20.264011+0000) 2024-03-04T11:55:21.239 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:55:21.263+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:55:21.264244+0000) 2024-03-04T11:55:22.239 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:55:22.263+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:55:22.264488+0000) 2024-03-04T11:55:23.239 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:55:23.263+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:55:23.264726+0000) 2024-03-04T11:55:24.239 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:55:24.264+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:55:24.265008+0000) 2024-03-04T11:55:25.239 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:55:25.264+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:55:25.265241+0000) 2024-03-04T11:55:26.240 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:55:26.265+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:55:26.265478+0000) 2024-03-04T11:55:27.240 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:55:27.265+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:55:27.265723+0000) 2024-03-04T11:55:28.240 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:55:28.265+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:55:28.265977+0000) 2024-03-04T11:55:29.240 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:55:29.265+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:55:29.266211+0000) 2024-03-04T11:55:30.241 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:55:30.265+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:55:30.266465+0000) 2024-03-04T11:55:31.241 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:55:31.265+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:55:31.266724+0000) 2024-03-04T11:55:32.241 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:55:32.266+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:55:32.266958+0000) 2024-03-04T11:55:33.058 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:55:33.066 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:55:33.241 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:55:33.266+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:55:33.267164+0000) 2024-03-04T11:55:34.241 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:55:34.266+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:55:34.267412+0000) 2024-03-04T11:55:35.242 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:55:35.267+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:55:35.267650+0000) 2024-03-04T11:55:36.242 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:55:36.267+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:55:36.267936+0000) 2024-03-04T11:55:37.243 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:55:37.267+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:55:37.268189+0000) 2024-03-04T11:55:38.243 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:55:38.267+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:55:38.268429+0000) 2024-03-04T11:55:39.243 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:55:39.268+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:55:39.268667+0000) 2024-03-04T11:55:40.243 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:55:40.268+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:55:40.268945+0000) 2024-03-04T11:55:41.243 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:55:41.268+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:55:41.269180+0000) 2024-03-04T11:55:42.244 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:55:42.268+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:55:42.269370+0000) 2024-03-04T11:55:43.244 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:55:43.269+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:55:43.269603+0000) 2024-03-04T11:55:44.244 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:55:44.269+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:55:44.269889+0000) 2024-03-04T11:55:45.244 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:55:45.269+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:55:45.270125+0000) 2024-03-04T11:55:46.244 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:55:46.269+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:55:46.270376+0000) 2024-03-04T11:55:47.245 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:55:47.269+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:55:47.270626+0000) 2024-03-04T11:55:48.245 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:55:48.269+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:55:48.270923+0000) 2024-03-04T11:55:49.245 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:55:49.270+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:55:49.271156+0000) 2024-03-04T11:55:50.245 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:55:50.270+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:55:50.271392+0000) 2024-03-04T11:55:51.246 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:55:51.270+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:55:51.271635+0000) 2024-03-04T11:55:52.246 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:55:52.270+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:55:52.271818+0000) 2024-03-04T11:55:53.246 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:55:53.270+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:55:53.272093+0000) 2024-03-04T11:55:54.246 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:55:54.271+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:55:54.272353+0000) 2024-03-04T11:55:55.247 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:55:55.271+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:55:55.272541+0000) 2024-03-04T11:55:56.247 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:55:56.271+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:55:56.272809+0000) 2024-03-04T11:55:57.247 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:55:57.272+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:55:57.273115+0000) 2024-03-04T11:55:58.248 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:55:58.272+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:55:58.273366+0000) 2024-03-04T11:55:59.248 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:55:59.272+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:55:59.273597+0000) 2024-03-04T11:56:00.248 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:56:00.273+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:56:00.273868+0000) 2024-03-04T11:56:01.248 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:56:01.273+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:56:01.274114+0000) 2024-03-04T11:56:02.248 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:56:02.273+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:56:02.274394+0000) 2024-03-04T11:56:03.249 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:56:03.273+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:56:03.274635+0000) 2024-03-04T11:56:03.269 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:56:03.281 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:56:04.249 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:56:04.274+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:56:04.274931+0000) 2024-03-04T11:56:05.249 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:56:05.274+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:56:05.275186+0000) 2024-03-04T11:56:06.249 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:56:06.274+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:56:06.275449+0000) 2024-03-04T11:56:07.250 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:56:07.274+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:56:07.275685+0000) 2024-03-04T11:56:08.250 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:56:08.275+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:56:08.275937+0000) 2024-03-04T11:56:09.250 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:56:09.275+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:56:09.276170+0000) 2024-03-04T11:56:10.251 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:56:10.275+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:56:10.276418+0000) 2024-03-04T11:56:11.251 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:56:11.275+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:56:11.276657+0000) 2024-03-04T11:56:12.251 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:56:12.276+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:56:12.276954+0000) 2024-03-04T11:56:13.252 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:56:13.276+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:56:13.277213+0000) 2024-03-04T11:56:14.252 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:56:14.276+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:56:14.277452+0000) 2024-03-04T11:56:15.252 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:56:15.277+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:56:15.277692+0000) 2024-03-04T11:56:16.252 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:56:16.277+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:56:16.277934+0000) 2024-03-04T11:56:17.253 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:56:17.277+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:56:17.278181+0000) 2024-03-04T11:56:18.253 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:56:18.277+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:56:18.278424+0000) 2024-03-04T11:56:19.253 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:56:19.278+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:56:19.278657+0000) 2024-03-04T11:56:20.254 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:56:20.278+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:56:20.278955+0000) 2024-03-04T11:56:21.254 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:56:21.278+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:56:21.279194+0000) 2024-03-04T11:56:22.254 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:56:22.278+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:56:22.279440+0000) 2024-03-04T11:56:23.255 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:56:23.279+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:56:23.279678+0000) 2024-03-04T11:56:24.255 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:56:24.279+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:56:24.279958+0000) 2024-03-04T11:56:25.255 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:56:25.279+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:56:25.280153+0000) 2024-03-04T11:56:26.256 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:56:26.279+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:56:26.280399+0000) 2024-03-04T11:56:27.256 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:56:27.280+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:56:27.280676+0000) 2024-03-04T11:56:28.256 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:56:28.280+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:56:28.280949+0000) 2024-03-04T11:56:29.257 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:56:29.280+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:56:29.281179+0000) 2024-03-04T11:56:30.257 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:56:30.281+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:56:30.281419+0000) 2024-03-04T11:56:31.257 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:56:31.281+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:56:31.281682+0000) 2024-03-04T11:56:32.258 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:56:32.281+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:56:32.281949+0000) 2024-03-04T11:56:33.258 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:56:33.281+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:56:33.282186+0000) 2024-03-04T11:56:33.423 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:56:33.431 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:56:34.258 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:56:34.282+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:56:34.282433+0000) 2024-03-04T11:56:35.258 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:56:35.281+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:56:35.282714+0000) 2024-03-04T11:56:36.259 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:56:36.282+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:56:36.282990+0000) 2024-03-04T11:56:37.259 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:56:37.282+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:56:37.283252+0000) 2024-03-04T11:56:38.260 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:56:38.283+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:56:38.283508+0000) 2024-03-04T11:56:39.260 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:56:39.282+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:56:39.283739+0000) 2024-03-04T11:56:40.260 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:56:40.283+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:56:40.283991+0000) 2024-03-04T11:56:41.260 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:56:41.283+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:56:41.284238+0000) 2024-03-04T11:56:42.261 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:56:42.284+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:56:42.284486+0000) 2024-03-04T11:56:43.261 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:56:43.284+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:56:43.284721+0000) 2024-03-04T11:56:44.261 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:56:44.284+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:56:44.284960+0000) 2024-03-04T11:56:45.261 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:56:45.284+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:56:45.285153+0000) 2024-03-04T11:56:46.262 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:56:46.285+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:56:46.285401+0000) 2024-03-04T11:56:47.262 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:56:47.285+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:56:47.285655+0000) 2024-03-04T11:56:48.262 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:56:48.285+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:56:48.285932+0000) 2024-03-04T11:56:49.263 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:56:49.284+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:56:49.286121+0000) 2024-03-04T11:56:50.263 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:56:50.284+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:56:50.286315+0000) 2024-03-04T11:56:51.263 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:56:51.285+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:56:51.286559+0000) 2024-03-04T11:56:52.263 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:56:52.285+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:56:52.286806+0000) 2024-03-04T11:56:53.264 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:56:53.285+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:56:53.287090+0000) 2024-03-04T11:56:54.264 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:56:54.285+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:56:54.287285+0000) 2024-03-04T11:56:55.264 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:56:55.285+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:56:55.287468+0000) 2024-03-04T11:56:56.264 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:56:56.286+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:56:56.287714+0000) 2024-03-04T11:56:57.265 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:56:57.287+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:56:57.288010+0000) 2024-03-04T11:56:58.265 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:56:58.286+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:56:58.288265+0000) 2024-03-04T11:56:59.265 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:56:59.287+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:56:59.288511+0000) 2024-03-04T11:57:00.266 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:57:00.287+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:57:00.288767+0000) 2024-03-04T11:57:01.266 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:57:01.288+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:57:01.289027+0000) 2024-03-04T11:57:02.266 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:57:02.288+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:57:02.289235+0000) 2024-03-04T11:57:03.266 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:57:03.288+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:57:03.289512+0000) 2024-03-04T11:57:03.658 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:57:03.666 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:57:04.267 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:57:04.288+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:57:04.289774+0000) 2024-03-04T11:57:05.267 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:57:05.289+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:57:05.289986+0000) 2024-03-04T11:57:06.267 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:57:06.289+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:57:06.290255+0000) 2024-03-04T11:57:07.268 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:57:07.289+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:57:07.290565+0000) 2024-03-04T11:57:08.268 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:57:08.290+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:57:08.290867+0000) 2024-03-04T11:57:09.268 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:57:09.290+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:57:09.291163+0000) 2024-03-04T11:57:10.269 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:57:10.290+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:57:10.291423+0000) 2024-03-04T11:57:11.269 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:57:11.290+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:57:11.291687+0000) 2024-03-04T11:57:12.269 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:57:12.291+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:57:12.291955+0000) 2024-03-04T11:57:13.270 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:57:13.291+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:57:13.292214+0000) 2024-03-04T11:57:14.270 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:57:14.291+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:57:14.292428+0000) 2024-03-04T11:57:15.270 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:57:15.291+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:57:15.292678+0000) 2024-03-04T11:57:16.271 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:57:16.292+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:57:16.292945+0000) 2024-03-04T11:57:17.271 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:57:17.292+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:57:17.293198+0000) 2024-03-04T11:57:18.271 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:57:18.292+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:57:18.293440+0000) 2024-03-04T11:57:19.271 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:57:19.293+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:57:19.293674+0000) 2024-03-04T11:57:20.272 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:57:20.293+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:57:20.293946+0000) 2024-03-04T11:57:21.272 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:57:21.293+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:57:21.294191+0000) 2024-03-04T11:57:22.272 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:57:22.293+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:57:22.294438+0000) 2024-03-04T11:57:23.273 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:57:23.294+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:57:23.294695+0000) 2024-03-04T11:57:24.273 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:57:24.294+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:57:24.294933+0000) 2024-03-04T11:57:25.273 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:57:25.294+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:57:25.295171+0000) 2024-03-04T11:57:26.273 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:57:26.294+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:57:26.295422+0000) 2024-03-04T11:57:27.274 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:57:27.295+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:57:27.295672+0000) 2024-03-04T11:57:28.274 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:57:28.295+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:57:28.295953+0000) 2024-03-04T11:57:29.274 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:57:29.295+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:57:29.296190+0000) 2024-03-04T11:57:30.275 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:57:30.295+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:57:30.296435+0000) 2024-03-04T11:57:31.275 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:57:31.296+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:57:31.296737+0000) 2024-03-04T11:57:32.275 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:57:32.296+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:57:32.297025+0000) 2024-03-04T11:57:33.276 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:57:33.296+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:57:33.297226+0000) 2024-03-04T11:57:33.846 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:57:33.855 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:57:34.276 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:57:34.297+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:57:34.297488+0000) 2024-03-04T11:57:35.276 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:57:35.296+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:57:35.297755+0000) 2024-03-04T11:57:36.277 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:57:36.297+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:57:36.298023+0000) 2024-03-04T11:57:37.277 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:57:37.297+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:57:37.298300+0000) 2024-03-04T11:57:38.277 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:57:38.298+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:57:38.298563+0000) 2024-03-04T11:57:39.277 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:57:39.298+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:57:39.298817+0000) 2024-03-04T11:57:40.278 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:57:40.298+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:57:40.299116+0000) 2024-03-04T11:57:41.278 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:57:41.299+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:57:41.299366+0000) 2024-03-04T11:57:42.278 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:57:42.299+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:57:42.299618+0000) 2024-03-04T11:57:43.279 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:57:43.299+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:57:43.299929+0000) 2024-03-04T11:57:44.279 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:57:44.299+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:57:44.300166+0000) 2024-03-04T11:57:45.279 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:57:45.300+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:57:45.300404+0000) 2024-03-04T11:57:46.280 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:57:46.300+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:57:46.300652+0000) 2024-03-04T11:57:47.280 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:57:47.300+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:57:47.300955+0000) 2024-03-04T11:57:48.280 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:57:48.301+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:57:48.301203+0000) 2024-03-04T11:57:49.281 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:57:49.301+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:57:49.301443+0000) 2024-03-04T11:57:50.281 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:57:50.301+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:57:50.301681+0000) 2024-03-04T11:57:51.281 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:57:51.300+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:57:51.301985+0000) 2024-03-04T11:57:52.282 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:57:52.301+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:57:52.302281+0000) 2024-03-04T11:57:53.282 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:57:53.301+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:57:53.302522+0000) 2024-03-04T11:57:54.282 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:57:54.301+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:57:54.302762+0000) 2024-03-04T11:57:55.282 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:57:55.301+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:57:55.302996+0000) 2024-03-04T11:57:56.283 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:57:56.302+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:57:56.303249+0000) 2024-03-04T11:57:57.283 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:57:57.302+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:57:57.303511+0000) 2024-03-04T11:57:58.284 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:57:58.301+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:57:58.303744+0000) 2024-03-04T11:57:59.284 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:57:59.302+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:57:59.303936+0000) 2024-03-04T11:58:00.284 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:58:00.303+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:58:00.304202+0000) 2024-03-04T11:58:01.284 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:58:01.303+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:58:01.304471+0000) 2024-03-04T11:58:02.284 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:58:02.303+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:58:02.304693+0000) 2024-03-04T11:58:03.285 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:58:03.304+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:58:03.304944+0000) 2024-03-04T11:58:03.978 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:58:03.990 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:58:04.285 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:58:04.304+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:58:04.305197+0000) 2024-03-04T11:58:05.285 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:58:05.304+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:58:05.305377+0000) 2024-03-04T11:58:06.285 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:58:06.304+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:58:06.305619+0000) 2024-03-04T11:58:07.286 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:58:07.305+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:58:07.305928+0000) 2024-03-04T11:58:08.286 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:58:08.305+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:58:08.306173+0000) 2024-03-04T11:58:09.286 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:58:09.305+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:58:09.306408+0000) 2024-03-04T11:58:10.287 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:58:10.305+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:58:10.306650+0000) 2024-03-04T11:58:11.287 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:58:11.306+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:58:11.306923+0000) 2024-03-04T11:58:12.288 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:58:12.306+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:58:12.307166+0000) 2024-03-04T11:58:13.288 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:58:13.306+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:58:13.307424+0000) 2024-03-04T11:58:14.288 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:58:14.306+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:58:14.307662+0000) 2024-03-04T11:58:15.288 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:58:15.307+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:58:15.307939+0000) 2024-03-04T11:58:16.289 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:58:16.307+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:58:16.308188+0000) 2024-03-04T11:58:17.289 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:58:17.307+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:58:17.308467+0000) 2024-03-04T11:58:18.289 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:58:18.307+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:58:18.308668+0000) 2024-03-04T11:58:19.289 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:58:19.308+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:58:19.308962+0000) 2024-03-04T11:58:20.290 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:58:20.308+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:58:20.309221+0000) 2024-03-04T11:58:21.290 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:58:21.308+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:58:21.309473+0000) 2024-03-04T11:58:22.290 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:58:22.309+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:58:22.309738+0000) 2024-03-04T11:58:23.290 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:58:23.309+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:58:23.309987+0000) 2024-03-04T11:58:24.291 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:58:24.309+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:58:24.310230+0000) 2024-03-04T11:58:25.291 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:58:25.309+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:58:25.310468+0000) 2024-03-04T11:58:26.291 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:58:26.310+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:58:26.310715+0000) 2024-03-04T11:58:27.292 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:58:27.310+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:58:27.310982+0000) 2024-03-04T11:58:28.292 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:58:28.310+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:58:28.311226+0000) 2024-03-04T11:58:29.293 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:58:29.310+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:58:29.311452+0000) 2024-03-04T11:58:30.293 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:58:30.310+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:58:30.311688+0000) 2024-03-04T11:58:31.293 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:58:31.311+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:58:31.311978+0000) 2024-03-04T11:58:32.293 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:58:32.311+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:58:32.312230+0000) 2024-03-04T11:58:33.293 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:58:33.312+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:58:33.312474+0000) 2024-03-04T11:58:34.201 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:58:34.209 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:58:34.294 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:58:34.311+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:58:34.312740+0000) 2024-03-04T11:58:35.294 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:58:35.312+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:58:35.312977+0000) 2024-03-04T11:58:36.294 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:58:36.312+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:58:36.313214+0000) 2024-03-04T11:58:37.295 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:58:37.312+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:58:37.313463+0000) 2024-03-04T11:58:38.295 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:58:38.313+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:58:38.313700+0000) 2024-03-04T11:58:39.295 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:58:39.313+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:58:39.313940+0000) 2024-03-04T11:58:40.295 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:58:40.313+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:58:40.314179+0000) 2024-03-04T11:58:41.296 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:58:41.314+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:58:41.314417+0000) 2024-03-04T11:58:42.296 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:58:42.313+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:58:42.314622+0000) 2024-03-04T11:58:43.296 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:58:43.314+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:58:43.314901+0000) 2024-03-04T11:58:44.297 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:58:44.314+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:58:44.315144+0000) 2024-03-04T11:58:45.297 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:58:45.315+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:58:45.315442+0000) 2024-03-04T11:58:46.297 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:58:46.315+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:58:46.315681+0000) 2024-03-04T11:58:47.297 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:58:47.315+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:58:47.315935+0000) 2024-03-04T11:58:48.298 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:58:48.315+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:58:48.316175+0000) 2024-03-04T11:58:49.298 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:58:49.316+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:58:49.316435+0000) 2024-03-04T11:58:50.298 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:58:50.316+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:58:50.316674+0000) 2024-03-04T11:58:51.299 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:58:51.316+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:58:51.316941+0000) 2024-03-04T11:58:52.299 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:58:52.317+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:58:52.317179+0000) 2024-03-04T11:58:53.299 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:58:53.317+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:58:53.317420+0000) 2024-03-04T11:58:54.300 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:58:54.316+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:58:54.317675+0000) 2024-03-04T11:58:55.300 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:58:55.316+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:58:55.317940+0000) 2024-03-04T11:58:56.300 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:58:56.317+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:58:56.318179+0000) 2024-03-04T11:58:57.300 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:58:57.317+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:58:57.318432+0000) 2024-03-04T11:58:58.301 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:58:58.317+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:58:58.318679+0000) 2024-03-04T11:58:59.301 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:58:59.317+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:58:59.318929+0000) 2024-03-04T11:59:00.301 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:59:00.318+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:59:00.319176+0000) 2024-03-04T11:59:01.302 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:59:01.318+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:59:01.319434+0000) 2024-03-04T11:59:02.302 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:59:02.318+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:59:02.319691+0000) 2024-03-04T11:59:03.302 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:59:03.318+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:59:03.319949+0000) 2024-03-04T11:59:04.302 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:59:04.319+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:59:04.320187+0000) 2024-03-04T11:59:04.416 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:59:04.424 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:59:05.303 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:59:05.319+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:59:05.320428+0000) 2024-03-04T11:59:06.303 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:59:06.319+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:59:06.320680+0000) 2024-03-04T11:59:07.303 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:59:07.320+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:59:07.320953+0000) 2024-03-04T11:59:08.304 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:59:08.320+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:59:08.321228+0000) 2024-03-04T11:59:09.304 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:59:09.320+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:59:09.321464+0000) 2024-03-04T11:59:10.304 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:59:10.320+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:59:10.321714+0000) 2024-03-04T11:59:11.304 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:59:11.321+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:59:11.321985+0000) 2024-03-04T11:59:12.305 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:59:12.321+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:59:12.322224+0000) 2024-03-04T11:59:13.305 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:59:13.321+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:59:13.322465+0000) 2024-03-04T11:59:14.305 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:59:14.321+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:59:14.322717+0000) 2024-03-04T11:59:15.306 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:59:15.322+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:59:15.322957+0000) 2024-03-04T11:59:16.306 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:59:16.322+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:59:16.323228+0000) 2024-03-04T11:59:17.306 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:59:17.322+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:59:17.323478+0000) 2024-03-04T11:59:18.307 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:59:18.322+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:59:18.323731+0000) 2024-03-04T11:59:19.307 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:59:19.323+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:59:19.323966+0000) 2024-03-04T11:59:20.307 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:59:20.323+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:59:20.324204+0000) 2024-03-04T11:59:21.307 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:59:21.323+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:59:21.324444+0000) 2024-03-04T11:59:22.308 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:59:22.323+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:59:22.324705+0000) 2024-03-04T11:59:23.308 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:59:23.324+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:59:23.324944+0000) 2024-03-04T11:59:24.308 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:59:24.324+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:59:24.325172+0000) 2024-03-04T11:59:25.308 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:59:25.324+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:59:25.325405+0000) 2024-03-04T11:59:26.309 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:59:26.325+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:59:26.325643+0000) 2024-03-04T11:59:27.309 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:59:27.325+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:59:27.325939+0000) 2024-03-04T11:59:28.309 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:59:28.325+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:59:28.326183+0000) 2024-03-04T11:59:29.310 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:59:29.325+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:59:29.326416+0000) 2024-03-04T11:59:30.310 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:59:30.326+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:59:30.326662+0000) 2024-03-04T11:59:31.310 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:59:31.326+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:59:31.326950+0000) 2024-03-04T11:59:32.311 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:59:32.326+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:59:32.327210+0000) 2024-03-04T11:59:33.311 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:59:33.326+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:59:33.327449+0000) 2024-03-04T11:59:34.311 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:59:34.327+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:59:34.327680+0000) 2024-03-04T11:59:34.558 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:59:34.566 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T11:59:35.311 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:59:35.327+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:59:35.327959+0000) 2024-03-04T11:59:36.312 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:59:36.327+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:59:36.328219+0000) 2024-03-04T11:59:37.312 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:59:37.328+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:59:37.328475+0000) 2024-03-04T11:59:38.312 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:59:38.328+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:59:38.328778+0000) 2024-03-04T11:59:39.313 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:59:39.328+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:59:39.329020+0000) 2024-03-04T11:59:40.313 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:59:40.328+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:59:40.329257+0000) 2024-03-04T11:59:41.313 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:59:41.329+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:59:41.329504+0000) 2024-03-04T11:59:42.313 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:59:42.329+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:59:42.329767+0000) 2024-03-04T11:59:43.314 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:59:43.329+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:59:43.330022+0000) 2024-03-04T11:59:44.314 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:59:44.329+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:59:44.330260+0000) 2024-03-04T11:59:45.314 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:59:45.330+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:59:45.330499+0000) 2024-03-04T11:59:46.315 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:59:46.330+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:59:46.330742+0000) 2024-03-04T11:59:47.315 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:59:47.330+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:59:47.330986+0000) 2024-03-04T11:59:48.315 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:59:48.330+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:59:48.331234+0000) 2024-03-04T11:59:49.315 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:59:49.330+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:59:49.331464+0000) 2024-03-04T11:59:50.316 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:59:50.331+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:59:50.331705+0000) 2024-03-04T11:59:51.316 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:59:51.331+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:59:51.331944+0000) 2024-03-04T11:59:52.316 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:59:52.332+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:59:52.332213+0000) 2024-03-04T11:59:53.317 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:59:53.332+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:59:53.332471+0000) 2024-03-04T11:59:54.317 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:59:54.332+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:59:54.332715+0000) 2024-03-04T11:59:55.317 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:59:55.332+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:59:55.332953+0000) 2024-03-04T11:59:56.317 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:59:56.332+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:59:56.333188+0000) 2024-03-04T11:59:57.318 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:59:57.332+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:59:57.333441+0000) 2024-03-04T11:59:58.318 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:59:58.332+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:59:58.333697+0000) 2024-03-04T11:59:59.318 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T11:59:59.332+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T10:59:59.333976+0000) 2024-03-04T12:00:00.319 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:00:00.333+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:00:00.334240+0000) 2024-03-04T12:00:01.319 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:00:01.333+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:00:01.334518+0000) 2024-03-04T12:00:02.319 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:00:02.332+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:00:02.334788+0000) 2024-03-04T12:00:03.320 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:00:03.334+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:00:03.335089+0000) 2024-03-04T12:00:04.320 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:00:04.334+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:00:04.335345+0000) 2024-03-04T12:00:04.795 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:00:04.803 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:00:05.320 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:00:05.334+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:00:05.335609+0000) 2024-03-04T12:00:06.323 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:00:06.334+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:00:06.335918+0000) 2024-03-04T12:00:07.322 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:00:07.335+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:00:07.336198+0000) 2024-03-04T12:00:08.321 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:00:08.335+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:00:08.336454+0000) 2024-03-04T12:00:09.321 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:00:09.335+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:00:09.336711+0000) 2024-03-04T12:00:10.322 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:00:10.336+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:00:10.336976+0000) 2024-03-04T12:00:11.322 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:00:11.336+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:00:11.337236+0000) 2024-03-04T12:00:12.322 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:00:12.336+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:00:12.337493+0000) 2024-03-04T12:00:13.322 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:00:13.336+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:00:13.337759+0000) 2024-03-04T12:00:14.323 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:00:14.337+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:00:14.338011+0000) 2024-03-04T12:00:15.323 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:00:15.337+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:00:15.338216+0000) 2024-03-04T12:00:16.323 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:00:16.337+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:00:16.338480+0000) 2024-03-04T12:00:17.324 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:00:17.337+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:00:17.338753+0000) 2024-03-04T12:00:18.324 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:00:18.338+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:00:18.339012+0000) 2024-03-04T12:00:19.324 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:00:19.338+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:00:19.339270+0000) 2024-03-04T12:00:20.325 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:00:20.338+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:00:20.339544+0000) 2024-03-04T12:00:21.325 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:00:21.338+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:00:21.339792+0000) 2024-03-04T12:00:22.325 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:00:22.339+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:00:22.340079+0000) 2024-03-04T12:00:23.326 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:00:23.339+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:00:23.340323+0000) 2024-03-04T12:00:24.326 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:00:24.339+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:00:24.340558+0000) 2024-03-04T12:00:25.326 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:00:25.340+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:00:25.340814+0000) 2024-03-04T12:00:26.326 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:00:26.340+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:00:26.341184+0000) 2024-03-04T12:00:27.327 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:00:27.340+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:00:27.341474+0000) 2024-03-04T12:00:28.327 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:00:28.341+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:00:28.341718+0000) 2024-03-04T12:00:29.327 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:00:29.341+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:00:29.341972+0000) 2024-03-04T12:00:30.328 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:00:30.341+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:00:30.342229+0000) 2024-03-04T12:00:31.328 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:00:31.341+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:00:31.342464+0000) 2024-03-04T12:00:32.328 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:00:32.342+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:00:32.342705+0000) 2024-03-04T12:00:33.328 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:00:33.342+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:00:33.342948+0000) 2024-03-04T12:00:34.329 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:00:34.342+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:00:34.343200+0000) 2024-03-04T12:00:35.009 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:00:35.018 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:00:35.329 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:00:35.342+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:00:35.343454+0000) 2024-03-04T12:00:36.329 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:00:36.343+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:00:36.343719+0000) 2024-03-04T12:00:37.330 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:00:37.343+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:00:37.343993+0000) 2024-03-04T12:00:38.330 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:00:38.343+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:00:38.344207+0000) 2024-03-04T12:00:39.330 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:00:39.344+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:00:39.344461+0000) 2024-03-04T12:00:40.330 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:00:40.344+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:00:40.344694+0000) 2024-03-04T12:00:41.331 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:00:41.344+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:00:41.344938+0000) 2024-03-04T12:00:42.331 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:00:42.344+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:00:42.345185+0000) 2024-03-04T12:00:43.331 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:00:43.345+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:00:43.345427+0000) 2024-03-04T12:00:44.332 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:00:44.345+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:00:44.345689+0000) 2024-03-04T12:00:45.332 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:00:45.345+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:00:45.345960+0000) 2024-03-04T12:00:46.332 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:00:46.345+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:00:46.346235+0000) 2024-03-04T12:00:47.332 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:00:47.346+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:00:47.346504+0000) 2024-03-04T12:00:48.333 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:00:48.345+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:00:48.346766+0000) 2024-03-04T12:00:49.333 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:00:49.346+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:00:49.347006+0000) 2024-03-04T12:00:50.333 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:00:50.346+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:00:50.347248+0000) 2024-03-04T12:00:51.334 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:00:51.347+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:00:51.347490+0000) 2024-03-04T12:00:52.334 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:00:52.347+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:00:52.347732+0000) 2024-03-04T12:00:53.334 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:00:53.347+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:00:53.347977+0000) 2024-03-04T12:00:54.336 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:00:54.348+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:00:54.348213+0000) 2024-03-04T12:00:55.335 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:00:55.348+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:00:55.348452+0000) 2024-03-04T12:00:56.335 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:00:56.348+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:00:56.348695+0000) 2024-03-04T12:00:57.335 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:00:57.348+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:00:57.348954+0000) 2024-03-04T12:00:58.336 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:00:58.348+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:00:58.349224+0000) 2024-03-04T12:00:59.336 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:00:59.348+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:00:59.349471+0000) 2024-03-04T12:01:00.336 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:01:00.348+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:01:00.349758+0000) 2024-03-04T12:01:01.336 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:01:01.348+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:01:01.350008+0000) 2024-03-04T12:01:02.337 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:01:02.349+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:01:02.350268+0000) 2024-03-04T12:01:03.337 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:01:03.349+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:01:03.350560+0000) 2024-03-04T12:01:04.337 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:01:04.349+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:01:04.350815+0000) 2024-03-04T12:01:05.224 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:01:05.231 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:01:05.338 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:01:05.350+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:01:05.351060+0000) 2024-03-04T12:01:06.338 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:01:06.350+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:01:06.351303+0000) 2024-03-04T12:01:07.338 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:01:07.349+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:01:07.351519+0000) 2024-03-04T12:01:08.338 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:01:08.350+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:01:08.351781+0000) 2024-03-04T12:01:09.339 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:01:09.351+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:01:09.352074+0000) 2024-03-04T12:01:10.339 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:01:10.351+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:01:10.352361+0000) 2024-03-04T12:01:11.339 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:01:11.351+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:01:11.352617+0000) 2024-03-04T12:01:12.340 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:01:12.352+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:01:12.352921+0000) 2024-03-04T12:01:13.340 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:01:13.352+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:01:13.353190+0000) 2024-03-04T12:01:14.343 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:01:14.352+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:01:14.353440+0000) 2024-03-04T12:01:15.341 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:01:15.352+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:01:15.353691+0000) 2024-03-04T12:01:16.341 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:01:16.353+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:01:16.353948+0000) 2024-03-04T12:01:17.341 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:01:17.353+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:01:17.354220+0000) 2024-03-04T12:01:18.341 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:01:18.353+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:01:18.354483+0000) 2024-03-04T12:01:19.342 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:01:19.353+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:01:19.354732+0000) 2024-03-04T12:01:20.342 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:01:20.354+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:01:20.355091+0000) 2024-03-04T12:01:21.342 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:01:21.354+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:01:21.355346+0000) 2024-03-04T12:01:22.343 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:01:22.354+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:01:22.355606+0000) 2024-03-04T12:01:23.343 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:01:23.355+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:01:23.355922+0000) 2024-03-04T12:01:24.343 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:01:24.355+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:01:24.356177+0000) 2024-03-04T12:01:25.344 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:01:25.355+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:01:25.356431+0000) 2024-03-04T12:01:26.344 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:01:26.356+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:01:26.356627+0000) 2024-03-04T12:01:27.344 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:01:27.356+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:01:27.356916+0000) 2024-03-04T12:01:28.345 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:01:28.356+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:01:28.357186+0000) 2024-03-04T12:01:29.345 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:01:29.356+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:01:29.357448+0000) 2024-03-04T12:01:30.345 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:01:30.357+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:01:30.357690+0000) 2024-03-04T12:01:31.345 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:01:31.357+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:01:31.357930+0000) 2024-03-04T12:01:32.346 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:01:32.357+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:01:32.358176+0000) 2024-03-04T12:01:33.346 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:01:33.357+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:01:33.358428+0000) 2024-03-04T12:01:34.346 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:01:34.358+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:01:34.358671+0000) 2024-03-04T12:01:35.346 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:01:35.358+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:01:35.358941+0000) 2024-03-04T12:01:35.429 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:01:35.438 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:01:36.347 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:01:36.358+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:01:36.359139+0000) 2024-03-04T12:01:37.347 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:01:37.358+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:01:37.359393+0000) 2024-03-04T12:01:38.347 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:01:38.359+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:01:38.359674+0000) 2024-03-04T12:01:39.347 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:01:39.359+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:01:39.359941+0000) 2024-03-04T12:01:40.348 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:01:40.359+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:01:40.360177+0000) 2024-03-04T12:01:41.348 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:01:41.359+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:01:41.360411+0000) 2024-03-04T12:01:42.348 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:01:42.360+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:01:42.360653+0000) 2024-03-04T12:01:43.349 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:01:43.360+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:01:43.360948+0000) 2024-03-04T12:01:44.349 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:01:44.360+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:01:44.361239+0000) 2024-03-04T12:01:45.349 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:01:45.361+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:01:45.361496+0000) 2024-03-04T12:01:46.349 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:01:46.361+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:01:46.361743+0000) 2024-03-04T12:01:47.350 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:01:47.361+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:01:47.361991+0000) 2024-03-04T12:01:48.350 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:01:48.361+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:01:48.362238+0000) 2024-03-04T12:01:49.350 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:01:49.362+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:01:49.362473+0000) 2024-03-04T12:01:50.351 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:01:50.362+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:01:50.362716+0000) 2024-03-04T12:01:51.351 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:01:51.362+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:01:51.362948+0000) 2024-03-04T12:01:52.351 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:01:52.362+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:01:52.363212+0000) 2024-03-04T12:01:53.352 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:01:53.363+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:01:53.363458+0000) 2024-03-04T12:01:54.352 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:01:54.363+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:01:54.363736+0000) 2024-03-04T12:01:55.352 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:01:55.363+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:01:55.363971+0000) 2024-03-04T12:01:56.353 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:01:56.363+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:01:56.364201+0000) 2024-03-04T12:01:57.353 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:01:57.364+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:01:57.364458+0000) 2024-03-04T12:01:58.353 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:01:58.363+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:01:58.364720+0000) 2024-03-04T12:01:59.353 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:01:59.364+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:01:59.364992+0000) 2024-03-04T12:02:00.353 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:02:00.365+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:02:00.365237+0000) 2024-03-04T12:02:01.354 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:02:01.364+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:02:01.365477+0000) 2024-03-04T12:02:02.354 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:02:02.364+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:02:02.365727+0000) 2024-03-04T12:02:03.354 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:02:03.364+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:02:03.365973+0000) 2024-03-04T12:02:04.355 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:02:04.365+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:02:04.366211+0000) 2024-03-04T12:02:05.355 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:02:05.365+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:02:05.366449+0000) 2024-03-04T12:02:05.652 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:02:05.660 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:02:06.355 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:02:06.365+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:02:06.366688+0000) 2024-03-04T12:02:07.355 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:02:07.365+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:02:07.366989+0000) 2024-03-04T12:02:08.356 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:02:08.366+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:02:08.367237+0000) 2024-03-04T12:02:09.356 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:02:09.366+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:02:09.367475+0000) 2024-03-04T12:02:10.356 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:02:10.366+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:02:10.367714+0000) 2024-03-04T12:02:11.356 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:02:11.367+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:02:11.367986+0000) 2024-03-04T12:02:12.357 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:02:12.367+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:02:12.368236+0000) 2024-03-04T12:02:13.357 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:02:13.367+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:02:13.368501+0000) 2024-03-04T12:02:14.357 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:02:14.367+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:02:14.368749+0000) 2024-03-04T12:02:15.358 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:02:15.368+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:02:15.368989+0000) 2024-03-04T12:02:16.358 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:02:16.368+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:02:16.369232+0000) 2024-03-04T12:02:17.358 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:02:17.368+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:02:17.369491+0000) 2024-03-04T12:02:18.359 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:02:18.369+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:02:18.369815+0000) 2024-03-04T12:02:19.359 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:02:19.369+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:02:19.370089+0000) 2024-03-04T12:02:20.359 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:02:20.369+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:02:20.370345+0000) 2024-03-04T12:02:21.359 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:02:21.369+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:02:21.370584+0000) 2024-03-04T12:02:22.363 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:02:22.370+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:02:22.370867+0000) 2024-03-04T12:02:23.360 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:02:23.370+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:02:23.371117+0000) 2024-03-04T12:02:24.360 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:02:24.370+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:02:24.371364+0000) 2024-03-04T12:02:25.361 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:02:25.370+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:02:25.371624+0000) 2024-03-04T12:02:26.361 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:02:26.371+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:02:26.371897+0000) 2024-03-04T12:02:27.361 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:02:27.371+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:02:27.372159+0000) 2024-03-04T12:02:28.361 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:02:28.371+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:02:28.372426+0000) 2024-03-04T12:02:29.362 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:02:29.372+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:02:29.372662+0000) 2024-03-04T12:02:30.362 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:02:30.372+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:02:30.372940+0000) 2024-03-04T12:02:31.362 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:02:31.372+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:02:31.373175+0000) 2024-03-04T12:02:32.362 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:02:32.372+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:02:32.373441+0000) 2024-03-04T12:02:33.363 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:02:33.373+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:02:33.373691+0000) 2024-03-04T12:02:34.363 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:02:34.373+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:02:34.373944+0000) 2024-03-04T12:02:35.363 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:02:35.373+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:02:35.374177+0000) 2024-03-04T12:02:35.859 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:02:35.868 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:02:36.364 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:02:36.373+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:02:36.374437+0000) 2024-03-04T12:02:37.364 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:02:37.374+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:02:37.374709+0000) 2024-03-04T12:02:38.364 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:02:38.374+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:02:38.374955+0000) 2024-03-04T12:02:39.364 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:02:39.374+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:02:39.375189+0000) 2024-03-04T12:02:40.365 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:02:40.374+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:02:40.375427+0000) 2024-03-04T12:02:41.365 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:02:41.375+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:02:41.375681+0000) 2024-03-04T12:02:42.365 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:02:42.375+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:02:42.375966+0000) 2024-03-04T12:02:43.366 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:02:43.375+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:02:43.376218+0000) 2024-03-04T12:02:44.366 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:02:44.376+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:02:44.376477+0000) 2024-03-04T12:02:45.366 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:02:45.376+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:02:45.376738+0000) 2024-03-04T12:02:46.366 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:02:46.376+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:02:46.377001+0000) 2024-03-04T12:02:47.367 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:02:47.376+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:02:47.377250+0000) 2024-03-04T12:02:48.367 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:02:48.377+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:02:48.377516+0000) 2024-03-04T12:02:49.367 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:02:49.377+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:02:49.377754+0000) 2024-03-04T12:02:50.368 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:02:50.377+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:02:50.378034+0000) 2024-03-04T12:02:51.368 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:02:51.377+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:02:51.378271+0000) 2024-03-04T12:02:52.368 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:02:52.378+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:02:52.378512+0000) 2024-03-04T12:02:53.368 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:02:53.378+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:02:53.378759+0000) 2024-03-04T12:02:54.369 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:02:54.378+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:02:54.378993+0000) 2024-03-04T12:02:55.369 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:02:55.378+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:02:55.379227+0000) 2024-03-04T12:02:56.369 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:02:56.379+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:02:56.379469+0000) 2024-03-04T12:02:57.369 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:02:57.379+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:02:57.379714+0000) 2024-03-04T12:02:58.370 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:02:58.379+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:02:58.379987+0000) 2024-03-04T12:02:59.370 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:02:59.380+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:02:59.380244+0000) 2024-03-04T12:03:00.370 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:03:00.380+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:03:00.380507+0000) 2024-03-04T12:03:01.371 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:03:01.380+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:03:01.380748+0000) 2024-03-04T12:03:02.371 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:03:02.380+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:03:02.380986+0000) 2024-03-04T12:03:03.371 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:03:03.380+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:03:03.381237+0000) 2024-03-04T12:03:04.371 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:03:04.379+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:03:04.381492+0000) 2024-03-04T12:03:05.372 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:03:05.380+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:03:05.381730+0000) 2024-03-04T12:03:06.030 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:03:06.038 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:03:06.372 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:03:06.380+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:03:06.381982+0000) 2024-03-04T12:03:07.372 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:03:07.381+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:03:07.382240+0000) 2024-03-04T12:03:08.372 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:03:08.381+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:03:08.382527+0000) 2024-03-04T12:03:09.373 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:03:09.381+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:03:09.382782+0000) 2024-03-04T12:03:10.373 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:03:10.381+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:03:10.383007+0000) 2024-03-04T12:03:11.373 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:03:11.382+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:03:11.383245+0000) 2024-03-04T12:03:12.374 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:03:12.382+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:03:12.383531+0000) 2024-03-04T12:03:13.374 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:03:13.382+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:03:13.383780+0000) 2024-03-04T12:03:14.374 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:03:14.383+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:03:14.384019+0000) 2024-03-04T12:03:15.374 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:03:15.383+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:03:15.384254+0000) 2024-03-04T12:03:16.375 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:03:16.383+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:03:16.384494+0000) 2024-03-04T12:03:17.375 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:03:17.383+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:03:17.384775+0000) 2024-03-04T12:03:18.375 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:03:18.384+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:03:18.385034+0000) 2024-03-04T12:03:19.375 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:03:19.384+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:03:19.385266+0000) 2024-03-04T12:03:20.376 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:03:20.384+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:03:20.385502+0000) 2024-03-04T12:03:21.376 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:03:21.384+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:03:21.385742+0000) 2024-03-04T12:03:22.376 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:03:22.385+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:03:22.386006+0000) 2024-03-04T12:03:23.377 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:03:23.385+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:03:23.386244+0000) 2024-03-04T12:03:24.377 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:03:24.385+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:03:24.386489+0000) 2024-03-04T12:03:25.380 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:03:25.385+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:03:25.386677+0000) 2024-03-04T12:03:26.377 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:03:26.386+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:03:26.386940+0000) 2024-03-04T12:03:27.378 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:03:27.386+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:03:27.387191+0000) 2024-03-04T12:03:28.378 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:03:28.386+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:03:28.387456+0000) 2024-03-04T12:03:29.378 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:03:29.387+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:03:29.387716+0000) 2024-03-04T12:03:30.379 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:03:30.387+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:03:30.387990+0000) 2024-03-04T12:03:31.379 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:03:31.387+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:03:31.388232+0000) 2024-03-04T12:03:32.379 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:03:32.387+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:03:32.388471+0000) 2024-03-04T12:03:33.379 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:03:33.388+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:03:33.388709+0000) 2024-03-04T12:03:34.380 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:03:34.388+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:03:34.388974+0000) 2024-03-04T12:03:35.380 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:03:35.388+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:03:35.389207+0000) 2024-03-04T12:03:36.262 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:03:36.270 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:03:36.380 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:03:36.388+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:03:36.389485+0000) 2024-03-04T12:03:37.380 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:03:37.389+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:03:37.389738+0000) 2024-03-04T12:03:38.381 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:03:38.389+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:03:38.389983+0000) 2024-03-04T12:03:39.381 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:03:39.389+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:03:39.390214+0000) 2024-03-04T12:03:40.381 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:03:40.389+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:03:40.390457+0000) 2024-03-04T12:03:41.382 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:03:41.390+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:03:41.390693+0000) 2024-03-04T12:03:42.382 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:03:42.390+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:03:42.390943+0000) 2024-03-04T12:03:43.382 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:03:43.390+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:03:43.391186+0000) 2024-03-04T12:03:44.382 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:03:44.390+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:03:44.391457+0000) 2024-03-04T12:03:45.383 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:03:45.391+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:03:45.391705+0000) 2024-03-04T12:03:46.383 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:03:46.391+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:03:46.391946+0000) 2024-03-04T12:03:47.383 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:03:47.391+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:03:47.392192+0000) 2024-03-04T12:03:48.383 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:03:48.392+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:03:48.392431+0000) 2024-03-04T12:03:49.384 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:03:49.392+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:03:49.392677+0000) 2024-03-04T12:03:50.384 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:03:50.392+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:03:50.392942+0000) 2024-03-04T12:03:51.384 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:03:51.392+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:03:51.393189+0000) 2024-03-04T12:03:52.385 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:03:52.393+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:03:52.393425+0000) 2024-03-04T12:03:53.385 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:03:53.393+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:03:53.393664+0000) 2024-03-04T12:03:54.385 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:03:54.393+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:03:54.393880+0000) 2024-03-04T12:03:55.385 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:03:55.393+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:03:55.394119+0000) 2024-03-04T12:03:56.386 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:03:56.394+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:03:56.394362+0000) 2024-03-04T12:03:57.386 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:03:57.394+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:03:57.394606+0000) 2024-03-04T12:03:58.387 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:03:58.394+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:03:58.394916+0000) 2024-03-04T12:03:59.386 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:03:59.394+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:03:59.395179+0000) 2024-03-04T12:04:00.387 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:04:00.395+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:04:00.395414+0000) 2024-03-04T12:04:01.387 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:04:01.394+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:04:01.395643+0000) 2024-03-04T12:04:02.387 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:04:02.395+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:04:02.395853+0000) 2024-03-04T12:04:03.387 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:04:03.395+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:04:03.396045+0000) 2024-03-04T12:04:04.388 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:04:04.395+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:04:04.396227+0000) 2024-03-04T12:04:05.388 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:04:05.395+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:04:05.396458+0000) 2024-03-04T12:04:06.388 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:04:06.394+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:04:06.396711+0000) 2024-03-04T12:04:06.576 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:04:06.581 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:04:07.388 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:04:07.395+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:04:07.396967+0000) 2024-03-04T12:04:08.389 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:04:08.396+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:04:08.397211+0000) 2024-03-04T12:04:09.389 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:04:09.396+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:04:09.397449+0000) 2024-03-04T12:04:10.389 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:04:10.395+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:04:10.397703+0000) 2024-03-04T12:04:11.390 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:04:11.396+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:04:11.397941+0000) 2024-03-04T12:04:12.390 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:04:12.397+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:04:12.398175+0000) 2024-03-04T12:04:13.390 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:04:13.397+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:04:13.398420+0000) 2024-03-04T12:04:14.390 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:04:14.397+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:04:14.398660+0000) 2024-03-04T12:04:15.390 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:04:15.397+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:04:15.398953+0000) 2024-03-04T12:04:16.391 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:04:16.398+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:04:16.399127+0000) 2024-03-04T12:04:17.391 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:04:17.398+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:04:17.399375+0000) 2024-03-04T12:04:18.391 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:04:18.398+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:04:18.399635+0000) 2024-03-04T12:04:19.392 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:04:19.399+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:04:19.399917+0000) 2024-03-04T12:04:20.395 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:04:20.399+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:04:20.400156+0000) 2024-03-04T12:04:21.392 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:04:21.399+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:04:21.400397+0000) 2024-03-04T12:04:22.392 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:04:22.399+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:04:22.400633+0000) 2024-03-04T12:04:23.393 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:04:23.400+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:04:23.400927+0000) 2024-03-04T12:04:24.393 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:04:24.400+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:04:24.401167+0000) 2024-03-04T12:04:25.393 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:04:25.400+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:04:25.401409+0000) 2024-03-04T12:04:26.393 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:04:26.400+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:04:26.401661+0000) 2024-03-04T12:04:27.394 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:04:27.401+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:04:27.401946+0000) 2024-03-04T12:04:28.394 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:04:28.401+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:04:28.402136+0000) 2024-03-04T12:04:29.394 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:04:29.401+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:04:29.402389+0000) 2024-03-04T12:04:30.394 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:04:30.401+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:04:30.402568+0000) 2024-03-04T12:04:31.395 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:04:31.402+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:04:31.402853+0000) 2024-03-04T12:04:32.395 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:04:32.402+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:04:32.403052+0000) 2024-03-04T12:04:33.395 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:04:33.402+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:04:33.403298+0000) 2024-03-04T12:04:34.396 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:04:34.402+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:04:34.403537+0000) 2024-03-04T12:04:35.396 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:04:35.403+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:04:35.403777+0000) 2024-03-04T12:04:36.396 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:04:36.403+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:04:36.404012+0000) 2024-03-04T12:04:36.790 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:04:36.798 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:04:37.396 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:04:37.403+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:04:37.404269+0000) 2024-03-04T12:04:38.397 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:04:38.403+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:04:38.404507+0000) 2024-03-04T12:04:39.397 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:04:39.404+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:04:39.404740+0000) 2024-03-04T12:04:40.397 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:04:40.404+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:04:40.404975+0000) 2024-03-04T12:04:41.397 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:04:41.404+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:04:41.405211+0000) 2024-03-04T12:04:42.398 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:04:42.404+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:04:42.405489+0000) 2024-03-04T12:04:43.398 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:04:43.405+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:04:43.405728+0000) 2024-03-04T12:04:44.398 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:04:44.405+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:04:44.405972+0000) 2024-03-04T12:04:45.398 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:04:45.405+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:04:45.406209+0000) 2024-03-04T12:04:46.399 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:04:46.405+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:04:46.406454+0000) 2024-03-04T12:04:47.399 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:04:47.406+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:04:47.406697+0000) 2024-03-04T12:04:48.399 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:04:48.406+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:04:48.406985+0000) 2024-03-04T12:04:49.400 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:04:49.406+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:04:49.407223+0000) 2024-03-04T12:04:50.400 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:04:50.407+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:04:50.407466+0000) 2024-03-04T12:04:51.400 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:04:51.407+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:04:51.407708+0000) 2024-03-04T12:04:52.401 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:04:52.407+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:04:52.407948+0000) 2024-03-04T12:04:53.401 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:04:53.407+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:04:53.408190+0000) 2024-03-04T12:04:54.401 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:04:54.408+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:04:54.408434+0000) 2024-03-04T12:04:55.402 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:04:55.408+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:04:55.408667+0000) 2024-03-04T12:04:56.401 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:04:56.408+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:04:56.408944+0000) 2024-03-04T12:04:57.402 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:04:57.408+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:04:57.409195+0000) 2024-03-04T12:04:58.402 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:04:58.408+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:04:58.409386+0000) 2024-03-04T12:04:59.402 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:04:59.409+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:04:59.409621+0000) 2024-03-04T12:05:00.402 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:05:00.409+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:05:00.409933+0000) 2024-03-04T12:05:01.403 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:05:01.409+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:05:01.410195+0000) 2024-03-04T12:05:02.403 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:05:02.410+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:05:02.410453+0000) 2024-03-04T12:05:03.403 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:05:03.410+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:05:03.410700+0000) 2024-03-04T12:05:04.404 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:05:04.410+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:05:04.410945+0000) 2024-03-04T12:05:05.404 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:05:05.411+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:05:05.411216+0000) 2024-03-04T12:05:06.404 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:05:06.411+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:05:06.411453+0000) 2024-03-04T12:05:07.011 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:05:07.020 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:05:07.404 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:05:07.410+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:05:07.411707+0000) 2024-03-04T12:05:08.405 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:05:08.410+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:05:08.411957+0000) 2024-03-04T12:05:09.405 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:05:09.411+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:05:09.412194+0000) 2024-03-04T12:05:10.405 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:05:10.411+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:05:10.412433+0000) 2024-03-04T12:05:11.405 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:05:11.411+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:05:11.412668+0000) 2024-03-04T12:05:12.406 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:05:12.411+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:05:12.412931+0000) 2024-03-04T12:05:13.406 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:05:13.412+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:05:13.413174+0000) 2024-03-04T12:05:14.406 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:05:14.412+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:05:14.413419+0000) 2024-03-04T12:05:15.407 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:05:15.412+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:05:15.413653+0000) 2024-03-04T12:05:16.407 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:05:16.412+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:05:16.413925+0000) 2024-03-04T12:05:17.407 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:05:17.413+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:05:17.414170+0000) 2024-03-04T12:05:18.407 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:05:18.413+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:05:18.414448+0000) 2024-03-04T12:05:19.408 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:05:19.413+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:05:19.414687+0000) 2024-03-04T12:05:20.408 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:05:20.414+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:05:20.414929+0000) 2024-03-04T12:05:21.408 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:05:21.414+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:05:21.415164+0000) 2024-03-04T12:05:22.408 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:05:22.414+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:05:22.415399+0000) 2024-03-04T12:05:23.409 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:05:23.414+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:05:23.415647+0000) 2024-03-04T12:05:24.409 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:05:24.415+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:05:24.415927+0000) 2024-03-04T12:05:25.409 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:05:25.415+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:05:25.416174+0000) 2024-03-04T12:05:26.409 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:05:26.415+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:05:26.416412+0000) 2024-03-04T12:05:27.410 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:05:27.415+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:05:27.416653+0000) 2024-03-04T12:05:28.410 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:05:28.416+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:05:28.416933+0000) 2024-03-04T12:05:29.410 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:05:29.416+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:05:29.417175+0000) 2024-03-04T12:05:30.410 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:05:30.416+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:05:30.417427+0000) 2024-03-04T12:05:31.412 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:05:31.416+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:05:31.417682+0000) 2024-03-04T12:05:32.411 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:05:32.417+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:05:32.417944+0000) 2024-03-04T12:05:33.411 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:05:33.417+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:05:33.418127+0000) 2024-03-04T12:05:34.412 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:05:34.417+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:05:34.418371+0000) 2024-03-04T12:05:35.412 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:05:35.417+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:05:35.418607+0000) 2024-03-04T12:05:36.412 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:05:36.418+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:05:36.418880+0000) 2024-03-04T12:05:37.149 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:05:37.157 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:05:37.412 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:05:37.418+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:05:37.419133+0000) 2024-03-04T12:05:38.413 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:05:38.418+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:05:38.419327+0000) 2024-03-04T12:05:39.413 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:05:39.418+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:05:39.419566+0000) 2024-03-04T12:05:40.413 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:05:40.419+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:05:40.419809+0000) 2024-03-04T12:05:41.413 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:05:41.419+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:05:41.420101+0000) 2024-03-04T12:05:42.416 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:05:42.419+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:05:42.420286+0000) 2024-03-04T12:05:43.414 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:05:43.419+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:05:43.420571+0000) 2024-03-04T12:05:44.414 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:05:44.420+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:05:44.420891+0000) 2024-03-04T12:05:45.415 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:05:45.420+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:05:45.421181+0000) 2024-03-04T12:05:46.415 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:05:46.420+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:05:46.421537+0000) 2024-03-04T12:05:47.415 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:05:47.421+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:05:47.421867+0000) 2024-03-04T12:05:48.416 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:05:48.421+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:05:48.422160+0000) 2024-03-04T12:05:49.416 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:05:49.421+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:05:49.422448+0000) 2024-03-04T12:05:50.417 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:05:50.421+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:05:50.422763+0000) 2024-03-04T12:05:51.417 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:05:51.422+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:05:51.423078+0000) 2024-03-04T12:05:52.417 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:05:52.422+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:05:52.423372+0000) 2024-03-04T12:05:53.417 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:05:53.423+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:05:53.423668+0000) 2024-03-04T12:05:54.418 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:05:54.423+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:05:54.424000+0000) 2024-03-04T12:05:55.418 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:05:55.423+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:05:55.424293+0000) 2024-03-04T12:05:56.418 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:05:56.424+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:05:56.424589+0000) 2024-03-04T12:05:57.419 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:05:57.424+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:05:57.424915+0000) 2024-03-04T12:05:58.419 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:05:58.424+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:05:58.425235+0000) 2024-03-04T12:05:59.419 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:05:59.425+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:05:59.425536+0000) 2024-03-04T12:06:00.420 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:06:00.425+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:06:00.425805+0000) 2024-03-04T12:06:01.420 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:06:01.425+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:06:01.426145+0000) 2024-03-04T12:06:02.420 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:06:02.426+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:06:02.426435+0000) 2024-03-04T12:06:03.421 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:06:03.425+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:06:03.426763+0000) 2024-03-04T12:06:04.421 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:06:04.426+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:06:04.427029+0000) 2024-03-04T12:06:05.421 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:06:05.427+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:06:05.427322+0000) 2024-03-04T12:06:06.422 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:06:06.427+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:06:06.427611+0000) 2024-03-04T12:06:07.365 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:06:07.374 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:06:07.422 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:06:07.427+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:06:07.427862+0000) 2024-03-04T12:06:08.422 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:06:08.427+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:06:08.428146+0000) 2024-03-04T12:06:09.422 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:06:09.427+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:06:09.428472+0000) 2024-03-04T12:06:10.423 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:06:10.427+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:06:10.428708+0000) 2024-03-04T12:06:11.423 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:06:11.427+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:06:11.428998+0000) 2024-03-04T12:06:12.423 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:06:12.428+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:06:12.429280+0000) 2024-03-04T12:06:13.424 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:06:13.428+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:06:13.429576+0000) 2024-03-04T12:06:14.424 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:06:14.428+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:06:14.429922+0000) 2024-03-04T12:06:15.424 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:06:15.429+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:06:15.430213+0000) 2024-03-04T12:06:16.425 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:06:16.429+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:06:16.430524+0000) 2024-03-04T12:06:17.425 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:06:17.429+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:06:17.430870+0000) 2024-03-04T12:06:18.425 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:06:18.430+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:06:18.431170+0000) 2024-03-04T12:06:19.426 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:06:19.430+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:06:19.431420+0000) 2024-03-04T12:06:20.426 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:06:20.430+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:06:20.431695+0000) 2024-03-04T12:06:21.426 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:06:21.431+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:06:21.432017+0000) 2024-03-04T12:06:22.427 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:06:22.431+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:06:22.432251+0000) 2024-03-04T12:06:23.427 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:06:23.431+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:06:23.432497+0000) 2024-03-04T12:06:24.427 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:06:24.432+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:06:24.432873+0000) 2024-03-04T12:06:25.427 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:06:25.432+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:06:25.433128+0000) 2024-03-04T12:06:26.428 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:06:26.432+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:06:26.433427+0000) 2024-03-04T12:06:27.428 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:06:27.432+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:06:27.433733+0000) 2024-03-04T12:06:28.428 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:06:28.433+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:06:28.434033+0000) 2024-03-04T12:06:29.429 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:06:29.433+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:06:29.434332+0000) 2024-03-04T12:06:30.429 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:06:30.433+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:06:30.434629+0000) 2024-03-04T12:06:31.429 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:06:31.434+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:06:31.434950+0000) 2024-03-04T12:06:32.430 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:06:32.434+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:06:32.435248+0000) 2024-03-04T12:06:33.430 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:06:33.434+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:06:33.435537+0000) 2024-03-04T12:06:34.430 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:06:34.435+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:06:34.435871+0000) 2024-03-04T12:06:35.431 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:06:35.435+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:06:35.436166+0000) 2024-03-04T12:06:36.431 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:06:36.435+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:06:36.436463+0000) 2024-03-04T12:06:37.431 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:06:37.436+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:06:37.436770+0000) 2024-03-04T12:06:37.548 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:06:37.560 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:06:38.432 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:06:38.436+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:06:38.437041+0000) 2024-03-04T12:06:39.432 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:06:39.436+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:06:39.437280+0000) 2024-03-04T12:06:40.432 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:06:40.436+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:06:40.437515+0000) 2024-03-04T12:06:41.432 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:06:41.437+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:06:41.437756+0000) 2024-03-04T12:06:42.433 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:06:42.437+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:06:42.438000+0000) 2024-03-04T12:06:43.433 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:06:43.437+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:06:43.438228+0000) 2024-03-04T12:06:44.433 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:06:44.437+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:06:44.438495+0000) 2024-03-04T12:06:45.433 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:06:45.438+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:06:45.438729+0000) 2024-03-04T12:06:46.434 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:06:46.438+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:06:46.439001+0000) 2024-03-04T12:06:47.434 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:06:47.438+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:06:47.439253+0000) 2024-03-04T12:06:48.434 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:06:48.439+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:06:48.439502+0000) 2024-03-04T12:06:49.435 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:06:49.439+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:06:49.439751+0000) 2024-03-04T12:06:50.435 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:06:50.439+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:06:50.439997+0000) 2024-03-04T12:06:51.435 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:06:51.439+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:06:51.440249+0000) 2024-03-04T12:06:52.435 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:06:52.440+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:06:52.440581+0000) 2024-03-04T12:06:53.436 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:06:53.440+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:06:53.440882+0000) 2024-03-04T12:06:54.436 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:06:54.440+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:06:54.441121+0000) 2024-03-04T12:06:55.436 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:06:55.441+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:06:55.441366+0000) 2024-03-04T12:06:56.436 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:06:56.440+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:06:56.441623+0000) 2024-03-04T12:06:57.437 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:06:57.441+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:06:57.441904+0000) 2024-03-04T12:06:58.437 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:06:58.441+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:06:58.442113+0000) 2024-03-04T12:06:59.437 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:06:59.441+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:06:59.442316+0000) 2024-03-04T12:07:00.437 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:07:00.442+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:07:00.442559+0000) 2024-03-04T12:07:01.438 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:07:01.442+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:07:01.442813+0000) 2024-03-04T12:07:02.438 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:07:02.442+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:07:02.443072+0000) 2024-03-04T12:07:03.438 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:07:03.443+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:07:03.443341+0000) 2024-03-04T12:07:04.438 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:07:04.442+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:07:04.443606+0000) 2024-03-04T12:07:05.439 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:07:05.443+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:07:05.443886+0000) 2024-03-04T12:07:06.439 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:07:06.443+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:07:06.444129+0000) 2024-03-04T12:07:07.439 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:07:07.444+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:07:07.444371+0000) 2024-03-04T12:07:07.712 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:07:07.721 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:07:08.440 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:07:08.444+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:07:08.444629+0000) 2024-03-04T12:07:09.440 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:07:09.444+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:07:09.444927+0000) 2024-03-04T12:07:10.441 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:07:10.444+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:07:10.445167+0000) 2024-03-04T12:07:11.440 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:07:11.444+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:07:11.445402+0000) 2024-03-04T12:07:12.441 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:07:12.444+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:07:12.445641+0000) 2024-03-04T12:07:13.441 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:07:13.444+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:07:13.445930+0000) 2024-03-04T12:07:14.441 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:07:14.445+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:07:14.446181+0000) 2024-03-04T12:07:15.442 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:07:15.445+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:07:15.446420+0000) 2024-03-04T12:07:16.442 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:07:16.445+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:07:16.446657+0000) 2024-03-04T12:07:17.442 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:07:17.445+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:07:17.446975+0000) 2024-03-04T12:07:18.442 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:07:18.446+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:07:18.447268+0000) 2024-03-04T12:07:19.443 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:07:19.446+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:07:19.447565+0000) 2024-03-04T12:07:20.443 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:07:20.446+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:07:20.447915+0000) 2024-03-04T12:07:21.443 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:07:21.447+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:07:21.448227+0000) 2024-03-04T12:07:22.444 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:07:22.447+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:07:22.448510+0000) 2024-03-04T12:07:23.444 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:07:23.447+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:07:23.448802+0000) 2024-03-04T12:07:24.444 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:07:24.448+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:07:24.449153+0000) 2024-03-04T12:07:25.445 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:07:25.448+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:07:25.449398+0000) 2024-03-04T12:07:26.445 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:07:26.448+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:07:26.449691+0000) 2024-03-04T12:07:27.445 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:07:27.449+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:07:27.449979+0000) 2024-03-04T12:07:28.446 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:07:28.449+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:07:28.450224+0000) 2024-03-04T12:07:29.446 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:07:29.449+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:07:29.450472+0000) 2024-03-04T12:07:30.446 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:07:30.449+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:07:30.450714+0000) 2024-03-04T12:07:31.446 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:07:31.450+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:07:31.450957+0000) 2024-03-04T12:07:32.447 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:07:32.450+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:07:32.451215+0000) 2024-03-04T12:07:33.447 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:07:33.450+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:07:33.451478+0000) 2024-03-04T12:07:34.447 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:07:34.450+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:07:34.451717+0000) 2024-03-04T12:07:35.447 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:07:35.451+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:07:35.451987+0000) 2024-03-04T12:07:36.448 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:07:36.451+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:07:36.452218+0000) 2024-03-04T12:07:37.448 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:07:37.451+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:07:37.452474+0000) 2024-03-04T12:07:37.860 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:07:37.868 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:07:38.448 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:07:38.452+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:07:38.452798+0000) 2024-03-04T12:07:39.448 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:07:39.452+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:07:39.453043+0000) 2024-03-04T12:07:40.449 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:07:40.452+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:07:40.453298+0000) 2024-03-04T12:07:41.449 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:07:41.452+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:07:41.453536+0000) 2024-03-04T12:07:42.449 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:07:42.452+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:07:42.453792+0000) 2024-03-04T12:07:43.450 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:07:43.453+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:07:43.454034+0000) 2024-03-04T12:07:44.450 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:07:44.453+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:07:44.454290+0000) 2024-03-04T12:07:45.450 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:07:45.453+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:07:45.454529+0000) 2024-03-04T12:07:46.450 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:07:46.454+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:07:46.454773+0000) 2024-03-04T12:07:47.451 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:07:47.454+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:07:47.455018+0000) 2024-03-04T12:07:48.451 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:07:48.454+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:07:48.455263+0000) 2024-03-04T12:07:49.451 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:07:49.455+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:07:49.455535+0000) 2024-03-04T12:07:50.452 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:07:50.454+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:07:50.455767+0000) 2024-03-04T12:07:51.452 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:07:51.455+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:07:51.455996+0000) 2024-03-04T12:07:52.452 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:07:52.455+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:07:52.456235+0000) 2024-03-04T12:07:53.452 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:07:53.456+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:07:53.456474+0000) 2024-03-04T12:07:54.452 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:07:54.456+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:07:54.456757+0000) 2024-03-04T12:07:55.453 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:07:55.456+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:07:55.457010+0000) 2024-03-04T12:07:56.453 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:07:56.456+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:07:56.457249+0000) 2024-03-04T12:07:57.453 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:07:57.457+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:07:57.457489+0000) 2024-03-04T12:07:58.453 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:07:58.457+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:07:58.457734+0000) 2024-03-04T12:07:59.455 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:07:59.457+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:07:59.458010+0000) 2024-03-04T12:08:00.454 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:08:00.457+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:08:00.458241+0000) 2024-03-04T12:08:01.454 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:08:01.458+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:08:01.458485+0000) 2024-03-04T12:08:02.455 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:08:02.458+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:08:02.458724+0000) 2024-03-04T12:08:03.455 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:08:03.458+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:08:03.458961+0000) 2024-03-04T12:08:04.455 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:08:04.458+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:08:04.459205+0000) 2024-03-04T12:08:05.455 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:08:05.459+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:08:05.459440+0000) 2024-03-04T12:08:06.456 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:08:06.459+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:08:06.459683+0000) 2024-03-04T12:08:07.456 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:08:07.459+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:08:07.459944+0000) 2024-03-04T12:08:08.000 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:08:08.009 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:08:08.456 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:08:08.460+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:08:08.460194+0000) 2024-03-04T12:08:09.456 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:08:09.459+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:08:09.460376+0000) 2024-03-04T12:08:10.457 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:08:10.460+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:08:10.460564+0000) 2024-03-04T12:08:11.457 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:08:11.459+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:08:11.460801+0000) 2024-03-04T12:08:12.457 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:08:12.459+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:08:12.461077+0000) 2024-03-04T12:08:13.457 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:08:13.460+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:08:13.461318+0000) 2024-03-04T12:08:14.458 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:08:14.460+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:08:14.461556+0000) 2024-03-04T12:08:15.458 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:08:15.460+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:08:15.461791+0000) 2024-03-04T12:08:16.458 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:08:16.461+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:08:16.462072+0000) 2024-03-04T12:08:17.458 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:08:17.460+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:08:17.462272+0000) 2024-03-04T12:08:18.459 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:08:18.461+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:08:18.462519+0000) 2024-03-04T12:08:19.459 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:08:19.460+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:08:19.462752+0000) 2024-03-04T12:08:20.459 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:08:20.462+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:08:20.463012+0000) 2024-03-04T12:08:21.459 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:08:21.462+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:08:21.463274+0000) 2024-03-04T12:08:22.460 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:08:22.462+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:08:22.463510+0000) 2024-03-04T12:08:23.460 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:08:23.462+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:08:23.463785+0000) 2024-03-04T12:08:24.460 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:08:24.463+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:08:24.464031+0000) 2024-03-04T12:08:25.460 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:08:25.463+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:08:25.464265+0000) 2024-03-04T12:08:26.461 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:08:26.463+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:08:26.464565+0000) 2024-03-04T12:08:27.461 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:08:27.464+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:08:27.464817+0000) 2024-03-04T12:08:28.461 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:08:28.464+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:08:28.465101+0000) 2024-03-04T12:08:29.462 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:08:29.464+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:08:29.465343+0000) 2024-03-04T12:08:30.462 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:08:30.464+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:08:30.465582+0000) 2024-03-04T12:08:31.462 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:08:31.465+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:08:31.465818+0000) 2024-03-04T12:08:32.462 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:08:32.465+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:08:32.466104+0000) 2024-03-04T12:08:33.463 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:08:33.465+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:08:33.466380+0000) 2024-03-04T12:08:34.463 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:08:34.465+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:08:34.466629+0000) 2024-03-04T12:08:35.463 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:08:35.466+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:08:35.466927+0000) 2024-03-04T12:08:36.464 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:08:36.466+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:08:36.467169+0000) 2024-03-04T12:08:37.464 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:08:37.466+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:08:37.467439+0000) 2024-03-04T12:08:38.153 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:08:38.161 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:08:38.464 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:08:38.466+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:08:38.467704+0000) 2024-03-04T12:08:39.464 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:08:39.467+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:08:39.467938+0000) 2024-03-04T12:08:40.465 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:08:40.467+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:08:40.468172+0000) 2024-03-04T12:08:41.465 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:08:41.467+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:08:41.468407+0000) 2024-03-04T12:08:42.465 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:08:42.468+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:08:42.468655+0000) 2024-03-04T12:08:43.465 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:08:43.468+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:08:43.468925+0000) 2024-03-04T12:08:44.466 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:08:44.468+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:08:44.469120+0000) 2024-03-04T12:08:45.466 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:08:45.468+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:08:45.469354+0000) 2024-03-04T12:08:46.466 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:08:46.469+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:08:46.469588+0000) 2024-03-04T12:08:47.466 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:08:47.469+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:08:47.469869+0000) 2024-03-04T12:08:48.467 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:08:48.469+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:08:48.470115+0000) 2024-03-04T12:08:49.467 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:08:49.469+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:08:49.470352+0000) 2024-03-04T12:08:50.467 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:08:50.470+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:08:50.470624+0000) 2024-03-04T12:08:51.467 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:08:51.470+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:08:51.470921+0000) 2024-03-04T12:08:52.468 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:08:52.470+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:08:52.471111+0000) 2024-03-04T12:08:53.468 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:08:53.470+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:08:53.471350+0000) 2024-03-04T12:08:54.468 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:08:54.471+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:08:54.471590+0000) 2024-03-04T12:08:55.468 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:08:55.471+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:08:55.471886+0000) 2024-03-04T12:08:56.469 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:08:56.471+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:08:56.472160+0000) 2024-03-04T12:08:57.469 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:08:57.472+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:08:57.472411+0000) 2024-03-04T12:08:58.469 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:08:58.472+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:08:58.472667+0000) 2024-03-04T12:08:59.470 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:08:59.472+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:08:59.472950+0000) 2024-03-04T12:09:00.470 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:09:00.472+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:09:00.473208+0000) 2024-03-04T12:09:01.470 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:09:01.473+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:09:01.473447+0000) 2024-03-04T12:09:02.472 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:09:02.473+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:09:02.473685+0000) 2024-03-04T12:09:03.471 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:09:03.473+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:09:03.473925+0000) 2024-03-04T12:09:04.471 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:09:04.473+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:09:04.474171+0000) 2024-03-04T12:09:05.471 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:09:05.473+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:09:05.474431+0000) 2024-03-04T12:09:06.472 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:09:06.474+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:09:06.474675+0000) 2024-03-04T12:09:07.472 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:09:07.474+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:09:07.474922+0000) 2024-03-04T12:09:08.351 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:09:08.359 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:09:08.478 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:09:08.474+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:09:08.475180+0000) 2024-03-04T12:09:09.472 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:09:09.475+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:09:09.475421+0000) 2024-03-04T12:09:10.473 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:09:10.475+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:09:10.475657+0000) 2024-03-04T12:09:11.473 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:09:11.475+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:09:11.475942+0000) 2024-03-04T12:09:12.473 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:09:12.476+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:09:12.476179+0000) 2024-03-04T12:09:13.473 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:09:13.475+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:09:13.476416+0000) 2024-03-04T12:09:14.474 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:09:14.475+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:09:14.476651+0000) 2024-03-04T12:09:15.474 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:09:15.475+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:09:15.476927+0000) 2024-03-04T12:09:16.474 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:09:16.476+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:09:16.477164+0000) 2024-03-04T12:09:17.475 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:09:17.476+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:09:17.477405+0000) 2024-03-04T12:09:18.475 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:09:18.476+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:09:18.477642+0000) 2024-03-04T12:09:19.475 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:09:19.476+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:09:19.477925+0000) 2024-03-04T12:09:20.475 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:09:20.477+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:09:20.478179+0000) 2024-03-04T12:09:21.475 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:09:21.477+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:09:21.478438+0000) 2024-03-04T12:09:22.476 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:09:22.477+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:09:22.478675+0000) 2024-03-04T12:09:23.476 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:09:23.477+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:09:23.478913+0000) 2024-03-04T12:09:24.476 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:09:24.478+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:09:24.479160+0000) 2024-03-04T12:09:25.477 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:09:25.478+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:09:25.479450+0000) 2024-03-04T12:09:26.477 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:09:26.478+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:09:26.479685+0000) 2024-03-04T12:09:27.477 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:09:27.479+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:09:27.479950+0000) 2024-03-04T12:09:28.477 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:09:28.479+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:09:28.480189+0000) 2024-03-04T12:09:29.478 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:09:29.479+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:09:29.480430+0000) 2024-03-04T12:09:30.478 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:09:30.479+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:09:30.480672+0000) 2024-03-04T12:09:31.478 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:09:31.480+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:09:31.480942+0000) 2024-03-04T12:09:32.479 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:09:32.480+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:09:32.481124+0000) 2024-03-04T12:09:33.479 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:09:33.480+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:09:33.481366+0000) 2024-03-04T12:09:34.479 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:09:34.480+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:09:34.481604+0000) 2024-03-04T12:09:35.479 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:09:35.481+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:09:35.481815+0000) 2024-03-04T12:09:36.479 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:09:36.481+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:09:36.482121+0000) 2024-03-04T12:09:37.480 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:09:37.481+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:09:37.482348+0000) 2024-03-04T12:09:38.480 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:09:38.481+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:09:38.482588+0000) 2024-03-04T12:09:38.548 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:09:38.558 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:09:39.480 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:09:39.482+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:09:39.482856+0000) 2024-03-04T12:09:40.480 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:09:40.482+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:09:40.483057+0000) 2024-03-04T12:09:41.481 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:09:41.482+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:09:41.483298+0000) 2024-03-04T12:09:42.481 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:09:42.482+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:09:42.483557+0000) 2024-03-04T12:09:43.481 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:09:43.483+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:09:43.483818+0000) 2024-03-04T12:09:44.481 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:09:44.483+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:09:44.484131+0000) 2024-03-04T12:09:45.482 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:09:45.483+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:09:45.484322+0000) 2024-03-04T12:09:46.482 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:09:46.483+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:09:46.484553+0000) 2024-03-04T12:09:47.482 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:09:47.484+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:09:47.484812+0000) 2024-03-04T12:09:48.482 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:09:48.484+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:09:48.485093+0000) 2024-03-04T12:09:49.483 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:09:49.484+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:09:49.485334+0000) 2024-03-04T12:09:50.483 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:09:50.485+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:09:50.485594+0000) 2024-03-04T12:09:51.483 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:09:51.485+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:09:51.485857+0000) 2024-03-04T12:09:52.483 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:09:52.485+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:09:52.486103+0000) 2024-03-04T12:09:53.484 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:09:53.485+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:09:53.486304+0000) 2024-03-04T12:09:54.487 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:09:54.486+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:09:54.486574+0000) 2024-03-04T12:09:55.484 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:09:55.486+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:09:55.486815+0000) 2024-03-04T12:09:56.485 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:09:56.486+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:09:56.487094+0000) 2024-03-04T12:09:57.485 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:09:57.486+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:09:57.487347+0000) 2024-03-04T12:09:58.485 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:09:58.487+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:09:58.487591+0000) 2024-03-04T12:09:59.485 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:09:59.487+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:09:59.487863+0000) 2024-03-04T12:10:00.486 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:10:00.487+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:10:00.488298+0000) 2024-03-04T12:10:01.486 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:10:01.488+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:10:01.488547+0000) 2024-03-04T12:10:02.486 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:10:02.488+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:10:02.488792+0000) 2024-03-04T12:10:03.487 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:10:03.488+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:10:03.489071+0000) 2024-03-04T12:10:04.487 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:10:04.488+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:10:04.489275+0000) 2024-03-04T12:10:05.487 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:10:05.489+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:10:05.489543+0000) 2024-03-04T12:10:06.487 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:10:06.488+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:10:06.489795+0000) 2024-03-04T12:10:07.488 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:10:07.489+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:10:07.490039+0000) 2024-03-04T12:10:08.488 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:10:08.490+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:10:08.490361+0000) 2024-03-04T12:10:08.695 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:10:08.703 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:10:09.488 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:10:09.490+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:10:09.490606+0000) 2024-03-04T12:10:10.488 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:10:10.490+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:10:10.490883+0000) 2024-03-04T12:10:11.491 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:10:11.490+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:10:11.491125+0000) 2024-03-04T12:10:12.489 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:10:12.491+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:10:12.491363+0000) 2024-03-04T12:10:13.489 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:10:13.491+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:10:13.491601+0000) 2024-03-04T12:10:14.490 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:10:14.490+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:10:14.491876+0000) 2024-03-04T12:10:15.490 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:10:15.491+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:10:15.492120+0000) 2024-03-04T12:10:16.490 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:10:16.491+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:10:16.492358+0000) 2024-03-04T12:10:17.491 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:10:17.491+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:10:17.492607+0000) 2024-03-04T12:10:18.491 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:10:18.491+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:10:18.492887+0000) 2024-03-04T12:10:19.491 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:10:19.492+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:10:19.493177+0000) 2024-03-04T12:10:20.491 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:10:20.492+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:10:20.493434+0000) 2024-03-04T12:10:21.492 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:10:21.492+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:10:21.493672+0000) 2024-03-04T12:10:22.492 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:10:22.492+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:10:22.493938+0000) 2024-03-04T12:10:23.492 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:10:23.493+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:10:23.494177+0000) 2024-03-04T12:10:24.492 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:10:24.493+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:10:24.494425+0000) 2024-03-04T12:10:25.493 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:10:25.493+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:10:25.494670+0000) 2024-03-04T12:10:26.493 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:10:26.494+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:10:26.494945+0000) 2024-03-04T12:10:27.493 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:10:27.494+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:10:27.495199+0000) 2024-03-04T12:10:28.493 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:10:28.494+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:10:28.495438+0000) 2024-03-04T12:10:29.494 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:10:29.494+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:10:29.495703+0000) 2024-03-04T12:10:30.494 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:10:30.495+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:10:30.495944+0000) 2024-03-04T12:10:31.494 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:10:31.495+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:10:31.496183+0000) 2024-03-04T12:10:32.495 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:10:32.495+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:10:32.496421+0000) 2024-03-04T12:10:33.495 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:10:33.495+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:10:33.496664+0000) 2024-03-04T12:10:34.495 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:10:34.496+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:10:34.496904+0000) 2024-03-04T12:10:35.495 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:10:35.496+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:10:35.497172+0000) 2024-03-04T12:10:36.495 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:10:36.496+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:10:36.497411+0000) 2024-03-04T12:10:37.496 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:10:37.496+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:10:37.497657+0000) 2024-03-04T12:10:38.496 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:10:38.497+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:10:38.497915+0000) 2024-03-04T12:10:38.910 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:10:38.918 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:10:39.496 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:10:39.497+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:10:39.498158+0000) 2024-03-04T12:10:40.497 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:10:40.497+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:10:40.498404+0000) 2024-03-04T12:10:41.497 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:10:41.497+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:10:41.498648+0000) 2024-03-04T12:10:42.497 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:10:42.498+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:10:42.498930+0000) 2024-03-04T12:10:43.497 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:10:43.498+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:10:43.499166+0000) 2024-03-04T12:10:44.498 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:10:44.498+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:10:44.499414+0000) 2024-03-04T12:10:45.498 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:10:45.499+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:10:45.499666+0000) 2024-03-04T12:10:46.498 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:10:46.499+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:10:46.499953+0000) 2024-03-04T12:10:47.498 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:10:47.499+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:10:47.500194+0000) 2024-03-04T12:10:48.499 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:10:48.499+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:10:48.500437+0000) 2024-03-04T12:10:49.499 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:10:49.500+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:10:49.500704+0000) 2024-03-04T12:10:50.499 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:10:50.500+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:10:50.500981+0000) 2024-03-04T12:10:51.499 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:10:51.500+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:10:51.501216+0000) 2024-03-04T12:10:52.500 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:10:52.500+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:10:52.501455+0000) 2024-03-04T12:10:53.500 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:10:53.500+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:10:53.501684+0000) 2024-03-04T12:10:54.500 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:10:54.501+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:10:54.501941+0000) 2024-03-04T12:10:55.500 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:10:55.501+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:10:55.502186+0000) 2024-03-04T12:10:56.501 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:10:56.501+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:10:56.502491+0000) 2024-03-04T12:10:57.501 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:10:57.502+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:10:57.502748+0000) 2024-03-04T12:10:58.501 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:10:58.502+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:10:58.502992+0000) 2024-03-04T12:10:59.502 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:10:59.502+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:10:59.503242+0000) 2024-03-04T12:11:00.502 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:11:00.503+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:11:00.503489+0000) 2024-03-04T12:11:01.502 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:11:01.502+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:11:01.503763+0000) 2024-03-04T12:11:02.502 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:11:02.503+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:11:02.504004+0000) 2024-03-04T12:11:03.502 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:11:03.503+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:11:03.504204+0000) 2024-03-04T12:11:04.503 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:11:04.504+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:11:04.504450+0000) 2024-03-04T12:11:05.503 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:11:05.504+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:11:05.504700+0000) 2024-03-04T12:11:06.503 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:11:06.504+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:11:06.504945+0000) 2024-03-04T12:11:07.504 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:11:07.504+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:11:07.505195+0000) 2024-03-04T12:11:08.504 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:11:08.505+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:11:08.505437+0000) 2024-03-04T12:11:09.039 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:11:09.047 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:11:09.504 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:11:09.505+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:11:09.505684+0000) 2024-03-04T12:11:10.504 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:11:10.505+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:11:10.505933+0000) 2024-03-04T12:11:11.505 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:11:11.505+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:11:11.506178+0000) 2024-03-04T12:11:12.505 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:11:12.506+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:11:12.506416+0000) 2024-03-04T12:11:13.505 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:11:13.506+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:11:13.506657+0000) 2024-03-04T12:11:14.505 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:11:14.506+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:11:14.506933+0000) 2024-03-04T12:11:15.506 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:11:15.506+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:11:15.507182+0000) 2024-03-04T12:11:16.506 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:11:16.505+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:11:16.507377+0000) 2024-03-04T12:11:17.506 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:11:17.505+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:11:17.507578+0000) 2024-03-04T12:11:18.506 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:11:18.506+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:11:18.507848+0000) 2024-03-04T12:11:19.507 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:11:19.507+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:11:19.508095+0000) 2024-03-04T12:11:20.507 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:11:20.507+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:11:20.508365+0000) 2024-03-04T12:11:21.507 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:11:21.506+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:11:21.508553+0000) 2024-03-04T12:11:22.507 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:11:22.507+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:11:22.508797+0000) 2024-03-04T12:11:23.508 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:11:23.508+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:11:23.509078+0000) 2024-03-04T12:11:24.508 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:11:24.508+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:11:24.509332+0000) 2024-03-04T12:11:25.508 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:11:25.508+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:11:25.509585+0000) 2024-03-04T12:11:26.509 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:11:26.509+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:11:26.509818+0000) 2024-03-04T12:11:27.509 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:11:27.509+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:11:27.510102+0000) 2024-03-04T12:11:28.509 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:11:28.509+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:11:28.510377+0000) 2024-03-04T12:11:29.511 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:11:29.509+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:11:29.510627+0000) 2024-03-04T12:11:30.509 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:11:30.510+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:11:30.510911+0000) 2024-03-04T12:11:31.510 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:11:31.510+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:11:31.511097+0000) 2024-03-04T12:11:32.510 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:11:32.510+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:11:32.511343+0000) 2024-03-04T12:11:33.510 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:11:33.510+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:11:33.511603+0000) 2024-03-04T12:11:34.511 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:11:34.511+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:11:34.511889+0000) 2024-03-04T12:11:35.511 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:11:35.511+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:11:35.512143+0000) 2024-03-04T12:11:36.511 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:11:36.511+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:11:36.512403+0000) 2024-03-04T12:11:37.512 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:11:37.511+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:11:37.512687+0000) 2024-03-04T12:11:38.512 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:11:38.512+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:11:38.512977+0000) 2024-03-04T12:11:39.253 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:11:39.265 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:11:39.512 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:11:39.512+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:11:39.513239+0000) 2024-03-04T12:11:40.512 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:11:40.512+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:11:40.513491+0000) 2024-03-04T12:11:41.512 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:11:41.512+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:11:41.513721+0000) 2024-03-04T12:11:42.513 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:11:42.513+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:11:42.514016+0000) 2024-03-04T12:11:43.513 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:11:43.513+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:11:43.514251+0000) 2024-03-04T12:11:44.513 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:11:44.513+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:11:44.514450+0000) 2024-03-04T12:11:45.514 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:11:45.514+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:11:45.514699+0000) 2024-03-04T12:11:46.514 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:11:46.514+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:11:46.514942+0000) 2024-03-04T12:11:47.514 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:11:47.514+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:11:47.515202+0000) 2024-03-04T12:11:48.514 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:11:48.514+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:11:48.515403+0000) 2024-03-04T12:11:49.515 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:11:49.514+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:11:49.515598+0000) 2024-03-04T12:11:50.515 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:11:50.515+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:11:50.515919+0000) 2024-03-04T12:11:51.515 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:11:51.515+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:11:51.516183+0000) 2024-03-04T12:11:52.515 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:11:52.515+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:11:52.516421+0000) 2024-03-04T12:11:53.516 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:11:53.516+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:11:53.516667+0000) 2024-03-04T12:11:54.516 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:11:54.516+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:11:54.516955+0000) 2024-03-04T12:11:55.516 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:11:55.516+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:11:55.517237+0000) 2024-03-04T12:11:56.516 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:11:56.517+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:11:56.517475+0000) 2024-03-04T12:11:57.517 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:11:57.517+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:11:57.517723+0000) 2024-03-04T12:11:58.517 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:11:58.517+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:11:58.517975+0000) 2024-03-04T12:11:59.517 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:11:59.517+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:11:59.518217+0000) 2024-03-04T12:12:00.517 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:12:00.517+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:12:00.518419+0000) 2024-03-04T12:12:01.518 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:12:01.518+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:12:01.518678+0000) 2024-03-04T12:12:02.518 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:12:02.518+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:12:02.518969+0000) 2024-03-04T12:12:03.518 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:12:03.518+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:12:03.519217+0000) 2024-03-04T12:12:04.518 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:12:04.519+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:12:04.519464+0000) 2024-03-04T12:12:05.519 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:12:05.519+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:12:05.519757+0000) 2024-03-04T12:12:06.519 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:12:06.519+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:12:06.519993+0000) 2024-03-04T12:12:07.519 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:12:07.519+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:12:07.520235+0000) 2024-03-04T12:12:08.520 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:12:08.520+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:12:08.520473+0000) 2024-03-04T12:12:09.443 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:12:09.451 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:12:09.520 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:12:09.519+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:12:09.520719+0000) 2024-03-04T12:12:10.520 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:12:10.520+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:12:10.521005+0000) 2024-03-04T12:12:11.520 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:12:11.520+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:12:11.521252+0000) 2024-03-04T12:12:12.521 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:12:12.520+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:12:12.521521+0000) 2024-03-04T12:12:13.521 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:12:13.521+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:12:13.521805+0000) 2024-03-04T12:12:14.521 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:12:14.521+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:12:14.522131+0000) 2024-03-04T12:12:15.521 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:12:15.522+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:12:15.522383+0000) 2024-03-04T12:12:16.522 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:12:16.522+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:12:16.522629+0000) 2024-03-04T12:12:17.522 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:12:17.521+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:12:17.522920+0000) 2024-03-04T12:12:18.522 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:12:18.522+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:12:18.523162+0000) 2024-03-04T12:12:19.523 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:12:19.522+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:12:19.523420+0000) 2024-03-04T12:12:20.523 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:12:20.522+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:12:20.523687+0000) 2024-03-04T12:12:21.523 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:12:21.522+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:12:21.523926+0000) 2024-03-04T12:12:22.523 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:12:22.523+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:12:22.524164+0000) 2024-03-04T12:12:23.524 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:12:23.522+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:12:23.524359+0000) 2024-03-04T12:12:24.524 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:12:24.523+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:12:24.524597+0000) 2024-03-04T12:12:25.524 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:12:25.523+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:12:25.524888+0000) 2024-03-04T12:12:26.524 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:12:26.524+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:12:26.525137+0000) 2024-03-04T12:12:27.525 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:12:27.524+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:12:27.525412+0000) 2024-03-04T12:12:28.525 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:12:28.524+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:12:28.525668+0000) 2024-03-04T12:12:29.525 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:12:29.525+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:12:29.525947+0000) 2024-03-04T12:12:30.525 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:12:30.525+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:12:30.526215+0000) 2024-03-04T12:12:31.526 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:12:31.525+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:12:31.526473+0000) 2024-03-04T12:12:32.526 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:12:32.525+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:12:32.526728+0000) 2024-03-04T12:12:33.526 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:12:33.526+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:12:33.526987+0000) 2024-03-04T12:12:34.527 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:12:34.526+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:12:34.527249+0000) 2024-03-04T12:12:35.527 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:12:35.526+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:12:35.527508+0000) 2024-03-04T12:12:36.527 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:12:36.526+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:12:36.527782+0000) 2024-03-04T12:12:37.527 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:12:37.527+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:12:37.528103+0000) 2024-03-04T12:12:38.528 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:12:38.527+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:12:38.528354+0000) 2024-03-04T12:12:39.528 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:12:39.527+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:12:39.528606+0000) 2024-03-04T12:12:39.581 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:12:39.589 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:12:40.528 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:12:40.528+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:12:40.528904+0000) 2024-03-04T12:12:41.528 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:12:41.528+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:12:41.529165+0000) 2024-03-04T12:12:42.529 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:12:42.528+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:12:42.529371+0000) 2024-03-04T12:12:43.529 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:12:43.528+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:12:43.529621+0000) 2024-03-04T12:12:44.529 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:12:44.529+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:12:44.529927+0000) 2024-03-04T12:12:45.530 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:12:45.529+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:12:45.530186+0000) 2024-03-04T12:12:46.530 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:12:46.529+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:12:46.530441+0000) 2024-03-04T12:12:47.530 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:12:47.530+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:12:47.530708+0000) 2024-03-04T12:12:48.531 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:12:48.530+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:12:48.530992+0000) 2024-03-04T12:12:49.531 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:12:49.530+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:12:49.531266+0000) 2024-03-04T12:12:50.531 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:12:50.530+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:12:50.531523+0000) 2024-03-04T12:12:51.531 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:12:51.530+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:12:51.531774+0000) 2024-03-04T12:12:52.532 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:12:52.531+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:12:52.532036+0000) 2024-03-04T12:12:53.532 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:12:53.531+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:12:53.532300+0000) 2024-03-04T12:12:54.532 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:12:54.532+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:12:54.532552+0000) 2024-03-04T12:12:55.532 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:12:55.532+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:12:55.532818+0000) 2024-03-04T12:12:56.533 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:12:56.532+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:12:56.533132+0000) 2024-03-04T12:12:57.533 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:12:57.532+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:12:57.533424+0000) 2024-03-04T12:12:58.533 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:12:58.533+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:12:58.533745+0000) 2024-03-04T12:12:59.534 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:12:59.533+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:12:59.534022+0000) 2024-03-04T12:13:00.534 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:13:00.533+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:13:00.534262+0000) 2024-03-04T12:13:01.534 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:13:01.534+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:13:01.534504+0000) 2024-03-04T12:13:02.534 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:13:02.534+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:13:02.534737+0000) 2024-03-04T12:13:03.535 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:13:03.534+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:13:03.534994+0000) 2024-03-04T12:13:04.535 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:13:04.534+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:13:04.535238+0000) 2024-03-04T12:13:05.535 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:13:05.535+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:13:05.535491+0000) 2024-03-04T12:13:06.536 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:13:06.535+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:13:06.535748+0000) 2024-03-04T12:13:07.536 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:13:07.535+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:13:07.536015+0000) 2024-03-04T12:13:08.536 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:13:08.535+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:13:08.536275+0000) 2024-03-04T12:13:09.536 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:13:09.536+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:13:09.536531+0000) 2024-03-04T12:13:09.795 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:13:09.803 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:13:10.536 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:13:10.536+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:13:10.536785+0000) 2024-03-04T12:13:11.537 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:13:11.536+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:13:11.537078+0000) 2024-03-04T12:13:12.537 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:13:12.537+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:13:12.537332+0000) 2024-03-04T12:13:13.537 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:13:13.537+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:13:13.537583+0000) 2024-03-04T12:13:14.538 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:13:14.537+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:13:14.537882+0000) 2024-03-04T12:13:15.538 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:13:15.537+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:13:15.538145+0000) 2024-03-04T12:13:16.538 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:13:16.538+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:13:16.538416+0000) 2024-03-04T12:13:17.538 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:13:17.538+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:13:17.538683+0000) 2024-03-04T12:13:18.539 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:13:18.537+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:13:18.538952+0000) 2024-03-04T12:13:19.539 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:13:19.537+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:13:19.539144+0000) 2024-03-04T12:13:20.539 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:13:20.538+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:13:20.539386+0000) 2024-03-04T12:13:21.539 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:13:21.537+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:13:21.539645+0000) 2024-03-04T12:13:22.540 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:13:22.538+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:13:22.539934+0000) 2024-03-04T12:13:23.540 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:13:23.538+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:13:23.540183+0000) 2024-03-04T12:13:24.540 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:13:24.539+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:13:24.540422+0000) 2024-03-04T12:13:25.540 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:13:25.539+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:13:25.540669+0000) 2024-03-04T12:13:26.541 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:13:26.539+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:13:26.540948+0000) 2024-03-04T12:13:27.541 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:13:27.539+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:13:27.541149+0000) 2024-03-04T12:13:28.541 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:13:28.540+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:13:28.541412+0000) 2024-03-04T12:13:29.542 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:13:29.540+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:13:29.541657+0000) 2024-03-04T12:13:30.542 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:13:30.541+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:13:30.541952+0000) 2024-03-04T12:13:31.542 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:13:31.541+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:13:31.542193+0000) 2024-03-04T12:13:32.542 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:13:32.541+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:13:32.542392+0000) 2024-03-04T12:13:33.543 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:13:33.541+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:13:33.542635+0000) 2024-03-04T12:13:34.543 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:13:34.542+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:13:34.542927+0000) 2024-03-04T12:13:35.543 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:13:35.542+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:13:35.543172+0000) 2024-03-04T12:13:36.543 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:13:36.542+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:13:36.543421+0000) 2024-03-04T12:13:37.544 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:13:37.542+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:13:37.543693+0000) 2024-03-04T12:13:38.544 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:13:38.543+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:13:38.543954+0000) 2024-03-04T12:13:39.544 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:13:39.543+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:13:39.544206+0000) 2024-03-04T12:13:40.009 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:13:40.018 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:13:40.545 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:13:40.543+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:13:40.544466+0000) 2024-03-04T12:13:41.545 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:13:41.543+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:13:41.544709+0000) 2024-03-04T12:13:42.545 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:13:42.544+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:13:42.544959+0000) 2024-03-04T12:13:43.545 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:13:43.544+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:13:43.545199+0000) 2024-03-04T12:13:44.546 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:13:44.544+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:13:44.545436+0000) 2024-03-04T12:13:45.546 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:13:45.544+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:13:45.545630+0000) 2024-03-04T12:13:46.546 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:13:46.545+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:13:46.545996+0000) 2024-03-04T12:13:47.546 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:13:47.545+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:13:47.546265+0000) 2024-03-04T12:13:48.547 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:13:48.545+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:13:48.546530+0000) 2024-03-04T12:13:49.547 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:13:49.545+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:13:49.546765+0000) 2024-03-04T12:13:50.547 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:13:50.546+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:13:50.547005+0000) 2024-03-04T12:13:51.547 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:13:51.546+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:13:51.547238+0000) 2024-03-04T12:13:52.548 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:13:52.546+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:13:52.547476+0000) 2024-03-04T12:13:53.548 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:13:53.547+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:13:53.547721+0000) 2024-03-04T12:13:54.548 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:13:54.547+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:13:54.547962+0000) 2024-03-04T12:13:55.548 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:13:55.547+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:13:55.548158+0000) 2024-03-04T12:13:56.549 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:13:56.547+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:13:56.548403+0000) 2024-03-04T12:13:57.549 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:13:57.547+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:13:57.548648+0000) 2024-03-04T12:13:58.549 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:13:58.548+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:13:58.548944+0000) 2024-03-04T12:13:59.549 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:13:59.548+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:13:59.549217+0000) 2024-03-04T12:14:00.550 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:14:00.549+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:14:00.549483+0000) 2024-03-04T12:14:01.550 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:14:01.549+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:14:01.549739+0000) 2024-03-04T12:14:02.550 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:14:02.549+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:14:02.550008+0000) 2024-03-04T12:14:03.551 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:14:03.549+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:14:03.550250+0000) 2024-03-04T12:14:04.551 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:14:04.550+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:14:04.550512+0000) 2024-03-04T12:14:05.551 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:14:05.550+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:14:05.550770+0000) 2024-03-04T12:14:06.551 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:14:06.550+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:14:06.551027+0000) 2024-03-04T12:14:07.552 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:14:07.550+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:14:07.551294+0000) 2024-03-04T12:14:08.552 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:14:08.551+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:14:08.551555+0000) 2024-03-04T12:14:09.552 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:14:09.551+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:14:09.551866+0000) 2024-03-04T12:14:10.231 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:14:10.239 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:14:10.552 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:14:10.551+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:14:10.552135+0000) 2024-03-04T12:14:11.553 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:14:11.551+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:14:11.552346+0000) 2024-03-04T12:14:12.553 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:14:12.552+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:14:12.552597+0000) 2024-03-04T12:14:13.553 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:14:13.552+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:14:13.552893+0000) 2024-03-04T12:14:14.553 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:14:14.552+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:14:14.553145+0000) 2024-03-04T12:14:15.554 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:14:15.553+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:14:15.553402+0000) 2024-03-04T12:14:16.554 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:14:16.553+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:14:16.553676+0000) 2024-03-04T12:14:17.555 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:14:17.553+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:14:17.553972+0000) 2024-03-04T12:14:18.555 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:14:18.554+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:14:18.554233+0000) 2024-03-04T12:14:19.555 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:14:19.552+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:14:19.554507+0000) 2024-03-04T12:14:20.555 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:14:20.552+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:14:20.554783+0000) 2024-03-04T12:14:21.555 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:14:21.553+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:14:21.555022+0000) 2024-03-04T12:14:22.556 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:14:22.553+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:14:22.555228+0000) 2024-03-04T12:14:23.556 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:14:23.554+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:14:23.555482+0000) 2024-03-04T12:14:24.556 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:14:24.554+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:14:24.555749+0000) 2024-03-04T12:14:25.556 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:14:25.554+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:14:25.556009+0000) 2024-03-04T12:14:26.557 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:14:26.555+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:14:26.556272+0000) 2024-03-04T12:14:27.557 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:14:27.555+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:14:27.556539+0000) 2024-03-04T12:14:28.557 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:14:28.555+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:14:28.556792+0000) 2024-03-04T12:14:29.557 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:14:29.556+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:14:29.557079+0000) 2024-03-04T12:14:30.558 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:14:30.556+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:14:30.557288+0000) 2024-03-04T12:14:31.558 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:14:31.556+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:14:31.557545+0000) 2024-03-04T12:14:32.558 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:14:32.556+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:14:32.557800+0000) 2024-03-04T12:14:33.558 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:14:33.557+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:14:33.558103+0000) 2024-03-04T12:14:34.559 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:14:34.557+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:14:34.558355+0000) 2024-03-04T12:14:35.559 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:14:35.557+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:14:35.558616+0000) 2024-03-04T12:14:36.559 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:14:36.558+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:14:36.558924+0000) 2024-03-04T12:14:37.561 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:14:37.558+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:14:37.559189+0000) 2024-03-04T12:14:38.560 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:14:38.558+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:14:38.559443+0000) 2024-03-04T12:14:39.560 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:14:39.558+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:14:39.559695+0000) 2024-03-04T12:14:40.427 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:14:40.439 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:14:40.573 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:14:40.559+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:14:40.559982+0000) 2024-03-04T12:14:41.561 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:14:41.559+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:14:41.560241+0000) 2024-03-04T12:14:42.561 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:14:42.559+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:14:42.560495+0000) 2024-03-04T12:14:43.561 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:14:43.559+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:14:43.560750+0000) 2024-03-04T12:14:44.562 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:14:44.560+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:14:44.561018+0000) 2024-03-04T12:14:45.562 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:14:45.560+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:14:45.561323+0000) 2024-03-04T12:14:46.562 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:14:46.560+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:14:46.561578+0000) 2024-03-04T12:14:47.563 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:14:47.560+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:14:47.561801+0000) 2024-03-04T12:14:48.563 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:14:48.561+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:14:48.562108+0000) 2024-03-04T12:14:49.563 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:14:49.561+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:14:49.562362+0000) 2024-03-04T12:14:50.563 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:14:50.562+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:14:50.562629+0000) 2024-03-04T12:14:51.564 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:14:51.562+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:14:51.562928+0000) 2024-03-04T12:14:52.564 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:14:52.562+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:14:52.563178+0000) 2024-03-04T12:14:53.564 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:14:53.562+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:14:53.563389+0000) 2024-03-04T12:14:54.564 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:14:54.562+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:14:54.563658+0000) 2024-03-04T12:14:55.565 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:14:55.563+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:14:55.563955+0000) 2024-03-04T12:14:56.565 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:14:56.563+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:14:56.564210+0000) 2024-03-04T12:14:57.565 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:14:57.563+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:14:57.564459+0000) 2024-03-04T12:14:58.565 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:14:58.563+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:14:58.564656+0000) 2024-03-04T12:14:59.566 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:14:59.564+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:14:59.564934+0000) 2024-03-04T12:15:00.566 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:15:00.564+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:15:00.565193+0000) 2024-03-04T12:15:01.566 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:15:01.565+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:15:01.565436+0000) 2024-03-04T12:15:02.567 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:15:02.565+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:15:02.565674+0000) 2024-03-04T12:15:03.567 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:15:03.565+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:15:03.565957+0000) 2024-03-04T12:15:04.567 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:15:04.565+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:15:04.566211+0000) 2024-03-04T12:15:05.569 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:15:05.565+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:15:05.566409+0000) 2024-03-04T12:15:06.567 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:15:06.566+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:15:06.566660+0000) 2024-03-04T12:15:07.568 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:15:07.566+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:15:07.566950+0000) 2024-03-04T12:15:08.568 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:15:08.566+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:15:08.567192+0000) 2024-03-04T12:15:09.568 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:15:09.567+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:15:09.567448+0000) 2024-03-04T12:15:10.569 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:15:10.567+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:15:10.567715+0000) 2024-03-04T12:15:10.648 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:15:10.657 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:15:11.569 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:15:11.567+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:15:11.567990+0000) 2024-03-04T12:15:12.569 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:15:12.568+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:15:12.568245+0000) 2024-03-04T12:15:13.569 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:15:13.568+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:15:13.568502+0000) 2024-03-04T12:15:14.570 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:15:14.567+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:15:14.568774+0000) 2024-03-04T12:15:15.570 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:15:15.568+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:15:15.569065+0000) 2024-03-04T12:15:16.570 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:15:16.569+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:15:16.569329+0000) 2024-03-04T12:15:17.571 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:15:17.569+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:15:17.569598+0000) 2024-03-04T12:15:18.571 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:15:18.569+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:15:18.569893+0000) 2024-03-04T12:15:19.571 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:15:19.570+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:15:19.570146+0000) 2024-03-04T12:15:20.571 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:15:20.569+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:15:20.570408+0000) 2024-03-04T12:15:21.572 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:15:21.569+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:15:21.570666+0000) 2024-03-04T12:15:22.572 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:15:22.569+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:15:22.570949+0000) 2024-03-04T12:15:23.572 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:15:23.570+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:15:23.571203+0000) 2024-03-04T12:15:24.572 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:15:24.570+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:15:24.571490+0000) 2024-03-04T12:15:25.573 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:15:25.570+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:15:25.571777+0000) 2024-03-04T12:15:26.573 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:15:26.571+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:15:26.572036+0000) 2024-03-04T12:15:27.573 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:15:27.571+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:15:27.572301+0000) 2024-03-04T12:15:28.574 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:15:28.571+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:15:28.572621+0000) 2024-03-04T12:15:29.574 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:15:29.571+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:15:29.572913+0000) 2024-03-04T12:15:30.574 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:15:30.572+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:15:30.573175+0000) 2024-03-04T12:15:31.574 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:15:31.572+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:15:31.573428+0000) 2024-03-04T12:15:32.575 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:15:32.572+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:15:32.573701+0000) 2024-03-04T12:15:33.575 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:15:33.573+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:15:33.573956+0000) 2024-03-04T12:15:34.575 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:15:34.573+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:15:34.574210+0000) 2024-03-04T12:15:35.575 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:15:35.573+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:15:35.574479+0000) 2024-03-04T12:15:36.576 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:15:36.573+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:15:36.574769+0000) 2024-03-04T12:15:37.576 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:15:37.574+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:15:37.575026+0000) 2024-03-04T12:15:38.576 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:15:38.574+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:15:38.575273+0000) 2024-03-04T12:15:39.577 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:15:39.574+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:15:39.575514+0000) 2024-03-04T12:15:40.577 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:15:40.574+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:15:40.575760+0000) 2024-03-04T12:15:40.804 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:15:40.811 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:15:41.577 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:15:41.575+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:15:41.576008+0000) 2024-03-04T12:15:42.577 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:15:42.575+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:15:42.576277+0000) 2024-03-04T12:15:43.578 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:15:43.575+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:15:43.576536+0000) 2024-03-04T12:15:44.578 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:15:44.575+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:15:44.576773+0000) 2024-03-04T12:15:45.578 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:15:45.576+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:15:45.577020+0000) 2024-03-04T12:15:46.578 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:15:46.576+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:15:46.577263+0000) 2024-03-04T12:15:47.579 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:15:47.576+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:15:47.577510+0000) 2024-03-04T12:15:48.579 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:15:48.577+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:15:48.577778+0000) 2024-03-04T12:15:49.579 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:15:49.577+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:15:49.577987+0000) 2024-03-04T12:15:50.579 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:15:50.577+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:15:50.578249+0000) 2024-03-04T12:15:51.580 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:15:51.577+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:15:51.578508+0000) 2024-03-04T12:15:52.580 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:15:52.578+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:15:52.578762+0000) 2024-03-04T12:15:53.580 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:15:53.578+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:15:53.579055+0000) 2024-03-04T12:15:54.581 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:15:54.578+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:15:54.579322+0000) 2024-03-04T12:15:55.581 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:15:55.579+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:15:55.579586+0000) 2024-03-04T12:15:56.581 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:15:56.579+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:15:56.579863+0000) 2024-03-04T12:15:57.581 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:15:57.579+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:15:57.580123+0000) 2024-03-04T12:15:58.582 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:15:58.579+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:15:58.580365+0000) 2024-03-04T12:15:59.582 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:15:59.579+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:15:59.580634+0000) 2024-03-04T12:16:00.582 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:16:00.580+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:16:00.580950+0000) 2024-03-04T12:16:01.582 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:16:01.580+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:16:01.581166+0000) 2024-03-04T12:16:02.583 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:16:02.580+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:16:02.581364+0000) 2024-03-04T12:16:03.583 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:16:03.580+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:16:03.581614+0000) 2024-03-04T12:16:04.583 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:16:04.581+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:16:04.581962+0000) 2024-03-04T12:16:05.583 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:16:05.581+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:16:05.582223+0000) 2024-03-04T12:16:06.584 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:16:06.582+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:16:06.582465+0000) 2024-03-04T12:16:07.584 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:16:07.582+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:16:07.582716+0000) 2024-03-04T12:16:08.584 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:16:08.582+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:16:08.582954+0000) 2024-03-04T12:16:09.584 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:16:09.582+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:16:09.583196+0000) 2024-03-04T12:16:10.585 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:16:10.583+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:16:10.583454+0000) 2024-03-04T12:16:11.038 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:16:11.050 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:16:11.585 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:16:11.583+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:16:11.583703+0000) 2024-03-04T12:16:12.585 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:16:12.583+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:16:12.583966+0000) 2024-03-04T12:16:13.586 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:16:13.583+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:16:13.584226+0000) 2024-03-04T12:16:14.586 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:16:14.584+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:16:14.584463+0000) 2024-03-04T12:16:15.586 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:16:15.584+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:16:15.584714+0000) 2024-03-04T12:16:16.586 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:16:16.584+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:16:16.585013+0000) 2024-03-04T12:16:17.587 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:16:17.585+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:16:17.585265+0000) 2024-03-04T12:16:18.587 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:16:18.585+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:16:18.585457+0000) 2024-03-04T12:16:19.587 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:16:19.585+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:16:19.585693+0000) 2024-03-04T12:16:20.587 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:16:20.585+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:16:20.585939+0000) 2024-03-04T12:16:21.587 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:16:21.585+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:16:21.586189+0000) 2024-03-04T12:16:22.590 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:16:22.585+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:16:22.586427+0000) 2024-03-04T12:16:23.588 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:16:23.585+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:16:23.586666+0000) 2024-03-04T12:16:24.588 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:16:24.585+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:16:24.586928+0000) 2024-03-04T12:16:25.589 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:16:25.586+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:16:25.587175+0000) 2024-03-04T12:16:26.589 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:16:26.586+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:16:26.587448+0000) 2024-03-04T12:16:27.589 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:16:27.586+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:16:27.587715+0000) 2024-03-04T12:16:28.590 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:16:28.586+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:16:28.587989+0000) 2024-03-04T12:16:29.590 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:16:29.587+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:16:29.588226+0000) 2024-03-04T12:16:30.590 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:16:30.587+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:16:30.588476+0000) 2024-03-04T12:16:31.590 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:16:31.587+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:16:31.588729+0000) 2024-03-04T12:16:32.590 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:16:32.588+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:16:32.588973+0000) 2024-03-04T12:16:33.591 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:16:33.588+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:16:33.589207+0000) 2024-03-04T12:16:34.591 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:16:34.588+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:16:34.589480+0000) 2024-03-04T12:16:35.591 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:16:35.589+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:16:35.589811+0000) 2024-03-04T12:16:36.592 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:16:36.589+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:16:36.590152+0000) 2024-03-04T12:16:37.592 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:16:37.589+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:16:37.590460+0000) 2024-03-04T12:16:38.592 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:16:38.589+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:16:38.590768+0000) 2024-03-04T12:16:39.592 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:16:39.590+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:16:39.591071+0000) 2024-03-04T12:16:40.593 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:16:40.590+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:16:40.591376+0000) 2024-03-04T12:16:41.248 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:16:41.256 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:16:41.593 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:16:41.590+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:16:41.591681+0000) 2024-03-04T12:16:42.593 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:16:42.591+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:16:42.591981+0000) 2024-03-04T12:16:43.594 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:16:43.591+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:16:43.592274+0000) 2024-03-04T12:16:44.594 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:16:44.591+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:16:44.592565+0000) 2024-03-04T12:16:45.595 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:16:45.592+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:16:45.592891+0000) 2024-03-04T12:16:46.595 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:16:46.592+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:16:46.593141+0000) 2024-03-04T12:16:47.595 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:16:47.592+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:16:47.593405+0000) 2024-03-04T12:16:48.595 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:16:48.593+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:16:48.593672+0000) 2024-03-04T12:16:49.595 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:16:49.593+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:16:49.593930+0000) 2024-03-04T12:16:50.596 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:16:50.593+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:16:50.594233+0000) 2024-03-04T12:16:51.596 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:16:51.593+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:16:51.594481+0000) 2024-03-04T12:16:52.597 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:16:52.594+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:16:52.594773+0000) 2024-03-04T12:16:53.597 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:16:53.594+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:16:53.595073+0000) 2024-03-04T12:16:54.597 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:16:54.594+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:16:54.595372+0000) 2024-03-04T12:16:55.597 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:16:55.594+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:16:55.595654+0000) 2024-03-04T12:16:56.598 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:16:56.595+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:16:56.595987+0000) 2024-03-04T12:16:57.598 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:16:57.595+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:16:57.596308+0000) 2024-03-04T12:16:58.598 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:16:58.596+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:16:58.596577+0000) 2024-03-04T12:16:59.599 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:16:59.596+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:16:59.596871+0000) 2024-03-04T12:17:00.599 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:17:00.596+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:17:00.597165+0000) 2024-03-04T12:17:01.599 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:17:01.596+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:17:01.597466+0000) 2024-03-04T12:17:02.600 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:17:02.597+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:17:02.597764+0000) 2024-03-04T12:17:03.600 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:17:03.597+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:17:03.598075+0000) 2024-03-04T12:17:04.600 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:17:04.597+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:17:04.598375+0000) 2024-03-04T12:17:05.600 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:17:05.597+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:17:05.598669+0000) 2024-03-04T12:17:06.601 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:17:06.598+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:17:06.598982+0000) 2024-03-04T12:17:07.601 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:17:07.598+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:17:07.599241+0000) 2024-03-04T12:17:08.601 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:17:08.599+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:17:08.599540+0000) 2024-03-04T12:17:09.602 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:17:09.599+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:17:09.599867+0000) 2024-03-04T12:17:10.602 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:17:10.599+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:17:10.600161+0000) 2024-03-04T12:17:11.471 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:17:11.479 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:17:11.602 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:17:11.600+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:17:11.600464+0000) 2024-03-04T12:17:12.603 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:17:12.600+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:17:12.600757+0000) 2024-03-04T12:17:13.603 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:17:13.600+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:17:13.601057+0000) 2024-03-04T12:17:14.604 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:17:14.600+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:17:14.601306+0000) 2024-03-04T12:17:15.604 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:17:15.601+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:17:15.601636+0000) 2024-03-04T12:17:16.604 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:17:16.601+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:17:16.601984+0000) 2024-03-04T12:17:17.604 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:17:17.602+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:17:17.602265+0000) 2024-03-04T12:17:18.604 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:17:18.601+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:17:18.602498+0000) 2024-03-04T12:17:19.605 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:17:19.602+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:17:19.602791+0000) 2024-03-04T12:17:20.605 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:17:20.602+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:17:20.603160+0000) 2024-03-04T12:17:21.605 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:17:21.602+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:17:21.603446+0000) 2024-03-04T12:17:22.606 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:17:22.602+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:17:22.603745+0000) 2024-03-04T12:17:23.606 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:17:23.602+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:17:23.604041+0000) 2024-03-04T12:17:24.606 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:17:24.603+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:17:24.604357+0000) 2024-03-04T12:17:25.607 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:17:25.603+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:17:25.604639+0000) 2024-03-04T12:17:26.608 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:17:26.603+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:17:26.604919+0000) 2024-03-04T12:17:27.607 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:17:27.604+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:17:27.605186+0000) 2024-03-04T12:17:28.607 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:17:28.604+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:17:28.605446+0000) 2024-03-04T12:17:29.608 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:17:29.604+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:17:29.605683+0000) 2024-03-04T12:17:30.608 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:17:30.604+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:17:30.605956+0000) 2024-03-04T12:17:31.608 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:17:31.604+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:17:31.606214+0000) 2024-03-04T12:17:32.608 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:17:32.605+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:17:32.606491+0000) 2024-03-04T12:17:33.609 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:17:33.605+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:17:33.606754+0000) 2024-03-04T12:17:34.609 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:17:34.606+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:17:34.607027+0000) 2024-03-04T12:17:35.610 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:17:35.606+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:17:35.607253+0000) 2024-03-04T12:17:36.609 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:17:36.606+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:17:36.607495+0000) 2024-03-04T12:17:37.610 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:17:37.606+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:17:37.607753+0000) 2024-03-04T12:17:38.610 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:17:38.607+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:17:38.608010+0000) 2024-03-04T12:17:39.610 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:17:39.607+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:17:39.608252+0000) 2024-03-04T12:17:40.610 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:17:40.607+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:17:40.608494+0000) 2024-03-04T12:17:41.611 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:17:41.607+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:17:41.608726+0000) 2024-03-04T12:17:41.668 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:17:41.676 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:17:42.611 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:17:42.608+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:17:42.608998+0000) 2024-03-04T12:17:43.611 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:17:43.608+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:17:43.609240+0000) 2024-03-04T12:17:44.611 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:17:44.608+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:17:44.609474+0000) 2024-03-04T12:17:45.612 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:17:45.608+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:17:45.609713+0000) 2024-03-04T12:17:46.612 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:17:46.609+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:17:46.609952+0000) 2024-03-04T12:17:47.612 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:17:47.609+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:17:47.610149+0000) 2024-03-04T12:17:48.613 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:17:48.609+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:17:48.610394+0000) 2024-03-04T12:17:49.613 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:17:49.609+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:17:49.610662+0000) 2024-03-04T12:17:50.613 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:17:50.610+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:17:50.610939+0000) 2024-03-04T12:17:51.613 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:17:51.610+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:17:51.611168+0000) 2024-03-04T12:17:52.614 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:17:52.610+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:17:52.611416+0000) 2024-03-04T12:17:53.614 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:17:53.611+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:17:53.611661+0000) 2024-03-04T12:17:54.614 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:17:54.611+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:17:54.611931+0000) 2024-03-04T12:17:55.614 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:17:55.611+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:17:55.612167+0000) 2024-03-04T12:17:56.614 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:17:56.611+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:17:56.612403+0000) 2024-03-04T12:17:57.615 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:17:57.612+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:17:57.612656+0000) 2024-03-04T12:17:58.615 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:17:58.612+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:17:58.612945+0000) 2024-03-04T12:17:59.615 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:17:59.612+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:17:59.613181+0000) 2024-03-04T12:18:00.616 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:18:00.612+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:18:00.613427+0000) 2024-03-04T12:18:01.616 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:18:01.612+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:18:01.613707+0000) 2024-03-04T12:18:02.616 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:18:02.613+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:18:02.614027+0000) 2024-03-04T12:18:03.616 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:18:03.613+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:18:03.614336+0000) 2024-03-04T12:18:04.617 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:18:04.614+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:18:04.614627+0000) 2024-03-04T12:18:05.617 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:18:05.614+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:18:05.614931+0000) 2024-03-04T12:18:06.617 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:18:06.614+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:18:06.615161+0000) 2024-03-04T12:18:07.617 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:18:07.614+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:18:07.615396+0000) 2024-03-04T12:18:08.618 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:18:08.614+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:18:08.615716+0000) 2024-03-04T12:18:09.618 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:18:09.615+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:18:09.615949+0000) 2024-03-04T12:18:10.618 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:18:10.615+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:18:10.616187+0000) 2024-03-04T12:18:11.619 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:18:11.615+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:18:11.616450+0000) 2024-03-04T12:18:11.814 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:18:11.822 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:18:12.619 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:18:12.616+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:18:12.616686+0000) 2024-03-04T12:18:13.619 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:18:13.616+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:18:13.616925+0000) 2024-03-04T12:18:14.619 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:18:14.616+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:18:14.617160+0000) 2024-03-04T12:18:15.620 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:18:15.617+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:18:15.617396+0000) 2024-03-04T12:18:16.620 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:18:16.616+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:18:16.617631+0000) 2024-03-04T12:18:17.620 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:18:17.617+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:18:17.617921+0000) 2024-03-04T12:18:18.620 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:18:18.617+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:18:18.618160+0000) 2024-03-04T12:18:19.621 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:18:19.618+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:18:19.618401+0000) 2024-03-04T12:18:20.621 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:18:20.618+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:18:20.618661+0000) 2024-03-04T12:18:21.621 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:18:21.618+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:18:21.618940+0000) 2024-03-04T12:18:22.621 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:18:22.618+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:18:22.619178+0000) 2024-03-04T12:18:23.622 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:18:23.618+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:18:23.619410+0000) 2024-03-04T12:18:24.622 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:18:24.617+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:18:24.619666+0000) 2024-03-04T12:18:25.622 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:18:25.618+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:18:25.619938+0000) 2024-03-04T12:18:26.622 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:18:26.618+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:18:26.620214+0000) 2024-03-04T12:18:27.623 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:18:27.619+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:18:27.620502+0000) 2024-03-04T12:18:28.623 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:18:28.618+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:18:28.620740+0000) 2024-03-04T12:18:29.623 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:18:29.619+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:18:29.620982+0000) 2024-03-04T12:18:30.624 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:18:30.620+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:18:30.621220+0000) 2024-03-04T12:18:31.624 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:18:31.620+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:18:31.621411+0000) 2024-03-04T12:18:32.624 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:18:32.620+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:18:32.621669+0000) 2024-03-04T12:18:33.624 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:18:33.621+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:18:33.621946+0000) 2024-03-04T12:18:34.624 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:18:34.621+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:18:34.622182+0000) 2024-03-04T12:18:35.625 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:18:35.621+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:18:35.622410+0000) 2024-03-04T12:18:36.625 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:18:36.621+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:18:36.622649+0000) 2024-03-04T12:18:37.625 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:18:37.622+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:18:37.622962+0000) 2024-03-04T12:18:38.625 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:18:38.622+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:18:38.623194+0000) 2024-03-04T12:18:39.626 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:18:39.622+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:18:39.623433+0000) 2024-03-04T12:18:40.626 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:18:40.622+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:18:40.623672+0000) 2024-03-04T12:18:41.626 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:18:41.623+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:18:41.623952+0000) 2024-03-04T12:18:42.027 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:18:42.035 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:18:42.627 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:18:42.623+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:18:42.624152+0000) 2024-03-04T12:18:43.627 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:18:43.623+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:18:43.624349+0000) 2024-03-04T12:18:44.627 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:18:44.623+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:18:44.624591+0000) 2024-03-04T12:18:45.627 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:18:45.624+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:18:45.624854+0000) 2024-03-04T12:18:46.627 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:18:46.624+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:18:46.625119+0000) 2024-03-04T12:18:47.628 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:18:47.624+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:18:47.625318+0000) 2024-03-04T12:18:48.628 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:18:48.624+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:18:48.625552+0000) 2024-03-04T12:18:49.628 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:18:49.624+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:18:49.625807+0000) 2024-03-04T12:18:50.628 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:18:50.625+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:18:50.626111+0000) 2024-03-04T12:18:51.629 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:18:51.625+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:18:51.626348+0000) 2024-03-04T12:18:52.629 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:18:52.625+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:18:52.626587+0000) 2024-03-04T12:18:53.631 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:18:53.626+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:18:53.626817+0000) 2024-03-04T12:18:54.630 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:18:54.626+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:18:54.627118+0000) 2024-03-04T12:18:55.630 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:18:55.626+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:18:55.627352+0000) 2024-03-04T12:18:56.630 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:18:56.627+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:18:56.627617+0000) 2024-03-04T12:18:57.630 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:18:57.627+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:18:57.627853+0000) 2024-03-04T12:18:58.631 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:18:58.627+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:18:58.628117+0000) 2024-03-04T12:18:59.631 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:18:59.627+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:18:59.628363+0000) 2024-03-04T12:19:00.631 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:19:00.628+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:19:00.628601+0000) 2024-03-04T12:19:01.631 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:19:01.628+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:19:01.628876+0000) 2024-03-04T12:19:02.632 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:19:02.628+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:19:02.629137+0000) 2024-03-04T12:19:03.632 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:19:03.628+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:19:03.629372+0000) 2024-03-04T12:19:04.632 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:19:04.629+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:19:04.629614+0000) 2024-03-04T12:19:05.633 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:19:05.629+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:19:05.629889+0000) 2024-03-04T12:19:06.633 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:19:06.629+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:19:06.630188+0000) 2024-03-04T12:19:07.633 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:19:07.630+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:19:07.630445+0000) 2024-03-04T12:19:08.633 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:19:08.630+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:19:08.630682+0000) 2024-03-04T12:19:09.634 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:19:09.630+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:19:09.630924+0000) 2024-03-04T12:19:10.635 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:19:10.630+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:19:10.631165+0000) 2024-03-04T12:19:11.634 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:19:11.631+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:19:11.631401+0000) 2024-03-04T12:19:12.250 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:19:12.258 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:19:12.634 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:19:12.630+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:19:12.631662+0000) 2024-03-04T12:19:13.635 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:19:13.631+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:19:13.631936+0000) 2024-03-04T12:19:14.635 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:19:14.631+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:19:14.632172+0000) 2024-03-04T12:19:15.635 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:19:15.632+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:19:15.632415+0000) 2024-03-04T12:19:16.635 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:19:16.632+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:19:16.632655+0000) 2024-03-04T12:19:17.636 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:19:17.632+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:19:17.632945+0000) 2024-03-04T12:19:18.636 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:19:18.632+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:19:18.633193+0000) 2024-03-04T12:19:19.636 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:19:19.633+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:19:19.633437+0000) 2024-03-04T12:19:20.636 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:19:20.633+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:19:20.633675+0000) 2024-03-04T12:19:21.637 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:19:21.633+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:19:21.633947+0000) 2024-03-04T12:19:22.637 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:19:22.633+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:19:22.634142+0000) 2024-03-04T12:19:23.637 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:19:23.633+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:19:23.634374+0000) 2024-03-04T12:19:24.637 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:19:24.633+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:19:24.634614+0000) 2024-03-04T12:19:25.637 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:19:25.633+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:19:25.634885+0000) 2024-03-04T12:19:26.638 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:19:26.633+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:19:26.635117+0000) 2024-03-04T12:19:27.638 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:19:27.634+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:19:27.635410+0000) 2024-03-04T12:19:28.638 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:19:28.634+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:19:28.635702+0000) 2024-03-04T12:19:29.639 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:19:29.634+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:19:29.635999+0000) 2024-03-04T12:19:30.639 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:19:30.635+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:19:30.636296+0000) 2024-03-04T12:19:31.639 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:19:31.635+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:19:31.636584+0000) 2024-03-04T12:19:32.640 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:19:32.635+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:19:32.636927+0000) 2024-03-04T12:19:33.640 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:19:33.636+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:19:33.637216+0000) 2024-03-04T12:19:34.640 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:19:34.636+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:19:34.637513+0000) 2024-03-04T12:19:35.641 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:19:35.636+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:19:35.637806+0000) 2024-03-04T12:19:36.641 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:19:36.637+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:19:36.638129+0000) 2024-03-04T12:19:37.641 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:19:37.637+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:19:37.638342+0000) 2024-03-04T12:19:38.641 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:19:38.637+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:19:38.638600+0000) 2024-03-04T12:19:39.642 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:19:39.638+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:19:39.638867+0000) 2024-03-04T12:19:40.642 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:19:40.638+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:19:40.639110+0000) 2024-03-04T12:19:41.642 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:19:41.638+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:19:41.639352+0000) 2024-03-04T12:19:42.405 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:19:42.414 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:19:42.642 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:19:42.638+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:19:42.639623+0000) 2024-03-04T12:19:43.643 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:19:43.639+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:19:43.639902+0000) 2024-03-04T12:19:44.643 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:19:44.639+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:19:44.640147+0000) 2024-03-04T12:19:45.643 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:19:45.639+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:19:45.640384+0000) 2024-03-04T12:19:46.643 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:19:46.639+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:19:46.640626+0000) 2024-03-04T12:19:47.644 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:19:47.640+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:19:47.640926+0000) 2024-03-04T12:19:48.644 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:19:48.640+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:19:48.641166+0000) 2024-03-04T12:19:49.644 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:19:49.640+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:19:49.641427+0000) 2024-03-04T12:19:50.645 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:19:50.640+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:19:50.641664+0000) 2024-03-04T12:19:51.645 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:19:51.641+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:19:51.641947+0000) 2024-03-04T12:19:52.645 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:19:52.641+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:19:52.642188+0000) 2024-03-04T12:19:53.645 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:19:53.641+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:19:53.642422+0000) 2024-03-04T12:19:54.646 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:19:54.642+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:19:54.642726+0000) 2024-03-04T12:19:55.646 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:19:55.642+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:19:55.642971+0000) 2024-03-04T12:19:56.646 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:19:56.642+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:19:56.643170+0000) 2024-03-04T12:19:57.646 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:19:57.642+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:19:57.643429+0000) 2024-03-04T12:19:58.647 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:19:58.643+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:19:58.643675+0000) 2024-03-04T12:19:59.647 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:19:59.643+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:19:59.643936+0000) 2024-03-04T12:20:00.647 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:20:00.643+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:20:00.644184+0000) 2024-03-04T12:20:01.648 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:20:01.643+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:20:01.644371+0000) 2024-03-04T12:20:02.648 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:20:02.644+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:20:02.644632+0000) 2024-03-04T12:20:03.648 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:20:03.644+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:20:03.644907+0000) 2024-03-04T12:20:04.648 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:20:04.644+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:20:04.645148+0000) 2024-03-04T12:20:05.648 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:20:05.644+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:20:05.645394+0000) 2024-03-04T12:20:06.648 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:20:06.644+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:20:06.645582+0000) 2024-03-04T12:20:07.649 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:20:07.645+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:20:07.645865+0000) 2024-03-04T12:20:08.649 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:20:08.645+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:20:08.646050+0000) 2024-03-04T12:20:09.650 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:20:09.645+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:20:09.646292+0000) 2024-03-04T12:20:10.649 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:20:10.646+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:20:10.646531+0000) 2024-03-04T12:20:11.650 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:20:11.645+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:20:11.646791+0000) 2024-03-04T12:20:12.617 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:20:12.625 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:20:12.650 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:20:12.646+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:20:12.646984+0000) 2024-03-04T12:20:13.650 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:20:13.646+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:20:13.647223+0000) 2024-03-04T12:20:14.651 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:20:14.647+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:20:14.647462+0000) 2024-03-04T12:20:15.651 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:20:15.647+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:20:15.647691+0000) 2024-03-04T12:20:16.651 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:20:16.647+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:20:16.647936+0000) 2024-03-04T12:20:17.651 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:20:17.647+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:20:17.648189+0000) 2024-03-04T12:20:18.651 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:20:18.648+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:20:18.648434+0000) 2024-03-04T12:20:19.652 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:20:19.647+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:20:19.648667+0000) 2024-03-04T12:20:20.652 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:20:20.648+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:20:20.648942+0000) 2024-03-04T12:20:21.652 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:20:21.649+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:20:21.649191+0000) 2024-03-04T12:20:22.652 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:20:22.648+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:20:22.649467+0000) 2024-03-04T12:20:23.653 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:20:23.649+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:20:23.649704+0000) 2024-03-04T12:20:24.653 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:20:24.648+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:20:24.649983+0000) 2024-03-04T12:20:25.653 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:20:25.648+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:20:25.650213+0000) 2024-03-04T12:20:26.654 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:20:26.648+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:20:26.650486+0000) 2024-03-04T12:20:27.654 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:20:27.649+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:20:27.650740+0000) 2024-03-04T12:20:28.654 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:20:28.649+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:20:28.650979+0000) 2024-03-04T12:20:29.654 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:20:29.650+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:20:29.651219+0000) 2024-03-04T12:20:30.654 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:20:30.650+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:20:30.651457+0000) 2024-03-04T12:20:31.655 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:20:31.649+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:20:31.651709+0000) 2024-03-04T12:20:32.655 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:20:32.650+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:20:32.652019+0000) 2024-03-04T12:20:33.655 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:20:33.651+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:20:33.652312+0000) 2024-03-04T12:20:34.656 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:20:34.651+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:20:34.652604+0000) 2024-03-04T12:20:35.656 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:20:35.652+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:20:35.652935+0000) 2024-03-04T12:20:36.656 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:20:36.652+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:20:36.653228+0000) 2024-03-04T12:20:37.657 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:20:37.652+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:20:37.653561+0000) 2024-03-04T12:20:38.657 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:20:38.653+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:20:38.653919+0000) 2024-03-04T12:20:39.657 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:20:39.653+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:20:39.654132+0000) 2024-03-04T12:20:40.657 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:20:40.653+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:20:40.654324+0000) 2024-03-04T12:20:41.658 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:20:41.653+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:20:41.654571+0000) 2024-03-04T12:20:42.658 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:20:42.654+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:20:42.654871+0000) 2024-03-04T12:20:42.791 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:20:42.799 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:20:43.660 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:20:43.654+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:20:43.655120+0000) 2024-03-04T12:20:44.658 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:20:44.654+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:20:44.655367+0000) 2024-03-04T12:20:45.659 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:20:45.654+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:20:45.655611+0000) 2024-03-04T12:20:46.659 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:20:46.654+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:20:46.655802+0000) 2024-03-04T12:20:47.659 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:20:47.655+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:20:47.656100+0000) 2024-03-04T12:20:48.660 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:20:48.655+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:20:48.656408+0000) 2024-03-04T12:20:49.660 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:20:49.655+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:20:49.656642+0000) 2024-03-04T12:20:50.660 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:20:50.656+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:20:50.656920+0000) 2024-03-04T12:20:51.660 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:20:51.656+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:20:51.657166+0000) 2024-03-04T12:20:52.661 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:20:52.656+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:20:52.657413+0000) 2024-03-04T12:20:53.661 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:20:53.657+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:20:53.657652+0000) 2024-03-04T12:20:54.661 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:20:54.657+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:20:54.657922+0000) 2024-03-04T12:20:55.661 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:20:55.657+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:20:55.658108+0000) 2024-03-04T12:20:56.662 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:20:56.657+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:20:56.658358+0000) 2024-03-04T12:20:57.662 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:20:57.658+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:20:57.658621+0000) 2024-03-04T12:20:58.662 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:20:58.658+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:20:58.658917+0000) 2024-03-04T12:20:59.662 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:20:59.658+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:20:59.659148+0000) 2024-03-04T12:21:00.663 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:21:00.658+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:21:00.659338+0000) 2024-03-04T12:21:01.663 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:21:01.659+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:21:01.659607+0000) 2024-03-04T12:21:02.663 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:21:02.659+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:21:02.659887+0000) 2024-03-04T12:21:03.663 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:21:03.659+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:21:03.660118+0000) 2024-03-04T12:21:04.664 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:21:04.659+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:21:04.660357+0000) 2024-03-04T12:21:05.664 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:21:05.660+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:21:05.660618+0000) 2024-03-04T12:21:06.664 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:21:06.660+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:21:06.660931+0000) 2024-03-04T12:21:07.665 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:21:07.660+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:21:07.661203+0000) 2024-03-04T12:21:08.665 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:21:08.660+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:21:08.661472+0000) 2024-03-04T12:21:09.665 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:21:09.660+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:21:09.661712+0000) 2024-03-04T12:21:10.665 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:21:10.661+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:21:10.661948+0000) 2024-03-04T12:21:11.666 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:21:11.661+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:21:11.662194+0000) 2024-03-04T12:21:12.666 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:21:12.662+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:21:12.662441+0000) 2024-03-04T12:21:13.053 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:21:13.061 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:21:13.666 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:21:13.661+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:21:13.662704+0000) 2024-03-04T12:21:14.666 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:21:14.662+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:21:14.662945+0000) 2024-03-04T12:21:15.666 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:21:15.662+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:21:15.663150+0000) 2024-03-04T12:21:16.667 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:21:16.663+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:21:16.663395+0000) 2024-03-04T12:21:17.667 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:21:17.662+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:21:17.663669+0000) 2024-03-04T12:21:18.667 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:21:18.663+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:21:18.663932+0000) 2024-03-04T12:21:19.668 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:21:19.663+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:21:19.664172+0000) 2024-03-04T12:21:20.668 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:21:20.664+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:21:20.664415+0000) 2024-03-04T12:21:21.669 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:21:21.664+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:21:21.664663+0000) 2024-03-04T12:21:22.668 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:21:22.664+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:21:22.664901+0000) 2024-03-04T12:21:23.668 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:21:23.665+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:21:23.665145+0000) 2024-03-04T12:21:24.669 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:21:24.665+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:21:24.665386+0000) 2024-03-04T12:21:25.669 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:21:25.664+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:21:25.665616+0000) 2024-03-04T12:21:26.669 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:21:26.664+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:21:26.665897+0000) 2024-03-04T12:21:27.670 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:21:27.665+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:21:27.666156+0000) 2024-03-04T12:21:28.670 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:21:28.665+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:21:28.666397+0000) 2024-03-04T12:21:29.670 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:21:29.665+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:21:29.666629+0000) 2024-03-04T12:21:30.670 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:21:30.665+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:21:30.666928+0000) 2024-03-04T12:21:31.671 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:21:31.666+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:21:31.667197+0000) 2024-03-04T12:21:32.671 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:21:32.666+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:21:32.667450+0000) 2024-03-04T12:21:33.671 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:21:33.666+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:21:33.667683+0000) 2024-03-04T12:21:34.672 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:21:34.666+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:21:34.667915+0000) 2024-03-04T12:21:35.672 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:21:35.667+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:21:35.668187+0000) 2024-03-04T12:21:36.672 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:21:36.667+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:21:36.668462+0000) 2024-03-04T12:21:37.672 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:21:37.667+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:21:37.668788+0000) 2024-03-04T12:21:38.672 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:21:38.668+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:21:38.669131+0000) 2024-03-04T12:21:39.673 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:21:39.668+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:21:39.669403+0000) 2024-03-04T12:21:40.673 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:21:40.668+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:21:40.669694+0000) 2024-03-04T12:21:41.674 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:21:41.669+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:21:41.669970+0000) 2024-03-04T12:21:42.674 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:21:42.669+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:21:42.670272+0000) 2024-03-04T12:21:43.267 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:21:43.279 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:21:43.674 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:21:43.669+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:21:43.670578+0000) 2024-03-04T12:21:44.674 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:21:44.670+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:21:44.670914+0000) 2024-03-04T12:21:45.675 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:21:45.670+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:21:45.671192+0000) 2024-03-04T12:21:46.675 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:21:46.670+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:21:46.671474+0000) 2024-03-04T12:21:47.675 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:21:47.670+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:21:47.671803+0000) 2024-03-04T12:21:48.676 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:21:48.671+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:21:48.672085+0000) 2024-03-04T12:21:49.676 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:21:49.671+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:21:49.672353+0000) 2024-03-04T12:21:50.676 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:21:50.671+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:21:50.672644+0000) 2024-03-04T12:21:51.676 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:21:51.672+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:21:51.672960+0000) 2024-03-04T12:21:52.677 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:21:52.672+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:21:52.673279+0000) 2024-03-04T12:21:53.677 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:21:53.672+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:21:53.673576+0000) 2024-03-04T12:21:54.677 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:21:54.673+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:21:54.673912+0000) 2024-03-04T12:21:55.678 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:21:55.673+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:21:55.674187+0000) 2024-03-04T12:21:56.678 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:21:56.673+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:21:56.674502+0000) 2024-03-04T12:21:57.678 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:21:57.674+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:21:57.674874+0000) 2024-03-04T12:21:58.679 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:21:58.674+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:21:58.675170+0000) 2024-03-04T12:21:59.679 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:21:59.674+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:21:59.675440+0000) 2024-03-04T12:22:00.679 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:22:00.675+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:22:00.675737+0000) 2024-03-04T12:22:01.680 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:22:01.675+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:22:01.676018+0000) 2024-03-04T12:22:02.680 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:22:02.675+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:22:02.676305+0000) 2024-03-04T12:22:03.680 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:22:03.676+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:22:03.676595+0000) 2024-03-04T12:22:04.680 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:22:04.676+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:22:04.676919+0000) 2024-03-04T12:22:05.681 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:22:05.676+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:22:05.677197+0000) 2024-03-04T12:22:06.681 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:22:06.677+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:22:06.677486+0000) 2024-03-04T12:22:07.681 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:22:07.677+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:22:07.677816+0000) 2024-03-04T12:22:08.682 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:22:08.677+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:22:08.678158+0000) 2024-03-04T12:22:09.682 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:22:09.678+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:22:09.678437+0000) 2024-03-04T12:22:10.683 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:22:10.678+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:22:10.678720+0000) 2024-03-04T12:22:11.683 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:22:11.678+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:22:11.678993+0000) 2024-03-04T12:22:12.683 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:22:12.678+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:22:12.679223+0000) 2024-03-04T12:22:13.413 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:22:13.419 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:22:13.683 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:22:13.678+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:22:13.679466+0000) 2024-03-04T12:22:14.683 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:22:14.678+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:22:14.679771+0000) 2024-03-04T12:22:15.684 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:22:15.679+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:22:15.679981+0000) 2024-03-04T12:22:16.684 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:22:16.679+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:22:16.680172+0000) 2024-03-04T12:22:17.684 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:22:17.680+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:22:17.680470+0000) 2024-03-04T12:22:18.684 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:22:18.679+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:22:18.680748+0000) 2024-03-04T12:22:19.685 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:22:19.680+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:22:19.680988+0000) 2024-03-04T12:22:20.685 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:22:20.681+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:22:20.681252+0000) 2024-03-04T12:22:21.685 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:22:21.681+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:22:21.681499+0000) 2024-03-04T12:22:22.685 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:22:22.681+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:22:22.681755+0000) 2024-03-04T12:22:23.686 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:22:23.681+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:22:23.682011+0000) 2024-03-04T12:22:24.686 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:22:24.682+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:22:24.682299+0000) 2024-03-04T12:22:25.686 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:22:25.682+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:22:25.682547+0000) 2024-03-04T12:22:26.687 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:22:26.681+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:22:26.682802+0000) 2024-03-04T12:22:27.687 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:22:27.682+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:22:27.683139+0000) 2024-03-04T12:22:28.687 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:22:28.681+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:22:28.683387+0000) 2024-03-04T12:22:29.687 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:22:29.682+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:22:29.683633+0000) 2024-03-04T12:22:30.688 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:22:30.682+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:22:30.683927+0000) 2024-03-04T12:22:31.688 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:22:31.683+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:22:31.684175+0000) 2024-03-04T12:22:32.688 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:22:32.682+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:22:32.684457+0000) 2024-03-04T12:22:33.688 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:22:33.683+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:22:33.684718+0000) 2024-03-04T12:22:34.689 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:22:34.683+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:22:34.684983+0000) 2024-03-04T12:22:35.689 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:22:35.684+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:22:35.685229+0000) 2024-03-04T12:22:36.689 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:22:36.684+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:22:36.685482+0000) 2024-03-04T12:22:37.690 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:22:37.684+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:22:37.685754+0000) 2024-03-04T12:22:38.690 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:22:38.685+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:22:38.686009+0000) 2024-03-04T12:22:39.690 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:22:39.685+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:22:39.686252+0000) 2024-03-04T12:22:40.690 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:22:40.685+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:22:40.686505+0000) 2024-03-04T12:22:41.690 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:22:41.685+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:22:41.686753+0000) 2024-03-04T12:22:42.691 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:22:42.686+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:22:42.687012+0000) 2024-03-04T12:22:43.620 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:22:43.628 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:22:43.691 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:22:43.686+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:22:43.687206+0000) 2024-03-04T12:22:44.691 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:22:44.686+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:22:44.687446+0000) 2024-03-04T12:22:45.692 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:22:45.686+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:22:45.687694+0000) 2024-03-04T12:22:46.692 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:22:46.687+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:22:46.687945+0000) 2024-03-04T12:22:47.692 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:22:47.687+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:22:47.688248+0000) 2024-03-04T12:22:48.692 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:22:48.687+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:22:48.688506+0000) 2024-03-04T12:22:49.692 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:22:49.688+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:22:49.688752+0000) 2024-03-04T12:22:50.693 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:22:50.688+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:22:50.689008+0000) 2024-03-04T12:22:51.693 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:22:51.688+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:22:51.689251+0000) 2024-03-04T12:22:52.693 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:22:52.688+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:22:52.689506+0000) 2024-03-04T12:22:53.694 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:22:53.689+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:22:53.689775+0000) 2024-03-04T12:22:54.694 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:22:54.689+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:22:54.690055+0000) 2024-03-04T12:22:55.694 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:22:55.689+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:22:55.690322+0000) 2024-03-04T12:22:56.694 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:22:56.689+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:22:56.690584+0000) 2024-03-04T12:22:57.695 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:22:57.690+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:22:57.690892+0000) 2024-03-04T12:22:58.695 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:22:58.690+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:22:58.691150+0000) 2024-03-04T12:22:59.695 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:22:59.690+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:22:59.691400+0000) 2024-03-04T12:23:00.696 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:23:00.691+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:23:00.691654+0000) 2024-03-04T12:23:01.696 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:23:01.691+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:23:01.691937+0000) 2024-03-04T12:23:02.696 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:23:02.691+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:23:02.692143+0000) 2024-03-04T12:23:03.696 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:23:03.691+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:23:03.692405+0000) 2024-03-04T12:23:04.697 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:23:04.692+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:23:04.692619+0000) 2024-03-04T12:23:05.697 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:23:05.692+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:23:05.692920+0000) 2024-03-04T12:23:06.697 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:23:06.692+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:23:06.693159+0000) 2024-03-04T12:23:07.697 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:23:07.692+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:23:07.693407+0000) 2024-03-04T12:23:08.698 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:23:08.692+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:23:08.693674+0000) 2024-03-04T12:23:09.698 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:23:09.693+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:23:09.693919+0000) 2024-03-04T12:23:10.698 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:23:10.693+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:23:10.694165+0000) 2024-03-04T12:23:11.698 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:23:11.694+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:23:11.694417+0000) 2024-03-04T12:23:12.699 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:23:12.694+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:23:12.694672+0000) 2024-03-04T12:23:13.699 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:23:13.694+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:23:13.694943+0000) 2024-03-04T12:23:13.815 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:23:13.827 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:23:14.699 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:23:14.694+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:23:14.695202+0000) 2024-03-04T12:23:15.700 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:23:15.695+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:23:15.695484+0000) 2024-03-04T12:23:16.700 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:23:16.695+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:23:16.695737+0000) 2024-03-04T12:23:17.700 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:23:17.695+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:23:17.696003+0000) 2024-03-04T12:23:18.700 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:23:18.695+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:23:18.696241+0000) 2024-03-04T12:23:19.700 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:23:19.696+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:23:19.696470+0000) 2024-03-04T12:23:20.701 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:23:20.696+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:23:20.696732+0000) 2024-03-04T12:23:21.701 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:23:21.696+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:23:21.696992+0000) 2024-03-04T12:23:22.701 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:23:22.697+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:23:22.697263+0000) 2024-03-04T12:23:23.702 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:23:23.697+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:23:23.697517+0000) 2024-03-04T12:23:24.702 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:23:24.697+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:23:24.697776+0000) 2024-03-04T12:23:25.702 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:23:25.697+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:23:25.698003+0000) 2024-03-04T12:23:26.702 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:23:26.698+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:23:26.698261+0000) 2024-03-04T12:23:27.703 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:23:27.698+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:23:27.698511+0000) 2024-03-04T12:23:28.703 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:23:28.697+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:23:28.698873+0000) 2024-03-04T12:23:29.703 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:23:29.698+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:23:29.699120+0000) 2024-03-04T12:23:30.703 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:23:30.698+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:23:30.699372+0000) 2024-03-04T12:23:31.704 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:23:31.697+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:23:31.699640+0000) 2024-03-04T12:23:32.704 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:23:32.698+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:23:32.699929+0000) 2024-03-04T12:23:33.704 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:23:33.699+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:23:33.700182+0000) 2024-03-04T12:23:34.704 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:23:34.698+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:23:34.700434+0000) 2024-03-04T12:23:35.705 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:23:35.699+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:23:35.700699+0000) 2024-03-04T12:23:36.705 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:23:36.699+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:23:36.700979+0000) 2024-03-04T12:23:37.705 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:23:37.700+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:23:37.701252+0000) 2024-03-04T12:23:38.705 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:23:38.699+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:23:38.701457+0000) 2024-03-04T12:23:39.706 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:23:39.700+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:23:39.701687+0000) 2024-03-04T12:23:40.706 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:23:40.701+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:23:40.701948+0000) 2024-03-04T12:23:41.706 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:23:41.701+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:23:41.702193+0000) 2024-03-04T12:23:42.707 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:23:42.701+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:23:42.702453+0000) 2024-03-04T12:23:43.707 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:23:43.701+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:23:43.702732+0000) 2024-03-04T12:23:44.030 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:23:44.038 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:23:44.707 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:23:44.702+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:23:44.702987+0000) 2024-03-04T12:23:45.707 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:23:45.702+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:23:45.703235+0000) 2024-03-04T12:23:46.707 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:23:46.702+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:23:46.703445+0000) 2024-03-04T12:23:47.708 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:23:47.702+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:23:47.703722+0000) 2024-03-04T12:23:48.708 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:23:48.703+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:23:48.703980+0000) 2024-03-04T12:23:49.712 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:23:49.703+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:23:49.704226+0000) 2024-03-04T12:23:50.709 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:23:50.703+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:23:50.704480+0000) 2024-03-04T12:23:51.709 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:23:51.703+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:23:51.704744+0000) 2024-03-04T12:23:52.709 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:23:52.704+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:23:52.704982+0000) 2024-03-04T12:23:53.709 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:23:53.704+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:23:53.705219+0000) 2024-03-04T12:23:54.710 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:23:54.704+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:23:54.705505+0000) 2024-03-04T12:23:55.710 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:23:55.705+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:23:55.705782+0000) 2024-03-04T12:23:56.710 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:23:56.705+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:23:56.706105+0000) 2024-03-04T12:23:57.711 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:23:57.705+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:23:57.706406+0000) 2024-03-04T12:23:58.711 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:23:58.705+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:23:58.706698+0000) 2024-03-04T12:23:59.711 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:23:59.706+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:23:59.706947+0000) 2024-03-04T12:24:00.711 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:24:00.706+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:24:00.707230+0000) 2024-03-04T12:24:01.712 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:24:01.706+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:24:01.707525+0000) 2024-03-04T12:24:02.712 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:24:02.706+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:24:02.707757+0000) 2024-03-04T12:24:03.712 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:24:03.707+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:24:03.708074+0000) 2024-03-04T12:24:04.712 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:24:04.707+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:24:04.708358+0000) 2024-03-04T12:24:05.713 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:24:05.707+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:24:05.708633+0000) 2024-03-04T12:24:06.713 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:24:06.708+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:24:06.708954+0000) 2024-03-04T12:24:07.713 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:24:07.708+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:24:07.709211+0000) 2024-03-04T12:24:08.714 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:24:08.708+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:24:08.709501+0000) 2024-03-04T12:24:09.714 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:24:09.709+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:24:09.709783+0000) 2024-03-04T12:24:10.714 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:24:10.709+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:24:10.710081+0000) 2024-03-04T12:24:11.715 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:24:11.709+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:24:11.710354+0000) 2024-03-04T12:24:12.715 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:24:12.709+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:24:12.710651+0000) 2024-03-04T12:24:13.715 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:24:13.710+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:24:13.710939+0000) 2024-03-04T12:24:14.243 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:24:14.251 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:24:14.715 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:24:14.710+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:24:14.711181+0000) 2024-03-04T12:24:15.716 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:24:15.711+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:24:15.711416+0000) 2024-03-04T12:24:16.716 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:24:16.710+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:24:16.711675+0000) 2024-03-04T12:24:17.716 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:24:17.711+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:24:17.711928+0000) 2024-03-04T12:24:18.716 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:24:18.711+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:24:18.712171+0000) 2024-03-04T12:24:19.717 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:24:19.712+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:24:19.712419+0000) 2024-03-04T12:24:20.717 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:24:20.712+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:24:20.712676+0000) 2024-03-04T12:24:21.717 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:24:21.712+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:24:21.712939+0000) 2024-03-04T12:24:22.718 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:24:22.712+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:24:22.713182+0000) 2024-03-04T12:24:23.718 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:24:23.712+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:24:23.713374+0000) 2024-03-04T12:24:24.718 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:24:24.712+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:24:24.713626+0000) 2024-03-04T12:24:25.718 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:24:25.713+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:24:25.713880+0000) 2024-03-04T12:24:26.718 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:24:26.713+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:24:26.714127+0000) 2024-03-04T12:24:27.719 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:24:27.713+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:24:27.714323+0000) 2024-03-04T12:24:28.719 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:24:28.714+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:24:28.714510+0000) 2024-03-04T12:24:29.719 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:24:29.713+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:24:29.714707+0000) 2024-03-04T12:24:30.719 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:24:30.713+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:24:30.714996+0000) 2024-03-04T12:24:31.720 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:24:31.714+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:24:31.715224+0000) 2024-03-04T12:24:32.720 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:24:32.714+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:24:32.715478+0000) 2024-03-04T12:24:33.720 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:24:33.714+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:24:33.715722+0000) 2024-03-04T12:24:34.720 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:24:34.714+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:24:34.715950+0000) 2024-03-04T12:24:35.721 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:24:35.715+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:24:35.716178+0000) 2024-03-04T12:24:36.721 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:24:36.715+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:24:36.716420+0000) 2024-03-04T12:24:37.721 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:24:37.715+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:24:37.716679+0000) 2024-03-04T12:24:38.721 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:24:38.715+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:24:38.716954+0000) 2024-03-04T12:24:39.721 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:24:39.716+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:24:39.717178+0000) 2024-03-04T12:24:40.722 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:24:40.716+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:24:40.717417+0000) 2024-03-04T12:24:41.722 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:24:41.716+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:24:41.717653+0000) 2024-03-04T12:24:42.722 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:24:42.717+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:24:42.717945+0000) 2024-03-04T12:24:43.722 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:24:43.717+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:24:43.718184+0000) 2024-03-04T12:24:44.394 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:24:44.402 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:24:44.723 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:24:44.717+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:24:44.718438+0000) 2024-03-04T12:24:45.723 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:24:45.717+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:24:45.718627+0000) 2024-03-04T12:24:46.723 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:24:46.718+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:24:46.718909+0000) 2024-03-04T12:24:47.723 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:24:47.718+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:24:47.719155+0000) 2024-03-04T12:24:48.724 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:24:48.718+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:24:48.719398+0000) 2024-03-04T12:24:49.724 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:24:49.718+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:24:49.719628+0000) 2024-03-04T12:24:50.724 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:24:50.719+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:24:50.719932+0000) 2024-03-04T12:24:51.725 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:24:51.719+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:24:51.720180+0000) 2024-03-04T12:24:52.725 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:24:52.719+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:24:52.720422+0000) 2024-03-04T12:24:53.725 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:24:53.719+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:24:53.720662+0000) 2024-03-04T12:24:54.725 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:24:54.720+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:24:54.720952+0000) 2024-03-04T12:24:55.725 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:24:55.720+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:24:55.721131+0000) 2024-03-04T12:24:56.726 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:24:56.720+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:24:56.721368+0000) 2024-03-04T12:24:57.726 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:24:57.720+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:24:57.721627+0000) 2024-03-04T12:24:58.726 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:24:58.721+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:24:58.721927+0000) 2024-03-04T12:24:59.727 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:24:59.721+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:24:59.722158+0000) 2024-03-04T12:25:00.727 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:25:00.721+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:25:00.722398+0000) 2024-03-04T12:25:01.727 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:25:01.722+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:25:01.722629+0000) 2024-03-04T12:25:02.727 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:25:02.722+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:25:02.722914+0000) 2024-03-04T12:25:03.728 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:25:03.722+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:25:03.723154+0000) 2024-03-04T12:25:04.728 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:25:04.722+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:25:04.723395+0000) 2024-03-04T12:25:05.728 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:25:05.722+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:25:05.723571+0000) 2024-03-04T12:25:06.735 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:25:06.723+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:25:06.723812+0000) 2024-03-04T12:25:07.729 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:25:07.723+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:25:07.724112+0000) 2024-03-04T12:25:08.729 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:25:08.723+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:25:08.724355+0000) 2024-03-04T12:25:09.729 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:25:09.724+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:25:09.724583+0000) 2024-03-04T12:25:10.729 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:25:10.723+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:25:10.724772+0000) 2024-03-04T12:25:11.729 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:25:11.724+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:25:11.725006+0000) 2024-03-04T12:25:12.730 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:25:12.724+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:25:12.725258+0000) 2024-03-04T12:25:13.730 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:25:13.725+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:25:13.725499+0000) 2024-03-04T12:25:14.603 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:25:14.611 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:25:14.765 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:25:14.725+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:25:14.725810+0000) 2024-03-04T12:25:15.731 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:25:15.725+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:25:15.726090+0000) 2024-03-04T12:25:16.731 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:25:16.725+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:25:16.726332+0000) 2024-03-04T12:25:17.731 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:25:17.726+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:25:17.726584+0000) 2024-03-04T12:25:18.731 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:25:18.726+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:25:18.726856+0000) 2024-03-04T12:25:19.731 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:25:19.726+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:25:19.727028+0000) 2024-03-04T12:25:20.732 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:25:20.726+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:25:20.727272+0000) 2024-03-04T12:25:21.732 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:25:21.727+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:25:21.727528+0000) 2024-03-04T12:25:22.732 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:25:22.726+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:25:22.727792+0000) 2024-03-04T12:25:23.733 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:25:23.727+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:25:23.728115+0000) 2024-03-04T12:25:24.733 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:25:24.728+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:25:24.728408+0000) 2024-03-04T12:25:25.733 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:25:25.728+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:25:25.728688+0000) 2024-03-04T12:25:26.733 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:25:26.728+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:25:26.728972+0000) 2024-03-04T12:25:27.734 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:25:27.728+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:25:27.729275+0000) 2024-03-04T12:25:28.734 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:25:28.729+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:25:28.729567+0000) 2024-03-04T12:25:29.734 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:25:29.729+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:25:29.729874+0000) 2024-03-04T12:25:30.735 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:25:30.729+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:25:30.730164+0000) 2024-03-04T12:25:31.735 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:25:31.730+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:25:31.730477+0000) 2024-03-04T12:25:32.735 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:25:32.729+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:25:32.730769+0000) 2024-03-04T12:25:33.736 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:25:33.729+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:25:33.731066+0000) 2024-03-04T12:25:34.736 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:25:34.730+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:25:34.731310+0000) 2024-03-04T12:25:35.736 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:25:35.730+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:25:35.731534+0000) 2024-03-04T12:25:36.737 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:25:36.730+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:25:36.731771+0000) 2024-03-04T12:25:37.737 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:25:37.731+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:25:37.732091+0000) 2024-03-04T12:25:38.737 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:25:38.731+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:25:38.732389+0000) 2024-03-04T12:25:39.737 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:25:39.731+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:25:39.732691+0000) 2024-03-04T12:25:40.737 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:25:40.731+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:25:40.732963+0000) 2024-03-04T12:25:41.738 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:25:41.732+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:25:41.733240+0000) 2024-03-04T12:25:42.738 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:25:42.732+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:25:42.733535+0000) 2024-03-04T12:25:43.739 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:25:43.732+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:25:43.733788+0000) 2024-03-04T12:25:44.739 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:25:44.733+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:25:44.734118+0000) 2024-03-04T12:25:44.833 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:25:44.841 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:25:45.739 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:25:45.733+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:25:45.734416+0000) 2024-03-04T12:25:46.739 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:25:46.733+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:25:46.734711+0000) 2024-03-04T12:25:47.740 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:25:47.734+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:25:47.735032+0000) 2024-03-04T12:25:48.740 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:25:48.734+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:25:48.735338+0000) 2024-03-04T12:25:49.740 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:25:49.734+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:25:49.735622+0000) 2024-03-04T12:25:50.740 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:25:50.735+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:25:50.735957+0000) 2024-03-04T12:25:51.741 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:25:51.735+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:25:51.736260+0000) 2024-03-04T12:25:52.741 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:25:52.735+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:25:52.736545+0000) 2024-03-04T12:25:53.741 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:25:53.736+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:25:53.736876+0000) 2024-03-04T12:25:54.742 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:25:54.736+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:25:54.737168+0000) 2024-03-04T12:25:55.742 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:25:55.736+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:25:55.737434+0000) 2024-03-04T12:25:56.742 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:25:56.736+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:25:56.737743+0000) 2024-03-04T12:25:57.743 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:25:57.737+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:25:57.738043+0000) 2024-03-04T12:25:58.743 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:25:58.737+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:25:58.738344+0000) 2024-03-04T12:25:59.743 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:25:59.737+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:25:59.738565+0000) 2024-03-04T12:26:00.744 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:26:00.738+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:26:00.738890+0000) 2024-03-04T12:26:01.744 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:26:01.738+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:26:01.739135+0000) 2024-03-04T12:26:02.744 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:26:02.738+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:26:02.739486+0000) 2024-03-04T12:26:03.744 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:26:03.738+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:26:03.739774+0000) 2024-03-04T12:26:04.745 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:26:04.739+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:26:04.740056+0000) 2024-03-04T12:26:05.745 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:26:05.739+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:26:05.740336+0000) 2024-03-04T12:26:06.745 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:26:06.740+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:26:06.740624+0000) 2024-03-04T12:26:07.746 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:26:07.740+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:26:07.740956+0000) 2024-03-04T12:26:08.746 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:26:08.740+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:26:08.741247+0000) 2024-03-04T12:26:09.746 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:26:09.740+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:26:09.741549+0000) 2024-03-04T12:26:10.746 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:26:10.741+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:26:10.741876+0000) 2024-03-04T12:26:11.747 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:26:11.741+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:26:11.742144+0000) 2024-03-04T12:26:12.747 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:26:12.741+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:26:12.742433+0000) 2024-03-04T12:26:13.748 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:26:13.741+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:26:13.742721+0000) 2024-03-04T12:26:14.748 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:26:14.742+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:26:14.743035+0000) 2024-03-04T12:26:14.980 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:26:14.988 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:26:15.748 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:26:15.742+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:26:15.743266+0000) 2024-03-04T12:26:16.748 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:26:16.743+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:26:16.743549+0000) 2024-03-04T12:26:17.748 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:26:17.742+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:26:17.743794+0000) 2024-03-04T12:26:18.749 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:26:18.743+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:26:18.744277+0000) 2024-03-04T12:26:19.749 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:26:19.743+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:26:19.744514+0000) 2024-03-04T12:26:20.749 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:26:20.744+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:26:20.744753+0000) 2024-03-04T12:26:21.750 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:26:21.744+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:26:21.744992+0000) 2024-03-04T12:26:22.750 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:26:22.744+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:26:22.745279+0000) 2024-03-04T12:26:23.750 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:26:23.744+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:26:23.745565+0000) 2024-03-04T12:26:24.751 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:26:24.745+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:26:24.745919+0000) 2024-03-04T12:26:25.751 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:26:25.745+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:26:25.746148+0000) 2024-03-04T12:26:26.751 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:26:26.746+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:26:26.746432+0000) 2024-03-04T12:26:27.751 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:26:27.746+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:26:27.746716+0000) 2024-03-04T12:26:28.752 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:26:28.746+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:26:28.747006+0000) 2024-03-04T12:26:29.756 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:26:29.747+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:26:29.747273+0000) 2024-03-04T12:26:30.752 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:26:30.747+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:26:30.747558+0000) 2024-03-04T12:26:31.752 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:26:31.747+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:26:31.747860+0000) 2024-03-04T12:26:32.753 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:26:32.747+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:26:32.748149+0000) 2024-03-04T12:26:33.753 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:26:33.748+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:26:33.748460+0000) 2024-03-04T12:26:34.753 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:26:34.747+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:26:34.748743+0000) 2024-03-04T12:26:35.754 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:26:35.747+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:26:35.749007+0000) 2024-03-04T12:26:36.754 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:26:36.748+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:26:36.749295+0000) 2024-03-04T12:26:37.754 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:26:37.748+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:26:37.749578+0000) 2024-03-04T12:26:38.755 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:26:38.748+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:26:38.749923+0000) 2024-03-04T12:26:39.755 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:26:39.748+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:26:39.750182+0000) 2024-03-04T12:26:40.755 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:26:40.749+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:26:40.750502+0000) 2024-03-04T12:26:41.756 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:26:41.749+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:26:41.750788+0000) 2024-03-04T12:26:42.756 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:26:42.750+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:26:42.751135+0000) 2024-03-04T12:26:43.756 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:26:43.750+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:26:43.751415+0000) 2024-03-04T12:26:44.757 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:26:44.750+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:26:44.751703+0000) 2024-03-04T12:26:45.118 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:26:45.126 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:26:45.757 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:26:45.751+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:26:45.751995+0000) 2024-03-04T12:26:46.757 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:26:46.751+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:26:46.752267+0000) 2024-03-04T12:26:47.757 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:26:47.751+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:26:47.752545+0000) 2024-03-04T12:26:48.758 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:26:48.752+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:26:48.752882+0000) 2024-03-04T12:26:49.758 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:26:49.752+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:26:49.753155+0000) 2024-03-04T12:26:50.758 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:26:50.752+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:26:50.753439+0000) 2024-03-04T12:26:51.759 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:26:51.752+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:26:51.753759+0000) 2024-03-04T12:26:52.759 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:26:52.753+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:26:52.754068+0000) 2024-03-04T12:26:53.759 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:26:53.753+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:26:53.754365+0000) 2024-03-04T12:26:54.760 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:26:54.753+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:26:54.754660+0000) 2024-03-04T12:26:55.760 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:26:55.754+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:26:55.754970+0000) 2024-03-04T12:26:56.760 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:26:56.754+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:26:56.755336+0000) 2024-03-04T12:26:57.761 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:26:57.754+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:26:57.755639+0000) 2024-03-04T12:26:58.761 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:26:58.755+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:26:58.755927+0000) 2024-03-04T12:26:59.761 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:26:59.755+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:26:59.756210+0000) 2024-03-04T12:27:00.762 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:27:00.755+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:27:00.756495+0000) 2024-03-04T12:27:01.762 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:27:01.756+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:27:01.756800+0000) 2024-03-04T12:27:02.762 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:27:02.756+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:27:02.757140+0000) 2024-03-04T12:27:03.762 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:27:03.756+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:27:03.757424+0000) 2024-03-04T12:27:04.763 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:27:04.756+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:27:04.757631+0000) 2024-03-04T12:27:05.763 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:27:05.757+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:27:05.757937+0000) 2024-03-04T12:27:06.763 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:27:06.757+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:27:06.758198+0000) 2024-03-04T12:27:07.763 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:27:07.757+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:27:07.758445+0000) 2024-03-04T12:27:08.764 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:27:08.757+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:27:08.758708+0000) 2024-03-04T12:27:09.764 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:27:09.758+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:27:09.758966+0000) 2024-03-04T12:27:10.764 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:27:10.758+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:27:10.759205+0000) 2024-03-04T12:27:11.764 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:27:11.758+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:27:11.759442+0000) 2024-03-04T12:27:12.765 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:27:12.759+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:27:12.759678+0000) 2024-03-04T12:27:13.765 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:27:13.759+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:27:13.759946+0000) 2024-03-04T12:27:14.765 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:27:14.759+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:27:14.760141+0000) 2024-03-04T12:27:15.331 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:27:15.339 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:27:15.765 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:27:15.759+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:27:15.760385+0000) 2024-03-04T12:27:16.766 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:27:16.760+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:27:16.760659+0000) 2024-03-04T12:27:17.766 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:27:17.760+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:27:17.760943+0000) 2024-03-04T12:27:18.766 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:27:18.760+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:27:18.761135+0000) 2024-03-04T12:27:19.766 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:27:19.760+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:27:19.761329+0000) 2024-03-04T12:27:20.767 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:27:20.761+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:27:20.761569+0000) 2024-03-04T12:27:21.767 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:27:21.761+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:27:21.761809+0000) 2024-03-04T12:27:22.767 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:27:22.761+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:27:22.762082+0000) 2024-03-04T12:27:23.768 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:27:23.762+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:27:23.762338+0000) 2024-03-04T12:27:24.768 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:27:24.762+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:27:24.762614+0000) 2024-03-04T12:27:25.768 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:27:25.762+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:27:25.762935+0000) 2024-03-04T12:27:26.768 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:27:26.762+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:27:26.763240+0000) 2024-03-04T12:27:27.768 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:27:27.763+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:27:27.763546+0000) 2024-03-04T12:27:28.769 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:27:28.763+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:27:28.763875+0000) 2024-03-04T12:27:29.769 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:27:29.763+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:27:29.764172+0000) 2024-03-04T12:27:30.770 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:27:30.764+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:27:30.764485+0000) 2024-03-04T12:27:31.770 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:27:31.764+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:27:31.764778+0000) 2024-03-04T12:27:32.770 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:27:32.764+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:27:32.765059+0000) 2024-03-04T12:27:33.770 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:27:33.765+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:27:33.765364+0000) 2024-03-04T12:27:34.771 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:27:34.765+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:27:34.765678+0000) 2024-03-04T12:27:35.771 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:27:35.764+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:27:35.765965+0000) 2024-03-04T12:27:36.771 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:27:36.765+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:27:36.766243+0000) 2024-03-04T12:27:37.772 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:27:37.765+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:27:37.766579+0000) 2024-03-04T12:27:38.772 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:27:38.765+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:27:38.766862+0000) 2024-03-04T12:27:39.772 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:27:39.766+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:27:39.767161+0000) 2024-03-04T12:27:40.772 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:27:40.766+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:27:40.767448+0000) 2024-03-04T12:27:41.773 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:27:41.766+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:27:41.767741+0000) 2024-03-04T12:27:42.773 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:27:42.766+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:27:42.768028+0000) 2024-03-04T12:27:43.773 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:27:43.767+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:27:43.768319+0000) 2024-03-04T12:27:44.774 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:27:44.767+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:27:44.768631+0000) 2024-03-04T12:27:45.464 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:27:45.472 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:27:45.774 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:27:45.768+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:27:45.768978+0000) 2024-03-04T12:27:46.775 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:27:46.768+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:27:46.769279+0000) 2024-03-04T12:27:47.775 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:27:47.768+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:27:47.769587+0000) 2024-03-04T12:27:48.775 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:27:48.769+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:27:48.769930+0000) 2024-03-04T12:27:49.775 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:27:49.769+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:27:49.770221+0000) 2024-03-04T12:27:50.776 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:27:50.769+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:27:50.770536+0000) 2024-03-04T12:27:51.776 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:27:51.770+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:27:51.770878+0000) 2024-03-04T12:27:52.776 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:27:52.770+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:27:52.771172+0000) 2024-03-04T12:27:53.777 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:27:53.770+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:27:53.771418+0000) 2024-03-04T12:27:54.777 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:27:54.770+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:27:54.771708+0000) 2024-03-04T12:27:55.777 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:27:55.771+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:27:55.771999+0000) 2024-03-04T12:27:56.777 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:27:56.771+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:27:56.772250+0000) 2024-03-04T12:27:57.778 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:27:57.771+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:27:57.772537+0000) 2024-03-04T12:27:58.778 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:27:58.772+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:27:58.772796+0000) 2024-03-04T12:27:59.778 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:27:59.772+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:27:59.773085+0000) 2024-03-04T12:28:00.778 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:28:00.772+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:28:00.773352+0000) 2024-03-04T12:28:01.779 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:28:01.772+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:28:01.773586+0000) 2024-03-04T12:28:02.779 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:28:02.773+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:28:02.773816+0000) 2024-03-04T12:28:03.779 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:28:03.773+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:28:03.774106+0000) 2024-03-04T12:28:04.780 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:28:04.773+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:28:04.774369+0000) 2024-03-04T12:28:05.780 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:28:05.773+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:28:05.774610+0000) 2024-03-04T12:28:06.780 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:28:06.774+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:28:06.774890+0000) 2024-03-04T12:28:07.780 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:28:07.774+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:28:07.775133+0000) 2024-03-04T12:28:08.781 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:28:08.774+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:28:08.775374+0000) 2024-03-04T12:28:09.781 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:28:09.775+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:28:09.775618+0000) 2024-03-04T12:28:10.781 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:28:10.775+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:28:10.775912+0000) 2024-03-04T12:28:11.781 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:28:11.775+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:28:11.776112+0000) 2024-03-04T12:28:12.781 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:28:12.775+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:28:12.776286+0000) 2024-03-04T12:28:13.782 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:28:13.776+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:28:13.776526+0000) 2024-03-04T12:28:14.782 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:28:14.775+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:28:14.776726+0000) 2024-03-04T12:28:15.675 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:28:15.683 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:28:15.811 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:28:15.776+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:28:15.777025+0000) 2024-03-04T12:28:16.783 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:28:16.776+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:28:16.777287+0000) 2024-03-04T12:28:17.783 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:28:17.777+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:28:17.777534+0000) 2024-03-04T12:28:18.783 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:28:18.777+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:28:18.777782+0000) 2024-03-04T12:28:19.783 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:28:19.777+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:28:19.778043+0000) 2024-03-04T12:28:20.784 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:28:20.777+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:28:20.778304+0000) 2024-03-04T12:28:21.784 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:28:21.778+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:28:21.778543+0000) 2024-03-04T12:28:22.784 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:28:22.778+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:28:22.778813+0000) 2024-03-04T12:28:23.784 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:28:23.778+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:28:23.779086+0000) 2024-03-04T12:28:24.784 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:28:24.779+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:28:24.779332+0000) 2024-03-04T12:28:25.785 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:28:25.779+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:28:25.779572+0000) 2024-03-04T12:28:26.785 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:28:26.779+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:28:26.779871+0000) 2024-03-04T12:28:27.785 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:28:27.779+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:28:27.780136+0000) 2024-03-04T12:28:28.786 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:28:28.780+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:28:28.780399+0000) 2024-03-04T12:28:29.786 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:28:29.780+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:28:29.780662+0000) 2024-03-04T12:28:30.786 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:28:30.780+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:28:30.780953+0000) 2024-03-04T12:28:31.787 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:28:31.781+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:28:31.781205+0000) 2024-03-04T12:28:32.787 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:28:32.780+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:28:32.781408+0000) 2024-03-04T12:28:33.787 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:28:33.781+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:28:33.781651+0000) 2024-03-04T12:28:34.787 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:28:34.781+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:28:34.781931+0000) 2024-03-04T12:28:35.788 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:28:35.781+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:28:35.782169+0000) 2024-03-04T12:28:36.788 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:28:36.781+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:28:36.782412+0000) 2024-03-04T12:28:37.788 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:28:37.781+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:28:37.782656+0000) 2024-03-04T12:28:38.788 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:28:38.781+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:28:38.782940+0000) 2024-03-04T12:28:39.788 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:28:39.781+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:28:39.783135+0000) 2024-03-04T12:28:40.789 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:28:40.782+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:28:40.783375+0000) 2024-03-04T12:28:41.789 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:28:41.782+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:28:41.783615+0000) 2024-03-04T12:28:42.789 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:28:42.782+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:28:42.783896+0000) 2024-03-04T12:28:43.790 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:28:43.783+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:28:43.784140+0000) 2024-03-04T12:28:44.790 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:28:44.783+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:28:44.784392+0000) 2024-03-04T12:28:45.790 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:28:45.783+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:28:45.784671+0000) 2024-03-04T12:28:45.879 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:28:45.890 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:28:46.790 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:28:46.784+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:28:46.785002+0000) 2024-03-04T12:28:47.791 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:28:47.784+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:28:47.785305+0000) 2024-03-04T12:28:48.791 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:28:48.784+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:28:48.785616+0000) 2024-03-04T12:28:49.791 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:28:49.785+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:28:49.785956+0000) 2024-03-04T12:28:50.792 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:28:50.785+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:28:50.786249+0000) 2024-03-04T12:28:51.792 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:28:51.785+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:28:51.786495+0000) 2024-03-04T12:28:52.792 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:28:52.785+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:28:52.786805+0000) 2024-03-04T12:28:53.793 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:28:53.786+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:28:53.787157+0000) 2024-03-04T12:28:54.793 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:28:54.786+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:28:54.787456+0000) 2024-03-04T12:28:55.793 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:28:55.786+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:28:55.787746+0000) 2024-03-04T12:28:56.793 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:28:56.787+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:28:56.788042+0000) 2024-03-04T12:28:57.794 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:28:57.787+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:28:57.788362+0000) 2024-03-04T12:28:58.794 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:28:58.787+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:28:58.788663+0000) 2024-03-04T12:28:59.794 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:28:59.788+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:28:59.788991+0000) 2024-03-04T12:29:00.795 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:29:00.788+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:29:00.789236+0000) 2024-03-04T12:29:01.795 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:29:01.788+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:29:01.789526+0000) 2024-03-04T12:29:02.795 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:29:02.789+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:29:02.789875+0000) 2024-03-04T12:29:03.796 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:29:03.789+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:29:03.790177+0000) 2024-03-04T12:29:04.796 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:29:04.789+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:29:04.790438+0000) 2024-03-04T12:29:05.796 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:29:05.790+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:29:05.790689+0000) 2024-03-04T12:29:06.796 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:29:06.790+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:29:06.790934+0000) 2024-03-04T12:29:07.797 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:29:07.790+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:29:07.791180+0000) 2024-03-04T12:29:08.797 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:29:08.790+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:29:08.791427+0000) 2024-03-04T12:29:09.797 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:29:09.791+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:29:09.791673+0000) 2024-03-04T12:29:10.797 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:29:10.791+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:29:10.791947+0000) 2024-03-04T12:29:11.798 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:29:11.791+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:29:11.792193+0000) 2024-03-04T12:29:12.798 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:29:12.791+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:29:12.792438+0000) 2024-03-04T12:29:13.798 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:29:13.791+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:29:13.792670+0000) 2024-03-04T12:29:14.798 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:29:14.792+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:29:14.792952+0000) 2024-03-04T12:29:15.799 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:29:15.792+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:29:15.793179+0000) 2024-03-04T12:29:16.025 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:29:16.034 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:29:16.799 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:29:16.792+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:29:16.793452+0000) 2024-03-04T12:29:17.799 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:29:17.792+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:29:17.793650+0000) 2024-03-04T12:29:18.799 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:29:18.793+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:29:18.793928+0000) 2024-03-04T12:29:19.800 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:29:19.793+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:29:19.794172+0000) 2024-03-04T12:29:20.800 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:29:20.794+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:29:20.794426+0000) 2024-03-04T12:29:21.800 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:29:21.794+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:29:21.794700+0000) 2024-03-04T12:29:22.801 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:29:22.794+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:29:22.794996+0000) 2024-03-04T12:29:23.801 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:29:23.794+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:29:23.795293+0000) 2024-03-04T12:29:24.801 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:29:24.795+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:29:24.795590+0000) 2024-03-04T12:29:25.801 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:29:25.795+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:29:25.795931+0000) 2024-03-04T12:29:26.802 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:29:26.795+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:29:26.796225+0000) 2024-03-04T12:29:27.802 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:29:27.796+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:29:27.796519+0000) 2024-03-04T12:29:28.802 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:29:28.795+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:29:28.796756+0000) 2024-03-04T12:29:29.803 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:29:29.796+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:29:29.797039+0000) 2024-03-04T12:29:30.803 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:29:30.797+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:29:30.797318+0000) 2024-03-04T12:29:31.803 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:29:31.797+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:29:31.797556+0000) 2024-03-04T12:29:32.803 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:29:32.797+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:29:32.797889+0000) 2024-03-04T12:29:33.804 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:29:33.797+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:29:33.798167+0000) 2024-03-04T12:29:34.804 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:29:34.798+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:29:34.798463+0000) 2024-03-04T12:29:35.805 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:29:35.797+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:29:35.798759+0000) 2024-03-04T12:29:36.805 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:29:36.797+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:29:36.799078+0000) 2024-03-04T12:29:37.805 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:29:37.797+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:29:37.799319+0000) 2024-03-04T12:29:38.805 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:29:38.798+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:29:38.799619+0000) 2024-03-04T12:29:39.805 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:29:39.798+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:29:39.799944+0000) 2024-03-04T12:29:40.806 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:29:40.798+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:29:40.800219+0000) 2024-03-04T12:29:41.806 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:29:41.799+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:29:41.800550+0000) 2024-03-04T12:29:42.807 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:29:42.799+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:29:42.800871+0000) 2024-03-04T12:29:43.807 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:29:43.800+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:29:43.801203+0000) 2024-03-04T12:29:44.807 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:29:44.800+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:29:44.801490+0000) 2024-03-04T12:29:45.807 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:29:45.800+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:29:45.801787+0000) 2024-03-04T12:29:46.240 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:29:46.251 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:29:46.808 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:29:46.801+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:29:46.802126+0000) 2024-03-04T12:29:47.808 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:29:47.801+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:29:47.802422+0000) 2024-03-04T12:29:48.808 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:29:48.801+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:29:48.802706+0000) 2024-03-04T12:29:49.809 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:29:49.802+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:29:49.802991+0000) 2024-03-04T12:29:50.809 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:29:50.802+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:29:50.803316+0000) 2024-03-04T12:29:51.809 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:29:51.802+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:29:51.803598+0000) 2024-03-04T12:29:52.809 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:29:52.803+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:29:52.803932+0000) 2024-03-04T12:29:53.810 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:29:53.803+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:29:53.804182+0000) 2024-03-04T12:29:54.810 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:29:54.803+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:29:54.804497+0000) 2024-03-04T12:29:55.810 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:29:55.803+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:29:55.804777+0000) 2024-03-04T12:29:56.811 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:29:56.804+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:29:56.805123+0000) 2024-03-04T12:29:57.811 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:29:57.804+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:29:57.805417+0000) 2024-03-04T12:29:58.811 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:29:58.804+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:29:58.805724+0000) 2024-03-04T12:29:59.812 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:29:59.805+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:29:59.806023+0000) 2024-03-04T12:30:00.812 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:30:00.805+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:30:00.806312+0000) 2024-03-04T12:30:01.812 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:30:01.805+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:30:01.806604+0000) 2024-03-04T12:30:02.813 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:30:02.806+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:30:02.806937+0000) 2024-03-04T12:30:03.813 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:30:03.806+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:30:03.807242+0000) 2024-03-04T12:30:04.813 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:30:04.806+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:30:04.807534+0000) 2024-03-04T12:30:05.813 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:30:05.807+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:30:05.807871+0000) 2024-03-04T12:30:06.814 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:30:06.807+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:30:06.808167+0000) 2024-03-04T12:30:07.814 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:30:07.807+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:30:07.808456+0000) 2024-03-04T12:30:08.815 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:30:08.808+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:30:08.808721+0000) 2024-03-04T12:30:09.815 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:30:09.808+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:30:09.809041+0000) 2024-03-04T12:30:10.815 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:30:10.808+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:30:10.809330+0000) 2024-03-04T12:30:11.816 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:30:11.809+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:30:11.809621+0000) 2024-03-04T12:30:12.816 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:30:12.809+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:30:12.809929+0000) 2024-03-04T12:30:13.816 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:30:13.809+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:30:13.810192+0000) 2024-03-04T12:30:14.816 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:30:14.809+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:30:14.810390+0000) 2024-03-04T12:30:15.816 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:30:15.810+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:30:15.810626+0000) 2024-03-04T12:30:16.462 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:30:16.469 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:30:16.816 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:30:16.810+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:30:16.810860+0000) 2024-03-04T12:30:17.817 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:30:17.810+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:30:17.811134+0000) 2024-03-04T12:30:18.817 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:30:18.810+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:30:18.811385+0000) 2024-03-04T12:30:19.819 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:30:19.811+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:30:19.811627+0000) 2024-03-04T12:30:20.818 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:30:20.811+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:30:20.811905+0000) 2024-03-04T12:30:21.818 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:30:21.811+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:30:21.812162+0000) 2024-03-04T12:30:22.818 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:30:22.812+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:30:22.812417+0000) 2024-03-04T12:30:23.818 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:30:23.811+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:30:23.812680+0000) 2024-03-04T12:30:24.819 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:30:24.812+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:30:24.812950+0000) 2024-03-04T12:30:25.819 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:30:25.812+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:30:25.813187+0000) 2024-03-04T12:30:26.820 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:30:26.813+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:30:26.813451+0000) 2024-03-04T12:30:27.819 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:30:27.813+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:30:27.813692+0000) 2024-03-04T12:30:28.820 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:30:28.813+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:30:28.813949+0000) 2024-03-04T12:30:29.820 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:30:29.813+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:30:29.814149+0000) 2024-03-04T12:30:30.820 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:30:30.814+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:30:30.814407+0000) 2024-03-04T12:30:31.820 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:30:31.814+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:30:31.814678+0000) 2024-03-04T12:30:32.821 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:30:32.814+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:30:32.814932+0000) 2024-03-04T12:30:33.821 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:30:33.815+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:30:33.815179+0000) 2024-03-04T12:30:34.821 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:30:34.814+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:30:34.815501+0000) 2024-03-04T12:30:35.821 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:30:35.815+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:30:35.815743+0000) 2024-03-04T12:30:36.822 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:30:36.814+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:30:36.815981+0000) 2024-03-04T12:30:37.824 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:30:37.815+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:30:37.816227+0000) 2024-03-04T12:30:38.822 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:30:38.815+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:30:38.816482+0000) 2024-03-04T12:30:39.822 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:30:39.815+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:30:39.816738+0000) 2024-03-04T12:30:40.823 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:30:40.815+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:30:40.816983+0000) 2024-03-04T12:30:41.823 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:30:41.816+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:30:41.817220+0000) 2024-03-04T12:30:42.823 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:30:42.816+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:30:42.817455+0000) 2024-03-04T12:30:43.823 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:30:43.816+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:30:43.817701+0000) 2024-03-04T12:30:44.823 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:30:44.816+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:30:44.817948+0000) 2024-03-04T12:30:45.824 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:30:45.817+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:30:45.818188+0000) 2024-03-04T12:30:46.602 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:30:46.610 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:30:46.824 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:30:46.817+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:30:46.818458+0000) 2024-03-04T12:30:47.824 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:30:47.817+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:30:47.818739+0000) 2024-03-04T12:30:48.825 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:30:48.818+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:30:48.818989+0000) 2024-03-04T12:30:49.825 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:30:49.818+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:30:49.819235+0000) 2024-03-04T12:30:50.825 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:30:50.818+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:30:50.819474+0000) 2024-03-04T12:30:51.825 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:30:51.818+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:30:51.819729+0000) 2024-03-04T12:30:52.825 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:30:52.819+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:30:52.819998+0000) 2024-03-04T12:30:53.826 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:30:53.819+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:30:53.820193+0000) 2024-03-04T12:30:54.826 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:30:54.819+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:30:54.820444+0000) 2024-03-04T12:30:55.826 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:30:55.819+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:30:55.820697+0000) 2024-03-04T12:30:56.826 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:30:56.820+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:30:56.820932+0000) 2024-03-04T12:30:57.827 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:30:57.820+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:30:57.821174+0000) 2024-03-04T12:30:58.827 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:30:58.820+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:30:58.821355+0000) 2024-03-04T12:30:59.827 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:30:59.820+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:30:59.821604+0000) 2024-03-04T12:31:00.828 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:31:00.821+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:31:00.821884+0000) 2024-03-04T12:31:01.828 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:31:01.821+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:31:01.822143+0000) 2024-03-04T12:31:02.828 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:31:02.821+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:31:02.822379+0000) 2024-03-04T12:31:03.828 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:31:03.821+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:31:03.822615+0000) 2024-03-04T12:31:04.829 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:31:04.822+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:31:04.822907+0000) 2024-03-04T12:31:05.829 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:31:05.822+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:31:05.823147+0000) 2024-03-04T12:31:06.829 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:31:06.822+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:31:06.823385+0000) 2024-03-04T12:31:07.829 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:31:07.822+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:31:07.823564+0000) 2024-03-04T12:31:08.829 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:31:08.822+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:31:08.823802+0000) 2024-03-04T12:31:09.829 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:31:09.823+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:31:09.824097+0000) 2024-03-04T12:31:10.830 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:31:10.823+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:31:10.824341+0000) 2024-03-04T12:31:11.830 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:31:11.824+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:31:11.824580+0000) 2024-03-04T12:31:12.830 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:31:12.824+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:31:12.824811+0000) 2024-03-04T12:31:13.830 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:31:13.824+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:31:13.825091+0000) 2024-03-04T12:31:14.831 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:31:14.824+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:31:14.825339+0000) 2024-03-04T12:31:15.831 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:31:15.825+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:31:15.825579+0000) 2024-03-04T12:31:16.812 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:31:16.821 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:31:16.831 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:31:16.825+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:31:16.825813+0000) 2024-03-04T12:31:17.832 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:31:17.825+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:31:17.826162+0000) 2024-03-04T12:31:18.832 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:31:18.826+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:31:18.826421+0000) 2024-03-04T12:31:19.832 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:31:19.826+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:31:19.826669+0000) 2024-03-04T12:31:20.832 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:31:20.826+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:31:20.826942+0000) 2024-03-04T12:31:21.833 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:31:21.826+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:31:21.827206+0000) 2024-03-04T12:31:22.833 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:31:22.827+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:31:22.827496+0000) 2024-03-04T12:31:23.834 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:31:23.827+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:31:23.827732+0000) 2024-03-04T12:31:24.833 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:31:24.827+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:31:24.827979+0000) 2024-03-04T12:31:25.834 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:31:25.827+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:31:25.828223+0000) 2024-03-04T12:31:26.834 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:31:26.828+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:31:26.828463+0000) 2024-03-04T12:31:27.834 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:31:27.828+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:31:27.828704+0000) 2024-03-04T12:31:28.834 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:31:28.828+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:31:28.828958+0000) 2024-03-04T12:31:29.834 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:31:29.828+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:31:29.829201+0000) 2024-03-04T12:31:30.835 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:31:30.828+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:31:30.829394+0000) 2024-03-04T12:31:31.835 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:31:31.829+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:31:31.829628+0000) 2024-03-04T12:31:32.835 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:31:32.829+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:31:32.829902+0000) 2024-03-04T12:31:33.835 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:31:33.829+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:31:33.830138+0000) 2024-03-04T12:31:34.836 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:31:34.829+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:31:34.830335+0000) 2024-03-04T12:31:35.836 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:31:35.830+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:31:35.830568+0000) 2024-03-04T12:31:36.836 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:31:36.830+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:31:36.830805+0000) 2024-03-04T12:31:37.836 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:31:37.829+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:31:37.831087+0000) 2024-03-04T12:31:38.837 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:31:38.830+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:31:38.831332+0000) 2024-03-04T12:31:39.837 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:31:39.830+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:31:39.831580+0000) 2024-03-04T12:31:40.837 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:31:40.830+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:31:40.831853+0000) 2024-03-04T12:31:41.837 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:31:41.831+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:31:41.832093+0000) 2024-03-04T12:31:42.838 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:31:42.831+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:31:42.832327+0000) 2024-03-04T12:31:43.838 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:31:43.831+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:31:43.832598+0000) 2024-03-04T12:31:44.838 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:31:44.831+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:31:44.832867+0000) 2024-03-04T12:31:45.838 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:31:45.832+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:31:45.833105+0000) 2024-03-04T12:31:46.839 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:31:46.832+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:31:46.833344+0000) 2024-03-04T12:31:46.951 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:31:46.959 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:31:47.839 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:31:47.832+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:31:47.833608+0000) 2024-03-04T12:31:48.839 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:31:48.833+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:31:48.833915+0000) 2024-03-04T12:31:49.839 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:31:49.833+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:31:49.834153+0000) 2024-03-04T12:31:50.840 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:31:50.833+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:31:50.834398+0000) 2024-03-04T12:31:51.840 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:31:51.833+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:31:51.834634+0000) 2024-03-04T12:31:52.840 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:31:52.834+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:31:52.834911+0000) 2024-03-04T12:31:53.840 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:31:53.834+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:31:53.835153+0000) 2024-03-04T12:31:54.841 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:31:54.834+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:31:54.835427+0000) 2024-03-04T12:31:55.841 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:31:55.834+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:31:55.835671+0000) 2024-03-04T12:31:56.841 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:31:56.835+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:31:56.835896+0000) 2024-03-04T12:31:57.841 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:31:57.835+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:31:57.836141+0000) 2024-03-04T12:31:58.842 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:31:58.835+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:31:58.836385+0000) 2024-03-04T12:31:59.842 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:31:59.835+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:31:59.836634+0000) 2024-03-04T12:32:00.842 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:32:00.836+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:32:00.836931+0000) 2024-03-04T12:32:01.842 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:32:01.836+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:32:01.837170+0000) 2024-03-04T12:32:02.843 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:32:02.836+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:32:02.837406+0000) 2024-03-04T12:32:03.843 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:32:03.836+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:32:03.837674+0000) 2024-03-04T12:32:04.843 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:32:04.837+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:32:04.837953+0000) 2024-03-04T12:32:05.843 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:32:05.837+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:32:05.838233+0000) 2024-03-04T12:32:06.844 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:32:06.837+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:32:06.838476+0000) 2024-03-04T12:32:07.844 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:32:07.837+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:32:07.838663+0000) 2024-03-04T12:32:08.844 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:32:08.838+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:32:08.838955+0000) 2024-03-04T12:32:09.844 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:32:09.838+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:32:09.839220+0000) 2024-03-04T12:32:10.845 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:32:10.838+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:32:10.839467+0000) 2024-03-04T12:32:11.845 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:32:11.839+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:32:11.839740+0000) 2024-03-04T12:32:12.845 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:32:12.839+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:32:12.839992+0000) 2024-03-04T12:32:13.845 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:32:13.839+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:32:13.840229+0000) 2024-03-04T12:32:14.846 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:32:14.839+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:32:14.840473+0000) 2024-03-04T12:32:15.846 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:32:15.840+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:32:15.840716+0000) 2024-03-04T12:32:16.846 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:32:16.840+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:32:16.841013+0000) 2024-03-04T12:32:17.215 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:32:17.223 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:32:17.846 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:32:17.840+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:32:17.841246+0000) 2024-03-04T12:32:18.847 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:32:18.841+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:32:18.841511+0000) 2024-03-04T12:32:19.847 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:32:19.841+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:32:19.841753+0000) 2024-03-04T12:32:20.847 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:32:20.841+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:32:20.841998+0000) 2024-03-04T12:32:21.847 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:32:21.841+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:32:21.842240+0000) 2024-03-04T12:32:22.848 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:32:22.842+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:32:22.842474+0000) 2024-03-04T12:32:23.848 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:32:23.842+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:32:23.842720+0000) 2024-03-04T12:32:24.848 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:32:24.842+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:32:24.842983+0000) 2024-03-04T12:32:25.848 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:32:25.842+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:32:25.843225+0000) 2024-03-04T12:32:26.849 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:32:26.843+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:32:26.843473+0000) 2024-03-04T12:32:27.849 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:32:27.843+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:32:27.843690+0000) 2024-03-04T12:32:28.849 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:32:28.843+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:32:28.843955+0000) 2024-03-04T12:32:29.849 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:32:29.843+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:32:29.844204+0000) 2024-03-04T12:32:30.849 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:32:30.844+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:32:30.844465+0000) 2024-03-04T12:32:31.850 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:32:31.843+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:32:31.844705+0000) 2024-03-04T12:32:32.850 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:32:32.844+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:32:32.844944+0000) 2024-03-04T12:32:33.850 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:32:33.844+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:32:33.845182+0000) 2024-03-04T12:32:34.850 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:32:34.845+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:32:34.845428+0000) 2024-03-04T12:32:35.851 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:32:35.845+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:32:35.845669+0000) 2024-03-04T12:32:36.851 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:32:36.845+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:32:36.845952+0000) 2024-03-04T12:32:37.851 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:32:37.845+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:32:37.846208+0000) 2024-03-04T12:32:38.851 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:32:38.845+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:32:38.846447+0000) 2024-03-04T12:32:39.852 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:32:39.845+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:32:39.846690+0000) 2024-03-04T12:32:40.852 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:32:40.845+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:32:40.846935+0000) 2024-03-04T12:32:41.852 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:32:41.846+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:32:41.847183+0000) 2024-03-04T12:32:42.852 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:32:42.846+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:32:42.847430+0000) 2024-03-04T12:32:43.853 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:32:43.846+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:32:43.847670+0000) 2024-03-04T12:32:44.853 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:32:44.846+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:32:44.847934+0000) 2024-03-04T12:32:45.853 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:32:45.846+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:32:45.848165+0000) 2024-03-04T12:32:46.853 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:32:46.847+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:32:46.848407+0000) 2024-03-04T12:32:47.452 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:32:47.460 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:32:47.854 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:32:47.847+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:32:47.848599+0000) 2024-03-04T12:32:48.854 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:32:48.847+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:32:48.848912+0000) 2024-03-04T12:32:49.854 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:32:49.848+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:32:49.849110+0000) 2024-03-04T12:32:50.854 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:32:50.848+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:32:50.849308+0000) 2024-03-04T12:32:51.854 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:32:51.848+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:32:51.849500+0000) 2024-03-04T12:32:52.855 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:32:52.848+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:32:52.849701+0000) 2024-03-04T12:32:53.855 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:32:53.849+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:32:53.849948+0000) 2024-03-04T12:32:54.855 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:32:54.849+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:32:54.850248+0000) 2024-03-04T12:32:55.855 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:32:55.849+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:32:55.850564+0000) 2024-03-04T12:32:56.856 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:32:56.850+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:32:56.850880+0000) 2024-03-04T12:32:57.856 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:32:57.850+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:32:57.851207+0000) 2024-03-04T12:32:58.856 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:32:58.850+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:32:58.851491+0000) 2024-03-04T12:32:59.857 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:32:59.850+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:32:59.851790+0000) 2024-03-04T12:33:00.857 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:33:00.851+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:33:00.852155+0000) 2024-03-04T12:33:01.857 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:33:01.851+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:33:01.852471+0000) 2024-03-04T12:33:02.858 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:33:02.851+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:33:02.852789+0000) 2024-03-04T12:33:03.858 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:33:03.852+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:33:03.853118+0000) 2024-03-04T12:33:04.858 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:33:04.852+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:33:04.853401+0000) 2024-03-04T12:33:05.859 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:33:05.852+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:33:05.853701+0000) 2024-03-04T12:33:06.859 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:33:06.853+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:33:06.853992+0000) 2024-03-04T12:33:07.859 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:33:07.853+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:33:07.854286+0000) 2024-03-04T12:33:08.859 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:33:08.853+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:33:08.854569+0000) 2024-03-04T12:33:09.860 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:33:09.854+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:33:09.854895+0000) 2024-03-04T12:33:10.860 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:33:10.854+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:33:10.855215+0000) 2024-03-04T12:33:11.861 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:33:11.854+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:33:11.855494+0000) 2024-03-04T12:33:12.861 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:33:12.855+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:33:12.855783+0000) 2024-03-04T12:33:13.861 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:33:13.855+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:33:13.856115+0000) 2024-03-04T12:33:14.861 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:33:14.855+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:33:14.856459+0000) 2024-03-04T12:33:15.862 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:33:15.856+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:33:15.856755+0000) 2024-03-04T12:33:16.862 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:33:16.856+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:33:16.857048+0000) 2024-03-04T12:33:17.666 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:33:17.674 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:33:17.862 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:33:17.856+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:33:17.857313+0000) 2024-03-04T12:33:18.862 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:33:18.856+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:33:18.857571+0000) 2024-03-04T12:33:19.863 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:33:19.857+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:33:19.857803+0000) 2024-03-04T12:33:20.863 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:33:20.857+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:33:20.858114+0000) 2024-03-04T12:33:21.863 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:33:21.857+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:33:21.858407+0000) 2024-03-04T12:33:22.864 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:33:22.857+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:33:22.858715+0000) 2024-03-04T12:33:23.864 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:33:23.858+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:33:23.859004+0000) 2024-03-04T12:33:24.864 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:33:24.858+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:33:24.859288+0000) 2024-03-04T12:33:25.864 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:33:25.859+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:33:25.859585+0000) 2024-03-04T12:33:26.865 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:33:26.859+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:33:26.859917+0000) 2024-03-04T12:33:27.865 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:33:27.859+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:33:27.860231+0000) 2024-03-04T12:33:28.865 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:33:28.859+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:33:28.860526+0000) 2024-03-04T12:33:29.865 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:33:29.860+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:33:29.860769+0000) 2024-03-04T12:33:30.866 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:33:30.860+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:33:30.861091+0000) 2024-03-04T12:33:31.866 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:33:31.860+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:33:31.861333+0000) 2024-03-04T12:33:32.866 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:33:32.860+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:33:32.861608+0000) 2024-03-04T12:33:33.867 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:33:33.861+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:33:33.861943+0000) 2024-03-04T12:33:34.867 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:33:34.861+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:33:34.862235+0000) 2024-03-04T12:33:35.867 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:33:35.862+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:33:35.862524+0000) 2024-03-04T12:33:36.868 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:33:36.862+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:33:36.862870+0000) 2024-03-04T12:33:37.868 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:33:37.862+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:33:37.863178+0000) 2024-03-04T12:33:38.868 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:33:38.863+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:33:38.863456+0000) 2024-03-04T12:33:39.869 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:33:39.862+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:33:39.863767+0000) 2024-03-04T12:33:40.869 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:33:40.862+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:33:40.864073+0000) 2024-03-04T12:33:41.869 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:33:41.862+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:33:41.864301+0000) 2024-03-04T12:33:42.869 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:33:42.863+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:33:42.864597+0000) 2024-03-04T12:33:43.870 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:33:43.863+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:33:43.864875+0000) 2024-03-04T12:33:44.870 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:33:44.863+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:33:44.865123+0000) 2024-03-04T12:33:45.870 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:33:45.864+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:33:45.865408+0000) 2024-03-04T12:33:46.870 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:33:46.864+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:33:46.865716+0000) 2024-03-04T12:33:47.871 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:33:47.872 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:33:47.865+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:33:47.866050+0000) 2024-03-04T12:33:47.879 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:33:48.871 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:33:48.865+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:33:48.866350+0000) 2024-03-04T12:33:49.872 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:33:49.864+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:33:49.866635+0000) 2024-03-04T12:33:50.872 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:33:50.866+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:33:50.866978+0000) 2024-03-04T12:33:51.872 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:33:51.866+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:33:51.867275+0000) 2024-03-04T12:33:52.872 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:33:52.866+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:33:52.867554+0000) 2024-03-04T12:33:53.873 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:33:53.867+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:33:53.867863+0000) 2024-03-04T12:33:54.873 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:33:54.867+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:33:54.868159+0000) 2024-03-04T12:33:55.873 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:33:55.867+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:33:55.868408+0000) 2024-03-04T12:33:56.873 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:33:56.867+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:33:56.868689+0000) 2024-03-04T12:33:57.874 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:33:57.868+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:33:57.869014+0000) 2024-03-04T12:33:58.874 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:33:58.868+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:33:58.869310+0000) 2024-03-04T12:33:59.874 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:33:59.868+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:33:59.869603+0000) 2024-03-04T12:34:00.875 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:34:00.869+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:34:00.869947+0000) 2024-03-04T12:34:01.875 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:34:01.869+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:34:01.870239+0000) 2024-03-04T12:34:02.875 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:34:02.869+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:34:02.870463+0000) 2024-03-04T12:34:03.875 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:34:03.869+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:34:03.870754+0000) 2024-03-04T12:34:04.876 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:34:04.870+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:34:04.871036+0000) 2024-03-04T12:34:05.876 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:34:05.870+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:34:05.871246+0000) 2024-03-04T12:34:06.876 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:34:06.870+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:34:06.871433+0000) 2024-03-04T12:34:07.876 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:34:07.871+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:34:07.871705+0000) 2024-03-04T12:34:08.877 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:34:08.871+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:34:08.871996+0000) 2024-03-04T12:34:09.877 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:34:09.871+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:34:09.872241+0000) 2024-03-04T12:34:10.877 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:34:10.871+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:34:10.872443+0000) 2024-03-04T12:34:11.877 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:34:11.872+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:34:11.872678+0000) 2024-03-04T12:34:12.878 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:34:12.872+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:34:12.872949+0000) 2024-03-04T12:34:13.878 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:34:13.872+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:34:13.873137+0000) 2024-03-04T12:34:14.878 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:34:14.872+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:34:14.873384+0000) 2024-03-04T12:34:15.878 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:34:15.873+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:34:15.873630+0000) 2024-03-04T12:34:16.879 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:34:16.873+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:34:16.873941+0000) 2024-03-04T12:34:17.879 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:34:17.873+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:34:17.874223+0000) 2024-03-04T12:34:18.009 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:34:18.018 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:34:18.879 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:34:18.873+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:34:18.874469+0000) 2024-03-04T12:34:19.879 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:34:19.874+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:34:19.874715+0000) 2024-03-04T12:34:20.880 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:34:20.874+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:34:20.874956+0000) 2024-03-04T12:34:21.880 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:34:21.874+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:34:21.875193+0000) 2024-03-04T12:34:22.880 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:34:22.875+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:34:22.875430+0000) 2024-03-04T12:34:23.880 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:34:23.875+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:34:23.875664+0000) 2024-03-04T12:34:24.881 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:34:24.875+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:34:24.875951+0000) 2024-03-04T12:34:25.881 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:34:25.875+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:34:25.876190+0000) 2024-03-04T12:34:26.881 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:34:26.876+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:34:26.876463+0000) 2024-03-04T12:34:27.881 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:34:27.876+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:34:27.876766+0000) 2024-03-04T12:34:28.882 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:34:28.876+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:34:28.877045+0000) 2024-03-04T12:34:29.882 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:34:29.876+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:34:29.877343+0000) 2024-03-04T12:34:30.882 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:34:30.877+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:34:30.877644+0000) 2024-03-04T12:34:31.883 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:34:31.877+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:34:31.877971+0000) 2024-03-04T12:34:32.883 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:34:32.877+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:34:32.878222+0000) 2024-03-04T12:34:33.883 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:34:33.878+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:34:33.878492+0000) 2024-03-04T12:34:34.883 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:34:34.878+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:34:34.878729+0000) 2024-03-04T12:34:35.884 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:34:35.878+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:34:35.878990+0000) 2024-03-04T12:34:36.884 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:34:36.879+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:34:36.879249+0000) 2024-03-04T12:34:37.884 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:34:37.879+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:34:37.879499+0000) 2024-03-04T12:34:38.884 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:34:38.879+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:34:38.879739+0000) 2024-03-04T12:34:39.885 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:34:39.879+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:34:39.879984+0000) 2024-03-04T12:34:40.885 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:34:40.880+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:34:40.880238+0000) 2024-03-04T12:34:41.885 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:34:41.879+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:34:41.880475+0000) 2024-03-04T12:34:42.885 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:34:42.879+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:34:42.880710+0000) 2024-03-04T12:34:43.885 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:34:43.879+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:34:43.880944+0000) 2024-03-04T12:34:44.886 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:34:44.880+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:34:44.881188+0000) 2024-03-04T12:34:45.886 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:34:45.880+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:34:45.881437+0000) 2024-03-04T12:34:46.886 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:34:46.880+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:34:46.881680+0000) 2024-03-04T12:34:47.886 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:34:47.880+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:34:47.881930+0000) 2024-03-04T12:34:48.149 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:34:48.157 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:34:48.887 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:34:48.881+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:34:48.882202+0000) 2024-03-04T12:34:49.887 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:34:49.881+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:34:49.882448+0000) 2024-03-04T12:34:50.887 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:34:50.881+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:34:50.882691+0000) 2024-03-04T12:34:51.887 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:34:51.882+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:34:51.882931+0000) 2024-03-04T12:34:52.888 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:34:52.882+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:34:52.883166+0000) 2024-03-04T12:34:53.888 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:34:53.882+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:34:53.883398+0000) 2024-03-04T12:34:54.891 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:34:54.882+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:34:54.883663+0000) 2024-03-04T12:34:55.889 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:34:55.883+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:34:55.883939+0000) 2024-03-04T12:34:56.889 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:34:56.883+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:34:56.884134+0000) 2024-03-04T12:34:57.889 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:34:57.883+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:34:57.884377+0000) 2024-03-04T12:34:58.889 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:34:58.883+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:34:58.884611+0000) 2024-03-04T12:34:59.889 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:34:59.884+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:34:59.884918+0000) 2024-03-04T12:35:00.890 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:35:00.884+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:35:00.885185+0000) 2024-03-04T12:35:01.890 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:35:01.884+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:35:01.885422+0000) 2024-03-04T12:35:02.890 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:35:02.884+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:35:02.885658+0000) 2024-03-04T12:35:03.890 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:35:03.885+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:35:03.885955+0000) 2024-03-04T12:35:04.891 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:35:04.885+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:35:04.886209+0000) 2024-03-04T12:35:05.891 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:35:05.885+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:35:05.886482+0000) 2024-03-04T12:35:06.891 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:35:06.886+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:35:06.886748+0000) 2024-03-04T12:35:07.892 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:35:07.886+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:35:07.887009+0000) 2024-03-04T12:35:08.892 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:35:08.886+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:35:08.887247+0000) 2024-03-04T12:35:09.892 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:35:09.886+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:35:09.887497+0000) 2024-03-04T12:35:10.892 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:35:10.886+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:35:10.887771+0000) 2024-03-04T12:35:11.893 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:35:11.887+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:35:11.888010+0000) 2024-03-04T12:35:12.893 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:35:12.887+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:35:12.888248+0000) 2024-03-04T12:35:13.893 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:35:13.887+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:35:13.888444+0000) 2024-03-04T12:35:14.893 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:35:14.888+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:35:14.888694+0000) 2024-03-04T12:35:15.893 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:35:15.888+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:35:15.888932+0000) 2024-03-04T12:35:16.894 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:35:16.888+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:35:16.889174+0000) 2024-03-04T12:35:17.894 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:35:17.888+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:35:17.889417+0000) 2024-03-04T12:35:18.327 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:35:18.339 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:35:18.894 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:35:18.888+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:35:18.889649+0000) 2024-03-04T12:35:19.894 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:35:19.889+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:35:19.889970+0000) 2024-03-04T12:35:20.895 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:35:20.889+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:35:20.890225+0000) 2024-03-04T12:35:21.895 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:35:21.890+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:35:21.890483+0000) 2024-03-04T12:35:22.895 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:35:22.889+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:35:22.890735+0000) 2024-03-04T12:35:23.895 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:35:23.890+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:35:23.890970+0000) 2024-03-04T12:35:24.896 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:35:24.890+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:35:24.891171+0000) 2024-03-04T12:35:25.896 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:35:25.890+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:35:25.891430+0000) 2024-03-04T12:35:26.896 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:35:26.891+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:35:26.891666+0000) 2024-03-04T12:35:27.896 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:35:27.891+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:35:27.891956+0000) 2024-03-04T12:35:28.897 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:35:28.891+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:35:28.892202+0000) 2024-03-04T12:35:29.897 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:35:29.892+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:35:29.892482+0000) 2024-03-04T12:35:30.897 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:35:30.892+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:35:30.892729+0000) 2024-03-04T12:35:31.899 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:35:31.892+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:35:31.892969+0000) 2024-03-04T12:35:32.898 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:35:32.892+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:35:32.893204+0000) 2024-03-04T12:35:33.898 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:35:33.893+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:35:33.893443+0000) 2024-03-04T12:35:34.898 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:35:34.892+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:35:34.893678+0000) 2024-03-04T12:35:35.898 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:35:35.893+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:35:35.893957+0000) 2024-03-04T12:35:36.899 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:35:36.893+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:35:36.894219+0000) 2024-03-04T12:35:37.899 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:35:37.893+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:35:37.894407+0000) 2024-03-04T12:35:38.900 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:35:38.893+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:35:38.894601+0000) 2024-03-04T12:35:39.899 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:35:39.894+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:35:39.894898+0000) 2024-03-04T12:35:40.900 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:35:40.894+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:35:40.895152+0000) 2024-03-04T12:35:41.900 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:35:41.895+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:35:41.895414+0000) 2024-03-04T12:35:42.900 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:35:42.895+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:35:42.895646+0000) 2024-03-04T12:35:43.900 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:35:43.894+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:35:43.895922+0000) 2024-03-04T12:35:44.901 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:35:44.895+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:35:44.896251+0000) 2024-03-04T12:35:45.901 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:35:45.895+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:35:45.896504+0000) 2024-03-04T12:35:46.901 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:35:46.895+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:35:46.896744+0000) 2024-03-04T12:35:47.901 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:35:47.895+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:35:47.897001+0000) 2024-03-04T12:35:48.491 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:35:48.499 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:35:48.902 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:35:48.896+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:35:48.897245+0000) 2024-03-04T12:35:49.902 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:35:49.896+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:35:49.897489+0000) 2024-03-04T12:35:50.902 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:35:50.896+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:35:50.897744+0000) 2024-03-04T12:35:51.902 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:35:51.896+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:35:51.897982+0000) 2024-03-04T12:35:52.903 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:35:52.897+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:35:52.898219+0000) 2024-03-04T12:35:53.903 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:35:53.897+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:35:53.898487+0000) 2024-03-04T12:35:54.903 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:35:54.897+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:35:54.898727+0000) 2024-03-04T12:35:55.903 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:35:55.898+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:35:55.899009+0000) 2024-03-04T12:35:56.904 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:35:56.898+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:35:56.899249+0000) 2024-03-04T12:35:57.904 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:35:57.898+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:35:57.899500+0000) 2024-03-04T12:35:58.904 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:35:58.898+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:35:58.899737+0000) 2024-03-04T12:35:59.904 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:35:59.899+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:35:59.899973+0000) 2024-03-04T12:36:00.905 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:36:00.899+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:36:00.900262+0000) 2024-03-04T12:36:01.905 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:36:01.899+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:36:01.900508+0000) 2024-03-04T12:36:02.905 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:36:02.899+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:36:02.900744+0000) 2024-03-04T12:36:03.906 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:36:03.900+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:36:03.900983+0000) 2024-03-04T12:36:04.906 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:36:04.900+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:36:04.901222+0000) 2024-03-04T12:36:05.906 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:36:05.900+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:36:05.901495+0000) 2024-03-04T12:36:06.906 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:36:06.901+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:36:06.901758+0000) 2024-03-04T12:36:07.906 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:36:07.901+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:36:07.902023+0000) 2024-03-04T12:36:08.907 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:36:08.901+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:36:08.902262+0000) 2024-03-04T12:36:09.907 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:36:09.901+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:36:09.902498+0000) 2024-03-04T12:36:10.907 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:36:10.902+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:36:10.902751+0000) 2024-03-04T12:36:11.907 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:36:11.902+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:36:11.902995+0000) 2024-03-04T12:36:12.908 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:36:12.902+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:36:12.903232+0000) 2024-03-04T12:36:13.908 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:36:13.902+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:36:13.903474+0000) 2024-03-04T12:36:14.908 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:36:14.902+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:36:14.903721+0000) 2024-03-04T12:36:15.908 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:36:15.903+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:36:15.904000+0000) 2024-03-04T12:36:16.908 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:36:16.903+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:36:16.904247+0000) 2024-03-04T12:36:17.909 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:36:17.903+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:36:17.904496+0000) 2024-03-04T12:36:18.619 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:36:18.631 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:36:18.909 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:36:18.904+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:36:18.904744+0000) 2024-03-04T12:36:19.909 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:36:19.904+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:36:19.904941+0000) 2024-03-04T12:36:20.909 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:36:20.904+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:36:20.905180+0000) 2024-03-04T12:36:21.910 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:36:21.904+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:36:21.905418+0000) 2024-03-04T12:36:22.910 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:36:22.905+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:36:22.905654+0000) 2024-03-04T12:36:23.910 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:36:23.905+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:36:23.905926+0000) 2024-03-04T12:36:24.910 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:36:24.905+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:36:24.906201+0000) 2024-03-04T12:36:25.911 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:36:25.905+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:36:25.906462+0000) 2024-03-04T12:36:26.911 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:36:26.906+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:36:26.906754+0000) 2024-03-04T12:36:27.911 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:36:27.906+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:36:27.907061+0000) 2024-03-04T12:36:28.912 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:36:28.906+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:36:28.907366+0000) 2024-03-04T12:36:29.912 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:36:29.907+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:36:29.907657+0000) 2024-03-04T12:36:30.912 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:36:30.907+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:36:30.907985+0000) 2024-03-04T12:36:31.913 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:36:31.907+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:36:31.908271+0000) 2024-03-04T12:36:32.913 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:36:32.907+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:36:32.908507+0000) 2024-03-04T12:36:33.913 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:36:33.908+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:36:33.908802+0000) 2024-03-04T12:36:34.913 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:36:34.908+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:36:34.909152+0000) 2024-03-04T12:36:35.914 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:36:35.909+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:36:35.909433+0000) 2024-03-04T12:36:36.914 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:36:36.909+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:36:36.909677+0000) 2024-03-04T12:36:37.914 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:36:37.909+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:36:37.909948+0000) 2024-03-04T12:36:38.914 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:36:38.909+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:36:38.910190+0000) 2024-03-04T12:36:39.915 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:36:39.910+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:36:39.910431+0000) 2024-03-04T12:36:40.915 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:36:40.910+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:36:40.910677+0000) 2024-03-04T12:36:41.915 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:36:41.910+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:36:41.910965+0000) 2024-03-04T12:36:42.916 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:36:42.911+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:36:42.911263+0000) 2024-03-04T12:36:43.916 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:36:43.911+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:36:43.911500+0000) 2024-03-04T12:36:44.916 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:36:44.910+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:36:44.911740+0000) 2024-03-04T12:36:45.916 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:36:45.910+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:36:45.912016+0000) 2024-03-04T12:36:46.917 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:36:46.911+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:36:46.912264+0000) 2024-03-04T12:36:47.917 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:36:47.911+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:36:47.912511+0000) 2024-03-04T12:36:48.825 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:36:48.833 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:36:48.917 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:36:48.910+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:36:48.912694+0000) 2024-03-04T12:36:49.917 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:36:49.911+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:36:49.912935+0000) 2024-03-04T12:36:50.917 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:36:50.911+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:36:50.913141+0000) 2024-03-04T12:36:51.918 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:36:51.912+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:36:51.913382+0000) 2024-03-04T12:36:52.918 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:36:52.912+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:36:52.913623+0000) 2024-03-04T12:36:53.918 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:36:53.912+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:36:53.913906+0000) 2024-03-04T12:36:54.918 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:36:54.913+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:36:54.914121+0000) 2024-03-04T12:36:55.919 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:36:55.913+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:36:55.914363+0000) 2024-03-04T12:36:56.919 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:36:56.913+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:36:56.914610+0000) 2024-03-04T12:36:57.919 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:36:57.914+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:36:57.914913+0000) 2024-03-04T12:36:58.920 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:36:58.914+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:36:58.915182+0000) 2024-03-04T12:36:59.920 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:36:59.914+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:36:59.915417+0000) 2024-03-04T12:37:00.920 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:37:00.914+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:37:00.915660+0000) 2024-03-04T12:37:01.920 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:37:01.915+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:37:01.915954+0000) 2024-03-04T12:37:02.921 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:37:02.915+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:37:02.916191+0000) 2024-03-04T12:37:03.921 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:37:03.915+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:37:03.916433+0000) 2024-03-04T12:37:04.921 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:37:04.915+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:37:04.916706+0000) 2024-03-04T12:37:05.921 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:37:05.916+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:37:05.916973+0000) 2024-03-04T12:37:06.921 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:37:06.916+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:37:06.917171+0000) 2024-03-04T12:37:07.922 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:37:07.916+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:37:07.917433+0000) 2024-03-04T12:37:08.922 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:37:08.916+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:37:08.917677+0000) 2024-03-04T12:37:09.922 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:37:09.917+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:37:09.917954+0000) 2024-03-04T12:37:10.922 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:37:10.917+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:37:10.918198+0000) 2024-03-04T12:37:11.923 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:37:11.917+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:37:11.918439+0000) 2024-03-04T12:37:12.923 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:37:12.918+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:37:12.918681+0000) 2024-03-04T12:37:13.923 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:37:13.918+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:37:13.918922+0000) 2024-03-04T12:37:14.923 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:37:14.918+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:37:14.919169+0000) 2024-03-04T12:37:15.924 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:37:15.918+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:37:15.919420+0000) 2024-03-04T12:37:16.924 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:37:16.918+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:37:16.919680+0000) 2024-03-04T12:37:17.924 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:37:17.919+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:37:17.919929+0000) 2024-03-04T12:37:18.924 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:37:18.919+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:37:18.920170+0000) 2024-03-04T12:37:19.056 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:37:19.064 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:37:19.925 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:37:19.919+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:37:19.920366+0000) 2024-03-04T12:37:20.925 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:37:20.919+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:37:20.920634+0000) 2024-03-04T12:37:21.925 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:37:21.920+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:37:21.920944+0000) 2024-03-04T12:37:22.925 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:37:22.920+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:37:22.921178+0000) 2024-03-04T12:37:23.926 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:37:23.920+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:37:23.921438+0000) 2024-03-04T12:37:24.926 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:37:24.920+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:37:24.921701+0000) 2024-03-04T12:37:25.926 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:37:25.921+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:37:25.921978+0000) 2024-03-04T12:37:26.926 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:37:26.921+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:37:26.922216+0000) 2024-03-04T12:37:27.927 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:37:27.922+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:37:27.922461+0000) 2024-03-04T12:37:28.927 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:37:28.921+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:37:28.922722+0000) 2024-03-04T12:37:29.927 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:37:29.922+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:37:29.922988+0000) 2024-03-04T12:37:30.927 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:37:30.922+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:37:30.923243+0000) 2024-03-04T12:37:31.928 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:37:31.922+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:37:31.923500+0000) 2024-03-04T12:37:32.928 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:37:32.922+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:37:32.923733+0000) 2024-03-04T12:37:33.928 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:37:33.923+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:37:33.923973+0000) 2024-03-04T12:37:34.928 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:37:34.923+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:37:34.924200+0000) 2024-03-04T12:37:35.929 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:37:35.924+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:37:35.924523+0000) 2024-03-04T12:37:36.929 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:37:36.924+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:37:36.924804+0000) 2024-03-04T12:37:37.929 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:37:37.924+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:37:37.925186+0000) 2024-03-04T12:37:38.930 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:37:38.925+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:37:38.925475+0000) 2024-03-04T12:37:39.930 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:37:39.924+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:37:39.925765+0000) 2024-03-04T12:37:40.930 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:37:40.925+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:37:40.926052+0000) 2024-03-04T12:37:41.930 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:37:41.926+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:37:41.926344+0000) 2024-03-04T12:37:42.931 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:37:42.926+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:37:42.926631+0000) 2024-03-04T12:37:43.931 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:37:43.926+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:37:43.926962+0000) 2024-03-04T12:37:44.931 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:37:44.927+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:37:44.927253+0000) 2024-03-04T12:37:45.932 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:37:45.927+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:37:45.927573+0000) 2024-03-04T12:37:46.932 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:37:46.927+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:37:46.927863+0000) 2024-03-04T12:37:47.932 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:37:47.927+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:37:47.928178+0000) 2024-03-04T12:37:48.933 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:37:48.927+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:37:48.928470+0000) 2024-03-04T12:37:49.286 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:37:49.295 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:37:49.933 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:37:49.927+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:37:49.928753+0000) 2024-03-04T12:37:50.933 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:37:50.927+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:37:50.929051+0000) 2024-03-04T12:37:51.934 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:37:51.927+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:37:51.929329+0000) 2024-03-04T12:37:52.934 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:37:52.928+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:37:52.929580+0000) 2024-03-04T12:37:53.934 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:37:53.928+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:37:53.929891+0000) 2024-03-04T12:37:54.934 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:37:54.929+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:37:54.930202+0000) 2024-03-04T12:37:55.935 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:37:55.928+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:37:55.930513+0000) 2024-03-04T12:37:56.935 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:37:56.929+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:37:56.930792+0000) 2024-03-04T12:37:57.935 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:37:57.930+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:37:57.931134+0000) 2024-03-04T12:37:58.935 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:37:58.930+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:37:58.931412+0000) 2024-03-04T12:37:59.936 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:37:59.930+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:37:59.931696+0000) 2024-03-04T12:38:00.936 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:38:00.931+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:38:00.931997+0000) 2024-03-04T12:38:01.936 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:38:01.931+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:38:01.932304+0000) 2024-03-04T12:38:02.937 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:38:02.931+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:38:02.932583+0000) 2024-03-04T12:38:03.937 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:38:03.932+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:38:03.932920+0000) 2024-03-04T12:38:04.937 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:38:04.932+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:38:04.933197+0000) 2024-03-04T12:38:05.938 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:38:05.932+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:38:05.933451+0000) 2024-03-04T12:38:06.938 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:38:06.932+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:38:06.933635+0000) 2024-03-04T12:38:07.938 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:38:07.933+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:38:07.933923+0000) 2024-03-04T12:38:08.938 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:38:08.933+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:38:08.934160+0000) 2024-03-04T12:38:09.938 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:38:09.933+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:38:09.934393+0000) 2024-03-04T12:38:10.939 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:38:10.933+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:38:10.934635+0000) 2024-03-04T12:38:11.939 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:38:11.934+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:38:11.934919+0000) 2024-03-04T12:38:12.939 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:38:12.934+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:38:12.935151+0000) 2024-03-04T12:38:13.940 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:38:13.934+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:38:13.935423+0000) 2024-03-04T12:38:14.940 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:38:14.934+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:38:14.935710+0000) 2024-03-04T12:38:15.940 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:38:15.935+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:38:15.935974+0000) 2024-03-04T12:38:16.940 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:38:16.935+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:38:16.936187+0000) 2024-03-04T12:38:17.941 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:38:17.935+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:38:17.936428+0000) 2024-03-04T12:38:18.941 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:38:18.936+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:38:18.936669+0000) 2024-03-04T12:38:19.482 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:38:19.490 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:38:19.941 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:38:19.936+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:38:19.936979+0000) 2024-03-04T12:38:20.941 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:38:20.936+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:38:20.937262+0000) 2024-03-04T12:38:21.942 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:38:21.936+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:38:21.937555+0000) 2024-03-04T12:38:22.942 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:38:22.937+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:38:22.937867+0000) 2024-03-04T12:38:23.942 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:38:23.937+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:38:23.938118+0000) 2024-03-04T12:38:24.942 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:38:24.937+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:38:24.938358+0000) 2024-03-04T12:38:25.943 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:38:25.938+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:38:25.938597+0000) 2024-03-04T12:38:26.943 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:38:26.938+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:38:26.938873+0000) 2024-03-04T12:38:27.943 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:38:27.938+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:38:27.939120+0000) 2024-03-04T12:38:28.944 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:38:28.938+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:38:28.939355+0000) 2024-03-04T12:38:29.944 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:38:29.938+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:38:29.939604+0000) 2024-03-04T12:38:30.944 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:38:30.939+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:38:30.939911+0000) 2024-03-04T12:38:31.944 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:38:31.939+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:38:31.940106+0000) 2024-03-04T12:38:32.944 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:38:32.939+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:38:32.940341+0000) 2024-03-04T12:38:33.945 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:38:33.940+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:38:33.940598+0000) 2024-03-04T12:38:34.945 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:38:34.940+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:38:34.940973+0000) 2024-03-04T12:38:35.945 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:38:35.940+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:38:35.941169+0000) 2024-03-04T12:38:36.945 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:38:36.941+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:38:36.941444+0000) 2024-03-04T12:38:37.946 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:38:37.941+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:38:37.941688+0000) 2024-03-04T12:38:38.946 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:38:38.941+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:38:38.941932+0000) 2024-03-04T12:38:39.946 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:38:39.941+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:38:39.942202+0000) 2024-03-04T12:38:40.947 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:38:40.942+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:38:40.942444+0000) 2024-03-04T12:38:41.947 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:38:41.942+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:38:41.942682+0000) 2024-03-04T12:38:42.947 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:38:42.942+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:38:42.942949+0000) 2024-03-04T12:38:43.948 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:38:43.942+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:38:43.943191+0000) 2024-03-04T12:38:44.947 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:38:44.943+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:38:44.943379+0000) 2024-03-04T12:38:45.948 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:38:45.943+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:38:45.943629+0000) 2024-03-04T12:38:46.948 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:38:46.943+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:38:46.943930+0000) 2024-03-04T12:38:47.948 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:38:47.944+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:38:47.944193+0000) 2024-03-04T12:38:48.949 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:38:48.943+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:38:48.944424+0000) 2024-03-04T12:38:49.671 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:38:49.679 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:38:49.949 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:38:49.943+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:38:49.944675+0000) 2024-03-04T12:38:50.949 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:38:50.943+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:38:50.944944+0000) 2024-03-04T12:38:51.949 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:38:51.944+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:38:51.945183+0000) 2024-03-04T12:38:52.949 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:38:52.944+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:38:52.945423+0000) 2024-03-04T12:38:53.950 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:38:53.944+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:38:53.945659+0000) 2024-03-04T12:38:54.950 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:38:54.944+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:38:54.945941+0000) 2024-03-04T12:38:55.950 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:38:55.945+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:38:55.946180+0000) 2024-03-04T12:38:56.950 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:38:56.945+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:38:56.946424+0000) 2024-03-04T12:38:57.951 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:38:57.945+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:38:57.946670+0000) 2024-03-04T12:38:58.951 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:38:58.945+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:38:58.946970+0000) 2024-03-04T12:38:59.951 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:38:59.946+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:38:59.947227+0000) 2024-03-04T12:39:00.952 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:39:00.946+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:39:00.947485+0000) 2024-03-04T12:39:01.952 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:39:01.946+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:39:01.947712+0000) 2024-03-04T12:39:02.952 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:39:02.947+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:39:02.947972+0000) 2024-03-04T12:39:03.952 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:39:03.947+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:39:03.948167+0000) 2024-03-04T12:39:04.952 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:39:04.947+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:39:04.948361+0000) 2024-03-04T12:39:05.953 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:39:05.947+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:39:05.948612+0000) 2024-03-04T12:39:06.953 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:39:06.948+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:39:06.948902+0000) 2024-03-04T12:39:07.953 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:39:07.948+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:39:07.949151+0000) 2024-03-04T12:39:08.953 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:39:08.948+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:39:08.949396+0000) 2024-03-04T12:39:09.954 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:39:09.948+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:39:09.949631+0000) 2024-03-04T12:39:10.954 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:39:10.949+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:39:10.949873+0000) 2024-03-04T12:39:11.954 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:39:11.949+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:39:11.950140+0000) 2024-03-04T12:39:12.954 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:39:12.949+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:39:12.950377+0000) 2024-03-04T12:39:13.955 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:39:13.949+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:39:13.950621+0000) 2024-03-04T12:39:14.955 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:39:14.950+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:39:14.950850+0000) 2024-03-04T12:39:15.955 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:39:15.950+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:39:15.951098+0000) 2024-03-04T12:39:16.955 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:39:16.950+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:39:16.951342+0000) 2024-03-04T12:39:17.956 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:39:17.950+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:39:17.951593+0000) 2024-03-04T12:39:18.956 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:39:18.951+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:39:18.951875+0000) 2024-03-04T12:39:19.893 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:39:19.901 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:39:19.956 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:39:19.951+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:39:19.952071+0000) 2024-03-04T12:39:20.956 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:39:20.951+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:39:20.952324+0000) 2024-03-04T12:39:21.957 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:39:21.951+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:39:21.952564+0000) 2024-03-04T12:39:22.957 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:39:22.952+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:39:22.952921+0000) 2024-03-04T12:39:23.957 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:39:23.952+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:39:23.953161+0000) 2024-03-04T12:39:24.957 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:39:24.952+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:39:24.953402+0000) 2024-03-04T12:39:25.958 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:39:25.953+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:39:25.953647+0000) 2024-03-04T12:39:26.958 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:39:26.953+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:39:26.953949+0000) 2024-03-04T12:39:27.958 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:39:27.953+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:39:27.954188+0000) 2024-03-04T12:39:28.958 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:39:28.953+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:39:28.954465+0000) 2024-03-04T12:39:29.959 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:39:29.954+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:39:29.954778+0000) 2024-03-04T12:39:30.959 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:39:30.954+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:39:30.955028+0000) 2024-03-04T12:39:31.959 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:39:31.954+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:39:31.955313+0000) 2024-03-04T12:39:32.960 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:39:32.954+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:39:32.955539+0000) 2024-03-04T12:39:33.960 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:39:33.954+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:39:33.955775+0000) 2024-03-04T12:39:34.960 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:39:34.955+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:39:34.956030+0000) 2024-03-04T12:39:35.960 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:39:35.955+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:39:35.956311+0000) 2024-03-04T12:39:36.961 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:39:36.956+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:39:36.956564+0000) 2024-03-04T12:39:37.961 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:39:37.956+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:39:37.956811+0000) 2024-03-04T12:39:38.961 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:39:38.956+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:39:38.957005+0000) 2024-03-04T12:39:39.961 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:39:39.956+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:39:39.957240+0000) 2024-03-04T12:39:40.962 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:39:40.957+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:39:40.957491+0000) 2024-03-04T12:39:41.962 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:39:41.957+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:39:41.957738+0000) 2024-03-04T12:39:42.962 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:39:42.957+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:39:42.957970+0000) 2024-03-04T12:39:43.962 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:39:43.957+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:39:43.958205+0000) 2024-03-04T12:39:44.962 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:39:44.957+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:39:44.958462+0000) 2024-03-04T12:39:45.963 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:39:45.958+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:39:45.958711+0000) 2024-03-04T12:39:46.963 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:39:46.958+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:39:46.958981+0000) 2024-03-04T12:39:47.963 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:39:47.959+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:39:47.959245+0000) 2024-03-04T12:39:48.963 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:39:48.959+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:39:48.959489+0000) 2024-03-04T12:39:49.964 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:39:49.959+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:39:49.959726+0000) 2024-03-04T12:39:50.072 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:39:50.080 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:39:50.964 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:39:50.959+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:39:50.959996+0000) 2024-03-04T12:39:51.964 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:39:51.959+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:39:51.960262+0000) 2024-03-04T12:39:52.964 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:39:52.959+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:39:52.960498+0000) 2024-03-04T12:39:53.965 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:39:53.958+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:39:53.960695+0000) 2024-03-04T12:39:54.965 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:39:54.959+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:39:54.960958+0000) 2024-03-04T12:39:55.965 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:39:55.959+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:39:55.961192+0000) 2024-03-04T12:39:56.965 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:39:56.960+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:39:56.961449+0000) 2024-03-04T12:39:57.966 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:39:57.959+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:39:57.961685+0000) 2024-03-04T12:39:58.966 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:39:58.960+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:39:58.961978+0000) 2024-03-04T12:39:59.967 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:39:59.961+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:39:59.962290+0000) 2024-03-04T12:40:00.967 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:40:00.961+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:40:00.962629+0000) 2024-03-04T12:40:01.967 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:40:01.962+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:40:01.962949+0000) 2024-03-04T12:40:02.967 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:40:02.962+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:40:02.963233+0000) 2024-03-04T12:40:03.968 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:40:03.962+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:40:03.963529+0000) 2024-03-04T12:40:04.968 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:40:04.963+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:40:04.963873+0000) 2024-03-04T12:40:05.968 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:40:05.963+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:40:05.964168+0000) 2024-03-04T12:40:06.968 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:40:06.963+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:40:06.964471+0000) 2024-03-04T12:40:07.969 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:40:07.963+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:40:07.964794+0000) 2024-03-04T12:40:08.969 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:40:08.964+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:40:08.965137+0000) 2024-03-04T12:40:09.969 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:40:09.964+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:40:09.965431+0000) 2024-03-04T12:40:10.970 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:40:10.964+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:40:10.965733+0000) 2024-03-04T12:40:11.970 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:40:11.965+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:40:11.966040+0000) 2024-03-04T12:40:12.970 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:40:12.965+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:40:12.966332+0000) 2024-03-04T12:40:13.971 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:40:13.965+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:40:13.966628+0000) 2024-03-04T12:40:14.971 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:40:14.966+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:40:14.966956+0000) 2024-03-04T12:40:15.971 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:40:15.966+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:40:15.967256+0000) 2024-03-04T12:40:16.972 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:40:16.966+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:40:16.967551+0000) 2024-03-04T12:40:17.972 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:40:17.967+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:40:17.967907+0000) 2024-03-04T12:40:18.972 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:40:18.967+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:40:18.968159+0000) 2024-03-04T12:40:19.972 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:40:19.967+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:40:19.968446+0000) 2024-03-04T12:40:20.287 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:40:20.295 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:40:20.973 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:40:20.968+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:40:20.968738+0000) 2024-03-04T12:40:21.973 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:40:21.968+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:40:21.969080+0000) 2024-03-04T12:40:22.973 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:40:22.968+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:40:22.969372+0000) 2024-03-04T12:40:23.974 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:40:23.968+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:40:23.969610+0000) 2024-03-04T12:40:24.974 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:40:24.969+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:40:24.969966+0000) 2024-03-04T12:40:25.974 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:40:25.969+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:40:25.970254+0000) 2024-03-04T12:40:26.975 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:40:26.969+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:40:26.970548+0000) 2024-03-04T12:40:27.975 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:40:27.970+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:40:27.970896+0000) 2024-03-04T12:40:28.975 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:40:28.970+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:40:28.971183+0000) 2024-03-04T12:40:29.975 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:40:29.970+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:40:29.971465+0000) 2024-03-04T12:40:30.976 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:40:30.971+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:40:30.971758+0000) 2024-03-04T12:40:31.976 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:40:31.971+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:40:31.972060+0000) 2024-03-04T12:40:32.976 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:40:32.971+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:40:32.972352+0000) 2024-03-04T12:40:33.977 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:40:33.972+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:40:33.972634+0000) 2024-03-04T12:40:34.977 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:40:34.972+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:40:34.972969+0000) 2024-03-04T12:40:35.977 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:40:35.972+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:40:35.973247+0000) 2024-03-04T12:40:36.978 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:40:36.973+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:40:36.973517+0000) 2024-03-04T12:40:37.978 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:40:37.972+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:40:37.973786+0000) 2024-03-04T12:40:38.978 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:40:38.973+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:40:38.974032+0000) 2024-03-04T12:40:39.978 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:40:39.973+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:40:39.974292+0000) 2024-03-04T12:40:40.979 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:40:40.973+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:40:40.974486+0000) 2024-03-04T12:40:41.979 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:40:41.974+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:40:41.974739+0000) 2024-03-04T12:40:42.979 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:40:42.974+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:40:42.974976+0000) 2024-03-04T12:40:43.979 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:40:43.974+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:40:43.975219+0000) 2024-03-04T12:40:44.979 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:40:44.975+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:40:44.975464+0000) 2024-03-04T12:40:45.980 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:40:45.974+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:40:45.975701+0000) 2024-03-04T12:40:46.980 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:40:46.975+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:40:46.975949+0000) 2024-03-04T12:40:47.980 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:40:47.976+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:40:47.976194+0000) 2024-03-04T12:40:48.980 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:40:48.976+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:40:48.976448+0000) 2024-03-04T12:40:49.981 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:40:49.976+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:40:49.976694+0000) 2024-03-04T12:40:50.508 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:40:50.516 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:40:50.981 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:40:50.976+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:40:50.976949+0000) 2024-03-04T12:40:51.981 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:40:51.975+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:40:51.977150+0000) 2024-03-04T12:40:52.981 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:40:52.976+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:40:52.977406+0000) 2024-03-04T12:40:53.982 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:40:53.975+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:40:53.977595+0000) 2024-03-04T12:40:54.982 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:40:54.976+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:40:54.977875+0000) 2024-03-04T12:40:55.982 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:40:55.976+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:40:55.978061+0000) 2024-03-04T12:40:56.982 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:40:56.977+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:40:56.978313+0000) 2024-03-04T12:40:57.983 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:40:57.977+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:40:57.978555+0000) 2024-03-04T12:40:58.983 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:40:58.977+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:40:58.978857+0000) 2024-03-04T12:40:59.983 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:40:59.978+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:40:59.979066+0000) 2024-03-04T12:41:00.983 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:41:00.978+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:41:00.979312+0000) 2024-03-04T12:41:01.984 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:41:01.978+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:41:01.979562+0000) 2024-03-04T12:41:02.984 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:41:02.978+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:41:02.979803+0000) 2024-03-04T12:41:03.984 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:41:03.979+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:41:03.980088+0000) 2024-03-04T12:41:04.984 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:41:04.979+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:41:04.980388+0000) 2024-03-04T12:41:05.985 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:41:05.979+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:41:05.980651+0000) 2024-03-04T12:41:06.985 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:41:06.980+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:41:06.980932+0000) 2024-03-04T12:41:07.985 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:41:07.980+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:41:07.981179+0000) 2024-03-04T12:41:08.986 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:41:08.980+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:41:08.981423+0000) 2024-03-04T12:41:09.986 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:41:09.980+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:41:09.981616+0000) 2024-03-04T12:41:10.986 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:41:10.981+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:41:10.981907+0000) 2024-03-04T12:41:11.986 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:41:11.981+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:41:11.982111+0000) 2024-03-04T12:41:12.986 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:41:12.981+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:41:12.982352+0000) 2024-03-04T12:41:13.987 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:41:13.981+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:41:13.982599+0000) 2024-03-04T12:41:14.987 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:41:14.982+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:41:14.982874+0000) 2024-03-04T12:41:15.987 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:41:15.982+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:41:15.983113+0000) 2024-03-04T12:41:16.987 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:41:16.982+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:41:16.983358+0000) 2024-03-04T12:41:17.988 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:41:17.982+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:41:17.983608+0000) 2024-03-04T12:41:18.988 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:41:18.983+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:41:18.983889+0000) 2024-03-04T12:41:19.988 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:41:19.983+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:41:19.984130+0000) 2024-03-04T12:41:20.739 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:41:20.749 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:41:20.988 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:41:20.983+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:41:20.984384+0000) 2024-03-04T12:41:21.989 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:41:21.983+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:41:21.984618+0000) 2024-03-04T12:41:22.989 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:41:22.984+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:41:22.984924+0000) 2024-03-04T12:41:23.989 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:41:23.984+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:41:23.985188+0000) 2024-03-04T12:41:24.989 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:41:24.984+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:41:24.985449+0000) 2024-03-04T12:41:25.990 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:41:25.985+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:41:25.985692+0000) 2024-03-04T12:41:26.991 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:41:26.985+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:41:26.985943+0000) 2024-03-04T12:41:27.990 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:41:27.985+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:41:27.986193+0000) 2024-03-04T12:41:28.990 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:41:28.985+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:41:28.986434+0000) 2024-03-04T12:41:29.991 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:41:29.985+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:41:29.986701+0000) 2024-03-04T12:41:30.991 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:41:30.986+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:41:30.986949+0000) 2024-03-04T12:41:31.991 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:41:31.986+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:41:31.987229+0000) 2024-03-04T12:41:32.991 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:41:32.987+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:41:32.987475+0000) 2024-03-04T12:41:33.992 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:41:33.987+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:41:33.987717+0000) 2024-03-04T12:41:34.992 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:41:34.987+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:41:34.987959+0000) 2024-03-04T12:41:35.992 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:41:35.987+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:41:35.988199+0000) 2024-03-04T12:41:36.992 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:41:36.987+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:41:36.988443+0000) 2024-03-04T12:41:37.993 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:41:37.987+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:41:37.988690+0000) 2024-03-04T12:41:38.993 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:41:38.988+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:41:38.988946+0000) 2024-03-04T12:41:39.993 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:41:39.988+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:41:39.989180+0000) 2024-03-04T12:41:40.993 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:41:40.988+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:41:40.989377+0000) 2024-03-04T12:41:41.994 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:41:41.989+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:41:41.989620+0000) 2024-03-04T12:41:42.994 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:41:42.989+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:41:42.989921+0000) 2024-03-04T12:41:43.994 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:41:43.989+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:41:43.990162+0000) 2024-03-04T12:41:44.994 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:41:44.990+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:41:44.990405+0000) 2024-03-04T12:41:45.995 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:41:45.990+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:41:45.990648+0000) 2024-03-04T12:41:46.995 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:41:46.990+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:41:46.990935+0000) 2024-03-04T12:41:47.995 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:41:47.990+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:41:47.991184+0000) 2024-03-04T12:41:48.995 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:41:48.991+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:41:48.991430+0000) 2024-03-04T12:41:49.996 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:41:49.990+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:41:49.991664+0000) 2024-03-04T12:41:50.892 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:41:50.900 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:41:50.996 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:41:50.991+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:41:50.991856+0000) 2024-03-04T12:41:51.996 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:41:51.990+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:41:51.992081+0000) 2024-03-04T12:41:52.996 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:41:52.990+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:41:52.992382+0000) 2024-03-04T12:41:53.997 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:41:53.990+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:41:53.992631+0000) 2024-03-04T12:41:54.997 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:41:54.991+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:41:54.992963+0000) 2024-03-04T12:41:55.997 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:41:55.992+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:41:55.993239+0000) 2024-03-04T12:41:56.998 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:41:56.992+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:41:56.993528+0000) 2024-03-04T12:41:57.998 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:41:57.992+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:41:57.993799+0000) 2024-03-04T12:41:58.998 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:41:58.993+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:41:58.994199+0000) 2024-03-04T12:41:59.998 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:41:59.993+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:41:59.994438+0000) 2024-03-04T12:42:00.999 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:42:00.993+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:42:00.994692+0000) 2024-03-04T12:42:01.999 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:42:01.993+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:42:01.994951+0000) 2024-03-04T12:42:03.003 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:42:02.994+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:42:02.995188+0000) 2024-03-04T12:42:03.999 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:42:03.994+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:42:03.995428+0000) 2024-03-04T12:42:05.000 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:42:04.994+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:42:04.995670+0000) 2024-03-04T12:42:06.000 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:42:05.995+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:42:05.995923+0000) 2024-03-04T12:42:07.000 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:42:06.995+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:42:06.996117+0000) 2024-03-04T12:42:08.000 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:42:07.995+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:42:07.996363+0000) 2024-03-04T12:42:09.000 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:42:08.995+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:42:08.996606+0000) 2024-03-04T12:42:10.001 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:42:09.996+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:42:09.996911+0000) 2024-03-04T12:42:11.001 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:42:10.996+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:42:10.997148+0000) 2024-03-04T12:42:12.001 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:42:11.996+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:42:11.997377+0000) 2024-03-04T12:42:13.002 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:42:12.996+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:42:12.997612+0000) 2024-03-04T12:42:14.002 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:42:13.997+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:42:13.997914+0000) 2024-03-04T12:42:15.002 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:42:14.997+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:42:14.998153+0000) 2024-03-04T12:42:16.002 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:42:15.997+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:42:15.998383+0000) 2024-03-04T12:42:17.003 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:42:16.997+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:42:16.998625+0000) 2024-03-04T12:42:18.003 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:42:17.998+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:42:17.998917+0000) 2024-03-04T12:42:19.004 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:42:18.998+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:42:18.999179+0000) 2024-03-04T12:42:20.003 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:42:19.998+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:42:19.999413+0000) 2024-03-04T12:42:21.004 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:42:20.999+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:42:20.999653+0000) 2024-03-04T12:42:21.107 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:42:21.116 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:42:22.004 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:42:21.999+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:42:21.999998+0000) 2024-03-04T12:42:23.004 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:42:22.999+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:42:23.000270+0000) 2024-03-04T12:42:24.004 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:42:23.999+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:42:24.000504+0000) 2024-03-04T12:42:25.005 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:42:25.000+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:42:25.000749+0000) 2024-03-04T12:42:26.005 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:42:26.000+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:42:26.000987+0000) 2024-03-04T12:42:27.005 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:42:27.000+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:42:27.001247+0000) 2024-03-04T12:42:28.005 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:42:28.000+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:42:28.001507+0000) 2024-03-04T12:42:29.006 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:42:29.001+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:42:29.001766+0000) 2024-03-04T12:42:30.006 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:42:30.001+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:42:30.002002+0000) 2024-03-04T12:42:31.006 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:42:31.001+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:42:31.002236+0000) 2024-03-04T12:42:32.006 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:42:32.002+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:42:32.002478+0000) 2024-03-04T12:42:33.007 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:42:33.002+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:42:33.002715+0000) 2024-03-04T12:42:34.007 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:42:34.002+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:42:34.002952+0000) 2024-03-04T12:42:35.007 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:42:35.002+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:42:35.003197+0000) 2024-03-04T12:42:36.007 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:42:36.003+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:42:36.003433+0000) 2024-03-04T12:42:37.008 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:42:37.003+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:42:37.003675+0000) 2024-03-04T12:42:38.008 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:42:38.003+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:42:38.003957+0000) 2024-03-04T12:42:39.008 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:42:39.003+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:42:39.004197+0000) 2024-03-04T12:42:40.008 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:42:40.004+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:42:40.004447+0000) 2024-03-04T12:42:41.009 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:42:41.003+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:42:41.004710+0000) 2024-03-04T12:42:42.009 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:42:42.004+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:42:42.004948+0000) 2024-03-04T12:42:43.009 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:42:43.004+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:42:43.005179+0000) 2024-03-04T12:42:44.009 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:42:44.005+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:42:44.005422+0000) 2024-03-04T12:42:45.010 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:42:45.005+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:42:45.005659+0000) 2024-03-04T12:42:46.010 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:42:46.005+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:42:46.005936+0000) 2024-03-04T12:42:47.010 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:42:47.005+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:42:47.006171+0000) 2024-03-04T12:42:48.010 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:42:48.006+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:42:48.006413+0000) 2024-03-04T12:42:49.011 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:42:49.005+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:42:49.006670+0000) 2024-03-04T12:42:50.011 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:42:50.006+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:42:50.006941+0000) 2024-03-04T12:42:51.011 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:42:51.007+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:42:51.007184+0000) 2024-03-04T12:42:51.287 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:42:51.296 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:42:52.011 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:42:52.007+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:42:52.007441+0000) 2024-03-04T12:42:53.012 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:42:53.006+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:42:53.007709+0000) 2024-03-04T12:42:54.012 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:42:54.006+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:42:54.007946+0000) 2024-03-04T12:42:55.012 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:42:55.007+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:42:55.008187+0000) 2024-03-04T12:42:56.012 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:42:56.007+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:42:56.008425+0000) 2024-03-04T12:42:57.013 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:42:57.007+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:42:57.008667+0000) 2024-03-04T12:42:58.013 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:42:58.007+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:42:58.008960+0000) 2024-03-04T12:42:59.013 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:42:59.008+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:42:59.009198+0000) 2024-03-04T12:43:00.013 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:43:00.008+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:43:00.009422+0000) 2024-03-04T12:43:01.014 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:43:01.008+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:43:01.009662+0000) 2024-03-04T12:43:02.014 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:43:02.008+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:43:02.009946+0000) 2024-03-04T12:43:03.014 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:43:03.009+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:43:03.010145+0000) 2024-03-04T12:43:04.014 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:43:04.009+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:43:04.010380+0000) 2024-03-04T12:43:05.015 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:43:05.009+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:43:05.010621+0000) 2024-03-04T12:43:06.016 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:43:06.010+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:43:06.010921+0000) 2024-03-04T12:43:07.015 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:43:07.010+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:43:07.011188+0000) 2024-03-04T12:43:08.015 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:43:08.010+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:43:08.011438+0000) 2024-03-04T12:43:09.016 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:43:09.010+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:43:09.011675+0000) 2024-03-04T12:43:10.016 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:43:10.011+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:43:10.011941+0000) 2024-03-04T12:43:11.016 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:43:11.011+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:43:11.012184+0000) 2024-03-04T12:43:12.016 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:43:12.011+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:43:12.012421+0000) 2024-03-04T12:43:13.017 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:43:13.011+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:43:13.012682+0000) 2024-03-04T12:43:14.017 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:43:14.012+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:43:14.012949+0000) 2024-03-04T12:43:15.017 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:43:15.012+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:43:15.013194+0000) 2024-03-04T12:43:16.018 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:43:16.012+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:43:16.013370+0000) 2024-03-04T12:43:17.017 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:43:17.012+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:43:17.013611+0000) 2024-03-04T12:43:18.018 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:43:18.013+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:43:18.013932+0000) 2024-03-04T12:43:19.018 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:43:19.013+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:43:19.014173+0000) 2024-03-04T12:43:20.018 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:43:20.013+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:43:20.014406+0000) 2024-03-04T12:43:21.019 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:43:21.013+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:43:21.014649+0000) 2024-03-04T12:43:21.493 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:43:21.502 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:43:22.019 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:43:22.014+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:43:22.014946+0000) 2024-03-04T12:43:23.019 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:43:23.014+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:43:23.015208+0000) 2024-03-04T12:43:24.019 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:43:24.014+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:43:24.015463+0000) 2024-03-04T12:43:25.020 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:43:25.015+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:43:25.015698+0000) 2024-03-04T12:43:26.020 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:43:26.015+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:43:26.015917+0000) 2024-03-04T12:43:27.020 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:43:27.015+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:43:27.016158+0000) 2024-03-04T12:43:28.020 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:43:28.015+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:43:28.016356+0000) 2024-03-04T12:43:29.021 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:43:29.016+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:43:29.016599+0000) 2024-03-04T12:43:30.021 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:43:30.016+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:43:30.016889+0000) 2024-03-04T12:43:31.021 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:43:31.016+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:43:31.017133+0000) 2024-03-04T12:43:32.021 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:43:32.016+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:43:32.017342+0000) 2024-03-04T12:43:33.022 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:43:33.017+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:43:33.017576+0000) 2024-03-04T12:43:34.022 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:43:34.017+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:43:34.017815+0000) 2024-03-04T12:43:35.022 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:43:35.017+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:43:35.018114+0000) 2024-03-04T12:43:36.022 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:43:36.017+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:43:36.018349+0000) 2024-03-04T12:43:37.023 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:43:37.018+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:43:37.018594+0000) 2024-03-04T12:43:38.023 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:43:38.018+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:43:38.018902+0000) 2024-03-04T12:43:39.023 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:43:39.018+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:43:39.019145+0000) 2024-03-04T12:43:40.023 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:43:40.018+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:43:40.019322+0000) 2024-03-04T12:43:41.024 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:43:41.019+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:43:41.019561+0000) 2024-03-04T12:43:42.024 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:43:42.019+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:43:42.019804+0000) 2024-03-04T12:43:43.024 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:43:43.019+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:43:43.020110+0000) 2024-03-04T12:43:44.024 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:43:44.020+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:43:44.020348+0000) 2024-03-04T12:43:45.025 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:43:45.020+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:43:45.020587+0000) 2024-03-04T12:43:46.025 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:43:46.020+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:43:46.020854+0000) 2024-03-04T12:43:47.025 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:43:47.020+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:43:47.021118+0000) 2024-03-04T12:43:48.025 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:43:48.021+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:43:48.021370+0000) 2024-03-04T12:43:49.026 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:43:49.020+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:43:49.021625+0000) 2024-03-04T12:43:50.026 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:43:50.021+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:43:50.021910+0000) 2024-03-04T12:43:51.026 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:43:51.021+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:43:51.022147+0000) 2024-03-04T12:43:51.690 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:43:51.698 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:43:52.026 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:43:52.022+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:43:52.022392+0000) 2024-03-04T12:43:53.027 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:43:53.021+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:43:53.022617+0000) 2024-03-04T12:43:54.027 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:43:54.021+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:43:54.022917+0000) 2024-03-04T12:43:55.027 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:43:55.021+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:43:55.023185+0000) 2024-03-04T12:43:56.027 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:43:56.022+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:43:56.023422+0000) 2024-03-04T12:43:57.028 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:43:57.022+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:43:57.023668+0000) 2024-03-04T12:43:58.028 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:43:58.022+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:43:58.023943+0000) 2024-03-04T12:43:59.028 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:43:59.023+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:43:59.024182+0000) 2024-03-04T12:44:00.028 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:44:00.023+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:44:00.024416+0000) 2024-03-04T12:44:01.029 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:44:01.023+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:44:01.024679+0000) 2024-03-04T12:44:02.029 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:44:02.023+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:44:02.024949+0000) 2024-03-04T12:44:03.029 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:44:03.024+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:44:03.025183+0000) 2024-03-04T12:44:04.029 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:44:04.024+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:44:04.025424+0000) 2024-03-04T12:44:05.030 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:44:05.024+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:44:05.025665+0000) 2024-03-04T12:44:06.030 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:44:06.025+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:44:06.025939+0000) 2024-03-04T12:44:07.030 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:44:07.025+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:44:07.026178+0000) 2024-03-04T12:44:08.030 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:44:08.025+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:44:08.026441+0000) 2024-03-04T12:44:09.031 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:44:09.025+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:44:09.026681+0000) 2024-03-04T12:44:10.031 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:44:10.026+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:44:10.026926+0000) 2024-03-04T12:44:11.031 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:44:11.026+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:44:11.027187+0000) 2024-03-04T12:44:12.031 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:44:12.026+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:44:12.027376+0000) 2024-03-04T12:44:13.032 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:44:13.026+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:44:13.027612+0000) 2024-03-04T12:44:14.032 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:44:14.027+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:44:14.027934+0000) 2024-03-04T12:44:15.032 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:44:15.027+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:44:15.028174+0000) 2024-03-04T12:44:16.032 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:44:16.027+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:44:16.028409+0000) 2024-03-04T12:44:17.033 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:44:17.027+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:44:17.028658+0000) 2024-03-04T12:44:18.033 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:44:18.028+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:44:18.028937+0000) 2024-03-04T12:44:19.033 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:44:19.028+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:44:19.029176+0000) 2024-03-04T12:44:20.034 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:44:20.028+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:44:20.029409+0000) 2024-03-04T12:44:21.034 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:44:21.028+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:44:21.029650+0000) 2024-03-04T12:44:21.919 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:44:21.927 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:44:22.034 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:44:22.029+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:44:22.029930+0000) 2024-03-04T12:44:23.034 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:44:23.029+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:44:23.030103+0000) 2024-03-04T12:44:24.034 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:44:24.029+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:44:24.030365+0000) 2024-03-04T12:44:25.035 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:44:25.029+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:44:25.030601+0000) 2024-03-04T12:44:26.035 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:44:26.030+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:44:26.030903+0000) 2024-03-04T12:44:27.035 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:44:27.030+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:44:27.031153+0000) 2024-03-04T12:44:28.035 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:44:28.030+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:44:28.031401+0000) 2024-03-04T12:44:29.036 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:44:29.031+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:44:29.031678+0000) 2024-03-04T12:44:30.036 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:44:30.031+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:44:30.031950+0000) 2024-03-04T12:44:31.037 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:44:31.031+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:44:31.032198+0000) 2024-03-04T12:44:32.037 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:44:32.031+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:44:32.032446+0000) 2024-03-04T12:44:33.037 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:44:33.032+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:44:33.032682+0000) 2024-03-04T12:44:34.037 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:44:34.032+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:44:34.032929+0000) 2024-03-04T12:44:35.037 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:44:35.032+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:44:35.033196+0000) 2024-03-04T12:44:36.038 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:44:36.032+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:44:36.033428+0000) 2024-03-04T12:44:37.038 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:44:37.033+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:44:37.033683+0000) 2024-03-04T12:44:38.038 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:44:38.033+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:44:38.033945+0000) 2024-03-04T12:44:39.038 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:44:39.033+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:44:39.034187+0000) 2024-03-04T12:44:40.039 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:44:40.034+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:44:40.034422+0000) 2024-03-04T12:44:41.039 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:44:41.034+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:44:41.034670+0000) 2024-03-04T12:44:42.039 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:44:42.034+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:44:42.034953+0000) 2024-03-04T12:44:43.039 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:44:43.034+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:44:43.035260+0000) 2024-03-04T12:44:44.040 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:44:44.035+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:44:44.035503+0000) 2024-03-04T12:44:45.040 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:44:45.035+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:44:45.035749+0000) 2024-03-04T12:44:46.040 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:44:46.035+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:44:46.035997+0000) 2024-03-04T12:44:47.040 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:44:47.036+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:44:47.036240+0000) 2024-03-04T12:44:48.040 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:44:48.036+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:44:48.036518+0000) 2024-03-04T12:44:49.041 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:44:49.036+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:44:49.036762+0000) 2024-03-04T12:44:50.041 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:44:50.036+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:44:50.037003+0000) 2024-03-04T12:44:51.041 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:44:51.037+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:44:51.037248+0000) 2024-03-04T12:44:52.041 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:44:52.037+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:44:52.037492+0000) 2024-03-04T12:44:52.057 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:44:52.065 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:44:53.042 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:44:53.037+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:44:53.037886+0000) 2024-03-04T12:44:54.042 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:44:54.037+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:44:54.038147+0000) 2024-03-04T12:44:55.042 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:44:55.037+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:44:55.038385+0000) 2024-03-04T12:44:56.043 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:44:56.037+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:44:56.038626+0000) 2024-03-04T12:44:57.043 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:44:57.037+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:44:57.038946+0000) 2024-03-04T12:44:58.043 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:44:58.038+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:44:58.039218+0000) 2024-03-04T12:44:59.043 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:44:59.038+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:44:59.039466+0000) 2024-03-04T12:45:00.044 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:45:00.038+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:45:00.039706+0000) 2024-03-04T12:45:01.044 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:45:01.038+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:45:01.039951+0000) 2024-03-04T12:45:02.044 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:45:02.039+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:45:02.040211+0000) 2024-03-04T12:45:03.044 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:45:03.039+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:45:03.040466+0000) 2024-03-04T12:45:04.045 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:45:04.039+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:45:04.040728+0000) 2024-03-04T12:45:05.045 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:45:05.040+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:45:05.040974+0000) 2024-03-04T12:45:06.045 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:45:06.040+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:45:06.041220+0000) 2024-03-04T12:45:07.045 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:45:07.040+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:45:07.041487+0000) 2024-03-04T12:45:08.046 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:45:08.040+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:45:08.041752+0000) 2024-03-04T12:45:09.046 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:45:09.041+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:45:09.041996+0000) 2024-03-04T12:45:10.046 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:45:10.041+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:45:10.042240+0000) 2024-03-04T12:45:11.046 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:45:11.041+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:45:11.042483+0000) 2024-03-04T12:45:12.047 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:45:12.041+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:45:12.042727+0000) 2024-03-04T12:45:13.048 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:45:13.042+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:45:13.042980+0000) 2024-03-04T12:45:14.047 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:45:14.042+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:45:14.043222+0000) 2024-03-04T12:45:15.048 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:45:15.042+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:45:15.043468+0000) 2024-03-04T12:45:16.048 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:45:16.042+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:45:16.043707+0000) 2024-03-04T12:45:17.048 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:45:17.043+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:45:17.043956+0000) 2024-03-04T12:45:18.048 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:45:18.043+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:45:18.044216+0000) 2024-03-04T12:45:19.048 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:45:19.043+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:45:19.044454+0000) 2024-03-04T12:45:20.049 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:45:20.044+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:45:20.044692+0000) 2024-03-04T12:45:21.049 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:45:21.044+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:45:21.044936+0000) 2024-03-04T12:45:22.049 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:45:22.044+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:45:22.045133+0000) 2024-03-04T12:45:22.212 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:45:22.224 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:45:23.049 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:45:23.044+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:45:23.045412+0000) 2024-03-04T12:45:24.050 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:45:24.045+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:45:24.045654+0000) 2024-03-04T12:45:25.050 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:45:25.045+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:45:25.045940+0000) 2024-03-04T12:45:26.050 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:45:26.045+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:45:26.046181+0000) 2024-03-04T12:45:27.050 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:45:27.045+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:45:27.046422+0000) 2024-03-04T12:45:28.051 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:45:28.046+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:45:28.046679+0000) 2024-03-04T12:45:29.051 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:45:29.046+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:45:29.046944+0000) 2024-03-04T12:45:30.051 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:45:30.046+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:45:30.047185+0000) 2024-03-04T12:45:31.051 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:45:31.046+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:45:31.047432+0000) 2024-03-04T12:45:32.052 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:45:32.046+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:45:32.047617+0000) 2024-03-04T12:45:33.052 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:45:33.047+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:45:33.047915+0000) 2024-03-04T12:45:34.052 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:45:34.047+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:45:34.048177+0000) 2024-03-04T12:45:35.053 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:45:35.047+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:45:35.048422+0000) 2024-03-04T12:45:36.053 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:45:36.047+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:45:36.048611+0000) 2024-03-04T12:45:37.053 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:45:37.048+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:45:37.048895+0000) 2024-03-04T12:45:38.053 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:45:38.048+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:45:38.049102+0000) 2024-03-04T12:45:39.053 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:45:39.048+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:45:39.049349+0000) 2024-03-04T12:45:40.054 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:45:40.049+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:45:40.049589+0000) 2024-03-04T12:45:41.054 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:45:41.049+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:45:41.049892+0000) 2024-03-04T12:45:42.054 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:45:42.049+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:45:42.050146+0000) 2024-03-04T12:45:43.054 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:45:43.049+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:45:43.050338+0000) 2024-03-04T12:45:44.055 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:45:44.049+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:45:44.050578+0000) 2024-03-04T12:45:45.055 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:45:45.050+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:45:45.050816+0000) 2024-03-04T12:45:46.055 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:45:46.050+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:45:46.051041+0000) 2024-03-04T12:45:47.055 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:45:47.050+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:45:47.051285+0000) 2024-03-04T12:45:48.056 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:45:48.051+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:45:48.051584+0000) 2024-03-04T12:45:49.056 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:45:49.051+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:45:49.051856+0000) 2024-03-04T12:45:50.056 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:45:50.051+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:45:50.052101+0000) 2024-03-04T12:45:51.056 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:45:51.051+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:45:51.052370+0000) 2024-03-04T12:45:52.057 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:45:52.052+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:45:52.052606+0000) 2024-03-04T12:45:52.452 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:45:52.460 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:45:53.057 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:45:53.052+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:45:53.052914+0000) 2024-03-04T12:45:54.057 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:45:54.053+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:45:54.053186+0000) 2024-03-04T12:45:55.057 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:45:55.051+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:45:55.053381+0000) 2024-03-04T12:45:56.058 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:45:56.052+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:45:56.053617+0000) 2024-03-04T12:45:57.058 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:45:57.052+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:45:57.053898+0000) 2024-03-04T12:45:58.058 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:45:58.053+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:45:58.054148+0000) 2024-03-04T12:45:59.058 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:45:59.053+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:45:59.054384+0000) 2024-03-04T12:46:00.059 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:46:00.053+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:46:00.054652+0000) 2024-03-04T12:46:01.059 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:46:01.053+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:46:01.054926+0000) 2024-03-04T12:46:02.059 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:46:02.054+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:46:02.055161+0000) 2024-03-04T12:46:03.059 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:46:03.054+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:46:03.055399+0000) 2024-03-04T12:46:04.060 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:46:04.054+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:46:04.055637+0000) 2024-03-04T12:46:05.060 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:46:05.055+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:46:05.055869+0000) 2024-03-04T12:46:06.060 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:46:06.055+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:46:06.056111+0000) 2024-03-04T12:46:07.060 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:46:07.055+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:46:07.056356+0000) 2024-03-04T12:46:08.061 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:46:08.055+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:46:08.056601+0000) 2024-03-04T12:46:09.061 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:46:09.056+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:46:09.056879+0000) 2024-03-04T12:46:10.061 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:46:10.056+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:46:10.057059+0000) 2024-03-04T12:46:11.061 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:46:11.056+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:46:11.057292+0000) 2024-03-04T12:46:12.062 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:46:12.056+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:46:12.057526+0000) 2024-03-04T12:46:13.062 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:46:13.056+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:46:13.057760+0000) 2024-03-04T12:46:14.062 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:46:14.057+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:46:14.058002+0000) 2024-03-04T12:46:15.062 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:46:15.057+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:46:15.058194+0000) 2024-03-04T12:46:16.062 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:46:16.057+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:46:16.058434+0000) 2024-03-04T12:46:17.063 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:46:17.057+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:46:17.058674+0000) 2024-03-04T12:46:18.063 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:46:18.058+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:46:18.058938+0000) 2024-03-04T12:46:19.063 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:46:19.058+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:46:19.059188+0000) 2024-03-04T12:46:20.064 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:46:20.058+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:46:20.059422+0000) 2024-03-04T12:46:21.064 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:46:21.058+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:46:21.059666+0000) 2024-03-04T12:46:22.064 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:46:22.059+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:46:22.059937+0000) 2024-03-04T12:46:22.689 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:46:22.697 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:46:23.064 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:46:23.059+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:46:23.060195+0000) 2024-03-04T12:46:24.065 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:46:24.059+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:46:24.060458+0000) 2024-03-04T12:46:25.065 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:46:25.060+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:46:25.060767+0000) 2024-03-04T12:46:26.065 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:46:26.060+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:46:26.060998+0000) 2024-03-04T12:46:27.065 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:46:27.060+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:46:27.061235+0000) 2024-03-04T12:46:28.066 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:46:28.060+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:46:28.061479+0000) 2024-03-04T12:46:29.066 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:46:29.061+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:46:29.061719+0000) 2024-03-04T12:46:30.066 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:46:30.061+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:46:30.061957+0000) 2024-03-04T12:46:31.066 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:46:31.061+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:46:31.062195+0000) 2024-03-04T12:46:32.066 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:46:32.061+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:46:32.062449+0000) 2024-03-04T12:46:33.067 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:46:33.062+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:46:33.062682+0000) 2024-03-04T12:46:34.067 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:46:34.062+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:46:34.062957+0000) 2024-03-04T12:46:35.067 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:46:35.062+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:46:35.063221+0000) 2024-03-04T12:46:36.067 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:46:36.063+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:46:36.063455+0000) 2024-03-04T12:46:37.068 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:46:37.063+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:46:37.063693+0000) 2024-03-04T12:46:38.068 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:46:38.063+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:46:38.063936+0000) 2024-03-04T12:46:39.068 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:46:39.063+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:46:39.064175+0000) 2024-03-04T12:46:40.068 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:46:40.064+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:46:40.064435+0000) 2024-03-04T12:46:41.069 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:46:41.063+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:46:41.064692+0000) 2024-03-04T12:46:42.069 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:46:42.064+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:46:42.064946+0000) 2024-03-04T12:46:43.069 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:46:43.064+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:46:43.065181+0000) 2024-03-04T12:46:44.070 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:46:44.064+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:46:44.065362+0000) 2024-03-04T12:46:45.070 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:46:45.065+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:46:45.065594+0000) 2024-03-04T12:46:46.070 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:46:46.065+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:46:46.065783+0000) 2024-03-04T12:46:47.070 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:46:47.065+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:46:47.066020+0000) 2024-03-04T12:46:48.070 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:46:48.066+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:46:48.066261+0000) 2024-03-04T12:46:49.071 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:46:49.066+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:46:49.066500+0000) 2024-03-04T12:46:50.071 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:46:50.066+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:46:50.066736+0000) 2024-03-04T12:46:51.071 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:46:51.066+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:46:51.066980+0000) 2024-03-04T12:46:52.071 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:46:52.067+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:46:52.067225+0000) 2024-03-04T12:46:52.918 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:46:52.926 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:46:53.071 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:46:53.066+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:46:53.067475+0000) 2024-03-04T12:46:54.072 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:46:54.066+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:46:54.067708+0000) 2024-03-04T12:46:55.072 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:46:55.066+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:46:55.067964+0000) 2024-03-04T12:46:56.072 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:46:56.067+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:46:56.068202+0000) 2024-03-04T12:46:57.073 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:46:57.067+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:46:57.068446+0000) 2024-03-04T12:46:58.073 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:46:58.066+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:46:58.068633+0000) 2024-03-04T12:46:59.073 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:46:59.067+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:46:59.068908+0000) 2024-03-04T12:47:00.073 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:47:00.068+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:47:00.069143+0000) 2024-03-04T12:47:01.073 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:47:01.068+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:47:01.069383+0000) 2024-03-04T12:47:02.074 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:47:02.068+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:47:02.069621+0000) 2024-03-04T12:47:03.074 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:47:03.068+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:47:03.069921+0000) 2024-03-04T12:47:04.074 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:47:04.069+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:47:04.070179+0000) 2024-03-04T12:47:05.075 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:47:05.069+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:47:05.070439+0000) 2024-03-04T12:47:06.075 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:47:06.069+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:47:06.070683+0000) 2024-03-04T12:47:07.075 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:47:07.070+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:47:07.070963+0000) 2024-03-04T12:47:08.075 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:47:08.070+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:47:08.071205+0000) 2024-03-04T12:47:09.075 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:47:09.070+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:47:09.071451+0000) 2024-03-04T12:47:10.076 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:47:10.070+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:47:10.071690+0000) 2024-03-04T12:47:11.076 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:47:11.071+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:47:11.071923+0000) 2024-03-04T12:47:12.076 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:47:12.071+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:47:12.072161+0000) 2024-03-04T12:47:13.077 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:47:13.071+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:47:13.072448+0000) 2024-03-04T12:47:14.077 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:47:14.071+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:47:14.072635+0000) 2024-03-04T12:47:15.077 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:47:15.072+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:47:15.072926+0000) 2024-03-04T12:47:16.077 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:47:16.072+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:47:16.073160+0000) 2024-03-04T12:47:17.078 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:47:17.072+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:47:17.073433+0000) 2024-03-04T12:47:18.078 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:47:18.072+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:47:18.073755+0000) 2024-03-04T12:47:19.078 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:47:19.073+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:47:19.074007+0000) 2024-03-04T12:47:20.079 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:47:20.073+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:47:20.074246+0000) 2024-03-04T12:47:21.079 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:47:21.073+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:47:21.074487+0000) 2024-03-04T12:47:22.079 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:47:22.073+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:47:22.074720+0000) 2024-03-04T12:47:23.056 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:47:23.064 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:47:23.079 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:47:23.074+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:47:23.074886+0000) 2024-03-04T12:47:24.079 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:47:24.074+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:47:24.075113+0000) 2024-03-04T12:47:25.080 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:47:25.074+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:47:25.075383+0000) 2024-03-04T12:47:26.080 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:47:26.075+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:47:26.075684+0000) 2024-03-04T12:47:27.080 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:47:27.075+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:47:27.076000+0000) 2024-03-04T12:47:28.080 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:47:28.075+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:47:28.076329+0000) 2024-03-04T12:47:29.081 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:47:29.076+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:47:29.076619+0000) 2024-03-04T12:47:30.081 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:47:30.076+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:47:30.076955+0000) 2024-03-04T12:47:31.081 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:47:31.076+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:47:31.077265+0000) 2024-03-04T12:47:32.082 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:47:32.076+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:47:32.077500+0000) 2024-03-04T12:47:33.082 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:47:33.077+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:47:33.077743+0000) 2024-03-04T12:47:34.082 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:47:34.077+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:47:34.077982+0000) 2024-03-04T12:47:35.082 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:47:35.077+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:47:35.078165+0000) 2024-03-04T12:47:36.083 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:47:36.077+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:47:36.078402+0000) 2024-03-04T12:47:37.083 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:47:37.078+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:47:37.078643+0000) 2024-03-04T12:47:38.083 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:47:38.078+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:47:38.078921+0000) 2024-03-04T12:47:39.083 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:47:39.078+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:47:39.079165+0000) 2024-03-04T12:47:40.083 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:47:40.078+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:47:40.079344+0000) 2024-03-04T12:47:41.084 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:47:41.079+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:47:41.079604+0000) 2024-03-04T12:47:42.084 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:47:42.079+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:47:42.079864+0000) 2024-03-04T12:47:43.084 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:47:43.079+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:47:43.080056+0000) 2024-03-04T12:47:44.084 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:47:44.079+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:47:44.080249+0000) 2024-03-04T12:47:45.085 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:47:45.079+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:47:45.080434+0000) 2024-03-04T12:47:46.085 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:47:46.079+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:47:46.080615+0000) 2024-03-04T12:47:47.085 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:47:47.080+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:47:47.080799+0000) 2024-03-04T12:47:48.085 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:47:48.080+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:47:48.081124+0000) 2024-03-04T12:47:49.086 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:47:49.081+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:47:49.081384+0000) 2024-03-04T12:47:50.086 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:47:50.081+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:47:50.081629+0000) 2024-03-04T12:47:51.086 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:47:51.081+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:47:51.081915+0000) 2024-03-04T12:47:52.086 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:47:52.081+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:47:52.082153+0000) 2024-03-04T12:47:53.087 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:47:53.081+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:47:53.082391+0000) 2024-03-04T12:47:53.198 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:47:53.209 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:47:54.087 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:47:54.082+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:47:54.082635+0000) 2024-03-04T12:47:55.087 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:47:55.082+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:47:55.082937+0000) 2024-03-04T12:47:56.087 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:47:56.081+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:47:56.083120+0000) 2024-03-04T12:47:57.087 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:47:57.082+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:47:57.083357+0000) 2024-03-04T12:47:58.088 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:47:58.081+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:47:58.083547+0000) 2024-03-04T12:47:59.088 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:47:59.082+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:47:59.083786+0000) 2024-03-04T12:48:00.088 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:48:00.082+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:48:00.084030+0000) 2024-03-04T12:48:01.088 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:48:01.083+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:48:01.084273+0000) 2024-03-04T12:48:02.088 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:48:02.083+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:48:02.084464+0000) 2024-03-04T12:48:03.089 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:48:03.083+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:48:03.084699+0000) 2024-03-04T12:48:04.089 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:48:04.083+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:48:04.084943+0000) 2024-03-04T12:48:05.089 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:48:05.084+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:48:05.085184+0000) 2024-03-04T12:48:06.090 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:48:06.084+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:48:06.085430+0000) 2024-03-04T12:48:07.090 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:48:07.084+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:48:07.085675+0000) 2024-03-04T12:48:08.090 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:48:08.085+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:48:08.085972+0000) 2024-03-04T12:48:09.090 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:48:09.085+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:48:09.086228+0000) 2024-03-04T12:48:10.091 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:48:10.085+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:48:10.086466+0000) 2024-03-04T12:48:11.091 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:48:11.085+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:48:11.086711+0000) 2024-03-04T12:48:12.091 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:48:12.086+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:48:12.086947+0000) 2024-03-04T12:48:13.091 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:48:13.086+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:48:13.087183+0000) 2024-03-04T12:48:14.092 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:48:14.086+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:48:14.087421+0000) 2024-03-04T12:48:15.092 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:48:15.086+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:48:15.087659+0000) 2024-03-04T12:48:16.092 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:48:16.087+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:48:16.087956+0000) 2024-03-04T12:48:17.092 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:48:17.087+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:48:17.088203+0000) 2024-03-04T12:48:18.093 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:48:18.087+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:48:18.088453+0000) 2024-03-04T12:48:19.093 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:48:19.087+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:48:19.088692+0000) 2024-03-04T12:48:20.093 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:48:20.088+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:48:20.088963+0000) 2024-03-04T12:48:21.093 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:48:21.088+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:48:21.089207+0000) 2024-03-04T12:48:22.094 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:48:22.088+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:48:22.089404+0000) 2024-03-04T12:48:23.094 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:48:23.088+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:48:23.089647+0000) 2024-03-04T12:48:23.341 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:48:23.353 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:48:24.094 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:48:24.089+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:48:24.089928+0000) 2024-03-04T12:48:25.094 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:48:25.089+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:48:25.090171+0000) 2024-03-04T12:48:26.095 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:48:26.089+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:48:26.090429+0000) 2024-03-04T12:48:27.095 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:48:27.090+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:48:27.090706+0000) 2024-03-04T12:48:28.095 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:48:28.090+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:48:28.090952+0000) 2024-03-04T12:48:29.095 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:48:29.090+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:48:29.091193+0000) 2024-03-04T12:48:30.096 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:48:30.090+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:48:30.091386+0000) 2024-03-04T12:48:31.096 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:48:31.091+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:48:31.091625+0000) 2024-03-04T12:48:32.096 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:48:32.091+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:48:32.091915+0000) 2024-03-04T12:48:33.096 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:48:33.091+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:48:33.092157+0000) 2024-03-04T12:48:34.097 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:48:34.091+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:48:34.092399+0000) 2024-03-04T12:48:35.097 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:48:35.092+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:48:35.092643+0000) 2024-03-04T12:48:36.097 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:48:36.092+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:48:36.092933+0000) 2024-03-04T12:48:37.097 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:48:37.092+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:48:37.093171+0000) 2024-03-04T12:48:38.098 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:48:38.092+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:48:38.093427+0000) 2024-03-04T12:48:39.098 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:48:39.092+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:48:39.093685+0000) 2024-03-04T12:48:40.098 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:48:40.093+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:48:40.093971+0000) 2024-03-04T12:48:41.098 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:48:41.093+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:48:41.094272+0000) 2024-03-04T12:48:42.099 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:48:42.094+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:48:42.094531+0000) 2024-03-04T12:48:43.099 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:48:43.094+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:48:43.094775+0000) 2024-03-04T12:48:44.099 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:48:44.094+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:48:44.095021+0000) 2024-03-04T12:48:45.099 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:48:45.094+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:48:45.095263+0000) 2024-03-04T12:48:46.100 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:48:46.094+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:48:46.095518+0000) 2024-03-04T12:48:47.100 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:48:47.094+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:48:47.095774+0000) 2024-03-04T12:48:48.100 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:48:48.095+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:48:48.096030+0000) 2024-03-04T12:48:49.100 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:48:49.095+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:48:49.096236+0000) 2024-03-04T12:48:50.101 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:48:50.096+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:48:50.096481+0000) 2024-03-04T12:48:51.101 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:48:51.096+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:48:51.096723+0000) 2024-03-04T12:48:52.101 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:48:52.096+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:48:52.096982+0000) 2024-03-04T12:48:53.101 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:48:53.096+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:48:53.097205+0000) 2024-03-04T12:48:53.564 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:48:53.572 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:48:54.101 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:48:54.097+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:48:54.097452+0000) 2024-03-04T12:48:55.102 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:48:55.097+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:48:55.097696+0000) 2024-03-04T12:48:56.102 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:48:56.097+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:48:56.097957+0000) 2024-03-04T12:48:57.102 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:48:57.098+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:48:57.098191+0000) 2024-03-04T12:48:58.103 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:48:58.097+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:48:58.098450+0000) 2024-03-04T12:48:59.103 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:48:59.097+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:48:59.098698+0000) 2024-03-04T12:49:00.103 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:49:00.097+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:49:00.098991+0000) 2024-03-04T12:49:01.103 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:49:01.098+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:49:01.099272+0000) 2024-03-04T12:49:02.104 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:49:02.098+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:49:02.099523+0000) 2024-03-04T12:49:03.104 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:49:03.098+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:49:03.099771+0000) 2024-03-04T12:49:04.104 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:49:04.098+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:49:04.100009+0000) 2024-03-04T12:49:05.104 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:49:05.099+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:49:05.100305+0000) 2024-03-04T12:49:06.105 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:49:06.099+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:49:06.100548+0000) 2024-03-04T12:49:07.105 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:49:07.099+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:49:07.100792+0000) 2024-03-04T12:49:08.105 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:49:08.100+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:49:08.101051+0000) 2024-03-04T12:49:09.106 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:49:09.100+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:49:09.101301+0000) 2024-03-04T12:49:10.106 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:49:10.100+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:49:10.101542+0000) 2024-03-04T12:49:11.106 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:49:11.100+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:49:11.101785+0000) 2024-03-04T12:49:12.106 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:49:12.101+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:49:12.101989+0000) 2024-03-04T12:49:13.106 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:49:13.101+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:49:13.102248+0000) 2024-03-04T12:49:14.107 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:49:14.101+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:49:14.102490+0000) 2024-03-04T12:49:15.107 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:49:15.101+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:49:15.102736+0000) 2024-03-04T12:49:16.107 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:49:16.102+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:49:16.102984+0000) 2024-03-04T12:49:17.107 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:49:17.102+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:49:17.103230+0000) 2024-03-04T12:49:18.108 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:49:18.102+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:49:18.103496+0000) 2024-03-04T12:49:19.108 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:49:19.102+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:49:19.103745+0000) 2024-03-04T12:49:20.108 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:49:20.103+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:49:20.103945+0000) 2024-03-04T12:49:21.108 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:49:21.103+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:49:21.104192+0000) 2024-03-04T12:49:22.109 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:49:22.103+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:49:22.104399+0000) 2024-03-04T12:49:23.109 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:49:23.103+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:49:23.104650+0000) 2024-03-04T12:49:23.752 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:49:23.760 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:49:24.109 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:49:24.104+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:49:24.104948+0000) 2024-03-04T12:49:25.109 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:49:25.104+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:49:25.105196+0000) 2024-03-04T12:49:26.110 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:49:26.104+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:49:26.105468+0000) 2024-03-04T12:49:27.110 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:49:27.105+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:49:27.105734+0000) 2024-03-04T12:49:28.110 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:49:28.105+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:49:28.105986+0000) 2024-03-04T12:49:29.110 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:49:29.105+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:49:29.106235+0000) 2024-03-04T12:49:30.111 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:49:30.105+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:49:30.106475+0000) 2024-03-04T12:49:31.111 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:49:31.105+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:49:31.106736+0000) 2024-03-04T12:49:32.111 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:49:32.106+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:49:32.106974+0000) 2024-03-04T12:49:33.111 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:49:33.106+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:49:33.107223+0000) 2024-03-04T12:49:34.112 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:49:34.106+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:49:34.107473+0000) 2024-03-04T12:49:35.112 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:49:35.106+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:49:35.107730+0000) 2024-03-04T12:49:36.112 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:49:36.107+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:49:36.108011+0000) 2024-03-04T12:49:37.112 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:49:37.107+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:49:37.108255+0000) 2024-03-04T12:49:38.113 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:49:38.108+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:49:38.108510+0000) 2024-03-04T12:49:39.113 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:49:39.108+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:49:39.108762+0000) 2024-03-04T12:49:40.113 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:49:40.108+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:49:40.109002+0000) 2024-03-04T12:49:41.113 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:49:41.108+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:49:41.109255+0000) 2024-03-04T12:49:42.114 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:49:42.108+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:49:42.109446+0000) 2024-03-04T12:49:43.114 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:49:43.108+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:49:43.109642+0000) 2024-03-04T12:49:44.114 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:49:44.109+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:49:44.109906+0000) 2024-03-04T12:49:45.114 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:49:45.109+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:49:45.110104+0000) 2024-03-04T12:49:46.115 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:49:46.109+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:49:46.110497+0000) 2024-03-04T12:49:47.116 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:49:47.109+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:49:47.110752+0000) 2024-03-04T12:49:48.115 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:49:48.110+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:49:48.110997+0000) 2024-03-04T12:49:49.116 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:49:49.110+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:49:49.111303+0000) 2024-03-04T12:49:50.116 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:49:50.111+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:49:50.111561+0000) 2024-03-04T12:49:51.116 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:49:51.111+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:49:51.111916+0000) 2024-03-04T12:49:52.116 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:49:52.111+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:49:52.112216+0000) 2024-03-04T12:49:53.117 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:49:53.112+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:49:53.112504+0000) 2024-03-04T12:49:53.957 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:49:53.967 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:49:54.117 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:49:54.112+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:49:54.112813+0000) 2024-03-04T12:49:55.117 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:49:55.112+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:49:55.113130+0000) 2024-03-04T12:49:56.118 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:49:56.113+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:49:56.113429+0000) 2024-03-04T12:49:57.118 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:49:57.113+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:49:57.113709+0000) 2024-03-04T12:49:58.118 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:49:58.113+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:49:58.113968+0000) 2024-03-04T12:49:59.119 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:49:59.113+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:49:59.114227+0000) 2024-03-04T12:50:00.119 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:50:00.113+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:50:00.114464+0000) 2024-03-04T12:50:01.119 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:50:01.112+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:50:01.114727+0000) 2024-03-04T12:50:02.119 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:50:02.113+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:50:02.114963+0000) 2024-03-04T12:50:03.119 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:50:03.114+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:50:03.115252+0000) 2024-03-04T12:50:04.120 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:50:04.114+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:50:04.115490+0000) 2024-03-04T12:50:05.120 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:50:05.113+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:50:05.115731+0000) 2024-03-04T12:50:06.120 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:50:06.114+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:50:06.115970+0000) 2024-03-04T12:50:07.120 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:50:07.115+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:50:07.116215+0000) 2024-03-04T12:50:08.121 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:50:08.115+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:50:08.116458+0000) 2024-03-04T12:50:09.121 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:50:09.115+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:50:09.116662+0000) 2024-03-04T12:50:10.121 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:50:10.116+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:50:10.116953+0000) 2024-03-04T12:50:11.121 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:50:11.116+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:50:11.117211+0000) 2024-03-04T12:50:12.122 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:50:12.116+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:50:12.117439+0000) 2024-03-04T12:50:13.122 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:50:13.116+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:50:13.117621+0000) 2024-03-04T12:50:14.122 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:50:14.117+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:50:14.117918+0000) 2024-03-04T12:50:15.122 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:50:15.117+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:50:15.118153+0000) 2024-03-04T12:50:16.123 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:50:16.117+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:50:16.118412+0000) 2024-03-04T12:50:17.123 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:50:17.117+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:50:17.118653+0000) 2024-03-04T12:50:18.123 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:50:18.118+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:50:18.118935+0000) 2024-03-04T12:50:19.124 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:50:19.118+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:50:19.119177+0000) 2024-03-04T12:50:20.124 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:50:20.118+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:50:20.119408+0000) 2024-03-04T12:50:21.124 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:50:21.118+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:50:21.119669+0000) 2024-03-04T12:50:22.124 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:50:22.119+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:50:22.119941+0000) 2024-03-04T12:50:23.124 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:50:23.119+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:50:23.120174+0000) 2024-03-04T12:50:24.093 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:50:24.101 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:50:24.124 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:50:24.119+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:50:24.120358+0000) 2024-03-04T12:50:25.125 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:50:25.119+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:50:25.120581+0000) 2024-03-04T12:50:26.125 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:50:26.120+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:50:26.120811+0000) 2024-03-04T12:50:27.125 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:50:27.120+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:50:27.121107+0000) 2024-03-04T12:50:28.126 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:50:28.120+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:50:28.121350+0000) 2024-03-04T12:50:29.126 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:50:29.120+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:50:29.121599+0000) 2024-03-04T12:50:30.126 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:50:30.121+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:50:30.121876+0000) 2024-03-04T12:50:31.127 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:50:31.121+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:50:31.122137+0000) 2024-03-04T12:50:32.127 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:50:32.121+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:50:32.122370+0000) 2024-03-04T12:50:33.127 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:50:33.121+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:50:33.122622+0000) 2024-03-04T12:50:34.127 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:50:34.122+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:50:34.122896+0000) 2024-03-04T12:50:35.127 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:50:35.122+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:50:35.123160+0000) 2024-03-04T12:50:36.128 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:50:36.122+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:50:36.123380+0000) 2024-03-04T12:50:37.128 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:50:37.123+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:50:37.123620+0000) 2024-03-04T12:50:38.128 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:50:38.123+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:50:38.123913+0000) 2024-03-04T12:50:39.129 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:50:39.123+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:50:39.124152+0000) 2024-03-04T12:50:40.129 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:50:40.123+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:50:40.124389+0000) 2024-03-04T12:50:41.129 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:50:41.124+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:50:41.124630+0000) 2024-03-04T12:50:42.129 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:50:42.124+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:50:42.124919+0000) 2024-03-04T12:50:43.129 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:50:43.124+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:50:43.125107+0000) 2024-03-04T12:50:44.130 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:50:44.124+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:50:44.125373+0000) 2024-03-04T12:50:45.130 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:50:45.125+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:50:45.125612+0000) 2024-03-04T12:50:46.130 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:50:46.125+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:50:46.125893+0000) 2024-03-04T12:50:47.130 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:50:47.125+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:50:47.126130+0000) 2024-03-04T12:50:48.131 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:50:48.125+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:50:48.126312+0000) 2024-03-04T12:50:49.131 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:50:49.125+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:50:49.126505+0000) 2024-03-04T12:50:50.131 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:50:50.126+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:50:50.126747+0000) 2024-03-04T12:50:51.131 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:50:51.126+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:50:51.127048+0000) 2024-03-04T12:50:52.132 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:50:52.126+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:50:52.127235+0000) 2024-03-04T12:50:53.132 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:50:53.126+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:50:53.127416+0000) 2024-03-04T12:50:54.132 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:50:54.126+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:50:54.127600+0000) 2024-03-04T12:50:54.258 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:50:54.267 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:50:55.132 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:50:55.127+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:50:55.127956+0000) 2024-03-04T12:50:56.133 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:50:56.128+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:50:56.128255+0000) 2024-03-04T12:50:57.133 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:50:57.128+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:50:57.128540+0000) 2024-03-04T12:50:58.133 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:50:58.128+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:50:58.128782+0000) 2024-03-04T12:50:59.133 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:50:59.128+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:50:59.129015+0000) 2024-03-04T12:51:00.134 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:51:00.129+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:51:00.129260+0000) 2024-03-04T12:51:01.134 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:51:01.127+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:51:01.129446+0000) 2024-03-04T12:51:02.134 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:51:02.127+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:51:02.129687+0000) 2024-03-04T12:51:03.134 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:51:03.128+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:51:03.129923+0000) 2024-03-04T12:51:04.135 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:51:04.129+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:51:04.130164+0000) 2024-03-04T12:51:05.135 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:51:05.129+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:51:05.130403+0000) 2024-03-04T12:51:06.135 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:51:06.129+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:51:06.130663+0000) 2024-03-04T12:51:07.135 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:51:07.129+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:51:07.130968+0000) 2024-03-04T12:51:08.136 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:51:08.130+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:51:08.131213+0000) 2024-03-04T12:51:09.136 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:51:09.130+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:51:09.131451+0000) 2024-03-04T12:51:10.136 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:51:10.130+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:51:10.131689+0000) 2024-03-04T12:51:11.136 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:51:11.130+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:51:11.131931+0000) 2024-03-04T12:51:12.136 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:51:12.131+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:51:12.132171+0000) 2024-03-04T12:51:13.137 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:51:13.131+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:51:13.132409+0000) 2024-03-04T12:51:14.137 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:51:14.131+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:51:14.132654+0000) 2024-03-04T12:51:15.137 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:51:15.132+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:51:15.132933+0000) 2024-03-04T12:51:16.137 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:51:16.132+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:51:16.133178+0000) 2024-03-04T12:51:17.138 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:51:17.132+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:51:17.133419+0000) 2024-03-04T12:51:18.138 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:51:18.132+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:51:18.133673+0000) 2024-03-04T12:51:19.138 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:51:19.133+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:51:19.133938+0000) 2024-03-04T12:51:20.138 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:51:20.133+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:51:20.134179+0000) 2024-03-04T12:51:21.139 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:51:21.133+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:51:21.134422+0000) 2024-03-04T12:51:22.139 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:51:22.133+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:51:22.134658+0000) 2024-03-04T12:51:23.139 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:51:23.134+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:51:23.134939+0000) 2024-03-04T12:51:24.139 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:51:24.134+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:51:24.135176+0000) 2024-03-04T12:51:24.455 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:51:24.463 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:51:25.140 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:51:25.134+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:51:25.135367+0000) 2024-03-04T12:51:26.140 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:51:26.134+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:51:26.135624+0000) 2024-03-04T12:51:27.140 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:51:27.135+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:51:27.135921+0000) 2024-03-04T12:51:28.141 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:51:28.135+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:51:28.136169+0000) 2024-03-04T12:51:29.141 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:51:29.135+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:51:29.136429+0000) 2024-03-04T12:51:30.141 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:51:30.136+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:51:30.136703+0000) 2024-03-04T12:51:31.141 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:51:31.136+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:51:31.136955+0000) 2024-03-04T12:51:32.141 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:51:32.136+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:51:32.137189+0000) 2024-03-04T12:51:33.142 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:51:33.136+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:51:33.137445+0000) 2024-03-04T12:51:34.142 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:51:34.137+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:51:34.137687+0000) 2024-03-04T12:51:35.142 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:51:35.137+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:51:35.137926+0000) 2024-03-04T12:51:36.143 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:51:36.137+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:51:36.138162+0000) 2024-03-04T12:51:37.143 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:51:37.137+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:51:37.138422+0000) 2024-03-04T12:51:38.143 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:51:38.138+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:51:38.138592+0000) 2024-03-04T12:51:39.143 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:51:39.138+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:51:39.138868+0000) 2024-03-04T12:51:40.143 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:51:40.138+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:51:40.139113+0000) 2024-03-04T12:51:41.144 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:51:41.138+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:51:41.139358+0000) 2024-03-04T12:51:42.144 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:51:42.139+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:51:42.139616+0000) 2024-03-04T12:51:43.144 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:51:43.139+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:51:43.139895+0000) 2024-03-04T12:51:44.144 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:51:44.139+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:51:44.140086+0000) 2024-03-04T12:51:45.145 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:51:45.139+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:51:45.140407+0000) 2024-03-04T12:51:46.145 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:51:46.139+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:51:46.140638+0000) 2024-03-04T12:51:47.145 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:51:47.140+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:51:47.140913+0000) 2024-03-04T12:51:48.145 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:51:48.140+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:51:48.141152+0000) 2024-03-04T12:51:49.146 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:51:49.141+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:51:49.141397+0000) 2024-03-04T12:51:50.146 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:51:50.141+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:51:50.141637+0000) 2024-03-04T12:51:51.146 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:51:51.141+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:51:51.141931+0000) 2024-03-04T12:51:52.147 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:51:52.141+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:51:52.142168+0000) 2024-03-04T12:51:53.147 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:51:53.141+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:51:53.142403+0000) 2024-03-04T12:51:54.147 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:51:54.141+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:51:54.142588+0000) 2024-03-04T12:51:54.659 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:51:54.667 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:51:55.147 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:51:55.142+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:51:55.142877+0000) 2024-03-04T12:51:56.148 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:51:56.142+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:51:56.143141+0000) 2024-03-04T12:51:57.148 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:51:57.143+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:51:57.143405+0000) 2024-03-04T12:51:58.148 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:51:58.143+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:51:58.143650+0000) 2024-03-04T12:51:59.148 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:51:59.143+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:51:59.143947+0000) 2024-03-04T12:52:00.149 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:52:00.144+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:52:00.144216+0000) 2024-03-04T12:52:01.152 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:52:01.144+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:52:01.144461+0000) 2024-03-04T12:52:02.149 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:52:02.144+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:52:02.144699+0000) 2024-03-04T12:52:03.149 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:52:03.143+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:52:03.144962+0000) 2024-03-04T12:52:04.150 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:52:04.143+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:52:04.145198+0000) 2024-03-04T12:52:05.150 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:52:05.144+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:52:05.145436+0000) 2024-03-04T12:52:06.150 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:52:06.144+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:52:06.145679+0000) 2024-03-04T12:52:07.150 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:52:07.144+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:52:07.145925+0000) 2024-03-04T12:52:08.151 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:52:08.145+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:52:08.146167+0000) 2024-03-04T12:52:09.151 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:52:09.145+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:52:09.146410+0000) 2024-03-04T12:52:10.151 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:52:10.145+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:52:10.146649+0000) 2024-03-04T12:52:11.151 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:52:11.145+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:52:11.146949+0000) 2024-03-04T12:52:12.152 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:52:12.146+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:52:12.147181+0000) 2024-03-04T12:52:13.152 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:52:13.146+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:52:13.147417+0000) 2024-03-04T12:52:14.153 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:52:14.146+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:52:14.147655+0000) 2024-03-04T12:52:15.152 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:52:15.147+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:52:15.147963+0000) 2024-03-04T12:52:16.152 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:52:16.147+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:52:16.148203+0000) 2024-03-04T12:52:17.153 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:52:17.147+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:52:17.148444+0000) 2024-03-04T12:52:18.153 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:52:18.147+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:52:18.148724+0000) 2024-03-04T12:52:19.153 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:52:19.148+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:52:19.148965+0000) 2024-03-04T12:52:20.154 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:52:20.148+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:52:20.149213+0000) 2024-03-04T12:52:21.154 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:52:21.148+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:52:21.149464+0000) 2024-03-04T12:52:22.154 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:52:22.148+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:52:22.149697+0000) 2024-03-04T12:52:23.154 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:52:23.149+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:52:23.149964+0000) 2024-03-04T12:52:24.155 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:52:24.149+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:52:24.150205+0000) 2024-03-04T12:52:24.873 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:52:24.881 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:52:25.155 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:52:25.149+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:52:25.150458+0000) 2024-03-04T12:52:26.155 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:52:26.149+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:52:26.150718+0000) 2024-03-04T12:52:27.155 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:52:27.150+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:52:27.150980+0000) 2024-03-04T12:52:28.156 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:52:28.150+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:52:28.151232+0000) 2024-03-04T12:52:29.156 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:52:29.150+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:52:29.151478+0000) 2024-03-04T12:52:30.156 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:52:30.150+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:52:30.151735+0000) 2024-03-04T12:52:31.156 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:52:31.151+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:52:31.151996+0000) 2024-03-04T12:52:32.157 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:52:32.151+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:52:32.152230+0000) 2024-03-04T12:52:33.157 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:52:33.151+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:52:33.152515+0000) 2024-03-04T12:52:34.157 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:52:34.152+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:52:34.152754+0000) 2024-03-04T12:52:35.157 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:52:35.152+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:52:35.153003+0000) 2024-03-04T12:52:36.158 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:52:36.152+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:52:36.153241+0000) 2024-03-04T12:52:37.158 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:52:37.152+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:52:37.153490+0000) 2024-03-04T12:52:38.158 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:52:38.152+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:52:38.153694+0000) 2024-03-04T12:52:39.158 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:52:39.153+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:52:39.153943+0000) 2024-03-04T12:52:40.159 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:52:40.153+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:52:40.154180+0000) 2024-03-04T12:52:41.159 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:52:41.153+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:52:41.154427+0000) 2024-03-04T12:52:42.159 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:52:42.153+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:52:42.154661+0000) 2024-03-04T12:52:43.159 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:52:43.154+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:52:43.154946+0000) 2024-03-04T12:52:44.160 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:52:44.154+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:52:44.155147+0000) 2024-03-04T12:52:45.160 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:52:45.154+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:52:45.155394+0000) 2024-03-04T12:52:46.160 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:52:46.155+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:52:46.155639+0000) 2024-03-04T12:52:47.160 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:52:47.155+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:52:47.155921+0000) 2024-03-04T12:52:48.161 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:52:48.155+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:52:48.156169+0000) 2024-03-04T12:52:49.161 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:52:49.156+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:52:49.156414+0000) 2024-03-04T12:52:50.161 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:52:50.156+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:52:50.156665+0000) 2024-03-04T12:52:51.161 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:52:51.156+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:52:51.156888+0000) 2024-03-04T12:52:52.161 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:52:52.156+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:52:52.157090+0000) 2024-03-04T12:52:53.162 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:52:53.157+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:52:53.157369+0000) 2024-03-04T12:52:54.162 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:52:54.157+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:52:54.157615+0000) 2024-03-04T12:52:55.056 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:52:55.063 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:52:55.192 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:52:55.157+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:52:55.157913+0000) 2024-03-04T12:52:56.163 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:52:56.157+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:52:56.158153+0000) 2024-03-04T12:52:57.163 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:52:57.157+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:52:57.158393+0000) 2024-03-04T12:52:58.163 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:52:58.158+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:52:58.158665+0000) 2024-03-04T12:52:59.163 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:52:59.158+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:52:59.158948+0000) 2024-03-04T12:53:00.165 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:53:00.158+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:53:00.159192+0000) 2024-03-04T12:53:01.164 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:53:01.159+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:53:01.159440+0000) 2024-03-04T12:53:02.164 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:53:02.158+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:53:02.159682+0000) 2024-03-04T12:53:03.164 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:53:03.159+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:53:03.159954+0000) 2024-03-04T12:53:04.165 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:53:04.160+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:53:04.160199+0000) 2024-03-04T12:53:05.165 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:53:05.158+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:53:05.160404+0000) 2024-03-04T12:53:06.165 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:53:06.159+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:53:06.160649+0000) 2024-03-04T12:53:07.165 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:53:07.159+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:53:07.160926+0000) 2024-03-04T12:53:08.166 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:53:08.160+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:53:08.161173+0000) 2024-03-04T12:53:09.166 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:53:09.159+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:53:09.161418+0000) 2024-03-04T12:53:10.166 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:53:10.160+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:53:10.161668+0000) 2024-03-04T12:53:11.166 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:53:11.160+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:53:11.161949+0000) 2024-03-04T12:53:12.167 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:53:12.160+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:53:12.162144+0000) 2024-03-04T12:53:13.167 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:53:13.161+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:53:13.162409+0000) 2024-03-04T12:53:14.167 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:53:14.161+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:53:14.162675+0000) 2024-03-04T12:53:15.167 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:53:15.161+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:53:15.162928+0000) 2024-03-04T12:53:16.167 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:53:16.162+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:53:16.163160+0000) 2024-03-04T12:53:17.168 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:53:17.162+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:53:17.163405+0000) 2024-03-04T12:53:18.168 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:53:18.162+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:53:18.163655+0000) 2024-03-04T12:53:19.168 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:53:19.163+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:53:19.163957+0000) 2024-03-04T12:53:20.169 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:53:20.163+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:53:20.164275+0000) 2024-03-04T12:53:21.169 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:53:21.163+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:53:21.164574+0000) 2024-03-04T12:53:22.169 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:53:22.164+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:53:22.164936+0000) 2024-03-04T12:53:23.170 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:53:23.164+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:53:23.165178+0000) 2024-03-04T12:53:24.170 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:53:24.164+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:53:24.165477+0000) 2024-03-04T12:53:25.170 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:53:25.164+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:53:25.165703+0000) 2024-03-04T12:53:25.257 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:53:25.265 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:53:26.170 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:53:26.165+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:53:26.165989+0000) 2024-03-04T12:53:27.171 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:53:27.165+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:53:27.166277+0000) 2024-03-04T12:53:28.171 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:53:28.165+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:53:28.166580+0000) 2024-03-04T12:53:29.171 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:53:29.166+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:53:29.166917+0000) 2024-03-04T12:53:30.172 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:53:30.166+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:53:30.167164+0000) 2024-03-04T12:53:31.172 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:53:31.166+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:53:31.167534+0000) 2024-03-04T12:53:32.172 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:53:32.167+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:53:32.167889+0000) 2024-03-04T12:53:33.173 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:53:33.167+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:53:33.168214+0000) 2024-03-04T12:53:34.173 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:53:34.167+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:53:34.168462+0000) 2024-03-04T12:53:35.173 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:53:35.168+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:53:35.168753+0000) 2024-03-04T12:53:36.173 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:53:36.168+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:53:36.169048+0000) 2024-03-04T12:53:37.174 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:53:37.168+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:53:37.169351+0000) 2024-03-04T12:53:38.174 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:53:38.168+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:53:38.169676+0000) 2024-03-04T12:53:39.174 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:53:39.169+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:53:39.169983+0000) 2024-03-04T12:53:40.175 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:53:40.169+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:53:40.170232+0000) 2024-03-04T12:53:41.175 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:53:41.169+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:53:41.170507+0000) 2024-03-04T12:53:42.175 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:53:42.170+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:53:42.170807+0000) 2024-03-04T12:53:43.175 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:53:43.170+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:53:43.171106+0000) 2024-03-04T12:53:44.176 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:53:44.170+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:53:44.171417+0000) 2024-03-04T12:53:45.176 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:53:45.170+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:53:45.171705+0000) 2024-03-04T12:53:46.176 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:53:46.171+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:53:46.171982+0000) 2024-03-04T12:53:47.177 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:53:47.171+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:53:47.172235+0000) 2024-03-04T12:53:48.177 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:53:48.171+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:53:48.172518+0000) 2024-03-04T12:53:49.177 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:53:49.171+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:53:49.172792+0000) 2024-03-04T12:53:50.177 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:53:50.172+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:53:50.172996+0000) 2024-03-04T12:53:51.178 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:53:51.172+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:53:51.173229+0000) 2024-03-04T12:53:52.178 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:53:52.173+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:53:52.173484+0000) 2024-03-04T12:53:53.178 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:53:53.172+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:53:53.173747+0000) 2024-03-04T12:53:54.179 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:53:54.173+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:53:54.173991+0000) 2024-03-04T12:53:55.179 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:53:55.173+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:53:55.174228+0000) 2024-03-04T12:53:55.398 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:53:55.406 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:53:56.179 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:53:56.174+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:53:56.174468+0000) 2024-03-04T12:53:57.179 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:53:57.173+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:53:57.174716+0000) 2024-03-04T12:53:58.180 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:53:58.174+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:53:58.174965+0000) 2024-03-04T12:53:59.180 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:53:59.174+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:53:59.175205+0000) 2024-03-04T12:54:00.180 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:54:00.175+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:54:00.175445+0000) 2024-03-04T12:54:01.180 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:54:01.175+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:54:01.175679+0000) 2024-03-04T12:54:02.180 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:54:02.175+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:54:02.175960+0000) 2024-03-04T12:54:03.181 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:54:03.175+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:54:03.176145+0000) 2024-03-04T12:54:04.181 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:54:04.176+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:54:04.176397+0000) 2024-03-04T12:54:05.181 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:54:05.176+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:54:05.176636+0000) 2024-03-04T12:54:06.181 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:54:06.176+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:54:06.176929+0000) 2024-03-04T12:54:07.182 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:54:07.177+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:54:07.177165+0000) 2024-03-04T12:54:08.182 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:54:08.176+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:54:08.177405+0000) 2024-03-04T12:54:09.182 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:54:09.176+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:54:09.177648+0000) 2024-03-04T12:54:10.182 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:54:10.176+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:54:10.177944+0000) 2024-03-04T12:54:11.183 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:54:11.176+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:54:11.178130+0000) 2024-03-04T12:54:12.183 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:54:12.177+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:54:12.178369+0000) 2024-03-04T12:54:13.183 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:54:13.177+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:54:13.178599+0000) 2024-03-04T12:54:14.183 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:54:14.177+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:54:14.178909+0000) 2024-03-04T12:54:15.183 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:54:15.178+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:54:15.179100+0000) 2024-03-04T12:54:16.184 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:54:16.178+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:54:16.179294+0000) 2024-03-04T12:54:17.184 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:54:17.178+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:54:17.179528+0000) 2024-03-04T12:54:18.184 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:54:18.178+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:54:18.179779+0000) 2024-03-04T12:54:19.184 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:54:19.179+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:54:19.180036+0000) 2024-03-04T12:54:20.185 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:54:20.179+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:54:20.180307+0000) 2024-03-04T12:54:21.185 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:54:21.179+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:54:21.180539+0000) 2024-03-04T12:54:22.185 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:54:22.179+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:54:22.180773+0000) 2024-03-04T12:54:23.186 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:54:23.180+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:54:23.181022+0000) 2024-03-04T12:54:24.186 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:54:24.180+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:54:24.181290+0000) 2024-03-04T12:54:25.186 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:54:25.180+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:54:25.181574+0000) 2024-03-04T12:54:25.551 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:54:25.559 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:54:26.186 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:54:26.181+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:54:26.181817+0000) 2024-03-04T12:54:27.186 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:54:27.181+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:54:27.182097+0000) 2024-03-04T12:54:28.187 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:54:28.181+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:54:28.182338+0000) 2024-03-04T12:54:29.187 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:54:29.181+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:54:29.182581+0000) 2024-03-04T12:54:30.187 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:54:30.182+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:54:30.182875+0000) 2024-03-04T12:54:31.188 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:54:31.182+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:54:31.183068+0000) 2024-03-04T12:54:32.188 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:54:32.182+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:54:32.183308+0000) 2024-03-04T12:54:33.188 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:54:33.182+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:54:33.183541+0000) 2024-03-04T12:54:34.188 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:54:34.183+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:54:34.183775+0000) 2024-03-04T12:54:35.188 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:54:35.183+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:54:35.183966+0000) 2024-03-04T12:54:36.189 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:54:36.183+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:54:36.184203+0000) 2024-03-04T12:54:37.189 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:54:37.183+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:54:37.184447+0000) 2024-03-04T12:54:38.189 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:54:38.184+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:54:38.184689+0000) 2024-03-04T12:54:39.189 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:54:39.184+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:54:39.184953+0000) 2024-03-04T12:54:40.190 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:54:40.184+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:54:40.185203+0000) 2024-03-04T12:54:41.190 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:54:41.184+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:54:41.185439+0000) 2024-03-04T12:54:42.190 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:54:42.184+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:54:42.185618+0000) 2024-03-04T12:54:43.190 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:54:43.185+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:54:43.185895+0000) 2024-03-04T12:54:44.191 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:54:44.185+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:54:44.186133+0000) 2024-03-04T12:54:45.191 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:54:45.185+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:54:45.186362+0000) 2024-03-04T12:54:46.191 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:54:46.186+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:54:46.186601+0000) 2024-03-04T12:54:47.191 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:54:47.186+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:54:47.186893+0000) 2024-03-04T12:54:48.192 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:54:48.186+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:54:48.187136+0000) 2024-03-04T12:54:49.192 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:54:49.186+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:54:49.187375+0000) 2024-03-04T12:54:50.192 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:54:50.187+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:54:50.187623+0000) 2024-03-04T12:54:51.192 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:54:51.187+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:54:51.187917+0000) 2024-03-04T12:54:52.193 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:54:52.187+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:54:52.188160+0000) 2024-03-04T12:54:53.193 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:54:53.187+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:54:53.188393+0000) 2024-03-04T12:54:54.193 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:54:54.188+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:54:54.188650+0000) 2024-03-04T12:54:55.193 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:54:55.188+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:54:55.188941+0000) 2024-03-04T12:54:55.765 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:54:55.773 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:54:56.194 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:54:56.188+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:54:56.189185+0000) 2024-03-04T12:54:57.194 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:54:57.189+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:54:57.189421+0000) 2024-03-04T12:54:58.194 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:54:58.188+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:54:58.189688+0000) 2024-03-04T12:54:59.194 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:54:59.189+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:54:59.189903+0000) 2024-03-04T12:55:00.195 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:55:00.189+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:55:00.190141+0000) 2024-03-04T12:55:01.195 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:55:01.190+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:55:01.190377+0000) 2024-03-04T12:55:02.195 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:55:02.190+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:55:02.190613+0000) 2024-03-04T12:55:03.195 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:55:03.190+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:55:03.190891+0000) 2024-03-04T12:55:04.196 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:55:04.190+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:55:04.191141+0000) 2024-03-04T12:55:05.196 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:55:05.191+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:55:05.191374+0000) 2024-03-04T12:55:06.196 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:55:06.191+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:55:06.191613+0000) 2024-03-04T12:55:07.196 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:55:07.191+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:55:07.191915+0000) 2024-03-04T12:55:08.197 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:55:08.192+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:55:08.192173+0000) 2024-03-04T12:55:09.197 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:55:09.192+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:55:09.192409+0000) 2024-03-04T12:55:10.197 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:55:10.192+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:55:10.192648+0000) 2024-03-04T12:55:11.197 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:55:11.191+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:55:11.192929+0000) 2024-03-04T12:55:12.198 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:55:12.191+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:55:12.193159+0000) 2024-03-04T12:55:13.200 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:55:13.191+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:55:13.193420+0000) 2024-03-04T12:55:14.198 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:55:14.192+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:55:14.193657+0000) 2024-03-04T12:55:15.198 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:55:15.192+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:55:15.193928+0000) 2024-03-04T12:55:16.199 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:55:16.192+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:55:16.194110+0000) 2024-03-04T12:55:17.199 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:55:17.193+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:55:17.194337+0000) 2024-03-04T12:55:18.199 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:55:18.192+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:55:18.194600+0000) 2024-03-04T12:55:19.199 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:55:19.193+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:55:19.194910+0000) 2024-03-04T12:55:20.200 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:55:20.194+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:55:20.195166+0000) 2024-03-04T12:55:21.200 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:55:21.194+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:55:21.195400+0000) 2024-03-04T12:55:22.200 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:55:22.194+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:55:22.195637+0000) 2024-03-04T12:55:23.200 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:55:23.195+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:55:23.195980+0000) 2024-03-04T12:55:24.201 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:55:24.195+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:55:24.196237+0000) 2024-03-04T12:55:25.201 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:55:25.195+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:55:25.196479+0000) 2024-03-04T12:55:25.953 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:55:25.961 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:55:26.201 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:55:26.195+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:55:26.196752+0000) 2024-03-04T12:55:27.201 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:55:27.196+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:55:27.196989+0000) 2024-03-04T12:55:28.202 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:55:28.196+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:55:28.197252+0000) 2024-03-04T12:55:29.202 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:55:29.196+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:55:29.197488+0000) 2024-03-04T12:55:30.202 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:55:30.196+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:55:30.197744+0000) 2024-03-04T12:55:31.203 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:55:31.197+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:55:31.197986+0000) 2024-03-04T12:55:32.203 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:55:32.197+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:55:32.198224+0000) 2024-03-04T12:55:33.203 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:55:33.197+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:55:33.198464+0000) 2024-03-04T12:55:34.203 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:55:34.197+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:55:34.198702+0000) 2024-03-04T12:55:35.203 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:55:35.198+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:55:35.198928+0000) 2024-03-04T12:55:36.204 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:55:36.198+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:55:36.199161+0000) 2024-03-04T12:55:37.204 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:55:37.198+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:55:37.199402+0000) 2024-03-04T12:55:38.204 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:55:38.198+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:55:38.199645+0000) 2024-03-04T12:55:39.204 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:55:39.199+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:55:39.199915+0000) 2024-03-04T12:55:40.205 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:55:40.199+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:55:40.200154+0000) 2024-03-04T12:55:41.205 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:55:41.199+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:55:41.200409+0000) 2024-03-04T12:55:42.205 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:55:42.199+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:55:42.200632+0000) 2024-03-04T12:55:43.205 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:55:43.200+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:55:43.200919+0000) 2024-03-04T12:55:44.206 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:55:44.200+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:55:44.201144+0000) 2024-03-04T12:55:45.206 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:55:45.200+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:55:45.201379+0000) 2024-03-04T12:55:46.206 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:55:46.201+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:55:46.201621+0000) 2024-03-04T12:55:47.206 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:55:47.201+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:55:47.201888+0000) 2024-03-04T12:55:48.207 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:55:48.201+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:55:48.202131+0000) 2024-03-04T12:55:49.207 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:55:49.201+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:55:49.202365+0000) 2024-03-04T12:55:50.207 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:55:50.201+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:55:50.202624+0000) 2024-03-04T12:55:51.208 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:55:51.202+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:55:51.202927+0000) 2024-03-04T12:55:52.208 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:55:52.202+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:55:52.203154+0000) 2024-03-04T12:55:53.208 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:55:53.202+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:55:53.203392+0000) 2024-03-04T12:55:54.208 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:55:54.203+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:55:54.203652+0000) 2024-03-04T12:55:55.208 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:55:55.203+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:55:55.203932+0000) 2024-03-04T12:55:56.168 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:55:56.177 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:55:56.209 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:55:56.203+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:55:56.204143+0000) 2024-03-04T12:55:57.209 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:55:57.203+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:55:57.204326+0000) 2024-03-04T12:55:58.209 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:55:58.204+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:55:58.204576+0000) 2024-03-04T12:55:59.209 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:55:59.204+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:55:59.204810+0000) 2024-03-04T12:56:00.210 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:56:00.204+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:56:00.205111+0000) 2024-03-04T12:56:01.210 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:56:01.205+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:56:01.205352+0000) 2024-03-04T12:56:02.210 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:56:02.204+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:56:02.205623+0000) 2024-03-04T12:56:03.210 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:56:03.205+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:56:03.205926+0000) 2024-03-04T12:56:04.211 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:56:04.205+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:56:04.206166+0000) 2024-03-04T12:56:05.211 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:56:05.206+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:56:05.206403+0000) 2024-03-04T12:56:06.211 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:56:06.206+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:56:06.206645+0000) 2024-03-04T12:56:07.212 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:56:07.206+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:56:07.206931+0000) 2024-03-04T12:56:08.212 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:56:08.206+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:56:08.207177+0000) 2024-03-04T12:56:09.212 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:56:09.207+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:56:09.207412+0000) 2024-03-04T12:56:10.212 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:56:10.207+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:56:10.207660+0000) 2024-03-04T12:56:11.212 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:56:11.207+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:56:11.208003+0000) 2024-03-04T12:56:12.213 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:56:12.207+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:56:12.208235+0000) 2024-03-04T12:56:13.213 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:56:13.208+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:56:13.208476+0000) 2024-03-04T12:56:14.213 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:56:14.207+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:56:14.208713+0000) 2024-03-04T12:56:15.213 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:56:15.207+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:56:15.208947+0000) 2024-03-04T12:56:16.214 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:56:16.208+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:56:16.209188+0000) 2024-03-04T12:56:17.214 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:56:17.208+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:56:17.209428+0000) 2024-03-04T12:56:18.214 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:56:18.208+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:56:18.209674+0000) 2024-03-04T12:56:19.214 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:56:19.208+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:56:19.209952+0000) 2024-03-04T12:56:20.215 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:56:20.209+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:56:20.210195+0000) 2024-03-04T12:56:21.216 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:56:21.209+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:56:21.210432+0000) 2024-03-04T12:56:22.215 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:56:22.209+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:56:22.210621+0000) 2024-03-04T12:56:23.215 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:56:23.209+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:56:23.210908+0000) 2024-03-04T12:56:24.216 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:56:24.210+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:56:24.211171+0000) 2024-03-04T12:56:25.216 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:56:25.210+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:56:25.211398+0000) 2024-03-04T12:56:26.216 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:56:26.210+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:56:26.211659+0000) 2024-03-04T12:56:26.330 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:56:26.341 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:56:27.216 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:56:27.211+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:56:27.211916+0000) 2024-03-04T12:56:28.217 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:56:28.211+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:56:28.212167+0000) 2024-03-04T12:56:29.217 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:56:29.211+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:56:29.212407+0000) 2024-03-04T12:56:30.217 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:56:30.211+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:56:30.212648+0000) 2024-03-04T12:56:31.217 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:56:31.212+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:56:31.212930+0000) 2024-03-04T12:56:32.218 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:56:32.212+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:56:32.213112+0000) 2024-03-04T12:56:33.218 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:56:33.212+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:56:33.213347+0000) 2024-03-04T12:56:34.218 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:56:34.212+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:56:34.213592+0000) 2024-03-04T12:56:35.219 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:56:35.213+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:56:35.213859+0000) 2024-03-04T12:56:36.219 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:56:36.213+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:56:36.214050+0000) 2024-03-04T12:56:37.219 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:56:37.213+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:56:37.214290+0000) 2024-03-04T12:56:38.219 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:56:38.213+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:56:38.214543+0000) 2024-03-04T12:56:39.219 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:56:39.214+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:56:39.214777+0000) 2024-03-04T12:56:40.220 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:56:40.214+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:56:40.215022+0000) 2024-03-04T12:56:41.220 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:56:41.214+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:56:41.215272+0000) 2024-03-04T12:56:42.220 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:56:42.214+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:56:42.215513+0000) 2024-03-04T12:56:43.220 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:56:43.215+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:56:43.215752+0000) 2024-03-04T12:56:44.221 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:56:44.215+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:56:44.215967+0000) 2024-03-04T12:56:45.221 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:56:45.215+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:56:45.216222+0000) 2024-03-04T12:56:46.221 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:56:46.215+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:56:46.216465+0000) 2024-03-04T12:56:47.221 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:56:47.216+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:56:47.216716+0000) 2024-03-04T12:56:48.222 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:56:48.216+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:56:48.216969+0000) 2024-03-04T12:56:49.222 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:56:49.216+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:56:49.217225+0000) 2024-03-04T12:56:50.222 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:56:50.216+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:56:50.217471+0000) 2024-03-04T12:56:51.222 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:56:51.217+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:56:51.217713+0000) 2024-03-04T12:56:52.223 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:56:52.217+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:56:52.217971+0000) 2024-03-04T12:56:53.223 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:56:53.217+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:56:53.218232+0000) 2024-03-04T12:56:54.223 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:56:54.217+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:56:54.218418+0000) 2024-03-04T12:56:55.223 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:56:55.217+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:56:55.218648+0000) 2024-03-04T12:56:56.224 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:56:56.218+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:56:56.218954+0000) 2024-03-04T12:56:56.527 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:56:56.539 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:56:57.224 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:56:57.218+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:56:57.219198+0000) 2024-03-04T12:56:58.224 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:56:58.219+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:56:58.219491+0000) 2024-03-04T12:56:59.224 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:56:59.219+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:56:59.219780+0000) 2024-03-04T12:57:00.225 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:57:00.219+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:57:00.220026+0000) 2024-03-04T12:57:01.225 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:57:01.219+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:57:01.220263+0000) 2024-03-04T12:57:02.225 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:57:02.219+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:57:02.220493+0000) 2024-03-04T12:57:03.225 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:57:03.220+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:57:03.220736+0000) 2024-03-04T12:57:04.226 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:57:04.220+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:57:04.220978+0000) 2024-03-04T12:57:05.226 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:57:05.220+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:57:05.221168+0000) 2024-03-04T12:57:06.226 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:57:06.221+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:57:06.221409+0000) 2024-03-04T12:57:07.226 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:57:07.220+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:57:07.221636+0000) 2024-03-04T12:57:08.226 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:57:08.221+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:57:08.221872+0000) 2024-03-04T12:57:09.227 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:57:09.221+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:57:09.222076+0000) 2024-03-04T12:57:10.227 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:57:10.222+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:57:10.222317+0000) 2024-03-04T12:57:11.227 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:57:11.222+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:57:11.222556+0000) 2024-03-04T12:57:12.228 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:57:12.221+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:57:12.222791+0000) 2024-03-04T12:57:13.228 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:57:13.222+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:57:13.223025+0000) 2024-03-04T12:57:14.228 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:57:14.223+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:57:14.223289+0000) 2024-03-04T12:57:15.228 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:57:15.223+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:57:15.223533+0000) 2024-03-04T12:57:16.228 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:57:16.222+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:57:16.223785+0000) 2024-03-04T12:57:17.229 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:57:17.222+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:57:17.224016+0000) 2024-03-04T12:57:18.229 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:57:18.223+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:57:18.224265+0000) 2024-03-04T12:57:19.229 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:57:19.222+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:57:19.224513+0000) 2024-03-04T12:57:20.229 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:57:20.223+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:57:20.224723+0000) 2024-03-04T12:57:21.230 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:57:21.223+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:57:21.224966+0000) 2024-03-04T12:57:22.230 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:57:22.223+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:57:22.225200+0000) 2024-03-04T12:57:23.230 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:57:23.224+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:57:23.225435+0000) 2024-03-04T12:57:24.230 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:57:24.223+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:57:24.225620+0000) 2024-03-04T12:57:25.230 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:57:25.224+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:57:25.225915+0000) 2024-03-04T12:57:26.231 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:57:26.225+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:57:26.226171+0000) 2024-03-04T12:57:26.733 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:57:26.741 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:57:27.231 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:57:27.225+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:57:27.226383+0000) 2024-03-04T12:57:28.231 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:57:28.225+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:57:28.226670+0000) 2024-03-04T12:57:29.232 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:57:29.226+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:57:29.226946+0000) 2024-03-04T12:57:30.232 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:57:30.226+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:57:30.227187+0000) 2024-03-04T12:57:31.232 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:57:31.226+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:57:31.227447+0000) 2024-03-04T12:57:32.232 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:57:32.226+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:57:32.227683+0000) 2024-03-04T12:57:33.233 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:57:33.227+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:57:33.227942+0000) 2024-03-04T12:57:34.233 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:57:34.227+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:57:34.228145+0000) 2024-03-04T12:57:35.233 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:57:35.227+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:57:35.228337+0000) 2024-03-04T12:57:36.233 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:57:36.227+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:57:36.228575+0000) 2024-03-04T12:57:37.233 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:57:37.228+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:57:37.228814+0000) 2024-03-04T12:57:38.234 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:57:38.228+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:57:38.229121+0000) 2024-03-04T12:57:39.234 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:57:39.228+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:57:39.229360+0000) 2024-03-04T12:57:40.234 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:57:40.228+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:57:40.229601+0000) 2024-03-04T12:57:41.234 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:57:41.229+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:57:41.229876+0000) 2024-03-04T12:57:42.235 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:57:42.229+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:57:42.230117+0000) 2024-03-04T12:57:43.235 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:57:43.229+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:57:43.230353+0000) 2024-03-04T12:57:44.235 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:57:44.229+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:57:44.230597+0000) 2024-03-04T12:57:45.236 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:57:45.230+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:57:45.230912+0000) 2024-03-04T12:57:46.236 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:57:46.230+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:57:46.231100+0000) 2024-03-04T12:57:47.236 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:57:47.230+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:57:47.231334+0000) 2024-03-04T12:57:48.236 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:57:48.230+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:57:48.231604+0000) 2024-03-04T12:57:49.237 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:57:49.231+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:57:49.231874+0000) 2024-03-04T12:57:50.237 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:57:50.231+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:57:50.232118+0000) 2024-03-04T12:57:51.237 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:57:51.231+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:57:51.232372+0000) 2024-03-04T12:57:52.237 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:57:52.232+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:57:52.232605+0000) 2024-03-04T12:57:53.238 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:57:53.232+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:57:53.232927+0000) 2024-03-04T12:57:54.238 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:57:54.232+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:57:54.233169+0000) 2024-03-04T12:57:55.238 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:57:55.232+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:57:55.233404+0000) 2024-03-04T12:57:56.238 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:57:56.233+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:57:56.233643+0000) 2024-03-04T12:57:56.929 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:57:56.941 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:57:57.238 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:57:57.233+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:57:57.233918+0000) 2024-03-04T12:57:58.239 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:57:58.233+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:57:58.234185+0000) 2024-03-04T12:57:59.239 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:57:59.233+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:57:59.234423+0000) 2024-03-04T12:58:00.239 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:58:00.234+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:58:00.234677+0000) 2024-03-04T12:58:01.240 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:58:01.234+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:58:01.234928+0000) 2024-03-04T12:58:02.241 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:58:02.234+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:58:02.235133+0000) 2024-03-04T12:58:03.240 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:58:03.235+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:58:03.235375+0000) 2024-03-04T12:58:04.240 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:58:04.235+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:58:04.235605+0000) 2024-03-04T12:58:05.240 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:58:05.235+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:58:05.235895+0000) 2024-03-04T12:58:06.241 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:58:06.235+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:58:06.236136+0000) 2024-03-04T12:58:07.241 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:58:07.236+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:58:07.236372+0000) 2024-03-04T12:58:08.241 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:58:08.236+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:58:08.236629+0000) 2024-03-04T12:58:09.242 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:58:09.236+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:58:09.236921+0000) 2024-03-04T12:58:10.242 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:58:10.236+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:58:10.237166+0000) 2024-03-04T12:58:11.242 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:58:11.237+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:58:11.237398+0000) 2024-03-04T12:58:12.242 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:58:12.237+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:58:12.237667+0000) 2024-03-04T12:58:13.243 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:58:13.237+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:58:13.237948+0000) 2024-03-04T12:58:14.243 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:58:14.237+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:58:14.238173+0000) 2024-03-04T12:58:15.243 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:58:15.238+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:58:15.238405+0000) 2024-03-04T12:58:16.243 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:58:16.238+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:58:16.238654+0000) 2024-03-04T12:58:17.244 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:58:17.238+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:58:17.238942+0000) 2024-03-04T12:58:18.244 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:58:18.239+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:58:18.239190+0000) 2024-03-04T12:58:19.244 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:58:19.238+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:58:19.239425+0000) 2024-03-04T12:58:20.244 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:58:20.237+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:58:20.239613+0000) 2024-03-04T12:58:21.245 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:58:21.238+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:58:21.239882+0000) 2024-03-04T12:58:22.245 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:58:22.239+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:58:22.240123+0000) 2024-03-04T12:58:23.245 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:58:23.239+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:58:23.240369+0000) 2024-03-04T12:58:24.245 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:58:24.239+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:58:24.240614+0000) 2024-03-04T12:58:25.246 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:58:25.239+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:58:25.240922+0000) 2024-03-04T12:58:26.246 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:58:26.240+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:58:26.241180+0000) 2024-03-04T12:58:27.152 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:58:27.163 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:58:27.246 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:58:27.239+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:58:27.241361+0000) 2024-03-04T12:58:28.246 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:58:28.240+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:58:28.241628+0000) 2024-03-04T12:58:29.247 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:58:29.240+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:58:29.241912+0000) 2024-03-04T12:58:30.247 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:58:30.241+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:58:30.242161+0000) 2024-03-04T12:58:31.247 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:58:31.241+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:58:31.242399+0000) 2024-03-04T12:58:32.247 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:58:32.241+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:58:32.242639+0000) 2024-03-04T12:58:33.248 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:58:33.242+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:58:33.242933+0000) 2024-03-04T12:58:34.248 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:58:34.242+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:58:34.243186+0000) 2024-03-04T12:58:35.248 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:58:35.242+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:58:35.243441+0000) 2024-03-04T12:58:36.248 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:58:36.242+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:58:36.243634+0000) 2024-03-04T12:58:37.249 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:58:37.243+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:58:37.243926+0000) 2024-03-04T12:58:38.249 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:58:38.243+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:58:38.244170+0000) 2024-03-04T12:58:39.249 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:58:39.243+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:58:39.244405+0000) 2024-03-04T12:58:40.249 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:58:40.243+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:58:40.244651+0000) 2024-03-04T12:58:41.250 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:58:41.244+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:58:41.244907+0000) 2024-03-04T12:58:42.250 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:58:42.244+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:58:42.245120+0000) 2024-03-04T12:58:43.250 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:58:43.244+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:58:43.245355+0000) 2024-03-04T12:58:44.250 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:58:44.244+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:58:44.245591+0000) 2024-03-04T12:58:45.251 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:58:45.245+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:58:45.245862+0000) 2024-03-04T12:58:46.251 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:58:46.245+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:58:46.246069+0000) 2024-03-04T12:58:47.251 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:58:47.245+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:58:47.246301+0000) 2024-03-04T12:58:48.251 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:58:48.245+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:58:48.246583+0000) 2024-03-04T12:58:49.252 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:58:49.246+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:58:49.246854+0000) 2024-03-04T12:58:50.252 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:58:50.246+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:58:50.247111+0000) 2024-03-04T12:58:51.252 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:58:51.246+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:58:51.247356+0000) 2024-03-04T12:58:52.252 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:58:52.246+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:58:52.247589+0000) 2024-03-04T12:58:53.253 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:58:53.247+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:58:53.247856+0000) 2024-03-04T12:58:54.253 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:58:54.247+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:58:54.248091+0000) 2024-03-04T12:58:55.253 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:58:55.247+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:58:55.248328+0000) 2024-03-04T12:58:56.253 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:58:56.247+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:58:56.248567+0000) 2024-03-04T12:58:57.254 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:58:57.247+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:58:57.248798+0000) 2024-03-04T12:58:57.289 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:58:57.296 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:58:58.254 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:58:58.248+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:58:58.249113+0000) 2024-03-04T12:58:59.254 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:58:59.248+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:58:59.249351+0000) 2024-03-04T12:59:00.254 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:59:00.249+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:59:00.249603+0000) 2024-03-04T12:59:01.255 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:59:01.249+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:59:01.249876+0000) 2024-03-04T12:59:02.255 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:59:02.249+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:59:02.250073+0000) 2024-03-04T12:59:03.255 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:59:03.249+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:59:03.250306+0000) 2024-03-04T12:59:04.255 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:59:04.249+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:59:04.250562+0000) 2024-03-04T12:59:05.256 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:59:05.249+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:59:05.250796+0000) 2024-03-04T12:59:06.256 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:59:06.250+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:59:06.251037+0000) 2024-03-04T12:59:07.256 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:59:07.250+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:59:07.251268+0000) 2024-03-04T12:59:08.256 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:59:08.251+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:59:08.251516+0000) 2024-03-04T12:59:09.256 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:59:09.251+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:59:09.251755+0000) 2024-03-04T12:59:10.257 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:59:10.251+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:59:10.252031+0000) 2024-03-04T12:59:11.257 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:59:11.251+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:59:11.252224+0000) 2024-03-04T12:59:12.257 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:59:12.252+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:59:12.252496+0000) 2024-03-04T12:59:13.257 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:59:13.252+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:59:13.252731+0000) 2024-03-04T12:59:14.258 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:59:14.252+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:59:14.252974+0000) 2024-03-04T12:59:15.258 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:59:15.252+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:59:15.253215+0000) 2024-03-04T12:59:16.258 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:59:16.252+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:59:16.253417+0000) 2024-03-04T12:59:17.258 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:59:17.253+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:59:17.253653+0000) 2024-03-04T12:59:18.259 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:59:18.253+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:59:18.253939+0000) 2024-03-04T12:59:19.259 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:59:19.254+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:59:19.254176+0000) 2024-03-04T12:59:20.259 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:59:20.254+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:59:20.254407+0000) 2024-03-04T12:59:21.259 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:59:21.253+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:59:21.254601+0000) 2024-03-04T12:59:22.260 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:59:22.253+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:59:22.254896+0000) 2024-03-04T12:59:23.260 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:59:23.253+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:59:23.255125+0000) 2024-03-04T12:59:24.260 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:59:24.254+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:59:24.255361+0000) 2024-03-04T12:59:25.260 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:59:25.253+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:59:25.255611+0000) 2024-03-04T12:59:26.261 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:59:26.254+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:59:26.255900+0000) 2024-03-04T12:59:27.261 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:59:27.255+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:59:27.256133+0000) 2024-03-04T12:59:27.511 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:59:27.519 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:59:28.261 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:59:28.254+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:59:28.256406+0000) 2024-03-04T12:59:29.261 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:59:29.255+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:59:29.256665+0000) 2024-03-04T12:59:30.262 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:59:30.255+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:59:30.256907+0000) 2024-03-04T12:59:31.262 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:59:31.255+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:59:31.257110+0000) 2024-03-04T12:59:32.262 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:59:32.256+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:59:32.257346+0000) 2024-03-04T12:59:33.262 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:59:33.256+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:59:33.257582+0000) 2024-03-04T12:59:34.263 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:59:34.257+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:59:34.257860+0000) 2024-03-04T12:59:35.263 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:59:35.257+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:59:35.258098+0000) 2024-03-04T12:59:36.263 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:59:36.257+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:59:36.258357+0000) 2024-03-04T12:59:37.263 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:59:37.257+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:59:37.258607+0000) 2024-03-04T12:59:38.264 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:59:38.258+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:59:38.258900+0000) 2024-03-04T12:59:39.264 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:59:39.258+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:59:39.259142+0000) 2024-03-04T12:59:40.264 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:59:40.258+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:59:40.259378+0000) 2024-03-04T12:59:41.264 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:59:41.258+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:59:41.259628+0000) 2024-03-04T12:59:42.265 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:59:42.259+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:59:42.259910+0000) 2024-03-04T12:59:43.265 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:59:43.259+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:59:43.260148+0000) 2024-03-04T12:59:44.265 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:59:44.259+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:59:44.260414+0000) 2024-03-04T12:59:45.265 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:59:45.259+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:59:45.260702+0000) 2024-03-04T12:59:46.266 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:59:46.260+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:59:46.260955+0000) 2024-03-04T12:59:47.266 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:59:47.260+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:59:47.261196+0000) 2024-03-04T12:59:48.266 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:59:48.260+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:59:48.261444+0000) 2024-03-04T12:59:49.266 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:59:49.260+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:59:49.261680+0000) 2024-03-04T12:59:50.267 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:59:50.261+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:59:50.261913+0000) 2024-03-04T12:59:51.267 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:59:51.261+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:59:51.262100+0000) 2024-03-04T12:59:52.267 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:59:52.261+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:59:52.262345+0000) 2024-03-04T12:59:53.268 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:59:53.261+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:59:53.262584+0000) 2024-03-04T12:59:54.268 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:59:54.262+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:59:54.262858+0000) 2024-03-04T12:59:55.268 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:59:55.262+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:59:55.263094+0000) 2024-03-04T12:59:56.268 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:59:56.262+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:59:56.263341+0000) 2024-03-04T12:59:57.268 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:59:57.262+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:59:57.263532+0000) 2024-03-04T12:59:57.699 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:59:57.707 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T12:59:58.269 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:59:58.263+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:59:58.263785+0000) 2024-03-04T12:59:59.269 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T12:59:59.263+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T11:59:59.264038+0000) 2024-03-04T13:00:00.269 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:00:00.263+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:00:00.264301+0000) 2024-03-04T13:00:01.269 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:00:01.263+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:00:01.264563+0000) 2024-03-04T13:00:02.270 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:00:02.264+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:00:02.264805+0000) 2024-03-04T13:00:03.270 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:00:03.264+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:00:03.265081+0000) 2024-03-04T13:00:04.270 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:00:04.264+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:00:04.265348+0000) 2024-03-04T13:00:05.270 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:00:05.264+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:00:05.265525+0000) 2024-03-04T13:00:06.271 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:00:06.264+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:00:06.265757+0000) 2024-03-04T13:00:07.271 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:00:07.265+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:00:07.266024+0000) 2024-03-04T13:00:08.271 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:00:08.265+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:00:08.266271+0000) 2024-03-04T13:00:09.271 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:00:09.265+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:00:09.266454+0000) 2024-03-04T13:00:10.271 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:00:10.266+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:00:10.266692+0000) 2024-03-04T13:00:11.272 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:00:11.266+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:00:11.266930+0000) 2024-03-04T13:00:12.272 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:00:12.266+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:00:12.267209+0000) 2024-03-04T13:00:13.272 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:00:13.267+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:00:13.267446+0000) 2024-03-04T13:00:14.272 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:00:14.267+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:00:14.267689+0000) 2024-03-04T13:00:15.273 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:00:15.267+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:00:15.267941+0000) 2024-03-04T13:00:16.273 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:00:16.267+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:00:16.268184+0000) 2024-03-04T13:00:17.273 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:00:17.268+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:00:17.268423+0000) 2024-03-04T13:00:18.273 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:00:18.267+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:00:18.268687+0000) 2024-03-04T13:00:19.274 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:00:19.268+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:00:19.268925+0000) 2024-03-04T13:00:20.274 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:00:20.268+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:00:20.269164+0000) 2024-03-04T13:00:21.274 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:00:21.269+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:00:21.269403+0000) 2024-03-04T13:00:22.274 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:00:22.269+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:00:22.269647+0000) 2024-03-04T13:00:23.275 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:00:23.269+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:00:23.269940+0000) 2024-03-04T13:00:24.275 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:00:24.270+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:00:24.270213+0000) 2024-03-04T13:00:25.275 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:00:25.269+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:00:25.270451+0000) 2024-03-04T13:00:26.275 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:00:26.269+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:00:26.270696+0000) 2024-03-04T13:00:27.276 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:00:27.269+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:00:27.270942+0000) 2024-03-04T13:00:27.896 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:00:27.907 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:00:28.276 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:00:28.270+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:00:28.271195+0000) 2024-03-04T13:00:29.276 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:00:29.270+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:00:29.271435+0000) 2024-03-04T13:00:30.276 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:00:30.270+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:00:30.271696+0000) 2024-03-04T13:00:31.277 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:00:31.270+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:00:31.271960+0000) 2024-03-04T13:00:32.277 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:00:32.271+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:00:32.272208+0000) 2024-03-04T13:00:33.277 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:00:33.271+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:00:33.272454+0000) 2024-03-04T13:00:34.278 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:00:34.271+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:00:34.272702+0000) 2024-03-04T13:00:35.278 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:00:35.271+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:00:35.272946+0000) 2024-03-04T13:00:36.278 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:00:36.272+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:00:36.273193+0000) 2024-03-04T13:00:37.278 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:00:37.272+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:00:37.273447+0000) 2024-03-04T13:00:38.279 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:00:38.272+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:00:38.273785+0000) 2024-03-04T13:00:39.279 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:00:39.273+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:00:39.274023+0000) 2024-03-04T13:00:40.279 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:00:40.273+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:00:40.274283+0000) 2024-03-04T13:00:41.279 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:00:41.273+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:00:41.274538+0000) 2024-03-04T13:00:42.280 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:00:42.273+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:00:42.274784+0000) 2024-03-04T13:00:43.280 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:00:43.274+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:00:43.275024+0000) 2024-03-04T13:00:44.280 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:00:44.274+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:00:44.275264+0000) 2024-03-04T13:00:45.280 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:00:45.274+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:00:45.275515+0000) 2024-03-04T13:00:46.281 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:00:46.274+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:00:46.275760+0000) 2024-03-04T13:00:47.281 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:00:47.275+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:00:47.275989+0000) 2024-03-04T13:00:48.281 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:00:48.275+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:00:48.276239+0000) 2024-03-04T13:00:49.281 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:00:49.275+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:00:49.276475+0000) 2024-03-04T13:00:50.282 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:00:50.276+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:00:50.276749+0000) 2024-03-04T13:00:51.282 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:00:51.276+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:00:51.276991+0000) 2024-03-04T13:00:52.282 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:00:52.276+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:00:52.277239+0000) 2024-03-04T13:00:53.282 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:00:53.276+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:00:53.277480+0000) 2024-03-04T13:00:54.282 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:00:54.277+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:00:54.277719+0000) 2024-03-04T13:00:55.283 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:00:55.277+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:00:55.277984+0000) 2024-03-04T13:00:56.283 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:00:56.277+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:00:56.278233+0000) 2024-03-04T13:00:57.283 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:00:57.277+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:00:57.278474+0000) 2024-03-04T13:00:58.118 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:00:58.126 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:00:58.283 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:00:58.278+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:00:58.278733+0000) 2024-03-04T13:00:59.284 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:00:59.278+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:00:59.278998+0000) 2024-03-04T13:01:00.284 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:01:00.278+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:01:00.279257+0000) 2024-03-04T13:01:01.284 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:01:01.278+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:01:01.279518+0000) 2024-03-04T13:01:02.285 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:01:02.278+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:01:02.279768+0000) 2024-03-04T13:01:03.285 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:01:03.279+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:01:03.279956+0000) 2024-03-04T13:01:04.285 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:01:04.279+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:01:04.280182+0000) 2024-03-04T13:01:05.285 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:01:05.279+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:01:05.280424+0000) 2024-03-04T13:01:06.286 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:01:06.279+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:01:06.280689+0000) 2024-03-04T13:01:07.286 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:01:07.280+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:01:07.280950+0000) 2024-03-04T13:01:08.287 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:01:08.280+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:01:08.281225+0000) 2024-03-04T13:01:09.286 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:01:09.281+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:01:09.281468+0000) 2024-03-04T13:01:10.287 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:01:10.280+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:01:10.281726+0000) 2024-03-04T13:01:11.287 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:01:11.281+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:01:11.281965+0000) 2024-03-04T13:01:12.287 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:01:12.281+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:01:12.282176+0000) 2024-03-04T13:01:13.287 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:01:13.281+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:01:13.282381+0000) 2024-03-04T13:01:14.287 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:01:14.281+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:01:14.282620+0000) 2024-03-04T13:01:15.288 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:01:15.282+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:01:15.282854+0000) 2024-03-04T13:01:16.288 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:01:16.282+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:01:16.283122+0000) 2024-03-04T13:01:17.288 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:01:17.282+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:01:17.283369+0000) 2024-03-04T13:01:18.289 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:01:18.283+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:01:18.283632+0000) 2024-03-04T13:01:19.289 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:01:19.283+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:01:19.283903+0000) 2024-03-04T13:01:20.289 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:01:20.283+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:01:20.284140+0000) 2024-03-04T13:01:21.289 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:01:21.284+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:01:21.284383+0000) 2024-03-04T13:01:22.290 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:01:22.283+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:01:22.284662+0000) 2024-03-04T13:01:23.290 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:01:23.284+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:01:23.284948+0000) 2024-03-04T13:01:24.290 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:01:24.284+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:01:24.285141+0000) 2024-03-04T13:01:25.290 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:01:25.285+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:01:25.285394+0000) 2024-03-04T13:01:26.291 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:01:26.283+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:01:26.285600+0000) 2024-03-04T13:01:27.291 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:01:27.284+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:01:27.285877+0000) 2024-03-04T13:01:28.268 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:01:28.276 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:01:28.291 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:01:28.284+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:01:28.286112+0000) 2024-03-04T13:01:29.291 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:01:29.284+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:01:29.286364+0000) 2024-03-04T13:01:30.292 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:01:30.285+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:01:30.286679+0000) 2024-03-04T13:01:31.292 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:01:31.285+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:01:31.287056+0000) 2024-03-04T13:01:32.292 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:01:32.286+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:01:32.287343+0000) 2024-03-04T13:01:33.293 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:01:33.285+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:01:33.287648+0000) 2024-03-04T13:01:34.293 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:01:34.286+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:01:34.287983+0000) 2024-03-04T13:01:35.293 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:01:35.287+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:01:35.288274+0000) 2024-03-04T13:01:36.293 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:01:36.287+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:01:36.288526+0000) 2024-03-04T13:01:37.294 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:01:37.287+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:01:37.288762+0000) 2024-03-04T13:01:38.294 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:01:38.288+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:01:38.289004+0000) 2024-03-04T13:01:39.294 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:01:39.288+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:01:39.289241+0000) 2024-03-04T13:01:40.294 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:01:40.288+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:01:40.289481+0000) 2024-03-04T13:01:41.295 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:01:41.288+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:01:41.289713+0000) 2024-03-04T13:01:42.295 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:01:42.289+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:01:42.289954+0000) 2024-03-04T13:01:43.295 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:01:43.289+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:01:43.290211+0000) 2024-03-04T13:01:44.295 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:01:44.289+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:01:44.290476+0000) 2024-03-04T13:01:45.296 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:01:45.289+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:01:45.290713+0000) 2024-03-04T13:01:46.296 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:01:46.290+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:01:46.290962+0000) 2024-03-04T13:01:47.296 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:01:47.290+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:01:47.291201+0000) 2024-03-04T13:01:48.296 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:01:48.290+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:01:48.291453+0000) 2024-03-04T13:01:49.297 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:01:49.290+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:01:49.291723+0000) 2024-03-04T13:01:50.297 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:01:50.291+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:01:50.291994+0000) 2024-03-04T13:01:51.297 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:01:51.291+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:01:51.292205+0000) 2024-03-04T13:01:52.297 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:01:52.291+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:01:52.292470+0000) 2024-03-04T13:01:53.298 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:01:53.292+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:01:53.292703+0000) 2024-03-04T13:01:54.298 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:01:54.292+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:01:54.292940+0000) 2024-03-04T13:01:55.298 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:01:55.292+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:01:55.293173+0000) 2024-03-04T13:01:56.298 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:01:56.292+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:01:56.293433+0000) 2024-03-04T13:01:57.299 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:01:57.293+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:01:57.293685+0000) 2024-03-04T13:01:58.299 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:01:58.293+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:01:58.293951+0000) 2024-03-04T13:01:58.420 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:01:58.428 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:01:59.299 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:01:59.293+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:01:59.294217+0000) 2024-03-04T13:02:00.299 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:02:00.293+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:02:00.294482+0000) 2024-03-04T13:02:01.300 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:02:01.293+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:02:01.294774+0000) 2024-03-04T13:02:02.300 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:02:02.294+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:02:02.295068+0000) 2024-03-04T13:02:03.300 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:02:03.294+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:02:03.295314+0000) 2024-03-04T13:02:04.301 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:02:04.295+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:02:04.295555+0000) 2024-03-04T13:02:05.301 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:02:05.294+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:02:05.295789+0000) 2024-03-04T13:02:06.301 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:02:06.295+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:02:06.296035+0000) 2024-03-04T13:02:07.301 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:02:07.295+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:02:07.296277+0000) 2024-03-04T13:02:08.302 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:02:08.296+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:02:08.296543+0000) 2024-03-04T13:02:09.302 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:02:09.296+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:02:09.296781+0000) 2024-03-04T13:02:10.302 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:02:10.296+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:02:10.297019+0000) 2024-03-04T13:02:11.302 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:02:11.296+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:02:11.297254+0000) 2024-03-04T13:02:12.302 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:02:12.297+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:02:12.297492+0000) 2024-03-04T13:02:13.303 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:02:13.297+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:02:13.297726+0000) 2024-03-04T13:02:14.303 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:02:14.297+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:02:14.298008+0000) 2024-03-04T13:02:15.303 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:02:15.297+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:02:15.298247+0000) 2024-03-04T13:02:16.303 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:02:16.298+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:02:16.298494+0000) 2024-03-04T13:02:17.304 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:02:17.298+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:02:17.298731+0000) 2024-03-04T13:02:18.304 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:02:18.298+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:02:18.298974+0000) 2024-03-04T13:02:19.304 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:02:19.298+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:02:19.299261+0000) 2024-03-04T13:02:20.304 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:02:20.299+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:02:20.299498+0000) 2024-03-04T13:02:21.305 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:02:21.299+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:02:21.299768+0000) 2024-03-04T13:02:22.305 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:02:22.299+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:02:22.300005+0000) 2024-03-04T13:02:23.305 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:02:23.300+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:02:23.300239+0000) 2024-03-04T13:02:24.305 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:02:24.300+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:02:24.300474+0000) 2024-03-04T13:02:25.306 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:02:25.300+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:02:25.300722+0000) 2024-03-04T13:02:26.306 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:02:26.299+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:02:26.300965+0000) 2024-03-04T13:02:27.306 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:02:27.300+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:02:27.301198+0000) 2024-03-04T13:02:28.306 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:02:28.300+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:02:28.301443+0000) 2024-03-04T13:02:28.625 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:02:28.633 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:02:29.307 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:02:29.300+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:02:29.301697+0000) 2024-03-04T13:02:30.307 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:02:30.300+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:02:30.301956+0000) 2024-03-04T13:02:31.307 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:02:31.301+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:02:31.302198+0000) 2024-03-04T13:02:32.307 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:02:32.300+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:02:32.302429+0000) 2024-03-04T13:02:33.308 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:02:33.301+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:02:33.302663+0000) 2024-03-04T13:02:34.308 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:02:34.301+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:02:34.302935+0000) 2024-03-04T13:02:35.308 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:02:35.302+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:02:35.303170+0000) 2024-03-04T13:02:36.308 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:02:36.302+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:02:36.303410+0000) 2024-03-04T13:02:37.309 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:02:37.302+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:02:37.303647+0000) 2024-03-04T13:02:38.309 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:02:38.303+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:02:38.303946+0000) 2024-03-04T13:02:39.310 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:02:39.303+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:02:39.304177+0000) 2024-03-04T13:02:40.310 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:02:40.303+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:02:40.304411+0000) 2024-03-04T13:02:41.310 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:02:41.303+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:02:41.304653+0000) 2024-03-04T13:02:42.310 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:02:42.304+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:02:42.304879+0000) 2024-03-04T13:02:43.310 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:02:43.304+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:02:43.305113+0000) 2024-03-04T13:02:44.310 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:02:44.304+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:02:44.305350+0000) 2024-03-04T13:02:45.311 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:02:45.304+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:02:45.305581+0000) 2024-03-04T13:02:46.311 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:02:46.305+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:02:46.305880+0000) 2024-03-04T13:02:47.311 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:02:47.305+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:02:47.306084+0000) 2024-03-04T13:02:48.311 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:02:48.305+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:02:48.306334+0000) 2024-03-04T13:02:49.312 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:02:49.305+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:02:49.306587+0000) 2024-03-04T13:02:50.312 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:02:50.306+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:02:50.306856+0000) 2024-03-04T13:02:51.312 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:02:51.306+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:02:51.307096+0000) 2024-03-04T13:02:52.312 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:02:52.306+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:02:52.307293+0000) 2024-03-04T13:02:53.313 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:02:53.306+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:02:53.307558+0000) 2024-03-04T13:02:54.313 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:02:54.307+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:02:54.307795+0000) 2024-03-04T13:02:55.313 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:02:55.307+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:02:55.308064+0000) 2024-03-04T13:02:56.313 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:02:56.307+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:02:56.308302+0000) 2024-03-04T13:02:57.314 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:02:57.307+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:02:57.308541+0000) 2024-03-04T13:02:58.314 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:02:58.308+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:02:58.308785+0000) 2024-03-04T13:02:58.846 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:02:58.854 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:02:59.314 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:02:59.308+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:02:59.309020+0000) 2024-03-04T13:03:00.314 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:03:00.308+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:03:00.309276+0000) 2024-03-04T13:03:01.315 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:03:01.308+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:03:01.309512+0000) 2024-03-04T13:03:02.315 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:03:02.309+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:03:02.309756+0000) 2024-03-04T13:03:03.315 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:03:03.309+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:03:03.309977+0000) 2024-03-04T13:03:04.315 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:03:04.309+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:03:04.310229+0000) 2024-03-04T13:03:05.316 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:03:05.309+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:03:05.310480+0000) 2024-03-04T13:03:06.316 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:03:06.310+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:03:06.310719+0000) 2024-03-04T13:03:07.316 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:03:07.310+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:03:07.310964+0000) 2024-03-04T13:03:08.316 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:03:08.310+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:03:08.311206+0000) 2024-03-04T13:03:09.316 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:03:09.311+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:03:09.311442+0000) 2024-03-04T13:03:10.317 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:03:10.311+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:03:10.311630+0000) 2024-03-04T13:03:11.317 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:03:11.311+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:03:11.311895+0000) 2024-03-04T13:03:12.317 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:03:12.311+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:03:12.312132+0000) 2024-03-04T13:03:13.317 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:03:13.312+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:03:13.312432+0000) 2024-03-04T13:03:14.318 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:03:14.312+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:03:14.312667+0000) 2024-03-04T13:03:15.318 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:03:15.312+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:03:15.312944+0000) 2024-03-04T13:03:16.318 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:03:16.312+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:03:16.313143+0000) 2024-03-04T13:03:17.318 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:03:17.313+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:03:17.313382+0000) 2024-03-04T13:03:18.319 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:03:18.313+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:03:18.313578+0000) 2024-03-04T13:03:19.319 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:03:19.313+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:03:19.313854+0000) 2024-03-04T13:03:20.319 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:03:20.313+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:03:20.314094+0000) 2024-03-04T13:03:21.319 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:03:21.314+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:03:21.314352+0000) 2024-03-04T13:03:22.320 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:03:22.314+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:03:22.314589+0000) 2024-03-04T13:03:23.320 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:03:23.314+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:03:23.314861+0000) 2024-03-04T13:03:24.320 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:03:24.314+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:03:24.315097+0000) 2024-03-04T13:03:25.321 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:03:25.315+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:03:25.315331+0000) 2024-03-04T13:03:26.321 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:03:26.313+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:03:26.315596+0000) 2024-03-04T13:03:27.321 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:03:27.313+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:03:27.315787+0000) 2024-03-04T13:03:28.321 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:03:28.314+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:03:28.316071+0000) 2024-03-04T13:03:29.076 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:03:29.084 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:03:29.321 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:03:29.314+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:03:29.316265+0000) 2024-03-04T13:03:30.322 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:03:30.315+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:03:30.316456+0000) 2024-03-04T13:03:31.322 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:03:31.315+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:03:31.316690+0000) 2024-03-04T13:03:32.322 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:03:32.315+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:03:32.316967+0000) 2024-03-04T13:03:33.322 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:03:33.316+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:03:33.317204+0000) 2024-03-04T13:03:34.323 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:03:34.316+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:03:34.317462+0000) 2024-03-04T13:03:35.323 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:03:35.316+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:03:35.317688+0000) 2024-03-04T13:03:36.323 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:03:36.317+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:03:36.317985+0000) 2024-03-04T13:03:37.323 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:03:37.317+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:03:37.318279+0000) 2024-03-04T13:03:38.324 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:03:38.317+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:03:38.318585+0000) 2024-03-04T13:03:39.324 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:03:39.318+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:03:39.318932+0000) 2024-03-04T13:03:40.324 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:03:40.318+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:03:40.319220+0000) 2024-03-04T13:03:41.325 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:03:41.318+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:03:41.319507+0000) 2024-03-04T13:03:42.325 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:03:42.318+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:03:42.319802+0000) 2024-03-04T13:03:43.325 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:03:43.319+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:03:43.320154+0000) 2024-03-04T13:03:44.326 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:03:44.319+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:03:44.320401+0000) 2024-03-04T13:03:45.326 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:03:45.319+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:03:45.320693+0000) 2024-03-04T13:03:46.326 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:03:46.320+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:03:46.320991+0000) 2024-03-04T13:03:47.326 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:03:47.320+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:03:47.321283+0000) 2024-03-04T13:03:48.327 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:03:48.320+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:03:48.321578+0000) 2024-03-04T13:03:49.327 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:03:49.321+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:03:49.321917+0000) 2024-03-04T13:03:50.327 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:03:50.321+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:03:50.322201+0000) 2024-03-04T13:03:51.328 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:03:51.321+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:03:51.322502+0000) 2024-03-04T13:03:52.328 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:03:52.322+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:03:52.322814+0000) 2024-03-04T13:03:53.328 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:03:53.322+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:03:53.323171+0000) 2024-03-04T13:03:54.328 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:03:54.322+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:03:54.323453+0000) 2024-03-04T13:03:55.329 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:03:55.323+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:03:55.323739+0000) 2024-03-04T13:03:56.329 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:03:56.323+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:03:56.324028+0000) 2024-03-04T13:03:57.330 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:03:57.323+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:03:57.324301+0000) 2024-03-04T13:03:58.330 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:03:58.324+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:03:58.324590+0000) 2024-03-04T13:03:59.223 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:03:59.231 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:03:59.330 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:03:59.324+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:03:59.324895+0000) 2024-03-04T13:04:00.330 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:04:00.324+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:04:00.325162+0000) 2024-03-04T13:04:01.331 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:04:01.324+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:04:01.325418+0000) 2024-03-04T13:04:02.331 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:04:02.325+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:04:02.325686+0000) 2024-03-04T13:04:03.331 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:04:03.325+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:04:03.325940+0000) 2024-03-04T13:04:04.331 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:04:04.325+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:04:04.326141+0000) 2024-03-04T13:04:05.332 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:04:05.325+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:04:05.326398+0000) 2024-03-04T13:04:06.332 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:04:06.326+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:04:06.326661+0000) 2024-03-04T13:04:07.332 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:04:07.326+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:04:07.326946+0000) 2024-03-04T13:04:08.333 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:04:08.326+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:04:08.327214+0000) 2024-03-04T13:04:09.333 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:04:09.327+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:04:09.327457+0000) 2024-03-04T13:04:10.333 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:04:10.327+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:04:10.327705+0000) 2024-03-04T13:04:11.333 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:04:11.327+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:04:11.327941+0000) 2024-03-04T13:04:12.333 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:04:12.327+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:04:12.328191+0000) 2024-03-04T13:04:13.334 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:04:13.328+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:04:13.328435+0000) 2024-03-04T13:04:14.334 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:04:14.328+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:04:14.328674+0000) 2024-03-04T13:04:15.334 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:04:15.328+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:04:15.328937+0000) 2024-03-04T13:04:16.334 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:04:16.328+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:04:16.329222+0000) 2024-03-04T13:04:17.335 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:04:17.328+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:04:17.329486+0000) 2024-03-04T13:04:18.335 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:04:18.329+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:04:18.329784+0000) 2024-03-04T13:04:19.335 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:04:19.329+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:04:19.330050+0000) 2024-03-04T13:04:20.335 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:04:20.330+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:04:20.330292+0000) 2024-03-04T13:04:21.336 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:04:21.330+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:04:21.330531+0000) 2024-03-04T13:04:22.336 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:04:22.330+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:04:22.330773+0000) 2024-03-04T13:04:23.336 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:04:23.330+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:04:23.330991+0000) 2024-03-04T13:04:24.336 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:04:24.331+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:04:24.331256+0000) 2024-03-04T13:04:25.337 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:04:25.330+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:04:25.331494+0000) 2024-03-04T13:04:26.337 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:04:26.330+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:04:26.331757+0000) 2024-03-04T13:04:27.337 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:04:27.330+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:04:27.332005+0000) 2024-03-04T13:04:28.337 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:04:28.331+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:04:28.332263+0000) 2024-03-04T13:04:29.338 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:04:29.330+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:04:29.332503+0000) 2024-03-04T13:04:29.361 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:04:29.368 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:04:30.338 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:04:30.331+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:04:30.332751+0000) 2024-03-04T13:04:31.338 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:04:31.331+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:04:31.332986+0000) 2024-03-04T13:04:32.338 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:04:32.332+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:04:32.333227+0000) 2024-03-04T13:04:33.339 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:04:33.332+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:04:33.333468+0000) 2024-03-04T13:04:34.339 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:04:34.332+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:04:34.333710+0000) 2024-03-04T13:04:35.339 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:04:35.332+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:04:35.333955+0000) 2024-03-04T13:04:36.339 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:04:36.333+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:04:36.334234+0000) 2024-03-04T13:04:37.340 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:04:37.333+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:04:37.334489+0000) 2024-03-04T13:04:38.340 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:04:38.333+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:04:38.334766+0000) 2024-03-04T13:04:39.340 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:04:39.334+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:04:39.335005+0000) 2024-03-04T13:04:40.340 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:04:40.334+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:04:40.335238+0000) 2024-03-04T13:04:41.340 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:04:41.334+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:04:41.335436+0000) 2024-03-04T13:04:42.341 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:04:42.334+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:04:42.335621+0000) 2024-03-04T13:04:43.341 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:04:43.335+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:04:43.335895+0000) 2024-03-04T13:04:44.341 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:04:44.335+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:04:44.336131+0000) 2024-03-04T13:04:45.341 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:04:45.335+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:04:45.336364+0000) 2024-03-04T13:04:46.343 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:04:46.335+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:04:46.336604+0000) 2024-03-04T13:04:47.342 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:04:47.336+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:04:47.336866+0000) 2024-03-04T13:04:48.342 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:04:48.336+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:04:48.337108+0000) 2024-03-04T13:04:49.342 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:04:49.336+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:04:49.337307+0000) 2024-03-04T13:04:50.342 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:04:50.336+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:04:50.337568+0000) 2024-03-04T13:04:51.342 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:04:51.337+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:04:51.337800+0000) 2024-03-04T13:04:52.343 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:04:52.337+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:04:52.338078+0000) 2024-03-04T13:04:53.343 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:04:53.337+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:04:53.338266+0000) 2024-03-04T13:04:54.343 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:04:54.337+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:04:54.338502+0000) 2024-03-04T13:04:55.343 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:04:55.338+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:04:55.338734+0000) 2024-03-04T13:04:56.344 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:04:56.338+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:04:56.338972+0000) 2024-03-04T13:04:57.344 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:04:57.338+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:04:57.339210+0000) 2024-03-04T13:04:58.344 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:04:58.338+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:04:58.339483+0000) 2024-03-04T13:04:59.344 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:04:59.339+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:04:59.339746+0000) 2024-03-04T13:04:59.499 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:04:59.507 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:05:00.344 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:05:00.339+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:05:00.339982+0000) 2024-03-04T13:05:01.345 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:05:01.339+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:05:01.340215+0000) 2024-03-04T13:05:02.345 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:05:02.339+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:05:02.340454+0000) 2024-03-04T13:05:03.345 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:05:03.340+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:05:03.340685+0000) 2024-03-04T13:05:04.345 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:05:04.340+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:05:04.340924+0000) 2024-03-04T13:05:05.346 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:05:05.340+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:05:05.341161+0000) 2024-03-04T13:05:06.346 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:05:06.340+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:05:06.341398+0000) 2024-03-04T13:05:07.346 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:05:07.341+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:05:07.341636+0000) 2024-03-04T13:05:08.347 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:05:08.341+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:05:08.341921+0000) 2024-03-04T13:05:09.347 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:05:09.341+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:05:09.342160+0000) 2024-03-04T13:05:10.347 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:05:10.342+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:05:10.342397+0000) 2024-03-04T13:05:11.347 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:05:11.341+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:05:11.342585+0000) 2024-03-04T13:05:12.347 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:05:12.342+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:05:12.342906+0000) 2024-03-04T13:05:13.347 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:05:13.342+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:05:13.343097+0000) 2024-03-04T13:05:14.348 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:05:14.342+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:05:14.343284+0000) 2024-03-04T13:05:15.348 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:05:15.342+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:05:15.343464+0000) 2024-03-04T13:05:16.348 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:05:16.343+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:05:16.343642+0000) 2024-03-04T13:05:17.348 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:05:17.343+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:05:17.343910+0000) 2024-03-04T13:05:18.348 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:05:18.343+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:05:18.344152+0000) 2024-03-04T13:05:19.349 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:05:19.343+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:05:19.344417+0000) 2024-03-04T13:05:20.349 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:05:20.344+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:05:20.344690+0000) 2024-03-04T13:05:21.349 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:05:21.344+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:05:21.344926+0000) 2024-03-04T13:05:22.349 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:05:22.344+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:05:22.345168+0000) 2024-03-04T13:05:23.350 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:05:23.345+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:05:23.345400+0000) 2024-03-04T13:05:24.350 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:05:24.344+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:05:24.345632+0000) 2024-03-04T13:05:25.350 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:05:25.345+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:05:25.345941+0000) 2024-03-04T13:05:26.350 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:05:26.345+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:05:26.346202+0000) 2024-03-04T13:05:27.351 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:05:27.345+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:05:27.346442+0000) 2024-03-04T13:05:28.351 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:05:28.345+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:05:28.346691+0000) 2024-03-04T13:05:29.351 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:05:29.345+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:05:29.346944+0000) 2024-03-04T13:05:29.712 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:05:29.720 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:05:30.352 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:05:30.346+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:05:30.347202+0000) 2024-03-04T13:05:31.352 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:05:31.346+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:05:31.347434+0000) 2024-03-04T13:05:32.352 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:05:32.345+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:05:32.347615+0000) 2024-03-04T13:05:33.352 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:05:33.346+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:05:33.347900+0000) 2024-03-04T13:05:34.352 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:05:34.347+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:05:34.348133+0000) 2024-03-04T13:05:35.352 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:05:35.347+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:05:35.348385+0000) 2024-03-04T13:05:36.353 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:05:36.347+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:05:36.348634+0000) 2024-03-04T13:05:37.353 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:05:37.348+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:05:37.348912+0000) 2024-03-04T13:05:38.353 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:05:38.348+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:05:38.349156+0000) 2024-03-04T13:05:39.353 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:05:39.348+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:05:39.349389+0000) 2024-03-04T13:05:40.354 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:05:40.348+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:05:40.349626+0000) 2024-03-04T13:05:41.354 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:05:41.349+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:05:41.349907+0000) 2024-03-04T13:05:42.354 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:05:42.349+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:05:42.350143+0000) 2024-03-04T13:05:43.354 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:05:43.349+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:05:43.350375+0000) 2024-03-04T13:05:44.355 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:05:44.349+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:05:44.350612+0000) 2024-03-04T13:05:45.355 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:05:45.350+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:05:45.350883+0000) 2024-03-04T13:05:46.355 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:05:46.350+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:05:46.351138+0000) 2024-03-04T13:05:47.355 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:05:47.350+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:05:47.351375+0000) 2024-03-04T13:05:48.356 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:05:48.350+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:05:48.351620+0000) 2024-03-04T13:05:49.356 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:05:49.351+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:05:49.351909+0000) 2024-03-04T13:05:50.356 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:05:50.351+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:05:50.352141+0000) 2024-03-04T13:05:51.356 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:05:51.351+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:05:51.352374+0000) 2024-03-04T13:05:52.356 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:05:52.351+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:05:52.352613+0000) 2024-03-04T13:05:53.357 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:05:53.352+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:05:53.352897+0000) 2024-03-04T13:05:54.357 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:05:54.352+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:05:54.353133+0000) 2024-03-04T13:05:55.357 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:05:55.352+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:05:55.353420+0000) 2024-03-04T13:05:56.357 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:05:56.353+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:05:56.353714+0000) 2024-03-04T13:05:57.357 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:05:57.353+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:05:57.353970+0000) 2024-03-04T13:05:58.358 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:05:58.353+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:05:58.354265+0000) 2024-03-04T13:05:59.358 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:05:59.353+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:05:59.354517+0000) 2024-03-04T13:05:59.932 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:05:59.940 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:06:00.359 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:06:00.354+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:06:00.354779+0000) 2024-03-04T13:06:01.358 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:06:01.354+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:06:01.354981+0000) 2024-03-04T13:06:02.359 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:06:02.354+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:06:02.355211+0000) 2024-03-04T13:06:03.359 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:06:03.354+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:06:03.355463+0000) 2024-03-04T13:06:04.359 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:06:04.355+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:06:04.355735+0000) 2024-03-04T13:06:05.359 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:06:05.355+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:06:05.356001+0000) 2024-03-04T13:06:06.360 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:06:06.355+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:06:06.356261+0000) 2024-03-04T13:06:07.360 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:06:07.356+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:06:07.356512+0000) 2024-03-04T13:06:08.360 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:06:08.356+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:06:08.356774+0000) 2024-03-04T13:06:09.360 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:06:09.356+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:06:09.357023+0000) 2024-03-04T13:06:10.361 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:06:10.356+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:06:10.357296+0000) 2024-03-04T13:06:11.361 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:06:11.356+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:06:11.357608+0000) 2024-03-04T13:06:12.361 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:06:12.357+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:06:12.357942+0000) 2024-03-04T13:06:13.362 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:06:13.357+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:06:13.358185+0000) 2024-03-04T13:06:14.362 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:06:14.357+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:06:14.358414+0000) 2024-03-04T13:06:15.362 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:06:15.357+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:06:15.358644+0000) 2024-03-04T13:06:16.362 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:06:16.358+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:06:16.358918+0000) 2024-03-04T13:06:17.363 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:06:17.358+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:06:17.359207+0000) 2024-03-04T13:06:18.363 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:06:18.358+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:06:18.359479+0000) 2024-03-04T13:06:19.363 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:06:19.359+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:06:19.359878+0000) 2024-03-04T13:06:20.363 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:06:20.359+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:06:20.360168+0000) 2024-03-04T13:06:21.364 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:06:21.360+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:06:21.360464+0000) 2024-03-04T13:06:22.364 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:06:22.360+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:06:22.360755+0000) 2024-03-04T13:06:23.364 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:06:23.360+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:06:23.361042+0000) 2024-03-04T13:06:24.365 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:06:24.361+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:06:24.361329+0000) 2024-03-04T13:06:25.365 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:06:25.361+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:06:25.361619+0000) 2024-03-04T13:06:26.365 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:06:26.361+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:06:26.361975+0000) 2024-03-04T13:06:27.365 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:06:27.362+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:06:27.362299+0000) 2024-03-04T13:06:28.366 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:06:28.360+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:06:28.362569+0000) 2024-03-04T13:06:29.366 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:06:29.361+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:06:29.362890+0000) 2024-03-04T13:06:30.146 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:06:30.154 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:06:30.366 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:06:30.362+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:06:30.363187+0000) 2024-03-04T13:06:31.367 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:06:31.362+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:06:31.363476+0000) 2024-03-04T13:06:32.367 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:06:32.362+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:06:32.363761+0000) 2024-03-04T13:06:33.367 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:06:33.362+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:06:33.364060+0000) 2024-03-04T13:06:34.367 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:06:34.362+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:06:34.364295+0000) 2024-03-04T13:06:35.368 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:06:35.362+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:06:35.364605+0000) 2024-03-04T13:06:36.368 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:06:36.363+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:06:36.364937+0000) 2024-03-04T13:06:37.368 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:06:37.364+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:06:37.365240+0000) 2024-03-04T13:06:38.369 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:06:38.364+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:06:38.365556+0000) 2024-03-04T13:06:39.369 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:06:39.365+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:06:39.365886+0000) 2024-03-04T13:06:40.369 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:06:40.365+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:06:40.366179+0000) 2024-03-04T13:06:41.370 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:06:41.365+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:06:41.366537+0000) 2024-03-04T13:06:42.370 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:06:42.365+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:06:42.366771+0000) 2024-03-04T13:06:43.370 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:06:43.366+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:06:43.367064+0000) 2024-03-04T13:06:44.370 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:06:44.366+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:06:44.367360+0000) 2024-03-04T13:06:45.371 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:06:45.366+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:06:45.367647+0000) 2024-03-04T13:06:46.371 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:06:46.367+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:06:46.367966+0000) 2024-03-04T13:06:47.371 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:06:47.367+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:06:47.368263+0000) 2024-03-04T13:06:48.371 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:06:48.367+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:06:48.368560+0000) 2024-03-04T13:06:49.372 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:06:49.368+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:06:49.368908+0000) 2024-03-04T13:06:50.372 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:06:50.368+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:06:50.369146+0000) 2024-03-04T13:06:51.372 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:06:51.368+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:06:51.369435+0000) 2024-03-04T13:06:52.373 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:06:52.368+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:06:52.369737+0000) 2024-03-04T13:06:53.373 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:06:53.369+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:06:53.370037+0000) 2024-03-04T13:06:54.373 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:06:54.369+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:06:54.370278+0000) 2024-03-04T13:06:55.373 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:06:55.369+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:06:55.370572+0000) 2024-03-04T13:06:56.374 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:06:56.370+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:06:56.370931+0000) 2024-03-04T13:06:57.374 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:06:57.370+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:06:57.371231+0000) 2024-03-04T13:06:58.374 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:06:58.370+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:06:58.371554+0000) 2024-03-04T13:06:59.374 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:06:59.371+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:06:59.371791+0000) 2024-03-04T13:07:00.375 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:07:00.371+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:07:00.372032+0000) 2024-03-04T13:07:00.375 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:07:00.383 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:07:01.375 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:07:01.371+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:07:01.372286+0000) 2024-03-04T13:07:02.375 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:07:02.371+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:07:02.372554+0000) 2024-03-04T13:07:03.375 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:07:03.372+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:07:03.372888+0000) 2024-03-04T13:07:04.376 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:07:04.372+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:07:04.373191+0000) 2024-03-04T13:07:05.376 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:07:05.372+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:07:05.373468+0000) 2024-03-04T13:07:06.376 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:07:06.373+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:07:06.373761+0000) 2024-03-04T13:07:07.376 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:07:07.373+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:07:07.374040+0000) 2024-03-04T13:07:08.377 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:07:08.373+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:07:08.374337+0000) 2024-03-04T13:07:09.377 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:07:09.374+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:07:09.374650+0000) 2024-03-04T13:07:10.377 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:07:10.374+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:07:10.374975+0000) 2024-03-04T13:07:11.378 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:07:11.374+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:07:11.375268+0000) 2024-03-04T13:07:12.379 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:07:12.375+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:07:12.375558+0000) 2024-03-04T13:07:13.378 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:07:13.375+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:07:13.375861+0000) 2024-03-04T13:07:14.379 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:07:14.375+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:07:14.376154+0000) 2024-03-04T13:07:15.379 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:07:15.376+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:07:15.376449+0000) 2024-03-04T13:07:16.379 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:07:16.376+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:07:16.376746+0000) 2024-03-04T13:07:17.379 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:07:17.376+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:07:17.377040+0000) 2024-03-04T13:07:18.380 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:07:18.376+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:07:18.377339+0000) 2024-03-04T13:07:19.380 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:07:19.377+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:07:19.377653+0000) 2024-03-04T13:07:20.380 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:07:20.377+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:07:20.377950+0000) 2024-03-04T13:07:21.380 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:07:21.377+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:07:21.378146+0000) 2024-03-04T13:07:22.381 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:07:22.377+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:07:22.378342+0000) 2024-03-04T13:07:23.381 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:07:23.378+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:07:23.378629+0000) 2024-03-04T13:07:24.381 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:07:24.378+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:07:24.378962+0000) 2024-03-04T13:07:25.381 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:07:25.379+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:07:25.379277+0000) 2024-03-04T13:07:26.382 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:07:26.379+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:07:26.379571+0000) 2024-03-04T13:07:27.382 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:07:27.379+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:07:27.379914+0000) 2024-03-04T13:07:28.383 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:07:28.379+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:07:28.380209+0000) 2024-03-04T13:07:29.383 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:07:29.380+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:07:29.380503+0000) 2024-03-04T13:07:30.383 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:07:30.379+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:07:30.380866+0000) 2024-03-04T13:07:30.516 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:07:30.523 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:07:31.383 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:07:31.380+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:07:31.381165+0000) 2024-03-04T13:07:32.384 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:07:32.380+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:07:32.381451+0000) 2024-03-04T13:07:33.384 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:07:33.380+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:07:33.381738+0000) 2024-03-04T13:07:34.384 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:07:34.380+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:07:34.382022+0000) 2024-03-04T13:07:35.384 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:07:35.381+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:07:35.382325+0000) 2024-03-04T13:07:36.385 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:07:36.381+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:07:36.382608+0000) 2024-03-04T13:07:37.385 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:07:37.381+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:07:37.382953+0000) 2024-03-04T13:07:38.385 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:07:38.382+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:07:38.383281+0000) 2024-03-04T13:07:39.386 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:07:39.382+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:07:39.383571+0000) 2024-03-04T13:07:40.386 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:07:40.382+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:07:40.383810+0000) 2024-03-04T13:07:41.386 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:07:41.383+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:07:41.384040+0000) 2024-03-04T13:07:42.386 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:07:42.383+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:07:42.384291+0000) 2024-03-04T13:07:43.387 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:07:43.383+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:07:43.384534+0000) 2024-03-04T13:07:44.387 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:07:44.383+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:07:44.384776+0000) 2024-03-04T13:07:45.387 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:07:45.384+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:07:45.385031+0000) 2024-03-04T13:07:46.387 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:07:46.384+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:07:46.385361+0000) 2024-03-04T13:07:47.388 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:07:47.384+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:07:47.385657+0000) 2024-03-04T13:07:48.388 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:07:48.385+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:07:48.385960+0000) 2024-03-04T13:07:49.388 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:07:49.385+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:07:49.386254+0000) 2024-03-04T13:07:50.388 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:07:50.385+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:07:50.386552+0000) 2024-03-04T13:07:51.389 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:07:51.386+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:07:51.386867+0000) 2024-03-04T13:07:52.389 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:07:52.386+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:07:52.387126+0000) 2024-03-04T13:07:53.389 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:07:53.386+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:07:53.387450+0000) 2024-03-04T13:07:54.390 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:07:54.386+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:07:54.387773+0000) 2024-03-04T13:07:55.390 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:07:55.387+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:07:55.388060+0000) 2024-03-04T13:07:56.390 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:07:56.387+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:07:56.388313+0000) 2024-03-04T13:07:57.390 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:07:57.387+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:07:57.388500+0000) 2024-03-04T13:07:58.391 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:07:58.387+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:07:58.388712+0000) 2024-03-04T13:07:59.391 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:07:59.388+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:07:59.388956+0000) 2024-03-04T13:08:00.391 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:08:00.388+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:08:00.389214+0000) 2024-03-04T13:08:00.718 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:08:00.726 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:08:01.391 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:08:01.388+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:08:01.389492+0000) 2024-03-04T13:08:02.391 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:08:02.388+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:08:02.389776+0000) 2024-03-04T13:08:03.392 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:08:03.389+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:08:03.390025+0000) 2024-03-04T13:08:04.392 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:08:04.389+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:08:04.390261+0000) 2024-03-04T13:08:05.392 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:08:05.389+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:08:05.390495+0000) 2024-03-04T13:08:06.392 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:08:06.390+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:08:06.390751+0000) 2024-03-04T13:08:07.393 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:08:07.390+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:08:07.391009+0000) 2024-03-04T13:08:08.393 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:08:08.390+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:08:08.391288+0000) 2024-03-04T13:08:09.393 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:08:09.390+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:08:09.391468+0000) 2024-03-04T13:08:10.393 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:08:10.390+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:08:10.391721+0000) 2024-03-04T13:08:11.394 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:08:11.391+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:08:11.391959+0000) 2024-03-04T13:08:12.394 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:08:12.391+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:08:12.392229+0000) 2024-03-04T13:08:13.394 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:08:13.392+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:08:13.392506+0000) 2024-03-04T13:08:14.394 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:08:14.392+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:08:14.392779+0000) 2024-03-04T13:08:15.395 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:08:15.392+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:08:15.393083+0000) 2024-03-04T13:08:16.395 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:08:16.392+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:08:16.393350+0000) 2024-03-04T13:08:17.395 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:08:17.392+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:08:17.393559+0000) 2024-03-04T13:08:18.395 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:08:18.393+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:08:18.393892+0000) 2024-03-04T13:08:19.396 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:08:19.393+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:08:19.394132+0000) 2024-03-04T13:08:20.396 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:08:20.394+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:08:20.394371+0000) 2024-03-04T13:08:21.396 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:08:21.394+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:08:21.394610+0000) 2024-03-04T13:08:22.396 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:08:22.394+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:08:22.394893+0000) 2024-03-04T13:08:23.397 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:08:23.394+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:08:23.395126+0000) 2024-03-04T13:08:24.397 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:08:24.395+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:08:24.395366+0000) 2024-03-04T13:08:25.397 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:08:25.394+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:08:25.395619+0000) 2024-03-04T13:08:26.397 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:08:26.395+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:08:26.395900+0000) 2024-03-04T13:08:27.398 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:08:27.395+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:08:27.396205+0000) 2024-03-04T13:08:28.398 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:08:28.395+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:08:28.396411+0000) 2024-03-04T13:08:29.398 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:08:29.396+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:08:29.396663+0000) 2024-03-04T13:08:30.398 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:08:30.396+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:08:30.396939+0000) 2024-03-04T13:08:30.889 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:08:30.901 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:08:31.398 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:08:31.397+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:08:31.397191+0000) 2024-03-04T13:08:32.399 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:08:32.397+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:08:32.397469+0000) 2024-03-04T13:08:33.399 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:08:33.395+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:08:33.397755+0000) 2024-03-04T13:08:34.399 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:08:34.396+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:08:34.398013+0000) 2024-03-04T13:08:35.400 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:08:35.397+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:08:35.398262+0000) 2024-03-04T13:08:36.400 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:08:36.397+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:08:36.398514+0000) 2024-03-04T13:08:37.400 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:08:37.397+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:08:37.398773+0000) 2024-03-04T13:08:38.400 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:08:38.398+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:08:38.399134+0000) 2024-03-04T13:08:39.403 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:08:39.397+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:08:39.399327+0000) 2024-03-04T13:08:40.401 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:08:40.398+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:08:40.399577+0000) 2024-03-04T13:08:41.401 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:08:41.397+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:08:41.399761+0000) 2024-03-04T13:08:42.401 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:08:42.399+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:08:42.399997+0000) 2024-03-04T13:08:43.402 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:08:43.399+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:08:43.400268+0000) 2024-03-04T13:08:44.402 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:08:44.399+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:08:44.400522+0000) 2024-03-04T13:08:45.402 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:08:45.399+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:08:45.400754+0000) 2024-03-04T13:08:46.402 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:08:46.400+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:08:46.400993+0000) 2024-03-04T13:08:47.402 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:08:47.400+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:08:47.401217+0000) 2024-03-04T13:08:48.403 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:08:48.400+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:08:48.401505+0000) 2024-03-04T13:08:49.403 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:08:49.400+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:08:49.401759+0000) 2024-03-04T13:08:50.403 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:08:50.401+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:08:50.401997+0000) 2024-03-04T13:08:51.403 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:08:51.401+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:08:51.402245+0000) 2024-03-04T13:08:52.404 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:08:52.401+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:08:52.402502+0000) 2024-03-04T13:08:53.404 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:08:53.401+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:08:53.402761+0000) 2024-03-04T13:08:54.404 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:08:54.402+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:08:54.403011+0000) 2024-03-04T13:08:55.404 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:08:55.402+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:08:55.403258+0000) 2024-03-04T13:08:56.405 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:08:56.402+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:08:56.403509+0000) 2024-03-04T13:08:57.405 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:08:57.403+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:08:57.403761+0000) 2024-03-04T13:08:58.405 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:08:58.403+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:08:58.404037+0000) 2024-03-04T13:08:59.405 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:08:59.403+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:08:59.404313+0000) 2024-03-04T13:09:00.406 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:09:00.403+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:09:00.404566+0000) 2024-03-04T13:09:01.085 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:09:01.093 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:09:01.406 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:09:01.403+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:09:01.404784+0000) 2024-03-04T13:09:02.406 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:09:02.404+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:09:02.405081+0000) 2024-03-04T13:09:03.406 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:09:03.404+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:09:03.405331+0000) 2024-03-04T13:09:04.407 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:09:04.404+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:09:04.405571+0000) 2024-03-04T13:09:05.407 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:09:05.404+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:09:05.405752+0000) 2024-03-04T13:09:06.407 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:09:06.405+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:09:06.406005+0000) 2024-03-04T13:09:07.407 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:09:07.405+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:09:07.406242+0000) 2024-03-04T13:09:08.407 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:09:08.405+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:09:08.406491+0000) 2024-03-04T13:09:09.408 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:09:09.406+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:09:09.406728+0000) 2024-03-04T13:09:10.408 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:09:10.406+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:09:10.406979+0000) 2024-03-04T13:09:11.408 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:09:11.406+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:09:11.407212+0000) 2024-03-04T13:09:12.409 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:09:12.406+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:09:12.407458+0000) 2024-03-04T13:09:13.409 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:09:13.406+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:09:13.407713+0000) 2024-03-04T13:09:14.409 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:09:14.407+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:09:14.407982+0000) 2024-03-04T13:09:15.409 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:09:15.407+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:09:15.408217+0000) 2024-03-04T13:09:16.409 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:09:16.408+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:09:16.408452+0000) 2024-03-04T13:09:17.410 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:09:17.408+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:09:17.408717+0000) 2024-03-04T13:09:18.410 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:09:18.408+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:09:18.408984+0000) 2024-03-04T13:09:19.410 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:09:19.408+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:09:19.409219+0000) 2024-03-04T13:09:20.410 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:09:20.409+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:09:20.409457+0000) 2024-03-04T13:09:21.411 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:09:21.408+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:09:21.409724+0000) 2024-03-04T13:09:22.411 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:09:22.409+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:09:22.409972+0000) 2024-03-04T13:09:23.411 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:09:23.409+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:09:23.410208+0000) 2024-03-04T13:09:24.411 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:09:24.409+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:09:24.410397+0000) 2024-03-04T13:09:25.411 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:09:25.410+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:09:25.410633+0000) 2024-03-04T13:09:26.412 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:09:26.410+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:09:26.410933+0000) 2024-03-04T13:09:27.412 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:09:27.410+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:09:27.411120+0000) 2024-03-04T13:09:28.412 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:09:28.411+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:09:28.411367+0000) 2024-03-04T13:09:29.412 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:09:29.410+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:09:29.411616+0000) 2024-03-04T13:09:30.412 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:09:30.411+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:09:30.411911+0000) 2024-03-04T13:09:31.212 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:09:31.221 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:09:31.413 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:09:31.411+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:09:31.412121+0000) 2024-03-04T13:09:32.413 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:09:32.412+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:09:32.412360+0000) 2024-03-04T13:09:33.413 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:09:33.412+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:09:33.412601+0000) 2024-03-04T13:09:34.413 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:09:34.412+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:09:34.412870+0000) 2024-03-04T13:09:35.414 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:09:35.412+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:09:35.413099+0000) 2024-03-04T13:09:36.414 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:09:36.413+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:09:36.413334+0000) 2024-03-04T13:09:37.414 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:09:37.412+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:09:37.413572+0000) 2024-03-04T13:09:38.414 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:09:38.412+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:09:38.413876+0000) 2024-03-04T13:09:39.415 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:09:39.413+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:09:39.414119+0000) 2024-03-04T13:09:40.415 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:09:40.413+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:09:40.414373+0000) 2024-03-04T13:09:41.415 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:09:41.412+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:09:41.414627+0000) 2024-03-04T13:09:42.415 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:09:42.413+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:09:42.414916+0000) 2024-03-04T13:09:43.416 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:09:43.414+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:09:43.415178+0000) 2024-03-04T13:09:44.416 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:09:44.414+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:09:44.415439+0000) 2024-03-04T13:09:45.416 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:09:45.414+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:09:45.415693+0000) 2024-03-04T13:09:46.417 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:09:46.414+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:09:46.415953+0000) 2024-03-04T13:09:47.417 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:09:47.415+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:09:47.416189+0000) 2024-03-04T13:09:48.417 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:09:48.415+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:09:48.416431+0000) 2024-03-04T13:09:49.417 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:09:49.415+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:09:49.416663+0000) 2024-03-04T13:09:50.417 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:09:50.416+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:09:50.416927+0000) 2024-03-04T13:09:51.418 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:09:51.416+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:09:51.417159+0000) 2024-03-04T13:09:52.418 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:09:52.416+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:09:52.417413+0000) 2024-03-04T13:09:53.418 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:09:53.416+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:09:53.417666+0000) 2024-03-04T13:09:54.418 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:09:54.417+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:09:54.417947+0000) 2024-03-04T13:09:55.419 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:09:55.417+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:09:55.418206+0000) 2024-03-04T13:09:56.419 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:09:56.417+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:09:56.418449+0000) 2024-03-04T13:09:57.419 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:09:57.417+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:09:57.418683+0000) 2024-03-04T13:09:58.419 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:09:58.418+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:09:58.418931+0000) 2024-03-04T13:09:59.419 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:09:59.418+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:09:59.419152+0000) 2024-03-04T13:10:00.420 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:10:00.418+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:10:00.419422+0000) 2024-03-04T13:10:01.420 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:10:01.418+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:10:01.419688+0000) 2024-03-04T13:10:01.436 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:10:01.444 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:10:02.420 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:10:02.419+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:10:02.419932+0000) 2024-03-04T13:10:03.420 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:10:03.419+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:10:03.420160+0000) 2024-03-04T13:10:04.421 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:10:04.419+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:10:04.420406+0000) 2024-03-04T13:10:05.421 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:10:05.419+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:10:05.420647+0000) 2024-03-04T13:10:06.421 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:10:06.420+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:10:06.420935+0000) 2024-03-04T13:10:07.421 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:10:07.420+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:10:07.421128+0000) 2024-03-04T13:10:08.422 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:10:08.420+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:10:08.421382+0000) 2024-03-04T13:10:09.422 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:10:09.421+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:10:09.421634+0000) 2024-03-04T13:10:10.422 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:10:10.421+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:10:10.421932+0000) 2024-03-04T13:10:11.422 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:10:11.421+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:10:11.422129+0000) 2024-03-04T13:10:12.423 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:10:12.421+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:10:12.422370+0000) 2024-03-04T13:10:13.423 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:10:13.422+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:10:13.422609+0000) 2024-03-04T13:10:14.423 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:10:14.422+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:10:14.422922+0000) 2024-03-04T13:10:15.423 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:10:15.422+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:10:15.423186+0000) 2024-03-04T13:10:16.424 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:10:16.422+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:10:16.423430+0000) 2024-03-04T13:10:17.424 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:10:17.423+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:10:17.423673+0000) 2024-03-04T13:10:18.424 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:10:18.423+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:10:18.423932+0000) 2024-03-04T13:10:19.424 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:10:19.423+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:10:19.424220+0000) 2024-03-04T13:10:20.424 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:10:20.424+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:10:20.424466+0000) 2024-03-04T13:10:21.425 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:10:21.423+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:10:21.424722+0000) 2024-03-04T13:10:22.425 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:10:22.424+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:10:22.424960+0000) 2024-03-04T13:10:23.425 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:10:23.424+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:10:23.425157+0000) 2024-03-04T13:10:24.425 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:10:24.424+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:10:24.425390+0000) 2024-03-04T13:10:25.426 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:10:25.425+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:10:25.425634+0000) 2024-03-04T13:10:26.426 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:10:26.425+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:10:26.425928+0000) 2024-03-04T13:10:27.426 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:10:27.425+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:10:27.426139+0000) 2024-03-04T13:10:28.426 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:10:28.425+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:10:28.426399+0000) 2024-03-04T13:10:29.427 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:10:29.425+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:10:29.426606+0000) 2024-03-04T13:10:30.427 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:10:30.426+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:10:30.426882+0000) 2024-03-04T13:10:31.427 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:10:31.426+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:10:31.427131+0000) 2024-03-04T13:10:31.573 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:10:31.582 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:10:32.428 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:10:32.426+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:10:32.427403+0000) 2024-03-04T13:10:33.428 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:10:33.427+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:10:33.427656+0000) 2024-03-04T13:10:34.428 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:10:34.427+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:10:34.427932+0000) 2024-03-04T13:10:35.428 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:10:35.427+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:10:35.428175+0000) 2024-03-04T13:10:36.428 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:10:36.428+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:10:36.428419+0000) 2024-03-04T13:10:37.429 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:10:37.428+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:10:37.428653+0000) 2024-03-04T13:10:38.429 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:10:38.428+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:10:38.428940+0000) 2024-03-04T13:10:39.429 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:10:39.429+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:10:39.429190+0000) 2024-03-04T13:10:40.429 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:10:40.428+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:10:40.429444+0000) 2024-03-04T13:10:41.430 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:10:41.427+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:10:41.429728+0000) 2024-03-04T13:10:42.430 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:10:42.428+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:10:42.429984+0000) 2024-03-04T13:10:43.430 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:10:43.429+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:10:43.430198+0000) 2024-03-04T13:10:44.430 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:10:44.429+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:10:44.430457+0000) 2024-03-04T13:10:45.431 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:10:45.429+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:10:45.430722+0000) 2024-03-04T13:10:46.431 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:10:46.429+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:10:46.430960+0000) 2024-03-04T13:10:47.431 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:10:47.430+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:10:47.431190+0000) 2024-03-04T13:10:48.431 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:10:48.430+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:10:48.431435+0000) 2024-03-04T13:10:49.432 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:10:49.430+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:10:49.431704+0000) 2024-03-04T13:10:50.432 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:10:50.430+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:10:50.431957+0000) 2024-03-04T13:10:51.432 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:10:51.431+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:10:51.432201+0000) 2024-03-04T13:10:52.432 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:10:52.431+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:10:52.432479+0000) 2024-03-04T13:10:53.432 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:10:53.431+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:10:53.432742+0000) 2024-03-04T13:10:54.433 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:10:54.432+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:10:54.432997+0000) 2024-03-04T13:10:55.433 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:10:55.432+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:10:55.433249+0000) 2024-03-04T13:10:56.433 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:10:56.432+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:10:56.433487+0000) 2024-03-04T13:10:57.433 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:10:57.432+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:10:57.433731+0000) 2024-03-04T13:10:58.434 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:10:58.433+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:10:58.433980+0000) 2024-03-04T13:10:59.434 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:10:59.433+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:10:59.434217+0000) 2024-03-04T13:11:00.434 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:11:00.433+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:11:00.434484+0000) 2024-03-04T13:11:01.434 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:11:01.433+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:11:01.434731+0000) 2024-03-04T13:11:01.711 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:11:01.719 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:11:02.434 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:11:02.434+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:11:02.434986+0000) 2024-03-04T13:11:03.435 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:11:03.434+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:11:03.435224+0000) 2024-03-04T13:11:04.435 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:11:04.434+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:11:04.435471+0000) 2024-03-04T13:11:05.435 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:11:05.434+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:11:05.435658+0000) 2024-03-04T13:11:06.435 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:11:06.435+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:11:06.435936+0000) 2024-03-04T13:11:07.436 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:11:07.435+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:11:07.436175+0000) 2024-03-04T13:11:08.436 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:11:08.435+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:11:08.436425+0000) 2024-03-04T13:11:09.436 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:11:09.436+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:11:09.436686+0000) 2024-03-04T13:11:10.436 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:11:10.436+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:11:10.436912+0000) 2024-03-04T13:11:11.437 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:11:11.436+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:11:11.437184+0000) 2024-03-04T13:11:12.437 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:11:12.436+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:11:12.437431+0000) 2024-03-04T13:11:13.437 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:11:13.436+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:11:13.437667+0000) 2024-03-04T13:11:14.438 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:11:14.437+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:11:14.437929+0000) 2024-03-04T13:11:15.438 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:11:15.437+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:11:15.438171+0000) 2024-03-04T13:11:16.438 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:11:16.437+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:11:16.438407+0000) 2024-03-04T13:11:17.438 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:11:17.438+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:11:17.438657+0000) 2024-03-04T13:11:18.438 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:11:18.438+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:11:18.439009+0000) 2024-03-04T13:11:19.439 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:11:19.438+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:11:19.439262+0000) 2024-03-04T13:11:20.439 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:11:20.439+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:11:20.439516+0000) 2024-03-04T13:11:21.439 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:11:21.438+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:11:21.439758+0000) 2024-03-04T13:11:22.439 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:11:22.439+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:11:22.440014+0000) 2024-03-04T13:11:23.440 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:11:23.439+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:11:23.440268+0000) 2024-03-04T13:11:24.440 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:11:24.440+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:11:24.440533+0000) 2024-03-04T13:11:25.440 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:11:25.440+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:11:25.440732+0000) 2024-03-04T13:11:26.440 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:11:26.440+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:11:26.440984+0000) 2024-03-04T13:11:27.441 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:11:27.440+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:11:27.441236+0000) 2024-03-04T13:11:28.441 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:11:28.441+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:11:28.441503+0000) 2024-03-04T13:11:29.441 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:11:29.440+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:11:29.441774+0000) 2024-03-04T13:11:30.441 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:11:30.441+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:11:30.442010+0000) 2024-03-04T13:11:31.442 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:11:31.441+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:11:31.442262+0000) 2024-03-04T13:11:31.924 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:11:31.932 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:11:32.442 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:11:32.442+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:11:32.442524+0000) 2024-03-04T13:11:33.442 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:11:33.442+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:11:33.442783+0000) 2024-03-04T13:11:34.442 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:11:34.442+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:11:34.443097+0000) 2024-03-04T13:11:35.443 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:11:35.443+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:11:35.443346+0000) 2024-03-04T13:11:36.443 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:11:36.443+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:11:36.443604+0000) 2024-03-04T13:11:37.443 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:11:37.443+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:11:37.443921+0000) 2024-03-04T13:11:38.443 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:11:38.443+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:11:38.444197+0000) 2024-03-04T13:11:39.444 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:11:39.444+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:11:39.444452+0000) 2024-03-04T13:11:40.444 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:11:40.444+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:11:40.444707+0000) 2024-03-04T13:11:41.444 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:11:41.444+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:11:41.444962+0000) 2024-03-04T13:11:42.445 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:11:42.445+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:11:42.445213+0000) 2024-03-04T13:11:43.448 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:11:43.445+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:11:43.445491+0000) 2024-03-04T13:11:44.445 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:11:44.444+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:11:44.445751+0000) 2024-03-04T13:11:45.445 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:11:45.444+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:11:45.446002+0000) 2024-03-04T13:11:46.445 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:11:46.445+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:11:46.446243+0000) 2024-03-04T13:11:47.446 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:11:47.445+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:11:47.446505+0000) 2024-03-04T13:11:48.446 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:11:48.445+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:11:48.446757+0000) 2024-03-04T13:11:49.446 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:11:49.445+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:11:49.446995+0000) 2024-03-04T13:11:50.446 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:11:50.446+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:11:50.447233+0000) 2024-03-04T13:11:51.447 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:11:51.446+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:11:51.447465+0000) 2024-03-04T13:11:52.447 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:11:52.445+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:11:52.447648+0000) 2024-03-04T13:11:53.447 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:11:53.446+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:11:53.447930+0000) 2024-03-04T13:11:54.447 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:11:54.446+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:11:54.448115+0000) 2024-03-04T13:11:55.447 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:11:55.447+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:11:55.448355+0000) 2024-03-04T13:11:56.448 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:11:56.447+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:11:56.448589+0000) 2024-03-04T13:11:57.448 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:11:57.448+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:11:57.448855+0000) 2024-03-04T13:11:58.448 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:11:58.448+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:11:58.449114+0000) 2024-03-04T13:11:59.448 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:11:59.448+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:11:59.449374+0000) 2024-03-04T13:12:00.449 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:12:00.448+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:12:00.449629+0000) 2024-03-04T13:12:01.449 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:12:01.449+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:12:01.449927+0000) 2024-03-04T13:12:02.061 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:12:02.070 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:12:02.449 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:12:02.449+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:12:02.450138+0000) 2024-03-04T13:12:03.450 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:12:03.449+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:12:03.450394+0000) 2024-03-04T13:12:04.450 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:12:04.449+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:12:04.450656+0000) 2024-03-04T13:12:05.450 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:12:05.450+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:12:05.450932+0000) 2024-03-04T13:12:06.450 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:12:06.450+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:12:06.451187+0000) 2024-03-04T13:12:07.451 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:12:07.450+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:12:07.451419+0000) 2024-03-04T13:12:08.451 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:12:08.450+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:12:08.451674+0000) 2024-03-04T13:12:09.451 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:12:09.451+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:12:09.451906+0000) 2024-03-04T13:12:10.451 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:12:10.451+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:12:10.452148+0000) 2024-03-04T13:12:11.451 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:12:11.451+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:12:11.452345+0000) 2024-03-04T13:12:12.452 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:12:12.451+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:12:12.452583+0000) 2024-03-04T13:12:13.452 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:12:13.452+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:12:13.452818+0000) 2024-03-04T13:12:14.452 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:12:14.452+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:12:14.453125+0000) 2024-03-04T13:12:15.452 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:12:15.452+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:12:15.453381+0000) 2024-03-04T13:12:16.453 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:12:16.453+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:12:16.453619+0000) 2024-03-04T13:12:17.453 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:12:17.453+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:12:17.453892+0000) 2024-03-04T13:12:18.453 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:12:18.453+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:12:18.454152+0000) 2024-03-04T13:12:19.454 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:12:19.453+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:12:19.454386+0000) 2024-03-04T13:12:20.454 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:12:20.454+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:12:20.454623+0000) 2024-03-04T13:12:21.454 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:12:21.454+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:12:21.454899+0000) 2024-03-04T13:12:22.454 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:12:22.454+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:12:22.455149+0000) 2024-03-04T13:12:23.454 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:12:23.454+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:12:23.455385+0000) 2024-03-04T13:12:24.455 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:12:24.455+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:12:24.455629+0000) 2024-03-04T13:12:25.456 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:12:25.455+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:12:25.455919+0000) 2024-03-04T13:12:26.455 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:12:26.455+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:12:26.456158+0000) 2024-03-04T13:12:27.455 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:12:27.455+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:12:27.456403+0000) 2024-03-04T13:12:28.456 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:12:28.456+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:12:28.456650+0000) 2024-03-04T13:12:29.456 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:12:29.456+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:12:29.456921+0000) 2024-03-04T13:12:30.456 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:12:30.456+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:12:30.457156+0000) 2024-03-04T13:12:31.456 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:12:31.457+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:12:31.457392+0000) 2024-03-04T13:12:32.292 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:12:32.300 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:12:32.456 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:12:32.456+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:12:32.457611+0000) 2024-03-04T13:12:33.459 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:12:33.457+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:12:33.457914+0000) 2024-03-04T13:12:34.457 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:12:34.457+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:12:34.458157+0000) 2024-03-04T13:12:35.457 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:12:35.457+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:12:35.458412+0000) 2024-03-04T13:12:36.457 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:12:36.458+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:12:36.458656+0000) 2024-03-04T13:12:37.458 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:12:37.458+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:12:37.458955+0000) 2024-03-04T13:12:38.458 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:12:38.458+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:12:38.459208+0000) 2024-03-04T13:12:39.458 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:12:39.459+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:12:39.459410+0000) 2024-03-04T13:12:40.458 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:12:40.459+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:12:40.459649+0000) 2024-03-04T13:12:41.459 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:12:41.459+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:12:41.459930+0000) 2024-03-04T13:12:42.459 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:12:42.459+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:12:42.460170+0000) 2024-03-04T13:12:43.459 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:12:43.460+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:12:43.460419+0000) 2024-03-04T13:12:44.459 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:12:44.459+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:12:44.460615+0000) 2024-03-04T13:12:45.460 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:12:45.460+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:12:45.460914+0000) 2024-03-04T13:12:46.460 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:12:46.460+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:12:46.461155+0000) 2024-03-04T13:12:47.460 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:12:47.460+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:12:47.461400+0000) 2024-03-04T13:12:48.460 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:12:48.460+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:12:48.461652+0000) 2024-03-04T13:12:49.461 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:12:49.460+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:12:49.461952+0000) 2024-03-04T13:12:50.461 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:12:50.461+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:12:50.462216+0000) 2024-03-04T13:12:51.461 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:12:51.460+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:12:51.462488+0000) 2024-03-04T13:12:52.461 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:12:52.461+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:12:52.462745+0000) 2024-03-04T13:12:53.462 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:12:53.461+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:12:53.463037+0000) 2024-03-04T13:12:54.462 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:12:54.462+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:12:54.463335+0000) 2024-03-04T13:12:55.462 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:12:55.462+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:12:55.463592+0000) 2024-03-04T13:12:56.462 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:12:56.462+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:12:56.463872+0000) 2024-03-04T13:12:57.463 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:12:57.463+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:12:57.464114+0000) 2024-03-04T13:12:58.463 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:12:58.463+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:12:58.464374+0000) 2024-03-04T13:12:59.463 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:12:59.463+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:12:59.464671+0000) 2024-03-04T13:13:00.464 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:13:00.464+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:13:00.464950+0000) 2024-03-04T13:13:01.464 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:13:01.464+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:13:01.465190+0000) 2024-03-04T13:13:02.464 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:13:02.464+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:13:02.465473+0000) 2024-03-04T13:13:02.497 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:13:02.505 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:13:03.464 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:13:03.464+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:13:03.465783+0000) 2024-03-04T13:13:04.465 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:13:04.465+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:13:04.466161+0000) 2024-03-04T13:13:05.465 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:13:05.465+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:13:05.466462+0000) 2024-03-04T13:13:06.465 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:13:06.465+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:13:06.466697+0000) 2024-03-04T13:13:07.466 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:13:07.466+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:13:07.467040+0000) 2024-03-04T13:13:08.466 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:13:08.466+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:13:08.467388+0000) 2024-03-04T13:13:09.466 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:13:09.466+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:13:09.467683+0000) 2024-03-04T13:13:10.466 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:13:10.467+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:13:10.467973+0000) 2024-03-04T13:13:11.467 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:13:11.467+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:13:11.468288+0000) 2024-03-04T13:13:12.467 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:13:12.467+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:13:12.468606+0000) 2024-03-04T13:13:13.467 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:13:13.468+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:13:13.468930+0000) 2024-03-04T13:13:14.468 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:13:14.468+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:13:14.469192+0000) 2024-03-04T13:13:15.468 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:13:15.468+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:13:15.469406+0000) 2024-03-04T13:13:16.468 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:13:16.468+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:13:16.469682+0000) 2024-03-04T13:13:17.468 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:13:17.469+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:13:17.469940+0000) 2024-03-04T13:13:18.469 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:13:18.469+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:13:18.470214+0000) 2024-03-04T13:13:19.469 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:13:19.469+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:13:19.470478+0000) 2024-03-04T13:13:20.469 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:13:20.469+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:13:20.470689+0000) 2024-03-04T13:13:21.469 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:13:21.470+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:13:21.470939+0000) 2024-03-04T13:13:22.470 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:13:22.470+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:13:22.471167+0000) 2024-03-04T13:13:23.470 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:13:23.470+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:13:23.471426+0000) 2024-03-04T13:13:24.470 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:13:24.470+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:13:24.471633+0000) 2024-03-04T13:13:25.472 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:13:25.471+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:13:25.471924+0000) 2024-03-04T13:13:26.471 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:13:26.471+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:13:26.472179+0000) 2024-03-04T13:13:27.471 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:13:27.471+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:13:27.472409+0000) 2024-03-04T13:13:28.471 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:13:28.472+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:13:28.472651+0000) 2024-03-04T13:13:29.471 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:13:29.472+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:13:29.472943+0000) 2024-03-04T13:13:30.472 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:13:30.472+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:13:30.473194+0000) 2024-03-04T13:13:31.472 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:13:31.473+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:13:31.473429+0000) 2024-03-04T13:13:32.472 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:13:32.473+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:13:32.473659+0000) 2024-03-04T13:13:32.710 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:13:32.717 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:13:33.472 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:13:33.473+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:13:33.473934+0000) 2024-03-04T13:13:34.472 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:13:34.473+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:13:34.474181+0000) 2024-03-04T13:13:35.473 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:13:35.474+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:13:35.474413+0000) 2024-03-04T13:13:36.473 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:13:36.474+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:13:36.474649+0000) 2024-03-04T13:13:37.473 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:13:37.474+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:13:37.474925+0000) 2024-03-04T13:13:38.473 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:13:38.474+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:13:38.475189+0000) 2024-03-04T13:13:39.474 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:13:39.475+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:13:39.475462+0000) 2024-03-04T13:13:40.474 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:13:40.475+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:13:40.475741+0000) 2024-03-04T13:13:41.474 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:13:41.475+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:13:41.476011+0000) 2024-03-04T13:13:42.475 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:13:42.476+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:13:42.476271+0000) 2024-03-04T13:13:43.475 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:13:43.476+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:13:43.476524+0000) 2024-03-04T13:13:44.475 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:13:44.476+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:13:44.476812+0000) 2024-03-04T13:13:45.475 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:13:45.476+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:13:45.477119+0000) 2024-03-04T13:13:46.476 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:13:46.477+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:13:46.477377+0000) 2024-03-04T13:13:47.476 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:13:47.477+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:13:47.477684+0000) 2024-03-04T13:13:48.476 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:13:48.477+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:13:48.477966+0000) 2024-03-04T13:13:49.477 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:13:49.478+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:13:49.478228+0000) 2024-03-04T13:13:50.477 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:13:50.476+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:13:50.478445+0000) 2024-03-04T13:13:51.477 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:13:51.477+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:13:51.478692+0000) 2024-03-04T13:13:52.477 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:13:52.477+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:13:52.478980+0000) 2024-03-04T13:13:53.477 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:13:53.478+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:13:53.479213+0000) 2024-03-04T13:13:54.478 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:13:54.478+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:13:54.479486+0000) 2024-03-04T13:13:55.478 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:13:55.478+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:13:55.479718+0000) 2024-03-04T13:13:56.478 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:13:56.478+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:13:56.479959+0000) 2024-03-04T13:13:57.478 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:13:57.479+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:13:57.480190+0000) 2024-03-04T13:13:58.479 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:13:58.479+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:13:58.480446+0000) 2024-03-04T13:13:59.479 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:13:59.479+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:13:59.480699+0000) 2024-03-04T13:14:00.479 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:14:00.480+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:14:00.480986+0000) 2024-03-04T13:14:01.479 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:14:01.480+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:14:01.481236+0000) 2024-03-04T13:14:02.480 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:14:02.480+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:14:02.481489+0000) 2024-03-04T13:14:02.931 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:14:02.939 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:14:03.480 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:14:03.480+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:14:03.481780+0000) 2024-03-04T13:14:04.480 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:14:04.481+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:14:04.482059+0000) 2024-03-04T13:14:05.480 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:14:05.481+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:14:05.482339+0000) 2024-03-04T13:14:06.481 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:14:06.481+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:14:06.482628+0000) 2024-03-04T13:14:07.481 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:14:07.482+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:14:07.482958+0000) 2024-03-04T13:14:08.481 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:14:08.482+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:14:08.483278+0000) 2024-03-04T13:14:09.482 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:14:09.482+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:14:09.483607+0000) 2024-03-04T13:14:10.482 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:14:10.483+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:14:10.483953+0000) 2024-03-04T13:14:11.482 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:14:11.483+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:14:11.484200+0000) 2024-03-04T13:14:12.482 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:14:12.483+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:14:12.484439+0000) 2024-03-04T13:14:13.483 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:14:13.483+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:14:13.484677+0000) 2024-03-04T13:14:14.483 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:14:14.484+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:14:14.484949+0000) 2024-03-04T13:14:15.483 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:14:15.484+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:14:15.485192+0000) 2024-03-04T13:14:16.483 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:14:16.484+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:14:16.485444+0000) 2024-03-04T13:14:17.484 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:14:17.484+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:14:17.485694+0000) 2024-03-04T13:14:18.484 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:14:18.485+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:14:18.485998+0000) 2024-03-04T13:14:19.484 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:14:19.485+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:14:19.486241+0000) 2024-03-04T13:14:20.484 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:14:20.485+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:14:20.486535+0000) 2024-03-04T13:14:21.485 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:14:21.486+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:14:21.486863+0000) 2024-03-04T13:14:22.485 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:14:22.486+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:14:22.487155+0000) 2024-03-04T13:14:23.486 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:14:23.486+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:14:23.487455+0000) 2024-03-04T13:14:24.486 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:14:24.487+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:14:24.487738+0000) 2024-03-04T13:14:25.486 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:14:25.487+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:14:25.488031+0000) 2024-03-04T13:14:26.486 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:14:26.487+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:14:26.488311+0000) 2024-03-04T13:14:27.487 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:14:27.487+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:14:27.488597+0000) 2024-03-04T13:14:28.487 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:14:28.488+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:14:28.488926+0000) 2024-03-04T13:14:29.487 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:14:29.488+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:14:29.489247+0000) 2024-03-04T13:14:30.487 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:14:30.488+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:14:30.489540+0000) 2024-03-04T13:14:31.488 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:14:31.489+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:14:31.489880+0000) 2024-03-04T13:14:32.488 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:14:32.489+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:14:32.490205+0000) 2024-03-04T13:14:33.085 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:14:33.093 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:14:33.488 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:14:33.489+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:14:33.490482+0000) 2024-03-04T13:14:34.488 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:14:34.489+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:14:34.490715+0000) 2024-03-04T13:14:35.489 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:14:35.490+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:14:35.491026+0000) 2024-03-04T13:14:36.489 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:14:36.490+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:14:36.491324+0000) 2024-03-04T13:14:37.489 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:14:37.490+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:14:37.491600+0000) 2024-03-04T13:14:38.490 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:14:38.491+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:14:38.491942+0000) 2024-03-04T13:14:39.490 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:14:39.491+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:14:39.492232+0000) 2024-03-04T13:14:40.490 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:14:40.492+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:14:40.492517+0000) 2024-03-04T13:14:41.491 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:14:41.492+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:14:41.492816+0000) 2024-03-04T13:14:42.491 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:14:42.492+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:14:42.493155+0000) 2024-03-04T13:14:43.491 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:14:43.493+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:14:43.493441+0000) 2024-03-04T13:14:44.491 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:14:44.493+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:14:44.493719+0000) 2024-03-04T13:14:45.492 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:14:45.493+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:14:45.494007+0000) 2024-03-04T13:14:46.492 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:14:46.493+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:14:46.494302+0000) 2024-03-04T13:14:47.492 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:14:47.493+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:14:47.494604+0000) 2024-03-04T13:14:48.493 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:14:48.494+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:14:48.494940+0000) 2024-03-04T13:14:49.493 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:14:49.494+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:14:49.495215+0000) 2024-03-04T13:14:50.493 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:14:50.495+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:14:50.495516+0000) 2024-03-04T13:14:51.493 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:14:51.495+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:14:51.495804+0000) 2024-03-04T13:14:52.494 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:14:52.495+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:14:52.496154+0000) 2024-03-04T13:14:53.494 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:14:53.496+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:14:53.496442+0000) 2024-03-04T13:14:54.494 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:14:54.496+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:14:54.496711+0000) 2024-03-04T13:14:55.495 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:14:55.496+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:14:55.497003+0000) 2024-03-04T13:14:56.495 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:14:56.496+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:14:56.497247+0000) 2024-03-04T13:14:57.495 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:14:57.496+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:14:57.497533+0000) 2024-03-04T13:14:58.496 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:14:58.496+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:14:58.497869+0000) 2024-03-04T13:14:59.496 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:14:59.497+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:14:59.498156+0000) 2024-03-04T13:15:00.496 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:15:00.497+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:15:00.498443+0000) 2024-03-04T13:15:01.496 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:15:01.497+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:15:01.498750+0000) 2024-03-04T13:15:02.497 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:15:02.497+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:15:02.499054+0000) 2024-03-04T13:15:03.272 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:15:03.280 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:15:03.497 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:15:03.497+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:15:03.499276+0000) 2024-03-04T13:15:04.497 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:15:04.498+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:15:04.499514+0000) 2024-03-04T13:15:05.497 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:15:05.498+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:15:05.499754+0000) 2024-03-04T13:15:06.498 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:15:06.499+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:15:06.499999+0000) 2024-03-04T13:15:07.498 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:15:07.499+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:15:07.500237+0000) 2024-03-04T13:15:08.498 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:15:08.499+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:15:08.500479+0000) 2024-03-04T13:15:09.498 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:15:09.499+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:15:09.500718+0000) 2024-03-04T13:15:10.498 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:15:10.500+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:15:10.500954+0000) 2024-03-04T13:15:11.499 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:15:11.500+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:15:11.501210+0000) 2024-03-04T13:15:12.499 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:15:12.500+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:15:12.501449+0000) 2024-03-04T13:15:13.499 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:15:13.500+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:15:13.501691+0000) 2024-03-04T13:15:14.499 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:15:14.501+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:15:14.501936+0000) 2024-03-04T13:15:15.500 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:15:15.501+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:15:15.502188+0000) 2024-03-04T13:15:16.500 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:15:16.501+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:15:16.502456+0000) 2024-03-04T13:15:17.500 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:15:17.501+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:15:17.502647+0000) 2024-03-04T13:15:18.500 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:15:18.502+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:15:18.502946+0000) 2024-03-04T13:15:19.501 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:15:19.502+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:15:19.503202+0000) 2024-03-04T13:15:20.501 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:15:20.502+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:15:20.503450+0000) 2024-03-04T13:15:21.501 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:15:21.502+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:15:21.503701+0000) 2024-03-04T13:15:22.501 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:15:22.503+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:15:22.503956+0000) 2024-03-04T13:15:23.502 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:15:23.503+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:15:23.504215+0000) 2024-03-04T13:15:24.502 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:15:24.503+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:15:24.504475+0000) 2024-03-04T13:15:25.502 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:15:25.504+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:15:25.504739+0000) 2024-03-04T13:15:26.502 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:15:26.504+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:15:26.505006+0000) 2024-03-04T13:15:27.503 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:15:27.504+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:15:27.505257+0000) 2024-03-04T13:15:28.503 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:15:28.504+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:15:28.505487+0000) 2024-03-04T13:15:29.503 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:15:29.505+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:15:29.505754+0000) 2024-03-04T13:15:30.503 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:15:30.505+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:15:30.506017+0000) 2024-03-04T13:15:31.504 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:15:31.505+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:15:31.506293+0000) 2024-03-04T13:15:32.504 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:15:32.505+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:15:32.506564+0000) 2024-03-04T13:15:33.478 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:15:33.486 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:15:33.504 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:15:33.505+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:15:33.506793+0000) 2024-03-04T13:15:34.504 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:15:34.506+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:15:34.507084+0000) 2024-03-04T13:15:35.505 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:15:35.506+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:15:35.507382+0000) 2024-03-04T13:15:36.505 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:15:36.507+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:15:36.507674+0000) 2024-03-04T13:15:37.505 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:15:37.507+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:15:37.507958+0000) 2024-03-04T13:15:38.506 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:15:38.507+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:15:38.508276+0000) 2024-03-04T13:15:39.506 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:15:39.507+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:15:39.508548+0000) 2024-03-04T13:15:40.506 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:15:40.508+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:15:40.508792+0000) 2024-03-04T13:15:41.506 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:15:41.508+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:15:41.509065+0000) 2024-03-04T13:15:42.507 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:15:42.508+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:15:42.509309+0000) 2024-03-04T13:15:43.507 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:15:43.509+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:15:43.509550+0000) 2024-03-04T13:15:44.507 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:15:44.508+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:15:44.509735+0000) 2024-03-04T13:15:45.507 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:15:45.509+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:15:45.509973+0000) 2024-03-04T13:15:46.508 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:15:46.509+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:15:46.510221+0000) 2024-03-04T13:15:47.508 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:15:47.509+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:15:47.510530+0000) 2024-03-04T13:15:48.508 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:15:48.510+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:15:48.510872+0000) 2024-03-04T13:15:49.509 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:15:49.510+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:15:49.511188+0000) 2024-03-04T13:15:50.509 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:15:50.511+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:15:50.511480+0000) 2024-03-04T13:15:51.509 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:15:51.511+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:15:51.511770+0000) 2024-03-04T13:15:52.509 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:15:52.511+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:15:52.512063+0000) 2024-03-04T13:15:53.509 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:15:53.512+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:15:53.512352+0000) 2024-03-04T13:15:54.510 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:15:54.512+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:15:54.512641+0000) 2024-03-04T13:15:55.510 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:15:55.512+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:15:55.512898+0000) 2024-03-04T13:15:56.510 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:15:56.512+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:15:56.513160+0000) 2024-03-04T13:15:57.511 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:15:57.513+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:15:57.513414+0000) 2024-03-04T13:15:58.512 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:15:58.513+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:15:58.513691+0000) 2024-03-04T13:15:59.511 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:15:59.513+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:15:59.513972+0000) 2024-03-04T13:16:00.511 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:16:00.514+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:16:00.514256+0000) 2024-03-04T13:16:01.512 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:16:01.514+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:16:01.514516+0000) 2024-03-04T13:16:02.512 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:16:02.514+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:16:02.514774+0000) 2024-03-04T13:16:03.512 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:16:03.514+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:16:03.515040+0000) 2024-03-04T13:16:03.641 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:16:03.649 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:16:04.513 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:16:04.514+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:16:04.515365+0000) 2024-03-04T13:16:05.513 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:16:05.515+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:16:05.515684+0000) 2024-03-04T13:16:06.513 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:16:06.515+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:16:06.515991+0000) 2024-03-04T13:16:07.513 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:16:07.514+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:16:07.516286+0000) 2024-03-04T13:16:08.514 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:16:08.515+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:16:08.516594+0000) 2024-03-04T13:16:09.514 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:16:09.515+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:16:09.516920+0000) 2024-03-04T13:16:10.514 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:16:10.516+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:16:10.517213+0000) 2024-03-04T13:16:11.514 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:16:11.516+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:16:11.517497+0000) 2024-03-04T13:16:12.515 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:16:12.515+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:16:12.517780+0000) 2024-03-04T13:16:13.515 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:16:13.516+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:16:13.518045+0000) 2024-03-04T13:16:14.515 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:16:14.516+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:16:14.518273+0000) 2024-03-04T13:16:15.516 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:16:15.516+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:16:15.518527+0000) 2024-03-04T13:16:16.516 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:16:16.516+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:16:16.518805+0000) 2024-03-04T13:16:17.516 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:16:17.518+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:16:17.519122+0000) 2024-03-04T13:16:18.516 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:16:18.518+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:16:18.519370+0000) 2024-03-04T13:16:19.516 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:16:19.518+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:16:19.519606+0000) 2024-03-04T13:16:20.517 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:16:20.518+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:16:20.519895+0000) 2024-03-04T13:16:21.517 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:16:21.519+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:16:21.520151+0000) 2024-03-04T13:16:22.517 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:16:22.519+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:16:22.520383+0000) 2024-03-04T13:16:23.517 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:16:23.519+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:16:23.520615+0000) 2024-03-04T13:16:24.518 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:16:24.520+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:16:24.520957+0000) 2024-03-04T13:16:25.518 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:16:25.520+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:16:25.521244+0000) 2024-03-04T13:16:26.518 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:16:26.520+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:16:26.521550+0000) 2024-03-04T13:16:27.520 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:16:27.521+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:16:27.521908+0000) 2024-03-04T13:16:28.519 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:16:28.521+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:16:28.522207+0000) 2024-03-04T13:16:29.519 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:16:29.521+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:16:29.522503+0000) 2024-03-04T13:16:30.519 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:16:30.521+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:16:30.522794+0000) 2024-03-04T13:16:31.520 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:16:31.522+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:16:31.523135+0000) 2024-03-04T13:16:32.520 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:16:32.522+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:16:32.523427+0000) 2024-03-04T13:16:33.520 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:16:33.522+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:16:33.523717+0000) 2024-03-04T13:16:33.863 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:16:33.871 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:16:34.521 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:16:34.523+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:16:34.524013+0000) 2024-03-04T13:16:35.521 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:16:35.523+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:16:35.524306+0000) 2024-03-04T13:16:36.522 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:16:36.523+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:16:36.524584+0000) 2024-03-04T13:16:37.522 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:16:37.524+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:16:37.524906+0000) 2024-03-04T13:16:38.522 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:16:38.524+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:16:38.525251+0000) 2024-03-04T13:16:39.522 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:16:39.524+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:16:39.525540+0000) 2024-03-04T13:16:40.522 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:16:40.525+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:16:40.525860+0000) 2024-03-04T13:16:41.523 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:16:41.525+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:16:41.526143+0000) 2024-03-04T13:16:42.523 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:16:42.525+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:16:42.526434+0000) 2024-03-04T13:16:43.523 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:16:43.525+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:16:43.526720+0000) 2024-03-04T13:16:44.524 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:16:44.526+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:16:44.527052+0000) 2024-03-04T13:16:45.524 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:16:45.526+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:16:45.527343+0000) 2024-03-04T13:16:46.524 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:16:46.526+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:16:46.527642+0000) 2024-03-04T13:16:47.524 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:16:47.527+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:16:47.527985+0000) 2024-03-04T13:16:48.525 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:16:48.527+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:16:48.528287+0000) 2024-03-04T13:16:49.525 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:16:49.527+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:16:49.528521+0000) 2024-03-04T13:16:50.525 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:16:50.528+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:16:50.528818+0000) 2024-03-04T13:16:51.526 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:16:51.528+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:16:51.529158+0000) 2024-03-04T13:16:52.526 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:16:52.528+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:16:52.529470+0000) 2024-03-04T13:16:53.526 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:16:53.529+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:16:53.529780+0000) 2024-03-04T13:16:54.527 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:16:54.529+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:16:54.530135+0000) 2024-03-04T13:16:55.527 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:16:55.529+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:16:55.530441+0000) 2024-03-04T13:16:56.527 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:16:56.530+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:16:56.530729+0000) 2024-03-04T13:16:57.528 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:16:57.530+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:16:57.531008+0000) 2024-03-04T13:16:58.528 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:16:58.530+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:16:58.531303+0000) 2024-03-04T13:16:59.528 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:16:59.530+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:16:59.531614+0000) 2024-03-04T13:17:00.528 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:17:00.531+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:17:00.531941+0000) 2024-03-04T13:17:01.529 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:17:01.531+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:17:01.532229+0000) 2024-03-04T13:17:02.529 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:17:02.531+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:17:02.532522+0000) 2024-03-04T13:17:03.529 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:17:03.532+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:17:03.532816+0000) 2024-03-04T13:17:03.999 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:17:04.007 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:17:04.529 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:17:04.532+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:17:04.533119+0000) 2024-03-04T13:17:05.530 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:17:05.532+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:17:05.533369+0000) 2024-03-04T13:17:06.530 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:17:06.533+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:17:06.533625+0000) 2024-03-04T13:17:07.530 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:17:07.533+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:17:07.533919+0000) 2024-03-04T13:17:08.530 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:17:08.533+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:17:08.534180+0000) 2024-03-04T13:17:09.531 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:17:09.533+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:17:09.534438+0000) 2024-03-04T13:17:10.531 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:17:10.533+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:17:10.534711+0000) 2024-03-04T13:17:11.531 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:17:11.534+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:17:11.534991+0000) 2024-03-04T13:17:12.531 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:17:12.534+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:17:12.535234+0000) 2024-03-04T13:17:13.532 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:17:13.534+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:17:13.535416+0000) 2024-03-04T13:17:14.532 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:17:14.535+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:17:14.535660+0000) 2024-03-04T13:17:15.532 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:17:15.535+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:17:15.535977+0000) 2024-03-04T13:17:16.532 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:17:16.535+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:17:16.536212+0000) 2024-03-04T13:17:17.533 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:17:17.536+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:17:17.536444+0000) 2024-03-04T13:17:18.533 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:17:18.536+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:17:18.536685+0000) 2024-03-04T13:17:19.533 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:17:19.536+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:17:19.536924+0000) 2024-03-04T13:17:20.533 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:17:20.536+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:17:20.537186+0000) 2024-03-04T13:17:21.534 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:17:21.536+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:17:21.537414+0000) 2024-03-04T13:17:22.534 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:17:22.537+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:17:22.537650+0000) 2024-03-04T13:17:23.534 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:17:23.537+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:17:23.537931+0000) 2024-03-04T13:17:24.534 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:17:24.537+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:17:24.538174+0000) 2024-03-04T13:17:25.534 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:17:25.538+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:17:25.538413+0000) 2024-03-04T13:17:26.535 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:17:26.538+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:17:26.538657+0000) 2024-03-04T13:17:27.535 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:17:27.538+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:17:27.538957+0000) 2024-03-04T13:17:28.535 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:17:28.538+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:17:28.539225+0000) 2024-03-04T13:17:29.535 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:17:29.539+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:17:29.539485+0000) 2024-03-04T13:17:30.536 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:17:30.539+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:17:30.539747+0000) 2024-03-04T13:17:31.536 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:17:31.539+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:17:31.540003+0000) 2024-03-04T13:17:32.536 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:17:32.539+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:17:32.540254+0000) 2024-03-04T13:17:33.536 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:17:33.540+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:17:33.540488+0000) 2024-03-04T13:17:34.147 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:17:34.155 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:17:34.537 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:17:34.540+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:17:34.540742+0000) 2024-03-04T13:17:35.537 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:17:35.540+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:17:35.541011+0000) 2024-03-04T13:17:36.537 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:17:36.540+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:17:36.541251+0000) 2024-03-04T13:17:37.537 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:17:37.541+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:17:37.541492+0000) 2024-03-04T13:17:38.538 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:17:38.541+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:17:38.541742+0000) 2024-03-04T13:17:39.538 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:17:39.541+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:17:39.541985+0000) 2024-03-04T13:17:40.538 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:17:40.542+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:17:40.542271+0000) 2024-03-04T13:17:41.538 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:17:41.542+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:17:41.542513+0000) 2024-03-04T13:17:42.538 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:17:42.542+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:17:42.542749+0000) 2024-03-04T13:17:43.539 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:17:43.542+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:17:43.542983+0000) 2024-03-04T13:17:44.539 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:17:44.543+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:17:44.543231+0000) 2024-03-04T13:17:45.539 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:17:45.543+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:17:45.543472+0000) 2024-03-04T13:17:46.540 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:17:46.543+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:17:46.543711+0000) 2024-03-04T13:17:47.540 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:17:47.543+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:17:47.544005+0000) 2024-03-04T13:17:48.540 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:17:48.544+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:17:48.544262+0000) 2024-03-04T13:17:49.540 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:17:49.544+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:17:49.544504+0000) 2024-03-04T13:17:50.540 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:17:50.544+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:17:50.544753+0000) 2024-03-04T13:17:51.541 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:17:51.544+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:17:51.545007+0000) 2024-03-04T13:17:52.541 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:17:52.545+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:17:52.545259+0000) 2024-03-04T13:17:53.541 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:17:53.545+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:17:53.545504+0000) 2024-03-04T13:17:54.541 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:17:54.545+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:17:54.545751+0000) 2024-03-04T13:17:55.542 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:17:55.544+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:17:55.545991+0000) 2024-03-04T13:17:56.542 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:17:56.545+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:17:56.546232+0000) 2024-03-04T13:17:57.542 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:17:57.545+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:17:57.546421+0000) 2024-03-04T13:17:58.542 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:17:58.544+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:17:58.546636+0000) 2024-03-04T13:17:59.542 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:17:59.545+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:17:59.546930+0000) 2024-03-04T13:18:00.543 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:18:00.546+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:18:00.547173+0000) 2024-03-04T13:18:01.543 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:18:01.546+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:18:01.547429+0000) 2024-03-04T13:18:02.543 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:18:02.546+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:18:02.547668+0000) 2024-03-04T13:18:03.543 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:18:03.546+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:18:03.547939+0000) 2024-03-04T13:18:04.295 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:18:04.307 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:18:04.544 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:18:04.547+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:18:04.548195+0000) 2024-03-04T13:18:05.544 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:18:05.547+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:18:05.548439+0000) 2024-03-04T13:18:06.544 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:18:06.547+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:18:06.548679+0000) 2024-03-04T13:18:07.544 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:18:07.547+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:18:07.548950+0000) 2024-03-04T13:18:08.545 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:18:08.548+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:18:08.549202+0000) 2024-03-04T13:18:09.545 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:18:09.548+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:18:09.549519+0000) 2024-03-04T13:18:10.545 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:18:10.548+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:18:10.549764+0000) 2024-03-04T13:18:11.545 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:18:11.549+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:18:11.550010+0000) 2024-03-04T13:18:12.546 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:18:12.549+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:18:12.550252+0000) 2024-03-04T13:18:13.546 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:18:13.549+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:18:13.550495+0000) 2024-03-04T13:18:14.546 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:18:14.549+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:18:14.550740+0000) 2024-03-04T13:18:15.546 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:18:15.550+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:18:15.550985+0000) 2024-03-04T13:18:16.547 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:18:16.550+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:18:16.551176+0000) 2024-03-04T13:18:17.547 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:18:17.550+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:18:17.551366+0000) 2024-03-04T13:18:18.547 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:18:18.550+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:18:18.551564+0000) 2024-03-04T13:18:19.547 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:18:19.550+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:18:19.551804+0000) 2024-03-04T13:18:20.547 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:18:20.551+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:18:20.552040+0000) 2024-03-04T13:18:21.548 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:18:21.551+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:18:21.552284+0000) 2024-03-04T13:18:22.548 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:18:22.551+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:18:22.552475+0000) 2024-03-04T13:18:23.548 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:18:23.551+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:18:23.552752+0000) 2024-03-04T13:18:24.548 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:18:24.552+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:18:24.553018+0000) 2024-03-04T13:18:25.549 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:18:25.552+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:18:25.553277+0000) 2024-03-04T13:18:26.549 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:18:26.552+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:18:26.553474+0000) 2024-03-04T13:18:27.549 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:18:27.552+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:18:27.553718+0000) 2024-03-04T13:18:28.549 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:18:28.553+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:18:28.553972+0000) 2024-03-04T13:18:29.549 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:18:29.553+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:18:29.554205+0000) 2024-03-04T13:18:30.550 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:18:30.553+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:18:30.554452+0000) 2024-03-04T13:18:31.550 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:18:31.553+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:18:31.554691+0000) 2024-03-04T13:18:32.550 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:18:32.554+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:18:32.554960+0000) 2024-03-04T13:18:33.551 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:18:33.554+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:18:33.555204+0000) 2024-03-04T13:18:34.439 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:18:34.447 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:18:34.551 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:18:34.554+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:18:34.555395+0000) 2024-03-04T13:18:35.551 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:18:35.555+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:18:35.555672+0000) 2024-03-04T13:18:36.551 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:18:36.555+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:18:36.555939+0000) 2024-03-04T13:18:37.551 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:18:37.555+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:18:37.556131+0000) 2024-03-04T13:18:38.552 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:18:38.555+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:18:38.556338+0000) 2024-03-04T13:18:39.552 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:18:39.555+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:18:39.556578+0000) 2024-03-04T13:18:40.552 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:18:40.556+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:18:40.556873+0000) 2024-03-04T13:18:41.552 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:18:41.556+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:18:41.557060+0000) 2024-03-04T13:18:42.553 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:18:42.556+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:18:42.557303+0000) 2024-03-04T13:18:43.553 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:18:43.556+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:18:43.557545+0000) 2024-03-04T13:18:44.553 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:18:44.556+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:18:44.557770+0000) 2024-03-04T13:18:45.553 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:18:45.557+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:18:45.557977+0000) 2024-03-04T13:18:46.553 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:18:46.557+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:18:46.558222+0000) 2024-03-04T13:18:47.554 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:18:47.557+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:18:47.558424+0000) 2024-03-04T13:18:48.554 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:18:48.557+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:18:48.558745+0000) 2024-03-04T13:18:49.554 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:18:49.558+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:18:49.559042+0000) 2024-03-04T13:18:50.555 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:18:50.558+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:18:50.559341+0000) 2024-03-04T13:18:51.555 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:18:51.559+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:18:51.559631+0000) 2024-03-04T13:18:52.556 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:18:52.559+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:18:52.559963+0000) 2024-03-04T13:18:53.555 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:18:53.559+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:18:53.560245+0000) 2024-03-04T13:18:54.556 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:18:54.560+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:18:54.560499+0000) 2024-03-04T13:18:55.556 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:18:55.560+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:18:55.560733+0000) 2024-03-04T13:18:56.556 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:18:56.560+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:18:56.560969+0000) 2024-03-04T13:18:57.556 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:18:57.560+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:18:57.561206+0000) 2024-03-04T13:18:58.557 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:18:58.561+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:18:58.561454+0000) 2024-03-04T13:18:59.557 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:18:59.561+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:18:59.561686+0000) 2024-03-04T13:19:00.557 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:19:00.561+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:19:00.561930+0000) 2024-03-04T13:19:01.557 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:19:01.561+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:19:01.562166+0000) 2024-03-04T13:19:02.557 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:19:02.561+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:19:02.562434+0000) 2024-03-04T13:19:03.558 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:19:03.562+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:19:03.562669+0000) 2024-03-04T13:19:04.558 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:19:04.562+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:19:04.562891+0000) 2024-03-04T13:19:04.595 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:19:04.603 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:19:05.558 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:19:05.562+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:19:05.563129+0000) 2024-03-04T13:19:06.558 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:19:06.563+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:19:06.563370+0000) 2024-03-04T13:19:07.559 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:19:07.563+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:19:07.563683+0000) 2024-03-04T13:19:08.559 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:19:08.563+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:19:08.563945+0000) 2024-03-04T13:19:09.560 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:19:09.563+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:19:09.564180+0000) 2024-03-04T13:19:10.559 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:19:10.564+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:19:10.564419+0000) 2024-03-04T13:19:11.560 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:19:11.564+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:19:11.564652+0000) 2024-03-04T13:19:12.560 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:19:12.564+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:19:12.564928+0000) 2024-03-04T13:19:13.560 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:19:13.565+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:19:13.565168+0000) 2024-03-04T13:19:14.560 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:19:14.565+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:19:14.565415+0000) 2024-03-04T13:19:15.561 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:19:15.565+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:19:15.565684+0000) 2024-03-04T13:19:16.561 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:19:16.564+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:19:16.565952+0000) 2024-03-04T13:19:17.561 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:19:17.565+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:19:17.566190+0000) 2024-03-04T13:19:18.561 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:19:18.565+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:19:18.566436+0000) 2024-03-04T13:19:19.562 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:19:19.565+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:19:19.566688+0000) 2024-03-04T13:19:20.562 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:19:20.565+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:19:20.566949+0000) 2024-03-04T13:19:21.562 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:19:21.566+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:19:21.567202+0000) 2024-03-04T13:19:22.562 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:19:22.566+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:19:22.567458+0000) 2024-03-04T13:19:23.563 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:19:23.566+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:19:23.567720+0000) 2024-03-04T13:19:24.563 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:19:24.566+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:19:24.568001+0000) 2024-03-04T13:19:25.563 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:19:25.567+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:19:25.568263+0000) 2024-03-04T13:19:26.563 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:19:26.567+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:19:26.568508+0000) 2024-03-04T13:19:27.564 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:19:27.567+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:19:27.568744+0000) 2024-03-04T13:19:28.564 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:19:28.568+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:19:28.569008+0000) 2024-03-04T13:19:29.564 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:19:29.568+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:19:29.569245+0000) 2024-03-04T13:19:30.564 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:19:30.568+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:19:30.569504+0000) 2024-03-04T13:19:31.565 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:19:31.568+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:19:31.569739+0000) 2024-03-04T13:19:32.565 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:19:32.569+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:19:32.569976+0000) 2024-03-04T13:19:33.565 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:19:33.569+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:19:33.570212+0000) 2024-03-04T13:19:34.565 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:19:34.569+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:19:34.570446+0000) 2024-03-04T13:19:34.732 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:19:34.744 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:19:35.566 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:19:35.569+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:19:35.570711+0000) 2024-03-04T13:19:36.566 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:19:36.570+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:19:36.570963+0000) 2024-03-04T13:19:37.566 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:19:37.570+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:19:37.571188+0000) 2024-03-04T13:19:38.566 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:19:38.570+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:19:38.571499+0000) 2024-03-04T13:19:39.567 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:19:39.570+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:19:39.571731+0000) 2024-03-04T13:19:40.567 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:19:40.571+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:19:40.571963+0000) 2024-03-04T13:19:41.567 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:19:41.571+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:19:41.572188+0000) 2024-03-04T13:19:42.567 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:19:42.571+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:19:42.572418+0000) 2024-03-04T13:19:43.567 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:19:43.571+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:19:43.572631+0000) 2024-03-04T13:19:44.568 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:19:44.572+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:19:44.572887+0000) 2024-03-04T13:19:45.568 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:19:45.572+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:19:45.573121+0000) 2024-03-04T13:19:46.568 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:19:46.572+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:19:46.573334+0000) 2024-03-04T13:19:47.568 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:19:47.572+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:19:47.573522+0000) 2024-03-04T13:19:48.569 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:19:48.572+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:19:48.573719+0000) 2024-03-04T13:19:49.569 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:19:49.573+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:19:49.573949+0000) 2024-03-04T13:19:50.569 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:19:50.573+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:19:50.574156+0000) 2024-03-04T13:19:51.569 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:19:51.573+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:19:51.574352+0000) 2024-03-04T13:19:52.569 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:19:52.573+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:19:52.574542+0000) 2024-03-04T13:19:53.569 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:19:53.573+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:19:53.574733+0000) 2024-03-04T13:19:54.570 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:19:54.574+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:19:54.574938+0000) 2024-03-04T13:19:55.570 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:19:55.574+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:19:55.575133+0000) 2024-03-04T13:19:56.570 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:19:56.574+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:19:56.575327+0000) 2024-03-04T13:19:57.570 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:19:57.574+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:19:57.575529+0000) 2024-03-04T13:19:58.570 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:19:58.575+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:19:58.575734+0000) 2024-03-04T13:19:59.570 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:19:59.575+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:19:59.575924+0000) 2024-03-04T13:20:00.571 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:20:00.575+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:20:00.576127+0000) 2024-03-04T13:20:01.571 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:20:01.575+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:20:01.576369+0000) 2024-03-04T13:20:02.571 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:20:02.575+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:20:02.576564+0000) 2024-03-04T13:20:03.571 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:20:03.575+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:20:03.576737+0000) 2024-03-04T13:20:04.572 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:20:04.576+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:20:04.576948+0000) 2024-03-04T13:20:04.938 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:20:04.946 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:20:05.572 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:20:05.576+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:20:05.577221+0000) 2024-03-04T13:20:06.572 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:20:06.576+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:20:06.577404+0000) 2024-03-04T13:20:07.572 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:20:07.576+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:20:07.577630+0000) 2024-03-04T13:20:08.572 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:20:08.577+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:20:08.577815+0000) 2024-03-04T13:20:09.573 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:20:09.577+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:20:09.578083+0000) 2024-03-04T13:20:10.573 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:20:10.577+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:20:10.578345+0000) 2024-03-04T13:20:11.573 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:20:11.577+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:20:11.578526+0000) 2024-03-04T13:20:12.573 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:20:12.577+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:20:12.578778+0000) 2024-03-04T13:20:13.574 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:20:13.578+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:20:13.578940+0000) 2024-03-04T13:20:14.574 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:20:14.578+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:20:14.579196+0000) 2024-03-04T13:20:15.574 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:20:15.578+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:20:15.579433+0000) 2024-03-04T13:20:16.574 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:20:16.578+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:20:16.579613+0000) 2024-03-04T13:20:17.574 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:20:17.579+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:20:17.579924+0000) 2024-03-04T13:20:18.575 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:20:18.579+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:20:18.580107+0000) 2024-03-04T13:20:19.575 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:20:19.580+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:20:19.580362+0000) 2024-03-04T13:20:20.575 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:20:20.580+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:20:20.580625+0000) 2024-03-04T13:20:21.575 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:20:21.580+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:20:21.580907+0000) 2024-03-04T13:20:22.576 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:20:22.580+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:20:22.581152+0000) 2024-03-04T13:20:23.576 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:20:23.581+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:20:23.581385+0000) 2024-03-04T13:20:24.576 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:20:24.581+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:20:24.581617+0000) 2024-03-04T13:20:25.576 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:20:25.580+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:20:25.581801+0000) 2024-03-04T13:20:26.577 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:20:26.581+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:20:26.582070+0000) 2024-03-04T13:20:27.577 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:20:27.582+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:20:27.582318+0000) 2024-03-04T13:20:28.577 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:20:28.581+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:20:28.582566+0000) 2024-03-04T13:20:29.577 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:20:29.580+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:20:29.582798+0000) 2024-03-04T13:20:30.578 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:20:30.582+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:20:30.583101+0000) 2024-03-04T13:20:31.578 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:20:31.581+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:20:31.583290+0000) 2024-03-04T13:20:32.578 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:20:32.581+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:20:32.583528+0000) 2024-03-04T13:20:33.578 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:20:33.582+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:20:33.583770+0000) 2024-03-04T13:20:34.578 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:20:34.582+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:20:34.584007+0000) 2024-03-04T13:20:35.076 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:20:35.084 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:20:35.579 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:20:35.582+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:20:35.584199+0000) 2024-03-04T13:20:36.579 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:20:36.583+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:20:36.584437+0000) 2024-03-04T13:20:37.579 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:20:37.583+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:20:37.584675+0000) 2024-03-04T13:20:38.579 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:20:38.583+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:20:38.584952+0000) 2024-03-04T13:20:39.580 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:20:39.584+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:20:39.585190+0000) 2024-03-04T13:20:40.580 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:20:40.584+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:20:40.585428+0000) 2024-03-04T13:20:41.580 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:20:41.584+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:20:41.585670+0000) 2024-03-04T13:20:42.581 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:20:42.585+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:20:42.585887+0000) 2024-03-04T13:20:43.580 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:20:43.585+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:20:43.586126+0000) 2024-03-04T13:20:44.581 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:20:44.585+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:20:44.586398+0000) 2024-03-04T13:20:45.581 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:20:45.585+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:20:45.586579+0000) 2024-03-04T13:20:46.581 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:20:46.586+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:20:46.586816+0000) 2024-03-04T13:20:47.581 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:20:47.586+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:20:47.587090+0000) 2024-03-04T13:20:48.582 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:20:48.586+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:20:48.587347+0000) 2024-03-04T13:20:49.582 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:20:49.586+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:20:49.587594+0000) 2024-03-04T13:20:50.582 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:20:50.587+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:20:50.587845+0000) 2024-03-04T13:20:51.582 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:20:51.587+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:20:51.588039+0000) 2024-03-04T13:20:52.583 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:20:52.587+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:20:52.588273+0000) 2024-03-04T13:20:53.583 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:20:53.587+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:20:53.588525+0000) 2024-03-04T13:20:54.583 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:20:54.588+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:20:54.588764+0000) 2024-03-04T13:20:55.583 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:20:55.588+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:20:55.589006+0000) 2024-03-04T13:20:56.584 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:20:56.588+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:20:56.589253+0000) 2024-03-04T13:20:57.584 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:20:57.588+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:20:57.589492+0000) 2024-03-04T13:20:58.585 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:20:58.589+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:20:58.589742+0000) 2024-03-04T13:20:59.584 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:20:59.589+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:20:59.590032+0000) 2024-03-04T13:21:00.585 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:21:00.589+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:21:00.590218+0000) 2024-03-04T13:21:02.052 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:21:01.589+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:21:01.590405+0000) 2024-03-04T13:21:02.585 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:21:02.590+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:21:02.590664+0000) 2024-03-04T13:21:03.585 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:21:03.590+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:21:03.590938+0000) 2024-03-04T13:21:04.585 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:21:04.590+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:21:04.591173+0000) 2024-03-04T13:21:05.280 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:21:05.288 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:21:05.586 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:21:05.590+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:21:05.591427+0000) 2024-03-04T13:21:06.586 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:21:06.591+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:21:06.591669+0000) 2024-03-04T13:21:07.586 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:21:07.591+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:21:07.591946+0000) 2024-03-04T13:21:08.586 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:21:08.591+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:21:08.592216+0000) 2024-03-04T13:21:09.587 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:21:09.591+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:21:09.592465+0000) 2024-03-04T13:21:10.587 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:21:10.591+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:21:10.592658+0000) 2024-03-04T13:21:11.587 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:21:11.592+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:21:11.592871+0000) 2024-03-04T13:21:12.587 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:21:12.592+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:21:12.593056+0000) 2024-03-04T13:21:13.587 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:21:13.592+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:21:13.593249+0000) 2024-03-04T13:21:14.588 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:21:14.592+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:21:14.593486+0000) 2024-03-04T13:21:15.588 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:21:15.593+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:21:15.593738+0000) 2024-03-04T13:21:16.589 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:21:16.593+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:21:16.593973+0000) 2024-03-04T13:21:17.588 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:21:17.593+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:21:17.594165+0000) 2024-03-04T13:21:18.588 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:21:18.593+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:21:18.594411+0000) 2024-03-04T13:21:19.589 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:21:19.594+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:21:19.594644+0000) 2024-03-04T13:21:20.589 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:21:20.594+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:21:20.594933+0000) 2024-03-04T13:21:21.589 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:21:21.594+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:21:21.595169+0000) 2024-03-04T13:21:22.590 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:21:22.595+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:21:22.595441+0000) 2024-03-04T13:21:23.590 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:21:23.595+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:21:23.595674+0000) 2024-03-04T13:21:24.590 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:21:24.595+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:21:24.595942+0000) 2024-03-04T13:21:25.590 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:21:25.595+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:21:25.596200+0000) 2024-03-04T13:21:26.591 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:21:26.596+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:21:26.596445+0000) 2024-03-04T13:21:27.591 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:21:27.596+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:21:27.596678+0000) 2024-03-04T13:21:28.591 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:21:28.596+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:21:28.596923+0000) 2024-03-04T13:21:29.591 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:21:29.596+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:21:29.597158+0000) 2024-03-04T13:21:30.591 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:21:30.597+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:21:30.597396+0000) 2024-03-04T13:21:31.592 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:21:31.596+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:21:31.597639+0000) 2024-03-04T13:21:32.592 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:21:32.597+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:21:32.597929+0000) 2024-03-04T13:21:33.592 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:21:33.597+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:21:33.598190+0000) 2024-03-04T13:21:34.592 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:21:34.597+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:21:34.598369+0000) 2024-03-04T13:21:35.468 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:21:35.476 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:21:35.610 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:21:35.597+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:21:35.598601+0000) 2024-03-04T13:21:36.593 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:21:36.598+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:21:36.598868+0000) 2024-03-04T13:21:37.593 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:21:37.598+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:21:37.599111+0000) 2024-03-04T13:21:38.593 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:21:38.598+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:21:38.599357+0000) 2024-03-04T13:21:39.594 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:21:39.598+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:21:39.599535+0000) 2024-03-04T13:21:40.594 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:21:40.597+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:21:40.599790+0000) 2024-03-04T13:21:41.594 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:21:41.598+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:21:41.600026+0000) 2024-03-04T13:21:42.595 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:21:42.599+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:21:42.600267+0000) 2024-03-04T13:21:43.595 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:21:43.599+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:21:43.600499+0000) 2024-03-04T13:21:44.595 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:21:44.599+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:21:44.600681+0000) 2024-03-04T13:21:45.595 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:21:45.599+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:21:45.600927+0000) 2024-03-04T13:21:46.595 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:21:46.599+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:21:46.601140+0000) 2024-03-04T13:21:47.595 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:21:47.600+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:21:47.601442+0000) 2024-03-04T13:21:48.596 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:21:48.600+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:21:48.601703+0000) 2024-03-04T13:21:49.596 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:21:49.601+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:21:49.601952+0000) 2024-03-04T13:21:50.596 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:21:50.601+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:21:50.602209+0000) 2024-03-04T13:21:51.596 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:21:51.601+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:21:51.602458+0000) 2024-03-04T13:21:52.597 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:21:52.601+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:21:52.602707+0000) 2024-03-04T13:21:53.597 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:21:53.602+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:21:53.602990+0000) 2024-03-04T13:21:54.597 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:21:54.602+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:21:54.603228+0000) 2024-03-04T13:21:55.597 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:21:55.602+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:21:55.603475+0000) 2024-03-04T13:21:56.598 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:21:56.602+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:21:56.603673+0000) 2024-03-04T13:21:57.598 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:21:57.603+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:21:57.603931+0000) 2024-03-04T13:21:58.598 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:21:58.603+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:21:58.604177+0000) 2024-03-04T13:21:59.598 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:21:59.603+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:21:59.604412+0000) 2024-03-04T13:22:00.599 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:22:00.603+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:22:00.604656+0000) 2024-03-04T13:22:01.599 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:22:01.604+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:22:01.604931+0000) 2024-03-04T13:22:02.599 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:22:02.604+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:22:02.605165+0000) 2024-03-04T13:22:03.600 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:22:03.604+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:22:03.605398+0000) 2024-03-04T13:22:04.600 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:22:04.604+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:22:04.605655+0000) 2024-03-04T13:22:05.600 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:22:05.605+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:22:05.605938+0000) 2024-03-04T13:22:05.674 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:22:05.685 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:22:06.600 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:22:06.605+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:22:06.606201+0000) 2024-03-04T13:22:07.600 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:22:07.605+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:22:07.606436+0000) 2024-03-04T13:22:08.601 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:22:08.606+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:22:08.606683+0000) 2024-03-04T13:22:09.601 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:22:09.606+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:22:09.606933+0000) 2024-03-04T13:22:10.601 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:22:10.606+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:22:10.607165+0000) 2024-03-04T13:22:11.601 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:22:11.606+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:22:11.607401+0000) 2024-03-04T13:22:12.602 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:22:12.607+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:22:12.607634+0000) 2024-03-04T13:22:13.602 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:22:13.607+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:22:13.607925+0000) 2024-03-04T13:22:14.602 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:22:14.607+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:22:14.608169+0000) 2024-03-04T13:22:15.602 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:22:15.607+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:22:15.608406+0000) 2024-03-04T13:22:16.603 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:22:16.607+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:22:16.608643+0000) 2024-03-04T13:22:17.603 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:22:17.608+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:22:17.608921+0000) 2024-03-04T13:22:18.605 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:22:18.608+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:22:18.609170+0000) 2024-03-04T13:22:19.603 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:22:19.608+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:22:19.609415+0000) 2024-03-04T13:22:20.604 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:22:20.609+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:22:20.609654+0000) 2024-03-04T13:22:21.604 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:22:21.609+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:22:21.609934+0000) 2024-03-04T13:22:22.604 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:22:22.609+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:22:22.610176+0000) 2024-03-04T13:22:23.604 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:22:23.609+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:22:23.610409+0000) 2024-03-04T13:22:24.604 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:22:24.610+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:22:24.610645+0000) 2024-03-04T13:22:25.605 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:22:25.610+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:22:25.610934+0000) 2024-03-04T13:22:26.605 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:22:26.610+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:22:26.611177+0000) 2024-03-04T13:22:27.605 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:22:27.611+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:22:27.611444+0000) 2024-03-04T13:22:28.606 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:22:28.610+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:22:28.611688+0000) 2024-03-04T13:22:29.606 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:22:29.611+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:22:29.611959+0000) 2024-03-04T13:22:30.606 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:22:30.611+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:22:30.612201+0000) 2024-03-04T13:22:31.606 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:22:31.612+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:22:31.612443+0000) 2024-03-04T13:22:32.607 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:22:32.612+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:22:32.612684+0000) 2024-03-04T13:22:33.607 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:22:33.612+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:22:33.612922+0000) 2024-03-04T13:22:34.608 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:22:34.612+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:22:34.613167+0000) 2024-03-04T13:22:35.607 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:22:35.613+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:22:35.613432+0000) 2024-03-04T13:22:35.820 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:22:35.828 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:22:36.607 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:22:36.613+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:22:36.613693+0000) 2024-03-04T13:22:37.608 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:22:37.613+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:22:37.613950+0000) 2024-03-04T13:22:38.608 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:22:38.613+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:22:38.614196+0000) 2024-03-04T13:22:39.608 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:22:39.614+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:22:39.614430+0000) 2024-03-04T13:22:40.609 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:22:40.614+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:22:40.614674+0000) 2024-03-04T13:22:41.609 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:22:41.614+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:22:41.614985+0000) 2024-03-04T13:22:42.609 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:22:42.615+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:22:42.615225+0000) 2024-03-04T13:22:43.609 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:22:43.615+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:22:43.615478+0000) 2024-03-04T13:22:44.609 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:22:44.614+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:22:44.615719+0000) 2024-03-04T13:22:45.610 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:22:45.614+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:22:45.615960+0000) 2024-03-04T13:22:46.610 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:22:46.615+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:22:46.616204+0000) 2024-03-04T13:22:47.610 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:22:47.614+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:22:47.616458+0000) 2024-03-04T13:22:48.611 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:22:48.615+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:22:48.616703+0000) 2024-03-04T13:22:49.611 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:22:49.615+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:22:49.616954+0000) 2024-03-04T13:22:50.611 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:22:50.615+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:22:50.617187+0000) 2024-03-04T13:22:51.611 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:22:51.616+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:22:51.617456+0000) 2024-03-04T13:22:52.611 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:22:52.616+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:22:52.617695+0000) 2024-03-04T13:22:53.612 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:22:53.616+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:22:53.617932+0000) 2024-03-04T13:22:54.612 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:22:54.617+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:22:54.618167+0000) 2024-03-04T13:22:55.612 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:22:55.617+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:22:55.618400+0000) 2024-03-04T13:22:56.612 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:22:56.617+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:22:56.618608+0000) 2024-03-04T13:22:57.613 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:22:57.618+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:22:57.618885+0000) 2024-03-04T13:22:58.613 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:22:58.618+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:22:58.619141+0000) 2024-03-04T13:22:59.613 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:22:59.618+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:22:59.619374+0000) 2024-03-04T13:23:00.614 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:23:00.618+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:23:00.619613+0000) 2024-03-04T13:23:01.614 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:23:01.619+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:23:01.619878+0000) 2024-03-04T13:23:02.614 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:23:02.619+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:23:02.620085+0000) 2024-03-04T13:23:03.614 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:23:03.619+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:23:03.620320+0000) 2024-03-04T13:23:04.614 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:23:04.619+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:23:04.620555+0000) 2024-03-04T13:23:05.615 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:23:05.620+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:23:05.620807+0000) 2024-03-04T13:23:06.098 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:23:06.106 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:23:06.615 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:23:06.620+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:23:06.621036+0000) 2024-03-04T13:23:07.615 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:23:07.620+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:23:07.621297+0000) 2024-03-04T13:23:08.615 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:23:08.620+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:23:08.621548+0000) 2024-03-04T13:23:09.616 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:23:09.620+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:23:09.621775+0000) 2024-03-04T13:23:10.616 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:23:10.621+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:23:10.621999+0000) 2024-03-04T13:23:11.616 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:23:11.621+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:23:11.622239+0000) 2024-03-04T13:23:12.619 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:23:12.621+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:23:12.622481+0000) 2024-03-04T13:23:13.616 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:23:13.622+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:23:13.622721+0000) 2024-03-04T13:23:14.617 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:23:14.622+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:23:14.622960+0000) 2024-03-04T13:23:15.617 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:23:15.622+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:23:15.623192+0000) 2024-03-04T13:23:16.617 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:23:16.622+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:23:16.623441+0000) 2024-03-04T13:23:17.617 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:23:17.622+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:23:17.623693+0000) 2024-03-04T13:23:18.618 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:23:18.623+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:23:18.623948+0000) 2024-03-04T13:23:19.618 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:23:19.623+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:23:19.624178+0000) 2024-03-04T13:23:20.618 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:23:20.623+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:23:20.624433+0000) 2024-03-04T13:23:21.618 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:23:21.623+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:23:21.624637+0000) 2024-03-04T13:23:22.619 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:23:22.624+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:23:22.624923+0000) 2024-03-04T13:23:23.619 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:23:23.624+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:23:23.625121+0000) 2024-03-04T13:23:24.619 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:23:24.624+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:23:24.625381+0000) 2024-03-04T13:23:25.619 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:23:25.625+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:23:25.625617+0000) 2024-03-04T13:23:26.619 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:23:26.625+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:23:26.625916+0000) 2024-03-04T13:23:27.620 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:23:27.625+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:23:27.626155+0000) 2024-03-04T13:23:28.620 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:23:28.625+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:23:28.626436+0000) 2024-03-04T13:23:29.620 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:23:29.625+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:23:29.626739+0000) 2024-03-04T13:23:30.620 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:23:30.626+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:23:30.626986+0000) 2024-03-04T13:23:31.621 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:23:31.626+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:23:31.627247+0000) 2024-03-04T13:23:32.621 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:23:32.626+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:23:32.627433+0000) 2024-03-04T13:23:33.621 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:23:33.626+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:23:33.627696+0000) 2024-03-04T13:23:34.621 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:23:34.627+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:23:34.627935+0000) 2024-03-04T13:23:35.622 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:23:35.627+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:23:35.628150+0000) 2024-03-04T13:23:36.238 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:23:36.246 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:23:36.622 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:23:36.627+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:23:36.628385+0000) 2024-03-04T13:23:37.622 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:23:37.628+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:23:37.628664+0000) 2024-03-04T13:23:38.624 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:23:38.628+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:23:38.628976+0000) 2024-03-04T13:23:39.623 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:23:39.628+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:23:39.629222+0000) 2024-03-04T13:23:40.623 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:23:40.629+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:23:40.629458+0000) 2024-03-04T13:23:41.623 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:23:41.629+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:23:41.629693+0000) 2024-03-04T13:23:42.623 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:23:42.629+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:23:42.629975+0000) 2024-03-04T13:23:43.624 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:23:43.629+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:23:43.630156+0000) 2024-03-04T13:23:44.624 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:23:44.629+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:23:44.630413+0000) 2024-03-04T13:23:45.624 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:23:45.630+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:23:45.630646+0000) 2024-03-04T13:23:46.625 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:23:46.630+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:23:46.630919+0000) 2024-03-04T13:23:47.625 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:23:47.630+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:23:47.631153+0000) 2024-03-04T13:23:48.625 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:23:48.630+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:23:48.631348+0000) 2024-03-04T13:23:49.625 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:23:49.631+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:23:49.631581+0000) 2024-03-04T13:23:50.625 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:23:50.631+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:23:50.631855+0000) 2024-03-04T13:23:51.626 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:23:51.630+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:23:51.632044+0000) 2024-03-04T13:23:52.626 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:23:52.631+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:23:52.632282+0000) 2024-03-04T13:23:53.626 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:23:53.631+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:23:53.632517+0000) 2024-03-04T13:23:54.627 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:23:54.631+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:23:54.632751+0000) 2024-03-04T13:23:55.626 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:23:55.631+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:23:55.632990+0000) 2024-03-04T13:23:56.627 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:23:56.632+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:23:56.633227+0000) 2024-03-04T13:23:57.627 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:23:57.632+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:23:57.633466+0000) 2024-03-04T13:23:58.627 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:23:58.632+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:23:58.633710+0000) 2024-03-04T13:23:59.627 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:23:59.632+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:23:59.633946+0000) 2024-03-04T13:24:00.628 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:24:00.633+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:24:00.634175+0000) 2024-03-04T13:24:01.628 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:24:01.633+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:24:01.634416+0000) 2024-03-04T13:24:02.628 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:24:02.633+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:24:02.634684+0000) 2024-03-04T13:24:03.628 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:24:03.634+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:24:03.634922+0000) 2024-03-04T13:24:04.629 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:24:04.634+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:24:04.635161+0000) 2024-03-04T13:24:05.629 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:24:05.634+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:24:05.635400+0000) 2024-03-04T13:24:06.377 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:24:06.385 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:24:06.630 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:24:06.634+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:24:06.635594+0000) 2024-03-04T13:24:07.629 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:24:07.635+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:24:07.635856+0000) 2024-03-04T13:24:08.629 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:24:08.635+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:24:08.636094+0000) 2024-03-04T13:24:09.630 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:24:09.635+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:24:09.636351+0000) 2024-03-04T13:24:10.630 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:24:10.635+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:24:10.636592+0000) 2024-03-04T13:24:11.630 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:24:11.636+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:24:11.636854+0000) 2024-03-04T13:24:12.631 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:24:12.636+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:24:12.637101+0000) 2024-03-04T13:24:13.631 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:24:13.636+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:24:13.637343+0000) 2024-03-04T13:24:14.631 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:24:14.636+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:24:14.637575+0000) 2024-03-04T13:24:15.631 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:24:15.637+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:24:15.637806+0000) 2024-03-04T13:24:16.631 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:24:16.637+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:24:16.638085+0000) 2024-03-04T13:24:17.632 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:24:17.637+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:24:17.638322+0000) 2024-03-04T13:24:18.636 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:24:18.637+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:24:18.638569+0000) 2024-03-04T13:24:19.632 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:24:19.638+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:24:19.638801+0000) 2024-03-04T13:24:20.632 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:24:20.638+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:24:20.639074+0000) 2024-03-04T13:24:21.633 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:24:21.638+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:24:21.639311+0000) 2024-03-04T13:24:22.633 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:24:22.638+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:24:22.639532+0000) 2024-03-04T13:24:23.633 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:24:23.639+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:24:23.639768+0000) 2024-03-04T13:24:24.633 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:24:24.639+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:24:24.640006+0000) 2024-03-04T13:24:25.634 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:24:25.639+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:24:25.640243+0000) 2024-03-04T13:24:26.634 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:24:26.639+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:24:26.640486+0000) 2024-03-04T13:24:27.634 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:24:27.640+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:24:27.640773+0000) 2024-03-04T13:24:28.634 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:24:28.640+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:24:28.641023+0000) 2024-03-04T13:24:29.634 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:24:29.640+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:24:29.641257+0000) 2024-03-04T13:24:30.635 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:24:30.640+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:24:30.641490+0000) 2024-03-04T13:24:31.635 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:24:31.641+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:24:31.641722+0000) 2024-03-04T13:24:32.635 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:24:32.641+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:24:32.641954+0000) 2024-03-04T13:24:33.635 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:24:33.641+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:24:33.642148+0000) 2024-03-04T13:24:34.636 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:24:34.641+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:24:34.642401+0000) 2024-03-04T13:24:35.636 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:24:35.641+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:24:35.642650+0000) 2024-03-04T13:24:36.581 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:24:36.589 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:24:36.636 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:24:36.642+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:24:36.642916+0000) 2024-03-04T13:24:37.636 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:24:37.642+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:24:37.643149+0000) 2024-03-04T13:24:38.637 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:24:38.642+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:24:38.643400+0000) 2024-03-04T13:24:39.637 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:24:39.643+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:24:39.643658+0000) 2024-03-04T13:24:40.637 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:24:40.643+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:24:40.643929+0000) 2024-03-04T13:24:41.637 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:24:41.643+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:24:41.644202+0000) 2024-03-04T13:24:42.638 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:24:42.643+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:24:42.644402+0000) 2024-03-04T13:24:43.638 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:24:43.644+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:24:43.644643+0000) 2024-03-04T13:24:44.638 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:24:44.644+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:24:44.644941+0000) 2024-03-04T13:24:45.638 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:24:45.644+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:24:45.645176+0000) 2024-03-04T13:24:46.639 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:24:46.645+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:24:46.645418+0000) 2024-03-04T13:24:47.639 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:24:47.645+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:24:47.645649+0000) 2024-03-04T13:24:48.639 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:24:48.645+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:24:48.645932+0000) 2024-03-04T13:24:49.639 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:24:49.645+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:24:49.646169+0000) 2024-03-04T13:24:50.639 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:24:50.645+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:24:50.646354+0000) 2024-03-04T13:24:51.640 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:24:51.646+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:24:51.646606+0000) 2024-03-04T13:24:52.640 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:24:52.646+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:24:52.646880+0000) 2024-03-04T13:24:53.640 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:24:53.646+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:24:53.647114+0000) 2024-03-04T13:24:54.641 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:24:54.647+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:24:54.647348+0000) 2024-03-04T13:24:55.641 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:24:55.647+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:24:55.647583+0000) 2024-03-04T13:24:56.641 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:24:56.647+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:24:56.647856+0000) 2024-03-04T13:24:57.641 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:24:57.646+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:24:57.648094+0000) 2024-03-04T13:24:58.641 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:24:58.647+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:24:58.648344+0000) 2024-03-04T13:24:59.642 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:24:59.646+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:24:59.648594+0000) 2024-03-04T13:25:00.642 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:25:00.647+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:25:00.648859+0000) 2024-03-04T13:25:01.642 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:25:01.648+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:25:01.649099+0000) 2024-03-04T13:25:02.642 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:25:02.648+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:25:02.649338+0000) 2024-03-04T13:25:03.643 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:25:03.647+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:25:03.649521+0000) 2024-03-04T13:25:04.643 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:25:04.648+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:25:04.649759+0000) 2024-03-04T13:25:05.643 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:25:05.648+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:25:05.649988+0000) 2024-03-04T13:25:06.643 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:25:06.649+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:25:06.650269+0000) 2024-03-04T13:25:06.719 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:25:06.728 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:25:07.644 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:25:07.648+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:25:07.650467+0000) 2024-03-04T13:25:08.644 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:25:08.649+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:25:08.650786+0000) 2024-03-04T13:25:09.644 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:25:09.650+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:25:09.650945+0000) 2024-03-04T13:25:10.644 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:25:10.650+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:25:10.651200+0000) 2024-03-04T13:25:11.644 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:25:11.650+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:25:11.651447+0000) 2024-03-04T13:25:12.645 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:25:12.650+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:25:12.651687+0000) 2024-03-04T13:25:13.645 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:25:13.651+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:25:13.651939+0000) 2024-03-04T13:25:14.645 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:25:14.651+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:25:14.652174+0000) 2024-03-04T13:25:15.645 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:25:15.651+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:25:15.652403+0000) 2024-03-04T13:25:16.646 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:25:16.651+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:25:16.652641+0000) 2024-03-04T13:25:17.646 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:25:17.652+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:25:17.652896+0000) 2024-03-04T13:25:18.646 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:25:18.652+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:25:18.653154+0000) 2024-03-04T13:25:19.646 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:25:19.652+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:25:19.653384+0000) 2024-03-04T13:25:20.647 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:25:20.652+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:25:20.653623+0000) 2024-03-04T13:25:21.647 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:25:21.653+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:25:21.653942+0000) 2024-03-04T13:25:22.647 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:25:22.653+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:25:22.654171+0000) 2024-03-04T13:25:23.648 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:25:23.653+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:25:23.654403+0000) 2024-03-04T13:25:24.648 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:25:24.653+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:25:24.654640+0000) 2024-03-04T13:25:25.648 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:25:25.654+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:25:25.654928+0000) 2024-03-04T13:25:26.648 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:25:26.654+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:25:26.655162+0000) 2024-03-04T13:25:27.648 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:25:27.654+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:25:27.655416+0000) 2024-03-04T13:25:28.649 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:25:28.654+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:25:28.655676+0000) 2024-03-04T13:25:29.649 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:25:29.655+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:25:29.655950+0000) 2024-03-04T13:25:30.649 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:25:30.655+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:25:30.656191+0000) 2024-03-04T13:25:31.649 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:25:31.655+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:25:31.656382+0000) 2024-03-04T13:25:32.650 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:25:32.655+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:25:32.656564+0000) 2024-03-04T13:25:33.650 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:25:33.656+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:25:33.656814+0000) 2024-03-04T13:25:34.650 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:25:34.656+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:25:34.657087+0000) 2024-03-04T13:25:35.650 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:25:35.656+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:25:35.657319+0000) 2024-03-04T13:25:36.650 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:25:36.657+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:25:36.657561+0000) 2024-03-04T13:25:36.949 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:25:36.957 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:25:37.651 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:25:37.657+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:25:37.657799+0000) 2024-03-04T13:25:38.651 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:25:38.657+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:25:38.658091+0000) 2024-03-04T13:25:39.651 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:25:39.657+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:25:39.658270+0000) 2024-03-04T13:25:40.651 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:25:40.657+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:25:40.658506+0000) 2024-03-04T13:25:41.652 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:25:41.657+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:25:41.658765+0000) 2024-03-04T13:25:42.652 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:25:42.658+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:25:42.658988+0000) 2024-03-04T13:25:43.652 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:25:43.658+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:25:43.659220+0000) 2024-03-04T13:25:44.652 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:25:44.658+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:25:44.659439+0000) 2024-03-04T13:25:45.653 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:25:45.658+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:25:45.659656+0000) 2024-03-04T13:25:46.653 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:25:46.659+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:25:46.659929+0000) 2024-03-04T13:25:47.653 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:25:47.659+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:25:47.660160+0000) 2024-03-04T13:25:48.653 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:25:48.660+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:25:48.660401+0000) 2024-03-04T13:25:49.654 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:25:49.660+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:25:49.660656+0000) 2024-03-04T13:25:50.654 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:25:50.660+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:25:50.660929+0000) 2024-03-04T13:25:51.654 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:25:51.660+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:25:51.661195+0000) 2024-03-04T13:25:52.654 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:25:52.661+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:25:52.661454+0000) 2024-03-04T13:25:53.655 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:25:53.661+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:25:53.661692+0000) 2024-03-04T13:25:54.655 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:25:54.661+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:25:54.661931+0000) 2024-03-04T13:25:55.655 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:25:55.661+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:25:55.662161+0000) 2024-03-04T13:25:56.655 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:25:56.662+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:25:56.662407+0000) 2024-03-04T13:25:57.656 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:25:57.661+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:25:57.662658+0000) 2024-03-04T13:25:58.656 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:25:58.662+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:25:58.662928+0000) 2024-03-04T13:25:59.660 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:25:59.663+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:25:59.663180+0000) 2024-03-04T13:26:00.658 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:26:00.663+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:26:00.663414+0000) 2024-03-04T13:26:01.657 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:26:01.663+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:26:01.663650+0000) 2024-03-04T13:26:02.657 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:26:02.662+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:26:02.663932+0000) 2024-03-04T13:26:03.657 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:26:03.663+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:26:03.664167+0000) 2024-03-04T13:26:04.657 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:26:04.663+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:26:04.664408+0000) 2024-03-04T13:26:05.658 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:26:05.663+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:26:05.664640+0000) 2024-03-04T13:26:06.658 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:26:06.663+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:26:06.664922+0000) 2024-03-04T13:26:07.104 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:26:07.112 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:26:07.658 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:26:07.664+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:26:07.665155+0000) 2024-03-04T13:26:08.658 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:26:08.664+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:26:08.665418+0000) 2024-03-04T13:26:09.659 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:26:09.664+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:26:09.665650+0000) 2024-03-04T13:26:10.659 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:26:10.664+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:26:10.665926+0000) 2024-03-04T13:26:11.659 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:26:11.665+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:26:11.666185+0000) 2024-03-04T13:26:12.659 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:26:12.665+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:26:12.666446+0000) 2024-03-04T13:26:13.659 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:26:13.665+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:26:13.666629+0000) 2024-03-04T13:26:14.660 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:26:14.666+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:26:14.666907+0000) 2024-03-04T13:26:15.660 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:26:15.666+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:26:15.667140+0000) 2024-03-04T13:26:16.660 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:26:16.666+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:26:16.667381+0000) 2024-03-04T13:26:17.660 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:26:17.666+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:26:17.667627+0000) 2024-03-04T13:26:18.661 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:26:18.667+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:26:18.667913+0000) 2024-03-04T13:26:19.661 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:26:19.667+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:26:19.668202+0000) 2024-03-04T13:26:20.665 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:26:20.667+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:26:20.668444+0000) 2024-03-04T13:26:21.662 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:26:21.667+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:26:21.668683+0000) 2024-03-04T13:26:22.662 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:26:22.668+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:26:22.668955+0000) 2024-03-04T13:26:23.662 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:26:23.668+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:26:23.669192+0000) 2024-03-04T13:26:24.662 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:26:24.668+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:26:24.669452+0000) 2024-03-04T13:26:25.664 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:26:25.668+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:26:25.669691+0000) 2024-03-04T13:26:26.663 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:26:26.669+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:26:26.669939+0000) 2024-03-04T13:26:27.663 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:26:27.669+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:26:27.670186+0000) 2024-03-04T13:26:28.663 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:26:28.669+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:26:28.670428+0000) 2024-03-04T13:26:29.663 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:26:29.669+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:26:29.670682+0000) 2024-03-04T13:26:30.664 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:26:30.670+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:26:30.670927+0000) 2024-03-04T13:26:31.664 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:26:31.670+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:26:31.671107+0000) 2024-03-04T13:26:32.664 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:26:32.670+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:26:32.671348+0000) 2024-03-04T13:26:33.664 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:26:33.670+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:26:33.671589+0000) 2024-03-04T13:26:34.668 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:26:34.671+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:26:34.671858+0000) 2024-03-04T13:26:35.665 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:26:35.671+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:26:35.672104+0000) 2024-03-04T13:26:36.665 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:26:36.671+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:26:36.672345+0000) 2024-03-04T13:26:37.318 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:26:37.327 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:26:37.665 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:26:37.672+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:26:37.672594+0000) 2024-03-04T13:26:38.666 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:26:38.672+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:26:38.672863+0000) 2024-03-04T13:26:39.666 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:26:39.672+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:26:39.673096+0000) 2024-03-04T13:26:40.666 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:26:40.672+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:26:40.673332+0000) 2024-03-04T13:26:41.667 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:26:41.672+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:26:41.673576+0000) 2024-03-04T13:26:42.667 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:26:42.673+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:26:42.673865+0000) 2024-03-04T13:26:43.667 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:26:43.673+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:26:43.674081+0000) 2024-03-04T13:26:44.667 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:26:44.673+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:26:44.674329+0000) 2024-03-04T13:26:45.667 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:26:45.674+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:26:45.674568+0000) 2024-03-04T13:26:46.668 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:26:46.674+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:26:46.674809+0000) 2024-03-04T13:26:47.668 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:26:47.674+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:26:47.675080+0000) 2024-03-04T13:26:48.668 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:26:48.674+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:26:48.675339+0000) 2024-03-04T13:26:49.668 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:26:49.675+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:26:49.675576+0000) 2024-03-04T13:26:50.668 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:26:50.675+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:26:50.675814+0000) 2024-03-04T13:26:51.669 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:26:51.675+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:26:51.676086+0000) 2024-03-04T13:26:52.669 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:26:52.675+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:26:52.676328+0000) 2024-03-04T13:26:53.669 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:26:53.676+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:26:53.676564+0000) 2024-03-04T13:26:54.670 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:26:54.676+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:26:54.676798+0000) 2024-03-04T13:26:55.670 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:26:55.676+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:26:55.677070+0000) 2024-03-04T13:26:56.670 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:26:56.677+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:26:56.677309+0000) 2024-03-04T13:26:57.670 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:26:57.677+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:26:57.677549+0000) 2024-03-04T13:26:58.670 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:26:58.677+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:26:58.677800+0000) 2024-03-04T13:26:59.671 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:26:59.677+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:26:59.678061+0000) 2024-03-04T13:27:00.671 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:27:00.678+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:27:00.678302+0000) 2024-03-04T13:27:01.671 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:27:01.678+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:27:01.678540+0000) 2024-03-04T13:27:02.672 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:27:02.678+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:27:02.678784+0000) 2024-03-04T13:27:03.672 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:27:03.678+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:27:03.679021+0000) 2024-03-04T13:27:04.672 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:27:04.679+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:27:04.679264+0000) 2024-03-04T13:27:05.672 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:27:05.679+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:27:05.679501+0000) 2024-03-04T13:27:06.672 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:27:06.679+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:27:06.679741+0000) 2024-03-04T13:27:07.556 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:27:07.564 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:27:07.711 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:27:07.678+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:27:07.680040+0000) 2024-03-04T13:27:08.673 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:27:08.678+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:27:08.680254+0000) 2024-03-04T13:27:09.673 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:27:09.678+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:27:09.680445+0000) 2024-03-04T13:27:10.674 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:27:10.679+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:27:10.680702+0000) 2024-03-04T13:27:11.674 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:27:11.679+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:27:11.680955+0000) 2024-03-04T13:27:12.674 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:27:12.679+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:27:12.681204+0000) 2024-03-04T13:27:13.674 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:27:13.680+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:27:13.681489+0000) 2024-03-04T13:27:14.674 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:27:14.680+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:27:14.681764+0000) 2024-03-04T13:27:15.675 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:27:15.681+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:27:15.682010+0000) 2024-03-04T13:27:16.675 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:27:16.681+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:27:16.682272+0000) 2024-03-04T13:27:17.686 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:27:17.681+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:27:17.682528+0000) 2024-03-04T13:27:18.675 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:27:18.681+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:27:18.682797+0000) 2024-03-04T13:27:19.676 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:27:19.682+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:27:19.683081+0000) 2024-03-04T13:27:20.676 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:27:20.682+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:27:20.683333+0000) 2024-03-04T13:27:21.676 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:27:21.682+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:27:21.683601+0000) 2024-03-04T13:27:22.677 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:27:22.683+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:27:22.683913+0000) 2024-03-04T13:27:23.677 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:27:23.683+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:27:23.684160+0000) 2024-03-04T13:27:24.677 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:27:24.683+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:27:24.684412+0000) 2024-03-04T13:27:25.677 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:27:25.683+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:27:25.684657+0000) 2024-03-04T13:27:26.678 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:27:26.684+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:27:26.684949+0000) 2024-03-04T13:27:27.678 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:27:27.684+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:27:27.685187+0000) 2024-03-04T13:27:28.678 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:27:28.684+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:27:28.685438+0000) 2024-03-04T13:27:29.678 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:27:29.684+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:27:29.685672+0000) 2024-03-04T13:27:30.679 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:27:30.685+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:27:30.685950+0000) 2024-03-04T13:27:31.679 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:27:31.685+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:27:31.686187+0000) 2024-03-04T13:27:32.679 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:27:32.685+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:27:32.686436+0000) 2024-03-04T13:27:33.679 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:27:33.685+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:27:33.686672+0000) 2024-03-04T13:27:34.680 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:27:34.686+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:27:34.686955+0000) 2024-03-04T13:27:35.680 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:27:35.686+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:27:35.687188+0000) 2024-03-04T13:27:36.680 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:27:36.686+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:27:36.687459+0000) 2024-03-04T13:27:37.680 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:27:37.686+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:27:37.687708+0000) 2024-03-04T13:27:37.780 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:27:37.797 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:27:38.681 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:27:38.687+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:27:38.687974+0000) 2024-03-04T13:27:39.681 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:27:39.687+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:27:39.688229+0000) 2024-03-04T13:27:40.681 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:27:40.687+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:27:40.688483+0000) 2024-03-04T13:27:41.681 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:27:41.688+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:27:41.688733+0000) 2024-03-04T13:27:42.682 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:27:42.688+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:27:42.688987+0000) 2024-03-04T13:27:43.682 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:27:43.688+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:27:43.689269+0000) 2024-03-04T13:27:44.682 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:27:44.688+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:27:44.689521+0000) 2024-03-04T13:27:45.682 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:27:45.689+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:27:45.689776+0000) 2024-03-04T13:27:46.683 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:27:46.689+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:27:46.690014+0000) 2024-03-04T13:27:47.684 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:27:47.689+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:27:47.690274+0000) 2024-03-04T13:27:48.683 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:27:48.690+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:27:48.690530+0000) 2024-03-04T13:27:49.683 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:27:49.690+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:27:49.690781+0000) 2024-03-04T13:27:50.684 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:27:50.690+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:27:50.691066+0000) 2024-03-04T13:27:51.684 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:27:51.690+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:27:51.691280+0000) 2024-03-04T13:27:52.684 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:27:52.690+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:27:52.691489+0000) 2024-03-04T13:27:53.684 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:27:53.690+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:27:53.691764+0000) 2024-03-04T13:27:54.685 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:27:54.691+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:27:54.692030+0000) 2024-03-04T13:27:55.685 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:27:55.691+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:27:55.692318+0000) 2024-03-04T13:27:56.685 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:27:56.692+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:27:56.692553+0000) 2024-03-04T13:27:57.685 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:27:57.692+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:27:57.692788+0000) 2024-03-04T13:27:58.686 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:27:58.692+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:27:58.693074+0000) 2024-03-04T13:27:59.686 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:27:59.693+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:27:59.693331+0000) 2024-03-04T13:28:00.686 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:28:00.693+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:28:00.693572+0000) 2024-03-04T13:28:01.686 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:28:01.693+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:28:01.693812+0000) 2024-03-04T13:28:02.686 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:28:02.693+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:28:02.694024+0000) 2024-03-04T13:28:03.687 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:28:03.694+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:28:03.694293+0000) 2024-03-04T13:28:04.687 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:28:04.693+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:28:04.694550+0000) 2024-03-04T13:28:05.687 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:28:05.694+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:28:05.694772+0000) 2024-03-04T13:28:06.687 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:28:06.694+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:28:06.695009+0000) 2024-03-04T13:28:07.688 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:28:07.695+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:28:07.695245+0000) 2024-03-04T13:28:07.941 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:28:07.949 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:28:08.688 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:28:08.695+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:28:08.695491+0000) 2024-03-04T13:28:09.688 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:28:09.695+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:28:09.695721+0000) 2024-03-04T13:28:10.688 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:28:10.695+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:28:10.695958+0000) 2024-03-04T13:28:11.689 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:28:11.696+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:28:11.696195+0000) 2024-03-04T13:28:12.689 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:28:12.695+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:28:12.696431+0000) 2024-03-04T13:28:13.689 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:28:13.695+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:28:13.696685+0000) 2024-03-04T13:28:14.689 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:28:14.695+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:28:14.696953+0000) 2024-03-04T13:28:15.690 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:28:15.696+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:28:15.697185+0000) 2024-03-04T13:28:16.690 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:28:16.696+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:28:16.697421+0000) 2024-03-04T13:28:17.690 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:28:17.696+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:28:17.697682+0000) 2024-03-04T13:28:18.692 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:28:18.696+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:28:18.697953+0000) 2024-03-04T13:28:19.691 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:28:19.697+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:28:19.698189+0000) 2024-03-04T13:28:20.691 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:28:20.697+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:28:20.698445+0000) 2024-03-04T13:28:21.691 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:28:21.696+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:28:21.698697+0000) 2024-03-04T13:28:22.691 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:28:22.697+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:28:22.698955+0000) 2024-03-04T13:28:23.692 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:28:23.698+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:28:23.699221+0000) 2024-03-04T13:28:24.692 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:28:24.698+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:28:24.699473+0000) 2024-03-04T13:28:25.692 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:28:25.698+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:28:25.699723+0000) 2024-03-04T13:28:26.692 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:28:26.699+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:28:26.699954+0000) 2024-03-04T13:28:27.693 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:28:27.699+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:28:27.700184+0000) 2024-03-04T13:28:28.693 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:28:28.699+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:28:28.700421+0000) 2024-03-04T13:28:29.693 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:28:29.699+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:28:29.700658+0000) 2024-03-04T13:28:30.693 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:28:30.700+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:28:30.700926+0000) 2024-03-04T13:28:31.694 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:28:31.700+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:28:31.701156+0000) 2024-03-04T13:28:32.694 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:28:32.700+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:28:32.701395+0000) 2024-03-04T13:28:33.694 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:28:33.700+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:28:33.701639+0000) 2024-03-04T13:28:34.694 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:28:34.701+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:28:34.701927+0000) 2024-03-04T13:28:35.694 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:28:35.701+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:28:35.702106+0000) 2024-03-04T13:28:36.695 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:28:36.701+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:28:36.702349+0000) 2024-03-04T13:28:37.695 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:28:37.701+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:28:37.702614+0000) 2024-03-04T13:28:38.095 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:28:38.103 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:28:38.695 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:28:38.702+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:28:38.702918+0000) 2024-03-04T13:28:39.695 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:28:39.702+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:28:39.703153+0000) 2024-03-04T13:28:40.696 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:28:40.702+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:28:40.703393+0000) 2024-03-04T13:28:41.696 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:28:41.702+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:28:41.703630+0000) 2024-03-04T13:28:42.696 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:28:42.703+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:28:42.703926+0000) 2024-03-04T13:28:43.697 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:28:43.703+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:28:43.704179+0000) 2024-03-04T13:28:44.697 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:28:44.703+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:28:44.704432+0000) 2024-03-04T13:28:45.697 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:28:45.704+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:28:45.704669+0000) 2024-03-04T13:28:46.697 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:28:46.704+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:28:46.704940+0000) 2024-03-04T13:28:47.697 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:28:47.704+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:28:47.705171+0000) 2024-03-04T13:28:48.698 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:28:48.704+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:28:48.705418+0000) 2024-03-04T13:28:49.698 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:28:49.704+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:28:49.705654+0000) 2024-03-04T13:28:50.699 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:28:50.705+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:28:50.705876+0000) 2024-03-04T13:28:51.698 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:28:51.705+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:28:51.706143+0000) 2024-03-04T13:28:52.699 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:28:52.705+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:28:52.706390+0000) 2024-03-04T13:28:53.699 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:28:53.706+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:28:53.706629+0000) 2024-03-04T13:28:54.699 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:28:54.706+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:28:54.706909+0000) 2024-03-04T13:28:55.699 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:28:55.706+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:28:55.707142+0000) 2024-03-04T13:28:56.700 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:28:56.706+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:28:56.707386+0000) 2024-03-04T13:28:57.700 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:28:57.707+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:28:57.707626+0000) 2024-03-04T13:28:58.700 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:28:58.707+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:28:58.707927+0000) 2024-03-04T13:28:59.700 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:28:59.707+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:28:59.708167+0000) 2024-03-04T13:29:00.701 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:29:00.707+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:29:00.708404+0000) 2024-03-04T13:29:01.701 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:29:01.708+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:29:01.708637+0000) 2024-03-04T13:29:02.701 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:29:02.708+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:29:02.708947+0000) 2024-03-04T13:29:03.701 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:29:03.708+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:29:03.709197+0000) 2024-03-04T13:29:04.702 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:29:04.709+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:29:04.709434+0000) 2024-03-04T13:29:05.702 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:29:05.709+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:29:05.709670+0000) 2024-03-04T13:29:06.702 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:29:06.709+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:29:06.709939+0000) 2024-03-04T13:29:07.702 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:29:07.709+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:29:07.710171+0000) 2024-03-04T13:29:08.292 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:29:08.300 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:29:08.703 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:29:08.709+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:29:08.710434+0000) 2024-03-04T13:29:09.703 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:29:09.710+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:29:09.710668+0000) 2024-03-04T13:29:10.703 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:29:10.710+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:29:10.710941+0000) 2024-03-04T13:29:11.703 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:29:11.710+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:29:11.711173+0000) 2024-03-04T13:29:12.704 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:29:12.711+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:29:12.711417+0000) 2024-03-04T13:29:13.704 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:29:13.711+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:29:13.711674+0000) 2024-03-04T13:29:14.704 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:29:14.711+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:29:14.711933+0000) 2024-03-04T13:29:15.705 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:29:15.712+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:29:15.712170+0000) 2024-03-04T13:29:16.705 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:29:16.712+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:29:16.712411+0000) 2024-03-04T13:29:17.705 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:29:17.711+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:29:17.712651+0000) 2024-03-04T13:29:18.705 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:29:18.711+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:29:18.712956+0000) 2024-03-04T13:29:19.705 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:29:19.712+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:29:19.713187+0000) 2024-03-04T13:29:20.706 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:29:20.712+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:29:20.713428+0000) 2024-03-04T13:29:21.706 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:29:21.712+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:29:21.713682+0000) 2024-03-04T13:29:22.706 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:29:22.712+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:29:22.713955+0000) 2024-03-04T13:29:23.707 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:29:23.713+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:29:23.714195+0000) 2024-03-04T13:29:24.707 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:29:24.713+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:29:24.714432+0000) 2024-03-04T13:29:25.708 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:29:25.713+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:29:25.714672+0000) 2024-03-04T13:29:26.707 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:29:26.713+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:29:26.714944+0000) 2024-03-04T13:29:27.707 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:29:27.714+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:29:27.715178+0000) 2024-03-04T13:29:28.708 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:29:28.714+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:29:28.715425+0000) 2024-03-04T13:29:29.708 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:29:29.714+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:29:29.715663+0000) 2024-03-04T13:29:30.708 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:29:30.715+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:29:30.715939+0000) 2024-03-04T13:29:31.708 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:29:31.715+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:29:31.716174+0000) 2024-03-04T13:29:32.708 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:29:32.715+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:29:32.716358+0000) 2024-03-04T13:29:33.709 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:29:33.715+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:29:33.716595+0000) 2024-03-04T13:29:34.709 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:29:34.716+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:29:34.716858+0000) 2024-03-04T13:29:35.709 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:29:35.716+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:29:35.717050+0000) 2024-03-04T13:29:36.709 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:29:36.716+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:29:36.717288+0000) 2024-03-04T13:29:37.710 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:29:37.716+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:29:37.717527+0000) 2024-03-04T13:29:38.520 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:29:38.528 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:29:38.710 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:29:38.716+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:29:38.717788+0000) 2024-03-04T13:29:39.710 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:29:39.717+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:29:39.718035+0000) 2024-03-04T13:29:40.710 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:29:40.717+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:29:40.718278+0000) 2024-03-04T13:29:41.711 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:29:41.717+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:29:41.718514+0000) 2024-03-04T13:29:42.711 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:29:42.718+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:29:42.718761+0000) 2024-03-04T13:29:43.711 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:29:43.718+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:29:43.718982+0000) 2024-03-04T13:29:44.711 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:29:44.718+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:29:44.719176+0000) 2024-03-04T13:29:45.711 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:29:45.718+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:29:45.719371+0000) 2024-03-04T13:29:46.712 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:29:46.718+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:29:46.719609+0000) 2024-03-04T13:29:47.712 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:29:47.719+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:29:47.719984+0000) 2024-03-04T13:29:48.712 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:29:48.719+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:29:48.720202+0000) 2024-03-04T13:29:49.713 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:29:49.719+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:29:49.720439+0000) 2024-03-04T13:29:50.713 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:29:50.720+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:29:50.720678+0000) 2024-03-04T13:29:51.713 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:29:51.720+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:29:51.720936+0000) 2024-03-04T13:29:52.713 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:29:52.720+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:29:52.721175+0000) 2024-03-04T13:29:53.714 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:29:53.720+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:29:53.721421+0000) 2024-03-04T13:29:54.714 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:29:54.721+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:29:54.721668+0000) 2024-03-04T13:29:55.714 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:29:55.721+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:29:55.721920+0000) 2024-03-04T13:29:56.714 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:29:56.721+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:29:56.722162+0000) 2024-03-04T13:29:57.715 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:29:57.721+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:29:57.722401+0000) 2024-03-04T13:29:58.715 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:29:58.722+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:29:58.722650+0000) 2024-03-04T13:29:59.715 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:29:59.722+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:29:59.722921+0000) 2024-03-04T13:30:00.715 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:30:00.722+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:30:00.723158+0000) 2024-03-04T13:30:01.716 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:30:01.722+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:30:01.723389+0000) 2024-03-04T13:30:02.716 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:30:02.723+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:30:02.723644+0000) 2024-03-04T13:30:03.716 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:30:03.723+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:30:03.723923+0000) 2024-03-04T13:30:04.716 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:30:04.723+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:30:04.724160+0000) 2024-03-04T13:30:05.717 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:30:05.724+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:30:05.724393+0000) 2024-03-04T13:30:06.717 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:30:06.724+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:30:06.724630+0000) 2024-03-04T13:30:07.717 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:30:07.724+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:30:07.724906+0000) 2024-03-04T13:30:08.717 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:30:08.724+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:30:08.725190+0000) 2024-03-04T13:30:08.726 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:30:08.735 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:30:09.718 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:30:09.725+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:30:09.725426+0000) 2024-03-04T13:30:10.718 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:30:10.725+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:30:10.725667+0000) 2024-03-04T13:30:11.718 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:30:11.725+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:30:11.725940+0000) 2024-03-04T13:30:12.718 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:30:12.725+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:30:12.726187+0000) 2024-03-04T13:30:13.718 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:30:13.726+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:30:13.726425+0000) 2024-03-04T13:30:14.719 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:30:14.725+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:30:14.726608+0000) 2024-03-04T13:30:15.719 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:30:15.726+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:30:15.726903+0000) 2024-03-04T13:30:16.719 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:30:16.726+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:30:16.727175+0000) 2024-03-04T13:30:17.720 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:30:17.727+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:30:17.727413+0000) 2024-03-04T13:30:18.720 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:30:18.727+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:30:18.727669+0000) 2024-03-04T13:30:19.720 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:30:19.727+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:30:19.727955+0000) 2024-03-04T13:30:20.720 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:30:20.728+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:30:20.728188+0000) 2024-03-04T13:30:21.721 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:30:21.727+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:30:21.728422+0000) 2024-03-04T13:30:22.721 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:30:22.727+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:30:22.728678+0000) 2024-03-04T13:30:23.721 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:30:23.727+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:30:23.728945+0000) 2024-03-04T13:30:24.721 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:30:24.728+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:30:24.729179+0000) 2024-03-04T13:30:25.721 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:30:25.728+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:30:25.729425+0000) 2024-03-04T13:30:26.722 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:30:26.728+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:30:26.729674+0000) 2024-03-04T13:30:27.722 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:30:27.728+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:30:27.729929+0000) 2024-03-04T13:30:28.722 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:30:28.729+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:30:28.730215+0000) 2024-03-04T13:30:29.723 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:30:29.729+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:30:29.730455+0000) 2024-03-04T13:30:30.723 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:30:30.729+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:30:30.730698+0000) 2024-03-04T13:30:31.723 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:30:31.729+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:30:31.730932+0000) 2024-03-04T13:30:32.723 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:30:32.730+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:30:32.731190+0000) 2024-03-04T13:30:33.723 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:30:33.730+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:30:33.731424+0000) 2024-03-04T13:30:34.724 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:30:34.730+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:30:34.731662+0000) 2024-03-04T13:30:35.724 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:30:35.731+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:30:35.731980+0000) 2024-03-04T13:30:36.724 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:30:36.731+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:30:36.732221+0000) 2024-03-04T13:30:37.725 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:30:37.731+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:30:37.732453+0000) 2024-03-04T13:30:38.725 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:30:38.731+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:30:38.732700+0000) 2024-03-04T13:30:38.859 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:30:38.868 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:30:39.725 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:30:39.732+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:30:39.732939+0000) 2024-03-04T13:30:40.725 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:30:40.732+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:30:40.733189+0000) 2024-03-04T13:30:41.725 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:30:41.732+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:30:41.733468+0000) 2024-03-04T13:30:42.726 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:30:42.732+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:30:42.733725+0000) 2024-03-04T13:30:43.726 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:30:43.733+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:30:43.733958+0000) 2024-03-04T13:30:44.726 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:30:44.733+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:30:44.734212+0000) 2024-03-04T13:30:45.726 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:30:45.733+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:30:45.734463+0000) 2024-03-04T13:30:46.727 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:30:46.733+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:30:46.734727+0000) 2024-03-04T13:30:47.727 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:30:47.734+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:30:47.734961+0000) 2024-03-04T13:30:48.727 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:30:48.734+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:30:48.735222+0000) 2024-03-04T13:30:49.727 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:30:49.734+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:30:49.735457+0000) 2024-03-04T13:30:50.728 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:30:50.734+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:30:50.735712+0000) 2024-03-04T13:30:51.728 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:30:51.735+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:30:51.735945+0000) 2024-03-04T13:30:52.728 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:30:52.735+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:30:52.736193+0000) 2024-03-04T13:30:53.728 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:30:53.735+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:30:53.736428+0000) 2024-03-04T13:30:54.729 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:30:54.735+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:30:54.736655+0000) 2024-03-04T13:30:55.729 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:30:55.736+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:30:55.736928+0000) 2024-03-04T13:30:56.729 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:30:56.736+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:30:56.737157+0000) 2024-03-04T13:30:57.729 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:30:57.736+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:30:57.737407+0000) 2024-03-04T13:30:58.730 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:30:58.737+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:30:58.737663+0000) 2024-03-04T13:30:59.730 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:30:59.737+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:30:59.737929+0000) 2024-03-04T13:31:00.730 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:31:00.737+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:31:00.738166+0000) 2024-03-04T13:31:01.730 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:31:01.737+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:31:01.738404+0000) 2024-03-04T13:31:02.731 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:31:02.738+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:31:02.738646+0000) 2024-03-04T13:31:03.731 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:31:03.738+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:31:03.738934+0000) 2024-03-04T13:31:04.731 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:31:04.738+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:31:04.739168+0000) 2024-03-04T13:31:05.731 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:31:05.738+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:31:05.739403+0000) 2024-03-04T13:31:06.732 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:31:06.739+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:31:06.739643+0000) 2024-03-04T13:31:07.732 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:31:07.739+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:31:07.739931+0000) 2024-03-04T13:31:08.732 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:31:08.739+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:31:08.740141+0000) 2024-03-04T13:31:09.055 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:31:09.063 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:31:09.732 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:31:09.740+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:31:09.740377+0000) 2024-03-04T13:31:10.733 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:31:10.740+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:31:10.740611+0000) 2024-03-04T13:31:11.733 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:31:11.740+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:31:11.740878+0000) 2024-03-04T13:31:12.733 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:31:12.740+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:31:12.741130+0000) 2024-03-04T13:31:13.733 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:31:13.741+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:31:13.741375+0000) 2024-03-04T13:31:14.734 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:31:14.741+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:31:14.741620+0000) 2024-03-04T13:31:15.736 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:31:15.741+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:31:15.741925+0000) 2024-03-04T13:31:16.734 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:31:16.741+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:31:16.742182+0000) 2024-03-04T13:31:17.734 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:31:17.742+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:31:17.742423+0000) 2024-03-04T13:31:18.735 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:31:18.742+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:31:18.742679+0000) 2024-03-04T13:31:19.735 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:31:19.742+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:31:19.742952+0000) 2024-03-04T13:31:20.735 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:31:20.742+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:31:20.743187+0000) 2024-03-04T13:31:21.735 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:31:21.743+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:31:21.743432+0000) 2024-03-04T13:31:22.736 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:31:22.743+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:31:22.743677+0000) 2024-03-04T13:31:23.736 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:31:23.743+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:31:23.743933+0000) 2024-03-04T13:31:24.736 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:31:24.744+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:31:24.744169+0000) 2024-03-04T13:31:25.736 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:31:25.743+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:31:25.744358+0000) 2024-03-04T13:31:26.737 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:31:26.743+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:31:26.744590+0000) 2024-03-04T13:31:27.737 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:31:27.743+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:31:27.744865+0000) 2024-03-04T13:31:28.737 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:31:28.744+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:31:28.745122+0000) 2024-03-04T13:31:29.737 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:31:29.744+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:31:29.745365+0000) 2024-03-04T13:31:30.738 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:31:30.744+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:31:30.745604+0000) 2024-03-04T13:31:31.738 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:31:31.744+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:31:31.745865+0000) 2024-03-04T13:31:32.740 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:31:32.745+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:31:32.746109+0000) 2024-03-04T13:31:33.738 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:31:33.745+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:31:33.746352+0000) 2024-03-04T13:31:34.739 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:31:34.744+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:31:34.746636+0000) 2024-03-04T13:31:35.739 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:31:35.745+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:31:35.746886+0000) 2024-03-04T13:31:36.739 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:31:36.746+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:31:36.747125+0000) 2024-03-04T13:31:37.739 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:31:37.746+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:31:37.747323+0000) 2024-03-04T13:31:38.739 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:31:38.746+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:31:38.747581+0000) 2024-03-04T13:31:39.210 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:31:39.218 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:31:39.740 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:31:39.746+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:31:39.747775+0000) 2024-03-04T13:31:40.740 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:31:40.747+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:31:40.748008+0000) 2024-03-04T13:31:41.740 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:31:41.747+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:31:41.748245+0000) 2024-03-04T13:31:42.740 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:31:42.747+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:31:42.748480+0000) 2024-03-04T13:31:43.741 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:31:43.747+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:31:43.748721+0000) 2024-03-04T13:31:44.741 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:31:44.748+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:31:44.748955+0000) 2024-03-04T13:31:45.741 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:31:45.748+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:31:45.749143+0000) 2024-03-04T13:31:46.741 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:31:46.748+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:31:46.749393+0000) 2024-03-04T13:31:47.742 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:31:47.748+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:31:47.749655+0000) 2024-03-04T13:31:48.742 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:31:48.749+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:31:48.749941+0000) 2024-03-04T13:31:49.742 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:31:49.749+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:31:49.750186+0000) 2024-03-04T13:31:50.742 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:31:50.749+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:31:50.750426+0000) 2024-03-04T13:31:51.742 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:31:51.749+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:31:51.750659+0000) 2024-03-04T13:31:52.743 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:31:52.750+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:31:52.750879+0000) 2024-03-04T13:31:53.743 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:31:53.750+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:31:53.751125+0000) 2024-03-04T13:31:54.743 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:31:54.750+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:31:54.751380+0000) 2024-03-04T13:31:55.744 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:31:55.750+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:31:55.751638+0000) 2024-03-04T13:31:56.744 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:31:56.751+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:31:56.751917+0000) 2024-03-04T13:31:57.744 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:31:57.751+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:31:57.752193+0000) 2024-03-04T13:31:58.747 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:31:58.751+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:31:58.752465+0000) 2024-03-04T13:31:59.744 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:31:59.752+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:31:59.752696+0000) 2024-03-04T13:32:00.745 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:32:00.752+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:32:00.752936+0000) 2024-03-04T13:32:01.745 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:32:01.752+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:32:01.753174+0000) 2024-03-04T13:32:02.745 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:32:02.752+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:32:02.753421+0000) 2024-03-04T13:32:03.745 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:32:03.753+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:32:03.753664+0000) 2024-03-04T13:32:04.746 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:32:04.753+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:32:04.753937+0000) 2024-03-04T13:32:05.746 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:32:05.753+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:32:05.754171+0000) 2024-03-04T13:32:06.746 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:32:06.753+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:32:06.754353+0000) 2024-03-04T13:32:07.746 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:32:07.754+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:32:07.754595+0000) 2024-03-04T13:32:08.747 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:32:08.754+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:32:08.754891+0000) 2024-03-04T13:32:09.423 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:32:09.431 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:32:09.747 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:32:09.754+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:32:09.755142+0000) 2024-03-04T13:32:10.747 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:32:10.754+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:32:10.755394+0000) 2024-03-04T13:32:11.747 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:32:11.755+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:32:11.755630+0000) 2024-03-04T13:32:12.748 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:32:12.755+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:32:12.755933+0000) 2024-03-04T13:32:13.748 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:32:13.755+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:32:13.756197+0000) 2024-03-04T13:32:14.748 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:32:14.756+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:32:14.756434+0000) 2024-03-04T13:32:15.749 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:32:15.756+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:32:15.756676+0000) 2024-03-04T13:32:16.749 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:32:16.756+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:32:16.756941+0000) 2024-03-04T13:32:17.749 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:32:17.756+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:32:17.757205+0000) 2024-03-04T13:32:18.749 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:32:18.757+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:32:18.757465+0000) 2024-03-04T13:32:19.749 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:32:19.757+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:32:19.757703+0000) 2024-03-04T13:32:20.750 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:32:20.757+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:32:20.757941+0000) 2024-03-04T13:32:21.750 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:32:21.757+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:32:21.758234+0000) 2024-03-04T13:32:22.750 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:32:22.758+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:32:22.758594+0000) 2024-03-04T13:32:23.750 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:32:23.758+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:32:23.758781+0000) 2024-03-04T13:32:24.751 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:32:24.758+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:32:24.759013+0000) 2024-03-04T13:32:25.751 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:32:25.759+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:32:25.759250+0000) 2024-03-04T13:32:26.751 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:32:26.759+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:32:26.759489+0000) 2024-03-04T13:32:27.752 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:32:27.759+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:32:27.759801+0000) 2024-03-04T13:32:28.752 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:32:28.759+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:32:28.760099+0000) 2024-03-04T13:32:29.752 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:32:29.760+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:32:29.760337+0000) 2024-03-04T13:32:30.752 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:32:30.759+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:32:30.760577+0000) 2024-03-04T13:32:31.752 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:32:31.759+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:32:31.760813+0000) 2024-03-04T13:32:32.753 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:32:32.759+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:32:32.761012+0000) 2024-03-04T13:32:33.753 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:32:33.760+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:32:33.761257+0000) 2024-03-04T13:32:34.753 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:32:34.760+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:32:34.761499+0000) 2024-03-04T13:32:35.754 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:32:35.760+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:32:35.761730+0000) 2024-03-04T13:32:36.754 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:32:36.760+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:32:36.761972+0000) 2024-03-04T13:32:37.754 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:32:37.761+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:32:37.762203+0000) 2024-03-04T13:32:38.754 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:32:38.761+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:32:38.762461+0000) 2024-03-04T13:32:39.636 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:32:39.644 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:32:39.782 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:32:39.761+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:32:39.762727+0000) 2024-03-04T13:32:40.755 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:32:40.762+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:32:40.762967+0000) 2024-03-04T13:32:41.755 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:32:41.762+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:32:41.763201+0000) 2024-03-04T13:32:42.755 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:32:42.762+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:32:42.763444+0000) 2024-03-04T13:32:43.755 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:32:43.762+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:32:43.763692+0000) 2024-03-04T13:32:44.756 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:32:44.763+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:32:44.763931+0000) 2024-03-04T13:32:45.756 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:32:45.763+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:32:45.764166+0000) 2024-03-04T13:32:46.756 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:32:46.763+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:32:46.764403+0000) 2024-03-04T13:32:47.757 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:32:47.763+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:32:47.764651+0000) 2024-03-04T13:32:48.757 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:32:48.764+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:32:48.764957+0000) 2024-03-04T13:32:49.757 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:32:49.764+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:32:49.765147+0000) 2024-03-04T13:32:50.757 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:32:50.764+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:32:50.765387+0000) 2024-03-04T13:32:51.760 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:32:51.764+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:32:51.765560+0000) 2024-03-04T13:32:52.757 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:32:52.765+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:32:52.765795+0000) 2024-03-04T13:32:53.758 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:32:53.765+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:32:53.766065+0000) 2024-03-04T13:32:54.758 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:32:54.765+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:32:54.766303+0000) 2024-03-04T13:32:55.758 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:32:55.765+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:32:55.766553+0000) 2024-03-04T13:32:56.758 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:32:56.766+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:32:56.766794+0000) 2024-03-04T13:32:57.759 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:32:57.766+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:32:57.767009+0000) 2024-03-04T13:32:58.759 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:32:58.766+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:32:58.767260+0000) 2024-03-04T13:32:59.759 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:32:59.766+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:32:59.767511+0000) 2024-03-04T13:33:00.760 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:33:00.767+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:33:00.767745+0000) 2024-03-04T13:33:01.760 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:33:01.767+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:33:01.767985+0000) 2024-03-04T13:33:02.760 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:33:02.767+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:33:02.768220+0000) 2024-03-04T13:33:03.760 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:33:03.767+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:33:03.768459+0000) 2024-03-04T13:33:04.761 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:33:04.767+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:33:04.768716+0000) 2024-03-04T13:33:05.760 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:33:05.768+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:33:05.768963+0000) 2024-03-04T13:33:06.761 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:33:06.768+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:33:06.769205+0000) 2024-03-04T13:33:07.761 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:33:07.768+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:33:07.769445+0000) 2024-03-04T13:33:08.764 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:33:08.769+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:33:08.769685+0000) 2024-03-04T13:33:09.762 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:33:09.769+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:33:09.769952+0000) 2024-03-04T13:33:09.850 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:33:09.858 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:33:10.762 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:33:10.769+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:33:10.770198+0000) 2024-03-04T13:33:11.762 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:33:11.769+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:33:11.770440+0000) 2024-03-04T13:33:12.762 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:33:12.770+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:33:12.770679+0000) 2024-03-04T13:33:13.762 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:33:13.770+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:33:13.770923+0000) 2024-03-04T13:33:14.763 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:33:14.770+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:33:14.771161+0000) 2024-03-04T13:33:15.763 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:33:15.770+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:33:15.771444+0000) 2024-03-04T13:33:16.763 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:33:16.771+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:33:16.771680+0000) 2024-03-04T13:33:17.763 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:33:17.771+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:33:17.771939+0000) 2024-03-04T13:33:18.764 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:33:18.771+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:33:18.772200+0000) 2024-03-04T13:33:19.764 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:33:19.772+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:33:19.772457+0000) 2024-03-04T13:33:20.764 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:33:20.772+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:33:20.772695+0000) 2024-03-04T13:33:21.765 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:33:21.772+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:33:21.772935+0000) 2024-03-04T13:33:22.765 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:33:22.772+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:33:22.773129+0000) 2024-03-04T13:33:23.765 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:33:23.773+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:33:23.773361+0000) 2024-03-04T13:33:24.765 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:33:24.773+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:33:24.773603+0000) 2024-03-04T13:33:25.765 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:33:25.773+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:33:25.773866+0000) 2024-03-04T13:33:26.766 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:33:26.773+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:33:26.774068+0000) 2024-03-04T13:33:27.766 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:33:27.773+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:33:27.774240+0000) 2024-03-04T13:33:28.766 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:33:28.774+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:33:28.774479+0000) 2024-03-04T13:33:29.766 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:33:29.774+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:33:29.774712+0000) 2024-03-04T13:33:30.766 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:33:30.774+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:33:30.774955+0000) 2024-03-04T13:33:31.767 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:33:31.775+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:33:31.775188+0000) 2024-03-04T13:33:32.767 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:33:32.775+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:33:32.775430+0000) 2024-03-04T13:33:33.768 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:33:33.775+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:33:33.775663+0000) 2024-03-04T13:33:34.768 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:33:34.774+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:33:34.775942+0000) 2024-03-04T13:33:35.768 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:33:35.775+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:33:35.776175+0000) 2024-03-04T13:33:36.768 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:33:36.775+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:33:36.776411+0000) 2024-03-04T13:33:37.768 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:33:37.775+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:33:37.776646+0000) 2024-03-04T13:33:38.768 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:33:38.775+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:33:38.776925+0000) 2024-03-04T13:33:39.769 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:33:39.776+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:33:39.777168+0000) 2024-03-04T13:33:39.988 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:33:39.998 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:33:40.769 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:33:40.776+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:33:40.777416+0000) 2024-03-04T13:33:41.769 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:33:41.776+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:33:41.777648+0000) 2024-03-04T13:33:42.769 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:33:42.776+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:33:42.777932+0000) 2024-03-04T13:33:43.770 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:33:43.777+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:33:43.778175+0000) 2024-03-04T13:33:44.770 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:33:44.777+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:33:44.778417+0000) 2024-03-04T13:33:45.770 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:33:45.777+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:33:45.778668+0000) 2024-03-04T13:33:46.770 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:33:46.778+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:33:46.778959+0000) 2024-03-04T13:33:47.771 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:33:47.778+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:33:47.779148+0000) 2024-03-04T13:33:48.771 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:33:48.778+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:33:48.779396+0000) 2024-03-04T13:33:49.771 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:33:49.778+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:33:49.779657+0000) 2024-03-04T13:33:50.771 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:33:50.779+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:33:50.779952+0000) 2024-03-04T13:33:51.772 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:33:51.779+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:33:51.780183+0000) 2024-03-04T13:33:52.772 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:33:52.779+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:33:52.780423+0000) 2024-03-04T13:33:53.772 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:33:53.779+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:33:53.780666+0000) 2024-03-04T13:33:54.773 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:33:54.780+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:33:54.780951+0000) 2024-03-04T13:33:55.773 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:33:55.780+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:33:55.781184+0000) 2024-03-04T13:33:56.773 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:33:56.780+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:33:56.781424+0000) 2024-03-04T13:33:57.773 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:33:57.780+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:33:57.781657+0000) 2024-03-04T13:33:58.773 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:33:58.781+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:33:58.781955+0000) 2024-03-04T13:33:59.774 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:33:59.781+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:33:59.782185+0000) 2024-03-04T13:34:00.774 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:34:00.781+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:34:00.782424+0000) 2024-03-04T13:34:01.774 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:34:01.781+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:34:01.782659+0000) 2024-03-04T13:34:02.774 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:34:02.782+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:34:02.782961+0000) 2024-03-04T13:34:03.775 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:34:03.782+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:34:03.783221+0000) 2024-03-04T13:34:04.775 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:34:04.782+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:34:04.783482+0000) 2024-03-04T13:34:05.775 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:34:05.783+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:34:05.783735+0000) 2024-03-04T13:34:06.775 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:34:06.783+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:34:06.783985+0000) 2024-03-04T13:34:07.776 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:34:07.783+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:34:07.784223+0000) 2024-03-04T13:34:08.776 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:34:08.783+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:34:08.784490+0000) 2024-03-04T13:34:09.776 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:34:09.784+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:34:09.784777+0000) 2024-03-04T13:34:10.209 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:34:10.218 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:34:10.776 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:34:10.784+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:34:10.785027+0000) 2024-03-04T13:34:11.777 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:34:11.784+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:34:11.785272+0000) 2024-03-04T13:34:12.777 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:34:12.785+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:34:12.785523+0000) 2024-03-04T13:34:13.777 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:34:13.784+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:34:13.785721+0000) 2024-03-04T13:34:14.777 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:34:14.785+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:34:14.785986+0000) 2024-03-04T13:34:15.778 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:34:15.785+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:34:15.786224+0000) 2024-03-04T13:34:16.778 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:34:16.785+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:34:16.786424+0000) 2024-03-04T13:34:17.778 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:34:17.786+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:34:17.786673+0000) 2024-03-04T13:34:18.778 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:34:18.786+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:34:18.786949+0000) 2024-03-04T13:34:19.779 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:34:19.786+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:34:19.787204+0000) 2024-03-04T13:34:20.779 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:34:20.787+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:34:20.787477+0000) 2024-03-04T13:34:21.779 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:34:21.787+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:34:21.787708+0000) 2024-03-04T13:34:22.779 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:34:22.787+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:34:22.787958+0000) 2024-03-04T13:34:23.779 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:34:23.787+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:34:23.788189+0000) 2024-03-04T13:34:24.780 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:34:24.787+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:34:24.788392+0000) 2024-03-04T13:34:25.780 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:34:25.788+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:34:25.788648+0000) 2024-03-04T13:34:26.780 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:34:26.788+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:34:26.788941+0000) 2024-03-04T13:34:27.781 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:34:27.788+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:34:27.789182+0000) 2024-03-04T13:34:28.781 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:34:28.789+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:34:28.789448+0000) 2024-03-04T13:34:29.781 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:34:29.789+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:34:29.789699+0000) 2024-03-04T13:34:30.781 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:34:30.789+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:34:30.789970+0000) 2024-03-04T13:34:31.782 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:34:31.789+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:34:31.790219+0000) 2024-03-04T13:34:32.782 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:34:32.790+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:34:32.790470+0000) 2024-03-04T13:34:33.782 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:34:33.790+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:34:33.790721+0000) 2024-03-04T13:34:34.783 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:34:34.790+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:34:34.790983+0000) 2024-03-04T13:34:35.783 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:34:35.791+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:34:35.791236+0000) 2024-03-04T13:34:36.783 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:34:36.791+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:34:36.791499+0000) 2024-03-04T13:34:37.783 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:34:37.791+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:34:37.791756+0000) 2024-03-04T13:34:38.783 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:34:38.790+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:34:38.792009+0000) 2024-03-04T13:34:39.784 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:34:39.791+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:34:39.792270+0000) 2024-03-04T13:34:40.441 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:34:40.450 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:34:40.784 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:34:40.791+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:34:40.792536+0000) 2024-03-04T13:34:41.784 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:34:41.791+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:34:41.792793+0000) 2024-03-04T13:34:42.784 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:34:42.792+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:34:42.793071+0000) 2024-03-04T13:34:43.785 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:34:43.792+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:34:43.793326+0000) 2024-03-04T13:34:44.785 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:34:44.792+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:34:44.793567+0000) 2024-03-04T13:34:45.785 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:34:45.792+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:34:45.793816+0000) 2024-03-04T13:34:46.785 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:34:46.793+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:34:46.794112+0000) 2024-03-04T13:34:47.786 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:34:47.793+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:34:47.794347+0000) 2024-03-04T13:34:48.786 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:34:48.793+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:34:48.794615+0000) 2024-03-04T13:34:49.786 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:34:49.794+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:34:49.794932+0000) 2024-03-04T13:34:50.787 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:34:50.794+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:34:50.795213+0000) 2024-03-04T13:34:51.787 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:34:51.794+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:34:51.795472+0000) 2024-03-04T13:34:52.787 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:34:52.794+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:34:52.795725+0000) 2024-03-04T13:34:53.787 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:34:53.795+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:34:53.796011+0000) 2024-03-04T13:34:54.787 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:34:54.795+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:34:54.796266+0000) 2024-03-04T13:34:55.788 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:34:55.795+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:34:55.796514+0000) 2024-03-04T13:34:56.788 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:34:56.795+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:34:56.796769+0000) 2024-03-04T13:34:57.788 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:34:57.796+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:34:57.797034+0000) 2024-03-04T13:34:58.789 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:34:58.796+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:34:58.797307+0000) 2024-03-04T13:34:59.789 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:34:59.796+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:34:59.797560+0000) 2024-03-04T13:35:00.789 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:35:00.797+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:35:00.797858+0000) 2024-03-04T13:35:01.789 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:35:01.797+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:35:01.798111+0000) 2024-03-04T13:35:02.790 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:35:02.797+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:35:02.798376+0000) 2024-03-04T13:35:03.790 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:35:03.797+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:35:03.798642+0000) 2024-03-04T13:35:04.790 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:35:04.798+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:35:04.798951+0000) 2024-03-04T13:35:05.791 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:35:05.798+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:35:05.799210+0000) 2024-03-04T13:35:06.791 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:35:06.798+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:35:06.799487+0000) 2024-03-04T13:35:07.791 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:35:07.798+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:35:07.799701+0000) 2024-03-04T13:35:08.791 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:35:08.799+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:35:08.799981+0000) 2024-03-04T13:35:09.792 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:35:09.799+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:35:09.800202+0000) 2024-03-04T13:35:10.660 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:35:10.668 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:35:10.817 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:35:10.799+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:35:10.800433+0000) 2024-03-04T13:35:11.792 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:35:11.800+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:35:11.800669+0000) 2024-03-04T13:35:12.792 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:35:12.800+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:35:12.800947+0000) 2024-03-04T13:35:13.792 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:35:13.800+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:35:13.801141+0000) 2024-03-04T13:35:14.793 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:35:14.800+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:35:14.801383+0000) 2024-03-04T13:35:15.793 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:35:15.801+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:35:15.801621+0000) 2024-03-04T13:35:16.793 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:35:16.801+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:35:16.801930+0000) 2024-03-04T13:35:17.793 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:35:17.801+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:35:17.802166+0000) 2024-03-04T13:35:18.794 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:35:18.801+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:35:18.802440+0000) 2024-03-04T13:35:19.794 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:35:19.802+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:35:19.802680+0000) 2024-03-04T13:35:20.794 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:35:20.802+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:35:20.802944+0000) 2024-03-04T13:35:21.794 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:35:21.802+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:35:21.803199+0000) 2024-03-04T13:35:22.795 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:35:22.802+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:35:22.803442+0000) 2024-03-04T13:35:23.795 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:35:23.802+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:35:23.803689+0000) 2024-03-04T13:35:24.795 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:35:24.803+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:35:24.803960+0000) 2024-03-04T13:35:25.795 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:35:25.803+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:35:25.804197+0000) 2024-03-04T13:35:26.796 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:35:26.804+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:35:26.804453+0000) 2024-03-04T13:35:27.796 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:35:27.804+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:35:27.804713+0000) 2024-03-04T13:35:28.796 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:35:28.804+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:35:28.805000+0000) 2024-03-04T13:35:29.796 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:35:29.804+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:35:29.805239+0000) 2024-03-04T13:35:30.797 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:35:30.805+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:35:30.805486+0000) 2024-03-04T13:35:31.797 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:35:31.805+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:35:31.805720+0000) 2024-03-04T13:35:32.797 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:35:32.805+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:35:32.805962+0000) 2024-03-04T13:35:33.800 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:35:33.805+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:35:33.806203+0000) 2024-03-04T13:35:34.798 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:35:34.806+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:35:34.806449+0000) 2024-03-04T13:35:35.798 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:35:35.806+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:35:35.806688+0000) 2024-03-04T13:35:36.798 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:35:36.806+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:35:36.806929+0000) 2024-03-04T13:35:37.798 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:35:37.806+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:35:37.807127+0000) 2024-03-04T13:35:38.799 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:35:38.807+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:35:38.807387+0000) 2024-03-04T13:35:39.799 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:35:39.807+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:35:39.807631+0000) 2024-03-04T13:35:40.799 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:35:40.807+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:35:40.807918+0000) 2024-03-04T13:35:40.891 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:35:40.899 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:35:41.799 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:35:41.808+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:35:41.808155+0000) 2024-03-04T13:35:42.799 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:35:42.807+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:35:42.808369+0000) 2024-03-04T13:35:43.800 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:35:43.807+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:35:43.808628+0000) 2024-03-04T13:35:44.800 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:35:44.807+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:35:44.808942+0000) 2024-03-04T13:35:45.800 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:35:45.808+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:35:45.809194+0000) 2024-03-04T13:35:46.801 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:35:46.808+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:35:46.809457+0000) 2024-03-04T13:35:47.801 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:35:47.808+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:35:47.809704+0000) 2024-03-04T13:35:48.801 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:35:48.808+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:35:48.810001+0000) 2024-03-04T13:35:49.801 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:35:49.809+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:35:49.810281+0000) 2024-03-04T13:35:50.802 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:35:50.809+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:35:50.810538+0000) 2024-03-04T13:35:51.802 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:35:51.809+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:35:51.810785+0000) 2024-03-04T13:35:52.803 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:35:52.810+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:35:52.811084+0000) 2024-03-04T13:35:53.803 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:35:53.810+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:35:53.811350+0000) 2024-03-04T13:35:54.803 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:35:54.810+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:35:54.811612+0000) 2024-03-04T13:35:55.803 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:35:55.811+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:35:55.811916+0000) 2024-03-04T13:35:56.803 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:35:56.811+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:35:56.812122+0000) 2024-03-04T13:35:57.803 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:35:57.811+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:35:57.812371+0000) 2024-03-04T13:35:58.804 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:35:58.811+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:35:58.812644+0000) 2024-03-04T13:35:59.804 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:35:59.812+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:35:59.812920+0000) 2024-03-04T13:36:00.804 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:36:00.812+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:36:00.813173+0000) 2024-03-04T13:36:01.805 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:36:01.812+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:36:01.813488+0000) 2024-03-04T13:36:02.805 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:36:02.812+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:36:02.813744+0000) 2024-03-04T13:36:03.805 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:36:03.813+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:36:03.814001+0000) 2024-03-04T13:36:04.805 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:36:04.813+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:36:04.814267+0000) 2024-03-04T13:36:05.806 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:36:05.813+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:36:05.814517+0000) 2024-03-04T13:36:06.806 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:36:06.814+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:36:06.814772+0000) 2024-03-04T13:36:07.806 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:36:07.814+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:36:07.815028+0000) 2024-03-04T13:36:08.806 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:36:08.814+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:36:08.815286+0000) 2024-03-04T13:36:09.807 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:36:09.814+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:36:09.815555+0000) 2024-03-04T13:36:10.807 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:36:10.815+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:36:10.815808+0000) 2024-03-04T13:36:11.037 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:36:11.046 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:36:11.807 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:36:11.815+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:36:11.816061+0000) 2024-03-04T13:36:12.807 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:36:12.815+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:36:12.816326+0000) 2024-03-04T13:36:13.808 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:36:13.815+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:36:13.816623+0000) 2024-03-04T13:36:14.808 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:36:14.816+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:36:14.816933+0000) 2024-03-04T13:36:15.808 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:36:15.816+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:36:15.817179+0000) 2024-03-04T13:36:16.808 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:36:16.816+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:36:16.817433+0000) 2024-03-04T13:36:17.809 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:36:17.817+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:36:17.817705+0000) 2024-03-04T13:36:18.809 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:36:18.817+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:36:18.817999+0000) 2024-03-04T13:36:19.809 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:36:19.817+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:36:19.818263+0000) 2024-03-04T13:36:20.810 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:36:20.817+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:36:20.818514+0000) 2024-03-04T13:36:21.810 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:36:21.818+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:36:21.818765+0000) 2024-03-04T13:36:22.810 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:36:22.818+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:36:22.819025+0000) 2024-03-04T13:36:23.810 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:36:23.818+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:36:23.819291+0000) 2024-03-04T13:36:24.810 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:36:24.818+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:36:24.819505+0000) 2024-03-04T13:36:25.811 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:36:25.819+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:36:25.819761+0000) 2024-03-04T13:36:26.811 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:36:26.819+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:36:26.820023+0000) 2024-03-04T13:36:27.811 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:36:27.819+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:36:27.820273+0000) 2024-03-04T13:36:28.812 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:36:28.820+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:36:28.820545+0000) 2024-03-04T13:36:29.812 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:36:29.820+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:36:29.820805+0000) 2024-03-04T13:36:30.812 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:36:30.820+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:36:30.821110+0000) 2024-03-04T13:36:31.812 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:36:31.821+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:36:31.821382+0000) 2024-03-04T13:36:32.813 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:36:32.820+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:36:32.821661+0000) 2024-03-04T13:36:33.813 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:36:33.821+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:36:33.821947+0000) 2024-03-04T13:36:34.813 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:36:34.821+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:36:34.822221+0000) 2024-03-04T13:36:35.813 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:36:35.822+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:36:35.822472+0000) 2024-03-04T13:36:36.814 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:36:36.821+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:36:36.822743+0000) 2024-03-04T13:36:37.814 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:36:37.822+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:36:37.823013+0000) 2024-03-04T13:36:38.814 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:36:38.822+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:36:38.823299+0000) 2024-03-04T13:36:39.815 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:36:39.822+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:36:39.823536+0000) 2024-03-04T13:36:40.815 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:36:40.823+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:36:40.823793+0000) 2024-03-04T13:36:41.242 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:36:41.251 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:36:41.815 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:36:41.823+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:36:41.824095+0000) 2024-03-04T13:36:42.815 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:36:42.824+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:36:42.824356+0000) 2024-03-04T13:36:43.816 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:36:43.824+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:36:43.824611+0000) 2024-03-04T13:36:44.820 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:36:44.824+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:36:44.824924+0000) 2024-03-04T13:36:45.816 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:36:45.824+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:36:45.825149+0000) 2024-03-04T13:36:46.816 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:36:46.824+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:36:46.825424+0000) 2024-03-04T13:36:47.817 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:36:47.824+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:36:47.825674+0000) 2024-03-04T13:36:48.817 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:36:48.824+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:36:48.825946+0000) 2024-03-04T13:36:49.817 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:36:49.825+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:36:49.826206+0000) 2024-03-04T13:36:50.817 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:36:50.825+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:36:50.826445+0000) 2024-03-04T13:36:51.818 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:36:51.824+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:36:51.826692+0000) 2024-03-04T13:36:52.818 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:36:52.825+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:36:52.826935+0000) 2024-03-04T13:36:53.818 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:36:53.826+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:36:53.827176+0000) 2024-03-04T13:36:54.819 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:36:54.825+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:36:54.827661+0000) 2024-03-04T13:36:55.819 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:36:55.826+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:36:55.827937+0000) 2024-03-04T13:36:56.819 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:36:56.827+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:36:56.828172+0000) 2024-03-04T13:36:57.819 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:36:57.827+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:36:57.828364+0000) 2024-03-04T13:36:58.820 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:36:58.827+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:36:58.828619+0000) 2024-03-04T13:36:59.820 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:36:59.828+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:36:59.828910+0000) 2024-03-04T13:37:00.820 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:37:00.828+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:37:00.829157+0000) 2024-03-04T13:37:01.820 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:37:01.828+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:37:01.829333+0000) 2024-03-04T13:37:02.821 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:37:02.828+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:37:02.829578+0000) 2024-03-04T13:37:03.821 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:37:03.829+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:37:03.829808+0000) 2024-03-04T13:37:04.821 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:37:04.829+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:37:04.830093+0000) 2024-03-04T13:37:05.821 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:37:05.829+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:37:05.830331+0000) 2024-03-04T13:37:06.822 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:37:06.829+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:37:06.830575+0000) 2024-03-04T13:37:07.822 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:37:07.829+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:37:07.830748+0000) 2024-03-04T13:37:08.822 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:37:08.830+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:37:08.830994+0000) 2024-03-04T13:37:09.822 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:37:09.830+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:37:09.831236+0000) 2024-03-04T13:37:10.823 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:37:10.830+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:37:10.831472+0000) 2024-03-04T13:37:11.388 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:37:11.396 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:37:11.823 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:37:11.830+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:37:11.831718+0000) 2024-03-04T13:37:12.823 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:37:12.831+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:37:12.831959+0000) 2024-03-04T13:37:13.823 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:37:13.831+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:37:13.832187+0000) 2024-03-04T13:37:14.823 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:37:14.831+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:37:14.832440+0000) 2024-03-04T13:37:15.824 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:37:15.831+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:37:15.832638+0000) 2024-03-04T13:37:16.824 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:37:16.832+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:37:16.832818+0000) 2024-03-04T13:37:17.824 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:37:17.832+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:37:17.833115+0000) 2024-03-04T13:37:18.825 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:37:18.832+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:37:18.833352+0000) 2024-03-04T13:37:19.824 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:37:19.832+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:37:19.833560+0000) 2024-03-04T13:37:20.825 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:37:20.833+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:37:20.833864+0000) 2024-03-04T13:37:21.825 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:37:21.833+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:37:21.834096+0000) 2024-03-04T13:37:22.825 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:37:22.833+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:37:22.834339+0000) 2024-03-04T13:37:23.825 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:37:23.833+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:37:23.834571+0000) 2024-03-04T13:37:24.826 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:37:24.834+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:37:24.834814+0000) 2024-03-04T13:37:25.826 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:37:25.834+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:37:25.835112+0000) 2024-03-04T13:37:26.826 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:37:26.834+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:37:26.835366+0000) 2024-03-04T13:37:27.827 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:37:27.835+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:37:27.835600+0000) 2024-03-04T13:37:28.827 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:37:28.835+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:37:28.835909+0000) 2024-03-04T13:37:29.827 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:37:29.835+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:37:29.836153+0000) 2024-03-04T13:37:30.827 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:37:30.835+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:37:30.836395+0000) 2024-03-04T13:37:31.828 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:37:31.836+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:37:31.836638+0000) 2024-03-04T13:37:32.828 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:37:32.836+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:37:32.836931+0000) 2024-03-04T13:37:33.828 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:37:33.836+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:37:33.837203+0000) 2024-03-04T13:37:34.828 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:37:34.836+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:37:34.837420+0000) 2024-03-04T13:37:35.829 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:37:35.837+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:37:35.837674+0000) 2024-03-04T13:37:36.829 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:37:36.837+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:37:36.837925+0000) 2024-03-04T13:37:37.829 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:37:37.837+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:37:37.838257+0000) 2024-03-04T13:37:38.829 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:37:38.838+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:37:38.838520+0000) 2024-03-04T13:37:39.830 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:37:39.838+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:37:39.838752+0000) 2024-03-04T13:37:40.830 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:37:40.838+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:37:40.838992+0000) 2024-03-04T13:37:41.601 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:37:41.610 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:37:41.830 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:37:41.838+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:37:41.839240+0000) 2024-03-04T13:37:42.830 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:37:42.839+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:37:42.839495+0000) 2024-03-04T13:37:43.831 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:37:43.839+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:37:43.839727+0000) 2024-03-04T13:37:44.831 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:37:44.839+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:37:44.839969+0000) 2024-03-04T13:37:45.831 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:37:45.839+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:37:45.840156+0000) 2024-03-04T13:37:46.831 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:37:46.840+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:37:46.840398+0000) 2024-03-04T13:37:47.832 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:37:47.840+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:37:47.840672+0000) 2024-03-04T13:37:48.832 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:37:48.840+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:37:48.840933+0000) 2024-03-04T13:37:49.832 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:37:49.841+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:37:49.841168+0000) 2024-03-04T13:37:50.832 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:37:50.840+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:37:50.841428+0000) 2024-03-04T13:37:51.833 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:37:51.840+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:37:51.841683+0000) 2024-03-04T13:37:52.833 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:37:52.840+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:37:52.841934+0000) 2024-03-04T13:37:53.833 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:37:53.841+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:37:53.842184+0000) 2024-03-04T13:37:54.833 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:37:54.841+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:37:54.842428+0000) 2024-03-04T13:37:55.833 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:37:55.841+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:37:55.842663+0000) 2024-03-04T13:37:56.834 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:37:56.841+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:37:56.842943+0000) 2024-03-04T13:37:57.836 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:37:57.842+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:37:57.843176+0000) 2024-03-04T13:37:58.834 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:37:58.842+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:37:58.843443+0000) 2024-03-04T13:37:59.834 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:37:59.842+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:37:59.843675+0000) 2024-03-04T13:38:00.835 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:38:00.842+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:38:00.843949+0000) 2024-03-04T13:38:01.835 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:38:01.843+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:38:01.844184+0000) 2024-03-04T13:38:02.835 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:38:02.843+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:38:02.844418+0000) 2024-03-04T13:38:03.836 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:38:03.843+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:38:03.844647+0000) 2024-03-04T13:38:04.836 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:38:04.844+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:38:04.844947+0000) 2024-03-04T13:38:05.836 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:38:05.844+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:38:05.845191+0000) 2024-03-04T13:38:06.836 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:38:06.844+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:38:06.845427+0000) 2024-03-04T13:38:07.840 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:38:07.844+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:38:07.845617+0000) 2024-03-04T13:38:08.837 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:38:08.845+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:38:08.845892+0000) 2024-03-04T13:38:09.837 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:38:09.845+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:38:09.846126+0000) 2024-03-04T13:38:10.837 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:38:10.845+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:38:10.846339+0000) 2024-03-04T13:38:11.757 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:38:11.765 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:38:11.837 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:38:11.845+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:38:11.846551+0000) 2024-03-04T13:38:12.838 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:38:12.845+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:38:12.846735+0000) 2024-03-04T13:38:13.838 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:38:13.846+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:38:13.846934+0000) 2024-03-04T13:38:14.838 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:38:14.846+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:38:14.847117+0000) 2024-03-04T13:38:15.838 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:38:15.846+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:38:15.847290+0000) 2024-03-04T13:38:16.838 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:38:16.846+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:38:16.847468+0000) 2024-03-04T13:38:17.838 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:38:17.846+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:38:17.847653+0000) 2024-03-04T13:38:18.839 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:38:18.847+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:38:18.847879+0000) 2024-03-04T13:38:19.839 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:38:19.847+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:38:19.848055+0000) 2024-03-04T13:38:20.839 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:38:20.847+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:38:20.848242+0000) 2024-03-04T13:38:21.839 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:38:21.847+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:38:21.848419+0000) 2024-03-04T13:38:22.839 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:38:22.847+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:38:22.848599+0000) 2024-03-04T13:38:23.840 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:38:23.847+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:38:23.848772+0000) 2024-03-04T13:38:24.840 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:38:24.848+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:38:24.848943+0000) 2024-03-04T13:38:25.840 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:38:25.848+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:38:25.849120+0000) 2024-03-04T13:38:26.840 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:38:26.848+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:38:26.849311+0000) 2024-03-04T13:38:27.840 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:38:27.848+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:38:27.849502+0000) 2024-03-04T13:38:28.840 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:38:28.848+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:38:28.849686+0000) 2024-03-04T13:38:29.841 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:38:29.849+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:38:29.849944+0000) 2024-03-04T13:38:30.841 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:38:30.849+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:38:30.850140+0000) 2024-03-04T13:38:31.841 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:38:31.849+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:38:31.850327+0000) 2024-03-04T13:38:32.841 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:38:32.849+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:38:32.850588+0000) 2024-03-04T13:38:33.853 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:38:33.850+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:38:33.850817+0000) 2024-03-04T13:38:34.842 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:38:34.850+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:38:34.851044+0000) 2024-03-04T13:38:35.842 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:38:35.850+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:38:35.851286+0000) 2024-03-04T13:38:36.842 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:38:36.851+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:38:36.851523+0000) 2024-03-04T13:38:37.843 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:38:37.851+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:38:37.851757+0000) 2024-03-04T13:38:38.843 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:38:38.851+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:38:38.852001+0000) 2024-03-04T13:38:39.843 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:38:39.851+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:38:39.852236+0000) 2024-03-04T13:38:40.843 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:38:40.852+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:38:40.852482+0000) 2024-03-04T13:38:41.844 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:38:41.852+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:38:41.852717+0000) 2024-03-04T13:38:41.961 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:38:41.969 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:38:42.844 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:38:42.852+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:38:42.853000+0000) 2024-03-04T13:38:43.844 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:38:43.852+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:38:43.853231+0000) 2024-03-04T13:38:44.845 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:38:44.853+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:38:44.853469+0000) 2024-03-04T13:38:45.848 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:38:45.853+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:38:45.853704+0000) 2024-03-04T13:38:46.845 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:38:46.853+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:38:46.853942+0000) 2024-03-04T13:38:47.845 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:38:47.853+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:38:47.854173+0000) 2024-03-04T13:38:48.845 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:38:48.854+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:38:48.854424+0000) 2024-03-04T13:38:49.846 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:38:49.854+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:38:49.854662+0000) 2024-03-04T13:38:50.846 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:38:50.854+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:38:50.854944+0000) 2024-03-04T13:38:51.846 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:38:51.855+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:38:51.855183+0000) 2024-03-04T13:38:52.847 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:38:52.855+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:38:52.855439+0000) 2024-03-04T13:38:53.847 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:38:53.855+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:38:53.855690+0000) 2024-03-04T13:38:54.847 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:38:54.854+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:38:54.855976+0000) 2024-03-04T13:38:55.847 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:38:55.855+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:38:55.856208+0000) 2024-03-04T13:38:56.848 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:38:56.855+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:38:56.856452+0000) 2024-03-04T13:38:57.848 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:38:57.855+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:38:57.856686+0000) 2024-03-04T13:38:58.848 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:38:58.855+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:38:58.856951+0000) 2024-03-04T13:38:59.848 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:38:59.856+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:38:59.857185+0000) 2024-03-04T13:39:00.849 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:39:00.856+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:39:00.857427+0000) 2024-03-04T13:39:01.849 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:39:01.856+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:39:01.857661+0000) 2024-03-04T13:39:02.849 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:39:02.856+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:39:02.857971+0000) 2024-03-04T13:39:03.850 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:39:03.857+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:39:03.858204+0000) 2024-03-04T13:39:04.850 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:39:04.857+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:39:04.858481+0000) 2024-03-04T13:39:05.850 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:39:05.857+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:39:05.858716+0000) 2024-03-04T13:39:06.850 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:39:06.858+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:39:06.858980+0000) 2024-03-04T13:39:07.851 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:39:07.858+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:39:07.859217+0000) 2024-03-04T13:39:08.851 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:39:08.858+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:39:08.859458+0000) 2024-03-04T13:39:09.851 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:39:09.858+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:39:09.859692+0000) 2024-03-04T13:39:10.851 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:39:10.859+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:39:10.859937+0000) 2024-03-04T13:39:11.852 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:39:11.859+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:39:11.860169+0000) 2024-03-04T13:39:12.199 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:39:12.207 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:39:12.852 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:39:12.859+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:39:12.860415+0000) 2024-03-04T13:39:13.852 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:39:13.859+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:39:13.860651+0000) 2024-03-04T13:39:14.852 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:39:14.860+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:39:14.860944+0000) 2024-03-04T13:39:15.853 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:39:15.860+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:39:15.861211+0000) 2024-03-04T13:39:16.853 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:39:16.860+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:39:16.861447+0000) 2024-03-04T13:39:17.853 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:39:17.860+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:39:17.861679+0000) 2024-03-04T13:39:18.854 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:39:18.861+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:39:18.861925+0000) 2024-03-04T13:39:19.854 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:39:19.861+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:39:19.862166+0000) 2024-03-04T13:39:20.854 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:39:20.861+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:39:20.862412+0000) 2024-03-04T13:39:21.854 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:39:21.861+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:39:21.862646+0000) 2024-03-04T13:39:22.855 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:39:22.862+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:39:22.862949+0000) 2024-03-04T13:39:23.855 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:39:23.862+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:39:23.863257+0000) 2024-03-04T13:39:24.855 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:39:24.862+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:39:24.863508+0000) 2024-03-04T13:39:25.855 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:39:25.863+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:39:25.863742+0000) 2024-03-04T13:39:26.856 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:39:26.863+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:39:26.863985+0000) 2024-03-04T13:39:27.856 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:39:27.863+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:39:27.864225+0000) 2024-03-04T13:39:28.856 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:39:28.863+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:39:28.864476+0000) 2024-03-04T13:39:29.856 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:39:29.864+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:39:29.864715+0000) 2024-03-04T13:39:30.857 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:39:30.864+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:39:30.864959+0000) 2024-03-04T13:39:31.857 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:39:31.864+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:39:31.865192+0000) 2024-03-04T13:39:32.857 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:39:32.864+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:39:32.865424+0000) 2024-03-04T13:39:33.858 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:39:33.865+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:39:33.865665+0000) 2024-03-04T13:39:34.858 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:39:34.865+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:39:34.865931+0000) 2024-03-04T13:39:35.858 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:39:35.865+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:39:35.866166+0000) 2024-03-04T13:39:36.858 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:39:36.865+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:39:36.866407+0000) 2024-03-04T13:39:37.859 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:39:37.865+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:39:37.866589+0000) 2024-03-04T13:39:38.859 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:39:38.866+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:39:38.866917+0000) 2024-03-04T13:39:39.859 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:39:39.866+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:39:39.867173+0000) 2024-03-04T13:39:40.860 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:39:40.867+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:39:40.867418+0000) 2024-03-04T13:39:41.860 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:39:41.867+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:39:41.867653+0000) 2024-03-04T13:39:42.339 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:39:42.347 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:39:42.860 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:39:42.867+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:39:42.867929+0000) 2024-03-04T13:39:43.860 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:39:43.867+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:39:43.868163+0000) 2024-03-04T13:39:44.860 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:39:44.868+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:39:44.868408+0000) 2024-03-04T13:39:45.861 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:39:45.868+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:39:45.868645+0000) 2024-03-04T13:39:46.861 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:39:46.868+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:39:46.868927+0000) 2024-03-04T13:39:47.861 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:39:47.868+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:39:47.869163+0000) 2024-03-04T13:39:48.862 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:39:48.869+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:39:48.869434+0000) 2024-03-04T13:39:49.862 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:39:49.869+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:39:49.869670+0000) 2024-03-04T13:39:50.862 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:39:50.869+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:39:50.869940+0000) 2024-03-04T13:39:51.862 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:39:51.869+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:39:51.870167+0000) 2024-03-04T13:39:52.863 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:39:52.869+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:39:52.870347+0000) 2024-03-04T13:39:53.863 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:39:53.870+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:39:53.870601+0000) 2024-03-04T13:39:54.863 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:39:54.870+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:39:54.870914+0000) 2024-03-04T13:39:55.863 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:39:55.870+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:39:55.871156+0000) 2024-03-04T13:39:56.864 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:39:56.871+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:39:56.871392+0000) 2024-03-04T13:39:57.864 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:39:57.871+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:39:57.871647+0000) 2024-03-04T13:39:58.864 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:39:58.870+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:39:58.871923+0000) 2024-03-04T13:39:59.864 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:39:59.871+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:39:59.872166+0000) 2024-03-04T13:40:00.865 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:40:00.871+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:40:00.872406+0000) 2024-03-04T13:40:01.865 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:40:01.871+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:40:01.872645+0000) 2024-03-04T13:40:02.865 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:40:02.871+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:40:02.872931+0000) 2024-03-04T13:40:03.866 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:40:03.872+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:40:03.873164+0000) 2024-03-04T13:40:04.866 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:40:04.872+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:40:04.873406+0000) 2024-03-04T13:40:05.866 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:40:05.872+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:40:05.873640+0000) 2024-03-04T13:40:06.866 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:40:06.872+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:40:06.873966+0000) 2024-03-04T13:40:07.867 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:40:07.873+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:40:07.874198+0000) 2024-03-04T13:40:08.867 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:40:08.873+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:40:08.874449+0000) 2024-03-04T13:40:09.867 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:40:09.873+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:40:09.874701+0000) 2024-03-04T13:40:10.867 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:40:10.874+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:40:10.874942+0000) 2024-03-04T13:40:11.868 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:40:11.874+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:40:11.875177+0000) 2024-03-04T13:40:12.569 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:40:12.577 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:40:12.868 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:40:12.874+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:40:12.875427+0000) 2024-03-04T13:40:13.868 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:40:13.874+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:40:13.875663+0000) 2024-03-04T13:40:14.868 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:40:14.875+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:40:14.875948+0000) 2024-03-04T13:40:15.869 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:40:15.875+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:40:15.876183+0000) 2024-03-04T13:40:16.869 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:40:16.875+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:40:16.876422+0000) 2024-03-04T13:40:17.869 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:40:17.875+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:40:17.876654+0000) 2024-03-04T13:40:18.870 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:40:18.876+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:40:18.876939+0000) 2024-03-04T13:40:19.870 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:40:19.876+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:40:19.877184+0000) 2024-03-04T13:40:20.870 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:40:20.876+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:40:20.877442+0000) 2024-03-04T13:40:21.870 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:40:21.876+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:40:21.877681+0000) 2024-03-04T13:40:22.871 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:40:22.877+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:40:22.877968+0000) 2024-03-04T13:40:23.871 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:40:23.877+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:40:23.878220+0000) 2024-03-04T13:40:24.871 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:40:24.877+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:40:24.878476+0000) 2024-03-04T13:40:25.871 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:40:25.878+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:40:25.878724+0000) 2024-03-04T13:40:26.872 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:40:26.878+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:40:26.878978+0000) 2024-03-04T13:40:27.872 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:40:27.878+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:40:27.879224+0000) 2024-03-04T13:40:28.872 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:40:28.878+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:40:28.879467+0000) 2024-03-04T13:40:29.872 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:40:29.879+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:40:29.879715+0000) 2024-03-04T13:40:30.873 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:40:30.879+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:40:30.879961+0000) 2024-03-04T13:40:31.873 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:40:31.879+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:40:31.880196+0000) 2024-03-04T13:40:32.873 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:40:32.879+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:40:32.880431+0000) 2024-03-04T13:40:33.874 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:40:33.880+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:40:33.880671+0000) 2024-03-04T13:40:34.874 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:40:34.880+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:40:34.880945+0000) 2024-03-04T13:40:35.874 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:40:35.880+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:40:35.881197+0000) 2024-03-04T13:40:36.874 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:40:36.880+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:40:36.881437+0000) 2024-03-04T13:40:37.875 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:40:37.881+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:40:37.881671+0000) 2024-03-04T13:40:38.875 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:40:38.881+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:40:38.881967+0000) 2024-03-04T13:40:39.875 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:40:39.881+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:40:39.882165+0000) 2024-03-04T13:40:40.875 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:40:40.881+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:40:40.882441+0000) 2024-03-04T13:40:41.876 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:40:41.881+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:40:41.882680+0000) 2024-03-04T13:40:42.782 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:40:42.789 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:40:42.876 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:40:42.882+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:40:42.882972+0000) 2024-03-04T13:40:43.876 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:40:43.882+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:40:43.883205+0000) 2024-03-04T13:40:44.876 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:40:44.883+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:40:44.883456+0000) 2024-03-04T13:40:45.879 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:40:45.883+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:40:45.883702+0000) 2024-03-04T13:40:46.877 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:40:46.883+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:40:46.883938+0000) 2024-03-04T13:40:47.877 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:40:47.883+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:40:47.884175+0000) 2024-03-04T13:40:48.877 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:40:48.883+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:40:48.884385+0000) 2024-03-04T13:40:49.878 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:40:49.884+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:40:49.884632+0000) 2024-03-04T13:40:50.878 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:40:50.884+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:40:50.884918+0000) 2024-03-04T13:40:51.878 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:40:51.884+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:40:51.885150+0000) 2024-03-04T13:40:52.879 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:40:52.885+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:40:52.885388+0000) 2024-03-04T13:40:53.879 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:40:53.885+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:40:53.885649+0000) 2024-03-04T13:40:54.879 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:40:54.885+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:40:54.885952+0000) 2024-03-04T13:40:55.879 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:40:55.885+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:40:55.886197+0000) 2024-03-04T13:40:56.880 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:40:56.886+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:40:56.886440+0000) 2024-03-04T13:40:57.880 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:40:57.886+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:40:57.886674+0000) 2024-03-04T13:40:58.880 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:40:58.886+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:40:58.886933+0000) 2024-03-04T13:40:59.881 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:40:59.887+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:40:59.887190+0000) 2024-03-04T13:41:00.881 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:41:00.887+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:41:00.887438+0000) 2024-03-04T13:41:01.881 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:41:01.886+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:41:01.887679+0000) 2024-03-04T13:41:02.881 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:41:02.886+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:41:02.887947+0000) 2024-03-04T13:41:03.881 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:41:03.886+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:41:03.888132+0000) 2024-03-04T13:41:04.882 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:41:04.887+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:41:04.888378+0000) 2024-03-04T13:41:05.882 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:41:05.887+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:41:05.888632+0000) 2024-03-04T13:41:06.882 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:41:06.887+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:41:06.888934+0000) 2024-03-04T13:41:07.883 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:41:07.888+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:41:07.889178+0000) 2024-03-04T13:41:08.883 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:41:08.888+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:41:08.889431+0000) 2024-03-04T13:41:09.883 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:41:09.888+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:41:09.889675+0000) 2024-03-04T13:41:10.883 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:41:10.888+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:41:10.889970+0000) 2024-03-04T13:41:11.884 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:41:11.889+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:41:11.890201+0000) 2024-03-04T13:41:12.884 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:41:12.889+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:41:12.890437+0000) 2024-03-04T13:41:12.938 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:41:12.946 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:41:13.884 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:41:13.889+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:41:13.890674+0000) 2024-03-04T13:41:14.884 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:41:14.890+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:41:14.890942+0000) 2024-03-04T13:41:15.885 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:41:15.890+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:41:15.891239+0000) 2024-03-04T13:41:16.885 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:41:16.890+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:41:16.891482+0000) 2024-03-04T13:41:17.885 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:41:17.890+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:41:17.891716+0000) 2024-03-04T13:41:18.885 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:41:18.891+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:41:18.891950+0000) 2024-03-04T13:41:19.886 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:41:19.891+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:41:19.892194+0000) 2024-03-04T13:41:20.886 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:41:20.891+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:41:20.892449+0000) 2024-03-04T13:41:21.886 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:41:21.891+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:41:21.892685+0000) 2024-03-04T13:41:22.886 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:41:22.892+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:41:22.892905+0000) 2024-03-04T13:41:23.887 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:41:23.892+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:41:23.893160+0000) 2024-03-04T13:41:24.887 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:41:24.892+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:41:24.893430+0000) 2024-03-04T13:41:25.887 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:41:25.892+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:41:25.893703+0000) 2024-03-04T13:41:26.887 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:41:26.893+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:41:26.893948+0000) 2024-03-04T13:41:27.888 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:41:27.893+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:41:27.894182+0000) 2024-03-04T13:41:28.888 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:41:28.893+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:41:28.894424+0000) 2024-03-04T13:41:29.888 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:41:29.893+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:41:29.894675+0000) 2024-03-04T13:41:30.889 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:41:30.894+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:41:30.894955+0000) 2024-03-04T13:41:31.889 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:41:31.894+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:41:31.895291+0000) 2024-03-04T13:41:32.889 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:41:32.894+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:41:32.895580+0000) 2024-03-04T13:41:33.890 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:41:33.895+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:41:33.895818+0000) 2024-03-04T13:41:34.890 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:41:34.895+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:41:34.896101+0000) 2024-03-04T13:41:35.890 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:41:35.895+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:41:35.896348+0000) 2024-03-04T13:41:36.890 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:41:36.896+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:41:36.896587+0000) 2024-03-04T13:41:37.891 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:41:37.896+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:41:37.896852+0000) 2024-03-04T13:41:38.891 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:41:38.896+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:41:38.897112+0000) 2024-03-04T13:41:39.891 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:41:39.896+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:41:39.897389+0000) 2024-03-04T13:41:40.891 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:41:40.897+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:41:40.897638+0000) 2024-03-04T13:41:41.892 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:41:41.897+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:41:41.897948+0000) 2024-03-04T13:41:42.892 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:41:42.897+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:41:42.898193+0000) 2024-03-04T13:41:43.135 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:41:43.143 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:41:43.892 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:41:43.897+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:41:43.898380+0000) 2024-03-04T13:41:44.892 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:41:44.898+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:41:44.898632+0000) 2024-03-04T13:41:45.893 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:41:45.898+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:41:45.898923+0000) 2024-03-04T13:41:46.893 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:41:46.898+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:41:46.899180+0000) 2024-03-04T13:41:47.893 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:41:47.899+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:41:47.899435+0000) 2024-03-04T13:41:48.894 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:41:48.899+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:41:48.899698+0000) 2024-03-04T13:41:49.894 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:41:49.899+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:41:49.899946+0000) 2024-03-04T13:41:50.894 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:41:50.899+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:41:50.900211+0000) 2024-03-04T13:41:51.894 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:41:51.900+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:41:51.900461+0000) 2024-03-04T13:41:52.895 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:41:52.900+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:41:52.900731+0000) 2024-03-04T13:41:53.895 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:41:53.900+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:41:53.901007+0000) 2024-03-04T13:41:54.895 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:41:54.900+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:41:54.901285+0000) 2024-03-04T13:41:55.896 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:41:55.901+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:41:55.901542+0000) 2024-03-04T13:41:56.896 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:41:56.901+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:41:56.901795+0000) 2024-03-04T13:41:57.896 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:41:57.901+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:41:57.902097+0000) 2024-03-04T13:41:58.896 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:41:58.902+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:41:58.902385+0000) 2024-03-04T13:41:59.897 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:41:59.902+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:41:59.902616+0000) 2024-03-04T13:42:00.897 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:42:00.902+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:42:00.902916+0000) 2024-03-04T13:42:01.897 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:42:01.902+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:42:01.903157+0000) 2024-03-04T13:42:02.897 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:42:02.902+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:42:02.903417+0000) 2024-03-04T13:42:03.898 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:42:03.902+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:42:03.903722+0000) 2024-03-04T13:42:04.898 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:42:04.903+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:42:04.904012+0000) 2024-03-04T13:42:05.898 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:42:05.903+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:42:05.904302+0000) 2024-03-04T13:42:06.899 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:42:06.903+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:42:06.904598+0000) 2024-03-04T13:42:07.900 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:42:07.903+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:42:07.904944+0000) 2024-03-04T13:42:08.899 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:42:08.904+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:42:08.905256+0000) 2024-03-04T13:42:09.900 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:42:09.904+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:42:09.905550+0000) 2024-03-04T13:42:10.900 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:42:10.904+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:42:10.905885+0000) 2024-03-04T13:42:11.900 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:42:11.905+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:42:11.906137+0000) 2024-03-04T13:42:12.901 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:42:12.905+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:42:12.906382+0000) 2024-03-04T13:42:13.265 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:42:13.273 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:42:13.901 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:42:13.905+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:42:13.906622+0000) 2024-03-04T13:42:14.901 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:42:14.905+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:42:14.906936+0000) 2024-03-04T13:42:15.901 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:42:15.906+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:42:15.907195+0000) 2024-03-04T13:42:16.902 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:42:16.906+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:42:16.907406+0000) 2024-03-04T13:42:17.902 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:42:17.906+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:42:17.907636+0000) 2024-03-04T13:42:18.902 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:42:18.907+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:42:18.907918+0000) 2024-03-04T13:42:19.902 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:42:19.907+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:42:19.908155+0000) 2024-03-04T13:42:20.903 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:42:20.907+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:42:20.908431+0000) 2024-03-04T13:42:21.903 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:42:21.907+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:42:21.908686+0000) 2024-03-04T13:42:22.903 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:42:22.908+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:42:22.908936+0000) 2024-03-04T13:42:23.904 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:42:23.908+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:42:23.909166+0000) 2024-03-04T13:42:24.904 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:42:24.908+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:42:24.909409+0000) 2024-03-04T13:42:25.904 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:42:25.908+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:42:25.909602+0000) 2024-03-04T13:42:26.904 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:42:26.909+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:42:26.909877+0000) 2024-03-04T13:42:27.905 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:42:27.909+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:42:27.910116+0000) 2024-03-04T13:42:28.905 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:42:28.909+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:42:28.910310+0000) 2024-03-04T13:42:29.905 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:42:29.909+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:42:29.910549+0000) 2024-03-04T13:42:30.906 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:42:30.910+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:42:30.910790+0000) 2024-03-04T13:42:31.906 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:42:31.910+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:42:31.911085+0000) 2024-03-04T13:42:32.906 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:42:32.910+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:42:32.911355+0000) 2024-03-04T13:42:33.906 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:42:33.910+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:42:33.911604+0000) 2024-03-04T13:42:34.906 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:42:34.911+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:42:34.911898+0000) 2024-03-04T13:42:35.907 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:42:35.911+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:42:35.912158+0000) 2024-03-04T13:42:36.908 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:42:36.911+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:42:36.912411+0000) 2024-03-04T13:42:37.907 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:42:37.912+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:42:37.912662+0000) 2024-03-04T13:42:38.907 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:42:38.912+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:42:38.912923+0000) 2024-03-04T13:42:39.908 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:42:39.912+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:42:39.913170+0000) 2024-03-04T13:42:40.908 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:42:40.912+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:42:40.913427+0000) 2024-03-04T13:42:41.908 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:42:41.913+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:42:41.913682+0000) 2024-03-04T13:42:42.908 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:42:42.913+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:42:42.913944+0000) 2024-03-04T13:42:43.404 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:42:43.412 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:42:43.909 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:42:43.913+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:42:43.914206+0000) 2024-03-04T13:42:44.909 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:42:44.913+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:42:44.914475+0000) 2024-03-04T13:42:45.909 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:42:45.914+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:42:45.914735+0000) 2024-03-04T13:42:46.910 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:42:46.914+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:42:46.914998+0000) 2024-03-04T13:42:47.910 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:42:47.914+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:42:47.915247+0000) 2024-03-04T13:42:48.910 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:42:48.915+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:42:48.915512+0000) 2024-03-04T13:42:49.910 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:42:49.915+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:42:49.915766+0000) 2024-03-04T13:42:50.911 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:42:50.915+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:42:50.916042+0000) 2024-03-04T13:42:51.911 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:42:51.915+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:42:51.916320+0000) 2024-03-04T13:42:52.911 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:42:52.916+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:42:52.916564+0000) 2024-03-04T13:42:53.912 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:42:53.916+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:42:53.916802+0000) 2024-03-04T13:42:54.912 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:42:54.916+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:42:54.917093+0000) 2024-03-04T13:42:55.912 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:42:55.917+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:42:55.917327+0000) 2024-03-04T13:42:56.912 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:42:56.917+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:42:56.917565+0000) 2024-03-04T13:42:57.913 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:42:57.917+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:42:57.917902+0000) 2024-03-04T13:42:58.913 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:42:58.917+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:42:58.918105+0000) 2024-03-04T13:42:59.913 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:42:59.917+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:42:59.918284+0000) 2024-03-04T13:43:00.913 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:43:00.918+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:43:00.918529+0000) 2024-03-04T13:43:01.914 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:43:01.918+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:43:01.918777+0000) 2024-03-04T13:43:02.914 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:43:02.918+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:43:02.919029+0000) 2024-03-04T13:43:03.914 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:43:03.919+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:43:03.919278+0000) 2024-03-04T13:43:04.914 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:43:04.919+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:43:04.919529+0000) 2024-03-04T13:43:05.915 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:43:05.919+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:43:05.919787+0000) 2024-03-04T13:43:06.915 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:43:06.919+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:43:06.920048+0000) 2024-03-04T13:43:07.915 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:43:07.920+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:43:07.920324+0000) 2024-03-04T13:43:08.916 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:43:08.919+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:43:08.920586+0000) 2024-03-04T13:43:09.916 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:43:09.919+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:43:09.920885+0000) 2024-03-04T13:43:10.916 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:43:10.920+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:43:10.921157+0000) 2024-03-04T13:43:11.916 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:43:11.920+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:43:11.921409+0000) 2024-03-04T13:43:12.917 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:43:12.919+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:43:12.921606+0000) 2024-03-04T13:43:13.543 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:43:13.551 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:43:13.917 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:43:13.920+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:43:13.921932+0000) 2024-03-04T13:43:14.917 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:43:14.921+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:43:14.922187+0000) 2024-03-04T13:43:15.917 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:43:15.921+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:43:15.922441+0000) 2024-03-04T13:43:16.918 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:43:16.921+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:43:16.922695+0000) 2024-03-04T13:43:17.918 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:43:17.921+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:43:17.922948+0000) 2024-03-04T13:43:18.918 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:43:18.922+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:43:18.923214+0000) 2024-03-04T13:43:19.918 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:43:19.922+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:43:19.923465+0000) 2024-03-04T13:43:20.919 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:43:20.922+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:43:20.923731+0000) 2024-03-04T13:43:21.919 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:43:21.923+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:43:21.923965+0000) 2024-03-04T13:43:22.919 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:43:22.923+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:43:22.924209+0000) 2024-03-04T13:43:23.920 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:43:23.923+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:43:23.924453+0000) 2024-03-04T13:43:24.920 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:43:24.923+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:43:24.924662+0000) 2024-03-04T13:43:25.920 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:43:25.924+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:43:25.924940+0000) 2024-03-04T13:43:26.920 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:43:26.924+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:43:26.925191+0000) 2024-03-04T13:43:27.920 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:43:27.924+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:43:27.925425+0000) 2024-03-04T13:43:28.921 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:43:28.924+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:43:28.925671+0000) 2024-03-04T13:43:29.921 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:43:29.925+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:43:29.925931+0000) 2024-03-04T13:43:30.921 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:43:30.925+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:43:30.926124+0000) 2024-03-04T13:43:31.921 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:43:31.925+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:43:31.926360+0000) 2024-03-04T13:43:32.922 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:43:32.925+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:43:32.926597+0000) 2024-03-04T13:43:33.922 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:43:33.926+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:43:33.926866+0000) 2024-03-04T13:43:34.922 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:43:34.926+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:43:34.927117+0000) 2024-03-04T13:43:35.923 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:43:35.926+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:43:35.927353+0000) 2024-03-04T13:43:36.923 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:43:36.926+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:43:36.927589+0000) 2024-03-04T13:43:37.923 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:43:37.927+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:43:37.927861+0000) 2024-03-04T13:43:38.923 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:43:38.927+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:43:38.928139+0000) 2024-03-04T13:43:39.924 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:43:39.927+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:43:39.928380+0000) 2024-03-04T13:43:40.924 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:43:40.928+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:43:40.928619+0000) 2024-03-04T13:43:41.924 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:43:41.928+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:43:41.928896+0000) 2024-03-04T13:43:42.924 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:43:42.928+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:43:42.929133+0000) 2024-03-04T13:43:43.683 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:43:43.691 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:43:43.925 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:43:43.928+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:43:43.929380+0000) 2024-03-04T13:43:44.925 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:43:44.929+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:43:44.929621+0000) 2024-03-04T13:43:45.925 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:43:45.929+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:43:45.929937+0000) 2024-03-04T13:43:46.926 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:43:46.929+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:43:46.930175+0000) 2024-03-04T13:43:47.926 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:43:47.929+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:43:47.930409+0000) 2024-03-04T13:43:48.926 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:43:48.929+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:43:48.930689+0000) 2024-03-04T13:43:49.926 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:43:49.930+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:43:49.930976+0000) 2024-03-04T13:43:50.927 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:43:50.930+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:43:50.931217+0000) 2024-03-04T13:43:51.927 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:43:51.931+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:43:51.931494+0000) 2024-03-04T13:43:52.927 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:43:52.931+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:43:52.931734+0000) 2024-03-04T13:43:53.927 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:43:53.931+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:43:53.931972+0000) 2024-03-04T13:43:54.928 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:43:54.931+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:43:54.932156+0000) 2024-03-04T13:43:55.928 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:43:55.931+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:43:55.932336+0000) 2024-03-04T13:43:56.928 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:43:56.932+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:43:56.932578+0000) 2024-03-04T13:43:57.928 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:43:57.932+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:43:57.932816+0000) 2024-03-04T13:43:58.928 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:43:58.932+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:43:58.933053+0000) 2024-03-04T13:43:59.929 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:43:59.932+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:43:59.933286+0000) 2024-03-04T13:44:00.929 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:44:00.932+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:44:00.933543+0000) 2024-03-04T13:44:01.929 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:44:01.933+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:44:01.933789+0000) 2024-03-04T13:44:02.929 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:44:02.933+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:44:02.934036+0000) 2024-03-04T13:44:03.930 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:44:03.934+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:44:03.934276+0000) 2024-03-04T13:44:04.930 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:44:04.934+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:44:04.934520+0000) 2024-03-04T13:44:05.930 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:44:05.934+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:44:05.934797+0000) 2024-03-04T13:44:06.931 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:44:06.934+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:44:06.935080+0000) 2024-03-04T13:44:07.931 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:44:07.935+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:44:07.935326+0000) 2024-03-04T13:44:08.931 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:44:08.935+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:44:08.935581+0000) 2024-03-04T13:44:09.932 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:44:09.935+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:44:09.936036+0000) 2024-03-04T13:44:10.932 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:44:10.936+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:44:10.936331+0000) 2024-03-04T13:44:11.932 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:44:11.935+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:44:11.936585+0000) 2024-03-04T13:44:12.933 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:44:12.935+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:44:12.936867+0000) 2024-03-04T13:44:13.836 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:44:13.848 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:44:13.933 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:44:13.935+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:44:13.937099+0000) 2024-03-04T13:44:14.933 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:44:14.936+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:44:14.937340+0000) 2024-03-04T13:44:15.933 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:44:15.936+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:44:15.937574+0000) 2024-03-04T13:44:16.934 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:44:16.936+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:44:16.938210+0000) 2024-03-04T13:44:17.934 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:44:17.937+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:44:17.938515+0000) 2024-03-04T13:44:18.935 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:44:18.937+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:44:18.938883+0000) 2024-03-04T13:44:19.935 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:44:19.938+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:44:19.939169+0000) 2024-03-04T13:44:20.935 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:44:20.938+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:44:20.939447+0000) 2024-03-04T13:44:21.935 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:44:21.938+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:44:21.939743+0000) 2024-03-04T13:44:22.936 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:44:22.939+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:44:22.940036+0000) 2024-03-04T13:44:23.936 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:44:23.939+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:44:23.940268+0000) 2024-03-04T13:44:24.936 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:44:24.939+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:44:24.940560+0000) 2024-03-04T13:44:25.937 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:44:25.940+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:44:25.940889+0000) 2024-03-04T13:44:26.937 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:44:26.940+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:44:26.941181+0000) 2024-03-04T13:44:27.937 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:44:27.940+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:44:27.941454+0000) 2024-03-04T13:44:28.938 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:44:28.940+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:44:28.941754+0000) 2024-03-04T13:44:29.938 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:44:29.941+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:44:29.942046+0000) 2024-03-04T13:44:30.938 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:44:30.941+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:44:30.942334+0000) 2024-03-04T13:44:31.939 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:44:31.941+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:44:31.942624+0000) 2024-03-04T13:44:32.939 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:44:32.942+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:44:32.942961+0000) 2024-03-04T13:44:33.939 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:44:33.942+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:44:33.943238+0000) 2024-03-04T13:44:34.939 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:44:34.942+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:44:34.943450+0000) 2024-03-04T13:44:35.940 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:44:35.942+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:44:35.943723+0000) 2024-03-04T13:44:36.940 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:44:36.943+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:44:36.944008+0000) 2024-03-04T13:44:37.940 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:44:37.943+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:44:37.944297+0000) 2024-03-04T13:44:38.941 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:44:38.943+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:44:38.944605+0000) 2024-03-04T13:44:39.941 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:44:39.944+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:44:39.944876+0000) 2024-03-04T13:44:40.941 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:44:40.944+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:44:40.945179+0000) 2024-03-04T13:44:41.941 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:44:41.944+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:44:41.945461+0000) 2024-03-04T13:44:42.942 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:44:42.945+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:44:42.945743+0000) 2024-03-04T13:44:43.942 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:44:43.945+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:44:43.946090+0000) 2024-03-04T13:44:44.049 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:44:44.061 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:44:44.942 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:44:44.945+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:44:44.946377+0000) 2024-03-04T13:44:45.943 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:44:45.945+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:44:45.946656+0000) 2024-03-04T13:44:46.943 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:44:46.946+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:44:46.946978+0000) 2024-03-04T13:44:47.943 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:44:47.946+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:44:47.947204+0000) 2024-03-04T13:44:48.943 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:44:48.946+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:44:48.947496+0000) 2024-03-04T13:44:49.944 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:44:49.946+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:44:49.947778+0000) 2024-03-04T13:44:50.944 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:44:50.947+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:44:50.948067+0000) 2024-03-04T13:44:51.944 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:44:51.947+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:44:51.948358+0000) 2024-03-04T13:44:52.945 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:44:52.947+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:44:52.948583+0000) 2024-03-04T13:44:53.945 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:44:53.948+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:44:53.948943+0000) 2024-03-04T13:44:54.945 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:44:54.948+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:44:54.949158+0000) 2024-03-04T13:44:55.945 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:44:55.948+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:44:55.949433+0000) 2024-03-04T13:44:56.946 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:44:56.948+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:44:56.949681+0000) 2024-03-04T13:44:57.946 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:44:57.949+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:44:57.949987+0000) 2024-03-04T13:44:58.946 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:44:58.949+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:44:58.950250+0000) 2024-03-04T13:44:59.946 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:44:59.949+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:44:59.950500+0000) 2024-03-04T13:45:00.947 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:45:00.949+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:45:00.950762+0000) 2024-03-04T13:45:01.947 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:45:01.950+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:45:01.950998+0000) 2024-03-04T13:45:02.947 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:45:02.950+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:45:02.951191+0000) 2024-03-04T13:45:03.947 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:45:03.950+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:45:03.951445+0000) 2024-03-04T13:45:04.948 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:45:04.951+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:45:04.951692+0000) 2024-03-04T13:45:05.948 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:45:05.951+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:45:05.951955+0000) 2024-03-04T13:45:06.948 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:45:06.951+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:45:06.952158+0000) 2024-03-04T13:45:07.949 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:45:07.951+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:45:07.952415+0000) 2024-03-04T13:45:08.949 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:45:08.952+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:45:08.952661+0000) 2024-03-04T13:45:09.949 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:45:09.952+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:45:09.952938+0000) 2024-03-04T13:45:10.949 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:45:10.952+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:45:10.953186+0000) 2024-03-04T13:45:11.950 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:45:11.953+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:45:11.953431+0000) 2024-03-04T13:45:12.950 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:45:12.952+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:45:12.953666+0000) 2024-03-04T13:45:13.950 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:45:13.953+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:45:13.953924+0000) 2024-03-04T13:45:14.196 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:45:14.204 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:45:14.950 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:45:14.953+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:45:14.954186+0000) 2024-03-04T13:45:15.951 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:45:15.952+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:45:15.954386+0000) 2024-03-04T13:45:16.951 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:45:16.953+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:45:16.954633+0000) 2024-03-04T13:45:17.951 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:45:17.953+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:45:17.954922+0000) 2024-03-04T13:45:18.951 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:45:18.954+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:45:18.955172+0000) 2024-03-04T13:45:19.952 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:45:19.954+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:45:19.955405+0000) 2024-03-04T13:45:20.952 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:45:20.954+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:45:20.955656+0000) 2024-03-04T13:45:21.952 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:45:21.954+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:45:21.955893+0000) 2024-03-04T13:45:22.952 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:45:22.955+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:45:22.956140+0000) 2024-03-04T13:45:23.953 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:45:23.955+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:45:23.956378+0000) 2024-03-04T13:45:24.953 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:45:24.955+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:45:24.956616+0000) 2024-03-04T13:45:25.953 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:45:25.956+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:45:25.956932+0000) 2024-03-04T13:45:26.953 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:45:26.956+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:45:26.957183+0000) 2024-03-04T13:45:27.954 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:45:27.956+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:45:27.957420+0000) 2024-03-04T13:45:28.954 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:45:28.956+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:45:28.957682+0000) 2024-03-04T13:45:29.954 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:45:29.957+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:45:29.957926+0000) 2024-03-04T13:45:30.954 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:45:30.957+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:45:30.958180+0000) 2024-03-04T13:45:31.955 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:45:31.957+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:45:31.958440+0000) 2024-03-04T13:45:32.955 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:45:32.957+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:45:32.958680+0000) 2024-03-04T13:45:33.955 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:45:33.958+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:45:33.958947+0000) 2024-03-04T13:45:34.955 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:45:34.958+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:45:34.959138+0000) 2024-03-04T13:45:35.956 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:45:35.958+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:45:35.959388+0000) 2024-03-04T13:45:36.956 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:45:36.958+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:45:36.959636+0000) 2024-03-04T13:45:37.956 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:45:37.959+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:45:37.959924+0000) 2024-03-04T13:45:38.956 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:45:38.959+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:45:38.960172+0000) 2024-03-04T13:45:39.957 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:45:39.959+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:45:39.960407+0000) 2024-03-04T13:45:40.957 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:45:40.959+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:45:40.960652+0000) 2024-03-04T13:45:41.957 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:45:41.960+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:45:41.960931+0000) 2024-03-04T13:45:42.958 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:45:42.960+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:45:42.961173+0000) 2024-03-04T13:45:43.958 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:45:43.960+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:45:43.961427+0000) 2024-03-04T13:45:44.361 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:45:44.369 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:45:44.958 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:45:44.960+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:45:44.961689+0000) 2024-03-04T13:45:45.958 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:45:45.961+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:45:45.961993+0000) 2024-03-04T13:45:46.959 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:45:46.961+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:45:46.962295+0000) 2024-03-04T13:45:47.959 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:45:47.961+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:45:47.962582+0000) 2024-03-04T13:45:48.959 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:45:48.962+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:45:48.962938+0000) 2024-03-04T13:45:49.960 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:45:49.962+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:45:49.963262+0000) 2024-03-04T13:45:50.960 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:45:50.962+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:45:50.963562+0000) 2024-03-04T13:45:51.960 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:45:51.963+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:45:51.963918+0000) 2024-03-04T13:45:52.961 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:45:52.963+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:45:52.964242+0000) 2024-03-04T13:45:53.961 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:45:53.963+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:45:53.964532+0000) 2024-03-04T13:45:54.961 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:45:54.964+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:45:54.964871+0000) 2024-03-04T13:45:55.962 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:45:55.964+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:45:55.965178+0000) 2024-03-04T13:45:56.962 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:45:56.964+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:45:56.965483+0000) 2024-03-04T13:45:57.962 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:45:57.965+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:45:57.965777+0000) 2024-03-04T13:45:58.963 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:45:58.965+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:45:58.966109+0000) 2024-03-04T13:45:59.963 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:45:59.965+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:45:59.966403+0000) 2024-03-04T13:46:00.963 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:46:00.965+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:46:00.966633+0000) 2024-03-04T13:46:01.963 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:46:01.966+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:46:01.966975+0000) 2024-03-04T13:46:02.964 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:46:02.966+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:46:02.967222+0000) 2024-03-04T13:46:03.964 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:46:03.967+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:46:03.967535+0000) 2024-03-04T13:46:04.965 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:46:04.967+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:46:04.967885+0000) 2024-03-04T13:46:05.965 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:46:05.967+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:46:05.968188+0000) 2024-03-04T13:46:06.965 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:46:06.968+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:46:06.968490+0000) 2024-03-04T13:46:07.966 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:46:07.967+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:46:07.968775+0000) 2024-03-04T13:46:08.966 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:46:08.968+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:46:08.969135+0000) 2024-03-04T13:46:09.966 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:46:09.969+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:46:09.969444+0000) 2024-03-04T13:46:10.966 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:46:10.969+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:46:10.969734+0000) 2024-03-04T13:46:11.968 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:46:11.969+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:46:11.970033+0000) 2024-03-04T13:46:12.967 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:46:12.970+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:46:12.970323+0000) 2024-03-04T13:46:13.967 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:46:13.969+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:46:13.970632+0000) 2024-03-04T13:46:14.591 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:46:14.600 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:46:14.967 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:46:14.970+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:46:14.970970+0000) 2024-03-04T13:46:15.968 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:46:15.971+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:46:15.971262+0000) 2024-03-04T13:46:16.968 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:46:16.971+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:46:16.971556+0000) 2024-03-04T13:46:17.969 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:46:17.970+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:46:17.971889+0000) 2024-03-04T13:46:18.969 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:46:18.971+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:46:18.972198+0000) 2024-03-04T13:46:19.969 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:46:19.971+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:46:19.972477+0000) 2024-03-04T13:46:20.970 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:46:20.971+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:46:20.972777+0000) 2024-03-04T13:46:21.970 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:46:21.972+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:46:21.973111+0000) 2024-03-04T13:46:22.970 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:46:22.971+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:46:22.973399+0000) 2024-03-04T13:46:23.971 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:46:23.972+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:46:23.973691+0000) 2024-03-04T13:46:24.971 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:46:24.972+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:46:24.973989+0000) 2024-03-04T13:46:25.971 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:46:25.973+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:46:25.974318+0000) 2024-03-04T13:46:26.971 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:46:26.973+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:46:26.974619+0000) 2024-03-04T13:46:27.972 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:46:27.973+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:46:27.974955+0000) 2024-03-04T13:46:28.972 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:46:28.974+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:46:28.975261+0000) 2024-03-04T13:46:29.972 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:46:29.974+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:46:29.975485+0000) 2024-03-04T13:46:30.973 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:46:30.974+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:46:30.975770+0000) 2024-03-04T13:46:31.973 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:46:31.975+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:46:31.976067+0000) 2024-03-04T13:46:32.973 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:46:32.975+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:46:32.976348+0000) 2024-03-04T13:46:33.974 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:46:33.975+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:46:33.976653+0000) 2024-03-04T13:46:34.974 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:46:34.976+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:46:34.976964+0000) 2024-03-04T13:46:35.974 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:46:35.976+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:46:35.977303+0000) 2024-03-04T13:46:36.975 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:46:36.976+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:46:36.977604+0000) 2024-03-04T13:46:37.975 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:46:37.977+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:46:37.977927+0000) 2024-03-04T13:46:38.975 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:46:38.977+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:46:38.978234+0000) 2024-03-04T13:46:39.975 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:46:39.977+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:46:39.978516+0000) 2024-03-04T13:46:40.976 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:46:40.978+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:46:40.978806+0000) 2024-03-04T13:46:41.976 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:46:41.978+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:46:41.979141+0000) 2024-03-04T13:46:42.976 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:46:42.978+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:46:42.979422+0000) 2024-03-04T13:46:43.977 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:46:43.978+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:46:43.979713+0000) 2024-03-04T13:46:44.822 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:46:44.830 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:46:44.977 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:46:44.979+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:46:44.980022+0000) 2024-03-04T13:46:45.977 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:46:45.979+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:46:45.980315+0000) 2024-03-04T13:46:46.978 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:46:46.979+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:46:46.980612+0000) 2024-03-04T13:46:47.978 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:46:47.980+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:46:47.980873+0000) 2024-03-04T13:46:48.978 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:46:48.980+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:46:48.981178+0000) 2024-03-04T13:46:49.978 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:46:49.980+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:46:49.981471+0000) 2024-03-04T13:46:50.979 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:46:50.981+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:46:50.981756+0000) 2024-03-04T13:46:51.979 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:46:51.981+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:46:51.982041+0000) 2024-03-04T13:46:52.979 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:46:52.981+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:46:52.982337+0000) 2024-03-04T13:46:53.980 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:46:53.982+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:46:53.982629+0000) 2024-03-04T13:46:54.980 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:46:54.982+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:46:54.982965+0000) 2024-03-04T13:46:55.980 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:46:55.982+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:46:55.983256+0000) 2024-03-04T13:46:56.981 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:46:56.983+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:46:56.983554+0000) 2024-03-04T13:46:57.981 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:46:57.983+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:46:57.983887+0000) 2024-03-04T13:46:58.981 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:46:58.983+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:46:58.984147+0000) 2024-03-04T13:46:59.981 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:46:59.983+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:46:59.984451+0000) 2024-03-04T13:47:00.982 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:47:00.984+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:47:00.984747+0000) 2024-03-04T13:47:01.982 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:47:01.984+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:47:01.985034+0000) 2024-03-04T13:47:02.982 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:47:02.984+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:47:02.985321+0000) 2024-03-04T13:47:03.983 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:47:03.984+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:47:03.985621+0000) 2024-03-04T13:47:04.983 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:47:04.985+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:47:04.985949+0000) 2024-03-04T13:47:05.984 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:47:05.985+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:47:05.986218+0000) 2024-03-04T13:47:06.984 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:47:06.986+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:47:06.986475+0000) 2024-03-04T13:47:07.984 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:47:07.986+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:47:07.986711+0000) 2024-03-04T13:47:08.984 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:47:08.986+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:47:08.986981+0000) 2024-03-04T13:47:09.984 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:47:09.986+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:47:09.987210+0000) 2024-03-04T13:47:10.985 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:47:10.987+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:47:10.987451+0000) 2024-03-04T13:47:11.985 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:47:11.987+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:47:11.987786+0000) 2024-03-04T13:47:12.985 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:47:12.987+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:47:12.988119+0000) 2024-03-04T13:47:13.985 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:47:13.988+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:47:13.988385+0000) 2024-03-04T13:47:14.986 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:47:14.988+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:47:14.988623+0000) 2024-03-04T13:47:15.053 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:47:15.061 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:47:15.986 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:47:15.988+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:47:15.988904+0000) 2024-03-04T13:47:16.986 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:47:16.988+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:47:16.989159+0000) 2024-03-04T13:47:17.987 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:47:17.989+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:47:17.989419+0000) 2024-03-04T13:47:18.987 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:47:18.989+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:47:18.989682+0000) 2024-03-04T13:47:19.987 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:47:19.989+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:47:19.989946+0000) 2024-03-04T13:47:20.987 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:47:20.989+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:47:20.990197+0000) 2024-03-04T13:47:21.988 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:47:21.988+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:47:21.990429+0000) 2024-03-04T13:47:22.988 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:47:22.988+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:47:22.990696+0000) 2024-03-04T13:47:23.988 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:47:23.989+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:47:23.990958+0000) 2024-03-04T13:47:24.988 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:47:24.990+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:47:24.991201+0000) 2024-03-04T13:47:25.989 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:47:25.990+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:47:25.991438+0000) 2024-03-04T13:47:26.989 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:47:26.989+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:47:26.991680+0000) 2024-03-04T13:47:27.989 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:47:27.990+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:47:27.991950+0000) 2024-03-04T13:47:28.989 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:47:28.991+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:47:28.992217+0000) 2024-03-04T13:47:29.990 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:47:29.991+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:47:29.992520+0000) 2024-03-04T13:47:30.990 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:47:30.991+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:47:30.992779+0000) 2024-03-04T13:47:31.990 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:47:31.992+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:47:31.993013+0000) 2024-03-04T13:47:32.991 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:47:32.992+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:47:32.993263+0000) 2024-03-04T13:47:33.991 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:47:33.992+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:47:33.993496+0000) 2024-03-04T13:47:34.991 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:47:34.992+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:47:34.993741+0000) 2024-03-04T13:47:35.991 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:47:35.993+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:47:35.993981+0000) 2024-03-04T13:47:36.991 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:47:36.993+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:47:36.994223+0000) 2024-03-04T13:47:37.992 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:47:37.993+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:47:37.994457+0000) 2024-03-04T13:47:38.992 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:47:38.993+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:47:38.994705+0000) 2024-03-04T13:47:39.992 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:47:39.994+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:47:39.994949+0000) 2024-03-04T13:47:40.992 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:47:40.994+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:47:40.995128+0000) 2024-03-04T13:47:41.993 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:47:41.994+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:47:41.995363+0000) 2024-03-04T13:47:42.993 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:47:42.994+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:47:42.995604+0000) 2024-03-04T13:47:43.993 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:47:43.995+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:47:43.995865+0000) 2024-03-04T13:47:44.993 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:47:44.995+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:47:44.996107+0000) 2024-03-04T13:47:45.274 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:47:45.283 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:47:45.994 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:47:45.995+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:47:45.996344+0000) 2024-03-04T13:47:46.994 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:47:46.995+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:47:46.996591+0000) 2024-03-04T13:47:47.994 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:47:47.996+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:47:47.996855+0000) 2024-03-04T13:47:48.994 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:47:48.996+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:47:48.997046+0000) 2024-03-04T13:47:49.995 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:47:49.996+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:47:49.997279+0000) 2024-03-04T13:47:50.995 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:47:50.996+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:47:50.997516+0000) 2024-03-04T13:47:51.995 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:47:51.997+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:47:51.997752+0000) 2024-03-04T13:47:52.995 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:47:52.997+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:47:52.997992+0000) 2024-03-04T13:47:53.996 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:47:53.997+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:47:53.998229+0000) 2024-03-04T13:47:54.996 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:47:54.997+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:47:54.998473+0000) 2024-03-04T13:47:55.996 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:47:55.998+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:47:55.998715+0000) 2024-03-04T13:47:56.996 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:47:56.998+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:47:56.998954+0000) 2024-03-04T13:47:57.997 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:47:57.998+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:47:57.999188+0000) 2024-03-04T13:47:58.997 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:47:58.998+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:47:58.999454+0000) 2024-03-04T13:47:59.997 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:47:59.999+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:47:59.999709+0000) 2024-03-04T13:48:00.997 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:48:00.999+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:48:00.999947+0000) 2024-03-04T13:48:01.998 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:48:01.999+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:48:02.000180+0000) 2024-03-04T13:48:02.998 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:48:02.999+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:48:03.000379+0000) 2024-03-04T13:48:03.998 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:48:04.000+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:48:04.000610+0000) 2024-03-04T13:48:04.998 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:48:05.000+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:48:05.000885+0000) 2024-03-04T13:48:05.999 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:48:06.000+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:48:06.001119+0000) 2024-03-04T13:48:06.999 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:48:07.000+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:48:07.001368+0000) 2024-03-04T13:48:07.999 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:48:08.001+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:48:08.001600+0000) 2024-03-04T13:48:08.999 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:48:09.001+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:48:09.001890+0000) 2024-03-04T13:48:10.000 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:48:10.001+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:48:10.002125+0000) 2024-03-04T13:48:11.000 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:48:11.001+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:48:11.002316+0000) 2024-03-04T13:48:12.000 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:48:12.002+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:48:12.002607+0000) 2024-03-04T13:48:13.000 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:48:13.002+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:48:13.002885+0000) 2024-03-04T13:48:14.001 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:48:14.002+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:48:14.003122+0000) 2024-03-04T13:48:15.001 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:48:15.003+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:48:15.003371+0000) 2024-03-04T13:48:15.413 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:48:15.422 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:48:16.001 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:48:16.003+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:48:16.003605+0000) 2024-03-04T13:48:17.001 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:48:17.003+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:48:17.003881+0000) 2024-03-04T13:48:18.002 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:48:18.003+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:48:18.004123+0000) 2024-03-04T13:48:19.002 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:48:19.004+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:48:19.004370+0000) 2024-03-04T13:48:20.002 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:48:20.004+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:48:20.004603+0000) 2024-03-04T13:48:21.002 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:48:21.004+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:48:21.004881+0000) 2024-03-04T13:48:22.003 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:48:22.005+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:48:22.005141+0000) 2024-03-04T13:48:23.003 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:48:23.003+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:48:23.005335+0000) 2024-03-04T13:48:24.003 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:48:24.004+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:48:24.005567+0000) 2024-03-04T13:48:25.004 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:48:25.004+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:48:25.005805+0000) 2024-03-04T13:48:26.004 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:48:26.005+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:48:26.006137+0000) 2024-03-04T13:48:27.004 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:48:27.005+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:48:27.006379+0000) 2024-03-04T13:48:28.004 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:48:28.005+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:48:28.006618+0000) 2024-03-04T13:48:29.004 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:48:29.005+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:48:29.006933+0000) 2024-03-04T13:48:30.005 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:48:30.006+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:48:30.007165+0000) 2024-03-04T13:48:31.005 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:48:31.006+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:48:31.007402+0000) 2024-03-04T13:48:32.005 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:48:32.006+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:48:32.007635+0000) 2024-03-04T13:48:33.006 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:48:33.006+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:48:33.007924+0000) 2024-03-04T13:48:34.006 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:48:34.007+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:48:34.008178+0000) 2024-03-04T13:48:35.006 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:48:35.007+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:48:35.008452+0000) 2024-03-04T13:48:36.006 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:48:36.007+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:48:36.008695+0000) 2024-03-04T13:48:37.007 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:48:37.008+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:48:37.008960+0000) 2024-03-04T13:48:38.007 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:48:38.008+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:48:38.009210+0000) 2024-03-04T13:48:39.007 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:48:39.008+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:48:39.009473+0000) 2024-03-04T13:48:40.008 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:48:40.008+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:48:40.009703+0000) 2024-03-04T13:48:41.008 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:48:41.009+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:48:41.009933+0000) 2024-03-04T13:48:42.008 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:48:42.009+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:48:42.010167+0000) 2024-03-04T13:48:43.008 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:48:43.009+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:48:43.010415+0000) 2024-03-04T13:48:44.008 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:48:44.009+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:48:44.010669+0000) 2024-03-04T13:48:45.009 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:48:45.010+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:48:45.010978+0000) 2024-03-04T13:48:45.558 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:48:45.566 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:48:46.009 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:48:46.010+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:48:46.011255+0000) 2024-03-04T13:48:47.009 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:48:47.010+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:48:47.011495+0000) 2024-03-04T13:48:48.010 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:48:48.011+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:48:48.011734+0000) 2024-03-04T13:48:49.010 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:48:49.011+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:48:49.011988+0000) 2024-03-04T13:48:50.010 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:48:50.011+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:48:50.012221+0000) 2024-03-04T13:48:51.010 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:48:51.011+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:48:51.012474+0000) 2024-03-04T13:48:52.011 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:48:52.012+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:48:52.012714+0000) 2024-03-04T13:48:53.011 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:48:53.012+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:48:53.012960+0000) 2024-03-04T13:48:54.011 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:48:54.012+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:48:54.013199+0000) 2024-03-04T13:48:55.011 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:48:55.012+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:48:55.013439+0000) 2024-03-04T13:48:56.012 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:48:56.013+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:48:56.013678+0000) 2024-03-04T13:48:57.012 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:48:57.013+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:48:57.013949+0000) 2024-03-04T13:48:58.012 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:48:58.013+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:48:58.014190+0000) 2024-03-04T13:48:59.012 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:48:59.013+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:48:59.014462+0000) 2024-03-04T13:49:00.013 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:49:00.013+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:49:00.014751+0000) 2024-03-04T13:49:01.013 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:49:01.014+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:49:01.014996+0000) 2024-03-04T13:49:02.013 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:49:02.014+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:49:02.015237+0000) 2024-03-04T13:49:03.013 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:49:03.014+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:49:03.015476+0000) 2024-03-04T13:49:04.014 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:49:04.015+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:49:04.015712+0000) 2024-03-04T13:49:05.014 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:49:05.015+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:49:05.015989+0000) 2024-03-04T13:49:06.014 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:49:06.015+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:49:06.016224+0000) 2024-03-04T13:49:07.014 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:49:07.015+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:49:07.016445+0000) 2024-03-04T13:49:08.015 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:49:08.016+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:49:08.016715+0000) 2024-03-04T13:49:09.015 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:49:09.016+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:49:09.016963+0000) 2024-03-04T13:49:10.015 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:49:10.016+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:49:10.017194+0000) 2024-03-04T13:49:11.015 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:49:11.017+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:49:11.017443+0000) 2024-03-04T13:49:12.016 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:49:12.017+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:49:12.017683+0000) 2024-03-04T13:49:13.016 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:49:13.017+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:49:13.017952+0000) 2024-03-04T13:49:14.016 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:49:14.017+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:49:14.018205+0000) 2024-03-04T13:49:15.016 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:49:15.017+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:49:15.018466+0000) 2024-03-04T13:49:15.747 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:49:15.756 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:49:16.016 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:49:16.018+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:49:16.018715+0000) 2024-03-04T13:49:17.017 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:49:17.018+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:49:17.018941+0000) 2024-03-04T13:49:18.017 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:49:18.018+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:49:18.019129+0000) 2024-03-04T13:49:19.017 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:49:19.019+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:49:19.019381+0000) 2024-03-04T13:49:20.018 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:49:20.019+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:49:20.019614+0000) 2024-03-04T13:49:21.018 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:49:21.019+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:49:21.019892+0000) 2024-03-04T13:49:22.018 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:49:22.019+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:49:22.020108+0000) 2024-03-04T13:49:23.018 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:49:23.020+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:49:23.020367+0000) 2024-03-04T13:49:24.019 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:49:24.020+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:49:24.020622+0000) 2024-03-04T13:49:25.019 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:49:25.020+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:49:25.020929+0000) 2024-03-04T13:49:26.019 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:49:26.020+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:49:26.021176+0000) 2024-03-04T13:49:27.019 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:49:27.020+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:49:27.021422+0000) 2024-03-04T13:49:28.020 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:49:28.020+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:49:28.021664+0000) 2024-03-04T13:49:29.020 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:49:29.020+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:49:29.021933+0000) 2024-03-04T13:49:30.020 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:49:30.021+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:49:30.022190+0000) 2024-03-04T13:49:31.021 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:49:31.021+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:49:31.022425+0000) 2024-03-04T13:49:32.021 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:49:32.021+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:49:32.022682+0000) 2024-03-04T13:49:33.021 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:49:33.021+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:49:33.022947+0000) 2024-03-04T13:49:34.021 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:49:34.022+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:49:34.023180+0000) 2024-03-04T13:49:35.022 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:49:35.022+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:49:35.023433+0000) 2024-03-04T13:49:36.022 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:49:36.022+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:49:36.023667+0000) 2024-03-04T13:49:37.022 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:49:37.023+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:49:37.023947+0000) 2024-03-04T13:49:38.022 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:49:38.023+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:49:38.024198+0000) 2024-03-04T13:49:39.023 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:49:39.023+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:49:39.024447+0000) 2024-03-04T13:49:40.023 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:49:40.023+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:49:40.024659+0000) 2024-03-04T13:49:41.024 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:49:41.024+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:49:41.024946+0000) 2024-03-04T13:49:42.023 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:49:42.024+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:49:42.025205+0000) 2024-03-04T13:49:43.024 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:49:43.024+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:49:43.025473+0000) 2024-03-04T13:49:44.024 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:49:44.024+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:49:44.025671+0000) 2024-03-04T13:49:45.024 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:49:45.025+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:49:45.025903+0000) 2024-03-04T13:49:45.944 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:49:45.953 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:49:46.024 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:49:46.025+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:49:46.026102+0000) 2024-03-04T13:49:47.024 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:49:47.025+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:49:47.026344+0000) 2024-03-04T13:49:48.025 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:49:48.025+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:49:48.026603+0000) 2024-03-04T13:49:49.025 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:49:49.026+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:49:49.026888+0000) 2024-03-04T13:49:50.025 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:49:50.026+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:49:50.027120+0000) 2024-03-04T13:49:51.026 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:49:51.026+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:49:51.027356+0000) 2024-03-04T13:49:52.026 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:49:52.026+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:49:52.027636+0000) 2024-03-04T13:49:53.026 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:49:53.027+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:49:53.027934+0000) 2024-03-04T13:49:54.026 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:49:54.027+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:49:54.028172+0000) 2024-03-04T13:49:55.027 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:49:55.027+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:49:55.028366+0000) 2024-03-04T13:49:56.027 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:49:56.027+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:49:56.028606+0000) 2024-03-04T13:49:57.027 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:49:57.028+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:49:57.028892+0000) 2024-03-04T13:49:58.027 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:49:58.028+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:49:58.029135+0000) 2024-03-04T13:49:59.028 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:49:59.028+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:49:59.029468+0000) 2024-03-04T13:50:00.028 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:50:00.029+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:50:00.029705+0000) 2024-03-04T13:50:01.028 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:50:01.029+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:50:01.029959+0000) 2024-03-04T13:50:02.028 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:50:02.029+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:50:02.030244+0000) 2024-03-04T13:50:03.029 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:50:03.029+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:50:03.030488+0000) 2024-03-04T13:50:04.030 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:50:04.030+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:50:04.030744+0000) 2024-03-04T13:50:05.029 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:50:05.030+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:50:05.031025+0000) 2024-03-04T13:50:06.029 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:50:06.030+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:50:06.031255+0000) 2024-03-04T13:50:07.030 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:50:07.031+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:50:07.031561+0000) 2024-03-04T13:50:08.030 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:50:08.031+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:50:08.031905+0000) 2024-03-04T13:50:09.030 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:50:09.031+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:50:09.032209+0000) 2024-03-04T13:50:10.031 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:50:10.032+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:50:10.032495+0000) 2024-03-04T13:50:11.031 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:50:11.032+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:50:11.032785+0000) 2024-03-04T13:50:12.031 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:50:12.032+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:50:12.033123+0000) 2024-03-04T13:50:13.032 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:50:13.033+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:50:13.033426+0000) 2024-03-04T13:50:14.032 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:50:14.033+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:50:14.033659+0000) 2024-03-04T13:50:15.032 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:50:15.033+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:50:15.033979+0000) 2024-03-04T13:50:16.033 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:50:16.033+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:50:16.034269+0000) 2024-03-04T13:50:16.084 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:50:16.092 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:50:17.033 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:50:17.033+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:50:17.034525+0000) 2024-03-04T13:50:18.033 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:50:18.034+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:50:18.034896+0000) 2024-03-04T13:50:19.034 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:50:19.034+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:50:19.035202+0000) 2024-03-04T13:50:20.034 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:50:20.035+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:50:20.035491+0000) 2024-03-04T13:50:21.034 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:50:21.035+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:50:21.035785+0000) 2024-03-04T13:50:22.034 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:50:22.035+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:50:22.036140+0000) 2024-03-04T13:50:23.035 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:50:23.036+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:50:23.036437+0000) 2024-03-04T13:50:24.037 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:50:24.036+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:50:24.036726+0000) 2024-03-04T13:50:25.035 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:50:25.036+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:50:25.037022+0000) 2024-03-04T13:50:26.036 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:50:26.037+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:50:26.037338+0000) 2024-03-04T13:50:27.036 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:50:27.036+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:50:27.037642+0000) 2024-03-04T13:50:28.036 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:50:28.036+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:50:28.037975+0000) 2024-03-04T13:50:29.037 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:50:29.037+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:50:29.038286+0000) 2024-03-04T13:50:30.037 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:50:30.036+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:50:30.038472+0000) 2024-03-04T13:50:31.037 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:50:31.037+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:50:31.038707+0000) 2024-03-04T13:50:32.037 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:50:32.037+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:50:32.038950+0000) 2024-03-04T13:50:33.038 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:50:33.038+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:50:33.039214+0000) 2024-03-04T13:50:34.038 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:50:34.038+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:50:34.039452+0000) 2024-03-04T13:50:35.038 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:50:35.038+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:50:35.039690+0000) 2024-03-04T13:50:36.038 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:50:36.038+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:50:36.039917+0000) 2024-03-04T13:50:37.039 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:50:37.038+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:50:37.040163+0000) 2024-03-04T13:50:38.039 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:50:38.039+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:50:38.040432+0000) 2024-03-04T13:50:39.039 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:50:39.039+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:50:39.040681+0000) 2024-03-04T13:50:40.039 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:50:40.040+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:50:40.040947+0000) 2024-03-04T13:50:41.040 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:50:41.040+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:50:41.041187+0000) 2024-03-04T13:50:42.040 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:50:42.040+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:50:42.041436+0000) 2024-03-04T13:50:43.040 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:50:43.040+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:50:43.041686+0000) 2024-03-04T13:50:44.040 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:50:44.041+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:50:44.041920+0000) 2024-03-04T13:50:45.041 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:50:45.041+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:50:45.042152+0000) 2024-03-04T13:50:46.041 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:50:46.041+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:50:46.042343+0000) 2024-03-04T13:50:46.221 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:50:46.230 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:50:47.041 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:50:47.041+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:50:47.042645+0000) 2024-03-04T13:50:48.041 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:50:48.042+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:50:48.042933+0000) 2024-03-04T13:50:49.042 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:50:49.042+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:50:49.043180+0000) 2024-03-04T13:50:50.042 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:50:50.042+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:50:50.043415+0000) 2024-03-04T13:50:51.042 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:50:51.042+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:50:51.043665+0000) 2024-03-04T13:50:52.042 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:50:52.043+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:50:52.043946+0000) 2024-03-04T13:50:53.043 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:50:53.043+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:50:53.044192+0000) 2024-03-04T13:50:54.043 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:50:54.043+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:50:54.044433+0000) 2024-03-04T13:50:55.043 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:50:55.044+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:50:55.044691+0000) 2024-03-04T13:50:56.043 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:50:56.044+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:50:56.044954+0000) 2024-03-04T13:50:57.044 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:50:57.044+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:50:57.045189+0000) 2024-03-04T13:50:58.044 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:50:58.044+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:50:58.045460+0000) 2024-03-04T13:50:59.044 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:50:59.044+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:50:59.045752+0000) 2024-03-04T13:51:00.044 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:51:00.045+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:51:00.046010+0000) 2024-03-04T13:51:01.045 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:51:01.045+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:51:01.046250+0000) 2024-03-04T13:51:02.046 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:51:02.045+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:51:02.046497+0000) 2024-03-04T13:51:03.045 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:51:03.046+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:51:03.046743+0000) 2024-03-04T13:51:04.046 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:51:04.046+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:51:04.046981+0000) 2024-03-04T13:51:05.046 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:51:05.046+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:51:05.047222+0000) 2024-03-04T13:51:06.046 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:51:06.046+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:51:06.047456+0000) 2024-03-04T13:51:07.047 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:51:07.046+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:51:07.047725+0000) 2024-03-04T13:51:08.047 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:51:08.047+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:51:08.048008+0000) 2024-03-04T13:51:09.047 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:51:09.047+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:51:09.048248+0000) 2024-03-04T13:51:10.047 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:51:10.048+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:51:10.048498+0000) 2024-03-04T13:51:11.047 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:51:11.048+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:51:11.048738+0000) 2024-03-04T13:51:12.048 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:51:12.048+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:51:12.048978+0000) 2024-03-04T13:51:13.048 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:51:13.048+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:51:13.049223+0000) 2024-03-04T13:51:14.048 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:51:14.049+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:51:14.049461+0000) 2024-03-04T13:51:15.048 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:51:15.049+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:51:15.049705+0000) 2024-03-04T13:51:16.048 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:51:16.049+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:51:16.049934+0000) 2024-03-04T13:51:16.368 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:51:16.376 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:51:17.049 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:51:17.049+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:51:17.050190+0000) 2024-03-04T13:51:18.049 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:51:18.049+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:51:18.050388+0000) 2024-03-04T13:51:19.049 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:51:19.049+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:51:19.050627+0000) 2024-03-04T13:51:20.049 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:51:20.049+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:51:20.050801+0000) 2024-03-04T13:51:21.050 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:51:21.050+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:51:21.051019+0000) 2024-03-04T13:51:22.050 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:51:22.051+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:51:22.051262+0000) 2024-03-04T13:51:23.050 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:51:23.051+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:51:23.051510+0000) 2024-03-04T13:51:24.050 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:51:24.051+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:51:24.051748+0000) 2024-03-04T13:51:25.051 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:51:25.051+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:51:25.051989+0000) 2024-03-04T13:51:26.051 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:51:26.052+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:51:26.052242+0000) 2024-03-04T13:51:27.051 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:51:27.052+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:51:27.052500+0000) 2024-03-04T13:51:28.051 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:51:28.051+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:51:28.052739+0000) 2024-03-04T13:51:29.052 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:51:29.051+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:51:29.052984+0000) 2024-03-04T13:51:30.052 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:51:30.051+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:51:30.053226+0000) 2024-03-04T13:51:31.052 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:51:31.052+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:51:31.053464+0000) 2024-03-04T13:51:32.052 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:51:32.052+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:51:32.053694+0000) 2024-03-04T13:51:33.053 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:51:33.052+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:51:33.053941+0000) 2024-03-04T13:51:34.053 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:51:34.053+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:51:34.054173+0000) 2024-03-04T13:51:35.053 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:51:35.053+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:51:35.054448+0000) 2024-03-04T13:51:36.053 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:51:36.053+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:51:36.054700+0000) 2024-03-04T13:51:37.054 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:51:37.053+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:51:37.054939+0000) 2024-03-04T13:51:38.054 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:51:38.054+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:51:38.055141+0000) 2024-03-04T13:51:39.054 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:51:39.054+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:51:39.055409+0000) 2024-03-04T13:51:40.054 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:51:40.054+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:51:40.055640+0000) 2024-03-04T13:51:41.055 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:51:41.055+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:51:41.055930+0000) 2024-03-04T13:51:42.055 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:51:42.055+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:51:42.056163+0000) 2024-03-04T13:51:43.055 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:51:43.055+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:51:43.056345+0000) 2024-03-04T13:51:44.056 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:51:44.055+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:51:44.056538+0000) 2024-03-04T13:51:45.056 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:51:45.056+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:51:45.056818+0000) 2024-03-04T13:51:46.056 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:51:46.056+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:51:46.057116+0000) 2024-03-04T13:51:46.515 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:51:46.522 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:51:47.056 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:51:47.056+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:51:47.057364+0000) 2024-03-04T13:51:48.056 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:51:48.056+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:51:48.057598+0000) 2024-03-04T13:51:49.057 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:51:49.057+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:51:49.057886+0000) 2024-03-04T13:51:50.057 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:51:50.057+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:51:50.058128+0000) 2024-03-04T13:51:51.057 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:51:51.057+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:51:51.058369+0000) 2024-03-04T13:51:52.057 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:51:52.057+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:51:52.058602+0000) 2024-03-04T13:51:53.058 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:51:53.058+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:51:53.058875+0000) 2024-03-04T13:51:54.058 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:51:54.058+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:51:54.059113+0000) 2024-03-04T13:51:55.058 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:51:55.058+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:51:55.059351+0000) 2024-03-04T13:51:56.058 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:51:56.058+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:51:56.059598+0000) 2024-03-04T13:51:57.059 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:51:57.059+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:51:57.059890+0000) 2024-03-04T13:51:58.059 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:51:58.059+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:51:58.060145+0000) 2024-03-04T13:51:59.059 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:51:59.059+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:51:59.060388+0000) 2024-03-04T13:52:00.059 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:52:00.060+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:52:00.060624+0000) 2024-03-04T13:52:01.060 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:52:01.060+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:52:01.060914+0000) 2024-03-04T13:52:02.060 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:52:02.060+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:52:02.061152+0000) 2024-03-04T13:52:03.060 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:52:03.060+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:52:03.061390+0000) 2024-03-04T13:52:04.061 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:52:04.061+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:52:04.061639+0000) 2024-03-04T13:52:05.061 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:52:05.061+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:52:05.061942+0000) 2024-03-04T13:52:06.062 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:52:06.061+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:52:06.062192+0000) 2024-03-04T13:52:07.061 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:52:07.061+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:52:07.062474+0000) 2024-03-04T13:52:08.062 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:52:08.061+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:52:08.062735+0000) 2024-03-04T13:52:09.062 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:52:09.062+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:52:09.062967+0000) 2024-03-04T13:52:10.062 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:52:10.062+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:52:10.063212+0000) 2024-03-04T13:52:11.062 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:52:11.063+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:52:11.063452+0000) 2024-03-04T13:52:12.063 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:52:12.063+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:52:12.063689+0000) 2024-03-04T13:52:13.063 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:52:13.063+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:52:13.063935+0000) 2024-03-04T13:52:14.063 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:52:14.063+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:52:14.064171+0000) 2024-03-04T13:52:15.063 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:52:15.064+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:52:15.064408+0000) 2024-03-04T13:52:16.064 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:52:16.064+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:52:16.064643+0000) 2024-03-04T13:52:16.745 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:52:16.753 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:52:17.064 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:52:17.064+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:52:17.064951+0000) 2024-03-04T13:52:18.064 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:52:18.064+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:52:18.065179+0000) 2024-03-04T13:52:19.064 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:52:19.065+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:52:19.065434+0000) 2024-03-04T13:52:20.065 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:52:20.065+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:52:20.065680+0000) 2024-03-04T13:52:21.065 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:52:21.065+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:52:21.065967+0000) 2024-03-04T13:52:22.065 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:52:22.065+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:52:22.066203+0000) 2024-03-04T13:52:23.068 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:52:23.066+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:52:23.066533+0000) 2024-03-04T13:52:24.066 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:52:24.065+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:52:24.066716+0000) 2024-03-04T13:52:25.066 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:52:25.066+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:52:25.066945+0000) 2024-03-04T13:52:26.066 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:52:26.067+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:52:26.067210+0000) 2024-03-04T13:52:27.066 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:52:27.066+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:52:27.067468+0000) 2024-03-04T13:52:28.067 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:52:28.066+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:52:28.067724+0000) 2024-03-04T13:52:29.067 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:52:29.067+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:52:29.067974+0000) 2024-03-04T13:52:30.068 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:52:30.067+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:52:30.068208+0000) 2024-03-04T13:52:31.067 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:52:31.067+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:52:31.068453+0000) 2024-03-04T13:52:32.068 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:52:32.067+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:52:32.068707+0000) 2024-03-04T13:52:33.068 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:52:33.067+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:52:33.068963+0000) 2024-03-04T13:52:34.068 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:52:34.068+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:52:34.069207+0000) 2024-03-04T13:52:35.069 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:52:35.068+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:52:35.069446+0000) 2024-03-04T13:52:36.069 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:52:36.068+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:52:36.069681+0000) 2024-03-04T13:52:37.069 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:52:37.068+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:52:37.069922+0000) 2024-03-04T13:52:38.069 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:52:38.069+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:52:38.070163+0000) 2024-03-04T13:52:39.069 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:52:39.069+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:52:39.070415+0000) 2024-03-04T13:52:40.070 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:52:40.069+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:52:40.070653+0000) 2024-03-04T13:52:41.070 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:52:41.070+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:52:41.070936+0000) 2024-03-04T13:52:42.070 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:52:42.070+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:52:42.071169+0000) 2024-03-04T13:52:43.070 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:52:43.070+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:52:43.071407+0000) 2024-03-04T13:52:44.071 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:52:44.070+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:52:44.071645+0000) 2024-03-04T13:52:45.071 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:52:45.071+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:52:45.071933+0000) 2024-03-04T13:52:46.071 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:52:46.071+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:52:46.072175+0000) 2024-03-04T13:52:46.950 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:52:46.958 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:52:47.071 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:52:47.071+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:52:47.072387+0000) 2024-03-04T13:52:48.072 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:52:48.071+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:52:48.072611+0000) 2024-03-04T13:52:49.072 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:52:49.072+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:52:49.072923+0000) 2024-03-04T13:52:50.072 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:52:50.072+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:52:50.073157+0000) 2024-03-04T13:52:51.072 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:52:51.072+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:52:51.073403+0000) 2024-03-04T13:52:52.073 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:52:52.072+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:52:52.073637+0000) 2024-03-04T13:52:53.073 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:52:53.073+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:52:53.073939+0000) 2024-03-04T13:52:54.073 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:52:54.073+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:52:54.074195+0000) 2024-03-04T13:52:55.074 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:52:55.073+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:52:55.074438+0000) 2024-03-04T13:52:56.074 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:52:56.073+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:52:56.074686+0000) 2024-03-04T13:52:57.074 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:52:57.074+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:52:57.074945+0000) 2024-03-04T13:52:58.074 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:52:58.074+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:52:58.075204+0000) 2024-03-04T13:52:59.075 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:52:59.074+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:52:59.075456+0000) 2024-03-04T13:53:00.075 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:53:00.075+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:53:00.075695+0000) 2024-03-04T13:53:01.075 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:53:01.075+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:53:01.075933+0000) 2024-03-04T13:53:02.075 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:53:02.075+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:53:02.076167+0000) 2024-03-04T13:53:03.075 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:53:03.075+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:53:03.076413+0000) 2024-03-04T13:53:04.076 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:53:04.076+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:53:04.076648+0000) 2024-03-04T13:53:05.076 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:53:05.076+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:53:05.076925+0000) 2024-03-04T13:53:06.076 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:53:06.076+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:53:06.077162+0000) 2024-03-04T13:53:07.077 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:53:07.076+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:53:07.077409+0000) 2024-03-04T13:53:08.077 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:53:08.077+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:53:08.077647+0000) 2024-03-04T13:53:09.077 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:53:09.077+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:53:09.077948+0000) 2024-03-04T13:53:10.077 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:53:10.077+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:53:10.078184+0000) 2024-03-04T13:53:11.078 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:53:11.078+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:53:11.078424+0000) 2024-03-04T13:53:12.078 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:53:12.078+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:53:12.078658+0000) 2024-03-04T13:53:13.078 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:53:13.078+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:53:13.078943+0000) 2024-03-04T13:53:14.078 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:53:14.078+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:53:14.079172+0000) 2024-03-04T13:53:15.079 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:53:15.079+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:53:15.079409+0000) 2024-03-04T13:53:16.079 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:53:16.079+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:53:16.079652+0000) 2024-03-04T13:53:17.079 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:53:17.079+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:53:17.079926+0000) 2024-03-04T13:53:17.172 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:53:17.180 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:53:18.079 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:53:18.079+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:53:18.080171+0000) 2024-03-04T13:53:19.080 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:53:19.080+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:53:19.080425+0000) 2024-03-04T13:53:20.080 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:53:20.080+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:53:20.080680+0000) 2024-03-04T13:53:21.080 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:53:21.080+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:53:21.080925+0000) 2024-03-04T13:53:22.080 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:53:22.080+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:53:22.081120+0000) 2024-03-04T13:53:23.081 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:53:23.081+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:53:23.081360+0000) 2024-03-04T13:53:24.081 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:53:24.081+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:53:24.081595+0000) 2024-03-04T13:53:25.081 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:53:25.081+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:53:25.081879+0000) 2024-03-04T13:53:26.081 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:53:26.081+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:53:26.082119+0000) 2024-03-04T13:53:27.082 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:53:27.082+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:53:27.082381+0000) 2024-03-04T13:53:28.082 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:53:28.082+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:53:28.082631+0000) 2024-03-04T13:53:29.082 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:53:29.081+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:53:29.082934+0000) 2024-03-04T13:53:30.082 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:53:30.082+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:53:30.083166+0000) 2024-03-04T13:53:31.083 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:53:31.082+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:53:31.083407+0000) 2024-03-04T13:53:32.084 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:53:32.082+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:53:32.083649+0000) 2024-03-04T13:53:33.083 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:53:33.082+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:53:33.083955+0000) 2024-03-04T13:53:34.084 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:53:34.083+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:53:34.084190+0000) 2024-03-04T13:53:35.084 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:53:35.083+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:53:35.084427+0000) 2024-03-04T13:53:36.084 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:53:36.083+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:53:36.084686+0000) 2024-03-04T13:53:37.084 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:53:37.083+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:53:37.084939+0000) 2024-03-04T13:53:38.085 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:53:38.084+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:53:38.085184+0000) 2024-03-04T13:53:39.085 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:53:39.084+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:53:39.085469+0000) 2024-03-04T13:53:40.086 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:53:40.084+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:53:40.085707+0000) 2024-03-04T13:53:41.085 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:53:41.085+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:53:41.085954+0000) 2024-03-04T13:53:42.085 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:53:42.085+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:53:42.086188+0000) 2024-03-04T13:53:43.086 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:53:43.085+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:53:43.086392+0000) 2024-03-04T13:53:44.086 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:53:44.085+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:53:44.086632+0000) 2024-03-04T13:53:45.086 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:53:45.086+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:53:45.086913+0000) 2024-03-04T13:53:46.086 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:53:46.086+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:53:46.087149+0000) 2024-03-04T13:53:47.087 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:53:47.086+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:53:47.087387+0000) 2024-03-04T13:53:47.386 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:53:47.394 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:53:48.087 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:53:48.086+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:53:48.087631+0000) 2024-03-04T13:53:49.087 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:53:49.087+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:53:49.087931+0000) 2024-03-04T13:53:50.087 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:53:50.087+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:53:50.088170+0000) 2024-03-04T13:53:51.089 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:53:51.087+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:53:51.088410+0000) 2024-03-04T13:53:52.088 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:53:52.087+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:53:52.088639+0000) 2024-03-04T13:53:53.088 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:53:53.088+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:53:53.088928+0000) 2024-03-04T13:53:54.089 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:53:54.088+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:53:54.089181+0000) 2024-03-04T13:53:55.089 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:53:55.088+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:53:55.089418+0000) 2024-03-04T13:53:56.089 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:53:56.089+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:53:56.089652+0000) 2024-03-04T13:53:57.089 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:53:57.089+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:53:57.089954+0000) 2024-03-04T13:53:58.090 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:53:58.089+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:53:58.090215+0000) 2024-03-04T13:53:59.090 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:53:59.089+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:53:59.090469+0000) 2024-03-04T13:54:00.090 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:54:00.090+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:54:00.090694+0000) 2024-03-04T13:54:01.090 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:54:01.090+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:54:01.090932+0000) 2024-03-04T13:54:02.091 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:54:02.090+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:54:02.091165+0000) 2024-03-04T13:54:03.091 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:54:03.090+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:54:03.091411+0000) 2024-03-04T13:54:04.091 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:54:04.091+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:54:04.091656+0000) 2024-03-04T13:54:05.091 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:54:05.091+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:54:05.091947+0000) 2024-03-04T13:54:06.092 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:54:06.091+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:54:06.092185+0000) 2024-03-04T13:54:07.092 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:54:07.091+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:54:07.092423+0000) 2024-03-04T13:54:08.092 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:54:08.092+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:54:08.092668+0000) 2024-03-04T13:54:09.092 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:54:09.092+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:54:09.092927+0000) 2024-03-04T13:54:10.093 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:54:10.092+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:54:10.093231+0000) 2024-03-04T13:54:11.093 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:54:11.093+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:54:11.093470+0000) 2024-03-04T13:54:12.093 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:54:12.093+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:54:12.093703+0000) 2024-03-04T13:54:13.093 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:54:13.093+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:54:13.093947+0000) 2024-03-04T13:54:14.094 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:54:14.093+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:54:14.094193+0000) 2024-03-04T13:54:15.094 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:54:15.094+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:54:15.094433+0000) 2024-03-04T13:54:16.094 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:54:16.094+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:54:16.094617+0000) 2024-03-04T13:54:17.094 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:54:17.094+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:54:17.094923+0000) 2024-03-04T13:54:17.523 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:54:17.531 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:54:18.095 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:54:18.094+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:54:18.095194+0000) 2024-03-04T13:54:19.095 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:54:19.095+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:54:19.095473+0000) 2024-03-04T13:54:20.095 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:54:20.095+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:54:20.095727+0000) 2024-03-04T13:54:21.096 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:54:21.095+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:54:21.095988+0000) 2024-03-04T13:54:22.096 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:54:22.096+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:54:22.096236+0000) 2024-03-04T13:54:23.096 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:54:23.096+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:54:23.096499+0000) 2024-03-04T13:54:24.096 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:54:24.096+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:54:24.096754+0000) 2024-03-04T13:54:25.097 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:54:25.095+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:54:25.097010+0000) 2024-03-04T13:54:26.097 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:54:26.096+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:54:26.097286+0000) 2024-03-04T13:54:27.097 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:54:27.096+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:54:27.097563+0000) 2024-03-04T13:54:28.097 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:54:28.096+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:54:28.097878+0000) 2024-03-04T13:54:29.098 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:54:29.097+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:54:29.098149+0000) 2024-03-04T13:54:30.098 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:54:30.097+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:54:30.098400+0000) 2024-03-04T13:54:31.098 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:54:31.097+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:54:31.098674+0000) 2024-03-04T13:54:32.099 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:54:32.097+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:54:32.098921+0000) 2024-03-04T13:54:33.099 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:54:33.098+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:54:33.099183+0000) 2024-03-04T13:54:34.100 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:54:34.098+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:54:34.099448+0000) 2024-03-04T13:54:35.100 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:54:35.098+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:54:35.099709+0000) 2024-03-04T13:54:36.099 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:54:36.099+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:54:36.099960+0000) 2024-03-04T13:54:37.100 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:54:37.099+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:54:37.100214+0000) 2024-03-04T13:54:38.100 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:54:38.099+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:54:38.100473+0000) 2024-03-04T13:54:39.100 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:54:39.099+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:54:39.100751+0000) 2024-03-04T13:54:40.101 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:54:40.100+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:54:40.100996+0000) 2024-03-04T13:54:41.101 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:54:41.100+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:54:41.101253+0000) 2024-03-04T13:54:42.101 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:54:42.100+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:54:42.101503+0000) 2024-03-04T13:54:43.101 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:54:43.100+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:54:43.101767+0000) 2024-03-04T13:54:44.102 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:54:44.101+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:54:44.102033+0000) 2024-03-04T13:54:45.102 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:54:45.101+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:54:45.102227+0000) 2024-03-04T13:54:46.102 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:54:46.101+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:54:46.102482+0000) 2024-03-04T13:54:47.103 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:54:47.101+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:54:47.102734+0000) 2024-03-04T13:54:47.737 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:54:47.746 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:54:48.103 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:54:48.102+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:54:48.102995+0000) 2024-03-04T13:54:49.103 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:54:49.102+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:54:49.103272+0000) 2024-03-04T13:54:50.103 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:54:50.102+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:54:50.103510+0000) 2024-03-04T13:54:51.103 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:54:51.103+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:54:51.103753+0000) 2024-03-04T13:54:52.104 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:54:52.103+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:54:52.103986+0000) 2024-03-04T13:54:53.104 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:54:53.103+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:54:53.104245+0000) 2024-03-04T13:54:54.104 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:54:54.103+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:54:54.104480+0000) 2024-03-04T13:54:55.104 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:54:55.104+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:54:55.104719+0000) 2024-03-04T13:54:56.105 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:54:56.104+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:54:56.104972+0000) 2024-03-04T13:54:57.105 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:54:57.104+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:54:57.105242+0000) 2024-03-04T13:54:58.105 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:54:58.104+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:54:58.105488+0000) 2024-03-04T13:54:59.105 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:54:59.104+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:54:59.105734+0000) 2024-03-04T13:55:00.106 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:55:00.105+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:55:00.105973+0000) 2024-03-04T13:55:01.106 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:55:01.105+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:55:01.106203+0000) 2024-03-04T13:55:02.106 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:55:02.105+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:55:02.106442+0000) 2024-03-04T13:55:03.107 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:55:03.106+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:55:03.106702+0000) 2024-03-04T13:55:04.107 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:55:04.106+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:55:04.106955+0000) 2024-03-04T13:55:05.107 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:55:05.106+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:55:05.107232+0000) 2024-03-04T13:55:06.107 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:55:06.107+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:55:06.107484+0000) 2024-03-04T13:55:07.107 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:55:07.107+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:55:07.107740+0000) 2024-03-04T13:55:08.108 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:55:08.107+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:55:08.108025+0000) 2024-03-04T13:55:09.108 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:55:09.107+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:55:09.108295+0000) 2024-03-04T13:55:10.109 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:55:10.108+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:55:10.108537+0000) 2024-03-04T13:55:11.109 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:55:11.108+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:55:11.108866+0000) 2024-03-04T13:55:12.109 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:55:12.108+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:55:12.109119+0000) 2024-03-04T13:55:13.109 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:55:13.109+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:55:13.109388+0000) 2024-03-04T13:55:14.109 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:55:14.109+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:55:14.109649+0000) 2024-03-04T13:55:15.110 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:55:15.109+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:55:15.109933+0000) 2024-03-04T13:55:16.110 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:55:16.109+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:55:16.110184+0000) 2024-03-04T13:55:17.110 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:55:17.110+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:55:17.110439+0000) 2024-03-04T13:55:17.867 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:55:17.875 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:55:18.110 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:55:18.109+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:55:18.110722+0000) 2024-03-04T13:55:19.111 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:55:19.110+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:55:19.111000+0000) 2024-03-04T13:55:20.111 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:55:20.111+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:55:20.111223+0000) 2024-03-04T13:55:21.112 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:55:21.111+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:55:21.111460+0000) 2024-03-04T13:55:22.112 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:55:22.111+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:55:22.111699+0000) 2024-03-04T13:55:23.112 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:55:23.110+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:55:23.111945+0000) 2024-03-04T13:55:24.112 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:55:24.111+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:55:24.112193+0000) 2024-03-04T13:55:25.112 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:55:25.111+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:55:25.112450+0000) 2024-03-04T13:55:26.113 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:55:26.111+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:55:26.112702+0000) 2024-03-04T13:55:27.113 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:55:27.111+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:55:27.112959+0000) 2024-03-04T13:55:28.113 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:55:28.112+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:55:28.113218+0000) 2024-03-04T13:55:29.113 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:55:29.112+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:55:29.113416+0000) 2024-03-04T13:55:30.113 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:55:30.112+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:55:30.113661+0000) 2024-03-04T13:55:31.114 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:55:31.112+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:55:31.113868+0000) 2024-03-04T13:55:32.114 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:55:32.113+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:55:32.114100+0000) 2024-03-04T13:55:33.114 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:55:33.112+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:55:33.114286+0000) 2024-03-04T13:55:34.114 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:55:34.113+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:55:34.114530+0000) 2024-03-04T13:55:35.115 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:55:35.113+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:55:35.114789+0000) 2024-03-04T13:55:36.115 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:55:36.114+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:55:36.115035+0000) 2024-03-04T13:55:37.115 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:55:37.114+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:55:37.115224+0000) 2024-03-04T13:55:38.115 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:55:38.114+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:55:38.115468+0000) 2024-03-04T13:55:39.116 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:55:39.114+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:55:39.115727+0000) 2024-03-04T13:55:40.116 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:55:40.115+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:55:40.116013+0000) 2024-03-04T13:55:41.116 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:55:41.115+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:55:41.116269+0000) 2024-03-04T13:55:42.116 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:55:42.115+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:55:42.116503+0000) 2024-03-04T13:55:43.117 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:55:43.115+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:55:43.116752+0000) 2024-03-04T13:55:44.117 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:55:44.116+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:55:44.117029+0000) 2024-03-04T13:55:45.117 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:55:45.116+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:55:45.117281+0000) 2024-03-04T13:55:46.119 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:55:46.116+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:55:46.117535+0000) 2024-03-04T13:55:47.118 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:55:47.117+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:55:47.117791+0000) 2024-03-04T13:55:48.089 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:55:48.097 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:55:48.118 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:55:48.117+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:55:48.118033+0000) 2024-03-04T13:55:49.118 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:55:49.117+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:55:49.118295+0000) 2024-03-04T13:55:50.118 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:55:50.117+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:55:50.118530+0000) 2024-03-04T13:55:51.119 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:55:51.117+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:55:51.118787+0000) 2024-03-04T13:55:52.119 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:55:52.118+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:55:52.119026+0000) 2024-03-04T13:55:53.120 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:55:53.118+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:55:53.119280+0000) 2024-03-04T13:55:54.120 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:55:54.118+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:55:54.119522+0000) 2024-03-04T13:55:55.120 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:55:55.119+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:55:55.119762+0000) 2024-03-04T13:55:56.120 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:55:56.119+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:55:56.120004+0000) 2024-03-04T13:55:57.120 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:55:57.119+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:55:57.120288+0000) 2024-03-04T13:55:58.120 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:55:58.120+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:55:58.120544+0000) 2024-03-04T13:55:59.121 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:55:59.120+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:55:59.120769+0000) 2024-03-04T13:56:00.121 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:56:00.120+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:56:00.121030+0000) 2024-03-04T13:56:01.121 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:56:01.120+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:56:01.121290+0000) 2024-03-04T13:56:02.122 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:56:02.121+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:56:02.121574+0000) 2024-03-04T13:56:03.122 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:56:03.121+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:56:03.121863+0000) 2024-03-04T13:56:04.122 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:56:04.121+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:56:04.122127+0000) 2024-03-04T13:56:05.122 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:56:05.121+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:56:05.122380+0000) 2024-03-04T13:56:06.123 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:56:06.121+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:56:06.122648+0000) 2024-03-04T13:56:07.123 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:56:07.122+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:56:07.122943+0000) 2024-03-04T13:56:08.123 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:56:08.122+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:56:08.123195+0000) 2024-03-04T13:56:09.124 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:56:09.123+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:56:09.123471+0000) 2024-03-04T13:56:10.124 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:56:10.123+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:56:10.123721+0000) 2024-03-04T13:56:11.125 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:56:11.123+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:56:11.123988+0000) 2024-03-04T13:56:12.124 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:56:12.123+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:56:12.124270+0000) 2024-03-04T13:56:13.125 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:56:13.124+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:56:13.124565+0000) 2024-03-04T13:56:14.125 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:56:14.124+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:56:14.124929+0000) 2024-03-04T13:56:15.125 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:56:15.124+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:56:15.125215+0000) 2024-03-04T13:56:16.125 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:56:16.125+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:56:16.125506+0000) 2024-03-04T13:56:17.126 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:56:17.125+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:56:17.125800+0000) 2024-03-04T13:56:18.126 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:56:18.125+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:56:18.126150+0000) 2024-03-04T13:56:18.259 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:56:18.268 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:56:19.126 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:56:19.125+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:56:19.126408+0000) 2024-03-04T13:56:20.127 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:56:20.126+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:56:20.126652+0000) 2024-03-04T13:56:21.127 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:56:21.126+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:56:21.126932+0000) 2024-03-04T13:56:22.127 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:56:22.127+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:56:22.127166+0000) 2024-03-04T13:56:23.127 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:56:23.126+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:56:23.127412+0000) 2024-03-04T13:56:24.128 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:56:24.126+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:56:24.127668+0000) 2024-03-04T13:56:25.128 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:56:25.126+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:56:25.127950+0000) 2024-03-04T13:56:26.128 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:56:26.127+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:56:26.128203+0000) 2024-03-04T13:56:27.128 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:56:27.127+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:56:27.128455+0000) 2024-03-04T13:56:28.129 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:56:28.126+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:56:28.128787+0000) 2024-03-04T13:56:29.129 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:56:29.128+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:56:29.129118+0000) 2024-03-04T13:56:30.129 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:56:30.128+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:56:30.129402+0000) 2024-03-04T13:56:31.130 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:56:31.128+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:56:31.129661+0000) 2024-03-04T13:56:32.130 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:56:32.128+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:56:32.129950+0000) 2024-03-04T13:56:33.130 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:56:33.129+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:56:33.130190+0000) 2024-03-04T13:56:34.130 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:56:34.129+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:56:34.130427+0000) 2024-03-04T13:56:35.131 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:56:35.129+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:56:35.130673+0000) 2024-03-04T13:56:36.131 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:56:36.130+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:56:36.130941+0000) 2024-03-04T13:56:37.131 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:56:37.130+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:56:37.131140+0000) 2024-03-04T13:56:38.132 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:56:38.130+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:56:38.131386+0000) 2024-03-04T13:56:39.132 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:56:39.130+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:56:39.131591+0000) 2024-03-04T13:56:40.132 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:56:40.131+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:56:40.131887+0000) 2024-03-04T13:56:41.132 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:56:41.131+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:56:41.132144+0000) 2024-03-04T13:56:42.132 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:56:42.131+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:56:42.132405+0000) 2024-03-04T13:56:43.133 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:56:43.131+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:56:43.132698+0000) 2024-03-04T13:56:44.133 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:56:44.132+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:56:44.132956+0000) 2024-03-04T13:56:45.133 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:56:45.132+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:56:45.133217+0000) 2024-03-04T13:56:46.133 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:56:46.132+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:56:46.133409+0000) 2024-03-04T13:56:47.134 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:56:47.132+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:56:47.133675+0000) 2024-03-04T13:56:48.134 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:56:48.133+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:56:48.133935+0000) 2024-03-04T13:56:48.415 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:56:48.428 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:56:49.134 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:56:49.133+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:56:49.134198+0000) 2024-03-04T13:56:50.135 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:56:50.133+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:56:50.134487+0000) 2024-03-04T13:56:51.135 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:56:51.133+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:56:51.134745+0000) 2024-03-04T13:56:52.135 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:56:52.134+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:56:52.135055+0000) 2024-03-04T13:56:53.137 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:56:53.134+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:56:53.135339+0000) 2024-03-04T13:56:54.136 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:56:54.134+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:56:54.135574+0000) 2024-03-04T13:56:55.136 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:56:55.135+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:56:55.135812+0000) 2024-03-04T13:56:56.136 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:56:56.135+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:56:56.136094+0000) 2024-03-04T13:56:57.136 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:56:57.135+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:56:57.136330+0000) 2024-03-04T13:56:58.137 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:56:58.136+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:56:58.136573+0000) 2024-03-04T13:56:59.137 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:56:59.136+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:56:59.136874+0000) 2024-03-04T13:57:00.137 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:57:00.136+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:57:00.137129+0000) 2024-03-04T13:57:01.138 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:57:01.136+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:57:01.137366+0000) 2024-03-04T13:57:02.138 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:57:02.137+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:57:02.137599+0000) 2024-03-04T13:57:03.138 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:57:03.137+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:57:03.137881+0000) 2024-03-04T13:57:04.138 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:57:04.137+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:57:04.138119+0000) 2024-03-04T13:57:05.139 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:57:05.137+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:57:05.138355+0000) 2024-03-04T13:57:06.139 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:57:06.138+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:57:06.138595+0000) 2024-03-04T13:57:07.139 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:57:07.138+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:57:07.138885+0000) 2024-03-04T13:57:08.139 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:57:08.138+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:57:08.139140+0000) 2024-03-04T13:57:09.140 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:57:09.138+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:57:09.139406+0000) 2024-03-04T13:57:10.141 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:57:10.139+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:57:10.139639+0000) 2024-03-04T13:57:11.140 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:57:11.139+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:57:11.139917+0000) 2024-03-04T13:57:12.140 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:57:12.139+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:57:12.140191+0000) 2024-03-04T13:57:13.141 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:57:13.140+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:57:13.140430+0000) 2024-03-04T13:57:14.141 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:57:14.140+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:57:14.140619+0000) 2024-03-04T13:57:15.141 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:57:15.140+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:57:15.140927+0000) 2024-03-04T13:57:16.142 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:57:16.140+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:57:16.141204+0000) 2024-03-04T13:57:17.142 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:57:17.141+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:57:17.141455+0000) 2024-03-04T13:57:18.142 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:57:18.141+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:57:18.141708+0000) 2024-03-04T13:57:18.559 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:57:18.568 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:57:19.143 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:57:19.141+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:57:19.141980+0000) 2024-03-04T13:57:20.143 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:57:20.142+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:57:20.142228+0000) 2024-03-04T13:57:21.143 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:57:21.142+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:57:21.142497+0000) 2024-03-04T13:57:22.143 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:57:22.142+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:57:22.142781+0000) 2024-03-04T13:57:23.143 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:57:23.141+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:57:23.143084+0000) 2024-03-04T13:57:24.144 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:57:24.142+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:57:24.143334+0000) 2024-03-04T13:57:25.144 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:57:25.142+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:57:25.143594+0000) 2024-03-04T13:57:26.144 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:57:26.142+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:57:26.143878+0000) 2024-03-04T13:57:27.144 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:57:27.143+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:57:27.144223+0000) 2024-03-04T13:57:28.145 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:57:28.142+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:57:28.144424+0000) 2024-03-04T13:57:29.145 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:57:29.142+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:57:29.144692+0000) 2024-03-04T13:57:30.145 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:57:30.143+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:57:30.144942+0000) 2024-03-04T13:57:31.146 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:57:31.144+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:57:31.145189+0000) 2024-03-04T13:57:32.146 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:57:32.144+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:57:32.145425+0000) 2024-03-04T13:57:33.146 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:57:33.144+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:57:33.145662+0000) 2024-03-04T13:57:34.146 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:57:34.145+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:57:34.145937+0000) 2024-03-04T13:57:35.147 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:57:35.145+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:57:35.146179+0000) 2024-03-04T13:57:36.147 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:57:36.145+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:57:36.146414+0000) 2024-03-04T13:57:37.147 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:57:37.145+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:57:37.146646+0000) 2024-03-04T13:57:38.148 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:57:38.146+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:57:38.146965+0000) 2024-03-04T13:57:39.148 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:57:39.146+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:57:39.147223+0000) 2024-03-04T13:57:40.148 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:57:40.146+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:57:40.147552+0000) 2024-03-04T13:57:41.148 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:57:41.147+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:57:41.147860+0000) 2024-03-04T13:57:42.149 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:57:42.147+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:57:42.148100+0000) 2024-03-04T13:57:43.149 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:57:43.147+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:57:43.148343+0000) 2024-03-04T13:57:44.149 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:57:44.147+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:57:44.148579+0000) 2024-03-04T13:57:45.149 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:57:45.148+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:57:45.148816+0000) 2024-03-04T13:57:46.149 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:57:46.148+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:57:46.149044+0000) 2024-03-04T13:57:47.150 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:57:47.148+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:57:47.149281+0000) 2024-03-04T13:57:48.150 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:57:48.148+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:57:48.149516+0000) 2024-03-04T13:57:48.766 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:57:48.778 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:57:49.150 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:57:49.149+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:57:49.149774+0000) 2024-03-04T13:57:50.151 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:57:50.149+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:57:50.150028+0000) 2024-03-04T13:57:51.151 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:57:51.149+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:57:51.150286+0000) 2024-03-04T13:57:52.151 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:57:52.149+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:57:52.150544+0000) 2024-03-04T13:57:53.151 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:57:53.150+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:57:53.150784+0000) 2024-03-04T13:57:54.151 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:57:54.150+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:57:54.151031+0000) 2024-03-04T13:57:55.152 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:57:55.150+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:57:55.151264+0000) 2024-03-04T13:57:56.152 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:57:56.150+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:57:56.151500+0000) 2024-03-04T13:57:57.152 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:57:57.151+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:57:57.151745+0000) 2024-03-04T13:57:58.152 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:57:58.151+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:57:58.151981+0000) 2024-03-04T13:57:59.153 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:57:59.151+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:57:59.152183+0000) 2024-03-04T13:58:00.153 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:58:00.151+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:58:00.152429+0000) 2024-03-04T13:58:01.153 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:58:01.152+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:58:01.152672+0000) 2024-03-04T13:58:02.154 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:58:02.152+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:58:02.152950+0000) 2024-03-04T13:58:03.154 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:58:03.152+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:58:03.153193+0000) 2024-03-04T13:58:04.154 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:58:04.153+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:58:04.153440+0000) 2024-03-04T13:58:05.154 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:58:05.153+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:58:05.153637+0000) 2024-03-04T13:58:06.155 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:58:06.153+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:58:06.153924+0000) 2024-03-04T13:58:07.155 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:58:07.153+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:58:07.154166+0000) 2024-03-04T13:58:08.155 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:58:08.154+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:58:08.154402+0000) 2024-03-04T13:58:09.155 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:58:09.154+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:58:09.154655+0000) 2024-03-04T13:58:10.156 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:58:10.154+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:58:10.154908+0000) 2024-03-04T13:58:11.156 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:58:11.154+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:58:11.155149+0000) 2024-03-04T13:58:12.156 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:58:12.155+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:58:12.155381+0000) 2024-03-04T13:58:13.156 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:58:13.155+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:58:13.155619+0000) 2024-03-04T13:58:14.157 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:58:14.155+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:58:14.155888+0000) 2024-03-04T13:58:15.157 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:58:15.155+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:58:15.156187+0000) 2024-03-04T13:58:16.157 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:58:16.156+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:58:16.156428+0000) 2024-03-04T13:58:17.157 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:58:17.156+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:58:17.156662+0000) 2024-03-04T13:58:18.158 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:58:18.156+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:58:18.156935+0000) 2024-03-04T13:58:18.972 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:58:18.980 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:58:19.158 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:58:19.157+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:58:19.157196+0000) 2024-03-04T13:58:20.158 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:58:20.157+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:58:20.157453+0000) 2024-03-04T13:58:21.158 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:58:21.157+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:58:21.157718+0000) 2024-03-04T13:58:22.158 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:58:22.156+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:58:22.157957+0000) 2024-03-04T13:58:23.159 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:58:23.157+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:58:23.158199+0000) 2024-03-04T13:58:24.159 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:58:24.157+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:58:24.158438+0000) 2024-03-04T13:58:25.159 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:58:25.157+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:58:25.158677+0000) 2024-03-04T13:58:26.160 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:58:26.157+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:58:26.158946+0000) 2024-03-04T13:58:27.160 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:58:27.157+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:58:27.159137+0000) 2024-03-04T13:58:28.160 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:58:28.158+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:58:28.159379+0000) 2024-03-04T13:58:29.160 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:58:29.158+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:58:29.159623+0000) 2024-03-04T13:58:30.160 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:58:30.158+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:58:30.159914+0000) 2024-03-04T13:58:31.161 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:58:31.158+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:58:31.160150+0000) 2024-03-04T13:58:32.161 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:58:32.158+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:58:32.160324+0000) 2024-03-04T13:58:33.161 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:58:33.159+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:58:33.160574+0000) 2024-03-04T13:58:34.162 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:58:34.160+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:58:34.160814+0000) 2024-03-04T13:58:35.162 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:58:35.160+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:58:35.161098+0000) 2024-03-04T13:58:36.162 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:58:36.160+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:58:36.161338+0000) 2024-03-04T13:58:37.162 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:58:37.160+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:58:37.161597+0000) 2024-03-04T13:58:38.162 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:58:38.161+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:58:38.161860+0000) 2024-03-04T13:58:39.163 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:58:39.161+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:58:39.162109+0000) 2024-03-04T13:58:40.163 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:58:40.161+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:58:40.162358+0000) 2024-03-04T13:58:41.163 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:58:41.161+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:58:41.162553+0000) 2024-03-04T13:58:42.164 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:58:42.161+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:58:42.162768+0000) 2024-03-04T13:58:43.164 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:58:43.162+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:58:43.163016+0000) 2024-03-04T13:58:44.164 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:58:44.162+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:58:44.163258+0000) 2024-03-04T13:58:45.165 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:58:45.162+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:58:45.163496+0000) 2024-03-04T13:58:46.165 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:58:46.162+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:58:46.163744+0000) 2024-03-04T13:58:47.164 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:58:47.163+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:58:47.163983+0000) 2024-03-04T13:58:48.165 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:58:48.163+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:58:48.164218+0000) 2024-03-04T13:58:49.165 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:58:49.163+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:58:49.164474+0000) 2024-03-04T13:58:49.176 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:58:49.185 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:58:50.165 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:58:50.164+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:58:50.164752+0000) 2024-03-04T13:58:51.166 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:58:51.164+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:58:51.165047+0000) 2024-03-04T13:58:52.166 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:58:52.164+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:58:52.165355+0000) 2024-03-04T13:58:53.166 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:58:53.164+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:58:53.165632+0000) 2024-03-04T13:58:54.166 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:58:54.165+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:58:54.165973+0000) 2024-03-04T13:58:55.167 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:58:55.165+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:58:55.166267+0000) 2024-03-04T13:58:56.168 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:58:56.165+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:58:56.166574+0000) 2024-03-04T13:58:57.168 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:58:57.166+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:58:57.166921+0000) 2024-03-04T13:58:58.168 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:58:58.166+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:58:58.167213+0000) 2024-03-04T13:58:59.168 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:58:59.166+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:58:59.167480+0000) 2024-03-04T13:59:00.168 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:59:00.166+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:59:00.167708+0000) 2024-03-04T13:59:01.169 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:59:01.167+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:59:01.167998+0000) 2024-03-04T13:59:02.169 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:59:02.167+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:59:02.168280+0000) 2024-03-04T13:59:03.169 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:59:03.168+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:59:03.168561+0000) 2024-03-04T13:59:04.169 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:59:04.168+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:59:04.168854+0000) 2024-03-04T13:59:05.170 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:59:05.168+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:59:05.169128+0000) 2024-03-04T13:59:06.170 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:59:06.168+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:59:06.169409+0000) 2024-03-04T13:59:07.171 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:59:07.169+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:59:07.169701+0000) 2024-03-04T13:59:08.171 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:59:08.169+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:59:08.170037+0000) 2024-03-04T13:59:09.171 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:59:09.169+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:59:09.170352+0000) 2024-03-04T13:59:10.171 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:59:10.170+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:59:10.170637+0000) 2024-03-04T13:59:11.171 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:59:11.170+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:59:11.170957+0000) 2024-03-04T13:59:12.172 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:59:12.170+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:59:12.171235+0000) 2024-03-04T13:59:13.172 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:59:13.171+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:59:13.171527+0000) 2024-03-04T13:59:14.172 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:59:14.171+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:59:14.171816+0000) 2024-03-04T13:59:15.173 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:59:15.171+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:59:15.172153+0000) 2024-03-04T13:59:16.174 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:59:16.172+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:59:16.172435+0000) 2024-03-04T13:59:17.173 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:59:17.172+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:59:17.172729+0000) 2024-03-04T13:59:18.174 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:59:18.172+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:59:18.173032+0000) 2024-03-04T13:59:19.174 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:59:19.173+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:59:19.173358+0000) 2024-03-04T13:59:19.331 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:59:19.339 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:59:20.174 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:59:20.173+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:59:20.173646+0000) 2024-03-04T13:59:21.174 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:59:21.173+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:59:21.173955+0000) 2024-03-04T13:59:22.175 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:59:22.173+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:59:22.174230+0000) 2024-03-04T13:59:23.175 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:59:23.173+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:59:23.174520+0000) 2024-03-04T13:59:24.175 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:59:24.173+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:59:24.174809+0000) 2024-03-04T13:59:25.176 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:59:25.174+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:59:25.175163+0000) 2024-03-04T13:59:26.176 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:59:26.173+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:59:26.175410+0000) 2024-03-04T13:59:27.176 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:59:27.174+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:59:27.175701+0000) 2024-03-04T13:59:28.177 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:59:28.174+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:59:28.175995+0000) 2024-03-04T13:59:29.177 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:59:29.175+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:59:29.176300+0000) 2024-03-04T13:59:30.177 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:59:30.175+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:59:30.176597+0000) 2024-03-04T13:59:31.177 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:59:31.175+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:59:31.176917+0000) 2024-03-04T13:59:32.178 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:59:32.176+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:59:32.177195+0000) 2024-03-04T13:59:33.178 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:59:33.176+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:59:33.177489+0000) 2024-03-04T13:59:34.178 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:59:34.176+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:59:34.177778+0000) 2024-03-04T13:59:35.179 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:59:35.177+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:59:35.178119+0000) 2024-03-04T13:59:36.179 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:59:36.177+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:59:36.178361+0000) 2024-03-04T13:59:37.179 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:59:37.177+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:59:37.178651+0000) 2024-03-04T13:59:38.180 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:59:38.178+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:59:38.178980+0000) 2024-03-04T13:59:39.180 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:59:39.178+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:59:39.179294+0000) 2024-03-04T13:59:40.180 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:59:40.178+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:59:40.179583+0000) 2024-03-04T13:59:41.181 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:59:41.179+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:59:41.179934+0000) 2024-03-04T13:59:42.181 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:59:42.179+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:59:42.180166+0000) 2024-03-04T13:59:43.181 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:59:43.179+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:59:43.180481+0000) 2024-03-04T13:59:44.182 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:59:44.179+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:59:44.180786+0000) 2024-03-04T13:59:45.182 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:59:45.180+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:59:45.181148+0000) 2024-03-04T13:59:46.182 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:59:46.180+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:59:46.181431+0000) 2024-03-04T13:59:47.182 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:59:47.180+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:59:47.181722+0000) 2024-03-04T13:59:48.183 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:59:48.181+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:59:48.182015+0000) 2024-03-04T13:59:49.183 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:59:49.181+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:59:49.182347+0000) 2024-03-04T13:59:49.477 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:59:49.485 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T13:59:50.183 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:59:50.181+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:59:50.182624+0000) 2024-03-04T13:59:51.184 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:59:51.182+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:59:51.183000+0000) 2024-03-04T13:59:52.184 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:59:52.182+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:59:52.183283+0000) 2024-03-04T13:59:53.184 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:59:53.182+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:59:53.183582+0000) 2024-03-04T13:59:54.184 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:59:54.183+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:59:54.183923+0000) 2024-03-04T13:59:55.185 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:59:55.183+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:59:55.184185+0000) 2024-03-04T13:59:56.185 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:59:56.183+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:59:56.184466+0000) 2024-03-04T13:59:57.186 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:59:57.183+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:59:57.184776+0000) 2024-03-04T13:59:58.186 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:59:58.184+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:59:58.185069+0000) 2024-03-04T13:59:59.186 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T13:59:59.184+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T12:59:59.185317+0000) 2024-03-04T14:00:00.187 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:00:00.185+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:00:00.185617+0000) 2024-03-04T14:00:01.187 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:00:01.185+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:00:01.185960+0000) 2024-03-04T14:00:02.187 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:00:02.185+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:00:02.186192+0000) 2024-03-04T14:00:03.187 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:00:03.185+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:00:03.186492+0000) 2024-03-04T14:00:04.188 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:00:04.186+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:00:04.186791+0000) 2024-03-04T14:00:05.188 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:00:05.186+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:00:05.187164+0000) 2024-03-04T14:00:06.188 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:00:06.187+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:00:06.187459+0000) 2024-03-04T14:00:07.189 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:00:07.187+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:00:07.187745+0000) 2024-03-04T14:00:08.189 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:00:08.187+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:00:08.188056+0000) 2024-03-04T14:00:09.189 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:00:09.187+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:00:09.188388+0000) 2024-03-04T14:00:10.190 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:00:10.188+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:00:10.188700+0000) 2024-03-04T14:00:11.190 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:00:11.188+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:00:11.188998+0000) 2024-03-04T14:00:12.190 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:00:12.188+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:00:12.189217+0000) 2024-03-04T14:00:13.190 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:00:13.188+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:00:13.189515+0000) 2024-03-04T14:00:14.191 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:00:14.189+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:00:14.189798+0000) 2024-03-04T14:00:15.191 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:00:15.189+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:00:15.190144+0000) 2024-03-04T14:00:16.191 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:00:16.190+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:00:16.190433+0000) 2024-03-04T14:00:17.192 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:00:17.190+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:00:17.190715+0000) 2024-03-04T14:00:18.192 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:00:18.190+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:00:18.191016+0000) 2024-03-04T14:00:19.192 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:00:19.191+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:00:19.191291+0000) 2024-03-04T14:00:19.624 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:00:19.632 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:00:20.192 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:00:20.190+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:00:20.191525+0000) 2024-03-04T14:00:21.193 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:00:21.191+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:00:21.191866+0000) 2024-03-04T14:00:22.193 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:00:22.191+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:00:22.192164+0000) 2024-03-04T14:00:23.193 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:00:23.191+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:00:23.192445+0000) 2024-03-04T14:00:24.193 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:00:24.191+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:00:24.192742+0000) 2024-03-04T14:00:25.194 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:00:25.191+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:00:25.192985+0000) 2024-03-04T14:00:26.194 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:00:26.192+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:00:26.193274+0000) 2024-03-04T14:00:27.194 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:00:27.192+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:00:27.193574+0000) 2024-03-04T14:00:28.195 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:00:28.192+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:00:28.193936+0000) 2024-03-04T14:00:29.195 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:00:29.192+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:00:29.194183+0000) 2024-03-04T14:00:30.195 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:00:30.193+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:00:30.194477+0000) 2024-03-04T14:00:31.195 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:00:31.193+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:00:31.194776+0000) 2024-03-04T14:00:32.196 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:00:32.194+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:00:32.195100+0000) 2024-03-04T14:00:33.196 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:00:33.194+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:00:33.195398+0000) 2024-03-04T14:00:34.196 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:00:34.194+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:00:34.195719+0000) 2024-03-04T14:00:35.197 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:00:35.195+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:00:35.196037+0000) 2024-03-04T14:00:36.197 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:00:36.195+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:00:36.196315+0000) 2024-03-04T14:00:37.197 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:00:37.195+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:00:37.196606+0000) 2024-03-04T14:00:38.198 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:00:38.196+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:00:38.196919+0000) 2024-03-04T14:00:39.198 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:00:39.196+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:00:39.197222+0000) 2024-03-04T14:00:40.198 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:00:40.196+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:00:40.197524+0000) 2024-03-04T14:00:41.199 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:00:41.197+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:00:41.197877+0000) 2024-03-04T14:00:42.199 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:00:42.197+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:00:42.198169+0000) 2024-03-04T14:00:43.199 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:00:43.197+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:00:43.198462+0000) 2024-03-04T14:00:44.199 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:00:44.197+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:00:44.198754+0000) 2024-03-04T14:00:45.200 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:00:45.198+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:00:45.199060+0000) 2024-03-04T14:00:46.200 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:00:46.198+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:00:46.199332+0000) 2024-03-04T14:00:47.200 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:00:47.198+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:00:47.199579+0000) 2024-03-04T14:00:48.201 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:00:48.199+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:00:48.199922+0000) 2024-03-04T14:00:49.201 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:00:49.199+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:00:49.200222+0000) 2024-03-04T14:00:49.822 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:00:49.830 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:00:50.201 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:00:50.199+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:00:50.200522+0000) 2024-03-04T14:00:51.202 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:00:51.200+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:00:51.200882+0000) 2024-03-04T14:00:52.202 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:00:52.200+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:00:52.201173+0000) 2024-03-04T14:00:53.202 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:00:53.200+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:00:53.201471+0000) 2024-03-04T14:00:54.202 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:00:54.201+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:00:54.201741+0000) 2024-03-04T14:00:55.203 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:00:55.201+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:00:55.201994+0000) 2024-03-04T14:00:56.203 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:00:56.201+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:00:56.202231+0000) 2024-03-04T14:00:57.203 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:00:57.201+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:00:57.202468+0000) 2024-03-04T14:00:58.204 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:00:58.201+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:00:58.202746+0000) 2024-03-04T14:00:59.204 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:00:59.202+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:00:59.203044+0000) 2024-03-04T14:01:00.204 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:01:00.202+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:01:00.203362+0000) 2024-03-04T14:01:01.204 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:01:01.203+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:01:01.203687+0000) 2024-03-04T14:01:02.205 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:01:02.203+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:01:02.203989+0000) 2024-03-04T14:01:03.205 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:01:03.203+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:01:03.204285+0000) 2024-03-04T14:01:04.205 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:01:04.204+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:01:04.204584+0000) 2024-03-04T14:01:05.206 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:01:05.204+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:01:05.204938+0000) 2024-03-04T14:01:06.206 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:01:06.204+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:01:06.205226+0000) 2024-03-04T14:01:07.206 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:01:07.204+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:01:07.205466+0000) 2024-03-04T14:01:08.207 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:01:08.205+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:01:08.205744+0000) 2024-03-04T14:01:09.207 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:01:09.205+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:01:09.206028+0000) 2024-03-04T14:01:10.207 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:01:10.205+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:01:10.206265+0000) 2024-03-04T14:01:11.207 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:01:11.206+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:01:11.206545+0000) 2024-03-04T14:01:12.208 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:01:12.206+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:01:12.206814+0000) 2024-03-04T14:01:13.208 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:01:13.206+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:01:13.207157+0000) 2024-03-04T14:01:14.208 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:01:14.207+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:01:14.207427+0000) 2024-03-04T14:01:15.209 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:01:15.207+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:01:15.207718+0000) 2024-03-04T14:01:16.209 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:01:16.207+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:01:16.208009+0000) 2024-03-04T14:01:17.209 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:01:17.208+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:01:17.208267+0000) 2024-03-04T14:01:18.209 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:01:18.207+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:01:18.208478+0000) 2024-03-04T14:01:19.210 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:01:19.208+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:01:19.208782+0000) 2024-03-04T14:01:20.010 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:01:20.018 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:01:20.210 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:01:20.208+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:01:20.209131+0000) 2024-03-04T14:01:21.210 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:01:21.209+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:01:21.209428+0000) 2024-03-04T14:01:22.211 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:01:22.209+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:01:22.209707+0000) 2024-03-04T14:01:23.211 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:01:23.209+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:01:23.210031+0000) 2024-03-04T14:01:24.211 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:01:24.209+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:01:24.210300+0000) 2024-03-04T14:01:25.211 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:01:25.209+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:01:25.210590+0000) 2024-03-04T14:01:26.212 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:01:26.209+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:01:26.210939+0000) 2024-03-04T14:01:27.212 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:01:27.210+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:01:27.211256+0000) 2024-03-04T14:01:28.212 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:01:28.209+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:01:28.211479+0000) 2024-03-04T14:01:29.213 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:01:29.210+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:01:29.211804+0000) 2024-03-04T14:01:30.213 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:01:30.211+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:01:30.212127+0000) 2024-03-04T14:01:31.213 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:01:31.211+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:01:31.212420+0000) 2024-03-04T14:01:32.214 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:01:32.211+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:01:32.212696+0000) 2024-03-04T14:01:33.214 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:01:33.211+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:01:33.212986+0000) 2024-03-04T14:01:34.214 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:01:34.212+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:01:34.213244+0000) 2024-03-04T14:01:35.215 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:01:35.212+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:01:35.213538+0000) 2024-03-04T14:01:36.215 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:01:36.213+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:01:36.213913+0000) 2024-03-04T14:01:37.216 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:01:37.213+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:01:37.214209+0000) 2024-03-04T14:01:38.215 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:01:38.213+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:01:38.214570+0000) 2024-03-04T14:01:39.216 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:01:39.214+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:01:39.214931+0000) 2024-03-04T14:01:40.216 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:01:40.214+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:01:40.215163+0000) 2024-03-04T14:01:41.217 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:01:41.214+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:01:41.215456+0000) 2024-03-04T14:01:42.217 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:01:42.214+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:01:42.215724+0000) 2024-03-04T14:01:43.217 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:01:43.215+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:01:43.216021+0000) 2024-03-04T14:01:44.217 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:01:44.215+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:01:44.216300+0000) 2024-03-04T14:01:45.219 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:01:45.215+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:01:45.216585+0000) 2024-03-04T14:01:46.218 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:01:46.216+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:01:46.216928+0000) 2024-03-04T14:01:47.218 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:01:47.216+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:01:47.217220+0000) 2024-03-04T14:01:48.218 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:01:48.216+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:01:48.217466+0000) 2024-03-04T14:01:49.219 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:01:49.217+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:01:49.217721+0000) 2024-03-04T14:01:50.219 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:01:50.217+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:01:50.217977+0000) 2024-03-04T14:01:50.231 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:01:50.239 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:01:51.219 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:01:51.217+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:01:51.218212+0000) 2024-03-04T14:01:52.219 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:01:52.217+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:01:52.218474+0000) 2024-03-04T14:01:53.220 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:01:53.217+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:01:53.218757+0000) 2024-03-04T14:01:54.220 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:01:54.218+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:01:54.219001+0000) 2024-03-04T14:01:55.220 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:01:55.218+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:01:55.219274+0000) 2024-03-04T14:01:56.220 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:01:56.218+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:01:56.219527+0000) 2024-03-04T14:01:57.221 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:01:57.219+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:01:57.219780+0000) 2024-03-04T14:01:58.221 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:01:58.219+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:01:58.220083+0000) 2024-03-04T14:01:59.221 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:01:59.219+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:01:59.220350+0000) 2024-03-04T14:02:00.221 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:02:00.220+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:02:00.220599+0000) 2024-03-04T14:02:01.222 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:02:01.220+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:02:01.220890+0000) 2024-03-04T14:02:02.222 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:02:02.220+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:02:02.221143+0000) 2024-03-04T14:02:03.222 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:02:03.220+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:02:03.221393+0000) 2024-03-04T14:02:04.222 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:02:04.221+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:02:04.221647+0000) 2024-03-04T14:02:05.223 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:02:05.221+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:02:05.221955+0000) 2024-03-04T14:02:06.223 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:02:06.221+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:02:06.222213+0000) 2024-03-04T14:02:07.223 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:02:07.222+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:02:07.222467+0000) 2024-03-04T14:02:08.224 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:02:08.222+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:02:08.222721+0000) 2024-03-04T14:02:09.224 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:02:09.222+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:02:09.222985+0000) 2024-03-04T14:02:10.224 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:02:10.222+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:02:10.223239+0000) 2024-03-04T14:02:11.224 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:02:11.223+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:02:11.223501+0000) 2024-03-04T14:02:12.225 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:02:12.223+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:02:12.223742+0000) 2024-03-04T14:02:13.225 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:02:13.223+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:02:13.224016+0000) 2024-03-04T14:02:14.225 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:02:14.223+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:02:14.224263+0000) 2024-03-04T14:02:15.226 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:02:15.224+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:02:15.224511+0000) 2024-03-04T14:02:16.226 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:02:16.224+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:02:16.224745+0000) 2024-03-04T14:02:17.227 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:02:17.224+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:02:17.224987+0000) 2024-03-04T14:02:18.226 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:02:18.225+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:02:18.225246+0000) 2024-03-04T14:02:19.226 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:02:19.225+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:02:19.225507+0000) 2024-03-04T14:02:20.227 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:02:20.225+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:02:20.225750+0000) 2024-03-04T14:02:20.369 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:02:20.377 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:02:21.227 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:02:21.225+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:02:21.226005+0000) 2024-03-04T14:02:22.227 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:02:22.226+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:02:22.226258+0000) 2024-03-04T14:02:23.227 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:02:23.226+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:02:23.226443+0000) 2024-03-04T14:02:24.228 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:02:24.226+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:02:24.226717+0000) 2024-03-04T14:02:25.228 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:02:25.225+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:02:25.226977+0000) 2024-03-04T14:02:26.228 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:02:26.225+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:02:26.227246+0000) 2024-03-04T14:02:27.228 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:02:27.226+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:02:27.227528+0000) 2024-03-04T14:02:28.229 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:02:28.226+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:02:28.227776+0000) 2024-03-04T14:02:29.229 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:02:29.227+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:02:29.228086+0000) 2024-03-04T14:02:30.229 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:02:30.227+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:02:30.228374+0000) 2024-03-04T14:02:31.230 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:02:31.227+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:02:31.228635+0000) 2024-03-04T14:02:32.230 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:02:32.227+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:02:32.228927+0000) 2024-03-04T14:02:33.230 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:02:33.228+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:02:33.229178+0000) 2024-03-04T14:02:34.230 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:02:34.228+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:02:34.229440+0000) 2024-03-04T14:02:35.231 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:02:35.228+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:02:35.229694+0000) 2024-03-04T14:02:36.231 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:02:36.229+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:02:36.229933+0000) 2024-03-04T14:02:37.231 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:02:37.229+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:02:37.230190+0000) 2024-03-04T14:02:38.231 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:02:38.229+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:02:38.230447+0000) 2024-03-04T14:02:39.232 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:02:39.229+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:02:39.230721+0000) 2024-03-04T14:02:40.232 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:02:40.230+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:02:40.230977+0000) 2024-03-04T14:02:41.232 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:02:41.230+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:02:41.231238+0000) 2024-03-04T14:02:42.233 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:02:42.230+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:02:42.231488+0000) 2024-03-04T14:02:43.233 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:02:43.230+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:02:43.231738+0000) 2024-03-04T14:02:44.233 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:02:44.231+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:02:44.231976+0000) 2024-03-04T14:02:45.233 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:02:45.231+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:02:45.232205+0000) 2024-03-04T14:02:46.233 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:02:46.231+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:02:46.232458+0000) 2024-03-04T14:02:47.234 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:02:47.231+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:02:47.232700+0000) 2024-03-04T14:02:48.234 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:02:48.232+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:02:48.232949+0000) 2024-03-04T14:02:49.235 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:02:49.232+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:02:49.233200+0000) 2024-03-04T14:02:50.235 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:02:50.232+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:02:50.233440+0000) 2024-03-04T14:02:50.608 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:02:50.616 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:02:51.235 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:02:51.233+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:02:51.233701+0000) 2024-03-04T14:02:52.235 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:02:52.233+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:02:52.233965+0000) 2024-03-04T14:02:53.235 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:02:53.233+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:02:53.234197+0000) 2024-03-04T14:02:54.235 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:02:54.233+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:02:54.234426+0000) 2024-03-04T14:02:55.236 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:02:55.234+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:02:55.234664+0000) 2024-03-04T14:02:56.236 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:02:56.234+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:02:56.234932+0000) 2024-03-04T14:02:57.237 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:02:57.234+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:02:57.235170+0000) 2024-03-04T14:02:58.236 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:02:58.234+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:02:58.235412+0000) 2024-03-04T14:02:59.237 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:02:59.235+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:02:59.235666+0000) 2024-03-04T14:03:00.237 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:03:00.235+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:03:00.235943+0000) 2024-03-04T14:03:01.237 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:03:01.235+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:03:01.236187+0000) 2024-03-04T14:03:02.237 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:03:02.235+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:03:02.236394+0000) 2024-03-04T14:03:03.238 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:03:03.236+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:03:03.236645+0000) 2024-03-04T14:03:04.238 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:03:04.236+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:03:04.236946+0000) 2024-03-04T14:03:05.238 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:03:05.236+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:03:05.237177+0000) 2024-03-04T14:03:06.238 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:03:06.236+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:03:06.237367+0000) 2024-03-04T14:03:07.239 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:03:07.237+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:03:07.237607+0000) 2024-03-04T14:03:08.239 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:03:08.237+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:03:08.237939+0000) 2024-03-04T14:03:09.239 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:03:09.237+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:03:09.238192+0000) 2024-03-04T14:03:10.240 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:03:10.238+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:03:10.238429+0000) 2024-03-04T14:03:11.240 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:03:11.238+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:03:11.238676+0000) 2024-03-04T14:03:12.240 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:03:12.238+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:03:12.238936+0000) 2024-03-04T14:03:13.240 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:03:13.238+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:03:13.239174+0000) 2024-03-04T14:03:14.240 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:03:14.239+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:03:14.239404+0000) 2024-03-04T14:03:15.241 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:03:15.239+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:03:15.239644+0000) 2024-03-04T14:03:16.241 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:03:16.239+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:03:16.239919+0000) 2024-03-04T14:03:17.241 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:03:17.239+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:03:17.240133+0000) 2024-03-04T14:03:18.241 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:03:18.239+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:03:18.240312+0000) 2024-03-04T14:03:19.242 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:03:19.239+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:03:19.240557+0000) 2024-03-04T14:03:20.242 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:03:20.240+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:03:20.240795+0000) 2024-03-04T14:03:20.829 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:03:20.837 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:03:21.242 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:03:21.240+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:03:21.241090+0000) 2024-03-04T14:03:22.242 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:03:22.241+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:03:22.241346+0000) 2024-03-04T14:03:23.243 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:03:23.241+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:03:23.241584+0000) 2024-03-04T14:03:24.243 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:03:24.241+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:03:24.241815+0000) 2024-03-04T14:03:25.243 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:03:25.241+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:03:25.242094+0000) 2024-03-04T14:03:26.243 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:03:26.240+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:03:26.242332+0000) 2024-03-04T14:03:27.244 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:03:27.240+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:03:27.242525+0000) 2024-03-04T14:03:28.244 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:03:28.241+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:03:28.242758+0000) 2024-03-04T14:03:29.244 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:03:29.241+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:03:29.243010+0000) 2024-03-04T14:03:30.244 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:03:30.242+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:03:30.243260+0000) 2024-03-04T14:03:31.245 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:03:31.242+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:03:31.243500+0000) 2024-03-04T14:03:32.245 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:03:32.242+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:03:32.243732+0000) 2024-03-04T14:03:33.245 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:03:33.242+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:03:33.243972+0000) 2024-03-04T14:03:34.245 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:03:34.243+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:03:34.244203+0000) 2024-03-04T14:03:35.246 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:03:35.243+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:03:35.244452+0000) 2024-03-04T14:03:36.246 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:03:36.243+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:03:36.244698+0000) 2024-03-04T14:03:37.246 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:03:37.244+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:03:37.244937+0000) 2024-03-04T14:03:38.246 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:03:38.244+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:03:38.245112+0000) 2024-03-04T14:03:39.247 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:03:39.244+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:03:39.245359+0000) 2024-03-04T14:03:40.247 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:03:40.244+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:03:40.245546+0000) 2024-03-04T14:03:41.247 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:03:41.244+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:03:41.245804+0000) 2024-03-04T14:03:42.247 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:03:42.245+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:03:42.246101+0000) 2024-03-04T14:03:43.247 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:03:43.245+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:03:43.246292+0000) 2024-03-04T14:03:44.248 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:03:44.245+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:03:44.246519+0000) 2024-03-04T14:03:45.248 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:03:45.245+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:03:45.246755+0000) 2024-03-04T14:03:46.248 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:03:46.246+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:03:46.247034+0000) 2024-03-04T14:03:47.248 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:03:47.246+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:03:47.247274+0000) 2024-03-04T14:03:48.249 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:03:48.246+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:03:48.247529+0000) 2024-03-04T14:03:49.249 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:03:49.246+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:03:49.247795+0000) 2024-03-04T14:03:50.249 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:03:50.247+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:03:50.248025+0000) 2024-03-04T14:03:51.249 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:03:51.247+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:03:51.248273+0000) 2024-03-04T14:03:51.251 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:03:51.259 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:03:52.250 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:03:52.247+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:03:52.248543+0000) 2024-03-04T14:03:53.250 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:03:53.248+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:03:53.248804+0000) 2024-03-04T14:03:54.250 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:03:54.248+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:03:54.249071+0000) 2024-03-04T14:03:55.250 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:03:55.248+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:03:55.249313+0000) 2024-03-04T14:03:56.251 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:03:56.248+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:03:56.249555+0000) 2024-03-04T14:03:57.251 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:03:57.249+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:03:57.249795+0000) 2024-03-04T14:03:58.251 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:03:58.249+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:03:58.250066+0000) 2024-03-04T14:03:59.251 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:03:59.249+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:03:59.250314+0000) 2024-03-04T14:04:00.252 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:04:00.249+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:04:00.250527+0000) 2024-03-04T14:04:01.252 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:04:01.249+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:04:01.250790+0000) 2024-03-04T14:04:02.252 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:04:02.250+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:04:02.251119+0000) 2024-03-04T14:04:03.252 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:04:03.250+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:04:03.251312+0000) 2024-03-04T14:04:04.253 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:04:04.251+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:04:04.251541+0000) 2024-03-04T14:04:05.253 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:04:05.251+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:04:05.251788+0000) 2024-03-04T14:04:06.253 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:04:06.251+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:04:06.252088+0000) 2024-03-04T14:04:07.253 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:04:07.251+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:04:07.252330+0000) 2024-03-04T14:04:08.254 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:04:08.252+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:04:08.252569+0000) 2024-03-04T14:04:09.254 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:04:09.252+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:04:09.252895+0000) 2024-03-04T14:04:10.254 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:04:10.252+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:04:10.253144+0000) 2024-03-04T14:04:11.254 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:04:11.253+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:04:11.253386+0000) 2024-03-04T14:04:12.255 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:04:12.253+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:04:12.253619+0000) 2024-03-04T14:04:13.255 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:04:13.253+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:04:13.253880+0000) 2024-03-04T14:04:14.255 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:04:14.253+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:04:14.254113+0000) 2024-03-04T14:04:15.256 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:04:15.254+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:04:15.254366+0000) 2024-03-04T14:04:16.256 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:04:16.254+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:04:16.254600+0000) 2024-03-04T14:04:17.256 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:04:17.254+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:04:17.254889+0000) 2024-03-04T14:04:18.256 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:04:18.254+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:04:18.255126+0000) 2024-03-04T14:04:19.257 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:04:19.255+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:04:19.255398+0000) 2024-03-04T14:04:20.257 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:04:20.255+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:04:20.255640+0000) 2024-03-04T14:04:21.257 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:04:21.255+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:04:21.255926+0000) 2024-03-04T14:04:21.389 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:04:21.397 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:04:22.257 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:04:22.255+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:04:22.256188+0000) 2024-03-04T14:04:23.258 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:04:23.256+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:04:23.256448+0000) 2024-03-04T14:04:24.259 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:04:24.256+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:04:24.256683+0000) 2024-03-04T14:04:25.258 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:04:25.256+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:04:25.256965+0000) 2024-03-04T14:04:26.258 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:04:26.257+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:04:26.257212+0000) 2024-03-04T14:04:27.259 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:04:27.256+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:04:27.257453+0000) 2024-03-04T14:04:28.259 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:04:28.256+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:04:28.257686+0000) 2024-03-04T14:04:29.259 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:04:29.256+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:04:29.257917+0000) 2024-03-04T14:04:30.259 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:04:30.257+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:04:30.258164+0000) 2024-03-04T14:04:31.260 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:04:31.256+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:04:31.258361+0000) 2024-03-04T14:04:32.260 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:04:32.257+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:04:32.258587+0000) 2024-03-04T14:04:33.260 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:04:33.257+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:04:33.258856+0000) 2024-03-04T14:04:34.260 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:04:34.258+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:04:34.259095+0000) 2024-03-04T14:04:35.260 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:04:35.258+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:04:35.259341+0000) 2024-03-04T14:04:36.261 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:04:36.258+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:04:36.259583+0000) 2024-03-04T14:04:37.261 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:04:37.259+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:04:37.259874+0000) 2024-03-04T14:04:38.261 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:04:38.259+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:04:38.260123+0000) 2024-03-04T14:04:39.262 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:04:39.259+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:04:39.260377+0000) 2024-03-04T14:04:40.262 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:04:40.259+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:04:40.260626+0000) 2024-03-04T14:04:41.262 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:04:41.260+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:04:41.260927+0000) 2024-03-04T14:04:42.262 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:04:42.260+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:04:42.261167+0000) 2024-03-04T14:04:43.263 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:04:43.260+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:04:43.261358+0000) 2024-03-04T14:04:44.263 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:04:44.260+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:04:44.261585+0000) 2024-03-04T14:04:45.263 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:04:45.261+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:04:45.261869+0000) 2024-03-04T14:04:46.263 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:04:46.261+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:04:46.262114+0000) 2024-03-04T14:04:47.264 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:04:47.261+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:04:47.262354+0000) 2024-03-04T14:04:48.264 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:04:48.261+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:04:48.262589+0000) 2024-03-04T14:04:49.264 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:04:49.262+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:04:49.262882+0000) 2024-03-04T14:04:50.264 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:04:50.262+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:04:50.263132+0000) 2024-03-04T14:04:51.265 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:04:51.262+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:04:51.263379+0000) 2024-03-04T14:04:51.628 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:04:51.637 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:04:52.265 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:04:52.262+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:04:52.263626+0000) 2024-03-04T14:04:53.265 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:04:53.263+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:04:53.263926+0000) 2024-03-04T14:04:54.266 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:04:54.263+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:04:54.264176+0000) 2024-03-04T14:04:55.266 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:04:55.263+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:04:55.264474+0000) 2024-03-04T14:04:56.266 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:04:56.264+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:04:56.264726+0000) 2024-03-04T14:04:57.266 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:04:57.264+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:04:57.264966+0000) 2024-03-04T14:04:58.266 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:04:58.264+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:04:58.265202+0000) 2024-03-04T14:04:59.267 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:04:59.264+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:04:59.265447+0000) 2024-03-04T14:05:00.267 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:05:00.265+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:05:00.265697+0000) 2024-03-04T14:05:01.267 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:05:01.265+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:05:01.265953+0000) 2024-03-04T14:05:02.267 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:05:02.265+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:05:02.266185+0000) 2024-03-04T14:05:03.268 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:05:03.265+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:05:03.266420+0000) 2024-03-04T14:05:04.268 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:05:04.266+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:05:04.266651+0000) 2024-03-04T14:05:05.268 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:05:05.266+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:05:05.266877+0000) 2024-03-04T14:05:06.268 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:05:06.266+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:05:06.267130+0000) 2024-03-04T14:05:07.269 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:05:07.267+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:05:07.267369+0000) 2024-03-04T14:05:08.269 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:05:08.267+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:05:08.267603+0000) 2024-03-04T14:05:09.269 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:05:09.267+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:05:09.267900+0000) 2024-03-04T14:05:10.269 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:05:10.267+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:05:10.268146+0000) 2024-03-04T14:05:11.270 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:05:11.268+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:05:11.268421+0000) 2024-03-04T14:05:12.270 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:05:12.267+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:05:12.268644+0000) 2024-03-04T14:05:13.271 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:05:13.268+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:05:13.268907+0000) 2024-03-04T14:05:14.271 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:05:14.268+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:05:14.269146+0000) 2024-03-04T14:05:15.271 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:05:15.268+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:05:15.269388+0000) 2024-03-04T14:05:16.271 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:05:16.269+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:05:16.269632+0000) 2024-03-04T14:05:17.271 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:05:17.269+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:05:17.269922+0000) 2024-03-04T14:05:18.271 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:05:18.270+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:05:18.270160+0000) 2024-03-04T14:05:19.272 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:05:19.270+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:05:19.270421+0000) 2024-03-04T14:05:20.272 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:05:20.269+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:05:20.270659+0000) 2024-03-04T14:05:21.272 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:05:21.269+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:05:21.270949+0000) 2024-03-04T14:05:21.833 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:05:21.841 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:05:22.273 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:05:22.270+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:05:22.271217+0000) 2024-03-04T14:05:23.273 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:05:23.269+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:05:23.271485+0000) 2024-03-04T14:05:24.273 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:05:24.270+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:05:24.271711+0000) 2024-03-04T14:05:25.273 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:05:25.270+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:05:25.271953+0000) 2024-03-04T14:05:26.274 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:05:26.271+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:05:26.272201+0000) 2024-03-04T14:05:27.274 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:05:27.271+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:05:27.272445+0000) 2024-03-04T14:05:28.274 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:05:28.271+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:05:28.272670+0000) 2024-03-04T14:05:29.274 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:05:29.271+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:05:29.272953+0000) 2024-03-04T14:05:30.274 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:05:30.272+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:05:30.273139+0000) 2024-03-04T14:05:31.275 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:05:31.272+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:05:31.273332+0000) 2024-03-04T14:05:32.275 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:05:32.272+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:05:32.273608+0000) 2024-03-04T14:05:33.275 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:05:33.273+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:05:33.273953+0000) 2024-03-04T14:05:34.276 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:05:34.273+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:05:34.274219+0000) 2024-03-04T14:05:35.276 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:05:35.273+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:05:35.274511+0000) 2024-03-04T14:05:36.276 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:05:36.274+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:05:36.274873+0000) 2024-03-04T14:05:37.277 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:05:37.274+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:05:37.275177+0000) 2024-03-04T14:05:38.277 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:05:38.274+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:05:38.275456+0000) 2024-03-04T14:05:39.277 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:05:39.274+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:05:39.275755+0000) 2024-03-04T14:05:40.277 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:05:40.275+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:05:40.276021+0000) 2024-03-04T14:05:41.278 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:05:41.275+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:05:41.276331+0000) 2024-03-04T14:05:42.278 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:05:42.275+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:05:42.276600+0000) 2024-03-04T14:05:43.278 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:05:43.276+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:05:43.276930+0000) 2024-03-04T14:05:44.279 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:05:44.276+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:05:44.277197+0000) 2024-03-04T14:05:45.279 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:05:45.276+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:05:45.277475+0000) 2024-03-04T14:05:46.279 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:05:46.277+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:05:46.277766+0000) 2024-03-04T14:05:47.292 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:05:47.277+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:05:47.278004+0000) 2024-03-04T14:05:48.280 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:05:48.277+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:05:48.278289+0000) 2024-03-04T14:05:49.280 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:05:49.277+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:05:49.278531+0000) 2024-03-04T14:05:50.280 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:05:50.278+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:05:50.278779+0000) 2024-03-04T14:05:51.281 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:05:51.278+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:05:51.279110+0000) 2024-03-04T14:05:52.055 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:05:52.063 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:05:52.281 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:05:52.278+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:05:52.279319+0000) 2024-03-04T14:05:53.281 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:05:53.279+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:05:53.279587+0000) 2024-03-04T14:05:54.281 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:05:54.279+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:05:54.279919+0000) 2024-03-04T14:05:55.282 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:05:55.279+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:05:55.280210+0000) 2024-03-04T14:05:56.282 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:05:56.279+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:05:56.280507+0000) 2024-03-04T14:05:57.282 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:05:57.279+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:05:57.280790+0000) 2024-03-04T14:05:58.283 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:05:58.280+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:05:58.281069+0000) 2024-03-04T14:05:59.283 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:05:59.280+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:05:59.281364+0000) 2024-03-04T14:06:00.284 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:06:00.281+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:06:00.281629+0000) 2024-03-04T14:06:01.284 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:06:01.281+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:06:01.281968+0000) 2024-03-04T14:06:02.284 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:06:02.281+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:06:02.282257+0000) 2024-03-04T14:06:03.284 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:06:03.281+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:06:03.282495+0000) 2024-03-04T14:06:04.284 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:06:04.282+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:06:04.282745+0000) 2024-03-04T14:06:05.285 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:06:05.282+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:06:05.283007+0000) 2024-03-04T14:06:06.285 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:06:06.282+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:06:06.283261+0000) 2024-03-04T14:06:07.285 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:06:07.283+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:06:07.283499+0000) 2024-03-04T14:06:08.285 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:06:08.283+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:06:08.283728+0000) 2024-03-04T14:06:09.286 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:06:09.283+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:06:09.283994+0000) 2024-03-04T14:06:10.287 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:06:10.284+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:06:10.284248+0000) 2024-03-04T14:06:11.286 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:06:11.283+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:06:11.284444+0000) 2024-03-04T14:06:12.286 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:06:12.284+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:06:12.284668+0000) 2024-03-04T14:06:13.286 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:06:13.284+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:06:13.284930+0000) 2024-03-04T14:06:14.287 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:06:14.285+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:06:14.285153+0000) 2024-03-04T14:06:15.287 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:06:15.285+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:06:15.285392+0000) 2024-03-04T14:06:16.287 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:06:16.285+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:06:16.285629+0000) 2024-03-04T14:06:17.287 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:06:17.284+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:06:17.285923+0000) 2024-03-04T14:06:18.288 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:06:18.284+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:06:18.286147+0000) 2024-03-04T14:06:19.288 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:06:19.285+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:06:19.286416+0000) 2024-03-04T14:06:20.288 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:06:20.285+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:06:20.286650+0000) 2024-03-04T14:06:21.288 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:06:21.285+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:06:21.286932+0000) 2024-03-04T14:06:22.284 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:06:22.289 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:06:22.286+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:06:22.287199+0000) 2024-03-04T14:06:22.292 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:06:23.289 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:06:23.286+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:06:23.287489+0000) 2024-03-04T14:06:24.289 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:06:24.286+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:06:24.287783+0000) 2024-03-04T14:06:25.290 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:06:25.287+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:06:25.288125+0000) 2024-03-04T14:06:26.290 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:06:26.287+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:06:26.288371+0000) 2024-03-04T14:06:27.290 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:06:27.287+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:06:27.288569+0000) 2024-03-04T14:06:28.290 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:06:28.288+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:06:28.288865+0000) 2024-03-04T14:06:29.291 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:06:29.288+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:06:29.289115+0000) 2024-03-04T14:06:30.292 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:06:30.288+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:06:30.289364+0000) 2024-03-04T14:06:31.291 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:06:31.288+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:06:31.289603+0000) 2024-03-04T14:06:32.291 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:06:32.289+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:06:32.289872+0000) 2024-03-04T14:06:33.292 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:06:33.289+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:06:33.290106+0000) 2024-03-04T14:06:34.292 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:06:34.289+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:06:34.290341+0000) 2024-03-04T14:06:35.292 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:06:35.289+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:06:35.290581+0000) 2024-03-04T14:06:36.292 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:06:36.290+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:06:36.290905+0000) 2024-03-04T14:06:37.293 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:06:37.290+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:06:37.291142+0000) 2024-03-04T14:06:38.293 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:06:38.290+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:06:38.291376+0000) 2024-03-04T14:06:39.294 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:06:39.290+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:06:39.291618+0000) 2024-03-04T14:06:40.294 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:06:40.291+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:06:40.291895+0000) 2024-03-04T14:06:41.294 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:06:41.291+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:06:41.292133+0000) 2024-03-04T14:06:42.294 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:06:42.291+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:06:42.292382+0000) 2024-03-04T14:06:43.294 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:06:43.291+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:06:43.292613+0000) 2024-03-04T14:06:44.295 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:06:44.292+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:06:44.292926+0000) 2024-03-04T14:06:45.295 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:06:45.292+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:06:45.293156+0000) 2024-03-04T14:06:46.295 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:06:46.292+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:06:46.293396+0000) 2024-03-04T14:06:47.295 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:06:47.293+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:06:47.293657+0000) 2024-03-04T14:06:48.296 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:06:48.293+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:06:48.293931+0000) 2024-03-04T14:06:49.296 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:06:49.293+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:06:49.294177+0000) 2024-03-04T14:06:50.296 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:06:50.293+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:06:50.294411+0000) 2024-03-04T14:06:51.296 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:06:51.294+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:06:51.294670+0000) 2024-03-04T14:06:52.297 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:06:52.294+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:06:52.294954+0000) 2024-03-04T14:06:52.422 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:06:52.430 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:06:53.297 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:06:53.294+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:06:53.295188+0000) 2024-03-04T14:06:54.297 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:06:54.294+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:06:54.295423+0000) 2024-03-04T14:06:55.297 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:06:55.295+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:06:55.295658+0000) 2024-03-04T14:06:56.297 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:06:56.295+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:06:56.295923+0000) 2024-03-04T14:06:57.298 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:06:57.295+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:06:57.296190+0000) 2024-03-04T14:06:58.298 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:06:58.296+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:06:58.296480+0000) 2024-03-04T14:06:59.298 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:06:59.296+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:06:59.296789+0000) 2024-03-04T14:07:00.299 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:07:00.296+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:07:00.297066+0000) 2024-03-04T14:07:01.299 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:07:01.296+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:07:01.297341+0000) 2024-03-04T14:07:02.299 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:07:02.297+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:07:02.297634+0000) 2024-03-04T14:07:03.300 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:07:03.297+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:07:03.297953+0000) 2024-03-04T14:07:04.300 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:07:04.297+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:07:04.298247+0000) 2024-03-04T14:07:05.300 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:07:05.297+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:07:05.298563+0000) 2024-03-04T14:07:06.301 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:07:06.298+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:07:06.298815+0000) 2024-03-04T14:07:07.301 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:07:07.298+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:07:07.299091+0000) 2024-03-04T14:07:08.301 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:07:08.298+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:07:08.299345+0000) 2024-03-04T14:07:09.301 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:07:09.299+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:07:09.299667+0000) 2024-03-04T14:07:10.302 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:07:10.299+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:07:10.299978+0000) 2024-03-04T14:07:11.302 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:07:11.300+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:07:11.300275+0000) 2024-03-04T14:07:12.302 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:07:12.300+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:07:12.300511+0000) 2024-03-04T14:07:13.303 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:07:13.299+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:07:13.300799+0000) 2024-03-04T14:07:14.303 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:07:14.299+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:07:14.301140+0000) 2024-03-04T14:07:15.303 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:07:15.300+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:07:15.301452+0000) 2024-03-04T14:07:16.305 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:07:16.300+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:07:16.301768+0000) 2024-03-04T14:07:17.304 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:07:17.300+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:07:17.302073+0000) 2024-03-04T14:07:18.304 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:07:18.301+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:07:18.302360+0000) 2024-03-04T14:07:19.304 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:07:19.301+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:07:19.302715+0000) 2024-03-04T14:07:20.305 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:07:20.301+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:07:20.303009+0000) 2024-03-04T14:07:21.305 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:07:21.301+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:07:21.303282+0000) 2024-03-04T14:07:22.305 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:07:22.302+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:07:22.303561+0000) 2024-03-04T14:07:22.619 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:07:22.626 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:07:23.305 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:07:23.302+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:07:23.303870+0000) 2024-03-04T14:07:24.306 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:07:24.303+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:07:24.304148+0000) 2024-03-04T14:07:25.306 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:07:25.303+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:07:25.304440+0000) 2024-03-04T14:07:26.306 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:07:26.303+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:07:26.304722+0000) 2024-03-04T14:07:27.307 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:07:27.304+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:07:27.305025+0000) 2024-03-04T14:07:28.307 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:07:28.304+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:07:28.305275+0000) 2024-03-04T14:07:29.307 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:07:29.304+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:07:29.305519+0000) 2024-03-04T14:07:30.308 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:07:30.304+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:07:30.305796+0000) 2024-03-04T14:07:31.308 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:07:31.305+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:07:31.306149+0000) 2024-03-04T14:07:32.308 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:07:32.305+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:07:32.306442+0000) 2024-03-04T14:07:33.308 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:07:33.305+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:07:33.306746+0000) 2024-03-04T14:07:34.309 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:07:34.306+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:07:34.307036+0000) 2024-03-04T14:07:35.310 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:07:35.306+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:07:35.307328+0000) 2024-03-04T14:07:36.309 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:07:36.306+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:07:36.307560+0000) 2024-03-04T14:07:37.310 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:07:37.307+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:07:37.307892+0000) 2024-03-04T14:07:38.310 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:07:38.307+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:07:38.308178+0000) 2024-03-04T14:07:39.310 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:07:39.307+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:07:39.308481+0000) 2024-03-04T14:07:40.311 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:07:40.308+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:07:40.308796+0000) 2024-03-04T14:07:41.311 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:07:41.308+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:07:41.309135+0000) 2024-03-04T14:07:42.311 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:07:42.308+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:07:42.309401+0000) 2024-03-04T14:07:43.311 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:07:43.309+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:07:43.309689+0000) 2024-03-04T14:07:44.312 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:07:44.309+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:07:44.309976+0000) 2024-03-04T14:07:45.312 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:07:45.309+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:07:45.310264+0000) 2024-03-04T14:07:46.312 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:07:46.309+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:07:46.310565+0000) 2024-03-04T14:07:47.313 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:07:47.310+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:07:47.310943+0000) 2024-03-04T14:07:48.313 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:07:48.310+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:07:48.311229+0000) 2024-03-04T14:07:49.313 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:07:49.310+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:07:49.311534+0000) 2024-03-04T14:07:50.314 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:07:50.311+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:07:50.311874+0000) 2024-03-04T14:07:51.314 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:07:51.311+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:07:51.312169+0000) 2024-03-04T14:07:52.314 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:07:52.311+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:07:52.312476+0000) 2024-03-04T14:07:52.782 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:07:52.790 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:07:53.314 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:07:53.312+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:07:53.312767+0000) 2024-03-04T14:07:54.315 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:07:54.312+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:07:54.313057+0000) 2024-03-04T14:07:55.315 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:07:55.312+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:07:55.313337+0000) 2024-03-04T14:07:56.315 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:07:56.313+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:07:56.313624+0000) 2024-03-04T14:07:57.316 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:07:57.313+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:07:57.313957+0000) 2024-03-04T14:07:58.316 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:07:58.313+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:07:58.314247+0000) 2024-03-04T14:07:59.316 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:07:59.314+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:07:59.314548+0000) 2024-03-04T14:08:00.317 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:08:00.314+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:08:00.314905+0000) 2024-03-04T14:08:01.317 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:08:01.314+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:08:01.315205+0000) 2024-03-04T14:08:02.317 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:08:02.314+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:08:02.315456+0000) 2024-03-04T14:08:03.317 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:08:03.315+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:08:03.315747+0000) 2024-03-04T14:08:04.318 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:08:04.315+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:08:04.316037+0000) 2024-03-04T14:08:05.318 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:08:05.315+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:08:05.316288+0000) 2024-03-04T14:08:06.318 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:08:06.316+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:08:06.316596+0000) 2024-03-04T14:08:07.319 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:08:07.316+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:08:07.316947+0000) 2024-03-04T14:08:08.319 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:08:08.316+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:08:08.317230+0000) 2024-03-04T14:08:09.319 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:08:09.317+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:08:09.317529+0000) 2024-03-04T14:08:10.320 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:08:10.317+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:08:10.317804+0000) 2024-03-04T14:08:11.320 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:08:11.317+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:08:11.318149+0000) 2024-03-04T14:08:12.320 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:08:12.318+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:08:12.318490+0000) 2024-03-04T14:08:13.321 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:08:13.318+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:08:13.318776+0000) 2024-03-04T14:08:14.321 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:08:14.318+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:08:14.319064+0000) 2024-03-04T14:08:15.321 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:08:15.318+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:08:15.319358+0000) 2024-03-04T14:08:16.321 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:08:16.318+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:08:16.319652+0000) 2024-03-04T14:08:17.322 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:08:17.318+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:08:17.319930+0000) 2024-03-04T14:08:18.322 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:08:18.319+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:08:18.320218+0000) 2024-03-04T14:08:19.322 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:08:19.319+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:08:19.320504+0000) 2024-03-04T14:08:20.323 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:08:20.319+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:08:20.320788+0000) 2024-03-04T14:08:21.323 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:08:21.320+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:08:21.321137+0000) 2024-03-04T14:08:22.323 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:08:22.320+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:08:22.321418+0000) 2024-03-04T14:08:23.004 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:08:23.012 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:08:23.324 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:08:23.320+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:08:23.321730+0000) 2024-03-04T14:08:24.324 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:08:24.321+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:08:24.322017+0000) 2024-03-04T14:08:25.324 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:08:25.321+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:08:25.322313+0000) 2024-03-04T14:08:26.324 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:08:26.321+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:08:26.322595+0000) 2024-03-04T14:08:27.325 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:08:27.322+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:08:27.322924+0000) 2024-03-04T14:08:28.325 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:08:28.322+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:08:28.323216+0000) 2024-03-04T14:08:29.325 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:08:29.322+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:08:29.323516+0000) 2024-03-04T14:08:30.326 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:08:30.322+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:08:30.323803+0000) 2024-03-04T14:08:31.326 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:08:31.323+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:08:31.324079+0000) 2024-03-04T14:08:32.326 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:08:32.323+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:08:32.324354+0000) 2024-03-04T14:08:33.326 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:08:33.323+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:08:33.324635+0000) 2024-03-04T14:08:34.327 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:08:34.324+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:08:34.324960+0000) 2024-03-04T14:08:35.327 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:08:35.324+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:08:35.325250+0000) 2024-03-04T14:08:36.327 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:08:36.324+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:08:36.325541+0000) 2024-03-04T14:08:37.328 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:08:37.325+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:08:37.325883+0000) 2024-03-04T14:08:38.328 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:08:38.325+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:08:38.326167+0000) 2024-03-04T14:08:39.328 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:08:39.325+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:08:39.326456+0000) 2024-03-04T14:08:40.329 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:08:40.325+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:08:40.326738+0000) 2024-03-04T14:08:41.329 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:08:41.326+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:08:41.327030+0000) 2024-03-04T14:08:42.329 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:08:42.326+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:08:42.327313+0000) 2024-03-04T14:08:43.329 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:08:43.326+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:08:43.327601+0000) 2024-03-04T14:08:44.330 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:08:44.327+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:08:44.327957+0000) 2024-03-04T14:08:45.330 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:08:45.327+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:08:45.328254+0000) 2024-03-04T14:08:46.330 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:08:46.327+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:08:46.328493+0000) 2024-03-04T14:08:47.331 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:08:47.327+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:08:47.328778+0000) 2024-03-04T14:08:48.331 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:08:48.328+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:08:48.329086+0000) 2024-03-04T14:08:49.331 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:08:49.328+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:08:49.329394+0000) 2024-03-04T14:08:50.332 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:08:50.329+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:08:50.329687+0000) 2024-03-04T14:08:51.332 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:08:51.329+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:08:51.329987+0000) 2024-03-04T14:08:52.332 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:08:52.329+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:08:52.330272+0000) 2024-03-04T14:08:53.301 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:08:53.309 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:08:53.332 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:08:53.329+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:08:53.330494+0000) 2024-03-04T14:08:54.333 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:08:54.329+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:08:54.330798+0000) 2024-03-04T14:08:55.333 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:08:55.330+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:08:55.331152+0000) 2024-03-04T14:08:56.333 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:08:56.330+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:08:56.331404+0000) 2024-03-04T14:08:57.333 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:08:57.331+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:08:57.331653+0000) 2024-03-04T14:08:58.334 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:08:58.331+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:08:58.331937+0000) 2024-03-04T14:08:59.334 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:08:59.331+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:08:59.332187+0000) 2024-03-04T14:09:00.334 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:09:00.332+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:09:00.332427+0000) 2024-03-04T14:09:01.335 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:09:01.332+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:09:01.332674+0000) 2024-03-04T14:09:02.335 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:09:02.332+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:09:02.332898+0000) 2024-03-04T14:09:03.335 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:09:03.332+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:09:03.333128+0000) 2024-03-04T14:09:04.335 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:09:04.333+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:09:04.333369+0000) 2024-03-04T14:09:05.335 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:09:05.332+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:09:05.333628+0000) 2024-03-04T14:09:06.336 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:09:06.333+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:09:06.333948+0000) 2024-03-04T14:09:07.336 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:09:07.333+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:09:07.334188+0000) 2024-03-04T14:09:08.336 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:09:08.334+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:09:08.334426+0000) 2024-03-04T14:09:09.336 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:09:09.333+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:09:09.334665+0000) 2024-03-04T14:09:10.337 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:09:10.334+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:09:10.334927+0000) 2024-03-04T14:09:11.338 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:09:11.334+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:09:11.335195+0000) 2024-03-04T14:09:12.337 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:09:12.335+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:09:12.335445+0000) 2024-03-04T14:09:13.338 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:09:13.335+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:09:13.335716+0000) 2024-03-04T14:09:14.338 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:09:14.335+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:09:14.336000+0000) 2024-03-04T14:09:15.338 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:09:15.335+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:09:15.336260+0000) 2024-03-04T14:09:16.338 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:09:16.336+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:09:16.336524+0000) 2024-03-04T14:09:17.339 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:09:17.336+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:09:17.336778+0000) 2024-03-04T14:09:18.339 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:09:18.335+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:09:18.336953+0000) 2024-03-04T14:09:19.339 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:09:19.336+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:09:19.337215+0000) 2024-03-04T14:09:20.339 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:09:20.336+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:09:20.337462+0000) 2024-03-04T14:09:21.340 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:09:21.335+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:09:21.337652+0000) 2024-03-04T14:09:22.340 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:09:22.336+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:09:22.337943+0000) 2024-03-04T14:09:23.340 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:09:23.336+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:09:23.338175+0000) 2024-03-04T14:09:23.441 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:09:23.449 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:09:24.340 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:09:24.337+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:09:24.338414+0000) 2024-03-04T14:09:25.341 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:09:25.337+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:09:25.338649+0000) 2024-03-04T14:09:26.341 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:09:26.337+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:09:26.338933+0000) 2024-03-04T14:09:27.341 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:09:27.338+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:09:27.339136+0000) 2024-03-04T14:09:28.341 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:09:28.338+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:09:28.339322+0000) 2024-03-04T14:09:29.341 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:09:29.338+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:09:29.339590+0000) 2024-03-04T14:09:30.342 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:09:30.339+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:09:30.339854+0000) 2024-03-04T14:09:31.342 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:09:31.339+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:09:31.340120+0000) 2024-03-04T14:09:32.342 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:09:32.339+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:09:32.340365+0000) 2024-03-04T14:09:33.342 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:09:33.339+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:09:33.340600+0000) 2024-03-04T14:09:34.343 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:09:34.340+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:09:34.340955+0000) 2024-03-04T14:09:35.343 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:09:35.340+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:09:35.341253+0000) 2024-03-04T14:09:36.343 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:09:36.340+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:09:36.341543+0000) 2024-03-04T14:09:37.344 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:09:37.340+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:09:37.341780+0000) 2024-03-04T14:09:38.344 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:09:38.341+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:09:38.342122+0000) 2024-03-04T14:09:39.345 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:09:39.341+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:09:39.342393+0000) 2024-03-04T14:09:40.345 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:09:40.341+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:09:40.342626+0000) 2024-03-04T14:09:41.345 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:09:41.342+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:09:41.342919+0000) 2024-03-04T14:09:42.345 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:09:42.342+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:09:42.343119+0000) 2024-03-04T14:09:43.345 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:09:43.342+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:09:43.343310+0000) 2024-03-04T14:09:44.345 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:09:44.342+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:09:44.343559+0000) 2024-03-04T14:09:45.346 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:09:45.343+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:09:45.343810+0000) 2024-03-04T14:09:46.346 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:09:46.343+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:09:46.344214+0000) 2024-03-04T14:09:47.346 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:09:47.343+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:09:47.344479+0000) 2024-03-04T14:09:48.347 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:09:48.344+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:09:48.344721+0000) 2024-03-04T14:09:49.347 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:09:49.344+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:09:49.344970+0000) 2024-03-04T14:09:50.347 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:09:50.344+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:09:50.345206+0000) 2024-03-04T14:09:51.347 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:09:51.344+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:09:51.345454+0000) 2024-03-04T14:09:52.348 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:09:52.344+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:09:52.345730+0000) 2024-03-04T14:09:53.348 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:09:53.345+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:09:53.345969+0000) 2024-03-04T14:09:53.628 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:09:53.636 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:09:54.348 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:09:54.345+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:09:54.346216+0000) 2024-03-04T14:09:55.349 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:09:55.345+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:09:55.346461+0000) 2024-03-04T14:09:56.349 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:09:56.346+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:09:56.346727+0000) 2024-03-04T14:09:57.349 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:09:57.346+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:09:57.347009+0000) 2024-03-04T14:09:58.350 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:09:58.346+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:09:58.347245+0000) 2024-03-04T14:09:59.350 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:09:59.347+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:09:59.347547+0000) 2024-03-04T14:10:00.350 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:10:00.347+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:10:00.347785+0000) 2024-03-04T14:10:01.350 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:10:01.347+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:10:01.348037+0000) 2024-03-04T14:10:02.350 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:10:02.347+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:10:02.348231+0000) 2024-03-04T14:10:03.350 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:10:03.348+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:10:03.348463+0000) 2024-03-04T14:10:04.350 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:10:04.348+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:10:04.348706+0000) 2024-03-04T14:10:05.351 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:10:05.348+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:10:05.348951+0000) 2024-03-04T14:10:06.351 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:10:06.348+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:10:06.349197+0000) 2024-03-04T14:10:07.351 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:10:07.349+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:10:07.349452+0000) 2024-03-04T14:10:08.352 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:10:08.349+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:10:08.349687+0000) 2024-03-04T14:10:09.352 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:10:09.349+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:10:09.349933+0000) 2024-03-04T14:10:10.352 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:10:10.349+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:10:10.350164+0000) 2024-03-04T14:10:11.352 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:10:11.350+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:10:11.350422+0000) 2024-03-04T14:10:12.353 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:10:12.350+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:10:12.350668+0000) 2024-03-04T14:10:13.353 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:10:13.350+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:10:13.350928+0000) 2024-03-04T14:10:14.353 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:10:14.350+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:10:14.351168+0000) 2024-03-04T14:10:15.353 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:10:15.351+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:10:15.351403+0000) 2024-03-04T14:10:16.354 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:10:16.351+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:10:16.351640+0000) 2024-03-04T14:10:17.354 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:10:17.351+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:10:17.351935+0000) 2024-03-04T14:10:18.354 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:10:18.352+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:10:18.352175+0000) 2024-03-04T14:10:19.354 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:10:19.352+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:10:19.352423+0000) 2024-03-04T14:10:20.355 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:10:20.351+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:10:20.352662+0000) 2024-03-04T14:10:21.355 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:10:21.351+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:10:21.352944+0000) 2024-03-04T14:10:22.355 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:10:22.352+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:10:22.353191+0000) 2024-03-04T14:10:23.355 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:10:23.352+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:10:23.353427+0000) 2024-03-04T14:10:23.808 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:10:23.816 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:10:24.356 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:10:24.352+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:10:24.353676+0000) 2024-03-04T14:10:25.356 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:10:25.352+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:10:25.353989+0000) 2024-03-04T14:10:26.356 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:10:26.353+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:10:26.354308+0000) 2024-03-04T14:10:27.356 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:10:27.352+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:10:27.354602+0000) 2024-03-04T14:10:28.357 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:10:28.353+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:10:28.354933+0000) 2024-03-04T14:10:29.359 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:10:29.353+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:10:29.355230+0000) 2024-03-04T14:10:30.357 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:10:30.354+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:10:30.355523+0000) 2024-03-04T14:10:31.358 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:10:31.355+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:10:31.355818+0000) 2024-03-04T14:10:32.358 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:10:32.355+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:10:32.356177+0000) 2024-03-04T14:10:33.358 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:10:33.355+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:10:33.356466+0000) 2024-03-04T14:10:34.359 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:10:34.355+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:10:34.356750+0000) 2024-03-04T14:10:35.359 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:10:35.356+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:10:35.357039+0000) 2024-03-04T14:10:36.359 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:10:36.356+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:10:36.357317+0000) 2024-03-04T14:10:37.360 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:10:37.356+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:10:37.357624+0000) 2024-03-04T14:10:38.360 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:10:38.357+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:10:38.357959+0000) 2024-03-04T14:10:39.360 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:10:39.357+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:10:39.358261+0000) 2024-03-04T14:10:40.360 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:10:40.357+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:10:40.358550+0000) 2024-03-04T14:10:41.361 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:10:41.358+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:10:41.358874+0000) 2024-03-04T14:10:42.361 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:10:42.358+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:10:42.359209+0000) 2024-03-04T14:10:43.361 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:10:43.358+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:10:43.359492+0000) 2024-03-04T14:10:44.362 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:10:44.359+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:10:44.359779+0000) 2024-03-04T14:10:45.362 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:10:45.359+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:10:45.360072+0000) 2024-03-04T14:10:46.362 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:10:46.359+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:10:46.360362+0000) 2024-03-04T14:10:47.363 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:10:47.359+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:10:47.360654+0000) 2024-03-04T14:10:48.363 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:10:48.360+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:10:48.360971+0000) 2024-03-04T14:10:49.363 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:10:49.360+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:10:49.361286+0000) 2024-03-04T14:10:50.363 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:10:50.360+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:10:50.361575+0000) 2024-03-04T14:10:51.364 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:10:51.361+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:10:51.361919+0000) 2024-03-04T14:10:52.364 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:10:52.361+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:10:52.362226+0000) 2024-03-04T14:10:53.364 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:10:53.361+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:10:53.362503+0000) 2024-03-04T14:10:54.004 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:10:54.012 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:10:54.365 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:10:54.362+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:10:54.362804+0000) 2024-03-04T14:10:55.365 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:10:55.362+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:10:55.363148+0000) 2024-03-04T14:10:56.365 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:10:56.362+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:10:56.363464+0000) 2024-03-04T14:10:57.366 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:10:57.363+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:10:57.363756+0000) 2024-03-04T14:10:58.366 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:10:58.363+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:10:58.364043+0000) 2024-03-04T14:10:59.366 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:10:59.363+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:10:59.364348+0000) 2024-03-04T14:11:00.367 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:11:00.364+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:11:00.364633+0000) 2024-03-04T14:11:01.367 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:11:01.364+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:11:01.364992+0000) 2024-03-04T14:11:02.367 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:11:02.364+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:11:02.365286+0000) 2024-03-04T14:11:03.368 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:11:03.365+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:11:03.365583+0000) 2024-03-04T14:11:04.368 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:11:04.365+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:11:04.365917+0000) 2024-03-04T14:11:05.368 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:11:05.365+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:11:05.366139+0000) 2024-03-04T14:11:06.368 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:11:06.366+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:11:06.366433+0000) 2024-03-04T14:11:07.369 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:11:07.366+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:11:07.366761+0000) 2024-03-04T14:11:08.369 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:11:08.366+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:11:08.367043+0000) 2024-03-04T14:11:09.369 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:11:09.366+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:11:09.367336+0000) 2024-03-04T14:11:10.375 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:11:10.367+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:11:10.367621+0000) 2024-03-04T14:11:11.370 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:11:11.367+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:11:11.367948+0000) 2024-03-04T14:11:12.370 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:11:12.367+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:11:12.368255+0000) 2024-03-04T14:11:13.371 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:11:13.368+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:11:13.368541+0000) 2024-03-04T14:11:14.371 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:11:14.368+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:11:14.368873+0000) 2024-03-04T14:11:15.371 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:11:15.368+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:11:15.369166+0000) 2024-03-04T14:11:16.371 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:11:16.369+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:11:16.369447+0000) 2024-03-04T14:11:17.372 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:11:17.369+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:11:17.369758+0000) 2024-03-04T14:11:18.372 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:11:18.369+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:11:18.370081+0000) 2024-03-04T14:11:19.372 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:11:19.369+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:11:19.370321+0000) 2024-03-04T14:11:20.373 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:11:20.369+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:11:20.370625+0000) 2024-03-04T14:11:21.373 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:11:21.370+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:11:21.370960+0000) 2024-03-04T14:11:22.374 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:11:22.371+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:11:22.371248+0000) 2024-03-04T14:11:23.374 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:11:23.370+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:11:23.371530+0000) 2024-03-04T14:11:24.218 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:11:24.226 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:11:24.374 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:11:24.370+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:11:24.371901+0000) 2024-03-04T14:11:25.375 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:11:25.371+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:11:25.372195+0000) 2024-03-04T14:11:26.375 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:11:26.371+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:11:26.372484+0000) 2024-03-04T14:11:27.375 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:11:27.371+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:11:27.372785+0000) 2024-03-04T14:11:28.375 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:11:28.372+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:11:28.373102+0000) 2024-03-04T14:11:29.375 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:11:29.372+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:11:29.373408+0000) 2024-03-04T14:11:30.376 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:11:30.372+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:11:30.373772+0000) 2024-03-04T14:11:31.376 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:11:31.373+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:11:31.374071+0000) 2024-03-04T14:11:32.376 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:11:32.373+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:11:32.374333+0000) 2024-03-04T14:11:33.376 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:11:33.373+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:11:33.374572+0000) 2024-03-04T14:11:34.377 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:11:34.374+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:11:34.374920+0000) 2024-03-04T14:11:35.377 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:11:35.374+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:11:35.375200+0000) 2024-03-04T14:11:36.378 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:11:36.374+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:11:36.375485+0000) 2024-03-04T14:11:37.378 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:11:37.374+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:11:37.375780+0000) 2024-03-04T14:11:38.378 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:11:38.375+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:11:38.376067+0000) 2024-03-04T14:11:39.378 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:11:39.375+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:11:39.376351+0000) 2024-03-04T14:11:40.379 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:11:40.375+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:11:40.376644+0000) 2024-03-04T14:11:41.379 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:11:41.376+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:11:41.376980+0000) 2024-03-04T14:11:42.379 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:11:42.376+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:11:42.377281+0000) 2024-03-04T14:11:43.379 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:11:43.376+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:11:43.377559+0000) 2024-03-04T14:11:44.380 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:11:44.377+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:11:44.377864+0000) 2024-03-04T14:11:45.380 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:11:45.377+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:11:45.378106+0000) 2024-03-04T14:11:46.380 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:11:46.377+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:11:46.378397+0000) 2024-03-04T14:11:47.381 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:11:47.377+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:11:47.378704+0000) 2024-03-04T14:11:48.381 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:11:48.378+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:11:48.378953+0000) 2024-03-04T14:11:49.381 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:11:49.378+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:11:49.379237+0000) 2024-03-04T14:11:50.382 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:11:50.378+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:11:50.379530+0000) 2024-03-04T14:11:51.382 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:11:51.379+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:11:51.379867+0000) 2024-03-04T14:11:52.382 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:11:52.379+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:11:52.380167+0000) 2024-03-04T14:11:53.382 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:11:53.379+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:11:53.380415+0000) 2024-03-04T14:11:54.383 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:11:54.380+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:11:54.380688+0000) 2024-03-04T14:11:54.413 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:11:54.421 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:11:55.383 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:11:55.380+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:11:55.380981+0000) 2024-03-04T14:11:56.383 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:11:56.380+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:11:56.381274+0000) 2024-03-04T14:11:57.383 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:11:57.380+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:11:57.381572+0000) 2024-03-04T14:11:58.384 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:11:58.381+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:11:58.381851+0000) 2024-03-04T14:11:59.384 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:11:59.381+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:11:59.382168+0000) 2024-03-04T14:12:00.384 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:12:00.381+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:12:00.382458+0000) 2024-03-04T14:12:01.385 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:12:01.382+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:12:01.382755+0000) 2024-03-04T14:12:02.385 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:12:02.382+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:12:02.383054+0000) 2024-03-04T14:12:03.386 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:12:03.382+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:12:03.383303+0000) 2024-03-04T14:12:04.386 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:12:04.383+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:12:04.383600+0000) 2024-03-04T14:12:05.386 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:12:05.383+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:12:05.383914+0000) 2024-03-04T14:12:06.388 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:12:06.383+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:12:06.384219+0000) 2024-03-04T14:12:07.387 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:12:07.384+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:12:07.384520+0000) 2024-03-04T14:12:08.387 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:12:08.384+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:12:08.384781+0000) 2024-03-04T14:12:09.387 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:12:09.384+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:12:09.385088+0000) 2024-03-04T14:12:10.387 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:12:10.384+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:12:10.385380+0000) 2024-03-04T14:12:11.388 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:12:11.385+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:12:11.385671+0000) 2024-03-04T14:12:12.388 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:12:12.385+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:12:12.385957+0000) 2024-03-04T14:12:13.388 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:12:13.385+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:12:13.386207+0000) 2024-03-04T14:12:14.388 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:12:14.385+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:12:14.386499+0000) 2024-03-04T14:12:15.389 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:12:15.386+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:12:15.386787+0000) 2024-03-04T14:12:16.389 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:12:16.386+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:12:16.387131+0000) 2024-03-04T14:12:17.389 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:12:17.387+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:12:17.387424+0000) 2024-03-04T14:12:18.390 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:12:18.387+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:12:18.387671+0000) 2024-03-04T14:12:19.390 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:12:19.387+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:12:19.387990+0000) 2024-03-04T14:12:20.390 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:12:20.388+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:12:20.388277+0000) 2024-03-04T14:12:21.391 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:12:21.388+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:12:21.388565+0000) 2024-03-04T14:12:22.391 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:12:22.388+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:12:22.388887+0000) 2024-03-04T14:12:23.391 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:12:23.388+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:12:23.389145+0000) 2024-03-04T14:12:24.391 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:12:24.389+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:12:24.389421+0000) 2024-03-04T14:12:24.619 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:12:24.627 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:12:25.392 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:12:25.389+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:12:25.389717+0000) 2024-03-04T14:12:26.392 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:12:26.389+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:12:26.390010+0000) 2024-03-04T14:12:27.392 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:12:27.389+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:12:27.390301+0000) 2024-03-04T14:12:28.393 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:12:28.389+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:12:28.390558+0000) 2024-03-04T14:12:29.393 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:12:29.389+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:12:29.390918+0000) 2024-03-04T14:12:30.393 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:12:30.390+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:12:30.391205+0000) 2024-03-04T14:12:31.393 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:12:31.390+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:12:31.391498+0000) 2024-03-04T14:12:32.394 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:12:32.390+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:12:32.391867+0000) 2024-03-04T14:12:33.394 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:12:33.391+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:12:33.392132+0000) 2024-03-04T14:12:34.394 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:12:34.391+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:12:34.392433+0000) 2024-03-04T14:12:35.396 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:12:35.391+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:12:35.392724+0000) 2024-03-04T14:12:36.395 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:12:36.391+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:12:36.393010+0000) 2024-03-04T14:12:37.396 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:12:37.392+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:12:37.393300+0000) 2024-03-04T14:12:38.395 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:12:38.392+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:12:38.393550+0000) 2024-03-04T14:12:39.396 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:12:39.392+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:12:39.393799+0000) 2024-03-04T14:12:40.396 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:12:40.393+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:12:40.394072+0000) 2024-03-04T14:12:41.396 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:12:41.393+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:12:41.394312+0000) 2024-03-04T14:12:42.397 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:12:42.393+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:12:42.394590+0000) 2024-03-04T14:12:43.397 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:12:43.394+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:12:43.394856+0000) 2024-03-04T14:12:44.397 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:12:44.394+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:12:44.395102+0000) 2024-03-04T14:12:45.397 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:12:45.394+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:12:45.395341+0000) 2024-03-04T14:12:46.398 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:12:46.394+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:12:46.395576+0000) 2024-03-04T14:12:47.398 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:12:47.395+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:12:47.395810+0000) 2024-03-04T14:12:48.398 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:12:48.395+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:12:48.396028+0000) 2024-03-04T14:12:49.398 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:12:49.395+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:12:49.396292+0000) 2024-03-04T14:12:50.399 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:12:50.395+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:12:50.396592+0000) 2024-03-04T14:12:51.399 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:12:51.396+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:12:51.396927+0000) 2024-03-04T14:12:52.399 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:12:52.396+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:12:52.397215+0000) 2024-03-04T14:12:53.399 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:12:53.396+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:12:53.397461+0000) 2024-03-04T14:12:54.400 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:12:54.397+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:12:54.397701+0000) 2024-03-04T14:12:54.798 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:12:54.806 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:12:55.400 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:12:55.397+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:12:55.397950+0000) 2024-03-04T14:12:56.400 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:12:56.397+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:12:56.398190+0000) 2024-03-04T14:12:57.400 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:12:57.397+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:12:57.398437+0000) 2024-03-04T14:12:58.401 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:12:58.398+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:12:58.398679+0000) 2024-03-04T14:12:59.401 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:12:59.398+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:12:59.398927+0000) 2024-03-04T14:13:00.401 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:13:00.398+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:13:00.399163+0000) 2024-03-04T14:13:01.401 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:13:01.398+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:13:01.399401+0000) 2024-03-04T14:13:02.401 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:13:02.398+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:13:02.399595+0000) 2024-03-04T14:13:03.402 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:13:03.399+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:13:03.399858+0000) 2024-03-04T14:13:04.402 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:13:04.399+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:13:04.400114+0000) 2024-03-04T14:13:05.402 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:13:05.399+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:13:05.400370+0000) 2024-03-04T14:13:06.402 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:13:06.399+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:13:06.400555+0000) 2024-03-04T14:13:07.403 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:13:07.400+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:13:07.400796+0000) 2024-03-04T14:13:08.403 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:13:08.400+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:13:08.401064+0000) 2024-03-04T14:13:09.403 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:13:09.400+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:13:09.401348+0000) 2024-03-04T14:13:10.403 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:13:10.401+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:13:10.401631+0000) 2024-03-04T14:13:11.404 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:13:11.401+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:13:11.401960+0000) 2024-03-04T14:13:12.404 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:13:12.401+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:13:12.402246+0000) 2024-03-04T14:13:13.404 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:13:13.401+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:13:13.402436+0000) 2024-03-04T14:13:14.405 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:13:14.402+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:13:14.402689+0000) 2024-03-04T14:13:15.405 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:13:15.402+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:13:15.402928+0000) 2024-03-04T14:13:16.405 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:13:16.402+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:13:16.403217+0000) 2024-03-04T14:13:17.405 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:13:17.403+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:13:17.403456+0000) 2024-03-04T14:13:18.406 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:13:18.403+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:13:18.403689+0000) 2024-03-04T14:13:19.406 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:13:19.403+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:13:19.403935+0000) 2024-03-04T14:13:20.406 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:13:20.403+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:13:20.404168+0000) 2024-03-04T14:13:21.406 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:13:21.404+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:13:21.404417+0000) 2024-03-04T14:13:22.406 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:13:22.404+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:13:22.404655+0000) 2024-03-04T14:13:23.407 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:13:23.404+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:13:23.404921+0000) 2024-03-04T14:13:24.407 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:13:24.404+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:13:24.405106+0000) 2024-03-04T14:13:25.004 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:13:25.015 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:13:25.407 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:13:25.405+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:13:25.405368+0000) 2024-03-04T14:13:26.407 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:13:26.405+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:13:26.405602+0000) 2024-03-04T14:13:27.408 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:13:27.405+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:13:27.405879+0000) 2024-03-04T14:13:28.408 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:13:28.405+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:13:28.406112+0000) 2024-03-04T14:13:29.408 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:13:29.406+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:13:29.406358+0000) 2024-03-04T14:13:30.408 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:13:30.406+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:13:30.406587+0000) 2024-03-04T14:13:31.408 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:13:31.405+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:13:31.406853+0000) 2024-03-04T14:13:32.409 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:13:32.406+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:13:32.407102+0000) 2024-03-04T14:13:33.409 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:13:33.406+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:13:33.407336+0000) 2024-03-04T14:13:34.409 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:13:34.405+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:13:34.407579+0000) 2024-03-04T14:13:35.409 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:13:35.406+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:13:35.407880+0000) 2024-03-04T14:13:36.410 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:13:36.407+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:13:36.408116+0000) 2024-03-04T14:13:37.410 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:13:37.407+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:13:37.408357+0000) 2024-03-04T14:13:38.410 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:13:38.407+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:13:38.408592+0000) 2024-03-04T14:13:39.411 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:13:39.407+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:13:39.408921+0000) 2024-03-04T14:13:40.411 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:13:40.408+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:13:40.409164+0000) 2024-03-04T14:13:41.411 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:13:41.408+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:13:41.409401+0000) 2024-03-04T14:13:42.411 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:13:42.408+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:13:42.409651+0000) 2024-03-04T14:13:43.412 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:13:43.409+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:13:43.409865+0000) 2024-03-04T14:13:44.412 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:13:44.409+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:13:44.410104+0000) 2024-03-04T14:13:45.412 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:13:45.409+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:13:45.410350+0000) 2024-03-04T14:13:46.412 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:13:46.409+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:13:46.410605+0000) 2024-03-04T14:13:47.412 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:13:47.410+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:13:47.410889+0000) 2024-03-04T14:13:48.413 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:13:48.410+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:13:48.411126+0000) 2024-03-04T14:13:49.413 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:13:49.410+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:13:49.411381+0000) 2024-03-04T14:13:50.413 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:13:50.410+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:13:50.411649+0000) 2024-03-04T14:13:51.413 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:13:51.411+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:13:51.411949+0000) 2024-03-04T14:13:52.414 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:13:52.411+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:13:52.412191+0000) 2024-03-04T14:13:53.414 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:13:53.411+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:13:53.412423+0000) 2024-03-04T14:13:54.414 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:13:54.411+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:13:54.412663+0000) 2024-03-04T14:13:55.207 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:13:55.216 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:13:55.414 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:13:55.412+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:13:55.412963+0000) 2024-03-04T14:13:56.415 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:13:56.412+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:13:56.413197+0000) 2024-03-04T14:13:57.415 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:13:57.412+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:13:57.413444+0000) 2024-03-04T14:13:58.415 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:13:58.412+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:13:58.413696+0000) 2024-03-04T14:13:59.415 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:13:59.413+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:13:59.413980+0000) 2024-03-04T14:14:00.417 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:14:00.413+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:14:00.414217+0000) 2024-03-04T14:14:01.416 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:14:01.413+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:14:01.414454+0000) 2024-03-04T14:14:02.416 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:14:02.413+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:14:02.414698+0000) 2024-03-04T14:14:03.416 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:14:03.414+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:14:03.414933+0000) 2024-03-04T14:14:04.417 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:14:04.414+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:14:04.415244+0000) 2024-03-04T14:14:05.417 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:14:05.414+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:14:05.415501+0000) 2024-03-04T14:14:06.417 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:14:06.415+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:14:06.415737+0000) 2024-03-04T14:14:07.417 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:14:07.415+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:14:07.415981+0000) 2024-03-04T14:14:08.418 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:14:08.415+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:14:08.416214+0000) 2024-03-04T14:14:09.418 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:14:09.415+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:14:09.416469+0000) 2024-03-04T14:14:10.418 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:14:10.416+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:14:10.416710+0000) 2024-03-04T14:14:11.418 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:14:11.416+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:14:11.416959+0000) 2024-03-04T14:14:12.419 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:14:12.416+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:14:12.417209+0000) 2024-03-04T14:14:13.419 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:14:13.416+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:14:13.417443+0000) 2024-03-04T14:14:14.419 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:14:14.417+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:14:14.417684+0000) 2024-03-04T14:14:15.419 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:14:15.417+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:14:15.417928+0000) 2024-03-04T14:14:16.420 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:14:16.417+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:14:16.418156+0000) 2024-03-04T14:14:17.420 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:14:17.417+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:14:17.418394+0000) 2024-03-04T14:14:18.420 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:14:18.417+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:14:18.418577+0000) 2024-03-04T14:14:19.421 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:14:19.417+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:14:19.418769+0000) 2024-03-04T14:14:20.420 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:14:20.418+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:14:20.419007+0000) 2024-03-04T14:14:21.421 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:14:21.418+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:14:21.419249+0000) 2024-03-04T14:14:22.421 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:14:22.419+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:14:22.419495+0000) 2024-03-04T14:14:23.421 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:14:23.419+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:14:23.419735+0000) 2024-03-04T14:14:24.421 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:14:24.419+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:14:24.419990+0000) 2024-03-04T14:14:25.412 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:14:25.420 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:14:25.422 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:14:25.419+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:14:25.420227+0000) 2024-03-04T14:14:26.422 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:14:26.420+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:14:26.420470+0000) 2024-03-04T14:14:27.422 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:14:27.420+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:14:27.420719+0000) 2024-03-04T14:14:28.422 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:14:28.420+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:14:28.420956+0000) 2024-03-04T14:14:29.422 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:14:29.421+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:14:29.421200+0000) 2024-03-04T14:14:30.423 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:14:30.420+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:14:30.421436+0000) 2024-03-04T14:14:31.423 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:14:31.421+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:14:31.421697+0000) 2024-03-04T14:14:32.423 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:14:32.421+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:14:32.421946+0000) 2024-03-04T14:14:33.424 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:14:33.422+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:14:33.422182+0000) 2024-03-04T14:14:34.424 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:14:34.421+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:14:34.422426+0000) 2024-03-04T14:14:35.424 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:14:35.421+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:14:35.422695+0000) 2024-03-04T14:14:36.424 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:14:36.421+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:14:36.422957+0000) 2024-03-04T14:14:37.424 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:14:37.422+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:14:37.423201+0000) 2024-03-04T14:14:38.425 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:14:38.422+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:14:38.423467+0000) 2024-03-04T14:14:39.425 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:14:39.422+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:14:39.423733+0000) 2024-03-04T14:14:40.425 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:14:40.422+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:14:40.423970+0000) 2024-03-04T14:14:41.425 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:14:41.423+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:14:41.424215+0000) 2024-03-04T14:14:42.426 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:14:42.423+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:14:42.424462+0000) 2024-03-04T14:14:43.426 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:14:43.423+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:14:43.424709+0000) 2024-03-04T14:14:44.426 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:14:44.423+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:14:44.424945+0000) 2024-03-04T14:14:45.427 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:14:45.424+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:14:45.425177+0000) 2024-03-04T14:14:46.427 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:14:46.424+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:14:46.425416+0000) 2024-03-04T14:14:47.427 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:14:47.424+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:14:47.425661+0000) 2024-03-04T14:14:48.427 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:14:48.425+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:14:48.425908+0000) 2024-03-04T14:14:49.427 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:14:49.425+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:14:49.426154+0000) 2024-03-04T14:14:50.428 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:14:50.425+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:14:50.426395+0000) 2024-03-04T14:14:51.428 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:14:51.425+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:14:51.426650+0000) 2024-03-04T14:14:52.428 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:14:52.426+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:14:52.426975+0000) 2024-03-04T14:14:53.429 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:14:53.426+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:14:53.427220+0000) 2024-03-04T14:14:54.429 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:14:54.426+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:14:54.427479+0000) 2024-03-04T14:14:55.429 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:14:55.426+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:14:55.427718+0000) 2024-03-04T14:14:55.550 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:14:55.558 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:14:56.429 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:14:56.427+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:14:56.427953+0000) 2024-03-04T14:14:57.429 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:14:57.427+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:14:57.428195+0000) 2024-03-04T14:14:58.430 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:14:58.427+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:14:58.428399+0000) 2024-03-04T14:14:59.430 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:14:59.427+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:14:59.428648+0000) 2024-03-04T14:15:00.430 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:15:00.428+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:15:00.428931+0000) 2024-03-04T14:15:01.430 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:15:01.428+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:15:01.429166+0000) 2024-03-04T14:15:02.431 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:15:02.428+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:15:02.429405+0000) 2024-03-04T14:15:03.431 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:15:03.429+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:15:03.429683+0000) 2024-03-04T14:15:04.431 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:15:04.429+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:15:04.429925+0000) 2024-03-04T14:15:05.431 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:15:05.429+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:15:05.430168+0000) 2024-03-04T14:15:06.433 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:15:06.429+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:15:06.430430+0000) 2024-03-04T14:15:07.432 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:15:07.429+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:15:07.430682+0000) 2024-03-04T14:15:08.432 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:15:08.430+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:15:08.430958+0000) 2024-03-04T14:15:09.432 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:15:09.430+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:15:09.431160+0000) 2024-03-04T14:15:10.432 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:15:10.430+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:15:10.431343+0000) 2024-03-04T14:15:11.433 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:15:11.430+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:15:11.431521+0000) 2024-03-04T14:15:12.433 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:15:12.431+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:15:12.431756+0000) 2024-03-04T14:15:13.433 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:15:13.431+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:15:13.432017+0000) 2024-03-04T14:15:14.433 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:15:14.431+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:15:14.432259+0000) 2024-03-04T14:15:15.433 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:15:15.431+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:15:15.432452+0000) 2024-03-04T14:15:16.434 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:15:16.432+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:15:16.432704+0000) 2024-03-04T14:15:17.434 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:15:17.432+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:15:17.432954+0000) 2024-03-04T14:15:18.434 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:15:18.432+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:15:18.433208+0000) 2024-03-04T14:15:19.435 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:15:19.432+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:15:19.433462+0000) 2024-03-04T14:15:20.435 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:15:20.433+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:15:20.433696+0000) 2024-03-04T14:15:21.435 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:15:21.433+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:15:21.433939+0000) 2024-03-04T14:15:22.435 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:15:22.433+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:15:22.434177+0000) 2024-03-04T14:15:23.436 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:15:23.433+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:15:23.434424+0000) 2024-03-04T14:15:24.436 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:15:24.434+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:15:24.434671+0000) 2024-03-04T14:15:25.436 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:15:25.434+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:15:25.434939+0000) 2024-03-04T14:15:25.697 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:15:25.705 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:15:26.436 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:15:26.434+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:15:26.435174+0000) 2024-03-04T14:15:27.436 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:15:27.435+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:15:27.435442+0000) 2024-03-04T14:15:28.437 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:15:28.435+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:15:28.435721+0000) 2024-03-04T14:15:29.437 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:15:29.435+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:15:29.436017+0000) 2024-03-04T14:15:30.437 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:15:30.436+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:15:30.436312+0000) 2024-03-04T14:15:31.438 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:15:31.436+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:15:31.436606+0000) 2024-03-04T14:15:32.438 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:15:32.436+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:15:32.436957+0000) 2024-03-04T14:15:33.438 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:15:33.437+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:15:33.437250+0000) 2024-03-04T14:15:34.438 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:15:34.437+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:15:34.437544+0000) 2024-03-04T14:15:35.439 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:15:35.437+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:15:35.437887+0000) 2024-03-04T14:15:36.439 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:15:36.438+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:15:36.438210+0000) 2024-03-04T14:15:37.439 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:15:37.437+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:15:37.438455+0000) 2024-03-04T14:15:38.440 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:15:38.437+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:15:38.438679+0000) 2024-03-04T14:15:39.440 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:15:39.437+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:15:39.439006+0000) 2024-03-04T14:15:40.440 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:15:40.438+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:15:40.439352+0000) 2024-03-04T14:15:41.441 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:15:41.438+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:15:41.439645+0000) 2024-03-04T14:15:42.441 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:15:42.438+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:15:42.439960+0000) 2024-03-04T14:15:43.441 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:15:43.439+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:15:43.440287+0000) 2024-03-04T14:15:44.441 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:15:44.439+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:15:44.440576+0000) 2024-03-04T14:15:45.442 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:15:45.439+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:15:45.440899+0000) 2024-03-04T14:15:46.442 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:15:46.440+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:15:46.441194+0000) 2024-03-04T14:15:47.442 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:15:47.440+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:15:47.441490+0000) 2024-03-04T14:15:48.443 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:15:48.440+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:15:48.441784+0000) 2024-03-04T14:15:49.444 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:15:49.441+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:15:49.442132+0000) 2024-03-04T14:15:50.443 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:15:50.441+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:15:50.442424+0000) 2024-03-04T14:15:51.444 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:15:51.441+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:15:51.442712+0000) 2024-03-04T14:15:52.444 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:15:52.442+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:15:52.442997+0000) 2024-03-04T14:15:53.444 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:15:53.442+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:15:53.443289+0000) 2024-03-04T14:15:54.445 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:15:54.442+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:15:54.443576+0000) 2024-03-04T14:15:55.445 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:15:55.443+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:15:55.443923+0000) 2024-03-04T14:15:55.909 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:15:55.921 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:15:56.445 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:15:56.443+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:15:56.444207+0000) 2024-03-04T14:15:57.445 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:15:57.443+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:15:57.444498+0000) 2024-03-04T14:15:58.446 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:15:58.443+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:15:58.444786+0000) 2024-03-04T14:15:59.446 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:15:59.444+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:15:59.445152+0000) 2024-03-04T14:16:00.446 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:16:00.444+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:16:00.445450+0000) 2024-03-04T14:16:01.447 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:16:01.444+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:16:01.445761+0000) 2024-03-04T14:16:02.447 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:16:02.445+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:16:02.446060+0000) 2024-03-04T14:16:03.447 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:16:03.445+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:16:03.446339+0000) 2024-03-04T14:16:04.448 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:16:04.445+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:16:04.446656+0000) 2024-03-04T14:16:05.448 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:16:05.446+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:16:05.446982+0000) 2024-03-04T14:16:06.448 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:16:06.446+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:16:06.447270+0000) 2024-03-04T14:16:07.448 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:16:07.446+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:16:07.447561+0000) 2024-03-04T14:16:08.449 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:16:08.447+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:16:08.447893+0000) 2024-03-04T14:16:09.449 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:16:09.447+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:16:09.448186+0000) 2024-03-04T14:16:10.449 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:16:10.447+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:16:10.448488+0000) 2024-03-04T14:16:11.450 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:16:11.448+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:16:11.448782+0000) 2024-03-04T14:16:12.450 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:16:12.448+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:16:12.449064+0000) 2024-03-04T14:16:13.450 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:16:13.448+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:16:13.449318+0000) 2024-03-04T14:16:14.450 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:16:14.449+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:16:14.449554+0000) 2024-03-04T14:16:15.451 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:16:15.448+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:16:15.449806+0000) 2024-03-04T14:16:16.451 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:16:16.449+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:16:16.450083+0000) 2024-03-04T14:16:17.451 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:16:17.449+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:16:17.450320+0000) 2024-03-04T14:16:18.451 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:16:18.450+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:16:18.450556+0000) 2024-03-04T14:16:19.452 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:16:19.450+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:16:19.450808+0000) 2024-03-04T14:16:20.452 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:16:20.450+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:16:20.451082+0000) 2024-03-04T14:16:21.452 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:16:21.450+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:16:21.451319+0000) 2024-03-04T14:16:22.452 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:16:22.451+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:16:22.451554+0000) 2024-03-04T14:16:23.456 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:16:23.451+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:16:23.451816+0000) 2024-03-04T14:16:24.453 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:16:24.451+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:16:24.452089+0000) 2024-03-04T14:16:25.453 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:16:25.451+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:16:25.452326+0000) 2024-03-04T14:16:26.050 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:16:26.058 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:16:26.453 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:16:26.452+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:16:26.452563+0000) 2024-03-04T14:16:27.454 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:16:27.451+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:16:27.452805+0000) 2024-03-04T14:16:28.454 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:16:28.452+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:16:28.453105+0000) 2024-03-04T14:16:29.454 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:16:29.452+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:16:29.453297+0000) 2024-03-04T14:16:30.454 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:16:30.453+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:16:30.453571+0000) 2024-03-04T14:16:31.455 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:16:31.453+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:16:31.453809+0000) 2024-03-04T14:16:32.455 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:16:32.453+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:16:32.454037+0000) 2024-03-04T14:16:33.455 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:16:33.453+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:16:33.454266+0000) 2024-03-04T14:16:34.455 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:16:34.454+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:16:34.454512+0000) 2024-03-04T14:16:35.455 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:16:35.454+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:16:35.454755+0000) 2024-03-04T14:16:36.456 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:16:36.454+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:16:36.454999+0000) 2024-03-04T14:16:37.456 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:16:37.455+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:16:37.455240+0000) 2024-03-04T14:16:38.456 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:16:38.455+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:16:38.455476+0000) 2024-03-04T14:16:39.456 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:16:39.455+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:16:39.455794+0000) 2024-03-04T14:16:40.457 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:16:40.455+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:16:40.456072+0000) 2024-03-04T14:16:41.457 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:16:41.455+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:16:41.456310+0000) 2024-03-04T14:16:42.457 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:16:42.455+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:16:42.456558+0000) 2024-03-04T14:16:43.457 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:16:43.455+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:16:43.456793+0000) 2024-03-04T14:16:44.458 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:16:44.455+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:16:44.457078+0000) 2024-03-04T14:16:45.458 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:16:45.456+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:16:45.457314+0000) 2024-03-04T14:16:46.458 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:16:46.456+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:16:46.457495+0000) 2024-03-04T14:16:47.458 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:16:47.456+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:16:47.457732+0000) 2024-03-04T14:16:48.459 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:16:48.456+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:16:48.457973+0000) 2024-03-04T14:16:49.459 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:16:49.457+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:16:49.458229+0000) 2024-03-04T14:16:50.459 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:16:50.457+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:16:50.458464+0000) 2024-03-04T14:16:51.459 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:16:51.457+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:16:51.458702+0000) 2024-03-04T14:16:52.460 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:16:52.458+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:16:52.458959+0000) 2024-03-04T14:16:53.460 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:16:53.458+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:16:53.459197+0000) 2024-03-04T14:16:54.460 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:16:54.458+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:16:54.459434+0000) 2024-03-04T14:16:55.460 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:16:55.458+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:16:55.459697+0000) 2024-03-04T14:16:56.253 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:16:56.264 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:16:56.460 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:16:56.459+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:16:56.459957+0000) 2024-03-04T14:16:57.461 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:16:57.459+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:16:57.460203+0000) 2024-03-04T14:16:58.461 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:16:58.459+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:16:58.460440+0000) 2024-03-04T14:16:59.461 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:16:59.459+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:16:59.460708+0000) 2024-03-04T14:17:00.462 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:17:00.460+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:17:00.460956+0000) 2024-03-04T14:17:01.462 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:17:01.460+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:17:01.461197+0000) 2024-03-04T14:17:02.462 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:17:02.460+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:17:02.461430+0000) 2024-03-04T14:17:03.462 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:17:03.460+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:17:03.461607+0000) 2024-03-04T14:17:04.462 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:17:04.461+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:17:04.461880+0000) 2024-03-04T14:17:05.463 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:17:05.461+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:17:05.462124+0000) 2024-03-04T14:17:06.463 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:17:06.461+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:17:06.462373+0000) 2024-03-04T14:17:07.463 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:17:07.461+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:17:07.462611+0000) 2024-03-04T14:17:08.463 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:17:08.462+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:17:08.462886+0000) 2024-03-04T14:17:09.464 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:17:09.462+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:17:09.463088+0000) 2024-03-04T14:17:10.464 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:17:10.462+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:17:10.463331+0000) 2024-03-04T14:17:11.464 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:17:11.462+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:17:11.463577+0000) 2024-03-04T14:17:12.464 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:17:12.463+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:17:12.463812+0000) 2024-03-04T14:17:13.465 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:17:13.463+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:17:13.464099+0000) 2024-03-04T14:17:14.465 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:17:14.463+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:17:14.464336+0000) 2024-03-04T14:17:15.465 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:17:15.463+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:17:15.464616+0000) 2024-03-04T14:17:16.466 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:17:16.464+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:17:16.464942+0000) 2024-03-04T14:17:17.466 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:17:17.464+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:17:17.465235+0000) 2024-03-04T14:17:18.466 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:17:18.464+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:17:18.465522+0000) 2024-03-04T14:17:19.468 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:17:19.465+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:17:19.465864+0000) 2024-03-04T14:17:20.467 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:17:20.465+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:17:20.466166+0000) 2024-03-04T14:17:21.467 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:17:21.465+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:17:21.466467+0000) 2024-03-04T14:17:22.467 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:17:22.466+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:17:22.466748+0000) 2024-03-04T14:17:23.468 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:17:23.466+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:17:23.467038+0000) 2024-03-04T14:17:24.468 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:17:24.466+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:17:24.467323+0000) 2024-03-04T14:17:25.468 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:17:25.467+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:17:25.467615+0000) 2024-03-04T14:17:26.396 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:17:26.404 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:17:26.468 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:17:26.467+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:17:26.467873+0000) 2024-03-04T14:17:27.469 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:17:27.467+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:17:27.468217+0000) 2024-03-04T14:17:28.469 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:17:28.468+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:17:28.468526+0000) 2024-03-04T14:17:29.472 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:17:29.468+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:17:29.468872+0000) 2024-03-04T14:17:30.470 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:17:30.468+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:17:30.469150+0000) 2024-03-04T14:17:31.470 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:17:31.469+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:17:31.469432+0000) 2024-03-04T14:17:32.470 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:17:32.469+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:17:32.469722+0000) 2024-03-04T14:17:33.471 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:17:33.469+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:17:33.470014+0000) 2024-03-04T14:17:34.471 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:17:34.469+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:17:34.470296+0000) 2024-03-04T14:17:35.471 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:17:35.470+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:17:35.470592+0000) 2024-03-04T14:17:36.471 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:17:36.470+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:17:36.470933+0000) 2024-03-04T14:17:37.472 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:17:37.470+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:17:37.471224+0000) 2024-03-04T14:17:38.472 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:17:38.470+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:17:38.471532+0000) 2024-03-04T14:17:39.472 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:17:39.471+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:17:39.471875+0000) 2024-03-04T14:17:40.473 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:17:40.471+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:17:40.472159+0000) 2024-03-04T14:17:41.473 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:17:41.472+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:17:41.472450+0000) 2024-03-04T14:17:42.473 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:17:42.472+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:17:42.472738+0000) 2024-03-04T14:17:43.474 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:17:43.472+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:17:43.473027+0000) 2024-03-04T14:17:44.474 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:17:44.473+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:17:44.473311+0000) 2024-03-04T14:17:45.474 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:17:45.472+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:17:45.473588+0000) 2024-03-04T14:17:46.474 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:17:46.472+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:17:46.473924+0000) 2024-03-04T14:17:47.475 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:17:47.473+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:17:47.474214+0000) 2024-03-04T14:17:48.475 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:17:48.472+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:17:48.474453+0000) 2024-03-04T14:17:49.475 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:17:49.473+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:17:49.474760+0000) 2024-03-04T14:17:50.475 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:17:50.473+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:17:50.475050+0000) 2024-03-04T14:17:51.476 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:17:51.474+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:17:51.475340+0000) 2024-03-04T14:17:52.476 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:17:52.474+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:17:52.475629+0000) 2024-03-04T14:17:53.476 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:17:53.474+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:17:53.475970+0000) 2024-03-04T14:17:54.477 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:17:54.474+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:17:54.476214+0000) 2024-03-04T14:17:55.477 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:17:55.475+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:17:55.476507+0000) 2024-03-04T14:17:56.478 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:17:56.475+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:17:56.476799+0000) 2024-03-04T14:17:56.579 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:17:56.591 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:17:57.477 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:17:57.476+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:17:57.477133+0000) 2024-03-04T14:17:58.478 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:17:58.476+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:17:58.477441+0000) 2024-03-04T14:17:59.478 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:17:59.476+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:17:59.477737+0000) 2024-03-04T14:18:00.478 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:18:00.477+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:18:00.478058+0000) 2024-03-04T14:18:01.479 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:18:01.477+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:18:01.478351+0000) 2024-03-04T14:18:02.479 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:18:02.477+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:18:02.478632+0000) 2024-03-04T14:18:03.479 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:18:03.478+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:18:03.478986+0000) 2024-03-04T14:18:04.480 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:18:04.478+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:18:04.479273+0000) 2024-03-04T14:18:05.480 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:18:05.478+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:18:05.479566+0000) 2024-03-04T14:18:06.480 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:18:06.479+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:18:06.479886+0000) 2024-03-04T14:18:07.480 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:18:07.479+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:18:07.480171+0000) 2024-03-04T14:18:08.481 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:18:08.479+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:18:08.480387+0000) 2024-03-04T14:18:09.481 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:18:09.479+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:18:09.480639+0000) 2024-03-04T14:18:10.481 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:18:10.480+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:18:10.480925+0000) 2024-03-04T14:18:11.482 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:18:11.480+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:18:11.481166+0000) 2024-03-04T14:18:12.482 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:18:12.480+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:18:12.481405+0000) 2024-03-04T14:18:13.482 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:18:13.480+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:18:13.481675+0000) 2024-03-04T14:18:14.482 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:18:14.481+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:18:14.481957+0000) 2024-03-04T14:18:15.483 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:18:15.481+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:18:15.482237+0000) 2024-03-04T14:18:16.483 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:18:16.481+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:18:16.482487+0000) 2024-03-04T14:18:17.483 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:18:17.482+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:18:17.482783+0000) 2024-03-04T14:18:18.483 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:18:18.482+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:18:18.483124+0000) 2024-03-04T14:18:19.484 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:18:19.482+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:18:19.483454+0000) 2024-03-04T14:18:20.484 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:18:20.483+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:18:20.483759+0000) 2024-03-04T14:18:21.484 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:18:21.483+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:18:21.484055+0000) 2024-03-04T14:18:22.485 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:18:22.483+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:18:22.484301+0000) 2024-03-04T14:18:23.485 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:18:23.484+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:18:23.484597+0000) 2024-03-04T14:18:24.485 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:18:24.484+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:18:24.484933+0000) 2024-03-04T14:18:25.486 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:18:25.484+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:18:25.485180+0000) 2024-03-04T14:18:26.486 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:18:26.484+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:18:26.485474+0000) 2024-03-04T14:18:26.751 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:18:26.758 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:18:27.486 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:18:27.484+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:18:27.485741+0000) 2024-03-04T14:18:28.486 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:18:28.485+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:18:28.485983+0000) 2024-03-04T14:18:29.486 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:18:29.485+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:18:29.486239+0000) 2024-03-04T14:18:30.487 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:18:30.486+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:18:30.486478+0000) 2024-03-04T14:18:31.487 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:18:31.485+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:18:31.486676+0000) 2024-03-04T14:18:32.487 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:18:32.486+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:18:32.486940+0000) 2024-03-04T14:18:33.487 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:18:33.486+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:18:33.487176+0000) 2024-03-04T14:18:34.488 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:18:34.487+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:18:34.487422+0000) 2024-03-04T14:18:35.488 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:18:35.487+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:18:35.487678+0000) 2024-03-04T14:18:36.488 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:18:36.487+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:18:36.487931+0000) 2024-03-04T14:18:37.488 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:18:37.487+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:18:37.488170+0000) 2024-03-04T14:18:38.489 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:18:38.488+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:18:38.488412+0000) 2024-03-04T14:18:39.489 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:18:39.487+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:18:39.488618+0000) 2024-03-04T14:18:40.489 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:18:40.488+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:18:40.488900+0000) 2024-03-04T14:18:41.490 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:18:41.488+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:18:41.489137+0000) 2024-03-04T14:18:42.490 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:18:42.488+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:18:42.489472+0000) 2024-03-04T14:18:43.490 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:18:43.489+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:18:43.489738+0000) 2024-03-04T14:18:44.490 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:18:44.489+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:18:44.490002+0000) 2024-03-04T14:18:45.491 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:18:45.490+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:18:45.490257+0000) 2024-03-04T14:18:46.491 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:18:46.490+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:18:46.490511+0000) 2024-03-04T14:18:47.491 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:18:47.490+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:18:47.490773+0000) 2024-03-04T14:18:48.491 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:18:48.489+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:18:48.491036+0000) 2024-03-04T14:18:49.492 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:18:49.490+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:18:49.491328+0000) 2024-03-04T14:18:50.492 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:18:50.490+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:18:50.491581+0000) 2024-03-04T14:18:51.492 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:18:51.490+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:18:51.491878+0000) 2024-03-04T14:18:52.492 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:18:52.491+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:18:52.492143+0000) 2024-03-04T14:18:53.493 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:18:53.491+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:18:53.492401+0000) 2024-03-04T14:18:54.493 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:18:54.491+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:18:54.492664+0000) 2024-03-04T14:18:55.493 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:18:55.491+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:18:55.492942+0000) 2024-03-04T14:18:56.493 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:18:56.492+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:18:56.493208+0000) 2024-03-04T14:18:56.897 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:18:56.905 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:18:57.493 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:18:57.492+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:18:57.493465+0000) 2024-03-04T14:18:58.494 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:18:58.492+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:18:58.493790+0000) 2024-03-04T14:18:59.494 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:18:59.493+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:18:59.494112+0000) 2024-03-04T14:19:00.495 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:19:00.493+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:19:00.494363+0000) 2024-03-04T14:19:01.495 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:19:01.493+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:19:01.494618+0000) 2024-03-04T14:19:02.495 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:19:02.494+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:19:02.494947+0000) 2024-03-04T14:19:03.495 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:19:03.494+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:19:03.495209+0000) 2024-03-04T14:19:04.496 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:19:04.494+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:19:04.495467+0000) 2024-03-04T14:19:05.496 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:19:05.494+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:19:05.495723+0000) 2024-03-04T14:19:06.496 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:19:06.495+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:19:06.495955+0000) 2024-03-04T14:19:07.496 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:19:07.495+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:19:07.496239+0000) 2024-03-04T14:19:08.497 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:19:08.495+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:19:08.496520+0000) 2024-03-04T14:19:09.497 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:19:09.495+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:19:09.496785+0000) 2024-03-04T14:19:10.497 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:19:10.496+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:19:10.497085+0000) 2024-03-04T14:19:11.498 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:19:11.496+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:19:11.497339+0000) 2024-03-04T14:19:12.498 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:19:12.496+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:19:12.497600+0000) 2024-03-04T14:19:13.499 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:19:13.497+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:19:13.497924+0000) 2024-03-04T14:19:14.498 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:19:14.497+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:19:14.498201+0000) 2024-03-04T14:19:15.499 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:19:15.497+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:19:15.498460+0000) 2024-03-04T14:19:16.500 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:19:16.497+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:19:16.498650+0000) 2024-03-04T14:19:17.499 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:19:17.498+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:19:17.498925+0000) 2024-03-04T14:19:18.499 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:19:18.498+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:19:18.499188+0000) 2024-03-04T14:19:19.500 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:19:19.498+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:19:19.499416+0000) 2024-03-04T14:19:20.500 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:19:20.499+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:19:20.499685+0000) 2024-03-04T14:19:21.500 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:19:21.499+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:19:21.499944+0000) 2024-03-04T14:19:22.500 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:19:22.499+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:19:22.500198+0000) 2024-03-04T14:19:23.501 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:19:23.499+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:19:23.500416+0000) 2024-03-04T14:19:24.501 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:19:24.500+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:19:24.500680+0000) 2024-03-04T14:19:25.501 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:19:25.500+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:19:25.500954+0000) 2024-03-04T14:19:26.501 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:19:26.500+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:19:26.501220+0000) 2024-03-04T14:19:27.128 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:19:27.140 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:19:27.502 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:19:27.500+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:19:27.501463+0000) 2024-03-04T14:19:28.502 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:19:28.501+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:19:28.501706+0000) 2024-03-04T14:19:29.502 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:19:29.501+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:19:29.501960+0000) 2024-03-04T14:19:30.502 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:19:30.501+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:19:30.502200+0000) 2024-03-04T14:19:31.503 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:19:31.502+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:19:31.502445+0000) 2024-03-04T14:19:32.503 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:19:32.502+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:19:32.502686+0000) 2024-03-04T14:19:33.503 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:19:33.502+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:19:33.502931+0000) 2024-03-04T14:19:34.503 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:19:34.502+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:19:34.503188+0000) 2024-03-04T14:19:35.504 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:19:35.503+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:19:35.503443+0000) 2024-03-04T14:19:36.504 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:19:36.503+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:19:36.503699+0000) 2024-03-04T14:19:37.504 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:19:37.503+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:19:37.503952+0000) 2024-03-04T14:19:38.504 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:19:38.503+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:19:38.504212+0000) 2024-03-04T14:19:39.505 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:19:39.504+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:19:39.504485+0000) 2024-03-04T14:19:40.505 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:19:40.503+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:19:40.504759+0000) 2024-03-04T14:19:41.505 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:19:41.504+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:19:41.505021+0000) 2024-03-04T14:19:42.505 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:19:42.504+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:19:42.505219+0000) 2024-03-04T14:19:43.506 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:19:43.505+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:19:43.505487+0000) 2024-03-04T14:19:44.506 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:19:44.505+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:19:44.505754+0000) 2024-03-04T14:19:45.506 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:19:45.505+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:19:45.506009+0000) 2024-03-04T14:19:46.506 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:19:46.506+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:19:46.506288+0000) 2024-03-04T14:19:47.506 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:19:47.506+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:19:47.506470+0000) 2024-03-04T14:19:48.507 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:19:48.505+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:19:48.506700+0000) 2024-03-04T14:19:49.507 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:19:49.506+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:19:49.506959+0000) 2024-03-04T14:19:50.507 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:19:50.507+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:19:50.507195+0000) 2024-03-04T14:19:51.507 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:19:51.506+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:19:51.507437+0000) 2024-03-04T14:19:52.508 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:19:52.506+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:19:52.507672+0000) 2024-03-04T14:19:53.508 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:19:53.506+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:19:53.507945+0000) 2024-03-04T14:19:54.508 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:19:54.507+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:19:54.508202+0000) 2024-03-04T14:19:55.509 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:19:55.507+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:19:55.508443+0000) 2024-03-04T14:19:56.511 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:19:56.507+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:19:56.508688+0000) 2024-03-04T14:19:57.273 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:19:57.281 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:19:57.509 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:19:57.507+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:19:57.508986+0000) 2024-03-04T14:19:58.509 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:19:58.508+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:19:58.509227+0000) 2024-03-04T14:19:59.510 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:19:59.508+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:19:59.509481+0000) 2024-03-04T14:20:00.510 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:20:00.508+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:20:00.509736+0000) 2024-03-04T14:20:01.510 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:20:01.509+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:20:01.509979+0000) 2024-03-04T14:20:02.510 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:20:02.509+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:20:02.510256+0000) 2024-03-04T14:20:03.511 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:20:03.509+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:20:03.510439+0000) 2024-03-04T14:20:04.511 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:20:04.509+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:20:04.510677+0000) 2024-03-04T14:20:05.511 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:20:05.510+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:20:05.510954+0000) 2024-03-04T14:20:06.511 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:20:06.510+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:20:06.511197+0000) 2024-03-04T14:20:07.512 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:20:07.510+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:20:07.511439+0000) 2024-03-04T14:20:08.512 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:20:08.510+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:20:08.511677+0000) 2024-03-04T14:20:09.512 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:20:09.511+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:20:09.511966+0000) 2024-03-04T14:20:10.512 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:20:10.511+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:20:10.512205+0000) 2024-03-04T14:20:11.512 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:20:11.511+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:20:11.512447+0000) 2024-03-04T14:20:12.514 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:20:12.511+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:20:12.512691+0000) 2024-03-04T14:20:13.513 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:20:13.512+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:20:13.512933+0000) 2024-03-04T14:20:14.513 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:20:14.512+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:20:14.513176+0000) 2024-03-04T14:20:15.513 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:20:15.512+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:20:15.513420+0000) 2024-03-04T14:20:16.514 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:20:16.512+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:20:16.513713+0000) 2024-03-04T14:20:17.514 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:20:17.513+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:20:17.513986+0000) 2024-03-04T14:20:18.514 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:20:18.513+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:20:18.514182+0000) 2024-03-04T14:20:19.514 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:20:19.513+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:20:19.514434+0000) 2024-03-04T14:20:20.515 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:20:20.513+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:20:20.514627+0000) 2024-03-04T14:20:21.515 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:20:21.514+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:20:21.514932+0000) 2024-03-04T14:20:22.515 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:20:22.514+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:20:22.515122+0000) 2024-03-04T14:20:23.515 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:20:23.514+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:20:23.515311+0000) 2024-03-04T14:20:24.516 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:20:24.514+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:20:24.515561+0000) 2024-03-04T14:20:25.516 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:20:25.515+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:20:25.515806+0000) 2024-03-04T14:20:26.520 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:20:26.515+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:20:26.516099+0000) 2024-03-04T14:20:27.487 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:20:27.495 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:20:27.516 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:20:27.515+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:20:27.516287+0000) 2024-03-04T14:20:28.516 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:20:28.516+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:20:28.516530+0000) 2024-03-04T14:20:29.517 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:20:29.516+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:20:29.516783+0000) 2024-03-04T14:20:30.517 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:20:30.516+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:20:30.517024+0000) 2024-03-04T14:20:31.517 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:20:31.516+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:20:31.517267+0000) 2024-03-04T14:20:32.517 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:20:32.517+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:20:32.517505+0000) 2024-03-04T14:20:33.518 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:20:33.517+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:20:33.517746+0000) 2024-03-04T14:20:34.518 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:20:34.517+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:20:34.517991+0000) 2024-03-04T14:20:35.518 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:20:35.517+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:20:35.518234+0000) 2024-03-04T14:20:36.519 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:20:36.518+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:20:36.518492+0000) 2024-03-04T14:20:37.519 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:20:37.517+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:20:37.518753+0000) 2024-03-04T14:20:38.519 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:20:38.518+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:20:38.518986+0000) 2024-03-04T14:20:39.520 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:20:39.518+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:20:39.519199+0000) 2024-03-04T14:20:40.519 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:20:40.519+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:20:40.519438+0000) 2024-03-04T14:20:41.520 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:20:41.518+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:20:41.519623+0000) 2024-03-04T14:20:42.520 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:20:42.519+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:20:42.519932+0000) 2024-03-04T14:20:43.520 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:20:43.519+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:20:43.520173+0000) 2024-03-04T14:20:44.520 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:20:44.519+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:20:44.520425+0000) 2024-03-04T14:20:45.521 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:20:45.520+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:20:45.520668+0000) 2024-03-04T14:20:46.521 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:20:46.520+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:20:46.520943+0000) 2024-03-04T14:20:47.521 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:20:47.521+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:20:47.521208+0000) 2024-03-04T14:20:48.521 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:20:48.521+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:20:48.521467+0000) 2024-03-04T14:20:49.522 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:20:49.520+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:20:49.521688+0000) 2024-03-04T14:20:50.522 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:20:50.521+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:20:50.521936+0000) 2024-03-04T14:20:51.522 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:20:51.522+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:20:51.522184+0000) 2024-03-04T14:20:52.523 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:20:52.521+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:20:52.522429+0000) 2024-03-04T14:20:53.523 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:20:53.521+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:20:53.522672+0000) 2024-03-04T14:20:54.523 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:20:54.521+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:20:54.522890+0000) 2024-03-04T14:20:55.523 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:20:55.522+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:20:55.523138+0000) 2024-03-04T14:20:56.523 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:20:56.522+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:20:56.523380+0000) 2024-03-04T14:20:57.524 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:20:57.522+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:20:57.523621+0000) 2024-03-04T14:20:57.642 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:20:57.650 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:20:58.524 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:20:58.522+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:20:58.523922+0000) 2024-03-04T14:20:59.524 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:20:59.522+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:20:59.524173+0000) 2024-03-04T14:21:00.524 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:21:00.523+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:21:00.524451+0000) 2024-03-04T14:21:01.525 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:21:01.523+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:21:01.524743+0000) 2024-03-04T14:21:02.525 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:21:02.524+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:21:02.524995+0000) 2024-03-04T14:21:03.525 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:21:03.524+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:21:03.525247+0000) 2024-03-04T14:21:04.525 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:21:04.524+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:21:04.525503+0000) 2024-03-04T14:21:05.526 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:21:05.524+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:21:05.525704+0000) 2024-03-04T14:21:06.526 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:21:06.525+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:21:06.525962+0000) 2024-03-04T14:21:07.526 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:21:07.525+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:21:07.526219+0000) 2024-03-04T14:21:08.526 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:21:08.525+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:21:08.526467+0000) 2024-03-04T14:21:09.526 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:21:09.525+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:21:09.526732+0000) 2024-03-04T14:21:10.527 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:21:10.526+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:21:10.526987+0000) 2024-03-04T14:21:11.527 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:21:11.526+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:21:11.527243+0000) 2024-03-04T14:21:12.527 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:21:12.526+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:21:12.527439+0000) 2024-03-04T14:21:13.528 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:21:13.526+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:21:13.527705+0000) 2024-03-04T14:21:14.528 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:21:14.527+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:21:14.527961+0000) 2024-03-04T14:21:15.528 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:21:15.527+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:21:15.528208+0000) 2024-03-04T14:21:16.529 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:21:16.527+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:21:16.528452+0000) 2024-03-04T14:21:17.528 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:21:17.527+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:21:17.528707+0000) 2024-03-04T14:21:18.529 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:21:18.528+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:21:18.528977+0000) 2024-03-04T14:21:19.529 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:21:19.528+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:21:19.529235+0000) 2024-03-04T14:21:20.529 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:21:20.528+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:21:20.529472+0000) 2024-03-04T14:21:21.530 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:21:21.529+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:21:21.529709+0000) 2024-03-04T14:21:22.530 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:21:22.529+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:21:22.529940+0000) 2024-03-04T14:21:23.530 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:21:23.529+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:21:23.530171+0000) 2024-03-04T14:21:24.530 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:21:24.529+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:21:24.530428+0000) 2024-03-04T14:21:25.530 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:21:25.530+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:21:25.530693+0000) 2024-03-04T14:21:26.531 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:21:26.530+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:21:26.530974+0000) 2024-03-04T14:21:27.531 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:21:27.530+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:21:27.531232+0000) 2024-03-04T14:21:27.872 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:21:27.880 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:21:28.531 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:21:28.530+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:21:28.531467+0000) 2024-03-04T14:21:29.532 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:21:29.531+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:21:29.531732+0000) 2024-03-04T14:21:30.532 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:21:30.531+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:21:30.531975+0000) 2024-03-04T14:21:31.532 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:21:31.531+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:21:31.532217+0000) 2024-03-04T14:21:32.532 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:21:32.532+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:21:32.532471+0000) 2024-03-04T14:21:33.533 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:21:33.531+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:21:33.532721+0000) 2024-03-04T14:21:34.533 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:21:34.532+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:21:34.532957+0000) 2024-03-04T14:21:35.534 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:21:35.532+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:21:35.533205+0000) 2024-03-04T14:21:36.533 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:21:36.533+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:21:36.533438+0000) 2024-03-04T14:21:37.534 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:21:37.532+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:21:37.533689+0000) 2024-03-04T14:21:38.534 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:21:38.533+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:21:38.533973+0000) 2024-03-04T14:21:39.534 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:21:39.533+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:21:39.534240+0000) 2024-03-04T14:21:40.534 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:21:40.533+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:21:40.534436+0000) 2024-03-04T14:21:41.535 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:21:41.534+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:21:41.534672+0000) 2024-03-04T14:21:42.535 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:21:42.534+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:21:42.534940+0000) 2024-03-04T14:21:43.535 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:21:43.534+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:21:43.535178+0000) 2024-03-04T14:21:44.535 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:21:44.535+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:21:44.535415+0000) 2024-03-04T14:21:45.536 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:21:45.535+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:21:45.535661+0000) 2024-03-04T14:21:46.536 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:21:46.535+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:21:46.535936+0000) 2024-03-04T14:21:47.536 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:21:47.535+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:21:47.536175+0000) 2024-03-04T14:21:48.536 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:21:48.536+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:21:48.536432+0000) 2024-03-04T14:21:49.537 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:21:49.535+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:21:49.536694+0000) 2024-03-04T14:21:50.537 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:21:50.536+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:21:50.536938+0000) 2024-03-04T14:21:51.537 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:21:51.537+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:21:51.537180+0000) 2024-03-04T14:21:52.537 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:21:52.537+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:21:52.537420+0000) 2024-03-04T14:21:53.538 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:21:53.536+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:21:53.537606+0000) 2024-03-04T14:21:54.538 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:21:54.536+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:21:54.537909+0000) 2024-03-04T14:21:55.538 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:21:55.537+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:21:55.538147+0000) 2024-03-04T14:21:56.539 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:21:56.537+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:21:56.538382+0000) 2024-03-04T14:21:57.539 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:21:57.536+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:21:57.538634+0000) 2024-03-04T14:21:58.075 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:21:58.084 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:21:58.539 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:21:58.537+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:21:58.538940+0000) 2024-03-04T14:21:59.539 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:21:59.538+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:21:59.539188+0000) 2024-03-04T14:22:00.539 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:22:00.538+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:22:00.539427+0000) 2024-03-04T14:22:01.540 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:22:01.538+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:22:01.539670+0000) 2024-03-04T14:22:02.540 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:22:02.538+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:22:02.539948+0000) 2024-03-04T14:22:03.540 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:22:03.539+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:22:03.540183+0000) 2024-03-04T14:22:04.540 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:22:04.539+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:22:04.540431+0000) 2024-03-04T14:22:05.541 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:22:05.539+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:22:05.540677+0000) 2024-03-04T14:22:06.541 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:22:06.540+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:22:06.540951+0000) 2024-03-04T14:22:07.541 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:22:07.540+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:22:07.541189+0000) 2024-03-04T14:22:08.541 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:22:08.540+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:22:08.541428+0000) 2024-03-04T14:22:09.542 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:22:09.540+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:22:09.541695+0000) 2024-03-04T14:22:10.542 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:22:10.541+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:22:10.541938+0000) 2024-03-04T14:22:11.542 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:22:11.541+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:22:11.542181+0000) 2024-03-04T14:22:12.542 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:22:12.541+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:22:12.542414+0000) 2024-03-04T14:22:13.543 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:22:13.541+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:22:13.542654+0000) 2024-03-04T14:22:14.543 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:22:14.542+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:22:14.542941+0000) 2024-03-04T14:22:15.543 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:22:15.542+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:22:15.543186+0000) 2024-03-04T14:22:16.543 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:22:16.542+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:22:16.543426+0000) 2024-03-04T14:22:17.544 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:22:17.542+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:22:17.543668+0000) 2024-03-04T14:22:18.544 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:22:18.543+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:22:18.543934+0000) 2024-03-04T14:22:19.544 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:22:19.543+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:22:19.544181+0000) 2024-03-04T14:22:20.544 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:22:20.543+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:22:20.544480+0000) 2024-03-04T14:22:21.544 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:22:21.543+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:22:21.544710+0000) 2024-03-04T14:22:22.545 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:22:22.544+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:22:22.544958+0000) 2024-03-04T14:22:23.545 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:22:23.544+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:22:23.545196+0000) 2024-03-04T14:22:24.545 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:22:24.544+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:22:24.545438+0000) 2024-03-04T14:22:25.546 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:22:25.545+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:22:25.545685+0000) 2024-03-04T14:22:26.546 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:22:26.545+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:22:26.545970+0000) 2024-03-04T14:22:27.546 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:22:27.545+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:22:27.546207+0000) 2024-03-04T14:22:28.216 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:22:28.224 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:22:28.546 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:22:28.545+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:22:28.546452+0000) 2024-03-04T14:22:29.547 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:22:29.546+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:22:29.546701+0000) 2024-03-04T14:22:30.547 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:22:30.546+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:22:30.546943+0000) 2024-03-04T14:22:31.547 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:22:31.546+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:22:31.547183+0000) 2024-03-04T14:22:32.547 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:22:32.546+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:22:32.547421+0000) 2024-03-04T14:22:33.548 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:22:33.547+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:22:33.547657+0000) 2024-03-04T14:22:34.548 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:22:34.547+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:22:34.547963+0000) 2024-03-04T14:22:35.548 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:22:35.547+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:22:35.548204+0000) 2024-03-04T14:22:36.548 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:22:36.548+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:22:36.548447+0000) 2024-03-04T14:22:37.549 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:22:37.548+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:22:37.548693+0000) 2024-03-04T14:22:38.549 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:22:38.548+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:22:38.548936+0000) 2024-03-04T14:22:39.549 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:22:39.548+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:22:39.549145+0000) 2024-03-04T14:22:40.549 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:22:40.549+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:22:40.549395+0000) 2024-03-04T14:22:41.550 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:22:41.549+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:22:41.549671+0000) 2024-03-04T14:22:42.550 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:22:42.549+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:22:42.549966+0000) 2024-03-04T14:22:43.550 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:22:43.549+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:22:43.550253+0000) 2024-03-04T14:22:44.551 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:22:44.550+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:22:44.550543+0000) 2024-03-04T14:22:45.551 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:22:45.550+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:22:45.550878+0000) 2024-03-04T14:22:46.551 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:22:46.550+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:22:46.551168+0000) 2024-03-04T14:22:47.551 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:22:47.551+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:22:47.551479+0000) 2024-03-04T14:22:48.553 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:22:48.551+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:22:48.551784+0000) 2024-03-04T14:22:49.552 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:22:49.551+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:22:49.552142+0000) 2024-03-04T14:22:50.552 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:22:50.552+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:22:50.552430+0000) 2024-03-04T14:22:51.553 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:22:51.551+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:22:51.552715+0000) 2024-03-04T14:22:52.553 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:22:52.552+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:22:52.552995+0000) 2024-03-04T14:22:53.553 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:22:53.553+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:22:53.553291+0000) 2024-03-04T14:22:54.553 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:22:54.552+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:22:54.553593+0000) 2024-03-04T14:22:55.554 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:22:55.553+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:22:55.553875+0000) 2024-03-04T14:22:56.554 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:22:56.552+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:22:56.554106+0000) 2024-03-04T14:22:57.554 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:22:57.553+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:22:57.554407+0000) 2024-03-04T14:22:58.355 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:22:58.363 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:22:58.554 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:22:58.552+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:22:58.554660+0000) 2024-03-04T14:22:59.555 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:22:59.553+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:22:59.554993+0000) 2024-03-04T14:23:00.555 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:23:00.554+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:23:00.555277+0000) 2024-03-04T14:23:01.556 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:23:01.554+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:23:01.555576+0000) 2024-03-04T14:23:02.556 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:23:02.554+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:23:02.555891+0000) 2024-03-04T14:23:03.556 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:23:03.555+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:23:03.556168+0000) 2024-03-04T14:23:04.556 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:23:04.555+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:23:04.556458+0000) 2024-03-04T14:23:05.557 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:23:05.555+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:23:05.556748+0000) 2024-03-04T14:23:06.557 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:23:06.556+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:23:06.557032+0000) 2024-03-04T14:23:07.557 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:23:07.556+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:23:07.557279+0000) 2024-03-04T14:23:08.558 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:23:08.556+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:23:08.557614+0000) 2024-03-04T14:23:09.558 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:23:09.557+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:23:09.557979+0000) 2024-03-04T14:23:10.558 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:23:10.557+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:23:10.558267+0000) 2024-03-04T14:23:11.558 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:23:11.557+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:23:11.558562+0000) 2024-03-04T14:23:12.559 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:23:12.558+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:23:12.558907+0000) 2024-03-04T14:23:13.559 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:23:13.558+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:23:13.559217+0000) 2024-03-04T14:23:14.559 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:23:14.558+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:23:14.559524+0000) 2024-03-04T14:23:15.560 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:23:15.559+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:23:15.559868+0000) 2024-03-04T14:23:16.560 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:23:16.559+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:23:16.560162+0000) 2024-03-04T14:23:17.560 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:23:17.559+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:23:17.560412+0000) 2024-03-04T14:23:18.561 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:23:18.559+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:23:18.560602+0000) 2024-03-04T14:23:19.561 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:23:19.560+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:23:19.560917+0000) 2024-03-04T14:23:20.561 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:23:20.560+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:23:20.561129+0000) 2024-03-04T14:23:21.561 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:23:21.560+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:23:21.561370+0000) 2024-03-04T14:23:22.562 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:23:22.560+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:23:22.561606+0000) 2024-03-04T14:23:23.562 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:23:23.561+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:23:23.561880+0000) 2024-03-04T14:23:24.562 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:23:24.561+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:23:24.562140+0000) 2024-03-04T14:23:25.562 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:23:25.561+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:23:25.562379+0000) 2024-03-04T14:23:26.562 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:23:26.562+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:23:26.562653+0000) 2024-03-04T14:23:27.563 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:23:27.562+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:23:27.562934+0000) 2024-03-04T14:23:28.489 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:23:28.497 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:23:28.562 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:23:28.562+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:23:28.563149+0000) 2024-03-04T14:23:29.563 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:23:29.562+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:23:29.563411+0000) 2024-03-04T14:23:30.564 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:23:30.563+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:23:30.563655+0000) 2024-03-04T14:23:31.563 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:23:31.563+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:23:31.563939+0000) 2024-03-04T14:23:32.564 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:23:32.563+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:23:32.564174+0000) 2024-03-04T14:23:33.564 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:23:33.563+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:23:33.564414+0000) 2024-03-04T14:23:34.565 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:23:34.564+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:23:34.564659+0000) 2024-03-04T14:23:35.565 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:23:35.564+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:23:35.564936+0000) 2024-03-04T14:23:36.565 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:23:36.564+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:23:36.565172+0000) 2024-03-04T14:23:37.565 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:23:37.564+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:23:37.565431+0000) 2024-03-04T14:23:38.566 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:23:38.565+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:23:38.565667+0000) 2024-03-04T14:23:39.566 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:23:39.565+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:23:39.565906+0000) 2024-03-04T14:23:40.566 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:23:40.565+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:23:40.566152+0000) 2024-03-04T14:23:41.566 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:23:41.565+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:23:41.566384+0000) 2024-03-04T14:23:42.566 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:23:42.565+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:23:42.566634+0000) 2024-03-04T14:23:43.567 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:23:43.566+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:23:43.566915+0000) 2024-03-04T14:23:44.567 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:23:44.566+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:23:44.567183+0000) 2024-03-04T14:23:45.567 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:23:45.566+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:23:45.567397+0000) 2024-03-04T14:23:46.568 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:23:46.567+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:23:46.567658+0000) 2024-03-04T14:23:47.568 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:23:47.567+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:23:47.567890+0000) 2024-03-04T14:23:48.568 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:23:48.567+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:23:48.568140+0000) 2024-03-04T14:23:49.568 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:23:49.567+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:23:49.568404+0000) 2024-03-04T14:23:50.568 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:23:50.568+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:23:50.568658+0000) 2024-03-04T14:23:51.569 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:23:51.568+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:23:51.568933+0000) 2024-03-04T14:23:52.569 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:23:52.568+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:23:52.569172+0000) 2024-03-04T14:23:53.569 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:23:53.568+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:23:53.569414+0000) 2024-03-04T14:23:54.570 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:23:54.568+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:23:54.569653+0000) 2024-03-04T14:23:55.570 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:23:55.569+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:23:55.569946+0000) 2024-03-04T14:23:56.570 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:23:56.570+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:23:56.570186+0000) 2024-03-04T14:23:57.570 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:23:57.570+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:23:57.570424+0000) 2024-03-04T14:23:58.570 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:23:58.570+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:23:58.570665+0000) 2024-03-04T14:23:58.695 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:23:58.703 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:23:59.571 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:23:59.569+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:23:59.570968+0000) 2024-03-04T14:24:00.571 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:24:00.570+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:24:00.571225+0000) 2024-03-04T14:24:01.571 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:24:01.569+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:24:01.571479+0000) 2024-03-04T14:24:02.571 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:24:02.570+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:24:02.571719+0000) 2024-03-04T14:24:03.572 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:24:03.570+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:24:03.571960+0000) 2024-03-04T14:24:04.572 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:24:04.571+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:24:04.572253+0000) 2024-03-04T14:24:05.572 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:24:05.571+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:24:05.572493+0000) 2024-03-04T14:24:06.572 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:24:06.571+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:24:06.572674+0000) 2024-03-04T14:24:07.573 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:24:07.571+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:24:07.573000+0000) 2024-03-04T14:24:08.573 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:24:08.572+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:24:08.573238+0000) 2024-03-04T14:24:09.573 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:24:09.572+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:24:09.573454+0000) 2024-03-04T14:24:10.573 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:24:10.572+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:24:10.573701+0000) 2024-03-04T14:24:11.574 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:24:11.573+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:24:11.573947+0000) 2024-03-04T14:24:12.574 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:24:12.573+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:24:12.574220+0000) 2024-03-04T14:24:13.574 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:24:13.573+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:24:13.574477+0000) 2024-03-04T14:24:14.574 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:24:14.573+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:24:14.574751+0000) 2024-03-04T14:24:15.574 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:24:15.574+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:24:15.575007+0000) 2024-03-04T14:24:16.575 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:24:16.574+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:24:16.575267+0000) 2024-03-04T14:24:17.575 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:24:17.574+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:24:17.575511+0000) 2024-03-04T14:24:18.576 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:24:18.574+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:24:18.575756+0000) 2024-03-04T14:24:19.576 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:24:19.575+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:24:19.576020+0000) 2024-03-04T14:24:20.576 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:24:20.575+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:24:20.576308+0000) 2024-03-04T14:24:21.577 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:24:21.575+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:24:21.576617+0000) 2024-03-04T14:24:22.577 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:24:22.576+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:24:22.576948+0000) 2024-03-04T14:24:23.577 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:24:23.576+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:24:23.577241+0000) 2024-03-04T14:24:24.577 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:24:24.576+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:24:24.577557+0000) 2024-03-04T14:24:25.578 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:24:25.577+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:24:25.577817+0000) 2024-03-04T14:24:26.578 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:24:26.577+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:24:26.578109+0000) 2024-03-04T14:24:27.578 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:24:27.577+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:24:27.578370+0000) 2024-03-04T14:24:28.579 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:24:28.577+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:24:28.578613+0000) 2024-03-04T14:24:28.923 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:24:28.932 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:24:29.578 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:24:29.578+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:24:29.578916+0000) 2024-03-04T14:24:30.579 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:24:30.578+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:24:30.579173+0000) 2024-03-04T14:24:31.579 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:24:31.578+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:24:31.579413+0000) 2024-03-04T14:24:32.580 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:24:32.579+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:24:32.579650+0000) 2024-03-04T14:24:33.580 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:24:33.579+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:24:33.579932+0000) 2024-03-04T14:24:34.580 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:24:34.579+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:24:34.580178+0000) 2024-03-04T14:24:35.580 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:24:35.579+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:24:35.580372+0000) 2024-03-04T14:24:36.580 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:24:36.580+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:24:36.580613+0000) 2024-03-04T14:24:37.580 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:24:37.580+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:24:37.580905+0000) 2024-03-04T14:24:38.581 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:24:38.580+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:24:38.581152+0000) 2024-03-04T14:24:39.581 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:24:39.580+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:24:39.581408+0000) 2024-03-04T14:24:40.581 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:24:40.581+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:24:40.581683+0000) 2024-03-04T14:24:41.581 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:24:41.581+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:24:41.581944+0000) 2024-03-04T14:24:42.582 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:24:42.581+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:24:42.582208+0000) 2024-03-04T14:24:43.582 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:24:43.582+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:24:43.582447+0000) 2024-03-04T14:24:44.582 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:24:44.581+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:24:44.582642+0000) 2024-03-04T14:24:45.582 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:24:45.582+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:24:45.582937+0000) 2024-03-04T14:24:46.583 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:24:46.582+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:24:46.583175+0000) 2024-03-04T14:24:47.583 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:24:47.582+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:24:47.583408+0000) 2024-03-04T14:24:48.583 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:24:48.583+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:24:48.583649+0000) 2024-03-04T14:24:49.583 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:24:49.583+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:24:49.583947+0000) 2024-03-04T14:24:50.584 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:24:50.583+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:24:50.584173+0000) 2024-03-04T14:24:51.584 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:24:51.584+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:24:51.584417+0000) 2024-03-04T14:24:52.584 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:24:52.584+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:24:52.584657+0000) 2024-03-04T14:24:53.585 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:24:53.584+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:24:53.584942+0000) 2024-03-04T14:24:54.585 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:24:54.584+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:24:54.585185+0000) 2024-03-04T14:24:55.585 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:24:55.585+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:24:55.585482+0000) 2024-03-04T14:24:56.585 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:24:56.585+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:24:56.585722+0000) 2024-03-04T14:24:57.585 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:24:57.585+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:24:57.585942+0000) 2024-03-04T14:24:58.586 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:24:58.586+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:24:58.586183+0000) 2024-03-04T14:24:59.129 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:24:59.139 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:24:59.586 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:24:59.586+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:24:59.586440+0000) 2024-03-04T14:25:00.586 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:25:00.586+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:25:00.586683+0000) 2024-03-04T14:25:01.586 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:25:01.585+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:25:01.586967+0000) 2024-03-04T14:25:02.587 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:25:02.586+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:25:02.587203+0000) 2024-03-04T14:25:03.587 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:25:03.586+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:25:03.587424+0000) 2024-03-04T14:25:04.587 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:25:04.586+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:25:04.587721+0000) 2024-03-04T14:25:05.588 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:25:05.586+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:25:05.588028+0000) 2024-03-04T14:25:06.588 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:25:06.587+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:25:06.588323+0000) 2024-03-04T14:25:07.588 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:25:07.587+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:25:07.588613+0000) 2024-03-04T14:25:08.589 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:25:08.587+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:25:08.588929+0000) 2024-03-04T14:25:09.589 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:25:09.587+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:25:09.589180+0000) 2024-03-04T14:25:10.589 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:25:10.588+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:25:10.589491+0000) 2024-03-04T14:25:11.590 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:25:11.588+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:25:11.589799+0000) 2024-03-04T14:25:12.590 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:25:12.589+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:25:12.590133+0000) 2024-03-04T14:25:13.590 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:25:13.589+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:25:13.590430+0000) 2024-03-04T14:25:14.590 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:25:14.589+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:25:14.590721+0000) 2024-03-04T14:25:15.590 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:25:15.590+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:25:15.591013+0000) 2024-03-04T14:25:16.591 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:25:16.590+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:25:16.591290+0000) 2024-03-04T14:25:17.591 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:25:17.590+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:25:17.591603+0000) 2024-03-04T14:25:18.592 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:25:18.591+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:25:18.591865+0000) 2024-03-04T14:25:19.592 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:25:19.591+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:25:19.592172+0000) 2024-03-04T14:25:20.592 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:25:20.591+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:25:20.592455+0000) 2024-03-04T14:25:21.592 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:25:21.591+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:25:21.592770+0000) 2024-03-04T14:25:22.593 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:25:22.592+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:25:22.593069+0000) 2024-03-04T14:25:23.593 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:25:23.592+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:25:23.593357+0000) 2024-03-04T14:25:24.593 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:25:24.592+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:25:24.593644+0000) 2024-03-04T14:25:25.594 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:25:25.593+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:25:25.593963+0000) 2024-03-04T14:25:26.594 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:25:26.593+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:25:26.594241+0000) 2024-03-04T14:25:27.595 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:25:27.593+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:25:27.594501+0000) 2024-03-04T14:25:28.594 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:25:28.594+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:25:28.594736+0000) 2024-03-04T14:25:29.292 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:25:29.301 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:25:29.594 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:25:29.594+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:25:29.594994+0000) 2024-03-04T14:25:30.595 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:25:30.594+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:25:30.595245+0000) 2024-03-04T14:25:31.595 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:25:31.594+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:25:31.595484+0000) 2024-03-04T14:25:32.595 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:25:32.595+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:25:32.595734+0000) 2024-03-04T14:25:33.596 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:25:33.595+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:25:33.595984+0000) 2024-03-04T14:25:34.596 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:25:34.595+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:25:34.596243+0000) 2024-03-04T14:25:35.596 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:25:35.595+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:25:35.596481+0000) 2024-03-04T14:25:36.596 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:25:36.596+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:25:36.596716+0000) 2024-03-04T14:25:37.597 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:25:37.596+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:25:37.596985+0000) 2024-03-04T14:25:38.597 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:25:38.596+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:25:38.597273+0000) 2024-03-04T14:25:39.597 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:25:39.597+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:25:39.597517+0000) 2024-03-04T14:25:40.598 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:25:40.596+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:25:40.597752+0000) 2024-03-04T14:25:41.597 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:25:41.597+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:25:41.597988+0000) 2024-03-04T14:25:42.598 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:25:42.597+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:25:42.598223+0000) 2024-03-04T14:25:43.598 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:25:43.598+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:25:43.598459+0000) 2024-03-04T14:25:44.598 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:25:44.598+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:25:44.598697+0000) 2024-03-04T14:25:45.599 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:25:45.598+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:25:45.598941+0000) 2024-03-04T14:25:46.599 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:25:46.598+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:25:46.599177+0000) 2024-03-04T14:25:47.599 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:25:47.599+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:25:47.599425+0000) 2024-03-04T14:25:48.600 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:25:48.599+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:25:48.599657+0000) 2024-03-04T14:25:49.600 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:25:49.599+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:25:49.599939+0000) 2024-03-04T14:25:50.600 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:25:50.599+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:25:50.600201+0000) 2024-03-04T14:25:51.600 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:25:51.600+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:25:51.600440+0000) 2024-03-04T14:25:52.600 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:25:52.599+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:25:52.600689+0000) 2024-03-04T14:25:53.600 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:25:53.600+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:25:53.600924+0000) 2024-03-04T14:25:54.601 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:25:54.600+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:25:54.601160+0000) 2024-03-04T14:25:55.601 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:25:55.601+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:25:55.601422+0000) 2024-03-04T14:25:56.601 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:25:56.601+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:25:56.601662+0000) 2024-03-04T14:25:57.602 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:25:57.601+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:25:57.601934+0000) 2024-03-04T14:25:58.601 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:25:58.601+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:25:58.602168+0000) 2024-03-04T14:25:59.506 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:25:59.518 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:25:59.602 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:25:59.601+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:25:59.602391+0000) 2024-03-04T14:26:00.602 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:26:00.602+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:26:00.602628+0000) 2024-03-04T14:26:01.602 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:26:01.602+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:26:01.602908+0000) 2024-03-04T14:26:02.603 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:26:02.602+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:26:02.603119+0000) 2024-03-04T14:26:03.603 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:26:03.603+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:26:03.603366+0000) 2024-03-04T14:26:04.603 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:26:04.602+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:26:04.603624+0000) 2024-03-04T14:26:05.603 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:26:05.602+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:26:05.603932+0000) 2024-03-04T14:26:06.604 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:26:06.603+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:26:06.604190+0000) 2024-03-04T14:26:07.604 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:26:07.603+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:26:07.604466+0000) 2024-03-04T14:26:08.604 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:26:08.603+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:26:08.604737+0000) 2024-03-04T14:26:09.605 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:26:09.603+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:26:09.604999+0000) 2024-03-04T14:26:10.605 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:26:10.603+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:26:10.605251+0000) 2024-03-04T14:26:11.605 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:26:11.604+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:26:11.605517+0000) 2024-03-04T14:26:12.606 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:26:12.604+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:26:12.605755+0000) 2024-03-04T14:26:13.605 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:26:13.605+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:26:13.605988+0000) 2024-03-04T14:26:14.606 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:26:14.605+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:26:14.606225+0000) 2024-03-04T14:26:15.606 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:26:15.605+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:26:15.606471+0000) 2024-03-04T14:26:16.606 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:26:16.605+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:26:16.606717+0000) 2024-03-04T14:26:17.607 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:26:17.606+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:26:17.606959+0000) 2024-03-04T14:26:18.607 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:26:18.606+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:26:18.607175+0000) 2024-03-04T14:26:19.607 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:26:19.606+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:26:19.607442+0000) 2024-03-04T14:26:20.608 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:26:20.606+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:26:20.607700+0000) 2024-03-04T14:26:21.609 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:26:21.607+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:26:21.607955+0000) 2024-03-04T14:26:22.608 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:26:22.607+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:26:22.608219+0000) 2024-03-04T14:26:23.608 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:26:23.607+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:26:23.608474+0000) 2024-03-04T14:26:24.608 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:26:24.607+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:26:24.608731+0000) 2024-03-04T14:26:25.609 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:26:25.608+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:26:25.608998+0000) 2024-03-04T14:26:26.609 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:26:26.608+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:26:26.609271+0000) 2024-03-04T14:26:27.609 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:26:27.608+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:26:27.609526+0000) 2024-03-04T14:26:28.610 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:26:28.608+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:26:28.609773+0000) 2024-03-04T14:26:29.610 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:26:29.609+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:26:29.610037+0000) 2024-03-04T14:26:29.711 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:26:29.720 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:26:30.610 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:26:30.609+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:26:30.610233+0000) 2024-03-04T14:26:31.611 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:26:31.609+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:26:31.610481+0000) 2024-03-04T14:26:32.611 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:26:32.610+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:26:32.610723+0000) 2024-03-04T14:26:33.611 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:26:33.610+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:26:33.610988+0000) 2024-03-04T14:26:34.611 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:26:34.610+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:26:34.611201+0000) 2024-03-04T14:26:35.611 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:26:35.610+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:26:35.611457+0000) 2024-03-04T14:26:36.611 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:26:36.611+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:26:36.611702+0000) 2024-03-04T14:26:37.612 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:26:37.611+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:26:37.611949+0000) 2024-03-04T14:26:38.612 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:26:38.611+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:26:38.612176+0000) 2024-03-04T14:26:39.612 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:26:39.611+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:26:39.612420+0000) 2024-03-04T14:26:40.613 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:26:40.612+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:26:40.612666+0000) 2024-03-04T14:26:41.614 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:26:41.612+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:26:41.612946+0000) 2024-03-04T14:26:42.613 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:26:42.612+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:26:42.613255+0000) 2024-03-04T14:26:43.613 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:26:43.612+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:26:43.613450+0000) 2024-03-04T14:26:44.613 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:26:44.612+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:26:44.613677+0000) 2024-03-04T14:26:45.613 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:26:45.613+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:26:45.613948+0000) 2024-03-04T14:26:46.614 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:26:46.613+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:26:46.614194+0000) 2024-03-04T14:26:47.614 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:26:47.613+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:26:47.614390+0000) 2024-03-04T14:26:48.614 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:26:48.614+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:26:48.614623+0000) 2024-03-04T14:26:49.615 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:26:49.614+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:26:49.614856+0000) 2024-03-04T14:26:50.615 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:26:50.614+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:26:50.615068+0000) 2024-03-04T14:26:51.615 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:26:51.614+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:26:51.615323+0000) 2024-03-04T14:26:52.615 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:26:52.615+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:26:52.615577+0000) 2024-03-04T14:26:53.615 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:26:53.615+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:26:53.615864+0000) 2024-03-04T14:26:54.615 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:26:54.615+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:26:54.616117+0000) 2024-03-04T14:26:55.616 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:26:55.616+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:26:55.616363+0000) 2024-03-04T14:26:56.616 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:26:56.616+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:26:56.616607+0000) 2024-03-04T14:26:57.616 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:26:57.616+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:26:57.616817+0000) 2024-03-04T14:26:58.616 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:26:58.616+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:26:58.617072+0000) 2024-03-04T14:26:59.616 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:26:59.617+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:26:59.617317+0000) 2024-03-04T14:26:59.932 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:26:59.941 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:27:00.617 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:27:00.616+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:27:00.617548+0000) 2024-03-04T14:27:01.617 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:27:01.617+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:27:01.617787+0000) 2024-03-04T14:27:02.617 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:27:02.617+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:27:02.618029+0000) 2024-03-04T14:27:03.618 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:27:03.618+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:27:03.618268+0000) 2024-03-04T14:27:04.618 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:27:04.617+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:27:04.618534+0000) 2024-03-04T14:27:05.618 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:27:05.617+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:27:05.618772+0000) 2024-03-04T14:27:06.618 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:27:06.618+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:27:06.618952+0000) 2024-03-04T14:27:07.619 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:27:07.618+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:27:07.619189+0000) 2024-03-04T14:27:08.619 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:27:08.618+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:27:08.619426+0000) 2024-03-04T14:27:09.619 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:27:09.618+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:27:09.619687+0000) 2024-03-04T14:27:10.619 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:27:10.618+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:27:10.619958+0000) 2024-03-04T14:27:11.619 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:27:11.619+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:27:11.620201+0000) 2024-03-04T14:27:12.620 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:27:12.619+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:27:12.620437+0000) 2024-03-04T14:27:13.620 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:27:13.618+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:27:13.620612+0000) 2024-03-04T14:27:14.620 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:27:14.619+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:27:14.620813+0000) 2024-03-04T14:27:15.620 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:27:15.620+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:27:15.621087+0000) 2024-03-04T14:27:16.621 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:27:16.620+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:27:16.621326+0000) 2024-03-04T14:27:17.621 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:27:17.620+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:27:17.621578+0000) 2024-03-04T14:27:18.621 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:27:18.620+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:27:18.621812+0000) 2024-03-04T14:27:19.622 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:27:19.621+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:27:19.622102+0000) 2024-03-04T14:27:20.622 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:27:20.621+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:27:20.622340+0000) 2024-03-04T14:27:21.622 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:27:21.621+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:27:21.622535+0000) 2024-03-04T14:27:22.622 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:27:22.621+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:27:22.622785+0000) 2024-03-04T14:27:23.622 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:27:23.622+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:27:23.623026+0000) 2024-03-04T14:27:24.622 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:27:24.622+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:27:24.623277+0000) 2024-03-04T14:27:25.623 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:27:25.622+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:27:25.623522+0000) 2024-03-04T14:27:26.623 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:27:26.622+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:27:26.623771+0000) 2024-03-04T14:27:27.623 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:27:27.623+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:27:27.624006+0000) 2024-03-04T14:27:28.624 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:27:28.623+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:27:28.624245+0000) 2024-03-04T14:27:29.626 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:27:29.623+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:27:29.624513+0000) 2024-03-04T14:27:30.137 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:27:30.149 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:27:30.624 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:27:30.623+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:27:30.624758+0000) 2024-03-04T14:27:31.624 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:27:31.624+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:27:31.624998+0000) 2024-03-04T14:27:32.625 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:27:32.624+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:27:32.625234+0000) 2024-03-04T14:27:33.626 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:27:33.624+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:27:33.625470+0000) 2024-03-04T14:27:34.625 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:27:34.624+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:27:34.625725+0000) 2024-03-04T14:27:35.625 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:27:35.625+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:27:35.625968+0000) 2024-03-04T14:27:36.625 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:27:36.625+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:27:36.626208+0000) 2024-03-04T14:27:37.626 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:27:37.625+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:27:37.626466+0000) 2024-03-04T14:27:38.626 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:27:38.625+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:27:38.626707+0000) 2024-03-04T14:27:39.626 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:27:39.626+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:27:39.626978+0000) 2024-03-04T14:27:40.627 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:27:40.626+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:27:40.627174+0000) 2024-03-04T14:27:41.627 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:27:41.626+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:27:41.627437+0000) 2024-03-04T14:27:42.627 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:27:42.627+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:27:42.627675+0000) 2024-03-04T14:27:43.627 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:27:43.627+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:27:43.627956+0000) 2024-03-04T14:27:44.627 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:27:44.627+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:27:44.628191+0000) 2024-03-04T14:27:45.628 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:27:45.627+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:27:45.628454+0000) 2024-03-04T14:27:46.628 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:27:46.628+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:27:46.628700+0000) 2024-03-04T14:27:47.628 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:27:47.628+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:27:47.628941+0000) 2024-03-04T14:27:48.629 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:27:48.628+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:27:48.629179+0000) 2024-03-04T14:27:49.630 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:27:49.628+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:27:49.629423+0000) 2024-03-04T14:27:50.629 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:27:50.629+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:27:50.629671+0000) 2024-03-04T14:27:51.629 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:27:51.629+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:27:51.629947+0000) 2024-03-04T14:27:52.629 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:27:52.629+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:27:52.630183+0000) 2024-03-04T14:27:53.630 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:27:53.630+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:27:53.630419+0000) 2024-03-04T14:27:54.630 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:27:54.630+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:27:54.630660+0000) 2024-03-04T14:27:55.630 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:27:55.630+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:27:55.630935+0000) 2024-03-04T14:27:56.630 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:27:56.630+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:27:56.631132+0000) 2024-03-04T14:27:57.631 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:27:57.630+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:27:57.631323+0000) 2024-03-04T14:27:58.631 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:27:58.631+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:27:58.631561+0000) 2024-03-04T14:27:59.631 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:27:59.631+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:27:59.631812+0000) 2024-03-04T14:28:00.360 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:28:00.368 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:28:00.631 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:28:00.631+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:28:00.632140+0000) 2024-03-04T14:28:01.632 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:28:01.632+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:28:01.632439+0000) 2024-03-04T14:28:02.632 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:28:02.631+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:28:02.632729+0000) 2024-03-04T14:28:03.632 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:28:03.632+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:28:03.632965+0000) 2024-03-04T14:28:04.632 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:28:04.632+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:28:04.633203+0000) 2024-03-04T14:28:05.633 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:28:05.633+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:28:05.633445+0000) 2024-03-04T14:28:06.633 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:28:06.633+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:28:06.633691+0000) 2024-03-04T14:28:07.633 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:28:07.633+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:28:07.633946+0000) 2024-03-04T14:28:08.633 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:28:08.634+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:28:08.634205+0000) 2024-03-04T14:28:09.634 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:28:09.633+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:28:09.634472+0000) 2024-03-04T14:28:10.634 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:28:10.633+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:28:10.634755+0000) 2024-03-04T14:28:11.634 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:28:11.633+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:28:11.635030+0000) 2024-03-04T14:28:12.635 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:28:12.634+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:28:12.635292+0000) 2024-03-04T14:28:13.635 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:28:13.634+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:28:13.635548+0000) 2024-03-04T14:28:14.635 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:28:14.634+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:28:14.635810+0000) 2024-03-04T14:28:15.636 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:28:15.634+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:28:15.636120+0000) 2024-03-04T14:28:16.636 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:28:16.635+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:28:16.636362+0000) 2024-03-04T14:28:17.636 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:28:17.635+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:28:17.636543+0000) 2024-03-04T14:28:18.636 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:28:18.635+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:28:18.636878+0000) 2024-03-04T14:28:19.637 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:28:19.636+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:28:19.637129+0000) 2024-03-04T14:28:20.637 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:28:20.636+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:28:20.637389+0000) 2024-03-04T14:28:21.637 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:28:21.636+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:28:21.637646+0000) 2024-03-04T14:28:22.637 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:28:22.636+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:28:22.637935+0000) 2024-03-04T14:28:23.637 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:28:23.637+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:28:23.638174+0000) 2024-03-04T14:28:24.637 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:28:24.637+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:28:24.638411+0000) 2024-03-04T14:28:25.638 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:28:25.637+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:28:25.638659+0000) 2024-03-04T14:28:26.638 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:28:26.638+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:28:26.638968+0000) 2024-03-04T14:28:27.638 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:28:27.638+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:28:27.639208+0000) 2024-03-04T14:28:28.638 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:28:28.638+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:28:28.639447+0000) 2024-03-04T14:28:29.639 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:28:29.638+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:28:29.639690+0000) 2024-03-04T14:28:30.506 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:28:30.515 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:28:30.645 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:28:30.639+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:28:30.639993+0000) 2024-03-04T14:28:31.640 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:28:31.639+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:28:31.640236+0000) 2024-03-04T14:28:32.640 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:28:32.639+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:28:32.640484+0000) 2024-03-04T14:28:33.640 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:28:33.639+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:28:33.640740+0000) 2024-03-04T14:28:34.640 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:28:34.640+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:28:34.640990+0000) 2024-03-04T14:28:35.641 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:28:35.640+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:28:35.641238+0000) 2024-03-04T14:28:36.641 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:28:36.640+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:28:36.641484+0000) 2024-03-04T14:28:37.641 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:28:37.641+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:28:37.641721+0000) 2024-03-04T14:28:38.641 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:28:38.641+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:28:38.641960+0000) 2024-03-04T14:28:39.641 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:28:39.641+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:28:39.642240+0000) 2024-03-04T14:28:40.642 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:28:40.641+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:28:40.642514+0000) 2024-03-04T14:28:41.642 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:28:41.642+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:28:41.642780+0000) 2024-03-04T14:28:42.642 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:28:42.642+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:28:42.643045+0000) 2024-03-04T14:28:43.643 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:28:43.642+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:28:43.643285+0000) 2024-03-04T14:28:44.643 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:28:44.642+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:28:44.643468+0000) 2024-03-04T14:28:45.643 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:28:45.643+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:28:45.643720+0000) 2024-03-04T14:28:46.643 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:28:46.643+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:28:46.643966+0000) 2024-03-04T14:28:47.644 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:28:47.643+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:28:47.644200+0000) 2024-03-04T14:28:48.644 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:28:48.643+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:28:48.644446+0000) 2024-03-04T14:28:49.644 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:28:49.644+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:28:49.644698+0000) 2024-03-04T14:28:50.644 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:28:50.644+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:28:50.644952+0000) 2024-03-04T14:28:51.644 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:28:51.644+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:28:51.645202+0000) 2024-03-04T14:28:52.645 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:28:52.644+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:28:52.645439+0000) 2024-03-04T14:28:53.645 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:28:53.644+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:28:53.645618+0000) 2024-03-04T14:28:54.645 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:28:54.645+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:28:54.645901+0000) 2024-03-04T14:28:55.645 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:28:55.645+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:28:55.646152+0000) 2024-03-04T14:28:56.645 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:28:56.645+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:28:56.646403+0000) 2024-03-04T14:28:57.646 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:28:57.646+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:28:57.646662+0000) 2024-03-04T14:28:58.646 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:28:58.646+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:28:58.646953+0000) 2024-03-04T14:28:59.646 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:28:59.646+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:28:59.647216+0000) 2024-03-04T14:29:00.647 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:29:00.647+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:29:00.647478+0000) 2024-03-04T14:29:00.710 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:29:00.718 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:29:01.647 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:29:01.647+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:29:01.647744+0000) 2024-03-04T14:29:02.647 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:29:02.647+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:29:02.647981+0000) 2024-03-04T14:29:03.648 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:29:03.647+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:29:03.648214+0000) 2024-03-04T14:29:04.648 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:29:04.648+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:29:04.648449+0000) 2024-03-04T14:29:05.648 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:29:05.648+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:29:05.648697+0000) 2024-03-04T14:29:06.648 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:29:06.648+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:29:06.648953+0000) 2024-03-04T14:29:07.648 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:29:07.648+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:29:07.649193+0000) 2024-03-04T14:29:08.649 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:29:08.649+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:29:08.649425+0000) 2024-03-04T14:29:09.649 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:29:09.649+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:29:09.649667+0000) 2024-03-04T14:29:10.649 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:29:10.649+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:29:10.649939+0000) 2024-03-04T14:29:11.649 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:29:11.649+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:29:11.650140+0000) 2024-03-04T14:29:12.650 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:29:12.650+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:29:12.650376+0000) 2024-03-04T14:29:13.650 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:29:13.650+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:29:13.650614+0000) 2024-03-04T14:29:14.650 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:29:14.649+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:29:14.650795+0000) 2024-03-04T14:29:15.650 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:29:15.650+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:29:15.651082+0000) 2024-03-04T14:29:16.650 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:29:16.650+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:29:16.651326+0000) 2024-03-04T14:29:17.651 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:29:17.650+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:29:17.651564+0000) 2024-03-04T14:29:18.651 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:29:18.650+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:29:18.651812+0000) 2024-03-04T14:29:19.651 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:29:19.651+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:29:19.652104+0000) 2024-03-04T14:29:20.652 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:29:20.651+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:29:20.652357+0000) 2024-03-04T14:29:21.652 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:29:21.650+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:29:21.652617+0000) 2024-03-04T14:29:22.652 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:29:22.651+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:29:22.652933+0000) 2024-03-04T14:29:23.652 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:29:23.652+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:29:23.653167+0000) 2024-03-04T14:29:24.652 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:29:24.652+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:29:24.653402+0000) 2024-03-04T14:29:25.653 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:29:25.652+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:29:25.653637+0000) 2024-03-04T14:29:26.653 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:29:26.652+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:29:26.653941+0000) 2024-03-04T14:29:27.653 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:29:27.653+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:29:27.654182+0000) 2024-03-04T14:29:28.653 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:29:28.653+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:29:28.654420+0000) 2024-03-04T14:29:29.654 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:29:29.653+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:29:29.654670+0000) 2024-03-04T14:29:30.654 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:29:30.654+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:29:30.654933+0000) 2024-03-04T14:29:30.849 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:29:30.857 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:29:31.654 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:29:31.654+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:29:31.655172+0000) 2024-03-04T14:29:32.655 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:29:32.654+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:29:32.655409+0000) 2024-03-04T14:29:33.655 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:29:33.654+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:29:33.655644+0000) 2024-03-04T14:29:34.655 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:29:34.655+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:29:34.655929+0000) 2024-03-04T14:29:35.655 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:29:35.655+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:29:35.656233+0000) 2024-03-04T14:29:36.656 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:29:36.655+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:29:36.656477+0000) 2024-03-04T14:29:37.656 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:29:37.655+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:29:37.656716+0000) 2024-03-04T14:29:38.656 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:29:38.656+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:29:38.656962+0000) 2024-03-04T14:29:39.656 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:29:39.656+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:29:39.657207+0000) 2024-03-04T14:29:40.656 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:29:40.656+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:29:40.657460+0000) 2024-03-04T14:29:41.657 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:29:41.656+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:29:41.657722+0000) 2024-03-04T14:29:42.657 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:29:42.657+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:29:42.657957+0000) 2024-03-04T14:29:43.657 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:29:43.657+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:29:43.658194+0000) 2024-03-04T14:29:44.657 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:29:44.657+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:29:44.658422+0000) 2024-03-04T14:29:45.658 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:29:45.657+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:29:45.658664+0000) 2024-03-04T14:29:46.658 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:29:46.658+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:29:46.659010+0000) 2024-03-04T14:29:47.658 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:29:47.658+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:29:47.659248+0000) 2024-03-04T14:29:48.658 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:29:48.658+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:29:48.659534+0000) 2024-03-04T14:29:49.659 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:29:49.658+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:29:49.659724+0000) 2024-03-04T14:29:50.659 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:29:50.659+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:29:50.659961+0000) 2024-03-04T14:29:51.659 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:29:51.659+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:29:51.660201+0000) 2024-03-04T14:29:52.659 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:29:52.659+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:29:52.660446+0000) 2024-03-04T14:29:53.660 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:29:53.659+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:29:53.660697+0000) 2024-03-04T14:29:54.660 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:29:54.660+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:29:54.660970+0000) 2024-03-04T14:29:55.660 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:29:55.660+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:29:55.661197+0000) 2024-03-04T14:29:56.660 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:29:56.660+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:29:56.661440+0000) 2024-03-04T14:29:57.661 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:29:57.661+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:29:57.661684+0000) 2024-03-04T14:29:58.661 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:29:58.661+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:29:58.661940+0000) 2024-03-04T14:29:59.661 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:29:59.661+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:29:59.662185+0000) 2024-03-04T14:30:00.661 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:30:00.661+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:30:00.662419+0000) 2024-03-04T14:30:01.062 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:30:01.070 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:30:01.662 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:30:01.661+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:30:01.662684+0000) 2024-03-04T14:30:02.662 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:30:02.662+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:30:02.662945+0000) 2024-03-04T14:30:03.662 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:30:03.662+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:30:03.663178+0000) 2024-03-04T14:30:04.662 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:30:04.662+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:30:04.663371+0000) 2024-03-04T14:30:05.663 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:30:05.662+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:30:05.663620+0000) 2024-03-04T14:30:06.663 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:30:06.663+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:30:06.663920+0000) 2024-03-04T14:30:07.663 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:30:07.663+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:30:07.664112+0000) 2024-03-04T14:30:08.663 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:30:08.664+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:30:08.664348+0000) 2024-03-04T14:30:09.663 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:30:09.663+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:30:09.664591+0000) 2024-03-04T14:30:10.664 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:30:10.664+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:30:10.664889+0000) 2024-03-04T14:30:11.664 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:30:11.664+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:30:11.665248+0000) 2024-03-04T14:30:12.664 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:30:12.665+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:30:12.665488+0000) 2024-03-04T14:30:13.665 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:30:13.665+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:30:13.665718+0000) 2024-03-04T14:30:14.665 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:30:14.665+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:30:14.665956+0000) 2024-03-04T14:30:15.665 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:30:15.665+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:30:15.666190+0000) 2024-03-04T14:30:16.665 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:30:16.666+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:30:16.666429+0000) 2024-03-04T14:30:17.665 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:30:17.665+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:30:17.666622+0000) 2024-03-04T14:30:18.666 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:30:18.666+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:30:18.666965+0000) 2024-03-04T14:30:19.666 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:30:19.667+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:30:19.667208+0000) 2024-03-04T14:30:20.666 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:30:20.667+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:30:20.667455+0000) 2024-03-04T14:30:21.667 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:30:21.667+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:30:21.667695+0000) 2024-03-04T14:30:22.667 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:30:22.666+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:30:22.667943+0000) 2024-03-04T14:30:23.667 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:30:23.667+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:30:23.668194+0000) 2024-03-04T14:30:24.667 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:30:24.667+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:30:24.668438+0000) 2024-03-04T14:30:25.668 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:30:25.666+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:30:25.668687+0000) 2024-03-04T14:30:26.668 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:30:26.667+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:30:26.668958+0000) 2024-03-04T14:30:27.668 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:30:27.668+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:30:27.669198+0000) 2024-03-04T14:30:28.668 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:30:28.668+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:30:28.669432+0000) 2024-03-04T14:30:29.669 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:30:29.668+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:30:29.669622+0000) 2024-03-04T14:30:30.669 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:30:30.668+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:30:30.669925+0000) 2024-03-04T14:30:31.208 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:30:31.216 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:30:31.669 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:30:31.669+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:30:31.670173+0000) 2024-03-04T14:30:32.669 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:30:32.669+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:30:32.670366+0000) 2024-03-04T14:30:33.669 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:30:33.669+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:30:33.670628+0000) 2024-03-04T14:30:34.670 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:30:34.670+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:30:34.670920+0000) 2024-03-04T14:30:35.670 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:30:35.670+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:30:35.671156+0000) 2024-03-04T14:30:36.670 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:30:36.670+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:30:36.671408+0000) 2024-03-04T14:30:37.671 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:30:37.670+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:30:37.671647+0000) 2024-03-04T14:30:38.671 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:30:38.671+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:30:38.671931+0000) 2024-03-04T14:30:39.671 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:30:39.671+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:30:39.672168+0000) 2024-03-04T14:30:40.671 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:30:40.671+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:30:40.672409+0000) 2024-03-04T14:30:41.672 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:30:41.671+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:30:41.672668+0000) 2024-03-04T14:30:42.672 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:30:42.672+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:30:42.672988+0000) 2024-03-04T14:30:43.672 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:30:43.672+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:30:43.673224+0000) 2024-03-04T14:30:44.672 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:30:44.672+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:30:44.673465+0000) 2024-03-04T14:30:45.673 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:30:45.672+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:30:45.673702+0000) 2024-03-04T14:30:46.673 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:30:46.673+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:30:46.673942+0000) 2024-03-04T14:30:47.673 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:30:47.673+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:30:47.674188+0000) 2024-03-04T14:30:48.673 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:30:48.673+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:30:48.674422+0000) 2024-03-04T14:30:49.674 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:30:49.673+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:30:49.674672+0000) 2024-03-04T14:30:50.674 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:30:50.674+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:30:50.674955+0000) 2024-03-04T14:30:51.674 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:30:51.674+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:30:51.675201+0000) 2024-03-04T14:30:52.674 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:30:52.674+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:30:52.675442+0000) 2024-03-04T14:30:53.675 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:30:53.674+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:30:53.675672+0000) 2024-03-04T14:30:54.675 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:30:54.675+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:30:54.675935+0000) 2024-03-04T14:30:55.675 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:30:55.675+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:30:55.676177+0000) 2024-03-04T14:30:56.675 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:30:56.675+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:30:56.676418+0000) 2024-03-04T14:30:57.675 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:30:57.676+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:30:57.676659+0000) 2024-03-04T14:30:58.676 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:30:58.676+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:30:58.676945+0000) 2024-03-04T14:30:59.676 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:30:59.676+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:30:59.677191+0000) 2024-03-04T14:31:00.676 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:31:00.676+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:31:00.677428+0000) 2024-03-04T14:31:01.430 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:31:01.442 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:31:01.676 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:31:01.677+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:31:01.677676+0000) 2024-03-04T14:31:02.677 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:31:02.677+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:31:02.677954+0000) 2024-03-04T14:31:03.677 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:31:03.677+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:31:03.678190+0000) 2024-03-04T14:31:04.677 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:31:04.677+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:31:04.678432+0000) 2024-03-04T14:31:05.677 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:31:05.678+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:31:05.678672+0000) 2024-03-04T14:31:06.678 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:31:06.678+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:31:06.678952+0000) 2024-03-04T14:31:07.678 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:31:07.678+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:31:07.679192+0000) 2024-03-04T14:31:08.678 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:31:08.679+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:31:08.679436+0000) 2024-03-04T14:31:09.678 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:31:09.679+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:31:09.679679+0000) 2024-03-04T14:31:10.679 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:31:10.679+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:31:10.679957+0000) 2024-03-04T14:31:11.679 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:31:11.679+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:31:11.680220+0000) 2024-03-04T14:31:12.679 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:31:12.680+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:31:12.680479+0000) 2024-03-04T14:31:13.679 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:31:13.679+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:31:13.680715+0000) 2024-03-04T14:31:14.680 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:31:14.680+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:31:14.680979+0000) 2024-03-04T14:31:15.680 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:31:15.680+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:31:15.681214+0000) 2024-03-04T14:31:16.680 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:31:16.681+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:31:16.681455+0000) 2024-03-04T14:31:17.680 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:31:17.680+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:31:17.681710+0000) 2024-03-04T14:31:18.681 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:31:18.681+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:31:18.681947+0000) 2024-03-04T14:31:19.681 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:31:19.681+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:31:19.682195+0000) 2024-03-04T14:31:20.681 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:31:20.682+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:31:20.682441+0000) 2024-03-04T14:31:21.681 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:31:21.682+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:31:21.682681+0000) 2024-03-04T14:31:22.682 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:31:22.682+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:31:22.682964+0000) 2024-03-04T14:31:23.682 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:31:23.682+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:31:23.683204+0000) 2024-03-04T14:31:24.682 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:31:24.682+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:31:24.683459+0000) 2024-03-04T14:31:25.682 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:31:25.683+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:31:25.683692+0000) 2024-03-04T14:31:26.683 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:31:26.683+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:31:26.683933+0000) 2024-03-04T14:31:27.683 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:31:27.683+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:31:27.684124+0000) 2024-03-04T14:31:28.683 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:31:28.683+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:31:28.684359+0000) 2024-03-04T14:31:29.683 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:31:29.683+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:31:29.684612+0000) 2024-03-04T14:31:30.684 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:31:30.683+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:31:30.684913+0000) 2024-03-04T14:31:31.576 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:31:31.585 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:31:31.684 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:31:31.683+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:31:31.685126+0000) 2024-03-04T14:31:32.684 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:31:32.684+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:31:32.685373+0000) 2024-03-04T14:31:33.684 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:31:33.683+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:31:33.685555+0000) 2024-03-04T14:31:34.685 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:31:34.684+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:31:34.685791+0000) 2024-03-04T14:31:35.685 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:31:35.685+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:31:35.686083+0000) 2024-03-04T14:31:36.685 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:31:36.685+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:31:36.686329+0000) 2024-03-04T14:31:37.685 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:31:37.685+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:31:37.686570+0000) 2024-03-04T14:31:38.685 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:31:38.685+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:31:38.686811+0000) 2024-03-04T14:31:39.686 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:31:39.686+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:31:39.687077+0000) 2024-03-04T14:31:40.686 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:31:40.686+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:31:40.687304+0000) 2024-03-04T14:31:41.686 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:31:41.686+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:31:41.687574+0000) 2024-03-04T14:31:42.687 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:31:42.687+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:31:42.687815+0000) 2024-03-04T14:31:43.687 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:31:43.687+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:31:43.688116+0000) 2024-03-04T14:31:44.687 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:31:44.687+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:31:44.688350+0000) 2024-03-04T14:31:45.687 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:31:45.687+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:31:45.688589+0000) 2024-03-04T14:31:46.687 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:31:46.688+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:31:46.688867+0000) 2024-03-04T14:31:47.688 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:31:47.688+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:31:47.689109+0000) 2024-03-04T14:31:48.688 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:31:48.688+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:31:48.689345+0000) 2024-03-04T14:31:49.688 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:31:49.688+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:31:49.689590+0000) 2024-03-04T14:31:50.688 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:31:50.689+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:31:50.689869+0000) 2024-03-04T14:31:51.689 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:31:51.689+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:31:51.690073+0000) 2024-03-04T14:31:52.689 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:31:52.689+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:31:52.690312+0000) 2024-03-04T14:31:53.689 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:31:53.689+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:31:53.690563+0000) 2024-03-04T14:31:54.696 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:31:54.690+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:31:54.690849+0000) 2024-03-04T14:31:55.690 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:31:55.690+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:31:55.691037+0000) 2024-03-04T14:31:56.690 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:31:56.690+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:31:56.691286+0000) 2024-03-04T14:31:57.690 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:31:57.690+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:31:57.691520+0000) 2024-03-04T14:31:58.690 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:31:58.691+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:31:58.691796+0000) 2024-03-04T14:31:59.691 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:31:59.691+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:31:59.692092+0000) 2024-03-04T14:32:00.691 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:32:00.691+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:32:00.692339+0000) 2024-03-04T14:32:01.691 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:32:01.691+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:32:01.692595+0000) 2024-03-04T14:32:01.824 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:32:01.832 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:32:02.691 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:32:02.692+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:32:02.692892+0000) 2024-03-04T14:32:03.692 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:32:03.692+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:32:03.693140+0000) 2024-03-04T14:32:04.692 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:32:04.692+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:32:04.693380+0000) 2024-03-04T14:32:05.692 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:32:05.693+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:32:05.693623+0000) 2024-03-04T14:32:06.693 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:32:06.693+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:32:06.693922+0000) 2024-03-04T14:32:07.693 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:32:07.693+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:32:07.694166+0000) 2024-03-04T14:32:08.693 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:32:08.693+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:32:08.694407+0000) 2024-03-04T14:32:09.694 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:32:09.694+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:32:09.694657+0000) 2024-03-04T14:32:10.694 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:32:10.694+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:32:10.694949+0000) 2024-03-04T14:32:11.694 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:32:11.694+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:32:11.695207+0000) 2024-03-04T14:32:12.696 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:32:12.695+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:32:12.695473+0000) 2024-03-04T14:32:13.695 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:32:13.694+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:32:13.695657+0000) 2024-03-04T14:32:14.695 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:32:14.695+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:32:14.695952+0000) 2024-03-04T14:32:15.695 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:32:15.695+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:32:15.696199+0000) 2024-03-04T14:32:16.695 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:32:16.695+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:32:16.696460+0000) 2024-03-04T14:32:17.695 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:32:17.696+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:32:17.696708+0000) 2024-03-04T14:32:18.696 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:32:18.696+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:32:18.696960+0000) 2024-03-04T14:32:19.696 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:32:19.696+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:32:19.697207+0000) 2024-03-04T14:32:20.696 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:32:20.697+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:32:20.697442+0000) 2024-03-04T14:32:21.697 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:32:21.697+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:32:21.697687+0000) 2024-03-04T14:32:22.697 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:32:22.697+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:32:22.697932+0000) 2024-03-04T14:32:23.697 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:32:23.697+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:32:23.698180+0000) 2024-03-04T14:32:24.697 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:32:24.698+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:32:24.698422+0000) 2024-03-04T14:32:25.697 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:32:25.698+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:32:25.698662+0000) 2024-03-04T14:32:26.698 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:32:26.698+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:32:26.698949+0000) 2024-03-04T14:32:27.698 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:32:27.698+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:32:27.699194+0000) 2024-03-04T14:32:28.698 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:32:28.698+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:32:28.699447+0000) 2024-03-04T14:32:29.698 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:32:29.699+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:32:29.699702+0000) 2024-03-04T14:32:30.699 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:32:30.699+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:32:30.699945+0000) 2024-03-04T14:32:31.699 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:32:31.699+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:32:31.700188+0000) 2024-03-04T14:32:32.028 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:32:32.036 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:32:32.699 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:32:32.700+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:32:32.700455+0000) 2024-03-04T14:32:33.700 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:32:33.699+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:32:33.700699+0000) 2024-03-04T14:32:34.699 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:32:34.699+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:32:34.700948+0000) 2024-03-04T14:32:35.700 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:32:35.699+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:32:35.701147+0000) 2024-03-04T14:32:36.700 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:32:36.700+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:32:36.701393+0000) 2024-03-04T14:32:37.700 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:32:37.700+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:32:37.701604+0000) 2024-03-04T14:32:38.700 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:32:38.700+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:32:38.701896+0000) 2024-03-04T14:32:39.701 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:32:39.701+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:32:39.702147+0000) 2024-03-04T14:32:40.701 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:32:40.701+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:32:40.702385+0000) 2024-03-04T14:32:41.701 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:32:41.700+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:32:41.702627+0000) 2024-03-04T14:32:42.701 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:32:42.701+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:32:42.702950+0000) 2024-03-04T14:32:43.702 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:32:43.702+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:32:43.703187+0000) 2024-03-04T14:32:44.702 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:32:44.702+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:32:44.703426+0000) 2024-03-04T14:32:45.702 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:32:45.702+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:32:45.703667+0000) 2024-03-04T14:32:46.703 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:32:46.703+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:32:46.703950+0000) 2024-03-04T14:32:47.703 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:32:47.703+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:32:47.704195+0000) 2024-03-04T14:32:48.703 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:32:48.703+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:32:48.704388+0000) 2024-03-04T14:32:49.703 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:32:49.703+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:32:49.704681+0000) 2024-03-04T14:32:50.703 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:32:50.704+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:32:50.704944+0000) 2024-03-04T14:32:51.704 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:32:51.704+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:32:51.705148+0000) 2024-03-04T14:32:52.704 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:32:52.704+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:32:52.705361+0000) 2024-03-04T14:32:53.704 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:32:53.704+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:32:53.705645+0000) 2024-03-04T14:32:54.704 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:32:54.705+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:32:54.705951+0000) 2024-03-04T14:32:55.705 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:32:55.705+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:32:55.706258+0000) 2024-03-04T14:32:56.705 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:32:56.705+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:32:56.706539+0000) 2024-03-04T14:32:57.705 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:32:57.706+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:32:57.706876+0000) 2024-03-04T14:32:58.706 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:32:58.706+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:32:58.707168+0000) 2024-03-04T14:32:59.706 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:32:59.706+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:32:59.707430+0000) 2024-03-04T14:33:00.706 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:33:00.706+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:33:00.707668+0000) 2024-03-04T14:33:01.706 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:33:01.707+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:33:01.707993+0000) 2024-03-04T14:33:02.250 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:33:02.258 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:33:02.707 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:33:02.707+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:33:02.708261+0000) 2024-03-04T14:33:03.707 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:33:03.707+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:33:03.708495+0000) 2024-03-04T14:33:04.707 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:33:04.707+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:33:04.708744+0000) 2024-03-04T14:33:05.707 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:33:05.708+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:33:05.708982+0000) 2024-03-04T14:33:06.708 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:33:06.708+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:33:06.709257+0000) 2024-03-04T14:33:07.708 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:33:07.708+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:33:07.709501+0000) 2024-03-04T14:33:08.708 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:33:08.709+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:33:08.709782+0000) 2024-03-04T14:33:09.709 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:33:09.709+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:33:09.710081+0000) 2024-03-04T14:33:10.709 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:33:10.709+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:33:10.710391+0000) 2024-03-04T14:33:11.709 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:33:11.710+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:33:11.710700+0000) 2024-03-04T14:33:12.710 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:33:12.710+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:33:12.710985+0000) 2024-03-04T14:33:13.710 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:33:13.710+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:33:13.711276+0000) 2024-03-04T14:33:14.710 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:33:14.711+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:33:14.711571+0000) 2024-03-04T14:33:15.710 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:33:15.711+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:33:15.711902+0000) 2024-03-04T14:33:16.711 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:33:16.711+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:33:16.712250+0000) 2024-03-04T14:33:17.711 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:33:17.712+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:33:17.712540+0000) 2024-03-04T14:33:18.711 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:33:18.712+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:33:18.712862+0000) 2024-03-04T14:33:19.712 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:33:19.712+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:33:19.713163+0000) 2024-03-04T14:33:20.712 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:33:20.713+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:33:20.713450+0000) 2024-03-04T14:33:21.712 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:33:21.713+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:33:21.713746+0000) 2024-03-04T14:33:22.712 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:33:22.713+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:33:22.714048+0000) 2024-03-04T14:33:23.713 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:33:23.713+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:33:23.714341+0000) 2024-03-04T14:33:24.713 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:33:24.713+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:33:24.714643+0000) 2024-03-04T14:33:25.713 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:33:25.714+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:33:25.714966+0000) 2024-03-04T14:33:26.714 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:33:26.714+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:33:26.715255+0000) 2024-03-04T14:33:27.714 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:33:27.715+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:33:27.715556+0000) 2024-03-04T14:33:28.714 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:33:28.715+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:33:28.715894+0000) 2024-03-04T14:33:29.715 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:33:29.715+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:33:29.716154+0000) 2024-03-04T14:33:30.715 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:33:30.716+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:33:30.716417+0000) 2024-03-04T14:33:31.715 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:33:31.716+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:33:31.716664+0000) 2024-03-04T14:33:32.446 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:33:32.454 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:33:32.715 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:33:32.716+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:33:32.716972+0000) 2024-03-04T14:33:33.716 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:33:33.717+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:33:33.717209+0000) 2024-03-04T14:33:34.716 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:33:34.717+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:33:34.717446+0000) 2024-03-04T14:33:35.716 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:33:35.717+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:33:35.717681+0000) 2024-03-04T14:33:36.716 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:33:36.717+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:33:36.717964+0000) 2024-03-04T14:33:37.717 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:33:37.718+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:33:37.718239+0000) 2024-03-04T14:33:38.717 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:33:38.717+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:33:38.718499+0000) 2024-03-04T14:33:39.717 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:33:39.717+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:33:39.718744+0000) 2024-03-04T14:33:40.717 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:33:40.717+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:33:40.718984+0000) 2024-03-04T14:33:41.718 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:33:41.718+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:33:41.719232+0000) 2024-03-04T14:33:42.718 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:33:42.718+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:33:42.719483+0000) 2024-03-04T14:33:43.718 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:33:43.718+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:33:43.719725+0000) 2024-03-04T14:33:44.719 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:33:44.718+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:33:44.719981+0000) 2024-03-04T14:33:45.719 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:33:45.719+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:33:45.720245+0000) 2024-03-04T14:33:46.719 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:33:46.718+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:33:46.720492+0000) 2024-03-04T14:33:47.719 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:33:47.719+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:33:47.720731+0000) 2024-03-04T14:33:48.719 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:33:48.720+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:33:48.720969+0000) 2024-03-04T14:33:49.720 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:33:49.720+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:33:49.721213+0000) 2024-03-04T14:33:50.720 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:33:50.720+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:33:50.721449+0000) 2024-03-04T14:33:51.720 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:33:51.720+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:33:51.721702+0000) 2024-03-04T14:33:52.720 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:33:52.721+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:33:52.721943+0000) 2024-03-04T14:33:53.721 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:33:53.721+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:33:53.722198+0000) 2024-03-04T14:33:54.721 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:33:54.721+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:33:54.722461+0000) 2024-03-04T14:33:55.721 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:33:55.721+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:33:55.722701+0000) 2024-03-04T14:33:56.721 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:33:56.722+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:33:56.722936+0000) 2024-03-04T14:33:57.722 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:33:57.722+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:33:57.723185+0000) 2024-03-04T14:33:58.722 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:33:58.722+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:33:58.723420+0000) 2024-03-04T14:33:59.722 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:33:59.722+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:33:59.723719+0000) 2024-03-04T14:34:00.722 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:34:00.723+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:34:00.723955+0000) 2024-03-04T14:34:01.723 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:34:01.723+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:34:01.724196+0000) 2024-03-04T14:34:02.586 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:34:02.594 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:34:02.727 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:34:02.723+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:34:02.724471+0000) 2024-03-04T14:34:03.723 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:34:03.723+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:34:03.724704+0000) 2024-03-04T14:34:04.723 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:34:04.724+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:34:04.724940+0000) 2024-03-04T14:34:05.724 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:34:05.724+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:34:05.725182+0000) 2024-03-04T14:34:06.724 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:34:06.724+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:34:06.725531+0000) 2024-03-04T14:34:07.725 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:34:07.725+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:34:07.725874+0000) 2024-03-04T14:34:08.725 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:34:08.725+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:34:08.726166+0000) 2024-03-04T14:34:09.725 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:34:09.725+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:34:09.726473+0000) 2024-03-04T14:34:10.726 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:34:10.725+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:34:10.726745+0000) 2024-03-04T14:34:11.725 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:34:11.726+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:34:11.727024+0000) 2024-03-04T14:34:12.726 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:34:12.726+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:34:12.727259+0000) 2024-03-04T14:34:13.726 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:34:13.726+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:34:13.727485+0000) 2024-03-04T14:34:14.726 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:34:14.727+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:34:14.727727+0000) 2024-03-04T14:34:15.726 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:34:15.727+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:34:15.727974+0000) 2024-03-04T14:34:16.727 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:34:16.727+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:34:16.728210+0000) 2024-03-04T14:34:17.727 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:34:17.727+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:34:17.728449+0000) 2024-03-04T14:34:18.727 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:34:18.728+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:34:18.728691+0000) 2024-03-04T14:34:19.727 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:34:19.728+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:34:19.728945+0000) 2024-03-04T14:34:20.728 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:34:20.728+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:34:20.729181+0000) 2024-03-04T14:34:21.728 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:34:21.728+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:34:21.729420+0000) 2024-03-04T14:34:22.728 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:34:22.728+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:34:22.729683+0000) 2024-03-04T14:34:23.728 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:34:23.729+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:34:23.729960+0000) 2024-03-04T14:34:24.729 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:34:24.729+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:34:24.730196+0000) 2024-03-04T14:34:25.729 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:34:25.730+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:34:25.730455+0000) 2024-03-04T14:34:26.729 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:34:26.730+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:34:26.730695+0000) 2024-03-04T14:34:27.729 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:34:27.730+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:34:27.730954+0000) 2024-03-04T14:34:28.729 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:34:28.730+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:34:28.731129+0000) 2024-03-04T14:34:29.730 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:34:29.731+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:34:29.731381+0000) 2024-03-04T14:34:30.730 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:34:30.731+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:34:30.731624+0000) 2024-03-04T14:34:31.730 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:34:31.731+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:34:31.731805+0000) 2024-03-04T14:34:32.730 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:34:32.731+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:34:32.732093+0000) 2024-03-04T14:34:32.799 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:34:32.807 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:34:33.731 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:34:33.732+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:34:33.732328+0000) 2024-03-04T14:34:34.731 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:34:34.732+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:34:34.732569+0000) 2024-03-04T14:34:35.731 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:34:35.732+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:34:35.732808+0000) 2024-03-04T14:34:36.731 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:34:36.732+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:34:36.733120+0000) 2024-03-04T14:34:37.732 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:34:37.733+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:34:37.733380+0000) 2024-03-04T14:34:38.732 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:34:38.732+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:34:38.733643+0000) 2024-03-04T14:34:39.732 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:34:39.733+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:34:39.733941+0000) 2024-03-04T14:34:40.733 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:34:40.734+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:34:40.734182+0000) 2024-03-04T14:34:41.733 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:34:41.734+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:34:41.734421+0000) 2024-03-04T14:34:42.733 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:34:42.734+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:34:42.734665+0000) 2024-03-04T14:34:43.733 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:34:43.733+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:34:43.734937+0000) 2024-03-04T14:34:44.734 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:34:44.734+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:34:44.735173+0000) 2024-03-04T14:34:45.734 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:34:45.734+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:34:45.735409+0000) 2024-03-04T14:34:46.734 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:34:46.734+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:34:46.735674+0000) 2024-03-04T14:34:47.734 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:34:47.734+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:34:47.735955+0000) 2024-03-04T14:34:48.735 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:34:48.735+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:34:48.736188+0000) 2024-03-04T14:34:49.735 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:34:49.735+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:34:49.736437+0000) 2024-03-04T14:34:50.735 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:34:50.735+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:34:50.736678+0000) 2024-03-04T14:34:51.736 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:34:51.735+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:34:51.736954+0000) 2024-03-04T14:34:52.736 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:34:52.736+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:34:52.737250+0000) 2024-03-04T14:34:53.736 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:34:53.736+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:34:53.737485+0000) 2024-03-04T14:34:54.736 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:34:54.736+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:34:54.737724+0000) 2024-03-04T14:34:55.736 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:34:55.737+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:34:55.737972+0000) 2024-03-04T14:34:56.737 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:34:56.737+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:34:56.738215+0000) 2024-03-04T14:34:57.737 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:34:57.737+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:34:57.738477+0000) 2024-03-04T14:34:58.737 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:34:58.737+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:34:58.738732+0000) 2024-03-04T14:34:59.737 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:34:59.738+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:34:59.738978+0000) 2024-03-04T14:35:00.737 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:35:00.738+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:35:00.739217+0000) 2024-03-04T14:35:01.738 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:35:01.738+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:35:01.739409+0000) 2024-03-04T14:35:02.738 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:35:02.738+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:35:02.739655+0000) 2024-03-04T14:35:02.945 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:35:02.953 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:35:03.738 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:35:03.739+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:35:03.739937+0000) 2024-03-04T14:35:04.738 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:35:04.739+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:35:04.740179+0000) 2024-03-04T14:35:05.739 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:35:05.739+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:35:05.740419+0000) 2024-03-04T14:35:06.739 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:35:06.739+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:35:06.740603+0000) 2024-03-04T14:35:07.739 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:35:07.740+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:35:07.740923+0000) 2024-03-04T14:35:08.739 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:35:08.740+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:35:08.741180+0000) 2024-03-04T14:35:09.740 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:35:09.740+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:35:09.741424+0000) 2024-03-04T14:35:10.740 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:35:10.740+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:35:10.741657+0000) 2024-03-04T14:35:11.740 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:35:11.741+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:35:11.741948+0000) 2024-03-04T14:35:12.740 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:35:12.741+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:35:12.742187+0000) 2024-03-04T14:35:13.741 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:35:13.741+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:35:13.742422+0000) 2024-03-04T14:35:14.741 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:35:14.741+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:35:14.742619+0000) 2024-03-04T14:35:15.741 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:35:15.742+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:35:15.742939+0000) 2024-03-04T14:35:16.741 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:35:16.742+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:35:16.743129+0000) 2024-03-04T14:35:17.743 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:35:17.742+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:35:17.743312+0000) 2024-03-04T14:35:18.742 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:35:18.742+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:35:18.743502+0000) 2024-03-04T14:35:19.742 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:35:19.743+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:35:19.743750+0000) 2024-03-04T14:35:20.742 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:35:20.743+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:35:20.744007+0000) 2024-03-04T14:35:21.743 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:35:21.743+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:35:21.744259+0000) 2024-03-04T14:35:22.743 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:35:22.744+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:35:22.744516+0000) 2024-03-04T14:35:23.744 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:35:23.744+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:35:23.744773+0000) 2024-03-04T14:35:24.744 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:35:24.744+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:35:24.745038+0000) 2024-03-04T14:35:25.744 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:35:25.744+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:35:25.745287+0000) 2024-03-04T14:35:26.744 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:35:26.745+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:35:26.745539+0000) 2024-03-04T14:35:27.745 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:35:27.745+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:35:27.745777+0000) 2024-03-04T14:35:28.745 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:35:28.745+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:35:28.746019+0000) 2024-03-04T14:35:29.745 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:35:29.745+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:35:29.746270+0000) 2024-03-04T14:35:30.745 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:35:30.746+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:35:30.746518+0000) 2024-03-04T14:35:31.745 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:35:31.746+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:35:31.746764+0000) 2024-03-04T14:35:32.745 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:35:32.746+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:35:32.747022+0000) 2024-03-04T14:35:33.158 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:35:33.170 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:35:33.746 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:35:33.746+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:35:33.747216+0000) 2024-03-04T14:35:34.746 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:35:34.746+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:35:34.747406+0000) 2024-03-04T14:35:35.746 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:35:35.747+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:35:35.747661+0000) 2024-03-04T14:35:36.746 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:35:36.747+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:35:36.747949+0000) 2024-03-04T14:35:37.747 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:35:37.747+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:35:37.748209+0000) 2024-03-04T14:35:38.747 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:35:38.747+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:35:38.748408+0000) 2024-03-04T14:35:39.747 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:35:39.748+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:35:39.748695+0000) 2024-03-04T14:35:40.748 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:35:40.748+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:35:40.748951+0000) 2024-03-04T14:35:41.748 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:35:41.748+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:35:41.749203+0000) 2024-03-04T14:35:42.748 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:35:42.749+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:35:42.749458+0000) 2024-03-04T14:35:43.748 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:35:43.749+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:35:43.749709+0000) 2024-03-04T14:35:44.748 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:35:44.749+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:35:44.749946+0000) 2024-03-04T14:35:45.749 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:35:45.750+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:35:45.750185+0000) 2024-03-04T14:35:46.750 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:35:46.750+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:35:46.750482+0000) 2024-03-04T14:35:47.749 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:35:47.749+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:35:47.750720+0000) 2024-03-04T14:35:48.750 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:35:48.749+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:35:48.750956+0000) 2024-03-04T14:35:49.750 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:35:49.750+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:35:49.751229+0000) 2024-03-04T14:35:50.750 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:35:50.750+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:35:50.751495+0000) 2024-03-04T14:35:51.750 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:35:51.749+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:35:51.751791+0000) 2024-03-04T14:35:52.751 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:35:52.751+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:35:52.752108+0000) 2024-03-04T14:35:53.751 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:35:53.750+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:35:53.752345+0000) 2024-03-04T14:35:54.751 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:35:54.751+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:35:54.752638+0000) 2024-03-04T14:35:55.751 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:35:55.751+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:35:55.752968+0000) 2024-03-04T14:35:56.752 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:35:56.752+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:35:56.753253+0000) 2024-03-04T14:35:57.752 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:35:57.752+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:35:57.753545+0000) 2024-03-04T14:35:58.752 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:35:58.753+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:35:58.753875+0000) 2024-03-04T14:35:59.752 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:35:59.753+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:35:59.754146+0000) 2024-03-04T14:36:00.753 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:36:00.753+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:36:00.754475+0000) 2024-03-04T14:36:01.753 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:36:01.753+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:36:01.754764+0000) 2024-03-04T14:36:02.754 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:36:02.754+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:36:02.755054+0000) 2024-03-04T14:36:03.371 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:36:03.380 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:36:03.754 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:36:03.754+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:36:03.755308+0000) 2024-03-04T14:36:04.754 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:36:04.754+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:36:04.755545+0000) 2024-03-04T14:36:05.754 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:36:05.754+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:36:05.755779+0000) 2024-03-04T14:36:06.754 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:36:06.755+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:36:06.756015+0000) 2024-03-04T14:36:07.754 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:36:07.755+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:36:07.756258+0000) 2024-03-04T14:36:08.755 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:36:08.755+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:36:08.756512+0000) 2024-03-04T14:36:09.755 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:36:09.755+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:36:09.756765+0000) 2024-03-04T14:36:10.755 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:36:10.756+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:36:10.757011+0000) 2024-03-04T14:36:11.755 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:36:11.756+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:36:11.757243+0000) 2024-03-04T14:36:12.756 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:36:12.756+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:36:12.757484+0000) 2024-03-04T14:36:13.756 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:36:13.757+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:36:13.757726+0000) 2024-03-04T14:36:14.757 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:36:14.757+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:36:14.757988+0000) 2024-03-04T14:36:15.756 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:36:15.757+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:36:15.758243+0000) 2024-03-04T14:36:16.757 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:36:16.757+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:36:16.758474+0000) 2024-03-04T14:36:17.757 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:36:17.758+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:36:17.758723+0000) 2024-03-04T14:36:18.758 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:36:18.758+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:36:18.758975+0000) 2024-03-04T14:36:19.758 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:36:19.758+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:36:19.759265+0000) 2024-03-04T14:36:20.760 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:36:20.758+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:36:20.759562+0000) 2024-03-04T14:36:21.758 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:36:21.758+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:36:21.759804+0000) 2024-03-04T14:36:22.758 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:36:22.759+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:36:22.760166+0000) 2024-03-04T14:36:23.759 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:36:23.759+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:36:23.760387+0000) 2024-03-04T14:36:24.759 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:36:24.760+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:36:24.760679+0000) 2024-03-04T14:36:25.760 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:36:25.760+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:36:25.760966+0000) 2024-03-04T14:36:26.760 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:36:26.760+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:36:26.761253+0000) 2024-03-04T14:36:27.760 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:36:27.761+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:36:27.761543+0000) 2024-03-04T14:36:28.760 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:36:28.761+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:36:28.761882+0000) 2024-03-04T14:36:29.760 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:36:29.761+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:36:29.762189+0000) 2024-03-04T14:36:30.761 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:36:30.761+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:36:30.762471+0000) 2024-03-04T14:36:31.761 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:36:31.762+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:36:31.762770+0000) 2024-03-04T14:36:32.762 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:36:32.762+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:36:32.763075+0000) 2024-03-04T14:36:33.520 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:36:33.528 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:36:33.762 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:36:33.762+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:36:33.763340+0000) 2024-03-04T14:36:34.762 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:36:34.762+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:36:34.763535+0000) 2024-03-04T14:36:35.762 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:36:35.763+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:36:35.763793+0000) 2024-03-04T14:36:36.763 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:36:36.763+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:36:36.764066+0000) 2024-03-04T14:36:37.762 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:36:37.763+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:36:37.764309+0000) 2024-03-04T14:36:38.763 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:36:38.764+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:36:38.764551+0000) 2024-03-04T14:36:39.763 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:36:39.764+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:36:39.764922+0000) 2024-03-04T14:36:40.764 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:36:40.764+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:36:40.765167+0000) 2024-03-04T14:36:41.764 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:36:41.765+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:36:41.765409+0000) 2024-03-04T14:36:42.764 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:36:42.764+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:36:42.765676+0000) 2024-03-04T14:36:43.764 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:36:43.765+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:36:43.765983+0000) 2024-03-04T14:36:44.764 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:36:44.766+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:36:44.766292+0000) 2024-03-04T14:36:45.765 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:36:45.765+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:36:45.766578+0000) 2024-03-04T14:36:46.765 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:36:46.766+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:36:46.766917+0000) 2024-03-04T14:36:47.765 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:36:47.766+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:36:47.767220+0000) 2024-03-04T14:36:48.766 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:36:48.766+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:36:48.767473+0000) 2024-03-04T14:36:49.766 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:36:49.767+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:36:49.767784+0000) 2024-03-04T14:36:50.766 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:36:50.767+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:36:50.768094+0000) 2024-03-04T14:36:51.767 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:36:51.767+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:36:51.768388+0000) 2024-03-04T14:36:52.767 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:36:52.766+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:36:52.768643+0000) 2024-03-04T14:36:53.767 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:36:53.767+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:36:53.768963+0000) 2024-03-04T14:36:54.767 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:36:54.767+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:36:54.769213+0000) 2024-03-04T14:36:55.768 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:36:55.767+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:36:55.769447+0000) 2024-03-04T14:36:56.768 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:36:56.768+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:36:56.769677+0000) 2024-03-04T14:36:57.768 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:36:57.768+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:36:57.769962+0000) 2024-03-04T14:36:58.769 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:36:58.769+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:36:58.770264+0000) 2024-03-04T14:36:59.769 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:36:59.769+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:36:59.770564+0000) 2024-03-04T14:37:00.769 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:37:00.769+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:37:00.770874+0000) 2024-03-04T14:37:01.770 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:37:01.770+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:37:01.771191+0000) 2024-03-04T14:37:02.770 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:37:02.770+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:37:02.771471+0000) 2024-03-04T14:37:03.741 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:37:03.749 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:37:03.770 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:37:03.770+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:37:03.771737+0000) 2024-03-04T14:37:04.770 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:37:04.771+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:37:04.771994+0000) 2024-03-04T14:37:05.770 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:37:05.771+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:37:05.772251+0000) 2024-03-04T14:37:06.771 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:37:06.771+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:37:06.772515+0000) 2024-03-04T14:37:07.771 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:37:07.771+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:37:07.772755+0000) 2024-03-04T14:37:08.771 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:37:08.772+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:37:08.772953+0000) 2024-03-04T14:37:09.772 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:37:09.772+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:37:09.773203+0000) 2024-03-04T14:37:10.772 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:37:10.772+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:37:10.773437+0000) 2024-03-04T14:37:11.772 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:37:11.772+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:37:11.773674+0000) 2024-03-04T14:37:12.772 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:37:12.773+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:37:12.773954+0000) 2024-03-04T14:37:13.773 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:37:13.773+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:37:13.774192+0000) 2024-03-04T14:37:14.773 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:37:14.773+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:37:14.774431+0000) 2024-03-04T14:37:15.773 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:37:15.773+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:37:15.774674+0000) 2024-03-04T14:37:16.773 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:37:16.774+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:37:16.774943+0000) 2024-03-04T14:37:17.773 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:37:17.774+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:37:17.775181+0000) 2024-03-04T14:37:18.774 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:37:18.774+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:37:18.775417+0000) 2024-03-04T14:37:19.774 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:37:19.774+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:37:19.775662+0000) 2024-03-04T14:37:20.776 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:37:20.775+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:37:20.775934+0000) 2024-03-04T14:37:21.775 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:37:21.775+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:37:21.776115+0000) 2024-03-04T14:37:22.775 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:37:22.775+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:37:22.776404+0000) 2024-03-04T14:37:23.775 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:37:23.776+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:37:23.776646+0000) 2024-03-04T14:37:24.775 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:37:24.776+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:37:24.776922+0000) 2024-03-04T14:37:25.775 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:37:25.776+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:37:25.777163+0000) 2024-03-04T14:37:26.776 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:37:26.776+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:37:26.777400+0000) 2024-03-04T14:37:27.776 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:37:27.776+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:37:27.777640+0000) 2024-03-04T14:37:28.776 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:37:28.777+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:37:28.777910+0000) 2024-03-04T14:37:29.776 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:37:29.777+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:37:29.778162+0000) 2024-03-04T14:37:30.777 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:37:30.777+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:37:30.778405+0000) 2024-03-04T14:37:31.777 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:37:31.778+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:37:31.778671+0000) 2024-03-04T14:37:32.777 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:37:32.778+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:37:32.778939+0000) 2024-03-04T14:37:33.778 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:37:33.778+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:37:33.779184+0000) 2024-03-04T14:37:33.880 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:37:33.889 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:37:34.778 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:37:34.778+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:37:34.779424+0000) 2024-03-04T14:37:35.778 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:37:35.779+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:37:35.779666+0000) 2024-03-04T14:37:36.778 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:37:36.779+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:37:36.779927+0000) 2024-03-04T14:37:37.779 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:37:37.779+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:37:37.780168+0000) 2024-03-04T14:37:38.779 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:37:38.780+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:37:38.780411+0000) 2024-03-04T14:37:39.779 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:37:39.780+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:37:39.780663+0000) 2024-03-04T14:37:40.779 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:37:40.780+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:37:40.780903+0000) 2024-03-04T14:37:41.779 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:37:41.780+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:37:41.781096+0000) 2024-03-04T14:37:42.779 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:37:42.780+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:37:42.781280+0000) 2024-03-04T14:37:43.780 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:37:43.780+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:37:43.781539+0000) 2024-03-04T14:37:44.780 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:37:44.781+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:37:44.781782+0000) 2024-03-04T14:37:45.780 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:37:45.781+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:37:45.782022+0000) 2024-03-04T14:37:46.781 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:37:46.781+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:37:46.782270+0000) 2024-03-04T14:37:47.781 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:37:47.782+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:37:47.782517+0000) 2024-03-04T14:37:48.781 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:37:48.782+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:37:48.782761+0000) 2024-03-04T14:37:49.781 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:37:49.782+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:37:49.783006+0000) 2024-03-04T14:37:50.781 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:37:50.783+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:37:50.783245+0000) 2024-03-04T14:37:51.782 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:37:51.782+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:37:51.783444+0000) 2024-03-04T14:37:52.782 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:37:52.783+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:37:52.783689+0000) 2024-03-04T14:37:53.782 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:37:53.783+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:37:53.783933+0000) 2024-03-04T14:37:54.782 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:37:54.784+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:37:54.784175+0000) 2024-03-04T14:37:55.783 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:37:55.784+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:37:55.784417+0000) 2024-03-04T14:37:56.783 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:37:56.782+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:37:56.784678+0000) 2024-03-04T14:37:57.783 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:37:57.783+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:37:57.784948+0000) 2024-03-04T14:37:58.783 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:37:58.784+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:37:58.785184+0000) 2024-03-04T14:37:59.784 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:37:59.784+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:37:59.785432+0000) 2024-03-04T14:38:00.784 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:38:00.784+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:38:00.785674+0000) 2024-03-04T14:38:01.784 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:38:01.784+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:38:01.785944+0000) 2024-03-04T14:38:02.784 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:38:02.785+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:38:02.786217+0000) 2024-03-04T14:38:03.785 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:38:03.784+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:38:03.786481+0000) 2024-03-04T14:38:04.101 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:38:04.109 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:38:04.785 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:38:04.785+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:38:04.786735+0000) 2024-03-04T14:38:05.785 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:38:05.786+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:38:05.786982+0000) 2024-03-04T14:38:06.785 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:38:06.786+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:38:06.787178+0000) 2024-03-04T14:38:07.786 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:38:07.786+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:38:07.787424+0000) 2024-03-04T14:38:08.786 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:38:08.786+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:38:08.787670+0000) 2024-03-04T14:38:09.787 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:38:09.787+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:38:09.787904+0000) 2024-03-04T14:38:10.786 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:38:10.787+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:38:10.788194+0000) 2024-03-04T14:38:11.787 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:38:11.787+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:38:11.788377+0000) 2024-03-04T14:38:12.787 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:38:12.787+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:38:12.788630+0000) 2024-03-04T14:38:13.787 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:38:13.788+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:38:13.788927+0000) 2024-03-04T14:38:14.787 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:38:14.788+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:38:14.789169+0000) 2024-03-04T14:38:15.787 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:38:15.788+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:38:15.789371+0000) 2024-03-04T14:38:16.788 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:38:16.788+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:38:16.789608+0000) 2024-03-04T14:38:17.788 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:38:17.789+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:38:17.789892+0000) 2024-03-04T14:38:18.788 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:38:18.789+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:38:18.790103+0000) 2024-03-04T14:38:19.788 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:38:19.789+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:38:19.790361+0000) 2024-03-04T14:38:20.789 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:38:20.789+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:38:20.790598+0000) 2024-03-04T14:38:21.789 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:38:21.790+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:38:21.790857+0000) 2024-03-04T14:38:22.789 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:38:22.790+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:38:22.791118+0000) 2024-03-04T14:38:23.789 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:38:23.790+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:38:23.791368+0000) 2024-03-04T14:38:24.790 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:38:24.790+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:38:24.791612+0000) 2024-03-04T14:38:25.790 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:38:25.791+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:38:25.791887+0000) 2024-03-04T14:38:26.790 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:38:26.791+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:38:26.792128+0000) 2024-03-04T14:38:27.790 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:38:27.791+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:38:27.792357+0000) 2024-03-04T14:38:28.791 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:38:28.791+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:38:28.792608+0000) 2024-03-04T14:38:29.791 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:38:29.792+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:38:29.792896+0000) 2024-03-04T14:38:30.791 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:38:30.792+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:38:30.793136+0000) 2024-03-04T14:38:31.791 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:38:31.792+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:38:31.793372+0000) 2024-03-04T14:38:32.792 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:38:32.793+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:38:32.793611+0000) 2024-03-04T14:38:33.792 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:38:33.793+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:38:33.793911+0000) 2024-03-04T14:38:34.296 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:38:34.305 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:38:34.792 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:38:34.793+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:38:34.794153+0000) 2024-03-04T14:38:35.793 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:38:35.793+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:38:35.794395+0000) 2024-03-04T14:38:36.793 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:38:36.794+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:38:36.794638+0000) 2024-03-04T14:38:37.793 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:38:37.794+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:38:37.794925+0000) 2024-03-04T14:38:38.794 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:38:38.794+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:38:38.795211+0000) 2024-03-04T14:38:39.793 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:38:39.794+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:38:39.795417+0000) 2024-03-04T14:38:40.794 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:38:40.795+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:38:40.795652+0000) 2024-03-04T14:38:41.794 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:38:41.795+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:38:41.795927+0000) 2024-03-04T14:38:42.794 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:38:42.795+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:38:42.796168+0000) 2024-03-04T14:38:43.794 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:38:43.796+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:38:43.796416+0000) 2024-03-04T14:38:44.795 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:38:44.796+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:38:44.796657+0000) 2024-03-04T14:38:45.795 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:38:45.796+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:38:45.796933+0000) 2024-03-04T14:38:46.795 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:38:46.796+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:38:46.797179+0000) 2024-03-04T14:38:47.795 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:38:47.797+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:38:47.797424+0000) 2024-03-04T14:38:48.796 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:38:48.797+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:38:48.797698+0000) 2024-03-04T14:38:49.796 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:38:49.797+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:38:49.797940+0000) 2024-03-04T14:38:50.796 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:38:50.797+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:38:50.798176+0000) 2024-03-04T14:38:51.797 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:38:51.798+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:38:51.798414+0000) 2024-03-04T14:38:52.797 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:38:52.798+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:38:52.798656+0000) 2024-03-04T14:38:53.797 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:38:53.798+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:38:53.798963+0000) 2024-03-04T14:38:54.797 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:38:54.798+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:38:54.799225+0000) 2024-03-04T14:38:55.798 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:38:55.799+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:38:55.799476+0000) 2024-03-04T14:38:56.798 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:38:56.798+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:38:56.799730+0000) 2024-03-04T14:38:57.798 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:38:57.799+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:38:57.799972+0000) 2024-03-04T14:38:58.798 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:38:58.800+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:38:58.800214+0000) 2024-03-04T14:38:59.799 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:38:59.800+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:38:59.800487+0000) 2024-03-04T14:39:00.799 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:39:00.799+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:39:00.800736+0000) 2024-03-04T14:39:01.799 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:39:01.799+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:39:01.800972+0000) 2024-03-04T14:39:02.800 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:39:02.800+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:39:02.801213+0000) 2024-03-04T14:39:03.800 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:39:03.800+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:39:03.801473+0000) 2024-03-04T14:39:04.502 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:39:04.512 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:39:04.800 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:39:04.800+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:39:04.801739+0000) 2024-03-04T14:39:05.800 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:39:05.800+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:39:05.801986+0000) 2024-03-04T14:39:06.801 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:39:06.801+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:39:06.802240+0000) 2024-03-04T14:39:07.801 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:39:07.801+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:39:07.802480+0000) 2024-03-04T14:39:08.801 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:39:08.800+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:39:08.802667+0000) 2024-03-04T14:39:09.801 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:39:09.801+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:39:09.802955+0000) 2024-03-04T14:39:10.801 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:39:10.802+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:39:10.803194+0000) 2024-03-04T14:39:11.801 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:39:11.802+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:39:11.803427+0000) 2024-03-04T14:39:12.802 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:39:12.802+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:39:12.803685+0000) 2024-03-04T14:39:13.802 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:39:13.803+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:39:13.803955+0000) 2024-03-04T14:39:14.802 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:39:14.803+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:39:14.804196+0000) 2024-03-04T14:39:15.802 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:39:15.803+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:39:15.804440+0000) 2024-03-04T14:39:16.803 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:39:16.803+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:39:16.804678+0000) 2024-03-04T14:39:17.803 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:39:17.804+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:39:17.804951+0000) 2024-03-04T14:39:18.803 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:39:18.804+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:39:18.805210+0000) 2024-03-04T14:39:19.804 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:39:19.804+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:39:19.805484+0000) 2024-03-04T14:39:20.804 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:39:20.804+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:39:20.805726+0000) 2024-03-04T14:39:21.804 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:39:21.805+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:39:21.805967+0000) 2024-03-04T14:39:22.804 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:39:22.805+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:39:22.806211+0000) 2024-03-04T14:39:23.804 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:39:23.805+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:39:23.806392+0000) 2024-03-04T14:39:24.805 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:39:24.805+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:39:24.806654+0000) 2024-03-04T14:39:25.805 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:39:25.806+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:39:25.806941+0000) 2024-03-04T14:39:26.806 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:39:26.806+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:39:26.807179+0000) 2024-03-04T14:39:27.806 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:39:27.806+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:39:27.807437+0000) 2024-03-04T14:39:28.807 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:39:28.807+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:39:28.807683+0000) 2024-03-04T14:39:29.806 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:39:29.807+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:39:29.807956+0000) 2024-03-04T14:39:30.806 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:39:30.807+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:39:30.808198+0000) 2024-03-04T14:39:31.806 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:39:31.807+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:39:31.808435+0000) 2024-03-04T14:39:32.807 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:39:32.808+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:39:32.808673+0000) 2024-03-04T14:39:33.807 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:39:33.808+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:39:33.808931+0000) 2024-03-04T14:39:34.738 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:39:34.746 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:39:34.807 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:39:34.808+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:39:34.809153+0000) 2024-03-04T14:39:35.807 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:39:35.808+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:39:35.809402+0000) 2024-03-04T14:39:36.808 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:39:36.809+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:39:36.809658+0000) 2024-03-04T14:39:37.808 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:39:37.809+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:39:37.809957+0000) 2024-03-04T14:39:38.808 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:39:38.809+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:39:38.810203+0000) 2024-03-04T14:39:39.808 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:39:39.809+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:39:39.810455+0000) 2024-03-04T14:39:40.809 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:39:40.810+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:39:40.810692+0000) 2024-03-04T14:39:41.809 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:39:41.810+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:39:41.810930+0000) 2024-03-04T14:39:42.809 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:39:42.810+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:39:42.811167+0000) 2024-03-04T14:39:43.810 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:39:43.810+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:39:43.811409+0000) 2024-03-04T14:39:44.810 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:39:44.811+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:39:44.811658+0000) 2024-03-04T14:39:45.810 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:39:45.811+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:39:45.811937+0000) 2024-03-04T14:39:46.810 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:39:46.811+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:39:46.812184+0000) 2024-03-04T14:39:47.811 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:39:47.811+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:39:47.812433+0000) 2024-03-04T14:39:48.811 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:39:48.812+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:39:48.812676+0000) 2024-03-04T14:39:49.811 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:39:49.812+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:39:49.812956+0000) 2024-03-04T14:39:50.811 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:39:50.812+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:39:50.813195+0000) 2024-03-04T14:39:51.812 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:39:51.812+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:39:51.813434+0000) 2024-03-04T14:39:52.812 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:39:52.813+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:39:52.813870+0000) 2024-03-04T14:39:53.812 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:39:53.813+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:39:53.814164+0000) 2024-03-04T14:39:54.813 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:39:54.814+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:39:54.814456+0000) 2024-03-04T14:39:55.813 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:39:55.814+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:39:55.814746+0000) 2024-03-04T14:39:56.813 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:39:56.814+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:39:56.815101+0000) 2024-03-04T14:39:57.814 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:39:57.815+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:39:57.815388+0000) 2024-03-04T14:39:58.814 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:39:58.815+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:39:58.815669+0000) 2024-03-04T14:39:59.814 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:39:59.815+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:39:59.815977+0000) 2024-03-04T14:40:00.814 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:40:00.816+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:40:00.816272+0000) 2024-03-04T14:40:01.815 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:40:01.816+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:40:01.816554+0000) 2024-03-04T14:40:02.815 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:40:02.816+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:40:02.816817+0000) 2024-03-04T14:40:03.815 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:40:03.816+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:40:03.817103+0000) 2024-03-04T14:40:04.815 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:40:04.816+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:40:04.817347+0000) 2024-03-04T14:40:04.886 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:40:04.894 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:40:05.816 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:40:05.816+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:40:05.817532+0000) 2024-03-04T14:40:06.816 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:40:06.815+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:40:06.817768+0000) 2024-03-04T14:40:07.816 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:40:07.816+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:40:07.818017+0000) 2024-03-04T14:40:08.816 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:40:08.817+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:40:08.818264+0000) 2024-03-04T14:40:09.817 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:40:09.817+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:40:09.818518+0000) 2024-03-04T14:40:10.817 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:40:10.817+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:40:10.818778+0000) 2024-03-04T14:40:11.817 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:40:11.818+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:40:11.819016+0000) 2024-03-04T14:40:12.817 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:40:12.818+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:40:12.819268+0000) 2024-03-04T14:40:13.818 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:40:13.818+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:40:13.819509+0000) 2024-03-04T14:40:14.818 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:40:14.818+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:40:14.819755+0000) 2024-03-04T14:40:15.818 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:40:15.819+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:40:15.819995+0000) 2024-03-04T14:40:16.818 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:40:16.819+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:40:16.820182+0000) 2024-03-04T14:40:17.818 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:40:17.819+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:40:17.820424+0000) 2024-03-04T14:40:18.818 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:40:18.819+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:40:18.820643+0000) 2024-03-04T14:40:19.819 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:40:19.820+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:40:19.820953+0000) 2024-03-04T14:40:20.819 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:40:20.820+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:40:20.821158+0000) 2024-03-04T14:40:21.819 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:40:21.820+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:40:21.821360+0000) 2024-03-04T14:40:22.820 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:40:22.820+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:40:22.821597+0000) 2024-03-04T14:40:23.820 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:40:23.821+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:40:23.821868+0000) 2024-03-04T14:40:24.820 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:40:24.821+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:40:24.822105+0000) 2024-03-04T14:40:25.820 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:40:25.821+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:40:25.822342+0000) 2024-03-04T14:40:26.821 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:40:26.821+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:40:26.822603+0000) 2024-03-04T14:40:27.821 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:40:27.822+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:40:27.822866+0000) 2024-03-04T14:40:28.821 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:40:28.822+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:40:28.823110+0000) 2024-03-04T14:40:29.822 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:40:29.822+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:40:29.823360+0000) 2024-03-04T14:40:30.822 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:40:30.822+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:40:30.823595+0000) 2024-03-04T14:40:31.822 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:40:31.823+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:40:31.823864+0000) 2024-03-04T14:40:32.822 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:40:32.823+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:40:32.824123+0000) 2024-03-04T14:40:33.823 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:40:33.823+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:40:33.824388+0000) 2024-03-04T14:40:34.823 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:40:34.823+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:40:34.824634+0000) 2024-03-04T14:40:35.084 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:40:35.092 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:40:35.823 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:40:35.824+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:40:35.824910+0000) 2024-03-04T14:40:36.823 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:40:36.824+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:40:36.825145+0000) 2024-03-04T14:40:37.824 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:40:37.824+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:40:37.825385+0000) 2024-03-04T14:40:38.824 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:40:38.825+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:40:38.825622+0000) 2024-03-04T14:40:39.824 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:40:39.825+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:40:39.825915+0000) 2024-03-04T14:40:40.824 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:40:40.825+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:40:40.826150+0000) 2024-03-04T14:40:41.824 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:40:41.825+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:40:41.826386+0000) 2024-03-04T14:40:42.825 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:40:42.826+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:40:42.826628+0000) 2024-03-04T14:40:43.825 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:40:43.826+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:40:43.826902+0000) 2024-03-04T14:40:44.825 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:40:44.826+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:40:44.827201+0000) 2024-03-04T14:40:45.825 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:40:45.826+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:40:45.827438+0000) 2024-03-04T14:40:46.826 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:40:46.827+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:40:46.827675+0000) 2024-03-04T14:40:47.826 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:40:47.827+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:40:47.827943+0000) 2024-03-04T14:40:48.826 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:40:48.827+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:40:48.828178+0000) 2024-03-04T14:40:49.827 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:40:49.827+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:40:49.828381+0000) 2024-03-04T14:40:50.827 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:40:50.827+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:40:50.828631+0000) 2024-03-04T14:40:51.827 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:40:51.828+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:40:51.828919+0000) 2024-03-04T14:40:52.827 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:40:52.828+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:40:52.829175+0000) 2024-03-04T14:40:53.827 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:40:53.829+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:40:53.829428+0000) 2024-03-04T14:40:54.828 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:40:54.829+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:40:54.829670+0000) 2024-03-04T14:40:55.828 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:40:55.829+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:40:55.829931+0000) 2024-03-04T14:40:56.828 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:40:56.829+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:40:56.830205+0000) 2024-03-04T14:40:57.828 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:40:57.829+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:40:57.830448+0000) 2024-03-04T14:40:58.829 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:40:58.830+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:40:58.830711+0000) 2024-03-04T14:40:59.829 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:40:59.830+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:40:59.830977+0000) 2024-03-04T14:41:00.829 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:41:00.831+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:41:00.831243+0000) 2024-03-04T14:41:01.830 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:41:01.831+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:41:01.831495+0000) 2024-03-04T14:41:02.830 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:41:02.831+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:41:02.831770+0000) 2024-03-04T14:41:03.830 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:41:03.831+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:41:03.832050+0000) 2024-03-04T14:41:04.831 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:41:04.832+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:41:04.832309+0000) 2024-03-04T14:41:05.314 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:41:05.322 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:41:05.830 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:41:05.831+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:41:05.832519+0000) 2024-03-04T14:41:06.831 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:41:06.831+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:41:06.832806+0000) 2024-03-04T14:41:07.831 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:41:07.832+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:41:07.833115+0000) 2024-03-04T14:41:08.832 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:41:08.832+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:41:08.833370+0000) 2024-03-04T14:41:09.832 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:41:09.832+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:41:09.833638+0000) 2024-03-04T14:41:10.832 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:41:10.832+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:41:10.833929+0000) 2024-03-04T14:41:11.832 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:41:11.833+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:41:11.834171+0000) 2024-03-04T14:41:12.832 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:41:12.833+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:41:12.834426+0000) 2024-03-04T14:41:13.833 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:41:13.833+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:41:13.834684+0000) 2024-03-04T14:41:14.833 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:41:14.833+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:41:14.834977+0000) 2024-03-04T14:41:15.833 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:41:15.833+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:41:15.835187+0000) 2024-03-04T14:41:16.833 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:41:16.834+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:41:16.835443+0000) 2024-03-04T14:41:17.834 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:41:17.834+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:41:17.835695+0000) 2024-03-04T14:41:18.834 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:41:18.834+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:41:18.835940+0000) 2024-03-04T14:41:19.834 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:41:19.835+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:41:19.836206+0000) 2024-03-04T14:41:20.835 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:41:20.835+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:41:20.836460+0000) 2024-03-04T14:41:21.835 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:41:21.835+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:41:21.836714+0000) 2024-03-04T14:41:22.835 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:41:22.836+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:41:22.836973+0000) 2024-03-04T14:41:23.835 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:41:23.836+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:41:23.837234+0000) 2024-03-04T14:41:24.835 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:41:24.836+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:41:24.837489+0000) 2024-03-04T14:41:25.836 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:41:25.836+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:41:25.837777+0000) 2024-03-04T14:41:26.836 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:41:26.837+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:41:26.838032+0000) 2024-03-04T14:41:27.836 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:41:27.837+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:41:27.838288+0000) 2024-03-04T14:41:28.837 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:41:28.837+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:41:28.838554+0000) 2024-03-04T14:41:29.837 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:41:29.838+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:41:29.838863+0000) 2024-03-04T14:41:30.837 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:41:30.838+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:41:30.839121+0000) 2024-03-04T14:41:31.838 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:41:31.838+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:41:31.839403+0000) 2024-03-04T14:41:32.838 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:41:32.838+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:41:32.839734+0000) 2024-03-04T14:41:33.838 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:41:33.839+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:41:33.839951+0000) 2024-03-04T14:41:34.838 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:41:34.839+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:41:34.840206+0000) 2024-03-04T14:41:35.445 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:41:35.453 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:41:35.838 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:41:35.839+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:41:35.840470+0000) 2024-03-04T14:41:36.839 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:41:36.840+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:41:36.840727+0000) 2024-03-04T14:41:37.839 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:41:37.840+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:41:37.840994+0000) 2024-03-04T14:41:38.839 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:41:38.840+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:41:38.841255+0000) 2024-03-04T14:41:39.840 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:41:39.840+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:41:39.841523+0000) 2024-03-04T14:41:40.840 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:41:40.841+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:41:40.841779+0000) 2024-03-04T14:41:41.840 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:41:41.841+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:41:41.842079+0000) 2024-03-04T14:41:42.840 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:41:42.841+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:41:42.842334+0000) 2024-03-04T14:41:43.840 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:41:43.841+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:41:43.842584+0000) 2024-03-04T14:41:44.841 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:41:44.842+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:41:44.842883+0000) 2024-03-04T14:41:45.841 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:41:45.842+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:41:45.843140+0000) 2024-03-04T14:41:46.841 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:41:46.842+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:41:46.843397+0000) 2024-03-04T14:41:47.842 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:41:47.843+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:41:47.843656+0000) 2024-03-04T14:41:48.842 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:41:48.843+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:41:48.843947+0000) 2024-03-04T14:41:49.842 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:41:49.843+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:41:49.844216+0000) 2024-03-04T14:41:50.842 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:41:50.843+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:41:50.844453+0000) 2024-03-04T14:41:51.843 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:41:51.844+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:41:51.844691+0000) 2024-03-04T14:41:52.843 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:41:52.844+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:41:52.845027+0000) 2024-03-04T14:41:53.843 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:41:53.844+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:41:53.845283+0000) 2024-03-04T14:41:54.844 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:41:54.845+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:41:54.845527+0000) 2024-03-04T14:41:55.844 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:41:55.845+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:41:55.845764+0000) 2024-03-04T14:41:56.844 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:41:56.845+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:41:56.846004+0000) 2024-03-04T14:41:57.844 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:41:57.845+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:41:57.846248+0000) 2024-03-04T14:41:58.845 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:41:58.846+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:41:58.846495+0000) 2024-03-04T14:41:59.845 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:41:59.846+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:41:59.846754+0000) 2024-03-04T14:42:00.845 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:42:00.846+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:42:00.846977+0000) 2024-03-04T14:42:01.845 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:42:01.846+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:42:01.847246+0000) 2024-03-04T14:42:02.845 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:42:02.847+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:42:02.847492+0000) 2024-03-04T14:42:03.846 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:42:03.847+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:42:03.847733+0000) 2024-03-04T14:42:04.846 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:42:04.847+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:42:04.847979+0000) 2024-03-04T14:42:05.665 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:42:05.676 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:42:05.846 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:42:05.847+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:42:05.848233+0000) 2024-03-04T14:42:06.846 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:42:06.848+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:42:06.848475+0000) 2024-03-04T14:42:07.847 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:42:07.848+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:42:07.848716+0000) 2024-03-04T14:42:08.847 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:42:08.848+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:42:08.848980+0000) 2024-03-04T14:42:09.847 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:42:09.849+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:42:09.849230+0000) 2024-03-04T14:42:10.847 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:42:10.849+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:42:10.849471+0000) 2024-03-04T14:42:11.848 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:42:11.849+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:42:11.849712+0000) 2024-03-04T14:42:12.848 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:42:12.849+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:42:12.849949+0000) 2024-03-04T14:42:13.848 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:42:13.850+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:42:13.850193+0000) 2024-03-04T14:42:14.848 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:42:14.849+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:42:14.850435+0000) 2024-03-04T14:42:15.849 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:42:15.849+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:42:15.850678+0000) 2024-03-04T14:42:16.850 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:42:16.849+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:42:16.850949+0000) 2024-03-04T14:42:17.849 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:42:17.850+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:42:17.851187+0000) 2024-03-04T14:42:18.849 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:42:18.850+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:42:18.851462+0000) 2024-03-04T14:42:19.850 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:42:19.850+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:42:19.851715+0000) 2024-03-04T14:42:20.851 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:42:20.850+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:42:20.852001+0000) 2024-03-04T14:42:21.851 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:42:21.851+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:42:21.852248+0000) 2024-03-04T14:42:22.851 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:42:22.851+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:42:22.852487+0000) 2024-03-04T14:42:23.851 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:42:23.851+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:42:23.852730+0000) 2024-03-04T14:42:24.851 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:42:24.852+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:42:24.853006+0000) 2024-03-04T14:42:25.851 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:42:25.852+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:42:25.853247+0000) 2024-03-04T14:42:26.851 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:42:26.851+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:42:26.853508+0000) 2024-03-04T14:42:27.852 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:42:27.852+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:42:27.853749+0000) 2024-03-04T14:42:28.852 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:42:28.853+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:42:28.853991+0000) 2024-03-04T14:42:29.852 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:42:29.853+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:42:29.854245+0000) 2024-03-04T14:42:30.852 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:42:30.853+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:42:30.854428+0000) 2024-03-04T14:42:31.853 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:42:31.853+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:42:31.854691+0000) 2024-03-04T14:42:32.853 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:42:32.854+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:42:32.854931+0000) 2024-03-04T14:42:33.853 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:42:33.854+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:42:33.855177+0000) 2024-03-04T14:42:34.853 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:42:34.854+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:42:34.855422+0000) 2024-03-04T14:42:35.853 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:42:35.854+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:42:35.855610+0000) 2024-03-04T14:42:35.887 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:42:35.894 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:42:36.854 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:42:36.855+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:42:36.855895+0000) 2024-03-04T14:42:37.854 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:42:37.855+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:42:37.856171+0000) 2024-03-04T14:42:38.854 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:42:38.855+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:42:38.856440+0000) 2024-03-04T14:42:39.855 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:42:39.855+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:42:39.856693+0000) 2024-03-04T14:42:40.855 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:42:40.856+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:42:40.856939+0000) 2024-03-04T14:42:41.855 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:42:41.856+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:42:41.857181+0000) 2024-03-04T14:42:42.855 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:42:42.856+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:42:42.857425+0000) 2024-03-04T14:42:43.856 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:42:43.856+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:42:43.857683+0000) 2024-03-04T14:42:44.856 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:42:44.857+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:42:44.857961+0000) 2024-03-04T14:42:45.856 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:42:45.857+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:42:45.858200+0000) 2024-03-04T14:42:46.856 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:42:46.857+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:42:46.858446+0000) 2024-03-04T14:42:47.857 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:42:47.858+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:42:47.858684+0000) 2024-03-04T14:42:48.857 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:42:48.858+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:42:48.858969+0000) 2024-03-04T14:42:49.857 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:42:49.858+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:42:49.859226+0000) 2024-03-04T14:42:50.857 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:42:50.858+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:42:50.859469+0000) 2024-03-04T14:42:51.858 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:42:51.859+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:42:51.859715+0000) 2024-03-04T14:42:52.858 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:42:52.859+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:42:52.859955+0000) 2024-03-04T14:42:53.858 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:42:53.859+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:42:53.860198+0000) 2024-03-04T14:42:54.858 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:42:54.859+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:42:54.860443+0000) 2024-03-04T14:42:55.858 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:42:55.859+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:42:55.860713+0000) 2024-03-04T14:42:56.859 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:42:56.860+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:42:56.860972+0000) 2024-03-04T14:42:57.859 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:42:57.860+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:42:57.861210+0000) 2024-03-04T14:42:58.859 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:42:58.860+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:42:58.861462+0000) 2024-03-04T14:42:59.859 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:42:59.861+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:42:59.861721+0000) 2024-03-04T14:43:00.860 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:43:00.861+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:43:00.861984+0000) 2024-03-04T14:43:01.860 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:43:01.861+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:43:01.862246+0000) 2024-03-04T14:43:02.860 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:43:02.862+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:43:02.862487+0000) 2024-03-04T14:43:03.860 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:43:03.862+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:43:03.862731+0000) 2024-03-04T14:43:04.861 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:43:04.862+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:43:04.862976+0000) 2024-03-04T14:43:05.861 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:43:05.862+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:43:05.863218+0000) 2024-03-04T14:43:06.042 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:43:06.051 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:43:06.861 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:43:06.863+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:43:06.863458+0000) 2024-03-04T14:43:07.862 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:43:07.863+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:43:07.863698+0000) 2024-03-04T14:43:08.862 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:43:08.863+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:43:08.863949+0000) 2024-03-04T14:43:09.862 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:43:09.863+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:43:09.864148+0000) 2024-03-04T14:43:10.862 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:43:10.864+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:43:10.864387+0000) 2024-03-04T14:43:11.862 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:43:11.863+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:43:11.864638+0000) 2024-03-04T14:43:12.863 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:43:12.864+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:43:12.864915+0000) 2024-03-04T14:43:13.863 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:43:13.864+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:43:13.865155+0000) 2024-03-04T14:43:14.863 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:43:14.865+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:43:14.865457+0000) 2024-03-04T14:43:15.864 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:43:15.865+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:43:15.865693+0000) 2024-03-04T14:43:16.864 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:43:16.865+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:43:16.865941+0000) 2024-03-04T14:43:17.864 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:43:17.865+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:43:17.866177+0000) 2024-03-04T14:43:18.865 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:43:18.866+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:43:18.866422+0000) 2024-03-04T14:43:19.865 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:43:19.866+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:43:19.866671+0000) 2024-03-04T14:43:20.865 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:43:20.866+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:43:20.866936+0000) 2024-03-04T14:43:21.865 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:43:21.867+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:43:21.867173+0000) 2024-03-04T14:43:22.865 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:43:22.867+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:43:22.867450+0000) 2024-03-04T14:43:23.866 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:43:23.866+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:43:23.867697+0000) 2024-03-04T14:43:24.866 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:43:24.866+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:43:24.867960+0000) 2024-03-04T14:43:25.866 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:43:25.867+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:43:25.868199+0000) 2024-03-04T14:43:26.866 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:43:26.867+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:43:26.868461+0000) 2024-03-04T14:43:27.867 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:43:27.867+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:43:27.868733+0000) 2024-03-04T14:43:28.867 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:43:28.867+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:43:28.868983+0000) 2024-03-04T14:43:29.867 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:43:29.868+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:43:29.869237+0000) 2024-03-04T14:43:30.867 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:43:30.867+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:43:30.869481+0000) 2024-03-04T14:43:31.868 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:43:31.868+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:43:31.869739+0000) 2024-03-04T14:43:32.868 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:43:32.869+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:43:32.869996+0000) 2024-03-04T14:43:33.868 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:43:33.869+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:43:33.870233+0000) 2024-03-04T14:43:34.868 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:43:34.869+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:43:34.870490+0000) 2024-03-04T14:43:35.869 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:43:35.869+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:43:35.870735+0000) 2024-03-04T14:43:36.173 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:43:36.181 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:43:36.869 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:43:36.870+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:43:36.870979+0000) 2024-03-04T14:43:37.869 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:43:37.870+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:43:37.871219+0000) 2024-03-04T14:43:38.869 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:43:38.870+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:43:38.871477+0000) 2024-03-04T14:43:39.869 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:43:39.870+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:43:39.871688+0000) 2024-03-04T14:43:40.870 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:43:40.871+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:43:40.871938+0000) 2024-03-04T14:43:41.870 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:43:41.871+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:43:41.872175+0000) 2024-03-04T14:43:42.870 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:43:42.871+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:43:42.872411+0000) 2024-03-04T14:43:43.870 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:43:43.871+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:43:43.872650+0000) 2024-03-04T14:43:44.871 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:43:44.872+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:43:44.872932+0000) 2024-03-04T14:43:45.871 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:43:45.872+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:43:45.873126+0000) 2024-03-04T14:43:46.871 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:43:46.872+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:43:46.873368+0000) 2024-03-04T14:43:47.871 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:43:47.872+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:43:47.873606+0000) 2024-03-04T14:43:48.872 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:43:48.873+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:43:48.873880+0000) 2024-03-04T14:43:49.872 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:43:49.873+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:43:49.874134+0000) 2024-03-04T14:43:50.872 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:43:50.873+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:43:50.874375+0000) 2024-03-04T14:43:51.872 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:43:51.873+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:43:51.874554+0000) 2024-03-04T14:43:52.872 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:43:52.874+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:43:52.874812+0000) 2024-03-04T14:43:53.873 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:43:53.874+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:43:53.875088+0000) 2024-03-04T14:43:54.873 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:43:54.874+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:43:54.875329+0000) 2024-03-04T14:43:55.873 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:43:55.874+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:43:55.875580+0000) 2024-03-04T14:43:56.874 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:43:56.875+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:43:56.875871+0000) 2024-03-04T14:43:57.875 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:43:57.875+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:43:57.876133+0000) 2024-03-04T14:43:58.874 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:43:58.875+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:43:58.876372+0000) 2024-03-04T14:43:59.874 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:43:59.876+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:43:59.876625+0000) 2024-03-04T14:44:00.875 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:44:00.876+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:44:00.876922+0000) 2024-03-04T14:44:01.875 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:44:01.876+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:44:01.877197+0000) 2024-03-04T14:44:02.875 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:44:02.876+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:44:02.877480+0000) 2024-03-04T14:44:03.875 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:44:03.877+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:44:03.877751+0000) 2024-03-04T14:44:04.876 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:44:04.877+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:44:04.878013+0000) 2024-03-04T14:44:05.876 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:44:05.877+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:44:05.878270+0000) 2024-03-04T14:44:06.402 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:44:06.410 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:44:06.876 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:44:06.877+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:44:06.878526+0000) 2024-03-04T14:44:07.877 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:44:07.878+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:44:07.878879+0000) 2024-03-04T14:44:08.877 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:44:08.878+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:44:08.879167+0000) 2024-03-04T14:44:09.877 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:44:09.878+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:44:09.879457+0000) 2024-03-04T14:44:10.877 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:44:10.879+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:44:10.879665+0000) 2024-03-04T14:44:11.878 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:44:11.879+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:44:11.879925+0000) 2024-03-04T14:44:12.878 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:44:12.879+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:44:12.880181+0000) 2024-03-04T14:44:13.878 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:44:13.880+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:44:13.880429+0000) 2024-03-04T14:44:14.878 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:44:14.879+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:44:14.880676+0000) 2024-03-04T14:44:15.879 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:44:15.880+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:44:15.880960+0000) 2024-03-04T14:44:16.879 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:44:16.880+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:44:16.881211+0000) 2024-03-04T14:44:17.879 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:44:17.881+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:44:17.881464+0000) 2024-03-04T14:44:18.879 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:44:18.881+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:44:18.881704+0000) 2024-03-04T14:44:19.880 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:44:19.881+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:44:19.881954+0000) 2024-03-04T14:44:20.880 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:44:20.881+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:44:20.882155+0000) 2024-03-04T14:44:21.880 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:44:21.882+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:44:21.882406+0000) 2024-03-04T14:44:22.880 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:44:22.882+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:44:22.882663+0000) 2024-03-04T14:44:23.881 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:44:23.882+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:44:23.882947+0000) 2024-03-04T14:44:24.881 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:44:24.882+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:44:24.883201+0000) 2024-03-04T14:44:25.881 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:44:25.882+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:44:25.883409+0000) 2024-03-04T14:44:26.881 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:44:26.882+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:44:26.883613+0000) 2024-03-04T14:44:27.882 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:44:27.883+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:44:27.883924+0000) 2024-03-04T14:44:28.882 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:44:28.884+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:44:28.884187+0000) 2024-03-04T14:44:29.882 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:44:29.884+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:44:29.884452+0000) 2024-03-04T14:44:30.883 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:44:30.883+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:44:30.884707+0000) 2024-03-04T14:44:31.883 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:44:31.883+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:44:31.884965+0000) 2024-03-04T14:44:32.883 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:44:32.884+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:44:32.885190+0000) 2024-03-04T14:44:33.883 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:44:33.884+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:44:33.885449+0000) 2024-03-04T14:44:34.883 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:44:34.884+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:44:34.885715+0000) 2024-03-04T14:44:35.883 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:44:35.884+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:44:35.885981+0000) 2024-03-04T14:44:36.606 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:44:36.615 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:44:36.884 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:44:36.885+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:44:36.886248+0000) 2024-03-04T14:44:37.884 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:44:37.885+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:44:37.886503+0000) 2024-03-04T14:44:38.884 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:44:38.885+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:44:38.886756+0000) 2024-03-04T14:44:39.884 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:44:39.886+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:44:39.887031+0000) 2024-03-04T14:44:40.885 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:44:40.886+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:44:40.887286+0000) 2024-03-04T14:44:41.885 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:44:41.886+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:44:41.887544+0000) 2024-03-04T14:44:42.885 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:44:42.887+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:44:42.887806+0000) 2024-03-04T14:44:43.886 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:44:43.887+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:44:43.888110+0000) 2024-03-04T14:44:44.886 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:44:44.887+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:44:44.888370+0000) 2024-03-04T14:44:45.886 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:44:45.887+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:44:45.888625+0000) 2024-03-04T14:44:46.887 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:44:46.888+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:44:46.888920+0000) 2024-03-04T14:44:47.887 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:44:47.888+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:44:47.889175+0000) 2024-03-04T14:44:48.887 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:44:48.888+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:44:48.889381+0000) 2024-03-04T14:44:49.887 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:44:49.888+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:44:49.889658+0000) 2024-03-04T14:44:50.888 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:44:50.889+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:44:50.889998+0000) 2024-03-04T14:44:51.889 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:44:51.889+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:44:51.890214+0000) 2024-03-04T14:44:52.888 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:44:52.889+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:44:52.890471+0000) 2024-03-04T14:44:53.889 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:44:53.889+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:44:53.890740+0000) 2024-03-04T14:44:54.889 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:44:54.890+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:44:54.890982+0000) 2024-03-04T14:44:55.889 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:44:55.890+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:44:55.891221+0000) 2024-03-04T14:44:56.889 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:44:56.890+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:44:56.891460+0000) 2024-03-04T14:44:57.889 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:44:57.891+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:44:57.891737+0000) 2024-03-04T14:44:58.889 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:44:58.891+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:44:58.891992+0000) 2024-03-04T14:44:59.890 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:44:59.891+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:44:59.892261+0000) 2024-03-04T14:45:00.890 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:45:00.891+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:45:00.892537+0000) 2024-03-04T14:45:01.890 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:45:01.892+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:45:01.892871+0000) 2024-03-04T14:45:02.891 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:45:02.892+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:45:02.893081+0000) 2024-03-04T14:45:03.891 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:45:03.892+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:45:03.893331+0000) 2024-03-04T14:45:04.891 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:45:04.892+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:45:04.893588+0000) 2024-03-04T14:45:05.891 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:45:05.893+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:45:05.893881+0000) 2024-03-04T14:45:06.844 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:45:06.853 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:45:06.891 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:45:06.893+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:45:06.894164+0000) 2024-03-04T14:45:07.892 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:45:07.893+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:45:07.894432+0000) 2024-03-04T14:45:08.892 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:45:08.894+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:45:08.894668+0000) 2024-03-04T14:45:09.892 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:45:09.894+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:45:09.894943+0000) 2024-03-04T14:45:10.893 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:45:10.894+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:45:10.895182+0000) 2024-03-04T14:45:11.893 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:45:11.894+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:45:11.895419+0000) 2024-03-04T14:45:12.893 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:45:12.895+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:45:12.895690+0000) 2024-03-04T14:45:13.893 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:45:13.895+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:45:13.895952+0000) 2024-03-04T14:45:14.894 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:45:14.895+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:45:14.896201+0000) 2024-03-04T14:45:15.894 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:45:15.895+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:45:15.896442+0000) 2024-03-04T14:45:16.894 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:45:16.896+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:45:16.896680+0000) 2024-03-04T14:45:17.894 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:45:17.896+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:45:17.896892+0000) 2024-03-04T14:45:18.895 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:45:18.896+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:45:18.897131+0000) 2024-03-04T14:45:19.895 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:45:19.896+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:45:19.897382+0000) 2024-03-04T14:45:20.895 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:45:20.897+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:45:20.897623+0000) 2024-03-04T14:45:21.895 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:45:21.897+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:45:21.897919+0000) 2024-03-04T14:45:22.896 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:45:22.897+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:45:22.898162+0000) 2024-03-04T14:45:23.896 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:45:23.898+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:45:23.898394+0000) 2024-03-04T14:45:24.896 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:45:24.898+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:45:24.898632+0000) 2024-03-04T14:45:25.896 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:45:25.898+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:45:25.898923+0000) 2024-03-04T14:45:26.897 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:45:26.898+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:45:26.899162+0000) 2024-03-04T14:45:27.897 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:45:27.899+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:45:27.899399+0000) 2024-03-04T14:45:28.897 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:45:28.899+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:45:28.899639+0000) 2024-03-04T14:45:29.897 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:45:29.899+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:45:29.899923+0000) 2024-03-04T14:45:30.898 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:45:30.899+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:45:30.900186+0000) 2024-03-04T14:45:31.898 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:45:31.900+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:45:31.900422+0000) 2024-03-04T14:45:32.898 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:45:32.900+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:45:32.900668+0000) 2024-03-04T14:45:33.898 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:45:33.900+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:45:33.900942+0000) 2024-03-04T14:45:34.898 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:45:34.901+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:45:34.901195+0000) 2024-03-04T14:45:35.899 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:45:35.901+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:45:35.901447+0000) 2024-03-04T14:45:36.899 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:45:36.901+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:45:36.901679+0000) 2024-03-04T14:45:37.000 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:45:37.008 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:45:37.899 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:45:37.900+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:45:37.901947+0000) 2024-03-04T14:45:38.900 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:45:38.901+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:45:38.902270+0000) 2024-03-04T14:45:39.900 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:45:39.901+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:45:39.902517+0000) 2024-03-04T14:45:40.900 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:45:40.901+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:45:40.902761+0000) 2024-03-04T14:45:41.900 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:45:41.901+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:45:41.902944+0000) 2024-03-04T14:45:42.901 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:45:42.902+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:45:42.903183+0000) 2024-03-04T14:45:43.901 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:45:43.901+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:45:43.903366+0000) 2024-03-04T14:45:44.901 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:45:44.902+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:45:44.903605+0000) 2024-03-04T14:45:45.901 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:45:45.901+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:45:45.903801+0000) 2024-03-04T14:45:46.901 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:45:46.903+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:45:46.904013+0000) 2024-03-04T14:45:47.902 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:45:47.903+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:45:47.904247+0000) 2024-03-04T14:45:48.902 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:45:48.903+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:45:48.904484+0000) 2024-03-04T14:45:49.902 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:45:49.903+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:45:49.904729+0000) 2024-03-04T14:45:50.902 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:45:50.904+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:45:50.904967+0000) 2024-03-04T14:45:51.903 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:45:51.904+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:45:51.905202+0000) 2024-03-04T14:45:52.903 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:45:52.904+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:45:52.905438+0000) 2024-03-04T14:45:53.903 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:45:53.904+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:45:53.905678+0000) 2024-03-04T14:45:54.904 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:45:54.905+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:45:54.905952+0000) 2024-03-04T14:45:55.904 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:45:55.905+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:45:55.906196+0000) 2024-03-04T14:45:56.904 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:45:56.905+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:45:56.906460+0000) 2024-03-04T14:45:57.904 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:45:57.905+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:45:57.906697+0000) 2024-03-04T14:45:58.904 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:45:58.906+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:45:58.906944+0000) 2024-03-04T14:45:59.905 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:45:59.906+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:45:59.907192+0000) 2024-03-04T14:46:00.905 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:46:00.906+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:46:00.907388+0000) 2024-03-04T14:46:01.905 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:46:01.906+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:46:01.907645+0000) 2024-03-04T14:46:02.905 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:46:02.907+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:46:02.907938+0000) 2024-03-04T14:46:03.906 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:46:03.907+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:46:03.908181+0000) 2024-03-04T14:46:04.906 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:46:04.907+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:46:04.908368+0000) 2024-03-04T14:46:05.906 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:46:05.907+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:46:05.908603+0000) 2024-03-04T14:46:06.906 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:46:06.908+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:46:06.908864+0000) 2024-03-04T14:46:07.154 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:46:07.162 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:46:07.907 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:46:07.908+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:46:07.909137+0000) 2024-03-04T14:46:08.907 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:46:08.908+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:46:08.909379+0000) 2024-03-04T14:46:09.907 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:46:09.908+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:46:09.909634+0000) 2024-03-04T14:46:10.907 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:46:10.909+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:46:10.909913+0000) 2024-03-04T14:46:11.908 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:46:11.909+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:46:11.910149+0000) 2024-03-04T14:46:12.908 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:46:12.909+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:46:12.910395+0000) 2024-03-04T14:46:13.908 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:46:13.910+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:46:13.910643+0000) 2024-03-04T14:46:14.908 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:46:14.910+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:46:14.910915+0000) 2024-03-04T14:46:15.908 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:46:15.910+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:46:15.911109+0000) 2024-03-04T14:46:16.909 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:46:16.910+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:46:16.911348+0000) 2024-03-04T14:46:17.909 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:46:17.911+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:46:17.911618+0000) 2024-03-04T14:46:18.909 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:46:18.911+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:46:18.911894+0000) 2024-03-04T14:46:19.910 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:46:19.911+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:46:19.912144+0000) 2024-03-04T14:46:20.910 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:46:20.911+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:46:20.912386+0000) 2024-03-04T14:46:21.910 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:46:21.912+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:46:21.912625+0000) 2024-03-04T14:46:22.910 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:46:22.912+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:46:22.912914+0000) 2024-03-04T14:46:23.911 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:46:23.912+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:46:23.913153+0000) 2024-03-04T14:46:24.911 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:46:24.912+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:46:24.913384+0000) 2024-03-04T14:46:25.911 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:46:25.913+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:46:25.913624+0000) 2024-03-04T14:46:26.911 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:46:26.913+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:46:26.913909+0000) 2024-03-04T14:46:27.911 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:46:27.913+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:46:27.914144+0000) 2024-03-04T14:46:28.912 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:46:28.914+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:46:28.914385+0000) 2024-03-04T14:46:29.912 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:46:29.914+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:46:29.914636+0000) 2024-03-04T14:46:30.912 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:46:30.914+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:46:30.914915+0000) 2024-03-04T14:46:31.912 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:46:31.914+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:46:31.915140+0000) 2024-03-04T14:46:32.913 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:46:32.915+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:46:32.915401+0000) 2024-03-04T14:46:33.913 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:46:33.915+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:46:33.915667+0000) 2024-03-04T14:46:34.913 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:46:34.915+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:46:34.915930+0000) 2024-03-04T14:46:35.914 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:46:35.915+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:46:35.916115+0000) 2024-03-04T14:46:36.914 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:46:36.916+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:46:36.916361+0000) 2024-03-04T14:46:37.294 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:46:37.302 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:46:37.914 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:46:37.916+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:46:37.916608+0000) 2024-03-04T14:46:38.914 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:46:38.916+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:46:38.916890+0000) 2024-03-04T14:46:39.915 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:46:39.916+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:46:39.917150+0000) 2024-03-04T14:46:40.915 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:46:40.917+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:46:40.917397+0000) 2024-03-04T14:46:41.915 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:46:41.917+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:46:41.917636+0000) 2024-03-04T14:46:42.915 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:46:42.917+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:46:42.917917+0000) 2024-03-04T14:46:43.915 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:46:43.918+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:46:43.918160+0000) 2024-03-04T14:46:44.916 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:46:44.918+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:46:44.918400+0000) 2024-03-04T14:46:45.916 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:46:45.917+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:46:45.918642+0000) 2024-03-04T14:46:46.916 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:46:46.917+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:46:46.918929+0000) 2024-03-04T14:46:47.917 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:46:47.917+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:46:47.919160+0000) 2024-03-04T14:46:48.917 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:46:48.918+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:46:48.919392+0000) 2024-03-04T14:46:49.917 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:46:49.918+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:46:49.919641+0000) 2024-03-04T14:46:50.917 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:46:50.918+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:46:50.919927+0000) 2024-03-04T14:46:51.917 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:46:51.919+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:46:51.920189+0000) 2024-03-04T14:46:52.918 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:46:52.919+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:46:52.920431+0000) 2024-03-04T14:46:53.918 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:46:53.919+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:46:53.920674+0000) 2024-03-04T14:46:54.918 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:46:54.919+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:46:54.920942+0000) 2024-03-04T14:46:55.919 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:46:55.920+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:46:55.921181+0000) 2024-03-04T14:46:56.919 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:46:56.920+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:46:56.921420+0000) 2024-03-04T14:46:57.919 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:46:57.920+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:46:57.921655+0000) 2024-03-04T14:46:58.919 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:46:58.921+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:46:58.921950+0000) 2024-03-04T14:46:59.920 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:46:59.921+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:46:59.922199+0000) 2024-03-04T14:47:00.920 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:47:00.921+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:47:00.922442+0000) 2024-03-04T14:47:01.920 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:47:01.921+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:47:01.922713+0000) 2024-03-04T14:47:02.921 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:47:02.922+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:47:02.923000+0000) 2024-03-04T14:47:03.921 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:47:03.922+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:47:03.923251+0000) 2024-03-04T14:47:04.921 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:47:04.922+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:47:04.923504+0000) 2024-03-04T14:47:05.921 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:47:05.922+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:47:05.923758+0000) 2024-03-04T14:47:06.921 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:47:06.923+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:47:06.924029+0000) 2024-03-04T14:47:07.507 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:47:07.515 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:47:07.921 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:47:07.923+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:47:07.924286+0000) 2024-03-04T14:47:08.922 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:47:08.923+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:47:08.924554+0000) 2024-03-04T14:47:09.922 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:47:09.924+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:47:09.924850+0000) 2024-03-04T14:47:10.923 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:47:10.924+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:47:10.925113+0000) 2024-03-04T14:47:11.923 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:47:11.924+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:47:11.925354+0000) 2024-03-04T14:47:12.923 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:47:12.924+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:47:12.925608+0000) 2024-03-04T14:47:13.924 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:47:13.925+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:47:13.925882+0000) 2024-03-04T14:47:14.924 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:47:14.925+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:47:14.926168+0000) 2024-03-04T14:47:15.924 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:47:15.925+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:47:15.926448+0000) 2024-03-04T14:47:16.925 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:47:16.925+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:47:16.926744+0000) 2024-03-04T14:47:17.925 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:47:17.926+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:47:17.927035+0000) 2024-03-04T14:47:18.925 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:47:18.926+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:47:18.927309+0000) 2024-03-04T14:47:19.925 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:47:19.926+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:47:19.927614+0000) 2024-03-04T14:47:20.925 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:47:20.927+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:47:20.927950+0000) 2024-03-04T14:47:21.926 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:47:21.927+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:47:21.928193+0000) 2024-03-04T14:47:22.926 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:47:22.927+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:47:22.928435+0000) 2024-03-04T14:47:23.926 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:47:23.928+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:47:23.928697+0000) 2024-03-04T14:47:24.927 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:47:24.928+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:47:24.929007+0000) 2024-03-04T14:47:25.927 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:47:25.928+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:47:25.929252+0000) 2024-03-04T14:47:26.927 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:47:26.928+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:47:26.929488+0000) 2024-03-04T14:47:27.927 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:47:27.929+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:47:27.929728+0000) 2024-03-04T14:47:28.928 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:47:28.929+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:47:28.929973+0000) 2024-03-04T14:47:29.928 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:47:29.929+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:47:29.930228+0000) 2024-03-04T14:47:30.928 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:47:30.930+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:47:30.930496+0000) 2024-03-04T14:47:31.928 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:47:31.930+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:47:31.930754+0000) 2024-03-04T14:47:32.929 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:47:32.930+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:47:32.931046+0000) 2024-03-04T14:47:33.929 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:47:33.930+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:47:33.931285+0000) 2024-03-04T14:47:34.929 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:47:34.931+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:47:34.931559+0000) 2024-03-04T14:47:35.930 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:47:35.931+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:47:35.931808+0000) 2024-03-04T14:47:36.930 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:47:36.931+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:47:36.932086+0000) 2024-03-04T14:47:37.721 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:47:37.730 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:47:37.930 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:47:37.931+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:47:37.932352+0000) 2024-03-04T14:47:38.931 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:47:38.932+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:47:38.932594+0000) 2024-03-04T14:47:39.931 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:47:39.932+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:47:39.932884+0000) 2024-03-04T14:47:40.931 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:47:40.932+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:47:40.933164+0000) 2024-03-04T14:47:41.931 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:47:41.933+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:47:41.933405+0000) 2024-03-04T14:47:42.931 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:47:42.933+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:47:42.933647+0000) 2024-03-04T14:47:43.931 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:47:43.933+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:47:43.933938+0000) 2024-03-04T14:47:44.932 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:47:44.933+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:47:44.934185+0000) 2024-03-04T14:47:45.932 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:47:45.933+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:47:45.934375+0000) 2024-03-04T14:47:46.933 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:47:46.933+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:47:46.934615+0000) 2024-03-04T14:47:47.933 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:47:47.933+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:47:47.934796+0000) 2024-03-04T14:47:48.933 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:47:48.934+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:47:48.935054+0000) 2024-03-04T14:47:49.933 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:47:49.934+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:47:49.935292+0000) 2024-03-04T14:47:50.933 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:47:50.935+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:47:50.935556+0000) 2024-03-04T14:47:51.933 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:47:51.935+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:47:51.935813+0000) 2024-03-04T14:47:52.934 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:47:52.935+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:47:52.936119+0000) 2024-03-04T14:47:53.934 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:47:53.936+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:47:53.936378+0000) 2024-03-04T14:47:54.935 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:47:54.935+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:47:54.936643+0000) 2024-03-04T14:47:55.935 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:47:55.935+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:47:55.936941+0000) 2024-03-04T14:47:56.935 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:47:56.936+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:47:56.937203+0000) 2024-03-04T14:47:57.935 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:47:57.936+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:47:57.937459+0000) 2024-03-04T14:47:58.936 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:47:58.936+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:47:58.937714+0000) 2024-03-04T14:47:59.936 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:47:59.936+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:47:59.937997+0000) 2024-03-04T14:48:00.936 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:48:00.937+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:48:00.938282+0000) 2024-03-04T14:48:01.936 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:48:01.937+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:48:01.938538+0000) 2024-03-04T14:48:02.937 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:48:02.937+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:48:02.938810+0000) 2024-03-04T14:48:03.937 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:48:03.938+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:48:03.939119+0000) 2024-03-04T14:48:04.937 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:48:04.938+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:48:04.939372+0000) 2024-03-04T14:48:05.938 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:48:05.937+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:48:05.939636+0000) 2024-03-04T14:48:06.938 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:48:06.939+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:48:06.939919+0000) 2024-03-04T14:48:07.938 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:48:07.939+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:48:07.940191+0000) 2024-03-04T14:48:07.943 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:48:07.951 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:48:08.939 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:48:08.939+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:48:08.940442+0000) 2024-03-04T14:48:09.939 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:48:09.939+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:48:09.940737+0000) 2024-03-04T14:48:10.939 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:48:10.940+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:48:10.941009+0000) 2024-03-04T14:48:11.939 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:48:11.940+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:48:11.941214+0000) 2024-03-04T14:48:12.939 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:48:12.940+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:48:12.941412+0000) 2024-03-04T14:48:13.939 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:48:13.940+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:48:13.941665+0000) 2024-03-04T14:48:14.940 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:48:14.941+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:48:14.941940+0000) 2024-03-04T14:48:15.940 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:48:15.941+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:48:15.942144+0000) 2024-03-04T14:48:16.940 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:48:16.941+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:48:16.942409+0000) 2024-03-04T14:48:17.941 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:48:17.941+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:48:17.942610+0000) 2024-03-04T14:48:18.941 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:48:18.942+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:48:18.943156+0000) 2024-03-04T14:48:19.942 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:48:19.942+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:48:19.943395+0000) 2024-03-04T14:48:20.944 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:48:20.942+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:48:20.943652+0000) 2024-03-04T14:48:21.942 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:48:21.943+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:48:21.943918+0000) 2024-03-04T14:48:22.942 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:48:22.943+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:48:22.944155+0000) 2024-03-04T14:48:23.942 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:48:23.943+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:48:23.944390+0000) 2024-03-04T14:48:24.942 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:48:24.943+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:48:24.944626+0000) 2024-03-04T14:48:25.943 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:48:25.944+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:48:25.944910+0000) 2024-03-04T14:48:26.943 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:48:26.944+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:48:26.945155+0000) 2024-03-04T14:48:27.944 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:48:27.944+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:48:27.945388+0000) 2024-03-04T14:48:28.943 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:48:28.945+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:48:28.945644+0000) 2024-03-04T14:48:29.944 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:48:29.945+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:48:29.945937+0000) 2024-03-04T14:48:30.944 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:48:30.945+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:48:30.946166+0000) 2024-03-04T14:48:31.944 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:48:31.945+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:48:31.946403+0000) 2024-03-04T14:48:32.945 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:48:32.946+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:48:32.946639+0000) 2024-03-04T14:48:33.945 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:48:33.946+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:48:33.946917+0000) 2024-03-04T14:48:34.945 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:48:34.946+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:48:34.947157+0000) 2024-03-04T14:48:35.945 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:48:35.946+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:48:35.947409+0000) 2024-03-04T14:48:36.946 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:48:36.947+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:48:36.947645+0000) 2024-03-04T14:48:37.946 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:48:37.947+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:48:37.947864+0000) 2024-03-04T14:48:38.165 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:48:38.173 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:48:38.946 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:48:38.947+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:48:38.948104+0000) 2024-03-04T14:48:39.946 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:48:39.947+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:48:39.948369+0000) 2024-03-04T14:48:40.947 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:48:40.948+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:48:40.948609+0000) 2024-03-04T14:48:41.947 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:48:41.948+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:48:41.948870+0000) 2024-03-04T14:48:42.947 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:48:42.948+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:48:42.949109+0000) 2024-03-04T14:48:43.947 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:48:43.948+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:48:43.949345+0000) 2024-03-04T14:48:44.948 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:48:44.949+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:48:44.949590+0000) 2024-03-04T14:48:45.948 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:48:45.949+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:48:45.949860+0000) 2024-03-04T14:48:46.948 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:48:46.949+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:48:46.950101+0000) 2024-03-04T14:48:47.948 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:48:47.949+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:48:47.950335+0000) 2024-03-04T14:48:48.949 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:48:48.950+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:48:48.950571+0000) 2024-03-04T14:48:49.949 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:48:49.950+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:48:49.950818+0000) 2024-03-04T14:48:50.949 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:48:50.950+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:48:50.951112+0000) 2024-03-04T14:48:51.949 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:48:51.951+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:48:51.951366+0000) 2024-03-04T14:48:52.950 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:48:52.950+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:48:52.951548+0000) 2024-03-04T14:48:53.950 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:48:53.950+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:48:53.951799+0000) 2024-03-04T14:48:54.950 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:48:54.951+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:48:54.952095+0000) 2024-03-04T14:48:55.951 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:48:55.952+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:48:55.952345+0000) 2024-03-04T14:48:56.951 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:48:56.952+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:48:56.952593+0000) 2024-03-04T14:48:57.951 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:48:57.952+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:48:57.952890+0000) 2024-03-04T14:48:58.951 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:48:58.952+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:48:58.953158+0000) 2024-03-04T14:48:59.951 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:48:59.953+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:48:59.953444+0000) 2024-03-04T14:49:00.952 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:49:00.953+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:49:00.953757+0000) 2024-03-04T14:49:01.952 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:49:01.953+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:49:01.954020+0000) 2024-03-04T14:49:02.952 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:49:02.954+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:49:02.954275+0000) 2024-03-04T14:49:03.953 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:49:03.953+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:49:03.954523+0000) 2024-03-04T14:49:04.953 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:49:04.953+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:49:04.954775+0000) 2024-03-04T14:49:05.953 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:49:05.953+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:49:05.955024+0000) 2024-03-04T14:49:06.954 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:49:06.953+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:49:06.955235+0000) 2024-03-04T14:49:07.956 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:49:07.954+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:49:07.955490+0000) 2024-03-04T14:49:08.298 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:49:08.309 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:49:08.954 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:49:08.954+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:49:08.955742+0000) 2024-03-04T14:49:09.954 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:49:09.954+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:49:09.956010+0000) 2024-03-04T14:49:10.954 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:49:10.955+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:49:10.956272+0000) 2024-03-04T14:49:11.955 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:49:11.955+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:49:11.956526+0000) 2024-03-04T14:49:12.955 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:49:12.955+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:49:12.956782+0000) 2024-03-04T14:49:13.955 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:49:13.955+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:49:13.957028+0000) 2024-03-04T14:49:14.956 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:49:14.956+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:49:14.957285+0000) 2024-03-04T14:49:15.958 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:49:15.956+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:49:15.957518+0000) 2024-03-04T14:49:16.956 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:49:16.956+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:49:16.957771+0000) 2024-03-04T14:49:17.956 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:49:17.957+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:49:17.958003+0000) 2024-03-04T14:49:18.957 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:49:18.957+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:49:18.958238+0000) 2024-03-04T14:49:19.957 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:49:19.957+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:49:19.958482+0000) 2024-03-04T14:49:20.957 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:49:20.957+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:49:20.958698+0000) 2024-03-04T14:49:21.957 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:49:21.958+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:49:21.958904+0000) 2024-03-04T14:49:22.957 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:49:22.958+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:49:22.959143+0000) 2024-03-04T14:49:23.957 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:49:23.958+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:49:23.959369+0000) 2024-03-04T14:49:24.958 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:49:24.958+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:49:24.959560+0000) 2024-03-04T14:49:25.958 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:49:25.958+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:49:25.959768+0000) 2024-03-04T14:49:26.958 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:49:26.959+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:49:26.959941+0000) 2024-03-04T14:49:27.958 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:49:27.959+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:49:27.960118+0000) 2024-03-04T14:49:28.959 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:49:28.959+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:49:28.960359+0000) 2024-03-04T14:49:29.959 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:49:29.959+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:49:29.960608+0000) 2024-03-04T14:49:30.960 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:49:30.960+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:49:30.960887+0000) 2024-03-04T14:49:31.961 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:49:31.960+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:49:31.961148+0000) 2024-03-04T14:49:32.960 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:49:32.960+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:49:32.961386+0000) 2024-03-04T14:49:33.960 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:49:33.960+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:49:33.961619+0000) 2024-03-04T14:49:34.960 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:49:34.961+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:49:34.961893+0000) 2024-03-04T14:49:35.961 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:49:35.961+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:49:35.962163+0000) 2024-03-04T14:49:36.964 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:49:36.961+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:49:36.962421+0000) 2024-03-04T14:49:37.961 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:49:37.961+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:49:37.962670+0000) 2024-03-04T14:49:38.428 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:49:38.438 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:49:38.961 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:49:38.962+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:49:38.962952+0000) 2024-03-04T14:49:39.961 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:49:39.962+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:49:39.963150+0000) 2024-03-04T14:49:40.964 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:49:40.962+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:49:40.963343+0000) 2024-03-04T14:49:41.962 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:49:41.962+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:49:41.963522+0000) 2024-03-04T14:49:42.963 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:49:42.962+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:49:42.963705+0000) 2024-03-04T14:49:43.963 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:49:43.963+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:49:43.963965+0000) 2024-03-04T14:49:44.963 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:49:44.963+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:49:44.964205+0000) 2024-03-04T14:49:45.963 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:49:45.963+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:49:45.964442+0000) 2024-03-04T14:49:46.963 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:49:46.964+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:49:46.964683+0000) 2024-03-04T14:49:47.963 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:49:47.964+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:49:47.964924+0000) 2024-03-04T14:49:48.964 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:49:48.964+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:49:48.965231+0000) 2024-03-04T14:49:49.964 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:49:49.965+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:49:49.965484+0000) 2024-03-04T14:49:50.964 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:49:50.965+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:49:50.965730+0000) 2024-03-04T14:49:51.964 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:49:51.965+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:49:51.965966+0000) 2024-03-04T14:49:52.965 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:49:52.965+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:49:52.966207+0000) 2024-03-04T14:49:53.965 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:49:53.966+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:49:53.966445+0000) 2024-03-04T14:49:54.965 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:49:54.966+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:49:54.966691+0000) 2024-03-04T14:49:55.965 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:49:55.966+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:49:55.966929+0000) 2024-03-04T14:49:56.968 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:49:56.966+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:49:56.967167+0000) 2024-03-04T14:49:57.966 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:49:57.966+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:49:57.967357+0000) 2024-03-04T14:49:58.966 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:49:58.967+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:49:58.967599+0000) 2024-03-04T14:49:59.966 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:49:59.967+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:49:59.967881+0000) 2024-03-04T14:50:00.967 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:50:00.967+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:50:00.968122+0000) 2024-03-04T14:50:01.967 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:50:01.968+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:50:01.968377+0000) 2024-03-04T14:50:02.967 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:50:02.968+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:50:02.968635+0000) 2024-03-04T14:50:03.968 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:50:03.968+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:50:03.968922+0000) 2024-03-04T14:50:04.968 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:50:04.968+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:50:04.969198+0000) 2024-03-04T14:50:05.968 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:50:05.969+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:50:05.969442+0000) 2024-03-04T14:50:06.968 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:50:06.969+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:50:06.969681+0000) 2024-03-04T14:50:07.969 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:50:07.969+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:50:07.969992+0000) 2024-03-04T14:50:08.643 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:50:08.652 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:50:08.969 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:50:08.970+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:50:08.970282+0000) 2024-03-04T14:50:09.969 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:50:09.969+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:50:09.970580+0000) 2024-03-04T14:50:10.970 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:50:10.970+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:50:10.970916+0000) 2024-03-04T14:50:11.970 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:50:11.970+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:50:11.971207+0000) 2024-03-04T14:50:12.970 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:50:12.970+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:50:12.971490+0000) 2024-03-04T14:50:13.970 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:50:13.970+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:50:13.971774+0000) 2024-03-04T14:50:14.971 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:50:14.970+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:50:14.972062+0000) 2024-03-04T14:50:15.971 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:50:15.971+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:50:15.972357+0000) 2024-03-04T14:50:16.971 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:50:16.971+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:50:16.972637+0000) 2024-03-04T14:50:17.971 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:50:17.971+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:50:17.972971+0000) 2024-03-04T14:50:18.972 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:50:18.972+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:50:18.973255+0000) 2024-03-04T14:50:19.972 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:50:19.972+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:50:19.973555+0000) 2024-03-04T14:50:20.973 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:50:20.972+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:50:20.973879+0000) 2024-03-04T14:50:21.973 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:50:21.973+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:50:21.974178+0000) 2024-03-04T14:50:22.973 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:50:22.973+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:50:22.974457+0000) 2024-03-04T14:50:23.973 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:50:23.973+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:50:23.974704+0000) 2024-03-04T14:50:24.974 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:50:24.974+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:50:24.974945+0000) 2024-03-04T14:50:25.974 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:50:25.974+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:50:25.975192+0000) 2024-03-04T14:50:26.974 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:50:26.974+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:50:26.975435+0000) 2024-03-04T14:50:27.975 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:50:27.974+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:50:27.975675+0000) 2024-03-04T14:50:28.974 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:50:28.975+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:50:28.975928+0000) 2024-03-04T14:50:29.975 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:50:29.975+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:50:29.976189+0000) 2024-03-04T14:50:30.976 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:50:30.975+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:50:30.976425+0000) 2024-03-04T14:50:31.975 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:50:31.975+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:50:31.976664+0000) 2024-03-04T14:50:32.976 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:50:32.976+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:50:32.976933+0000) 2024-03-04T14:50:33.976 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:50:33.976+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:50:33.977119+0000) 2024-03-04T14:50:34.976 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:50:34.976+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:50:34.977378+0000) 2024-03-04T14:50:35.976 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:50:35.976+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:50:35.977603+0000) 2024-03-04T14:50:36.976 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:50:36.977+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:50:36.977881+0000) 2024-03-04T14:50:37.976 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:50:37.977+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:50:37.978118+0000) 2024-03-04T14:50:38.821 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:50:38.830 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:50:38.978 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:50:38.977+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:50:38.978368+0000) 2024-03-04T14:50:39.979 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:50:39.977+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:50:39.978612+0000) 2024-03-04T14:50:40.978 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:50:40.978+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:50:40.978913+0000) 2024-03-04T14:50:41.978 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:50:41.978+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:50:41.979152+0000) 2024-03-04T14:50:42.978 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:50:42.978+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:50:42.979395+0000) 2024-03-04T14:50:43.978 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:50:43.978+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:50:43.979629+0000) 2024-03-04T14:50:44.979 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:50:44.979+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:50:44.979923+0000) 2024-03-04T14:50:45.979 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:50:45.979+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:50:45.980164+0000) 2024-03-04T14:50:46.979 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:50:46.979+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:50:46.980403+0000) 2024-03-04T14:50:47.979 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:50:47.979+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:50:47.980689+0000) 2024-03-04T14:50:48.981 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:50:48.980+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:50:48.980975+0000) 2024-03-04T14:50:49.980 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:50:49.980+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:50:49.981220+0000) 2024-03-04T14:50:50.980 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:50:50.980+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:50:50.981477+0000) 2024-03-04T14:50:51.980 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:50:51.981+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:50:51.981710+0000) 2024-03-04T14:50:52.981 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:50:52.981+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:50:52.981952+0000) 2024-03-04T14:50:53.981 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:50:53.981+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:50:53.982133+0000) 2024-03-04T14:50:54.983 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:50:54.981+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:50:54.982372+0000) 2024-03-04T14:50:55.981 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:50:55.982+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:50:55.982604+0000) 2024-03-04T14:50:56.982 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:50:56.982+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:50:56.982878+0000) 2024-03-04T14:50:57.982 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:50:57.982+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:50:57.983065+0000) 2024-03-04T14:50:58.982 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:50:58.982+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:50:58.983258+0000) 2024-03-04T14:50:59.982 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:50:59.982+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:50:59.983447+0000) 2024-03-04T14:51:00.982 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:51:00.982+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:51:00.983629+0000) 2024-03-04T14:51:01.983 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:51:01.983+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:51:01.983978+0000) 2024-03-04T14:51:02.983 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:51:02.983+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:51:02.984251+0000) 2024-03-04T14:51:03.983 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:51:03.983+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:51:03.984559+0000) 2024-03-04T14:51:04.984 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:51:04.984+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:51:04.984903+0000) 2024-03-04T14:51:05.984 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:51:05.984+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:51:05.985148+0000) 2024-03-04T14:51:06.984 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:51:06.985+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:51:06.985429+0000) 2024-03-04T14:51:07.984 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:51:07.985+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:51:07.985724+0000) 2024-03-04T14:51:08.985 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:51:08.985+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:51:08.986006+0000) 2024-03-04T14:51:09.035 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:51:09.043 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:51:09.985 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:51:09.986+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:51:09.986294+0000) 2024-03-04T14:51:10.985 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:51:10.986+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:51:10.986577+0000) 2024-03-04T14:51:11.986 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:51:11.986+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:51:11.986893+0000) 2024-03-04T14:51:12.986 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:51:12.986+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:51:12.987188+0000) 2024-03-04T14:51:13.986 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:51:13.987+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:51:13.987468+0000) 2024-03-04T14:51:14.986 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:51:14.987+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:51:14.987752+0000) 2024-03-04T14:51:15.987 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:51:15.987+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:51:15.988042+0000) 2024-03-04T14:51:16.987 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:51:16.987+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:51:16.988323+0000) 2024-03-04T14:51:17.987 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:51:17.988+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:51:17.988630+0000) 2024-03-04T14:51:18.988 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:51:18.987+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:51:18.988968+0000) 2024-03-04T14:51:19.988 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:51:19.988+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:51:19.989301+0000) 2024-03-04T14:51:20.988 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:51:20.988+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:51:20.989594+0000) 2024-03-04T14:51:21.988 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:51:21.988+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:51:21.989909+0000) 2024-03-04T14:51:22.989 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:51:22.989+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:51:22.990190+0000) 2024-03-04T14:51:23.989 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:51:23.988+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:51:23.990482+0000) 2024-03-04T14:51:24.989 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:51:24.988+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:51:24.990763+0000) 2024-03-04T14:51:25.990 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:51:25.989+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:51:25.991055+0000) 2024-03-04T14:51:26.990 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:51:26.990+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:51:26.991365+0000) 2024-03-04T14:51:27.990 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:51:27.990+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:51:27.991667+0000) 2024-03-04T14:51:28.991 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:51:28.990+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:51:28.991987+0000) 2024-03-04T14:51:29.991 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:51:29.991+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:51:29.992290+0000) 2024-03-04T14:51:30.991 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:51:30.991+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:51:30.992593+0000) 2024-03-04T14:51:31.992 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:51:31.992+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:51:31.992917+0000) 2024-03-04T14:51:32.992 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:51:32.992+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:51:32.993167+0000) 2024-03-04T14:51:33.992 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:51:33.992+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:51:33.993477+0000) 2024-03-04T14:51:34.993 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:51:34.992+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:51:34.993757+0000) 2024-03-04T14:51:35.993 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:51:35.993+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:51:35.994091+0000) 2024-03-04T14:51:36.993 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:51:36.993+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:51:36.994373+0000) 2024-03-04T14:51:37.994 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:51:37.993+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:51:37.994657+0000) 2024-03-04T14:51:38.994 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:51:38.994+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:51:38.995002+0000) 2024-03-04T14:51:39.181 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:51:39.189 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:51:39.994 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:51:39.994+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:51:39.995307+0000) 2024-03-04T14:51:40.994 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:51:40.994+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:51:40.995600+0000) 2024-03-04T14:51:41.995 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:51:41.995+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:51:41.995932+0000) 2024-03-04T14:51:42.995 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:51:42.995+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:51:42.996176+0000) 2024-03-04T14:51:43.995 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:51:43.995+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:51:43.996472+0000) 2024-03-04T14:51:44.996 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:51:44.995+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:51:44.996752+0000) 2024-03-04T14:51:45.996 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:51:45.996+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:51:45.997014+0000) 2024-03-04T14:51:46.996 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:51:46.996+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:51:46.997290+0000) 2024-03-04T14:51:47.996 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:51:47.996+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:51:47.997586+0000) 2024-03-04T14:51:48.997 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:51:48.997+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:51:48.997942+0000) 2024-03-04T14:51:49.997 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:51:49.997+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:51:49.998263+0000) 2024-03-04T14:51:50.997 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:51:50.997+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:51:50.998557+0000) 2024-03-04T14:51:51.998 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:51:51.998+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:51:51.998929+0000) 2024-03-04T14:51:52.998 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:51:52.998+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:51:52.999230+0000) 2024-03-04T14:51:53.999 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:51:53.998+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:51:53.999511+0000) 2024-03-04T14:51:54.999 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:51:54.999+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:51:54.999816+0000) 2024-03-04T14:51:55.999 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:51:55.999+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:51:56.000107+0000) 2024-03-04T14:51:56.999 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:51:56.999+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:51:57.000394+0000) 2024-03-04T14:51:57.999 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:51:58.000+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:51:58.000671+0000) 2024-03-04T14:51:59.000 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:51:59.000+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:51:59.000979+0000) 2024-03-04T14:52:00.000 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:52:00.000+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:52:00.001306+0000) 2024-03-04T14:52:01.000 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:52:01.001+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:52:01.001588+0000) 2024-03-04T14:52:02.001 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:52:02.001+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:52:02.001908+0000) 2024-03-04T14:52:03.001 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:52:03.001+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:52:03.002226+0000) 2024-03-04T14:52:04.001 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:52:04.002+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:52:04.002521+0000) 2024-03-04T14:52:05.002 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:52:05.002+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:52:05.002806+0000) 2024-03-04T14:52:06.002 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:52:06.002+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:52:06.003170+0000) 2024-03-04T14:52:07.002 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:52:07.003+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:52:07.003460+0000) 2024-03-04T14:52:08.003 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:52:08.003+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:52:08.003752+0000) 2024-03-04T14:52:09.003 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:52:09.003+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:52:09.004047+0000) 2024-03-04T14:52:09.377 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:52:09.389 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:52:10.003 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:52:10.003+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:52:10.004349+0000) 2024-03-04T14:52:11.004 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:52:11.004+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:52:11.004640+0000) 2024-03-04T14:52:12.004 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:52:12.004+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:52:12.004917+0000) 2024-03-04T14:52:13.004 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:52:13.004+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:52:13.005219+0000) 2024-03-04T14:52:14.004 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:52:14.005+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:52:14.005515+0000) 2024-03-04T14:52:15.005 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:52:15.005+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:52:15.005867+0000) 2024-03-04T14:52:16.005 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:52:16.005+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:52:16.006164+0000) 2024-03-04T14:52:17.005 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:52:17.006+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:52:17.006475+0000) 2024-03-04T14:52:18.006 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:52:18.006+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:52:18.006808+0000) 2024-03-04T14:52:19.006 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:52:19.006+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:52:19.007186+0000) 2024-03-04T14:52:20.006 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:52:20.006+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:52:20.007392+0000) 2024-03-04T14:52:21.007 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:52:21.007+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:52:21.007596+0000) 2024-03-04T14:52:22.007 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:52:22.006+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:52:22.007795+0000) 2024-03-04T14:52:23.007 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:52:23.007+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:52:23.008030+0000) 2024-03-04T14:52:24.007 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:52:24.007+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:52:24.008222+0000) 2024-03-04T14:52:25.007 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:52:25.006+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:52:25.008405+0000) 2024-03-04T14:52:26.008 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:52:26.007+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:52:26.008681+0000) 2024-03-04T14:52:27.008 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:52:27.007+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:52:27.008950+0000) 2024-03-04T14:52:28.008 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:52:28.008+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:52:28.009186+0000) 2024-03-04T14:52:29.009 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:52:29.007+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:52:29.009373+0000) 2024-03-04T14:52:30.009 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:52:30.007+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:52:30.009639+0000) 2024-03-04T14:52:31.009 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:52:31.008+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:52:31.009935+0000) 2024-03-04T14:52:32.009 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:52:32.008+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:52:32.010177+0000) 2024-03-04T14:52:33.009 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:52:33.008+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:52:33.010422+0000) 2024-03-04T14:52:34.010 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:52:34.008+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:52:34.010605+0000) 2024-03-04T14:52:35.010 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:52:35.009+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:52:35.010882+0000) 2024-03-04T14:52:36.010 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:52:36.010+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:52:36.011099+0000) 2024-03-04T14:52:37.010 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:52:37.010+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:52:37.011346+0000) 2024-03-04T14:52:38.011 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:52:38.010+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:52:38.011583+0000) 2024-03-04T14:52:39.011 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:52:39.011+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:52:39.011856+0000) 2024-03-04T14:52:39.532 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:52:39.541 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:52:40.011 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:52:40.011+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:52:40.012061+0000) 2024-03-04T14:52:41.011 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:52:41.011+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:52:41.012310+0000) 2024-03-04T14:52:42.012 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:52:42.011+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:52:42.012501+0000) 2024-03-04T14:52:43.012 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:52:43.011+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:52:43.012758+0000) 2024-03-04T14:52:44.012 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:52:44.012+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:52:44.012944+0000) 2024-03-04T14:52:45.013 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:52:45.012+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:52:45.013188+0000) 2024-03-04T14:52:46.012 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:52:46.012+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:52:46.013429+0000) 2024-03-04T14:52:47.013 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:52:47.012+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:52:47.013690+0000) 2024-03-04T14:52:48.013 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:52:48.013+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:52:48.013945+0000) 2024-03-04T14:52:49.014 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:52:49.013+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:52:49.014202+0000) 2024-03-04T14:52:50.013 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:52:50.013+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:52:50.014453+0000) 2024-03-04T14:52:51.014 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:52:51.013+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:52:51.014702+0000) 2024-03-04T14:52:52.014 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:52:52.014+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:52:52.014938+0000) 2024-03-04T14:52:53.014 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:52:53.014+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:52:53.015182+0000) 2024-03-04T14:52:54.015 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:52:54.014+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:52:54.015419+0000) 2024-03-04T14:52:55.015 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:52:55.014+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:52:55.015655+0000) 2024-03-04T14:52:56.015 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:52:56.015+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:52:56.015931+0000) 2024-03-04T14:52:57.015 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:52:57.015+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:52:57.016176+0000) 2024-03-04T14:52:58.016 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:52:58.015+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:52:58.016414+0000) 2024-03-04T14:52:59.016 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:52:59.016+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:52:59.016652+0000) 2024-03-04T14:53:00.016 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:53:00.016+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:53:00.016921+0000) 2024-03-04T14:53:01.016 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:53:01.016+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:53:01.017163+0000) 2024-03-04T14:53:02.016 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:53:02.016+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:53:02.017411+0000) 2024-03-04T14:53:03.017 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:53:03.017+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:53:03.017650+0000) 2024-03-04T14:53:04.017 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:53:04.017+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:53:04.017936+0000) 2024-03-04T14:53:05.017 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:53:05.017+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:53:05.018181+0000) 2024-03-04T14:53:06.017 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:53:06.017+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:53:06.018414+0000) 2024-03-04T14:53:07.018 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:53:07.017+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:53:07.018612+0000) 2024-03-04T14:53:08.018 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:53:08.018+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:53:08.018913+0000) 2024-03-04T14:53:09.018 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:53:09.018+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:53:09.019168+0000) 2024-03-04T14:53:09.738 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:53:09.746 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:53:10.018 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:53:10.018+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:53:10.019415+0000) 2024-03-04T14:53:11.019 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:53:11.019+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:53:11.019663+0000) 2024-03-04T14:53:12.019 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:53:12.019+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:53:12.019989+0000) 2024-03-04T14:53:13.019 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:53:13.019+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:53:13.020182+0000) 2024-03-04T14:53:14.019 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:53:14.020+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:53:14.020417+0000) 2024-03-04T14:53:15.020 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:53:15.019+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:53:15.020725+0000) 2024-03-04T14:53:16.020 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:53:16.020+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:53:16.021015+0000) 2024-03-04T14:53:17.020 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:53:17.020+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:53:17.021294+0000) 2024-03-04T14:53:18.021 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:53:18.021+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:53:18.021605+0000) 2024-03-04T14:53:19.021 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:53:19.021+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:53:19.021935+0000) 2024-03-04T14:53:20.021 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:53:20.021+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:53:20.022218+0000) 2024-03-04T14:53:21.021 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:53:21.022+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:53:21.022517+0000) 2024-03-04T14:53:22.022 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:53:22.022+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:53:22.022813+0000) 2024-03-04T14:53:23.022 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:53:23.022+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:53:23.023153+0000) 2024-03-04T14:53:24.023 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:53:24.023+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:53:24.023407+0000) 2024-03-04T14:53:25.023 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:53:25.022+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:53:25.023600+0000) 2024-03-04T14:53:26.023 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:53:26.023+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:53:26.023863+0000) 2024-03-04T14:53:27.023 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:53:27.023+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:53:27.024112+0000) 2024-03-04T14:53:28.023 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:53:28.024+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:53:28.024384+0000) 2024-03-04T14:53:29.024 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:53:29.023+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:53:29.024632+0000) 2024-03-04T14:53:30.024 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:53:30.023+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:53:30.024915+0000) 2024-03-04T14:53:31.024 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:53:31.024+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:53:31.025160+0000) 2024-03-04T14:53:32.024 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:53:32.024+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:53:32.025399+0000) 2024-03-04T14:53:33.025 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:53:33.024+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:53:33.025643+0000) 2024-03-04T14:53:34.025 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:53:34.024+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:53:34.025909+0000) 2024-03-04T14:53:35.025 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:53:35.025+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:53:35.026149+0000) 2024-03-04T14:53:36.025 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:53:36.025+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:53:36.026384+0000) 2024-03-04T14:53:37.026 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:53:37.025+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:53:37.026623+0000) 2024-03-04T14:53:38.027 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:53:38.025+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:53:38.026912+0000) 2024-03-04T14:53:39.026 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:53:39.025+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:53:39.027092+0000) 2024-03-04T14:53:39.959 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:53:39.967 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:53:40.026 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:53:40.026+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:53:40.027268+0000) 2024-03-04T14:53:41.027 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:53:41.026+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:53:41.027545+0000) 2024-03-04T14:53:42.027 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:53:42.026+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:53:42.027763+0000) 2024-03-04T14:53:43.027 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:53:43.027+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:53:43.028044+0000) 2024-03-04T14:53:44.027 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:53:44.027+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:53:44.028298+0000) 2024-03-04T14:53:45.028 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:53:45.027+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:53:45.028555+0000) 2024-03-04T14:53:46.028 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:53:46.027+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:53:46.028794+0000) 2024-03-04T14:53:47.028 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:53:47.028+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:53:47.029102+0000) 2024-03-04T14:53:48.029 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:53:48.028+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:53:48.029363+0000) 2024-03-04T14:53:49.029 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:53:49.028+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:53:49.029607+0000) 2024-03-04T14:53:50.029 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:53:50.029+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:53:50.029881+0000) 2024-03-04T14:53:51.029 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:53:51.029+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:53:51.030148+0000) 2024-03-04T14:53:52.030 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:53:52.029+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:53:52.030384+0000) 2024-03-04T14:53:53.030 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:53:53.029+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:53:53.030632+0000) 2024-03-04T14:53:54.030 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:53:54.030+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:53:54.030903+0000) 2024-03-04T14:53:55.030 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:53:55.030+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:53:55.031145+0000) 2024-03-04T14:53:56.031 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:53:56.030+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:53:56.031380+0000) 2024-03-04T14:53:57.031 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:53:57.030+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:53:57.031620+0000) 2024-03-04T14:53:58.031 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:53:58.031+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:53:58.031897+0000) 2024-03-04T14:53:59.031 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:53:59.031+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:53:59.032136+0000) 2024-03-04T14:54:00.032 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:54:00.031+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:54:00.032436+0000) 2024-03-04T14:54:01.032 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:54:01.032+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:54:01.032728+0000) 2024-03-04T14:54:02.032 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:54:02.032+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:54:02.033013+0000) 2024-03-04T14:54:03.032 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:54:03.032+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:54:03.033302+0000) 2024-03-04T14:54:04.033 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:54:04.032+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:54:04.033554+0000) 2024-03-04T14:54:05.033 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:54:05.033+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:54:05.033858+0000) 2024-03-04T14:54:06.033 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:54:06.033+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:54:06.034101+0000) 2024-03-04T14:54:07.033 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:54:07.033+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:54:07.034350+0000) 2024-03-04T14:54:08.034 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:54:08.033+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:54:08.034578+0000) 2024-03-04T14:54:09.034 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:54:09.034+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:54:09.034864+0000) 2024-03-04T14:54:10.035 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:54:10.034+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:54:10.035121+0000) 2024-03-04T14:54:10.114 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:54:10.121 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:54:11.035 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:54:11.034+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:54:11.035368+0000) 2024-03-04T14:54:12.035 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:54:12.035+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:54:12.035609+0000) 2024-03-04T14:54:13.035 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:54:13.035+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:54:13.035903+0000) 2024-03-04T14:54:14.035 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:54:14.035+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:54:14.036144+0000) 2024-03-04T14:54:15.036 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:54:15.035+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:54:15.036387+0000) 2024-03-04T14:54:16.036 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:54:16.036+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:54:16.036629+0000) 2024-03-04T14:54:17.036 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:54:17.036+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:54:17.036938+0000) 2024-03-04T14:54:18.036 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:54:18.036+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:54:18.037178+0000) 2024-03-04T14:54:19.037 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:54:19.037+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:54:19.037443+0000) 2024-03-04T14:54:20.037 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:54:20.036+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:54:20.037703+0000) 2024-03-04T14:54:21.037 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:54:21.037+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:54:21.037959+0000) 2024-03-04T14:54:22.037 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:54:22.037+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:54:22.038241+0000) 2024-03-04T14:54:23.038 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:54:23.038+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:54:23.038519+0000) 2024-03-04T14:54:24.038 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:54:24.038+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:54:24.038811+0000) 2024-03-04T14:54:25.039 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:54:25.038+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:54:25.039147+0000) 2024-03-04T14:54:26.039 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:54:26.039+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:54:26.039436+0000) 2024-03-04T14:54:27.039 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:54:27.039+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:54:27.039721+0000) 2024-03-04T14:54:28.040 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:54:28.039+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:54:28.040014+0000) 2024-03-04T14:54:29.040 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:54:29.039+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:54:29.040242+0000) 2024-03-04T14:54:30.040 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:54:30.040+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:54:30.040542+0000) 2024-03-04T14:54:31.040 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:54:31.040+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:54:31.040877+0000) 2024-03-04T14:54:32.040 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:54:32.041+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:54:32.041177+0000) 2024-03-04T14:54:33.041 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:54:33.040+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:54:33.041464+0000) 2024-03-04T14:54:34.041 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:54:34.040+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:54:34.041754+0000) 2024-03-04T14:54:35.041 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:54:35.040+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:54:35.042040+0000) 2024-03-04T14:54:36.042 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:54:36.041+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:54:36.042329+0000) 2024-03-04T14:54:37.042 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:54:37.041+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:54:37.042626+0000) 2024-03-04T14:54:38.044 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:54:38.041+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:54:38.042943+0000) 2024-03-04T14:54:39.043 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:54:39.042+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:54:39.043184+0000) 2024-03-04T14:54:40.043 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:54:40.042+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:54:40.043428+0000) 2024-03-04T14:54:40.291 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:54:40.303 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:54:41.043 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:54:41.042+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:54:41.043670+0000) 2024-03-04T14:54:42.043 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:54:42.042+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:54:42.043948+0000) 2024-03-04T14:54:43.044 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:54:43.043+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:54:43.044196+0000) 2024-03-04T14:54:44.044 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:54:44.043+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:54:44.044451+0000) 2024-03-04T14:54:45.044 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:54:45.043+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:54:45.044691+0000) 2024-03-04T14:54:46.044 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:54:46.044+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:54:46.044983+0000) 2024-03-04T14:54:47.045 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:54:47.044+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:54:47.045265+0000) 2024-03-04T14:54:48.045 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:54:48.044+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:54:48.045581+0000) 2024-03-04T14:54:49.045 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:54:49.045+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:54:49.045938+0000) 2024-03-04T14:54:50.046 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:54:50.045+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:54:50.046248+0000) 2024-03-04T14:54:51.046 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:54:51.045+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:54:51.046535+0000) 2024-03-04T14:54:52.046 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:54:52.046+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:54:52.046865+0000) 2024-03-04T14:54:53.046 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:54:53.046+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:54:53.047116+0000) 2024-03-04T14:54:54.047 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:54:54.046+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:54:54.047407+0000) 2024-03-04T14:54:55.047 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:54:55.046+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:54:55.047688+0000) 2024-03-04T14:54:56.047 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:54:56.047+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:54:56.048030+0000) 2024-03-04T14:54:57.048 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:54:57.047+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:54:57.048290+0000) 2024-03-04T14:54:58.048 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:54:58.047+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:54:58.048527+0000) 2024-03-04T14:54:59.048 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:54:59.048+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:54:59.048819+0000) 2024-03-04T14:55:00.048 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:55:00.048+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:55:00.049104+0000) 2024-03-04T14:55:01.049 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:55:01.048+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:55:01.049342+0000) 2024-03-04T14:55:02.049 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:55:02.048+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:55:02.049580+0000) 2024-03-04T14:55:03.049 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:55:03.049+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:55:03.049871+0000) 2024-03-04T14:55:04.049 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:55:04.049+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:55:04.050062+0000) 2024-03-04T14:55:05.050 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:55:05.049+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:55:05.050301+0000) 2024-03-04T14:55:06.050 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:55:06.049+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:55:06.050563+0000) 2024-03-04T14:55:07.050 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:55:07.050+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:55:07.050878+0000) 2024-03-04T14:55:08.051 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:55:08.050+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:55:08.051190+0000) 2024-03-04T14:55:09.051 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:55:09.050+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:55:09.051472+0000) 2024-03-04T14:55:10.051 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:55:10.051+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:55:10.051763+0000) 2024-03-04T14:55:10.441 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:55:10.449 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:55:11.051 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:55:11.051+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:55:11.052067+0000) 2024-03-04T14:55:12.052 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:55:12.051+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:55:12.052377+0000) 2024-03-04T14:55:13.052 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:55:13.052+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:55:13.052682+0000) 2024-03-04T14:55:14.052 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:55:14.052+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:55:14.052974+0000) 2024-03-04T14:55:15.053 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:55:15.052+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:55:15.053268+0000) 2024-03-04T14:55:16.053 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:55:16.053+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:55:16.053564+0000) 2024-03-04T14:55:17.053 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:55:17.053+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:55:17.053881+0000) 2024-03-04T14:55:18.053 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:55:18.053+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:55:18.054124+0000) 2024-03-04T14:55:19.054 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:55:19.053+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:55:19.054416+0000) 2024-03-04T14:55:20.054 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:55:20.054+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:55:20.054717+0000) 2024-03-04T14:55:21.054 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:55:21.054+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:55:21.054998+0000) 2024-03-04T14:55:22.055 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:55:22.054+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:55:22.055287+0000) 2024-03-04T14:55:23.055 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:55:23.055+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:55:23.055570+0000) 2024-03-04T14:55:24.055 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:55:24.055+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:55:24.055913+0000) 2024-03-04T14:55:25.058 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:55:25.055+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:55:25.056194+0000) 2024-03-04T14:55:26.056 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:55:26.056+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:55:26.056484+0000) 2024-03-04T14:55:27.056 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:55:27.056+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:55:27.056773+0000) 2024-03-04T14:55:28.057 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:55:28.056+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:55:28.057063+0000) 2024-03-04T14:55:29.057 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:55:29.057+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:55:29.057351+0000) 2024-03-04T14:55:30.057 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:55:30.057+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:55:30.057644+0000) 2024-03-04T14:55:31.057 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:55:31.057+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:55:31.057960+0000) 2024-03-04T14:55:32.058 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:55:32.058+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:55:32.058247+0000) 2024-03-04T14:55:33.058 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:55:33.057+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:55:33.058475+0000) 2024-03-04T14:55:34.058 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:55:34.058+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:55:34.058765+0000) 2024-03-04T14:55:35.059 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:55:35.058+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:55:35.059051+0000) 2024-03-04T14:55:36.059 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:55:36.058+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:55:36.059342+0000) 2024-03-04T14:55:37.059 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:55:37.058+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:55:37.059647+0000) 2024-03-04T14:55:38.059 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:55:38.058+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:55:38.059977+0000) 2024-03-04T14:55:39.060 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:55:39.059+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:55:39.060261+0000) 2024-03-04T14:55:40.060 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:55:40.059+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:55:40.060555+0000) 2024-03-04T14:55:40.663 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:55:40.677 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:55:41.061 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:55:41.059+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:55:41.060897+0000) 2024-03-04T14:55:42.061 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:55:42.060+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:55:42.061186+0000) 2024-03-04T14:55:43.061 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:55:43.060+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:55:43.061494+0000) 2024-03-04T14:55:44.062 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:55:44.060+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:55:44.061785+0000) 2024-03-04T14:55:45.062 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:55:45.061+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:55:45.062093+0000) 2024-03-04T14:55:46.062 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:55:46.061+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:55:46.062361+0000) 2024-03-04T14:55:47.062 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:55:47.061+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:55:47.062656+0000) 2024-03-04T14:55:48.062 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:55:48.062+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:55:48.062947+0000) 2024-03-04T14:55:49.063 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:55:49.062+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:55:49.063202+0000) 2024-03-04T14:55:50.063 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:55:50.062+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:55:50.063464+0000) 2024-03-04T14:55:51.063 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:55:51.062+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:55:51.063704+0000) 2024-03-04T14:55:52.063 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:55:52.063+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:55:52.063972+0000) 2024-03-04T14:55:53.064 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:55:53.063+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:55:53.064216+0000) 2024-03-04T14:55:54.064 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:55:54.063+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:55:54.064459+0000) 2024-03-04T14:55:55.064 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:55:55.063+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:55:55.064704+0000) 2024-03-04T14:55:56.065 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:55:56.064+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:55:56.064957+0000) 2024-03-04T14:55:57.065 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:55:57.064+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:55:57.065223+0000) 2024-03-04T14:55:58.065 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:55:58.064+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:55:58.065461+0000) 2024-03-04T14:55:59.065 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:55:59.064+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:55:59.065709+0000) 2024-03-04T14:56:00.065 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:56:00.065+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:56:00.065960+0000) 2024-03-04T14:56:01.066 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:56:01.065+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:56:01.066199+0000) 2024-03-04T14:56:02.066 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:56:02.065+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:56:02.066439+0000) 2024-03-04T14:56:03.066 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:56:03.065+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:56:03.066681+0000) 2024-03-04T14:56:04.066 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:56:04.066+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:56:04.066924+0000) 2024-03-04T14:56:05.067 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:56:05.066+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:56:05.067206+0000) 2024-03-04T14:56:06.067 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:56:06.066+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:56:06.067461+0000) 2024-03-04T14:56:07.067 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:56:07.067+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:56:07.067724+0000) 2024-03-04T14:56:08.067 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:56:08.067+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:56:08.067971+0000) 2024-03-04T14:56:09.068 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:56:09.067+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:56:09.068228+0000) 2024-03-04T14:56:10.068 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:56:10.067+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:56:10.068491+0000) 2024-03-04T14:56:10.877 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:56:10.885 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:56:11.068 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:56:11.068+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:56:11.068734+0000) 2024-03-04T14:56:12.069 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:56:12.068+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:56:12.068970+0000) 2024-03-04T14:56:13.069 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:56:13.068+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:56:13.069222+0000) 2024-03-04T14:56:14.069 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:56:14.068+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:56:14.069428+0000) 2024-03-04T14:56:15.069 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:56:15.069+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:56:15.069691+0000) 2024-03-04T14:56:16.069 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:56:16.069+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:56:16.069942+0000) 2024-03-04T14:56:17.070 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:56:17.069+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:56:17.070198+0000) 2024-03-04T14:56:18.070 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:56:18.069+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:56:18.070449+0000) 2024-03-04T14:56:19.070 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:56:19.070+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:56:19.070707+0000) 2024-03-04T14:56:20.070 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:56:20.070+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:56:20.070993+0000) 2024-03-04T14:56:21.071 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:56:21.070+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:56:21.071249+0000) 2024-03-04T14:56:22.071 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:56:22.071+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:56:22.071509+0000) 2024-03-04T14:56:23.071 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:56:23.071+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:56:23.071767+0000) 2024-03-04T14:56:24.072 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:56:24.071+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:56:24.072023+0000) 2024-03-04T14:56:25.072 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:56:25.071+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:56:25.072293+0000) 2024-03-04T14:56:26.072 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:56:26.072+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:56:26.072569+0000) 2024-03-04T14:56:27.076 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:56:27.072+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:56:27.072865+0000) 2024-03-04T14:56:28.073 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:56:28.072+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:56:28.073126+0000) 2024-03-04T14:56:29.073 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:56:29.073+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:56:29.073439+0000) 2024-03-04T14:56:30.073 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:56:30.073+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:56:30.073697+0000) 2024-03-04T14:56:31.073 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:56:31.073+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:56:31.073957+0000) 2024-03-04T14:56:32.074 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:56:32.073+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:56:32.074213+0000) 2024-03-04T14:56:33.074 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:56:33.074+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:56:33.074472+0000) 2024-03-04T14:56:34.074 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:56:34.074+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:56:34.074754+0000) 2024-03-04T14:56:35.074 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:56:35.074+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:56:35.075019+0000) 2024-03-04T14:56:36.075 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:56:36.075+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:56:36.075273+0000) 2024-03-04T14:56:37.075 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:56:37.075+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:56:37.075530+0000) 2024-03-04T14:56:38.076 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:56:38.075+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:56:38.075781+0000) 2024-03-04T14:56:39.076 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:56:39.074+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:56:39.076083+0000) 2024-03-04T14:56:40.076 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:56:40.075+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:56:40.076348+0000) 2024-03-04T14:56:41.022 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:56:41.030 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:56:41.076 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:56:41.074+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:56:41.076546+0000) 2024-03-04T14:56:42.077 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:56:42.074+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:56:42.076794+0000) 2024-03-04T14:56:43.077 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:56:43.076+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:56:43.077068+0000) 2024-03-04T14:56:44.077 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:56:44.076+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:56:44.077302+0000) 2024-03-04T14:56:45.077 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:56:45.076+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:56:45.077547+0000) 2024-03-04T14:56:46.077 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:56:46.076+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:56:46.077785+0000) 2024-03-04T14:56:47.078 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:56:47.077+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:56:47.078031+0000) 2024-03-04T14:56:48.079 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:56:48.077+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:56:48.078221+0000) 2024-03-04T14:56:49.078 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:56:49.077+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:56:49.078463+0000) 2024-03-04T14:56:50.078 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:56:50.077+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:56:50.078730+0000) 2024-03-04T14:56:51.079 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:56:51.078+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:56:51.078996+0000) 2024-03-04T14:56:52.079 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:56:52.078+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:56:52.079293+0000) 2024-03-04T14:56:53.079 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:56:53.078+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:56:53.079586+0000) 2024-03-04T14:56:54.080 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:56:54.079+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:56:54.079902+0000) 2024-03-04T14:56:55.080 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:56:55.079+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:56:55.080151+0000) 2024-03-04T14:56:56.080 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:56:56.079+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:56:56.080393+0000) 2024-03-04T14:56:57.080 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:56:57.079+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:56:57.080642+0000) 2024-03-04T14:56:58.081 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:56:58.080+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:56:58.080919+0000) 2024-03-04T14:56:59.081 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:56:59.080+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:56:59.081196+0000) 2024-03-04T14:57:00.081 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:57:00.080+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:57:00.081402+0000) 2024-03-04T14:57:01.081 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:57:01.080+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:57:01.081647+0000) 2024-03-04T14:57:02.082 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:57:02.081+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:57:02.081923+0000) 2024-03-04T14:57:03.082 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:57:03.081+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:57:03.082120+0000) 2024-03-04T14:57:04.082 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:57:04.081+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:57:04.082371+0000) 2024-03-04T14:57:05.082 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:57:05.081+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:57:05.082631+0000) 2024-03-04T14:57:06.083 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:57:06.082+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:57:06.082909+0000) 2024-03-04T14:57:07.083 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:57:07.082+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:57:07.083157+0000) 2024-03-04T14:57:08.083 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:57:08.082+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:57:08.083413+0000) 2024-03-04T14:57:09.084 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:57:09.082+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:57:09.083677+0000) 2024-03-04T14:57:10.084 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:57:10.083+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:57:10.083956+0000) 2024-03-04T14:57:11.084 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:57:11.083+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:57:11.084213+0000) 2024-03-04T14:57:11.211 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:57:11.219 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:57:12.084 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:57:12.083+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:57:12.084469+0000) 2024-03-04T14:57:13.084 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:57:13.084+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:57:13.084729+0000) 2024-03-04T14:57:14.085 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:57:14.084+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:57:14.084998+0000) 2024-03-04T14:57:15.085 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:57:15.084+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:57:15.085251+0000) 2024-03-04T14:57:16.085 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:57:16.084+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:57:16.085504+0000) 2024-03-04T14:57:17.085 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:57:17.085+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:57:17.085768+0000) 2024-03-04T14:57:18.086 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:57:18.085+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:57:18.086016+0000) 2024-03-04T14:57:19.086 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:57:19.085+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:57:19.086267+0000) 2024-03-04T14:57:20.086 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:57:20.085+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:57:20.086517+0000) 2024-03-04T14:57:21.087 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:57:21.086+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:57:21.086769+0000) 2024-03-04T14:57:22.087 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:57:22.086+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:57:22.087024+0000) 2024-03-04T14:57:23.087 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:57:23.086+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:57:23.087347+0000) 2024-03-04T14:57:24.087 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:57:24.087+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:57:24.087596+0000) 2024-03-04T14:57:25.088 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:57:25.087+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:57:25.087880+0000) 2024-03-04T14:57:26.088 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:57:26.087+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:57:26.088126+0000) 2024-03-04T14:57:27.088 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:57:27.088+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:57:27.088386+0000) 2024-03-04T14:57:28.089 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:57:28.088+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:57:28.088642+0000) 2024-03-04T14:57:29.089 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:57:29.088+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:57:29.088946+0000) 2024-03-04T14:57:30.089 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:57:30.088+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:57:30.089206+0000) 2024-03-04T14:57:31.089 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:57:31.089+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:57:31.089458+0000) 2024-03-04T14:57:32.089 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:57:32.089+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:57:32.089706+0000) 2024-03-04T14:57:33.090 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:57:33.089+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:57:33.089983+0000) 2024-03-04T14:57:34.090 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:57:34.089+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:57:34.090228+0000) 2024-03-04T14:57:35.090 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:57:35.090+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:57:35.090489+0000) 2024-03-04T14:57:36.090 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:57:36.090+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:57:36.090739+0000) 2024-03-04T14:57:37.091 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:57:37.090+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:57:37.090997+0000) 2024-03-04T14:57:38.091 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:57:38.091+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:57:38.091281+0000) 2024-03-04T14:57:39.092 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:57:39.091+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:57:39.091531+0000) 2024-03-04T14:57:40.092 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:57:40.091+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:57:40.091791+0000) 2024-03-04T14:57:41.092 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:57:41.091+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:57:41.092075+0000) 2024-03-04T14:57:41.376 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:57:41.385 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:57:42.092 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:57:42.091+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:57:42.092322+0000) 2024-03-04T14:57:43.093 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:57:43.090+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:57:43.092538+0000) 2024-03-04T14:57:44.093 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:57:44.090+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:57:44.092805+0000) 2024-03-04T14:57:45.093 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:57:45.092+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:57:45.093100+0000) 2024-03-04T14:57:46.094 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:57:46.092+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:57:46.093348+0000) 2024-03-04T14:57:47.094 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:57:47.092+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:57:47.093603+0000) 2024-03-04T14:57:48.094 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:57:48.092+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:57:48.093888+0000) 2024-03-04T14:57:49.094 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:57:49.093+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:57:49.094146+0000) 2024-03-04T14:57:50.095 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:57:50.093+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:57:50.094406+0000) 2024-03-04T14:57:51.096 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:57:51.093+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:57:51.094658+0000) 2024-03-04T14:57:52.095 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:57:52.093+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:57:52.094934+0000) 2024-03-04T14:57:53.095 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:57:53.094+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:57:53.095186+0000) 2024-03-04T14:57:54.096 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:57:54.094+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:57:54.095439+0000) 2024-03-04T14:57:55.096 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:57:55.094+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:57:55.095705+0000) 2024-03-04T14:57:56.096 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:57:56.095+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:57:56.095953+0000) 2024-03-04T14:57:57.096 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:57:57.095+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:57:57.096207+0000) 2024-03-04T14:57:58.096 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:57:58.095+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:57:58.096457+0000) 2024-03-04T14:57:59.097 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:57:59.095+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:57:59.096715+0000) 2024-03-04T14:58:00.099 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:58:00.096+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:58:00.096976+0000) 2024-03-04T14:58:01.097 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:58:01.096+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:58:01.097218+0000) 2024-03-04T14:58:02.097 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:58:02.096+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:58:02.097408+0000) 2024-03-04T14:58:03.098 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:58:03.096+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:58:03.097686+0000) 2024-03-04T14:58:04.098 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:58:04.097+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:58:04.097936+0000) 2024-03-04T14:58:05.098 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:58:05.097+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:58:05.098197+0000) 2024-03-04T14:58:06.099 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:58:06.097+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:58:06.098448+0000) 2024-03-04T14:58:07.099 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:58:07.097+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:58:07.098708+0000) 2024-03-04T14:58:08.099 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:58:08.098+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:58:08.098961+0000) 2024-03-04T14:58:09.099 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:58:09.098+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:58:09.099167+0000) 2024-03-04T14:58:10.100 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:58:10.098+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:58:10.099433+0000) 2024-03-04T14:58:11.100 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:58:11.099+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:58:11.099695+0000) 2024-03-04T14:58:11.589 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:58:11.597 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:58:12.100 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:58:12.099+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:58:12.099945+0000) 2024-03-04T14:58:13.100 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:58:13.099+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:58:13.100227+0000) 2024-03-04T14:58:14.101 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:58:14.099+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:58:14.100475+0000) 2024-03-04T14:58:15.101 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:58:15.100+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:58:15.100713+0000) 2024-03-04T14:58:16.101 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:58:16.100+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:58:16.100997+0000) 2024-03-04T14:58:17.101 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:58:17.100+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:58:17.101237+0000) 2024-03-04T14:58:18.102 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:58:18.100+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:58:18.101472+0000) 2024-03-04T14:58:19.102 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:58:19.101+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:58:19.101719+0000) 2024-03-04T14:58:20.102 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:58:20.101+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:58:20.101962+0000) 2024-03-04T14:58:21.102 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:58:21.101+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:58:21.102220+0000) 2024-03-04T14:58:22.102 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:58:22.101+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:58:22.102470+0000) 2024-03-04T14:58:23.103 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:58:23.102+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:58:23.102727+0000) 2024-03-04T14:58:24.103 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:58:24.102+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:58:24.102976+0000) 2024-03-04T14:58:25.103 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:58:25.102+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:58:25.103218+0000) 2024-03-04T14:58:26.103 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:58:26.103+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:58:26.103465+0000) 2024-03-04T14:58:27.104 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:58:27.103+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:58:27.103704+0000) 2024-03-04T14:58:28.104 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:58:28.103+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:58:28.103940+0000) 2024-03-04T14:58:29.105 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:58:29.103+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:58:29.104180+0000) 2024-03-04T14:58:30.105 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:58:30.104+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:58:30.104445+0000) 2024-03-04T14:58:31.105 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:58:31.104+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:58:31.104687+0000) 2024-03-04T14:58:32.105 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:58:32.104+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:58:32.104967+0000) 2024-03-04T14:58:33.105 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:58:33.104+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:58:33.105252+0000) 2024-03-04T14:58:34.106 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:58:34.105+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:58:34.105500+0000) 2024-03-04T14:58:35.106 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:58:35.104+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:58:35.105681+0000) 2024-03-04T14:58:36.106 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:58:36.105+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:58:36.105945+0000) 2024-03-04T14:58:37.106 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:58:37.105+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:58:37.106183+0000) 2024-03-04T14:58:38.106 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:58:38.106+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:58:38.106437+0000) 2024-03-04T14:58:39.107 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:58:39.106+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:58:39.106699+0000) 2024-03-04T14:58:40.107 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:58:40.106+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:58:40.106966+0000) 2024-03-04T14:58:41.107 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:58:41.107+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:58:41.107219+0000) 2024-03-04T14:58:41.810 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:58:41.823 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:58:42.107 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:58:42.106+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:58:42.107491+0000) 2024-03-04T14:58:43.108 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:58:43.107+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:58:43.107754+0000) 2024-03-04T14:58:44.108 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:58:44.107+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:58:44.108009+0000) 2024-03-04T14:58:45.109 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:58:45.108+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:58:45.108262+0000) 2024-03-04T14:58:46.108 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:58:46.107+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:58:46.108512+0000) 2024-03-04T14:58:47.109 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:58:47.107+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:58:47.108770+0000) 2024-03-04T14:58:48.109 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:58:48.107+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:58:48.109027+0000) 2024-03-04T14:58:49.110 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:58:49.108+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:58:49.109307+0000) 2024-03-04T14:58:50.110 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:58:50.108+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:58:50.109527+0000) 2024-03-04T14:58:51.110 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:58:51.108+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:58:51.109756+0000) 2024-03-04T14:58:52.110 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:58:52.108+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:58:52.110014+0000) 2024-03-04T14:58:53.110 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:58:53.109+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:58:53.110273+0000) 2024-03-04T14:58:54.110 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:58:54.109+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:58:54.110531+0000) 2024-03-04T14:58:55.111 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:58:55.109+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:58:55.110732+0000) 2024-03-04T14:58:56.111 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:58:56.110+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:58:56.110985+0000) 2024-03-04T14:58:57.111 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:58:57.110+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:58:57.111178+0000) 2024-03-04T14:58:58.112 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:58:58.110+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:58:58.111437+0000) 2024-03-04T14:58:59.112 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:58:59.110+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:58:59.111680+0000) 2024-03-04T14:59:00.112 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:59:00.111+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:59:00.111958+0000) 2024-03-04T14:59:01.112 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:59:01.111+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:59:01.112214+0000) 2024-03-04T14:59:02.113 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:59:02.111+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:59:02.112487+0000) 2024-03-04T14:59:03.113 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:59:03.111+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:59:03.112760+0000) 2024-03-04T14:59:04.113 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:59:04.112+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:59:04.113009+0000) 2024-03-04T14:59:05.113 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:59:05.112+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:59:05.113264+0000) 2024-03-04T14:59:06.114 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:59:06.112+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:59:06.113513+0000) 2024-03-04T14:59:07.114 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:59:07.112+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:59:07.113770+0000) 2024-03-04T14:59:08.114 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:59:08.113+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:59:08.114030+0000) 2024-03-04T14:59:09.114 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:59:09.113+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:59:09.114325+0000) 2024-03-04T14:59:10.115 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:59:10.113+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:59:10.114567+0000) 2024-03-04T14:59:11.115 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:59:11.114+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:59:11.114818+0000) 2024-03-04T14:59:11.965 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:59:11.974 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:59:12.115 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:59:12.114+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:59:12.115095+0000) 2024-03-04T14:59:13.116 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:59:13.114+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:59:13.115336+0000) 2024-03-04T14:59:14.116 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:59:14.114+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:59:14.115571+0000) 2024-03-04T14:59:15.116 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:59:15.115+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:59:15.115813+0000) 2024-03-04T14:59:16.116 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:59:16.115+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:59:16.116075+0000) 2024-03-04T14:59:17.116 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:59:17.115+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:59:17.116312+0000) 2024-03-04T14:59:18.117 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:59:18.115+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:59:18.116549+0000) 2024-03-04T14:59:19.117 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:59:19.116+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:59:19.116795+0000) 2024-03-04T14:59:20.117 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:59:20.116+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:59:20.117082+0000) 2024-03-04T14:59:21.118 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:59:21.116+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:59:21.117324+0000) 2024-03-04T14:59:22.118 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:59:22.117+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:59:22.117586+0000) 2024-03-04T14:59:23.118 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:59:23.117+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:59:23.117853+0000) 2024-03-04T14:59:24.119 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:59:24.117+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:59:24.118090+0000) 2024-03-04T14:59:25.118 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:59:25.117+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:59:25.118356+0000) 2024-03-04T14:59:26.119 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:59:26.117+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:59:26.118604+0000) 2024-03-04T14:59:27.119 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:59:27.118+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:59:27.118886+0000) 2024-03-04T14:59:28.119 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:59:28.118+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:59:28.119122+0000) 2024-03-04T14:59:29.120 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:59:29.118+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:59:29.119381+0000) 2024-03-04T14:59:30.120 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:59:30.119+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:59:30.119628+0000) 2024-03-04T14:59:31.120 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:59:31.119+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:59:31.119909+0000) 2024-03-04T14:59:32.120 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:59:32.119+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:59:32.120162+0000) 2024-03-04T14:59:33.121 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:59:33.120+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:59:33.120425+0000) 2024-03-04T14:59:34.121 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:59:34.120+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:59:34.120679+0000) 2024-03-04T14:59:35.121 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:59:35.120+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:59:35.120918+0000) 2024-03-04T14:59:36.121 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:59:36.120+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:59:36.121188+0000) 2024-03-04T14:59:37.122 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:59:37.121+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:59:37.121427+0000) 2024-03-04T14:59:38.122 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:59:38.121+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:59:38.121664+0000) 2024-03-04T14:59:39.122 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:59:39.121+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:59:39.121950+0000) 2024-03-04T14:59:40.122 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:59:40.121+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:59:40.122151+0000) 2024-03-04T14:59:41.122 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:59:41.121+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:59:41.122400+0000) 2024-03-04T14:59:42.112 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:59:42.121 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T14:59:42.122 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:59:42.121+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:59:42.122620+0000) 2024-03-04T14:59:43.123 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:59:43.122+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:59:43.122890+0000) 2024-03-04T14:59:44.123 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:59:44.122+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:59:44.123124+0000) 2024-03-04T14:59:45.124 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:59:45.123+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:59:45.123386+0000) 2024-03-04T14:59:46.124 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:59:46.123+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:59:46.123620+0000) 2024-03-04T14:59:47.124 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:59:47.122+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:59:47.123803+0000) 2024-03-04T14:59:48.124 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:59:48.123+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:59:48.124069+0000) 2024-03-04T14:59:49.124 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:59:49.124+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:59:49.124310+0000) 2024-03-04T14:59:50.125 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:59:50.123+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:59:50.124554+0000) 2024-03-04T14:59:51.125 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:59:51.123+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:59:51.124791+0000) 2024-03-04T14:59:52.125 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:59:52.123+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:59:52.125047+0000) 2024-03-04T14:59:53.125 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:59:53.124+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:59:53.125260+0000) 2024-03-04T14:59:54.126 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:59:54.124+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:59:54.125496+0000) 2024-03-04T14:59:55.126 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:59:55.124+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:59:55.125736+0000) 2024-03-04T14:59:56.126 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:59:56.124+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:59:56.126006+0000) 2024-03-04T14:59:57.127 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:59:57.125+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:59:57.126302+0000) 2024-03-04T14:59:58.127 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:59:58.125+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:59:58.126559+0000) 2024-03-04T14:59:59.127 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T14:59:59.125+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T13:59:59.126807+0000) 2024-03-04T15:00:00.127 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:00:00.126+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:00:00.127094+0000) 2024-03-04T15:00:01.127 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:00:01.126+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:00:01.127332+0000) 2024-03-04T15:00:02.132 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:00:02.126+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:00:02.127558+0000) 2024-03-04T15:00:03.128 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:00:03.127+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:00:03.127851+0000) 2024-03-04T15:00:04.128 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:00:04.127+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:00:04.128112+0000) 2024-03-04T15:00:05.129 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:00:05.127+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:00:05.128356+0000) 2024-03-04T15:00:06.129 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:00:06.127+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:00:06.128585+0000) 2024-03-04T15:00:07.129 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:00:07.128+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:00:07.128908+0000) 2024-03-04T15:00:08.129 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:00:08.128+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:00:08.129150+0000) 2024-03-04T15:00:09.129 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:00:09.128+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:00:09.129388+0000) 2024-03-04T15:00:10.130 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:00:10.128+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:00:10.129631+0000) 2024-03-04T15:00:11.130 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:00:11.129+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:00:11.129917+0000) 2024-03-04T15:00:12.130 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:00:12.129+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:00:12.130166+0000) 2024-03-04T15:00:12.252 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:00:12.260 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:00:13.131 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:00:13.129+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:00:13.130405+0000) 2024-03-04T15:00:14.131 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:00:14.129+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:00:14.130661+0000) 2024-03-04T15:00:15.131 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:00:15.130+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:00:15.130968+0000) 2024-03-04T15:00:16.131 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:00:16.130+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:00:16.131202+0000) 2024-03-04T15:00:17.132 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:00:17.130+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:00:17.131443+0000) 2024-03-04T15:00:18.132 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:00:18.131+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:00:18.131685+0000) 2024-03-04T15:00:19.132 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:00:19.131+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:00:19.131930+0000) 2024-03-04T15:00:20.132 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:00:20.131+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:00:20.132183+0000) 2024-03-04T15:00:21.133 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:00:21.131+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:00:21.132427+0000) 2024-03-04T15:00:22.133 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:00:22.132+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:00:22.132666+0000) 2024-03-04T15:00:23.133 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:00:23.132+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:00:23.132962+0000) 2024-03-04T15:00:24.133 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:00:24.132+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:00:24.133228+0000) 2024-03-04T15:00:25.133 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:00:25.132+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:00:25.133467+0000) 2024-03-04T15:00:26.134 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:00:26.133+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:00:26.133708+0000) 2024-03-04T15:00:27.134 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:00:27.133+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:00:27.133948+0000) 2024-03-04T15:00:28.134 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:00:28.133+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:00:28.134184+0000) 2024-03-04T15:00:29.134 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:00:29.133+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:00:29.134427+0000) 2024-03-04T15:00:30.135 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:00:30.134+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:00:30.134672+0000) 2024-03-04T15:00:31.135 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:00:31.134+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:00:31.134955+0000) 2024-03-04T15:00:32.135 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:00:32.134+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:00:32.135212+0000) 2024-03-04T15:00:33.136 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:00:33.135+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:00:33.135474+0000) 2024-03-04T15:00:34.136 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:00:34.135+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:00:34.135710+0000) 2024-03-04T15:00:35.136 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:00:35.135+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:00:35.135956+0000) 2024-03-04T15:00:36.136 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:00:36.135+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:00:36.136148+0000) 2024-03-04T15:00:37.136 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:00:37.136+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:00:37.136389+0000) 2024-03-04T15:00:38.137 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:00:38.136+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:00:38.136625+0000) 2024-03-04T15:00:39.137 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:00:39.136+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:00:39.136925+0000) 2024-03-04T15:00:40.137 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:00:40.136+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:00:40.137189+0000) 2024-03-04T15:00:41.138 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:00:41.137+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:00:41.137430+0000) 2024-03-04T15:00:42.138 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:00:42.137+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:00:42.137673+0000) 2024-03-04T15:00:42.398 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:00:42.406 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:00:43.138 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:00:43.137+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:00:43.137979+0000) 2024-03-04T15:00:44.138 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:00:44.137+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:00:44.138230+0000) 2024-03-04T15:00:45.139 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:00:45.138+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:00:45.138475+0000) 2024-03-04T15:00:46.139 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:00:46.138+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:00:46.138714+0000) 2024-03-04T15:00:47.139 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:00:47.138+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:00:47.139053+0000) 2024-03-04T15:00:48.139 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:00:48.138+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:00:48.139259+0000) 2024-03-04T15:00:49.140 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:00:49.138+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:00:49.139452+0000) 2024-03-04T15:00:50.140 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:00:50.138+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:00:50.139643+0000) 2024-03-04T15:00:51.140 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:00:51.139+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:00:51.139954+0000) 2024-03-04T15:00:52.140 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:00:52.140+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:00:52.140204+0000) 2024-03-04T15:00:53.141 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:00:53.140+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:00:53.140447+0000) 2024-03-04T15:00:54.141 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:00:54.139+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:00:54.140682+0000) 2024-03-04T15:00:55.141 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:00:55.139+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:00:55.140954+0000) 2024-03-04T15:00:56.141 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:00:56.140+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:00:56.141199+0000) 2024-03-04T15:00:57.142 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:00:57.140+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:00:57.141436+0000) 2024-03-04T15:00:58.142 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:00:58.140+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:00:58.141676+0000) 2024-03-04T15:00:59.142 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:00:59.140+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:00:59.141952+0000) 2024-03-04T15:01:00.143 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:01:00.141+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:01:00.142215+0000) 2024-03-04T15:01:01.143 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:01:01.141+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:01:01.142465+0000) 2024-03-04T15:01:02.143 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:01:02.141+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:01:02.142709+0000) 2024-03-04T15:01:03.143 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:01:03.141+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:01:03.142975+0000) 2024-03-04T15:01:04.143 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:01:04.142+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:01:04.143227+0000) 2024-03-04T15:01:05.144 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:01:05.142+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:01:05.143466+0000) 2024-03-04T15:01:06.144 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:01:06.142+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:01:06.143698+0000) 2024-03-04T15:01:07.144 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:01:07.143+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:01:07.143943+0000) 2024-03-04T15:01:08.145 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:01:08.143+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:01:08.144201+0000) 2024-03-04T15:01:09.145 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:01:09.143+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:01:09.144441+0000) 2024-03-04T15:01:10.145 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:01:10.143+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:01:10.144689+0000) 2024-03-04T15:01:11.145 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:01:11.144+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:01:11.144950+0000) 2024-03-04T15:01:12.146 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:01:12.144+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:01:12.145186+0000) 2024-03-04T15:01:12.603 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:01:12.612 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:01:13.146 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:01:13.144+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:01:13.145428+0000) 2024-03-04T15:01:14.146 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:01:14.144+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:01:14.145662+0000) 2024-03-04T15:01:15.146 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:01:15.145+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:01:15.145945+0000) 2024-03-04T15:01:16.146 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:01:16.145+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:01:16.146183+0000) 2024-03-04T15:01:17.147 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:01:17.145+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:01:17.146427+0000) 2024-03-04T15:01:18.147 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:01:18.145+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:01:18.146623+0000) 2024-03-04T15:01:19.147 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:01:19.146+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:01:19.146916+0000) 2024-03-04T15:01:20.147 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:01:20.146+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:01:20.147160+0000) 2024-03-04T15:01:21.147 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:01:21.146+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:01:21.147399+0000) 2024-03-04T15:01:22.148 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:01:22.146+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:01:22.147634+0000) 2024-03-04T15:01:23.148 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:01:23.147+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:01:23.147927+0000) 2024-03-04T15:01:24.148 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:01:24.147+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:01:24.148137+0000) 2024-03-04T15:01:25.149 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:01:25.147+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:01:25.148375+0000) 2024-03-04T15:01:26.149 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:01:26.147+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:01:26.148624+0000) 2024-03-04T15:01:27.149 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:01:27.148+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:01:27.148904+0000) 2024-03-04T15:01:28.149 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:01:28.148+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:01:28.149146+0000) 2024-03-04T15:01:29.149 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:01:29.148+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:01:29.149379+0000) 2024-03-04T15:01:30.150 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:01:30.149+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:01:30.149624+0000) 2024-03-04T15:01:31.150 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:01:31.149+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:01:31.149920+0000) 2024-03-04T15:01:32.150 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:01:32.149+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:01:32.150159+0000) 2024-03-04T15:01:33.151 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:01:33.149+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:01:33.150478+0000) 2024-03-04T15:01:34.151 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:01:34.149+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:01:34.150711+0000) 2024-03-04T15:01:35.151 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:01:35.150+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:01:35.150967+0000) 2024-03-04T15:01:36.152 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:01:36.150+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:01:36.151354+0000) 2024-03-04T15:01:37.152 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:01:37.151+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:01:37.151616+0000) 2024-03-04T15:01:38.152 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:01:38.151+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:01:38.151886+0000) 2024-03-04T15:01:39.152 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:01:39.151+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:01:39.152134+0000) 2024-03-04T15:01:40.153 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:01:40.151+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:01:40.152375+0000) 2024-03-04T15:01:41.153 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:01:41.152+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:01:41.152615+0000) 2024-03-04T15:01:42.153 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:01:42.152+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:01:42.152887+0000) 2024-03-04T15:01:42.826 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:01:42.834 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:01:43.153 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:01:43.152+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:01:43.153138+0000) 2024-03-04T15:01:44.154 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:01:44.153+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:01:44.153380+0000) 2024-03-04T15:01:45.154 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:01:45.153+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:01:45.153617+0000) 2024-03-04T15:01:46.154 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:01:46.153+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:01:46.153875+0000) 2024-03-04T15:01:47.154 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:01:47.153+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:01:47.154119+0000) 2024-03-04T15:01:48.154 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:01:48.153+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:01:48.154371+0000) 2024-03-04T15:01:49.155 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:01:49.153+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:01:49.154634+0000) 2024-03-04T15:01:50.155 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:01:50.154+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:01:50.154919+0000) 2024-03-04T15:01:51.155 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:01:51.154+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:01:51.155160+0000) 2024-03-04T15:01:52.156 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:01:52.155+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:01:52.155396+0000) 2024-03-04T15:01:53.156 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:01:53.155+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:01:53.155641+0000) 2024-03-04T15:01:54.156 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:01:54.155+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:01:54.155933+0000) 2024-03-04T15:01:55.157 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:01:55.156+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:01:55.156172+0000) 2024-03-04T15:01:56.157 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:01:56.156+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:01:56.156408+0000) 2024-03-04T15:01:57.157 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:01:57.156+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:01:57.156647+0000) 2024-03-04T15:01:58.157 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:01:58.155+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:01:58.156958+0000) 2024-03-04T15:01:59.158 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:01:59.156+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:01:59.157199+0000) 2024-03-04T15:02:00.159 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:02:00.156+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:02:00.157449+0000) 2024-03-04T15:02:01.158 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:02:01.155+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:02:01.157683+0000) 2024-03-04T15:02:02.158 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:02:02.156+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:02:02.157942+0000) 2024-03-04T15:02:03.159 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:02:03.157+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:02:03.158211+0000) 2024-03-04T15:02:04.159 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:02:04.156+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:02:04.158412+0000) 2024-03-04T15:02:05.159 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:02:05.157+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:02:05.158653+0000) 2024-03-04T15:02:06.159 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:02:06.157+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:02:06.158946+0000) 2024-03-04T15:02:07.160 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:02:07.158+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:02:07.159213+0000) 2024-03-04T15:02:08.160 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:02:08.158+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:02:08.159456+0000) 2024-03-04T15:02:09.160 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:02:09.158+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:02:09.159695+0000) 2024-03-04T15:02:10.162 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:02:10.159+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:02:10.159946+0000) 2024-03-04T15:02:11.161 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:02:11.159+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:02:11.160198+0000) 2024-03-04T15:02:12.161 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:02:12.159+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:02:12.160432+0000) 2024-03-04T15:02:13.013 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:02:13.026 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:02:13.161 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:02:13.159+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:02:13.160686+0000) 2024-03-04T15:02:14.161 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:02:14.160+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:02:14.160934+0000) 2024-03-04T15:02:15.161 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:02:15.160+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:02:15.161175+0000) 2024-03-04T15:02:16.162 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:02:16.160+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:02:16.161428+0000) 2024-03-04T15:02:17.162 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:02:17.160+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:02:17.161665+0000) 2024-03-04T15:02:18.162 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:02:18.161+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:02:18.161955+0000) 2024-03-04T15:02:19.162 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:02:19.161+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:02:19.162203+0000) 2024-03-04T15:02:20.163 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:02:20.161+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:02:20.162457+0000) 2024-03-04T15:02:21.163 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:02:21.161+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:02:21.162700+0000) 2024-03-04T15:02:22.163 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:02:22.162+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:02:22.162936+0000) 2024-03-04T15:02:23.163 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:02:23.162+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:02:23.163140+0000) 2024-03-04T15:02:24.164 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:02:24.162+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:02:24.163387+0000) 2024-03-04T15:02:25.164 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:02:25.162+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:02:25.163628+0000) 2024-03-04T15:02:26.164 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:02:26.163+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:02:26.163912+0000) 2024-03-04T15:02:27.164 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:02:27.163+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:02:27.164231+0000) 2024-03-04T15:02:28.165 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:02:28.163+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:02:28.164475+0000) 2024-03-04T15:02:29.165 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:02:29.164+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:02:29.164711+0000) 2024-03-04T15:02:30.165 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:02:30.164+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:02:30.164979+0000) 2024-03-04T15:02:31.165 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:02:31.164+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:02:31.165162+0000) 2024-03-04T15:02:32.166 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:02:32.164+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:02:32.165413+0000) 2024-03-04T15:02:33.166 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:02:33.165+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:02:33.165672+0000) 2024-03-04T15:02:34.166 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:02:34.165+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:02:34.165961+0000) 2024-03-04T15:02:35.166 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:02:35.165+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:02:35.166225+0000) 2024-03-04T15:02:36.167 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:02:36.165+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:02:36.166485+0000) 2024-03-04T15:02:37.169 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:02:37.166+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:02:37.166769+0000) 2024-03-04T15:02:38.170 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:02:38.166+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:02:38.167022+0000) 2024-03-04T15:02:39.168 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:02:39.166+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:02:39.167278+0000) 2024-03-04T15:02:40.168 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:02:40.166+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:02:40.167556+0000) 2024-03-04T15:02:41.168 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:02:41.167+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:02:41.167814+0000) 2024-03-04T15:02:42.169 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:02:42.167+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:02:42.168125+0000) 2024-03-04T15:02:43.153 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:02:43.163 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:02:43.172 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:02:43.167+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:02:43.168338+0000) 2024-03-04T15:02:44.169 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:02:44.167+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:02:44.168625+0000) 2024-03-04T15:02:45.169 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:02:45.168+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:02:45.168923+0000) 2024-03-04T15:02:46.169 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:02:46.168+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:02:46.169124+0000) 2024-03-04T15:02:47.170 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:02:47.168+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:02:47.169328+0000) 2024-03-04T15:02:48.170 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:02:48.168+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:02:48.169585+0000) 2024-03-04T15:02:49.171 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:02:49.169+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:02:49.169876+0000) 2024-03-04T15:02:50.171 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:02:50.169+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:02:50.170178+0000) 2024-03-04T15:02:51.171 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:02:51.169+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:02:51.170451+0000) 2024-03-04T15:02:52.171 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:02:52.170+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:02:52.170685+0000) 2024-03-04T15:02:53.171 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:02:53.170+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:02:53.170951+0000) 2024-03-04T15:02:54.172 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:02:54.170+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:02:54.171191+0000) 2024-03-04T15:02:55.172 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:02:55.171+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:02:55.171430+0000) 2024-03-04T15:02:56.172 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:02:56.171+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:02:56.171669+0000) 2024-03-04T15:02:57.172 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:02:57.171+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:02:57.171943+0000) 2024-03-04T15:02:58.172 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:02:58.171+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:02:58.172175+0000) 2024-03-04T15:02:59.173 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:02:59.172+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:02:59.172419+0000) 2024-03-04T15:03:00.173 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:03:00.171+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:03:00.172657+0000) 2024-03-04T15:03:01.173 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:03:01.172+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:03:01.172915+0000) 2024-03-04T15:03:02.173 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:03:02.173+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:03:02.173174+0000) 2024-03-04T15:03:03.174 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:03:03.172+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:03:03.173433+0000) 2024-03-04T15:03:04.174 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:03:04.172+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:03:04.173667+0000) 2024-03-04T15:03:05.174 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:03:05.172+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:03:05.173940+0000) 2024-03-04T15:03:06.175 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:03:06.173+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:03:06.174207+0000) 2024-03-04T15:03:07.175 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:03:07.173+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:03:07.174439+0000) 2024-03-04T15:03:08.175 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:03:08.173+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:03:08.174686+0000) 2024-03-04T15:03:09.175 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:03:09.173+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:03:09.174933+0000) 2024-03-04T15:03:10.176 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:03:10.174+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:03:10.175176+0000) 2024-03-04T15:03:11.176 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:03:11.174+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:03:11.175413+0000) 2024-03-04T15:03:12.176 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:03:12.174+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:03:12.175645+0000) 2024-03-04T15:03:13.176 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:03:13.174+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:03:13.175917+0000) 2024-03-04T15:03:13.340 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:03:13.350 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:03:14.177 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:03:14.175+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:03:14.176151+0000) 2024-03-04T15:03:15.177 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:03:15.175+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:03:15.176434+0000) 2024-03-04T15:03:16.177 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:03:16.175+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:03:16.176667+0000) 2024-03-04T15:03:17.177 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:03:17.176+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:03:17.176938+0000) 2024-03-04T15:03:18.178 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:03:18.176+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:03:18.177175+0000) 2024-03-04T15:03:19.178 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:03:19.176+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:03:19.177356+0000) 2024-03-04T15:03:20.178 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:03:20.176+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:03:20.177604+0000) 2024-03-04T15:03:21.178 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:03:21.177+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:03:21.177915+0000) 2024-03-04T15:03:22.178 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:03:22.177+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:03:22.178152+0000) 2024-03-04T15:03:23.178 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:03:23.177+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:03:23.178390+0000) 2024-03-04T15:03:24.179 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:03:24.177+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:03:24.178644+0000) 2024-03-04T15:03:25.179 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:03:25.178+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:03:25.178938+0000) 2024-03-04T15:03:26.180 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:03:26.178+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:03:26.179190+0000) 2024-03-04T15:03:27.180 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:03:27.178+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:03:27.179445+0000) 2024-03-04T15:03:28.180 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:03:28.178+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:03:28.179706+0000) 2024-03-04T15:03:29.180 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:03:29.179+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:03:29.179950+0000) 2024-03-04T15:03:30.180 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:03:30.179+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:03:30.180199+0000) 2024-03-04T15:03:31.181 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:03:31.179+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:03:31.180410+0000) 2024-03-04T15:03:32.181 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:03:32.179+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:03:32.180654+0000) 2024-03-04T15:03:33.181 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:03:33.180+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:03:33.180933+0000) 2024-03-04T15:03:34.182 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:03:34.180+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:03:34.181170+0000) 2024-03-04T15:03:35.182 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:03:35.180+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:03:35.181414+0000) 2024-03-04T15:03:36.182 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:03:36.180+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:03:36.181669+0000) 2024-03-04T15:03:37.182 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:03:37.181+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:03:37.181950+0000) 2024-03-04T15:03:38.182 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:03:38.181+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:03:38.182188+0000) 2024-03-04T15:03:39.183 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:03:39.181+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:03:39.182441+0000) 2024-03-04T15:03:40.183 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:03:40.182+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:03:40.182687+0000) 2024-03-04T15:03:41.183 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:03:41.182+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:03:41.182937+0000) 2024-03-04T15:03:42.184 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:03:42.182+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:03:42.183170+0000) 2024-03-04T15:03:43.184 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:03:43.182+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:03:43.183371+0000) 2024-03-04T15:03:43.561 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:03:43.569 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:03:44.184 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:03:44.183+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:03:44.183612+0000) 2024-03-04T15:03:45.184 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:03:45.183+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:03:45.183925+0000) 2024-03-04T15:03:46.184 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:03:46.183+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:03:46.184128+0000) 2024-03-04T15:03:47.185 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:03:47.183+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:03:47.184376+0000) 2024-03-04T15:03:48.185 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:03:48.183+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:03:48.184635+0000) 2024-03-04T15:03:49.185 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:03:49.184+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:03:49.184925+0000) 2024-03-04T15:03:50.185 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:03:50.184+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:03:50.185173+0000) 2024-03-04T15:03:51.186 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:03:51.184+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:03:51.185435+0000) 2024-03-04T15:03:52.186 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:03:52.184+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:03:52.185696+0000) 2024-03-04T15:03:53.186 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:03:53.185+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:03:53.185954+0000) 2024-03-04T15:03:54.187 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:03:54.185+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:03:54.186171+0000) 2024-03-04T15:03:55.188 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:03:55.186+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:03:55.186448+0000) 2024-03-04T15:03:56.187 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:03:56.186+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:03:56.186701+0000) 2024-03-04T15:03:57.187 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:03:57.186+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:03:57.186951+0000) 2024-03-04T15:03:58.188 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:03:58.186+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:03:58.187142+0000) 2024-03-04T15:03:59.188 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:03:59.187+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:03:59.187390+0000) 2024-03-04T15:04:00.188 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:04:00.186+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:04:00.187592+0000) 2024-03-04T15:04:01.189 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:04:01.186+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:04:01.187785+0000) 2024-03-04T15:04:02.188 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:04:02.187+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:04:02.188026+0000) 2024-03-04T15:04:03.189 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:04:03.188+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:04:03.188342+0000) 2024-03-04T15:04:04.189 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:04:04.188+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:04:04.188602+0000) 2024-03-04T15:04:05.189 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:04:05.188+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:04:05.188886+0000) 2024-03-04T15:04:06.190 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:04:06.188+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:04:06.189130+0000) 2024-03-04T15:04:07.190 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:04:07.189+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:04:07.189380+0000) 2024-03-04T15:04:08.190 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:04:08.188+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:04:08.189620+0000) 2024-03-04T15:04:09.191 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:04:09.188+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:04:09.189938+0000) 2024-03-04T15:04:10.191 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:04:10.189+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:04:10.190187+0000) 2024-03-04T15:04:11.191 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:04:11.189+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:04:11.190432+0000) 2024-03-04T15:04:12.191 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:04:12.188+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:04:12.190682+0000) 2024-03-04T15:04:13.191 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:04:13.189+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:04:13.190938+0000) 2024-03-04T15:04:13.732 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:04:13.742 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:04:14.191 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:04:14.190+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:04:14.191183+0000) 2024-03-04T15:04:15.192 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:04:15.190+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:04:15.191431+0000) 2024-03-04T15:04:16.192 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:04:16.190+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:04:16.191694+0000) 2024-03-04T15:04:17.192 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:04:17.190+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:04:17.191947+0000) 2024-03-04T15:04:18.193 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:04:18.191+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:04:18.192207+0000) 2024-03-04T15:04:19.193 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:04:19.191+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:04:19.192462+0000) 2024-03-04T15:04:20.193 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:04:20.191+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:04:20.192708+0000) 2024-03-04T15:04:21.193 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:04:21.192+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:04:21.192950+0000) 2024-03-04T15:04:22.193 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:04:22.192+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:04:22.193191+0000) 2024-03-04T15:04:23.194 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:04:23.192+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:04:23.193468+0000) 2024-03-04T15:04:24.194 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:04:24.192+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:04:24.193721+0000) 2024-03-04T15:04:25.194 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:04:25.193+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:04:25.193971+0000) 2024-03-04T15:04:26.195 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:04:26.193+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:04:26.194213+0000) 2024-03-04T15:04:27.195 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:04:27.193+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:04:27.194484+0000) 2024-03-04T15:04:28.195 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:04:28.193+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:04:28.194721+0000) 2024-03-04T15:04:29.196 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:04:29.194+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:04:29.194992+0000) 2024-03-04T15:04:30.195 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:04:30.194+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:04:30.195237+0000) 2024-03-04T15:04:31.196 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:04:31.194+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:04:31.195481+0000) 2024-03-04T15:04:32.196 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:04:32.194+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:04:32.195725+0000) 2024-03-04T15:04:33.198 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:04:33.195+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:04:33.195990+0000) 2024-03-04T15:04:34.197 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:04:34.195+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:04:34.196233+0000) 2024-03-04T15:04:35.197 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:04:35.195+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:04:35.196493+0000) 2024-03-04T15:04:36.197 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:04:36.196+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:04:36.196727+0000) 2024-03-04T15:04:37.197 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:04:37.196+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:04:37.196980+0000) 2024-03-04T15:04:38.198 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:04:38.196+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:04:38.197214+0000) 2024-03-04T15:04:39.198 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:04:39.196+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:04:39.197405+0000) 2024-03-04T15:04:40.198 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:04:40.196+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:04:40.197671+0000) 2024-03-04T15:04:41.198 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:04:41.197+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:04:41.197938+0000) 2024-03-04T15:04:42.198 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:04:42.197+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:04:42.198177+0000) 2024-03-04T15:04:43.199 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:04:43.197+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:04:43.198418+0000) 2024-03-04T15:04:43.954 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:04:43.962 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:04:44.199 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:04:44.197+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:04:44.198698+0000) 2024-03-04T15:04:45.199 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:04:45.198+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:04:45.198973+0000) 2024-03-04T15:04:46.200 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:04:46.198+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:04:46.199201+0000) 2024-03-04T15:04:47.200 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:04:47.198+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:04:47.199444+0000) 2024-03-04T15:04:48.200 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:04:48.199+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:04:48.199680+0000) 2024-03-04T15:04:49.201 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:04:49.199+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:04:49.199949+0000) 2024-03-04T15:04:50.201 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:04:50.199+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:04:50.200189+0000) 2024-03-04T15:04:51.201 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:04:51.199+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:04:51.200425+0000) 2024-03-04T15:04:52.201 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:04:52.199+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:04:52.200610+0000) 2024-03-04T15:04:53.201 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:04:53.200+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:04:53.200901+0000) 2024-03-04T15:04:54.201 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:04:54.200+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:04:54.201086+0000) 2024-03-04T15:04:55.201 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:04:55.200+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:04:55.201336+0000) 2024-03-04T15:04:56.202 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:04:56.201+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:04:56.201596+0000) 2024-03-04T15:04:57.202 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:04:57.201+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:04:57.201871+0000) 2024-03-04T15:04:58.202 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:04:58.201+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:04:58.202106+0000) 2024-03-04T15:04:59.203 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:04:59.202+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:04:59.202360+0000) 2024-03-04T15:05:00.203 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:05:00.202+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:05:00.202605+0000) 2024-03-04T15:05:01.203 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:05:01.202+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:05:01.202901+0000) 2024-03-04T15:05:02.203 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:05:02.202+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:05:02.203136+0000) 2024-03-04T15:05:03.204 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:05:03.203+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:05:03.203406+0000) 2024-03-04T15:05:04.204 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:05:04.203+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:05:04.203661+0000) 2024-03-04T15:05:05.205 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:05:05.203+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:05:05.203956+0000) 2024-03-04T15:05:06.205 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:05:06.203+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:05:06.204190+0000) 2024-03-04T15:05:07.205 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:05:07.204+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:05:07.204441+0000) 2024-03-04T15:05:08.205 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:05:08.204+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:05:08.204679+0000) 2024-03-04T15:05:09.206 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:05:09.204+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:05:09.204935+0000) 2024-03-04T15:05:10.206 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:05:10.204+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:05:10.205176+0000) 2024-03-04T15:05:11.206 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:05:11.205+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:05:11.205442+0000) 2024-03-04T15:05:12.206 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:05:12.205+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:05:12.205676+0000) 2024-03-04T15:05:13.206 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:05:13.205+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:05:13.205956+0000) 2024-03-04T15:05:14.151 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:05:14.160 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:05:14.206 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:05:14.205+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:05:14.206124+0000) 2024-03-04T15:05:15.207 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:05:15.205+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:05:15.206360+0000) 2024-03-04T15:05:16.207 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:05:16.205+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:05:16.206600+0000) 2024-03-04T15:05:17.207 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:05:17.205+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:05:17.206915+0000) 2024-03-04T15:05:18.208 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:05:18.206+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:05:18.207158+0000) 2024-03-04T15:05:19.208 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:05:19.206+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:05:19.207413+0000) 2024-03-04T15:05:20.208 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:05:20.206+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:05:20.207608+0000) 2024-03-04T15:05:21.208 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:05:21.206+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:05:21.207908+0000) 2024-03-04T15:05:22.208 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:05:22.207+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:05:22.208144+0000) 2024-03-04T15:05:23.209 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:05:23.207+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:05:23.208406+0000) 2024-03-04T15:05:24.209 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:05:24.207+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:05:24.208585+0000) 2024-03-04T15:05:25.209 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:05:25.207+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:05:25.208887+0000) 2024-03-04T15:05:26.210 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:05:26.208+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:05:26.209122+0000) 2024-03-04T15:05:27.210 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:05:27.208+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:05:27.209376+0000) 2024-03-04T15:05:28.210 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:05:28.208+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:05:28.209618+0000) 2024-03-04T15:05:29.210 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:05:29.209+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:05:29.209920+0000) 2024-03-04T15:05:30.211 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:05:30.209+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:05:30.210126+0000) 2024-03-04T15:05:31.211 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:05:31.209+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:05:31.210383+0000) 2024-03-04T15:05:32.211 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:05:32.209+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:05:32.210616+0000) 2024-03-04T15:05:33.211 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:05:33.210+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:05:33.210933+0000) 2024-03-04T15:05:34.212 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:05:34.210+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:05:34.211124+0000) 2024-03-04T15:05:35.212 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:05:35.210+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:05:35.211398+0000) 2024-03-04T15:05:36.212 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:05:36.210+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:05:36.211641+0000) 2024-03-04T15:05:37.212 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:05:37.211+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:05:37.211931+0000) 2024-03-04T15:05:38.212 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:05:38.211+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:05:38.212108+0000) 2024-03-04T15:05:39.213 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:05:39.211+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:05:39.212359+0000) 2024-03-04T15:05:40.213 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:05:40.211+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:05:40.212610+0000) 2024-03-04T15:05:41.213 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:05:41.212+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:05:41.212897+0000) 2024-03-04T15:05:42.213 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:05:42.212+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:05:42.213150+0000) 2024-03-04T15:05:43.214 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:05:43.212+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:05:43.213420+0000) 2024-03-04T15:05:44.214 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:05:44.212+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:05:44.213695+0000) 2024-03-04T15:05:44.323 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:05:44.331 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:05:45.214 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:05:45.213+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:05:45.213952+0000) 2024-03-04T15:05:46.215 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:05:46.213+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:05:46.214199+0000) 2024-03-04T15:05:47.215 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:05:47.213+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:05:47.214463+0000) 2024-03-04T15:05:48.215 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:05:48.214+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:05:48.214752+0000) 2024-03-04T15:05:49.215 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:05:49.214+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:05:49.215050+0000) 2024-03-04T15:05:50.216 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:05:50.214+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:05:50.215304+0000) 2024-03-04T15:05:51.216 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:05:51.214+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:05:51.215552+0000) 2024-03-04T15:05:52.216 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:05:52.215+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:05:52.215789+0000) 2024-03-04T15:05:53.217 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:05:53.215+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:05:53.216074+0000) 2024-03-04T15:05:54.217 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:05:54.215+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:05:54.216318+0000) 2024-03-04T15:05:55.217 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:05:55.216+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:05:55.216572+0000) 2024-03-04T15:05:56.217 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:05:56.216+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:05:56.216807+0000) 2024-03-04T15:05:57.218 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:05:57.216+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:05:57.217090+0000) 2024-03-04T15:05:58.218 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:05:58.216+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:05:58.217322+0000) 2024-03-04T15:05:59.218 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:05:59.217+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:05:59.217580+0000) 2024-03-04T15:06:00.218 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:06:00.217+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:06:00.217880+0000) 2024-03-04T15:06:01.219 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:06:01.217+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:06:01.218126+0000) 2024-03-04T15:06:02.219 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:06:02.217+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:06:02.218304+0000) 2024-03-04T15:06:03.219 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:06:03.218+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:06:03.218567+0000) 2024-03-04T15:06:04.219 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:06:04.218+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:06:04.218860+0000) 2024-03-04T15:06:05.220 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:06:05.218+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:06:05.219129+0000) 2024-03-04T15:06:06.220 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:06:06.219+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:06:06.219370+0000) 2024-03-04T15:06:07.220 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:06:07.218+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:06:07.219612+0000) 2024-03-04T15:06:08.220 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:06:08.219+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:06:08.219884+0000) 2024-03-04T15:06:09.221 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:06:09.219+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:06:09.220135+0000) 2024-03-04T15:06:10.221 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:06:10.219+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:06:10.220399+0000) 2024-03-04T15:06:11.221 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:06:11.220+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:06:11.220644+0000) 2024-03-04T15:06:12.221 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:06:12.220+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:06:12.220915+0000) 2024-03-04T15:06:13.222 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:06:13.220+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:06:13.221159+0000) 2024-03-04T15:06:14.222 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:06:14.220+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:06:14.221424+0000) 2024-03-04T15:06:14.478 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:06:14.489 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:06:15.222 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:06:15.221+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:06:15.221700+0000) 2024-03-04T15:06:16.222 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:06:16.221+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:06:16.221930+0000) 2024-03-04T15:06:17.226 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:06:17.221+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:06:17.222192+0000) 2024-03-04T15:06:18.226 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:06:18.222+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:06:18.222451+0000) 2024-03-04T15:06:19.223 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:06:19.221+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:06:19.222736+0000) 2024-03-04T15:06:20.225 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:06:20.222+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:06:20.223002+0000) 2024-03-04T15:06:21.224 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:06:21.223+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:06:21.223281+0000) 2024-03-04T15:06:22.224 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:06:22.222+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:06:22.223493+0000) 2024-03-04T15:06:23.224 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:06:23.221+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:06:23.223761+0000) 2024-03-04T15:06:24.224 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:06:24.222+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:06:24.224034+0000) 2024-03-04T15:06:25.225 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:06:25.223+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:06:25.224298+0000) 2024-03-04T15:06:26.225 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:06:26.222+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:06:26.224585+0000) 2024-03-04T15:06:27.225 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:06:27.223+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:06:27.224881+0000) 2024-03-04T15:06:28.226 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:06:28.224+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:06:28.225112+0000) 2024-03-04T15:06:29.226 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:06:29.223+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:06:29.225356+0000) 2024-03-04T15:06:30.226 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:06:30.224+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:06:30.225631+0000) 2024-03-04T15:06:31.227 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:06:31.224+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:06:31.225935+0000) 2024-03-04T15:06:32.227 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:06:32.224+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:06:32.226166+0000) 2024-03-04T15:06:33.227 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:06:33.224+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:06:33.226433+0000) 2024-03-04T15:06:34.227 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:06:34.225+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:06:34.226684+0000) 2024-03-04T15:06:35.228 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:06:35.226+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:06:35.226940+0000) 2024-03-04T15:06:36.228 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:06:36.226+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:06:36.227195+0000) 2024-03-04T15:06:37.228 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:06:37.226+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:06:37.227528+0000) 2024-03-04T15:06:38.228 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:06:38.226+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:06:38.227783+0000) 2024-03-04T15:06:39.228 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:06:39.227+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:06:39.228093+0000) 2024-03-04T15:06:40.229 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:06:40.227+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:06:40.228376+0000) 2024-03-04T15:06:41.229 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:06:41.227+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:06:41.228657+0000) 2024-03-04T15:06:42.229 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:06:42.228+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:06:42.228946+0000) 2024-03-04T15:06:43.230 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:06:43.228+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:06:43.229206+0000) 2024-03-04T15:06:44.230 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:06:44.228+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:06:44.229462+0000) 2024-03-04T15:06:44.708 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:06:44.716 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:06:45.230 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:06:45.228+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:06:45.229656+0000) 2024-03-04T15:06:46.230 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:06:46.229+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:06:46.229949+0000) 2024-03-04T15:06:47.231 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:06:47.229+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:06:47.230198+0000) 2024-03-04T15:06:48.231 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:06:48.229+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:06:48.230440+0000) 2024-03-04T15:06:49.231 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:06:49.229+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:06:49.230691+0000) 2024-03-04T15:06:50.233 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:06:50.230+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:06:50.230949+0000) 2024-03-04T15:06:51.232 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:06:51.230+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:06:51.231193+0000) 2024-03-04T15:06:52.232 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:06:52.230+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:06:52.231430+0000) 2024-03-04T15:06:53.232 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:06:53.230+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:06:53.231691+0000) 2024-03-04T15:06:54.232 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:06:54.231+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:06:54.231937+0000) 2024-03-04T15:06:55.233 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:06:55.231+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:06:55.232187+0000) 2024-03-04T15:06:56.233 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:06:56.231+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:06:56.232440+0000) 2024-03-04T15:06:57.233 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:06:57.232+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:06:57.232683+0000) 2024-03-04T15:06:58.233 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:06:58.232+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:06:58.232954+0000) 2024-03-04T15:06:59.234 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:06:59.232+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:06:59.233208+0000) 2024-03-04T15:07:00.234 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:07:00.232+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:07:00.233459+0000) 2024-03-04T15:07:01.234 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:07:01.232+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:07:01.233698+0000) 2024-03-04T15:07:02.234 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:07:02.233+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:07:02.233950+0000) 2024-03-04T15:07:03.235 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:07:03.233+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:07:03.234196+0000) 2024-03-04T15:07:04.235 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:07:04.233+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:07:04.234444+0000) 2024-03-04T15:07:05.235 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:07:05.234+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:07:05.234690+0000) 2024-03-04T15:07:06.235 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:07:06.234+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:07:06.234925+0000) 2024-03-04T15:07:07.236 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:07:07.234+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:07:07.235127+0000) 2024-03-04T15:07:08.236 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:07:08.234+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:07:08.235368+0000) 2024-03-04T15:07:09.236 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:07:09.235+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:07:09.235614+0000) 2024-03-04T15:07:10.236 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:07:10.235+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:07:10.235929+0000) 2024-03-04T15:07:11.237 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:07:11.235+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:07:11.236182+0000) 2024-03-04T15:07:12.237 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:07:12.236+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:07:12.236425+0000) 2024-03-04T15:07:13.237 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:07:13.235+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:07:13.236686+0000) 2024-03-04T15:07:14.238 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:07:14.236+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:07:14.236982+0000) 2024-03-04T15:07:14.914 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:07:14.922 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:07:15.238 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:07:15.236+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:07:15.237286+0000) 2024-03-04T15:07:16.238 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:07:16.237+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:07:16.237575+0000) 2024-03-04T15:07:17.239 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:07:17.237+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:07:17.237918+0000) 2024-03-04T15:07:18.239 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:07:18.237+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:07:18.238212+0000) 2024-03-04T15:07:19.239 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:07:19.238+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:07:19.238504+0000) 2024-03-04T15:07:20.239 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:07:20.238+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:07:20.238808+0000) 2024-03-04T15:07:21.240 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:07:21.238+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:07:21.239159+0000) 2024-03-04T15:07:22.240 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:07:22.239+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:07:22.239472+0000) 2024-03-04T15:07:23.240 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:07:23.238+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:07:23.239769+0000) 2024-03-04T15:07:24.241 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:07:24.239+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:07:24.240066+0000) 2024-03-04T15:07:25.241 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:07:25.240+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:07:25.240422+0000) 2024-03-04T15:07:26.241 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:07:26.239+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:07:26.240735+0000) 2024-03-04T15:07:27.242 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:07:27.240+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:07:27.241007+0000) 2024-03-04T15:07:28.242 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:07:28.240+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:07:28.241218+0000) 2024-03-04T15:07:29.242 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:07:29.240+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:07:29.241458+0000) 2024-03-04T15:07:30.242 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:07:30.240+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:07:30.241726+0000) 2024-03-04T15:07:31.242 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:07:31.240+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:07:31.241979+0000) 2024-03-04T15:07:32.243 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:07:32.241+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:07:32.242212+0000) 2024-03-04T15:07:33.243 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:07:33.241+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:07:33.242458+0000) 2024-03-04T15:07:34.243 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:07:34.241+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:07:34.242696+0000) 2024-03-04T15:07:35.243 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:07:35.241+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:07:35.242950+0000) 2024-03-04T15:07:36.244 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:07:36.242+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:07:36.243199+0000) 2024-03-04T15:07:37.244 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:07:37.241+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:07:37.243444+0000) 2024-03-04T15:07:38.244 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:07:38.242+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:07:38.243708+0000) 2024-03-04T15:07:39.244 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:07:39.242+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:07:39.243953+0000) 2024-03-04T15:07:40.244 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:07:40.243+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:07:40.244204+0000) 2024-03-04T15:07:41.245 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:07:41.243+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:07:41.244473+0000) 2024-03-04T15:07:42.245 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:07:42.243+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:07:42.244714+0000) 2024-03-04T15:07:43.245 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:07:43.244+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:07:43.244982+0000) 2024-03-04T15:07:44.246 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:07:44.244+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:07:44.245223+0000) 2024-03-04T15:07:45.119 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:07:45.127 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:07:45.245 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:07:45.244+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:07:45.245429+0000) 2024-03-04T15:07:46.246 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:07:46.244+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:07:46.245615+0000) 2024-03-04T15:07:47.246 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:07:47.245+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:07:47.245903+0000) 2024-03-04T15:07:48.246 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:07:48.245+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:07:48.246140+0000) 2024-03-04T15:07:49.247 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:07:49.245+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:07:49.246386+0000) 2024-03-04T15:07:50.247 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:07:50.245+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:07:50.246636+0000) 2024-03-04T15:07:51.247 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:07:51.246+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:07:51.246945+0000) 2024-03-04T15:07:52.247 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:07:52.246+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:07:52.247186+0000) 2024-03-04T15:07:53.248 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:07:53.246+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:07:53.247430+0000) 2024-03-04T15:07:54.248 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:07:54.246+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:07:54.247664+0000) 2024-03-04T15:07:55.248 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:07:55.247+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:07:55.247944+0000) 2024-03-04T15:07:56.248 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:07:56.247+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:07:56.248183+0000) 2024-03-04T15:07:57.249 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:07:57.247+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:07:57.248433+0000) 2024-03-04T15:07:58.249 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:07:58.247+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:07:58.248675+0000) 2024-03-04T15:07:59.249 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:07:59.248+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:07:59.248939+0000) 2024-03-04T15:08:00.249 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:08:00.248+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:08:00.249215+0000) 2024-03-04T15:08:01.250 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:08:01.248+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:08:01.249464+0000) 2024-03-04T15:08:02.250 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:08:02.249+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:08:02.249705+0000) 2024-03-04T15:08:03.250 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:08:03.249+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:08:03.249953+0000) 2024-03-04T15:08:04.251 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:08:04.249+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:08:04.250192+0000) 2024-03-04T15:08:05.251 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:08:05.249+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:08:05.250442+0000) 2024-03-04T15:08:06.251 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:08:06.250+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:08:06.250683+0000) 2024-03-04T15:08:07.251 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:08:07.250+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:08:07.250936+0000) 2024-03-04T15:08:08.251 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:08:08.250+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:08:08.251176+0000) 2024-03-04T15:08:09.252 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:08:09.250+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:08:09.251438+0000) 2024-03-04T15:08:10.252 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:08:10.251+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:08:10.251685+0000) 2024-03-04T15:08:11.252 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:08:11.251+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:08:11.251939+0000) 2024-03-04T15:08:12.252 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:08:12.251+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:08:12.252175+0000) 2024-03-04T15:08:13.253 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:08:13.251+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:08:13.252473+0000) 2024-03-04T15:08:14.253 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:08:14.251+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:08:14.252728+0000) 2024-03-04T15:08:15.253 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:08:15.252+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:08:15.252976+0000) 2024-03-04T15:08:15.256 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:08:15.264 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:08:16.253 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:08:16.252+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:08:16.253216+0000) 2024-03-04T15:08:17.254 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:08:17.253+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:08:17.253459+0000) 2024-03-04T15:08:18.254 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:08:18.253+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:08:18.253699+0000) 2024-03-04T15:08:19.254 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:08:19.253+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:08:19.253950+0000) 2024-03-04T15:08:20.255 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:08:20.253+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:08:20.254201+0000) 2024-03-04T15:08:21.255 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:08:21.254+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:08:21.254454+0000) 2024-03-04T15:08:22.255 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:08:22.254+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:08:22.254692+0000) 2024-03-04T15:08:23.255 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:08:23.254+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:08:23.254945+0000) 2024-03-04T15:08:24.255 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:08:24.254+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:08:24.255183+0000) 2024-03-04T15:08:25.256 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:08:25.255+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:08:25.255430+0000) 2024-03-04T15:08:26.256 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:08:26.254+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:08:26.255624+0000) 2024-03-04T15:08:27.256 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:08:27.255+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:08:27.255943+0000) 2024-03-04T15:08:28.257 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:08:28.255+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:08:28.256185+0000) 2024-03-04T15:08:29.257 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:08:29.255+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:08:29.256473+0000) 2024-03-04T15:08:30.257 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:08:30.255+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:08:30.256748+0000) 2024-03-04T15:08:31.257 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:08:31.256+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:08:31.257008+0000) 2024-03-04T15:08:32.257 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:08:32.257+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:08:32.257261+0000) 2024-03-04T15:08:33.258 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:08:33.256+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:08:33.257444+0000) 2024-03-04T15:08:34.258 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:08:34.257+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:08:34.257676+0000) 2024-03-04T15:08:35.258 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:08:35.257+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:08:35.257961+0000) 2024-03-04T15:08:36.258 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:08:36.258+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:08:36.258196+0000) 2024-03-04T15:08:37.259 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:08:37.258+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:08:37.258426+0000) 2024-03-04T15:08:38.259 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:08:38.256+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:08:38.258677+0000) 2024-03-04T15:08:39.259 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:08:39.257+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:08:39.258924+0000) 2024-03-04T15:08:40.259 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:08:40.258+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:08:40.259174+0000) 2024-03-04T15:08:41.260 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:08:41.258+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:08:41.259414+0000) 2024-03-04T15:08:42.260 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:08:42.257+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:08:42.259662+0000) 2024-03-04T15:08:43.260 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:08:43.258+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:08:43.259950+0000) 2024-03-04T15:08:44.260 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:08:44.259+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:08:44.260184+0000) 2024-03-04T15:08:45.261 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:08:45.259+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:08:45.260426+0000) 2024-03-04T15:08:45.401 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:08:45.409 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:08:46.261 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:08:46.259+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:08:46.260666+0000) 2024-03-04T15:08:47.261 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:08:47.259+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:08:47.260944+0000) 2024-03-04T15:08:48.262 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:08:48.260+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:08:48.261179+0000) 2024-03-04T15:08:49.262 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:08:49.260+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:08:49.261420+0000) 2024-03-04T15:08:50.262 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:08:50.260+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:08:50.261667+0000) 2024-03-04T15:08:51.262 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:08:51.261+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:08:51.261950+0000) 2024-03-04T15:08:52.262 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:08:52.261+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:08:52.262190+0000) 2024-03-04T15:08:53.263 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:08:53.261+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:08:53.262423+0000) 2024-03-04T15:08:54.263 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:08:54.261+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:08:54.262657+0000) 2024-03-04T15:08:55.263 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:08:55.262+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:08:55.262942+0000) 2024-03-04T15:08:56.263 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:08:56.262+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:08:56.263180+0000) 2024-03-04T15:08:57.264 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:08:57.262+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:08:57.263422+0000) 2024-03-04T15:08:58.264 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:08:58.262+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:08:58.263654+0000) 2024-03-04T15:08:59.264 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:08:59.263+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:08:59.263930+0000) 2024-03-04T15:09:00.264 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:09:00.263+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:09:00.264192+0000) 2024-03-04T15:09:01.265 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:09:01.263+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:09:01.264507+0000) 2024-03-04T15:09:02.265 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:09:02.263+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:09:02.264738+0000) 2024-03-04T15:09:03.265 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:09:03.264+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:09:03.264990+0000) 2024-03-04T15:09:04.265 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:09:04.264+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:09:04.265222+0000) 2024-03-04T15:09:05.266 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:09:05.264+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:09:05.265466+0000) 2024-03-04T15:09:06.266 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:09:06.264+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:09:06.265708+0000) 2024-03-04T15:09:07.266 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:09:07.265+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:09:07.265956+0000) 2024-03-04T15:09:08.266 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:09:08.265+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:09:08.266199+0000) 2024-03-04T15:09:09.267 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:09:09.265+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:09:09.266443+0000) 2024-03-04T15:09:10.267 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:09:10.266+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:09:10.266695+0000) 2024-03-04T15:09:11.267 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:09:11.266+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:09:11.266931+0000) 2024-03-04T15:09:12.267 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:09:12.266+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:09:12.267173+0000) 2024-03-04T15:09:13.268 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:09:13.266+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:09:13.267417+0000) 2024-03-04T15:09:14.268 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:09:14.267+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:09:14.267655+0000) 2024-03-04T15:09:15.268 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:09:15.267+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:09:15.267945+0000) 2024-03-04T15:09:15.617 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:09:15.625 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:09:16.268 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:09:16.267+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:09:16.268189+0000) 2024-03-04T15:09:17.269 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:09:17.267+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:09:17.268456+0000) 2024-03-04T15:09:18.269 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:09:18.268+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:09:18.268701+0000) 2024-03-04T15:09:19.269 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:09:19.268+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:09:19.268945+0000) 2024-03-04T15:09:20.269 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:09:20.268+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:09:20.269191+0000) 2024-03-04T15:09:21.270 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:09:21.268+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:09:21.269436+0000) 2024-03-04T15:09:22.270 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:09:22.268+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:09:22.269693+0000) 2024-03-04T15:09:23.270 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:09:23.269+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:09:23.269939+0000) 2024-03-04T15:09:24.270 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:09:24.269+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:09:24.270179+0000) 2024-03-04T15:09:25.271 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:09:25.269+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:09:25.270378+0000) 2024-03-04T15:09:26.271 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:09:26.270+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:09:26.270619+0000) 2024-03-04T15:09:27.271 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:09:27.270+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:09:27.270891+0000) 2024-03-04T15:09:28.271 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:09:28.270+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:09:28.271133+0000) 2024-03-04T15:09:29.272 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:09:29.271+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:09:29.271374+0000) 2024-03-04T15:09:30.272 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:09:30.271+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:09:30.271649+0000) 2024-03-04T15:09:31.272 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:09:31.271+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:09:31.271949+0000) 2024-03-04T15:09:32.272 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:09:32.271+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:09:32.272191+0000) 2024-03-04T15:09:33.273 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:09:33.271+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:09:33.272469+0000) 2024-03-04T15:09:34.273 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:09:34.272+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:09:34.272703+0000) 2024-03-04T15:09:35.273 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:09:35.272+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:09:35.272954+0000) 2024-03-04T15:09:36.273 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:09:36.272+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:09:36.273195+0000) 2024-03-04T15:09:37.274 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:09:37.273+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:09:37.273442+0000) 2024-03-04T15:09:38.274 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:09:38.272+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:09:38.273703+0000) 2024-03-04T15:09:39.274 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:09:39.273+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:09:39.274008+0000) 2024-03-04T15:09:40.274 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:09:40.274+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:09:40.274278+0000) 2024-03-04T15:09:41.275 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:09:41.273+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:09:41.274540+0000) 2024-03-04T15:09:42.275 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:09:42.274+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:09:42.274775+0000) 2024-03-04T15:09:43.275 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:09:43.274+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:09:43.275019+0000) 2024-03-04T15:09:44.275 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:09:44.275+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:09:44.275262+0000) 2024-03-04T15:09:45.276 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:09:45.275+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:09:45.275543+0000) 2024-03-04T15:09:45.763 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:09:45.771 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:09:46.276 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:09:46.274+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:09:46.275781+0000) 2024-03-04T15:09:47.276 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:09:47.274+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:09:47.276028+0000) 2024-03-04T15:09:48.276 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:09:48.275+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:09:48.276269+0000) 2024-03-04T15:09:49.277 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:09:49.275+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:09:49.276512+0000) 2024-03-04T15:09:50.277 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:09:50.275+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:09:50.276766+0000) 2024-03-04T15:09:51.277 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:09:51.275+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:09:51.277017+0000) 2024-03-04T15:09:52.277 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:09:52.276+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:09:52.277196+0000) 2024-03-04T15:09:53.278 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:09:53.276+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:09:53.277452+0000) 2024-03-04T15:09:54.278 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:09:54.276+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:09:54.277690+0000) 2024-03-04T15:09:55.278 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:09:55.276+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:09:55.277994+0000) 2024-03-04T15:09:56.278 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:09:56.277+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:09:56.278235+0000) 2024-03-04T15:09:57.279 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:09:57.277+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:09:57.278419+0000) 2024-03-04T15:09:58.279 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:09:58.277+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:09:58.278668+0000) 2024-03-04T15:09:59.279 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:09:59.278+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:09:59.278935+0000) 2024-03-04T15:10:00.279 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:10:00.278+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:10:00.279185+0000) 2024-03-04T15:10:01.280 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:10:01.278+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:10:01.279452+0000) 2024-03-04T15:10:02.280 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:10:02.278+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:10:02.279664+0000) 2024-03-04T15:10:03.280 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:10:03.279+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:10:03.279951+0000) 2024-03-04T15:10:04.280 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:10:04.279+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:10:04.280188+0000) 2024-03-04T15:10:05.281 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:10:05.279+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:10:05.280434+0000) 2024-03-04T15:10:06.281 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:10:06.279+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:10:06.280620+0000) 2024-03-04T15:10:07.281 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:10:07.280+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:10:07.280913+0000) 2024-03-04T15:10:08.282 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:10:08.280+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:10:08.281132+0000) 2024-03-04T15:10:09.282 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:10:09.280+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:10:09.281331+0000) 2024-03-04T15:10:10.282 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:10:10.280+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:10:10.281529+0000) 2024-03-04T15:10:11.282 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:10:11.281+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:10:11.281797+0000) 2024-03-04T15:10:12.282 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:10:12.281+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:10:12.282115+0000) 2024-03-04T15:10:13.283 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:10:13.281+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:10:13.282374+0000) 2024-03-04T15:10:14.283 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:10:14.281+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:10:14.282569+0000) 2024-03-04T15:10:15.283 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:10:15.282+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:10:15.282863+0000) 2024-03-04T15:10:15.976 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:10:15.987 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:10:16.283 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:10:16.282+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:10:16.283122+0000) 2024-03-04T15:10:17.284 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:10:17.282+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:10:17.283397+0000) 2024-03-04T15:10:18.284 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:10:18.282+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:10:18.283641+0000) 2024-03-04T15:10:19.284 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:10:19.283+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:10:19.283954+0000) 2024-03-04T15:10:20.284 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:10:20.283+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:10:20.284195+0000) 2024-03-04T15:10:21.285 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:10:21.283+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:10:21.284404+0000) 2024-03-04T15:10:22.285 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:10:22.283+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:10:22.284651+0000) 2024-03-04T15:10:23.285 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:10:23.284+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:10:23.284934+0000) 2024-03-04T15:10:24.285 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:10:24.284+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:10:24.285171+0000) 2024-03-04T15:10:25.286 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:10:25.284+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:10:25.285419+0000) 2024-03-04T15:10:26.286 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:10:26.285+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:10:26.285654+0000) 2024-03-04T15:10:27.286 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:10:27.285+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:10:27.285953+0000) 2024-03-04T15:10:28.286 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:10:28.285+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:10:28.286191+0000) 2024-03-04T15:10:29.287 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:10:29.285+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:10:29.286426+0000) 2024-03-04T15:10:30.287 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:10:30.286+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:10:30.286673+0000) 2024-03-04T15:10:31.287 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:10:31.286+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:10:31.286941+0000) 2024-03-04T15:10:32.287 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:10:32.286+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:10:32.287129+0000) 2024-03-04T15:10:33.288 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:10:33.286+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:10:33.287404+0000) 2024-03-04T15:10:34.288 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:10:34.287+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:10:34.287636+0000) 2024-03-04T15:10:35.288 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:10:35.287+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:10:35.287936+0000) 2024-03-04T15:10:36.288 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:10:36.287+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:10:36.288193+0000) 2024-03-04T15:10:37.289 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:10:37.287+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:10:37.288451+0000) 2024-03-04T15:10:38.289 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:10:38.288+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:10:38.288687+0000) 2024-03-04T15:10:39.289 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:10:39.288+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:10:39.288966+0000) 2024-03-04T15:10:40.290 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:10:40.288+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:10:40.289200+0000) 2024-03-04T15:10:41.290 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:10:41.289+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:10:41.289444+0000) 2024-03-04T15:10:42.290 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:10:42.289+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:10:42.289712+0000) 2024-03-04T15:10:43.290 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:10:43.289+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:10:43.290007+0000) 2024-03-04T15:10:44.290 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:10:44.289+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:10:44.290232+0000) 2024-03-04T15:10:45.291 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:10:45.290+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:10:45.290482+0000) 2024-03-04T15:10:46.189 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:10:46.198 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:10:46.291 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:10:46.289+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:10:46.290726+0000) 2024-03-04T15:10:47.291 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:10:47.290+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:10:47.291017+0000) 2024-03-04T15:10:48.291 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:10:48.291+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:10:48.291308+0000) 2024-03-04T15:10:49.292 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:10:49.291+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:10:49.291581+0000) 2024-03-04T15:10:50.292 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:10:50.291+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:10:50.291869+0000) 2024-03-04T15:10:51.292 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:10:51.291+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:10:51.292118+0000) 2024-03-04T15:10:52.293 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:10:52.292+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:10:52.292354+0000) 2024-03-04T15:10:53.293 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:10:53.292+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:10:53.292609+0000) 2024-03-04T15:10:54.293 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:10:54.292+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:10:54.292887+0000) 2024-03-04T15:10:55.293 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:10:55.292+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:10:55.293137+0000) 2024-03-04T15:10:56.293 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:10:56.292+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:10:56.293384+0000) 2024-03-04T15:10:57.294 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:10:57.292+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:10:57.293633+0000) 2024-03-04T15:10:58.294 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:10:58.292+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:10:58.293936+0000) 2024-03-04T15:10:59.294 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:10:59.292+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:10:59.294178+0000) 2024-03-04T15:11:00.295 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:11:00.293+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:11:00.294442+0000) 2024-03-04T15:11:01.295 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:11:01.293+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:11:01.294707+0000) 2024-03-04T15:11:02.295 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:11:02.293+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:11:02.294930+0000) 2024-03-04T15:11:03.295 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:11:03.294+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:11:03.295182+0000) 2024-03-04T15:11:04.296 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:11:04.294+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:11:04.295365+0000) 2024-03-04T15:11:05.296 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:11:05.294+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:11:05.295612+0000) 2024-03-04T15:11:06.296 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:11:06.294+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:11:06.295913+0000) 2024-03-04T15:11:07.296 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:11:07.295+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:11:07.296162+0000) 2024-03-04T15:11:08.296 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:11:08.295+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:11:08.296404+0000) 2024-03-04T15:11:09.297 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:11:09.295+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:11:09.296648+0000) 2024-03-04T15:11:10.297 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:11:10.296+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:11:10.296938+0000) 2024-03-04T15:11:11.297 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:11:11.296+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:11:11.297150+0000) 2024-03-04T15:11:12.297 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:11:12.296+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:11:12.297387+0000) 2024-03-04T15:11:13.298 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:11:13.296+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:11:13.297587+0000) 2024-03-04T15:11:14.298 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:11:14.297+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:11:14.297867+0000) 2024-03-04T15:11:15.298 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:11:15.297+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:11:15.298125+0000) 2024-03-04T15:11:16.299 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:11:16.297+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:11:16.298314+0000) 2024-03-04T15:11:16.337 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:11:16.345 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:11:17.299 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:11:17.297+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:11:17.298563+0000) 2024-03-04T15:11:18.299 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:11:18.298+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:11:18.298817+0000) 2024-03-04T15:11:19.299 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:11:19.298+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:11:19.299101+0000) 2024-03-04T15:11:20.300 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:11:20.298+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:11:20.299380+0000) 2024-03-04T15:11:21.300 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:11:21.298+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:11:21.299626+0000) 2024-03-04T15:11:22.300 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:11:22.299+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:11:22.299933+0000) 2024-03-04T15:11:23.300 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:11:23.299+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:11:23.300181+0000) 2024-03-04T15:11:24.301 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:11:24.299+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:11:24.300447+0000) 2024-03-04T15:11:25.301 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:11:25.300+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:11:25.300697+0000) 2024-03-04T15:11:26.301 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:11:26.300+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:11:26.300961+0000) 2024-03-04T15:11:27.302 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:11:27.300+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:11:27.301211+0000) 2024-03-04T15:11:28.302 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:11:28.300+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:11:28.301454+0000) 2024-03-04T15:11:29.302 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:11:29.301+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:11:29.301694+0000) 2024-03-04T15:11:30.302 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:11:30.301+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:11:30.301945+0000) 2024-03-04T15:11:31.302 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:11:31.301+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:11:31.302260+0000) 2024-03-04T15:11:32.303 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:11:32.301+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:11:32.302505+0000) 2024-03-04T15:11:33.303 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:11:33.302+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:11:33.302751+0000) 2024-03-04T15:11:34.303 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:11:34.302+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:11:34.302985+0000) 2024-03-04T15:11:35.304 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:11:35.302+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:11:35.303192+0000) 2024-03-04T15:11:36.304 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:11:36.302+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:11:36.303383+0000) 2024-03-04T15:11:37.304 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:11:37.302+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:11:37.303621+0000) 2024-03-04T15:11:38.304 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:11:38.303+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:11:38.303924+0000) 2024-03-04T15:11:39.304 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:11:39.303+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:11:39.304169+0000) 2024-03-04T15:11:40.305 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:11:40.303+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:11:40.304448+0000) 2024-03-04T15:11:41.305 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:11:41.304+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:11:41.304703+0000) 2024-03-04T15:11:42.305 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:11:42.304+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:11:42.304942+0000) 2024-03-04T15:11:43.305 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:11:43.304+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:11:43.305132+0000) 2024-03-04T15:11:44.306 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:11:44.304+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:11:44.305395+0000) 2024-03-04T15:11:45.306 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:11:45.305+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:11:45.305639+0000) 2024-03-04T15:11:46.306 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:11:46.305+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:11:46.305947+0000) 2024-03-04T15:11:46.550 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:11:46.558 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:11:47.306 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:11:47.305+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:11:47.306230+0000) 2024-03-04T15:11:48.307 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:11:48.305+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:11:48.306492+0000) 2024-03-04T15:11:49.307 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:11:49.305+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:11:49.306692+0000) 2024-03-04T15:11:50.307 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:11:50.306+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:11:50.306990+0000) 2024-03-04T15:11:51.307 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:11:51.306+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:11:51.307206+0000) 2024-03-04T15:11:52.308 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:11:52.306+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:11:52.307475+0000) 2024-03-04T15:11:53.308 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:11:53.307+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:11:53.307770+0000) 2024-03-04T15:11:54.308 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:11:54.307+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:11:54.308070+0000) 2024-03-04T15:11:55.309 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:11:55.308+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:11:55.308351+0000) 2024-03-04T15:11:56.311 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:11:56.308+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:11:56.308638+0000) 2024-03-04T15:11:57.309 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:11:57.308+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:11:57.308976+0000) 2024-03-04T15:11:58.309 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:11:58.309+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:11:58.309265+0000) 2024-03-04T15:11:59.310 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:11:59.308+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:11:59.309552+0000) 2024-03-04T15:12:00.310 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:12:00.309+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:12:00.309915+0000) 2024-03-04T15:12:01.311 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:12:01.309+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:12:01.310207+0000) 2024-03-04T15:12:02.311 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:12:02.309+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:12:02.310496+0000) 2024-03-04T15:12:03.311 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:12:03.309+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:12:03.310796+0000) 2024-03-04T15:12:04.311 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:12:04.310+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:12:04.311143+0000) 2024-03-04T15:12:05.312 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:12:05.310+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:12:05.311441+0000) 2024-03-04T15:12:06.312 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:12:06.309+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:12:06.311756+0000) 2024-03-04T15:12:07.312 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:12:07.310+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:12:07.311974+0000) 2024-03-04T15:12:08.312 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:12:08.311+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:12:08.312247+0000) 2024-03-04T15:12:09.313 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:12:09.311+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:12:09.312535+0000) 2024-03-04T15:12:10.313 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:12:10.311+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:12:10.312920+0000) 2024-03-04T15:12:11.314 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:12:11.312+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:12:11.313207+0000) 2024-03-04T15:12:12.314 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:12:12.312+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:12:12.313426+0000) 2024-03-04T15:12:13.314 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:12:13.312+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:12:13.313717+0000) 2024-03-04T15:12:14.314 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:12:14.313+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:12:14.314033+0000) 2024-03-04T15:12:15.315 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:12:15.313+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:12:15.314324+0000) 2024-03-04T15:12:16.315 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:12:16.313+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:12:16.314632+0000) 2024-03-04T15:12:16.758 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:12:16.766 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:12:17.315 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:12:17.314+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:12:17.314911+0000) 2024-03-04T15:12:18.315 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:12:18.314+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:12:18.315203+0000) 2024-03-04T15:12:19.316 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:12:19.314+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:12:19.315576+0000) 2024-03-04T15:12:20.316 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:12:20.315+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:12:20.315926+0000) 2024-03-04T15:12:21.316 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:12:21.315+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:12:21.316230+0000) 2024-03-04T15:12:22.317 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:12:22.315+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:12:22.316523+0000) 2024-03-04T15:12:23.317 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:12:23.316+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:12:23.316817+0000) 2024-03-04T15:12:24.317 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:12:24.316+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:12:24.317145+0000) 2024-03-04T15:12:25.318 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:12:25.316+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:12:25.317454+0000) 2024-03-04T15:12:26.318 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:12:26.316+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:12:26.317761+0000) 2024-03-04T15:12:27.318 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:12:27.317+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:12:27.317991+0000) 2024-03-04T15:12:28.318 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:12:28.317+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:12:28.318283+0000) 2024-03-04T15:12:29.319 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:12:29.317+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:12:29.318580+0000) 2024-03-04T15:12:30.319 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:12:30.318+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:12:30.318914+0000) 2024-03-04T15:12:31.319 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:12:31.318+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:12:31.319198+0000) 2024-03-04T15:12:32.320 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:12:32.318+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:12:32.319528+0000) 2024-03-04T15:12:33.320 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:12:33.319+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:12:33.319873+0000) 2024-03-04T15:12:34.320 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:12:34.319+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:12:34.320190+0000) 2024-03-04T15:12:35.321 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:12:35.319+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:12:35.320495+0000) 2024-03-04T15:12:36.321 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:12:36.319+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:12:36.320806+0000) 2024-03-04T15:12:37.321 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:12:37.320+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:12:37.321166+0000) 2024-03-04T15:12:38.322 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:12:38.320+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:12:38.321467+0000) 2024-03-04T15:12:39.322 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:12:39.321+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:12:39.321762+0000) 2024-03-04T15:12:40.322 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:12:40.321+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:12:40.322038+0000) 2024-03-04T15:12:41.322 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:12:41.321+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:12:41.322275+0000) 2024-03-04T15:12:42.323 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:12:42.321+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:12:42.322523+0000) 2024-03-04T15:12:43.323 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:12:43.322+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:12:43.322884+0000) 2024-03-04T15:12:44.323 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:12:44.322+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:12:44.323168+0000) 2024-03-04T15:12:45.324 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:12:45.322+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:12:45.323449+0000) 2024-03-04T15:12:46.324 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:12:46.322+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:12:46.323755+0000) 2024-03-04T15:12:46.975 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:12:46.984 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:12:47.324 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:12:47.323+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:12:47.324033+0000) 2024-03-04T15:12:48.325 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:12:48.323+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:12:48.324298+0000) 2024-03-04T15:12:49.325 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:12:49.324+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:12:49.324541+0000) 2024-03-04T15:12:50.325 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:12:50.324+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:12:50.324861+0000) 2024-03-04T15:12:51.325 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:12:51.324+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:12:51.325167+0000) 2024-03-04T15:12:52.326 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:12:52.325+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:12:52.325464+0000) 2024-03-04T15:12:53.326 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:12:53.325+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:12:53.325771+0000) 2024-03-04T15:12:54.326 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:12:54.325+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:12:54.326065+0000) 2024-03-04T15:12:55.327 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:12:55.326+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:12:55.326383+0000) 2024-03-04T15:12:56.327 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:12:56.325+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:12:56.326662+0000) 2024-03-04T15:12:57.327 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:12:57.326+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:12:57.326987+0000) 2024-03-04T15:12:58.327 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:12:58.326+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:12:58.327285+0000) 2024-03-04T15:12:59.328 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:12:59.327+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:12:59.327590+0000) 2024-03-04T15:13:00.328 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:13:00.327+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:13:00.327934+0000) 2024-03-04T15:13:01.328 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:13:01.327+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:13:01.328191+0000) 2024-03-04T15:13:02.329 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:13:02.327+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:13:02.328484+0000) 2024-03-04T15:13:03.329 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:13:03.328+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:13:03.328781+0000) 2024-03-04T15:13:04.329 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:13:04.328+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:13:04.329106+0000) 2024-03-04T15:13:05.329 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:13:05.328+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:13:05.329403+0000) 2024-03-04T15:13:06.330 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:13:06.328+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:13:06.329715+0000) 2024-03-04T15:13:07.330 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:13:07.328+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:13:07.330018+0000) 2024-03-04T15:13:08.330 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:13:08.328+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:13:08.330307+0000) 2024-03-04T15:13:09.331 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:13:09.329+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:13:09.330599+0000) 2024-03-04T15:13:10.331 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:13:10.329+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:13:10.330937+0000) 2024-03-04T15:13:11.331 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:13:11.330+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:13:11.331233+0000) 2024-03-04T15:13:12.332 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:13:12.330+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:13:12.331516+0000) 2024-03-04T15:13:13.332 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:13:13.330+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:13:13.331857+0000) 2024-03-04T15:13:14.332 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:13:14.331+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:13:14.332151+0000) 2024-03-04T15:13:15.333 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:13:15.331+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:13:15.332456+0000) 2024-03-04T15:13:16.333 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:13:16.330+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:13:16.332767+0000) 2024-03-04T15:13:17.198 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:13:17.206 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:13:17.333 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:13:17.332+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:13:17.333062+0000) 2024-03-04T15:13:18.334 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:13:18.332+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:13:18.333382+0000) 2024-03-04T15:13:19.334 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:13:19.332+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:13:19.333677+0000) 2024-03-04T15:13:20.334 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:13:20.333+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:13:20.333976+0000) 2024-03-04T15:13:21.334 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:13:21.333+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:13:21.334275+0000) 2024-03-04T15:13:22.335 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:13:22.333+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:13:22.334583+0000) 2024-03-04T15:13:23.335 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:13:23.334+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:13:23.334860+0000) 2024-03-04T15:13:24.335 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:13:24.334+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:13:24.335110+0000) 2024-03-04T15:13:25.335 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:13:25.334+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:13:25.335336+0000) 2024-03-04T15:13:26.336 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:13:26.334+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:13:26.335579+0000) 2024-03-04T15:13:27.336 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:13:27.335+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:13:27.335872+0000) 2024-03-04T15:13:28.336 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:13:28.335+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:13:28.336172+0000) 2024-03-04T15:13:29.337 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:13:29.335+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:13:29.336430+0000) 2024-03-04T15:13:30.337 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:13:30.335+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:13:30.336695+0000) 2024-03-04T15:13:31.337 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:13:31.336+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:13:31.336944+0000) 2024-03-04T15:13:32.337 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:13:32.336+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:13:32.337201+0000) 2024-03-04T15:13:33.338 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:13:33.336+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:13:33.337477+0000) 2024-03-04T15:13:34.338 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:13:34.337+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:13:34.337713+0000) 2024-03-04T15:13:35.338 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:13:35.337+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:13:35.337957+0000) 2024-03-04T15:13:36.338 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:13:36.337+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:13:36.338153+0000) 2024-03-04T15:13:37.339 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:13:37.337+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:13:37.338410+0000) 2024-03-04T15:13:38.339 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:13:38.337+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:13:38.338679+0000) 2024-03-04T15:13:39.339 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:13:39.338+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:13:39.338934+0000) 2024-03-04T15:13:40.339 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:13:40.338+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:13:40.339218+0000) 2024-03-04T15:13:41.340 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:13:41.338+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:13:41.339470+0000) 2024-03-04T15:13:42.340 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:13:42.339+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:13:42.339720+0000) 2024-03-04T15:13:43.340 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:13:43.339+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:13:43.339965+0000) 2024-03-04T15:13:44.340 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:13:44.339+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:13:44.340224+0000) 2024-03-04T15:13:45.341 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:13:45.339+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:13:45.340493+0000) 2024-03-04T15:13:46.341 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:13:46.340+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:13:46.340731+0000) 2024-03-04T15:13:47.341 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:13:47.340+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:13:47.340970+0000) 2024-03-04T15:13:47.395 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:13:47.406 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:13:48.341 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:13:48.340+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:13:48.341213+0000) 2024-03-04T15:13:49.342 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:13:49.340+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:13:49.341483+0000) 2024-03-04T15:13:50.342 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:13:50.341+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:13:50.341726+0000) 2024-03-04T15:13:51.342 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:13:51.341+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:13:51.341967+0000) 2024-03-04T15:13:52.343 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:13:52.341+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:13:52.342200+0000) 2024-03-04T15:13:53.343 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:13:53.342+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:13:53.342445+0000) 2024-03-04T15:13:54.343 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:13:54.342+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:13:54.342686+0000) 2024-03-04T15:13:55.343 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:13:55.342+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:13:55.342969+0000) 2024-03-04T15:13:56.343 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:13:56.342+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:13:56.343204+0000) 2024-03-04T15:13:57.343 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:13:57.342+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:13:57.343396+0000) 2024-03-04T15:13:58.344 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:13:58.343+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:13:58.343634+0000) 2024-03-04T15:13:59.344 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:13:59.343+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:13:59.343932+0000) 2024-03-04T15:14:00.344 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:14:00.343+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:14:00.344178+0000) 2024-03-04T15:14:01.345 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:14:01.343+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:14:01.344366+0000) 2024-03-04T15:14:02.345 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:14:02.344+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:14:02.344602+0000) 2024-03-04T15:14:03.345 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:14:03.344+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:14:03.344885+0000) 2024-03-04T15:14:04.345 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:14:04.344+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:14:04.345146+0000) 2024-03-04T15:14:05.346 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:14:05.345+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:14:05.345446+0000) 2024-03-04T15:14:06.346 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:14:06.345+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:14:06.345706+0000) 2024-03-04T15:14:07.346 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:14:07.345+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:14:07.345959+0000) 2024-03-04T15:14:08.346 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:14:08.345+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:14:08.346196+0000) 2024-03-04T15:14:09.347 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:14:09.346+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:14:09.346428+0000) 2024-03-04T15:14:10.347 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:14:10.344+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:14:10.346696+0000) 2024-03-04T15:14:11.347 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:14:11.345+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:14:11.346966+0000) 2024-03-04T15:14:12.348 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:14:12.346+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:14:12.347243+0000) 2024-03-04T15:14:13.348 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:14:13.345+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:14:13.347459+0000) 2024-03-04T15:14:14.348 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:14:14.346+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:14:14.347709+0000) 2024-03-04T15:14:15.348 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:14:15.346+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:14:15.347995+0000) 2024-03-04T15:14:16.348 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:14:16.347+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:14:16.348257+0000) 2024-03-04T15:14:17.349 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:14:17.347+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:14:17.348523+0000) 2024-03-04T15:14:17.582 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:14:17.590 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:14:18.349 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:14:18.347+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:14:18.348784+0000) 2024-03-04T15:14:19.349 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:14:19.348+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:14:19.349110+0000) 2024-03-04T15:14:20.349 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:14:20.348+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:14:20.349405+0000) 2024-03-04T15:14:21.350 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:14:21.348+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:14:21.349701+0000) 2024-03-04T15:14:22.350 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:14:22.349+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:14:22.349995+0000) 2024-03-04T15:14:23.351 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:14:23.349+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:14:23.350283+0000) 2024-03-04T15:14:24.351 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:14:24.349+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:14:24.350574+0000) 2024-03-04T15:14:25.351 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:14:25.350+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:14:25.350929+0000) 2024-03-04T15:14:26.351 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:14:26.350+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:14:26.351223+0000) 2024-03-04T15:14:27.352 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:14:27.350+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:14:27.351526+0000) 2024-03-04T15:14:28.352 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:14:28.351+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:14:28.351866+0000) 2024-03-04T15:14:29.353 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:14:29.351+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:14:29.352191+0000) 2024-03-04T15:14:30.353 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:14:30.351+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:14:30.352488+0000) 2024-03-04T15:14:31.353 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:14:31.351+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:14:31.352790+0000) 2024-03-04T15:14:32.353 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:14:32.352+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:14:32.353134+0000) 2024-03-04T15:14:33.354 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:14:33.352+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:14:33.353388+0000) 2024-03-04T15:14:34.354 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:14:34.352+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:14:34.353680+0000) 2024-03-04T15:14:35.354 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:14:35.353+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:14:35.354000+0000) 2024-03-04T15:14:36.355 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:14:36.353+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:14:36.354297+0000) 2024-03-04T15:14:37.355 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:14:37.353+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:14:37.354599+0000) 2024-03-04T15:14:38.355 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:14:38.354+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:14:38.354922+0000) 2024-03-04T15:14:39.356 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:14:39.354+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:14:39.355224+0000) 2024-03-04T15:14:40.356 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:14:40.354+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:14:40.355525+0000) 2024-03-04T15:14:41.356 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:14:41.355+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:14:41.355891+0000) 2024-03-04T15:14:42.357 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:14:42.355+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:14:42.356217+0000) 2024-03-04T15:14:43.357 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:14:43.355+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:14:43.356502+0000) 2024-03-04T15:14:44.357 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:14:44.356+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:14:44.356869+0000) 2024-03-04T15:14:45.357 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:14:45.356+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:14:45.357174+0000) 2024-03-04T15:14:46.358 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:14:46.356+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:14:46.357472+0000) 2024-03-04T15:14:47.358 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:14:47.357+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:14:47.357762+0000) 2024-03-04T15:14:47.786 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:14:47.794 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:14:48.358 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:14:48.357+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:14:48.357959+0000) 2024-03-04T15:14:49.358 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:14:49.357+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:14:49.358205+0000) 2024-03-04T15:14:50.359 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:14:50.357+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:14:50.358454+0000) 2024-03-04T15:14:51.359 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:14:51.358+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:14:51.358705+0000) 2024-03-04T15:14:52.359 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:14:52.358+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:14:52.358917+0000) 2024-03-04T15:14:53.360 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:14:53.358+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:14:53.359152+0000) 2024-03-04T15:14:54.360 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:14:54.358+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:14:54.359343+0000) 2024-03-04T15:14:55.360 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:14:55.358+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:14:55.359579+0000) 2024-03-04T15:14:56.360 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:14:56.359+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:14:56.359899+0000) 2024-03-04T15:14:57.360 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:14:57.359+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:14:57.360143+0000) 2024-03-04T15:14:58.361 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:14:58.360+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:14:58.360393+0000) 2024-03-04T15:14:59.361 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:14:59.359+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:14:59.360629+0000) 2024-03-04T15:15:00.361 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:15:00.360+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:15:00.360928+0000) 2024-03-04T15:15:01.361 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:15:01.360+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:15:01.361175+0000) 2024-03-04T15:15:02.362 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:15:02.361+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:15:02.361413+0000) 2024-03-04T15:15:03.362 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:15:03.361+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:15:03.361653+0000) 2024-03-04T15:15:04.362 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:15:04.361+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:15:04.361943+0000) 2024-03-04T15:15:05.363 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:15:05.361+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:15:05.362172+0000) 2024-03-04T15:15:06.363 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:15:06.362+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:15:06.362414+0000) 2024-03-04T15:15:07.363 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:15:07.361+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:15:07.362653+0000) 2024-03-04T15:15:08.363 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:15:08.362+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:15:08.362948+0000) 2024-03-04T15:15:09.364 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:15:09.363+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:15:09.363213+0000) 2024-03-04T15:15:10.364 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:15:10.363+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:15:10.363447+0000) 2024-03-04T15:15:11.364 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:15:11.363+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:15:11.363696+0000) 2024-03-04T15:15:12.364 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:15:12.362+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:15:12.363932+0000) 2024-03-04T15:15:13.364 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:15:13.363+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:15:13.364173+0000) 2024-03-04T15:15:14.365 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:15:14.363+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:15:14.364410+0000) 2024-03-04T15:15:15.365 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:15:15.363+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:15:15.364649+0000) 2024-03-04T15:15:16.365 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:15:16.363+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:15:16.364928+0000) 2024-03-04T15:15:17.365 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:15:17.364+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:15:17.365173+0000) 2024-03-04T15:15:18.008 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:15:18.016 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:15:18.366 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:15:18.364+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:15:18.365422+0000) 2024-03-04T15:15:19.366 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:15:19.364+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:15:19.365660+0000) 2024-03-04T15:15:20.366 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:15:20.364+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:15:20.365948+0000) 2024-03-04T15:15:21.367 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:15:21.365+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:15:21.366185+0000) 2024-03-04T15:15:22.367 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:15:22.365+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:15:22.366421+0000) 2024-03-04T15:15:23.367 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:15:23.365+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:15:23.366671+0000) 2024-03-04T15:15:24.367 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:15:24.366+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:15:24.366941+0000) 2024-03-04T15:15:25.368 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:15:25.366+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:15:25.367182+0000) 2024-03-04T15:15:26.368 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:15:26.366+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:15:26.367426+0000) 2024-03-04T15:15:27.368 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:15:27.366+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:15:27.367689+0000) 2024-03-04T15:15:28.368 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:15:28.367+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:15:28.367928+0000) 2024-03-04T15:15:29.369 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:15:29.367+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:15:29.368165+0000) 2024-03-04T15:15:30.369 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:15:30.367+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:15:30.368400+0000) 2024-03-04T15:15:31.369 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:15:31.367+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:15:31.368603+0000) 2024-03-04T15:15:32.369 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:15:32.367+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:15:32.368782+0000) 2024-03-04T15:15:33.370 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:15:33.368+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:15:33.369029+0000) 2024-03-04T15:15:34.370 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:15:34.368+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:15:34.369266+0000) 2024-03-04T15:15:35.370 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:15:35.368+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:15:35.369510+0000) 2024-03-04T15:15:36.370 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:15:36.369+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:15:36.369756+0000) 2024-03-04T15:15:37.370 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:15:37.369+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:15:37.370002+0000) 2024-03-04T15:15:38.370 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:15:38.369+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:15:38.370250+0000) 2024-03-04T15:15:39.371 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:15:39.369+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:15:39.370487+0000) 2024-03-04T15:15:40.371 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:15:40.370+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:15:40.370764+0000) 2024-03-04T15:15:41.371 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:15:41.370+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:15:41.371074+0000) 2024-03-04T15:15:42.372 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:15:42.370+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:15:42.371321+0000) 2024-03-04T15:15:43.372 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:15:43.370+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:15:43.371587+0000) 2024-03-04T15:15:44.372 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:15:44.371+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:15:44.371854+0000) 2024-03-04T15:15:45.372 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:15:45.371+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:15:45.372089+0000) 2024-03-04T15:15:46.373 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:15:46.371+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:15:46.372341+0000) 2024-03-04T15:15:47.374 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:15:47.372+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:15:47.372597+0000) 2024-03-04T15:15:48.222 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:15:48.231 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:15:48.373 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:15:48.372+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:15:48.372915+0000) 2024-03-04T15:15:49.373 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:15:49.372+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:15:49.373161+0000) 2024-03-04T15:15:50.374 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:15:50.372+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:15:50.373427+0000) 2024-03-04T15:15:51.374 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:15:51.373+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:15:51.373684+0000) 2024-03-04T15:15:52.374 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:15:52.373+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:15:52.373933+0000) 2024-03-04T15:15:53.375 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:15:53.373+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:15:53.374183+0000) 2024-03-04T15:15:54.375 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:15:54.373+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:15:54.374421+0000) 2024-03-04T15:15:55.375 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:15:55.373+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:15:55.374702+0000) 2024-03-04T15:15:56.376 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:15:56.374+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:15:56.374950+0000) 2024-03-04T15:15:57.375 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:15:57.374+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:15:57.375200+0000) 2024-03-04T15:15:58.376 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:15:58.375+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:15:58.375444+0000) 2024-03-04T15:15:59.376 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:15:59.374+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:15:59.375681+0000) 2024-03-04T15:16:00.376 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:16:00.375+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:16:00.375941+0000) 2024-03-04T15:16:01.376 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:16:01.375+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:16:01.376144+0000) 2024-03-04T15:16:02.377 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:16:02.376+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:16:02.376392+0000) 2024-03-04T15:16:03.377 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:16:03.376+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:16:03.376641+0000) 2024-03-04T15:16:04.377 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:16:04.376+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:16:04.376935+0000) 2024-03-04T15:16:05.377 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:16:05.376+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:16:05.377177+0000) 2024-03-04T15:16:06.378 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:16:06.376+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:16:06.377381+0000) 2024-03-04T15:16:07.378 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:16:07.377+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:16:07.377629+0000) 2024-03-04T15:16:08.378 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:16:08.377+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:16:08.377922+0000) 2024-03-04T15:16:09.378 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:16:09.377+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:16:09.378195+0000) 2024-03-04T15:16:10.379 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:16:10.377+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:16:10.378459+0000) 2024-03-04T15:16:11.379 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:16:11.377+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:16:11.378733+0000) 2024-03-04T15:16:12.379 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:16:12.378+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:16:12.378960+0000) 2024-03-04T15:16:13.380 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:16:13.379+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:16:13.379209+0000) 2024-03-04T15:16:14.380 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:16:14.379+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:16:14.379443+0000) 2024-03-04T15:16:15.380 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:16:15.377+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:16:15.379710+0000) 2024-03-04T15:16:16.380 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:16:16.378+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:16:16.379978+0000) 2024-03-04T15:16:17.380 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:16:17.379+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:16:17.380237+0000) 2024-03-04T15:16:18.350 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:16:18.363 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:16:18.384 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:16:18.378+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:16:18.380433+0000) 2024-03-04T15:16:19.381 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:16:19.379+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:16:19.380696+0000) 2024-03-04T15:16:20.381 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:16:20.379+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:16:20.380933+0000) 2024-03-04T15:16:21.381 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:16:21.379+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:16:21.381125+0000) 2024-03-04T15:16:22.382 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:16:22.380+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:16:22.381373+0000) 2024-03-04T15:16:23.382 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:16:23.380+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:16:23.381627+0000) 2024-03-04T15:16:24.382 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:16:24.380+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:16:24.381904+0000) 2024-03-04T15:16:25.382 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:16:25.380+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:16:25.382168+0000) 2024-03-04T15:16:26.383 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:16:26.381+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:16:26.382427+0000) 2024-03-04T15:16:27.383 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:16:27.381+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:16:27.382620+0000) 2024-03-04T15:16:28.383 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:16:28.382+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:16:28.382883+0000) 2024-03-04T15:16:29.383 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:16:29.382+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:16:29.383190+0000) 2024-03-04T15:16:30.384 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:16:30.382+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:16:30.383444+0000) 2024-03-04T15:16:31.384 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:16:31.382+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:16:31.383684+0000) 2024-03-04T15:16:32.384 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:16:32.383+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:16:32.383934+0000) 2024-03-04T15:16:33.385 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:16:33.383+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:16:33.384175+0000) 2024-03-04T15:16:34.385 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:16:34.383+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:16:34.384409+0000) 2024-03-04T15:16:35.385 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:16:35.383+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:16:35.384650+0000) 2024-03-04T15:16:36.385 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:16:36.384+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:16:36.384951+0000) 2024-03-04T15:16:37.385 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:16:37.384+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:16:37.385165+0000) 2024-03-04T15:16:38.386 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:16:38.384+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:16:38.385406+0000) 2024-03-04T15:16:39.386 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:16:39.384+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:16:39.385660+0000) 2024-03-04T15:16:40.386 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:16:40.385+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:16:40.385935+0000) 2024-03-04T15:16:41.386 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:16:41.385+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:16:41.386175+0000) 2024-03-04T15:16:42.387 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:16:42.385+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:16:42.386361+0000) 2024-03-04T15:16:43.387 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:16:43.385+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:16:43.386608+0000) 2024-03-04T15:16:44.387 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:16:44.386+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:16:44.386873+0000) 2024-03-04T15:16:45.387 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:16:45.386+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:16:45.387120+0000) 2024-03-04T15:16:46.388 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:16:46.386+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:16:46.387361+0000) 2024-03-04T15:16:47.388 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:16:47.387+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:16:47.387622+0000) 2024-03-04T15:16:48.388 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:16:48.387+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:16:48.387920+0000) 2024-03-04T15:16:48.500 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:16:48.509 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:16:49.388 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:16:49.387+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:16:49.388164+0000) 2024-03-04T15:16:50.389 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:16:50.387+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:16:50.388408+0000) 2024-03-04T15:16:51.389 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:16:51.388+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:16:51.388647+0000) 2024-03-04T15:16:52.389 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:16:52.388+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:16:52.388930+0000) 2024-03-04T15:16:53.389 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:16:53.388+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:16:53.389171+0000) 2024-03-04T15:16:54.390 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:16:54.388+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:16:54.389422+0000) 2024-03-04T15:16:55.390 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:16:55.389+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:16:55.389679+0000) 2024-03-04T15:16:56.390 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:16:56.389+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:16:56.389926+0000) 2024-03-04T15:16:57.390 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:16:57.389+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:16:57.390185+0000) 2024-03-04T15:16:58.391 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:16:58.389+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:16:58.390420+0000) 2024-03-04T15:16:59.391 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:16:59.389+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:16:59.390677+0000) 2024-03-04T15:17:00.391 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:17:00.390+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:17:00.390926+0000) 2024-03-04T15:17:01.392 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:17:01.390+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:17:01.391175+0000) 2024-03-04T15:17:02.392 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:17:02.391+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:17:02.391419+0000) 2024-03-04T15:17:03.392 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:17:03.391+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:17:03.391668+0000) 2024-03-04T15:17:04.392 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:17:04.391+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:17:04.391936+0000) 2024-03-04T15:17:05.393 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:17:05.391+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:17:05.392181+0000) 2024-03-04T15:17:06.393 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:17:06.392+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:17:06.392421+0000) 2024-03-04T15:17:07.393 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:17:07.392+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:17:07.392669+0000) 2024-03-04T15:17:08.393 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:17:08.392+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:17:08.392945+0000) 2024-03-04T15:17:09.393 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:17:09.392+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:17:09.393208+0000) 2024-03-04T15:17:10.394 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:17:10.393+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:17:10.393473+0000) 2024-03-04T15:17:11.394 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:17:11.392+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:17:11.393706+0000) 2024-03-04T15:17:12.394 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:17:12.393+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:17:12.393972+0000) 2024-03-04T15:17:13.395 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:17:13.394+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:17:13.394217+0000) 2024-03-04T15:17:14.395 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:17:14.394+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:17:14.394459+0000) 2024-03-04T15:17:15.395 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:17:15.393+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:17:15.394720+0000) 2024-03-04T15:17:16.395 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:17:16.394+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:17:16.394979+0000) 2024-03-04T15:17:17.395 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:17:17.395+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:17:17.395229+0000) 2024-03-04T15:17:18.396 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:17:18.394+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:17:18.395472+0000) 2024-03-04T15:17:18.757 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:17:18.765 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:17:19.396 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:17:19.394+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:17:19.395726+0000) 2024-03-04T15:17:20.396 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:17:20.394+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:17:20.395976+0000) 2024-03-04T15:17:21.397 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:17:21.395+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:17:21.396222+0000) 2024-03-04T15:17:22.397 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:17:22.395+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:17:22.396464+0000) 2024-03-04T15:17:23.397 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:17:23.395+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:17:23.396714+0000) 2024-03-04T15:17:24.397 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:17:24.395+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:17:24.396953+0000) 2024-03-04T15:17:25.398 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:17:25.396+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:17:25.397201+0000) 2024-03-04T15:17:26.398 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:17:26.396+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:17:26.397436+0000) 2024-03-04T15:17:27.398 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:17:27.396+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:17:27.397690+0000) 2024-03-04T15:17:28.398 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:17:28.396+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:17:28.398056+0000) 2024-03-04T15:17:29.399 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:17:29.397+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:17:29.398299+0000) 2024-03-04T15:17:30.399 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:17:30.397+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:17:30.398564+0000) 2024-03-04T15:17:31.399 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:17:31.398+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:17:31.398865+0000) 2024-03-04T15:17:32.400 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:17:32.398+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:17:32.399108+0000) 2024-03-04T15:17:33.400 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:17:33.398+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:17:33.399361+0000) 2024-03-04T15:17:34.400 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:17:34.398+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:17:34.399598+0000) 2024-03-04T15:17:35.400 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:17:35.399+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:17:35.399870+0000) 2024-03-04T15:17:36.400 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:17:36.399+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:17:36.400116+0000) 2024-03-04T15:17:37.401 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:17:37.399+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:17:37.400364+0000) 2024-03-04T15:17:38.401 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:17:38.399+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:17:38.400612+0000) 2024-03-04T15:17:39.401 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:17:39.400+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:17:39.400891+0000) 2024-03-04T15:17:40.402 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:17:40.400+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:17:40.401098+0000) 2024-03-04T15:17:41.402 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:17:41.400+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:17:41.401295+0000) 2024-03-04T15:17:42.402 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:17:42.400+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:17:42.401534+0000) 2024-03-04T15:17:43.402 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:17:43.400+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:17:43.401793+0000) 2024-03-04T15:17:44.402 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:17:44.401+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:17:44.402083+0000) 2024-03-04T15:17:45.403 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:17:45.401+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:17:45.402335+0000) 2024-03-04T15:17:46.403 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:17:46.401+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:17:46.402625+0000) 2024-03-04T15:17:47.403 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:17:47.402+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:17:47.402937+0000) 2024-03-04T15:17:48.404 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:17:48.402+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:17:48.403175+0000) 2024-03-04T15:17:48.892 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:17:48.900 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:17:49.404 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:17:49.402+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:17:49.403423+0000) 2024-03-04T15:17:50.404 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:17:50.403+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:17:50.403664+0000) 2024-03-04T15:17:51.404 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:17:51.403+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:17:51.403934+0000) 2024-03-04T15:17:52.404 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:17:52.403+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:17:52.404161+0000) 2024-03-04T15:17:53.405 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:17:53.403+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:17:53.404395+0000) 2024-03-04T15:17:54.405 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:17:54.403+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:17:54.404583+0000) 2024-03-04T15:17:55.405 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:17:55.404+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:17:55.404852+0000) 2024-03-04T15:17:56.405 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:17:56.404+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:17:56.405088+0000) 2024-03-04T15:17:57.406 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:17:57.404+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:17:57.405321+0000) 2024-03-04T15:17:58.406 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:17:58.404+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:17:58.405552+0000) 2024-03-04T15:17:59.406 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:17:59.404+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:17:59.405790+0000) 2024-03-04T15:18:00.407 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:18:00.405+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:18:00.406094+0000) 2024-03-04T15:18:01.407 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:18:01.405+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:18:01.406332+0000) 2024-03-04T15:18:02.407 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:18:02.405+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:18:02.406568+0000) 2024-03-04T15:18:03.407 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:18:03.406+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:18:03.406856+0000) 2024-03-04T15:18:04.407 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:18:04.406+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:18:04.407091+0000) 2024-03-04T15:18:05.408 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:18:05.406+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:18:05.407334+0000) 2024-03-04T15:18:06.408 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:18:06.407+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:18:06.407599+0000) 2024-03-04T15:18:07.408 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:18:07.407+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:18:07.407887+0000) 2024-03-04T15:18:08.408 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:18:08.407+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:18:08.408131+0000) 2024-03-04T15:18:09.409 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:18:09.408+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:18:09.408385+0000) 2024-03-04T15:18:10.409 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:18:10.408+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:18:10.408657+0000) 2024-03-04T15:18:11.409 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:18:11.408+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:18:11.408948+0000) 2024-03-04T15:18:12.409 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:18:12.408+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:18:12.409126+0000) 2024-03-04T15:18:13.410 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:18:13.409+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:18:13.409379+0000) 2024-03-04T15:18:14.410 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:18:14.408+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:18:14.409625+0000) 2024-03-04T15:18:15.410 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:18:15.409+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:18:15.409915+0000) 2024-03-04T15:18:16.411 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:18:16.409+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:18:16.410204+0000) 2024-03-04T15:18:17.411 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:18:17.410+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:18:17.410441+0000) 2024-03-04T15:18:18.411 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:18:18.410+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:18:18.410678+0000) 2024-03-04T15:18:19.123 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:18:19.134 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:18:19.411 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:18:19.410+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:18:19.410963+0000) 2024-03-04T15:18:20.412 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:18:20.410+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:18:20.411165+0000) 2024-03-04T15:18:21.412 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:18:21.410+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:18:21.411412+0000) 2024-03-04T15:18:22.412 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:18:22.410+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:18:22.411656+0000) 2024-03-04T15:18:23.412 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:18:23.410+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:18:23.411945+0000) 2024-03-04T15:18:24.413 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:18:24.410+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:18:24.412133+0000) 2024-03-04T15:18:25.413 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:18:25.411+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:18:25.412371+0000) 2024-03-04T15:18:26.413 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:18:26.410+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:18:26.412634+0000) 2024-03-04T15:18:27.413 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:18:27.411+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:18:27.412937+0000) 2024-03-04T15:18:28.413 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:18:28.412+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:18:28.413201+0000) 2024-03-04T15:18:29.414 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:18:29.412+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:18:29.413395+0000) 2024-03-04T15:18:30.414 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:18:30.412+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:18:30.413641+0000) 2024-03-04T15:18:31.414 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:18:31.412+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:18:31.413932+0000) 2024-03-04T15:18:32.415 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:18:32.413+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:18:32.414184+0000) 2024-03-04T15:18:33.415 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:18:33.413+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:18:33.414436+0000) 2024-03-04T15:18:34.416 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:18:34.413+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:18:34.414626+0000) 2024-03-04T15:18:35.415 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:18:35.414+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:18:35.414930+0000) 2024-03-04T15:18:36.416 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:18:36.414+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:18:36.415194+0000) 2024-03-04T15:18:37.416 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:18:37.414+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:18:37.415433+0000) 2024-03-04T15:18:38.416 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:18:38.414+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:18:38.415700+0000) 2024-03-04T15:18:39.416 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:18:39.415+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:18:39.415963+0000) 2024-03-04T15:18:40.417 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:18:40.415+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:18:40.416224+0000) 2024-03-04T15:18:41.418 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:18:41.415+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:18:41.416420+0000) 2024-03-04T15:18:42.417 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:18:42.415+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:18:42.416669+0000) 2024-03-04T15:18:43.417 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:18:43.416+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:18:43.416914+0000) 2024-03-04T15:18:44.417 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:18:44.416+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:18:44.417124+0000) 2024-03-04T15:18:45.418 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:18:45.416+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:18:45.417365+0000) 2024-03-04T15:18:46.418 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:18:46.416+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:18:46.417606+0000) 2024-03-04T15:18:47.418 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:18:47.417+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:18:47.417889+0000) 2024-03-04T15:18:48.419 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:18:48.417+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:18:48.418131+0000) 2024-03-04T15:18:49.353 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:18:49.361 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:18:49.419 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:18:49.417+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:18:49.418288+0000) 2024-03-04T15:18:50.419 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:18:50.417+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:18:50.418540+0000) 2024-03-04T15:18:51.420 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:18:51.417+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:18:51.418808+0000) 2024-03-04T15:18:52.419 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:18:52.418+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:18:52.419091+0000) 2024-03-04T15:18:53.420 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:18:53.418+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:18:53.419346+0000) 2024-03-04T15:18:54.420 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:18:54.418+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:18:54.419584+0000) 2024-03-04T15:18:55.420 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:18:55.419+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:18:55.419886+0000) 2024-03-04T15:18:56.421 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:18:56.419+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:18:56.420120+0000) 2024-03-04T15:18:57.421 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:18:57.419+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:18:57.420359+0000) 2024-03-04T15:18:58.421 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:18:58.419+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:18:58.420595+0000) 2024-03-04T15:18:59.421 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:18:59.420+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:18:59.420861+0000) 2024-03-04T15:19:00.421 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:19:00.420+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:19:00.421056+0000) 2024-03-04T15:19:01.422 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:19:01.420+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:19:01.421246+0000) 2024-03-04T15:19:02.422 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:19:02.420+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:19:02.421421+0000) 2024-03-04T15:19:03.422 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:19:03.420+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:19:03.421680+0000) 2024-03-04T15:19:04.422 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:19:04.421+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:19:04.421938+0000) 2024-03-04T15:19:05.423 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:19:05.421+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:19:05.422130+0000) 2024-03-04T15:19:06.423 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:19:06.421+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:19:06.422310+0000) 2024-03-04T15:19:07.423 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:19:07.421+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:19:07.422494+0000) 2024-03-04T15:19:08.423 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:19:08.422+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:19:08.422728+0000) 2024-03-04T15:19:09.423 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:19:09.422+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:19:09.422977+0000) 2024-03-04T15:19:10.424 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:19:10.422+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:19:10.423224+0000) 2024-03-04T15:19:11.424 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:19:11.423+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:19:11.423489+0000) 2024-03-04T15:19:12.424 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:19:12.423+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:19:12.423745+0000) 2024-03-04T15:19:13.424 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:19:13.423+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:19:13.423946+0000) 2024-03-04T15:19:14.425 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:19:14.423+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:19:14.424181+0000) 2024-03-04T15:19:15.425 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:19:15.424+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:19:15.424428+0000) 2024-03-04T15:19:16.425 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:19:16.424+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:19:16.424662+0000) 2024-03-04T15:19:17.425 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:19:17.424+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:19:17.424964+0000) 2024-03-04T15:19:18.426 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:19:18.424+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:19:18.425209+0000) 2024-03-04T15:19:19.426 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:19:19.425+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:19:19.425443+0000) 2024-03-04T15:19:19.499 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:19:19.507 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:19:20.426 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:19:20.425+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:19:20.425698+0000) 2024-03-04T15:19:21.426 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:19:21.425+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:19:21.425948+0000) 2024-03-04T15:19:22.427 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:19:22.425+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:19:22.426159+0000) 2024-03-04T15:19:23.427 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:19:23.426+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:19:23.426395+0000) 2024-03-04T15:19:24.427 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:19:24.425+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:19:24.426642+0000) 2024-03-04T15:19:25.427 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:19:25.425+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:19:25.426935+0000) 2024-03-04T15:19:26.428 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:19:26.426+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:19:26.427167+0000) 2024-03-04T15:19:27.428 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:19:27.426+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:19:27.427422+0000) 2024-03-04T15:19:28.428 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:19:28.426+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:19:28.427662+0000) 2024-03-04T15:19:29.428 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:19:29.426+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:19:29.427936+0000) 2024-03-04T15:19:30.429 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:19:30.427+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:19:30.428180+0000) 2024-03-04T15:19:31.429 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:19:31.427+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:19:31.428428+0000) 2024-03-04T15:19:32.429 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:19:32.426+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:19:32.428630+0000) 2024-03-04T15:19:33.429 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:19:33.427+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:19:33.428854+0000) 2024-03-04T15:19:34.430 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:19:34.428+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:19:34.429087+0000) 2024-03-04T15:19:35.430 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:19:35.428+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:19:35.429349+0000) 2024-03-04T15:19:36.430 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:19:36.428+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:19:36.429586+0000) 2024-03-04T15:19:37.430 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:19:37.429+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:19:37.429864+0000) 2024-03-04T15:19:38.431 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:19:38.429+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:19:38.430116+0000) 2024-03-04T15:19:39.431 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:19:39.429+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:19:39.430358+0000) 2024-03-04T15:19:40.431 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:19:40.429+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:19:40.430550+0000) 2024-03-04T15:19:41.431 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:19:41.429+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:19:41.430798+0000) 2024-03-04T15:19:42.432 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:19:42.430+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:19:42.431103+0000) 2024-03-04T15:19:43.432 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:19:43.430+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:19:43.431307+0000) 2024-03-04T15:19:44.432 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:19:44.430+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:19:44.431549+0000) 2024-03-04T15:19:45.432 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:19:45.430+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:19:45.431790+0000) 2024-03-04T15:19:46.433 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:19:46.431+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:19:46.432041+0000) 2024-03-04T15:19:47.433 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:19:47.431+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:19:47.432291+0000) 2024-03-04T15:19:48.433 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:19:48.431+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:19:48.432538+0000) 2024-03-04T15:19:49.433 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:19:49.432+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:19:49.432781+0000) 2024-03-04T15:19:49.787 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:19:49.795 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:19:50.433 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:19:50.432+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:19:50.433041+0000) 2024-03-04T15:19:51.434 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:19:51.432+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:19:51.433284+0000) 2024-03-04T15:19:52.434 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:19:52.432+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:19:52.433537+0000) 2024-03-04T15:19:53.434 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:19:53.433+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:19:53.433784+0000) 2024-03-04T15:19:54.435 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:19:54.433+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:19:54.434031+0000) 2024-03-04T15:19:55.435 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:19:55.433+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:19:55.434268+0000) 2024-03-04T15:19:56.435 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:19:56.433+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:19:56.434503+0000) 2024-03-04T15:19:57.436 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:19:57.434+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:19:57.434754+0000) 2024-03-04T15:19:58.436 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:19:58.434+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:19:58.435020+0000) 2024-03-04T15:19:59.436 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:19:59.434+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:19:59.435206+0000) 2024-03-04T15:20:00.436 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:20:00.434+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:20:00.435456+0000) 2024-03-04T15:20:01.436 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:20:01.434+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:20:01.435648+0000) 2024-03-04T15:20:02.436 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:20:02.435+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:20:02.435935+0000) 2024-03-04T15:20:03.437 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:20:03.435+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:20:03.436188+0000) 2024-03-04T15:20:04.437 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:20:04.435+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:20:04.436428+0000) 2024-03-04T15:20:05.440 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:20:05.435+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:20:05.436685+0000) 2024-03-04T15:20:06.437 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:20:06.436+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:20:06.436925+0000) 2024-03-04T15:20:07.438 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:20:07.436+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:20:07.437171+0000) 2024-03-04T15:20:08.439 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:20:08.436+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:20:08.437408+0000) 2024-03-04T15:20:09.438 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:20:09.436+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:20:09.437655+0000) 2024-03-04T15:20:10.438 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:20:10.437+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:20:10.437916+0000) 2024-03-04T15:20:11.439 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:20:11.437+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:20:11.438172+0000) 2024-03-04T15:20:12.439 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:20:12.438+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:20:12.438425+0000) 2024-03-04T15:20:13.439 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:20:13.438+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:20:13.438704+0000) 2024-03-04T15:20:14.439 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:20:14.438+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:20:14.438936+0000) 2024-03-04T15:20:15.440 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:20:15.438+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:20:15.439132+0000) 2024-03-04T15:20:16.440 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:20:16.439+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:20:16.439364+0000) 2024-03-04T15:20:17.440 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:20:17.438+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:20:17.439672+0000) 2024-03-04T15:20:18.440 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:20:18.439+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:20:18.439932+0000) 2024-03-04T15:20:19.441 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:20:19.439+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:20:19.440215+0000) 2024-03-04T15:20:20.001 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:20:20.013 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:20:20.441 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:20:20.440+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:20:20.440483+0000) 2024-03-04T15:20:21.441 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:20:21.439+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:20:21.440736+0000) 2024-03-04T15:20:22.441 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:20:22.440+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:20:22.440983+0000) 2024-03-04T15:20:23.442 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:20:23.440+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:20:23.441190+0000) 2024-03-04T15:20:24.442 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:20:24.441+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:20:24.441424+0000) 2024-03-04T15:20:25.442 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:20:25.440+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:20:25.441676+0000) 2024-03-04T15:20:26.442 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:20:26.441+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:20:26.441956+0000) 2024-03-04T15:20:27.443 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:20:27.441+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:20:27.442163+0000) 2024-03-04T15:20:28.443 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:20:28.441+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:20:28.442356+0000) 2024-03-04T15:20:29.443 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:20:29.440+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:20:29.442548+0000) 2024-03-04T15:20:30.443 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:20:30.440+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:20:30.442797+0000) 2024-03-04T15:20:31.443 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:20:31.441+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:20:31.443067+0000) 2024-03-04T15:20:32.444 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:20:32.442+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:20:32.443291+0000) 2024-03-04T15:20:33.444 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:20:33.441+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:20:33.443566+0000) 2024-03-04T15:20:34.444 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:20:34.442+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:20:34.443814+0000) 2024-03-04T15:20:35.445 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:20:35.442+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:20:35.444032+0000) 2024-03-04T15:20:36.445 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:20:36.442+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:20:36.444216+0000) 2024-03-04T15:20:37.445 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:20:37.443+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:20:37.444411+0000) 2024-03-04T15:20:38.445 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:20:38.443+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:20:38.444675+0000) 2024-03-04T15:20:39.445 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:20:39.443+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:20:39.444968+0000) 2024-03-04T15:20:40.446 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:20:40.444+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:20:40.445221+0000) 2024-03-04T15:20:41.446 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:20:41.444+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:20:41.445411+0000) 2024-03-04T15:20:42.446 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:20:42.444+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:20:42.445657+0000) 2024-03-04T15:20:43.446 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:20:43.445+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:20:43.445951+0000) 2024-03-04T15:20:44.447 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:20:44.445+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:20:44.446223+0000) 2024-03-04T15:20:45.447 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:20:45.445+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:20:45.446501+0000) 2024-03-04T15:20:46.447 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:20:46.445+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:20:46.446738+0000) 2024-03-04T15:20:47.448 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:20:47.446+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:20:47.446993+0000) 2024-03-04T15:20:48.448 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:20:48.446+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:20:48.447233+0000) 2024-03-04T15:20:49.448 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:20:49.446+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:20:49.447479+0000) 2024-03-04T15:20:50.206 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:20:50.213 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:20:50.448 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:20:50.446+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:20:50.447764+0000) 2024-03-04T15:20:51.448 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:20:51.447+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:20:51.448017+0000) 2024-03-04T15:20:52.449 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:20:52.447+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:20:52.448233+0000) 2024-03-04T15:20:53.449 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:20:53.447+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:20:53.448497+0000) 2024-03-04T15:20:54.449 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:20:54.448+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:20:54.448736+0000) 2024-03-04T15:20:55.449 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:20:55.448+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:20:55.449005+0000) 2024-03-04T15:20:56.450 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:20:56.448+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:20:56.449248+0000) 2024-03-04T15:20:57.450 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:20:57.448+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:20:57.449495+0000) 2024-03-04T15:20:58.450 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:20:58.448+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:20:58.449740+0000) 2024-03-04T15:20:59.450 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:20:59.449+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:20:59.450040+0000) 2024-03-04T15:21:00.451 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:21:00.449+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:21:00.450248+0000) 2024-03-04T15:21:01.451 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:21:01.449+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:21:01.450448+0000) 2024-03-04T15:21:02.451 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:21:02.449+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:21:02.450708+0000) 2024-03-04T15:21:03.451 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:21:03.450+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:21:03.450974+0000) 2024-03-04T15:21:04.452 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:21:04.450+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:21:04.451172+0000) 2024-03-04T15:21:05.452 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:21:05.450+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:21:05.451385+0000) 2024-03-04T15:21:06.452 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:21:06.451+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:21:06.451624+0000) 2024-03-04T15:21:07.452 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:21:07.451+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:21:07.451920+0000) 2024-03-04T15:21:08.453 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:21:08.451+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:21:08.452123+0000) 2024-03-04T15:21:09.453 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:21:09.451+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:21:09.452318+0000) 2024-03-04T15:21:10.453 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:21:10.451+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:21:10.452523+0000) 2024-03-04T15:21:11.453 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:21:11.451+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:21:11.452751+0000) 2024-03-04T15:21:12.453 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:21:12.452+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:21:12.453017+0000) 2024-03-04T15:21:13.454 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:21:13.452+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:21:13.453224+0000) 2024-03-04T15:21:14.454 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:21:14.453+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:21:14.453462+0000) 2024-03-04T15:21:15.454 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:21:15.452+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:21:15.453737+0000) 2024-03-04T15:21:16.454 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:21:16.453+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:21:16.453995+0000) 2024-03-04T15:21:17.456 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:21:17.453+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:21:17.454241+0000) 2024-03-04T15:21:18.455 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:21:18.454+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:21:18.454496+0000) 2024-03-04T15:21:19.455 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:21:19.454+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:21:19.454750+0000) 2024-03-04T15:21:20.419 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:21:20.427 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:21:20.455 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:21:20.454+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:21:20.454946+0000) 2024-03-04T15:21:21.456 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:21:21.454+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:21:21.455139+0000) 2024-03-04T15:21:22.456 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:21:22.455+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:21:22.455371+0000) 2024-03-04T15:21:23.456 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:21:23.454+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:21:23.455610+0000) 2024-03-04T15:21:24.456 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:21:24.455+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:21:24.455874+0000) 2024-03-04T15:21:25.457 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:21:25.455+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:21:25.456113+0000) 2024-03-04T15:21:26.457 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:21:26.456+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:21:26.456363+0000) 2024-03-04T15:21:27.457 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:21:27.456+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:21:27.456564+0000) 2024-03-04T15:21:28.457 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:21:28.456+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:21:28.456869+0000) 2024-03-04T15:21:29.458 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:21:29.456+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:21:29.457121+0000) 2024-03-04T15:21:30.458 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:21:30.457+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:21:30.457362+0000) 2024-03-04T15:21:31.461 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:21:31.456+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:21:31.457628+0000) 2024-03-04T15:21:32.458 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:21:32.457+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:21:32.457919+0000) 2024-03-04T15:21:33.459 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:21:33.456+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:21:33.458118+0000) 2024-03-04T15:21:34.459 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:21:34.457+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:21:34.458352+0000) 2024-03-04T15:21:35.459 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:21:35.456+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:21:35.458553+0000) 2024-03-04T15:21:36.459 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:21:36.457+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:21:36.458785+0000) 2024-03-04T15:21:37.459 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:21:37.457+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:21:37.459010+0000) 2024-03-04T15:21:38.460 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:21:38.458+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:21:38.459258+0000) 2024-03-04T15:21:39.460 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:21:39.457+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:21:39.459543+0000) 2024-03-04T15:21:40.460 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:21:40.458+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:21:40.459810+0000) 2024-03-04T15:21:41.460 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:21:41.458+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:21:41.460066+0000) 2024-03-04T15:21:42.461 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:21:42.459+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:21:42.460312+0000) 2024-03-04T15:21:43.464 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:21:43.458+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:21:43.460568+0000) 2024-03-04T15:21:44.461 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:21:44.460+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:21:44.460865+0000) 2024-03-04T15:21:45.462 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:21:45.460+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:21:45.461083+0000) 2024-03-04T15:21:46.462 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:21:46.460+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:21:46.461366+0000) 2024-03-04T15:21:47.462 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:21:47.460+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:21:47.461621+0000) 2024-03-04T15:21:48.462 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:21:48.461+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:21:48.461951+0000) 2024-03-04T15:21:49.463 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:21:49.461+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:21:49.462152+0000) 2024-03-04T15:21:50.463 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:21:50.461+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:21:50.462420+0000) 2024-03-04T15:21:50.557 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:21:50.566 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:21:51.463 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:21:51.461+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:21:51.462624+0000) 2024-03-04T15:21:52.463 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:21:52.462+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:21:52.462919+0000) 2024-03-04T15:21:53.464 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:21:53.462+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:21:53.463187+0000) 2024-03-04T15:21:54.465 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:21:54.462+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:21:54.463428+0000) 2024-03-04T15:21:55.464 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:21:55.462+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:21:55.463676+0000) 2024-03-04T15:21:56.464 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:21:56.463+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:21:56.463956+0000) 2024-03-04T15:21:57.465 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:21:57.463+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:21:57.464199+0000) 2024-03-04T15:21:58.465 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:21:58.463+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:21:58.464444+0000) 2024-03-04T15:21:59.466 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:21:59.463+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:21:59.464638+0000) 2024-03-04T15:22:00.468 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:22:00.464+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:22:00.464928+0000) 2024-03-04T15:22:01.466 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:22:01.464+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:22:01.465139+0000) 2024-03-04T15:22:02.467 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:22:02.464+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:22:02.465410+0000) 2024-03-04T15:22:03.466 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:22:03.464+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:22:03.465697+0000) 2024-03-04T15:22:04.466 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:22:04.465+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:22:04.465953+0000) 2024-03-04T15:22:05.467 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:22:05.465+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:22:05.466216+0000) 2024-03-04T15:22:06.467 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:22:06.465+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:22:06.466453+0000) 2024-03-04T15:22:07.467 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:22:07.466+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:22:07.466697+0000) 2024-03-04T15:22:08.467 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:22:08.466+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:22:08.466948+0000) 2024-03-04T15:22:09.468 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:22:09.466+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:22:09.467190+0000) 2024-03-04T15:22:10.468 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:22:10.466+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:22:10.467446+0000) 2024-03-04T15:22:11.471 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:22:11.466+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:22:11.467711+0000) 2024-03-04T15:22:12.469 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:22:12.467+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:22:12.467950+0000) 2024-03-04T15:22:13.469 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:22:13.467+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:22:13.468187+0000) 2024-03-04T15:22:14.469 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:22:14.467+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:22:14.468388+0000) 2024-03-04T15:22:15.469 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:22:15.467+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:22:15.468628+0000) 2024-03-04T15:22:16.469 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:22:16.468+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:22:16.468812+0000) 2024-03-04T15:22:17.472 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:22:17.468+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:22:17.469038+0000) 2024-03-04T15:22:18.471 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:22:18.468+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:22:18.469279+0000) 2024-03-04T15:22:19.470 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:22:19.469+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:22:19.469534+0000) 2024-03-04T15:22:20.470 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:22:20.469+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:22:20.469798+0000) 2024-03-04T15:22:20.696 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:22:20.704 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:22:21.471 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:22:21.469+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:22:21.470089+0000) 2024-03-04T15:22:22.471 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:22:22.469+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:22:22.470329+0000) 2024-03-04T15:22:23.471 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:22:23.470+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:22:23.470571+0000) 2024-03-04T15:22:24.472 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:22:24.470+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:22:24.470811+0000) 2024-03-04T15:22:25.472 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:22:25.470+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:22:25.471087+0000) 2024-03-04T15:22:26.473 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:22:26.471+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:22:26.471321+0000) 2024-03-04T15:22:27.472 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:22:27.471+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:22:27.471665+0000) 2024-03-04T15:22:28.472 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:22:28.471+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:22:28.471897+0000) 2024-03-04T15:22:29.473 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:22:29.471+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:22:29.472137+0000) 2024-03-04T15:22:30.473 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:22:30.471+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:22:30.472324+0000) 2024-03-04T15:22:31.473 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:22:31.472+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:22:31.472558+0000) 2024-03-04T15:22:32.474 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:22:32.471+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:22:32.472784+0000) 2024-03-04T15:22:33.473 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:22:33.472+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:22:33.473036+0000) 2024-03-04T15:22:34.474 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:22:34.472+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:22:34.473224+0000) 2024-03-04T15:22:35.474 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:22:35.473+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:22:35.473466+0000) 2024-03-04T15:22:36.475 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:22:36.472+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:22:36.473720+0000) 2024-03-04T15:22:37.474 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:22:37.472+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:22:37.473963+0000) 2024-03-04T15:22:38.475 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:22:38.473+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:22:38.474200+0000) 2024-03-04T15:22:39.475 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:22:39.472+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:22:39.474394+0000) 2024-03-04T15:22:40.475 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:22:40.472+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:22:40.474656+0000) 2024-03-04T15:22:41.476 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:22:41.473+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:22:41.474959+0000) 2024-03-04T15:22:42.476 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:22:42.473+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:22:42.475149+0000) 2024-03-04T15:22:43.476 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:22:43.474+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:22:43.475393+0000) 2024-03-04T15:22:44.476 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:22:44.474+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:22:44.475629+0000) 2024-03-04T15:22:45.479 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:22:45.474+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:22:45.475915+0000) 2024-03-04T15:22:46.477 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:22:46.474+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:22:46.476101+0000) 2024-03-04T15:22:47.477 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:22:47.475+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:22:47.476298+0000) 2024-03-04T15:22:48.477 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:22:48.475+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:22:48.476550+0000) 2024-03-04T15:22:49.477 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:22:49.475+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:22:49.476795+0000) 2024-03-04T15:22:50.478 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:22:50.476+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:22:50.477090+0000) 2024-03-04T15:22:50.918 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:22:50.927 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:22:51.478 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:22:51.476+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:22:51.477336+0000) 2024-03-04T15:22:52.481 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:22:52.476+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:22:52.477575+0000) 2024-03-04T15:22:53.484 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:22:53.477+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:22:53.477855+0000) 2024-03-04T15:22:54.479 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:22:54.477+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:22:54.478099+0000) 2024-03-04T15:22:55.479 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:22:55.477+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:22:55.478339+0000) 2024-03-04T15:22:56.479 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:22:56.477+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:22:56.478576+0000) 2024-03-04T15:22:57.479 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:22:57.478+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:22:57.478818+0000) 2024-03-04T15:22:58.480 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:22:58.478+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:22:58.479061+0000) 2024-03-04T15:22:59.480 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:22:59.478+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:22:59.479318+0000) 2024-03-04T15:23:00.480 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:23:00.478+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:23:00.479568+0000) 2024-03-04T15:23:01.481 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:23:01.479+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:23:01.479811+0000) 2024-03-04T15:23:02.484 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:23:02.479+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:23:02.480094+0000) 2024-03-04T15:23:03.481 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:23:03.479+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:23:03.480391+0000) 2024-03-04T15:23:04.481 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:23:04.479+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:23:04.480671+0000) 2024-03-04T15:23:05.481 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:23:05.480+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:23:05.480963+0000) 2024-03-04T15:23:06.482 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:23:06.480+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:23:06.481157+0000) 2024-03-04T15:23:07.484 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:23:07.480+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:23:07.481432+0000) 2024-03-04T15:23:08.482 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:23:08.481+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:23:08.481733+0000) 2024-03-04T15:23:09.483 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:23:09.481+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:23:09.482087+0000) 2024-03-04T15:23:10.483 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:23:10.481+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:23:10.482398+0000) 2024-03-04T15:23:11.484 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:23:11.481+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:23:11.482708+0000) 2024-03-04T15:23:12.483 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:23:12.482+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:23:12.482999+0000) 2024-03-04T15:23:13.484 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:23:13.482+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:23:13.483258+0000) 2024-03-04T15:23:14.489 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:23:14.482+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:23:14.483494+0000) 2024-03-04T15:23:15.484 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:23:15.483+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:23:15.483871+0000) 2024-03-04T15:23:16.485 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:23:16.483+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:23:16.484123+0000) 2024-03-04T15:23:17.485 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:23:17.483+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:23:17.484431+0000) 2024-03-04T15:23:18.487 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:23:18.484+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:23:18.484728+0000) 2024-03-04T15:23:19.486 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:23:19.484+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:23:19.485019+0000) 2024-03-04T15:23:20.488 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:23:20.484+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:23:20.485326+0000) 2024-03-04T15:23:21.092 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:23:21.100 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:23:21.486 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:23:21.485+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:23:21.485624+0000) 2024-03-04T15:23:22.486 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:23:22.485+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:23:22.485951+0000) 2024-03-04T15:23:23.487 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:23:23.485+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:23:23.486245+0000) 2024-03-04T15:23:24.487 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:23:24.486+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:23:24.486527+0000) 2024-03-04T15:23:25.487 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:23:25.486+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:23:25.486871+0000) 2024-03-04T15:23:26.488 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:23:26.486+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:23:26.487116+0000) 2024-03-04T15:23:27.488 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:23:27.487+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:23:27.487421+0000) 2024-03-04T15:23:28.488 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:23:28.486+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:23:28.487671+0000) 2024-03-04T15:23:29.488 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:23:29.487+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:23:29.487928+0000) 2024-03-04T15:23:30.491 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:23:30.487+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:23:30.488171+0000) 2024-03-04T15:23:31.489 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:23:31.488+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:23:31.488432+0000) 2024-03-04T15:23:32.489 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:23:32.487+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:23:32.488610+0000) 2024-03-04T15:23:33.489 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:23:33.488+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:23:33.488918+0000) 2024-03-04T15:23:34.492 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:23:34.488+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:23:34.489159+0000) 2024-03-04T15:23:35.490 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:23:35.489+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:23:35.489394+0000) 2024-03-04T15:23:36.490 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:23:36.489+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:23:36.489630+0000) 2024-03-04T15:23:37.490 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:23:37.489+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:23:37.489917+0000) 2024-03-04T15:23:38.491 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:23:38.490+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:23:38.490156+0000) 2024-03-04T15:23:39.491 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:23:39.490+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:23:39.490397+0000) 2024-03-04T15:23:40.491 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:23:40.489+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:23:40.490641+0000) 2024-03-04T15:23:41.499 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:23:41.489+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:23:41.490922+0000) 2024-03-04T15:23:42.492 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:23:42.490+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:23:42.491156+0000) 2024-03-04T15:23:43.492 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:23:43.490+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:23:43.491420+0000) 2024-03-04T15:23:44.492 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:23:44.490+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:23:44.491647+0000) 2024-03-04T15:23:45.492 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:23:45.490+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:23:45.491945+0000) 2024-03-04T15:23:46.493 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:23:46.491+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:23:46.492178+0000) 2024-03-04T15:23:47.493 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:23:47.491+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:23:47.492434+0000) 2024-03-04T15:23:48.493 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:23:48.491+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:23:48.492703+0000) 2024-03-04T15:23:49.493 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:23:49.491+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:23:49.492942+0000) 2024-03-04T15:23:50.494 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:23:50.492+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:23:50.493190+0000) 2024-03-04T15:23:51.319 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:23:51.328 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:23:51.494 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:23:51.492+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:23:51.493469+0000) 2024-03-04T15:23:52.494 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:23:52.492+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:23:52.493715+0000) 2024-03-04T15:23:53.494 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:23:53.493+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:23:53.493958+0000) 2024-03-04T15:23:54.496 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:23:54.493+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:23:54.494196+0000) 2024-03-04T15:23:55.495 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:23:55.493+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:23:55.494448+0000) 2024-03-04T15:23:56.495 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:23:56.493+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:23:56.494680+0000) 2024-03-04T15:23:57.495 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:23:57.494+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:23:57.494929+0000) 2024-03-04T15:23:58.496 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:23:58.494+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:23:58.495219+0000) 2024-03-04T15:23:59.496 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:23:59.494+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:23:59.495415+0000) 2024-03-04T15:24:00.496 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:24:00.494+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:24:00.495692+0000) 2024-03-04T15:24:01.496 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:24:01.495+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:24:01.495954+0000) 2024-03-04T15:24:02.497 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:24:02.495+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:24:02.496200+0000) 2024-03-04T15:24:03.497 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:24:03.495+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:24:03.496498+0000) 2024-03-04T15:24:04.499 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:24:04.495+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:24:04.496753+0000) 2024-03-04T15:24:05.498 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:24:05.496+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:24:05.496996+0000) 2024-03-04T15:24:06.498 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:24:06.496+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:24:06.497239+0000) 2024-03-04T15:24:07.498 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:24:07.496+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:24:07.497489+0000) 2024-03-04T15:24:08.498 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:24:08.496+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:24:08.497756+0000) 2024-03-04T15:24:09.501 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:24:09.497+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:24:09.498005+0000) 2024-03-04T15:24:10.499 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:24:10.497+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:24:10.498254+0000) 2024-03-04T15:24:11.499 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:24:11.497+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:24:11.498503+0000) 2024-03-04T15:24:12.499 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:24:12.497+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:24:12.498738+0000) 2024-03-04T15:24:13.499 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:24:13.498+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:24:13.499009+0000) 2024-03-04T15:24:14.502 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:24:14.498+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:24:14.499251+0000) 2024-03-04T15:24:15.503 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:24:15.498+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:24:15.499495+0000) 2024-03-04T15:24:16.500 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:24:16.499+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:24:16.499741+0000) 2024-03-04T15:24:17.501 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:24:17.499+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:24:17.499984+0000) 2024-03-04T15:24:18.501 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:24:18.499+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:24:18.500242+0000) 2024-03-04T15:24:19.501 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:24:19.499+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:24:19.500485+0000) 2024-03-04T15:24:20.501 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:24:20.499+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:24:20.500716+0000) 2024-03-04T15:24:21.501 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:24:21.500+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:24:21.500973+0000) 2024-03-04T15:24:21.534 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:24:21.543 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:24:22.502 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:24:22.500+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:24:22.501167+0000) 2024-03-04T15:24:23.502 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:24:23.500+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:24:23.501413+0000) 2024-03-04T15:24:24.502 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:24:24.501+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:24:24.501649+0000) 2024-03-04T15:24:25.503 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:24:25.501+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:24:25.501927+0000) 2024-03-04T15:24:26.503 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:24:26.501+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:24:26.502161+0000) 2024-03-04T15:24:27.503 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:24:27.502+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:24:27.502403+0000) 2024-03-04T15:24:28.503 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:24:28.502+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:24:28.502647+0000) 2024-03-04T15:24:29.503 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:24:29.502+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:24:29.502938+0000) 2024-03-04T15:24:30.504 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:24:30.502+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:24:30.503187+0000) 2024-03-04T15:24:31.504 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:24:31.503+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:24:31.503425+0000) 2024-03-04T15:24:32.504 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:24:32.503+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:24:32.503659+0000) 2024-03-04T15:24:33.504 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:24:33.503+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:24:33.503943+0000) 2024-03-04T15:24:34.505 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:24:34.503+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:24:34.504182+0000) 2024-03-04T15:24:35.505 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:24:35.504+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:24:35.504425+0000) 2024-03-04T15:24:36.506 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:24:36.504+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:24:36.504658+0000) 2024-03-04T15:24:37.505 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:24:37.504+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:24:37.504934+0000) 2024-03-04T15:24:38.506 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:24:38.504+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:24:38.505180+0000) 2024-03-04T15:24:39.506 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:24:39.505+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:24:39.505427+0000) 2024-03-04T15:24:40.506 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:24:40.505+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:24:40.505673+0000) 2024-03-04T15:24:41.506 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:24:41.505+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:24:41.505925+0000) 2024-03-04T15:24:42.507 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:24:42.505+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:24:42.506154+0000) 2024-03-04T15:24:43.507 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:24:43.506+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:24:43.506402+0000) 2024-03-04T15:24:44.507 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:24:44.506+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:24:44.506682+0000) 2024-03-04T15:24:45.507 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:24:45.505+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:24:45.506946+0000) 2024-03-04T15:24:46.508 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:24:46.506+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:24:46.507178+0000) 2024-03-04T15:24:47.508 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:24:47.506+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:24:47.507414+0000) 2024-03-04T15:24:48.508 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:24:48.506+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:24:48.507663+0000) 2024-03-04T15:24:49.508 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:24:49.506+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:24:49.507934+0000) 2024-03-04T15:24:50.509 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:24:50.507+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:24:50.508178+0000) 2024-03-04T15:24:51.510 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:24:51.507+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:24:51.508439+0000) 2024-03-04T15:24:51.672 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:24:51.680 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:24:52.509 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:24:52.507+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:24:52.508676+0000) 2024-03-04T15:24:53.510 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:24:53.507+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:24:53.508951+0000) 2024-03-04T15:24:54.510 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:24:54.508+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:24:54.509206+0000) 2024-03-04T15:24:55.510 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:24:55.508+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:24:55.509450+0000) 2024-03-04T15:24:56.510 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:24:56.508+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:24:56.509740+0000) 2024-03-04T15:24:57.510 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:24:57.509+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:24:57.509998+0000) 2024-03-04T15:24:58.511 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:24:58.509+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:24:58.510250+0000) 2024-03-04T15:24:59.511 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:24:59.509+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:24:59.510473+0000) 2024-03-04T15:25:00.511 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:25:00.509+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:25:00.510727+0000) 2024-03-04T15:25:01.512 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:25:01.510+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:25:01.510994+0000) 2024-03-04T15:25:02.512 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:25:02.510+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:25:02.511234+0000) 2024-03-04T15:25:03.514 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:25:03.510+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:25:03.511435+0000) 2024-03-04T15:25:04.512 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:25:04.510+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:25:04.511684+0000) 2024-03-04T15:25:05.513 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:25:05.511+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:25:05.511996+0000) 2024-03-04T15:25:06.513 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:25:06.511+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:25:06.512292+0000) 2024-03-04T15:25:07.515 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:25:07.511+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:25:07.512591+0000) 2024-03-04T15:25:08.513 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:25:08.512+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:25:08.512947+0000) 2024-03-04T15:25:09.514 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:25:09.512+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:25:09.513203+0000) 2024-03-04T15:25:10.514 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:25:10.512+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:25:10.513441+0000) 2024-03-04T15:25:11.514 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:25:11.512+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:25:11.513644+0000) 2024-03-04T15:25:12.514 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:25:12.513+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:25:12.513927+0000) 2024-03-04T15:25:13.515 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:25:13.513+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:25:13.514211+0000) 2024-03-04T15:25:14.515 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:25:14.513+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:25:14.514456+0000) 2024-03-04T15:25:15.515 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:25:15.514+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:25:15.514696+0000) 2024-03-04T15:25:16.515 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:25:16.514+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:25:16.514928+0000) 2024-03-04T15:25:17.516 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:25:17.514+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:25:17.515169+0000) 2024-03-04T15:25:18.516 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:25:18.514+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:25:18.515434+0000) 2024-03-04T15:25:19.516 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:25:19.514+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:25:19.515637+0000) 2024-03-04T15:25:20.516 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:25:20.515+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:25:20.515923+0000) 2024-03-04T15:25:21.517 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:25:21.515+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:25:21.516157+0000) 2024-03-04T15:25:21.892 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:25:21.900 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:25:22.517 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:25:22.515+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:25:22.516393+0000) 2024-03-04T15:25:23.517 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:25:23.516+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:25:23.516648+0000) 2024-03-04T15:25:24.517 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:25:24.516+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:25:24.516941+0000) 2024-03-04T15:25:25.518 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:25:25.516+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:25:25.517195+0000) 2024-03-04T15:25:26.518 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:25:26.516+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:25:26.517428+0000) 2024-03-04T15:25:27.518 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:25:27.517+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:25:27.517666+0000) 2024-03-04T15:25:28.518 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:25:28.517+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:25:28.517959+0000) 2024-03-04T15:25:29.519 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:25:29.517+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:25:29.518165+0000) 2024-03-04T15:25:30.519 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:25:30.517+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:25:30.518506+0000) 2024-03-04T15:25:31.519 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:25:31.518+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:25:31.518761+0000) 2024-03-04T15:25:32.520 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:25:32.518+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:25:32.519001+0000) 2024-03-04T15:25:33.520 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:25:33.518+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:25:33.519257+0000) 2024-03-04T15:25:34.520 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:25:34.519+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:25:34.519501+0000) 2024-03-04T15:25:35.520 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:25:35.519+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:25:35.519701+0000) 2024-03-04T15:25:36.521 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:25:36.519+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:25:36.519937+0000) 2024-03-04T15:25:37.521 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:25:37.519+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:25:37.520175+0000) 2024-03-04T15:25:38.521 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:25:38.520+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:25:38.520452+0000) 2024-03-04T15:25:39.521 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:25:39.520+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:25:39.520703+0000) 2024-03-04T15:25:40.522 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:25:40.520+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:25:40.520955+0000) 2024-03-04T15:25:41.522 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:25:41.520+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:25:41.521191+0000) 2024-03-04T15:25:42.522 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:25:42.521+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:25:42.521432+0000) 2024-03-04T15:25:43.522 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:25:43.521+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:25:43.521700+0000) 2024-03-04T15:25:44.523 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:25:44.521+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:25:44.521986+0000) 2024-03-04T15:25:45.523 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:25:45.522+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:25:45.522227+0000) 2024-03-04T15:25:46.523 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:25:46.522+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:25:46.522473+0000) 2024-03-04T15:25:47.523 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:25:47.522+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:25:47.522715+0000) 2024-03-04T15:25:48.524 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:25:48.522+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:25:48.522968+0000) 2024-03-04T15:25:49.524 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:25:49.522+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:25:49.523229+0000) 2024-03-04T15:25:50.524 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:25:50.522+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:25:50.523477+0000) 2024-03-04T15:25:51.524 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:25:51.522+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:25:51.523721+0000) 2024-03-04T15:25:52.098 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:25:52.106 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:25:52.524 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:25:52.522+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:25:52.523967+0000) 2024-03-04T15:25:53.525 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:25:53.523+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:25:53.524213+0000) 2024-03-04T15:25:54.525 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:25:54.523+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:25:54.524461+0000) 2024-03-04T15:25:55.525 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:25:55.523+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:25:55.524708+0000) 2024-03-04T15:25:56.526 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:25:56.523+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:25:56.524945+0000) 2024-03-04T15:25:57.526 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:25:57.523+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:25:57.525180+0000) 2024-03-04T15:25:58.526 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:25:58.523+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:25:58.525373+0000) 2024-03-04T15:25:59.526 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:25:59.523+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:25:59.525618+0000) 2024-03-04T15:26:00.526 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:26:00.524+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:26:00.525915+0000) 2024-03-04T15:26:01.527 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:26:01.525+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:26:01.526152+0000) 2024-03-04T15:26:02.527 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:26:02.525+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:26:02.526419+0000) 2024-03-04T15:26:03.527 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:26:03.525+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:26:03.526666+0000) 2024-03-04T15:26:04.528 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:26:04.526+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:26:04.526957+0000) 2024-03-04T15:26:05.529 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:26:05.526+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:26:05.527199+0000) 2024-03-04T15:26:06.528 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:26:06.526+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:26:06.527430+0000) 2024-03-04T15:26:07.528 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:26:07.526+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:26:07.527674+0000) 2024-03-04T15:26:08.529 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:26:08.527+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:26:08.527955+0000) 2024-03-04T15:26:09.529 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:26:09.527+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:26:09.528204+0000) 2024-03-04T15:26:10.529 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:26:10.527+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:26:10.528452+0000) 2024-03-04T15:26:11.529 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:26:11.527+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:26:11.528697+0000) 2024-03-04T15:26:12.530 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:26:12.528+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:26:12.528934+0000) 2024-03-04T15:26:13.530 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:26:13.528+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:26:13.529179+0000) 2024-03-04T15:26:14.530 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:26:14.528+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:26:14.529434+0000) 2024-03-04T15:26:15.530 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:26:15.528+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:26:15.529693+0000) 2024-03-04T15:26:16.530 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:26:16.529+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:26:16.529934+0000) 2024-03-04T15:26:17.531 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:26:17.529+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:26:17.530169+0000) 2024-03-04T15:26:18.531 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:26:18.529+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:26:18.530404+0000) 2024-03-04T15:26:19.532 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:26:19.529+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:26:19.530642+0000) 2024-03-04T15:26:20.531 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:26:20.530+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:26:20.530939+0000) 2024-03-04T15:26:21.532 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:26:21.530+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:26:21.531182+0000) 2024-03-04T15:26:22.295 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:26:22.303 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:26:22.532 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:26:22.530+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:26:22.531451+0000) 2024-03-04T15:26:23.532 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:26:23.531+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:26:23.531690+0000) 2024-03-04T15:26:24.533 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:26:24.531+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:26:24.531951+0000) 2024-03-04T15:26:25.533 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:26:25.531+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:26:25.532191+0000) 2024-03-04T15:26:26.533 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:26:26.531+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:26:26.532430+0000) 2024-03-04T15:26:27.536 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:26:27.532+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:26:27.532677+0000) 2024-03-04T15:26:28.534 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:26:28.532+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:26:28.532942+0000) 2024-03-04T15:26:29.534 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:26:29.532+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:26:29.533180+0000) 2024-03-04T15:26:30.534 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:26:30.532+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:26:30.533431+0000) 2024-03-04T15:26:31.534 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:26:31.533+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:26:31.533680+0000) 2024-03-04T15:26:32.535 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:26:32.533+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:26:32.533939+0000) 2024-03-04T15:26:33.535 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:26:33.533+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:26:33.534134+0000) 2024-03-04T15:26:34.535 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:26:34.533+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:26:34.534330+0000) 2024-03-04T15:26:35.535 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:26:35.534+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:26:35.534574+0000) 2024-03-04T15:26:36.536 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:26:36.534+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:26:36.534812+0000) 2024-03-04T15:26:37.536 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:26:37.534+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:26:37.535143+0000) 2024-03-04T15:26:38.536 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:26:38.534+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:26:38.535375+0000) 2024-03-04T15:26:39.536 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:26:39.535+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:26:39.535642+0000) 2024-03-04T15:26:40.537 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:26:40.535+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:26:40.535943+0000) 2024-03-04T15:26:41.537 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:26:41.535+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:26:41.536146+0000) 2024-03-04T15:26:42.537 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:26:42.535+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:26:42.536375+0000) 2024-03-04T15:26:43.537 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:26:43.535+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:26:43.536576+0000) 2024-03-04T15:26:44.538 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:26:44.536+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:26:44.536859+0000) 2024-03-04T15:26:45.538 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:26:45.536+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:26:45.537099+0000) 2024-03-04T15:26:46.538 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:26:46.536+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:26:46.537301+0000) 2024-03-04T15:26:47.538 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:26:47.537+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:26:47.537570+0000) 2024-03-04T15:26:48.538 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:26:48.537+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:26:48.537805+0000) 2024-03-04T15:26:49.539 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:26:49.537+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:26:49.538053+0000) 2024-03-04T15:26:50.539 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:26:50.538+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:26:50.538300+0000) 2024-03-04T15:26:51.539 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:26:51.538+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:26:51.538539+0000) 2024-03-04T15:26:52.525 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:26:52.533 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:26:52.544 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:26:52.537+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:26:52.538764+0000) 2024-03-04T15:26:53.540 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:26:53.538+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:26:53.539022+0000) 2024-03-04T15:26:54.540 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:26:54.539+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:26:54.539255+0000) 2024-03-04T15:26:55.540 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:26:55.539+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:26:55.539494+0000) 2024-03-04T15:26:56.540 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:26:56.539+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:26:56.539719+0000) 2024-03-04T15:26:57.541 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:26:57.539+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:26:57.539955+0000) 2024-03-04T15:26:58.541 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:26:58.539+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:26:58.540187+0000) 2024-03-04T15:26:59.541 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:26:59.539+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:26:59.540431+0000) 2024-03-04T15:27:00.541 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:27:00.539+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:27:00.540679+0000) 2024-03-04T15:27:01.542 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:27:01.539+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:27:01.540949+0000) 2024-03-04T15:27:02.542 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:27:02.540+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:27:02.541183+0000) 2024-03-04T15:27:03.542 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:27:03.539+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:27:03.541423+0000) 2024-03-04T15:27:04.542 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:27:04.540+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:27:04.541669+0000) 2024-03-04T15:27:05.542 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:27:05.540+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:27:05.541928+0000) 2024-03-04T15:27:06.543 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:27:06.541+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:27:06.542181+0000) 2024-03-04T15:27:07.543 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:27:07.541+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:27:07.542417+0000) 2024-03-04T15:27:08.543 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:27:08.541+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:27:08.542651+0000) 2024-03-04T15:27:09.543 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:27:09.541+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:27:09.542939+0000) 2024-03-04T15:27:10.544 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:27:10.542+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:27:10.543183+0000) 2024-03-04T15:27:11.544 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:27:11.542+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:27:11.543424+0000) 2024-03-04T15:27:12.544 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:27:12.542+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:27:12.543660+0000) 2024-03-04T15:27:13.545 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:27:13.543+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:27:13.543952+0000) 2024-03-04T15:27:14.545 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:27:14.543+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:27:14.544145+0000) 2024-03-04T15:27:15.545 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:27:15.543+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:27:15.544391+0000) 2024-03-04T15:27:16.545 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:27:16.543+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:27:16.544647+0000) 2024-03-04T15:27:17.545 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:27:17.544+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:27:17.544938+0000) 2024-03-04T15:27:18.546 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:27:18.544+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:27:18.545178+0000) 2024-03-04T15:27:19.546 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:27:19.544+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:27:19.545447+0000) 2024-03-04T15:27:20.546 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:27:20.544+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:27:20.545751+0000) 2024-03-04T15:27:21.547 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:27:21.545+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:27:21.545997+0000) 2024-03-04T15:27:22.547 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:27:22.545+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:27:22.546227+0000) 2024-03-04T15:27:22.663 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:27:22.672 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:27:23.547 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:27:23.545+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:27:23.546422+0000) 2024-03-04T15:27:24.547 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:27:24.545+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:27:24.546605+0000) 2024-03-04T15:27:25.547 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:27:25.546+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:27:25.546887+0000) 2024-03-04T15:27:26.548 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:27:26.546+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:27:26.547134+0000) 2024-03-04T15:27:27.548 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:27:27.546+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:27:27.547324+0000) 2024-03-04T15:27:28.548 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:27:28.546+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:27:28.547553+0000) 2024-03-04T15:27:29.548 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:27:29.546+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:27:29.547736+0000) 2024-03-04T15:27:30.548 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:27:30.547+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:27:30.547954+0000) 2024-03-04T15:27:31.549 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:27:31.547+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:27:31.548198+0000) 2024-03-04T15:27:32.549 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:27:32.547+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:27:32.548435+0000) 2024-03-04T15:27:33.549 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:27:33.548+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:27:33.548676+0000) 2024-03-04T15:27:34.549 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:27:34.548+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:27:34.548944+0000) 2024-03-04T15:27:35.550 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:27:35.548+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:27:35.549188+0000) 2024-03-04T15:27:36.550 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:27:36.548+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:27:36.549430+0000) 2024-03-04T15:27:37.550 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:27:37.549+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:27:37.549673+0000) 2024-03-04T15:27:38.550 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:27:38.549+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:27:38.549955+0000) 2024-03-04T15:27:39.551 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:27:39.549+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:27:39.550199+0000) 2024-03-04T15:27:40.551 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:27:40.549+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:27:40.550447+0000) 2024-03-04T15:27:41.551 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:27:41.550+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:27:41.550687+0000) 2024-03-04T15:27:42.552 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:27:42.550+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:27:42.550918+0000) 2024-03-04T15:27:43.552 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:27:43.550+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:27:43.551162+0000) 2024-03-04T15:27:44.552 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:27:44.550+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:27:44.551344+0000) 2024-03-04T15:27:45.552 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:27:45.550+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:27:45.551539+0000) 2024-03-04T15:27:46.552 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:27:46.551+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:27:46.551775+0000) 2024-03-04T15:27:47.553 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:27:47.551+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:27:47.552034+0000) 2024-03-04T15:27:48.553 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:27:48.551+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:27:48.552296+0000) 2024-03-04T15:27:49.555 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:27:49.551+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:27:49.552496+0000) 2024-03-04T15:27:50.553 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:27:50.551+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:27:50.552687+0000) 2024-03-04T15:27:51.553 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:27:51.552+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:27:51.552922+0000) 2024-03-04T15:27:52.554 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:27:52.552+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:27:52.553160+0000) 2024-03-04T15:27:52.796 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:27:52.803 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:27:53.554 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:27:53.553+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:27:53.553407+0000) 2024-03-04T15:27:54.554 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:27:54.552+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:27:54.553634+0000) 2024-03-04T15:27:55.555 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:27:55.553+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:27:55.553923+0000) 2024-03-04T15:27:56.555 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:27:56.553+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:27:56.554157+0000) 2024-03-04T15:27:57.555 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:27:57.554+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:27:57.554399+0000) 2024-03-04T15:27:58.555 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:27:58.554+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:27:58.554632+0000) 2024-03-04T15:27:59.555 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:27:59.554+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:27:59.554913+0000) 2024-03-04T15:28:00.556 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:28:00.554+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:28:00.555188+0000) 2024-03-04T15:28:01.556 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:28:01.554+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:28:01.555454+0000) 2024-03-04T15:28:02.556 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:28:02.555+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:28:02.555691+0000) 2024-03-04T15:28:03.556 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:28:03.555+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:28:03.555937+0000) 2024-03-04T15:28:04.557 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:28:04.556+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:28:04.556181+0000) 2024-03-04T15:28:05.557 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:28:05.555+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:28:05.556431+0000) 2024-03-04T15:28:06.557 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:28:06.556+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:28:06.556676+0000) 2024-03-04T15:28:07.557 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:28:07.556+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:28:07.556961+0000) 2024-03-04T15:28:08.558 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:28:08.555+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:28:08.557183+0000) 2024-03-04T15:28:09.558 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:28:09.555+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:28:09.557397+0000) 2024-03-04T15:28:10.558 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:28:10.555+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:28:10.557592+0000) 2024-03-04T15:28:11.558 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:28:11.556+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:28:11.557867+0000) 2024-03-04T15:28:12.559 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:28:12.557+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:28:12.558125+0000) 2024-03-04T15:28:13.559 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:28:13.557+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:28:13.558414+0000) 2024-03-04T15:28:14.559 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:28:14.557+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:28:14.558658+0000) 2024-03-04T15:28:15.559 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:28:15.557+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:28:15.558940+0000) 2024-03-04T15:28:16.560 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:28:16.558+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:28:16.559173+0000) 2024-03-04T15:28:17.560 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:28:17.558+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:28:17.559415+0000) 2024-03-04T15:28:18.560 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:28:18.558+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:28:18.559669+0000) 2024-03-04T15:28:19.561 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:28:19.558+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:28:19.559938+0000) 2024-03-04T15:28:20.561 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:28:20.559+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:28:20.560186+0000) 2024-03-04T15:28:21.561 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:28:21.559+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:28:21.560421+0000) 2024-03-04T15:28:22.561 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:28:22.559+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:28:22.560653+0000) 2024-03-04T15:28:23.024 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:28:23.033 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:28:23.561 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:28:23.560+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:28:23.560940+0000) 2024-03-04T15:28:24.562 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:28:24.560+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:28:24.561176+0000) 2024-03-04T15:28:25.562 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:28:25.560+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:28:25.561417+0000) 2024-03-04T15:28:26.562 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:28:26.560+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:28:26.561650+0000) 2024-03-04T15:28:27.562 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:28:27.561+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:28:27.561933+0000) 2024-03-04T15:28:28.563 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:28:28.561+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:28:28.562168+0000) 2024-03-04T15:28:29.563 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:28:29.561+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:28:29.562406+0000) 2024-03-04T15:28:30.563 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:28:30.561+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:28:30.562651+0000) 2024-03-04T15:28:31.563 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:28:31.562+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:28:31.562935+0000) 2024-03-04T15:28:32.564 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:28:32.562+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:28:32.563174+0000) 2024-03-04T15:28:33.564 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:28:33.562+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:28:33.563413+0000) 2024-03-04T15:28:34.564 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:28:34.562+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:28:34.563647+0000) 2024-03-04T15:28:35.564 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:28:35.563+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:28:35.563930+0000) 2024-03-04T15:28:36.565 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:28:36.563+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:28:36.564155+0000) 2024-03-04T15:28:37.565 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:28:37.563+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:28:37.564393+0000) 2024-03-04T15:28:38.565 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:28:38.563+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:28:38.564641+0000) 2024-03-04T15:28:39.565 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:28:39.564+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:28:39.564930+0000) 2024-03-04T15:28:40.566 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:28:40.564+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:28:40.565186+0000) 2024-03-04T15:28:41.566 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:28:41.564+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:28:41.565425+0000) 2024-03-04T15:28:42.566 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:28:42.564+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:28:42.565660+0000) 2024-03-04T15:28:43.566 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:28:43.565+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:28:43.565942+0000) 2024-03-04T15:28:44.567 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:28:44.565+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:28:44.566181+0000) 2024-03-04T15:28:45.567 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:28:45.565+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:28:45.566368+0000) 2024-03-04T15:28:46.567 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:28:46.565+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:28:46.566564+0000) 2024-03-04T15:28:47.568 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:28:47.566+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:28:47.566907+0000) 2024-03-04T15:28:48.568 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:28:48.566+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:28:48.567199+0000) 2024-03-04T15:28:49.568 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:28:49.566+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:28:49.567428+0000) 2024-03-04T15:28:50.568 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:28:50.567+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:28:50.567734+0000) 2024-03-04T15:28:51.568 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:28:51.567+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:28:51.568034+0000) 2024-03-04T15:28:52.569 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:28:52.567+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:28:52.568271+0000) 2024-03-04T15:28:53.170 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:28:53.179 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:28:53.569 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:28:53.568+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:28:53.568568+0000) 2024-03-04T15:28:54.569 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:28:54.568+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:28:54.568914+0000) 2024-03-04T15:28:55.570 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:28:55.568+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:28:55.569206+0000) 2024-03-04T15:28:56.570 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:28:56.569+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:28:56.569506+0000) 2024-03-04T15:28:57.570 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:28:57.569+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:28:57.569786+0000) 2024-03-04T15:28:58.571 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:28:58.569+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:28:58.570131+0000) 2024-03-04T15:28:59.571 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:28:59.569+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:28:59.570420+0000) 2024-03-04T15:29:00.571 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:29:00.570+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:29:00.570729+0000) 2024-03-04T15:29:01.572 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:29:01.570+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:29:01.571022+0000) 2024-03-04T15:29:02.572 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:29:02.570+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:29:02.571259+0000) 2024-03-04T15:29:03.572 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:29:03.570+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:29:03.571521+0000) 2024-03-04T15:29:04.572 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:29:04.571+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:29:04.571760+0000) 2024-03-04T15:29:05.573 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:29:05.571+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:29:05.572002+0000) 2024-03-04T15:29:06.573 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:29:06.571+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:29:06.572233+0000) 2024-03-04T15:29:07.573 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:29:07.572+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:29:07.572525+0000) 2024-03-04T15:29:08.573 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:29:08.571+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:29:08.572778+0000) 2024-03-04T15:29:09.573 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:29:09.572+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:29:09.573026+0000) 2024-03-04T15:29:10.574 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:29:10.572+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:29:10.573227+0000) 2024-03-04T15:29:11.574 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:29:11.572+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:29:11.573491+0000) 2024-03-04T15:29:12.574 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:29:12.573+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:29:12.573726+0000) 2024-03-04T15:29:13.574 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:29:13.573+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:29:13.573966+0000) 2024-03-04T15:29:14.575 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:29:14.574+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:29:14.574260+0000) 2024-03-04T15:29:15.575 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:29:15.574+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:29:15.574559+0000) 2024-03-04T15:29:16.575 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:29:16.574+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:29:16.574929+0000) 2024-03-04T15:29:17.576 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:29:17.575+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:29:17.575251+0000) 2024-03-04T15:29:18.576 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:29:18.573+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:29:18.575488+0000) 2024-03-04T15:29:19.576 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:29:19.573+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:29:19.575733+0000) 2024-03-04T15:29:20.576 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:29:20.574+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:29:20.576002+0000) 2024-03-04T15:29:21.577 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:29:21.574+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:29:21.576211+0000) 2024-03-04T15:29:22.577 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:29:22.575+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:29:22.576494+0000) 2024-03-04T15:29:23.493 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:29:23.501 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:29:23.577 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:29:23.574+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:29:23.576791+0000) 2024-03-04T15:29:24.578 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:29:24.576+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:29:24.577169+0000) 2024-03-04T15:29:25.578 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:29:25.575+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:29:25.577461+0000) 2024-03-04T15:29:26.578 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:29:26.576+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:29:26.577740+0000) 2024-03-04T15:29:27.578 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:29:27.577+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:29:27.578030+0000) 2024-03-04T15:29:28.579 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:29:28.577+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:29:28.578313+0000) 2024-03-04T15:29:29.579 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:29:29.577+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:29:29.578607+0000) 2024-03-04T15:29:30.579 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:29:30.577+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:29:30.578951+0000) 2024-03-04T15:29:31.580 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:29:31.578+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:29:31.579239+0000) 2024-03-04T15:29:32.580 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:29:32.578+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:29:32.579532+0000) 2024-03-04T15:29:33.580 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:29:33.579+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:29:33.579813+0000) 2024-03-04T15:29:34.581 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:29:34.579+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:29:34.580157+0000) 2024-03-04T15:29:35.583 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:29:35.579+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:29:35.580445+0000) 2024-03-04T15:29:36.581 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:29:36.579+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:29:36.580732+0000) 2024-03-04T15:29:37.581 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:29:37.580+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:29:37.581019+0000) 2024-03-04T15:29:38.582 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:29:38.580+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:29:38.581312+0000) 2024-03-04T15:29:39.582 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:29:39.580+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:29:39.581610+0000) 2024-03-04T15:29:40.582 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:29:40.581+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:29:40.581962+0000) 2024-03-04T15:29:41.583 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:29:41.581+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:29:41.582252+0000) 2024-03-04T15:29:42.583 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:29:42.581+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:29:42.582561+0000) 2024-03-04T15:29:43.583 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:29:43.582+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:29:43.582918+0000) 2024-03-04T15:29:44.584 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:29:44.582+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:29:44.583196+0000) 2024-03-04T15:29:45.584 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:29:45.582+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:29:45.583467+0000) 2024-03-04T15:29:46.585 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:29:46.582+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:29:46.583700+0000) 2024-03-04T15:29:47.584 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:29:47.583+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:29:47.583942+0000) 2024-03-04T15:29:48.585 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:29:48.583+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:29:48.584175+0000) 2024-03-04T15:29:49.585 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:29:49.583+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:29:49.584429+0000) 2024-03-04T15:29:50.585 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:29:50.584+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:29:50.584694+0000) 2024-03-04T15:29:51.585 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:29:51.584+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:29:51.584964+0000) 2024-03-04T15:29:52.586 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:29:52.584+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:29:52.585153+0000) 2024-03-04T15:29:53.586 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:29:53.584+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:29:53.585394+0000) 2024-03-04T15:29:53.705 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:29:53.713 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:29:54.586 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:29:54.584+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:29:54.585636+0000) 2024-03-04T15:29:55.586 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:29:55.585+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:29:55.585997+0000) 2024-03-04T15:29:56.587 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:29:56.585+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:29:56.586245+0000) 2024-03-04T15:29:57.587 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:29:57.585+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:29:57.586525+0000) 2024-03-04T15:29:58.587 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:29:58.586+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:29:58.586811+0000) 2024-03-04T15:29:59.588 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:29:59.586+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:29:59.587162+0000) 2024-03-04T15:30:00.588 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:30:00.586+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:30:00.587471+0000) 2024-03-04T15:30:01.588 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:30:01.587+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:30:01.587760+0000) 2024-03-04T15:30:02.589 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:30:02.587+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:30:02.588051+0000) 2024-03-04T15:30:03.589 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:30:03.587+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:30:03.588341+0000) 2024-03-04T15:30:04.589 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:30:04.588+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:30:04.588656+0000) 2024-03-04T15:30:05.589 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:30:05.588+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:30:05.588991+0000) 2024-03-04T15:30:06.590 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:30:06.588+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:30:06.589271+0000) 2024-03-04T15:30:07.590 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:30:07.588+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:30:07.589498+0000) 2024-03-04T15:30:08.590 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:30:08.589+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:30:08.589782+0000) 2024-03-04T15:30:09.591 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:30:09.589+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:30:09.590132+0000) 2024-03-04T15:30:10.592 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:30:10.590+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:30:10.590471+0000) 2024-03-04T15:30:11.591 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:30:11.590+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:30:11.590754+0000) 2024-03-04T15:30:12.591 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:30:12.590+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:30:12.590991+0000) 2024-03-04T15:30:13.592 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:30:13.590+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:30:13.591283+0000) 2024-03-04T15:30:14.592 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:30:14.591+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:30:14.591548+0000) 2024-03-04T15:30:15.592 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:30:15.591+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:30:15.591794+0000) 2024-03-04T15:30:16.592 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:30:16.591+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:30:16.592083+0000) 2024-03-04T15:30:17.593 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:30:17.592+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:30:17.592321+0000) 2024-03-04T15:30:18.593 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:30:18.592+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:30:18.592553+0000) 2024-03-04T15:30:19.593 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:30:19.592+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:30:19.592802+0000) 2024-03-04T15:30:20.594 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:30:20.592+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:30:20.593094+0000) 2024-03-04T15:30:21.594 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:30:21.593+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:30:21.593334+0000) 2024-03-04T15:30:22.594 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:30:22.593+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:30:22.593571+0000) 2024-03-04T15:30:23.594 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:30:23.593+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:30:23.593867+0000) 2024-03-04T15:30:23.832 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:30:23.840 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:30:24.594 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:30:24.593+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:30:24.594068+0000) 2024-03-04T15:30:25.595 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:30:25.593+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:30:25.594304+0000) 2024-03-04T15:30:26.595 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:30:26.593+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:30:26.594492+0000) 2024-03-04T15:30:27.595 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:30:27.593+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:30:27.594756+0000) 2024-03-04T15:30:28.595 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:30:28.593+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:30:28.595013+0000) 2024-03-04T15:30:29.596 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:30:29.593+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:30:29.595217+0000) 2024-03-04T15:30:30.596 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:30:30.594+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:30:30.595488+0000) 2024-03-04T15:30:31.596 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:30:31.594+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:30:31.595746+0000) 2024-03-04T15:30:32.596 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:30:32.594+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:30:32.595998+0000) 2024-03-04T15:30:33.597 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:30:33.595+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:30:33.596256+0000) 2024-03-04T15:30:34.597 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:30:34.595+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:30:34.596504+0000) 2024-03-04T15:30:35.597 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:30:35.595+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:30:35.596800+0000) 2024-03-04T15:30:36.597 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:30:36.596+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:30:36.597111+0000) 2024-03-04T15:30:37.598 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:30:37.596+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:30:37.597362+0000) 2024-03-04T15:30:38.598 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:30:38.596+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:30:38.597615+0000) 2024-03-04T15:30:39.598 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:30:39.596+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:30:39.597940+0000) 2024-03-04T15:30:40.599 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:30:40.597+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:30:40.598207+0000) 2024-03-04T15:30:41.599 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:30:41.597+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:30:41.598468+0000) 2024-03-04T15:30:42.599 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:30:42.597+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:30:42.598719+0000) 2024-03-04T15:30:43.599 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:30:43.598+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:30:43.599107+0000) 2024-03-04T15:30:44.600 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:30:44.598+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:30:44.599363+0000) 2024-03-04T15:30:45.600 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:30:45.598+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:30:45.599658+0000) 2024-03-04T15:30:46.600 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:30:46.599+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:30:46.599945+0000) 2024-03-04T15:30:47.601 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:30:47.599+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:30:47.600202+0000) 2024-03-04T15:30:48.601 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:30:48.599+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:30:48.600455+0000) 2024-03-04T15:30:49.601 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:30:49.599+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:30:49.600741+0000) 2024-03-04T15:30:50.601 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:30:50.600+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:30:50.600995+0000) 2024-03-04T15:30:51.602 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:30:51.600+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:30:51.601238+0000) 2024-03-04T15:30:52.602 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:30:52.600+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:30:52.601425+0000) 2024-03-04T15:30:53.602 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:30:53.600+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:30:53.601661+0000) 2024-03-04T15:30:54.057 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:30:54.066 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:30:54.602 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:30:54.601+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:30:54.601920+0000) 2024-03-04T15:30:55.603 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:30:55.601+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:30:55.602164+0000) 2024-03-04T15:30:56.603 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:30:56.601+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:30:56.602426+0000) 2024-03-04T15:30:57.603 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:30:57.601+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:30:57.602612+0000) 2024-03-04T15:30:58.603 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:30:58.602+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:30:58.602862+0000) 2024-03-04T15:30:59.603 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:30:59.602+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:30:59.603085+0000) 2024-03-04T15:31:00.604 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:31:00.602+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:31:00.603309+0000) 2024-03-04T15:31:01.604 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:31:01.602+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:31:01.603551+0000) 2024-03-04T15:31:02.604 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:31:02.603+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:31:02.603786+0000) 2024-03-04T15:31:03.604 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:31:03.603+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:31:03.604031+0000) 2024-03-04T15:31:04.605 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:31:04.603+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:31:04.604268+0000) 2024-03-04T15:31:05.605 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:31:05.603+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:31:05.604515+0000) 2024-03-04T15:31:06.605 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:31:06.604+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:31:06.604748+0000) 2024-03-04T15:31:07.605 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:31:07.604+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:31:07.604990+0000) 2024-03-04T15:31:08.606 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:31:08.604+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:31:08.605227+0000) 2024-03-04T15:31:09.606 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:31:09.604+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:31:09.605467+0000) 2024-03-04T15:31:10.606 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:31:10.605+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:31:10.605730+0000) 2024-03-04T15:31:11.607 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:31:11.605+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:31:11.605973+0000) 2024-03-04T15:31:12.607 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:31:12.605+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:31:12.606214+0000) 2024-03-04T15:31:13.607 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:31:13.606+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:31:13.606471+0000) 2024-03-04T15:31:14.607 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:31:14.606+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:31:14.606704+0000) 2024-03-04T15:31:15.607 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:31:15.606+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:31:15.606949+0000) 2024-03-04T15:31:16.608 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:31:16.606+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:31:16.607184+0000) 2024-03-04T15:31:17.608 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:31:17.607+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:31:17.607419+0000) 2024-03-04T15:31:18.608 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:31:18.607+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:31:18.607653+0000) 2024-03-04T15:31:19.608 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:31:19.607+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:31:19.607923+0000) 2024-03-04T15:31:20.609 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:31:20.607+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:31:20.608168+0000) 2024-03-04T15:31:21.609 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:31:21.607+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:31:21.608360+0000) 2024-03-04T15:31:22.609 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:31:22.608+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:31:22.608605+0000) 2024-03-04T15:31:23.609 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:31:23.608+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:31:23.608884+0000) 2024-03-04T15:31:24.245 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:31:24.253 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:31:24.609 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:31:24.608+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:31:24.609150+0000) 2024-03-04T15:31:25.610 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:31:25.609+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:31:25.609397+0000) 2024-03-04T15:31:26.610 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:31:26.608+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:31:26.609672+0000) 2024-03-04T15:31:27.610 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:31:27.609+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:31:27.609960+0000) 2024-03-04T15:31:28.611 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:31:28.609+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:31:28.610176+0000) 2024-03-04T15:31:29.611 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:31:29.610+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:31:29.610459+0000) 2024-03-04T15:31:30.611 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:31:30.610+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:31:30.610765+0000) 2024-03-04T15:31:31.612 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:31:31.610+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:31:31.611171+0000) 2024-03-04T15:31:32.612 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:31:32.611+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:31:32.611471+0000) 2024-03-04T15:31:33.612 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:31:33.610+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:31:33.611759+0000) 2024-03-04T15:31:34.612 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:31:34.611+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:31:34.612043+0000) 2024-03-04T15:31:35.613 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:31:35.612+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:31:35.612332+0000) 2024-03-04T15:31:36.613 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:31:36.612+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:31:36.612622+0000) 2024-03-04T15:31:37.613 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:31:37.611+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:31:37.612949+0000) 2024-03-04T15:31:38.614 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:31:38.612+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:31:38.613257+0000) 2024-03-04T15:31:39.614 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:31:39.612+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:31:39.613553+0000) 2024-03-04T15:31:40.614 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:31:40.612+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:31:40.613899+0000) 2024-03-04T15:31:41.614 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:31:41.612+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:31:41.614134+0000) 2024-03-04T15:31:42.615 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:31:42.613+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:31:42.614462+0000) 2024-03-04T15:31:43.615 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:31:43.613+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:31:43.614760+0000) 2024-03-04T15:31:44.615 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:31:44.614+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:31:44.615056+0000) 2024-03-04T15:31:45.616 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:31:45.613+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:31:45.615332+0000) 2024-03-04T15:31:46.616 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:31:46.614+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:31:46.615652+0000) 2024-03-04T15:31:47.616 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:31:47.614+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:31:47.615962+0000) 2024-03-04T15:31:48.617 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:31:48.615+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:31:48.616252+0000) 2024-03-04T15:31:49.617 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:31:49.615+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:31:49.616544+0000) 2024-03-04T15:31:50.620 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:31:50.616+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:31:50.616876+0000) 2024-03-04T15:31:51.618 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:31:51.616+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:31:51.617158+0000) 2024-03-04T15:31:52.618 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:31:52.616+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:31:52.617448+0000) 2024-03-04T15:31:53.618 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:31:53.616+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:31:53.617727+0000) 2024-03-04T15:31:54.450 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:31:54.458 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:31:54.618 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:31:54.617+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:31:54.617974+0000) 2024-03-04T15:31:55.618 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:31:55.617+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:31:55.618212+0000) 2024-03-04T15:31:56.619 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:31:56.617+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:31:56.618498+0000) 2024-03-04T15:31:57.619 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:31:57.617+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:31:57.618792+0000) 2024-03-04T15:31:58.619 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:31:58.618+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:31:58.619136+0000) 2024-03-04T15:31:59.620 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:31:59.618+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:31:59.619398+0000) 2024-03-04T15:32:00.620 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:32:00.618+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:32:00.619641+0000) 2024-03-04T15:32:01.620 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:32:01.619+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:32:01.619893+0000) 2024-03-04T15:32:02.620 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:32:02.619+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:32:02.620134+0000) 2024-03-04T15:32:03.621 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:32:03.619+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:32:03.620394+0000) 2024-03-04T15:32:04.621 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:32:04.619+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:32:04.620661+0000) 2024-03-04T15:32:05.621 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:32:05.620+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:32:05.620942+0000) 2024-03-04T15:32:06.621 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:32:06.620+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:32:06.621195+0000) 2024-03-04T15:32:07.622 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:32:07.620+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:32:07.621452+0000) 2024-03-04T15:32:08.622 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:32:08.621+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:32:08.621711+0000) 2024-03-04T15:32:09.622 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:32:09.621+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:32:09.621958+0000) 2024-03-04T15:32:10.623 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:32:10.621+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:32:10.622198+0000) 2024-03-04T15:32:11.623 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:32:11.621+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:32:11.622442+0000) 2024-03-04T15:32:12.623 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:32:12.622+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:32:12.622682+0000) 2024-03-04T15:32:13.623 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:32:13.622+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:32:13.622956+0000) 2024-03-04T15:32:14.624 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:32:14.622+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:32:14.623191+0000) 2024-03-04T15:32:15.624 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:32:15.622+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:32:15.623442+0000) 2024-03-04T15:32:16.624 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:32:16.622+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:32:16.623637+0000) 2024-03-04T15:32:17.624 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:32:17.623+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:32:17.623926+0000) 2024-03-04T15:32:18.624 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:32:18.623+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:32:18.624139+0000) 2024-03-04T15:32:19.625 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:32:19.623+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:32:19.624340+0000) 2024-03-04T15:32:20.625 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:32:20.623+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:32:20.624549+0000) 2024-03-04T15:32:21.625 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:32:21.624+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:32:21.624867+0000) 2024-03-04T15:32:22.625 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:32:22.624+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:32:22.625118+0000) 2024-03-04T15:32:23.626 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:32:23.624+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:32:23.625372+0000) 2024-03-04T15:32:24.598 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:32:24.606 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:32:24.626 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:32:24.624+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:32:24.625573+0000) 2024-03-04T15:32:25.626 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:32:25.625+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:32:25.625815+0000) 2024-03-04T15:32:26.626 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:32:26.625+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:32:26.626082+0000) 2024-03-04T15:32:27.627 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:32:27.625+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:32:27.626342+0000) 2024-03-04T15:32:28.627 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:32:28.626+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:32:28.626582+0000) 2024-03-04T15:32:29.627 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:32:29.626+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:32:29.626852+0000) 2024-03-04T15:32:30.627 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:32:30.626+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:32:30.627049+0000) 2024-03-04T15:32:31.627 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:32:31.626+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:32:31.627288+0000) 2024-03-04T15:32:32.628 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:32:32.627+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:32:32.627528+0000) 2024-03-04T15:32:33.628 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:32:33.627+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:32:33.627785+0000) 2024-03-04T15:32:34.628 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:32:34.627+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:32:34.628073+0000) 2024-03-04T15:32:35.629 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:32:35.628+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:32:35.628353+0000) 2024-03-04T15:32:36.629 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:32:36.628+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:32:36.628627+0000) 2024-03-04T15:32:37.629 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:32:37.628+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:32:37.628907+0000) 2024-03-04T15:32:38.629 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:32:38.628+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:32:38.629181+0000) 2024-03-04T15:32:39.630 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:32:39.629+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:32:39.629435+0000) 2024-03-04T15:32:40.630 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:32:40.629+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:32:40.629705+0000) 2024-03-04T15:32:41.630 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:32:41.629+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:32:41.629963+0000) 2024-03-04T15:32:42.630 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:32:42.630+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:32:42.630233+0000) 2024-03-04T15:32:43.631 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:32:43.630+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:32:43.630475+0000) 2024-03-04T15:32:44.631 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:32:44.630+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:32:44.630703+0000) 2024-03-04T15:32:45.631 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:32:45.630+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:32:45.630953+0000) 2024-03-04T15:32:46.632 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:32:46.631+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:32:46.631191+0000) 2024-03-04T15:32:47.632 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:32:47.629+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:32:47.631450+0000) 2024-03-04T15:32:48.632 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:32:48.630+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:32:48.631687+0000) 2024-03-04T15:32:49.632 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:32:49.630+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:32:49.631952+0000) 2024-03-04T15:32:50.633 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:32:50.631+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:32:50.632197+0000) 2024-03-04T15:32:51.633 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:32:51.630+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:32:51.632463+0000) 2024-03-04T15:32:52.633 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:32:52.631+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:32:52.632705+0000) 2024-03-04T15:32:53.633 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:32:53.631+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:32:53.632944+0000) 2024-03-04T15:32:54.633 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:32:54.632+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:32:54.633189+0000) 2024-03-04T15:32:54.737 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:32:54.745 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:32:55.634 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:32:55.632+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:32:55.633436+0000) 2024-03-04T15:32:56.634 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:32:56.632+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:32:56.633668+0000) 2024-03-04T15:32:57.634 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:32:57.632+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:32:57.633947+0000) 2024-03-04T15:32:58.634 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:32:58.633+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:32:58.634159+0000) 2024-03-04T15:32:59.635 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:32:59.633+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:32:59.634397+0000) 2024-03-04T15:33:00.635 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:33:00.633+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:33:00.634646+0000) 2024-03-04T15:33:01.635 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:33:01.634+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:33:01.634934+0000) 2024-03-04T15:33:02.635 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:33:02.634+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:33:02.635168+0000) 2024-03-04T15:33:03.636 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:33:03.634+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:33:03.635412+0000) 2024-03-04T15:33:04.637 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:33:04.634+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:33:04.635651+0000) 2024-03-04T15:33:05.636 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:33:05.635+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:33:05.635958+0000) 2024-03-04T15:33:06.636 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:33:06.635+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:33:06.636217+0000) 2024-03-04T15:33:07.637 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:33:07.635+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:33:07.636459+0000) 2024-03-04T15:33:08.637 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:33:08.635+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:33:08.636694+0000) 2024-03-04T15:33:09.637 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:33:09.636+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:33:09.636966+0000) 2024-03-04T15:33:10.637 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:33:10.636+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:33:10.637210+0000) 2024-03-04T15:33:11.638 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:33:11.636+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:33:11.637459+0000) 2024-03-04T15:33:12.638 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:33:12.636+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:33:12.637699+0000) 2024-03-04T15:33:13.638 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:33:13.637+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:33:13.637930+0000) 2024-03-04T15:33:14.639 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:33:14.637+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:33:14.638186+0000) 2024-03-04T15:33:15.639 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:33:15.637+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:33:15.638434+0000) 2024-03-04T15:33:16.639 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:33:16.638+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:33:16.638704+0000) 2024-03-04T15:33:17.639 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:33:17.638+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:33:17.639028+0000) 2024-03-04T15:33:18.640 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:33:18.638+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:33:18.639263+0000) 2024-03-04T15:33:19.640 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:33:19.638+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:33:19.639520+0000) 2024-03-04T15:33:20.640 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:33:20.639+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:33:20.639772+0000) 2024-03-04T15:33:21.640 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:33:21.639+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:33:21.640016+0000) 2024-03-04T15:33:22.640 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:33:22.639+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:33:22.640207+0000) 2024-03-04T15:33:23.641 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:33:23.639+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:33:23.640458+0000) 2024-03-04T15:33:24.641 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:33:24.640+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:33:24.640692+0000) 2024-03-04T15:33:24.950 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:33:24.958 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:33:25.641 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:33:25.640+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:33:25.640940+0000) 2024-03-04T15:33:26.641 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:33:26.640+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:33:26.641199+0000) 2024-03-04T15:33:27.642 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:33:27.640+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:33:27.641438+0000) 2024-03-04T15:33:28.642 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:33:28.641+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:33:28.641677+0000) 2024-03-04T15:33:29.642 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:33:29.641+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:33:29.641964+0000) 2024-03-04T15:33:30.642 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:33:30.641+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:33:30.642254+0000) 2024-03-04T15:33:31.643 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:33:31.641+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:33:31.642511+0000) 2024-03-04T15:33:32.643 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:33:32.642+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:33:32.642748+0000) 2024-03-04T15:33:33.643 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:33:33.642+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:33:33.642999+0000) 2024-03-04T15:33:34.644 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:33:34.642+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:33:34.643239+0000) 2024-03-04T15:33:35.644 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:33:35.643+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:33:35.643505+0000) 2024-03-04T15:33:36.644 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:33:36.643+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:33:36.643773+0000) 2024-03-04T15:33:37.645 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:33:37.643+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:33:37.644016+0000) 2024-03-04T15:33:38.644 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:33:38.643+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:33:38.644204+0000) 2024-03-04T15:33:39.645 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:33:39.644+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:33:39.644467+0000) 2024-03-04T15:33:40.645 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:33:40.643+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:33:40.644752+0000) 2024-03-04T15:33:41.645 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:33:41.644+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:33:41.644996+0000) 2024-03-04T15:33:42.646 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:33:42.644+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:33:42.645188+0000) 2024-03-04T15:33:43.646 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:33:43.645+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:33:43.645427+0000) 2024-03-04T15:33:44.646 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:33:44.644+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:33:44.645654+0000) 2024-03-04T15:33:45.646 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:33:45.645+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:33:45.645942+0000) 2024-03-04T15:33:46.647 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:33:46.645+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:33:46.646174+0000) 2024-03-04T15:33:47.647 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:33:47.646+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:33:47.646412+0000) 2024-03-04T15:33:48.647 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:33:48.646+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:33:48.646656+0000) 2024-03-04T15:33:49.647 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:33:49.646+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:33:49.646932+0000) 2024-03-04T15:33:50.647 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:33:50.646+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:33:50.647178+0000) 2024-03-04T15:33:51.648 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:33:51.647+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:33:51.647420+0000) 2024-03-04T15:33:52.648 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:33:52.646+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:33:52.647657+0000) 2024-03-04T15:33:53.648 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:33:53.646+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:33:53.647934+0000) 2024-03-04T15:33:54.648 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:33:54.647+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:33:54.648210+0000) 2024-03-04T15:33:55.163 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:33:55.171 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:33:55.649 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:33:55.647+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:33:55.648481+0000) 2024-03-04T15:33:56.649 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:33:56.646+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:33:56.648743+0000) 2024-03-04T15:33:57.649 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:33:57.647+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:33:57.649019+0000) 2024-03-04T15:33:58.650 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:33:58.648+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:33:58.649268+0000) 2024-03-04T15:33:59.650 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:33:59.648+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:33:59.649502+0000) 2024-03-04T15:34:00.650 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:34:00.648+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:34:00.649762+0000) 2024-03-04T15:34:01.650 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:34:01.649+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:34:01.650019+0000) 2024-03-04T15:34:02.651 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:34:02.649+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:34:02.650270+0000) 2024-03-04T15:34:03.651 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:34:03.649+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:34:03.650510+0000) 2024-03-04T15:34:04.651 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:34:04.649+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:34:04.650748+0000) 2024-03-04T15:34:05.651 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:34:05.650+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:34:05.651082+0000) 2024-03-04T15:34:06.652 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:34:06.650+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:34:06.651335+0000) 2024-03-04T15:34:07.652 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:34:07.650+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:34:07.651574+0000) 2024-03-04T15:34:08.652 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:34:08.651+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:34:08.651814+0000) 2024-03-04T15:34:09.652 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:34:09.651+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:34:09.652102+0000) 2024-03-04T15:34:10.653 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:34:10.651+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:34:10.652358+0000) 2024-03-04T15:34:11.653 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:34:11.651+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:34:11.652643+0000) 2024-03-04T15:34:12.653 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:34:12.652+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:34:12.652917+0000) 2024-03-04T15:34:13.653 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:34:13.652+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:34:13.653157+0000) 2024-03-04T15:34:14.654 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:34:14.652+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:34:14.653391+0000) 2024-03-04T15:34:15.654 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:34:15.652+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:34:15.653640+0000) 2024-03-04T15:34:16.654 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:34:16.653+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:34:16.653939+0000) 2024-03-04T15:34:17.654 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:34:17.653+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:34:17.654175+0000) 2024-03-04T15:34:18.655 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:34:18.653+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:34:18.654409+0000) 2024-03-04T15:34:19.655 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:34:19.653+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:34:19.654647+0000) 2024-03-04T15:34:20.655 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:34:20.654+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:34:20.654947+0000) 2024-03-04T15:34:21.656 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:34:21.654+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:34:21.655198+0000) 2024-03-04T15:34:22.656 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:34:22.654+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:34:22.655451+0000) 2024-03-04T15:34:23.656 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:34:23.655+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:34:23.655688+0000) 2024-03-04T15:34:24.656 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:34:24.655+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:34:24.655927+0000) 2024-03-04T15:34:25.301 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:34:25.313 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:34:25.656 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:34:25.655+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:34:25.656175+0000) 2024-03-04T15:34:26.657 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:34:26.655+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:34:26.656432+0000) 2024-03-04T15:34:27.657 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:34:27.656+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:34:27.656676+0000) 2024-03-04T15:34:28.657 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:34:28.656+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:34:28.656940+0000) 2024-03-04T15:34:29.657 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:34:29.656+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:34:29.657172+0000) 2024-03-04T15:34:30.658 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:34:30.656+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:34:30.657425+0000) 2024-03-04T15:34:31.658 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:34:31.657+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:34:31.657673+0000) 2024-03-04T15:34:32.658 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:34:32.657+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:34:32.657933+0000) 2024-03-04T15:34:33.659 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:34:33.657+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:34:33.658171+0000) 2024-03-04T15:34:34.659 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:34:34.657+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:34:34.658403+0000) 2024-03-04T15:34:35.659 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:34:35.657+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:34:35.658664+0000) 2024-03-04T15:34:36.659 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:34:36.658+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:34:36.658961+0000) 2024-03-04T15:34:37.659 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:34:37.658+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:34:37.659205+0000) 2024-03-04T15:34:38.660 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:34:38.659+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:34:38.659439+0000) 2024-03-04T15:34:39.660 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:34:39.659+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:34:39.659674+0000) 2024-03-04T15:34:40.660 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:34:40.659+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:34:40.659954+0000) 2024-03-04T15:34:41.660 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:34:41.659+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:34:41.660225+0000) 2024-03-04T15:34:42.661 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:34:42.660+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:34:42.660466+0000) 2024-03-04T15:34:43.661 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:34:43.660+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:34:43.660716+0000) 2024-03-04T15:34:44.661 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:34:44.660+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:34:44.660950+0000) 2024-03-04T15:34:45.661 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:34:45.660+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:34:45.661204+0000) 2024-03-04T15:34:46.662 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:34:46.661+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:34:46.661451+0000) 2024-03-04T15:34:47.662 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:34:47.661+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:34:47.661694+0000) 2024-03-04T15:34:48.662 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:34:48.661+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:34:48.661999+0000) 2024-03-04T15:34:49.663 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:34:49.661+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:34:49.662238+0000) 2024-03-04T15:34:50.663 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:34:50.662+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:34:50.662495+0000) 2024-03-04T15:34:51.663 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:34:51.662+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:34:51.662743+0000) 2024-03-04T15:34:52.663 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:34:52.662+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:34:52.662975+0000) 2024-03-04T15:34:53.663 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:34:53.663+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:34:53.663222+0000) 2024-03-04T15:34:54.664 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:34:54.663+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:34:54.663463+0000) 2024-03-04T15:34:55.523 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:34:55.531 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:34:55.664 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:34:55.663+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:34:55.663662+0000) 2024-03-04T15:34:56.664 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:34:56.663+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:34:56.663955+0000) 2024-03-04T15:34:57.664 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:34:57.664+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:34:57.664195+0000) 2024-03-04T15:34:58.665 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:34:58.663+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:34:58.664444+0000) 2024-03-04T15:34:59.665 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:34:59.663+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:34:59.664695+0000) 2024-03-04T15:35:00.665 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:35:00.663+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:35:00.664964+0000) 2024-03-04T15:35:01.666 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:35:01.664+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:35:01.665237+0000) 2024-03-04T15:35:02.666 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:35:02.664+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:35:02.665493+0000) 2024-03-04T15:35:03.666 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:35:03.664+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:35:03.665751+0000) 2024-03-04T15:35:04.666 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:35:04.664+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:35:04.666010+0000) 2024-03-04T15:35:05.667 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:35:05.665+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:35:05.666304+0000) 2024-03-04T15:35:06.667 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:35:06.665+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:35:06.666567+0000) 2024-03-04T15:35:07.667 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:35:07.665+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:35:07.666818+0000) 2024-03-04T15:35:08.668 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:35:08.666+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:35:08.667138+0000) 2024-03-04T15:35:09.668 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:35:09.666+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:35:09.667386+0000) 2024-03-04T15:35:10.668 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:35:10.666+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:35:10.667663+0000) 2024-03-04T15:35:11.668 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:35:11.667+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:35:11.667950+0000) 2024-03-04T15:35:12.668 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:35:12.667+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:35:12.668187+0000) 2024-03-04T15:35:13.669 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:35:13.667+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:35:13.668429+0000) 2024-03-04T15:35:14.669 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:35:14.667+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:35:14.668664+0000) 2024-03-04T15:35:15.669 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:35:15.668+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:35:15.668907+0000) 2024-03-04T15:35:16.669 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:35:16.668+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:35:16.669149+0000) 2024-03-04T15:35:17.670 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:35:17.668+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:35:17.669385+0000) 2024-03-04T15:35:18.670 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:35:18.668+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:35:18.669619+0000) 2024-03-04T15:35:19.670 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:35:19.669+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:35:19.669905+0000) 2024-03-04T15:35:20.670 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:35:20.669+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:35:20.670151+0000) 2024-03-04T15:35:21.671 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:35:21.669+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:35:21.670394+0000) 2024-03-04T15:35:22.671 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:35:22.669+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:35:22.670642+0000) 2024-03-04T15:35:23.671 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:35:23.670+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:35:23.670929+0000) 2024-03-04T15:35:24.671 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:35:24.670+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:35:24.671165+0000) 2024-03-04T15:35:25.671 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:35:25.672 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:35:25.670+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:35:25.671405+0000) 2024-03-04T15:35:25.679 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:35:26.672 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:35:26.670+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:35:26.671649+0000) 2024-03-04T15:35:27.672 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:35:27.671+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:35:27.671937+0000) 2024-03-04T15:35:28.673 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:35:28.671+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:35:28.672170+0000) 2024-03-04T15:35:29.673 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:35:29.671+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:35:29.672407+0000) 2024-03-04T15:35:30.673 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:35:30.672+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:35:30.672649+0000) 2024-03-04T15:35:31.673 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:35:31.672+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:35:31.672931+0000) 2024-03-04T15:35:32.673 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:35:32.672+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:35:32.673165+0000) 2024-03-04T15:35:33.674 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:35:33.672+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:35:33.673406+0000) 2024-03-04T15:35:34.674 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:35:34.673+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:35:34.673638+0000) 2024-03-04T15:35:35.674 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:35:35.673+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:35:35.673940+0000) 2024-03-04T15:35:36.674 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:35:36.673+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:35:36.674174+0000) 2024-03-04T15:35:37.675 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:35:37.673+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:35:37.674416+0000) 2024-03-04T15:35:38.675 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:35:38.674+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:35:38.674682+0000) 2024-03-04T15:35:39.675 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:35:39.674+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:35:39.674940+0000) 2024-03-04T15:35:40.675 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:35:40.674+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:35:40.675201+0000) 2024-03-04T15:35:41.676 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:35:41.674+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:35:41.675492+0000) 2024-03-04T15:35:42.676 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:35:42.675+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:35:42.675734+0000) 2024-03-04T15:35:43.676 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:35:43.675+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:35:43.675971+0000) 2024-03-04T15:35:44.676 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:35:44.675+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:35:44.676202+0000) 2024-03-04T15:35:45.677 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:35:45.675+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:35:45.676451+0000) 2024-03-04T15:35:46.677 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:35:46.675+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:35:46.676710+0000) 2024-03-04T15:35:47.677 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:35:47.676+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:35:47.676957+0000) 2024-03-04T15:35:48.678 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:35:48.676+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:35:48.677195+0000) 2024-03-04T15:35:49.678 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:35:49.677+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:35:49.677436+0000) 2024-03-04T15:35:50.678 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:35:50.677+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:35:50.677678+0000) 2024-03-04T15:35:51.678 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:35:51.677+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:35:51.677957+0000) 2024-03-04T15:35:52.679 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:35:52.677+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:35:52.678209+0000) 2024-03-04T15:35:53.679 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:35:53.677+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:35:53.678479+0000) 2024-03-04T15:35:54.679 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:35:54.678+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:35:54.678717+0000) 2024-03-04T15:35:55.679 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:35:55.678+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:35:55.678957+0000) 2024-03-04T15:35:55.818 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:35:55.826 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:35:56.680 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:35:56.678+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:35:56.679233+0000) 2024-03-04T15:35:57.680 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:35:57.678+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:35:57.679495+0000) 2024-03-04T15:35:58.680 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:35:58.679+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:35:58.679795+0000) 2024-03-04T15:35:59.681 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:35:59.679+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:35:59.680134+0000) 2024-03-04T15:36:00.681 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:36:00.680+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:36:00.680436+0000) 2024-03-04T15:36:01.681 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:36:01.680+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:36:01.680732+0000) 2024-03-04T15:36:02.681 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:36:02.680+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:36:02.680957+0000) 2024-03-04T15:36:03.682 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:36:03.679+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:36:03.681216+0000) 2024-03-04T15:36:04.682 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:36:04.679+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:36:04.681397+0000) 2024-03-04T15:36:05.682 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:36:05.680+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:36:05.681658+0000) 2024-03-04T15:36:06.682 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:36:06.680+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:36:06.681939+0000) 2024-03-04T15:36:07.683 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:36:07.681+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:36:07.682227+0000) 2024-03-04T15:36:08.683 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:36:08.681+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:36:08.682503+0000) 2024-03-04T15:36:09.683 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:36:09.681+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:36:09.682755+0000) 2024-03-04T15:36:10.683 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:36:10.681+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:36:10.682995+0000) 2024-03-04T15:36:11.684 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:36:11.681+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:36:11.683228+0000) 2024-03-04T15:36:12.684 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:36:12.682+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:36:12.683464+0000) 2024-03-04T15:36:13.684 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:36:13.682+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:36:13.683706+0000) 2024-03-04T15:36:14.684 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:36:14.683+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:36:14.683944+0000) 2024-03-04T15:36:15.684 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:36:15.683+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:36:15.684206+0000) 2024-03-04T15:36:16.685 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:36:16.683+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:36:16.684442+0000) 2024-03-04T15:36:17.685 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:36:17.683+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:36:17.684680+0000) 2024-03-04T15:36:18.685 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:36:18.684+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:36:18.684937+0000) 2024-03-04T15:36:19.685 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:36:19.684+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:36:19.685169+0000) 2024-03-04T15:36:20.686 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:36:20.684+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:36:20.685434+0000) 2024-03-04T15:36:21.686 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:36:21.684+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:36:21.685693+0000) 2024-03-04T15:36:22.686 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:36:22.685+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:36:22.685947+0000) 2024-03-04T15:36:23.686 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:36:23.685+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:36:23.686199+0000) 2024-03-04T15:36:24.687 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:36:24.685+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:36:24.686457+0000) 2024-03-04T15:36:25.687 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:36:25.685+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:36:25.686708+0000) 2024-03-04T15:36:26.030 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:36:26.038 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:36:26.687 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:36:26.686+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:36:26.686960+0000) 2024-03-04T15:36:27.687 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:36:27.686+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:36:27.687200+0000) 2024-03-04T15:36:28.688 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:36:28.686+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:36:28.687438+0000) 2024-03-04T15:36:29.688 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:36:29.686+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:36:29.687676+0000) 2024-03-04T15:36:30.688 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:36:30.687+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:36:30.687911+0000) 2024-03-04T15:36:31.689 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:36:31.687+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:36:31.688156+0000) 2024-03-04T15:36:32.689 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:36:32.687+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:36:32.688389+0000) 2024-03-04T15:36:33.689 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:36:33.687+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:36:33.688629+0000) 2024-03-04T15:36:34.689 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:36:34.688+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:36:34.688935+0000) 2024-03-04T15:36:35.690 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:36:35.688+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:36:35.689267+0000) 2024-03-04T15:36:36.690 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:36:36.688+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:36:36.689531+0000) 2024-03-04T15:36:37.690 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:36:37.689+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:36:37.689793+0000) 2024-03-04T15:36:38.690 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:36:38.689+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:36:38.690098+0000) 2024-03-04T15:36:39.691 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:36:39.689+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:36:39.690357+0000) 2024-03-04T15:36:40.691 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:36:40.689+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:36:40.690609+0000) 2024-03-04T15:36:41.691 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:36:41.690+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:36:41.690914+0000) 2024-03-04T15:36:42.691 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:36:42.690+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:36:42.691171+0000) 2024-03-04T15:36:43.692 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:36:43.690+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:36:43.691423+0000) 2024-03-04T15:36:44.692 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:36:44.690+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:36:44.691629+0000) 2024-03-04T15:36:45.692 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:36:45.691+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:36:45.691931+0000) 2024-03-04T15:36:46.693 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:36:46.691+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:36:46.692184+0000) 2024-03-04T15:36:47.693 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:36:47.691+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:36:47.692445+0000) 2024-03-04T15:36:48.693 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:36:48.692+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:36:48.692704+0000) 2024-03-04T15:36:49.693 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:36:49.692+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:36:49.692963+0000) 2024-03-04T15:36:50.694 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:36:50.692+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:36:50.693224+0000) 2024-03-04T15:36:51.694 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:36:51.693+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:36:51.693488+0000) 2024-03-04T15:36:52.694 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:36:52.693+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:36:52.693729+0000) 2024-03-04T15:36:53.694 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:36:53.693+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:36:53.693978+0000) 2024-03-04T15:36:54.694 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:36:54.693+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:36:54.694217+0000) 2024-03-04T15:36:55.695 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:36:55.694+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:36:55.694469+0000) 2024-03-04T15:36:56.186 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:36:56.194 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:36:56.695 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:36:56.694+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:36:56.694733+0000) 2024-03-04T15:36:57.695 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:36:57.694+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:36:57.694990+0000) 2024-03-04T15:36:58.696 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:36:58.694+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:36:58.695240+0000) 2024-03-04T15:36:59.696 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:36:59.695+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:36:59.695495+0000) 2024-03-04T15:37:00.696 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:37:00.695+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:37:00.695754+0000) 2024-03-04T15:37:01.696 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:37:01.695+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:37:01.696036+0000) 2024-03-04T15:37:02.697 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:37:02.696+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:37:02.696287+0000) 2024-03-04T15:37:03.697 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:37:03.695+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:37:03.696514+0000) 2024-03-04T15:37:04.697 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:37:04.696+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:37:04.696787+0000) 2024-03-04T15:37:05.697 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:37:05.696+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:37:05.697086+0000) 2024-03-04T15:37:06.698 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:37:06.697+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:37:06.697318+0000) 2024-03-04T15:37:07.698 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:37:07.697+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:37:07.697563+0000) 2024-03-04T15:37:08.698 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:37:08.696+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:37:08.697794+0000) 2024-03-04T15:37:09.698 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:37:09.696+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:37:09.698070+0000) 2024-03-04T15:37:10.699 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:37:10.697+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:37:10.698317+0000) 2024-03-04T15:37:11.699 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:37:11.697+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:37:11.698566+0000) 2024-03-04T15:37:12.699 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:37:12.697+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:37:12.698760+0000) 2024-03-04T15:37:13.699 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:37:13.697+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:37:13.698998+0000) 2024-03-04T15:37:14.700 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:37:14.698+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:37:14.699249+0000) 2024-03-04T15:37:15.700 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:37:15.697+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:37:15.699433+0000) 2024-03-04T15:37:16.700 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:37:16.698+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:37:16.699666+0000) 2024-03-04T15:37:17.700 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:37:17.698+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:37:17.699953+0000) 2024-03-04T15:37:18.701 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:37:18.699+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:37:18.700187+0000) 2024-03-04T15:37:19.701 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:37:19.699+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:37:19.700424+0000) 2024-03-04T15:37:20.701 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:37:20.699+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:37:20.700645+0000) 2024-03-04T15:37:21.701 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:37:21.700+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:37:21.700933+0000) 2024-03-04T15:37:22.701 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:37:22.700+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:37:22.701186+0000) 2024-03-04T15:37:23.702 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:37:23.700+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:37:23.701425+0000) 2024-03-04T15:37:24.702 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:37:24.700+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:37:24.701686+0000) 2024-03-04T15:37:25.702 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:37:25.701+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:37:25.701929+0000) 2024-03-04T15:37:26.415 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:37:26.423 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:37:26.702 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:37:26.701+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:37:26.702176+0000) 2024-03-04T15:37:27.703 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:37:27.701+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:37:27.702416+0000) 2024-03-04T15:37:28.703 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:37:28.701+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:37:28.702701+0000) 2024-03-04T15:37:29.703 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:37:29.702+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:37:29.702943+0000) 2024-03-04T15:37:30.703 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:37:30.702+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:37:30.703188+0000) 2024-03-04T15:37:31.704 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:37:31.702+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:37:31.703449+0000) 2024-03-04T15:37:32.704 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:37:32.702+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:37:32.703681+0000) 2024-03-04T15:37:33.704 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:37:33.703+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:37:33.703947+0000) 2024-03-04T15:37:34.705 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:37:34.703+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:37:34.704189+0000) 2024-03-04T15:37:35.705 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:37:35.703+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:37:35.704424+0000) 2024-03-04T15:37:36.705 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:37:36.703+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:37:36.704661+0000) 2024-03-04T15:37:37.705 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:37:37.704+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:37:37.704953+0000) 2024-03-04T15:37:38.706 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:37:38.704+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:37:38.705188+0000) 2024-03-04T15:37:39.706 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:37:39.704+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:37:39.705438+0000) 2024-03-04T15:37:40.706 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:37:40.704+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:37:40.705694+0000) 2024-03-04T15:37:41.706 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:37:41.705+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:37:41.705935+0000) 2024-03-04T15:37:42.707 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:37:42.705+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:37:42.706177+0000) 2024-03-04T15:37:43.707 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:37:43.705+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:37:43.706359+0000) 2024-03-04T15:37:44.707 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:37:44.705+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:37:44.706614+0000) 2024-03-04T15:37:45.707 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:37:45.706+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:37:45.706887+0000) 2024-03-04T15:37:46.707 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:37:46.706+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:37:46.707122+0000) 2024-03-04T15:37:47.708 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:37:47.706+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:37:47.707367+0000) 2024-03-04T15:37:48.708 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:37:48.707+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:37:48.707601+0000) 2024-03-04T15:37:49.708 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:37:49.707+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:37:49.707879+0000) 2024-03-04T15:37:50.708 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:37:50.707+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:37:50.708123+0000) 2024-03-04T15:37:51.709 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:37:51.707+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:37:51.708382+0000) 2024-03-04T15:37:52.709 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:37:52.708+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:37:52.708632+0000) 2024-03-04T15:37:53.709 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:37:53.708+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:37:53.708910+0000) 2024-03-04T15:37:54.709 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:37:54.708+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:37:54.709146+0000) 2024-03-04T15:37:55.710 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:37:55.708+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:37:55.709385+0000) 2024-03-04T15:37:56.627 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:37:56.638 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:37:56.710 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:37:56.708+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:37:56.709621+0000) 2024-03-04T15:37:57.710 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:37:57.709+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:37:57.709937+0000) 2024-03-04T15:37:58.710 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:37:58.709+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:37:58.710139+0000) 2024-03-04T15:37:59.711 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:37:59.710+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:37:59.710376+0000) 2024-03-04T15:38:00.711 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:38:00.710+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:38:00.710629+0000) 2024-03-04T15:38:01.711 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:38:01.710+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:38:01.710899+0000) 2024-03-04T15:38:02.711 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:38:02.710+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:38:02.711142+0000) 2024-03-04T15:38:03.712 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:38:03.711+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:38:03.711380+0000) 2024-03-04T15:38:04.712 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:38:04.710+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:38:04.711623+0000) 2024-03-04T15:38:05.712 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:38:05.711+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:38:05.711931+0000) 2024-03-04T15:38:06.713 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:38:06.711+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:38:06.712169+0000) 2024-03-04T15:38:07.713 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:38:07.712+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:38:07.712413+0000) 2024-03-04T15:38:08.713 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:38:08.712+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:38:08.712650+0000) 2024-03-04T15:38:09.713 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:38:09.712+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:38:09.712923+0000) 2024-03-04T15:38:10.713 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:38:10.713+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:38:10.713164+0000) 2024-03-04T15:38:11.714 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:38:11.713+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:38:11.713462+0000) 2024-03-04T15:38:12.714 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:38:12.712+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:38:12.713690+0000) 2024-03-04T15:38:13.714 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:38:13.712+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:38:13.713923+0000) 2024-03-04T15:38:14.715 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:38:14.713+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:38:14.714174+0000) 2024-03-04T15:38:15.715 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:38:15.713+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:38:15.714415+0000) 2024-03-04T15:38:16.715 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:38:16.712+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:38:16.714654+0000) 2024-03-04T15:38:17.715 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:38:17.713+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:38:17.714945+0000) 2024-03-04T15:38:18.715 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:38:18.714+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:38:18.715139+0000) 2024-03-04T15:38:19.716 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:38:19.714+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:38:19.715378+0000) 2024-03-04T15:38:20.716 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:38:20.713+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:38:20.715622+0000) 2024-03-04T15:38:21.716 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:38:21.714+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:38:21.715808+0000) 2024-03-04T15:38:22.716 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:38:22.714+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:38:22.716065+0000) 2024-03-04T15:38:23.717 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:38:23.715+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:38:23.716301+0000) 2024-03-04T15:38:24.717 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:38:24.715+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:38:24.716527+0000) 2024-03-04T15:38:25.717 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:38:25.715+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:38:25.716763+0000) 2024-03-04T15:38:26.717 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:38:26.716+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:38:26.717006+0000) 2024-03-04T15:38:26.850 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:38:26.859 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:38:27.717 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:38:27.716+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:38:27.717251+0000) 2024-03-04T15:38:28.718 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:38:28.716+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:38:28.717480+0000) 2024-03-04T15:38:29.718 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:38:29.716+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:38:29.717716+0000) 2024-03-04T15:38:30.718 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:38:30.717+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:38:30.717963+0000) 2024-03-04T15:38:31.719 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:38:31.717+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:38:31.718211+0000) 2024-03-04T15:38:32.719 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:38:32.717+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:38:32.718459+0000) 2024-03-04T15:38:33.719 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:38:33.717+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:38:33.718693+0000) 2024-03-04T15:38:34.719 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:38:34.718+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:38:34.718923+0000) 2024-03-04T15:38:35.719 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:38:35.718+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:38:35.719180+0000) 2024-03-04T15:38:36.720 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:38:36.718+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:38:36.719450+0000) 2024-03-04T15:38:37.720 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:38:37.718+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:38:37.719705+0000) 2024-03-04T15:38:38.720 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:38:38.719+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:38:38.719959+0000) 2024-03-04T15:38:39.720 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:38:39.719+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:38:39.720200+0000) 2024-03-04T15:38:40.721 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:38:40.719+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:38:40.720440+0000) 2024-03-04T15:38:41.721 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:38:41.720+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:38:41.720690+0000) 2024-03-04T15:38:42.721 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:38:42.720+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:38:42.720934+0000) 2024-03-04T15:38:43.721 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:38:43.720+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:38:43.721170+0000) 2024-03-04T15:38:44.722 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:38:44.720+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:38:44.721394+0000) 2024-03-04T15:38:45.722 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:38:45.721+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:38:45.721632+0000) 2024-03-04T15:38:46.722 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:38:46.721+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:38:46.721934+0000) 2024-03-04T15:38:47.723 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:38:47.721+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:38:47.722130+0000) 2024-03-04T15:38:48.723 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:38:48.721+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:38:48.722362+0000) 2024-03-04T15:38:49.723 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:38:49.721+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:38:49.722614+0000) 2024-03-04T15:38:50.724 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:38:50.722+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:38:50.722907+0000) 2024-03-04T15:38:51.724 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:38:51.722+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:38:51.723159+0000) 2024-03-04T15:38:52.724 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:38:52.722+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:38:52.723404+0000) 2024-03-04T15:38:53.724 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:38:53.723+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:38:53.723642+0000) 2024-03-04T15:38:54.724 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:38:54.723+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:38:54.723923+0000) 2024-03-04T15:38:55.724 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:38:55.723+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:38:55.724160+0000) 2024-03-04T15:38:56.725 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:38:56.723+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:38:56.724403+0000) 2024-03-04T15:38:57.141 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:38:57.149 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:38:57.725 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:38:57.723+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:38:57.724652+0000) 2024-03-04T15:38:58.725 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:38:58.724+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:38:58.724936+0000) 2024-03-04T15:38:59.725 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:38:59.724+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:38:59.725217+0000) 2024-03-04T15:39:00.726 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:39:00.725+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:39:00.725455+0000) 2024-03-04T15:39:01.726 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:39:01.725+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:39:01.725710+0000) 2024-03-04T15:39:02.726 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:39:02.725+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:39:02.725956+0000) 2024-03-04T15:39:03.726 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:39:03.725+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:39:03.726193+0000) 2024-03-04T15:39:04.727 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:39:04.726+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:39:04.726423+0000) 2024-03-04T15:39:05.727 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:39:05.725+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:39:05.726691+0000) 2024-03-04T15:39:06.727 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:39:06.726+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:39:06.726931+0000) 2024-03-04T15:39:07.728 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:39:07.726+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:39:07.727191+0000) 2024-03-04T15:39:08.728 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:39:08.727+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:39:08.727424+0000) 2024-03-04T15:39:09.728 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:39:09.727+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:39:09.727672+0000) 2024-03-04T15:39:10.728 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:39:10.727+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:39:10.727919+0000) 2024-03-04T15:39:11.728 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:39:11.727+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:39:11.728167+0000) 2024-03-04T15:39:12.729 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:39:12.728+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:39:12.728415+0000) 2024-03-04T15:39:13.729 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:39:13.728+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:39:13.728650+0000) 2024-03-04T15:39:14.729 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:39:14.728+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:39:14.728943+0000) 2024-03-04T15:39:15.730 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:39:15.729+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:39:15.729214+0000) 2024-03-04T15:39:16.731 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:39:16.728+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:39:16.729403+0000) 2024-03-04T15:39:17.730 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:39:17.728+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:39:17.729660+0000) 2024-03-04T15:39:18.730 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:39:18.728+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:39:18.729932+0000) 2024-03-04T15:39:19.730 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:39:19.729+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:39:19.730174+0000) 2024-03-04T15:39:20.731 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:39:20.728+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:39:20.730430+0000) 2024-03-04T15:39:21.731 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:39:21.729+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:39:21.730677+0000) 2024-03-04T15:39:22.731 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:39:22.729+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:39:22.730961+0000) 2024-03-04T15:39:23.732 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:39:23.730+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:39:23.731194+0000) 2024-03-04T15:39:24.732 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:39:24.730+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:39:24.731423+0000) 2024-03-04T15:39:25.732 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:39:25.730+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:39:25.731666+0000) 2024-03-04T15:39:26.732 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:39:26.730+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:39:26.731967+0000) 2024-03-04T15:39:27.352 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:39:27.360 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:39:27.732 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:39:27.731+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:39:27.732224+0000) 2024-03-04T15:39:28.733 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:39:28.731+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:39:28.732456+0000) 2024-03-04T15:39:29.733 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:39:29.731+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:39:29.732703+0000) 2024-03-04T15:39:30.733 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:39:30.732+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:39:30.732943+0000) 2024-03-04T15:39:31.733 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:39:31.732+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:39:31.733196+0000) 2024-03-04T15:39:32.734 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:39:32.732+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:39:32.733430+0000) 2024-03-04T15:39:33.734 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:39:33.732+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:39:33.733671+0000) 2024-03-04T15:39:34.734 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:39:34.733+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:39:34.733940+0000) 2024-03-04T15:39:35.735 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:39:35.733+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:39:35.734197+0000) 2024-03-04T15:39:36.735 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:39:36.733+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:39:36.734479+0000) 2024-03-04T15:39:37.735 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:39:37.733+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:39:37.734750+0000) 2024-03-04T15:39:38.735 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:39:38.734+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:39:38.734986+0000) 2024-03-04T15:39:39.736 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:39:39.734+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:39:39.735231+0000) 2024-03-04T15:39:40.736 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:39:40.734+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:39:40.735467+0000) 2024-03-04T15:39:41.736 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:39:41.735+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:39:41.735749+0000) 2024-03-04T15:39:42.736 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:39:42.735+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:39:42.735995+0000) 2024-03-04T15:39:43.737 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:39:43.735+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:39:43.736248+0000) 2024-03-04T15:39:44.737 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:39:44.735+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:39:44.736477+0000) 2024-03-04T15:39:45.737 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:39:45.736+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:39:45.736714+0000) 2024-03-04T15:39:46.737 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:39:46.736+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:39:46.736955+0000) 2024-03-04T15:39:47.738 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:39:47.736+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:39:47.737206+0000) 2024-03-04T15:39:48.738 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:39:48.736+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:39:48.737434+0000) 2024-03-04T15:39:49.738 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:39:49.737+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:39:49.737668+0000) 2024-03-04T15:39:50.739 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:39:50.737+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:39:50.737941+0000) 2024-03-04T15:39:51.739 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:39:51.737+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:39:51.738178+0000) 2024-03-04T15:39:52.739 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:39:52.737+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:39:52.738427+0000) 2024-03-04T15:39:53.739 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:39:53.738+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:39:53.738660+0000) 2024-03-04T15:39:54.739 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:39:54.738+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:39:54.738924+0000) 2024-03-04T15:39:55.739 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:39:55.738+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:39:55.739162+0000) 2024-03-04T15:39:56.740 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:39:56.738+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:39:56.739387+0000) 2024-03-04T15:39:57.532 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:39:57.540 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:39:57.740 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:39:57.738+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:39:57.739624+0000) 2024-03-04T15:39:58.740 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:39:58.739+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:39:58.739910+0000) 2024-03-04T15:39:59.741 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:39:59.739+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:39:59.740145+0000) 2024-03-04T15:40:00.741 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:40:00.739+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:40:00.740387+0000) 2024-03-04T15:40:01.741 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:40:01.740+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:40:01.740626+0000) 2024-03-04T15:40:02.741 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:40:02.740+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:40:02.740918+0000) 2024-03-04T15:40:03.741 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:40:03.740+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:40:03.741162+0000) 2024-03-04T15:40:04.742 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:40:04.740+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:40:04.741330+0000) 2024-03-04T15:40:05.742 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:40:05.741+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:40:05.741573+0000) 2024-03-04T15:40:06.742 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:40:06.741+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:40:06.741854+0000) 2024-03-04T15:40:07.743 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:40:07.741+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:40:07.742131+0000) 2024-03-04T15:40:08.743 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:40:08.742+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:40:08.742371+0000) 2024-03-04T15:40:09.743 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:40:09.742+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:40:09.742607+0000) 2024-03-04T15:40:10.743 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:40:10.742+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:40:10.742913+0000) 2024-03-04T15:40:11.744 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:40:11.742+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:40:11.743162+0000) 2024-03-04T15:40:12.744 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:40:12.742+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:40:12.743384+0000) 2024-03-04T15:40:13.744 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:40:13.743+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:40:13.743620+0000) 2024-03-04T15:40:14.744 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:40:14.743+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:40:14.743890+0000) 2024-03-04T15:40:15.744 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:40:15.743+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:40:15.744123+0000) 2024-03-04T15:40:16.745 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:40:16.744+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:40:16.744368+0000) 2024-03-04T15:40:17.745 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:40:17.744+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:40:17.744611+0000) 2024-03-04T15:40:18.745 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:40:18.744+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:40:18.744907+0000) 2024-03-04T15:40:19.746 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:40:19.745+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:40:19.745147+0000) 2024-03-04T15:40:20.746 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:40:20.744+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:40:20.745387+0000) 2024-03-04T15:40:21.746 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:40:21.744+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:40:21.745621+0000) 2024-03-04T15:40:22.746 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:40:22.744+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:40:22.745922+0000) 2024-03-04T15:40:23.747 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:40:23.745+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:40:23.746161+0000) 2024-03-04T15:40:24.747 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:40:24.745+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:40:24.746392+0000) 2024-03-04T15:40:25.747 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:40:25.745+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:40:25.746630+0000) 2024-03-04T15:40:26.747 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:40:26.745+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:40:26.746897+0000) 2024-03-04T15:40:27.745 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:40:27.747 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:40:27.746+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:40:27.747132+0000) 2024-03-04T15:40:27.753 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:40:28.748 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:40:28.746+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:40:28.747368+0000) 2024-03-04T15:40:29.748 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:40:29.746+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:40:29.747611+0000) 2024-03-04T15:40:30.748 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:40:30.746+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:40:30.747895+0000) 2024-03-04T15:40:31.748 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:40:31.747+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:40:31.748140+0000) 2024-03-04T15:40:32.749 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:40:32.747+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:40:32.748378+0000) 2024-03-04T15:40:33.749 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:40:33.747+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:40:33.748612+0000) 2024-03-04T15:40:34.749 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:40:34.748+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:40:34.748880+0000) 2024-03-04T15:40:35.750 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:40:35.748+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:40:35.749124+0000) 2024-03-04T15:40:36.750 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:40:36.748+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:40:36.749359+0000) 2024-03-04T15:40:37.750 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:40:37.748+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:40:37.749616+0000) 2024-03-04T15:40:38.750 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:40:38.749+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:40:38.749887+0000) 2024-03-04T15:40:39.750 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:40:39.749+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:40:39.750128+0000) 2024-03-04T15:40:40.751 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:40:40.749+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:40:40.750327+0000) 2024-03-04T15:40:41.751 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:40:41.749+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:40:41.750586+0000) 2024-03-04T15:40:42.751 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:40:42.750+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:40:42.750848+0000) 2024-03-04T15:40:43.751 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:40:43.750+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:40:43.751090+0000) 2024-03-04T15:40:44.752 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:40:44.750+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:40:44.751323+0000) 2024-03-04T15:40:45.752 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:40:45.750+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:40:45.751613+0000) 2024-03-04T15:40:46.752 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:40:46.751+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:40:46.751878+0000) 2024-03-04T15:40:47.753 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:40:47.751+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:40:47.752172+0000) 2024-03-04T15:40:48.753 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:40:48.751+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:40:48.752430+0000) 2024-03-04T15:40:49.753 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:40:49.751+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:40:49.752684+0000) 2024-03-04T15:40:50.753 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:40:50.752+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:40:50.752963+0000) 2024-03-04T15:40:51.754 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:40:51.752+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:40:51.753211+0000) 2024-03-04T15:40:52.754 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:40:52.752+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:40:52.753441+0000) 2024-03-04T15:40:53.754 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:40:53.753+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:40:53.753709+0000) 2024-03-04T15:40:54.754 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:40:54.753+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:40:54.753932+0000) 2024-03-04T15:40:55.755 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:40:55.753+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:40:55.754169+0000) 2024-03-04T15:40:56.755 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:40:56.753+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:40:56.754396+0000) 2024-03-04T15:40:57.755 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:40:57.754+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:40:57.754642+0000) 2024-03-04T15:40:57.883 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:40:57.892 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:40:58.756 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:40:58.754+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:40:58.754948+0000) 2024-03-04T15:40:59.756 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:40:59.754+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:40:59.755191+0000) 2024-03-04T15:41:00.756 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:41:00.754+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:41:00.755435+0000) 2024-03-04T15:41:01.756 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:41:01.755+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:41:01.755677+0000) 2024-03-04T15:41:02.756 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:41:02.755+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:41:02.755948+0000) 2024-03-04T15:41:03.757 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:41:03.755+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:41:03.756182+0000) 2024-03-04T15:41:04.757 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:41:04.756+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:41:04.756409+0000) 2024-03-04T15:41:05.757 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:41:05.756+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:41:05.756647+0000) 2024-03-04T15:41:06.757 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:41:06.756+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:41:06.756938+0000) 2024-03-04T15:41:07.758 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:41:07.756+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:41:07.757197+0000) 2024-03-04T15:41:08.758 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:41:08.757+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:41:08.757436+0000) 2024-03-04T15:41:09.758 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:41:09.757+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:41:09.757674+0000) 2024-03-04T15:41:10.759 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:41:10.757+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:41:10.757945+0000) 2024-03-04T15:41:11.759 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:41:11.757+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:41:11.758183+0000) 2024-03-04T15:41:12.759 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:41:12.758+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:41:12.758419+0000) 2024-03-04T15:41:13.759 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:41:13.757+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:41:13.758665+0000) 2024-03-04T15:41:14.760 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:41:14.758+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:41:14.758931+0000) 2024-03-04T15:41:15.760 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:41:15.758+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:41:15.759123+0000) 2024-03-04T15:41:16.760 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:41:16.758+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:41:16.759296+0000) 2024-03-04T15:41:17.760 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:41:17.759+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:41:17.759498+0000) 2024-03-04T15:41:18.760 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:41:18.759+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:41:18.759765+0000) 2024-03-04T15:41:19.760 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:41:19.759+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:41:19.760005+0000) 2024-03-04T15:41:20.761 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:41:20.760+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:41:20.760240+0000) 2024-03-04T15:41:21.761 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:41:21.758+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:41:21.760420+0000) 2024-03-04T15:41:22.761 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:41:22.759+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:41:22.760656+0000) 2024-03-04T15:41:23.761 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:41:23.759+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:41:23.760887+0000) 2024-03-04T15:41:24.762 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:41:24.760+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:41:24.761114+0000) 2024-03-04T15:41:25.762 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:41:25.760+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:41:25.761350+0000) 2024-03-04T15:41:26.762 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:41:26.760+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:41:26.761579+0000) 2024-03-04T15:41:27.762 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:41:27.760+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:41:27.761864+0000) 2024-03-04T15:41:28.123 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:41:28.131 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:41:28.762 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:41:28.761+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:41:28.762107+0000) 2024-03-04T15:41:29.763 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:41:29.761+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:41:29.762345+0000) 2024-03-04T15:41:30.763 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:41:30.761+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:41:30.762605+0000) 2024-03-04T15:41:31.763 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:41:31.761+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:41:31.762891+0000) 2024-03-04T15:41:32.764 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:41:32.762+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:41:32.763127+0000) 2024-03-04T15:41:33.764 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:41:33.762+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:41:33.763367+0000) 2024-03-04T15:41:34.764 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:41:34.762+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:41:34.763613+0000) 2024-03-04T15:41:35.764 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:41:35.763+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:41:35.763931+0000) 2024-03-04T15:41:36.765 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:41:36.763+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:41:36.764150+0000) 2024-03-04T15:41:37.765 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:41:37.763+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:41:37.764413+0000) 2024-03-04T15:41:38.765 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:41:38.763+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:41:38.764677+0000) 2024-03-04T15:41:39.765 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:41:39.764+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:41:39.764962+0000) 2024-03-04T15:41:40.766 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:41:40.764+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:41:40.765164+0000) 2024-03-04T15:41:41.766 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:41:41.764+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:41:41.765416+0000) 2024-03-04T15:41:42.766 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:41:42.764+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:41:42.765657+0000) 2024-03-04T15:41:43.766 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:41:43.765+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:41:43.765930+0000) 2024-03-04T15:41:44.767 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:41:44.765+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:41:44.766159+0000) 2024-03-04T15:41:45.767 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:41:45.765+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:41:45.766387+0000) 2024-03-04T15:41:46.767 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:41:46.765+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:41:46.766631+0000) 2024-03-04T15:41:47.767 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:41:47.766+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:41:47.766935+0000) 2024-03-04T15:41:48.768 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:41:48.766+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:41:48.767115+0000) 2024-03-04T15:41:49.768 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:41:49.766+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:41:49.767353+0000) 2024-03-04T15:41:50.768 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:41:50.766+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:41:50.767599+0000) 2024-03-04T15:41:51.768 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:41:51.767+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:41:51.767908+0000) 2024-03-04T15:41:52.769 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:41:52.767+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:41:52.768142+0000) 2024-03-04T15:41:53.769 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:41:53.767+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:41:53.768329+0000) 2024-03-04T15:41:54.769 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:41:54.767+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:41:54.768555+0000) 2024-03-04T15:41:55.769 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:41:55.768+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:41:55.768790+0000) 2024-03-04T15:41:56.769 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:41:56.768+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:41:56.769021+0000) 2024-03-04T15:41:57.770 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:41:57.768+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:41:57.769263+0000) 2024-03-04T15:41:58.259 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:41:58.271 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:41:58.770 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:41:58.768+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:41:58.769501+0000) 2024-03-04T15:41:59.770 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:41:59.769+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:41:59.769763+0000) 2024-03-04T15:42:00.770 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:42:00.769+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:42:00.770022+0000) 2024-03-04T15:42:01.771 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:42:01.769+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:42:01.770280+0000) 2024-03-04T15:42:02.771 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:42:02.770+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:42:02.770531+0000) 2024-03-04T15:42:03.771 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:42:03.770+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:42:03.770782+0000) 2024-03-04T15:42:04.772 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:42:04.770+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:42:04.771058+0000) 2024-03-04T15:42:05.772 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:42:05.770+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:42:05.771348+0000) 2024-03-04T15:42:06.772 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:42:06.770+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:42:06.771571+0000) 2024-03-04T15:42:07.772 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:42:07.771+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:42:07.771814+0000) 2024-03-04T15:42:08.773 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:42:08.771+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:42:08.772037+0000) 2024-03-04T15:42:09.773 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:42:09.771+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:42:09.772297+0000) 2024-03-04T15:42:10.773 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:42:10.772+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:42:10.772536+0000) 2024-03-04T15:42:11.773 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:42:11.772+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:42:11.772779+0000) 2024-03-04T15:42:12.774 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:42:12.772+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:42:12.773012+0000) 2024-03-04T15:42:13.774 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:42:13.772+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:42:13.773252+0000) 2024-03-04T15:42:14.774 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:42:14.773+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:42:14.773477+0000) 2024-03-04T15:42:15.775 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:42:15.773+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:42:15.773715+0000) 2024-03-04T15:42:16.774 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:42:16.773+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:42:16.773969+0000) 2024-03-04T15:42:17.775 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:42:17.774+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:42:17.774216+0000) 2024-03-04T15:42:18.775 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:42:18.774+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:42:18.774449+0000) 2024-03-04T15:42:19.775 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:42:19.774+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:42:19.774691+0000) 2024-03-04T15:42:20.775 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:42:20.774+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:42:20.774925+0000) 2024-03-04T15:42:21.776 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:42:21.775+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:42:21.775216+0000) 2024-03-04T15:42:22.776 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:42:22.775+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:42:22.775456+0000) 2024-03-04T15:42:23.776 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:42:23.773+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:42:23.775704+0000) 2024-03-04T15:42:24.776 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:42:24.774+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:42:24.775939+0000) 2024-03-04T15:42:25.777 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:42:25.775+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:42:25.776178+0000) 2024-03-04T15:42:26.777 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:42:26.775+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:42:26.776406+0000) 2024-03-04T15:42:27.777 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:42:27.774+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:42:27.776615+0000) 2024-03-04T15:42:28.456 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:42:28.465 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:42:28.777 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:42:28.775+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:42:28.776879+0000) 2024-03-04T15:42:29.778 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:42:29.775+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:42:29.777131+0000) 2024-03-04T15:42:30.778 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:42:30.776+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:42:30.777407+0000) 2024-03-04T15:42:31.778 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:42:31.776+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:42:31.777638+0000) 2024-03-04T15:42:32.779 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:42:32.776+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:42:32.777937+0000) 2024-03-04T15:42:33.779 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:42:33.777+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:42:33.778157+0000) 2024-03-04T15:42:34.779 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:42:34.777+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:42:34.778414+0000) 2024-03-04T15:42:35.779 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:42:35.777+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:42:35.778659+0000) 2024-03-04T15:42:36.779 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:42:36.778+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:42:36.778925+0000) 2024-03-04T15:42:37.780 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:42:37.778+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:42:37.779178+0000) 2024-03-04T15:42:38.780 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:42:38.778+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:42:38.779437+0000) 2024-03-04T15:42:39.780 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:42:39.778+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:42:39.779706+0000) 2024-03-04T15:42:40.780 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:42:40.779+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:42:40.779946+0000) 2024-03-04T15:42:41.781 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:42:41.779+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:42:41.780188+0000) 2024-03-04T15:42:42.781 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:42:42.779+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:42:42.780427+0000) 2024-03-04T15:42:43.781 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:42:43.779+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:42:43.780668+0000) 2024-03-04T15:42:44.781 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:42:44.780+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:42:44.780934+0000) 2024-03-04T15:42:45.782 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:42:45.780+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:42:45.781172+0000) 2024-03-04T15:42:46.782 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:42:46.780+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:42:46.781423+0000) 2024-03-04T15:42:47.782 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:42:47.780+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:42:47.781669+0000) 2024-03-04T15:42:48.782 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:42:48.781+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:42:48.781944+0000) 2024-03-04T15:42:49.783 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:42:49.781+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:42:49.782187+0000) 2024-03-04T15:42:50.783 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:42:50.781+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:42:50.782373+0000) 2024-03-04T15:42:51.783 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:42:51.781+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:42:51.782614+0000) 2024-03-04T15:42:52.783 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:42:52.781+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:42:52.782803+0000) 2024-03-04T15:42:53.784 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:42:53.782+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:42:53.783097+0000) 2024-03-04T15:42:54.784 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:42:54.782+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:42:54.783322+0000) 2024-03-04T15:42:55.784 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:42:55.782+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:42:55.783566+0000) 2024-03-04T15:42:56.784 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:42:56.783+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:42:56.783813+0000) 2024-03-04T15:42:57.785 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:42:57.783+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:42:57.784100+0000) 2024-03-04T15:42:58.661 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:42:58.669 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:42:58.806 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:42:58.783+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:42:58.784345+0000) 2024-03-04T15:42:59.785 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:42:59.784+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:42:59.784585+0000) 2024-03-04T15:43:00.785 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:43:00.784+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:43:00.784865+0000) 2024-03-04T15:43:01.786 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:43:01.784+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:43:01.785105+0000) 2024-03-04T15:43:02.786 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:43:02.784+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:43:02.785343+0000) 2024-03-04T15:43:03.786 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:43:03.785+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:43:03.785579+0000) 2024-03-04T15:43:04.786 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:43:04.785+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:43:04.785878+0000) 2024-03-04T15:43:05.787 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:43:05.785+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:43:05.786114+0000) 2024-03-04T15:43:06.787 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:43:06.785+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:43:06.786361+0000) 2024-03-04T15:43:07.788 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:43:07.786+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:43:07.786570+0000) 2024-03-04T15:43:08.787 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:43:08.786+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:43:08.786809+0000) 2024-03-04T15:43:09.788 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:43:09.786+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:43:09.787163+0000) 2024-03-04T15:43:10.788 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:43:10.787+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:43:10.787421+0000) 2024-03-04T15:43:11.788 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:43:11.787+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:43:11.787657+0000) 2024-03-04T15:43:12.788 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:43:12.787+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:43:12.787941+0000) 2024-03-04T15:43:13.789 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:43:13.787+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:43:13.788173+0000) 2024-03-04T15:43:14.789 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:43:14.788+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:43:14.788402+0000) 2024-03-04T15:43:15.789 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:43:15.787+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:43:15.788590+0000) 2024-03-04T15:43:16.790 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:43:16.788+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:43:16.788815+0000) 2024-03-04T15:43:17.790 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:43:17.788+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:43:17.789134+0000) 2024-03-04T15:43:18.790 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:43:18.789+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:43:18.789367+0000) 2024-03-04T15:43:19.790 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:43:19.789+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:43:19.789602+0000) 2024-03-04T15:43:20.790 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:43:20.789+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:43:20.789902+0000) 2024-03-04T15:43:21.791 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:43:21.789+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:43:21.790105+0000) 2024-03-04T15:43:22.791 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:43:22.790+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:43:22.790349+0000) 2024-03-04T15:43:23.791 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:43:23.790+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:43:23.790587+0000) 2024-03-04T15:43:24.791 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:43:24.790+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:43:24.790760+0000) 2024-03-04T15:43:25.792 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:43:25.790+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:43:25.791000+0000) 2024-03-04T15:43:26.792 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:43:26.790+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:43:26.791227+0000) 2024-03-04T15:43:27.792 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:43:27.790+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:43:27.791481+0000) 2024-03-04T15:43:28.792 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:43:28.790+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:43:28.791721+0000) 2024-03-04T15:43:28.875 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:43:28.883 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:43:29.793 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:43:29.790+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:43:29.791959+0000) 2024-03-04T15:43:30.793 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:43:30.791+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:43:30.792202+0000) 2024-03-04T15:43:31.793 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:43:31.791+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:43:31.792444+0000) 2024-03-04T15:43:32.793 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:43:32.791+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:43:32.792716+0000) 2024-03-04T15:43:33.794 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:43:33.791+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:43:33.792954+0000) 2024-03-04T15:43:34.794 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:43:34.792+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:43:34.793188+0000) 2024-03-04T15:43:35.794 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:43:35.792+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:43:35.793430+0000) 2024-03-04T15:43:36.794 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:43:36.792+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:43:36.793663+0000) 2024-03-04T15:43:37.794 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:43:37.793+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:43:37.793897+0000) 2024-03-04T15:43:38.795 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:43:38.793+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:43:38.794137+0000) 2024-03-04T15:43:39.795 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:43:39.793+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:43:39.794372+0000) 2024-03-04T15:43:40.795 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:43:40.793+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:43:40.794634+0000) 2024-03-04T15:43:41.796 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:43:41.794+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:43:41.794931+0000) 2024-03-04T15:43:42.796 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:43:42.794+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:43:42.795182+0000) 2024-03-04T15:43:43.796 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:43:43.794+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:43:43.795417+0000) 2024-03-04T15:43:44.796 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:43:44.794+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:43:44.795649+0000) 2024-03-04T15:43:45.796 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:43:45.795+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:43:45.795926+0000) 2024-03-04T15:43:46.797 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:43:46.795+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:43:46.796159+0000) 2024-03-04T15:43:47.797 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:43:47.795+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:43:47.796411+0000) 2024-03-04T15:43:48.797 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:43:48.795+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:43:48.796652+0000) 2024-03-04T15:43:49.798 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:43:49.796+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:43:49.796934+0000) 2024-03-04T15:43:50.798 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:43:50.796+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:43:50.797138+0000) 2024-03-04T15:43:51.798 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:43:51.796+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:43:51.797377+0000) 2024-03-04T15:43:52.798 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:43:52.796+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:43:52.797577+0000) 2024-03-04T15:43:53.799 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:43:53.797+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:43:53.797814+0000) 2024-03-04T15:43:54.799 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:43:54.797+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:43:54.798098+0000) 2024-03-04T15:43:55.799 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:43:55.797+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:43:55.798292+0000) 2024-03-04T15:43:56.799 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:43:56.797+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:43:56.798529+0000) 2024-03-04T15:43:57.799 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:43:57.797+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:43:57.798750+0000) 2024-03-04T15:43:58.800 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:43:58.798+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:43:58.798993+0000) 2024-03-04T15:43:59.013 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:43:59.021 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:43:59.800 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:43:59.798+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:43:59.799193+0000) 2024-03-04T15:44:00.800 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:44:00.798+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:44:00.799438+0000) 2024-03-04T15:44:01.800 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:44:01.799+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:44:01.799679+0000) 2024-03-04T15:44:02.801 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:44:02.799+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:44:02.799954+0000) 2024-03-04T15:44:03.801 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:44:03.799+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:44:03.800193+0000) 2024-03-04T15:44:04.801 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:44:04.799+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:44:04.800432+0000) 2024-03-04T15:44:05.802 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:44:05.800+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:44:05.800678+0000) 2024-03-04T15:44:06.801 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:44:06.800+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:44:06.800947+0000) 2024-03-04T15:44:07.802 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:44:07.800+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:44:07.801151+0000) 2024-03-04T15:44:08.802 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:44:08.800+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:44:08.801444+0000) 2024-03-04T15:44:09.802 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:44:09.801+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:44:09.801673+0000) 2024-03-04T15:44:10.803 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:44:10.801+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:44:10.801920+0000) 2024-03-04T15:44:11.803 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:44:11.801+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:44:11.802179+0000) 2024-03-04T15:44:12.803 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:44:12.801+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:44:12.802378+0000) 2024-03-04T15:44:13.803 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:44:13.802+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:44:13.802623+0000) 2024-03-04T15:44:14.804 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:44:14.802+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:44:14.802902+0000) 2024-03-04T15:44:15.804 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:44:15.802+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:44:15.803140+0000) 2024-03-04T15:44:16.804 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:44:16.803+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:44:16.803375+0000) 2024-03-04T15:44:17.804 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:44:17.803+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:44:17.803624+0000) 2024-03-04T15:44:18.805 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:44:18.803+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:44:18.803900+0000) 2024-03-04T15:44:19.805 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:44:19.803+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:44:19.804142+0000) 2024-03-04T15:44:20.805 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:44:20.804+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:44:20.804380+0000) 2024-03-04T15:44:21.805 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:44:21.804+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:44:21.804624+0000) 2024-03-04T15:44:22.806 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:44:22.804+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:44:22.804906+0000) 2024-03-04T15:44:23.806 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:44:23.804+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:44:23.805145+0000) 2024-03-04T15:44:24.806 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:44:24.805+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:44:24.805381+0000) 2024-03-04T15:44:25.806 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:44:25.805+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:44:25.805620+0000) 2024-03-04T15:44:26.806 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:44:26.805+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:44:26.805906+0000) 2024-03-04T15:44:27.807 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:44:27.806+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:44:27.806156+0000) 2024-03-04T15:44:28.807 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:44:28.805+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:44:28.806393+0000) 2024-03-04T15:44:29.251 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:44:29.260 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:44:29.807 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:44:29.805+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:44:29.806673+0000) 2024-03-04T15:44:30.807 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:44:30.805+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:44:30.806948+0000) 2024-03-04T15:44:31.808 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:44:31.805+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:44:31.807187+0000) 2024-03-04T15:44:32.808 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:44:32.806+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:44:32.807421+0000) 2024-03-04T15:44:33.808 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:44:33.806+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:44:33.807667+0000) 2024-03-04T15:44:34.809 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:44:34.806+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:44:34.807940+0000) 2024-03-04T15:44:35.809 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:44:35.807+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:44:35.808175+0000) 2024-03-04T15:44:36.809 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:44:36.807+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:44:36.808408+0000) 2024-03-04T15:44:37.809 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:44:37.807+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:44:37.808644+0000) 2024-03-04T15:44:38.810 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:44:38.807+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:44:38.808935+0000) 2024-03-04T15:44:39.810 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:44:39.808+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:44:39.809179+0000) 2024-03-04T15:44:40.810 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:44:40.808+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:44:40.809425+0000) 2024-03-04T15:44:41.810 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:44:41.808+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:44:41.809610+0000) 2024-03-04T15:44:42.811 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:44:42.809+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:44:42.809912+0000) 2024-03-04T15:44:43.811 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:44:43.809+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:44:43.810153+0000) 2024-03-04T15:44:44.811 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:44:44.809+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:44:44.810333+0000) 2024-03-04T15:44:45.811 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:44:45.809+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:44:45.810578+0000) 2024-03-04T15:44:46.811 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:44:46.810+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:44:46.810810+0000) 2024-03-04T15:44:47.812 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:44:47.810+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:44:47.811091+0000) 2024-03-04T15:44:48.812 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:44:48.810+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:44:48.811329+0000) 2024-03-04T15:44:49.812 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:44:49.810+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:44:49.811582+0000) 2024-03-04T15:44:50.812 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:44:50.811+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:44:50.811859+0000) 2024-03-04T15:44:51.813 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:44:51.811+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:44:51.812093+0000) 2024-03-04T15:44:52.813 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:44:52.811+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:44:52.812325+0000) 2024-03-04T15:44:53.813 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:44:53.811+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:44:53.812566+0000) 2024-03-04T15:44:54.813 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:44:54.812+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:44:54.812804+0000) 2024-03-04T15:44:55.814 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:44:55.812+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:44:55.813094+0000) 2024-03-04T15:44:56.814 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:44:56.812+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:44:56.813383+0000) 2024-03-04T15:44:57.814 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:44:57.812+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:44:57.813620+0000) 2024-03-04T15:44:58.815 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:44:58.813+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:44:58.813896+0000) 2024-03-04T15:44:59.464 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:44:59.473 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:44:59.815 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:44:59.813+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:44:59.814150+0000) 2024-03-04T15:45:00.815 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:45:00.813+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:45:00.814411+0000) 2024-03-04T15:45:01.815 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:45:01.814+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:45:01.814653+0000) 2024-03-04T15:45:02.816 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:45:02.814+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:45:02.814929+0000) 2024-03-04T15:45:03.816 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:45:03.814+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:45:03.815162+0000) 2024-03-04T15:45:04.816 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:45:04.814+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:45:04.815395+0000) 2024-03-04T15:45:05.816 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:45:05.815+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:45:05.815635+0000) 2024-03-04T15:45:06.816 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:45:06.815+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:45:06.815813+0000) 2024-03-04T15:45:07.817 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:45:07.815+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:45:07.816126+0000) 2024-03-04T15:45:08.817 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:45:08.815+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:45:08.816358+0000) 2024-03-04T15:45:09.817 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:45:09.816+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:45:09.816602+0000) 2024-03-04T15:45:10.817 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:45:10.816+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:45:10.816899+0000) 2024-03-04T15:45:11.818 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:45:11.816+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:45:11.817145+0000) 2024-03-04T15:45:12.818 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:45:12.816+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:45:12.817396+0000) 2024-03-04T15:45:13.818 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:45:13.816+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:45:13.817646+0000) 2024-03-04T15:45:14.819 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:45:14.817+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:45:14.817928+0000) 2024-03-04T15:45:15.819 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:45:15.817+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:45:15.818165+0000) 2024-03-04T15:45:16.819 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:45:16.818+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:45:16.818400+0000) 2024-03-04T15:45:17.819 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:45:17.818+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:45:17.818642+0000) 2024-03-04T15:45:18.820 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:45:18.818+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:45:18.818917+0000) 2024-03-04T15:45:19.820 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:45:19.818+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:45:19.819156+0000) 2024-03-04T15:45:20.820 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:45:20.819+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:45:20.819407+0000) 2024-03-04T15:45:21.820 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:45:21.819+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:45:21.819657+0000) 2024-03-04T15:45:22.821 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:45:22.819+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:45:22.819945+0000) 2024-03-04T15:45:23.821 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:45:23.819+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:45:23.820197+0000) 2024-03-04T15:45:24.821 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:45:24.820+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:45:24.820451+0000) 2024-03-04T15:45:25.822 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:45:25.820+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:45:25.820709+0000) 2024-03-04T15:45:26.822 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:45:26.820+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:45:26.820991+0000) 2024-03-04T15:45:27.822 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:45:27.821+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:45:27.821247+0000) 2024-03-04T15:45:28.822 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:45:28.821+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:45:28.821489+0000) 2024-03-04T15:45:29.670 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:45:29.678 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:45:29.822 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:45:29.821+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:45:29.821782+0000) 2024-03-04T15:45:30.823 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:45:30.821+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:45:30.822088+0000) 2024-03-04T15:45:31.823 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:45:31.822+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:45:31.822339+0000) 2024-03-04T15:45:32.823 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:45:32.821+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:45:32.822587+0000) 2024-03-04T15:45:33.823 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:45:33.821+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:45:33.822879+0000) 2024-03-04T15:45:34.824 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:45:34.822+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:45:34.823139+0000) 2024-03-04T15:45:35.824 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:45:35.822+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:45:35.823392+0000) 2024-03-04T15:45:36.824 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:45:36.822+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:45:36.823654+0000) 2024-03-04T15:45:37.825 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:45:37.822+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:45:37.823953+0000) 2024-03-04T15:45:38.825 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:45:38.823+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:45:38.824227+0000) 2024-03-04T15:45:39.825 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:45:39.823+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:45:39.824550+0000) 2024-03-04T15:45:40.825 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:45:40.823+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:45:40.824865+0000) 2024-03-04T15:45:41.826 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:45:41.824+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:45:41.825148+0000) 2024-03-04T15:45:42.826 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:45:42.824+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:45:42.825418+0000) 2024-03-04T15:45:43.826 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:45:43.824+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:45:43.825672+0000) 2024-03-04T15:45:44.827 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:45:44.825+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:45:44.825927+0000) 2024-03-04T15:45:45.827 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:45:45.825+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:45:45.826179+0000) 2024-03-04T15:45:46.827 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:45:46.825+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:45:46.826433+0000) 2024-03-04T15:45:47.827 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:45:47.825+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:45:47.826689+0000) 2024-03-04T15:45:48.828 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:45:48.826+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:45:48.826941+0000) 2024-03-04T15:45:49.828 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:45:49.826+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:45:49.827197+0000) 2024-03-04T15:45:50.828 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:45:50.826+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:45:50.827464+0000) 2024-03-04T15:45:51.828 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:45:51.826+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:45:51.827717+0000) 2024-03-04T15:45:52.829 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:45:52.827+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:45:52.827978+0000) 2024-03-04T15:45:53.829 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:45:53.827+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:45:53.828231+0000) 2024-03-04T15:45:54.829 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:45:54.827+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:45:54.828485+0000) 2024-03-04T15:45:55.829 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:45:55.827+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:45:55.828731+0000) 2024-03-04T15:45:56.830 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:45:56.828+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:45:56.828967+0000) 2024-03-04T15:45:57.830 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:45:57.828+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:45:57.829155+0000) 2024-03-04T15:45:58.830 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:45:58.828+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:45:58.829389+0000) 2024-03-04T15:45:59.830 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:45:59.828+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:45:59.829630+0000) 2024-03-04T15:45:59.866 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:45:59.874 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:46:00.831 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:46:00.829+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:46:00.829963+0000) 2024-03-04T15:46:01.831 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:46:01.829+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:46:01.830223+0000) 2024-03-04T15:46:02.831 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:46:02.829+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:46:02.830475+0000) 2024-03-04T15:46:03.831 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:46:03.830+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:46:03.830728+0000) 2024-03-04T15:46:04.832 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:46:04.830+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:46:04.830983+0000) 2024-03-04T15:46:05.832 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:46:05.830+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:46:05.831238+0000) 2024-03-04T15:46:06.832 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:46:06.830+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:46:06.831442+0000) 2024-03-04T15:46:07.832 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:46:07.830+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:46:07.831716+0000) 2024-03-04T15:46:08.833 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:46:08.831+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:46:08.831963+0000) 2024-03-04T15:46:09.833 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:46:09.831+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:46:09.832209+0000) 2024-03-04T15:46:10.833 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:46:10.831+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:46:10.832492+0000) 2024-03-04T15:46:11.833 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:46:11.832+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:46:11.832768+0000) 2024-03-04T15:46:12.833 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:46:12.832+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:46:12.832970+0000) 2024-03-04T15:46:13.834 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:46:13.832+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:46:13.833227+0000) 2024-03-04T15:46:14.834 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:46:14.833+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:46:14.833470+0000) 2024-03-04T15:46:15.834 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:46:15.833+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:46:15.833744+0000) 2024-03-04T15:46:16.835 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:46:16.833+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:46:16.834019+0000) 2024-03-04T15:46:17.835 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:46:17.833+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:46:17.834258+0000) 2024-03-04T15:46:18.835 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:46:18.833+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:46:18.834508+0000) 2024-03-04T15:46:19.835 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:46:19.834+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:46:19.834752+0000) 2024-03-04T15:46:20.836 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:46:20.834+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:46:20.835007+0000) 2024-03-04T15:46:21.836 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:46:21.834+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:46:21.835255+0000) 2024-03-04T15:46:22.836 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:46:22.835+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:46:22.835493+0000) 2024-03-04T15:46:23.836 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:46:23.835+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:46:23.835730+0000) 2024-03-04T15:46:24.837 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:46:24.835+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:46:24.835971+0000) 2024-03-04T15:46:25.837 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:46:25.835+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:46:25.836212+0000) 2024-03-04T15:46:26.837 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:46:26.836+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:46:26.836453+0000) 2024-03-04T15:46:27.837 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:46:27.836+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:46:27.836698+0000) 2024-03-04T15:46:28.838 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:46:28.836+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:46:28.836933+0000) 2024-03-04T15:46:29.838 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:46:29.836+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:46:29.837176+0000) 2024-03-04T15:46:30.003 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:46:30.012 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:46:30.838 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:46:30.837+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:46:30.837438+0000) 2024-03-04T15:46:31.838 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:46:31.837+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:46:31.837674+0000) 2024-03-04T15:46:32.839 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:46:32.837+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:46:32.838003+0000) 2024-03-04T15:46:33.839 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:46:33.837+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:46:33.838180+0000) 2024-03-04T15:46:34.839 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:46:34.838+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:46:34.838419+0000) 2024-03-04T15:46:35.839 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:46:35.838+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:46:35.838657+0000) 2024-03-04T15:46:36.840 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:46:36.837+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:46:36.838927+0000) 2024-03-04T15:46:37.840 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:46:37.838+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:46:37.839166+0000) 2024-03-04T15:46:38.840 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:46:38.838+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:46:38.839401+0000) 2024-03-04T15:46:39.840 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:46:39.838+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:46:39.839644+0000) 2024-03-04T15:46:40.840 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:46:40.838+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:46:40.839939+0000) 2024-03-04T15:46:41.841 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:46:41.838+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:46:41.840182+0000) 2024-03-04T15:46:42.841 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:46:42.839+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:46:42.840435+0000) 2024-03-04T15:46:43.841 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:46:43.839+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:46:43.840628+0000) 2024-03-04T15:46:44.841 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:46:44.839+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:46:44.840940+0000) 2024-03-04T15:46:45.842 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:46:45.840+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:46:45.841182+0000) 2024-03-04T15:46:46.842 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:46:46.840+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:46:46.841426+0000) 2024-03-04T15:46:47.842 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:46:47.840+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:46:47.841661+0000) 2024-03-04T15:46:48.843 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:46:48.841+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:46:48.841934+0000) 2024-03-04T15:46:49.843 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:46:49.841+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:46:49.842189+0000) 2024-03-04T15:46:50.843 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:46:50.841+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:46:50.842457+0000) 2024-03-04T15:46:51.843 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:46:51.841+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:46:51.842713+0000) 2024-03-04T15:46:52.844 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:46:52.842+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:46:52.842948+0000) 2024-03-04T15:46:53.844 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:46:53.842+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:46:53.843185+0000) 2024-03-04T15:46:54.844 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:46:54.842+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:46:54.843415+0000) 2024-03-04T15:46:55.844 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:46:55.842+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:46:55.843660+0000) 2024-03-04T15:46:56.845 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:46:56.843+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:46:56.843936+0000) 2024-03-04T15:46:57.845 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:46:57.843+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:46:57.844177+0000) 2024-03-04T15:46:58.845 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:46:58.843+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:46:58.844436+0000) 2024-03-04T15:46:59.845 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:46:59.843+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:46:59.844709+0000) 2024-03-04T15:47:00.217 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:47:00.226 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:47:00.846 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:47:00.844+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:47:00.844979+0000) 2024-03-04T15:47:01.846 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:47:01.844+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:47:01.845236+0000) 2024-03-04T15:47:02.846 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:47:02.844+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:47:02.845502+0000) 2024-03-04T15:47:03.846 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:47:03.845+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:47:03.845753+0000) 2024-03-04T15:47:04.847 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:47:04.845+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:47:04.846014+0000) 2024-03-04T15:47:05.847 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:47:05.845+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:47:05.846352+0000) 2024-03-04T15:47:06.847 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:47:06.845+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:47:06.846616+0000) 2024-03-04T15:47:07.847 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:47:07.846+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:47:07.846898+0000) 2024-03-04T15:47:08.848 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:47:08.846+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:47:08.847134+0000) 2024-03-04T15:47:09.848 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:47:09.846+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:47:09.847344+0000) 2024-03-04T15:47:10.848 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:47:10.846+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:47:10.847591+0000) 2024-03-04T15:47:11.849 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:47:11.847+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:47:11.847893+0000) 2024-03-04T15:47:12.849 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:47:12.847+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:47:12.848136+0000) 2024-03-04T15:47:13.849 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:47:13.847+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:47:13.848363+0000) 2024-03-04T15:47:14.849 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:47:14.847+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:47:14.848573+0000) 2024-03-04T15:47:15.849 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:47:15.848+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:47:15.848853+0000) 2024-03-04T15:47:16.850 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:47:16.848+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:47:16.849099+0000) 2024-03-04T15:47:17.850 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:47:17.848+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:47:17.849343+0000) 2024-03-04T15:47:18.850 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:47:18.848+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:47:18.849595+0000) 2024-03-04T15:47:19.851 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:47:19.849+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:47:19.849904+0000) 2024-03-04T15:47:20.851 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:47:20.849+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:47:20.850155+0000) 2024-03-04T15:47:21.851 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:47:21.849+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:47:21.850340+0000) 2024-03-04T15:47:22.851 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:47:22.850+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:47:22.850586+0000) 2024-03-04T15:47:23.852 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:47:23.850+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:47:23.850864+0000) 2024-03-04T15:47:24.852 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:47:24.850+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:47:24.851114+0000) 2024-03-04T15:47:25.852 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:47:25.850+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:47:25.851357+0000) 2024-03-04T15:47:26.852 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:47:26.850+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:47:26.851595+0000) 2024-03-04T15:47:27.853 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:47:27.851+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:47:27.851878+0000) 2024-03-04T15:47:28.853 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:47:28.851+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:47:28.852113+0000) 2024-03-04T15:47:29.853 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:47:29.851+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:47:29.852314+0000) 2024-03-04T15:47:30.422 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:47:30.434 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:47:30.853 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:47:30.851+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:47:30.852586+0000) 2024-03-04T15:47:31.853 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:47:31.852+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:47:31.852814+0000) 2024-03-04T15:47:32.854 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:47:32.852+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:47:32.853097+0000) 2024-03-04T15:47:33.855 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:47:33.853+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:47:33.853332+0000) 2024-03-04T15:47:34.854 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:47:34.853+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:47:34.853578+0000) 2024-03-04T15:47:35.854 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:47:35.853+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:47:35.853818+0000) 2024-03-04T15:47:36.855 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:47:36.853+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:47:36.854089+0000) 2024-03-04T15:47:37.855 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:47:37.853+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:47:37.854285+0000) 2024-03-04T15:47:38.855 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:47:38.853+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:47:38.854559+0000) 2024-03-04T15:47:39.855 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:47:39.854+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:47:39.854862+0000) 2024-03-04T15:47:40.856 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:47:40.855+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:47:40.855162+0000) 2024-03-04T15:47:41.856 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:47:41.854+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:47:41.855428+0000) 2024-03-04T15:47:42.856 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:47:42.853+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:47:42.855669+0000) 2024-03-04T15:47:43.857 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:47:43.854+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:47:43.855947+0000) 2024-03-04T15:47:44.861 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:47:44.855+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:47:44.856191+0000) 2024-03-04T15:47:45.857 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:47:45.855+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:47:45.856427+0000) 2024-03-04T15:47:46.858 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:47:46.854+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:47:46.856668+0000) 2024-03-04T15:47:47.858 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:47:47.855+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:47:47.856933+0000) 2024-03-04T15:47:48.858 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:47:48.856+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:47:48.857171+0000) 2024-03-04T15:47:49.858 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:47:49.856+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:47:49.857416+0000) 2024-03-04T15:47:50.858 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:47:50.856+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:47:50.857662+0000) 2024-03-04T15:47:51.859 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:47:51.856+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:47:51.857942+0000) 2024-03-04T15:47:52.859 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:47:52.856+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:47:52.858122+0000) 2024-03-04T15:47:53.861 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:47:53.857+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:47:53.858361+0000) 2024-03-04T15:47:54.860 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:47:54.857+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:47:54.858614+0000) 2024-03-04T15:47:55.860 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:47:55.858+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:47:55.858916+0000) 2024-03-04T15:47:56.860 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:47:56.858+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:47:56.859194+0000) 2024-03-04T15:47:57.860 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:47:57.858+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:47:57.859502+0000) 2024-03-04T15:47:58.860 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:47:58.858+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:47:58.859755+0000) 2024-03-04T15:47:59.861 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:47:59.859+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:47:59.860000+0000) 2024-03-04T15:48:00.627 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:48:00.635 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:48:00.861 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:48:00.859+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:48:00.860299+0000) 2024-03-04T15:48:01.861 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:48:01.859+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:48:01.860541+0000) 2024-03-04T15:48:02.862 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:48:02.860+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:48:02.860813+0000) 2024-03-04T15:48:03.862 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:48:03.860+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:48:03.861093+0000) 2024-03-04T15:48:04.862 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:48:04.860+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:48:04.861386+0000) 2024-03-04T15:48:05.862 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:48:05.860+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:48:05.861681+0000) 2024-03-04T15:48:06.863 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:48:06.861+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:48:06.861976+0000) 2024-03-04T15:48:07.863 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:48:07.861+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:48:07.862268+0000) 2024-03-04T15:48:08.863 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:48:08.861+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:48:08.862529+0000) 2024-03-04T15:48:09.864 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:48:09.862+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:48:09.862873+0000) 2024-03-04T15:48:10.864 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:48:10.862+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:48:10.863156+0000) 2024-03-04T15:48:11.864 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:48:11.862+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:48:11.863415+0000) 2024-03-04T15:48:12.864 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:48:12.862+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:48:12.863663+0000) 2024-03-04T15:48:13.865 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:48:13.863+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:48:13.863951+0000) 2024-03-04T15:48:14.865 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:48:14.863+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:48:14.864198+0000) 2024-03-04T15:48:15.865 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:48:15.863+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:48:15.864439+0000) 2024-03-04T15:48:16.865 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:48:16.864+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:48:16.864702+0000) 2024-03-04T15:48:17.866 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:48:17.864+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:48:17.864951+0000) 2024-03-04T15:48:18.867 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:48:18.864+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:48:18.865194+0000) 2024-03-04T15:48:19.866 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:48:19.864+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:48:19.865544+0000) 2024-03-04T15:48:20.867 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:48:20.865+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:48:20.865858+0000) 2024-03-04T15:48:21.867 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:48:21.865+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:48:21.866123+0000) 2024-03-04T15:48:22.867 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:48:22.865+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:48:22.866382+0000) 2024-03-04T15:48:23.867 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:48:23.865+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:48:23.866644+0000) 2024-03-04T15:48:24.868 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:48:24.866+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:48:24.866946+0000) 2024-03-04T15:48:25.868 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:48:25.866+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:48:25.867189+0000) 2024-03-04T15:48:26.868 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:48:26.866+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:48:26.867436+0000) 2024-03-04T15:48:27.868 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:48:27.867+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:48:27.867699+0000) 2024-03-04T15:48:28.869 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:48:28.867+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:48:28.867966+0000) 2024-03-04T15:48:29.869 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:48:29.867+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:48:29.868225+0000) 2024-03-04T15:48:30.869 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:48:30.868+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:48:30.868493+0000) 2024-03-04T15:48:30.874 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:48:30.882 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:48:31.869 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:48:31.868+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:48:31.868760+0000) 2024-03-04T15:48:32.870 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:48:32.868+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:48:32.869024+0000) 2024-03-04T15:48:33.870 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:48:33.868+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:48:33.869278+0000) 2024-03-04T15:48:34.870 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:48:34.868+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:48:34.869571+0000) 2024-03-04T15:48:35.871 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:48:35.869+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:48:35.869914+0000) 2024-03-04T15:48:36.871 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:48:36.869+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:48:36.870148+0000) 2024-03-04T15:48:37.871 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:48:37.869+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:48:37.870382+0000) 2024-03-04T15:48:38.871 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:48:38.869+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:48:38.870592+0000) 2024-03-04T15:48:39.871 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:48:39.870+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:48:39.870806+0000) 2024-03-04T15:48:40.872 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:48:40.870+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:48:40.871127+0000) 2024-03-04T15:48:41.872 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:48:41.870+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:48:41.871331+0000) 2024-03-04T15:48:42.872 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:48:42.870+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:48:42.871537+0000) 2024-03-04T15:48:43.873 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:48:43.871+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:48:43.871802+0000) 2024-03-04T15:48:44.873 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:48:44.871+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:48:44.872117+0000) 2024-03-04T15:48:45.873 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:48:45.872+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:48:45.872408+0000) 2024-03-04T15:48:46.873 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:48:46.871+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:48:46.872668+0000) 2024-03-04T15:48:47.874 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:48:47.871+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:48:47.872941+0000) 2024-03-04T15:48:48.874 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:48:48.871+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:48:48.873138+0000) 2024-03-04T15:48:49.874 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:48:49.872+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:48:49.873385+0000) 2024-03-04T15:48:50.875 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:48:50.871+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:48:50.873575+0000) 2024-03-04T15:48:51.875 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:48:51.872+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:48:51.873866+0000) 2024-03-04T15:48:52.875 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:48:52.872+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:48:52.874101+0000) 2024-03-04T15:48:53.875 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:48:53.873+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:48:53.874352+0000) 2024-03-04T15:48:54.875 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:48:54.873+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:48:54.874612+0000) 2024-03-04T15:48:55.876 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:48:55.873+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:48:55.874875+0000) 2024-03-04T15:48:56.876 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:48:56.873+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:48:56.875062+0000) 2024-03-04T15:48:57.876 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:48:57.874+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:48:57.875257+0000) 2024-03-04T15:48:58.876 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:48:58.874+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:48:58.875462+0000) 2024-03-04T15:48:59.876 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:48:59.874+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:48:59.875714+0000) 2024-03-04T15:49:00.877 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:49:00.875+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:49:00.875997+0000) 2024-03-04T15:49:01.021 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:49:01.029 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:49:01.877 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:49:01.875+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:49:01.876203+0000) 2024-03-04T15:49:02.877 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:49:02.875+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:49:02.876403+0000) 2024-03-04T15:49:03.877 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:49:03.875+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:49:03.876669+0000) 2024-03-04T15:49:04.878 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:49:04.876+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:49:04.876938+0000) 2024-03-04T15:49:05.878 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:49:05.876+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:49:05.877196+0000) 2024-03-04T15:49:06.878 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:49:06.876+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:49:06.877453+0000) 2024-03-04T15:49:07.879 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:49:07.877+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:49:07.877791+0000) 2024-03-04T15:49:08.879 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:49:08.877+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:49:08.878123+0000) 2024-03-04T15:49:09.880 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:49:09.877+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:49:09.878354+0000) 2024-03-04T15:49:10.879 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:49:10.877+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:49:10.878619+0000) 2024-03-04T15:49:11.880 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:49:11.878+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:49:11.878925+0000) 2024-03-04T15:49:12.880 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:49:12.878+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:49:12.879201+0000) 2024-03-04T15:49:13.880 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:49:13.878+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:49:13.879471+0000) 2024-03-04T15:49:14.880 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:49:14.879+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:49:14.879738+0000) 2024-03-04T15:49:15.881 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:49:15.879+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:49:15.879991+0000) 2024-03-04T15:49:16.881 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:49:16.879+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:49:16.880238+0000) 2024-03-04T15:49:17.881 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:49:17.879+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:49:17.880495+0000) 2024-03-04T15:49:18.881 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:49:18.880+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:49:18.880744+0000) 2024-03-04T15:49:19.882 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:49:19.880+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:49:19.881013+0000) 2024-03-04T15:49:20.882 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:49:20.880+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:49:20.881283+0000) 2024-03-04T15:49:21.882 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:49:21.880+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:49:21.881539+0000) 2024-03-04T15:49:22.883 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:49:22.881+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:49:22.881793+0000) 2024-03-04T15:49:23.883 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:49:23.881+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:49:23.882090+0000) 2024-03-04T15:49:24.883 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:49:24.881+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:49:24.882363+0000) 2024-03-04T15:49:25.883 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:49:25.882+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:49:25.882617+0000) 2024-03-04T15:49:26.884 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:49:26.882+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:49:26.882919+0000) 2024-03-04T15:49:27.884 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:49:27.882+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:49:27.883178+0000) 2024-03-04T15:49:28.884 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:49:28.882+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:49:28.883449+0000) 2024-03-04T15:49:29.885 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:49:29.883+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:49:29.883720+0000) 2024-03-04T15:49:30.885 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:49:30.883+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:49:30.884011+0000) 2024-03-04T15:49:31.162 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:49:31.170 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:49:31.885 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:49:31.883+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:49:31.884269+0000) 2024-03-04T15:49:32.885 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:49:32.884+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:49:32.884522+0000) 2024-03-04T15:49:33.885 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:49:33.883+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:49:33.884728+0000) 2024-03-04T15:49:34.886 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:49:34.884+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:49:34.884992+0000) 2024-03-04T15:49:35.886 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:49:35.884+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:49:35.885247+0000) 2024-03-04T15:49:36.886 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:49:36.885+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:49:36.885529+0000) 2024-03-04T15:49:37.887 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:49:37.885+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:49:37.885806+0000) 2024-03-04T15:49:38.887 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:49:38.885+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:49:38.886111+0000) 2024-03-04T15:49:39.887 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:49:39.886+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:49:39.886371+0000) 2024-03-04T15:49:40.887 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:49:40.885+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:49:40.886649+0000) 2024-03-04T15:49:41.888 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:49:41.886+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:49:41.886927+0000) 2024-03-04T15:49:42.888 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:49:42.886+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:49:42.887181+0000) 2024-03-04T15:49:43.888 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:49:43.887+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:49:43.887434+0000) 2024-03-04T15:49:44.888 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:49:44.887+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:49:44.887711+0000) 2024-03-04T15:49:45.889 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:49:45.887+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:49:45.887959+0000) 2024-03-04T15:49:46.889 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:49:46.887+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:49:46.888153+0000) 2024-03-04T15:49:47.889 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:49:47.888+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:49:47.888421+0000) 2024-03-04T15:49:48.889 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:49:48.888+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:49:48.888711+0000) 2024-03-04T15:49:49.890 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:49:49.888+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:49:49.889070+0000) 2024-03-04T15:49:50.890 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:49:50.888+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:49:50.889385+0000) 2024-03-04T15:49:51.890 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:49:51.887+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:49:51.889700+0000) 2024-03-04T15:49:52.891 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:49:52.888+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:49:52.889996+0000) 2024-03-04T15:49:53.891 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:49:53.889+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:49:53.890279+0000) 2024-03-04T15:49:54.891 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:49:54.888+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:49:54.890519+0000) 2024-03-04T15:49:55.892 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:49:55.889+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:49:55.890812+0000) 2024-03-04T15:49:56.892 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:49:56.890+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:49:56.891172+0000) 2024-03-04T15:49:57.892 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:49:57.890+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:49:57.891467+0000) 2024-03-04T15:49:58.893 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:49:58.890+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:49:58.891762+0000) 2024-03-04T15:49:59.893 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:49:59.891+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:49:59.892055+0000) 2024-03-04T15:50:00.893 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:50:00.891+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:50:00.892363+0000) 2024-03-04T15:50:01.372 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:50:01.380 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:50:01.893 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:50:01.891+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:50:01.892650+0000) 2024-03-04T15:50:02.894 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:50:02.892+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:50:02.892960+0000) 2024-03-04T15:50:03.894 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:50:03.892+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:50:03.893288+0000) 2024-03-04T15:50:04.894 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:50:04.892+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:50:04.893606+0000) 2024-03-04T15:50:05.895 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:50:05.893+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:50:05.893912+0000) 2024-03-04T15:50:06.895 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:50:06.893+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:50:06.894146+0000) 2024-03-04T15:50:07.895 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:50:07.893+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:50:07.894388+0000) 2024-03-04T15:50:08.895 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:50:08.893+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:50:08.894621+0000) 2024-03-04T15:50:09.896 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:50:09.894+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:50:09.894903+0000) 2024-03-04T15:50:10.896 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:50:10.894+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:50:10.895114+0000) 2024-03-04T15:50:11.896 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:50:11.894+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:50:11.895354+0000) 2024-03-04T15:50:12.896 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:50:12.894+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:50:12.895589+0000) 2024-03-04T15:50:13.897 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:50:13.895+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:50:13.895852+0000) 2024-03-04T15:50:14.897 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:50:14.895+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:50:14.896046+0000) 2024-03-04T15:50:15.897 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:50:15.895+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:50:15.896294+0000) 2024-03-04T15:50:16.897 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:50:16.895+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:50:16.896531+0000) 2024-03-04T15:50:17.898 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:50:17.895+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:50:17.896764+0000) 2024-03-04T15:50:18.898 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:50:18.896+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:50:18.897008+0000) 2024-03-04T15:50:19.898 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:50:19.896+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:50:19.897252+0000) 2024-03-04T15:50:20.898 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:50:20.896+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:50:20.897437+0000) 2024-03-04T15:50:21.898 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:50:21.896+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:50:21.897694+0000) 2024-03-04T15:50:22.899 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:50:22.897+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:50:22.897937+0000) 2024-03-04T15:50:23.899 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:50:23.897+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:50:23.898175+0000) 2024-03-04T15:50:24.899 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:50:24.897+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:50:24.898440+0000) 2024-03-04T15:50:25.899 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:50:25.897+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:50:25.898644+0000) 2024-03-04T15:50:26.900 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:50:26.898+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:50:26.898921+0000) 2024-03-04T15:50:27.900 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:50:27.898+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:50:27.899172+0000) 2024-03-04T15:50:28.900 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:50:28.898+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:50:28.899372+0000) 2024-03-04T15:50:29.901 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:50:29.899+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:50:29.899618+0000) 2024-03-04T15:50:30.901 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:50:30.899+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:50:30.899915+0000) 2024-03-04T15:50:31.516 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:50:31.524 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:50:31.901 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:50:31.899+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:50:31.900167+0000) 2024-03-04T15:50:32.901 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:50:32.899+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:50:32.900415+0000) 2024-03-04T15:50:33.901 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:50:33.899+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:50:33.900673+0000) 2024-03-04T15:50:34.902 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:50:34.900+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:50:34.900938+0000) 2024-03-04T15:50:35.902 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:50:35.900+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:50:35.901177+0000) 2024-03-04T15:50:36.902 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:50:36.901+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:50:36.901410+0000) 2024-03-04T15:50:37.902 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:50:37.901+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:50:37.901651+0000) 2024-03-04T15:50:38.903 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:50:38.901+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:50:38.901924+0000) 2024-03-04T15:50:39.903 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:50:39.901+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:50:39.902183+0000) 2024-03-04T15:50:40.903 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:50:40.902+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:50:40.902456+0000) 2024-03-04T15:50:41.903 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:50:41.902+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:50:41.902695+0000) 2024-03-04T15:50:42.904 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:50:42.902+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:50:42.902962+0000) 2024-03-04T15:50:43.904 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:50:43.902+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:50:43.903197+0000) 2024-03-04T15:50:44.904 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:50:44.902+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:50:44.903450+0000) 2024-03-04T15:50:45.905 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:50:45.903+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:50:45.903725+0000) 2024-03-04T15:50:46.905 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:50:46.903+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:50:46.903962+0000) 2024-03-04T15:50:47.905 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:50:47.903+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:50:47.904149+0000) 2024-03-04T15:50:48.905 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:50:48.904+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:50:48.904387+0000) 2024-03-04T15:50:49.905 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:50:49.904+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:50:49.904633+0000) 2024-03-04T15:50:50.906 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:50:50.904+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:50:50.904917+0000) 2024-03-04T15:50:51.906 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:50:51.904+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:50:51.905155+0000) 2024-03-04T15:50:52.906 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:50:52.905+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:50:52.905398+0000) 2024-03-04T15:50:53.906 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:50:53.905+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:50:53.905638+0000) 2024-03-04T15:50:54.907 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:50:54.904+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:50:54.905956+0000) 2024-03-04T15:50:55.907 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:50:55.905+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:50:55.906204+0000) 2024-03-04T15:50:56.907 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:50:56.904+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:50:56.906405+0000) 2024-03-04T15:50:57.908 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:50:57.905+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:50:57.906643+0000) 2024-03-04T15:50:58.908 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:50:58.905+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:50:58.906971+0000) 2024-03-04T15:50:59.908 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:50:59.906+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:50:59.907236+0000) 2024-03-04T15:51:00.908 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:51:00.906+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:51:00.907492+0000) 2024-03-04T15:51:01.724 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:51:01.732 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:51:01.908 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:51:01.906+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:51:01.907784+0000) 2024-03-04T15:51:02.909 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:51:02.907+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:51:02.907990+0000) 2024-03-04T15:51:03.909 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:51:03.907+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:51:03.908248+0000) 2024-03-04T15:51:04.909 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:51:04.907+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:51:04.908512+0000) 2024-03-04T15:51:05.910 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:51:05.907+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:51:05.908752+0000) 2024-03-04T15:51:06.910 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:51:06.908+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:51:06.909012+0000) 2024-03-04T15:51:07.910 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:51:07.908+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:51:07.909251+0000) 2024-03-04T15:51:08.910 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:51:08.908+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:51:08.909493+0000) 2024-03-04T15:51:09.911 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:51:09.908+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:51:09.909740+0000) 2024-03-04T15:51:10.911 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:51:10.909+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:51:10.909985+0000) 2024-03-04T15:51:11.911 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:51:11.909+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:51:11.910224+0000) 2024-03-04T15:51:12.911 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:51:12.909+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:51:12.910410+0000) 2024-03-04T15:51:13.911 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:51:13.909+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:51:13.910683+0000) 2024-03-04T15:51:14.912 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:51:14.910+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:51:14.910909+0000) 2024-03-04T15:51:15.912 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:51:15.910+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:51:15.911154+0000) 2024-03-04T15:51:16.912 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:51:16.910+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:51:16.911395+0000) 2024-03-04T15:51:17.913 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:51:17.910+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:51:17.911621+0000) 2024-03-04T15:51:18.913 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:51:18.911+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:51:18.911942+0000) 2024-03-04T15:51:19.913 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:51:19.911+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:51:19.912183+0000) 2024-03-04T15:51:20.913 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:51:20.911+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:51:20.912438+0000) 2024-03-04T15:51:21.913 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:51:21.911+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:51:21.912677+0000) 2024-03-04T15:51:22.914 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:51:22.912+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:51:22.912973+0000) 2024-03-04T15:51:23.914 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:51:23.912+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:51:23.913218+0000) 2024-03-04T15:51:24.914 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:51:24.912+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:51:24.913446+0000) 2024-03-04T15:51:25.915 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:51:25.913+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:51:25.913694+0000) 2024-03-04T15:51:26.915 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:51:26.913+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:51:26.913931+0000) 2024-03-04T15:51:27.915 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:51:27.913+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:51:27.914178+0000) 2024-03-04T15:51:28.915 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:51:28.913+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:51:28.914372+0000) 2024-03-04T15:51:29.915 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:51:29.914+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:51:29.914631+0000) 2024-03-04T15:51:30.916 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:51:30.914+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:51:30.914945+0000) 2024-03-04T15:51:31.916 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:51:31.914+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:51:31.915186+0000) 2024-03-04T15:51:31.955 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:51:31.963 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:51:32.916 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:51:32.914+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:51:32.915382+0000) 2024-03-04T15:51:33.916 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:51:33.915+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:51:33.915617+0000) 2024-03-04T15:51:34.917 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:51:34.915+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:51:34.915915+0000) 2024-03-04T15:51:35.917 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:51:35.915+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:51:35.916180+0000) 2024-03-04T15:51:36.917 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:51:36.915+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:51:36.916473+0000) 2024-03-04T15:51:37.918 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:51:37.916+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:51:37.916717+0000) 2024-03-04T15:51:38.918 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:51:38.916+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:51:38.916981+0000) 2024-03-04T15:51:39.918 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:51:39.916+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:51:39.917231+0000) 2024-03-04T15:51:40.918 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:51:40.917+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:51:40.917488+0000) 2024-03-04T15:51:41.919 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:51:41.916+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:51:41.917723+0000) 2024-03-04T15:51:42.919 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:51:42.917+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:51:42.917964+0000) 2024-03-04T15:51:43.919 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:51:43.917+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:51:43.918215+0000) 2024-03-04T15:51:44.920 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:51:44.918+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:51:44.918468+0000) 2024-03-04T15:51:45.920 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:51:45.918+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:51:45.918712+0000) 2024-03-04T15:51:46.920 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:51:46.918+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:51:46.919012+0000) 2024-03-04T15:51:47.920 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:51:47.918+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:51:47.919255+0000) 2024-03-04T15:51:48.920 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:51:48.919+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:51:48.919490+0000) 2024-03-04T15:51:49.920 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:51:49.918+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:51:49.919727+0000) 2024-03-04T15:51:50.921 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:51:50.919+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:51:50.919983+0000) 2024-03-04T15:51:51.921 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:51:51.919+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:51:51.920226+0000) 2024-03-04T15:51:52.921 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:51:52.920+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:51:52.920466+0000) 2024-03-04T15:51:53.922 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:51:53.920+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:51:53.920701+0000) 2024-03-04T15:51:54.922 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:51:54.920+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:51:54.920943+0000) 2024-03-04T15:51:55.922 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:51:55.921+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:51:55.921183+0000) 2024-03-04T15:51:56.922 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:51:56.920+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:51:56.921474+0000) 2024-03-04T15:51:57.922 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:51:57.920+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:51:57.921670+0000) 2024-03-04T15:51:58.923 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:51:58.921+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:51:58.921937+0000) 2024-03-04T15:51:59.923 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:51:59.921+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:51:59.922177+0000) 2024-03-04T15:52:00.923 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:52:00.921+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:52:00.922428+0000) 2024-03-04T15:52:01.924 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:52:01.921+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:52:01.922669+0000) 2024-03-04T15:52:02.092 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:52:02.101 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:52:02.924 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:52:02.921+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:52:02.922944+0000) 2024-03-04T15:52:03.924 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:52:03.922+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:52:03.923188+0000) 2024-03-04T15:52:04.924 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:52:04.921+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:52:04.923385+0000) 2024-03-04T15:52:05.925 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:52:05.922+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:52:05.923652+0000) 2024-03-04T15:52:06.925 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:52:06.922+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:52:06.923931+0000) 2024-03-04T15:52:07.925 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:52:07.923+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:52:07.924173+0000) 2024-03-04T15:52:08.925 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:52:08.923+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:52:08.924427+0000) 2024-03-04T15:52:09.925 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:52:09.923+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:52:09.924670+0000) 2024-03-04T15:52:10.926 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:52:10.924+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:52:10.924959+0000) 2024-03-04T15:52:11.926 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:52:11.924+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:52:11.925214+0000) 2024-03-04T15:52:12.927 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:52:12.924+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:52:12.925466+0000) 2024-03-04T15:52:13.927 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:52:13.924+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:52:13.925715+0000) 2024-03-04T15:52:14.927 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:52:14.925+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:52:14.925998+0000) 2024-03-04T15:52:15.927 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:52:15.925+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:52:15.926244+0000) 2024-03-04T15:52:16.927 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:52:16.925+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:52:16.926472+0000) 2024-03-04T15:52:17.928 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:52:17.925+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:52:17.926715+0000) 2024-03-04T15:52:18.928 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:52:18.926+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:52:18.926992+0000) 2024-03-04T15:52:19.928 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:52:19.926+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:52:19.927254+0000) 2024-03-04T15:52:20.928 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:52:20.926+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:52:20.927522+0000) 2024-03-04T15:52:21.929 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:52:21.926+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:52:21.927774+0000) 2024-03-04T15:52:22.929 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:52:22.927+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:52:22.928038+0000) 2024-03-04T15:52:23.929 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:52:23.927+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:52:23.928292+0000) 2024-03-04T15:52:24.929 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:52:24.927+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:52:24.928500+0000) 2024-03-04T15:52:25.930 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:52:25.927+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:52:25.928715+0000) 2024-03-04T15:52:26.930 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:52:26.928+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:52:26.929005+0000) 2024-03-04T15:52:27.930 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:52:27.928+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:52:27.929266+0000) 2024-03-04T15:52:28.930 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:52:28.928+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:52:28.929522+0000) 2024-03-04T15:52:29.931 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:52:29.929+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:52:29.929892+0000) 2024-03-04T15:52:30.931 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:52:30.929+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:52:30.930163+0000) 2024-03-04T15:52:31.931 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:52:31.929+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:52:31.930360+0000) 2024-03-04T15:52:32.314 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:52:32.325 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:52:32.931 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:52:32.929+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:52:32.930603+0000) 2024-03-04T15:52:33.932 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:52:33.930+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:52:33.930880+0000) 2024-03-04T15:52:34.932 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:52:34.930+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:52:34.931152+0000) 2024-03-04T15:52:35.932 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:52:35.930+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:52:35.931417+0000) 2024-03-04T15:52:36.933 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:52:36.931+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:52:36.931670+0000) 2024-03-04T15:52:37.933 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:52:37.931+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:52:37.931958+0000) 2024-03-04T15:52:38.933 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:52:38.931+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:52:38.932241+0000) 2024-03-04T15:52:39.933 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:52:39.931+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:52:39.932528+0000) 2024-03-04T15:52:40.934 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:52:40.932+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:52:40.932880+0000) 2024-03-04T15:52:41.934 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:52:41.932+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:52:41.933178+0000) 2024-03-04T15:52:42.934 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:52:42.932+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:52:42.933430+0000) 2024-03-04T15:52:43.935 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:52:43.933+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:52:43.933732+0000) 2024-03-04T15:52:44.935 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:52:44.933+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:52:44.934040+0000) 2024-03-04T15:52:45.935 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:52:45.933+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:52:45.934357+0000) 2024-03-04T15:52:46.936 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:52:46.934+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:52:46.934648+0000) 2024-03-04T15:52:47.936 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:52:47.934+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:52:47.934974+0000) 2024-03-04T15:52:48.936 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:52:48.934+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:52:48.935280+0000) 2024-03-04T15:52:49.936 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:52:49.935+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:52:49.935526+0000) 2024-03-04T15:52:50.937 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:52:50.934+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:52:50.935804+0000) 2024-03-04T15:52:51.937 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:52:51.935+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:52:51.936093+0000) 2024-03-04T15:52:52.937 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:52:52.935+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:52:52.936283+0000) 2024-03-04T15:52:53.937 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:52:53.936+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:52:53.936523+0000) 2024-03-04T15:52:54.938 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:52:54.936+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:52:54.936761+0000) 2024-03-04T15:52:55.938 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:52:55.936+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:52:55.937009+0000) 2024-03-04T15:52:56.938 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:52:56.937+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:52:56.937252+0000) 2024-03-04T15:52:57.938 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:52:57.937+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:52:57.937490+0000) 2024-03-04T15:52:58.939 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:52:58.937+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:52:58.937740+0000) 2024-03-04T15:52:59.939 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:52:59.937+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:52:59.938006+0000) 2024-03-04T15:53:00.939 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:53:00.938+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:53:00.938287+0000) 2024-03-04T15:53:01.939 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:53:01.938+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:53:01.938553+0000) 2024-03-04T15:53:02.519 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:53:02.529 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:53:02.940 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:53:02.938+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:53:02.938857+0000) 2024-03-04T15:53:03.940 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:53:03.938+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:53:03.939109+0000) 2024-03-04T15:53:04.940 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:53:04.939+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:53:04.939391+0000) 2024-03-04T15:53:05.941 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:53:05.937+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:53:05.939671+0000) 2024-03-04T15:53:06.941 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:53:06.938+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:53:06.939964+0000) 2024-03-04T15:53:07.941 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:53:07.939+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:53:07.940227+0000) 2024-03-04T15:53:08.941 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:53:08.939+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:53:08.940479+0000) 2024-03-04T15:53:09.942 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:53:09.939+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:53:09.940754+0000) 2024-03-04T15:53:10.942 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:53:10.939+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:53:10.940997+0000) 2024-03-04T15:53:11.942 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:53:11.940+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:53:11.941255+0000) 2024-03-04T15:53:12.943 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:53:12.940+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:53:12.941520+0000) 2024-03-04T15:53:13.943 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:53:13.940+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:53:13.941782+0000) 2024-03-04T15:53:14.943 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:53:14.941+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:53:14.942085+0000) 2024-03-04T15:53:15.943 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:53:15.941+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:53:15.942349+0000) 2024-03-04T15:53:16.943 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:53:16.941+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:53:16.942626+0000) 2024-03-04T15:53:17.944 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:53:17.942+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:53:17.942913+0000) 2024-03-04T15:53:18.944 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:53:18.942+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:53:18.943168+0000) 2024-03-04T15:53:19.944 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:53:19.942+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:53:19.943364+0000) 2024-03-04T15:53:20.945 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:53:20.942+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:53:20.943623+0000) 2024-03-04T15:53:21.945 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:53:21.943+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:53:21.943922+0000) 2024-03-04T15:53:22.945 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:53:22.943+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:53:22.944175+0000) 2024-03-04T15:53:23.945 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:53:23.943+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:53:23.944455+0000) 2024-03-04T15:53:24.946 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:53:24.943+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:53:24.944748+0000) 2024-03-04T15:53:25.946 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:53:25.944+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:53:25.945010+0000) 2024-03-04T15:53:26.946 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:53:26.944+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:53:26.945251+0000) 2024-03-04T15:53:27.946 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:53:27.944+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:53:27.945494+0000) 2024-03-04T15:53:28.947 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:53:28.944+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:53:28.945737+0000) 2024-03-04T15:53:29.947 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:53:29.945+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:53:29.945977+0000) 2024-03-04T15:53:30.948 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:53:30.945+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:53:30.946260+0000) 2024-03-04T15:53:31.947 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:53:31.945+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:53:31.946527+0000) 2024-03-04T15:53:32.657 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:53:32.666 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:53:32.948 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:53:32.946+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:53:32.946799+0000) 2024-03-04T15:53:33.948 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:53:33.946+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:53:33.947066+0000) 2024-03-04T15:53:34.948 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:53:34.946+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:53:34.947286+0000) 2024-03-04T15:53:35.948 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:53:35.946+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:53:35.947553+0000) 2024-03-04T15:53:36.949 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:53:36.947+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:53:36.947810+0000) 2024-03-04T15:53:37.949 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:53:37.947+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:53:37.948133+0000) 2024-03-04T15:53:38.949 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:53:38.947+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:53:38.948395+0000) 2024-03-04T15:53:39.950 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:53:39.948+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:53:39.948660+0000) 2024-03-04T15:53:40.950 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:53:40.948+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:53:40.948946+0000) 2024-03-04T15:53:41.950 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:53:41.948+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:53:41.949158+0000) 2024-03-04T15:53:42.950 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:53:42.948+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:53:42.949413+0000) 2024-03-04T15:53:43.951 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:53:43.949+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:53:43.949672+0000) 2024-03-04T15:53:44.951 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:53:44.949+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:53:44.949980+0000) 2024-03-04T15:53:45.951 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:53:45.949+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:53:45.950193+0000) 2024-03-04T15:53:46.951 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:53:46.949+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:53:46.950411+0000) 2024-03-04T15:53:47.952 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:53:47.949+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:53:47.950659+0000) 2024-03-04T15:53:48.952 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:53:48.950+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:53:48.950938+0000) 2024-03-04T15:53:49.952 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:53:49.950+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:53:49.951188+0000) 2024-03-04T15:53:50.952 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:53:50.951+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:53:50.951445+0000) 2024-03-04T15:53:51.953 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:53:51.951+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:53:51.951684+0000) 2024-03-04T15:53:52.953 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:53:52.951+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:53:52.951925+0000) 2024-03-04T15:53:53.953 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:53:53.951+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:53:53.952166+0000) 2024-03-04T15:53:54.953 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:53:54.952+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:53:54.952416+0000) 2024-03-04T15:53:55.954 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:53:55.952+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:53:55.952668+0000) 2024-03-04T15:53:56.954 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:53:56.952+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:53:56.952946+0000) 2024-03-04T15:53:57.954 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:53:57.952+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:53:57.953186+0000) 2024-03-04T15:53:58.954 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:53:58.953+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:53:58.953418+0000) 2024-03-04T15:53:59.955 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:53:59.953+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:53:59.953699+0000) 2024-03-04T15:54:00.955 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:54:00.953+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:54:00.954023+0000) 2024-03-04T15:54:01.955 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:54:01.954+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:54:01.954262+0000) 2024-03-04T15:54:02.812 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:54:02.818 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:54:02.955 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:54:02.954+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:54:02.954521+0000) 2024-03-04T15:54:03.956 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:54:03.954+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:54:03.954756+0000) 2024-03-04T15:54:04.956 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:54:04.954+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:54:04.954991+0000) 2024-03-04T15:54:05.956 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:54:05.955+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:54:05.955281+0000) 2024-03-04T15:54:06.956 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:54:06.955+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:54:06.955517+0000) 2024-03-04T15:54:07.957 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:54:07.955+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:54:07.955757+0000) 2024-03-04T15:54:08.957 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:54:08.955+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:54:08.955997+0000) 2024-03-04T15:54:09.957 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:54:09.955+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:54:09.956245+0000) 2024-03-04T15:54:10.957 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:54:10.954+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:54:10.956523+0000) 2024-03-04T15:54:11.958 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:54:11.955+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:54:11.956772+0000) 2024-03-04T15:54:12.958 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:54:12.955+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:54:12.957008+0000) 2024-03-04T15:54:13.958 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:54:13.956+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:54:13.957243+0000) 2024-03-04T15:54:14.958 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:54:14.956+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:54:14.957477+0000) 2024-03-04T15:54:15.959 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:54:15.956+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:54:15.957730+0000) 2024-03-04T15:54:16.959 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:54:16.956+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:54:16.957987+0000) 2024-03-04T15:54:17.959 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:54:17.957+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:54:17.958248+0000) 2024-03-04T15:54:18.959 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:54:18.957+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:54:18.958486+0000) 2024-03-04T15:54:19.960 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:54:19.957+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:54:19.958728+0000) 2024-03-04T15:54:20.960 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:54:20.958+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:54:20.959003+0000) 2024-03-04T15:54:21.960 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:54:21.958+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:54:21.959257+0000) 2024-03-04T15:54:22.960 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:54:22.958+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:54:22.959507+0000) 2024-03-04T15:54:23.961 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:54:23.958+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:54:23.959740+0000) 2024-03-04T15:54:24.961 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:54:24.959+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:54:24.960010+0000) 2024-03-04T15:54:25.961 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:54:25.959+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:54:25.960247+0000) 2024-03-04T15:54:26.961 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:54:26.959+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:54:26.960501+0000) 2024-03-04T15:54:27.962 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:54:27.959+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:54:27.960751+0000) 2024-03-04T15:54:28.962 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:54:28.960+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:54:28.960987+0000) 2024-03-04T15:54:29.962 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:54:29.960+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:54:29.961243+0000) 2024-03-04T15:54:30.962 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:54:30.960+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:54:30.961507+0000) 2024-03-04T15:54:31.963 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:54:31.960+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:54:31.961747+0000) 2024-03-04T15:54:32.963 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:54:32.961+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:54:32.961949+0000) 2024-03-04T15:54:33.018 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:54:33.026 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:54:33.963 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:54:33.961+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:54:33.962184+0000) 2024-03-04T15:54:34.964 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:54:34.961+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:54:34.962445+0000) 2024-03-04T15:54:35.964 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:54:35.961+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:54:35.962685+0000) 2024-03-04T15:54:36.964 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:54:36.962+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:54:36.962924+0000) 2024-03-04T15:54:37.964 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:54:37.962+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:54:37.963162+0000) 2024-03-04T15:54:38.964 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:54:38.962+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:54:38.963395+0000) 2024-03-04T15:54:39.965 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:54:39.962+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:54:39.963634+0000) 2024-03-04T15:54:40.965 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:54:40.963+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:54:40.963923+0000) 2024-03-04T15:54:41.965 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:54:41.963+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:54:41.964164+0000) 2024-03-04T15:54:42.965 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:54:42.963+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:54:42.964427+0000) 2024-03-04T15:54:43.966 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:54:43.964+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:54:43.964664+0000) 2024-03-04T15:54:44.966 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:54:44.964+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:54:44.964943+0000) 2024-03-04T15:54:45.966 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:54:45.964+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:54:45.965192+0000) 2024-03-04T15:54:46.966 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:54:46.964+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:54:46.965427+0000) 2024-03-04T15:54:47.967 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:54:47.965+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:54:47.965662+0000) 2024-03-04T15:54:48.967 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:54:48.965+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:54:48.965955+0000) 2024-03-04T15:54:49.967 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:54:49.965+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:54:49.966145+0000) 2024-03-04T15:54:50.967 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:54:50.965+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:54:50.966392+0000) 2024-03-04T15:54:51.967 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:54:51.966+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:54:51.966631+0000) 2024-03-04T15:54:52.968 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:54:52.966+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:54:52.966917+0000) 2024-03-04T15:54:53.968 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:54:53.966+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:54:53.967207+0000) 2024-03-04T15:54:54.968 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:54:54.967+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:54:54.967470+0000) 2024-03-04T15:54:55.969 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:54:55.967+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:54:55.967710+0000) 2024-03-04T15:54:56.969 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:54:56.967+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:54:56.967982+0000) 2024-03-04T15:54:57.969 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:54:57.967+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:54:57.968220+0000) 2024-03-04T15:54:58.969 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:54:58.967+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:54:58.968455+0000) 2024-03-04T15:54:59.970 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:54:59.968+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:54:59.968700+0000) 2024-03-04T15:55:00.970 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:55:00.968+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:55:00.968970+0000) 2024-03-04T15:55:01.970 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:55:01.968+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:55:01.969237+0000) 2024-03-04T15:55:02.970 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:55:02.969+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:55:02.969481+0000) 2024-03-04T15:55:03.163 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:55:03.171 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:55:03.971 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:55:03.969+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:55:03.969715+0000) 2024-03-04T15:55:04.971 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:55:04.969+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:55:04.969966+0000) 2024-03-04T15:55:05.971 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:55:05.969+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:55:05.970209+0000) 2024-03-04T15:55:06.971 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:55:06.970+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:55:06.970447+0000) 2024-03-04T15:55:07.972 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:55:07.969+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:55:07.970706+0000) 2024-03-04T15:55:08.972 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:55:08.970+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:55:08.970944+0000) 2024-03-04T15:55:09.972 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:55:09.971+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:55:09.971191+0000) 2024-03-04T15:55:10.972 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:55:10.971+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:55:10.971436+0000) 2024-03-04T15:55:11.973 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:55:11.971+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:55:11.971675+0000) 2024-03-04T15:55:12.973 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:55:12.971+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:55:12.971948+0000) 2024-03-04T15:55:13.973 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:55:13.971+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:55:13.972182+0000) 2024-03-04T15:55:14.973 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:55:14.971+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:55:14.972424+0000) 2024-03-04T15:55:15.974 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:55:15.971+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:55:15.972669+0000) 2024-03-04T15:55:16.974 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:55:16.971+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:55:16.972950+0000) 2024-03-04T15:55:17.974 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:55:17.971+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:55:17.973139+0000) 2024-03-04T15:55:18.974 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:55:18.972+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:55:18.973376+0000) 2024-03-04T15:55:19.975 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:55:19.971+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:55:19.973634+0000) 2024-03-04T15:55:20.975 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:55:20.972+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:55:20.973942+0000) 2024-03-04T15:55:21.975 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:55:21.973+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:55:21.974180+0000) 2024-03-04T15:55:22.975 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:55:22.973+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:55:22.974429+0000) 2024-03-04T15:55:23.976 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:55:23.973+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:55:23.974668+0000) 2024-03-04T15:55:24.976 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:55:24.973+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:55:24.974943+0000) 2024-03-04T15:55:25.976 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:55:25.974+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:55:25.975186+0000) 2024-03-04T15:55:26.976 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:55:26.974+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:55:26.975425+0000) 2024-03-04T15:55:27.976 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:55:27.974+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:55:27.975667+0000) 2024-03-04T15:55:28.977 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:55:28.975+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:55:28.975947+0000) 2024-03-04T15:55:29.977 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:55:29.975+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:55:29.976186+0000) 2024-03-04T15:55:30.977 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:55:30.975+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:55:30.976461+0000) 2024-03-04T15:55:31.978 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:55:31.975+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:55:31.976759+0000) 2024-03-04T15:55:32.978 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:55:32.976+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:55:32.977061+0000) 2024-03-04T15:55:33.386 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:55:33.394 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:55:33.978 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:55:33.976+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:55:33.977348+0000) 2024-03-04T15:55:34.979 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:55:34.976+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:55:34.977642+0000) 2024-03-04T15:55:35.979 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:55:35.977+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:55:35.977963+0000) 2024-03-04T15:55:36.979 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:55:36.977+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:55:36.978247+0000) 2024-03-04T15:55:37.979 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:55:37.977+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:55:37.978553+0000) 2024-03-04T15:55:38.980 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:55:38.978+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:55:38.978903+0000) 2024-03-04T15:55:39.980 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:55:39.978+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:55:39.979196+0000) 2024-03-04T15:55:40.980 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:55:40.978+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:55:40.979504+0000) 2024-03-04T15:55:41.981 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:55:41.979+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:55:41.979793+0000) 2024-03-04T15:55:42.981 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:55:42.979+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:55:42.980141+0000) 2024-03-04T15:55:43.981 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:55:43.979+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:55:43.980463+0000) 2024-03-04T15:55:44.981 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:55:44.980+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:55:44.980754+0000) 2024-03-04T15:55:45.983 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:55:45.980+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:55:45.981047+0000) 2024-03-04T15:55:46.982 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:55:46.980+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:55:46.981350+0000) 2024-03-04T15:55:47.982 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:55:47.981+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:55:47.981722+0000) 2024-03-04T15:55:48.983 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:55:48.981+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:55:48.982008+0000) 2024-03-04T15:55:49.983 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:55:49.981+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:55:49.982319+0000) 2024-03-04T15:55:50.983 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:55:50.981+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:55:50.982615+0000) 2024-03-04T15:55:51.984 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:55:51.982+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:55:51.982956+0000) 2024-03-04T15:55:52.984 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:55:52.982+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:55:52.983202+0000) 2024-03-04T15:55:53.984 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:55:53.982+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:55:53.983436+0000) 2024-03-04T15:55:54.984 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:55:54.982+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:55:54.983670+0000) 2024-03-04T15:55:55.987 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:55:55.983+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:55:55.983956+0000) 2024-03-04T15:55:56.985 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:55:56.983+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:55:56.984223+0000) 2024-03-04T15:55:57.985 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:55:57.984+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:55:57.984471+0000) 2024-03-04T15:55:58.985 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:55:58.984+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:55:58.984701+0000) 2024-03-04T15:55:59.986 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:55:59.984+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:55:59.984934+0000) 2024-03-04T15:56:00.986 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:56:00.984+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:56:00.985184+0000) 2024-03-04T15:56:01.986 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:56:01.985+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:56:01.985420+0000) 2024-03-04T15:56:02.986 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:56:02.985+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:56:02.985663+0000) 2024-03-04T15:56:03.615 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:56:03.623 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:56:03.987 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:56:03.985+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:56:03.985942+0000) 2024-03-04T15:56:04.987 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:56:04.985+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:56:04.986183+0000) 2024-03-04T15:56:05.987 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:56:05.985+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:56:05.986370+0000) 2024-03-04T15:56:06.987 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:56:06.986+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:56:06.986642+0000) 2024-03-04T15:56:07.988 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:56:07.986+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:56:07.986933+0000) 2024-03-04T15:56:08.988 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:56:08.986+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:56:08.987165+0000) 2024-03-04T15:56:09.988 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:56:09.987+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:56:09.987400+0000) 2024-03-04T15:56:10.988 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:56:10.986+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:56:10.987601+0000) 2024-03-04T15:56:11.988 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:56:11.987+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:56:11.987865+0000) 2024-03-04T15:56:12.989 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:56:12.987+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:56:12.988104+0000) 2024-03-04T15:56:13.989 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:56:13.988+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:56:13.988374+0000) 2024-03-04T15:56:14.989 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:56:14.988+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:56:14.988615+0000) 2024-03-04T15:56:15.989 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:56:15.988+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:56:15.988892+0000) 2024-03-04T15:56:16.990 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:56:16.988+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:56:16.989084+0000) 2024-03-04T15:56:17.990 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:56:17.989+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:56:17.989326+0000) 2024-03-04T15:56:18.990 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:56:18.988+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:56:18.989563+0000) 2024-03-04T15:56:19.991 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:56:19.988+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:56:19.989804+0000) 2024-03-04T15:56:20.991 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:56:20.989+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:56:20.990093+0000) 2024-03-04T15:56:21.991 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:56:21.989+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:56:21.990333+0000) 2024-03-04T15:56:22.991 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:56:22.989+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:56:22.990574+0000) 2024-03-04T15:56:23.991 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:56:23.989+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:56:23.990805+0000) 2024-03-04T15:56:24.992 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:56:24.990+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:56:24.991088+0000) 2024-03-04T15:56:25.992 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:56:25.990+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:56:25.991327+0000) 2024-03-04T15:56:26.992 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:56:26.990+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:56:26.991589+0000) 2024-03-04T15:56:27.992 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:56:27.990+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:56:27.991895+0000) 2024-03-04T15:56:28.993 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:56:28.991+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:56:28.992127+0000) 2024-03-04T15:56:29.993 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:56:29.991+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:56:29.992373+0000) 2024-03-04T15:56:30.993 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:56:30.991+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:56:30.992585+0000) 2024-03-04T15:56:31.994 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:56:31.992+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:56:31.992876+0000) 2024-03-04T15:56:32.994 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:56:32.992+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:56:32.993116+0000) 2024-03-04T15:56:33.855 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:56:33.863 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:56:33.994 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:56:33.992+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:56:33.993370+0000) 2024-03-04T15:56:34.994 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:56:34.992+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:56:34.993651+0000) 2024-03-04T15:56:35.995 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:56:35.993+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:56:35.994017+0000) 2024-03-04T15:56:36.995 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:56:36.993+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:56:36.994302+0000) 2024-03-04T15:56:37.995 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:56:37.993+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:56:37.994596+0000) 2024-03-04T15:56:38.996 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:56:38.994+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:56:38.994920+0000) 2024-03-04T15:56:39.996 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:56:39.994+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:56:39.995216+0000) 2024-03-04T15:56:40.996 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:56:40.994+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:56:40.995518+0000) 2024-03-04T15:56:41.996 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:56:41.995+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:56:41.995859+0000) 2024-03-04T15:56:42.997 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:56:42.995+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:56:42.996158+0000) 2024-03-04T15:56:43.997 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:56:43.995+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:56:43.996449+0000) 2024-03-04T15:56:44.997 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:56:44.995+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:56:44.996740+0000) 2024-03-04T15:56:45.998 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:56:45.996+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:56:45.997029+0000) 2024-03-04T15:56:46.998 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:56:46.996+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:56:46.997323+0000) 2024-03-04T15:56:47.998 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:56:47.996+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:56:47.997615+0000) 2024-03-04T15:56:48.998 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:56:48.997+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:56:48.997939+0000) 2024-03-04T15:56:49.999 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:56:49.997+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:56:49.998233+0000) 2024-03-04T15:56:50.999 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:56:50.997+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:56:50.998530+0000) 2024-03-04T15:56:51.999 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:56:51.998+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:56:51.998814+0000) 2024-03-04T15:56:53.000 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:56:52.998+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:56:52.999172+0000) 2024-03-04T15:56:54.000 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:56:53.998+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:56:53.999493+0000) 2024-03-04T15:56:55.000 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:56:54.999+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:56:54.999785+0000) 2024-03-04T15:56:56.001 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:56:55.999+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:56:56.000068+0000) 2024-03-04T15:56:57.001 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:56:56.999+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:56:57.000350+0000) 2024-03-04T15:56:58.001 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:56:58.000+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:56:58.000637+0000) 2024-03-04T15:56:59.001 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:56:59.000+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:56:59.000960+0000) 2024-03-04T15:57:00.002 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:57:00.000+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:57:00.001246+0000) 2024-03-04T15:57:01.002 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:57:01.000+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:57:01.001500+0000) 2024-03-04T15:57:02.002 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:57:02.001+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:57:02.001810+0000) 2024-03-04T15:57:03.003 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:57:03.001+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:57:03.002162+0000) 2024-03-04T15:57:04.003 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:57:04.001+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:57:04.002452+0000) 2024-03-04T15:57:04.067 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:57:04.079 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:57:05.003 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:57:05.002+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:57:05.002752+0000) 2024-03-04T15:57:06.004 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:57:06.002+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:57:06.003026+0000) 2024-03-04T15:57:07.004 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:57:07.002+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:57:07.003318+0000) 2024-03-04T15:57:08.004 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:57:08.003+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:57:08.003619+0000) 2024-03-04T15:57:09.005 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:57:09.003+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:57:09.003949+0000) 2024-03-04T15:57:10.005 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:57:10.003+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:57:10.004199+0000) 2024-03-04T15:57:11.005 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:57:11.004+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:57:11.004449+0000) 2024-03-04T15:57:12.005 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:57:12.004+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:57:12.004690+0000) 2024-03-04T15:57:13.005 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:57:13.004+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:57:13.004937+0000) 2024-03-04T15:57:14.006 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:57:14.004+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:57:14.005175+0000) 2024-03-04T15:57:15.006 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:57:15.004+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:57:15.005372+0000) 2024-03-04T15:57:16.006 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:57:16.005+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:57:16.005614+0000) 2024-03-04T15:57:17.008 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:57:17.005+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:57:17.005885+0000) 2024-03-04T15:57:18.007 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:57:18.005+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:57:18.006142+0000) 2024-03-04T15:57:19.007 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:57:19.006+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:57:19.006376+0000) 2024-03-04T15:57:20.007 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:57:20.006+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:57:20.006612+0000) 2024-03-04T15:57:21.007 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:57:21.006+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:57:21.006904+0000) 2024-03-04T15:57:22.008 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:57:22.007+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:57:22.007167+0000) 2024-03-04T15:57:23.008 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:57:23.006+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:57:23.007419+0000) 2024-03-04T15:57:24.008 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:57:24.006+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:57:24.007732+0000) 2024-03-04T15:57:25.008 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:57:25.006+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:57:25.007970+0000) 2024-03-04T15:57:26.009 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:57:26.007+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:57:26.008241+0000) 2024-03-04T15:57:27.009 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:57:27.007+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:57:27.008515+0000) 2024-03-04T15:57:28.009 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:57:28.007+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:57:28.008803+0000) 2024-03-04T15:57:29.010 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:57:29.008+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:57:29.009151+0000) 2024-03-04T15:57:30.010 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:57:30.008+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:57:30.009406+0000) 2024-03-04T15:57:31.010 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:57:31.008+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:57:31.009667+0000) 2024-03-04T15:57:32.010 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:57:32.008+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:57:32.010008+0000) 2024-03-04T15:57:33.011 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:57:33.009+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:57:33.010216+0000) 2024-03-04T15:57:34.011 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:57:34.009+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:57:34.010473+0000) 2024-03-04T15:57:34.298 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:57:34.306 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:57:35.011 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:57:35.009+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:57:35.010719+0000) 2024-03-04T15:57:36.011 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:57:36.010+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:57:36.010975+0000) 2024-03-04T15:57:37.012 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:57:37.010+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:57:37.011213+0000) 2024-03-04T15:57:38.013 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:57:38.010+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:57:38.011414+0000) 2024-03-04T15:57:39.012 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:57:39.010+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:57:39.011666+0000) 2024-03-04T15:57:40.013 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:57:40.011+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:57:40.011939+0000) 2024-03-04T15:57:41.013 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:57:41.011+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:57:41.012197+0000) 2024-03-04T15:57:42.013 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:57:42.011+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:57:42.012441+0000) 2024-03-04T15:57:43.013 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:57:43.011+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:57:43.012688+0000) 2024-03-04T15:57:44.013 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:57:44.012+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:57:44.012934+0000) 2024-03-04T15:57:45.014 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:57:45.012+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:57:45.013136+0000) 2024-03-04T15:57:46.014 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:57:46.012+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:57:46.013391+0000) 2024-03-04T15:57:47.014 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:57:47.012+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:57:47.013639+0000) 2024-03-04T15:57:48.014 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:57:48.013+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:57:48.013933+0000) 2024-03-04T15:57:49.015 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:57:49.013+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:57:49.014171+0000) 2024-03-04T15:57:50.015 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:57:50.013+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:57:50.014427+0000) 2024-03-04T15:57:51.016 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:57:51.014+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:57:51.014706+0000) 2024-03-04T15:57:52.015 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:57:52.014+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:57:52.014959+0000) 2024-03-04T15:57:53.016 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:57:53.014+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:57:53.015199+0000) 2024-03-04T15:57:54.016 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:57:54.014+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:57:54.015388+0000) 2024-03-04T15:57:55.016 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:57:55.015+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:57:55.015641+0000) 2024-03-04T15:57:56.016 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:57:56.015+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:57:56.015942+0000) 2024-03-04T15:57:57.016 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:57:57.015+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:57:57.016186+0000) 2024-03-04T15:57:58.017 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:57:58.015+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:57:58.016431+0000) 2024-03-04T15:57:59.017 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:57:59.016+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:57:59.016687+0000) 2024-03-04T15:58:00.017 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:58:00.016+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:58:00.016935+0000) 2024-03-04T15:58:01.017 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:58:01.016+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:58:01.017145+0000) 2024-03-04T15:58:02.018 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:58:02.016+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:58:02.017388+0000) 2024-03-04T15:58:03.018 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:58:03.016+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:58:03.017583+0000) 2024-03-04T15:58:04.018 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:58:04.017+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:58:04.017880+0000) 2024-03-04T15:58:04.443 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:58:04.454 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:58:05.019 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:58:05.017+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:58:05.018126+0000) 2024-03-04T15:58:06.019 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:58:06.017+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:58:06.018400+0000) 2024-03-04T15:58:07.019 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:58:07.018+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:58:07.018653+0000) 2024-03-04T15:58:08.019 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:58:08.018+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:58:08.018942+0000) 2024-03-04T15:58:09.020 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:58:09.018+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:58:09.019190+0000) 2024-03-04T15:58:10.020 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:58:10.019+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:58:10.019434+0000) 2024-03-04T15:58:11.020 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:58:11.019+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:58:11.019939+0000) 2024-03-04T15:58:12.021 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:58:12.019+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:58:12.020285+0000) 2024-03-04T15:58:13.021 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:58:13.019+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:58:13.020558+0000) 2024-03-04T15:58:14.021 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:58:14.020+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:58:14.020796+0000) 2024-03-04T15:58:15.021 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:58:15.020+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:58:15.021090+0000) 2024-03-04T15:58:16.022 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:58:16.020+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:58:16.021334+0000) 2024-03-04T15:58:17.022 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:58:17.020+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:58:17.021564+0000) 2024-03-04T15:58:18.022 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:58:18.020+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:58:18.021807+0000) 2024-03-04T15:58:19.022 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:58:19.021+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:58:19.022050+0000) 2024-03-04T15:58:20.023 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:58:20.021+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:58:20.022259+0000) 2024-03-04T15:58:21.023 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:58:21.021+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:58:21.022546+0000) 2024-03-04T15:58:22.024 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:58:22.021+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:58:22.022772+0000) 2024-03-04T15:58:23.023 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:58:23.022+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:58:23.022993+0000) 2024-03-04T15:58:24.024 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:58:24.022+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:58:24.023220+0000) 2024-03-04T15:58:25.024 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:58:25.022+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:58:25.023459+0000) 2024-03-04T15:58:26.024 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:58:26.022+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:58:26.023720+0000) 2024-03-04T15:58:27.024 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:58:27.023+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:58:27.024005+0000) 2024-03-04T15:58:28.024 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:58:28.022+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:58:28.024268+0000) 2024-03-04T15:58:29.025 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:58:29.022+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:58:29.024468+0000) 2024-03-04T15:58:30.025 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:58:30.022+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:58:30.024711+0000) 2024-03-04T15:58:31.025 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:58:31.023+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:58:31.024959+0000) 2024-03-04T15:58:32.025 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:58:32.023+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:58:32.025164+0000) 2024-03-04T15:58:33.026 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:58:33.023+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:58:33.025368+0000) 2024-03-04T15:58:34.026 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:58:34.023+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:58:34.025562+0000) 2024-03-04T15:58:34.591 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:58:34.599 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:58:35.026 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:58:35.023+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:58:35.025805+0000) 2024-03-04T15:58:36.026 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:58:36.024+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:58:36.026056+0000) 2024-03-04T15:58:37.026 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:58:37.024+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:58:37.026254+0000) 2024-03-04T15:58:38.027 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:58:38.024+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:58:38.026482+0000) 2024-03-04T15:58:39.027 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:58:39.025+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:58:39.026735+0000) 2024-03-04T15:58:40.027 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:58:40.026+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:58:40.026997+0000) 2024-03-04T15:58:41.028 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:58:41.026+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:58:41.027301+0000) 2024-03-04T15:58:42.028 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:58:42.026+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:58:42.027577+0000) 2024-03-04T15:58:43.028 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:58:43.027+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:58:43.027856+0000) 2024-03-04T15:58:44.028 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:58:44.027+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:58:44.028100+0000) 2024-03-04T15:58:45.029 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:58:45.027+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:58:45.028340+0000) 2024-03-04T15:58:46.029 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:58:46.027+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:58:46.028560+0000) 2024-03-04T15:58:47.029 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:58:47.028+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:58:47.028855+0000) 2024-03-04T15:58:48.029 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:58:48.028+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:58:48.029092+0000) 2024-03-04T15:58:49.030 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:58:49.028+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:58:49.029329+0000) 2024-03-04T15:58:50.030 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:58:50.028+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:58:50.029589+0000) 2024-03-04T15:58:51.030 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:58:51.029+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:58:51.029921+0000) 2024-03-04T15:58:52.030 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:58:52.029+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:58:52.030176+0000) 2024-03-04T15:58:53.031 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:58:53.029+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:58:53.030419+0000) 2024-03-04T15:58:54.031 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:58:54.029+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:58:54.030667+0000) 2024-03-04T15:58:55.031 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:58:55.030+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:58:55.030936+0000) 2024-03-04T15:58:56.031 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:58:56.030+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:58:56.031176+0000) 2024-03-04T15:58:57.032 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:58:57.030+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:58:57.031444+0000) 2024-03-04T15:58:58.032 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:58:58.030+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:58:58.031715+0000) 2024-03-04T15:58:59.032 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:58:59.031+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:58:59.031956+0000) 2024-03-04T15:59:00.033 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:59:00.031+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:59:00.032190+0000) 2024-03-04T15:59:01.033 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:59:01.031+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:59:01.032441+0000) 2024-03-04T15:59:02.033 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:59:02.031+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:59:02.032700+0000) 2024-03-04T15:59:03.033 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:59:03.032+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:59:03.032939+0000) 2024-03-04T15:59:04.033 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:59:04.032+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:59:04.033232+0000) 2024-03-04T15:59:04.812 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:59:04.821 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:59:05.034 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:59:05.033+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:59:05.033867+0000) 2024-03-04T15:59:06.034 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:59:06.033+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:59:06.034139+0000) 2024-03-04T15:59:07.035 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:59:07.033+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:59:07.034372+0000) 2024-03-04T15:59:08.035 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:59:08.034+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:59:08.034633+0000) 2024-03-04T15:59:09.035 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:59:09.034+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:59:09.034919+0000) 2024-03-04T15:59:10.035 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:59:10.034+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:59:10.035185+0000) 2024-03-04T15:59:11.036 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:59:11.035+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:59:11.035446+0000) 2024-03-04T15:59:12.036 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:59:12.035+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:59:12.035704+0000) 2024-03-04T15:59:13.036 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:59:13.035+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:59:13.035963+0000) 2024-03-04T15:59:14.036 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:59:14.035+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:59:14.036213+0000) 2024-03-04T15:59:15.037 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:59:15.035+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:59:15.036412+0000) 2024-03-04T15:59:16.037 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:59:16.036+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:59:16.036671+0000) 2024-03-04T15:59:17.037 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:59:17.036+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:59:17.036947+0000) 2024-03-04T15:59:18.037 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:59:18.036+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:59:18.037181+0000) 2024-03-04T15:59:19.038 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:59:19.037+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:59:19.037440+0000) 2024-03-04T15:59:20.038 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:59:20.037+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:59:20.037681+0000) 2024-03-04T15:59:21.038 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:59:21.037+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:59:21.037970+0000) 2024-03-04T15:59:22.038 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:59:22.037+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:59:22.038243+0000) 2024-03-04T15:59:23.039 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:59:23.038+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:59:23.038506+0000) 2024-03-04T15:59:24.039 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:59:24.038+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:59:24.038755+0000) 2024-03-04T15:59:25.039 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:59:25.038+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:59:25.039009+0000) 2024-03-04T15:59:26.039 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:59:26.039+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:59:26.039269+0000) 2024-03-04T15:59:27.040 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:59:27.039+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:59:27.039504+0000) 2024-03-04T15:59:28.040 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:59:28.039+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:59:28.039748+0000) 2024-03-04T15:59:29.041 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:59:29.039+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:59:29.039992+0000) 2024-03-04T15:59:30.040 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:59:30.039+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:59:30.040182+0000) 2024-03-04T15:59:31.040 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:59:31.040+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:59:31.040406+0000) 2024-03-04T15:59:32.041 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:59:32.039+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:59:32.040669+0000) 2024-03-04T15:59:33.041 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:59:33.039+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:59:33.040950+0000) 2024-03-04T15:59:34.041 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:59:34.040+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:59:34.041215+0000) 2024-03-04T15:59:35.008 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:59:35.017 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T15:59:35.042 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:59:35.039+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:59:35.041417+0000) 2024-03-04T15:59:36.042 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:59:36.040+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:59:36.041645+0000) 2024-03-04T15:59:37.042 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:59:37.040+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:59:37.041909+0000) 2024-03-04T15:59:38.042 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:59:38.041+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:59:38.042153+0000) 2024-03-04T15:59:39.043 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:59:39.041+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:59:39.042392+0000) 2024-03-04T15:59:40.043 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:59:40.041+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:59:40.042630+0000) 2024-03-04T15:59:41.043 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:59:41.041+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:59:41.042921+0000) 2024-03-04T15:59:42.043 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:59:42.042+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:59:42.043165+0000) 2024-03-04T15:59:43.044 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:59:43.042+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:59:43.043405+0000) 2024-03-04T15:59:44.044 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:59:44.042+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:59:44.043595+0000) 2024-03-04T15:59:45.044 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:59:45.042+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:59:45.043781+0000) 2024-03-04T15:59:46.044 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:59:46.043+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:59:46.044033+0000) 2024-03-04T15:59:47.044 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:59:47.043+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:59:47.044279+0000) 2024-03-04T15:59:48.045 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:59:48.043+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:59:48.044522+0000) 2024-03-04T15:59:49.045 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:59:49.043+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:59:49.044775+0000) 2024-03-04T15:59:50.045 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:59:50.044+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:59:50.045035+0000) 2024-03-04T15:59:51.045 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:59:51.044+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:59:51.045284+0000) 2024-03-04T15:59:52.046 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:59:52.044+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:59:52.045560+0000) 2024-03-04T15:59:53.046 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:59:53.045+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:59:53.045860+0000) 2024-03-04T15:59:54.046 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:59:54.045+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:59:54.046102+0000) 2024-03-04T15:59:55.047 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:59:55.045+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:59:55.046346+0000) 2024-03-04T15:59:56.047 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:59:56.045+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:59:56.046588+0000) 2024-03-04T15:59:57.047 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:59:57.046+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:59:57.046869+0000) 2024-03-04T15:59:58.047 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:59:58.046+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:59:58.047205+0000) 2024-03-04T15:59:59.047 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T15:59:59.046+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T14:59:59.047390+0000) 2024-03-04T16:00:00.048 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:00:00.046+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:00:00.047670+0000) 2024-03-04T16:00:01.048 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:00:01.047+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:00:01.047949+0000) 2024-03-04T16:00:02.048 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:00:02.047+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:00:02.048148+0000) 2024-03-04T16:00:03.048 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:00:03.047+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:00:03.048399+0000) 2024-03-04T16:00:04.049 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:00:04.048+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:00:04.048638+0000) 2024-03-04T16:00:05.049 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:00:05.048+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:00:05.048921+0000) 2024-03-04T16:00:05.172 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:00:05.180 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:00:06.049 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:00:06.048+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:00:06.049161+0000) 2024-03-04T16:00:07.050 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:00:07.048+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:00:07.049397+0000) 2024-03-04T16:00:08.050 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:00:08.048+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:00:08.049640+0000) 2024-03-04T16:00:09.050 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:00:09.049+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:00:09.049927+0000) 2024-03-04T16:00:10.050 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:00:10.049+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:00:10.050170+0000) 2024-03-04T16:00:11.050 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:00:11.049+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:00:11.050429+0000) 2024-03-04T16:00:12.051 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:00:12.050+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:00:12.050682+0000) 2024-03-04T16:00:13.051 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:00:13.050+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:00:13.050959+0000) 2024-03-04T16:00:14.051 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:00:14.050+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:00:14.051198+0000) 2024-03-04T16:00:15.052 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:00:15.050+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:00:15.051449+0000) 2024-03-04T16:00:16.052 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:00:16.051+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:00:16.051715+0000) 2024-03-04T16:00:17.052 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:00:17.051+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:00:17.051955+0000) 2024-03-04T16:00:18.052 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:00:18.051+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:00:18.052229+0000) 2024-03-04T16:00:19.053 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:00:19.052+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:00:19.052491+0000) 2024-03-04T16:00:20.056 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:00:20.052+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:00:20.052727+0000) 2024-03-04T16:00:21.053 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:00:21.052+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:00:21.052975+0000) 2024-03-04T16:00:22.053 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:00:22.052+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:00:22.053220+0000) 2024-03-04T16:00:23.054 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:00:23.053+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:00:23.053468+0000) 2024-03-04T16:00:24.054 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:00:24.053+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:00:24.053703+0000) 2024-03-04T16:00:25.054 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:00:25.053+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:00:25.053979+0000) 2024-03-04T16:00:26.054 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:00:26.053+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:00:26.054162+0000) 2024-03-04T16:00:27.054 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:00:27.054+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:00:27.054431+0000) 2024-03-04T16:00:28.055 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:00:28.054+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:00:28.054680+0000) 2024-03-04T16:00:29.055 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:00:29.054+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:00:29.054937+0000) 2024-03-04T16:00:30.055 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:00:30.054+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:00:30.055179+0000) 2024-03-04T16:00:31.055 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:00:31.055+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:00:31.055427+0000) 2024-03-04T16:00:32.056 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:00:32.055+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:00:32.055667+0000) 2024-03-04T16:00:33.056 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:00:33.055+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:00:33.055951+0000) 2024-03-04T16:00:34.056 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:00:34.056+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:00:34.056187+0000) 2024-03-04T16:00:35.056 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:00:35.056+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:00:35.056436+0000) 2024-03-04T16:00:35.378 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:00:35.391 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:00:36.057 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:00:36.055+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:00:36.056686+0000) 2024-03-04T16:00:37.057 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:00:37.055+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:00:37.056925+0000) 2024-03-04T16:00:38.057 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:00:38.056+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:00:38.057172+0000) 2024-03-04T16:00:39.057 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:00:39.056+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:00:39.057414+0000) 2024-03-04T16:00:40.058 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:00:40.056+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:00:40.057651+0000) 2024-03-04T16:00:41.058 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:00:41.056+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:00:41.057947+0000) 2024-03-04T16:00:42.058 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:00:42.057+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:00:42.058190+0000) 2024-03-04T16:00:43.058 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:00:43.057+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:00:43.058438+0000) 2024-03-04T16:00:44.059 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:00:44.056+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:00:44.058615+0000) 2024-03-04T16:00:45.059 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:00:45.057+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:00:45.058891+0000) 2024-03-04T16:00:46.059 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:00:46.058+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:00:46.059190+0000) 2024-03-04T16:00:47.060 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:00:47.058+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:00:47.059429+0000) 2024-03-04T16:00:48.060 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:00:48.058+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:00:48.059667+0000) 2024-03-04T16:00:49.060 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:00:49.059+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:00:49.059920+0000) 2024-03-04T16:00:50.060 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:00:50.059+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:00:50.060179+0000) 2024-03-04T16:00:51.060 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:00:51.059+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:00:51.060432+0000) 2024-03-04T16:00:52.061 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:00:52.059+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:00:52.060671+0000) 2024-03-04T16:00:53.061 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:00:53.060+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:00:53.060958+0000) 2024-03-04T16:00:54.061 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:00:54.060+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:00:54.061157+0000) 2024-03-04T16:00:55.061 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:00:55.060+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:00:55.061396+0000) 2024-03-04T16:00:56.062 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:00:56.060+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:00:56.061635+0000) 2024-03-04T16:00:57.062 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:00:57.061+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:00:57.061918+0000) 2024-03-04T16:00:58.062 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:00:58.061+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:00:58.062178+0000) 2024-03-04T16:00:59.062 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:00:59.061+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:00:59.062388+0000) 2024-03-04T16:01:00.063 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:01:00.061+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:01:00.062629+0000) 2024-03-04T16:01:01.063 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:01:01.062+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:01:01.062925+0000) 2024-03-04T16:01:02.063 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:01:02.062+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:01:02.063173+0000) 2024-03-04T16:01:03.064 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:01:03.062+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:01:03.063417+0000) 2024-03-04T16:01:04.064 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:01:04.062+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:01:04.063652+0000) 2024-03-04T16:01:05.064 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:01:05.063+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:01:05.063934+0000) 2024-03-04T16:01:05.609 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:01:05.617 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:01:06.064 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:01:06.063+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:01:06.064178+0000) 2024-03-04T16:01:07.064 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:01:07.063+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:01:07.064426+0000) 2024-03-04T16:01:08.065 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:01:08.063+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:01:08.064683+0000) 2024-03-04T16:01:09.065 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:01:09.064+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:01:09.064920+0000) 2024-03-04T16:01:10.065 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:01:10.064+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:01:10.065159+0000) 2024-03-04T16:01:11.065 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:01:11.064+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:01:11.065409+0000) 2024-03-04T16:01:12.066 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:01:12.065+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:01:12.065653+0000) 2024-03-04T16:01:13.066 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:01:13.065+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:01:13.065936+0000) 2024-03-04T16:01:14.066 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:01:14.065+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:01:14.066161+0000) 2024-03-04T16:01:15.066 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:01:15.065+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:01:15.066399+0000) 2024-03-04T16:01:16.067 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:01:16.066+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:01:16.066638+0000) 2024-03-04T16:01:17.069 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:01:17.066+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:01:17.066932+0000) 2024-03-04T16:01:18.067 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:01:18.066+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:01:18.067181+0000) 2024-03-04T16:01:19.068 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:01:19.066+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:01:19.067434+0000) 2024-03-04T16:01:20.068 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:01:20.067+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:01:20.067690+0000) 2024-03-04T16:01:21.068 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:01:21.067+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:01:21.067955+0000) 2024-03-04T16:01:22.068 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:01:22.067+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:01:22.068270+0000) 2024-03-04T16:01:23.068 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:01:23.068+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:01:23.068516+0000) 2024-03-04T16:01:24.072 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:01:24.067+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:01:24.068772+0000) 2024-03-04T16:01:25.069 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:01:25.068+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:01:25.069023+0000) 2024-03-04T16:01:26.069 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:01:26.068+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:01:26.069252+0000) 2024-03-04T16:01:27.070 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:01:27.068+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:01:27.069432+0000) 2024-03-04T16:01:28.070 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:01:28.069+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:01:28.069675+0000) 2024-03-04T16:01:29.070 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:01:29.069+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:01:29.069995+0000) 2024-03-04T16:01:30.070 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:01:30.069+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:01:30.070232+0000) 2024-03-04T16:01:31.070 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:01:31.070+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:01:31.070484+0000) 2024-03-04T16:01:32.071 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:01:32.070+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:01:32.070726+0000) 2024-03-04T16:01:33.072 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:01:33.070+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:01:33.071010+0000) 2024-03-04T16:01:34.071 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:01:34.071+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:01:34.071266+0000) 2024-03-04T16:01:35.072 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:01:35.071+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:01:35.071505+0000) 2024-03-04T16:01:35.830 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:01:35.839 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:01:36.072 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:01:36.071+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:01:36.071751+0000) 2024-03-04T16:01:37.072 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:01:37.071+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:01:37.072014+0000) 2024-03-04T16:01:38.072 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:01:38.072+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:01:38.072276+0000) 2024-03-04T16:01:39.072 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:01:39.072+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:01:39.072503+0000) 2024-03-04T16:01:40.073 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:01:40.071+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:01:40.072746+0000) 2024-03-04T16:01:41.073 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:01:41.071+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:01:41.072992+0000) 2024-03-04T16:01:42.073 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:01:42.072+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:01:42.073234+0000) 2024-03-04T16:01:43.073 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:01:43.072+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:01:43.073479+0000) 2024-03-04T16:01:44.074 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:01:44.072+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:01:44.073719+0000) 2024-03-04T16:01:45.074 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:01:45.072+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:01:45.073956+0000) 2024-03-04T16:01:46.074 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:01:46.073+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:01:46.074194+0000) 2024-03-04T16:01:47.074 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:01:47.072+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:01:47.074394+0000) 2024-03-04T16:01:48.075 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:01:48.073+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:01:48.074647+0000) 2024-03-04T16:01:49.075 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:01:49.073+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:01:49.074942+0000) 2024-03-04T16:01:50.075 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:01:50.074+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:01:50.075188+0000) 2024-03-04T16:01:51.075 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:01:51.074+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:01:51.075438+0000) 2024-03-04T16:01:52.076 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:01:52.074+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:01:52.075685+0000) 2024-03-04T16:01:53.076 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:01:53.075+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:01:53.075947+0000) 2024-03-04T16:01:54.076 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:01:54.075+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:01:54.076250+0000) 2024-03-04T16:01:55.076 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:01:55.075+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:01:55.076534+0000) 2024-03-04T16:01:56.077 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:01:56.076+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:01:56.076875+0000) 2024-03-04T16:01:57.077 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:01:57.076+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:01:57.077126+0000) 2024-03-04T16:01:58.077 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:01:58.076+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:01:58.077418+0000) 2024-03-04T16:01:59.078 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:01:59.076+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:01:59.077713+0000) 2024-03-04T16:02:00.078 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:02:00.077+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:02:00.077991+0000) 2024-03-04T16:02:01.078 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:02:01.077+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:02:01.078292+0000) 2024-03-04T16:02:02.079 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:02:02.077+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:02:02.078548+0000) 2024-03-04T16:02:03.084 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:02:03.078+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:02:03.078803+0000) 2024-03-04T16:02:04.079 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:02:04.078+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:02:04.079080+0000) 2024-03-04T16:02:05.079 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:02:05.078+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:02:05.079318+0000) 2024-03-04T16:02:06.025 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:02:06.033 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:02:06.079 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:02:06.078+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:02:06.079525+0000) 2024-03-04T16:02:07.080 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:02:07.079+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:02:07.079754+0000) 2024-03-04T16:02:08.080 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:02:08.079+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:02:08.080009+0000) 2024-03-04T16:02:09.081 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:02:09.079+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:02:09.080244+0000) 2024-03-04T16:02:10.080 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:02:10.079+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:02:10.080483+0000) 2024-03-04T16:02:11.081 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:02:11.080+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:02:11.080736+0000) 2024-03-04T16:02:12.081 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:02:12.080+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:02:12.080980+0000) 2024-03-04T16:02:13.081 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:02:13.080+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:02:13.081229+0000) 2024-03-04T16:02:14.083 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:02:14.080+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:02:14.081465+0000) 2024-03-04T16:02:15.082 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:02:15.081+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:02:15.081700+0000) 2024-03-04T16:02:16.082 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:02:16.081+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:02:16.082021+0000) 2024-03-04T16:02:17.082 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:02:17.081+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:02:17.082271+0000) 2024-03-04T16:02:18.082 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:02:18.082+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:02:18.082534+0000) 2024-03-04T16:02:19.083 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:02:19.082+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:02:19.082798+0000) 2024-03-04T16:02:20.083 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:02:20.082+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:02:20.083073+0000) 2024-03-04T16:02:21.083 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:02:21.082+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:02:21.083322+0000) 2024-03-04T16:02:22.083 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:02:22.083+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:02:22.083631+0000) 2024-03-04T16:02:23.084 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:02:23.083+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:02:23.083929+0000) 2024-03-04T16:02:24.084 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:02:24.083+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:02:24.084164+0000) 2024-03-04T16:02:25.084 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:02:25.083+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:02:25.084419+0000) 2024-03-04T16:02:26.085 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:02:26.084+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:02:26.084659+0000) 2024-03-04T16:02:27.085 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:02:27.084+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:02:27.084950+0000) 2024-03-04T16:02:28.085 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:02:28.084+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:02:28.085200+0000) 2024-03-04T16:02:29.085 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:02:29.084+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:02:29.085459+0000) 2024-03-04T16:02:30.086 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:02:30.085+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:02:30.085697+0000) 2024-03-04T16:02:31.086 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:02:31.085+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:02:31.085967+0000) 2024-03-04T16:02:32.086 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:02:32.085+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:02:32.086215+0000) 2024-03-04T16:02:33.086 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:02:33.086+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:02:33.086471+0000) 2024-03-04T16:02:34.086 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:02:34.086+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:02:34.086707+0000) 2024-03-04T16:02:35.087 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:02:35.086+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:02:35.086949+0000) 2024-03-04T16:02:36.087 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:02:36.086+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:02:36.087194+0000) 2024-03-04T16:02:36.159 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:02:36.168 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:02:37.087 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:02:37.087+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:02:37.087435+0000) 2024-03-04T16:02:38.087 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:02:38.086+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:02:38.087692+0000) 2024-03-04T16:02:39.088 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:02:39.087+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:02:39.087929+0000) 2024-03-04T16:02:40.088 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:02:40.087+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:02:40.088175+0000) 2024-03-04T16:02:41.088 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:02:41.088+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:02:41.088429+0000) 2024-03-04T16:02:42.088 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:02:42.088+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:02:42.088664+0000) 2024-03-04T16:02:43.089 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:02:43.088+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:02:43.088952+0000) 2024-03-04T16:02:44.089 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:02:44.088+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:02:44.089190+0000) 2024-03-04T16:02:45.089 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:02:45.088+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:02:45.089426+0000) 2024-03-04T16:02:46.090 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:02:46.088+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:02:46.089670+0000) 2024-03-04T16:02:47.090 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:02:47.088+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:02:47.089946+0000) 2024-03-04T16:02:48.090 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:02:48.089+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:02:48.090212+0000) 2024-03-04T16:02:49.090 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:02:49.088+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:02:49.090443+0000) 2024-03-04T16:02:50.090 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:02:50.089+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:02:50.090686+0000) 2024-03-04T16:02:51.091 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:02:51.089+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:02:51.090952+0000) 2024-03-04T16:02:52.091 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:02:52.090+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:02:52.091181+0000) 2024-03-04T16:02:53.091 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:02:53.090+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:02:53.091427+0000) 2024-03-04T16:02:54.092 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:02:54.090+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:02:54.091655+0000) 2024-03-04T16:02:55.092 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:02:55.091+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:02:55.091931+0000) 2024-03-04T16:02:56.092 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:02:56.091+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:02:56.092205+0000) 2024-03-04T16:02:57.092 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:02:57.091+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:02:57.092459+0000) 2024-03-04T16:02:58.093 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:02:58.091+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:02:58.092706+0000) 2024-03-04T16:02:59.093 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:02:59.092+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:02:59.092943+0000) 2024-03-04T16:03:00.093 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:03:00.092+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:03:00.093223+0000) 2024-03-04T16:03:01.093 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:03:01.092+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:03:01.093479+0000) 2024-03-04T16:03:02.094 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:03:02.092+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:03:02.093728+0000) 2024-03-04T16:03:03.094 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:03:03.093+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:03:03.093980+0000) 2024-03-04T16:03:04.094 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:03:04.093+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:03:04.094211+0000) 2024-03-04T16:03:05.094 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:03:05.093+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:03:05.094456+0000) 2024-03-04T16:03:06.095 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:03:06.093+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:03:06.094704+0000) 2024-03-04T16:03:06.320 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:03:06.328 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:03:07.095 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:03:07.094+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:03:07.094956+0000) 2024-03-04T16:03:08.096 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:03:08.094+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:03:08.095210+0000) 2024-03-04T16:03:09.095 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:03:09.094+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:03:09.095445+0000) 2024-03-04T16:03:10.096 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:03:10.094+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:03:10.095683+0000) 2024-03-04T16:03:11.096 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:03:11.095+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:03:11.095953+0000) 2024-03-04T16:03:12.096 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:03:12.095+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:03:12.096203+0000) 2024-03-04T16:03:13.096 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:03:13.095+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:03:13.096456+0000) 2024-03-04T16:03:14.096 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:03:14.096+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:03:14.096701+0000) 2024-03-04T16:03:15.097 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:03:15.096+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:03:15.096942+0000) 2024-03-04T16:03:16.097 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:03:16.096+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:03:16.097241+0000) 2024-03-04T16:03:17.097 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:03:17.096+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:03:17.097497+0000) 2024-03-04T16:03:18.098 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:03:18.097+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:03:18.097783+0000) 2024-03-04T16:03:19.098 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:03:19.097+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:03:19.098059+0000) 2024-03-04T16:03:20.098 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:03:20.097+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:03:20.098306+0000) 2024-03-04T16:03:21.098 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:03:21.097+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:03:21.098508+0000) 2024-03-04T16:03:22.099 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:03:22.097+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:03:22.098778+0000) 2024-03-04T16:03:23.099 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:03:23.098+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:03:23.099038+0000) 2024-03-04T16:03:24.099 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:03:24.098+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:03:24.099275+0000) 2024-03-04T16:03:25.099 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:03:25.099+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:03:25.099514+0000) 2024-03-04T16:03:26.099 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:03:26.099+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:03:26.099760+0000) 2024-03-04T16:03:27.100 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:03:27.099+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:03:27.100003+0000) 2024-03-04T16:03:28.100 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:03:28.099+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:03:28.100264+0000) 2024-03-04T16:03:29.100 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:03:29.099+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:03:29.100521+0000) 2024-03-04T16:03:30.100 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:03:30.099+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:03:30.100754+0000) 2024-03-04T16:03:31.101 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:03:31.100+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:03:31.101006+0000) 2024-03-04T16:03:32.101 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:03:32.100+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:03:32.101260+0000) 2024-03-04T16:03:33.101 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:03:33.101+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:03:33.101507+0000) 2024-03-04T16:03:34.102 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:03:34.101+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:03:34.101745+0000) 2024-03-04T16:03:35.102 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:03:35.101+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:03:35.101983+0000) 2024-03-04T16:03:36.102 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:03:36.101+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:03:36.102221+0000) 2024-03-04T16:03:36.550 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:03:36.558 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:03:37.102 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:03:37.101+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:03:37.102482+0000) 2024-03-04T16:03:38.102 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:03:38.102+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:03:38.102767+0000) 2024-03-04T16:03:39.103 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:03:39.102+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:03:39.103028+0000) 2024-03-04T16:03:40.103 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:03:40.103+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:03:40.103273+0000) 2024-03-04T16:03:41.104 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:03:41.103+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:03:41.103521+0000) 2024-03-04T16:03:42.104 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:03:42.103+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:03:42.103769+0000) 2024-03-04T16:03:43.104 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:03:43.103+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:03:43.104027+0000) 2024-03-04T16:03:44.104 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:03:44.104+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:03:44.104262+0000) 2024-03-04T16:03:45.104 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:03:45.104+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:03:45.104526+0000) 2024-03-04T16:03:46.104 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:03:46.104+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:03:46.104767+0000) 2024-03-04T16:03:47.105 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:03:47.104+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:03:47.105004+0000) 2024-03-04T16:03:48.105 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:03:48.103+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:03:48.105206+0000) 2024-03-04T16:03:49.105 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:03:49.104+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:03:49.105462+0000) 2024-03-04T16:03:50.105 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:03:50.104+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:03:50.105700+0000) 2024-03-04T16:03:51.106 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:03:51.104+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:03:51.105949+0000) 2024-03-04T16:03:52.106 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:03:52.105+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:03:52.106186+0000) 2024-03-04T16:03:53.106 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:03:53.105+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:03:53.106428+0000) 2024-03-04T16:03:54.106 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:03:54.104+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:03:54.106686+0000) 2024-03-04T16:03:55.107 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:03:55.105+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:03:55.106930+0000) 2024-03-04T16:03:56.107 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:03:56.106+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:03:56.107122+0000) 2024-03-04T16:03:57.107 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:03:57.106+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:03:57.107360+0000) 2024-03-04T16:03:58.108 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:03:58.106+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:03:58.107612+0000) 2024-03-04T16:03:59.108 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:03:59.107+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:03:59.107895+0000) 2024-03-04T16:04:00.108 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:04:00.107+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:04:00.108132+0000) 2024-03-04T16:04:01.108 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:04:01.107+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:04:01.108330+0000) 2024-03-04T16:04:02.108 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:04:02.107+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:04:02.108585+0000) 2024-03-04T16:04:03.109 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:04:03.108+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:04:03.108867+0000) 2024-03-04T16:04:04.109 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:04:04.108+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:04:04.109114+0000) 2024-03-04T16:04:05.109 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:04:05.108+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:04:05.109354+0000) 2024-03-04T16:04:06.109 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:04:06.108+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:04:06.109649+0000) 2024-03-04T16:04:06.754 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:04:06.763 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:04:07.110 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:04:07.109+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:04:07.109978+0000) 2024-03-04T16:04:08.110 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:04:08.109+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:04:08.110265+0000) 2024-03-04T16:04:09.110 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:04:09.109+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:04:09.110566+0000) 2024-03-04T16:04:10.111 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:04:10.110+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:04:10.110894+0000) 2024-03-04T16:04:11.111 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:04:11.110+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:04:11.111260+0000) 2024-03-04T16:04:12.111 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:04:12.110+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:04:12.111574+0000) 2024-03-04T16:04:13.112 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:04:13.111+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:04:13.111917+0000) 2024-03-04T16:04:14.112 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:04:14.111+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:04:14.112166+0000) 2024-03-04T16:04:15.112 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:04:15.111+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:04:15.112394+0000) 2024-03-04T16:04:16.112 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:04:16.111+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:04:16.112678+0000) 2024-03-04T16:04:17.113 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:04:17.112+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:04:17.112980+0000) 2024-03-04T16:04:18.113 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:04:18.112+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:04:18.113231+0000) 2024-03-04T16:04:19.113 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:04:19.112+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:04:19.113516+0000) 2024-03-04T16:04:20.114 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:04:20.113+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:04:20.113803+0000) 2024-03-04T16:04:21.114 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:04:21.113+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:04:21.114176+0000) 2024-03-04T16:04:22.114 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:04:22.113+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:04:22.114444+0000) 2024-03-04T16:04:23.114 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:04:23.114+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:04:23.114685+0000) 2024-03-04T16:04:24.115 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:04:24.114+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:04:24.114931+0000) 2024-03-04T16:04:25.115 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:04:25.114+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:04:25.115135+0000) 2024-03-04T16:04:26.115 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:04:26.114+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:04:26.115385+0000) 2024-03-04T16:04:27.115 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:04:27.114+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:04:27.115642+0000) 2024-03-04T16:04:28.116 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:04:28.115+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:04:28.115932+0000) 2024-03-04T16:04:29.116 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:04:29.115+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:04:29.116223+0000) 2024-03-04T16:04:30.116 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:04:30.115+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:04:30.116464+0000) 2024-03-04T16:04:31.117 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:04:31.116+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:04:31.116778+0000) 2024-03-04T16:04:32.117 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:04:32.116+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:04:32.117122+0000) 2024-03-04T16:04:33.117 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:04:33.116+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:04:33.117378+0000) 2024-03-04T16:04:34.117 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:04:34.116+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:04:34.117659+0000) 2024-03-04T16:04:35.118 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:04:35.117+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:04:35.117988+0000) 2024-03-04T16:04:36.118 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:04:36.117+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:04:36.118235+0000) 2024-03-04T16:04:36.976 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:04:36.985 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:04:37.118 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:04:37.117+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:04:37.118446+0000) 2024-03-04T16:04:38.118 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:04:38.118+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:04:38.118704+0000) 2024-03-04T16:04:39.119 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:04:39.118+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:04:39.118947+0000) 2024-03-04T16:04:40.119 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:04:40.118+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:04:40.119189+0000) 2024-03-04T16:04:41.119 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:04:41.119+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:04:41.119449+0000) 2024-03-04T16:04:42.120 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:04:42.119+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:04:42.119733+0000) 2024-03-04T16:04:43.120 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:04:43.119+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:04:43.120036+0000) 2024-03-04T16:04:44.120 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:04:44.119+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:04:44.120259+0000) 2024-03-04T16:04:45.120 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:04:45.120+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:04:45.120583+0000) 2024-03-04T16:04:46.121 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:04:46.120+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:04:46.120954+0000) 2024-03-04T16:04:47.121 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:04:47.121+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:04:47.121246+0000) 2024-03-04T16:04:48.121 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:04:48.121+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:04:48.121543+0000) 2024-03-04T16:04:49.122 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:04:49.121+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:04:49.121870+0000) 2024-03-04T16:04:50.122 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:04:50.121+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:04:50.122122+0000) 2024-03-04T16:04:51.122 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:04:51.121+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:04:51.122383+0000) 2024-03-04T16:04:52.122 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:04:52.121+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:04:52.122761+0000) 2024-03-04T16:04:53.123 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:04:53.121+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:04:53.123058+0000) 2024-03-04T16:04:54.123 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:04:54.122+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:04:54.123351+0000) 2024-03-04T16:04:55.124 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:04:55.122+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:04:55.123653+0000) 2024-03-04T16:04:56.124 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:04:56.122+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:04:56.123971+0000) 2024-03-04T16:04:57.124 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:04:57.122+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:04:57.124258+0000) 2024-03-04T16:04:58.124 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:04:58.123+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:04:58.124559+0000) 2024-03-04T16:04:59.125 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:04:59.122+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:04:59.124791+0000) 2024-03-04T16:05:00.125 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:05:00.124+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:05:00.125112+0000) 2024-03-04T16:05:01.125 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:05:01.124+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:05:01.125413+0000) 2024-03-04T16:05:02.125 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:05:02.124+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:05:02.125703+0000) 2024-03-04T16:05:03.126 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:05:03.125+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:05:03.125994+0000) 2024-03-04T16:05:04.126 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:05:04.125+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:05:04.126289+0000) 2024-03-04T16:05:05.128 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:05:05.125+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:05:05.126579+0000) 2024-03-04T16:05:06.127 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:05:06.126+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:05:06.126912+0000) 2024-03-04T16:05:07.114 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:05:07.123 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:05:07.131 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:05:07.126+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:05:07.127133+0000) 2024-03-04T16:05:08.127 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:05:08.126+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:05:08.127406+0000) 2024-03-04T16:05:09.127 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:05:09.126+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:05:09.127637+0000) 2024-03-04T16:05:10.127 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:05:10.127+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:05:10.127914+0000) 2024-03-04T16:05:11.128 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:05:11.127+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:05:11.128166+0000) 2024-03-04T16:05:12.128 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:05:12.127+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:05:12.128344+0000) 2024-03-04T16:05:13.132 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:05:13.127+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:05:13.128608+0000) 2024-03-04T16:05:14.128 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:05:14.128+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:05:14.128867+0000) 2024-03-04T16:05:15.129 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:05:15.128+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:05:15.129063+0000) 2024-03-04T16:05:16.129 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:05:16.128+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:05:16.129297+0000) 2024-03-04T16:05:17.129 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:05:17.128+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:05:17.129570+0000) 2024-03-04T16:05:18.130 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:05:18.129+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:05:18.129854+0000) 2024-03-04T16:05:19.130 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:05:19.129+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:05:19.130104+0000) 2024-03-04T16:05:20.130 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:05:20.129+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:05:20.130350+0000) 2024-03-04T16:05:21.132 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:05:21.129+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:05:21.130635+0000) 2024-03-04T16:05:22.131 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:05:22.130+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:05:22.130968+0000) 2024-03-04T16:05:23.131 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:05:23.130+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:05:23.131264+0000) 2024-03-04T16:05:24.131 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:05:24.130+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:05:24.131561+0000) 2024-03-04T16:05:25.131 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:05:25.131+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:05:25.131904+0000) 2024-03-04T16:05:26.132 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:05:26.131+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:05:26.132129+0000) 2024-03-04T16:05:27.132 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:05:27.131+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:05:27.132423+0000) 2024-03-04T16:05:28.132 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:05:28.131+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:05:28.132723+0000) 2024-03-04T16:05:29.133 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:05:29.132+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:05:29.133002+0000) 2024-03-04T16:05:30.133 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:05:30.132+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:05:30.133302+0000) 2024-03-04T16:05:31.133 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:05:31.132+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:05:31.133594+0000) 2024-03-04T16:05:32.133 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:05:32.133+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:05:32.133918+0000) 2024-03-04T16:05:33.134 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:05:33.133+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:05:33.134217+0000) 2024-03-04T16:05:34.134 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:05:34.134+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:05:34.134496+0000) 2024-03-04T16:05:35.134 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:05:35.133+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:05:35.134777+0000) 2024-03-04T16:05:36.135 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:05:36.134+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:05:36.135111+0000) 2024-03-04T16:05:37.135 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:05:37.134+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:05:37.135404+0000) 2024-03-04T16:05:37.252 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:05:37.260 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:05:38.135 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:05:38.135+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:05:38.135715+0000) 2024-03-04T16:05:39.136 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:05:39.135+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:05:39.136029+0000) 2024-03-04T16:05:40.136 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:05:40.135+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:05:40.136411+0000) 2024-03-04T16:05:41.136 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:05:41.135+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:05:41.136737+0000) 2024-03-04T16:05:42.137 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:05:42.136+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:05:42.137051+0000) 2024-03-04T16:05:43.137 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:05:43.136+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:05:43.137349+0000) 2024-03-04T16:05:44.137 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:05:44.136+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:05:44.137656+0000) 2024-03-04T16:05:45.137 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:05:45.137+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:05:45.137975+0000) 2024-03-04T16:05:46.138 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:05:46.137+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:05:46.138270+0000) 2024-03-04T16:05:47.138 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:05:47.137+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:05:47.138572+0000) 2024-03-04T16:05:48.139 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:05:48.138+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:05:48.138917+0000) 2024-03-04T16:05:49.139 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:05:49.138+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:05:49.139202+0000) 2024-03-04T16:05:50.139 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:05:50.139+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:05:50.139515+0000) 2024-03-04T16:05:51.140 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:05:51.139+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:05:51.139805+0000) 2024-03-04T16:05:52.140 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:05:52.139+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:05:52.140082+0000) 2024-03-04T16:05:53.140 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:05:53.140+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:05:53.140371+0000) 2024-03-04T16:05:54.140 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:05:54.140+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:05:54.140669+0000) 2024-03-04T16:05:55.141 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:05:55.140+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:05:55.140989+0000) 2024-03-04T16:05:56.141 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:05:56.141+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:05:56.141279+0000) 2024-03-04T16:05:57.141 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:05:57.140+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:05:57.141571+0000) 2024-03-04T16:05:58.141 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:05:58.140+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:05:58.141881+0000) 2024-03-04T16:05:59.142 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:05:59.141+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:05:59.142174+0000) 2024-03-04T16:06:00.142 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:06:00.141+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:06:00.142454+0000) 2024-03-04T16:06:01.142 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:06:01.141+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:06:01.142748+0000) 2024-03-04T16:06:02.143 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:06:02.141+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:06:02.142976+0000) 2024-03-04T16:06:03.143 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:06:03.142+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:06:03.143291+0000) 2024-03-04T16:06:04.143 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:06:04.142+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:06:04.143605+0000) 2024-03-04T16:06:05.143 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:06:05.142+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:06:05.143932+0000) 2024-03-04T16:06:06.144 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:06:06.142+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:06:06.144211+0000) 2024-03-04T16:06:07.144 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:06:07.143+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:06:07.144502+0000) 2024-03-04T16:06:07.466 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:06:07.474 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:06:08.144 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:06:08.143+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:06:08.144790+0000) 2024-03-04T16:06:09.145 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:06:09.144+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:06:09.145129+0000) 2024-03-04T16:06:10.145 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:06:10.144+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:06:10.145411+0000) 2024-03-04T16:06:11.145 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:06:11.144+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:06:11.145716+0000) 2024-03-04T16:06:12.146 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:06:12.145+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:06:12.146004+0000) 2024-03-04T16:06:13.146 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:06:13.145+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:06:13.146289+0000) 2024-03-04T16:06:14.146 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:06:14.145+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:06:14.146577+0000) 2024-03-04T16:06:15.146 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:06:15.146+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:06:15.146929+0000) 2024-03-04T16:06:16.147 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:06:16.146+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:06:16.147211+0000) 2024-03-04T16:06:17.147 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:06:17.146+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:06:17.147490+0000) 2024-03-04T16:06:18.147 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:06:18.146+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:06:18.147776+0000) 2024-03-04T16:06:19.148 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:06:19.147+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:06:19.148068+0000) 2024-03-04T16:06:20.148 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:06:20.147+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:06:20.148348+0000) 2024-03-04T16:06:21.148 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:06:21.147+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:06:21.148659+0000) 2024-03-04T16:06:22.148 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:06:22.148+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:06:22.148957+0000) 2024-03-04T16:06:23.151 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:06:23.148+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:06:23.149199+0000) 2024-03-04T16:06:24.149 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:06:24.148+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:06:24.149484+0000) 2024-03-04T16:06:25.149 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:06:25.148+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:06:25.149771+0000) 2024-03-04T16:06:26.150 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:06:26.149+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:06:26.150070+0000) 2024-03-04T16:06:27.150 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:06:27.149+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:06:27.150357+0000) 2024-03-04T16:06:28.150 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:06:28.149+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:06:28.150648+0000) 2024-03-04T16:06:29.151 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:06:29.150+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:06:29.150928+0000) 2024-03-04T16:06:30.151 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:06:30.150+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:06:30.151133+0000) 2024-03-04T16:06:31.151 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:06:31.150+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:06:31.151334+0000) 2024-03-04T16:06:32.151 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:06:32.150+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:06:32.151598+0000) 2024-03-04T16:06:33.151 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:06:33.151+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:06:33.151877+0000) 2024-03-04T16:06:34.152 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:06:34.151+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:06:34.152122+0000) 2024-03-04T16:06:35.152 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:06:35.151+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:06:35.152394+0000) 2024-03-04T16:06:36.152 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:06:36.151+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:06:36.152629+0000) 2024-03-04T16:06:37.153 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:06:37.152+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:06:37.152944+0000) 2024-03-04T16:06:37.604 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:06:37.612 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:06:38.153 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:06:38.152+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:06:38.153185+0000) 2024-03-04T16:06:39.153 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:06:39.152+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:06:39.153432+0000) 2024-03-04T16:06:40.153 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:06:40.152+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:06:40.153657+0000) 2024-03-04T16:06:41.153 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:06:41.153+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:06:41.153926+0000) 2024-03-04T16:06:42.154 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:06:42.153+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:06:42.154156+0000) 2024-03-04T16:06:43.154 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:06:43.153+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:06:43.154398+0000) 2024-03-04T16:06:44.154 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:06:44.153+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:06:44.154629+0000) 2024-03-04T16:06:45.154 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:06:45.154+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:06:45.154914+0000) 2024-03-04T16:06:46.155 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:06:46.154+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:06:46.155141+0000) 2024-03-04T16:06:47.155 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:06:47.154+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:06:47.155367+0000) 2024-03-04T16:06:48.155 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:06:48.154+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:06:48.155602+0000) 2024-03-04T16:06:49.156 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:06:49.155+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:06:49.155951+0000) 2024-03-04T16:06:50.156 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:06:50.155+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:06:50.156170+0000) 2024-03-04T16:06:51.156 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:06:51.155+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:06:51.156410+0000) 2024-03-04T16:06:52.156 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:06:52.155+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:06:52.156642+0000) 2024-03-04T16:06:53.156 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:06:53.156+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:06:53.156963+0000) 2024-03-04T16:06:54.157 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:06:54.156+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:06:54.157184+0000) 2024-03-04T16:06:55.157 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:06:55.157+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:06:55.157481+0000) 2024-03-04T16:06:56.157 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:06:56.156+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:06:56.157753+0000) 2024-03-04T16:06:57.158 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:06:57.157+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:06:57.158051+0000) 2024-03-04T16:06:58.158 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:06:58.157+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:06:58.158279+0000) 2024-03-04T16:06:59.158 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:06:59.157+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:06:59.158589+0000) 2024-03-04T16:07:00.158 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:07:00.158+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:07:00.158902+0000) 2024-03-04T16:07:01.159 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:07:01.158+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:07:01.159226+0000) 2024-03-04T16:07:02.159 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:07:02.157+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:07:02.159466+0000) 2024-03-04T16:07:03.159 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:07:03.157+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:07:03.159756+0000) 2024-03-04T16:07:04.160 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:07:04.158+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:07:04.160017+0000) 2024-03-04T16:07:05.160 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:07:05.158+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:07:05.160322+0000) 2024-03-04T16:07:06.160 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:07:06.158+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:07:06.160602+0000) 2024-03-04T16:07:07.160 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:07:07.159+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:07:07.160817+0000) 2024-03-04T16:07:07.818 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:07:07.826 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:07:08.161 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:07:08.159+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:07:08.161059+0000) 2024-03-04T16:07:09.161 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:07:09.159+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:07:09.161349+0000) 2024-03-04T16:07:10.161 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:07:10.159+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:07:10.161696+0000) 2024-03-04T16:07:11.162 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:07:11.160+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:07:11.162003+0000) 2024-03-04T16:07:12.162 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:07:12.161+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:07:12.162209+0000) 2024-03-04T16:07:13.162 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:07:13.161+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:07:13.162448+0000) 2024-03-04T16:07:14.162 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:07:14.161+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:07:14.162683+0000) 2024-03-04T16:07:15.163 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:07:15.162+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:07:15.162979+0000) 2024-03-04T16:07:16.163 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:07:16.162+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:07:16.163180+0000) 2024-03-04T16:07:17.163 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:07:17.162+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:07:17.163383+0000) 2024-03-04T16:07:18.163 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:07:18.162+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:07:18.163602+0000) 2024-03-04T16:07:19.163 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:07:19.163+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:07:19.163856+0000) 2024-03-04T16:07:20.164 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:07:20.163+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:07:20.164103+0000) 2024-03-04T16:07:21.164 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:07:21.163+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:07:21.164337+0000) 2024-03-04T16:07:22.164 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:07:22.163+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:07:22.164525+0000) 2024-03-04T16:07:23.164 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:07:23.163+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:07:23.164706+0000) 2024-03-04T16:07:24.165 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:07:24.164+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:07:24.164981+0000) 2024-03-04T16:07:25.165 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:07:25.164+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:07:25.165214+0000) 2024-03-04T16:07:26.165 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:07:26.164+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:07:26.165486+0000) 2024-03-04T16:07:27.165 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:07:27.164+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:07:27.165716+0000) 2024-03-04T16:07:28.165 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:07:28.165+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:07:28.165963+0000) 2024-03-04T16:07:29.166 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:07:29.165+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:07:29.166190+0000) 2024-03-04T16:07:30.166 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:07:30.165+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:07:30.166416+0000) 2024-03-04T16:07:31.166 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:07:31.165+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:07:31.166681+0000) 2024-03-04T16:07:32.166 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:07:32.166+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:07:32.166946+0000) 2024-03-04T16:07:33.167 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:07:33.166+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:07:33.167187+0000) 2024-03-04T16:07:34.167 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:07:34.166+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:07:34.167423+0000) 2024-03-04T16:07:35.168 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:07:35.166+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:07:35.167729+0000) 2024-03-04T16:07:36.167 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:07:36.167+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:07:36.167964+0000) 2024-03-04T16:07:37.168 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:07:37.167+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:07:37.168216+0000) 2024-03-04T16:07:38.022 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:07:38.030 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:07:38.168 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:07:38.167+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:07:38.168474+0000) 2024-03-04T16:07:39.168 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:07:39.167+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:07:39.168713+0000) 2024-03-04T16:07:40.168 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:07:40.168+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:07:40.168959+0000) 2024-03-04T16:07:41.169 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:07:41.168+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:07:41.169218+0000) 2024-03-04T16:07:42.169 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:07:42.168+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:07:42.169450+0000) 2024-03-04T16:07:43.169 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:07:43.168+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:07:43.169674+0000) 2024-03-04T16:07:44.169 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:07:44.169+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:07:44.169917+0000) 2024-03-04T16:07:45.170 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:07:45.169+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:07:45.170144+0000) 2024-03-04T16:07:46.170 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:07:46.169+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:07:46.170501+0000) 2024-03-04T16:07:47.170 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:07:47.169+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:07:47.170733+0000) 2024-03-04T16:07:48.170 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:07:48.170+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:07:48.170964+0000) 2024-03-04T16:07:49.171 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:07:49.170+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:07:49.171222+0000) 2024-03-04T16:07:50.171 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:07:50.170+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:07:50.171448+0000) 2024-03-04T16:07:51.171 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:07:51.170+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:07:51.171661+0000) 2024-03-04T16:07:52.171 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:07:52.171+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:07:52.171916+0000) 2024-03-04T16:07:53.172 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:07:53.171+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:07:53.172126+0000) 2024-03-04T16:07:54.172 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:07:54.171+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:07:54.172324+0000) 2024-03-04T16:07:55.172 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:07:55.171+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:07:55.172505+0000) 2024-03-04T16:07:56.172 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:07:56.171+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:07:56.172728+0000) 2024-03-04T16:07:57.172 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:07:57.172+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:07:57.172924+0000) 2024-03-04T16:07:58.173 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:07:58.172+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:07:58.173159+0000) 2024-03-04T16:07:59.173 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:07:59.172+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:07:59.173373+0000) 2024-03-04T16:08:00.173 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:08:00.172+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:08:00.173555+0000) 2024-03-04T16:08:01.173 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:08:01.172+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:08:01.173748+0000) 2024-03-04T16:08:02.174 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:08:02.173+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:08:02.174035+0000) 2024-03-04T16:08:03.177 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:08:03.173+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:08:03.174222+0000) 2024-03-04T16:08:04.174 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:08:04.173+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:08:04.174414+0000) 2024-03-04T16:08:05.174 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:08:05.173+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:08:05.174649+0000) 2024-03-04T16:08:06.174 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:08:06.173+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:08:06.174921+0000) 2024-03-04T16:08:07.175 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:08:07.173+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:08:07.175102+0000) 2024-03-04T16:08:08.151 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:08:08.163 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:08:08.174 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:08:08.173+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:08:08.175284+0000) 2024-03-04T16:08:09.175 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:08:09.173+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:08:09.175473+0000) 2024-03-04T16:08:10.175 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:08:10.173+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:08:10.175657+0000) 2024-03-04T16:08:11.175 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:08:11.174+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:08:11.175929+0000) 2024-03-04T16:08:12.176 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:08:12.174+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:08:12.176113+0000) 2024-03-04T16:08:13.176 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:08:13.174+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:08:13.176251+0000) 2024-03-04T16:08:14.176 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:08:14.174+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:08:14.176420+0000) 2024-03-04T16:08:15.176 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:08:15.174+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:08:15.176643+0000) 2024-03-04T16:08:16.176 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:08:16.175+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:08:16.176915+0000) 2024-03-04T16:08:17.177 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:08:17.176+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:08:17.177107+0000) 2024-03-04T16:08:18.177 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:08:18.176+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:08:18.177290+0000) 2024-03-04T16:08:19.177 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:08:19.176+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:08:19.177475+0000) 2024-03-04T16:08:20.177 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:08:20.176+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:08:20.177758+0000) 2024-03-04T16:08:21.177 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:08:21.177+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:08:21.177933+0000) 2024-03-04T16:08:22.178 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:08:22.177+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:08:22.178125+0000) 2024-03-04T16:08:23.178 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:08:23.177+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:08:23.178314+0000) 2024-03-04T16:08:24.178 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:08:24.177+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:08:24.178499+0000) 2024-03-04T16:08:25.178 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:08:25.177+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:08:25.178740+0000) 2024-03-04T16:08:26.178 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:08:26.178+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:08:26.178990+0000) 2024-03-04T16:08:27.179 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:08:27.178+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:08:27.179192+0000) 2024-03-04T16:08:28.179 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:08:28.178+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:08:28.179380+0000) 2024-03-04T16:08:29.179 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:08:29.178+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:08:29.179615+0000) 2024-03-04T16:08:30.179 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:08:30.179+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:08:30.179929+0000) 2024-03-04T16:08:31.180 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:08:31.179+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:08:31.180248+0000) 2024-03-04T16:08:32.180 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:08:32.179+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:08:32.180500+0000) 2024-03-04T16:08:33.180 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:08:33.179+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:08:33.180793+0000) 2024-03-04T16:08:34.180 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:08:34.180+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:08:34.181088+0000) 2024-03-04T16:08:35.181 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:08:35.180+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:08:35.181408+0000) 2024-03-04T16:08:36.181 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:08:36.181+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:08:36.181721+0000) 2024-03-04T16:08:37.182 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:08:37.181+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:08:37.182016+0000) 2024-03-04T16:08:38.184 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:08:38.181+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:08:38.182269+0000) 2024-03-04T16:08:38.315 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:08:38.327 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:08:39.182 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:08:39.181+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:08:39.182521+0000) 2024-03-04T16:08:40.182 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:08:40.181+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:08:40.182803+0000) 2024-03-04T16:08:41.182 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:08:41.182+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:08:41.183107+0000) 2024-03-04T16:08:42.183 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:08:42.182+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:08:42.183407+0000) 2024-03-04T16:08:43.183 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:08:43.182+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:08:43.183664+0000) 2024-03-04T16:08:44.183 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:08:44.183+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:08:44.183976+0000) 2024-03-04T16:08:45.184 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:08:45.183+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:08:45.184228+0000) 2024-03-04T16:08:46.184 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:08:46.183+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:08:46.184435+0000) 2024-03-04T16:08:47.184 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:08:47.184+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:08:47.184673+0000) 2024-03-04T16:08:48.184 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:08:48.184+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:08:48.184919+0000) 2024-03-04T16:08:49.185 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:08:49.184+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:08:49.185114+0000) 2024-03-04T16:08:50.185 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:08:50.184+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:08:50.185356+0000) 2024-03-04T16:08:51.185 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:08:51.184+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:08:51.185619+0000) 2024-03-04T16:08:52.185 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:08:52.185+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:08:52.185905+0000) 2024-03-04T16:08:53.186 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:08:53.185+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:08:53.186143+0000) 2024-03-04T16:08:54.186 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:08:54.185+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:08:54.186355+0000) 2024-03-04T16:08:55.186 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:08:55.185+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:08:55.186612+0000) 2024-03-04T16:08:56.186 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:08:56.186+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:08:56.186904+0000) 2024-03-04T16:08:57.187 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:08:57.186+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:08:57.187101+0000) 2024-03-04T16:08:58.187 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:08:58.186+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:08:58.187298+0000) 2024-03-04T16:08:59.187 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:08:59.186+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:08:59.187496+0000) 2024-03-04T16:09:00.187 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:09:00.186+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:09:00.187769+0000) 2024-03-04T16:09:01.187 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:09:01.187+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:09:01.188018+0000) 2024-03-04T16:09:02.188 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:09:02.188+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:09:02.188309+0000) 2024-03-04T16:09:03.188 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:09:03.187+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:09:03.188542+0000) 2024-03-04T16:09:04.188 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:09:04.187+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:09:04.188792+0000) 2024-03-04T16:09:05.188 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:09:05.188+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:09:05.189047+0000) 2024-03-04T16:09:06.189 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:09:06.188+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:09:06.189276+0000) 2024-03-04T16:09:07.189 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:09:07.188+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:09:07.189505+0000) 2024-03-04T16:09:08.189 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:09:08.188+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:09:08.189699+0000) 2024-03-04T16:09:08.528 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:09:08.536 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:09:09.189 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:09:09.189+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:09:09.189961+0000) 2024-03-04T16:09:10.190 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:09:10.188+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:09:10.190191+0000) 2024-03-04T16:09:11.190 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:09:11.188+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:09:11.190450+0000) 2024-03-04T16:09:12.190 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:09:12.188+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:09:12.190705+0000) 2024-03-04T16:09:13.190 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:09:13.189+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:09:13.190945+0000) 2024-03-04T16:09:14.191 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:09:14.189+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:09:14.191172+0000) 2024-03-04T16:09:15.191 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:09:15.189+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:09:15.191364+0000) 2024-03-04T16:09:16.191 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:09:16.189+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:09:16.191625+0000) 2024-03-04T16:09:17.191 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:09:17.190+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:09:17.191927+0000) 2024-03-04T16:09:18.192 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:09:18.190+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:09:18.192126+0000) 2024-03-04T16:09:19.192 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:09:19.191+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:09:19.192326+0000) 2024-03-04T16:09:20.192 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:09:20.191+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:09:20.192596+0000) 2024-03-04T16:09:21.192 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:09:21.192+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:09:21.192887+0000) 2024-03-04T16:09:22.193 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:09:22.192+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:09:22.193126+0000) 2024-03-04T16:09:23.193 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:09:23.192+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:09:23.193321+0000) 2024-03-04T16:09:24.193 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:09:24.192+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:09:24.193557+0000) 2024-03-04T16:09:25.193 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:09:25.192+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:09:25.193738+0000) 2024-03-04T16:09:26.193 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:09:26.193+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:09:26.193973+0000) 2024-03-04T16:09:27.194 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:09:27.193+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:09:27.194215+0000) 2024-03-04T16:09:28.194 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:09:28.193+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:09:28.194459+0000) 2024-03-04T16:09:29.194 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:09:29.193+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:09:29.194708+0000) 2024-03-04T16:09:30.196 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:09:30.194+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:09:30.194942+0000) 2024-03-04T16:09:31.195 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:09:31.194+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:09:31.195195+0000) 2024-03-04T16:09:32.195 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:09:32.194+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:09:32.195437+0000) 2024-03-04T16:09:33.195 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:09:33.194+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:09:33.195678+0000) 2024-03-04T16:09:34.195 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:09:34.195+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:09:34.195951+0000) 2024-03-04T16:09:35.196 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:09:35.195+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:09:35.196134+0000) 2024-03-04T16:09:36.196 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:09:36.195+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:09:36.196391+0000) 2024-03-04T16:09:37.196 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:09:37.195+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:09:37.196584+0000) 2024-03-04T16:09:38.196 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:09:38.196+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:09:38.196878+0000) 2024-03-04T16:09:38.666 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:09:38.674 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:09:39.201 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:09:39.196+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:09:39.197119+0000) 2024-03-04T16:09:40.197 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:09:40.196+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:09:40.197362+0000) 2024-03-04T16:09:41.197 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:09:41.196+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:09:41.197606+0000) 2024-03-04T16:09:42.197 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:09:42.197+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:09:42.197890+0000) 2024-03-04T16:09:43.197 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:09:43.197+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:09:43.198136+0000) 2024-03-04T16:09:44.198 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:09:44.197+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:09:44.198371+0000) 2024-03-04T16:09:45.198 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:09:45.198+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:09:45.198615+0000) 2024-03-04T16:09:46.198 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:09:46.198+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:09:46.198886+0000) 2024-03-04T16:09:47.199 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:09:47.198+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:09:47.199127+0000) 2024-03-04T16:09:48.199 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:09:48.198+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:09:48.199381+0000) 2024-03-04T16:09:49.199 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:09:49.199+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:09:49.199632+0000) 2024-03-04T16:09:50.199 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:09:50.199+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:09:50.199966+0000) 2024-03-04T16:09:51.200 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:09:51.199+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:09:51.200217+0000) 2024-03-04T16:09:52.200 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:09:52.199+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:09:52.200452+0000) 2024-03-04T16:09:53.200 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:09:53.199+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:09:53.200684+0000) 2024-03-04T16:09:54.200 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:09:54.200+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:09:54.200922+0000) 2024-03-04T16:09:55.201 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:09:55.200+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:09:55.201159+0000) 2024-03-04T16:09:56.201 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:09:56.201+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:09:56.201401+0000) 2024-03-04T16:09:57.201 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:09:57.201+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:09:57.201634+0000) 2024-03-04T16:09:58.201 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:09:58.201+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:09:58.201916+0000) 2024-03-04T16:09:59.202 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:09:59.201+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:09:59.202153+0000) 2024-03-04T16:10:00.202 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:10:00.202+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:10:00.202392+0000) 2024-03-04T16:10:01.202 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:10:01.201+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:10:01.202653+0000) 2024-03-04T16:10:02.202 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:10:02.202+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:10:02.202953+0000) 2024-03-04T16:10:03.203 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:10:03.202+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:10:03.203195+0000) 2024-03-04T16:10:04.203 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:10:04.203+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:10:04.203432+0000) 2024-03-04T16:10:05.203 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:10:05.203+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:10:05.203674+0000) 2024-03-04T16:10:06.203 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:10:06.203+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:10:06.203934+0000) 2024-03-04T16:10:07.204 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:10:07.203+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:10:07.204189+0000) 2024-03-04T16:10:08.204 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:10:08.204+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:10:08.204466+0000) 2024-03-04T16:10:08.888 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:10:08.896 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:10:09.204 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:10:09.203+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:10:09.204699+0000) 2024-03-04T16:10:10.204 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:10:10.204+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:10:10.204977+0000) 2024-03-04T16:10:11.205 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:10:11.204+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:10:11.205171+0000) 2024-03-04T16:10:12.205 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:10:12.204+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:10:12.205356+0000) 2024-03-04T16:10:13.205 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:10:13.205+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:10:13.205636+0000) 2024-03-04T16:10:14.205 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:10:14.204+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:10:14.205911+0000) 2024-03-04T16:10:15.206 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:10:15.205+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:10:15.206149+0000) 2024-03-04T16:10:16.206 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:10:16.204+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:10:16.206381+0000) 2024-03-04T16:10:17.206 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:10:17.204+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:10:17.206676+0000) 2024-03-04T16:10:18.206 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:10:18.205+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:10:18.206968+0000) 2024-03-04T16:10:19.206 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:10:19.205+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:10:19.207197+0000) 2024-03-04T16:10:20.207 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:10:20.206+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:10:20.207473+0000) 2024-03-04T16:10:21.207 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:10:21.205+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:10:21.207669+0000) 2024-03-04T16:10:22.207 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:10:22.206+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:10:22.207950+0000) 2024-03-04T16:10:23.208 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:10:23.206+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:10:23.208182+0000) 2024-03-04T16:10:24.208 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:10:24.207+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:10:24.208418+0000) 2024-03-04T16:10:25.208 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:10:25.207+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:10:25.208656+0000) 2024-03-04T16:10:26.208 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:10:26.208+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:10:26.208958+0000) 2024-03-04T16:10:27.209 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:10:27.208+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:10:27.209183+0000) 2024-03-04T16:10:28.209 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:10:28.208+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:10:28.209432+0000) 2024-03-04T16:10:29.209 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:10:29.208+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:10:29.209676+0000) 2024-03-04T16:10:30.209 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:10:30.209+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:10:30.209967+0000) 2024-03-04T16:10:31.210 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:10:31.209+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:10:31.210166+0000) 2024-03-04T16:10:32.210 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:10:32.209+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:10:32.210424+0000) 2024-03-04T16:10:33.210 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:10:33.209+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:10:33.210668+0000) 2024-03-04T16:10:34.210 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:10:34.210+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:10:34.210953+0000) 2024-03-04T16:10:35.211 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:10:35.210+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:10:35.211193+0000) 2024-03-04T16:10:36.211 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:10:36.210+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:10:36.211452+0000) 2024-03-04T16:10:37.211 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:10:37.210+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:10:37.211687+0000) 2024-03-04T16:10:38.211 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:10:38.211+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:10:38.211996+0000) 2024-03-04T16:10:39.094 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:10:39.102 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:10:39.212 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:10:39.211+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:10:39.212228+0000) 2024-03-04T16:10:40.212 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:10:40.211+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:10:40.212420+0000) 2024-03-04T16:10:41.212 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:10:41.211+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:10:41.212670+0000) 2024-03-04T16:10:42.212 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:10:42.212+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:10:42.212952+0000) 2024-03-04T16:10:43.213 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:10:43.212+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:10:43.213202+0000) 2024-03-04T16:10:44.213 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:10:44.212+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:10:44.213434+0000) 2024-03-04T16:10:45.213 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:10:45.213+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:10:45.213672+0000) 2024-03-04T16:10:46.213 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:10:46.213+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:10:46.213951+0000) 2024-03-04T16:10:47.214 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:10:47.213+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:10:47.214221+0000) 2024-03-04T16:10:48.214 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:10:48.213+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:10:48.214502+0000) 2024-03-04T16:10:49.214 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:10:49.214+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:10:49.214809+0000) 2024-03-04T16:10:50.214 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:10:50.214+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:10:50.215091+0000) 2024-03-04T16:10:51.215 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:10:51.214+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:10:51.215359+0000) 2024-03-04T16:10:52.215 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:10:52.215+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:10:52.215598+0000) 2024-03-04T16:10:53.215 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:10:53.215+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:10:53.215904+0000) 2024-03-04T16:10:54.216 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:10:54.215+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:10:54.216143+0000) 2024-03-04T16:10:55.216 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:10:55.215+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:10:55.216404+0000) 2024-03-04T16:10:56.216 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:10:56.216+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:10:56.216668+0000) 2024-03-04T16:10:57.216 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:10:57.216+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:10:57.216949+0000) 2024-03-04T16:10:58.217 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:10:58.216+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:10:58.217185+0000) 2024-03-04T16:10:59.217 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:10:59.216+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:10:59.217425+0000) 2024-03-04T16:11:00.217 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:11:00.217+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:11:00.217673+0000) 2024-03-04T16:11:01.217 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:11:01.217+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:11:01.217958+0000) 2024-03-04T16:11:02.218 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:11:02.217+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:11:02.218243+0000) 2024-03-04T16:11:03.218 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:11:03.218+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:11:03.218496+0000) 2024-03-04T16:11:04.218 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:11:04.217+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:11:04.218732+0000) 2024-03-04T16:11:05.218 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:11:05.218+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:11:05.218992+0000) 2024-03-04T16:11:06.219 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:11:06.218+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:11:06.219250+0000) 2024-03-04T16:11:07.219 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:11:07.219+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:11:07.219487+0000) 2024-03-04T16:11:08.219 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:11:08.219+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:11:08.219761+0000) 2024-03-04T16:11:09.220 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:11:09.219+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:11:09.220056+0000) 2024-03-04T16:11:09.289 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:11:09.297 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:11:10.220 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:11:10.219+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:11:10.220345+0000) 2024-03-04T16:11:11.220 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:11:11.220+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:11:11.220584+0000) 2024-03-04T16:11:12.220 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:11:12.220+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:11:12.220852+0000) 2024-03-04T16:11:13.220 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:11:13.220+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:11:13.221096+0000) 2024-03-04T16:11:14.221 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:11:14.221+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:11:14.221339+0000) 2024-03-04T16:11:15.221 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:11:15.221+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:11:15.221580+0000) 2024-03-04T16:11:16.221 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:11:16.221+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:11:16.221818+0000) 2024-03-04T16:11:17.222 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:11:17.221+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:11:17.222090+0000) 2024-03-04T16:11:18.222 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:11:18.222+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:11:18.222335+0000) 2024-03-04T16:11:19.222 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:11:19.221+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:11:19.222571+0000) 2024-03-04T16:11:20.222 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:11:20.221+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:11:20.222807+0000) 2024-03-04T16:11:21.222 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:11:21.222+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:11:21.223091+0000) 2024-03-04T16:11:22.223 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:11:22.221+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:11:22.223343+0000) 2024-03-04T16:11:23.223 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:11:23.221+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:11:23.223537+0000) 2024-03-04T16:11:24.224 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:11:24.222+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:11:24.223774+0000) 2024-03-04T16:11:25.223 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:11:25.222+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:11:25.224021+0000) 2024-03-04T16:11:26.224 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:11:26.222+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:11:26.224255+0000) 2024-03-04T16:11:27.224 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:11:27.223+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:11:27.224481+0000) 2024-03-04T16:11:28.224 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:11:28.222+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:11:28.224736+0000) 2024-03-04T16:11:29.228 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:11:29.224+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:11:29.224975+0000) 2024-03-04T16:11:30.225 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:11:30.224+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:11:30.225217+0000) 2024-03-04T16:11:31.225 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:11:31.224+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:11:31.225465+0000) 2024-03-04T16:11:32.225 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:11:32.224+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:11:32.225702+0000) 2024-03-04T16:11:33.225 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:11:33.225+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:11:33.225979+0000) 2024-03-04T16:11:34.226 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:11:34.225+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:11:34.226235+0000) 2024-03-04T16:11:35.226 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:11:35.225+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:11:35.226494+0000) 2024-03-04T16:11:36.226 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:11:36.225+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:11:36.226680+0000) 2024-03-04T16:11:37.226 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:11:37.226+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:11:37.226963+0000) 2024-03-04T16:11:38.226 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:11:38.226+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:11:38.227178+0000) 2024-03-04T16:11:39.227 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:11:39.226+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:11:39.227372+0000) 2024-03-04T16:11:39.444 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:11:39.452 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:11:40.227 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:11:40.226+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:11:40.227613+0000) 2024-03-04T16:11:41.227 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:11:41.227+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:11:41.227889+0000) 2024-03-04T16:11:42.227 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:11:42.227+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:11:42.228130+0000) 2024-03-04T16:11:43.228 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:11:43.227+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:11:43.228369+0000) 2024-03-04T16:11:44.228 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:11:44.227+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:11:44.228624+0000) 2024-03-04T16:11:45.228 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:11:45.228+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:11:45.228931+0000) 2024-03-04T16:11:46.229 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:11:46.228+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:11:46.229174+0000) 2024-03-04T16:11:47.229 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:11:47.228+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:11:47.229368+0000) 2024-03-04T16:11:48.229 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:11:48.228+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:11:48.229634+0000) 2024-03-04T16:11:49.229 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:11:49.229+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:11:49.229927+0000) 2024-03-04T16:11:50.229 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:11:50.229+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:11:50.230119+0000) 2024-03-04T16:11:51.230 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:11:51.229+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:11:51.230370+0000) 2024-03-04T16:11:52.230 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:11:52.229+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:11:52.230604+0000) 2024-03-04T16:11:53.230 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:11:53.230+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:11:53.230906+0000) 2024-03-04T16:11:54.230 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:11:54.230+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:11:54.231145+0000) 2024-03-04T16:11:55.231 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:11:55.230+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:11:55.231392+0000) 2024-03-04T16:11:56.231 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:11:56.231+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:11:56.231629+0000) 2024-03-04T16:11:57.231 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:11:57.231+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:11:57.231940+0000) 2024-03-04T16:11:58.232 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:11:58.231+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:11:58.232224+0000) 2024-03-04T16:11:59.232 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:11:59.231+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:11:59.232517+0000) 2024-03-04T16:12:00.232 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:12:00.232+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:12:00.232814+0000) 2024-03-04T16:12:01.232 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:12:01.232+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:12:01.233175+0000) 2024-03-04T16:12:02.236 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:12:02.232+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:12:02.233471+0000) 2024-03-04T16:12:03.233 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:12:03.233+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:12:03.233766+0000) 2024-03-04T16:12:04.233 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:12:04.233+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:12:04.234059+0000) 2024-03-04T16:12:05.234 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:12:05.233+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:12:05.234280+0000) 2024-03-04T16:12:06.234 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:12:06.233+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:12:06.234593+0000) 2024-03-04T16:12:07.234 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:12:07.234+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:12:07.234923+0000) 2024-03-04T16:12:08.234 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:12:08.234+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:12:08.235172+0000) 2024-03-04T16:12:09.235 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:12:09.235+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:12:09.235455+0000) 2024-03-04T16:12:09.676 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:12:09.684 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:12:10.235 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:12:10.235+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:12:10.235755+0000) 2024-03-04T16:12:11.235 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:12:11.235+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:12:11.236054+0000) 2024-03-04T16:12:12.236 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:12:12.236+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:12:12.236359+0000) 2024-03-04T16:12:13.236 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:12:13.236+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:12:13.236655+0000) 2024-03-04T16:12:14.236 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:12:14.236+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:12:14.236965+0000) 2024-03-04T16:12:15.237 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:12:15.236+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:12:15.237252+0000) 2024-03-04T16:12:16.237 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:12:16.237+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:12:16.237536+0000) 2024-03-04T16:12:17.238 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:12:17.237+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:12:17.237785+0000) 2024-03-04T16:12:18.237 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:12:18.237+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:12:18.238036+0000) 2024-03-04T16:12:19.238 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:12:19.237+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:12:19.238235+0000) 2024-03-04T16:12:20.238 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:12:20.237+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:12:20.238538+0000) 2024-03-04T16:12:21.239 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:12:21.238+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:12:21.238806+0000) 2024-03-04T16:12:22.238 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:12:22.238+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:12:22.239051+0000) 2024-03-04T16:12:23.239 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:12:23.238+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:12:23.239248+0000) 2024-03-04T16:12:24.239 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:12:24.238+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:12:24.239491+0000) 2024-03-04T16:12:25.239 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:12:25.238+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:12:25.239731+0000) 2024-03-04T16:12:26.239 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:12:26.238+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:12:26.239980+0000) 2024-03-04T16:12:27.239 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:12:27.238+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:12:27.240178+0000) 2024-03-04T16:12:28.240 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:12:28.239+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:12:28.240411+0000) 2024-03-04T16:12:29.240 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:12:29.239+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:12:29.240652+0000) 2024-03-04T16:12:30.240 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:12:30.239+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:12:30.240876+0000) 2024-03-04T16:12:31.240 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:12:31.239+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:12:31.241158+0000) 2024-03-04T16:12:32.241 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:12:32.239+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:12:32.241445+0000) 2024-03-04T16:12:33.241 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:12:33.240+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:12:33.241756+0000) 2024-03-04T16:12:34.241 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:12:34.241+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:12:34.242039+0000) 2024-03-04T16:12:35.242 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:12:35.241+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:12:35.242311+0000) 2024-03-04T16:12:36.242 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:12:36.241+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:12:36.242623+0000) 2024-03-04T16:12:37.242 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:12:37.242+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:12:37.242906+0000) 2024-03-04T16:12:38.243 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:12:38.242+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:12:38.243191+0000) 2024-03-04T16:12:39.243 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:12:39.242+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:12:39.243487+0000) 2024-03-04T16:12:39.845 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:12:39.853 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:12:40.243 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:12:40.242+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:12:40.243807+0000) 2024-03-04T16:12:41.244 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:12:41.243+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:12:41.244188+0000) 2024-03-04T16:12:42.244 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:12:42.243+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:12:42.244480+0000) 2024-03-04T16:12:43.244 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:12:43.243+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:12:43.244773+0000) 2024-03-04T16:12:44.244 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:12:44.244+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:12:44.245087+0000) 2024-03-04T16:12:45.245 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:12:45.244+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:12:45.245342+0000) 2024-03-04T16:12:46.245 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:12:46.244+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:12:46.245641+0000) 2024-03-04T16:12:47.245 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:12:47.245+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:12:47.245967+0000) 2024-03-04T16:12:48.246 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:12:48.245+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:12:48.246250+0000) 2024-03-04T16:12:49.246 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:12:49.245+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:12:49.246493+0000) 2024-03-04T16:12:50.246 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:12:50.246+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:12:50.246855+0000) 2024-03-04T16:12:51.247 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:12:51.246+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:12:51.247196+0000) 2024-03-04T16:12:52.247 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:12:52.246+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:12:52.247512+0000) 2024-03-04T16:12:53.247 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:12:53.246+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:12:53.247798+0000) 2024-03-04T16:12:54.247 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:12:54.247+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:12:54.248142+0000) 2024-03-04T16:12:55.248 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:12:55.247+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:12:55.248443+0000) 2024-03-04T16:12:56.248 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:12:56.248+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:12:56.248743+0000) 2024-03-04T16:12:57.248 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:12:57.248+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:12:57.249022+0000) 2024-03-04T16:12:58.249 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:12:58.248+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:12:58.249336+0000) 2024-03-04T16:12:59.249 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:12:59.249+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:12:59.249653+0000) 2024-03-04T16:13:00.249 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:13:00.249+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:13:00.249986+0000) 2024-03-04T16:13:01.250 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:13:01.249+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:13:01.250298+0000) 2024-03-04T16:13:02.250 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:13:02.249+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:13:02.250528+0000) 2024-03-04T16:13:03.250 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:13:03.250+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:13:03.250875+0000) 2024-03-04T16:13:04.250 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:13:04.250+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:13:04.251170+0000) 2024-03-04T16:13:05.251 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:13:05.250+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:13:05.251464+0000) 2024-03-04T16:13:06.251 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:13:06.251+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:13:06.251755+0000) 2024-03-04T16:13:07.251 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:13:07.251+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:13:07.251975+0000) 2024-03-04T16:13:08.251 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:13:08.251+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:13:08.252275+0000) 2024-03-04T16:13:09.252 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:13:09.252+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:13:09.252568+0000) 2024-03-04T16:13:10.076 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:13:10.084 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:13:10.252 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:13:10.252+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:13:10.252917+0000) 2024-03-04T16:13:11.252 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:13:11.252+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:13:11.253184+0000) 2024-03-04T16:13:12.253 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:13:12.253+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:13:12.253440+0000) 2024-03-04T16:13:13.253 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:13:13.252+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:13:13.253740+0000) 2024-03-04T16:13:14.253 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:13:14.253+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:13:14.253996+0000) 2024-03-04T16:13:15.254 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:13:15.253+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:13:15.254251+0000) 2024-03-04T16:13:16.254 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:13:16.254+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:13:16.254504+0000) 2024-03-04T16:13:17.254 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:13:17.254+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:13:17.254708+0000) 2024-03-04T16:13:18.254 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:13:18.254+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:13:18.254957+0000) 2024-03-04T16:13:19.255 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:13:19.254+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:13:19.255186+0000) 2024-03-04T16:13:20.255 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:13:20.254+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:13:20.255422+0000) 2024-03-04T16:13:21.255 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:13:21.255+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:13:21.255675+0000) 2024-03-04T16:13:22.255 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:13:22.255+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:13:22.255924+0000) 2024-03-04T16:13:23.255 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:13:23.255+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:13:23.256119+0000) 2024-03-04T16:13:24.256 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:13:24.256+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:13:24.256356+0000) 2024-03-04T16:13:25.256 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:13:25.256+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:13:25.256597+0000) 2024-03-04T16:13:26.256 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:13:26.256+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:13:26.256862+0000) 2024-03-04T16:13:27.256 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:13:27.256+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:13:27.257119+0000) 2024-03-04T16:13:28.257 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:13:28.256+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:13:28.257363+0000) 2024-03-04T16:13:29.257 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:13:29.256+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:13:29.257620+0000) 2024-03-04T16:13:30.257 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:13:30.256+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:13:30.257915+0000) 2024-03-04T16:13:31.257 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:13:31.257+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:13:31.258193+0000) 2024-03-04T16:13:32.258 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:13:32.257+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:13:32.258374+0000) 2024-03-04T16:13:33.258 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:13:33.257+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:13:33.258630+0000) 2024-03-04T16:13:34.258 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:13:34.257+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:13:34.258933+0000) 2024-03-04T16:13:35.258 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:13:35.258+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:13:35.259170+0000) 2024-03-04T16:13:36.259 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:13:36.258+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:13:36.259406+0000) 2024-03-04T16:13:37.259 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:13:37.258+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:13:37.259651+0000) 2024-03-04T16:13:38.259 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:13:38.258+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:13:38.260003+0000) 2024-03-04T16:13:39.259 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:13:39.259+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:13:39.260249+0000) 2024-03-04T16:13:40.260 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:13:40.259+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:13:40.260491+0000) 2024-03-04T16:13:40.298 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:13:40.306 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:13:41.260 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:13:41.259+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:13:41.260749+0000) 2024-03-04T16:13:42.260 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:13:42.260+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:13:42.260983+0000) 2024-03-04T16:13:43.260 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:13:43.260+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:13:43.261228+0000) 2024-03-04T16:13:44.261 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:13:44.260+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:13:44.261460+0000) 2024-03-04T16:13:45.261 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:13:45.260+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:13:45.261718+0000) 2024-03-04T16:13:46.261 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:13:46.261+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:13:46.261953+0000) 2024-03-04T16:13:47.261 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:13:47.261+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:13:47.262195+0000) 2024-03-04T16:13:48.262 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:13:48.261+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:13:48.262440+0000) 2024-03-04T16:13:49.262 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:13:49.261+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:13:49.262671+0000) 2024-03-04T16:13:50.262 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:13:50.262+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:13:50.262948+0000) 2024-03-04T16:13:51.263 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:13:51.262+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:13:51.263203+0000) 2024-03-04T16:13:52.263 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:13:52.262+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:13:52.263439+0000) 2024-03-04T16:13:53.263 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:13:53.262+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:13:53.263682+0000) 2024-03-04T16:13:54.263 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:13:54.263+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:13:54.263921+0000) 2024-03-04T16:13:55.263 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:13:55.263+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:13:55.264162+0000) 2024-03-04T16:13:56.264 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:13:56.263+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:13:56.264402+0000) 2024-03-04T16:13:57.264 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:13:57.263+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:13:57.264657+0000) 2024-03-04T16:13:58.264 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:13:58.264+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:13:58.264947+0000) 2024-03-04T16:13:59.264 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:13:59.264+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:13:59.265134+0000) 2024-03-04T16:14:00.265 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:14:00.264+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:14:00.265374+0000) 2024-03-04T16:14:01.265 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:14:01.265+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:14:01.265627+0000) 2024-03-04T16:14:02.265 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:14:02.265+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:14:02.265919+0000) 2024-03-04T16:14:03.265 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:14:03.265+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:14:03.266160+0000) 2024-03-04T16:14:04.266 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:14:04.265+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:14:04.266394+0000) 2024-03-04T16:14:05.266 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:14:05.266+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:14:05.266647+0000) 2024-03-04T16:14:06.266 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:14:06.266+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:14:06.266852+0000) 2024-03-04T16:14:07.266 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:14:07.266+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:14:07.267097+0000) 2024-03-04T16:14:08.267 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:14:08.266+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:14:08.267343+0000) 2024-03-04T16:14:09.267 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:14:09.266+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:14:09.267574+0000) 2024-03-04T16:14:10.267 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:14:10.267+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:14:10.267866+0000) 2024-03-04T16:14:10.436 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:14:10.447 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:14:11.268 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:14:11.267+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:14:11.268149+0000) 2024-03-04T16:14:12.268 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:14:12.267+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:14:12.268386+0000) 2024-03-04T16:14:13.268 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:14:13.268+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:14:13.268627+0000) 2024-03-04T16:14:14.268 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:14:14.267+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:14:14.268808+0000) 2024-03-04T16:14:15.268 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:14:15.268+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:14:15.269086+0000) 2024-03-04T16:14:16.269 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:14:16.268+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:14:16.269324+0000) 2024-03-04T16:14:17.269 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:14:17.268+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:14:17.269556+0000) 2024-03-04T16:14:18.269 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:14:18.269+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:14:18.269802+0000) 2024-03-04T16:14:19.269 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:14:19.269+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:14:19.270080+0000) 2024-03-04T16:14:20.270 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:14:20.270+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:14:20.270321+0000) 2024-03-04T16:14:21.270 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:14:21.270+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:14:21.270572+0000) 2024-03-04T16:14:22.270 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:14:22.270+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:14:22.270814+0000) 2024-03-04T16:14:23.270 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:14:23.270+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:14:23.271086+0000) 2024-03-04T16:14:24.271 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:14:24.271+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:14:24.271324+0000) 2024-03-04T16:14:25.271 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:14:25.271+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:14:25.271565+0000) 2024-03-04T16:14:26.272 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:14:26.271+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:14:26.271803+0000) 2024-03-04T16:14:27.271 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:14:27.271+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:14:27.272101+0000) 2024-03-04T16:14:28.272 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:14:28.271+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:14:28.272364+0000) 2024-03-04T16:14:29.272 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:14:29.271+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:14:29.272616+0000) 2024-03-04T16:14:30.272 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:14:30.272+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:14:30.272893+0000) 2024-03-04T16:14:31.272 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:14:31.273+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:14:31.273144+0000) 2024-03-04T16:14:32.273 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:14:32.273+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:14:32.273402+0000) 2024-03-04T16:14:33.273 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:14:33.272+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:14:33.273645+0000) 2024-03-04T16:14:34.273 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:14:34.272+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:14:34.273925+0000) 2024-03-04T16:14:35.273 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:14:35.273+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:14:35.274166+0000) 2024-03-04T16:14:36.274 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:14:36.273+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:14:36.274410+0000) 2024-03-04T16:14:37.274 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:14:37.273+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:14:37.274655+0000) 2024-03-04T16:14:38.274 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:14:38.273+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:14:38.274936+0000) 2024-03-04T16:14:39.274 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:14:39.274+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:14:39.275172+0000) 2024-03-04T16:14:40.275 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:14:40.274+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:14:40.275410+0000) 2024-03-04T16:14:40.648 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:14:40.660 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:14:41.276 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:14:41.274+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:14:41.275658+0000) 2024-03-04T16:14:42.275 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:14:42.274+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:14:42.275937+0000) 2024-03-04T16:14:43.275 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:14:43.275+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:14:43.276182+0000) 2024-03-04T16:14:44.276 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:14:44.275+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:14:44.276429+0000) 2024-03-04T16:14:45.276 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:14:45.275+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:14:45.276663+0000) 2024-03-04T16:14:46.276 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:14:46.276+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:14:46.276939+0000) 2024-03-04T16:14:47.276 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:14:47.276+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:14:47.277185+0000) 2024-03-04T16:14:48.277 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:14:48.276+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:14:48.277427+0000) 2024-03-04T16:14:49.277 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:14:49.276+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:14:49.277673+0000) 2024-03-04T16:14:50.277 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:14:50.277+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:14:50.277930+0000) 2024-03-04T16:14:51.278 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:14:51.277+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:14:51.278184+0000) 2024-03-04T16:14:52.278 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:14:52.277+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:14:52.278423+0000) 2024-03-04T16:14:53.278 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:14:53.277+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:14:53.278661+0000) 2024-03-04T16:14:54.278 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:14:54.278+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:14:54.278934+0000) 2024-03-04T16:14:55.278 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:14:55.278+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:14:55.279171+0000) 2024-03-04T16:14:56.279 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:14:56.278+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:14:56.279412+0000) 2024-03-04T16:14:57.279 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:14:57.278+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:14:57.279670+0000) 2024-03-04T16:14:58.279 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:14:58.279+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:14:58.279904+0000) 2024-03-04T16:14:59.279 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:14:59.279+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:14:59.280144+0000) 2024-03-04T16:15:00.280 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:15:00.279+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:15:00.280430+0000) 2024-03-04T16:15:01.280 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:15:01.279+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:15:01.280678+0000) 2024-03-04T16:15:02.280 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:15:02.280+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:15:02.280956+0000) 2024-03-04T16:15:03.280 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:15:03.280+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:15:03.281192+0000) 2024-03-04T16:15:04.281 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:15:04.280+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:15:04.281429+0000) 2024-03-04T16:15:05.281 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:15:05.281+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:15:05.281672+0000) 2024-03-04T16:15:06.281 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:15:06.281+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:15:06.281943+0000) 2024-03-04T16:15:07.281 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:15:07.281+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:15:07.282188+0000) 2024-03-04T16:15:08.282 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:15:08.281+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:15:08.282449+0000) 2024-03-04T16:15:09.282 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:15:09.282+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:15:09.282685+0000) 2024-03-04T16:15:10.282 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:15:10.282+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:15:10.282935+0000) 2024-03-04T16:15:10.870 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:15:10.882 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:15:11.282 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:15:11.282+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:15:11.283180+0000) 2024-03-04T16:15:12.283 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:15:12.282+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:15:12.283431+0000) 2024-03-04T16:15:13.283 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:15:13.283+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:15:13.283675+0000) 2024-03-04T16:15:14.283 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:15:14.283+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:15:14.283946+0000) 2024-03-04T16:15:15.283 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:15:15.283+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:15:15.284185+0000) 2024-03-04T16:15:16.284 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:15:16.283+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:15:16.284426+0000) 2024-03-04T16:15:17.284 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:15:17.283+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:15:17.284616+0000) 2024-03-04T16:15:18.284 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:15:18.284+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:15:18.284894+0000) 2024-03-04T16:15:19.284 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:15:19.284+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:15:19.285133+0000) 2024-03-04T16:15:20.285 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:15:20.284+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:15:20.285371+0000) 2024-03-04T16:15:21.285 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:15:21.284+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:15:21.285649+0000) 2024-03-04T16:15:22.285 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:15:22.285+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:15:22.285934+0000) 2024-03-04T16:15:23.285 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:15:23.285+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:15:23.286127+0000) 2024-03-04T16:15:24.285 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:15:24.286+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:15:24.286363+0000) 2024-03-04T16:15:25.286 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:15:25.286+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:15:25.286602+0000) 2024-03-04T16:15:26.286 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:15:26.286+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:15:26.286876+0000) 2024-03-04T16:15:27.286 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:15:27.286+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:15:27.287142+0000) 2024-03-04T16:15:28.287 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:15:28.287+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:15:28.287406+0000) 2024-03-04T16:15:29.287 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:15:29.286+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:15:29.287633+0000) 2024-03-04T16:15:30.287 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:15:30.287+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:15:30.287937+0000) 2024-03-04T16:15:31.287 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:15:31.287+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:15:31.288181+0000) 2024-03-04T16:15:32.288 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:15:32.288+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:15:32.288433+0000) 2024-03-04T16:15:33.288 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:15:33.288+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:15:33.288674+0000) 2024-03-04T16:15:34.288 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:15:34.288+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:15:34.288949+0000) 2024-03-04T16:15:35.288 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:15:35.288+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:15:35.289140+0000) 2024-03-04T16:15:36.289 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:15:36.289+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:15:36.289378+0000) 2024-03-04T16:15:37.289 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:15:37.288+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:15:37.289622+0000) 2024-03-04T16:15:38.289 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:15:38.288+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:15:38.289925+0000) 2024-03-04T16:15:39.289 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:15:39.288+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:15:39.290157+0000) 2024-03-04T16:15:40.290 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:15:40.288+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:15:40.290395+0000) 2024-03-04T16:15:41.092 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:15:41.101 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:15:41.290 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:15:41.289+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:15:41.290646+0000) 2024-03-04T16:15:42.291 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:15:42.289+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:15:42.290938+0000) 2024-03-04T16:15:43.290 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:15:43.290+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:15:43.291177+0000) 2024-03-04T16:15:44.291 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:15:44.289+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:15:44.291366+0000) 2024-03-04T16:15:45.291 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:15:45.290+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:15:45.291602+0000) 2024-03-04T16:15:46.291 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:15:46.290+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:15:46.291885+0000) 2024-03-04T16:15:47.291 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:15:47.291+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:15:47.292129+0000) 2024-03-04T16:15:48.292 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:15:48.291+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:15:48.292432+0000) 2024-03-04T16:15:49.292 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:15:49.291+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:15:49.292667+0000) 2024-03-04T16:15:50.292 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:15:50.292+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:15:50.292946+0000) 2024-03-04T16:15:51.292 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:15:51.292+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:15:51.293197+0000) 2024-03-04T16:15:52.293 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:15:52.292+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:15:52.293435+0000) 2024-03-04T16:15:53.293 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:15:53.292+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:15:53.293668+0000) 2024-03-04T16:15:54.293 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:15:54.293+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:15:54.293952+0000) 2024-03-04T16:15:55.293 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:15:55.293+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:15:55.294194+0000) 2024-03-04T16:15:56.294 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:15:56.293+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:15:56.294431+0000) 2024-03-04T16:15:57.294 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:15:57.293+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:15:57.294672+0000) 2024-03-04T16:15:58.294 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:15:58.294+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:15:58.294955+0000) 2024-03-04T16:15:59.294 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:15:59.294+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:15:59.295212+0000) 2024-03-04T16:16:00.295 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:16:00.294+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:16:00.295455+0000) 2024-03-04T16:16:01.295 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:16:01.294+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:16:01.295708+0000) 2024-03-04T16:16:02.295 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:16:02.295+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:16:02.295939+0000) 2024-03-04T16:16:03.295 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:16:03.295+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:16:03.296179+0000) 2024-03-04T16:16:04.296 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:16:04.295+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:16:04.296434+0000) 2024-03-04T16:16:05.296 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:16:05.295+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:16:05.296675+0000) 2024-03-04T16:16:06.296 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:16:06.296+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:16:06.296932+0000) 2024-03-04T16:16:07.296 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:16:07.296+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:16:07.297192+0000) 2024-03-04T16:16:08.297 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:16:08.296+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:16:08.297452+0000) 2024-03-04T16:16:09.300 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:16:09.297+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:16:09.297706+0000) 2024-03-04T16:16:10.297 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:16:10.297+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:16:10.297997+0000) 2024-03-04T16:16:11.298 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:16:11.299 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:16:11.297+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:16:11.298266+0000) 2024-03-04T16:16:11.306 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:16:12.298 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:16:12.297+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:16:12.298467+0000) 2024-03-04T16:16:13.298 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:16:13.298+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:16:13.298718+0000) 2024-03-04T16:16:14.298 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:16:14.298+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:16:14.298971+0000) 2024-03-04T16:16:15.298 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:16:15.298+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:16:15.299235+0000) 2024-03-04T16:16:16.299 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:16:16.298+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:16:16.299498+0000) 2024-03-04T16:16:17.299 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:16:17.298+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:16:17.299745+0000) 2024-03-04T16:16:18.299 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:16:18.299+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:16:18.300009+0000) 2024-03-04T16:16:19.299 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:16:19.299+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:16:19.300293+0000) 2024-03-04T16:16:20.300 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:16:20.299+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:16:20.300549+0000) 2024-03-04T16:16:21.300 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:16:21.300+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:16:21.300857+0000) 2024-03-04T16:16:22.300 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:16:22.300+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:16:22.301071+0000) 2024-03-04T16:16:23.301 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:16:23.300+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:16:23.301327+0000) 2024-03-04T16:16:24.301 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:16:24.301+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:16:24.301578+0000) 2024-03-04T16:16:25.301 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:16:25.301+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:16:25.301878+0000) 2024-03-04T16:16:26.303 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:16:26.301+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:16:26.302143+0000) 2024-03-04T16:16:27.302 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:16:27.302+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:16:27.302428+0000) 2024-03-04T16:16:28.302 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:16:28.302+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:16:28.302704+0000) 2024-03-04T16:16:29.302 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:16:29.302+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:16:29.302961+0000) 2024-03-04T16:16:30.302 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:16:30.302+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:16:30.303276+0000) 2024-03-04T16:16:31.303 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:16:31.303+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:16:31.303546+0000) 2024-03-04T16:16:32.303 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:16:32.303+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:16:32.303859+0000) 2024-03-04T16:16:33.303 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:16:33.303+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:16:33.304121+0000) 2024-03-04T16:16:34.303 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:16:34.304+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:16:34.304375+0000) 2024-03-04T16:16:35.304 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:16:35.304+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:16:35.304644+0000) 2024-03-04T16:16:36.304 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:16:36.304+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:16:36.304895+0000) 2024-03-04T16:16:37.304 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:16:37.304+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:16:37.305153+0000) 2024-03-04T16:16:38.305 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:16:38.305+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:16:38.305408+0000) 2024-03-04T16:16:39.305 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:16:39.304+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:16:39.305676+0000) 2024-03-04T16:16:40.305 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:16:40.305+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:16:40.305953+0000) 2024-03-04T16:16:41.305 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:16:41.305+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:16:41.306188+0000) 2024-03-04T16:16:41.561 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:16:41.569 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:16:42.306 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:16:42.305+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:16:42.306444+0000) 2024-03-04T16:16:43.306 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:16:43.305+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:16:43.306708+0000) 2024-03-04T16:16:44.306 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:16:44.305+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:16:44.306969+0000) 2024-03-04T16:16:45.306 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:16:45.306+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:16:45.307224+0000) 2024-03-04T16:16:46.307 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:16:46.306+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:16:46.307478+0000) 2024-03-04T16:16:47.307 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:16:47.306+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:16:47.307730+0000) 2024-03-04T16:16:48.307 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:16:48.306+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:16:48.307985+0000) 2024-03-04T16:16:49.307 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:16:49.307+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:16:49.308242+0000) 2024-03-04T16:16:50.308 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:16:50.307+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:16:50.308499+0000) 2024-03-04T16:16:51.308 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:16:51.307+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:16:51.308769+0000) 2024-03-04T16:16:52.308 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:16:52.308+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:16:52.309034+0000) 2024-03-04T16:16:53.309 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:16:53.308+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:16:53.309312+0000) 2024-03-04T16:16:54.309 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:16:54.308+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:16:54.309524+0000) 2024-03-04T16:16:55.309 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:16:55.308+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:16:55.309786+0000) 2024-03-04T16:16:56.309 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:16:56.309+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:16:56.310102+0000) 2024-03-04T16:16:57.310 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:16:57.309+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:16:57.310373+0000) 2024-03-04T16:16:58.310 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:16:58.309+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:16:58.310642+0000) 2024-03-04T16:16:59.310 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:16:59.310+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:16:59.310931+0000) 2024-03-04T16:17:00.310 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:17:00.310+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:17:00.311171+0000) 2024-03-04T16:17:01.311 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:17:01.310+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:17:01.311422+0000) 2024-03-04T16:17:02.311 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:17:02.310+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:17:02.311657+0000) 2024-03-04T16:17:03.311 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:17:03.311+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:17:03.311934+0000) 2024-03-04T16:17:04.311 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:17:04.311+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:17:04.312167+0000) 2024-03-04T16:17:05.312 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:17:05.311+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:17:05.312432+0000) 2024-03-04T16:17:06.312 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:17:06.311+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:17:06.312671+0000) 2024-03-04T16:17:07.312 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:17:07.312+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:17:07.312948+0000) 2024-03-04T16:17:08.312 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:17:08.312+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:17:08.313142+0000) 2024-03-04T16:17:09.313 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:17:09.312+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:17:09.313376+0000) 2024-03-04T16:17:10.313 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:17:10.312+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:17:10.313636+0000) 2024-03-04T16:17:11.313 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:17:11.313+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:17:11.313949+0000) 2024-03-04T16:17:11.768 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:17:11.777 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:17:12.313 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:17:12.313+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:17:12.314211+0000) 2024-03-04T16:17:13.314 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:17:13.313+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:17:13.314472+0000) 2024-03-04T16:17:14.314 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:17:14.313+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:17:14.314711+0000) 2024-03-04T16:17:15.314 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:17:15.314+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:17:15.314984+0000) 2024-03-04T16:17:16.314 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:17:16.314+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:17:16.315240+0000) 2024-03-04T16:17:17.315 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:17:17.314+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:17:17.315492+0000) 2024-03-04T16:17:18.315 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:17:18.315+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:17:18.315745+0000) 2024-03-04T16:17:19.315 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:17:19.315+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:17:19.315946+0000) 2024-03-04T16:17:20.315 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:17:20.315+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:17:20.316226+0000) 2024-03-04T16:17:21.316 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:17:21.315+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:17:21.316505+0000) 2024-03-04T16:17:22.316 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:17:22.315+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:17:22.316726+0000) 2024-03-04T16:17:23.316 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:17:23.316+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:17:23.316983+0000) 2024-03-04T16:17:24.316 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:17:24.316+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:17:24.317299+0000) 2024-03-04T16:17:25.317 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:17:25.317+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:17:25.317576+0000) 2024-03-04T16:17:26.317 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:17:26.317+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:17:26.317865+0000) 2024-03-04T16:17:27.317 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:17:27.317+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:17:27.318144+0000) 2024-03-04T16:17:28.317 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:17:28.317+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:17:28.318350+0000) 2024-03-04T16:17:29.318 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:17:29.318+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:17:29.318614+0000) 2024-03-04T16:17:30.318 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:17:30.318+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:17:30.318929+0000) 2024-03-04T16:17:31.318 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:17:31.318+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:17:31.319201+0000) 2024-03-04T16:17:32.318 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:17:32.319+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:17:32.319453+0000) 2024-03-04T16:17:33.319 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:17:33.318+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:17:33.319720+0000) 2024-03-04T16:17:34.319 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:17:34.319+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:17:34.319974+0000) 2024-03-04T16:17:35.319 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:17:35.319+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:17:35.320226+0000) 2024-03-04T16:17:36.320 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:17:36.320+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:17:36.320484+0000) 2024-03-04T16:17:37.320 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:17:37.319+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:17:37.320760+0000) 2024-03-04T16:17:38.320 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:17:38.320+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:17:38.321034+0000) 2024-03-04T16:17:39.320 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:17:39.320+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:17:39.321278+0000) 2024-03-04T16:17:40.321 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:17:40.321+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:17:40.321539+0000) 2024-03-04T16:17:41.321 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:17:41.321+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:17:41.321805+0000) 2024-03-04T16:17:41.980 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:17:41.988 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:17:42.321 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:17:42.321+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:17:42.322093+0000) 2024-03-04T16:17:43.321 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:17:43.322+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:17:43.322339+0000) 2024-03-04T16:17:44.322 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:17:44.322+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:17:44.322591+0000) 2024-03-04T16:17:45.322 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:17:45.322+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:17:45.322861+0000) 2024-03-04T16:17:46.322 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:17:46.322+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:17:46.323110+0000) 2024-03-04T16:17:47.323 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:17:47.322+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:17:47.323397+0000) 2024-03-04T16:17:48.323 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:17:48.322+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:17:48.323652+0000) 2024-03-04T16:17:49.323 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:17:49.322+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:17:49.323891+0000) 2024-03-04T16:17:50.323 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:17:50.323+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:17:50.324150+0000) 2024-03-04T16:17:51.324 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:17:51.323+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:17:51.324420+0000) 2024-03-04T16:17:52.324 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:17:52.323+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:17:52.324659+0000) 2024-03-04T16:17:53.324 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:17:53.323+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:17:53.324936+0000) 2024-03-04T16:17:54.324 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:17:54.323+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:17:54.325165+0000) 2024-03-04T16:17:55.324 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:17:55.324+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:17:55.325428+0000) 2024-03-04T16:17:56.325 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:17:56.324+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:17:56.325674+0000) 2024-03-04T16:17:57.325 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:17:57.324+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:17:57.325943+0000) 2024-03-04T16:17:58.325 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:17:58.325+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:17:58.326179+0000) 2024-03-04T16:17:59.325 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:17:59.325+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:17:59.326369+0000) 2024-03-04T16:18:00.326 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:18:00.325+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:18:00.326611+0000) 2024-03-04T16:18:01.326 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:18:01.326+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:18:01.326948+0000) 2024-03-04T16:18:02.326 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:18:02.326+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:18:02.327139+0000) 2024-03-04T16:18:03.327 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:18:03.326+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:18:03.327376+0000) 2024-03-04T16:18:04.327 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:18:04.326+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:18:04.327613+0000) 2024-03-04T16:18:05.327 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:18:05.327+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:18:05.327893+0000) 2024-03-04T16:18:06.327 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:18:06.327+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:18:06.328130+0000) 2024-03-04T16:18:07.328 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:18:07.327+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:18:07.328375+0000) 2024-03-04T16:18:08.328 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:18:08.327+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:18:08.328614+0000) 2024-03-04T16:18:09.328 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:18:09.328+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:18:09.328908+0000) 2024-03-04T16:18:10.328 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:18:10.328+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:18:10.329152+0000) 2024-03-04T16:18:11.329 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:18:11.328+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:18:11.329403+0000) 2024-03-04T16:18:12.177 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:18:12.185 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:18:12.329 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:18:12.328+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:18:12.329655+0000) 2024-03-04T16:18:13.329 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:18:13.329+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:18:13.329937+0000) 2024-03-04T16:18:14.329 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:18:14.329+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:18:14.330116+0000) 2024-03-04T16:18:15.329 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:18:15.329+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:18:15.330349+0000) 2024-03-04T16:18:16.330 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:18:16.329+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:18:16.330547+0000) 2024-03-04T16:18:17.330 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:18:17.330+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:18:17.330913+0000) 2024-03-04T16:18:18.330 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:18:18.330+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:18:18.331172+0000) 2024-03-04T16:18:19.331 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:18:19.330+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:18:19.331414+0000) 2024-03-04T16:18:20.331 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:18:20.331+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:18:20.331664+0000) 2024-03-04T16:18:21.331 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:18:21.331+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:18:21.331933+0000) 2024-03-04T16:18:22.331 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:18:22.331+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:18:22.332190+0000) 2024-03-04T16:18:23.332 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:18:23.331+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:18:23.332443+0000) 2024-03-04T16:18:24.335 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:18:24.332+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:18:24.332715+0000) 2024-03-04T16:18:25.332 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:18:25.332+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:18:25.332969+0000) 2024-03-04T16:18:26.332 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:18:26.332+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:18:26.333225+0000) 2024-03-04T16:18:27.333 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:18:27.332+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:18:27.333474+0000) 2024-03-04T16:18:28.333 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:18:28.332+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:18:28.333743+0000) 2024-03-04T16:18:29.333 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:18:29.333+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:18:29.333996+0000) 2024-03-04T16:18:30.333 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:18:30.333+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:18:30.334248+0000) 2024-03-04T16:18:31.334 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:18:31.334+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:18:31.334510+0000) 2024-03-04T16:18:32.334 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:18:32.334+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:18:32.334768+0000) 2024-03-04T16:18:33.334 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:18:33.334+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:18:33.335008+0000) 2024-03-04T16:18:34.334 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:18:34.334+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:18:34.335275+0000) 2024-03-04T16:18:35.335 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:18:35.335+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:18:35.335514+0000) 2024-03-04T16:18:36.335 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:18:36.335+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:18:36.335759+0000) 2024-03-04T16:18:37.335 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:18:37.335+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:18:37.336016+0000) 2024-03-04T16:18:38.335 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:18:38.335+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:18:38.336263+0000) 2024-03-04T16:18:39.336 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:18:39.336+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:18:39.336504+0000) 2024-03-04T16:18:40.336 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:18:40.336+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:18:40.336755+0000) 2024-03-04T16:18:41.336 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:18:41.336+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:18:41.337003+0000) 2024-03-04T16:18:42.336 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:18:42.336+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:18:42.337194+0000) 2024-03-04T16:18:42.391 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:18:42.400 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:18:43.337 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:18:43.337+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:18:43.337436+0000) 2024-03-04T16:18:44.337 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:18:44.337+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:18:44.337677+0000) 2024-03-04T16:18:45.337 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:18:45.337+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:18:45.337937+0000) 2024-03-04T16:18:46.337 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:18:46.337+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:18:46.338178+0000) 2024-03-04T16:18:47.337 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:18:47.337+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:18:47.338359+0000) 2024-03-04T16:18:48.338 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:18:48.338+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:18:48.338599+0000) 2024-03-04T16:18:49.338 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:18:49.338+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:18:49.338892+0000) 2024-03-04T16:18:50.338 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:18:50.338+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:18:50.339129+0000) 2024-03-04T16:18:51.338 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:18:51.338+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:18:51.339370+0000) 2024-03-04T16:18:52.339 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:18:52.338+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:18:52.339611+0000) 2024-03-04T16:18:53.339 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:18:53.338+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:18:53.339897+0000) 2024-03-04T16:18:54.339 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:18:54.339+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:18:54.340132+0000) 2024-03-04T16:18:55.339 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:18:55.338+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:18:55.340323+0000) 2024-03-04T16:18:56.340 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:18:56.339+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:18:56.340584+0000) 2024-03-04T16:18:57.340 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:18:57.339+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:18:57.340870+0000) 2024-03-04T16:18:58.340 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:18:58.340+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:18:58.341123+0000) 2024-03-04T16:18:59.340 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:18:59.340+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:18:59.341360+0000) 2024-03-04T16:19:00.341 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:19:00.340+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:19:00.341601+0000) 2024-03-04T16:19:01.341 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:19:01.340+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:19:01.341912+0000) 2024-03-04T16:19:02.341 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:19:02.341+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:19:02.342166+0000) 2024-03-04T16:19:03.342 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:19:03.341+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:19:03.342405+0000) 2024-03-04T16:19:04.342 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:19:04.341+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:19:04.342641+0000) 2024-03-04T16:19:05.342 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:19:05.342+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:19:05.342985+0000) 2024-03-04T16:19:06.342 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:19:06.342+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:19:06.343240+0000) 2024-03-04T16:19:07.343 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:19:07.342+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:19:07.343497+0000) 2024-03-04T16:19:08.343 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:19:08.342+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:19:08.343767+0000) 2024-03-04T16:19:09.343 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:19:09.343+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:19:09.344001+0000) 2024-03-04T16:19:10.343 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:19:10.343+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:19:10.344242+0000) 2024-03-04T16:19:11.344 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:19:11.343+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:19:11.344491+0000) 2024-03-04T16:19:12.344 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:19:12.343+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:19:12.344738+0000) 2024-03-04T16:19:12.622 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:19:12.629 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:19:13.344 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:19:13.344+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:19:13.344996+0000) 2024-03-04T16:19:14.344 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:19:14.344+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:19:14.345177+0000) 2024-03-04T16:19:15.345 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:19:15.344+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:19:15.345432+0000) 2024-03-04T16:19:16.345 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:19:16.344+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:19:16.345681+0000) 2024-03-04T16:19:17.345 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:19:17.345+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:19:17.345954+0000) 2024-03-04T16:19:18.345 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:19:18.345+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:19:18.346205+0000) 2024-03-04T16:19:19.346 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:19:19.345+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:19:19.346387+0000) 2024-03-04T16:19:20.346 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:19:20.345+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:19:20.346625+0000) 2024-03-04T16:19:21.346 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:19:21.346+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:19:21.346908+0000) 2024-03-04T16:19:22.346 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:19:22.346+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:19:22.347163+0000) 2024-03-04T16:19:23.346 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:19:23.346+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:19:23.347372+0000) 2024-03-04T16:19:24.347 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:19:24.347+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:19:24.347623+0000) 2024-03-04T16:19:25.347 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:19:25.347+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:19:25.347919+0000) 2024-03-04T16:19:26.347 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:19:26.347+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:19:26.348187+0000) 2024-03-04T16:19:27.347 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:19:27.347+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:19:27.348438+0000) 2024-03-04T16:19:28.348 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:19:28.347+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:19:28.348712+0000) 2024-03-04T16:19:29.348 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:19:29.348+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:19:29.348977+0000) 2024-03-04T16:19:30.348 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:19:30.348+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:19:30.349231+0000) 2024-03-04T16:19:31.349 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:19:31.348+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:19:31.349489+0000) 2024-03-04T16:19:32.349 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:19:32.348+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:19:32.349760+0000) 2024-03-04T16:19:33.349 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:19:33.349+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:19:33.349995+0000) 2024-03-04T16:19:34.349 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:19:34.349+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:19:34.350230+0000) 2024-03-04T16:19:35.350 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:19:35.349+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:19:35.350469+0000) 2024-03-04T16:19:36.350 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:19:36.349+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:19:36.350673+0000) 2024-03-04T16:19:37.350 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:19:37.350+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:19:37.350966+0000) 2024-03-04T16:19:38.350 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:19:38.350+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:19:38.351177+0000) 2024-03-04T16:19:39.350 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:19:39.350+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:19:39.351370+0000) 2024-03-04T16:19:40.351 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:19:40.350+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:19:40.351601+0000) 2024-03-04T16:19:41.351 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:19:41.351+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:19:41.351905+0000) 2024-03-04T16:19:42.351 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:19:42.351+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:19:42.352143+0000) 2024-03-04T16:19:42.818 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:19:42.826 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:19:43.352 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:19:43.352+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:19:43.352381+0000) 2024-03-04T16:19:44.352 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:19:44.351+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:19:44.352560+0000) 2024-03-04T16:19:45.352 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:19:45.352+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:19:45.352865+0000) 2024-03-04T16:19:46.352 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:19:46.352+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:19:46.353126+0000) 2024-03-04T16:19:47.353 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:19:47.352+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:19:47.353384+0000) 2024-03-04T16:19:48.353 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:19:48.353+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:19:48.353625+0000) 2024-03-04T16:19:49.353 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:19:49.353+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:19:49.353917+0000) 2024-03-04T16:19:50.353 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:19:50.353+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:19:50.354152+0000) 2024-03-04T16:19:51.353 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:19:51.353+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:19:51.354354+0000) 2024-03-04T16:19:52.354 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:19:52.354+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:19:52.354624+0000) 2024-03-04T16:19:53.354 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:19:53.354+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:19:53.354923+0000) 2024-03-04T16:19:54.354 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:19:54.355+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:19:54.355158+0000) 2024-03-04T16:19:55.355 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:19:55.355+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:19:55.355415+0000) 2024-03-04T16:19:56.355 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:19:56.354+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:19:56.355663+0000) 2024-03-04T16:19:57.355 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:19:57.354+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:19:57.355930+0000) 2024-03-04T16:19:58.355 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:19:58.355+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:19:58.356236+0000) 2024-03-04T16:19:59.356 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:19:59.354+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:19:59.356474+0000) 2024-03-04T16:20:00.356 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:20:00.355+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:20:00.356716+0000) 2024-03-04T16:20:01.356 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:20:01.355+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:20:01.356968+0000) 2024-03-04T16:20:02.356 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:20:02.356+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:20:02.357211+0000) 2024-03-04T16:20:03.357 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:20:03.356+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:20:03.357464+0000) 2024-03-04T16:20:04.357 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:20:04.356+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:20:04.357704+0000) 2024-03-04T16:20:05.357 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:20:05.356+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:20:05.357946+0000) 2024-03-04T16:20:06.357 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:20:06.357+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:20:06.358191+0000) 2024-03-04T16:20:07.358 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:20:07.357+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:20:07.358432+0000) 2024-03-04T16:20:08.358 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:20:08.357+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:20:08.358673+0000) 2024-03-04T16:20:09.358 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:20:09.358+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:20:09.358960+0000) 2024-03-04T16:20:10.358 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:20:10.358+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:20:10.359204+0000) 2024-03-04T16:20:11.359 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:20:11.358+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:20:11.359456+0000) 2024-03-04T16:20:12.359 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:20:12.358+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:20:12.359701+0000) 2024-03-04T16:20:13.015 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:20:13.026 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:20:13.359 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:20:13.359+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:20:13.359948+0000) 2024-03-04T16:20:14.359 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:20:14.359+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:20:14.360194+0000) 2024-03-04T16:20:15.360 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:20:15.359+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:20:15.360388+0000) 2024-03-04T16:20:16.360 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:20:16.359+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:20:16.360630+0000) 2024-03-04T16:20:17.360 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:20:17.360+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:20:17.360927+0000) 2024-03-04T16:20:18.360 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:20:18.360+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:20:18.361111+0000) 2024-03-04T16:20:19.360 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:20:19.360+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:20:19.361359+0000) 2024-03-04T16:20:20.361 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:20:20.360+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:20:20.361611+0000) 2024-03-04T16:20:21.361 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:20:21.361+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:20:21.361913+0000) 2024-03-04T16:20:22.361 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:20:22.361+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:20:22.362156+0000) 2024-03-04T16:20:23.361 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:20:23.361+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:20:23.362392+0000) 2024-03-04T16:20:24.362 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:20:24.361+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:20:24.362632+0000) 2024-03-04T16:20:25.362 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:20:25.362+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:20:25.362910+0000) 2024-03-04T16:20:26.362 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:20:26.362+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:20:26.363156+0000) 2024-03-04T16:20:27.363 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:20:27.362+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:20:27.363405+0000) 2024-03-04T16:20:28.363 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:20:28.363+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:20:28.363643+0000) 2024-03-04T16:20:29.363 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:20:29.363+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:20:29.363940+0000) 2024-03-04T16:20:30.363 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:20:30.363+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:20:30.364181+0000) 2024-03-04T16:20:31.364 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:20:31.363+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:20:31.364429+0000) 2024-03-04T16:20:32.364 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:20:32.363+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:20:32.364609+0000) 2024-03-04T16:20:33.364 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:20:33.364+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:20:33.364905+0000) 2024-03-04T16:20:34.364 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:20:34.364+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:20:34.365141+0000) 2024-03-04T16:20:35.364 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:20:35.364+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:20:35.365380+0000) 2024-03-04T16:20:36.365 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:20:36.365+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:20:36.365623+0000) 2024-03-04T16:20:37.365 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:20:37.365+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:20:37.365900+0000) 2024-03-04T16:20:38.365 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:20:38.365+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:20:38.366131+0000) 2024-03-04T16:20:39.365 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:20:39.365+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:20:39.366319+0000) 2024-03-04T16:20:40.366 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:20:40.366+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:20:40.366562+0000) 2024-03-04T16:20:41.366 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:20:41.366+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:20:41.366906+0000) 2024-03-04T16:20:42.366 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:20:42.366+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:20:42.367145+0000) 2024-03-04T16:20:43.153 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:20:43.164 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:20:43.366 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:20:43.366+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:20:43.367388+0000) 2024-03-04T16:20:44.367 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:20:44.367+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:20:44.367623+0000) 2024-03-04T16:20:45.367 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:20:45.367+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:20:45.367910+0000) 2024-03-04T16:20:46.367 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:20:46.367+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:20:46.368159+0000) 2024-03-04T16:20:47.367 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:20:47.368+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:20:47.368401+0000) 2024-03-04T16:20:48.368 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:20:48.367+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:20:48.368660+0000) 2024-03-04T16:20:49.368 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:20:49.368+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:20:49.368928+0000) 2024-03-04T16:20:50.368 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:20:50.368+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:20:50.369171+0000) 2024-03-04T16:20:51.369 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:20:51.369+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:20:51.369443+0000) 2024-03-04T16:20:52.369 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:20:52.369+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:20:52.369722+0000) 2024-03-04T16:20:53.371 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:20:53.369+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:20:53.369963+0000) 2024-03-04T16:20:54.369 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:20:54.369+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:20:54.370193+0000) 2024-03-04T16:20:55.370 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:20:55.370+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:20:55.370459+0000) 2024-03-04T16:20:56.370 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:20:56.370+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:20:56.370721+0000) 2024-03-04T16:20:57.370 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:20:57.370+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:20:57.370962+0000) 2024-03-04T16:20:58.370 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:20:58.371+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:20:58.371201+0000) 2024-03-04T16:20:59.371 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:20:59.371+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:20:59.371439+0000) 2024-03-04T16:21:00.371 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:21:00.371+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:21:00.371674+0000) 2024-03-04T16:21:01.371 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:21:01.370+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:21:01.371928+0000) 2024-03-04T16:21:02.371 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:21:02.371+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:21:02.372174+0000) 2024-03-04T16:21:03.372 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:21:03.371+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:21:03.372413+0000) 2024-03-04T16:21:04.372 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:21:04.371+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:21:04.372663+0000) 2024-03-04T16:21:05.372 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:21:05.371+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:21:05.372946+0000) 2024-03-04T16:21:06.372 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:21:06.372+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:21:06.373191+0000) 2024-03-04T16:21:07.373 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:21:07.371+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:21:07.373451+0000) 2024-03-04T16:21:08.373 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:21:08.372+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:21:08.373692+0000) 2024-03-04T16:21:09.373 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:21:09.372+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:21:09.373936+0000) 2024-03-04T16:21:10.373 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:21:10.373+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:21:10.374131+0000) 2024-03-04T16:21:11.374 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:21:11.373+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:21:11.374378+0000) 2024-03-04T16:21:12.374 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:21:12.373+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:21:12.374632+0000) 2024-03-04T16:21:13.299 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:21:13.307 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:21:13.374 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:21:13.374+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:21:13.374848+0000) 2024-03-04T16:21:14.374 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:21:14.374+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:21:14.375023+0000) 2024-03-04T16:21:15.374 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:21:15.374+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:21:15.375261+0000) 2024-03-04T16:21:16.375 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:21:16.374+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:21:16.375507+0000) 2024-03-04T16:21:17.375 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:21:17.374+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:21:17.375782+0000) 2024-03-04T16:21:18.375 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:21:18.375+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:21:18.375987+0000) 2024-03-04T16:21:19.375 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:21:19.375+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:21:19.376219+0000) 2024-03-04T16:21:20.376 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:21:20.375+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:21:20.376458+0000) 2024-03-04T16:21:21.376 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:21:21.375+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:21:21.376707+0000) 2024-03-04T16:21:22.376 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:21:22.376+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:21:22.376946+0000) 2024-03-04T16:21:23.376 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:21:23.376+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:21:23.377203+0000) 2024-03-04T16:21:24.376 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:21:24.376+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:21:24.377440+0000) 2024-03-04T16:21:25.377 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:21:25.376+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:21:25.377673+0000) 2024-03-04T16:21:26.377 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:21:26.377+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:21:26.377950+0000) 2024-03-04T16:21:27.377 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:21:27.377+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:21:27.378168+0000) 2024-03-04T16:21:28.378 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:21:28.377+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:21:28.378427+0000) 2024-03-04T16:21:29.378 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:21:29.377+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:21:29.378697+0000) 2024-03-04T16:21:30.378 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:21:30.378+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:21:30.378941+0000) 2024-03-04T16:21:31.378 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:21:31.378+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:21:31.379195+0000) 2024-03-04T16:21:32.378 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:21:32.378+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:21:32.379445+0000) 2024-03-04T16:21:33.379 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:21:33.379+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:21:33.379680+0000) 2024-03-04T16:21:34.379 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:21:34.379+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:21:34.380004+0000) 2024-03-04T16:21:35.379 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:21:35.379+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:21:35.380260+0000) 2024-03-04T16:21:36.380 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:21:36.379+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:21:36.380500+0000) 2024-03-04T16:21:37.380 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:21:37.380+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:21:37.380772+0000) 2024-03-04T16:21:38.380 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:21:38.380+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:21:38.381031+0000) 2024-03-04T16:21:39.380 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:21:39.380+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:21:39.381263+0000) 2024-03-04T16:21:40.381 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:21:40.381+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:21:40.381526+0000) 2024-03-04T16:21:41.381 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:21:41.381+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:21:41.381778+0000) 2024-03-04T16:21:42.381 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:21:42.381+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:21:42.382052+0000) 2024-03-04T16:21:43.381 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:21:43.381+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:21:43.382293+0000) 2024-03-04T16:21:43.472 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:21:43.481 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:21:44.382 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:21:44.382+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:21:44.382529+0000) 2024-03-04T16:21:45.382 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:21:45.381+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:21:45.382706+0000) 2024-03-04T16:21:46.382 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:21:46.382+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:21:46.382937+0000) 2024-03-04T16:21:47.382 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:21:47.382+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:21:47.383183+0000) 2024-03-04T16:21:48.383 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:21:48.382+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:21:48.383438+0000) 2024-03-04T16:21:49.383 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:21:49.383+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:21:49.383675+0000) 2024-03-04T16:21:50.383 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:21:50.383+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:21:50.383944+0000) 2024-03-04T16:21:51.383 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:21:51.383+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:21:51.384198+0000) 2024-03-04T16:21:52.384 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:21:52.384+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:21:52.384443+0000) 2024-03-04T16:21:53.384 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:21:53.384+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:21:53.384682+0000) 2024-03-04T16:21:54.384 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:21:54.384+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:21:54.384936+0000) 2024-03-04T16:21:55.384 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:21:55.384+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:21:55.385170+0000) 2024-03-04T16:21:56.385 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:21:56.385+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:21:56.385413+0000) 2024-03-04T16:21:57.385 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:21:57.385+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:21:57.385684+0000) 2024-03-04T16:21:58.385 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:21:58.385+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:21:58.385944+0000) 2024-03-04T16:21:59.385 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:21:59.385+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:21:59.386183+0000) 2024-03-04T16:22:00.386 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:22:00.386+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:22:00.386418+0000) 2024-03-04T16:22:01.386 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:22:01.386+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:22:01.386709+0000) 2024-03-04T16:22:02.386 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:22:02.386+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:22:02.386947+0000) 2024-03-04T16:22:03.386 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:22:03.387+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:22:03.387190+0000) 2024-03-04T16:22:04.387 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:22:04.387+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:22:04.387427+0000) 2024-03-04T16:22:05.387 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:22:05.386+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:22:05.387662+0000) 2024-03-04T16:22:06.387 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:22:06.386+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:22:06.387938+0000) 2024-03-04T16:22:07.387 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:22:07.387+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:22:07.388178+0000) 2024-03-04T16:22:08.388 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:22:08.387+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:22:08.388421+0000) 2024-03-04T16:22:09.388 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:22:09.387+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:22:09.388673+0000) 2024-03-04T16:22:10.388 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:22:10.387+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:22:10.388881+0000) 2024-03-04T16:22:11.389 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:22:11.388+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:22:11.389136+0000) 2024-03-04T16:22:12.389 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:22:12.387+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:22:12.389368+0000) 2024-03-04T16:22:13.389 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:22:13.388+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:22:13.389626+0000) 2024-03-04T16:22:13.660 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:22:13.672 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:22:14.389 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:22:14.388+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:22:14.389927+0000) 2024-03-04T16:22:15.389 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:22:15.389+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:22:15.390163+0000) 2024-03-04T16:22:16.390 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:22:16.389+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:22:16.390400+0000) 2024-03-04T16:22:17.390 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:22:17.389+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:22:17.390647+0000) 2024-03-04T16:22:18.390 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:22:18.390+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:22:18.390936+0000) 2024-03-04T16:22:19.390 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:22:19.390+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:22:19.391174+0000) 2024-03-04T16:22:20.391 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:22:20.390+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:22:20.391427+0000) 2024-03-04T16:22:21.391 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:22:21.390+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:22:21.391675+0000) 2024-03-04T16:22:22.391 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:22:22.391+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:22:22.391951+0000) 2024-03-04T16:22:23.391 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:22:23.391+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:22:23.392142+0000) 2024-03-04T16:22:24.391 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:22:24.391+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:22:24.392379+0000) 2024-03-04T16:22:25.392 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:22:25.391+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:22:25.392614+0000) 2024-03-04T16:22:26.392 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:22:26.392+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:22:26.392893+0000) 2024-03-04T16:22:27.392 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:22:27.392+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:22:27.393157+0000) 2024-03-04T16:22:28.393 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:22:28.392+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:22:28.393489+0000) 2024-03-04T16:22:29.393 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:22:29.392+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:22:29.393725+0000) 2024-03-04T16:22:30.393 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:22:30.393+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:22:30.393963+0000) 2024-03-04T16:22:31.393 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:22:31.393+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:22:31.394208+0000) 2024-03-04T16:22:32.395 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:22:32.393+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:22:32.394482+0000) 2024-03-04T16:22:33.394 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:22:33.394+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:22:33.394727+0000) 2024-03-04T16:22:34.394 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:22:34.394+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:22:34.394970+0000) 2024-03-04T16:22:35.394 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:22:35.394+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:22:35.395207+0000) 2024-03-04T16:22:36.395 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:22:36.394+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:22:36.395443+0000) 2024-03-04T16:22:37.395 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:22:37.395+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:22:37.395688+0000) 2024-03-04T16:22:38.395 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:22:38.395+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:22:38.395930+0000) 2024-03-04T16:22:39.395 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:22:39.395+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:22:39.396167+0000) 2024-03-04T16:22:40.395 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:22:40.395+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:22:40.396407+0000) 2024-03-04T16:22:41.396 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:22:41.395+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:22:41.396652+0000) 2024-03-04T16:22:42.396 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:22:42.396+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:22:42.396926+0000) 2024-03-04T16:22:43.396 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:22:43.396+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:22:43.397167+0000) 2024-03-04T16:22:43.875 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:22:43.883 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:22:44.397 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:22:44.396+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:22:44.397418+0000) 2024-03-04T16:22:45.397 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:22:45.397+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:22:45.397654+0000) 2024-03-04T16:22:46.397 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:22:46.397+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:22:46.397925+0000) 2024-03-04T16:22:47.397 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:22:47.397+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:22:47.398166+0000) 2024-03-04T16:22:48.398 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:22:48.397+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:22:48.398414+0000) 2024-03-04T16:22:49.399 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:22:49.397+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:22:49.398668+0000) 2024-03-04T16:22:50.398 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:22:50.398+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:22:50.398930+0000) 2024-03-04T16:22:51.398 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:22:51.398+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:22:51.399176+0000) 2024-03-04T16:22:52.398 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:22:52.398+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:22:52.399413+0000) 2024-03-04T16:22:53.399 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:22:53.399+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:22:53.399653+0000) 2024-03-04T16:22:54.399 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:22:54.399+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:22:54.399861+0000) 2024-03-04T16:22:55.399 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:22:55.399+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:22:55.400127+0000) 2024-03-04T16:22:56.399 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:22:56.399+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:22:56.400361+0000) 2024-03-04T16:22:57.400 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:22:57.399+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:22:57.400595+0000) 2024-03-04T16:22:58.400 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:22:58.400+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:22:58.400887+0000) 2024-03-04T16:22:59.400 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:22:59.400+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:22:59.401147+0000) 2024-03-04T16:23:00.400 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:23:00.401+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:23:00.401387+0000) 2024-03-04T16:23:01.401 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:23:01.401+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:23:01.401636+0000) 2024-03-04T16:23:02.401 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:23:02.401+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:23:02.401908+0000) 2024-03-04T16:23:03.401 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:23:03.401+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:23:03.402147+0000) 2024-03-04T16:23:04.401 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:23:04.402+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:23:04.402388+0000) 2024-03-04T16:23:05.402 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:23:05.401+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:23:05.402641+0000) 2024-03-04T16:23:06.402 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:23:06.402+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:23:06.402921+0000) 2024-03-04T16:23:07.402 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:23:07.402+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:23:07.403099+0000) 2024-03-04T16:23:08.403 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:23:08.403+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:23:08.403364+0000) 2024-03-04T16:23:09.403 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:23:09.403+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:23:09.403543+0000) 2024-03-04T16:23:10.403 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:23:10.402+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:23:10.403786+0000) 2024-03-04T16:23:11.403 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:23:11.402+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:23:11.404061+0000) 2024-03-04T16:23:12.403 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:23:12.403+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:23:12.404301+0000) 2024-03-04T16:23:13.404 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:23:13.403+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:23:13.404538+0000) 2024-03-04T16:23:14.071 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:23:14.079 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:23:14.404 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:23:14.403+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:23:14.404789+0000) 2024-03-04T16:23:15.404 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:23:15.403+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:23:15.405034+0000) 2024-03-04T16:23:16.404 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:23:16.404+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:23:16.405319+0000) 2024-03-04T16:23:17.405 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:23:17.404+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:23:17.405557+0000) 2024-03-04T16:23:18.405 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:23:18.404+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:23:18.405793+0000) 2024-03-04T16:23:19.405 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:23:19.405+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:23:19.406037+0000) 2024-03-04T16:23:20.405 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:23:20.404+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:23:20.406217+0000) 2024-03-04T16:23:21.406 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:23:21.405+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:23:21.406464+0000) 2024-03-04T16:23:22.406 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:23:22.405+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:23:22.406665+0000) 2024-03-04T16:23:23.406 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:23:23.406+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:23:23.406944+0000) 2024-03-04T16:23:24.406 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:23:24.406+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:23:24.407182+0000) 2024-03-04T16:23:25.407 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:23:25.406+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:23:25.407421+0000) 2024-03-04T16:23:26.407 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:23:26.406+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:23:26.407620+0000) 2024-03-04T16:23:27.407 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:23:27.407+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:23:27.407903+0000) 2024-03-04T16:23:28.407 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:23:28.407+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:23:28.408144+0000) 2024-03-04T16:23:29.407 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:23:29.407+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:23:29.408400+0000) 2024-03-04T16:23:30.408 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:23:30.407+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:23:30.408658+0000) 2024-03-04T16:23:31.408 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:23:31.408+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:23:31.408908+0000) 2024-03-04T16:23:32.408 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:23:32.408+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:23:32.409147+0000) 2024-03-04T16:23:33.409 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:23:33.408+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:23:33.409423+0000) 2024-03-04T16:23:34.409 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:23:34.408+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:23:34.409660+0000) 2024-03-04T16:23:35.409 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:23:35.409+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:23:35.409940+0000) 2024-03-04T16:23:36.409 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:23:36.409+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:23:36.410184+0000) 2024-03-04T16:23:37.409 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:23:37.409+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:23:37.410430+0000) 2024-03-04T16:23:38.410 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:23:38.409+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:23:38.410675+0000) 2024-03-04T16:23:39.410 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:23:39.410+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:23:39.410943+0000) 2024-03-04T16:23:40.410 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:23:40.410+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:23:40.411173+0000) 2024-03-04T16:23:41.411 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:23:41.410+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:23:41.411422+0000) 2024-03-04T16:23:42.411 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:23:42.411+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:23:42.411663+0000) 2024-03-04T16:23:43.411 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:23:43.411+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:23:43.411924+0000) 2024-03-04T16:23:44.208 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:23:44.217 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:23:44.411 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:23:44.411+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:23:44.412185+0000) 2024-03-04T16:23:45.412 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:23:45.411+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:23:45.412422+0000) 2024-03-04T16:23:46.412 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:23:46.412+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:23:46.412664+0000) 2024-03-04T16:23:47.412 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:23:47.412+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:23:47.412931+0000) 2024-03-04T16:23:48.412 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:23:48.412+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:23:48.413133+0000) 2024-03-04T16:23:49.415 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:23:49.412+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:23:49.413371+0000) 2024-03-04T16:23:50.413 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:23:50.413+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:23:50.413608+0000) 2024-03-04T16:23:51.413 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:23:51.413+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:23:51.413886+0000) 2024-03-04T16:23:52.413 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:23:52.413+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:23:52.414130+0000) 2024-03-04T16:23:53.414 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:23:53.413+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:23:53.414371+0000) 2024-03-04T16:23:54.414 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:23:54.413+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:23:54.414562+0000) 2024-03-04T16:23:55.416 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:23:55.414+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:23:55.414803+0000) 2024-03-04T16:23:56.414 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:23:56.414+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:23:56.415088+0000) 2024-03-04T16:23:57.415 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:23:57.414+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:23:57.415317+0000) 2024-03-04T16:23:58.415 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:23:58.414+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:23:58.415550+0000) 2024-03-04T16:23:59.415 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:23:59.415+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:23:59.415817+0000) 2024-03-04T16:24:00.415 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:24:00.415+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:24:00.416141+0000) 2024-03-04T16:24:01.415 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:24:01.415+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:24:01.416405+0000) 2024-03-04T16:24:02.416 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:24:02.415+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:24:02.416664+0000) 2024-03-04T16:24:03.416 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:24:03.416+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:24:03.416992+0000) 2024-03-04T16:24:04.416 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:24:04.416+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:24:04.417249+0000) 2024-03-04T16:24:05.416 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:24:05.416+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:24:05.417501+0000) 2024-03-04T16:24:06.417 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:24:06.416+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:24:06.417753+0000) 2024-03-04T16:24:07.417 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:24:07.417+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:24:07.418082+0000) 2024-03-04T16:24:08.417 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:24:08.417+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:24:08.418314+0000) 2024-03-04T16:24:09.418 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:24:09.417+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:24:09.418579+0000) 2024-03-04T16:24:10.418 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:24:10.418+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:24:10.418812+0000) 2024-03-04T16:24:11.418 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:24:11.418+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:24:11.419109+0000) 2024-03-04T16:24:12.418 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:24:12.418+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:24:12.419305+0000) 2024-03-04T16:24:13.419 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:24:13.418+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:24:13.419499+0000) 2024-03-04T16:24:14.406 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:24:14.414 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:24:14.422 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:24:14.418+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:24:14.419704+0000) 2024-03-04T16:24:15.419 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:24:15.418+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:24:15.419965+0000) 2024-03-04T16:24:16.419 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:24:16.418+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:24:16.420195+0000) 2024-03-04T16:24:17.419 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:24:17.418+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:24:17.420463+0000) 2024-03-04T16:24:18.420 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:24:18.418+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:24:18.420717+0000) 2024-03-04T16:24:19.420 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:24:19.419+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:24:19.420979+0000) 2024-03-04T16:24:20.420 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:24:20.420+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:24:20.421225+0000) 2024-03-04T16:24:21.421 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:24:21.419+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:24:21.421484+0000) 2024-03-04T16:24:22.421 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:24:22.419+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:24:22.421717+0000) 2024-03-04T16:24:23.421 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:24:23.420+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:24:23.421958+0000) 2024-03-04T16:24:24.421 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:24:24.421+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:24:24.422193+0000) 2024-03-04T16:24:25.421 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:24:25.421+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:24:25.422459+0000) 2024-03-04T16:24:26.423 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:24:26.421+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:24:26.422714+0000) 2024-03-04T16:24:27.422 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:24:27.422+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:24:27.422963+0000) 2024-03-04T16:24:28.422 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:24:28.422+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:24:28.423204+0000) 2024-03-04T16:24:29.422 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:24:29.422+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:24:29.423389+0000) 2024-03-04T16:24:30.423 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:24:30.422+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:24:30.423691+0000) 2024-03-04T16:24:31.424 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:24:31.423+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:24:31.423973+0000) 2024-03-04T16:24:32.423 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:24:32.423+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:24:32.424213+0000) 2024-03-04T16:24:33.424 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:24:33.423+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:24:33.424456+0000) 2024-03-04T16:24:34.424 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:24:34.423+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:24:34.424722+0000) 2024-03-04T16:24:35.424 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:24:35.424+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:24:35.424994+0000) 2024-03-04T16:24:36.424 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:24:36.424+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:24:36.425188+0000) 2024-03-04T16:24:37.424 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:24:37.424+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:24:37.425430+0000) 2024-03-04T16:24:38.425 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:24:38.424+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:24:38.425684+0000) 2024-03-04T16:24:39.425 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:24:39.425+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:24:39.425940+0000) 2024-03-04T16:24:40.425 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:24:40.425+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:24:40.426178+0000) 2024-03-04T16:24:41.425 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:24:41.425+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:24:41.426432+0000) 2024-03-04T16:24:42.426 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:24:42.425+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:24:42.426693+0000) 2024-03-04T16:24:43.426 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:24:43.426+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:24:43.426950+0000) 2024-03-04T16:24:44.426 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:24:44.426+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:24:44.427224+0000) 2024-03-04T16:24:44.544 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:24:44.552 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:24:45.426 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:24:45.426+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:24:45.427463+0000) 2024-03-04T16:24:46.427 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:24:46.427+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:24:46.427708+0000) 2024-03-04T16:24:47.427 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:24:47.427+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:24:47.427949+0000) 2024-03-04T16:24:48.427 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:24:48.427+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:24:48.428190+0000) 2024-03-04T16:24:49.427 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:24:49.427+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:24:49.428372+0000) 2024-03-04T16:24:50.428 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:24:50.427+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:24:50.428631+0000) 2024-03-04T16:24:51.428 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:24:51.428+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:24:51.428921+0000) 2024-03-04T16:24:52.428 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:24:52.428+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:24:52.429201+0000) 2024-03-04T16:24:53.429 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:24:53.428+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:24:53.429441+0000) 2024-03-04T16:24:54.429 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:24:54.429+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:24:54.429683+0000) 2024-03-04T16:24:55.429 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:24:55.429+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:24:55.429938+0000) 2024-03-04T16:24:56.429 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:24:56.429+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:24:56.430175+0000) 2024-03-04T16:24:57.429 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:24:57.429+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:24:57.430433+0000) 2024-03-04T16:24:58.430 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:24:58.430+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:24:58.430678+0000) 2024-03-04T16:24:59.430 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:24:59.430+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:24:59.430946+0000) 2024-03-04T16:25:00.430 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:25:00.430+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:25:00.431142+0000) 2024-03-04T16:25:01.430 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:25:01.431+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:25:01.431416+0000) 2024-03-04T16:25:02.431 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:25:02.431+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:25:02.431693+0000) 2024-03-04T16:25:03.431 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:25:03.431+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:25:03.431951+0000) 2024-03-04T16:25:04.431 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:25:04.431+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:25:04.432187+0000) 2024-03-04T16:25:05.431 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:25:05.431+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:25:05.432436+0000) 2024-03-04T16:25:06.432 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:25:06.432+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:25:06.432675+0000) 2024-03-04T16:25:07.432 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:25:07.432+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:25:07.432945+0000) 2024-03-04T16:25:08.432 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:25:08.432+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:25:08.433195+0000) 2024-03-04T16:25:09.432 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:25:09.433+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:25:09.433436+0000) 2024-03-04T16:25:10.433 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:25:10.433+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:25:10.433672+0000) 2024-03-04T16:25:11.433 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:25:11.433+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:25:11.433954+0000) 2024-03-04T16:25:12.433 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:25:12.433+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:25:12.434192+0000) 2024-03-04T16:25:13.434 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:25:13.434+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:25:13.434433+0000) 2024-03-04T16:25:14.434 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:25:14.434+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:25:14.434674+0000) 2024-03-04T16:25:14.748 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:25:14.757 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:25:15.434 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:25:15.434+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:25:15.434954+0000) 2024-03-04T16:25:16.434 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:25:16.435+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:25:16.435199+0000) 2024-03-04T16:25:17.434 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:25:17.435+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:25:17.435404+0000) 2024-03-04T16:25:18.435 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:25:18.435+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:25:18.435646+0000) 2024-03-04T16:25:19.435 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:25:19.434+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:25:19.435931+0000) 2024-03-04T16:25:20.435 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:25:20.435+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:25:20.436170+0000) 2024-03-04T16:25:21.435 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:25:21.435+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:25:21.436414+0000) 2024-03-04T16:25:22.436 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:25:22.435+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:25:22.436654+0000) 2024-03-04T16:25:23.436 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:25:23.435+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:25:23.436939+0000) 2024-03-04T16:25:24.436 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:25:24.436+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:25:24.437186+0000) 2024-03-04T16:25:25.436 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:25:25.435+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:25:25.437440+0000) 2024-03-04T16:25:26.437 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:25:26.436+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:25:26.437683+0000) 2024-03-04T16:25:27.437 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:25:27.436+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:25:27.437955+0000) 2024-03-04T16:25:28.437 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:25:28.436+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:25:28.438138+0000) 2024-03-04T16:25:29.437 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:25:29.437+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:25:29.438382+0000) 2024-03-04T16:25:30.438 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:25:30.437+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:25:30.438640+0000) 2024-03-04T16:25:31.438 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:25:31.438+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:25:31.438951+0000) 2024-03-04T16:25:32.439 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:25:32.438+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:25:32.439180+0000) 2024-03-04T16:25:33.439 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:25:33.438+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:25:33.439386+0000) 2024-03-04T16:25:34.439 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:25:34.438+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:25:34.439662+0000) 2024-03-04T16:25:35.439 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:25:35.439+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:25:35.439936+0000) 2024-03-04T16:25:36.439 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:25:36.439+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:25:36.440170+0000) 2024-03-04T16:25:37.439 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:25:37.439+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:25:37.440400+0000) 2024-03-04T16:25:38.440 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:25:38.439+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:25:38.440638+0000) 2024-03-04T16:25:39.440 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:25:39.440+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:25:39.440928+0000) 2024-03-04T16:25:40.440 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:25:40.440+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:25:40.441159+0000) 2024-03-04T16:25:41.440 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:25:41.440+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:25:41.441350+0000) 2024-03-04T16:25:42.441 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:25:42.440+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:25:42.441615+0000) 2024-03-04T16:25:43.441 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:25:43.441+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:25:43.441927+0000) 2024-03-04T16:25:44.441 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:25:44.441+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:25:44.442169+0000) 2024-03-04T16:25:44.887 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:25:44.895 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:25:45.441 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:25:45.441+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:25:45.442456+0000) 2024-03-04T16:25:46.442 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:25:46.441+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:25:46.442697+0000) 2024-03-04T16:25:47.442 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:25:47.442+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:25:47.442939+0000) 2024-03-04T16:25:48.442 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:25:48.442+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:25:48.443178+0000) 2024-03-04T16:25:49.442 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:25:49.442+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:25:49.443413+0000) 2024-03-04T16:25:50.443 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:25:50.443+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:25:50.443674+0000) 2024-03-04T16:25:51.443 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:25:51.443+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:25:51.443925+0000) 2024-03-04T16:25:52.443 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:25:52.443+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:25:52.444171+0000) 2024-03-04T16:25:53.443 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:25:53.443+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:25:53.444400+0000) 2024-03-04T16:25:54.444 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:25:54.444+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:25:54.444639+0000) 2024-03-04T16:25:55.444 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:25:55.444+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:25:55.444926+0000) 2024-03-04T16:25:56.444 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:25:56.444+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:25:56.445109+0000) 2024-03-04T16:25:57.444 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:25:57.444+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:25:57.445348+0000) 2024-03-04T16:25:58.445 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:25:58.445+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:25:58.445585+0000) 2024-03-04T16:25:59.445 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:25:59.445+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:25:59.445862+0000) 2024-03-04T16:26:00.445 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:26:00.445+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:26:00.446124+0000) 2024-03-04T16:26:01.445 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:26:01.445+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:26:01.446400+0000) 2024-03-04T16:26:02.446 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:26:02.446+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:26:02.446644+0000) 2024-03-04T16:26:03.446 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:26:03.446+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:26:03.446931+0000) 2024-03-04T16:26:04.446 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:26:04.446+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:26:04.447168+0000) 2024-03-04T16:26:05.446 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:26:05.446+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:26:05.447404+0000) 2024-03-04T16:26:06.447 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:26:06.446+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:26:06.447659+0000) 2024-03-04T16:26:07.447 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:26:07.447+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:26:07.447938+0000) 2024-03-04T16:26:08.447 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:26:08.447+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:26:08.448179+0000) 2024-03-04T16:26:09.448 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:26:09.448+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:26:09.448412+0000) 2024-03-04T16:26:10.448 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:26:10.448+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:26:10.448596+0000) 2024-03-04T16:26:11.448 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:26:11.448+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:26:11.448904+0000) 2024-03-04T16:26:12.448 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:26:12.448+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:26:12.449150+0000) 2024-03-04T16:26:13.448 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:26:13.449+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:26:13.449388+0000) 2024-03-04T16:26:14.449 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:26:14.449+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:26:14.449620+0000) 2024-03-04T16:26:15.217 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:26:15.226 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:26:15.449 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:26:15.449+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:26:15.449890+0000) 2024-03-04T16:26:16.449 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:26:16.449+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:26:16.450130+0000) 2024-03-04T16:26:17.449 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:26:17.450+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:26:17.450370+0000) 2024-03-04T16:26:18.450 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:26:18.450+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:26:18.450615+0000) 2024-03-04T16:26:19.450 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:26:19.450+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:26:19.450917+0000) 2024-03-04T16:26:20.450 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:26:20.450+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:26:20.451153+0000) 2024-03-04T16:26:21.450 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:26:21.451+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:26:21.451398+0000) 2024-03-04T16:26:22.451 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:26:22.451+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:26:22.451677+0000) 2024-03-04T16:26:23.451 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:26:23.451+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:26:23.451958+0000) 2024-03-04T16:26:24.451 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:26:24.451+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:26:24.452197+0000) 2024-03-04T16:26:25.451 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:26:25.450+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:26:25.452425+0000) 2024-03-04T16:26:26.452 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:26:26.451+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:26:26.452686+0000) 2024-03-04T16:26:27.452 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:26:27.451+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:26:27.452926+0000) 2024-03-04T16:26:28.452 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:26:28.452+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:26:28.453164+0000) 2024-03-04T16:26:29.452 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:26:29.452+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:26:29.453402+0000) 2024-03-04T16:26:30.453 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:26:30.452+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:26:30.453657+0000) 2024-03-04T16:26:31.453 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:26:31.452+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:26:31.453928+0000) 2024-03-04T16:26:32.453 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:26:32.453+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:26:32.454165+0000) 2024-03-04T16:26:33.453 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:26:33.453+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:26:33.454411+0000) 2024-03-04T16:26:34.454 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:26:34.453+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:26:34.454665+0000) 2024-03-04T16:26:35.454 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:26:35.454+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:26:35.454951+0000) 2024-03-04T16:26:36.454 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:26:36.454+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:26:36.455189+0000) 2024-03-04T16:26:37.454 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:26:37.454+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:26:37.455429+0000) 2024-03-04T16:26:38.455 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:26:38.454+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:26:38.455743+0000) 2024-03-04T16:26:39.455 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:26:39.455+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:26:39.456009+0000) 2024-03-04T16:26:40.455 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:26:40.455+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:26:40.456245+0000) 2024-03-04T16:26:41.455 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:26:41.455+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:26:41.456491+0000) 2024-03-04T16:26:42.456 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:26:42.455+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:26:42.456737+0000) 2024-03-04T16:26:43.456 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:26:43.456+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:26:43.456977+0000) 2024-03-04T16:26:44.457 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:26:44.456+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:26:44.457219+0000) 2024-03-04T16:26:45.438 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:26:45.448 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:26:45.458 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:26:45.456+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:26:45.457409+0000) 2024-03-04T16:26:46.457 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:26:46.456+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:26:46.457648+0000) 2024-03-04T16:26:47.457 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:26:47.457+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:26:47.457954+0000) 2024-03-04T16:26:48.457 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:26:48.457+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:26:48.458148+0000) 2024-03-04T16:26:49.457 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:26:49.457+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:26:49.458389+0000) 2024-03-04T16:26:50.458 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:26:50.457+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:26:50.458649+0000) 2024-03-04T16:26:51.458 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:26:51.458+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:26:51.458939+0000) 2024-03-04T16:26:52.458 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:26:52.458+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:26:52.459205+0000) 2024-03-04T16:26:53.459 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:26:53.458+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:26:53.459486+0000) 2024-03-04T16:26:54.459 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:26:54.459+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:26:54.459740+0000) 2024-03-04T16:26:55.459 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:26:55.459+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:26:55.459978+0000) 2024-03-04T16:26:56.459 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:26:56.459+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:26:56.460250+0000) 2024-03-04T16:26:57.460 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:26:57.459+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:26:57.460544+0000) 2024-03-04T16:26:58.460 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:26:58.460+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:26:58.460872+0000) 2024-03-04T16:26:59.460 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:26:59.460+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:26:59.461133+0000) 2024-03-04T16:27:00.460 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:27:00.460+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:27:00.461398+0000) 2024-03-04T16:27:01.461 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:27:01.461+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:27:01.461668+0000) 2024-03-04T16:27:02.461 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:27:02.461+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:27:02.461938+0000) 2024-03-04T16:27:03.461 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:27:03.461+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:27:03.462178+0000) 2024-03-04T16:27:04.461 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:27:04.461+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:27:04.462408+0000) 2024-03-04T16:27:05.462 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:27:05.462+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:27:05.462648+0000) 2024-03-04T16:27:06.463 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:27:06.462+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:27:06.462889+0000) 2024-03-04T16:27:07.462 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:27:07.462+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:27:07.463127+0000) 2024-03-04T16:27:08.462 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:27:08.462+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:27:08.463321+0000) 2024-03-04T16:27:09.462 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:27:09.463+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:27:09.463564+0000) 2024-03-04T16:27:10.463 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:27:10.463+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:27:10.463802+0000) 2024-03-04T16:27:11.463 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:27:11.463+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:27:11.464113+0000) 2024-03-04T16:27:12.463 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:27:12.463+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:27:12.464353+0000) 2024-03-04T16:27:13.464 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:27:13.464+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:27:13.464597+0000) 2024-03-04T16:27:14.464 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:27:14.463+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:27:14.464776+0000) 2024-03-04T16:27:15.464 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:27:15.464+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:27:15.465008+0000) 2024-03-04T16:27:15.600 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:27:15.608 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:27:16.464 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:27:16.464+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:27:16.465250+0000) 2024-03-04T16:27:17.464 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:27:17.465+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:27:17.465488+0000) 2024-03-04T16:27:18.465 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:27:18.465+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:27:18.465733+0000) 2024-03-04T16:27:19.465 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:27:19.465+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:27:19.465979+0000) 2024-03-04T16:27:20.465 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:27:20.465+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:27:20.466220+0000) 2024-03-04T16:27:21.465 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:27:21.466+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:27:21.466467+0000) 2024-03-04T16:27:22.466 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:27:22.466+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:27:22.466702+0000) 2024-03-04T16:27:23.466 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:27:23.466+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:27:23.466944+0000) 2024-03-04T16:27:24.466 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:27:24.466+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:27:24.467186+0000) 2024-03-04T16:27:25.467 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:27:25.467+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:27:25.467421+0000) 2024-03-04T16:27:26.467 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:27:26.467+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:27:26.467704+0000) 2024-03-04T16:27:27.467 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:27:27.467+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:27:27.467941+0000) 2024-03-04T16:27:28.467 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:27:28.468+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:27:28.468202+0000) 2024-03-04T16:27:29.467 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:27:29.467+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:27:29.468438+0000) 2024-03-04T16:27:30.468 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:27:30.466+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:27:30.468678+0000) 2024-03-04T16:27:31.468 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:27:31.467+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:27:31.468947+0000) 2024-03-04T16:27:32.468 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:27:32.468+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:27:32.469196+0000) 2024-03-04T16:27:33.468 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:27:33.468+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:27:33.469446+0000) 2024-03-04T16:27:34.469 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:27:34.468+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:27:34.469683+0000) 2024-03-04T16:27:35.469 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:27:35.468+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:27:35.469910+0000) 2024-03-04T16:27:36.470 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:27:36.469+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:27:36.470152+0000) 2024-03-04T16:27:37.469 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:27:37.469+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:27:37.470392+0000) 2024-03-04T16:27:38.470 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:27:38.469+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:27:38.470633+0000) 2024-03-04T16:27:39.470 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:27:39.469+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:27:39.470920+0000) 2024-03-04T16:27:40.470 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:27:40.470+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:27:40.471160+0000) 2024-03-04T16:27:41.470 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:27:41.470+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:27:41.471356+0000) 2024-03-04T16:27:42.470 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:27:42.470+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:27:42.471595+0000) 2024-03-04T16:27:43.471 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:27:43.471+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:27:43.471874+0000) 2024-03-04T16:27:44.471 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:27:44.471+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:27:44.472116+0000) 2024-03-04T16:27:45.471 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:27:45.471+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:27:45.472351+0000) 2024-03-04T16:27:45.823 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:27:45.833 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:27:46.471 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:27:46.471+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:27:46.472592+0000) 2024-03-04T16:27:47.472 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:27:47.471+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:27:47.472778+0000) 2024-03-04T16:27:48.472 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:27:48.472+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:27:48.473018+0000) 2024-03-04T16:27:49.472 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:27:49.472+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:27:49.473254+0000) 2024-03-04T16:27:50.472 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:27:50.472+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:27:50.473515+0000) 2024-03-04T16:27:51.473 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:27:51.472+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:27:51.473761+0000) 2024-03-04T16:27:52.473 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:27:52.473+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:27:52.473999+0000) 2024-03-04T16:27:53.473 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:27:53.473+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:27:53.474243+0000) 2024-03-04T16:27:54.473 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:27:54.473+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:27:54.474486+0000) 2024-03-04T16:27:55.474 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:27:55.474+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:27:55.474732+0000) 2024-03-04T16:27:56.474 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:27:56.474+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:27:56.475058+0000) 2024-03-04T16:27:57.474 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:27:57.474+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:27:57.475316+0000) 2024-03-04T16:27:58.475 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:27:58.474+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:27:58.475561+0000) 2024-03-04T16:27:59.475 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:27:59.474+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:27:59.475793+0000) 2024-03-04T16:28:00.475 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:28:00.475+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:28:00.476084+0000) 2024-03-04T16:28:01.475 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:28:01.475+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:28:01.476279+0000) 2024-03-04T16:28:02.475 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:28:02.475+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:28:02.476539+0000) 2024-03-04T16:28:03.476 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:28:03.475+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:28:03.476796+0000) 2024-03-04T16:28:04.476 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:28:04.476+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:28:04.477085+0000) 2024-03-04T16:28:05.476 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:28:05.476+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:28:05.477328+0000) 2024-03-04T16:28:06.477 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:28:06.477+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:28:06.477570+0000) 2024-03-04T16:28:07.477 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:28:07.477+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:28:07.477807+0000) 2024-03-04T16:28:08.477 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:28:08.477+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:28:08.478094+0000) 2024-03-04T16:28:09.477 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:28:09.477+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:28:09.478391+0000) 2024-03-04T16:28:10.480 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:28:10.478+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:28:10.478633+0000) 2024-03-04T16:28:11.478 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:28:11.478+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:28:11.478950+0000) 2024-03-04T16:28:12.478 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:28:12.478+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:28:12.479158+0000) 2024-03-04T16:28:13.478 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:28:13.478+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:28:13.479396+0000) 2024-03-04T16:28:14.479 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:28:14.478+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:28:14.479646+0000) 2024-03-04T16:28:15.479 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:28:15.479+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:28:15.479932+0000) 2024-03-04T16:28:15.969 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:28:15.978 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:28:16.479 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:28:16.479+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:28:16.480180+0000) 2024-03-04T16:28:17.479 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:28:17.479+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:28:17.480382+0000) 2024-03-04T16:28:18.480 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:28:18.480+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:28:18.480620+0000) 2024-03-04T16:28:19.480 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:28:19.480+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:28:19.480906+0000) 2024-03-04T16:28:20.480 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:28:20.480+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:28:20.481172+0000) 2024-03-04T16:28:21.480 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:28:21.481+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:28:21.481413+0000) 2024-03-04T16:28:22.481 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:28:22.481+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:28:22.481648+0000) 2024-03-04T16:28:23.481 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:28:23.481+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:28:23.481928+0000) 2024-03-04T16:28:24.481 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:28:24.481+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:28:24.482168+0000) 2024-03-04T16:28:25.481 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:28:25.482+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:28:25.482402+0000) 2024-03-04T16:28:26.482 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:28:26.482+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:28:26.482646+0000) 2024-03-04T16:28:27.482 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:28:27.482+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:28:27.482921+0000) 2024-03-04T16:28:28.482 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:28:28.482+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:28:28.483162+0000) 2024-03-04T16:28:29.483 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:28:29.482+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:28:29.483346+0000) 2024-03-04T16:28:30.482 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:28:30.482+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:28:30.483540+0000) 2024-03-04T16:28:31.483 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:28:31.483+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:28:31.483784+0000) 2024-03-04T16:28:32.483 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:28:32.483+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:28:32.484039+0000) 2024-03-04T16:28:33.484 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:28:33.484+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:28:33.484297+0000) 2024-03-04T16:28:34.483 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:28:34.483+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:28:34.484542+0000) 2024-03-04T16:28:35.484 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:28:35.483+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:28:35.484781+0000) 2024-03-04T16:28:36.484 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:28:36.483+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:28:36.485027+0000) 2024-03-04T16:28:37.484 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:28:37.484+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:28:37.485261+0000) 2024-03-04T16:28:38.484 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:28:38.484+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:28:38.485500+0000) 2024-03-04T16:28:39.485 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:28:39.484+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:28:39.485737+0000) 2024-03-04T16:28:40.485 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:28:40.484+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:28:40.485975+0000) 2024-03-04T16:28:41.485 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:28:41.485+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:28:41.486242+0000) 2024-03-04T16:28:42.485 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:28:42.485+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:28:42.486479+0000) 2024-03-04T16:28:43.486 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:28:43.484+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:28:43.486712+0000) 2024-03-04T16:28:44.486 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:28:44.486+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:28:44.486956+0000) 2024-03-04T16:28:45.487 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:28:45.486+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:28:45.487191+0000) 2024-03-04T16:28:46.142 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:28:46.150 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:28:46.486 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:28:46.486+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:28:46.487434+0000) 2024-03-04T16:28:47.488 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:28:47.486+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:28:47.487677+0000) 2024-03-04T16:28:48.487 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:28:48.487+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:28:48.487948+0000) 2024-03-04T16:28:49.487 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:28:49.487+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:28:49.488135+0000) 2024-03-04T16:28:50.487 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:28:50.487+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:28:50.488316+0000) 2024-03-04T16:28:51.488 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:28:51.487+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:28:51.488563+0000) 2024-03-04T16:28:52.488 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:28:52.487+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:28:52.488802+0000) 2024-03-04T16:28:53.489 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:28:53.488+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:28:53.489080+0000) 2024-03-04T16:28:54.488 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:28:54.488+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:28:54.489323+0000) 2024-03-04T16:28:55.489 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:28:55.488+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:28:55.489563+0000) 2024-03-04T16:28:56.489 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:28:56.489+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:28:56.489816+0000) 2024-03-04T16:28:57.489 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:28:57.489+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:28:57.490095+0000) 2024-03-04T16:28:58.490 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:28:58.489+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:28:58.490286+0000) 2024-03-04T16:28:59.489 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:28:59.489+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:28:59.490527+0000) 2024-03-04T16:29:00.490 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:29:00.489+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:29:00.490761+0000) 2024-03-04T16:29:01.490 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:29:01.490+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:29:01.491009+0000) 2024-03-04T16:29:02.491 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:29:02.490+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:29:02.491302+0000) 2024-03-04T16:29:03.491 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:29:03.490+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:29:03.491574+0000) 2024-03-04T16:29:04.491 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:29:04.491+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:29:04.491855+0000) 2024-03-04T16:29:05.491 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:29:05.491+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:29:05.492109+0000) 2024-03-04T16:29:06.491 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:29:06.491+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:29:06.492350+0000) 2024-03-04T16:29:07.492 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:29:07.491+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:29:07.492587+0000) 2024-03-04T16:29:08.492 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:29:08.491+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:29:08.492800+0000) 2024-03-04T16:29:09.492 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:29:09.492+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:29:09.493093+0000) 2024-03-04T16:29:10.492 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:29:10.492+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:29:10.493332+0000) 2024-03-04T16:29:11.493 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:29:11.493+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:29:11.493580+0000) 2024-03-04T16:29:12.493 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:29:12.493+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:29:12.493853+0000) 2024-03-04T16:29:13.493 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:29:13.493+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:29:13.494055+0000) 2024-03-04T16:29:14.494 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:29:14.493+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:29:14.494248+0000) 2024-03-04T16:29:15.493 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:29:15.494+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:29:15.494489+0000) 2024-03-04T16:29:16.288 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:29:16.297 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:29:16.494 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:29:16.493+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:29:16.494674+0000) 2024-03-04T16:29:17.494 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:29:17.494+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:29:17.494948+0000) 2024-03-04T16:29:18.494 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:29:18.494+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:29:18.495189+0000) 2024-03-04T16:29:19.494 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:29:19.495+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:29:19.495447+0000) 2024-03-04T16:29:20.494 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:29:20.494+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:29:20.495703+0000) 2024-03-04T16:29:21.494 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:29:21.495+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:29:21.495949+0000) 2024-03-04T16:29:22.495 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:29:22.495+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:29:22.496226+0000) 2024-03-04T16:29:23.495 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:29:23.496+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:29:23.496520+0000) 2024-03-04T16:29:24.495 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:29:24.496+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:29:24.496778+0000) 2024-03-04T16:29:25.495 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:29:25.496+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:29:25.496990+0000) 2024-03-04T16:29:26.496 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:29:26.496+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:29:26.497225+0000) 2024-03-04T16:29:27.496 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:29:27.497+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:29:27.497459+0000) 2024-03-04T16:29:28.496 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:29:28.497+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:29:28.497705+0000) 2024-03-04T16:29:29.496 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:29:29.497+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:29:29.497945+0000) 2024-03-04T16:29:30.496 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:29:30.497+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:29:30.498185+0000) 2024-03-04T16:29:31.497 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:29:31.498+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:29:31.498431+0000) 2024-03-04T16:29:32.497 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:29:32.497+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:29:32.498711+0000) 2024-03-04T16:29:33.497 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:29:33.498+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:29:33.498989+0000) 2024-03-04T16:29:34.497 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:29:34.499+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:29:34.499253+0000) 2024-03-04T16:29:35.498 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:29:35.499+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:29:35.499490+0000) 2024-03-04T16:29:36.498 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:29:36.499+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:29:36.499726+0000) 2024-03-04T16:29:37.498 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:29:37.499+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:29:37.499961+0000) 2024-03-04T16:29:38.503 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:29:38.500+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:29:38.500200+0000) 2024-03-04T16:29:39.499 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:29:39.499+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:29:39.500442+0000) 2024-03-04T16:29:40.499 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:29:40.499+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:29:40.500625+0000) 2024-03-04T16:29:41.499 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:29:41.499+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:29:41.500936+0000) 2024-03-04T16:29:42.499 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:29:42.500+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:29:42.501199+0000) 2024-03-04T16:29:43.499 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:29:43.500+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:29:43.501436+0000) 2024-03-04T16:29:44.499 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:29:44.500+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:29:44.501677+0000) 2024-03-04T16:29:45.500 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:29:45.500+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:29:45.501950+0000) 2024-03-04T16:29:46.500 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:29:46.500+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:29:46.502148+0000) 2024-03-04T16:29:46.526 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:29:46.534 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:29:47.500 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:29:47.501+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:29:47.502387+0000) 2024-03-04T16:29:48.500 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:29:48.501+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:29:48.502625+0000) 2024-03-04T16:29:49.500 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:29:49.501+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:29:49.502914+0000) 2024-03-04T16:29:50.501 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:29:50.502+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:29:50.503162+0000) 2024-03-04T16:29:51.501 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:29:51.502+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:29:51.503408+0000) 2024-03-04T16:29:52.505 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:29:52.502+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:29:52.503667+0000) 2024-03-04T16:29:53.502 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:29:53.503+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:29:53.503938+0000) 2024-03-04T16:29:54.502 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:29:54.503+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:29:54.504172+0000) 2024-03-04T16:29:55.504 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:29:55.503+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:29:55.504406+0000) 2024-03-04T16:29:56.502 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:29:56.503+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:29:56.504706+0000) 2024-03-04T16:29:57.502 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:29:57.504+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:29:57.504943+0000) 2024-03-04T16:29:58.502 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:29:58.504+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:29:58.505182+0000) 2024-03-04T16:29:59.503 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:29:59.504+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:29:59.505422+0000) 2024-03-04T16:30:00.503 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:30:00.504+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:30:00.505670+0000) 2024-03-04T16:30:01.503 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:30:01.505+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:30:01.505955+0000) 2024-03-04T16:30:02.503 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:30:02.505+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:30:02.506151+0000) 2024-03-04T16:30:03.503 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:30:03.505+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:30:03.506393+0000) 2024-03-04T16:30:04.505 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:30:04.505+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:30:04.506655+0000) 2024-03-04T16:30:05.504 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:30:05.506+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:30:05.506950+0000) 2024-03-04T16:30:06.504 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:30:06.506+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:30:06.507198+0000) 2024-03-04T16:30:07.505 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:30:07.506+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:30:07.507436+0000) 2024-03-04T16:30:08.505 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:30:08.506+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:30:08.507694+0000) 2024-03-04T16:30:09.505 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:30:09.507+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:30:09.507940+0000) 2024-03-04T16:30:10.505 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:30:10.507+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:30:10.508132+0000) 2024-03-04T16:30:11.505 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:30:11.507+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:30:11.508381+0000) 2024-03-04T16:30:12.506 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:30:12.508+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:30:12.508627+0000) 2024-03-04T16:30:13.506 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:30:13.508+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:30:13.508918+0000) 2024-03-04T16:30:14.506 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:30:14.508+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:30:14.509194+0000) 2024-03-04T16:30:15.510 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:30:15.508+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:30:15.509432+0000) 2024-03-04T16:30:16.506 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:30:16.508+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:30:16.509696+0000) 2024-03-04T16:30:16.680 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:30:16.688 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:30:17.507 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:30:17.509+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:30:17.509943+0000) 2024-03-04T16:30:18.507 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:30:18.509+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:30:18.510182+0000) 2024-03-04T16:30:19.507 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:30:19.509+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:30:19.510423+0000) 2024-03-04T16:30:20.508 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:30:20.510+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:30:20.510668+0000) 2024-03-04T16:30:21.508 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:30:21.510+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:30:21.510962+0000) 2024-03-04T16:30:22.508 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:30:22.510+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:30:22.511241+0000) 2024-03-04T16:30:23.508 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:30:23.511+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:30:23.511484+0000) 2024-03-04T16:30:24.508 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:30:24.511+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:30:24.511730+0000) 2024-03-04T16:30:25.508 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:30:25.511+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:30:25.511968+0000) 2024-03-04T16:30:26.509 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:30:26.511+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:30:26.512206+0000) 2024-03-04T16:30:27.509 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:30:27.512+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:30:27.512443+0000) 2024-03-04T16:30:28.509 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:30:28.512+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:30:28.512678+0000) 2024-03-04T16:30:29.509 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:30:29.512+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:30:29.512954+0000) 2024-03-04T16:30:30.510 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:30:30.512+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:30:30.513208+0000) 2024-03-04T16:30:31.510 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:30:31.513+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:30:31.513491+0000) 2024-03-04T16:30:32.510 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:30:32.513+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:30:32.513730+0000) 2024-03-04T16:30:33.510 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:30:33.513+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:30:33.513972+0000) 2024-03-04T16:30:34.510 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:30:34.513+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:30:34.514232+0000) 2024-03-04T16:30:35.511 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:30:35.513+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:30:35.514488+0000) 2024-03-04T16:30:36.511 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:30:36.513+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:30:36.514683+0000) 2024-03-04T16:30:37.511 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:30:37.514+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:30:37.514958+0000) 2024-03-04T16:30:38.511 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:30:38.514+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:30:38.515153+0000) 2024-03-04T16:30:39.511 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:30:39.515+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:30:39.515399+0000) 2024-03-04T16:30:40.512 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:30:40.515+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:30:40.515640+0000) 2024-03-04T16:30:41.512 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:30:41.515+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:30:41.515932+0000) 2024-03-04T16:30:42.512 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:30:42.516+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:30:42.516175+0000) 2024-03-04T16:30:43.512 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:30:43.516+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:30:43.516420+0000) 2024-03-04T16:30:44.513 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:30:44.514+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:30:44.516680+0000) 2024-03-04T16:30:45.513 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:30:45.515+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:30:45.516956+0000) 2024-03-04T16:30:46.513 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:30:46.516+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:30:46.517200+0000) 2024-03-04T16:30:46.884 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:30:46.892 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:30:47.514 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:30:47.516+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:30:47.517439+0000) 2024-03-04T16:30:48.513 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:30:48.515+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:30:48.517679+0000) 2024-03-04T16:30:49.514 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:30:49.516+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:30:49.518009+0000) 2024-03-04T16:30:50.514 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:30:50.517+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:30:50.518248+0000) 2024-03-04T16:30:51.514 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:30:51.517+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:30:51.518498+0000) 2024-03-04T16:30:52.514 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:30:52.517+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:30:52.518738+0000) 2024-03-04T16:30:53.515 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:30:53.518+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:30:53.519007+0000) 2024-03-04T16:30:54.515 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:30:54.518+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:30:54.519205+0000) 2024-03-04T16:30:55.515 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:30:55.518+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:30:55.519439+0000) 2024-03-04T16:30:56.515 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:30:56.518+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:30:56.519676+0000) 2024-03-04T16:30:57.515 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:30:57.519+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:30:57.519949+0000) 2024-03-04T16:30:58.516 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:30:58.519+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:30:58.520231+0000) 2024-03-04T16:30:59.516 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:30:59.519+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:30:59.520486+0000) 2024-03-04T16:31:00.516 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:31:00.519+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:31:00.520732+0000) 2024-03-04T16:31:01.516 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:31:01.520+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:31:01.521002+0000) 2024-03-04T16:31:02.517 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:31:02.520+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:31:02.521239+0000) 2024-03-04T16:31:03.517 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:31:03.520+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:31:03.521478+0000) 2024-03-04T16:31:04.517 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:31:04.520+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:31:04.521717+0000) 2024-03-04T16:31:05.517 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:31:05.521+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:31:05.521976+0000) 2024-03-04T16:31:06.517 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:31:06.521+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:31:06.522238+0000) 2024-03-04T16:31:07.518 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:31:07.521+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:31:07.522478+0000) 2024-03-04T16:31:08.518 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:31:08.521+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:31:08.522721+0000) 2024-03-04T16:31:09.518 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:31:09.522+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:31:09.522966+0000) 2024-03-04T16:31:10.518 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:31:10.522+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:31:10.523208+0000) 2024-03-04T16:31:11.519 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:31:11.522+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:31:11.523455+0000) 2024-03-04T16:31:12.519 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:31:12.522+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:31:12.523687+0000) 2024-03-04T16:31:13.519 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:31:13.523+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:31:13.523929+0000) 2024-03-04T16:31:14.519 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:31:14.523+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:31:14.524168+0000) 2024-03-04T16:31:15.519 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:31:15.523+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:31:15.524414+0000) 2024-03-04T16:31:16.520 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:31:16.523+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:31:16.524668+0000) 2024-03-04T16:31:17.028 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:31:17.036 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:31:17.520 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:31:17.524+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:31:17.524938+0000) 2024-03-04T16:31:18.520 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:31:18.524+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:31:18.525202+0000) 2024-03-04T16:31:19.520 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:31:19.524+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:31:19.525442+0000) 2024-03-04T16:31:20.520 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:31:20.524+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:31:20.525696+0000) 2024-03-04T16:31:21.521 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:31:21.525+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:31:21.525945+0000) 2024-03-04T16:31:22.521 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:31:22.525+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:31:22.526175+0000) 2024-03-04T16:31:23.521 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:31:23.525+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:31:23.526460+0000) 2024-03-04T16:31:24.521 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:31:24.525+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:31:24.526659+0000) 2024-03-04T16:31:25.522 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:31:25.526+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:31:25.526958+0000) 2024-03-04T16:31:26.522 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:31:26.526+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:31:26.527198+0000) 2024-03-04T16:31:27.522 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:31:27.526+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:31:27.527439+0000) 2024-03-04T16:31:28.522 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:31:28.527+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:31:28.527688+0000) 2024-03-04T16:31:29.522 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:31:29.527+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:31:29.527928+0000) 2024-03-04T16:31:30.523 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:31:30.527+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:31:30.528163+0000) 2024-03-04T16:31:31.523 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:31:31.528+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:31:31.528415+0000) 2024-03-04T16:31:32.523 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:31:32.527+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:31:32.528700+0000) 2024-03-04T16:31:33.523 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:31:33.528+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:31:33.528952+0000) 2024-03-04T16:31:34.524 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:31:34.528+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:31:34.529233+0000) 2024-03-04T16:31:35.524 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:31:35.529+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:31:35.529508+0000) 2024-03-04T16:31:36.524 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:31:36.529+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:31:36.529745+0000) 2024-03-04T16:31:37.524 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:31:37.529+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:31:37.529942+0000) 2024-03-04T16:31:38.524 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:31:38.529+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:31:38.530178+0000) 2024-03-04T16:31:39.525 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:31:39.530+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:31:39.530414+0000) 2024-03-04T16:31:40.525 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:31:40.530+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:31:40.530652+0000) 2024-03-04T16:31:41.528 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:31:41.530+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:31:41.530940+0000) 2024-03-04T16:31:42.525 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:31:42.530+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:31:42.531244+0000) 2024-03-04T16:31:43.526 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:31:43.531+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:31:43.531529+0000) 2024-03-04T16:31:44.526 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:31:44.531+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:31:44.531815+0000) 2024-03-04T16:31:45.526 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:31:45.531+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:31:45.532176+0000) 2024-03-04T16:31:46.526 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:31:46.532+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:31:46.532464+0000) 2024-03-04T16:31:47.250 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:31:47.265 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:31:47.529 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:31:47.531+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:31:47.532759+0000) 2024-03-04T16:31:48.527 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:31:48.532+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:31:48.533022+0000) 2024-03-04T16:31:49.528 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:31:49.531+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:31:49.533231+0000) 2024-03-04T16:31:50.527 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:31:50.531+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:31:50.533420+0000) 2024-03-04T16:31:51.528 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:31:51.532+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:31:51.533670+0000) 2024-03-04T16:31:52.528 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:31:52.532+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:31:52.533946+0000) 2024-03-04T16:31:53.528 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:31:53.533+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:31:53.534184+0000) 2024-03-04T16:31:54.528 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:31:54.533+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:31:54.534422+0000) 2024-03-04T16:31:55.528 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:31:55.533+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:31:55.534685+0000) 2024-03-04T16:31:56.529 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:31:56.533+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:31:56.534938+0000) 2024-03-04T16:31:57.529 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:31:57.534+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:31:57.535189+0000) 2024-03-04T16:31:58.529 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:31:58.534+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:31:58.535426+0000) 2024-03-04T16:31:59.529 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:31:59.534+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:31:59.535704+0000) 2024-03-04T16:32:00.530 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:32:00.535+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:32:00.535945+0000) 2024-03-04T16:32:01.530 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:32:01.535+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:32:01.536191+0000) 2024-03-04T16:32:02.530 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:32:02.535+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:32:02.536431+0000) 2024-03-04T16:32:03.530 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:32:03.535+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:32:03.536673+0000) 2024-03-04T16:32:04.530 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:32:04.536+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:32:04.536935+0000) 2024-03-04T16:32:05.531 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:32:05.536+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:32:05.537170+0000) 2024-03-04T16:32:06.531 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:32:06.536+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:32:06.537408+0000) 2024-03-04T16:32:07.531 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:32:07.536+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:32:07.537647+0000) 2024-03-04T16:32:08.532 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:32:08.537+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:32:08.537924+0000) 2024-03-04T16:32:09.532 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:32:09.537+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:32:09.538172+0000) 2024-03-04T16:32:10.532 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:32:10.537+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:32:10.538366+0000) 2024-03-04T16:32:11.532 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:32:11.537+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:32:11.538612+0000) 2024-03-04T16:32:12.532 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:32:12.538+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:32:12.538913+0000) 2024-03-04T16:32:13.532 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:32:13.538+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:32:13.539177+0000) 2024-03-04T16:32:14.533 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:32:14.538+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:32:14.539416+0000) 2024-03-04T16:32:15.533 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:32:15.538+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:32:15.539655+0000) 2024-03-04T16:32:16.533 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:32:16.539+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:32:16.539954+0000) 2024-03-04T16:32:17.480 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:32:17.489 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:32:17.533 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:32:17.539+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:32:17.540123+0000) 2024-03-04T16:32:18.533 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:32:18.539+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:32:18.540389+0000) 2024-03-04T16:32:19.534 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:32:19.539+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:32:19.540648+0000) 2024-03-04T16:32:20.534 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:32:20.540+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:32:20.540953+0000) 2024-03-04T16:32:21.534 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:32:21.540+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:32:21.541200+0000) 2024-03-04T16:32:22.534 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:32:22.540+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:32:22.541435+0000) 2024-03-04T16:32:23.535 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:32:23.541+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:32:23.541674+0000) 2024-03-04T16:32:24.535 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:32:24.541+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:32:24.541958+0000) 2024-03-04T16:32:25.535 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:32:25.541+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:32:25.542199+0000) 2024-03-04T16:32:26.535 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:32:26.541+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:32:26.542443+0000) 2024-03-04T16:32:27.536 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:32:27.542+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:32:27.542683+0000) 2024-03-04T16:32:28.536 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:32:28.542+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:32:28.542969+0000) 2024-03-04T16:32:29.536 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:32:29.542+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:32:29.543213+0000) 2024-03-04T16:32:30.536 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:32:30.542+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:32:30.543457+0000) 2024-03-04T16:32:31.537 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:32:31.543+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:32:31.543735+0000) 2024-03-04T16:32:32.537 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:32:32.543+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:32:32.543973+0000) 2024-03-04T16:32:33.537 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:32:33.543+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:32:33.544210+0000) 2024-03-04T16:32:34.537 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:32:34.543+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:32:34.544451+0000) 2024-03-04T16:32:35.537 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:32:35.544+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:32:35.544712+0000) 2024-03-04T16:32:36.538 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:32:36.544+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:32:36.544941+0000) 2024-03-04T16:32:37.538 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:32:37.544+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:32:37.545180+0000) 2024-03-04T16:32:38.538 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:32:38.545+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:32:38.545417+0000) 2024-03-04T16:32:39.539 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:32:39.545+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:32:39.545654+0000) 2024-03-04T16:32:40.538 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:32:40.545+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:32:40.545941+0000) 2024-03-04T16:32:41.539 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:32:41.545+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:32:41.546178+0000) 2024-03-04T16:32:42.539 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:32:42.546+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:32:42.546416+0000) 2024-03-04T16:32:43.539 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:32:43.546+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:32:43.546666+0000) 2024-03-04T16:32:44.539 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:32:44.546+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:32:44.546956+0000) 2024-03-04T16:32:45.540 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:32:45.546+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:32:45.547194+0000) 2024-03-04T16:32:46.540 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:32:46.546+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:32:46.547387+0000) 2024-03-04T16:32:47.540 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:32:47.546+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:32:47.547574+0000) 2024-03-04T16:32:47.642 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:32:47.653 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:32:48.540 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:32:48.547+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:32:48.547862+0000) 2024-03-04T16:32:49.540 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:32:49.547+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:32:49.548111+0000) 2024-03-04T16:32:50.541 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:32:50.548+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:32:50.548347+0000) 2024-03-04T16:32:51.541 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:32:51.548+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:32:51.548599+0000) 2024-03-04T16:32:52.541 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:32:52.548+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:32:52.548873+0000) 2024-03-04T16:32:53.541 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:32:53.548+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:32:53.549126+0000) 2024-03-04T16:32:54.542 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:32:54.548+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:32:54.549402+0000) 2024-03-04T16:32:55.542 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:32:55.548+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:32:55.549653+0000) 2024-03-04T16:32:56.542 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:32:56.548+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:32:56.549930+0000) 2024-03-04T16:32:57.542 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:32:57.549+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:32:57.550198+0000) 2024-03-04T16:32:58.543 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:32:58.549+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:32:58.550485+0000) 2024-03-04T16:32:59.543 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:32:59.549+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:32:59.550744+0000) 2024-03-04T16:33:00.543 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:33:00.549+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:33:00.550983+0000) 2024-03-04T16:33:01.543 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:33:01.549+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:33:01.551219+0000) 2024-03-04T16:33:02.544 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:33:02.550+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:33:02.551460+0000) 2024-03-04T16:33:03.544 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:33:03.550+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:33:03.551694+0000) 2024-03-04T16:33:04.544 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:33:04.550+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:33:04.551935+0000) 2024-03-04T16:33:05.544 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:33:05.551+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:33:05.552171+0000) 2024-03-04T16:33:06.545 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:33:06.551+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:33:06.552432+0000) 2024-03-04T16:33:07.545 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:33:07.551+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:33:07.552667+0000) 2024-03-04T16:33:08.545 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:33:08.552+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:33:08.552944+0000) 2024-03-04T16:33:09.545 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:33:09.552+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:33:09.553182+0000) 2024-03-04T16:33:10.545 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:33:10.552+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:33:10.553418+0000) 2024-03-04T16:33:11.546 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:33:11.552+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:33:11.553662+0000) 2024-03-04T16:33:12.546 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:33:12.553+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:33:12.553971+0000) 2024-03-04T16:33:13.546 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:33:13.553+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:33:13.554231+0000) 2024-03-04T16:33:14.546 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:33:14.553+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:33:14.554470+0000) 2024-03-04T16:33:15.546 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:33:15.553+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:33:15.554722+0000) 2024-03-04T16:33:16.547 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:33:16.554+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:33:16.554958+0000) 2024-03-04T16:33:17.547 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:33:17.554+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:33:17.555194+0000) 2024-03-04T16:33:17.862 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:33:17.874 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:33:18.547 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:33:18.554+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:33:18.555437+0000) 2024-03-04T16:33:19.547 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:33:19.554+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:33:19.555616+0000) 2024-03-04T16:33:20.547 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:33:20.555+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:33:20.555902+0000) 2024-03-04T16:33:21.548 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:33:21.555+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:33:21.556144+0000) 2024-03-04T16:33:22.548 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:33:22.555+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:33:22.556383+0000) 2024-03-04T16:33:23.548 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:33:23.555+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:33:23.556649+0000) 2024-03-04T16:33:24.548 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:33:24.556+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:33:24.556941+0000) 2024-03-04T16:33:25.549 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:33:25.556+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:33:25.557183+0000) 2024-03-04T16:33:26.549 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:33:26.556+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:33:26.557420+0000) 2024-03-04T16:33:27.549 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:33:27.556+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:33:27.557600+0000) 2024-03-04T16:33:28.550 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:33:28.557+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:33:28.557923+0000) 2024-03-04T16:33:29.550 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:33:29.557+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:33:29.558180+0000) 2024-03-04T16:33:30.550 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:33:30.557+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:33:30.558357+0000) 2024-03-04T16:33:31.550 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:33:31.558+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:33:31.558605+0000) 2024-03-04T16:33:32.550 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:33:32.558+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:33:32.558898+0000) 2024-03-04T16:33:33.550 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:33:33.558+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:33:33.559181+0000) 2024-03-04T16:33:34.551 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:33:34.558+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:33:34.559436+0000) 2024-03-04T16:33:35.551 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:33:35.559+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:33:35.559692+0000) 2024-03-04T16:33:36.551 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:33:36.559+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:33:36.559949+0000) 2024-03-04T16:33:37.551 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:33:37.559+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:33:37.560187+0000) 2024-03-04T16:33:38.552 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:33:38.559+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:33:38.560422+0000) 2024-03-04T16:33:39.552 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:33:39.560+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:33:39.560662+0000) 2024-03-04T16:33:40.552 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:33:40.560+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:33:40.560937+0000) 2024-03-04T16:33:41.552 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:33:41.560+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:33:41.561181+0000) 2024-03-04T16:33:42.553 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:33:42.561+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:33:42.561418+0000) 2024-03-04T16:33:43.553 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:33:43.561+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:33:43.561663+0000) 2024-03-04T16:33:44.553 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:33:44.561+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:33:44.561930+0000) 2024-03-04T16:33:45.553 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:33:45.561+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:33:45.562184+0000) 2024-03-04T16:33:46.553 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:33:46.561+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:33:46.562412+0000) 2024-03-04T16:33:47.554 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:33:47.562+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:33:47.562647+0000) 2024-03-04T16:33:48.083 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:33:48.091 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:33:48.554 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:33:48.562+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:33:48.562934+0000) 2024-03-04T16:33:49.554 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:33:49.562+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:33:49.563142+0000) 2024-03-04T16:33:50.554 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:33:50.562+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:33:50.563342+0000) 2024-03-04T16:33:51.554 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:33:51.562+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:33:51.563550+0000) 2024-03-04T16:33:52.555 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:33:52.562+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:33:52.563750+0000) 2024-03-04T16:33:53.555 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:33:53.563+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:33:53.564011+0000) 2024-03-04T16:33:54.555 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:33:54.564+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:33:54.564266+0000) 2024-03-04T16:33:55.555 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:33:55.564+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:33:55.564521+0000) 2024-03-04T16:33:56.555 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:33:56.564+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:33:56.564780+0000) 2024-03-04T16:33:57.556 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:33:57.564+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:33:57.565072+0000) 2024-03-04T16:33:58.556 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:33:58.564+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:33:58.565328+0000) 2024-03-04T16:33:59.556 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:33:59.564+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:33:59.565528+0000) 2024-03-04T16:34:00.556 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:34:00.564+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:34:00.565783+0000) 2024-03-04T16:34:01.557 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:34:01.564+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:34:01.566090+0000) 2024-03-04T16:34:02.557 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:34:02.565+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:34:02.566339+0000) 2024-03-04T16:34:03.557 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:34:03.565+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:34:03.566573+0000) 2024-03-04T16:34:04.557 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:34:04.565+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:34:04.566804+0000) 2024-03-04T16:34:05.558 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:34:05.566+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:34:05.567081+0000) 2024-03-04T16:34:06.558 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:34:06.566+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:34:06.567391+0000) 2024-03-04T16:34:07.558 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:34:07.566+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:34:07.567652+0000) 2024-03-04T16:34:08.558 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:34:08.566+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:34:08.567927+0000) 2024-03-04T16:34:09.559 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:34:09.566+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:34:09.568108+0000) 2024-03-04T16:34:10.559 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:34:10.567+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:34:10.568347+0000) 2024-03-04T16:34:11.559 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:34:11.567+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:34:11.568601+0000) 2024-03-04T16:34:12.559 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:34:12.568+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:34:12.568889+0000) 2024-03-04T16:34:13.560 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:34:13.568+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:34:13.569128+0000) 2024-03-04T16:34:14.560 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:34:14.568+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:34:14.569309+0000) 2024-03-04T16:34:15.560 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:34:15.568+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:34:15.569493+0000) 2024-03-04T16:34:16.560 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:34:16.568+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:34:16.569752+0000) 2024-03-04T16:34:17.560 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:34:17.569+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:34:17.569994+0000) 2024-03-04T16:34:18.221 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:34:18.229 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:34:18.560 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:34:18.569+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:34:18.570237+0000) 2024-03-04T16:34:19.561 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:34:19.569+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:34:19.570476+0000) 2024-03-04T16:34:20.561 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:34:20.569+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:34:20.570731+0000) 2024-03-04T16:34:21.561 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:34:21.570+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:34:21.570946+0000) 2024-03-04T16:34:22.561 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:34:22.570+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:34:22.571177+0000) 2024-03-04T16:34:23.562 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:34:23.570+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:34:23.571416+0000) 2024-03-04T16:34:24.562 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:34:24.570+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:34:24.571656+0000) 2024-03-04T16:34:25.562 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:34:25.571+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:34:25.571935+0000) 2024-03-04T16:34:26.563 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:34:26.571+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:34:26.572173+0000) 2024-03-04T16:34:27.563 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:34:27.571+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:34:27.572412+0000) 2024-03-04T16:34:28.563 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:34:28.571+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:34:28.572654+0000) 2024-03-04T16:34:29.563 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:34:29.572+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:34:29.572929+0000) 2024-03-04T16:34:30.563 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:34:30.572+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:34:30.573175+0000) 2024-03-04T16:34:31.563 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:34:31.572+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:34:31.573422+0000) 2024-03-04T16:34:32.564 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:34:32.573+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:34:32.573664+0000) 2024-03-04T16:34:33.564 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:34:33.573+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:34:33.573938+0000) 2024-03-04T16:34:34.564 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:34:34.573+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:34:34.574182+0000) 2024-03-04T16:34:35.564 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:34:35.573+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:34:35.574425+0000) 2024-03-04T16:34:36.565 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:34:36.574+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:34:36.574664+0000) 2024-03-04T16:34:37.565 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:34:37.574+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:34:37.574952+0000) 2024-03-04T16:34:38.565 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:34:38.574+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:34:38.575217+0000) 2024-03-04T16:34:39.565 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:34:39.574+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:34:39.575456+0000) 2024-03-04T16:34:40.566 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:34:40.575+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:34:40.575698+0000) 2024-03-04T16:34:41.566 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:34:41.575+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:34:41.575945+0000) 2024-03-04T16:34:42.566 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:34:42.575+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:34:42.576188+0000) 2024-03-04T16:34:43.566 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:34:43.575+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:34:43.576368+0000) 2024-03-04T16:34:44.566 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:34:44.576+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:34:44.576608+0000) 2024-03-04T16:34:45.566 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:34:45.576+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:34:45.576923+0000) 2024-03-04T16:34:46.567 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:34:46.576+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:34:46.577166+0000) 2024-03-04T16:34:47.567 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:34:47.577+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:34:47.577403+0000) 2024-03-04T16:34:48.365 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:34:48.373 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:34:48.567 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:34:48.576+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:34:48.577590+0000) 2024-03-04T16:34:49.568 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:34:49.576+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:34:49.577808+0000) 2024-03-04T16:34:50.568 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:34:50.577+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:34:50.578073+0000) 2024-03-04T16:34:51.568 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:34:51.577+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:34:51.578319+0000) 2024-03-04T16:34:52.568 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:34:52.578+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:34:52.578558+0000) 2024-03-04T16:34:53.568 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:34:53.578+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:34:53.578794+0000) 2024-03-04T16:34:54.569 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:34:54.578+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:34:54.579069+0000) 2024-03-04T16:34:55.569 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:34:55.579+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:34:55.579308+0000) 2024-03-04T16:34:56.569 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:34:56.579+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:34:56.579552+0000) 2024-03-04T16:34:57.569 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:34:57.579+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:34:57.579809+0000) 2024-03-04T16:34:58.569 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:34:58.579+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:34:58.580095+0000) 2024-03-04T16:34:59.570 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:34:59.579+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:34:59.580327+0000) 2024-03-04T16:35:00.570 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:35:00.580+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:35:00.580615+0000) 2024-03-04T16:35:01.570 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:35:01.580+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:35:01.580888+0000) 2024-03-04T16:35:02.570 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:35:02.580+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:35:02.581130+0000) 2024-03-04T16:35:03.571 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:35:03.581+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:35:03.581370+0000) 2024-03-04T16:35:04.571 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:35:04.580+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:35:04.581605+0000) 2024-03-04T16:35:05.571 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:35:05.581+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:35:05.581887+0000) 2024-03-04T16:35:06.571 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:35:06.581+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:35:06.582126+0000) 2024-03-04T16:35:07.572 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:35:07.581+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:35:07.582385+0000) 2024-03-04T16:35:08.572 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:35:08.581+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:35:08.582654+0000) 2024-03-04T16:35:09.572 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:35:09.581+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:35:09.582912+0000) 2024-03-04T16:35:10.573 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:35:10.582+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:35:10.583162+0000) 2024-03-04T16:35:11.573 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:35:11.582+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:35:11.583441+0000) 2024-03-04T16:35:12.573 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:35:12.582+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:35:12.583621+0000) 2024-03-04T16:35:13.573 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:35:13.582+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:35:13.583915+0000) 2024-03-04T16:35:14.573 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:35:14.582+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:35:14.584148+0000) 2024-03-04T16:35:15.573 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:35:15.583+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:35:15.584387+0000) 2024-03-04T16:35:16.574 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:35:16.583+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:35:16.584629+0000) 2024-03-04T16:35:17.574 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:35:17.583+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:35:17.584918+0000) 2024-03-04T16:35:18.571 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:35:18.574 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:35:18.584+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:35:18.585157+0000) 2024-03-04T16:35:18.580 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:35:19.574 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:35:19.584+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:35:19.585417+0000) 2024-03-04T16:35:20.575 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:35:20.584+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:35:20.585658+0000) 2024-03-04T16:35:21.575 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:35:21.585+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:35:21.585958+0000) 2024-03-04T16:35:22.575 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:35:22.585+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:35:22.586196+0000) 2024-03-04T16:35:23.575 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:35:23.585+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:35:23.586453+0000) 2024-03-04T16:35:24.576 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:35:24.585+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:35:24.586694+0000) 2024-03-04T16:35:25.576 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:35:25.586+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:35:25.586941+0000) 2024-03-04T16:35:26.576 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:35:26.586+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:35:26.587183+0000) 2024-03-04T16:35:27.576 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:35:27.586+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:35:27.587441+0000) 2024-03-04T16:35:28.576 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:35:28.586+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:35:28.587677+0000) 2024-03-04T16:35:29.577 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:35:29.587+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:35:29.587936+0000) 2024-03-04T16:35:30.577 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:35:30.587+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:35:30.588134+0000) 2024-03-04T16:35:31.577 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:35:31.587+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:35:31.588397+0000) 2024-03-04T16:35:32.577 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:35:32.587+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:35:32.588656+0000) 2024-03-04T16:35:33.578 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:35:33.588+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:35:33.588934+0000) 2024-03-04T16:35:34.578 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:35:34.588+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:35:34.589129+0000) 2024-03-04T16:35:35.578 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:35:35.588+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:35:35.589371+0000) 2024-03-04T16:35:36.578 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:35:36.588+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:35:36.589614+0000) 2024-03-04T16:35:37.578 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:35:37.589+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:35:37.589894+0000) 2024-03-04T16:35:38.579 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:35:38.589+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:35:38.590127+0000) 2024-03-04T16:35:39.579 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:35:39.589+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:35:39.590395+0000) 2024-03-04T16:35:40.579 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:35:40.589+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:35:40.590639+0000) 2024-03-04T16:35:41.579 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:35:41.590+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:35:41.590940+0000) 2024-03-04T16:35:42.580 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:35:42.590+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:35:42.591210+0000) 2024-03-04T16:35:43.580 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:35:43.590+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:35:43.591469+0000) 2024-03-04T16:35:44.580 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:35:44.590+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:35:44.591736+0000) 2024-03-04T16:35:45.580 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:35:45.591+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:35:45.592017+0000) 2024-03-04T16:35:46.581 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:35:46.591+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:35:46.592273+0000) 2024-03-04T16:35:47.581 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:35:47.591+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:35:47.592538+0000) 2024-03-04T16:35:48.581 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:35:48.591+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:35:48.592802+0000) 2024-03-04T16:35:48.716 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:35:48.724 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:35:49.582 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:35:49.592+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:35:49.593133+0000) 2024-03-04T16:35:50.582 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:35:50.592+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:35:50.593337+0000) 2024-03-04T16:35:51.582 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:35:51.592+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:35:51.593548+0000) 2024-03-04T16:35:52.582 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:35:52.592+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:35:52.593804+0000) 2024-03-04T16:35:53.582 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:35:53.593+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:35:53.594114+0000) 2024-03-04T16:35:54.583 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:35:54.593+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:35:54.594306+0000) 2024-03-04T16:35:55.583 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:35:55.593+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:35:55.594557+0000) 2024-03-04T16:35:56.583 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:35:56.594+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:35:56.594815+0000) 2024-03-04T16:35:57.583 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:35:57.594+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:35:57.595074+0000) 2024-03-04T16:35:58.584 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:35:58.594+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:35:58.595313+0000) 2024-03-04T16:35:59.584 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:35:59.594+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:35:59.595515+0000) 2024-03-04T16:36:00.584 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:36:00.594+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:36:00.595710+0000) 2024-03-04T16:36:01.584 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:36:01.595+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:36:01.595981+0000) 2024-03-04T16:36:02.584 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:36:02.595+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:36:02.596186+0000) 2024-03-04T16:36:03.584 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:36:03.595+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:36:03.596393+0000) 2024-03-04T16:36:04.585 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:36:04.595+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:36:04.596659+0000) 2024-03-04T16:36:05.585 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:36:05.596+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:36:05.596982+0000) 2024-03-04T16:36:06.585 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:36:06.596+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:36:06.597173+0000) 2024-03-04T16:36:07.585 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:36:07.596+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:36:07.597402+0000) 2024-03-04T16:36:08.586 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:36:08.596+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:36:08.597661+0000) 2024-03-04T16:36:09.586 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:36:09.597+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:36:09.597975+0000) 2024-03-04T16:36:10.586 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:36:10.597+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:36:10.598213+0000) 2024-03-04T16:36:11.586 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:36:11.597+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:36:11.598458+0000) 2024-03-04T16:36:12.587 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:36:12.597+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:36:12.598717+0000) 2024-03-04T16:36:13.587 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:36:13.597+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:36:13.598980+0000) 2024-03-04T16:36:14.587 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:36:14.597+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:36:14.599218+0000) 2024-03-04T16:36:15.587 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:36:15.597+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:36:15.599480+0000) 2024-03-04T16:36:16.588 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:36:16.597+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:36:16.599735+0000) 2024-03-04T16:36:17.588 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:36:17.598+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:36:17.599975+0000) 2024-03-04T16:36:18.588 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:36:18.599+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:36:18.600218+0000) 2024-03-04T16:36:18.921 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:36:18.928 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:36:19.588 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:36:19.598+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:36:19.600467+0000) 2024-03-04T16:36:20.589 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:36:20.598+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:36:20.600694+0000) 2024-03-04T16:36:21.589 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:36:21.599+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:36:21.600969+0000) 2024-03-04T16:36:22.589 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:36:22.600+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:36:22.601214+0000) 2024-03-04T16:36:23.590 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:36:23.600+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:36:23.601485+0000) 2024-03-04T16:36:24.589 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:36:24.600+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:36:24.601738+0000) 2024-03-04T16:36:25.590 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:36:25.601+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:36:25.601989+0000) 2024-03-04T16:36:26.590 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:36:26.601+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:36:26.602186+0000) 2024-03-04T16:36:27.590 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:36:27.601+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:36:27.602418+0000) 2024-03-04T16:36:28.590 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:36:28.601+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:36:28.602679+0000) 2024-03-04T16:36:29.591 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:36:29.602+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:36:29.602939+0000) 2024-03-04T16:36:30.591 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:36:30.602+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:36:30.603195+0000) 2024-03-04T16:36:31.591 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:36:31.602+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:36:31.603452+0000) 2024-03-04T16:36:32.591 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:36:32.602+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:36:32.603687+0000) 2024-03-04T16:36:33.592 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:36:33.603+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:36:33.603946+0000) 2024-03-04T16:36:34.592 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:36:34.603+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:36:34.604238+0000) 2024-03-04T16:36:35.592 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:36:35.603+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:36:35.604506+0000) 2024-03-04T16:36:36.592 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:36:36.603+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:36:36.604749+0000) 2024-03-04T16:36:37.592 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:36:37.604+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:36:37.604990+0000) 2024-03-04T16:36:38.593 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:36:38.604+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:36:38.605210+0000) 2024-03-04T16:36:39.593 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:36:39.604+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:36:39.605411+0000) 2024-03-04T16:36:40.593 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:36:40.604+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:36:40.605687+0000) 2024-03-04T16:36:41.593 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:36:41.605+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:36:41.605946+0000) 2024-03-04T16:36:42.594 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:36:42.605+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:36:42.606187+0000) 2024-03-04T16:36:43.594 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:36:43.605+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:36:43.606392+0000) 2024-03-04T16:36:44.594 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:36:44.605+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:36:44.606648+0000) 2024-03-04T16:36:45.594 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:36:45.606+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:36:45.606947+0000) 2024-03-04T16:36:46.594 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:36:46.606+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:36:46.607144+0000) 2024-03-04T16:36:47.595 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:36:47.606+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:36:47.607350+0000) 2024-03-04T16:36:48.595 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:36:48.606+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:36:48.607550+0000) 2024-03-04T16:36:49.133 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:36:49.141 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:36:49.595 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:36:49.607+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:36:49.607857+0000) 2024-03-04T16:36:50.595 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:36:50.607+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:36:50.608079+0000) 2024-03-04T16:36:51.596 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:36:51.607+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:36:51.608303+0000) 2024-03-04T16:36:52.596 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:36:52.607+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:36:52.608579+0000) 2024-03-04T16:36:53.596 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:36:53.608+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:36:53.608867+0000) 2024-03-04T16:36:54.596 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:36:54.608+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:36:54.609100+0000) 2024-03-04T16:36:55.596 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:36:55.608+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:36:55.609311+0000) 2024-03-04T16:36:56.597 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:36:56.608+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:36:56.609513+0000) 2024-03-04T16:36:57.597 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:36:57.608+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:36:57.609780+0000) 2024-03-04T16:36:58.597 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:36:58.609+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:36:58.610015+0000) 2024-03-04T16:36:59.597 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:36:59.609+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:36:59.610247+0000) 2024-03-04T16:37:00.598 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:37:00.609+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:37:00.610451+0000) 2024-03-04T16:37:01.598 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:37:01.609+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:37:01.610719+0000) 2024-03-04T16:37:02.598 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:37:02.610+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:37:02.610981+0000) 2024-03-04T16:37:03.598 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:37:03.610+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:37:03.611211+0000) 2024-03-04T16:37:04.598 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:37:04.610+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:37:04.611409+0000) 2024-03-04T16:37:05.599 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:37:05.610+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:37:05.611670+0000) 2024-03-04T16:37:06.599 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:37:06.611+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:37:06.611943+0000) 2024-03-04T16:37:07.599 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:37:07.611+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:37:07.612218+0000) 2024-03-04T16:37:08.599 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:37:08.611+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:37:08.612475+0000) 2024-03-04T16:37:09.600 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:37:09.611+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:37:09.612744+0000) 2024-03-04T16:37:10.600 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:37:10.612+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:37:10.612992+0000) 2024-03-04T16:37:11.600 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:37:11.613+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:37:11.613246+0000) 2024-03-04T16:37:12.600 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:37:12.612+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:37:12.613444+0000) 2024-03-04T16:37:13.602 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:37:13.612+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:37:13.613707+0000) 2024-03-04T16:37:14.601 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:37:14.613+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:37:14.614027+0000) 2024-03-04T16:37:15.601 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:37:15.614+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:37:15.614288+0000) 2024-03-04T16:37:16.601 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:37:16.614+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:37:16.614530+0000) 2024-03-04T16:37:17.602 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:37:17.613+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:37:17.614786+0000) 2024-03-04T16:37:18.602 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:37:18.613+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:37:18.615034+0000) 2024-03-04T16:37:19.287 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:37:19.295 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:37:19.602 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:37:19.614+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:37:19.615282+0000) 2024-03-04T16:37:20.602 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:37:20.614+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:37:20.615532+0000) 2024-03-04T16:37:21.603 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:37:21.614+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:37:21.615782+0000) 2024-03-04T16:37:22.603 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:37:22.614+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:37:22.616018+0000) 2024-03-04T16:37:23.603 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:37:23.615+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:37:23.616262+0000) 2024-03-04T16:37:24.603 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:37:24.615+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:37:24.616503+0000) 2024-03-04T16:37:25.604 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:37:25.614+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:37:25.616762+0000) 2024-03-04T16:37:26.604 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:37:26.616+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:37:26.617003+0000) 2024-03-04T16:37:27.604 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:37:27.616+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:37:27.617237+0000) 2024-03-04T16:37:28.604 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:37:28.616+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:37:28.617507+0000) 2024-03-04T16:37:29.604 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:37:29.616+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:37:29.617799+0000) 2024-03-04T16:37:30.605 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:37:30.617+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:37:30.618072+0000) 2024-03-04T16:37:31.605 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:37:31.617+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:37:31.618277+0000) 2024-03-04T16:37:32.605 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:37:32.617+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:37:32.618476+0000) 2024-03-04T16:37:33.605 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:37:33.617+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:37:33.618736+0000) 2024-03-04T16:37:34.606 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:37:34.618+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:37:34.618985+0000) 2024-03-04T16:37:35.606 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:37:35.618+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:37:35.619235+0000) 2024-03-04T16:37:36.606 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:37:36.618+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:37:36.619492+0000) 2024-03-04T16:37:37.606 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:37:37.618+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:37:37.619728+0000) 2024-03-04T16:37:38.606 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:37:38.619+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:37:38.619977+0000) 2024-03-04T16:37:39.607 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:37:39.619+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:37:39.620237+0000) 2024-03-04T16:37:40.607 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:37:40.619+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:37:40.620508+0000) 2024-03-04T16:37:41.607 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:37:41.619+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:37:41.620776+0000) 2024-03-04T16:37:42.607 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:37:42.620+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:37:42.621015+0000) 2024-03-04T16:37:43.608 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:37:43.620+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:37:43.621257+0000) 2024-03-04T16:37:44.608 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:37:44.620+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:37:44.621494+0000) 2024-03-04T16:37:45.608 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:37:45.621+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:37:45.621747+0000) 2024-03-04T16:37:46.608 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:37:46.621+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:37:46.621997+0000) 2024-03-04T16:37:47.608 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:37:47.621+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:37:47.622194+0000) 2024-03-04T16:37:48.609 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:37:48.621+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:37:48.622439+0000) 2024-03-04T16:37:49.500 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:37:49.509 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:37:49.609 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:37:49.621+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:37:49.622680+0000) 2024-03-04T16:37:50.609 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:37:50.622+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:37:50.622950+0000) 2024-03-04T16:37:51.609 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:37:51.622+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:37:51.623201+0000) 2024-03-04T16:37:52.609 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:37:52.622+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:37:52.623443+0000) 2024-03-04T16:37:53.610 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:37:53.623+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:37:53.623684+0000) 2024-03-04T16:37:54.610 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:37:54.623+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:37:54.623957+0000) 2024-03-04T16:37:55.610 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:37:55.623+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:37:55.624202+0000) 2024-03-04T16:37:56.611 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:37:56.623+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:37:56.624444+0000) 2024-03-04T16:37:57.611 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:37:57.624+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:37:57.624713+0000) 2024-03-04T16:37:58.611 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:37:58.624+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:37:58.624954+0000) 2024-03-04T16:37:59.611 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:37:59.624+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:37:59.625219+0000) 2024-03-04T16:38:00.612 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:38:00.625+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:38:00.625486+0000) 2024-03-04T16:38:01.612 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:38:01.624+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:38:01.625754+0000) 2024-03-04T16:38:02.612 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:38:02.625+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:38:02.625997+0000) 2024-03-04T16:38:03.612 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:38:03.625+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:38:03.626241+0000) 2024-03-04T16:38:04.613 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:38:04.626+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:38:04.626521+0000) 2024-03-04T16:38:05.613 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:38:05.626+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:38:05.626782+0000) 2024-03-04T16:38:06.613 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:38:06.626+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:38:06.627032+0000) 2024-03-04T16:38:07.613 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:38:07.626+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:38:07.627291+0000) 2024-03-04T16:38:08.613 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:38:08.626+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:38:08.627535+0000) 2024-03-04T16:38:09.614 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:38:09.627+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:38:09.627809+0000) 2024-03-04T16:38:10.614 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:38:10.627+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:38:10.628162+0000) 2024-03-04T16:38:11.614 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:38:11.628+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:38:11.628448+0000) 2024-03-04T16:38:12.615 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:38:12.628+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:38:12.628641+0000) 2024-03-04T16:38:13.615 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:38:13.628+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:38:13.628939+0000) 2024-03-04T16:38:14.615 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:38:14.628+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:38:14.629182+0000) 2024-03-04T16:38:15.615 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:38:15.628+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:38:15.629381+0000) 2024-03-04T16:38:16.616 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:38:16.629+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:38:16.629622+0000) 2024-03-04T16:38:17.616 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:38:17.629+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:38:17.629993+0000) 2024-03-04T16:38:18.616 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:38:18.630+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:38:18.630235+0000) 2024-03-04T16:38:19.616 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:38:19.630+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:38:19.630477+0000) 2024-03-04T16:38:19.697 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:38:19.705 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:38:20.616 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:38:20.630+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:38:20.630727+0000) 2024-03-04T16:38:21.617 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:38:21.630+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:38:21.630978+0000) 2024-03-04T16:38:22.617 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:38:22.631+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:38:22.631219+0000) 2024-03-04T16:38:23.617 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:38:23.631+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:38:23.631465+0000) 2024-03-04T16:38:24.617 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:38:24.630+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:38:24.631706+0000) 2024-03-04T16:38:25.618 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:38:25.630+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:38:25.631950+0000) 2024-03-04T16:38:26.618 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:38:26.631+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:38:26.632193+0000) 2024-03-04T16:38:27.618 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:38:27.631+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:38:27.632434+0000) 2024-03-04T16:38:28.618 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:38:28.631+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:38:28.632685+0000) 2024-03-04T16:38:29.619 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:38:29.631+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:38:29.632925+0000) 2024-03-04T16:38:30.619 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:38:30.632+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:38:30.633173+0000) 2024-03-04T16:38:31.619 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:38:31.631+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:38:31.633442+0000) 2024-03-04T16:38:32.619 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:38:32.632+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:38:32.633681+0000) 2024-03-04T16:38:33.619 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:38:33.632+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:38:33.633960+0000) 2024-03-04T16:38:34.620 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:38:34.633+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:38:34.634155+0000) 2024-03-04T16:38:35.620 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:38:35.633+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:38:35.634401+0000) 2024-03-04T16:38:36.620 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:38:36.633+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:38:36.634666+0000) 2024-03-04T16:38:37.620 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:38:37.634+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:38:37.634954+0000) 2024-03-04T16:38:38.621 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:38:38.634+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:38:38.635198+0000) 2024-03-04T16:38:39.621 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:38:39.634+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:38:39.635451+0000) 2024-03-04T16:38:40.621 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:38:40.634+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:38:40.635716+0000) 2024-03-04T16:38:41.621 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:38:41.635+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:38:41.635967+0000) 2024-03-04T16:38:42.622 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:38:42.635+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:38:42.636212+0000) 2024-03-04T16:38:43.622 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:38:43.635+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:38:43.636456+0000) 2024-03-04T16:38:44.622 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:38:44.635+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:38:44.636700+0000) 2024-03-04T16:38:45.622 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:38:45.636+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:38:45.636955+0000) 2024-03-04T16:38:46.622 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:38:46.636+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:38:46.637198+0000) 2024-03-04T16:38:47.623 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:38:47.636+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:38:47.637440+0000) 2024-03-04T16:38:48.623 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:38:48.636+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:38:48.637684+0000) 2024-03-04T16:38:49.623 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:38:49.637+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:38:49.637926+0000) 2024-03-04T16:38:49.910 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:38:49.918 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:38:50.623 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:38:50.637+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:38:50.638176+0000) 2024-03-04T16:38:51.624 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:38:51.637+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:38:51.638454+0000) 2024-03-04T16:38:52.624 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:38:52.638+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:38:52.638689+0000) 2024-03-04T16:38:53.624 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:38:53.638+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:38:53.638931+0000) 2024-03-04T16:38:54.624 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:38:54.638+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:38:54.639172+0000) 2024-03-04T16:38:55.625 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:38:55.638+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:38:55.639424+0000) 2024-03-04T16:38:56.625 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:38:56.639+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:38:56.639674+0000) 2024-03-04T16:38:57.625 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:38:57.639+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:38:57.639957+0000) 2024-03-04T16:38:58.625 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:38:58.639+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:38:58.640200+0000) 2024-03-04T16:38:59.626 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:38:59.639+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:38:59.640448+0000) 2024-03-04T16:39:00.626 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:39:00.639+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:39:00.640760+0000) 2024-03-04T16:39:01.626 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:39:01.640+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:39:01.641011+0000) 2024-03-04T16:39:02.626 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:39:02.640+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:39:02.641255+0000) 2024-03-04T16:39:03.627 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:39:03.640+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:39:03.641527+0000) 2024-03-04T16:39:04.627 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:39:04.641+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:39:04.641773+0000) 2024-03-04T16:39:05.627 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:39:05.641+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:39:05.642020+0000) 2024-03-04T16:39:06.627 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:39:06.641+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:39:06.642264+0000) 2024-03-04T16:39:07.628 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:39:07.641+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:39:07.642593+0000) 2024-03-04T16:39:08.628 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:39:08.642+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:39:08.642944+0000) 2024-03-04T16:39:09.628 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:39:09.642+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:39:09.643125+0000) 2024-03-04T16:39:10.628 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:39:10.642+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:39:10.643375+0000) 2024-03-04T16:39:11.629 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:39:11.643+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:39:11.643629+0000) 2024-03-04T16:39:12.629 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:39:12.643+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:39:12.643930+0000) 2024-03-04T16:39:13.629 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:39:13.643+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:39:13.644174+0000) 2024-03-04T16:39:14.629 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:39:14.644+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:39:14.644429+0000) 2024-03-04T16:39:15.630 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:39:15.644+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:39:15.644675+0000) 2024-03-04T16:39:16.630 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:39:16.644+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:39:16.644950+0000) 2024-03-04T16:39:17.630 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:39:17.644+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:39:17.645190+0000) 2024-03-04T16:39:18.630 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:39:18.644+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:39:18.645449+0000) 2024-03-04T16:39:19.631 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:39:19.645+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:39:19.645689+0000) 2024-03-04T16:39:20.348 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:39:20.356 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:39:20.631 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:39:20.645+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:39:20.645945+0000) 2024-03-04T16:39:21.631 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:39:21.645+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:39:21.646204+0000) 2024-03-04T16:39:22.631 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:39:22.646+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:39:22.646449+0000) 2024-03-04T16:39:23.632 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:39:23.646+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:39:23.646683+0000) 2024-03-04T16:39:24.632 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:39:24.646+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:39:24.646954+0000) 2024-03-04T16:39:25.632 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:39:25.646+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:39:25.647192+0000) 2024-03-04T16:39:26.632 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:39:26.647+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:39:26.647443+0000) 2024-03-04T16:39:27.632 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:39:27.647+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:39:27.647680+0000) 2024-03-04T16:39:28.633 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:39:28.647+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:39:28.647953+0000) 2024-03-04T16:39:29.633 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:39:29.647+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:39:29.648172+0000) 2024-03-04T16:39:30.633 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:39:30.646+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:39:30.648433+0000) 2024-03-04T16:39:31.633 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:39:31.647+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:39:31.648686+0000) 2024-03-04T16:39:32.634 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:39:32.647+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:39:32.648936+0000) 2024-03-04T16:39:33.634 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:39:33.648+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:39:33.649177+0000) 2024-03-04T16:39:34.634 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:39:34.648+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:39:34.649416+0000) 2024-03-04T16:39:35.634 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:39:35.648+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:39:35.649660+0000) 2024-03-04T16:39:36.635 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:39:36.648+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:39:36.649947+0000) 2024-03-04T16:39:37.635 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:39:37.649+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:39:37.650193+0000) 2024-03-04T16:39:38.635 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:39:38.649+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:39:38.650474+0000) 2024-03-04T16:39:39.635 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:39:39.649+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:39:39.650757+0000) 2024-03-04T16:39:40.636 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:39:40.650+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:39:40.651020+0000) 2024-03-04T16:39:41.636 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:39:41.650+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:39:41.651295+0000) 2024-03-04T16:39:42.636 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:39:42.650+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:39:42.651512+0000) 2024-03-04T16:39:43.636 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:39:43.650+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:39:43.651776+0000) 2024-03-04T16:39:44.637 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:39:44.651+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:39:44.652042+0000) 2024-03-04T16:39:45.637 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:39:45.651+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:39:45.652303+0000) 2024-03-04T16:39:46.637 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:39:46.651+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:39:46.652571+0000) 2024-03-04T16:39:47.637 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:39:47.652+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:39:47.652883+0000) 2024-03-04T16:39:48.638 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:39:48.652+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:39:48.653144+0000) 2024-03-04T16:39:49.638 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:39:49.652+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:39:49.653417+0000) 2024-03-04T16:39:50.494 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:39:50.502 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:39:50.638 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:39:50.652+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:39:50.653677+0000) 2024-03-04T16:39:51.638 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:39:51.653+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:39:51.653983+0000) 2024-03-04T16:39:52.639 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:39:52.653+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:39:52.654247+0000) 2024-03-04T16:39:53.639 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:39:53.653+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:39:53.654559+0000) 2024-03-04T16:39:54.639 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:39:54.654+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:39:54.654871+0000) 2024-03-04T16:39:55.639 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:39:55.654+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:39:55.655092+0000) 2024-03-04T16:39:56.640 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:39:56.654+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:39:56.655355+0000) 2024-03-04T16:39:57.640 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:39:57.654+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:39:57.655543+0000) 2024-03-04T16:39:58.640 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:39:58.655+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:39:58.655786+0000) 2024-03-04T16:39:59.640 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:39:59.655+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:39:59.656033+0000) 2024-03-04T16:40:00.641 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:40:00.655+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:40:00.656293+0000) 2024-03-04T16:40:01.641 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:40:01.655+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:40:01.656560+0000) 2024-03-04T16:40:02.641 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:40:02.656+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:40:02.656799+0000) 2024-03-04T16:40:03.641 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:40:03.656+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:40:03.657090+0000) 2024-03-04T16:40:04.642 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:40:04.656+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:40:04.657340+0000) 2024-03-04T16:40:05.642 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:40:05.657+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:40:05.657584+0000) 2024-03-04T16:40:06.642 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:40:06.657+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:40:06.657859+0000) 2024-03-04T16:40:07.642 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:40:07.657+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:40:07.658120+0000) 2024-03-04T16:40:08.643 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:40:08.657+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:40:08.658387+0000) 2024-03-04T16:40:09.643 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:40:09.657+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:40:09.658591+0000) 2024-03-04T16:40:10.643 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:40:10.658+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:40:10.658894+0000) 2024-03-04T16:40:11.643 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:40:11.658+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:40:11.659147+0000) 2024-03-04T16:40:12.644 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:40:12.658+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:40:12.659393+0000) 2024-03-04T16:40:13.644 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:40:13.659+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:40:13.659636+0000) 2024-03-04T16:40:14.644 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:40:14.659+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:40:14.659932+0000) 2024-03-04T16:40:15.644 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:40:15.659+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:40:15.660156+0000) 2024-03-04T16:40:16.644 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:40:16.659+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:40:16.660401+0000) 2024-03-04T16:40:17.645 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:40:17.660+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:40:17.660639+0000) 2024-03-04T16:40:18.645 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:40:18.660+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:40:18.660928+0000) 2024-03-04T16:40:19.645 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:40:19.660+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:40:19.661169+0000) 2024-03-04T16:40:20.645 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:40:20.660+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:40:20.661372+0000) 2024-03-04T16:40:20.698 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:40:20.706 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:40:21.646 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:40:21.661+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:40:21.661628+0000) 2024-03-04T16:40:22.648 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:40:22.661+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:40:22.661911+0000) 2024-03-04T16:40:23.646 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:40:23.661+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:40:23.662155+0000) 2024-03-04T16:40:24.646 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:40:24.661+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:40:24.662351+0000) 2024-03-04T16:40:25.647 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:40:25.661+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:40:25.662542+0000) 2024-03-04T16:40:26.647 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:40:26.662+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:40:26.662784+0000) 2024-03-04T16:40:27.647 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:40:27.662+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:40:27.663032+0000) 2024-03-04T16:40:28.648 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:40:28.663+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:40:28.663277+0000) 2024-03-04T16:40:29.647 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:40:29.663+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:40:29.663511+0000) 2024-03-04T16:40:30.648 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:40:30.662+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:40:30.663776+0000) 2024-03-04T16:40:31.648 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:40:31.663+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:40:31.664038+0000) 2024-03-04T16:40:32.648 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:40:32.663+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:40:32.664292+0000) 2024-03-04T16:40:33.648 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:40:33.664+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:40:33.664537+0000) 2024-03-04T16:40:34.649 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:40:34.664+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:40:34.664779+0000) 2024-03-04T16:40:35.649 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:40:35.663+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:40:35.665106+0000) 2024-03-04T16:40:36.649 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:40:36.664+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:40:36.665369+0000) 2024-03-04T16:40:37.649 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:40:37.664+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:40:37.665631+0000) 2024-03-04T16:40:38.650 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:40:38.664+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:40:38.665941+0000) 2024-03-04T16:40:39.650 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:40:39.664+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:40:39.666192+0000) 2024-03-04T16:40:40.650 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:40:40.665+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:40:40.666452+0000) 2024-03-04T16:40:41.651 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:40:41.665+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:40:41.666732+0000) 2024-03-04T16:40:42.651 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:40:42.665+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:40:42.666991+0000) 2024-03-04T16:40:43.651 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:40:43.666+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:40:43.667249+0000) 2024-03-04T16:40:44.651 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:40:44.666+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:40:44.667503+0000) 2024-03-04T16:40:45.651 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:40:45.666+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:40:45.667765+0000) 2024-03-04T16:40:46.652 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:40:46.667+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:40:46.667988+0000) 2024-03-04T16:40:47.652 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:40:47.667+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:40:47.668242+0000) 2024-03-04T16:40:48.652 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:40:48.667+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:40:48.668506+0000) 2024-03-04T16:40:49.652 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:40:49.667+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:40:49.668742+0000) 2024-03-04T16:40:50.653 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:40:50.668+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:40:50.668986+0000) 2024-03-04T16:40:50.905 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:40:50.913 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:40:51.653 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:40:51.668+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:40:51.669261+0000) 2024-03-04T16:40:52.653 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:40:52.668+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:40:52.669507+0000) 2024-03-04T16:40:53.653 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:40:53.668+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:40:53.669750+0000) 2024-03-04T16:40:54.654 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:40:54.669+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:40:54.670033+0000) 2024-03-04T16:40:55.654 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:40:55.669+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:40:55.670332+0000) 2024-03-04T16:40:56.654 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:40:56.669+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:40:56.670579+0000) 2024-03-04T16:40:57.654 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:40:57.670+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:40:57.670882+0000) 2024-03-04T16:40:58.655 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:40:58.670+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:40:58.671126+0000) 2024-03-04T16:40:59.655 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:40:59.670+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:40:59.671384+0000) 2024-03-04T16:41:00.655 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:41:00.670+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:41:00.671656+0000) 2024-03-04T16:41:01.655 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:41:01.671+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:41:01.671958+0000) 2024-03-04T16:41:02.656 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:41:02.671+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:41:02.672215+0000) 2024-03-04T16:41:03.656 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:41:03.671+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:41:03.672483+0000) 2024-03-04T16:41:04.656 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:41:04.672+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:41:04.672738+0000) 2024-03-04T16:41:05.656 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:41:05.672+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:41:05.673002+0000) 2024-03-04T16:41:06.657 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:41:06.672+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:41:06.673283+0000) 2024-03-04T16:41:07.657 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:41:07.672+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:41:07.673565+0000) 2024-03-04T16:41:08.657 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:41:08.673+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:41:08.673863+0000) 2024-03-04T16:41:09.657 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:41:09.673+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:41:09.674082+0000) 2024-03-04T16:41:10.658 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:41:10.673+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:41:10.674332+0000) 2024-03-04T16:41:11.658 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:41:11.674+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:41:11.674600+0000) 2024-03-04T16:41:12.658 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:41:12.674+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:41:12.674901+0000) 2024-03-04T16:41:13.659 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:41:13.674+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:41:13.675159+0000) 2024-03-04T16:41:14.659 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:41:14.674+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:41:14.675413+0000) 2024-03-04T16:41:15.659 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:41:15.674+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:41:15.675669+0000) 2024-03-04T16:41:16.659 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:41:16.675+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:41:16.675962+0000) 2024-03-04T16:41:17.660 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:41:17.675+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:41:17.676225+0000) 2024-03-04T16:41:18.660 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:41:18.676+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:41:18.676489+0000) 2024-03-04T16:41:19.660 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:41:19.676+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:41:19.676742+0000) 2024-03-04T16:41:20.660 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:41:20.676+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:41:20.676950+0000) 2024-03-04T16:41:21.041 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:41:21.049 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:41:21.660 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:41:21.676+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:41:21.677167+0000) 2024-03-04T16:41:22.661 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:41:22.677+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:41:22.677425+0000) 2024-03-04T16:41:23.661 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:41:23.676+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:41:23.677702+0000) 2024-03-04T16:41:24.661 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:41:24.677+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:41:24.677967+0000) 2024-03-04T16:41:25.661 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:41:25.677+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:41:25.678164+0000) 2024-03-04T16:41:26.662 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:41:26.678+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:41:26.678462+0000) 2024-03-04T16:41:27.662 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:41:27.678+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:41:27.678719+0000) 2024-03-04T16:41:28.662 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:41:28.678+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:41:28.678976+0000) 2024-03-04T16:41:29.662 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:41:29.678+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:41:29.679239+0000) 2024-03-04T16:41:30.663 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:41:30.678+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:41:30.679440+0000) 2024-03-04T16:41:31.663 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:41:31.679+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:41:31.679748+0000) 2024-03-04T16:41:32.663 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:41:32.679+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:41:32.680003+0000) 2024-03-04T16:41:33.663 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:41:33.680+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:41:33.680260+0000) 2024-03-04T16:41:34.664 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:41:34.680+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:41:34.680520+0000) 2024-03-04T16:41:35.664 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:41:35.679+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:41:35.680768+0000) 2024-03-04T16:41:36.664 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:41:36.680+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:41:36.681046+0000) 2024-03-04T16:41:37.665 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:41:37.681+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:41:37.681310+0000) 2024-03-04T16:41:38.665 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:41:38.681+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:41:38.681562+0000) 2024-03-04T16:41:39.665 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:41:39.680+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:41:39.681818+0000) 2024-03-04T16:41:40.665 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:41:40.680+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:41:40.682156+0000) 2024-03-04T16:41:41.665 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:41:41.680+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:41:41.682363+0000) 2024-03-04T16:41:42.666 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:41:42.681+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:41:42.682603+0000) 2024-03-04T16:41:43.666 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:41:43.681+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:41:43.682884+0000) 2024-03-04T16:41:44.666 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:41:44.681+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:41:44.683087+0000) 2024-03-04T16:41:45.666 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:41:45.682+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:41:45.683332+0000) 2024-03-04T16:41:46.667 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:41:46.682+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:41:46.683566+0000) 2024-03-04T16:41:47.667 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:41:47.682+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:41:47.683809+0000) 2024-03-04T16:41:48.668 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:41:48.683+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:41:48.684114+0000) 2024-03-04T16:41:49.667 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:41:49.683+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:41:49.684361+0000) 2024-03-04T16:41:50.668 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:41:50.683+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:41:50.684607+0000) 2024-03-04T16:41:51.254 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:41:51.262 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:41:51.668 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:41:51.684+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:41:51.684918+0000) 2024-03-04T16:41:52.668 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:41:52.684+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:41:52.685161+0000) 2024-03-04T16:41:53.668 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:41:53.684+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:41:53.685424+0000) 2024-03-04T16:41:54.669 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:41:54.684+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:41:54.685666+0000) 2024-03-04T16:41:55.669 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:41:55.685+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:41:55.685950+0000) 2024-03-04T16:41:56.669 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:41:56.685+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:41:56.686194+0000) 2024-03-04T16:41:57.669 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:41:57.685+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:41:57.686440+0000) 2024-03-04T16:41:58.670 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:41:58.685+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:41:58.686709+0000) 2024-03-04T16:41:59.670 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:41:59.686+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:41:59.686947+0000) 2024-03-04T16:42:00.670 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:42:00.686+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:42:00.687186+0000) 2024-03-04T16:42:01.670 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:42:01.686+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:42:01.687452+0000) 2024-03-04T16:42:02.671 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:42:02.686+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:42:02.687696+0000) 2024-03-04T16:42:03.671 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:42:03.687+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:42:03.687947+0000) 2024-03-04T16:42:04.671 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:42:04.687+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:42:04.688185+0000) 2024-03-04T16:42:05.671 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:42:05.687+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:42:05.688429+0000) 2024-03-04T16:42:06.671 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:42:06.687+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:42:06.688695+0000) 2024-03-04T16:42:07.672 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:42:07.688+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:42:07.688942+0000) 2024-03-04T16:42:08.672 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:42:08.688+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:42:08.689186+0000) 2024-03-04T16:42:09.672 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:42:09.688+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:42:09.689423+0000) 2024-03-04T16:42:10.673 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:42:10.689+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:42:10.689684+0000) 2024-03-04T16:42:11.673 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:42:11.689+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:42:11.689945+0000) 2024-03-04T16:42:12.673 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:42:12.689+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:42:12.690192+0000) 2024-03-04T16:42:13.673 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:42:13.689+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:42:13.690453+0000) 2024-03-04T16:42:14.673 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:42:14.690+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:42:14.690691+0000) 2024-03-04T16:42:15.674 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:42:15.690+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:42:15.690950+0000) 2024-03-04T16:42:16.674 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:42:16.690+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:42:16.691197+0000) 2024-03-04T16:42:17.674 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:42:17.690+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:42:17.691442+0000) 2024-03-04T16:42:18.674 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:42:18.691+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:42:18.691686+0000) 2024-03-04T16:42:19.675 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:42:19.691+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:42:19.691948+0000) 2024-03-04T16:42:20.675 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:42:20.691+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:42:20.692193+0000) 2024-03-04T16:42:21.468 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:42:21.479 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:42:21.675 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:42:21.691+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:42:21.692447+0000) 2024-03-04T16:42:22.675 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:42:22.692+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:42:22.692699+0000) 2024-03-04T16:42:23.676 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:42:23.692+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:42:23.692944+0000) 2024-03-04T16:42:24.676 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:42:24.692+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:42:24.693188+0000) 2024-03-04T16:42:25.676 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:42:25.693+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:42:25.693430+0000) 2024-03-04T16:42:26.676 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:42:26.692+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:42:26.693685+0000) 2024-03-04T16:42:27.676 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:42:27.693+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:42:27.693933+0000) 2024-03-04T16:42:28.677 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:42:28.693+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:42:28.694227+0000) 2024-03-04T16:42:29.677 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:42:29.694+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:42:29.694466+0000) 2024-03-04T16:42:30.677 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:42:30.694+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:42:30.694706+0000) 2024-03-04T16:42:31.678 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:42:31.694+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:42:31.694981+0000) 2024-03-04T16:42:32.678 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:42:32.694+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:42:32.695242+0000) 2024-03-04T16:42:33.678 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:42:33.695+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:42:33.695486+0000) 2024-03-04T16:42:34.678 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:42:34.695+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:42:34.695729+0000) 2024-03-04T16:42:35.678 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:42:35.695+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:42:35.695990+0000) 2024-03-04T16:42:36.679 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:42:36.696+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:42:36.696254+0000) 2024-03-04T16:42:37.679 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:42:37.696+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:42:37.696518+0000) 2024-03-04T16:42:38.679 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:42:38.696+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:42:38.696785+0000) 2024-03-04T16:42:39.679 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:42:39.696+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:42:39.697047+0000) 2024-03-04T16:42:40.680 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:42:40.697+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:42:40.697295+0000) 2024-03-04T16:42:41.680 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:42:41.697+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:42:41.697548+0000) 2024-03-04T16:42:42.680 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:42:42.697+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:42:42.697793+0000) 2024-03-04T16:42:43.681 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:42:43.696+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:42:43.698079+0000) 2024-03-04T16:42:44.681 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:42:44.697+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:42:44.698320+0000) 2024-03-04T16:42:45.681 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:42:45.697+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:42:45.698558+0000) 2024-03-04T16:42:46.681 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:42:46.697+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:42:46.698788+0000) 2024-03-04T16:42:47.681 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:42:47.697+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:42:47.699050+0000) 2024-03-04T16:42:48.682 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:42:48.698+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:42:48.699319+0000) 2024-03-04T16:42:49.682 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:42:49.698+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:42:49.699560+0000) 2024-03-04T16:42:50.682 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:42:50.698+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:42:50.699803+0000) 2024-03-04T16:42:51.606 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:42:51.617 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:42:51.682 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:42:51.698+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:42:51.700053+0000) 2024-03-04T16:42:52.682 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:42:52.699+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:42:52.700296+0000) 2024-03-04T16:42:53.683 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:42:53.699+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:42:53.700537+0000) 2024-03-04T16:42:54.683 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:42:54.699+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:42:54.700790+0000) 2024-03-04T16:42:55.683 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:42:55.700+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:42:55.700990+0000) 2024-03-04T16:42:56.683 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:42:56.700+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:42:56.701232+0000) 2024-03-04T16:42:57.684 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:42:57.700+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:42:57.701484+0000) 2024-03-04T16:42:58.684 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:42:58.700+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:42:58.701726+0000) 2024-03-04T16:42:59.684 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:42:59.701+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:42:59.702012+0000) 2024-03-04T16:43:00.685 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:43:00.701+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:43:00.702260+0000) 2024-03-04T16:43:01.685 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:43:01.701+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:43:01.702534+0000) 2024-03-04T16:43:02.685 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:43:02.701+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:43:02.702777+0000) 2024-03-04T16:43:03.685 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:43:03.702+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:43:03.703022+0000) 2024-03-04T16:43:04.685 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:43:04.702+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:43:04.703218+0000) 2024-03-04T16:43:05.685 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:43:05.702+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:43:05.703461+0000) 2024-03-04T16:43:06.686 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:43:06.702+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:43:06.703724+0000) 2024-03-04T16:43:07.686 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:43:07.703+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:43:07.703990+0000) 2024-03-04T16:43:08.686 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:43:08.703+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:43:08.704180+0000) 2024-03-04T16:43:09.686 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:43:09.703+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:43:09.704377+0000) 2024-03-04T16:43:10.687 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:43:10.703+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:43:10.704688+0000) 2024-03-04T16:43:11.687 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:43:11.704+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:43:11.704939+0000) 2024-03-04T16:43:12.687 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:43:12.704+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:43:12.705184+0000) 2024-03-04T16:43:13.687 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:43:13.704+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:43:13.705424+0000) 2024-03-04T16:43:14.688 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:43:14.704+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:43:14.705625+0000) 2024-03-04T16:43:15.688 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:43:15.705+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:43:15.705918+0000) 2024-03-04T16:43:16.688 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:43:16.705+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:43:16.706166+0000) 2024-03-04T16:43:17.688 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:43:17.705+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:43:17.706410+0000) 2024-03-04T16:43:18.689 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:43:18.706+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:43:18.706652+0000) 2024-03-04T16:43:19.689 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:43:19.706+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:43:19.706893+0000) 2024-03-04T16:43:20.689 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:43:20.706+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:43:20.707127+0000) 2024-03-04T16:43:21.689 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:43:21.706+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:43:21.707381+0000) 2024-03-04T16:43:21.778 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:43:21.786 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:43:22.690 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:43:22.707+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:43:22.707627+0000) 2024-03-04T16:43:23.690 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:43:23.707+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:43:23.707924+0000) 2024-03-04T16:43:24.690 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:43:24.707+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:43:24.708115+0000) 2024-03-04T16:43:25.690 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:43:25.707+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:43:25.708353+0000) 2024-03-04T16:43:26.690 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:43:26.707+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:43:26.708545+0000) 2024-03-04T16:43:27.691 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:43:27.708+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:43:27.708790+0000) 2024-03-04T16:43:28.692 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:43:28.708+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:43:28.709034+0000) 2024-03-04T16:43:29.691 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:43:29.708+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:43:29.709226+0000) 2024-03-04T16:43:30.691 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:43:30.709+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:43:30.709470+0000) 2024-03-04T16:43:31.691 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:43:31.709+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:43:31.709722+0000) 2024-03-04T16:43:32.692 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:43:32.709+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:43:32.709970+0000) 2024-03-04T16:43:33.692 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:43:33.709+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:43:33.710166+0000) 2024-03-04T16:43:34.692 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:43:34.709+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:43:34.710347+0000) 2024-03-04T16:43:35.692 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:43:35.709+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:43:35.710591+0000) 2024-03-04T16:43:36.693 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:43:36.710+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:43:36.710880+0000) 2024-03-04T16:43:37.693 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:43:37.710+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:43:37.711154+0000) 2024-03-04T16:43:38.696 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:43:38.711+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:43:38.711422+0000) 2024-03-04T16:43:39.693 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:43:39.710+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:43:39.711699+0000) 2024-03-04T16:43:40.694 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:43:40.711+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:43:40.711956+0000) 2024-03-04T16:43:41.694 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:43:41.711+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:43:41.712207+0000) 2024-03-04T16:43:42.694 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:43:42.712+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:43:42.712457+0000) 2024-03-04T16:43:43.694 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:43:43.712+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:43:43.712697+0000) 2024-03-04T16:43:44.695 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:43:44.712+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:43:44.712936+0000) 2024-03-04T16:43:45.695 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:43:45.713+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:43:45.713172+0000) 2024-03-04T16:43:46.695 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:43:46.713+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:43:46.713416+0000) 2024-03-04T16:43:47.695 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:43:47.713+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:43:47.713659+0000) 2024-03-04T16:43:48.696 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:43:48.712+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:43:48.713947+0000) 2024-03-04T16:43:49.696 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:43:49.713+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:43:49.714182+0000) 2024-03-04T16:43:50.696 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:43:50.713+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:43:50.714419+0000) 2024-03-04T16:43:51.696 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:43:51.713+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:43:51.714707+0000) 2024-03-04T16:43:51.999 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:43:52.007 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:43:52.696 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:43:52.713+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:43:52.714954+0000) 2024-03-04T16:43:53.697 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:43:53.714+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:43:53.715200+0000) 2024-03-04T16:43:54.697 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:43:54.714+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:43:54.715441+0000) 2024-03-04T16:43:55.697 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:43:55.714+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:43:55.715690+0000) 2024-03-04T16:43:56.698 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:43:56.714+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:43:56.715940+0000) 2024-03-04T16:43:57.698 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:43:57.715+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:43:57.716186+0000) 2024-03-04T16:43:58.698 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:43:58.715+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:43:58.716434+0000) 2024-03-04T16:43:59.698 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:43:59.715+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:43:59.716691+0000) 2024-03-04T16:44:00.698 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:44:00.716+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:44:00.716975+0000) 2024-03-04T16:44:01.699 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:44:01.716+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:44:01.717229+0000) 2024-03-04T16:44:02.699 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:44:02.716+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:44:02.717476+0000) 2024-03-04T16:44:03.699 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:44:03.716+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:44:03.717723+0000) 2024-03-04T16:44:04.699 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:44:04.717+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:44:04.718000+0000) 2024-03-04T16:44:05.700 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:44:05.717+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:44:05.718239+0000) 2024-03-04T16:44:06.700 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:44:06.717+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:44:06.718484+0000) 2024-03-04T16:44:07.700 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:44:07.717+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:44:07.718746+0000) 2024-03-04T16:44:08.703 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:44:08.718+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:44:08.718985+0000) 2024-03-04T16:44:09.701 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:44:09.718+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:44:09.719226+0000) 2024-03-04T16:44:10.701 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:44:10.718+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:44:10.719468+0000) 2024-03-04T16:44:11.701 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:44:11.718+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:44:11.719726+0000) 2024-03-04T16:44:12.701 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:44:12.719+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:44:12.719971+0000) 2024-03-04T16:44:13.701 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:44:13.719+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:44:13.720157+0000) 2024-03-04T16:44:14.702 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:44:14.719+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:44:14.720401+0000) 2024-03-04T16:44:15.702 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:44:15.719+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:44:15.720636+0000) 2024-03-04T16:44:16.702 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:44:16.720+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:44:16.720935+0000) 2024-03-04T16:44:17.702 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:44:17.720+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:44:17.721179+0000) 2024-03-04T16:44:18.703 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:44:18.720+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:44:18.721427+0000) 2024-03-04T16:44:19.703 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:44:19.720+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:44:19.721686+0000) 2024-03-04T16:44:20.703 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:44:20.721+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:44:20.721917+0000) 2024-03-04T16:44:21.703 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:44:21.721+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:44:21.722171+0000) 2024-03-04T16:44:22.229 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:44:22.236 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:44:22.704 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:44:22.721+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:44:22.722405+0000) 2024-03-04T16:44:23.704 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:44:23.722+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:44:23.722656+0000) 2024-03-04T16:44:24.704 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:44:24.722+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:44:24.722941+0000) 2024-03-04T16:44:25.704 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:44:25.722+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:44:25.723180+0000) 2024-03-04T16:44:26.705 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:44:26.722+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:44:26.723441+0000) 2024-03-04T16:44:27.705 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:44:27.722+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:44:27.723681+0000) 2024-03-04T16:44:28.705 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:44:28.723+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:44:28.723927+0000) 2024-03-04T16:44:29.705 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:44:29.723+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:44:29.724130+0000) 2024-03-04T16:44:30.706 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:44:30.723+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:44:30.724375+0000) 2024-03-04T16:44:31.706 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:44:31.724+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:44:31.724656+0000) 2024-03-04T16:44:32.706 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:44:32.724+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:44:32.724947+0000) 2024-03-04T16:44:33.706 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:44:33.724+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:44:33.725192+0000) 2024-03-04T16:44:34.707 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:44:34.725+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:44:34.725432+0000) 2024-03-04T16:44:35.707 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:44:35.725+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:44:35.725683+0000) 2024-03-04T16:44:36.708 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:44:36.725+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:44:36.725938+0000) 2024-03-04T16:44:37.707 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:44:37.725+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:44:37.726183+0000) 2024-03-04T16:44:38.708 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:44:38.726+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:44:38.726440+0000) 2024-03-04T16:44:39.708 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:44:39.726+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:44:39.726699+0000) 2024-03-04T16:44:40.708 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:44:40.726+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:44:40.726937+0000) 2024-03-04T16:44:41.708 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:44:41.726+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:44:41.727193+0000) 2024-03-04T16:44:42.709 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:44:42.726+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:44:42.727375+0000) 2024-03-04T16:44:43.709 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:44:43.727+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:44:43.727618+0000) 2024-03-04T16:44:44.709 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:44:44.727+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:44:44.727896+0000) 2024-03-04T16:44:45.709 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:44:45.727+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:44:45.728134+0000) 2024-03-04T16:44:46.709 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:44:46.728+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:44:46.728370+0000) 2024-03-04T16:44:47.710 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:44:47.727+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:44:47.728633+0000) 2024-03-04T16:44:48.710 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:44:48.728+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:44:48.728864+0000) 2024-03-04T16:44:49.710 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:44:49.728+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:44:49.729105+0000) 2024-03-04T16:44:50.710 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:44:50.729+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:44:50.729340+0000) 2024-03-04T16:44:51.711 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:44:51.729+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:44:51.729595+0000) 2024-03-04T16:44:52.359 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:44:52.367 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:44:52.711 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:44:52.728+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:44:52.729865+0000) 2024-03-04T16:44:53.711 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:44:53.728+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:44:53.730108+0000) 2024-03-04T16:44:54.711 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:44:54.729+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:44:54.730341+0000) 2024-03-04T16:44:55.711 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:44:55.729+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:44:55.730586+0000) 2024-03-04T16:44:56.712 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:44:56.729+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:44:56.730857+0000) 2024-03-04T16:44:57.712 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:44:57.729+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:44:57.731119+0000) 2024-03-04T16:44:58.712 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:44:58.730+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:44:58.731356+0000) 2024-03-04T16:44:59.713 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:44:59.730+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:44:59.731597+0000) 2024-03-04T16:45:00.713 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:45:00.730+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:45:00.731864+0000) 2024-03-04T16:45:01.713 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:45:01.731+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:45:01.732140+0000) 2024-03-04T16:45:02.713 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:45:02.731+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:45:02.732373+0000) 2024-03-04T16:45:03.714 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:45:03.731+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:45:03.732607+0000) 2024-03-04T16:45:04.714 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:45:04.732+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:45:04.732877+0000) 2024-03-04T16:45:05.714 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:45:05.732+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:45:05.733127+0000) 2024-03-04T16:45:06.714 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:45:06.732+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:45:06.733360+0000) 2024-03-04T16:45:07.715 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:45:07.732+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:45:07.733605+0000) 2024-03-04T16:45:08.715 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:45:08.733+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:45:08.733901+0000) 2024-03-04T16:45:09.715 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:45:09.733+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:45:09.734164+0000) 2024-03-04T16:45:10.715 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:45:10.733+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:45:10.734395+0000) 2024-03-04T16:45:11.715 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:45:11.733+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:45:11.734649+0000) 2024-03-04T16:45:12.716 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:45:12.734+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:45:12.734925+0000) 2024-03-04T16:45:13.716 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:45:13.734+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:45:13.735170+0000) 2024-03-04T16:45:14.716 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:45:14.734+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:45:14.735400+0000) 2024-03-04T16:45:15.717 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:45:15.734+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:45:15.735640+0000) 2024-03-04T16:45:16.717 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:45:16.735+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:45:16.735927+0000) 2024-03-04T16:45:17.717 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:45:17.735+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:45:17.736126+0000) 2024-03-04T16:45:18.717 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:45:18.735+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:45:18.736364+0000) 2024-03-04T16:45:19.717 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:45:19.735+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:45:19.736552+0000) 2024-03-04T16:45:20.718 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:45:20.736+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:45:20.736782+0000) 2024-03-04T16:45:21.718 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:45:21.736+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:45:21.737046+0000) 2024-03-04T16:45:22.580 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:45:22.588 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:45:22.718 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:45:22.736+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:45:22.737276+0000) 2024-03-04T16:45:23.718 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:45:23.736+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:45:23.737521+0000) 2024-03-04T16:45:24.719 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:45:24.737+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:45:24.737756+0000) 2024-03-04T16:45:25.719 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:45:25.737+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:45:25.737996+0000) 2024-03-04T16:45:26.719 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:45:26.737+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:45:26.738227+0000) 2024-03-04T16:45:27.719 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:45:27.737+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:45:27.738479+0000) 2024-03-04T16:45:28.719 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:45:28.738+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:45:28.738712+0000) 2024-03-04T16:45:29.720 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:45:29.738+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:45:29.738952+0000) 2024-03-04T16:45:30.720 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:45:30.738+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:45:30.739187+0000) 2024-03-04T16:45:31.720 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:45:31.738+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:45:31.739443+0000) 2024-03-04T16:45:32.720 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:45:32.739+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:45:32.739675+0000) 2024-03-04T16:45:33.721 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:45:33.739+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:45:33.739953+0000) 2024-03-04T16:45:34.721 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:45:34.739+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:45:34.740190+0000) 2024-03-04T16:45:35.721 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:45:35.739+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:45:35.740430+0000) 2024-03-04T16:45:36.721 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:45:36.740+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:45:36.740667+0000) 2024-03-04T16:45:37.721 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:45:37.740+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:45:37.740943+0000) 2024-03-04T16:45:38.722 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:45:38.740+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:45:38.741182+0000) 2024-03-04T16:45:39.722 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:45:39.741+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:45:39.741453+0000) 2024-03-04T16:45:40.722 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:45:40.741+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:45:40.741710+0000) 2024-03-04T16:45:41.722 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:45:41.741+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:45:41.741965+0000) 2024-03-04T16:45:42.723 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:45:42.741+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:45:42.742207+0000) 2024-03-04T16:45:43.723 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:45:43.742+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:45:43.742450+0000) 2024-03-04T16:45:44.723 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:45:44.742+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:45:44.742683+0000) 2024-03-04T16:45:45.724 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:45:45.742+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:45:45.743018+0000) 2024-03-04T16:45:46.724 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:45:46.742+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:45:46.743257+0000) 2024-03-04T16:45:47.724 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:45:47.743+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:45:47.743508+0000) 2024-03-04T16:45:48.724 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:45:48.743+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:45:48.743742+0000) 2024-03-04T16:45:49.724 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:45:49.743+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:45:49.743980+0000) 2024-03-04T16:45:50.725 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:45:50.743+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:45:50.744165+0000) 2024-03-04T16:45:51.725 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:45:51.744+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:45:51.744451+0000) 2024-03-04T16:45:52.725 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:45:52.744+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:45:52.744685+0000) 2024-03-04T16:45:52.793 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:45:52.801 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:45:53.725 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:45:53.744+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:45:53.744963+0000) 2024-03-04T16:45:54.726 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:45:54.745+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:45:54.745195+0000) 2024-03-04T16:45:55.726 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:45:55.745+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:45:55.745443+0000) 2024-03-04T16:45:56.726 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:45:56.744+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:45:56.745678+0000) 2024-03-04T16:45:57.726 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:45:57.744+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:45:57.745944+0000) 2024-03-04T16:45:58.727 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:45:58.744+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:45:58.746179+0000) 2024-03-04T16:45:59.727 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:45:59.745+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:45:59.746428+0000) 2024-03-04T16:46:00.728 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:46:00.745+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:46:00.746661+0000) 2024-03-04T16:46:01.727 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:46:01.745+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:46:01.746951+0000) 2024-03-04T16:46:02.727 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:46:02.745+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:46:02.747138+0000) 2024-03-04T16:46:03.728 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:46:03.746+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:46:03.747381+0000) 2024-03-04T16:46:04.728 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:46:04.746+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:46:04.747614+0000) 2024-03-04T16:46:05.728 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:46:05.746+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:46:05.747890+0000) 2024-03-04T16:46:06.728 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:46:06.747+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:46:06.748129+0000) 2024-03-04T16:46:07.729 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:46:07.747+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:46:07.748395+0000) 2024-03-04T16:46:08.729 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:46:08.747+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:46:08.748630+0000) 2024-03-04T16:46:09.729 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:46:09.748+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:46:09.748928+0000) 2024-03-04T16:46:10.729 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:46:10.748+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:46:10.749110+0000) 2024-03-04T16:46:11.730 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:46:11.748+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:46:11.749382+0000) 2024-03-04T16:46:12.730 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:46:12.748+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:46:12.749626+0000) 2024-03-04T16:46:13.730 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:46:13.749+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:46:13.749932+0000) 2024-03-04T16:46:14.731 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:46:14.749+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:46:14.750188+0000) 2024-03-04T16:46:15.731 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:46:15.749+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:46:15.750447+0000) 2024-03-04T16:46:16.731 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:46:16.749+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:46:16.750683+0000) 2024-03-04T16:46:17.731 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:46:17.750+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:46:17.750935+0000) 2024-03-04T16:46:18.731 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:46:18.750+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:46:18.751162+0000) 2024-03-04T16:46:19.732 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:46:19.750+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:46:19.751405+0000) 2024-03-04T16:46:20.732 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:46:20.750+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:46:20.751637+0000) 2024-03-04T16:46:21.732 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:46:21.751+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:46:21.751919+0000) 2024-03-04T16:46:22.732 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:46:22.751+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:46:22.752155+0000) 2024-03-04T16:46:23.007 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:46:23.015 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:46:23.733 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:46:23.751+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:46:23.752401+0000) 2024-03-04T16:46:24.733 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:46:24.751+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:46:24.752638+0000) 2024-03-04T16:46:25.733 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:46:25.752+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:46:25.752929+0000) 2024-03-04T16:46:26.733 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:46:26.752+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:46:26.753165+0000) 2024-03-04T16:46:27.734 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:46:27.752+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:46:27.753429+0000) 2024-03-04T16:46:28.734 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:46:28.753+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:46:28.753670+0000) 2024-03-04T16:46:29.734 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:46:29.753+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:46:29.753888+0000) 2024-03-04T16:46:30.734 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:46:30.753+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:46:30.754120+0000) 2024-03-04T16:46:31.734 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:46:31.753+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:46:31.754368+0000) 2024-03-04T16:46:32.735 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:46:32.753+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:46:32.754618+0000) 2024-03-04T16:46:33.735 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:46:33.754+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:46:33.754924+0000) 2024-03-04T16:46:34.735 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:46:34.754+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:46:34.755163+0000) 2024-03-04T16:46:35.735 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:46:35.754+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:46:35.755400+0000) 2024-03-04T16:46:36.736 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:46:36.755+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:46:36.755635+0000) 2024-03-04T16:46:37.736 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:46:37.755+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:46:37.755931+0000) 2024-03-04T16:46:38.736 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:46:38.755+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:46:38.756171+0000) 2024-03-04T16:46:39.736 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:46:39.755+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:46:39.756407+0000) 2024-03-04T16:46:40.737 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:46:40.756+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:46:40.756658+0000) 2024-03-04T16:46:41.737 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:46:41.756+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:46:41.756936+0000) 2024-03-04T16:46:42.737 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:46:42.756+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:46:42.757174+0000) 2024-03-04T16:46:43.737 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:46:43.757+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:46:43.757438+0000) 2024-03-04T16:46:44.738 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:46:44.757+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:46:44.757674+0000) 2024-03-04T16:46:45.738 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:46:45.757+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:46:45.757927+0000) 2024-03-04T16:46:46.738 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:46:46.757+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:46:46.758159+0000) 2024-03-04T16:46:47.738 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:46:47.758+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:46:47.758404+0000) 2024-03-04T16:46:48.739 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:46:48.758+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:46:48.758639+0000) 2024-03-04T16:46:49.739 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:46:49.758+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:46:49.758946+0000) 2024-03-04T16:46:50.739 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:46:50.758+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:46:50.759131+0000) 2024-03-04T16:46:51.739 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:46:51.758+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:46:51.759331+0000) 2024-03-04T16:46:52.739 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:46:52.759+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:46:52.759556+0000) 2024-03-04T16:46:53.153 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:46:53.161 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:46:53.740 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:46:53.759+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:46:53.759799+0000) 2024-03-04T16:46:54.740 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:46:54.759+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:46:54.760092+0000) 2024-03-04T16:46:55.740 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:46:55.760+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:46:55.760328+0000) 2024-03-04T16:46:56.740 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:46:56.760+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:46:56.760569+0000) 2024-03-04T16:46:57.741 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:46:57.760+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:46:57.760810+0000) 2024-03-04T16:46:58.741 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:46:58.760+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:46:58.761099+0000) 2024-03-04T16:46:59.741 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:46:59.761+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:46:59.761350+0000) 2024-03-04T16:47:00.741 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:47:00.760+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:47:00.761585+0000) 2024-03-04T16:47:01.742 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:47:01.760+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:47:01.761870+0000) 2024-03-04T16:47:02.742 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:47:02.761+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:47:02.762113+0000) 2024-03-04T16:47:03.742 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:47:03.761+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:47:03.762366+0000) 2024-03-04T16:47:04.742 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:47:04.761+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:47:04.762607+0000) 2024-03-04T16:47:05.743 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:47:05.761+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:47:05.762915+0000) 2024-03-04T16:47:06.743 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:47:06.762+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:47:06.763149+0000) 2024-03-04T16:47:07.743 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:47:07.761+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:47:07.763339+0000) 2024-03-04T16:47:08.743 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:47:08.762+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:47:08.763574+0000) 2024-03-04T16:47:09.744 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:47:09.762+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:47:09.763812+0000) 2024-03-04T16:47:10.744 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:47:10.763+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:47:10.764124+0000) 2024-03-04T16:47:11.744 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:47:11.763+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:47:11.764393+0000) 2024-03-04T16:47:12.744 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:47:12.763+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:47:12.764630+0000) 2024-03-04T16:47:13.745 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:47:13.764+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:47:13.764928+0000) 2024-03-04T16:47:14.745 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:47:14.764+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:47:14.765160+0000) 2024-03-04T16:47:15.745 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:47:15.764+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:47:15.765405+0000) 2024-03-04T16:47:16.746 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:47:16.764+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:47:16.765641+0000) 2024-03-04T16:47:17.746 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:47:17.765+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:47:17.765949+0000) 2024-03-04T16:47:18.746 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:47:18.765+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:47:18.766185+0000) 2024-03-04T16:47:19.746 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:47:19.765+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:47:19.766423+0000) 2024-03-04T16:47:20.746 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:47:20.765+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:47:20.766659+0000) 2024-03-04T16:47:21.747 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:47:21.766+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:47:21.767019+0000) 2024-03-04T16:47:22.747 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:47:22.766+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:47:22.767287+0000) 2024-03-04T16:47:23.300 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:47:23.309 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:47:23.747 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:47:23.766+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:47:23.767535+0000) 2024-03-04T16:47:24.747 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:47:24.766+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:47:24.767761+0000) 2024-03-04T16:47:25.748 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:47:25.767+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:47:25.767999+0000) 2024-03-04T16:47:26.748 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:47:26.767+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:47:26.768233+0000) 2024-03-04T16:47:27.748 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:47:27.767+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:47:27.768482+0000) 2024-03-04T16:47:28.748 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:47:28.768+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:47:28.768715+0000) 2024-03-04T16:47:29.749 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:47:29.768+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:47:29.768959+0000) 2024-03-04T16:47:30.749 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:47:30.768+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:47:30.769191+0000) 2024-03-04T16:47:31.749 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:47:31.768+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:47:31.769441+0000) 2024-03-04T16:47:32.749 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:47:32.769+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:47:32.769676+0000) 2024-03-04T16:47:33.752 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:47:33.769+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:47:33.770009+0000) 2024-03-04T16:47:34.750 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:47:34.769+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:47:34.770256+0000) 2024-03-04T16:47:35.750 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:47:35.769+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:47:35.770494+0000) 2024-03-04T16:47:36.750 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:47:36.770+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:47:36.770726+0000) 2024-03-04T16:47:37.751 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:47:37.770+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:47:37.770969+0000) 2024-03-04T16:47:38.751 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:47:38.770+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:47:38.771205+0000) 2024-03-04T16:47:39.751 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:47:39.770+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:47:39.771457+0000) 2024-03-04T16:47:40.751 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:47:40.771+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:47:40.771713+0000) 2024-03-04T16:47:41.752 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:47:41.771+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:47:41.771984+0000) 2024-03-04T16:47:42.752 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:47:42.771+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:47:42.772220+0000) 2024-03-04T16:47:43.752 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:47:43.772+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:47:43.772463+0000) 2024-03-04T16:47:44.752 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:47:44.772+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:47:44.772699+0000) 2024-03-04T16:47:45.752 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:47:45.772+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:47:45.772939+0000) 2024-03-04T16:47:46.753 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:47:46.772+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:47:46.773173+0000) 2024-03-04T16:47:47.753 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:47:47.773+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:47:47.773417+0000) 2024-03-04T16:47:48.753 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:47:48.773+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:47:48.773652+0000) 2024-03-04T16:47:49.753 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:47:49.773+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:47:49.773932+0000) 2024-03-04T16:47:50.754 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:47:50.773+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:47:50.774159+0000) 2024-03-04T16:47:51.754 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:47:51.774+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:47:51.774402+0000) 2024-03-04T16:47:52.754 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:47:52.774+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:47:52.774595+0000) 2024-03-04T16:47:53.512 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:47:53.521 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:47:53.789 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:47:53.774+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:47:53.774885+0000) 2024-03-04T16:47:54.755 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:47:54.774+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:47:54.775124+0000) 2024-03-04T16:47:55.755 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:47:55.775+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:47:55.775378+0000) 2024-03-04T16:47:56.756 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:47:56.775+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:47:56.775614+0000) 2024-03-04T16:47:57.755 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:47:57.775+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:47:57.775927+0000) 2024-03-04T16:47:58.756 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:47:58.775+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:47:58.776171+0000) 2024-03-04T16:47:59.756 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:47:59.775+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:47:59.776376+0000) 2024-03-04T16:48:00.756 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:48:00.776+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:48:00.776645+0000) 2024-03-04T16:48:01.756 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:48:01.776+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:48:01.776928+0000) 2024-03-04T16:48:02.757 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:48:02.777+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:48:02.777163+0000) 2024-03-04T16:48:03.757 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:48:03.776+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:48:03.777405+0000) 2024-03-04T16:48:04.757 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:48:04.775+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:48:04.777650+0000) 2024-03-04T16:48:05.757 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:48:05.776+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:48:05.777935+0000) 2024-03-04T16:48:06.758 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:48:06.777+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:48:06.778186+0000) 2024-03-04T16:48:07.758 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:48:07.777+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:48:07.778433+0000) 2024-03-04T16:48:08.758 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:48:08.777+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:48:08.778667+0000) 2024-03-04T16:48:09.758 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:48:09.777+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:48:09.778930+0000) 2024-03-04T16:48:10.759 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:48:10.777+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:48:10.779159+0000) 2024-03-04T16:48:11.760 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:48:11.777+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:48:11.779427+0000) 2024-03-04T16:48:12.759 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:48:12.778+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:48:12.779661+0000) 2024-03-04T16:48:13.759 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:48:13.778+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:48:13.779948+0000) 2024-03-04T16:48:14.760 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:48:14.779+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:48:14.780186+0000) 2024-03-04T16:48:15.760 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:48:15.779+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:48:15.780415+0000) 2024-03-04T16:48:16.760 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:48:16.779+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:48:16.780649+0000) 2024-03-04T16:48:17.760 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:48:17.780+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:48:17.780932+0000) 2024-03-04T16:48:18.761 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:48:18.780+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:48:18.781170+0000) 2024-03-04T16:48:19.761 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:48:19.780+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:48:19.781460+0000) 2024-03-04T16:48:20.761 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:48:20.780+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:48:20.781705+0000) 2024-03-04T16:48:21.761 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:48:21.781+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:48:21.781962+0000) 2024-03-04T16:48:22.761 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:48:22.781+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:48:22.782142+0000) 2024-03-04T16:48:23.652 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:48:23.660 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:48:23.761 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:48:23.781+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:48:23.782313+0000) 2024-03-04T16:48:24.762 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:48:24.781+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:48:24.782530+0000) 2024-03-04T16:48:25.762 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:48:25.781+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:48:25.782727+0000) 2024-03-04T16:48:26.762 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:48:26.782+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:48:26.782952+0000) 2024-03-04T16:48:27.763 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:48:27.782+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:48:27.783213+0000) 2024-03-04T16:48:28.763 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:48:28.782+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:48:28.783464+0000) 2024-03-04T16:48:29.763 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:48:29.782+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:48:29.783709+0000) 2024-03-04T16:48:30.763 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:48:30.783+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:48:30.783988+0000) 2024-03-04T16:48:31.764 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:48:31.783+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:48:31.784258+0000) 2024-03-04T16:48:32.764 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:48:32.783+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:48:32.784528+0000) 2024-03-04T16:48:33.764 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:48:33.783+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:48:33.784725+0000) 2024-03-04T16:48:34.764 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:48:34.784+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:48:34.785012+0000) 2024-03-04T16:48:35.764 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:48:35.784+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:48:35.785208+0000) 2024-03-04T16:48:36.765 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:48:36.784+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:48:36.785391+0000) 2024-03-04T16:48:37.765 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:48:37.784+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:48:37.785629+0000) 2024-03-04T16:48:38.765 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:48:38.785+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:48:38.785889+0000) 2024-03-04T16:48:39.765 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:48:39.785+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:48:39.786132+0000) 2024-03-04T16:48:40.765 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:48:40.785+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:48:40.786325+0000) 2024-03-04T16:48:41.766 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:48:41.786+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:48:41.786576+0000) 2024-03-04T16:48:42.766 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:48:42.786+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:48:42.786857+0000) 2024-03-04T16:48:43.766 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:48:43.786+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:48:43.787108+0000) 2024-03-04T16:48:44.766 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:48:44.786+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:48:44.787347+0000) 2024-03-04T16:48:45.767 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:48:45.786+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:48:45.787521+0000) 2024-03-04T16:48:46.767 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:48:46.787+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:48:46.787777+0000) 2024-03-04T16:48:47.767 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:48:47.787+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:48:47.788021+0000) 2024-03-04T16:48:48.767 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:48:48.787+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:48:48.788255+0000) 2024-03-04T16:48:49.767 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:48:49.788+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:48:49.788495+0000) 2024-03-04T16:48:50.768 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:48:50.787+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:48:50.788742+0000) 2024-03-04T16:48:51.768 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:48:51.788+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:48:51.788992+0000) 2024-03-04T16:48:52.768 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:48:52.788+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:48:52.789227+0000) 2024-03-04T16:48:53.768 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:48:53.789+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:48:53.789489+0000) 2024-03-04T16:48:53.831 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:48:53.839 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:48:54.769 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:48:54.789+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:48:54.789723+0000) 2024-03-04T16:48:55.769 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:48:55.789+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:48:55.789962+0000) 2024-03-04T16:48:56.769 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:48:56.789+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:48:56.790191+0000) 2024-03-04T16:48:57.769 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:48:57.789+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:48:57.790433+0000) 2024-03-04T16:48:58.770 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:48:58.790+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:48:58.790669+0000) 2024-03-04T16:48:59.770 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:48:59.790+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:48:59.790951+0000) 2024-03-04T16:49:00.770 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:49:00.790+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:49:00.791183+0000) 2024-03-04T16:49:01.770 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:49:01.791+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:49:01.791457+0000) 2024-03-04T16:49:02.771 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:49:02.791+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:49:02.791693+0000) 2024-03-04T16:49:03.771 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:49:03.791+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:49:03.791964+0000) 2024-03-04T16:49:04.771 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:49:04.791+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:49:04.792202+0000) 2024-03-04T16:49:05.771 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:49:05.792+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:49:05.792459+0000) 2024-03-04T16:49:06.772 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:49:06.792+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:49:06.792691+0000) 2024-03-04T16:49:07.772 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:49:07.791+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:49:07.792948+0000) 2024-03-04T16:49:08.772 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:49:08.792+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:49:08.793237+0000) 2024-03-04T16:49:09.772 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:49:09.792+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:49:09.793477+0000) 2024-03-04T16:49:10.773 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:49:10.792+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:49:10.793713+0000) 2024-03-04T16:49:11.773 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:49:11.792+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:49:11.793966+0000) 2024-03-04T16:49:12.773 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:49:12.793+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:49:12.794221+0000) 2024-03-04T16:49:13.773 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:49:13.793+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:49:13.794490+0000) 2024-03-04T16:49:14.774 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:49:14.793+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:49:14.794731+0000) 2024-03-04T16:49:15.774 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:49:15.794+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:49:15.794992+0000) 2024-03-04T16:49:16.774 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:49:16.794+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:49:16.795226+0000) 2024-03-04T16:49:17.774 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:49:17.794+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:49:17.795494+0000) 2024-03-04T16:49:18.775 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:49:18.794+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:49:18.795724+0000) 2024-03-04T16:49:19.775 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:49:19.795+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:49:19.795960+0000) 2024-03-04T16:49:20.775 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:49:20.795+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:49:20.796186+0000) 2024-03-04T16:49:21.775 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:49:21.795+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:49:21.796394+0000) 2024-03-04T16:49:22.775 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:49:22.795+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:49:22.796624+0000) 2024-03-04T16:49:23.776 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:49:23.796+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:49:23.796929+0000) 2024-03-04T16:49:23.970 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:49:23.978 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:49:24.776 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:49:24.796+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:49:24.797164+0000) 2024-03-04T16:49:25.776 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:49:25.796+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:49:25.797399+0000) 2024-03-04T16:49:26.777 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:49:26.796+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:49:26.797632+0000) 2024-03-04T16:49:27.777 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:49:27.797+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:49:27.797911+0000) 2024-03-04T16:49:28.777 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:49:28.797+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:49:28.798151+0000) 2024-03-04T16:49:29.777 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:49:29.797+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:49:29.798395+0000) 2024-03-04T16:49:30.779 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:49:30.797+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:49:30.798625+0000) 2024-03-04T16:49:31.778 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:49:31.798+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:49:31.798858+0000) 2024-03-04T16:49:32.778 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:49:32.798+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:49:32.799057+0000) 2024-03-04T16:49:33.778 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:49:33.798+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:49:33.799305+0000) 2024-03-04T16:49:34.778 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:49:34.798+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:49:34.799545+0000) 2024-03-04T16:49:35.778 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:49:35.799+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:49:35.799781+0000) 2024-03-04T16:49:36.779 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:49:36.799+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:49:36.800014+0000) 2024-03-04T16:49:37.779 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:49:37.799+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:49:37.800257+0000) 2024-03-04T16:49:38.779 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:49:38.799+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:49:38.800517+0000) 2024-03-04T16:49:39.780 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:49:39.800+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:49:39.800792+0000) 2024-03-04T16:49:40.780 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:49:40.800+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:49:40.801064+0000) 2024-03-04T16:49:41.780 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:49:41.800+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:49:41.801298+0000) 2024-03-04T16:49:42.780 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:49:42.800+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:49:42.801534+0000) 2024-03-04T16:49:43.781 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:49:43.801+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:49:43.801798+0000) 2024-03-04T16:49:44.781 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:49:44.801+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:49:44.802043+0000) 2024-03-04T16:49:45.781 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:49:45.801+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:49:45.802282+0000) 2024-03-04T16:49:46.781 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:49:46.802+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:49:46.802517+0000) 2024-03-04T16:49:47.781 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:49:47.802+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:49:47.802762+0000) 2024-03-04T16:49:48.782 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:49:48.802+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:49:48.802999+0000) 2024-03-04T16:49:49.784 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:49:49.802+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:49:49.803179+0000) 2024-03-04T16:49:50.782 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:49:50.802+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:49:50.803419+0000) 2024-03-04T16:49:51.782 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:49:51.803+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:49:51.803666+0000) 2024-03-04T16:49:52.782 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:49:52.803+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:49:52.803946+0000) 2024-03-04T16:49:53.783 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:49:53.803+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:49:53.804197+0000) 2024-03-04T16:49:54.334 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:49:54.342 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:49:54.783 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:49:54.804+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:49:54.804437+0000) 2024-03-04T16:49:55.784 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:49:55.804+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:49:55.804681+0000) 2024-03-04T16:49:56.784 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:49:56.804+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:49:56.804985+0000) 2024-03-04T16:49:57.784 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:49:57.804+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:49:57.805231+0000) 2024-03-04T16:49:58.784 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:49:58.805+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:49:58.805468+0000) 2024-03-04T16:49:59.784 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:49:59.805+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:49:59.805709+0000) 2024-03-04T16:50:00.785 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:50:00.805+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:50:00.805946+0000) 2024-03-04T16:50:01.785 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:50:01.805+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:50:01.806203+0000) 2024-03-04T16:50:02.785 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:50:02.806+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:50:02.806447+0000) 2024-03-04T16:50:03.785 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:50:03.806+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:50:03.806656+0000) 2024-03-04T16:50:04.785 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:50:04.806+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:50:04.806925+0000) 2024-03-04T16:50:05.786 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:50:05.806+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:50:05.807200+0000) 2024-03-04T16:50:06.786 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:50:06.807+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:50:06.807438+0000) 2024-03-04T16:50:07.786 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:50:07.807+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:50:07.807683+0000) 2024-03-04T16:50:08.786 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:50:08.807+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:50:08.807921+0000) 2024-03-04T16:50:09.787 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:50:09.808+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:50:09.808162+0000) 2024-03-04T16:50:10.787 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:50:10.808+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:50:10.808404+0000) 2024-03-04T16:50:11.787 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:50:11.807+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:50:11.808653+0000) 2024-03-04T16:50:12.787 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:50:12.807+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:50:12.808915+0000) 2024-03-04T16:50:13.788 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:50:13.807+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:50:13.809170+0000) 2024-03-04T16:50:14.788 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:50:14.808+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:50:14.809425+0000) 2024-03-04T16:50:15.788 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:50:15.807+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:50:15.809609+0000) 2024-03-04T16:50:16.788 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:50:16.808+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:50:16.809883+0000) 2024-03-04T16:50:17.789 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:50:17.809+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:50:17.810138+0000) 2024-03-04T16:50:18.789 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:50:18.809+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:50:18.810330+0000) 2024-03-04T16:50:19.789 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:50:19.809+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:50:19.810570+0000) 2024-03-04T16:50:20.789 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:50:20.809+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:50:20.810806+0000) 2024-03-04T16:50:21.790 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:50:21.810+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:50:21.811090+0000) 2024-03-04T16:50:22.790 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:50:22.810+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:50:22.811328+0000) 2024-03-04T16:50:23.790 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:50:23.810+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:50:23.811590+0000) 2024-03-04T16:50:24.554 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:50:24.562 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:50:24.790 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:50:24.811+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:50:24.811884+0000) 2024-03-04T16:50:25.791 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:50:25.811+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:50:25.812126+0000) 2024-03-04T16:50:26.791 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:50:26.811+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:50:26.812384+0000) 2024-03-04T16:50:27.791 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:50:27.811+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:50:27.812635+0000) 2024-03-04T16:50:28.791 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:50:28.812+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:50:28.812872+0000) 2024-03-04T16:50:29.792 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:50:29.812+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:50:29.813111+0000) 2024-03-04T16:50:30.792 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:50:30.812+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:50:30.813373+0000) 2024-03-04T16:50:31.792 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:50:31.812+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:50:31.813622+0000) 2024-03-04T16:50:32.792 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:50:32.813+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:50:32.813908+0000) 2024-03-04T16:50:33.792 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:50:33.813+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:50:33.814113+0000) 2024-03-04T16:50:34.793 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:50:34.813+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:50:34.814344+0000) 2024-03-04T16:50:35.793 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:50:35.813+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:50:35.814605+0000) 2024-03-04T16:50:36.794 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:50:36.814+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:50:36.814898+0000) 2024-03-04T16:50:37.793 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:50:37.814+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:50:37.815095+0000) 2024-03-04T16:50:38.794 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:50:38.814+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:50:38.815344+0000) 2024-03-04T16:50:39.794 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:50:39.814+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:50:39.815582+0000) 2024-03-04T16:50:40.794 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:50:40.815+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:50:40.815817+0000) 2024-03-04T16:50:41.794 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:50:41.815+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:50:41.816097+0000) 2024-03-04T16:50:42.795 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:50:42.815+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:50:42.816338+0000) 2024-03-04T16:50:43.795 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:50:43.815+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:50:43.816550+0000) 2024-03-04T16:50:44.795 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:50:44.816+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:50:44.816788+0000) 2024-03-04T16:50:45.795 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:50:45.816+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:50:45.817059+0000) 2024-03-04T16:50:46.796 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:50:46.816+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:50:46.817291+0000) 2024-03-04T16:50:47.796 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:50:47.816+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:50:47.817532+0000) 2024-03-04T16:50:48.796 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:50:48.816+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:50:48.817767+0000) 2024-03-04T16:50:49.796 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:50:49.817+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:50:49.818008+0000) 2024-03-04T16:50:50.797 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:50:50.817+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:50:50.818240+0000) 2024-03-04T16:50:51.797 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:50:51.818+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:50:51.818492+0000) 2024-03-04T16:50:52.797 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:50:52.817+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:50:52.818738+0000) 2024-03-04T16:50:53.797 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:50:53.818+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:50:53.818959+0000) 2024-03-04T16:50:54.768 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:50:54.776 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:50:54.797 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:50:54.818+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:50:54.819169+0000) 2024-03-04T16:50:55.798 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:50:55.818+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:50:55.819419+0000) 2024-03-04T16:50:56.798 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:50:56.818+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:50:56.819610+0000) 2024-03-04T16:50:57.798 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:50:57.819+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:50:57.819889+0000) 2024-03-04T16:50:58.798 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:50:58.819+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:50:58.820091+0000) 2024-03-04T16:50:59.799 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:50:59.819+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:50:59.820327+0000) 2024-03-04T16:51:00.799 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:51:00.819+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:51:00.820570+0000) 2024-03-04T16:51:01.799 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:51:01.820+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:51:01.820852+0000) 2024-03-04T16:51:02.799 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:51:02.820+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:51:02.821096+0000) 2024-03-04T16:51:03.799 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:51:03.820+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:51:03.821297+0000) 2024-03-04T16:51:04.800 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:51:04.820+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:51:04.821547+0000) 2024-03-04T16:51:05.800 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:51:05.821+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:51:05.821793+0000) 2024-03-04T16:51:06.800 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:51:06.821+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:51:06.822064+0000) 2024-03-04T16:51:07.801 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:51:07.822+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:51:07.822309+0000) 2024-03-04T16:51:08.801 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:51:08.821+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:51:08.822497+0000) 2024-03-04T16:51:09.801 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:51:09.822+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:51:09.822740+0000) 2024-03-04T16:51:10.801 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:51:10.822+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:51:10.823001+0000) 2024-03-04T16:51:11.801 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:51:11.823+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:51:11.823259+0000) 2024-03-04T16:51:12.801 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:51:12.822+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:51:12.823445+0000) 2024-03-04T16:51:13.802 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:51:13.822+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:51:13.823637+0000) 2024-03-04T16:51:14.802 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:51:14.822+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:51:14.823814+0000) 2024-03-04T16:51:15.802 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:51:15.822+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:51:15.824094+0000) 2024-03-04T16:51:16.802 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:51:16.822+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:51:16.824286+0000) 2024-03-04T16:51:17.803 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:51:17.823+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:51:17.824485+0000) 2024-03-04T16:51:18.803 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:51:18.822+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:51:18.824714+0000) 2024-03-04T16:51:19.803 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:51:19.823+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:51:19.824960+0000) 2024-03-04T16:51:20.803 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:51:20.823+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:51:20.825148+0000) 2024-03-04T16:51:21.803 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:51:21.824+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:51:21.825394+0000) 2024-03-04T16:51:22.804 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:51:22.824+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:51:22.825582+0000) 2024-03-04T16:51:23.804 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:51:23.823+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:51:23.825785+0000) 2024-03-04T16:51:24.804 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:51:24.825+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:51:24.826018+0000) 2024-03-04T16:51:24.938 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:51:24.946 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:51:25.804 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:51:25.825+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:51:25.826218+0000) 2024-03-04T16:51:26.804 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:51:26.825+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:51:26.826461+0000) 2024-03-04T16:51:27.805 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:51:27.825+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:51:27.826723+0000) 2024-03-04T16:51:28.805 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:51:28.826+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:51:28.826961+0000) 2024-03-04T16:51:29.805 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:51:29.826+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:51:29.827202+0000) 2024-03-04T16:51:30.805 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:51:30.826+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:51:30.827437+0000) 2024-03-04T16:51:31.806 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:51:31.826+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:51:31.827690+0000) 2024-03-04T16:51:32.806 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:51:32.827+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:51:32.827978+0000) 2024-03-04T16:51:33.806 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:51:33.827+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:51:33.828223+0000) 2024-03-04T16:51:34.806 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:51:34.827+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:51:34.828452+0000) 2024-03-04T16:51:35.807 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:51:35.827+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:51:35.828694+0000) 2024-03-04T16:51:36.807 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:51:36.828+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:51:36.828927+0000) 2024-03-04T16:51:37.807 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:51:37.828+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:51:37.829182+0000) 2024-03-04T16:51:38.807 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:51:38.828+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:51:38.829416+0000) 2024-03-04T16:51:39.808 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:51:39.828+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:51:39.829656+0000) 2024-03-04T16:51:40.808 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:51:40.829+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:51:40.829953+0000) 2024-03-04T16:51:41.808 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:51:41.829+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:51:41.830203+0000) 2024-03-04T16:51:42.808 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:51:42.829+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:51:42.830440+0000) 2024-03-04T16:51:43.808 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:51:43.829+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:51:43.830636+0000) 2024-03-04T16:51:44.809 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:51:44.830+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:51:44.830917+0000) 2024-03-04T16:51:45.809 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:51:45.830+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:51:45.831155+0000) 2024-03-04T16:51:46.809 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:51:46.830+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:51:46.831422+0000) 2024-03-04T16:51:47.810 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:51:47.831+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:51:47.831684+0000) 2024-03-04T16:51:48.810 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:51:48.831+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:51:48.831949+0000) 2024-03-04T16:51:49.810 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:51:49.831+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:51:49.832183+0000) 2024-03-04T16:51:50.810 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:51:50.831+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:51:50.832417+0000) 2024-03-04T16:51:51.811 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:51:51.832+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:51:51.832672+0000) 2024-03-04T16:51:52.811 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:51:52.832+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:51:52.832926+0000) 2024-03-04T16:51:53.811 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:51:53.832+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:51:53.833172+0000) 2024-03-04T16:51:54.811 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:51:54.832+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:51:54.833399+0000) 2024-03-04T16:51:55.160 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:51:55.168 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:51:55.811 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:51:55.832+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:51:55.833656+0000) 2024-03-04T16:51:56.812 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:51:56.833+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:51:56.833924+0000) 2024-03-04T16:51:57.812 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:51:57.833+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:51:57.834113+0000) 2024-03-04T16:51:58.812 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:51:58.833+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:51:58.834351+0000) 2024-03-04T16:51:59.812 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:51:59.833+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:51:59.834590+0000) 2024-03-04T16:52:00.813 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:52:00.834+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:52:00.834854+0000) 2024-03-04T16:52:01.813 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:52:01.834+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:52:01.835107+0000) 2024-03-04T16:52:02.813 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:52:02.834+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:52:02.835341+0000) 2024-03-04T16:52:03.813 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:52:03.835+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:52:03.835587+0000) 2024-03-04T16:52:04.814 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:52:04.835+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:52:04.835856+0000) 2024-03-04T16:52:05.814 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:52:05.835+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:52:05.836115+0000) 2024-03-04T16:52:06.814 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:52:06.836+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:52:06.836351+0000) 2024-03-04T16:52:07.815 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:52:07.836+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:52:07.836614+0000) 2024-03-04T16:52:08.815 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:52:08.836+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:52:08.836876+0000) 2024-03-04T16:52:09.815 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:52:09.836+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:52:09.837124+0000) 2024-03-04T16:52:10.815 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:52:10.836+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:52:10.837313+0000) 2024-03-04T16:52:11.816 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:52:11.836+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:52:11.837569+0000) 2024-03-04T16:52:12.816 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:52:12.837+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:52:12.837857+0000) 2024-03-04T16:52:13.816 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:52:13.837+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:52:13.838107+0000) 2024-03-04T16:52:14.816 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:52:14.837+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:52:14.838345+0000) 2024-03-04T16:52:15.816 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:52:15.838+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:52:15.838583+0000) 2024-03-04T16:52:16.820 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:52:16.838+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:52:16.838881+0000) 2024-03-04T16:52:17.817 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:52:17.838+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:52:17.839120+0000) 2024-03-04T16:52:18.817 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:52:18.838+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:52:18.839353+0000) 2024-03-04T16:52:19.818 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:52:19.837+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:52:19.839598+0000) 2024-03-04T16:52:20.818 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:52:20.838+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:52:20.839902+0000) 2024-03-04T16:52:21.818 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:52:21.839+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:52:21.840151+0000) 2024-03-04T16:52:22.818 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:52:22.839+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:52:22.840383+0000) 2024-03-04T16:52:23.818 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:52:23.839+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:52:23.840651+0000) 2024-03-04T16:52:24.819 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:52:24.839+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:52:24.840939+0000) 2024-03-04T16:52:25.382 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:52:25.390 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:52:25.819 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:52:25.840+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:52:25.841251+0000) 2024-03-04T16:52:26.819 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:52:26.840+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:52:26.841485+0000) 2024-03-04T16:52:27.820 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:52:27.840+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:52:27.841747+0000) 2024-03-04T16:52:28.820 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:52:28.841+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:52:28.841981+0000) 2024-03-04T16:52:29.820 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:52:29.841+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:52:29.842227+0000) 2024-03-04T16:52:30.820 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:52:30.841+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:52:30.842462+0000) 2024-03-04T16:52:31.820 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:52:31.841+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:52:31.842701+0000) 2024-03-04T16:52:32.821 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:52:32.842+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:52:32.842957+0000) 2024-03-04T16:52:33.824 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:52:33.842+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:52:33.843200+0000) 2024-03-04T16:52:34.821 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:52:34.842+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:52:34.843453+0000) 2024-03-04T16:52:35.821 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:52:35.842+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:52:35.843694+0000) 2024-03-04T16:52:36.822 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:52:36.843+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:52:36.843930+0000) 2024-03-04T16:52:37.822 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:52:37.843+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:52:37.844173+0000) 2024-03-04T16:52:38.822 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:52:38.843+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:52:38.844424+0000) 2024-03-04T16:52:39.822 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:52:39.843+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:52:39.844682+0000) 2024-03-04T16:52:40.823 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:52:40.844+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:52:40.844923+0000) 2024-03-04T16:52:41.823 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:52:41.844+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:52:41.845171+0000) 2024-03-04T16:52:42.823 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:52:42.844+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:52:42.845405+0000) 2024-03-04T16:52:43.823 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:52:43.844+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:52:43.845650+0000) 2024-03-04T16:52:44.823 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:52:44.845+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:52:44.845864+0000) 2024-03-04T16:52:45.823 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:52:45.845+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:52:45.846081+0000) 2024-03-04T16:52:46.824 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:52:46.845+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:52:46.846259+0000) 2024-03-04T16:52:47.824 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:52:47.845+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:52:47.846506+0000) 2024-03-04T16:52:48.824 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:52:48.846+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:52:48.846745+0000) 2024-03-04T16:52:49.825 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:52:49.846+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:52:49.846995+0000) 2024-03-04T16:52:50.825 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:52:50.846+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:52:50.847254+0000) 2024-03-04T16:52:51.825 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:52:51.846+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:52:51.847458+0000) 2024-03-04T16:52:52.825 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:52:52.847+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:52:52.847702+0000) 2024-03-04T16:52:53.825 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:52:53.847+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:52:53.847943+0000) 2024-03-04T16:52:54.826 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:52:54.847+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:52:54.848176+0000) 2024-03-04T16:52:55.586 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:52:55.598 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:52:55.826 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:52:55.847+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:52:55.848378+0000) 2024-03-04T16:52:56.826 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:52:56.848+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:52:56.848613+0000) 2024-03-04T16:52:57.826 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:52:57.848+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:52:57.848915+0000) 2024-03-04T16:52:58.827 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:52:58.848+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:52:58.849151+0000) 2024-03-04T16:52:59.827 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:52:59.848+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:52:59.849394+0000) 2024-03-04T16:53:00.828 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:53:00.848+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:53:00.849632+0000) 2024-03-04T16:53:01.827 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:53:01.849+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:53:01.849940+0000) 2024-03-04T16:53:02.828 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:53:02.849+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:53:02.850175+0000) 2024-03-04T16:53:03.828 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:53:03.850+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:53:03.850416+0000) 2024-03-04T16:53:04.828 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:53:04.849+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:53:04.850606+0000) 2024-03-04T16:53:05.828 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:53:05.850+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:53:05.850893+0000) 2024-03-04T16:53:06.828 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:53:06.850+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:53:06.851130+0000) 2024-03-04T16:53:07.829 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:53:07.850+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:53:07.851366+0000) 2024-03-04T16:53:08.829 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:53:08.850+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:53:08.851548+0000) 2024-03-04T16:53:09.829 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:53:09.850+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:53:09.851743+0000) 2024-03-04T16:53:10.829 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:53:10.851+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:53:10.851978+0000) 2024-03-04T16:53:11.829 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:53:11.851+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:53:11.852237+0000) 2024-03-04T16:53:12.830 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:53:12.852+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:53:12.852472+0000) 2024-03-04T16:53:13.830 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:53:13.852+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:53:13.852726+0000) 2024-03-04T16:53:14.830 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:53:14.852+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:53:14.852961+0000) 2024-03-04T16:53:15.831 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:53:15.853+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:53:15.853214+0000) 2024-03-04T16:53:16.831 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:53:16.853+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:53:16.853451+0000) 2024-03-04T16:53:17.831 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:53:17.853+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:53:17.853697+0000) 2024-03-04T16:53:18.832 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:53:18.853+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:53:18.854171+0000) 2024-03-04T16:53:19.832 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:53:19.854+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:53:19.854421+0000) 2024-03-04T16:53:20.832 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:53:20.853+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:53:20.854677+0000) 2024-03-04T16:53:21.832 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:53:21.853+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:53:21.854948+0000) 2024-03-04T16:53:22.833 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:53:22.854+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:53:22.855188+0000) 2024-03-04T16:53:23.833 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:53:23.854+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:53:23.855438+0000) 2024-03-04T16:53:24.833 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:53:24.854+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:53:24.855682+0000) 2024-03-04T16:53:25.815 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:53:25.824 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:53:25.834 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:53:25.854+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:53:25.855951+0000) 2024-03-04T16:53:26.834 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:53:26.855+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:53:26.856190+0000) 2024-03-04T16:53:27.834 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:53:27.855+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:53:27.856440+0000) 2024-03-04T16:53:28.834 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:53:28.855+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:53:28.856683+0000) 2024-03-04T16:53:29.834 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:53:29.855+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:53:29.856924+0000) 2024-03-04T16:53:30.834 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:53:30.856+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:53:30.857191+0000) 2024-03-04T16:53:31.835 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:53:31.856+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:53:31.857424+0000) 2024-03-04T16:53:32.835 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:53:32.856+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:53:32.857652+0000) 2024-03-04T16:53:33.835 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:53:33.857+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:53:33.857947+0000) 2024-03-04T16:53:34.836 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:53:34.857+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:53:34.858204+0000) 2024-03-04T16:53:35.836 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:53:35.857+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:53:35.858442+0000) 2024-03-04T16:53:36.836 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:53:36.857+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:53:36.858704+0000) 2024-03-04T16:53:37.836 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:53:37.858+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:53:37.858980+0000) 2024-03-04T16:53:38.837 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:53:38.858+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:53:38.859251+0000) 2024-03-04T16:53:39.837 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:53:39.858+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:53:39.859453+0000) 2024-03-04T16:53:40.837 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:53:40.858+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:53:40.859634+0000) 2024-03-04T16:53:41.837 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:53:41.859+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:53:41.859951+0000) 2024-03-04T16:53:42.837 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:53:42.859+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:53:42.860155+0000) 2024-03-04T16:53:43.838 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:53:43.859+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:53:43.860445+0000) 2024-03-04T16:53:44.838 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:53:44.859+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:53:44.860682+0000) 2024-03-04T16:53:45.838 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:53:45.860+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:53:45.860945+0000) 2024-03-04T16:53:46.838 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:53:46.860+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:53:46.861151+0000) 2024-03-04T16:53:47.839 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:53:47.860+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:53:47.861418+0000) 2024-03-04T16:53:48.839 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:53:48.860+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:53:48.861667+0000) 2024-03-04T16:53:49.839 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:53:49.861+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:53:49.861961+0000) 2024-03-04T16:53:50.839 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:53:50.861+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:53:50.862232+0000) 2024-03-04T16:53:51.840 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:53:51.861+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:53:51.862517+0000) 2024-03-04T16:53:52.840 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:53:52.862+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:53:52.862798+0000) 2024-03-04T16:53:53.840 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:53:53.862+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:53:53.863095+0000) 2024-03-04T16:53:54.840 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:53:54.862+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:53:54.863344+0000) 2024-03-04T16:53:55.841 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:53:55.863+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:53:55.863600+0000) 2024-03-04T16:53:55.953 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:53:55.959 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:53:56.841 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:53:56.863+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:53:56.863886+0000) 2024-03-04T16:53:57.841 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:53:57.863+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:53:57.864140+0000) 2024-03-04T16:53:58.842 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:53:58.863+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:53:58.864374+0000) 2024-03-04T16:53:59.842 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:53:59.864+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:53:59.864623+0000) 2024-03-04T16:54:00.842 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:54:00.864+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:54:00.864887+0000) 2024-03-04T16:54:01.842 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:54:01.864+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:54:01.865136+0000) 2024-03-04T16:54:02.842 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:54:02.864+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:54:02.865368+0000) 2024-03-04T16:54:03.843 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:54:03.865+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:54:03.865618+0000) 2024-03-04T16:54:04.843 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:54:04.865+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:54:04.865894+0000) 2024-03-04T16:54:05.843 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:54:05.865+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:54:05.866101+0000) 2024-03-04T16:54:06.844 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:54:06.865+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:54:06.866386+0000) 2024-03-04T16:54:07.844 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:54:07.866+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:54:07.866627+0000) 2024-03-04T16:54:08.844 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:54:08.866+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:54:08.866914+0000) 2024-03-04T16:54:09.844 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:54:09.866+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:54:09.867177+0000) 2024-03-04T16:54:10.844 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:54:10.867+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:54:10.867427+0000) 2024-03-04T16:54:11.845 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:54:11.867+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:54:11.867693+0000) 2024-03-04T16:54:12.845 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:54:12.867+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:54:12.867947+0000) 2024-03-04T16:54:13.845 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:54:13.867+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:54:13.868236+0000) 2024-03-04T16:54:14.845 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:54:14.868+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:54:14.868494+0000) 2024-03-04T16:54:15.846 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:54:15.868+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:54:15.868745+0000) 2024-03-04T16:54:16.846 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:54:16.868+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:54:16.869002+0000) 2024-03-04T16:54:17.846 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:54:17.869+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:54:17.869254+0000) 2024-03-04T16:54:18.846 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:54:18.869+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:54:18.869501+0000) 2024-03-04T16:54:19.847 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:54:19.869+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:54:19.869810+0000) 2024-03-04T16:54:20.847 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:54:20.869+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:54:20.870176+0000) 2024-03-04T16:54:21.848 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:54:21.869+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:54:21.870422+0000) 2024-03-04T16:54:22.848 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:54:22.869+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:54:22.870724+0000) 2024-03-04T16:54:23.848 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:54:23.869+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:54:23.870983+0000) 2024-03-04T16:54:24.848 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:54:24.870+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:54:24.871269+0000) 2024-03-04T16:54:25.848 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:54:25.870+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:54:25.871565+0000) 2024-03-04T16:54:26.100 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:54:26.108 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:54:26.849 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:54:26.870+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:54:26.871813+0000) 2024-03-04T16:54:27.849 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:54:27.871+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:54:27.872139+0000) 2024-03-04T16:54:28.849 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:54:28.870+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:54:28.872390+0000) 2024-03-04T16:54:29.850 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:54:29.871+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:54:29.872677+0000) 2024-03-04T16:54:30.850 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:54:30.871+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:54:30.872930+0000) 2024-03-04T16:54:31.850 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:54:31.872+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:54:31.873212+0000) 2024-03-04T16:54:32.850 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:54:32.872+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:54:32.873518+0000) 2024-03-04T16:54:33.851 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:54:33.872+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:54:33.873815+0000) 2024-03-04T16:54:34.851 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:54:34.873+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:54:34.874149+0000) 2024-03-04T16:54:35.851 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:54:35.873+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:54:35.874448+0000) 2024-03-04T16:54:36.852 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:54:36.873+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:54:36.874675+0000) 2024-03-04T16:54:37.852 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:54:37.874+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:54:37.874971+0000) 2024-03-04T16:54:38.852 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:54:38.874+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:54:38.875262+0000) 2024-03-04T16:54:39.852 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:54:39.874+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:54:39.875564+0000) 2024-03-04T16:54:40.853 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:54:40.875+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:54:40.875892+0000) 2024-03-04T16:54:41.854 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:54:41.875+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:54:41.876191+0000) 2024-03-04T16:54:42.854 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:54:42.875+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:54:42.876439+0000) 2024-03-04T16:54:43.854 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:54:43.875+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:54:43.876698+0000) 2024-03-04T16:54:44.854 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:54:44.876+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:54:44.876952+0000) 2024-03-04T16:54:45.854 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:54:45.876+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:54:45.877209+0000) 2024-03-04T16:54:46.854 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:54:46.876+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:54:46.877480+0000) 2024-03-04T16:54:47.855 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:54:47.876+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:54:47.877736+0000) 2024-03-04T16:54:48.855 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:54:48.877+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:54:48.877989+0000) 2024-03-04T16:54:49.855 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:54:49.877+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:54:49.878307+0000) 2024-03-04T16:54:50.855 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:54:50.877+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:54:50.878564+0000) 2024-03-04T16:54:51.856 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:54:51.878+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:54:51.878888+0000) 2024-03-04T16:54:52.856 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:54:52.878+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:54:52.879144+0000) 2024-03-04T16:54:53.856 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:54:53.878+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:54:53.879407+0000) 2024-03-04T16:54:54.856 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:54:54.879+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:54:54.879658+0000) 2024-03-04T16:54:55.857 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:54:55.879+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:54:55.879956+0000) 2024-03-04T16:54:56.296 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:54:56.305 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:54:56.857 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:54:56.879+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:54:56.880212+0000) 2024-03-04T16:54:57.857 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:54:57.879+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:54:57.880475+0000) 2024-03-04T16:54:58.858 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:54:58.880+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:54:58.880725+0000) 2024-03-04T16:54:59.858 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:54:59.880+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:54:59.880971+0000) 2024-03-04T16:55:00.858 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:55:00.880+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:55:00.881210+0000) 2024-03-04T16:55:01.858 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:55:01.880+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:55:01.881480+0000) 2024-03-04T16:55:02.859 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:55:02.881+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:55:02.881720+0000) 2024-03-04T16:55:03.859 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:55:03.881+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:55:03.881971+0000) 2024-03-04T16:55:04.859 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:55:04.881+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:55:04.882207+0000) 2024-03-04T16:55:05.859 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:55:05.881+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:55:05.882404+0000) 2024-03-04T16:55:06.859 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:55:06.882+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:55:06.882639+0000) 2024-03-04T16:55:07.860 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:55:07.882+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:55:07.882965+0000) 2024-03-04T16:55:08.861 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:55:08.882+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:55:08.883232+0000) 2024-03-04T16:55:09.861 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:55:09.882+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:55:09.883439+0000) 2024-03-04T16:55:10.861 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:55:10.882+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:55:10.883708+0000) 2024-03-04T16:55:11.861 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:55:11.883+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:55:11.883983+0000) 2024-03-04T16:55:12.861 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:55:12.883+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:55:12.884238+0000) 2024-03-04T16:55:13.861 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:55:13.884+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:55:13.884505+0000) 2024-03-04T16:55:14.861 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:55:14.883+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:55:14.884763+0000) 2024-03-04T16:55:15.862 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:55:15.884+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:55:15.885043+0000) 2024-03-04T16:55:16.862 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:55:16.885+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:55:16.885312+0000) 2024-03-04T16:55:17.862 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:55:17.885+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:55:17.885569+0000) 2024-03-04T16:55:18.863 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:55:18.885+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:55:18.885812+0000) 2024-03-04T16:55:19.863 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:55:19.885+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:55:19.886124+0000) 2024-03-04T16:55:20.863 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:55:20.885+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:55:20.886379+0000) 2024-03-04T16:55:21.863 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:55:21.886+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:55:21.886626+0000) 2024-03-04T16:55:22.865 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:55:22.886+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:55:22.886901+0000) 2024-03-04T16:55:23.864 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:55:23.886+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:55:23.887147+0000) 2024-03-04T16:55:24.865 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:55:24.887+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:55:24.887421+0000) 2024-03-04T16:55:25.864 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:55:25.887+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:55:25.887625+0000) 2024-03-04T16:55:26.526 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:55:26.534 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:55:26.864 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:55:26.886+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:55:26.887927+0000) 2024-03-04T16:55:27.865 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:55:27.886+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:55:27.888136+0000) 2024-03-04T16:55:28.865 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:55:28.887+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:55:28.888387+0000) 2024-03-04T16:55:29.865 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:55:29.887+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:55:29.888647+0000) 2024-03-04T16:55:30.866 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:55:30.887+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:55:30.888936+0000) 2024-03-04T16:55:31.866 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:55:31.888+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:55:31.889204+0000) 2024-03-04T16:55:32.866 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:55:32.888+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:55:32.889452+0000) 2024-03-04T16:55:33.867 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:55:33.888+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:55:33.889725+0000) 2024-03-04T16:55:34.867 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:55:34.888+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:55:34.889961+0000) 2024-03-04T16:55:35.867 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:55:35.889+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:55:35.890225+0000) 2024-03-04T16:55:36.867 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:55:36.889+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:55:36.890480+0000) 2024-03-04T16:55:37.867 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:55:37.889+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:55:37.890745+0000) 2024-03-04T16:55:38.868 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:55:38.890+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:55:38.891003+0000) 2024-03-04T16:55:39.868 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:55:39.890+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:55:39.891259+0000) 2024-03-04T16:55:40.868 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:55:40.890+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:55:40.891506+0000) 2024-03-04T16:55:41.868 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:55:41.890+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:55:41.891779+0000) 2024-03-04T16:55:42.869 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:55:42.891+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:55:42.892136+0000) 2024-03-04T16:55:43.869 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:55:43.891+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:55:43.892394+0000) 2024-03-04T16:55:44.869 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:55:44.891+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:55:44.892679+0000) 2024-03-04T16:55:45.869 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:55:45.892+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:55:45.892962+0000) 2024-03-04T16:55:46.870 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:55:46.892+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:55:46.893232+0000) 2024-03-04T16:55:47.870 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:55:47.892+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:55:47.893481+0000) 2024-03-04T16:55:48.870 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:55:48.892+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:55:48.893724+0000) 2024-03-04T16:55:49.871 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:55:49.893+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:55:49.893961+0000) 2024-03-04T16:55:50.871 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:55:50.893+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:55:50.894201+0000) 2024-03-04T16:55:51.871 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:55:51.893+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:55:51.894457+0000) 2024-03-04T16:55:52.871 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:55:52.893+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:55:52.894729+0000) 2024-03-04T16:55:53.871 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:55:53.894+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:55:53.894994+0000) 2024-03-04T16:55:54.872 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:55:54.894+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:55:54.895213+0000) 2024-03-04T16:55:55.872 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:55:55.894+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:55:55.895468+0000) 2024-03-04T16:55:56.766 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:55:56.774 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:55:56.872 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:55:56.894+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:55:56.895682+0000) 2024-03-04T16:55:57.872 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:55:57.895+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:55:57.895959+0000) 2024-03-04T16:55:58.873 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:55:58.895+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:55:58.896215+0000) 2024-03-04T16:55:59.873 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:55:59.895+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:55:59.896465+0000) 2024-03-04T16:56:00.873 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:56:00.896+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:56:00.896702+0000) 2024-03-04T16:56:01.873 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:56:01.896+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:56:01.896967+0000) 2024-03-04T16:56:02.874 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:56:02.896+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:56:02.897219+0000) 2024-03-04T16:56:03.874 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:56:03.896+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:56:03.897475+0000) 2024-03-04T16:56:04.874 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:56:04.897+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:56:04.897732+0000) 2024-03-04T16:56:05.875 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:56:05.897+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:56:05.897989+0000) 2024-03-04T16:56:06.875 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:56:06.897+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:56:06.898241+0000) 2024-03-04T16:56:07.875 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:56:07.898+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:56:07.898505+0000) 2024-03-04T16:56:08.876 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:56:08.897+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:56:08.898711+0000) 2024-03-04T16:56:09.875 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:56:09.898+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:56:09.898971+0000) 2024-03-04T16:56:10.876 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:56:10.898+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:56:10.899215+0000) 2024-03-04T16:56:11.876 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:56:11.899+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:56:11.899479+0000) 2024-03-04T16:56:12.876 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:56:12.899+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:56:12.899712+0000) 2024-03-04T16:56:13.877 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:56:13.899+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:56:13.899984+0000) 2024-03-04T16:56:14.877 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:56:14.899+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:56:14.900236+0000) 2024-03-04T16:56:15.877 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:56:15.900+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:56:15.900497+0000) 2024-03-04T16:56:16.877 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:56:16.900+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:56:16.900720+0000) 2024-03-04T16:56:17.877 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:56:17.900+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:56:17.900957+0000) 2024-03-04T16:56:18.878 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:56:18.900+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:56:18.901198+0000) 2024-03-04T16:56:19.878 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:56:19.901+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:56:19.901439+0000) 2024-03-04T16:56:20.878 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:56:20.901+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:56:20.901680+0000) 2024-03-04T16:56:21.878 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:56:21.901+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:56:21.901925+0000) 2024-03-04T16:56:22.879 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:56:22.901+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:56:22.902156+0000) 2024-03-04T16:56:23.879 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:56:23.902+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:56:23.902404+0000) 2024-03-04T16:56:24.879 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:56:24.902+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:56:24.902641+0000) 2024-03-04T16:56:25.879 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:56:25.902+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:56:25.902883+0000) 2024-03-04T16:56:26.879 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:56:26.902+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:56:26.903140+0000) 2024-03-04T16:56:26.928 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:56:26.936 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:56:27.880 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:56:27.902+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:56:27.903339+0000) 2024-03-04T16:56:28.880 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:56:28.903+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:56:28.903582+0000) 2024-03-04T16:56:29.880 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:56:29.902+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:56:29.903887+0000) 2024-03-04T16:56:30.880 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:56:30.902+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:56:30.904138+0000) 2024-03-04T16:56:31.881 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:56:31.903+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:56:31.904423+0000) 2024-03-04T16:56:32.882 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:56:32.903+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:56:32.904672+0000) 2024-03-04T16:56:33.881 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:56:33.903+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:56:33.904962+0000) 2024-03-04T16:56:34.882 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:56:34.904+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:56:34.905210+0000) 2024-03-04T16:56:35.883 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:56:35.904+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:56:35.905468+0000) 2024-03-04T16:56:36.882 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:56:36.904+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:56:36.905801+0000) 2024-03-04T16:56:37.884 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:56:37.904+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:56:37.906106+0000) 2024-03-04T16:56:38.883 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:56:38.905+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:56:38.906386+0000) 2024-03-04T16:56:39.883 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:56:39.905+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:56:39.906645+0000) 2024-03-04T16:56:40.883 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:56:40.906+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:56:40.906950+0000) 2024-03-04T16:56:41.883 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:56:41.906+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:56:41.907214+0000) 2024-03-04T16:56:42.884 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:56:42.906+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:56:42.907467+0000) 2024-03-04T16:56:43.884 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:56:43.906+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:56:43.907757+0000) 2024-03-04T16:56:44.884 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:56:44.907+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:56:44.908012+0000) 2024-03-04T16:56:45.886 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:56:45.907+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:56:45.908254+0000) 2024-03-04T16:56:46.885 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:56:46.907+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:56:46.908489+0000) 2024-03-04T16:56:47.885 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:56:47.907+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:56:47.908725+0000) 2024-03-04T16:56:48.885 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:56:48.908+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:56:48.908960+0000) 2024-03-04T16:56:49.885 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:56:49.908+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:56:49.909209+0000) 2024-03-04T16:56:50.886 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:56:50.908+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:56:50.909399+0000) 2024-03-04T16:56:51.886 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:56:51.908+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:56:51.909666+0000) 2024-03-04T16:56:52.886 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:56:52.909+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:56:52.909941+0000) 2024-03-04T16:56:53.887 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:56:53.909+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:56:53.910185+0000) 2024-03-04T16:56:54.887 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:56:54.909+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:56:54.910412+0000) 2024-03-04T16:56:55.887 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:56:55.909+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:56:55.910670+0000) 2024-03-04T16:56:56.887 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:56:56.910+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:56:56.910921+0000) 2024-03-04T16:56:57.125 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:56:57.133 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:56:57.887 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:56:57.910+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:56:57.911176+0000) 2024-03-04T16:56:58.888 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:56:58.910+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:56:58.911398+0000) 2024-03-04T16:56:59.888 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:56:59.911+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:56:59.911660+0000) 2024-03-04T16:57:00.888 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:57:00.911+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:57:00.911934+0000) 2024-03-04T16:57:01.889 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:57:01.911+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:57:01.912192+0000) 2024-03-04T16:57:02.889 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:57:02.911+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:57:02.912446+0000) 2024-03-04T16:57:03.889 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:57:03.911+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:57:03.912699+0000) 2024-03-04T16:57:04.889 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:57:04.912+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:57:04.912930+0000) 2024-03-04T16:57:05.889 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:57:05.912+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:57:05.913198+0000) 2024-03-04T16:57:06.890 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:57:06.912+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:57:06.913435+0000) 2024-03-04T16:57:07.890 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:57:07.913+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:57:07.913682+0000) 2024-03-04T16:57:08.890 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:57:08.913+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:57:08.913921+0000) 2024-03-04T16:57:09.890 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:57:09.913+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:57:09.914181+0000) 2024-03-04T16:57:10.891 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:57:10.913+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:57:10.914413+0000) 2024-03-04T16:57:11.892 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:57:11.914+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:57:11.914668+0000) 2024-03-04T16:57:12.891 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:57:12.914+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:57:12.914941+0000) 2024-03-04T16:57:13.891 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:57:13.914+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:57:13.915181+0000) 2024-03-04T16:57:14.891 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:57:14.914+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:57:14.915423+0000) 2024-03-04T16:57:15.892 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:57:15.914+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:57:15.915679+0000) 2024-03-04T16:57:16.892 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:57:16.915+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:57:16.915965+0000) 2024-03-04T16:57:17.892 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:57:17.915+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:57:17.916255+0000) 2024-03-04T16:57:18.893 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:57:18.916+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:57:18.916512+0000) 2024-03-04T16:57:19.893 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:57:19.915+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:57:19.916779+0000) 2024-03-04T16:57:20.893 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:57:20.916+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:57:20.917070+0000) 2024-03-04T16:57:21.893 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:57:21.917+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:57:21.917352+0000) 2024-03-04T16:57:22.894 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:57:22.916+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:57:22.917591+0000) 2024-03-04T16:57:23.894 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:57:23.917+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:57:23.917937+0000) 2024-03-04T16:57:24.894 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:57:24.917+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:57:24.918140+0000) 2024-03-04T16:57:25.894 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:57:25.918+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:57:25.918397+0000) 2024-03-04T16:57:26.895 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:57:26.918+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:57:26.918650+0000) 2024-03-04T16:57:27.304 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:57:27.313 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:57:27.895 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:57:27.918+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:57:27.918885+0000) 2024-03-04T16:57:28.895 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:57:28.918+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:57:28.919159+0000) 2024-03-04T16:57:29.895 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:57:29.919+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:57:29.919471+0000) 2024-03-04T16:57:30.896 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:57:30.919+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:57:30.919722+0000) 2024-03-04T16:57:31.896 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:57:31.919+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:57:31.919990+0000) 2024-03-04T16:57:32.896 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:57:32.919+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:57:32.920246+0000) 2024-03-04T16:57:33.896 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:57:33.919+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:57:33.920518+0000) 2024-03-04T16:57:34.897 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:57:34.918+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:57:34.920713+0000) 2024-03-04T16:57:35.897 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:57:35.919+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:57:35.920976+0000) 2024-03-04T16:57:36.897 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:57:36.920+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:57:36.921225+0000) 2024-03-04T16:57:37.897 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:57:37.920+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:57:37.921478+0000) 2024-03-04T16:57:38.898 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:57:38.920+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:57:38.921734+0000) 2024-03-04T16:57:39.898 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:57:39.920+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:57:39.921993+0000) 2024-03-04T16:57:40.898 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:57:40.921+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:57:40.922252+0000) 2024-03-04T16:57:41.898 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:57:41.921+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:57:41.922523+0000) 2024-03-04T16:57:42.899 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:57:42.921+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:57:42.922767+0000) 2024-03-04T16:57:43.899 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:57:43.922+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:57:43.923047+0000) 2024-03-04T16:57:44.899 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:57:44.922+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:57:44.923298+0000) 2024-03-04T16:57:45.899 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:57:45.922+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:57:45.923564+0000) 2024-03-04T16:57:46.900 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:57:46.922+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:57:46.923796+0000) 2024-03-04T16:57:47.900 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:57:47.923+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:57:47.924050+0000) 2024-03-04T16:57:48.900 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:57:48.923+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:57:48.924290+0000) 2024-03-04T16:57:49.900 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:57:49.923+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:57:49.924548+0000) 2024-03-04T16:57:50.901 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:57:50.923+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:57:50.924791+0000) 2024-03-04T16:57:51.901 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:57:51.924+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:57:51.925092+0000) 2024-03-04T16:57:52.901 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:57:52.924+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:57:52.925341+0000) 2024-03-04T16:57:53.901 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:57:53.924+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:57:53.925596+0000) 2024-03-04T16:57:54.902 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:57:54.925+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:57:54.925891+0000) 2024-03-04T16:57:55.902 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:57:55.925+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:57:55.926191+0000) 2024-03-04T16:57:56.902 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:57:56.925+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:57:56.926407+0000) 2024-03-04T16:57:57.525 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:57:57.533 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:57:57.902 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:57:57.925+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:57:57.926673+0000) 2024-03-04T16:57:58.903 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:57:58.926+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:57:58.926923+0000) 2024-03-04T16:57:59.903 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:57:59.926+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:57:59.927182+0000) 2024-03-04T16:58:00.903 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:58:00.926+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:58:00.927437+0000) 2024-03-04T16:58:01.904 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:58:01.927+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:58:01.927708+0000) 2024-03-04T16:58:02.904 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:58:02.927+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:58:02.927989+0000) 2024-03-04T16:58:03.904 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:58:03.927+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:58:03.928193+0000) 2024-03-04T16:58:04.904 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:58:04.927+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:58:04.928386+0000) 2024-03-04T16:58:05.907 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:58:05.928+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:58:05.928638+0000) 2024-03-04T16:58:06.905 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:58:06.928+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:58:06.928951+0000) 2024-03-04T16:58:07.905 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:58:07.928+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:58:07.929221+0000) 2024-03-04T16:58:08.905 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:58:08.928+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:58:08.929453+0000) 2024-03-04T16:58:09.905 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:58:09.929+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:58:09.929703+0000) 2024-03-04T16:58:10.906 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:58:10.929+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:58:10.929949+0000) 2024-03-04T16:58:11.906 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:58:11.929+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:58:11.930205+0000) 2024-03-04T16:58:12.906 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:58:12.930+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:58:12.930517+0000) 2024-03-04T16:58:13.907 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:58:13.930+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:58:13.930781+0000) 2024-03-04T16:58:14.907 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:58:14.930+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:58:14.931028+0000) 2024-03-04T16:58:15.907 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:58:15.930+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:58:15.931275+0000) 2024-03-04T16:58:16.907 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:58:16.931+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:58:16.931520+0000) 2024-03-04T16:58:17.908 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:58:17.930+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:58:17.931765+0000) 2024-03-04T16:58:18.909 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:58:18.931+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:58:18.931997+0000) 2024-03-04T16:58:19.908 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:58:19.931+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:58:19.932241+0000) 2024-03-04T16:58:20.908 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:58:20.932+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:58:20.932479+0000) 2024-03-04T16:58:21.909 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:58:21.932+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:58:21.932749+0000) 2024-03-04T16:58:22.909 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:58:22.932+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:58:22.932990+0000) 2024-03-04T16:58:23.909 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:58:23.932+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:58:23.933231+0000) 2024-03-04T16:58:24.909 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:58:24.933+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:58:24.933469+0000) 2024-03-04T16:58:25.909 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:58:25.933+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:58:25.933720+0000) 2024-03-04T16:58:26.910 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:58:26.933+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:58:26.933971+0000) 2024-03-04T16:58:27.747 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:58:27.759 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:58:27.910 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:58:27.933+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:58:27.934190+0000) 2024-03-04T16:58:28.910 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:58:28.934+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:58:28.934446+0000) 2024-03-04T16:58:29.910 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:58:29.934+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:58:29.934706+0000) 2024-03-04T16:58:30.911 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:58:30.934+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:58:30.934954+0000) 2024-03-04T16:58:31.911 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:58:31.934+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:58:31.935219+0000) 2024-03-04T16:58:32.911 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:58:32.935+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:58:32.935457+0000) 2024-03-04T16:58:33.911 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:58:33.935+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:58:33.935707+0000) 2024-03-04T16:58:34.912 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:58:34.935+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:58:34.935942+0000) 2024-03-04T16:58:35.912 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:58:35.936+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:58:35.936193+0000) 2024-03-04T16:58:36.912 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:58:36.935+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:58:36.936428+0000) 2024-03-04T16:58:37.912 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:58:37.935+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:58:37.936671+0000) 2024-03-04T16:58:38.913 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:58:38.935+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:58:38.936947+0000) 2024-03-04T16:58:39.913 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:58:39.935+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:58:39.937139+0000) 2024-03-04T16:58:40.913 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:58:40.936+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:58:40.937375+0000) 2024-03-04T16:58:41.913 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:58:41.936+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:58:41.937623+0000) 2024-03-04T16:58:42.913 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:58:42.936+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:58:42.937931+0000) 2024-03-04T16:58:43.914 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:58:43.936+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:58:43.938166+0000) 2024-03-04T16:58:44.914 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:58:44.936+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:58:44.938419+0000) 2024-03-04T16:58:45.915 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:58:45.937+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:58:45.938664+0000) 2024-03-04T16:58:46.914 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:58:46.937+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:58:46.938950+0000) 2024-03-04T16:58:47.915 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:58:47.938+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:58:47.939190+0000) 2024-03-04T16:58:48.915 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:58:48.938+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:58:48.939430+0000) 2024-03-04T16:58:49.916 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:58:49.938+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:58:49.939697+0000) 2024-03-04T16:58:50.916 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:58:50.939+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:58:50.939983+0000) 2024-03-04T16:58:51.916 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:58:51.939+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:58:51.940247+0000) 2024-03-04T16:58:52.916 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:58:52.939+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:58:52.940483+0000) 2024-03-04T16:58:53.916 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:58:53.939+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:58:53.940670+0000) 2024-03-04T16:58:54.917 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:58:54.940+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:58:54.940939+0000) 2024-03-04T16:58:55.917 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:58:55.940+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:58:55.941184+0000) 2024-03-04T16:58:56.917 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:58:56.940+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:58:56.941419+0000) 2024-03-04T16:58:57.917 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:58:57.940+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:58:57.941656+0000) 2024-03-04T16:58:57.953 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:58:57.961 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:58:58.917 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:58:58.941+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:58:58.941923+0000) 2024-03-04T16:58:59.918 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:58:59.941+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:58:59.942167+0000) 2024-03-04T16:59:00.918 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:59:00.941+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:59:00.942405+0000) 2024-03-04T16:59:01.918 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:59:01.941+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:59:01.942658+0000) 2024-03-04T16:59:02.918 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:59:02.942+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:59:02.942927+0000) 2024-03-04T16:59:03.919 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:59:03.942+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:59:03.943169+0000) 2024-03-04T16:59:04.919 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:59:04.942+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:59:04.943405+0000) 2024-03-04T16:59:05.919 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:59:05.942+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:59:05.943660+0000) 2024-03-04T16:59:06.919 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:59:06.943+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:59:06.943936+0000) 2024-03-04T16:59:07.920 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:59:07.943+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:59:07.944168+0000) 2024-03-04T16:59:08.920 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:59:08.943+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:59:08.944403+0000) 2024-03-04T16:59:09.920 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:59:09.943+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:59:09.944642+0000) 2024-03-04T16:59:10.920 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:59:10.944+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:59:10.944936+0000) 2024-03-04T16:59:11.921 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:59:11.944+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:59:11.945188+0000) 2024-03-04T16:59:12.921 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:59:12.944+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:59:12.945457+0000) 2024-03-04T16:59:13.921 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:59:13.945+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:59:13.945735+0000) 2024-03-04T16:59:14.921 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:59:14.945+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:59:14.945974+0000) 2024-03-04T16:59:15.922 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:59:15.945+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:59:15.946222+0000) 2024-03-04T16:59:16.922 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:59:16.945+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:59:16.946464+0000) 2024-03-04T16:59:17.922 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:59:17.946+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:59:17.946699+0000) 2024-03-04T16:59:18.922 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:59:18.946+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:59:18.946935+0000) 2024-03-04T16:59:19.923 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:59:19.946+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:59:19.947184+0000) 2024-03-04T16:59:20.923 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:59:20.946+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:59:20.947418+0000) 2024-03-04T16:59:21.923 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:59:21.947+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:59:21.947714+0000) 2024-03-04T16:59:22.923 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:59:22.947+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:59:22.947972+0000) 2024-03-04T16:59:23.924 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:59:23.947+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:59:23.948231+0000) 2024-03-04T16:59:24.924 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:59:24.947+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:59:24.948479+0000) 2024-03-04T16:59:25.924 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:59:25.947+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:59:25.948732+0000) 2024-03-04T16:59:26.924 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:59:26.948+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:59:26.948977+0000) 2024-03-04T16:59:27.925 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:59:27.948+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:59:27.949210+0000) 2024-03-04T16:59:28.107 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:59:28.115 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:59:28.925 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:59:28.949+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:59:28.949449+0000) 2024-03-04T16:59:29.925 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:59:29.949+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:59:29.949690+0000) 2024-03-04T16:59:30.925 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:59:30.949+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:59:30.949947+0000) 2024-03-04T16:59:31.925 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:59:31.949+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:59:31.950215+0000) 2024-03-04T16:59:32.926 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:59:32.949+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:59:32.950470+0000) 2024-03-04T16:59:33.926 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:59:33.950+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:59:33.950715+0000) 2024-03-04T16:59:34.926 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:59:34.950+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:59:34.950948+0000) 2024-03-04T16:59:35.927 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:59:35.950+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:59:35.951195+0000) 2024-03-04T16:59:36.927 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:59:36.951+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:59:36.951430+0000) 2024-03-04T16:59:37.927 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:59:37.951+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:59:37.951686+0000) 2024-03-04T16:59:38.927 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:59:38.951+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:59:38.951918+0000) 2024-03-04T16:59:39.927 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:59:39.952+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:59:39.952190+0000) 2024-03-04T16:59:40.928 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:59:40.951+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:59:40.952427+0000) 2024-03-04T16:59:41.928 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:59:41.951+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:59:41.952670+0000) 2024-03-04T16:59:42.928 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:59:42.951+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:59:42.952935+0000) 2024-03-04T16:59:43.928 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:59:43.951+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:59:43.953128+0000) 2024-03-04T16:59:44.929 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:59:44.952+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:59:44.953389+0000) 2024-03-04T16:59:45.929 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:59:45.952+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:59:45.953636+0000) 2024-03-04T16:59:46.929 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:59:46.952+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:59:46.953928+0000) 2024-03-04T16:59:47.930 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:59:47.952+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:59:47.954178+0000) 2024-03-04T16:59:48.930 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:59:48.953+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:59:48.954417+0000) 2024-03-04T16:59:49.930 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:59:49.953+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:59:49.954679+0000) 2024-03-04T16:59:50.930 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:59:50.953+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:59:50.954941+0000) 2024-03-04T16:59:51.930 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:59:51.954+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:59:51.955198+0000) 2024-03-04T16:59:52.931 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:59:52.954+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:59:52.955435+0000) 2024-03-04T16:59:53.931 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:59:53.954+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:59:53.955681+0000) 2024-03-04T16:59:54.931 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:59:54.955+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:59:54.955954+0000) 2024-03-04T16:59:55.931 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:59:55.955+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:59:55.956230+0000) 2024-03-04T16:59:56.932 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:59:56.955+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:59:56.956513+0000) 2024-03-04T16:59:57.932 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:59:57.956+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:59:57.956816+0000) 2024-03-04T16:59:58.346 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:59:58.354 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T16:59:58.932 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:59:58.956+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:59:58.957132+0000) 2024-03-04T16:59:59.933 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T16:59:59.956+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T15:59:59.957419+0000) 2024-03-04T17:00:00.933 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:00:00.956+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:00:00.957712+0000) 2024-03-04T17:00:01.933 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:00:01.957+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:00:01.958020+0000) 2024-03-04T17:00:02.933 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:00:02.957+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:00:02.958309+0000) 2024-03-04T17:00:03.934 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:00:03.957+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:00:03.958566+0000) 2024-03-04T17:00:04.934 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:00:04.958+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:00:04.958807+0000) 2024-03-04T17:00:05.934 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:00:05.958+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:00:05.959097+0000) 2024-03-04T17:00:06.935 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:00:06.958+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:00:06.959331+0000) 2024-03-04T17:00:07.935 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:00:07.958+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:00:07.959572+0000) 2024-03-04T17:00:08.935 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:00:08.959+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:00:08.959887+0000) 2024-03-04T17:00:09.935 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:00:09.959+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:00:09.960155+0000) 2024-03-04T17:00:10.936 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:00:10.959+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:00:10.960436+0000) 2024-03-04T17:00:11.936 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:00:11.960+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:00:11.960726+0000) 2024-03-04T17:00:12.936 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:00:12.960+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:00:12.961026+0000) 2024-03-04T17:00:13.936 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:00:13.960+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:00:13.961236+0000) 2024-03-04T17:00:14.937 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:00:14.960+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:00:14.961512+0000) 2024-03-04T17:00:15.937 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:00:15.961+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:00:15.961802+0000) 2024-03-04T17:00:16.937 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:00:16.961+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:00:16.962135+0000) 2024-03-04T17:00:17.938 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:00:17.961+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:00:17.962424+0000) 2024-03-04T17:00:18.938 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:00:18.962+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:00:18.962714+0000) 2024-03-04T17:00:19.938 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:00:19.962+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:00:19.962984+0000) 2024-03-04T17:00:20.938 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:00:20.962+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:00:20.963275+0000) 2024-03-04T17:00:21.939 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:00:21.963+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:00:21.963573+0000) 2024-03-04T17:00:22.939 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:00:22.963+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:00:22.963909+0000) 2024-03-04T17:00:23.939 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:00:23.963+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:00:23.964203+0000) 2024-03-04T17:00:24.940 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:00:24.964+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:00:24.964496+0000) 2024-03-04T17:00:25.940 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:00:25.964+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:00:25.964796+0000) 2024-03-04T17:00:26.940 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:00:26.964+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:00:26.965140+0000) 2024-03-04T17:00:27.940 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:00:27.965+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:00:27.965425+0000) 2024-03-04T17:00:28.559 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:00:28.567 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:00:28.941 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:00:28.965+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:00:28.965703+0000) 2024-03-04T17:00:29.941 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:00:29.965+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:00:29.966004+0000) 2024-03-04T17:00:30.941 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:00:30.965+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:00:30.966290+0000) 2024-03-04T17:00:31.942 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:00:31.966+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:00:31.966601+0000) 2024-03-04T17:00:32.942 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:00:32.966+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:00:32.966924+0000) 2024-03-04T17:00:33.942 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:00:33.966+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:00:33.967207+0000) 2024-03-04T17:00:34.942 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:00:34.966+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:00:34.967446+0000) 2024-03-04T17:00:35.943 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:00:35.967+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:00:35.967687+0000) 2024-03-04T17:00:36.943 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:00:36.967+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:00:36.967960+0000) 2024-03-04T17:00:37.943 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:00:37.968+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:00:37.968289+0000) 2024-03-04T17:00:38.944 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:00:38.968+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:00:38.968599+0000) 2024-03-04T17:00:39.945 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:00:39.968+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:00:39.968926+0000) 2024-03-04T17:00:40.944 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:00:40.969+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:00:40.969219+0000) 2024-03-04T17:00:41.945 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:00:41.969+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:00:41.969535+0000) 2024-03-04T17:00:42.945 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:00:42.969+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:00:42.969873+0000) 2024-03-04T17:00:43.945 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:00:43.970+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:00:43.970189+0000) 2024-03-04T17:00:44.945 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:00:44.968+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:00:44.970432+0000) 2024-03-04T17:00:45.946 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:00:45.968+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:00:45.970613+0000) 2024-03-04T17:00:46.946 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:00:46.969+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:00:46.970947+0000) 2024-03-04T17:00:47.946 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:00:47.969+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:00:47.971135+0000) 2024-03-04T17:00:48.946 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:00:48.970+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:00:48.971374+0000) 2024-03-04T17:00:49.947 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:00:49.969+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:00:49.971629+0000) 2024-03-04T17:00:50.947 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:00:50.969+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:00:50.971807+0000) 2024-03-04T17:00:51.947 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:00:51.971+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:00:51.972105+0000) 2024-03-04T17:00:52.947 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:00:52.971+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:00:52.972342+0000) 2024-03-04T17:00:53.948 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:00:53.970+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:00:53.972601+0000) 2024-03-04T17:00:54.948 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:00:54.971+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:00:54.972862+0000) 2024-03-04T17:00:55.948 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:00:55.972+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:00:55.973111+0000) 2024-03-04T17:00:56.948 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:00:56.972+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:00:56.973352+0000) 2024-03-04T17:00:57.949 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:00:57.972+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:00:57.973599+0000) 2024-03-04T17:00:58.713 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:00:58.721 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:00:58.949 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:00:58.973+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:00:58.973878+0000) 2024-03-04T17:00:59.949 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:00:59.973+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:00:59.974137+0000) 2024-03-04T17:01:00.950 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:01:00.973+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:01:00.974393+0000) 2024-03-04T17:01:01.950 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:01:01.973+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:01:01.974645+0000) 2024-03-04T17:01:02.950 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:01:02.974+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:01:02.974926+0000) 2024-03-04T17:01:03.950 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:01:03.974+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:01:03.975169+0000) 2024-03-04T17:01:04.950 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:01:04.974+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:01:04.975406+0000) 2024-03-04T17:01:05.950 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:01:05.974+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:01:05.975651+0000) 2024-03-04T17:01:06.951 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:01:06.975+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:01:06.975930+0000) 2024-03-04T17:01:07.951 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:01:07.975+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:01:07.976191+0000) 2024-03-04T17:01:08.951 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:01:08.975+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:01:08.976427+0000) 2024-03-04T17:01:09.952 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:01:09.975+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:01:09.976670+0000) 2024-03-04T17:01:10.952 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:01:10.976+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:01:10.976943+0000) 2024-03-04T17:01:11.952 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:01:11.976+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:01:11.977192+0000) 2024-03-04T17:01:12.952 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:01:12.976+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:01:12.977426+0000) 2024-03-04T17:01:13.952 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:01:13.977+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:01:13.977668+0000) 2024-03-04T17:01:14.953 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:01:14.977+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:01:14.977948+0000) 2024-03-04T17:01:15.953 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:01:15.977+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:01:15.978193+0000) 2024-03-04T17:01:16.953 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:01:16.977+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:01:16.978428+0000) 2024-03-04T17:01:17.954 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:01:17.977+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:01:17.978683+0000) 2024-03-04T17:01:18.954 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:01:18.978+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:01:18.978927+0000) 2024-03-04T17:01:19.954 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:01:19.978+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:01:19.979164+0000) 2024-03-04T17:01:20.954 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:01:20.978+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:01:20.979358+0000) 2024-03-04T17:01:21.954 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:01:21.978+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:01:21.979562+0000) 2024-03-04T17:01:22.955 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:01:22.979+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:01:22.979789+0000) 2024-03-04T17:01:23.955 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:01:23.979+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:01:23.980084+0000) 2024-03-04T17:01:24.955 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:01:24.979+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:01:24.980320+0000) 2024-03-04T17:01:25.955 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:01:25.980+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:01:25.980564+0000) 2024-03-04T17:01:26.956 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:01:26.980+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:01:26.980813+0000) 2024-03-04T17:01:27.956 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:01:27.980+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:01:27.981112+0000) 2024-03-04T17:01:28.851 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:01:28.860 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:01:28.956 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:01:28.980+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:01:28.981295+0000) 2024-03-04T17:01:29.956 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:01:29.981+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:01:29.981563+0000) 2024-03-04T17:01:30.957 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:01:30.981+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:01:30.981903+0000) 2024-03-04T17:01:31.957 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:01:31.981+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:01:31.982222+0000) 2024-03-04T17:01:32.957 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:01:32.982+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:01:32.982507+0000) 2024-03-04T17:01:33.958 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:01:33.982+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:01:33.982798+0000) 2024-03-04T17:01:34.958 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:01:34.982+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:01:34.983136+0000) 2024-03-04T17:01:35.958 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:01:35.983+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:01:35.983452+0000) 2024-03-04T17:01:36.958 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:01:36.983+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:01:36.983757+0000) 2024-03-04T17:01:37.959 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:01:37.983+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:01:37.984044+0000) 2024-03-04T17:01:38.959 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:01:38.984+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:01:38.984328+0000) 2024-03-04T17:01:39.960 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:01:39.984+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:01:39.984620+0000) 2024-03-04T17:01:40.960 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:01:40.984+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:01:40.984959+0000) 2024-03-04T17:01:41.960 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:01:41.984+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:01:41.985265+0000) 2024-03-04T17:01:42.960 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:01:42.984+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:01:42.985529+0000) 2024-03-04T17:01:43.961 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:01:43.985+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:01:43.985776+0000) 2024-03-04T17:01:44.961 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:01:44.985+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:01:44.986014+0000) 2024-03-04T17:01:45.961 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:01:45.986+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:01:45.986258+0000) 2024-03-04T17:01:46.962 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:01:46.986+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:01:46.986510+0000) 2024-03-04T17:01:47.961 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:01:47.986+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:01:47.986750+0000) 2024-03-04T17:01:48.962 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:01:48.985+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:01:48.986989+0000) 2024-03-04T17:01:49.962 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:01:49.986+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:01:49.987232+0000) 2024-03-04T17:01:50.962 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:01:50.986+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:01:50.987477+0000) 2024-03-04T17:01:51.963 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:01:51.986+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:01:51.987727+0000) 2024-03-04T17:01:52.963 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:01:52.986+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:01:52.987968+0000) 2024-03-04T17:01:53.963 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:01:53.987+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:01:53.988208+0000) 2024-03-04T17:01:54.964 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:01:54.987+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:01:54.988446+0000) 2024-03-04T17:01:55.963 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:01:55.987+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:01:55.988691+0000) 2024-03-04T17:01:56.963 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:01:56.987+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:01:56.988934+0000) 2024-03-04T17:01:57.964 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:01:57.988+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:01:57.989196+0000) 2024-03-04T17:01:58.964 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:01:58.988+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:01:58.989431+0000) 2024-03-04T17:01:58.994 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:01:59.003 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:01:59.964 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:01:59.988+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:01:59.989673+0000) 2024-03-04T17:02:00.964 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:02:00.989+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:02:00.989938+0000) 2024-03-04T17:02:01.965 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:02:01.989+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:02:01.990212+0000) 2024-03-04T17:02:02.965 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:02:02.989+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:02:02.990495+0000) 2024-03-04T17:02:03.965 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:02:03.989+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:02:03.990777+0000) 2024-03-04T17:02:04.966 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:02:04.990+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:02:04.991091+0000) 2024-03-04T17:02:05.966 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:02:05.990+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:02:05.991394+0000) 2024-03-04T17:02:06.966 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:02:06.990+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:02:06.991614+0000) 2024-03-04T17:02:07.966 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:02:07.991+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:02:07.991924+0000) 2024-03-04T17:02:08.967 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:02:08.991+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:02:08.992124+0000) 2024-03-04T17:02:09.967 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:02:09.991+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:02:09.992366+0000) 2024-03-04T17:02:10.967 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:02:10.991+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:02:10.992600+0000) 2024-03-04T17:02:11.967 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:02:11.992+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:02:11.992895+0000) 2024-03-04T17:02:12.968 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:02:12.992+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:02:12.993140+0000) 2024-03-04T17:02:13.968 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:02:13.992+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:02:13.993380+0000) 2024-03-04T17:02:14.968 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:02:14.992+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:02:14.993613+0000) 2024-03-04T17:02:15.970 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:02:15.993+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:02:15.993902+0000) 2024-03-04T17:02:16.969 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:02:16.993+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:02:16.994135+0000) 2024-03-04T17:02:17.969 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:02:17.993+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:02:17.994390+0000) 2024-03-04T17:02:18.969 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:02:18.993+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:02:18.994651+0000) 2024-03-04T17:02:19.970 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:02:19.994+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:02:19.994942+0000) 2024-03-04T17:02:20.970 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:02:20.994+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:02:20.995176+0000) 2024-03-04T17:02:21.970 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:02:21.994+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:02:21.995432+0000) 2024-03-04T17:02:22.970 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:02:22.995+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:02:22.995717+0000) 2024-03-04T17:02:23.971 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:02:23.995+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:02:23.995963+0000) 2024-03-04T17:02:24.971 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:02:24.995+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:02:24.996198+0000) 2024-03-04T17:02:25.971 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:02:25.995+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:02:25.996440+0000) 2024-03-04T17:02:26.971 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:02:26.996+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:02:26.996677+0000) 2024-03-04T17:02:27.971 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:02:27.996+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:02:27.996949+0000) 2024-03-04T17:02:28.972 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:02:28.996+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:02:28.997186+0000) 2024-03-04T17:02:29.136 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:02:29.144 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:02:29.972 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:02:29.996+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:02:29.997432+0000) 2024-03-04T17:02:30.972 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:02:30.996+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:02:30.997611+0000) 2024-03-04T17:02:31.972 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:02:31.997+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:02:31.997916+0000) 2024-03-04T17:02:32.973 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:02:32.997+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:02:32.998155+0000) 2024-03-04T17:02:33.973 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:02:33.997+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:02:33.998349+0000) 2024-03-04T17:02:34.973 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:02:34.997+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:02:34.998537+0000) 2024-03-04T17:02:35.973 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:02:35.997+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:02:35.998728+0000) 2024-03-04T17:02:36.973 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:02:36.998+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:02:36.998941+0000) 2024-03-04T17:02:37.974 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:02:37.998+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:02:37.999154+0000) 2024-03-04T17:02:38.974 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:02:38.998+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:02:38.999510+0000) 2024-03-04T17:02:39.974 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:02:39.999+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:02:39.999766+0000) 2024-03-04T17:02:40.974 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:02:40.999+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:02:41.000011+0000) 2024-03-04T17:02:41.975 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:02:41.999+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:02:42.000264+0000) 2024-03-04T17:02:42.975 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:02:43.000+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:02:43.000504+0000) 2024-03-04T17:02:43.975 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:02:44.000+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:02:44.000772+0000) 2024-03-04T17:02:44.975 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:02:45.000+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:02:45.001014+0000) 2024-03-04T17:02:45.976 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:02:46.001+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:02:46.001265+0000) 2024-03-04T17:02:46.976 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:02:47.001+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:02:47.001487+0000) 2024-03-04T17:02:47.976 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:02:48.001+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:02:48.001730+0000) 2024-03-04T17:02:48.976 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:02:49.001+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:02:49.002004+0000) 2024-03-04T17:02:49.977 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:02:50.002+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:02:50.002238+0000) 2024-03-04T17:02:50.977 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:02:51.002+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:02:51.002482+0000) 2024-03-04T17:02:51.977 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:02:52.002+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:02:52.002732+0000) 2024-03-04T17:02:52.977 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:02:53.001+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:02:53.002971+0000) 2024-03-04T17:02:53.978 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:02:54.002+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:02:54.003214+0000) 2024-03-04T17:02:54.978 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:02:55.002+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:02:55.003460+0000) 2024-03-04T17:02:55.978 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:02:56.002+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:02:56.003692+0000) 2024-03-04T17:02:56.978 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:02:57.002+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:02:57.003946+0000) 2024-03-04T17:02:57.979 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:02:58.003+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:02:58.004205+0000) 2024-03-04T17:02:58.979 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:02:59.003+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:02:59.004447+0000) 2024-03-04T17:02:59.285 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:02:59.293 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:02:59.979 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:03:00.003+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:03:00.004691+0000) 2024-03-04T17:03:00.980 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:03:01.003+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:03:01.004961+0000) 2024-03-04T17:03:01.979 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:03:02.004+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:03:02.005208+0000) 2024-03-04T17:03:02.980 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:03:03.004+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:03:03.005395+0000) 2024-03-04T17:03:03.980 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:03:04.004+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:03:04.005642+0000) 2024-03-04T17:03:04.980 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:03:05.005+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:03:05.005923+0000) 2024-03-04T17:03:05.980 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:03:06.005+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:03:06.006189+0000) 2024-03-04T17:03:06.981 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:03:07.005+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:03:07.006448+0000) 2024-03-04T17:03:07.981 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:03:08.005+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:03:08.006692+0000) 2024-03-04T17:03:08.981 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:03:09.006+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:03:09.006928+0000) 2024-03-04T17:03:09.981 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:03:10.006+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:03:10.007163+0000) 2024-03-04T17:03:10.982 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:03:11.006+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:03:11.007488+0000) 2024-03-04T17:03:11.982 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:03:12.006+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:03:12.007760+0000) 2024-03-04T17:03:12.982 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:03:13.007+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:03:13.008000+0000) 2024-03-04T17:03:13.982 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:03:14.007+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:03:14.008184+0000) 2024-03-04T17:03:14.983 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:03:15.007+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:03:15.008416+0000) 2024-03-04T17:03:15.983 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:03:16.007+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:03:16.008663+0000) 2024-03-04T17:03:16.983 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:03:17.008+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:03:17.008937+0000) 2024-03-04T17:03:17.983 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:03:18.008+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:03:18.009181+0000) 2024-03-04T17:03:18.984 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:03:19.008+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:03:19.009362+0000) 2024-03-04T17:03:19.984 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:03:20.008+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:03:20.009603+0000) 2024-03-04T17:03:20.985 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:03:21.009+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:03:21.009867+0000) 2024-03-04T17:03:21.985 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:03:22.009+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:03:22.010084+0000) 2024-03-04T17:03:22.985 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:03:23.009+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:03:23.010362+0000) 2024-03-04T17:03:23.985 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:03:24.009+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:03:24.010603+0000) 2024-03-04T17:03:24.985 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:03:25.010+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:03:25.010874+0000) 2024-03-04T17:03:25.986 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:03:26.010+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:03:26.011139+0000) 2024-03-04T17:03:26.986 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:03:27.010+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:03:27.011381+0000) 2024-03-04T17:03:27.986 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:03:28.011+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:03:28.011624+0000) 2024-03-04T17:03:28.986 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:03:29.011+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:03:29.011912+0000) 2024-03-04T17:03:29.412 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:03:29.420 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:03:29.987 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:03:30.011+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:03:30.012101+0000) 2024-03-04T17:03:30.987 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:03:31.011+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:03:31.012347+0000) 2024-03-04T17:03:31.988 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:03:32.011+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:03:32.012602+0000) 2024-03-04T17:03:32.988 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:03:33.012+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:03:33.012910+0000) 2024-03-04T17:03:33.988 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:03:34.012+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:03:34.013157+0000) 2024-03-04T17:03:34.988 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:03:35.012+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:03:35.013395+0000) 2024-03-04T17:03:35.989 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:03:36.013+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:03:36.013665+0000) 2024-03-04T17:03:36.989 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:03:37.013+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:03:37.013932+0000) 2024-03-04T17:03:37.989 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:03:38.013+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:03:38.014179+0000) 2024-03-04T17:03:38.990 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:03:39.014+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:03:39.014427+0000) 2024-03-04T17:03:39.990 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:03:40.014+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:03:40.014671+0000) 2024-03-04T17:03:40.990 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:03:41.014+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:03:41.014958+0000) 2024-03-04T17:03:41.990 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:03:42.014+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:03:42.015209+0000) 2024-03-04T17:03:42.991 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:03:43.015+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:03:43.015451+0000) 2024-03-04T17:03:43.991 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:03:44.015+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:03:44.015695+0000) 2024-03-04T17:03:44.991 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:03:45.015+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:03:45.015934+0000) 2024-03-04T17:03:45.992 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:03:46.015+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:03:46.016176+0000) 2024-03-04T17:03:46.992 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:03:47.016+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:03:47.016416+0000) 2024-03-04T17:03:47.992 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:03:48.016+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:03:48.016660+0000) 2024-03-04T17:03:48.992 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:03:49.016+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:03:49.016949+0000) 2024-03-04T17:03:49.993 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:03:50.016+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:03:50.017185+0000) 2024-03-04T17:03:50.993 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:03:51.017+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:03:51.017425+0000) 2024-03-04T17:03:51.993 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:03:52.017+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:03:52.017684+0000) 2024-03-04T17:03:52.994 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:03:53.017+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:03:53.017957+0000) 2024-03-04T17:03:53.994 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:03:54.018+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:03:54.018193+0000) 2024-03-04T17:03:54.994 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:03:55.018+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:03:55.018429+0000) 2024-03-04T17:03:55.995 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:03:56.018+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:03:56.018688+0000) 2024-03-04T17:03:56.995 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:03:57.017+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:03:57.018928+0000) 2024-03-04T17:03:57.995 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:03:58.018+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:03:58.019176+0000) 2024-03-04T17:03:58.995 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:03:59.018+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:03:59.019364+0000) 2024-03-04T17:03:59.561 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:03:59.569 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:03:59.996 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:04:00.017+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:04:00.019628+0000) 2024-03-04T17:04:00.996 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:04:01.018+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:04:01.019918+0000) 2024-03-04T17:04:01.996 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:04:02.019+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:04:02.020170+0000) 2024-03-04T17:04:02.997 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:04:03.019+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:04:03.020405+0000) 2024-03-04T17:04:03.997 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:04:04.019+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:04:04.020642+0000) 2024-03-04T17:04:04.997 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:04:05.019+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:04:05.020934+0000) 2024-03-04T17:04:05.997 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:04:06.020+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:04:06.021201+0000) 2024-03-04T17:04:06.998 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:04:07.020+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:04:07.021455+0000) 2024-03-04T17:04:07.998 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:04:08.020+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:04:08.021689+0000) 2024-03-04T17:04:08.998 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:04:09.021+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:04:09.021937+0000) 2024-03-04T17:04:09.999 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:04:10.021+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:04:10.022234+0000) 2024-03-04T17:04:10.999 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:04:11.021+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:04:11.022490+0000) 2024-03-04T17:04:11.999 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:04:12.021+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:04:12.022748+0000) 2024-03-04T17:04:13.000 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:04:13.022+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:04:13.022949+0000) 2024-03-04T17:04:14.000 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:04:14.022+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:04:14.023166+0000) 2024-03-04T17:04:15.000 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:04:15.022+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:04:15.023423+0000) 2024-03-04T17:04:16.000 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:04:16.022+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:04:16.023671+0000) 2024-03-04T17:04:17.001 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:04:17.023+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:04:17.023954+0000) 2024-03-04T17:04:18.001 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:04:18.023+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:04:18.024194+0000) 2024-03-04T17:04:19.001 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:04:19.023+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:04:19.024437+0000) 2024-03-04T17:04:20.002 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:04:20.023+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:04:20.024633+0000) 2024-03-04T17:04:21.002 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:04:21.024+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:04:21.024945+0000) 2024-03-04T17:04:22.002 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:04:22.024+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:04:22.025198+0000) 2024-03-04T17:04:23.002 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:04:23.024+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:04:23.025449+0000) 2024-03-04T17:04:24.003 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:04:24.024+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:04:24.025707+0000) 2024-03-04T17:04:25.003 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:04:25.025+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:04:25.025969+0000) 2024-03-04T17:04:26.003 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:04:26.025+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:04:26.026235+0000) 2024-03-04T17:04:27.004 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:04:27.025+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:04:27.026495+0000) 2024-03-04T17:04:28.004 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:04:28.026+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:04:28.026738+0000) 2024-03-04T17:04:29.004 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:04:29.026+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:04:29.026977+0000) 2024-03-04T17:04:29.784 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:04:29.792 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:04:30.004 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:04:30.026+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:04:30.027161+0000) 2024-03-04T17:04:31.005 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:04:31.026+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:04:31.027410+0000) 2024-03-04T17:04:32.005 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:04:32.027+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:04:32.027691+0000) 2024-03-04T17:04:33.005 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:04:33.027+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:04:33.027947+0000) 2024-03-04T17:04:34.006 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:04:34.027+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:04:34.028199+0000) 2024-03-04T17:04:35.006 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:04:35.027+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:04:35.028455+0000) 2024-03-04T17:04:36.006 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:04:36.028+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:04:36.028720+0000) 2024-03-04T17:04:37.006 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:04:37.028+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:04:37.029012+0000) 2024-03-04T17:04:38.007 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:04:38.028+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:04:38.029270+0000) 2024-03-04T17:04:39.007 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:04:39.029+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:04:39.029518+0000) 2024-03-04T17:04:40.007 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:04:40.029+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:04:40.029744+0000) 2024-03-04T17:04:41.008 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:04:41.029+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:04:41.030025+0000) 2024-03-04T17:04:42.008 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:04:42.029+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:04:42.030288+0000) 2024-03-04T17:04:43.008 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:04:43.030+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:04:43.030539+0000) 2024-03-04T17:04:44.009 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:04:44.030+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:04:44.030795+0000) 2024-03-04T17:04:45.009 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:04:45.030+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:04:45.031095+0000) 2024-03-04T17:04:46.009 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:04:46.030+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:04:46.031356+0000) 2024-03-04T17:04:47.010 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:04:47.030+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:04:47.031634+0000) 2024-03-04T17:04:48.010 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:04:48.031+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:04:48.031938+0000) 2024-03-04T17:04:49.010 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:04:49.031+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:04:49.032196+0000) 2024-03-04T17:04:50.011 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:04:50.031+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:04:50.032470+0000) 2024-03-04T17:04:51.011 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:04:51.032+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:04:51.032730+0000) 2024-03-04T17:04:52.011 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:04:52.032+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:04:52.033001+0000) 2024-03-04T17:04:53.011 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:04:53.032+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:04:53.033247+0000) 2024-03-04T17:04:54.012 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:04:54.033+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:04:54.033506+0000) 2024-03-04T17:04:55.012 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:04:55.033+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:04:55.033754+0000) 2024-03-04T17:04:56.012 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:04:56.033+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:04:56.033948+0000) 2024-03-04T17:04:57.013 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:04:57.034+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:04:57.034248+0000) 2024-03-04T17:04:58.013 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:04:58.034+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:04:58.034554+0000) 2024-03-04T17:04:59.013 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:04:59.034+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:04:59.034855+0000) 2024-03-04T17:04:59.998 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:05:00.006 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:05:00.015 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:05:00.034+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:05:00.035070+0000) 2024-03-04T17:05:01.014 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:05:01.035+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:05:01.035334+0000) 2024-03-04T17:05:02.014 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:05:02.033+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:05:02.035604+0000) 2024-03-04T17:05:03.015 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:05:03.034+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:05:03.035910+0000) 2024-03-04T17:05:04.015 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:05:04.035+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:05:04.036167+0000) 2024-03-04T17:05:05.015 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:05:05.035+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:05:05.036440+0000) 2024-03-04T17:05:06.016 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:05:06.035+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:05:06.036697+0000) 2024-03-04T17:05:07.016 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:05:07.035+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:05:07.036950+0000) 2024-03-04T17:05:08.016 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:05:08.036+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:05:08.037211+0000) 2024-03-04T17:05:09.016 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:05:09.035+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:05:09.037479+0000) 2024-03-04T17:05:10.017 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:05:10.036+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:05:10.037735+0000) 2024-03-04T17:05:11.021 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:05:11.037+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:05:11.037986+0000) 2024-03-04T17:05:12.017 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:05:12.037+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:05:12.038291+0000) 2024-03-04T17:05:13.019 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:05:13.037+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:05:13.038547+0000) 2024-03-04T17:05:14.019 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:05:14.037+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:05:14.038802+0000) 2024-03-04T17:05:15.018 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:05:15.038+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:05:15.039099+0000) 2024-03-04T17:05:16.019 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:05:16.038+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:05:16.039394+0000) 2024-03-04T17:05:17.019 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:05:17.038+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:05:17.039647+0000) 2024-03-04T17:05:18.020 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:05:18.039+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:05:18.039948+0000) 2024-03-04T17:05:19.020 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:05:19.039+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:05:19.040199+0000) 2024-03-04T17:05:20.020 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:05:20.039+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:05:20.040469+0000) 2024-03-04T17:05:21.023 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:05:21.039+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:05:21.040724+0000) 2024-03-04T17:05:22.020 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:05:22.040+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:05:22.040991+0000) 2024-03-04T17:05:23.021 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:05:23.040+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:05:23.041253+0000) 2024-03-04T17:05:24.021 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:05:24.040+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:05:24.041511+0000) 2024-03-04T17:05:25.022 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:05:25.040+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:05:25.041758+0000) 2024-03-04T17:05:26.023 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:05:26.041+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:05:26.042019+0000) 2024-03-04T17:05:27.022 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:05:27.041+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:05:27.042264+0000) 2024-03-04T17:05:28.023 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:05:28.041+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:05:28.042524+0000) 2024-03-04T17:05:29.025 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:05:29.042+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:05:29.042782+0000) 2024-03-04T17:05:30.023 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:05:30.042+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:05:30.043084+0000) 2024-03-04T17:05:30.137 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:05:30.145 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:05:31.023 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:05:31.042+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:05:31.043300+0000) 2024-03-04T17:05:32.023 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:05:32.042+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:05:32.043544+0000) 2024-03-04T17:05:33.024 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:05:33.043+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:05:33.043803+0000) 2024-03-04T17:05:34.024 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:05:34.043+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:05:34.044114+0000) 2024-03-04T17:05:35.024 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:05:35.043+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:05:35.044374+0000) 2024-03-04T17:05:36.025 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:05:36.044+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:05:36.044609+0000) 2024-03-04T17:05:37.025 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:05:37.044+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:05:37.044914+0000) 2024-03-04T17:05:38.025 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:05:38.044+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:05:38.045180+0000) 2024-03-04T17:05:39.026 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:05:39.044+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:05:39.045439+0000) 2024-03-04T17:05:40.026 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:05:40.045+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:05:40.045694+0000) 2024-03-04T17:05:41.026 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:05:41.045+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:05:41.046031+0000) 2024-03-04T17:05:42.027 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:05:42.045+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:05:42.046300+0000) 2024-03-04T17:05:43.027 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:05:43.046+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:05:43.046560+0000) 2024-03-04T17:05:44.027 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:05:44.046+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:05:44.046857+0000) 2024-03-04T17:05:45.027 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:05:45.046+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:05:45.047118+0000) 2024-03-04T17:05:46.028 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:05:46.046+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:05:46.047374+0000) 2024-03-04T17:05:47.029 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:05:47.047+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:05:47.047633+0000) 2024-03-04T17:05:48.028 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:05:48.047+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:05:48.047939+0000) 2024-03-04T17:05:49.032 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:05:49.047+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:05:49.048231+0000) 2024-03-04T17:05:50.029 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:05:50.048+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:05:50.048487+0000) 2024-03-04T17:05:51.029 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:05:51.047+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:05:51.048739+0000) 2024-03-04T17:05:52.030 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:05:52.048+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:05:52.049001+0000) 2024-03-04T17:05:53.030 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:05:53.048+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:05:53.049252+0000) 2024-03-04T17:05:54.030 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:05:54.049+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:05:54.049495+0000) 2024-03-04T17:05:55.030 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:05:55.049+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:05:55.049737+0000) 2024-03-04T17:05:56.031 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:05:56.049+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:05:56.049979+0000) 2024-03-04T17:05:57.031 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:05:57.049+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:05:57.050276+0000) 2024-03-04T17:05:58.031 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:05:58.050+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:05:58.050576+0000) 2024-03-04T17:05:59.032 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:05:59.050+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:05:59.050808+0000) 2024-03-04T17:06:00.032 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:06:00.050+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:06:00.051142+0000) 2024-03-04T17:06:00.285 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:06:00.293 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:06:01.032 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:06:01.051+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:06:01.051443+0000) 2024-03-04T17:06:02.033 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:06:02.051+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:06:02.051763+0000) 2024-03-04T17:06:03.033 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:06:03.051+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:06:03.052063+0000) 2024-03-04T17:06:04.033 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:06:04.051+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:06:04.052353+0000) 2024-03-04T17:06:05.034 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:06:05.052+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:06:05.052649+0000) 2024-03-04T17:06:06.034 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:06:06.051+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:06:06.052993+0000) 2024-03-04T17:06:07.034 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:06:07.052+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:06:07.053286+0000) 2024-03-04T17:06:08.035 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:06:08.052+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:06:08.053557+0000) 2024-03-04T17:06:09.035 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:06:09.052+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:06:09.053815+0000) 2024-03-04T17:06:10.035 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:06:10.053+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:06:10.054112+0000) 2024-03-04T17:06:11.036 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:06:11.053+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:06:11.054369+0000) 2024-03-04T17:06:12.036 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:06:12.053+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:06:12.054638+0000) 2024-03-04T17:06:13.036 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:06:13.053+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:06:13.054949+0000) 2024-03-04T17:06:14.037 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:06:14.054+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:06:14.055216+0000) 2024-03-04T17:06:15.037 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:06:15.054+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:06:15.055458+0000) 2024-03-04T17:06:16.038 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:06:16.054+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:06:16.055698+0000) 2024-03-04T17:06:17.037 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:06:17.055+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:06:17.055941+0000) 2024-03-04T17:06:18.038 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:06:18.055+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:06:18.056192+0000) 2024-03-04T17:06:19.038 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:06:19.055+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:06:19.056434+0000) 2024-03-04T17:06:20.038 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:06:20.055+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:06:20.056667+0000) 2024-03-04T17:06:21.038 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:06:21.056+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:06:21.056957+0000) 2024-03-04T17:06:22.039 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:06:22.056+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:06:22.057210+0000) 2024-03-04T17:06:23.039 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:06:23.056+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:06:23.057452+0000) 2024-03-04T17:06:24.039 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:06:24.056+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:06:24.057700+0000) 2024-03-04T17:06:25.040 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:06:25.057+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:06:25.057941+0000) 2024-03-04T17:06:26.040 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:06:26.057+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:06:26.058194+0000) 2024-03-04T17:06:27.040 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:06:27.057+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:06:27.058386+0000) 2024-03-04T17:06:28.040 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:06:28.057+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:06:28.058630+0000) 2024-03-04T17:06:29.041 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:06:29.058+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:06:29.058938+0000) 2024-03-04T17:06:30.041 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:06:30.058+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:06:30.059203+0000) 2024-03-04T17:06:30.433 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:06:30.442 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:06:31.041 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:06:31.058+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:06:31.059460+0000) 2024-03-04T17:06:32.042 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:06:32.059+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:06:32.059716+0000) 2024-03-04T17:06:33.042 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:06:33.059+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:06:33.059954+0000) 2024-03-04T17:06:34.043 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:06:34.059+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:06:34.060249+0000) 2024-03-04T17:06:35.042 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:06:35.059+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:06:35.060478+0000) 2024-03-04T17:06:36.043 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:06:36.060+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:06:36.060726+0000) 2024-03-04T17:06:37.044 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:06:37.060+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:06:37.060969+0000) 2024-03-04T17:06:38.047 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:06:38.060+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:06:38.061212+0000) 2024-03-04T17:06:39.044 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:06:39.060+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:06:39.061456+0000) 2024-03-04T17:06:40.044 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:06:40.061+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:06:40.061715+0000) 2024-03-04T17:06:41.044 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:06:41.061+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:06:41.061978+0000) 2024-03-04T17:06:42.044 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:06:42.061+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:06:42.062185+0000) 2024-03-04T17:06:43.045 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:06:43.061+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:06:43.062451+0000) 2024-03-04T17:06:44.045 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:06:44.062+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:06:44.062701+0000) 2024-03-04T17:06:45.045 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:06:45.062+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:06:45.062946+0000) 2024-03-04T17:06:46.046 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:06:46.062+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:06:46.063194+0000) 2024-03-04T17:06:47.046 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:06:47.062+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:06:47.063435+0000) 2024-03-04T17:06:48.047 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:06:48.063+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:06:48.063682+0000) 2024-03-04T17:06:49.046 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:06:49.063+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:06:49.063955+0000) 2024-03-04T17:06:50.047 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:06:50.063+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:06:50.064221+0000) 2024-03-04T17:06:51.047 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:06:51.064+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:06:51.064463+0000) 2024-03-04T17:06:52.047 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:06:52.064+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:06:52.064716+0000) 2024-03-04T17:06:53.048 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:06:53.064+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:06:53.064976+0000) 2024-03-04T17:06:54.048 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:06:54.064+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:06:54.065172+0000) 2024-03-04T17:06:55.050 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:06:55.064+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:06:55.065407+0000) 2024-03-04T17:06:56.049 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:06:56.065+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:06:56.065655+0000) 2024-03-04T17:06:57.049 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:06:57.065+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:06:57.065945+0000) 2024-03-04T17:06:58.049 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:06:58.065+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:06:58.066192+0000) 2024-03-04T17:06:59.049 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:06:59.065+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:06:59.066427+0000) 2024-03-04T17:07:00.050 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:07:00.066+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:07:00.066692+0000) 2024-03-04T17:07:00.581 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:07:00.590 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:07:01.050 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:07:01.066+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:07:01.066968+0000) 2024-03-04T17:07:02.050 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:07:02.066+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:07:02.067244+0000) 2024-03-04T17:07:03.050 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:07:03.067+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:07:03.067473+0000) 2024-03-04T17:07:04.051 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:07:04.067+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:07:04.067657+0000) 2024-03-04T17:07:05.051 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:07:05.067+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:07:05.067938+0000) 2024-03-04T17:07:06.051 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:07:06.067+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:07:06.068180+0000) 2024-03-04T17:07:07.052 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:07:07.068+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:07:07.068420+0000) 2024-03-04T17:07:08.052 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:07:08.068+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:07:08.068669+0000) 2024-03-04T17:07:09.052 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:07:09.068+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:07:09.068955+0000) 2024-03-04T17:07:10.052 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:07:10.068+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:07:10.069192+0000) 2024-03-04T17:07:11.053 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:07:11.068+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:07:11.069424+0000) 2024-03-04T17:07:12.053 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:07:12.067+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:07:12.069690+0000) 2024-03-04T17:07:13.053 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:07:13.068+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:07:13.069934+0000) 2024-03-04T17:07:14.053 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:07:14.068+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:07:14.070188+0000) 2024-03-04T17:07:15.054 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:07:15.069+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:07:15.070458+0000) 2024-03-04T17:07:16.054 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:07:16.069+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:07:16.070713+0000) 2024-03-04T17:07:17.054 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:07:17.069+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:07:17.070994+0000) 2024-03-04T17:07:18.055 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:07:18.069+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:07:18.071197+0000) 2024-03-04T17:07:19.055 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:07:19.070+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:07:19.071454+0000) 2024-03-04T17:07:20.055 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:07:20.070+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:07:20.071709+0000) 2024-03-04T17:07:21.055 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:07:21.071+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:07:21.071967+0000) 2024-03-04T17:07:22.056 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:07:22.071+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:07:22.072232+0000) 2024-03-04T17:07:23.056 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:07:23.071+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:07:23.072550+0000) 2024-03-04T17:07:24.057 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:07:24.072+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:07:24.072807+0000) 2024-03-04T17:07:25.057 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:07:25.072+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:07:25.073110+0000) 2024-03-04T17:07:26.057 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:07:26.072+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:07:26.073394+0000) 2024-03-04T17:07:27.057 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:07:27.072+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:07:27.073654+0000) 2024-03-04T17:07:28.058 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:07:28.073+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:07:28.073955+0000) 2024-03-04T17:07:29.058 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:07:29.073+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:07:29.074209+0000) 2024-03-04T17:07:30.059 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:07:30.073+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:07:30.074477+0000) 2024-03-04T17:07:30.786 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:07:30.795 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:07:31.058 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:07:31.073+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:07:31.074751+0000) 2024-03-04T17:07:32.059 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:07:32.074+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:07:32.075047+0000) 2024-03-04T17:07:33.059 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:07:33.074+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:07:33.075308+0000) 2024-03-04T17:07:34.060 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:07:34.074+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:07:34.075572+0000) 2024-03-04T17:07:35.060 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:07:35.075+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:07:35.075863+0000) 2024-03-04T17:07:36.060 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:07:36.075+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:07:36.076126+0000) 2024-03-04T17:07:37.060 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:07:37.075+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:07:37.076338+0000) 2024-03-04T17:07:38.061 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:07:38.075+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:07:38.076558+0000) 2024-03-04T17:07:39.061 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:07:39.076+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:07:39.076876+0000) 2024-03-04T17:07:40.061 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:07:40.076+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:07:40.077087+0000) 2024-03-04T17:07:41.061 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:07:41.076+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:07:41.077373+0000) 2024-03-04T17:07:42.062 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:07:42.076+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:07:42.077638+0000) 2024-03-04T17:07:43.062 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:07:43.077+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:07:43.077943+0000) 2024-03-04T17:07:44.062 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:07:44.077+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:07:44.078189+0000) 2024-03-04T17:07:45.063 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:07:45.077+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:07:45.078428+0000) 2024-03-04T17:07:46.063 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:07:46.077+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:07:46.078670+0000) 2024-03-04T17:07:47.063 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:07:47.078+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:07:47.078972+0000) 2024-03-04T17:07:48.063 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:07:48.078+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:07:48.079241+0000) 2024-03-04T17:07:49.064 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:07:49.078+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:07:49.079489+0000) 2024-03-04T17:07:50.064 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:07:50.079+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:07:50.079728+0000) 2024-03-04T17:07:51.064 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:07:51.079+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:07:51.079968+0000) 2024-03-04T17:07:52.065 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:07:52.079+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:07:52.080218+0000) 2024-03-04T17:07:53.065 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:07:53.080+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:07:53.080465+0000) 2024-03-04T17:07:54.065 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:07:54.079+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:07:54.080706+0000) 2024-03-04T17:07:55.065 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:07:55.080+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:07:55.080950+0000) 2024-03-04T17:07:56.066 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:07:56.080+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:07:56.081196+0000) 2024-03-04T17:07:57.066 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:07:57.081+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:07:57.081441+0000) 2024-03-04T17:07:58.066 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:07:58.080+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:07:58.081674+0000) 2024-03-04T17:07:59.067 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:07:59.081+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:07:59.081943+0000) 2024-03-04T17:08:00.067 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:08:00.081+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:08:00.082205+0000) 2024-03-04T17:08:01.027 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:08:01.035 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:08:01.067 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:08:01.081+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:08:01.082411+0000) 2024-03-04T17:08:02.067 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:08:02.081+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:08:02.082620+0000) 2024-03-04T17:08:03.067 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:08:03.081+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:08:03.082805+0000) 2024-03-04T17:08:04.068 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:08:04.082+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:08:04.083107+0000) 2024-03-04T17:08:05.068 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:08:05.083+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:08:05.083353+0000) 2024-03-04T17:08:06.069 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:08:06.083+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:08:06.083594+0000) 2024-03-04T17:08:07.069 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:08:07.083+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:08:07.083909+0000) 2024-03-04T17:08:08.069 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:08:08.083+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:08:08.084174+0000) 2024-03-04T17:08:09.069 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:08:09.083+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:08:09.084431+0000) 2024-03-04T17:08:10.070 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:08:10.083+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:08:10.084694+0000) 2024-03-04T17:08:11.070 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:08:11.084+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:08:11.084945+0000) 2024-03-04T17:08:12.070 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:08:12.085+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:08:12.085202+0000) 2024-03-04T17:08:13.071 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:08:13.085+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:08:13.085452+0000) 2024-03-04T17:08:14.071 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:08:14.084+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:08:14.085685+0000) 2024-03-04T17:08:15.071 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:08:15.084+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:08:15.085988+0000) 2024-03-04T17:08:16.071 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:08:16.085+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:08:16.086231+0000) 2024-03-04T17:08:17.072 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:08:17.085+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:08:17.086493+0000) 2024-03-04T17:08:18.072 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:08:18.085+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:08:18.086734+0000) 2024-03-04T17:08:19.072 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:08:19.085+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:08:19.086980+0000) 2024-03-04T17:08:20.073 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:08:20.086+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:08:20.087223+0000) 2024-03-04T17:08:21.073 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:08:21.086+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:08:21.087458+0000) 2024-03-04T17:08:22.073 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:08:22.086+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:08:22.087712+0000) 2024-03-04T17:08:23.073 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:08:23.086+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:08:23.087955+0000) 2024-03-04T17:08:24.074 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:08:24.087+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:08:24.088195+0000) 2024-03-04T17:08:25.074 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:08:25.087+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:08:25.088437+0000) 2024-03-04T17:08:26.074 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:08:26.087+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:08:26.088705+0000) 2024-03-04T17:08:27.074 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:08:27.088+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:08:27.088943+0000) 2024-03-04T17:08:28.075 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:08:28.088+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:08:28.089191+0000) 2024-03-04T17:08:29.075 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:08:29.088+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:08:29.089451+0000) 2024-03-04T17:08:30.075 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:08:30.088+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:08:30.089711+0000) 2024-03-04T17:08:31.076 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:08:31.089+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:08:31.089958+0000) 2024-03-04T17:08:31.184 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:08:31.192 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:08:32.076 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:08:32.089+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:08:32.090159+0000) 2024-03-04T17:08:33.076 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:08:33.089+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:08:33.090426+0000) 2024-03-04T17:08:34.076 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:08:34.089+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:08:34.090672+0000) 2024-03-04T17:08:35.077 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:08:35.090+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:08:35.090930+0000) 2024-03-04T17:08:36.077 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:08:36.090+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:08:36.091173+0000) 2024-03-04T17:08:37.077 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:08:37.090+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:08:37.091425+0000) 2024-03-04T17:08:38.077 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:08:38.090+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:08:38.091683+0000) 2024-03-04T17:08:39.078 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:08:39.091+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:08:39.091925+0000) 2024-03-04T17:08:40.078 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:08:40.091+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:08:40.092167+0000) 2024-03-04T17:08:41.078 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:08:41.091+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:08:41.092407+0000) 2024-03-04T17:08:42.078 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:08:42.091+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:08:42.092669+0000) 2024-03-04T17:08:43.079 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:08:43.092+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:08:43.092955+0000) 2024-03-04T17:08:44.079 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:08:44.092+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:08:44.093198+0000) 2024-03-04T17:08:45.079 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:08:45.092+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:08:45.093459+0000) 2024-03-04T17:08:46.079 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:08:46.092+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:08:46.093709+0000) 2024-03-04T17:08:47.080 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:08:47.093+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:08:47.093959+0000) 2024-03-04T17:08:48.080 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:08:48.093+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:08:48.094174+0000) 2024-03-04T17:08:49.080 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:08:49.093+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:08:49.094414+0000) 2024-03-04T17:08:50.081 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:08:50.094+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:08:50.094661+0000) 2024-03-04T17:08:51.081 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:08:51.094+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:08:51.094959+0000) 2024-03-04T17:08:52.081 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:08:52.094+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:08:52.095215+0000) 2024-03-04T17:08:53.081 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:08:53.094+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:08:53.095436+0000) 2024-03-04T17:08:54.082 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:08:54.095+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:08:54.095681+0000) 2024-03-04T17:08:55.082 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:08:55.095+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:08:55.095955+0000) 2024-03-04T17:08:56.082 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:08:56.095+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:08:56.096138+0000) 2024-03-04T17:08:57.083 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:08:57.095+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:08:57.096384+0000) 2024-03-04T17:08:58.083 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:08:58.095+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:08:58.096635+0000) 2024-03-04T17:08:59.083 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:08:59.096+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:08:59.096942+0000) 2024-03-04T17:09:00.083 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:09:00.096+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:09:00.097181+0000) 2024-03-04T17:09:01.084 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:09:01.097+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:09:01.097448+0000) 2024-03-04T17:09:01.420 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:09:01.428 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:09:02.084 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:09:02.097+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:09:02.097708+0000) 2024-03-04T17:09:03.084 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:09:03.097+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:09:03.097989+0000) 2024-03-04T17:09:04.085 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:09:04.097+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:09:04.098271+0000) 2024-03-04T17:09:05.085 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:09:05.097+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:09:05.098529+0000) 2024-03-04T17:09:06.087 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:09:06.097+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:09:06.098787+0000) 2024-03-04T17:09:07.085 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:09:07.098+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:09:07.099047+0000) 2024-03-04T17:09:08.086 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:09:08.098+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:09:08.099237+0000) 2024-03-04T17:09:09.086 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:09:09.099+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:09:09.099531+0000) 2024-03-04T17:09:10.086 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:09:10.098+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:09:10.099718+0000) 2024-03-04T17:09:11.087 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:09:11.099+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:09:11.099954+0000) 2024-03-04T17:09:12.087 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:09:12.100+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:09:12.100217+0000) 2024-03-04T17:09:13.087 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:09:13.099+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:09:13.100456+0000) 2024-03-04T17:09:14.087 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:09:14.099+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:09:14.100646+0000) 2024-03-04T17:09:15.088 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:09:15.100+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:09:15.100932+0000) 2024-03-04T17:09:16.088 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:09:16.101+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:09:16.101176+0000) 2024-03-04T17:09:17.088 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:09:17.101+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:09:17.101428+0000) 2024-03-04T17:09:18.088 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:09:18.099+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:09:18.101693+0000) 2024-03-04T17:09:19.089 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:09:19.100+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:09:19.101959+0000) 2024-03-04T17:09:20.089 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:09:20.101+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:09:20.102211+0000) 2024-03-04T17:09:21.089 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:09:21.101+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:09:21.102471+0000) 2024-03-04T17:09:22.090 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:09:22.100+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:09:22.102762+0000) 2024-03-04T17:09:23.090 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:09:23.101+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:09:23.103046+0000) 2024-03-04T17:09:24.090 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:09:24.101+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:09:24.103268+0000) 2024-03-04T17:09:25.090 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:09:25.102+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:09:25.103516+0000) 2024-03-04T17:09:26.091 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:09:26.102+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:09:26.103762+0000) 2024-03-04T17:09:27.091 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:09:27.103+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:09:27.104045+0000) 2024-03-04T17:09:28.091 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:09:28.102+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:09:28.104272+0000) 2024-03-04T17:09:29.092 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:09:29.103+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:09:29.104571+0000) 2024-03-04T17:09:30.092 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:09:30.104+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:09:30.104867+0000) 2024-03-04T17:09:31.092 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:09:31.104+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:09:31.105128+0000) 2024-03-04T17:09:31.643 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:09:31.652 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:09:32.093 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:09:32.104+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:09:32.105409+0000) 2024-03-04T17:09:33.093 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:09:33.104+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:09:33.105657+0000) 2024-03-04T17:09:34.093 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:09:34.105+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:09:34.105942+0000) 2024-03-04T17:09:35.093 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:09:35.105+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:09:35.106203+0000) 2024-03-04T17:09:36.094 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:09:36.105+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:09:36.106460+0000) 2024-03-04T17:09:37.094 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:09:37.105+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:09:37.106738+0000) 2024-03-04T17:09:38.094 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:09:38.106+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:09:38.106989+0000) 2024-03-04T17:09:39.094 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:09:39.106+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:09:39.107251+0000) 2024-03-04T17:09:40.095 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:09:40.106+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:09:40.107512+0000) 2024-03-04T17:09:41.095 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:09:41.107+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:09:41.107773+0000) 2024-03-04T17:09:42.095 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:09:42.107+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:09:42.108078+0000) 2024-03-04T17:09:43.096 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:09:43.107+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:09:43.108271+0000) 2024-03-04T17:09:44.096 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:09:44.107+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:09:44.108525+0000) 2024-03-04T17:09:45.097 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:09:45.108+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:09:45.108765+0000) 2024-03-04T17:09:46.096 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:09:46.108+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:09:46.109016+0000) 2024-03-04T17:09:47.097 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:09:47.108+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:09:47.109262+0000) 2024-03-04T17:09:48.097 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:09:48.108+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:09:48.109460+0000) 2024-03-04T17:09:49.097 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:09:49.109+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:09:49.109698+0000) 2024-03-04T17:09:50.098 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:09:50.109+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:09:50.109942+0000) 2024-03-04T17:09:51.098 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:09:51.109+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:09:51.110240+0000) 2024-03-04T17:09:52.098 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:09:52.109+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:09:52.110495+0000) 2024-03-04T17:09:53.098 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:09:53.109+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:09:53.110757+0000) 2024-03-04T17:09:54.099 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:09:54.110+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:09:54.111001+0000) 2024-03-04T17:09:55.099 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:09:55.110+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:09:55.111259+0000) 2024-03-04T17:09:56.099 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:09:56.110+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:09:56.111515+0000) 2024-03-04T17:09:57.099 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:09:57.111+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:09:57.111756+0000) 2024-03-04T17:09:58.100 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:09:58.111+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:09:58.111990+0000) 2024-03-04T17:09:59.100 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:09:59.111+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:09:59.112252+0000) 2024-03-04T17:10:00.100 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:10:00.112+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:10:00.112511+0000) 2024-03-04T17:10:01.101 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:10:01.112+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:10:01.112766+0000) 2024-03-04T17:10:01.858 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:10:01.866 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:10:02.101 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:10:02.112+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:10:02.113028+0000) 2024-03-04T17:10:03.101 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:10:03.112+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:10:03.113392+0000) 2024-03-04T17:10:04.101 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:10:04.113+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:10:04.113652+0000) 2024-03-04T17:10:05.102 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:10:05.113+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:10:05.113922+0000) 2024-03-04T17:10:06.102 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:10:06.113+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:10:06.114166+0000) 2024-03-04T17:10:07.102 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:10:07.114+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:10:07.114408+0000) 2024-03-04T17:10:08.102 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:10:08.113+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:10:08.114606+0000) 2024-03-04T17:10:09.103 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:10:09.114+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:10:09.114880+0000) 2024-03-04T17:10:10.103 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:10:10.114+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:10:10.115131+0000) 2024-03-04T17:10:11.103 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:10:11.114+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:10:11.115389+0000) 2024-03-04T17:10:12.104 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:10:12.114+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:10:12.115605+0000) 2024-03-04T17:10:13.104 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:10:13.115+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:10:13.115894+0000) 2024-03-04T17:10:14.104 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:10:14.115+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:10:14.116136+0000) 2024-03-04T17:10:15.104 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:10:15.116+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:10:15.116376+0000) 2024-03-04T17:10:16.105 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:10:16.116+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:10:16.116618+0000) 2024-03-04T17:10:17.105 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:10:17.116+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:10:17.116896+0000) 2024-03-04T17:10:18.105 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:10:18.116+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:10:18.117152+0000) 2024-03-04T17:10:19.106 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:10:19.116+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:10:19.117408+0000) 2024-03-04T17:10:20.106 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:10:20.117+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:10:20.117664+0000) 2024-03-04T17:10:21.106 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:10:21.117+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:10:21.117940+0000) 2024-03-04T17:10:22.106 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:10:22.116+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:10:22.118178+0000) 2024-03-04T17:10:23.107 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:10:23.117+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:10:23.118426+0000) 2024-03-04T17:10:24.107 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:10:24.117+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:10:24.118668+0000) 2024-03-04T17:10:25.107 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:10:25.117+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:10:25.118956+0000) 2024-03-04T17:10:26.107 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:10:26.118+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:10:26.119204+0000) 2024-03-04T17:10:27.108 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:10:27.117+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:10:27.119690+0000) 2024-03-04T17:10:28.108 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:10:28.118+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:10:28.119970+0000) 2024-03-04T17:10:29.109 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:10:29.119+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:10:29.120219+0000) 2024-03-04T17:10:30.109 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:10:30.119+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:10:30.120460+0000) 2024-03-04T17:10:31.110 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:10:31.118+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:10:31.120718+0000) 2024-03-04T17:10:32.073 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:10:32.085 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:10:32.109 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:10:32.119+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:10:32.120923+0000) 2024-03-04T17:10:33.110 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:10:33.120+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:10:33.121169+0000) 2024-03-04T17:10:34.110 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:10:34.120+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:10:34.121408+0000) 2024-03-04T17:10:35.111 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:10:35.120+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:10:35.121716+0000) 2024-03-04T17:10:36.110 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:10:36.121+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:10:36.122021+0000) 2024-03-04T17:10:37.111 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:10:37.121+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:10:37.122317+0000) 2024-03-04T17:10:38.111 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:10:38.121+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:10:38.122572+0000) 2024-03-04T17:10:39.112 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:10:39.122+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:10:39.122952+0000) 2024-03-04T17:10:40.112 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:10:40.122+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:10:40.123244+0000) 2024-03-04T17:10:41.113 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:10:41.122+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:10:41.123543+0000) 2024-03-04T17:10:42.112 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:10:42.123+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:10:42.123938+0000) 2024-03-04T17:10:43.113 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:10:43.123+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:10:43.124197+0000) 2024-03-04T17:10:44.113 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:10:44.123+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:10:44.124516+0000) 2024-03-04T17:10:45.113 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:10:45.124+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:10:45.124814+0000) 2024-03-04T17:10:46.114 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:10:46.124+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:10:46.125165+0000) 2024-03-04T17:10:47.114 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:10:47.124+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:10:47.125461+0000) 2024-03-04T17:10:48.115 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:10:48.124+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:10:48.125702+0000) 2024-03-04T17:10:49.115 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:10:49.125+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:10:49.126008+0000) 2024-03-04T17:10:50.115 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:10:50.125+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:10:50.126260+0000) 2024-03-04T17:10:51.115 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:10:51.125+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:10:51.126494+0000) 2024-03-04T17:10:52.116 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:10:52.126+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:10:52.126885+0000) 2024-03-04T17:10:53.116 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:10:53.126+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:10:53.127165+0000) 2024-03-04T17:10:54.116 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:10:54.126+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:10:54.127457+0000) 2024-03-04T17:10:55.117 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:10:55.126+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:10:55.127737+0000) 2024-03-04T17:10:56.117 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:10:56.127+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:10:56.128024+0000) 2024-03-04T17:10:57.117 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:10:57.127+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:10:57.128315+0000) 2024-03-04T17:10:58.118 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:10:58.127+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:10:58.128612+0000) 2024-03-04T17:10:59.118 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:10:59.128+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:10:59.128938+0000) 2024-03-04T17:11:00.118 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:11:00.128+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:11:00.129175+0000) 2024-03-04T17:11:01.118 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:11:01.128+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:11:01.129478+0000) 2024-03-04T17:11:02.119 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:11:02.129+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:11:02.129771+0000) 2024-03-04T17:11:02.218 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:11:02.227 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:11:03.119 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:11:03.129+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:11:03.130033+0000) 2024-03-04T17:11:04.119 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:11:04.129+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:11:04.130332+0000) 2024-03-04T17:11:05.120 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:11:05.130+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:11:05.130631+0000) 2024-03-04T17:11:06.120 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:11:06.130+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:11:06.130972+0000) 2024-03-04T17:11:07.120 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:11:07.130+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:11:07.131231+0000) 2024-03-04T17:11:08.121 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:11:08.130+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:11:08.131478+0000) 2024-03-04T17:11:09.121 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:11:09.130+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:11:09.131733+0000) 2024-03-04T17:11:10.121 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:11:10.131+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:11:10.131981+0000) 2024-03-04T17:11:11.121 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:11:11.131+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:11:11.132228+0000) 2024-03-04T17:11:12.122 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:11:12.131+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:11:12.132421+0000) 2024-03-04T17:11:13.122 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:11:13.131+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:11:13.132614+0000) 2024-03-04T17:11:14.122 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:11:14.132+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:11:14.132912+0000) 2024-03-04T17:11:15.122 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:11:15.132+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:11:15.133141+0000) 2024-03-04T17:11:16.123 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:11:16.133+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:11:16.133370+0000) 2024-03-04T17:11:17.123 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:11:17.133+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:11:17.133620+0000) 2024-03-04T17:11:18.123 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:11:18.133+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:11:18.133871+0000) 2024-03-04T17:11:19.124 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:11:19.133+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:11:19.134155+0000) 2024-03-04T17:11:20.124 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:11:20.134+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:11:20.134395+0000) 2024-03-04T17:11:21.124 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:11:21.134+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:11:21.134643+0000) 2024-03-04T17:11:22.124 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:11:22.134+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:11:22.134952+0000) 2024-03-04T17:11:23.125 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:11:23.134+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:11:23.135147+0000) 2024-03-04T17:11:24.125 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:11:24.135+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:11:24.135390+0000) 2024-03-04T17:11:25.126 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:11:25.135+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:11:25.135637+0000) 2024-03-04T17:11:26.125 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:11:26.134+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:11:26.135934+0000) 2024-03-04T17:11:27.126 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:11:27.135+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:11:27.136181+0000) 2024-03-04T17:11:28.126 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:11:28.134+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:11:28.136380+0000) 2024-03-04T17:11:29.126 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:11:29.135+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:11:29.136624+0000) 2024-03-04T17:11:30.126 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:11:30.135+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:11:30.136921+0000) 2024-03-04T17:11:31.127 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:11:31.135+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:11:31.137160+0000) 2024-03-04T17:11:32.127 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:11:32.136+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:11:32.137411+0000) 2024-03-04T17:11:32.459 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:11:32.467 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:11:33.127 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:11:33.135+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:11:33.137600+0000) 2024-03-04T17:11:34.128 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:11:34.136+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:11:34.137872+0000) 2024-03-04T17:11:35.128 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:11:35.137+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:11:35.138107+0000) 2024-03-04T17:11:36.128 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:11:36.137+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:11:36.138343+0000) 2024-03-04T17:11:37.128 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:11:37.137+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:11:37.138591+0000) 2024-03-04T17:11:38.129 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:11:38.138+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:11:38.138929+0000) 2024-03-04T17:11:39.129 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:11:39.138+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:11:39.139182+0000) 2024-03-04T17:11:40.129 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:11:40.138+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:11:40.139420+0000) 2024-03-04T17:11:41.130 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:11:41.138+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:11:41.139664+0000) 2024-03-04T17:11:42.130 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:11:42.139+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:11:42.139945+0000) 2024-03-04T17:11:43.130 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:11:43.139+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:11:43.140192+0000) 2024-03-04T17:11:44.130 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:11:44.139+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:11:44.140440+0000) 2024-03-04T17:11:45.131 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:11:45.139+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:11:45.140704+0000) 2024-03-04T17:11:46.131 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:11:46.140+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:11:46.140956+0000) 2024-03-04T17:11:47.131 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:11:47.140+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:11:47.141159+0000) 2024-03-04T17:11:48.131 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:11:48.140+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:11:48.141408+0000) 2024-03-04T17:11:49.132 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:11:49.140+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:11:49.141653+0000) 2024-03-04T17:11:50.132 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:11:50.141+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:11:50.141923+0000) 2024-03-04T17:11:51.132 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:11:51.141+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:11:51.142161+0000) 2024-03-04T17:11:52.132 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:11:52.141+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:11:52.142409+0000) 2024-03-04T17:11:53.133 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:11:53.141+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:11:53.142681+0000) 2024-03-04T17:11:54.133 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:11:54.142+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:11:54.142954+0000) 2024-03-04T17:11:55.133 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:11:55.142+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:11:55.143193+0000) 2024-03-04T17:11:56.133 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:11:56.142+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:11:56.143439+0000) 2024-03-04T17:11:57.134 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:11:57.143+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:11:57.143689+0000) 2024-03-04T17:11:58.134 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:11:58.143+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:11:58.143958+0000) 2024-03-04T17:11:59.134 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:11:59.143+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:11:59.144208+0000) 2024-03-04T17:12:00.135 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:12:00.143+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:12:00.144457+0000) 2024-03-04T17:12:01.135 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:12:01.144+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:12:01.144702+0000) 2024-03-04T17:12:02.135 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:12:02.144+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:12:02.144954+0000) 2024-03-04T17:12:02.605 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:12:02.617 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:12:03.135 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:12:03.144+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:12:03.145202+0000) 2024-03-04T17:12:04.136 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:12:04.144+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:12:04.145393+0000) 2024-03-04T17:12:05.136 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:12:05.145+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:12:05.145637+0000) 2024-03-04T17:12:06.136 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:12:06.145+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:12:06.145847+0000) 2024-03-04T17:12:07.136 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:12:07.145+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:12:07.146104+0000) 2024-03-04T17:12:08.137 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:12:08.145+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:12:08.146349+0000) 2024-03-04T17:12:09.137 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:12:09.146+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:12:09.146602+0000) 2024-03-04T17:12:10.137 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:12:10.146+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:12:10.146884+0000) 2024-03-04T17:12:11.137 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:12:11.146+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:12:11.147124+0000) 2024-03-04T17:12:12.138 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:12:12.146+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:12:12.147381+0000) 2024-03-04T17:12:13.138 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:12:13.147+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:12:13.147629+0000) 2024-03-04T17:12:14.138 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:12:14.147+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:12:14.147925+0000) 2024-03-04T17:12:15.138 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:12:15.147+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:12:15.148180+0000) 2024-03-04T17:12:16.139 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:12:16.148+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:12:16.148412+0000) 2024-03-04T17:12:17.139 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:12:17.148+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:12:17.148604+0000) 2024-03-04T17:12:18.140 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:12:18.148+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:12:18.148800+0000) 2024-03-04T17:12:19.140 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:12:19.148+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:12:19.149089+0000) 2024-03-04T17:12:20.140 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:12:20.149+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:12:20.149327+0000) 2024-03-04T17:12:21.140 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:12:21.149+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:12:21.149573+0000) 2024-03-04T17:12:22.140 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:12:22.149+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:12:22.149856+0000) 2024-03-04T17:12:23.141 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:12:23.149+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:12:23.150125+0000) 2024-03-04T17:12:24.141 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:12:24.150+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:12:24.150386+0000) 2024-03-04T17:12:25.141 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:12:25.150+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:12:25.150628+0000) 2024-03-04T17:12:26.141 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:12:26.150+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:12:26.150972+0000) 2024-03-04T17:12:27.142 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:12:27.151+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:12:27.151220+0000) 2024-03-04T17:12:28.142 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:12:28.151+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:12:28.151474+0000) 2024-03-04T17:12:29.142 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:12:29.151+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:12:29.151717+0000) 2024-03-04T17:12:30.143 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:12:30.151+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:12:30.151953+0000) 2024-03-04T17:12:31.143 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:12:31.150+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:12:31.152146+0000) 2024-03-04T17:12:32.143 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:12:32.151+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:12:32.152397+0000) 2024-03-04T17:12:32.811 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:12:32.819 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:12:33.143 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:12:33.151+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:12:33.152663+0000) 2024-03-04T17:12:34.144 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:12:34.151+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:12:34.152943+0000) 2024-03-04T17:12:35.144 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:12:35.152+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:12:35.153195+0000) 2024-03-04T17:12:36.144 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:12:36.152+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:12:36.153434+0000) 2024-03-04T17:12:37.144 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:12:37.152+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:12:37.153673+0000) 2024-03-04T17:12:38.145 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:12:38.152+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:12:38.153951+0000) 2024-03-04T17:12:39.145 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:12:39.153+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:12:39.154193+0000) 2024-03-04T17:12:40.145 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:12:40.153+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:12:40.154430+0000) 2024-03-04T17:12:41.146 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:12:41.153+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:12:41.154670+0000) 2024-03-04T17:12:42.146 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:12:42.154+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:12:42.154932+0000) 2024-03-04T17:12:43.146 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:12:43.154+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:12:43.155213+0000) 2024-03-04T17:12:44.146 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:12:44.154+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:12:44.155463+0000) 2024-03-04T17:12:45.147 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:12:45.154+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:12:45.155670+0000) 2024-03-04T17:12:46.147 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:12:46.155+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:12:46.155932+0000) 2024-03-04T17:12:47.147 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:12:47.155+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:12:47.156135+0000) 2024-03-04T17:12:48.147 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:12:48.155+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:12:48.156406+0000) 2024-03-04T17:12:49.148 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:12:49.155+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:12:49.156660+0000) 2024-03-04T17:12:50.148 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:12:50.156+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:12:50.156930+0000) 2024-03-04T17:12:51.148 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:12:51.156+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:12:51.157176+0000) 2024-03-04T17:12:52.148 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:12:52.156+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:12:52.157431+0000) 2024-03-04T17:12:53.149 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:12:53.156+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:12:53.157698+0000) 2024-03-04T17:12:54.149 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:12:54.157+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:12:54.157970+0000) 2024-03-04T17:12:55.149 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:12:55.157+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:12:55.158213+0000) 2024-03-04T17:12:56.150 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:12:56.157+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:12:56.158461+0000) 2024-03-04T17:12:57.150 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:12:57.157+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:12:57.158701+0000) 2024-03-04T17:12:58.150 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:12:58.158+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:12:58.158956+0000) 2024-03-04T17:12:59.150 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:12:59.158+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:12:59.159159+0000) 2024-03-04T17:13:00.151 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:13:00.158+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:13:00.159399+0000) 2024-03-04T17:13:01.151 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:13:01.158+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:13:01.159648+0000) 2024-03-04T17:13:02.151 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:13:02.159+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:13:02.159960+0000) 2024-03-04T17:13:03.034 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:13:03.043 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:13:03.151 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:13:03.159+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:13:03.160163+0000) 2024-03-04T17:13:04.151 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:13:04.159+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:13:04.160410+0000) 2024-03-04T17:13:05.152 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:13:05.160+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:13:05.160652+0000) 2024-03-04T17:13:06.152 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:13:06.160+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:13:06.160941+0000) 2024-03-04T17:13:07.152 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:13:07.160+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:13:07.161185+0000) 2024-03-04T17:13:08.153 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:13:08.160+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:13:08.161429+0000) 2024-03-04T17:13:09.153 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:13:09.161+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:13:09.161678+0000) 2024-03-04T17:13:10.153 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:13:10.161+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:13:10.161948+0000) 2024-03-04T17:13:11.154 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:13:11.161+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:13:11.162163+0000) 2024-03-04T17:13:12.154 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:13:12.161+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:13:12.162367+0000) 2024-03-04T17:13:13.154 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:13:13.162+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:13:13.162620+0000) 2024-03-04T17:13:14.154 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:13:14.162+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:13:14.162895+0000) 2024-03-04T17:13:15.155 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:13:15.162+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:13:15.163142+0000) 2024-03-04T17:13:16.155 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:13:16.162+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:13:16.163389+0000) 2024-03-04T17:13:17.155 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:13:17.162+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:13:17.163590+0000) 2024-03-04T17:13:18.155 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:13:18.163+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:13:18.163869+0000) 2024-03-04T17:13:19.156 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:13:19.163+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:13:19.164185+0000) 2024-03-04T17:13:20.156 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:13:20.163+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:13:20.164424+0000) 2024-03-04T17:13:21.156 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:13:21.163+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:13:21.164662+0000) 2024-03-04T17:13:22.156 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:13:22.164+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:13:22.164950+0000) 2024-03-04T17:13:23.157 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:13:23.164+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:13:23.165156+0000) 2024-03-04T17:13:24.157 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:13:24.165+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:13:24.165412+0000) 2024-03-04T17:13:25.157 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:13:25.164+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:13:25.165667+0000) 2024-03-04T17:13:26.157 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:13:26.165+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:13:26.165933+0000) 2024-03-04T17:13:27.158 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:13:27.165+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:13:27.166166+0000) 2024-03-04T17:13:28.158 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:13:28.166+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:13:28.166442+0000) 2024-03-04T17:13:29.158 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:13:29.166+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:13:29.166682+0000) 2024-03-04T17:13:30.159 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:13:30.166+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:13:30.166959+0000) 2024-03-04T17:13:31.159 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:13:31.166+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:13:31.167199+0000) 2024-03-04T17:13:32.159 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:13:32.167+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:13:32.167449+0000) 2024-03-04T17:13:33.159 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:13:33.167+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:13:33.167699+0000) 2024-03-04T17:13:33.194 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:13:33.203 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:13:34.160 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:13:34.167+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:13:34.167939+0000) 2024-03-04T17:13:35.160 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:13:35.166+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:13:35.168175+0000) 2024-03-04T17:13:36.160 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:13:36.167+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:13:36.168416+0000) 2024-03-04T17:13:37.160 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:13:37.167+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:13:37.168654+0000) 2024-03-04T17:13:38.161 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:13:38.167+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:13:38.168940+0000) 2024-03-04T17:13:39.164 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:13:39.167+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:13:39.169178+0000) 2024-03-04T17:13:40.161 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:13:40.168+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:13:40.169436+0000) 2024-03-04T17:13:41.162 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:13:41.167+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:13:41.169705+0000) 2024-03-04T17:13:42.162 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:13:42.168+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:13:42.169950+0000) 2024-03-04T17:13:43.162 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:13:43.169+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:13:43.170204+0000) 2024-03-04T17:13:44.162 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:13:44.169+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:13:44.170436+0000) 2024-03-04T17:13:45.164 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:13:45.168+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:13:45.170669+0000) 2024-03-04T17:13:46.163 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:13:46.170+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:13:46.170947+0000) 2024-03-04T17:13:47.163 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:13:47.170+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:13:47.171147+0000) 2024-03-04T17:13:48.163 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:13:48.170+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:13:48.171394+0000) 2024-03-04T17:13:49.164 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:13:49.170+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:13:49.171638+0000) 2024-03-04T17:13:50.164 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:13:50.171+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:13:50.171920+0000) 2024-03-04T17:13:51.164 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:13:51.171+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:13:51.172167+0000) 2024-03-04T17:13:52.164 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:13:52.171+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:13:52.172374+0000) 2024-03-04T17:13:53.165 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:13:53.171+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:13:53.172573+0000) 2024-03-04T17:13:54.165 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:13:54.172+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:13:54.172864+0000) 2024-03-04T17:13:55.165 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:13:55.172+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:13:55.173071+0000) 2024-03-04T17:13:56.165 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:13:56.172+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:13:56.173267+0000) 2024-03-04T17:13:57.166 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:13:57.172+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:13:57.173517+0000) 2024-03-04T17:13:58.166 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:13:58.173+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:13:58.173766+0000) 2024-03-04T17:13:59.166 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:13:59.173+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:13:59.174003+0000) 2024-03-04T17:14:00.166 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:14:00.173+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:14:00.174255+0000) 2024-03-04T17:14:01.167 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:14:01.173+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:14:01.174529+0000) 2024-03-04T17:14:02.167 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:14:02.174+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:14:02.174928+0000) 2024-03-04T17:14:03.167 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:14:03.174+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:14:03.175190+0000) 2024-03-04T17:14:03.403 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:14:03.412 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:14:04.168 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:14:04.174+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:14:04.175492+0000) 2024-03-04T17:14:05.168 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:14:05.174+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:14:05.175788+0000) 2024-03-04T17:14:06.168 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:14:06.175+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:14:06.176148+0000) 2024-03-04T17:14:07.168 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:14:07.175+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:14:07.176424+0000) 2024-03-04T17:14:08.169 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:14:08.175+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:14:08.176737+0000) 2024-03-04T17:14:09.169 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:14:09.176+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:14:09.177012+0000) 2024-03-04T17:14:10.170 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:14:10.176+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:14:10.177312+0000) 2024-03-04T17:14:11.170 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:14:11.176+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:14:11.177551+0000) 2024-03-04T17:14:12.170 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:14:12.177+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:14:12.177862+0000) 2024-03-04T17:14:13.170 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:14:13.177+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:14:13.178166+0000) 2024-03-04T17:14:14.171 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:14:14.177+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:14:14.178385+0000) 2024-03-04T17:14:15.171 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:14:15.178+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:14:15.178686+0000) 2024-03-04T17:14:16.171 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:14:16.178+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:14:16.178982+0000) 2024-03-04T17:14:17.172 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:14:17.178+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:14:17.179276+0000) 2024-03-04T17:14:18.172 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:14:18.179+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:14:18.179599+0000) 2024-03-04T17:14:19.172 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:14:19.179+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:14:19.179964+0000) 2024-03-04T17:14:20.173 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:14:20.179+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:14:20.180262+0000) 2024-03-04T17:14:21.173 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:14:21.180+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:14:21.180579+0000) 2024-03-04T17:14:22.173 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:14:22.180+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:14:22.180930+0000) 2024-03-04T17:14:23.174 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:14:23.180+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:14:23.181253+0000) 2024-03-04T17:14:24.174 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:14:24.180+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:14:24.181561+0000) 2024-03-04T17:14:25.174 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:14:25.181+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:14:25.181882+0000) 2024-03-04T17:14:26.175 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:14:26.181+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:14:26.182137+0000) 2024-03-04T17:14:27.175 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:14:27.182+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:14:27.182428+0000) 2024-03-04T17:14:28.175 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:14:28.182+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:14:28.182728+0000) 2024-03-04T17:14:29.176 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:14:29.182+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:14:29.183040+0000) 2024-03-04T17:14:30.176 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:14:30.183+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:14:30.183335+0000) 2024-03-04T17:14:31.176 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:14:31.183+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:14:31.183631+0000) 2024-03-04T17:14:32.179 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:14:32.183+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:14:32.183979+0000) 2024-03-04T17:14:33.177 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:14:33.184+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:14:33.184275+0000) 2024-03-04T17:14:33.617 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:14:33.626 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:14:34.177 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:14:34.184+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:14:34.184557+0000) 2024-03-04T17:14:35.178 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:14:35.184+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:14:35.184902+0000) 2024-03-04T17:14:36.178 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:14:36.184+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:14:36.185186+0000) 2024-03-04T17:14:37.178 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:14:37.185+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:14:37.185502+0000) 2024-03-04T17:14:38.178 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:14:38.185+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:14:38.185804+0000) 2024-03-04T17:14:39.179 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:14:39.184+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:14:39.186144+0000) 2024-03-04T17:14:40.179 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:14:40.185+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:14:40.186442+0000) 2024-03-04T17:14:41.179 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:14:41.185+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:14:41.186736+0000) 2024-03-04T17:14:42.180 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:14:42.185+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:14:42.187037+0000) 2024-03-04T17:14:43.180 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:14:43.186+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:14:43.187346+0000) 2024-03-04T17:14:44.180 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:14:44.185+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:14:44.187641+0000) 2024-03-04T17:14:45.181 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:14:45.186+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:14:45.187978+0000) 2024-03-04T17:14:46.181 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:14:46.187+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:14:46.188294+0000) 2024-03-04T17:14:47.181 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:14:47.187+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:14:47.188611+0000) 2024-03-04T17:14:48.182 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:14:48.187+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:14:48.188940+0000) 2024-03-04T17:14:49.182 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:14:49.188+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:14:49.189216+0000) 2024-03-04T17:14:50.183 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:14:50.188+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:14:50.189528+0000) 2024-03-04T17:14:51.183 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:14:51.189+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:14:51.189864+0000) 2024-03-04T17:14:52.183 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:14:52.189+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:14:52.190169+0000) 2024-03-04T17:14:53.183 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:14:53.189+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:14:53.190465+0000) 2024-03-04T17:14:54.184 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:14:54.189+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:14:54.190762+0000) 2024-03-04T17:14:55.184 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:14:55.190+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:14:55.191062+0000) 2024-03-04T17:14:56.185 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:14:56.190+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:14:56.191355+0000) 2024-03-04T17:14:57.185 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:14:57.190+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:14:57.191646+0000) 2024-03-04T17:14:58.185 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:14:58.191+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:14:58.191987+0000) 2024-03-04T17:14:59.185 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:14:59.191+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:14:59.192290+0000) 2024-03-04T17:15:00.186 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:15:00.191+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:15:00.192567+0000) 2024-03-04T17:15:01.186 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:15:01.192+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:15:01.192885+0000) 2024-03-04T17:15:02.186 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:15:02.192+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:15:02.193194+0000) 2024-03-04T17:15:03.187 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:15:03.192+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:15:03.193496+0000) 2024-03-04T17:15:03.814 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:15:03.820 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:15:04.187 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:15:04.193+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:15:04.193787+0000) 2024-03-04T17:15:05.187 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:15:05.193+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:15:05.194146+0000) 2024-03-04T17:15:06.187 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:15:06.193+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:15:06.194439+0000) 2024-03-04T17:15:07.188 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:15:07.194+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:15:07.194743+0000) 2024-03-04T17:15:08.188 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:15:08.194+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:15:08.195026+0000) 2024-03-04T17:15:09.188 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:15:09.194+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:15:09.195320+0000) 2024-03-04T17:15:10.189 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:15:10.194+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:15:10.195639+0000) 2024-03-04T17:15:11.189 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:15:11.195+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:15:11.195989+0000) 2024-03-04T17:15:12.189 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:15:12.195+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:15:12.196305+0000) 2024-03-04T17:15:13.190 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:15:13.195+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:15:13.196600+0000) 2024-03-04T17:15:14.190 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:15:14.196+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:15:14.196926+0000) 2024-03-04T17:15:15.190 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:15:15.196+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:15:15.197226+0000) 2024-03-04T17:15:16.191 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:15:16.196+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:15:16.197497+0000) 2024-03-04T17:15:17.191 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:15:17.197+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:15:17.197798+0000) 2024-03-04T17:15:18.192 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:15:18.197+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:15:18.198148+0000) 2024-03-04T17:15:19.192 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:15:19.197+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:15:19.198392+0000) 2024-03-04T17:15:20.192 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:15:20.198+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:15:20.198674+0000) 2024-03-04T17:15:21.192 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:15:21.198+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:15:21.198976+0000) 2024-03-04T17:15:22.192 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:15:22.198+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:15:22.199266+0000) 2024-03-04T17:15:23.193 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:15:23.198+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:15:23.199501+0000) 2024-03-04T17:15:24.193 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:15:24.199+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:15:24.199800+0000) 2024-03-04T17:15:25.193 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:15:25.199+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:15:25.200090+0000) 2024-03-04T17:15:26.193 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:15:26.199+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:15:26.200325+0000) 2024-03-04T17:15:27.194 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:15:27.200+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:15:27.200619+0000) 2024-03-04T17:15:28.194 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:15:28.200+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:15:28.200939+0000) 2024-03-04T17:15:29.195 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:15:29.200+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:15:29.201232+0000) 2024-03-04T17:15:30.195 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:15:30.201+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:15:30.201531+0000) 2024-03-04T17:15:31.195 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:15:31.201+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:15:31.201864+0000) 2024-03-04T17:15:32.196 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:15:32.201+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:15:32.202209+0000) 2024-03-04T17:15:33.196 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:15:33.202+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:15:33.202526+0000) 2024-03-04T17:15:34.028 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:15:34.036 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:15:34.196 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:15:34.201+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:15:34.202802+0000) 2024-03-04T17:15:35.196 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:15:35.202+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:15:35.203144+0000) 2024-03-04T17:15:36.197 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:15:36.203+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:15:36.203464+0000) 2024-03-04T17:15:37.197 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:15:37.202+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:15:37.203775+0000) 2024-03-04T17:15:38.198 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:15:38.203+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:15:38.204085+0000) 2024-03-04T17:15:39.198 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:15:39.203+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:15:39.204326+0000) 2024-03-04T17:15:40.198 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:15:40.203+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:15:40.204556+0000) 2024-03-04T17:15:41.198 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:15:41.203+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:15:41.204794+0000) 2024-03-04T17:15:42.198 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:15:42.204+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:15:42.205174+0000) 2024-03-04T17:15:43.199 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:15:43.204+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:15:43.205424+0000) 2024-03-04T17:15:44.199 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:15:44.203+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:15:44.205685+0000) 2024-03-04T17:15:45.199 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:15:45.204+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:15:45.205938+0000) 2024-03-04T17:15:46.200 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:15:46.204+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:15:46.206162+0000) 2024-03-04T17:15:47.200 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:15:47.204+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:15:47.206424+0000) 2024-03-04T17:15:48.200 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:15:48.205+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:15:48.206670+0000) 2024-03-04T17:15:49.201 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:15:49.205+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:15:49.206975+0000) 2024-03-04T17:15:50.201 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:15:50.206+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:15:50.207219+0000) 2024-03-04T17:15:51.201 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:15:51.205+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:15:51.207418+0000) 2024-03-04T17:15:52.201 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:15:52.206+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:15:52.207675+0000) 2024-03-04T17:15:53.202 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:15:53.206+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:15:53.207970+0000) 2024-03-04T17:15:54.202 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:15:54.207+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:15:54.208154+0000) 2024-03-04T17:15:55.202 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:15:55.207+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:15:55.208403+0000) 2024-03-04T17:15:56.202 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:15:56.207+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:15:56.208631+0000) 2024-03-04T17:15:57.202 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:15:57.208+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:15:57.208923+0000) 2024-03-04T17:15:58.203 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:15:58.208+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:15:58.209129+0000) 2024-03-04T17:15:59.203 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:15:59.208+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:15:59.209368+0000) 2024-03-04T17:16:00.203 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:16:00.208+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:16:00.209617+0000) 2024-03-04T17:16:01.204 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:16:01.209+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:16:01.209927+0000) 2024-03-04T17:16:02.204 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:16:02.209+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:16:02.210177+0000) 2024-03-04T17:16:03.204 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:16:03.209+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:16:03.210421+0000) 2024-03-04T17:16:04.192 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:16:04.200 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:16:04.209 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:16:04.209+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:16:04.210643+0000) 2024-03-04T17:16:05.205 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:16:05.210+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:16:05.210972+0000) 2024-03-04T17:16:06.205 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:16:06.210+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:16:06.211237+0000) 2024-03-04T17:16:07.205 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:16:07.210+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:16:07.211483+0000) 2024-03-04T17:16:08.205 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:16:08.211+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:16:08.211720+0000) 2024-03-04T17:16:09.206 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:16:09.211+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:16:09.211959+0000) 2024-03-04T17:16:10.206 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:16:10.211+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:16:10.212206+0000) 2024-03-04T17:16:11.206 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:16:11.211+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:16:11.212448+0000) 2024-03-04T17:16:12.207 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:16:12.212+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:16:12.212682+0000) 2024-03-04T17:16:13.207 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:16:13.212+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:16:13.212975+0000) 2024-03-04T17:16:14.207 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:16:14.212+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:16:14.213208+0000) 2024-03-04T17:16:15.207 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:16:15.212+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:16:15.213449+0000) 2024-03-04T17:16:16.208 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:16:16.212+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:16:16.213690+0000) 2024-03-04T17:16:17.208 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:16:17.213+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:16:17.213931+0000) 2024-03-04T17:16:18.208 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:16:18.213+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:16:18.214178+0000) 2024-03-04T17:16:19.209 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:16:19.213+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:16:19.214451+0000) 2024-03-04T17:16:20.209 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:16:20.214+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:16:20.214746+0000) 2024-03-04T17:16:21.209 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:16:21.214+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:16:21.215054+0000) 2024-03-04T17:16:22.209 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:16:22.214+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:16:22.215334+0000) 2024-03-04T17:16:23.210 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:16:23.214+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:16:23.215569+0000) 2024-03-04T17:16:24.210 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:16:24.215+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:16:24.215905+0000) 2024-03-04T17:16:25.210 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:16:25.215+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:16:25.216214+0000) 2024-03-04T17:16:26.210 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:16:26.216+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:16:26.216496+0000) 2024-03-04T17:16:27.211 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:16:27.216+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:16:27.216801+0000) 2024-03-04T17:16:28.211 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:16:28.216+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:16:28.217149+0000) 2024-03-04T17:16:29.211 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:16:29.217+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:16:29.217438+0000) 2024-03-04T17:16:30.212 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:16:30.217+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:16:30.217716+0000) 2024-03-04T17:16:31.212 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:16:31.217+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:16:31.218010+0000) 2024-03-04T17:16:32.212 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:16:32.217+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:16:32.218292+0000) 2024-03-04T17:16:33.213 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:16:33.217+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:16:33.218600+0000) 2024-03-04T17:16:34.213 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:16:34.218+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:16:34.218938+0000) 2024-03-04T17:16:34.329 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:16:34.338 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:16:35.214 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:16:35.218+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:16:35.219241+0000) 2024-03-04T17:16:36.214 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:16:36.219+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:16:36.219519+0000) 2024-03-04T17:16:37.214 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:16:37.219+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:16:37.219929+0000) 2024-03-04T17:16:38.214 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:16:38.219+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:16:38.220208+0000) 2024-03-04T17:16:39.215 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:16:39.220+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:16:39.220500+0000) 2024-03-04T17:16:40.215 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:16:40.220+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:16:40.220793+0000) 2024-03-04T17:16:41.215 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:16:41.220+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:16:41.221101+0000) 2024-03-04T17:16:42.215 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:16:42.221+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:16:42.221406+0000) 2024-03-04T17:16:43.216 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:16:43.221+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:16:43.221706+0000) 2024-03-04T17:16:44.216 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:16:44.221+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:16:44.221998+0000) 2024-03-04T17:16:45.216 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:16:45.221+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:16:45.222246+0000) 2024-03-04T17:16:46.217 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:16:46.222+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:16:46.222523+0000) 2024-03-04T17:16:47.217 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:16:47.221+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:16:47.222873+0000) 2024-03-04T17:16:48.217 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:16:48.221+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:16:48.223161+0000) 2024-03-04T17:16:49.218 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:16:49.221+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:16:49.223392+0000) 2024-03-04T17:16:50.218 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:16:50.222+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:16:50.223688+0000) 2024-03-04T17:16:51.218 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:16:51.222+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:16:51.223989+0000) 2024-03-04T17:16:52.219 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:16:52.223+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:16:52.224265+0000) 2024-03-04T17:16:53.219 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:16:53.223+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:16:53.224574+0000) 2024-03-04T17:16:54.219 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:16:54.223+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:16:54.224913+0000) 2024-03-04T17:16:55.219 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:16:55.224+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:16:55.225216+0000) 2024-03-04T17:16:56.220 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:16:56.224+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:16:56.225485+0000) 2024-03-04T17:16:57.220 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:16:57.224+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:16:57.225719+0000) 2024-03-04T17:16:58.220 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:16:58.225+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:16:58.226015+0000) 2024-03-04T17:16:59.220 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:16:59.225+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:16:59.226266+0000) 2024-03-04T17:17:00.221 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:17:00.225+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:17:00.226559+0000) 2024-03-04T17:17:01.221 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:17:01.226+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:17:01.226908+0000) 2024-03-04T17:17:02.222 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:17:02.226+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:17:02.227188+0000) 2024-03-04T17:17:03.222 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:17:03.226+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:17:03.227436+0000) 2024-03-04T17:17:04.222 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:17:04.226+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:17:04.227723+0000) 2024-03-04T17:17:04.478 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:17:04.486 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:17:05.222 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:17:05.227+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:17:05.228018+0000) 2024-03-04T17:17:06.223 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:17:06.227+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:17:06.228297+0000) 2024-03-04T17:17:07.223 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:17:07.227+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:17:07.228531+0000) 2024-03-04T17:17:08.223 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:17:08.228+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:17:08.228806+0000) 2024-03-04T17:17:09.224 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:17:09.228+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:17:09.229149+0000) 2024-03-04T17:17:10.224 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:17:10.228+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:17:10.229444+0000) 2024-03-04T17:17:11.224 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:17:11.228+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:17:11.229749+0000) 2024-03-04T17:17:12.225 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:17:12.229+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:17:12.230031+0000) 2024-03-04T17:17:13.225 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:17:13.229+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:17:13.230320+0000) 2024-03-04T17:17:14.225 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:17:14.229+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:17:14.230625+0000) 2024-03-04T17:17:15.225 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:17:15.230+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:17:15.230968+0000) 2024-03-04T17:17:16.226 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:17:16.230+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:17:16.231247+0000) 2024-03-04T17:17:17.226 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:17:17.230+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:17:17.231477+0000) 2024-03-04T17:17:18.226 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:17:18.231+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:17:18.231754+0000) 2024-03-04T17:17:19.227 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:17:19.231+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:17:19.232123+0000) 2024-03-04T17:17:20.227 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:17:20.231+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:17:20.232446+0000) 2024-03-04T17:17:21.227 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:17:21.231+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:17:21.232694+0000) 2024-03-04T17:17:22.228 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:17:22.232+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:17:22.232987+0000) 2024-03-04T17:17:23.228 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:17:23.232+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:17:23.233280+0000) 2024-03-04T17:17:24.228 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:17:24.233+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:17:24.233572+0000) 2024-03-04T17:17:25.229 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:17:25.233+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:17:25.233928+0000) 2024-03-04T17:17:26.229 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:17:26.233+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:17:26.234203+0000) 2024-03-04T17:17:27.229 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:17:27.233+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:17:27.234435+0000) 2024-03-04T17:17:28.230 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:17:28.234+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:17:28.234731+0000) 2024-03-04T17:17:29.230 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:17:29.234+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:17:29.235005+0000) 2024-03-04T17:17:30.231 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:17:30.234+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:17:30.235294+0000) 2024-03-04T17:17:31.230 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:17:31.234+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:17:31.235589+0000) 2024-03-04T17:17:32.231 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:17:32.235+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:17:32.235913+0000) 2024-03-04T17:17:33.231 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:17:33.235+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:17:33.236206+0000) 2024-03-04T17:17:34.231 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:17:34.236+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:17:34.236493+0000) 2024-03-04T17:17:34.707 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:17:34.719 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:17:35.231 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:17:35.235+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:17:35.236736+0000) 2024-03-04T17:17:36.232 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:17:36.236+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:17:36.237021+0000) 2024-03-04T17:17:37.232 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:17:37.236+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:17:37.237265+0000) 2024-03-04T17:17:38.232 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:17:38.236+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:17:38.237574+0000) 2024-03-04T17:17:39.233 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:17:39.237+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:17:39.237816+0000) 2024-03-04T17:17:40.233 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:17:40.237+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:17:40.238165+0000) 2024-03-04T17:17:41.233 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:17:41.238+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:17:41.238473+0000) 2024-03-04T17:17:42.234 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:17:42.238+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:17:42.238762+0000) 2024-03-04T17:17:43.234 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:17:43.238+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:17:43.239071+0000) 2024-03-04T17:17:44.234 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:17:44.239+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:17:44.239337+0000) 2024-03-04T17:17:45.234 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:17:45.238+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:17:45.239560+0000) 2024-03-04T17:17:46.235 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:17:46.239+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:17:46.239904+0000) 2024-03-04T17:17:47.235 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:17:47.239+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:17:47.240194+0000) 2024-03-04T17:17:48.235 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:17:48.240+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:17:48.240488+0000) 2024-03-04T17:17:49.236 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:17:49.240+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:17:49.240785+0000) 2024-03-04T17:17:50.236 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:17:50.239+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:17:50.241122+0000) 2024-03-04T17:17:51.237 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:17:51.240+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:17:51.241428+0000) 2024-03-04T17:17:52.237 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:17:52.239+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:17:52.241730+0000) 2024-03-04T17:17:53.237 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:17:53.240+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:17:53.242021+0000) 2024-03-04T17:17:54.237 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:17:54.241+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:17:54.242335+0000) 2024-03-04T17:17:55.238 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:17:55.241+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:17:55.242614+0000) 2024-03-04T17:17:56.238 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:17:56.241+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:17:56.242958+0000) 2024-03-04T17:17:57.238 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:17:57.241+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:17:57.243251+0000) 2024-03-04T17:17:58.239 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:17:58.242+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:17:58.243548+0000) 2024-03-04T17:17:59.239 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:17:59.242+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:17:59.243780+0000) 2024-03-04T17:18:00.239 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:18:00.243+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:18:00.244098+0000) 2024-03-04T17:18:01.240 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:18:01.243+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:18:01.244390+0000) 2024-03-04T17:18:02.240 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:18:02.243+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:18:02.244696+0000) 2024-03-04T17:18:03.240 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:18:03.244+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:18:03.244950+0000) 2024-03-04T17:18:04.240 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:18:04.244+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:18:04.245183+0000) 2024-03-04T17:18:04.930 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:18:04.939 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:18:05.240 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:18:05.244+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:18:05.245431+0000) 2024-03-04T17:18:06.241 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:18:06.244+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:18:06.245680+0000) 2024-03-04T17:18:07.241 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:18:07.245+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:18:07.245987+0000) 2024-03-04T17:18:08.242 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:18:08.245+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:18:08.246287+0000) 2024-03-04T17:18:09.242 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:18:09.245+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:18:09.246574+0000) 2024-03-04T17:18:10.242 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:18:10.246+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:18:10.246924+0000) 2024-03-04T17:18:11.242 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:18:11.246+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:18:11.247142+0000) 2024-03-04T17:18:12.243 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:18:12.246+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:18:12.247377+0000) 2024-03-04T17:18:13.243 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:18:13.246+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:18:13.247654+0000) 2024-03-04T17:18:14.243 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:18:14.247+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:18:14.247983+0000) 2024-03-04T17:18:15.244 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:18:15.247+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:18:15.248288+0000) 2024-03-04T17:18:16.244 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:18:16.247+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:18:16.248569+0000) 2024-03-04T17:18:17.244 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:18:17.248+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:18:17.248913+0000) 2024-03-04T17:18:18.244 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:18:18.248+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:18:18.249193+0000) 2024-03-04T17:18:19.245 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:18:19.248+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:18:19.249469+0000) 2024-03-04T17:18:20.245 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:18:20.248+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:18:20.249784+0000) 2024-03-04T17:18:21.245 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:18:21.249+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:18:21.250102+0000) 2024-03-04T17:18:22.246 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:18:22.249+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:18:22.250415+0000) 2024-03-04T17:18:23.246 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:18:23.250+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:18:23.250708+0000) 2024-03-04T17:18:24.246 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:18:24.250+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:18:24.251014+0000) 2024-03-04T17:18:25.247 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:18:25.250+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:18:25.251291+0000) 2024-03-04T17:18:26.247 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:18:26.250+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:18:26.251469+0000) 2024-03-04T17:18:27.247 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:18:27.250+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:18:27.251732+0000) 2024-03-04T17:18:28.247 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:18:28.251+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:18:28.251973+0000) 2024-03-04T17:18:29.247 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:18:29.251+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:18:29.252212+0000) 2024-03-04T17:18:30.248 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:18:30.251+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:18:30.252457+0000) 2024-03-04T17:18:31.248 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:18:31.252+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:18:31.252740+0000) 2024-03-04T17:18:32.248 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:18:32.252+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:18:32.253038+0000) 2024-03-04T17:18:33.249 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:18:33.252+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:18:33.253339+0000) 2024-03-04T17:18:34.249 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:18:34.252+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:18:34.253576+0000) 2024-03-04T17:18:35.144 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:18:35.153 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:18:35.249 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:18:35.253+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:18:35.253933+0000) 2024-03-04T17:18:36.250 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:18:36.253+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:18:36.254233+0000) 2024-03-04T17:18:37.250 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:18:37.254+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:18:37.254522+0000) 2024-03-04T17:18:38.250 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:18:38.254+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:18:38.254818+0000) 2024-03-04T17:18:39.251 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:18:39.254+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:18:39.255169+0000) 2024-03-04T17:18:40.251 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:18:40.255+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:18:40.255472+0000) 2024-03-04T17:18:41.251 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:18:41.255+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:18:41.255762+0000) 2024-03-04T17:18:42.251 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:18:42.255+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:18:42.256057+0000) 2024-03-04T17:18:43.252 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:18:43.256+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:18:43.256346+0000) 2024-03-04T17:18:44.252 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:18:44.255+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:18:44.256655+0000) 2024-03-04T17:18:45.252 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:18:45.256+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:18:45.256982+0000) 2024-03-04T17:18:46.253 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:18:46.257+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:18:46.257271+0000) 2024-03-04T17:18:47.253 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:18:47.257+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:18:47.257568+0000) 2024-03-04T17:18:48.253 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:18:48.257+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:18:48.257930+0000) 2024-03-04T17:18:49.254 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:18:49.257+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:18:49.258219+0000) 2024-03-04T17:18:50.254 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:18:50.257+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:18:50.258532+0000) 2024-03-04T17:18:51.254 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:18:51.258+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:18:51.258886+0000) 2024-03-04T17:18:52.255 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:18:52.259+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:18:52.259193+0000) 2024-03-04T17:18:53.255 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:18:53.259+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:18:53.259483+0000) 2024-03-04T17:18:54.255 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:18:54.258+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:18:54.259782+0000) 2024-03-04T17:18:55.256 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:18:55.258+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:18:55.260126+0000) 2024-03-04T17:18:56.256 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:18:56.259+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:18:56.260408+0000) 2024-03-04T17:18:57.256 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:18:57.258+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:18:57.260662+0000) 2024-03-04T17:18:58.256 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:18:58.259+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:18:58.260981+0000) 2024-03-04T17:18:59.257 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:18:59.259+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:18:59.261430+0000) 2024-03-04T17:19:00.257 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:19:00.260+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:19:00.261733+0000) 2024-03-04T17:19:01.258 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:19:01.260+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:19:01.262033+0000) 2024-03-04T17:19:02.258 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:19:02.261+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:19:02.262311+0000) 2024-03-04T17:19:03.258 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:19:03.261+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:19:03.262634+0000) 2024-03-04T17:19:04.259 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:19:04.261+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:19:04.262961+0000) 2024-03-04T17:19:05.259 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:19:05.262+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:19:05.263258+0000) 2024-03-04T17:19:05.273 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:19:05.281 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:19:06.259 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:19:06.262+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:19:06.263600+0000) 2024-03-04T17:19:07.260 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:19:07.263+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:19:07.263937+0000) 2024-03-04T17:19:08.260 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:19:08.263+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:19:08.264248+0000) 2024-03-04T17:19:09.260 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:19:09.263+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:19:09.264560+0000) 2024-03-04T17:19:10.261 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:19:10.264+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:19:10.264879+0000) 2024-03-04T17:19:11.261 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:19:11.264+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:19:11.265128+0000) 2024-03-04T17:19:12.261 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:19:12.264+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:19:12.265442+0000) 2024-03-04T17:19:13.261 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:19:13.264+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:19:13.265730+0000) 2024-03-04T17:19:14.262 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:19:14.265+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:19:14.266028+0000) 2024-03-04T17:19:15.262 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:19:15.265+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:19:15.266333+0000) 2024-03-04T17:19:16.262 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:19:16.265+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:19:16.266643+0000) 2024-03-04T17:19:17.263 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:19:17.266+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:19:17.266985+0000) 2024-03-04T17:19:18.263 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:19:18.266+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:19:18.267278+0000) 2024-03-04T17:19:19.263 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:19:19.266+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:19:19.267565+0000) 2024-03-04T17:19:20.264 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:19:20.267+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:19:20.267929+0000) 2024-03-04T17:19:21.264 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:19:21.267+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:19:21.268216+0000) 2024-03-04T17:19:22.264 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:19:22.267+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:19:22.268516+0000) 2024-03-04T17:19:23.265 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:19:23.268+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:19:23.268808+0000) 2024-03-04T17:19:24.265 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:19:24.268+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:19:24.269123+0000) 2024-03-04T17:19:25.265 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:19:25.268+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:19:25.269388+0000) 2024-03-04T17:19:26.265 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:19:26.268+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:19:26.269619+0000) 2024-03-04T17:19:27.266 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:19:27.269+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:19:27.269899+0000) 2024-03-04T17:19:28.266 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:19:28.269+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:19:28.270143+0000) 2024-03-04T17:19:29.266 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:19:29.269+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:19:29.270382+0000) 2024-03-04T17:19:30.266 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:19:30.270+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:19:30.270622+0000) 2024-03-04T17:19:31.267 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:19:31.270+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:19:31.270930+0000) 2024-03-04T17:19:32.267 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:19:32.270+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:19:32.271211+0000) 2024-03-04T17:19:33.267 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:19:33.270+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:19:33.271522+0000) 2024-03-04T17:19:34.268 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:19:34.271+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:19:34.271816+0000) 2024-03-04T17:19:35.268 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:19:35.271+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:19:35.272177+0000) 2024-03-04T17:19:35.405 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:19:35.413 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:19:36.268 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:19:36.271+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:19:36.272468+0000) 2024-03-04T17:19:37.269 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:19:37.271+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:19:37.272792+0000) 2024-03-04T17:19:38.269 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:19:38.272+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:19:38.273376+0000) 2024-03-04T17:19:39.269 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:19:39.273+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:19:39.273628+0000) 2024-03-04T17:19:40.270 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:19:40.273+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:19:40.273913+0000) 2024-03-04T17:19:41.270 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:19:41.273+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:19:41.274156+0000) 2024-03-04T17:19:42.270 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:19:42.274+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:19:42.274393+0000) 2024-03-04T17:19:43.271 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:19:43.274+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:19:43.274643+0000) 2024-03-04T17:19:44.271 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:19:44.274+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:19:44.274912+0000) 2024-03-04T17:19:45.271 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:19:45.274+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:19:45.275177+0000) 2024-03-04T17:19:46.271 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:19:46.275+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:19:46.275434+0000) 2024-03-04T17:19:47.272 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:19:47.275+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:19:47.275687+0000) 2024-03-04T17:19:48.272 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:19:48.275+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:19:48.275931+0000) 2024-03-04T17:19:49.272 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:19:49.275+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:19:49.276164+0000) 2024-03-04T17:19:50.272 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:19:50.276+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:19:50.276424+0000) 2024-03-04T17:19:51.273 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:19:51.276+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:19:51.276675+0000) 2024-03-04T17:19:52.273 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:19:52.276+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:19:52.276958+0000) 2024-03-04T17:19:53.273 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:19:53.277+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:19:53.277245+0000) 2024-03-04T17:19:54.273 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:19:54.277+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:19:54.277525+0000) 2024-03-04T17:19:55.274 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:19:55.277+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:19:55.277816+0000) 2024-03-04T17:19:56.274 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:19:56.278+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:19:56.278183+0000) 2024-03-04T17:19:57.274 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:19:57.277+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:19:57.278479+0000) 2024-03-04T17:19:58.275 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:19:58.277+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:19:58.278786+0000) 2024-03-04T17:19:59.275 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:19:59.278+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:19:59.279172+0000) 2024-03-04T17:20:00.276 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:20:00.277+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:20:00.279481+0000) 2024-03-04T17:20:01.276 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:20:01.278+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:20:01.279775+0000) 2024-03-04T17:20:02.276 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:20:02.279+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:20:02.280080+0000) 2024-03-04T17:20:03.276 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:20:03.279+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:20:03.280347+0000) 2024-03-04T17:20:04.277 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:20:04.279+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:20:04.280575+0000) 2024-03-04T17:20:05.277 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:20:05.279+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:20:05.280909+0000) 2024-03-04T17:20:05.609 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:20:05.617 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:20:06.277 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:20:06.280+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:20:06.281208+0000) 2024-03-04T17:20:07.278 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:20:07.279+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:20:07.281446+0000) 2024-03-04T17:20:08.278 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:20:08.280+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:20:08.281736+0000) 2024-03-04T17:20:09.278 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:20:09.281+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:20:09.282015+0000) 2024-03-04T17:20:10.278 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:20:10.281+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:20:10.282266+0000) 2024-03-04T17:20:11.279 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:20:11.281+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:20:11.282576+0000) 2024-03-04T17:20:12.279 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:20:12.282+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:20:12.282921+0000) 2024-03-04T17:20:13.279 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:20:13.282+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:20:13.283211+0000) 2024-03-04T17:20:14.280 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:20:14.282+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:20:14.283484+0000) 2024-03-04T17:20:15.280 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:20:15.282+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:20:15.283749+0000) 2024-03-04T17:20:16.280 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:20:16.283+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:20:16.283988+0000) 2024-03-04T17:20:17.280 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:20:17.283+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:20:17.284233+0000) 2024-03-04T17:20:18.281 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:20:18.283+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:20:18.284477+0000) 2024-03-04T17:20:19.281 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:20:19.283+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:20:19.284740+0000) 2024-03-04T17:20:20.281 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:20:20.284+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:20:20.284985+0000) 2024-03-04T17:20:21.281 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:20:21.284+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:20:21.285235+0000) 2024-03-04T17:20:22.282 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:20:22.284+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:20:22.285476+0000) 2024-03-04T17:20:23.282 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:20:23.284+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:20:23.285722+0000) 2024-03-04T17:20:24.282 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:20:24.285+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:20:24.285950+0000) 2024-03-04T17:20:25.283 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:20:25.285+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:20:25.286198+0000) 2024-03-04T17:20:26.283 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:20:26.285+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:20:26.286456+0000) 2024-03-04T17:20:27.283 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:20:27.286+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:20:27.286703+0000) 2024-03-04T17:20:28.283 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:20:28.286+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:20:28.286956+0000) 2024-03-04T17:20:29.283 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:20:29.286+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:20:29.287236+0000) 2024-03-04T17:20:30.284 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:20:30.286+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:20:30.287468+0000) 2024-03-04T17:20:31.284 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:20:31.287+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:20:31.287720+0000) 2024-03-04T17:20:32.285 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:20:32.287+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:20:32.287960+0000) 2024-03-04T17:20:33.285 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:20:33.287+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:20:33.288203+0000) 2024-03-04T17:20:34.285 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:20:34.287+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:20:34.288435+0000) 2024-03-04T17:20:35.285 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:20:35.287+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:20:35.288704+0000) 2024-03-04T17:20:35.806 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:20:35.815 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:20:36.285 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:20:36.288+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:20:36.288934+0000) 2024-03-04T17:20:37.286 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:20:37.288+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:20:37.289183+0000) 2024-03-04T17:20:38.286 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:20:38.288+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:20:38.289428+0000) 2024-03-04T17:20:39.286 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:20:39.289+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:20:39.289708+0000) 2024-03-04T17:20:40.286 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:20:40.289+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:20:40.290000+0000) 2024-03-04T17:20:41.287 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:20:41.289+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:20:41.290282+0000) 2024-03-04T17:20:42.287 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:20:42.290+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:20:42.290579+0000) 2024-03-04T17:20:43.287 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:20:43.290+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:20:43.290927+0000) 2024-03-04T17:20:44.288 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:20:44.290+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:20:44.291143+0000) 2024-03-04T17:20:45.288 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:20:45.291+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:20:45.291432+0000) 2024-03-04T17:20:46.288 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:20:46.291+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:20:46.291727+0000) 2024-03-04T17:20:47.288 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:20:47.291+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:20:47.292100+0000) 2024-03-04T17:20:48.289 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:20:48.291+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:20:48.292388+0000) 2024-03-04T17:20:49.290 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:20:49.292+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:20:49.292674+0000) 2024-03-04T17:20:50.289 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:20:50.292+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:20:50.292974+0000) 2024-03-04T17:20:51.290 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:20:51.292+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:20:51.293217+0000) 2024-03-04T17:20:52.290 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:20:52.293+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:20:52.293500+0000) 2024-03-04T17:20:53.290 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:20:53.293+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:20:53.293815+0000) 2024-03-04T17:20:54.291 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:20:54.293+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:20:54.294124+0000) 2024-03-04T17:20:55.291 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:20:55.293+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:20:55.294414+0000) 2024-03-04T17:20:56.291 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:20:56.294+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:20:56.294727+0000) 2024-03-04T17:20:57.292 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:20:57.294+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:20:57.295027+0000) 2024-03-04T17:20:58.292 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:20:58.294+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:20:58.295276+0000) 2024-03-04T17:20:59.292 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:20:59.295+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:20:59.295559+0000) 2024-03-04T17:21:00.292 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:21:00.295+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:21:00.295885+0000) 2024-03-04T17:21:01.293 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:21:01.295+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:21:01.296174+0000) 2024-03-04T17:21:02.293 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:21:02.295+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:21:02.296461+0000) 2024-03-04T17:21:03.293 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:21:03.294+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:21:03.296769+0000) 2024-03-04T17:21:04.294 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:21:04.295+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:21:04.297058+0000) 2024-03-04T17:21:05.294 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:21:05.296+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:21:05.297346+0000) 2024-03-04T17:21:06.010 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:21:06.019 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:21:06.294 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:21:06.296+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:21:06.297648+0000) 2024-03-04T17:21:07.295 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:21:07.296+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:21:07.297978+0000) 2024-03-04T17:21:08.295 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:21:08.297+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:21:08.298275+0000) 2024-03-04T17:21:09.295 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:21:09.297+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:21:09.298580+0000) 2024-03-04T17:21:10.295 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:21:10.297+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:21:10.298915+0000) 2024-03-04T17:21:11.296 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:21:11.298+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:21:11.299243+0000) 2024-03-04T17:21:12.296 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:21:12.298+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:21:12.299479+0000) 2024-03-04T17:21:13.296 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:21:13.298+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:21:13.299768+0000) 2024-03-04T17:21:14.297 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:21:14.299+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:21:14.300064+0000) 2024-03-04T17:21:15.297 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:21:15.299+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:21:15.300323+0000) 2024-03-04T17:21:16.299 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:21:16.299+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:21:16.300575+0000) 2024-03-04T17:21:17.297 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:21:17.299+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:21:17.300803+0000) 2024-03-04T17:21:18.298 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:21:18.300+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:21:18.301084+0000) 2024-03-04T17:21:19.298 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:21:19.300+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:21:19.301333+0000) 2024-03-04T17:21:20.298 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:21:20.300+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:21:20.301586+0000) 2024-03-04T17:21:21.299 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:21:21.301+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:21:21.301889+0000) 2024-03-04T17:21:22.299 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:21:22.301+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:21:22.302204+0000) 2024-03-04T17:21:23.299 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:21:23.301+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:21:23.302487+0000) 2024-03-04T17:21:24.300 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:21:24.301+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:21:24.302728+0000) 2024-03-04T17:21:25.300 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:21:25.302+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:21:25.302974+0000) 2024-03-04T17:21:26.300 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:21:26.302+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:21:26.303173+0000) 2024-03-04T17:21:27.300 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:21:27.302+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:21:27.303365+0000) 2024-03-04T17:21:28.300 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:21:28.302+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:21:28.303650+0000) 2024-03-04T17:21:29.301 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:21:29.303+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:21:29.303934+0000) 2024-03-04T17:21:30.301 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:21:30.303+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:21:30.304236+0000) 2024-03-04T17:21:31.301 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:21:31.303+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:21:31.304496+0000) 2024-03-04T17:21:32.302 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:21:32.304+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:21:32.304783+0000) 2024-03-04T17:21:33.302 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:21:33.304+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:21:33.304997+0000) 2024-03-04T17:21:34.302 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:21:34.304+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:21:34.305241+0000) 2024-03-04T17:21:35.302 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:21:35.304+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:21:35.305491+0000) 2024-03-04T17:21:36.174 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:21:36.183 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:21:36.303 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:21:36.305+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:21:36.305734+0000) 2024-03-04T17:21:37.303 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:21:37.305+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:21:37.305984+0000) 2024-03-04T17:21:38.303 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:21:38.305+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:21:38.306236+0000) 2024-03-04T17:21:39.303 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:21:39.305+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:21:39.306479+0000) 2024-03-04T17:21:40.304 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:21:40.306+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:21:40.306723+0000) 2024-03-04T17:21:41.304 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:21:41.306+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:21:41.306963+0000) 2024-03-04T17:21:42.304 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:21:42.306+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:21:42.307201+0000) 2024-03-04T17:21:43.304 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:21:43.306+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:21:43.307387+0000) 2024-03-04T17:21:44.304 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:21:44.306+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:21:44.307619+0000) 2024-03-04T17:21:45.305 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:21:45.307+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:21:45.307914+0000) 2024-03-04T17:21:46.305 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:21:46.307+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:21:46.308147+0000) 2024-03-04T17:21:47.305 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:21:47.307+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:21:47.308334+0000) 2024-03-04T17:21:48.305 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:21:48.308+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:21:48.308588+0000) 2024-03-04T17:21:49.306 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:21:49.307+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:21:49.308783+0000) 2024-03-04T17:21:50.306 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:21:50.308+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:21:50.309044+0000) 2024-03-04T17:21:51.306 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:21:51.308+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:21:51.309245+0000) 2024-03-04T17:21:52.306 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:21:52.309+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:21:52.309526+0000) 2024-03-04T17:21:53.307 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:21:53.309+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:21:53.309873+0000) 2024-03-04T17:21:54.307 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:21:54.309+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:21:54.310120+0000) 2024-03-04T17:21:55.307 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:21:55.310+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:21:55.310432+0000) 2024-03-04T17:21:56.308 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:21:56.310+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:21:56.310748+0000) 2024-03-04T17:21:57.308 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:21:57.310+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:21:57.311041+0000) 2024-03-04T17:21:58.308 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:21:58.311+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:21:58.311337+0000) 2024-03-04T17:21:59.309 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:21:59.311+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:21:59.311624+0000) 2024-03-04T17:22:00.309 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:22:00.311+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:22:00.311951+0000) 2024-03-04T17:22:01.309 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:22:01.312+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:22:01.312245+0000) 2024-03-04T17:22:02.310 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:22:02.312+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:22:02.312541+0000) 2024-03-04T17:22:03.310 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:22:03.312+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:22:03.312872+0000) 2024-03-04T17:22:04.310 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:22:04.312+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:22:04.313162+0000) 2024-03-04T17:22:05.311 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:22:05.312+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:22:05.313452+0000) 2024-03-04T17:22:06.311 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:22:06.312+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:22:06.313749+0000) 2024-03-04T17:22:06.312 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:22:06.324 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:22:07.311 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:22:07.312+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:22:07.314039+0000) 2024-03-04T17:22:08.311 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:22:08.313+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:22:08.314333+0000) 2024-03-04T17:22:09.312 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:22:09.312+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:22:09.314567+0000) 2024-03-04T17:22:10.312 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:22:10.313+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:22:10.314915+0000) 2024-03-04T17:22:11.312 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:22:11.313+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:22:11.315160+0000) 2024-03-04T17:22:12.312 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:22:12.314+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:22:12.315468+0000) 2024-03-04T17:22:13.313 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:22:13.313+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:22:13.315777+0000) 2024-03-04T17:22:14.313 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:22:14.314+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:22:14.316026+0000) 2024-03-04T17:22:15.314 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:22:15.315+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:22:15.316309+0000) 2024-03-04T17:22:16.314 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:22:16.315+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:22:16.316602+0000) 2024-03-04T17:22:17.314 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:22:17.316+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:22:17.316921+0000) 2024-03-04T17:22:18.314 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:22:18.316+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:22:18.317208+0000) 2024-03-04T17:22:19.315 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:22:19.316+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:22:19.317439+0000) 2024-03-04T17:22:20.315 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:22:20.316+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:22:20.317743+0000) 2024-03-04T17:22:21.315 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:22:21.317+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:22:21.317984+0000) 2024-03-04T17:22:22.315 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:22:22.317+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:22:22.318288+0000) 2024-03-04T17:22:23.316 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:22:23.317+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:22:23.318632+0000) 2024-03-04T17:22:24.316 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:22:24.318+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:22:24.318924+0000) 2024-03-04T17:22:25.316 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:22:25.318+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:22:25.319160+0000) 2024-03-04T17:22:26.317 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:22:26.318+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:22:26.319422+0000) 2024-03-04T17:22:27.317 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:22:27.318+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:22:27.319612+0000) 2024-03-04T17:22:28.317 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:22:28.319+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:22:28.319892+0000) 2024-03-04T17:22:29.317 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:22:29.319+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:22:29.320081+0000) 2024-03-04T17:22:30.317 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:22:30.319+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:22:30.320341+0000) 2024-03-04T17:22:31.318 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:22:31.319+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:22:31.320618+0000) 2024-03-04T17:22:32.318 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:22:32.320+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:22:32.320902+0000) 2024-03-04T17:22:33.318 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:22:33.320+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:22:33.321138+0000) 2024-03-04T17:22:34.319 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:22:34.320+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:22:34.321373+0000) 2024-03-04T17:22:35.319 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:22:35.320+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:22:35.321631+0000) 2024-03-04T17:22:36.319 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:22:36.321+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:22:36.321916+0000) 2024-03-04T17:22:36.453 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:22:36.464 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:22:37.319 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:22:37.321+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:22:37.322126+0000) 2024-03-04T17:22:38.320 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:22:38.321+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:22:38.322364+0000) 2024-03-04T17:22:39.320 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:22:39.322+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:22:39.322596+0000) 2024-03-04T17:22:40.320 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:22:40.322+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:22:40.322896+0000) 2024-03-04T17:22:41.320 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:22:41.322+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:22:41.323099+0000) 2024-03-04T17:22:42.321 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:22:42.322+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:22:42.323344+0000) 2024-03-04T17:22:43.321 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:22:43.323+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:22:43.323582+0000) 2024-03-04T17:22:44.321 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:22:44.323+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:22:44.323847+0000) 2024-03-04T17:22:45.321 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:22:45.323+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:22:45.324095+0000) 2024-03-04T17:22:46.322 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:22:46.323+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:22:46.324335+0000) 2024-03-04T17:22:47.322 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:22:47.324+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:22:47.324566+0000) 2024-03-04T17:22:48.322 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:22:48.324+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:22:48.324804+0000) 2024-03-04T17:22:49.322 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:22:49.324+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:22:49.325101+0000) 2024-03-04T17:22:50.323 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:22:50.324+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:22:50.325368+0000) 2024-03-04T17:22:51.323 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:22:51.325+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:22:51.325620+0000) 2024-03-04T17:22:52.323 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:22:52.325+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:22:52.325918+0000) 2024-03-04T17:22:53.323 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:22:53.325+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:22:53.326160+0000) 2024-03-04T17:22:54.324 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:22:54.326+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:22:54.326394+0000) 2024-03-04T17:22:55.324 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:22:55.325+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:22:55.326651+0000) 2024-03-04T17:22:56.324 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:22:56.326+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:22:56.326935+0000) 2024-03-04T17:22:57.325 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:22:57.326+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:22:57.327200+0000) 2024-03-04T17:22:58.325 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:22:58.327+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:22:58.327444+0000) 2024-03-04T17:22:59.325 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:22:59.326+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:22:59.327707+0000) 2024-03-04T17:23:00.325 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:23:00.327+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:23:00.327976+0000) 2024-03-04T17:23:01.326 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:23:01.327+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:23:01.328219+0000) 2024-03-04T17:23:02.326 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:23:02.328+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:23:02.328463+0000) 2024-03-04T17:23:03.326 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:23:03.328+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:23:03.328697+0000) 2024-03-04T17:23:04.326 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:23:04.328+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:23:04.328929+0000) 2024-03-04T17:23:05.327 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:23:05.329+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:23:05.329173+0000) 2024-03-04T17:23:06.327 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:23:06.328+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:23:06.329437+0000) 2024-03-04T17:23:06.666 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:23:06.679 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:23:07.327 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:23:07.329+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:23:07.329681+0000) 2024-03-04T17:23:08.327 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:23:08.328+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:23:08.329927+0000) 2024-03-04T17:23:09.328 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:23:09.328+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:23:09.330124+0000) 2024-03-04T17:23:10.328 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:23:10.329+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:23:10.330405+0000) 2024-03-04T17:23:11.328 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:23:11.329+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:23:11.330657+0000) 2024-03-04T17:23:12.328 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:23:12.329+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:23:12.330948+0000) 2024-03-04T17:23:13.329 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:23:13.330+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:23:13.331186+0000) 2024-03-04T17:23:14.329 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:23:14.329+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:23:14.331433+0000) 2024-03-04T17:23:15.329 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:23:15.330+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:23:15.331681+0000) 2024-03-04T17:23:16.329 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:23:16.330+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:23:16.331961+0000) 2024-03-04T17:23:17.330 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:23:17.331+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:23:17.332251+0000) 2024-03-04T17:23:18.330 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:23:18.331+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:23:18.332509+0000) 2024-03-04T17:23:19.330 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:23:19.331+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:23:19.332752+0000) 2024-03-04T17:23:20.331 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:23:20.332+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:23:20.333008+0000) 2024-03-04T17:23:21.331 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:23:21.332+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:23:21.333253+0000) 2024-03-04T17:23:22.331 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:23:22.332+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:23:22.333505+0000) 2024-03-04T17:23:23.331 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:23:23.332+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:23:23.333744+0000) 2024-03-04T17:23:24.332 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:23:24.333+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:23:24.333975+0000) 2024-03-04T17:23:25.332 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:23:25.333+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:23:25.334219+0000) 2024-03-04T17:23:26.332 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:23:26.333+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:23:26.334466+0000) 2024-03-04T17:23:27.332 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:23:27.333+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:23:27.334712+0000) 2024-03-04T17:23:28.333 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:23:28.334+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:23:28.334942+0000) 2024-03-04T17:23:29.333 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:23:29.334+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:23:29.335181+0000) 2024-03-04T17:23:30.333 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:23:30.334+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:23:30.335429+0000) 2024-03-04T17:23:31.335 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:23:31.334+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:23:31.335686+0000) 2024-03-04T17:23:32.334 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:23:32.335+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:23:32.335935+0000) 2024-03-04T17:23:33.334 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:23:33.335+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:23:33.336173+0000) 2024-03-04T17:23:34.334 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:23:34.335+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:23:34.336408+0000) 2024-03-04T17:23:35.335 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:23:35.335+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:23:35.336686+0000) 2024-03-04T17:23:36.335 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:23:36.336+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:23:36.336935+0000) 2024-03-04T17:23:36.872 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:23:36.880 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:23:37.335 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:23:37.336+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:23:37.337173+0000) 2024-03-04T17:23:38.335 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:23:38.336+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:23:38.337441+0000) 2024-03-04T17:23:39.335 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:23:39.337+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:23:39.337675+0000) 2024-03-04T17:23:40.336 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:23:40.337+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:23:40.337991+0000) 2024-03-04T17:23:41.336 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:23:41.337+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:23:41.338235+0000) 2024-03-04T17:23:42.336 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:23:42.337+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:23:42.338441+0000) 2024-03-04T17:23:43.336 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:23:43.337+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:23:43.338668+0000) 2024-03-04T17:23:44.337 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:23:44.338+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:23:44.338942+0000) 2024-03-04T17:23:45.337 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:23:45.338+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:23:45.339221+0000) 2024-03-04T17:23:46.337 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:23:46.338+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:23:46.339468+0000) 2024-03-04T17:23:47.337 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:23:47.339+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:23:47.339709+0000) 2024-03-04T17:23:48.338 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:23:48.339+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:23:48.339954+0000) 2024-03-04T17:23:49.338 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:23:49.339+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:23:49.340188+0000) 2024-03-04T17:23:50.338 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:23:50.339+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:23:50.340381+0000) 2024-03-04T17:23:51.338 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:23:51.339+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:23:51.340573+0000) 2024-03-04T17:23:52.339 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:23:52.340+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:23:52.340848+0000) 2024-03-04T17:23:53.339 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:23:53.340+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:23:53.341090+0000) 2024-03-04T17:23:54.340 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:23:54.340+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:23:54.341325+0000) 2024-03-04T17:23:55.339 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:23:55.341+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:23:55.341572+0000) 2024-03-04T17:23:56.340 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:23:56.341+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:23:56.341817+0000) 2024-03-04T17:23:57.340 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:23:57.341+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:23:57.342087+0000) 2024-03-04T17:23:58.340 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:23:58.341+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:23:58.342323+0000) 2024-03-04T17:23:59.340 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:23:59.342+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:23:59.342559+0000) 2024-03-04T17:24:00.341 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:24:00.342+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:24:00.342797+0000) 2024-03-04T17:24:01.341 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:24:01.342+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:24:01.343079+0000) 2024-03-04T17:24:02.341 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:24:02.342+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:24:02.343320+0000) 2024-03-04T17:24:03.341 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:24:03.343+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:24:03.343558+0000) 2024-03-04T17:24:04.342 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:24:04.343+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:24:04.343794+0000) 2024-03-04T17:24:05.342 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:24:05.343+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:24:05.344068+0000) 2024-03-04T17:24:06.342 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:24:06.344+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:24:06.344310+0000) 2024-03-04T17:24:07.018 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:24:07.029 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:24:07.342 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:24:07.344+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:24:07.344555+0000) 2024-03-04T17:24:08.343 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:24:08.344+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:24:08.344800+0000) 2024-03-04T17:24:09.343 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:24:09.344+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:24:09.345080+0000) 2024-03-04T17:24:10.343 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:24:10.344+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:24:10.345374+0000) 2024-03-04T17:24:11.343 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:24:11.344+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:24:11.345568+0000) 2024-03-04T17:24:12.344 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:24:12.344+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:24:12.345815+0000) 2024-03-04T17:24:13.344 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:24:13.344+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:24:13.346052+0000) 2024-03-04T17:24:14.344 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:24:14.344+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:24:14.346278+0000) 2024-03-04T17:24:15.344 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:24:15.345+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:24:15.346513+0000) 2024-03-04T17:24:16.345 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:24:16.345+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:24:16.346747+0000) 2024-03-04T17:24:17.345 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:24:17.345+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:24:17.346987+0000) 2024-03-04T17:24:18.345 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:24:18.346+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:24:18.347231+0000) 2024-03-04T17:24:19.345 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:24:19.346+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:24:19.347490+0000) 2024-03-04T17:24:20.346 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:24:20.346+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:24:20.347765+0000) 2024-03-04T17:24:21.346 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:24:21.347+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:24:21.348024+0000) 2024-03-04T17:24:22.346 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:24:22.347+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:24:22.348271+0000) 2024-03-04T17:24:23.346 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:24:23.347+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:24:23.348506+0000) 2024-03-04T17:24:24.347 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:24:24.347+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:24:24.348738+0000) 2024-03-04T17:24:25.347 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:24:25.348+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:24:25.348971+0000) 2024-03-04T17:24:26.347 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:24:26.348+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:24:26.349218+0000) 2024-03-04T17:24:27.347 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:24:27.348+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:24:27.349454+0000) 2024-03-04T17:24:28.348 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:24:28.348+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:24:28.349695+0000) 2024-03-04T17:24:29.348 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:24:29.349+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:24:29.349933+0000) 2024-03-04T17:24:30.348 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:24:30.349+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:24:30.350173+0000) 2024-03-04T17:24:31.348 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:24:31.349+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:24:31.350410+0000) 2024-03-04T17:24:32.349 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:24:32.349+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:24:32.350659+0000) 2024-03-04T17:24:33.349 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:24:33.350+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:24:33.350946+0000) 2024-03-04T17:24:34.349 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:24:34.350+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:24:34.351203+0000) 2024-03-04T17:24:35.349 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:24:35.350+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:24:35.351441+0000) 2024-03-04T17:24:36.350 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:24:36.350+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:24:36.351701+0000) 2024-03-04T17:24:37.223 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:24:37.232 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:24:37.366 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:24:37.351+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:24:37.352025+0000) 2024-03-04T17:24:38.350 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:24:38.351+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:24:38.352299+0000) 2024-03-04T17:24:39.351 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:24:39.351+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:24:39.352534+0000) 2024-03-04T17:24:40.351 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:24:40.352+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:24:40.352814+0000) 2024-03-04T17:24:41.351 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:24:41.352+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:24:41.353098+0000) 2024-03-04T17:24:42.352 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:24:42.352+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:24:42.353347+0000) 2024-03-04T17:24:43.352 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:24:43.352+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:24:43.353588+0000) 2024-03-04T17:24:44.352 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:24:44.353+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:24:44.353856+0000) 2024-03-04T17:24:45.353 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:24:45.353+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:24:45.354091+0000) 2024-03-04T17:24:46.352 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:24:46.353+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:24:46.354328+0000) 2024-03-04T17:24:47.353 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:24:47.354+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:24:47.354569+0000) 2024-03-04T17:24:48.353 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:24:48.354+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:24:48.354808+0000) 2024-03-04T17:24:49.353 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:24:49.354+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:24:49.355097+0000) 2024-03-04T17:24:50.353 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:24:50.354+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:24:50.355288+0000) 2024-03-04T17:24:51.354 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:24:51.355+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:24:51.355549+0000) 2024-03-04T17:24:52.354 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:24:52.355+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:24:52.355795+0000) 2024-03-04T17:24:53.354 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:24:53.355+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:24:53.356112+0000) 2024-03-04T17:24:54.354 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:24:54.355+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:24:54.356345+0000) 2024-03-04T17:24:55.355 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:24:55.355+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:24:55.356633+0000) 2024-03-04T17:24:56.355 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:24:56.356+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:24:56.356922+0000) 2024-03-04T17:24:57.396 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:24:57.356+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:24:57.357156+0000) 2024-03-04T17:24:58.355 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:24:58.357+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:24:58.357406+0000) 2024-03-04T17:24:59.356 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:24:59.357+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:24:59.357645+0000) 2024-03-04T17:25:00.356 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:25:00.357+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:25:00.357920+0000) 2024-03-04T17:25:01.356 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:25:01.357+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:25:01.358153+0000) 2024-03-04T17:25:02.356 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:25:02.357+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:25:02.358392+0000) 2024-03-04T17:25:03.357 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:25:03.357+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:25:03.358630+0000) 2024-03-04T17:25:04.357 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:25:04.358+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:25:04.358916+0000) 2024-03-04T17:25:05.357 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:25:05.358+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:25:05.359154+0000) 2024-03-04T17:25:06.358 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:25:06.359+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:25:06.359393+0000) 2024-03-04T17:25:07.358 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:25:07.359+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:25:07.359630+0000) 2024-03-04T17:25:07.428 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:25:07.437 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:25:08.358 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:25:08.359+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:25:08.359908+0000) 2024-03-04T17:25:09.358 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:25:09.359+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:25:09.360145+0000) 2024-03-04T17:25:10.358 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:25:10.359+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:25:10.360344+0000) 2024-03-04T17:25:11.359 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:25:11.360+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:25:11.360580+0000) 2024-03-04T17:25:12.359 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:25:12.360+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:25:12.360856+0000) 2024-03-04T17:25:13.359 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:25:13.360+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:25:13.361097+0000) 2024-03-04T17:25:14.360 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:25:14.361+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:25:14.361373+0000) 2024-03-04T17:25:15.360 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:25:15.360+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:25:15.361611+0000) 2024-03-04T17:25:16.360 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:25:16.360+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:25:16.361885+0000) 2024-03-04T17:25:17.360 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:25:17.360+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:25:17.362113+0000) 2024-03-04T17:25:18.361 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:25:18.360+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:25:18.362342+0000) 2024-03-04T17:25:19.361 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:25:19.361+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:25:19.362576+0000) 2024-03-04T17:25:20.361 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:25:20.361+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:25:20.362873+0000) 2024-03-04T17:25:21.361 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:25:21.362+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:25:21.363129+0000) 2024-03-04T17:25:22.362 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:25:22.362+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:25:22.363375+0000) 2024-03-04T17:25:23.362 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:25:23.362+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:25:23.363618+0000) 2024-03-04T17:25:24.362 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:25:24.362+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:25:24.363903+0000) 2024-03-04T17:25:25.362 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:25:25.363+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:25:25.364143+0000) 2024-03-04T17:25:26.363 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:25:26.363+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:25:26.364388+0000) 2024-03-04T17:25:27.363 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:25:27.363+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:25:27.364624+0000) 2024-03-04T17:25:28.363 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:25:28.364+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:25:28.364908+0000) 2024-03-04T17:25:29.363 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:25:29.364+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:25:29.365160+0000) 2024-03-04T17:25:30.364 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:25:30.364+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:25:30.365406+0000) 2024-03-04T17:25:31.364 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:25:31.364+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:25:31.365645+0000) 2024-03-04T17:25:32.364 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:25:32.365+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:25:32.365943+0000) 2024-03-04T17:25:33.364 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:25:33.365+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:25:33.366158+0000) 2024-03-04T17:25:34.365 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:25:34.365+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:25:34.366427+0000) 2024-03-04T17:25:35.365 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:25:35.365+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:25:35.366664+0000) 2024-03-04T17:25:36.365 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:25:36.366+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:25:36.366967+0000) 2024-03-04T17:25:37.366 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:25:37.366+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:25:37.367212+0000) 2024-03-04T17:25:37.576 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:25:37.585 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:25:38.366 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:25:38.366+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:25:38.367406+0000) 2024-03-04T17:25:39.366 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:25:39.366+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:25:39.367638+0000) 2024-03-04T17:25:40.366 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:25:40.367+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:25:40.367929+0000) 2024-03-04T17:25:41.367 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:25:41.367+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:25:41.368215+0000) 2024-03-04T17:25:42.367 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:25:42.367+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:25:42.368462+0000) 2024-03-04T17:25:43.367 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:25:43.368+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:25:43.368696+0000) 2024-03-04T17:25:44.367 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:25:44.368+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:25:44.368930+0000) 2024-03-04T17:25:45.367 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:25:45.368+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:25:45.369186+0000) 2024-03-04T17:25:46.368 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:25:46.368+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:25:46.369429+0000) 2024-03-04T17:25:47.368 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:25:47.368+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:25:47.369687+0000) 2024-03-04T17:25:48.368 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:25:48.369+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:25:48.369926+0000) 2024-03-04T17:25:49.368 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:25:49.369+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:25:49.370166+0000) 2024-03-04T17:25:50.369 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:25:50.369+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:25:50.370424+0000) 2024-03-04T17:25:51.369 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:25:51.369+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:25:51.370687+0000) 2024-03-04T17:25:52.369 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:25:52.370+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:25:52.370936+0000) 2024-03-04T17:25:53.370 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:25:53.370+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:25:53.371185+0000) 2024-03-04T17:25:54.370 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:25:54.370+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:25:54.371418+0000) 2024-03-04T17:25:55.370 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:25:55.371+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:25:55.371659+0000) 2024-03-04T17:25:56.370 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:25:56.371+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:25:56.371940+0000) 2024-03-04T17:25:57.371 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:25:57.371+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:25:57.372179+0000) 2024-03-04T17:25:58.371 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:25:58.371+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:25:58.372417+0000) 2024-03-04T17:25:59.371 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:25:59.372+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:25:59.372604+0000) 2024-03-04T17:26:00.372 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:26:00.372+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:26:00.372911+0000) 2024-03-04T17:26:01.371 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:26:01.372+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:26:01.373154+0000) 2024-03-04T17:26:02.372 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:26:02.373+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:26:02.373406+0000) 2024-03-04T17:26:03.372 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:26:03.373+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:26:03.373645+0000) 2024-03-04T17:26:04.372 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:26:04.373+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:26:04.373914+0000) 2024-03-04T17:26:05.373 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:26:05.373+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:26:05.374151+0000) 2024-03-04T17:26:06.373 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:26:06.374+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:26:06.374385+0000) 2024-03-04T17:26:07.373 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:26:07.374+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:26:07.374625+0000) 2024-03-04T17:26:07.773 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:26:07.781 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:26:08.373 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:26:08.374+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:26:08.374958+0000) 2024-03-04T17:26:09.374 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:26:09.374+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:26:09.375193+0000) 2024-03-04T17:26:10.374 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:26:10.375+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:26:10.375436+0000) 2024-03-04T17:26:11.374 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:26:11.374+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:26:11.375680+0000) 2024-03-04T17:26:12.374 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:26:12.375+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:26:12.375963+0000) 2024-03-04T17:26:13.375 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:26:13.375+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:26:13.376205+0000) 2024-03-04T17:26:14.375 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:26:14.376+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:26:14.376439+0000) 2024-03-04T17:26:15.375 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:26:15.376+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:26:15.376710+0000) 2024-03-04T17:26:16.375 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:26:16.376+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:26:16.376961+0000) 2024-03-04T17:26:17.376 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:26:17.377+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:26:17.377206+0000) 2024-03-04T17:26:18.376 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:26:18.377+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:26:18.377451+0000) 2024-03-04T17:26:19.376 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:26:19.376+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:26:19.377689+0000) 2024-03-04T17:26:20.376 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:26:20.376+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:26:20.377932+0000) 2024-03-04T17:26:21.377 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:26:21.376+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:26:21.378168+0000) 2024-03-04T17:26:22.377 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:26:22.376+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:26:22.378421+0000) 2024-03-04T17:26:23.377 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:26:23.377+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:26:23.378660+0000) 2024-03-04T17:26:24.377 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:26:24.377+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:26:24.378931+0000) 2024-03-04T17:26:25.378 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:26:25.378+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:26:25.379190+0000) 2024-03-04T17:26:26.378 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:26:26.378+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:26:26.379439+0000) 2024-03-04T17:26:27.378 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:26:27.378+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:26:27.379684+0000) 2024-03-04T17:26:28.378 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:26:28.378+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:26:28.379955+0000) 2024-03-04T17:26:29.379 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:26:29.379+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:26:29.380194+0000) 2024-03-04T17:26:30.380 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:26:30.379+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:26:30.380375+0000) 2024-03-04T17:26:31.379 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:26:31.379+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:26:31.380613+0000) 2024-03-04T17:26:32.379 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:26:32.380+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:26:32.380913+0000) 2024-03-04T17:26:33.380 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:26:33.380+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:26:33.381153+0000) 2024-03-04T17:26:34.380 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:26:34.380+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:26:34.381412+0000) 2024-03-04T17:26:35.380 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:26:35.380+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:26:35.381601+0000) 2024-03-04T17:26:36.381 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:26:36.381+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:26:36.381885+0000) 2024-03-04T17:26:37.384 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:26:37.381+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:26:37.382126+0000) 2024-03-04T17:26:37.988 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:26:37.996 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:26:38.381 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:26:38.381+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:26:38.382401+0000) 2024-03-04T17:26:39.381 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:26:39.381+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:26:39.382640+0000) 2024-03-04T17:26:40.382 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:26:40.382+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:26:40.382948+0000) 2024-03-04T17:26:41.382 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:26:41.382+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:26:41.383191+0000) 2024-03-04T17:26:42.382 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:26:42.382+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:26:42.383447+0000) 2024-03-04T17:26:43.383 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:26:43.382+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:26:43.383695+0000) 2024-03-04T17:26:44.382 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:26:44.383+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:26:44.383933+0000) 2024-03-04T17:26:45.383 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:26:45.383+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:26:45.384174+0000) 2024-03-04T17:26:46.383 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:26:46.383+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:26:46.384372+0000) 2024-03-04T17:26:47.383 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:26:47.383+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:26:47.384617+0000) 2024-03-04T17:26:48.384 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:26:48.384+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:26:48.384892+0000) 2024-03-04T17:26:49.384 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:26:49.384+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:26:49.385075+0000) 2024-03-04T17:26:50.384 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:26:50.384+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:26:50.385346+0000) 2024-03-04T17:26:51.384 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:26:51.384+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:26:51.385612+0000) 2024-03-04T17:26:52.385 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:26:52.385+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:26:52.385942+0000) 2024-03-04T17:26:53.385 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:26:53.385+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:26:53.386187+0000) 2024-03-04T17:26:54.385 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:26:54.385+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:26:54.386417+0000) 2024-03-04T17:26:55.385 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:26:55.386+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:26:55.386659+0000) 2024-03-04T17:26:56.386 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:26:56.386+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:26:56.386950+0000) 2024-03-04T17:26:57.386 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:26:57.386+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:26:57.387164+0000) 2024-03-04T17:26:58.386 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:26:58.386+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:26:58.387354+0000) 2024-03-04T17:26:59.386 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:26:59.386+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:26:59.387537+0000) 2024-03-04T17:27:00.387 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:27:00.386+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:27:00.387767+0000) 2024-03-04T17:27:01.387 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:27:01.387+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:27:01.387946+0000) 2024-03-04T17:27:02.388 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:27:02.387+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:27:02.388211+0000) 2024-03-04T17:27:03.387 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:27:03.387+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:27:03.388415+0000) 2024-03-04T17:27:04.387 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:27:04.388+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:27:04.388662+0000) 2024-03-04T17:27:05.388 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:27:05.388+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:27:05.388956+0000) 2024-03-04T17:27:06.389 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:27:06.388+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:27:06.389214+0000) 2024-03-04T17:27:07.388 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:27:07.389+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:27:07.389479+0000) 2024-03-04T17:27:08.135 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:27:08.155 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:27:08.388 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:27:08.388+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:27:08.389742+0000) 2024-03-04T17:27:09.389 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:27:09.389+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:27:09.390008+0000) 2024-03-04T17:27:10.389 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:27:10.389+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:27:10.390243+0000) 2024-03-04T17:27:11.389 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:27:11.390+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:27:11.390486+0000) 2024-03-04T17:27:12.390 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:27:12.390+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:27:12.390750+0000) 2024-03-04T17:27:13.390 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:27:13.390+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:27:13.390982+0000) 2024-03-04T17:27:14.390 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:27:14.390+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:27:14.391229+0000) 2024-03-04T17:27:15.390 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:27:15.391+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:27:15.391480+0000) 2024-03-04T17:27:16.390 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:27:16.391+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:27:16.391732+0000) 2024-03-04T17:27:17.391 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:27:17.391+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:27:17.391992+0000) 2024-03-04T17:27:18.391 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:27:18.392+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:27:18.392247+0000) 2024-03-04T17:27:19.391 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:27:19.391+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:27:19.392488+0000) 2024-03-04T17:27:20.394 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:27:20.392+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:27:20.392744+0000) 2024-03-04T17:27:21.392 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:27:21.392+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:27:21.393004+0000) 2024-03-04T17:27:22.392 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:27:22.392+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:27:22.393299+0000) 2024-03-04T17:27:23.392 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:27:23.392+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:27:23.393551+0000) 2024-03-04T17:27:24.393 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:27:24.392+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:27:24.393801+0000) 2024-03-04T17:27:25.393 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:27:25.393+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:27:25.394099+0000) 2024-03-04T17:27:26.393 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:27:26.393+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:27:26.394352+0000) 2024-03-04T17:27:27.393 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:27:27.392+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:27:27.394680+0000) 2024-03-04T17:27:28.394 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:27:28.393+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:27:28.394934+0000) 2024-03-04T17:27:29.396 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:27:29.394+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:27:29.395182+0000) 2024-03-04T17:27:30.394 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:27:30.394+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:27:30.395451+0000) 2024-03-04T17:27:31.395 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:27:31.394+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:27:31.395712+0000) 2024-03-04T17:27:32.395 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:27:32.395+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:27:32.395982+0000) 2024-03-04T17:27:33.395 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:27:33.395+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:27:33.396187+0000) 2024-03-04T17:27:34.395 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:27:34.395+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:27:34.396454+0000) 2024-03-04T17:27:35.396 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:27:35.395+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:27:35.396693+0000) 2024-03-04T17:27:36.396 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:27:36.396+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:27:36.396955+0000) 2024-03-04T17:27:37.396 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:27:37.396+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:27:37.397206+0000) 2024-03-04T17:27:38.373 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:27:38.382 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:27:38.396 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:27:38.396+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:27:38.397385+0000) 2024-03-04T17:27:39.396 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:27:39.396+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:27:39.397616+0000) 2024-03-04T17:27:40.397 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:27:40.397+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:27:40.397895+0000) 2024-03-04T17:27:41.397 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:27:41.397+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:27:41.398158+0000) 2024-03-04T17:27:42.397 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:27:42.397+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:27:42.398374+0000) 2024-03-04T17:27:43.398 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:27:43.397+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:27:43.398631+0000) 2024-03-04T17:27:44.398 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:27:44.398+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:27:44.398918+0000) 2024-03-04T17:27:45.398 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:27:45.398+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:27:45.399153+0000) 2024-03-04T17:27:46.398 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:27:46.398+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:27:46.399402+0000) 2024-03-04T17:27:47.399 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:27:47.398+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:27:47.399662+0000) 2024-03-04T17:27:48.399 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:27:48.399+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:27:48.399943+0000) 2024-03-04T17:27:49.399 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:27:49.399+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:27:49.400192+0000) 2024-03-04T17:27:50.399 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:27:50.399+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:27:50.400457+0000) 2024-03-04T17:27:51.400 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:27:51.400+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:27:51.400718+0000) 2024-03-04T17:27:52.400 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:27:52.400+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:27:52.400992+0000) 2024-03-04T17:27:53.400 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:27:53.400+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:27:53.401229+0000) 2024-03-04T17:27:54.400 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:27:54.400+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:27:54.401465+0000) 2024-03-04T17:27:55.401 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:27:55.401+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:27:55.401706+0000) 2024-03-04T17:27:56.403 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:27:56.401+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:27:56.401960+0000) 2024-03-04T17:27:57.401 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:27:57.401+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:27:57.402204+0000) 2024-03-04T17:27:58.401 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:27:58.401+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:27:58.402442+0000) 2024-03-04T17:27:59.402 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:27:59.402+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:27:59.402681+0000) 2024-03-04T17:28:00.402 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:28:00.402+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:28:00.402936+0000) 2024-03-04T17:28:01.402 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:28:01.402+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:28:01.403184+0000) 2024-03-04T17:28:02.402 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:28:02.402+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:28:02.403445+0000) 2024-03-04T17:28:03.403 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:28:03.403+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:28:03.403685+0000) 2024-03-04T17:28:04.403 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:28:04.403+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:28:04.403936+0000) 2024-03-04T17:28:05.403 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:28:05.403+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:28:05.404184+0000) 2024-03-04T17:28:06.403 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:28:06.403+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:28:06.404431+0000) 2024-03-04T17:28:07.404 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:28:07.404+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:28:07.404674+0000) 2024-03-04T17:28:08.404 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:28:08.404+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:28:08.404945+0000) 2024-03-04T17:28:08.513 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:28:08.526 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:28:09.404 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:28:09.404+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:28:09.405199+0000) 2024-03-04T17:28:10.404 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:28:10.404+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:28:10.405401+0000) 2024-03-04T17:28:11.405 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:28:11.405+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:28:11.405665+0000) 2024-03-04T17:28:12.405 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:28:12.405+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:28:12.405940+0000) 2024-03-04T17:28:13.408 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:28:13.405+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:28:13.406208+0000) 2024-03-04T17:28:14.406 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:28:14.405+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:28:14.406405+0000) 2024-03-04T17:28:15.406 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:28:15.406+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:28:15.406744+0000) 2024-03-04T17:28:16.406 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:28:16.406+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:28:16.407045+0000) 2024-03-04T17:28:17.406 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:28:17.407+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:28:17.407306+0000) 2024-03-04T17:28:18.407 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:28:18.407+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:28:18.407542+0000) 2024-03-04T17:28:19.407 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:28:19.407+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:28:19.407770+0000) 2024-03-04T17:28:20.407 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:28:20.407+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:28:20.408020+0000) 2024-03-04T17:28:21.407 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:28:21.408+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:28:21.408281+0000) 2024-03-04T17:28:22.408 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:28:22.408+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:28:22.408565+0000) 2024-03-04T17:28:23.408 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:28:23.408+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:28:23.408816+0000) 2024-03-04T17:28:24.408 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:28:24.408+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:28:24.409077+0000) 2024-03-04T17:28:25.408 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:28:25.409+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:28:25.409336+0000) 2024-03-04T17:28:26.409 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:28:26.408+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:28:26.409625+0000) 2024-03-04T17:28:27.409 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:28:27.408+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:28:27.409924+0000) 2024-03-04T17:28:28.409 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:28:28.409+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:28:28.410179+0000) 2024-03-04T17:28:29.410 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:28:29.408+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:28:29.410445+0000) 2024-03-04T17:28:30.410 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:28:30.408+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:28:30.410720+0000) 2024-03-04T17:28:31.410 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:28:31.409+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:28:31.410960+0000) 2024-03-04T17:28:32.410 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:28:32.410+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:28:32.411235+0000) 2024-03-04T17:28:33.411 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:28:33.410+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:28:33.411480+0000) 2024-03-04T17:28:34.411 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:28:34.410+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:28:34.411722+0000) 2024-03-04T17:28:35.411 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:28:35.411+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:28:35.412010+0000) 2024-03-04T17:28:36.411 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:28:36.411+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:28:36.412260+0000) 2024-03-04T17:28:37.412 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:28:37.411+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:28:37.412543+0000) 2024-03-04T17:28:38.412 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:28:38.411+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:28:38.412801+0000) 2024-03-04T17:28:38.716 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:28:38.725 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:28:39.412 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:28:39.412+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:28:39.413098+0000) 2024-03-04T17:28:40.412 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:28:40.412+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:28:40.413357+0000) 2024-03-04T17:28:41.413 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:28:41.412+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:28:41.413590+0000) 2024-03-04T17:28:42.413 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:28:42.413+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:28:42.413883+0000) 2024-03-04T17:28:43.413 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:28:43.413+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:28:43.414113+0000) 2024-03-04T17:28:44.413 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:28:44.413+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:28:44.414347+0000) 2024-03-04T17:28:45.414 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:28:45.413+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:28:45.414587+0000) 2024-03-04T17:28:46.414 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:28:46.414+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:28:46.414851+0000) 2024-03-04T17:28:47.414 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:28:47.414+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:28:47.415098+0000) 2024-03-04T17:28:48.415 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:28:48.414+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:28:48.415338+0000) 2024-03-04T17:28:49.415 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:28:49.414+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:28:49.415532+0000) 2024-03-04T17:28:50.416 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:28:50.415+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:28:50.415793+0000) 2024-03-04T17:28:51.415 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:28:51.415+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:28:51.416008+0000) 2024-03-04T17:28:52.416 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:28:52.415+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:28:52.416257+0000) 2024-03-04T17:28:53.416 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:28:53.415+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:28:53.416566+0000) 2024-03-04T17:28:54.420 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:28:54.416+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:28:54.416858+0000) 2024-03-04T17:28:55.416 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:28:55.416+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:28:55.417106+0000) 2024-03-04T17:28:56.417 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:28:56.416+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:28:56.417361+0000) 2024-03-04T17:28:57.421 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:28:57.416+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:28:57.417609+0000) 2024-03-04T17:28:58.417 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:28:58.417+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:28:58.417911+0000) 2024-03-04T17:28:59.417 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:28:59.417+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:28:59.418156+0000) 2024-03-04T17:29:00.418 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:29:00.417+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:29:00.418420+0000) 2024-03-04T17:29:01.418 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:29:01.418+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:29:01.418671+0000) 2024-03-04T17:29:02.419 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:29:02.418+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:29:02.418937+0000) 2024-03-04T17:29:03.418 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:29:03.418+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:29:03.419236+0000) 2024-03-04T17:29:04.419 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:29:04.418+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:29:04.419490+0000) 2024-03-04T17:29:05.419 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:29:05.419+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:29:05.419736+0000) 2024-03-04T17:29:06.419 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:29:06.419+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:29:06.420008+0000) 2024-03-04T17:29:07.419 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:29:07.419+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:29:07.420257+0000) 2024-03-04T17:29:08.420 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:29:08.420+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:29:08.420507+0000) 2024-03-04T17:29:08.932 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:29:08.940 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:29:09.420 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:29:09.420+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:29:09.420787+0000) 2024-03-04T17:29:10.420 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:29:10.420+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:29:10.421113+0000) 2024-03-04T17:29:11.421 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:29:11.420+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:29:11.421378+0000) 2024-03-04T17:29:12.421 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:29:12.421+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:29:12.421643+0000) 2024-03-04T17:29:13.421 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:29:13.421+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:29:13.421923+0000) 2024-03-04T17:29:14.421 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:29:14.421+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:29:14.422168+0000) 2024-03-04T17:29:15.422 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:29:15.422+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:29:15.422454+0000) 2024-03-04T17:29:16.423 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:29:16.422+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:29:16.422781+0000) 2024-03-04T17:29:17.422 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:29:17.422+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:29:17.423114+0000) 2024-03-04T17:29:18.423 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:29:18.423+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:29:18.423426+0000) 2024-03-04T17:29:19.423 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:29:19.423+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:29:19.423713+0000) 2024-03-04T17:29:20.423 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:29:20.423+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:29:20.423952+0000) 2024-03-04T17:29:21.425 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:29:21.423+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:29:21.424184+0000) 2024-03-04T17:29:22.424 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:29:22.424+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:29:22.424427+0000) 2024-03-04T17:29:23.424 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:29:23.424+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:29:23.424661+0000) 2024-03-04T17:29:24.424 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:29:24.424+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:29:24.424926+0000) 2024-03-04T17:29:25.424 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:29:25.424+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:29:25.425122+0000) 2024-03-04T17:29:26.425 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:29:26.425+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:29:26.425381+0000) 2024-03-04T17:29:27.425 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:29:27.425+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:29:27.425613+0000) 2024-03-04T17:29:28.425 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:29:28.425+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:29:28.425884+0000) 2024-03-04T17:29:29.425 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:29:29.425+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:29:29.426139+0000) 2024-03-04T17:29:30.426 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:29:30.424+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:29:30.426402+0000) 2024-03-04T17:29:31.426 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:29:31.424+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:29:31.426629+0000) 2024-03-04T17:29:32.427 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:29:32.425+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:29:32.426923+0000) 2024-03-04T17:29:33.427 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:29:33.426+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:29:33.427163+0000) 2024-03-04T17:29:34.427 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:29:34.426+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:29:34.427398+0000) 2024-03-04T17:29:35.427 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:29:35.426+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:29:35.427641+0000) 2024-03-04T17:29:36.427 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:29:36.426+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:29:36.427930+0000) 2024-03-04T17:29:37.428 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:29:37.427+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:29:37.428164+0000) 2024-03-04T17:29:38.429 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:29:38.427+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:29:38.428400+0000) 2024-03-04T17:29:39.119 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:29:39.133 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:29:39.428 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:29:39.427+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:29:39.428643+0000) 2024-03-04T17:29:40.428 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:29:40.428+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:29:40.428925+0000) 2024-03-04T17:29:41.428 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:29:41.428+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:29:41.429154+0000) 2024-03-04T17:29:42.429 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:29:42.428+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:29:42.429406+0000) 2024-03-04T17:29:43.429 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:29:43.428+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:29:43.429640+0000) 2024-03-04T17:29:44.429 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:29:44.429+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:29:44.429926+0000) 2024-03-04T17:29:45.433 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:29:45.429+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:29:45.430182+0000) 2024-03-04T17:29:46.430 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:29:46.429+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:29:46.430424+0000) 2024-03-04T17:29:47.431 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:29:47.429+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:29:47.430610+0000) 2024-03-04T17:29:48.430 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:29:48.430+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:29:48.430919+0000) 2024-03-04T17:29:49.431 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:29:49.430+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:29:49.431169+0000) 2024-03-04T17:29:50.431 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:29:50.430+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:29:50.431396+0000) 2024-03-04T17:29:51.435 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:29:51.430+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:29:51.431631+0000) 2024-03-04T17:29:52.432 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:29:52.431+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:29:52.431935+0000) 2024-03-04T17:29:53.432 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:29:53.431+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:29:53.432121+0000) 2024-03-04T17:29:54.432 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:29:54.431+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:29:54.432352+0000) 2024-03-04T17:29:55.433 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:29:55.431+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:29:55.432602+0000) 2024-03-04T17:29:56.432 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:29:56.432+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:29:56.432867+0000) 2024-03-04T17:29:57.435 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:29:57.432+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:29:57.433106+0000) 2024-03-04T17:29:58.435 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:29:58.432+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:29:58.433348+0000) 2024-03-04T17:29:59.435 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:29:59.432+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:29:59.433609+0000) 2024-03-04T17:30:00.433 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:30:00.433+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:30:00.433878+0000) 2024-03-04T17:30:01.434 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:30:01.433+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:30:01.434114+0000) 2024-03-04T17:30:02.434 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:30:02.433+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:30:02.434363+0000) 2024-03-04T17:30:03.434 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:30:03.433+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:30:03.434542+0000) 2024-03-04T17:30:04.435 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:30:04.434+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:30:04.434779+0000) 2024-03-04T17:30:05.434 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:30:05.434+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:30:05.435021+0000) 2024-03-04T17:30:06.436 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:30:06.434+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:30:06.435250+0000) 2024-03-04T17:30:07.435 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:30:07.434+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:30:07.435489+0000) 2024-03-04T17:30:08.436 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:30:08.435+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:30:08.435727+0000) 2024-03-04T17:30:09.374 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:30:09.383 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:30:09.435 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:30:09.435+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:30:09.435894+0000) 2024-03-04T17:30:10.436 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:30:10.435+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:30:10.436140+0000) 2024-03-04T17:30:11.436 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:30:11.435+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:30:11.436376+0000) 2024-03-04T17:30:12.436 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:30:12.435+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:30:12.436570+0000) 2024-03-04T17:30:13.437 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:30:13.435+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:30:13.436804+0000) 2024-03-04T17:30:14.437 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:30:14.436+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:30:14.437082+0000) 2024-03-04T17:30:15.437 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:30:15.436+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:30:15.437276+0000) 2024-03-04T17:30:16.437 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:30:16.437+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:30:16.437513+0000) 2024-03-04T17:30:17.441 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:30:17.437+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:30:17.437750+0000) 2024-03-04T17:30:18.438 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:30:18.437+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:30:18.437994+0000) 2024-03-04T17:30:19.439 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:30:19.437+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:30:19.438224+0000) 2024-03-04T17:30:20.438 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:30:20.438+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:30:20.438483+0000) 2024-03-04T17:30:21.438 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:30:21.438+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:30:21.438752+0000) 2024-03-04T17:30:22.439 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:30:22.438+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:30:22.438999+0000) 2024-03-04T17:30:23.439 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:30:23.438+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:30:23.439239+0000) 2024-03-04T17:30:24.439 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:30:24.439+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:30:24.439470+0000) 2024-03-04T17:30:25.440 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:30:25.439+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:30:25.439725+0000) 2024-03-04T17:30:26.440 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:30:26.439+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:30:26.439971+0000) 2024-03-04T17:30:27.440 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:30:27.440+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:30:27.440209+0000) 2024-03-04T17:30:28.440 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:30:28.440+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:30:28.440449+0000) 2024-03-04T17:30:29.440 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:30:29.440+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:30:29.440682+0000) 2024-03-04T17:30:30.441 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:30:30.440+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:30:30.440956+0000) 2024-03-04T17:30:31.443 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:30:31.441+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:30:31.441192+0000) 2024-03-04T17:30:32.441 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:30:32.439+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:30:32.441401+0000) 2024-03-04T17:30:33.445 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:30:33.440+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:30:33.441641+0000) 2024-03-04T17:30:34.442 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:30:34.440+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:30:34.441922+0000) 2024-03-04T17:30:35.442 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:30:35.441+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:30:35.442161+0000) 2024-03-04T17:30:36.442 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:30:36.441+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:30:36.442401+0000) 2024-03-04T17:30:37.442 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:30:37.440+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:30:37.442591+0000) 2024-03-04T17:30:38.442 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:30:38.441+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:30:38.442854+0000) 2024-03-04T17:30:39.443 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:30:39.442+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:30:39.443120+0000) 2024-03-04T17:30:39.528 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:30:39.536 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:30:40.443 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:30:40.441+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:30:40.443350+0000) 2024-03-04T17:30:41.443 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:30:41.442+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:30:41.443576+0000) 2024-03-04T17:30:42.443 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:30:42.442+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:30:42.443811+0000) 2024-03-04T17:30:43.444 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:30:43.443+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:30:43.444142+0000) 2024-03-04T17:30:44.444 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:30:44.443+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:30:44.444377+0000) 2024-03-04T17:30:45.444 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:30:45.443+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:30:45.444610+0000) 2024-03-04T17:30:46.445 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:30:46.444+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:30:46.444893+0000) 2024-03-04T17:30:47.445 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:30:47.444+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:30:47.445133+0000) 2024-03-04T17:30:48.447 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:30:48.444+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:30:48.445368+0000) 2024-03-04T17:30:49.445 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:30:49.444+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:30:49.445608+0000) 2024-03-04T17:30:50.446 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:30:50.445+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:30:50.445957+0000) 2024-03-04T17:30:51.446 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:30:51.445+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:30:51.446223+0000) 2024-03-04T17:30:52.446 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:30:52.445+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:30:52.446470+0000) 2024-03-04T17:30:53.446 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:30:53.445+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:30:53.446707+0000) 2024-03-04T17:30:54.447 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:30:54.446+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:30:54.446943+0000) 2024-03-04T17:30:55.447 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:30:55.446+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:30:55.447186+0000) 2024-03-04T17:30:56.447 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:30:56.446+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:30:56.447426+0000) 2024-03-04T17:30:57.447 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:30:57.446+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:30:57.447665+0000) 2024-03-04T17:30:58.448 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:30:58.447+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:30:58.447948+0000) 2024-03-04T17:30:59.448 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:30:59.447+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:30:59.448188+0000) 2024-03-04T17:31:00.451 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:31:00.447+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:31:00.448428+0000) 2024-03-04T17:31:01.448 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:31:01.448+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:31:01.448668+0000) 2024-03-04T17:31:02.449 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:31:02.448+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:31:02.448951+0000) 2024-03-04T17:31:03.449 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:31:03.448+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:31:03.449191+0000) 2024-03-04T17:31:04.449 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:31:04.448+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:31:04.449433+0000) 2024-03-04T17:31:05.450 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:31:05.449+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:31:05.449671+0000) 2024-03-04T17:31:06.450 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:31:06.449+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:31:06.449945+0000) 2024-03-04T17:31:07.456 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:31:07.449+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:31:07.450217+0000) 2024-03-04T17:31:08.450 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:31:08.449+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:31:08.450456+0000) 2024-03-04T17:31:09.451 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:31:09.449+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:31:09.450707+0000) 2024-03-04T17:31:09.742 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:31:09.750 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:31:10.451 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:31:10.450+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:31:10.450953+0000) 2024-03-04T17:31:11.451 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:31:11.450+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:31:11.451145+0000) 2024-03-04T17:31:12.451 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:31:12.450+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:31:12.451395+0000) 2024-03-04T17:31:13.451 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:31:13.451+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:31:13.451636+0000) 2024-03-04T17:31:14.452 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:31:14.451+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:31:14.451946+0000) 2024-03-04T17:31:15.452 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:31:15.451+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:31:15.452194+0000) 2024-03-04T17:31:16.452 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:31:16.452+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:31:16.452439+0000) 2024-03-04T17:31:17.452 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:31:17.451+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:31:17.452683+0000) 2024-03-04T17:31:18.453 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:31:18.452+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:31:18.452947+0000) 2024-03-04T17:31:19.454 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:31:19.452+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:31:19.453184+0000) 2024-03-04T17:31:20.453 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:31:20.453+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:31:20.453444+0000) 2024-03-04T17:31:21.459 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:31:21.453+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:31:21.453702+0000) 2024-03-04T17:31:22.454 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:31:22.453+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:31:22.453980+0000) 2024-03-04T17:31:23.454 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:31:23.453+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:31:23.454275+0000) 2024-03-04T17:31:24.454 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:31:24.454+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:31:24.454522+0000) 2024-03-04T17:31:25.455 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:31:25.454+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:31:25.454763+0000) 2024-03-04T17:31:26.455 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:31:26.454+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:31:26.454997+0000) 2024-03-04T17:31:27.455 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:31:27.454+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:31:27.455235+0000) 2024-03-04T17:31:28.455 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:31:28.455+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:31:28.455483+0000) 2024-03-04T17:31:29.456 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:31:29.455+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:31:29.455772+0000) 2024-03-04T17:31:30.456 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:31:30.455+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:31:30.456075+0000) 2024-03-04T17:31:31.456 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:31:31.456+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:31:31.456372+0000) 2024-03-04T17:31:32.457 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:31:32.456+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:31:32.456687+0000) 2024-03-04T17:31:33.457 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:31:33.456+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:31:33.456985+0000) 2024-03-04T17:31:34.457 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:31:34.456+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:31:34.457287+0000) 2024-03-04T17:31:35.457 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:31:35.455+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:31:35.457538+0000) 2024-03-04T17:31:36.458 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:31:36.456+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:31:36.457864+0000) 2024-03-04T17:31:37.458 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:31:37.457+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:31:37.458196+0000) 2024-03-04T17:31:38.460 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:31:38.457+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:31:38.458542+0000) 2024-03-04T17:31:39.459 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:31:39.457+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:31:39.458765+0000) 2024-03-04T17:31:39.954 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:31:39.965 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:31:40.459 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:31:40.457+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:31:40.459081+0000) 2024-03-04T17:31:41.459 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:31:41.458+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:31:41.459375+0000) 2024-03-04T17:31:42.460 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:31:42.458+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:31:42.459685+0000) 2024-03-04T17:31:43.460 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:31:43.458+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:31:43.459991+0000) 2024-03-04T17:31:44.460 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:31:44.459+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:31:44.460284+0000) 2024-03-04T17:31:45.461 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:31:45.459+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:31:45.460575+0000) 2024-03-04T17:31:46.461 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:31:46.460+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:31:46.460924+0000) 2024-03-04T17:31:47.461 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:31:47.460+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:31:47.461230+0000) 2024-03-04T17:31:48.461 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:31:48.460+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:31:48.461541+0000) 2024-03-04T17:31:49.462 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:31:49.461+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:31:49.461864+0000) 2024-03-04T17:31:50.462 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:31:50.461+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:31:50.462160+0000) 2024-03-04T17:31:51.462 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:31:51.461+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:31:51.462437+0000) 2024-03-04T17:31:52.463 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:31:52.461+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:31:52.462733+0000) 2024-03-04T17:31:53.463 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:31:53.462+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:31:53.463043+0000) 2024-03-04T17:31:54.463 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:31:54.462+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:31:54.463318+0000) 2024-03-04T17:31:55.464 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:31:55.462+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:31:55.463606+0000) 2024-03-04T17:31:56.464 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:31:56.463+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:31:56.463927+0000) 2024-03-04T17:31:57.464 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:31:57.463+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:31:57.464222+0000) 2024-03-04T17:31:58.464 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:31:58.463+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:31:58.464507+0000) 2024-03-04T17:31:59.465 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:31:59.464+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:31:59.464792+0000) 2024-03-04T17:32:00.466 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:32:00.464+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:32:00.465081+0000) 2024-03-04T17:32:01.465 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:32:01.464+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:32:01.465368+0000) 2024-03-04T17:32:02.466 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:32:02.464+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:32:02.465676+0000) 2024-03-04T17:32:03.466 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:32:03.465+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:32:03.465997+0000) 2024-03-04T17:32:04.466 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:32:04.465+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:32:04.466278+0000) 2024-03-04T17:32:05.467 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:32:05.465+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:32:05.466578+0000) 2024-03-04T17:32:06.467 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:32:06.466+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:32:06.466923+0000) 2024-03-04T17:32:07.467 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:32:07.466+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:32:07.467220+0000) 2024-03-04T17:32:08.467 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:32:08.466+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:32:08.467449+0000) 2024-03-04T17:32:09.468 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:32:09.467+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:32:09.467740+0000) 2024-03-04T17:32:10.161 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:32:10.169 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:32:10.468 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:32:10.467+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:32:10.468031+0000) 2024-03-04T17:32:11.468 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:32:11.467+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:32:11.468326+0000) 2024-03-04T17:32:12.469 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:32:12.468+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:32:12.468629+0000) 2024-03-04T17:32:13.469 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:32:13.468+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:32:13.468964+0000) 2024-03-04T17:32:14.469 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:32:14.468+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:32:14.469257+0000) 2024-03-04T17:32:15.470 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:32:15.469+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:32:15.469537+0000) 2024-03-04T17:32:16.470 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:32:16.469+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:32:16.469863+0000) 2024-03-04T17:32:17.471 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:32:17.469+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:32:17.470151+0000) 2024-03-04T17:32:18.471 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:32:18.469+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:32:18.470426+0000) 2024-03-04T17:32:19.471 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:32:19.470+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:32:19.470720+0000) 2024-03-04T17:32:20.471 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:32:20.470+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:32:20.471017+0000) 2024-03-04T17:32:21.472 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:32:21.471+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:32:21.471420+0000) 2024-03-04T17:32:22.472 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:32:22.471+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:32:22.471727+0000) 2024-03-04T17:32:23.472 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:32:23.471+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:32:23.472021+0000) 2024-03-04T17:32:24.473 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:32:24.471+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:32:24.472302+0000) 2024-03-04T17:32:25.473 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:32:25.472+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:32:25.472594+0000) 2024-03-04T17:32:26.473 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:32:26.472+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:32:26.472931+0000) 2024-03-04T17:32:27.473 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:32:27.472+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:32:27.473161+0000) 2024-03-04T17:32:28.474 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:32:28.473+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:32:28.473463+0000) 2024-03-04T17:32:29.474 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:32:29.472+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:32:29.473690+0000) 2024-03-04T17:32:30.474 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:32:30.473+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:32:30.473983+0000) 2024-03-04T17:32:31.475 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:32:31.474+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:32:31.474271+0000) 2024-03-04T17:32:32.475 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:32:32.474+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:32:32.474582+0000) 2024-03-04T17:32:33.476 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:32:33.474+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:32:33.474927+0000) 2024-03-04T17:32:34.475 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:32:34.474+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:32:34.475173+0000) 2024-03-04T17:32:35.476 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:32:35.475+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:32:35.475464+0000) 2024-03-04T17:32:36.476 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:32:36.474+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:32:36.475757+0000) 2024-03-04T17:32:37.476 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:32:37.474+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:32:37.476063+0000) 2024-03-04T17:32:38.477 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:32:38.475+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:32:38.476380+0000) 2024-03-04T17:32:39.477 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:32:39.475+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:32:39.476685+0000) 2024-03-04T17:32:40.382 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:32:40.391 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:32:40.477 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:32:40.475+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:32:40.476908+0000) 2024-03-04T17:32:41.477 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:32:41.475+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:32:41.477106+0000) 2024-03-04T17:32:42.477 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:32:42.476+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:32:42.477374+0000) 2024-03-04T17:32:43.478 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:32:43.476+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:32:43.477619+0000) 2024-03-04T17:32:44.478 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:32:44.476+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:32:44.477894+0000) 2024-03-04T17:32:45.478 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:32:45.477+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:32:45.478128+0000) 2024-03-04T17:32:46.478 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:32:46.477+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:32:46.478371+0000) 2024-03-04T17:32:47.479 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:32:47.477+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:32:47.478644+0000) 2024-03-04T17:32:48.479 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:32:48.478+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:32:48.478961+0000) 2024-03-04T17:32:49.479 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:32:49.478+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:32:49.479196+0000) 2024-03-04T17:32:50.480 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:32:50.478+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:32:50.479438+0000) 2024-03-04T17:32:51.481 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:32:51.478+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:32:51.479674+0000) 2024-03-04T17:32:52.480 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:32:52.479+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:32:52.479972+0000) 2024-03-04T17:32:53.480 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:32:53.479+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:32:53.480214+0000) 2024-03-04T17:32:54.481 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:32:54.479+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:32:54.480472+0000) 2024-03-04T17:32:55.481 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:32:55.479+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:32:55.480736+0000) 2024-03-04T17:32:56.481 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:32:56.480+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:32:56.480977+0000) 2024-03-04T17:32:57.481 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:32:57.480+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:32:57.481210+0000) 2024-03-04T17:32:58.482 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:32:58.480+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:32:58.481468+0000) 2024-03-04T17:32:59.482 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:32:59.480+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:32:59.481711+0000) 2024-03-04T17:33:00.482 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:33:00.481+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:33:00.481986+0000) 2024-03-04T17:33:01.482 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:33:01.481+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:33:01.482224+0000) 2024-03-04T17:33:02.483 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:33:02.481+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:33:02.482485+0000) 2024-03-04T17:33:03.483 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:33:03.482+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:33:03.482731+0000) 2024-03-04T17:33:04.483 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:33:04.482+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:33:04.482972+0000) 2024-03-04T17:33:05.483 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:33:05.482+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:33:05.483160+0000) 2024-03-04T17:33:06.484 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:33:06.482+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:33:06.483352+0000) 2024-03-04T17:33:07.485 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:33:07.482+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:33:07.483599+0000) 2024-03-04T17:33:08.484 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:33:08.483+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:33:08.483886+0000) 2024-03-04T17:33:09.484 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:33:09.483+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:33:09.484198+0000) 2024-03-04T17:33:10.485 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:33:10.483+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:33:10.484455+0000) 2024-03-04T17:33:10.588 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:33:10.596 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:33:11.485 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:33:11.483+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:33:11.484642+0000) 2024-03-04T17:33:12.485 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:33:12.484+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:33:12.484933+0000) 2024-03-04T17:33:13.486 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:33:13.484+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:33:13.485198+0000) 2024-03-04T17:33:14.486 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:33:14.484+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:33:14.485433+0000) 2024-03-04T17:33:15.486 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:33:15.484+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:33:15.485683+0000) 2024-03-04T17:33:16.486 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:33:16.485+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:33:16.485925+0000) 2024-03-04T17:33:17.487 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:33:17.485+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:33:17.486188+0000) 2024-03-04T17:33:18.487 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:33:18.485+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:33:18.486436+0000) 2024-03-04T17:33:19.487 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:33:19.486+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:33:19.486675+0000) 2024-03-04T17:33:20.487 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:33:20.486+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:33:20.486955+0000) 2024-03-04T17:33:21.487 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:33:21.486+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:33:21.487182+0000) 2024-03-04T17:33:22.488 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:33:22.487+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:33:22.487425+0000) 2024-03-04T17:33:23.488 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:33:23.487+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:33:23.487673+0000) 2024-03-04T17:33:24.488 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:33:24.487+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:33:24.487946+0000) 2024-03-04T17:33:25.488 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:33:25.487+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:33:25.488138+0000) 2024-03-04T17:33:26.489 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:33:26.488+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:33:26.488374+0000) 2024-03-04T17:33:27.489 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:33:27.488+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:33:27.488607+0000) 2024-03-04T17:33:28.489 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:33:28.488+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:33:28.488888+0000) 2024-03-04T17:33:29.489 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:33:29.488+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:33:29.489125+0000) 2024-03-04T17:33:30.490 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:33:30.489+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:33:30.489366+0000) 2024-03-04T17:33:31.490 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:33:31.489+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:33:31.489601+0000) 2024-03-04T17:33:32.490 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:33:32.489+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:33:32.489924+0000) 2024-03-04T17:33:33.490 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:33:33.489+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:33:33.490110+0000) 2024-03-04T17:33:34.491 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:33:34.490+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:33:34.490351+0000) 2024-03-04T17:33:35.491 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:33:35.490+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:33:35.490590+0000) 2024-03-04T17:33:36.491 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:33:36.490+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:33:36.490861+0000) 2024-03-04T17:33:37.492 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:33:37.490+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:33:37.491100+0000) 2024-03-04T17:33:38.492 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:33:38.490+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:33:38.491358+0000) 2024-03-04T17:33:39.492 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:33:39.489+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:33:39.491608+0000) 2024-03-04T17:33:40.493 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:33:40.490+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:33:40.491886+0000) 2024-03-04T17:33:40.817 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:33:40.826 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:33:41.492 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:33:41.491+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:33:41.492125+0000) 2024-03-04T17:33:42.493 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:33:42.491+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:33:42.492403+0000) 2024-03-04T17:33:43.493 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:33:43.491+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:33:43.492637+0000) 2024-03-04T17:33:44.498 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:33:44.491+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:33:44.492916+0000) 2024-03-04T17:33:45.494 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:33:45.492+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:33:45.493165+0000) 2024-03-04T17:33:46.494 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:33:46.492+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:33:46.493404+0000) 2024-03-04T17:33:47.494 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:33:47.492+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:33:47.493646+0000) 2024-03-04T17:33:48.494 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:33:48.492+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:33:48.493856+0000) 2024-03-04T17:33:49.495 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:33:49.493+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:33:49.494094+0000) 2024-03-04T17:33:50.495 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:33:50.493+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:33:50.494338+0000) 2024-03-04T17:33:51.495 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:33:51.493+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:33:51.494575+0000) 2024-03-04T17:33:52.495 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:33:52.494+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:33:52.494857+0000) 2024-03-04T17:33:53.496 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:33:53.494+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:33:53.495107+0000) 2024-03-04T17:33:54.496 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:33:54.494+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:33:54.495341+0000) 2024-03-04T17:33:55.496 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:33:55.494+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:33:55.495578+0000) 2024-03-04T17:33:56.496 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:33:56.495+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:33:56.495866+0000) 2024-03-04T17:33:57.497 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:33:57.495+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:33:57.496103+0000) 2024-03-04T17:33:58.497 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:33:58.495+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:33:58.496351+0000) 2024-03-04T17:33:59.497 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:33:59.495+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:33:59.496584+0000) 2024-03-04T17:34:00.497 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:34:00.496+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:34:00.496850+0000) 2024-03-04T17:34:01.497 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:34:01.496+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:34:01.497029+0000) 2024-03-04T17:34:02.498 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:34:02.496+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:34:02.497285+0000) 2024-03-04T17:34:03.498 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:34:03.496+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:34:03.497471+0000) 2024-03-04T17:34:04.498 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:34:04.497+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:34:04.497713+0000) 2024-03-04T17:34:05.499 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:34:05.497+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:34:05.497983+0000) 2024-03-04T17:34:06.499 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:34:06.497+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:34:06.498222+0000) 2024-03-04T17:34:07.499 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:34:07.497+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:34:07.498461+0000) 2024-03-04T17:34:08.499 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:34:08.497+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:34:08.498716+0000) 2024-03-04T17:34:09.500 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:34:09.498+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:34:09.498947+0000) 2024-03-04T17:34:10.500 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:34:10.498+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:34:10.499205+0000) 2024-03-04T17:34:10.965 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:34:10.973 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:34:11.500 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:34:11.498+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:34:11.499420+0000) 2024-03-04T17:34:12.500 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:34:12.499+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:34:12.499667+0000) 2024-03-04T17:34:13.500 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:34:13.499+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:34:13.499950+0000) 2024-03-04T17:34:14.501 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:34:14.499+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:34:14.500176+0000) 2024-03-04T17:34:15.501 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:34:15.499+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:34:15.500416+0000) 2024-03-04T17:34:16.501 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:34:16.500+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:34:16.500659+0000) 2024-03-04T17:34:17.502 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:34:17.500+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:34:17.500945+0000) 2024-03-04T17:34:18.502 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:34:18.500+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:34:18.501193+0000) 2024-03-04T17:34:19.502 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:34:19.500+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:34:19.501374+0000) 2024-03-04T17:34:20.502 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:34:20.501+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:34:20.501609+0000) 2024-03-04T17:34:21.503 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:34:21.501+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:34:21.501878+0000) 2024-03-04T17:34:22.503 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:34:22.501+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:34:22.502123+0000) 2024-03-04T17:34:23.503 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:34:23.501+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:34:23.502363+0000) 2024-03-04T17:34:24.503 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:34:24.502+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:34:24.502600+0000) 2024-03-04T17:34:25.503 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:34:25.502+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:34:25.502865+0000) 2024-03-04T17:34:26.504 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:34:26.502+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:34:26.503115+0000) 2024-03-04T17:34:27.504 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:34:27.502+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:34:27.503354+0000) 2024-03-04T17:34:28.504 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:34:28.503+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:34:28.503596+0000) 2024-03-04T17:34:29.505 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:34:29.503+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:34:29.503854+0000) 2024-03-04T17:34:30.505 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:34:30.503+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:34:30.504110+0000) 2024-03-04T17:34:31.505 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:34:31.503+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:34:31.504298+0000) 2024-03-04T17:34:32.506 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:34:32.503+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:34:32.504502+0000) 2024-03-04T17:34:33.505 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:34:33.504+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:34:33.504755+0000) 2024-03-04T17:34:34.506 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:34:34.504+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:34:34.505011+0000) 2024-03-04T17:34:35.506 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:34:35.505+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:34:35.505252+0000) 2024-03-04T17:34:36.506 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:34:36.504+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:34:36.505465+0000) 2024-03-04T17:34:37.506 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:34:37.504+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:34:37.505695+0000) 2024-03-04T17:34:38.506 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:34:38.505+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:34:38.505931+0000) 2024-03-04T17:34:39.507 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:34:39.506+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:34:39.506170+0000) 2024-03-04T17:34:40.507 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:34:40.505+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:34:40.506419+0000) 2024-03-04T17:34:41.111 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:34:41.119 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:34:41.507 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:34:41.505+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:34:41.506697+0000) 2024-03-04T17:34:42.508 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:34:42.505+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:34:42.506952+0000) 2024-03-04T17:34:43.508 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:34:43.506+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:34:43.507225+0000) 2024-03-04T17:34:44.508 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:34:44.506+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:34:44.507460+0000) 2024-03-04T17:34:45.508 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:34:45.506+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:34:45.507698+0000) 2024-03-04T17:34:46.509 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:34:46.506+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:34:46.507962+0000) 2024-03-04T17:34:47.512 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:34:47.506+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:34:47.508155+0000) 2024-03-04T17:34:48.509 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:34:48.507+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:34:48.508394+0000) 2024-03-04T17:34:49.509 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:34:49.507+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:34:49.508631+0000) 2024-03-04T17:34:50.510 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:34:50.507+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:34:50.508916+0000) 2024-03-04T17:34:51.510 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:34:51.508+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:34:51.509157+0000) 2024-03-04T17:34:52.510 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:34:52.508+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:34:52.509430+0000) 2024-03-04T17:34:53.510 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:34:53.508+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:34:53.509620+0000) 2024-03-04T17:34:54.511 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:34:54.509+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:34:54.509912+0000) 2024-03-04T17:34:55.511 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:34:55.509+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:34:55.510209+0000) 2024-03-04T17:34:56.511 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:34:56.509+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:34:56.510447+0000) 2024-03-04T17:34:57.511 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:34:57.509+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:34:57.510684+0000) 2024-03-04T17:34:58.512 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:34:58.510+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:34:58.510927+0000) 2024-03-04T17:34:59.512 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:34:59.510+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:34:59.511168+0000) 2024-03-04T17:35:00.512 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:35:00.510+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:35:00.511408+0000) 2024-03-04T17:35:01.512 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:35:01.510+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:35:01.511646+0000) 2024-03-04T17:35:02.513 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:35:02.511+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:35:02.511940+0000) 2024-03-04T17:35:03.513 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:35:03.511+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:35:03.512182+0000) 2024-03-04T17:35:04.513 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:35:04.511+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:35:04.512420+0000) 2024-03-04T17:35:05.513 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:35:05.511+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:35:05.512652+0000) 2024-03-04T17:35:06.514 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:35:06.512+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:35:06.512938+0000) 2024-03-04T17:35:07.514 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:35:07.512+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:35:07.513178+0000) 2024-03-04T17:35:08.514 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:35:08.512+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:35:08.513423+0000) 2024-03-04T17:35:09.514 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:35:09.513+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:35:09.513660+0000) 2024-03-04T17:35:10.515 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:35:10.513+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:35:10.513932+0000) 2024-03-04T17:35:11.316 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:35:11.325 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:35:11.515 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:35:11.513+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:35:11.514186+0000) 2024-03-04T17:35:12.515 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:35:12.513+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:35:12.514455+0000) 2024-03-04T17:35:13.515 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:35:13.513+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:35:13.514711+0000) 2024-03-04T17:35:14.516 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:35:14.514+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:35:14.514948+0000) 2024-03-04T17:35:15.516 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:35:15.514+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:35:15.515211+0000) 2024-03-04T17:35:16.516 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:35:16.514+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:35:16.515451+0000) 2024-03-04T17:35:17.516 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:35:17.515+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:35:17.515696+0000) 2024-03-04T17:35:18.517 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:35:18.515+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:35:18.515932+0000) 2024-03-04T17:35:19.517 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:35:19.515+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:35:19.516167+0000) 2024-03-04T17:35:20.517 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:35:20.515+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:35:20.516354+0000) 2024-03-04T17:35:21.517 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:35:21.516+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:35:21.516596+0000) 2024-03-04T17:35:22.520 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:35:22.516+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:35:22.516917+0000) 2024-03-04T17:35:23.518 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:35:23.516+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:35:23.517164+0000) 2024-03-04T17:35:24.518 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:35:24.516+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:35:24.517406+0000) 2024-03-04T17:35:25.518 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:35:25.517+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:35:25.517643+0000) 2024-03-04T17:35:26.519 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:35:26.517+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:35:26.517937+0000) 2024-03-04T17:35:27.519 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:35:27.517+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:35:27.518176+0000) 2024-03-04T17:35:28.519 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:35:28.518+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:35:28.518417+0000) 2024-03-04T17:35:29.520 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:35:29.518+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:35:29.518651+0000) 2024-03-04T17:35:30.520 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:35:30.518+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:35:30.518936+0000) 2024-03-04T17:35:31.520 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:35:31.518+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:35:31.519179+0000) 2024-03-04T17:35:32.520 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:35:32.519+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:35:32.519430+0000) 2024-03-04T17:35:33.521 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:35:33.519+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:35:33.519670+0000) 2024-03-04T17:35:34.521 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:35:34.519+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:35:34.519978+0000) 2024-03-04T17:35:35.521 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:35:35.519+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:35:35.520236+0000) 2024-03-04T17:35:36.521 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:35:36.520+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:35:36.520477+0000) 2024-03-04T17:35:37.522 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:35:37.520+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:35:37.520715+0000) 2024-03-04T17:35:38.522 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:35:38.520+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:35:38.520962+0000) 2024-03-04T17:35:39.522 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:35:39.521+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:35:39.521204+0000) 2024-03-04T17:35:40.522 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:35:40.521+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:35:40.521447+0000) 2024-03-04T17:35:41.488 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:35:41.500 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:35:41.522 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:35:41.520+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:35:41.521683+0000) 2024-03-04T17:35:42.523 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:35:42.521+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:35:42.521924+0000) 2024-03-04T17:35:43.523 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:35:43.521+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:35:43.522169+0000) 2024-03-04T17:35:44.523 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:35:44.520+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:35:44.522360+0000) 2024-03-04T17:35:45.524 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:35:45.520+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:35:45.522615+0000) 2024-03-04T17:35:46.524 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:35:46.521+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:35:46.522916+0000) 2024-03-04T17:35:47.524 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:35:47.522+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:35:47.523163+0000) 2024-03-04T17:35:48.524 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:35:48.522+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:35:48.523409+0000) 2024-03-04T17:35:49.525 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:35:49.522+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:35:49.523702+0000) 2024-03-04T17:35:50.525 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:35:50.522+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:35:50.523976+0000) 2024-03-04T17:35:51.525 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:35:51.523+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:35:51.524212+0000) 2024-03-04T17:35:52.525 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:35:52.523+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:35:52.524463+0000) 2024-03-04T17:35:53.525 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:35:53.523+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:35:53.524702+0000) 2024-03-04T17:35:54.526 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:35:54.524+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:35:54.524950+0000) 2024-03-04T17:35:55.526 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:35:55.524+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:35:55.525193+0000) 2024-03-04T17:35:56.526 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:35:56.524+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:35:56.525423+0000) 2024-03-04T17:35:57.526 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:35:57.524+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:35:57.525662+0000) 2024-03-04T17:35:58.527 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:35:58.525+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:35:58.525940+0000) 2024-03-04T17:35:59.527 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:35:59.525+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:35:59.526185+0000) 2024-03-04T17:36:00.527 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:36:00.525+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:36:00.526382+0000) 2024-03-04T17:36:01.528 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:36:01.525+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:36:01.526627+0000) 2024-03-04T17:36:02.528 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:36:02.526+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:36:02.526931+0000) 2024-03-04T17:36:03.528 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:36:03.526+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:36:03.527176+0000) 2024-03-04T17:36:04.528 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:36:04.526+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:36:04.527415+0000) 2024-03-04T17:36:05.529 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:36:05.526+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:36:05.527661+0000) 2024-03-04T17:36:06.529 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:36:06.527+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:36:06.527943+0000) 2024-03-04T17:36:07.529 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:36:07.527+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:36:07.528182+0000) 2024-03-04T17:36:08.529 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:36:08.527+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:36:08.528432+0000) 2024-03-04T17:36:09.530 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:36:09.527+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:36:09.528667+0000) 2024-03-04T17:36:10.530 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:36:10.528+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:36:10.528956+0000) 2024-03-04T17:36:11.530 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:36:11.528+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:36:11.529213+0000) 2024-03-04T17:36:11.660 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:36:11.668 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:36:12.530 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:36:12.528+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:36:12.529491+0000) 2024-03-04T17:36:13.531 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:36:13.529+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:36:13.529733+0000) 2024-03-04T17:36:14.531 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:36:14.529+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:36:14.529973+0000) 2024-03-04T17:36:15.531 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:36:15.529+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:36:15.530229+0000) 2024-03-04T17:36:16.531 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:36:16.529+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:36:16.530466+0000) 2024-03-04T17:36:17.532 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:36:17.529+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:36:17.530648+0000) 2024-03-04T17:36:18.532 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:36:18.530+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:36:18.530922+0000) 2024-03-04T17:36:19.532 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:36:19.530+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:36:19.531153+0000) 2024-03-04T17:36:20.532 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:36:20.530+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:36:20.531345+0000) 2024-03-04T17:36:21.533 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:36:21.531+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:36:21.531581+0000) 2024-03-04T17:36:22.533 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:36:22.531+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:36:22.531859+0000) 2024-03-04T17:36:23.533 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:36:23.531+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:36:23.532101+0000) 2024-03-04T17:36:24.533 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:36:24.531+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:36:24.532340+0000) 2024-03-04T17:36:25.534 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:36:25.532+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:36:25.532582+0000) 2024-03-04T17:36:26.534 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:36:26.532+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:36:26.532817+0000) 2024-03-04T17:36:27.534 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:36:27.532+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:36:27.533115+0000) 2024-03-04T17:36:28.534 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:36:28.532+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:36:28.533372+0000) 2024-03-04T17:36:29.535 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:36:29.533+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:36:29.533607+0000) 2024-03-04T17:36:30.535 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:36:30.533+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:36:30.533886+0000) 2024-03-04T17:36:31.535 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:36:31.533+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:36:31.534186+0000) 2024-03-04T17:36:32.535 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:36:32.533+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:36:32.534426+0000) 2024-03-04T17:36:33.536 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:36:33.534+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:36:33.534665+0000) 2024-03-04T17:36:34.536 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:36:34.534+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:36:34.534940+0000) 2024-03-04T17:36:35.536 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:36:35.534+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:36:35.535176+0000) 2024-03-04T17:36:36.536 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:36:36.535+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:36:36.535415+0000) 2024-03-04T17:36:37.537 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:36:37.535+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:36:37.535652+0000) 2024-03-04T17:36:38.537 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:36:38.535+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:36:38.535929+0000) 2024-03-04T17:36:39.537 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:36:39.535+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:36:39.536163+0000) 2024-03-04T17:36:40.537 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:36:40.536+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:36:40.536395+0000) 2024-03-04T17:36:41.538 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:36:41.536+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:36:41.536659+0000) 2024-03-04T17:36:41.882 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:36:41.890 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:36:42.538 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:36:42.536+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:36:42.536947+0000) 2024-03-04T17:36:43.538 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:36:43.536+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:36:43.537180+0000) 2024-03-04T17:36:44.539 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:36:44.536+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:36:44.537367+0000) 2024-03-04T17:36:45.539 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:36:45.537+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:36:45.537551+0000) 2024-03-04T17:36:46.539 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:36:46.536+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:36:46.537863+0000) 2024-03-04T17:36:47.539 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:36:47.537+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:36:47.538163+0000) 2024-03-04T17:36:48.539 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:36:48.537+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:36:48.538447+0000) 2024-03-04T17:36:49.540 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:36:49.537+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:36:49.538737+0000) 2024-03-04T17:36:50.540 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:36:50.537+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:36:50.539052+0000) 2024-03-04T17:36:51.541 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:36:51.538+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:36:51.539346+0000) 2024-03-04T17:36:52.541 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:36:52.538+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:36:52.539655+0000) 2024-03-04T17:36:53.541 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:36:53.538+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:36:53.539961+0000) 2024-03-04T17:36:54.541 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:36:54.539+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:36:54.540206+0000) 2024-03-04T17:36:55.541 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:36:55.538+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:36:55.540442+0000) 2024-03-04T17:36:56.542 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:36:56.539+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:36:56.540667+0000) 2024-03-04T17:36:57.542 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:36:57.540+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:36:57.540932+0000) 2024-03-04T17:36:58.542 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:36:58.540+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:36:58.541223+0000) 2024-03-04T17:36:59.542 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:36:59.540+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:36:59.541471+0000) 2024-03-04T17:37:00.543 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:37:00.540+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:37:00.541780+0000) 2024-03-04T17:37:01.543 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:37:01.541+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:37:01.542118+0000) 2024-03-04T17:37:02.543 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:37:02.541+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:37:02.542420+0000) 2024-03-04T17:37:03.544 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:37:03.541+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:37:03.542709+0000) 2024-03-04T17:37:04.544 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:37:04.542+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:37:04.542996+0000) 2024-03-04T17:37:05.544 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:37:05.542+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:37:05.543306+0000) 2024-03-04T17:37:06.545 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:37:06.542+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:37:06.543620+0000) 2024-03-04T17:37:07.545 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:37:07.543+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:37:07.543939+0000) 2024-03-04T17:37:08.545 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:37:08.543+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:37:08.544233+0000) 2024-03-04T17:37:09.546 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:37:09.543+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:37:09.544560+0000) 2024-03-04T17:37:10.546 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:37:10.544+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:37:10.544903+0000) 2024-03-04T17:37:11.546 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:37:11.544+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:37:11.545184+0000) 2024-03-04T17:37:12.037 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:37:12.045 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:37:12.547 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:37:12.544+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:37:12.545515+0000) 2024-03-04T17:37:13.547 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:37:13.545+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:37:13.545872+0000) 2024-03-04T17:37:14.547 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:37:14.545+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:37:14.546112+0000) 2024-03-04T17:37:15.547 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:37:15.545+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:37:15.546401+0000) 2024-03-04T17:37:16.548 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:37:16.546+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:37:16.546696+0000) 2024-03-04T17:37:17.548 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:37:17.546+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:37:17.546985+0000) 2024-03-04T17:37:18.548 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:37:18.546+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:37:18.547301+0000) 2024-03-04T17:37:19.549 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:37:19.546+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:37:19.547636+0000) 2024-03-04T17:37:20.549 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:37:20.547+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:37:20.547957+0000) 2024-03-04T17:37:21.549 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:37:21.547+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:37:21.548251+0000) 2024-03-04T17:37:22.550 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:37:22.547+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:37:22.548555+0000) 2024-03-04T17:37:23.550 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:37:23.548+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:37:23.548896+0000) 2024-03-04T17:37:24.550 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:37:24.548+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:37:24.549201+0000) 2024-03-04T17:37:25.551 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:37:25.548+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:37:25.549494+0000) 2024-03-04T17:37:26.551 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:37:26.549+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:37:26.549785+0000) 2024-03-04T17:37:27.551 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:37:27.549+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:37:27.550120+0000) 2024-03-04T17:37:28.551 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:37:28.549+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:37:28.550337+0000) 2024-03-04T17:37:29.552 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:37:29.549+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:37:29.550597+0000) 2024-03-04T17:37:30.552 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:37:30.550+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:37:30.550903+0000) 2024-03-04T17:37:31.552 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:37:31.550+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:37:31.551144+0000) 2024-03-04T17:37:32.553 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:37:32.551+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:37:32.551391+0000) 2024-03-04T17:37:33.553 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:37:33.551+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:37:33.551638+0000) 2024-03-04T17:37:34.553 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:37:34.551+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:37:34.551941+0000) 2024-03-04T17:37:35.553 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:37:35.551+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:37:35.552202+0000) 2024-03-04T17:37:36.554 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:37:36.552+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:37:36.552462+0000) 2024-03-04T17:37:37.554 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:37:37.552+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:37:37.552707+0000) 2024-03-04T17:37:38.554 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:37:38.552+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:37:38.552958+0000) 2024-03-04T17:37:39.554 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:37:39.552+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:37:39.553203+0000) 2024-03-04T17:37:40.555 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:37:40.553+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:37:40.553441+0000) 2024-03-04T17:37:41.555 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:37:41.553+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:37:41.553682+0000) 2024-03-04T17:37:42.258 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:37:42.266 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:37:42.555 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:37:42.553+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:37:42.553940+0000) 2024-03-04T17:37:43.555 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:37:43.553+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:37:43.554178+0000) 2024-03-04T17:37:44.555 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:37:44.554+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:37:44.554422+0000) 2024-03-04T17:37:45.556 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:37:45.554+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:37:45.554667+0000) 2024-03-04T17:37:46.556 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:37:46.554+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:37:46.554954+0000) 2024-03-04T17:37:47.556 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:37:47.555+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:37:47.555208+0000) 2024-03-04T17:37:48.556 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:37:48.555+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:37:48.555456+0000) 2024-03-04T17:37:49.557 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:37:49.553+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:37:49.555694+0000) 2024-03-04T17:37:50.557 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:37:50.554+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:37:50.555938+0000) 2024-03-04T17:37:51.557 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:37:51.554+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:37:51.556169+0000) 2024-03-04T17:37:52.557 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:37:52.555+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:37:52.556413+0000) 2024-03-04T17:37:53.558 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:37:53.554+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:37:53.556662+0000) 2024-03-04T17:37:54.558 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:37:54.555+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:37:54.556935+0000) 2024-03-04T17:37:55.558 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:37:55.556+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:37:55.557130+0000) 2024-03-04T17:37:56.558 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:37:56.556+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:37:56.557366+0000) 2024-03-04T17:37:57.558 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:37:57.555+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:37:57.557552+0000) 2024-03-04T17:37:58.559 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:37:58.556+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:37:58.557811+0000) 2024-03-04T17:37:59.559 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:37:59.556+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:37:59.558098+0000) 2024-03-04T17:38:00.559 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:38:00.557+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:38:00.558284+0000) 2024-03-04T17:38:01.559 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:38:01.557+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:38:01.558520+0000) 2024-03-04T17:38:02.560 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:38:02.557+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:38:02.558766+0000) 2024-03-04T17:38:03.560 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:38:03.558+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:38:03.558997+0000) 2024-03-04T17:38:04.560 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:38:04.558+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:38:04.559245+0000) 2024-03-04T17:38:05.560 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:38:05.558+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:38:05.559501+0000) 2024-03-04T17:38:06.561 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:38:06.558+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:38:06.559755+0000) 2024-03-04T17:38:07.561 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:38:07.559+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:38:07.560046+0000) 2024-03-04T17:38:08.561 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:38:08.559+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:38:08.560333+0000) 2024-03-04T17:38:09.561 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:38:09.559+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:38:09.560606+0000) 2024-03-04T17:38:10.562 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:38:10.560+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:38:10.560882+0000) 2024-03-04T17:38:11.562 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:38:11.560+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:38:11.561141+0000) 2024-03-04T17:38:12.479 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:38:12.488 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:38:12.562 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:38:12.560+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:38:12.561339+0000) 2024-03-04T17:38:13.562 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:38:13.560+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:38:13.561592+0000) 2024-03-04T17:38:14.563 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:38:14.561+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:38:14.561852+0000) 2024-03-04T17:38:15.563 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:38:15.561+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:38:15.562081+0000) 2024-03-04T17:38:16.563 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:38:16.561+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:38:16.562319+0000) 2024-03-04T17:38:17.563 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:38:17.561+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:38:17.562574+0000) 2024-03-04T17:38:18.564 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:38:18.562+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:38:18.562898+0000) 2024-03-04T17:38:19.564 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:38:19.562+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:38:19.563133+0000) 2024-03-04T17:38:20.564 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:38:20.562+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:38:20.563368+0000) 2024-03-04T17:38:21.565 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:38:21.562+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:38:21.563606+0000) 2024-03-04T17:38:22.565 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:38:22.563+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:38:22.563889+0000) 2024-03-04T17:38:23.565 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:38:23.563+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:38:23.564130+0000) 2024-03-04T17:38:24.565 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:38:24.563+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:38:24.564371+0000) 2024-03-04T17:38:25.566 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:38:25.563+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:38:25.564624+0000) 2024-03-04T17:38:26.566 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:38:26.564+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:38:26.564913+0000) 2024-03-04T17:38:27.566 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:38:27.564+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:38:27.565150+0000) 2024-03-04T17:38:28.566 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:38:28.564+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:38:28.565398+0000) 2024-03-04T17:38:29.566 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:38:29.565+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:38:29.565634+0000) 2024-03-04T17:38:30.567 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:38:30.565+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:38:30.565934+0000) 2024-03-04T17:38:31.567 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:38:31.565+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:38:31.566161+0000) 2024-03-04T17:38:32.567 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:38:32.565+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:38:32.566414+0000) 2024-03-04T17:38:33.568 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:38:33.566+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:38:33.566659+0000) 2024-03-04T17:38:34.568 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:38:34.566+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:38:34.566952+0000) 2024-03-04T17:38:35.568 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:38:35.566+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:38:35.567211+0000) 2024-03-04T17:38:36.568 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:38:36.567+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:38:36.567452+0000) 2024-03-04T17:38:37.568 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:38:37.566+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:38:37.567691+0000) 2024-03-04T17:38:38.569 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:38:38.567+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:38:38.567937+0000) 2024-03-04T17:38:39.569 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:38:39.567+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:38:39.568174+0000) 2024-03-04T17:38:40.569 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:38:40.568+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:38:40.568418+0000) 2024-03-04T17:38:41.570 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:38:41.568+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:38:41.568658+0000) 2024-03-04T17:38:42.570 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:38:42.568+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:38:42.568945+0000) 2024-03-04T17:38:42.618 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:38:42.626 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:38:43.570 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:38:43.568+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:38:43.569191+0000) 2024-03-04T17:38:44.570 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:38:44.569+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:38:44.569431+0000) 2024-03-04T17:38:45.570 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:38:45.569+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:38:45.569670+0000) 2024-03-04T17:38:46.571 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:38:46.569+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:38:46.569944+0000) 2024-03-04T17:38:47.571 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:38:47.570+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:38:47.570200+0000) 2024-03-04T17:38:48.571 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:38:48.570+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:38:48.570446+0000) 2024-03-04T17:38:49.571 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:38:49.570+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:38:49.570677+0000) 2024-03-04T17:38:50.572 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:38:50.570+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:38:50.570946+0000) 2024-03-04T17:38:51.572 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:38:51.571+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:38:51.571186+0000) 2024-03-04T17:38:52.572 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:38:52.569+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:38:52.571449+0000) 2024-03-04T17:38:53.572 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:38:53.570+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:38:53.571719+0000) 2024-03-04T17:38:54.573 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:38:54.570+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:38:54.571958+0000) 2024-03-04T17:38:55.573 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:38:55.571+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:38:55.572203+0000) 2024-03-04T17:38:56.573 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:38:56.571+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:38:56.572444+0000) 2024-03-04T17:38:57.573 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:38:57.571+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:38:57.572684+0000) 2024-03-04T17:38:58.574 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:38:58.571+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:38:58.572954+0000) 2024-03-04T17:38:59.574 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:38:59.571+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:38:59.573150+0000) 2024-03-04T17:39:00.574 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:39:00.572+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:39:00.573389+0000) 2024-03-04T17:39:01.574 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:39:01.572+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:39:01.573630+0000) 2024-03-04T17:39:02.575 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:39:02.572+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:39:02.573929+0000) 2024-03-04T17:39:03.575 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:39:03.573+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:39:03.574167+0000) 2024-03-04T17:39:04.575 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:39:04.573+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:39:04.574405+0000) 2024-03-04T17:39:05.575 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:39:05.573+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:39:05.574664+0000) 2024-03-04T17:39:06.576 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:39:06.574+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:39:06.574986+0000) 2024-03-04T17:39:07.576 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:39:07.574+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:39:07.575225+0000) 2024-03-04T17:39:08.576 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:39:08.574+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:39:08.575463+0000) 2024-03-04T17:39:09.576 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:39:09.574+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:39:09.575648+0000) 2024-03-04T17:39:10.576 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:39:10.575+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:39:10.575929+0000) 2024-03-04T17:39:11.577 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:39:11.575+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:39:11.576168+0000) 2024-03-04T17:39:12.577 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:39:12.575+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:39:12.576424+0000) 2024-03-04T17:39:12.832 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:39:12.840 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:39:13.577 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:39:13.575+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:39:13.576665+0000) 2024-03-04T17:39:14.578 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:39:14.576+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:39:14.576937+0000) 2024-03-04T17:39:15.578 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:39:15.576+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:39:15.577173+0000) 2024-03-04T17:39:16.578 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:39:16.576+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:39:16.577415+0000) 2024-03-04T17:39:17.578 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:39:17.576+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:39:17.577665+0000) 2024-03-04T17:39:18.578 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:39:18.577+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:39:18.577943+0000) 2024-03-04T17:39:19.579 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:39:19.577+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:39:19.578184+0000) 2024-03-04T17:39:20.579 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:39:20.577+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:39:20.578425+0000) 2024-03-04T17:39:21.579 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:39:21.578+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:39:21.578663+0000) 2024-03-04T17:39:22.580 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:39:22.578+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:39:22.578951+0000) 2024-03-04T17:39:23.580 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:39:23.578+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:39:23.579189+0000) 2024-03-04T17:39:24.580 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:39:24.578+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:39:24.579431+0000) 2024-03-04T17:39:25.580 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:39:25.578+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:39:25.579688+0000) 2024-03-04T17:39:26.581 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:39:26.579+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:39:26.579927+0000) 2024-03-04T17:39:27.581 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:39:27.579+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:39:27.580122+0000) 2024-03-04T17:39:28.581 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:39:28.579+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:39:28.580361+0000) 2024-03-04T17:39:29.581 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:39:29.580+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:39:29.580597+0000) 2024-03-04T17:39:30.581 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:39:30.580+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:39:30.580879+0000) 2024-03-04T17:39:31.582 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:39:31.580+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:39:31.581124+0000) 2024-03-04T17:39:32.582 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:39:32.580+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:39:32.581371+0000) 2024-03-04T17:39:33.582 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:39:33.581+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:39:33.581619+0000) 2024-03-04T17:39:34.582 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:39:34.581+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:39:34.581902+0000) 2024-03-04T17:39:35.583 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:39:35.581+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:39:35.582087+0000) 2024-03-04T17:39:36.583 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:39:36.581+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:39:36.582347+0000) 2024-03-04T17:39:37.583 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:39:37.582+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:39:37.582583+0000) 2024-03-04T17:39:38.583 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:39:38.582+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:39:38.582856+0000) 2024-03-04T17:39:39.584 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:39:39.582+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:39:39.583050+0000) 2024-03-04T17:39:40.584 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:39:40.582+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:39:40.583321+0000) 2024-03-04T17:39:41.584 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:39:41.583+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:39:41.583558+0000) 2024-03-04T17:39:42.584 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:39:42.583+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:39:42.583805+0000) 2024-03-04T17:39:42.969 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:39:42.978 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:39:43.585 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:39:43.583+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:39:43.584086+0000) 2024-03-04T17:39:44.585 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:39:44.584+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:39:44.584325+0000) 2024-03-04T17:39:45.585 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:39:45.584+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:39:45.584563+0000) 2024-03-04T17:39:46.585 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:39:46.584+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:39:46.584800+0000) 2024-03-04T17:39:47.585 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:39:47.584+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:39:47.585076+0000) 2024-03-04T17:39:48.586 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:39:48.584+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:39:48.585262+0000) 2024-03-04T17:39:49.586 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:39:49.584+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:39:49.585488+0000) 2024-03-04T17:39:50.586 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:39:50.585+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:39:50.585724+0000) 2024-03-04T17:39:51.586 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:39:51.585+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:39:51.585959+0000) 2024-03-04T17:39:52.587 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:39:52.586+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:39:52.586205+0000) 2024-03-04T17:39:53.587 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:39:53.586+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:39:53.586444+0000) 2024-03-04T17:39:54.587 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:39:54.586+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:39:54.586721+0000) 2024-03-04T17:39:55.587 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:39:55.585+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:39:55.586964+0000) 2024-03-04T17:39:56.588 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:39:56.585+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:39:56.587160+0000) 2024-03-04T17:39:57.588 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:39:57.585+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:39:57.587347+0000) 2024-03-04T17:39:58.588 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:39:58.586+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:39:58.587596+0000) 2024-03-04T17:39:59.588 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:39:59.586+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:39:59.587891+0000) 2024-03-04T17:40:00.589 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:40:00.586+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:40:00.588128+0000) 2024-03-04T17:40:01.589 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:40:01.587+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:40:01.588318+0000) 2024-03-04T17:40:02.589 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:40:02.587+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:40:02.588562+0000) 2024-03-04T17:40:03.589 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:40:03.587+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:40:03.588799+0000) 2024-03-04T17:40:04.590 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:40:04.588+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:40:04.589077+0000) 2024-03-04T17:40:05.590 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:40:05.588+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:40:05.589319+0000) 2024-03-04T17:40:06.590 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:40:06.588+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:40:06.589559+0000) 2024-03-04T17:40:07.590 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:40:07.589+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:40:07.589810+0000) 2024-03-04T17:40:08.590 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:40:08.589+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:40:08.590089+0000) 2024-03-04T17:40:09.591 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:40:09.589+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:40:09.590340+0000) 2024-03-04T17:40:10.591 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:40:10.589+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:40:10.590581+0000) 2024-03-04T17:40:11.591 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:40:11.590+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:40:11.590852+0000) 2024-03-04T17:40:12.591 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:40:12.590+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:40:12.591100+0000) 2024-03-04T17:40:13.183 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:40:13.191 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:40:13.591 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:40:13.590+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:40:13.591348+0000) 2024-03-04T17:40:14.592 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:40:14.590+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:40:14.591580+0000) 2024-03-04T17:40:15.592 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:40:15.591+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:40:15.591817+0000) 2024-03-04T17:40:16.592 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:40:16.591+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:40:16.592086+0000) 2024-03-04T17:40:17.593 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:40:17.591+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:40:17.592326+0000) 2024-03-04T17:40:18.593 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:40:18.591+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:40:18.592569+0000) 2024-03-04T17:40:19.593 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:40:19.592+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:40:19.592854+0000) 2024-03-04T17:40:20.593 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:40:20.592+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:40:20.593053+0000) 2024-03-04T17:40:21.594 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:40:21.592+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:40:21.593311+0000) 2024-03-04T17:40:22.594 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:40:22.592+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:40:22.593553+0000) 2024-03-04T17:40:23.594 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:40:23.593+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:40:23.593787+0000) 2024-03-04T17:40:24.594 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:40:24.593+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:40:24.594060+0000) 2024-03-04T17:40:25.594 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:40:25.593+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:40:25.594299+0000) 2024-03-04T17:40:26.595 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:40:26.593+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:40:26.594497+0000) 2024-03-04T17:40:27.595 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:40:27.594+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:40:27.594750+0000) 2024-03-04T17:40:28.595 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:40:28.594+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:40:28.595009+0000) 2024-03-04T17:40:29.596 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:40:29.594+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:40:29.595295+0000) 2024-03-04T17:40:30.596 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:40:30.594+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:40:30.595551+0000) 2024-03-04T17:40:31.596 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:40:31.595+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:40:31.595806+0000) 2024-03-04T17:40:32.596 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:40:32.595+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:40:32.596129+0000) 2024-03-04T17:40:33.597 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:40:33.595+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:40:33.596380+0000) 2024-03-04T17:40:34.597 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:40:34.596+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:40:34.596631+0000) 2024-03-04T17:40:35.597 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:40:35.596+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:40:35.596936+0000) 2024-03-04T17:40:36.598 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:40:36.596+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:40:36.597203+0000) 2024-03-04T17:40:37.598 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:40:37.597+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:40:37.597480+0000) 2024-03-04T17:40:38.598 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:40:38.597+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:40:38.597729+0000) 2024-03-04T17:40:39.598 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:40:39.597+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:40:39.597972+0000) 2024-03-04T17:40:40.598 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:40:40.597+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:40:40.598226+0000) 2024-03-04T17:40:41.599 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:40:41.598+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:40:41.598476+0000) 2024-03-04T17:40:42.599 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:40:42.598+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:40:42.598794+0000) 2024-03-04T17:40:43.330 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:40:43.340 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:40:43.599 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:40:43.598+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:40:43.599104+0000) 2024-03-04T17:40:44.600 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:40:44.598+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:40:44.599374+0000) 2024-03-04T17:40:45.600 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:40:45.599+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:40:45.599630+0000) 2024-03-04T17:40:46.600 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:40:46.599+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:40:46.599929+0000) 2024-03-04T17:40:47.600 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:40:47.599+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:40:47.600176+0000) 2024-03-04T17:40:48.601 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:40:48.600+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:40:48.600427+0000) 2024-03-04T17:40:49.601 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:40:49.600+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:40:49.600675+0000) 2024-03-04T17:40:50.601 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:40:50.600+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:40:50.600968+0000) 2024-03-04T17:40:51.601 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:40:51.600+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:40:51.601146+0000) 2024-03-04T17:40:52.602 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:40:52.601+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:40:52.601399+0000) 2024-03-04T17:40:53.602 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:40:53.601+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:40:53.601636+0000) 2024-03-04T17:40:54.602 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:40:54.601+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:40:54.601911+0000) 2024-03-04T17:40:55.602 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:40:55.601+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:40:55.602155+0000) 2024-03-04T17:40:56.602 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:40:56.602+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:40:56.602383+0000) 2024-03-04T17:40:57.603 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:40:57.600+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:40:57.602629+0000) 2024-03-04T17:40:58.603 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:40:58.601+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:40:58.602924+0000) 2024-03-04T17:40:59.603 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:40:59.602+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:40:59.603163+0000) 2024-03-04T17:41:00.604 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:41:00.602+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:41:00.603434+0000) 2024-03-04T17:41:01.604 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:41:01.601+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:41:01.603663+0000) 2024-03-04T17:41:02.604 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:41:02.602+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:41:02.603906+0000) 2024-03-04T17:41:03.604 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:41:03.603+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:41:03.604163+0000) 2024-03-04T17:41:04.605 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:41:04.603+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:41:04.604420+0000) 2024-03-04T17:41:05.605 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:41:05.603+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:41:05.604687+0000) 2024-03-04T17:41:06.605 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:41:06.603+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:41:06.604962+0000) 2024-03-04T17:41:07.605 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:41:07.604+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:41:07.605227+0000) 2024-03-04T17:41:08.606 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:41:08.604+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:41:08.605501+0000) 2024-03-04T17:41:09.606 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:41:09.604+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:41:09.605759+0000) 2024-03-04T17:41:10.606 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:41:10.605+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:41:10.606013+0000) 2024-03-04T17:41:11.606 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:41:11.605+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:41:11.606269+0000) 2024-03-04T17:41:12.607 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:41:12.605+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:41:12.606532+0000) 2024-03-04T17:41:13.543 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:41:13.551 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:41:13.607 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:41:13.605+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:41:13.606722+0000) 2024-03-04T17:41:14.607 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:41:14.606+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:41:14.606950+0000) 2024-03-04T17:41:15.607 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:41:15.606+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:41:15.607143+0000) 2024-03-04T17:41:16.607 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:41:16.606+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:41:16.607378+0000) 2024-03-04T17:41:17.608 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:41:17.606+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:41:17.607618+0000) 2024-03-04T17:41:18.608 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:41:18.607+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:41:18.607922+0000) 2024-03-04T17:41:19.608 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:41:19.607+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:41:19.608172+0000) 2024-03-04T17:41:20.609 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:41:20.607+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:41:20.608410+0000) 2024-03-04T17:41:21.609 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:41:21.607+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:41:21.608641+0000) 2024-03-04T17:41:22.609 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:41:22.608+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:41:22.608918+0000) 2024-03-04T17:41:23.609 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:41:23.608+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:41:23.609158+0000) 2024-03-04T17:41:24.609 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:41:24.608+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:41:24.609398+0000) 2024-03-04T17:41:25.610 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:41:25.608+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:41:25.609640+0000) 2024-03-04T17:41:26.610 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:41:26.609+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:41:26.609926+0000) 2024-03-04T17:41:27.610 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:41:27.609+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:41:27.610106+0000) 2024-03-04T17:41:28.610 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:41:28.609+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:41:28.610352+0000) 2024-03-04T17:41:29.611 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:41:29.609+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:41:29.610599+0000) 2024-03-04T17:41:30.611 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:41:30.610+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:41:30.610879+0000) 2024-03-04T17:41:31.611 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:41:31.610+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:41:31.611125+0000) 2024-03-04T17:41:32.611 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:41:32.610+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:41:32.611320+0000) 2024-03-04T17:41:33.612 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:41:33.611+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:41:33.611556+0000) 2024-03-04T17:41:34.612 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:41:34.611+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:41:34.611794+0000) 2024-03-04T17:41:35.612 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:41:35.611+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:41:35.612076+0000) 2024-03-04T17:41:36.612 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:41:36.611+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:41:36.612371+0000) 2024-03-04T17:41:37.613 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:41:37.612+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:41:37.612647+0000) 2024-03-04T17:41:38.613 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:41:38.612+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:41:38.612925+0000) 2024-03-04T17:41:39.613 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:41:39.612+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:41:39.613163+0000) 2024-03-04T17:41:40.613 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:41:40.612+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:41:40.613410+0000) 2024-03-04T17:41:41.614 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:41:41.613+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:41:41.613646+0000) 2024-03-04T17:41:42.614 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:41:42.613+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:41:42.613934+0000) 2024-03-04T17:41:43.614 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:41:43.613+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:41:43.614174+0000) 2024-03-04T17:41:43.689 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:41:43.697 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:41:44.614 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:41:44.613+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:41:44.614368+0000) 2024-03-04T17:41:45.615 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:41:45.614+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:41:45.614608+0000) 2024-03-04T17:41:46.615 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:41:46.614+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:41:46.614886+0000) 2024-03-04T17:41:47.615 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:41:47.614+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:41:47.615080+0000) 2024-03-04T17:41:48.615 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:41:48.615+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:41:48.615326+0000) 2024-03-04T17:41:49.616 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:41:49.615+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:41:49.615572+0000) 2024-03-04T17:41:50.616 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:41:50.615+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:41:50.615809+0000) 2024-03-04T17:41:51.616 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:41:51.615+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:41:51.616094+0000) 2024-03-04T17:41:52.616 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:41:52.616+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:41:52.616349+0000) 2024-03-04T17:41:53.617 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:41:53.616+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:41:53.616585+0000) 2024-03-04T17:41:54.617 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:41:54.616+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:41:54.616853+0000) 2024-03-04T17:41:55.617 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:41:55.616+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:41:55.617096+0000) 2024-03-04T17:41:56.617 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:41:56.617+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:41:56.617356+0000) 2024-03-04T17:41:57.618 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:41:57.616+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:41:57.617609+0000) 2024-03-04T17:41:58.618 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:41:58.617+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:41:58.617893+0000) 2024-03-04T17:41:59.618 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:41:59.617+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:41:59.618083+0000) 2024-03-04T17:42:00.618 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:42:00.617+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:42:00.618329+0000) 2024-03-04T17:42:01.619 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:42:01.617+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:42:01.618568+0000) 2024-03-04T17:42:02.619 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:42:02.617+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:42:02.618818+0000) 2024-03-04T17:42:03.619 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:42:03.618+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:42:03.619090+0000) 2024-03-04T17:42:04.619 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:42:04.618+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:42:04.619338+0000) 2024-03-04T17:42:05.620 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:42:05.618+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:42:05.619586+0000) 2024-03-04T17:42:06.620 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:42:06.618+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:42:06.619882+0000) 2024-03-04T17:42:07.620 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:42:07.619+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:42:07.620131+0000) 2024-03-04T17:42:08.620 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:42:08.619+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:42:08.620376+0000) 2024-03-04T17:42:09.621 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:42:09.619+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:42:09.620646+0000) 2024-03-04T17:42:10.621 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:42:10.619+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:42:10.620894+0000) 2024-03-04T17:42:11.621 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:42:11.620+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:42:11.621176+0000) 2024-03-04T17:42:12.621 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:42:12.620+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:42:12.621393+0000) 2024-03-04T17:42:13.621 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:42:13.620+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:42:13.621645+0000) 2024-03-04T17:42:13.894 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:42:13.902 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:42:14.622 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:42:14.621+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:42:14.621952+0000) 2024-03-04T17:42:15.622 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:42:15.621+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:42:15.622226+0000) 2024-03-04T17:42:16.622 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:42:16.621+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:42:16.622484+0000) 2024-03-04T17:42:17.623 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:42:17.621+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:42:17.622736+0000) 2024-03-04T17:42:18.623 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:42:18.622+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:42:18.623016+0000) 2024-03-04T17:42:19.623 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:42:19.622+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:42:19.623279+0000) 2024-03-04T17:42:20.623 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:42:20.622+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:42:20.623547+0000) 2024-03-04T17:42:21.624 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:42:21.623+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:42:21.623857+0000) 2024-03-04T17:42:22.624 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:42:22.623+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:42:22.624165+0000) 2024-03-04T17:42:23.624 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:42:23.623+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:42:23.624363+0000) 2024-03-04T17:42:24.624 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:42:24.623+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:42:24.624587+0000) 2024-03-04T17:42:25.625 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:42:25.624+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:42:25.624861+0000) 2024-03-04T17:42:26.625 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:42:26.624+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:42:26.625098+0000) 2024-03-04T17:42:27.625 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:42:27.624+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:42:27.625278+0000) 2024-03-04T17:42:28.625 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:42:28.624+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:42:28.625516+0000) 2024-03-04T17:42:29.626 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:42:29.624+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:42:29.625757+0000) 2024-03-04T17:42:30.626 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:42:30.625+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:42:30.626012+0000) 2024-03-04T17:42:31.626 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:42:31.625+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:42:31.626249+0000) 2024-03-04T17:42:32.626 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:42:32.625+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:42:32.626494+0000) 2024-03-04T17:42:33.627 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:42:33.626+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:42:33.626738+0000) 2024-03-04T17:42:34.627 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:42:34.626+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:42:34.626989+0000) 2024-03-04T17:42:35.627 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:42:35.626+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:42:35.627240+0000) 2024-03-04T17:42:36.627 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:42:36.626+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:42:36.627496+0000) 2024-03-04T17:42:37.627 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:42:37.627+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:42:37.627753+0000) 2024-03-04T17:42:38.628 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:42:38.627+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:42:38.627999+0000) 2024-03-04T17:42:39.628 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:42:39.627+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:42:39.628286+0000) 2024-03-04T17:42:40.628 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:42:40.628+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:42:40.628533+0000) 2024-03-04T17:42:41.629 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:42:41.628+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:42:41.628769+0000) 2024-03-04T17:42:42.629 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:42:42.628+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:42:42.629014+0000) 2024-03-04T17:42:43.629 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:42:43.628+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:42:43.629253+0000) 2024-03-04T17:42:44.099 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:42:44.107 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:42:44.629 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:42:44.629+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:42:44.629490+0000) 2024-03-04T17:42:45.630 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:42:45.629+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:42:45.629746+0000) 2024-03-04T17:42:46.630 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:42:46.629+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:42:46.629989+0000) 2024-03-04T17:42:47.630 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:42:47.629+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:42:47.630227+0000) 2024-03-04T17:42:48.630 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:42:48.630+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:42:48.630465+0000) 2024-03-04T17:42:49.631 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:42:49.629+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:42:49.630694+0000) 2024-03-04T17:42:50.631 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:42:50.630+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:42:50.630979+0000) 2024-03-04T17:42:51.631 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:42:51.630+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:42:51.631215+0000) 2024-03-04T17:42:52.631 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:42:52.630+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:42:52.631417+0000) 2024-03-04T17:42:53.631 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:42:53.631+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:42:53.631656+0000) 2024-03-04T17:42:54.632 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:42:54.631+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:42:54.631937+0000) 2024-03-04T17:42:55.632 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:42:55.631+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:42:55.632177+0000) 2024-03-04T17:42:56.632 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:42:56.631+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:42:56.632412+0000) 2024-03-04T17:42:57.632 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:42:57.632+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:42:57.632643+0000) 2024-03-04T17:42:58.633 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:42:58.632+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:42:58.632932+0000) 2024-03-04T17:42:59.633 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:42:59.632+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:42:59.633168+0000) 2024-03-04T17:43:00.633 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:43:00.632+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:43:00.633415+0000) 2024-03-04T17:43:01.634 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:43:01.633+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:43:01.633653+0000) 2024-03-04T17:43:02.634 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:43:02.633+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:43:02.633868+0000) 2024-03-04T17:43:03.634 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:43:03.633+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:43:03.634101+0000) 2024-03-04T17:43:04.634 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:43:04.633+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:43:04.634352+0000) 2024-03-04T17:43:05.634 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:43:05.633+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:43:05.634592+0000) 2024-03-04T17:43:06.635 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:43:06.633+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:43:06.634858+0000) 2024-03-04T17:43:07.635 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:43:07.633+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:43:07.635098+0000) 2024-03-04T17:43:08.635 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:43:08.634+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:43:08.635356+0000) 2024-03-04T17:43:09.635 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:43:09.634+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:43:09.635593+0000) 2024-03-04T17:43:10.636 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:43:10.634+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:43:10.635870+0000) 2024-03-04T17:43:11.636 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:43:11.634+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:43:11.636049+0000) 2024-03-04T17:43:12.636 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:43:12.635+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:43:12.636292+0000) 2024-03-04T17:43:13.636 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:43:13.635+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:43:13.636532+0000) 2024-03-04T17:43:14.329 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:43:14.337 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:43:14.636 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:43:14.635+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:43:14.636779+0000) 2024-03-04T17:43:15.637 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:43:15.636+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:43:15.637040+0000) 2024-03-04T17:43:16.637 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:43:16.636+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:43:16.637278+0000) 2024-03-04T17:43:17.637 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:43:17.636+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:43:17.637563+0000) 2024-03-04T17:43:18.638 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:43:18.637+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:43:18.637809+0000) 2024-03-04T17:43:19.638 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:43:19.637+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:43:19.638082+0000) 2024-03-04T17:43:20.638 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:43:20.637+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:43:20.638324+0000) 2024-03-04T17:43:21.638 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:43:21.637+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:43:21.638559+0000) 2024-03-04T17:43:22.638 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:43:22.638+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:43:22.638813+0000) 2024-03-04T17:43:23.639 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:43:23.638+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:43:23.639089+0000) 2024-03-04T17:43:24.639 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:43:24.638+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:43:24.639324+0000) 2024-03-04T17:43:25.639 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:43:25.638+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:43:25.639581+0000) 2024-03-04T17:43:26.639 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:43:26.639+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:43:26.639818+0000) 2024-03-04T17:43:27.640 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:43:27.639+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:43:27.640089+0000) 2024-03-04T17:43:28.640 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:43:28.639+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:43:28.640330+0000) 2024-03-04T17:43:29.640 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:43:29.639+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:43:29.640563+0000) 2024-03-04T17:43:30.640 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:43:30.640+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:43:30.640813+0000) 2024-03-04T17:43:31.641 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:43:31.640+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:43:31.641090+0000) 2024-03-04T17:43:32.641 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:43:32.640+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:43:32.641336+0000) 2024-03-04T17:43:33.641 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:43:33.640+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:43:33.641590+0000) 2024-03-04T17:43:34.642 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:43:34.641+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:43:34.641911+0000) 2024-03-04T17:43:35.642 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:43:35.641+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:43:35.642161+0000) 2024-03-04T17:43:36.642 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:43:36.641+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:43:36.642414+0000) 2024-03-04T17:43:37.642 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:43:37.642+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:43:37.642683+0000) 2024-03-04T17:43:38.643 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:43:38.642+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:43:38.642960+0000) 2024-03-04T17:43:39.643 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:43:39.642+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:43:39.643164+0000) 2024-03-04T17:43:40.643 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:43:40.642+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:43:40.643431+0000) 2024-03-04T17:43:41.644 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:43:41.643+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:43:41.643688+0000) 2024-03-04T17:43:42.644 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:43:42.643+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:43:42.643950+0000) 2024-03-04T17:43:43.644 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:43:43.643+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:43:43.644181+0000) 2024-03-04T17:43:44.558 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:43:44.567 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:43:44.644 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:43:44.643+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:43:44.644433+0000) 2024-03-04T17:43:45.644 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:43:45.644+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:43:45.644672+0000) 2024-03-04T17:43:46.645 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:43:46.644+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:43:46.644934+0000) 2024-03-04T17:43:47.645 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:43:47.644+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:43:47.645183+0000) 2024-03-04T17:43:48.645 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:43:48.645+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:43:48.645440+0000) 2024-03-04T17:43:49.645 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:43:49.644+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:43:49.645687+0000) 2024-03-04T17:43:50.646 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:43:50.645+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:43:50.645947+0000) 2024-03-04T17:43:51.646 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:43:51.645+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:43:51.646152+0000) 2024-03-04T17:43:52.646 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:43:52.646+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:43:52.646437+0000) 2024-03-04T17:43:53.646 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:43:53.646+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:43:53.646692+0000) 2024-03-04T17:43:54.647 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:43:54.646+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:43:54.646991+0000) 2024-03-04T17:43:55.647 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:43:55.646+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:43:55.647250+0000) 2024-03-04T17:43:56.647 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:43:56.647+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:43:56.647504+0000) 2024-03-04T17:43:57.647 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:43:57.647+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:43:57.647753+0000) 2024-03-04T17:43:58.648 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:43:58.647+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:43:58.648028+0000) 2024-03-04T17:43:59.648 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:43:59.648+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:43:59.648352+0000) 2024-03-04T17:44:00.648 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:44:00.648+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:44:00.648596+0000) 2024-03-04T17:44:01.648 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:44:01.648+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:44:01.648899+0000) 2024-03-04T17:44:02.649 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:44:02.648+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:44:02.649162+0000) 2024-03-04T17:44:03.649 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:44:03.648+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:44:03.649370+0000) 2024-03-04T17:44:04.649 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:44:04.649+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:44:04.649618+0000) 2024-03-04T17:44:05.649 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:44:05.649+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:44:05.649905+0000) 2024-03-04T17:44:06.650 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:44:06.650+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:44:06.650186+0000) 2024-03-04T17:44:07.650 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:44:07.648+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:44:07.650451+0000) 2024-03-04T17:44:08.650 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:44:08.649+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:44:08.650711+0000) 2024-03-04T17:44:09.651 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:44:09.649+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:44:09.650959+0000) 2024-03-04T17:44:10.651 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:44:10.650+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:44:10.651211+0000) 2024-03-04T17:44:11.651 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:44:11.650+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:44:11.651461+0000) 2024-03-04T17:44:12.651 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:44:12.650+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:44:12.651745+0000) 2024-03-04T17:44:13.652 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:44:13.650+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:44:13.652001+0000) 2024-03-04T17:44:14.652 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:44:14.651+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:44:14.652257+0000) 2024-03-04T17:44:14.706 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:44:14.714 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:44:15.652 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:44:15.651+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:44:15.652515+0000) 2024-03-04T17:44:16.652 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:44:16.651+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:44:16.652752+0000) 2024-03-04T17:44:17.652 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:44:17.652+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:44:17.652984+0000) 2024-03-04T17:44:18.653 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:44:18.652+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:44:18.653183+0000) 2024-03-04T17:44:19.653 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:44:19.652+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:44:19.653417+0000) 2024-03-04T17:44:20.653 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:44:20.652+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:44:20.653683+0000) 2024-03-04T17:44:21.654 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:44:21.653+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:44:21.653921+0000) 2024-03-04T17:44:22.654 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:44:22.653+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:44:22.654168+0000) 2024-03-04T17:44:23.654 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:44:23.653+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:44:23.654406+0000) 2024-03-04T17:44:24.654 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:44:24.653+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:44:24.654647+0000) 2024-03-04T17:44:25.654 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:44:25.654+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:44:25.654936+0000) 2024-03-04T17:44:26.655 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:44:26.654+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:44:26.655178+0000) 2024-03-04T17:44:27.655 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:44:27.654+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:44:27.655470+0000) 2024-03-04T17:44:28.655 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:44:28.654+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:44:28.655787+0000) 2024-03-04T17:44:29.656 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:44:29.655+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:44:29.656068+0000) 2024-03-04T17:44:30.656 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:44:30.655+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:44:30.656313+0000) 2024-03-04T17:44:31.656 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:44:31.655+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:44:31.656570+0000) 2024-03-04T17:44:32.656 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:44:32.656+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:44:32.656814+0000) 2024-03-04T17:44:33.657 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:44:33.656+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:44:33.657081+0000) 2024-03-04T17:44:34.657 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:44:34.656+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:44:34.657319+0000) 2024-03-04T17:44:35.657 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:44:35.656+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:44:35.657572+0000) 2024-03-04T17:44:36.657 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:44:36.657+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:44:36.657887+0000) 2024-03-04T17:44:37.658 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:44:37.657+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:44:37.658121+0000) 2024-03-04T17:44:38.659 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:44:38.657+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:44:38.658324+0000) 2024-03-04T17:44:39.658 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:44:39.657+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:44:39.658581+0000) 2024-03-04T17:44:40.658 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:44:40.658+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:44:40.658867+0000) 2024-03-04T17:44:41.659 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:44:41.658+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:44:41.659102+0000) 2024-03-04T17:44:42.659 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:44:42.658+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:44:42.659351+0000) 2024-03-04T17:44:43.659 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:44:43.658+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:44:43.659611+0000) 2024-03-04T17:44:44.659 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:44:44.659+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:44:44.659900+0000) 2024-03-04T17:44:44.910 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:44:44.918 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:44:45.660 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:44:45.659+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:44:45.660159+0000) 2024-03-04T17:44:46.660 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:44:46.659+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:44:46.660414+0000) 2024-03-04T17:44:47.660 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:44:47.660+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:44:47.660669+0000) 2024-03-04T17:44:48.661 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:44:48.660+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:44:48.660956+0000) 2024-03-04T17:44:49.661 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:44:49.660+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:44:49.661204+0000) 2024-03-04T17:44:50.661 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:44:50.661+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:44:50.661456+0000) 2024-03-04T17:44:51.661 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:44:51.661+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:44:51.661718+0000) 2024-03-04T17:44:52.661 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:44:52.661+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:44:52.661980+0000) 2024-03-04T17:44:53.662 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:44:53.661+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:44:53.662292+0000) 2024-03-04T17:44:54.662 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:44:54.662+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:44:54.662544+0000) 2024-03-04T17:44:55.662 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:44:55.662+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:44:55.662794+0000) 2024-03-04T17:44:56.663 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:44:56.662+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:44:56.663098+0000) 2024-03-04T17:44:57.663 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:44:57.663+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:44:57.663346+0000) 2024-03-04T17:44:58.663 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:44:58.663+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:44:58.663609+0000) 2024-03-04T17:44:59.663 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:44:59.663+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:44:59.663888+0000) 2024-03-04T17:45:00.664 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:45:00.663+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:45:00.664173+0000) 2024-03-04T17:45:01.664 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:45:01.663+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:45:01.664442+0000) 2024-03-04T17:45:02.664 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:45:02.664+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:45:02.664707+0000) 2024-03-04T17:45:03.664 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:45:03.664+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:45:03.664948+0000) 2024-03-04T17:45:04.665 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:45:04.664+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:45:04.665184+0000) 2024-03-04T17:45:05.665 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:45:05.665+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:45:05.665445+0000) 2024-03-04T17:45:06.665 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:45:06.664+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:45:06.665709+0000) 2024-03-04T17:45:07.666 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:45:07.665+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:45:07.665945+0000) 2024-03-04T17:45:08.666 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:45:08.666+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:45:08.666185+0000) 2024-03-04T17:45:09.666 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:45:09.666+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:45:09.666420+0000) 2024-03-04T17:45:10.666 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:45:10.665+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:45:10.666665+0000) 2024-03-04T17:45:11.666 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:45:11.665+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:45:11.666944+0000) 2024-03-04T17:45:12.667 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:45:12.666+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:45:12.667218+0000) 2024-03-04T17:45:13.667 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:45:13.666+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:45:13.667470+0000) 2024-03-04T17:45:14.667 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:45:14.666+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:45:14.667716+0000) 2024-03-04T17:45:15.115 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:45:15.123 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:45:15.667 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:45:15.666+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:45:15.667988+0000) 2024-03-04T17:45:16.668 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:45:16.666+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:45:16.668197+0000) 2024-03-04T17:45:17.668 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:45:17.667+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:45:17.668475+0000) 2024-03-04T17:45:18.668 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:45:18.667+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:45:18.668769+0000) 2024-03-04T17:45:19.668 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:45:19.668+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:45:19.669020+0000) 2024-03-04T17:45:20.669 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:45:20.668+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:45:20.669275+0000) 2024-03-04T17:45:21.669 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:45:21.668+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:45:21.669537+0000) 2024-03-04T17:45:22.669 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:45:22.669+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:45:22.669886+0000) 2024-03-04T17:45:23.670 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:45:23.669+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:45:23.670142+0000) 2024-03-04T17:45:24.670 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:45:24.669+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:45:24.670410+0000) 2024-03-04T17:45:25.670 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:45:25.669+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:45:25.670672+0000) 2024-03-04T17:45:26.670 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:45:26.670+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:45:26.670954+0000) 2024-03-04T17:45:27.671 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:45:27.670+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:45:27.671210+0000) 2024-03-04T17:45:28.671 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:45:28.670+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:45:28.671490+0000) 2024-03-04T17:45:29.671 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:45:29.670+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:45:29.671740+0000) 2024-03-04T17:45:30.671 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:45:30.671+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:45:30.672019+0000) 2024-03-04T17:45:31.672 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:45:31.671+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:45:31.672232+0000) 2024-03-04T17:45:32.672 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:45:32.671+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:45:32.672502+0000) 2024-03-04T17:45:33.672 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:45:33.671+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:45:33.672749+0000) 2024-03-04T17:45:34.672 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:45:34.672+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:45:34.673018+0000) 2024-03-04T17:45:35.673 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:45:35.672+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:45:35.673281+0000) 2024-03-04T17:45:36.673 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:45:36.672+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:45:36.673520+0000) 2024-03-04T17:45:37.673 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:45:37.672+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:45:37.673785+0000) 2024-03-04T17:45:38.673 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:45:38.673+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:45:38.674076+0000) 2024-03-04T17:45:39.674 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:45:39.673+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:45:39.674277+0000) 2024-03-04T17:45:40.674 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:45:40.673+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:45:40.674514+0000) 2024-03-04T17:45:41.674 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:45:41.673+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:45:41.674786+0000) 2024-03-04T17:45:42.674 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:45:42.674+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:45:42.675073+0000) 2024-03-04T17:45:43.675 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:45:43.674+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:45:43.675311+0000) 2024-03-04T17:45:44.675 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:45:44.674+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:45:44.675548+0000) 2024-03-04T17:45:45.297 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:45:45.305 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:45:45.675 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:45:45.675+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:45:45.675788+0000) 2024-03-04T17:45:46.675 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:45:46.675+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:45:46.676093+0000) 2024-03-04T17:45:47.676 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:45:47.675+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:45:47.676331+0000) 2024-03-04T17:45:48.676 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:45:48.676+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:45:48.676574+0000) 2024-03-04T17:45:49.676 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:45:49.676+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:45:49.676813+0000) 2024-03-04T17:45:50.676 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:45:50.676+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:45:50.677044+0000) 2024-03-04T17:45:51.677 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:45:51.676+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:45:51.677285+0000) 2024-03-04T17:45:52.677 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:45:52.677+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:45:52.677531+0000) 2024-03-04T17:45:53.677 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:45:53.677+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:45:53.677769+0000) 2024-03-04T17:45:54.677 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:45:54.677+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:45:54.678007+0000) 2024-03-04T17:45:55.678 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:45:55.677+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:45:55.678249+0000) 2024-03-04T17:45:56.678 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:45:56.678+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:45:56.678487+0000) 2024-03-04T17:45:57.678 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:45:57.678+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:45:57.678726+0000) 2024-03-04T17:45:58.678 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:45:58.678+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:45:58.678968+0000) 2024-03-04T17:45:59.679 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:45:59.678+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:45:59.679199+0000) 2024-03-04T17:46:00.679 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:46:00.678+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:46:00.679391+0000) 2024-03-04T17:46:01.679 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:46:01.679+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:46:01.679638+0000) 2024-03-04T17:46:02.679 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:46:02.679+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:46:02.679932+0000) 2024-03-04T17:46:03.679 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:46:03.679+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:46:03.680174+0000) 2024-03-04T17:46:04.680 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:46:04.680+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:46:04.680428+0000) 2024-03-04T17:46:05.680 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:46:05.680+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:46:05.680696+0000) 2024-03-04T17:46:06.680 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:46:06.680+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:46:06.680935+0000) 2024-03-04T17:46:07.680 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:46:07.680+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:46:07.681174+0000) 2024-03-04T17:46:08.681 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:46:08.681+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:46:08.681411+0000) 2024-03-04T17:46:09.681 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:46:09.681+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:46:09.681649+0000) 2024-03-04T17:46:10.681 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:46:10.681+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:46:10.681935+0000) 2024-03-04T17:46:11.681 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:46:11.682+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:46:11.682174+0000) 2024-03-04T17:46:12.682 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:46:12.682+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:46:12.682419+0000) 2024-03-04T17:46:13.682 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:46:13.681+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:46:13.682659+0000) 2024-03-04T17:46:14.682 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:46:14.681+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:46:14.682935+0000) 2024-03-04T17:46:15.516 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:46:15.524 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:46:15.682 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:46:15.682+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:46:15.683194+0000) 2024-03-04T17:46:16.683 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:46:16.682+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:46:16.683439+0000) 2024-03-04T17:46:17.683 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:46:17.682+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:46:17.683675+0000) 2024-03-04T17:46:18.683 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:46:18.682+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:46:18.683956+0000) 2024-03-04T17:46:19.683 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:46:19.682+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:46:19.684144+0000) 2024-03-04T17:46:20.684 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:46:20.682+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:46:20.684380+0000) 2024-03-04T17:46:21.684 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:46:21.683+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:46:21.684623+0000) 2024-03-04T17:46:22.684 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:46:22.683+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:46:22.684897+0000) 2024-03-04T17:46:23.684 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:46:23.684+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:46:23.685093+0000) 2024-03-04T17:46:24.685 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:46:24.684+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:46:24.685328+0000) 2024-03-04T17:46:25.685 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:46:25.684+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:46:25.685598+0000) 2024-03-04T17:46:26.685 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:46:26.685+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:46:26.685866+0000) 2024-03-04T17:46:27.685 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:46:27.685+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:46:27.686113+0000) 2024-03-04T17:46:28.686 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:46:28.685+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:46:28.686377+0000) 2024-03-04T17:46:29.686 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:46:29.685+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:46:29.686635+0000) 2024-03-04T17:46:30.686 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:46:30.686+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:46:30.686940+0000) 2024-03-04T17:46:31.686 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:46:31.686+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:46:31.687121+0000) 2024-03-04T17:46:32.687 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:46:32.686+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:46:32.687376+0000) 2024-03-04T17:46:33.687 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:46:33.686+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:46:33.687631+0000) 2024-03-04T17:46:34.687 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:46:34.687+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:46:34.687961+0000) 2024-03-04T17:46:35.687 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:46:35.687+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:46:35.688238+0000) 2024-03-04T17:46:36.688 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:46:36.687+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:46:36.688494+0000) 2024-03-04T17:46:37.688 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:46:37.688+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:46:37.688753+0000) 2024-03-04T17:46:38.688 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:46:38.688+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:46:38.689005+0000) 2024-03-04T17:46:39.688 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:46:39.688+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:46:39.689315+0000) 2024-03-04T17:46:40.689 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:46:40.688+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:46:40.689582+0000) 2024-03-04T17:46:41.689 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:46:41.689+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:46:41.689913+0000) 2024-03-04T17:46:42.690 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:46:42.689+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:46:42.690186+0000) 2024-03-04T17:46:43.690 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:46:43.689+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:46:43.690442+0000) 2024-03-04T17:46:44.690 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:46:44.689+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:46:44.690709+0000) 2024-03-04T17:46:45.650 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:46:45.662 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:46:45.690 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:46:45.690+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:46:45.690962+0000) 2024-03-04T17:46:46.690 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:46:46.690+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:46:46.691222+0000) 2024-03-04T17:46:47.691 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:46:47.690+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:46:47.691490+0000) 2024-03-04T17:46:48.691 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:46:48.691+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:46:48.691734+0000) 2024-03-04T17:46:49.692 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:46:49.691+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:46:49.691978+0000) 2024-03-04T17:46:50.692 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:46:50.691+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:46:50.692247+0000) 2024-03-04T17:46:51.692 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:46:51.691+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:46:51.692493+0000) 2024-03-04T17:46:52.692 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:46:52.692+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:46:52.692746+0000) 2024-03-04T17:46:53.692 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:46:53.692+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:46:53.693007+0000) 2024-03-04T17:46:54.695 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:46:54.692+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:46:54.693259+0000) 2024-03-04T17:46:55.693 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:46:55.693+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:46:55.693527+0000) 2024-03-04T17:46:56.693 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:46:56.693+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:46:56.693790+0000) 2024-03-04T17:46:57.693 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:46:57.693+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:46:57.694121+0000) 2024-03-04T17:46:58.694 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:46:58.693+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:46:58.694379+0000) 2024-03-04T17:46:59.694 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:46:59.694+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:46:59.694640+0000) 2024-03-04T17:47:00.694 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:47:00.694+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:47:00.694916+0000) 2024-03-04T17:47:01.694 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:47:01.694+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:47:01.695164+0000) 2024-03-04T17:47:02.695 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:47:02.695+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:47:02.695436+0000) 2024-03-04T17:47:03.695 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:47:03.695+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:47:03.695714+0000) 2024-03-04T17:47:04.695 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:47:04.695+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:47:04.695969+0000) 2024-03-04T17:47:05.696 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:47:05.695+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:47:05.696294+0000) 2024-03-04T17:47:06.696 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:47:06.696+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:47:06.696563+0000) 2024-03-04T17:47:07.696 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:47:07.696+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:47:07.696864+0000) 2024-03-04T17:47:08.696 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:47:08.696+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:47:08.697119+0000) 2024-03-04T17:47:09.697 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:47:09.697+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:47:09.697372+0000) 2024-03-04T17:47:10.697 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:47:10.697+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:47:10.697658+0000) 2024-03-04T17:47:11.697 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:47:11.697+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:47:11.697968+0000) 2024-03-04T17:47:12.697 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:47:12.697+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:47:12.698193+0000) 2024-03-04T17:47:13.698 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:47:13.698+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:47:13.698440+0000) 2024-03-04T17:47:14.698 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:47:14.698+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:47:14.698691+0000) 2024-03-04T17:47:15.698 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:47:15.698+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:47:15.698964+0000) 2024-03-04T17:47:15.927 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:47:15.935 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:47:16.698 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:47:16.698+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:47:16.699231+0000) 2024-03-04T17:47:17.699 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:47:17.698+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:47:17.699489+0000) 2024-03-04T17:47:18.699 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:47:18.698+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:47:18.699761+0000) 2024-03-04T17:47:19.699 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:47:19.698+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:47:19.700002+0000) 2024-03-04T17:47:20.699 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:47:20.699+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:47:20.700259+0000) 2024-03-04T17:47:21.700 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:47:21.699+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:47:21.700515+0000) 2024-03-04T17:47:22.700 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:47:22.699+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:47:22.700738+0000) 2024-03-04T17:47:23.700 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:47:23.699+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:47:23.700988+0000) 2024-03-04T17:47:24.700 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:47:24.699+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:47:24.701247+0000) 2024-03-04T17:47:25.701 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:47:25.700+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:47:25.701502+0000) 2024-03-04T17:47:26.701 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:47:26.700+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:47:26.701753+0000) 2024-03-04T17:47:27.701 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:47:27.701+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:47:27.702053+0000) 2024-03-04T17:47:28.701 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:47:28.701+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:47:28.702296+0000) 2024-03-04T17:47:29.702 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:47:29.701+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:47:29.702474+0000) 2024-03-04T17:47:30.702 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:47:30.701+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:47:30.702652+0000) 2024-03-04T17:47:31.702 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:47:31.702+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:47:31.702854+0000) 2024-03-04T17:47:32.702 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:47:32.702+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:47:32.703043+0000) 2024-03-04T17:47:33.702 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:47:33.702+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:47:33.703221+0000) 2024-03-04T17:47:34.703 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:47:34.702+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:47:34.703393+0000) 2024-03-04T17:47:35.703 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:47:35.702+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:47:35.703568+0000) 2024-03-04T17:47:36.703 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:47:36.702+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:47:36.703765+0000) 2024-03-04T17:47:37.703 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:47:37.703+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:47:37.703936+0000) 2024-03-04T17:47:38.703 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:47:38.703+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:47:38.704125+0000) 2024-03-04T17:47:39.703 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:47:39.703+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:47:39.704313+0000) 2024-03-04T17:47:40.704 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:47:40.703+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:47:40.704504+0000) 2024-03-04T17:47:41.704 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:47:41.703+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:47:41.704702+0000) 2024-03-04T17:47:42.704 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:47:42.704+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:47:42.704936+0000) 2024-03-04T17:47:43.705 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:47:43.704+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:47:43.705128+0000) 2024-03-04T17:47:44.705 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:47:44.704+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:47:44.705318+0000) 2024-03-04T17:47:45.705 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:47:45.704+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:47:45.705518+0000) 2024-03-04T17:47:46.072 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:47:46.080 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:47:46.705 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:47:46.704+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:47:46.705715+0000) 2024-03-04T17:47:47.705 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:47:47.705+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:47:47.705918+0000) 2024-03-04T17:47:48.705 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:47:48.705+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:47:48.706095+0000) 2024-03-04T17:47:49.706 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:47:49.705+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:47:49.706277+0000) 2024-03-04T17:47:50.706 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:47:50.705+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:47:50.706453+0000) 2024-03-04T17:47:51.706 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:47:51.705+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:47:51.706625+0000) 2024-03-04T17:47:52.706 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:47:52.706+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:47:52.706815+0000) 2024-03-04T17:47:53.706 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:47:53.706+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:47:53.707029+0000) 2024-03-04T17:47:54.706 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:47:54.706+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:47:54.707208+0000) 2024-03-04T17:47:55.706 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:47:55.706+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:47:55.707383+0000) 2024-03-04T17:47:56.707 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:47:56.706+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:47:56.707583+0000) 2024-03-04T17:47:57.707 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:47:57.706+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:47:57.707784+0000) 2024-03-04T17:47:58.707 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:47:58.707+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:47:58.708063+0000) 2024-03-04T17:47:59.707 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:47:59.707+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:47:59.708253+0000) 2024-03-04T17:48:00.708 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:48:00.707+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:48:00.708451+0000) 2024-03-04T17:48:01.708 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:48:01.707+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:48:01.708648+0000) 2024-03-04T17:48:02.708 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:48:02.708+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:48:02.708893+0000) 2024-03-04T17:48:03.708 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:48:03.708+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:48:03.709089+0000) 2024-03-04T17:48:04.708 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:48:04.708+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:48:04.709283+0000) 2024-03-04T17:48:05.709 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:48:05.708+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:48:05.709477+0000) 2024-03-04T17:48:06.709 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:48:06.708+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:48:06.709675+0000) 2024-03-04T17:48:07.709 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:48:07.709+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:48:07.709911+0000) 2024-03-04T17:48:08.709 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:48:08.709+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:48:08.710103+0000) 2024-03-04T17:48:09.710 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:48:09.709+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:48:09.710300+0000) 2024-03-04T17:48:10.710 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:48:10.709+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:48:10.710550+0000) 2024-03-04T17:48:11.710 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:48:11.709+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:48:11.710747+0000) 2024-03-04T17:48:12.710 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:48:12.710+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:48:12.710947+0000) 2024-03-04T17:48:13.710 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:48:13.710+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:48:13.711140+0000) 2024-03-04T17:48:14.710 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:48:14.710+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:48:14.711334+0000) 2024-03-04T17:48:15.711 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:48:15.710+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:48:15.711528+0000) 2024-03-04T17:48:16.211 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:48:16.219 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:48:16.711 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:48:16.710+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:48:16.711727+0000) 2024-03-04T17:48:17.711 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:48:17.711+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:48:17.711953+0000) 2024-03-04T17:48:18.711 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:48:18.711+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:48:18.712156+0000) 2024-03-04T17:48:19.711 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:48:19.710+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:48:19.712369+0000) 2024-03-04T17:48:20.712 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:48:20.711+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:48:20.712560+0000) 2024-03-04T17:48:21.715 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:48:21.711+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:48:21.712770+0000) 2024-03-04T17:48:22.712 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:48:22.711+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:48:22.713013+0000) 2024-03-04T17:48:23.712 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:48:23.711+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:48:23.713225+0000) 2024-03-04T17:48:24.713 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:48:24.711+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:48:24.713487+0000) 2024-03-04T17:48:25.713 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:48:25.712+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:48:25.713732+0000) 2024-03-04T17:48:26.713 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:48:26.712+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:48:26.714005+0000) 2024-03-04T17:48:27.713 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:48:27.712+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:48:27.714214+0000) 2024-03-04T17:48:28.714 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:48:28.712+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:48:28.714424+0000) 2024-03-04T17:48:29.715 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:48:29.713+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:48:29.714619+0000) 2024-03-04T17:48:30.714 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:48:30.713+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:48:30.714804+0000) 2024-03-04T17:48:31.714 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:48:31.714+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:48:31.715027+0000) 2024-03-04T17:48:32.714 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:48:32.714+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:48:32.715266+0000) 2024-03-04T17:48:33.715 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:48:33.714+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:48:33.715537+0000) 2024-03-04T17:48:34.715 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:48:34.714+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:48:34.715787+0000) 2024-03-04T17:48:35.715 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:48:35.715+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:48:35.716049+0000) 2024-03-04T17:48:36.715 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:48:36.715+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:48:36.716241+0000) 2024-03-04T17:48:37.716 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:48:37.715+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:48:37.716430+0000) 2024-03-04T17:48:38.716 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:48:38.715+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:48:38.716679+0000) 2024-03-04T17:48:39.716 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:48:39.716+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:48:39.716951+0000) 2024-03-04T17:48:40.716 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:48:40.716+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:48:40.717142+0000) 2024-03-04T17:48:41.716 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:48:41.716+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:48:41.717332+0000) 2024-03-04T17:48:42.717 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:48:42.716+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:48:42.717528+0000) 2024-03-04T17:48:43.717 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:48:43.717+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:48:43.717765+0000) 2024-03-04T17:48:44.717 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:48:44.717+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:48:44.718012+0000) 2024-03-04T17:48:45.718 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:48:45.717+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:48:45.718247+0000) 2024-03-04T17:48:46.374 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:48:46.382 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:48:46.717 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:48:46.717+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:48:46.718486+0000) 2024-03-04T17:48:47.718 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:48:47.718+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:48:47.718723+0000) 2024-03-04T17:48:48.718 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:48:48.718+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:48:48.718985+0000) 2024-03-04T17:48:49.718 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:48:49.718+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:48:49.719214+0000) 2024-03-04T17:48:50.718 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:48:50.718+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:48:50.719447+0000) 2024-03-04T17:48:51.719 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:48:51.719+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:48:51.719686+0000) 2024-03-04T17:48:52.719 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:48:52.719+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:48:52.719935+0000) 2024-03-04T17:48:53.719 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:48:53.719+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:48:53.720172+0000) 2024-03-04T17:48:54.720 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:48:54.719+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:48:54.720410+0000) 2024-03-04T17:48:55.720 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:48:55.720+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:48:55.720643+0000) 2024-03-04T17:48:56.720 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:48:56.720+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:48:56.720921+0000) 2024-03-04T17:48:57.720 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:48:57.720+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:48:57.721158+0000) 2024-03-04T17:48:58.720 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:48:58.720+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:48:58.721417+0000) 2024-03-04T17:48:59.721 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:48:59.721+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:48:59.721651+0000) 2024-03-04T17:49:00.721 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:49:00.721+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:49:00.721942+0000) 2024-03-04T17:49:01.721 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:49:01.721+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:49:01.722180+0000) 2024-03-04T17:49:02.721 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:49:02.721+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:49:02.722426+0000) 2024-03-04T17:49:03.722 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:49:03.722+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:49:03.722723+0000) 2024-03-04T17:49:04.723 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:49:04.722+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:49:04.722983+0000) 2024-03-04T17:49:05.722 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:49:05.722+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:49:05.723164+0000) 2024-03-04T17:49:06.722 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:49:06.722+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:49:06.723357+0000) 2024-03-04T17:49:07.723 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:49:07.722+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:49:07.723549+0000) 2024-03-04T17:49:08.723 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:49:08.723+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:49:08.723790+0000) 2024-03-04T17:49:09.723 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:49:09.723+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:49:09.724062+0000) 2024-03-04T17:49:10.724 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:49:10.723+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:49:10.724302+0000) 2024-03-04T17:49:11.724 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:49:11.723+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:49:11.724482+0000) 2024-03-04T17:49:12.724 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:49:12.724+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:49:12.724730+0000) 2024-03-04T17:49:13.724 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:49:13.724+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:49:13.724972+0000) 2024-03-04T17:49:14.724 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:49:14.724+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:49:14.725211+0000) 2024-03-04T17:49:15.725 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:49:15.725+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:49:15.725476+0000) 2024-03-04T17:49:16.596 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:49:16.607 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:49:16.725 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:49:16.725+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:49:16.725742+0000) 2024-03-04T17:49:17.725 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:49:17.725+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:49:17.726004+0000) 2024-03-04T17:49:18.725 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:49:18.726+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:49:18.726242+0000) 2024-03-04T17:49:19.725 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:49:19.726+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:49:19.726485+0000) 2024-03-04T17:49:20.726 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:49:20.725+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:49:20.726714+0000) 2024-03-04T17:49:21.726 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:49:21.726+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:49:21.726948+0000) 2024-03-04T17:49:22.727 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:49:22.726+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:49:22.727192+0000) 2024-03-04T17:49:23.726 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:49:23.726+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:49:23.727432+0000) 2024-03-04T17:49:24.727 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:49:24.726+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:49:24.727668+0000) 2024-03-04T17:49:25.727 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:49:25.726+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:49:25.727951+0000) 2024-03-04T17:49:26.727 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:49:26.726+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:49:26.728133+0000) 2024-03-04T17:49:27.727 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:49:27.727+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:49:27.728369+0000) 2024-03-04T17:49:28.728 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:49:28.727+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:49:28.728614+0000) 2024-03-04T17:49:29.728 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:49:29.727+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:49:29.728879+0000) 2024-03-04T17:49:30.728 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:49:30.728+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:49:30.729118+0000) 2024-03-04T17:49:31.728 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:49:31.728+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:49:31.729356+0000) 2024-03-04T17:49:32.729 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:49:32.727+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:49:32.729628+0000) 2024-03-04T17:49:33.729 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:49:33.729+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:49:33.729910+0000) 2024-03-04T17:49:34.729 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:49:34.729+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:49:34.730157+0000) 2024-03-04T17:49:35.730 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:49:35.729+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:49:35.730391+0000) 2024-03-04T17:49:36.730 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:49:36.729+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:49:36.730630+0000) 2024-03-04T17:49:37.730 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:49:37.730+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:49:37.730928+0000) 2024-03-04T17:49:38.730 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:49:38.730+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:49:38.731185+0000) 2024-03-04T17:49:39.730 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:49:39.730+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:49:39.731421+0000) 2024-03-04T17:49:40.731 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:49:40.730+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:49:40.731660+0000) 2024-03-04T17:49:41.731 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:49:41.731+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:49:41.731951+0000) 2024-03-04T17:49:42.731 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:49:42.731+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:49:42.732200+0000) 2024-03-04T17:49:43.732 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:49:43.731+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:49:43.732440+0000) 2024-03-04T17:49:44.732 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:49:44.731+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:49:44.732672+0000) 2024-03-04T17:49:45.732 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:49:45.732+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:49:45.732953+0000) 2024-03-04T17:49:46.732 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:49:46.732+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:49:46.733216+0000) 2024-03-04T17:49:46.754 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:49:46.762 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:49:47.733 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:49:47.732+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:49:47.733473+0000) 2024-03-04T17:49:48.733 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:49:48.733+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:49:48.733711+0000) 2024-03-04T17:49:49.733 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:49:49.733+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:49:49.733943+0000) 2024-03-04T17:49:50.733 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:49:50.733+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:49:50.734122+0000) 2024-03-04T17:49:51.733 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:49:51.733+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:49:51.734364+0000) 2024-03-04T17:49:52.734 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:49:52.733+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:49:52.734614+0000) 2024-03-04T17:49:53.734 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:49:53.734+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:49:53.734908+0000) 2024-03-04T17:49:54.734 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:49:54.734+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:49:54.735159+0000) 2024-03-04T17:49:55.735 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:49:55.734+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:49:55.735418+0000) 2024-03-04T17:49:56.735 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:49:56.734+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:49:56.735621+0000) 2024-03-04T17:49:57.735 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:49:57.735+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:49:57.735950+0000) 2024-03-04T17:49:58.735 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:49:58.735+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:49:58.736196+0000) 2024-03-04T17:49:59.735 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:49:59.735+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:49:59.736436+0000) 2024-03-04T17:50:00.736 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:50:00.735+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:50:00.736696+0000) 2024-03-04T17:50:01.736 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:50:01.736+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:50:01.736955+0000) 2024-03-04T17:50:02.736 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:50:02.736+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:50:02.737215+0000) 2024-03-04T17:50:03.736 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:50:03.736+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:50:03.737460+0000) 2024-03-04T17:50:04.737 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:50:04.737+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:50:04.737696+0000) 2024-03-04T17:50:05.739 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:50:05.737+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:50:05.737952+0000) 2024-03-04T17:50:06.737 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:50:06.737+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:50:06.738191+0000) 2024-03-04T17:50:07.737 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:50:07.738+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:50:07.738430+0000) 2024-03-04T17:50:08.738 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:50:08.738+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:50:08.738670+0000) 2024-03-04T17:50:09.738 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:50:09.738+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:50:09.738929+0000) 2024-03-04T17:50:10.738 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:50:10.738+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:50:10.739168+0000) 2024-03-04T17:50:11.738 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:50:11.739+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:50:11.739408+0000) 2024-03-04T17:50:12.739 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:50:12.739+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:50:12.739660+0000) 2024-03-04T17:50:13.739 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:50:13.739+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:50:13.739948+0000) 2024-03-04T17:50:14.739 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:50:14.739+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:50:14.740184+0000) 2024-03-04T17:50:15.739 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:50:15.740+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:50:15.740422+0000) 2024-03-04T17:50:16.740 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:50:16.740+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:50:16.740686+0000) 2024-03-04T17:50:16.888 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:50:16.896 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:50:17.740 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:50:17.740+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:50:17.740948+0000) 2024-03-04T17:50:18.740 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:50:18.740+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:50:18.741191+0000) 2024-03-04T17:50:19.740 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:50:19.741+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:50:19.741429+0000) 2024-03-04T17:50:20.741 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:50:20.741+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:50:20.741667+0000) 2024-03-04T17:50:21.741 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:50:21.741+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:50:21.741946+0000) 2024-03-04T17:50:22.741 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:50:22.742+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:50:22.742203+0000) 2024-03-04T17:50:23.741 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:50:23.742+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:50:23.742400+0000) 2024-03-04T17:50:24.742 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:50:24.741+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:50:24.742576+0000) 2024-03-04T17:50:25.742 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:50:25.741+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:50:25.742854+0000) 2024-03-04T17:50:26.742 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:50:26.741+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:50:26.743095+0000) 2024-03-04T17:50:27.742 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:50:27.742+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:50:27.743333+0000) 2024-03-04T17:50:28.743 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:50:28.742+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:50:28.743577+0000) 2024-03-04T17:50:29.743 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:50:29.742+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:50:29.743807+0000) 2024-03-04T17:50:30.743 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:50:30.743+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:50:30.744086+0000) 2024-03-04T17:50:31.743 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:50:31.743+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:50:31.744326+0000) 2024-03-04T17:50:32.743 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:50:32.743+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:50:32.744576+0000) 2024-03-04T17:50:33.744 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:50:33.743+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:50:33.744868+0000) 2024-03-04T17:50:34.744 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:50:34.744+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:50:34.745119+0000) 2024-03-04T17:50:35.744 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:50:35.744+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:50:35.745359+0000) 2024-03-04T17:50:36.745 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:50:36.744+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:50:36.745607+0000) 2024-03-04T17:50:37.745 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:50:37.745+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:50:37.745896+0000) 2024-03-04T17:50:38.745 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:50:38.745+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:50:38.746156+0000) 2024-03-04T17:50:39.745 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:50:39.745+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:50:39.746409+0000) 2024-03-04T17:50:40.746 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:50:40.745+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:50:40.746660+0000) 2024-03-04T17:50:41.746 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:50:41.746+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:50:41.746950+0000) 2024-03-04T17:50:42.746 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:50:42.746+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:50:42.747217+0000) 2024-03-04T17:50:43.747 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:50:43.746+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:50:43.747470+0000) 2024-03-04T17:50:44.747 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:50:44.746+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:50:44.747675+0000) 2024-03-04T17:50:45.748 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:50:45.747+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:50:45.747941+0000) 2024-03-04T17:50:46.747 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:50:46.747+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:50:46.748226+0000) 2024-03-04T17:50:47.086 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:50:47.097 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:50:47.747 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:50:47.747+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:50:47.748507+0000) 2024-03-04T17:50:48.748 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:50:48.748+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:50:48.748798+0000) 2024-03-04T17:50:49.748 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:50:49.748+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:50:49.749098+0000) 2024-03-04T17:50:50.748 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:50:50.748+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:50:50.749407+0000) 2024-03-04T17:50:51.749 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:50:51.748+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:50:51.749678+0000) 2024-03-04T17:50:52.749 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:50:52.749+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:50:52.749941+0000) 2024-03-04T17:50:53.749 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:50:53.749+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:50:53.750188+0000) 2024-03-04T17:50:54.750 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:50:54.749+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:50:54.750424+0000) 2024-03-04T17:50:55.750 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:50:55.750+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:50:55.750685+0000) 2024-03-04T17:50:56.750 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:50:56.750+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:50:56.750942+0000) 2024-03-04T17:50:57.751 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:50:57.750+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:50:57.751140+0000) 2024-03-04T17:50:58.750 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:50:58.750+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:50:58.751370+0000) 2024-03-04T17:50:59.751 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:50:59.751+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:50:59.751627+0000) 2024-03-04T17:51:00.751 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:51:00.751+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:51:00.751931+0000) 2024-03-04T17:51:01.751 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:51:01.751+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:51:01.752174+0000) 2024-03-04T17:51:02.751 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:51:02.751+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:51:02.752429+0000) 2024-03-04T17:51:03.752 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:51:03.751+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:51:03.752666+0000) 2024-03-04T17:51:04.752 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:51:04.752+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:51:04.752950+0000) 2024-03-04T17:51:05.752 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:51:05.752+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:51:05.753140+0000) 2024-03-04T17:51:06.752 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:51:06.752+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:51:06.753336+0000) 2024-03-04T17:51:07.753 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:51:07.753+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:51:07.753598+0000) 2024-03-04T17:51:08.753 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:51:08.752+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:51:08.753799+0000) 2024-03-04T17:51:09.753 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:51:09.753+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:51:09.754012+0000) 2024-03-04T17:51:10.753 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:51:10.753+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:51:10.754262+0000) 2024-03-04T17:51:11.754 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:51:11.753+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:51:11.754533+0000) 2024-03-04T17:51:12.754 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:51:12.754+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:51:12.754784+0000) 2024-03-04T17:51:13.755 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:51:13.754+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:51:13.755092+0000) 2024-03-04T17:51:14.754 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:51:14.755+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:51:14.755340+0000) 2024-03-04T17:51:15.755 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:51:15.755+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:51:15.755592+0000) 2024-03-04T17:51:16.755 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:51:16.755+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:51:16.755906+0000) 2024-03-04T17:51:17.281 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:51:17.289 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:51:17.755 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:51:17.755+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:51:17.756189+0000) 2024-03-04T17:51:18.755 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:51:18.756+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:51:18.756438+0000) 2024-03-04T17:51:19.756 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:51:19.756+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:51:19.756684+0000) 2024-03-04T17:51:20.756 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:51:20.756+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:51:20.756940+0000) 2024-03-04T17:51:21.756 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:51:21.756+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:51:21.757199+0000) 2024-03-04T17:51:22.759 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:51:22.757+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:51:22.757488+0000) 2024-03-04T17:51:23.757 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:51:23.757+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:51:23.757731+0000) 2024-03-04T17:51:24.757 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:51:24.757+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:51:24.757955+0000) 2024-03-04T17:51:25.757 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:51:25.757+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:51:25.758193+0000) 2024-03-04T17:51:26.758 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:51:26.757+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:51:26.758389+0000) 2024-03-04T17:51:27.757 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:51:27.757+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:51:27.758580+0000) 2024-03-04T17:51:28.758 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:51:28.757+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:51:28.758851+0000) 2024-03-04T17:51:29.758 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:51:29.757+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:51:29.759088+0000) 2024-03-04T17:51:30.758 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:51:30.758+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:51:30.759364+0000) 2024-03-04T17:51:31.759 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:51:31.758+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:51:31.759625+0000) 2024-03-04T17:51:32.759 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:51:32.758+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:51:32.759948+0000) 2024-03-04T17:51:33.759 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:51:33.759+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:51:33.760270+0000) 2024-03-04T17:51:34.759 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:51:34.759+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:51:34.760517+0000) 2024-03-04T17:51:35.760 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:51:35.759+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:51:35.760776+0000) 2024-03-04T17:51:36.760 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:51:36.760+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:51:36.761035+0000) 2024-03-04T17:51:37.760 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:51:37.760+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:51:37.761294+0000) 2024-03-04T17:51:38.761 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:51:38.760+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:51:38.761552+0000) 2024-03-04T17:51:39.761 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:51:39.761+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:51:39.761807+0000) 2024-03-04T17:51:40.764 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:51:40.761+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:51:40.762109+0000) 2024-03-04T17:51:41.761 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:51:41.761+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:51:41.762370+0000) 2024-03-04T17:51:42.762 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:51:42.761+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:51:42.762643+0000) 2024-03-04T17:51:43.762 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:51:43.762+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:51:43.762932+0000) 2024-03-04T17:51:44.762 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:51:44.762+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:51:44.763184+0000) 2024-03-04T17:51:45.762 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:51:45.762+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:51:45.763461+0000) 2024-03-04T17:51:46.763 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:51:46.762+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:51:46.763742+0000) 2024-03-04T17:51:47.512 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:51:47.520 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:51:47.763 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:51:47.763+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:51:47.764008+0000) 2024-03-04T17:51:48.763 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:51:48.763+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:51:48.764263+0000) 2024-03-04T17:51:49.763 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:51:49.763+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:51:49.764510+0000) 2024-03-04T17:51:50.764 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:51:50.763+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:51:50.764769+0000) 2024-03-04T17:51:51.764 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:51:51.764+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:51:51.765029+0000) 2024-03-04T17:51:52.764 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:51:52.764+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:51:52.765282+0000) 2024-03-04T17:51:53.764 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:51:53.764+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:51:53.765527+0000) 2024-03-04T17:51:54.765 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:51:54.764+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:51:54.765705+0000) 2024-03-04T17:51:55.765 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:51:55.765+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:51:55.765945+0000) 2024-03-04T17:51:56.765 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:51:56.765+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:51:56.766191+0000) 2024-03-04T17:51:57.765 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:51:57.765+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:51:57.766465+0000) 2024-03-04T17:51:58.766 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:51:58.765+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:51:58.766725+0000) 2024-03-04T17:51:59.766 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:51:59.766+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:51:59.766960+0000) 2024-03-04T17:52:00.766 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:52:00.766+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:52:00.767204+0000) 2024-03-04T17:52:01.766 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:52:01.766+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:52:01.767450+0000) 2024-03-04T17:52:02.767 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:52:02.767+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:52:02.767707+0000) 2024-03-04T17:52:03.767 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:52:03.767+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:52:03.767944+0000) 2024-03-04T17:52:04.767 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:52:04.767+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:52:04.768179+0000) 2024-03-04T17:52:05.767 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:52:05.767+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:52:05.768418+0000) 2024-03-04T17:52:06.768 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:52:06.768+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:52:06.768670+0000) 2024-03-04T17:52:07.768 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:52:07.768+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:52:07.768946+0000) 2024-03-04T17:52:08.768 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:52:08.768+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:52:08.769139+0000) 2024-03-04T17:52:09.768 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:52:09.768+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:52:09.769373+0000) 2024-03-04T17:52:10.769 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:52:10.769+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:52:10.769615+0000) 2024-03-04T17:52:11.769 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:52:11.769+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:52:11.769885+0000) 2024-03-04T17:52:12.769 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:52:12.769+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:52:12.770135+0000) 2024-03-04T17:52:13.769 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:52:13.769+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:52:13.770377+0000) 2024-03-04T17:52:14.770 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:52:14.770+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:52:14.770610+0000) 2024-03-04T17:52:15.770 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:52:15.770+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:52:15.770886+0000) 2024-03-04T17:52:16.770 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:52:16.770+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:52:16.771124+0000) 2024-03-04T17:52:17.725 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:52:17.733 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:52:17.770 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:52:17.770+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:52:17.771293+0000) 2024-03-04T17:52:18.770 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:52:18.770+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:52:18.771581+0000) 2024-03-04T17:52:19.771 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:52:19.771+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:52:19.771914+0000) 2024-03-04T17:52:20.771 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:52:20.771+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:52:20.772208+0000) 2024-03-04T17:52:21.771 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:52:21.772+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:52:21.772567+0000) 2024-03-04T17:52:22.772 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:52:22.772+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:52:22.772912+0000) 2024-03-04T17:52:23.772 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:52:23.772+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:52:23.773159+0000) 2024-03-04T17:52:24.772 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:52:24.773+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:52:24.773393+0000) 2024-03-04T17:52:25.773 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:52:25.773+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:52:25.773653+0000) 2024-03-04T17:52:26.773 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:52:26.773+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:52:26.773941+0000) 2024-03-04T17:52:27.773 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:52:27.773+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:52:27.774180+0000) 2024-03-04T17:52:28.773 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:52:28.774+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:52:28.774422+0000) 2024-03-04T17:52:29.774 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:52:29.774+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:52:29.774692+0000) 2024-03-04T17:52:30.774 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:52:30.774+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:52:30.774936+0000) 2024-03-04T17:52:31.774 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:52:31.775+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:52:31.775178+0000) 2024-03-04T17:52:32.774 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:52:32.774+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:52:32.775432+0000) 2024-03-04T17:52:33.775 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:52:33.774+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:52:33.775672+0000) 2024-03-04T17:52:34.775 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:52:34.774+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:52:34.775948+0000) 2024-03-04T17:52:35.775 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:52:35.775+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:52:35.776186+0000) 2024-03-04T17:52:36.775 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:52:36.775+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:52:36.776424+0000) 2024-03-04T17:52:37.775 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:52:37.774+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:52:37.776609+0000) 2024-03-04T17:52:38.776 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:52:38.775+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:52:38.776892+0000) 2024-03-04T17:52:39.776 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:52:39.776+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:52:39.777126+0000) 2024-03-04T17:52:40.776 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:52:40.776+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:52:40.777401+0000) 2024-03-04T17:52:41.777 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:52:41.776+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:52:41.777645+0000) 2024-03-04T17:52:42.777 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:52:42.776+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:52:42.777937+0000) 2024-03-04T17:52:43.777 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:52:43.777+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:52:43.778181+0000) 2024-03-04T17:52:44.777 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:52:44.777+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:52:44.778417+0000) 2024-03-04T17:52:45.778 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:52:45.777+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:52:45.778672+0000) 2024-03-04T17:52:46.778 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:52:46.778+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:52:46.778936+0000) 2024-03-04T17:52:47.778 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:52:47.778+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:52:47.779179+0000) 2024-03-04T17:52:47.853 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:52:47.862 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:52:48.778 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:52:48.778+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:52:48.779423+0000) 2024-03-04T17:52:49.779 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:52:49.778+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:52:49.779661+0000) 2024-03-04T17:52:50.779 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:52:50.779+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:52:50.779947+0000) 2024-03-04T17:52:51.779 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:52:51.779+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:52:51.780193+0000) 2024-03-04T17:52:52.779 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:52:52.779+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:52:52.780442+0000) 2024-03-04T17:52:53.780 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:52:53.779+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:52:53.780687+0000) 2024-03-04T17:52:54.780 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:52:54.780+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:52:54.780922+0000) 2024-03-04T17:52:55.780 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:52:55.780+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:52:55.781152+0000) 2024-03-04T17:52:56.780 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:52:56.780+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:52:56.781391+0000) 2024-03-04T17:52:57.781 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:52:57.780+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:52:57.781645+0000) 2024-03-04T17:52:58.781 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:52:58.781+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:52:58.781923+0000) 2024-03-04T17:52:59.781 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:52:59.781+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:52:59.782153+0000) 2024-03-04T17:53:00.781 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:53:00.781+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:53:00.782388+0000) 2024-03-04T17:53:01.781 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:53:01.781+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:53:01.782642+0000) 2024-03-04T17:53:02.782 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:53:02.782+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:53:02.782946+0000) 2024-03-04T17:53:03.782 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:53:03.782+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:53:03.783188+0000) 2024-03-04T17:53:04.782 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:53:04.782+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:53:04.783429+0000) 2024-03-04T17:53:05.783 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:53:05.783+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:53:05.783670+0000) 2024-03-04T17:53:06.783 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:53:06.783+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:53:06.783954+0000) 2024-03-04T17:53:07.783 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:53:07.783+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:53:07.784199+0000) 2024-03-04T17:53:08.783 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:53:08.783+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:53:08.784459+0000) 2024-03-04T17:53:09.784 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:53:09.784+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:53:09.784700+0000) 2024-03-04T17:53:10.784 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:53:10.784+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:53:10.784940+0000) 2024-03-04T17:53:11.784 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:53:11.784+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:53:11.785181+0000) 2024-03-04T17:53:12.784 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:53:12.784+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:53:12.785381+0000) 2024-03-04T17:53:13.785 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:53:13.785+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:53:13.785652+0000) 2024-03-04T17:53:14.785 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:53:14.785+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:53:14.785927+0000) 2024-03-04T17:53:15.785 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:53:15.785+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:53:15.786161+0000) 2024-03-04T17:53:16.785 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:53:16.786+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:53:16.786419+0000) 2024-03-04T17:53:17.786 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:53:17.786+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:53:17.786655+0000) 2024-03-04T17:53:18.068 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:53:18.079 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:53:18.786 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:53:18.786+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:53:18.786918+0000) 2024-03-04T17:53:19.786 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:53:19.786+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:53:19.787160+0000) 2024-03-04T17:53:20.786 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:53:20.786+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:53:20.787342+0000) 2024-03-04T17:53:21.786 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:53:21.787+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:53:21.787578+0000) 2024-03-04T17:53:22.787 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:53:22.787+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:53:22.787863+0000) 2024-03-04T17:53:23.787 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:53:23.787+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:53:23.788105+0000) 2024-03-04T17:53:24.787 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:53:24.788+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:53:24.788339+0000) 2024-03-04T17:53:25.787 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:53:25.788+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:53:25.788583+0000) 2024-03-04T17:53:26.788 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:53:26.788+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:53:26.788855+0000) 2024-03-04T17:53:27.788 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:53:27.788+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:53:27.789100+0000) 2024-03-04T17:53:28.788 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:53:28.789+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:53:28.789338+0000) 2024-03-04T17:53:29.788 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:53:29.789+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:53:29.789574+0000) 2024-03-04T17:53:30.789 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:53:30.789+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:53:30.789813+0000) 2024-03-04T17:53:31.789 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:53:31.789+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:53:31.790095+0000) 2024-03-04T17:53:32.789 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:53:32.790+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:53:32.790342+0000) 2024-03-04T17:53:33.789 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:53:33.789+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:53:33.790567+0000) 2024-03-04T17:53:34.790 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:53:34.790+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:53:34.790905+0000) 2024-03-04T17:53:35.790 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:53:35.790+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:53:35.791187+0000) 2024-03-04T17:53:36.790 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:53:36.790+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:53:36.791504+0000) 2024-03-04T17:53:37.791 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:53:37.790+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:53:37.791785+0000) 2024-03-04T17:53:38.791 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:53:38.791+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:53:38.792125+0000) 2024-03-04T17:53:39.791 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:53:39.791+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:53:39.792416+0000) 2024-03-04T17:53:40.792 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:53:40.790+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:53:40.792725+0000) 2024-03-04T17:53:41.792 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:53:41.791+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:53:41.793014+0000) 2024-03-04T17:53:42.792 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:53:42.792+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:53:42.793320+0000) 2024-03-04T17:53:43.792 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:53:43.792+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:53:43.793637+0000) 2024-03-04T17:53:44.793 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:53:44.792+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:53:44.793964+0000) 2024-03-04T17:53:45.793 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:53:45.793+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:53:45.794262+0000) 2024-03-04T17:53:46.793 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:53:46.793+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:53:46.794531+0000) 2024-03-04T17:53:47.794 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:53:47.793+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:53:47.794784+0000) 2024-03-04T17:53:48.272 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:53:48.280 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:53:48.794 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:53:48.794+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:53:48.795099+0000) 2024-03-04T17:53:49.794 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:53:49.794+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:53:49.795356+0000) 2024-03-04T17:53:50.795 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:53:50.794+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:53:50.795609+0000) 2024-03-04T17:53:51.795 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:53:51.795+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:53:51.795911+0000) 2024-03-04T17:53:52.795 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:53:52.795+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:53:52.796178+0000) 2024-03-04T17:53:53.795 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:53:53.795+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:53:53.796435+0000) 2024-03-04T17:53:54.796 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:53:54.795+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:53:54.796698+0000) 2024-03-04T17:53:55.796 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:53:55.796+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:53:55.796972+0000) 2024-03-04T17:53:56.796 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:53:56.796+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:53:56.797225+0000) 2024-03-04T17:53:57.796 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:53:57.796+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:53:57.797516+0000) 2024-03-04T17:53:58.797 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:53:58.797+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:53:58.797780+0000) 2024-03-04T17:53:59.797 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:53:59.797+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:53:59.798118+0000) 2024-03-04T17:54:00.797 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:54:00.797+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:54:00.798375+0000) 2024-03-04T17:54:01.797 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:54:01.797+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:54:01.798617+0000) 2024-03-04T17:54:02.798 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:54:02.798+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:54:02.798908+0000) 2024-03-04T17:54:03.798 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:54:03.798+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:54:03.799146+0000) 2024-03-04T17:54:04.798 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:54:04.798+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:54:04.799381+0000) 2024-03-04T17:54:05.799 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:54:05.798+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:54:05.799633+0000) 2024-03-04T17:54:06.799 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:54:06.799+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:54:06.799917+0000) 2024-03-04T17:54:07.799 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:54:07.799+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:54:07.800269+0000) 2024-03-04T17:54:08.799 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:54:08.799+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:54:08.800506+0000) 2024-03-04T17:54:09.800 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:54:09.800+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:54:09.800748+0000) 2024-03-04T17:54:10.800 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:54:10.800+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:54:10.800989+0000) 2024-03-04T17:54:11.800 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:54:11.800+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:54:11.801247+0000) 2024-03-04T17:54:12.800 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:54:12.800+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:54:12.801513+0000) 2024-03-04T17:54:13.801 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:54:13.801+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:54:13.801755+0000) 2024-03-04T17:54:14.801 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:54:14.801+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:54:14.801997+0000) 2024-03-04T17:54:15.801 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:54:15.801+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:54:15.802251+0000) 2024-03-04T17:54:16.801 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:54:16.802+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:54:16.802495+0000) 2024-03-04T17:54:17.802 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:54:17.802+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:54:17.802733+0000) 2024-03-04T17:54:18.402 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:54:18.410 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:54:18.802 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:54:18.802+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:54:18.802973+0000) 2024-03-04T17:54:19.802 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:54:19.802+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:54:19.803208+0000) 2024-03-04T17:54:20.802 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:54:20.803+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:54:20.803445+0000) 2024-03-04T17:54:21.803 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:54:21.802+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:54:21.803681+0000) 2024-03-04T17:54:22.803 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:54:22.803+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:54:22.803948+0000) 2024-03-04T17:54:23.803 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:54:23.803+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:54:23.804143+0000) 2024-03-04T17:54:24.803 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:54:24.804+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:54:24.804384+0000) 2024-03-04T17:54:25.803 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:54:25.803+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:54:25.804615+0000) 2024-03-04T17:54:26.804 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:54:26.804+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:54:26.804890+0000) 2024-03-04T17:54:27.804 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:54:27.804+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:54:27.805129+0000) 2024-03-04T17:54:28.804 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:54:28.805+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:54:28.805366+0000) 2024-03-04T17:54:29.804 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:54:29.805+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:54:29.805631+0000) 2024-03-04T17:54:30.805 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:54:30.805+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:54:30.805926+0000) 2024-03-04T17:54:31.805 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:54:31.805+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:54:31.806159+0000) 2024-03-04T17:54:32.805 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:54:32.806+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:54:32.806410+0000) 2024-03-04T17:54:33.806 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:54:33.806+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:54:33.806649+0000) 2024-03-04T17:54:34.806 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:54:34.806+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:54:34.806931+0000) 2024-03-04T17:54:35.806 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:54:35.807+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:54:35.807168+0000) 2024-03-04T17:54:36.806 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:54:36.807+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:54:36.807405+0000) 2024-03-04T17:54:37.807 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:54:37.807+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:54:37.807643+0000) 2024-03-04T17:54:38.807 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:54:38.806+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:54:38.807949+0000) 2024-03-04T17:54:39.807 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:54:39.807+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:54:39.808186+0000) 2024-03-04T17:54:40.807 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:54:40.807+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:54:40.808427+0000) 2024-03-04T17:54:41.808 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:54:41.807+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:54:41.808690+0000) 2024-03-04T17:54:42.808 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:54:42.807+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:54:42.808935+0000) 2024-03-04T17:54:43.808 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:54:43.808+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:54:43.809213+0000) 2024-03-04T17:54:44.808 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:54:44.808+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:54:44.809455+0000) 2024-03-04T17:54:45.812 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:54:45.808+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:54:45.809694+0000) 2024-03-04T17:54:46.809 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:54:46.808+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:54:46.809939+0000) 2024-03-04T17:54:47.809 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:54:47.809+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:54:47.810139+0000) 2024-03-04T17:54:48.616 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:54:48.624 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:54:48.809 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:54:48.809+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:54:48.810377+0000) 2024-03-04T17:54:49.809 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:54:49.809+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:54:49.810612+0000) 2024-03-04T17:54:50.810 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:54:50.810+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:54:50.810888+0000) 2024-03-04T17:54:51.810 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:54:51.810+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:54:51.811130+0000) 2024-03-04T17:54:52.810 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:54:52.810+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:54:52.811392+0000) 2024-03-04T17:54:53.810 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:54:53.810+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:54:53.811647+0000) 2024-03-04T17:54:54.811 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:54:54.811+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:54:54.811951+0000) 2024-03-04T17:54:55.811 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:54:55.811+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:54:55.812255+0000) 2024-03-04T17:54:56.811 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:54:56.811+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:54:56.812461+0000) 2024-03-04T17:54:57.812 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:54:57.811+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:54:57.812720+0000) 2024-03-04T17:54:58.812 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:54:58.812+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:54:58.812985+0000) 2024-03-04T17:54:59.812 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:54:59.812+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:54:59.813254+0000) 2024-03-04T17:55:00.812 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:55:00.812+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:55:00.813533+0000) 2024-03-04T17:55:01.813 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:55:01.813+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:55:01.813781+0000) 2024-03-04T17:55:02.813 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:55:02.813+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:55:02.814039+0000) 2024-03-04T17:55:03.813 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:55:03.813+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:55:03.814275+0000) 2024-03-04T17:55:04.813 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:55:04.813+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:55:04.814509+0000) 2024-03-04T17:55:05.814 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:55:05.814+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:55:05.814747+0000) 2024-03-04T17:55:06.814 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:55:06.814+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:55:06.814986+0000) 2024-03-04T17:55:07.814 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:55:07.814+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:55:07.815237+0000) 2024-03-04T17:55:08.814 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:55:08.814+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:55:08.815478+0000) 2024-03-04T17:55:09.815 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:55:09.815+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:55:09.815712+0000) 2024-03-04T17:55:10.815 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:55:10.815+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:55:10.815951+0000) 2024-03-04T17:55:11.815 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:55:11.815+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:55:11.816191+0000) 2024-03-04T17:55:12.815 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:55:12.815+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:55:12.816460+0000) 2024-03-04T17:55:13.815 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:55:13.816+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:55:13.816687+0000) 2024-03-04T17:55:14.816 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:55:14.816+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:55:14.816924+0000) 2024-03-04T17:55:15.816 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:55:15.816+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:55:15.817156+0000) 2024-03-04T17:55:16.816 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:55:16.816+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:55:16.817426+0000) 2024-03-04T17:55:17.817 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:55:17.817+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:55:17.817667+0000) 2024-03-04T17:55:18.817 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:55:18.817+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:55:18.817951+0000) 2024-03-04T17:55:18.821 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:55:18.830 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:55:19.817 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:55:19.817+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:55:19.818185+0000) 2024-03-04T17:55:20.817 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:55:20.817+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:55:20.818412+0000) 2024-03-04T17:55:21.818 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:55:21.818+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:55:21.818647+0000) 2024-03-04T17:55:22.818 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:55:22.818+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:55:22.818955+0000) 2024-03-04T17:55:23.818 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:55:23.818+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:55:23.819192+0000) 2024-03-04T17:55:24.818 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:55:24.819+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:55:24.819429+0000) 2024-03-04T17:55:25.819 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:55:25.818+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:55:25.819679+0000) 2024-03-04T17:55:26.819 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:55:26.819+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:55:26.819916+0000) 2024-03-04T17:55:27.819 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:55:27.819+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:55:27.820097+0000) 2024-03-04T17:55:28.819 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:55:28.820+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:55:28.820340+0000) 2024-03-04T17:55:29.819 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:55:29.820+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:55:29.820571+0000) 2024-03-04T17:55:30.820 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:55:30.820+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:55:30.820814+0000) 2024-03-04T17:55:31.820 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:55:31.820+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:55:31.821119+0000) 2024-03-04T17:55:32.820 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:55:32.821+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:55:32.821365+0000) 2024-03-04T17:55:33.821 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:55:33.821+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:55:33.821609+0000) 2024-03-04T17:55:34.821 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:55:34.821+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:55:34.821873+0000) 2024-03-04T17:55:35.821 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:55:35.821+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:55:35.822116+0000) 2024-03-04T17:55:36.821 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:55:36.822+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:55:36.822356+0000) 2024-03-04T17:55:37.822 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:55:37.822+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:55:37.822601+0000) 2024-03-04T17:55:38.822 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:55:38.822+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:55:38.822894+0000) 2024-03-04T17:55:39.822 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:55:39.822+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:55:39.823136+0000) 2024-03-04T17:55:40.822 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:55:40.823+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:55:40.823373+0000) 2024-03-04T17:55:41.822 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:55:41.821+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:55:41.823604+0000) 2024-03-04T17:55:42.823 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:55:42.822+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:55:42.823921+0000) 2024-03-04T17:55:43.823 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:55:43.823+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:55:43.824244+0000) 2024-03-04T17:55:44.823 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:55:44.823+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:55:44.824503+0000) 2024-03-04T17:55:45.824 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:55:45.823+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:55:45.824791+0000) 2024-03-04T17:55:46.824 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:55:46.824+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:55:46.825143+0000) 2024-03-04T17:55:47.824 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:55:47.824+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:55:47.825435+0000) 2024-03-04T17:55:48.825 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:55:48.824+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:55:48.825728+0000) 2024-03-04T17:55:48.960 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:55:48.972 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:55:49.825 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:55:49.824+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:55:49.826032+0000) 2024-03-04T17:55:50.825 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:55:50.824+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:55:50.826314+0000) 2024-03-04T17:55:51.825 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:55:51.825+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:55:51.826592+0000) 2024-03-04T17:55:52.826 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:55:52.826+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:55:52.826942+0000) 2024-03-04T17:55:53.826 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:55:53.826+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:55:53.827240+0000) 2024-03-04T17:55:54.826 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:55:54.826+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:55:54.827535+0000) 2024-03-04T17:55:55.827 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:55:55.826+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:55:55.827794+0000) 2024-03-04T17:55:56.827 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:55:56.827+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:55:56.828091+0000) 2024-03-04T17:55:57.827 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:55:57.827+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:55:57.828336+0000) 2024-03-04T17:55:58.827 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:55:58.827+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:55:58.828578+0000) 2024-03-04T17:55:59.828 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:55:59.828+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:55:59.828817+0000) 2024-03-04T17:56:00.828 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:56:00.828+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:56:00.829087+0000) 2024-03-04T17:56:01.828 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:56:01.828+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:56:01.829322+0000) 2024-03-04T17:56:02.828 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:56:02.828+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:56:02.829577+0000) 2024-03-04T17:56:03.829 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:56:03.829+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:56:03.829817+0000) 2024-03-04T17:56:04.829 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:56:04.829+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:56:04.830094+0000) 2024-03-04T17:56:05.829 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:56:05.829+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:56:05.830331+0000) 2024-03-04T17:56:06.829 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:56:06.829+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:56:06.830570+0000) 2024-03-04T17:56:07.830 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:56:07.830+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:56:07.830811+0000) 2024-03-04T17:56:08.830 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:56:08.830+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:56:08.831077+0000) 2024-03-04T17:56:09.830 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:56:09.830+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:56:09.831317+0000) 2024-03-04T17:56:10.830 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:56:10.830+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:56:10.831558+0000) 2024-03-04T17:56:11.831 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:56:11.831+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:56:11.831816+0000) 2024-03-04T17:56:12.831 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:56:12.831+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:56:12.832070+0000) 2024-03-04T17:56:13.831 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:56:13.831+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:56:13.832316+0000) 2024-03-04T17:56:14.831 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:56:14.831+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:56:14.832550+0000) 2024-03-04T17:56:15.832 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:56:15.832+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:56:15.832810+0000) 2024-03-04T17:56:16.832 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:56:16.832+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:56:16.833094+0000) 2024-03-04T17:56:17.832 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:56:17.832+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:56:17.833279+0000) 2024-03-04T17:56:18.832 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:56:18.832+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:56:18.833518+0000) 2024-03-04T17:56:19.181 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:56:19.189 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:56:19.833 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:56:19.833+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:56:19.833752+0000) 2024-03-04T17:56:20.833 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:56:20.833+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:56:20.833989+0000) 2024-03-04T17:56:21.833 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:56:21.833+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:56:21.834229+0000) 2024-03-04T17:56:22.833 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:56:22.834+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:56:22.834477+0000) 2024-03-04T17:56:23.834 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:56:23.833+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:56:23.834715+0000) 2024-03-04T17:56:24.834 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:56:24.834+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:56:24.834948+0000) 2024-03-04T17:56:25.834 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:56:25.834+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:56:25.835213+0000) 2024-03-04T17:56:26.834 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:56:26.835+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:56:26.835456+0000) 2024-03-04T17:56:27.835 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:56:27.835+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:56:27.835721+0000) 2024-03-04T17:56:28.835 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:56:28.835+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:56:28.835995+0000) 2024-03-04T17:56:29.835 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:56:29.835+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:56:29.836244+0000) 2024-03-04T17:56:30.835 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:56:30.836+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:56:30.836496+0000) 2024-03-04T17:56:31.836 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:56:31.836+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:56:31.836866+0000) 2024-03-04T17:56:32.836 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:56:32.836+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:56:32.837168+0000) 2024-03-04T17:56:33.836 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:56:33.837+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:56:33.837425+0000) 2024-03-04T17:56:34.837 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:56:34.837+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:56:34.837698+0000) 2024-03-04T17:56:35.837 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:56:35.837+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:56:35.837981+0000) 2024-03-04T17:56:36.837 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:56:36.837+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:56:36.838252+0000) 2024-03-04T17:56:37.837 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:56:37.838+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:56:37.838472+0000) 2024-03-04T17:56:38.838 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:56:38.838+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:56:38.838758+0000) 2024-03-04T17:56:39.838 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:56:39.838+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:56:39.839011+0000) 2024-03-04T17:56:40.838 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:56:40.839+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:56:40.839281+0000) 2024-03-04T17:56:41.838 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:56:41.839+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:56:41.839570+0000) 2024-03-04T17:56:42.839 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:56:42.839+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:56:42.839889+0000) 2024-03-04T17:56:43.839 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:56:43.840+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:56:43.840164+0000) 2024-03-04T17:56:44.839 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:56:44.839+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:56:44.840460+0000) 2024-03-04T17:56:45.840 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:56:45.839+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:56:45.840709+0000) 2024-03-04T17:56:46.840 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:56:46.839+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:56:46.840943+0000) 2024-03-04T17:56:47.840 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:56:47.840+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:56:47.841191+0000) 2024-03-04T17:56:48.840 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:56:48.840+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:56:48.841430+0000) 2024-03-04T17:56:49.395 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:56:49.403 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:56:49.840 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:56:49.840+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:56:49.841669+0000) 2024-03-04T17:56:50.841 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:56:50.840+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:56:50.841952+0000) 2024-03-04T17:56:51.841 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:56:51.840+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:56:51.842194+0000) 2024-03-04T17:56:52.841 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:56:52.840+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:56:52.842466+0000) 2024-03-04T17:56:53.841 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:56:53.840+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:56:53.842661+0000) 2024-03-04T17:56:54.842 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:56:54.841+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:56:54.842934+0000) 2024-03-04T17:56:55.842 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:56:55.841+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:56:55.843116+0000) 2024-03-04T17:56:56.842 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:56:56.842+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:56:56.843351+0000) 2024-03-04T17:56:57.844 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:56:57.842+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:56:57.843608+0000) 2024-03-04T17:56:58.843 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:56:58.843+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:56:58.843887+0000) 2024-03-04T17:56:59.843 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:56:59.843+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:56:59.844123+0000) 2024-03-04T17:57:00.844 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:57:00.843+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:57:00.844359+0000) 2024-03-04T17:57:01.843 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:57:01.843+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:57:01.844595+0000) 2024-03-04T17:57:02.844 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:57:02.844+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:57:02.844879+0000) 2024-03-04T17:57:03.844 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:57:03.844+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:57:03.845127+0000) 2024-03-04T17:57:04.844 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:57:04.844+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:57:04.845385+0000) 2024-03-04T17:57:05.844 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:57:05.844+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:57:05.845568+0000) 2024-03-04T17:57:06.844 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:57:06.845+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:57:06.845856+0000) 2024-03-04T17:57:07.845 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:57:07.845+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:57:07.846103+0000) 2024-03-04T17:57:08.845 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:57:08.845+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:57:08.846344+0000) 2024-03-04T17:57:09.845 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:57:09.845+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:57:09.846612+0000) 2024-03-04T17:57:10.846 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:57:10.846+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:57:10.846928+0000) 2024-03-04T17:57:11.846 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:57:11.846+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:57:11.847162+0000) 2024-03-04T17:57:12.846 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:57:12.846+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:57:12.847409+0000) 2024-03-04T17:57:13.846 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:57:13.846+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:57:13.847648+0000) 2024-03-04T17:57:14.847 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:57:14.847+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:57:14.847911+0000) 2024-03-04T17:57:15.847 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:57:15.847+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:57:15.848153+0000) 2024-03-04T17:57:16.847 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:57:16.847+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:57:16.848389+0000) 2024-03-04T17:57:17.848 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:57:17.848+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:57:17.848627+0000) 2024-03-04T17:57:18.848 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:57:18.848+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:57:18.848894+0000) 2024-03-04T17:57:19.607 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:57:19.616 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:57:19.848 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:57:19.848+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:57:19.849146+0000) 2024-03-04T17:57:20.848 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:57:20.848+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:57:20.849403+0000) 2024-03-04T17:57:21.848 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:57:21.848+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:57:21.849593+0000) 2024-03-04T17:57:22.849 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:57:22.849+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:57:22.849854+0000) 2024-03-04T17:57:23.849 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:57:23.849+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:57:23.850101+0000) 2024-03-04T17:57:24.849 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:57:24.849+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:57:24.850337+0000) 2024-03-04T17:57:25.849 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:57:25.850+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:57:25.850591+0000) 2024-03-04T17:57:26.850 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:57:26.850+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:57:26.850851+0000) 2024-03-04T17:57:27.850 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:57:27.850+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:57:27.851087+0000) 2024-03-04T17:57:28.850 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:57:28.850+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:57:28.851328+0000) 2024-03-04T17:57:29.850 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:57:29.850+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:57:29.851558+0000) 2024-03-04T17:57:30.851 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:57:30.851+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:57:30.851901+0000) 2024-03-04T17:57:31.851 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:57:31.851+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:57:31.852155+0000) 2024-03-04T17:57:32.851 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:57:32.851+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:57:32.852417+0000) 2024-03-04T17:57:33.851 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:57:33.852+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:57:33.852674+0000) 2024-03-04T17:57:34.852 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:57:34.852+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:57:34.852928+0000) 2024-03-04T17:57:35.852 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:57:35.852+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:57:35.853181+0000) 2024-03-04T17:57:36.852 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:57:36.852+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:57:36.853395+0000) 2024-03-04T17:57:37.853 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:57:37.853+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:57:37.853618+0000) 2024-03-04T17:57:38.853 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:57:38.853+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:57:38.853934+0000) 2024-03-04T17:57:39.853 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:57:39.853+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:57:39.854206+0000) 2024-03-04T17:57:40.854 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:57:40.854+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:57:40.854481+0000) 2024-03-04T17:57:41.854 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:57:41.854+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:57:41.854731+0000) 2024-03-04T17:57:42.854 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:57:42.854+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:57:42.854993+0000) 2024-03-04T17:57:43.854 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:57:43.855+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:57:43.855278+0000) 2024-03-04T17:57:44.854 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:57:44.855+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:57:44.855541+0000) 2024-03-04T17:57:45.855 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:57:45.855+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:57:45.855790+0000) 2024-03-04T17:57:46.855 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:57:46.855+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:57:46.856097+0000) 2024-03-04T17:57:47.855 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:57:47.856+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:57:47.856349+0000) 2024-03-04T17:57:48.855 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:57:48.854+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:57:48.856619+0000) 2024-03-04T17:57:49.754 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:57:49.762 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:57:49.855 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:57:49.855+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:57:49.856859+0000) 2024-03-04T17:57:50.856 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:57:50.856+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:57:50.857103+0000) 2024-03-04T17:57:51.856 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:57:51.855+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:57:51.857376+0000) 2024-03-04T17:57:52.856 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:57:52.856+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:57:52.857645+0000) 2024-03-04T17:57:53.857 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:57:53.856+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:57:53.857928+0000) 2024-03-04T17:57:54.857 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:57:54.857+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:57:54.858167+0000) 2024-03-04T17:57:55.857 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:57:55.857+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:57:55.858405+0000) 2024-03-04T17:57:56.857 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:57:56.857+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:57:56.858642+0000) 2024-03-04T17:57:57.858 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:57:57.857+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:57:57.858913+0000) 2024-03-04T17:57:58.858 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:57:58.858+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:57:58.859154+0000) 2024-03-04T17:57:59.858 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:57:59.858+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:57:59.859388+0000) 2024-03-04T17:58:00.858 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:58:00.858+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:58:00.859639+0000) 2024-03-04T17:58:01.859 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:58:01.859+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:58:01.859926+0000) 2024-03-04T17:58:02.860 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:58:02.859+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:58:02.860174+0000) 2024-03-04T17:58:03.859 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:58:03.859+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:58:03.860411+0000) 2024-03-04T17:58:04.859 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:58:04.859+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:58:04.860649+0000) 2024-03-04T17:58:05.860 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:58:05.860+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:58:05.860946+0000) 2024-03-04T17:58:06.860 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:58:06.860+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:58:06.861184+0000) 2024-03-04T17:58:07.860 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:58:07.860+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:58:07.861423+0000) 2024-03-04T17:58:08.860 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:58:08.860+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:58:08.861609+0000) 2024-03-04T17:58:09.861 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:58:09.861+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:58:09.861910+0000) 2024-03-04T17:58:10.861 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:58:10.861+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:58:10.862168+0000) 2024-03-04T17:58:11.861 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:58:11.861+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:58:11.862406+0000) 2024-03-04T17:58:12.861 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:58:12.861+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:58:12.862673+0000) 2024-03-04T17:58:13.862 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:58:13.862+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:58:13.862938+0000) 2024-03-04T17:58:14.862 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:58:14.862+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:58:14.863169+0000) 2024-03-04T17:58:15.862 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:58:15.862+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:58:15.863413+0000) 2024-03-04T17:58:16.862 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:58:16.863+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:58:16.863675+0000) 2024-03-04T17:58:17.863 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:58:17.863+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:58:17.863944+0000) 2024-03-04T17:58:18.863 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:58:18.863+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:58:18.864239+0000) 2024-03-04T17:58:19.863 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:58:19.863+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:58:19.864479+0000) 2024-03-04T17:58:19.918 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:58:19.927 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:58:20.864 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:58:20.863+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:58:20.864665+0000) 2024-03-04T17:58:21.864 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:58:21.864+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:58:21.864944+0000) 2024-03-04T17:58:22.864 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:58:22.864+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:58:22.865194+0000) 2024-03-04T17:58:23.864 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:58:23.864+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:58:23.865435+0000) 2024-03-04T17:58:24.865 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:58:24.865+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:58:24.865671+0000) 2024-03-04T17:58:25.865 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:58:25.865+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:58:25.865955+0000) 2024-03-04T17:58:26.865 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:58:26.865+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:58:26.866201+0000) 2024-03-04T17:58:27.865 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:58:27.865+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:58:27.866437+0000) 2024-03-04T17:58:28.866 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:58:28.865+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:58:28.866696+0000) 2024-03-04T17:58:29.866 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:58:29.866+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:58:29.866952+0000) 2024-03-04T17:58:30.866 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:58:30.866+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:58:30.867193+0000) 2024-03-04T17:58:31.866 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:58:31.866+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:58:31.867422+0000) 2024-03-04T17:58:32.867 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:58:32.867+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:58:32.867667+0000) 2024-03-04T17:58:33.867 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:58:33.867+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:58:33.867948+0000) 2024-03-04T17:58:34.867 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:58:34.867+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:58:34.868140+0000) 2024-03-04T17:58:35.867 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:58:35.868+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:58:35.868374+0000) 2024-03-04T17:58:36.868 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:58:36.868+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:58:36.868618+0000) 2024-03-04T17:58:37.868 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:58:37.868+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:58:37.868882+0000) 2024-03-04T17:58:38.868 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:58:38.868+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:58:38.869126+0000) 2024-03-04T17:58:39.868 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:58:39.869+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:58:39.869383+0000) 2024-03-04T17:58:40.869 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:58:40.869+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:58:40.869639+0000) 2024-03-04T17:58:41.869 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:58:41.869+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:58:41.869913+0000) 2024-03-04T17:58:42.869 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:58:42.869+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:58:42.870157+0000) 2024-03-04T17:58:43.870 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:58:43.870+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:58:43.870397+0000) 2024-03-04T17:58:44.870 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:58:44.870+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:58:44.870635+0000) 2024-03-04T17:58:45.870 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:58:45.870+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:58:45.870922+0000) 2024-03-04T17:58:46.870 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:58:46.870+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:58:46.871177+0000) 2024-03-04T17:58:47.870 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:58:47.871+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:58:47.871408+0000) 2024-03-04T17:58:48.871 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:58:48.870+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:58:48.871663+0000) 2024-03-04T17:58:49.871 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:58:49.871+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:58:49.871941+0000) 2024-03-04T17:58:50.123 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:58:50.130 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:58:50.871 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:58:50.872+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:58:50.872183+0000) 2024-03-04T17:58:51.871 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:58:51.871+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:58:51.872432+0000) 2024-03-04T17:58:52.871 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:58:52.871+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:58:52.872682+0000) 2024-03-04T17:58:53.872 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:58:53.871+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:58:53.872928+0000) 2024-03-04T17:58:54.872 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:58:54.872+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:58:54.873167+0000) 2024-03-04T17:58:55.873 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:58:55.872+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:58:55.873402+0000) 2024-03-04T17:58:56.872 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:58:56.872+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:58:56.873637+0000) 2024-03-04T17:58:57.873 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:58:57.872+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:58:57.873910+0000) 2024-03-04T17:58:58.873 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:58:58.872+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:58:58.874099+0000) 2024-03-04T17:58:59.873 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:58:59.872+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:58:59.874356+0000) 2024-03-04T17:59:00.873 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:59:00.873+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:59:00.874553+0000) 2024-03-04T17:59:01.874 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:59:01.873+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:59:01.874885+0000) 2024-03-04T17:59:02.874 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:59:02.873+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:59:02.875070+0000) 2024-03-04T17:59:03.874 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:59:03.874+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:59:03.875309+0000) 2024-03-04T17:59:04.874 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:59:04.874+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:59:04.875538+0000) 2024-03-04T17:59:05.875 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:59:05.874+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:59:05.875785+0000) 2024-03-04T17:59:06.875 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:59:06.875+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:59:06.876027+0000) 2024-03-04T17:59:07.875 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:59:07.875+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:59:07.876261+0000) 2024-03-04T17:59:08.875 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:59:08.875+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:59:08.876499+0000) 2024-03-04T17:59:09.876 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:59:09.875+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:59:09.876766+0000) 2024-03-04T17:59:10.876 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:59:10.876+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:59:10.877021+0000) 2024-03-04T17:59:11.876 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:59:11.876+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:59:11.877260+0000) 2024-03-04T17:59:12.876 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:59:12.876+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:59:12.877463+0000) 2024-03-04T17:59:13.876 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:59:13.876+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:59:13.877739+0000) 2024-03-04T17:59:14.877 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:59:14.877+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:59:14.877976+0000) 2024-03-04T17:59:15.877 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:59:15.877+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:59:15.878213+0000) 2024-03-04T17:59:16.877 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:59:16.877+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:59:16.878452+0000) 2024-03-04T17:59:17.878 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:59:17.877+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:59:17.878629+0000) 2024-03-04T17:59:18.878 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:59:18.878+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:59:18.878960+0000) 2024-03-04T17:59:19.878 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:59:19.878+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:59:19.879194+0000) 2024-03-04T17:59:20.328 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:59:20.336 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:59:20.878 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:59:20.878+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:59:20.879432+0000) 2024-03-04T17:59:21.879 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:59:21.879+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:59:21.879671+0000) 2024-03-04T17:59:22.879 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:59:22.879+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:59:22.879907+0000) 2024-03-04T17:59:23.879 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:59:23.879+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:59:23.880145+0000) 2024-03-04T17:59:24.879 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:59:24.879+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:59:24.880379+0000) 2024-03-04T17:59:25.879 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:59:25.879+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:59:25.880557+0000) 2024-03-04T17:59:26.880 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:59:26.880+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:59:26.880811+0000) 2024-03-04T17:59:27.880 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:59:27.880+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:59:27.881106+0000) 2024-03-04T17:59:28.880 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:59:28.880+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:59:28.881344+0000) 2024-03-04T17:59:29.881 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:59:29.881+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:59:29.881604+0000) 2024-03-04T17:59:30.881 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:59:30.881+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:59:30.881920+0000) 2024-03-04T17:59:31.881 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:59:31.881+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:59:31.882150+0000) 2024-03-04T17:59:32.881 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:59:32.881+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:59:32.882385+0000) 2024-03-04T17:59:33.881 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:59:33.882+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:59:33.882626+0000) 2024-03-04T17:59:34.882 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:59:34.882+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:59:34.882893+0000) 2024-03-04T17:59:35.882 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:59:35.882+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:59:35.883128+0000) 2024-03-04T17:59:36.882 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:59:36.882+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:59:36.883364+0000) 2024-03-04T17:59:37.882 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:59:37.882+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:59:37.883542+0000) 2024-03-04T17:59:38.883 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:59:38.883+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:59:38.883780+0000) 2024-03-04T17:59:39.883 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:59:39.883+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:59:39.884020+0000) 2024-03-04T17:59:40.883 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:59:40.883+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:59:40.884260+0000) 2024-03-04T17:59:41.884 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:59:41.884+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:59:41.884515+0000) 2024-03-04T17:59:42.884 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:59:42.884+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:59:42.884779+0000) 2024-03-04T17:59:43.884 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:59:43.884+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:59:43.885020+0000) 2024-03-04T17:59:44.884 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:59:44.884+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:59:44.885257+0000) 2024-03-04T17:59:45.884 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:59:45.885+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:59:45.885498+0000) 2024-03-04T17:59:46.885 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:59:46.885+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:59:46.885735+0000) 2024-03-04T17:59:47.885 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:59:47.885+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:59:47.885974+0000) 2024-03-04T17:59:48.885 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:59:48.885+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:59:48.886211+0000) 2024-03-04T17:59:49.885 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:59:49.886+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:59:49.886536+0000) 2024-03-04T17:59:50.541 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:59:50.549 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T17:59:50.885 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:59:50.886+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:59:50.886776+0000) 2024-03-04T17:59:51.886 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:59:51.886+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:59:51.887016+0000) 2024-03-04T17:59:52.886 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:59:52.887+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:59:52.887254+0000) 2024-03-04T17:59:53.886 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:59:53.886+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:59:53.887507+0000) 2024-03-04T17:59:54.887 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:59:54.887+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:59:54.887748+0000) 2024-03-04T17:59:55.887 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:59:55.886+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:59:55.887987+0000) 2024-03-04T17:59:56.887 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:59:56.887+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:59:56.888229+0000) 2024-03-04T17:59:57.888 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:59:57.887+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:59:57.888471+0000) 2024-03-04T17:59:58.888 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:59:58.887+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:59:58.888708+0000) 2024-03-04T17:59:59.888 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T17:59:59.887+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T16:59:59.888952+0000) 2024-03-04T18:00:00.888 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:00:00.888+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:00:00.889192+0000) 2024-03-04T18:00:01.888 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:00:01.887+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:00:01.889374+0000) 2024-03-04T18:00:02.889 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:00:02.888+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:00:02.889630+0000) 2024-03-04T18:00:03.889 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:00:03.888+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:00:03.889919+0000) 2024-03-04T18:00:04.889 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:00:04.889+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:00:04.890159+0000) 2024-03-04T18:00:05.889 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:00:05.889+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:00:05.890399+0000) 2024-03-04T18:00:06.890 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:00:06.889+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:00:06.890654+0000) 2024-03-04T18:00:07.890 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:00:07.890+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:00:07.890942+0000) 2024-03-04T18:00:08.890 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:00:08.890+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:00:08.891184+0000) 2024-03-04T18:00:09.890 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:00:09.890+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:00:09.891423+0000) 2024-03-04T18:00:10.891 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:00:10.890+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:00:10.891659+0000) 2024-03-04T18:00:11.891 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:00:11.891+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:00:11.891954+0000) 2024-03-04T18:00:12.891 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:00:12.891+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:00:12.892222+0000) 2024-03-04T18:00:13.891 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:00:13.891+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:00:13.892462+0000) 2024-03-04T18:00:14.891 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:00:14.891+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:00:14.892710+0000) 2024-03-04T18:00:15.892 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:00:15.892+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:00:15.892939+0000) 2024-03-04T18:00:16.892 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:00:16.892+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:00:16.893183+0000) 2024-03-04T18:00:17.893 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:00:17.892+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:00:17.893420+0000) 2024-03-04T18:00:18.893 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:00:18.892+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:00:18.893658+0000) 2024-03-04T18:00:19.893 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:00:19.893+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:00:19.893954+0000) 2024-03-04T18:00:20.755 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:00:20.763 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:00:20.893 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:00:20.893+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:00:20.894228+0000) 2024-03-04T18:00:21.893 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:00:21.893+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:00:21.894476+0000) 2024-03-04T18:00:22.894 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:00:22.894+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:00:22.894728+0000) 2024-03-04T18:00:23.894 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:00:23.894+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:00:23.894969+0000) 2024-03-04T18:00:24.894 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:00:24.894+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:00:24.895210+0000) 2024-03-04T18:00:25.894 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:00:25.894+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:00:25.895467+0000) 2024-03-04T18:00:26.895 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:00:26.895+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:00:26.895708+0000) 2024-03-04T18:00:27.895 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:00:27.895+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:00:27.895983+0000) 2024-03-04T18:00:28.895 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:00:28.895+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:00:28.896229+0000) 2024-03-04T18:00:29.896 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:00:29.895+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:00:29.896468+0000) 2024-03-04T18:00:30.896 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:00:30.895+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:00:30.896652+0000) 2024-03-04T18:00:31.896 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:00:31.896+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:00:31.896955+0000) 2024-03-04T18:00:32.896 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:00:32.896+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:00:32.897199+0000) 2024-03-04T18:00:33.897 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:00:33.896+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:00:33.897442+0000) 2024-03-04T18:00:34.897 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:00:34.897+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:00:34.897686+0000) 2024-03-04T18:00:35.897 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:00:35.897+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:00:35.897923+0000) 2024-03-04T18:00:36.897 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:00:36.897+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:00:36.898155+0000) 2024-03-04T18:00:37.897 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:00:37.897+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:00:37.898345+0000) 2024-03-04T18:00:38.897 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:00:38.898+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:00:38.898587+0000) 2024-03-04T18:00:39.898 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:00:39.898+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:00:39.898855+0000) 2024-03-04T18:00:40.898 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:00:40.898+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:00:40.899102+0000) 2024-03-04T18:00:41.898 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:00:41.898+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:00:41.899338+0000) 2024-03-04T18:00:42.899 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:00:42.899+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:00:42.899616+0000) 2024-03-04T18:00:43.899 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:00:43.899+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:00:43.899922+0000) 2024-03-04T18:00:44.900 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:00:44.899+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:00:44.900116+0000) 2024-03-04T18:00:45.899 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:00:45.900+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:00:45.900352+0000) 2024-03-04T18:00:46.899 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:00:46.900+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:00:46.900610+0000) 2024-03-04T18:00:47.900 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:00:47.900+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:00:47.900816+0000) 2024-03-04T18:00:48.900 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:00:48.900+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:00:48.901090+0000) 2024-03-04T18:00:49.900 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:00:49.901+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:00:49.901333+0000) 2024-03-04T18:00:50.900 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:00:50.901+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:00:50.901572+0000) 2024-03-04T18:00:50.909 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:00:50.918 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:00:51.901 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:00:51.901+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:00:51.901811+0000) 2024-03-04T18:00:52.901 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:00:52.901+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:00:52.902095+0000) 2024-03-04T18:00:53.901 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:00:53.902+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:00:53.902341+0000) 2024-03-04T18:00:54.901 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:00:54.902+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:00:54.902593+0000) 2024-03-04T18:00:55.902 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:00:55.902+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:00:55.902866+0000) 2024-03-04T18:00:56.902 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:00:56.902+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:00:56.903107+0000) 2024-03-04T18:00:57.902 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:00:57.903+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:00:57.903349+0000) 2024-03-04T18:00:58.902 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:00:58.902+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:00:58.903599+0000) 2024-03-04T18:00:59.903 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:00:59.902+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:00:59.903921+0000) 2024-03-04T18:01:00.903 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:01:00.903+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:01:00.904170+0000) 2024-03-04T18:01:01.903 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:01:01.903+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:01:01.904408+0000) 2024-03-04T18:01:02.904 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:01:02.903+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:01:02.904656+0000) 2024-03-04T18:01:03.904 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:01:03.903+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:01:03.904932+0000) 2024-03-04T18:01:04.905 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:01:04.904+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:01:04.905174+0000) 2024-03-04T18:01:05.904 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:01:05.904+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:01:05.905414+0000) 2024-03-04T18:01:06.905 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:01:06.904+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:01:06.905660+0000) 2024-03-04T18:01:07.905 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:01:07.904+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:01:07.905951+0000) 2024-03-04T18:01:08.906 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:01:08.905+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:01:08.906195+0000) 2024-03-04T18:01:09.905 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:01:09.905+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:01:09.906445+0000) 2024-03-04T18:01:10.906 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:01:10.905+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:01:10.906678+0000) 2024-03-04T18:01:11.906 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:01:11.906+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:01:11.906960+0000) 2024-03-04T18:01:12.906 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:01:12.906+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:01:12.907203+0000) 2024-03-04T18:01:13.906 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:01:13.906+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:01:13.907445+0000) 2024-03-04T18:01:14.907 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:01:14.906+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:01:14.907695+0000) 2024-03-04T18:01:15.907 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:01:15.907+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:01:15.907972+0000) 2024-03-04T18:01:16.907 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:01:16.907+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:01:16.908213+0000) 2024-03-04T18:01:17.907 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:01:17.907+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:01:17.908457+0000) 2024-03-04T18:01:18.908 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:01:18.907+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:01:18.908696+0000) 2024-03-04T18:01:19.908 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:01:19.908+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:01:19.908939+0000) 2024-03-04T18:01:20.908 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:01:20.908+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:01:20.909182+0000) 2024-03-04T18:01:21.051 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:01:21.059 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:01:21.908 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:01:21.908+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:01:21.909422+0000) 2024-03-04T18:01:22.909 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:01:22.908+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:01:22.909676+0000) 2024-03-04T18:01:23.909 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:01:23.909+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:01:23.909927+0000) 2024-03-04T18:01:24.909 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:01:24.909+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:01:24.910167+0000) 2024-03-04T18:01:25.910 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:01:25.909+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:01:25.910406+0000) 2024-03-04T18:01:26.910 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:01:26.909+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:01:26.910644+0000) 2024-03-04T18:01:27.910 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:01:27.910+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:01:27.910973+0000) 2024-03-04T18:01:28.910 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:01:28.910+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:01:28.911209+0000) 2024-03-04T18:01:29.910 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:01:29.910+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:01:29.911460+0000) 2024-03-04T18:01:30.911 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:01:30.910+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:01:30.911700+0000) 2024-03-04T18:01:31.911 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:01:31.911+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:01:31.911940+0000) 2024-03-04T18:01:32.911 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:01:32.911+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:01:32.912208+0000) 2024-03-04T18:01:33.911 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:01:33.911+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:01:33.912453+0000) 2024-03-04T18:01:34.912 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:01:34.912+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:01:34.912693+0000) 2024-03-04T18:01:35.912 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:01:35.912+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:01:35.912990+0000) 2024-03-04T18:01:36.912 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:01:36.912+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:01:36.913247+0000) 2024-03-04T18:01:37.912 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:01:37.912+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:01:37.913501+0000) 2024-03-04T18:01:38.912 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:01:38.913+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:01:38.913706+0000) 2024-03-04T18:01:39.913 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:01:39.913+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:01:39.913994+0000) 2024-03-04T18:01:40.913 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:01:40.913+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:01:40.914230+0000) 2024-03-04T18:01:41.913 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:01:41.913+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:01:41.914467+0000) 2024-03-04T18:01:42.914 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:01:42.914+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:01:42.914744+0000) 2024-03-04T18:01:43.914 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:01:43.914+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:01:43.914980+0000) 2024-03-04T18:01:44.914 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:01:44.914+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:01:44.915179+0000) 2024-03-04T18:01:45.915 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:01:45.914+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:01:45.915433+0000) 2024-03-04T18:01:46.915 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:01:46.915+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:01:46.915713+0000) 2024-03-04T18:01:47.915 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:01:47.915+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:01:47.915959+0000) 2024-03-04T18:01:48.915 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:01:48.915+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:01:48.916198+0000) 2024-03-04T18:01:49.915 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:01:49.916+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:01:49.916438+0000) 2024-03-04T18:01:50.916 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:01:50.916+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:01:50.916709+0000) 2024-03-04T18:01:51.277 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:01:51.289 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:01:51.916 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:01:51.916+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:01:51.916951+0000) 2024-03-04T18:01:52.916 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:01:52.916+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:01:52.917193+0000) 2024-03-04T18:01:53.916 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:01:53.917+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:01:53.917429+0000) 2024-03-04T18:01:54.917 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:01:54.916+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:01:54.917699+0000) 2024-03-04T18:01:55.917 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:01:55.917+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:01:55.917952+0000) 2024-03-04T18:01:56.917 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:01:56.917+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:01:56.918188+0000) 2024-03-04T18:01:57.917 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:01:57.917+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:01:57.918415+0000) 2024-03-04T18:01:58.918 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:01:58.918+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:01:58.918971+0000) 2024-03-04T18:01:59.918 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:01:59.919+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:01:59.919267+0000) 2024-03-04T18:02:00.918 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:02:00.919+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:02:00.919563+0000) 2024-03-04T18:02:01.919 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:02:01.918+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:02:01.919901+0000) 2024-03-04T18:02:02.919 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:02:02.919+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:02:02.920191+0000) 2024-03-04T18:02:03.919 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:02:03.919+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:02:03.920492+0000) 2024-03-04T18:02:04.920 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:02:04.919+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:02:04.920877+0000) 2024-03-04T18:02:05.920 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:02:05.920+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:02:05.921167+0000) 2024-03-04T18:02:06.920 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:02:06.919+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:02:06.921415+0000) 2024-03-04T18:02:07.921 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:02:07.919+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:02:07.921711+0000) 2024-03-04T18:02:08.921 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:02:08.920+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:02:08.922004+0000) 2024-03-04T18:02:09.921 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:02:09.920+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:02:09.922304+0000) 2024-03-04T18:02:10.922 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:02:10.921+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:02:10.922617+0000) 2024-03-04T18:02:11.922 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:02:11.921+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:02:11.922936+0000) 2024-03-04T18:02:12.922 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:02:12.922+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:02:12.923231+0000) 2024-03-04T18:02:13.922 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:02:13.922+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:02:13.923528+0000) 2024-03-04T18:02:14.923 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:02:14.923+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:02:14.923817+0000) 2024-03-04T18:02:15.923 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:02:15.923+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:02:15.924207+0000) 2024-03-04T18:02:16.923 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:02:16.923+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:02:16.924500+0000) 2024-03-04T18:02:17.924 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:02:17.923+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:02:17.924790+0000) 2024-03-04T18:02:18.924 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:02:18.924+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:02:18.925117+0000) 2024-03-04T18:02:19.924 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:02:19.924+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:02:19.925405+0000) 2024-03-04T18:02:20.925 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:02:20.924+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:02:20.925677+0000) 2024-03-04T18:02:21.499 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:02:21.506 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:02:21.925 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:02:21.925+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:02:21.925959+0000) 2024-03-04T18:02:22.926 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:02:22.925+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:02:22.926264+0000) 2024-03-04T18:02:23.925 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:02:23.925+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:02:23.926553+0000) 2024-03-04T18:02:24.926 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:02:24.926+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:02:24.926863+0000) 2024-03-04T18:02:25.926 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:02:25.926+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:02:25.927146+0000) 2024-03-04T18:02:26.926 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:02:26.926+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:02:26.927437+0000) 2024-03-04T18:02:27.926 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:02:27.926+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:02:27.927722+0000) 2024-03-04T18:02:28.927 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:02:28.927+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:02:28.928018+0000) 2024-03-04T18:02:29.927 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:02:29.927+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:02:29.928377+0000) 2024-03-04T18:02:30.928 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:02:30.927+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:02:30.928690+0000) 2024-03-04T18:02:31.928 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:02:31.928+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:02:31.928984+0000) 2024-03-04T18:02:32.928 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:02:32.928+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:02:32.929285+0000) 2024-03-04T18:02:33.928 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:02:33.928+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:02:33.929522+0000) 2024-03-04T18:02:34.929 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:02:34.929+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:02:34.929893+0000) 2024-03-04T18:02:35.929 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:02:35.929+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:02:35.930206+0000) 2024-03-04T18:02:36.929 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:02:36.929+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:02:36.930452+0000) 2024-03-04T18:02:37.930 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:02:37.929+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:02:37.930682+0000) 2024-03-04T18:02:38.930 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:02:38.930+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:02:38.930962+0000) 2024-03-04T18:02:39.930 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:02:39.930+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:02:39.931243+0000) 2024-03-04T18:02:40.932 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:02:40.930+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:02:40.931525+0000) 2024-03-04T18:02:41.931 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:02:41.931+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:02:41.931803+0000) 2024-03-04T18:02:42.931 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:02:42.931+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:02:42.932176+0000) 2024-03-04T18:02:43.931 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:02:43.932+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:02:43.932493+0000) 2024-03-04T18:02:44.932 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:02:44.932+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:02:44.932790+0000) 2024-03-04T18:02:45.932 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:02:45.932+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:02:45.933116+0000) 2024-03-04T18:02:46.932 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:02:46.932+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:02:46.933413+0000) 2024-03-04T18:02:47.933 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:02:47.932+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:02:47.933715+0000) 2024-03-04T18:02:48.933 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:02:48.933+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:02:48.934009+0000) 2024-03-04T18:02:49.933 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:02:49.933+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:02:49.934305+0000) 2024-03-04T18:02:50.934 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:02:50.933+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:02:50.934605+0000) 2024-03-04T18:02:51.713 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:02:51.721 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:02:51.934 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:02:51.934+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:02:51.934958+0000) 2024-03-04T18:02:52.934 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:02:52.934+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:02:52.935253+0000) 2024-03-04T18:02:53.935 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:02:53.935+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:02:53.935542+0000) 2024-03-04T18:02:54.935 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:02:54.935+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:02:54.935863+0000) 2024-03-04T18:02:55.935 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:02:55.935+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:02:55.936144+0000) 2024-03-04T18:02:56.936 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:02:56.936+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:02:56.936434+0000) 2024-03-04T18:02:57.936 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:02:57.936+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:02:57.936722+0000) 2024-03-04T18:02:58.936 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:02:58.936+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:02:58.937012+0000) 2024-03-04T18:02:59.936 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:02:59.936+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:02:59.937257+0000) 2024-03-04T18:03:00.936 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:03:00.937+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:03:00.937576+0000) 2024-03-04T18:03:01.938 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:03:01.937+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:03:01.937926+0000) 2024-03-04T18:03:02.937 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:03:02.938+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:03:02.938229+0000) 2024-03-04T18:03:03.937 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:03:03.937+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:03:03.938454+0000) 2024-03-04T18:03:04.938 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:03:04.937+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:03:04.938752+0000) 2024-03-04T18:03:05.938 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:03:05.937+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:03:05.938969+0000) 2024-03-04T18:03:06.938 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:03:06.938+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:03:06.939256+0000) 2024-03-04T18:03:07.938 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:03:07.938+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:03:07.939585+0000) 2024-03-04T18:03:08.939 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:03:08.938+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:03:08.939933+0000) 2024-03-04T18:03:09.939 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:03:09.939+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:03:09.940242+0000) 2024-03-04T18:03:10.939 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:03:10.939+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:03:10.940532+0000) 2024-03-04T18:03:11.940 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:03:11.939+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:03:11.940914+0000) 2024-03-04T18:03:12.940 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:03:12.940+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:03:12.941211+0000) 2024-03-04T18:03:13.940 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:03:13.940+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:03:13.941490+0000) 2024-03-04T18:03:14.943 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:03:14.940+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:03:14.941775+0000) 2024-03-04T18:03:15.941 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:03:15.941+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:03:15.942053+0000) 2024-03-04T18:03:16.941 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:03:16.941+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:03:16.942333+0000) 2024-03-04T18:03:17.941 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:03:17.941+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:03:17.942623+0000) 2024-03-04T18:03:18.942 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:03:18.942+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:03:18.942939+0000) 2024-03-04T18:03:19.942 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:03:19.942+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:03:19.943179+0000) 2024-03-04T18:03:20.942 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:03:20.942+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:03:20.943484+0000) 2024-03-04T18:03:21.876 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:03:21.884 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:03:21.942 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:03:21.942+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:03:21.943710+0000) 2024-03-04T18:03:22.943 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:03:22.943+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:03:22.943975+0000) 2024-03-04T18:03:23.943 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:03:23.943+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:03:23.944267+0000) 2024-03-04T18:03:24.944 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:03:24.943+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:03:24.944565+0000) 2024-03-04T18:03:25.944 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:03:25.944+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:03:25.944902+0000) 2024-03-04T18:03:26.944 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:03:26.944+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:03:26.945219+0000) 2024-03-04T18:03:27.944 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:03:27.944+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:03:27.945524+0000) 2024-03-04T18:03:28.945 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:03:28.945+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:03:28.945812+0000) 2024-03-04T18:03:29.945 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:03:29.945+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:03:29.946154+0000) 2024-03-04T18:03:30.945 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:03:30.945+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:03:30.946448+0000) 2024-03-04T18:03:31.946 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:03:31.946+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:03:31.946739+0000) 2024-03-04T18:03:32.946 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:03:32.946+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:03:32.947044+0000) 2024-03-04T18:03:33.946 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:03:33.946+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:03:33.947340+0000) 2024-03-04T18:03:34.946 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:03:34.946+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:03:34.947634+0000) 2024-03-04T18:03:35.947 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:03:35.947+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:03:35.947959+0000) 2024-03-04T18:03:36.947 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:03:36.947+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:03:36.948240+0000) 2024-03-04T18:03:37.947 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:03:37.947+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:03:37.948531+0000) 2024-03-04T18:03:38.948 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:03:38.948+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:03:38.948813+0000) 2024-03-04T18:03:39.948 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:03:39.948+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:03:39.949205+0000) 2024-03-04T18:03:40.948 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:03:40.948+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:03:40.949496+0000) 2024-03-04T18:03:41.949 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:03:41.949+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:03:41.949786+0000) 2024-03-04T18:03:42.949 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:03:42.949+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:03:42.950192+0000) 2024-03-04T18:03:43.949 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:03:43.949+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:03:43.950432+0000) 2024-03-04T18:03:44.950 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:03:44.950+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:03:44.950700+0000) 2024-03-04T18:03:45.950 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:03:45.950+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:03:45.950967+0000) 2024-03-04T18:03:46.951 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:03:46.950+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:03:46.951197+0000) 2024-03-04T18:03:47.950 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:03:47.951+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:03:47.951476+0000) 2024-03-04T18:03:48.951 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:03:48.951+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:03:48.951767+0000) 2024-03-04T18:03:49.951 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:03:49.951+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:03:49.952060+0000) 2024-03-04T18:03:50.951 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:03:50.951+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:03:50.952344+0000) 2024-03-04T18:03:51.951 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:03:51.951+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:03:51.952569+0000) 2024-03-04T18:03:52.089 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:03:52.097 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:03:52.955 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:03:52.952+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:03:52.952927+0000) 2024-03-04T18:03:53.952 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:03:53.952+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:03:53.953179+0000) 2024-03-04T18:03:54.952 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:03:54.952+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:03:54.953400+0000) 2024-03-04T18:03:55.952 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:03:55.952+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:03:55.953635+0000) 2024-03-04T18:03:56.953 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:03:56.953+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:03:56.953965+0000) 2024-03-04T18:03:57.953 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:03:57.953+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:03:57.954255+0000) 2024-03-04T18:03:58.953 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:03:58.954+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:03:58.954546+0000) 2024-03-04T18:03:59.954 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:03:59.954+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:03:59.954921+0000) 2024-03-04T18:04:00.954 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:04:00.954+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:04:00.955221+0000) 2024-03-04T18:04:01.954 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:04:01.955+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:04:01.955513+0000) 2024-03-04T18:04:02.955 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:04:02.955+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:04:02.955891+0000) 2024-03-04T18:04:03.955 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:04:03.955+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:04:03.956148+0000) 2024-03-04T18:04:04.955 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:04:04.956+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:04:04.956433+0000) 2024-03-04T18:04:05.956 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:04:05.955+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:04:05.956752+0000) 2024-03-04T18:04:06.956 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:04:06.956+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:04:06.957043+0000) 2024-03-04T18:04:07.956 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:04:07.956+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:04:07.957333+0000) 2024-03-04T18:04:08.957 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:04:08.956+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:04:08.957625+0000) 2024-03-04T18:04:09.958 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:04:09.956+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:04:09.957920+0000) 2024-03-04T18:04:10.957 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:04:10.957+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:04:10.958176+0000) 2024-03-04T18:04:11.957 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:04:11.957+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:04:11.958412+0000) 2024-03-04T18:04:12.957 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:04:12.957+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:04:12.958660+0000) 2024-03-04T18:04:13.958 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:04:13.957+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:04:13.958943+0000) 2024-03-04T18:04:14.958 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:04:14.957+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:04:14.959156+0000) 2024-03-04T18:04:15.958 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:04:15.958+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:04:15.959436+0000) 2024-03-04T18:04:16.959 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:04:16.958+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:04:16.959744+0000) 2024-03-04T18:04:17.960 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:04:17.959+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:04:17.960021+0000) 2024-03-04T18:04:18.959 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:04:18.959+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:04:18.960325+0000) 2024-03-04T18:04:19.959 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:04:19.959+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:04:19.960586+0000) 2024-03-04T18:04:20.960 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:04:20.960+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:04:20.960875+0000) 2024-03-04T18:04:21.960 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:04:21.960+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:04:21.961121+0000) 2024-03-04T18:04:22.235 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:04:22.243 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:04:22.960 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:04:22.960+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:04:22.961370+0000) 2024-03-04T18:04:23.961 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:04:23.960+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:04:23.961608+0000) 2024-03-04T18:04:24.961 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:04:24.961+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:04:24.961899+0000) 2024-03-04T18:04:25.961 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:04:25.961+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:04:25.962155+0000) 2024-03-04T18:04:26.961 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:04:26.961+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:04:26.962408+0000) 2024-03-04T18:04:27.962 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:04:27.961+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:04:27.962661+0000) 2024-03-04T18:04:28.962 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:04:28.962+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:04:28.962954+0000) 2024-03-04T18:04:29.962 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:04:29.962+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:04:29.963157+0000) 2024-03-04T18:04:30.962 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:04:30.962+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:04:30.963413+0000) 2024-03-04T18:04:31.963 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:04:31.962+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:04:31.963668+0000) 2024-03-04T18:04:32.963 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:04:32.963+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:04:32.963932+0000) 2024-03-04T18:04:33.963 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:04:33.963+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:04:33.964190+0000) 2024-03-04T18:04:34.963 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:04:34.963+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:04:34.964440+0000) 2024-03-04T18:04:35.964 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:04:35.964+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:04:35.964714+0000) 2024-03-04T18:04:36.964 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:04:36.964+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:04:36.964990+0000) 2024-03-04T18:04:37.964 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:04:37.964+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:04:37.965239+0000) 2024-03-04T18:04:38.964 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:04:38.964+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:04:38.965492+0000) 2024-03-04T18:04:39.965 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:04:39.965+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:04:39.965746+0000) 2024-03-04T18:04:40.965 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:04:40.965+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:04:40.966037+0000) 2024-03-04T18:04:41.965 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:04:41.965+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:04:41.966292+0000) 2024-03-04T18:04:42.965 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:04:42.965+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:04:42.966563+0000) 2024-03-04T18:04:43.966 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:04:43.965+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:04:43.966761+0000) 2024-03-04T18:04:44.966 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:04:44.966+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:04:44.967017+0000) 2024-03-04T18:04:45.966 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:04:45.966+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:04:45.967282+0000) 2024-03-04T18:04:46.967 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:04:46.967+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:04:46.967541+0000) 2024-03-04T18:04:47.967 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:04:47.967+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:04:47.967804+0000) 2024-03-04T18:04:48.967 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:04:48.967+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:04:48.968095+0000) 2024-03-04T18:04:49.967 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:04:49.967+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:04:49.968348+0000) 2024-03-04T18:04:50.968 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:04:50.968+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:04:50.968607+0000) 2024-03-04T18:04:51.968 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:04:51.968+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:04:51.968895+0000) 2024-03-04T18:04:52.440 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:04:52.449 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:04:52.968 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:04:52.968+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:04:52.969160+0000) 2024-03-04T18:04:53.968 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:04:53.969+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:04:53.969422+0000) 2024-03-04T18:04:54.969 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:04:54.968+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:04:54.969681+0000) 2024-03-04T18:04:55.969 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:04:55.969+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:04:55.969954+0000) 2024-03-04T18:04:56.969 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:04:56.969+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:04:56.970196+0000) 2024-03-04T18:04:57.969 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:04:57.970+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:04:57.970432+0000) 2024-03-04T18:04:58.970 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:04:58.970+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:04:58.970737+0000) 2024-03-04T18:04:59.970 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:04:59.970+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:04:59.970976+0000) 2024-03-04T18:05:00.970 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:05:00.970+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:05:00.971218+0000) 2024-03-04T18:05:01.970 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:05:01.970+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:05:01.971401+0000) 2024-03-04T18:05:02.971 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:05:02.970+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:05:02.971667+0000) 2024-03-04T18:05:03.971 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:05:03.971+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:05:03.971935+0000) 2024-03-04T18:05:04.971 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:05:04.971+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:05:04.972171+0000) 2024-03-04T18:05:05.972 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:05:05.972+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:05:05.972416+0000) 2024-03-04T18:05:06.971 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:05:06.972+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:05:06.972655+0000) 2024-03-04T18:05:07.972 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:05:07.972+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:05:07.972934+0000) 2024-03-04T18:05:08.972 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:05:08.973+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:05:08.973171+0000) 2024-03-04T18:05:09.972 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:05:09.972+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:05:09.973414+0000) 2024-03-04T18:05:10.972 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:05:10.971+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:05:10.973667+0000) 2024-03-04T18:05:11.973 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:05:11.972+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:05:11.973954+0000) 2024-03-04T18:05:12.973 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:05:12.973+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:05:12.974201+0000) 2024-03-04T18:05:13.973 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:05:13.973+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:05:13.974441+0000) 2024-03-04T18:05:14.973 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:05:14.973+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:05:14.974677+0000) 2024-03-04T18:05:15.974 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:05:15.973+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:05:15.974956+0000) 2024-03-04T18:05:16.974 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:05:16.973+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:05:16.975136+0000) 2024-03-04T18:05:17.974 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:05:17.974+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:05:17.975405+0000) 2024-03-04T18:05:18.975 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:05:18.974+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:05:18.975675+0000) 2024-03-04T18:05:19.975 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:05:19.974+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:05:19.975958+0000) 2024-03-04T18:05:20.975 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:05:20.975+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:05:20.976248+0000) 2024-03-04T18:05:21.975 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:05:21.975+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:05:21.976540+0000) 2024-03-04T18:05:22.654 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:05:22.666 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:05:22.976 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:05:22.976+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:05:22.976865+0000) 2024-03-04T18:05:23.976 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:05:23.976+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:05:23.977102+0000) 2024-03-04T18:05:24.976 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:05:24.976+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:05:24.977397+0000) 2024-03-04T18:05:25.979 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:05:25.976+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:05:25.977630+0000) 2024-03-04T18:05:26.977 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:05:26.977+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:05:26.977960+0000) 2024-03-04T18:05:27.977 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:05:27.977+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:05:27.978207+0000) 2024-03-04T18:05:28.977 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:05:28.977+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:05:28.978452+0000) 2024-03-04T18:05:29.978 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:05:29.977+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:05:29.978707+0000) 2024-03-04T18:05:30.978 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:05:30.978+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:05:30.978961+0000) 2024-03-04T18:05:31.978 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:05:31.978+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:05:31.979219+0000) 2024-03-04T18:05:32.978 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:05:32.978+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:05:32.979482+0000) 2024-03-04T18:05:33.979 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:05:33.978+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:05:33.979734+0000) 2024-03-04T18:05:34.979 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:05:34.979+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:05:34.979988+0000) 2024-03-04T18:05:35.979 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:05:35.979+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:05:35.980272+0000) 2024-03-04T18:05:36.980 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:05:36.979+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:05:36.980523+0000) 2024-03-04T18:05:37.980 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:05:37.980+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:05:37.980758+0000) 2024-03-04T18:05:38.980 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:05:38.980+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:05:38.980996+0000) 2024-03-04T18:05:39.981 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:05:39.980+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:05:39.981258+0000) 2024-03-04T18:05:40.980 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:05:40.980+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:05:40.981509+0000) 2024-03-04T18:05:41.981 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:05:41.981+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:05:41.981767+0000) 2024-03-04T18:05:42.981 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:05:42.981+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:05:42.982035+0000) 2024-03-04T18:05:43.981 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:05:43.981+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:05:43.982282+0000) 2024-03-04T18:05:44.981 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:05:44.981+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:05:44.982534+0000) 2024-03-04T18:05:45.982 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:05:45.982+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:05:45.982790+0000) 2024-03-04T18:05:46.982 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:05:46.982+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:05:46.983126+0000) 2024-03-04T18:05:47.982 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:05:47.982+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:05:47.983327+0000) 2024-03-04T18:05:48.982 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:05:48.982+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:05:48.983527+0000) 2024-03-04T18:05:49.983 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:05:49.983+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:05:49.983729+0000) 2024-03-04T18:05:50.983 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:05:50.983+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:05:50.984013+0000) 2024-03-04T18:05:51.983 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:05:51.983+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:05:51.984287+0000) 2024-03-04T18:05:52.865 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:05:52.873 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:05:52.983 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:05:52.983+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:05:52.984561+0000) 2024-03-04T18:05:53.984 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:05:53.984+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:05:53.984809+0000) 2024-03-04T18:05:54.984 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:05:54.984+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:05:54.985065+0000) 2024-03-04T18:05:55.984 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:05:55.984+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:05:55.985322+0000) 2024-03-04T18:05:56.985 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:05:56.985+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:05:56.985573+0000) 2024-03-04T18:05:57.985 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:05:57.985+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:05:57.985862+0000) 2024-03-04T18:05:58.985 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:05:58.985+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:05:58.986119+0000) 2024-03-04T18:05:59.985 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:05:59.985+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:05:59.986329+0000) 2024-03-04T18:06:00.986 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:06:00.985+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:06:00.986614+0000) 2024-03-04T18:06:01.986 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:06:01.986+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:06:01.986931+0000) 2024-03-04T18:06:02.986 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:06:02.986+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:06:02.987198+0000) 2024-03-04T18:06:03.986 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:06:03.987+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:06:03.987450+0000) 2024-03-04T18:06:04.987 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:06:04.987+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:06:04.987705+0000) 2024-03-04T18:06:05.987 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:06:05.987+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:06:05.987966+0000) 2024-03-04T18:06:06.987 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:06:06.988+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:06:06.988213+0000) 2024-03-04T18:06:07.987 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:06:07.987+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:06:07.988445+0000) 2024-03-04T18:06:08.988 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:06:08.988+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:06:08.988690+0000) 2024-03-04T18:06:09.988 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:06:09.988+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:06:09.988927+0000) 2024-03-04T18:06:10.988 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:06:10.989+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:06:10.989164+0000) 2024-03-04T18:06:11.988 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:06:11.988+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:06:11.989402+0000) 2024-03-04T18:06:12.989 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:06:12.988+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:06:12.989592+0000) 2024-03-04T18:06:13.989 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:06:13.988+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:06:13.989854+0000) 2024-03-04T18:06:14.989 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:06:14.989+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:06:14.990093+0000) 2024-03-04T18:06:15.989 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:06:15.989+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:06:15.990334+0000) 2024-03-04T18:06:16.990 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:06:16.989+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:06:16.990568+0000) 2024-03-04T18:06:17.990 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:06:17.989+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:06:17.990868+0000) 2024-03-04T18:06:18.990 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:06:18.989+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:06:18.991076+0000) 2024-03-04T18:06:19.990 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:06:19.990+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:06:19.991318+0000) 2024-03-04T18:06:20.990 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:06:20.990+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:06:20.991573+0000) 2024-03-04T18:06:21.991 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:06:21.991+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:06:21.991866+0000) 2024-03-04T18:06:22.991 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:06:22.991+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:06:22.992164+0000) 2024-03-04T18:06:23.005 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:06:23.016 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:06:23.991 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:06:23.991+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:06:23.992424+0000) 2024-03-04T18:06:24.992 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:06:24.991+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:06:24.992685+0000) 2024-03-04T18:06:25.992 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:06:25.992+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:06:25.992946+0000) 2024-03-04T18:06:26.992 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:06:26.992+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:06:26.993225+0000) 2024-03-04T18:06:27.992 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:06:27.992+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:06:27.993479+0000) 2024-03-04T18:06:28.993 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:06:28.992+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:06:28.993739+0000) 2024-03-04T18:06:29.993 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:06:29.993+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:06:29.994110+0000) 2024-03-04T18:06:30.994 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:06:30.993+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:06:30.994380+0000) 2024-03-04T18:06:31.994 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:06:31.993+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:06:31.994636+0000) 2024-03-04T18:06:32.994 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:06:32.994+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:06:32.994921+0000) 2024-03-04T18:06:33.994 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:06:33.994+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:06:33.995156+0000) 2024-03-04T18:06:34.994 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:06:34.994+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:06:34.995400+0000) 2024-03-04T18:06:35.995 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:06:35.994+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:06:35.995637+0000) 2024-03-04T18:06:36.995 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:06:36.995+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:06:36.995906+0000) 2024-03-04T18:06:37.995 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:06:37.995+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:06:37.996146+0000) 2024-03-04T18:06:38.996 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:06:38.995+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:06:38.996383+0000) 2024-03-04T18:06:39.996 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:06:39.995+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:06:39.996616+0000) 2024-03-04T18:06:40.996 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:06:40.996+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:06:40.996887+0000) 2024-03-04T18:06:41.997 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:06:41.996+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:06:41.997132+0000) 2024-03-04T18:06:42.996 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:06:42.996+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:06:42.997386+0000) 2024-03-04T18:06:43.997 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:06:43.997+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:06:43.997625+0000) 2024-03-04T18:06:44.997 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:06:44.997+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:06:44.997897+0000) 2024-03-04T18:06:45.997 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:06:45.997+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:06:45.998136+0000) 2024-03-04T18:06:46.997 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:06:46.997+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:06:46.998397+0000) 2024-03-04T18:06:47.998 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:06:47.997+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:06:47.998624+0000) 2024-03-04T18:06:48.998 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:06:48.998+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:06:48.998912+0000) 2024-03-04T18:06:49.998 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:06:49.998+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:06:49.999150+0000) 2024-03-04T18:06:50.998 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:06:50.998+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:06:50.999368+0000) 2024-03-04T18:06:51.999 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:06:51.999+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:06:51.999642+0000) 2024-03-04T18:06:52.999 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:06:52.999+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:06:52.999881+0000) 2024-03-04T18:06:53.147 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:06:53.156 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:06:53.999 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:06:53.999+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:06:54.000121+0000) 2024-03-04T18:06:54.999 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:06:54.999+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:06:55.000359+0000) 2024-03-04T18:06:55.999 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:06:55.999+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:06:56.000545+0000) 2024-03-04T18:06:57.000 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:06:57.000+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:06:57.000786+0000) 2024-03-04T18:06:58.000 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:06:58.000+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:06:58.001028+0000) 2024-03-04T18:06:59.000 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:06:59.000+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:06:59.001276+0000) 2024-03-04T18:07:00.000 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:07:00.000+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:07:00.001471+0000) 2024-03-04T18:07:01.001 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:07:01.001+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:07:01.001716+0000) 2024-03-04T18:07:02.001 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:07:02.001+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:07:02.002000+0000) 2024-03-04T18:07:03.001 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:07:03.001+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:07:03.002206+0000) 2024-03-04T18:07:04.001 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:07:04.001+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:07:04.002457+0000) 2024-03-04T18:07:05.002 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:07:05.001+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:07:05.002709+0000) 2024-03-04T18:07:06.002 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:07:06.002+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:07:06.002962+0000) 2024-03-04T18:07:07.002 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:07:07.002+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:07:07.003206+0000) 2024-03-04T18:07:08.002 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:07:08.003+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:07:08.003446+0000) 2024-03-04T18:07:09.003 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:07:09.003+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:07:09.003687+0000) 2024-03-04T18:07:10.003 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:07:10.003+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:07:10.003936+0000) 2024-03-04T18:07:11.003 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:07:11.004+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:07:11.004176+0000) 2024-03-04T18:07:12.003 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:07:12.003+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:07:12.004415+0000) 2024-03-04T18:07:13.004 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:07:13.003+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:07:13.004663+0000) 2024-03-04T18:07:14.004 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:07:14.003+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:07:14.004943+0000) 2024-03-04T18:07:15.004 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:07:15.003+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:07:15.005184+0000) 2024-03-04T18:07:16.004 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:07:16.004+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:07:16.005419+0000) 2024-03-04T18:07:17.005 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:07:17.004+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:07:17.005692+0000) 2024-03-04T18:07:18.005 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:07:18.004+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:07:18.005950+0000) 2024-03-04T18:07:19.005 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:07:19.005+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:07:19.006185+0000) 2024-03-04T18:07:20.005 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:07:20.005+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:07:20.006423+0000) 2024-03-04T18:07:21.006 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:07:21.005+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:07:21.006661+0000) 2024-03-04T18:07:22.006 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:07:22.005+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:07:22.006953+0000) 2024-03-04T18:07:23.006 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:07:23.006+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:07:23.007260+0000) 2024-03-04T18:07:23.363 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:07:23.371 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:07:24.006 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:07:24.006+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:07:24.007507+0000) 2024-03-04T18:07:25.007 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:07:25.006+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:07:25.007746+0000) 2024-03-04T18:07:26.007 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:07:26.007+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:07:26.007991+0000) 2024-03-04T18:07:27.007 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:07:27.007+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:07:27.008231+0000) 2024-03-04T18:07:28.008 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:07:28.007+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:07:28.008481+0000) 2024-03-04T18:07:29.008 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:07:29.007+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:07:29.008660+0000) 2024-03-04T18:07:30.008 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:07:30.008+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:07:30.008937+0000) 2024-03-04T18:07:31.008 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:07:31.008+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:07:31.009177+0000) 2024-03-04T18:07:32.008 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:07:32.008+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:07:32.009425+0000) 2024-03-04T18:07:33.009 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:07:33.008+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:07:33.009673+0000) 2024-03-04T18:07:34.009 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:07:34.009+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:07:34.009930+0000) 2024-03-04T18:07:35.009 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:07:35.009+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:07:35.010183+0000) 2024-03-04T18:07:36.009 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:07:36.009+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:07:36.010390+0000) 2024-03-04T18:07:37.010 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:07:37.009+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:07:37.010603+0000) 2024-03-04T18:07:38.010 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:07:38.010+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:07:38.010900+0000) 2024-03-04T18:07:39.010 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:07:39.010+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:07:39.011110+0000) 2024-03-04T18:07:40.010 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:07:40.010+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:07:40.011316+0000) 2024-03-04T18:07:41.010 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:07:41.010+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:07:41.011583+0000) 2024-03-04T18:07:42.011 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:07:42.011+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:07:42.011879+0000) 2024-03-04T18:07:43.011 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:07:43.011+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:07:43.012139+0000) 2024-03-04T18:07:44.011 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:07:44.011+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:07:44.012374+0000) 2024-03-04T18:07:45.012 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:07:45.012+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:07:45.012612+0000) 2024-03-04T18:07:46.012 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:07:46.012+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:07:46.012896+0000) 2024-03-04T18:07:47.012 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:07:47.012+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:07:47.013172+0000) 2024-03-04T18:07:48.012 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:07:48.012+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:07:48.013438+0000) 2024-03-04T18:07:49.013 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:07:49.013+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:07:49.013678+0000) 2024-03-04T18:07:50.013 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:07:50.013+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:07:50.013977+0000) 2024-03-04T18:07:51.013 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:07:51.013+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:07:51.014172+0000) 2024-03-04T18:07:52.013 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:07:52.013+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:07:52.014416+0000) 2024-03-04T18:07:53.014 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:07:53.014+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:07:53.014664+0000) 2024-03-04T18:07:53.509 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:07:53.517 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:07:54.014 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:07:54.014+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:07:54.014945+0000) 2024-03-04T18:07:55.014 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:07:55.014+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:07:55.015178+0000) 2024-03-04T18:07:56.014 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:07:56.015+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:07:56.015418+0000) 2024-03-04T18:07:57.015 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:07:57.015+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:07:57.015663+0000) 2024-03-04T18:07:58.015 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:07:58.015+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:07:58.015950+0000) 2024-03-04T18:07:59.016 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:07:59.015+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:07:59.016194+0000) 2024-03-04T18:08:00.016 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:08:00.016+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:08:00.016434+0000) 2024-03-04T18:08:01.016 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:08:01.016+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:08:01.016671+0000) 2024-03-04T18:08:02.016 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:08:02.016+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:08:02.016950+0000) 2024-03-04T18:08:03.016 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:08:03.016+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:08:03.017199+0000) 2024-03-04T18:08:04.016 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:08:04.017+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:08:04.017437+0000) 2024-03-04T18:08:05.017 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:08:05.017+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:08:05.017673+0000) 2024-03-04T18:08:06.017 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:08:06.017+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:08:06.017951+0000) 2024-03-04T18:08:07.017 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:08:07.017+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:08:07.018183+0000) 2024-03-04T18:08:08.017 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:08:08.017+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:08:08.018432+0000) 2024-03-04T18:08:09.018 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:08:09.017+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:08:09.018667+0000) 2024-03-04T18:08:10.018 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:08:10.018+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:08:10.018960+0000) 2024-03-04T18:08:11.018 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:08:11.018+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:08:11.019154+0000) 2024-03-04T18:08:12.018 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:08:12.019+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:08:12.019411+0000) 2024-03-04T18:08:13.019 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:08:13.019+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:08:13.019686+0000) 2024-03-04T18:08:14.019 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:08:14.018+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:08:14.019949+0000) 2024-03-04T18:08:15.019 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:08:15.019+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:08:15.020184+0000) 2024-03-04T18:08:16.019 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:08:16.019+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:08:16.020427+0000) 2024-03-04T18:08:17.020 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:08:17.018+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:08:17.020678+0000) 2024-03-04T18:08:18.020 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:08:18.019+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:08:18.020925+0000) 2024-03-04T18:08:19.020 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:08:19.020+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:08:19.021163+0000) 2024-03-04T18:08:20.020 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:08:20.020+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:08:20.021407+0000) 2024-03-04T18:08:21.021 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:08:21.020+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:08:21.021588+0000) 2024-03-04T18:08:22.021 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:08:22.020+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:08:22.021872+0000) 2024-03-04T18:08:23.021 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:08:23.021+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:08:23.022142+0000) 2024-03-04T18:08:23.647 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:08:23.655 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:08:24.021 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:08:24.021+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:08:24.022374+0000) 2024-03-04T18:08:25.022 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:08:25.021+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:08:25.022617+0000) 2024-03-04T18:08:26.022 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:08:26.022+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:08:26.022919+0000) 2024-03-04T18:08:27.022 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:08:27.022+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:08:27.023158+0000) 2024-03-04T18:08:28.022 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:08:28.022+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:08:28.023397+0000) 2024-03-04T18:08:29.023 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:08:29.022+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:08:29.023635+0000) 2024-03-04T18:08:30.023 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:08:30.023+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:08:30.023920+0000) 2024-03-04T18:08:31.023 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:08:31.023+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:08:31.024180+0000) 2024-03-04T18:08:32.023 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:08:32.023+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:08:32.024426+0000) 2024-03-04T18:08:33.024 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:08:33.023+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:08:33.024676+0000) 2024-03-04T18:08:34.024 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:08:34.024+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:08:34.024959+0000) 2024-03-04T18:08:35.024 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:08:35.024+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:08:35.025198+0000) 2024-03-04T18:08:36.024 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:08:36.024+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:08:36.025452+0000) 2024-03-04T18:08:37.025 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:08:37.024+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:08:37.025691+0000) 2024-03-04T18:08:38.025 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:08:38.025+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:08:38.025970+0000) 2024-03-04T18:08:39.025 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:08:39.025+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:08:39.026216+0000) 2024-03-04T18:08:40.026 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:08:40.025+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:08:40.026467+0000) 2024-03-04T18:08:41.026 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:08:41.025+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:08:41.026720+0000) 2024-03-04T18:08:42.026 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:08:42.026+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:08:42.026966+0000) 2024-03-04T18:08:43.026 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:08:43.026+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:08:43.027216+0000) 2024-03-04T18:08:44.027 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:08:44.026+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:08:44.027456+0000) 2024-03-04T18:08:45.027 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:08:45.027+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:08:45.027693+0000) 2024-03-04T18:08:46.027 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:08:46.027+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:08:46.027938+0000) 2024-03-04T18:08:47.027 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:08:47.027+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:08:47.028183+0000) 2024-03-04T18:08:48.027 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:08:48.027+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:08:48.028372+0000) 2024-03-04T18:08:49.028 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:08:49.027+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:08:49.028640+0000) 2024-03-04T18:08:50.028 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:08:50.028+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:08:50.028922+0000) 2024-03-04T18:08:51.028 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:08:51.028+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:08:51.029131+0000) 2024-03-04T18:08:52.029 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:08:52.028+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:08:52.029375+0000) 2024-03-04T18:08:53.029 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:08:53.029+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:08:53.029619+0000) 2024-03-04T18:08:53.794 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:08:53.802 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:08:54.029 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:08:54.029+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:08:54.029960+0000) 2024-03-04T18:08:55.029 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:08:55.029+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:08:55.030209+0000) 2024-03-04T18:08:56.030 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:08:56.030+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:08:56.030468+0000) 2024-03-04T18:08:57.030 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:08:57.030+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:08:57.030715+0000) 2024-03-04T18:08:58.030 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:08:58.030+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:08:58.030955+0000) 2024-03-04T18:08:59.030 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:08:59.030+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:08:59.031199+0000) 2024-03-04T18:09:00.031 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:09:00.030+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:09:00.031432+0000) 2024-03-04T18:09:01.031 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:09:01.031+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:09:01.031683+0000) 2024-03-04T18:09:02.031 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:09:02.031+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:09:02.031967+0000) 2024-03-04T18:09:03.031 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:09:03.031+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:09:03.032158+0000) 2024-03-04T18:09:04.031 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:09:04.032+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:09:04.032408+0000) 2024-03-04T18:09:05.032 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:09:05.032+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:09:05.032651+0000) 2024-03-04T18:09:06.032 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:09:06.032+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:09:06.032944+0000) 2024-03-04T18:09:07.033 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:09:07.032+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:09:07.033189+0000) 2024-03-04T18:09:08.033 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:09:08.033+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:09:08.033432+0000) 2024-03-04T18:09:09.033 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:09:09.032+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:09:09.033632+0000) 2024-03-04T18:09:10.033 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:09:10.033+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:09:10.033930+0000) 2024-03-04T18:09:11.033 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:09:11.033+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:09:11.034136+0000) 2024-03-04T18:09:12.033 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:09:12.033+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:09:12.034403+0000) 2024-03-04T18:09:13.034 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:09:13.033+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:09:13.034671+0000) 2024-03-04T18:09:14.034 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:09:14.034+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:09:14.034931+0000) 2024-03-04T18:09:15.034 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:09:15.035+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:09:15.035179+0000) 2024-03-04T18:09:16.034 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:09:16.034+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:09:16.035423+0000) 2024-03-04T18:09:17.035 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:09:17.034+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:09:17.035676+0000) 2024-03-04T18:09:18.035 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:09:18.034+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:09:18.035938+0000) 2024-03-04T18:09:19.035 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:09:19.035+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:09:19.036212+0000) 2024-03-04T18:09:20.035 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:09:20.035+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:09:20.036455+0000) 2024-03-04T18:09:21.036 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:09:21.035+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:09:21.036701+0000) 2024-03-04T18:09:22.036 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:09:22.035+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:09:22.036972+0000) 2024-03-04T18:09:23.036 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:09:23.036+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:09:23.037227+0000) 2024-03-04T18:09:23.933 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:09:23.941 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:09:24.036 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:09:24.035+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:09:24.037403+0000) 2024-03-04T18:09:25.039 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:09:25.036+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:09:25.037637+0000) 2024-03-04T18:09:26.037 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:09:26.036+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:09:26.037947+0000) 2024-03-04T18:09:27.037 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:09:27.037+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:09:27.038145+0000) 2024-03-04T18:09:28.037 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:09:28.037+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:09:28.038387+0000) 2024-03-04T18:09:29.038 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:09:29.037+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:09:29.038627+0000) 2024-03-04T18:09:30.038 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:09:30.038+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:09:30.038921+0000) 2024-03-04T18:09:31.039 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:09:31.038+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:09:31.039163+0000) 2024-03-04T18:09:32.038 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:09:32.038+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:09:32.039407+0000) 2024-03-04T18:09:33.039 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:09:33.038+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:09:33.039660+0000) 2024-03-04T18:09:34.039 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:09:34.039+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:09:34.039963+0000) 2024-03-04T18:09:35.039 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:09:35.039+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:09:35.040202+0000) 2024-03-04T18:09:36.039 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:09:36.039+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:09:36.040448+0000) 2024-03-04T18:09:37.040 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:09:37.039+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:09:37.040642+0000) 2024-03-04T18:09:38.040 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:09:38.040+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:09:38.040929+0000) 2024-03-04T18:09:39.040 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:09:39.040+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:09:39.041174+0000) 2024-03-04T18:09:40.040 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:09:40.040+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:09:40.041418+0000) 2024-03-04T18:09:41.041 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:09:41.040+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:09:41.041663+0000) 2024-03-04T18:09:42.041 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:09:42.041+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:09:42.041965+0000) 2024-03-04T18:09:43.041 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:09:43.041+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:09:43.042203+0000) 2024-03-04T18:09:44.041 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:09:44.041+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:09:44.042395+0000) 2024-03-04T18:09:45.042 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:09:45.041+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:09:45.042650+0000) 2024-03-04T18:09:46.042 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:09:46.042+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:09:46.042943+0000) 2024-03-04T18:09:47.042 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:09:47.042+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:09:47.043187+0000) 2024-03-04T18:09:48.042 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:09:48.042+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:09:48.043459+0000) 2024-03-04T18:09:49.043 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:09:49.043+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:09:49.043702+0000) 2024-03-04T18:09:50.043 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:09:50.043+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:09:50.043970+0000) 2024-03-04T18:09:51.043 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:09:51.043+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:09:51.044237+0000) 2024-03-04T18:09:52.044 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:09:52.043+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:09:52.044491+0000) 2024-03-04T18:09:53.044 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:09:53.044+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:09:53.044737+0000) 2024-03-04T18:09:54.044 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:09:54.044+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:09:54.045000+0000) 2024-03-04T18:09:54.088 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:09:54.096 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:09:55.044 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:09:55.044+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:09:55.045250+0000) 2024-03-04T18:09:56.044 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:09:56.045+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:09:56.045517+0000) 2024-03-04T18:09:57.045 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:09:57.045+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:09:57.045760+0000) 2024-03-04T18:09:58.045 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:09:58.045+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:09:58.046102+0000) 2024-03-04T18:09:59.045 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:09:59.045+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:09:59.046345+0000) 2024-03-04T18:10:00.046 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:10:00.046+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:10:00.046588+0000) 2024-03-04T18:10:01.046 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:10:01.046+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:10:01.046920+0000) 2024-03-04T18:10:02.046 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:10:02.046+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:10:02.047170+0000) 2024-03-04T18:10:03.046 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:10:03.047+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:10:03.047427+0000) 2024-03-04T18:10:04.047 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:10:04.047+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:10:04.047674+0000) 2024-03-04T18:10:05.047 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:10:05.047+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:10:05.047954+0000) 2024-03-04T18:10:06.047 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:10:06.047+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:10:06.048223+0000) 2024-03-04T18:10:07.047 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:10:07.048+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:10:07.048490+0000) 2024-03-04T18:10:08.048 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:10:08.048+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:10:08.048745+0000) 2024-03-04T18:10:09.048 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:10:09.048+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:10:09.048989+0000) 2024-03-04T18:10:10.048 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:10:10.048+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:10:10.049233+0000) 2024-03-04T18:10:11.049 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:10:11.049+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:10:11.049489+0000) 2024-03-04T18:10:12.049 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:10:12.048+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:10:12.049754+0000) 2024-03-04T18:10:13.049 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:10:13.049+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:10:13.050010+0000) 2024-03-04T18:10:14.049 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:10:14.049+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:10:14.050247+0000) 2024-03-04T18:10:15.049 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:10:15.049+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:10:15.050482+0000) 2024-03-04T18:10:16.050 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:10:16.050+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:10:16.050729+0000) 2024-03-04T18:10:17.050 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:10:17.050+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:10:17.050986+0000) 2024-03-04T18:10:18.050 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:10:18.050+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:10:18.051251+0000) 2024-03-04T18:10:19.050 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:10:19.050+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:10:19.051493+0000) 2024-03-04T18:10:20.051 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:10:20.050+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:10:20.051747+0000) 2024-03-04T18:10:21.051 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:10:21.050+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:10:21.051996+0000) 2024-03-04T18:10:22.051 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:10:22.051+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:10:22.052249+0000) 2024-03-04T18:10:23.052 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:10:23.051+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:10:23.052500+0000) 2024-03-04T18:10:24.052 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:10:24.051+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:10:24.052740+0000) 2024-03-04T18:10:24.225 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:10:24.237 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:10:25.052 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:10:25.051+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:10:25.052978+0000) 2024-03-04T18:10:26.052 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:10:26.052+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:10:26.053224+0000) 2024-03-04T18:10:27.052 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:10:27.052+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:10:27.053470+0000) 2024-03-04T18:10:28.053 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:10:28.052+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:10:28.053713+0000) 2024-03-04T18:10:29.053 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:10:29.053+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:10:29.053992+0000) 2024-03-04T18:10:30.053 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:10:30.053+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:10:30.054233+0000) 2024-03-04T18:10:31.054 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:10:31.053+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:10:31.054481+0000) 2024-03-04T18:10:32.054 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:10:32.053+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:10:32.054727+0000) 2024-03-04T18:10:33.054 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:10:33.054+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:10:33.054977+0000) 2024-03-04T18:10:34.054 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:10:34.054+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:10:34.055220+0000) 2024-03-04T18:10:35.055 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:10:35.054+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:10:35.055477+0000) 2024-03-04T18:10:36.055 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:10:36.054+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:10:36.055723+0000) 2024-03-04T18:10:37.055 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:10:37.055+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:10:37.055964+0000) 2024-03-04T18:10:38.055 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:10:38.055+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:10:38.056207+0000) 2024-03-04T18:10:39.056 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:10:39.055+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:10:39.056454+0000) 2024-03-04T18:10:40.056 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:10:40.055+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:10:40.056773+0000) 2024-03-04T18:10:41.056 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:10:41.056+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:10:41.057020+0000) 2024-03-04T18:10:42.056 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:10:42.056+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:10:42.057297+0000) 2024-03-04T18:10:43.057 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:10:43.056+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:10:43.057566+0000) 2024-03-04T18:10:44.057 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:10:44.056+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:10:44.057756+0000) 2024-03-04T18:10:45.057 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:10:45.057+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:10:45.057997+0000) 2024-03-04T18:10:46.057 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:10:46.057+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:10:46.058195+0000) 2024-03-04T18:10:47.058 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:10:47.057+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:10:47.058474+0000) 2024-03-04T18:10:48.058 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:10:48.057+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:10:48.058739+0000) 2024-03-04T18:10:49.058 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:10:49.058+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:10:49.058981+0000) 2024-03-04T18:10:50.058 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:10:50.058+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:10:50.059225+0000) 2024-03-04T18:10:51.059 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:10:51.058+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:10:51.059477+0000) 2024-03-04T18:10:52.059 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:10:52.059+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:10:52.059748+0000) 2024-03-04T18:10:53.059 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:10:53.059+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:10:53.060004+0000) 2024-03-04T18:10:54.059 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:10:54.059+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:10:54.060243+0000) 2024-03-04T18:10:54.373 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:10:54.384 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:10:55.060 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:10:55.059+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:10:55.060490+0000) 2024-03-04T18:10:56.060 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:10:56.060+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:10:56.060739+0000) 2024-03-04T18:10:57.060 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:10:57.060+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:10:57.060987+0000) 2024-03-04T18:10:58.060 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:10:58.060+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:10:58.061167+0000) 2024-03-04T18:10:59.061 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:10:59.060+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:10:59.061352+0000) 2024-03-04T18:11:00.061 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:11:00.060+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:11:00.061535+0000) 2024-03-04T18:11:01.061 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:11:01.060+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:11:01.061745+0000) 2024-03-04T18:11:02.061 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:11:02.061+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:11:02.061991+0000) 2024-03-04T18:11:03.061 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:11:03.061+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:11:03.062203+0000) 2024-03-04T18:11:04.061 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:11:04.062+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:11:04.062440+0000) 2024-03-04T18:11:05.062 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:11:05.062+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:11:05.062680+0000) 2024-03-04T18:11:06.062 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:11:06.062+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:11:06.062943+0000) 2024-03-04T18:11:07.062 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:11:07.062+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:11:07.063190+0000) 2024-03-04T18:11:08.063 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:11:08.063+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:11:08.063492+0000) 2024-03-04T18:11:09.063 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:11:09.062+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:11:09.063787+0000) 2024-03-04T18:11:10.063 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:11:10.063+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:11:10.064134+0000) 2024-03-04T18:11:11.064 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:11:11.063+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:11:11.064385+0000) 2024-03-04T18:11:12.064 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:11:12.064+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:11:12.064693+0000) 2024-03-04T18:11:13.064 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:11:13.064+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:11:13.065030+0000) 2024-03-04T18:11:14.064 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:11:14.064+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:11:14.065308+0000) 2024-03-04T18:11:15.065 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:11:15.065+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:11:15.065606+0000) 2024-03-04T18:11:16.065 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:11:16.065+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:11:16.065926+0000) 2024-03-04T18:11:17.065 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:11:17.065+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:11:17.066212+0000) 2024-03-04T18:11:18.066 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:11:18.066+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:11:18.066481+0000) 2024-03-04T18:11:19.066 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:11:19.065+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:11:19.066720+0000) 2024-03-04T18:11:20.066 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:11:20.065+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:11:20.066959+0000) 2024-03-04T18:11:21.066 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:11:21.066+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:11:21.067222+0000) 2024-03-04T18:11:22.067 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:11:22.065+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:11:22.067463+0000) 2024-03-04T18:11:23.067 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:11:23.066+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:11:23.067718+0000) 2024-03-04T18:11:24.067 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:11:24.066+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:11:24.067936+0000) 2024-03-04T18:11:24.569 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:11:24.579 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:11:25.067 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:11:25.067+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:11:25.068188+0000) 2024-03-04T18:11:26.068 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:11:26.067+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:11:26.068438+0000) 2024-03-04T18:11:27.068 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:11:27.067+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:11:27.068683+0000) 2024-03-04T18:11:28.068 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:11:28.067+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:11:28.068957+0000) 2024-03-04T18:11:29.069 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:11:29.068+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:11:29.069221+0000) 2024-03-04T18:11:30.069 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:11:30.068+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:11:30.069462+0000) 2024-03-04T18:11:31.069 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:11:31.068+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:11:31.069706+0000) 2024-03-04T18:11:32.069 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:11:32.069+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:11:32.069952+0000) 2024-03-04T18:11:33.069 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:11:33.069+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:11:33.070207+0000) 2024-03-04T18:11:34.070 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:11:34.069+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:11:34.070568+0000) 2024-03-04T18:11:35.070 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:11:35.070+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:11:35.070867+0000) 2024-03-04T18:11:36.071 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:11:36.070+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:11:36.071109+0000) 2024-03-04T18:11:37.070 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:11:37.070+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:11:37.071374+0000) 2024-03-04T18:11:38.071 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:11:38.070+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:11:38.071613+0000) 2024-03-04T18:11:39.071 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:11:39.071+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:11:39.071924+0000) 2024-03-04T18:11:40.071 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:11:40.071+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:11:40.072159+0000) 2024-03-04T18:11:41.071 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:11:41.071+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:11:41.072405+0000) 2024-03-04T18:11:42.072 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:11:42.071+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:11:42.072664+0000) 2024-03-04T18:11:43.072 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:11:43.072+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:11:43.072945+0000) 2024-03-04T18:11:44.072 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:11:44.072+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:11:44.073142+0000) 2024-03-04T18:11:45.072 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:11:45.072+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:11:45.073386+0000) 2024-03-04T18:11:46.073 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:11:46.072+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:11:46.073648+0000) 2024-03-04T18:11:47.073 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:11:47.073+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:11:47.073947+0000) 2024-03-04T18:11:48.073 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:11:48.073+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:11:48.074189+0000) 2024-03-04T18:11:49.073 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:11:49.073+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:11:49.074432+0000) 2024-03-04T18:11:50.074 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:11:50.074+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:11:50.074676+0000) 2024-03-04T18:11:51.074 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:11:51.074+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:11:51.074942+0000) 2024-03-04T18:11:52.074 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:11:52.074+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:11:52.075186+0000) 2024-03-04T18:11:53.075 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:11:53.074+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:11:53.075439+0000) 2024-03-04T18:11:54.075 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:11:54.075+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:11:54.075681+0000) 2024-03-04T18:11:54.732 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:11:54.740 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:11:55.075 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:11:55.075+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:11:55.075936+0000) 2024-03-04T18:11:56.075 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:11:56.075+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:11:56.076179+0000) 2024-03-04T18:11:57.076 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:11:57.075+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:11:57.076441+0000) 2024-03-04T18:11:58.076 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:11:58.076+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:11:58.076685+0000) 2024-03-04T18:11:59.076 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:11:59.076+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:11:59.076929+0000) 2024-03-04T18:12:00.076 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:12:00.076+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:12:00.077167+0000) 2024-03-04T18:12:01.076 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:12:01.076+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:12:01.077413+0000) 2024-03-04T18:12:02.077 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:12:02.077+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:12:02.077653+0000) 2024-03-04T18:12:03.077 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:12:03.077+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:12:03.077948+0000) 2024-03-04T18:12:04.077 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:12:04.077+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:12:04.078214+0000) 2024-03-04T18:12:05.077 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:12:05.078+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:12:05.078473+0000) 2024-03-04T18:12:06.078 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:12:06.078+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:12:06.078714+0000) 2024-03-04T18:12:07.078 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:12:07.078+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:12:07.078958+0000) 2024-03-04T18:12:08.078 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:12:08.078+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:12:08.079200+0000) 2024-03-04T18:12:09.078 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:12:09.078+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:12:09.079396+0000) 2024-03-04T18:12:10.079 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:12:10.079+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:12:10.079610+0000) 2024-03-04T18:12:11.079 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:12:11.079+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:12:11.079894+0000) 2024-03-04T18:12:12.079 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:12:12.079+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:12:12.080091+0000) 2024-03-04T18:12:13.079 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:12:13.080+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:12:13.080344+0000) 2024-03-04T18:12:14.080 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:12:14.080+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:12:14.080582+0000) 2024-03-04T18:12:15.080 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:12:15.080+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:12:15.080858+0000) 2024-03-04T18:12:16.080 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:12:16.080+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:12:16.081123+0000) 2024-03-04T18:12:17.080 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:12:17.081+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:12:17.081384+0000) 2024-03-04T18:12:18.081 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:12:18.081+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:12:18.081629+0000) 2024-03-04T18:12:19.081 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:12:19.081+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:12:19.081921+0000) 2024-03-04T18:12:20.081 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:12:20.081+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:12:20.082194+0000) 2024-03-04T18:12:21.082 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:12:21.081+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:12:21.082504+0000) 2024-03-04T18:12:22.082 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:12:22.081+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:12:22.082818+0000) 2024-03-04T18:12:23.082 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:12:23.082+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:12:23.083162+0000) 2024-03-04T18:12:24.082 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:12:24.082+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:12:24.083459+0000) 2024-03-04T18:12:24.955 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:12:24.963 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:12:25.083 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:12:25.082+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:12:25.083755+0000) 2024-03-04T18:12:26.083 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:12:26.082+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:12:26.084062+0000) 2024-03-04T18:12:27.084 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:12:27.083+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:12:27.084419+0000) 2024-03-04T18:12:28.084 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:12:28.083+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:12:28.084677+0000) 2024-03-04T18:12:29.084 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:12:29.083+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:12:29.084951+0000) 2024-03-04T18:12:30.084 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:12:30.084+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:12:30.085192+0000) 2024-03-04T18:12:31.084 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:12:31.084+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:12:31.085442+0000) 2024-03-04T18:12:32.085 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:12:32.084+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:12:32.085686+0000) 2024-03-04T18:12:33.087 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:12:33.085+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:12:33.085948+0000) 2024-03-04T18:12:34.085 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:12:34.085+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:12:34.086187+0000) 2024-03-04T18:12:35.085 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:12:35.085+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:12:35.086384+0000) 2024-03-04T18:12:36.086 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:12:36.085+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:12:36.086651+0000) 2024-03-04T18:12:37.086 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:12:37.086+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:12:37.086928+0000) 2024-03-04T18:12:38.086 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:12:38.086+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:12:38.087172+0000) 2024-03-04T18:12:39.086 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:12:39.086+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:12:39.087437+0000) 2024-03-04T18:12:40.087 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:12:40.086+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:12:40.087667+0000) 2024-03-04T18:12:41.087 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:12:41.087+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:12:41.087950+0000) 2024-03-04T18:12:42.087 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:12:42.087+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:12:42.088149+0000) 2024-03-04T18:12:43.087 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:12:43.087+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:12:43.088414+0000) 2024-03-04T18:12:44.088 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:12:44.087+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:12:44.088677+0000) 2024-03-04T18:12:45.088 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:12:45.088+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:12:45.088945+0000) 2024-03-04T18:12:46.088 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:12:46.088+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:12:46.089185+0000) 2024-03-04T18:12:47.089 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:12:47.088+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:12:47.089434+0000) 2024-03-04T18:12:48.089 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:12:48.088+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:12:48.089712+0000) 2024-03-04T18:12:49.089 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:12:49.089+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:12:49.089947+0000) 2024-03-04T18:12:50.089 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:12:50.089+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:12:50.090182+0000) 2024-03-04T18:12:51.089 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:12:51.089+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:12:51.090416+0000) 2024-03-04T18:12:52.090 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:12:52.089+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:12:52.090673+0000) 2024-03-04T18:12:53.090 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:12:53.090+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:12:53.090944+0000) 2024-03-04T18:12:54.090 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:12:54.090+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:12:54.091202+0000) 2024-03-04T18:12:55.090 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:12:55.090+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:12:55.091447+0000) 2024-03-04T18:12:55.093 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:12:55.104 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:12:56.091 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:12:56.090+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:12:56.091677+0000) 2024-03-04T18:12:57.091 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:12:57.091+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:12:57.091955+0000) 2024-03-04T18:12:58.091 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:12:58.091+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:12:58.092194+0000) 2024-03-04T18:12:59.091 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:12:59.091+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:12:59.092437+0000) 2024-03-04T18:13:00.092 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:13:00.091+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:13:00.092707+0000) 2024-03-04T18:13:01.092 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:13:01.092+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:13:01.092947+0000) 2024-03-04T18:13:02.092 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:13:02.092+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:13:02.093193+0000) 2024-03-04T18:13:03.092 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:13:03.092+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:13:03.093446+0000) 2024-03-04T18:13:04.093 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:13:04.092+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:13:04.093708+0000) 2024-03-04T18:13:05.093 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:13:05.093+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:13:05.093954+0000) 2024-03-04T18:13:06.093 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:13:06.093+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:13:06.094187+0000) 2024-03-04T18:13:07.093 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:13:07.094+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:13:07.094427+0000) 2024-03-04T18:13:08.094 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:13:08.093+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:13:08.094684+0000) 2024-03-04T18:13:09.094 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:13:09.094+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:13:09.094947+0000) 2024-03-04T18:13:10.094 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:13:10.094+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:13:10.095188+0000) 2024-03-04T18:13:11.094 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:13:11.094+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:13:11.095382+0000) 2024-03-04T18:13:12.095 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:13:12.095+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:13:12.095619+0000) 2024-03-04T18:13:13.095 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:13:13.095+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:13:13.095915+0000) 2024-03-04T18:13:14.095 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:13:14.095+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:13:14.096161+0000) 2024-03-04T18:13:15.095 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:13:15.096+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:13:15.096424+0000) 2024-03-04T18:13:16.096 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:13:16.096+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:13:16.096657+0000) 2024-03-04T18:13:17.096 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:13:17.096+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:13:17.096945+0000) 2024-03-04T18:13:18.096 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:13:18.096+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:13:18.097178+0000) 2024-03-04T18:13:19.096 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:13:19.097+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:13:19.097422+0000) 2024-03-04T18:13:20.097 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:13:20.097+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:13:20.097713+0000) 2024-03-04T18:13:21.097 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:13:21.097+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:13:21.097963+0000) 2024-03-04T18:13:22.097 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:13:22.098+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:13:22.098209+0000) 2024-03-04T18:13:23.097 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:13:23.097+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:13:23.098459+0000) 2024-03-04T18:13:24.098 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:13:24.096+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:13:24.098692+0000) 2024-03-04T18:13:25.098 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:13:25.097+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:13:25.098935+0000) 2024-03-04T18:13:25.223 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:13:25.231 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:13:26.098 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:13:26.098+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:13:26.099178+0000) 2024-03-04T18:13:27.098 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:13:27.097+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:13:27.099437+0000) 2024-03-04T18:13:28.099 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:13:28.098+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:13:28.099678+0000) 2024-03-04T18:13:29.099 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:13:29.098+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:13:29.099949+0000) 2024-03-04T18:13:30.099 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:13:30.099+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:13:30.100184+0000) 2024-03-04T18:13:31.099 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:13:31.099+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:13:31.100427+0000) 2024-03-04T18:13:32.099 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:13:32.099+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:13:32.100676+0000) 2024-03-04T18:13:33.100 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:13:33.099+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:13:33.100956+0000) 2024-03-04T18:13:34.100 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:13:34.100+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:13:34.101194+0000) 2024-03-04T18:13:35.100 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:13:35.100+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:13:35.101430+0000) 2024-03-04T18:13:36.100 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:13:36.100+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:13:36.101671+0000) 2024-03-04T18:13:37.101 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:13:37.101+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:13:37.101944+0000) 2024-03-04T18:13:38.101 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:13:38.101+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:13:38.102192+0000) 2024-03-04T18:13:39.101 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:13:39.101+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:13:39.102431+0000) 2024-03-04T18:13:40.102 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:13:40.101+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:13:40.102701+0000) 2024-03-04T18:13:41.102 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:13:41.102+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:13:41.102955+0000) 2024-03-04T18:13:42.102 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:13:42.102+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:13:42.103200+0000) 2024-03-04T18:13:43.102 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:13:43.102+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:13:43.103463+0000) 2024-03-04T18:13:44.103 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:13:44.102+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:13:44.103709+0000) 2024-03-04T18:13:45.103 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:13:45.103+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:13:45.103972+0000) 2024-03-04T18:13:46.104 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:13:46.103+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:13:46.104248+0000) 2024-03-04T18:13:47.103 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:13:47.103+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:13:47.104450+0000) 2024-03-04T18:13:48.104 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:13:48.103+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:13:48.104695+0000) 2024-03-04T18:13:49.104 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:13:49.104+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:13:49.104944+0000) 2024-03-04T18:13:50.104 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:13:50.104+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:13:50.105178+0000) 2024-03-04T18:13:51.104 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:13:51.104+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:13:51.105422+0000) 2024-03-04T18:13:52.105 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:13:52.104+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:13:52.105683+0000) 2024-03-04T18:13:53.105 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:13:53.105+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:13:53.105948+0000) 2024-03-04T18:13:54.105 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:13:54.105+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:13:54.106189+0000) 2024-03-04T18:13:55.105 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:13:55.105+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:13:55.106381+0000) 2024-03-04T18:13:55.436 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:13:55.447 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:13:56.106 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:13:56.105+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:13:56.106651+0000) 2024-03-04T18:13:57.106 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:13:57.106+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:13:57.106936+0000) 2024-03-04T18:13:58.106 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:13:58.106+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:13:58.107183+0000) 2024-03-04T18:13:59.106 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:13:59.106+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:13:59.107425+0000) 2024-03-04T18:14:00.106 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:14:00.107+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:14:00.107666+0000) 2024-03-04T18:14:01.107 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:14:01.107+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:14:01.107951+0000) 2024-03-04T18:14:02.107 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:14:02.107+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:14:02.108199+0000) 2024-03-04T18:14:03.107 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:14:03.107+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:14:03.108411+0000) 2024-03-04T18:14:04.107 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:14:04.108+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:14:04.108658+0000) 2024-03-04T18:14:05.108 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:14:05.108+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:14:05.108934+0000) 2024-03-04T18:14:06.108 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:14:06.108+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:14:06.109179+0000) 2024-03-04T18:14:07.108 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:14:07.109+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:14:07.109423+0000) 2024-03-04T18:14:08.108 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:14:08.109+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:14:08.109734+0000) 2024-03-04T18:14:09.109 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:14:09.109+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:14:09.110000+0000) 2024-03-04T18:14:10.109 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:14:10.109+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:14:10.110215+0000) 2024-03-04T18:14:11.109 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:14:11.110+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:14:11.110487+0000) 2024-03-04T18:14:12.110 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:14:12.110+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:14:12.110728+0000) 2024-03-04T18:14:13.110 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:14:13.110+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:14:13.110990+0000) 2024-03-04T18:14:14.110 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:14:14.110+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:14:14.111253+0000) 2024-03-04T18:14:15.110 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:14:15.111+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:14:15.111519+0000) 2024-03-04T18:14:16.111 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:14:16.111+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:14:16.111763+0000) 2024-03-04T18:14:17.111 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:14:17.111+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:14:17.111966+0000) 2024-03-04T18:14:18.111 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:14:18.111+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:14:18.112232+0000) 2024-03-04T18:14:19.111 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:14:19.111+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:14:19.112509+0000) 2024-03-04T18:14:20.112 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:14:20.112+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:14:20.112741+0000) 2024-03-04T18:14:21.112 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:14:21.112+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:14:21.112985+0000) 2024-03-04T18:14:22.112 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:14:22.113+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:14:22.113237+0000) 2024-03-04T18:14:23.112 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:14:23.113+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:14:23.113495+0000) 2024-03-04T18:14:24.112 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:14:24.112+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:14:24.113755+0000) 2024-03-04T18:14:25.113 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:14:25.112+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:14:25.114001+0000) 2024-03-04T18:14:25.642 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:14:25.650 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:14:26.113 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:14:26.112+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:14:26.114208+0000) 2024-03-04T18:14:27.113 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:14:27.113+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:14:27.114463+0000) 2024-03-04T18:14:28.114 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:14:28.113+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:14:28.114712+0000) 2024-03-04T18:14:29.114 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:14:29.113+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:14:29.114971+0000) 2024-03-04T18:14:30.114 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:14:30.113+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:14:30.115168+0000) 2024-03-04T18:14:31.114 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:14:31.114+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:14:31.115409+0000) 2024-03-04T18:14:32.115 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:14:32.114+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:14:32.115653+0000) 2024-03-04T18:14:33.115 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:14:33.114+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:14:33.115955+0000) 2024-03-04T18:14:34.115 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:14:34.115+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:14:34.116202+0000) 2024-03-04T18:14:35.115 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:14:35.115+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:14:35.116447+0000) 2024-03-04T18:14:36.115 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:14:36.115+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:14:36.116691+0000) 2024-03-04T18:14:37.116 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:14:37.116+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:14:37.116935+0000) 2024-03-04T18:14:38.116 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:14:38.116+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:14:38.117184+0000) 2024-03-04T18:14:39.116 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:14:39.116+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:14:39.117426+0000) 2024-03-04T18:14:40.116 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:14:40.116+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:14:40.117683+0000) 2024-03-04T18:14:41.117 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:14:41.117+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:14:41.117961+0000) 2024-03-04T18:14:42.117 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:14:42.117+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:14:42.118210+0000) 2024-03-04T18:14:43.117 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:14:43.117+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:14:43.118469+0000) 2024-03-04T18:14:44.117 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:14:44.117+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:14:44.118656+0000) 2024-03-04T18:14:45.118 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:14:45.118+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:14:45.118962+0000) 2024-03-04T18:14:46.118 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:14:46.118+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:14:46.119226+0000) 2024-03-04T18:14:47.118 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:14:47.118+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:14:47.119474+0000) 2024-03-04T18:14:48.118 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:14:48.118+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:14:48.119719+0000) 2024-03-04T18:14:49.119 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:14:49.119+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:14:49.119965+0000) 2024-03-04T18:14:50.119 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:14:50.119+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:14:50.120202+0000) 2024-03-04T18:14:51.119 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:14:51.119+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:14:51.120495+0000) 2024-03-04T18:14:52.119 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:14:52.119+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:14:52.120728+0000) 2024-03-04T18:14:53.120 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:14:53.120+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:14:53.120985+0000) 2024-03-04T18:14:54.120 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:14:54.120+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:14:54.121232+0000) 2024-03-04T18:14:55.120 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:14:55.120+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:14:55.121477+0000) 2024-03-04T18:14:55.855 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:14:55.863 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:14:56.120 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:14:56.121+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:14:56.121730+0000) 2024-03-04T18:14:57.121 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:14:57.121+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:14:57.122012+0000) 2024-03-04T18:14:58.121 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:14:58.121+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:14:58.122216+0000) 2024-03-04T18:14:59.121 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:14:59.121+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:14:59.122467+0000) 2024-03-04T18:15:00.121 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:15:00.121+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:15:00.122705+0000) 2024-03-04T18:15:01.122 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:15:01.122+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:15:01.122950+0000) 2024-03-04T18:15:02.122 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:15:02.122+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:15:02.123221+0000) 2024-03-04T18:15:03.122 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:15:03.123+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:15:03.123512+0000) 2024-03-04T18:15:04.123 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:15:04.123+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:15:04.123773+0000) 2024-03-04T18:15:05.123 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:15:05.123+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:15:05.124015+0000) 2024-03-04T18:15:06.123 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:15:06.123+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:15:06.124263+0000) 2024-03-04T18:15:07.123 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:15:07.124+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:15:07.124505+0000) 2024-03-04T18:15:08.123 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:15:08.123+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:15:08.124720+0000) 2024-03-04T18:15:09.124 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:15:09.124+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:15:09.124959+0000) 2024-03-04T18:15:10.124 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:15:10.124+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:15:10.125204+0000) 2024-03-04T18:15:11.124 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:15:11.124+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:15:11.125432+0000) 2024-03-04T18:15:12.124 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:15:12.124+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:15:12.125623+0000) 2024-03-04T18:15:13.125 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:15:13.125+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:15:13.125852+0000) 2024-03-04T18:15:14.125 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:15:14.125+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:15:14.126029+0000) 2024-03-04T18:15:15.125 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:15:15.125+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:15:15.126220+0000) 2024-03-04T18:15:16.125 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:15:16.125+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:15:16.126418+0000) 2024-03-04T18:15:17.125 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:15:17.125+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:15:17.126670+0000) 2024-03-04T18:15:18.126 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:15:18.126+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:15:18.126950+0000) 2024-03-04T18:15:19.126 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:15:19.126+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:15:19.127150+0000) 2024-03-04T18:15:20.126 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:15:20.126+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:15:20.127333+0000) 2024-03-04T18:15:21.126 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:15:21.126+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:15:21.127524+0000) 2024-03-04T18:15:22.126 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:15:22.126+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:15:22.127716+0000) 2024-03-04T18:15:23.127 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:15:23.127+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:15:23.127965+0000) 2024-03-04T18:15:24.127 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:15:24.127+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:15:24.128180+0000) 2024-03-04T18:15:25.127 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:15:25.127+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:15:25.128421+0000) 2024-03-04T18:15:26.059 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:15:26.067 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:15:26.127 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:15:26.126+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:15:26.128677+0000) 2024-03-04T18:15:27.128 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:15:27.127+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:15:27.128963+0000) 2024-03-04T18:15:28.128 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:15:28.127+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:15:28.129165+0000) 2024-03-04T18:15:29.128 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:15:29.127+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:15:29.129352+0000) 2024-03-04T18:15:30.128 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:15:30.127+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:15:30.129531+0000) 2024-03-04T18:15:31.128 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:15:31.127+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:15:31.129718+0000) 2024-03-04T18:15:32.129 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:15:32.128+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:15:32.129931+0000) 2024-03-04T18:15:33.129 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:15:33.128+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:15:33.130136+0000) 2024-03-04T18:15:34.129 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:15:34.129+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:15:34.130331+0000) 2024-03-04T18:15:35.129 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:15:35.128+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:15:35.130508+0000) 2024-03-04T18:15:36.129 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:15:36.128+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:15:36.130743+0000) 2024-03-04T18:15:37.130 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:15:37.130+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:15:37.130987+0000) 2024-03-04T18:15:38.130 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:15:38.130+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:15:38.131241+0000) 2024-03-04T18:15:39.130 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:15:39.130+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:15:39.131437+0000) 2024-03-04T18:15:40.130 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:15:40.130+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:15:40.131627+0000) 2024-03-04T18:15:41.130 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:15:41.131+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:15:41.131900+0000) 2024-03-04T18:15:42.131 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:15:42.131+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:15:42.132143+0000) 2024-03-04T18:15:43.131 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:15:43.131+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:15:43.132425+0000) 2024-03-04T18:15:44.132 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:15:44.131+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:15:44.132716+0000) 2024-03-04T18:15:45.132 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:15:45.132+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:15:45.132952+0000) 2024-03-04T18:15:46.132 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:15:46.132+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:15:46.133211+0000) 2024-03-04T18:15:47.132 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:15:47.132+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:15:47.133396+0000) 2024-03-04T18:15:48.132 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:15:48.132+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:15:48.133638+0000) 2024-03-04T18:15:49.133 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:15:49.133+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:15:49.133939+0000) 2024-03-04T18:15:50.133 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:15:50.133+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:15:50.134205+0000) 2024-03-04T18:15:51.133 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:15:51.133+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:15:51.134466+0000) 2024-03-04T18:15:52.133 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:15:52.134+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:15:52.134710+0000) 2024-03-04T18:15:53.134 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:15:53.134+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:15:53.134987+0000) 2024-03-04T18:15:54.134 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:15:54.134+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:15:54.135230+0000) 2024-03-04T18:15:55.134 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:15:55.134+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:15:55.135426+0000) 2024-03-04T18:15:56.134 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:15:56.135+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:15:56.135685+0000) 2024-03-04T18:15:56.189 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:15:56.198 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:15:57.135 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:15:57.135+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:15:57.135936+0000) 2024-03-04T18:15:58.135 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:15:58.135+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:15:58.136183+0000) 2024-03-04T18:15:59.135 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:15:59.135+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:15:59.136379+0000) 2024-03-04T18:16:00.135 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:16:00.136+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:16:00.136621+0000) 2024-03-04T18:16:01.136 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:16:01.136+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:16:01.136895+0000) 2024-03-04T18:16:02.136 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:16:02.136+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:16:02.137134+0000) 2024-03-04T18:16:03.136 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:16:03.136+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:16:03.137340+0000) 2024-03-04T18:16:04.136 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:16:04.137+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:16:04.137585+0000) 2024-03-04T18:16:05.136 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:16:05.137+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:16:05.137885+0000) 2024-03-04T18:16:06.137 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:16:06.137+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:16:06.138119+0000) 2024-03-04T18:16:07.138 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:16:07.137+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:16:07.138357+0000) 2024-03-04T18:16:08.137 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:16:08.138+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:16:08.138659+0000) 2024-03-04T18:16:09.137 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:16:09.138+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:16:09.138959+0000) 2024-03-04T18:16:10.138 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:16:10.138+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:16:10.139200+0000) 2024-03-04T18:16:11.139 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:16:11.138+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:16:11.139427+0000) 2024-03-04T18:16:12.138 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:16:12.139+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:16:12.139680+0000) 2024-03-04T18:16:13.139 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:16:13.139+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:16:13.139982+0000) 2024-03-04T18:16:14.139 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:16:14.139+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:16:14.140248+0000) 2024-03-04T18:16:15.139 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:16:15.140+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:16:15.140498+0000) 2024-03-04T18:16:16.139 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:16:16.140+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:16:16.140760+0000) 2024-03-04T18:16:17.140 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:16:17.140+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:16:17.141015+0000) 2024-03-04T18:16:18.140 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:16:18.140+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:16:18.141271+0000) 2024-03-04T18:16:19.140 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:16:19.140+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:16:19.141519+0000) 2024-03-04T18:16:20.140 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:16:20.141+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:16:20.141767+0000) 2024-03-04T18:16:21.141 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:16:21.141+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:16:21.142021+0000) 2024-03-04T18:16:22.141 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:16:22.142+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:16:22.142276+0000) 2024-03-04T18:16:23.141 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:16:23.142+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:16:23.142547+0000) 2024-03-04T18:16:24.141 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:16:24.142+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:16:24.142801+0000) 2024-03-04T18:16:25.142 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:16:25.142+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:16:25.143103+0000) 2024-03-04T18:16:26.142 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:16:26.143+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:16:26.143361+0000) 2024-03-04T18:16:26.394 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:16:26.402 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:16:27.142 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:16:27.142+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:16:27.143642+0000) 2024-03-04T18:16:28.142 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:16:28.142+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:16:28.143951+0000) 2024-03-04T18:16:29.143 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:16:29.143+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:16:29.144223+0000) 2024-03-04T18:16:30.143 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:16:30.143+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:16:30.144487+0000) 2024-03-04T18:16:31.143 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:16:31.143+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:16:31.144743+0000) 2024-03-04T18:16:32.144 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:16:32.143+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:16:32.145002+0000) 2024-03-04T18:16:33.144 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:16:33.144+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:16:33.145275+0000) 2024-03-04T18:16:34.144 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:16:34.144+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:16:34.145520+0000) 2024-03-04T18:16:35.144 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:16:35.143+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:16:35.145752+0000) 2024-03-04T18:16:36.144 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:16:36.145+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:16:36.146015+0000) 2024-03-04T18:16:37.145 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:16:37.145+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:16:37.146241+0000) 2024-03-04T18:16:38.145 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:16:38.145+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:16:38.146557+0000) 2024-03-04T18:16:39.145 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:16:39.146+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:16:39.146816+0000) 2024-03-04T18:16:40.146 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:16:40.146+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:16:40.147094+0000) 2024-03-04T18:16:41.146 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:16:41.146+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:16:41.147353+0000) 2024-03-04T18:16:42.146 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:16:42.146+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:16:42.147598+0000) 2024-03-04T18:16:43.146 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:16:43.147+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:16:43.147960+0000) 2024-03-04T18:16:44.147 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:16:44.147+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:16:44.148183+0000) 2024-03-04T18:16:45.147 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:16:45.147+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:16:45.148444+0000) 2024-03-04T18:16:46.147 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:16:46.147+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:16:46.148698+0000) 2024-03-04T18:16:47.147 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:16:47.148+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:16:47.148951+0000) 2024-03-04T18:16:48.148 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:16:48.148+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:16:48.149173+0000) 2024-03-04T18:16:49.148 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:16:49.148+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:16:49.149433+0000) 2024-03-04T18:16:50.148 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:16:50.148+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:16:50.149693+0000) 2024-03-04T18:16:51.149 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:16:51.149+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:16:51.149953+0000) 2024-03-04T18:16:52.149 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:16:52.149+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:16:52.150208+0000) 2024-03-04T18:16:53.149 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:16:53.149+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:16:53.150422+0000) 2024-03-04T18:16:54.149 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:16:54.149+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:16:54.150692+0000) 2024-03-04T18:16:55.149 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:16:55.150+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:16:55.150946+0000) 2024-03-04T18:16:56.150 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:16:56.150+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:16:56.151200+0000) 2024-03-04T18:16:56.533 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:16:56.542 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:16:57.150 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:16:57.150+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:16:57.151457+0000) 2024-03-04T18:16:58.150 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:16:58.151+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:16:58.151710+0000) 2024-03-04T18:16:59.151 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:16:59.151+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:16:59.151970+0000) 2024-03-04T18:17:00.151 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:17:00.151+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:17:00.152225+0000) 2024-03-04T18:17:01.151 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:17:01.151+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:17:01.152485+0000) 2024-03-04T18:17:02.151 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:17:02.152+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:17:02.152739+0000) 2024-03-04T18:17:03.152 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:17:03.152+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:17:03.153031+0000) 2024-03-04T18:17:04.152 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:17:04.152+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:17:04.153304+0000) 2024-03-04T18:17:05.152 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:17:05.152+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:17:05.153576+0000) 2024-03-04T18:17:06.152 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:17:06.153+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:17:06.153877+0000) 2024-03-04T18:17:07.153 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:17:07.153+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:17:07.154096+0000) 2024-03-04T18:17:08.153 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:17:08.153+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:17:08.154346+0000) 2024-03-04T18:17:09.153 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:17:09.153+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:17:09.154617+0000) 2024-03-04T18:17:10.153 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:17:10.154+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:17:10.154922+0000) 2024-03-04T18:17:11.154 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:17:11.154+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:17:11.155116+0000) 2024-03-04T18:17:12.154 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:17:12.154+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:17:12.155348+0000) 2024-03-04T18:17:13.154 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:17:13.155+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:17:13.155597+0000) 2024-03-04T18:17:14.154 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:17:14.155+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:17:14.155866+0000) 2024-03-04T18:17:15.155 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:17:15.155+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:17:15.156106+0000) 2024-03-04T18:17:16.155 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:17:16.156+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:17:16.156351+0000) 2024-03-04T18:17:17.155 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:17:17.156+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:17:17.156607+0000) 2024-03-04T18:17:18.155 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:17:18.156+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:17:18.156915+0000) 2024-03-04T18:17:19.156 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:17:19.156+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:17:19.157158+0000) 2024-03-04T18:17:20.156 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:17:20.156+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:17:20.157349+0000) 2024-03-04T18:17:21.156 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:17:21.157+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:17:21.157587+0000) 2024-03-04T18:17:22.156 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:17:22.157+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:17:22.157864+0000) 2024-03-04T18:17:23.157 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:17:23.157+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:17:23.158128+0000) 2024-03-04T18:17:24.157 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:17:24.158+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:17:24.158408+0000) 2024-03-04T18:17:25.157 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:17:25.158+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:17:25.158641+0000) 2024-03-04T18:17:26.157 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:17:26.158+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:17:26.158957+0000) 2024-03-04T18:17:26.738 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:17:26.746 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:17:27.158 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:17:27.158+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:17:27.159190+0000) 2024-03-04T18:17:28.158 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:17:28.159+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:17:28.159435+0000) 2024-03-04T18:17:29.158 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:17:29.158+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:17:29.159678+0000) 2024-03-04T18:17:30.158 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:17:30.158+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:17:30.159954+0000) 2024-03-04T18:17:31.159 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:17:31.158+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:17:31.160202+0000) 2024-03-04T18:17:32.159 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:17:32.159+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:17:32.160453+0000) 2024-03-04T18:17:33.159 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:17:33.159+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:17:33.160713+0000) 2024-03-04T18:17:34.159 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:17:34.159+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:17:34.160935+0000) 2024-03-04T18:17:35.160 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:17:35.160+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:17:35.161171+0000) 2024-03-04T18:17:36.160 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:17:36.160+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:17:36.161414+0000) 2024-03-04T18:17:37.160 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:17:37.160+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:17:37.161668+0000) 2024-03-04T18:17:38.160 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:17:38.160+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:17:38.161933+0000) 2024-03-04T18:17:39.161 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:17:39.160+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:17:39.162144+0000) 2024-03-04T18:17:40.161 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:17:40.161+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:17:40.162325+0000) 2024-03-04T18:17:41.161 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:17:41.161+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:17:41.162582+0000) 2024-03-04T18:17:42.161 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:17:42.162+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:17:42.162868+0000) 2024-03-04T18:17:43.162 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:17:43.162+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:17:43.163127+0000) 2024-03-04T18:17:44.162 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:17:44.162+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:17:44.163312+0000) 2024-03-04T18:17:45.162 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:17:45.162+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:17:45.163549+0000) 2024-03-04T18:17:46.162 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:17:46.162+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:17:46.163790+0000) 2024-03-04T18:17:47.163 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:17:47.163+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:17:47.164043+0000) 2024-03-04T18:17:48.163 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:17:48.163+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:17:48.164299+0000) 2024-03-04T18:17:49.163 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:17:49.163+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:17:49.164561+0000) 2024-03-04T18:17:50.163 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:17:50.164+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:17:50.164806+0000) 2024-03-04T18:17:51.164 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:17:51.164+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:17:51.165078+0000) 2024-03-04T18:17:52.164 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:17:52.164+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:17:52.165316+0000) 2024-03-04T18:17:53.165 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:17:53.164+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:17:53.165600+0000) 2024-03-04T18:17:54.164 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:17:54.165+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:17:54.165884+0000) 2024-03-04T18:17:55.165 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:17:55.165+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:17:55.166159+0000) 2024-03-04T18:17:56.165 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:17:56.165+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:17:56.166394+0000) 2024-03-04T18:17:56.935 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:17:56.943 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:17:57.165 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:17:57.165+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:17:57.166601+0000) 2024-03-04T18:17:58.165 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:17:58.166+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:17:58.166893+0000) 2024-03-04T18:17:59.166 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:17:59.166+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:17:59.167132+0000) 2024-03-04T18:18:00.166 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:18:00.166+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:18:00.167369+0000) 2024-03-04T18:18:01.166 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:18:01.167+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:18:01.167604+0000) 2024-03-04T18:18:02.166 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:18:02.167+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:18:02.167885+0000) 2024-03-04T18:18:03.167 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:18:03.167+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:18:03.168142+0000) 2024-03-04T18:18:04.167 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:18:04.167+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:18:04.168377+0000) 2024-03-04T18:18:05.167 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:18:05.168+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:18:05.168615+0000) 2024-03-04T18:18:06.167 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:18:06.168+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:18:06.168890+0000) 2024-03-04T18:18:07.168 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:18:07.168+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:18:07.169127+0000) 2024-03-04T18:18:08.168 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:18:08.168+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:18:08.169373+0000) 2024-03-04T18:18:09.168 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:18:09.169+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:18:09.169609+0000) 2024-03-04T18:18:10.168 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:18:10.169+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:18:10.169906+0000) 2024-03-04T18:18:11.169 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:18:11.169+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:18:11.170143+0000) 2024-03-04T18:18:12.169 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:18:12.169+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:18:12.170403+0000) 2024-03-04T18:18:13.169 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:18:13.169+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:18:13.170668+0000) 2024-03-04T18:18:14.169 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:18:14.170+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:18:14.170997+0000) 2024-03-04T18:18:15.170 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:18:15.170+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:18:15.171234+0000) 2024-03-04T18:18:16.170 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:18:16.170+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:18:16.171464+0000) 2024-03-04T18:18:17.170 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:18:17.171+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:18:17.171702+0000) 2024-03-04T18:18:18.170 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:18:18.171+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:18:18.171973+0000) 2024-03-04T18:18:19.171 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:18:19.171+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:18:19.172212+0000) 2024-03-04T18:18:20.171 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:18:20.172+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:18:20.172452+0000) 2024-03-04T18:18:21.171 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:18:21.172+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:18:21.172713+0000) 2024-03-04T18:18:22.171 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:18:22.172+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:18:22.172952+0000) 2024-03-04T18:18:23.172 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:18:23.172+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:18:23.173212+0000) 2024-03-04T18:18:24.172 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:18:24.173+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:18:24.173455+0000) 2024-03-04T18:18:25.172 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:18:25.173+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:18:25.173691+0000) 2024-03-04T18:18:26.172 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:18:26.173+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:18:26.173930+0000) 2024-03-04T18:18:27.072 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:18:27.080 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:18:27.172 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:18:27.173+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:18:27.174131+0000) 2024-03-04T18:18:28.173 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:18:28.174+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:18:28.174354+0000) 2024-03-04T18:18:29.173 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:18:29.174+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:18:29.174601+0000) 2024-03-04T18:18:30.173 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:18:30.174+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:18:30.174866+0000) 2024-03-04T18:18:31.174 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:18:31.173+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:18:31.175117+0000) 2024-03-04T18:18:32.174 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:18:32.174+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:18:32.175353+0000) 2024-03-04T18:18:33.174 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:18:33.174+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:18:33.175606+0000) 2024-03-04T18:18:34.174 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:18:34.174+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:18:34.175920+0000) 2024-03-04T18:18:35.174 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:18:35.175+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:18:35.176162+0000) 2024-03-04T18:18:36.175 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:18:36.175+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:18:36.176432+0000) 2024-03-04T18:18:37.175 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:18:37.175+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:18:37.176674+0000) 2024-03-04T18:18:38.175 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:18:38.175+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:18:38.176946+0000) 2024-03-04T18:18:39.176 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:18:39.175+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:18:39.177182+0000) 2024-03-04T18:18:40.176 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:18:40.176+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:18:40.177423+0000) 2024-03-04T18:18:41.176 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:18:41.176+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:18:41.177662+0000) 2024-03-04T18:18:42.176 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:18:42.177+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:18:42.177941+0000) 2024-03-04T18:18:43.177 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:18:43.177+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:18:43.178154+0000) 2024-03-04T18:18:44.177 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:18:44.177+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:18:44.178347+0000) 2024-03-04T18:18:45.177 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:18:45.177+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:18:45.178610+0000) 2024-03-04T18:18:46.177 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:18:46.178+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:18:46.178886+0000) 2024-03-04T18:18:47.177 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:18:47.178+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:18:47.179137+0000) 2024-03-04T18:18:48.178 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:18:48.178+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:18:48.179397+0000) 2024-03-04T18:18:49.178 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:18:49.178+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:18:49.179672+0000) 2024-03-04T18:18:50.178 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:18:50.179+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:18:50.179932+0000) 2024-03-04T18:18:51.179 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:18:51.179+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:18:51.180176+0000) 2024-03-04T18:18:52.179 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:18:52.179+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:18:52.180422+0000) 2024-03-04T18:18:53.179 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:18:53.179+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:18:53.180669+0000) 2024-03-04T18:18:54.179 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:18:54.180+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:18:54.180932+0000) 2024-03-04T18:18:55.179 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:18:55.180+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:18:55.181169+0000) 2024-03-04T18:18:56.180 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:18:56.180+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:18:56.181415+0000) 2024-03-04T18:18:57.180 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:18:57.180+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:18:57.181654+0000) 2024-03-04T18:18:57.285 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:18:57.293 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:18:58.180 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:18:58.181+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:18:58.181945+0000) 2024-03-04T18:18:59.181 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:18:59.181+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:18:59.182183+0000) 2024-03-04T18:19:00.181 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:19:00.181+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:19:00.182423+0000) 2024-03-04T18:19:01.181 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:19:01.182+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:19:01.182668+0000) 2024-03-04T18:19:02.181 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:19:02.182+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:19:02.182972+0000) 2024-03-04T18:19:03.182 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:19:03.182+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:19:03.183226+0000) 2024-03-04T18:19:04.182 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:19:04.182+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:19:04.183475+0000) 2024-03-04T18:19:05.182 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:19:05.183+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:19:05.183715+0000) 2024-03-04T18:19:06.182 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:19:06.183+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:19:06.183961+0000) 2024-03-04T18:19:07.183 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:19:07.183+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:19:07.184205+0000) 2024-03-04T18:19:08.183 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:19:08.183+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:19:08.184450+0000) 2024-03-04T18:19:09.183 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:19:09.184+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:19:09.184733+0000) 2024-03-04T18:19:10.183 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:19:10.184+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:19:10.184994+0000) 2024-03-04T18:19:11.184 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:19:11.184+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:19:11.185240+0000) 2024-03-04T18:19:12.184 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:19:12.185+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:19:12.185480+0000) 2024-03-04T18:19:13.184 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:19:13.185+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:19:13.185737+0000) 2024-03-04T18:19:14.184 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:19:14.185+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:19:14.186044+0000) 2024-03-04T18:19:15.185 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:19:15.185+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:19:15.186239+0000) 2024-03-04T18:19:16.185 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:19:16.185+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:19:16.186437+0000) 2024-03-04T18:19:17.185 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:19:17.185+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:19:17.186619+0000) 2024-03-04T18:19:18.185 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:19:18.186+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:19:18.186926+0000) 2024-03-04T18:19:19.185 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:19:19.186+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:19:19.187173+0000) 2024-03-04T18:19:20.186 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:19:20.187+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:19:20.187418+0000) 2024-03-04T18:19:21.186 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:19:21.187+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:19:21.187660+0000) 2024-03-04T18:19:22.186 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:19:22.187+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:19:22.187932+0000) 2024-03-04T18:19:23.187 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:19:23.187+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:19:23.188188+0000) 2024-03-04T18:19:24.187 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:19:24.188+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:19:24.188435+0000) 2024-03-04T18:19:25.188 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:19:25.188+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:19:25.188673+0000) 2024-03-04T18:19:26.187 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:19:26.188+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:19:26.188951+0000) 2024-03-04T18:19:27.188 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:19:27.188+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:19:27.189189+0000) 2024-03-04T18:19:27.491 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:19:27.499 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:19:28.188 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:19:28.188+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:19:28.189454+0000) 2024-03-04T18:19:29.188 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:19:29.189+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:19:29.189700+0000) 2024-03-04T18:19:30.188 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:19:30.189+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:19:30.189943+0000) 2024-03-04T18:19:31.189 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:19:31.190+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:19:31.190191+0000) 2024-03-04T18:19:32.189 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:19:32.190+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:19:32.190432+0000) 2024-03-04T18:19:33.189 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:19:33.189+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:19:33.190689+0000) 2024-03-04T18:19:34.189 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:19:34.189+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:19:34.190942+0000) 2024-03-04T18:19:35.189 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:19:35.190+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:19:35.191182+0000) 2024-03-04T18:19:36.190 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:19:36.190+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:19:36.191422+0000) 2024-03-04T18:19:37.190 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:19:37.190+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:19:37.191659+0000) 2024-03-04T18:19:38.190 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:19:38.190+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:19:38.191954+0000) 2024-03-04T18:19:39.191 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:19:39.191+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:19:39.192206+0000) 2024-03-04T18:19:40.191 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:19:40.191+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:19:40.192438+0000) 2024-03-04T18:19:41.191 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:19:41.191+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:19:41.192683+0000) 2024-03-04T18:19:42.192 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:19:42.191+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:19:42.192957+0000) 2024-03-04T18:19:43.192 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:19:43.192+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:19:43.193231+0000) 2024-03-04T18:19:44.192 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:19:44.192+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:19:44.193482+0000) 2024-03-04T18:19:45.192 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:19:45.192+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:19:45.193719+0000) 2024-03-04T18:19:46.192 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:19:46.193+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:19:46.193985+0000) 2024-03-04T18:19:47.193 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:19:47.193+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:19:47.194220+0000) 2024-03-04T18:19:48.193 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:19:48.193+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:19:48.194460+0000) 2024-03-04T18:19:49.193 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:19:49.193+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:19:49.194729+0000) 2024-03-04T18:19:50.193 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:19:50.194+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:19:50.194972+0000) 2024-03-04T18:19:51.194 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:19:51.194+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:19:51.195213+0000) 2024-03-04T18:19:52.194 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:19:52.194+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:19:52.195453+0000) 2024-03-04T18:19:53.194 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:19:53.194+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:19:53.195721+0000) 2024-03-04T18:19:54.194 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:19:54.195+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:19:54.195973+0000) 2024-03-04T18:19:55.195 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:19:55.195+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:19:55.196212+0000) 2024-03-04T18:19:56.195 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:19:56.195+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:19:56.196454+0000) 2024-03-04T18:19:57.195 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:19:57.195+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:19:57.196689+0000) 2024-03-04T18:19:57.645 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:19:57.654 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:19:58.195 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:19:58.196+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:19:58.196934+0000) 2024-03-04T18:19:59.196 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:19:59.196+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:19:59.197172+0000) 2024-03-04T18:20:00.196 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:20:00.196+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:20:00.197436+0000) 2024-03-04T18:20:01.196 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:20:01.197+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:20:01.197684+0000) 2024-03-04T18:20:02.196 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:20:02.197+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:20:02.197950+0000) 2024-03-04T18:20:03.197 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:20:03.197+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:20:03.198207+0000) 2024-03-04T18:20:04.197 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:20:04.197+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:20:04.198409+0000) 2024-03-04T18:20:05.197 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:20:05.198+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:20:05.198646+0000) 2024-03-04T18:20:06.197 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:20:06.198+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:20:06.198926+0000) 2024-03-04T18:20:07.198 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:20:07.198+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:20:07.199115+0000) 2024-03-04T18:20:08.198 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:20:08.198+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:20:08.199306+0000) 2024-03-04T18:20:09.198 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:20:09.198+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:20:09.199498+0000) 2024-03-04T18:20:10.198 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:20:10.198+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:20:10.199779+0000) 2024-03-04T18:20:11.198 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:20:11.199+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:20:11.200019+0000) 2024-03-04T18:20:12.198 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:20:12.199+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:20:12.200207+0000) 2024-03-04T18:20:13.199 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:20:13.199+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:20:13.200478+0000) 2024-03-04T18:20:14.199 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:20:14.200+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:20:14.200728+0000) 2024-03-04T18:20:15.199 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:20:15.200+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:20:15.200962+0000) 2024-03-04T18:20:16.200 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:20:16.200+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:20:16.201220+0000) 2024-03-04T18:20:17.200 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:20:17.201+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:20:17.201460+0000) 2024-03-04T18:20:18.200 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:20:18.200+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:20:18.201703+0000) 2024-03-04T18:20:19.200 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:20:19.201+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:20:19.201989+0000) 2024-03-04T18:20:20.201 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:20:20.201+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:20:20.202220+0000) 2024-03-04T18:20:21.201 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:20:21.201+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:20:21.202475+0000) 2024-03-04T18:20:22.201 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:20:22.201+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:20:22.202733+0000) 2024-03-04T18:20:23.201 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:20:23.202+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:20:23.203001+0000) 2024-03-04T18:20:24.201 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:20:24.202+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:20:24.203195+0000) 2024-03-04T18:20:25.202 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:20:25.203+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:20:25.203430+0000) 2024-03-04T18:20:26.202 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:20:26.203+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:20:26.203670+0000) 2024-03-04T18:20:27.202 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:20:27.203+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:20:27.203946+0000) 2024-03-04T18:20:27.842 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:20:27.851 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:20:28.202 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:20:28.203+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:20:28.204189+0000) 2024-03-04T18:20:29.203 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:20:29.204+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:20:29.204432+0000) 2024-03-04T18:20:30.203 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:20:30.204+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:20:30.204674+0000) 2024-03-04T18:20:31.203 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:20:31.204+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:20:31.204963+0000) 2024-03-04T18:20:32.203 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:20:32.205+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:20:32.205202+0000) 2024-03-04T18:20:33.204 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:20:33.205+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:20:33.205448+0000) 2024-03-04T18:20:34.204 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:20:34.205+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:20:34.205718+0000) 2024-03-04T18:20:35.204 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:20:35.204+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:20:35.205956+0000) 2024-03-04T18:20:36.205 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:20:36.205+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:20:36.206219+0000) 2024-03-04T18:20:37.205 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:20:37.205+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:20:37.206463+0000) 2024-03-04T18:20:38.205 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:20:38.205+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:20:38.206703+0000) 2024-03-04T18:20:39.205 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:20:39.205+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:20:39.207000+0000) 2024-03-04T18:20:40.206 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:20:40.206+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:20:40.207292+0000) 2024-03-04T18:20:41.206 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:20:41.206+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:20:41.207590+0000) 2024-03-04T18:20:42.206 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:20:42.206+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:20:42.207929+0000) 2024-03-04T18:20:43.207 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:20:43.207+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:20:43.208220+0000) 2024-03-04T18:20:44.207 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:20:44.207+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:20:44.208481+0000) 2024-03-04T18:20:45.207 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:20:45.207+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:20:45.208728+0000) 2024-03-04T18:20:46.207 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:20:46.208+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:20:46.208970+0000) 2024-03-04T18:20:47.208 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:20:47.208+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:20:47.209209+0000) 2024-03-04T18:20:48.208 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:20:48.208+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:20:48.209441+0000) 2024-03-04T18:20:49.208 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:20:49.208+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:20:49.209677+0000) 2024-03-04T18:20:50.208 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:20:50.209+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:20:50.209915+0000) 2024-03-04T18:20:51.209 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:20:51.209+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:20:51.210149+0000) 2024-03-04T18:20:52.209 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:20:52.209+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:20:52.210390+0000) 2024-03-04T18:20:53.209 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:20:53.209+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:20:53.210658+0000) 2024-03-04T18:20:54.209 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:20:54.210+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:20:54.210948+0000) 2024-03-04T18:20:55.209 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:20:55.210+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:20:55.211181+0000) 2024-03-04T18:20:56.210 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:20:56.210+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:20:56.211424+0000) 2024-03-04T18:20:57.210 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:20:57.210+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:20:57.211670+0000) 2024-03-04T18:20:58.064 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:20:58.072 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:20:58.210 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:20:58.211+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:20:58.211962+0000) 2024-03-04T18:20:59.211 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:20:59.211+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:20:59.212219+0000) 2024-03-04T18:21:00.211 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:21:00.211+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:21:00.212467+0000) 2024-03-04T18:21:01.211 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:21:01.212+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:21:01.212708+0000) 2024-03-04T18:21:02.211 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:21:02.212+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:21:02.212950+0000) 2024-03-04T18:21:03.212 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:21:03.212+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:21:03.213200+0000) 2024-03-04T18:21:04.212 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:21:04.212+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:21:04.213451+0000) 2024-03-04T18:21:05.212 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:21:05.213+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:21:05.213691+0000) 2024-03-04T18:21:06.212 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:21:06.213+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:21:06.213933+0000) 2024-03-04T18:21:07.212 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:21:07.213+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:21:07.214179+0000) 2024-03-04T18:21:08.213 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:21:08.213+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:21:08.214422+0000) 2024-03-04T18:21:09.213 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:21:09.214+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:21:09.214668+0000) 2024-03-04T18:21:10.213 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:21:10.214+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:21:10.214949+0000) 2024-03-04T18:21:11.213 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:21:11.214+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:21:11.215190+0000) 2024-03-04T18:21:12.214 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:21:12.214+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:21:12.215429+0000) 2024-03-04T18:21:13.214 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:21:13.214+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:21:13.215691+0000) 2024-03-04T18:21:14.214 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:21:14.215+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:21:14.215943+0000) 2024-03-04T18:21:15.214 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:21:15.215+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:21:15.216194+0000) 2024-03-04T18:21:16.215 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:21:16.215+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:21:16.216479+0000) 2024-03-04T18:21:17.215 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:21:17.216+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:21:17.216767+0000) 2024-03-04T18:21:18.216 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:21:18.216+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:21:18.217047+0000) 2024-03-04T18:21:19.216 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:21:19.216+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:21:19.217365+0000) 2024-03-04T18:21:20.216 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:21:20.216+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:21:20.217558+0000) 2024-03-04T18:21:21.216 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:21:21.217+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:21:21.217795+0000) 2024-03-04T18:21:22.217 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:21:22.217+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:21:22.218074+0000) 2024-03-04T18:21:23.217 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:21:23.217+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:21:23.218298+0000) 2024-03-04T18:21:24.217 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:21:24.218+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:21:24.218559+0000) 2024-03-04T18:21:25.217 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:21:25.218+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:21:25.218862+0000) 2024-03-04T18:21:26.218 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:21:26.218+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:21:26.219122+0000) 2024-03-04T18:21:27.218 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:21:27.219+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:21:27.219375+0000) 2024-03-04T18:21:28.201 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:21:28.209 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:21:28.218 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:21:28.218+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:21:28.219630+0000) 2024-03-04T18:21:29.218 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:21:29.219+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:21:29.219956+0000) 2024-03-04T18:21:30.219 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:21:30.219+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:21:30.220221+0000) 2024-03-04T18:21:31.219 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:21:31.220+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:21:31.220502+0000) 2024-03-04T18:21:32.219 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:21:32.220+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:21:32.220760+0000) 2024-03-04T18:21:33.219 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:21:33.220+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:21:33.221033+0000) 2024-03-04T18:21:34.220 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:21:34.221+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:21:34.221294+0000) 2024-03-04T18:21:35.220 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:21:35.221+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:21:35.221527+0000) 2024-03-04T18:21:36.223 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:21:36.221+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:21:36.221871+0000) 2024-03-04T18:21:37.221 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:21:37.220+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:21:37.222113+0000) 2024-03-04T18:21:38.221 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:21:38.221+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:21:38.222367+0000) 2024-03-04T18:21:39.221 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:21:39.221+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:21:39.222606+0000) 2024-03-04T18:21:40.221 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:21:40.221+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:21:40.222911+0000) 2024-03-04T18:21:41.222 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:21:41.222+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:21:41.223150+0000) 2024-03-04T18:21:42.222 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:21:42.222+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:21:42.223393+0000) 2024-03-04T18:21:43.222 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:21:43.222+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:21:43.223643+0000) 2024-03-04T18:21:44.222 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:21:44.222+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:21:44.223885+0000) 2024-03-04T18:21:45.222 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:21:45.223+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:21:45.224129+0000) 2024-03-04T18:21:46.223 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:21:46.223+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:21:46.224374+0000) 2024-03-04T18:21:47.223 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:21:47.223+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:21:47.224631+0000) 2024-03-04T18:21:48.223 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:21:48.224+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:21:48.224943+0000) 2024-03-04T18:21:49.224 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:21:49.224+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:21:49.225187+0000) 2024-03-04T18:21:50.224 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:21:50.224+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:21:50.225433+0000) 2024-03-04T18:21:51.224 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:21:51.224+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:21:51.225676+0000) 2024-03-04T18:21:52.225 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:21:52.225+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:21:52.225979+0000) 2024-03-04T18:21:53.225 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:21:53.225+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:21:53.226228+0000) 2024-03-04T18:21:54.225 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:21:54.225+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:21:54.226492+0000) 2024-03-04T18:21:55.225 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:21:55.225+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:21:55.226733+0000) 2024-03-04T18:21:56.225 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:21:56.226+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:21:56.226977+0000) 2024-03-04T18:21:57.226 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:21:57.226+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:21:57.227228+0000) 2024-03-04T18:21:58.226 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:21:58.226+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:21:58.227485+0000) 2024-03-04T18:21:58.356 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:21:58.364 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:21:59.226 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:21:59.226+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:21:59.227703+0000) 2024-03-04T18:22:00.226 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:22:00.227+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:22:00.227962+0000) 2024-03-04T18:22:01.226 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:22:01.227+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:22:01.228223+0000) 2024-03-04T18:22:02.227 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:22:02.227+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:22:02.228453+0000) 2024-03-04T18:22:03.227 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:22:03.227+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:22:03.228656+0000) 2024-03-04T18:22:04.227 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:22:04.228+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:22:04.228948+0000) 2024-03-04T18:22:05.227 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:22:05.228+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:22:05.229188+0000) 2024-03-04T18:22:06.228 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:22:06.228+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:22:06.229439+0000) 2024-03-04T18:22:07.228 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:22:07.228+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:22:07.229701+0000) 2024-03-04T18:22:08.228 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:22:08.229+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:22:08.229988+0000) 2024-03-04T18:22:09.229 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:22:09.229+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:22:09.230181+0000) 2024-03-04T18:22:10.229 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:22:10.229+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:22:10.230428+0000) 2024-03-04T18:22:11.229 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:22:11.230+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:22:11.230672+0000) 2024-03-04T18:22:12.229 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:22:12.230+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:22:12.230945+0000) 2024-03-04T18:22:13.229 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:22:13.230+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:22:13.231200+0000) 2024-03-04T18:22:14.230 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:22:14.230+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:22:14.231449+0000) 2024-03-04T18:22:15.230 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:22:15.231+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:22:15.231692+0000) 2024-03-04T18:22:16.230 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:22:16.231+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:22:16.231933+0000) 2024-03-04T18:22:17.231 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:22:17.231+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:22:17.232203+0000) 2024-03-04T18:22:18.231 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:22:18.231+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:22:18.232402+0000) 2024-03-04T18:22:19.231 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:22:19.231+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:22:19.232641+0000) 2024-03-04T18:22:20.231 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:22:20.232+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:22:20.232933+0000) 2024-03-04T18:22:21.231 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:22:21.232+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:22:21.233173+0000) 2024-03-04T18:22:22.232 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:22:22.233+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:22:22.233416+0000) 2024-03-04T18:22:23.232 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:22:23.233+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:22:23.233666+0000) 2024-03-04T18:22:24.232 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:22:24.233+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:22:24.234045+0000) 2024-03-04T18:22:25.232 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:22:25.233+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:22:25.234284+0000) 2024-03-04T18:22:26.233 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:22:26.233+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:22:26.234543+0000) 2024-03-04T18:22:27.233 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:22:27.233+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:22:27.234801+0000) 2024-03-04T18:22:28.233 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:22:28.234+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:22:28.235073+0000) 2024-03-04T18:22:28.578 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:22:28.586 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:22:29.233 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:22:29.235+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:22:29.235316+0000) 2024-03-04T18:22:30.234 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:22:30.235+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:22:30.235558+0000) 2024-03-04T18:22:31.234 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:22:31.235+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:22:31.235800+0000) 2024-03-04T18:22:32.234 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:22:32.235+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:22:32.236037+0000) 2024-03-04T18:22:33.235 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:22:33.236+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:22:33.236300+0000) 2024-03-04T18:22:34.235 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:22:34.235+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:22:34.236560+0000) 2024-03-04T18:22:35.235 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:22:35.236+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:22:35.236811+0000) 2024-03-04T18:22:36.236 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:22:36.236+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:22:36.237104+0000) 2024-03-04T18:22:37.236 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:22:37.236+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:22:37.237368+0000) 2024-03-04T18:22:38.236 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:22:38.237+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:22:38.237614+0000) 2024-03-04T18:22:39.239 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:22:39.236+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:22:39.237891+0000) 2024-03-04T18:22:40.237 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:22:40.237+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:22:40.238134+0000) 2024-03-04T18:22:41.237 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:22:41.236+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:22:41.238364+0000) 2024-03-04T18:22:42.237 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:22:42.236+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:22:42.238621+0000) 2024-03-04T18:22:43.238 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:22:43.237+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:22:43.238922+0000) 2024-03-04T18:22:44.237 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:22:44.237+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:22:44.239156+0000) 2024-03-04T18:22:45.238 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:22:45.237+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:22:45.239390+0000) 2024-03-04T18:22:46.238 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:22:46.238+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:22:46.239674+0000) 2024-03-04T18:22:47.238 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:22:47.238+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:22:47.239933+0000) 2024-03-04T18:22:48.239 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:22:48.239+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:22:48.240143+0000) 2024-03-04T18:22:49.239 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:22:49.239+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:22:49.240341+0000) 2024-03-04T18:22:50.239 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:22:50.239+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:22:50.240580+0000) 2024-03-04T18:22:51.239 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:22:51.240+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:22:51.240865+0000) 2024-03-04T18:22:52.239 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:22:52.240+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:22:52.241111+0000) 2024-03-04T18:22:53.240 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:22:53.240+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:22:53.241396+0000) 2024-03-04T18:22:54.241 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:22:54.240+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:22:54.241641+0000) 2024-03-04T18:22:55.240 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:22:55.241+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:22:55.241921+0000) 2024-03-04T18:22:56.240 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:22:56.241+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:22:56.242157+0000) 2024-03-04T18:22:57.241 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:22:57.241+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:22:57.242351+0000) 2024-03-04T18:22:58.241 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:22:58.241+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:22:58.242611+0000) 2024-03-04T18:22:58.793 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:22:58.801 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:22:59.241 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:22:59.242+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:22:59.242914+0000) 2024-03-04T18:23:00.242 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:23:00.242+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:23:00.243124+0000) 2024-03-04T18:23:01.242 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:23:01.242+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:23:01.243360+0000) 2024-03-04T18:23:02.242 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:23:02.242+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:23:02.243624+0000) 2024-03-04T18:23:03.242 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:23:03.243+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:23:03.243920+0000) 2024-03-04T18:23:04.242 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:23:04.243+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:23:04.244171+0000) 2024-03-04T18:23:05.243 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:23:05.243+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:23:05.244415+0000) 2024-03-04T18:23:06.243 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:23:06.243+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:23:06.244669+0000) 2024-03-04T18:23:07.243 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:23:07.244+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:23:07.244960+0000) 2024-03-04T18:23:08.243 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:23:08.244+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:23:08.245217+0000) 2024-03-04T18:23:09.244 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:23:09.244+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:23:09.245462+0000) 2024-03-04T18:23:10.244 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:23:10.245+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:23:10.245706+0000) 2024-03-04T18:23:11.244 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:23:11.245+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:23:11.245950+0000) 2024-03-04T18:23:12.244 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:23:12.245+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:23:12.246246+0000) 2024-03-04T18:23:13.245 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:23:13.245+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:23:13.246501+0000) 2024-03-04T18:23:14.245 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:23:14.246+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:23:14.246750+0000) 2024-03-04T18:23:15.245 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:23:15.246+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:23:15.247071+0000) 2024-03-04T18:23:16.246 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:23:16.246+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:23:16.247349+0000) 2024-03-04T18:23:17.246 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:23:17.247+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:23:17.247591+0000) 2024-03-04T18:23:18.246 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:23:18.247+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:23:18.247869+0000) 2024-03-04T18:23:19.246 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:23:19.247+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:23:19.248124+0000) 2024-03-04T18:23:20.247 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:23:20.247+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:23:20.248371+0000) 2024-03-04T18:23:21.247 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:23:21.247+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:23:21.248627+0000) 2024-03-04T18:23:22.247 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:23:22.248+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:23:22.248938+0000) 2024-03-04T18:23:23.248 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:23:23.248+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:23:23.249254+0000) 2024-03-04T18:23:24.248 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:23:24.249+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:23:24.249574+0000) 2024-03-04T18:23:25.248 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:23:25.249+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:23:25.249918+0000) 2024-03-04T18:23:26.248 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:23:26.249+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:23:26.250196+0000) 2024-03-04T18:23:27.249 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:23:27.249+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:23:27.250442+0000) 2024-03-04T18:23:28.249 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:23:28.249+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:23:28.250756+0000) 2024-03-04T18:23:28.996 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:23:29.005 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:23:29.249 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:23:29.250+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:23:29.251047+0000) 2024-03-04T18:23:30.249 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:23:30.250+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:23:30.251329+0000) 2024-03-04T18:23:31.250 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:23:31.251+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:23:31.251624+0000) 2024-03-04T18:23:32.250 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:23:32.251+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:23:32.251962+0000) 2024-03-04T18:23:33.251 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:23:33.251+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:23:33.252262+0000) 2024-03-04T18:23:34.251 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:23:34.252+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:23:34.252549+0000) 2024-03-04T18:23:35.251 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:23:35.252+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:23:35.252890+0000) 2024-03-04T18:23:36.251 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:23:36.252+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:23:36.253173+0000) 2024-03-04T18:23:37.252 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:23:37.253+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:23:37.253463+0000) 2024-03-04T18:23:38.252 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:23:38.252+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:23:38.253691+0000) 2024-03-04T18:23:39.252 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:23:39.253+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:23:39.253985+0000) 2024-03-04T18:23:40.253 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:23:40.254+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:23:40.254264+0000) 2024-03-04T18:23:41.253 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:23:41.253+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:23:41.254563+0000) 2024-03-04T18:23:42.253 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:23:42.253+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:23:42.254923+0000) 2024-03-04T18:23:43.253 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:23:43.254+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:23:43.255249+0000) 2024-03-04T18:23:44.254 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:23:44.253+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:23:44.255557+0000) 2024-03-04T18:23:45.255 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:23:45.254+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:23:45.255792+0000) 2024-03-04T18:23:46.254 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:23:46.254+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:23:46.256111+0000) 2024-03-04T18:23:47.255 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:23:47.255+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:23:47.256397+0000) 2024-03-04T18:23:48.255 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:23:48.255+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:23:48.256644+0000) 2024-03-04T18:23:49.255 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:23:49.255+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:23:49.256976+0000) 2024-03-04T18:23:50.256 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:23:50.256+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:23:50.257285+0000) 2024-03-04T18:23:51.256 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:23:51.256+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:23:51.257592+0000) 2024-03-04T18:23:52.256 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:23:52.257+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:23:52.257931+0000) 2024-03-04T18:23:53.256 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:23:53.257+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:23:53.258205+0000) 2024-03-04T18:23:54.257 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:23:54.257+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:23:54.258504+0000) 2024-03-04T18:23:55.257 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:23:55.257+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:23:55.258795+0000) 2024-03-04T18:23:56.257 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:23:56.258+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:23:56.259168+0000) 2024-03-04T18:23:57.258 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:23:57.258+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:23:57.259463+0000) 2024-03-04T18:23:58.258 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:23:58.258+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:23:58.259700+0000) 2024-03-04T18:23:59.134 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:23:59.142 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:23:59.272 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:23:59.259+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:23:59.259958+0000) 2024-03-04T18:24:00.258 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:24:00.259+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:24:00.260200+0000) 2024-03-04T18:24:01.259 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:24:01.259+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:24:01.260438+0000) 2024-03-04T18:24:02.259 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:24:02.259+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:24:02.260674+0000) 2024-03-04T18:24:03.259 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:24:03.260+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:24:03.260953+0000) 2024-03-04T18:24:04.260 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:24:04.260+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:24:04.261201+0000) 2024-03-04T18:24:05.260 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:24:05.260+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:24:05.261452+0000) 2024-03-04T18:24:06.260 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:24:06.260+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:24:06.261684+0000) 2024-03-04T18:24:07.260 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:24:07.261+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:24:07.261967+0000) 2024-03-04T18:24:08.260 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:24:08.261+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:24:08.262236+0000) 2024-03-04T18:24:09.261 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:24:09.261+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:24:09.262517+0000) 2024-03-04T18:24:10.262 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:24:10.262+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:24:10.262785+0000) 2024-03-04T18:24:11.261 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:24:11.262+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:24:11.263067+0000) 2024-03-04T18:24:12.262 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:24:12.262+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:24:12.263301+0000) 2024-03-04T18:24:13.262 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:24:13.262+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:24:13.263546+0000) 2024-03-04T18:24:14.262 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:24:14.263+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:24:14.263800+0000) 2024-03-04T18:24:15.262 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:24:15.263+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:24:15.264082+0000) 2024-03-04T18:24:16.262 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:24:16.263+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:24:16.264279+0000) 2024-03-04T18:24:17.263 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:24:17.263+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:24:17.264534+0000) 2024-03-04T18:24:18.263 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:24:18.263+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:24:18.264732+0000) 2024-03-04T18:24:19.263 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:24:19.264+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:24:19.265000+0000) 2024-03-04T18:24:20.264 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:24:20.264+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:24:20.265245+0000) 2024-03-04T18:24:21.264 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:24:21.264+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:24:21.265501+0000) 2024-03-04T18:24:22.264 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:24:22.265+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:24:22.265742+0000) 2024-03-04T18:24:23.264 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:24:23.265+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:24:23.265992+0000) 2024-03-04T18:24:24.265 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:24:24.265+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:24:24.266209+0000) 2024-03-04T18:24:25.265 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:24:25.265+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:24:25.266462+0000) 2024-03-04T18:24:26.265 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:24:26.265+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:24:26.266696+0000) 2024-03-04T18:24:27.265 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:24:27.266+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:24:27.266964+0000) 2024-03-04T18:24:28.265 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:24:28.266+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:24:28.267157+0000) 2024-03-04T18:24:29.266 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:24:29.267+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:24:29.267403+0000) 2024-03-04T18:24:29.330 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:24:29.341 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:24:30.266 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:24:30.266+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:24:30.267669+0000) 2024-03-04T18:24:31.266 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:24:31.267+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:24:31.267947+0000) 2024-03-04T18:24:32.266 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:24:32.267+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:24:32.268206+0000) 2024-03-04T18:24:33.267 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:24:33.267+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:24:33.268418+0000) 2024-03-04T18:24:34.267 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:24:34.267+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:24:34.268624+0000) 2024-03-04T18:24:35.267 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:24:35.268+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:24:35.268941+0000) 2024-03-04T18:24:36.267 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:24:36.268+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:24:36.269197+0000) 2024-03-04T18:24:37.268 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:24:37.269+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:24:37.269452+0000) 2024-03-04T18:24:38.268 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:24:38.268+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:24:38.269705+0000) 2024-03-04T18:24:39.268 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:24:39.269+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:24:39.269970+0000) 2024-03-04T18:24:40.268 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:24:40.270+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:24:40.270244+0000) 2024-03-04T18:24:41.269 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:24:41.270+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:24:41.270498+0000) 2024-03-04T18:24:42.269 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:24:42.269+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:24:42.270745+0000) 2024-03-04T18:24:43.269 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:24:43.269+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:24:43.270994+0000) 2024-03-04T18:24:44.270 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:24:44.270+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:24:44.271238+0000) 2024-03-04T18:24:45.270 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:24:45.269+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:24:45.271439+0000) 2024-03-04T18:24:46.270 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:24:46.270+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:24:46.271681+0000) 2024-03-04T18:24:47.270 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:24:47.270+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:24:47.271943+0000) 2024-03-04T18:24:48.272 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:24:48.270+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:24:48.272139+0000) 2024-03-04T18:24:49.271 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:24:49.271+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:24:49.272400+0000) 2024-03-04T18:24:50.271 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:24:50.271+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:24:50.272683+0000) 2024-03-04T18:24:51.271 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:24:51.271+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:24:51.272939+0000) 2024-03-04T18:24:52.271 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:24:52.272+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:24:52.273177+0000) 2024-03-04T18:24:53.272 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:24:53.271+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:24:53.273386+0000) 2024-03-04T18:24:54.272 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:24:54.272+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:24:54.273683+0000) 2024-03-04T18:24:55.272 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:24:55.273+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:24:55.273991+0000) 2024-03-04T18:24:56.272 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:24:56.273+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:24:56.274193+0000) 2024-03-04T18:24:57.273 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:24:57.273+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:24:57.274381+0000) 2024-03-04T18:24:58.273 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:24:58.273+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:24:58.274588+0000) 2024-03-04T18:24:59.273 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:24:59.274+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:24:59.274951+0000) 2024-03-04T18:24:59.477 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:24:59.488 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:25:00.273 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:25:00.274+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:25:00.275158+0000) 2024-03-04T18:25:01.274 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:25:01.274+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:25:01.275401+0000) 2024-03-04T18:25:02.274 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:25:02.274+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:25:02.275644+0000) 2024-03-04T18:25:03.274 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:25:03.275+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:25:03.275921+0000) 2024-03-04T18:25:04.274 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:25:04.275+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:25:04.276166+0000) 2024-03-04T18:25:05.275 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:25:05.275+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:25:05.276429+0000) 2024-03-04T18:25:06.275 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:25:06.275+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:25:06.276678+0000) 2024-03-04T18:25:07.275 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:25:07.276+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:25:07.276983+0000) 2024-03-04T18:25:08.275 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:25:08.276+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:25:08.277258+0000) 2024-03-04T18:25:09.276 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:25:09.276+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:25:09.277521+0000) 2024-03-04T18:25:10.276 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:25:10.277+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:25:10.277773+0000) 2024-03-04T18:25:11.276 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:25:11.277+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:25:11.278013+0000) 2024-03-04T18:25:12.276 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:25:12.277+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:25:12.278258+0000) 2024-03-04T18:25:13.277 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:25:13.277+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:25:13.278507+0000) 2024-03-04T18:25:14.277 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:25:14.278+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:25:14.278762+0000) 2024-03-04T18:25:15.277 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:25:15.278+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:25:15.279023+0000) 2024-03-04T18:25:16.278 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:25:16.278+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:25:16.279264+0000) 2024-03-04T18:25:17.278 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:25:17.278+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:25:17.279504+0000) 2024-03-04T18:25:18.278 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:25:18.279+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:25:18.279752+0000) 2024-03-04T18:25:19.278 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:25:19.279+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:25:19.279982+0000) 2024-03-04T18:25:20.278 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:25:20.279+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:25:20.280226+0000) 2024-03-04T18:25:21.279 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:25:21.279+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:25:21.280485+0000) 2024-03-04T18:25:22.279 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:25:22.280+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:25:22.280725+0000) 2024-03-04T18:25:23.279 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:25:23.280+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:25:23.280984+0000) 2024-03-04T18:25:24.279 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:25:24.280+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:25:24.281229+0000) 2024-03-04T18:25:25.280 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:25:25.281+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:25:25.281468+0000) 2024-03-04T18:25:26.280 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:25:26.281+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:25:26.281723+0000) 2024-03-04T18:25:27.280 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:25:27.281+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:25:27.281993+0000) 2024-03-04T18:25:28.280 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:25:28.281+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:25:28.282195+0000) 2024-03-04T18:25:29.281 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:25:29.282+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:25:29.282440+0000) 2024-03-04T18:25:29.707 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:25:29.717 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:25:30.281 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:25:30.282+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:25:30.282686+0000) 2024-03-04T18:25:31.281 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:25:31.282+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:25:31.282969+0000) 2024-03-04T18:25:32.281 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:25:32.282+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:25:32.283148+0000) 2024-03-04T18:25:33.282 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:25:33.283+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:25:33.283407+0000) 2024-03-04T18:25:34.282 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:25:34.283+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:25:34.283655+0000) 2024-03-04T18:25:35.282 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:25:35.283+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:25:35.283933+0000) 2024-03-04T18:25:36.283 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:25:36.283+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:25:36.284174+0000) 2024-03-04T18:25:37.283 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:25:37.284+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:25:37.284407+0000) 2024-03-04T18:25:38.283 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:25:38.283+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:25:38.284646+0000) 2024-03-04T18:25:39.283 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:25:39.284+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:25:39.284925+0000) 2024-03-04T18:25:40.284 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:25:40.284+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:25:40.285163+0000) 2024-03-04T18:25:41.284 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:25:41.285+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:25:41.285418+0000) 2024-03-04T18:25:42.284 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:25:42.285+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:25:42.285703+0000) 2024-03-04T18:25:43.284 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:25:43.285+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:25:43.285958+0000) 2024-03-04T18:25:44.284 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:25:44.286+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:25:44.286204+0000) 2024-03-04T18:25:45.285 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:25:45.285+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:25:45.286443+0000) 2024-03-04T18:25:46.285 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:25:46.285+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:25:46.286687+0000) 2024-03-04T18:25:47.285 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:25:47.285+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:25:47.286933+0000) 2024-03-04T18:25:48.286 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:25:48.286+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:25:48.287179+0000) 2024-03-04T18:25:49.286 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:25:49.286+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:25:49.287416+0000) 2024-03-04T18:25:50.286 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:25:50.286+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:25:50.287660+0000) 2024-03-04T18:25:51.287 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:25:51.286+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:25:51.287936+0000) 2024-03-04T18:25:52.286 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:25:52.287+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:25:52.288191+0000) 2024-03-04T18:25:53.287 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:25:53.287+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:25:53.288447+0000) 2024-03-04T18:25:54.287 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:25:54.287+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:25:54.288711+0000) 2024-03-04T18:25:55.287 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:25:55.288+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:25:55.288967+0000) 2024-03-04T18:25:56.288 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:25:56.288+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:25:56.289212+0000) 2024-03-04T18:25:57.288 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:25:57.288+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:25:57.289444+0000) 2024-03-04T18:25:58.288 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:25:58.288+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:25:58.289693+0000) 2024-03-04T18:25:59.288 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:25:59.289+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:25:59.289930+0000) 2024-03-04T18:25:59.863 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:25:59.870 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:26:00.288 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:26:00.289+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:26:00.290173+0000) 2024-03-04T18:26:01.289 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:26:01.289+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:26:01.290435+0000) 2024-03-04T18:26:02.289 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:26:02.289+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:26:02.290690+0000) 2024-03-04T18:26:03.289 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:26:03.290+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:26:03.290942+0000) 2024-03-04T18:26:04.290 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:26:04.290+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:26:04.291181+0000) 2024-03-04T18:26:05.290 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:26:05.290+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:26:05.291414+0000) 2024-03-04T18:26:06.290 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:26:06.290+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:26:06.291655+0000) 2024-03-04T18:26:07.290 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:26:07.291+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:26:07.291959+0000) 2024-03-04T18:26:08.290 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:26:08.291+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:26:08.292223+0000) 2024-03-04T18:26:09.291 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:26:09.291+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:26:09.292461+0000) 2024-03-04T18:26:10.291 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:26:10.292+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:26:10.292723+0000) 2024-03-04T18:26:11.292 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:26:11.292+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:26:11.292984+0000) 2024-03-04T18:26:12.291 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:26:12.292+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:26:12.293167+0000) 2024-03-04T18:26:13.292 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:26:13.292+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:26:13.293415+0000) 2024-03-04T18:26:14.292 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:26:14.293+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:26:14.293655+0000) 2024-03-04T18:26:15.292 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:26:15.293+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:26:15.293869+0000) 2024-03-04T18:26:16.292 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:26:16.293+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:26:16.294117+0000) 2024-03-04T18:26:17.293 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:26:17.293+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:26:17.294352+0000) 2024-03-04T18:26:18.293 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:26:18.293+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:26:18.294584+0000) 2024-03-04T18:26:19.294 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:26:19.294+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:26:19.294903+0000) 2024-03-04T18:26:20.293 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:26:20.294+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:26:20.295136+0000) 2024-03-04T18:26:21.294 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:26:21.294+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:26:21.295375+0000) 2024-03-04T18:26:22.294 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:26:22.294+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:26:22.295610+0000) 2024-03-04T18:26:23.294 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:26:23.295+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:26:23.295926+0000) 2024-03-04T18:26:24.294 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:26:24.295+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:26:24.296191+0000) 2024-03-04T18:26:25.295 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:26:25.295+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:26:25.296465+0000) 2024-03-04T18:26:26.295 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:26:26.295+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:26:26.296716+0000) 2024-03-04T18:26:27.295 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:26:27.296+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:26:27.296997+0000) 2024-03-04T18:26:28.296 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:26:28.296+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:26:28.297243+0000) 2024-03-04T18:26:29.296 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:26:29.297+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:26:29.297483+0000) 2024-03-04T18:26:30.059 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:26:30.067 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:26:30.296 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:26:30.297+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:26:30.297792+0000) 2024-03-04T18:26:31.296 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:26:31.297+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:26:31.298057+0000) 2024-03-04T18:26:32.297 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:26:32.297+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:26:32.298299+0000) 2024-03-04T18:26:33.297 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:26:33.297+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:26:33.298567+0000) 2024-03-04T18:26:34.297 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:26:34.298+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:26:34.298802+0000) 2024-03-04T18:26:35.297 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:26:35.298+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:26:35.299154+0000) 2024-03-04T18:26:36.298 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:26:36.299+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:26:36.299412+0000) 2024-03-04T18:26:37.298 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:26:37.299+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:26:37.299659+0000) 2024-03-04T18:26:38.298 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:26:38.299+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:26:38.299957+0000) 2024-03-04T18:26:39.298 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:26:39.299+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:26:39.300296+0000) 2024-03-04T18:26:40.299 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:26:40.299+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:26:40.300506+0000) 2024-03-04T18:26:41.299 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:26:41.300+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:26:41.300759+0000) 2024-03-04T18:26:42.299 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:26:42.300+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:26:42.301002+0000) 2024-03-04T18:26:43.300 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:26:43.301+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:26:43.301352+0000) 2024-03-04T18:26:44.300 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:26:44.301+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:26:44.301653+0000) 2024-03-04T18:26:45.300 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:26:45.301+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:26:45.301989+0000) 2024-03-04T18:26:46.301 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:26:46.302+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:26:46.302293+0000) 2024-03-04T18:26:47.301 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:26:47.300+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:26:47.302532+0000) 2024-03-04T18:26:48.301 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:26:48.301+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:26:48.302863+0000) 2024-03-04T18:26:49.301 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:26:49.302+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:26:49.303189+0000) 2024-03-04T18:26:50.301 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:26:50.301+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:26:50.303418+0000) 2024-03-04T18:26:51.302 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:26:51.301+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:26:51.303649+0000) 2024-03-04T18:26:52.302 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:26:52.302+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:26:52.303981+0000) 2024-03-04T18:26:53.303 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:26:53.302+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:26:53.304282+0000) 2024-03-04T18:26:54.303 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:26:54.303+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:26:54.304579+0000) 2024-03-04T18:26:55.303 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:26:55.303+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:26:55.304926+0000) 2024-03-04T18:26:56.304 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:26:56.304+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:26:56.305246+0000) 2024-03-04T18:26:57.304 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:26:57.304+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:26:57.305547+0000) 2024-03-04T18:26:58.304 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:26:58.304+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:26:58.305791+0000) 2024-03-04T18:26:59.304 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:26:59.305+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:26:59.306125+0000) 2024-03-04T18:27:00.197 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:27:00.205 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:27:00.304 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:27:00.305+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:27:00.306343+0000) 2024-03-04T18:27:01.305 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:27:01.305+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:27:01.306652+0000) 2024-03-04T18:27:02.305 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:27:02.306+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:27:02.306987+0000) 2024-03-04T18:27:03.305 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:27:03.306+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:27:03.307299+0000) 2024-03-04T18:27:04.306 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:27:04.306+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:27:04.307591+0000) 2024-03-04T18:27:05.306 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:27:05.307+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:27:05.307955+0000) 2024-03-04T18:27:06.306 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:27:06.307+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:27:06.308201+0000) 2024-03-04T18:27:07.307 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:27:07.307+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:27:07.308437+0000) 2024-03-04T18:27:08.307 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:27:08.307+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:27:08.308739+0000) 2024-03-04T18:27:09.307 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:27:09.308+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:27:09.309031+0000) 2024-03-04T18:27:10.308 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:27:10.308+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:27:10.309319+0000) 2024-03-04T18:27:11.308 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:27:11.308+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:27:11.309617+0000) 2024-03-04T18:27:12.308 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:27:12.309+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:27:12.309968+0000) 2024-03-04T18:27:13.308 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:27:13.309+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:27:13.310268+0000) 2024-03-04T18:27:14.309 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:27:14.309+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:27:14.310549+0000) 2024-03-04T18:27:15.309 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:27:15.310+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:27:15.310876+0000) 2024-03-04T18:27:16.311 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:27:16.310+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:27:16.311173+0000) 2024-03-04T18:27:17.310 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:27:17.310+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:27:17.311482+0000) 2024-03-04T18:27:18.310 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:27:18.310+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:27:18.311797+0000) 2024-03-04T18:27:19.310 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:27:19.311+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:27:19.312080+0000) 2024-03-04T18:27:20.310 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:27:20.311+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:27:20.312316+0000) 2024-03-04T18:27:21.311 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:27:21.311+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:27:21.312612+0000) 2024-03-04T18:27:22.311 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:27:22.312+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:27:22.312940+0000) 2024-03-04T18:27:23.311 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:27:23.312+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:27:23.313243+0000) 2024-03-04T18:27:24.312 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:27:24.313+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:27:24.313577+0000) 2024-03-04T18:27:25.312 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:27:25.313+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:27:25.313911+0000) 2024-03-04T18:27:26.312 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:27:26.313+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:27:26.314207+0000) 2024-03-04T18:27:27.313 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:27:27.313+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:27:27.314486+0000) 2024-03-04T18:27:28.313 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:27:28.314+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:27:28.314775+0000) 2024-03-04T18:27:29.313 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:27:29.314+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:27:29.315077+0000) 2024-03-04T18:27:30.314 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:27:30.314+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:27:30.315320+0000) 2024-03-04T18:27:30.444 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:27:30.452 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:27:31.314 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:27:31.314+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:27:31.315634+0000) 2024-03-04T18:27:32.314 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:27:32.315+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:27:32.315911+0000) 2024-03-04T18:27:33.314 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:27:33.315+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:27:33.316221+0000) 2024-03-04T18:27:34.315 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:27:34.316+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:27:34.316519+0000) 2024-03-04T18:27:35.315 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:27:35.316+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:27:35.316867+0000) 2024-03-04T18:27:36.315 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:27:36.316+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:27:36.317179+0000) 2024-03-04T18:27:37.316 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:27:37.317+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:27:37.317482+0000) 2024-03-04T18:27:38.316 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:27:38.317+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:27:38.317775+0000) 2024-03-04T18:27:39.316 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:27:39.317+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:27:39.318070+0000) 2024-03-04T18:27:40.317 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:27:40.317+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:27:40.318371+0000) 2024-03-04T18:27:41.317 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:27:41.318+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:27:41.318671+0000) 2024-03-04T18:27:42.317 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:27:42.318+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:27:42.318973+0000) 2024-03-04T18:27:43.317 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:27:43.319+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:27:43.319284+0000) 2024-03-04T18:27:44.318 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:27:44.319+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:27:44.319568+0000) 2024-03-04T18:27:45.318 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:27:45.319+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:27:45.319891+0000) 2024-03-04T18:27:46.319 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:27:46.319+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:27:46.320178+0000) 2024-03-04T18:27:47.319 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:27:47.320+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:27:47.320471+0000) 2024-03-04T18:27:48.319 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:27:48.320+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:27:48.320766+0000) 2024-03-04T18:27:49.319 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:27:49.319+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:27:49.321067+0000) 2024-03-04T18:27:50.320 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:27:50.319+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:27:50.321361+0000) 2024-03-04T18:27:51.320 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:27:51.320+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:27:51.321676+0000) 2024-03-04T18:27:52.320 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:27:52.320+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:27:52.321976+0000) 2024-03-04T18:27:53.321 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:27:53.321+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:27:53.322308+0000) 2024-03-04T18:27:54.321 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:27:54.320+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:27:54.322531+0000) 2024-03-04T18:27:55.321 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:27:55.321+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:27:55.322866+0000) 2024-03-04T18:27:56.321 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:27:56.322+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:27:56.323173+0000) 2024-03-04T18:27:57.322 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:27:57.322+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:27:57.323466+0000) 2024-03-04T18:27:58.322 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:27:58.321+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:27:58.323779+0000) 2024-03-04T18:27:59.322 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:27:59.323+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:27:59.324107+0000) 2024-03-04T18:28:00.323 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:28:00.323+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:28:00.324393+0000) 2024-03-04T18:28:00.649 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:28:00.656 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:28:01.323 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:28:01.323+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:28:01.324681+0000) 2024-03-04T18:28:02.323 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:28:02.324+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:28:02.325000+0000) 2024-03-04T18:28:03.324 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:28:03.324+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:28:03.325310+0000) 2024-03-04T18:28:04.324 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:28:04.324+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:28:04.325605+0000) 2024-03-04T18:28:05.324 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:28:05.325+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:28:05.325946+0000) 2024-03-04T18:28:06.324 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:28:06.325+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:28:06.326247+0000) 2024-03-04T18:28:07.327 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:28:07.325+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:28:07.326535+0000) 2024-03-04T18:28:08.325 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:28:08.326+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:28:08.326906+0000) 2024-03-04T18:28:09.325 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:28:09.326+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:28:09.327208+0000) 2024-03-04T18:28:10.326 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:28:10.326+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:28:10.327522+0000) 2024-03-04T18:28:11.326 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:28:11.327+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:28:11.327815+0000) 2024-03-04T18:28:12.326 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:28:12.327+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:28:12.328168+0000) 2024-03-04T18:28:13.327 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:28:13.327+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:28:13.328501+0000) 2024-03-04T18:28:14.328 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:28:14.328+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:28:14.328791+0000) 2024-03-04T18:28:15.327 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:28:15.328+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:28:15.329141+0000) 2024-03-04T18:28:16.328 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:28:16.328+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:28:16.329478+0000) 2024-03-04T18:28:17.328 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:28:17.329+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:28:17.329768+0000) 2024-03-04T18:28:18.328 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:28:18.329+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:28:18.330055+0000) 2024-03-04T18:28:19.330 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:28:19.329+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:28:19.330347+0000) 2024-03-04T18:28:20.329 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:28:20.330+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:28:20.330626+0000) 2024-03-04T18:28:21.329 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:28:21.330+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:28:21.330940+0000) 2024-03-04T18:28:22.329 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:28:22.330+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:28:22.331171+0000) 2024-03-04T18:28:23.330 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:28:23.330+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:28:23.331443+0000) 2024-03-04T18:28:24.330 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:28:24.331+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:28:24.331685+0000) 2024-03-04T18:28:25.330 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:28:25.331+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:28:25.331945+0000) 2024-03-04T18:28:26.330 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:28:26.331+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:28:26.332195+0000) 2024-03-04T18:28:27.331 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:28:27.331+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:28:27.332443+0000) 2024-03-04T18:28:28.331 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:28:28.332+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:28:28.332716+0000) 2024-03-04T18:28:29.331 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:28:29.332+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:28:29.332951+0000) 2024-03-04T18:28:30.331 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:28:30.332+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:28:30.333180+0000) 2024-03-04T18:28:30.861 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:28:30.870 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:28:31.331 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:28:31.332+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:28:31.333434+0000) 2024-03-04T18:28:32.332 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:28:32.333+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:28:32.333677+0000) 2024-03-04T18:28:33.332 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:28:33.333+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:28:33.333933+0000) 2024-03-04T18:28:34.332 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:28:34.333+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:28:34.334183+0000) 2024-03-04T18:28:35.333 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:28:35.334+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:28:35.334452+0000) 2024-03-04T18:28:36.333 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:28:36.333+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:28:36.334653+0000) 2024-03-04T18:28:37.333 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:28:37.334+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:28:37.334933+0000) 2024-03-04T18:28:38.333 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:28:38.334+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:28:38.335178+0000) 2024-03-04T18:28:39.334 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:28:39.335+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:28:39.335425+0000) 2024-03-04T18:28:40.334 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:28:40.334+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:28:40.335701+0000) 2024-03-04T18:28:41.334 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:28:41.335+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:28:41.335945+0000) 2024-03-04T18:28:42.334 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:28:42.335+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:28:42.336197+0000) 2024-03-04T18:28:43.335 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:28:43.335+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:28:43.336470+0000) 2024-03-04T18:28:44.335 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:28:44.336+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:28:44.336712+0000) 2024-03-04T18:28:45.335 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:28:45.336+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:28:45.336960+0000) 2024-03-04T18:28:46.335 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:28:46.337+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:28:46.337213+0000) 2024-03-04T18:28:47.336 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:28:47.337+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:28:47.337458+0000) 2024-03-04T18:28:48.336 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:28:48.337+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:28:48.337707+0000) 2024-03-04T18:28:49.336 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:28:49.337+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:28:49.337943+0000) 2024-03-04T18:28:50.336 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:28:50.338+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:28:50.338214+0000) 2024-03-04T18:28:51.337 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:28:51.337+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:28:51.338489+0000) 2024-03-04T18:28:52.337 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:28:52.337+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:28:52.338734+0000) 2024-03-04T18:28:53.337 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:28:53.337+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:28:53.338988+0000) 2024-03-04T18:28:54.337 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:28:54.338+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:28:54.339229+0000) 2024-03-04T18:28:55.338 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:28:55.338+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:28:55.339479+0000) 2024-03-04T18:28:56.338 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:28:56.338+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:28:56.339737+0000) 2024-03-04T18:28:57.338 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:28:57.338+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:28:57.339983+0000) 2024-03-04T18:28:58.339 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:28:58.339+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:28:58.340234+0000) 2024-03-04T18:28:59.339 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:28:59.339+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:28:59.340474+0000) 2024-03-04T18:29:00.339 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:29:00.339+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:29:00.340718+0000) 2024-03-04T18:29:01.050 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:29:01.058 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:29:01.339 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:29:01.340+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:29:01.340991+0000) 2024-03-04T18:29:02.340 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:29:02.340+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:29:02.341238+0000) 2024-03-04T18:29:03.340 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:29:03.340+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:29:03.341512+0000) 2024-03-04T18:29:04.340 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:29:04.340+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:29:04.341756+0000) 2024-03-04T18:29:05.340 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:29:05.341+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:29:05.342026+0000) 2024-03-04T18:29:06.341 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:29:06.341+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:29:06.342278+0000) 2024-03-04T18:29:07.341 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:29:07.341+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:29:07.342523+0000) 2024-03-04T18:29:08.341 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:29:08.341+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:29:08.342794+0000) 2024-03-04T18:29:09.341 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:29:09.342+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:29:09.343093+0000) 2024-03-04T18:29:10.342 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:29:10.342+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:29:10.343412+0000) 2024-03-04T18:29:11.342 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:29:11.342+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:29:11.343682+0000) 2024-03-04T18:29:12.342 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:29:12.343+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:29:12.343977+0000) 2024-03-04T18:29:13.343 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:29:13.343+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:29:13.344300+0000) 2024-03-04T18:29:14.343 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:29:14.343+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:29:14.344594+0000) 2024-03-04T18:29:15.343 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:29:15.344+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:29:15.344900+0000) 2024-03-04T18:29:16.343 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:29:16.344+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:29:16.345151+0000) 2024-03-04T18:29:17.344 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:29:17.344+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:29:17.345397+0000) 2024-03-04T18:29:18.344 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:29:18.344+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:29:18.345589+0000) 2024-03-04T18:29:19.344 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:29:19.345+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:29:19.345883+0000) 2024-03-04T18:29:20.344 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:29:20.345+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:29:20.346157+0000) 2024-03-04T18:29:21.345 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:29:21.345+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:29:21.346397+0000) 2024-03-04T18:29:22.345 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:29:22.345+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:29:22.346595+0000) 2024-03-04T18:29:23.345 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:29:23.346+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:29:23.346891+0000) 2024-03-04T18:29:24.345 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:29:24.346+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:29:24.347132+0000) 2024-03-04T18:29:25.346 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:29:25.346+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:29:25.347411+0000) 2024-03-04T18:29:26.346 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:29:26.347+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:29:26.347664+0000) 2024-03-04T18:29:27.346 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:29:27.347+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:29:27.347957+0000) 2024-03-04T18:29:28.346 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:29:28.347+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:29:28.348197+0000) 2024-03-04T18:29:29.347 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:29:29.347+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:29:29.348441+0000) 2024-03-04T18:29:30.347 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:29:30.347+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:29:30.348708+0000) 2024-03-04T18:29:31.246 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:29:31.255 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:29:31.347 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:29:31.348+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:29:31.348930+0000) 2024-03-04T18:29:32.347 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:29:32.348+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:29:32.349170+0000) 2024-03-04T18:29:33.348 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:29:33.349+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:29:33.349439+0000) 2024-03-04T18:29:34.348 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:29:34.349+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:29:34.349679+0000) 2024-03-04T18:29:35.348 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:29:35.349+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:29:35.349951+0000) 2024-03-04T18:29:36.348 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:29:36.349+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:29:36.350214+0000) 2024-03-04T18:29:37.349 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:29:37.350+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:29:37.350457+0000) 2024-03-04T18:29:38.349 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:29:38.349+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:29:38.350722+0000) 2024-03-04T18:29:39.349 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:29:39.350+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:29:39.350961+0000) 2024-03-04T18:29:40.349 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:29:40.350+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:29:40.351199+0000) 2024-03-04T18:29:41.350 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:29:41.351+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:29:41.351452+0000) 2024-03-04T18:29:42.350 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:29:42.350+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:29:42.351714+0000) 2024-03-04T18:29:43.350 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:29:43.351+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:29:43.351962+0000) 2024-03-04T18:29:44.351 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:29:44.351+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:29:44.352200+0000) 2024-03-04T18:29:45.351 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:29:45.352+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:29:45.352435+0000) 2024-03-04T18:29:46.351 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:29:46.351+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:29:46.352698+0000) 2024-03-04T18:29:47.351 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:29:47.352+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:29:47.352940+0000) 2024-03-04T18:29:48.351 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:29:48.352+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:29:48.353181+0000) 2024-03-04T18:29:49.352 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:29:49.353+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:29:49.353438+0000) 2024-03-04T18:29:50.352 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:29:50.353+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:29:50.353672+0000) 2024-03-04T18:29:51.352 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:29:51.353+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:29:51.353969+0000) 2024-03-04T18:29:52.352 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:29:52.354+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:29:52.354258+0000) 2024-03-04T18:29:53.353 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:29:53.353+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:29:53.354510+0000) 2024-03-04T18:29:54.353 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:29:54.353+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:29:54.354759+0000) 2024-03-04T18:29:55.353 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:29:55.353+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:29:55.354995+0000) 2024-03-04T18:29:56.354 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:29:56.354+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:29:56.355248+0000) 2024-03-04T18:29:57.354 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:29:57.354+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:29:57.355492+0000) 2024-03-04T18:29:58.354 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:29:58.354+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:29:58.355737+0000) 2024-03-04T18:29:59.354 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:29:59.354+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:29:59.355944+0000) 2024-03-04T18:30:00.354 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:30:00.355+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:30:00.356182+0000) 2024-03-04T18:30:01.355 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:30:01.355+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:30:01.356435+0000) 2024-03-04T18:30:01.460 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:30:01.468 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:30:02.355 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:30:02.355+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:30:02.356680+0000) 2024-03-04T18:30:03.355 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:30:03.355+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:30:03.356953+0000) 2024-03-04T18:30:04.356 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:30:04.356+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:30:04.357191+0000) 2024-03-04T18:30:05.356 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:30:05.356+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:30:05.357431+0000) 2024-03-04T18:30:06.356 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:30:06.356+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:30:06.357691+0000) 2024-03-04T18:30:07.356 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:30:07.357+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:30:07.357937+0000) 2024-03-04T18:30:08.356 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:30:08.357+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:30:08.358191+0000) 2024-03-04T18:30:09.357 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:30:09.357+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:30:09.358438+0000) 2024-03-04T18:30:10.357 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:30:10.357+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:30:10.358704+0000) 2024-03-04T18:30:11.357 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:30:11.358+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:30:11.358956+0000) 2024-03-04T18:30:12.357 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:30:12.358+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:30:12.359194+0000) 2024-03-04T18:30:13.358 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:30:13.358+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:30:13.359454+0000) 2024-03-04T18:30:14.358 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:30:14.358+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:30:14.359696+0000) 2024-03-04T18:30:15.358 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:30:15.359+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:30:15.359937+0000) 2024-03-04T18:30:16.359 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:30:16.359+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:30:16.360182+0000) 2024-03-04T18:30:17.359 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:30:17.359+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:30:17.360424+0000) 2024-03-04T18:30:18.359 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:30:18.359+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:30:18.360673+0000) 2024-03-04T18:30:19.359 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:30:19.360+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:30:19.360942+0000) 2024-03-04T18:30:20.359 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:30:20.360+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:30:20.361203+0000) 2024-03-04T18:30:21.360 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:30:21.360+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:30:21.361450+0000) 2024-03-04T18:30:22.360 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:30:22.361+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:30:22.361699+0000) 2024-03-04T18:30:23.360 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:30:23.361+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:30:23.361968+0000) 2024-03-04T18:30:24.360 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:30:24.361+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:30:24.362210+0000) 2024-03-04T18:30:25.361 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:30:25.361+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:30:25.362448+0000) 2024-03-04T18:30:26.361 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:30:26.362+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:30:26.362695+0000) 2024-03-04T18:30:27.361 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:30:27.362+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:30:27.362936+0000) 2024-03-04T18:30:28.361 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:30:28.362+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:30:28.363175+0000) 2024-03-04T18:30:29.362 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:30:29.362+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:30:29.363416+0000) 2024-03-04T18:30:30.362 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:30:30.363+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:30:30.363658+0000) 2024-03-04T18:30:31.362 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:30:31.363+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:30:31.363961+0000) 2024-03-04T18:30:31.657 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:30:31.665 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:30:32.362 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:30:32.363+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:30:32.364160+0000) 2024-03-04T18:30:33.363 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:30:33.363+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:30:33.364360+0000) 2024-03-04T18:30:34.363 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:30:34.364+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:30:34.364605+0000) 2024-03-04T18:30:35.363 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:30:35.364+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:30:35.364875+0000) 2024-03-04T18:30:36.363 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:30:36.364+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:30:36.365161+0000) 2024-03-04T18:30:37.364 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:30:37.364+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:30:37.365410+0000) 2024-03-04T18:30:38.364 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:30:38.365+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:30:38.365657+0000) 2024-03-04T18:30:39.364 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:30:39.365+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:30:39.365944+0000) 2024-03-04T18:30:40.364 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:30:40.365+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:30:40.366258+0000) 2024-03-04T18:30:41.365 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:30:41.366+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:30:41.366502+0000) 2024-03-04T18:30:42.365 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:30:42.366+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:30:42.366747+0000) 2024-03-04T18:30:43.365 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:30:43.366+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:30:43.367007+0000) 2024-03-04T18:30:44.365 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:30:44.366+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:30:44.367273+0000) 2024-03-04T18:30:45.366 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:30:45.366+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:30:45.367526+0000) 2024-03-04T18:30:46.366 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:30:46.367+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:30:46.367779+0000) 2024-03-04T18:30:47.366 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:30:47.367+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:30:47.368019+0000) 2024-03-04T18:30:48.366 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:30:48.368+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:30:48.368263+0000) 2024-03-04T18:30:49.367 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:30:49.368+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:30:49.368524+0000) 2024-03-04T18:30:50.367 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:30:50.368+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:30:50.368806+0000) 2024-03-04T18:30:51.367 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:30:51.368+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:30:51.369173+0000) 2024-03-04T18:30:52.368 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:30:52.369+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:30:52.369470+0000) 2024-03-04T18:30:53.368 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:30:53.369+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:30:53.369724+0000) 2024-03-04T18:30:54.368 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:30:54.369+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:30:54.369971+0000) 2024-03-04T18:30:55.368 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:30:55.369+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:30:55.370210+0000) 2024-03-04T18:30:56.369 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:30:56.369+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:30:56.370459+0000) 2024-03-04T18:30:57.369 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:30:57.369+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:30:57.370703+0000) 2024-03-04T18:30:58.369 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:30:58.369+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:30:58.370948+0000) 2024-03-04T18:30:59.370 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:30:59.369+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:30:59.371162+0000) 2024-03-04T18:31:00.370 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:31:00.369+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:31:00.371376+0000) 2024-03-04T18:31:01.370 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:31:01.370+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:31:01.371633+0000) 2024-03-04T18:31:01.852 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:31:01.860 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:31:02.370 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:31:02.370+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:31:02.371940+0000) 2024-03-04T18:31:03.370 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:31:03.371+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:31:03.372209+0000) 2024-03-04T18:31:04.371 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:31:04.371+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:31:04.372461+0000) 2024-03-04T18:31:05.371 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:31:05.371+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:31:05.372747+0000) 2024-03-04T18:31:06.371 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:31:06.372+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:31:06.373009+0000) 2024-03-04T18:31:07.372 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:31:07.372+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:31:07.373334+0000) 2024-03-04T18:31:08.372 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:31:08.372+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:31:08.373637+0000) 2024-03-04T18:31:09.372 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:31:09.373+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:31:09.373984+0000) 2024-03-04T18:31:10.372 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:31:10.373+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:31:10.374225+0000) 2024-03-04T18:31:11.373 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:31:11.373+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:31:11.374457+0000) 2024-03-04T18:31:12.373 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:31:12.373+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:31:12.374738+0000) 2024-03-04T18:31:13.373 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:31:13.374+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:31:13.374993+0000) 2024-03-04T18:31:14.373 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:31:14.374+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:31:14.375233+0000) 2024-03-04T18:31:15.374 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:31:15.374+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:31:15.375463+0000) 2024-03-04T18:31:16.374 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:31:16.374+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:31:16.375718+0000) 2024-03-04T18:31:17.374 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:31:17.375+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:31:17.375978+0000) 2024-03-04T18:31:18.374 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:31:18.375+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:31:18.376201+0000) 2024-03-04T18:31:19.375 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:31:19.375+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:31:19.376458+0000) 2024-03-04T18:31:20.375 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:31:20.375+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:31:20.376700+0000) 2024-03-04T18:31:21.375 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:31:21.376+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:31:21.376939+0000) 2024-03-04T18:31:22.375 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:31:22.376+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:31:22.377178+0000) 2024-03-04T18:31:23.376 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:31:23.376+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:31:23.377423+0000) 2024-03-04T18:31:24.376 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:31:24.376+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:31:24.377621+0000) 2024-03-04T18:31:25.376 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:31:25.377+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:31:25.377905+0000) 2024-03-04T18:31:26.376 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:31:26.377+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:31:26.378145+0000) 2024-03-04T18:31:27.377 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:31:27.377+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:31:27.378400+0000) 2024-03-04T18:31:28.377 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:31:28.377+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:31:28.378658+0000) 2024-03-04T18:31:29.377 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:31:29.378+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:31:29.378955+0000) 2024-03-04T18:31:30.377 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:31:30.378+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:31:30.379192+0000) 2024-03-04T18:31:31.378 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:31:31.378+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:31:31.379413+0000) 2024-03-04T18:31:32.049 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:31:32.057 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:31:32.378 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:31:32.379+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:31:32.379693+0000) 2024-03-04T18:31:33.379 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:31:33.379+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:31:33.379949+0000) 2024-03-04T18:31:34.378 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:31:34.379+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:31:34.380143+0000) 2024-03-04T18:31:35.379 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:31:35.379+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:31:35.380379+0000) 2024-03-04T18:31:36.379 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:31:36.380+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:31:36.380624+0000) 2024-03-04T18:31:37.379 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:31:37.380+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:31:37.380905+0000) 2024-03-04T18:31:38.379 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:31:38.380+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:31:38.381151+0000) 2024-03-04T18:31:39.381 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:31:39.380+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:31:39.381499+0000) 2024-03-04T18:31:40.380 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:31:40.381+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:31:40.381755+0000) 2024-03-04T18:31:41.380 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:31:41.381+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:31:41.381998+0000) 2024-03-04T18:31:42.380 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:31:42.381+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:31:42.382250+0000) 2024-03-04T18:31:43.381 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:31:43.382+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:31:43.382501+0000) 2024-03-04T18:31:44.381 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:31:44.382+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:31:44.382748+0000) 2024-03-04T18:31:45.381 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:31:45.382+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:31:45.382990+0000) 2024-03-04T18:31:46.381 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:31:46.382+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:31:46.383239+0000) 2024-03-04T18:31:47.382 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:31:47.382+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:31:47.383428+0000) 2024-03-04T18:31:48.382 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:31:48.383+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:31:48.383708+0000) 2024-03-04T18:31:49.382 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:31:49.383+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:31:49.383976+0000) 2024-03-04T18:31:50.382 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:31:50.383+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:31:50.384222+0000) 2024-03-04T18:31:51.383 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:31:51.384+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:31:51.384459+0000) 2024-03-04T18:31:52.383 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:31:52.384+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:31:52.384704+0000) 2024-03-04T18:31:53.383 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:31:53.384+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:31:53.384956+0000) 2024-03-04T18:31:54.384 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:31:54.385+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:31:54.385210+0000) 2024-03-04T18:31:55.384 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:31:55.384+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:31:55.385402+0000) 2024-03-04T18:31:56.384 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:31:56.385+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:31:56.385652+0000) 2024-03-04T18:31:57.384 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:31:57.384+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:31:57.385949+0000) 2024-03-04T18:31:58.384 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:31:58.385+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:31:58.386200+0000) 2024-03-04T18:31:59.385 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:31:59.385+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:31:59.386438+0000) 2024-03-04T18:32:00.385 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:32:00.385+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:32:00.386674+0000) 2024-03-04T18:32:01.385 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:32:01.385+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:32:01.386944+0000) 2024-03-04T18:32:02.263 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:32:02.275 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:32:02.412 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:32:02.385+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:32:02.387110+0000) 2024-03-04T18:32:03.386 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:32:03.385+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:32:03.387400+0000) 2024-03-04T18:32:04.386 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:32:04.385+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:32:04.387655+0000) 2024-03-04T18:32:05.386 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:32:05.386+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:32:05.387927+0000) 2024-03-04T18:32:06.387 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:32:06.387+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:32:06.388175+0000) 2024-03-04T18:32:07.387 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:32:07.387+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:32:07.388410+0000) 2024-03-04T18:32:08.387 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:32:08.387+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:32:08.388599+0000) 2024-03-04T18:32:09.387 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:32:09.388+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:32:09.388871+0000) 2024-03-04T18:32:10.387 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:32:10.388+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:32:10.389119+0000) 2024-03-04T18:32:11.388 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:32:11.388+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:32:11.389370+0000) 2024-03-04T18:32:12.388 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:32:12.388+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:32:12.389620+0000) 2024-03-04T18:32:13.388 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:32:13.389+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:32:13.389907+0000) 2024-03-04T18:32:14.388 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:32:14.389+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:32:14.390146+0000) 2024-03-04T18:32:15.389 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:32:15.389+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:32:15.390331+0000) 2024-03-04T18:32:16.389 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:32:16.389+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:32:16.390580+0000) 2024-03-04T18:32:17.389 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:32:17.390+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:32:17.390854+0000) 2024-03-04T18:32:18.389 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:32:18.390+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:32:18.391144+0000) 2024-03-04T18:32:19.390 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:32:19.390+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:32:19.391343+0000) 2024-03-04T18:32:20.390 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:32:20.390+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:32:20.391583+0000) 2024-03-04T18:32:21.390 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:32:21.391+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:32:21.391852+0000) 2024-03-04T18:32:22.391 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:32:22.391+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:32:22.392119+0000) 2024-03-04T18:32:23.391 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:32:23.391+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:32:23.392319+0000) 2024-03-04T18:32:24.391 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:32:24.391+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:32:24.392554+0000) 2024-03-04T18:32:25.391 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:32:25.392+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:32:25.392801+0000) 2024-03-04T18:32:26.391 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:32:26.392+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:32:26.393116+0000) 2024-03-04T18:32:27.392 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:32:27.392+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:32:27.393367+0000) 2024-03-04T18:32:28.392 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:32:28.392+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:32:28.393594+0000) 2024-03-04T18:32:29.392 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:32:29.393+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:32:29.393864+0000) 2024-03-04T18:32:30.392 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:32:30.393+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:32:30.394116+0000) 2024-03-04T18:32:31.393 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:32:31.393+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:32:31.394320+0000) 2024-03-04T18:32:32.393 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:32:32.394+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:32:32.394564+0000) 2024-03-04T18:32:32.477 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:32:32.486 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:32:33.393 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:32:33.394+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:32:33.394813+0000) 2024-03-04T18:32:34.393 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:32:34.394+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:32:34.395105+0000) 2024-03-04T18:32:35.394 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:32:35.394+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:32:35.395348+0000) 2024-03-04T18:32:36.394 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:32:36.395+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:32:36.395597+0000) 2024-03-04T18:32:37.394 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:32:37.395+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:32:37.395875+0000) 2024-03-04T18:32:38.394 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:32:38.395+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:32:38.396119+0000) 2024-03-04T18:32:39.394 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:32:39.395+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:32:39.396355+0000) 2024-03-04T18:32:40.395 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:32:40.396+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:32:40.396592+0000) 2024-03-04T18:32:41.395 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:32:41.396+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:32:41.396861+0000) 2024-03-04T18:32:42.395 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:32:42.396+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:32:42.397105+0000) 2024-03-04T18:32:43.396 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:32:43.396+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:32:43.397361+0000) 2024-03-04T18:32:44.396 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:32:44.397+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:32:44.397607+0000) 2024-03-04T18:32:45.396 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:32:45.397+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:32:45.397883+0000) 2024-03-04T18:32:46.396 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:32:46.397+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:32:46.398126+0000) 2024-03-04T18:32:47.397 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:32:47.398+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:32:47.398373+0000) 2024-03-04T18:32:48.397 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:32:48.397+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:32:48.398562+0000) 2024-03-04T18:32:49.397 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:32:49.398+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:32:49.398853+0000) 2024-03-04T18:32:50.397 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:32:50.398+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:32:50.399051+0000) 2024-03-04T18:32:51.398 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:32:51.398+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:32:51.399298+0000) 2024-03-04T18:32:52.398 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:32:52.398+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:32:52.399484+0000) 2024-03-04T18:32:53.398 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:32:53.399+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:32:53.399735+0000) 2024-03-04T18:32:54.398 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:32:54.399+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:32:54.399983+0000) 2024-03-04T18:32:55.398 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:32:55.399+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:32:55.400209+0000) 2024-03-04T18:32:56.399 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:32:56.400+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:32:56.400453+0000) 2024-03-04T18:32:57.399 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:32:57.400+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:32:57.400694+0000) 2024-03-04T18:32:58.399 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:32:58.400+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:32:58.400936+0000) 2024-03-04T18:32:59.399 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:32:59.399+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:32:59.401173+0000) 2024-03-04T18:33:00.400 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:33:00.399+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:33:00.401427+0000) 2024-03-04T18:33:01.400 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:33:01.400+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:33:01.401630+0000) 2024-03-04T18:33:02.400 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:33:02.400+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:33:02.401923+0000) 2024-03-04T18:33:02.624 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:33:02.631 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:33:03.400 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:33:03.401+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:33:03.402199+0000) 2024-03-04T18:33:04.401 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:33:04.401+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:33:04.402443+0000) 2024-03-04T18:33:05.401 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:33:05.401+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:33:05.402681+0000) 2024-03-04T18:33:06.401 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:33:06.401+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:33:06.402958+0000) 2024-03-04T18:33:07.402 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:33:07.402+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:33:07.403212+0000) 2024-03-04T18:33:08.402 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:33:08.402+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:33:08.403459+0000) 2024-03-04T18:33:09.402 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:33:09.402+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:33:09.403702+0000) 2024-03-04T18:33:10.402 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:33:10.403+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:33:10.403991+0000) 2024-03-04T18:33:11.403 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:33:11.403+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:33:11.404240+0000) 2024-03-04T18:33:12.403 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:33:12.403+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:33:12.404487+0000) 2024-03-04T18:33:13.403 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:33:13.403+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:33:13.404764+0000) 2024-03-04T18:33:14.404 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:33:14.404+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:33:14.405009+0000) 2024-03-04T18:33:15.404 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:33:15.404+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:33:15.405259+0000) 2024-03-04T18:33:16.404 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:33:16.404+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:33:16.405512+0000) 2024-03-04T18:33:17.404 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:33:17.404+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:33:17.405701+0000) 2024-03-04T18:33:18.404 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:33:18.405+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:33:18.405949+0000) 2024-03-04T18:33:19.405 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:33:19.405+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:33:19.406209+0000) 2024-03-04T18:33:20.405 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:33:20.405+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:33:20.406475+0000) 2024-03-04T18:33:21.405 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:33:21.405+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:33:21.406746+0000) 2024-03-04T18:33:22.405 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:33:22.406+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:33:22.406994+0000) 2024-03-04T18:33:23.406 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:33:23.406+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:33:23.407254+0000) 2024-03-04T18:33:24.406 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:33:24.406+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:33:24.407493+0000) 2024-03-04T18:33:25.406 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:33:25.407+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:33:25.407729+0000) 2024-03-04T18:33:26.406 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:33:26.407+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:33:26.408017+0000) 2024-03-04T18:33:27.406 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:33:27.407+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:33:27.408264+0000) 2024-03-04T18:33:28.407 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:33:28.407+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:33:28.408508+0000) 2024-03-04T18:33:29.407 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:33:29.408+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:33:29.408749+0000) 2024-03-04T18:33:30.407 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:33:30.408+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:33:30.408989+0000) 2024-03-04T18:33:31.407 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:33:31.408+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:33:31.409181+0000) 2024-03-04T18:33:32.408 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:33:32.408+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:33:32.409426+0000) 2024-03-04T18:33:32.821 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:33:32.830 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:33:33.408 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:33:33.409+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:33:33.409687+0000) 2024-03-04T18:33:34.408 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:33:34.409+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:33:34.409933+0000) 2024-03-04T18:33:35.408 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:33:35.409+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:33:35.410174+0000) 2024-03-04T18:33:36.409 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:33:36.409+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:33:36.410421+0000) 2024-03-04T18:33:37.411 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:33:37.409+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:33:37.410687+0000) 2024-03-04T18:33:38.409 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:33:38.410+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:33:38.410968+0000) 2024-03-04T18:33:39.409 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:33:39.410+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:33:39.411214+0000) 2024-03-04T18:33:40.410 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:33:40.411+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:33:40.411459+0000) 2024-03-04T18:33:41.410 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:33:41.411+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:33:41.411705+0000) 2024-03-04T18:33:42.410 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:33:42.411+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:33:42.411961+0000) 2024-03-04T18:33:43.411 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:33:43.411+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:33:43.412220+0000) 2024-03-04T18:33:44.411 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:33:44.412+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:33:44.412460+0000) 2024-03-04T18:33:45.411 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:33:45.411+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:33:45.412715+0000) 2024-03-04T18:33:46.411 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:33:46.412+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:33:46.412969+0000) 2024-03-04T18:33:47.411 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:33:47.412+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:33:47.413224+0000) 2024-03-04T18:33:48.412 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:33:48.413+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:33:48.413469+0000) 2024-03-04T18:33:49.412 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:33:49.413+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:33:49.413725+0000) 2024-03-04T18:33:50.413 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:33:50.413+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:33:50.413943+0000) 2024-03-04T18:33:51.412 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:33:51.413+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:33:51.414196+0000) 2024-03-04T18:33:52.413 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:33:52.414+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:33:52.414447+0000) 2024-03-04T18:33:53.413 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:33:53.414+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:33:53.414703+0000) 2024-03-04T18:33:54.413 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:33:54.414+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:33:54.414948+0000) 2024-03-04T18:33:55.413 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:33:55.414+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:33:55.415153+0000) 2024-03-04T18:33:56.414 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:33:56.415+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:33:56.415400+0000) 2024-03-04T18:33:57.415 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:33:57.415+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:33:57.415644+0000) 2024-03-04T18:33:58.414 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:33:58.415+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:33:58.415940+0000) 2024-03-04T18:33:59.414 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:33:59.416+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:33:59.416178+0000) 2024-03-04T18:34:00.415 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:34:00.416+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:34:00.416414+0000) 2024-03-04T18:34:01.415 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:34:01.415+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:34:01.416662+0000) 2024-03-04T18:34:02.415 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:34:02.415+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:34:02.416947+0000) 2024-03-04T18:34:03.041 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:34:03.049 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:34:03.415 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:34:03.415+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:34:03.417270+0000) 2024-03-04T18:34:04.416 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:34:04.415+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:34:04.417462+0000) 2024-03-04T18:34:05.416 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:34:05.416+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:34:05.417697+0000) 2024-03-04T18:34:06.416 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:34:06.416+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:34:06.417940+0000) 2024-03-04T18:34:07.416 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:34:07.417+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:34:07.418186+0000) 2024-03-04T18:34:08.417 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:34:08.417+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:34:08.418433+0000) 2024-03-04T18:34:09.417 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:34:09.417+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:34:09.418683+0000) 2024-03-04T18:34:10.417 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:34:10.417+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:34:10.418946+0000) 2024-03-04T18:34:11.417 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:34:11.418+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:34:11.419192+0000) 2024-03-04T18:34:12.418 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:34:12.418+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:34:12.419435+0000) 2024-03-04T18:34:13.418 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:34:13.418+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:34:13.419692+0000) 2024-03-04T18:34:14.418 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:34:14.419+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:34:14.419930+0000) 2024-03-04T18:34:15.418 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:34:15.419+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:34:15.420162+0000) 2024-03-04T18:34:16.419 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:34:16.419+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:34:16.420406+0000) 2024-03-04T18:34:17.419 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:34:17.419+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:34:17.420592+0000) 2024-03-04T18:34:18.419 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:34:18.420+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:34:18.420868+0000) 2024-03-04T18:34:19.419 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:34:19.420+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:34:19.421096+0000) 2024-03-04T18:34:20.419 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:34:20.420+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:34:20.421344+0000) 2024-03-04T18:34:21.420 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:34:21.420+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:34:21.421606+0000) 2024-03-04T18:34:22.420 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:34:22.421+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:34:22.421883+0000) 2024-03-04T18:34:23.420 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:34:23.421+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:34:23.422153+0000) 2024-03-04T18:34:24.421 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:34:24.421+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:34:24.422393+0000) 2024-03-04T18:34:25.422 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:34:25.421+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:34:25.422627+0000) 2024-03-04T18:34:26.421 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:34:26.422+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:34:26.422935+0000) 2024-03-04T18:34:27.421 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:34:27.422+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:34:27.423157+0000) 2024-03-04T18:34:28.422 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:34:28.422+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:34:28.423464+0000) 2024-03-04T18:34:29.422 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:34:29.423+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:34:29.423715+0000) 2024-03-04T18:34:30.422 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:34:30.423+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:34:30.423937+0000) 2024-03-04T18:34:31.422 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:34:31.423+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:34:31.424181+0000) 2024-03-04T18:34:32.423 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:34:32.423+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:34:32.424470+0000) 2024-03-04T18:34:33.188 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:34:33.198 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:34:33.423 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:34:33.423+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:34:33.424760+0000) 2024-03-04T18:34:34.423 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:34:34.424+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:34:34.425006+0000) 2024-03-04T18:34:35.424 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:34:35.424+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:34:35.425248+0000) 2024-03-04T18:34:36.424 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:34:36.424+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:34:36.425483+0000) 2024-03-04T18:34:37.424 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:34:37.425+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:34:37.425767+0000) 2024-03-04T18:34:38.424 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:34:38.425+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:34:38.426013+0000) 2024-03-04T18:34:39.425 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:34:39.425+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:34:39.426259+0000) 2024-03-04T18:34:40.425 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:34:40.426+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:34:40.426505+0000) 2024-03-04T18:34:41.425 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:34:41.425+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:34:41.426740+0000) 2024-03-04T18:34:42.425 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:34:42.426+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:34:42.426979+0000) 2024-03-04T18:34:43.425 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:34:43.426+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:34:43.427239+0000) 2024-03-04T18:34:44.426 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:34:44.427+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:34:44.427480+0000) 2024-03-04T18:34:45.426 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:34:45.427+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:34:45.427720+0000) 2024-03-04T18:34:46.426 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:34:46.427+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:34:46.427969+0000) 2024-03-04T18:34:47.427 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:34:47.427+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:34:47.428234+0000) 2024-03-04T18:34:48.427 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:34:48.427+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:34:48.428437+0000) 2024-03-04T18:34:49.427 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:34:49.428+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:34:49.428677+0000) 2024-03-04T18:34:50.427 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:34:50.428+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:34:50.428973+0000) 2024-03-04T18:34:51.427 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:34:51.428+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:34:51.429189+0000) 2024-03-04T18:34:52.428 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:34:52.429+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:34:52.429435+0000) 2024-03-04T18:34:53.428 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:34:53.429+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:34:53.429690+0000) 2024-03-04T18:34:54.428 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:34:54.429+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:34:54.429952+0000) 2024-03-04T18:34:55.428 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:34:55.429+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:34:55.430162+0000) 2024-03-04T18:34:56.429 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:34:56.429+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:34:56.430360+0000) 2024-03-04T18:34:57.429 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:34:57.429+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:34:57.430546+0000) 2024-03-04T18:34:58.429 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:34:58.429+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:34:58.430731+0000) 2024-03-04T18:34:59.429 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:34:59.430+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:34:59.430958+0000) 2024-03-04T18:35:00.430 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:35:00.431+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:35:00.431206+0000) 2024-03-04T18:35:01.431 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:35:01.431+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:35:01.431445+0000) 2024-03-04T18:35:02.430 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:35:02.431+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:35:02.431701+0000) 2024-03-04T18:35:03.325 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:35:03.333 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:35:03.430 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:35:03.430+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:35:03.431966+0000) 2024-03-04T18:35:04.430 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:35:04.431+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:35:04.432212+0000) 2024-03-04T18:35:05.431 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:35:05.430+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:35:05.432467+0000) 2024-03-04T18:35:06.431 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:35:06.431+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:35:06.432720+0000) 2024-03-04T18:35:07.431 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:35:07.431+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:35:07.432992+0000) 2024-03-04T18:35:08.431 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:35:08.432+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:35:08.433231+0000) 2024-03-04T18:35:09.432 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:35:09.432+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:35:09.433483+0000) 2024-03-04T18:35:10.432 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:35:10.432+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:35:10.433724+0000) 2024-03-04T18:35:11.432 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:35:11.432+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:35:11.433968+0000) 2024-03-04T18:35:12.432 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:35:12.433+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:35:12.434208+0000) 2024-03-04T18:35:13.433 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:35:13.433+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:35:13.434498+0000) 2024-03-04T18:35:14.433 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:35:14.433+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:35:14.434796+0000) 2024-03-04T18:35:15.434 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:35:15.434+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:35:15.435132+0000) 2024-03-04T18:35:16.434 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:35:16.434+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:35:16.435426+0000) 2024-03-04T18:35:17.434 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:35:17.434+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:35:17.435724+0000) 2024-03-04T18:35:18.434 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:35:18.435+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:35:18.435984+0000) 2024-03-04T18:35:19.434 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:35:19.435+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:35:19.436224+0000) 2024-03-04T18:35:20.435 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:35:20.435+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:35:20.436481+0000) 2024-03-04T18:35:21.435 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:35:21.435+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:35:21.436736+0000) 2024-03-04T18:35:22.435 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:35:22.436+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:35:22.436979+0000) 2024-03-04T18:35:23.436 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:35:23.436+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:35:23.437249+0000) 2024-03-04T18:35:24.436 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:35:24.436+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:35:24.437504+0000) 2024-03-04T18:35:25.436 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:35:25.436+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:35:25.437747+0000) 2024-03-04T18:35:26.436 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:35:26.437+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:35:26.437983+0000) 2024-03-04T18:35:27.436 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:35:27.437+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:35:27.438242+0000) 2024-03-04T18:35:28.437 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:35:28.437+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:35:28.438453+0000) 2024-03-04T18:35:29.437 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:35:29.437+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:35:29.438699+0000) 2024-03-04T18:35:30.437 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:35:30.438+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:35:30.438948+0000) 2024-03-04T18:35:31.438 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:35:31.438+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:35:31.439225+0000) 2024-03-04T18:35:32.438 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:35:32.438+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:35:32.439489+0000) 2024-03-04T18:35:33.438 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:35:33.439+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:35:33.439741+0000) 2024-03-04T18:35:33.490 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:35:33.498 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:35:34.438 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:35:34.439+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:35:34.440016+0000) 2024-03-04T18:35:35.439 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:35:35.439+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:35:35.440314+0000) 2024-03-04T18:35:36.439 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:35:36.439+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:35:36.440604+0000) 2024-03-04T18:35:37.439 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:35:37.440+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:35:37.440921+0000) 2024-03-04T18:35:38.439 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:35:38.440+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:35:38.441211+0000) 2024-03-04T18:35:39.440 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:35:39.440+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:35:39.441506+0000) 2024-03-04T18:35:40.440 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:35:40.441+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:35:40.441807+0000) 2024-03-04T18:35:41.440 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:35:41.441+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:35:41.442142+0000) 2024-03-04T18:35:42.441 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:35:42.441+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:35:42.442419+0000) 2024-03-04T18:35:43.441 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:35:43.442+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:35:43.442726+0000) 2024-03-04T18:35:44.441 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:35:44.442+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:35:44.443013+0000) 2024-03-04T18:35:45.442 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:35:45.442+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:35:45.443303+0000) 2024-03-04T18:35:46.442 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:35:46.443+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:35:46.443584+0000) 2024-03-04T18:35:47.442 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:35:47.443+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:35:47.443929+0000) 2024-03-04T18:35:48.443 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:35:48.443+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:35:48.444210+0000) 2024-03-04T18:35:49.443 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:35:49.444+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:35:49.444497+0000) 2024-03-04T18:35:50.443 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:35:50.444+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:35:50.444924+0000) 2024-03-04T18:35:51.444 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:35:51.444+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:35:51.445216+0000) 2024-03-04T18:35:52.444 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:35:52.445+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:35:52.445507+0000) 2024-03-04T18:35:53.444 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:35:53.445+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:35:53.445808+0000) 2024-03-04T18:35:54.446 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:35:54.445+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:35:54.446142+0000) 2024-03-04T18:35:55.445 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:35:55.446+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:35:55.446424+0000) 2024-03-04T18:35:56.445 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:35:56.446+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:35:56.446716+0000) 2024-03-04T18:35:57.445 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:35:57.446+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:35:57.446995+0000) 2024-03-04T18:35:58.446 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:35:58.447+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:35:58.447313+0000) 2024-03-04T18:35:59.446 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:35:59.447+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:35:59.447591+0000) 2024-03-04T18:36:00.446 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:36:00.447+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:36:00.447925+0000) 2024-03-04T18:36:01.447 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:36:01.447+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:36:01.448216+0000) 2024-03-04T18:36:02.447 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:36:02.448+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:36:02.448470+0000) 2024-03-04T18:36:03.447 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:36:03.448+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:36:03.448723+0000) 2024-03-04T18:36:03.721 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:36:03.729 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:36:04.447 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:36:04.448+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:36:04.449011+0000) 2024-03-04T18:36:05.448 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:36:05.447+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:36:05.449251+0000) 2024-03-04T18:36:06.448 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:36:06.448+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:36:06.449494+0000) 2024-03-04T18:36:07.448 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:36:07.448+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:36:07.449738+0000) 2024-03-04T18:36:08.448 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:36:08.448+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:36:08.449984+0000) 2024-03-04T18:36:09.449 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:36:09.449+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:36:09.450288+0000) 2024-03-04T18:36:10.449 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:36:10.449+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:36:10.450592+0000) 2024-03-04T18:36:11.449 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:36:11.449+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:36:11.450939+0000) 2024-03-04T18:36:12.450 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:36:12.450+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:36:12.451238+0000) 2024-03-04T18:36:13.450 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:36:13.450+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:36:13.451563+0000) 2024-03-04T18:36:14.450 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:36:14.450+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:36:14.451925+0000) 2024-03-04T18:36:15.451 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:36:15.451+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:36:15.452221+0000) 2024-03-04T18:36:16.451 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:36:16.451+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:36:16.452517+0000) 2024-03-04T18:36:17.451 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:36:17.452+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:36:17.452811+0000) 2024-03-04T18:36:18.452 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:36:18.452+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:36:18.453150+0000) 2024-03-04T18:36:19.452 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:36:19.452+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:36:19.453397+0000) 2024-03-04T18:36:20.452 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:36:20.452+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:36:20.453695+0000) 2024-03-04T18:36:21.452 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:36:21.453+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:36:21.453985+0000) 2024-03-04T18:36:22.453 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:36:22.453+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:36:22.454282+0000) 2024-03-04T18:36:23.453 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:36:23.453+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:36:23.454591+0000) 2024-03-04T18:36:24.453 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:36:24.454+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:36:24.454927+0000) 2024-03-04T18:36:25.454 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:36:25.454+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:36:25.455220+0000) 2024-03-04T18:36:26.454 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:36:26.454+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:36:26.455519+0000) 2024-03-04T18:36:27.454 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:36:27.455+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:36:27.455872+0000) 2024-03-04T18:36:28.454 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:36:28.455+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:36:28.456170+0000) 2024-03-04T18:36:29.455 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:36:29.455+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:36:29.456490+0000) 2024-03-04T18:36:30.455 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:36:30.456+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:36:30.456780+0000) 2024-03-04T18:36:31.455 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:36:31.456+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:36:31.457111+0000) 2024-03-04T18:36:32.456 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:36:32.456+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:36:32.457406+0000) 2024-03-04T18:36:33.456 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:36:33.457+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:36:33.457718+0000) 2024-03-04T18:36:33.934 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:36:33.942 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:36:34.456 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:36:34.457+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:36:34.458017+0000) 2024-03-04T18:36:35.456 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:36:35.457+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:36:35.458272+0000) 2024-03-04T18:36:36.457 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:36:36.457+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:36:36.458567+0000) 2024-03-04T18:36:37.457 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:36:37.458+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:36:37.458894+0000) 2024-03-04T18:36:38.457 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:36:38.458+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:36:38.459225+0000) 2024-03-04T18:36:39.458 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:36:39.458+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:36:39.459539+0000) 2024-03-04T18:36:40.458 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:36:40.459+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:36:40.459894+0000) 2024-03-04T18:36:41.458 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:36:41.459+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:36:41.460194+0000) 2024-03-04T18:36:42.459 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:36:42.459+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:36:42.460485+0000) 2024-03-04T18:36:43.461 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:36:43.460+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:36:43.460790+0000) 2024-03-04T18:36:44.460 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:36:44.460+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:36:44.461136+0000) 2024-03-04T18:36:45.463 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:36:45.460+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:36:45.461436+0000) 2024-03-04T18:36:46.460 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:36:46.461+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:36:46.461712+0000) 2024-03-04T18:36:47.460 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:36:47.461+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:36:47.462033+0000) 2024-03-04T18:36:48.461 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:36:48.461+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:36:48.462320+0000) 2024-03-04T18:36:49.461 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:36:49.462+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:36:49.462609+0000) 2024-03-04T18:36:50.461 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:36:50.462+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:36:50.462940+0000) 2024-03-04T18:36:51.462 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:36:51.462+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:36:51.463234+0000) 2024-03-04T18:36:52.462 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:36:52.463+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:36:52.463524+0000) 2024-03-04T18:36:53.462 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:36:53.463+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:36:53.463803+0000) 2024-03-04T18:36:54.462 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:36:54.463+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:36:54.464154+0000) 2024-03-04T18:36:55.463 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:36:55.464+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:36:55.464443+0000) 2024-03-04T18:36:56.463 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:36:56.464+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:36:56.464735+0000) 2024-03-04T18:36:57.463 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:36:57.464+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:36:57.465026+0000) 2024-03-04T18:36:58.464 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:36:58.464+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:36:58.465253+0000) 2024-03-04T18:36:59.464 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:36:59.465+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:36:59.465513+0000) 2024-03-04T18:37:00.464 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:37:00.465+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:37:00.465784+0000) 2024-03-04T18:37:01.464 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:37:01.465+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:37:01.466057+0000) 2024-03-04T18:37:02.465 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:37:02.466+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:37:02.466343+0000) 2024-03-04T18:37:03.465 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:37:03.466+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:37:03.466650+0000) 2024-03-04T18:37:04.147 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:37:04.155 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:37:04.465 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:37:04.466+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:37:04.466993+0000) 2024-03-04T18:37:05.466 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:37:05.467+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:37:05.467281+0000) 2024-03-04T18:37:06.466 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:37:06.467+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:37:06.467581+0000) 2024-03-04T18:37:07.466 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:37:07.466+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:37:07.467916+0000) 2024-03-04T18:37:08.467 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:37:08.467+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:37:08.468248+0000) 2024-03-04T18:37:09.467 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:37:09.467+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:37:09.468544+0000) 2024-03-04T18:37:10.467 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:37:10.467+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:37:10.468877+0000) 2024-03-04T18:37:11.468 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:37:11.468+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:37:11.469169+0000) 2024-03-04T18:37:12.468 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:37:12.468+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:37:12.469443+0000) 2024-03-04T18:37:13.471 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:37:13.468+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:37:13.469696+0000) 2024-03-04T18:37:14.468 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:37:14.468+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:37:14.469943+0000) 2024-03-04T18:37:15.468 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:37:15.469+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:37:15.470216+0000) 2024-03-04T18:37:16.469 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:37:16.469+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:37:16.470498+0000) 2024-03-04T18:37:17.469 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:37:17.469+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:37:17.470792+0000) 2024-03-04T18:37:18.469 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:37:18.470+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:37:18.471124+0000) 2024-03-04T18:37:19.470 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:37:19.470+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:37:19.471418+0000) 2024-03-04T18:37:20.470 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:37:20.470+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:37:20.471758+0000) 2024-03-04T18:37:21.470 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:37:21.471+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:37:21.472053+0000) 2024-03-04T18:37:22.471 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:37:22.471+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:37:22.472293+0000) 2024-03-04T18:37:23.472 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:37:23.471+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:37:23.472604+0000) 2024-03-04T18:37:24.471 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:37:24.472+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:37:24.472964+0000) 2024-03-04T18:37:25.472 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:37:25.472+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:37:25.473261+0000) 2024-03-04T18:37:26.472 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:37:26.472+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:37:26.473591+0000) 2024-03-04T18:37:27.472 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:37:27.473+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:37:27.473865+0000) 2024-03-04T18:37:28.472 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:37:28.473+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:37:28.474164+0000) 2024-03-04T18:37:29.473 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:37:29.473+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:37:29.474401+0000) 2024-03-04T18:37:30.473 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:37:30.473+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:37:30.474719+0000) 2024-03-04T18:37:31.473 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:37:31.474+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:37:31.475022+0000) 2024-03-04T18:37:32.474 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:37:32.474+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:37:32.475315+0000) 2024-03-04T18:37:33.474 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:37:33.474+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:37:33.475627+0000) 2024-03-04T18:37:34.353 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:37:34.361 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:37:34.496 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:37:34.475+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:37:34.475909+0000) 2024-03-04T18:37:35.474 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:37:35.475+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:37:35.476204+0000) 2024-03-04T18:37:36.475 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:37:36.475+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:37:36.476500+0000) 2024-03-04T18:37:37.475 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:37:37.475+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:37:37.476795+0000) 2024-03-04T18:37:38.476 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:37:38.476+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:37:38.477127+0000) 2024-03-04T18:37:39.476 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:37:39.476+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:37:39.477429+0000) 2024-03-04T18:37:40.477 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:37:40.477+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:37:40.477720+0000) 2024-03-04T18:37:41.476 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:37:41.477+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:37:41.478042+0000) 2024-03-04T18:37:42.477 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:37:42.477+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:37:42.478344+0000) 2024-03-04T18:37:43.477 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:37:43.478+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:37:43.478656+0000) 2024-03-04T18:37:44.477 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:37:44.478+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:37:44.478969+0000) 2024-03-04T18:37:45.478 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:37:45.478+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:37:45.479279+0000) 2024-03-04T18:37:46.478 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:37:46.478+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:37:46.479557+0000) 2024-03-04T18:37:47.478 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:37:47.479+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:37:47.479896+0000) 2024-03-04T18:37:48.479 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:37:48.479+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:37:48.480200+0000) 2024-03-04T18:37:49.479 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:37:49.479+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:37:49.480468+0000) 2024-03-04T18:37:50.479 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:37:50.480+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:37:50.480748+0000) 2024-03-04T18:37:51.479 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:37:51.480+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:37:51.481040+0000) 2024-03-04T18:37:52.480 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:37:52.480+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:37:52.481335+0000) 2024-03-04T18:37:53.480 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:37:53.481+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:37:53.481645+0000) 2024-03-04T18:37:54.480 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:37:54.481+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:37:54.481921+0000) 2024-03-04T18:37:55.481 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:37:55.481+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:37:55.482235+0000) 2024-03-04T18:37:56.481 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:37:56.482+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:37:56.482530+0000) 2024-03-04T18:37:57.481 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:37:57.482+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:37:57.482865+0000) 2024-03-04T18:37:58.481 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:37:58.482+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:37:58.483168+0000) 2024-03-04T18:37:59.482 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:37:59.483+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:37:59.483460+0000) 2024-03-04T18:38:00.482 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:38:00.483+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:38:00.483774+0000) 2024-03-04T18:38:01.482 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:38:01.483+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:38:01.484099+0000) 2024-03-04T18:38:02.483 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:38:02.484+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:38:02.484400+0000) 2024-03-04T18:38:03.483 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:38:03.484+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:38:03.484717+0000) 2024-03-04T18:38:04.483 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:38:04.484+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:38:04.485012+0000) 2024-03-04T18:38:04.565 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:38:04.573 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:38:05.483 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:38:05.485+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:38:05.485320+0000) 2024-03-04T18:38:06.484 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:38:06.484+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:38:06.485601+0000) 2024-03-04T18:38:07.484 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:38:07.485+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:38:07.485956+0000) 2024-03-04T18:38:08.485 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:38:08.486+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:38:08.486250+0000) 2024-03-04T18:38:09.485 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:38:09.485+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:38:09.486546+0000) 2024-03-04T18:38:10.485 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:38:10.485+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:38:10.486783+0000) 2024-03-04T18:38:11.485 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:38:11.486+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:38:11.487128+0000) 2024-03-04T18:38:12.486 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:38:12.485+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:38:12.487449+0000) 2024-03-04T18:38:13.486 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:38:13.486+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:38:13.487748+0000) 2024-03-04T18:38:14.486 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:38:14.487+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:38:14.488119+0000) 2024-03-04T18:38:15.487 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:38:15.487+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:38:15.488358+0000) 2024-03-04T18:38:16.487 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:38:16.487+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:38:16.488594+0000) 2024-03-04T18:38:17.487 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:38:17.487+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:38:17.488876+0000) 2024-03-04T18:38:18.488 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:38:18.488+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:38:18.489131+0000) 2024-03-04T18:38:19.488 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:38:19.488+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:38:19.489373+0000) 2024-03-04T18:38:20.488 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:38:20.488+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:38:20.489631+0000) 2024-03-04T18:38:21.488 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:38:21.489+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:38:21.489922+0000) 2024-03-04T18:38:22.489 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:38:22.489+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:38:22.490166+0000) 2024-03-04T18:38:23.489 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:38:23.489+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:38:23.490423+0000) 2024-03-04T18:38:24.489 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:38:24.489+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:38:24.490665+0000) 2024-03-04T18:38:25.489 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:38:25.490+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:38:25.490954+0000) 2024-03-04T18:38:26.489 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:38:26.490+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:38:26.491152+0000) 2024-03-04T18:38:27.490 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:38:27.490+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:38:27.491431+0000) 2024-03-04T18:38:28.490 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:38:28.490+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:38:28.491725+0000) 2024-03-04T18:38:29.490 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:38:29.491+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:38:29.492017+0000) 2024-03-04T18:38:30.491 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:38:30.491+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:38:30.492319+0000) 2024-03-04T18:38:31.491 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:38:31.491+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:38:31.492607+0000) 2024-03-04T18:38:32.491 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:38:32.492+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:38:32.492937+0000) 2024-03-04T18:38:33.491 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:38:33.492+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:38:33.493238+0000) 2024-03-04T18:38:34.492 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:38:34.492+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:38:34.493538+0000) 2024-03-04T18:38:34.712 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:38:34.720 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:38:35.492 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:38:35.493+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:38:35.493874+0000) 2024-03-04T18:38:36.493 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:38:36.493+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:38:36.494185+0000) 2024-03-04T18:38:37.493 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:38:37.493+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:38:37.494472+0000) 2024-03-04T18:38:38.493 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:38:38.494+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:38:38.494771+0000) 2024-03-04T18:38:39.493 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:38:39.494+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:38:39.495080+0000) 2024-03-04T18:38:40.494 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:38:40.494+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:38:40.495373+0000) 2024-03-04T18:38:41.494 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:38:41.495+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:38:41.495668+0000) 2024-03-04T18:38:42.494 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:38:42.495+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:38:42.495987+0000) 2024-03-04T18:38:43.495 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:38:43.495+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:38:43.496291+0000) 2024-03-04T18:38:44.495 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:38:44.495+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:38:44.496593+0000) 2024-03-04T18:38:45.495 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:38:45.496+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:38:45.496884+0000) 2024-03-04T18:38:46.496 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:38:46.496+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:38:46.497131+0000) 2024-03-04T18:38:47.496 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:38:47.496+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:38:47.497398+0000) 2024-03-04T18:38:48.496 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:38:48.497+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:38:48.497667+0000) 2024-03-04T18:38:49.496 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:38:49.497+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:38:49.497940+0000) 2024-03-04T18:38:50.496 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:38:50.497+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:38:50.498185+0000) 2024-03-04T18:38:51.497 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:38:51.497+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:38:51.498435+0000) 2024-03-04T18:38:52.497 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:38:52.498+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:38:52.498686+0000) 2024-03-04T18:38:53.497 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:38:53.498+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:38:53.498945+0000) 2024-03-04T18:38:54.498 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:38:54.498+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:38:54.499186+0000) 2024-03-04T18:38:55.498 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:38:55.498+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:38:55.499372+0000) 2024-03-04T18:38:56.498 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:38:56.499+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:38:56.499618+0000) 2024-03-04T18:38:57.498 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:38:57.499+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:38:57.499940+0000) 2024-03-04T18:38:58.499 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:38:58.499+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:38:58.500268+0000) 2024-03-04T18:38:59.499 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:38:59.500+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:38:59.500594+0000) 2024-03-04T18:39:00.499 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:39:00.500+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:39:00.500919+0000) 2024-03-04T18:39:01.500 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:39:01.500+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:39:01.501216+0000) 2024-03-04T18:39:02.500 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:39:02.501+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:39:02.501466+0000) 2024-03-04T18:39:03.500 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:39:03.501+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:39:03.501727+0000) 2024-03-04T18:39:04.500 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:39:04.501+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:39:04.501966+0000) 2024-03-04T18:39:04.865 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:39:04.876 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:39:05.500 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:39:05.501+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:39:05.502206+0000) 2024-03-04T18:39:06.501 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:39:06.502+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:39:06.502446+0000) 2024-03-04T18:39:07.501 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:39:07.502+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:39:07.502767+0000) 2024-03-04T18:39:08.501 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:39:08.502+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:39:08.502951+0000) 2024-03-04T18:39:09.502 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:39:09.503+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:39:09.503186+0000) 2024-03-04T18:39:10.502 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:39:10.503+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:39:10.503428+0000) 2024-03-04T18:39:11.503 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:39:11.502+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:39:11.503661+0000) 2024-03-04T18:39:12.502 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:39:12.502+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:39:12.503934+0000) 2024-03-04T18:39:13.503 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:39:13.502+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:39:13.504177+0000) 2024-03-04T18:39:14.503 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:39:14.503+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:39:14.504459+0000) 2024-03-04T18:39:15.503 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:39:15.502+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:39:15.504690+0000) 2024-03-04T18:39:16.503 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:39:16.503+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:39:16.504992+0000) 2024-03-04T18:39:17.504 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:39:17.504+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:39:17.505265+0000) 2024-03-04T18:39:18.504 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:39:18.504+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:39:18.505509+0000) 2024-03-04T18:39:19.507 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:39:19.504+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:39:19.505752+0000) 2024-03-04T18:39:20.505 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:39:20.505+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:39:20.506028+0000) 2024-03-04T18:39:21.505 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:39:21.504+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:39:21.506228+0000) 2024-03-04T18:39:22.505 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:39:22.505+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:39:22.506454+0000) 2024-03-04T18:39:23.505 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:39:23.505+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:39:23.506685+0000) 2024-03-04T18:39:24.505 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:39:24.506+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:39:24.506935+0000) 2024-03-04T18:39:25.505 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:39:25.506+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:39:25.507123+0000) 2024-03-04T18:39:26.506 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:39:26.506+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:39:26.507384+0000) 2024-03-04T18:39:27.506 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:39:27.506+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:39:27.507634+0000) 2024-03-04T18:39:28.506 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:39:28.507+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:39:28.507920+0000) 2024-03-04T18:39:29.506 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:39:29.507+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:39:29.508152+0000) 2024-03-04T18:39:30.507 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:39:30.507+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:39:30.508399+0000) 2024-03-04T18:39:31.507 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:39:31.507+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:39:31.508631+0000) 2024-03-04T18:39:32.507 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:39:32.508+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:39:32.508925+0000) 2024-03-04T18:39:33.508 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:39:33.508+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:39:33.509170+0000) 2024-03-04T18:39:34.508 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:39:34.508+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:39:34.509418+0000) 2024-03-04T18:39:35.081 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:39:35.089 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:39:35.508 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:39:35.508+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:39:35.509652+0000) 2024-03-04T18:39:36.508 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:39:36.509+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:39:36.509916+0000) 2024-03-04T18:39:37.509 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:39:37.509+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:39:37.510164+0000) 2024-03-04T18:39:38.511 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:39:38.509+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:39:38.510415+0000) 2024-03-04T18:39:39.509 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:39:39.510+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:39:39.510688+0000) 2024-03-04T18:39:40.509 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:39:40.510+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:39:40.510927+0000) 2024-03-04T18:39:41.509 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:39:41.510+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:39:41.511164+0000) 2024-03-04T18:39:42.510 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:39:42.510+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:39:42.511407+0000) 2024-03-04T18:39:43.510 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:39:43.511+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:39:43.511658+0000) 2024-03-04T18:39:44.510 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:39:44.511+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:39:44.511940+0000) 2024-03-04T18:39:45.510 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:39:45.511+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:39:45.512180+0000) 2024-03-04T18:39:46.511 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:39:46.511+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:39:46.512455+0000) 2024-03-04T18:39:47.511 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:39:47.512+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:39:47.512709+0000) 2024-03-04T18:39:48.511 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:39:48.512+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:39:48.512955+0000) 2024-03-04T18:39:49.512 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:39:49.512+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:39:49.513189+0000) 2024-03-04T18:39:50.512 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:39:50.512+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:39:50.513378+0000) 2024-03-04T18:39:51.512 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:39:51.513+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:39:51.513612+0000) 2024-03-04T18:39:52.512 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:39:52.513+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:39:52.513888+0000) 2024-03-04T18:39:53.512 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:39:53.513+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:39:53.514134+0000) 2024-03-04T18:39:54.513 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:39:54.513+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:39:54.514380+0000) 2024-03-04T18:39:55.513 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:39:55.514+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:39:55.514618+0000) 2024-03-04T18:39:56.513 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:39:56.514+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:39:56.514905+0000) 2024-03-04T18:39:57.513 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:39:57.514+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:39:57.515080+0000) 2024-03-04T18:39:58.514 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:39:58.514+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:39:58.515353+0000) 2024-03-04T18:39:59.514 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:39:59.515+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:39:59.515589+0000) 2024-03-04T18:40:00.514 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:40:00.515+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:40:00.515925+0000) 2024-03-04T18:40:01.515 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:40:01.515+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:40:01.516161+0000) 2024-03-04T18:40:02.515 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:40:02.516+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:40:02.516402+0000) 2024-03-04T18:40:03.515 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:40:03.516+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:40:03.516654+0000) 2024-03-04T18:40:04.515 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:40:04.516+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:40:04.516989+0000) 2024-03-04T18:40:05.261 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:40:05.269 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:40:05.515 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:40:05.516+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:40:05.517217+0000) 2024-03-04T18:40:06.516 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:40:06.517+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:40:06.517462+0000) 2024-03-04T18:40:07.516 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:40:07.517+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:40:07.517642+0000) 2024-03-04T18:40:08.516 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:40:08.517+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:40:08.517933+0000) 2024-03-04T18:40:09.516 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:40:09.517+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:40:09.518166+0000) 2024-03-04T18:40:10.517 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:40:10.518+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:40:10.518411+0000) 2024-03-04T18:40:11.517 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:40:11.517+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:40:11.518661+0000) 2024-03-04T18:40:12.517 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:40:12.518+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:40:12.518947+0000) 2024-03-04T18:40:13.517 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:40:13.518+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:40:13.519233+0000) 2024-03-04T18:40:14.518 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:40:14.518+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:40:14.519488+0000) 2024-03-04T18:40:15.518 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:40:15.518+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:40:15.519735+0000) 2024-03-04T18:40:16.518 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:40:16.518+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:40:16.520028+0000) 2024-03-04T18:40:17.519 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:40:17.519+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:40:17.520304+0000) 2024-03-04T18:40:18.519 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:40:18.518+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:40:18.520503+0000) 2024-03-04T18:40:19.519 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:40:19.518+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:40:19.520765+0000) 2024-03-04T18:40:20.519 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:40:20.520+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:40:20.521022+0000) 2024-03-04T18:40:21.520 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:40:21.519+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:40:21.521222+0000) 2024-03-04T18:40:22.520 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:40:22.519+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:40:22.521422+0000) 2024-03-04T18:40:23.520 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:40:23.520+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:40:23.521628+0000) 2024-03-04T18:40:24.520 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:40:24.521+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:40:24.521946+0000) 2024-03-04T18:40:25.521 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:40:25.521+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:40:25.522222+0000) 2024-03-04T18:40:26.521 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:40:26.521+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:40:26.522510+0000) 2024-03-04T18:40:27.521 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:40:27.521+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:40:27.522795+0000) 2024-03-04T18:40:28.521 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:40:28.522+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:40:28.523116+0000) 2024-03-04T18:40:29.522 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:40:29.522+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:40:29.523401+0000) 2024-03-04T18:40:30.522 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:40:30.522+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:40:30.523684+0000) 2024-03-04T18:40:31.522 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:40:31.523+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:40:31.523968+0000) 2024-03-04T18:40:32.522 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:40:32.523+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:40:32.524247+0000) 2024-03-04T18:40:33.523 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:40:33.523+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:40:33.524547+0000) 2024-03-04T18:40:34.523 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:40:34.524+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:40:34.524879+0000) 2024-03-04T18:40:35.398 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:40:35.409 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:40:35.526 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:40:35.524+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:40:35.525128+0000) 2024-03-04T18:40:36.524 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:40:36.524+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:40:36.525347+0000) 2024-03-04T18:40:37.524 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:40:37.524+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:40:37.525626+0000) 2024-03-04T18:40:38.524 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:40:38.525+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:40:38.525965+0000) 2024-03-04T18:40:39.525 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:40:39.525+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:40:39.526257+0000) 2024-03-04T18:40:40.525 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:40:40.525+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:40:40.526542+0000) 2024-03-04T18:40:41.525 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:40:41.526+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:40:41.526862+0000) 2024-03-04T18:40:42.526 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:40:42.526+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:40:42.527124+0000) 2024-03-04T18:40:43.526 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:40:43.526+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:40:43.527370+0000) 2024-03-04T18:40:44.526 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:40:44.527+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:40:44.527631+0000) 2024-03-04T18:40:45.526 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:40:45.527+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:40:45.527916+0000) 2024-03-04T18:40:46.526 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:40:46.527+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:40:46.528152+0000) 2024-03-04T18:40:47.527 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:40:47.527+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:40:47.528376+0000) 2024-03-04T18:40:48.527 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:40:48.528+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:40:48.528725+0000) 2024-03-04T18:40:49.527 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:40:49.528+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:40:49.528966+0000) 2024-03-04T18:40:50.528 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:40:50.528+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:40:50.529209+0000) 2024-03-04T18:40:51.528 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:40:51.528+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:40:51.529443+0000) 2024-03-04T18:40:52.528 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:40:52.529+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:40:52.529686+0000) 2024-03-04T18:40:53.528 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:40:53.529+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:40:53.529930+0000) 2024-03-04T18:40:54.529 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:40:54.529+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:40:54.530179+0000) 2024-03-04T18:40:55.529 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:40:55.529+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:40:55.530402+0000) 2024-03-04T18:40:56.529 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:40:56.530+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:40:56.530641+0000) 2024-03-04T18:40:57.529 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:40:57.530+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:40:57.530922+0000) 2024-03-04T18:40:58.530 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:40:58.530+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:40:58.531168+0000) 2024-03-04T18:40:59.530 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:40:59.531+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:40:59.531440+0000) 2024-03-04T18:41:00.530 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:41:00.531+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:41:00.531679+0000) 2024-03-04T18:41:01.530 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:41:01.531+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:41:01.531942+0000) 2024-03-04T18:41:02.530 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:41:02.531+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:41:02.532185+0000) 2024-03-04T18:41:03.531 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:41:03.532+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:41:03.532431+0000) 2024-03-04T18:41:04.531 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:41:04.532+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:41:04.532676+0000) 2024-03-04T18:41:05.531 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:41:05.532+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:41:05.532937+0000) 2024-03-04T18:41:05.595 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:41:05.603 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:41:06.532 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:41:06.532+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:41:06.533185+0000) 2024-03-04T18:41:07.532 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:41:07.533+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:41:07.533418+0000) 2024-03-04T18:41:08.532 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:41:08.533+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:41:08.533657+0000) 2024-03-04T18:41:09.535 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:41:09.533+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:41:09.533924+0000) 2024-03-04T18:41:10.533 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:41:10.533+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:41:10.534169+0000) 2024-03-04T18:41:11.533 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:41:11.534+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:41:11.534458+0000) 2024-03-04T18:41:12.533 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:41:12.534+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:41:12.534749+0000) 2024-03-04T18:41:13.534 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:41:13.534+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:41:13.535066+0000) 2024-03-04T18:41:14.534 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:41:14.534+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:41:14.535295+0000) 2024-03-04T18:41:15.534 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:41:15.534+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:41:15.535573+0000) 2024-03-04T18:41:16.534 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:41:16.534+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:41:16.535895+0000) 2024-03-04T18:41:17.535 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:41:17.535+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:41:17.536188+0000) 2024-03-04T18:41:18.535 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:41:18.534+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:41:18.536489+0000) 2024-03-04T18:41:19.535 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:41:19.535+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:41:19.536738+0000) 2024-03-04T18:41:20.535 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:41:20.535+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:41:20.537027+0000) 2024-03-04T18:41:21.536 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:41:21.536+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:41:21.537317+0000) 2024-03-04T18:41:22.536 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:41:22.536+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:41:22.537596+0000) 2024-03-04T18:41:23.537 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:41:23.536+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:41:23.537925+0000) 2024-03-04T18:41:24.537 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:41:24.537+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:41:24.538210+0000) 2024-03-04T18:41:25.537 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:41:25.537+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:41:25.538501+0000) 2024-03-04T18:41:26.537 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:41:26.537+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:41:26.538798+0000) 2024-03-04T18:41:27.538 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:41:27.538+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:41:27.539136+0000) 2024-03-04T18:41:28.538 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:41:28.538+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:41:28.539450+0000) 2024-03-04T18:41:29.538 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:41:29.538+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:41:29.539739+0000) 2024-03-04T18:41:30.538 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:41:30.539+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:41:30.540038+0000) 2024-03-04T18:41:31.539 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:41:31.539+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:41:31.540374+0000) 2024-03-04T18:41:32.539 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:41:32.539+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:41:32.540670+0000) 2024-03-04T18:41:33.540 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:41:33.540+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:41:33.540976+0000) 2024-03-04T18:41:34.540 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:41:34.540+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:41:34.541272+0000) 2024-03-04T18:41:35.540 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:41:35.540+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:41:35.541563+0000) 2024-03-04T18:41:35.750 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:41:35.757 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:41:36.540 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:41:36.541+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:41:36.541891+0000) 2024-03-04T18:41:37.540 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:41:37.541+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:41:37.542182+0000) 2024-03-04T18:41:38.541 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:41:38.541+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:41:38.542479+0000) 2024-03-04T18:41:39.541 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:41:39.542+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:41:39.542785+0000) 2024-03-04T18:41:40.542 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:41:40.542+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:41:40.543152+0000) 2024-03-04T18:41:41.542 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:41:41.542+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:41:41.543459+0000) 2024-03-04T18:41:42.542 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:41:42.543+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:41:42.543752+0000) 2024-03-04T18:41:43.542 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:41:43.543+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:41:43.544052+0000) 2024-03-04T18:41:44.543 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:41:44.543+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:41:44.544315+0000) 2024-03-04T18:41:45.543 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:41:45.543+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:41:45.544565+0000) 2024-03-04T18:41:46.543 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:41:46.544+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:41:46.544813+0000) 2024-03-04T18:41:47.543 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:41:47.544+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:41:47.545044+0000) 2024-03-04T18:41:48.544 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:41:48.544+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:41:48.545290+0000) 2024-03-04T18:41:49.544 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:41:49.544+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:41:49.545526+0000) 2024-03-04T18:41:50.544 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:41:50.545+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:41:50.545775+0000) 2024-03-04T18:41:51.544 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:41:51.545+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:41:51.546009+0000) 2024-03-04T18:41:52.545 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:41:52.545+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:41:52.546251+0000) 2024-03-04T18:41:53.545 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:41:53.545+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:41:53.546511+0000) 2024-03-04T18:41:54.545 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:41:54.545+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:41:54.546779+0000) 2024-03-04T18:41:55.545 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:41:55.546+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:41:55.547011+0000) 2024-03-04T18:41:56.546 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:41:56.546+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:41:56.547253+0000) 2024-03-04T18:41:57.546 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:41:57.547+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:41:57.547486+0000) 2024-03-04T18:41:58.546 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:41:58.547+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:41:58.547729+0000) 2024-03-04T18:41:59.547 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:41:59.547+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:41:59.547964+0000) 2024-03-04T18:42:00.547 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:42:00.547+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:42:00.548194+0000) 2024-03-04T18:42:01.547 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:42:01.548+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:42:01.548428+0000) 2024-03-04T18:42:02.547 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:42:02.548+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:42:02.548665+0000) 2024-03-04T18:42:03.547 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:42:03.548+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:42:03.548934+0000) 2024-03-04T18:42:04.548 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:42:04.548+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:42:04.549183+0000) 2024-03-04T18:42:05.548 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:42:05.549+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:42:05.549421+0000) 2024-03-04T18:42:05.983 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:42:05.995 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:42:06.548 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:42:06.549+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:42:06.549659+0000) 2024-03-04T18:42:07.548 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:42:07.549+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:42:07.549920+0000) 2024-03-04T18:42:08.549 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:42:08.549+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:42:08.550198+0000) 2024-03-04T18:42:09.549 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:42:09.550+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:42:09.550429+0000) 2024-03-04T18:42:10.549 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:42:10.550+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:42:10.550675+0000) 2024-03-04T18:42:11.549 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:42:11.550+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:42:11.550886+0000) 2024-03-04T18:42:12.550 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:42:12.550+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:42:12.551129+0000) 2024-03-04T18:42:13.550 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:42:13.551+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:42:13.551395+0000) 2024-03-04T18:42:14.550 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:42:14.550+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:42:14.551697+0000) 2024-03-04T18:42:15.550 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:42:15.551+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:42:15.551981+0000) 2024-03-04T18:42:16.551 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:42:16.551+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:42:16.552211+0000) 2024-03-04T18:42:17.551 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:42:17.551+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:42:17.552500+0000) 2024-03-04T18:42:18.551 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:42:18.551+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:42:18.552800+0000) 2024-03-04T18:42:19.552 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:42:19.552+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:42:19.553143+0000) 2024-03-04T18:42:20.552 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:42:20.552+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:42:20.553442+0000) 2024-03-04T18:42:21.552 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:42:21.552+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:42:21.553730+0000) 2024-03-04T18:42:22.552 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:42:22.552+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:42:22.554028+0000) 2024-03-04T18:42:23.553 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:42:23.553+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:42:23.554325+0000) 2024-03-04T18:42:24.554 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:42:24.553+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:42:24.554665+0000) 2024-03-04T18:42:25.554 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:42:25.553+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:42:25.554963+0000) 2024-03-04T18:42:26.554 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:42:26.554+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:42:26.555210+0000) 2024-03-04T18:42:27.554 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:42:27.554+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:42:27.555498+0000) 2024-03-04T18:42:28.554 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:42:28.554+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:42:28.555801+0000) 2024-03-04T18:42:29.554 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:42:29.555+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:42:29.556082+0000) 2024-03-04T18:42:30.555 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:42:30.555+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:42:30.556379+0000) 2024-03-04T18:42:31.555 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:42:31.555+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:42:31.556667+0000) 2024-03-04T18:42:32.555 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:42:32.556+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:42:32.556993+0000) 2024-03-04T18:42:33.556 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:42:33.556+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:42:33.557293+0000) 2024-03-04T18:42:34.556 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:42:34.556+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:42:34.557596+0000) 2024-03-04T18:42:35.556 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:42:35.557+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:42:35.557941+0000) 2024-03-04T18:42:36.160 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:42:36.171 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:42:36.556 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:42:36.557+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:42:36.558264+0000) 2024-03-04T18:42:37.557 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:42:37.557+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:42:37.558514+0000) 2024-03-04T18:42:38.557 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:42:38.557+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:42:38.558762+0000) 2024-03-04T18:42:39.557 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:42:39.558+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:42:39.559001+0000) 2024-03-04T18:42:40.558 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:42:40.558+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:42:40.559253+0000) 2024-03-04T18:42:41.558 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:42:41.558+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:42:41.559493+0000) 2024-03-04T18:42:42.558 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:42:42.559+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:42:42.559742+0000) 2024-03-04T18:42:43.558 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:42:43.559+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:42:43.559995+0000) 2024-03-04T18:42:44.559 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:42:44.559+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:42:44.560239+0000) 2024-03-04T18:42:45.559 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:42:45.559+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:42:45.560516+0000) 2024-03-04T18:42:46.559 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:42:46.559+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:42:46.560747+0000) 2024-03-04T18:42:47.559 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:42:47.560+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:42:47.560980+0000) 2024-03-04T18:42:48.560 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:42:48.560+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:42:48.561233+0000) 2024-03-04T18:42:49.560 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:42:49.560+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:42:49.561478+0000) 2024-03-04T18:42:50.560 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:42:50.561+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:42:50.561719+0000) 2024-03-04T18:42:51.560 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:42:51.561+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:42:51.561955+0000) 2024-03-04T18:42:52.561 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:42:52.561+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:42:52.562202+0000) 2024-03-04T18:42:53.561 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:42:53.561+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:42:53.562446+0000) 2024-03-04T18:42:54.561 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:42:54.562+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:42:54.562694+0000) 2024-03-04T18:42:55.561 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:42:55.562+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:42:55.562936+0000) 2024-03-04T18:42:56.562 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:42:56.562+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:42:56.563188+0000) 2024-03-04T18:42:57.562 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:42:57.562+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:42:57.563420+0000) 2024-03-04T18:42:58.562 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:42:58.563+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:42:58.563675+0000) 2024-03-04T18:42:59.562 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:42:59.563+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:42:59.563920+0000) 2024-03-04T18:43:00.563 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:43:00.563+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:43:00.564156+0000) 2024-03-04T18:43:01.563 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:43:01.563+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:43:01.564383+0000) 2024-03-04T18:43:02.563 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:43:02.564+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:43:02.564625+0000) 2024-03-04T18:43:03.563 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:43:03.564+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:43:03.564902+0000) 2024-03-04T18:43:04.563 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:43:04.564+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:43:04.565142+0000) 2024-03-04T18:43:05.564 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:43:05.565+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:43:05.565397+0000) 2024-03-04T18:43:06.374 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:43:06.382 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:43:06.564 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:43:06.564+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:43:06.565659+0000) 2024-03-04T18:43:07.564 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:43:07.565+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:43:07.565944+0000) 2024-03-04T18:43:08.565 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:43:08.565+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:43:08.566213+0000) 2024-03-04T18:43:09.565 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:43:09.566+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:43:09.566454+0000) 2024-03-04T18:43:10.566 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:43:10.565+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:43:10.566716+0000) 2024-03-04T18:43:11.565 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:43:11.566+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:43:11.566938+0000) 2024-03-04T18:43:12.566 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:43:12.566+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:43:12.567202+0000) 2024-03-04T18:43:13.566 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:43:13.567+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:43:13.567466+0000) 2024-03-04T18:43:14.566 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:43:14.567+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:43:14.567734+0000) 2024-03-04T18:43:15.566 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:43:15.567+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:43:15.567943+0000) 2024-03-04T18:43:16.567 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:43:16.568+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:43:16.568208+0000) 2024-03-04T18:43:17.567 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:43:17.568+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:43:17.568451+0000) 2024-03-04T18:43:18.567 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:43:18.568+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:43:18.568746+0000) 2024-03-04T18:43:19.567 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:43:19.567+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:43:19.568984+0000) 2024-03-04T18:43:20.568 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:43:20.568+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:43:20.569222+0000) 2024-03-04T18:43:21.568 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:43:21.568+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:43:21.569451+0000) 2024-03-04T18:43:22.568 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:43:22.567+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:43:22.569717+0000) 2024-03-04T18:43:23.568 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:43:23.568+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:43:23.569976+0000) 2024-03-04T18:43:24.568 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:43:24.568+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:43:24.570171+0000) 2024-03-04T18:43:25.569 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:43:25.568+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:43:25.570423+0000) 2024-03-04T18:43:26.569 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:43:26.569+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:43:26.570667+0000) 2024-03-04T18:43:27.569 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:43:27.569+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:43:27.570934+0000) 2024-03-04T18:43:28.570 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:43:28.570+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:43:28.571219+0000) 2024-03-04T18:43:29.570 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:43:29.569+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:43:29.571468+0000) 2024-03-04T18:43:30.570 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:43:30.570+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:43:30.571712+0000) 2024-03-04T18:43:31.570 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:43:31.571+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:43:31.571943+0000) 2024-03-04T18:43:32.571 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:43:32.571+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:43:32.572188+0000) 2024-03-04T18:43:33.571 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:43:33.571+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:43:33.572410+0000) 2024-03-04T18:43:34.571 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:43:34.571+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:43:34.572677+0000) 2024-03-04T18:43:35.571 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:43:35.572+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:43:35.572932+0000) 2024-03-04T18:43:36.546 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:43:36.555 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:43:36.571 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:43:36.572+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:43:36.573101+0000) 2024-03-04T18:43:37.572 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:43:37.572+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:43:37.573330+0000) 2024-03-04T18:43:38.572 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:43:38.572+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:43:38.573590+0000) 2024-03-04T18:43:39.572 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:43:39.573+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:43:39.573864+0000) 2024-03-04T18:43:40.573 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:43:40.573+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:43:40.574098+0000) 2024-03-04T18:43:41.573 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:43:41.573+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:43:41.574330+0000) 2024-03-04T18:43:42.573 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:43:42.573+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:43:42.574606+0000) 2024-03-04T18:43:43.573 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:43:43.574+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:43:43.574883+0000) 2024-03-04T18:43:44.574 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:43:44.574+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:43:44.575153+0000) 2024-03-04T18:43:45.574 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:43:45.574+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:43:45.575385+0000) 2024-03-04T18:43:46.574 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:43:46.574+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:43:46.575626+0000) 2024-03-04T18:43:47.574 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:43:47.575+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:43:47.575888+0000) 2024-03-04T18:43:48.575 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:43:48.575+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:43:48.576131+0000) 2024-03-04T18:43:49.575 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:43:49.575+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:43:49.576374+0000) 2024-03-04T18:43:50.575 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:43:50.575+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:43:50.576616+0000) 2024-03-04T18:43:51.575 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:43:51.576+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:43:51.576887+0000) 2024-03-04T18:43:52.576 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:43:52.576+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:43:52.577124+0000) 2024-03-04T18:43:53.576 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:43:53.576+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:43:53.577358+0000) 2024-03-04T18:43:54.576 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:43:54.577+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:43:54.577608+0000) 2024-03-04T18:43:55.576 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:43:55.577+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:43:55.577920+0000) 2024-03-04T18:43:56.576 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:43:56.577+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:43:56.578206+0000) 2024-03-04T18:43:57.577 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:43:57.577+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:43:57.578494+0000) 2024-03-04T18:43:58.577 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:43:58.578+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:43:58.578775+0000) 2024-03-04T18:43:59.578 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:43:59.578+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:43:59.579074+0000) 2024-03-04T18:44:00.578 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:44:00.578+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:44:00.579368+0000) 2024-03-04T18:44:01.578 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:44:01.578+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:44:01.579630+0000) 2024-03-04T18:44:02.579 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:44:02.579+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:44:02.579917+0000) 2024-03-04T18:44:03.579 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:44:03.579+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:44:03.580162+0000) 2024-03-04T18:44:04.579 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:44:04.580+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:44:04.580409+0000) 2024-03-04T18:44:05.579 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:44:05.580+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:44:05.580662+0000) 2024-03-04T18:44:06.579 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:44:06.580+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:44:06.580952+0000) 2024-03-04T18:44:06.692 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:44:06.700 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:44:07.579 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:44:07.580+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:44:07.581129+0000) 2024-03-04T18:44:08.580 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:44:08.581+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:44:08.581369+0000) 2024-03-04T18:44:09.580 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:44:09.581+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:44:09.581621+0000) 2024-03-04T18:44:10.580 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:44:10.581+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:44:10.581906+0000) 2024-03-04T18:44:11.581 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:44:11.581+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:44:11.582142+0000) 2024-03-04T18:44:12.581 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:44:12.582+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:44:12.582383+0000) 2024-03-04T18:44:13.581 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:44:13.581+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:44:13.582689+0000) 2024-03-04T18:44:14.583 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:44:14.582+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:44:14.583003+0000) 2024-03-04T18:44:15.582 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:44:15.582+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:44:15.583284+0000) 2024-03-04T18:44:16.582 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:44:16.583+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:44:16.583578+0000) 2024-03-04T18:44:17.582 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:44:17.583+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:44:17.583888+0000) 2024-03-04T18:44:18.583 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:44:18.583+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:44:18.584192+0000) 2024-03-04T18:44:19.583 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:44:19.584+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:44:19.584479+0000) 2024-03-04T18:44:20.583 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:44:20.584+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:44:20.584782+0000) 2024-03-04T18:44:21.584 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:44:21.583+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:44:21.585101+0000) 2024-03-04T18:44:22.584 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:44:22.584+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:44:22.585396+0000) 2024-03-04T18:44:23.584 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:44:23.584+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:44:23.585692+0000) 2024-03-04T18:44:24.584 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:44:24.584+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:44:24.585991+0000) 2024-03-04T18:44:25.585 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:44:25.585+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:44:25.586315+0000) 2024-03-04T18:44:26.585 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:44:26.585+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:44:26.586604+0000) 2024-03-04T18:44:27.586 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:44:27.585+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:44:27.586916+0000) 2024-03-04T18:44:28.586 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:44:28.586+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:44:28.587208+0000) 2024-03-04T18:44:29.586 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:44:29.586+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:44:29.587495+0000) 2024-03-04T18:44:30.586 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:44:30.586+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:44:30.587796+0000) 2024-03-04T18:44:31.587 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:44:31.587+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:44:31.588114+0000) 2024-03-04T18:44:32.587 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:44:32.587+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:44:32.588358+0000) 2024-03-04T18:44:33.587 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:44:33.587+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:44:33.588555+0000) 2024-03-04T18:44:34.587 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:44:34.587+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:44:34.588794+0000) 2024-03-04T18:44:35.587 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:44:35.588+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:44:35.589062+0000) 2024-03-04T18:44:36.588 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:44:36.588+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:44:36.589256+0000) 2024-03-04T18:44:36.914 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:44:36.922 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:44:37.588 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:44:37.588+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:44:37.589486+0000) 2024-03-04T18:44:38.588 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:44:38.588+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:44:38.589725+0000) 2024-03-04T18:44:39.588 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:44:39.589+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:44:39.589956+0000) 2024-03-04T18:44:40.589 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:44:40.589+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:44:40.590207+0000) 2024-03-04T18:44:41.589 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:44:41.589+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:44:41.590437+0000) 2024-03-04T18:44:42.589 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:44:42.589+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:44:42.590683+0000) 2024-03-04T18:44:43.589 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:44:43.590+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:44:43.590927+0000) 2024-03-04T18:44:44.590 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:44:44.590+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:44:44.591171+0000) 2024-03-04T18:44:45.590 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:44:45.590+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:44:45.591402+0000) 2024-03-04T18:44:46.590 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:44:46.590+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:44:46.591673+0000) 2024-03-04T18:44:47.590 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:44:47.591+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:44:47.591937+0000) 2024-03-04T18:44:48.591 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:44:48.591+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:44:48.592180+0000) 2024-03-04T18:44:49.591 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:44:49.591+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:44:49.592414+0000) 2024-03-04T18:44:50.591 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:44:50.592+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:44:50.592657+0000) 2024-03-04T18:44:51.591 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:44:51.592+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:44:51.592930+0000) 2024-03-04T18:44:52.591 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:44:52.592+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:44:52.593166+0000) 2024-03-04T18:44:53.592 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:44:53.592+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:44:53.593408+0000) 2024-03-04T18:44:54.592 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:44:54.593+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:44:54.593652+0000) 2024-03-04T18:44:55.592 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:44:55.593+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:44:55.593960+0000) 2024-03-04T18:44:56.593 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:44:56.593+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:44:56.594257+0000) 2024-03-04T18:44:57.593 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:44:57.593+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:44:57.594534+0000) 2024-03-04T18:44:58.593 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:44:58.594+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:44:58.594882+0000) 2024-03-04T18:44:59.594 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:44:59.594+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:44:59.595219+0000) 2024-03-04T18:45:00.594 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:45:00.594+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:45:00.595515+0000) 2024-03-04T18:45:01.594 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:45:01.595+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:45:01.595856+0000) 2024-03-04T18:45:02.595 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:45:02.595+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:45:02.596160+0000) 2024-03-04T18:45:03.595 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:45:03.595+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:45:03.596463+0000) 2024-03-04T18:45:04.595 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:45:04.596+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:45:04.596752+0000) 2024-03-04T18:45:05.595 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:45:05.596+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:45:05.597051+0000) 2024-03-04T18:45:06.596 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:45:06.596+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:45:06.597351+0000) 2024-03-04T18:45:07.060 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:45:07.068 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:45:07.596 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:45:07.597+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:45:07.597638+0000) 2024-03-04T18:45:08.597 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:45:08.597+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:45:08.597959+0000) 2024-03-04T18:45:09.597 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:45:09.597+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:45:09.598248+0000) 2024-03-04T18:45:10.597 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:45:10.598+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:45:10.598547+0000) 2024-03-04T18:45:11.597 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:45:11.598+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:45:11.598857+0000) 2024-03-04T18:45:12.598 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:45:12.598+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:45:12.599154+0000) 2024-03-04T18:45:13.598 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:45:13.599+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:45:13.599460+0000) 2024-03-04T18:45:14.598 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:45:14.599+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:45:14.599759+0000) 2024-03-04T18:45:15.599 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:45:15.599+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:45:15.600035+0000) 2024-03-04T18:45:16.599 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:45:16.600+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:45:16.600307+0000) 2024-03-04T18:45:17.599 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:45:17.600+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:45:17.600591+0000) 2024-03-04T18:45:18.599 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:45:18.600+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:45:18.600910+0000) 2024-03-04T18:45:19.600 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:45:19.600+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:45:19.601207+0000) 2024-03-04T18:45:20.600 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:45:20.601+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:45:20.601493+0000) 2024-03-04T18:45:21.600 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:45:21.601+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:45:21.601778+0000) 2024-03-04T18:45:22.600 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:45:22.601+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:45:22.602061+0000) 2024-03-04T18:45:23.601 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:45:23.601+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:45:23.602354+0000) 2024-03-04T18:45:24.601 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:45:24.601+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:45:24.602647+0000) 2024-03-04T18:45:25.601 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:45:25.601+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:45:25.602971+0000) 2024-03-04T18:45:26.602 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:45:26.602+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:45:26.603283+0000) 2024-03-04T18:45:27.602 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:45:27.602+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:45:27.603597+0000) 2024-03-04T18:45:28.602 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:45:28.602+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:45:28.603918+0000) 2024-03-04T18:45:29.603 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:45:29.602+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:45:29.604140+0000) 2024-03-04T18:45:30.603 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:45:30.603+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:45:30.604379+0000) 2024-03-04T18:45:31.603 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:45:31.603+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:45:31.604676+0000) 2024-03-04T18:45:32.603 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:45:32.603+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:45:32.604972+0000) 2024-03-04T18:45:33.604 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:45:33.604+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:45:33.605267+0000) 2024-03-04T18:45:34.604 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:45:34.604+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:45:34.605564+0000) 2024-03-04T18:45:35.604 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:45:35.605+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:45:35.605881+0000) 2024-03-04T18:45:36.605 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:45:36.605+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:45:36.606199+0000) 2024-03-04T18:45:37.274 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:45:37.282 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:45:37.605 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:45:37.605+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:45:37.606495+0000) 2024-03-04T18:45:38.605 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:45:38.605+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:45:38.606789+0000) 2024-03-04T18:45:39.606 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:45:39.606+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:45:39.607127+0000) 2024-03-04T18:45:40.606 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:45:40.606+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:45:40.607430+0000) 2024-03-04T18:45:41.606 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:45:41.606+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:45:41.607718+0000) 2024-03-04T18:45:42.607 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:45:42.607+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:45:42.608101+0000) 2024-03-04T18:45:43.607 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:45:43.607+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:45:43.608350+0000) 2024-03-04T18:45:44.607 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:45:44.607+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:45:44.608593+0000) 2024-03-04T18:45:45.607 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:45:45.608+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:45:45.608907+0000) 2024-03-04T18:45:46.608 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:45:46.608+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:45:46.609194+0000) 2024-03-04T18:45:47.608 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:45:47.608+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:45:47.609470+0000) 2024-03-04T18:45:48.608 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:45:48.609+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:45:48.609764+0000) 2024-03-04T18:45:49.608 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:45:49.609+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:45:49.610055+0000) 2024-03-04T18:45:50.609 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:45:50.609+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:45:50.610299+0000) 2024-03-04T18:45:51.609 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:45:51.609+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:45:51.610532+0000) 2024-03-04T18:45:52.609 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:45:52.610+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:45:52.610870+0000) 2024-03-04T18:45:53.610 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:45:53.610+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:45:53.611207+0000) 2024-03-04T18:45:54.610 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:45:54.610+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:45:54.611492+0000) 2024-03-04T18:45:55.610 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:45:55.611+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:45:55.611780+0000) 2024-03-04T18:45:56.611 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:45:56.611+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:45:56.612086+0000) 2024-03-04T18:45:57.611 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:45:57.611+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:45:57.612362+0000) 2024-03-04T18:45:58.611 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:45:58.612+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:45:58.612641+0000) 2024-03-04T18:45:59.611 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:45:59.612+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:45:59.612950+0000) 2024-03-04T18:46:00.612 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:46:00.612+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:46:00.613261+0000) 2024-03-04T18:46:01.612 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:46:01.613+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:46:01.613548+0000) 2024-03-04T18:46:02.612 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:46:02.613+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:46:02.613873+0000) 2024-03-04T18:46:03.613 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:46:03.613+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:46:03.614175+0000) 2024-03-04T18:46:04.613 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:46:04.613+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:46:04.614462+0000) 2024-03-04T18:46:05.613 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:46:05.614+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:46:05.614749+0000) 2024-03-04T18:46:06.614 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:46:06.614+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:46:06.615037+0000) 2024-03-04T18:46:07.418 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:46:07.426 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:46:07.614 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:46:07.614+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:46:07.615286+0000) 2024-03-04T18:46:08.614 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:46:08.615+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:46:08.615569+0000) 2024-03-04T18:46:09.614 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:46:09.615+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:46:09.615903+0000) 2024-03-04T18:46:10.615 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:46:10.615+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:46:10.616197+0000) 2024-03-04T18:46:11.615 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:46:11.616+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:46:11.616479+0000) 2024-03-04T18:46:12.615 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:46:12.616+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:46:12.616773+0000) 2024-03-04T18:46:13.616 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:46:13.616+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:46:13.617117+0000) 2024-03-04T18:46:14.616 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:46:14.617+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:46:14.617417+0000) 2024-03-04T18:46:15.616 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:46:15.617+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:46:15.617704+0000) 2024-03-04T18:46:16.616 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:46:16.617+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:46:16.617958+0000) 2024-03-04T18:46:17.617 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:46:17.617+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:46:17.618193+0000) 2024-03-04T18:46:18.617 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:46:18.618+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:46:18.618460+0000) 2024-03-04T18:46:19.617 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:46:19.618+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:46:19.618698+0000) 2024-03-04T18:46:20.617 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:46:20.618+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:46:20.618940+0000) 2024-03-04T18:46:21.618 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:46:21.619+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:46:21.619190+0000) 2024-03-04T18:46:22.618 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:46:22.619+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:46:22.619425+0000) 2024-03-04T18:46:23.618 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:46:23.619+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:46:23.619692+0000) 2024-03-04T18:46:24.619 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:46:24.619+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:46:24.619943+0000) 2024-03-04T18:46:25.619 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:46:25.620+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:46:25.620176+0000) 2024-03-04T18:46:26.619 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:46:26.618+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:46:26.620366+0000) 2024-03-04T18:46:27.619 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:46:27.619+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:46:27.620600+0000) 2024-03-04T18:46:28.619 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:46:28.619+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:46:28.620899+0000) 2024-03-04T18:46:29.620 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:46:29.619+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:46:29.621085+0000) 2024-03-04T18:46:30.620 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:46:30.620+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:46:30.621335+0000) 2024-03-04T18:46:31.620 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:46:31.620+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:46:31.621569+0000) 2024-03-04T18:46:32.620 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:46:32.620+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:46:32.621808+0000) 2024-03-04T18:46:33.621 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:46:33.621+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:46:33.622092+0000) 2024-03-04T18:46:34.621 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:46:34.621+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:46:34.622334+0000) 2024-03-04T18:46:35.621 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:46:35.621+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:46:35.622602+0000) 2024-03-04T18:46:36.621 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:46:36.621+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:46:36.622879+0000) 2024-03-04T18:46:37.616 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:46:37.621 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:46:37.622+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:46:37.623130+0000) 2024-03-04T18:46:37.624 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:46:38.622 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:46:38.622+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:46:38.623368+0000) 2024-03-04T18:46:39.622 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:46:39.622+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:46:39.623639+0000) 2024-03-04T18:46:40.622 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:46:40.623+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:46:40.623940+0000) 2024-03-04T18:46:41.623 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:46:41.623+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:46:41.624174+0000) 2024-03-04T18:46:42.623 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:46:42.623+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:46:42.624448+0000) 2024-03-04T18:46:43.623 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:46:43.623+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:46:43.624692+0000) 2024-03-04T18:46:44.623 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:46:44.624+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:46:44.624935+0000) 2024-03-04T18:46:45.624 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:46:45.624+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:46:45.625193+0000) 2024-03-04T18:46:46.624 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:46:46.624+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:46:46.625435+0000) 2024-03-04T18:46:47.624 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:46:47.624+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:46:47.625665+0000) 2024-03-04T18:46:48.624 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:46:48.625+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:46:48.625944+0000) 2024-03-04T18:46:49.625 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:46:49.625+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:46:49.626212+0000) 2024-03-04T18:46:50.625 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:46:50.625+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:46:50.626479+0000) 2024-03-04T18:46:51.625 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:46:51.625+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:46:51.626658+0000) 2024-03-04T18:46:52.625 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:46:52.626+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:46:52.626932+0000) 2024-03-04T18:46:53.626 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:46:53.626+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:46:53.627176+0000) 2024-03-04T18:46:54.626 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:46:54.626+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:46:54.627433+0000) 2024-03-04T18:46:55.626 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:46:55.627+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:46:55.627669+0000) 2024-03-04T18:46:56.626 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:46:56.627+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:46:56.627949+0000) 2024-03-04T18:46:57.627 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:46:57.627+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:46:57.628131+0000) 2024-03-04T18:46:58.628 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:46:58.627+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:46:58.628380+0000) 2024-03-04T18:46:59.627 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:46:59.627+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:46:59.628608+0000) 2024-03-04T18:47:00.627 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:47:00.628+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:47:00.628894+0000) 2024-03-04T18:47:01.628 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:47:01.628+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:47:01.629127+0000) 2024-03-04T18:47:02.628 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:47:02.628+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:47:02.629370+0000) 2024-03-04T18:47:03.628 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:47:03.629+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:47:03.629618+0000) 2024-03-04T18:47:04.629 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:47:04.629+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:47:04.629934+0000) 2024-03-04T18:47:05.629 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:47:05.629+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:47:05.630167+0000) 2024-03-04T18:47:06.629 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:47:06.629+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:47:06.630419+0000) 2024-03-04T18:47:07.629 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:47:07.630+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:47:07.630661+0000) 2024-03-04T18:47:07.746 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:47:07.754 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:47:08.629 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:47:08.630+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:47:08.630937+0000) 2024-03-04T18:47:09.630 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:47:09.630+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:47:09.631170+0000) 2024-03-04T18:47:10.630 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:47:10.630+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:47:10.631413+0000) 2024-03-04T18:47:11.630 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:47:11.631+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:47:11.631644+0000) 2024-03-04T18:47:12.630 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:47:12.631+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:47:12.631933+0000) 2024-03-04T18:47:13.631 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:47:13.631+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:47:13.632176+0000) 2024-03-04T18:47:14.631 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:47:14.632+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:47:14.632413+0000) 2024-03-04T18:47:15.631 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:47:15.632+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:47:15.632650+0000) 2024-03-04T18:47:16.631 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:47:16.632+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:47:16.632938+0000) 2024-03-04T18:47:17.631 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:47:17.632+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:47:17.633167+0000) 2024-03-04T18:47:18.632 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:47:18.633+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:47:18.633412+0000) 2024-03-04T18:47:19.632 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:47:19.633+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:47:19.633666+0000) 2024-03-04T18:47:20.632 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:47:20.633+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:47:20.633949+0000) 2024-03-04T18:47:21.633 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:47:21.633+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:47:21.634173+0000) 2024-03-04T18:47:22.633 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:47:22.633+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:47:22.634415+0000) 2024-03-04T18:47:23.633 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:47:23.633+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:47:23.634602+0000) 2024-03-04T18:47:24.633 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:47:24.634+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:47:24.634863+0000) 2024-03-04T18:47:25.634 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:47:25.634+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:47:25.635098+0000) 2024-03-04T18:47:26.634 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:47:26.635+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:47:26.635339+0000) 2024-03-04T18:47:27.634 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:47:27.635+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:47:27.635578+0000) 2024-03-04T18:47:28.635 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:47:28.634+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:47:28.635927+0000) 2024-03-04T18:47:29.635 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:47:29.635+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:47:29.636163+0000) 2024-03-04T18:47:30.635 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:47:30.635+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:47:30.636413+0000) 2024-03-04T18:47:31.635 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:47:31.635+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:47:31.636648+0000) 2024-03-04T18:47:32.635 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:47:32.635+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:47:32.636928+0000) 2024-03-04T18:47:33.636 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:47:33.636+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:47:33.637170+0000) 2024-03-04T18:47:34.636 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:47:34.636+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:47:34.637410+0000) 2024-03-04T18:47:35.636 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:47:35.636+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:47:35.637644+0000) 2024-03-04T18:47:36.636 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:47:36.636+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:47:36.637935+0000) 2024-03-04T18:47:37.637 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:47:37.636+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:47:37.638162+0000) 2024-03-04T18:47:37.968 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:47:37.976 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:47:38.637 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:47:38.637+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:47:38.638430+0000) 2024-03-04T18:47:39.637 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:47:39.637+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:47:39.638667+0000) 2024-03-04T18:47:40.637 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:47:40.638+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:47:40.638946+0000) 2024-03-04T18:47:41.638 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:47:41.638+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:47:41.639184+0000) 2024-03-04T18:47:42.638 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:47:42.638+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:47:42.639420+0000) 2024-03-04T18:47:43.638 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:47:43.638+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:47:43.639671+0000) 2024-03-04T18:47:44.639 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:47:44.639+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:47:44.639907+0000) 2024-03-04T18:47:45.639 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:47:45.639+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:47:45.640144+0000) 2024-03-04T18:47:46.639 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:47:46.639+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:47:46.640390+0000) 2024-03-04T18:47:47.639 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:47:47.639+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:47:47.640626+0000) 2024-03-04T18:47:48.640 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:47:48.640+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:47:48.640915+0000) 2024-03-04T18:47:49.640 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:47:49.640+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:47:49.641169+0000) 2024-03-04T18:47:50.640 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:47:50.640+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:47:50.641374+0000) 2024-03-04T18:47:51.640 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:47:51.640+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:47:51.641609+0000) 2024-03-04T18:47:52.640 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:47:52.641+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:47:52.641894+0000) 2024-03-04T18:47:53.641 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:47:53.641+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:47:53.642138+0000) 2024-03-04T18:47:54.641 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:47:54.641+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:47:54.642384+0000) 2024-03-04T18:47:55.641 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:47:55.641+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:47:55.642645+0000) 2024-03-04T18:47:56.642 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:47:56.642+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:47:56.642907+0000) 2024-03-04T18:47:57.642 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:47:57.642+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:47:57.643135+0000) 2024-03-04T18:47:58.642 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:47:58.642+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:47:58.643377+0000) 2024-03-04T18:47:59.642 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:47:59.643+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:47:59.643612+0000) 2024-03-04T18:48:00.642 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:48:00.643+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:48:00.643905+0000) 2024-03-04T18:48:01.643 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:48:01.643+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:48:01.644143+0000) 2024-03-04T18:48:02.643 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:48:02.643+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:48:02.644390+0000) 2024-03-04T18:48:03.643 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:48:03.644+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:48:03.644634+0000) 2024-03-04T18:48:04.643 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:48:04.644+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:48:04.644926+0000) 2024-03-04T18:48:05.644 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:48:05.644+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:48:05.645177+0000) 2024-03-04T18:48:06.644 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:48:06.644+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:48:06.645422+0000) 2024-03-04T18:48:07.644 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:48:07.645+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:48:07.645653+0000) 2024-03-04T18:48:08.182 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:48:08.190 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:48:08.644 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:48:08.645+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:48:08.645939+0000) 2024-03-04T18:48:09.645 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:48:09.645+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:48:09.646201+0000) 2024-03-04T18:48:10.645 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:48:10.646+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:48:10.646448+0000) 2024-03-04T18:48:11.646 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:48:11.646+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:48:11.646685+0000) 2024-03-04T18:48:12.645 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:48:12.646+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:48:12.646953+0000) 2024-03-04T18:48:13.646 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:48:13.646+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:48:13.647194+0000) 2024-03-04T18:48:14.646 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:48:14.647+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:48:14.647440+0000) 2024-03-04T18:48:15.646 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:48:15.647+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:48:15.647678+0000) 2024-03-04T18:48:16.647 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:48:16.647+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:48:16.647988+0000) 2024-03-04T18:48:17.647 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:48:17.647+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:48:17.648234+0000) 2024-03-04T18:48:18.677 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:48:18.648+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:48:18.648477+0000) 2024-03-04T18:48:19.647 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:48:19.648+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:48:19.648729+0000) 2024-03-04T18:48:20.648 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:48:20.648+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:48:20.649004+0000) 2024-03-04T18:48:21.648 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:48:21.648+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:48:21.649228+0000) 2024-03-04T18:48:22.648 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:48:22.649+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:48:22.649469+0000) 2024-03-04T18:48:23.648 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:48:23.649+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:48:23.649715+0000) 2024-03-04T18:48:24.649 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:48:24.649+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:48:24.650023+0000) 2024-03-04T18:48:25.649 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:48:25.650+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:48:25.650308+0000) 2024-03-04T18:48:26.649 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:48:26.650+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:48:26.650590+0000) 2024-03-04T18:48:27.650 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:48:27.650+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:48:27.650934+0000) 2024-03-04T18:48:28.650 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:48:28.651+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:48:28.651227+0000) 2024-03-04T18:48:29.650 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:48:29.651+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:48:29.651544+0000) 2024-03-04T18:48:30.651 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:48:30.650+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:48:30.651899+0000) 2024-03-04T18:48:31.651 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:48:31.651+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:48:31.652187+0000) 2024-03-04T18:48:32.651 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:48:32.651+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:48:32.652480+0000) 2024-03-04T18:48:33.651 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:48:33.651+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:48:33.652783+0000) 2024-03-04T18:48:34.652 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:48:34.652+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:48:34.653100+0000) 2024-03-04T18:48:35.652 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:48:35.652+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:48:35.653400+0000) 2024-03-04T18:48:36.652 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:48:36.652+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:48:36.653694+0000) 2024-03-04T18:48:37.653 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:48:37.652+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:48:37.653983+0000) 2024-03-04T18:48:38.320 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:48:38.329 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:48:38.653 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:48:38.653+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:48:38.654292+0000) 2024-03-04T18:48:39.653 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:48:39.653+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:48:39.654582+0000) 2024-03-04T18:48:40.653 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:48:40.653+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:48:40.654923+0000) 2024-03-04T18:48:41.654 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:48:41.654+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:48:41.655170+0000) 2024-03-04T18:48:42.654 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:48:42.654+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:48:42.655415+0000) 2024-03-04T18:48:43.654 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:48:43.654+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:48:43.655686+0000) 2024-03-04T18:48:44.654 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:48:44.655+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:48:44.655978+0000) 2024-03-04T18:48:45.655 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:48:45.655+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:48:45.656294+0000) 2024-03-04T18:48:46.655 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:48:46.655+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:48:46.656609+0000) 2024-03-04T18:48:47.655 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:48:47.656+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:48:47.656936+0000) 2024-03-04T18:48:48.656 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:48:48.656+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:48:48.657223+0000) 2024-03-04T18:48:49.656 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:48:49.656+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:48:49.657507+0000) 2024-03-04T18:48:50.656 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:48:50.656+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:48:50.657796+0000) 2024-03-04T18:48:51.657 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:48:51.657+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:48:51.658132+0000) 2024-03-04T18:48:52.657 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:48:52.657+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:48:52.658421+0000) 2024-03-04T18:48:53.657 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:48:53.657+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:48:53.658723+0000) 2024-03-04T18:48:54.658 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:48:54.658+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:48:54.659020+0000) 2024-03-04T18:48:55.658 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:48:55.658+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:48:55.659317+0000) 2024-03-04T18:48:56.658 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:48:56.658+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:48:56.659622+0000) 2024-03-04T18:48:57.658 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:48:57.659+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:48:57.659971+0000) 2024-03-04T18:48:58.659 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:48:58.659+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:48:58.660257+0000) 2024-03-04T18:48:59.659 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:48:59.659+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:48:59.660551+0000) 2024-03-04T18:49:00.659 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:49:00.660+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:49:00.660879+0000) 2024-03-04T18:49:01.660 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:49:01.660+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:49:01.661165+0000) 2024-03-04T18:49:02.660 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:49:02.660+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:49:02.661461+0000) 2024-03-04T18:49:03.661 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:49:03.661+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:49:03.661754+0000) 2024-03-04T18:49:04.661 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:49:04.661+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:49:04.662084+0000) 2024-03-04T18:49:05.661 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:49:05.661+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:49:05.662371+0000) 2024-03-04T18:49:06.661 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:49:06.662+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:49:06.662653+0000) 2024-03-04T18:49:07.662 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:49:07.662+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:49:07.662929+0000) 2024-03-04T18:49:08.524 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:49:08.536 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:49:08.662 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:49:08.662+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:49:08.663200+0000) 2024-03-04T18:49:09.662 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:49:09.662+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:49:09.663419+0000) 2024-03-04T18:49:10.663 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:49:10.663+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:49:10.663718+0000) 2024-03-04T18:49:11.663 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:49:11.663+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:49:11.664011+0000) 2024-03-04T18:49:12.663 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:49:12.663+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:49:12.664331+0000) 2024-03-04T18:49:13.663 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:49:13.664+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:49:13.664636+0000) 2024-03-04T18:49:14.663 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:49:14.664+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:49:14.664931+0000) 2024-03-04T18:49:15.664 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:49:15.664+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:49:15.665170+0000) 2024-03-04T18:49:16.664 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:49:16.665+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:49:16.665412+0000) 2024-03-04T18:49:17.665 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:49:17.665+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:49:17.665643+0000) 2024-03-04T18:49:18.665 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:49:18.665+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:49:18.665932+0000) 2024-03-04T18:49:19.665 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:49:19.665+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:49:19.666162+0000) 2024-03-04T18:49:20.665 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:49:20.666+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:49:20.666408+0000) 2024-03-04T18:49:21.665 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:49:21.665+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:49:21.666655+0000) 2024-03-04T18:49:22.666 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:49:22.666+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:49:22.666928+0000) 2024-03-04T18:49:23.666 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:49:23.666+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:49:23.667169+0000) 2024-03-04T18:49:24.666 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:49:24.667+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:49:24.667408+0000) 2024-03-04T18:49:25.666 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:49:25.667+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:49:25.667685+0000) 2024-03-04T18:49:26.667 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:49:26.667+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:49:26.667986+0000) 2024-03-04T18:49:27.667 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:49:27.668+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:49:27.668273+0000) 2024-03-04T18:49:28.667 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:49:28.668+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:49:28.668563+0000) 2024-03-04T18:49:29.667 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:49:29.668+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:49:29.668886+0000) 2024-03-04T18:49:30.668 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:49:30.668+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:49:30.669178+0000) 2024-03-04T18:49:31.668 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:49:31.669+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:49:31.669454+0000) 2024-03-04T18:49:32.668 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:49:32.669+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:49:32.669749+0000) 2024-03-04T18:49:33.669 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:49:33.668+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:49:33.670048+0000) 2024-03-04T18:49:34.669 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:49:34.669+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:49:34.670321+0000) 2024-03-04T18:49:35.669 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:49:35.669+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:49:35.670603+0000) 2024-03-04T18:49:36.669 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:49:36.669+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:49:36.670929+0000) 2024-03-04T18:49:37.670 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:49:37.670+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:49:37.671209+0000) 2024-03-04T18:49:38.670 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:49:38.670+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:49:38.671509+0000) 2024-03-04T18:49:38.730 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:49:38.739 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:49:39.670 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:49:39.670+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:49:39.671817+0000) 2024-03-04T18:49:40.671 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:49:40.671+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:49:40.672141+0000) 2024-03-04T18:49:41.671 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:49:41.671+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:49:41.672424+0000) 2024-03-04T18:49:42.671 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:49:42.671+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:49:42.672725+0000) 2024-03-04T18:49:43.672 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:49:43.672+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:49:43.673025+0000) 2024-03-04T18:49:44.672 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:49:44.672+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:49:44.673300+0000) 2024-03-04T18:49:45.672 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:49:45.672+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:49:45.673587+0000) 2024-03-04T18:49:46.672 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:49:46.673+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:49:46.673925+0000) 2024-03-04T18:49:47.673 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:49:47.673+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:49:47.674213+0000) 2024-03-04T18:49:48.673 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:49:48.673+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:49:48.674507+0000) 2024-03-04T18:49:49.673 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:49:49.673+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:49:49.674777+0000) 2024-03-04T18:49:50.674 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:49:50.674+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:49:50.674988+0000) 2024-03-04T18:49:51.674 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:49:51.674+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:49:51.675266+0000) 2024-03-04T18:49:52.674 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:49:52.674+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:49:52.675610+0000) 2024-03-04T18:49:53.675 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:49:53.675+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:49:53.675964+0000) 2024-03-04T18:49:54.675 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:49:54.675+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:49:54.676254+0000) 2024-03-04T18:49:55.675 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:49:55.675+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:49:55.676486+0000) 2024-03-04T18:49:56.676 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:49:56.676+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:49:56.676787+0000) 2024-03-04T18:49:57.676 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:49:57.676+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:49:57.677156+0000) 2024-03-04T18:49:58.676 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:49:58.676+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:49:58.677435+0000) 2024-03-04T18:49:59.677 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:49:59.676+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:49:59.677713+0000) 2024-03-04T18:50:00.677 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:50:00.677+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:50:00.678010+0000) 2024-03-04T18:50:01.677 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:50:01.677+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:50:01.678249+0000) 2024-03-04T18:50:02.677 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:50:02.677+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:50:02.678494+0000) 2024-03-04T18:50:03.677 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:50:03.678+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:50:03.678783+0000) 2024-03-04T18:50:04.678 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:50:04.678+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:50:04.679139+0000) 2024-03-04T18:50:05.678 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:50:05.678+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:50:05.679446+0000) 2024-03-04T18:50:06.678 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:50:06.678+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:50:06.679762+0000) 2024-03-04T18:50:07.679 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:50:07.679+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:50:07.680044+0000) 2024-03-04T18:50:08.679 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:50:08.679+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:50:08.680328+0000) 2024-03-04T18:50:08.876 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:50:08.885 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:50:09.679 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:50:09.680+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:50:09.680607+0000) 2024-03-04T18:50:10.680 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:50:10.680+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:50:10.680926+0000) 2024-03-04T18:50:11.680 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:50:11.680+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:50:11.681214+0000) 2024-03-04T18:50:12.680 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:50:12.680+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:50:12.681502+0000) 2024-03-04T18:50:13.681 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:50:13.681+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:50:13.681805+0000) 2024-03-04T18:50:14.681 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:50:14.681+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:50:14.682084+0000) 2024-03-04T18:50:15.681 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:50:15.681+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:50:15.682341+0000) 2024-03-04T18:50:16.681 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:50:16.682+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:50:16.682585+0000) 2024-03-04T18:50:17.682 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:50:17.682+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:50:17.682817+0000) 2024-03-04T18:50:18.682 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:50:18.682+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:50:18.683045+0000) 2024-03-04T18:50:19.682 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:50:19.682+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:50:19.683279+0000) 2024-03-04T18:50:20.682 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:50:20.683+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:50:20.683522+0000) 2024-03-04T18:50:21.683 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:50:21.683+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:50:21.683755+0000) 2024-03-04T18:50:22.683 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:50:22.683+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:50:22.683986+0000) 2024-03-04T18:50:23.683 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:50:23.683+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:50:23.684232+0000) 2024-03-04T18:50:24.683 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:50:24.684+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:50:24.684468+0000) 2024-03-04T18:50:25.683 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:50:25.684+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:50:25.684707+0000) 2024-03-04T18:50:26.684 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:50:26.684+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:50:26.684958+0000) 2024-03-04T18:50:27.684 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:50:27.684+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:50:27.685193+0000) 2024-03-04T18:50:28.684 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:50:28.685+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:50:28.685433+0000) 2024-03-04T18:50:29.684 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:50:29.685+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:50:29.685665+0000) 2024-03-04T18:50:30.685 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:50:30.685+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:50:30.685939+0000) 2024-03-04T18:50:31.685 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:50:31.685+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:50:31.686171+0000) 2024-03-04T18:50:32.685 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:50:32.686+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:50:32.686409+0000) 2024-03-04T18:50:33.685 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:50:33.686+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:50:33.686677+0000) 2024-03-04T18:50:34.686 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:50:34.686+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:50:34.686936+0000) 2024-03-04T18:50:35.686 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:50:35.686+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:50:35.687177+0000) 2024-03-04T18:50:36.686 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:50:36.686+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:50:36.687415+0000) 2024-03-04T18:50:37.687 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:50:37.686+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:50:37.687646+0000) 2024-03-04T18:50:38.687 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:50:38.686+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:50:38.687935+0000) 2024-03-04T18:50:39.024 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:50:39.032 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:50:39.687 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:50:39.687+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:50:39.688173+0000) 2024-03-04T18:50:40.687 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:50:40.687+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:50:40.688420+0000) 2024-03-04T18:50:41.687 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:50:41.687+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:50:41.688660+0000) 2024-03-04T18:50:42.688 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:50:42.687+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:50:42.688941+0000) 2024-03-04T18:50:43.688 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:50:43.688+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:50:43.689185+0000) 2024-03-04T18:50:44.688 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:50:44.688+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:50:44.689416+0000) 2024-03-04T18:50:45.688 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:50:45.688+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:50:45.689646+0000) 2024-03-04T18:50:46.689 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:50:46.689+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:50:46.689937+0000) 2024-03-04T18:50:47.689 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:50:47.689+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:50:47.690175+0000) 2024-03-04T18:50:48.689 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:50:48.689+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:50:48.690438+0000) 2024-03-04T18:50:49.690 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:50:49.689+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:50:49.690672+0000) 2024-03-04T18:50:50.689 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:50:50.690+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:50:50.690953+0000) 2024-03-04T18:50:51.690 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:50:51.690+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:50:51.691187+0000) 2024-03-04T18:50:52.690 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:50:52.690+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:50:52.691425+0000) 2024-03-04T18:50:53.690 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:50:53.690+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:50:53.691673+0000) 2024-03-04T18:50:54.691 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:50:54.691+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:50:54.691960+0000) 2024-03-04T18:50:55.691 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:50:55.691+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:50:55.692191+0000) 2024-03-04T18:50:56.691 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:50:56.691+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:50:56.692433+0000) 2024-03-04T18:50:57.692 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:50:57.691+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:50:57.692665+0000) 2024-03-04T18:50:58.692 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:50:58.692+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:50:58.692945+0000) 2024-03-04T18:50:59.692 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:50:59.692+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:50:59.693200+0000) 2024-03-04T18:51:00.692 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:51:00.692+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:51:00.693440+0000) 2024-03-04T18:51:01.693 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:51:01.692+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:51:01.693673+0000) 2024-03-04T18:51:02.693 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:51:02.693+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:51:02.693969+0000) 2024-03-04T18:51:03.693 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:51:03.693+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:51:03.694232+0000) 2024-03-04T18:51:04.693 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:51:04.693+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:51:04.694484+0000) 2024-03-04T18:51:05.694 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:51:05.694+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:51:05.694735+0000) 2024-03-04T18:51:06.694 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:51:06.694+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:51:06.694976+0000) 2024-03-04T18:51:07.694 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:51:07.694+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:51:07.695215+0000) 2024-03-04T18:51:08.694 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:51:08.694+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:51:08.695457+0000) 2024-03-04T18:51:09.170 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:51:09.179 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:51:09.694 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:51:09.695+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:51:09.695689+0000) 2024-03-04T18:51:10.695 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:51:10.695+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:51:10.695930+0000) 2024-03-04T18:51:11.695 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:51:11.695+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:51:11.696165+0000) 2024-03-04T18:51:12.695 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:51:12.695+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:51:12.696412+0000) 2024-03-04T18:51:13.695 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:51:13.696+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:51:13.696655+0000) 2024-03-04T18:51:14.696 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:51:14.696+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:51:14.696936+0000) 2024-03-04T18:51:15.696 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:51:15.696+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:51:15.697174+0000) 2024-03-04T18:51:16.696 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:51:16.696+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:51:16.697415+0000) 2024-03-04T18:51:17.696 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:51:17.697+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:51:17.697651+0000) 2024-03-04T18:51:18.696 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:51:18.697+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:51:18.697930+0000) 2024-03-04T18:51:19.697 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:51:19.697+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:51:19.698163+0000) 2024-03-04T18:51:20.697 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:51:20.698+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:51:20.698402+0000) 2024-03-04T18:51:21.698 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:51:21.698+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:51:21.698643+0000) 2024-03-04T18:51:22.698 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:51:22.698+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:51:22.698931+0000) 2024-03-04T18:51:23.698 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:51:23.698+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:51:23.699178+0000) 2024-03-04T18:51:24.698 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:51:24.699+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:51:24.699412+0000) 2024-03-04T18:51:25.698 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:51:25.699+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:51:25.699649+0000) 2024-03-04T18:51:26.699 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:51:26.699+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:51:26.699934+0000) 2024-03-04T18:51:27.699 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:51:27.699+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:51:27.700172+0000) 2024-03-04T18:51:28.699 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:51:28.700+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:51:28.700421+0000) 2024-03-04T18:51:29.699 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:51:29.700+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:51:29.700651+0000) 2024-03-04T18:51:30.700 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:51:30.700+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:51:30.700959+0000) 2024-03-04T18:51:31.700 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:51:31.700+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:51:31.701210+0000) 2024-03-04T18:51:32.700 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:51:32.701+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:51:32.701449+0000) 2024-03-04T18:51:33.700 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:51:33.701+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:51:33.701705+0000) 2024-03-04T18:51:34.700 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:51:34.701+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:51:34.701961+0000) 2024-03-04T18:51:35.701 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:51:35.702+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:51:35.702227+0000) 2024-03-04T18:51:36.701 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:51:36.702+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:51:36.702471+0000) 2024-03-04T18:51:37.701 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:51:37.701+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:51:37.702707+0000) 2024-03-04T18:51:38.702 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:51:38.701+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:51:38.702951+0000) 2024-03-04T18:51:39.316 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:51:39.327 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:51:39.702 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:51:39.702+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:51:39.703235+0000) 2024-03-04T18:51:40.702 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:51:40.702+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:51:40.703482+0000) 2024-03-04T18:51:41.702 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:51:41.702+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:51:41.703723+0000) 2024-03-04T18:51:42.703 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:51:42.702+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:51:42.703978+0000) 2024-03-04T18:51:43.703 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:51:43.703+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:51:43.704226+0000) 2024-03-04T18:51:44.703 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:51:44.703+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:51:44.704466+0000) 2024-03-04T18:51:45.703 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:51:45.703+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:51:45.704701+0000) 2024-03-04T18:51:46.703 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:51:46.703+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:51:46.704941+0000) 2024-03-04T18:51:47.704 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:51:47.704+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:51:47.705151+0000) 2024-03-04T18:51:48.704 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:51:48.704+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:51:48.705389+0000) 2024-03-04T18:51:49.704 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:51:49.704+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:51:49.705624+0000) 2024-03-04T18:51:50.704 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:51:50.705+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:51:50.705929+0000) 2024-03-04T18:51:51.705 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:51:51.705+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:51:51.706164+0000) 2024-03-04T18:51:52.705 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:51:52.705+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:51:52.706407+0000) 2024-03-04T18:51:53.705 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:51:53.705+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:51:53.706652+0000) 2024-03-04T18:51:54.705 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:51:54.706+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:51:54.706933+0000) 2024-03-04T18:51:55.706 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:51:55.706+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:51:55.707177+0000) 2024-03-04T18:51:56.706 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:51:56.706+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:51:56.707417+0000) 2024-03-04T18:51:57.706 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:51:57.706+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:51:57.707674+0000) 2024-03-04T18:51:58.707 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:51:58.707+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:51:58.707943+0000) 2024-03-04T18:51:59.707 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:51:59.707+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:51:59.708120+0000) 2024-03-04T18:52:00.707 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:52:00.707+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:52:00.708387+0000) 2024-03-04T18:52:01.707 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:52:01.707+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:52:01.708623+0000) 2024-03-04T18:52:02.708 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:52:02.708+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:52:02.708911+0000) 2024-03-04T18:52:03.708 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:52:03.708+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:52:03.709160+0000) 2024-03-04T18:52:04.708 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:52:04.708+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:52:04.709413+0000) 2024-03-04T18:52:05.708 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:52:05.708+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:52:05.709667+0000) 2024-03-04T18:52:06.708 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:52:06.709+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:52:06.709942+0000) 2024-03-04T18:52:07.709 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:52:07.709+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:52:07.710171+0000) 2024-03-04T18:52:08.709 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:52:08.709+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:52:08.710372+0000) 2024-03-04T18:52:09.647 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:52:09.656 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:52:09.709 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:52:09.709+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:52:09.710617+0000) 2024-03-04T18:52:10.709 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:52:10.710+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:52:10.710926+0000) 2024-03-04T18:52:11.710 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:52:11.710+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:52:11.711182+0000) 2024-03-04T18:52:12.710 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:52:12.710+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:52:12.711419+0000) 2024-03-04T18:52:13.710 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:52:13.711+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:52:13.711672+0000) 2024-03-04T18:52:14.710 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:52:14.711+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:52:14.711938+0000) 2024-03-04T18:52:15.711 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:52:15.711+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:52:15.712178+0000) 2024-03-04T18:52:16.711 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:52:16.711+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:52:16.712436+0000) 2024-03-04T18:52:17.712 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:52:17.712+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:52:17.712687+0000) 2024-03-04T18:52:18.712 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:52:18.712+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:52:18.712945+0000) 2024-03-04T18:52:19.712 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:52:19.712+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:52:19.713194+0000) 2024-03-04T18:52:20.712 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:52:20.713+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:52:20.713476+0000) 2024-03-04T18:52:21.713 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:52:21.713+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:52:21.713729+0000) 2024-03-04T18:52:22.712 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:52:22.713+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:52:22.713981+0000) 2024-03-04T18:52:23.713 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:52:23.713+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:52:23.714238+0000) 2024-03-04T18:52:24.713 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:52:24.713+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:52:24.714491+0000) 2024-03-04T18:52:25.713 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:52:25.713+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:52:25.714755+0000) 2024-03-04T18:52:26.714 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:52:26.714+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:52:26.715002+0000) 2024-03-04T18:52:27.714 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:52:27.714+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:52:27.715235+0000) 2024-03-04T18:52:28.714 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:52:28.715+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:52:28.715475+0000) 2024-03-04T18:52:29.715 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:52:29.715+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:52:29.715714+0000) 2024-03-04T18:52:30.715 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:52:30.715+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:52:30.715964+0000) 2024-03-04T18:52:31.715 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:52:31.715+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:52:31.716197+0000) 2024-03-04T18:52:32.715 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:52:32.716+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:52:32.716438+0000) 2024-03-04T18:52:33.715 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:52:33.716+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:52:33.716680+0000) 2024-03-04T18:52:34.715 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:52:34.716+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:52:34.716950+0000) 2024-03-04T18:52:35.716 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:52:35.717+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:52:35.717214+0000) 2024-03-04T18:52:36.716 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:52:36.717+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:52:36.717466+0000) 2024-03-04T18:52:37.717 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:52:37.717+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:52:37.717699+0000) 2024-03-04T18:52:38.717 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:52:38.717+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:52:38.717938+0000) 2024-03-04T18:52:39.717 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:52:39.718+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:52:39.718195+0000) 2024-03-04T18:52:39.810 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:52:39.819 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:52:40.717 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:52:40.717+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:52:40.718450+0000) 2024-03-04T18:52:41.718 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:52:41.717+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:52:41.718689+0000) 2024-03-04T18:52:42.717 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:52:42.717+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:52:42.718967+0000) 2024-03-04T18:52:43.718 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:52:43.718+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:52:43.719213+0000) 2024-03-04T18:52:44.718 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:52:44.718+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:52:44.719457+0000) 2024-03-04T18:52:45.718 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:52:45.718+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:52:45.719700+0000) 2024-03-04T18:52:46.718 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:52:46.718+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:52:46.719945+0000) 2024-03-04T18:52:47.719 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:52:47.719+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:52:47.720182+0000) 2024-03-04T18:52:48.719 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:52:48.719+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:52:48.720434+0000) 2024-03-04T18:52:49.719 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:52:49.719+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:52:49.720663+0000) 2024-03-04T18:52:50.720 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:52:50.719+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:52:50.720953+0000) 2024-03-04T18:52:51.720 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:52:51.720+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:52:51.721200+0000) 2024-03-04T18:52:52.720 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:52:52.720+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:52:52.721435+0000) 2024-03-04T18:52:53.720 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:52:53.720+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:52:53.721679+0000) 2024-03-04T18:52:54.721 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:52:54.721+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:52:54.721921+0000) 2024-03-04T18:52:55.721 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:52:55.721+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:52:55.722157+0000) 2024-03-04T18:52:56.721 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:52:56.721+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:52:56.722413+0000) 2024-03-04T18:52:57.721 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:52:57.721+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:52:57.722636+0000) 2024-03-04T18:52:58.721 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:52:58.722+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:52:58.722926+0000) 2024-03-04T18:52:59.722 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:52:59.722+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:52:59.723167+0000) 2024-03-04T18:53:00.722 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:53:00.722+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:53:00.723408+0000) 2024-03-04T18:53:01.722 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:53:01.722+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:53:01.723654+0000) 2024-03-04T18:53:02.724 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:53:02.723+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:53:02.723929+0000) 2024-03-04T18:53:03.723 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:53:03.723+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:53:03.724191+0000) 2024-03-04T18:53:04.723 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:53:04.723+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:53:04.724422+0000) 2024-03-04T18:53:05.723 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:53:05.723+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:53:05.724683+0000) 2024-03-04T18:53:06.723 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:53:06.724+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:53:06.724948+0000) 2024-03-04T18:53:07.724 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:53:07.724+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:53:07.725204+0000) 2024-03-04T18:53:08.724 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:53:08.724+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:53:08.725479+0000) 2024-03-04T18:53:09.724 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:53:09.724+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:53:09.725748+0000) 2024-03-04T18:53:10.015 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:53:10.023 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:53:10.724 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:53:10.725+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:53:10.726058+0000) 2024-03-04T18:53:11.725 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:53:11.725+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:53:11.726322+0000) 2024-03-04T18:53:12.725 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:53:12.725+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:53:12.726530+0000) 2024-03-04T18:53:13.726 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:53:13.726+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:53:13.726807+0000) 2024-03-04T18:53:14.726 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:53:14.726+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:53:14.727101+0000) 2024-03-04T18:53:15.726 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:53:15.726+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:53:15.727358+0000) 2024-03-04T18:53:16.726 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:53:16.727+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:53:16.727624+0000) 2024-03-04T18:53:17.726 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:53:17.727+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:53:17.727879+0000) 2024-03-04T18:53:18.727 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:53:18.727+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:53:18.728142+0000) 2024-03-04T18:53:19.727 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:53:19.727+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:53:19.728395+0000) 2024-03-04T18:53:20.727 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:53:20.728+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:53:20.728664+0000) 2024-03-04T18:53:21.728 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:53:21.728+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:53:21.728955+0000) 2024-03-04T18:53:22.728 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:53:22.728+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:53:22.729212+0000) 2024-03-04T18:53:23.728 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:53:23.729+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:53:23.729474+0000) 2024-03-04T18:53:24.729 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:53:24.729+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:53:24.729749+0000) 2024-03-04T18:53:25.729 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:53:25.729+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:53:25.730000+0000) 2024-03-04T18:53:26.729 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:53:26.729+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:53:26.730274+0000) 2024-03-04T18:53:27.729 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:53:27.730+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:53:27.730522+0000) 2024-03-04T18:53:28.730 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:53:28.729+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:53:28.730790+0000) 2024-03-04T18:53:29.730 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:53:29.730+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:53:29.731084+0000) 2024-03-04T18:53:30.730 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:53:30.730+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:53:30.731290+0000) 2024-03-04T18:53:31.730 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:53:31.730+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:53:31.731536+0000) 2024-03-04T18:53:32.730 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:53:32.731+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:53:32.731793+0000) 2024-03-04T18:53:33.731 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:53:33.731+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:53:33.732114+0000) 2024-03-04T18:53:34.731 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:53:34.732+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:53:34.732381+0000) 2024-03-04T18:53:35.731 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:53:35.732+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:53:35.732629+0000) 2024-03-04T18:53:36.732 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:53:36.732+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:53:36.732933+0000) 2024-03-04T18:53:37.732 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:53:37.733+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:53:37.733202+0000) 2024-03-04T18:53:38.732 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:53:38.733+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:53:38.733474+0000) 2024-03-04T18:53:39.732 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:53:39.733+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:53:39.733709+0000) 2024-03-04T18:53:40.229 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:53:40.237 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:53:40.732 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:53:40.733+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:53:40.733957+0000) 2024-03-04T18:53:41.733 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:53:41.734+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:53:41.734199+0000) 2024-03-04T18:53:42.733 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:53:42.734+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:53:42.734456+0000) 2024-03-04T18:53:43.733 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:53:43.733+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:53:43.734716+0000) 2024-03-04T18:53:44.734 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:53:44.733+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:53:44.735001+0000) 2024-03-04T18:53:45.734 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:53:45.734+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:53:45.735248+0000) 2024-03-04T18:53:46.734 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:53:46.734+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:53:46.735505+0000) 2024-03-04T18:53:47.734 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:53:47.734+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:53:47.735766+0000) 2024-03-04T18:53:48.735 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:53:48.734+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:53:48.736030+0000) 2024-03-04T18:53:49.735 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:53:49.735+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:53:49.736283+0000) 2024-03-04T18:53:50.735 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:53:50.735+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:53:50.736545+0000) 2024-03-04T18:53:51.735 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:53:51.735+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:53:51.736794+0000) 2024-03-04T18:53:52.736 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:53:52.736+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:53:52.737104+0000) 2024-03-04T18:53:53.736 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:53:53.736+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:53:53.737321+0000) 2024-03-04T18:53:54.736 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:53:54.736+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:53:54.737570+0000) 2024-03-04T18:53:55.736 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:53:55.737+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:53:55.737856+0000) 2024-03-04T18:53:56.737 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:53:56.737+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:53:56.738117+0000) 2024-03-04T18:53:57.737 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:53:57.737+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:53:57.738350+0000) 2024-03-04T18:53:58.737 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:53:58.737+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:53:58.738592+0000) 2024-03-04T18:53:59.737 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:53:59.738+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:53:59.738850+0000) 2024-03-04T18:54:00.738 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:54:00.738+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:54:00.739111+0000) 2024-03-04T18:54:01.738 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:54:01.738+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:54:01.739349+0000) 2024-03-04T18:54:02.738 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:54:02.738+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:54:02.739610+0000) 2024-03-04T18:54:03.738 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:54:03.739+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:54:03.739923+0000) 2024-03-04T18:54:04.739 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:54:04.739+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:54:04.740148+0000) 2024-03-04T18:54:05.739 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:54:05.739+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:54:05.740379+0000) 2024-03-04T18:54:06.739 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:54:06.739+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:54:06.740639+0000) 2024-03-04T18:54:07.740 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:54:07.740+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:54:07.740918+0000) 2024-03-04T18:54:08.740 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:54:08.740+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:54:08.741173+0000) 2024-03-04T18:54:09.740 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:54:09.740+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:54:09.741421+0000) 2024-03-04T18:54:10.365 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:54:10.375 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:54:10.740 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:54:10.741+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:54:10.741701+0000) 2024-03-04T18:54:11.741 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:54:11.741+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:54:11.741942+0000) 2024-03-04T18:54:12.741 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:54:12.741+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:54:12.742191+0000) 2024-03-04T18:54:13.741 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:54:13.741+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:54:13.742450+0000) 2024-03-04T18:54:14.741 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:54:14.742+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:54:14.742704+0000) 2024-03-04T18:54:15.741 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:54:15.742+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:54:15.742986+0000) 2024-03-04T18:54:16.742 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:54:16.742+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:54:16.743239+0000) 2024-03-04T18:54:17.742 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:54:17.742+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:54:17.743490+0000) 2024-03-04T18:54:18.742 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:54:18.743+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:54:18.743754+0000) 2024-03-04T18:54:19.742 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:54:19.743+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:54:19.744005+0000) 2024-03-04T18:54:20.743 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:54:20.743+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:54:20.744246+0000) 2024-03-04T18:54:21.743 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:54:21.743+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:54:21.744500+0000) 2024-03-04T18:54:22.743 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:54:22.744+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:54:22.744754+0000) 2024-03-04T18:54:23.743 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:54:23.744+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:54:23.745013+0000) 2024-03-04T18:54:24.744 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:54:24.744+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:54:24.745263+0000) 2024-03-04T18:54:25.744 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:54:25.745+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:54:25.745514+0000) 2024-03-04T18:54:26.744 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:54:26.745+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:54:26.745772+0000) 2024-03-04T18:54:27.745 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:54:27.745+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:54:27.746031+0000) 2024-03-04T18:54:28.745 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:54:28.745+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:54:28.746236+0000) 2024-03-04T18:54:29.745 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:54:29.746+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:54:29.746487+0000) 2024-03-04T18:54:30.745 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:54:30.746+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:54:30.746744+0000) 2024-03-04T18:54:31.745 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:54:31.746+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:54:31.746998+0000) 2024-03-04T18:54:32.746 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:54:32.746+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:54:32.747275+0000) 2024-03-04T18:54:33.746 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:54:33.747+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:54:33.747556+0000) 2024-03-04T18:54:34.746 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:54:34.747+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:54:34.747803+0000) 2024-03-04T18:54:35.747 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:54:35.747+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:54:35.748110+0000) 2024-03-04T18:54:36.747 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:54:36.748+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:54:36.748363+0000) 2024-03-04T18:54:37.747 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:54:37.748+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:54:37.748612+0000) 2024-03-04T18:54:38.748 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:54:38.748+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:54:38.748890+0000) 2024-03-04T18:54:39.748 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:54:39.748+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:54:39.749146+0000) 2024-03-04T18:54:40.571 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:54:40.583 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:54:40.748 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:54:40.749+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:54:40.749422+0000) 2024-03-04T18:54:41.748 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:54:41.749+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:54:41.749672+0000) 2024-03-04T18:54:42.749 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:54:42.749+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:54:42.749929+0000) 2024-03-04T18:54:43.749 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:54:43.750+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:54:43.750195+0000) 2024-03-04T18:54:44.749 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:54:44.750+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:54:44.750446+0000) 2024-03-04T18:54:45.749 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:54:45.749+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:54:45.750696+0000) 2024-03-04T18:54:46.750 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:54:46.749+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:54:46.750952+0000) 2024-03-04T18:54:47.750 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:54:47.750+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:54:47.751204+0000) 2024-03-04T18:54:48.750 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:54:48.750+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:54:48.751459+0000) 2024-03-04T18:54:49.750 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:54:49.750+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:54:49.751712+0000) 2024-03-04T18:54:50.751 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:54:50.750+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:54:50.751966+0000) 2024-03-04T18:54:51.751 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:54:51.751+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:54:51.752217+0000) 2024-03-04T18:54:52.751 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:54:52.751+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:54:52.752470+0000) 2024-03-04T18:54:53.751 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:54:53.751+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:54:53.752735+0000) 2024-03-04T18:54:54.751 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:54:54.752+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:54:54.752987+0000) 2024-03-04T18:54:55.752 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:54:55.752+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:54:55.753235+0000) 2024-03-04T18:54:56.752 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:54:56.752+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:54:56.753546+0000) 2024-03-04T18:54:57.752 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:54:57.752+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:54:57.753800+0000) 2024-03-04T18:54:58.753 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:54:58.753+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:54:58.754091+0000) 2024-03-04T18:54:59.753 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:54:59.753+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:54:59.754297+0000) 2024-03-04T18:55:00.753 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:55:00.753+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:55:00.754541+0000) 2024-03-04T18:55:01.754 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:55:01.754+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:55:01.754807+0000) 2024-03-04T18:55:02.754 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:55:02.754+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:55:02.755141+0000) 2024-03-04T18:55:03.754 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:55:03.754+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:55:03.755402+0000) 2024-03-04T18:55:04.754 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:55:04.754+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:55:04.755628+0000) 2024-03-04T18:55:05.754 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:55:05.755+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:55:05.755931+0000) 2024-03-04T18:55:06.755 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:55:06.755+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:55:06.756168+0000) 2024-03-04T18:55:07.755 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:55:07.755+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:55:07.756334+0000) 2024-03-04T18:55:08.755 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:55:08.755+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:55:08.756571+0000) 2024-03-04T18:55:09.755 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:55:09.756+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:55:09.756818+0000) 2024-03-04T18:55:10.757 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:55:10.756+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:55:10.757114+0000) 2024-03-04T18:55:10.768 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:55:10.776 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:55:11.756 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:55:11.756+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:55:11.757366+0000) 2024-03-04T18:55:12.756 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:55:12.756+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:55:12.757575+0000) 2024-03-04T18:55:13.756 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:55:13.757+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:55:13.757891+0000) 2024-03-04T18:55:14.757 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:55:14.757+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:55:14.758143+0000) 2024-03-04T18:55:15.757 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:55:15.757+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:55:15.758410+0000) 2024-03-04T18:55:16.757 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:55:16.758+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:55:16.758673+0000) 2024-03-04T18:55:17.757 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:55:17.758+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:55:17.758931+0000) 2024-03-04T18:55:18.758 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:55:18.758+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:55:18.759141+0000) 2024-03-04T18:55:19.758 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:55:19.758+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:55:19.759389+0000) 2024-03-04T18:55:20.758 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:55:20.758+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:55:20.759654+0000) 2024-03-04T18:55:21.758 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:55:21.759+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:55:21.759929+0000) 2024-03-04T18:55:22.759 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:55:22.759+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:55:22.760174+0000) 2024-03-04T18:55:23.759 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:55:23.759+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:55:23.760403+0000) 2024-03-04T18:55:24.759 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:55:24.760+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:55:24.760650+0000) 2024-03-04T18:55:25.759 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:55:25.760+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:55:25.760934+0000) 2024-03-04T18:55:26.760 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:55:26.760+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:55:26.761192+0000) 2024-03-04T18:55:27.760 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:55:27.760+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:55:27.761443+0000) 2024-03-04T18:55:28.760 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:55:28.761+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:55:28.761695+0000) 2024-03-04T18:55:29.761 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:55:29.761+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:55:29.761954+0000) 2024-03-04T18:55:30.761 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:55:30.761+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:55:30.762210+0000) 2024-03-04T18:55:31.761 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:55:31.762+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:55:31.762490+0000) 2024-03-04T18:55:32.761 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:55:32.761+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:55:32.762734+0000) 2024-03-04T18:55:33.762 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:55:33.762+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:55:33.763000+0000) 2024-03-04T18:55:34.762 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:55:34.762+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:55:34.763242+0000) 2024-03-04T18:55:35.763 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:55:35.763+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:55:35.763500+0000) 2024-03-04T18:55:36.763 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:55:36.763+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:55:36.763777+0000) 2024-03-04T18:55:37.763 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:55:37.763+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:55:37.764038+0000) 2024-03-04T18:55:38.763 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:55:38.764+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:55:38.764291+0000) 2024-03-04T18:55:39.763 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:55:39.764+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:55:39.764541+0000) 2024-03-04T18:55:40.763 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:55:40.764+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:55:40.764793+0000) 2024-03-04T18:55:40.902 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:55:40.910 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:55:41.763 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:55:41.764+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:55:41.765088+0000) 2024-03-04T18:55:42.764 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:55:42.765+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:55:42.765350+0000) 2024-03-04T18:55:43.764 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:55:43.765+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:55:43.765609+0000) 2024-03-04T18:55:44.764 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:55:44.765+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:55:44.765892+0000) 2024-03-04T18:55:45.765 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:55:45.765+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:55:45.766144+0000) 2024-03-04T18:55:46.765 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:55:46.765+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:55:46.766404+0000) 2024-03-04T18:55:47.765 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:55:47.766+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:55:47.766602+0000) 2024-03-04T18:55:48.765 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:55:48.765+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:55:48.766864+0000) 2024-03-04T18:55:49.766 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:55:49.765+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:55:49.767067+0000) 2024-03-04T18:55:50.766 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:55:50.766+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:55:50.767324+0000) 2024-03-04T18:55:51.766 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:55:51.766+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:55:51.767574+0000) 2024-03-04T18:55:52.766 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:55:52.766+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:55:52.767864+0000) 2024-03-04T18:55:53.767 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:55:53.767+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:55:53.768142+0000) 2024-03-04T18:55:54.767 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:55:54.767+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:55:54.768407+0000) 2024-03-04T18:55:55.767 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:55:55.767+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:55:55.768659+0000) 2024-03-04T18:55:56.767 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:55:56.767+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:55:56.768944+0000) 2024-03-04T18:55:57.768 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:55:57.768+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:55:57.769199+0000) 2024-03-04T18:55:58.768 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:55:58.768+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:55:58.769447+0000) 2024-03-04T18:55:59.768 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:55:59.768+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:55:59.769694+0000) 2024-03-04T18:56:00.769 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:56:00.769+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:56:00.769962+0000) 2024-03-04T18:56:01.769 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:56:01.769+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:56:01.770216+0000) 2024-03-04T18:56:02.769 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:56:02.769+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:56:02.770428+0000) 2024-03-04T18:56:03.769 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:56:03.769+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:56:03.770689+0000) 2024-03-04T18:56:04.769 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:56:04.770+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:56:04.770951+0000) 2024-03-04T18:56:05.770 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:56:05.770+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:56:05.771203+0000) 2024-03-04T18:56:06.770 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:56:06.770+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:56:06.771460+0000) 2024-03-04T18:56:07.770 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:56:07.770+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:56:07.771715+0000) 2024-03-04T18:56:08.770 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:56:08.771+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:56:08.771969+0000) 2024-03-04T18:56:09.771 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:56:09.771+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:56:09.772219+0000) 2024-03-04T18:56:10.771 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:56:10.771+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:56:10.772476+0000) 2024-03-04T18:56:11.087 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:56:11.095 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:56:11.771 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:56:11.771+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:56:11.772727+0000) 2024-03-04T18:56:12.772 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:56:12.772+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:56:12.772987+0000) 2024-03-04T18:56:13.772 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:56:13.772+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:56:13.773256+0000) 2024-03-04T18:56:14.773 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:56:14.772+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:56:14.773500+0000) 2024-03-04T18:56:15.772 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:56:15.773+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:56:15.773749+0000) 2024-03-04T18:56:16.773 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:56:16.773+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:56:16.774009+0000) 2024-03-04T18:56:17.773 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:56:17.773+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:56:17.774260+0000) 2024-03-04T18:56:18.773 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:56:18.773+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:56:18.774511+0000) 2024-03-04T18:56:19.773 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:56:19.774+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:56:19.774764+0000) 2024-03-04T18:56:20.774 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:56:20.774+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:56:20.775009+0000) 2024-03-04T18:56:21.774 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:56:21.774+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:56:21.775257+0000) 2024-03-04T18:56:22.774 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:56:22.774+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:56:22.775504+0000) 2024-03-04T18:56:23.774 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:56:23.775+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:56:23.775745+0000) 2024-03-04T18:56:24.774 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:56:24.775+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:56:24.775972+0000) 2024-03-04T18:56:25.775 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:56:25.775+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:56:25.776203+0000) 2024-03-04T18:56:26.776 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:56:26.775+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:56:26.776476+0000) 2024-03-04T18:56:27.775 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:56:27.776+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:56:27.776714+0000) 2024-03-04T18:56:28.776 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:56:28.776+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:56:28.776947+0000) 2024-03-04T18:56:29.776 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:56:29.776+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:56:29.777188+0000) 2024-03-04T18:56:30.776 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:56:30.776+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:56:30.777466+0000) 2024-03-04T18:56:31.776 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:56:31.777+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:56:31.777737+0000) 2024-03-04T18:56:32.776 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:56:32.777+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:56:32.778049+0000) 2024-03-04T18:56:33.777 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:56:33.777+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:56:33.778311+0000) 2024-03-04T18:56:34.778 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:56:34.778+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:56:34.778557+0000) 2024-03-04T18:56:35.777 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:56:35.778+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:56:35.778811+0000) 2024-03-04T18:56:36.778 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:56:36.778+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:56:36.779124+0000) 2024-03-04T18:56:37.778 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:56:37.779+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:56:37.779374+0000) 2024-03-04T18:56:38.778 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:56:38.779+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:56:38.779627+0000) 2024-03-04T18:56:39.778 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:56:39.779+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:56:39.779941+0000) 2024-03-04T18:56:40.779 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:56:40.779+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:56:40.780207+0000) 2024-03-04T18:56:41.294 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:56:41.302 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:56:41.779 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:56:41.780+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:56:41.780460+0000) 2024-03-04T18:56:42.779 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:56:42.779+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:56:42.780731+0000) 2024-03-04T18:56:43.780 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:56:43.780+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:56:43.780996+0000) 2024-03-04T18:56:44.780 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:56:44.780+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:56:44.781184+0000) 2024-03-04T18:56:45.780 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:56:45.781+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:56:45.781437+0000) 2024-03-04T18:56:46.780 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:56:46.781+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:56:46.781699+0000) 2024-03-04T18:56:47.781 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:56:47.781+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:56:47.781926+0000) 2024-03-04T18:56:48.781 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:56:48.782+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:56:48.782212+0000) 2024-03-04T18:56:49.781 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:56:49.782+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:56:49.782457+0000) 2024-03-04T18:56:50.781 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:56:50.782+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:56:50.782700+0000) 2024-03-04T18:56:51.782 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:56:51.781+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:56:51.782947+0000) 2024-03-04T18:56:52.783 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:56:52.782+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:56:52.783204+0000) 2024-03-04T18:56:53.782 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:56:53.782+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:56:53.783466+0000) 2024-03-04T18:56:54.783 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:56:54.782+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:56:54.783714+0000) 2024-03-04T18:56:55.782 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:56:55.782+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:56:55.783959+0000) 2024-03-04T18:56:56.783 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:56:56.783+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:56:56.784218+0000) 2024-03-04T18:56:57.783 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:56:57.783+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:56:57.784465+0000) 2024-03-04T18:56:58.783 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:56:58.783+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:56:58.784722+0000) 2024-03-04T18:56:59.784 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:56:59.783+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:56:59.784992+0000) 2024-03-04T18:57:00.784 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:57:00.784+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:57:00.785254+0000) 2024-03-04T18:57:01.784 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:57:01.784+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:57:01.785525+0000) 2024-03-04T18:57:02.784 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:57:02.784+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:57:02.785783+0000) 2024-03-04T18:57:03.785 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:57:03.785+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:57:03.786058+0000) 2024-03-04T18:57:04.785 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:57:04.785+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:57:04.786298+0000) 2024-03-04T18:57:05.785 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:57:05.785+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:57:05.786478+0000) 2024-03-04T18:57:06.785 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:57:06.785+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:57:06.786729+0000) 2024-03-04T18:57:07.786 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:57:07.786+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:57:07.786965+0000) 2024-03-04T18:57:08.786 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:57:08.786+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:57:08.787194+0000) 2024-03-04T18:57:09.786 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:57:09.786+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:57:09.787425+0000) 2024-03-04T18:57:10.786 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:57:10.786+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:57:10.787712+0000) 2024-03-04T18:57:11.524 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:57:11.532 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:57:11.786 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:57:11.787+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:57:11.787978+0000) 2024-03-04T18:57:12.788 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:57:12.787+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:57:12.788238+0000) 2024-03-04T18:57:13.787 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:57:13.787+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:57:13.788482+0000) 2024-03-04T18:57:14.787 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:57:14.787+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:57:14.788711+0000) 2024-03-04T18:57:15.788 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:57:15.788+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:57:15.788965+0000) 2024-03-04T18:57:16.788 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:57:16.788+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:57:16.789214+0000) 2024-03-04T18:57:17.788 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:57:17.788+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:57:17.789454+0000) 2024-03-04T18:57:18.788 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:57:18.788+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:57:18.789688+0000) 2024-03-04T18:57:19.788 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:57:19.789+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:57:19.789921+0000) 2024-03-04T18:57:20.789 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:57:20.789+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:57:20.790211+0000) 2024-03-04T18:57:21.789 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:57:21.789+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:57:21.790457+0000) 2024-03-04T18:57:22.789 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:57:22.789+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:57:22.790648+0000) 2024-03-04T18:57:23.789 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:57:23.790+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:57:23.790927+0000) 2024-03-04T18:57:24.790 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:57:24.790+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:57:24.791158+0000) 2024-03-04T18:57:25.790 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:57:25.790+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:57:25.791389+0000) 2024-03-04T18:57:26.790 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:57:26.791+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:57:26.791635+0000) 2024-03-04T18:57:27.790 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:57:27.791+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:57:27.791933+0000) 2024-03-04T18:57:28.791 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:57:28.791+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:57:28.792114+0000) 2024-03-04T18:57:29.791 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:57:29.791+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:57:29.792344+0000) 2024-03-04T18:57:30.791 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:57:30.791+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:57:30.792569+0000) 2024-03-04T18:57:31.791 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:57:31.792+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:57:31.792867+0000) 2024-03-04T18:57:32.792 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:57:32.792+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:57:32.793108+0000) 2024-03-04T18:57:33.792 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:57:33.792+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:57:33.793363+0000) 2024-03-04T18:57:34.792 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:57:34.793+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:57:34.793590+0000) 2024-03-04T18:57:35.793 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:57:35.793+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:57:35.793854+0000) 2024-03-04T18:57:36.793 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:57:36.793+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:57:36.794109+0000) 2024-03-04T18:57:37.793 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:57:37.793+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:57:37.794375+0000) 2024-03-04T18:57:38.793 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:57:38.793+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:57:38.794611+0000) 2024-03-04T18:57:39.793 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:57:39.794+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:57:39.794878+0000) 2024-03-04T18:57:40.794 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:57:40.794+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:57:40.795106+0000) 2024-03-04T18:57:41.666 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:57:41.674 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:57:41.794 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:57:41.795+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:57:41.795363+0000) 2024-03-04T18:57:42.794 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:57:42.795+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:57:42.795621+0000) 2024-03-04T18:57:43.795 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:57:43.795+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:57:43.795912+0000) 2024-03-04T18:57:44.795 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:57:44.795+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:57:44.796139+0000) 2024-03-04T18:57:45.795 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:57:45.796+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:57:45.796371+0000) 2024-03-04T18:57:46.795 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:57:46.796+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:57:46.796607+0000) 2024-03-04T18:57:47.795 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:57:47.796+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:57:47.796891+0000) 2024-03-04T18:57:48.796 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:57:48.796+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:57:48.797129+0000) 2024-03-04T18:57:49.796 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:57:49.797+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:57:49.797365+0000) 2024-03-04T18:57:50.796 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:57:50.797+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:57:50.797594+0000) 2024-03-04T18:57:51.796 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:57:51.796+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:57:51.797773+0000) 2024-03-04T18:57:52.796 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:57:52.797+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:57:52.798057+0000) 2024-03-04T18:57:53.797 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:57:53.798+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:57:53.798301+0000) 2024-03-04T18:57:54.797 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:57:54.798+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:57:54.798535+0000) 2024-03-04T18:57:55.797 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:57:55.797+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:57:55.798770+0000) 2024-03-04T18:57:56.798 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:57:56.797+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:57:56.798999+0000) 2024-03-04T18:57:57.798 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:57:57.797+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:57:57.799242+0000) 2024-03-04T18:57:58.798 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:57:58.798+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:57:58.799478+0000) 2024-03-04T18:57:59.798 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:57:59.798+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:57:59.799716+0000) 2024-03-04T18:58:00.798 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:58:00.798+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:58:00.799944+0000) 2024-03-04T18:58:01.799 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:58:01.799+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:58:01.800196+0000) 2024-03-04T18:58:02.799 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:58:02.799+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:58:02.800463+0000) 2024-03-04T18:58:03.799 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:58:03.799+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:58:03.800709+0000) 2024-03-04T18:58:04.800 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:58:04.799+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:58:04.800933+0000) 2024-03-04T18:58:05.800 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:58:05.800+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:58:05.801163+0000) 2024-03-04T18:58:06.800 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:58:06.800+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:58:06.801396+0000) 2024-03-04T18:58:07.800 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:58:07.800+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:58:07.801641+0000) 2024-03-04T18:58:08.800 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:58:08.801+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:58:08.801920+0000) 2024-03-04T18:58:09.801 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:58:09.801+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:58:09.802192+0000) 2024-03-04T18:58:10.801 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:58:10.801+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:58:10.802442+0000) 2024-03-04T18:58:11.801 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:58:11.801+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:58:11.802696+0000) 2024-03-04T18:58:11.909 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:58:11.917 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:58:12.801 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:58:12.802+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:58:12.802937+0000) 2024-03-04T18:58:13.802 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:58:13.802+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:58:13.803187+0000) 2024-03-04T18:58:14.802 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:58:14.802+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:58:14.803416+0000) 2024-03-04T18:58:15.802 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:58:15.802+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:58:15.803651+0000) 2024-03-04T18:58:16.803 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:58:16.803+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:58:16.803932+0000) 2024-03-04T18:58:17.803 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:58:17.803+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:58:17.804166+0000) 2024-03-04T18:58:18.803 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:58:18.803+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:58:18.804402+0000) 2024-03-04T18:58:19.803 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:58:19.804+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:58:19.804875+0000) 2024-03-04T18:58:20.804 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:58:20.804+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:58:20.805134+0000) 2024-03-04T18:58:21.804 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:58:21.804+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:58:21.805325+0000) 2024-03-04T18:58:22.804 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:58:22.804+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:58:22.805598+0000) 2024-03-04T18:58:23.805 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:58:23.805+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:58:23.805953+0000) 2024-03-04T18:58:24.805 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:58:24.805+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:58:24.806182+0000) 2024-03-04T18:58:25.805 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:58:25.805+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:58:25.806374+0000) 2024-03-04T18:58:26.805 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:58:26.805+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:58:26.806607+0000) 2024-03-04T18:58:27.805 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:58:27.806+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:58:27.806866+0000) 2024-03-04T18:58:28.806 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:58:28.806+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:58:28.807106+0000) 2024-03-04T18:58:29.806 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:58:29.806+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:58:29.807339+0000) 2024-03-04T18:58:30.806 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:58:30.807+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:58:30.807590+0000) 2024-03-04T18:58:31.806 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:58:31.807+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:58:31.807881+0000) 2024-03-04T18:58:32.807 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:58:32.807+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:58:32.808125+0000) 2024-03-04T18:58:33.807 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:58:33.807+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:58:33.808366+0000) 2024-03-04T18:58:34.807 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:58:34.808+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:58:34.808596+0000) 2024-03-04T18:58:35.807 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:58:35.808+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:58:35.808863+0000) 2024-03-04T18:58:36.808 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:58:36.808+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:58:36.809094+0000) 2024-03-04T18:58:37.808 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:58:37.808+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:58:37.809331+0000) 2024-03-04T18:58:38.808 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:58:38.809+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:58:38.809582+0000) 2024-03-04T18:58:39.808 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:58:39.809+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:58:39.809812+0000) 2024-03-04T18:58:40.809 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:58:40.809+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:58:40.810033+0000) 2024-03-04T18:58:41.809 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:58:41.809+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:58:41.810262+0000) 2024-03-04T18:58:42.054 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:58:42.424 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:58:42.809 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:58:42.810+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:58:42.810505+0000) 2024-03-04T18:58:43.809 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:58:43.810+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:58:43.810749+0000) 2024-03-04T18:58:44.810 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:58:44.810+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:58:44.810993+0000) 2024-03-04T18:58:45.810 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:58:45.810+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:58:45.811226+0000) 2024-03-04T18:58:46.810 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:58:46.811+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:58:46.811462+0000) 2024-03-04T18:58:47.810 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:58:47.811+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:58:47.811697+0000) 2024-03-04T18:58:48.810 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:58:48.811+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:58:48.811944+0000) 2024-03-04T18:58:49.811 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:58:49.811+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:58:49.812175+0000) 2024-03-04T18:58:50.811 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:58:50.812+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:58:50.812411+0000) 2024-03-04T18:58:51.811 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:58:51.812+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:58:51.812648+0000) 2024-03-04T18:58:52.811 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:58:52.812+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:58:52.812915+0000) 2024-03-04T18:58:53.812 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:58:53.812+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:58:53.813197+0000) 2024-03-04T18:58:54.812 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:58:54.813+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:58:54.813470+0000) 2024-03-04T18:58:55.812 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:58:55.813+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:58:55.813755+0000) 2024-03-04T18:58:56.813 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:58:56.813+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:58:56.814041+0000) 2024-03-04T18:58:57.813 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:58:57.814+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:58:57.814284+0000) 2024-03-04T18:58:58.813 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:58:58.813+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:58:58.814529+0000) 2024-03-04T18:58:59.813 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:58:59.813+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:58:59.814764+0000) 2024-03-04T18:59:00.814 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:59:00.813+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:59:00.814986+0000) 2024-03-04T18:59:01.815 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:59:01.814+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:59:01.815238+0000) 2024-03-04T18:59:02.814 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:59:02.814+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:59:02.815510+0000) 2024-03-04T18:59:03.814 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:59:03.814+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:59:03.815752+0000) 2024-03-04T18:59:04.815 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:59:04.814+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:59:04.815984+0000) 2024-03-04T18:59:05.815 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:59:05.815+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:59:05.816220+0000) 2024-03-04T18:59:06.815 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:59:06.815+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:59:06.816452+0000) 2024-03-04T18:59:07.815 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:59:07.815+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:59:07.816745+0000) 2024-03-04T18:59:08.815 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:59:08.815+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:59:08.816993+0000) 2024-03-04T18:59:09.816 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:59:09.816+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:59:09.817224+0000) 2024-03-04T18:59:10.816 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:59:10.816+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:59:10.817480+0000) 2024-03-04T18:59:11.816 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:59:11.816+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:59:11.817727+0000) 2024-03-04T18:59:12.559 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:59:12.568 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:59:12.816 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:59:12.817+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:59:12.817935+0000) 2024-03-04T18:59:13.817 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:59:13.817+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:59:13.818142+0000) 2024-03-04T18:59:14.817 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:59:14.817+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:59:14.818388+0000) 2024-03-04T18:59:15.817 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:59:15.817+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:59:15.818628+0000) 2024-03-04T18:59:16.817 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:59:16.818+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:59:16.818896+0000) 2024-03-04T18:59:17.818 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:59:17.818+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:59:17.819128+0000) 2024-03-04T18:59:18.818 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:59:18.818+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:59:18.819372+0000) 2024-03-04T18:59:19.818 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:59:19.818+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:59:19.819602+0000) 2024-03-04T18:59:20.818 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:59:20.819+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:59:20.819883+0000) 2024-03-04T18:59:21.819 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:59:21.819+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:59:21.820122+0000) 2024-03-04T18:59:22.819 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:59:22.819+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:59:22.820362+0000) 2024-03-04T18:59:23.819 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:59:23.819+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:59:23.820640+0000) 2024-03-04T18:59:24.819 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:59:24.820+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:59:24.820950+0000) 2024-03-04T18:59:25.820 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:59:25.820+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:59:25.821240+0000) 2024-03-04T18:59:26.820 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:59:26.820+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:59:26.821523+0000) 2024-03-04T18:59:27.820 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:59:27.821+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:59:27.821810+0000) 2024-03-04T18:59:28.821 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:59:28.821+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:59:28.822166+0000) 2024-03-04T18:59:29.821 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:59:29.821+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:59:29.822370+0000) 2024-03-04T18:59:30.821 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:59:30.822+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:59:30.822602+0000) 2024-03-04T18:59:31.821 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:59:31.822+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:59:31.822908+0000) 2024-03-04T18:59:32.822 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:59:32.822+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:59:32.823165+0000) 2024-03-04T18:59:33.822 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:59:33.822+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:59:33.823410+0000) 2024-03-04T18:59:34.822 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:59:34.823+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:59:34.823646+0000) 2024-03-04T18:59:35.822 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:59:35.823+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:59:35.823917+0000) 2024-03-04T18:59:36.823 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:59:36.823+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:59:36.824167+0000) 2024-03-04T18:59:37.823 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:59:37.823+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:59:37.824400+0000) 2024-03-04T18:59:38.823 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:59:38.824+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:59:38.824680+0000) 2024-03-04T18:59:39.823 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:59:39.824+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:59:39.824974+0000) 2024-03-04T18:59:40.824 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:59:40.824+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:59:40.825243+0000) 2024-03-04T18:59:41.824 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:59:41.825+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:59:41.825533+0000) 2024-03-04T18:59:42.702 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:59:42.710 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T18:59:42.824 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:59:42.824+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:59:42.825783+0000) 2024-03-04T18:59:43.824 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:59:43.825+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:59:43.826084+0000) 2024-03-04T18:59:44.825 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:59:44.825+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:59:44.826267+0000) 2024-03-04T18:59:45.825 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:59:45.826+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:59:45.826510+0000) 2024-03-04T18:59:46.825 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:59:46.826+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:59:46.826746+0000) 2024-03-04T18:59:47.826 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:59:47.826+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:59:47.826989+0000) 2024-03-04T18:59:48.826 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:59:48.826+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:59:48.827218+0000) 2024-03-04T18:59:49.826 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:59:49.827+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:59:49.827455+0000) 2024-03-04T18:59:50.826 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:59:50.827+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:59:50.827686+0000) 2024-03-04T18:59:51.827 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:59:51.827+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:59:51.827947+0000) 2024-03-04T18:59:52.827 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:59:52.827+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:59:52.828188+0000) 2024-03-04T18:59:53.827 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:59:53.828+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:59:53.828454+0000) 2024-03-04T18:59:54.827 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:59:54.828+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:59:54.828686+0000) 2024-03-04T18:59:55.827 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:59:55.828+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:59:55.828930+0000) 2024-03-04T18:59:56.828 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:59:56.828+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:59:56.829157+0000) 2024-03-04T18:59:57.828 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:59:57.829+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:59:57.829427+0000) 2024-03-04T18:59:58.828 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:59:58.829+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:59:58.829669+0000) 2024-03-04T18:59:59.828 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T18:59:59.829+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T17:59:59.829942+0000) 2024-03-04T19:00:00.829 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:00:00.829+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:00:00.830232+0000) 2024-03-04T19:00:01.829 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:00:01.829+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:00:01.830492+0000) 2024-03-04T19:00:02.829 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:00:02.829+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:00:02.830752+0000) 2024-03-04T19:00:03.829 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:00:03.829+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:00:03.831034+0000) 2024-03-04T19:00:04.830 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:00:04.830+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:00:04.831303+0000) 2024-03-04T19:00:05.830 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:00:05.830+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:00:05.831582+0000) 2024-03-04T19:00:06.830 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:00:06.830+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:00:06.831917+0000) 2024-03-04T19:00:07.831 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:00:07.831+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:00:07.832214+0000) 2024-03-04T19:00:08.831 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:00:08.830+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:00:08.832513+0000) 2024-03-04T19:00:09.831 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:00:09.831+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:00:09.832793+0000) 2024-03-04T19:00:10.832 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:00:10.832+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:00:10.833138+0000) 2024-03-04T19:00:11.832 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:00:11.832+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:00:11.833422+0000) 2024-03-04T19:00:12.832 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:00:12.832+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:00:12.833730+0000) 2024-03-04T19:00:12.919 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:00:12.927 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:00:13.833 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:00:13.833+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:00:13.834063+0000) 2024-03-04T19:00:14.833 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:00:14.833+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:00:14.834346+0000) 2024-03-04T19:00:15.833 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:00:15.833+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:00:15.834634+0000) 2024-03-04T19:00:16.835 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:00:16.834+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:00:16.834959+0000) 2024-03-04T19:00:17.834 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:00:17.834+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:00:17.835247+0000) 2024-03-04T19:00:18.835 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:00:18.834+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:00:18.835566+0000) 2024-03-04T19:00:19.834 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:00:19.834+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:00:19.835755+0000) 2024-03-04T19:00:20.835 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:00:20.835+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:00:20.835982+0000) 2024-03-04T19:00:21.835 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:00:21.835+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:00:21.836215+0000) 2024-03-04T19:00:22.835 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:00:22.835+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:00:22.836493+0000) 2024-03-04T19:00:23.835 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:00:23.835+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:00:23.836792+0000) 2024-03-04T19:00:24.836 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:00:24.836+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:00:24.837110+0000) 2024-03-04T19:00:25.836 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:00:25.836+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:00:25.837389+0000) 2024-03-04T19:00:26.836 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:00:26.836+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:00:26.837672+0000) 2024-03-04T19:00:27.836 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:00:27.837+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:00:27.837967+0000) 2024-03-04T19:00:28.837 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:00:28.837+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:00:28.838274+0000) 2024-03-04T19:00:29.838 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:00:29.837+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:00:29.838559+0000) 2024-03-04T19:00:30.837 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:00:30.838+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:00:30.838877+0000) 2024-03-04T19:00:31.838 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:00:31.838+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:00:31.839168+0000) 2024-03-04T19:00:32.839 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:00:32.838+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:00:32.839466+0000) 2024-03-04T19:00:33.839 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:00:33.839+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:00:33.839778+0000) 2024-03-04T19:00:34.841 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:00:34.839+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:00:34.840054+0000) 2024-03-04T19:00:35.839 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:00:35.839+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:00:35.840342+0000) 2024-03-04T19:00:36.840 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:00:36.840+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:00:36.840639+0000) 2024-03-04T19:00:37.840 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:00:37.840+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:00:37.841004+0000) 2024-03-04T19:00:38.840 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:00:38.840+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:00:38.841290+0000) 2024-03-04T19:00:39.840 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:00:39.841+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:00:39.841569+0000) 2024-03-04T19:00:40.841 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:00:40.841+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:00:40.841927+0000) 2024-03-04T19:00:41.841 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:00:41.841+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:00:41.842203+0000) 2024-03-04T19:00:42.841 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:00:42.841+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:00:42.842504+0000) 2024-03-04T19:00:43.141 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:00:43.149 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:00:43.841 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:00:43.842+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:00:43.842893+0000) 2024-03-04T19:00:44.842 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:00:44.842+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:00:44.843186+0000) 2024-03-04T19:00:45.842 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:00:45.843+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:00:45.843466+0000) 2024-03-04T19:00:46.842 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:00:46.843+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:00:46.843780+0000) 2024-03-04T19:00:47.843 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:00:47.843+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:00:47.844108+0000) 2024-03-04T19:00:48.843 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:00:48.843+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:00:48.844348+0000) 2024-03-04T19:00:49.843 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:00:49.844+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:00:49.844629+0000) 2024-03-04T19:00:50.843 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:00:50.844+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:00:50.844963+0000) 2024-03-04T19:00:51.845 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:00:51.844+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:00:51.845251+0000) 2024-03-04T19:00:52.846 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:00:52.845+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:00:52.845566+0000) 2024-03-04T19:00:53.846 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:00:53.845+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:00:53.845924+0000) 2024-03-04T19:00:54.845 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:00:54.845+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:00:54.846215+0000) 2024-03-04T19:00:55.845 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:00:55.846+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:00:55.846491+0000) 2024-03-04T19:00:56.845 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:00:56.845+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:00:56.846719+0000) 2024-03-04T19:00:57.846 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:00:57.846+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:00:57.847007+0000) 2024-03-04T19:00:58.846 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:00:58.847+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:00:58.847296+0000) 2024-03-04T19:00:59.846 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:00:59.847+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:00:59.847572+0000) 2024-03-04T19:01:00.846 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:01:00.847+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:01:00.847923+0000) 2024-03-04T19:01:01.847 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:01:01.848+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:01:01.848215+0000) 2024-03-04T19:01:02.847 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:01:02.848+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:01:02.848497+0000) 2024-03-04T19:01:03.847 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:01:03.848+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:01:03.848730+0000) 2024-03-04T19:01:04.848 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:01:04.847+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:01:04.849019+0000) 2024-03-04T19:01:05.848 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:01:05.847+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:01:05.849300+0000) 2024-03-04T19:01:06.848 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:01:06.847+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:01:06.849615+0000) 2024-03-04T19:01:07.848 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:01:07.848+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:01:07.849946+0000) 2024-03-04T19:01:08.849 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:01:08.849+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:01:08.850190+0000) 2024-03-04T19:01:09.849 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:01:09.849+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:01:09.850422+0000) 2024-03-04T19:01:10.849 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:01:10.849+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:01:10.850663+0000) 2024-03-04T19:01:11.850 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:01:11.849+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:01:11.850979+0000) 2024-03-04T19:01:12.850 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:01:12.850+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:01:12.851279+0000) 2024-03-04T19:01:13.362 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:01:13.370 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:01:13.850 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:01:13.850+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:01:13.851583+0000) 2024-03-04T19:01:14.851 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:01:14.850+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:01:14.851924+0000) 2024-03-04T19:01:15.851 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:01:15.851+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:01:15.852198+0000) 2024-03-04T19:01:16.851 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:01:16.851+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:01:16.852475+0000) 2024-03-04T19:01:17.851 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:01:17.851+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:01:17.852728+0000) 2024-03-04T19:01:18.851 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:01:18.852+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:01:18.852949+0000) 2024-03-04T19:01:19.852 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:01:19.852+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:01:19.853267+0000) 2024-03-04T19:01:20.852 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:01:20.852+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:01:20.853564+0000) 2024-03-04T19:01:21.852 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:01:21.853+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:01:21.853902+0000) 2024-03-04T19:01:22.853 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:01:22.853+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:01:22.854205+0000) 2024-03-04T19:01:23.853 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:01:23.853+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:01:23.854488+0000) 2024-03-04T19:01:24.853 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:01:24.853+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:01:24.854740+0000) 2024-03-04T19:01:25.854 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:01:25.854+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:01:25.854974+0000) 2024-03-04T19:01:26.854 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:01:26.854+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:01:26.855217+0000) 2024-03-04T19:01:27.855 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:01:27.854+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:01:27.855453+0000) 2024-03-04T19:01:28.857 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:01:28.854+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:01:28.855692+0000) 2024-03-04T19:01:29.854 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:01:29.855+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:01:29.855926+0000) 2024-03-04T19:01:30.855 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:01:30.855+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:01:30.856171+0000) 2024-03-04T19:01:31.856 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:01:31.855+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:01:31.856408+0000) 2024-03-04T19:01:32.855 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:01:32.855+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:01:32.856651+0000) 2024-03-04T19:01:33.855 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:01:33.856+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:01:33.856945+0000) 2024-03-04T19:01:34.856 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:01:34.856+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:01:34.857178+0000) 2024-03-04T19:01:35.856 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:01:35.856+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:01:35.857458+0000) 2024-03-04T19:01:36.856 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:01:36.857+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:01:36.857738+0000) 2024-03-04T19:01:37.856 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:01:37.857+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:01:37.858038+0000) 2024-03-04T19:01:38.857 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:01:38.857+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:01:38.858325+0000) 2024-03-04T19:01:39.857 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:01:39.857+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:01:39.858643+0000) 2024-03-04T19:01:40.857 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:01:40.858+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:01:40.858966+0000) 2024-03-04T19:01:41.859 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:01:41.858+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:01:41.859255+0000) 2024-03-04T19:01:42.858 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:01:42.858+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:01:42.859580+0000) 2024-03-04T19:01:43.601 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:01:43.610 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:01:43.858 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:01:43.859+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:01:43.859931+0000) 2024-03-04T19:01:44.859 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:01:44.859+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:01:44.860160+0000) 2024-03-04T19:01:45.859 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:01:45.859+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:01:45.860395+0000) 2024-03-04T19:01:46.859 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:01:46.860+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:01:46.860640+0000) 2024-03-04T19:01:47.859 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:01:47.860+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:01:47.860879+0000) 2024-03-04T19:01:48.860 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:01:48.860+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:01:48.861173+0000) 2024-03-04T19:01:49.860 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:01:49.861+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:01:49.861455+0000) 2024-03-04T19:01:50.860 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:01:50.861+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:01:50.861751+0000) 2024-03-04T19:01:51.860 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:01:51.861+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:01:51.862028+0000) 2024-03-04T19:01:52.861 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:01:52.861+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:01:52.862331+0000) 2024-03-04T19:01:53.861 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:01:53.862+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:01:53.862631+0000) 2024-03-04T19:01:54.862 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:01:54.862+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:01:54.862907+0000) 2024-03-04T19:01:55.862 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:01:55.862+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:01:55.863200+0000) 2024-03-04T19:01:56.862 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:01:56.863+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:01:56.863496+0000) 2024-03-04T19:01:57.862 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:01:57.863+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:01:57.863787+0000) 2024-03-04T19:01:58.863 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:01:58.863+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:01:58.864139+0000) 2024-03-04T19:01:59.863 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:01:59.864+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:01:59.864414+0000) 2024-03-04T19:02:00.863 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:02:00.863+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:02:00.864730+0000) 2024-03-04T19:02:01.864 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:02:01.864+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:02:01.865018+0000) 2024-03-04T19:02:02.864 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:02:02.865+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:02:02.865302+0000) 2024-03-04T19:02:03.864 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:02:03.865+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:02:03.865602+0000) 2024-03-04T19:02:04.864 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:02:04.865+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:02:04.865918+0000) 2024-03-04T19:02:05.865 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:02:05.866+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:02:05.866199+0000) 2024-03-04T19:02:06.865 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:02:06.866+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:02:06.866491+0000) 2024-03-04T19:02:07.865 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:02:07.865+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:02:07.866798+0000) 2024-03-04T19:02:08.866 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:02:08.865+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:02:08.867099+0000) 2024-03-04T19:02:09.866 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:02:09.866+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:02:09.867385+0000) 2024-03-04T19:02:10.866 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:02:10.866+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:02:10.867683+0000) 2024-03-04T19:02:11.867 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:02:11.866+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:02:11.867971+0000) 2024-03-04T19:02:12.867 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:02:12.867+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:02:12.868276+0000) 2024-03-04T19:02:13.855 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:02:13.864 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:02:13.872 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:02:13.866+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:02:13.868545+0000) 2024-03-04T19:02:14.867 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:02:14.867+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:02:14.868857+0000) 2024-03-04T19:02:15.868 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:02:15.868+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:02:15.869097+0000) 2024-03-04T19:02:16.868 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:02:16.868+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:02:16.869337+0000) 2024-03-04T19:02:17.868 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:02:17.868+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:02:17.869585+0000) 2024-03-04T19:02:18.868 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:02:18.869+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:02:18.869857+0000) 2024-03-04T19:02:19.869 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:02:19.869+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:02:19.870072+0000) 2024-03-04T19:02:20.869 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:02:20.869+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:02:20.870323+0000) 2024-03-04T19:02:21.869 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:02:21.869+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:02:21.870567+0000) 2024-03-04T19:02:22.869 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:02:22.869+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:02:22.870776+0000) 2024-03-04T19:02:23.870 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:02:23.870+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:02:23.871051+0000) 2024-03-04T19:02:24.870 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:02:24.870+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:02:24.871289+0000) 2024-03-04T19:02:25.870 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:02:25.870+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:02:25.871526+0000) 2024-03-04T19:02:26.870 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:02:26.870+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:02:26.871788+0000) 2024-03-04T19:02:27.871 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:02:27.871+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:02:27.872033+0000) 2024-03-04T19:02:28.871 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:02:28.871+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:02:28.872278+0000) 2024-03-04T19:02:29.871 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:02:29.871+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:02:29.872519+0000) 2024-03-04T19:02:30.871 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:02:30.872+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:02:30.872805+0000) 2024-03-04T19:02:31.872 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:02:31.872+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:02:31.873121+0000) 2024-03-04T19:02:32.872 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:02:32.872+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:02:32.873315+0000) 2024-03-04T19:02:33.872 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:02:33.872+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:02:33.873575+0000) 2024-03-04T19:02:34.872 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:02:34.873+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:02:34.873851+0000) 2024-03-04T19:02:35.873 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:02:35.873+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:02:35.874096+0000) 2024-03-04T19:02:36.873 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:02:36.873+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:02:36.874335+0000) 2024-03-04T19:02:37.873 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:02:37.873+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:02:37.874588+0000) 2024-03-04T19:02:38.873 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:02:38.874+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:02:38.874885+0000) 2024-03-04T19:02:39.874 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:02:39.874+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:02:39.875128+0000) 2024-03-04T19:02:40.874 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:02:40.874+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:02:40.875380+0000) 2024-03-04T19:02:41.874 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:02:41.875+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:02:41.875622+0000) 2024-03-04T19:02:42.874 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:02:42.875+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:02:42.875925+0000) 2024-03-04T19:02:43.875 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:02:43.875+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:02:43.876210+0000) 2024-03-04T19:02:44.027 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:02:44.036 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:02:44.875 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:02:44.875+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:02:44.876454+0000) 2024-03-04T19:02:45.875 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:02:45.876+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:02:45.876690+0000) 2024-03-04T19:02:46.875 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:02:46.876+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:02:46.876937+0000) 2024-03-04T19:02:47.876 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:02:47.876+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:02:47.877197+0000) 2024-03-04T19:02:48.876 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:02:48.876+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:02:48.877458+0000) 2024-03-04T19:02:49.876 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:02:49.877+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:02:49.877698+0000) 2024-03-04T19:02:50.877 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:02:50.877+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:02:50.878272+0000) 2024-03-04T19:02:51.877 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:02:51.878+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:02:51.878548+0000) 2024-03-04T19:02:52.877 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:02:52.877+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:02:52.878782+0000) 2024-03-04T19:02:53.877 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:02:53.878+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:02:53.879068+0000) 2024-03-04T19:02:54.878 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:02:54.878+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:02:54.879306+0000) 2024-03-04T19:02:55.878 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:02:55.879+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:02:55.879542+0000) 2024-03-04T19:02:56.878 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:02:56.878+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:02:56.879729+0000) 2024-03-04T19:02:57.879 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:02:57.879+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:02:57.879974+0000) 2024-03-04T19:02:58.879 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:02:58.879+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:02:58.880211+0000) 2024-03-04T19:02:59.879 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:02:59.880+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:02:59.880451+0000) 2024-03-04T19:03:00.879 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:03:00.880+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:03:00.880692+0000) 2024-03-04T19:03:01.879 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:03:01.880+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:03:01.880935+0000) 2024-03-04T19:03:02.880 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:03:02.880+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:03:02.881178+0000) 2024-03-04T19:03:03.880 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:03:03.881+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:03:03.881426+0000) 2024-03-04T19:03:04.880 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:03:04.881+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:03:04.881666+0000) 2024-03-04T19:03:05.881 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:03:05.881+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:03:05.881971+0000) 2024-03-04T19:03:06.881 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:03:06.882+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:03:06.882234+0000) 2024-03-04T19:03:07.881 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:03:07.882+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:03:07.882478+0000) 2024-03-04T19:03:08.881 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:03:08.882+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:03:08.882720+0000) 2024-03-04T19:03:09.881 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:03:09.882+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:03:09.882984+0000) 2024-03-04T19:03:10.882 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:03:10.881+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:03:10.883166+0000) 2024-03-04T19:03:11.882 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:03:11.881+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:03:11.883346+0000) 2024-03-04T19:03:12.882 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:03:12.881+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:03:12.883538+0000) 2024-03-04T19:03:13.882 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:03:13.882+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:03:13.883801+0000) 2024-03-04T19:03:14.182 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:03:14.190 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:03:14.883 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:03:14.882+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:03:14.884065+0000) 2024-03-04T19:03:15.883 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:03:15.883+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:03:15.884315+0000) 2024-03-04T19:03:16.883 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:03:16.883+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:03:16.884555+0000) 2024-03-04T19:03:17.883 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:03:17.883+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:03:17.884802+0000) 2024-03-04T19:03:18.884 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:03:18.884+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:03:18.885168+0000) 2024-03-04T19:03:19.884 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:03:19.884+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:03:19.885420+0000) 2024-03-04T19:03:20.884 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:03:20.884+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:03:20.885679+0000) 2024-03-04T19:03:21.884 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:03:21.885+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:03:21.885931+0000) 2024-03-04T19:03:22.885 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:03:22.885+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:03:22.886191+0000) 2024-03-04T19:03:23.885 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:03:23.885+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:03:23.886459+0000) 2024-03-04T19:03:24.885 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:03:24.885+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:03:24.886710+0000) 2024-03-04T19:03:25.885 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:03:25.886+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:03:25.886989+0000) 2024-03-04T19:03:26.886 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:03:26.886+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:03:26.887249+0000) 2024-03-04T19:03:27.886 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:03:27.886+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:03:27.887501+0000) 2024-03-04T19:03:28.886 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:03:28.886+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:03:28.887759+0000) 2024-03-04T19:03:29.886 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:03:29.887+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:03:29.888006+0000) 2024-03-04T19:03:30.887 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:03:30.887+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:03:30.888213+0000) 2024-03-04T19:03:31.887 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:03:31.887+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:03:31.888455+0000) 2024-03-04T19:03:32.887 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:03:32.887+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:03:32.888692+0000) 2024-03-04T19:03:33.887 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:03:33.888+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:03:33.888943+0000) 2024-03-04T19:03:34.888 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:03:34.888+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:03:34.889181+0000) 2024-03-04T19:03:35.888 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:03:35.888+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:03:35.889415+0000) 2024-03-04T19:03:36.888 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:03:36.888+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:03:36.889650+0000) 2024-03-04T19:03:37.888 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:03:37.889+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:03:37.889950+0000) 2024-03-04T19:03:38.889 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:03:38.889+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:03:38.890204+0000) 2024-03-04T19:03:39.889 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:03:39.889+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:03:39.890515+0000) 2024-03-04T19:03:40.889 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:03:40.889+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:03:40.890753+0000) 2024-03-04T19:03:41.890 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:03:41.890+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:03:41.891066+0000) 2024-03-04T19:03:42.890 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:03:42.890+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:03:42.891365+0000) 2024-03-04T19:03:43.890 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:03:43.891+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:03:43.891663+0000) 2024-03-04T19:03:44.379 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:03:44.387 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:03:44.890 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:03:44.891+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:03:44.891990+0000) 2024-03-04T19:03:45.891 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:03:45.891+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:03:45.892280+0000) 2024-03-04T19:03:46.891 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:03:46.891+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:03:46.892573+0000) 2024-03-04T19:03:47.891 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:03:47.892+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:03:47.892921+0000) 2024-03-04T19:03:48.892 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:03:48.892+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:03:48.893164+0000) 2024-03-04T19:03:49.892 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:03:49.892+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:03:49.893414+0000) 2024-03-04T19:03:50.892 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:03:50.893+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:03:50.893656+0000) 2024-03-04T19:03:51.893 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:03:51.893+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:03:51.893937+0000) 2024-03-04T19:03:52.893 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:03:52.893+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:03:52.894191+0000) 2024-03-04T19:03:53.893 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:03:53.894+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:03:53.894453+0000) 2024-03-04T19:03:54.893 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:03:54.894+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:03:54.894711+0000) 2024-03-04T19:03:55.893 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:03:55.894+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:03:55.894959+0000) 2024-03-04T19:03:56.894 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:03:56.894+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:03:56.895213+0000) 2024-03-04T19:03:57.894 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:03:57.895+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:03:57.895472+0000) 2024-03-04T19:03:58.894 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:03:58.895+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:03:58.895732+0000) 2024-03-04T19:03:59.894 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:03:59.895+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:03:59.895982+0000) 2024-03-04T19:04:00.895 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:04:00.895+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:04:00.896276+0000) 2024-03-04T19:04:01.896 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:04:01.895+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:04:01.896501+0000) 2024-03-04T19:04:02.895 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:04:02.896+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:04:02.896743+0000) 2024-03-04T19:04:03.895 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:04:03.896+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:04:03.897006+0000) 2024-03-04T19:04:04.896 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:04:04.897+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:04:04.897256+0000) 2024-03-04T19:04:05.896 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:04:05.896+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:04:05.897523+0000) 2024-03-04T19:04:06.897 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:04:06.897+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:04:06.897781+0000) 2024-03-04T19:04:07.897 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:04:07.897+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:04:07.898073+0000) 2024-03-04T19:04:08.897 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:04:08.898+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:04:08.898316+0000) 2024-03-04T19:04:09.897 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:04:09.898+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:04:09.898561+0000) 2024-03-04T19:04:10.897 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:04:10.898+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:04:10.898849+0000) 2024-03-04T19:04:11.898 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:04:11.898+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:04:11.899087+0000) 2024-03-04T19:04:12.898 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:04:12.899+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:04:12.899332+0000) 2024-03-04T19:04:13.898 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:04:13.898+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:04:13.899574+0000) 2024-03-04T19:04:14.592 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:04:14.600 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:04:14.898 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:04:14.898+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:04:14.899818+0000) 2024-03-04T19:04:15.899 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:04:15.899+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:04:15.900090+0000) 2024-03-04T19:04:16.899 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:04:16.899+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:04:16.900333+0000) 2024-03-04T19:04:17.899 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:04:17.899+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:04:17.900566+0000) 2024-03-04T19:04:18.899 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:04:18.899+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:04:18.900807+0000) 2024-03-04T19:04:19.900 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:04:19.900+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:04:19.901105+0000) 2024-03-04T19:04:20.900 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:04:20.900+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:04:20.901359+0000) 2024-03-04T19:04:21.900 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:04:21.899+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:04:21.901561+0000) 2024-03-04T19:04:22.900 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:04:22.900+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:04:22.901817+0000) 2024-03-04T19:04:23.901 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:04:23.901+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:04:23.902150+0000) 2024-03-04T19:04:24.901 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:04:24.901+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:04:24.902399+0000) 2024-03-04T19:04:25.901 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:04:25.901+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:04:25.902654+0000) 2024-03-04T19:04:26.903 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:04:26.902+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:04:26.902937+0000) 2024-03-04T19:04:27.902 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:04:27.902+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:04:27.903195+0000) 2024-03-04T19:04:28.902 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:04:28.902+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:04:28.903404+0000) 2024-03-04T19:04:29.902 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:04:29.902+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:04:29.903651+0000) 2024-03-04T19:04:30.902 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:04:30.903+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:04:30.903928+0000) 2024-03-04T19:04:31.903 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:04:31.903+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:04:31.904182+0000) 2024-03-04T19:04:32.903 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:04:32.903+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:04:32.904444+0000) 2024-03-04T19:04:33.903 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:04:33.903+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:04:33.904692+0000) 2024-03-04T19:04:34.904 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:04:34.904+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:04:34.904933+0000) 2024-03-04T19:04:35.904 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:04:35.904+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:04:35.905164+0000) 2024-03-04T19:04:36.904 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:04:36.904+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:04:36.905401+0000) 2024-03-04T19:04:37.904 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:04:37.904+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:04:37.905637+0000) 2024-03-04T19:04:38.904 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:04:38.905+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:04:38.905938+0000) 2024-03-04T19:04:39.905 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:04:39.905+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:04:39.906197+0000) 2024-03-04T19:04:40.905 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:04:40.905+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:04:40.906400+0000) 2024-03-04T19:04:41.905 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:04:41.905+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:04:41.906580+0000) 2024-03-04T19:04:42.905 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:04:42.906+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:04:42.906870+0000) 2024-03-04T19:04:43.906 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:04:43.906+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:04:43.907133+0000) 2024-03-04T19:04:44.813 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:04:44.825 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:04:44.906 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:04:44.906+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:04:44.907325+0000) 2024-03-04T19:04:45.906 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:04:45.906+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:04:45.907555+0000) 2024-03-04T19:04:46.907 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:04:46.907+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:04:46.907796+0000) 2024-03-04T19:04:47.907 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:04:47.907+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:04:47.908067+0000) 2024-03-04T19:04:48.908 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:04:48.907+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:04:48.908315+0000) 2024-03-04T19:04:49.907 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:04:49.908+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:04:49.908546+0000) 2024-03-04T19:04:50.907 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:04:50.908+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:04:50.908783+0000) 2024-03-04T19:04:51.908 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:04:51.908+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:04:51.909016+0000) 2024-03-04T19:04:52.908 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:04:52.908+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:04:52.909253+0000) 2024-03-04T19:04:53.908 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:04:53.908+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:04:53.909450+0000) 2024-03-04T19:04:54.908 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:04:54.908+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:04:54.909726+0000) 2024-03-04T19:04:55.909 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:04:55.909+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:04:55.909960+0000) 2024-03-04T19:04:56.909 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:04:56.909+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:04:56.910196+0000) 2024-03-04T19:04:57.909 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:04:57.910+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:04:57.910430+0000) 2024-03-04T19:04:58.909 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:04:58.910+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:04:58.910671+0000) 2024-03-04T19:04:59.909 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:04:59.910+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:04:59.910938+0000) 2024-03-04T19:05:00.910 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:05:00.910+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:05:00.911173+0000) 2024-03-04T19:05:01.910 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:05:01.911+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:05:01.911412+0000) 2024-03-04T19:05:02.910 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:05:02.910+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:05:02.911639+0000) 2024-03-04T19:05:03.910 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:05:03.911+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:05:03.911931+0000) 2024-03-04T19:05:04.911 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:05:04.911+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:05:04.912164+0000) 2024-03-04T19:05:05.911 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:05:05.911+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:05:05.912395+0000) 2024-03-04T19:05:06.911 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:05:06.912+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:05:06.912635+0000) 2024-03-04T19:05:07.911 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:05:07.912+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:05:07.912913+0000) 2024-03-04T19:05:08.912 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:05:08.912+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:05:08.913156+0000) 2024-03-04T19:05:09.912 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:05:09.913+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:05:09.913422+0000) 2024-03-04T19:05:10.912 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:05:10.913+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:05:10.913661+0000) 2024-03-04T19:05:11.912 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:05:11.913+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:05:11.913946+0000) 2024-03-04T19:05:12.913 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:05:12.914+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:05:12.914203+0000) 2024-03-04T19:05:13.913 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:05:13.914+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:05:13.914448+0000) 2024-03-04T19:05:14.913 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:05:14.913+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:05:14.914676+0000) 2024-03-04T19:05:15.044 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:05:15.052 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:05:15.913 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:05:15.914+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:05:15.914943+0000) 2024-03-04T19:05:16.914 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:05:16.914+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:05:16.915184+0000) 2024-03-04T19:05:17.914 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:05:17.914+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:05:17.915415+0000) 2024-03-04T19:05:18.914 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:05:18.914+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:05:18.915657+0000) 2024-03-04T19:05:19.915 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:05:19.914+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:05:19.915925+0000) 2024-03-04T19:05:20.915 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:05:20.915+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:05:20.916166+0000) 2024-03-04T19:05:21.915 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:05:21.915+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:05:21.916399+0000) 2024-03-04T19:05:22.915 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:05:22.915+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:05:22.916643+0000) 2024-03-04T19:05:23.916 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:05:23.915+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:05:23.916965+0000) 2024-03-04T19:05:24.916 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:05:24.916+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:05:24.917200+0000) 2024-03-04T19:05:25.916 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:05:25.916+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:05:25.917432+0000) 2024-03-04T19:05:26.916 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:05:26.916+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:05:26.917676+0000) 2024-03-04T19:05:27.917 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:05:27.917+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:05:27.917953+0000) 2024-03-04T19:05:28.917 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:05:28.917+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:05:28.918192+0000) 2024-03-04T19:05:29.917 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:05:29.917+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:05:29.918427+0000) 2024-03-04T19:05:30.917 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:05:30.917+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:05:30.918668+0000) 2024-03-04T19:05:31.918 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:05:31.918+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:05:31.918943+0000) 2024-03-04T19:05:32.918 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:05:32.918+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:05:32.919182+0000) 2024-03-04T19:05:33.918 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:05:33.918+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:05:33.919427+0000) 2024-03-04T19:05:34.918 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:05:34.918+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:05:34.919663+0000) 2024-03-04T19:05:35.918 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:05:35.919+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:05:35.919943+0000) 2024-03-04T19:05:36.919 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:05:36.919+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:05:36.920181+0000) 2024-03-04T19:05:37.919 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:05:37.919+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:05:37.920418+0000) 2024-03-04T19:05:38.919 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:05:38.919+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:05:38.920656+0000) 2024-03-04T19:05:39.920 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:05:39.920+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:05:39.920922+0000) 2024-03-04T19:05:40.920 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:05:40.920+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:05:40.921158+0000) 2024-03-04T19:05:41.920 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:05:41.920+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:05:41.921417+0000) 2024-03-04T19:05:42.920 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:05:42.920+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:05:42.921602+0000) 2024-03-04T19:05:43.921 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:05:43.921+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:05:43.921878+0000) 2024-03-04T19:05:44.921 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:05:44.921+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:05:44.922121+0000) 2024-03-04T19:05:45.232 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:05:45.240 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:05:45.921 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:05:45.921+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:05:45.922358+0000) 2024-03-04T19:05:46.921 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:05:46.921+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:05:46.922598+0000) 2024-03-04T19:05:47.922 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:05:47.922+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:05:47.922918+0000) 2024-03-04T19:05:48.922 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:05:48.922+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:05:48.923158+0000) 2024-03-04T19:05:49.922 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:05:49.922+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:05:49.923394+0000) 2024-03-04T19:05:50.922 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:05:50.923+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:05:50.923646+0000) 2024-03-04T19:05:51.922 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:05:51.923+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:05:51.923921+0000) 2024-03-04T19:05:52.923 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:05:52.923+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:05:52.924179+0000) 2024-03-04T19:05:53.923 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:05:53.923+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:05:53.924423+0000) 2024-03-04T19:05:54.923 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:05:54.924+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:05:54.924659+0000) 2024-03-04T19:05:55.923 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:05:55.924+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:05:55.924926+0000) 2024-03-04T19:05:56.924 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:05:56.924+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:05:56.925171+0000) 2024-03-04T19:05:57.924 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:05:57.924+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:05:57.925404+0000) 2024-03-04T19:05:58.924 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:05:58.925+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:05:58.925648+0000) 2024-03-04T19:05:59.924 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:05:59.925+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:05:59.925917+0000) 2024-03-04T19:06:00.925 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:06:00.925+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:06:00.926187+0000) 2024-03-04T19:06:01.925 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:06:01.926+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:06:01.926479+0000) 2024-03-04T19:06:02.925 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:06:02.925+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:06:02.926756+0000) 2024-03-04T19:06:03.926 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:06:03.926+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:06:03.927067+0000) 2024-03-04T19:06:04.926 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:06:04.926+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:06:04.927356+0000) 2024-03-04T19:06:05.926 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:06:05.927+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:06:05.927640+0000) 2024-03-04T19:06:06.926 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:06:06.927+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:06:06.927968+0000) 2024-03-04T19:06:07.927 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:06:07.927+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:06:07.928263+0000) 2024-03-04T19:06:08.927 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:06:08.927+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:06:08.928559+0000) 2024-03-04T19:06:09.928 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:06:09.928+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:06:09.928872+0000) 2024-03-04T19:06:10.928 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:06:10.928+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:06:10.929158+0000) 2024-03-04T19:06:11.928 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:06:11.929+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:06:11.929446+0000) 2024-03-04T19:06:12.928 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:06:12.929+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:06:12.929742+0000) 2024-03-04T19:06:13.928 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:06:13.929+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:06:13.930052+0000) 2024-03-04T19:06:14.929 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:06:14.929+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:06:14.930284+0000) 2024-03-04T19:06:15.429 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:06:15.437 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:06:15.929 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:06:15.930+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:06:15.930579+0000) 2024-03-04T19:06:16.930 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:06:16.930+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:06:16.930926+0000) 2024-03-04T19:06:17.930 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:06:17.931+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:06:17.931220+0000) 2024-03-04T19:06:18.930 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:06:18.930+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:06:18.931516+0000) 2024-03-04T19:06:19.930 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:06:19.930+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:06:19.931799+0000) 2024-03-04T19:06:20.931 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:06:20.930+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:06:20.932093+0000) 2024-03-04T19:06:21.931 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:06:21.931+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:06:21.932354+0000) 2024-03-04T19:06:22.931 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:06:22.931+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:06:22.932611+0000) 2024-03-04T19:06:23.931 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:06:23.931+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:06:23.932932+0000) 2024-03-04T19:06:24.932 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:06:24.932+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:06:24.933178+0000) 2024-03-04T19:06:25.932 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:06:25.932+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:06:25.933417+0000) 2024-03-04T19:06:26.932 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:06:26.932+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:06:26.933666+0000) 2024-03-04T19:06:27.933 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:06:27.932+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:06:27.933935+0000) 2024-03-04T19:06:28.933 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:06:28.933+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:06:28.934185+0000) 2024-03-04T19:06:29.933 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:06:29.933+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:06:29.934426+0000) 2024-03-04T19:06:30.933 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:06:30.933+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:06:30.934674+0000) 2024-03-04T19:06:31.933 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:06:31.934+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:06:31.934943+0000) 2024-03-04T19:06:32.934 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:06:32.934+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:06:32.935188+0000) 2024-03-04T19:06:33.934 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:06:33.934+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:06:33.935457+0000) 2024-03-04T19:06:34.934 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:06:34.934+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:06:34.935709+0000) 2024-03-04T19:06:35.934 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:06:35.935+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:06:35.935939+0000) 2024-03-04T19:06:36.935 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:06:36.935+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:06:36.936179+0000) 2024-03-04T19:06:37.935 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:06:37.935+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:06:37.936436+0000) 2024-03-04T19:06:38.935 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:06:38.935+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:06:38.936704+0000) 2024-03-04T19:06:39.936 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:06:39.936+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:06:39.936957+0000) 2024-03-04T19:06:40.936 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:06:40.936+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:06:40.937255+0000) 2024-03-04T19:06:41.936 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:06:41.936+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:06:41.937495+0000) 2024-03-04T19:06:42.936 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:06:42.936+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:06:42.937739+0000) 2024-03-04T19:06:43.936 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:06:43.937+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:06:43.937998+0000) 2024-03-04T19:06:44.937 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:06:44.937+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:06:44.938247+0000) 2024-03-04T19:06:45.658 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:06:45.666 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:06:45.937 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:06:45.937+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:06:45.938494+0000) 2024-03-04T19:06:46.937 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:06:46.937+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:06:46.938801+0000) 2024-03-04T19:06:47.938 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:06:47.938+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:06:47.939154+0000) 2024-03-04T19:06:48.938 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:06:48.938+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:06:48.939456+0000) 2024-03-04T19:06:49.938 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:06:49.939+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:06:49.939750+0000) 2024-03-04T19:06:50.938 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:06:50.939+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:06:50.940027+0000) 2024-03-04T19:06:51.939 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:06:51.939+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:06:51.940309+0000) 2024-03-04T19:06:52.939 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:06:52.939+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:06:52.940610+0000) 2024-03-04T19:06:53.939 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:06:53.940+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:06:53.940979+0000) 2024-03-04T19:06:54.940 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:06:54.940+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:06:54.941293+0000) 2024-03-04T19:06:55.940 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:06:55.941+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:06:55.941584+0000) 2024-03-04T19:06:56.940 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:06:56.941+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:06:56.941947+0000) 2024-03-04T19:06:57.941 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:06:57.941+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:06:57.942242+0000) 2024-03-04T19:06:58.941 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:06:58.942+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:06:58.942540+0000) 2024-03-04T19:06:59.942 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:06:59.942+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:06:59.942877+0000) 2024-03-04T19:07:00.942 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:07:00.942+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:07:00.943179+0000) 2024-03-04T19:07:01.942 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:07:01.942+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:07:01.943420+0000) 2024-03-04T19:07:02.942 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:07:02.942+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:07:02.943673+0000) 2024-03-04T19:07:03.942 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:07:03.943+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:07:03.943977+0000) 2024-03-04T19:07:04.943 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:07:04.943+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:07:04.944235+0000) 2024-03-04T19:07:05.943 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:07:05.944+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:07:05.944543+0000) 2024-03-04T19:07:06.943 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:07:06.944+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:07:06.944875+0000) 2024-03-04T19:07:07.944 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:07:07.944+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:07:07.945169+0000) 2024-03-04T19:07:08.944 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:07:08.945+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:07:08.945459+0000) 2024-03-04T19:07:09.944 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:07:09.945+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:07:09.945739+0000) 2024-03-04T19:07:10.945 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:07:10.945+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:07:10.946036+0000) 2024-03-04T19:07:11.945 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:07:11.946+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:07:11.946325+0000) 2024-03-04T19:07:12.945 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:07:12.946+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:07:12.946616+0000) 2024-03-04T19:07:13.946 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:07:13.946+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:07:13.946990+0000) 2024-03-04T19:07:14.946 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:07:14.947+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:07:14.947310+0000) 2024-03-04T19:07:15.797 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:07:15.805 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:07:15.946 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:07:15.946+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:07:15.947622+0000) 2024-03-04T19:07:16.946 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:07:16.947+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:07:16.947953+0000) 2024-03-04T19:07:17.947 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:07:17.948+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:07:17.948255+0000) 2024-03-04T19:07:18.947 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:07:18.948+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:07:18.948543+0000) 2024-03-04T19:07:19.947 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:07:19.948+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:07:19.948875+0000) 2024-03-04T19:07:20.948 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:07:20.949+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:07:20.949196+0000) 2024-03-04T19:07:21.948 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:07:21.948+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:07:21.949486+0000) 2024-03-04T19:07:22.948 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:07:22.947+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:07:22.949742+0000) 2024-03-04T19:07:23.949 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:07:23.948+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:07:23.950032+0000) 2024-03-04T19:07:24.949 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:07:24.948+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:07:24.950286+0000) 2024-03-04T19:07:25.949 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:07:25.948+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:07:25.950509+0000) 2024-03-04T19:07:26.949 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:07:26.949+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:07:26.950798+0000) 2024-03-04T19:07:27.950 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:07:27.950+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:07:27.951133+0000) 2024-03-04T19:07:28.950 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:07:28.950+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:07:28.951477+0000) 2024-03-04T19:07:29.951 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:07:29.950+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:07:29.951794+0000) 2024-03-04T19:07:30.951 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:07:30.951+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:07:30.952160+0000) 2024-03-04T19:07:31.951 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:07:31.951+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:07:31.952447+0000) 2024-03-04T19:07:32.951 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:07:32.951+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:07:32.952737+0000) 2024-03-04T19:07:33.952 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:07:33.952+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:07:33.953041+0000) 2024-03-04T19:07:34.952 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:07:34.952+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:07:34.953333+0000) 2024-03-04T19:07:35.952 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:07:35.952+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:07:35.953617+0000) 2024-03-04T19:07:36.952 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:07:36.953+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:07:36.953955+0000) 2024-03-04T19:07:37.953 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:07:37.953+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:07:37.954193+0000) 2024-03-04T19:07:38.953 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:07:38.953+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:07:38.954425+0000) 2024-03-04T19:07:39.953 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:07:39.953+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:07:39.954686+0000) 2024-03-04T19:07:40.953 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:07:40.954+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:07:40.954923+0000) 2024-03-04T19:07:41.954 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:07:41.954+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:07:41.955158+0000) 2024-03-04T19:07:42.954 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:07:42.954+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:07:42.955403+0000) 2024-03-04T19:07:43.954 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:07:43.954+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:07:43.955646+0000) 2024-03-04T19:07:44.955 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:07:44.955+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:07:44.955972+0000) 2024-03-04T19:07:45.955 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:07:45.955+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:07:45.956253+0000) 2024-03-04T19:07:45.963 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:07:45.970 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:07:46.955 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:07:46.955+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:07:46.956534+0000) 2024-03-04T19:07:47.955 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:07:47.955+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:07:47.956768+0000) 2024-03-04T19:07:48.956 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:07:48.956+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:07:48.957032+0000) 2024-03-04T19:07:49.956 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:07:49.956+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:07:49.957318+0000) 2024-03-04T19:07:50.956 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:07:50.956+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:07:50.957600+0000) 2024-03-04T19:07:51.957 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:07:51.957+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:07:51.957938+0000) 2024-03-04T19:07:52.957 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:07:52.957+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:07:52.958226+0000) 2024-03-04T19:07:53.957 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:07:53.957+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:07:53.958524+0000) 2024-03-04T19:07:54.957 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:07:54.958+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:07:54.958812+0000) 2024-03-04T19:07:55.958 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:07:55.958+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:07:55.959143+0000) 2024-03-04T19:07:56.958 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:07:56.958+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:07:56.959454+0000) 2024-03-04T19:07:57.958 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:07:57.959+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:07:57.959761+0000) 2024-03-04T19:07:58.959 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:07:58.959+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:07:58.960042+0000) 2024-03-04T19:07:59.959 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:07:59.959+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:07:59.960329+0000) 2024-03-04T19:08:00.959 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:08:00.959+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:08:00.960607+0000) 2024-03-04T19:08:01.959 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:08:01.960+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:08:01.960951+0000) 2024-03-04T19:08:02.960 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:08:02.960+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:08:02.961231+0000) 2024-03-04T19:08:03.960 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:08:03.961+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:08:03.961537+0000) 2024-03-04T19:08:04.960 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:08:04.961+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:08:04.961881+0000) 2024-03-04T19:08:05.961 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:08:05.961+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:08:05.962165+0000) 2024-03-04T19:08:06.961 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:08:06.962+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:08:06.962458+0000) 2024-03-04T19:08:07.961 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:08:07.962+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:08:07.962750+0000) 2024-03-04T19:08:08.961 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:08:08.962+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:08:08.963040+0000) 2024-03-04T19:08:09.962 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:08:09.962+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:08:09.963319+0000) 2024-03-04T19:08:10.962 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:08:10.962+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:08:10.963592+0000) 2024-03-04T19:08:11.963 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:08:11.963+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:08:11.963997+0000) 2024-03-04T19:08:12.963 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:08:12.963+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:08:12.964284+0000) 2024-03-04T19:08:13.963 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:08:13.964+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:08:13.964589+0000) 2024-03-04T19:08:14.963 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:08:14.964+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:08:14.964917+0000) 2024-03-04T19:08:15.964 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:08:15.964+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:08:15.965187+0000) 2024-03-04T19:08:16.093 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:08:16.101 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:08:16.964 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:08:16.965+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:08:16.965490+0000) 2024-03-04T19:08:17.964 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:08:17.965+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:08:17.965790+0000) 2024-03-04T19:08:18.965 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:08:18.965+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:08:18.966128+0000) 2024-03-04T19:08:19.965 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:08:19.966+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:08:19.966410+0000) 2024-03-04T19:08:20.965 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:08:20.965+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:08:20.966646+0000) 2024-03-04T19:08:21.966 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:08:21.966+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:08:21.966983+0000) 2024-03-04T19:08:22.966 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:08:22.967+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:08:22.967288+0000) 2024-03-04T19:08:23.967 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:08:23.967+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:08:23.967587+0000) 2024-03-04T19:08:24.967 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:08:24.966+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:08:24.967914+0000) 2024-03-04T19:08:25.967 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:08:25.966+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:08:25.968192+0000) 2024-03-04T19:08:26.967 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:08:26.967+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:08:26.968472+0000) 2024-03-04T19:08:27.967 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:08:27.967+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:08:27.968764+0000) 2024-03-04T19:08:28.968 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:08:28.967+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:08:28.969041+0000) 2024-03-04T19:08:29.968 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:08:29.968+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:08:29.969330+0000) 2024-03-04T19:08:30.968 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:08:30.968+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:08:30.969618+0000) 2024-03-04T19:08:31.968 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:08:31.968+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:08:31.969931+0000) 2024-03-04T19:08:32.969 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:08:32.968+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:08:32.970154+0000) 2024-03-04T19:08:33.969 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:08:33.969+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:08:33.970454+0000) 2024-03-04T19:08:34.969 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:08:34.969+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:08:34.970745+0000) 2024-03-04T19:08:35.970 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:08:35.970+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:08:35.971059+0000) 2024-03-04T19:08:36.970 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:08:36.970+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:08:36.971369+0000) 2024-03-04T19:08:37.970 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:08:37.970+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:08:37.971655+0000) 2024-03-04T19:08:38.970 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:08:38.971+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:08:38.971981+0000) 2024-03-04T19:08:39.971 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:08:39.971+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:08:39.972271+0000) 2024-03-04T19:08:40.971 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:08:40.971+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:08:40.972619+0000) 2024-03-04T19:08:41.972 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:08:41.972+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:08:41.972975+0000) 2024-03-04T19:08:42.972 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:08:42.972+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:08:42.973255+0000) 2024-03-04T19:08:43.972 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:08:43.972+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:08:43.973555+0000) 2024-03-04T19:08:44.972 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:08:44.973+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:08:44.973891+0000) 2024-03-04T19:08:45.973 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:08:45.973+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:08:45.974184+0000) 2024-03-04T19:08:46.312 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:08:46.330 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:08:46.973 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:08:46.973+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:08:46.974495+0000) 2024-03-04T19:08:47.973 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:08:47.974+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:08:47.974808+0000) 2024-03-04T19:08:48.974 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:08:48.974+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:08:48.975143+0000) 2024-03-04T19:08:49.974 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:08:49.974+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:08:49.975425+0000) 2024-03-04T19:08:50.974 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:08:50.974+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:08:50.975705+0000) 2024-03-04T19:08:51.975 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:08:51.975+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:08:51.975989+0000) 2024-03-04T19:08:52.975 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:08:52.975+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:08:52.976278+0000) 2024-03-04T19:08:53.975 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:08:53.975+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:08:53.976584+0000) 2024-03-04T19:08:54.976 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:08:54.976+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:08:54.976940+0000) 2024-03-04T19:08:55.976 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:08:55.976+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:08:55.977232+0000) 2024-03-04T19:08:56.976 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:08:56.976+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:08:56.977522+0000) 2024-03-04T19:08:57.977 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:08:57.977+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:08:57.977810+0000) 2024-03-04T19:08:58.977 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:08:58.977+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:08:58.978110+0000) 2024-03-04T19:08:59.977 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:08:59.977+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:08:59.978404+0000) 2024-03-04T19:09:00.977 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:09:00.978+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:09:00.978660+0000) 2024-03-04T19:09:01.978 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:09:01.978+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:09:01.978951+0000) 2024-03-04T19:09:02.978 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:09:02.978+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:09:02.979190+0000) 2024-03-04T19:09:03.978 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:09:03.978+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:09:03.979432+0000) 2024-03-04T19:09:04.978 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:09:04.979+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:09:04.979726+0000) 2024-03-04T19:09:05.979 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:09:05.979+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:09:05.979958+0000) 2024-03-04T19:09:06.979 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:09:06.979+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:09:06.980216+0000) 2024-03-04T19:09:07.979 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:09:07.980+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:09:07.980451+0000) 2024-03-04T19:09:08.979 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:09:08.980+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:09:08.980690+0000) 2024-03-04T19:09:09.979 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:09:09.980+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:09:09.980923+0000) 2024-03-04T19:09:10.980 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:09:10.980+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:09:10.981163+0000) 2024-03-04T19:09:11.980 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:09:11.981+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:09:11.981398+0000) 2024-03-04T19:09:12.980 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:09:12.981+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:09:12.981633+0000) 2024-03-04T19:09:13.980 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:09:13.981+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:09:13.981912+0000) 2024-03-04T19:09:14.981 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:09:14.981+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:09:14.982167+0000) 2024-03-04T19:09:15.981 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:09:15.982+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:09:15.982426+0000) 2024-03-04T19:09:16.534 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:09:16.542 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:09:16.981 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:09:16.982+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:09:16.982618+0000) 2024-03-04T19:09:17.981 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:09:17.982+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:09:17.982888+0000) 2024-03-04T19:09:18.982 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:09:18.982+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:09:18.983126+0000) 2024-03-04T19:09:19.982 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:09:19.982+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:09:19.983375+0000) 2024-03-04T19:09:20.982 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:09:20.982+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:09:20.983637+0000) 2024-03-04T19:09:21.982 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:09:21.983+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:09:21.983915+0000) 2024-03-04T19:09:22.983 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:09:22.983+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:09:22.984159+0000) 2024-03-04T19:09:23.983 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:09:23.984+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:09:23.984399+0000) 2024-03-04T19:09:24.983 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:09:24.984+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:09:24.984675+0000) 2024-03-04T19:09:25.983 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:09:25.984+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:09:25.984928+0000) 2024-03-04T19:09:26.984 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:09:26.985+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:09:26.985167+0000) 2024-03-04T19:09:27.984 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:09:27.984+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:09:27.985399+0000) 2024-03-04T19:09:28.984 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:09:28.984+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:09:28.985638+0000) 2024-03-04T19:09:29.985 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:09:29.984+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:09:29.985926+0000) 2024-03-04T19:09:30.985 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:09:30.984+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:09:30.986110+0000) 2024-03-04T19:09:31.985 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:09:31.985+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:09:31.986348+0000) 2024-03-04T19:09:32.985 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:09:32.984+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:09:32.986597+0000) 2024-03-04T19:09:33.986 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:09:33.985+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:09:33.986867+0000) 2024-03-04T19:09:34.986 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:09:34.986+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:09:34.987109+0000) 2024-03-04T19:09:35.986 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:09:35.986+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:09:35.987343+0000) 2024-03-04T19:09:36.986 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:09:36.986+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:09:36.987578+0000) 2024-03-04T19:09:37.986 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:09:37.986+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:09:37.987809+0000) 2024-03-04T19:09:38.987 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:09:38.987+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:09:38.988098+0000) 2024-03-04T19:09:39.987 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:09:39.987+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:09:39.988326+0000) 2024-03-04T19:09:40.987 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:09:40.987+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:09:40.988566+0000) 2024-03-04T19:09:41.988 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:09:41.988+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:09:41.988812+0000) 2024-03-04T19:09:42.988 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:09:42.988+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:09:42.989085+0000) 2024-03-04T19:09:43.988 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:09:43.988+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:09:43.989326+0000) 2024-03-04T19:09:44.988 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:09:44.988+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:09:44.989566+0000) 2024-03-04T19:09:45.988 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:09:45.989+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:09:45.989867+0000) 2024-03-04T19:09:46.681 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:09:46.689 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:09:46.988 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:09:46.989+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:09:46.990066+0000) 2024-03-04T19:09:47.989 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:09:47.989+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:09:47.990302+0000) 2024-03-04T19:09:48.990 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:09:48.989+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:09:48.990538+0000) 2024-03-04T19:09:49.992 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:09:49.990+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:09:49.990778+0000) 2024-03-04T19:09:50.990 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:09:50.990+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:09:50.991017+0000) 2024-03-04T19:09:51.990 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:09:51.990+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:09:51.991255+0000) 2024-03-04T19:09:52.990 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:09:52.990+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:09:52.991493+0000) 2024-03-04T19:09:53.990 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:09:53.991+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:09:53.991734+0000) 2024-03-04T19:09:54.991 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:09:54.991+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:09:54.991969+0000) 2024-03-04T19:09:55.991 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:09:55.991+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:09:55.992201+0000) 2024-03-04T19:09:56.991 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:09:56.991+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:09:56.992438+0000) 2024-03-04T19:09:57.991 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:09:57.992+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:09:57.992668+0000) 2024-03-04T19:09:58.992 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:09:58.992+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:09:58.992959+0000) 2024-03-04T19:09:59.992 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:09:59.992+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:09:59.993213+0000) 2024-03-04T19:10:00.992 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:10:00.992+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:10:00.993404+0000) 2024-03-04T19:10:01.992 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:10:01.993+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:10:01.993635+0000) 2024-03-04T19:10:02.993 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:10:02.993+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:10:02.993915+0000) 2024-03-04T19:10:03.993 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:10:03.993+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:10:03.994162+0000) 2024-03-04T19:10:04.993 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:10:04.993+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:10:04.994404+0000) 2024-03-04T19:10:05.993 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:10:05.994+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:10:05.994644+0000) 2024-03-04T19:10:06.993 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:10:06.994+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:10:06.994935+0000) 2024-03-04T19:10:07.994 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:10:07.994+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:10:07.995196+0000) 2024-03-04T19:10:08.994 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:10:08.995+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:10:08.995477+0000) 2024-03-04T19:10:09.994 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:10:09.994+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:10:09.995671+0000) 2024-03-04T19:10:10.994 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:10:10.995+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:10:10.995897+0000) 2024-03-04T19:10:11.995 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:10:11.995+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:10:11.996146+0000) 2024-03-04T19:10:12.995 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:10:12.995+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:10:12.996387+0000) 2024-03-04T19:10:13.995 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:10:13.996+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:10:13.996638+0000) 2024-03-04T19:10:14.995 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:10:14.996+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:10:14.996923+0000) 2024-03-04T19:10:15.996 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:10:15.996+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:10:15.997159+0000) 2024-03-04T19:10:16.878 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:10:16.886 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:10:17.004 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:10:16.996+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:10:16.997429+0000) 2024-03-04T19:10:17.996 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:10:17.997+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:10:17.997672+0000) 2024-03-04T19:10:18.996 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:10:18.997+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:10:18.997953+0000) 2024-03-04T19:10:19.997 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:10:19.997+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:10:19.998193+0000) 2024-03-04T19:10:20.997 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:10:20.998+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:10:20.998444+0000) 2024-03-04T19:10:21.997 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:10:21.998+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:10:21.998682+0000) 2024-03-04T19:10:22.997 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:10:22.998+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:10:22.998942+0000) 2024-03-04T19:10:23.998 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:10:23.998+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:10:23.999192+0000) 2024-03-04T19:10:24.998 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:10:24.999+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:10:24.999438+0000) 2024-03-04T19:10:25.998 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:10:25.999+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:10:25.999675+0000) 2024-03-04T19:10:26.998 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:10:26.999+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:10:26.999956+0000) 2024-03-04T19:10:27.999 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:10:27.999+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:10:28.000148+0000) 2024-03-04T19:10:28.999 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:10:29.000+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:10:29.000412+0000) 2024-03-04T19:10:29.999 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:10:30.000+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:10:30.000657+0000) 2024-03-04T19:10:30.999 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:10:30.999+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:10:31.000943+0000) 2024-03-04T19:10:32.000 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:10:32.000+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:10:32.001218+0000) 2024-03-04T19:10:33.000 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:10:32.999+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:10:33.001456+0000) 2024-03-04T19:10:34.000 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:10:34.000+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:10:34.001706+0000) 2024-03-04T19:10:35.000 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:10:35.000+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:10:35.001963+0000) 2024-03-04T19:10:36.001 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:10:36.001+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:10:36.002212+0000) 2024-03-04T19:10:37.001 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:10:37.001+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:10:37.002452+0000) 2024-03-04T19:10:38.001 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:10:38.001+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:10:38.002690+0000) 2024-03-04T19:10:39.001 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:10:39.001+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:10:39.002935+0000) 2024-03-04T19:10:40.002 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:10:40.002+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:10:40.003172+0000) 2024-03-04T19:10:41.002 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:10:41.002+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:10:41.003432+0000) 2024-03-04T19:10:42.002 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:10:42.002+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:10:42.003669+0000) 2024-03-04T19:10:43.002 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:10:43.003+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:10:43.003962+0000) 2024-03-04T19:10:44.003 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:10:44.003+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:10:44.004233+0000) 2024-03-04T19:10:45.003 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:10:45.003+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:10:45.004479+0000) 2024-03-04T19:10:46.004 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:10:46.003+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:10:46.004717+0000) 2024-03-04T19:10:47.004 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:10:47.004+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:10:47.005034+0000) 2024-03-04T19:10:47.066 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:10:47.075 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:10:48.004 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:10:48.004+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:10:48.005275+0000) 2024-03-04T19:10:49.004 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:10:49.004+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:10:49.005516+0000) 2024-03-04T19:10:50.004 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:10:50.004+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:10:50.005782+0000) 2024-03-04T19:10:51.005 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:10:51.005+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:10:51.006023+0000) 2024-03-04T19:10:52.005 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:10:52.005+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:10:52.006264+0000) 2024-03-04T19:10:53.005 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:10:53.005+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:10:53.006504+0000) 2024-03-04T19:10:54.005 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:10:54.006+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:10:54.006752+0000) 2024-03-04T19:10:55.006 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:10:55.006+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:10:55.007002+0000) 2024-03-04T19:10:56.006 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:10:56.006+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:10:56.007183+0000) 2024-03-04T19:10:57.006 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:10:57.006+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:10:57.007436+0000) 2024-03-04T19:10:58.006 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:10:58.006+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:10:58.007689+0000) 2024-03-04T19:10:59.006 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:10:59.007+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:10:59.007934+0000) 2024-03-04T19:11:00.007 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:11:00.007+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:11:00.008143+0000) 2024-03-04T19:11:01.007 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:11:01.007+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:11:01.008393+0000) 2024-03-04T19:11:02.007 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:11:02.008+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:11:02.008633+0000) 2024-03-04T19:11:03.007 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:11:03.008+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:11:03.008957+0000) 2024-03-04T19:11:04.008 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:11:04.008+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:11:04.009252+0000) 2024-03-04T19:11:05.008 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:11:05.008+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:11:05.009509+0000) 2024-03-04T19:11:06.008 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:11:06.008+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:11:06.009729+0000) 2024-03-04T19:11:07.009 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:11:07.009+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:11:07.010008+0000) 2024-03-04T19:11:08.009 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:11:08.009+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:11:08.010209+0000) 2024-03-04T19:11:09.009 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:11:09.009+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:11:09.010455+0000) 2024-03-04T19:11:10.009 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:11:10.009+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:11:10.010708+0000) 2024-03-04T19:11:11.009 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:11:11.010+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:11:11.010955+0000) 2024-03-04T19:11:12.010 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:11:12.010+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:11:12.011191+0000) 2024-03-04T19:11:13.010 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:11:13.010+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:11:13.011456+0000) 2024-03-04T19:11:14.010 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:11:14.011+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:11:14.011717+0000) 2024-03-04T19:11:15.011 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:11:15.011+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:11:15.011970+0000) 2024-03-04T19:11:16.011 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:11:16.011+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:11:16.012179+0000) 2024-03-04T19:11:17.011 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:11:17.012+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:11:17.012455+0000) 2024-03-04T19:11:17.197 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:11:17.209 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:11:18.011 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:11:18.012+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:11:18.012724+0000) 2024-03-04T19:11:19.012 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:11:19.012+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:11:19.012977+0000) 2024-03-04T19:11:20.012 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:11:20.012+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:11:20.013241+0000) 2024-03-04T19:11:21.012 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:11:21.012+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:11:21.013454+0000) 2024-03-04T19:11:22.012 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:11:22.013+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:11:22.013713+0000) 2024-03-04T19:11:23.012 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:11:23.013+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:11:23.013970+0000) 2024-03-04T19:11:24.013 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:11:24.013+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:11:24.014174+0000) 2024-03-04T19:11:25.013 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:11:25.014+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:11:25.014437+0000) 2024-03-04T19:11:26.013 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:11:26.014+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:11:26.014692+0000) 2024-03-04T19:11:27.013 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:11:27.014+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:11:27.014933+0000) 2024-03-04T19:11:28.014 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:11:28.015+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:11:28.015201+0000) 2024-03-04T19:11:29.014 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:11:29.015+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:11:29.015446+0000) 2024-03-04T19:11:30.014 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:11:30.014+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:11:30.015641+0000) 2024-03-04T19:11:31.014 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:11:31.015+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:11:31.015890+0000) 2024-03-04T19:11:32.015 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:11:32.015+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:11:32.016140+0000) 2024-03-04T19:11:33.015 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:11:33.015+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:11:33.016406+0000) 2024-03-04T19:11:34.015 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:11:34.015+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:11:34.016667+0000) 2024-03-04T19:11:35.016 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:11:35.015+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:11:35.016962+0000) 2024-03-04T19:11:36.016 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:11:36.016+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:11:36.017233+0000) 2024-03-04T19:11:37.016 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:11:37.016+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:11:37.017511+0000) 2024-03-04T19:11:38.016 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:11:38.015+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:11:38.017774+0000) 2024-03-04T19:11:39.017 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:11:39.017+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:11:39.018043+0000) 2024-03-04T19:11:40.017 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:11:40.016+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:11:40.018302+0000) 2024-03-04T19:11:41.017 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:11:41.017+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:11:41.018563+0000) 2024-03-04T19:11:42.018 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:11:42.017+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:11:42.018880+0000) 2024-03-04T19:11:43.018 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:11:43.018+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:11:43.019135+0000) 2024-03-04T19:11:44.018 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:11:44.018+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:11:44.019399+0000) 2024-03-04T19:11:45.018 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:11:45.018+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:11:45.019665+0000) 2024-03-04T19:11:46.019 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:11:46.019+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:11:46.019961+0000) 2024-03-04T19:11:47.019 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:11:47.019+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:11:47.020244+0000) 2024-03-04T19:11:47.426 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:11:47.434 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:11:48.019 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:11:48.019+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:11:48.020493+0000) 2024-03-04T19:11:49.019 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:11:49.019+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:11:49.020748+0000) 2024-03-04T19:11:50.021 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:11:50.020+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:11:50.021038+0000) 2024-03-04T19:11:51.020 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:11:51.020+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:11:51.021308+0000) 2024-03-04T19:11:52.020 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:11:52.020+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:11:52.021559+0000) 2024-03-04T19:11:53.020 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:11:53.021+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:11:53.021887+0000) 2024-03-04T19:11:54.021 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:11:54.021+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:11:54.022166+0000) 2024-03-04T19:11:55.021 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:11:55.021+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:11:55.022417+0000) 2024-03-04T19:11:56.021 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:11:56.021+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:11:56.022686+0000) 2024-03-04T19:11:57.021 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:11:57.022+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:11:57.022937+0000) 2024-03-04T19:11:58.022 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:11:58.022+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:11:58.023178+0000) 2024-03-04T19:11:59.022 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:11:59.022+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:11:59.023422+0000) 2024-03-04T19:12:00.022 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:12:00.022+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:12:00.023658+0000) 2024-03-04T19:12:01.023 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:12:01.023+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:12:01.023953+0000) 2024-03-04T19:12:02.023 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:12:02.023+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:12:02.024204+0000) 2024-03-04T19:12:03.023 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:12:03.023+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:12:03.024457+0000) 2024-03-04T19:12:04.023 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:12:04.024+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:12:04.024726+0000) 2024-03-04T19:12:05.024 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:12:05.024+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:12:05.024981+0000) 2024-03-04T19:12:06.024 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:12:06.024+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:12:06.025219+0000) 2024-03-04T19:12:07.024 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:12:07.024+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:12:07.025471+0000) 2024-03-04T19:12:08.024 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:12:08.025+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:12:08.025725+0000) 2024-03-04T19:12:09.025 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:12:09.025+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:12:09.025982+0000) 2024-03-04T19:12:10.025 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:12:10.025+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:12:10.026221+0000) 2024-03-04T19:12:11.025 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:12:11.025+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:12:11.026424+0000) 2024-03-04T19:12:12.025 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:12:12.025+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:12:12.026691+0000) 2024-03-04T19:12:13.026 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:12:13.026+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:12:13.026940+0000) 2024-03-04T19:12:14.026 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:12:14.026+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:12:14.027185+0000) 2024-03-04T19:12:15.026 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:12:15.027+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:12:15.027449+0000) 2024-03-04T19:12:16.026 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:12:16.027+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:12:16.027709+0000) 2024-03-04T19:12:17.026 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:12:17.027+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:12:17.027949+0000) 2024-03-04T19:12:17.640 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:12:17.647 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:12:18.027 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:12:18.027+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:12:18.028185+0000) 2024-03-04T19:12:19.027 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:12:19.027+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:12:19.028615+0000) 2024-03-04T19:12:20.027 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:12:20.028+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:12:20.028955+0000) 2024-03-04T19:12:21.028 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:12:21.028+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:12:21.029157+0000) 2024-03-04T19:12:22.028 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:12:22.029+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:12:22.029436+0000) 2024-03-04T19:12:23.029 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:12:23.029+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:12:23.029797+0000) 2024-03-04T19:12:24.029 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:12:24.029+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:12:24.030147+0000) 2024-03-04T19:12:25.029 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:12:25.030+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:12:25.030425+0000) 2024-03-04T19:12:26.029 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:12:26.030+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:12:26.030715+0000) 2024-03-04T19:12:27.030 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:12:27.030+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:12:27.031001+0000) 2024-03-04T19:12:28.030 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:12:28.031+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:12:28.031281+0000) 2024-03-04T19:12:29.030 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:12:29.031+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:12:29.031537+0000) 2024-03-04T19:12:30.030 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:12:30.031+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:12:30.031770+0000) 2024-03-04T19:12:31.031 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:12:31.031+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:12:31.032053+0000) 2024-03-04T19:12:32.031 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:12:32.032+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:12:32.032293+0000) 2024-03-04T19:12:33.031 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:12:33.032+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:12:33.032564+0000) 2024-03-04T19:12:34.031 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:12:34.032+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:12:34.032807+0000) 2024-03-04T19:12:35.032 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:12:35.032+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:12:35.033085+0000) 2024-03-04T19:12:36.032 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:12:36.031+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:12:36.033263+0000) 2024-03-04T19:12:37.032 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:12:37.032+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:12:37.033501+0000) 2024-03-04T19:12:38.032 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:12:38.032+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:12:38.033731+0000) 2024-03-04T19:12:39.033 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:12:39.032+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:12:39.034009+0000) 2024-03-04T19:12:40.033 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:12:40.032+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:12:40.034197+0000) 2024-03-04T19:12:41.033 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:12:41.032+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:12:41.034426+0000) 2024-03-04T19:12:42.033 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:12:42.032+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:12:42.034656+0000) 2024-03-04T19:12:43.034 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:12:43.033+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:12:43.034930+0000) 2024-03-04T19:12:44.034 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:12:44.034+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:12:44.035175+0000) 2024-03-04T19:12:45.034 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:12:45.034+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:12:45.035409+0000) 2024-03-04T19:12:46.034 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:12:46.034+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:12:46.035645+0000) 2024-03-04T19:12:47.034 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:12:47.035+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:12:47.035926+0000) 2024-03-04T19:12:47.803 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:12:47.814 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:12:48.034 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:12:48.035+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:12:48.036134+0000) 2024-03-04T19:12:49.035 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:12:49.035+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:12:49.036382+0000) 2024-03-04T19:12:50.035 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:12:50.035+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:12:50.036613+0000) 2024-03-04T19:12:51.036 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:12:51.036+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:12:51.036934+0000) 2024-03-04T19:12:52.036 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:12:52.036+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:12:52.037168+0000) 2024-03-04T19:12:53.036 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:12:53.036+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:12:53.037408+0000) 2024-03-04T19:12:54.036 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:12:54.036+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:12:54.037654+0000) 2024-03-04T19:12:55.036 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:12:55.037+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:12:55.037924+0000) 2024-03-04T19:12:56.037 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:12:56.037+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:12:56.038158+0000) 2024-03-04T19:12:57.037 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:12:57.037+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:12:57.038396+0000) 2024-03-04T19:12:58.037 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:12:58.037+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:12:58.038631+0000) 2024-03-04T19:12:59.037 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:12:59.038+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:12:59.038923+0000) 2024-03-04T19:13:00.038 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:13:00.038+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:13:00.039166+0000) 2024-03-04T19:13:01.038 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:13:01.038+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:13:01.039405+0000) 2024-03-04T19:13:02.038 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:13:02.038+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:13:02.039639+0000) 2024-03-04T19:13:03.038 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:13:03.039+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:13:03.039916+0000) 2024-03-04T19:13:04.039 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:13:04.039+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:13:04.040159+0000) 2024-03-04T19:13:05.039 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:13:05.039+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:13:05.040350+0000) 2024-03-04T19:13:06.039 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:13:06.039+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:13:06.040615+0000) 2024-03-04T19:13:07.039 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:13:07.040+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:13:07.040890+0000) 2024-03-04T19:13:08.040 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:13:08.040+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:13:08.041127+0000) 2024-03-04T19:13:09.040 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:13:09.040+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:13:09.041313+0000) 2024-03-04T19:13:10.040 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:13:10.040+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:13:10.041550+0000) 2024-03-04T19:13:11.040 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:13:11.041+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:13:11.041796+0000) 2024-03-04T19:13:12.041 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:13:12.041+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:13:12.042074+0000) 2024-03-04T19:13:13.041 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:13:13.041+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:13:13.042312+0000) 2024-03-04T19:13:14.041 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:13:14.041+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:13:14.042552+0000) 2024-03-04T19:13:15.041 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:13:15.042+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:13:15.042809+0000) 2024-03-04T19:13:16.042 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:13:16.042+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:13:16.043086+0000) 2024-03-04T19:13:17.042 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:13:17.042+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:13:17.043326+0000) 2024-03-04T19:13:18.008 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:13:18.016 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:13:18.042 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:13:18.042+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:13:18.043529+0000) 2024-03-04T19:13:19.042 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:13:19.043+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:13:19.043791+0000) 2024-03-04T19:13:20.043 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:13:20.043+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:13:20.044106+0000) 2024-03-04T19:13:21.043 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:13:21.043+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:13:21.044396+0000) 2024-03-04T19:13:22.043 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:13:22.044+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:13:22.044683+0000) 2024-03-04T19:13:23.044 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:13:23.044+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:13:23.044984+0000) 2024-03-04T19:13:24.044 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:13:24.044+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:13:24.045290+0000) 2024-03-04T19:13:25.044 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:13:25.045+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:13:25.045580+0000) 2024-03-04T19:13:26.044 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:13:26.045+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:13:26.045899+0000) 2024-03-04T19:13:27.045 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:13:27.045+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:13:27.046183+0000) 2024-03-04T19:13:28.045 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:13:28.046+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:13:28.046471+0000) 2024-03-04T19:13:29.045 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:13:29.046+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:13:29.046768+0000) 2024-03-04T19:13:30.046 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:13:30.046+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:13:30.047065+0000) 2024-03-04T19:13:31.046 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:13:31.047+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:13:31.047368+0000) 2024-03-04T19:13:32.046 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:13:32.047+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:13:32.047665+0000) 2024-03-04T19:13:33.047 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:13:33.047+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:13:33.047986+0000) 2024-03-04T19:13:34.047 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:13:34.048+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:13:34.048289+0000) 2024-03-04T19:13:35.047 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:13:35.048+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:13:35.048584+0000) 2024-03-04T19:13:36.048 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:13:36.048+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:13:36.048917+0000) 2024-03-04T19:13:37.048 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:13:37.049+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:13:37.049206+0000) 2024-03-04T19:13:38.048 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:13:38.049+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:13:38.049493+0000) 2024-03-04T19:13:39.048 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:13:39.048+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:13:39.049774+0000) 2024-03-04T19:13:40.049 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:13:40.048+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:13:40.050075+0000) 2024-03-04T19:13:41.049 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:13:41.049+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:13:41.050355+0000) 2024-03-04T19:13:42.049 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:13:42.049+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:13:42.050665+0000) 2024-03-04T19:13:43.050 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:13:43.049+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:13:43.050982+0000) 2024-03-04T19:13:44.050 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:13:44.050+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:13:44.051291+0000) 2024-03-04T19:13:45.050 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:13:45.050+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:13:45.051544+0000) 2024-03-04T19:13:46.050 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:13:46.050+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:13:46.051873+0000) 2024-03-04T19:13:47.051 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:13:47.051+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:13:47.052125+0000) 2024-03-04T19:13:48.051 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:13:48.050+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:13:48.052420+0000) 2024-03-04T19:13:48.142 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:13:48.150 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:13:49.051 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:13:49.051+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:13:49.052721+0000) 2024-03-04T19:13:50.052 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:13:50.052+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:13:50.053013+0000) 2024-03-04T19:13:51.052 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:13:51.052+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:13:51.053294+0000) 2024-03-04T19:13:52.052 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:13:52.052+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:13:52.053585+0000) 2024-03-04T19:13:53.052 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:13:53.053+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:13:53.053866+0000) 2024-03-04T19:13:54.053 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:13:54.053+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:13:54.054182+0000) 2024-03-04T19:13:55.053 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:13:55.053+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:13:55.054491+0000) 2024-03-04T19:13:56.053 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:13:56.053+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:13:56.054779+0000) 2024-03-04T19:13:57.054 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:13:57.054+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:13:57.055118+0000) 2024-03-04T19:13:58.054 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:13:58.054+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:13:58.055404+0000) 2024-03-04T19:13:59.054 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:13:59.054+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:13:59.055685+0000) 2024-03-04T19:14:00.054 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:14:00.055+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:14:00.055969+0000) 2024-03-04T19:14:01.055 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:14:01.055+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:14:01.056254+0000) 2024-03-04T19:14:02.055 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:14:02.055+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:14:02.056546+0000) 2024-03-04T19:14:03.055 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:14:03.056+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:14:03.056890+0000) 2024-03-04T19:14:04.056 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:14:04.056+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:14:04.057190+0000) 2024-03-04T19:14:05.056 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:14:05.056+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:14:05.057489+0000) 2024-03-04T19:14:06.057 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:14:06.057+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:14:06.057791+0000) 2024-03-04T19:14:07.057 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:14:07.057+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:14:07.058154+0000) 2024-03-04T19:14:08.057 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:14:08.057+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:14:08.058534+0000) 2024-03-04T19:14:09.057 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:14:09.058+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:14:09.058897+0000) 2024-03-04T19:14:10.058 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:14:10.058+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:14:10.059142+0000) 2024-03-04T19:14:11.058 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:14:11.058+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:14:11.059392+0000) 2024-03-04T19:14:12.058 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:14:12.059+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:14:12.059633+0000) 2024-03-04T19:14:13.058 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:14:13.059+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:14:13.059911+0000) 2024-03-04T19:14:14.059 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:14:14.059+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:14:14.060164+0000) 2024-03-04T19:14:15.059 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:14:15.059+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:14:15.060430+0000) 2024-03-04T19:14:16.059 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:14:16.060+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:14:16.060672+0000) 2024-03-04T19:14:17.059 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:14:17.060+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:14:17.060949+0000) 2024-03-04T19:14:18.060 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:14:18.060+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:14:18.061191+0000) 2024-03-04T19:14:18.276 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:14:18.284 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:14:19.060 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:14:19.060+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:14:19.061438+0000) 2024-03-04T19:14:20.060 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:14:20.061+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:14:20.061683+0000) 2024-03-04T19:14:21.061 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:14:21.061+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:14:21.061938+0000) 2024-03-04T19:14:22.061 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:14:22.061+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:14:22.062178+0000) 2024-03-04T19:14:23.061 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:14:23.062+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:14:23.062425+0000) 2024-03-04T19:14:24.061 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:14:24.062+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:14:24.062680+0000) 2024-03-04T19:14:25.062 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:14:25.062+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:14:25.062955+0000) 2024-03-04T19:14:26.062 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:14:26.062+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:14:26.063201+0000) 2024-03-04T19:14:27.062 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:14:27.063+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:14:27.063452+0000) 2024-03-04T19:14:28.062 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:14:28.063+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:14:28.063712+0000) 2024-03-04T19:14:29.063 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:14:29.063+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:14:29.063975+0000) 2024-03-04T19:14:30.063 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:14:30.063+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:14:30.064230+0000) 2024-03-04T19:14:31.063 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:14:31.063+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:14:31.064422+0000) 2024-03-04T19:14:32.063 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:14:32.063+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:14:32.064680+0000) 2024-03-04T19:14:33.063 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:14:33.064+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:14:33.064937+0000) 2024-03-04T19:14:34.064 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:14:34.064+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:14:34.065178+0000) 2024-03-04T19:14:35.064 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:14:35.064+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:14:35.065406+0000) 2024-03-04T19:14:36.064 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:14:36.064+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:14:36.065660+0000) 2024-03-04T19:14:37.065 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:14:37.065+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:14:37.065951+0000) 2024-03-04T19:14:38.065 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:14:38.066+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:14:38.066187+0000) 2024-03-04T19:14:39.065 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:14:39.066+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:14:39.066426+0000) 2024-03-04T19:14:40.065 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:14:40.066+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:14:40.066660+0000) 2024-03-04T19:14:41.065 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:14:41.065+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:14:41.066949+0000) 2024-03-04T19:14:42.066 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:14:42.066+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:14:42.067188+0000) 2024-03-04T19:14:43.066 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:14:43.066+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:14:43.067429+0000) 2024-03-04T19:14:44.066 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:14:44.065+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:14:44.067620+0000) 2024-03-04T19:14:45.066 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:14:45.066+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:14:45.067907+0000) 2024-03-04T19:14:46.067 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:14:46.067+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:14:46.068149+0000) 2024-03-04T19:14:47.067 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:14:47.067+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:14:47.068394+0000) 2024-03-04T19:14:48.067 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:14:48.067+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:14:48.068640+0000) 2024-03-04T19:14:48.465 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:14:48.473 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:14:49.067 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:14:49.067+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:14:49.068964+0000) 2024-03-04T19:14:50.068 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:14:50.067+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:14:50.069178+0000) 2024-03-04T19:14:51.068 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:14:51.067+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:14:51.069421+0000) 2024-03-04T19:14:52.068 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:14:52.068+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:14:52.069667+0000) 2024-03-04T19:14:53.068 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:14:53.069+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:14:53.069926+0000) 2024-03-04T19:14:54.069 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:14:54.069+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:14:54.070174+0000) 2024-03-04T19:14:55.069 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:14:55.069+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:14:55.070428+0000) 2024-03-04T19:14:56.069 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:14:56.069+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:14:56.070681+0000) 2024-03-04T19:14:57.069 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:14:57.070+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:14:57.070998+0000) 2024-03-04T19:14:58.070 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:14:58.070+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:14:58.071255+0000) 2024-03-04T19:14:59.071 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:14:59.070+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:14:59.071528+0000) 2024-03-04T19:15:00.070 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:15:00.070+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:15:00.071778+0000) 2024-03-04T19:15:01.071 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:15:01.071+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:15:01.072034+0000) 2024-03-04T19:15:02.071 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:15:02.071+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:15:02.072290+0000) 2024-03-04T19:15:03.071 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:15:03.071+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:15:03.072544+0000) 2024-03-04T19:15:04.071 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:15:04.072+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:15:04.072803+0000) 2024-03-04T19:15:05.072 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:15:05.072+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:15:05.073097+0000) 2024-03-04T19:15:06.072 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:15:06.072+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:15:06.073357+0000) 2024-03-04T19:15:07.072 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:15:07.072+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:15:07.073631+0000) 2024-03-04T19:15:08.073 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:15:08.073+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:15:08.073920+0000) 2024-03-04T19:15:09.073 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:15:09.073+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:15:09.074163+0000) 2024-03-04T19:15:10.073 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:15:10.073+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:15:10.074397+0000) 2024-03-04T19:15:11.073 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:15:11.073+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:15:11.074641+0000) 2024-03-04T19:15:12.073 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:15:12.074+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:15:12.074931+0000) 2024-03-04T19:15:13.074 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:15:13.074+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:15:13.075115+0000) 2024-03-04T19:15:14.074 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:15:14.074+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:15:14.075358+0000) 2024-03-04T19:15:15.074 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:15:15.075+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:15:15.075595+0000) 2024-03-04T19:15:16.074 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:15:16.075+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:15:16.075886+0000) 2024-03-04T19:15:17.075 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:15:17.075+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:15:17.076142+0000) 2024-03-04T19:15:18.075 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:15:18.075+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:15:18.076398+0000) 2024-03-04T19:15:18.671 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:15:18.679 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:15:19.075 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:15:19.076+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:15:19.076654+0000) 2024-03-04T19:15:20.075 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:15:20.076+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:15:20.076932+0000) 2024-03-04T19:15:21.076 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:15:21.076+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:15:21.077184+0000) 2024-03-04T19:15:22.076 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:15:22.076+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:15:22.077431+0000) 2024-03-04T19:15:23.076 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:15:23.076+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:15:23.077624+0000) 2024-03-04T19:15:24.077 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:15:24.077+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:15:24.077909+0000) 2024-03-04T19:15:25.077 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:15:25.077+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:15:25.078144+0000) 2024-03-04T19:15:26.077 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:15:26.077+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:15:26.078387+0000) 2024-03-04T19:15:27.077 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:15:27.078+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:15:27.078629+0000) 2024-03-04T19:15:28.078 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:15:28.078+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:15:28.078909+0000) 2024-03-04T19:15:29.078 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:15:29.078+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:15:29.079153+0000) 2024-03-04T19:15:30.078 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:15:30.078+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:15:30.079387+0000) 2024-03-04T19:15:31.078 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:15:31.079+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:15:31.079632+0000) 2024-03-04T19:15:32.078 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:15:32.079+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:15:32.079930+0000) 2024-03-04T19:15:33.079 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:15:33.079+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:15:33.080167+0000) 2024-03-04T19:15:34.079 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:15:34.080+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:15:34.080438+0000) 2024-03-04T19:15:35.079 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:15:35.080+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:15:35.080697+0000) 2024-03-04T19:15:36.080 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:15:36.080+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:15:36.080958+0000) 2024-03-04T19:15:37.080 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:15:37.080+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:15:37.081197+0000) 2024-03-04T19:15:38.080 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:15:38.081+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:15:38.081433+0000) 2024-03-04T19:15:39.080 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:15:39.080+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:15:39.081690+0000) 2024-03-04T19:15:40.081 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:15:40.081+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:15:40.082012+0000) 2024-03-04T19:15:41.081 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:15:41.082+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:15:41.082253+0000) 2024-03-04T19:15:42.081 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:15:42.081+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:15:42.082436+0000) 2024-03-04T19:15:43.082 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:15:43.082+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:15:43.082672+0000) 2024-03-04T19:15:44.081 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:15:44.081+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:15:44.082946+0000) 2024-03-04T19:15:45.082 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:15:45.082+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:15:45.083181+0000) 2024-03-04T19:15:46.082 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:15:46.081+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:15:46.083373+0000) 2024-03-04T19:15:47.082 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:15:47.082+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:15:47.083617+0000) 2024-03-04T19:15:48.082 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:15:48.082+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:15:48.083933+0000) 2024-03-04T19:15:48.885 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:15:48.893 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:15:49.082 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:15:49.083+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:15:49.084180+0000) 2024-03-04T19:15:50.083 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:15:50.083+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:15:50.084415+0000) 2024-03-04T19:15:51.083 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:15:51.083+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:15:51.084656+0000) 2024-03-04T19:15:52.083 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:15:52.083+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:15:52.084933+0000) 2024-03-04T19:15:53.084 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:15:53.084+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:15:53.085193+0000) 2024-03-04T19:15:54.084 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:15:54.084+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:15:54.085435+0000) 2024-03-04T19:15:55.084 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:15:55.084+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:15:55.085677+0000) 2024-03-04T19:15:56.084 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:15:56.085+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:15:56.085971+0000) 2024-03-04T19:15:57.085 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:15:57.085+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:15:57.086220+0000) 2024-03-04T19:15:58.085 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:15:58.085+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:15:58.086471+0000) 2024-03-04T19:15:59.085 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:15:59.085+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:15:59.086727+0000) 2024-03-04T19:16:00.086 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:16:00.086+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:16:00.086976+0000) 2024-03-04T19:16:01.086 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:16:01.086+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:16:01.087239+0000) 2024-03-04T19:16:02.087 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:16:02.086+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:16:02.087500+0000) 2024-03-04T19:16:03.086 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:16:03.086+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:16:03.087757+0000) 2024-03-04T19:16:04.087 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:16:04.087+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:16:04.088054+0000) 2024-03-04T19:16:05.087 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:16:05.087+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:16:05.088248+0000) 2024-03-04T19:16:06.087 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:16:06.087+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:16:06.088531+0000) 2024-03-04T19:16:07.087 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:16:07.087+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:16:07.088802+0000) 2024-03-04T19:16:08.088 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:16:08.088+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:16:08.089089+0000) 2024-03-04T19:16:09.088 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:16:09.088+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:16:09.089344+0000) 2024-03-04T19:16:10.088 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:16:10.088+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:16:10.089597+0000) 2024-03-04T19:16:11.089 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:16:11.089+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:16:11.089896+0000) 2024-03-04T19:16:12.089 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:16:12.089+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:16:12.090145+0000) 2024-03-04T19:16:13.089 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:16:13.089+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:16:13.090376+0000) 2024-03-04T19:16:14.089 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:16:14.090+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:16:14.090642+0000) 2024-03-04T19:16:15.089 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:16:15.090+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:16:15.090854+0000) 2024-03-04T19:16:16.090 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:16:16.090+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:16:16.091103+0000) 2024-03-04T19:16:17.090 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:16:17.090+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:16:17.091302+0000) 2024-03-04T19:16:18.090 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:16:18.090+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:16:18.091563+0000) 2024-03-04T19:16:19.090 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:16:19.090+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:16:19.091750+0000) 2024-03-04T19:16:19.141 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:16:19.149 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:16:20.090 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:16:20.091+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:16:20.091982+0000) 2024-03-04T19:16:21.091 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:16:21.091+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:16:21.092170+0000) 2024-03-04T19:16:22.091 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:16:22.091+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:16:22.092415+0000) 2024-03-04T19:16:23.091 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:16:23.091+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:16:23.092608+0000) 2024-03-04T19:16:24.092 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:16:24.092+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:16:24.092938+0000) 2024-03-04T19:16:25.092 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:16:25.092+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:16:25.093221+0000) 2024-03-04T19:16:26.092 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:16:26.092+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:16:26.093509+0000) 2024-03-04T19:16:27.092 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:16:27.093+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:16:27.093810+0000) 2024-03-04T19:16:28.093 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:16:28.093+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:16:28.094146+0000) 2024-03-04T19:16:29.093 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:16:29.093+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:16:29.094389+0000) 2024-03-04T19:16:30.093 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:16:30.093+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:16:30.094567+0000) 2024-03-04T19:16:31.093 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:16:31.094+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:16:31.094814+0000) 2024-03-04T19:16:32.094 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:16:32.094+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:16:32.095097+0000) 2024-03-04T19:16:33.094 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:16:33.095+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:16:33.095399+0000) 2024-03-04T19:16:34.094 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:16:34.095+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:16:34.095664+0000) 2024-03-04T19:16:35.095 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:16:35.095+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:16:35.095921+0000) 2024-03-04T19:16:36.095 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:16:36.095+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:16:36.096163+0000) 2024-03-04T19:16:37.095 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:16:37.095+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:16:37.096359+0000) 2024-03-04T19:16:38.095 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:16:38.096+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:16:38.096585+0000) 2024-03-04T19:16:39.096 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:16:39.096+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:16:39.096877+0000) 2024-03-04T19:16:40.096 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:16:40.096+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:16:40.097118+0000) 2024-03-04T19:16:41.096 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:16:41.097+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:16:41.097355+0000) 2024-03-04T19:16:42.096 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:16:42.097+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:16:42.097599+0000) 2024-03-04T19:16:43.096 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:16:43.097+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:16:43.097868+0000) 2024-03-04T19:16:44.097 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:16:44.097+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:16:44.098120+0000) 2024-03-04T19:16:45.097 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:16:45.098+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:16:45.098353+0000) 2024-03-04T19:16:46.097 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:16:46.098+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:16:46.098585+0000) 2024-03-04T19:16:47.097 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:16:47.097+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:16:47.098882+0000) 2024-03-04T19:16:48.098 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:16:48.098+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:16:48.099119+0000) 2024-03-04T19:16:49.098 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:16:49.098+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:16:49.099356+0000) 2024-03-04T19:16:49.294 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:16:49.302 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:16:50.098 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:16:50.098+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:16:50.099588+0000) 2024-03-04T19:16:51.098 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:16:51.098+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:16:51.099847+0000) 2024-03-04T19:16:52.098 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:16:52.098+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:16:52.100039+0000) 2024-03-04T19:16:53.099 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:16:53.099+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:16:53.100276+0000) 2024-03-04T19:16:54.099 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:16:54.099+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:16:54.100519+0000) 2024-03-04T19:16:55.099 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:16:55.099+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:16:55.100754+0000) 2024-03-04T19:16:56.099 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:16:56.100+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:16:56.100994+0000) 2024-03-04T19:16:57.100 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:16:57.100+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:16:57.101237+0000) 2024-03-04T19:16:58.100 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:16:58.100+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:16:58.101470+0000) 2024-03-04T19:16:59.100 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:16:59.100+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:16:59.101707+0000) 2024-03-04T19:17:00.100 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:17:00.101+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:17:00.101942+0000) 2024-03-04T19:17:01.101 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:17:01.101+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:17:01.102178+0000) 2024-03-04T19:17:02.101 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:17:02.101+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:17:02.102441+0000) 2024-03-04T19:17:03.101 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:17:03.101+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:17:03.102685+0000) 2024-03-04T19:17:04.102 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:17:04.102+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:17:04.102951+0000) 2024-03-04T19:17:05.102 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:17:05.102+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:17:05.103207+0000) 2024-03-04T19:17:06.102 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:17:06.102+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:17:06.103460+0000) 2024-03-04T19:17:07.102 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:17:07.102+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:17:07.103723+0000) 2024-03-04T19:17:08.102 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:17:08.103+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:17:08.103919+0000) 2024-03-04T19:17:09.103 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:17:09.103+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:17:09.104160+0000) 2024-03-04T19:17:10.103 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:17:10.103+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:17:10.104428+0000) 2024-03-04T19:17:11.103 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:17:11.103+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:17:11.104683+0000) 2024-03-04T19:17:12.104 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:17:12.104+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:17:12.104924+0000) 2024-03-04T19:17:13.104 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:17:13.104+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:17:13.105162+0000) 2024-03-04T19:17:14.107 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:17:14.104+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:17:14.105405+0000) 2024-03-04T19:17:15.105 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:17:15.104+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:17:15.105648+0000) 2024-03-04T19:17:16.105 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:17:16.105+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:17:16.105930+0000) 2024-03-04T19:17:17.105 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:17:17.105+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:17:17.106187+0000) 2024-03-04T19:17:18.105 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:17:18.105+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:17:18.106421+0000) 2024-03-04T19:17:19.105 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:17:19.105+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:17:19.106603+0000) 2024-03-04T19:17:19.507 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:17:19.515 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:17:20.105 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:17:20.106+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:17:20.106864+0000) 2024-03-04T19:17:21.106 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:17:21.106+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:17:21.107108+0000) 2024-03-04T19:17:22.106 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:17:22.106+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:17:22.107344+0000) 2024-03-04T19:17:23.106 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:17:23.107+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:17:23.107584+0000) 2024-03-04T19:17:24.106 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:17:24.107+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:17:24.107861+0000) 2024-03-04T19:17:25.107 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:17:25.107+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:17:25.108107+0000) 2024-03-04T19:17:26.107 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:17:26.107+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:17:26.108340+0000) 2024-03-04T19:17:27.107 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:17:27.108+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:17:27.108638+0000) 2024-03-04T19:17:28.108 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:17:28.108+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:17:28.108929+0000) 2024-03-04T19:17:29.108 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:17:29.108+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:17:29.109188+0000) 2024-03-04T19:17:30.108 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:17:30.109+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:17:30.109421+0000) 2024-03-04T19:17:31.108 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:17:31.109+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:17:31.109663+0000) 2024-03-04T19:17:32.108 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:17:32.109+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:17:32.109866+0000) 2024-03-04T19:17:33.109 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:17:33.109+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:17:33.110105+0000) 2024-03-04T19:17:34.109 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:17:34.109+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:17:34.110353+0000) 2024-03-04T19:17:35.109 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:17:35.110+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:17:35.110591+0000) 2024-03-04T19:17:36.109 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:17:36.110+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:17:36.110881+0000) 2024-03-04T19:17:37.110 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:17:37.110+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:17:37.111146+0000) 2024-03-04T19:17:38.110 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:17:38.111+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:17:38.111383+0000) 2024-03-04T19:17:39.110 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:17:39.111+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:17:39.111624+0000) 2024-03-04T19:17:40.110 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:17:40.111+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:17:40.111910+0000) 2024-03-04T19:17:41.111 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:17:41.111+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:17:41.112163+0000) 2024-03-04T19:17:42.111 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:17:42.112+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:17:42.112431+0000) 2024-03-04T19:17:43.113 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:17:43.112+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:17:43.112675+0000) 2024-03-04T19:17:44.112 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:17:44.112+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:17:44.112924+0000) 2024-03-04T19:17:45.112 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:17:45.112+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:17:45.113156+0000) 2024-03-04T19:17:46.112 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:17:46.113+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:17:46.113397+0000) 2024-03-04T19:17:47.113 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:17:47.113+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:17:47.113636+0000) 2024-03-04T19:17:48.113 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:17:48.113+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:17:48.113919+0000) 2024-03-04T19:17:49.113 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:17:49.113+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:17:49.114158+0000) 2024-03-04T19:17:49.721 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:17:49.729 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:17:50.113 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:17:50.113+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:17:50.114397+0000) 2024-03-04T19:17:51.113 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:17:51.112+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:17:51.114689+0000) 2024-03-04T19:17:52.114 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:17:52.113+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:17:52.114941+0000) 2024-03-04T19:17:53.114 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:17:53.114+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:17:53.115187+0000) 2024-03-04T19:17:54.114 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:17:54.114+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:17:54.115435+0000) 2024-03-04T19:17:55.114 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:17:55.114+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:17:55.115683+0000) 2024-03-04T19:17:56.115 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:17:56.114+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:17:56.115945+0000) 2024-03-04T19:17:57.115 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:17:57.115+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:17:57.116184+0000) 2024-03-04T19:17:58.115 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:17:58.115+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:17:58.116421+0000) 2024-03-04T19:17:59.115 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:17:59.115+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:17:59.116662+0000) 2024-03-04T19:18:00.116 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:18:00.115+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:18:00.116923+0000) 2024-03-04T19:18:01.116 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:18:01.116+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:18:01.117211+0000) 2024-03-04T19:18:02.116 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:18:02.116+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:18:02.117472+0000) 2024-03-04T19:18:03.116 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:18:03.116+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:18:03.117740+0000) 2024-03-04T19:18:04.117 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:18:04.117+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:18:04.118011+0000) 2024-03-04T19:18:05.119 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:18:05.117+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:18:05.118288+0000) 2024-03-04T19:18:06.117 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:18:06.117+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:18:06.118573+0000) 2024-03-04T19:18:07.118 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:18:07.118+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:18:07.118867+0000) 2024-03-04T19:18:08.118 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:18:08.118+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:18:08.119128+0000) 2024-03-04T19:18:09.118 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:18:09.118+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:18:09.119460+0000) 2024-03-04T19:18:10.118 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:18:10.118+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:18:10.119721+0000) 2024-03-04T19:18:11.119 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:18:11.119+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:18:11.119988+0000) 2024-03-04T19:18:12.119 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:18:12.119+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:18:12.120228+0000) 2024-03-04T19:18:13.119 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:18:13.119+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:18:13.120476+0000) 2024-03-04T19:18:14.119 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:18:14.120+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:18:14.120759+0000) 2024-03-04T19:18:15.120 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:18:15.120+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:18:15.121003+0000) 2024-03-04T19:18:16.120 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:18:16.120+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:18:16.121308+0000) 2024-03-04T19:18:17.120 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:18:17.120+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:18:17.121568+0000) 2024-03-04T19:18:18.120 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:18:18.121+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:18:18.121807+0000) 2024-03-04T19:18:19.121 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:18:19.121+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:18:19.122103+0000) 2024-03-04T19:18:19.867 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:18:19.875 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:18:20.121 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:18:20.121+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:18:20.122402+0000) 2024-03-04T19:18:21.121 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:18:21.122+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:18:21.122647+0000) 2024-03-04T19:18:22.122 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:18:22.122+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:18:22.122922+0000) 2024-03-04T19:18:23.122 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:18:23.122+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:18:23.123173+0000) 2024-03-04T19:18:24.122 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:18:24.122+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:18:24.123418+0000) 2024-03-04T19:18:25.122 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:18:25.123+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:18:25.123661+0000) 2024-03-04T19:18:26.122 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:18:26.123+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:18:26.123937+0000) 2024-03-04T19:18:27.123 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:18:27.123+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:18:27.124183+0000) 2024-03-04T19:18:28.123 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:18:28.123+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:18:28.124416+0000) 2024-03-04T19:18:29.123 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:18:29.124+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:18:29.124662+0000) 2024-03-04T19:18:30.124 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:18:30.124+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:18:30.124938+0000) 2024-03-04T19:18:31.124 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:18:31.124+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:18:31.125174+0000) 2024-03-04T19:18:32.124 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:18:32.124+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:18:32.125413+0000) 2024-03-04T19:18:33.124 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:18:33.125+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:18:33.125656+0000) 2024-03-04T19:18:34.125 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:18:34.125+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:18:34.125925+0000) 2024-03-04T19:18:35.125 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:18:35.125+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:18:35.126163+0000) 2024-03-04T19:18:36.125 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:18:36.126+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:18:36.126408+0000) 2024-03-04T19:18:37.125 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:18:37.125+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:18:37.126599+0000) 2024-03-04T19:18:38.125 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:18:38.126+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:18:38.126850+0000) 2024-03-04T19:18:39.126 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:18:39.126+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:18:39.127051+0000) 2024-03-04T19:18:40.126 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:18:40.126+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:18:40.127315+0000) 2024-03-04T19:18:41.126 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:18:41.126+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:18:41.127534+0000) 2024-03-04T19:18:42.126 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:18:42.127+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:18:42.127772+0000) 2024-03-04T19:18:43.127 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:18:43.127+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:18:43.128019+0000) 2024-03-04T19:18:44.127 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:18:44.128+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:18:44.128263+0000) 2024-03-04T19:18:45.127 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:18:45.128+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:18:45.128494+0000) 2024-03-04T19:18:46.127 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:18:46.128+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:18:46.128727+0000) 2024-03-04T19:18:47.128 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:18:47.128+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:18:47.128999+0000) 2024-03-04T19:18:48.128 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:18:48.129+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:18:48.129264+0000) 2024-03-04T19:18:49.128 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:18:49.129+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:18:49.129507+0000) 2024-03-04T19:18:50.097 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:18:50.106 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:18:50.128 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:18:50.128+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:18:50.129698+0000) 2024-03-04T19:18:51.129 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:18:51.129+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:18:51.129989+0000) 2024-03-04T19:18:52.129 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:18:52.129+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:18:52.130242+0000) 2024-03-04T19:18:53.129 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:18:53.129+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:18:53.130484+0000) 2024-03-04T19:18:54.129 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:18:54.129+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:18:54.130730+0000) 2024-03-04T19:18:55.129 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:18:55.129+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:18:55.130964+0000) 2024-03-04T19:18:56.130 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:18:56.130+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:18:56.131198+0000) 2024-03-04T19:18:57.130 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:18:57.130+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:18:57.131441+0000) 2024-03-04T19:18:58.130 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:18:58.129+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:18:58.131694+0000) 2024-03-04T19:18:59.131 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:18:59.130+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:18:59.131933+0000) 2024-03-04T19:19:00.131 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:19:00.131+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:19:00.132165+0000) 2024-03-04T19:19:01.131 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:19:01.131+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:19:01.132425+0000) 2024-03-04T19:19:02.131 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:19:02.131+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:19:02.132673+0000) 2024-03-04T19:19:03.132 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:19:03.132+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:19:03.132918+0000) 2024-03-04T19:19:04.132 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:19:04.132+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:19:04.133133+0000) 2024-03-04T19:19:05.132 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:19:05.132+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:19:05.133395+0000) 2024-03-04T19:19:06.132 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:19:06.132+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:19:06.133659+0000) 2024-03-04T19:19:07.132 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:19:07.133+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:19:07.133945+0000) 2024-03-04T19:19:08.133 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:19:08.133+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:19:08.134255+0000) 2024-03-04T19:19:09.133 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:19:09.133+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:19:09.134443+0000) 2024-03-04T19:19:10.133 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:19:10.133+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:19:10.134676+0000) 2024-03-04T19:19:11.133 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:19:11.134+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:19:11.134950+0000) 2024-03-04T19:19:12.134 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:19:12.134+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:19:12.135185+0000) 2024-03-04T19:19:13.134 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:19:13.134+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:19:13.135444+0000) 2024-03-04T19:19:14.134 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:19:14.134+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:19:14.135719+0000) 2024-03-04T19:19:15.135 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:19:15.135+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:19:15.135976+0000) 2024-03-04T19:19:16.135 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:19:16.135+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:19:16.136215+0000) 2024-03-04T19:19:17.135 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:19:17.135+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:19:17.136462+0000) 2024-03-04T19:19:18.135 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:19:18.136+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:19:18.136699+0000) 2024-03-04T19:19:19.136 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:19:19.136+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:19:19.136957+0000) 2024-03-04T19:19:20.136 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:19:20.136+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:19:20.137190+0000) 2024-03-04T19:19:20.252 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:19:20.260 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:19:21.136 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:19:21.136+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:19:21.137428+0000) 2024-03-04T19:19:22.136 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:19:22.137+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:19:22.137668+0000) 2024-03-04T19:19:23.137 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:19:23.137+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:19:23.137946+0000) 2024-03-04T19:19:24.137 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:19:24.137+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:19:24.138194+0000) 2024-03-04T19:19:25.137 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:19:25.137+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:19:25.138434+0000) 2024-03-04T19:19:26.137 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:19:26.138+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:19:26.138685+0000) 2024-03-04T19:19:27.137 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:19:27.138+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:19:27.138931+0000) 2024-03-04T19:19:28.138 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:19:28.138+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:19:28.139164+0000) 2024-03-04T19:19:29.138 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:19:29.138+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:19:29.139413+0000) 2024-03-04T19:19:30.138 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:19:30.139+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:19:30.139650+0000) 2024-03-04T19:19:31.139 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:19:31.139+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:19:31.139928+0000) 2024-03-04T19:19:32.139 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:19:32.139+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:19:32.140167+0000) 2024-03-04T19:19:33.139 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:19:33.139+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:19:33.140413+0000) 2024-03-04T19:19:34.139 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:19:34.139+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:19:34.140658+0000) 2024-03-04T19:19:35.140 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:19:35.140+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:19:35.140921+0000) 2024-03-04T19:19:36.140 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:19:36.140+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:19:36.141182+0000) 2024-03-04T19:19:37.140 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:19:37.141+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:19:37.141425+0000) 2024-03-04T19:19:38.141 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:19:38.141+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:19:38.141666+0000) 2024-03-04T19:19:39.140 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:19:39.141+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:19:39.141942+0000) 2024-03-04T19:19:40.141 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:19:40.141+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:19:40.142191+0000) 2024-03-04T19:19:41.141 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:19:41.142+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:19:41.142427+0000) 2024-03-04T19:19:42.141 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:19:42.142+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:19:42.142666+0000) 2024-03-04T19:19:43.141 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:19:43.142+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:19:43.142947+0000) 2024-03-04T19:19:44.142 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:19:44.142+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:19:44.143192+0000) 2024-03-04T19:19:45.142 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:19:45.143+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:19:45.143428+0000) 2024-03-04T19:19:46.142 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:19:46.143+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:19:46.143663+0000) 2024-03-04T19:19:47.143 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:19:47.143+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:19:47.143942+0000) 2024-03-04T19:19:48.143 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:19:48.143+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:19:48.144183+0000) 2024-03-04T19:19:49.143 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:19:49.144+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:19:49.144424+0000) 2024-03-04T19:19:50.143 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:19:50.143+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:19:50.144678+0000) 2024-03-04T19:19:50.574 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:19:50.585 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:19:51.143 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:19:51.144+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:19:51.144968+0000) 2024-03-04T19:19:52.144 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:19:52.145+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:19:52.145207+0000) 2024-03-04T19:19:53.144 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:19:53.145+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:19:53.145448+0000) 2024-03-04T19:19:54.144 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:19:54.145+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:19:54.145697+0000) 2024-03-04T19:19:55.144 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:19:55.144+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:19:55.145971+0000) 2024-03-04T19:19:56.145 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:19:56.145+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:19:56.146216+0000) 2024-03-04T19:19:57.145 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:19:57.145+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:19:57.146462+0000) 2024-03-04T19:19:58.145 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:19:58.145+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:19:58.146702+0000) 2024-03-04T19:19:59.146 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:19:59.145+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:19:59.146969+0000) 2024-03-04T19:20:00.146 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:20:00.146+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:20:00.147211+0000) 2024-03-04T19:20:01.146 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:20:01.146+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:20:01.147461+0000) 2024-03-04T19:20:02.146 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:20:02.146+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:20:02.147710+0000) 2024-03-04T19:20:03.147 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:20:03.146+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:20:03.147955+0000) 2024-03-04T19:20:04.147 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:20:04.147+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:20:04.148161+0000) 2024-03-04T19:20:05.147 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:20:05.147+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:20:05.148432+0000) 2024-03-04T19:20:06.147 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:20:06.147+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:20:06.148690+0000) 2024-03-04T19:20:07.148 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:20:07.148+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:20:07.148956+0000) 2024-03-04T19:20:08.148 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:20:08.148+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:20:08.149174+0000) 2024-03-04T19:20:09.148 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:20:09.148+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:20:09.149419+0000) 2024-03-04T19:20:10.148 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:20:10.148+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:20:10.149664+0000) 2024-03-04T19:20:11.148 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:20:11.149+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:20:11.149996+0000) 2024-03-04T19:20:12.149 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:20:12.149+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:20:12.150203+0000) 2024-03-04T19:20:13.149 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:20:13.149+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:20:13.150447+0000) 2024-03-04T19:20:14.149 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:20:14.149+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:20:14.150698+0000) 2024-03-04T19:20:15.150 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:20:15.150+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:20:15.150935+0000) 2024-03-04T19:20:16.150 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:20:16.150+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:20:16.151177+0000) 2024-03-04T19:20:17.150 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:20:17.150+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:20:17.151378+0000) 2024-03-04T19:20:18.150 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:20:18.150+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:20:18.151604+0000) 2024-03-04T19:20:19.151 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:20:19.151+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:20:19.151868+0000) 2024-03-04T19:20:20.151 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:20:20.151+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:20:20.152071+0000) 2024-03-04T19:20:20.754 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:20:20.762 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:20:21.151 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:20:21.151+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:20:21.152347+0000) 2024-03-04T19:20:22.151 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:20:22.151+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:20:22.152605+0000) 2024-03-04T19:20:23.151 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:20:23.152+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:20:23.152922+0000) 2024-03-04T19:20:24.152 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:20:24.152+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:20:24.153194+0000) 2024-03-04T19:20:25.152 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:20:25.152+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:20:25.153453+0000) 2024-03-04T19:20:26.152 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:20:26.153+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:20:26.153701+0000) 2024-03-04T19:20:27.153 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:20:27.153+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:20:27.153960+0000) 2024-03-04T19:20:28.153 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:20:28.153+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:20:28.154174+0000) 2024-03-04T19:20:29.153 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:20:29.153+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:20:29.154436+0000) 2024-03-04T19:20:30.153 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:20:30.154+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:20:30.154684+0000) 2024-03-04T19:20:31.154 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:20:31.154+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:20:31.154936+0000) 2024-03-04T19:20:32.154 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:20:32.154+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:20:32.155199+0000) 2024-03-04T19:20:33.154 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:20:33.154+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:20:33.155458+0000) 2024-03-04T19:20:34.154 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:20:34.155+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:20:34.155720+0000) 2024-03-04T19:20:35.155 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:20:35.155+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:20:35.155989+0000) 2024-03-04T19:20:36.155 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:20:36.155+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:20:36.156264+0000) 2024-03-04T19:20:37.155 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:20:37.156+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:20:37.156531+0000) 2024-03-04T19:20:38.155 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:20:38.156+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:20:38.156790+0000) 2024-03-04T19:20:39.156 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:20:39.156+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:20:39.157123+0000) 2024-03-04T19:20:40.156 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:20:40.156+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:20:40.157318+0000) 2024-03-04T19:20:41.156 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:20:41.157+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:20:41.157574+0000) 2024-03-04T19:20:42.156 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:20:42.157+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:20:42.157874+0000) 2024-03-04T19:20:43.157 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:20:43.157+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:20:43.158138+0000) 2024-03-04T19:20:44.157 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:20:44.158+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:20:44.158403+0000) 2024-03-04T19:20:45.157 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:20:45.158+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:20:45.158653+0000) 2024-03-04T19:20:46.158 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:20:46.158+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:20:46.158942+0000) 2024-03-04T19:20:47.158 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:20:47.158+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:20:47.159228+0000) 2024-03-04T19:20:48.158 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:20:48.159+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:20:48.159497+0000) 2024-03-04T19:20:49.158 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:20:49.159+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:20:49.159773+0000) 2024-03-04T19:20:50.159 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:20:50.159+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:20:50.160065+0000) 2024-03-04T19:20:50.900 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:20:50.908 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:20:51.159 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:20:51.160+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:20:51.160350+0000) 2024-03-04T19:20:52.159 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:20:52.159+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:20:52.160606+0000) 2024-03-04T19:20:53.160 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:20:53.160+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:20:53.160854+0000) 2024-03-04T19:20:54.160 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:20:54.160+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:20:54.161055+0000) 2024-03-04T19:20:55.160 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:20:55.160+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:20:55.161245+0000) 2024-03-04T19:20:56.160 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:20:56.160+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:20:56.161434+0000) 2024-03-04T19:20:57.160 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:20:57.161+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:20:57.161633+0000) 2024-03-04T19:20:58.160 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:20:58.160+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:20:58.161855+0000) 2024-03-04T19:20:59.161 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:20:59.160+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:20:59.162093+0000) 2024-03-04T19:21:00.161 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:21:00.160+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:21:00.162319+0000) 2024-03-04T19:21:01.161 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:21:01.161+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:21:01.162556+0000) 2024-03-04T19:21:02.162 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:21:02.161+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:21:02.162806+0000) 2024-03-04T19:21:03.162 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:21:03.162+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:21:03.163094+0000) 2024-03-04T19:21:04.162 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:21:04.162+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:21:04.163346+0000) 2024-03-04T19:21:05.162 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:21:05.161+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:21:05.163604+0000) 2024-03-04T19:21:06.162 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:21:06.162+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:21:06.163785+0000) 2024-03-04T19:21:07.163 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:21:07.162+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:21:07.163951+0000) 2024-03-04T19:21:08.163 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:21:08.163+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:21:08.164191+0000) 2024-03-04T19:21:09.163 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:21:09.163+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:21:09.164431+0000) 2024-03-04T19:21:10.164 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:21:10.163+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:21:10.164683+0000) 2024-03-04T19:21:11.164 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:21:11.164+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:21:11.164944+0000) 2024-03-04T19:21:12.164 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:21:12.164+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:21:12.165209+0000) 2024-03-04T19:21:13.164 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:21:13.164+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:21:13.165472+0000) 2024-03-04T19:21:14.165 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:21:14.164+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:21:14.165710+0000) 2024-03-04T19:21:15.165 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:21:15.165+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:21:15.165968+0000) 2024-03-04T19:21:16.165 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:21:16.165+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:21:16.166205+0000) 2024-03-04T19:21:17.165 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:21:17.165+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:21:17.166447+0000) 2024-03-04T19:21:18.165 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:21:18.165+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:21:18.166696+0000) 2024-03-04T19:21:19.166 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:21:19.166+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:21:19.166972+0000) 2024-03-04T19:21:20.166 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:21:20.166+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:21:20.167166+0000) 2024-03-04T19:21:21.106 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:21:21.114 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:21:21.166 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:21:21.166+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:21:21.167383+0000) 2024-03-04T19:21:22.166 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:21:22.166+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:21:22.167631+0000) 2024-03-04T19:21:23.167 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:21:23.167+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:21:23.167939+0000) 2024-03-04T19:21:24.167 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:21:24.167+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:21:24.168185+0000) 2024-03-04T19:21:25.167 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:21:25.167+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:21:25.168416+0000) 2024-03-04T19:21:26.167 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:21:26.167+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:21:26.168653+0000) 2024-03-04T19:21:27.168 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:21:27.168+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:21:27.168951+0000) 2024-03-04T19:21:28.168 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:21:28.168+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:21:28.169190+0000) 2024-03-04T19:21:29.168 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:21:29.168+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:21:29.169449+0000) 2024-03-04T19:21:30.168 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:21:30.169+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:21:30.169688+0000) 2024-03-04T19:21:31.169 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:21:31.169+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:21:31.169928+0000) 2024-03-04T19:21:32.169 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:21:32.169+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:21:32.170173+0000) 2024-03-04T19:21:33.169 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:21:33.169+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:21:33.170416+0000) 2024-03-04T19:21:34.169 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:21:34.170+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:21:34.170668+0000) 2024-03-04T19:21:35.170 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:21:35.170+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:21:35.170931+0000) 2024-03-04T19:21:36.170 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:21:36.170+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:21:36.171123+0000) 2024-03-04T19:21:37.170 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:21:37.170+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:21:37.171413+0000) 2024-03-04T19:21:38.170 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:21:38.171+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:21:38.171659+0000) 2024-03-04T19:21:39.171 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:21:39.171+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:21:39.171938+0000) 2024-03-04T19:21:40.171 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:21:40.171+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:21:40.172164+0000) 2024-03-04T19:21:41.171 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:21:41.171+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:21:41.172400+0000) 2024-03-04T19:21:42.171 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:21:42.171+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:21:42.172589+0000) 2024-03-04T19:21:43.172 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:21:43.172+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:21:43.172866+0000) 2024-03-04T19:21:44.172 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:21:44.172+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:21:44.173114+0000) 2024-03-04T19:21:45.172 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:21:45.172+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:21:45.173347+0000) 2024-03-04T19:21:46.172 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:21:46.173+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:21:46.173587+0000) 2024-03-04T19:21:47.173 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:21:47.173+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:21:47.173869+0000) 2024-03-04T19:21:48.173 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:21:48.173+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:21:48.174092+0000) 2024-03-04T19:21:49.174 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:21:49.173+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:21:49.174280+0000) 2024-03-04T19:21:50.174 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:21:50.174+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:21:50.174514+0000) 2024-03-04T19:21:51.174 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:21:51.174+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:21:51.174760+0000) 2024-03-04T19:21:51.269 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:21:51.277 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:21:52.174 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:21:52.174+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:21:52.174996+0000) 2024-03-04T19:21:53.174 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:21:53.175+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:21:53.175240+0000) 2024-03-04T19:21:54.174 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:21:54.175+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:21:54.175491+0000) 2024-03-04T19:21:55.175 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:21:55.175+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:21:55.175730+0000) 2024-03-04T19:21:56.175 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:21:56.175+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:21:56.175974+0000) 2024-03-04T19:21:57.175 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:21:57.176+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:21:57.176211+0000) 2024-03-04T19:21:58.175 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:21:58.176+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:21:58.176452+0000) 2024-03-04T19:21:59.175 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:21:59.176+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:21:59.176707+0000) 2024-03-04T19:22:00.176 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:22:00.175+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:22:00.176943+0000) 2024-03-04T19:22:01.176 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:22:01.176+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:22:01.177189+0000) 2024-03-04T19:22:02.176 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:22:02.175+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:22:02.177442+0000) 2024-03-04T19:22:03.177 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:22:03.176+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:22:03.177689+0000) 2024-03-04T19:22:04.177 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:22:04.176+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:22:04.177941+0000) 2024-03-04T19:22:05.177 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:22:05.177+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:22:05.178181+0000) 2024-03-04T19:22:06.178 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:22:06.177+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:22:06.178417+0000) 2024-03-04T19:22:07.177 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:22:07.176+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:22:07.178679+0000) 2024-03-04T19:22:08.178 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:22:08.177+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:22:08.178941+0000) 2024-03-04T19:22:09.178 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:22:09.178+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:22:09.179185+0000) 2024-03-04T19:22:10.178 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:22:10.178+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:22:10.179413+0000) 2024-03-04T19:22:11.179 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:22:11.178+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:22:11.179678+0000) 2024-03-04T19:22:12.179 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:22:12.179+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:22:12.179949+0000) 2024-03-04T19:22:13.179 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:22:13.179+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:22:13.180191+0000) 2024-03-04T19:22:14.179 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:22:14.179+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:22:14.180463+0000) 2024-03-04T19:22:15.180 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:22:15.179+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:22:15.180705+0000) 2024-03-04T19:22:16.180 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:22:16.180+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:22:16.180964+0000) 2024-03-04T19:22:17.183 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:22:17.180+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:22:17.181154+0000) 2024-03-04T19:22:18.180 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:22:18.180+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:22:18.181425+0000) 2024-03-04T19:22:19.180 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:22:19.180+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:22:19.181623+0000) 2024-03-04T19:22:20.181 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:22:20.181+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:22:20.181906+0000) 2024-03-04T19:22:21.181 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:22:21.181+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:22:21.182147+0000) 2024-03-04T19:22:21.465 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:22:21.473 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:22:22.181 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:22:22.181+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:22:22.182409+0000) 2024-03-04T19:22:23.181 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:22:23.181+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:22:23.182660+0000) 2024-03-04T19:22:24.182 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:22:24.182+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:22:24.182948+0000) 2024-03-04T19:22:25.183 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:22:25.182+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:22:25.183134+0000) 2024-03-04T19:22:26.182 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:22:26.182+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:22:26.183367+0000) 2024-03-04T19:22:27.182 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:22:27.182+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:22:27.183608+0000) 2024-03-04T19:22:28.183 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:22:28.183+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:22:28.183877+0000) 2024-03-04T19:22:29.183 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:22:29.183+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:22:29.184142+0000) 2024-03-04T19:22:30.183 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:22:30.183+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:22:30.184375+0000) 2024-03-04T19:22:31.183 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:22:31.183+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:22:31.184655+0000) 2024-03-04T19:22:32.184 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:22:32.184+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:22:32.184933+0000) 2024-03-04T19:22:33.184 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:22:33.184+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:22:33.185194+0000) 2024-03-04T19:22:34.184 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:22:34.184+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:22:34.185444+0000) 2024-03-04T19:22:35.184 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:22:35.185+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:22:35.185683+0000) 2024-03-04T19:22:36.185 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:22:36.185+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:22:36.185923+0000) 2024-03-04T19:22:37.185 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:22:37.185+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:22:37.186192+0000) 2024-03-04T19:22:38.186 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:22:38.185+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:22:38.186464+0000) 2024-03-04T19:22:39.185 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:22:39.186+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:22:39.186714+0000) 2024-03-04T19:22:40.186 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:22:40.186+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:22:40.186947+0000) 2024-03-04T19:22:41.186 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:22:41.186+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:22:41.187186+0000) 2024-03-04T19:22:42.186 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:22:42.186+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:22:42.187421+0000) 2024-03-04T19:22:43.186 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:22:43.187+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:22:43.187659+0000) 2024-03-04T19:22:44.187 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:22:44.187+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:22:44.187941+0000) 2024-03-04T19:22:45.187 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:22:45.187+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:22:45.188128+0000) 2024-03-04T19:22:46.187 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:22:46.187+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:22:46.188361+0000) 2024-03-04T19:22:47.187 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:22:47.188+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:22:47.188602+0000) 2024-03-04T19:22:48.188 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:22:48.188+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:22:48.188870+0000) 2024-03-04T19:22:49.188 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:22:49.188+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:22:49.189114+0000) 2024-03-04T19:22:50.188 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:22:50.189+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:22:50.189351+0000) 2024-03-04T19:22:51.188 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:22:51.189+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:22:51.189591+0000) 2024-03-04T19:22:51.695 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:22:51.704 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:22:52.189 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:22:52.189+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:22:52.189867+0000) 2024-03-04T19:22:53.189 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:22:53.189+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:22:53.190104+0000) 2024-03-04T19:22:54.189 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:22:54.190+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:22:54.190354+0000) 2024-03-04T19:22:55.189 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:22:55.190+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:22:55.190608+0000) 2024-03-04T19:22:56.190 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:22:56.190+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:22:56.190878+0000) 2024-03-04T19:22:57.190 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:22:57.190+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:22:57.191116+0000) 2024-03-04T19:22:58.190 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:22:58.191+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:22:58.191344+0000) 2024-03-04T19:22:59.190 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:22:59.191+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:22:59.191589+0000) 2024-03-04T19:23:00.191 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:23:00.191+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:23:00.191869+0000) 2024-03-04T19:23:01.191 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:23:01.191+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:23:01.192108+0000) 2024-03-04T19:23:02.191 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:23:02.192+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:23:02.192341+0000) 2024-03-04T19:23:03.191 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:23:03.191+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:23:03.192582+0000) 2024-03-04T19:23:04.192 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:23:04.191+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:23:04.192854+0000) 2024-03-04T19:23:05.192 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:23:05.192+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:23:05.193093+0000) 2024-03-04T19:23:06.192 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:23:06.192+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:23:06.193330+0000) 2024-03-04T19:23:07.192 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:23:07.192+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:23:07.193571+0000) 2024-03-04T19:23:08.193 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:23:08.192+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:23:08.193858+0000) 2024-03-04T19:23:09.193 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:23:09.193+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:23:09.194123+0000) 2024-03-04T19:23:10.193 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:23:10.193+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:23:10.194360+0000) 2024-03-04T19:23:11.193 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:23:11.193+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:23:11.194602+0000) 2024-03-04T19:23:12.194 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:23:12.193+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:23:12.194880+0000) 2024-03-04T19:23:13.194 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:23:13.194+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:23:13.195196+0000) 2024-03-04T19:23:14.194 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:23:14.194+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:23:14.195445+0000) 2024-03-04T19:23:15.194 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:23:15.194+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:23:15.195685+0000) 2024-03-04T19:23:16.195 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:23:16.195+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:23:16.195922+0000) 2024-03-04T19:23:17.195 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:23:17.195+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:23:17.196159+0000) 2024-03-04T19:23:18.196 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:23:18.195+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:23:18.196399+0000) 2024-03-04T19:23:19.196 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:23:19.195+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:23:19.196637+0000) 2024-03-04T19:23:20.196 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:23:20.196+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:23:20.196914+0000) 2024-03-04T19:23:21.196 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:23:21.196+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:23:21.197148+0000) 2024-03-04T19:23:21.908 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:23:21.917 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:23:22.196 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:23:22.196+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:23:22.197344+0000) 2024-03-04T19:23:23.196 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:23:23.196+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:23:23.197582+0000) 2024-03-04T19:23:24.197 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:23:24.197+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:23:24.197863+0000) 2024-03-04T19:23:25.197 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:23:25.197+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:23:25.198105+0000) 2024-03-04T19:23:26.197 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:23:26.197+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:23:26.198283+0000) 2024-03-04T19:23:27.198 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:23:27.197+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:23:27.198556+0000) 2024-03-04T19:23:28.198 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:23:28.198+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:23:28.198788+0000) 2024-03-04T19:23:29.198 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:23:29.198+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:23:29.198953+0000) 2024-03-04T19:23:30.199 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:23:30.198+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:23:30.199218+0000) 2024-03-04T19:23:31.198 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:23:31.198+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:23:31.199455+0000) 2024-03-04T19:23:32.199 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:23:32.198+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:23:32.199680+0000) 2024-03-04T19:23:33.199 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:23:33.199+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:23:33.199927+0000) 2024-03-04T19:23:34.199 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:23:34.199+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:23:34.200170+0000) 2024-03-04T19:23:35.200 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:23:35.199+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:23:35.200410+0000) 2024-03-04T19:23:36.200 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:23:36.200+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:23:36.200644+0000) 2024-03-04T19:23:37.200 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:23:37.200+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:23:37.200916+0000) 2024-03-04T19:23:38.200 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:23:38.200+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:23:38.201103+0000) 2024-03-04T19:23:39.200 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:23:39.200+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:23:39.201365+0000) 2024-03-04T19:23:40.201 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:23:40.201+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:23:40.201628+0000) 2024-03-04T19:23:41.201 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:23:41.201+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:23:41.201901+0000) 2024-03-04T19:23:42.201 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:23:42.201+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:23:42.202135+0000) 2024-03-04T19:23:43.201 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:23:43.201+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:23:43.202378+0000) 2024-03-04T19:23:44.202 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:23:44.202+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:23:44.202622+0000) 2024-03-04T19:23:45.202 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:23:45.202+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:23:45.202908+0000) 2024-03-04T19:23:46.202 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:23:46.202+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:23:46.203142+0000) 2024-03-04T19:23:47.202 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:23:47.202+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:23:47.203382+0000) 2024-03-04T19:23:48.202 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:23:48.203+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:23:48.203617+0000) 2024-03-04T19:23:49.203 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:23:49.203+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:23:49.203891+0000) 2024-03-04T19:23:50.203 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:23:50.203+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:23:50.204123+0000) 2024-03-04T19:23:51.203 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:23:51.204+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:23:51.204371+0000) 2024-03-04T19:23:52.114 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:23:52.122 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:23:52.203 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:23:52.203+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:23:52.204560+0000) 2024-03-04T19:23:53.204 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:23:53.204+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:23:53.204793+0000) 2024-03-04T19:23:54.204 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:23:54.204+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:23:54.205075+0000) 2024-03-04T19:23:55.204 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:23:55.204+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:23:55.205300+0000) 2024-03-04T19:23:56.204 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:23:56.205+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:23:56.205537+0000) 2024-03-04T19:23:57.205 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:23:57.205+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:23:57.205774+0000) 2024-03-04T19:23:58.205 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:23:58.205+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:23:58.206001+0000) 2024-03-04T19:23:59.205 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:23:59.206+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:23:59.206242+0000) 2024-03-04T19:24:00.205 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:24:00.206+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:24:00.206473+0000) 2024-03-04T19:24:01.206 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:24:01.206+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:24:01.206764+0000) 2024-03-04T19:24:02.206 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:24:02.206+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:24:02.206997+0000) 2024-03-04T19:24:03.206 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:24:03.207+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:24:03.207243+0000) 2024-03-04T19:24:04.206 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:24:04.207+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:24:04.207489+0000) 2024-03-04T19:24:05.207 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:24:05.205+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:24:05.207737+0000) 2024-03-04T19:24:06.207 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:24:06.206+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:24:06.207989+0000) 2024-03-04T19:24:07.207 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:24:07.207+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:24:07.208225+0000) 2024-03-04T19:24:08.207 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:24:08.207+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:24:08.208462+0000) 2024-03-04T19:24:09.208 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:24:09.207+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:24:09.208701+0000) 2024-03-04T19:24:10.208 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:24:10.207+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:24:10.208958+0000) 2024-03-04T19:24:11.208 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:24:11.208+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:24:11.209218+0000) 2024-03-04T19:24:12.209 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:24:12.207+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:24:12.209396+0000) 2024-03-04T19:24:13.209 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:24:13.208+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:24:13.209655+0000) 2024-03-04T19:24:14.209 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:24:14.208+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:24:14.209924+0000) 2024-03-04T19:24:15.209 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:24:15.209+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:24:15.210197+0000) 2024-03-04T19:24:16.210 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:24:16.209+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:24:16.210410+0000) 2024-03-04T19:24:17.210 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:24:17.209+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:24:17.210666+0000) 2024-03-04T19:24:18.210 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:24:18.210+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:24:18.210969+0000) 2024-03-04T19:24:19.210 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:24:19.210+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:24:19.211218+0000) 2024-03-04T19:24:20.211 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:24:20.210+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:24:20.211508+0000) 2024-03-04T19:24:21.211 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:24:21.210+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:24:21.211757+0000) 2024-03-04T19:24:22.211 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:24:22.211+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:24:22.212051+0000) 2024-03-04T19:24:22.343 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:24:22.352 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:24:23.211 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:24:23.211+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:24:23.212344+0000) 2024-03-04T19:24:24.212 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:24:24.211+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:24:24.212651+0000) 2024-03-04T19:24:25.212 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:24:25.212+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:24:25.212967+0000) 2024-03-04T19:24:26.212 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:24:26.212+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:24:26.213265+0000) 2024-03-04T19:24:27.213 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:24:27.212+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:24:27.213587+0000) 2024-03-04T19:24:28.213 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:24:28.213+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:24:28.213937+0000) 2024-03-04T19:24:29.213 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:24:29.213+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:24:29.214184+0000) 2024-03-04T19:24:30.213 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:24:30.213+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:24:30.214416+0000) 2024-03-04T19:24:31.213 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:24:31.213+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:24:31.214636+0000) 2024-03-04T19:24:32.214 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:24:32.214+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:24:32.214910+0000) 2024-03-04T19:24:33.214 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:24:33.214+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:24:33.215196+0000) 2024-03-04T19:24:34.215 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:24:34.214+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:24:34.215501+0000) 2024-03-04T19:24:35.215 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:24:35.215+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:24:35.215791+0000) 2024-03-04T19:24:36.215 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:24:36.215+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:24:36.216133+0000) 2024-03-04T19:24:37.215 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:24:37.215+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:24:37.216434+0000) 2024-03-04T19:24:38.216 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:24:38.216+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:24:38.216744+0000) 2024-03-04T19:24:39.216 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:24:39.216+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:24:39.217041+0000) 2024-03-04T19:24:40.216 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:24:40.216+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:24:40.217283+0000) 2024-03-04T19:24:41.216 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:24:41.216+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:24:41.217530+0000) 2024-03-04T19:24:42.217 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:24:42.217+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:24:42.217777+0000) 2024-03-04T19:24:43.217 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:24:43.217+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:24:43.218074+0000) 2024-03-04T19:24:44.218 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:24:44.217+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:24:44.218348+0000) 2024-03-04T19:24:45.218 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:24:45.217+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:24:45.218541+0000) 2024-03-04T19:24:46.218 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:24:46.218+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:24:46.218809+0000) 2024-03-04T19:24:47.218 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:24:47.218+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:24:47.219118+0000) 2024-03-04T19:24:48.219 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:24:48.218+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:24:48.219376+0000) 2024-03-04T19:24:49.219 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:24:49.219+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:24:49.219633+0000) 2024-03-04T19:24:50.219 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:24:50.219+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:24:50.219914+0000) 2024-03-04T19:24:51.219 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:24:51.219+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:24:51.220172+0000) 2024-03-04T19:24:52.220 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:24:52.220+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:24:52.220421+0000) 2024-03-04T19:24:52.565 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:24:52.574 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:24:53.220 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:24:53.220+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:24:53.220679+0000) 2024-03-04T19:24:54.220 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:24:54.220+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:24:54.220948+0000) 2024-03-04T19:24:55.220 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:24:55.220+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:24:55.221199+0000) 2024-03-04T19:24:56.221 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:24:56.221+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:24:56.221448+0000) 2024-03-04T19:24:57.221 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:24:57.221+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:24:57.221719+0000) 2024-03-04T19:24:58.221 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:24:58.221+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:24:58.221975+0000) 2024-03-04T19:24:59.221 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:24:59.221+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:24:59.222232+0000) 2024-03-04T19:25:00.221 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:25:00.222+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:25:00.222532+0000) 2024-03-04T19:25:01.222 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:25:01.222+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:25:01.222781+0000) 2024-03-04T19:25:02.222 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:25:02.222+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:25:02.223048+0000) 2024-03-04T19:25:03.222 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:25:03.223+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:25:03.223297+0000) 2024-03-04T19:25:04.223 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:25:04.222+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:25:04.223538+0000) 2024-03-04T19:25:05.223 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:25:05.223+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:25:05.223777+0000) 2024-03-04T19:25:06.223 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:25:06.223+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:25:06.224034+0000) 2024-03-04T19:25:07.223 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:25:07.224+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:25:07.224292+0000) 2024-03-04T19:25:08.223 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:25:08.222+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:25:08.224523+0000) 2024-03-04T19:25:09.224 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:25:09.223+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:25:09.224764+0000) 2024-03-04T19:25:10.224 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:25:10.223+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:25:10.224995+0000) 2024-03-04T19:25:11.224 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:25:11.223+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:25:11.225273+0000) 2024-03-04T19:25:12.224 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:25:12.224+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:25:12.225529+0000) 2024-03-04T19:25:13.225 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:25:13.224+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:25:13.225768+0000) 2024-03-04T19:25:14.225 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:25:14.224+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:25:14.226016+0000) 2024-03-04T19:25:15.225 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:25:15.225+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:25:15.226254+0000) 2024-03-04T19:25:16.225 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:25:16.225+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:25:16.226495+0000) 2024-03-04T19:25:17.226 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:25:17.225+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:25:17.226729+0000) 2024-03-04T19:25:18.226 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:25:18.226+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:25:18.226986+0000) 2024-03-04T19:25:19.226 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:25:19.226+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:25:19.227243+0000) 2024-03-04T19:25:20.227 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:25:20.226+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:25:20.227492+0000) 2024-03-04T19:25:21.227 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:25:21.226+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:25:21.227735+0000) 2024-03-04T19:25:22.227 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:25:22.227+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:25:22.227973+0000) 2024-03-04T19:25:22.779 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:25:22.787 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:25:23.227 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:25:23.227+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:25:23.228215+0000) 2024-03-04T19:25:24.228 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:25:24.227+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:25:24.228461+0000) 2024-03-04T19:25:25.228 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:25:25.227+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:25:25.228637+0000) 2024-03-04T19:25:26.228 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:25:26.228+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:25:26.228945+0000) 2024-03-04T19:25:27.228 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:25:27.228+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:25:27.229140+0000) 2024-03-04T19:25:28.228 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:25:28.228+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:25:28.229382+0000) 2024-03-04T19:25:29.229 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:25:29.228+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:25:29.229622+0000) 2024-03-04T19:25:30.229 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:25:30.229+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:25:30.229950+0000) 2024-03-04T19:25:31.229 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:25:31.229+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:25:31.230230+0000) 2024-03-04T19:25:32.230 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:25:32.229+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:25:32.230519+0000) 2024-03-04T19:25:33.230 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:25:33.230+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:25:33.230814+0000) 2024-03-04T19:25:34.230 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:25:34.230+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:25:34.231176+0000) 2024-03-04T19:25:35.231 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:25:35.230+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:25:35.231471+0000) 2024-03-04T19:25:36.231 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:25:36.230+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:25:36.231770+0000) 2024-03-04T19:25:37.231 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:25:37.231+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:25:37.232094+0000) 2024-03-04T19:25:38.231 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:25:38.231+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:25:38.232391+0000) 2024-03-04T19:25:39.232 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:25:39.231+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:25:39.232697+0000) 2024-03-04T19:25:40.232 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:25:40.232+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:25:40.232990+0000) 2024-03-04T19:25:41.232 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:25:41.232+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:25:41.233287+0000) 2024-03-04T19:25:42.233 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:25:42.232+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:25:42.233585+0000) 2024-03-04T19:25:43.233 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:25:43.233+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:25:43.233934+0000) 2024-03-04T19:25:44.233 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:25:44.233+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:25:44.234239+0000) 2024-03-04T19:25:45.234 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:25:45.233+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:25:45.234534+0000) 2024-03-04T19:25:46.234 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:25:46.234+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:25:46.234814+0000) 2024-03-04T19:25:47.234 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:25:47.234+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:25:47.235146+0000) 2024-03-04T19:25:48.234 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:25:48.234+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:25:48.235521+0000) 2024-03-04T19:25:49.235 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:25:49.234+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:25:49.235779+0000) 2024-03-04T19:25:50.235 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:25:50.235+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:25:50.236015+0000) 2024-03-04T19:25:51.235 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:25:51.235+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:25:51.236258+0000) 2024-03-04T19:25:52.236 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:25:52.236+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:25:52.236496+0000) 2024-03-04T19:25:52.992 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:25:53.000 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:25:53.236 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:25:53.235+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:25:53.236731+0000) 2024-03-04T19:25:54.236 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:25:54.236+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:25:54.237008+0000) 2024-03-04T19:25:55.236 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:25:55.236+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:25:55.237247+0000) 2024-03-04T19:25:56.236 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:25:56.237+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:25:56.237490+0000) 2024-03-04T19:25:57.237 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:25:57.237+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:25:57.237729+0000) 2024-03-04T19:25:58.237 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:25:58.237+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:25:58.237978+0000) 2024-03-04T19:25:59.237 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:25:59.237+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:25:59.238224+0000) 2024-03-04T19:26:00.237 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:26:00.237+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:26:00.238456+0000) 2024-03-04T19:26:01.238 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:26:01.238+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:26:01.238708+0000) 2024-03-04T19:26:02.238 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:26:02.238+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:26:02.238962+0000) 2024-03-04T19:26:03.238 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:26:03.238+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:26:03.239173+0000) 2024-03-04T19:26:04.239 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:26:04.238+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:26:04.239444+0000) 2024-03-04T19:26:05.239 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:26:05.238+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:26:05.239705+0000) 2024-03-04T19:26:06.239 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:26:06.239+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:26:06.239943+0000) 2024-03-04T19:26:07.239 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:26:07.240+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:26:07.240177+0000) 2024-03-04T19:26:08.240 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:26:08.239+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:26:08.240373+0000) 2024-03-04T19:26:09.240 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:26:09.240+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:26:09.240648+0000) 2024-03-04T19:26:10.240 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:26:10.239+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:26:10.240972+0000) 2024-03-04T19:26:11.241 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:26:11.240+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:26:11.241267+0000) 2024-03-04T19:26:12.241 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:26:12.239+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:26:12.241495+0000) 2024-03-04T19:26:13.241 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:26:13.240+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:26:13.241783+0000) 2024-03-04T19:26:14.241 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:26:14.241+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:26:14.242142+0000) 2024-03-04T19:26:15.242 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:26:15.240+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:26:15.242437+0000) 2024-03-04T19:26:16.242 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:26:16.240+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:26:16.242666+0000) 2024-03-04T19:26:17.242 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:26:17.241+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:26:17.242986+0000) 2024-03-04T19:26:18.242 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:26:18.242+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:26:18.243237+0000) 2024-03-04T19:26:19.243 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:26:19.241+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:26:19.243426+0000) 2024-03-04T19:26:20.243 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:26:20.242+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:26:20.243697+0000) 2024-03-04T19:26:21.243 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:26:21.243+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:26:21.243943+0000) 2024-03-04T19:26:22.243 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:26:22.243+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:26:22.244178+0000) 2024-03-04T19:26:23.124 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:26:23.132 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:26:23.266 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:26:23.243+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:26:23.244367+0000) 2024-03-04T19:26:24.244 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:26:24.243+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:26:24.244626+0000) 2024-03-04T19:26:25.244 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:26:25.244+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:26:25.244907+0000) 2024-03-04T19:26:26.244 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:26:26.244+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:26:26.245142+0000) 2024-03-04T19:26:27.244 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:26:27.244+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:26:27.245387+0000) 2024-03-04T19:26:28.245 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:26:28.244+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:26:28.245622+0000) 2024-03-04T19:26:29.245 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:26:29.245+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:26:29.245931+0000) 2024-03-04T19:26:30.245 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:26:30.245+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:26:30.246186+0000) 2024-03-04T19:26:31.246 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:26:31.245+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:26:31.246428+0000) 2024-03-04T19:26:32.246 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:26:32.245+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:26:32.246684+0000) 2024-03-04T19:26:33.246 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:26:33.246+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:26:33.246944+0000) 2024-03-04T19:26:34.246 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:26:34.246+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:26:34.247185+0000) 2024-03-04T19:26:35.247 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:26:35.246+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:26:35.247441+0000) 2024-03-04T19:26:36.247 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:26:36.246+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:26:36.247728+0000) 2024-03-04T19:26:37.247 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:26:37.247+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:26:37.247969+0000) 2024-03-04T19:26:38.247 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:26:38.247+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:26:38.248203+0000) 2024-03-04T19:26:39.247 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:26:39.247+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:26:39.248443+0000) 2024-03-04T19:26:40.248 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:26:40.248+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:26:40.248674+0000) 2024-03-04T19:26:41.248 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:26:41.248+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:26:41.248929+0000) 2024-03-04T19:26:42.248 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:26:42.248+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:26:42.249160+0000) 2024-03-04T19:26:43.248 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:26:43.248+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:26:43.249407+0000) 2024-03-04T19:26:44.249 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:26:44.249+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:26:44.249656+0000) 2024-03-04T19:26:45.249 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:26:45.249+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:26:45.249923+0000) 2024-03-04T19:26:46.250 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:26:46.249+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:26:46.250163+0000) 2024-03-04T19:26:47.251 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:26:47.249+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:26:47.250402+0000) 2024-03-04T19:26:48.250 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:26:48.249+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:26:48.250641+0000) 2024-03-04T19:26:49.250 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:26:49.250+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:26:49.250932+0000) 2024-03-04T19:26:50.250 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:26:50.250+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:26:50.251163+0000) 2024-03-04T19:26:51.250 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:26:51.250+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:26:51.251411+0000) 2024-03-04T19:26:52.251 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:26:52.251+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:26:52.251649+0000) 2024-03-04T19:26:53.251 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:26:53.251+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:26:53.251931+0000) 2024-03-04T19:26:53.335 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:26:53.345 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:26:54.251 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:26:54.251+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:26:54.252185+0000) 2024-03-04T19:26:55.251 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:26:55.251+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:26:55.252364+0000) 2024-03-04T19:26:56.252 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:26:56.252+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:26:56.252603+0000) 2024-03-04T19:26:57.252 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:26:57.252+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:26:57.252855+0000) 2024-03-04T19:26:58.252 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:26:58.252+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:26:58.253102+0000) 2024-03-04T19:26:59.253 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:26:59.252+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:26:59.253352+0000) 2024-03-04T19:27:00.253 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:27:00.253+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:27:00.253610+0000) 2024-03-04T19:27:01.253 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:27:01.253+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:27:01.253880+0000) 2024-03-04T19:27:02.253 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:27:02.253+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:27:02.254112+0000) 2024-03-04T19:27:03.253 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:27:03.253+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:27:03.254342+0000) 2024-03-04T19:27:04.254 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:27:04.253+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:27:04.254606+0000) 2024-03-04T19:27:05.254 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:27:05.254+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:27:05.254801+0000) 2024-03-04T19:27:06.254 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:27:06.254+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:27:06.255077+0000) 2024-03-04T19:27:07.254 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:27:07.255+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:27:07.255311+0000) 2024-03-04T19:27:08.255 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:27:08.255+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:27:08.255549+0000) 2024-03-04T19:27:09.255 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:27:09.255+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:27:09.255788+0000) 2024-03-04T19:27:10.255 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:27:10.255+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:27:10.256029+0000) 2024-03-04T19:27:11.256 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:27:11.256+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:27:11.256305+0000) 2024-03-04T19:27:12.256 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:27:12.256+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:27:12.256540+0000) 2024-03-04T19:27:13.256 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:27:13.255+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:27:13.256779+0000) 2024-03-04T19:27:14.256 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:27:14.255+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:27:14.257027+0000) 2024-03-04T19:27:15.256 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:27:15.255+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:27:15.257216+0000) 2024-03-04T19:27:16.256 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:27:16.256+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:27:16.257457+0000) 2024-03-04T19:27:17.257 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:27:17.256+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:27:17.257692+0000) 2024-03-04T19:27:18.257 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:27:18.256+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:27:18.257963+0000) 2024-03-04T19:27:19.257 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:27:19.257+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:27:19.258204+0000) 2024-03-04T19:27:20.258 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:27:20.257+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:27:20.258441+0000) 2024-03-04T19:27:21.258 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:27:21.257+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:27:21.258682+0000) 2024-03-04T19:27:22.258 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:27:22.257+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:27:22.258943+0000) 2024-03-04T19:27:23.258 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:27:23.258+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:27:23.259188+0000) 2024-03-04T19:27:23.472 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:27:23.481 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:27:24.258 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:27:24.258+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:27:24.259482+0000) 2024-03-04T19:27:25.259 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:27:25.258+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:27:25.259721+0000) 2024-03-04T19:27:26.259 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:27:26.259+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:27:26.259956+0000) 2024-03-04T19:27:27.259 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:27:27.259+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:27:27.260213+0000) 2024-03-04T19:27:28.259 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:27:28.259+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:27:28.260446+0000) 2024-03-04T19:27:29.260 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:27:29.259+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:27:29.260683+0000) 2024-03-04T19:27:30.260 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:27:30.260+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:27:30.260946+0000) 2024-03-04T19:27:31.260 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:27:31.260+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:27:31.261180+0000) 2024-03-04T19:27:32.260 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:27:32.260+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:27:32.261419+0000) 2024-03-04T19:27:33.261 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:27:33.260+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:27:33.261650+0000) 2024-03-04T19:27:34.262 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:27:34.261+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:27:34.261945+0000) 2024-03-04T19:27:35.262 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:27:35.261+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:27:35.262200+0000) 2024-03-04T19:27:36.262 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:27:36.261+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:27:36.262450+0000) 2024-03-04T19:27:37.262 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:27:37.261+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:27:37.262683+0000) 2024-03-04T19:27:38.264 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:27:38.262+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:27:38.262938+0000) 2024-03-04T19:27:39.263 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:27:39.262+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:27:39.263195+0000) 2024-03-04T19:27:40.263 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:27:40.262+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:27:40.263431+0000) 2024-03-04T19:27:41.263 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:27:41.263+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:27:41.263672+0000) 2024-03-04T19:27:42.264 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:27:42.263+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:27:42.263946+0000) 2024-03-04T19:27:43.263 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:27:43.263+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:27:43.264185+0000) 2024-03-04T19:27:44.264 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:27:44.263+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:27:44.264428+0000) 2024-03-04T19:27:45.264 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:27:45.264+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:27:45.264664+0000) 2024-03-04T19:27:46.264 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:27:46.264+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:27:46.264953+0000) 2024-03-04T19:27:47.264 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:27:47.264+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:27:47.265189+0000) 2024-03-04T19:27:48.265 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:27:48.264+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:27:48.265421+0000) 2024-03-04T19:27:49.265 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:27:49.265+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:27:49.265658+0000) 2024-03-04T19:27:50.265 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:27:50.265+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:27:50.265924+0000) 2024-03-04T19:27:51.266 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:27:51.265+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:27:51.266150+0000) 2024-03-04T19:27:52.267 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:27:52.265+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:27:52.266388+0000) 2024-03-04T19:27:53.266 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:27:53.266+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:27:53.266623+0000) 2024-03-04T19:27:53.687 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:27:53.696 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:27:54.266 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:27:54.266+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:27:54.266920+0000) 2024-03-04T19:27:55.267 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:27:55.266+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:27:55.267161+0000) 2024-03-04T19:27:56.267 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:27:56.267+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:27:56.267432+0000) 2024-03-04T19:27:57.267 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:27:57.266+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:27:57.267618+0000) 2024-03-04T19:27:58.267 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:27:58.267+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:27:58.267938+0000) 2024-03-04T19:27:59.267 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:27:59.267+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:27:59.268183+0000) 2024-03-04T19:28:00.267 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:28:00.268+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:28:00.268414+0000) 2024-03-04T19:28:01.268 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:28:01.268+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:28:01.268651+0000) 2024-03-04T19:28:02.268 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:28:02.268+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:28:02.268926+0000) 2024-03-04T19:28:03.268 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:28:03.268+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:28:03.269158+0000) 2024-03-04T19:28:04.269 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:28:04.269+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:28:04.269426+0000) 2024-03-04T19:28:05.269 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:28:05.269+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:28:05.269686+0000) 2024-03-04T19:28:06.269 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:28:06.269+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:28:06.269926+0000) 2024-03-04T19:28:07.269 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:28:07.269+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:28:07.270166+0000) 2024-03-04T19:28:08.270 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:28:08.269+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:28:08.270418+0000) 2024-03-04T19:28:09.270 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:28:09.270+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:28:09.270654+0000) 2024-03-04T19:28:10.270 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:28:10.270+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:28:10.270920+0000) 2024-03-04T19:28:11.270 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:28:11.270+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:28:11.271103+0000) 2024-03-04T19:28:12.270 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:28:12.271+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:28:12.271339+0000) 2024-03-04T19:28:13.270 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:28:13.271+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:28:13.271574+0000) 2024-03-04T19:28:14.271 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:28:14.271+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:28:14.271852+0000) 2024-03-04T19:28:15.271 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:28:15.270+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:28:15.272092+0000) 2024-03-04T19:28:16.271 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:28:16.270+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:28:16.272280+0000) 2024-03-04T19:28:17.273 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:28:17.271+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:28:17.272521+0000) 2024-03-04T19:28:18.272 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:28:18.271+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:28:18.272759+0000) 2024-03-04T19:28:19.272 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:28:19.271+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:28:19.273003+0000) 2024-03-04T19:28:20.273 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:28:20.272+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:28:20.273235+0000) 2024-03-04T19:28:21.273 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:28:21.272+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:28:21.273493+0000) 2024-03-04T19:28:22.273 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:28:22.272+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:28:22.273730+0000) 2024-03-04T19:28:23.273 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:28:23.272+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:28:23.273973+0000) 2024-03-04T19:28:23.832 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:28:23.841 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:28:24.273 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:28:24.273+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:28:24.274178+0000) 2024-03-04T19:28:25.273 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:28:25.273+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:28:25.274418+0000) 2024-03-04T19:28:26.274 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:28:26.273+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:28:26.274650+0000) 2024-03-04T19:28:27.274 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:28:27.274+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:28:27.274919+0000) 2024-03-04T19:28:28.274 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:28:28.274+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:28:28.275154+0000) 2024-03-04T19:28:29.275 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:28:29.274+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:28:29.275393+0000) 2024-03-04T19:28:30.275 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:28:30.274+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:28:30.275625+0000) 2024-03-04T19:28:31.275 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:28:31.275+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:28:31.275903+0000) 2024-03-04T19:28:32.276 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:28:32.275+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:28:32.276178+0000) 2024-03-04T19:28:33.276 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:28:33.275+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:28:33.276465+0000) 2024-03-04T19:28:34.276 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:28:34.275+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:28:34.276773+0000) 2024-03-04T19:28:35.276 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:28:35.276+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:28:35.277094+0000) 2024-03-04T19:28:36.277 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:28:36.276+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:28:36.277341+0000) 2024-03-04T19:28:37.277 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:28:37.276+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:28:37.277621+0000) 2024-03-04T19:28:38.277 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:28:38.277+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:28:38.277941+0000) 2024-03-04T19:28:39.277 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:28:39.277+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:28:39.278220+0000) 2024-03-04T19:28:40.278 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:28:40.277+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:28:40.278507+0000) 2024-03-04T19:28:41.278 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:28:41.278+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:28:41.278805+0000) 2024-03-04T19:28:42.278 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:28:42.278+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:28:42.279146+0000) 2024-03-04T19:28:43.278 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:28:43.278+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:28:43.279381+0000) 2024-03-04T19:28:44.279 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:28:44.278+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:28:44.279670+0000) 2024-03-04T19:28:45.279 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:28:45.279+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:28:45.279964+0000) 2024-03-04T19:28:46.279 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:28:46.279+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:28:46.280252+0000) 2024-03-04T19:28:47.279 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:28:47.279+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:28:47.280535+0000) 2024-03-04T19:28:48.280 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:28:48.280+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:28:48.280860+0000) 2024-03-04T19:28:49.280 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:28:49.280+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:28:49.281150+0000) 2024-03-04T19:28:50.281 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:28:50.280+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:28:50.281437+0000) 2024-03-04T19:28:51.281 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:28:51.281+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:28:51.281718+0000) 2024-03-04T19:28:52.283 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:28:52.281+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:28:52.282019+0000) 2024-03-04T19:28:53.283 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:28:53.281+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:28:53.282294+0000) 2024-03-04T19:28:54.029 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:28:54.037 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:28:54.282 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:28:54.282+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:28:54.282623+0000) 2024-03-04T19:28:55.282 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:28:55.282+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:28:55.282906+0000) 2024-03-04T19:28:56.282 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:28:56.282+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:28:56.283151+0000) 2024-03-04T19:28:57.282 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:28:57.282+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:28:57.283400+0000) 2024-03-04T19:28:58.283 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:28:58.282+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:28:58.283593+0000) 2024-03-04T19:28:59.283 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:28:59.283+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:28:59.283854+0000) 2024-03-04T19:29:00.283 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:29:00.283+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:29:00.284092+0000) 2024-03-04T19:29:01.283 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:29:01.283+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:29:01.284292+0000) 2024-03-04T19:29:02.285 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:29:02.284+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:29:02.284549+0000) 2024-03-04T19:29:03.284 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:29:03.284+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:29:03.284804+0000) 2024-03-04T19:29:04.284 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:29:04.284+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:29:04.285091+0000) 2024-03-04T19:29:05.284 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:29:05.285+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:29:05.285330+0000) 2024-03-04T19:29:06.285 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:29:06.284+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:29:06.285508+0000) 2024-03-04T19:29:07.285 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:29:07.285+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:29:07.285751+0000) 2024-03-04T19:29:08.285 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:29:08.285+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:29:08.285992+0000) 2024-03-04T19:29:09.285 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:29:09.285+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:29:09.286233+0000) 2024-03-04T19:29:10.286 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:29:10.286+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:29:10.286472+0000) 2024-03-04T19:29:11.286 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:29:11.286+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:29:11.286758+0000) 2024-03-04T19:29:12.286 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:29:12.286+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:29:12.286994+0000) 2024-03-04T19:29:13.286 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:29:13.287+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:29:13.287233+0000) 2024-03-04T19:29:14.287 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:29:14.287+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:29:14.287478+0000) 2024-03-04T19:29:15.287 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:29:15.287+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:29:15.287756+0000) 2024-03-04T19:29:16.287 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:29:16.287+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:29:16.287997+0000) 2024-03-04T19:29:17.287 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:29:17.288+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:29:17.288245+0000) 2024-03-04T19:29:18.288 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:29:18.287+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:29:18.288516+0000) 2024-03-04T19:29:19.288 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:29:19.287+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:29:19.288755+0000) 2024-03-04T19:29:20.288 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:29:20.287+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:29:20.289028+0000) 2024-03-04T19:29:21.289 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:29:21.288+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:29:21.289321+0000) 2024-03-04T19:29:22.289 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:29:22.288+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:29:22.289613+0000) 2024-03-04T19:29:23.289 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:29:23.288+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:29:23.289918+0000) 2024-03-04T19:29:24.236 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:29:24.249 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:29:24.289 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:29:24.288+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:29:24.290093+0000) 2024-03-04T19:29:25.290 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:29:25.289+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:29:25.290336+0000) 2024-03-04T19:29:26.290 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:29:26.289+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:29:26.290574+0000) 2024-03-04T19:29:27.290 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:29:27.289+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:29:27.290855+0000) 2024-03-04T19:29:28.290 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:29:28.290+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:29:28.291094+0000) 2024-03-04T19:29:29.291 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:29:29.290+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:29:29.291422+0000) 2024-03-04T19:29:30.291 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:29:30.290+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:29:30.291717+0000) 2024-03-04T19:29:31.292 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:29:31.291+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:29:31.292008+0000) 2024-03-04T19:29:32.291 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:29:32.291+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:29:32.292307+0000) 2024-03-04T19:29:33.292 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:29:33.291+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:29:33.292601+0000) 2024-03-04T19:29:34.292 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:29:34.292+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:29:34.292935+0000) 2024-03-04T19:29:35.292 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:29:35.292+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:29:35.293222+0000) 2024-03-04T19:29:36.293 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:29:36.292+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:29:36.293512+0000) 2024-03-04T19:29:37.293 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:29:37.292+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:29:37.293797+0000) 2024-03-04T19:29:38.293 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:29:38.293+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:29:38.294121+0000) 2024-03-04T19:29:39.294 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:29:39.293+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:29:39.294412+0000) 2024-03-04T19:29:40.294 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:29:40.293+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:29:40.294711+0000) 2024-03-04T19:29:41.294 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:29:41.294+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:29:41.295010+0000) 2024-03-04T19:29:42.296 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:29:42.294+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:29:42.295307+0000) 2024-03-04T19:29:43.295 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:29:43.294+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:29:43.295605+0000) 2024-03-04T19:29:44.295 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:29:44.295+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:29:44.295944+0000) 2024-03-04T19:29:45.296 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:29:45.295+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:29:45.296239+0000) 2024-03-04T19:29:46.295 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:29:46.295+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:29:46.296533+0000) 2024-03-04T19:29:47.296 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:29:47.296+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:29:47.296867+0000) 2024-03-04T19:29:48.296 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:29:48.296+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:29:48.297158+0000) 2024-03-04T19:29:49.296 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:29:49.296+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:29:49.297435+0000) 2024-03-04T19:29:50.297 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:29:50.296+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:29:50.297679+0000) 2024-03-04T19:29:51.297 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:29:51.297+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:29:51.297948+0000) 2024-03-04T19:29:52.297 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:29:52.297+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:29:52.298199+0000) 2024-03-04T19:29:53.298 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:29:53.297+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:29:53.298443+0000) 2024-03-04T19:29:54.298 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:29:54.298+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:29:54.298697+0000) 2024-03-04T19:29:54.406 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:29:54.415 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:29:55.298 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:29:55.298+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:29:55.298942+0000) 2024-03-04T19:29:56.299 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:29:56.298+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:29:56.299174+0000) 2024-03-04T19:29:57.299 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:29:57.298+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:29:57.299411+0000) 2024-03-04T19:29:58.299 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:29:58.299+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:29:58.299673+0000) 2024-03-04T19:29:59.299 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:29:59.299+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:29:59.299984+0000) 2024-03-04T19:30:00.299 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:30:00.299+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:30:00.300223+0000) 2024-03-04T19:30:01.300 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:30:01.300+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:30:01.300462+0000) 2024-03-04T19:30:02.300 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:30:02.300+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:30:02.300701+0000) 2024-03-04T19:30:03.300 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:30:03.300+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:30:03.300956+0000) 2024-03-04T19:30:04.304 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:30:04.300+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:30:04.301215+0000) 2024-03-04T19:30:05.303 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:30:05.301+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:30:05.301462+0000) 2024-03-04T19:30:06.301 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:30:06.301+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:30:06.301732+0000) 2024-03-04T19:30:07.301 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:30:07.301+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:30:07.301974+0000) 2024-03-04T19:30:08.302 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:30:08.301+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:30:08.302219+0000) 2024-03-04T19:30:09.302 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:30:09.301+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:30:09.302417+0000) 2024-03-04T19:30:10.302 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:30:10.302+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:30:10.302655+0000) 2024-03-04T19:30:11.302 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:30:11.302+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:30:11.302925+0000) 2024-03-04T19:30:12.302 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:30:12.302+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:30:12.303163+0000) 2024-03-04T19:30:13.304 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:30:13.303+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:30:13.303403+0000) 2024-03-04T19:30:14.303 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:30:14.303+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:30:14.303653+0000) 2024-03-04T19:30:15.303 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:30:15.303+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:30:15.303938+0000) 2024-03-04T19:30:16.305 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:30:16.303+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:30:16.304175+0000) 2024-03-04T19:30:17.304 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:30:17.304+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:30:17.304411+0000) 2024-03-04T19:30:18.304 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:30:18.304+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:30:18.304650+0000) 2024-03-04T19:30:19.305 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:30:19.304+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:30:19.304932+0000) 2024-03-04T19:30:20.304 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:30:20.304+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:30:20.305178+0000) 2024-03-04T19:30:21.305 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:30:21.304+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:30:21.305421+0000) 2024-03-04T19:30:22.306 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:30:22.304+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:30:22.305668+0000) 2024-03-04T19:30:23.305 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:30:23.304+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:30:23.305935+0000) 2024-03-04T19:30:24.305 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:30:24.305+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:30:24.306184+0000) 2024-03-04T19:30:24.603 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:30:24.612 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:30:25.306 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:30:25.305+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:30:25.306426+0000) 2024-03-04T19:30:26.307 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:30:26.305+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:30:26.306663+0000) 2024-03-04T19:30:27.306 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:30:27.305+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:30:27.306931+0000) 2024-03-04T19:30:28.306 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:30:28.306+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:30:28.307183+0000) 2024-03-04T19:30:29.307 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:30:29.306+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:30:29.307449+0000) 2024-03-04T19:30:30.307 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:30:30.306+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:30:30.307712+0000) 2024-03-04T19:30:31.307 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:30:31.307+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:30:31.307978+0000) 2024-03-04T19:30:32.307 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:30:32.307+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:30:32.308214+0000) 2024-03-04T19:30:33.308 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:30:33.307+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:30:33.308450+0000) 2024-03-04T19:30:34.308 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:30:34.307+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:30:34.308705+0000) 2024-03-04T19:30:35.308 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:30:35.308+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:30:35.308946+0000) 2024-03-04T19:30:36.309 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:30:36.308+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:30:36.309141+0000) 2024-03-04T19:30:37.309 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:30:37.308+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:30:37.309375+0000) 2024-03-04T19:30:38.309 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:30:38.308+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:30:38.309614+0000) 2024-03-04T19:30:39.312 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:30:39.309+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:30:39.309931+0000) 2024-03-04T19:30:40.310 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:30:40.309+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:30:40.310169+0000) 2024-03-04T19:30:41.310 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:30:41.309+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:30:41.310460+0000) 2024-03-04T19:30:42.310 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:30:42.309+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:30:42.310701+0000) 2024-03-04T19:30:43.310 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:30:43.310+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:30:43.310938+0000) 2024-03-04T19:30:44.310 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:30:44.310+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:30:44.311195+0000) 2024-03-04T19:30:45.311 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:30:45.310+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:30:45.311435+0000) 2024-03-04T19:30:46.311 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:30:46.310+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:30:46.311608+0000) 2024-03-04T19:30:47.311 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:30:47.311+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:30:47.311872+0000) 2024-03-04T19:30:48.311 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:30:48.311+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:30:48.312149+0000) 2024-03-04T19:30:49.312 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:30:49.311+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:30:49.312396+0000) 2024-03-04T19:30:50.312 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:30:50.311+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:30:50.312633+0000) 2024-03-04T19:30:51.312 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:30:51.312+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:30:51.312929+0000) 2024-03-04T19:30:52.312 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:30:52.312+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:30:52.313167+0000) 2024-03-04T19:30:53.313 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:30:53.312+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:30:53.313424+0000) 2024-03-04T19:30:54.313 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:30:54.312+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:30:54.313685+0000) 2024-03-04T19:30:54.816 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:30:54.824 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:30:55.313 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:30:55.313+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:30:55.313934+0000) 2024-03-04T19:30:56.313 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:30:56.313+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:30:56.314174+0000) 2024-03-04T19:30:57.314 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:30:57.313+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:30:57.314410+0000) 2024-03-04T19:30:58.314 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:30:58.314+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:30:58.314647+0000) 2024-03-04T19:30:59.314 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:30:59.314+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:30:59.314955+0000) 2024-03-04T19:31:00.314 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:31:00.314+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:31:00.315158+0000) 2024-03-04T19:31:01.314 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:31:01.314+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:31:01.315410+0000) 2024-03-04T19:31:02.315 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:31:02.314+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:31:02.315636+0000) 2024-03-04T19:31:03.315 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:31:03.315+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:31:03.315929+0000) 2024-03-04T19:31:04.315 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:31:04.315+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:31:04.316172+0000) 2024-03-04T19:31:05.315 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:31:05.316+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:31:05.316414+0000) 2024-03-04T19:31:06.317 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:31:06.315+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:31:06.316644+0000) 2024-03-04T19:31:07.316 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:31:07.316+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:31:07.316907+0000) 2024-03-04T19:31:08.317 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:31:08.316+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:31:08.317149+0000) 2024-03-04T19:31:09.319 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:31:09.316+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:31:09.317338+0000) 2024-03-04T19:31:10.317 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:31:10.317+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:31:10.317579+0000) 2024-03-04T19:31:11.317 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:31:11.317+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:31:11.317852+0000) 2024-03-04T19:31:12.317 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:31:12.317+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:31:12.318094+0000) 2024-03-04T19:31:13.317 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:31:13.318+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:31:13.318334+0000) 2024-03-04T19:31:14.318 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:31:14.318+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:31:14.318583+0000) 2024-03-04T19:31:15.318 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:31:15.318+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:31:15.318853+0000) 2024-03-04T19:31:16.318 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:31:16.318+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:31:16.319092+0000) 2024-03-04T19:31:17.320 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:31:17.319+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:31:17.319330+0000) 2024-03-04T19:31:18.319 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:31:18.318+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:31:18.319583+0000) 2024-03-04T19:31:19.319 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:31:19.319+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:31:19.319854+0000) 2024-03-04T19:31:20.319 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:31:20.319+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:31:20.320087+0000) 2024-03-04T19:31:21.320 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:31:21.320+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:31:21.320327+0000) 2024-03-04T19:31:22.320 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:31:22.320+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:31:22.320559+0000) 2024-03-04T19:31:23.320 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:31:23.318+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:31:23.320739+0000) 2024-03-04T19:31:24.320 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:31:24.319+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:31:24.320988+0000) 2024-03-04T19:31:24.954 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:31:24.962 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:31:25.324 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:31:25.320+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:31:25.321226+0000) 2024-03-04T19:31:26.321 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:31:26.320+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:31:26.321460+0000) 2024-03-04T19:31:27.321 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:31:27.320+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:31:27.321709+0000) 2024-03-04T19:31:28.322 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:31:28.320+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:31:28.321944+0000) 2024-03-04T19:31:29.325 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:31:29.321+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:31:29.322183+0000) 2024-03-04T19:31:30.322 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:31:30.321+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:31:30.322436+0000) 2024-03-04T19:31:31.322 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:31:31.321+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:31:31.322702+0000) 2024-03-04T19:31:32.322 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:31:32.321+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:31:32.322934+0000) 2024-03-04T19:31:33.322 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:31:33.322+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:31:33.323176+0000) 2024-03-04T19:31:34.322 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:31:34.322+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:31:34.323438+0000) 2024-03-04T19:31:35.323 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:31:35.322+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:31:35.323754+0000) 2024-03-04T19:31:36.323 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:31:36.323+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:31:36.323985+0000) 2024-03-04T19:31:37.324 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:31:37.323+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:31:37.324218+0000) 2024-03-04T19:31:38.324 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:31:38.323+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:31:38.324458+0000) 2024-03-04T19:31:39.324 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:31:39.323+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:31:39.324701+0000) 2024-03-04T19:31:40.324 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:31:40.324+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:31:40.324935+0000) 2024-03-04T19:31:41.324 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:31:41.324+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:31:41.325171+0000) 2024-03-04T19:31:42.324 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:31:42.324+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:31:42.325406+0000) 2024-03-04T19:31:43.325 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:31:43.324+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:31:43.325639+0000) 2024-03-04T19:31:44.325 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:31:44.325+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:31:44.325919+0000) 2024-03-04T19:31:45.326 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:31:45.325+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:31:45.326164+0000) 2024-03-04T19:31:46.325 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:31:46.325+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:31:46.326405+0000) 2024-03-04T19:31:47.326 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:31:47.325+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:31:47.326647+0000) 2024-03-04T19:31:48.326 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:31:48.326+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:31:48.326934+0000) 2024-03-04T19:31:49.326 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:31:49.326+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:31:49.327122+0000) 2024-03-04T19:31:50.326 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:31:50.326+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:31:50.327367+0000) 2024-03-04T19:31:51.327 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:31:51.326+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:31:51.327607+0000) 2024-03-04T19:31:52.327 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:31:52.327+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:31:52.327865+0000) 2024-03-04T19:31:53.327 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:31:53.327+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:31:53.328103+0000) 2024-03-04T19:31:54.328 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:31:54.327+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:31:54.328350+0000) 2024-03-04T19:31:55.102 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:31:55.110 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:31:55.328 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:31:55.327+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:31:55.328590+0000) 2024-03-04T19:31:56.328 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:31:56.328+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:31:56.328867+0000) 2024-03-04T19:31:57.328 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:31:57.328+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:31:57.329109+0000) 2024-03-04T19:31:58.328 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:31:58.328+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:31:58.329347+0000) 2024-03-04T19:31:59.329 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:31:59.329+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:31:59.329589+0000) 2024-03-04T19:32:00.329 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:32:00.329+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:32:00.329818+0000) 2024-03-04T19:32:01.329 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:32:01.329+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:32:01.330119+0000) 2024-03-04T19:32:02.329 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:32:02.329+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:32:02.330352+0000) 2024-03-04T19:32:03.330 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:32:03.330+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:32:03.330589+0000) 2024-03-04T19:32:04.331 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:32:04.330+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:32:04.330893+0000) 2024-03-04T19:32:05.331 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:32:05.330+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:32:05.331143+0000) 2024-03-04T19:32:06.331 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:32:06.330+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:32:06.331378+0000) 2024-03-04T19:32:07.332 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:32:07.331+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:32:07.331613+0000) 2024-03-04T19:32:08.331 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:32:08.331+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:32:08.331901+0000) 2024-03-04T19:32:09.331 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:32:09.331+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:32:09.332140+0000) 2024-03-04T19:32:10.331 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:32:10.332+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:32:10.332375+0000) 2024-03-04T19:32:11.332 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:32:11.332+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:32:11.332617+0000) 2024-03-04T19:32:12.332 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:32:12.332+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:32:12.332890+0000) 2024-03-04T19:32:13.332 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:32:13.332+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:32:13.333128+0000) 2024-03-04T19:32:14.332 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:32:14.333+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:32:14.333371+0000) 2024-03-04T19:32:15.333 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:32:15.333+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:32:15.333608+0000) 2024-03-04T19:32:16.333 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:32:16.333+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:32:16.333910+0000) 2024-03-04T19:32:17.333 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:32:17.333+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:32:17.334164+0000) 2024-03-04T19:32:18.334 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:32:18.333+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:32:18.334391+0000) 2024-03-04T19:32:19.334 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:32:19.333+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:32:19.334630+0000) 2024-03-04T19:32:20.334 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:32:20.334+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:32:20.334854+0000) 2024-03-04T19:32:21.334 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:32:21.334+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:32:21.335112+0000) 2024-03-04T19:32:22.334 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:32:22.334+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:32:22.335336+0000) 2024-03-04T19:32:23.335 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:32:23.335+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:32:23.335576+0000) 2024-03-04T19:32:24.335 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:32:24.335+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:32:24.335878+0000) 2024-03-04T19:32:25.256 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:32:25.264 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:32:25.335 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:32:25.334+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:32:25.336094+0000) 2024-03-04T19:32:26.335 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:32:26.334+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:32:26.336338+0000) 2024-03-04T19:32:27.336 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:32:27.335+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:32:27.336574+0000) 2024-03-04T19:32:28.336 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:32:28.335+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:32:28.336906+0000) 2024-03-04T19:32:29.336 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:32:29.336+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:32:29.337146+0000) 2024-03-04T19:32:30.337 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:32:30.336+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:32:30.337379+0000) 2024-03-04T19:32:31.337 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:32:31.336+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:32:31.337616+0000) 2024-03-04T19:32:32.337 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:32:32.336+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:32:32.337797+0000) 2024-03-04T19:32:33.337 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:32:33.337+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:32:33.338072+0000) 2024-03-04T19:32:34.337 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:32:34.337+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:32:34.338270+0000) 2024-03-04T19:32:35.338 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:32:35.337+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:32:35.338458+0000) 2024-03-04T19:32:36.338 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:32:36.337+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:32:36.338711+0000) 2024-03-04T19:32:37.338 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:32:37.338+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:32:37.338949+0000) 2024-03-04T19:32:38.338 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:32:38.338+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:32:38.339184+0000) 2024-03-04T19:32:39.339 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:32:39.338+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:32:39.339424+0000) 2024-03-04T19:32:40.339 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:32:40.338+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:32:40.339661+0000) 2024-03-04T19:32:41.339 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:32:41.339+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:32:41.339941+0000) 2024-03-04T19:32:42.339 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:32:42.339+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:32:42.340176+0000) 2024-03-04T19:32:43.340 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:32:43.339+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:32:43.340420+0000) 2024-03-04T19:32:44.340 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:32:44.339+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:32:44.340668+0000) 2024-03-04T19:32:45.340 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:32:45.340+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:32:45.340950+0000) 2024-03-04T19:32:46.340 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:32:46.340+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:32:46.341186+0000) 2024-03-04T19:32:47.341 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:32:47.340+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:32:47.341440+0000) 2024-03-04T19:32:48.341 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:32:48.340+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:32:48.341688+0000) 2024-03-04T19:32:49.341 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:32:49.341+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:32:49.341927+0000) 2024-03-04T19:32:50.341 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:32:50.341+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:32:50.342162+0000) 2024-03-04T19:32:51.342 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:32:51.341+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:32:51.342403+0000) 2024-03-04T19:32:52.342 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:32:52.341+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:32:52.342639+0000) 2024-03-04T19:32:53.342 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:32:53.342+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:32:53.342927+0000) 2024-03-04T19:32:54.342 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:32:54.342+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:32:54.343167+0000) 2024-03-04T19:32:55.343 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:32:55.342+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:32:55.343404+0000) 2024-03-04T19:32:55.481 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:32:55.491 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:32:56.343 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:32:56.343+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:32:56.343641+0000) 2024-03-04T19:32:57.343 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:32:57.343+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:32:57.343925+0000) 2024-03-04T19:32:58.343 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:32:58.343+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:32:58.344169+0000) 2024-03-04T19:32:59.343 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:32:59.343+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:32:59.344409+0000) 2024-03-04T19:33:00.344 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:33:00.343+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:33:00.344663+0000) 2024-03-04T19:33:01.344 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:33:01.344+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:33:01.344951+0000) 2024-03-04T19:33:02.344 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:33:02.344+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:33:02.345137+0000) 2024-03-04T19:33:03.345 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:33:03.344+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:33:03.345431+0000) 2024-03-04T19:33:04.345 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:33:04.345+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:33:04.345708+0000) 2024-03-04T19:33:05.345 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:33:05.345+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:33:05.345948+0000) 2024-03-04T19:33:06.346 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:33:06.345+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:33:06.346231+0000) 2024-03-04T19:33:07.346 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:33:07.345+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:33:07.346503+0000) 2024-03-04T19:33:08.346 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:33:08.346+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:33:08.346795+0000) 2024-03-04T19:33:09.346 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:33:09.346+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:33:09.347141+0000) 2024-03-04T19:33:10.347 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:33:10.346+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:33:10.347363+0000) 2024-03-04T19:33:11.347 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:33:11.346+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:33:11.347667+0000) 2024-03-04T19:33:12.347 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:33:12.347+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:33:12.347989+0000) 2024-03-04T19:33:13.347 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:33:13.347+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:33:13.348280+0000) 2024-03-04T19:33:14.348 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:33:14.348+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:33:14.348582+0000) 2024-03-04T19:33:15.348 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:33:15.348+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:33:15.348888+0000) 2024-03-04T19:33:16.348 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:33:16.348+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:33:16.349176+0000) 2024-03-04T19:33:17.349 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:33:17.348+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:33:17.349407+0000) 2024-03-04T19:33:18.349 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:33:18.348+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:33:18.349726+0000) 2024-03-04T19:33:19.349 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:33:19.349+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:33:19.349972+0000) 2024-03-04T19:33:20.349 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:33:20.349+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:33:20.350195+0000) 2024-03-04T19:33:21.350 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:33:21.350+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:33:21.350482+0000) 2024-03-04T19:33:22.350 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:33:22.350+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:33:22.350783+0000) 2024-03-04T19:33:23.350 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:33:23.350+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:33:23.351114+0000) 2024-03-04T19:33:24.351 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:33:24.351+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:33:24.351421+0000) 2024-03-04T19:33:25.351 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:33:25.351+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:33:25.351652+0000) 2024-03-04T19:33:25.709 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:33:25.717 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:33:26.351 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:33:26.351+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:33:26.351970+0000) 2024-03-04T19:33:27.351 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:33:27.352+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:33:27.352267+0000) 2024-03-04T19:33:28.352 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:33:28.351+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:33:28.352559+0000) 2024-03-04T19:33:29.352 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:33:29.351+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:33:29.352913+0000) 2024-03-04T19:33:30.352 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:33:30.352+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:33:30.353204+0000) 2024-03-04T19:33:31.353 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:33:31.352+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:33:31.353485+0000) 2024-03-04T19:33:32.353 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:33:32.351+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:33:32.353777+0000) 2024-03-04T19:33:33.353 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:33:33.353+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:33:33.354097+0000) 2024-03-04T19:33:34.354 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:33:34.353+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:33:34.354397+0000) 2024-03-04T19:33:35.354 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:33:35.353+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:33:35.354677+0000) 2024-03-04T19:33:36.354 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:33:36.353+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:33:36.354965+0000) 2024-03-04T19:33:37.354 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:33:37.354+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:33:37.355241+0000) 2024-03-04T19:33:38.355 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:33:38.354+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:33:38.355539+0000) 2024-03-04T19:33:39.355 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:33:39.355+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:33:39.355891+0000) 2024-03-04T19:33:40.355 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:33:40.355+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:33:40.356171+0000) 2024-03-04T19:33:41.356 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:33:41.355+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:33:41.356464+0000) 2024-03-04T19:33:42.356 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:33:42.355+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:33:42.356755+0000) 2024-03-04T19:33:43.356 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:33:43.356+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:33:43.357047+0000) 2024-03-04T19:33:44.356 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:33:44.356+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:33:44.357350+0000) 2024-03-04T19:33:45.357 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:33:45.356+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:33:45.357646+0000) 2024-03-04T19:33:46.357 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:33:46.357+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:33:46.357979+0000) 2024-03-04T19:33:47.358 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:33:47.357+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:33:47.358247+0000) 2024-03-04T19:33:48.358 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:33:48.357+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:33:48.358524+0000) 2024-03-04T19:33:49.358 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:33:49.358+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:33:49.358814+0000) 2024-03-04T19:33:50.358 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:33:50.358+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:33:50.359157+0000) 2024-03-04T19:33:51.359 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:33:51.358+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:33:51.359462+0000) 2024-03-04T19:33:52.359 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:33:52.358+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:33:52.359750+0000) 2024-03-04T19:33:53.359 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:33:53.359+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:33:53.360028+0000) 2024-03-04T19:33:54.360 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:33:54.359+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:33:54.360354+0000) 2024-03-04T19:33:55.360 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:33:55.359+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:33:55.360660+0000) 2024-03-04T19:33:55.875 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:33:55.883 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:33:56.360 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:33:56.360+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:33:56.360969+0000) 2024-03-04T19:33:57.360 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:33:57.360+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:33:57.361252+0000) 2024-03-04T19:33:58.361 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:33:58.360+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:33:58.361542+0000) 2024-03-04T19:33:59.363 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:33:59.361+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:33:59.361888+0000) 2024-03-04T19:34:00.361 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:34:00.361+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:34:00.362180+0000) 2024-03-04T19:34:01.361 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:34:01.361+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:34:01.362409+0000) 2024-03-04T19:34:02.362 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:34:02.362+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:34:02.362700+0000) 2024-03-04T19:34:03.362 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:34:03.362+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:34:03.362945+0000) 2024-03-04T19:34:04.362 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:34:04.362+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:34:04.363195+0000) 2024-03-04T19:34:05.363 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:34:05.362+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:34:05.363434+0000) 2024-03-04T19:34:06.363 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:34:06.363+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:34:06.363668+0000) 2024-03-04T19:34:07.363 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:34:07.363+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:34:07.363937+0000) 2024-03-04T19:34:08.363 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:34:08.363+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:34:08.364177+0000) 2024-03-04T19:34:09.363 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:34:09.363+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:34:09.364472+0000) 2024-03-04T19:34:10.364 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:34:10.364+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:34:10.364710+0000) 2024-03-04T19:34:11.364 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:34:11.364+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:34:11.364958+0000) 2024-03-04T19:34:12.364 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:34:12.364+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:34:12.365193+0000) 2024-03-04T19:34:13.365 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:34:13.365+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:34:13.365435+0000) 2024-03-04T19:34:14.365 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:34:14.365+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:34:14.365681+0000) 2024-03-04T19:34:15.365 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:34:15.365+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:34:15.365955+0000) 2024-03-04T19:34:16.365 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:34:16.365+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:34:16.366185+0000) 2024-03-04T19:34:17.366 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:34:17.365+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:34:17.366412+0000) 2024-03-04T19:34:18.366 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:34:18.366+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:34:18.366710+0000) 2024-03-04T19:34:19.366 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:34:19.366+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:34:19.367003+0000) 2024-03-04T19:34:20.367 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:34:20.366+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:34:20.367290+0000) 2024-03-04T19:34:21.367 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:34:21.367+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:34:21.367578+0000) 2024-03-04T19:34:22.367 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:34:22.367+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:34:22.367923+0000) 2024-03-04T19:34:23.367 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:34:23.367+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:34:23.368208+0000) 2024-03-04T19:34:24.368 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:34:24.368+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:34:24.368512+0000) 2024-03-04T19:34:25.368 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:34:25.368+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:34:25.368803+0000) 2024-03-04T19:34:26.102 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:34:26.110 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:34:26.369 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:34:26.368+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:34:26.369139+0000) 2024-03-04T19:34:27.368 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:34:27.369+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:34:27.369419+0000) 2024-03-04T19:34:28.369 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:34:28.369+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:34:28.369747+0000) 2024-03-04T19:34:29.369 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:34:29.369+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:34:29.370045+0000) 2024-03-04T19:34:30.369 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:34:30.369+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:34:30.370342+0000) 2024-03-04T19:34:31.370 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:34:31.368+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:34:31.370630+0000) 2024-03-04T19:34:32.370 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:34:32.369+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:34:32.370952+0000) 2024-03-04T19:34:33.370 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:34:33.370+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:34:33.371261+0000) 2024-03-04T19:34:34.371 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:34:34.370+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:34:34.371541+0000) 2024-03-04T19:34:35.371 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:34:35.370+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:34:35.371875+0000) 2024-03-04T19:34:36.371 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:34:36.371+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:34:36.372175+0000) 2024-03-04T19:34:37.372 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:34:37.370+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:34:37.372482+0000) 2024-03-04T19:34:38.372 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:34:38.371+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:34:38.372771+0000) 2024-03-04T19:34:39.372 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:34:39.372+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:34:39.373062+0000) 2024-03-04T19:34:40.372 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:34:40.372+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:34:40.373359+0000) 2024-03-04T19:34:41.373 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:34:41.372+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:34:41.373660+0000) 2024-03-04T19:34:42.373 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:34:42.373+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:34:42.373973+0000) 2024-03-04T19:34:43.373 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:34:43.373+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:34:43.374252+0000) 2024-03-04T19:34:44.374 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:34:44.373+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:34:44.374554+0000) 2024-03-04T19:34:45.374 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:34:45.374+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:34:45.374890+0000) 2024-03-04T19:34:46.374 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:34:46.374+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:34:46.375185+0000) 2024-03-04T19:34:47.375 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:34:47.374+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:34:47.375487+0000) 2024-03-04T19:34:48.375 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:34:48.374+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:34:48.375793+0000) 2024-03-04T19:34:49.375 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:34:49.375+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:34:49.376138+0000) 2024-03-04T19:34:50.376 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:34:50.375+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:34:50.376431+0000) 2024-03-04T19:34:51.376 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:34:51.375+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:34:51.376730+0000) 2024-03-04T19:34:52.377 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:34:52.376+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:34:52.377087+0000) 2024-03-04T19:34:53.377 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:34:53.376+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:34:53.377329+0000) 2024-03-04T19:34:54.377 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:34:54.376+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:34:54.377643+0000) 2024-03-04T19:34:55.378 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:34:55.377+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:34:55.377979+0000) 2024-03-04T19:34:56.248 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:34:56.257 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:34:56.377 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:34:56.377+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:34:56.378287+0000) 2024-03-04T19:34:57.378 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:34:57.377+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:34:57.378568+0000) 2024-03-04T19:34:58.378 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:34:58.377+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:34:58.378801+0000) 2024-03-04T19:34:59.378 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:34:59.378+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:34:59.379122+0000) 2024-03-04T19:35:00.379 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:35:00.378+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:35:00.379425+0000) 2024-03-04T19:35:01.379 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:35:01.379+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:35:01.379708+0000) 2024-03-04T19:35:02.379 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:35:02.379+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:35:02.379985+0000) 2024-03-04T19:35:03.380 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:35:03.379+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:35:03.380275+0000) 2024-03-04T19:35:04.380 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:35:04.379+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:35:04.380579+0000) 2024-03-04T19:35:05.380 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:35:05.380+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:35:05.380903+0000) 2024-03-04T19:35:06.383 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:35:06.380+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:35:06.381176+0000) 2024-03-04T19:35:07.381 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:35:07.380+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:35:07.381473+0000) 2024-03-04T19:35:08.381 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:35:08.381+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:35:08.381758+0000) 2024-03-04T19:35:09.381 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:35:09.381+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:35:09.382048+0000) 2024-03-04T19:35:10.382 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:35:10.381+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:35:10.382328+0000) 2024-03-04T19:35:11.382 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:35:11.381+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:35:11.382613+0000) 2024-03-04T19:35:12.382 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:35:12.382+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:35:12.382933+0000) 2024-03-04T19:35:13.382 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:35:13.382+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:35:13.383207+0000) 2024-03-04T19:35:14.383 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:35:14.382+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:35:14.383404+0000) 2024-03-04T19:35:15.383 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:35:15.382+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:35:15.383642+0000) 2024-03-04T19:35:16.383 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:35:16.383+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:35:16.383937+0000) 2024-03-04T19:35:17.383 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:35:17.383+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:35:17.384172+0000) 2024-03-04T19:35:18.384 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:35:18.384+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:35:18.384411+0000) 2024-03-04T19:35:19.384 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:35:19.384+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:35:19.384649+0000) 2024-03-04T19:35:20.384 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:35:20.384+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:35:20.384926+0000) 2024-03-04T19:35:21.384 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:35:21.384+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:35:21.385168+0000) 2024-03-04T19:35:22.385 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:35:22.385+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:35:22.385402+0000) 2024-03-04T19:35:23.385 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:35:23.385+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:35:23.385638+0000) 2024-03-04T19:35:24.385 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:35:24.385+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:35:24.385933+0000) 2024-03-04T19:35:25.385 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:35:25.385+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:35:25.386175+0000) 2024-03-04T19:35:26.386 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:35:26.386+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:35:26.386413+0000) 2024-03-04T19:35:26.386 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:35:26.394 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:35:27.386 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:35:27.386+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:35:27.386661+0000) 2024-03-04T19:35:28.387 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:35:28.386+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:35:28.386934+0000) 2024-03-04T19:35:29.386 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:35:29.386+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:35:29.387123+0000) 2024-03-04T19:35:30.387 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:35:30.387+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:35:30.387360+0000) 2024-03-04T19:35:31.387 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:35:31.387+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:35:31.387600+0000) 2024-03-04T19:35:32.387 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:35:32.387+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:35:32.387865+0000) 2024-03-04T19:35:33.387 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:35:33.386+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:35:33.388104+0000) 2024-03-04T19:35:34.388 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:35:34.387+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:35:34.388345+0000) 2024-03-04T19:35:35.388 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:35:35.387+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:35:35.388591+0000) 2024-03-04T19:35:36.388 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:35:36.387+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:35:36.388854+0000) 2024-03-04T19:35:37.388 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:35:37.388+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:35:37.389093+0000) 2024-03-04T19:35:38.389 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:35:38.388+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:35:38.389327+0000) 2024-03-04T19:35:39.389 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:35:39.388+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:35:39.389568+0000) 2024-03-04T19:35:40.389 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:35:40.388+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:35:40.389854+0000) 2024-03-04T19:35:41.389 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:35:41.389+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:35:41.390099+0000) 2024-03-04T19:35:42.390 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:35:42.389+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:35:42.390335+0000) 2024-03-04T19:35:43.390 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:35:43.388+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:35:43.390519+0000) 2024-03-04T19:35:44.390 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:35:44.389+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:35:44.390791+0000) 2024-03-04T19:35:45.390 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:35:45.390+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:35:45.391023+0000) 2024-03-04T19:35:46.391 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:35:46.390+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:35:46.391264+0000) 2024-03-04T19:35:47.391 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:35:47.390+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:35:47.391503+0000) 2024-03-04T19:35:48.391 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:35:48.390+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:35:48.391741+0000) 2024-03-04T19:35:49.391 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:35:49.391+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:35:49.391981+0000) 2024-03-04T19:35:50.391 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:35:50.391+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:35:50.392214+0000) 2024-03-04T19:35:51.392 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:35:51.391+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:35:51.392458+0000) 2024-03-04T19:35:52.392 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:35:52.391+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:35:52.392700+0000) 2024-03-04T19:35:53.392 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:35:53.392+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:35:53.392941+0000) 2024-03-04T19:35:54.392 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:35:54.392+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:35:54.393193+0000) 2024-03-04T19:35:55.393 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:35:55.392+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:35:55.393427+0000) 2024-03-04T19:35:56.393 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:35:56.392+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:35:56.393676+0000) 2024-03-04T19:35:56.525 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:35:56.533 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:35:57.393 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:35:57.393+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:35:57.393907+0000) 2024-03-04T19:35:58.393 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:35:58.393+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:35:58.394151+0000) 2024-03-04T19:35:59.394 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:35:59.393+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:35:59.394384+0000) 2024-03-04T19:36:00.394 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:36:00.393+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:36:00.394560+0000) 2024-03-04T19:36:01.394 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:36:01.393+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:36:01.394752+0000) 2024-03-04T19:36:02.394 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:36:02.394+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:36:02.394941+0000) 2024-03-04T19:36:03.394 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:36:03.394+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:36:03.395135+0000) 2024-03-04T19:36:04.395 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:36:04.394+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:36:04.395384+0000) 2024-03-04T19:36:05.395 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:36:05.394+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:36:05.395635+0000) 2024-03-04T19:36:06.395 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:36:06.395+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:36:06.395939+0000) 2024-03-04T19:36:07.396 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:36:07.395+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:36:07.396197+0000) 2024-03-04T19:36:08.396 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:36:08.395+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:36:08.396457+0000) 2024-03-04T19:36:09.396 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:36:09.396+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:36:09.396691+0000) 2024-03-04T19:36:10.399 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:36:10.396+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:36:10.396924+0000) 2024-03-04T19:36:11.396 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:36:11.396+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:36:11.397165+0000) 2024-03-04T19:36:12.397 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:36:12.396+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:36:12.397403+0000) 2024-03-04T19:36:13.397 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:36:13.397+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:36:13.397637+0000) 2024-03-04T19:36:14.397 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:36:14.397+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:36:14.397934+0000) 2024-03-04T19:36:15.398 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:36:15.397+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:36:15.398184+0000) 2024-03-04T19:36:16.398 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:36:16.398+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:36:16.398443+0000) 2024-03-04T19:36:17.398 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:36:17.398+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:36:17.398705+0000) 2024-03-04T19:36:18.398 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:36:18.398+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:36:18.398945+0000) 2024-03-04T19:36:19.399 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:36:19.398+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:36:19.399180+0000) 2024-03-04T19:36:20.399 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:36:20.399+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:36:20.399414+0000) 2024-03-04T19:36:21.399 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:36:21.399+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:36:21.399649+0000) 2024-03-04T19:36:22.399 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:36:22.399+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:36:22.399922+0000) 2024-03-04T19:36:23.399 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:36:23.399+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:36:23.400160+0000) 2024-03-04T19:36:24.400 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:36:24.400+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:36:24.400401+0000) 2024-03-04T19:36:25.400 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:36:25.399+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:36:25.400636+0000) 2024-03-04T19:36:26.400 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:36:26.400+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:36:26.400913+0000) 2024-03-04T19:36:26.738 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:36:26.750 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:36:27.400 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:36:27.400+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:36:27.401152+0000) 2024-03-04T19:36:28.401 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:36:28.401+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:36:28.401391+0000) 2024-03-04T19:36:29.401 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:36:29.401+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:36:29.401622+0000) 2024-03-04T19:36:30.401 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:36:30.401+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:36:30.401896+0000) 2024-03-04T19:36:31.401 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:36:31.401+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:36:31.402150+0000) 2024-03-04T19:36:32.402 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:36:32.401+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:36:32.402419+0000) 2024-03-04T19:36:33.402 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:36:33.402+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:36:33.402659+0000) 2024-03-04T19:36:34.402 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:36:34.402+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:36:34.402939+0000) 2024-03-04T19:36:35.402 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:36:35.402+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:36:35.403181+0000) 2024-03-04T19:36:36.403 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:36:36.402+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:36:36.403414+0000) 2024-03-04T19:36:37.403 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:36:37.402+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:36:37.403688+0000) 2024-03-04T19:36:38.403 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:36:38.402+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:36:38.403924+0000) 2024-03-04T19:36:39.404 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:36:39.403+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:36:39.404211+0000) 2024-03-04T19:36:40.404 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:36:40.403+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:36:40.404446+0000) 2024-03-04T19:36:41.404 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:36:41.403+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:36:41.404679+0000) 2024-03-04T19:36:42.404 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:36:42.403+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:36:42.404945+0000) 2024-03-04T19:36:43.404 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:36:43.404+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:36:43.405184+0000) 2024-03-04T19:36:44.405 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:36:44.403+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:36:44.405432+0000) 2024-03-04T19:36:45.405 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:36:45.404+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:36:45.405682+0000) 2024-03-04T19:36:46.405 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:36:46.405+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:36:46.405943+0000) 2024-03-04T19:36:47.405 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:36:47.405+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:36:47.406182+0000) 2024-03-04T19:36:48.406 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:36:48.405+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:36:48.406425+0000) 2024-03-04T19:36:49.406 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:36:49.405+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:36:49.406665+0000) 2024-03-04T19:36:50.406 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:36:50.406+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:36:50.406936+0000) 2024-03-04T19:36:51.407 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:36:51.406+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:36:51.407181+0000) 2024-03-04T19:36:52.407 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:36:52.406+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:36:52.407414+0000) 2024-03-04T19:36:53.407 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:36:53.406+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:36:53.407655+0000) 2024-03-04T19:36:54.407 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:36:54.407+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:36:54.407938+0000) 2024-03-04T19:36:55.408 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:36:55.407+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:36:55.408208+0000) 2024-03-04T19:36:56.408 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:36:56.407+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:36:56.408450+0000) 2024-03-04T19:36:56.891 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:36:56.899 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:36:57.408 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:36:57.407+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:36:57.408691+0000) 2024-03-04T19:36:58.408 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:36:58.408+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:36:58.408960+0000) 2024-03-04T19:36:59.409 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:36:59.408+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:36:59.409150+0000) 2024-03-04T19:37:00.409 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:37:00.408+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:37:00.409391+0000) 2024-03-04T19:37:01.409 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:37:01.408+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:37:01.409656+0000) 2024-03-04T19:37:02.409 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:37:02.409+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:37:02.409923+0000) 2024-03-04T19:37:03.409 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:37:03.409+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:37:03.410198+0000) 2024-03-04T19:37:04.410 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:37:04.409+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:37:04.410504+0000) 2024-03-04T19:37:05.410 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:37:05.410+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:37:05.410794+0000) 2024-03-04T19:37:06.410 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:37:06.410+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:37:06.411118+0000) 2024-03-04T19:37:07.411 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:37:07.410+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:37:07.411414+0000) 2024-03-04T19:37:08.412 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:37:08.410+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:37:08.411714+0000) 2024-03-04T19:37:09.411 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:37:09.411+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:37:09.411999+0000) 2024-03-04T19:37:10.412 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:37:10.411+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:37:10.412243+0000) 2024-03-04T19:37:11.412 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:37:11.411+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:37:11.412525+0000) 2024-03-04T19:37:12.412 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:37:12.412+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:37:12.412816+0000) 2024-03-04T19:37:13.412 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:37:13.412+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:37:13.413153+0000) 2024-03-04T19:37:14.413 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:37:14.412+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:37:14.413472+0000) 2024-03-04T19:37:15.413 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:37:15.413+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:37:15.413763+0000) 2024-03-04T19:37:16.413 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:37:16.413+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:37:16.414054+0000) 2024-03-04T19:37:17.413 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:37:17.413+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:37:17.414296+0000) 2024-03-04T19:37:18.414 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:37:18.413+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:37:18.414496+0000) 2024-03-04T19:37:19.414 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:37:19.414+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:37:19.414731+0000) 2024-03-04T19:37:20.414 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:37:20.414+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:37:20.414992+0000) 2024-03-04T19:37:21.415 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:37:21.414+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:37:21.415236+0000) 2024-03-04T19:37:22.416 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:37:22.414+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:37:22.415427+0000) 2024-03-04T19:37:23.415 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:37:23.415+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:37:23.415666+0000) 2024-03-04T19:37:24.415 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:37:24.415+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:37:24.415894+0000) 2024-03-04T19:37:25.415 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:37:25.415+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:37:25.416123+0000) 2024-03-04T19:37:26.415 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:37:26.415+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:37:26.416310+0000) 2024-03-04T19:37:27.056 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:37:27.064 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:37:27.416 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:37:27.416+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:37:27.416616+0000) 2024-03-04T19:37:28.416 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:37:28.416+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:37:28.416858+0000) 2024-03-04T19:37:29.416 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:37:29.416+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:37:29.417110+0000) 2024-03-04T19:37:30.417 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:37:30.416+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:37:30.417288+0000) 2024-03-04T19:37:31.417 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:37:31.417+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:37:31.417550+0000) 2024-03-04T19:37:32.417 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:37:32.416+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:37:32.417767+0000) 2024-03-04T19:37:33.418 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:37:33.417+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:37:33.418061+0000) 2024-03-04T19:37:34.418 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:37:34.417+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:37:34.418308+0000) 2024-03-04T19:37:35.418 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:37:35.418+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:37:35.418597+0000) 2024-03-04T19:37:36.418 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:37:36.418+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:37:36.418867+0000) 2024-03-04T19:37:37.418 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:37:37.417+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:37:37.419125+0000) 2024-03-04T19:37:38.419 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:37:38.417+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:37:38.419405+0000) 2024-03-04T19:37:39.419 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:37:39.418+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:37:39.419690+0000) 2024-03-04T19:37:40.419 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:37:40.418+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:37:40.419969+0000) 2024-03-04T19:37:41.419 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:37:41.419+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:37:41.420250+0000) 2024-03-04T19:37:42.420 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:37:42.419+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:37:42.420530+0000) 2024-03-04T19:37:43.420 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:37:43.419+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:37:43.420878+0000) 2024-03-04T19:37:44.421 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:37:44.420+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:37:44.421160+0000) 2024-03-04T19:37:45.421 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:37:45.420+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:37:45.421391+0000) 2024-03-04T19:37:46.421 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:37:46.420+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:37:46.421629+0000) 2024-03-04T19:37:47.421 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:37:47.420+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:37:47.421920+0000) 2024-03-04T19:37:48.421 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:37:48.421+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:37:48.422180+0000) 2024-03-04T19:37:49.422 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:37:49.421+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:37:49.422411+0000) 2024-03-04T19:37:50.422 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:37:50.421+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:37:50.422642+0000) 2024-03-04T19:37:51.422 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:37:51.422+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:37:51.422934+0000) 2024-03-04T19:37:52.422 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:37:52.422+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:37:52.423168+0000) 2024-03-04T19:37:53.423 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:37:53.422+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:37:53.423409+0000) 2024-03-04T19:37:54.423 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:37:54.422+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:37:54.423659+0000) 2024-03-04T19:37:55.423 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:37:55.423+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:37:55.423932+0000) 2024-03-04T19:37:56.423 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:37:56.423+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:37:56.424169+0000) 2024-03-04T19:37:57.270 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:37:57.278 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:37:57.425 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:37:57.423+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:37:57.424406+0000) 2024-03-04T19:37:58.424 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:37:58.423+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:37:58.424641+0000) 2024-03-04T19:37:59.424 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:37:59.424+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:37:59.424928+0000) 2024-03-04T19:38:00.424 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:38:00.424+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:38:00.425160+0000) 2024-03-04T19:38:01.425 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:38:01.424+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:38:01.425399+0000) 2024-03-04T19:38:02.425 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:38:02.424+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:38:02.425675+0000) 2024-03-04T19:38:03.425 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:38:03.425+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:38:03.425974+0000) 2024-03-04T19:38:04.425 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:38:04.425+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:38:04.426273+0000) 2024-03-04T19:38:05.426 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:38:05.425+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:38:05.426557+0000) 2024-03-04T19:38:06.426 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:38:06.426+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:38:06.426874+0000) 2024-03-04T19:38:07.427 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:38:07.426+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:38:07.427183+0000) 2024-03-04T19:38:08.427 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:38:08.426+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:38:08.427478+0000) 2024-03-04T19:38:09.427 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:38:09.427+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:38:09.427768+0000) 2024-03-04T19:38:10.428 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:38:10.427+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:38:10.428067+0000) 2024-03-04T19:38:11.428 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:38:11.427+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:38:11.428373+0000) 2024-03-04T19:38:12.428 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:38:12.428+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:38:12.428694+0000) 2024-03-04T19:38:13.428 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:38:13.428+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:38:13.428990+0000) 2024-03-04T19:38:14.429 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:38:14.428+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:38:14.429283+0000) 2024-03-04T19:38:15.429 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:38:15.428+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:38:15.429604+0000) 2024-03-04T19:38:16.430 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:38:16.429+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:38:16.429928+0000) 2024-03-04T19:38:17.430 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:38:17.429+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:38:17.430222+0000) 2024-03-04T19:38:18.430 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:38:18.429+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:38:18.430452+0000) 2024-03-04T19:38:19.430 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:38:19.429+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:38:19.430748+0000) 2024-03-04T19:38:20.431 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:38:20.430+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:38:20.431094+0000) 2024-03-04T19:38:21.431 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:38:21.430+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:38:21.431384+0000) 2024-03-04T19:38:22.431 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:38:22.431+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:38:22.431683+0000) 2024-03-04T19:38:23.431 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:38:23.431+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:38:23.431995+0000) 2024-03-04T19:38:24.432 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:38:24.431+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:38:24.432287+0000) 2024-03-04T19:38:25.432 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:38:25.432+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:38:25.432576+0000) 2024-03-04T19:38:26.432 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:38:26.432+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:38:26.432909+0000) 2024-03-04T19:38:27.432 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:38:27.432+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:38:27.433211+0000) 2024-03-04T19:38:27.499 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:38:27.506 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:38:28.433 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:38:28.433+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:38:28.433501+0000) 2024-03-04T19:38:29.433 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:38:29.433+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:38:29.433793+0000) 2024-03-04T19:38:30.433 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:38:30.433+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:38:30.434177+0000) 2024-03-04T19:38:31.434 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:38:31.434+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:38:31.434469+0000) 2024-03-04T19:38:32.434 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:38:32.434+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:38:32.434762+0000) 2024-03-04T19:38:33.434 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:38:33.434+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:38:33.435052+0000) 2024-03-04T19:38:34.435 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:38:34.435+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:38:34.435352+0000) 2024-03-04T19:38:35.435 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:38:35.435+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:38:35.435648+0000) 2024-03-04T19:38:36.435 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:38:36.435+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:38:36.435975+0000) 2024-03-04T19:38:37.436 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:38:37.436+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:38:37.436281+0000) 2024-03-04T19:38:38.436 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:38:38.436+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:38:38.436570+0000) 2024-03-04T19:38:39.436 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:38:39.436+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:38:39.436902+0000) 2024-03-04T19:38:40.436 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:38:40.435+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:38:40.437132+0000) 2024-03-04T19:38:41.437 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:38:41.436+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:38:41.437418+0000) 2024-03-04T19:38:42.437 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:38:42.436+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:38:42.437719+0000) 2024-03-04T19:38:43.437 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:38:43.436+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:38:43.437977+0000) 2024-03-04T19:38:44.438 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:38:44.437+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:38:44.438255+0000) 2024-03-04T19:38:45.438 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:38:45.437+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:38:45.438488+0000) 2024-03-04T19:38:46.438 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:38:46.436+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:38:46.438718+0000) 2024-03-04T19:38:47.438 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:38:47.437+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:38:47.438996+0000) 2024-03-04T19:38:48.439 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:38:48.438+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:38:48.439234+0000) 2024-03-04T19:38:49.439 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:38:49.438+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:38:49.439467+0000) 2024-03-04T19:38:50.439 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:38:50.438+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:38:50.439735+0000) 2024-03-04T19:38:51.439 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:38:51.439+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:38:51.439982+0000) 2024-03-04T19:38:52.439 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:38:52.439+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:38:52.440226+0000) 2024-03-04T19:38:53.439 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:38:53.439+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:38:53.440416+0000) 2024-03-04T19:38:54.440 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:38:54.439+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:38:54.440604+0000) 2024-03-04T19:38:55.440 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:38:55.439+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:38:55.440780+0000) 2024-03-04T19:38:56.440 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:38:56.440+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:38:56.441064+0000) 2024-03-04T19:38:57.440 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:38:57.440+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:38:57.441308+0000) 2024-03-04T19:38:57.646 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:38:57.655 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:38:58.441 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:38:58.440+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:38:58.441543+0000) 2024-03-04T19:38:59.441 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:38:59.440+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:38:59.441796+0000) 2024-03-04T19:39:00.441 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:39:00.441+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:39:00.442067+0000) 2024-03-04T19:39:01.441 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:39:01.441+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:39:01.442318+0000) 2024-03-04T19:39:02.442 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:39:02.441+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:39:02.442579+0000) 2024-03-04T19:39:03.442 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:39:03.442+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:39:03.442910+0000) 2024-03-04T19:39:04.442 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:39:04.442+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:39:04.443157+0000) 2024-03-04T19:39:05.443 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:39:05.442+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:39:05.443387+0000) 2024-03-04T19:39:06.443 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:39:06.442+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:39:06.443647+0000) 2024-03-04T19:39:07.443 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:39:07.443+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:39:07.443942+0000) 2024-03-04T19:39:08.443 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:39:08.443+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:39:08.444181+0000) 2024-03-04T19:39:09.444 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:39:09.443+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:39:09.444419+0000) 2024-03-04T19:39:10.444 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:39:10.444+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:39:10.444653+0000) 2024-03-04T19:39:11.444 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:39:11.444+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:39:11.444948+0000) 2024-03-04T19:39:12.445 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:39:12.444+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:39:12.445187+0000) 2024-03-04T19:39:13.445 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:39:13.444+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:39:13.445427+0000) 2024-03-04T19:39:14.445 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:39:14.445+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:39:14.445670+0000) 2024-03-04T19:39:15.445 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:39:15.445+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:39:15.445943+0000) 2024-03-04T19:39:16.445 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:39:16.445+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:39:16.446191+0000) 2024-03-04T19:39:17.446 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:39:17.445+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:39:17.446429+0000) 2024-03-04T19:39:18.446 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:39:18.446+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:39:18.446669+0000) 2024-03-04T19:39:19.446 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:39:19.446+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:39:19.446927+0000) 2024-03-04T19:39:20.446 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:39:20.446+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:39:20.447157+0000) 2024-03-04T19:39:21.447 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:39:21.446+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:39:21.447397+0000) 2024-03-04T19:39:22.447 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:39:22.447+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:39:22.447648+0000) 2024-03-04T19:39:23.447 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:39:23.447+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:39:23.447938+0000) 2024-03-04T19:39:24.447 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:39:24.447+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:39:24.448204+0000) 2024-03-04T19:39:25.448 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:39:25.447+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:39:25.448437+0000) 2024-03-04T19:39:26.448 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:39:26.447+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:39:26.448663+0000) 2024-03-04T19:39:27.448 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:39:27.448+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:39:27.448939+0000) 2024-03-04T19:39:27.818 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:39:27.826 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:39:28.448 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:39:28.448+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:39:28.449176+0000) 2024-03-04T19:39:29.449 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:39:29.449+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:39:29.449411+0000) 2024-03-04T19:39:30.449 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:39:30.449+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:39:30.449647+0000) 2024-03-04T19:39:31.449 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:39:31.449+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:39:31.449942+0000) 2024-03-04T19:39:32.449 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:39:32.449+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:39:32.450183+0000) 2024-03-04T19:39:33.450 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:39:33.449+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:39:33.450422+0000) 2024-03-04T19:39:34.450 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:39:34.450+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:39:34.450666+0000) 2024-03-04T19:39:35.450 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:39:35.450+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:39:35.450945+0000) 2024-03-04T19:39:36.450 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:39:36.450+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:39:36.451171+0000) 2024-03-04T19:39:37.451 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:39:37.450+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:39:37.451468+0000) 2024-03-04T19:39:38.451 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:39:38.450+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:39:38.451752+0000) 2024-03-04T19:39:39.451 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:39:39.451+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:39:39.452030+0000) 2024-03-04T19:39:40.452 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:39:40.452+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:39:40.452334+0000) 2024-03-04T19:39:41.452 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:39:41.452+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:39:41.452619+0000) 2024-03-04T19:39:42.452 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:39:42.451+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:39:42.452933+0000) 2024-03-04T19:39:43.452 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:39:43.452+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:39:43.453186+0000) 2024-03-04T19:39:44.453 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:39:44.451+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:39:44.453491+0000) 2024-03-04T19:39:45.453 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:39:45.452+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:39:45.453771+0000) 2024-03-04T19:39:46.453 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:39:46.452+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:39:46.454069+0000) 2024-03-04T19:39:47.454 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:39:47.453+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:39:47.454358+0000) 2024-03-04T19:39:48.454 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:39:48.453+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:39:48.454655+0000) 2024-03-04T19:39:49.454 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:39:49.453+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:39:49.454977+0000) 2024-03-04T19:39:50.455 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:39:50.454+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:39:50.455270+0000) 2024-03-04T19:39:51.455 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:39:51.454+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:39:51.455562+0000) 2024-03-04T19:39:52.455 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:39:52.454+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:39:52.455879+0000) 2024-03-04T19:39:53.455 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:39:53.455+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:39:53.456165+0000) 2024-03-04T19:39:54.456 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:39:54.455+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:39:54.456465+0000) 2024-03-04T19:39:55.456 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:39:55.455+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:39:55.456752+0000) 2024-03-04T19:39:56.456 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:39:56.456+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:39:56.457121+0000) 2024-03-04T19:39:57.457 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:39:57.456+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:39:57.457406+0000) 2024-03-04T19:39:57.956 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:39:57.968 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:39:58.457 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:39:58.456+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:39:58.457690+0000) 2024-03-04T19:39:59.457 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:39:59.457+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:39:59.457985+0000) 2024-03-04T19:40:00.458 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:40:00.457+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:40:00.458276+0000) 2024-03-04T19:40:01.458 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:40:01.457+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:40:01.458571+0000) 2024-03-04T19:40:02.458 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:40:02.458+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:40:02.458808+0000) 2024-03-04T19:40:03.459 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:40:03.458+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:40:03.459168+0000) 2024-03-04T19:40:04.459 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:40:04.458+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:40:04.459465+0000) 2024-03-04T19:40:05.459 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:40:05.458+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:40:05.459748+0000) 2024-03-04T19:40:06.459 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:40:06.459+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:40:06.460009+0000) 2024-03-04T19:40:07.460 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:40:07.459+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:40:07.460256+0000) 2024-03-04T19:40:08.460 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:40:08.459+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:40:08.460487+0000) 2024-03-04T19:40:09.460 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:40:09.459+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:40:09.460751+0000) 2024-03-04T19:40:10.461 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:40:10.460+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:40:10.460985+0000) 2024-03-04T19:40:11.460 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:40:11.460+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:40:11.461229+0000) 2024-03-04T19:40:12.461 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:40:12.460+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:40:12.461468+0000) 2024-03-04T19:40:13.461 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:40:13.461+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:40:13.461707+0000) 2024-03-04T19:40:14.461 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:40:14.461+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:40:14.461954+0000) 2024-03-04T19:40:15.461 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:40:15.461+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:40:15.462198+0000) 2024-03-04T19:40:16.462 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:40:16.461+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:40:16.462436+0000) 2024-03-04T19:40:17.462 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:40:17.461+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:40:17.462696+0000) 2024-03-04T19:40:18.462 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:40:18.462+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:40:18.462935+0000) 2024-03-04T19:40:19.463 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:40:19.462+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:40:19.463173+0000) 2024-03-04T19:40:20.463 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:40:20.462+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:40:20.463408+0000) 2024-03-04T19:40:21.463 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:40:21.463+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:40:21.463644+0000) 2024-03-04T19:40:22.463 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:40:22.463+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:40:22.463935+0000) 2024-03-04T19:40:23.463 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:40:23.463+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:40:23.464168+0000) 2024-03-04T19:40:24.464 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:40:24.463+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:40:24.464415+0000) 2024-03-04T19:40:25.464 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:40:25.464+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:40:25.464647+0000) 2024-03-04T19:40:26.464 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:40:26.464+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:40:26.464934+0000) 2024-03-04T19:40:27.464 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:40:27.464+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:40:27.465180+0000) 2024-03-04T19:40:28.178 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:40:28.186 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:40:28.464 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:40:28.465+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:40:28.465425+0000) 2024-03-04T19:40:29.465 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:40:29.465+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:40:29.465665+0000) 2024-03-04T19:40:30.465 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:40:30.465+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:40:30.465948+0000) 2024-03-04T19:40:31.465 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:40:31.465+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:40:31.466210+0000) 2024-03-04T19:40:32.466 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:40:32.466+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:40:32.466472+0000) 2024-03-04T19:40:33.466 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:40:33.465+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:40:33.466731+0000) 2024-03-04T19:40:34.466 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:40:34.466+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:40:34.466994+0000) 2024-03-04T19:40:35.466 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:40:35.466+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:40:35.467232+0000) 2024-03-04T19:40:36.467 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:40:36.467+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:40:36.467468+0000) 2024-03-04T19:40:37.467 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:40:37.466+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:40:37.467666+0000) 2024-03-04T19:40:38.467 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:40:38.467+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:40:38.467939+0000) 2024-03-04T19:40:39.467 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:40:39.467+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:40:39.468184+0000) 2024-03-04T19:40:40.468 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:40:40.468+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:40:40.468424+0000) 2024-03-04T19:40:41.468 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:40:41.468+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:40:41.468662+0000) 2024-03-04T19:40:42.468 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:40:42.468+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:40:42.468944+0000) 2024-03-04T19:40:43.468 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:40:43.469+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:40:43.469181+0000) 2024-03-04T19:40:44.469 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:40:44.469+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:40:44.469424+0000) 2024-03-04T19:40:45.469 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:40:45.468+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:40:45.469669+0000) 2024-03-04T19:40:46.469 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:40:46.468+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:40:46.469943+0000) 2024-03-04T19:40:47.470 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:40:47.469+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:40:47.470185+0000) 2024-03-04T19:40:48.470 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:40:48.469+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:40:48.470421+0000) 2024-03-04T19:40:49.470 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:40:49.468+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:40:49.470704+0000) 2024-03-04T19:40:50.470 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:40:50.469+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:40:50.470962+0000) 2024-03-04T19:40:51.470 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:40:51.470+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:40:51.471207+0000) 2024-03-04T19:40:52.471 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:40:52.470+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:40:52.471400+0000) 2024-03-04T19:40:53.471 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:40:53.469+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:40:53.471589+0000) 2024-03-04T19:40:54.471 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:40:54.470+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:40:54.471871+0000) 2024-03-04T19:40:55.471 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:40:55.471+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:40:55.472128+0000) 2024-03-04T19:40:56.471 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:40:56.471+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:40:56.472383+0000) 2024-03-04T19:40:57.472 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:40:57.471+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:40:57.472636+0000) 2024-03-04T19:40:58.366 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:40:58.373 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:40:58.500 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:40:58.472+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:40:58.472921+0000) 2024-03-04T19:40:59.472 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:40:59.472+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:40:59.473176+0000) 2024-03-04T19:41:00.473 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:41:00.472+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:41:00.473430+0000) 2024-03-04T19:41:01.473 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:41:01.472+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:41:01.473683+0000) 2024-03-04T19:41:02.473 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:41:02.473+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:41:02.473961+0000) 2024-03-04T19:41:03.474 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:41:03.473+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:41:03.474209+0000) 2024-03-04T19:41:04.474 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:41:04.473+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:41:04.474477+0000) 2024-03-04T19:41:05.474 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:41:05.473+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:41:05.474729+0000) 2024-03-04T19:41:06.474 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:41:06.474+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:41:06.474982+0000) 2024-03-04T19:41:07.474 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:41:07.474+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:41:07.475236+0000) 2024-03-04T19:41:08.475 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:41:08.474+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:41:08.475488+0000) 2024-03-04T19:41:09.475 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:41:09.475+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:41:09.475731+0000) 2024-03-04T19:41:10.475 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:41:10.475+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:41:10.475970+0000) 2024-03-04T19:41:11.475 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:41:11.475+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:41:11.476211+0000) 2024-03-04T19:41:12.476 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:41:12.475+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:41:12.476467+0000) 2024-03-04T19:41:13.476 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:41:13.476+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:41:13.476753+0000) 2024-03-04T19:41:14.476 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:41:14.476+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:41:14.477054+0000) 2024-03-04T19:41:15.477 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:41:15.476+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:41:15.477346+0000) 2024-03-04T19:41:16.477 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:41:16.476+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:41:16.477626+0000) 2024-03-04T19:41:17.477 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:41:17.477+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:41:17.477944+0000) 2024-03-04T19:41:18.478 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:41:18.477+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:41:18.478235+0000) 2024-03-04T19:41:19.478 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:41:19.477+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:41:19.478508+0000) 2024-03-04T19:41:20.478 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:41:20.477+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:41:20.478738+0000) 2024-03-04T19:41:21.478 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:41:21.478+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:41:21.479022+0000) 2024-03-04T19:41:22.478 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:41:22.478+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:41:22.479311+0000) 2024-03-04T19:41:23.479 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:41:23.479+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:41:23.479594+0000) 2024-03-04T19:41:24.479 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:41:24.479+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:41:24.479933+0000) 2024-03-04T19:41:25.479 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:41:25.479+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:41:25.480222+0000) 2024-03-04T19:41:26.480 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:41:26.480+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:41:26.480520+0000) 2024-03-04T19:41:27.480 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:41:27.480+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:41:27.480811+0000) 2024-03-04T19:41:28.480 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:41:28.480+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:41:28.481136+0000) 2024-03-04T19:41:28.563 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:41:28.571 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:41:29.481 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:41:29.481+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:41:29.481446+0000) 2024-03-04T19:41:30.481 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:41:30.481+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:41:30.481757+0000) 2024-03-04T19:41:31.481 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:41:31.481+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:41:31.482010+0000) 2024-03-04T19:41:32.482 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:41:32.481+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:41:32.482307+0000) 2024-03-04T19:41:33.482 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:41:33.482+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:41:33.482617+0000) 2024-03-04T19:41:34.482 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:41:34.482+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:41:34.482953+0000) 2024-03-04T19:41:35.482 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:41:35.482+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:41:35.483179+0000) 2024-03-04T19:41:36.483 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:41:36.483+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:41:36.483472+0000) 2024-03-04T19:41:37.483 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:41:37.483+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:41:37.483807+0000) 2024-03-04T19:41:38.483 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:41:38.483+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:41:38.484147+0000) 2024-03-04T19:41:39.484 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:41:39.484+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:41:39.484401+0000) 2024-03-04T19:41:40.484 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:41:40.484+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:41:40.484640+0000) 2024-03-04T19:41:41.484 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:41:41.484+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:41:41.484931+0000) 2024-03-04T19:41:42.485 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:41:42.484+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:41:42.485164+0000) 2024-03-04T19:41:43.485 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:41:43.485+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:41:43.485408+0000) 2024-03-04T19:41:44.485 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:41:44.485+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:41:44.485713+0000) 2024-03-04T19:41:45.485 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:41:45.485+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:41:45.485963+0000) 2024-03-04T19:41:46.485 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:41:46.486+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:41:46.486211+0000) 2024-03-04T19:41:47.486 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:41:47.485+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:41:47.486466+0000) 2024-03-04T19:41:48.486 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:41:48.485+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:41:48.486720+0000) 2024-03-04T19:41:49.486 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:41:49.485+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:41:49.486973+0000) 2024-03-04T19:41:50.486 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:41:50.486+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:41:50.487204+0000) 2024-03-04T19:41:51.487 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:41:51.486+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:41:51.487441+0000) 2024-03-04T19:41:52.487 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:41:52.485+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:41:52.487689+0000) 2024-03-04T19:41:53.487 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:41:53.486+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:41:53.487936+0000) 2024-03-04T19:41:54.487 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:41:54.486+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:41:54.488126+0000) 2024-03-04T19:41:55.488 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:41:55.487+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:41:55.488363+0000) 2024-03-04T19:41:56.488 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:41:56.487+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:41:56.488597+0000) 2024-03-04T19:41:57.488 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:41:57.487+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:41:57.488780+0000) 2024-03-04T19:41:58.488 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:41:58.488+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:41:58.489012+0000) 2024-03-04T19:41:58.715 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:41:58.724 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:41:59.488 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:41:59.488+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:41:59.489282+0000) 2024-03-04T19:42:00.489 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:42:00.488+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:42:00.489511+0000) 2024-03-04T19:42:01.489 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:42:01.488+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:42:01.489745+0000) 2024-03-04T19:42:02.489 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:42:02.489+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:42:02.489958+0000) 2024-03-04T19:42:03.489 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:42:03.489+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:42:03.490196+0000) 2024-03-04T19:42:04.490 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:42:04.489+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:42:04.490442+0000) 2024-03-04T19:42:05.490 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:42:05.489+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:42:05.490680+0000) 2024-03-04T19:42:06.490 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:42:06.490+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:42:06.490957+0000) 2024-03-04T19:42:07.490 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:42:07.490+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:42:07.491238+0000) 2024-03-04T19:42:08.491 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:42:08.490+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:42:08.491489+0000) 2024-03-04T19:42:09.491 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:42:09.490+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:42:09.491706+0000) 2024-03-04T19:42:10.491 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:42:10.491+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:42:10.491970+0000) 2024-03-04T19:42:11.492 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:42:11.491+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:42:11.492228+0000) 2024-03-04T19:42:12.492 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:42:12.491+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:42:12.492480+0000) 2024-03-04T19:42:13.492 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:42:13.492+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:42:13.492742+0000) 2024-03-04T19:42:14.492 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:42:14.492+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:42:14.493038+0000) 2024-03-04T19:42:15.492 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:42:15.492+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:42:15.493319+0000) 2024-03-04T19:42:16.493 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:42:16.492+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:42:16.493571+0000) 2024-03-04T19:42:17.493 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:42:17.493+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:42:17.493812+0000) 2024-03-04T19:42:18.493 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:42:18.493+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:42:18.494042+0000) 2024-03-04T19:42:19.494 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:42:19.493+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:42:19.494277+0000) 2024-03-04T19:42:20.494 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:42:20.493+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:42:20.494564+0000) 2024-03-04T19:42:21.494 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:42:21.494+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:42:21.494800+0000) 2024-03-04T19:42:22.494 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:42:22.494+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:42:22.495066+0000) 2024-03-04T19:42:23.495 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:42:23.494+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:42:23.495305+0000) 2024-03-04T19:42:24.495 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:42:24.495+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:42:24.495556+0000) 2024-03-04T19:42:25.495 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:42:25.495+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:42:25.495789+0000) 2024-03-04T19:42:26.495 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:42:26.495+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:42:26.496030+0000) 2024-03-04T19:42:27.496 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:42:27.495+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:42:27.496291+0000) 2024-03-04T19:42:28.496 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:42:28.496+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:42:28.496556+0000) 2024-03-04T19:42:28.914 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:42:28.925 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:42:29.496 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:42:29.496+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:42:29.496849+0000) 2024-03-04T19:42:30.496 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:42:30.496+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:42:30.497094+0000) 2024-03-04T19:42:31.497 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:42:31.496+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:42:31.497335+0000) 2024-03-04T19:42:32.497 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:42:32.497+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:42:32.497572+0000) 2024-03-04T19:42:33.497 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:42:33.496+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:42:33.497759+0000) 2024-03-04T19:42:34.497 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:42:34.497+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:42:34.498004+0000) 2024-03-04T19:42:35.497 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:42:35.497+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:42:35.498231+0000) 2024-03-04T19:42:36.498 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:42:36.498+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:42:36.498491+0000) 2024-03-04T19:42:37.498 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:42:37.498+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:42:37.498765+0000) 2024-03-04T19:42:38.498 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:42:38.498+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:42:38.499018+0000) 2024-03-04T19:42:39.498 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:42:39.498+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:42:39.499269+0000) 2024-03-04T19:42:40.499 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:42:40.499+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:42:40.499524+0000) 2024-03-04T19:42:41.499 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:42:41.499+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:42:41.499776+0000) 2024-03-04T19:42:42.499 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:42:42.499+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:42:42.500035+0000) 2024-03-04T19:42:43.500 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:42:43.500+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:42:43.500291+0000) 2024-03-04T19:42:44.500 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:42:44.500+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:42:44.500564+0000) 2024-03-04T19:42:45.500 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:42:45.500+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:42:45.500817+0000) 2024-03-04T19:42:46.500 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:42:46.500+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:42:46.501108+0000) 2024-03-04T19:42:47.501 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:42:47.501+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:42:47.501365+0000) 2024-03-04T19:42:48.501 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:42:48.501+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:42:48.501616+0000) 2024-03-04T19:42:49.501 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:42:49.500+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:42:49.501923+0000) 2024-03-04T19:42:50.502 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:42:50.501+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:42:50.502169+0000) 2024-03-04T19:42:51.502 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:42:51.501+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:42:51.502429+0000) 2024-03-04T19:42:52.502 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:42:52.501+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:42:52.502686+0000) 2024-03-04T19:42:53.502 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:42:53.501+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:42:53.502943+0000) 2024-03-04T19:42:54.502 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:42:54.502+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:42:54.503206+0000) 2024-03-04T19:42:55.503 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:42:55.502+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:42:55.503455+0000) 2024-03-04T19:42:56.503 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:42:56.502+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:42:56.503707+0000) 2024-03-04T19:42:57.503 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:42:57.502+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:42:57.503965+0000) 2024-03-04T19:42:58.504 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:42:58.503+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:42:58.504241+0000) 2024-03-04T19:42:59.052 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:42:59.060 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:42:59.504 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:42:59.503+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:42:59.504515+0000) 2024-03-04T19:43:00.504 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:43:00.503+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:43:00.504762+0000) 2024-03-04T19:43:01.504 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:43:01.504+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:43:01.505014+0000) 2024-03-04T19:43:02.504 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:43:02.504+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:43:02.505271+0000) 2024-03-04T19:43:03.505 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:43:03.504+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:43:03.505528+0000) 2024-03-04T19:43:04.505 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:43:04.504+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:43:04.505802+0000) 2024-03-04T19:43:05.505 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:43:05.505+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:43:05.506115+0000) 2024-03-04T19:43:06.506 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:43:06.505+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:43:06.506404+0000) 2024-03-04T19:43:07.506 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:43:07.505+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:43:07.506698+0000) 2024-03-04T19:43:08.506 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:43:08.506+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:43:08.507000+0000) 2024-03-04T19:43:09.507 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:43:09.506+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:43:09.507292+0000) 2024-03-04T19:43:10.507 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:43:10.506+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:43:10.507567+0000) 2024-03-04T19:43:11.507 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:43:11.507+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:43:11.507882+0000) 2024-03-04T19:43:12.507 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:43:12.507+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:43:12.508175+0000) 2024-03-04T19:43:13.508 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:43:13.507+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:43:13.508516+0000) 2024-03-04T19:43:14.508 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:43:14.507+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:43:14.508778+0000) 2024-03-04T19:43:15.508 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:43:15.508+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:43:15.509011+0000) 2024-03-04T19:43:16.509 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:43:16.508+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:43:16.509284+0000) 2024-03-04T19:43:17.509 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:43:17.508+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:43:17.509569+0000) 2024-03-04T19:43:18.509 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:43:18.509+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:43:18.509897+0000) 2024-03-04T19:43:19.509 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:43:19.509+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:43:19.510190+0000) 2024-03-04T19:43:20.510 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:43:20.509+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:43:20.510436+0000) 2024-03-04T19:43:21.510 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:43:21.510+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:43:21.510690+0000) 2024-03-04T19:43:22.510 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:43:22.510+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:43:22.510952+0000) 2024-03-04T19:43:23.510 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:43:23.510+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:43:23.511183+0000) 2024-03-04T19:43:24.511 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:43:24.510+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:43:24.511432+0000) 2024-03-04T19:43:25.511 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:43:25.511+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:43:25.511686+0000) 2024-03-04T19:43:26.511 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:43:26.511+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:43:26.511944+0000) 2024-03-04T19:43:27.511 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:43:27.511+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:43:27.512190+0000) 2024-03-04T19:43:28.512 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:43:28.511+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:43:28.512437+0000) 2024-03-04T19:43:29.315 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:43:29.322 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:43:29.512 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:43:29.512+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:43:29.512690+0000) 2024-03-04T19:43:30.512 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:43:30.512+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:43:30.512955+0000) 2024-03-04T19:43:31.513 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:43:31.512+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:43:31.513251+0000) 2024-03-04T19:43:32.513 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:43:32.513+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:43:32.513535+0000) 2024-03-04T19:43:33.513 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:43:33.513+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:43:33.513870+0000) 2024-03-04T19:43:34.513 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:43:34.513+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:43:34.514193+0000) 2024-03-04T19:43:35.514 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:43:35.513+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:43:35.514503+0000) 2024-03-04T19:43:36.514 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:43:36.514+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:43:36.514792+0000) 2024-03-04T19:43:37.515 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:43:37.514+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:43:37.515094+0000) 2024-03-04T19:43:38.515 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:43:38.514+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:43:38.515330+0000) 2024-03-04T19:43:39.515 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:43:39.515+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:43:39.515581+0000) 2024-03-04T19:43:40.515 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:43:40.515+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:43:40.515817+0000) 2024-03-04T19:43:41.515 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:43:41.515+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:43:41.516091+0000) 2024-03-04T19:43:42.516 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:43:42.516+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:43:42.516335+0000) 2024-03-04T19:43:43.516 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:43:43.516+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:43:43.516578+0000) 2024-03-04T19:43:44.516 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:43:44.516+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:43:44.516852+0000) 2024-03-04T19:43:45.516 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:43:45.516+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:43:45.517096+0000) 2024-03-04T19:43:46.517 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:43:46.516+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:43:46.517477+0000) 2024-03-04T19:43:47.517 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:43:47.517+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:43:47.517734+0000) 2024-03-04T19:43:48.517 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:43:48.517+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:43:48.517975+0000) 2024-03-04T19:43:49.518 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:43:49.518+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:43:49.518210+0000) 2024-03-04T19:43:50.518 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:43:50.518+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:43:50.518447+0000) 2024-03-04T19:43:51.518 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:43:51.518+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:43:51.518691+0000) 2024-03-04T19:43:52.518 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:43:52.517+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:43:52.518960+0000) 2024-03-04T19:43:53.519 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:43:53.518+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:43:53.519253+0000) 2024-03-04T19:43:54.519 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:43:54.517+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:43:54.519508+0000) 2024-03-04T19:43:55.519 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:43:55.517+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:43:55.519787+0000) 2024-03-04T19:43:56.519 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:43:56.518+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:43:56.520121+0000) 2024-03-04T19:43:57.520 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:43:57.518+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:43:57.520401+0000) 2024-03-04T19:43:58.520 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:43:58.519+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:43:58.520683+0000) 2024-03-04T19:43:59.462 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:43:59.470 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:43:59.520 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:43:59.519+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:43:59.520920+0000) 2024-03-04T19:44:00.520 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:44:00.520+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:44:00.521195+0000) 2024-03-04T19:44:01.521 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:44:01.520+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:44:01.521477+0000) 2024-03-04T19:44:02.521 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:44:02.519+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:44:02.521703+0000) 2024-03-04T19:44:03.521 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:44:03.521+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:44:03.521997+0000) 2024-03-04T19:44:04.522 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:44:04.521+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:44:04.522305+0000) 2024-03-04T19:44:05.522 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:44:05.521+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:44:05.522619+0000) 2024-03-04T19:44:06.522 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:44:06.522+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:44:06.522944+0000) 2024-03-04T19:44:07.523 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:44:07.522+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:44:07.523271+0000) 2024-03-04T19:44:08.523 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:44:08.522+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:44:08.523560+0000) 2024-03-04T19:44:09.523 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:44:09.523+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:44:09.523890+0000) 2024-03-04T19:44:10.523 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:44:10.523+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:44:10.524150+0000) 2024-03-04T19:44:11.524 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:44:11.523+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:44:11.524436+0000) 2024-03-04T19:44:12.524 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:44:12.523+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:44:12.524731+0000) 2024-03-04T19:44:13.524 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:44:13.524+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:44:13.525014+0000) 2024-03-04T19:44:14.525 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:44:14.524+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:44:14.525321+0000) 2024-03-04T19:44:15.525 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:44:15.524+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:44:15.525606+0000) 2024-03-04T19:44:16.525 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:44:16.525+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:44:16.525944+0000) 2024-03-04T19:44:17.526 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:44:17.525+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:44:17.526255+0000) 2024-03-04T19:44:18.526 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:44:18.525+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:44:18.526533+0000) 2024-03-04T19:44:19.526 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:44:19.526+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:44:19.526898+0000) 2024-03-04T19:44:20.526 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:44:20.526+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:44:20.527214+0000) 2024-03-04T19:44:21.527 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:44:21.526+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:44:21.527504+0000) 2024-03-04T19:44:22.527 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:44:22.527+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:44:22.527756+0000) 2024-03-04T19:44:23.527 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:44:23.527+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:44:23.528001+0000) 2024-03-04T19:44:24.528 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:44:24.527+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:44:24.528245+0000) 2024-03-04T19:44:25.528 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:44:25.527+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:44:25.528479+0000) 2024-03-04T19:44:26.528 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:44:26.528+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:44:26.528723+0000) 2024-03-04T19:44:27.528 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:44:27.528+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:44:27.528962+0000) 2024-03-04T19:44:28.529 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:44:28.528+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:44:28.529212+0000) 2024-03-04T19:44:29.529 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:44:29.528+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:44:29.529451+0000) 2024-03-04T19:44:29.658 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:44:29.666 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:44:30.529 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:44:30.529+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:44:30.529701+0000) 2024-03-04T19:44:31.529 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:44:31.529+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:44:31.529955+0000) 2024-03-04T19:44:32.529 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:44:32.529+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:44:32.530210+0000) 2024-03-04T19:44:33.530 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:44:33.530+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:44:33.530471+0000) 2024-03-04T19:44:34.530 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:44:34.530+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:44:34.530737+0000) 2024-03-04T19:44:35.530 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:44:35.530+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:44:35.530989+0000) 2024-03-04T19:44:36.531 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:44:36.530+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:44:36.531239+0000) 2024-03-04T19:44:37.531 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:44:37.531+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:44:37.531501+0000) 2024-03-04T19:44:38.531 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:44:38.530+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:44:38.531773+0000) 2024-03-04T19:44:39.531 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:44:39.531+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:44:39.532036+0000) 2024-03-04T19:44:40.531 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:44:40.531+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:44:40.532230+0000) 2024-03-04T19:44:41.532 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:44:41.532+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:44:41.532483+0000) 2024-03-04T19:44:42.532 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:44:42.532+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:44:42.532744+0000) 2024-03-04T19:44:43.532 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:44:43.532+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:44:43.533019+0000) 2024-03-04T19:44:44.532 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:44:44.533+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:44:44.533287+0000) 2024-03-04T19:44:45.533 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:44:45.533+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:44:45.533542+0000) 2024-03-04T19:44:46.533 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:44:46.533+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:44:46.533812+0000) 2024-03-04T19:44:47.533 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:44:47.533+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:44:47.534136+0000) 2024-03-04T19:44:48.534 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:44:48.534+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:44:48.534393+0000) 2024-03-04T19:44:49.534 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:44:49.533+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:44:49.534746+0000) 2024-03-04T19:44:50.534 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:44:50.534+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:44:50.535032+0000) 2024-03-04T19:44:51.535 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:44:51.535+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:44:51.535293+0000) 2024-03-04T19:44:52.535 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:44:52.535+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:44:52.535538+0000) 2024-03-04T19:44:53.535 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:44:53.535+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:44:53.535784+0000) 2024-03-04T19:44:54.535 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:44:54.534+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:44:54.536076+0000) 2024-03-04T19:44:55.536 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:44:55.535+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:44:55.536316+0000) 2024-03-04T19:44:56.536 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:44:56.535+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:44:56.536557+0000) 2024-03-04T19:44:57.536 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:44:57.535+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:44:57.536795+0000) 2024-03-04T19:44:58.536 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:44:58.536+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:44:58.537108+0000) 2024-03-04T19:44:59.537 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:44:59.536+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:44:59.537389+0000) 2024-03-04T19:44:59.846 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:44:59.854 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:45:00.537 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:45:00.536+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:45:00.537705+0000) 2024-03-04T19:45:01.537 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:45:01.536+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:45:01.537944+0000) 2024-03-04T19:45:02.537 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:45:02.536+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:45:02.538166+0000) 2024-03-04T19:45:03.538 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:45:03.537+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:45:03.538426+0000) 2024-03-04T19:45:04.538 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:45:04.537+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:45:04.538692+0000) 2024-03-04T19:45:05.538 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:45:05.538+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:45:05.538956+0000) 2024-03-04T19:45:06.539 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:45:06.538+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:45:06.539209+0000) 2024-03-04T19:45:07.539 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:45:07.538+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:45:07.539461+0000) 2024-03-04T19:45:08.539 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:45:08.538+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:45:08.539748+0000) 2024-03-04T19:45:09.539 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:45:09.539+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:45:09.539961+0000) 2024-03-04T19:45:10.539 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:45:10.539+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:45:10.540170+0000) 2024-03-04T19:45:11.540 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:45:11.539+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:45:11.540428+0000) 2024-03-04T19:45:12.540 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:45:12.539+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:45:12.540681+0000) 2024-03-04T19:45:13.540 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:45:13.540+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:45:13.540972+0000) 2024-03-04T19:45:14.541 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:45:14.540+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:45:14.541244+0000) 2024-03-04T19:45:15.541 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:45:15.540+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:45:15.541446+0000) 2024-03-04T19:45:16.541 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:45:16.540+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:45:16.541704+0000) 2024-03-04T19:45:17.541 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:45:17.541+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:45:17.541955+0000) 2024-03-04T19:45:18.541 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:45:18.541+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:45:18.542164+0000) 2024-03-04T19:45:19.542 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:45:19.541+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:45:19.542437+0000) 2024-03-04T19:45:20.542 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:45:20.541+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:45:20.542698+0000) 2024-03-04T19:45:21.542 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:45:21.542+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:45:21.542949+0000) 2024-03-04T19:45:22.543 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:45:22.542+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:45:22.543207+0000) 2024-03-04T19:45:23.543 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:45:23.542+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:45:23.543445+0000) 2024-03-04T19:45:24.543 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:45:24.543+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:45:24.543693+0000) 2024-03-04T19:45:25.543 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:45:25.543+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:45:25.543938+0000) 2024-03-04T19:45:26.543 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:45:26.543+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:45:26.544181+0000) 2024-03-04T19:45:27.544 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:45:27.543+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:45:27.544485+0000) 2024-03-04T19:45:28.544 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:45:28.544+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:45:28.544743+0000) 2024-03-04T19:45:29.544 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:45:29.544+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:45:29.544989+0000) 2024-03-04T19:45:30.043 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:45:30.052 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:45:30.544 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:45:30.544+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:45:30.545231+0000) 2024-03-04T19:45:31.545 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:45:31.544+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:45:31.545472+0000) 2024-03-04T19:45:32.545 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:45:32.545+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:45:32.545708+0000) 2024-03-04T19:45:33.545 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:45:33.545+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:45:33.545947+0000) 2024-03-04T19:45:34.545 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:45:34.545+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:45:34.546132+0000) 2024-03-04T19:45:35.546 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:45:35.545+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:45:35.546365+0000) 2024-03-04T19:45:36.546 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:45:36.546+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:45:36.546604+0000) 2024-03-04T19:45:37.546 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:45:37.546+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:45:37.546900+0000) 2024-03-04T19:45:38.546 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:45:38.546+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:45:38.547130+0000) 2024-03-04T19:45:39.547 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:45:39.546+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:45:39.547367+0000) 2024-03-04T19:45:40.547 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:45:40.546+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:45:40.547543+0000) 2024-03-04T19:45:41.547 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:45:41.547+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:45:41.547778+0000) 2024-03-04T19:45:42.547 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:45:42.547+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:45:42.548011+0000) 2024-03-04T19:45:43.548 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:45:43.547+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:45:43.548251+0000) 2024-03-04T19:45:44.548 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:45:44.548+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:45:44.548505+0000) 2024-03-04T19:45:45.548 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:45:45.548+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:45:45.548740+0000) 2024-03-04T19:45:46.548 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:45:46.548+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:45:46.548977+0000) 2024-03-04T19:45:47.549 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:45:47.548+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:45:47.549166+0000) 2024-03-04T19:45:48.549 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:45:48.549+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:45:48.549466+0000) 2024-03-04T19:45:49.549 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:45:49.549+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:45:49.549724+0000) 2024-03-04T19:45:50.549 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:45:50.549+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:45:50.549957+0000) 2024-03-04T19:45:51.549 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:45:51.550+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:45:51.550219+0000) 2024-03-04T19:45:52.550 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:45:52.550+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:45:52.550457+0000) 2024-03-04T19:45:53.550 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:45:53.550+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:45:53.550698+0000) 2024-03-04T19:45:54.550 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:45:54.550+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:45:54.550945+0000) 2024-03-04T19:45:55.550 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:45:55.550+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:45:55.551135+0000) 2024-03-04T19:45:56.551 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:45:56.549+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:45:56.551363+0000) 2024-03-04T19:45:57.551 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:45:57.550+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:45:57.551621+0000) 2024-03-04T19:45:58.551 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:45:58.550+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:45:58.551903+0000) 2024-03-04T19:45:59.551 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:45:59.551+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:45:59.552144+0000) 2024-03-04T19:46:00.181 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:46:00.189 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:46:00.552 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:46:00.551+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:46:00.552328+0000) 2024-03-04T19:46:01.552 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:46:01.551+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:46:01.552571+0000) 2024-03-04T19:46:02.552 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:46:02.551+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:46:02.552855+0000) 2024-03-04T19:46:03.552 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:46:03.552+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:46:03.553095+0000) 2024-03-04T19:46:04.553 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:46:04.552+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:46:04.553341+0000) 2024-03-04T19:46:05.553 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:46:05.551+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:46:05.553590+0000) 2024-03-04T19:46:06.553 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:46:06.552+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:46:06.553921+0000) 2024-03-04T19:46:07.553 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:46:07.553+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:46:07.554211+0000) 2024-03-04T19:46:08.554 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:46:08.553+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:46:08.554509+0000) 2024-03-04T19:46:09.554 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:46:09.553+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:46:09.554785+0000) 2024-03-04T19:46:10.554 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:46:10.554+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:46:10.555123+0000) 2024-03-04T19:46:11.555 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:46:11.554+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:46:11.555423+0000) 2024-03-04T19:46:12.555 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:46:12.554+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:46:12.555688+0000) 2024-03-04T19:46:13.555 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:46:13.555+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:46:13.555942+0000) 2024-03-04T19:46:14.555 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:46:14.555+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:46:14.556187+0000) 2024-03-04T19:46:15.556 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:46:15.555+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:46:15.556430+0000) 2024-03-04T19:46:16.556 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:46:16.555+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:46:16.556664+0000) 2024-03-04T19:46:17.556 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:46:17.556+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:46:17.556941+0000) 2024-03-04T19:46:18.559 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:46:18.556+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:46:18.557198+0000) 2024-03-04T19:46:19.557 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:46:19.556+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:46:19.557444+0000) 2024-03-04T19:46:20.557 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:46:20.556+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:46:20.557708+0000) 2024-03-04T19:46:21.557 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:46:21.557+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:46:21.557964+0000) 2024-03-04T19:46:22.557 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:46:22.557+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:46:22.558200+0000) 2024-03-04T19:46:23.558 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:46:23.557+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:46:23.558483+0000) 2024-03-04T19:46:24.558 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:46:24.558+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:46:24.558750+0000) 2024-03-04T19:46:25.559 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:46:25.558+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:46:25.558988+0000) 2024-03-04T19:46:26.558 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:46:26.558+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:46:26.559226+0000) 2024-03-04T19:46:27.559 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:46:27.558+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:46:27.559465+0000) 2024-03-04T19:46:28.559 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:46:28.559+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:46:28.559705+0000) 2024-03-04T19:46:29.559 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:46:29.559+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:46:29.559948+0000) 2024-03-04T19:46:30.378 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:46:30.388 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:46:30.559 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:46:30.559+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:46:30.560200+0000) 2024-03-04T19:46:31.560 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:46:31.559+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:46:31.560488+0000) 2024-03-04T19:46:32.560 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:46:32.560+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:46:32.560723+0000) 2024-03-04T19:46:33.560 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:46:33.560+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:46:33.560964+0000) 2024-03-04T19:46:34.560 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:46:34.560+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:46:34.561171+0000) 2024-03-04T19:46:35.561 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:46:35.560+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:46:35.561409+0000) 2024-03-04T19:46:36.561 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:46:36.561+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:46:36.561648+0000) 2024-03-04T19:46:37.561 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:46:37.561+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:46:37.561922+0000) 2024-03-04T19:46:38.562 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:46:38.561+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:46:38.562160+0000) 2024-03-04T19:46:39.562 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:46:39.561+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:46:39.562405+0000) 2024-03-04T19:46:40.562 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:46:40.562+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:46:40.562638+0000) 2024-03-04T19:46:41.562 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:46:41.562+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:46:41.562920+0000) 2024-03-04T19:46:42.562 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:46:42.562+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:46:42.563160+0000) 2024-03-04T19:46:43.563 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:46:43.563+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:46:43.563417+0000) 2024-03-04T19:46:44.563 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:46:44.563+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:46:44.563695+0000) 2024-03-04T19:46:45.563 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:46:45.563+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:46:45.563961+0000) 2024-03-04T19:46:46.564 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:46:46.563+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:46:46.564186+0000) 2024-03-04T19:46:47.564 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:46:47.564+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:46:47.564438+0000) 2024-03-04T19:46:48.564 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:46:48.564+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:46:48.564714+0000) 2024-03-04T19:46:49.564 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:46:49.564+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:46:49.564993+0000) 2024-03-04T19:46:50.564 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:46:50.564+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:46:50.565235+0000) 2024-03-04T19:46:51.565 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:46:51.565+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:46:51.565492+0000) 2024-03-04T19:46:52.565 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:46:52.564+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:46:52.565775+0000) 2024-03-04T19:46:53.565 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:46:53.565+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:46:53.566035+0000) 2024-03-04T19:46:54.566 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:46:54.566+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:46:54.566306+0000) 2024-03-04T19:46:55.566 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:46:55.566+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:46:55.566558+0000) 2024-03-04T19:46:56.566 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:46:56.566+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:46:56.566808+0000) 2024-03-04T19:46:57.566 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:46:57.566+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:46:57.567108+0000) 2024-03-04T19:46:58.567 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:46:58.565+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:46:58.567323+0000) 2024-03-04T19:46:59.567 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:46:59.566+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:46:59.567580+0000) 2024-03-04T19:47:00.511 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:47:00.520 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:47:00.567 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:47:00.565+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:47:00.567782+0000) 2024-03-04T19:47:01.567 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:47:01.566+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:47:01.568032+0000) 2024-03-04T19:47:02.567 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:47:02.567+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:47:02.568268+0000) 2024-03-04T19:47:03.568 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:47:03.567+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:47:03.568513+0000) 2024-03-04T19:47:04.568 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:47:04.567+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:47:04.568758+0000) 2024-03-04T19:47:05.568 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:47:05.567+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:47:05.568998+0000) 2024-03-04T19:47:06.569 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:47:06.567+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:47:06.569227+0000) 2024-03-04T19:47:07.569 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:47:07.568+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:47:07.569500+0000) 2024-03-04T19:47:08.569 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:47:08.568+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:47:08.569740+0000) 2024-03-04T19:47:09.569 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:47:09.569+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:47:09.569998+0000) 2024-03-04T19:47:10.570 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:47:10.569+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:47:10.570251+0000) 2024-03-04T19:47:11.570 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:47:11.569+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:47:11.570505+0000) 2024-03-04T19:47:12.570 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:47:12.569+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:47:12.570772+0000) 2024-03-04T19:47:13.570 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:47:13.570+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:47:13.571028+0000) 2024-03-04T19:47:14.571 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:47:14.570+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:47:14.571252+0000) 2024-03-04T19:47:15.571 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:47:15.570+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:47:15.571485+0000) 2024-03-04T19:47:16.571 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:47:16.570+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:47:16.571722+0000) 2024-03-04T19:47:17.571 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:47:17.571+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:47:17.571943+0000) 2024-03-04T19:47:18.572 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:47:18.571+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:47:18.572209+0000) 2024-03-04T19:47:19.572 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:47:19.571+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:47:19.572466+0000) 2024-03-04T19:47:20.572 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:47:20.571+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:47:20.572704+0000) 2024-03-04T19:47:21.572 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:47:21.572+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:47:21.572949+0000) 2024-03-04T19:47:22.573 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:47:22.572+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:47:22.573191+0000) 2024-03-04T19:47:23.573 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:47:23.572+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:47:23.573460+0000) 2024-03-04T19:47:24.573 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:47:24.572+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:47:24.573729+0000) 2024-03-04T19:47:25.573 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:47:25.573+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:47:25.573983+0000) 2024-03-04T19:47:26.573 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:47:26.573+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:47:26.574238+0000) 2024-03-04T19:47:27.574 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:47:27.573+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:47:27.574497+0000) 2024-03-04T19:47:28.574 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:47:28.573+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:47:28.574739+0000) 2024-03-04T19:47:29.574 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:47:29.574+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:47:29.574989+0000) 2024-03-04T19:47:30.574 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:47:30.574+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:47:30.575227+0000) 2024-03-04T19:47:30.713 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:47:30.721 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:47:31.575 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:47:31.574+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:47:31.575504+0000) 2024-03-04T19:47:32.575 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:47:32.575+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:47:32.575742+0000) 2024-03-04T19:47:33.575 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:47:33.575+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:47:33.575995+0000) 2024-03-04T19:47:34.576 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:47:34.575+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:47:34.576256+0000) 2024-03-04T19:47:35.576 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:47:35.575+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:47:35.576449+0000) 2024-03-04T19:47:36.576 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:47:36.576+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:47:36.576691+0000) 2024-03-04T19:47:37.576 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:47:37.576+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:47:37.576926+0000) 2024-03-04T19:47:38.576 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:47:38.576+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:47:38.577179+0000) 2024-03-04T19:47:39.577 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:47:39.576+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:47:39.577419+0000) 2024-03-04T19:47:40.577 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:47:40.577+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:47:40.577667+0000) 2024-03-04T19:47:41.577 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:47:41.577+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:47:41.577979+0000) 2024-03-04T19:47:42.577 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:47:42.577+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:47:42.578179+0000) 2024-03-04T19:47:43.578 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:47:43.578+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:47:43.578428+0000) 2024-03-04T19:47:44.578 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:47:44.578+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:47:44.578681+0000) 2024-03-04T19:47:45.578 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:47:45.578+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:47:45.578929+0000) 2024-03-04T19:47:46.579 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:47:46.578+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:47:46.579177+0000) 2024-03-04T19:47:47.579 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:47:47.579+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:47:47.579419+0000) 2024-03-04T19:47:48.579 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:47:48.579+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:47:48.579673+0000) 2024-03-04T19:47:49.580 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:47:49.579+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:47:49.579937+0000) 2024-03-04T19:47:50.580 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:47:50.579+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:47:50.580181+0000) 2024-03-04T19:47:51.580 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:47:51.580+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:47:51.580453+0000) 2024-03-04T19:47:52.580 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:47:52.580+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:47:52.580700+0000) 2024-03-04T19:47:53.580 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:47:53.580+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:47:53.580946+0000) 2024-03-04T19:47:54.580 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:47:54.580+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:47:54.581200+0000) 2024-03-04T19:47:55.581 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:47:55.581+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:47:55.581439+0000) 2024-03-04T19:47:56.581 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:47:56.580+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:47:56.581676+0000) 2024-03-04T19:47:57.581 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:47:57.581+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:47:57.581945+0000) 2024-03-04T19:47:58.581 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:47:58.582+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:47:58.582193+0000) 2024-03-04T19:47:59.582 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:47:59.582+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:47:59.582440+0000) 2024-03-04T19:48:00.582 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:48:00.582+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:48:00.582682+0000) 2024-03-04T19:48:00.909 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:48:00.917 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:48:01.582 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:48:01.581+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:48:01.582964+0000) 2024-03-04T19:48:02.582 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:48:02.582+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:48:02.583207+0000) 2024-03-04T19:48:03.583 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:48:03.582+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:48:03.583455+0000) 2024-03-04T19:48:04.583 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:48:04.582+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:48:04.583712+0000) 2024-03-04T19:48:05.583 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:48:05.582+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:48:05.583969+0000) 2024-03-04T19:48:06.584 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:48:06.583+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:48:06.584212+0000) 2024-03-04T19:48:07.584 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:48:07.583+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:48:07.584451+0000) 2024-03-04T19:48:08.585 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:48:08.583+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:48:08.584700+0000) 2024-03-04T19:48:09.587 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:48:09.583+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:48:09.584948+0000) 2024-03-04T19:48:10.585 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:48:10.584+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:48:10.585224+0000) 2024-03-04T19:48:11.585 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:48:11.583+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:48:11.585490+0000) 2024-03-04T19:48:12.585 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:48:12.584+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:48:12.585724+0000) 2024-03-04T19:48:13.585 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:48:13.585+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:48:13.585981+0000) 2024-03-04T19:48:14.586 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:48:14.585+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:48:14.586242+0000) 2024-03-04T19:48:15.586 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:48:15.585+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:48:15.586483+0000) 2024-03-04T19:48:16.586 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:48:16.585+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:48:16.586719+0000) 2024-03-04T19:48:17.586 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:48:17.586+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:48:17.587035+0000) 2024-03-04T19:48:18.587 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:48:18.586+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:48:18.587310+0000) 2024-03-04T19:48:19.587 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:48:19.586+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:48:19.587581+0000) 2024-03-04T19:48:20.587 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:48:20.587+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:48:20.587858+0000) 2024-03-04T19:48:21.587 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:48:21.587+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:48:21.588098+0000) 2024-03-04T19:48:22.588 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:48:22.587+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:48:22.588335+0000) 2024-03-04T19:48:23.588 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:48:23.587+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:48:23.588592+0000) 2024-03-04T19:48:24.588 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:48:24.588+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:48:24.588903+0000) 2024-03-04T19:48:25.588 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:48:25.588+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:48:25.589145+0000) 2024-03-04T19:48:26.589 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:48:26.588+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:48:26.589393+0000) 2024-03-04T19:48:27.589 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:48:27.588+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:48:27.589632+0000) 2024-03-04T19:48:28.589 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:48:28.589+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:48:28.589928+0000) 2024-03-04T19:48:29.589 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:48:29.589+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:48:29.590182+0000) 2024-03-04T19:48:30.590 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:48:30.589+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:48:30.590416+0000) 2024-03-04T19:48:31.132 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:48:31.140 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:48:31.590 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:48:31.590+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:48:31.590667+0000) 2024-03-04T19:48:32.590 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:48:32.590+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:48:32.590939+0000) 2024-03-04T19:48:33.590 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:48:33.590+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:48:33.591196+0000) 2024-03-04T19:48:34.591 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:48:34.590+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:48:34.591457+0000) 2024-03-04T19:48:35.591 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:48:35.591+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:48:35.591697+0000) 2024-03-04T19:48:36.591 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:48:36.591+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:48:36.591939+0000) 2024-03-04T19:48:37.592 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:48:37.591+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:48:37.592176+0000) 2024-03-04T19:48:38.592 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:48:38.591+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:48:38.592425+0000) 2024-03-04T19:48:39.592 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:48:39.592+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:48:39.592678+0000) 2024-03-04T19:48:40.592 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:48:40.592+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:48:40.592983+0000) 2024-03-04T19:48:41.593 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:48:41.592+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:48:41.593234+0000) 2024-03-04T19:48:42.593 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:48:42.593+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:48:42.593477+0000) 2024-03-04T19:48:43.593 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:48:43.592+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:48:43.593719+0000) 2024-03-04T19:48:44.593 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:48:44.593+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:48:44.594001+0000) 2024-03-04T19:48:45.594 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:48:45.593+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:48:45.594235+0000) 2024-03-04T19:48:46.594 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:48:46.594+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:48:46.594468+0000) 2024-03-04T19:48:47.594 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:48:47.594+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:48:47.594707+0000) 2024-03-04T19:48:48.595 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:48:48.594+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:48:48.594977+0000) 2024-03-04T19:48:49.595 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:48:49.594+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:48:49.595246+0000) 2024-03-04T19:48:50.595 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:48:50.595+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:48:50.595486+0000) 2024-03-04T19:48:51.595 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:48:51.595+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:48:51.595729+0000) 2024-03-04T19:48:52.595 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:48:52.595+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:48:52.595981+0000) 2024-03-04T19:48:53.596 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:48:53.595+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:48:53.596250+0000) 2024-03-04T19:48:54.596 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:48:54.596+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:48:54.596533+0000) 2024-03-04T19:48:55.596 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:48:55.596+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:48:55.596795+0000) 2024-03-04T19:48:56.596 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:48:56.596+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:48:56.597098+0000) 2024-03-04T19:48:57.597 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:48:57.597+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:48:57.597378+0000) 2024-03-04T19:48:58.597 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:48:58.597+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:48:58.597645+0000) 2024-03-04T19:48:59.597 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:48:59.597+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:48:59.597929+0000) 2024-03-04T19:49:00.598 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:49:00.598+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:49:00.598184+0000) 2024-03-04T19:49:01.277 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:49:01.285 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:49:01.597 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:49:01.597+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:49:01.598427+0000) 2024-03-04T19:49:02.598 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:49:02.598+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:49:02.598673+0000) 2024-03-04T19:49:03.598 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:49:03.597+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:49:03.598967+0000) 2024-03-04T19:49:04.599 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:49:04.598+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:49:04.599230+0000) 2024-03-04T19:49:05.599 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:49:05.598+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:49:05.599490+0000) 2024-03-04T19:49:06.599 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:49:06.598+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:49:06.599752+0000) 2024-03-04T19:49:07.599 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:49:07.598+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:49:07.600019+0000) 2024-03-04T19:49:08.600 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:49:08.599+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:49:08.600287+0000) 2024-03-04T19:49:09.600 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:49:09.599+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:49:09.600533+0000) 2024-03-04T19:49:10.600 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:49:10.599+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:49:10.600773+0000) 2024-03-04T19:49:11.600 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:49:11.599+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:49:11.601089+0000) 2024-03-04T19:49:12.601 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:49:12.600+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:49:12.601343+0000) 2024-03-04T19:49:13.601 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:49:13.600+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:49:13.601609+0000) 2024-03-04T19:49:14.601 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:49:14.601+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:49:14.601940+0000) 2024-03-04T19:49:15.601 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:49:15.601+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:49:15.602178+0000) 2024-03-04T19:49:16.602 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:49:16.601+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:49:16.602455+0000) 2024-03-04T19:49:17.602 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:49:17.601+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:49:17.602729+0000) 2024-03-04T19:49:18.602 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:49:18.602+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:49:18.602999+0000) 2024-03-04T19:49:19.603 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:49:19.602+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:49:19.603271+0000) 2024-03-04T19:49:20.603 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:49:20.602+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:49:20.603569+0000) 2024-03-04T19:49:21.603 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:49:21.603+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:49:21.603865+0000) 2024-03-04T19:49:22.603 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:49:22.603+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:49:22.604120+0000) 2024-03-04T19:49:23.604 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:49:23.603+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:49:23.604393+0000) 2024-03-04T19:49:24.604 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:49:24.603+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:49:24.604756+0000) 2024-03-04T19:49:25.604 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:49:25.604+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:49:25.605015+0000) 2024-03-04T19:49:26.605 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:49:26.604+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:49:26.605251+0000) 2024-03-04T19:49:27.605 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:49:27.604+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:49:27.605493+0000) 2024-03-04T19:49:28.607 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:49:28.604+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:49:28.605777+0000) 2024-03-04T19:49:29.605 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:49:29.605+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:49:29.606045+0000) 2024-03-04T19:49:30.606 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:49:30.605+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:49:30.606282+0000) 2024-03-04T19:49:31.424 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:49:31.433 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:49:31.606 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:49:31.605+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:49:31.606608+0000) 2024-03-04T19:49:32.606 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:49:32.606+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:49:32.606916+0000) 2024-03-04T19:49:33.607 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:49:33.606+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:49:33.607159+0000) 2024-03-04T19:49:34.607 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:49:34.606+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:49:34.607420+0000) 2024-03-04T19:49:35.607 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:49:35.607+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:49:35.607655+0000) 2024-03-04T19:49:36.607 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:49:36.607+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:49:36.607929+0000) 2024-03-04T19:49:37.608 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:49:37.607+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:49:37.608170+0000) 2024-03-04T19:49:38.608 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:49:38.607+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:49:38.608417+0000) 2024-03-04T19:49:39.608 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:49:39.607+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:49:39.608673+0000) 2024-03-04T19:49:40.608 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:49:40.608+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:49:40.609006+0000) 2024-03-04T19:49:41.609 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:49:41.608+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:49:41.609187+0000) 2024-03-04T19:49:42.609 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:49:42.608+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:49:42.609425+0000) 2024-03-04T19:49:43.609 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:49:43.608+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:49:43.609671+0000) 2024-03-04T19:49:44.609 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:49:44.609+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:49:44.609924+0000) 2024-03-04T19:49:45.610 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:49:45.609+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:49:45.610187+0000) 2024-03-04T19:49:46.610 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:49:46.609+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:49:46.610415+0000) 2024-03-04T19:49:47.610 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:49:47.609+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:49:47.610669+0000) 2024-03-04T19:49:48.610 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:49:48.610+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:49:48.610969+0000) 2024-03-04T19:49:49.611 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:49:49.610+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:49:49.611223+0000) 2024-03-04T19:49:50.611 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:49:50.610+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:49:50.611473+0000) 2024-03-04T19:49:51.611 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:49:51.611+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:49:51.611715+0000) 2024-03-04T19:49:52.611 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:49:52.611+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:49:52.611950+0000) 2024-03-04T19:49:53.612 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:49:53.611+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:49:53.612236+0000) 2024-03-04T19:49:54.612 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:49:54.612+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:49:54.612492+0000) 2024-03-04T19:49:55.612 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:49:55.611+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:49:55.612722+0000) 2024-03-04T19:49:56.612 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:49:56.612+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:49:56.612980+0000) 2024-03-04T19:49:57.613 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:49:57.612+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:49:57.613237+0000) 2024-03-04T19:49:58.613 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:49:58.612+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:49:58.613467+0000) 2024-03-04T19:49:59.613 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:49:59.613+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:49:59.613718+0000) 2024-03-04T19:50:00.613 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:50:00.613+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:50:00.613952+0000) 2024-03-04T19:50:01.614 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:50:01.614+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:50:01.614192+0000) 2024-03-04T19:50:01.654 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:50:01.662 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:50:02.614 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:50:02.614+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:50:02.614428+0000) 2024-03-04T19:50:03.614 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:50:03.613+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:50:03.614619+0000) 2024-03-04T19:50:04.614 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:50:04.614+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:50:04.614926+0000) 2024-03-04T19:50:05.614 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:50:05.614+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:50:05.615160+0000) 2024-03-04T19:50:06.615 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:50:06.614+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:50:06.615398+0000) 2024-03-04T19:50:07.615 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:50:07.614+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:50:07.615640+0000) 2024-03-04T19:50:08.615 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:50:08.614+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:50:08.615939+0000) 2024-03-04T19:50:09.616 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:50:09.615+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:50:09.616189+0000) 2024-03-04T19:50:10.616 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:50:10.615+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:50:10.616426+0000) 2024-03-04T19:50:11.616 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:50:11.614+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:50:11.616612+0000) 2024-03-04T19:50:12.617 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:50:12.615+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:50:12.616892+0000) 2024-03-04T19:50:13.617 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:50:13.616+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:50:13.617137+0000) 2024-03-04T19:50:14.617 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:50:14.616+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:50:14.617385+0000) 2024-03-04T19:50:15.617 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:50:15.616+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:50:15.617643+0000) 2024-03-04T19:50:16.617 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:50:16.617+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:50:16.617924+0000) 2024-03-04T19:50:17.617 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:50:17.617+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:50:17.618156+0000) 2024-03-04T19:50:18.618 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:50:18.617+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:50:18.618394+0000) 2024-03-04T19:50:19.618 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:50:19.617+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:50:19.618594+0000) 2024-03-04T19:50:20.618 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:50:20.618+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:50:20.618868+0000) 2024-03-04T19:50:21.618 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:50:21.618+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:50:21.619108+0000) 2024-03-04T19:50:22.619 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:50:22.618+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:50:22.619340+0000) 2024-03-04T19:50:23.619 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:50:23.618+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:50:23.619520+0000) 2024-03-04T19:50:24.619 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:50:24.618+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:50:24.619773+0000) 2024-03-04T19:50:25.619 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:50:25.619+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:50:25.620032+0000) 2024-03-04T19:50:26.620 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:50:26.619+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:50:26.620269+0000) 2024-03-04T19:50:27.620 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:50:27.619+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:50:27.620527+0000) 2024-03-04T19:50:28.620 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:50:28.620+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:50:28.620771+0000) 2024-03-04T19:50:29.620 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:50:29.620+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:50:29.621010+0000) 2024-03-04T19:50:30.622 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:50:30.620+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:50:30.621247+0000) 2024-03-04T19:50:31.621 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:50:31.620+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:50:31.621505+0000) 2024-03-04T19:50:31.868 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:50:31.880 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:50:32.621 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:50:32.621+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:50:32.621750+0000) 2024-03-04T19:50:33.621 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:50:33.621+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:50:33.621991+0000) 2024-03-04T19:50:34.622 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:50:34.621+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:50:34.622239+0000) 2024-03-04T19:50:35.622 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:50:35.621+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:50:35.622505+0000) 2024-03-04T19:50:36.622 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:50:36.621+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:50:36.622767+0000) 2024-03-04T19:50:37.622 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:50:37.622+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:50:37.623005+0000) 2024-03-04T19:50:38.623 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:50:38.622+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:50:38.623271+0000) 2024-03-04T19:50:39.623 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:50:39.622+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:50:39.623522+0000) 2024-03-04T19:50:40.623 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:50:40.623+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:50:40.623762+0000) 2024-03-04T19:50:41.623 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:50:41.623+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:50:41.624057+0000) 2024-03-04T19:50:42.624 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:50:42.623+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:50:42.624301+0000) 2024-03-04T19:50:43.624 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:50:43.624+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:50:43.624547+0000) 2024-03-04T19:50:44.624 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:50:44.624+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:50:44.624800+0000) 2024-03-04T19:50:45.624 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:50:45.624+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:50:45.625029+0000) 2024-03-04T19:50:46.625 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:50:46.624+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:50:46.625291+0000) 2024-03-04T19:50:47.625 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:50:47.625+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:50:47.625553+0000) 2024-03-04T19:50:48.625 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:50:48.625+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:50:48.625795+0000) 2024-03-04T19:50:49.625 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:50:49.625+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:50:49.626068+0000) 2024-03-04T19:50:50.626 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:50:50.625+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:50:50.626316+0000) 2024-03-04T19:50:51.626 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:50:51.626+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:50:51.626560+0000) 2024-03-04T19:50:52.626 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:50:52.626+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:50:52.626811+0000) 2024-03-04T19:50:53.626 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:50:53.626+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:50:53.627093+0000) 2024-03-04T19:50:54.627 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:50:54.627+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:50:54.627352+0000) 2024-03-04T19:50:55.627 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:50:55.627+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:50:55.627598+0000) 2024-03-04T19:50:56.627 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:50:56.627+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:50:56.627858+0000) 2024-03-04T19:50:57.627 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:50:57.627+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:50:57.628097+0000) 2024-03-04T19:50:58.628 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:50:58.628+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:50:58.628340+0000) 2024-03-04T19:50:59.628 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:50:59.627+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:50:59.628608+0000) 2024-03-04T19:51:00.628 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:51:00.628+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:51:00.628881+0000) 2024-03-04T19:51:01.629 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:51:01.628+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:51:01.629117+0000) 2024-03-04T19:51:02.081 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:51:02.089 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:51:02.629 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:51:02.629+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:51:02.629365+0000) 2024-03-04T19:51:03.629 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:51:03.629+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:51:03.629608+0000) 2024-03-04T19:51:04.629 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:51:04.629+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:51:04.629915+0000) 2024-03-04T19:51:05.630 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:51:05.630+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:51:05.630157+0000) 2024-03-04T19:51:06.630 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:51:06.630+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:51:06.630438+0000) 2024-03-04T19:51:07.631 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:51:07.630+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:51:07.630678+0000) 2024-03-04T19:51:08.630 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:51:08.629+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:51:08.630948+0000) 2024-03-04T19:51:09.631 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:51:09.630+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:51:09.631187+0000) 2024-03-04T19:51:10.631 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:51:10.630+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:51:10.631469+0000) 2024-03-04T19:51:11.631 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:51:11.630+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:51:11.631752+0000) 2024-03-04T19:51:12.631 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:51:12.630+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:51:12.631969+0000) 2024-03-04T19:51:13.632 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:51:13.631+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:51:13.632249+0000) 2024-03-04T19:51:14.632 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:51:14.631+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:51:14.632550+0000) 2024-03-04T19:51:15.632 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:51:15.631+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:51:15.632907+0000) 2024-03-04T19:51:16.633 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:51:16.632+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:51:16.633202+0000) 2024-03-04T19:51:17.633 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:51:17.632+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:51:17.633500+0000) 2024-03-04T19:51:18.633 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:51:18.632+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:51:18.633796+0000) 2024-03-04T19:51:19.634 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:51:19.633+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:51:19.634127+0000) 2024-03-04T19:51:20.634 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:51:20.633+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:51:20.634418+0000) 2024-03-04T19:51:21.634 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:51:21.633+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:51:21.634709+0000) 2024-03-04T19:51:22.634 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:51:22.634+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:51:22.635025+0000) 2024-03-04T19:51:23.635 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:51:23.634+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:51:23.635313+0000) 2024-03-04T19:51:24.635 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:51:24.634+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:51:24.635624+0000) 2024-03-04T19:51:25.635 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:51:25.635+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:51:25.635967+0000) 2024-03-04T19:51:26.636 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:51:26.635+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:51:26.636260+0000) 2024-03-04T19:51:27.636 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:51:27.635+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:51:27.636555+0000) 2024-03-04T19:51:28.636 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:51:28.636+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:51:28.636898+0000) 2024-03-04T19:51:29.637 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:51:29.636+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:51:29.637188+0000) 2024-03-04T19:51:30.637 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:51:30.636+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:51:30.637478+0000) 2024-03-04T19:51:31.637 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:51:31.636+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:51:31.637774+0000) 2024-03-04T19:51:32.277 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:51:32.285 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:51:32.638 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:51:32.637+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:51:32.638083+0000) 2024-03-04T19:51:33.638 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:51:33.637+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:51:33.638373+0000) 2024-03-04T19:51:34.638 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:51:34.637+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:51:34.638701+0000) 2024-03-04T19:51:35.638 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:51:35.638+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:51:35.638995+0000) 2024-03-04T19:51:36.639 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:51:36.638+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:51:36.639290+0000) 2024-03-04T19:51:37.639 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:51:37.638+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:51:37.639594+0000) 2024-03-04T19:51:38.639 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:51:38.639+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:51:38.639930+0000) 2024-03-04T19:51:39.640 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:51:39.639+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:51:39.640222+0000) 2024-03-04T19:51:40.640 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:51:40.639+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:51:40.640592+0000) 2024-03-04T19:51:41.640 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:51:41.640+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:51:41.640921+0000) 2024-03-04T19:51:42.641 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:51:42.640+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:51:42.641212+0000) 2024-03-04T19:51:43.641 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:51:43.640+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:51:43.641501+0000) 2024-03-04T19:51:44.641 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:51:44.641+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:51:44.641807+0000) 2024-03-04T19:51:45.641 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:51:45.641+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:51:45.642133+0000) 2024-03-04T19:51:46.642 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:51:46.641+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:51:46.642432+0000) 2024-03-04T19:51:47.642 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:51:47.642+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:51:47.642733+0000) 2024-03-04T19:51:48.642 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:51:48.642+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:51:48.643024+0000) 2024-03-04T19:51:49.643 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:51:49.642+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:51:49.643322+0000) 2024-03-04T19:51:50.643 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:51:50.643+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:51:50.643567+0000) 2024-03-04T19:51:51.643 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:51:51.643+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:51:51.643800+0000) 2024-03-04T19:51:52.643 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:51:52.643+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:51:52.644032+0000) 2024-03-04T19:51:53.644 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:51:53.643+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:51:53.644275+0000) 2024-03-04T19:51:54.644 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:51:54.644+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:51:54.644535+0000) 2024-03-04T19:51:55.644 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:51:55.643+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:51:55.644727+0000) 2024-03-04T19:51:56.644 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:51:56.644+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:51:56.644971+0000) 2024-03-04T19:51:57.645 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:51:57.644+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:51:57.645248+0000) 2024-03-04T19:51:58.645 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:51:58.644+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:51:58.645483+0000) 2024-03-04T19:51:59.645 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:51:59.645+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:51:59.645731+0000) 2024-03-04T19:52:00.648 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:52:00.645+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:52:00.646021+0000) 2024-03-04T19:52:01.647 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:52:01.645+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:52:01.646259+0000) 2024-03-04T19:52:02.416 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:52:02.424 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:52:02.646 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:52:02.645+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:52:02.646508+0000) 2024-03-04T19:52:03.646 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:52:03.646+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:52:03.646812+0000) 2024-03-04T19:52:04.647 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:52:04.646+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:52:04.647140+0000) 2024-03-04T19:52:05.647 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:52:05.646+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:52:05.647381+0000) 2024-03-04T19:52:06.647 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:52:06.647+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:52:06.647680+0000) 2024-03-04T19:52:07.647 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:52:07.647+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:52:07.647990+0000) 2024-03-04T19:52:08.648 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:52:08.647+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:52:08.648237+0000) 2024-03-04T19:52:09.648 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:52:09.648+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:52:09.648541+0000) 2024-03-04T19:52:10.648 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:52:10.647+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:52:10.648916+0000) 2024-03-04T19:52:11.649 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:52:11.647+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:52:11.649199+0000) 2024-03-04T19:52:12.649 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:52:12.648+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:52:12.649502+0000) 2024-03-04T19:52:13.649 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:52:13.648+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:52:13.649784+0000) 2024-03-04T19:52:14.650 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:52:14.649+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:52:14.650148+0000) 2024-03-04T19:52:15.650 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:52:15.649+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:52:15.650438+0000) 2024-03-04T19:52:16.650 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:52:16.649+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:52:16.650763+0000) 2024-03-04T19:52:17.651 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:52:17.650+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:52:17.651065+0000) 2024-03-04T19:52:18.651 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:52:18.650+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:52:18.651361+0000) 2024-03-04T19:52:19.651 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:52:19.650+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:52:19.651615+0000) 2024-03-04T19:52:20.651 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:52:20.650+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:52:20.651958+0000) 2024-03-04T19:52:21.652 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:52:21.651+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:52:21.652265+0000) 2024-03-04T19:52:22.652 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:52:22.651+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:52:22.652538+0000) 2024-03-04T19:52:23.652 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:52:23.652+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:52:23.652864+0000) 2024-03-04T19:52:24.653 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:52:24.652+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:52:24.653180+0000) 2024-03-04T19:52:25.653 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:52:25.652+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:52:25.653476+0000) 2024-03-04T19:52:26.653 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:52:26.652+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:52:26.653770+0000) 2024-03-04T19:52:27.654 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:52:27.653+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:52:27.654077+0000) 2024-03-04T19:52:28.654 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:52:28.653+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:52:28.654416+0000) 2024-03-04T19:52:29.654 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:52:29.653+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:52:29.654722+0000) 2024-03-04T19:52:30.654 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:52:30.654+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:52:30.655021+0000) 2024-03-04T19:52:31.655 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:52:31.654+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:52:31.655303+0000) 2024-03-04T19:52:32.612 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:52:32.624 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:52:32.655 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:52:32.654+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:52:32.655490+0000) 2024-03-04T19:52:33.655 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:52:33.654+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:52:33.655775+0000) 2024-03-04T19:52:34.656 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:52:34.655+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:52:34.656164+0000) 2024-03-04T19:52:35.656 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:52:35.655+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:52:35.656453+0000) 2024-03-04T19:52:36.656 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:52:36.656+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:52:36.656747+0000) 2024-03-04T19:52:37.656 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:52:37.656+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:52:37.657048+0000) 2024-03-04T19:52:38.657 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:52:38.656+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:52:38.657295+0000) 2024-03-04T19:52:39.657 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:52:39.656+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:52:39.657594+0000) 2024-03-04T19:52:40.657 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:52:40.657+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:52:40.657890+0000) 2024-03-04T19:52:41.658 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:52:41.657+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:52:41.658126+0000) 2024-03-04T19:52:42.658 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:52:42.657+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:52:42.658367+0000) 2024-03-04T19:52:43.658 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:52:43.658+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:52:43.658633+0000) 2024-03-04T19:52:44.658 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:52:44.658+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:52:44.658932+0000) 2024-03-04T19:52:45.659 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:52:45.658+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:52:45.659219+0000) 2024-03-04T19:52:46.659 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:52:46.658+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:52:46.659509+0000) 2024-03-04T19:52:47.659 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:52:47.659+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:52:47.659802+0000) 2024-03-04T19:52:48.660 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:52:48.659+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:52:48.660166+0000) 2024-03-04T19:52:49.663 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:52:49.659+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:52:49.660459+0000) 2024-03-04T19:52:50.660 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:52:50.660+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:52:50.660764+0000) 2024-03-04T19:52:51.660 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:52:51.660+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:52:51.661029+0000) 2024-03-04T19:52:52.661 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:52:52.660+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:52:52.661317+0000) 2024-03-04T19:52:53.662 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:52:53.661+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:52:53.661608+0000) 2024-03-04T19:52:54.662 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:52:54.661+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:52:54.661964+0000) 2024-03-04T19:52:55.663 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:52:55.661+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:52:55.662288+0000) 2024-03-04T19:52:56.662 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:52:56.662+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:52:56.662569+0000) 2024-03-04T19:52:57.662 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:52:57.662+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:52:57.662913+0000) 2024-03-04T19:52:58.663 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:52:58.662+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:52:58.663228+0000) 2024-03-04T19:52:59.663 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:52:59.662+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:52:59.663480+0000) 2024-03-04T19:53:00.665 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:53:00.662+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:53:00.663732+0000) 2024-03-04T19:53:01.665 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:53:01.663+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:53:01.663987+0000) 2024-03-04T19:53:02.665 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:53:02.663+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:53:02.664190+0000) 2024-03-04T19:53:02.784 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:53:02.793 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:53:03.665 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:53:03.663+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:53:03.664428+0000) 2024-03-04T19:53:04.665 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:53:04.664+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:53:04.664679+0000) 2024-03-04T19:53:05.665 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:53:05.664+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:53:05.664995+0000) 2024-03-04T19:53:06.665 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:53:06.665+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:53:06.665313+0000) 2024-03-04T19:53:07.665 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:53:07.665+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:53:07.665631+0000) 2024-03-04T19:53:08.665 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:53:08.665+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:53:08.665982+0000) 2024-03-04T19:53:09.666 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:53:09.666+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:53:09.666270+0000) 2024-03-04T19:53:10.666 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:53:10.666+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:53:10.666577+0000) 2024-03-04T19:53:11.667 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:53:11.666+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:53:11.666921+0000) 2024-03-04T19:53:12.667 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:53:12.667+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:53:12.667224+0000) 2024-03-04T19:53:13.667 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:53:13.666+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:53:13.667528+0000) 2024-03-04T19:53:14.667 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:53:14.666+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:53:14.667897+0000) 2024-03-04T19:53:15.668 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:53:15.667+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:53:15.668197+0000) 2024-03-04T19:53:16.668 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:53:16.667+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:53:16.668539+0000) 2024-03-04T19:53:17.668 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:53:17.667+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:53:17.668900+0000) 2024-03-04T19:53:18.669 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:53:18.668+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:53:18.669196+0000) 2024-03-04T19:53:19.669 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:53:19.668+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:53:19.669497+0000) 2024-03-04T19:53:20.669 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:53:20.668+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:53:20.669797+0000) 2024-03-04T19:53:21.670 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:53:21.669+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:53:21.670115+0000) 2024-03-04T19:53:22.670 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:53:22.669+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:53:22.670356+0000) 2024-03-04T19:53:23.670 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:53:23.669+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:53:23.670600+0000) 2024-03-04T19:53:24.671 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:53:24.670+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:53:24.670925+0000) 2024-03-04T19:53:25.671 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:53:25.670+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:53:25.671194+0000) 2024-03-04T19:53:26.671 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:53:26.670+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:53:26.671436+0000) 2024-03-04T19:53:27.671 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:53:27.670+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:53:27.671685+0000) 2024-03-04T19:53:28.672 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:53:28.671+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:53:28.671950+0000) 2024-03-04T19:53:29.672 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:53:29.671+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:53:29.672195+0000) 2024-03-04T19:53:30.672 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:53:30.671+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:53:30.672444+0000) 2024-03-04T19:53:31.672 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:53:31.671+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:53:31.672690+0000) 2024-03-04T19:53:32.673 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:53:32.672+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:53:32.672934+0000) 2024-03-04T19:53:33.005 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:53:33.014 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:53:33.673 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:53:33.672+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:53:33.673175+0000) 2024-03-04T19:53:34.673 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:53:34.672+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:53:34.673443+0000) 2024-03-04T19:53:35.673 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:53:35.672+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:53:35.673712+0000) 2024-03-04T19:53:36.673 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:53:36.673+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:53:36.673990+0000) 2024-03-04T19:53:37.673 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:53:37.673+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:53:37.674265+0000) 2024-03-04T19:53:38.674 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:53:38.673+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:53:38.674512+0000) 2024-03-04T19:53:39.674 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:53:39.674+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:53:39.674782+0000) 2024-03-04T19:53:40.675 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:53:40.674+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:53:40.675092+0000) 2024-03-04T19:53:41.675 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:53:41.674+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:53:41.675378+0000) 2024-03-04T19:53:42.675 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:53:42.674+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:53:42.675613+0000) 2024-03-04T19:53:43.675 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:53:43.675+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:53:43.675894+0000) 2024-03-04T19:53:44.676 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:53:44.675+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:53:44.676155+0000) 2024-03-04T19:53:45.676 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:53:45.675+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:53:45.676406+0000) 2024-03-04T19:53:46.676 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:53:46.676+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:53:46.676650+0000) 2024-03-04T19:53:47.676 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:53:47.676+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:53:47.676929+0000) 2024-03-04T19:53:48.677 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:53:48.676+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:53:48.677168+0000) 2024-03-04T19:53:49.677 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:53:49.676+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:53:49.677415+0000) 2024-03-04T19:53:50.677 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:53:50.677+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:53:50.677670+0000) 2024-03-04T19:53:51.677 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:53:51.677+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:53:51.677941+0000) 2024-03-04T19:53:52.678 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:53:52.677+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:53:52.678175+0000) 2024-03-04T19:53:53.678 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:53:53.677+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:53:53.678441+0000) 2024-03-04T19:53:54.678 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:53:54.678+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:53:54.678723+0000) 2024-03-04T19:53:55.678 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:53:55.678+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:53:55.678975+0000) 2024-03-04T19:53:56.679 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:53:56.678+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:53:56.679155+0000) 2024-03-04T19:53:57.679 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:53:57.678+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:53:57.679357+0000) 2024-03-04T19:53:58.679 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:53:58.679+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:53:58.679613+0000) 2024-03-04T19:53:59.679 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:53:59.679+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:53:59.679910+0000) 2024-03-04T19:54:00.680 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:54:00.679+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:54:00.680162+0000) 2024-03-04T19:54:01.680 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:54:01.680+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:54:01.680404+0000) 2024-03-04T19:54:02.680 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:54:02.680+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:54:02.680640+0000) 2024-03-04T19:54:03.210 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:54:03.218 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:54:03.680 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:54:03.680+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:54:03.680935+0000) 2024-03-04T19:54:04.681 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:54:04.680+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:54:04.681260+0000) 2024-03-04T19:54:05.681 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:54:05.681+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:54:05.681523+0000) 2024-03-04T19:54:06.681 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:54:06.681+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:54:06.681780+0000) 2024-03-04T19:54:07.681 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:54:07.681+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:54:07.682073+0000) 2024-03-04T19:54:08.682 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:54:08.682+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:54:08.682330+0000) 2024-03-04T19:54:09.682 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:54:09.682+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:54:09.682617+0000) 2024-03-04T19:54:10.682 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:54:10.682+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:54:10.682924+0000) 2024-03-04T19:54:11.682 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:54:11.682+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:54:11.683124+0000) 2024-03-04T19:54:12.683 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:54:12.683+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:54:12.683388+0000) 2024-03-04T19:54:13.683 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:54:13.683+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:54:13.683643+0000) 2024-03-04T19:54:14.683 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:54:14.683+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:54:14.683966+0000) 2024-03-04T19:54:15.683 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:54:15.683+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:54:15.684249+0000) 2024-03-04T19:54:16.683 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:54:16.683+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:54:16.684505+0000) 2024-03-04T19:54:17.684 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:54:17.683+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:54:17.684763+0000) 2024-03-04T19:54:18.684 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:54:18.683+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:54:18.685026+0000) 2024-03-04T19:54:19.684 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:54:19.684+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:54:19.685293+0000) 2024-03-04T19:54:20.684 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:54:20.684+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:54:20.685561+0000) 2024-03-04T19:54:21.684 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:54:21.684+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:54:21.685866+0000) 2024-03-04T19:54:22.685 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:54:22.685+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:54:22.686154+0000) 2024-03-04T19:54:23.685 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:54:23.685+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:54:23.686410+0000) 2024-03-04T19:54:24.686 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:54:24.685+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:54:24.686685+0000) 2024-03-04T19:54:25.686 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:54:25.686+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:54:25.686960+0000) 2024-03-04T19:54:26.687 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:54:26.686+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:54:26.687218+0000) 2024-03-04T19:54:27.686 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:54:27.686+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:54:27.687476+0000) 2024-03-04T19:54:28.686 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:54:28.686+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:54:28.687727+0000) 2024-03-04T19:54:29.686 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:54:29.687+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:54:29.687978+0000) 2024-03-04T19:54:30.686 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:54:30.687+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:54:30.688246+0000) 2024-03-04T19:54:31.687 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:54:31.687+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:54:31.688506+0000) 2024-03-04T19:54:32.687 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:54:32.687+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:54:32.688760+0000) 2024-03-04T19:54:33.340 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:54:33.349 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:54:33.687 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:54:33.688+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:54:33.689030+0000) 2024-03-04T19:54:34.688 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:54:34.688+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:54:34.689305+0000) 2024-03-04T19:54:35.688 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:54:35.688+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:54:35.689586+0000) 2024-03-04T19:54:36.688 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:54:36.689+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:54:36.689870+0000) 2024-03-04T19:54:37.688 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:54:37.689+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:54:37.690126+0000) 2024-03-04T19:54:38.688 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:54:38.689+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:54:38.690402+0000) 2024-03-04T19:54:39.689 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:54:39.689+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:54:39.690692+0000) 2024-03-04T19:54:40.689 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:54:40.690+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:54:40.690981+0000) 2024-03-04T19:54:41.689 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:54:41.690+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:54:41.691244+0000) 2024-03-04T19:54:42.690 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:54:42.690+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:54:42.691495+0000) 2024-03-04T19:54:43.689 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:54:43.691+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:54:43.691738+0000) 2024-03-04T19:54:44.690 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:54:44.691+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:54:44.691999+0000) 2024-03-04T19:54:45.690 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:54:45.691+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:54:45.692247+0000) 2024-03-04T19:54:46.690 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:54:46.691+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:54:46.692489+0000) 2024-03-04T19:54:47.691 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:54:47.692+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:54:47.692749+0000) 2024-03-04T19:54:48.691 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:54:48.692+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:54:48.692987+0000) 2024-03-04T19:54:49.691 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:54:49.692+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:54:49.693226+0000) 2024-03-04T19:54:50.691 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:54:50.692+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:54:50.693498+0000) 2024-03-04T19:54:51.691 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:54:51.693+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:54:51.693766+0000) 2024-03-04T19:54:52.691 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:54:52.693+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:54:52.694010+0000) 2024-03-04T19:54:53.692 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:54:53.693+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:54:53.694257+0000) 2024-03-04T19:54:54.692 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:54:54.694+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:54:54.694509+0000) 2024-03-04T19:54:55.692 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:54:55.694+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:54:55.694766+0000) 2024-03-04T19:54:56.692 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:54:56.694+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:54:56.695011+0000) 2024-03-04T19:54:57.693 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:54:57.694+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:54:57.695194+0000) 2024-03-04T19:54:58.693 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:54:58.695+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:54:58.695432+0000) 2024-03-04T19:54:59.693 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:54:59.695+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:54:59.695683+0000) 2024-03-04T19:55:00.693 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:55:00.695+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:55:00.695949+0000) 2024-03-04T19:55:01.693 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:55:01.695+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:55:01.696143+0000) 2024-03-04T19:55:02.694 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:55:02.696+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:55:02.696385+0000) 2024-03-04T19:55:03.560 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:55:03.569 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:55:03.716 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:55:03.696+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:55:03.696637+0000) 2024-03-04T19:55:04.694 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:55:04.696+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:55:04.696939+0000) 2024-03-04T19:55:05.695 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:55:05.696+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:55:05.697179+0000) 2024-03-04T19:55:06.695 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:55:06.697+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:55:06.697419+0000) 2024-03-04T19:55:07.695 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:55:07.697+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:55:07.697658+0000) 2024-03-04T19:55:08.695 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:55:08.697+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:55:08.697938+0000) 2024-03-04T19:55:09.695 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:55:09.697+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:55:09.698183+0000) 2024-03-04T19:55:10.695 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:55:10.698+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:55:10.698430+0000) 2024-03-04T19:55:11.696 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:55:11.698+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:55:11.698687+0000) 2024-03-04T19:55:12.696 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:55:12.698+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:55:12.698944+0000) 2024-03-04T19:55:13.696 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:55:13.699+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:55:13.699215+0000) 2024-03-04T19:55:14.696 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:55:14.698+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:55:14.699478+0000) 2024-03-04T19:55:15.696 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:55:15.699+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:55:15.699692+0000) 2024-03-04T19:55:16.697 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:55:16.699+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:55:16.699951+0000) 2024-03-04T19:55:17.697 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:55:17.700+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:55:17.700206+0000) 2024-03-04T19:55:18.697 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:55:18.699+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:55:18.700465+0000) 2024-03-04T19:55:19.697 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:55:19.699+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:55:19.700723+0000) 2024-03-04T19:55:20.697 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:55:20.699+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:55:20.700973+0000) 2024-03-04T19:55:21.698 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:55:21.700+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:55:21.701247+0000) 2024-03-04T19:55:22.698 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:55:22.700+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:55:22.701495+0000) 2024-03-04T19:55:23.698 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:55:23.700+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:55:23.701755+0000) 2024-03-04T19:55:24.698 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:55:24.701+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:55:24.702028+0000) 2024-03-04T19:55:25.698 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:55:25.701+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:55:25.702287+0000) 2024-03-04T19:55:26.699 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:55:26.700+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:55:26.702554+0000) 2024-03-04T19:55:27.699 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:55:27.701+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:55:27.702808+0000) 2024-03-04T19:55:28.699 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:55:28.702+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:55:28.703115+0000) 2024-03-04T19:55:29.700 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:55:29.702+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:55:29.703372+0000) 2024-03-04T19:55:30.700 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:55:30.702+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:55:30.703631+0000) 2024-03-04T19:55:31.700 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:55:31.703+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:55:31.703935+0000) 2024-03-04T19:55:32.700 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:55:32.703+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:55:32.704091+0000) 2024-03-04T19:55:33.700 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:55:33.703+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:55:33.704317+0000) 2024-03-04T19:55:33.780 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:55:33.792 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:55:34.701 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:55:34.703+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:55:34.704590+0000) 2024-03-04T19:55:35.701 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:55:35.703+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:55:35.704789+0000) 2024-03-04T19:55:36.701 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:55:36.704+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:55:36.704997+0000) 2024-03-04T19:55:37.701 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:55:37.704+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:55:37.705238+0000) 2024-03-04T19:55:38.701 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:55:38.704+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:55:38.705514+0000) 2024-03-04T19:55:39.701 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:55:39.704+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:55:39.705762+0000) 2024-03-04T19:55:40.702 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:55:40.705+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:55:40.706011+0000) 2024-03-04T19:55:41.702 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:55:41.705+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:55:41.706249+0000) 2024-03-04T19:55:42.702 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:55:42.705+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:55:42.706482+0000) 2024-03-04T19:55:43.703 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:55:43.706+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:55:43.706716+0000) 2024-03-04T19:55:44.703 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:55:44.706+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:55:44.706966+0000) 2024-03-04T19:55:45.703 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:55:45.706+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:55:45.707207+0000) 2024-03-04T19:55:46.703 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:55:46.706+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:55:46.707444+0000) 2024-03-04T19:55:47.703 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:55:47.707+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:55:47.707684+0000) 2024-03-04T19:55:48.703 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:55:48.707+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:55:48.707922+0000) 2024-03-04T19:55:49.704 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:55:49.707+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:55:49.708157+0000) 2024-03-04T19:55:50.704 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:55:50.707+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:55:50.708392+0000) 2024-03-04T19:55:51.704 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:55:51.708+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:55:51.708705+0000) 2024-03-04T19:55:52.704 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:55:52.708+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:55:52.708977+0000) 2024-03-04T19:55:53.705 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:55:53.708+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:55:53.709215+0000) 2024-03-04T19:55:54.705 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:55:54.708+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:55:54.709469+0000) 2024-03-04T19:55:55.705 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:55:55.708+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:55:55.709667+0000) 2024-03-04T19:55:56.705 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:55:56.709+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:55:56.709945+0000) 2024-03-04T19:55:57.706 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:55:57.709+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:55:57.710173+0000) 2024-03-04T19:55:58.706 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:55:58.709+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:55:58.710407+0000) 2024-03-04T19:55:59.706 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:55:59.710+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:55:59.710641+0000) 2024-03-04T19:56:00.706 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:56:00.710+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:56:00.710947+0000) 2024-03-04T19:56:01.706 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:56:01.710+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:56:01.711195+0000) 2024-03-04T19:56:02.707 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:56:02.711+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:56:02.711435+0000) 2024-03-04T19:56:03.707 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:56:03.710+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:56:03.711623+0000) 2024-03-04T19:56:03.950 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:56:03.959 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:56:04.707 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:56:04.711+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:56:04.711934+0000) 2024-03-04T19:56:05.707 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:56:05.711+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:56:05.712188+0000) 2024-03-04T19:56:06.707 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:56:06.712+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:56:06.712442+0000) 2024-03-04T19:56:07.708 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:56:07.712+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:56:07.712712+0000) 2024-03-04T19:56:08.708 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:56:08.712+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:56:08.712949+0000) 2024-03-04T19:56:09.708 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:56:09.712+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:56:09.713187+0000) 2024-03-04T19:56:10.708 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:56:10.712+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:56:10.713376+0000) 2024-03-04T19:56:11.711 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:56:11.713+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:56:11.713656+0000) 2024-03-04T19:56:12.709 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:56:12.713+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:56:12.713942+0000) 2024-03-04T19:56:13.709 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:56:13.713+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:56:13.714213+0000) 2024-03-04T19:56:14.709 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:56:14.714+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:56:14.714462+0000) 2024-03-04T19:56:15.710 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:56:15.713+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:56:15.714701+0000) 2024-03-04T19:56:16.710 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:56:16.714+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:56:16.714936+0000) 2024-03-04T19:56:17.710 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:56:17.715+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:56:17.715173+0000) 2024-03-04T19:56:18.710 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:56:18.715+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:56:18.715439+0000) 2024-03-04T19:56:19.710 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:56:19.715+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:56:19.715681+0000) 2024-03-04T19:56:20.710 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:56:20.714+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:56:20.715926+0000) 2024-03-04T19:56:21.710 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:56:21.715+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:56:21.716182+0000) 2024-03-04T19:56:22.711 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:56:22.715+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:56:22.716445+0000) 2024-03-04T19:56:23.711 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:56:23.715+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:56:23.716682+0000) 2024-03-04T19:56:24.711 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:56:24.715+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:56:24.716955+0000) 2024-03-04T19:56:25.711 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:56:25.716+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:56:25.717200+0000) 2024-03-04T19:56:26.712 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:56:26.716+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:56:26.717440+0000) 2024-03-04T19:56:27.712 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:56:27.716+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:56:27.717694+0000) 2024-03-04T19:56:28.712 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:56:28.716+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:56:28.717932+0000) 2024-03-04T19:56:29.712 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:56:29.716+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:56:29.718113+0000) 2024-03-04T19:56:30.713 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:56:30.717+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:56:30.718357+0000) 2024-03-04T19:56:31.713 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:56:31.717+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:56:31.718601+0000) 2024-03-04T19:56:32.713 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:56:32.718+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:56:32.718892+0000) 2024-03-04T19:56:33.713 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:56:33.718+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:56:33.719151+0000) 2024-03-04T19:56:34.163 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:56:34.172 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:56:34.713 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:56:34.718+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:56:34.719440+0000) 2024-03-04T19:56:35.714 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:56:35.718+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:56:35.719685+0000) 2024-03-04T19:56:36.714 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:56:36.719+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:56:36.719927+0000) 2024-03-04T19:56:37.714 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:56:37.719+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:56:37.720163+0000) 2024-03-04T19:56:38.714 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:56:38.719+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:56:38.720400+0000) 2024-03-04T19:56:39.715 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:56:39.719+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:56:39.720687+0000) 2024-03-04T19:56:40.715 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:56:40.720+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:56:40.720934+0000) 2024-03-04T19:56:41.715 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:56:41.720+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:56:41.721178+0000) 2024-03-04T19:56:42.715 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:56:42.720+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:56:42.721417+0000) 2024-03-04T19:56:43.717 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:56:43.720+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:56:43.721672+0000) 2024-03-04T19:56:44.716 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:56:44.721+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:56:44.721949+0000) 2024-03-04T19:56:45.716 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:56:45.721+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:56:45.722202+0000) 2024-03-04T19:56:46.716 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:56:46.721+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:56:46.722455+0000) 2024-03-04T19:56:47.717 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:56:47.722+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:56:47.722692+0000) 2024-03-04T19:56:48.719 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:56:48.722+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:56:48.722932+0000) 2024-03-04T19:56:49.717 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:56:49.722+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:56:49.723168+0000) 2024-03-04T19:56:50.717 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:56:50.722+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:56:50.723413+0000) 2024-03-04T19:56:51.717 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:56:51.723+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:56:51.723677+0000) 2024-03-04T19:56:52.717 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:56:52.723+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:56:52.723906+0000) 2024-03-04T19:56:53.717 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:56:53.723+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:56:53.724161+0000) 2024-03-04T19:56:54.718 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:56:54.723+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:56:54.724407+0000) 2024-03-04T19:56:55.718 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:56:55.724+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:56:55.724664+0000) 2024-03-04T19:56:56.718 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:56:56.724+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:56:56.724957+0000) 2024-03-04T19:56:57.719 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:56:57.724+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:56:57.725210+0000) 2024-03-04T19:56:58.719 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:56:58.724+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:56:58.725463+0000) 2024-03-04T19:56:59.719 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:56:59.725+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:56:59.725721+0000) 2024-03-04T19:57:00.719 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:57:00.725+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:57:00.725977+0000) 2024-03-04T19:57:01.720 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:57:01.725+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:57:01.726219+0000) 2024-03-04T19:57:02.719 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:57:02.726+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:57:02.726459+0000) 2024-03-04T19:57:03.720 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:57:03.726+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:57:03.726698+0000) 2024-03-04T19:57:04.375 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:57:04.384 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:57:04.720 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:57:04.726+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:57:04.726963+0000) 2024-03-04T19:57:05.720 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:57:05.726+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:57:05.727211+0000) 2024-03-04T19:57:06.720 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:57:06.727+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:57:06.727455+0000) 2024-03-04T19:57:07.721 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:57:07.727+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:57:07.727702+0000) 2024-03-04T19:57:08.722 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:57:08.727+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:57:08.727961+0000) 2024-03-04T19:57:09.721 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:57:09.727+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:57:09.728223+0000) 2024-03-04T19:57:10.722 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:57:10.727+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:57:10.728438+0000) 2024-03-04T19:57:11.721 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:57:11.728+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:57:11.728697+0000) 2024-03-04T19:57:12.722 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:57:12.728+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:57:12.728953+0000) 2024-03-04T19:57:13.722 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:57:13.728+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:57:13.729198+0000) 2024-03-04T19:57:14.723 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:57:14.729+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:57:14.729488+0000) 2024-03-04T19:57:15.722 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:57:15.729+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:57:15.729752+0000) 2024-03-04T19:57:16.725 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:57:16.729+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:57:16.730039+0000) 2024-03-04T19:57:17.724 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:57:17.730+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:57:17.730297+0000) 2024-03-04T19:57:18.723 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:57:18.729+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:57:18.730567+0000) 2024-03-04T19:57:19.724 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:57:19.730+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:57:19.730858+0000) 2024-03-04T19:57:20.726 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:57:20.730+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:57:20.731128+0000) 2024-03-04T19:57:21.724 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:57:21.731+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:57:21.731436+0000) 2024-03-04T19:57:22.725 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:57:22.730+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:57:22.731714+0000) 2024-03-04T19:57:23.725 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:57:23.730+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:57:23.731971+0000) 2024-03-04T19:57:24.725 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:57:24.731+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:57:24.732237+0000) 2024-03-04T19:57:25.725 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:57:25.731+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:57:25.732498+0000) 2024-03-04T19:57:26.725 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:57:26.731+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:57:26.732752+0000) 2024-03-04T19:57:27.725 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:57:27.731+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:57:27.733002+0000) 2024-03-04T19:57:28.727 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:57:28.732+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:57:28.733258+0000) 2024-03-04T19:57:29.727 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:57:29.732+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:57:29.733544+0000) 2024-03-04T19:57:30.726 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:57:30.732+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:57:30.733804+0000) 2024-03-04T19:57:31.726 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:57:31.733+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:57:31.734120+0000) 2024-03-04T19:57:32.728 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:57:32.733+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:57:32.734381+0000) 2024-03-04T19:57:33.730 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:57:33.733+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:57:33.734631+0000) 2024-03-04T19:57:34.523 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:57:34.534 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:57:34.727 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:57:34.734+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:57:34.734945+0000) 2024-03-04T19:57:35.727 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:57:35.734+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:57:35.735195+0000) 2024-03-04T19:57:36.728 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:57:36.734+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:57:36.735438+0000) 2024-03-04T19:57:37.728 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:57:37.734+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:57:37.735681+0000) 2024-03-04T19:57:38.728 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:57:38.735+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:57:38.735947+0000) 2024-03-04T19:57:39.728 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:57:39.735+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:57:39.736141+0000) 2024-03-04T19:57:40.728 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:57:40.735+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:57:40.736386+0000) 2024-03-04T19:57:41.728 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:57:41.735+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:57:41.736632+0000) 2024-03-04T19:57:42.729 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:57:42.736+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:57:42.736920+0000) 2024-03-04T19:57:43.729 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:57:43.736+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:57:43.737182+0000) 2024-03-04T19:57:44.730 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:57:44.736+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:57:44.737447+0000) 2024-03-04T19:57:45.729 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:57:45.736+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:57:45.737698+0000) 2024-03-04T19:57:46.730 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:57:46.737+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:57:46.737948+0000) 2024-03-04T19:57:47.730 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:57:47.737+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:57:47.738197+0000) 2024-03-04T19:57:48.731 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:57:48.737+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:57:48.738435+0000) 2024-03-04T19:57:49.731 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:57:49.737+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:57:49.738669+0000) 2024-03-04T19:57:50.731 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:57:50.738+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:57:50.738974+0000) 2024-03-04T19:57:51.731 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:57:51.738+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:57:51.739293+0000) 2024-03-04T19:57:52.731 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:57:52.738+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:57:52.739598+0000) 2024-03-04T19:57:53.732 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:57:53.739+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:57:53.739927+0000) 2024-03-04T19:57:54.732 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:57:54.739+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:57:54.740251+0000) 2024-03-04T19:57:55.732 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:57:55.739+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:57:55.740558+0000) 2024-03-04T19:57:56.732 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:57:56.739+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:57:56.740803+0000) 2024-03-04T19:57:57.732 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:57:57.740+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:57:57.741141+0000) 2024-03-04T19:57:58.734 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:57:58.740+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:57:58.741433+0000) 2024-03-04T19:57:59.733 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:57:59.741+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:57:59.741730+0000) 2024-03-04T19:58:00.733 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:58:00.741+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:58:00.742022+0000) 2024-03-04T19:58:01.734 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:58:01.741+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:58:01.742322+0000) 2024-03-04T19:58:02.734 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:58:02.742+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:58:02.742623+0000) 2024-03-04T19:58:03.734 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:58:03.742+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:58:03.742965+0000) 2024-03-04T19:58:04.735 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:58:04.742+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:58:04.743233+0000) 2024-03-04T19:58:04.741 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:58:04.750 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:58:05.735 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:58:05.743+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:58:05.743497+0000) 2024-03-04T19:58:06.735 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:58:06.743+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:58:06.743741+0000) 2024-03-04T19:58:07.735 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:58:07.743+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:58:07.743985+0000) 2024-03-04T19:58:08.736 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:58:08.743+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:58:08.744230+0000) 2024-03-04T19:58:09.736 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:58:09.744+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:58:09.744473+0000) 2024-03-04T19:58:10.738 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:58:10.744+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:58:10.744741+0000) 2024-03-04T19:58:11.736 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:58:11.744+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:58:11.745013+0000) 2024-03-04T19:58:12.736 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:58:12.744+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:58:12.745249+0000) 2024-03-04T19:58:13.737 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:58:13.745+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:58:13.745494+0000) 2024-03-04T19:58:14.737 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:58:14.745+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:58:14.745747+0000) 2024-03-04T19:58:15.737 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:58:15.745+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:58:15.746094+0000) 2024-03-04T19:58:16.737 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:58:16.745+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:58:16.746293+0000) 2024-03-04T19:58:17.737 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:58:17.746+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:58:17.746555+0000) 2024-03-04T19:58:18.738 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:58:18.746+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:58:18.746858+0000) 2024-03-04T19:58:19.738 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:58:19.746+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:58:19.747103+0000) 2024-03-04T19:58:20.738 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:58:20.747+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:58:20.747354+0000) 2024-03-04T19:58:21.739 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:58:21.746+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:58:21.747618+0000) 2024-03-04T19:58:22.739 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:58:22.747+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:58:22.747892+0000) 2024-03-04T19:58:23.740 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:58:23.747+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:58:23.748133+0000) 2024-03-04T19:58:24.739 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:58:24.747+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:58:24.748334+0000) 2024-03-04T19:58:25.740 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:58:25.746+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:58:25.748530+0000) 2024-03-04T19:58:26.742 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:58:26.746+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:58:26.748774+0000) 2024-03-04T19:58:27.740 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:58:27.747+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:58:27.749017+0000) 2024-03-04T19:58:28.740 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:58:28.748+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:58:28.749256+0000) 2024-03-04T19:58:29.740 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:58:29.748+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:58:29.749495+0000) 2024-03-04T19:58:30.740 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:58:30.748+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:58:30.749752+0000) 2024-03-04T19:58:31.740 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:58:31.748+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:58:31.750000+0000) 2024-03-04T19:58:32.741 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:58:32.748+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:58:32.750209+0000) 2024-03-04T19:58:33.741 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:58:33.748+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:58:33.750474+0000) 2024-03-04T19:58:34.741 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:58:34.749+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:58:34.750714+0000) 2024-03-04T19:58:34.878 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:58:34.887 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:58:35.741 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:58:35.750+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:58:35.750973+0000) 2024-03-04T19:58:36.742 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:58:36.750+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:58:36.751226+0000) 2024-03-04T19:58:37.742 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:58:37.750+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:58:37.751463+0000) 2024-03-04T19:58:38.742 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:58:38.750+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:58:38.751696+0000) 2024-03-04T19:58:39.742 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:58:39.751+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:58:39.751934+0000) 2024-03-04T19:58:40.742 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:58:40.751+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:58:40.752183+0000) 2024-03-04T19:58:41.743 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:58:41.751+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:58:41.752435+0000) 2024-03-04T19:58:42.743 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:58:42.751+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:58:42.752696+0000) 2024-03-04T19:58:43.743 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:58:43.752+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:58:43.752952+0000) 2024-03-04T19:58:44.743 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:58:44.752+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:58:44.753218+0000) 2024-03-04T19:58:45.744 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:58:45.752+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:58:45.753482+0000) 2024-03-04T19:58:46.744 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:58:46.752+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:58:46.753752+0000) 2024-03-04T19:58:47.744 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:58:47.753+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:58:47.754029+0000) 2024-03-04T19:58:48.744 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:58:48.753+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:58:48.754238+0000) 2024-03-04T19:58:49.745 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:58:49.753+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:58:49.754492+0000) 2024-03-04T19:58:50.745 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:58:50.753+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:58:50.754758+0000) 2024-03-04T19:58:51.745 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:58:51.754+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:58:51.755007+0000) 2024-03-04T19:58:52.745 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:58:52.754+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:58:52.755264+0000) 2024-03-04T19:58:53.746 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:58:53.754+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:58:53.755516+0000) 2024-03-04T19:58:54.746 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:58:54.754+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:58:54.755800+0000) 2024-03-04T19:58:55.746 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:58:55.755+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:58:55.756149+0000) 2024-03-04T19:58:56.746 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:58:56.755+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:58:56.756407+0000) 2024-03-04T19:58:57.746 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:58:57.756+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:58:57.756675+0000) 2024-03-04T19:58:58.747 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:58:58.756+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:58:58.756934+0000) 2024-03-04T19:58:59.749 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:58:59.756+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:58:59.757196+0000) 2024-03-04T19:59:00.748 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:59:00.756+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:59:00.757460+0000) 2024-03-04T19:59:01.750 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:59:01.756+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:59:01.757673+0000) 2024-03-04T19:59:02.749 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:59:02.757+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:59:02.757928+0000) 2024-03-04T19:59:03.748 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:59:03.757+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:59:03.758189+0000) 2024-03-04T19:59:04.748 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:59:04.757+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:59:04.758439+0000) 2024-03-04T19:59:05.091 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:59:05.099 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:59:05.748 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:59:05.757+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:59:05.758740+0000) 2024-03-04T19:59:06.749 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:59:06.758+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:59:06.759000+0000) 2024-03-04T19:59:07.749 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:59:07.758+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:59:07.759239+0000) 2024-03-04T19:59:08.749 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:59:08.759+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:59:08.759574+0000) 2024-03-04T19:59:09.750 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:59:09.759+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:59:09.759915+0000) 2024-03-04T19:59:10.750 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:59:10.759+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:59:10.760219+0000) 2024-03-04T19:59:11.750 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:59:11.759+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:59:11.760466+0000) 2024-03-04T19:59:12.750 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:59:12.760+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:59:12.760765+0000) 2024-03-04T19:59:13.751 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:59:13.760+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:59:13.761059+0000) 2024-03-04T19:59:14.751 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:59:14.761+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:59:14.761376+0000) 2024-03-04T19:59:15.751 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:59:15.761+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:59:15.761695+0000) 2024-03-04T19:59:16.751 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:59:16.761+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:59:16.762004+0000) 2024-03-04T19:59:17.751 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:59:17.761+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:59:17.762305+0000) 2024-03-04T19:59:18.752 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:59:18.762+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:59:18.762558+0000) 2024-03-04T19:59:19.752 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:59:19.762+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:59:19.762867+0000) 2024-03-04T19:59:20.752 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:59:20.762+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:59:20.763109+0000) 2024-03-04T19:59:21.752 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:59:21.763+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:59:21.763356+0000) 2024-03-04T19:59:22.754 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:59:22.763+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:59:22.763600+0000) 2024-03-04T19:59:23.753 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:59:23.763+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:59:23.763873+0000) 2024-03-04T19:59:24.753 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:59:24.763+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:59:24.764118+0000) 2024-03-04T19:59:25.753 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:59:25.764+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:59:25.764351+0000) 2024-03-04T19:59:26.754 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:59:26.764+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:59:26.764594+0000) 2024-03-04T19:59:27.754 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:59:27.763+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:59:27.764856+0000) 2024-03-04T19:59:28.754 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:59:28.763+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:59:28.765096+0000) 2024-03-04T19:59:29.754 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:59:29.763+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:59:29.765362+0000) 2024-03-04T19:59:30.755 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:59:30.764+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:59:30.765603+0000) 2024-03-04T19:59:31.755 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:59:31.764+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:59:31.765883+0000) 2024-03-04T19:59:32.755 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:59:32.765+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:59:32.766126+0000) 2024-03-04T19:59:33.755 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:59:33.765+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:59:33.766363+0000) 2024-03-04T19:59:34.756 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:59:34.765+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:59:34.766596+0000) 2024-03-04T19:59:35.303 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:59:35.312 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T19:59:35.756 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:59:35.765+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:59:35.766889+0000) 2024-03-04T19:59:36.756 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:59:36.766+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:59:36.767135+0000) 2024-03-04T19:59:37.760 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:59:37.766+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:59:37.767374+0000) 2024-03-04T19:59:38.762 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:59:38.766+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:59:38.767606+0000) 2024-03-04T19:59:39.757 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:59:39.767+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:59:39.767890+0000) 2024-03-04T19:59:40.757 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:59:40.767+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:59:40.768127+0000) 2024-03-04T19:59:41.757 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:59:41.767+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:59:41.768371+0000) 2024-03-04T19:59:42.758 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:59:42.767+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:59:42.768608+0000) 2024-03-04T19:59:43.758 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:59:43.768+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:59:43.768888+0000) 2024-03-04T19:59:44.758 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:59:44.768+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:59:44.769156+0000) 2024-03-04T19:59:45.758 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:59:45.768+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:59:45.769419+0000) 2024-03-04T19:59:46.758 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:59:46.768+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:59:46.769668+0000) 2024-03-04T19:59:47.759 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:59:47.769+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:59:47.769958+0000) 2024-03-04T19:59:48.759 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:59:48.769+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:59:48.770201+0000) 2024-03-04T19:59:49.760 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:59:49.769+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:59:49.770436+0000) 2024-03-04T19:59:50.759 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:59:50.769+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:59:50.770671+0000) 2024-03-04T19:59:51.759 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:59:51.770+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:59:51.770944+0000) 2024-03-04T19:59:52.759 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:59:52.770+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:59:52.771186+0000) 2024-03-04T19:59:53.760 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:59:53.770+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:59:53.771422+0000) 2024-03-04T19:59:54.760 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:59:54.770+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:59:54.771676+0000) 2024-03-04T19:59:55.760 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:59:55.771+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:59:55.771952+0000) 2024-03-04T19:59:56.761 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:59:56.771+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:59:56.772240+0000) 2024-03-04T19:59:57.761 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:59:57.771+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:59:57.772498+0000) 2024-03-04T19:59:58.761 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:59:58.772+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:59:58.772734+0000) 2024-03-04T19:59:59.762 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T19:59:59.772+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T18:59:59.773011+0000) 2024-03-04T20:00:00.762 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:00:00.772+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:00:00.773303+0000) 2024-03-04T20:00:01.762 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:00:01.772+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:00:01.773605+0000) 2024-03-04T20:00:02.762 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:00:02.773+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:00:02.773872+0000) 2024-03-04T20:00:03.762 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:00:03.773+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:00:03.774165+0000) 2024-03-04T20:00:04.763 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:00:04.773+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:00:04.774476+0000) 2024-03-04T20:00:05.442 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:00:05.452 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:00:05.763 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:00:05.774+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:00:05.774775+0000) 2024-03-04T20:00:06.763 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:00:06.774+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:00:06.775027+0000) 2024-03-04T20:00:07.763 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:00:07.774+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:00:07.775268+0000) 2024-03-04T20:00:08.765 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:00:08.775+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:00:08.775508+0000) 2024-03-04T20:00:09.766 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:00:09.775+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:00:09.775744+0000) 2024-03-04T20:00:10.765 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:00:10.775+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:00:10.775989+0000) 2024-03-04T20:00:11.765 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:00:11.775+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:00:11.776233+0000) 2024-03-04T20:00:12.766 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:00:12.775+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:00:12.776469+0000) 2024-03-04T20:00:13.766 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:00:13.776+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:00:13.776741+0000) 2024-03-04T20:00:14.765 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:00:14.776+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:00:14.777010+0000) 2024-03-04T20:00:15.765 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:00:15.776+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:00:15.777268+0000) 2024-03-04T20:00:16.765 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:00:16.777+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:00:16.777512+0000) 2024-03-04T20:00:17.766 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:00:17.777+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:00:17.777762+0000) 2024-03-04T20:00:18.766 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:00:18.777+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:00:18.778013+0000) 2024-03-04T20:00:19.766 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:00:19.777+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:00:19.778253+0000) 2024-03-04T20:00:20.766 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:00:20.778+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:00:20.778490+0000) 2024-03-04T20:00:21.767 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:00:21.778+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:00:21.778735+0000) 2024-03-04T20:00:22.767 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:00:22.778+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:00:22.778973+0000) 2024-03-04T20:00:23.767 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:00:23.778+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:00:23.779215+0000) 2024-03-04T20:00:24.767 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:00:24.779+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:00:24.779456+0000) 2024-03-04T20:00:25.768 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:00:25.779+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:00:25.779692+0000) 2024-03-04T20:00:26.767 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:00:26.779+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:00:26.779934+0000) 2024-03-04T20:00:27.768 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:00:27.780+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:00:27.780174+0000) 2024-03-04T20:00:28.768 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:00:28.779+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:00:28.780411+0000) 2024-03-04T20:00:29.768 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:00:29.780+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:00:29.780661+0000) 2024-03-04T20:00:30.768 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:00:30.779+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:00:30.780934+0000) 2024-03-04T20:00:31.769 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:00:31.779+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:00:31.781225+0000) 2024-03-04T20:00:32.769 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:00:32.779+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:00:32.781432+0000) 2024-03-04T20:00:33.770 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:00:33.779+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:00:33.781636+0000) 2024-03-04T20:00:34.769 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:00:34.780+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:00:34.781933+0000) 2024-03-04T20:00:35.646 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:00:35.657 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:00:35.808 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:00:35.781+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:00:35.782188+0000) 2024-03-04T20:00:36.770 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:00:36.781+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:00:36.782449+0000) 2024-03-04T20:00:37.770 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:00:37.781+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:00:37.782711+0000) 2024-03-04T20:00:38.770 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:00:38.781+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:00:38.782954+0000) 2024-03-04T20:00:39.771 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:00:39.782+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:00:39.783216+0000) 2024-03-04T20:00:40.771 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:00:40.782+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:00:40.783495+0000) 2024-03-04T20:00:41.771 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:00:41.782+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:00:41.783746+0000) 2024-03-04T20:00:42.771 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:00:42.783+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:00:42.784003+0000) 2024-03-04T20:00:43.771 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:00:43.783+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:00:43.784278+0000) 2024-03-04T20:00:44.772 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:00:44.783+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:00:44.784561+0000) 2024-03-04T20:00:45.772 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:00:45.784+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:00:45.784812+0000) 2024-03-04T20:00:46.772 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:00:46.784+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:00:46.785091+0000) 2024-03-04T20:00:47.772 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:00:47.784+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:00:47.785331+0000) 2024-03-04T20:00:48.773 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:00:48.784+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:00:48.785568+0000) 2024-03-04T20:00:49.773 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:00:49.785+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:00:49.785916+0000) 2024-03-04T20:00:50.773 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:00:50.785+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:00:50.786114+0000) 2024-03-04T20:00:51.773 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:00:51.785+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:00:51.786364+0000) 2024-03-04T20:00:52.775 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:00:52.785+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:00:52.786603+0000) 2024-03-04T20:00:53.775 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:00:53.786+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:00:53.786937+0000) 2024-03-04T20:00:54.778 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:00:54.786+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:00:54.787243+0000) 2024-03-04T20:00:55.778 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:00:55.786+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:00:55.787491+0000) 2024-03-04T20:00:56.776 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:00:56.786+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:00:56.787714+0000) 2024-03-04T20:00:57.775 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:00:57.787+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:00:57.788003+0000) 2024-03-04T20:00:58.775 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:00:58.787+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:00:58.788302+0000) 2024-03-04T20:00:59.776 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:00:59.787+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:00:59.788589+0000) 2024-03-04T20:01:00.776 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:01:00.788+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:01:00.788929+0000) 2024-03-04T20:01:01.776 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:01:01.788+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:01:01.789216+0000) 2024-03-04T20:01:02.776 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:01:02.788+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:01:02.789512+0000) 2024-03-04T20:01:03.777 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:01:03.789+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:01:03.789817+0000) 2024-03-04T20:01:04.777 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:01:04.789+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:01:04.790223+0000) 2024-03-04T20:01:05.777 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:01:05.789+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:01:05.790495+0000) 2024-03-04T20:01:05.894 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:01:05.905 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:01:06.778 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:01:06.790+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:01:06.790785+0000) 2024-03-04T20:01:07.778 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:01:07.790+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:01:07.791101+0000) 2024-03-04T20:01:08.778 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:01:08.790+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:01:08.791355+0000) 2024-03-04T20:01:09.778 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:01:09.791+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:01:09.791600+0000) 2024-03-04T20:01:10.778 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:01:10.791+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:01:10.791894+0000) 2024-03-04T20:01:11.779 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:01:11.791+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:01:11.792160+0000) 2024-03-04T20:01:12.779 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:01:12.791+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:01:12.792405+0000) 2024-03-04T20:01:13.779 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:01:13.792+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:01:13.792643+0000) 2024-03-04T20:01:14.780 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:01:14.792+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:01:14.792944+0000) 2024-03-04T20:01:15.779 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:01:15.792+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:01:15.793181+0000) 2024-03-04T20:01:16.780 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:01:16.793+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:01:16.793428+0000) 2024-03-04T20:01:17.780 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:01:17.793+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:01:17.793665+0000) 2024-03-04T20:01:18.780 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:01:18.793+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:01:18.793954+0000) 2024-03-04T20:01:19.781 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:01:19.793+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:01:19.794206+0000) 2024-03-04T20:01:20.781 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:01:20.794+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:01:20.794437+0000) 2024-03-04T20:01:21.781 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:01:21.794+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:01:21.794681+0000) 2024-03-04T20:01:22.782 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:01:22.794+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:01:22.794958+0000) 2024-03-04T20:01:23.782 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:01:23.794+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:01:23.795208+0000) 2024-03-04T20:01:24.782 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:01:24.795+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:01:24.795456+0000) 2024-03-04T20:01:25.783 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:01:25.795+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:01:25.795690+0000) 2024-03-04T20:01:26.782 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:01:26.795+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:01:26.795933+0000) 2024-03-04T20:01:27.782 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:01:27.795+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:01:27.796174+0000) 2024-03-04T20:01:28.783 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:01:28.795+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:01:28.796353+0000) 2024-03-04T20:01:29.783 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:01:29.796+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:01:29.796587+0000) 2024-03-04T20:01:30.788 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:01:30.796+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:01:30.796863+0000) 2024-03-04T20:01:31.783 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:01:31.796+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:01:31.797155+0000) 2024-03-04T20:01:32.785 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:01:32.796+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:01:32.797485+0000) 2024-03-04T20:01:33.784 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:01:33.796+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:01:33.797776+0000) 2024-03-04T20:01:34.784 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:01:34.797+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:01:34.798131+0000) 2024-03-04T20:01:35.784 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:01:35.796+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:01:35.798365+0000) 2024-03-04T20:01:36.027 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:01:36.036 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:01:36.785 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:01:36.797+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:01:36.798667+0000) 2024-03-04T20:01:37.785 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:01:37.797+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:01:37.798970+0000) 2024-03-04T20:01:38.790 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:01:38.798+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:01:38.799264+0000) 2024-03-04T20:01:39.786 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:01:39.798+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:01:39.799571+0000) 2024-03-04T20:01:40.786 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:01:40.798+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:01:40.799919+0000) 2024-03-04T20:01:41.788 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:01:41.799+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:01:41.800222+0000) 2024-03-04T20:01:42.788 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:01:42.799+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:01:42.800515+0000) 2024-03-04T20:01:43.787 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:01:43.799+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:01:43.800743+0000) 2024-03-04T20:01:44.787 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:01:44.800+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:01:44.801051+0000) 2024-03-04T20:01:45.787 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:01:45.800+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:01:45.801340+0000) 2024-03-04T20:01:46.788 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:01:46.800+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:01:46.801644+0000) 2024-03-04T20:01:47.788 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:01:47.801+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:01:47.801961+0000) 2024-03-04T20:01:48.788 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:01:48.801+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:01:48.802241+0000) 2024-03-04T20:01:49.789 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:01:49.801+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:01:49.802543+0000) 2024-03-04T20:01:50.788 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:01:50.802+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:01:50.802879+0000) 2024-03-04T20:01:51.790 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:01:51.802+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:01:51.803174+0000) 2024-03-04T20:01:52.789 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:01:52.802+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:01:52.803471+0000) 2024-03-04T20:01:53.790 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:01:53.802+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:01:53.803750+0000) 2024-03-04T20:01:54.790 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:01:54.803+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:01:54.804055+0000) 2024-03-04T20:01:55.791 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:01:55.803+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:01:55.804308+0000) 2024-03-04T20:01:56.790 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:01:56.803+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:01:56.804506+0000) 2024-03-04T20:01:57.790 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:01:57.804+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:01:57.804748+0000) 2024-03-04T20:01:58.791 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:01:58.804+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:01:58.805002+0000) 2024-03-04T20:01:59.791 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:01:59.804+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:01:59.805258+0000) 2024-03-04T20:02:00.791 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:02:00.804+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:02:00.805463+0000) 2024-03-04T20:02:01.791 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:02:01.804+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:02:01.805671+0000) 2024-03-04T20:02:02.792 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:02:02.805+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:02:02.805928+0000) 2024-03-04T20:02:03.792 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:02:03.805+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:02:03.806189+0000) 2024-03-04T20:02:04.792 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:02:04.805+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:02:04.806457+0000) 2024-03-04T20:02:05.792 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:02:05.806+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:02:05.806712+0000) 2024-03-04T20:02:06.258 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:02:06.266 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:02:06.792 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:02:06.806+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:02:06.807039+0000) 2024-03-04T20:02:07.793 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:02:07.806+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:02:07.807315+0000) 2024-03-04T20:02:08.793 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:02:08.807+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:02:08.807615+0000) 2024-03-04T20:02:09.795 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:02:09.807+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:02:09.807940+0000) 2024-03-04T20:02:10.794 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:02:10.807+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:02:10.808234+0000) 2024-03-04T20:02:11.794 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:02:11.807+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:02:11.808526+0000) 2024-03-04T20:02:12.794 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:02:12.808+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:02:12.808868+0000) 2024-03-04T20:02:13.795 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:02:13.808+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:02:13.809165+0000) 2024-03-04T20:02:14.795 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:02:14.808+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:02:14.809477+0000) 2024-03-04T20:02:15.795 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:02:15.809+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:02:15.809777+0000) 2024-03-04T20:02:16.795 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:02:16.809+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:02:16.810076+0000) 2024-03-04T20:02:17.796 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:02:17.809+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:02:17.810377+0000) 2024-03-04T20:02:18.796 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:02:18.810+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:02:18.810676+0000) 2024-03-04T20:02:19.796 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:02:19.810+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:02:19.810973+0000) 2024-03-04T20:02:20.797 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:02:20.810+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:02:20.811217+0000) 2024-03-04T20:02:21.797 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:02:21.811+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:02:21.811514+0000) 2024-03-04T20:02:22.797 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:02:22.811+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:02:22.811812+0000) 2024-03-04T20:02:23.797 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:02:23.811+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:02:23.812154+0000) 2024-03-04T20:02:24.798 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:02:24.812+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:02:24.812462+0000) 2024-03-04T20:02:25.798 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:02:25.811+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:02:25.812696+0000) 2024-03-04T20:02:26.798 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:02:26.812+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:02:26.813014+0000) 2024-03-04T20:02:27.799 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:02:27.813+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:02:27.813277+0000) 2024-03-04T20:02:28.801 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:02:28.813+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:02:28.813590+0000) 2024-03-04T20:02:29.799 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:02:29.813+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:02:29.813923+0000) 2024-03-04T20:02:30.799 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:02:30.813+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:02:30.814213+0000) 2024-03-04T20:02:31.800 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:02:31.814+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:02:31.814515+0000) 2024-03-04T20:02:32.800 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:02:32.814+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:02:32.814866+0000) 2024-03-04T20:02:33.802 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:02:33.814+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:02:33.815160+0000) 2024-03-04T20:02:34.800 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:02:34.815+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:02:34.815467+0000) 2024-03-04T20:02:35.801 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:02:35.814+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:02:35.815779+0000) 2024-03-04T20:02:36.495 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:02:36.504 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:02:36.801 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:02:36.814+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:02:36.816131+0000) 2024-03-04T20:02:37.801 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:02:37.815+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:02:37.816438+0000) 2024-03-04T20:02:38.801 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:02:38.815+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:02:38.816740+0000) 2024-03-04T20:02:39.802 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:02:39.815+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:02:39.817067+0000) 2024-03-04T20:02:40.802 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:02:40.816+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:02:40.817361+0000) 2024-03-04T20:02:41.803 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:02:41.815+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:02:41.817673+0000) 2024-03-04T20:02:42.803 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:02:42.816+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:02:42.817989+0000) 2024-03-04T20:02:43.803 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:02:43.817+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:02:43.818287+0000) 2024-03-04T20:02:44.803 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:02:44.817+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:02:44.818613+0000) 2024-03-04T20:02:45.804 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:02:45.817+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:02:45.818939+0000) 2024-03-04T20:02:46.804 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:02:46.818+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:02:46.819243+0000) 2024-03-04T20:02:47.804 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:02:47.818+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:02:47.819548+0000) 2024-03-04T20:02:48.805 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:02:48.819+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:02:48.819873+0000) 2024-03-04T20:02:49.805 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:02:49.819+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:02:49.820186+0000) 2024-03-04T20:02:50.805 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:02:50.819+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:02:50.820488+0000) 2024-03-04T20:02:51.806 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:02:51.819+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:02:51.820786+0000) 2024-03-04T20:02:52.806 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:02:52.820+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:02:52.821133+0000) 2024-03-04T20:02:53.806 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:02:53.820+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:02:53.821424+0000) 2024-03-04T20:02:54.806 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:02:54.820+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:02:54.821722+0000) 2024-03-04T20:02:55.807 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:02:55.821+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:02:55.822010+0000) 2024-03-04T20:02:56.807 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:02:56.821+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:02:56.822334+0000) 2024-03-04T20:02:57.807 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:02:57.821+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:02:57.822643+0000) 2024-03-04T20:02:58.807 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:02:58.822+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:02:58.822926+0000) 2024-03-04T20:02:59.807 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:02:59.822+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:02:59.823229+0000) 2024-03-04T20:03:00.808 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:03:00.822+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:03:00.823552+0000) 2024-03-04T20:03:01.808 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:03:01.823+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:03:01.823904+0000) 2024-03-04T20:03:02.809 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:03:02.823+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:03:02.824211+0000) 2024-03-04T20:03:03.809 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:03:03.823+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:03:03.824504+0000) 2024-03-04T20:03:04.809 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:03:04.824+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:03:04.824865+0000) 2024-03-04T20:03:05.809 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:03:05.824+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:03:05.825165+0000) 2024-03-04T20:03:06.691 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:03:06.699 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:03:06.840 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:03:06.824+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:03:06.825460+0000) 2024-03-04T20:03:07.810 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:03:07.825+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:03:07.825762+0000) 2024-03-04T20:03:08.810 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:03:08.825+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:03:08.826035+0000) 2024-03-04T20:03:09.811 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:03:09.825+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:03:09.826351+0000) 2024-03-04T20:03:10.811 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:03:10.825+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:03:10.826660+0000) 2024-03-04T20:03:11.811 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:03:11.826+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:03:11.826971+0000) 2024-03-04T20:03:12.812 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:03:12.826+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:03:12.827215+0000) 2024-03-04T20:03:13.812 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:03:13.826+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:03:13.827502+0000) 2024-03-04T20:03:14.812 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:03:14.826+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:03:14.827771+0000) 2024-03-04T20:03:15.812 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:03:15.827+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:03:15.828000+0000) 2024-03-04T20:03:16.812 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:03:16.827+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:03:16.828249+0000) 2024-03-04T20:03:17.813 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:03:17.828+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:03:17.828495+0000) 2024-03-04T20:03:18.813 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:03:18.828+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:03:18.828732+0000) 2024-03-04T20:03:19.813 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:03:19.828+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:03:19.829026+0000) 2024-03-04T20:03:20.813 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:03:20.828+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:03:20.829260+0000) 2024-03-04T20:03:21.814 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:03:21.828+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:03:21.829461+0000) 2024-03-04T20:03:22.814 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:03:22.829+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:03:22.829712+0000) 2024-03-04T20:03:23.814 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:03:23.829+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:03:23.829990+0000) 2024-03-04T20:03:24.814 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:03:24.829+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:03:24.830275+0000) 2024-03-04T20:03:25.815 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:03:25.829+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:03:25.830554+0000) 2024-03-04T20:03:26.815 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:03:26.830+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:03:26.830885+0000) 2024-03-04T20:03:27.815 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:03:27.830+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:03:27.831185+0000) 2024-03-04T20:03:28.815 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:03:28.831+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:03:28.831436+0000) 2024-03-04T20:03:29.816 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:03:29.831+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:03:29.831739+0000) 2024-03-04T20:03:30.816 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:03:30.831+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:03:30.832038+0000) 2024-03-04T20:03:31.816 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:03:31.832+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:03:31.832334+0000) 2024-03-04T20:03:32.816 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:03:32.832+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:03:32.832644+0000) 2024-03-04T20:03:33.817 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:03:33.832+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:03:33.832974+0000) 2024-03-04T20:03:34.817 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:03:34.833+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:03:34.833282+0000) 2024-03-04T20:03:35.817 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:03:35.833+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:03:35.833579+0000) 2024-03-04T20:03:36.818 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:03:36.833+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:03:36.833927+0000) 2024-03-04T20:03:36.912 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:03:36.921 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:03:37.818 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:03:37.834+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:03:37.834234+0000) 2024-03-04T20:03:38.818 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:03:38.833+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:03:38.834530+0000) 2024-03-04T20:03:39.819 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:03:39.833+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:03:39.834816+0000) 2024-03-04T20:03:40.819 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:03:40.834+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:03:40.835123+0000) 2024-03-04T20:03:41.819 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:03:41.834+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:03:41.835389+0000) 2024-03-04T20:03:42.819 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:03:42.834+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:03:42.835674+0000) 2024-03-04T20:03:43.820 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:03:43.834+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:03:43.835965+0000) 2024-03-04T20:03:44.820 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:03:44.835+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:03:44.836272+0000) 2024-03-04T20:03:45.820 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:03:45.835+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:03:45.836588+0000) 2024-03-04T20:03:46.821 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:03:46.835+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:03:46.836917+0000) 2024-03-04T20:03:47.821 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:03:47.836+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:03:47.837223+0000) 2024-03-04T20:03:48.821 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:03:48.836+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:03:48.837533+0000) 2024-03-04T20:03:49.822 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:03:49.837+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:03:49.837894+0000) 2024-03-04T20:03:50.822 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:03:50.837+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:03:50.838190+0000) 2024-03-04T20:03:51.822 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:03:51.837+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:03:51.838496+0000) 2024-03-04T20:03:52.822 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:03:52.837+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:03:52.838790+0000) 2024-03-04T20:03:53.823 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:03:53.838+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:03:53.839098+0000) 2024-03-04T20:03:54.823 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:03:54.838+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:03:54.839400+0000) 2024-03-04T20:03:55.823 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:03:55.838+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:03:55.839684+0000) 2024-03-04T20:03:56.824 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:03:56.839+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:03:56.839939+0000) 2024-03-04T20:03:57.824 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:03:57.839+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:03:57.840189+0000) 2024-03-04T20:03:58.824 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:03:58.839+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:03:58.840426+0000) 2024-03-04T20:03:59.824 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:03:59.839+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:03:59.840663+0000) 2024-03-04T20:04:00.824 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:04:00.840+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:04:00.840924+0000) 2024-03-04T20:04:01.825 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:04:01.840+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:04:01.841156+0000) 2024-03-04T20:04:02.825 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:04:02.840+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:04:02.841368+0000) 2024-03-04T20:04:03.825 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:04:03.840+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:04:03.841630+0000) 2024-03-04T20:04:04.825 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:04:04.841+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:04:04.841932+0000) 2024-03-04T20:04:05.826 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:04:05.841+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:04:05.842143+0000) 2024-03-04T20:04:06.826 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:04:06.841+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:04:06.842374+0000) 2024-03-04T20:04:07.050 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:04:07.062 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:04:07.826 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:04:07.841+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:04:07.842640+0000) 2024-03-04T20:04:08.826 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:04:08.842+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:04:08.842938+0000) 2024-03-04T20:04:09.826 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:04:09.842+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:04:09.843129+0000) 2024-03-04T20:04:10.827 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:04:10.842+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:04:10.843376+0000) 2024-03-04T20:04:11.827 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:04:11.843+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:04:11.843620+0000) 2024-03-04T20:04:12.827 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:04:12.843+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:04:12.843939+0000) 2024-03-04T20:04:13.827 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:04:13.843+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:04:13.844178+0000) 2024-03-04T20:04:14.828 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:04:14.843+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:04:14.844379+0000) 2024-03-04T20:04:15.828 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:04:15.844+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:04:15.844620+0000) 2024-03-04T20:04:16.828 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:04:16.844+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:04:16.844915+0000) 2024-03-04T20:04:17.828 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:04:17.844+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:04:17.845135+0000) 2024-03-04T20:04:18.828 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:04:18.844+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:04:18.845394+0000) 2024-03-04T20:04:19.829 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:04:19.845+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:04:19.845639+0000) 2024-03-04T20:04:20.829 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:04:20.845+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:04:20.845927+0000) 2024-03-04T20:04:21.829 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:04:21.845+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:04:21.846166+0000) 2024-03-04T20:04:22.829 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:04:22.846+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:04:22.846431+0000) 2024-03-04T20:04:23.830 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:04:23.845+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:04:23.846666+0000) 2024-03-04T20:04:24.830 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:04:24.846+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:04:24.846943+0000) 2024-03-04T20:04:25.830 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:04:25.846+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:04:25.847180+0000) 2024-03-04T20:04:26.830 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:04:26.847+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:04:26.847423+0000) 2024-03-04T20:04:27.830 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:04:27.847+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:04:27.847672+0000) 2024-03-04T20:04:28.831 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:04:28.847+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:04:28.847947+0000) 2024-03-04T20:04:29.831 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:04:29.847+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:04:29.848199+0000) 2024-03-04T20:04:30.831 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:04:30.848+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:04:30.848440+0000) 2024-03-04T20:04:31.832 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:04:31.848+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:04:31.848680+0000) 2024-03-04T20:04:32.832 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:04:32.848+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:04:32.848949+0000) 2024-03-04T20:04:33.832 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:04:33.848+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:04:33.849192+0000) 2024-03-04T20:04:34.832 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:04:34.849+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:04:34.849444+0000) 2024-03-04T20:04:35.832 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:04:35.848+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:04:35.849632+0000) 2024-03-04T20:04:36.832 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:04:36.849+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:04:36.849904+0000) 2024-03-04T20:04:37.270 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:04:37.278 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:04:37.833 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:04:37.849+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:04:37.850172+0000) 2024-03-04T20:04:38.833 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:04:38.850+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:04:38.850433+0000) 2024-03-04T20:04:39.834 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:04:39.850+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:04:39.850689+0000) 2024-03-04T20:04:40.834 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:04:40.849+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:04:40.850942+0000) 2024-03-04T20:04:41.834 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:04:41.850+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:04:41.851177+0000) 2024-03-04T20:04:42.834 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:04:42.849+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:04:42.851373+0000) 2024-03-04T20:04:43.834 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:04:43.849+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:04:43.851641+0000) 2024-03-04T20:04:44.834 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:04:44.850+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:04:44.851938+0000) 2024-03-04T20:04:45.835 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:04:45.851+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:04:45.852180+0000) 2024-03-04T20:04:46.835 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:04:46.851+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:04:46.852416+0000) 2024-03-04T20:04:47.835 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:04:47.851+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:04:47.852685+0000) 2024-03-04T20:04:48.835 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:04:48.851+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:04:48.852945+0000) 2024-03-04T20:04:49.835 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:04:49.852+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:04:49.853187+0000) 2024-03-04T20:04:50.836 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:04:50.852+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:04:50.853430+0000) 2024-03-04T20:04:51.836 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:04:51.852+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:04:51.853670+0000) 2024-03-04T20:04:52.836 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:04:52.853+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:04:52.853946+0000) 2024-03-04T20:04:53.837 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:04:53.853+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:04:53.854203+0000) 2024-03-04T20:04:54.837 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:04:54.853+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:04:54.854472+0000) 2024-03-04T20:04:55.837 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:04:55.853+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:04:55.854732+0000) 2024-03-04T20:04:56.837 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:04:56.854+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:04:56.854948+0000) 2024-03-04T20:04:57.838 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:04:57.854+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:04:57.855190+0000) 2024-03-04T20:04:58.838 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:04:58.854+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:04:58.855430+0000) 2024-03-04T20:04:59.838 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:04:59.854+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:04:59.855668+0000) 2024-03-04T20:05:00.838 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:05:00.855+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:05:00.855984+0000) 2024-03-04T20:05:01.838 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:05:01.855+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:05:01.856218+0000) 2024-03-04T20:05:02.839 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:05:02.855+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:05:02.856457+0000) 2024-03-04T20:05:03.839 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:05:03.855+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:05:03.856698+0000) 2024-03-04T20:05:04.839 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:05:04.856+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:05:04.856943+0000) 2024-03-04T20:05:05.839 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:05:05.856+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:05:05.857185+0000) 2024-03-04T20:05:06.839 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:05:06.856+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:05:06.857423+0000) 2024-03-04T20:05:07.522 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:05:07.534 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:05:07.840 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:05:07.856+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:05:07.857675+0000) 2024-03-04T20:05:08.840 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:05:08.857+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:05:08.857989+0000) 2024-03-04T20:05:09.841 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:05:09.857+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:05:09.858288+0000) 2024-03-04T20:05:10.841 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:05:10.857+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:05:10.858487+0000) 2024-03-04T20:05:11.841 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:05:11.858+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:05:11.858724+0000) 2024-03-04T20:05:12.841 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:05:12.858+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:05:12.858938+0000) 2024-03-04T20:05:13.841 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:05:13.858+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:05:13.859174+0000) 2024-03-04T20:05:14.841 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:05:14.858+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:05:14.859420+0000) 2024-03-04T20:05:15.842 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:05:15.859+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:05:15.859674+0000) 2024-03-04T20:05:16.842 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:05:16.859+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:05:16.859948+0000) 2024-03-04T20:05:17.842 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:05:17.859+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:05:17.860208+0000) 2024-03-04T20:05:18.843 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:05:18.859+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:05:18.860450+0000) 2024-03-04T20:05:19.843 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:05:19.860+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:05:19.860689+0000) 2024-03-04T20:05:20.843 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:05:20.860+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:05:20.860933+0000) 2024-03-04T20:05:21.843 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:05:21.860+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:05:21.861298+0000) 2024-03-04T20:05:22.843 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:05:22.861+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:05:22.861590+0000) 2024-03-04T20:05:23.844 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:05:23.861+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:05:23.861864+0000) 2024-03-04T20:05:24.844 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:05:24.861+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:05:24.862159+0000) 2024-03-04T20:05:25.844 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:05:25.861+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:05:25.862426+0000) 2024-03-04T20:05:26.844 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:05:26.862+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:05:26.862673+0000) 2024-03-04T20:05:27.845 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:05:27.862+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:05:27.862949+0000) 2024-03-04T20:05:28.845 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:05:28.862+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:05:28.863191+0000) 2024-03-04T20:05:29.845 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:05:29.863+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:05:29.863429+0000) 2024-03-04T20:05:30.846 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:05:30.863+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:05:30.863675+0000) 2024-03-04T20:05:31.846 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:05:31.863+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:05:31.863974+0000) 2024-03-04T20:05:32.846 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:05:32.863+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:05:32.864212+0000) 2024-03-04T20:05:33.846 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:05:33.864+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:05:33.864448+0000) 2024-03-04T20:05:34.846 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:05:34.864+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:05:34.864694+0000) 2024-03-04T20:05:35.846 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:05:35.864+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:05:35.864931+0000) 2024-03-04T20:05:36.847 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:05:36.864+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:05:36.865172+0000) 2024-03-04T20:05:37.672 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:05:37.681 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:05:37.847 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:05:37.865+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:05:37.865429+0000) 2024-03-04T20:05:38.847 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:05:38.864+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:05:38.865691+0000) 2024-03-04T20:05:39.847 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:05:39.865+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:05:39.865997+0000) 2024-03-04T20:05:40.848 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:05:40.866+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:05:40.866245+0000) 2024-03-04T20:05:41.848 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:05:41.866+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:05:41.866487+0000) 2024-03-04T20:05:42.848 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:05:42.866+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:05:42.866740+0000) 2024-03-04T20:05:43.848 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:05:43.865+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:05:43.866984+0000) 2024-03-04T20:05:44.849 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:05:44.866+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:05:44.867230+0000) 2024-03-04T20:05:45.849 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:05:45.865+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:05:45.867481+0000) 2024-03-04T20:05:46.850 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:05:46.866+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:05:46.867761+0000) 2024-03-04T20:05:47.850 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:05:47.866+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:05:47.867991+0000) 2024-03-04T20:05:48.850 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:05:48.867+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:05:48.868234+0000) 2024-03-04T20:05:49.850 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:05:49.867+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:05:49.868476+0000) 2024-03-04T20:05:50.850 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:05:50.867+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:05:50.868712+0000) 2024-03-04T20:05:51.851 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:05:51.867+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:05:51.868950+0000) 2024-03-04T20:05:52.851 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:05:52.868+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:05:52.869138+0000) 2024-03-04T20:05:53.851 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:05:53.868+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:05:53.869326+0000) 2024-03-04T20:05:54.851 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:05:54.868+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:05:54.869572+0000) 2024-03-04T20:05:55.851 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:05:55.869+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:05:55.869812+0000) 2024-03-04T20:05:56.852 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:05:56.869+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:05:56.870109+0000) 2024-03-04T20:05:57.852 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:05:57.869+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:05:57.870348+0000) 2024-03-04T20:05:58.852 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:05:58.869+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:05:58.870592+0000) 2024-03-04T20:05:59.852 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:05:59.870+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:05:59.870889+0000) 2024-03-04T20:06:00.853 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:06:00.870+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:06:00.871126+0000) 2024-03-04T20:06:01.853 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:06:01.870+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:06:01.871366+0000) 2024-03-04T20:06:02.853 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:06:02.870+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:06:02.871612+0000) 2024-03-04T20:06:03.853 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:06:03.871+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:06:03.871933+0000) 2024-03-04T20:06:04.854 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:06:04.871+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:06:04.872181+0000) 2024-03-04T20:06:05.854 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:06:05.871+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:06:05.872416+0000) 2024-03-04T20:06:06.854 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:06:06.871+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:06:06.872651+0000) 2024-03-04T20:06:07.854 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:06:07.872+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:06:07.872935+0000) 2024-03-04T20:06:07.910 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:06:07.918 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:06:08.854 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:06:08.872+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:06:08.873178+0000) 2024-03-04T20:06:09.855 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:06:09.872+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:06:09.873420+0000) 2024-03-04T20:06:10.855 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:06:10.872+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:06:10.873660+0000) 2024-03-04T20:06:11.855 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:06:11.873+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:06:11.873953+0000) 2024-03-04T20:06:12.856 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:06:12.873+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:06:12.874188+0000) 2024-03-04T20:06:13.856 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:06:13.873+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:06:13.874433+0000) 2024-03-04T20:06:14.856 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:06:14.874+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:06:14.874719+0000) 2024-03-04T20:06:15.856 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:06:15.874+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:06:15.874958+0000) 2024-03-04T20:06:16.857 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:06:16.874+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:06:16.875230+0000) 2024-03-04T20:06:17.857 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:06:17.874+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:06:17.875501+0000) 2024-03-04T20:06:18.857 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:06:18.875+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:06:18.875764+0000) 2024-03-04T20:06:19.857 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:06:19.875+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:06:19.876007+0000) 2024-03-04T20:06:20.857 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:06:20.875+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:06:20.876249+0000) 2024-03-04T20:06:21.858 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:06:21.875+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:06:21.876430+0000) 2024-03-04T20:06:22.858 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:06:22.875+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:06:22.876673+0000) 2024-03-04T20:06:23.858 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:06:23.876+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:06:23.876959+0000) 2024-03-04T20:06:24.858 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:06:24.876+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:06:24.877208+0000) 2024-03-04T20:06:25.859 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:06:25.877+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:06:25.877446+0000) 2024-03-04T20:06:26.859 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:06:26.877+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:06:26.877689+0000) 2024-03-04T20:06:27.859 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:06:27.877+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:06:27.877936+0000) 2024-03-04T20:06:28.859 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:06:28.877+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:06:28.878119+0000) 2024-03-04T20:06:29.860 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:06:29.878+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:06:29.878375+0000) 2024-03-04T20:06:30.860 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:06:30.878+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:06:30.878613+0000) 2024-03-04T20:06:31.860 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:06:31.878+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:06:31.878927+0000) 2024-03-04T20:06:32.860 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:06:32.878+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:06:32.879166+0000) 2024-03-04T20:06:33.860 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:06:33.879+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:06:33.879414+0000) 2024-03-04T20:06:34.861 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:06:34.879+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:06:34.879664+0000) 2024-03-04T20:06:35.861 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:06:35.879+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:06:35.879947+0000) 2024-03-04T20:06:36.861 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:06:36.879+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:06:36.880242+0000) 2024-03-04T20:06:37.861 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:06:37.880+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:06:37.880484+0000) 2024-03-04T20:06:38.054 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:06:38.063 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:06:38.861 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:06:38.880+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:06:38.880742+0000) 2024-03-04T20:06:39.862 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:06:39.880+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:06:39.880980+0000) 2024-03-04T20:06:40.862 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:06:40.881+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:06:40.881225+0000) 2024-03-04T20:06:41.862 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:06:41.881+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:06:41.881464+0000) 2024-03-04T20:06:42.863 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:06:42.881+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:06:42.881708+0000) 2024-03-04T20:06:43.863 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:06:43.881+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:06:43.881987+0000) 2024-03-04T20:06:44.863 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:06:44.882+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:06:44.882235+0000) 2024-03-04T20:06:45.863 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:06:45.881+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:06:45.882476+0000) 2024-03-04T20:06:46.863 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:06:46.880+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:06:46.882652+0000) 2024-03-04T20:06:47.864 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:06:47.881+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:06:47.882989+0000) 2024-03-04T20:06:48.865 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:06:48.882+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:06:48.883252+0000) 2024-03-04T20:06:49.864 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:06:49.882+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:06:49.883495+0000) 2024-03-04T20:06:50.865 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:06:50.882+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:06:50.883699+0000) 2024-03-04T20:06:51.864 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:06:51.882+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:06:51.883941+0000) 2024-03-04T20:06:52.865 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:06:52.883+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:06:52.884214+0000) 2024-03-04T20:06:53.865 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:06:53.882+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:06:53.884415+0000) 2024-03-04T20:06:54.865 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:06:54.883+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:06:54.884664+0000) 2024-03-04T20:06:55.866 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:06:55.883+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:06:55.884956+0000) 2024-03-04T20:06:56.866 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:06:56.884+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:06:56.885198+0000) 2024-03-04T20:06:57.866 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:06:57.884+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:06:57.885389+0000) 2024-03-04T20:06:58.866 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:06:58.884+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:06:58.885639+0000) 2024-03-04T20:06:59.866 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:06:59.885+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:06:59.885932+0000) 2024-03-04T20:07:00.867 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:07:00.885+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:07:00.886174+0000) 2024-03-04T20:07:01.867 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:07:01.885+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:07:01.886415+0000) 2024-03-04T20:07:02.867 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:07:02.885+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:07:02.886660+0000) 2024-03-04T20:07:03.868 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:07:03.886+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:07:03.886952+0000) 2024-03-04T20:07:04.867 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:07:04.886+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:07:04.887201+0000) 2024-03-04T20:07:05.868 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:07:05.886+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:07:05.887435+0000) 2024-03-04T20:07:06.868 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:07:06.886+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:07:06.887699+0000) 2024-03-04T20:07:07.868 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:07:07.887+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:07:07.887948+0000) 2024-03-04T20:07:08.210 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:07:08.218 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:07:08.868 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:07:08.887+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:07:08.888195+0000) 2024-03-04T20:07:09.869 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:07:09.887+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:07:09.888438+0000) 2024-03-04T20:07:10.869 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:07:10.887+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:07:10.888677+0000) 2024-03-04T20:07:11.869 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:07:11.888+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:07:11.888925+0000) 2024-03-04T20:07:12.870 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:07:12.888+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:07:12.889193+0000) 2024-03-04T20:07:13.870 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:07:13.888+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:07:13.889440+0000) 2024-03-04T20:07:14.870 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:07:14.888+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:07:14.889713+0000) 2024-03-04T20:07:15.870 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:07:15.889+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:07:15.889989+0000) 2024-03-04T20:07:16.871 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:07:16.889+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:07:16.890243+0000) 2024-03-04T20:07:17.871 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:07:17.889+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:07:17.890526+0000) 2024-03-04T20:07:18.871 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:07:18.890+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:07:18.890800+0000) 2024-03-04T20:07:19.871 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:07:19.890+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:07:19.891086+0000) 2024-03-04T20:07:20.872 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:07:20.890+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:07:20.891324+0000) 2024-03-04T20:07:21.872 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:07:21.891+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:07:21.891562+0000) 2024-03-04T20:07:22.872 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:07:22.891+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:07:22.891868+0000) 2024-03-04T20:07:23.873 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:07:23.891+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:07:23.892123+0000) 2024-03-04T20:07:24.873 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:07:24.891+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:07:24.892380+0000) 2024-03-04T20:07:25.873 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:07:25.892+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:07:25.892618+0000) 2024-03-04T20:07:26.873 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:07:26.892+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:07:26.892925+0000) 2024-03-04T20:07:27.873 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:07:27.892+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:07:27.893196+0000) 2024-03-04T20:07:28.874 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:07:28.893+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:07:28.893457+0000) 2024-03-04T20:07:29.874 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:07:29.893+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:07:29.893760+0000) 2024-03-04T20:07:30.874 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:07:30.893+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:07:30.894008+0000) 2024-03-04T20:07:31.874 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:07:31.893+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:07:31.894300+0000) 2024-03-04T20:07:32.875 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:07:32.894+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:07:32.894581+0000) 2024-03-04T20:07:33.875 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:07:33.894+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:07:33.894860+0000) 2024-03-04T20:07:34.875 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:07:34.894+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:07:34.895115+0000) 2024-03-04T20:07:35.875 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:07:35.895+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:07:35.895362+0000) 2024-03-04T20:07:36.876 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:07:36.895+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:07:36.895602+0000) 2024-03-04T20:07:37.876 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:07:37.895+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:07:37.895898+0000) 2024-03-04T20:07:38.355 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:07:38.364 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:07:38.876 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:07:38.895+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:07:38.896146+0000) 2024-03-04T20:07:39.877 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:07:39.896+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:07:39.896389+0000) 2024-03-04T20:07:40.877 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:07:40.895+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:07:40.896618+0000) 2024-03-04T20:07:41.877 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:07:41.896+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:07:41.896892+0000) 2024-03-04T20:07:42.877 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:07:42.896+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:07:42.897139+0000) 2024-03-04T20:07:43.877 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:07:43.897+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:07:43.897393+0000) 2024-03-04T20:07:44.878 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:07:44.897+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:07:44.897646+0000) 2024-03-04T20:07:45.878 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:07:45.897+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:07:45.897945+0000) 2024-03-04T20:07:46.878 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:07:46.898+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:07:46.898210+0000) 2024-03-04T20:07:47.878 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:07:47.898+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:07:47.898463+0000) 2024-03-04T20:07:48.879 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:07:48.896+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:07:48.898663+0000) 2024-03-04T20:07:49.879 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:07:49.897+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:07:49.898944+0000) 2024-03-04T20:07:50.879 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:07:50.898+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:07:50.899189+0000) 2024-03-04T20:07:51.879 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:07:51.898+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:07:51.899434+0000) 2024-03-04T20:07:52.880 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:07:52.898+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:07:52.899689+0000) 2024-03-04T20:07:53.880 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:07:53.898+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:07:53.899946+0000) 2024-03-04T20:07:54.880 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:07:54.899+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:07:54.900205+0000) 2024-03-04T20:07:55.880 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:07:55.899+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:07:55.900462+0000) 2024-03-04T20:07:56.881 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:07:56.899+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:07:56.900700+0000) 2024-03-04T20:07:57.881 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:07:57.899+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:07:57.900951+0000) 2024-03-04T20:07:58.881 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:07:58.900+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:07:58.901197+0000) 2024-03-04T20:07:59.882 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:07:59.900+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:07:59.901435+0000) 2024-03-04T20:08:00.883 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:08:00.900+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:08:00.901680+0000) 2024-03-04T20:08:01.882 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:08:01.901+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:08:01.901959+0000) 2024-03-04T20:08:02.882 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:08:02.901+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:08:02.902210+0000) 2024-03-04T20:08:03.882 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:08:03.901+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:08:03.902463+0000) 2024-03-04T20:08:04.883 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:08:04.901+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:08:04.902747+0000) 2024-03-04T20:08:05.883 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:08:05.902+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:08:05.902981+0000) 2024-03-04T20:08:06.883 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:08:06.902+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:08:06.903220+0000) 2024-03-04T20:08:07.883 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:08:07.902+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:08:07.903493+0000) 2024-03-04T20:08:08.573 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:08:08.581 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:08:08.883 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:08:08.902+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:08:08.903740+0000) 2024-03-04T20:08:09.883 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:08:09.903+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:08:09.903987+0000) 2024-03-04T20:08:10.884 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:08:10.903+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:08:10.904228+0000) 2024-03-04T20:08:11.884 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:08:11.903+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:08:11.904426+0000) 2024-03-04T20:08:12.884 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:08:12.903+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:08:12.904692+0000) 2024-03-04T20:08:13.885 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:08:13.904+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:08:13.904943+0000) 2024-03-04T20:08:14.885 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:08:14.904+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:08:14.905189+0000) 2024-03-04T20:08:15.885 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:08:15.904+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:08:15.905458+0000) 2024-03-04T20:08:16.885 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:08:16.904+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:08:16.905713+0000) 2024-03-04T20:08:17.889 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:08:17.905+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:08:17.906051+0000) 2024-03-04T20:08:18.887 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:08:18.905+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:08:18.906299+0000) 2024-03-04T20:08:19.886 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:08:19.905+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:08:19.906544+0000) 2024-03-04T20:08:20.886 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:08:20.906+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:08:20.906786+0000) 2024-03-04T20:08:21.887 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:08:21.906+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:08:21.907054+0000) 2024-03-04T20:08:22.887 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:08:22.906+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:08:22.907302+0000) 2024-03-04T20:08:23.887 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:08:23.907+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:08:23.907554+0000) 2024-03-04T20:08:24.887 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:08:24.906+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:08:24.907804+0000) 2024-03-04T20:08:25.888 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:08:25.907+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:08:25.908127+0000) 2024-03-04T20:08:26.888 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:08:26.907+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:08:26.908374+0000) 2024-03-04T20:08:27.888 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:08:27.907+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:08:27.908642+0000) 2024-03-04T20:08:28.888 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:08:28.908+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:08:28.908939+0000) 2024-03-04T20:08:29.888 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:08:29.908+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:08:29.909196+0000) 2024-03-04T20:08:30.889 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:08:30.908+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:08:30.909465+0000) 2024-03-04T20:08:31.889 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:08:31.909+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:08:31.909697+0000) 2024-03-04T20:08:32.889 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:08:32.909+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:08:32.909942+0000) 2024-03-04T20:08:33.889 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:08:33.909+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:08:33.910194+0000) 2024-03-04T20:08:34.890 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:08:34.909+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:08:34.910447+0000) 2024-03-04T20:08:35.890 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:08:35.910+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:08:35.910723+0000) 2024-03-04T20:08:36.890 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:08:36.910+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:08:36.910967+0000) 2024-03-04T20:08:37.890 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:08:37.910+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:08:37.911210+0000) 2024-03-04T20:08:38.815 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:08:38.823 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:08:38.890 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:08:38.910+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:08:38.911421+0000) 2024-03-04T20:08:39.891 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:08:39.911+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:08:39.911663+0000) 2024-03-04T20:08:40.891 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:08:40.911+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:08:40.911982+0000) 2024-03-04T20:08:41.892 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:08:41.911+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:08:41.912234+0000) 2024-03-04T20:08:42.892 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:08:42.912+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:08:42.912490+0000) 2024-03-04T20:08:43.892 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:08:43.912+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:08:43.912755+0000) 2024-03-04T20:08:44.892 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:08:44.912+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:08:44.913039+0000) 2024-03-04T20:08:45.892 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:08:45.913+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:08:45.913297+0000) 2024-03-04T20:08:46.893 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:08:46.913+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:08:46.913555+0000) 2024-03-04T20:08:47.893 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:08:47.913+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:08:47.913796+0000) 2024-03-04T20:08:48.893 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:08:48.913+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:08:48.914079+0000) 2024-03-04T20:08:49.893 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:08:49.913+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:08:49.914265+0000) 2024-03-04T20:08:50.894 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:08:50.913+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:08:50.914511+0000) 2024-03-04T20:08:51.895 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:08:51.913+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:08:51.914754+0000) 2024-03-04T20:08:52.894 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:08:52.913+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:08:52.914999+0000) 2024-03-04T20:08:53.894 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:08:53.914+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:08:53.915268+0000) 2024-03-04T20:08:54.895 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:08:54.914+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:08:54.915518+0000) 2024-03-04T20:08:55.895 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:08:55.914+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:08:55.915758+0000) 2024-03-04T20:08:56.895 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:08:56.914+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:08:56.915996+0000) 2024-03-04T20:08:57.896 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:08:57.915+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:08:57.916232+0000) 2024-03-04T20:08:58.898 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:08:58.915+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:08:58.916480+0000) 2024-03-04T20:08:59.895 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:08:59.914+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:08:59.916737+0000) 2024-03-04T20:09:00.896 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:09:00.916+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:09:00.917035+0000) 2024-03-04T20:09:01.896 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:09:01.916+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:09:01.917279+0000) 2024-03-04T20:09:02.896 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:09:02.916+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:09:02.917540+0000) 2024-03-04T20:09:03.900 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:09:03.917+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:09:03.917813+0000) 2024-03-04T20:09:04.898 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:09:04.917+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:09:04.918163+0000) 2024-03-04T20:09:05.899 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:09:05.917+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:09:05.918425+0000) 2024-03-04T20:09:06.900 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:09:06.917+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:09:06.918676+0000) 2024-03-04T20:09:07.898 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:09:07.918+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:09:07.918953+0000) 2024-03-04T20:09:08.898 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:09:08.918+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:09:08.919207+0000) 2024-03-04T20:09:09.030 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:09:09.039 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:09:09.898 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:09:09.918+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:09:09.919457+0000) 2024-03-04T20:09:10.899 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:09:10.918+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:09:10.919707+0000) 2024-03-04T20:09:11.899 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:09:11.919+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:09:11.919952+0000) 2024-03-04T20:09:12.899 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:09:12.919+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:09:12.920186+0000) 2024-03-04T20:09:13.899 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:09:13.919+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:09:13.920427+0000) 2024-03-04T20:09:14.900 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:09:14.919+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:09:14.920710+0000) 2024-03-04T20:09:15.900 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:09:15.920+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:09:15.921006+0000) 2024-03-04T20:09:16.900 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:09:16.920+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:09:16.921285+0000) 2024-03-04T20:09:17.900 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:09:17.920+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:09:17.921541+0000) 2024-03-04T20:09:18.900 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:09:18.921+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:09:18.921868+0000) 2024-03-04T20:09:19.901 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:09:19.921+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:09:19.922117+0000) 2024-03-04T20:09:20.901 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:09:20.921+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:09:20.922402+0000) 2024-03-04T20:09:21.901 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:09:21.922+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:09:21.922682+0000) 2024-03-04T20:09:22.902 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:09:22.922+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:09:22.922976+0000) 2024-03-04T20:09:23.902 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:09:23.922+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:09:23.923280+0000) 2024-03-04T20:09:24.902 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:09:24.922+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:09:24.923585+0000) 2024-03-04T20:09:25.903 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:09:25.923+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:09:25.923920+0000) 2024-03-04T20:09:26.903 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:09:26.923+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:09:26.924173+0000) 2024-03-04T20:09:27.903 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:09:27.923+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:09:27.924428+0000) 2024-03-04T20:09:28.904 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:09:28.924+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:09:28.924720+0000) 2024-03-04T20:09:29.903 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:09:29.924+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:09:29.924983+0000) 2024-03-04T20:09:30.904 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:09:30.924+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:09:30.925246+0000) 2024-03-04T20:09:31.904 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:09:31.925+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:09:31.925504+0000) 2024-03-04T20:09:32.904 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:09:32.925+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:09:32.925761+0000) 2024-03-04T20:09:33.905 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:09:33.925+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:09:33.926016+0000) 2024-03-04T20:09:34.905 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:09:34.925+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:09:34.926283+0000) 2024-03-04T20:09:35.905 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:09:35.926+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:09:35.926543+0000) 2024-03-04T20:09:36.905 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:09:36.926+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:09:36.926878+0000) 2024-03-04T20:09:37.905 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:09:37.926+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:09:37.927188+0000) 2024-03-04T20:09:38.906 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:09:38.927+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:09:38.927493+0000) 2024-03-04T20:09:39.234 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:09:39.242 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:09:39.906 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:09:39.927+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:09:39.927780+0000) 2024-03-04T20:09:40.906 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:09:40.927+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:09:40.928118+0000) 2024-03-04T20:09:41.907 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:09:41.928+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:09:41.928428+0000) 2024-03-04T20:09:42.907 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:09:42.928+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:09:42.928737+0000) 2024-03-04T20:09:43.907 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:09:43.928+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:09:43.929039+0000) 2024-03-04T20:09:44.911 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:09:44.929+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:09:44.929338+0000) 2024-03-04T20:09:45.908 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:09:45.929+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:09:45.929624+0000) 2024-03-04T20:09:46.908 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:09:46.929+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:09:46.929963+0000) 2024-03-04T20:09:47.909 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:09:47.930+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:09:47.930274+0000) 2024-03-04T20:09:48.909 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:09:48.930+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:09:48.930556+0000) 2024-03-04T20:09:49.909 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:09:49.930+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:09:49.930903+0000) 2024-03-04T20:09:50.910 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:09:50.931+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:09:50.931198+0000) 2024-03-04T20:09:51.910 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:09:51.931+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:09:51.931479+0000) 2024-03-04T20:09:52.910 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:09:52.930+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:09:52.931768+0000) 2024-03-04T20:09:53.910 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:09:53.930+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:09:53.932112+0000) 2024-03-04T20:09:54.911 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:09:54.931+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:09:54.932415+0000) 2024-03-04T20:09:55.911 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:09:55.931+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:09:55.932710+0000) 2024-03-04T20:09:56.911 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:09:56.931+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:09:56.933008+0000) 2024-03-04T20:09:57.912 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:09:57.932+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:09:57.933298+0000) 2024-03-04T20:09:58.912 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:09:58.932+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:09:58.933587+0000) 2024-03-04T20:09:59.912 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:09:59.932+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:09:59.933935+0000) 2024-03-04T20:10:00.912 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:10:00.933+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:10:00.934224+0000) 2024-03-04T20:10:01.913 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:10:01.933+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:10:01.934536+0000) 2024-03-04T20:10:02.913 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:10:02.933+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:10:02.934876+0000) 2024-03-04T20:10:03.913 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:10:03.934+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:10:03.935167+0000) 2024-03-04T20:10:04.914 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:10:04.934+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:10:04.935465+0000) 2024-03-04T20:10:05.914 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:10:05.934+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:10:05.935755+0000) 2024-03-04T20:10:06.914 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:10:06.935+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:10:06.936067+0000) 2024-03-04T20:10:07.914 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:10:07.935+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:10:07.936348+0000) 2024-03-04T20:10:08.915 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:10:08.935+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:10:08.936560+0000) 2024-03-04T20:10:09.430 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:10:09.438 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:10:09.915 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:10:09.936+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:10:09.936911+0000) 2024-03-04T20:10:10.915 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:10:10.936+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:10:10.937226+0000) 2024-03-04T20:10:11.916 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:10:11.936+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:10:11.937521+0000) 2024-03-04T20:10:12.916 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:10:12.937+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:10:12.937867+0000) 2024-03-04T20:10:13.916 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:10:13.937+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:10:13.938162+0000) 2024-03-04T20:10:14.916 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:10:14.937+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:10:14.938457+0000) 2024-03-04T20:10:15.917 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:10:15.937+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:10:15.938749+0000) 2024-03-04T20:10:16.917 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:10:16.938+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:10:16.939049+0000) 2024-03-04T20:10:17.917 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:10:17.938+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:10:17.939340+0000) 2024-03-04T20:10:18.918 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:10:18.938+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:10:18.939626+0000) 2024-03-04T20:10:19.918 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:10:19.939+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:10:19.939967+0000) 2024-03-04T20:10:20.918 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:10:20.939+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:10:20.940268+0000) 2024-03-04T20:10:21.919 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:10:21.939+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:10:21.940557+0000) 2024-03-04T20:10:22.919 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:10:22.940+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:10:22.940891+0000) 2024-03-04T20:10:23.919 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:10:23.940+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:10:23.941169+0000) 2024-03-04T20:10:24.919 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:10:24.940+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:10:24.941470+0000) 2024-03-04T20:10:25.919 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:10:25.940+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:10:25.941707+0000) 2024-03-04T20:10:26.920 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:10:26.941+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:10:26.942001+0000) 2024-03-04T20:10:27.920 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:10:27.941+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:10:27.942302+0000) 2024-03-04T20:10:28.921 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:10:28.942+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:10:28.942592+0000) 2024-03-04T20:10:29.921 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:10:29.942+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:10:29.942934+0000) 2024-03-04T20:10:30.921 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:10:30.942+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:10:30.943191+0000) 2024-03-04T20:10:31.921 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:10:31.942+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:10:31.943488+0000) 2024-03-04T20:10:32.922 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:10:32.943+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:10:32.943783+0000) 2024-03-04T20:10:33.922 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:10:33.943+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:10:33.944121+0000) 2024-03-04T20:10:34.922 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:10:34.943+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:10:34.944433+0000) 2024-03-04T20:10:35.922 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:10:35.944+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:10:35.944730+0000) 2024-03-04T20:10:36.922 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:10:36.944+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:10:36.945011+0000) 2024-03-04T20:10:37.923 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:10:37.944+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:10:37.945304+0000) 2024-03-04T20:10:38.923 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:10:38.945+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:10:38.945599+0000) 2024-03-04T20:10:39.651 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:10:39.659 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:10:39.924 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:10:39.945+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:10:39.945948+0000) 2024-03-04T20:10:40.924 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:10:40.945+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:10:40.946241+0000) 2024-03-04T20:10:41.924 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:10:41.946+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:10:41.946522+0000) 2024-03-04T20:10:42.925 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:10:42.946+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:10:42.946818+0000) 2024-03-04T20:10:43.925 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:10:43.946+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:10:43.947163+0000) 2024-03-04T20:10:44.925 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:10:44.947+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:10:44.947471+0000) 2024-03-04T20:10:45.925 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:10:45.947+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:10:45.947762+0000) 2024-03-04T20:10:46.925 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:10:46.947+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:10:46.948053+0000) 2024-03-04T20:10:47.926 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:10:47.948+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:10:47.948455+0000) 2024-03-04T20:10:48.926 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:10:48.948+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:10:48.948744+0000) 2024-03-04T20:10:49.926 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:10:49.948+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:10:49.949037+0000) 2024-03-04T20:10:50.927 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:10:50.949+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:10:50.949332+0000) 2024-03-04T20:10:51.927 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:10:51.949+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:10:51.949616+0000) 2024-03-04T20:10:52.927 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:10:52.949+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:10:52.949947+0000) 2024-03-04T20:10:53.927 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:10:53.950+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:10:53.950202+0000) 2024-03-04T20:10:54.928 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:10:54.950+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:10:54.950427+0000) 2024-03-04T20:10:55.928 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:10:55.949+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:10:55.950716+0000) 2024-03-04T20:10:56.929 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:10:56.949+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:10:56.951011+0000) 2024-03-04T20:10:57.929 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:10:57.949+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:10:57.951251+0000) 2024-03-04T20:10:58.929 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:10:58.950+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:10:58.951567+0000) 2024-03-04T20:10:59.929 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:10:59.950+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:10:59.951901+0000) 2024-03-04T20:11:00.929 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:11:00.951+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:11:00.952192+0000) 2024-03-04T20:11:01.930 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:11:01.951+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:11:01.952430+0000) 2024-03-04T20:11:02.931 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:11:02.951+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:11:02.952726+0000) 2024-03-04T20:11:03.930 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:11:03.952+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:11:03.953030+0000) 2024-03-04T20:11:04.931 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:11:04.952+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:11:04.953354+0000) 2024-03-04T20:11:05.931 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:11:05.952+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:11:05.953675+0000) 2024-03-04T20:11:06.931 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:11:06.953+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:11:06.953965+0000) 2024-03-04T20:11:07.933 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:11:07.953+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:11:07.954277+0000) 2024-03-04T20:11:08.932 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:11:08.953+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:11:08.954566+0000) 2024-03-04T20:11:09.865 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:11:09.879 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:11:09.932 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:11:09.953+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:11:09.954799+0000) 2024-03-04T20:11:10.933 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:11:10.954+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:11:10.955110+0000) 2024-03-04T20:11:11.933 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:11:11.954+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:11:11.955397+0000) 2024-03-04T20:11:12.933 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:11:12.954+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:11:12.955685+0000) 2024-03-04T20:11:13.934 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:11:13.955+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:11:13.955973+0000) 2024-03-04T20:11:14.934 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:11:14.955+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:11:14.956282+0000) 2024-03-04T20:11:15.934 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:11:15.955+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:11:15.956585+0000) 2024-03-04T20:11:16.935 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:11:16.956+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:11:16.956924+0000) 2024-03-04T20:11:17.934 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:11:17.956+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:11:17.957158+0000) 2024-03-04T20:11:18.935 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:11:18.956+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:11:18.957457+0000) 2024-03-04T20:11:19.935 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:11:19.956+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:11:19.957739+0000) 2024-03-04T20:11:20.935 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:11:20.957+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:11:20.958033+0000) 2024-03-04T20:11:21.936 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:11:21.957+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:11:21.958318+0000) 2024-03-04T20:11:22.936 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:11:22.957+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:11:22.958612+0000) 2024-03-04T20:11:23.936 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:11:23.958+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:11:23.958978+0000) 2024-03-04T20:11:24.937 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:11:24.958+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:11:24.959281+0000) 2024-03-04T20:11:25.937 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:11:25.958+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:11:25.959571+0000) 2024-03-04T20:11:26.937 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:11:26.959+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:11:26.959903+0000) 2024-03-04T20:11:27.937 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:11:27.959+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:11:27.960202+0000) 2024-03-04T20:11:28.938 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:11:28.959+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:11:28.960501+0000) 2024-03-04T20:11:29.938 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:11:29.960+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:11:29.960810+0000) 2024-03-04T20:11:30.940 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:11:30.960+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:11:30.961190+0000) 2024-03-04T20:11:31.939 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:11:31.960+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:11:31.961486+0000) 2024-03-04T20:11:32.939 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:11:32.960+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:11:32.961791+0000) 2024-03-04T20:11:33.940 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:11:33.961+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:11:33.962139+0000) 2024-03-04T20:11:34.940 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:11:34.961+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:11:34.962442+0000) 2024-03-04T20:11:35.940 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:11:35.962+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:11:35.962742+0000) 2024-03-04T20:11:36.940 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:11:36.962+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:11:36.963059+0000) 2024-03-04T20:11:37.940 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:11:37.962+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:11:37.963345+0000) 2024-03-04T20:11:38.941 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:11:38.963+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:11:38.963646+0000) 2024-03-04T20:11:39.941 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:11:39.963+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:11:39.963971+0000) 2024-03-04T20:11:40.052 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:11:40.061 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:11:40.942 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:11:40.963+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:11:40.964312+0000) 2024-03-04T20:11:41.942 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:11:41.964+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:11:41.964605+0000) 2024-03-04T20:11:42.945 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:11:42.964+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:11:42.964933+0000) 2024-03-04T20:11:43.942 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:11:43.964+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:11:43.965256+0000) 2024-03-04T20:11:44.944 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:11:44.965+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:11:44.965581+0000) 2024-03-04T20:11:45.946 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:11:45.965+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:11:45.965922+0000) 2024-03-04T20:11:46.948 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:11:46.965+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:11:46.966209+0000) 2024-03-04T20:11:47.943 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:11:47.966+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:11:47.966519+0000) 2024-03-04T20:11:48.944 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:11:48.966+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:11:48.966877+0000) 2024-03-04T20:11:49.945 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:11:49.966+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:11:49.967161+0000) 2024-03-04T20:11:50.944 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:11:50.967+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:11:50.967453+0000) 2024-03-04T20:11:51.944 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:11:51.967+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:11:51.967745+0000) 2024-03-04T20:11:52.945 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:11:52.967+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:11:52.968034+0000) 2024-03-04T20:11:53.945 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:11:53.968+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:11:53.968332+0000) 2024-03-04T20:11:54.945 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:11:54.968+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:11:54.968646+0000) 2024-03-04T20:11:55.946 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:11:55.968+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:11:55.968971+0000) 2024-03-04T20:11:56.946 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:11:56.969+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:11:56.969258+0000) 2024-03-04T20:11:57.946 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:11:57.968+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:11:57.969582+0000) 2024-03-04T20:11:58.947 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:11:58.968+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:11:58.969925+0000) 2024-03-04T20:11:59.947 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:11:59.969+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:11:59.970225+0000) 2024-03-04T20:12:00.948 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:12:00.969+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:12:00.970516+0000) 2024-03-04T20:12:01.948 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:12:01.969+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:12:01.970799+0000) 2024-03-04T20:12:02.948 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:12:02.970+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:12:02.971166+0000) 2024-03-04T20:12:03.948 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:12:03.970+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:12:03.971465+0000) 2024-03-04T20:12:04.949 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:12:04.970+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:12:04.971773+0000) 2024-03-04T20:12:05.949 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:12:05.971+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:12:05.972068+0000) 2024-03-04T20:12:06.949 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:12:06.971+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:12:06.972362+0000) 2024-03-04T20:12:07.949 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:12:07.971+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:12:07.972663+0000) 2024-03-04T20:12:08.950 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:12:08.972+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:12:08.973015+0000) 2024-03-04T20:12:09.950 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:12:09.972+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:12:09.973319+0000) 2024-03-04T20:12:10.194 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:12:10.203 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:12:10.951 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:12:10.972+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:12:10.973613+0000) 2024-03-04T20:12:11.951 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:12:11.973+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:12:11.973943+0000) 2024-03-04T20:12:12.951 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:12:12.973+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:12:12.974255+0000) 2024-03-04T20:12:13.951 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:12:13.973+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:12:13.974567+0000) 2024-03-04T20:12:14.951 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:12:14.974+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:12:14.974868+0000) 2024-03-04T20:12:15.952 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:12:15.974+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:12:15.975131+0000) 2024-03-04T20:12:16.952 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:12:16.974+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:12:16.975394+0000) 2024-03-04T20:12:17.953 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:12:17.974+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:12:17.975652+0000) 2024-03-04T20:12:18.952 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:12:18.975+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:12:18.975941+0000) 2024-03-04T20:12:19.953 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:12:19.975+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:12:19.976142+0000) 2024-03-04T20:12:20.953 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:12:20.975+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:12:20.976376+0000) 2024-03-04T20:12:21.953 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:12:21.975+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:12:21.976617+0000) 2024-03-04T20:12:22.953 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:12:22.976+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:12:22.976910+0000) 2024-03-04T20:12:23.954 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:12:23.976+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:12:23.977157+0000) 2024-03-04T20:12:24.954 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:12:24.976+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:12:24.977413+0000) 2024-03-04T20:12:25.954 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:12:25.976+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:12:25.977653+0000) 2024-03-04T20:12:26.955 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:12:26.977+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:12:26.977928+0000) 2024-03-04T20:12:27.955 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:12:27.977+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:12:27.978171+0000) 2024-03-04T20:12:28.955 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:12:28.977+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:12:28.978466+0000) 2024-03-04T20:12:29.955 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:12:29.978+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:12:29.978712+0000) 2024-03-04T20:12:30.956 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:12:30.978+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:12:30.978949+0000) 2024-03-04T20:12:31.956 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:12:31.978+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:12:31.979188+0000) 2024-03-04T20:12:32.956 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:12:32.978+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:12:32.979378+0000) 2024-03-04T20:12:33.957 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:12:33.979+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:12:33.979626+0000) 2024-03-04T20:12:34.956 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:12:34.979+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:12:34.979939+0000) 2024-03-04T20:12:35.957 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:12:35.979+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:12:35.980199+0000) 2024-03-04T20:12:36.957 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:12:36.979+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:12:36.980452+0000) 2024-03-04T20:12:37.958 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:12:37.979+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:12:37.980649+0000) 2024-03-04T20:12:38.958 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:12:38.980+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:12:38.980946+0000) 2024-03-04T20:12:39.959 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:12:39.980+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:12:39.981197+0000) 2024-03-04T20:12:40.403 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:12:40.413 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:12:40.958 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:12:40.981+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:12:40.981439+0000) 2024-03-04T20:12:41.960 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:12:41.981+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:12:41.981699+0000) 2024-03-04T20:12:42.958 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:12:42.981+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:12:42.981960+0000) 2024-03-04T20:12:43.960 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:12:43.981+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:12:43.982213+0000) 2024-03-04T20:12:44.959 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:12:44.982+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:12:44.982472+0000) 2024-03-04T20:12:45.959 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:12:45.982+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:12:45.982712+0000) 2024-03-04T20:12:46.959 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:12:46.982+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:12:46.982947+0000) 2024-03-04T20:12:47.960 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:12:47.982+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:12:47.983213+0000) 2024-03-04T20:12:48.960 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:12:48.983+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:12:48.983477+0000) 2024-03-04T20:12:49.960 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:12:49.983+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:12:49.983734+0000) 2024-03-04T20:12:50.960 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:12:50.983+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:12:50.983943+0000) 2024-03-04T20:12:51.961 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:12:51.983+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:12:51.984195+0000) 2024-03-04T20:12:52.961 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:12:52.984+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:12:52.984438+0000) 2024-03-04T20:12:53.961 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:12:53.984+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:12:53.984686+0000) 2024-03-04T20:12:54.961 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:12:54.984+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:12:54.984940+0000) 2024-03-04T20:12:55.962 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:12:55.984+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:12:55.985180+0000) 2024-03-04T20:12:56.965 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:12:56.985+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:12:56.985422+0000) 2024-03-04T20:12:57.962 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:12:57.985+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:12:57.985694+0000) 2024-03-04T20:12:58.962 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:12:58.985+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:12:58.985940+0000) 2024-03-04T20:12:59.962 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:12:59.985+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:12:59.986192+0000) 2024-03-04T20:13:00.963 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:13:00.985+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:13:00.986435+0000) 2024-03-04T20:13:01.963 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:13:01.985+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:13:01.986679+0000) 2024-03-04T20:13:02.963 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:13:02.985+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:13:02.986952+0000) 2024-03-04T20:13:03.965 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:13:03.986+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:13:03.987200+0000) 2024-03-04T20:13:04.964 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:13:04.986+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:13:04.987458+0000) 2024-03-04T20:13:05.964 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:13:05.986+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:13:05.987705+0000) 2024-03-04T20:13:06.964 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:13:06.986+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:13:06.987952+0000) 2024-03-04T20:13:07.964 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:13:07.987+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:13:07.988192+0000) 2024-03-04T20:13:08.965 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:13:08.987+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:13:08.988442+0000) 2024-03-04T20:13:09.965 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:13:09.987+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:13:09.988691+0000) 2024-03-04T20:13:10.640 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:13:10.648 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:13:10.965 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:13:10.988+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:13:10.988939+0000) 2024-03-04T20:13:11.966 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:13:11.988+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:13:11.989246+0000) 2024-03-04T20:13:12.966 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:13:12.988+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:13:12.989504+0000) 2024-03-04T20:13:13.966 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:13:13.988+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:13:13.989746+0000) 2024-03-04T20:13:14.966 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:13:14.989+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:13:14.989997+0000) 2024-03-04T20:13:15.966 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:13:15.989+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:13:15.990237+0000) 2024-03-04T20:13:16.967 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:13:16.989+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:13:16.990482+0000) 2024-03-04T20:13:17.967 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:13:17.989+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:13:17.990727+0000) 2024-03-04T20:13:18.967 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:13:18.990+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:13:18.990969+0000) 2024-03-04T20:13:19.967 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:13:19.990+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:13:19.991212+0000) 2024-03-04T20:13:20.968 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:13:20.990+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:13:20.991478+0000) 2024-03-04T20:13:21.968 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:13:21.990+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:13:21.991718+0000) 2024-03-04T20:13:22.968 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:13:22.991+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:13:22.991954+0000) 2024-03-04T20:13:23.968 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:13:23.991+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:13:23.992195+0000) 2024-03-04T20:13:24.969 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:13:24.991+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:13:24.992449+0000) 2024-03-04T20:13:25.969 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:13:25.991+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:13:25.992687+0000) 2024-03-04T20:13:26.969 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:13:26.992+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:13:26.992924+0000) 2024-03-04T20:13:27.969 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:13:27.992+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:13:27.993161+0000) 2024-03-04T20:13:28.970 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:13:28.992+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:13:28.993400+0000) 2024-03-04T20:13:29.970 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:13:29.993+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:13:29.993684+0000) 2024-03-04T20:13:30.970 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:13:30.993+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:13:30.993925+0000) 2024-03-04T20:13:31.970 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:13:31.993+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:13:31.994162+0000) 2024-03-04T20:13:32.970 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:13:32.993+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:13:32.994413+0000) 2024-03-04T20:13:33.971 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:13:33.993+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:13:33.994666+0000) 2024-03-04T20:13:34.971 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:13:34.994+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:13:34.994939+0000) 2024-03-04T20:13:35.971 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:13:35.994+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:13:35.995198+0000) 2024-03-04T20:13:36.971 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:13:36.994+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:13:36.995435+0000) 2024-03-04T20:13:37.971 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:13:37.995+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:13:37.995682+0000) 2024-03-04T20:13:38.972 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:13:38.995+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:13:38.995948+0000) 2024-03-04T20:13:39.972 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:13:39.995+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:13:39.996200+0000) 2024-03-04T20:13:40.827 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:13:40.835 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:13:40.972 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:13:40.995+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:13:40.996449+0000) 2024-03-04T20:13:41.972 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:13:41.995+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:13:41.996712+0000) 2024-03-04T20:13:42.972 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:13:42.996+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:13:42.996974+0000) 2024-03-04T20:13:43.973 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:13:43.996+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:13:43.997212+0000) 2024-03-04T20:13:44.973 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:13:44.997+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:13:44.997501+0000) 2024-03-04T20:13:45.973 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:13:45.997+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:13:45.997740+0000) 2024-03-04T20:13:46.974 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:13:46.997+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:13:46.998017+0000) 2024-03-04T20:13:47.974 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:13:47.997+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:13:47.998229+0000) 2024-03-04T20:13:48.974 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:13:48.997+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:13:48.998421+0000) 2024-03-04T20:13:49.974 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:13:49.997+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:13:49.998674+0000) 2024-03-04T20:13:50.975 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:13:50.998+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:13:50.998950+0000) 2024-03-04T20:13:51.975 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:13:51.998+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:13:51.999188+0000) 2024-03-04T20:13:52.975 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:13:52.999+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:13:52.999426+0000) 2024-03-04T20:13:53.975 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:13:53.999+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:13:53.999674+0000) 2024-03-04T20:13:54.976 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:13:54.999+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:13:54.999962+0000) 2024-03-04T20:13:55.976 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:13:55.999+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:13:56.000204+0000) 2024-03-04T20:13:56.976 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:13:56.999+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:13:57.000413+0000) 2024-03-04T20:13:57.976 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:13:58.000+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:13:58.000656+0000) 2024-03-04T20:13:58.977 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:13:59.000+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:13:59.000942+0000) 2024-03-04T20:13:59.977 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:14:00.001+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:14:00.001186+0000) 2024-03-04T20:14:00.977 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:14:01.001+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:14:01.001425+0000) 2024-03-04T20:14:01.977 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:14:02.001+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:14:02.001664+0000) 2024-03-04T20:14:02.978 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:14:03.000+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:14:03.001943+0000) 2024-03-04T20:14:03.979 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:14:04.001+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:14:04.002203+0000) 2024-03-04T20:14:04.979 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:14:05.001+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:14:05.002456+0000) 2024-03-04T20:14:05.979 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:14:06.001+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:14:06.002695+0000) 2024-03-04T20:14:06.979 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:14:07.001+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:14:07.002940+0000) 2024-03-04T20:14:07.979 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:14:08.002+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:14:08.003180+0000) 2024-03-04T20:14:08.982 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:14:09.002+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:14:09.003419+0000) 2024-03-04T20:14:09.981 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:14:10.002+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:14:10.003663+0000) 2024-03-04T20:14:10.984 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:14:11.002+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:14:11.003978+0000) 2024-03-04T20:14:11.032 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:14:11.044 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:14:11.981 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:14:12.003+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:14:12.004242+0000) 2024-03-04T20:14:12.981 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:14:13.003+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:14:13.004513+0000) 2024-03-04T20:14:13.981 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:14:14.003+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:14:14.004751+0000) 2024-03-04T20:14:14.980 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:14:15.004+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:14:15.005037+0000) 2024-03-04T20:14:15.981 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:14:16.004+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:14:16.005279+0000) 2024-03-04T20:14:16.981 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:14:17.004+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:14:17.005520+0000) 2024-03-04T20:14:17.981 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:14:18.004+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:14:18.005756+0000) 2024-03-04T20:14:18.981 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:14:19.005+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:14:19.005995+0000) 2024-03-04T20:14:19.982 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:14:20.005+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:14:20.006190+0000) 2024-03-04T20:14:20.982 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:14:21.005+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:14:21.006430+0000) 2024-03-04T20:14:21.982 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:14:22.005+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:14:22.006663+0000) 2024-03-04T20:14:22.982 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:14:23.006+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:14:23.006960+0000) 2024-03-04T20:14:23.982 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:14:24.006+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:14:24.007193+0000) 2024-03-04T20:14:24.983 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:14:25.006+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:14:25.007442+0000) 2024-03-04T20:14:25.983 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:14:26.006+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:14:26.007703+0000) 2024-03-04T20:14:26.983 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:14:27.007+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:14:27.007977+0000) 2024-03-04T20:14:27.983 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:14:28.007+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:14:28.008238+0000) 2024-03-04T20:14:28.984 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:14:29.007+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:14:29.008485+0000) 2024-03-04T20:14:29.984 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:14:30.007+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:14:30.008697+0000) 2024-03-04T20:14:30.986 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:14:31.008+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:14:31.008935+0000) 2024-03-04T20:14:31.984 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:14:32.008+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:14:32.009174+0000) 2024-03-04T20:14:32.985 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:14:33.008+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:14:33.009413+0000) 2024-03-04T20:14:33.985 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:14:34.009+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:14:34.009651+0000) 2024-03-04T20:14:34.985 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:14:35.009+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:14:35.009929+0000) 2024-03-04T20:14:35.985 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:14:36.009+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:14:36.010126+0000) 2024-03-04T20:14:36.985 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:14:37.009+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:14:37.010394+0000) 2024-03-04T20:14:37.986 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:14:38.010+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:14:38.010630+0000) 2024-03-04T20:14:38.986 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:14:39.010+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:14:39.010955+0000) 2024-03-04T20:14:39.986 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:14:40.010+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:14:40.011136+0000) 2024-03-04T20:14:40.987 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:14:41.010+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:14:41.011377+0000) 2024-03-04T20:14:41.204 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:14:41.212 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:14:41.987 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:14:42.011+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:14:42.011622+0000) 2024-03-04T20:14:42.987 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:14:43.011+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:14:43.011931+0000) 2024-03-04T20:14:43.987 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:14:44.011+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:14:44.012191+0000) 2024-03-04T20:14:44.988 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:14:45.012+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:14:45.012469+0000) 2024-03-04T20:14:45.988 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:14:46.012+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:14:46.012715+0000) 2024-03-04T20:14:46.988 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:14:47.012+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:14:47.012955+0000) 2024-03-04T20:14:47.988 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:14:48.012+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:14:48.013192+0000) 2024-03-04T20:14:48.989 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:14:49.013+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:14:49.013430+0000) 2024-03-04T20:14:49.989 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:14:50.013+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:14:50.013669+0000) 2024-03-04T20:14:50.989 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:14:51.013+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:14:51.013946+0000) 2024-03-04T20:14:51.989 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:14:52.013+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:14:52.014137+0000) 2024-03-04T20:14:52.990 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:14:53.014+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:14:53.014375+0000) 2024-03-04T20:14:53.990 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:14:54.013+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:14:54.014563+0000) 2024-03-04T20:14:54.990 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:14:55.014+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:14:55.014816+0000) 2024-03-04T20:14:55.990 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:14:56.014+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:14:56.015113+0000) 2024-03-04T20:14:56.991 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:14:57.015+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:14:57.015354+0000) 2024-03-04T20:14:57.991 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:14:58.015+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:14:58.015646+0000) 2024-03-04T20:14:58.991 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:14:59.015+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:14:59.015937+0000) 2024-03-04T20:14:59.991 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:15:00.015+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:15:00.016146+0000) 2024-03-04T20:15:00.992 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:15:01.016+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:15:01.016393+0000) 2024-03-04T20:15:01.992 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:15:02.016+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:15:02.016626+0000) 2024-03-04T20:15:02.992 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:15:03.016+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:15:03.016931+0000) 2024-03-04T20:15:03.992 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:15:04.017+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:15:04.017176+0000) 2024-03-04T20:15:04.993 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:15:05.015+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:15:05.017419+0000) 2024-03-04T20:15:05.995 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:15:06.016+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:15:06.017659+0000) 2024-03-04T20:15:06.993 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:15:07.016+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:15:07.017951+0000) 2024-03-04T20:15:07.993 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:15:08.017+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:15:08.018202+0000) 2024-03-04T20:15:08.995 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:15:09.016+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:15:09.018394+0000) 2024-03-04T20:15:09.995 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:15:10.017+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:15:10.018635+0000) 2024-03-04T20:15:10.994 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:15:11.017+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:15:11.018928+0000) 2024-03-04T20:15:11.354 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:15:11.365 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:15:11.994 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:15:12.017+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:15:12.019127+0000) 2024-03-04T20:15:12.994 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:15:13.017+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:15:13.019374+0000) 2024-03-04T20:15:13.994 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:15:14.018+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:15:14.019655+0000) 2024-03-04T20:15:14.995 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:15:15.018+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:15:15.019922+0000) 2024-03-04T20:15:15.995 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:15:16.019+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:15:16.020182+0000) 2024-03-04T20:15:16.995 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:15:17.019+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:15:17.020421+0000) 2024-03-04T20:15:17.995 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:15:18.019+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:15:18.020677+0000) 2024-03-04T20:15:18.996 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:15:19.020+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:15:19.020984+0000) 2024-03-04T20:15:19.996 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:15:20.020+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:15:20.021237+0000) 2024-03-04T20:15:20.996 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:15:21.020+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:15:21.021475+0000) 2024-03-04T20:15:21.997 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:15:22.020+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:15:22.021716+0000) 2024-03-04T20:15:22.997 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:15:23.021+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:15:23.021954+0000) 2024-03-04T20:15:23.997 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:15:24.021+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:15:24.022189+0000) 2024-03-04T20:15:24.998 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:15:25.021+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:15:25.022443+0000) 2024-03-04T20:15:25.998 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:15:26.021+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:15:26.022682+0000) 2024-03-04T20:15:26.998 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:15:27.022+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:15:27.022950+0000) 2024-03-04T20:15:27.998 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:15:28.022+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:15:28.023188+0000) 2024-03-04T20:15:28.998 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:15:29.022+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:15:29.023426+0000) 2024-03-04T20:15:29.998 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:15:30.022+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:15:30.023672+0000) 2024-03-04T20:15:30.999 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:15:31.023+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:15:31.023945+0000) 2024-03-04T20:15:31.999 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:15:32.023+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:15:32.024189+0000) 2024-03-04T20:15:32.999 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:15:33.023+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:15:33.024426+0000) 2024-03-04T20:15:33.999 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:15:34.024+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:15:34.024670+0000) 2024-03-04T20:15:35.000 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:15:35.024+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:15:35.024960+0000) 2024-03-04T20:15:36.000 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:15:36.024+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:15:36.025223+0000) 2024-03-04T20:15:37.000 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:15:37.024+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:15:37.025475+0000) 2024-03-04T20:15:38.000 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:15:38.024+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:15:38.025708+0000) 2024-03-04T20:15:39.000 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:15:39.025+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:15:39.025939+0000) 2024-03-04T20:15:40.001 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:15:40.025+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:15:40.026179+0000) 2024-03-04T20:15:41.001 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:15:41.025+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:15:41.026425+0000) 2024-03-04T20:15:41.522 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:15:41.530 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:15:42.001 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:15:42.026+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:15:42.026670+0000) 2024-03-04T20:15:43.001 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:15:43.026+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:15:43.026929+0000) 2024-03-04T20:15:44.002 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:15:44.026+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:15:44.027173+0000) 2024-03-04T20:15:45.002 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:15:45.026+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:15:45.027424+0000) 2024-03-04T20:15:46.002 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:15:46.027+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:15:46.027663+0000) 2024-03-04T20:15:47.003 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:15:47.027+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:15:47.027939+0000) 2024-03-04T20:15:48.003 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:15:48.027+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:15:48.028179+0000) 2024-03-04T20:15:49.003 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:15:49.028+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:15:49.028422+0000) 2024-03-04T20:15:50.004 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:15:50.028+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:15:50.028660+0000) 2024-03-04T20:15:51.004 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:15:51.028+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:15:51.028947+0000) 2024-03-04T20:15:52.004 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:15:52.028+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:15:52.029201+0000) 2024-03-04T20:15:53.004 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:15:53.028+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:15:53.029449+0000) 2024-03-04T20:15:54.004 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:15:54.029+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:15:54.029700+0000) 2024-03-04T20:15:55.004 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:15:55.029+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:15:55.029954+0000) 2024-03-04T20:15:56.004 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:15:56.029+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:15:56.030194+0000) 2024-03-04T20:15:57.005 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:15:57.029+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:15:57.030448+0000) 2024-03-04T20:15:58.005 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:15:58.030+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:15:58.030683+0000) 2024-03-04T20:15:59.007 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:15:59.030+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:15:59.030923+0000) 2024-03-04T20:16:00.005 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:16:00.030+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:16:00.031166+0000) 2024-03-04T20:16:01.006 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:16:01.030+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:16:01.031361+0000) 2024-03-04T20:16:02.006 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:16:02.031+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:16:02.031602+0000) 2024-03-04T20:16:03.006 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:16:03.031+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:16:03.031925+0000) 2024-03-04T20:16:04.007 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:16:04.031+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:16:04.032165+0000) 2024-03-04T20:16:05.007 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:16:05.032+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:16:05.032417+0000) 2024-03-04T20:16:06.007 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:16:06.031+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:16:06.032670+0000) 2024-03-04T20:16:07.007 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:16:07.031+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:16:07.032945+0000) 2024-03-04T20:16:08.008 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:16:08.031+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:16:08.033135+0000) 2024-03-04T20:16:09.008 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:16:09.032+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:16:09.033372+0000) 2024-03-04T20:16:10.008 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:16:10.032+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:16:10.033613+0000) 2024-03-04T20:16:11.008 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:16:11.032+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:16:11.033910+0000) 2024-03-04T20:16:11.659 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:16:11.669 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:16:12.008 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:16:12.033+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:16:12.034161+0000) 2024-03-04T20:16:13.009 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:16:13.033+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:16:13.034421+0000) 2024-03-04T20:16:14.009 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:16:14.033+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:16:14.034689+0000) 2024-03-04T20:16:15.009 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:16:15.033+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:16:15.034961+0000) 2024-03-04T20:16:16.010 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:16:16.034+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:16:16.035202+0000) 2024-03-04T20:16:17.011 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:16:17.033+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:16:17.035427+0000) 2024-03-04T20:16:18.010 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:16:18.034+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:16:18.035666+0000) 2024-03-04T20:16:19.010 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:16:19.035+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:16:19.035949+0000) 2024-03-04T20:16:20.010 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:16:20.035+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:16:20.036201+0000) 2024-03-04T20:16:21.011 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:16:21.035+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:16:21.036440+0000) 2024-03-04T20:16:22.011 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:16:22.035+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:16:22.036683+0000) 2024-03-04T20:16:23.011 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:16:23.036+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:16:23.036950+0000) 2024-03-04T20:16:24.011 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:16:24.036+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:16:24.037130+0000) 2024-03-04T20:16:25.012 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:16:25.036+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:16:25.037385+0000) 2024-03-04T20:16:26.012 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:16:26.036+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:16:26.037623+0000) 2024-03-04T20:16:27.012 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:16:27.037+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:16:27.037904+0000) 2024-03-04T20:16:28.012 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:16:28.037+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:16:28.038149+0000) 2024-03-04T20:16:29.013 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:16:29.037+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:16:29.038391+0000) 2024-03-04T20:16:30.013 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:16:30.037+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:16:30.038636+0000) 2024-03-04T20:16:31.013 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:16:31.038+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:16:31.038925+0000) 2024-03-04T20:16:32.013 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:16:32.038+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:16:32.039170+0000) 2024-03-04T20:16:33.013 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:16:33.038+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:16:33.039409+0000) 2024-03-04T20:16:34.014 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:16:34.038+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:16:34.039702+0000) 2024-03-04T20:16:35.014 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:16:35.039+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:16:35.039961+0000) 2024-03-04T20:16:36.014 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:16:36.039+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:16:36.040214+0000) 2024-03-04T20:16:37.015 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:16:37.039+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:16:37.040449+0000) 2024-03-04T20:16:38.015 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:16:38.040+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:16:38.040692+0000) 2024-03-04T20:16:39.015 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:16:39.040+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:16:39.040965+0000) 2024-03-04T20:16:40.015 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:16:40.040+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:16:40.041210+0000) 2024-03-04T20:16:41.016 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:16:41.040+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:16:41.041454+0000) 2024-03-04T20:16:41.872 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:16:41.881 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:16:42.033 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:16:42.041+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:16:42.041702+0000) 2024-03-04T20:16:43.016 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:16:43.041+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:16:43.041966+0000) 2024-03-04T20:16:44.016 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:16:44.041+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:16:44.042230+0000) 2024-03-04T20:16:45.017 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:16:45.042+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:16:45.042525+0000) 2024-03-04T20:16:46.017 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:16:46.041+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:16:46.042793+0000) 2024-03-04T20:16:47.017 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:16:47.042+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:16:47.043064+0000) 2024-03-04T20:16:48.017 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:16:48.042+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:16:48.043303+0000) 2024-03-04T20:16:49.017 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:16:49.043+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:16:49.043543+0000) 2024-03-04T20:16:50.018 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:16:50.043+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:16:50.043794+0000) 2024-03-04T20:16:51.018 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:16:51.043+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:16:51.044067+0000) 2024-03-04T20:16:52.018 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:16:52.043+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:16:52.044308+0000) 2024-03-04T20:16:53.018 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:16:53.043+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:16:53.044495+0000) 2024-03-04T20:16:54.019 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:16:54.044+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:16:54.044731+0000) 2024-03-04T20:16:55.019 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:16:55.044+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:16:55.044993+0000) 2024-03-04T20:16:56.019 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:16:56.044+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:16:56.045243+0000) 2024-03-04T20:16:57.019 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:16:57.045+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:16:57.045508+0000) 2024-03-04T20:16:58.020 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:16:58.044+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:16:58.045697+0000) 2024-03-04T20:16:59.020 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:16:59.045+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:16:59.045931+0000) 2024-03-04T20:17:00.020 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:17:00.045+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:17:00.046189+0000) 2024-03-04T20:17:01.020 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:17:01.046+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:17:01.046439+0000) 2024-03-04T20:17:02.021 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:17:02.046+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:17:02.046678+0000) 2024-03-04T20:17:03.021 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:17:03.046+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:17:03.046960+0000) 2024-03-04T20:17:04.021 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:17:04.046+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:17:04.047152+0000) 2024-03-04T20:17:05.021 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:17:05.046+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:17:05.047432+0000) 2024-03-04T20:17:06.021 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:17:06.047+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:17:06.047683+0000) 2024-03-04T20:17:07.022 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:17:07.047+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:17:07.047928+0000) 2024-03-04T20:17:08.022 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:17:08.048+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:17:08.048169+0000) 2024-03-04T20:17:09.022 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:17:09.047+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:17:09.048410+0000) 2024-03-04T20:17:10.023 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:17:10.047+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:17:10.048658+0000) 2024-03-04T20:17:11.023 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:17:11.047+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:17:11.048944+0000) 2024-03-04T20:17:12.023 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:17:12.048+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:17:12.049189+0000) 2024-03-04T20:17:12.094 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:17:12.103 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:17:13.023 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:17:13.048+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:17:13.049459+0000) 2024-03-04T20:17:14.023 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:17:14.048+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:17:14.049713+0000) 2024-03-04T20:17:15.024 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:17:15.048+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:17:15.049999+0000) 2024-03-04T20:17:16.024 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:17:16.049+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:17:16.050273+0000) 2024-03-04T20:17:17.024 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:17:17.049+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:17:17.050510+0000) 2024-03-04T20:17:18.024 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:17:18.049+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:17:18.050752+0000) 2024-03-04T20:17:19.025 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:17:19.050+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:17:19.051023+0000) 2024-03-04T20:17:20.025 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:17:20.050+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:17:20.051227+0000) 2024-03-04T20:17:21.025 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:17:21.050+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:17:21.051480+0000) 2024-03-04T20:17:22.026 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:17:22.050+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:17:22.051799+0000) 2024-03-04T20:17:23.026 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:17:23.051+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:17:23.052102+0000) 2024-03-04T20:17:24.026 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:17:24.051+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:17:24.052364+0000) 2024-03-04T20:17:25.026 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:17:25.051+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:17:25.052649+0000) 2024-03-04T20:17:26.026 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:17:26.052+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:17:26.052950+0000) 2024-03-04T20:17:27.027 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:17:27.052+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:17:27.053206+0000) 2024-03-04T20:17:28.027 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:17:28.052+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:17:28.053463+0000) 2024-03-04T20:17:29.027 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:17:29.052+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:17:29.053714+0000) 2024-03-04T20:17:30.028 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:17:30.053+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:17:30.053978+0000) 2024-03-04T20:17:31.028 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:17:31.053+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:17:31.054245+0000) 2024-03-04T20:17:32.028 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:17:32.053+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:17:32.054502+0000) 2024-03-04T20:17:33.028 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:17:33.054+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:17:33.054764+0000) 2024-03-04T20:17:34.029 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:17:34.054+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:17:34.055021+0000) 2024-03-04T20:17:35.029 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:17:35.054+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:17:35.055292+0000) 2024-03-04T20:17:36.029 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:17:36.054+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:17:36.055554+0000) 2024-03-04T20:17:37.029 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:17:37.055+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:17:37.055812+0000) 2024-03-04T20:17:38.030 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:17:38.055+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:17:38.056113+0000) 2024-03-04T20:17:39.030 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:17:39.055+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:17:39.056366+0000) 2024-03-04T20:17:40.030 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:17:40.055+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:17:40.056596+0000) 2024-03-04T20:17:41.030 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:17:41.056+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:17:41.056916+0000) 2024-03-04T20:17:42.030 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:17:42.056+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:17:42.057205+0000) 2024-03-04T20:17:42.232 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:17:42.239 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:17:43.031 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:17:43.056+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:17:43.057480+0000) 2024-03-04T20:17:44.031 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:17:44.057+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:17:44.057737+0000) 2024-03-04T20:17:45.031 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:17:45.057+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:17:45.058016+0000) 2024-03-04T20:17:46.032 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:17:46.057+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:17:46.058282+0000) 2024-03-04T20:17:47.032 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:17:47.057+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:17:47.058486+0000) 2024-03-04T20:17:48.032 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:17:48.058+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:17:48.058744+0000) 2024-03-04T20:17:49.033 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:17:49.058+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:17:49.059011+0000) 2024-03-04T20:17:50.033 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:17:50.058+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:17:50.059257+0000) 2024-03-04T20:17:51.033 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:17:51.059+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:17:51.059523+0000) 2024-03-04T20:17:52.033 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:17:52.059+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:17:52.059780+0000) 2024-03-04T20:17:53.033 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:17:53.059+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:17:53.060067+0000) 2024-03-04T20:17:54.033 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:17:54.059+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:17:54.060310+0000) 2024-03-04T20:17:55.034 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:17:55.060+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:17:55.060566+0000) 2024-03-04T20:17:56.034 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:17:56.060+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:17:56.060869+0000) 2024-03-04T20:17:57.034 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:17:57.060+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:17:57.061098+0000) 2024-03-04T20:17:58.035 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:17:58.061+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:17:58.061332+0000) 2024-03-04T20:17:59.035 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:17:59.061+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:17:59.061568+0000) 2024-03-04T20:18:00.035 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:18:00.061+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:18:00.061883+0000) 2024-03-04T20:18:01.035 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:18:01.061+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:18:01.062133+0000) 2024-03-04T20:18:02.036 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:18:02.062+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:18:02.062376+0000) 2024-03-04T20:18:03.036 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:18:03.061+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:18:03.062669+0000) 2024-03-04T20:18:04.036 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:18:04.062+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:18:04.062957+0000) 2024-03-04T20:18:05.037 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:18:05.063+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:18:05.063273+0000) 2024-03-04T20:18:06.037 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:18:06.062+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:18:06.063591+0000) 2024-03-04T20:18:07.037 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:18:07.063+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:18:07.063942+0000) 2024-03-04T20:18:08.038 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:18:08.063+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:18:08.064182+0000) 2024-03-04T20:18:09.038 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:18:09.064+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:18:09.064477+0000) 2024-03-04T20:18:10.038 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:18:10.064+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:18:10.064791+0000) 2024-03-04T20:18:11.038 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:18:11.064+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:18:11.065150+0000) 2024-03-04T20:18:12.039 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:18:12.064+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:18:12.065432+0000) 2024-03-04T20:18:12.453 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:18:12.462 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:18:13.039 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:18:13.064+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:18:13.065719+0000) 2024-03-04T20:18:14.039 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:18:14.064+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:18:14.066004+0000) 2024-03-04T20:18:15.039 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:18:15.065+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:18:15.066307+0000) 2024-03-04T20:18:16.040 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:18:16.065+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:18:16.066598+0000) 2024-03-04T20:18:17.040 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:18:17.065+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:18:17.066924+0000) 2024-03-04T20:18:18.040 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:18:18.066+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:18:18.067220+0000) 2024-03-04T20:18:19.041 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:18:19.066+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:18:19.067497+0000) 2024-03-04T20:18:20.041 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:18:20.066+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:18:20.067785+0000) 2024-03-04T20:18:21.041 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:18:21.067+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:18:21.068214+0000) 2024-03-04T20:18:22.042 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:18:22.067+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:18:22.068524+0000) 2024-03-04T20:18:23.042 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:18:23.068+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:18:23.068873+0000) 2024-03-04T20:18:24.042 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:18:24.068+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:18:24.069155+0000) 2024-03-04T20:18:25.043 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:18:25.068+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:18:25.069465+0000) 2024-03-04T20:18:26.043 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:18:26.068+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:18:26.069763+0000) 2024-03-04T20:18:27.043 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:18:27.069+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:18:27.070061+0000) 2024-03-04T20:18:28.044 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:18:28.069+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:18:28.070361+0000) 2024-03-04T20:18:29.044 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:18:29.069+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:18:29.070601+0000) 2024-03-04T20:18:30.044 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:18:30.070+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:18:30.070932+0000) 2024-03-04T20:18:31.044 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:18:31.070+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:18:31.071228+0000) 2024-03-04T20:18:32.044 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:18:32.070+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:18:32.071485+0000) 2024-03-04T20:18:33.045 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:18:33.070+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:18:33.071714+0000) 2024-03-04T20:18:34.045 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:18:34.071+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:18:34.071959+0000) 2024-03-04T20:18:35.045 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:18:35.071+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:18:35.072266+0000) 2024-03-04T20:18:36.046 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:18:36.071+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:18:36.072571+0000) 2024-03-04T20:18:37.046 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:18:37.072+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:18:37.072925+0000) 2024-03-04T20:18:38.049 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:18:38.072+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:18:38.073198+0000) 2024-03-04T20:18:39.047 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:18:39.072+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:18:39.073494+0000) 2024-03-04T20:18:40.047 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:18:40.073+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:18:40.073809+0000) 2024-03-04T20:18:41.047 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:18:41.073+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:18:41.074133+0000) 2024-03-04T20:18:42.048 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:18:42.073+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:18:42.074367+0000) 2024-03-04T20:18:42.659 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:18:42.671 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:18:43.047 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:18:43.073+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:18:43.074593+0000) 2024-03-04T20:18:44.048 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:18:44.074+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:18:44.074878+0000) 2024-03-04T20:18:45.048 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:18:45.074+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:18:45.075125+0000) 2024-03-04T20:18:46.048 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:18:46.074+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:18:46.075309+0000) 2024-03-04T20:18:47.049 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:18:47.075+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:18:47.075552+0000) 2024-03-04T20:18:48.054 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:18:48.074+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:18:48.075806+0000) 2024-03-04T20:18:49.051 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:18:49.075+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:18:49.076091+0000) 2024-03-04T20:18:50.050 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:18:50.075+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:18:50.076330+0000) 2024-03-04T20:18:51.050 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:18:51.075+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:18:51.076510+0000) 2024-03-04T20:18:52.050 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:18:52.076+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:18:52.076754+0000) 2024-03-04T20:18:53.051 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:18:53.076+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:18:53.077015+0000) 2024-03-04T20:18:54.050 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:18:54.076+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:18:54.077259+0000) 2024-03-04T20:18:55.050 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:18:55.076+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:18:55.077459+0000) 2024-03-04T20:18:56.050 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:18:56.077+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:18:56.077700+0000) 2024-03-04T20:18:57.051 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:18:57.077+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:18:57.077938+0000) 2024-03-04T20:18:58.051 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:18:58.077+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:18:58.078133+0000) 2024-03-04T20:18:59.052 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:18:59.078+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:18:59.078364+0000) 2024-03-04T20:19:00.051 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:19:00.077+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:19:00.078559+0000) 2024-03-04T20:19:01.052 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:19:01.078+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:19:01.078802+0000) 2024-03-04T20:19:02.052 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:19:02.078+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:19:02.079074+0000) 2024-03-04T20:19:03.052 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:19:03.078+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:19:03.079266+0000) 2024-03-04T20:19:04.052 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:19:04.078+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:19:04.079471+0000) 2024-03-04T20:19:05.053 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:19:05.079+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:19:05.079716+0000) 2024-03-04T20:19:06.053 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:19:06.079+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:19:06.079985+0000) 2024-03-04T20:19:07.053 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:19:07.080+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:19:07.080246+0000) 2024-03-04T20:19:08.053 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:19:08.079+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:19:08.080444+0000) 2024-03-04T20:19:09.053 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:19:09.080+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:19:09.080740+0000) 2024-03-04T20:19:10.054 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:19:10.080+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:19:10.080976+0000) 2024-03-04T20:19:11.054 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:19:11.080+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:19:11.081207+0000) 2024-03-04T20:19:12.054 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:19:12.081+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:19:12.081446+0000) 2024-03-04T20:19:12.813 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:19:12.822 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:19:13.055 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:19:13.081+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:19:13.081731+0000) 2024-03-04T20:19:14.055 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:19:14.080+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:19:14.081994+0000) 2024-03-04T20:19:15.055 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:19:15.081+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:19:15.082276+0000) 2024-03-04T20:19:16.055 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:19:16.081+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:19:16.082562+0000) 2024-03-04T20:19:17.056 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:19:17.081+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:19:17.082905+0000) 2024-03-04T20:19:18.056 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:19:18.082+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:19:18.083202+0000) 2024-03-04T20:19:19.056 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:19:19.081+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:19:19.083493+0000) 2024-03-04T20:19:20.057 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:19:20.082+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:19:20.083817+0000) 2024-03-04T20:19:21.057 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:19:21.083+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:19:21.084190+0000) 2024-03-04T20:19:22.057 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:19:22.083+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:19:22.084497+0000) 2024-03-04T20:19:23.057 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:19:23.083+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:19:23.084796+0000) 2024-03-04T20:19:24.058 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:19:24.084+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:19:24.085129+0000) 2024-03-04T20:19:25.058 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:19:25.084+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:19:25.085434+0000) 2024-03-04T20:19:26.058 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:19:26.084+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:19:26.085719+0000) 2024-03-04T20:19:27.058 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:19:27.085+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:19:27.086004+0000) 2024-03-04T20:19:28.059 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:19:28.085+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:19:28.086253+0000) 2024-03-04T20:19:29.059 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:19:29.085+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:19:29.086544+0000) 2024-03-04T20:19:30.062 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:19:30.086+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:19:30.086870+0000) 2024-03-04T20:19:31.060 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:19:31.086+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:19:31.087153+0000) 2024-03-04T20:19:32.060 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:19:32.086+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:19:32.087456+0000) 2024-03-04T20:19:33.060 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:19:33.086+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:19:33.087767+0000) 2024-03-04T20:19:34.060 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:19:34.087+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:19:34.087976+0000) 2024-03-04T20:19:35.061 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:19:35.087+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:19:35.088275+0000) 2024-03-04T20:19:36.061 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:19:36.087+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:19:36.088561+0000) 2024-03-04T20:19:37.061 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:19:37.088+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:19:37.088880+0000) 2024-03-04T20:19:38.061 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:19:38.088+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:19:38.089176+0000) 2024-03-04T20:19:39.062 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:19:39.088+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:19:39.089469+0000) 2024-03-04T20:19:40.062 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:19:40.089+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:19:40.089766+0000) 2024-03-04T20:19:41.062 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:19:41.089+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:19:41.090062+0000) 2024-03-04T20:19:42.063 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:19:42.089+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:19:42.090298+0000) 2024-03-04T20:19:43.026 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:19:43.035 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:19:43.063 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:19:43.089+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:19:43.090493+0000) 2024-03-04T20:19:44.063 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:19:44.090+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:19:44.090749+0000) 2024-03-04T20:19:45.063 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:19:45.090+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:19:45.091038+0000) 2024-03-04T20:19:46.065 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:19:46.090+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:19:46.091327+0000) 2024-03-04T20:19:47.064 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:19:47.091+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:19:47.091625+0000) 2024-03-04T20:19:48.064 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:19:48.091+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:19:48.091954+0000) 2024-03-04T20:19:49.065 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:19:49.091+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:19:49.092258+0000) 2024-03-04T20:19:50.065 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:19:50.091+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:19:50.092542+0000) 2024-03-04T20:19:51.065 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:19:51.092+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:19:51.092873+0000) 2024-03-04T20:19:52.065 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:19:52.092+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:19:52.093178+0000) 2024-03-04T20:19:53.066 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:19:53.092+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:19:53.093425+0000) 2024-03-04T20:19:54.066 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:19:54.092+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:19:54.093674+0000) 2024-03-04T20:19:55.066 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:19:55.093+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:19:55.093922+0000) 2024-03-04T20:19:56.066 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:19:56.093+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:19:56.094167+0000) 2024-03-04T20:19:57.067 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:19:57.093+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:19:57.094459+0000) 2024-03-04T20:19:58.067 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:19:58.094+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:19:58.094700+0000) 2024-03-04T20:19:59.068 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:19:59.094+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:19:59.094943+0000) 2024-03-04T20:20:00.067 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:20:00.094+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:20:00.095204+0000) 2024-03-04T20:20:01.068 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:20:01.095+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:20:01.095465+0000) 2024-03-04T20:20:02.068 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:20:02.095+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:20:02.095706+0000) 2024-03-04T20:20:03.068 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:20:03.095+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:20:03.095945+0000) 2024-03-04T20:20:04.068 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:20:04.095+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:20:04.096187+0000) 2024-03-04T20:20:05.069 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:20:05.096+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:20:05.096429+0000) 2024-03-04T20:20:06.069 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:20:06.096+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:20:06.096670+0000) 2024-03-04T20:20:07.069 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:20:07.096+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:20:07.096967+0000) 2024-03-04T20:20:08.069 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:20:08.096+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:20:08.097161+0000) 2024-03-04T20:20:09.070 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:20:09.097+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:20:09.097404+0000) 2024-03-04T20:20:10.070 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:20:10.097+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:20:10.097639+0000) 2024-03-04T20:20:11.070 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:20:11.097+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:20:11.097939+0000) 2024-03-04T20:20:12.070 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:20:12.098+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:20:12.098191+0000) 2024-03-04T20:20:13.071 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:20:13.098+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:20:13.098432+0000) 2024-03-04T20:20:13.189 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:20:13.197 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:20:14.071 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:20:14.098+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:20:14.098671+0000) 2024-03-04T20:20:15.071 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:20:15.098+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:20:15.098952+0000) 2024-03-04T20:20:16.072 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:20:16.098+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:20:16.099193+0000) 2024-03-04T20:20:17.072 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:20:17.098+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:20:17.099431+0000) 2024-03-04T20:20:18.072 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:20:18.098+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:20:18.099678+0000) 2024-03-04T20:20:19.072 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:20:19.098+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:20:19.099954+0000) 2024-03-04T20:20:20.073 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:20:20.098+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:20:20.100238+0000) 2024-03-04T20:20:21.073 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:20:21.099+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:20:21.100495+0000) 2024-03-04T20:20:22.073 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:20:22.099+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:20:22.100734+0000) 2024-03-04T20:20:23.073 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:20:23.099+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:20:23.100979+0000) 2024-03-04T20:20:24.074 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:20:24.100+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:20:24.101225+0000) 2024-03-04T20:20:25.073 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:20:25.100+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:20:25.101473+0000) 2024-03-04T20:20:26.074 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:20:26.100+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:20:26.101712+0000) 2024-03-04T20:20:27.074 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:20:27.101+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:20:27.102049+0000) 2024-03-04T20:20:28.074 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:20:28.101+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:20:28.102333+0000) 2024-03-04T20:20:29.075 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:20:29.101+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:20:29.102631+0000) 2024-03-04T20:20:30.075 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:20:30.102+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:20:30.102964+0000) 2024-03-04T20:20:31.075 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:20:31.102+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:20:31.103210+0000) 2024-03-04T20:20:32.076 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:20:32.102+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:20:32.103503+0000) 2024-03-04T20:20:33.076 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:20:33.102+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:20:33.103788+0000) 2024-03-04T20:20:34.076 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:20:34.103+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:20:34.104115+0000) 2024-03-04T20:20:35.077 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:20:35.103+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:20:35.104441+0000) 2024-03-04T20:20:36.077 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:20:36.103+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:20:36.104739+0000) 2024-03-04T20:20:37.077 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:20:37.104+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:20:37.105002+0000) 2024-03-04T20:20:38.077 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:20:38.104+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:20:38.105281+0000) 2024-03-04T20:20:39.078 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:20:39.104+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:20:39.105579+0000) 2024-03-04T20:20:40.078 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:20:40.105+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:20:40.105917+0000) 2024-03-04T20:20:41.078 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:20:41.105+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:20:41.106205+0000) 2024-03-04T20:20:42.078 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:20:42.105+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:20:42.106450+0000) 2024-03-04T20:20:43.079 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:20:43.106+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:20:43.106769+0000) 2024-03-04T20:20:43.343 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:20:43.351 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:20:44.079 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:20:44.106+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:20:44.107056+0000) 2024-03-04T20:20:45.079 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:20:45.106+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:20:45.107398+0000) 2024-03-04T20:20:46.080 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:20:46.107+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:20:46.107692+0000) 2024-03-04T20:20:47.083 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:20:47.107+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:20:47.107979+0000) 2024-03-04T20:20:48.080 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:20:48.107+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:20:48.108289+0000) 2024-03-04T20:20:49.082 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:20:49.107+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:20:49.108599+0000) 2024-03-04T20:20:50.084 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:20:50.108+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:20:50.108924+0000) 2024-03-04T20:20:51.081 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:20:51.108+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:20:51.109221+0000) 2024-03-04T20:20:52.081 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:20:52.108+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:20:52.109525+0000) 2024-03-04T20:20:53.082 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:20:53.109+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:20:53.109812+0000) 2024-03-04T20:20:54.082 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:20:54.109+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:20:54.110161+0000) 2024-03-04T20:20:55.082 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:20:55.109+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:20:55.110480+0000) 2024-03-04T20:20:56.083 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:20:56.110+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:20:56.110904+0000) 2024-03-04T20:20:57.083 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:20:57.110+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:20:57.111184+0000) 2024-03-04T20:20:58.083 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:20:58.110+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:20:58.111441+0000) 2024-03-04T20:20:59.084 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:20:59.111+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:20:59.111700+0000) 2024-03-04T20:21:00.084 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:21:00.111+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:21:00.111999+0000) 2024-03-04T20:21:01.084 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:21:01.111+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:21:01.112257+0000) 2024-03-04T20:21:02.084 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:21:02.112+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:21:02.112497+0000) 2024-03-04T20:21:03.084 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:21:03.112+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:21:03.112733+0000) 2024-03-04T20:21:04.085 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:21:04.112+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:21:04.112972+0000) 2024-03-04T20:21:05.085 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:21:05.112+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:21:05.113216+0000) 2024-03-04T20:21:06.085 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:21:06.113+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:21:06.113450+0000) 2024-03-04T20:21:07.086 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:21:07.113+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:21:07.113690+0000) 2024-03-04T20:21:08.086 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:21:08.113+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:21:08.113933+0000) 2024-03-04T20:21:09.086 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:21:09.113+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:21:09.114194+0000) 2024-03-04T20:21:10.086 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:21:10.113+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:21:10.114375+0000) 2024-03-04T20:21:11.087 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:21:11.113+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:21:11.114578+0000) 2024-03-04T20:21:12.088 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:21:12.114+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:21:12.114873+0000) 2024-03-04T20:21:13.088 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:21:13.114+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:21:13.115110+0000) 2024-03-04T20:21:13.560 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:21:13.569 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:21:14.087 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:21:14.114+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:21:14.115370+0000) 2024-03-04T20:21:15.088 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:21:15.115+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:21:15.115617+0000) 2024-03-04T20:21:16.088 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:21:16.115+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:21:16.115900+0000) 2024-03-04T20:21:17.088 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:21:17.116+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:21:17.116150+0000) 2024-03-04T20:21:18.089 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:21:18.115+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:21:18.116387+0000) 2024-03-04T20:21:19.089 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:21:19.115+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:21:19.116632+0000) 2024-03-04T20:21:20.089 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:21:20.115+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:21:20.116926+0000) 2024-03-04T20:21:21.089 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:21:21.116+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:21:21.117176+0000) 2024-03-04T20:21:22.089 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:21:22.116+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:21:22.117420+0000) 2024-03-04T20:21:23.089 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:21:23.116+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:21:23.117659+0000) 2024-03-04T20:21:24.090 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:21:24.116+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:21:24.117927+0000) 2024-03-04T20:21:25.090 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:21:25.117+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:21:25.118187+0000) 2024-03-04T20:21:26.090 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:21:26.117+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:21:26.118434+0000) 2024-03-04T20:21:27.090 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:21:27.117+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:21:27.118696+0000) 2024-03-04T20:21:28.091 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:21:28.117+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:21:28.118936+0000) 2024-03-04T20:21:29.091 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:21:29.118+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:21:29.119167+0000) 2024-03-04T20:21:30.091 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:21:30.118+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:21:30.119404+0000) 2024-03-04T20:21:31.091 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:21:31.118+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:21:31.119649+0000) 2024-03-04T20:21:32.092 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:21:32.119+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:21:32.119943+0000) 2024-03-04T20:21:33.092 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:21:33.119+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:21:33.120179+0000) 2024-03-04T20:21:34.092 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:21:34.119+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:21:34.120425+0000) 2024-03-04T20:21:35.092 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:21:35.119+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:21:35.120667+0000) 2024-03-04T20:21:36.093 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:21:36.120+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:21:36.120951+0000) 2024-03-04T20:21:37.094 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:21:37.120+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:21:37.121201+0000) 2024-03-04T20:21:38.093 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:21:38.120+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:21:38.121449+0000) 2024-03-04T20:21:39.093 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:21:39.120+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:21:39.121680+0000) 2024-03-04T20:21:40.093 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:21:40.121+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:21:40.121944+0000) 2024-03-04T20:21:41.094 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:21:41.121+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:21:41.122193+0000) 2024-03-04T20:21:42.094 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:21:42.121+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:21:42.122441+0000) 2024-03-04T20:21:43.095 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:21:43.121+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:21:43.122679+0000) 2024-03-04T20:21:43.697 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:21:43.710 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:21:44.095 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:21:44.122+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:21:44.122938+0000) 2024-03-04T20:21:45.096 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:21:45.122+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:21:45.123183+0000) 2024-03-04T20:21:46.095 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:21:46.122+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:21:46.123431+0000) 2024-03-04T20:21:47.095 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:21:47.123+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:21:47.123685+0000) 2024-03-04T20:21:48.095 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:21:48.123+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:21:48.123945+0000) 2024-03-04T20:21:49.095 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:21:49.123+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:21:49.124201+0000) 2024-03-04T20:21:50.096 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:21:50.123+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:21:50.124433+0000) 2024-03-04T20:21:51.096 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:21:51.124+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:21:51.124679+0000) 2024-03-04T20:21:52.096 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:21:52.124+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:21:52.124961+0000) 2024-03-04T20:21:53.096 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:21:53.124+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:21:53.125205+0000) 2024-03-04T20:21:54.097 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:21:54.124+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:21:54.125465+0000) 2024-03-04T20:21:55.097 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:21:55.125+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:21:55.125727+0000) 2024-03-04T20:21:56.097 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:21:56.125+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:21:56.126009+0000) 2024-03-04T20:21:57.098 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:21:57.125+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:21:57.126259+0000) 2024-03-04T20:21:58.098 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:21:58.126+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:21:58.126498+0000) 2024-03-04T20:21:59.098 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:21:59.125+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:21:59.126753+0000) 2024-03-04T20:22:00.098 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:22:00.126+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:22:00.126988+0000) 2024-03-04T20:22:01.099 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:22:01.126+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:22:01.127238+0000) 2024-03-04T20:22:02.099 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:22:02.127+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:22:02.127496+0000) 2024-03-04T20:22:03.099 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:22:03.127+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:22:03.127739+0000) 2024-03-04T20:22:04.099 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:22:04.127+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:22:04.127981+0000) 2024-03-04T20:22:05.099 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:22:05.127+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:22:05.128173+0000) 2024-03-04T20:22:06.100 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:22:06.128+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:22:06.128421+0000) 2024-03-04T20:22:07.100 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:22:07.128+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:22:07.128676+0000) 2024-03-04T20:22:08.100 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:22:08.128+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:22:08.128954+0000) 2024-03-04T20:22:09.100 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:22:09.128+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:22:09.129189+0000) 2024-03-04T20:22:10.101 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:22:10.128+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:22:10.129416+0000) 2024-03-04T20:22:11.101 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:22:11.129+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:22:11.129663+0000) 2024-03-04T20:22:12.101 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:22:12.129+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:22:12.129949+0000) 2024-03-04T20:22:13.102 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:22:13.129+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:22:13.130142+0000) 2024-03-04T20:22:13.849 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:22:13.857 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:22:14.101 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:22:14.129+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:22:14.130423+0000) 2024-03-04T20:22:15.102 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:22:15.129+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:22:15.130694+0000) 2024-03-04T20:22:16.102 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:22:16.130+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:22:16.130951+0000) 2024-03-04T20:22:17.102 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:22:17.131+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:22:17.131194+0000) 2024-03-04T20:22:18.103 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:22:18.131+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:22:18.131436+0000) 2024-03-04T20:22:19.103 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:22:19.131+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:22:19.131684+0000) 2024-03-04T20:22:20.103 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:22:20.131+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:22:20.131927+0000) 2024-03-04T20:22:21.103 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:22:21.131+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:22:21.132179+0000) 2024-03-04T20:22:22.104 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:22:22.131+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:22:22.132429+0000) 2024-03-04T20:22:23.104 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:22:23.131+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:22:23.132670+0000) 2024-03-04T20:22:24.104 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:22:24.131+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:22:24.132930+0000) 2024-03-04T20:22:25.104 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:22:25.131+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:22:25.133125+0000) 2024-03-04T20:22:26.104 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:22:26.132+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:22:26.133376+0000) 2024-03-04T20:22:27.105 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:22:27.132+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:22:27.133629+0000) 2024-03-04T20:22:28.105 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:22:28.132+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:22:28.133909+0000) 2024-03-04T20:22:29.105 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:22:29.133+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:22:29.134142+0000) 2024-03-04T20:22:30.106 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:22:30.133+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:22:30.134375+0000) 2024-03-04T20:22:31.106 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:22:31.133+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:22:31.134619+0000) 2024-03-04T20:22:32.106 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:22:32.134+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:22:32.134954+0000) 2024-03-04T20:22:33.107 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:22:33.134+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:22:33.135193+0000) 2024-03-04T20:22:34.107 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:22:34.134+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:22:34.135430+0000) 2024-03-04T20:22:35.107 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:22:35.134+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:22:35.135675+0000) 2024-03-04T20:22:36.107 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:22:36.135+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:22:36.135954+0000) 2024-03-04T20:22:37.107 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:22:37.135+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:22:37.136243+0000) 2024-03-04T20:22:38.108 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:22:38.135+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:22:38.136482+0000) 2024-03-04T20:22:39.108 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:22:39.135+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:22:39.136714+0000) 2024-03-04T20:22:40.108 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:22:40.136+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:22:40.136948+0000) 2024-03-04T20:22:41.108 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:22:41.136+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:22:41.137185+0000) 2024-03-04T20:22:42.108 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:22:42.136+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:22:42.137437+0000) 2024-03-04T20:22:43.109 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:22:43.136+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:22:43.137679+0000) 2024-03-04T20:22:44.078 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:22:44.087 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:22:44.109 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:22:44.137+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:22:44.137893+0000) 2024-03-04T20:22:45.109 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:22:45.137+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:22:45.138095+0000) 2024-03-04T20:22:46.110 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:22:46.137+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:22:46.138342+0000) 2024-03-04T20:22:47.109 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:22:47.137+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:22:47.138540+0000) 2024-03-04T20:22:48.110 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:22:48.137+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:22:48.138783+0000) 2024-03-04T20:22:49.110 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:22:49.138+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:22:49.139039+0000) 2024-03-04T20:22:50.110 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:22:50.138+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:22:50.139275+0000) 2024-03-04T20:22:51.110 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:22:51.138+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:22:51.139514+0000) 2024-03-04T20:22:52.111 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:22:52.139+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:22:52.139769+0000) 2024-03-04T20:22:53.111 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:22:53.139+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:22:53.139998+0000) 2024-03-04T20:22:54.111 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:22:54.139+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:22:54.140242+0000) 2024-03-04T20:22:55.112 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:22:55.139+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:22:55.140504+0000) 2024-03-04T20:22:56.112 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:22:56.140+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:22:56.140765+0000) 2024-03-04T20:22:57.112 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:22:57.140+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:22:57.141011+0000) 2024-03-04T20:22:58.112 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:22:58.140+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:22:58.141252+0000) 2024-03-04T20:22:59.112 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:22:59.140+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:22:59.141518+0000) 2024-03-04T20:23:00.113 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:23:00.141+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:23:00.141755+0000) 2024-03-04T20:23:01.113 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:23:01.141+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:23:01.141995+0000) 2024-03-04T20:23:02.113 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:23:02.141+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:23:02.142236+0000) 2024-03-04T20:23:03.113 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:23:03.142+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:23:03.142492+0000) 2024-03-04T20:23:04.114 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:23:04.142+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:23:04.142733+0000) 2024-03-04T20:23:05.114 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:23:05.142+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:23:05.142972+0000) 2024-03-04T20:23:06.114 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:23:06.142+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:23:06.143220+0000) 2024-03-04T20:23:07.114 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:23:07.143+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:23:07.143465+0000) 2024-03-04T20:23:08.115 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:23:08.143+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:23:08.143710+0000) 2024-03-04T20:23:09.115 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:23:09.143+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:23:09.143949+0000) 2024-03-04T20:23:10.115 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:23:10.143+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:23:10.144184+0000) 2024-03-04T20:23:11.115 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:23:11.144+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:23:11.144426+0000) 2024-03-04T20:23:12.115 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:23:12.144+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:23:12.144680+0000) 2024-03-04T20:23:13.116 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:23:13.144+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:23:13.144952+0000) 2024-03-04T20:23:14.116 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:23:14.144+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:23:14.145196+0000) 2024-03-04T20:23:14.241 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:23:14.249 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:23:15.116 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:23:15.145+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:23:15.145398+0000) 2024-03-04T20:23:16.116 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:23:16.144+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:23:16.145589+0000) 2024-03-04T20:23:17.117 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:23:17.145+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:23:17.145857+0000) 2024-03-04T20:23:18.117 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:23:18.145+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:23:18.146095+0000) 2024-03-04T20:23:19.117 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:23:19.146+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:23:19.146338+0000) 2024-03-04T20:23:20.117 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:23:20.145+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:23:20.146629+0000) 2024-03-04T20:23:21.118 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:23:21.146+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:23:21.146947+0000) 2024-03-04T20:23:22.118 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:23:22.147+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:23:22.147185+0000) 2024-03-04T20:23:23.118 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:23:23.146+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:23:23.147427+0000) 2024-03-04T20:23:24.118 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:23:24.146+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:23:24.147673+0000) 2024-03-04T20:23:25.119 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:23:25.146+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:23:25.147921+0000) 2024-03-04T20:23:26.119 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:23:26.147+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:23:26.148182+0000) 2024-03-04T20:23:27.119 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:23:27.147+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:23:27.148444+0000) 2024-03-04T20:23:28.119 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:23:28.147+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:23:28.148683+0000) 2024-03-04T20:23:29.120 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:23:29.147+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:23:29.148927+0000) 2024-03-04T20:23:30.120 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:23:30.148+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:23:30.149163+0000) 2024-03-04T20:23:31.120 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:23:31.148+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:23:31.149401+0000) 2024-03-04T20:23:32.120 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:23:32.148+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:23:32.149635+0000) 2024-03-04T20:23:33.121 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:23:33.148+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:23:33.149923+0000) 2024-03-04T20:23:34.121 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:23:34.149+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:23:34.150178+0000) 2024-03-04T20:23:35.121 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:23:35.149+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:23:35.150375+0000) 2024-03-04T20:23:36.121 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:23:36.149+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:23:36.150649+0000) 2024-03-04T20:23:37.122 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:23:37.150+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:23:37.150936+0000) 2024-03-04T20:23:38.122 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:23:38.150+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:23:38.151173+0000) 2024-03-04T20:23:39.122 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:23:39.150+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:23:39.151411+0000) 2024-03-04T20:23:40.123 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:23:40.150+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:23:40.151646+0000) 2024-03-04T20:23:41.122 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:23:41.151+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:23:41.151930+0000) 2024-03-04T20:23:42.123 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:23:42.151+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:23:42.152158+0000) 2024-03-04T20:23:43.123 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:23:43.151+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:23:43.152434+0000) 2024-03-04T20:23:44.123 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:23:44.151+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:23:44.152671+0000) 2024-03-04T20:23:44.429 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:23:44.437 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:23:45.124 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:23:45.152+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:23:45.152950+0000) 2024-03-04T20:23:46.124 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:23:46.152+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:23:46.153190+0000) 2024-03-04T20:23:47.124 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:23:47.152+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:23:47.153425+0000) 2024-03-04T20:23:48.124 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:23:48.152+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:23:48.153666+0000) 2024-03-04T20:23:49.124 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:23:49.153+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:23:49.153928+0000) 2024-03-04T20:23:50.125 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:23:50.153+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:23:50.154192+0000) 2024-03-04T20:23:51.125 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:23:51.153+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:23:51.154432+0000) 2024-03-04T20:23:52.125 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:23:52.154+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:23:52.154693+0000) 2024-03-04T20:23:53.125 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:23:53.154+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:23:53.154933+0000) 2024-03-04T20:23:54.125 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:23:54.154+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:23:54.155175+0000) 2024-03-04T20:23:55.126 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:23:55.154+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:23:55.155372+0000) 2024-03-04T20:23:56.126 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:23:56.155+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:23:56.155616+0000) 2024-03-04T20:23:57.126 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:23:57.155+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:23:57.155921+0000) 2024-03-04T20:23:58.127 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:23:58.155+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:23:58.156158+0000) 2024-03-04T20:23:59.127 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:23:59.155+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:23:59.156396+0000) 2024-03-04T20:24:00.127 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:24:00.156+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:24:00.156629+0000) 2024-03-04T20:24:01.127 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:24:01.156+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:24:01.156907+0000) 2024-03-04T20:24:02.127 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:24:02.156+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:24:02.157201+0000) 2024-03-04T20:24:03.128 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:24:03.156+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:24:03.157448+0000) 2024-03-04T20:24:04.128 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:24:04.157+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:24:04.157694+0000) 2024-03-04T20:24:05.128 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:24:05.157+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:24:05.157938+0000) 2024-03-04T20:24:06.128 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:24:06.157+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:24:06.158177+0000) 2024-03-04T20:24:07.129 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:24:07.158+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:24:07.158414+0000) 2024-03-04T20:24:08.129 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:24:08.158+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:24:08.158653+0000) 2024-03-04T20:24:09.129 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:24:09.158+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:24:09.158914+0000) 2024-03-04T20:24:10.130 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:24:10.158+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:24:10.159149+0000) 2024-03-04T20:24:11.130 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:24:11.159+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:24:11.159390+0000) 2024-03-04T20:24:12.130 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:24:12.159+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:24:12.159631+0000) 2024-03-04T20:24:13.130 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:24:13.159+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:24:13.159913+0000) 2024-03-04T20:24:14.130 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:24:14.159+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:24:14.160154+0000) 2024-03-04T20:24:14.576 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:24:14.584 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:24:15.131 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:24:15.160+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:24:15.160417+0000) 2024-03-04T20:24:16.131 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:24:16.160+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:24:16.160656+0000) 2024-03-04T20:24:17.131 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:24:17.160+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:24:17.160927+0000) 2024-03-04T20:24:18.131 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:24:18.160+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:24:18.161176+0000) 2024-03-04T20:24:19.132 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:24:19.161+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:24:19.161411+0000) 2024-03-04T20:24:20.132 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:24:20.161+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:24:20.161646+0000) 2024-03-04T20:24:21.132 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:24:21.161+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:24:21.161924+0000) 2024-03-04T20:24:22.132 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:24:22.161+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:24:22.162163+0000) 2024-03-04T20:24:23.133 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:24:23.162+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:24:23.162403+0000) 2024-03-04T20:24:24.133 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:24:24.161+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:24:24.162636+0000) 2024-03-04T20:24:25.133 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:24:25.161+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:24:25.162856+0000) 2024-03-04T20:24:26.133 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:24:26.162+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:24:26.163114+0000) 2024-03-04T20:24:27.134 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:24:27.161+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:24:27.163379+0000) 2024-03-04T20:24:28.134 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:24:28.162+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:24:28.163619+0000) 2024-03-04T20:24:29.134 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:24:29.162+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:24:29.163895+0000) 2024-03-04T20:24:30.134 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:24:30.163+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:24:30.164126+0000) 2024-03-04T20:24:31.135 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:24:31.163+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:24:31.164367+0000) 2024-03-04T20:24:32.135 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:24:32.163+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:24:32.164606+0000) 2024-03-04T20:24:33.135 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:24:33.163+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:24:33.164879+0000) 2024-03-04T20:24:34.135 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:24:34.164+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:24:34.165123+0000) 2024-03-04T20:24:35.136 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:24:35.164+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:24:35.165373+0000) 2024-03-04T20:24:36.136 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:24:36.164+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:24:36.165606+0000) 2024-03-04T20:24:37.136 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:24:37.165+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:24:37.165889+0000) 2024-03-04T20:24:38.136 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:24:38.165+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:24:38.166130+0000) 2024-03-04T20:24:39.136 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:24:39.165+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:24:39.166312+0000) 2024-03-04T20:24:40.137 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:24:40.165+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:24:40.166548+0000) 2024-03-04T20:24:41.137 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:24:41.165+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:24:41.166781+0000) 2024-03-04T20:24:42.137 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:24:42.166+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:24:42.167028+0000) 2024-03-04T20:24:43.137 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:24:43.166+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:24:43.167274+0000) 2024-03-04T20:24:44.138 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:24:44.166+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:24:44.167512+0000) 2024-03-04T20:24:44.848 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:24:44.856 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:24:45.138 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:24:45.166+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:24:45.167780+0000) 2024-03-04T20:24:46.138 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:24:46.167+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:24:46.168022+0000) 2024-03-04T20:24:47.138 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:24:47.167+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:24:47.168259+0000) 2024-03-04T20:24:48.138 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:24:48.167+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:24:48.168504+0000) 2024-03-04T20:24:49.139 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:24:49.167+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:24:49.168697+0000) 2024-03-04T20:24:50.139 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:24:50.168+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:24:50.168983+0000) 2024-03-04T20:24:51.139 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:24:51.168+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:24:51.169223+0000) 2024-03-04T20:24:52.140 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:24:52.168+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:24:52.169463+0000) 2024-03-04T20:24:53.140 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:24:53.169+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:24:53.169703+0000) 2024-03-04T20:24:54.140 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:24:54.169+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:24:54.169948+0000) 2024-03-04T20:24:55.141 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:24:55.169+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:24:55.170206+0000) 2024-03-04T20:24:56.141 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:24:56.169+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:24:56.170450+0000) 2024-03-04T20:24:57.141 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:24:57.169+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:24:57.170676+0000) 2024-03-04T20:24:58.141 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:24:58.170+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:24:58.170933+0000) 2024-03-04T20:24:59.141 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:24:59.170+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:24:59.171171+0000) 2024-03-04T20:25:00.141 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:25:00.170+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:25:00.171403+0000) 2024-03-04T20:25:01.142 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:25:01.171+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:25:01.171652+0000) 2024-03-04T20:25:02.142 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:25:02.171+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:25:02.171914+0000) 2024-03-04T20:25:03.142 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:25:03.171+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:25:03.172163+0000) 2024-03-04T20:25:04.142 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:25:04.171+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:25:04.172400+0000) 2024-03-04T20:25:05.142 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:25:05.172+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:25:05.172641+0000) 2024-03-04T20:25:06.143 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:25:06.172+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:25:06.172920+0000) 2024-03-04T20:25:07.143 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:25:07.172+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:25:07.173158+0000) 2024-03-04T20:25:08.143 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:25:08.172+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:25:08.173400+0000) 2024-03-04T20:25:09.143 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:25:09.173+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:25:09.173638+0000) 2024-03-04T20:25:10.144 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:25:10.173+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:25:10.173902+0000) 2024-03-04T20:25:11.144 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:25:11.173+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:25:11.174164+0000) 2024-03-04T20:25:12.144 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:25:12.174+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:25:12.174407+0000) 2024-03-04T20:25:13.145 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:25:13.174+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:25:13.174648+0000) 2024-03-04T20:25:14.145 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:25:14.174+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:25:14.174932+0000) 2024-03-04T20:25:15.060 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:25:15.069 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:25:15.145 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:25:15.174+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:25:15.175193+0000) 2024-03-04T20:25:16.145 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:25:16.175+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:25:16.175445+0000) 2024-03-04T20:25:17.145 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:25:17.175+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:25:17.175689+0000) 2024-03-04T20:25:18.146 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:25:18.175+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:25:18.175980+0000) 2024-03-04T20:25:19.146 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:25:19.175+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:25:19.176215+0000) 2024-03-04T20:25:20.146 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:25:20.176+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:25:20.176446+0000) 2024-03-04T20:25:21.146 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:25:21.176+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:25:21.176686+0000) 2024-03-04T20:25:22.147 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:25:22.176+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:25:22.176943+0000) 2024-03-04T20:25:23.147 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:25:23.176+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:25:23.177183+0000) 2024-03-04T20:25:24.147 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:25:24.177+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:25:24.177423+0000) 2024-03-04T20:25:25.147 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:25:25.177+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:25:25.177663+0000) 2024-03-04T20:25:26.148 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:25:26.177+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:25:26.177955+0000) 2024-03-04T20:25:27.148 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:25:27.177+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:25:27.178221+0000) 2024-03-04T20:25:28.148 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:25:28.176+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:25:28.178489+0000) 2024-03-04T20:25:29.148 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:25:29.177+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:25:29.178726+0000) 2024-03-04T20:25:30.149 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:25:30.177+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:25:30.178957+0000) 2024-03-04T20:25:31.149 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:25:31.178+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:25:31.179197+0000) 2024-03-04T20:25:32.149 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:25:32.178+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:25:32.179471+0000) 2024-03-04T20:25:33.149 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:25:33.177+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:25:33.179728+0000) 2024-03-04T20:25:34.150 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:25:34.178+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:25:34.179966+0000) 2024-03-04T20:25:35.150 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:25:35.178+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:25:35.180151+0000) 2024-03-04T20:25:36.150 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:25:36.179+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:25:36.180393+0000) 2024-03-04T20:25:37.150 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:25:37.179+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:25:37.180633+0000) 2024-03-04T20:25:38.151 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:25:38.180+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:25:38.180971+0000) 2024-03-04T20:25:39.151 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:25:39.180+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:25:39.181209+0000) 2024-03-04T20:25:40.151 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:25:40.180+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:25:40.181452+0000) 2024-03-04T20:25:41.151 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:25:41.180+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:25:41.181704+0000) 2024-03-04T20:25:42.152 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:25:42.181+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:25:42.181942+0000) 2024-03-04T20:25:43.152 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:25:43.181+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:25:43.182188+0000) 2024-03-04T20:25:44.152 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:25:44.181+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:25:44.182430+0000) 2024-03-04T20:25:45.152 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:25:45.181+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:25:45.182674+0000) 2024-03-04T20:25:45.281 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:25:45.289 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:25:46.153 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:25:46.182+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:25:46.182950+0000) 2024-03-04T20:25:47.153 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:25:47.182+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:25:47.183188+0000) 2024-03-04T20:25:48.153 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:25:48.182+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:25:48.183429+0000) 2024-03-04T20:25:49.153 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:25:49.182+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:25:49.183679+0000) 2024-03-04T20:25:50.153 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:25:50.183+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:25:50.183948+0000) 2024-03-04T20:25:51.154 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:25:51.183+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:25:51.184192+0000) 2024-03-04T20:25:52.154 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:25:52.183+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:25:52.184429+0000) 2024-03-04T20:25:53.154 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:25:53.183+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:25:53.184670+0000) 2024-03-04T20:25:54.154 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:25:54.184+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:25:54.184948+0000) 2024-03-04T20:25:55.155 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:25:55.184+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:25:55.185186+0000) 2024-03-04T20:25:56.155 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:25:56.184+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:25:56.185428+0000) 2024-03-04T20:25:57.156 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:25:57.185+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:25:57.185687+0000) 2024-03-04T20:25:58.156 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:25:58.185+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:25:58.185948+0000) 2024-03-04T20:25:59.157 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:25:59.185+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:25:59.186184+0000) 2024-03-04T20:26:00.157 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:26:00.185+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:26:00.186441+0000) 2024-03-04T20:26:01.156 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:26:01.186+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:26:01.186683+0000) 2024-03-04T20:26:02.157 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:26:02.186+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:26:02.186946+0000) 2024-03-04T20:26:03.157 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:26:03.186+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:26:03.187188+0000) 2024-03-04T20:26:04.157 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:26:04.186+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:26:04.187426+0000) 2024-03-04T20:26:05.157 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:26:05.187+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:26:05.187667+0000) 2024-03-04T20:26:06.157 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:26:06.187+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:26:06.187932+0000) 2024-03-04T20:26:07.158 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:26:07.187+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:26:07.188169+0000) 2024-03-04T20:26:08.158 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:26:08.187+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:26:08.188412+0000) 2024-03-04T20:26:09.158 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:26:09.188+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:26:09.188621+0000) 2024-03-04T20:26:10.158 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:26:10.188+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:26:10.188886+0000) 2024-03-04T20:26:11.159 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:26:11.188+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:26:11.189125+0000) 2024-03-04T20:26:12.159 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:26:12.188+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:26:12.189367+0000) 2024-03-04T20:26:13.159 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:26:13.189+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:26:13.189609+0000) 2024-03-04T20:26:14.159 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:26:14.189+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:26:14.189878+0000) 2024-03-04T20:26:15.160 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:26:15.189+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:26:15.190122+0000) 2024-03-04T20:26:15.461 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:26:15.469 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:26:16.160 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:26:16.190+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:26:16.190368+0000) 2024-03-04T20:26:17.160 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:26:17.189+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:26:17.190607+0000) 2024-03-04T20:26:18.160 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:26:18.190+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:26:18.190888+0000) 2024-03-04T20:26:19.161 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:26:19.190+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:26:19.191134+0000) 2024-03-04T20:26:20.161 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:26:20.191+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:26:20.191368+0000) 2024-03-04T20:26:21.161 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:26:21.191+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:26:21.191603+0000) 2024-03-04T20:26:22.162 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:26:22.191+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:26:22.191873+0000) 2024-03-04T20:26:23.162 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:26:23.191+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:26:23.192115+0000) 2024-03-04T20:26:24.162 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:26:24.192+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:26:24.192354+0000) 2024-03-04T20:26:25.162 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:26:25.192+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:26:25.192591+0000) 2024-03-04T20:26:26.162 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:26:26.192+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:26:26.192869+0000) 2024-03-04T20:26:27.163 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:26:27.192+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:26:27.193117+0000) 2024-03-04T20:26:28.163 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:26:28.193+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:26:28.193380+0000) 2024-03-04T20:26:29.163 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:26:29.193+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:26:29.193638+0000) 2024-03-04T20:26:30.163 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:26:30.192+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:26:30.193915+0000) 2024-03-04T20:26:31.164 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:26:31.193+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:26:31.194154+0000) 2024-03-04T20:26:32.164 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:26:32.193+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:26:32.194404+0000) 2024-03-04T20:26:33.164 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:26:33.193+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:26:33.194669+0000) 2024-03-04T20:26:34.164 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:26:34.193+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:26:34.194944+0000) 2024-03-04T20:26:35.164 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:26:35.193+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:26:35.195142+0000) 2024-03-04T20:26:36.165 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:26:36.193+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:26:36.195339+0000) 2024-03-04T20:26:37.165 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:26:37.193+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:26:37.195634+0000) 2024-03-04T20:26:38.165 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:26:38.194+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:26:38.195927+0000) 2024-03-04T20:26:39.165 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:26:39.195+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:26:39.196165+0000) 2024-03-04T20:26:40.166 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:26:40.195+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:26:40.196405+0000) 2024-03-04T20:26:41.166 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:26:41.195+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:26:41.196648+0000) 2024-03-04T20:26:42.166 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:26:42.196+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:26:42.196938+0000) 2024-03-04T20:26:43.167 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:26:43.196+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:26:43.197132+0000) 2024-03-04T20:26:44.167 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:26:44.196+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:26:44.197381+0000) 2024-03-04T20:26:45.167 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:26:45.196+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:26:45.197637+0000) 2024-03-04T20:26:45.667 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:26:45.675 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:26:46.167 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:26:46.197+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:26:46.197938+0000) 2024-03-04T20:26:47.168 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:26:47.197+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:26:47.198204+0000) 2024-03-04T20:26:48.168 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:26:48.197+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:26:48.198471+0000) 2024-03-04T20:26:49.168 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:26:49.197+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:26:49.198695+0000) 2024-03-04T20:26:50.168 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:26:50.198+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:26:50.198929+0000) 2024-03-04T20:26:51.168 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:26:51.198+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:26:51.199165+0000) 2024-03-04T20:26:52.168 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:26:52.198+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:26:52.199417+0000) 2024-03-04T20:26:53.169 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:26:53.198+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:26:53.199660+0000) 2024-03-04T20:26:54.169 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:26:54.199+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:26:54.199947+0000) 2024-03-04T20:26:55.170 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:26:55.199+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:26:55.200191+0000) 2024-03-04T20:26:56.170 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:26:56.199+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:26:56.200432+0000) 2024-03-04T20:26:57.170 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:26:57.199+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:26:57.200679+0000) 2024-03-04T20:26:58.170 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:26:58.200+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:26:58.200953+0000) 2024-03-04T20:26:59.170 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:26:59.200+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:26:59.201210+0000) 2024-03-04T20:27:00.170 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:27:00.200+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:27:00.201443+0000) 2024-03-04T20:27:01.171 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:27:01.201+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:27:01.201685+0000) 2024-03-04T20:27:02.171 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:27:02.201+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:27:02.201965+0000) 2024-03-04T20:27:03.171 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:27:03.201+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:27:03.202234+0000) 2024-03-04T20:27:04.171 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:27:04.201+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:27:04.202435+0000) 2024-03-04T20:27:05.172 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:27:05.202+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:27:05.202675+0000) 2024-03-04T20:27:06.172 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:27:06.202+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:27:06.202957+0000) 2024-03-04T20:27:07.172 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:27:07.202+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:27:07.203204+0000) 2024-03-04T20:27:08.173 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:27:08.202+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:27:08.203408+0000) 2024-03-04T20:27:09.173 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:27:09.203+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:27:09.203656+0000) 2024-03-04T20:27:10.173 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:27:10.203+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:27:10.203926+0000) 2024-03-04T20:27:11.173 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:27:11.203+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:27:11.204160+0000) 2024-03-04T20:27:12.173 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:27:12.203+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:27:12.204403+0000) 2024-03-04T20:27:13.174 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:27:13.204+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:27:13.204659+0000) 2024-03-04T20:27:14.174 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:27:14.204+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:27:14.204951+0000) 2024-03-04T20:27:15.174 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:27:15.204+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:27:15.205184+0000) 2024-03-04T20:27:15.803 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:27:15.811 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:27:16.174 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:27:16.205+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:27:16.205428+0000) 2024-03-04T20:27:17.175 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:27:17.205+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:27:17.205676+0000) 2024-03-04T20:27:18.175 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:27:18.205+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:27:18.205973+0000) 2024-03-04T20:27:19.175 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:27:19.205+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:27:19.206209+0000) 2024-03-04T20:27:20.176 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:27:20.206+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:27:20.206443+0000) 2024-03-04T20:27:21.176 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:27:21.206+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:27:21.206680+0000) 2024-03-04T20:27:22.176 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:27:22.206+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:27:22.206969+0000) 2024-03-04T20:27:23.176 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:27:23.206+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:27:23.207218+0000) 2024-03-04T20:27:24.177 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:27:24.207+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:27:24.207459+0000) 2024-03-04T20:27:25.177 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:27:25.207+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:27:25.207703+0000) 2024-03-04T20:27:26.177 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:27:26.207+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:27:26.207956+0000) 2024-03-04T20:27:27.177 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:27:27.207+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:27:27.208204+0000) 2024-03-04T20:27:28.177 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:27:28.207+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:27:28.208444+0000) 2024-03-04T20:27:29.178 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:27:29.208+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:27:29.208755+0000) 2024-03-04T20:27:30.178 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:27:30.208+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:27:30.209061+0000) 2024-03-04T20:27:31.178 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:27:31.208+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:27:31.209266+0000) 2024-03-04T20:27:32.179 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:27:32.208+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:27:32.209516+0000) 2024-03-04T20:27:33.179 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:27:33.208+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:27:33.209761+0000) 2024-03-04T20:27:34.179 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:27:34.208+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:27:34.210002+0000) 2024-03-04T20:27:35.179 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:27:35.209+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:27:35.210243+0000) 2024-03-04T20:27:36.179 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:27:36.209+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:27:36.210439+0000) 2024-03-04T20:27:37.180 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:27:37.209+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:27:37.210673+0000) 2024-03-04T20:27:38.180 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:27:38.209+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:27:38.210952+0000) 2024-03-04T20:27:39.180 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:27:39.210+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:27:39.211190+0000) 2024-03-04T20:27:40.180 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:27:40.210+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:27:40.211431+0000) 2024-03-04T20:27:41.180 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:27:41.209+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:27:41.211630+0000) 2024-03-04T20:27:42.181 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:27:42.210+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:27:42.211931+0000) 2024-03-04T20:27:43.181 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:27:43.211+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:27:43.212210+0000) 2024-03-04T20:27:44.181 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:27:44.211+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:27:44.212391+0000) 2024-03-04T20:27:45.182 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:27:45.211+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:27:45.212636+0000) 2024-03-04T20:27:46.001 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:27:46.010 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:27:46.182 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:27:46.212+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:27:46.212964+0000) 2024-03-04T20:27:47.182 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:27:47.212+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:27:47.213196+0000) 2024-03-04T20:27:48.182 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:27:48.212+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:27:48.213476+0000) 2024-03-04T20:27:49.183 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:27:49.212+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:27:49.213707+0000) 2024-03-04T20:27:50.183 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:27:50.213+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:27:50.213939+0000) 2024-03-04T20:27:51.183 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:27:51.213+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:27:51.214207+0000) 2024-03-04T20:27:52.183 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:27:52.213+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:27:52.214447+0000) 2024-03-04T20:27:53.184 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:27:53.213+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:27:53.214708+0000) 2024-03-04T20:27:54.184 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:27:54.214+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:27:54.214944+0000) 2024-03-04T20:27:55.184 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:27:55.214+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:27:55.215195+0000) 2024-03-04T20:27:56.185 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:27:56.214+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:27:56.215428+0000) 2024-03-04T20:27:57.185 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:27:57.214+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:27:57.215668+0000) 2024-03-04T20:27:58.185 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:27:58.215+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:27:58.215941+0000) 2024-03-04T20:27:59.185 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:27:59.215+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:27:59.216193+0000) 2024-03-04T20:28:00.185 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:28:00.215+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:28:00.216426+0000) 2024-03-04T20:28:01.186 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:28:01.216+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:28:01.216665+0000) 2024-03-04T20:28:02.186 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:28:02.216+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:28:02.216954+0000) 2024-03-04T20:28:03.186 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:28:03.216+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:28:03.217199+0000) 2024-03-04T20:28:04.186 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:28:04.216+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:28:04.217462+0000) 2024-03-04T20:28:05.186 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:28:05.217+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:28:05.217696+0000) 2024-03-04T20:28:06.187 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:28:06.217+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:28:06.217962+0000) 2024-03-04T20:28:07.187 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:28:07.217+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:28:07.218200+0000) 2024-03-04T20:28:08.187 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:28:08.217+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:28:08.218460+0000) 2024-03-04T20:28:09.187 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:28:09.217+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:28:09.218770+0000) 2024-03-04T20:28:10.188 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:28:10.218+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:28:10.219018+0000) 2024-03-04T20:28:11.188 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:28:11.218+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:28:11.219244+0000) 2024-03-04T20:28:12.188 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:28:12.219+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:28:12.219488+0000) 2024-03-04T20:28:13.189 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:28:13.218+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:28:13.219804+0000) 2024-03-04T20:28:14.189 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:28:14.219+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:28:14.220089+0000) 2024-03-04T20:28:15.189 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:28:15.219+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:28:15.220332+0000) 2024-03-04T20:28:16.147 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:28:16.155 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:28:16.189 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:28:16.219+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:28:16.220612+0000) 2024-03-04T20:28:17.190 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:28:17.220+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:28:17.220937+0000) 2024-03-04T20:28:18.190 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:28:18.220+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:28:18.221227+0000) 2024-03-04T20:28:19.191 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:28:19.221+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:28:19.221523+0000) 2024-03-04T20:28:20.191 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:28:20.221+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:28:20.221776+0000) 2024-03-04T20:28:21.191 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:28:21.221+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:28:21.222010+0000) 2024-03-04T20:28:22.192 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:28:22.221+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:28:22.222244+0000) 2024-03-04T20:28:23.193 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:28:23.222+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:28:23.222484+0000) 2024-03-04T20:28:24.192 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:28:24.222+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:28:24.222723+0000) 2024-03-04T20:28:25.193 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:28:25.222+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:28:25.222963+0000) 2024-03-04T20:28:26.193 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:28:26.222+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:28:26.223195+0000) 2024-03-04T20:28:27.193 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:28:27.222+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:28:27.223419+0000) 2024-03-04T20:28:28.193 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:28:28.223+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:28:28.223707+0000) 2024-03-04T20:28:29.194 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:28:29.223+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:28:29.223995+0000) 2024-03-04T20:28:30.194 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:28:30.224+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:28:30.224241+0000) 2024-03-04T20:28:31.195 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:28:31.224+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:28:31.224478+0000) 2024-03-04T20:28:32.195 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:28:32.224+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:28:32.224732+0000) 2024-03-04T20:28:33.195 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:28:33.224+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:28:33.224970+0000) 2024-03-04T20:28:34.196 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:28:34.224+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:28:34.225210+0000) 2024-03-04T20:28:35.196 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:28:35.224+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:28:35.225445+0000) 2024-03-04T20:28:36.196 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:28:36.224+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:28:36.225684+0000) 2024-03-04T20:28:37.196 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:28:37.224+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:28:37.225961+0000) 2024-03-04T20:28:38.196 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:28:38.224+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:28:38.226192+0000) 2024-03-04T20:28:39.197 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:28:39.225+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:28:39.226433+0000) 2024-03-04T20:28:40.197 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:28:40.225+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:28:40.226670+0000) 2024-03-04T20:28:41.198 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:28:41.225+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:28:41.226949+0000) 2024-03-04T20:28:42.198 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:28:42.225+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:28:42.227142+0000) 2024-03-04T20:28:43.198 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:28:43.226+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:28:43.227378+0000) 2024-03-04T20:28:44.199 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:28:44.226+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:28:44.227623+0000) 2024-03-04T20:28:45.199 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:28:45.226+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:28:45.227916+0000) 2024-03-04T20:28:46.199 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:28:46.227+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:28:46.228159+0000) 2024-03-04T20:28:46.336 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:28:46.344 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:28:47.199 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:28:47.227+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:28:47.228401+0000) 2024-03-04T20:28:48.200 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:28:48.227+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:28:48.228653+0000) 2024-03-04T20:28:49.200 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:28:49.228+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:28:49.228933+0000) 2024-03-04T20:28:50.201 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:28:50.228+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:28:50.229176+0000) 2024-03-04T20:28:51.200 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:28:51.228+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:28:51.229428+0000) 2024-03-04T20:28:52.201 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:28:52.228+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:28:52.229701+0000) 2024-03-04T20:28:53.201 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:28:53.229+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:28:53.229943+0000) 2024-03-04T20:28:54.201 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:28:54.229+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:28:54.230181+0000) 2024-03-04T20:28:55.202 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:28:55.229+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:28:55.230421+0000) 2024-03-04T20:28:56.202 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:28:56.229+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:28:56.230669+0000) 2024-03-04T20:28:57.202 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:28:57.230+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:28:57.230955+0000) 2024-03-04T20:28:58.203 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:28:58.230+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:28:58.231217+0000) 2024-03-04T20:28:59.203 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:28:59.230+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:28:59.231475+0000) 2024-03-04T20:29:00.203 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:29:00.231+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:29:00.231707+0000) 2024-03-04T20:29:01.204 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:29:01.231+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:29:01.231947+0000) 2024-03-04T20:29:02.204 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:29:02.231+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:29:02.232179+0000) 2024-03-04T20:29:03.204 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:29:03.231+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:29:03.232434+0000) 2024-03-04T20:29:04.205 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:29:04.231+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:29:04.232663+0000) 2024-03-04T20:29:05.205 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:29:05.232+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:29:05.232947+0000) 2024-03-04T20:29:06.205 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:29:06.232+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:29:06.233180+0000) 2024-03-04T20:29:07.206 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:29:07.232+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:29:07.233415+0000) 2024-03-04T20:29:08.206 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:29:08.233+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:29:08.233656+0000) 2024-03-04T20:29:09.206 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:29:09.233+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:29:09.233941+0000) 2024-03-04T20:29:10.206 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:29:10.233+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:29:10.234117+0000) 2024-03-04T20:29:11.207 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:29:11.233+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:29:11.234311+0000) 2024-03-04T20:29:12.207 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:29:12.233+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:29:12.234545+0000) 2024-03-04T20:29:13.207 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:29:13.234+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:29:13.234810+0000) 2024-03-04T20:29:14.208 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:29:14.234+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:29:14.235091+0000) 2024-03-04T20:29:15.208 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:29:15.234+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:29:15.235336+0000) 2024-03-04T20:29:16.208 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:29:16.235+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:29:16.235575+0000) 2024-03-04T20:29:16.544 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:29:16.552 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:29:17.208 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:29:17.234+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:29:17.235762+0000) 2024-03-04T20:29:18.209 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:29:18.235+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:29:18.236010+0000) 2024-03-04T20:29:19.209 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:29:19.235+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:29:19.236250+0000) 2024-03-04T20:29:20.209 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:29:20.235+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:29:20.236402+0000) 2024-03-04T20:29:21.210 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:29:21.235+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:29:21.236657+0000) 2024-03-04T20:29:22.210 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:29:22.236+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:29:22.236935+0000) 2024-03-04T20:29:23.210 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:29:23.236+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:29:23.237174+0000) 2024-03-04T20:29:24.211 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:29:24.236+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:29:24.237432+0000) 2024-03-04T20:29:25.211 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:29:25.236+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:29:25.237691+0000) 2024-03-04T20:29:26.211 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:29:26.237+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:29:26.237931+0000) 2024-03-04T20:29:27.211 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:29:27.237+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:29:27.238113+0000) 2024-03-04T20:29:28.212 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:29:28.238+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:29:28.238357+0000) 2024-03-04T20:29:29.212 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:29:29.238+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:29:29.238617+0000) 2024-03-04T20:29:30.212 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:29:30.238+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:29:30.238921+0000) 2024-03-04T20:29:31.213 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:29:31.238+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:29:31.239159+0000) 2024-03-04T20:29:32.213 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:29:32.239+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:29:32.239402+0000) 2024-03-04T20:29:33.213 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:29:33.239+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:29:33.239657+0000) 2024-03-04T20:29:34.213 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:29:34.239+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:29:34.239952+0000) 2024-03-04T20:29:35.214 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:29:35.240+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:29:35.240196+0000) 2024-03-04T20:29:36.214 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:29:36.240+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:29:36.240435+0000) 2024-03-04T20:29:37.214 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:29:37.239+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:29:37.240671+0000) 2024-03-04T20:29:38.215 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:29:38.239+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:29:38.240956+0000) 2024-03-04T20:29:39.216 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:29:39.240+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:29:39.241198+0000) 2024-03-04T20:29:40.215 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:29:40.240+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:29:40.241418+0000) 2024-03-04T20:29:41.216 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:29:41.240+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:29:41.241651+0000) 2024-03-04T20:29:42.216 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:29:42.240+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:29:42.241923+0000) 2024-03-04T20:29:43.216 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:29:43.241+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:29:43.242162+0000) 2024-03-04T20:29:44.216 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:29:44.241+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:29:44.242402+0000) 2024-03-04T20:29:45.217 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:29:45.241+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:29:45.242667+0000) 2024-03-04T20:29:46.217 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:29:46.241+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:29:46.242948+0000) 2024-03-04T20:29:46.766 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:29:46.777 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:29:47.217 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:29:47.242+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:29:47.243171+0000) 2024-03-04T20:29:48.218 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:29:48.242+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:29:48.243459+0000) 2024-03-04T20:29:49.218 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:29:49.242+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:29:49.243756+0000) 2024-03-04T20:29:50.218 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:29:50.243+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:29:50.244001+0000) 2024-03-04T20:29:51.219 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:29:51.243+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:29:51.244241+0000) 2024-03-04T20:29:52.219 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:29:52.243+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:29:52.244487+0000) 2024-03-04T20:29:53.219 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:29:53.243+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:29:53.244744+0000) 2024-03-04T20:29:54.219 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:29:54.244+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:29:54.244989+0000) 2024-03-04T20:29:55.220 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:29:55.244+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:29:55.245231+0000) 2024-03-04T20:29:56.220 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:29:56.244+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:29:56.245469+0000) 2024-03-04T20:29:57.221 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:29:57.244+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:29:57.245708+0000) 2024-03-04T20:29:58.221 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:29:58.245+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:29:58.245980+0000) 2024-03-04T20:29:59.221 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:29:59.245+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:29:59.246235+0000) 2024-03-04T20:30:00.221 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:30:00.245+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:30:00.246537+0000) 2024-03-04T20:30:01.222 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:30:01.246+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:30:01.246776+0000) 2024-03-04T20:30:02.222 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:30:02.246+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:30:02.247021+0000) 2024-03-04T20:30:03.222 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:30:03.246+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:30:03.247260+0000) 2024-03-04T20:30:04.222 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:30:04.246+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:30:04.247497+0000) 2024-03-04T20:30:05.223 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:30:05.247+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:30:05.247740+0000) 2024-03-04T20:30:06.223 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:30:06.247+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:30:06.247973+0000) 2024-03-04T20:30:07.223 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:30:07.247+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:30:07.248206+0000) 2024-03-04T20:30:08.224 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:30:08.247+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:30:08.248450+0000) 2024-03-04T20:30:09.224 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:30:09.248+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:30:09.248692+0000) 2024-03-04T20:30:10.224 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:30:10.248+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:30:10.248930+0000) 2024-03-04T20:30:11.225 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:30:11.248+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:30:11.249167+0000) 2024-03-04T20:30:12.225 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:30:12.248+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:30:12.249401+0000) 2024-03-04T20:30:13.225 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:30:13.249+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:30:13.249638+0000) 2024-03-04T20:30:14.225 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:30:14.249+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:30:14.249921+0000) 2024-03-04T20:30:15.226 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:30:15.249+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:30:15.250179+0000) 2024-03-04T20:30:16.226 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:30:16.249+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:30:16.250437+0000) 2024-03-04T20:30:16.922 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:30:16.930 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:30:17.226 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:30:17.250+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:30:17.250694+0000) 2024-03-04T20:30:18.227 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:30:18.250+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:30:18.250935+0000) 2024-03-04T20:30:19.227 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:30:19.250+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:30:19.251179+0000) 2024-03-04T20:30:20.227 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:30:20.250+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:30:20.251414+0000) 2024-03-04T20:30:21.228 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:30:21.251+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:30:21.251653+0000) 2024-03-04T20:30:22.228 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:30:22.251+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:30:22.251932+0000) 2024-03-04T20:30:23.228 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:30:23.251+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:30:23.252169+0000) 2024-03-04T20:30:24.228 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:30:24.252+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:30:24.252424+0000) 2024-03-04T20:30:25.229 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:30:25.252+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:30:25.252663+0000) 2024-03-04T20:30:26.229 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:30:26.252+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:30:26.252935+0000) 2024-03-04T20:30:27.230 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:30:27.252+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:30:27.253170+0000) 2024-03-04T20:30:28.229 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:30:28.253+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:30:28.253410+0000) 2024-03-04T20:30:29.230 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:30:29.253+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:30:29.253667+0000) 2024-03-04T20:30:30.230 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:30:30.253+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:30:30.253930+0000) 2024-03-04T20:30:31.230 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:30:31.253+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:30:31.254209+0000) 2024-03-04T20:30:32.231 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:30:32.254+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:30:32.254497+0000) 2024-03-04T20:30:33.231 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:30:33.254+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:30:33.254795+0000) 2024-03-04T20:30:34.232 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:30:34.254+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:30:34.255151+0000) 2024-03-04T20:30:35.232 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:30:35.254+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:30:35.255441+0000) 2024-03-04T20:30:36.232 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:30:36.255+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:30:36.255716+0000) 2024-03-04T20:30:37.233 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:30:37.255+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:30:37.256014+0000) 2024-03-04T20:30:38.233 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:30:38.255+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:30:38.256265+0000) 2024-03-04T20:30:39.233 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:30:39.255+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:30:39.256545+0000) 2024-03-04T20:30:40.233 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:30:40.255+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:30:40.256791+0000) 2024-03-04T20:30:41.234 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:30:41.255+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:30:41.257085+0000) 2024-03-04T20:30:42.234 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:30:42.256+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:30:42.257363+0000) 2024-03-04T20:30:43.235 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:30:43.256+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:30:43.257600+0000) 2024-03-04T20:30:44.235 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:30:44.256+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:30:44.257880+0000) 2024-03-04T20:30:45.235 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:30:45.257+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:30:45.258132+0000) 2024-03-04T20:30:46.235 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:30:46.257+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:30:46.258376+0000) 2024-03-04T20:30:47.145 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:30:47.154 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:30:47.235 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:30:47.256+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:30:47.258539+0000) 2024-03-04T20:30:48.236 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:30:48.257+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:30:48.258811+0000) 2024-03-04T20:30:49.236 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:30:49.258+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:30:49.259093+0000) 2024-03-04T20:30:50.236 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:30:50.258+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:30:50.259330+0000) 2024-03-04T20:30:51.237 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:30:51.258+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:30:51.259567+0000) 2024-03-04T20:30:52.237 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:30:52.258+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:30:52.259759+0000) 2024-03-04T20:30:53.237 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:30:53.259+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:30:53.260007+0000) 2024-03-04T20:30:54.238 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:30:54.259+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:30:54.260248+0000) 2024-03-04T20:30:55.238 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:30:55.259+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:30:55.260502+0000) 2024-03-04T20:30:56.238 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:30:56.259+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:30:56.260738+0000) 2024-03-04T20:30:57.238 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:30:57.260+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:30:57.260977+0000) 2024-03-04T20:30:58.239 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:30:58.260+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:30:58.261256+0000) 2024-03-04T20:30:59.239 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:30:59.260+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:30:59.261517+0000) 2024-03-04T20:31:00.239 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:31:00.260+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:31:00.261751+0000) 2024-03-04T20:31:01.240 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:31:01.261+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:31:01.261989+0000) 2024-03-04T20:31:02.240 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:31:02.261+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:31:02.262221+0000) 2024-03-04T20:31:03.240 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:31:03.261+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:31:03.262481+0000) 2024-03-04T20:31:04.240 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:31:04.262+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:31:04.262751+0000) 2024-03-04T20:31:05.241 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:31:05.262+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:31:05.262994+0000) 2024-03-04T20:31:06.241 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:31:06.262+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:31:06.263231+0000) 2024-03-04T20:31:07.242 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:31:07.262+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:31:07.263476+0000) 2024-03-04T20:31:08.242 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:31:08.263+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:31:08.263728+0000) 2024-03-04T20:31:09.242 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:31:09.263+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:31:09.263977+0000) 2024-03-04T20:31:10.242 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:31:10.263+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:31:10.264219+0000) 2024-03-04T20:31:11.242 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:31:11.263+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:31:11.264459+0000) 2024-03-04T20:31:12.243 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:31:12.264+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:31:12.264693+0000) 2024-03-04T20:31:13.243 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:31:13.264+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:31:13.264945+0000) 2024-03-04T20:31:14.244 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:31:14.264+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:31:14.265196+0000) 2024-03-04T20:31:15.244 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:31:15.264+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:31:15.265467+0000) 2024-03-04T20:31:16.245 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:31:16.265+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:31:16.265724+0000) 2024-03-04T20:31:17.244 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:31:17.265+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:31:17.265972+0000) 2024-03-04T20:31:17.351 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:31:17.362 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:31:18.244 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:31:18.265+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:31:18.266216+0000) 2024-03-04T20:31:19.245 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:31:19.265+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:31:19.266457+0000) 2024-03-04T20:31:20.245 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:31:20.266+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:31:20.266693+0000) 2024-03-04T20:31:21.245 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:31:21.266+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:31:21.266940+0000) 2024-03-04T20:31:22.246 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:31:22.266+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:31:22.267201+0000) 2024-03-04T20:31:23.246 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:31:23.267+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:31:23.267453+0000) 2024-03-04T20:31:24.246 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:31:24.267+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:31:24.267699+0000) 2024-03-04T20:31:25.247 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:31:25.267+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:31:25.267952+0000) 2024-03-04T20:31:26.247 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:31:26.267+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:31:26.268195+0000) 2024-03-04T20:31:27.247 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:31:27.268+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:31:27.268434+0000) 2024-03-04T20:31:28.248 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:31:28.268+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:31:28.268723+0000) 2024-03-04T20:31:29.248 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:31:29.268+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:31:29.268985+0000) 2024-03-04T20:31:30.248 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:31:30.268+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:31:30.269290+0000) 2024-03-04T20:31:31.249 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:31:31.269+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:31:31.269551+0000) 2024-03-04T20:31:32.249 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:31:32.268+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:31:32.269795+0000) 2024-03-04T20:31:33.249 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:31:33.269+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:31:33.270095+0000) 2024-03-04T20:31:34.250 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:31:34.270+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:31:34.270353+0000) 2024-03-04T20:31:35.250 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:31:35.269+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:31:35.270645+0000) 2024-03-04T20:31:36.250 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:31:36.270+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:31:36.270937+0000) 2024-03-04T20:31:37.250 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:31:37.271+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:31:37.271198+0000) 2024-03-04T20:31:38.251 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:31:38.270+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:31:38.271477+0000) 2024-03-04T20:31:39.251 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:31:39.271+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:31:39.271745+0000) 2024-03-04T20:31:40.251 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:31:40.271+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:31:40.271996+0000) 2024-03-04T20:31:41.251 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:31:41.271+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:31:41.272201+0000) 2024-03-04T20:31:42.252 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:31:42.271+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:31:42.272455+0000) 2024-03-04T20:31:43.252 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:31:43.271+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:31:43.272729+0000) 2024-03-04T20:31:44.253 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:31:44.271+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:31:44.273004+0000) 2024-03-04T20:31:45.253 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:31:45.272+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:31:45.273289+0000) 2024-03-04T20:31:46.253 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:31:46.271+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:31:46.273564+0000) 2024-03-04T20:31:47.253 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:31:47.272+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:31:47.273807+0000) 2024-03-04T20:31:47.558 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:31:47.566 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:31:48.254 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:31:48.273+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:31:48.274118+0000) 2024-03-04T20:31:49.254 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:31:49.272+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:31:49.274364+0000) 2024-03-04T20:31:50.254 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:31:50.273+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:31:50.274545+0000) 2024-03-04T20:31:51.254 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:31:51.273+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:31:51.274779+0000) 2024-03-04T20:31:52.255 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:31:52.274+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:31:52.275032+0000) 2024-03-04T20:31:53.255 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:31:53.274+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:31:53.275232+0000) 2024-03-04T20:31:54.256 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:31:54.274+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:31:54.275491+0000) 2024-03-04T20:31:55.256 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:31:55.274+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:31:55.275738+0000) 2024-03-04T20:31:56.256 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:31:56.275+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:31:56.275995+0000) 2024-03-04T20:31:57.256 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:31:57.275+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:31:57.276276+0000) 2024-03-04T20:31:58.257 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:31:58.275+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:31:58.276521+0000) 2024-03-04T20:31:59.257 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:31:59.276+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:31:59.276812+0000) 2024-03-04T20:32:00.257 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:32:00.276+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:32:00.277112+0000) 2024-03-04T20:32:01.257 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:32:01.276+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:32:01.277365+0000) 2024-03-04T20:32:02.258 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:32:02.276+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:32:02.277641+0000) 2024-03-04T20:32:03.258 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:32:03.277+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:32:03.277946+0000) 2024-03-04T20:32:04.259 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:32:04.277+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:32:04.278195+0000) 2024-03-04T20:32:05.259 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:32:05.277+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:32:05.278459+0000) 2024-03-04T20:32:06.259 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:32:06.277+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:32:06.278745+0000) 2024-03-04T20:32:07.259 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:32:07.278+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:32:07.278990+0000) 2024-03-04T20:32:08.260 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:32:08.278+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:32:08.279248+0000) 2024-03-04T20:32:09.260 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:32:09.278+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:32:09.279517+0000) 2024-03-04T20:32:10.260 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:32:10.279+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:32:10.279791+0000) 2024-03-04T20:32:11.261 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:32:11.279+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:32:11.280093+0000) 2024-03-04T20:32:12.261 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:32:12.279+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:32:12.280351+0000) 2024-03-04T20:32:13.261 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:32:13.279+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:32:13.280624+0000) 2024-03-04T20:32:14.261 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:32:14.280+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:32:14.280918+0000) 2024-03-04T20:32:15.262 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:32:15.280+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:32:15.281181+0000) 2024-03-04T20:32:16.262 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:32:16.280+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:32:16.281460+0000) 2024-03-04T20:32:17.263 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:32:17.281+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:32:17.281720+0000) 2024-03-04T20:32:17.765 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:32:17.773 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:32:18.263 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:32:18.281+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:32:18.281978+0000) 2024-03-04T20:32:19.263 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:32:19.281+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:32:19.282186+0000) 2024-03-04T20:32:20.263 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:32:20.281+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:32:20.282452+0000) 2024-03-04T20:32:21.264 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:32:21.282+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:32:21.282693+0000) 2024-03-04T20:32:22.264 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:32:22.282+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:32:22.282980+0000) 2024-03-04T20:32:23.264 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:32:23.282+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:32:23.283220+0000) 2024-03-04T20:32:24.265 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:32:24.283+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:32:24.283514+0000) 2024-03-04T20:32:25.265 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:32:25.283+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:32:25.283757+0000) 2024-03-04T20:32:26.265 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:32:26.283+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:32:26.284013+0000) 2024-03-04T20:32:27.266 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:32:27.283+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:32:27.284270+0000) 2024-03-04T20:32:28.265 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:32:28.284+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:32:28.284516+0000) 2024-03-04T20:32:29.266 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:32:29.284+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:32:29.284798+0000) 2024-03-04T20:32:30.266 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:32:30.284+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:32:30.285095+0000) 2024-03-04T20:32:31.266 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:32:31.284+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:32:31.285336+0000) 2024-03-04T20:32:32.267 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:32:32.284+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:32:32.285579+0000) 2024-03-04T20:32:33.267 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:32:33.285+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:32:33.285812+0000) 2024-03-04T20:32:34.267 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:32:34.285+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:32:34.286077+0000) 2024-03-04T20:32:35.268 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:32:35.286+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:32:35.286319+0000) 2024-03-04T20:32:36.268 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:32:36.286+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:32:36.286557+0000) 2024-03-04T20:32:37.268 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:32:37.286+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:32:37.286793+0000) 2024-03-04T20:32:38.268 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:32:38.286+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:32:38.287076+0000) 2024-03-04T20:32:39.269 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:32:39.287+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:32:39.287319+0000) 2024-03-04T20:32:40.269 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:32:40.287+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:32:40.287558+0000) 2024-03-04T20:32:41.270 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:32:41.287+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:32:41.287795+0000) 2024-03-04T20:32:42.269 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:32:42.287+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:32:42.288073+0000) 2024-03-04T20:32:43.270 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:32:43.288+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:32:43.288316+0000) 2024-03-04T20:32:44.270 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:32:44.287+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:32:44.288552+0000) 2024-03-04T20:32:45.271 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:32:45.287+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:32:45.288795+0000) 2024-03-04T20:32:46.271 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:32:46.287+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:32:46.289076+0000) 2024-03-04T20:32:47.271 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:32:47.288+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:32:47.289311+0000) 2024-03-04T20:32:47.906 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:32:47.914 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:32:48.271 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:32:48.287+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:32:48.289506+0000) 2024-03-04T20:32:49.272 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:32:49.288+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:32:49.289754+0000) 2024-03-04T20:32:50.272 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:32:50.289+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:32:50.290022+0000) 2024-03-04T20:32:51.272 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:32:51.289+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:32:51.290253+0000) 2024-03-04T20:32:52.272 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:32:52.289+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:32:52.290494+0000) 2024-03-04T20:32:53.273 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:32:53.289+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:32:53.290746+0000) 2024-03-04T20:32:54.273 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:32:54.290+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:32:54.290999+0000) 2024-03-04T20:32:55.273 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:32:55.290+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:32:55.291258+0000) 2024-03-04T20:32:56.274 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:32:56.290+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:32:56.291539+0000) 2024-03-04T20:32:57.274 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:32:57.290+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:32:57.291784+0000) 2024-03-04T20:32:58.274 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:32:58.291+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:32:58.292090+0000) 2024-03-04T20:32:59.275 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:32:59.291+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:32:59.292350+0000) 2024-03-04T20:33:00.275 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:33:00.291+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:33:00.292604+0000) 2024-03-04T20:33:01.275 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:33:01.292+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:33:01.292888+0000) 2024-03-04T20:33:02.275 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:33:02.292+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:33:02.293141+0000) 2024-03-04T20:33:03.276 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:33:03.292+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:33:03.293335+0000) 2024-03-04T20:33:04.276 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:33:04.292+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:33:04.293589+0000) 2024-03-04T20:33:05.276 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:33:05.293+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:33:05.293862+0000) 2024-03-04T20:33:06.277 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:33:06.293+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:33:06.294128+0000) 2024-03-04T20:33:07.277 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:33:07.293+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:33:07.294366+0000) 2024-03-04T20:33:08.277 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:33:08.293+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:33:08.294613+0000) 2024-03-04T20:33:09.278 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:33:09.294+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:33:09.294886+0000) 2024-03-04T20:33:10.278 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:33:10.294+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:33:10.295141+0000) 2024-03-04T20:33:11.278 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:33:11.294+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:33:11.295376+0000) 2024-03-04T20:33:12.278 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:33:12.294+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:33:12.295623+0000) 2024-03-04T20:33:13.279 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:33:13.295+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:33:13.295894+0000) 2024-03-04T20:33:14.279 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:33:14.295+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:33:14.296128+0000) 2024-03-04T20:33:15.279 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:33:15.295+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:33:15.296370+0000) 2024-03-04T20:33:16.279 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:33:16.296+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:33:16.296608+0000) 2024-03-04T20:33:17.280 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:33:17.296+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:33:17.296927+0000) 2024-03-04T20:33:18.135 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:33:18.147 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:33:18.280 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:33:18.296+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:33:18.297121+0000) 2024-03-04T20:33:19.281 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:33:19.296+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:33:19.297378+0000) 2024-03-04T20:33:20.281 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:33:20.297+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:33:20.297616+0000) 2024-03-04T20:33:21.281 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:33:21.297+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:33:21.297902+0000) 2024-03-04T20:33:22.281 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:33:22.297+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:33:22.298140+0000) 2024-03-04T20:33:23.282 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:33:23.297+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:33:23.298381+0000) 2024-03-04T20:33:24.282 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:33:24.298+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:33:24.298625+0000) 2024-03-04T20:33:25.282 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:33:25.298+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:33:25.298910+0000) 2024-03-04T20:33:26.282 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:33:26.298+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:33:26.299177+0000) 2024-03-04T20:33:27.283 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:33:27.299+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:33:27.299435+0000) 2024-03-04T20:33:28.283 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:33:28.299+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:33:28.299678+0000) 2024-03-04T20:33:29.283 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:33:29.299+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:33:29.299957+0000) 2024-03-04T20:33:30.284 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:33:30.299+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:33:30.300198+0000) 2024-03-04T20:33:31.284 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:33:31.299+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:33:31.300374+0000) 2024-03-04T20:33:32.284 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:33:32.299+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:33:32.300601+0000) 2024-03-04T20:33:33.284 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:33:33.300+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:33:33.300874+0000) 2024-03-04T20:33:34.285 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:33:34.300+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:33:34.301116+0000) 2024-03-04T20:33:35.285 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:33:35.301+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:33:35.301365+0000) 2024-03-04T20:33:36.285 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:33:36.301+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:33:36.301602+0000) 2024-03-04T20:33:37.285 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:33:37.301+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:33:37.301858+0000) 2024-03-04T20:33:38.286 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:33:38.301+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:33:38.302100+0000) 2024-03-04T20:33:39.286 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:33:39.302+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:33:39.302347+0000) 2024-03-04T20:33:40.286 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:33:40.302+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:33:40.302585+0000) 2024-03-04T20:33:41.286 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:33:41.302+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:33:41.302812+0000) 2024-03-04T20:33:42.287 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:33:42.302+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:33:42.303082+0000) 2024-03-04T20:33:43.287 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:33:43.303+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:33:43.303315+0000) 2024-03-04T20:33:44.287 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:33:44.303+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:33:44.303554+0000) 2024-03-04T20:33:45.288 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:33:45.303+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:33:45.303794+0000) 2024-03-04T20:33:46.288 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:33:46.302+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:33:46.304074+0000) 2024-03-04T20:33:47.288 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:33:47.303+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:33:47.304306+0000) 2024-03-04T20:33:48.288 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:33:48.303+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:33:48.304547+0000) 2024-03-04T20:33:48.341 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:33:48.349 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:33:49.289 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:33:49.303+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:33:49.304789+0000) 2024-03-04T20:33:50.289 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:33:50.303+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:33:50.305028+0000) 2024-03-04T20:33:51.289 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:33:51.304+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:33:51.305252+0000) 2024-03-04T20:33:52.289 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:33:52.304+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:33:52.305486+0000) 2024-03-04T20:33:53.290 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:33:53.304+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:33:53.305715+0000) 2024-03-04T20:33:54.290 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:33:54.304+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:33:54.305959+0000) 2024-03-04T20:33:55.290 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:33:55.305+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:33:55.306199+0000) 2024-03-04T20:33:56.291 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:33:56.305+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:33:56.306435+0000) 2024-03-04T20:33:57.291 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:33:57.305+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:33:57.306688+0000) 2024-03-04T20:33:58.291 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:33:58.306+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:33:58.306947+0000) 2024-03-04T20:33:59.291 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:33:59.306+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:33:59.307192+0000) 2024-03-04T20:34:00.292 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:34:00.306+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:34:00.307435+0000) 2024-03-04T20:34:01.292 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:34:01.306+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:34:01.307668+0000) 2024-03-04T20:34:02.292 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:34:02.307+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:34:02.307936+0000) 2024-03-04T20:34:03.292 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:34:03.307+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:34:03.308127+0000) 2024-03-04T20:34:04.293 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:34:04.307+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:34:04.308365+0000) 2024-03-04T20:34:05.293 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:34:05.307+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:34:05.308609+0000) 2024-03-04T20:34:06.293 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:34:06.308+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:34:06.308874+0000) 2024-03-04T20:34:07.294 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:34:07.308+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:34:07.309100+0000) 2024-03-04T20:34:08.294 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:34:08.308+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:34:08.309328+0000) 2024-03-04T20:34:09.294 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:34:09.308+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:34:09.309588+0000) 2024-03-04T20:34:10.294 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:34:10.309+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:34:10.309909+0000) 2024-03-04T20:34:11.295 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:34:11.309+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:34:11.310144+0000) 2024-03-04T20:34:12.295 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:34:12.309+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:34:12.310386+0000) 2024-03-04T20:34:13.295 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:34:13.309+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:34:13.310618+0000) 2024-03-04T20:34:14.296 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:34:14.310+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:34:14.310904+0000) 2024-03-04T20:34:15.296 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:34:15.310+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:34:15.311145+0000) 2024-03-04T20:34:16.296 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:34:16.310+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:34:16.311390+0000) 2024-03-04T20:34:17.297 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:34:17.311+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:34:17.311616+0000) 2024-03-04T20:34:18.297 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:34:18.311+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:34:18.311908+0000) 2024-03-04T20:34:18.555 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:34:18.563 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:34:19.297 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:34:19.311+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:34:19.312148+0000) 2024-03-04T20:34:20.297 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:34:20.311+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:34:20.312398+0000) 2024-03-04T20:34:21.297 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:34:21.311+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:34:21.312583+0000) 2024-03-04T20:34:22.298 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:34:22.312+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:34:22.312816+0000) 2024-03-04T20:34:23.298 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:34:23.312+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:34:23.313080+0000) 2024-03-04T20:34:24.298 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:34:24.312+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:34:24.313315+0000) 2024-03-04T20:34:25.299 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:34:25.313+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:34:25.313593+0000) 2024-03-04T20:34:26.299 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:34:26.313+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:34:26.313863+0000) 2024-03-04T20:34:27.299 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:34:27.313+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:34:27.314115+0000) 2024-03-04T20:34:28.299 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:34:28.313+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:34:28.314384+0000) 2024-03-04T20:34:29.300 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:34:29.314+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:34:29.314627+0000) 2024-03-04T20:34:30.300 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:34:30.314+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:34:30.314902+0000) 2024-03-04T20:34:31.300 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:34:31.314+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:34:31.315131+0000) 2024-03-04T20:34:32.301 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:34:32.314+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:34:32.315372+0000) 2024-03-04T20:34:33.301 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:34:33.315+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:34:33.315605+0000) 2024-03-04T20:34:34.301 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:34:34.315+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:34:34.315891+0000) 2024-03-04T20:34:35.302 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:34:35.315+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:34:35.316137+0000) 2024-03-04T20:34:36.302 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:34:36.316+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:34:36.316374+0000) 2024-03-04T20:34:37.302 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:34:37.315+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:34:37.316619+0000) 2024-03-04T20:34:38.303 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:34:38.316+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:34:38.316899+0000) 2024-03-04T20:34:39.303 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:34:39.316+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:34:39.317137+0000) 2024-03-04T20:34:40.303 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:34:40.317+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:34:40.317371+0000) 2024-03-04T20:34:41.303 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:34:41.317+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:34:41.317604+0000) 2024-03-04T20:34:42.304 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:34:42.317+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:34:42.317884+0000) 2024-03-04T20:34:43.304 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:34:43.317+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:34:43.318126+0000) 2024-03-04T20:34:44.304 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:34:44.318+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:34:44.318377+0000) 2024-03-04T20:34:45.304 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:34:45.318+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:34:45.318625+0000) 2024-03-04T20:34:46.305 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:34:46.318+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:34:46.318888+0000) 2024-03-04T20:34:47.305 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:34:47.319+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:34:47.319148+0000) 2024-03-04T20:34:48.305 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:34:48.319+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:34:48.319416+0000) 2024-03-04T20:34:48.762 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:34:48.774 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:34:49.305 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:34:49.318+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:34:49.319657+0000) 2024-03-04T20:34:50.306 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:34:50.318+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:34:50.319933+0000) 2024-03-04T20:34:51.306 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:34:51.319+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:34:51.320166+0000) 2024-03-04T20:34:52.306 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:34:52.319+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:34:52.320403+0000) 2024-03-04T20:34:53.306 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:34:53.318+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:34:53.320654+0000) 2024-03-04T20:34:54.307 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:34:54.319+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:34:54.320925+0000) 2024-03-04T20:34:55.307 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:34:55.320+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:34:55.321172+0000) 2024-03-04T20:34:56.307 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:34:56.320+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:34:56.321413+0000) 2024-03-04T20:34:57.308 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:34:57.320+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:34:57.321671+0000) 2024-03-04T20:34:58.308 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:34:58.320+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:34:58.321976+0000) 2024-03-04T20:34:59.308 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:34:59.321+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:34:59.322218+0000) 2024-03-04T20:35:00.308 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:35:00.321+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:35:00.322481+0000) 2024-03-04T20:35:01.309 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:35:01.321+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:35:01.322724+0000) 2024-03-04T20:35:02.309 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:35:02.322+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:35:02.322979+0000) 2024-03-04T20:35:03.309 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:35:03.322+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:35:03.323213+0000) 2024-03-04T20:35:04.310 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:35:04.322+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:35:04.323455+0000) 2024-03-04T20:35:05.310 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:35:05.322+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:35:05.323697+0000) 2024-03-04T20:35:06.310 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:35:06.323+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:35:06.323966+0000) 2024-03-04T20:35:07.311 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:35:07.323+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:35:07.324204+0000) 2024-03-04T20:35:08.311 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:35:08.323+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:35:08.324441+0000) 2024-03-04T20:35:09.311 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:35:09.323+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:35:09.324681+0000) 2024-03-04T20:35:10.311 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:35:10.324+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:35:10.324935+0000) 2024-03-04T20:35:11.312 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:35:11.324+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:35:11.325175+0000) 2024-03-04T20:35:12.312 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:35:12.324+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:35:12.325416+0000) 2024-03-04T20:35:13.312 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:35:13.324+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:35:13.325654+0000) 2024-03-04T20:35:14.313 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:35:14.325+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:35:14.325918+0000) 2024-03-04T20:35:15.313 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:35:15.325+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:35:15.326160+0000) 2024-03-04T20:35:16.313 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:35:16.325+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:35:16.326403+0000) 2024-03-04T20:35:17.313 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:35:17.325+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:35:17.326632+0000) 2024-03-04T20:35:18.313 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:35:18.326+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:35:18.326902+0000) 2024-03-04T20:35:18.916 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:35:18.924 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:35:19.313 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:35:19.326+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:35:19.327155+0000) 2024-03-04T20:35:20.314 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:35:20.326+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:35:20.327394+0000) 2024-03-04T20:35:21.315 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:35:21.327+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:35:21.327621+0000) 2024-03-04T20:35:22.315 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:35:22.327+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:35:22.327888+0000) 2024-03-04T20:35:23.315 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:35:23.327+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:35:23.328129+0000) 2024-03-04T20:35:24.315 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:35:24.327+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:35:24.328325+0000) 2024-03-04T20:35:25.316 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:35:25.328+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:35:25.328596+0000) 2024-03-04T20:35:26.316 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:35:26.328+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:35:26.328857+0000) 2024-03-04T20:35:27.316 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:35:27.328+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:35:27.329084+0000) 2024-03-04T20:35:28.316 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:35:28.328+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:35:28.329339+0000) 2024-03-04T20:35:29.317 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:35:29.329+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:35:29.329602+0000) 2024-03-04T20:35:30.317 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:35:30.329+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:35:30.329874+0000) 2024-03-04T20:35:31.317 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:35:31.329+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:35:31.330111+0000) 2024-03-04T20:35:32.317 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:35:32.329+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:35:32.330344+0000) 2024-03-04T20:35:33.318 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:35:33.330+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:35:33.330573+0000) 2024-03-04T20:35:34.318 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:35:34.330+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:35:34.330812+0000) 2024-03-04T20:35:35.318 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:35:35.330+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:35:35.331089+0000) 2024-03-04T20:35:36.318 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:35:36.330+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:35:36.331327+0000) 2024-03-04T20:35:37.319 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:35:37.331+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:35:37.331560+0000) 2024-03-04T20:35:38.319 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:35:38.331+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:35:38.331794+0000) 2024-03-04T20:35:39.319 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:35:39.331+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:35:39.332085+0000) 2024-03-04T20:35:40.319 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:35:40.332+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:35:40.332322+0000) 2024-03-04T20:35:41.320 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:35:41.332+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:35:41.332600+0000) 2024-03-04T20:35:42.320 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:35:42.332+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:35:42.332891+0000) 2024-03-04T20:35:43.320 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:35:43.332+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:35:43.333120+0000) 2024-03-04T20:35:44.320 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:35:44.333+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:35:44.333357+0000) 2024-03-04T20:35:45.321 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:35:45.333+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:35:45.333604+0000) 2024-03-04T20:35:46.321 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:35:46.333+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:35:46.333880+0000) 2024-03-04T20:35:47.322 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:35:47.333+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:35:47.334111+0000) 2024-03-04T20:35:48.322 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:35:48.334+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:35:48.334347+0000) 2024-03-04T20:35:49.123 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:35:49.131 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:35:49.322 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:35:49.334+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:35:49.334612+0000) 2024-03-04T20:35:50.322 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:35:50.334+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:35:50.334876+0000) 2024-03-04T20:35:51.322 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:35:51.333+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:35:51.335109+0000) 2024-03-04T20:35:52.323 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:35:52.334+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:35:52.335343+0000) 2024-03-04T20:35:53.324 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:35:53.334+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:35:53.335574+0000) 2024-03-04T20:35:54.323 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:35:54.334+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:35:54.335754+0000) 2024-03-04T20:35:55.323 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:35:55.334+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:35:55.336000+0000) 2024-03-04T20:35:56.324 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:35:56.334+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:35:56.336192+0000) 2024-03-04T20:35:57.324 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:35:57.335+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:35:57.336420+0000) 2024-03-04T20:35:58.324 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:35:58.335+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:35:58.336684+0000) 2024-03-04T20:35:59.324 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:35:59.335+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:35:59.336950+0000) 2024-03-04T20:36:00.325 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:36:00.336+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:36:00.337210+0000) 2024-03-04T20:36:01.325 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:36:01.336+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:36:01.337436+0000) 2024-03-04T20:36:02.326 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:36:02.336+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:36:02.337678+0000) 2024-03-04T20:36:03.326 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:36:03.337+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:36:03.337924+0000) 2024-03-04T20:36:04.326 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:36:04.337+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:36:04.338162+0000) 2024-03-04T20:36:05.326 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:36:05.337+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:36:05.338415+0000) 2024-03-04T20:36:06.327 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:36:06.337+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:36:06.338676+0000) 2024-03-04T20:36:07.327 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:36:07.338+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:36:07.338929+0000) 2024-03-04T20:36:08.327 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:36:08.338+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:36:08.339169+0000) 2024-03-04T20:36:09.327 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:36:09.338+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:36:09.339421+0000) 2024-03-04T20:36:10.327 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:36:10.338+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:36:10.339657+0000) 2024-03-04T20:36:11.328 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:36:11.339+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:36:11.339877+0000) 2024-03-04T20:36:12.328 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:36:12.339+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:36:12.340113+0000) 2024-03-04T20:36:13.328 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:36:13.339+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:36:13.340345+0000) 2024-03-04T20:36:14.329 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:36:14.339+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:36:14.340580+0000) 2024-03-04T20:36:15.329 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:36:15.340+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:36:15.340868+0000) 2024-03-04T20:36:16.329 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:36:16.340+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:36:16.341119+0000) 2024-03-04T20:36:17.329 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:36:17.340+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:36:17.341364+0000) 2024-03-04T20:36:18.330 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:36:18.340+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:36:18.341624+0000) 2024-03-04T20:36:19.330 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:36:19.341+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:36:19.341933+0000) 2024-03-04T20:36:19.345 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:36:19.353 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:36:20.330 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:36:20.341+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:36:20.342191+0000) 2024-03-04T20:36:21.330 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:36:21.341+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:36:21.342443+0000) 2024-03-04T20:36:22.331 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:36:22.341+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:36:22.342708+0000) 2024-03-04T20:36:23.331 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:36:23.342+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:36:23.342956+0000) 2024-03-04T20:36:24.331 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:36:24.342+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:36:24.343220+0000) 2024-03-04T20:36:25.332 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:36:25.342+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:36:25.343481+0000) 2024-03-04T20:36:26.332 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:36:26.343+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:36:26.343735+0000) 2024-03-04T20:36:27.332 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:36:27.343+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:36:27.343990+0000) 2024-03-04T20:36:28.332 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:36:28.343+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:36:28.344256+0000) 2024-03-04T20:36:29.333 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:36:29.344+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:36:29.344541+0000) 2024-03-04T20:36:30.333 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:36:30.344+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:36:30.344787+0000) 2024-03-04T20:36:31.333 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:36:31.344+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:36:31.345072+0000) 2024-03-04T20:36:32.334 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:36:32.344+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:36:32.345328+0000) 2024-03-04T20:36:33.334 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:36:33.345+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:36:33.345600+0000) 2024-03-04T20:36:34.334 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:36:34.345+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:36:34.345951+0000) 2024-03-04T20:36:35.335 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:36:35.345+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:36:35.346220+0000) 2024-03-04T20:36:36.335 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:36:36.346+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:36:36.346479+0000) 2024-03-04T20:36:37.335 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:36:37.346+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:36:37.346722+0000) 2024-03-04T20:36:38.335 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:36:38.346+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:36:38.346979+0000) 2024-03-04T20:36:39.336 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:36:39.346+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:36:39.347245+0000) 2024-03-04T20:36:40.336 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:36:40.347+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:36:40.347502+0000) 2024-03-04T20:36:41.336 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:36:41.347+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:36:41.347741+0000) 2024-03-04T20:36:42.337 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:36:42.347+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:36:42.347993+0000) 2024-03-04T20:36:43.337 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:36:43.347+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:36:43.348265+0000) 2024-03-04T20:36:44.337 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:36:44.348+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:36:44.348526+0000) 2024-03-04T20:36:45.337 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:36:45.348+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:36:45.348799+0000) 2024-03-04T20:36:46.338 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:36:46.348+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:36:46.349102+0000) 2024-03-04T20:36:47.338 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:36:47.349+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:36:47.349358+0000) 2024-03-04T20:36:48.338 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:36:48.349+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:36:48.349620+0000) 2024-03-04T20:36:49.339 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:36:49.349+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:36:49.349935+0000) 2024-03-04T20:36:49.567 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:36:49.579 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:36:50.339 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:36:50.350+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:36:50.350206+0000) 2024-03-04T20:36:51.339 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:36:51.350+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:36:51.350439+0000) 2024-03-04T20:36:52.340 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:36:52.349+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:36:52.350670+0000) 2024-03-04T20:36:53.340 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:36:53.350+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:36:53.350927+0000) 2024-03-04T20:36:54.340 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:36:54.350+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:36:54.351165+0000) 2024-03-04T20:36:55.340 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:36:55.350+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:36:55.351432+0000) 2024-03-04T20:36:56.341 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:36:56.350+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:36:56.351673+0000) 2024-03-04T20:36:57.341 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:36:57.350+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:36:57.351919+0000) 2024-03-04T20:36:58.341 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:36:58.351+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:36:58.352178+0000) 2024-03-04T20:36:59.341 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:36:59.351+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:36:59.352422+0000) 2024-03-04T20:37:00.342 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:37:00.350+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:37:00.352678+0000) 2024-03-04T20:37:01.342 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:37:01.351+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:37:01.352937+0000) 2024-03-04T20:37:02.342 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:37:02.352+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:37:02.353179+0000) 2024-03-04T20:37:03.343 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:37:03.352+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:37:03.353412+0000) 2024-03-04T20:37:04.343 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:37:04.352+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:37:04.353656+0000) 2024-03-04T20:37:05.343 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:37:05.353+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:37:05.353949+0000) 2024-03-04T20:37:06.343 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:37:06.353+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:37:06.354192+0000) 2024-03-04T20:37:07.344 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:37:07.353+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:37:07.354424+0000) 2024-03-04T20:37:08.344 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:37:08.353+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:37:08.354683+0000) 2024-03-04T20:37:09.344 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:37:09.354+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:37:09.354920+0000) 2024-03-04T20:37:10.344 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:37:10.354+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:37:10.355108+0000) 2024-03-04T20:37:11.345 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:37:11.354+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:37:11.355346+0000) 2024-03-04T20:37:12.345 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:37:12.354+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:37:12.355582+0000) 2024-03-04T20:37:13.345 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:37:13.355+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:37:13.355811+0000) 2024-03-04T20:37:14.346 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:37:14.355+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:37:14.356082+0000) 2024-03-04T20:37:15.346 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:37:15.355+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:37:15.356337+0000) 2024-03-04T20:37:16.346 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:37:16.355+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:37:16.356574+0000) 2024-03-04T20:37:17.346 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:37:17.356+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:37:17.356854+0000) 2024-03-04T20:37:18.347 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:37:18.356+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:37:18.357090+0000) 2024-03-04T20:37:19.347 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:37:19.356+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:37:19.357317+0000) 2024-03-04T20:37:19.706 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:37:19.716 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:37:20.347 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:37:20.356+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:37:20.357503+0000) 2024-03-04T20:37:21.347 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:37:21.357+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:37:21.357741+0000) 2024-03-04T20:37:22.348 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:37:22.357+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:37:22.357986+0000) 2024-03-04T20:37:23.348 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:37:23.357+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:37:23.358214+0000) 2024-03-04T20:37:24.350 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:37:24.357+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:37:24.358448+0000) 2024-03-04T20:37:25.349 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:37:25.358+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:37:25.358701+0000) 2024-03-04T20:37:26.349 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:37:26.358+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:37:26.358940+0000) 2024-03-04T20:37:27.349 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:37:27.358+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:37:27.359166+0000) 2024-03-04T20:37:28.349 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:37:28.358+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:37:28.359502+0000) 2024-03-04T20:37:29.350 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:37:29.359+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:37:29.359755+0000) 2024-03-04T20:37:30.350 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:37:30.359+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:37:30.359994+0000) 2024-03-04T20:37:31.350 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:37:31.359+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:37:31.360187+0000) 2024-03-04T20:37:32.350 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:37:32.359+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:37:32.360422+0000) 2024-03-04T20:37:33.351 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:37:33.360+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:37:33.360659+0000) 2024-03-04T20:37:34.351 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:37:34.360+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:37:34.360927+0000) 2024-03-04T20:37:35.351 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:37:35.360+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:37:35.361153+0000) 2024-03-04T20:37:36.352 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:37:36.360+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:37:36.361386+0000) 2024-03-04T20:37:37.352 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:37:37.361+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:37:37.361665+0000) 2024-03-04T20:37:38.352 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:37:38.361+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:37:38.361938+0000) 2024-03-04T20:37:39.352 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:37:39.361+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:37:39.362173+0000) 2024-03-04T20:37:40.352 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:37:40.361+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:37:40.362366+0000) 2024-03-04T20:37:41.352 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:37:41.361+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:37:41.362625+0000) 2024-03-04T20:37:42.353 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:37:42.362+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:37:42.362920+0000) 2024-03-04T20:37:43.353 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:37:43.362+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:37:43.363111+0000) 2024-03-04T20:37:44.353 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:37:44.363+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:37:44.363351+0000) 2024-03-04T20:37:45.354 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:37:45.362+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:37:45.363593+0000) 2024-03-04T20:37:46.354 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:37:46.363+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:37:46.363871+0000) 2024-03-04T20:37:47.354 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:37:47.363+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:37:47.364118+0000) 2024-03-04T20:37:48.354 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:37:48.364+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:37:48.364373+0000) 2024-03-04T20:37:49.355 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:37:49.364+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:37:49.364616+0000) 2024-03-04T20:37:49.844 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:37:49.853 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:37:50.355 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:37:50.364+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:37:50.364891+0000) 2024-03-04T20:37:51.355 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:37:51.364+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:37:51.365145+0000) 2024-03-04T20:37:52.355 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:37:52.365+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:37:52.365407+0000) 2024-03-04T20:37:53.356 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:37:53.364+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:37:53.365675+0000) 2024-03-04T20:37:54.356 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:37:54.365+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:37:54.365933+0000) 2024-03-04T20:37:55.356 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:37:55.366+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:37:55.366192+0000) 2024-03-04T20:37:56.356 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:37:56.366+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:37:56.366447+0000) 2024-03-04T20:37:57.357 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:37:57.365+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:37:57.366684+0000) 2024-03-04T20:37:58.357 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:37:58.365+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:37:58.366968+0000) 2024-03-04T20:37:59.357 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:37:59.366+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:37:59.367236+0000) 2024-03-04T20:38:00.358 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:38:00.366+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:38:00.367491+0000) 2024-03-04T20:38:01.358 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:38:01.366+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:38:01.367745+0000) 2024-03-04T20:38:02.358 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:38:02.366+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:38:02.367996+0000) 2024-03-04T20:38:03.359 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:38:03.367+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:38:03.368247+0000) 2024-03-04T20:38:04.359 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:38:04.366+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:38:04.368511+0000) 2024-03-04T20:38:05.359 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:38:05.366+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:38:05.368717+0000) 2024-03-04T20:38:06.359 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:38:06.368+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:38:06.368967+0000) 2024-03-04T20:38:07.360 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:38:07.368+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:38:07.369213+0000) 2024-03-04T20:38:08.360 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:38:08.368+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:38:08.369473+0000) 2024-03-04T20:38:09.360 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:38:09.368+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:38:09.369725+0000) 2024-03-04T20:38:10.361 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:38:10.369+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:38:10.369975+0000) 2024-03-04T20:38:11.361 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:38:11.369+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:38:11.370247+0000) 2024-03-04T20:38:12.361 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:38:12.369+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:38:12.370496+0000) 2024-03-04T20:38:13.361 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:38:13.369+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:38:13.370760+0000) 2024-03-04T20:38:14.361 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:38:14.370+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:38:14.370987+0000) 2024-03-04T20:38:15.362 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:38:15.370+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:38:15.371245+0000) 2024-03-04T20:38:16.362 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:38:16.370+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:38:16.371492+0000) 2024-03-04T20:38:17.363 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:38:17.370+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:38:17.371733+0000) 2024-03-04T20:38:18.363 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:38:18.371+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:38:18.371985+0000) 2024-03-04T20:38:19.363 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:38:19.371+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:38:19.372226+0000) 2024-03-04T20:38:19.976 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:38:19.985 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:38:20.363 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:38:20.371+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:38:20.372490+0000) 2024-03-04T20:38:21.364 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:38:21.372+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:38:21.372797+0000) 2024-03-04T20:38:22.364 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:38:22.372+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:38:22.373092+0000) 2024-03-04T20:38:23.364 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:38:23.372+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:38:23.373347+0000) 2024-03-04T20:38:24.364 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:38:24.372+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:38:24.373599+0000) 2024-03-04T20:38:25.365 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:38:25.373+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:38:25.373929+0000) 2024-03-04T20:38:26.365 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:38:26.373+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:38:26.374182+0000) 2024-03-04T20:38:27.365 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:38:27.373+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:38:27.374428+0000) 2024-03-04T20:38:28.366 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:38:28.374+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:38:28.374695+0000) 2024-03-04T20:38:29.366 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:38:29.374+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:38:29.374958+0000) 2024-03-04T20:38:30.366 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:38:30.374+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:38:30.375192+0000) 2024-03-04T20:38:31.366 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:38:31.374+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:38:31.375436+0000) 2024-03-04T20:38:32.367 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:38:32.375+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:38:32.375670+0000) 2024-03-04T20:38:33.367 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:38:33.375+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:38:33.375931+0000) 2024-03-04T20:38:34.367 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:38:34.375+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:38:34.376156+0000) 2024-03-04T20:38:35.367 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:38:35.375+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:38:35.376400+0000) 2024-03-04T20:38:36.368 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:38:36.376+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:38:36.376658+0000) 2024-03-04T20:38:37.368 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:38:37.376+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:38:37.376921+0000) 2024-03-04T20:38:38.368 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:38:38.376+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:38:38.377156+0000) 2024-03-04T20:38:39.368 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:38:39.376+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:38:39.377379+0000) 2024-03-04T20:38:40.369 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:38:40.377+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:38:40.377614+0000) 2024-03-04T20:38:41.369 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:38:41.377+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:38:41.377921+0000) 2024-03-04T20:38:42.369 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:38:42.377+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:38:42.378174+0000) 2024-03-04T20:38:43.369 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:38:43.377+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:38:43.378412+0000) 2024-03-04T20:38:44.370 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:38:44.378+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:38:44.378662+0000) 2024-03-04T20:38:45.370 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:38:45.378+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:38:45.378960+0000) 2024-03-04T20:38:46.371 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:38:46.378+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:38:46.379213+0000) 2024-03-04T20:38:47.371 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:38:47.379+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:38:47.379450+0000) 2024-03-04T20:38:48.371 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:38:48.379+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:38:48.379697+0000) 2024-03-04T20:38:49.371 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:38:49.379+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:38:49.379935+0000) 2024-03-04T20:38:50.173 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:38:50.180 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:38:50.371 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:38:50.379+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:38:50.380144+0000) 2024-03-04T20:38:51.372 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:38:51.380+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:38:51.380416+0000) 2024-03-04T20:38:52.372 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:38:52.380+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:38:52.380672+0000) 2024-03-04T20:38:53.372 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:38:53.380+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:38:53.380927+0000) 2024-03-04T20:38:54.372 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:38:54.380+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:38:54.381179+0000) 2024-03-04T20:38:55.373 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:38:55.380+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:38:55.381432+0000) 2024-03-04T20:38:56.373 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:38:56.380+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:38:56.381684+0000) 2024-03-04T20:38:57.373 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:38:57.381+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:38:57.381943+0000) 2024-03-04T20:38:58.373 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:38:58.382+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:38:58.382215+0000) 2024-03-04T20:38:59.374 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:38:59.381+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:38:59.382506+0000) 2024-03-04T20:39:00.374 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:39:00.381+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:39:00.382755+0000) 2024-03-04T20:39:01.374 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:39:01.381+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:39:01.383012+0000) 2024-03-04T20:39:02.375 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:39:02.382+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:39:02.383261+0000) 2024-03-04T20:39:03.375 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:39:03.382+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:39:03.383514+0000) 2024-03-04T20:39:04.375 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:39:04.382+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:39:04.383711+0000) 2024-03-04T20:39:05.375 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:39:05.382+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:39:05.383972+0000) 2024-03-04T20:39:06.376 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:39:06.382+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:39:06.384167+0000) 2024-03-04T20:39:07.376 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:39:07.383+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:39:07.384416+0000) 2024-03-04T20:39:08.376 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:39:08.383+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:39:08.384668+0000) 2024-03-04T20:39:09.377 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:39:09.383+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:39:09.385013+0000) 2024-03-04T20:39:10.377 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:39:10.384+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:39:10.385276+0000) 2024-03-04T20:39:11.377 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:39:11.384+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:39:11.385519+0000) 2024-03-04T20:39:12.377 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:39:12.384+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:39:12.385752+0000) 2024-03-04T20:39:13.378 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:39:13.385+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:39:13.385987+0000) 2024-03-04T20:39:14.378 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:39:14.385+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:39:14.386222+0000) 2024-03-04T20:39:15.378 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:39:15.385+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:39:15.386465+0000) 2024-03-04T20:39:16.378 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:39:16.385+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:39:16.386709+0000) 2024-03-04T20:39:17.379 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:39:17.386+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:39:17.386982+0000) 2024-03-04T20:39:18.379 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:39:18.386+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:39:18.387212+0000) 2024-03-04T20:39:19.379 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:39:19.386+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:39:19.387449+0000) 2024-03-04T20:39:20.370 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:39:20.378 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:39:20.379 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:39:20.386+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:39:20.387677+0000) 2024-03-04T20:39:21.380 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:39:21.387+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:39:21.387949+0000) 2024-03-04T20:39:22.380 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:39:22.387+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:39:22.388184+0000) 2024-03-04T20:39:23.380 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:39:23.387+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:39:23.388418+0000) 2024-03-04T20:39:24.380 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:39:24.387+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:39:24.388651+0000) 2024-03-04T20:39:25.381 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:39:25.388+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:39:25.388945+0000) 2024-03-04T20:39:26.381 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:39:26.388+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:39:26.389177+0000) 2024-03-04T20:39:27.381 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:39:27.388+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:39:27.389417+0000) 2024-03-04T20:39:28.382 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:39:28.389+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:39:28.389670+0000) 2024-03-04T20:39:29.382 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:39:29.389+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:39:29.389902+0000) 2024-03-04T20:39:30.382 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:39:30.389+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:39:30.390141+0000) 2024-03-04T20:39:31.382 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:39:31.389+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:39:31.390379+0000) 2024-03-04T20:39:32.383 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:39:32.390+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:39:32.390614+0000) 2024-03-04T20:39:33.383 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:39:33.390+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:39:33.390920+0000) 2024-03-04T20:39:34.383 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:39:34.390+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:39:34.391160+0000) 2024-03-04T20:39:35.383 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:39:35.390+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:39:35.391408+0000) 2024-03-04T20:39:36.384 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:39:36.391+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:39:36.391643+0000) 2024-03-04T20:39:37.384 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:39:37.391+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:39:37.391934+0000) 2024-03-04T20:39:38.384 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:39:38.391+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:39:38.392166+0000) 2024-03-04T20:39:39.384 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:39:39.391+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:39:39.392403+0000) 2024-03-04T20:39:40.385 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:39:40.391+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:39:40.392583+0000) 2024-03-04T20:39:41.385 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:39:41.392+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:39:41.392783+0000) 2024-03-04T20:39:42.385 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:39:42.392+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:39:42.393020+0000) 2024-03-04T20:39:43.385 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:39:43.392+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:39:43.393269+0000) 2024-03-04T20:39:44.386 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:39:44.393+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:39:44.393513+0000) 2024-03-04T20:39:45.386 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:39:45.393+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:39:45.393755+0000) 2024-03-04T20:39:46.386 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:39:46.393+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:39:46.393996+0000) 2024-03-04T20:39:47.386 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:39:47.393+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:39:47.394208+0000) 2024-03-04T20:39:48.387 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:39:48.394+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:39:48.394478+0000) 2024-03-04T20:39:49.387 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:39:49.393+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:39:49.394707+0000) 2024-03-04T20:39:50.387 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:39:50.394+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:39:50.394940+0000) 2024-03-04T20:39:50.517 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:39:50.525 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:39:51.387 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:39:51.394+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:39:51.395181+0000) 2024-03-04T20:39:52.388 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:39:52.395+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:39:52.395485+0000) 2024-03-04T20:39:53.388 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:39:53.394+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:39:53.395739+0000) 2024-03-04T20:39:54.388 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:39:54.395+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:39:54.395974+0000) 2024-03-04T20:39:55.389 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:39:55.396+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:39:55.396255+0000) 2024-03-04T20:39:56.389 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:39:56.396+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:39:56.396492+0000) 2024-03-04T20:39:57.389 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:39:57.396+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:39:57.396729+0000) 2024-03-04T20:39:58.389 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:39:58.396+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:39:58.396963+0000) 2024-03-04T20:39:59.390 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:39:59.397+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:39:59.397233+0000) 2024-03-04T20:40:00.390 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:40:00.397+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:40:00.397495+0000) 2024-03-04T20:40:01.390 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:40:01.397+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:40:01.397735+0000) 2024-03-04T20:40:02.390 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:40:02.396+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:40:02.397979+0000) 2024-03-04T20:40:03.391 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:40:03.396+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:40:03.398165+0000) 2024-03-04T20:40:04.391 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:40:04.397+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:40:04.398401+0000) 2024-03-04T20:40:05.391 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:40:05.397+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:40:05.398645+0000) 2024-03-04T20:40:06.391 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:40:06.397+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:40:06.398902+0000) 2024-03-04T20:40:07.392 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:40:07.398+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:40:07.399141+0000) 2024-03-04T20:40:08.392 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:40:08.398+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:40:08.399379+0000) 2024-03-04T20:40:09.392 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:40:09.398+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:40:09.399627+0000) 2024-03-04T20:40:10.393 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:40:10.398+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:40:10.399908+0000) 2024-03-04T20:40:11.393 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:40:11.399+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:40:11.400157+0000) 2024-03-04T20:40:12.393 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:40:12.399+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:40:12.400426+0000) 2024-03-04T20:40:13.393 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:40:13.399+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:40:13.400614+0000) 2024-03-04T20:40:14.394 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:40:14.400+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:40:14.400885+0000) 2024-03-04T20:40:15.394 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:40:15.400+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:40:15.401162+0000) 2024-03-04T20:40:16.394 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:40:16.400+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:40:16.401401+0000) 2024-03-04T20:40:17.395 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:40:17.400+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:40:17.401645+0000) 2024-03-04T20:40:18.395 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:40:18.401+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:40:18.401931+0000) 2024-03-04T20:40:19.395 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:40:19.401+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:40:19.402169+0000) 2024-03-04T20:40:20.395 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:40:20.401+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:40:20.402397+0000) 2024-03-04T20:40:20.781 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:40:20.790 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:40:21.395 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:40:21.401+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:40:21.402663+0000) 2024-03-04T20:40:22.396 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:40:22.402+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:40:22.402960+0000) 2024-03-04T20:40:23.396 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:40:23.402+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:40:23.403210+0000) 2024-03-04T20:40:24.396 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:40:24.402+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:40:24.403451+0000) 2024-03-04T20:40:25.397 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:40:25.402+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:40:25.403712+0000) 2024-03-04T20:40:26.397 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:40:26.403+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:40:26.403958+0000) 2024-03-04T20:40:27.397 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:40:27.403+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:40:27.404196+0000) 2024-03-04T20:40:28.397 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:40:28.403+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:40:28.404430+0000) 2024-03-04T20:40:29.398 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:40:29.403+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:40:29.404667+0000) 2024-03-04T20:40:30.398 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:40:30.404+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:40:30.404928+0000) 2024-03-04T20:40:31.398 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:40:31.404+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:40:31.405117+0000) 2024-03-04T20:40:32.399 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:40:32.404+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:40:32.405355+0000) 2024-03-04T20:40:33.399 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:40:33.404+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:40:33.405602+0000) 2024-03-04T20:40:34.399 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:40:34.405+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:40:34.405872+0000) 2024-03-04T20:40:35.399 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:40:35.405+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:40:35.406147+0000) 2024-03-04T20:40:36.400 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:40:36.405+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:40:36.406384+0000) 2024-03-04T20:40:37.400 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:40:37.406+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:40:37.406621+0000) 2024-03-04T20:40:38.400 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:40:38.406+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:40:38.406925+0000) 2024-03-04T20:40:39.400 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:40:39.406+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:40:39.407165+0000) 2024-03-04T20:40:40.400 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:40:40.406+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:40:40.407421+0000) 2024-03-04T20:40:41.401 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:40:41.406+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:40:41.407609+0000) 2024-03-04T20:40:42.401 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:40:42.407+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:40:42.407893+0000) 2024-03-04T20:40:43.401 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:40:43.407+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:40:43.408171+0000) 2024-03-04T20:40:44.402 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:40:44.407+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:40:44.408421+0000) 2024-03-04T20:40:45.402 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:40:45.407+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:40:45.408665+0000) 2024-03-04T20:40:46.402 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:40:46.408+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:40:46.408927+0000) 2024-03-04T20:40:47.402 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:40:47.408+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:40:47.409211+0000) 2024-03-04T20:40:48.403 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:40:48.409+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:40:48.409466+0000) 2024-03-04T20:40:49.403 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:40:49.409+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:40:49.409704+0000) 2024-03-04T20:40:50.403 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:40:50.409+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:40:50.409942+0000) 2024-03-04T20:40:50.919 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:40:50.927 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:40:51.403 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:40:51.409+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:40:51.410186+0000) 2024-03-04T20:40:52.404 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:40:52.410+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:40:52.410423+0000) 2024-03-04T20:40:53.404 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:40:53.410+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:40:53.410664+0000) 2024-03-04T20:40:54.404 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:40:54.410+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:40:54.410928+0000) 2024-03-04T20:40:55.405 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:40:55.410+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:40:55.411175+0000) 2024-03-04T20:40:56.405 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:40:56.411+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:40:56.411416+0000) 2024-03-04T20:40:57.405 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:40:57.411+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:40:57.411664+0000) 2024-03-04T20:40:58.405 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:40:58.411+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:40:58.411933+0000) 2024-03-04T20:40:59.406 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:40:59.411+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:40:59.412177+0000) 2024-03-04T20:41:00.406 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:41:00.412+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:41:00.412430+0000) 2024-03-04T20:41:01.406 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:41:01.412+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:41:01.412702+0000) 2024-03-04T20:41:02.406 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:41:02.412+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:41:02.412939+0000) 2024-03-04T20:41:03.406 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:41:03.413+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:41:03.413185+0000) 2024-03-04T20:41:04.407 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:41:04.412+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:41:04.413421+0000) 2024-03-04T20:41:05.407 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:41:05.412+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:41:05.413684+0000) 2024-03-04T20:41:06.407 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:41:06.412+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:41:06.413928+0000) 2024-03-04T20:41:07.408 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:41:07.413+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:41:07.414172+0000) 2024-03-04T20:41:08.408 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:41:08.412+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:41:08.414367+0000) 2024-03-04T20:41:09.408 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:41:09.412+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:41:09.414550+0000) 2024-03-04T20:41:10.408 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:41:10.412+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:41:10.414806+0000) 2024-03-04T20:41:11.408 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:41:11.414+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:41:11.415128+0000) 2024-03-04T20:41:12.409 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:41:12.413+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:41:12.415323+0000) 2024-03-04T20:41:13.409 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:41:13.414+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:41:13.415576+0000) 2024-03-04T20:41:14.409 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:41:14.414+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:41:14.415810+0000) 2024-03-04T20:41:15.410 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:41:15.415+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:41:15.416119+0000) 2024-03-04T20:41:16.410 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:41:16.415+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:41:16.416363+0000) 2024-03-04T20:41:17.410 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:41:17.415+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:41:17.416601+0000) 2024-03-04T20:41:18.410 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:41:18.416+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:41:18.416878+0000) 2024-03-04T20:41:19.411 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:41:19.416+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:41:19.417073+0000) 2024-03-04T20:41:20.411 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:41:20.416+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:41:20.417313+0000) 2024-03-04T20:41:21.059 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:41:21.067 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:41:21.411 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:41:21.416+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:41:21.417566+0000) 2024-03-04T20:41:22.411 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:41:22.416+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:41:22.417808+0000) 2024-03-04T20:41:23.412 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:41:23.417+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:41:23.418095+0000) 2024-03-04T20:41:24.412 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:41:24.417+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:41:24.418334+0000) 2024-03-04T20:41:25.412 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:41:25.417+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:41:25.418591+0000) 2024-03-04T20:41:26.413 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:41:26.418+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:41:26.418893+0000) 2024-03-04T20:41:27.413 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:41:27.418+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:41:27.419135+0000) 2024-03-04T20:41:28.413 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:41:28.418+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:41:28.419379+0000) 2024-03-04T20:41:29.413 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:41:29.418+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:41:29.419617+0000) 2024-03-04T20:41:30.413 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:41:30.419+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:41:30.419809+0000) 2024-03-04T20:41:31.414 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:41:31.419+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:41:31.420043+0000) 2024-03-04T20:41:32.414 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:41:32.419+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:41:32.420277+0000) 2024-03-04T20:41:33.414 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:41:33.419+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:41:33.420470+0000) 2024-03-04T20:41:34.415 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:41:34.419+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:41:34.420697+0000) 2024-03-04T20:41:35.415 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:41:35.420+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:41:35.420983+0000) 2024-03-04T20:41:36.415 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:41:36.420+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:41:36.421218+0000) 2024-03-04T20:41:37.415 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:41:37.420+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:41:37.421446+0000) 2024-03-04T20:41:38.415 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:41:38.420+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:41:38.421697+0000) 2024-03-04T20:41:39.416 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:41:39.421+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:41:39.421961+0000) 2024-03-04T20:41:40.416 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:41:40.421+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:41:40.422204+0000) 2024-03-04T20:41:41.416 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:41:41.421+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:41:41.422451+0000) 2024-03-04T20:41:42.417 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:41:42.421+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:41:42.422743+0000) 2024-03-04T20:41:43.417 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:41:43.422+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:41:43.423001+0000) 2024-03-04T20:41:44.417 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:41:44.422+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:41:44.423258+0000) 2024-03-04T20:41:45.417 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:41:45.423+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:41:45.423560+0000) 2024-03-04T20:41:46.418 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:41:46.423+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:41:46.423817+0000) 2024-03-04T20:41:47.418 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:41:47.423+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:41:47.424115+0000) 2024-03-04T20:41:48.418 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:41:48.423+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:41:48.424373+0000) 2024-03-04T20:41:49.419 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:41:49.424+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:41:49.424651+0000) 2024-03-04T20:41:50.419 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:41:50.424+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:41:50.424951+0000) 2024-03-04T20:41:51.264 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:41:51.276 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:41:51.419 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:41:51.424+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:41:51.425234+0000) 2024-03-04T20:41:52.419 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:41:52.425+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:41:52.425497+0000) 2024-03-04T20:41:53.420 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:41:53.424+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:41:53.425779+0000) 2024-03-04T20:41:54.420 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:41:54.425+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:41:54.426072+0000) 2024-03-04T20:41:55.420 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:41:55.425+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:41:55.426326+0000) 2024-03-04T20:41:56.421 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:41:56.426+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:41:56.426563+0000) 2024-03-04T20:41:57.421 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:41:57.426+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:41:57.426859+0000) 2024-03-04T20:41:58.421 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:41:58.426+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:41:58.427117+0000) 2024-03-04T20:41:59.421 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:41:59.426+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:41:59.427328+0000) 2024-03-04T20:42:00.422 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:42:00.427+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:42:00.427598+0000) 2024-03-04T20:42:01.422 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:42:01.427+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:42:01.427940+0000) 2024-03-04T20:42:02.422 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:42:02.427+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:42:02.428188+0000) 2024-03-04T20:42:03.422 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:42:03.428+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:42:03.428430+0000) 2024-03-04T20:42:04.423 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:42:04.427+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:42:04.428689+0000) 2024-03-04T20:42:05.423 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:42:05.428+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:42:05.428937+0000) 2024-03-04T20:42:06.423 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:42:06.428+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:42:06.429133+0000) 2024-03-04T20:42:07.424 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:42:07.428+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:42:07.429371+0000) 2024-03-04T20:42:08.424 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:42:08.428+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:42:08.429614+0000) 2024-03-04T20:42:09.424 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:42:09.428+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:42:09.429879+0000) 2024-03-04T20:42:10.425 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:42:10.429+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:42:10.430123+0000) 2024-03-04T20:42:11.425 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:42:11.429+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:42:11.430338+0000) 2024-03-04T20:42:12.425 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:42:12.429+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:42:12.430575+0000) 2024-03-04T20:42:13.425 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:42:13.429+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:42:13.430813+0000) 2024-03-04T20:42:14.426 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:42:14.430+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:42:14.431083+0000) 2024-03-04T20:42:15.426 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:42:15.430+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:42:15.431278+0000) 2024-03-04T20:42:16.426 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:42:16.429+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:42:16.431545+0000) 2024-03-04T20:42:17.426 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:42:17.430+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:42:17.431785+0000) 2024-03-04T20:42:18.426 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:42:18.431+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:42:18.432022+0000) 2024-03-04T20:42:19.427 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:42:19.431+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:42:19.432257+0000) 2024-03-04T20:42:20.427 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:42:20.431+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:42:20.432517+0000) 2024-03-04T20:42:21.427 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:42:21.431+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:42:21.432761+0000) 2024-03-04T20:42:21.478 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:42:21.485 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:42:22.427 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:42:22.432+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:42:22.432996+0000) 2024-03-04T20:42:23.427 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:42:23.432+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:42:23.433234+0000) 2024-03-04T20:42:24.428 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:42:24.432+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:42:24.433498+0000) 2024-03-04T20:42:25.428 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:42:25.432+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:42:25.433738+0000) 2024-03-04T20:42:26.428 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:42:26.433+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:42:26.434030+0000) 2024-03-04T20:42:27.429 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:42:27.433+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:42:27.434271+0000) 2024-03-04T20:42:28.429 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:42:28.433+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:42:28.434509+0000) 2024-03-04T20:42:29.429 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:42:29.433+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:42:29.434742+0000) 2024-03-04T20:42:30.429 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:42:30.434+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:42:30.434992+0000) 2024-03-04T20:42:31.430 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:42:31.434+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:42:31.435248+0000) 2024-03-04T20:42:32.430 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:42:32.434+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:42:32.435507+0000) 2024-03-04T20:42:33.431 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:42:33.434+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:42:33.435761+0000) 2024-03-04T20:42:34.431 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:42:34.435+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:42:34.435995+0000) 2024-03-04T20:42:35.431 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:42:35.435+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:42:35.436236+0000) 2024-03-04T20:42:36.431 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:42:36.435+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:42:36.436476+0000) 2024-03-04T20:42:37.432 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:42:37.436+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:42:37.436716+0000) 2024-03-04T20:42:38.432 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:42:38.436+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:42:38.436957+0000) 2024-03-04T20:42:39.432 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:42:39.436+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:42:39.437215+0000) 2024-03-04T20:42:40.432 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:42:40.436+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:42:40.437451+0000) 2024-03-04T20:42:41.432 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:42:41.437+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:42:41.437693+0000) 2024-03-04T20:42:42.432 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:42:42.437+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:42:42.437936+0000) 2024-03-04T20:42:43.433 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:42:43.437+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:42:43.438181+0000) 2024-03-04T20:42:44.433 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:42:44.437+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:42:44.438418+0000) 2024-03-04T20:42:45.433 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:42:45.438+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:42:45.438662+0000) 2024-03-04T20:42:46.434 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:42:46.438+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:42:46.438952+0000) 2024-03-04T20:42:47.434 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:42:47.438+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:42:47.439192+0000) 2024-03-04T20:42:48.434 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:42:48.438+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:42:48.439433+0000) 2024-03-04T20:42:49.434 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:42:49.439+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:42:49.439668+0000) 2024-03-04T20:42:50.435 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:42:50.439+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:42:50.439951+0000) 2024-03-04T20:42:51.435 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:42:51.439+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:42:51.440192+0000) 2024-03-04T20:42:51.617 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:42:51.625 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:42:52.435 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:42:52.440+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:42:52.440449+0000) 2024-03-04T20:42:53.436 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:42:53.440+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:42:53.440711+0000) 2024-03-04T20:42:54.436 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:42:54.440+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:42:54.440958+0000) 2024-03-04T20:42:55.436 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:42:55.440+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:42:55.441218+0000) 2024-03-04T20:42:56.437 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:42:56.441+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:42:56.441466+0000) 2024-03-04T20:42:57.437 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:42:57.441+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:42:57.441720+0000) 2024-03-04T20:42:58.437 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:42:58.441+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:42:58.441957+0000) 2024-03-04T20:42:59.437 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:42:59.441+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:42:59.442195+0000) 2024-03-04T20:43:00.437 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:43:00.442+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:43:00.442430+0000) 2024-03-04T20:43:01.438 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:43:01.442+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:43:01.442688+0000) 2024-03-04T20:43:02.438 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:43:02.442+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:43:02.442963+0000) 2024-03-04T20:43:03.438 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:43:03.442+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:43:03.443217+0000) 2024-03-04T20:43:04.438 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:43:04.442+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:43:04.443417+0000) 2024-03-04T20:43:05.438 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:43:05.443+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:43:05.443693+0000) 2024-03-04T20:43:06.439 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:43:06.443+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:43:06.443945+0000) 2024-03-04T20:43:07.439 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:43:07.444+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:43:07.444199+0000) 2024-03-04T20:43:08.439 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:43:08.444+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:43:08.444450+0000) 2024-03-04T20:43:09.440 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:43:09.443+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:43:09.444702+0000) 2024-03-04T20:43:10.440 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:43:10.443+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:43:10.444983+0000) 2024-03-04T20:43:11.440 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:43:11.444+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:43:11.445280+0000) 2024-03-04T20:43:12.441 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:43:12.444+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:43:12.445567+0000) 2024-03-04T20:43:13.441 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:43:13.444+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:43:13.445903+0000) 2024-03-04T20:43:14.441 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:43:14.445+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:43:14.446241+0000) 2024-03-04T20:43:15.442 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:43:15.445+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:43:15.446546+0000) 2024-03-04T20:43:16.442 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:43:16.445+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:43:16.446870+0000) 2024-03-04T20:43:17.442 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:43:17.446+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:43:17.447164+0000) 2024-03-04T20:43:18.443 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:43:18.445+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:43:18.447442+0000) 2024-03-04T20:43:19.443 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:43:19.446+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:43:19.447737+0000) 2024-03-04T20:43:20.443 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:43:20.447+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:43:20.448052+0000) 2024-03-04T20:43:21.443 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:43:21.447+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:43:21.448342+0000) 2024-03-04T20:43:21.831 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:43:21.840 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:43:22.444 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:43:22.447+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:43:22.448633+0000) 2024-03-04T20:43:23.444 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:43:23.448+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:43:23.448969+0000) 2024-03-04T20:43:24.445 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:43:24.448+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:43:24.449283+0000) 2024-03-04T20:43:25.445 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:43:25.448+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:43:25.449601+0000) 2024-03-04T20:43:26.445 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:43:26.449+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:43:26.449919+0000) 2024-03-04T20:43:27.445 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:43:27.449+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:43:27.450167+0000) 2024-03-04T20:43:28.446 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:43:28.449+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:43:28.450449+0000) 2024-03-04T20:43:29.446 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:43:29.449+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:43:29.450641+0000) 2024-03-04T20:43:30.446 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:43:30.450+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:43:30.450935+0000) 2024-03-04T20:43:31.446 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:43:31.450+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:43:31.451190+0000) 2024-03-04T20:43:32.446 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:43:32.450+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:43:32.451439+0000) 2024-03-04T20:43:33.447 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:43:33.450+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:43:33.451691+0000) 2024-03-04T20:43:34.447 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:43:34.451+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:43:34.451941+0000) 2024-03-04T20:43:35.447 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:43:35.451+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:43:35.452201+0000) 2024-03-04T20:43:36.448 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:43:36.451+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:43:36.452453+0000) 2024-03-04T20:43:37.448 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:43:37.452+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:43:37.452706+0000) 2024-03-04T20:43:38.448 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:43:38.452+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:43:38.452957+0000) 2024-03-04T20:43:39.448 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:43:39.452+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:43:39.453215+0000) 2024-03-04T20:43:40.449 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:43:40.452+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:43:40.453494+0000) 2024-03-04T20:43:41.449 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:43:41.452+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:43:41.453765+0000) 2024-03-04T20:43:42.449 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:43:42.453+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:43:42.454013+0000) 2024-03-04T20:43:43.450 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:43:43.453+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:43:43.454262+0000) 2024-03-04T20:43:44.450 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:43:44.453+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:43:44.454516+0000) 2024-03-04T20:43:45.450 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:43:45.453+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:43:45.454791+0000) 2024-03-04T20:43:46.450 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:43:46.454+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:43:46.455067+0000) 2024-03-04T20:43:47.451 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:43:47.454+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:43:47.455342+0000) 2024-03-04T20:43:48.451 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:43:48.455+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:43:48.455591+0000) 2024-03-04T20:43:49.451 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:43:49.455+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:43:49.455880+0000) 2024-03-04T20:43:50.452 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:43:50.455+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:43:50.456134+0000) 2024-03-04T20:43:51.452 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:43:51.455+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:43:51.456384+0000) 2024-03-04T20:43:52.036 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:43:52.044 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:43:52.452 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:43:52.456+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:43:52.456637+0000) 2024-03-04T20:43:53.452 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:43:53.456+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:43:53.456933+0000) 2024-03-04T20:43:54.453 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:43:54.456+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:43:54.457181+0000) 2024-03-04T20:43:55.453 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:43:55.457+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:43:55.457447+0000) 2024-03-04T20:43:56.453 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:43:56.456+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:43:56.457642+0000) 2024-03-04T20:43:57.453 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:43:57.457+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:43:57.457950+0000) 2024-03-04T20:43:58.454 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:43:58.457+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:43:58.458246+0000) 2024-03-04T20:43:59.454 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:43:59.458+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:43:59.458535+0000) 2024-03-04T20:44:00.454 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:44:00.458+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:44:00.458867+0000) 2024-03-04T20:44:01.455 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:44:01.458+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:44:01.459163+0000) 2024-03-04T20:44:02.455 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:44:02.459+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:44:02.459497+0000) 2024-03-04T20:44:03.455 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:44:03.459+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:44:03.459789+0000) 2024-03-04T20:44:04.456 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:44:04.459+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:44:04.460137+0000) 2024-03-04T20:44:05.456 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:44:05.460+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:44:05.460425+0000) 2024-03-04T20:44:06.457 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:44:06.460+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:44:06.460723+0000) 2024-03-04T20:44:07.457 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:44:07.460+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:44:07.461018+0000) 2024-03-04T20:44:08.457 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:44:08.461+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:44:08.461309+0000) 2024-03-04T20:44:09.457 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:44:09.461+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:44:09.461604+0000) 2024-03-04T20:44:10.458 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:44:10.461+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:44:10.461939+0000) 2024-03-04T20:44:11.458 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:44:11.462+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:44:11.462220+0000) 2024-03-04T20:44:12.459 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:44:12.461+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:44:12.462510+0000) 2024-03-04T20:44:13.459 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:44:13.461+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:44:13.462796+0000) 2024-03-04T20:44:14.459 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:44:14.462+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:44:14.463135+0000) 2024-03-04T20:44:15.459 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:44:15.462+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:44:15.463434+0000) 2024-03-04T20:44:16.461 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:44:16.462+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:44:16.463738+0000) 2024-03-04T20:44:17.460 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:44:17.462+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:44:17.464019+0000) 2024-03-04T20:44:18.460 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:44:18.463+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:44:18.464307+0000) 2024-03-04T20:44:19.460 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:44:19.463+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:44:19.464588+0000) 2024-03-04T20:44:20.461 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:44:20.463+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:44:20.464924+0000) 2024-03-04T20:44:21.461 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:44:21.464+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:44:21.465217+0000) 2024-03-04T20:44:22.258 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:44:22.266 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:44:22.461 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:44:22.464+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:44:22.465543+0000) 2024-03-04T20:44:23.462 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:44:23.465+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:44:23.465888+0000) 2024-03-04T20:44:24.462 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:44:24.465+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:44:24.466179+0000) 2024-03-04T20:44:25.462 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:44:25.465+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:44:25.466439+0000) 2024-03-04T20:44:26.462 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:44:26.465+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:44:26.466672+0000) 2024-03-04T20:44:27.463 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:44:27.466+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:44:27.466939+0000) 2024-03-04T20:44:28.463 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:44:28.466+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:44:28.467122+0000) 2024-03-04T20:44:29.463 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:44:29.466+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:44:29.467318+0000) 2024-03-04T20:44:30.463 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:44:30.466+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:44:30.467496+0000) 2024-03-04T20:44:31.464 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:44:31.466+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:44:31.467736+0000) 2024-03-04T20:44:32.464 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:44:32.467+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:44:32.467999+0000) 2024-03-04T20:44:33.464 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:44:33.467+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:44:33.468240+0000) 2024-03-04T20:44:34.464 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:44:34.467+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:44:34.468484+0000) 2024-03-04T20:44:35.465 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:44:35.467+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:44:35.468727+0000) 2024-03-04T20:44:36.465 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:44:36.468+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:44:36.468963+0000) 2024-03-04T20:44:37.465 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:44:37.468+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:44:37.469201+0000) 2024-03-04T20:44:38.465 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:44:38.468+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:44:38.469436+0000) 2024-03-04T20:44:39.466 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:44:39.468+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:44:39.469674+0000) 2024-03-04T20:44:40.466 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:44:40.469+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:44:40.469943+0000) 2024-03-04T20:44:41.466 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:44:41.469+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:44:41.470183+0000) 2024-03-04T20:44:42.466 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:44:42.469+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:44:42.470416+0000) 2024-03-04T20:44:43.467 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:44:43.470+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:44:43.470658+0000) 2024-03-04T20:44:44.467 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:44:44.470+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:44:44.470937+0000) 2024-03-04T20:44:45.467 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:44:45.470+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:44:45.471191+0000) 2024-03-04T20:44:46.467 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:44:46.470+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:44:46.471431+0000) 2024-03-04T20:44:47.468 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:44:47.471+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:44:47.471678+0000) 2024-03-04T20:44:48.468 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:44:48.471+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:44:48.471951+0000) 2024-03-04T20:44:49.469 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:44:49.471+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:44:49.472136+0000) 2024-03-04T20:44:50.470 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:44:50.471+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:44:50.472382+0000) 2024-03-04T20:44:51.469 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:44:51.471+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:44:51.472623+0000) 2024-03-04T20:44:52.446 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:44:52.455 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:44:52.469 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:44:52.471+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:44:52.472774+0000) 2024-03-04T20:44:53.469 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:44:53.472+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:44:53.473006+0000) 2024-03-04T20:44:54.470 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:44:54.472+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:44:54.473265+0000) 2024-03-04T20:44:55.469 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:44:55.473+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:44:55.473509+0000) 2024-03-04T20:44:56.470 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:44:56.472+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:44:56.473761+0000) 2024-03-04T20:44:57.470 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:44:57.473+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:44:57.474003+0000) 2024-03-04T20:44:58.470 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:44:58.473+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:44:58.474187+0000) 2024-03-04T20:44:59.470 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:44:59.474+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:44:59.474448+0000) 2024-03-04T20:45:00.471 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:45:00.474+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:45:00.474734+0000) 2024-03-04T20:45:01.471 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:45:01.474+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:45:01.475102+0000) 2024-03-04T20:45:02.472 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:45:02.475+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:45:02.475410+0000) 2024-03-04T20:45:03.472 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:45:03.475+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:45:03.475699+0000) 2024-03-04T20:45:04.472 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:45:04.475+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:45:04.475997+0000) 2024-03-04T20:45:05.473 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:45:05.476+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:45:05.476301+0000) 2024-03-04T20:45:06.473 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:45:06.476+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:45:06.476579+0000) 2024-03-04T20:45:07.473 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:45:07.476+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:45:07.476930+0000) 2024-03-04T20:45:08.474 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:45:08.476+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:45:08.477228+0000) 2024-03-04T20:45:09.474 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:45:09.477+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:45:09.477520+0000) 2024-03-04T20:45:10.474 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:45:10.477+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:45:10.477817+0000) 2024-03-04T20:45:11.475 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:45:11.477+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:45:11.478169+0000) 2024-03-04T20:45:12.475 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:45:12.478+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:45:12.478464+0000) 2024-03-04T20:45:13.475 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:45:13.478+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:45:13.478762+0000) 2024-03-04T20:45:14.476 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:45:14.477+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:45:14.479052+0000) 2024-03-04T20:45:15.476 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:45:15.478+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:45:15.479360+0000) 2024-03-04T20:45:16.476 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:45:16.478+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:45:16.479655+0000) 2024-03-04T20:45:17.477 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:45:17.478+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:45:17.479980+0000) 2024-03-04T20:45:18.477 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:45:18.479+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:45:18.480271+0000) 2024-03-04T20:45:19.477 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:45:19.479+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:45:19.480591+0000) 2024-03-04T20:45:20.477 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:45:20.479+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:45:20.480930+0000) 2024-03-04T20:45:21.478 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:45:21.480+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:45:21.481214+0000) 2024-03-04T20:45:22.478 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:45:22.480+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:45:22.481510+0000) 2024-03-04T20:45:22.585 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:45:22.594 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:45:23.478 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:45:23.480+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:45:23.481794+0000) 2024-03-04T20:45:24.479 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:45:24.481+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:45:24.482131+0000) 2024-03-04T20:45:25.479 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:45:25.481+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:45:25.482434+0000) 2024-03-04T20:45:26.479 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:45:26.481+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:45:26.482741+0000) 2024-03-04T20:45:27.480 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:45:27.482+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:45:27.483037+0000) 2024-03-04T20:45:28.480 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:45:28.482+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:45:28.483332+0000) 2024-03-04T20:45:29.480 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:45:29.482+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:45:29.483630+0000) 2024-03-04T20:45:30.480 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:45:30.483+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:45:30.483963+0000) 2024-03-04T20:45:31.481 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:45:31.483+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:45:31.484211+0000) 2024-03-04T20:45:32.481 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:45:32.483+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:45:32.484504+0000) 2024-03-04T20:45:33.482 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:45:33.483+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:45:33.484784+0000) 2024-03-04T20:45:34.482 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:45:34.484+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:45:34.485102+0000) 2024-03-04T20:45:35.482 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:45:35.484+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:45:35.485397+0000) 2024-03-04T20:45:36.482 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:45:36.484+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:45:36.485717+0000) 2024-03-04T20:45:37.483 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:45:37.485+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:45:37.486011+0000) 2024-03-04T20:45:38.483 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:45:38.485+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:45:38.486295+0000) 2024-03-04T20:45:39.483 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:45:39.485+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:45:39.486622+0000) 2024-03-04T20:45:40.483 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:45:40.486+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:45:40.486927+0000) 2024-03-04T20:45:41.484 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:45:41.486+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:45:41.487183+0000) 2024-03-04T20:45:42.484 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:45:42.486+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:45:42.487419+0000) 2024-03-04T20:45:43.485 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:45:43.487+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:45:43.487668+0000) 2024-03-04T20:45:44.485 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:45:44.487+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:45:44.487948+0000) 2024-03-04T20:45:45.485 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:45:45.487+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:45:45.488219+0000) 2024-03-04T20:45:46.485 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:45:46.487+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:45:46.488475+0000) 2024-03-04T20:45:47.486 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:45:47.487+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:45:47.488750+0000) 2024-03-04T20:45:48.486 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:45:48.488+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:45:48.488987+0000) 2024-03-04T20:45:49.486 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:45:49.488+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:45:49.489283+0000) 2024-03-04T20:45:50.486 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:45:50.488+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:45:50.489518+0000) 2024-03-04T20:45:51.487 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:45:51.489+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:45:51.489802+0000) 2024-03-04T20:45:52.487 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:45:52.489+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:45:52.490078+0000) 2024-03-04T20:45:52.733 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:45:52.742 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:45:53.487 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:45:53.489+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:45:53.490326+0000) 2024-03-04T20:45:54.488 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:45:54.490+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:45:54.490587+0000) 2024-03-04T20:45:55.488 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:45:55.490+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:45:55.490863+0000) 2024-03-04T20:45:56.488 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:45:56.490+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:45:56.491107+0000) 2024-03-04T20:45:57.488 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:45:57.490+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:45:57.491350+0000) 2024-03-04T20:45:58.488 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:45:58.490+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:45:58.491578+0000) 2024-03-04T20:45:59.489 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:45:59.491+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:45:59.491854+0000) 2024-03-04T20:46:00.490 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:46:00.491+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:46:00.492051+0000) 2024-03-04T20:46:01.489 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:46:01.491+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:46:01.492303+0000) 2024-03-04T20:46:02.489 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:46:02.491+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:46:02.492530+0000) 2024-03-04T20:46:03.490 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:46:03.492+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:46:03.492779+0000) 2024-03-04T20:46:04.490 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:46:04.492+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:46:04.493023+0000) 2024-03-04T20:46:05.490 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:46:05.492+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:46:05.493274+0000) 2024-03-04T20:46:06.491 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:46:06.493+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:46:06.493525+0000) 2024-03-04T20:46:07.490 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:46:07.493+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:46:07.493782+0000) 2024-03-04T20:46:08.490 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:46:08.493+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:46:08.493979+0000) 2024-03-04T20:46:09.491 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:46:09.493+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:46:09.494223+0000) 2024-03-04T20:46:10.491 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:46:10.494+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:46:10.494464+0000) 2024-03-04T20:46:11.492 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:46:11.494+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:46:11.494718+0000) 2024-03-04T20:46:12.492 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:46:12.494+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:46:12.494962+0000) 2024-03-04T20:46:13.492 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:46:13.495+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:46:13.495211+0000) 2024-03-04T20:46:14.493 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:46:14.495+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:46:14.495451+0000) 2024-03-04T20:46:15.494 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:46:15.495+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:46:15.495725+0000) 2024-03-04T20:46:16.493 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:46:16.494+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:46:16.496006+0000) 2024-03-04T20:46:17.493 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:46:17.495+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:46:17.496263+0000) 2024-03-04T20:46:18.493 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:46:18.495+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:46:18.496505+0000) 2024-03-04T20:46:19.494 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:46:19.494+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:46:19.496764+0000) 2024-03-04T20:46:20.494 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:46:20.495+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:46:20.497003+0000) 2024-03-04T20:46:21.494 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:46:21.496+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:46:21.497256+0000) 2024-03-04T20:46:22.494 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:46:22.496+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:46:22.497519+0000) 2024-03-04T20:46:22.945 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:46:22.954 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:46:23.494 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:46:23.496+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:46:23.497783+0000) 2024-03-04T20:46:24.495 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:46:24.497+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:46:24.498042+0000) 2024-03-04T20:46:25.495 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:46:25.497+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:46:25.498292+0000) 2024-03-04T20:46:26.495 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:46:26.497+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:46:26.498542+0000) 2024-03-04T20:46:27.496 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:46:27.497+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:46:27.498799+0000) 2024-03-04T20:46:28.496 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:46:28.498+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:46:28.499079+0000) 2024-03-04T20:46:29.496 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:46:29.498+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:46:29.499326+0000) 2024-03-04T20:46:30.497 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:46:30.498+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:46:30.499603+0000) 2024-03-04T20:46:31.497 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:46:31.499+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:46:31.499816+0000) 2024-03-04T20:46:32.497 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:46:32.499+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:46:32.500094+0000) 2024-03-04T20:46:33.497 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:46:33.499+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:46:33.500284+0000) 2024-03-04T20:46:34.498 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:46:34.499+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:46:34.500573+0000) 2024-03-04T20:46:35.498 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:46:35.500+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:46:35.500901+0000) 2024-03-04T20:46:36.498 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:46:36.500+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:46:36.501144+0000) 2024-03-04T20:46:37.499 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:46:37.500+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:46:37.501386+0000) 2024-03-04T20:46:38.499 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:46:38.500+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:46:38.501637+0000) 2024-03-04T20:46:39.499 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:46:39.501+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:46:39.501929+0000) 2024-03-04T20:46:40.499 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:46:40.501+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:46:40.502189+0000) 2024-03-04T20:46:41.499 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:46:41.501+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:46:41.502458+0000) 2024-03-04T20:46:42.500 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:46:42.501+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:46:42.502699+0000) 2024-03-04T20:46:43.500 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:46:43.502+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:46:43.502965+0000) 2024-03-04T20:46:44.500 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:46:44.502+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:46:44.503164+0000) 2024-03-04T20:46:45.501 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:46:45.502+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:46:45.503439+0000) 2024-03-04T20:46:46.501 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:46:46.503+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:46:46.503676+0000) 2024-03-04T20:46:47.501 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:46:47.503+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:46:47.503956+0000) 2024-03-04T20:46:48.501 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:46:48.503+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:46:48.504222+0000) 2024-03-04T20:46:49.502 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:46:49.503+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:46:49.504437+0000) 2024-03-04T20:46:50.502 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:46:50.503+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:46:50.504660+0000) 2024-03-04T20:46:51.502 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:46:51.504+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:46:51.504984+0000) 2024-03-04T20:46:52.503 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:46:52.504+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:46:52.505273+0000) 2024-03-04T20:46:53.084 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:46:53.092 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:46:53.502 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:46:53.504+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:46:53.505522+0000) 2024-03-04T20:46:54.503 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:46:54.504+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:46:54.505770+0000) 2024-03-04T20:46:55.503 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:46:55.505+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:46:55.506014+0000) 2024-03-04T20:46:56.504 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:46:56.505+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:46:56.506249+0000) 2024-03-04T20:46:57.504 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:46:57.505+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:46:57.506445+0000) 2024-03-04T20:46:58.504 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:46:58.505+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:46:58.506640+0000) 2024-03-04T20:46:59.504 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:46:59.506+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:46:59.506916+0000) 2024-03-04T20:47:00.504 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:47:00.506+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:47:00.507097+0000) 2024-03-04T20:47:01.504 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:47:01.506+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:47:01.507335+0000) 2024-03-04T20:47:02.505 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:47:02.507+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:47:02.507566+0000) 2024-03-04T20:47:03.505 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:47:03.507+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:47:03.507813+0000) 2024-03-04T20:47:04.505 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:47:04.507+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:47:04.508083+0000) 2024-03-04T20:47:05.506 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:47:05.508+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:47:05.508331+0000) 2024-03-04T20:47:06.506 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:47:06.508+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:47:06.508567+0000) 2024-03-04T20:47:07.506 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:47:07.508+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:47:07.508803+0000) 2024-03-04T20:47:08.506 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:47:08.508+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:47:08.509096+0000) 2024-03-04T20:47:09.507 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:47:09.509+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:47:09.509340+0000) 2024-03-04T20:47:10.507 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:47:10.509+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:47:10.509572+0000) 2024-03-04T20:47:11.507 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:47:11.509+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:47:11.509810+0000) 2024-03-04T20:47:12.508 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:47:12.509+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:47:12.510078+0000) 2024-03-04T20:47:13.508 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:47:13.510+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:47:13.510321+0000) 2024-03-04T20:47:14.508 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:47:14.510+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:47:14.510563+0000) 2024-03-04T20:47:15.508 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:47:15.510+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:47:15.510860+0000) 2024-03-04T20:47:16.510 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:47:16.510+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:47:16.511118+0000) 2024-03-04T20:47:17.509 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:47:17.511+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:47:17.511357+0000) 2024-03-04T20:47:18.509 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:47:18.511+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:47:18.511592+0000) 2024-03-04T20:47:19.509 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:47:19.510+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:47:19.511861+0000) 2024-03-04T20:47:20.509 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:47:20.510+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:47:20.512054+0000) 2024-03-04T20:47:21.510 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:47:21.511+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:47:21.512292+0000) 2024-03-04T20:47:22.510 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:47:22.511+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:47:22.512524+0000) 2024-03-04T20:47:23.219 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:47:23.228 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:47:23.510 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:47:23.511+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:47:23.512790+0000) 2024-03-04T20:47:24.510 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:47:24.512+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:47:24.513030+0000) 2024-03-04T20:47:25.511 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:47:25.512+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:47:25.513323+0000) 2024-03-04T20:47:26.511 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:47:26.512+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:47:26.513511+0000) 2024-03-04T20:47:27.511 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:47:27.512+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:47:27.513753+0000) 2024-03-04T20:47:28.511 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:47:28.513+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:47:28.513989+0000) 2024-03-04T20:47:29.512 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:47:29.513+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:47:29.514224+0000) 2024-03-04T20:47:30.512 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:47:30.513+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:47:30.514478+0000) 2024-03-04T20:47:31.512 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:47:31.513+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:47:31.514700+0000) 2024-03-04T20:47:32.512 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:47:32.514+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:47:32.514941+0000) 2024-03-04T20:47:33.513 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:47:33.514+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:47:33.515217+0000) 2024-03-04T20:47:34.513 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:47:34.514+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:47:34.515449+0000) 2024-03-04T20:47:35.513 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:47:35.514+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:47:35.515682+0000) 2024-03-04T20:47:36.513 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:47:36.515+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:47:36.515947+0000) 2024-03-04T20:47:37.514 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:47:37.515+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:47:37.516184+0000) 2024-03-04T20:47:38.514 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:47:38.515+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:47:38.516419+0000) 2024-03-04T20:47:39.514 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:47:39.515+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:47:39.516656+0000) 2024-03-04T20:47:40.514 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:47:40.516+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:47:40.516933+0000) 2024-03-04T20:47:41.515 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:47:41.516+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:47:41.517171+0000) 2024-03-04T20:47:42.515 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:47:42.516+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:47:42.517403+0000) 2024-03-04T20:47:43.515 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:47:43.516+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:47:43.517640+0000) 2024-03-04T20:47:44.516 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:47:44.517+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:47:44.517907+0000) 2024-03-04T20:47:45.516 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:47:45.517+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:47:45.518215+0000) 2024-03-04T20:47:46.516 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:47:46.517+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:47:46.518526+0000) 2024-03-04T20:47:47.516 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:47:47.518+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:47:47.518818+0000) 2024-03-04T20:47:48.517 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:47:48.518+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:47:48.519156+0000) 2024-03-04T20:47:49.517 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:47:49.518+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:47:49.519403+0000) 2024-03-04T20:47:50.517 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:47:50.519+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:47:50.519681+0000) 2024-03-04T20:47:51.517 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:47:51.519+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:47:51.519953+0000) 2024-03-04T20:47:52.518 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:47:52.519+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:47:52.520233+0000) 2024-03-04T20:47:53.359 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:47:53.368 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:47:53.518 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:47:53.519+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:47:53.520523+0000) 2024-03-04T20:47:54.518 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:47:54.520+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:47:54.520813+0000) 2024-03-04T20:47:55.519 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:47:55.520+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:47:55.521183+0000) 2024-03-04T20:47:56.519 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:47:56.520+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:47:56.521464+0000) 2024-03-04T20:47:57.519 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:47:57.521+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:47:57.521762+0000) 2024-03-04T20:47:58.520 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:47:58.521+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:47:58.522043+0000) 2024-03-04T20:47:59.520 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:47:59.521+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:47:59.522329+0000) 2024-03-04T20:48:00.521 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:48:00.522+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:48:00.522621+0000) 2024-03-04T20:48:01.521 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:48:01.522+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:48:01.522951+0000) 2024-03-04T20:48:02.521 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:48:02.522+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:48:02.523239+0000) 2024-03-04T20:48:03.521 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:48:03.522+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:48:03.523563+0000) 2024-03-04T20:48:04.522 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:48:04.523+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:48:04.523909+0000) 2024-03-04T20:48:05.522 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:48:05.523+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:48:05.524209+0000) 2024-03-04T20:48:06.522 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:48:06.524+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:48:06.524533+0000) 2024-03-04T20:48:07.523 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:48:07.524+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:48:07.524868+0000) 2024-03-04T20:48:08.523 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:48:08.524+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:48:08.525158+0000) 2024-03-04T20:48:09.523 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:48:09.525+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:48:09.525449+0000) 2024-03-04T20:48:10.523 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:48:10.525+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:48:10.525740+0000) 2024-03-04T20:48:11.524 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:48:11.525+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:48:11.526051+0000) 2024-03-04T20:48:12.524 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:48:12.526+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:48:12.526343+0000) 2024-03-04T20:48:13.525 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:48:13.525+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:48:13.526616+0000) 2024-03-04T20:48:14.525 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:48:14.526+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:48:14.526937+0000) 2024-03-04T20:48:15.525 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:48:15.526+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:48:15.527237+0000) 2024-03-04T20:48:16.526 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:48:16.527+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:48:16.527524+0000) 2024-03-04T20:48:17.526 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:48:17.527+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:48:17.527866+0000) 2024-03-04T20:48:18.526 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:48:18.527+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:48:18.528139+0000) 2024-03-04T20:48:19.526 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:48:19.528+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:48:19.528426+0000) 2024-03-04T20:48:20.527 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:48:20.528+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:48:20.528710+0000) 2024-03-04T20:48:21.527 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:48:21.527+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:48:21.529018+0000) 2024-03-04T20:48:22.527 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:48:22.528+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:48:22.529304+0000) 2024-03-04T20:48:23.527 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:48:23.528+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:48:23.529598+0000) 2024-03-04T20:48:23.566 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:48:23.574 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:48:24.528 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:48:24.528+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:48:24.529945+0000) 2024-03-04T20:48:25.528 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:48:25.529+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:48:25.530246+0000) 2024-03-04T20:48:26.528 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:48:26.529+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:48:26.530525+0000) 2024-03-04T20:48:27.528 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:48:27.529+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:48:27.530808+0000) 2024-03-04T20:48:28.529 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:48:28.530+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:48:28.531157+0000) 2024-03-04T20:48:29.529 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:48:29.530+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:48:29.531431+0000) 2024-03-04T20:48:30.530 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:48:30.530+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:48:30.531708+0000) 2024-03-04T20:48:31.530 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:48:31.531+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:48:31.532001+0000) 2024-03-04T20:48:32.531 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:48:32.531+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:48:32.532316+0000) 2024-03-04T20:48:33.530 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:48:33.531+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:48:33.532616+0000) 2024-03-04T20:48:34.531 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:48:34.532+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:48:34.532943+0000) 2024-03-04T20:48:35.531 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:48:35.532+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:48:35.533247+0000) 2024-03-04T20:48:36.531 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:48:36.532+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:48:36.533556+0000) 2024-03-04T20:48:37.532 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:48:37.533+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:48:37.533930+0000) 2024-03-04T20:48:38.532 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:48:38.533+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:48:38.534218+0000) 2024-03-04T20:48:39.532 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:48:39.533+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:48:39.534549+0000) 2024-03-04T20:48:40.533 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:48:40.534+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:48:40.534865+0000) 2024-03-04T20:48:41.533 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:48:41.534+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:48:41.535172+0000) 2024-03-04T20:48:42.533 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:48:42.534+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:48:42.535460+0000) 2024-03-04T20:48:43.534 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:48:43.534+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:48:43.535752+0000) 2024-03-04T20:48:44.534 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:48:44.535+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:48:44.536041+0000) 2024-03-04T20:48:45.534 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:48:45.535+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:48:45.536331+0000) 2024-03-04T20:48:46.535 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:48:46.535+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:48:46.536615+0000) 2024-03-04T20:48:47.535 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:48:47.536+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:48:47.536931+0000) 2024-03-04T20:48:48.535 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:48:48.536+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:48:48.537207+0000) 2024-03-04T20:48:49.536 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:48:49.536+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:48:49.537489+0000) 2024-03-04T20:48:50.536 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:48:50.537+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:48:50.537778+0000) 2024-03-04T20:48:51.536 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:48:51.537+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:48:51.538060+0000) 2024-03-04T20:48:52.537 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:48:52.537+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:48:52.538285+0000) 2024-03-04T20:48:53.537 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:48:53.537+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:48:53.538565+0000) 2024-03-04T20:48:53.722 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:48:53.730 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:48:54.537 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:48:54.538+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:48:54.538912+0000) 2024-03-04T20:48:55.538 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:48:55.538+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:48:55.539232+0000) 2024-03-04T20:48:56.538 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:48:56.538+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:48:56.539527+0000) 2024-03-04T20:48:57.538 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:48:57.539+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:48:57.539865+0000) 2024-03-04T20:48:58.538 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:48:58.539+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:48:58.540171+0000) 2024-03-04T20:48:59.538 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:48:59.539+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:48:59.540457+0000) 2024-03-04T20:49:00.539 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:49:00.540+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:49:00.540748+0000) 2024-03-04T20:49:01.539 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:49:01.540+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:49:01.541020+0000) 2024-03-04T20:49:02.539 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:49:02.540+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:49:02.541230+0000) 2024-03-04T20:49:03.540 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:49:03.541+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:49:03.541486+0000) 2024-03-04T20:49:04.540 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:49:04.541+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:49:04.541720+0000) 2024-03-04T20:49:05.540 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:49:05.541+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:49:05.541961+0000) 2024-03-04T20:49:06.540 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:49:06.541+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:49:06.542254+0000) 2024-03-04T20:49:07.540 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:49:07.541+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:49:07.542493+0000) 2024-03-04T20:49:08.541 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:49:08.542+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:49:08.542728+0000) 2024-03-04T20:49:09.541 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:49:09.542+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:49:09.542966+0000) 2024-03-04T20:49:10.541 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:49:10.542+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:49:10.543192+0000) 2024-03-04T20:49:11.542 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:49:11.543+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:49:11.543434+0000) 2024-03-04T20:49:12.542 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:49:12.543+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:49:12.543669+0000) 2024-03-04T20:49:13.542 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:49:13.543+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:49:13.543945+0000) 2024-03-04T20:49:14.543 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:49:14.543+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:49:14.544179+0000) 2024-03-04T20:49:15.543 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:49:15.544+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:49:15.544422+0000) 2024-03-04T20:49:16.543 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:49:16.544+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:49:16.544655+0000) 2024-03-04T20:49:17.543 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:49:17.544+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:49:17.544940+0000) 2024-03-04T20:49:18.543 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:49:18.544+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:49:18.545174+0000) 2024-03-04T20:49:19.543 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:49:19.545+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:49:19.545412+0000) 2024-03-04T20:49:20.544 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:49:20.545+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:49:20.545645+0000) 2024-03-04T20:49:21.544 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:49:21.545+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:49:21.545917+0000) 2024-03-04T20:49:22.544 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:49:22.545+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:49:22.546159+0000) 2024-03-04T20:49:23.545 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:49:23.545+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:49:23.546408+0000) 2024-03-04T20:49:23.944 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:49:23.956 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:49:24.545 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:49:24.545+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:49:24.546641+0000) 2024-03-04T20:49:25.545 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:49:25.545+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:49:25.546928+0000) 2024-03-04T20:49:26.546 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:49:26.546+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:49:26.547167+0000) 2024-03-04T20:49:27.546 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:49:27.545+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:49:27.547427+0000) 2024-03-04T20:49:28.546 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:49:28.546+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:49:28.547662+0000) 2024-03-04T20:49:29.546 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:49:29.546+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:49:29.547940+0000) 2024-03-04T20:49:30.546 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:49:30.547+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:49:30.548175+0000) 2024-03-04T20:49:31.547 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:49:31.547+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:49:31.548417+0000) 2024-03-04T20:49:32.547 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:49:32.547+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:49:32.548669+0000) 2024-03-04T20:49:33.547 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:49:33.547+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:49:33.548939+0000) 2024-03-04T20:49:34.547 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:49:34.548+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:49:34.549171+0000) 2024-03-04T20:49:35.548 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:49:35.548+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:49:35.549414+0000) 2024-03-04T20:49:36.548 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:49:36.548+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:49:36.549646+0000) 2024-03-04T20:49:37.548 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:49:37.549+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:49:37.549929+0000) 2024-03-04T20:49:38.548 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:49:38.549+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:49:38.550162+0000) 2024-03-04T20:49:39.549 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:49:39.549+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:49:39.550398+0000) 2024-03-04T20:49:40.549 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:49:40.549+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:49:40.550640+0000) 2024-03-04T20:49:41.549 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:49:41.550+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:49:41.550929+0000) 2024-03-04T20:49:42.549 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:49:42.550+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:49:42.551173+0000) 2024-03-04T20:49:43.550 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:49:43.550+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:49:43.551414+0000) 2024-03-04T20:49:44.550 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:49:44.550+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:49:44.551649+0000) 2024-03-04T20:49:45.550 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:49:45.551+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:49:45.551930+0000) 2024-03-04T20:49:46.550 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:49:46.551+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:49:46.552224+0000) 2024-03-04T20:49:47.551 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:49:47.551+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:49:47.552516+0000) 2024-03-04T20:49:48.551 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:49:48.552+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:49:48.552764+0000) 2024-03-04T20:49:49.551 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:49:49.552+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:49:49.553001+0000) 2024-03-04T20:49:50.552 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:49:50.552+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:49:50.553237+0000) 2024-03-04T20:49:51.552 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:49:51.552+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:49:51.553419+0000) 2024-03-04T20:49:52.552 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:49:52.553+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:49:52.553660+0000) 2024-03-04T20:49:53.552 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:49:53.553+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:49:53.553936+0000) 2024-03-04T20:49:54.090 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:49:54.098 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:49:54.552 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:49:54.553+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:49:54.554173+0000) 2024-03-04T20:49:55.553 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:49:55.553+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:49:55.554424+0000) 2024-03-04T20:49:56.553 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:49:56.554+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:49:56.554661+0000) 2024-03-04T20:49:57.553 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:49:57.554+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:49:57.554942+0000) 2024-03-04T20:49:58.553 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:49:58.554+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:49:58.555184+0000) 2024-03-04T20:49:59.554 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:49:59.554+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:49:59.555418+0000) 2024-03-04T20:50:00.554 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:50:00.554+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:50:00.555704+0000) 2024-03-04T20:50:01.554 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:50:01.555+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:50:01.555946+0000) 2024-03-04T20:50:02.555 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:50:02.555+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:50:02.556208+0000) 2024-03-04T20:50:03.555 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:50:03.555+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:50:03.556451+0000) 2024-03-04T20:50:04.555 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:50:04.556+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:50:04.556707+0000) 2024-03-04T20:50:05.555 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:50:05.556+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:50:05.556956+0000) 2024-03-04T20:50:06.556 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:50:06.556+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:50:06.557190+0000) 2024-03-04T20:50:07.556 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:50:07.557+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:50:07.557433+0000) 2024-03-04T20:50:08.556 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:50:08.557+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:50:08.557670+0000) 2024-03-04T20:50:09.556 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:50:09.557+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:50:09.557944+0000) 2024-03-04T20:50:10.556 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:50:10.557+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:50:10.558192+0000) 2024-03-04T20:50:11.557 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:50:11.558+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:50:11.558456+0000) 2024-03-04T20:50:12.558 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:50:12.558+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:50:12.558728+0000) 2024-03-04T20:50:13.557 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:50:13.558+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:50:13.559003+0000) 2024-03-04T20:50:14.558 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:50:14.558+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:50:14.559260+0000) 2024-03-04T20:50:15.559 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:50:15.559+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:50:15.559506+0000) 2024-03-04T20:50:16.558 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:50:16.559+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:50:16.559753+0000) 2024-03-04T20:50:17.558 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:50:17.559+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:50:17.559997+0000) 2024-03-04T20:50:18.559 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:50:18.559+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:50:18.560231+0000) 2024-03-04T20:50:19.559 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:50:19.560+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:50:19.560469+0000) 2024-03-04T20:50:20.559 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:50:20.560+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:50:20.560667+0000) 2024-03-04T20:50:21.559 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:50:21.560+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:50:21.560942+0000) 2024-03-04T20:50:22.560 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:50:22.561+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:50:22.561203+0000) 2024-03-04T20:50:23.560 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:50:23.561+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:50:23.561445+0000) 2024-03-04T20:50:24.287 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:50:24.298 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:50:24.560 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:50:24.561+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:50:24.561688+0000) 2024-03-04T20:50:25.561 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:50:25.561+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:50:25.561936+0000) 2024-03-04T20:50:26.561 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:50:26.561+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:50:26.562180+0000) 2024-03-04T20:50:27.561 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:50:27.560+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:50:27.562434+0000) 2024-03-04T20:50:28.561 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:50:28.561+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:50:28.562669+0000) 2024-03-04T20:50:29.561 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:50:29.561+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:50:29.562938+0000) 2024-03-04T20:50:30.562 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:50:30.562+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:50:30.563174+0000) 2024-03-04T20:50:31.562 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:50:31.562+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:50:31.563412+0000) 2024-03-04T20:50:32.562 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:50:32.562+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:50:32.563652+0000) 2024-03-04T20:50:33.563 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:50:33.562+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:50:33.563950+0000) 2024-03-04T20:50:34.563 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:50:34.563+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:50:34.564138+0000) 2024-03-04T20:50:35.563 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:50:35.563+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:50:35.564383+0000) 2024-03-04T20:50:36.564 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:50:36.563+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:50:36.564619+0000) 2024-03-04T20:50:37.563 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:50:37.564+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:50:37.564910+0000) 2024-03-04T20:50:38.565 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:50:38.564+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:50:38.565146+0000) 2024-03-04T20:50:39.564 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:50:39.564+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:50:39.565328+0000) 2024-03-04T20:50:40.564 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:50:40.564+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:50:40.565560+0000) 2024-03-04T20:50:41.564 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:50:41.564+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:50:41.565795+0000) 2024-03-04T20:50:42.564 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:50:42.565+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:50:42.566135+0000) 2024-03-04T20:50:43.565 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:50:43.565+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:50:43.566385+0000) 2024-03-04T20:50:44.565 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:50:44.565+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:50:44.566636+0000) 2024-03-04T20:50:45.566 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:50:45.566+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:50:45.566917+0000) 2024-03-04T20:50:46.566 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:50:46.566+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:50:46.567157+0000) 2024-03-04T20:50:47.566 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:50:47.566+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:50:47.567407+0000) 2024-03-04T20:50:48.566 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:50:48.566+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:50:48.567653+0000) 2024-03-04T20:50:49.566 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:50:49.567+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:50:49.567956+0000) 2024-03-04T20:50:50.567 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:50:50.567+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:50:50.568227+0000) 2024-03-04T20:50:51.567 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:50:51.567+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:50:51.568470+0000) 2024-03-04T20:50:52.567 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:50:52.568+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:50:52.568725+0000) 2024-03-04T20:50:53.568 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:50:53.568+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:50:53.568998+0000) 2024-03-04T20:50:54.450 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:50:54.458 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:50:54.568 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:50:54.568+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:50:54.569176+0000) 2024-03-04T20:50:55.568 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:50:55.568+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:50:55.569418+0000) 2024-03-04T20:50:56.568 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:50:56.569+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:50:56.569658+0000) 2024-03-04T20:50:57.569 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:50:57.569+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:50:57.569940+0000) 2024-03-04T20:50:58.569 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:50:58.569+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:50:58.570144+0000) 2024-03-04T20:50:59.569 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:50:59.569+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:50:59.570386+0000) 2024-03-04T20:51:00.569 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:51:00.569+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:51:00.570642+0000) 2024-03-04T20:51:01.570 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:51:01.570+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:51:01.570935+0000) 2024-03-04T20:51:02.570 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:51:02.570+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:51:02.571181+0000) 2024-03-04T20:51:03.570 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:51:03.570+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:51:03.571390+0000) 2024-03-04T20:51:04.571 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:51:04.571+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:51:04.571663+0000) 2024-03-04T20:51:05.572 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:51:05.571+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:51:05.571947+0000) 2024-03-04T20:51:06.571 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:51:06.571+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:51:06.572187+0000) 2024-03-04T20:51:07.571 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:51:07.571+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:51:07.572435+0000) 2024-03-04T20:51:08.571 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:51:08.572+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:51:08.572700+0000) 2024-03-04T20:51:09.572 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:51:09.572+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:51:09.572982+0000) 2024-03-04T20:51:10.572 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:51:10.572+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:51:10.573224+0000) 2024-03-04T20:51:11.572 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:51:11.573+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:51:11.573464+0000) 2024-03-04T20:51:12.572 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:51:12.573+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:51:12.573708+0000) 2024-03-04T20:51:13.573 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:51:13.573+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:51:13.573955+0000) 2024-03-04T20:51:14.573 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:51:14.573+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:51:14.574199+0000) 2024-03-04T20:51:15.573 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:51:15.574+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:51:15.574446+0000) 2024-03-04T20:51:16.573 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:51:16.574+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:51:16.574720+0000) 2024-03-04T20:51:17.574 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:51:17.574+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:51:17.574984+0000) 2024-03-04T20:51:18.574 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:51:18.574+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:51:18.575185+0000) 2024-03-04T20:51:19.575 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:51:19.575+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:51:19.575425+0000) 2024-03-04T20:51:20.575 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:51:20.575+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:51:20.575664+0000) 2024-03-04T20:51:21.577 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:51:21.575+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:51:21.575945+0000) 2024-03-04T20:51:22.575 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:51:22.575+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:51:22.576199+0000) 2024-03-04T20:51:23.575 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:51:23.576+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:51:23.576448+0000) 2024-03-04T20:51:24.575 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:51:24.576+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:51:24.576695+0000) 2024-03-04T20:51:24.680 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:51:24.689 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:51:25.576 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:51:25.576+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:51:25.576971+0000) 2024-03-04T20:51:26.576 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:51:26.577+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:51:26.577207+0000) 2024-03-04T20:51:27.576 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:51:27.576+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:51:27.577465+0000) 2024-03-04T20:51:28.577 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:51:28.576+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:51:28.577715+0000) 2024-03-04T20:51:29.577 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:51:29.576+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:51:29.577981+0000) 2024-03-04T20:51:30.577 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:51:30.577+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:51:30.578312+0000) 2024-03-04T20:51:31.577 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:51:31.577+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:51:31.578560+0000) 2024-03-04T20:51:32.577 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:51:32.576+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:51:32.578803+0000) 2024-03-04T20:51:33.578 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:51:33.577+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:51:33.579036+0000) 2024-03-04T20:51:34.578 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:51:34.578+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:51:34.579294+0000) 2024-03-04T20:51:35.578 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:51:35.578+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:51:35.579542+0000) 2024-03-04T20:51:36.579 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:51:36.578+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:51:36.579818+0000) 2024-03-04T20:51:37.579 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:51:37.579+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:51:37.580111+0000) 2024-03-04T20:51:38.580 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:51:38.579+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:51:38.580405+0000) 2024-03-04T20:51:39.580 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:51:39.579+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:51:39.580649+0000) 2024-03-04T20:51:40.580 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:51:40.580+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:51:40.580916+0000) 2024-03-04T20:51:41.580 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:51:41.580+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:51:41.581150+0000) 2024-03-04T20:51:42.581 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:51:42.580+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:51:42.581399+0000) 2024-03-04T20:51:43.581 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:51:43.580+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:51:43.581673+0000) 2024-03-04T20:51:44.581 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:51:44.581+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:51:44.581965+0000) 2024-03-04T20:51:45.581 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:51:45.581+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:51:45.582237+0000) 2024-03-04T20:51:46.582 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:51:46.581+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:51:46.582422+0000) 2024-03-04T20:51:47.582 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:51:47.581+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:51:47.582658+0000) 2024-03-04T20:51:48.582 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:51:48.582+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:51:48.582950+0000) 2024-03-04T20:51:49.582 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:51:49.582+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:51:49.583202+0000) 2024-03-04T20:51:50.582 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:51:50.582+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:51:50.583436+0000) 2024-03-04T20:51:51.583 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:51:51.582+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:51:51.583671+0000) 2024-03-04T20:51:52.583 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:51:52.583+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:51:52.583915+0000) 2024-03-04T20:51:53.583 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:51:53.583+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:51:53.584153+0000) 2024-03-04T20:51:54.583 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:51:54.583+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:51:54.584393+0000) 2024-03-04T20:51:54.927 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:51:54.935 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:51:55.584 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:51:55.583+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:51:55.584635+0000) 2024-03-04T20:51:56.584 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:51:56.584+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:51:56.584931+0000) 2024-03-04T20:51:57.584 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:51:57.584+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:51:57.585168+0000) 2024-03-04T20:51:58.584 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:51:58.584+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:51:58.585408+0000) 2024-03-04T20:51:59.585 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:51:59.585+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:51:59.585651+0000) 2024-03-04T20:52:00.585 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:52:00.585+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:52:00.585940+0000) 2024-03-04T20:52:01.585 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:52:01.585+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:52:01.586177+0000) 2024-03-04T20:52:02.585 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:52:02.585+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:52:02.586430+0000) 2024-03-04T20:52:03.586 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:52:03.585+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:52:03.586660+0000) 2024-03-04T20:52:04.586 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:52:04.586+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:52:04.586936+0000) 2024-03-04T20:52:05.586 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:52:05.586+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:52:05.587177+0000) 2024-03-04T20:52:06.586 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:52:06.586+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:52:06.587429+0000) 2024-03-04T20:52:07.587 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:52:07.586+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:52:07.587655+0000) 2024-03-04T20:52:08.587 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:52:08.587+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:52:08.587931+0000) 2024-03-04T20:52:09.587 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:52:09.587+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:52:09.588120+0000) 2024-03-04T20:52:10.587 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:52:10.587+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:52:10.588351+0000) 2024-03-04T20:52:11.588 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:52:11.588+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:52:11.588590+0000) 2024-03-04T20:52:12.588 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:52:12.588+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:52:12.588870+0000) 2024-03-04T20:52:13.588 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:52:13.588+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:52:13.589131+0000) 2024-03-04T20:52:14.588 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:52:14.589+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:52:14.589382+0000) 2024-03-04T20:52:15.589 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:52:15.589+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:52:15.589641+0000) 2024-03-04T20:52:16.589 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:52:16.589+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:52:16.589875+0000) 2024-03-04T20:52:17.589 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:52:17.589+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:52:17.590135+0000) 2024-03-04T20:52:18.589 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:52:18.589+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:52:18.590331+0000) 2024-03-04T20:52:19.590 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:52:19.590+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:52:19.590587+0000) 2024-03-04T20:52:20.590 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:52:20.590+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:52:20.590906+0000) 2024-03-04T20:52:21.590 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:52:21.590+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:52:21.591153+0000) 2024-03-04T20:52:22.590 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:52:22.591+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:52:22.591392+0000) 2024-03-04T20:52:23.591 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:52:23.591+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:52:23.591635+0000) 2024-03-04T20:52:24.591 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:52:24.591+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:52:24.591910+0000) 2024-03-04T20:52:25.149 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:52:25.158 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:52:25.591 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:52:25.591+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:52:25.592164+0000) 2024-03-04T20:52:26.591 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:52:26.592+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:52:26.592406+0000) 2024-03-04T20:52:27.592 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:52:27.592+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:52:27.592640+0000) 2024-03-04T20:52:28.592 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:52:28.592+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:52:28.592931+0000) 2024-03-04T20:52:29.592 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:52:29.593+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:52:29.593221+0000) 2024-03-04T20:52:30.593 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:52:30.593+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:52:30.593456+0000) 2024-03-04T20:52:31.593 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:52:31.592+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:52:31.593696+0000) 2024-03-04T20:52:32.593 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:52:32.592+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:52:32.593931+0000) 2024-03-04T20:52:33.593 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:52:33.592+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:52:33.594122+0000) 2024-03-04T20:52:34.594 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:52:34.593+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:52:34.594376+0000) 2024-03-04T20:52:35.594 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:52:35.592+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:52:35.594630+0000) 2024-03-04T20:52:36.594 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:52:36.593+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:52:36.594898+0000) 2024-03-04T20:52:37.594 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:52:37.594+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:52:37.595137+0000) 2024-03-04T20:52:38.595 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:52:38.594+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:52:38.595375+0000) 2024-03-04T20:52:39.595 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:52:39.594+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:52:39.595611+0000) 2024-03-04T20:52:40.595 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:52:40.594+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:52:40.595933+0000) 2024-03-04T20:52:41.595 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:52:41.595+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:52:41.596230+0000) 2024-03-04T20:52:42.596 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:52:42.595+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:52:42.596524+0000) 2024-03-04T20:52:43.596 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:52:43.596+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:52:43.596815+0000) 2024-03-04T20:52:44.597 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:52:44.596+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:52:44.597154+0000) 2024-03-04T20:52:45.597 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:52:45.596+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:52:45.597456+0000) 2024-03-04T20:52:46.597 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:52:46.596+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:52:46.597746+0000) 2024-03-04T20:52:47.597 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:52:47.597+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:52:47.598041+0000) 2024-03-04T20:52:48.598 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:52:48.597+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:52:48.598330+0000) 2024-03-04T20:52:49.598 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:52:49.597+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:52:49.598617+0000) 2024-03-04T20:52:50.598 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:52:50.598+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:52:50.598943+0000) 2024-03-04T20:52:51.599 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:52:51.598+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:52:51.599236+0000) 2024-03-04T20:52:52.599 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:52:52.598+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:52:52.599524+0000) 2024-03-04T20:52:53.599 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:52:53.599+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:52:53.599873+0000) 2024-03-04T20:52:54.599 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:52:54.599+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:52:54.600107+0000) 2024-03-04T20:52:55.363 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:52:55.371 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:52:55.599 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:52:55.599+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:52:55.600352+0000) 2024-03-04T20:52:56.600 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:52:56.599+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:52:56.600649+0000) 2024-03-04T20:52:57.601 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:52:57.600+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:52:57.600960+0000) 2024-03-04T20:52:58.601 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:52:58.600+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:52:58.601252+0000) 2024-03-04T20:52:59.601 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:52:59.600+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:52:59.601555+0000) 2024-03-04T20:53:00.601 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:53:00.601+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:53:00.601912+0000) 2024-03-04T20:53:01.601 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:53:01.601+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:53:01.602092+0000) 2024-03-04T20:53:02.601 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:53:02.601+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:53:02.602329+0000) 2024-03-04T20:53:03.602 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:53:03.601+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:53:03.602573+0000) 2024-03-04T20:53:04.602 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:53:04.602+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:53:04.602855+0000) 2024-03-04T20:53:05.602 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:53:05.602+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:53:05.603101+0000) 2024-03-04T20:53:06.603 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:53:06.602+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:53:06.603339+0000) 2024-03-04T20:53:07.603 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:53:07.603+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:53:07.603611+0000) 2024-03-04T20:53:08.603 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:53:08.603+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:53:08.603892+0000) 2024-03-04T20:53:09.603 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:53:09.603+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:53:09.604130+0000) 2024-03-04T20:53:10.604 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:53:10.603+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:53:10.604366+0000) 2024-03-04T20:53:11.604 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:53:11.603+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:53:11.604553+0000) 2024-03-04T20:53:12.604 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:53:12.604+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:53:12.604795+0000) 2024-03-04T20:53:13.604 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:53:13.604+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:53:13.605080+0000) 2024-03-04T20:53:14.605 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:53:14.604+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:53:14.605330+0000) 2024-03-04T20:53:15.605 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:53:15.605+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:53:15.605580+0000) 2024-03-04T20:53:16.605 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:53:16.605+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:53:16.605759+0000) 2024-03-04T20:53:17.605 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:53:17.605+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:53:17.606051+0000) 2024-03-04T20:53:18.605 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:53:18.605+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:53:18.606286+0000) 2024-03-04T20:53:19.606 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:53:19.605+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:53:19.606522+0000) 2024-03-04T20:53:20.606 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:53:20.606+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:53:20.606755+0000) 2024-03-04T20:53:21.606 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:53:21.606+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:53:21.606995+0000) 2024-03-04T20:53:22.606 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:53:22.606+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:53:22.607229+0000) 2024-03-04T20:53:23.607 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:53:23.607+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:53:23.607475+0000) 2024-03-04T20:53:24.607 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:53:24.607+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:53:24.607717+0000) 2024-03-04T20:53:25.559 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:53:25.571 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:53:25.607 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:53:25.607+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:53:25.607967+0000) 2024-03-04T20:53:26.607 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:53:26.607+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:53:26.608192+0000) 2024-03-04T20:53:27.608 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:53:27.608+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:53:27.608428+0000) 2024-03-04T20:53:28.608 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:53:28.607+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:53:28.608664+0000) 2024-03-04T20:53:29.608 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:53:29.608+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:53:29.608937+0000) 2024-03-04T20:53:30.608 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:53:30.609+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:53:30.609190+0000) 2024-03-04T20:53:31.609 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:53:31.609+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:53:31.609458+0000) 2024-03-04T20:53:32.609 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:53:32.609+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:53:32.609691+0000) 2024-03-04T20:53:33.609 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:53:33.609+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:53:33.609936+0000) 2024-03-04T20:53:34.609 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:53:34.609+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:53:34.610169+0000) 2024-03-04T20:53:35.610 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:53:35.608+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:53:35.610355+0000) 2024-03-04T20:53:36.610 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:53:36.608+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:53:36.610606+0000) 2024-03-04T20:53:37.610 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:53:37.609+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:53:37.610921+0000) 2024-03-04T20:53:38.610 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:53:38.610+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:53:38.611159+0000) 2024-03-04T20:53:39.611 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:53:39.610+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:53:39.611399+0000) 2024-03-04T20:53:40.611 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:53:40.610+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:53:40.611637+0000) 2024-03-04T20:53:41.612 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:53:41.610+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:53:41.611950+0000) 2024-03-04T20:53:42.612 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:53:42.611+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:53:42.612238+0000) 2024-03-04T20:53:43.612 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:53:43.610+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:53:43.612508+0000) 2024-03-04T20:53:44.612 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:53:44.611+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:53:44.612750+0000) 2024-03-04T20:53:45.612 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:53:45.612+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:53:45.613036+0000) 2024-03-04T20:53:46.613 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:53:46.612+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:53:46.613322+0000) 2024-03-04T20:53:47.613 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:53:47.612+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:53:47.613620+0000) 2024-03-04T20:53:48.613 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:53:48.613+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:53:48.613965+0000) 2024-03-04T20:53:49.614 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:53:49.613+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:53:49.614274+0000) 2024-03-04T20:53:50.614 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:53:50.613+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:53:50.614570+0000) 2024-03-04T20:53:51.614 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:53:51.614+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:53:51.614905+0000) 2024-03-04T20:53:52.615 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:53:52.614+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:53:52.615196+0000) 2024-03-04T20:53:53.615 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:53:53.614+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:53:53.615475+0000) 2024-03-04T20:53:54.615 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:53:54.614+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:53:54.615756+0000) 2024-03-04T20:53:55.616 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:53:55.615+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:53:55.616020+0000) 2024-03-04T20:53:55.697 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:53:55.705 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:53:56.616 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:53:56.615+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:53:56.616255+0000) 2024-03-04T20:53:57.616 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:53:57.615+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:53:57.616494+0000) 2024-03-04T20:53:58.616 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:53:58.616+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:53:58.616748+0000) 2024-03-04T20:53:59.616 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:53:59.616+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:53:59.617012+0000) 2024-03-04T20:54:00.617 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:54:00.616+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:54:00.617245+0000) 2024-03-04T20:54:01.617 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:54:01.616+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:54:01.617487+0000) 2024-03-04T20:54:02.618 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:54:02.616+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:54:02.617797+0000) 2024-03-04T20:54:03.618 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:54:03.617+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:54:03.618121+0000) 2024-03-04T20:54:04.618 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:54:04.617+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:54:04.618356+0000) 2024-03-04T20:54:05.618 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:54:05.617+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:54:05.618600+0000) 2024-03-04T20:54:06.618 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:54:06.618+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:54:06.618863+0000) 2024-03-04T20:54:07.619 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:54:07.618+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:54:07.619113+0000) 2024-03-04T20:54:08.619 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:54:08.618+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:54:08.619347+0000) 2024-03-04T20:54:09.619 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:54:09.618+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:54:09.619580+0000) 2024-03-04T20:54:10.619 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:54:10.619+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:54:10.619904+0000) 2024-03-04T20:54:11.620 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:54:11.619+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:54:11.620140+0000) 2024-03-04T20:54:12.625 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:54:12.619+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:54:12.620392+0000) 2024-03-04T20:54:13.620 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:54:13.619+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:54:13.620666+0000) 2024-03-04T20:54:14.620 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:54:14.620+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:54:14.620938+0000) 2024-03-04T20:54:15.621 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:54:15.620+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:54:15.621204+0000) 2024-03-04T20:54:16.621 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:54:16.621+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:54:16.621439+0000) 2024-03-04T20:54:17.621 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:54:17.621+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:54:17.621674+0000) 2024-03-04T20:54:18.621 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:54:18.621+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:54:18.621947+0000) 2024-03-04T20:54:19.622 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:54:19.621+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:54:19.622178+0000) 2024-03-04T20:54:20.622 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:54:20.622+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:54:20.622414+0000) 2024-03-04T20:54:21.622 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:54:21.622+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:54:21.622658+0000) 2024-03-04T20:54:22.623 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:54:22.622+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:54:22.622935+0000) 2024-03-04T20:54:23.623 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:54:23.622+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:54:23.623175+0000) 2024-03-04T20:54:24.623 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:54:24.623+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:54:24.623408+0000) 2024-03-04T20:54:25.624 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:54:25.623+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:54:25.623650+0000) 2024-03-04T20:54:25.837 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:54:25.845 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:54:26.623 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:54:26.623+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:54:26.623927+0000) 2024-03-04T20:54:27.624 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:54:27.623+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:54:27.624158+0000) 2024-03-04T20:54:28.624 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:54:28.624+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:54:28.624414+0000) 2024-03-04T20:54:29.624 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:54:29.624+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:54:29.624673+0000) 2024-03-04T20:54:30.624 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:54:30.624+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:54:30.624944+0000) 2024-03-04T20:54:31.625 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:54:31.624+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:54:31.625152+0000) 2024-03-04T20:54:32.625 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:54:32.625+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:54:32.625389+0000) 2024-03-04T20:54:33.625 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:54:33.625+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:54:33.625636+0000) 2024-03-04T20:54:34.625 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:54:34.625+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:54:34.625923+0000) 2024-03-04T20:54:35.626 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:54:35.625+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:54:35.626124+0000) 2024-03-04T20:54:36.626 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:54:36.624+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:54:36.626316+0000) 2024-03-04T20:54:37.626 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:54:37.625+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:54:37.626553+0000) 2024-03-04T20:54:38.626 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:54:38.625+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:54:38.626790+0000) 2024-03-04T20:54:39.627 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:54:39.625+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:54:39.627031+0000) 2024-03-04T20:54:40.627 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:54:40.626+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:54:40.627266+0000) 2024-03-04T20:54:41.627 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:54:41.626+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:54:41.627511+0000) 2024-03-04T20:54:42.627 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:54:42.626+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:54:42.627750+0000) 2024-03-04T20:54:43.628 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:54:43.626+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:54:43.627998+0000) 2024-03-04T20:54:44.628 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:54:44.627+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:54:44.628234+0000) 2024-03-04T20:54:45.628 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:54:45.627+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:54:45.628476+0000) 2024-03-04T20:54:46.628 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:54:46.627+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:54:46.628711+0000) 2024-03-04T20:54:47.628 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:54:47.628+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:54:47.628949+0000) 2024-03-04T20:54:48.629 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:54:48.628+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:54:48.629190+0000) 2024-03-04T20:54:49.629 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:54:49.628+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:54:49.629431+0000) 2024-03-04T20:54:50.629 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:54:50.628+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:54:50.629700+0000) 2024-03-04T20:54:51.629 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:54:51.629+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:54:51.629943+0000) 2024-03-04T20:54:52.630 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:54:52.629+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:54:52.630189+0000) 2024-03-04T20:54:53.630 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:54:53.629+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:54:53.630497+0000) 2024-03-04T20:54:54.630 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:54:54.629+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:54:54.630754+0000) 2024-03-04T20:54:55.631 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:54:55.630+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:54:55.630957+0000) 2024-03-04T20:54:55.975 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:54:55.983 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:54:56.631 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:54:56.630+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:54:56.631185+0000) 2024-03-04T20:54:57.631 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:54:57.630+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:54:57.631427+0000) 2024-03-04T20:54:58.631 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:54:58.630+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:54:58.631672+0000) 2024-03-04T20:54:59.632 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:54:59.631+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:54:59.631935+0000) 2024-03-04T20:55:00.632 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:55:00.631+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:55:00.632195+0000) 2024-03-04T20:55:01.632 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:55:01.631+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:55:01.632377+0000) 2024-03-04T20:55:02.633 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:55:02.631+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:55:02.632624+0000) 2024-03-04T20:55:03.633 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:55:03.632+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:55:03.632917+0000) 2024-03-04T20:55:04.633 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:55:04.632+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:55:04.633160+0000) 2024-03-04T20:55:05.633 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:55:05.632+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:55:05.633357+0000) 2024-03-04T20:55:06.633 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:55:06.632+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:55:06.633601+0000) 2024-03-04T20:55:07.634 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:55:07.633+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:55:07.633867+0000) 2024-03-04T20:55:08.634 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:55:08.633+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:55:08.634118+0000) 2024-03-04T20:55:09.634 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:55:09.633+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:55:09.634357+0000) 2024-03-04T20:55:10.634 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:55:10.634+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:55:10.634595+0000) 2024-03-04T20:55:11.634 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:55:11.634+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:55:11.634863+0000) 2024-03-04T20:55:12.635 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:55:12.634+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:55:12.635112+0000) 2024-03-04T20:55:13.635 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:55:13.634+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:55:13.635358+0000) 2024-03-04T20:55:14.635 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:55:14.634+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:55:14.635542+0000) 2024-03-04T20:55:15.636 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:55:15.635+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:55:15.635814+0000) 2024-03-04T20:55:16.636 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:55:16.635+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:55:16.636101+0000) 2024-03-04T20:55:17.636 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:55:17.635+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:55:17.636348+0000) 2024-03-04T20:55:18.636 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:55:18.635+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:55:18.636584+0000) 2024-03-04T20:55:19.637 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:55:19.636+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:55:19.636887+0000) 2024-03-04T20:55:20.637 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:55:20.636+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:55:20.637124+0000) 2024-03-04T20:55:21.637 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:55:21.636+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:55:21.637365+0000) 2024-03-04T20:55:22.637 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:55:22.637+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:55:22.637604+0000) 2024-03-04T20:55:23.638 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:55:23.637+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:55:23.637886+0000) 2024-03-04T20:55:24.638 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:55:24.637+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:55:24.638131+0000) 2024-03-04T20:55:25.638 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:55:25.638+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:55:25.638382+0000) 2024-03-04T20:55:26.106 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:55:26.114 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:55:26.639 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:55:26.638+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:55:26.638623+0000) 2024-03-04T20:55:27.639 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:55:27.638+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:55:27.638903+0000) 2024-03-04T20:55:28.639 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:55:28.638+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:55:28.639160+0000) 2024-03-04T20:55:29.639 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:55:29.639+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:55:29.639407+0000) 2024-03-04T20:55:30.640 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:55:30.639+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:55:30.639675+0000) 2024-03-04T20:55:31.640 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:55:31.639+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:55:31.639961+0000) 2024-03-04T20:55:32.640 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:55:32.639+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:55:32.640156+0000) 2024-03-04T20:55:33.640 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:55:33.640+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:55:33.640421+0000) 2024-03-04T20:55:34.640 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:55:34.640+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:55:34.640667+0000) 2024-03-04T20:55:35.641 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:55:35.640+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:55:35.640978+0000) 2024-03-04T20:55:36.641 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:55:36.641+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:55:36.641222+0000) 2024-03-04T20:55:37.641 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:55:37.640+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:55:37.641407+0000) 2024-03-04T20:55:38.642 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:55:38.641+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:55:38.641658+0000) 2024-03-04T20:55:39.642 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:55:39.640+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:55:39.641959+0000) 2024-03-04T20:55:40.642 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:55:40.640+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:55:40.642190+0000) 2024-03-04T20:55:41.642 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:55:41.641+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:55:41.642435+0000) 2024-03-04T20:55:42.642 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:55:42.640+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:55:42.642690+0000) 2024-03-04T20:55:43.643 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:55:43.641+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:55:43.642949+0000) 2024-03-04T20:55:44.643 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:55:44.642+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:55:44.643200+0000) 2024-03-04T20:55:45.643 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:55:45.642+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:55:45.643460+0000) 2024-03-04T20:55:46.644 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:55:46.642+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:55:46.643702+0000) 2024-03-04T20:55:47.644 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:55:47.642+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:55:47.643950+0000) 2024-03-04T20:55:48.644 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:55:48.643+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:55:48.644217+0000) 2024-03-04T20:55:49.644 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:55:49.643+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:55:49.644463+0000) 2024-03-04T20:55:50.645 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:55:50.643+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:55:50.644727+0000) 2024-03-04T20:55:51.645 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:55:51.644+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:55:51.644997+0000) 2024-03-04T20:55:52.645 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:55:52.644+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:55:52.645241+0000) 2024-03-04T20:55:53.645 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:55:53.644+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:55:53.645442+0000) 2024-03-04T20:55:54.646 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:55:54.644+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:55:54.645693+0000) 2024-03-04T20:55:55.646 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:55:55.645+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:55:55.645947+0000) 2024-03-04T20:55:56.236 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:55:56.244 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:55:56.646 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:55:56.645+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:55:56.646224+0000) 2024-03-04T20:55:57.646 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:55:57.645+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:55:57.646466+0000) 2024-03-04T20:55:58.647 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:55:58.645+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:55:58.646712+0000) 2024-03-04T20:55:59.647 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:55:59.646+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:55:59.646962+0000) 2024-03-04T20:56:00.647 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:56:00.646+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:56:00.647224+0000) 2024-03-04T20:56:01.647 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:56:01.646+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:56:01.647462+0000) 2024-03-04T20:56:02.648 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:56:02.646+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:56:02.647706+0000) 2024-03-04T20:56:03.648 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:56:03.647+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:56:03.647973+0000) 2024-03-04T20:56:04.648 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:56:04.647+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:56:04.648180+0000) 2024-03-04T20:56:05.648 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:56:05.647+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:56:05.648388+0000) 2024-03-04T20:56:06.649 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:56:06.647+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:56:06.648634+0000) 2024-03-04T20:56:07.649 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:56:07.648+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:56:07.648936+0000) 2024-03-04T20:56:08.649 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:56:08.648+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:56:08.649158+0000) 2024-03-04T20:56:09.649 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:56:09.648+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:56:09.649429+0000) 2024-03-04T20:56:10.650 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:56:10.648+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:56:10.649683+0000) 2024-03-04T20:56:11.650 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:56:11.649+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:56:11.649959+0000) 2024-03-04T20:56:12.650 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:56:12.649+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:56:12.650160+0000) 2024-03-04T20:56:13.650 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:56:13.649+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:56:13.650426+0000) 2024-03-04T20:56:14.651 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:56:14.649+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:56:14.650697+0000) 2024-03-04T20:56:15.651 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:56:15.650+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:56:15.650939+0000) 2024-03-04T20:56:16.651 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:56:16.650+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:56:16.651177+0000) 2024-03-04T20:56:17.651 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:56:17.650+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:56:17.651363+0000) 2024-03-04T20:56:18.651 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:56:18.650+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:56:18.651552+0000) 2024-03-04T20:56:19.652 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:56:19.651+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:56:19.651863+0000) 2024-03-04T20:56:20.652 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:56:20.651+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:56:20.652118+0000) 2024-03-04T20:56:21.652 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:56:21.651+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:56:21.652332+0000) 2024-03-04T20:56:22.653 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:56:22.652+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:56:22.652585+0000) 2024-03-04T20:56:23.653 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:56:23.652+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:56:23.652853+0000) 2024-03-04T20:56:24.653 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:56:24.652+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:56:24.653059+0000) 2024-03-04T20:56:25.653 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:56:25.652+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:56:25.653307+0000) 2024-03-04T20:56:26.365 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:56:26.374 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:56:26.653 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:56:26.653+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:56:26.653553+0000) 2024-03-04T20:56:27.654 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:56:27.652+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:56:27.653806+0000) 2024-03-04T20:56:28.654 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:56:28.653+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:56:28.654074+0000) 2024-03-04T20:56:29.654 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:56:29.653+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:56:29.654267+0000) 2024-03-04T20:56:30.655 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:56:30.654+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:56:30.654522+0000) 2024-03-04T20:56:31.655 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:56:31.653+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:56:31.654793+0000) 2024-03-04T20:56:32.655 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:56:32.654+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:56:32.655092+0000) 2024-03-04T20:56:33.655 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:56:33.655+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:56:33.655345+0000) 2024-03-04T20:56:34.656 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:56:34.655+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:56:34.655626+0000) 2024-03-04T20:56:35.656 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:56:35.655+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:56:35.655870+0000) 2024-03-04T20:56:36.656 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:56:36.655+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:56:36.656108+0000) 2024-03-04T20:56:37.656 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:56:37.656+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:56:37.656352+0000) 2024-03-04T20:56:38.657 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:56:38.656+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:56:38.656585+0000) 2024-03-04T20:56:39.657 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:56:39.656+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:56:39.656853+0000) 2024-03-04T20:56:40.657 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:56:40.657+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:56:40.657144+0000) 2024-03-04T20:56:41.657 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:56:41.657+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:56:41.657386+0000) 2024-03-04T20:56:42.658 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:56:42.656+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:56:42.657621+0000) 2024-03-04T20:56:43.658 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:56:43.656+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:56:43.657913+0000) 2024-03-04T20:56:44.658 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:56:44.657+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:56:44.658183+0000) 2024-03-04T20:56:45.658 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:56:45.657+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:56:45.658393+0000) 2024-03-04T20:56:46.659 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:56:46.657+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:56:46.658630+0000) 2024-03-04T20:56:47.659 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:56:47.657+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:56:47.658898+0000) 2024-03-04T20:56:48.659 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:56:48.657+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:56:48.659094+0000) 2024-03-04T20:56:49.659 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:56:49.658+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:56:49.659333+0000) 2024-03-04T20:56:50.660 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:56:50.658+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:56:50.659568+0000) 2024-03-04T20:56:51.660 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:56:51.658+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:56:51.659806+0000) 2024-03-04T20:56:52.660 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:56:52.659+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:56:52.660074+0000) 2024-03-04T20:56:53.660 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:56:53.659+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:56:53.660310+0000) 2024-03-04T20:56:54.661 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:56:54.659+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:56:54.660545+0000) 2024-03-04T20:56:55.661 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:56:55.659+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:56:55.660801+0000) 2024-03-04T20:56:56.504 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:56:56.511 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:56:56.661 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:56:56.660+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:56:56.661110+0000) 2024-03-04T20:56:57.661 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:56:57.660+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:56:57.661349+0000) 2024-03-04T20:56:58.662 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:56:58.660+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:56:58.661580+0000) 2024-03-04T20:56:59.662 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:56:59.661+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:56:59.661856+0000) 2024-03-04T20:57:00.662 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:57:00.661+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:57:00.662115+0000) 2024-03-04T20:57:01.663 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:57:01.661+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:57:01.662374+0000) 2024-03-04T20:57:02.663 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:57:02.661+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:57:02.662605+0000) 2024-03-04T20:57:03.663 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:57:03.662+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:57:03.662878+0000) 2024-03-04T20:57:04.663 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:57:04.662+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:57:04.663118+0000) 2024-03-04T20:57:05.663 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:57:05.662+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:57:05.663357+0000) 2024-03-04T20:57:06.664 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:57:06.662+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:57:06.663593+0000) 2024-03-04T20:57:07.664 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:57:07.663+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:57:07.663871+0000) 2024-03-04T20:57:08.664 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:57:08.663+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:57:08.664062+0000) 2024-03-04T20:57:09.664 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:57:09.663+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:57:09.664323+0000) 2024-03-04T20:57:10.665 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:57:10.663+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:57:10.664558+0000) 2024-03-04T20:57:11.665 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:57:11.664+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:57:11.664795+0000) 2024-03-04T20:57:12.665 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:57:12.664+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:57:12.665115+0000) 2024-03-04T20:57:13.665 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:57:13.664+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:57:13.665351+0000) 2024-03-04T20:57:14.666 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:57:14.664+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:57:14.665588+0000) 2024-03-04T20:57:15.666 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:57:15.665+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:57:15.665874+0000) 2024-03-04T20:57:16.666 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:57:16.665+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:57:16.666115+0000) 2024-03-04T20:57:17.666 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:57:17.665+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:57:17.666359+0000) 2024-03-04T20:57:18.667 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:57:18.666+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:57:18.666593+0000) 2024-03-04T20:57:19.667 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:57:19.666+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:57:19.666872+0000) 2024-03-04T20:57:20.667 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:57:20.666+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:57:20.667152+0000) 2024-03-04T20:57:21.668 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:57:21.666+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:57:21.667443+0000) 2024-03-04T20:57:22.668 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:57:22.667+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:57:22.667721+0000) 2024-03-04T20:57:23.668 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:57:23.667+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:57:23.668000+0000) 2024-03-04T20:57:24.669 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:57:24.667+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:57:24.668277+0000) 2024-03-04T20:57:25.669 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:57:25.668+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:57:25.668570+0000) 2024-03-04T20:57:26.648 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:57:26.655 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:57:26.669 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:57:26.668+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:57:26.668906+0000) 2024-03-04T20:57:27.669 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:57:27.668+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:57:27.669196+0000) 2024-03-04T20:57:28.670 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:57:28.669+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:57:28.669571+0000) 2024-03-04T20:57:29.670 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:57:29.669+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:57:29.669923+0000) 2024-03-04T20:57:30.670 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:57:30.669+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:57:30.670212+0000) 2024-03-04T20:57:31.671 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:57:31.670+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:57:31.670495+0000) 2024-03-04T20:57:32.671 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:57:32.670+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:57:32.670788+0000) 2024-03-04T20:57:33.671 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:57:33.670+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:57:33.671129+0000) 2024-03-04T20:57:34.672 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:57:34.671+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:57:34.671414+0000) 2024-03-04T20:57:35.672 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:57:35.671+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:57:35.671720+0000) 2024-03-04T20:57:36.672 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:57:36.671+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:57:36.672008+0000) 2024-03-04T20:57:37.673 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:57:37.672+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:57:37.672299+0000) 2024-03-04T20:57:38.673 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:57:38.672+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:57:38.672589+0000) 2024-03-04T20:57:39.673 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:57:39.672+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:57:39.672916+0000) 2024-03-04T20:57:40.674 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:57:40.672+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:57:40.673210+0000) 2024-03-04T20:57:41.674 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:57:41.673+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:57:41.673499+0000) 2024-03-04T20:57:42.674 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:57:42.673+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:57:42.673787+0000) 2024-03-04T20:57:43.674 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:57:43.673+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:57:43.674154+0000) 2024-03-04T20:57:44.675 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:57:44.673+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:57:44.674452+0000) 2024-03-04T20:57:45.675 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:57:45.673+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:57:45.674746+0000) 2024-03-04T20:57:46.675 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:57:46.673+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:57:46.675033+0000) 2024-03-04T20:57:47.675 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:57:47.674+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:57:47.675321+0000) 2024-03-04T20:57:48.676 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:57:48.674+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:57:48.675616+0000) 2024-03-04T20:57:49.676 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:57:49.674+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:57:49.675950+0000) 2024-03-04T20:57:50.676 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:57:50.675+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:57:50.676240+0000) 2024-03-04T20:57:51.677 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:57:51.674+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:57:51.676545+0000) 2024-03-04T20:57:52.677 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:57:52.675+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:57:52.676872+0000) 2024-03-04T20:57:53.677 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:57:53.676+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:57:53.677155+0000) 2024-03-04T20:57:54.678 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:57:54.676+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:57:54.677457+0000) 2024-03-04T20:57:55.678 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:57:55.676+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:57:55.677759+0000) 2024-03-04T20:57:56.678 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:57:56.677+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:57:56.678045+0000) 2024-03-04T20:57:56.782 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:57:56.790 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:57:57.679 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:57:57.677+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:57:57.678338+0000) 2024-03-04T20:57:58.679 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:57:58.677+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:57:58.678650+0000) 2024-03-04T20:57:59.679 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:57:59.678+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:57:59.678971+0000) 2024-03-04T20:58:00.680 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:58:00.678+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:58:00.679299+0000) 2024-03-04T20:58:01.680 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:58:01.678+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:58:01.679590+0000) 2024-03-04T20:58:02.680 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:58:02.679+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:58:02.679923+0000) 2024-03-04T20:58:03.680 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:58:03.679+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:58:03.680158+0000) 2024-03-04T20:58:04.681 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:58:04.679+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:58:04.680442+0000) 2024-03-04T20:58:05.681 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:58:05.679+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:58:05.680741+0000) 2024-03-04T20:58:06.681 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:58:06.680+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:58:06.681039+0000) 2024-03-04T20:58:07.682 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:58:07.680+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:58:07.681328+0000) 2024-03-04T20:58:08.682 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:58:08.680+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:58:08.681616+0000) 2024-03-04T20:58:09.683 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:58:09.681+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:58:09.681944+0000) 2024-03-04T20:58:10.683 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:58:10.681+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:58:10.682277+0000) 2024-03-04T20:58:11.683 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:58:11.681+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:58:11.682568+0000) 2024-03-04T20:58:12.683 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:58:12.682+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:58:12.682901+0000) 2024-03-04T20:58:13.684 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:58:13.682+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:58:13.683188+0000) 2024-03-04T20:58:14.684 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:58:14.682+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:58:14.683483+0000) 2024-03-04T20:58:15.684 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:58:15.683+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:58:15.683777+0000) 2024-03-04T20:58:16.686 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:58:16.683+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:58:16.684114+0000) 2024-03-04T20:58:17.685 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:58:17.683+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:58:17.684410+0000) 2024-03-04T20:58:18.685 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:58:18.683+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:58:18.684687+0000) 2024-03-04T20:58:19.685 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:58:19.684+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:58:19.684990+0000) 2024-03-04T20:58:20.686 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:58:20.684+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:58:20.685305+0000) 2024-03-04T20:58:21.686 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:58:21.685+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:58:21.685601+0000) 2024-03-04T20:58:22.686 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:58:22.685+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:58:22.685917+0000) 2024-03-04T20:58:23.687 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:58:23.685+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:58:23.686211+0000) 2024-03-04T20:58:24.687 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:58:24.685+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:58:24.686501+0000) 2024-03-04T20:58:25.687 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:58:25.686+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:58:25.686789+0000) 2024-03-04T20:58:26.688 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:58:26.686+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:58:26.687073+0000) 2024-03-04T20:58:26.920 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:58:26.929 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:58:27.688 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:58:27.686+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:58:27.687307+0000) 2024-03-04T20:58:28.688 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:58:28.686+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:58:28.687533+0000) 2024-03-04T20:58:29.688 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:58:29.687+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:58:29.687764+0000) 2024-03-04T20:58:30.689 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:58:30.687+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:58:30.688076+0000) 2024-03-04T20:58:31.689 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:58:31.687+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:58:31.688373+0000) 2024-03-04T20:58:32.689 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:58:32.688+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:58:32.688666+0000) 2024-03-04T20:58:33.689 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:58:33.688+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:58:33.688969+0000) 2024-03-04T20:58:34.690 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:58:34.688+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:58:34.689219+0000) 2024-03-04T20:58:35.690 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:58:35.689+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:58:35.689466+0000) 2024-03-04T20:58:36.690 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:58:36.689+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:58:36.689703+0000) 2024-03-04T20:58:37.691 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:58:37.689+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:58:37.689941+0000) 2024-03-04T20:58:38.691 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:58:38.689+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:58:38.690177+0000) 2024-03-04T20:58:39.691 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:58:39.689+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:58:39.690435+0000) 2024-03-04T20:58:40.691 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:58:40.690+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:58:40.690674+0000) 2024-03-04T20:58:41.692 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:58:41.690+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:58:41.690956+0000) 2024-03-04T20:58:42.692 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:58:42.690+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:58:42.691150+0000) 2024-03-04T20:58:43.692 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:58:43.690+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:58:43.691344+0000) 2024-03-04T20:58:44.692 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:58:44.690+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:58:44.691591+0000) 2024-03-04T20:58:45.692 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:58:45.691+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:58:45.691886+0000) 2024-03-04T20:58:46.693 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:58:46.692+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:58:46.692130+0000) 2024-03-04T20:58:47.693 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:58:47.691+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:58:47.692370+0000) 2024-03-04T20:58:48.693 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:58:48.691+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:58:48.692633+0000) 2024-03-04T20:58:49.693 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:58:49.691+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:58:49.692946+0000) 2024-03-04T20:58:50.694 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:58:50.692+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:58:50.693192+0000) 2024-03-04T20:58:51.694 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:58:51.692+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:58:51.693443+0000) 2024-03-04T20:58:52.694 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:58:52.692+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:58:52.693685+0000) 2024-03-04T20:58:53.694 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:58:53.692+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:58:53.693928+0000) 2024-03-04T20:58:54.695 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:58:54.693+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:58:54.694166+0000) 2024-03-04T20:58:55.695 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:58:55.693+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:58:55.694370+0000) 2024-03-04T20:58:56.695 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:58:56.693+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:58:56.694613+0000) 2024-03-04T20:58:57.151 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:58:57.163 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:58:57.696 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:58:57.693+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:58:57.694913+0000) 2024-03-04T20:58:58.696 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:58:58.694+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:58:58.695162+0000) 2024-03-04T20:58:59.696 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:58:59.694+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:58:59.695403+0000) 2024-03-04T20:59:00.696 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:59:00.694+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:59:00.695648+0000) 2024-03-04T20:59:01.697 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:59:01.695+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:59:01.695936+0000) 2024-03-04T20:59:02.697 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:59:02.695+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:59:02.696175+0000) 2024-03-04T20:59:03.697 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:59:03.695+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:59:03.696434+0000) 2024-03-04T20:59:04.697 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:59:04.695+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:59:04.696712+0000) 2024-03-04T20:59:05.698 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:59:05.696+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:59:05.696998+0000) 2024-03-04T20:59:06.698 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:59:06.696+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:59:06.697293+0000) 2024-03-04T20:59:07.698 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:59:07.696+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:59:07.697612+0000) 2024-03-04T20:59:08.699 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:59:08.697+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:59:08.697981+0000) 2024-03-04T20:59:09.699 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:59:09.697+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:59:09.698280+0000) 2024-03-04T20:59:10.699 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:59:10.697+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:59:10.698591+0000) 2024-03-04T20:59:11.700 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:59:11.698+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:59:11.698931+0000) 2024-03-04T20:59:12.700 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:59:12.698+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:59:12.699175+0000) 2024-03-04T20:59:13.700 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:59:13.698+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:59:13.699425+0000) 2024-03-04T20:59:14.700 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:59:14.698+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:59:14.699674+0000) 2024-03-04T20:59:15.700 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:59:15.699+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:59:15.699952+0000) 2024-03-04T20:59:16.701 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:59:16.699+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:59:16.700224+0000) 2024-03-04T20:59:17.701 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:59:17.699+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:59:17.700488+0000) 2024-03-04T20:59:18.701 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:59:18.700+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:59:18.700723+0000) 2024-03-04T20:59:19.701 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:59:19.700+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:59:19.700994+0000) 2024-03-04T20:59:20.702 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:59:20.700+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:59:20.701239+0000) 2024-03-04T20:59:21.702 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:59:21.700+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:59:21.701478+0000) 2024-03-04T20:59:22.702 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:59:22.700+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:59:22.701731+0000) 2024-03-04T20:59:23.703 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:59:23.701+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:59:23.701981+0000) 2024-03-04T20:59:24.703 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:59:24.701+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:59:24.702224+0000) 2024-03-04T20:59:25.703 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:59:25.701+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:59:25.702484+0000) 2024-03-04T20:59:26.703 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:59:26.702+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:59:26.702758+0000) 2024-03-04T20:59:27.297 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:59:27.303 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:59:27.704 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:59:27.702+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:59:27.703022+0000) 2024-03-04T20:59:28.704 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:59:28.702+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:59:28.703265+0000) 2024-03-04T20:59:29.704 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:59:29.703+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:59:29.703511+0000) 2024-03-04T20:59:30.704 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:59:30.703+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:59:30.703706+0000) 2024-03-04T20:59:31.704 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:59:31.703+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:59:31.703957+0000) 2024-03-04T20:59:32.705 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:59:32.703+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:59:32.704195+0000) 2024-03-04T20:59:33.705 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:59:33.704+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:59:33.704438+0000) 2024-03-04T20:59:34.705 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:59:34.704+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:59:34.704674+0000) 2024-03-04T20:59:35.705 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:59:35.704+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:59:35.704949+0000) 2024-03-04T20:59:36.706 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:59:36.704+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:59:36.705201+0000) 2024-03-04T20:59:37.706 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:59:37.705+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:59:37.705447+0000) 2024-03-04T20:59:38.706 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:59:38.705+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:59:38.705685+0000) 2024-03-04T20:59:39.706 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:59:39.705+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:59:39.705952+0000) 2024-03-04T20:59:40.707 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:59:40.705+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:59:40.706211+0000) 2024-03-04T20:59:41.707 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:59:41.705+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:59:41.706442+0000) 2024-03-04T20:59:42.707 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:59:42.706+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:59:42.706677+0000) 2024-03-04T20:59:43.708 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:59:43.706+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:59:43.706960+0000) 2024-03-04T20:59:44.708 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:59:44.706+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:59:44.707165+0000) 2024-03-04T20:59:45.708 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:59:45.707+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:59:45.707421+0000) 2024-03-04T20:59:46.708 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:59:46.707+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:59:46.707701+0000) 2024-03-04T20:59:47.709 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:59:47.707+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:59:47.707976+0000) 2024-03-04T20:59:48.709 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:59:48.708+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:59:48.708239+0000) 2024-03-04T20:59:49.709 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:59:49.707+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:59:49.708493+0000) 2024-03-04T20:59:50.709 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:59:50.706+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:59:50.708744+0000) 2024-03-04T20:59:51.710 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:59:51.707+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:59:51.709005+0000) 2024-03-04T20:59:52.710 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:59:52.707+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:59:52.709254+0000) 2024-03-04T20:59:53.710 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:59:53.707+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:59:53.709474+0000) 2024-03-04T20:59:54.710 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:59:54.708+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:59:54.709753+0000) 2024-03-04T20:59:55.711 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:59:55.708+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:59:55.710053+0000) 2024-03-04T20:59:56.711 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:59:56.709+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:59:56.710342+0000) 2024-03-04T20:59:57.528 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:59:57.536 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T20:59:57.711 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:59:57.709+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:59:57.710701+0000) 2024-03-04T20:59:58.712 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:59:58.709+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:59:58.710978+0000) 2024-03-04T20:59:59.712 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T20:59:59.710+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T19:59:59.711301+0000) 2024-03-04T21:00:00.712 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:00:00.710+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:00:00.711554+0000) 2024-03-04T21:00:01.713 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:00:01.710+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:00:01.711791+0000) 2024-03-04T21:00:02.713 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:00:02.711+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:00:02.712025+0000) 2024-03-04T21:00:03.713 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:00:03.711+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:00:03.712263+0000) 2024-03-04T21:00:04.713 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:00:04.711+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:00:04.712505+0000) 2024-03-04T21:00:05.713 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:00:05.711+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:00:05.712753+0000) 2024-03-04T21:00:06.714 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:00:06.712+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:00:06.712985+0000) 2024-03-04T21:00:07.714 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:00:07.712+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:00:07.713220+0000) 2024-03-04T21:00:08.714 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:00:08.712+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:00:08.713453+0000) 2024-03-04T21:00:09.714 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:00:09.712+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:00:09.713704+0000) 2024-03-04T21:00:10.715 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:00:10.713+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:00:10.713946+0000) 2024-03-04T21:00:11.715 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:00:11.713+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:00:11.714203+0000) 2024-03-04T21:00:12.715 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:00:12.713+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:00:12.714462+0000) 2024-03-04T21:00:13.715 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:00:13.713+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:00:13.714702+0000) 2024-03-04T21:00:14.716 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:00:14.714+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:00:14.714950+0000) 2024-03-04T21:00:15.716 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:00:15.714+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:00:15.715226+0000) 2024-03-04T21:00:16.716 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:00:16.714+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:00:16.715491+0000) 2024-03-04T21:00:17.716 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:00:17.715+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:00:17.715731+0000) 2024-03-04T21:00:18.717 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:00:18.715+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:00:18.715968+0000) 2024-03-04T21:00:19.717 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:00:19.715+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:00:19.716215+0000) 2024-03-04T21:00:20.717 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:00:20.715+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:00:20.716447+0000) 2024-03-04T21:00:21.717 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:00:21.716+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:00:21.716681+0000) 2024-03-04T21:00:22.718 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:00:22.716+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:00:22.716960+0000) 2024-03-04T21:00:23.718 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:00:23.716+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:00:23.717204+0000) 2024-03-04T21:00:24.718 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:00:24.716+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:00:24.717391+0000) 2024-03-04T21:00:25.718 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:00:25.717+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:00:25.717650+0000) 2024-03-04T21:00:26.719 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:00:26.717+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:00:26.717931+0000) 2024-03-04T21:00:27.719 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:00:27.717+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:00:27.718166+0000) 2024-03-04T21:00:27.741 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T21:00:27.749 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T21:00:28.719 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:00:28.717+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:00:28.718407+0000) 2024-03-04T21:00:29.719 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:00:29.718+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:00:29.718658+0000) 2024-03-04T21:00:30.720 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:00:30.718+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:00:30.718950+0000) 2024-03-04T21:00:31.720 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:00:31.718+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:00:31.719186+0000) 2024-03-04T21:00:32.720 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:00:32.718+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:00:32.719438+0000) 2024-03-04T21:00:33.720 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:00:33.719+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:00:33.719707+0000) 2024-03-04T21:00:34.721 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:00:34.719+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:00:34.719954+0000) 2024-03-04T21:00:35.721 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:00:35.719+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:00:35.720208+0000) 2024-03-04T21:00:36.721 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:00:36.720+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:00:36.720443+0000) 2024-03-04T21:00:37.721 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:00:37.720+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:00:37.720685+0000) 2024-03-04T21:00:38.722 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:00:38.720+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:00:38.720926+0000) 2024-03-04T21:00:39.722 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:00:39.720+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:00:39.721253+0000) 2024-03-04T21:00:40.722 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:00:40.721+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:00:40.721501+0000) 2024-03-04T21:00:41.722 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:00:41.721+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:00:41.721741+0000) 2024-03-04T21:00:42.723 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:00:42.721+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:00:42.721994+0000) 2024-03-04T21:00:43.723 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:00:43.721+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:00:43.722231+0000) 2024-03-04T21:00:44.723 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:00:44.722+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:00:44.722474+0000) 2024-03-04T21:00:45.723 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:00:45.722+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:00:45.722747+0000) 2024-03-04T21:00:46.724 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:00:46.722+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:00:46.723011+0000) 2024-03-04T21:00:47.724 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:00:47.723+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:00:47.723250+0000) 2024-03-04T21:00:48.724 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:00:48.723+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:00:48.723490+0000) 2024-03-04T21:00:49.724 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:00:49.723+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:00:49.723737+0000) 2024-03-04T21:00:50.725 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:00:50.723+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:00:50.724090+0000) 2024-03-04T21:00:51.725 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:00:51.724+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:00:51.724351+0000) 2024-03-04T21:00:52.725 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:00:52.724+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:00:52.724586+0000) 2024-03-04T21:00:53.726 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:00:53.723+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:00:53.724854+0000) 2024-03-04T21:00:54.726 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:00:54.723+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:00:54.725046+0000) 2024-03-04T21:00:55.727 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:00:55.724+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:00:55.725298+0000) 2024-03-04T21:00:56.726 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:00:56.724+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:00:56.725536+0000) 2024-03-04T21:00:57.727 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:00:57.724+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:00:57.725781+0000) 2024-03-04T21:00:57.883 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T21:00:57.891 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T21:00:58.727 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:00:58.724+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:00:58.725981+0000) 2024-03-04T21:00:59.727 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:00:59.725+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:00:59.726220+0000) 2024-03-04T21:01:00.727 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:01:00.725+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:01:00.726474+0000) 2024-03-04T21:01:01.728 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:01:01.725+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:01:01.726722+0000) 2024-03-04T21:01:02.728 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:01:02.725+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:01:02.726959+0000) 2024-03-04T21:01:03.728 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:01:03.726+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:01:03.727202+0000) 2024-03-04T21:01:04.728 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:01:04.726+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:01:04.727437+0000) 2024-03-04T21:01:05.729 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:01:05.726+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:01:05.727689+0000) 2024-03-04T21:01:06.729 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:01:06.727+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:01:06.727952+0000) 2024-03-04T21:01:07.729 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:01:07.727+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:01:07.728219+0000) 2024-03-04T21:01:08.729 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:01:08.727+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:01:08.728459+0000) 2024-03-04T21:01:09.730 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:01:09.727+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:01:09.728701+0000) 2024-03-04T21:01:10.730 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:01:10.728+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:01:10.728936+0000) 2024-03-04T21:01:11.730 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:01:11.728+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:01:11.729208+0000) 2024-03-04T21:01:12.730 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:01:12.728+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:01:12.729463+0000) 2024-03-04T21:01:13.730 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:01:13.728+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:01:13.729663+0000) 2024-03-04T21:01:14.731 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:01:14.729+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:01:14.729900+0000) 2024-03-04T21:01:15.731 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:01:15.729+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:01:15.730171+0000) 2024-03-04T21:01:16.731 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:01:16.729+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:01:16.730429+0000) 2024-03-04T21:01:17.731 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:01:17.729+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:01:17.730708+0000) 2024-03-04T21:01:18.732 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:01:18.730+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:01:18.730944+0000) 2024-03-04T21:01:19.732 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:01:19.730+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:01:19.731183+0000) 2024-03-04T21:01:20.732 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:01:20.730+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:01:20.731425+0000) 2024-03-04T21:01:21.733 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:01:21.730+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:01:21.731663+0000) 2024-03-04T21:01:22.733 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:01:22.731+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:01:22.731972+0000) 2024-03-04T21:01:23.734 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:01:23.732+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:01:23.732667+0000) 2024-03-04T21:01:24.734 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:01:24.732+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:01:24.732927+0000) 2024-03-04T21:01:25.734 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:01:25.732+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:01:25.733178+0000) 2024-03-04T21:01:26.734 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:01:26.732+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:01:26.733417+0000) 2024-03-04T21:01:27.735 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:01:27.733+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:01:27.733652+0000) 2024-03-04T21:01:28.026 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T21:01:28.038 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T21:01:28.735 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:01:28.733+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:01:28.733931+0000) 2024-03-04T21:01:29.735 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:01:29.733+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:01:29.734172+0000) 2024-03-04T21:01:30.736 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:01:30.733+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:01:30.734460+0000) 2024-03-04T21:01:31.736 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:01:31.734+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:01:31.734765+0000) 2024-03-04T21:01:32.736 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:01:32.734+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:01:32.735046+0000) 2024-03-04T21:01:33.736 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:01:33.734+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:01:33.735329+0000) 2024-03-04T21:01:34.736 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:01:34.735+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:01:34.735627+0000) 2024-03-04T21:01:35.737 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:01:35.735+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:01:35.735939+0000) 2024-03-04T21:01:36.737 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:01:36.735+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:01:36.736175+0000) 2024-03-04T21:01:37.737 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:01:37.736+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:01:37.736414+0000) 2024-03-04T21:01:38.738 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:01:38.736+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:01:38.736809+0000) 2024-03-04T21:01:39.738 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:01:39.736+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:01:39.737123+0000) 2024-03-04T21:01:40.738 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:01:40.736+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:01:40.737363+0000) 2024-03-04T21:01:41.739 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:01:41.737+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:01:41.737603+0000) 2024-03-04T21:01:42.739 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:01:42.737+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:01:42.737917+0000) 2024-03-04T21:01:43.739 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:01:43.737+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:01:43.738184+0000) 2024-03-04T21:01:44.739 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:01:44.738+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:01:44.738432+0000) 2024-03-04T21:01:45.740 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:01:45.737+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:01:45.738628+0000) 2024-03-04T21:01:46.740 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:01:46.738+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:01:46.738818+0000) 2024-03-04T21:01:47.740 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:01:47.738+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:01:47.739086+0000) 2024-03-04T21:01:48.740 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:01:48.739+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:01:48.739325+0000) 2024-03-04T21:01:49.740 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:01:49.739+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:01:49.739570+0000) 2024-03-04T21:01:50.741 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:01:50.739+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:01:50.739815+0000) 2024-03-04T21:01:51.741 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:01:51.739+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:01:51.740008+0000) 2024-03-04T21:01:52.741 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:01:52.740+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:01:52.740248+0000) 2024-03-04T21:01:53.741 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:01:53.740+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:01:53.740490+0000) 2024-03-04T21:01:54.742 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:01:54.739+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:01:54.740747+0000) 2024-03-04T21:01:55.742 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:01:55.740+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:01:55.740994+0000) 2024-03-04T21:01:56.742 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:01:56.740+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:01:56.741233+0000) 2024-03-04T21:01:57.742 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:01:57.740+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:01:57.741477+0000) 2024-03-04T21:01:58.166 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T21:01:58.174 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T21:01:58.743 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:01:58.740+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:01:58.741724+0000) 2024-03-04T21:01:59.743 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:01:59.740+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:01:59.741996+0000) 2024-03-04T21:02:00.743 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:02:00.741+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:02:00.742274+0000) 2024-03-04T21:02:01.744 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:02:01.741+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:02:01.742542+0000) 2024-03-04T21:02:02.744 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:02:02.741+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:02:02.742864+0000) 2024-03-04T21:02:03.744 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:02:03.742+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:02:03.743163+0000) 2024-03-04T21:02:04.744 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:02:04.742+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:02:04.743458+0000) 2024-03-04T21:02:05.745 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:02:05.742+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:02:05.743728+0000) 2024-03-04T21:02:06.745 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:02:06.743+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:02:06.744039+0000) 2024-03-04T21:02:07.746 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:02:07.743+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:02:07.744294+0000) 2024-03-04T21:02:08.746 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:02:08.743+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:02:08.744588+0000) 2024-03-04T21:02:09.746 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:02:09.744+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:02:09.744933+0000) 2024-03-04T21:02:10.746 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:02:10.744+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:02:10.745176+0000) 2024-03-04T21:02:11.746 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:02:11.744+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:02:11.745477+0000) 2024-03-04T21:02:12.747 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:02:12.744+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:02:12.745791+0000) 2024-03-04T21:02:13.747 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:02:13.745+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:02:13.746144+0000) 2024-03-04T21:02:14.747 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:02:14.745+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:02:14.746394+0000) 2024-03-04T21:02:15.748 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:02:15.745+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:02:15.746672+0000) 2024-03-04T21:02:16.748 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:02:16.746+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:02:16.746977+0000) 2024-03-04T21:02:17.748 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:02:17.746+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:02:17.747274+0000) 2024-03-04T21:02:18.749 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:02:18.746+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:02:18.747549+0000) 2024-03-04T21:02:19.749 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:02:19.747+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:02:19.747873+0000) 2024-03-04T21:02:20.749 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:02:20.747+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:02:20.748164+0000) 2024-03-04T21:02:21.749 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:02:21.747+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:02:21.748531+0000) 2024-03-04T21:02:22.750 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:02:22.748+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:02:22.748861+0000) 2024-03-04T21:02:23.750 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:02:23.748+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:02:23.749161+0000) 2024-03-04T21:02:24.750 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:02:24.748+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:02:24.749458+0000) 2024-03-04T21:02:25.751 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:02:25.749+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:02:25.749727+0000) 2024-03-04T21:02:26.751 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:02:26.749+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:02:26.750014+0000) 2024-03-04T21:02:27.751 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:02:27.749+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:02:27.750312+0000) 2024-03-04T21:02:28.295 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T21:02:28.303 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T21:02:28.751 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:02:28.749+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:02:28.750612+0000) 2024-03-04T21:02:29.752 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:02:29.750+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:02:29.750940+0000) 2024-03-04T21:02:30.752 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:02:30.750+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:02:30.751248+0000) 2024-03-04T21:02:31.752 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:02:31.750+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:02:31.751480+0000) 2024-03-04T21:02:32.752 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:02:32.751+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:02:32.751771+0000) 2024-03-04T21:02:33.753 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:02:33.751+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:02:33.752105+0000) 2024-03-04T21:02:34.753 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:02:34.751+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:02:34.752339+0000) 2024-03-04T21:02:35.753 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:02:35.752+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:02:35.752645+0000) 2024-03-04T21:02:36.753 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:02:36.752+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:02:36.752985+0000) 2024-03-04T21:02:37.754 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:02:37.752+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:02:37.753280+0000) 2024-03-04T21:02:38.754 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:02:38.753+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:02:38.753571+0000) 2024-03-04T21:02:39.754 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:02:39.752+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:02:39.753804+0000) 2024-03-04T21:02:40.754 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:02:40.753+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:02:40.754068+0000) 2024-03-04T21:02:41.755 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:02:41.753+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:02:41.754303+0000) 2024-03-04T21:02:42.755 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:02:42.753+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:02:42.754530+0000) 2024-03-04T21:02:43.755 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:02:43.753+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:02:43.754762+0000) 2024-03-04T21:02:44.755 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:02:44.754+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:02:44.755019+0000) 2024-03-04T21:02:45.756 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:02:45.754+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:02:45.755269+0000) 2024-03-04T21:02:46.756 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:02:46.755+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:02:46.755503+0000) 2024-03-04T21:02:47.756 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:02:47.754+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:02:47.755752+0000) 2024-03-04T21:02:48.756 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:02:48.755+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:02:48.755990+0000) 2024-03-04T21:02:49.756 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:02:49.755+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:02:49.756174+0000) 2024-03-04T21:02:50.756 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:02:50.756+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:02:50.756410+0000) 2024-03-04T21:02:51.757 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:02:51.755+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:02:51.756676+0000) 2024-03-04T21:02:52.757 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:02:52.756+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:02:52.756941+0000) 2024-03-04T21:02:53.757 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:02:53.756+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:02:53.757181+0000) 2024-03-04T21:02:54.758 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:02:54.756+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:02:54.757440+0000) 2024-03-04T21:02:55.758 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:02:55.756+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:02:55.757764+0000) 2024-03-04T21:02:56.758 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:02:56.757+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:02:56.758048+0000) 2024-03-04T21:02:57.758 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:02:57.758+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:02:57.758338+0000) 2024-03-04T21:02:58.434 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T21:02:58.442 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T21:02:58.758 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:02:58.757+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:02:58.758614+0000) 2024-03-04T21:02:59.759 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:02:59.757+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:02:59.758903+0000) 2024-03-04T21:03:00.759 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:03:00.758+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:03:00.759171+0000) 2024-03-04T21:03:01.759 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:03:01.758+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:03:01.759463+0000) 2024-03-04T21:03:02.760 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:03:02.758+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:03:02.759745+0000) 2024-03-04T21:03:03.760 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:03:03.758+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:03:03.760023+0000) 2024-03-04T21:03:04.760 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:03:04.759+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:03:04.760314+0000) 2024-03-04T21:03:05.760 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:03:05.759+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:03:05.760638+0000) 2024-03-04T21:03:06.761 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:03:06.759+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:03:06.760975+0000) 2024-03-04T21:03:07.761 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:03:07.759+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:03:07.761138+0000) 2024-03-04T21:03:08.761 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:03:08.760+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:03:08.761363+0000) 2024-03-04T21:03:09.761 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:03:09.760+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:03:09.761603+0000) 2024-03-04T21:03:10.761 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:03:10.761+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:03:10.761867+0000) 2024-03-04T21:03:11.762 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:03:11.761+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:03:11.762118+0000) 2024-03-04T21:03:12.762 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:03:12.761+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:03:12.762405+0000) 2024-03-04T21:03:13.762 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:03:13.761+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:03:13.762694+0000) 2024-03-04T21:03:14.763 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:03:14.762+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:03:14.763051+0000) 2024-03-04T21:03:15.763 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:03:15.762+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:03:15.763350+0000) 2024-03-04T21:03:16.763 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:03:16.762+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:03:16.763643+0000) 2024-03-04T21:03:17.764 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:03:17.763+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:03:17.763985+0000) 2024-03-04T21:03:18.764 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:03:18.763+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:03:18.764286+0000) 2024-03-04T21:03:19.764 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:03:19.763+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:03:19.764530+0000) 2024-03-04T21:03:20.764 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:03:20.764+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:03:20.764788+0000) 2024-03-04T21:03:21.764 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:03:21.764+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:03:21.765035+0000) 2024-03-04T21:03:22.765 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:03:22.764+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:03:22.765273+0000) 2024-03-04T21:03:23.765 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:03:23.764+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:03:23.765517+0000) 2024-03-04T21:03:24.768 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:03:24.765+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:03:24.765750+0000) 2024-03-04T21:03:25.765 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:03:25.765+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:03:25.766001+0000) 2024-03-04T21:03:26.767 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:03:26.765+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:03:26.766258+0000) 2024-03-04T21:03:27.766 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:03:27.765+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:03:27.766498+0000) 2024-03-04T21:03:28.582 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T21:03:28.591 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T21:03:28.766 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:03:28.765+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:03:28.766767+0000) 2024-03-04T21:03:29.766 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:03:29.766+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:03:29.767011+0000) 2024-03-04T21:03:30.766 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:03:30.766+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:03:30.767245+0000) 2024-03-04T21:03:31.767 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:03:31.766+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:03:31.767487+0000) 2024-03-04T21:03:32.767 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:03:32.766+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:03:32.767728+0000) 2024-03-04T21:03:33.767 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:03:33.767+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:03:33.767972+0000) 2024-03-04T21:03:34.767 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:03:34.767+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:03:34.768225+0000) 2024-03-04T21:03:35.767 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:03:35.767+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:03:35.768494+0000) 2024-03-04T21:03:36.768 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:03:36.768+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:03:36.768731+0000) 2024-03-04T21:03:37.768 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:03:37.768+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:03:37.768968+0000) 2024-03-04T21:03:38.768 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:03:38.768+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:03:38.769203+0000) 2024-03-04T21:03:39.768 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:03:39.768+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:03:39.769443+0000) 2024-03-04T21:03:40.769 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:03:40.769+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:03:40.769685+0000) 2024-03-04T21:03:41.769 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:03:41.769+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:03:41.769932+0000) 2024-03-04T21:03:42.769 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:03:42.769+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:03:42.770128+0000) 2024-03-04T21:03:43.769 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:03:43.769+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:03:43.770391+0000) 2024-03-04T21:03:44.770 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:03:44.769+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:03:44.770614+0000) 2024-03-04T21:03:45.770 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:03:45.770+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:03:45.770916+0000) 2024-03-04T21:03:46.770 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:03:46.770+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:03:46.771111+0000) 2024-03-04T21:03:47.770 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:03:47.770+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:03:47.771301+0000) 2024-03-04T21:03:48.770 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:03:48.771+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:03:48.771537+0000) 2024-03-04T21:03:49.770 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:03:49.771+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:03:49.771773+0000) 2024-03-04T21:03:50.771 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:03:50.771+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:03:50.772030+0000) 2024-03-04T21:03:51.771 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:03:51.771+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:03:51.772271+0000) 2024-03-04T21:03:52.771 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:03:52.772+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:03:52.772510+0000) 2024-03-04T21:03:53.772 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:03:53.772+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:03:53.772755+0000) 2024-03-04T21:03:54.772 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:03:54.772+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:03:54.773002+0000) 2024-03-04T21:03:55.772 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:03:55.773+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:03:55.773255+0000) 2024-03-04T21:03:56.772 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:03:56.773+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:03:56.773493+0000) 2024-03-04T21:03:57.772 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:03:57.773+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:03:57.773732+0000) 2024-03-04T21:03:58.773 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:03:58.773+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:03:58.773977+0000) 2024-03-04T21:03:58.783 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T21:03:58.791 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T21:03:59.773 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:03:59.774+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:03:59.774218+0000) 2024-03-04T21:04:00.773 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:04:00.773+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:04:00.774410+0000) 2024-03-04T21:04:01.773 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:04:01.772+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:04:01.774642+0000) 2024-03-04T21:04:02.773 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:04:02.773+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:04:02.774930+0000) 2024-03-04T21:04:03.774 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:04:03.774+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:04:03.775167+0000) 2024-03-04T21:04:04.774 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:04:04.774+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:04:04.775403+0000) 2024-03-04T21:04:05.774 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:04:05.774+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:04:05.775644+0000) 2024-03-04T21:04:06.774 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:04:06.774+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:04:06.775950+0000) 2024-03-04T21:04:07.774 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:04:07.775+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:04:07.776209+0000) 2024-03-04T21:04:08.775 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:04:08.775+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:04:08.776491+0000) 2024-03-04T21:04:09.775 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:04:09.775+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:04:09.776729+0000) 2024-03-04T21:04:10.775 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:04:10.775+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:04:10.776964+0000) 2024-03-04T21:04:11.775 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:04:11.776+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:04:11.777209+0000) 2024-03-04T21:04:12.776 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:04:12.776+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:04:12.777446+0000) 2024-03-04T21:04:13.776 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:04:13.776+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:04:13.777685+0000) 2024-03-04T21:04:14.776 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:04:14.777+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:04:14.777920+0000) 2024-03-04T21:04:15.776 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:04:15.777+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:04:15.778166+0000) 2024-03-04T21:04:16.777 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:04:16.777+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:04:16.778414+0000) 2024-03-04T21:04:17.777 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:04:17.777+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:04:17.778671+0000) 2024-03-04T21:04:18.777 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:04:18.778+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:04:18.778931+0000) 2024-03-04T21:04:19.777 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:04:19.778+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:04:19.779171+0000) 2024-03-04T21:04:20.777 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:04:20.778+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:04:20.779410+0000) 2024-03-04T21:04:21.778 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:04:21.778+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:04:21.779655+0000) 2024-03-04T21:04:22.778 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:04:22.779+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:04:22.779941+0000) 2024-03-04T21:04:23.778 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:04:23.779+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:04:23.780180+0000) 2024-03-04T21:04:24.778 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:04:24.779+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:04:24.780419+0000) 2024-03-04T21:04:25.779 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:04:25.779+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:04:25.780677+0000) 2024-03-04T21:04:26.779 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:04:26.780+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:04:26.780936+0000) 2024-03-04T21:04:27.779 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:04:27.780+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:04:27.781176+0000) 2024-03-04T21:04:28.779 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:04:28.780+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:04:28.781359+0000) 2024-03-04T21:04:28.929 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T21:04:28.937 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T21:04:29.779 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:04:29.780+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:04:29.781630+0000) 2024-03-04T21:04:30.780 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:04:30.781+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:04:30.781935+0000) 2024-03-04T21:04:31.780 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:04:31.781+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:04:31.782196+0000) 2024-03-04T21:04:32.780 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:04:32.781+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:04:32.782447+0000) 2024-03-04T21:04:33.780 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:04:33.782+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:04:33.782707+0000) 2024-03-04T21:04:34.781 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:04:34.782+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:04:34.782986+0000) 2024-03-04T21:04:35.781 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:04:35.782+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:04:35.783280+0000) 2024-03-04T21:04:36.781 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:04:36.782+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:04:36.783558+0000) 2024-03-04T21:04:37.781 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:04:37.783+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:04:37.783811+0000) 2024-03-04T21:04:38.782 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:04:38.783+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:04:38.784134+0000) 2024-03-04T21:04:39.782 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:04:39.783+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:04:39.784389+0000) 2024-03-04T21:04:40.782 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:04:40.784+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:04:40.784644+0000) 2024-03-04T21:04:41.782 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:04:41.784+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:04:41.784944+0000) 2024-03-04T21:04:42.783 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:04:42.784+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:04:42.785191+0000) 2024-03-04T21:04:43.783 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:04:43.784+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:04:43.785448+0000) 2024-03-04T21:04:44.783 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:04:44.785+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:04:44.785702+0000) 2024-03-04T21:04:45.784 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:04:45.785+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:04:45.785971+0000) 2024-03-04T21:04:46.784 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:04:46.785+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:04:46.786210+0000) 2024-03-04T21:04:47.784 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:04:47.786+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:04:47.786446+0000) 2024-03-04T21:04:48.784 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:04:48.786+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:04:48.786686+0000) 2024-03-04T21:04:49.784 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:04:49.786+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:04:49.786928+0000) 2024-03-04T21:04:50.785 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:04:50.786+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:04:50.787152+0000) 2024-03-04T21:04:51.785 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:04:51.787+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:04:51.787395+0000) 2024-03-04T21:04:52.785 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:04:52.787+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:04:52.787630+0000) 2024-03-04T21:04:53.785 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:04:53.787+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:04:53.787919+0000) 2024-03-04T21:04:54.785 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:04:54.787+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:04:54.788162+0000) 2024-03-04T21:04:55.786 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:04:55.787+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:04:55.788411+0000) 2024-03-04T21:04:56.786 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:04:56.788+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:04:56.788653+0000) 2024-03-04T21:04:57.786 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:04:57.788+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:04:57.788938+0000) 2024-03-04T21:04:58.786 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:04:58.788+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:04:58.789194+0000) 2024-03-04T21:04:59.076 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T21:04:59.085 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T21:04:59.787 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:04:59.789+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:04:59.789442+0000) 2024-03-04T21:05:00.787 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:05:00.789+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:05:00.789688+0000) 2024-03-04T21:05:01.787 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:05:01.789+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:05:01.789993+0000) 2024-03-04T21:05:02.787 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:05:02.790+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:05:02.790229+0000) 2024-03-04T21:05:03.788 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:05:03.790+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:05:03.790466+0000) 2024-03-04T21:05:04.788 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:05:04.789+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:05:04.790707+0000) 2024-03-04T21:05:05.788 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:05:05.789+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:05:05.790985+0000) 2024-03-04T21:05:06.788 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:05:06.790+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:05:06.791250+0000) 2024-03-04T21:05:07.788 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:05:07.790+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:05:07.791497+0000) 2024-03-04T21:05:08.789 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:05:08.790+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:05:08.791733+0000) 2024-03-04T21:05:09.789 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:05:09.790+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:05:09.791974+0000) 2024-03-04T21:05:10.789 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:05:10.791+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:05:10.792222+0000) 2024-03-04T21:05:11.789 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:05:11.791+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:05:11.792475+0000) 2024-03-04T21:05:12.790 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:05:12.791+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:05:12.792715+0000) 2024-03-04T21:05:13.790 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:05:13.791+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:05:13.792960+0000) 2024-03-04T21:05:14.790 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:05:14.792+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:05:14.793223+0000) 2024-03-04T21:05:15.790 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:05:15.792+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:05:15.793486+0000) 2024-03-04T21:05:16.791 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:05:16.792+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:05:16.793727+0000) 2024-03-04T21:05:17.791 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:05:17.793+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:05:17.793989+0000) 2024-03-04T21:05:18.791 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:05:18.793+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:05:18.794176+0000) 2024-03-04T21:05:19.791 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:05:19.793+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:05:19.794438+0000) 2024-03-04T21:05:20.792 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:05:20.793+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:05:20.794684+0000) 2024-03-04T21:05:21.792 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:05:21.794+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:05:21.794989+0000) 2024-03-04T21:05:22.792 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:05:22.794+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:05:22.795249+0000) 2024-03-04T21:05:23.792 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:05:23.794+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:05:23.795560+0000) 2024-03-04T21:05:24.793 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:05:24.795+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:05:24.795810+0000) 2024-03-04T21:05:25.793 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:05:25.795+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:05:25.796123+0000) 2024-03-04T21:05:26.793 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:05:26.795+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:05:26.796418+0000) 2024-03-04T21:05:27.793 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:05:27.795+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:05:27.796630+0000) 2024-03-04T21:05:28.793 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:05:28.796+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:05:28.796922+0000) 2024-03-04T21:05:29.222 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T21:05:29.230 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T21:05:29.794 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:05:29.796+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:05:29.797191+0000) 2024-03-04T21:05:30.794 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:05:30.796+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:05:30.797444+0000) 2024-03-04T21:05:31.794 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:05:31.797+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:05:31.797713+0000) 2024-03-04T21:05:32.795 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:05:32.797+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:05:32.797939+0000) 2024-03-04T21:05:33.795 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:05:33.797+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:05:33.798197+0000) 2024-03-04T21:05:34.795 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:05:34.797+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:05:34.798479+0000) 2024-03-04T21:05:35.795 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:05:35.798+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:05:35.798779+0000) 2024-03-04T21:05:36.795 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:05:36.798+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:05:36.799033+0000) 2024-03-04T21:05:37.796 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:05:37.798+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:05:37.799292+0000) 2024-03-04T21:05:38.796 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:05:38.798+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:05:38.799555+0000) 2024-03-04T21:05:39.796 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:05:39.799+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:05:39.799809+0000) 2024-03-04T21:05:40.797 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:05:40.799+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:05:40.800116+0000) 2024-03-04T21:05:41.797 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:05:41.799+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:05:41.800361+0000) 2024-03-04T21:05:42.797 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:05:42.800+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:05:42.800601+0000) 2024-03-04T21:05:43.797 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:05:43.800+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:05:43.800884+0000) 2024-03-04T21:05:44.797 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:05:44.800+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:05:44.801127+0000) 2024-03-04T21:05:45.798 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:05:45.800+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:05:45.801385+0000) 2024-03-04T21:05:46.798 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:05:46.800+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:05:46.801672+0000) 2024-03-04T21:05:47.798 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:05:47.801+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:05:47.801974+0000) 2024-03-04T21:05:48.798 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:05:48.801+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:05:48.802177+0000) 2024-03-04T21:05:49.799 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:05:49.801+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:05:49.802511+0000) 2024-03-04T21:05:50.799 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:05:50.801+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:05:50.802742+0000) 2024-03-04T21:05:51.799 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:05:51.802+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:05:51.803028+0000) 2024-03-04T21:05:52.800 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:05:52.802+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:05:52.803306+0000) 2024-03-04T21:05:53.800 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:05:53.803+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:05:53.803568+0000) 2024-03-04T21:05:54.800 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:05:54.803+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:05:54.803868+0000) 2024-03-04T21:05:55.800 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:05:55.803+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:05:55.804152+0000) 2024-03-04T21:05:56.800 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:05:56.803+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:05:56.804396+0000) 2024-03-04T21:05:57.801 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:05:57.804+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:05:57.804637+0000) 2024-03-04T21:05:58.801 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:05:58.804+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:05:58.804930+0000) 2024-03-04T21:05:59.375 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T21:05:59.384 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T21:05:59.801 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:05:59.804+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:05:59.805175+0000) 2024-03-04T21:06:00.801 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:06:00.805+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:06:00.805420+0000) 2024-03-04T21:06:01.802 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:06:01.805+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:06:01.805662+0000) 2024-03-04T21:06:02.802 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:06:02.805+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:06:02.805941+0000) 2024-03-04T21:06:03.802 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:06:03.805+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:06:03.806132+0000) 2024-03-04T21:06:04.802 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:06:04.806+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:06:04.806394+0000) 2024-03-04T21:06:05.803 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:06:05.806+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:06:05.806645+0000) 2024-03-04T21:06:06.803 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:06:06.806+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:06:06.806932+0000) 2024-03-04T21:06:07.803 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:06:07.806+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:06:07.807188+0000) 2024-03-04T21:06:08.803 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:06:08.806+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:06:08.807445+0000) 2024-03-04T21:06:09.804 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:06:09.805+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:06:09.807716+0000) 2024-03-04T21:06:10.804 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:06:10.806+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:06:10.807979+0000) 2024-03-04T21:06:11.804 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:06:11.807+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:06:11.808238+0000) 2024-03-04T21:06:12.804 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:06:12.806+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:06:12.808450+0000) 2024-03-04T21:06:13.805 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:06:13.807+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:06:13.808707+0000) 2024-03-04T21:06:14.805 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:06:14.807+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:06:14.808980+0000) 2024-03-04T21:06:15.805 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:06:15.808+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:06:15.809249+0000) 2024-03-04T21:06:16.805 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:06:16.808+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:06:16.809496+0000) 2024-03-04T21:06:17.805 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:06:17.808+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:06:17.809743+0000) 2024-03-04T21:06:18.806 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:06:18.809+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:06:18.809985+0000) 2024-03-04T21:06:19.806 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:06:19.809+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:06:19.810224+0000) 2024-03-04T21:06:20.806 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:06:20.809+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:06:20.810463+0000) 2024-03-04T21:06:21.806 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:06:21.809+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:06:21.810655+0000) 2024-03-04T21:06:22.806 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:06:22.810+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:06:22.810946+0000) 2024-03-04T21:06:23.807 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:06:23.810+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:06:23.811189+0000) 2024-03-04T21:06:24.807 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:06:24.810+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:06:24.811430+0000) 2024-03-04T21:06:25.807 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:06:25.810+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:06:25.811676+0000) 2024-03-04T21:06:26.807 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:06:26.811+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:06:26.811960+0000) 2024-03-04T21:06:27.808 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:06:27.811+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:06:27.812204+0000) 2024-03-04T21:06:28.808 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:06:28.811+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:06:28.812469+0000) 2024-03-04T21:06:29.513 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T21:06:29.522 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T21:06:29.808 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:06:29.811+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:06:29.812732+0000) 2024-03-04T21:06:30.808 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:06:30.812+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:06:30.812987+0000) 2024-03-04T21:06:31.809 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:06:31.812+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:06:31.813247+0000) 2024-03-04T21:06:32.809 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:06:32.812+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:06:32.813621+0000) 2024-03-04T21:06:33.809 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:06:33.813+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:06:33.813926+0000) 2024-03-04T21:06:34.810 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:06:34.813+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:06:34.814177+0000) 2024-03-04T21:06:35.810 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:06:35.813+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:06:35.814429+0000) 2024-03-04T21:06:36.810 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:06:36.814+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:06:36.814691+0000) 2024-03-04T21:06:37.810 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:06:37.814+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:06:37.814935+0000) 2024-03-04T21:06:38.811 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:06:38.814+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:06:38.815183+0000) 2024-03-04T21:06:39.811 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:06:39.814+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:06:39.815446+0000) 2024-03-04T21:06:40.811 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:06:40.815+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:06:40.815707+0000) 2024-03-04T21:06:41.811 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:06:41.815+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:06:41.815971+0000) 2024-03-04T21:06:42.812 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:06:42.815+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:06:42.816252+0000) 2024-03-04T21:06:43.812 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:06:43.815+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:06:43.816454+0000) 2024-03-04T21:06:44.812 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:06:44.816+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:06:44.816717+0000) 2024-03-04T21:06:45.812 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:06:45.816+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:06:45.817071+0000) 2024-03-04T21:06:46.813 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:06:46.816+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:06:46.817365+0000) 2024-03-04T21:06:47.813 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:06:47.817+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:06:47.817629+0000) 2024-03-04T21:06:48.813 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:06:48.817+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:06:48.817927+0000) 2024-03-04T21:06:49.814 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:06:49.817+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:06:49.818189+0000) 2024-03-04T21:06:50.814 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:06:50.818+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:06:50.818448+0000) 2024-03-04T21:06:51.814 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:06:51.817+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:06:51.818650+0000) 2024-03-04T21:06:52.814 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:06:52.818+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:06:52.818947+0000) 2024-03-04T21:06:53.817 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:06:53.818+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:06:53.819165+0000) 2024-03-04T21:06:54.815 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:06:54.818+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:06:54.819418+0000) 2024-03-04T21:06:55.815 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:06:55.819+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:06:55.819684+0000) 2024-03-04T21:06:56.815 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:06:56.819+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:06:56.819948+0000) 2024-03-04T21:06:57.815 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:06:57.819+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:06:57.820208+0000) 2024-03-04T21:06:58.815 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:06:58.819+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:06:58.820421+0000) 2024-03-04T21:06:59.651 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T21:06:59.659 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T21:06:59.816 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:06:59.819+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:06:59.820637+0000) 2024-03-04T21:07:00.816 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:07:00.820+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:07:00.820927+0000) 2024-03-04T21:07:01.816 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:07:01.820+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:07:01.821203+0000) 2024-03-04T21:07:02.816 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:07:02.820+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:07:02.821455+0000) 2024-03-04T21:07:03.817 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:07:03.820+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:07:03.821663+0000) 2024-03-04T21:07:04.817 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:07:04.821+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:07:04.821930+0000) 2024-03-04T21:07:05.817 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:07:05.821+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:07:05.822121+0000) 2024-03-04T21:07:06.817 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:07:06.822+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:07:06.822369+0000) 2024-03-04T21:07:07.818 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:07:07.822+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:07:07.822608+0000) 2024-03-04T21:07:08.818 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:07:08.822+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:07:08.822878+0000) 2024-03-04T21:07:09.818 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:07:09.822+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:07:09.823118+0000) 2024-03-04T21:07:10.818 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:07:10.822+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:07:10.823351+0000) 2024-03-04T21:07:11.818 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:07:11.822+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:07:11.823593+0000) 2024-03-04T21:07:12.819 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:07:12.822+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:07:12.823864+0000) 2024-03-04T21:07:13.819 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:07:13.823+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:07:13.824108+0000) 2024-03-04T21:07:14.819 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:07:14.823+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:07:14.824350+0000) 2024-03-04T21:07:15.819 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:07:15.823+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:07:15.824620+0000) 2024-03-04T21:07:16.820 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:07:16.823+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:07:16.824902+0000) 2024-03-04T21:07:17.820 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:07:17.823+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:07:17.825083+0000) 2024-03-04T21:07:18.820 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:07:18.824+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:07:18.825318+0000) 2024-03-04T21:07:19.820 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:07:19.823+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:07:19.825493+0000) 2024-03-04T21:07:20.820 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:07:20.824+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:07:20.825728+0000) 2024-03-04T21:07:21.821 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:07:21.825+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:07:21.825994+0000) 2024-03-04T21:07:22.821 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:07:22.825+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:07:22.826233+0000) 2024-03-04T21:07:23.821 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:07:23.825+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:07:23.826471+0000) 2024-03-04T21:07:24.821 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:07:24.825+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:07:24.826704+0000) 2024-03-04T21:07:25.822 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:07:25.826+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:07:25.826976+0000) 2024-03-04T21:07:26.822 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:07:26.826+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:07:26.827229+0000) 2024-03-04T21:07:27.822 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:07:27.826+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:07:27.827475+0000) 2024-03-04T21:07:28.822 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:07:28.826+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:07:28.827712+0000) 2024-03-04T21:07:29.798 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T21:07:29.806 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T21:07:29.822 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:07:29.827+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:07:29.827902+0000) 2024-03-04T21:07:30.823 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:07:30.827+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:07:30.828132+0000) 2024-03-04T21:07:31.823 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:07:31.827+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:07:31.828371+0000) 2024-03-04T21:07:32.823 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:07:32.827+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:07:32.828611+0000) 2024-03-04T21:07:33.823 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:07:33.828+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:07:33.828907+0000) 2024-03-04T21:07:34.827 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:07:34.828+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:07:34.829144+0000) 2024-03-04T21:07:35.824 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:07:35.828+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:07:35.829387+0000) 2024-03-04T21:07:36.824 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:07:36.828+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:07:36.829623+0000) 2024-03-04T21:07:37.824 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:07:37.829+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:07:37.829902+0000) 2024-03-04T21:07:38.825 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:07:38.829+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:07:38.830160+0000) 2024-03-04T21:07:39.825 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:07:39.829+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:07:39.830527+0000) 2024-03-04T21:07:40.825 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:07:40.830+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:07:40.830815+0000) 2024-03-04T21:07:41.825 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:07:41.830+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:07:41.831156+0000) 2024-03-04T21:07:42.826 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:07:42.830+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:07:42.831448+0000) 2024-03-04T21:07:43.826 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:07:43.831+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:07:43.831747+0000) 2024-03-04T21:07:44.826 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:07:44.831+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:07:44.832043+0000) 2024-03-04T21:07:45.827 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:07:45.831+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:07:45.832332+0000) 2024-03-04T21:07:46.827 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:07:46.831+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:07:46.832631+0000) 2024-03-04T21:07:47.827 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:07:47.832+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:07:47.832915+0000) 2024-03-04T21:07:48.827 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:07:48.832+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:07:48.833165+0000) 2024-03-04T21:07:49.828 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:07:49.832+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:07:49.833400+0000) 2024-03-04T21:07:50.828 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:07:50.833+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:07:50.833639+0000) 2024-03-04T21:07:51.828 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:07:51.833+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:07:51.833930+0000) 2024-03-04T21:07:52.828 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:07:52.833+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:07:52.834172+0000) 2024-03-04T21:07:53.829 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:07:53.833+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:07:53.834411+0000) 2024-03-04T21:07:54.829 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:07:54.834+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:07:54.834670+0000) 2024-03-04T21:07:55.829 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:07:55.834+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:07:55.834955+0000) 2024-03-04T21:07:56.829 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:07:56.834+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:07:56.835206+0000) 2024-03-04T21:07:57.830 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:07:57.834+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:07:57.835445+0000) 2024-03-04T21:07:58.830 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:07:58.835+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:07:58.835684+0000) 2024-03-04T21:07:59.830 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:07:59.835+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:07:59.835929+0000) 2024-03-04T21:07:59.950 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T21:07:59.959 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T21:08:00.830 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:08:00.835+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:08:00.836194+0000) 2024-03-04T21:08:01.830 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:08:01.836+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:08:01.836473+0000) 2024-03-04T21:08:02.831 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:08:02.836+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:08:02.836737+0000) 2024-03-04T21:08:03.831 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:08:03.836+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:08:03.837197+0000) 2024-03-04T21:08:04.831 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:08:04.837+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:08:04.837492+0000) 2024-03-04T21:08:05.832 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:08:05.837+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:08:05.837790+0000) 2024-03-04T21:08:06.832 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:08:06.837+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:08:06.838103+0000) 2024-03-04T21:08:07.832 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:08:07.837+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:08:07.838344+0000) 2024-03-04T21:08:08.833 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:08:08.838+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:08:08.838603+0000) 2024-03-04T21:08:09.836 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:08:09.838+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:08:09.838937+0000) 2024-03-04T21:08:10.833 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:08:10.839+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:08:10.839264+0000) 2024-03-04T21:08:11.833 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:08:11.838+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:08:11.839498+0000) 2024-03-04T21:08:12.834 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:08:12.838+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:08:12.839795+0000) 2024-03-04T21:08:13.834 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:08:13.838+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:08:13.840147+0000) 2024-03-04T21:08:14.835 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:08:14.839+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:08:14.840444+0000) 2024-03-04T21:08:15.835 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:08:15.839+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:08:15.840737+0000) 2024-03-04T21:08:16.835 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:08:16.839+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:08:16.841032+0000) 2024-03-04T21:08:17.835 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:08:17.840+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:08:17.841320+0000) 2024-03-04T21:08:18.835 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:08:18.839+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:08:18.841680+0000) 2024-03-04T21:08:19.836 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:08:19.840+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:08:19.841940+0000) 2024-03-04T21:08:20.836 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:08:20.840+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:08:20.842171+0000) 2024-03-04T21:08:21.836 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:08:21.841+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:08:21.842462+0000) 2024-03-04T21:08:22.837 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:08:22.841+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:08:22.842745+0000) 2024-03-04T21:08:23.837 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:08:23.842+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:08:23.843038+0000) 2024-03-04T21:08:24.839 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:08:24.842+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:08:24.843329+0000) 2024-03-04T21:08:25.837 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:08:25.842+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:08:25.843628+0000) 2024-03-04T21:08:26.838 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:08:26.843+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:08:26.843926+0000) 2024-03-04T21:08:27.839 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:08:27.843+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:08:27.844173+0000) 2024-03-04T21:08:28.839 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:08:28.843+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:08:28.844426+0000) 2024-03-04T21:08:29.839 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:08:29.843+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:08:29.844701+0000) 2024-03-04T21:08:30.089 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T21:08:30.101 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T21:08:30.839 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:08:30.844+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:08:30.844985+0000) 2024-03-04T21:08:31.839 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:08:31.844+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:08:31.845282+0000) 2024-03-04T21:08:32.839 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:08:32.844+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:08:32.845563+0000) 2024-03-04T21:08:33.839 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:08:33.845+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:08:33.845900+0000) 2024-03-04T21:08:34.840 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:08:34.845+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:08:34.846157+0000) 2024-03-04T21:08:35.840 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:08:35.845+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:08:35.846348+0000) 2024-03-04T21:08:36.840 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:08:36.845+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:08:36.846586+0000) 2024-03-04T21:08:37.840 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:08:37.846+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:08:37.846868+0000) 2024-03-04T21:08:38.841 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:08:38.846+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:08:38.847108+0000) 2024-03-04T21:08:39.841 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:08:39.846+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:08:39.847350+0000) 2024-03-04T21:08:40.841 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:08:40.846+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:08:40.847591+0000) 2024-03-04T21:08:41.842 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:08:41.847+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:08:41.847870+0000) 2024-03-04T21:08:42.842 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:08:42.847+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:08:42.848113+0000) 2024-03-04T21:08:43.842 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:08:43.847+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:08:43.848322+0000) 2024-03-04T21:08:44.842 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:08:44.847+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:08:44.848518+0000) 2024-03-04T21:08:45.843 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:08:45.848+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:08:45.848766+0000) 2024-03-04T21:08:46.843 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:08:46.848+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:08:46.849011+0000) 2024-03-04T21:08:47.843 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:08:47.848+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:08:47.849248+0000) 2024-03-04T21:08:48.843 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:08:48.848+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:08:48.849492+0000) 2024-03-04T21:08:49.843 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:08:49.849+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:08:49.849728+0000) 2024-03-04T21:08:50.843 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:08:50.849+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:08:50.849970+0000) 2024-03-04T21:08:51.844 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:08:51.849+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:08:51.850162+0000) 2024-03-04T21:08:52.844 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:08:52.849+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:08:52.850402+0000) 2024-03-04T21:08:53.844 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:08:53.850+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:08:53.850640+0000) 2024-03-04T21:08:54.844 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:08:54.850+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:08:54.850906+0000) 2024-03-04T21:08:55.844 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:08:55.850+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:08:55.851150+0000) 2024-03-04T21:08:56.845 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:08:56.850+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:08:56.851393+0000) 2024-03-04T21:08:57.845 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:08:57.851+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:08:57.851660+0000) 2024-03-04T21:08:58.845 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:08:58.851+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:08:58.851932+0000) 2024-03-04T21:08:59.845 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:08:59.851+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:08:59.852126+0000) 2024-03-04T21:09:00.230 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T21:09:00.240 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T21:09:00.846 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:09:00.851+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:09:00.852369+0000) 2024-03-04T21:09:01.846 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:09:01.851+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:09:01.852629+0000) 2024-03-04T21:09:02.850 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:09:02.852+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:09:02.852938+0000) 2024-03-04T21:09:03.846 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:09:03.852+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:09:03.853179+0000) 2024-03-04T21:09:04.847 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:09:04.853+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:09:04.853420+0000) 2024-03-04T21:09:05.847 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:09:05.853+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:09:05.853662+0000) 2024-03-04T21:09:06.847 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:09:06.853+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:09:06.854007+0000) 2024-03-04T21:09:07.847 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:09:07.853+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:09:07.854215+0000) 2024-03-04T21:09:08.848 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:09:08.854+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:09:08.854458+0000) 2024-03-04T21:09:09.848 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:09:09.854+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:09:09.854702+0000) 2024-03-04T21:09:10.848 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:09:10.854+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:09:10.854952+0000) 2024-03-04T21:09:11.849 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:09:11.854+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:09:11.855205+0000) 2024-03-04T21:09:12.849 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:09:12.855+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:09:12.855541+0000) 2024-03-04T21:09:13.849 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:09:13.855+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:09:13.855782+0000) 2024-03-04T21:09:14.849 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:09:14.855+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:09:14.856025+0000) 2024-03-04T21:09:15.849 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:09:15.856+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:09:15.856279+0000) 2024-03-04T21:09:16.849 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:09:16.855+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:09:16.856522+0000) 2024-03-04T21:09:17.850 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:09:17.855+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:09:17.856770+0000) 2024-03-04T21:09:18.850 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:09:18.855+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:09:18.857012+0000) 2024-03-04T21:09:19.850 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:09:19.855+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:09:19.857245+0000) 2024-03-04T21:09:20.851 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:09:20.856+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:09:20.857478+0000) 2024-03-04T21:09:21.851 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:09:21.856+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:09:21.857720+0000) 2024-03-04T21:09:22.851 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:09:22.856+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:09:22.857969+0000) 2024-03-04T21:09:23.851 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:09:23.856+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:09:23.858167+0000) 2024-03-04T21:09:24.851 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:09:24.856+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:09:24.858439+0000) 2024-03-04T21:09:25.852 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:09:25.857+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:09:25.858692+0000) 2024-03-04T21:09:26.852 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:09:26.858+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:09:26.858974+0000) 2024-03-04T21:09:27.852 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:09:27.858+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:09:27.859253+0000) 2024-03-04T21:09:28.852 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:09:28.858+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:09:28.859439+0000) 2024-03-04T21:09:29.853 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:09:29.858+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:09:29.859677+0000) 2024-03-04T21:09:30.374 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T21:09:30.383 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T21:09:30.853 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:09:30.859+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:09:30.859944+0000) 2024-03-04T21:09:31.854 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:09:31.859+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:09:31.860182+0000) 2024-03-04T21:09:32.853 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:09:32.859+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:09:32.860430+0000) 2024-03-04T21:09:33.854 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:09:33.859+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:09:33.860674+0000) 2024-03-04T21:09:34.856 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:09:34.860+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:09:34.860933+0000) 2024-03-04T21:09:35.856 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:09:35.860+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:09:35.861185+0000) 2024-03-04T21:09:36.854 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:09:36.860+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:09:36.861431+0000) 2024-03-04T21:09:37.862 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:09:37.860+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:09:37.861674+0000) 2024-03-04T21:09:38.855 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:09:38.861+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:09:38.861953+0000) 2024-03-04T21:09:39.857 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:09:39.861+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:09:39.862156+0000) 2024-03-04T21:09:40.856 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:09:40.861+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:09:40.862357+0000) 2024-03-04T21:09:41.859 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:09:41.861+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:09:41.862605+0000) 2024-03-04T21:09:42.856 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:09:42.862+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:09:42.862920+0000) 2024-03-04T21:09:43.856 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:09:43.862+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:09:43.863215+0000) 2024-03-04T21:09:44.857 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:09:44.862+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:09:44.863460+0000) 2024-03-04T21:09:45.857 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:09:45.863+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:09:45.863706+0000) 2024-03-04T21:09:46.863 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:09:46.863+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:09:46.863949+0000) 2024-03-04T21:09:47.857 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:09:47.863+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:09:47.864197+0000) 2024-03-04T21:09:48.858 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:09:48.863+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:09:48.864438+0000) 2024-03-04T21:09:49.857 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:09:49.864+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:09:49.864679+0000) 2024-03-04T21:09:50.858 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:09:50.864+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:09:50.864958+0000) 2024-03-04T21:09:51.859 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:09:51.864+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:09:51.865205+0000) 2024-03-04T21:09:52.858 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:09:52.864+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:09:52.865454+0000) 2024-03-04T21:09:53.858 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:09:53.865+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:09:53.865696+0000) 2024-03-04T21:09:54.859 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:09:54.865+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:09:54.865993+0000) 2024-03-04T21:09:55.859 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:09:55.865+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:09:55.866252+0000) 2024-03-04T21:09:56.859 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:09:56.866+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:09:56.866518+0000) 2024-03-04T21:09:57.860 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:09:57.866+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:09:57.866859+0000) 2024-03-04T21:09:58.860 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:09:58.866+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:09:58.867131+0000) 2024-03-04T21:09:59.860 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:09:59.866+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:09:59.867391+0000) 2024-03-04T21:10:00.512 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T21:10:00.521 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T21:10:00.860 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:10:00.866+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:10:00.867594+0000) 2024-03-04T21:10:01.861 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:10:01.867+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:10:01.867891+0000) 2024-03-04T21:10:02.861 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:10:02.867+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:10:02.868142+0000) 2024-03-04T21:10:03.868 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:10:03.868+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:10:03.868391+0000) 2024-03-04T21:10:04.861 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:10:04.868+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:10:04.868634+0000) 2024-03-04T21:10:05.861 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:10:05.868+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:10:05.868932+0000) 2024-03-04T21:10:06.862 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:10:06.868+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:10:06.869184+0000) 2024-03-04T21:10:07.862 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:10:07.869+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:10:07.869439+0000) 2024-03-04T21:10:08.862 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:10:08.869+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:10:08.869691+0000) 2024-03-04T21:10:09.862 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:10:09.869+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:10:09.869946+0000) 2024-03-04T21:10:10.863 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:10:10.869+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:10:10.870198+0000) 2024-03-04T21:10:11.864 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:10:11.869+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:10:11.870404+0000) 2024-03-04T21:10:12.863 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:10:12.870+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:10:12.870665+0000) 2024-03-04T21:10:13.864 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:10:13.870+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:10:13.870962+0000) 2024-03-04T21:10:14.865 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:10:14.870+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:10:14.871194+0000) 2024-03-04T21:10:15.864 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:10:15.870+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:10:15.871393+0000) 2024-03-04T21:10:16.864 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:10:16.870+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:10:16.871636+0000) 2024-03-04T21:10:17.864 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:10:17.871+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:10:17.871962+0000) 2024-03-04T21:10:18.865 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:10:18.871+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:10:18.872175+0000) 2024-03-04T21:10:19.865 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:10:19.870+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:10:19.872371+0000) 2024-03-04T21:10:20.866 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:10:20.871+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:10:20.872622+0000) 2024-03-04T21:10:21.866 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:10:21.871+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:10:21.872917+0000) 2024-03-04T21:10:22.865 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:10:22.871+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:10:22.873138+0000) 2024-03-04T21:10:23.866 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:10:23.872+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:10:23.873389+0000) 2024-03-04T21:10:24.866 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:10:24.872+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:10:24.873637+0000) 2024-03-04T21:10:25.867 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:10:25.872+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:10:25.873935+0000) 2024-03-04T21:10:26.866 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:10:26.872+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:10:26.874176+0000) 2024-03-04T21:10:27.867 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:10:27.872+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:10:27.874426+0000) 2024-03-04T21:10:28.867 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:10:28.873+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:10:28.874629+0000) 2024-03-04T21:10:29.867 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:10:29.873+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:10:29.874934+0000) 2024-03-04T21:10:30.651 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T21:10:30.661 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T21:10:30.867 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:10:30.874+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:10:30.875196+0000) 2024-03-04T21:10:31.868 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:10:31.874+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:10:31.875455+0000) 2024-03-04T21:10:32.868 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:10:32.874+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:10:32.875722+0000) 2024-03-04T21:10:33.868 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:10:33.875+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:10:33.875982+0000) 2024-03-04T21:10:34.868 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:10:34.875+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:10:34.876239+0000) 2024-03-04T21:10:35.869 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:10:35.875+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:10:35.876510+0000) 2024-03-04T21:10:36.869 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:10:36.875+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:10:36.876767+0000) 2024-03-04T21:10:37.869 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:10:37.876+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:10:37.877055+0000) 2024-03-04T21:10:38.869 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:10:38.876+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:10:38.877306+0000) 2024-03-04T21:10:39.870 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:10:39.876+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:10:39.877573+0000) 2024-03-04T21:10:40.873 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:10:40.877+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:10:40.877861+0000) 2024-03-04T21:10:41.870 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:10:41.877+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:10:41.878123+0000) 2024-03-04T21:10:42.870 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:10:42.877+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:10:42.878434+0000) 2024-03-04T21:10:43.871 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:10:43.877+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:10:43.878690+0000) 2024-03-04T21:10:44.871 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:10:44.878+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:10:44.878947+0000) 2024-03-04T21:10:45.871 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:10:45.878+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:10:45.879204+0000) 2024-03-04T21:10:46.872 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:10:46.878+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:10:46.879464+0000) 2024-03-04T21:10:47.875 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:10:47.879+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:10:47.879726+0000) 2024-03-04T21:10:48.875 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:10:48.879+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:10:48.879985+0000) 2024-03-04T21:10:49.872 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:10:49.879+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:10:49.880242+0000) 2024-03-04T21:10:50.872 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:10:50.879+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:10:50.880507+0000) 2024-03-04T21:10:51.873 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:10:51.879+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:10:51.880779+0000) 2024-03-04T21:10:52.873 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:10:52.880+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:10:52.881077+0000) 2024-03-04T21:10:53.873 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:10:53.880+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:10:53.881337+0000) 2024-03-04T21:10:54.874 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:10:54.880+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:10:54.881586+0000) 2024-03-04T21:10:55.874 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:10:55.881+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:10:55.881926+0000) 2024-03-04T21:10:56.874 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:10:56.881+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:10:56.882173+0000) 2024-03-04T21:10:57.874 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:10:57.881+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:10:57.882413+0000) 2024-03-04T21:10:58.875 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:10:58.881+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:10:58.882651+0000) 2024-03-04T21:10:59.875 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:10:59.882+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:10:59.882943+0000) 2024-03-04T21:11:00.789 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T21:11:00.799 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T21:11:00.875 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:11:00.882+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:11:00.883159+0000) 2024-03-04T21:11:01.875 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:11:01.882+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:11:01.883385+0000) 2024-03-04T21:11:02.876 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:11:02.883+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:11:02.883625+0000) 2024-03-04T21:11:03.876 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:11:03.883+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:11:03.883957+0000) 2024-03-04T21:11:04.876 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:11:04.883+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:11:04.884226+0000) 2024-03-04T21:11:05.876 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:11:05.884+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:11:05.884471+0000) 2024-03-04T21:11:06.877 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:11:06.884+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:11:06.884722+0000) 2024-03-04T21:11:07.877 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:11:07.884+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:11:07.884982+0000) 2024-03-04T21:11:08.877 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:11:08.884+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:11:08.885231+0000) 2024-03-04T21:11:09.877 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:11:09.884+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:11:09.885505+0000) 2024-03-04T21:11:10.883 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:11:10.884+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:11:10.885762+0000) 2024-03-04T21:11:11.878 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:11:11.885+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:11:11.886033+0000) 2024-03-04T21:11:12.878 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:11:12.886+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:11:12.886304+0000) 2024-03-04T21:11:13.878 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:11:13.886+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:11:13.886557+0000) 2024-03-04T21:11:14.878 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:11:14.886+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:11:14.886813+0000) 2024-03-04T21:11:15.879 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:11:15.886+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:11:15.887134+0000) 2024-03-04T21:11:16.879 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:11:16.887+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:11:16.887388+0000) 2024-03-04T21:11:17.879 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:11:17.887+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:11:17.887645+0000) 2024-03-04T21:11:18.880 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:11:18.887+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:11:18.887977+0000) 2024-03-04T21:11:19.880 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:11:19.888+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:11:19.888266+0000) 2024-03-04T21:11:20.880 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:11:20.888+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:11:20.888517+0000) 2024-03-04T21:11:21.880 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:11:21.888+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:11:21.888760+0000) 2024-03-04T21:11:22.881 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:11:22.887+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:11:22.888997+0000) 2024-03-04T21:11:23.881 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:11:23.887+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:11:23.889192+0000) 2024-03-04T21:11:24.881 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:11:24.888+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:11:24.889445+0000) 2024-03-04T21:11:25.881 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:11:25.888+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:11:25.889712+0000) 2024-03-04T21:11:26.882 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:11:26.888+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:11:26.889944+0000) 2024-03-04T21:11:27.882 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:11:27.889+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:11:27.890189+0000) 2024-03-04T21:11:28.882 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:11:28.889+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:11:28.890422+0000) 2024-03-04T21:11:29.882 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:11:29.889+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:11:29.890656+0000) 2024-03-04T21:11:30.883 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:11:30.889+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:11:30.890929+0000) 2024-03-04T21:11:30.929 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T21:11:30.938 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T21:11:31.883 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:11:31.890+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:11:31.891171+0000) 2024-03-04T21:11:32.883 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:11:32.890+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:11:32.891410+0000) 2024-03-04T21:11:33.883 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:11:33.890+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:11:33.891648+0000) 2024-03-04T21:11:34.883 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:11:34.891+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:11:34.891927+0000) 2024-03-04T21:11:35.884 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:11:35.891+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:11:35.892206+0000) 2024-03-04T21:11:36.884 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:11:36.891+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:11:36.892441+0000) 2024-03-04T21:11:37.884 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:11:37.891+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:11:37.892677+0000) 2024-03-04T21:11:38.884 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:11:38.892+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:11:38.892946+0000) 2024-03-04T21:11:39.885 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:11:39.892+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:11:39.893183+0000) 2024-03-04T21:11:40.888 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:11:40.892+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:11:40.893421+0000) 2024-03-04T21:11:41.885 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:11:41.892+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:11:41.893720+0000) 2024-03-04T21:11:42.888 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:11:42.893+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:11:42.893953+0000) 2024-03-04T21:11:43.886 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:11:43.893+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:11:43.894192+0000) 2024-03-04T21:11:44.886 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:11:44.893+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:11:44.894427+0000) 2024-03-04T21:11:45.886 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:11:45.893+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:11:45.894670+0000) 2024-03-04T21:11:46.886 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:11:46.894+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:11:46.894956+0000) 2024-03-04T21:11:47.887 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:11:47.894+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:11:47.895158+0000) 2024-03-04T21:11:48.887 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:11:48.894+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:11:48.895393+0000) 2024-03-04T21:11:49.887 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:11:49.894+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:11:49.895632+0000) 2024-03-04T21:11:50.888 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:11:50.895+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:11:50.895902+0000) 2024-03-04T21:11:51.887 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:11:51.895+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:11:51.896141+0000) 2024-03-04T21:11:52.888 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:11:52.895+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:11:52.896332+0000) 2024-03-04T21:11:53.888 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:11:53.895+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:11:53.896571+0000) 2024-03-04T21:11:54.888 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:11:54.896+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:11:54.896861+0000) 2024-03-04T21:11:55.888 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:11:55.896+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:11:55.897102+0000) 2024-03-04T21:11:56.889 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:11:56.896+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:11:56.897334+0000) 2024-03-04T21:11:57.889 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:11:57.897+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:11:57.897574+0000) 2024-03-04T21:11:58.889 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:11:58.897+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:11:58.897888+0000) 2024-03-04T21:11:59.890 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:11:59.897+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:11:59.898139+0000) 2024-03-04T21:12:00.889 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:12:00.897+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:12:00.898365+0000) 2024-03-04T21:12:01.157 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T21:12:01.166 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T21:12:01.890 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:12:01.898+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:12:01.898607+0000) 2024-03-04T21:12:02.890 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:12:02.898+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:12:02.898923+0000) 2024-03-04T21:12:03.890 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:12:03.898+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:12:03.899165+0000) 2024-03-04T21:12:04.891 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:12:04.898+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:12:04.899402+0000) 2024-03-04T21:12:05.891 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:12:05.899+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:12:05.899642+0000) 2024-03-04T21:12:06.891 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:12:06.899+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:12:06.899941+0000) 2024-03-04T21:12:07.891 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:12:07.899+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:12:07.900121+0000) 2024-03-04T21:12:08.892 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:12:08.899+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:12:08.900354+0000) 2024-03-04T21:12:09.892 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:12:09.900+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:12:09.900595+0000) 2024-03-04T21:12:10.892 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:12:10.900+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:12:10.900875+0000) 2024-03-04T21:12:11.895 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:12:11.900+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:12:11.901127+0000) 2024-03-04T21:12:12.892 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:12:12.901+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:12:12.901362+0000) 2024-03-04T21:12:13.893 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:12:13.901+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:12:13.901599+0000) 2024-03-04T21:12:14.893 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:12:14.901+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:12:14.901877+0000) 2024-03-04T21:12:15.893 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:12:15.901+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:12:15.902123+0000) 2024-03-04T21:12:16.894 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:12:16.902+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:12:16.902361+0000) 2024-03-04T21:12:17.894 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:12:17.902+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:12:17.902602+0000) 2024-03-04T21:12:18.894 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:12:18.902+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:12:18.902905+0000) 2024-03-04T21:12:19.894 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:12:19.902+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:12:19.903145+0000) 2024-03-04T21:12:20.894 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:12:20.903+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:12:20.903380+0000) 2024-03-04T21:12:21.895 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:12:21.902+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:12:21.903615+0000) 2024-03-04T21:12:22.895 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:12:22.903+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:12:22.903928+0000) 2024-03-04T21:12:23.897 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:12:23.904+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:12:23.904212+0000) 2024-03-04T21:12:24.896 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:12:24.904+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:12:24.904525+0000) 2024-03-04T21:12:25.896 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:12:25.904+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:12:25.904898+0000) 2024-03-04T21:12:26.896 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:12:26.904+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:12:26.905196+0000) 2024-03-04T21:12:27.896 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:12:27.904+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:12:27.905478+0000) 2024-03-04T21:12:28.897 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:12:28.904+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:12:28.905764+0000) 2024-03-04T21:12:29.897 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:12:29.904+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:12:29.906147+0000) 2024-03-04T21:12:30.897 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:12:30.905+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:12:30.906437+0000) 2024-03-04T21:12:31.357 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T21:12:31.365 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T21:12:31.898 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:12:31.905+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:12:31.906729+0000) 2024-03-04T21:12:32.898 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:12:32.905+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:12:32.906994+0000) 2024-03-04T21:12:33.898 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:12:33.906+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:12:33.907239+0000) 2024-03-04T21:12:34.898 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:12:34.905+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:12:34.907483+0000) 2024-03-04T21:12:35.899 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:12:35.906+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:12:35.907765+0000) 2024-03-04T21:12:36.899 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:12:36.906+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:12:36.907970+0000) 2024-03-04T21:12:37.899 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:12:37.907+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:12:37.908253+0000) 2024-03-04T21:12:38.899 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:12:38.907+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:12:38.908543+0000) 2024-03-04T21:12:39.900 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:12:39.908+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:12:39.908867+0000) 2024-03-04T21:12:40.900 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:12:40.908+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:12:40.909169+0000) 2024-03-04T21:12:41.900 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:12:41.908+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:12:41.909461+0000) 2024-03-04T21:12:42.901 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:12:42.908+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:12:42.909747+0000) 2024-03-04T21:12:43.901 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:12:43.909+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:12:43.910038+0000) 2024-03-04T21:12:44.901 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:12:44.909+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:12:44.910334+0000) 2024-03-04T21:12:45.902 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:12:45.909+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:12:45.910636+0000) 2024-03-04T21:12:46.902 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:12:46.910+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:12:46.910955+0000) 2024-03-04T21:12:47.902 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:12:47.910+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:12:47.911238+0000) 2024-03-04T21:12:48.903 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-03-04T21:12:48.910+0000 7f6abfe00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-04T20:12:48.911524+0000) 2024-03-04T21:12:49.357 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2024-03-04T21:12:49.358 DEBUG:teuthology.task.console_log:Killing console logger for smithi006 2024-03-04T21:12:49.358 DEBUG:teuthology.task.console_log:Killing console logger for smithi146 2024-03-04T21:12:49.358 DEBUG:teuthology.exit:Finished running handlers